<?xml version='1.0' encoding='utf-8'?>
<gexf xmlns="http://www.gexf.net/1.2draft" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.gexf.net/1.2draft http://www.gexf.net/1.2draft/gexf.xsd" version="1.2">
  <meta lastmodifieddate="2025-02-28">
    <creator>NetworkX 3.3</creator>
  </meta>
  <graph defaultedgetype="directed" mode="static" name="">
    <attributes mode="static" class="edge">
      <attribute id="5" title="sentence" type="string" />
    </attributes>
    <attributes mode="static" class="node">
      <attribute id="0" title="title" type="string" />
      <attribute id="1" title="abstract" type="string" />
      <attribute id="2" title="introduction" type="string" />
      <attribute id="3" title="related" type="string" />
      <attribute id="4" title="concepts" type="string" />
    </attributes>
    <nodes>
      <node id="2406.16838" label="2406.16838">
        <attvalues>
          <attvalue for="0" value="From Decoding to Meta-Generation: Inference-time Algorithms for Large&#10;  Language Models" />
          <attvalue for="1" value="  One of the most striking findings in modern research on large language models&#10;(LLMs) is that scaling up compute during training leads to better results.&#10;However, less attention has been given to the benefits of scaling compute&#10;during inference. This survey focuses on these inference-time approaches. We&#10;explore three areas under a unified mathematical formalism: token-level&#10;generation algorithms, meta-generation algorithms, and efficient generation.&#10;Token-level generation algorithms, often called decoding algorithms, operate by&#10;sampling a single token at a time or constructing a token-level search space&#10;and then selecting an output. These methods typically assume access to a&#10;language model's logits, next-token distributions, or probability scores.&#10;Meta-generation algorithms work on partial or full sequences, incorporating&#10;domain knowledge, enabling backtracking, and integrating external information.&#10;Efficient generation methods aim to reduce token costs and improve the speed of&#10;generation. Our survey unifies perspectives from three research communities:&#10;traditional natural language processing, modern LLMs, and machine learning&#10;systems.&#10;" />
          <attvalue for="2" value="&#10;&#10;One of the most striking findings in modern research on large language models (LLMs) is that, given a model and dataset of sufficient scale, scaling up the compute used at training time leads to better final results \cite{kaplan2020scaling,hoffmann2022an}.&#10;However, &#10;there is another, lesser-mentioned scaling phenomenon, where adopting more sophisticated methods or scaling compute at inference time~\cite{jones2021scaling} can result in substantially better outputs from LLMs.&#10;This survey focuses on these approaches by exploring three connected themes: token-level generation algorithms, meta-generation algorithms, and efficient generation.&#10;&#10;Token-level generation algorithms, often called decoding algorithms, have a rich history in natural language processing, ranging from classical greedy decoding and beam search to modern sampling algorithms such as nucleus~\cite{Holtzman2020The} and $\eta$-sampling~\cite{hewitt-etal-2022-truncation}. &#10;These methods operate by sampling one token at a time or constructing a token-level search space. They assume varying levels of access to a language model's internals, such as logits, next-token distributions, or probability scores. &#10;&#10;Recently there has been growing interest in meta-generation algorithms---algorithms that operate on partial or full sequences, and treat the LLM as a black box that is called as part of a larger generation program (\ref{fig:roadmap};~\cite{khattab2022demonstrate,dohan2022language,schlag2023large}). &#10;For example, a meta-generation algorithm&#10;for solving a math problem might generate multiple solution paths, evaluate the solutions with a calculator,&#10;then select the most common answer.&#10;Meta-generators can &#10; increase the compute resources devoted to generation by making multiple model calls, augmenting the model with search algorithms~\cite{yao2023tree,madaan2023selfrefine}, or incorporating external data sources. &#10;Doing so has seen success in improving task performance (e.g., problem solving~\cite{lewkowycz2022solving}) and steering the output distribution (e.g., with human preferences~\cite{stiennon2020}), and may offer a way to overcome limitations &#10;of standard LLMs &#10;such as error accumulation~\cite{dziri2023faith} and computational capacity~\cite{merrill2024the}.&#10;Moreover, meta-generation research is widely accessible, as it often only requires black-box LLM access.&#10;&#10;Finally, generation needs to be fast and cost-effective.&#10;Fast generation becomes increasingly challenging as models grow in size, while cost becomes a critical factor in meta-generation algorithms that call models many times. &#10;On the other hand, meta-generation algorithms open new kinds of shared computation that can be leveraged for improved efficiency.&#10;As a result, there is growing interest in efficient generation algorithms that speed up generation and reduce token costs by drawing on ideas from machine learning systems and related areas.&#10;Efficient generation in turn expands the frontier of algorithms that are feasible to experiment with and develop, leading to a virtuous cycle of algorithmic development.&#10;&#10;Our survey provides a unified treatment of these three themes: token-level generation algorithms, meta-generation algorithms, and techniques for making generation fast and cost-effective.&#10;We integrate ideas from traditional natural language processing, modern LLMs, and machine learning systems, and present a mathematical formalism that includes both classical generation algorithms and modern meta-generators.&#10;This unified view is particularly important as the field expands. For example, practitioners working on novel meta-generation algorithms may benefit from learning about the historical context of generation algorithms or practical efficiency constraints, while researchers interested in efficiency may benefit from learning about major algorithmic patterns.&#10;More broadly, we aim to promote further research on inference-time approaches. &#10;&#10;Comparison to existing surveys. Several prior surveys have focused on training-time methods for better text generation \cite{ijcai2021p612, lu2018neuraltextgenerationpast}. \cite{wiher2022decodingstrategiesneuraltext} presents a detailed analysis of a smaller set of decoding strategies, while \cite{info12090355} spotlight token-level methods, with a particular focus on considerations for encoder-decoder models. In parallel, several surveys have addressed prompting and related methods \cite{liu2021pretrainpromptpredictsystematic,sahoo2024systematicsurveypromptengineering}, though these works do not address token-level methods.&#10;Recent surveys have also considered strategies for speeding up inference \cite{chittyvenkata2023surveytechniquesoptimizingtransformer,miao2023efficientgenerativelargelanguage,khoshnoodi2024comprehensivesurveyacceleratedgeneration, wang2024modelcompressionefficientinference}. However, these works focus primarily on token-level generation, not meta-generation; as a result, the discussion of inference-time compute-performance tradeoffs is limited. Our survey unifies and draws connections across these three areas.&#10;Finally, \cite{xiao2023surveynonautoregressivegenerationneural} focus on non-autoregressive generation, while our survey&#10;focuses on autoregressive generation.&#10;&#10;Roadmap.&#10;This paper provides a survey of algorithms for token-level generation, meta-generation, and efficient generation, summarized in \ref{fig:roadmap}. &#10;First, we consider why we use generation algorithms at all.&#10;Generally, a user's intent is to surface a high-quality output from the model, which we formalize and discuss in \S\ref{sec:prelims}. &#10;Readers who would like to review terminology or follow the mathematical formulation of the survey in depth should start in this section. &#10;Next, we discuss token-level generation algorithms in detail in \S\ref{sec:gen}. Most algorithms referred to as ``decoding algorithms'' in the literature are covered in this section. We discuss these methods' theoretical motivation, practical impact, commonalities, and provide a unified frame for discussion. These methods generally require some degree of access to the model's internals.&#10;&#10;A growing set of methods operate over partial or full sequences rather than individual tokens. These meta-generation algorithms have emerged from several communitites, including researchers interested in designing new decoding algorithms or prompting methods, as well as researchers interested in language model alignment and reasoning. Works from these communities often have different motivations and use different terminology. We present a unified picture in \S\ref{sec:metagen}, classifying them according to their programmatic structure (e.g., parallel generation, search, or refinement), and discussing their motivations.&#10;&#10;In addition to wanting a high-quality output, we often care about the efficiency of generation. &#10;We consider two definitions of efficient generation. &#10;In \S\ref{sec:analysis} we consider the token cost of generation algorithms, which is especially relevant for studying cost-performance tradeoffs as the amount of computation allocated to generation is scaled up, and for those using API-access models that charge by the token. &#10;In \S\ref{sec:speeding}, we discuss methods for speeding up generation primarily from a systems perspective, where access to the model weights is assumed and latency and throughput are the key considerations. In this section, we draw upon work primarily from the machine learning systems (MLSys) community. The sectionserves as both an introduction to this area for machine learning researchers whose work does not focus on systems, and a practical exploration of tools for speeding up generation. We include a review of libraries that implement the described techniques. &#10;&#10;We conclude the survey by discussing takeaways, broader directions, and future work in \S\ref{sec:discussion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Scaling, Machine Learning, Computer Science, Efficient Inference Methods, Linguistics, Generation Algorithms, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2001.08361" label="2001.08361">
        <attvalues>
          <attvalue for="0" value="Scaling Laws for Neural Language Models" />
          <attvalue for="1" value="  We study empirical scaling laws for language model performance on the&#10;cross-entropy loss. The loss scales as a power-law with model size, dataset&#10;size, and the amount of compute used for training, with some trends spanning&#10;more than seven orders of magnitude. Other architectural details such as&#10;network width or depth have minimal effects within a wide range. Simple&#10;equations govern the dependence of overfitting on model/dataset size and the&#10;dependence of training speed on model size. These relationships allow us to&#10;determine the optimal allocation of a fixed compute budget. Larger models are&#10;significantly more sample-efficient, such that optimally compute-efficient&#10;training involves training very large models on a relatively modest amount of&#10;data and stopping significantly before convergence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.09751" label="1904.09751">
        <attvalues>
          <attvalue for="0" value="The Curious Case of Neural Text Degeneration" />
          <attvalue for="1" value="  Despite considerable advancements with deep neural language models, the&#10;enigma of neural text degeneration persists when these models are tested as&#10;text generators. The counter-intuitive empirical observation is that even&#10;though the use of likelihood as training objective leads to high quality models&#10;for a broad range of language understanding tasks, using likelihood as a&#10;decoding objective leads to text that is bland and strangely repetitive.&#10;  In this paper, we reveal surprising distributional differences between human&#10;text and machine text. In addition, we find that decoding strategies alone can&#10;dramatically effect the quality of machine text, even when generated from&#10;exactly the same neural language model. Our findings motivate Nucleus Sampling,&#10;a simple but effective method to draw the best out of neural generation. By&#10;sampling text from the dynamic nucleus of the probability distribution, which&#10;allows for diversity while effectively truncating the less reliable tail of the&#10;distribution, the resulting text better demonstrates the quality of human text,&#10;yielding enhanced diversity without sacrificing fluency and coherence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.15191" label="2210.15191">
        <attvalues>
          <attvalue for="0" value="Truncation Sampling as Language Model Desmoothing" />
          <attvalue for="1" value="  Long samples of text from neural language models can be of poor quality.&#10;Truncation sampling algorithms--like top-$p$ or top-$k$ -- address this by&#10;setting some words' probabilities to zero at each step. This work provides&#10;framing for the aim of truncation, and an improved algorithm for that aim. We&#10;propose thinking of a neural language model as a mixture of a true distribution&#10;and a smoothing distribution that avoids infinite perplexity. In this light,&#10;truncation algorithms aim to perform desmoothing, estimating a subset of the&#10;support of the true distribution. Finding a good subset is crucial: we show&#10;that top-$p$ unnecessarily truncates high-probability words, for example&#10;causing it to truncate all words but Trump for a document that starts with&#10;Donald. We introduce $\eta$-sampling, which truncates words below an&#10;entropy-dependent probability threshold. Compared to previous algorithms,&#10;$\eta$-sampling generates more plausible long English documents according to&#10;humans, is better at breaking out of repetition, and behaves more reasonably on&#10;a battery of test distributions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.14024" label="2212.14024">
        <attvalues>
          <attvalue for="0" value="Demonstrate-Search-Predict: Composing retrieval and language models for&#10;  knowledge-intensive NLP" />
          <attvalue for="1" value="  Retrieval-augmented in-context learning has emerged as a powerful approach&#10;for addressing knowledge-intensive tasks using frozen language models (LM) and&#10;retrieval models (RM). Existing work has combined these in simple&#10;&quot;retrieve-then-read&quot; pipelines in which the RM retrieves passages that are&#10;inserted into the LM prompt. To begin to fully realize the potential of frozen&#10;LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that&#10;relies on passing natural language texts in sophisticated pipelines between an&#10;LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware&#10;demonstrations, search for relevant passages, and generate grounded&#10;predictions, systematically breaking down problems into small transformations&#10;that the LM and RM can handle more reliably. We have written novel DSP programs&#10;for answering questions in open-domain, multi-hop, and conversational settings,&#10;establishing in early evaluations new state-of-the-art in-context learning&#10;results and delivering 37-120%, 8-39%, and 80-290% relative gains against the&#10;vanilla LM (GPT-3.5), a standard retrieve-then-read pipeline, and a&#10;contemporaneous self-ask pipeline, respectively. We release DSP at&#10;https://github.com/stanfordnlp/dsp&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.10342" label="2207.10342">
        <attvalues>
          <attvalue for="0" value="Language Model Cascades" />
          <attvalue for="1" value="  Prompted models have demonstrated impressive few-shot learning abilities.&#10;Repeated interactions at test-time with a single model, or the composition of&#10;multiple models together, further expands capabilities. These compositions are&#10;probabilistic models, and may be expressed in the language of graphical models&#10;with random variables whose values are complex data types such as strings.&#10;Cases with control flow and dynamic structure require techniques from&#10;probabilistic programming, which allow implementing disparate model structures&#10;and inference strategies in a unified language. We formalize several existing&#10;techniques from this perspective, including scratchpads / chain of thought,&#10;verifiers, STaR, selection-inference, and tool use. We refer to the resulting&#10;programs as language model cascades.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;There is a rich prior literature on &#10;probabilistic programming languages (PPLs),&#10;which extend probabilistic graphical models to&#10;support more complex joint distributions whose size and ``shape''&#10;can itself be stochastic (e.g., a graph&#10;unrolled for a random number of iterations,&#10;until a data-dependent stopping criterion is met).&#10;PPLs extend traditional programming languages with the ability to {sample} from distributions and {observe} values of variables based on data (i.e. condition the model).&#10;The semantics of sample and observe vary depending on the inference algorithm.&#10;For more details, see \cite{intro_ppl}.&#10;&#10;Recently there has been an explosion of interest in large language models, such as &#10;GPT-3 \cite{gpt3} and PaLM \cite{palm}.&#10;These can be used for tasks such as ``zero-shot&quot;&#10;question-answering. In this setting, we &#10;provide the question $Q$ as a prompt to the LM,&#10;and then sample answers from the model, &#10;which we denote by $p(A|Q,\theta)$,&#10;where $\theta$ are the pre-trained model parameters.&#10;Alternatively, we can compute the MAP answer,&#10;$\hat{A} = \argmax_A p(A|Q,\theta)$.&#10;&#10;To ensure the model ``does the right thing'',&#10;we can provide a small training set of question-answer pairs,&#10;$D = \{ (Q^m,A^m): m=1:M\}$ pairs.&#10;This can be provided as extra context to the model,&#10;provided in the text prompt, followed by sampling&#10;from $p(A|Q,D,\theta)$.&#10;We refer to this as ``few-shot prompting''.&#10;We can also fine-tune the model parameters on $D$ to&#10;get $\theta'$, and then sample&#10;from $p(A|Q,\theta')$.&#10;&#10;We can improve performance by introducing an additional auxiliary ``thought'' variable,&#10;and then extend the model to have the form $p(A,T|Q) = p(A|T,Q)p(T|Q)$, where each conditional is computed using an LM which includes its conditioning variables as a part of its input.&#10;Work on scratchpads \cite{scratchpads} and chain of thought \cite{chainofthought} illustrate this, and finetune or prompt the LM to produce this auxiliary thought before answering.&#10;&#10;We typically condition this on a small set&#10;$D_S$ of $(A^m,T^m,Q^m)$ triples,&#10;and optionally a larger set $D_L$ of $(A^m, Q^m)$ pairs.&#10;We then compute a distribution over answers to a test question&#10;using&#10;\begin{align}&#10;\hat{p}(A|Q) = \sum_T &#10;\hat{p}(A|Q, T) \hat{p}(T|Q)&#10;\end{align}&#10;where $\hat{p}(\cdot) = p(\cdot|D_L,D_S,\theta)$&#10;is the prior predictive distribution. (Scratchpad creates its prior predictive by fine-tuning, while Chain of Thought adds $D_S$ to the LM prompt.)&#10;&#10;In practice, we cannot sum over all possible strings $T$&#10;in \ref{eqn:probQA}.&#10;The most common approach is to compute the MAP estimate&#10;$\hat{T} = \argmax \hat{p}(T)$ using beam search,&#10;and then to approximate the sum over $T$ with this single&#10;value.&#10;More recently, Self Consistency \cite{selfconsistency} &#10;proposed to sample multiple values for $T$&#10;using forward sampling of $(A,T)$ given $Q$,&#10;and then taking the answer $A$ that is most common&#10;in this set.&#10;&#10;PromptChainer \cite{promptchainer} proposes a visual interface for composing language models together, specifying control flow and prompting strategies for each node in a chain. Nodes may query language models or external systems. &#10;Socratic models \cite{socraticmodels} extends model chaining to the multimodal setting and demonstrates zero-shot abilities on tasks for which no single model exists.&#10;&#10;The Eliciting Latent Knowledge proposal \cite{ELK} suggests making latent variables explicit, modelled using a Bayesian network, to improve interpretability and safety for advanced AI systems.&#10;&#10;\cite{ortega2021shaking} explains a formalism for LM finetuning with causal graphical models in order to extend the predictive capabilities of AI agents towards more adaptive behaviour. They focus on analysing an auto-regressive action (random variable) prediction scheme in the interactive setting of RL where a model is simultaneously a generator and predictor of data.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.05364" label="2305.05364">
        <attvalues>
          <attvalue for="0" value="Large Language Model Programs" />
          <attvalue for="1" value="  In recent years, large pre-trained language models (LLMs) have demonstrated&#10;the ability to follow instructions and perform novel tasks from a few examples.&#10;The possibility to parameterise an LLM through such in-context examples widens&#10;their capability at a much lower cost than finetuning. We extend this line of&#10;reasoning and present a method which further expands the capabilities of an LLM&#10;by embedding it within an algorithm or program. To demonstrate the benefits of&#10;this approach, we present an illustrative example of evidence-supported&#10;question-answering. We obtain a 6.4\% improvement over the chain of thought&#10;baseline through a more algorithmic approach without any finetuning.&#10;Furthermore, we highlight recent work from this perspective and discuss the&#10;advantages and disadvantages in comparison to the standard approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10601" label="2305.10601">
        <attvalues>
          <attvalue for="0" value="Tree of Thoughts: Deliberate Problem Solving with Large Language Models" />
          <attvalue for="1" value="  Language models are increasingly being deployed for general problem solving&#10;across a wide range of tasks, but are still confined to token-level,&#10;left-to-right decision-making processes during inference. This means they can&#10;fall short in tasks that require exploration, strategic lookahead, or where&#10;initial decisions play a pivotal role. To surmount these challenges, we&#10;introduce a new framework for language model inference, Tree of Thoughts (ToT),&#10;which generalizes over the popular Chain of Thought approach to prompting&#10;language models, and enables exploration over coherent units of text (thoughts)&#10;that serve as intermediate steps toward problem solving. ToT allows LMs to&#10;perform deliberate decision making by considering multiple different reasoning&#10;paths and self-evaluating choices to decide the next course of action, as well&#10;as looking ahead or backtracking when necessary to make global choices. Our&#10;experiments show that ToT significantly enhances language models'&#10;problem-solving abilities on three novel tasks requiring non-trivial planning&#10;or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in&#10;Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of&#10;tasks, our method achieved a success rate of 74%. Code repo with all prompts:&#10;https://github.com/princeton-nlp/tree-of-thought-llm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.17651" label="2303.17651">
        <attvalues>
          <attvalue for="0" value="Self-Refine: Iterative Refinement with Self-Feedback" />
          <attvalue for="1" value="  Like humans, large language models (LLMs) do not always generate the best&#10;output on their first try. Motivated by how humans refine their written text,&#10;we introduce Self-Refine, an approach for improving initial outputs from LLMs&#10;through iterative feedback and refinement. The main idea is to generate an&#10;initial output using an LLMs; then, the same LLMs provides feedback for its&#10;output and uses it to refine itself, iteratively. Self-Refine does not require&#10;any supervised training data, additional training, or reinforcement learning,&#10;and instead uses a single LLM as the generator, refiner, and feedback provider.&#10;We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response&#10;generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT,&#10;and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine&#10;are preferred by humans and automatic metrics over those generated with the&#10;same LLM using conventional one-step generation, improving by ~20% absolute on&#10;average in task performance. Our work demonstrates that even state-of-the-art&#10;LLMs like GPT-4 can be further improved at test time using our simple,&#10;standalone approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.01325" label="2009.01325">
        <attvalues>
          <attvalue for="0" value="Learning to summarize from human feedback" />
          <attvalue for="1" value="  As language models become more powerful, training and evaluation are&#10;increasingly bottlenecked by the data and metrics used for a particular task.&#10;For example, summarization models are often trained to predict human reference&#10;summaries and evaluated using ROUGE, but both of these metrics are rough&#10;proxies for what we really care about -- summary quality. In this work, we show&#10;that it is possible to significantly improve summary quality by training a&#10;model to optimize for human preferences. We collect a large, high-quality&#10;dataset of human comparisons between summaries, train a model to predict the&#10;human-preferred summary, and use that model as a reward function to fine-tune a&#10;summarization policy using reinforcement learning. We apply our method to a&#10;version of the TL;DR dataset of Reddit posts and find that our models&#10;significantly outperform both human reference summaries and much larger models&#10;fine-tuned with supervised learning alone. Our models also transfer to CNN/DM&#10;news articles, producing summaries nearly as good as the human reference&#10;without any news-specific fine-tuning. We conduct extensive analyses to&#10;understand our human feedback dataset and fine-tuned models We establish that&#10;our reward model generalizes to new datasets, and that optimizing our reward&#10;model results in better summaries than optimizing ROUGE according to humans. We&#10;hope the evidence from our paper motivates machine learning researchers to pay&#10;closer attention to how their training loss affects the model behavior they&#10;actually want.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18654" label="2305.18654">
        <attvalues>
          <attvalue for="0" value="Faith and Fate: Limits of Transformers on Compositionality" />
          <attvalue for="1" value="  Transformer large language models (LLMs) have sparked admiration for their&#10;exceptional performance on tasks that demand intricate multi-step reasoning.&#10;Yet, these models simultaneously show failures on surprisingly trivial&#10;problems. This begs the question: Are these errors incidental, or do they&#10;signal more substantial limitations? In an attempt to demystify transformer&#10;LLMs, we investigate the limits of these models across three representative&#10;compositional tasks -- multi-digit multiplication, logic grid puzzles, and a&#10;classic dynamic programming problem. These tasks require breaking problems down&#10;into sub-steps and synthesizing these steps into a precise answer. We formulate&#10;compositional tasks as computation graphs to systematically quantify the level&#10;of complexity, and break down reasoning steps into intermediate sub-procedures.&#10;Our empirical findings suggest that transformer LLMs solve compositional tasks&#10;by reducing multi-step compositional reasoning into linearized subgraph&#10;matching, without necessarily developing systematic problem-solving skills. To&#10;round off our empirical study, we provide theoretical arguments on abstract&#10;multi-step reasoning problems that highlight how autoregressive generations'&#10;performance can rapidly decay with\,increased\,task\,complexity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.07923" label="2310.07923">
        <attvalues>
          <attvalue for="0" value="The Expressive Power of Transformers with Chain of Thought" />
          <attvalue for="1" value="  Recent theoretical work has identified surprisingly simple reasoning&#10;problems, such as checking if two nodes in a graph are connected or simulating&#10;finite-state machines, that are provably unsolvable by standard transformers&#10;that answer immediately after reading their input. However, in practice,&#10;transformers' reasoning can be improved by allowing them to use a &quot;chain of&#10;thought&quot; or &quot;scratchpad&quot;, i.e., generate and condition on a sequence of&#10;intermediate tokens before answering. Motivated by this, we ask: Does such&#10;intermediate generation fundamentally extend the computational power of a&#10;decoder-only transformer? We show that the answer is yes, but the amount of&#10;increase depends crucially on the amount of intermediate generation. For&#10;instance, we find that transformer decoders with a logarithmic number of&#10;decoding steps (w.r.t. the input length) push the limits of standard&#10;transformers only slightly, while a linear number of decoding steps, assuming&#10;projected pre-norm (a slight generalization of standard pre-norm), adds a clear&#10;new ability (under standard complexity conjectures): recognizing all regular&#10;languages. Our results also imply that linear steps keep transformer decoders&#10;within context-sensitive languages, and polynomial steps with generalized&#10;pre-norm make them recognize exactly the class of polynomial-time solvable&#10;problems -- the first exact characterization of a type of transformers in terms&#10;of standard complexity classes. Together, this provides a nuanced framework for&#10;understanding how the length of a transformer's chain of thought or scratchpad&#10;impacts its reasoning power.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15721" label="2203.15721">
        <attvalues>
          <attvalue for="0" value="On Decoding Strategies for Neural Text Generators" />
          <attvalue for="1" value="  When generating text from probabilistic models, the chosen decoding strategy&#10;has a profound effect on the resulting text. Yet the properties elicited by&#10;various decoding strategies do not always transfer across natural language&#10;generation tasks. For example, while mode-seeking methods like beam search&#10;perform remarkably well for machine translation, they have been observed to&#10;lead to incoherent and repetitive text in story generation. Despite such&#10;observations, the effectiveness of decoding strategies is often assessed with&#10;respect to only a single task. This work -- in contrast -- provides a&#10;comprehensive analysis of the interaction between language generation tasks and&#10;decoding strategies. Specifically, we measure changes in attributes of&#10;generated text as a function of both decoding strategy and task using human and&#10;automatic evaluation. Our results reveal both previously-observed and&#10;surprising findings. For example, the nature of the diversity-quality trade-off&#10;in language generation is very task-specific; the length bias often attributed&#10;to beam search is not constant across tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.13586" label="2107.13586">
        <attvalues>
          <attvalue for="0" value="Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods&#10;  in Natural Language Processing" />
          <attvalue for="1" value="  This paper surveys and organizes research works in a new paradigm in natural&#10;language processing, which we dub &quot;prompt-based learning&quot;. Unlike traditional&#10;supervised learning, which trains a model to take in an input x and predict an&#10;output y as P(y|x), prompt-based learning is based on language models that&#10;model the probability of text directly. To use these models to perform&#10;prediction tasks, the original input x is modified using a template into a&#10;textual string prompt x' that has some unfilled slots, and then the language&#10;model is used to probabilistically fill the unfilled information to obtain a&#10;final string x, from which the final output y can be derived. This framework is&#10;powerful and attractive for a number of reasons: it allows the language model&#10;to be pre-trained on massive amounts of raw text, and by defining a new&#10;prompting function the model is able to perform few-shot or even zero-shot&#10;learning, adapting to new scenarios with few or no labeled data. In this paper&#10;we introduce the basics of this promising paradigm, describe a unified set of&#10;mathematical notations that can cover a wide variety of existing work, and&#10;organize existing work along several dimensions, e.g.the choice of pre-trained&#10;models, prompts, and tuning strategies. To make the field more accessible to&#10;interested beginners, we not only make a systematic review of existing works&#10;and a highly structured typology of prompt-based concepts, but also release&#10;other resources, e.g., a website http://pretrain.nlpedia.ai/ including&#10;constantly-updated survey, and paperlist.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.07927" label="2402.07927">
        <attvalues>
          <attvalue for="0" value="A Systematic Survey of Prompt Engineering in Large Language Models:&#10;  Techniques and Applications" />
          <attvalue for="1" value="  Prompt engineering has emerged as an indispensable technique for extending&#10;the capabilities of large language models (LLMs) and vision-language models&#10;(VLMs). This approach leverages task-specific instructions, known as prompts,&#10;to enhance model efficacy without modifying the core model parameters. Rather&#10;than updating the model parameters, prompts allow seamless integration of&#10;pre-trained models into downstream tasks by eliciting desired model behaviors&#10;solely based on the given prompt. Prompts can be natural language instructions&#10;that provide context to guide the model or learned vector representations that&#10;activate relevant knowledge. This burgeoning field has enabled success across&#10;various applications, from question-answering to commonsense reasoning.&#10;However, there remains a lack of systematic organization and understanding of&#10;the diverse prompt engineering methods and techniques. This survey paper&#10;addresses the gap by providing a structured overview of recent advancements in&#10;prompt engineering, categorized by application area. For each prompting&#10;approach, we provide a summary detailing the prompting methodology, its&#10;applications, the models involved, and the datasets utilized. We also delve&#10;into the strengths and limitations of each approach and include a taxonomy&#10;diagram and table summarizing datasets, models, and critical points of each&#10;prompting technique. This systematic analysis enables a better understanding of&#10;this rapidly developing field and facilitates future research by illuminating&#10;open challenges and opportunities for prompt engineering.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.07982" label="2307.07982">
        <attvalues>
          <attvalue for="0" value="A Survey of Techniques for Optimizing Transformer Inference" />
          <attvalue for="1" value="  Recent years have seen a phenomenal rise in performance and applications of&#10;transformer neural networks. The family of transformer networks, including&#10;Bidirectional Encoder Representations from Transformer (BERT), Generative&#10;Pretrained Transformer (GPT) and Vision Transformer (ViT), have shown their&#10;effectiveness across Natural Language Processing (NLP) and Computer Vision (CV)&#10;domains. Transformer-based networks such as ChatGPT have impacted the lives of&#10;common men. However, the quest for high predictive performance has led to an&#10;exponential increase in transformers' memory and compute footprint. Researchers&#10;have proposed techniques to optimize transformer inference at all levels of&#10;abstraction. This paper presents a comprehensive survey of techniques for&#10;optimizing the inference phase of transformer networks. We survey techniques&#10;such as knowledge distillation, pruning, quantization, neural architecture&#10;search and lightweight network design at the algorithmic level. We further&#10;review hardware-level optimization techniques and the design of novel hardware&#10;accelerators for transformers. We summarize the quantitative results on the&#10;number of parameters/FLOPs and accuracy of several models/techniques to&#10;showcase the tradeoff exercised by them. We also outline future directions in&#10;this rapidly evolving field of research. We believe that this survey will&#10;educate both novice and seasoned researchers and also spark a plethora of&#10;research efforts in this field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.15234" label="2312.15234">
        <attvalues>
          <attvalue for="0" value="Towards Efficient Generative Large Language Model Serving: A Survey from&#10;  Algorithms to Systems" />
          <attvalue for="1" value="  In the rapidly evolving landscape of artificial intelligence (AI), generative&#10;large language models (LLMs) stand at the forefront, revolutionizing how we&#10;interact with our data. However, the computational intensity and memory&#10;consumption of deploying these models present substantial challenges in terms&#10;of serving efficiency, particularly in scenarios demanding low latency and high&#10;throughput. This survey addresses the imperative need for efficient LLM serving&#10;methodologies from a machine learning system (MLSys) research perspective,&#10;standing at the crux of advanced AI innovations and practical system&#10;optimizations. We provide in-depth analysis, covering a spectrum of solutions,&#10;ranging from cutting-edge algorithmic modifications to groundbreaking changes&#10;in system designs. The survey aims to provide a comprehensive understanding of&#10;the current state and future directions in efficient LLM serving, offering&#10;valuable insights for researchers and practitioners in overcoming the barriers&#10;of effective LLM deployment, thereby reshaping the future of AI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.13019" label="2405.13019">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey of Accelerated Generation Techniques in Large&#10;  Language Models" />
          <attvalue for="1" value="  Despite the crucial importance of accelerating text generation in large&#10;language models (LLMs) for efficiently producing content, the sequential nature&#10;of this process often leads to high inference latency, posing challenges for&#10;real-time applications. Various techniques have been proposed and developed to&#10;address these challenges and improve efficiency. This paper presents a&#10;comprehensive survey of accelerated generation techniques in autoregressive&#10;language models, aiming to understand the state-of-the-art methods and their&#10;applications. We categorize these techniques into several key areas:&#10;speculative decoding, early exiting mechanisms, and non-autoregressive methods.&#10;We discuss each category's underlying principles, advantages, limitations, and&#10;recent advancements. Through this survey, we aim to offer insights into the&#10;current landscape of techniques in LLMs and provide guidance for future&#10;research directions in this critical area of natural language processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.09269" label="2204.09269">
        <attvalues>
          <attvalue for="0" value="A Survey on Non-Autoregressive Generation for Neural Machine Translation&#10;  and Beyond" />
          <attvalue for="1" value="  Non-autoregressive (NAR) generation, which is first proposed in neural&#10;machine translation (NMT) to speed up inference, has attracted much attention&#10;in both machine learning and natural language processing communities. While NAR&#10;generation can significantly accelerate inference speed for machine&#10;translation, the speedup comes at the cost of sacrificed translation accuracy&#10;compared to its counterpart, autoregressive (AR) generation. In recent years,&#10;many new models and algorithms have been designed/proposed to bridge the&#10;accuracy gap between NAR generation and AR generation. In this paper, we&#10;conduct a systematic survey with comparisons and discussions of various&#10;non-autoregressive translation (NAT) models from different aspects.&#10;Specifically, we categorize the efforts of NAT into several groups, including&#10;data manipulation, modeling methods, training criterion, decoding algorithms,&#10;and the benefit from pre-trained models. Furthermore, we briefly review other&#10;applications of NAR models beyond machine translation, such as grammatical&#10;error correction, text summarization, text style transfer, dialogue, semantic&#10;parsing, automatic speech recognition, and so on. In addition, we also discuss&#10;potential directions for future exploration, including releasing the dependency&#10;of KD, reasonable training objectives, pre-training for NAR, and wider&#10;applications, etc. We hope this survey can help researchers capture the latest&#10;progress in NAR generation, inspire the design of advanced NAR models and&#10;algorithms, and enable industry practitioners to choose appropriate solutions&#10;for their applications. The web page of this survey is at&#10;\url{https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.03741" label="1706.03741">
        <attvalues>
          <attvalue for="0" value="Deep reinforcement learning from human preferences" />
          <attvalue for="1" value="  For sophisticated reinforcement learning (RL) systems to interact usefully&#10;with real-world environments, we need to communicate complex goals to these&#10;systems. In this work, we explore goals defined in terms of (non-expert) human&#10;preferences between pairs of trajectory segments. We show that this approach&#10;can effectively solve complex RL tasks without access to the reward function,&#10;including Atari games and simulated robot locomotion, while providing feedback&#10;on less than one percent of our agent's interactions with the environment. This&#10;reduces the cost of human oversight far enough that it can be practically&#10;applied to state-of-the-art RL systems. To demonstrate the flexibility of our&#10;approach, we show that we can successfully train complex novel behaviors with&#10;about an hour of human time. These behaviors and environments are considerably&#10;more complex than any that have been previously learned from human feedback.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.10020" label="2401.10020">
        <attvalues>
          <attvalue for="0" value="Self-Rewarding Language Models" />
          <attvalue for="1" value="  We posit that to achieve superhuman agents, future models require superhuman&#10;feedback in order to provide an adequate training signal. Current approaches&#10;commonly train reward models from human preferences, which may then be&#10;bottlenecked by human performance level, and secondly these separate frozen&#10;reward models cannot then learn to improve during LLM training. In this work,&#10;we study Self-Rewarding Language Models, where the language model itself is&#10;used via LLM-as-a-Judge prompting to provide its own rewards during training.&#10;We show that during Iterative DPO training that not only does instruction&#10;following ability improve, but also the ability to provide high-quality rewards&#10;to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a&#10;model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard,&#10;including Claude 2, Gemini Pro, and GPT-4 0613. While there is much left still&#10;to explore, this work opens the door to the possibility of models that can&#10;continually improve in both axes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.19159" label="2403.19159">
        <attvalues>
          <attvalue for="0" value="Disentangling Length from Quality in Direct Preference Optimization" />
          <attvalue for="1" value="  Reinforcement Learning from Human Feedback (RLHF) has been a crucial&#10;component in the recent success of Large Language Models. However, RLHF is know&#10;to exploit biases in human preferences, such as verbosity. A well-formatted and&#10;eloquent answer is often more highly rated by users, even when it is less&#10;helpful and objective. A number of approaches have been developed to control&#10;those biases in the classical RLHF literature, but the problem remains&#10;relatively under-explored for Direct Alignment Algorithms such as Direct&#10;Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a&#10;separate reward model or use reinforcement learning directly, so previous&#10;approaches developed to control verbosity cannot be directly applied to this&#10;setting. Our work makes several contributions. For the first time, we study the&#10;length problem in the DPO setting, showing significant exploitation in DPO and&#10;linking it to out-of-distribution bootstrapping. We then develop a principled&#10;but simple regularization strategy that prevents length exploitation, while&#10;still maintaining improvements in model quality. We demonstrate these effects&#10;across datasets on summarization and dialogue, where we achieve up to 20\%&#10;improvement in win rates when controlling for length, despite the GPT4 judge's&#10;well-known verbosity bias.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.02243" label="1906.02243">
        <attvalues>
          <attvalue for="0" value="Energy and Policy Considerations for Deep Learning in NLP" />
          <attvalue for="1" value="  Recent progress in hardware and methodology for training neural networks has&#10;ushered in a new generation of large networks trained on abundant data. These&#10;models have obtained notable gains in accuracy across many NLP tasks. However,&#10;these accuracy improvements depend on the availability of exceptionally large&#10;computational resources that necessitate similarly substantial energy&#10;consumption. As a result these models are costly to train and develop, both&#10;financially, due to the cost of hardware and electricity or cloud compute time,&#10;and environmentally, due to the carbon footprint required to fuel modern tensor&#10;processing hardware. In this paper we bring this issue to the attention of NLP&#10;researchers by quantifying the approximate financial and environmental costs of&#10;training a variety of recently successful neural network models for NLP. Based&#10;on these findings, we propose actionable recommendations to reduce costs and&#10;improve equity in NLP research and practice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.12036" label="2407.12036">
        <attvalues>
          <attvalue for="0" value="Exploring Advanced Large Language Models with LLMsuite" />
          <attvalue for="1" value="  This tutorial explores the advancements and challenges in the development of&#10;Large Language Models (LLMs) such as ChatGPT and Gemini. It addresses inherent&#10;limitations like temporal knowledge cutoffs, mathematical inaccuracies, and the&#10;generation of incorrect information, proposing solutions like Retrieval&#10;Augmented Generation (RAG), Program-Aided Language Models (PAL), and frameworks&#10;such as ReAct and LangChain. The integration of these techniques enhances LLM&#10;performance and reliability, especially in multi-step reasoning and complex&#10;task execution. The paper also covers fine-tuning strategies, including&#10;instruction fine-tuning, parameter-efficient methods like LoRA, and&#10;Reinforcement Learning from Human Feedback (RLHF) as well as Reinforced&#10;Self-Training (ReST). Additionally, it provides a comprehensive survey of&#10;transformer architectures and training techniques for LLMs. The toolbox for&#10;implementing these techniques is publicly available at&#10;https://github.com/giorgioroffo/large_language_models_open_suite&#10;" />
          <attvalue for="2" value="&#10;In recent years, the field of natural language processing (NLP) has witnessed groundbreaking advancements with the development and deployment of Large Language Models (LLMs) such as ChatGPT and Gemini. These models, characterized by their ability to generate human-like text, have set new benchmarks in a variety of applications, from automated customer support to creative writing. Despite their impressive capabilities, LLMs are not without limitations. They often struggle with the temporal limitations of their knowledge base, complex mathematical computations, and a tendency to produce plausible but incorrect information—commonly referred to as &quot;hallucinations&quot;.&#10;&#10;Addressing these limitations has become a focal point of contemporary research. One promising approach is the integration of LLMs with external data sources and applications, which can significantly enhance their accuracy and relevance without the need for costly retraining. Retrieval Augmented Generation (RAG) \cite{lewis2020retrieval} is a prominent technique in this regard, augmenting LLMs by linking them to up-to-date external databases to improve the precision of their outputs. Additionally, advanced prompting strategies like chain-of-thought prompting \cite{wei2022chain} aid in improving the reasoning capabilities of LLMs, especially for tasks requiring multi-step logic.&#10;&#10;Moreover, specialized frameworks such as Program-Aided Language Models (PAL) \cite{gao2023pal} and ReAct \cite{yao2022react} have been developed to extend the functionalities of LLMs. PAL leverages external code interpreters to handle precise mathematical computations, while ReAct integrates reasoning with action planning to manage complex workflows. LangChain \cite{topsakal2023creating}, another significant development, offers modular components and agents to facilitate the integration of LLMs into diverse applications, enhancing their ability to execute complex tasks efficiently.&#10;&#10;This tutorial paper provides a comprehensive exploration of the challenges and solutions associated with LLMs. It begins by examining the inherent limitations of LLMs, such as temporal knowledge cutoffs, mathematical inaccuracies, and hallucinations. The paper then introduces Retrieval Augmented Generation (RAG) as a means to access real-time external information, thus improving LLM performance across various applications. For instance, in customer service bots, RAG can enable real-time interactions with databases and APIs, enhancing the relevance and accuracy of responses.&#10;&#10;The paper also discusses the integration of LLMs with external applications to perform complex tasks. Additionally, the concept of chain-of-thought prompting is introduced to enhance the reasoning capabilities of LLMs in multi-step tasks. This approach encourages the model to break down complex problems into intermediate steps, improving the coherence and logic of the generated responses.&#10;&#10;Further, the paper delves into the Program-Aided Language Model (PAL) framework \cite{gao2023pal}, which pairs LLMs with external code interpreters to execute accurate calculations. This integration is crucial for improving the mathematical capabilities of LLMs, allowing them to handle tasks that require precise numerical computations. Recent advancements such as the ReAct \cite{yao2022react} framework and LangChain \cite{topsakal2023creating} are also explored, highlighting their potential in guiding LLMs through structured prompts to solve intricate problems and support advanced techniques like PAL.&#10;&#10;The architectural components necessary for developing LLM-powered applications are thoroughly outlined, covering aspects such as infrastructure, deployment, and the integration of external information sources. The paper reviews various transformer-based models and recent advancements aimed at enhancing performance and efficiency. Techniques for scaling model training beyond a single GPU, including PyTorch’s Distributed Data Parallel (DDP) and Fully Sharded Data Parallel (FSDP) \cite{zhao2023pytorch}, are discussed, along with the ZeRO \cite{rajbhandari2019zero} stages that optimize memory usage during training.&#10;&#10;Fine-tuning strategies are a significant focus, exploring methods to enhance LLM performance for specific use cases. The paper examines instruction fine-tuning, multitask fine-tuning, and parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) \cite{hu2021lora} and prompt tuning \cite{lester2021prompt}. These techniques address the issue of catastrophic forgetting and suggest ways to mitigate it. Reinforcement Learning from Human Feedback (RLHF) \cite{ziegler2019,glaese2022improving} and Reinforced Self-Training \cite{gulcehre2023reinforced} are also explored as a method to align LLMs with human preferences. The use of Proximal Policy Optimization (PPO) \cite{schulman2017proximal} to update LLM weights based on human evaluations is discussed, along with challenges like reward hacking and the importance of maintaining model quality through techniques such as KL divergence.&#10;&#10;Finally, the paper provides an introduction to PPO \cite{schulman2017proximal}, its phases, and its objectives, including policy loss, value function loss, and entropy loss. It discusses how PPO ensures stable learning by constraining updates within a trust region and its application in fine-tuning LLMs for human alignment. The integration of external data sources and the application of advanced frameworks and fine-tuning strategies significantly enhance the performance and reliability of LLMs, offering an overview of the current state and future directions in LLM-based applications.&#10;&#10;The toolbox includes not only the code but also comprehensive tutorial slides to aid understanding and application. For fine-tuning models, we often need multi-task settings and diverse datasets to ensure robust performance across various tasks. Having references to the available datasets is crucial for researchers and practitioners in selecting appropriate data for their specific fine-tuning needs. Table \ref{tab:datasets} presents an overview of datasets and benchmarks commonly used in natural language processing tasks, including their type, citation, and a brief description. For additional resources and information, please contact or visit the \href{ Page} or \href{ Roffo}. The source code can be accessed by contacting the author via email for a request.&#10;&#10;In summary, this report serves as a guide to the techniques, architectures, and practical applications of Large Language Models. By addressing the inherent limitations of LLMs and introducing innovative frameworks and strategies, it aims to enhance the capabilities and reliability of these powerful tools in various real-world applications.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Limitations Solutions, Computer Science, Large Language Models, Linguistics, Artificial Intelligence, Mathematics, Training Techniques Optimization, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2005.11401" label="2005.11401">
        <attvalues>
          <attvalue for="0" value="Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" />
          <attvalue for="1" value="  Large pre-trained language models have been shown to store factual knowledge&#10;in their parameters, and achieve state-of-the-art results when fine-tuned on&#10;downstream NLP tasks. However, their ability to access and precisely manipulate&#10;knowledge is still limited, and hence on knowledge-intensive tasks, their&#10;performance lags behind task-specific architectures. Additionally, providing&#10;provenance for their decisions and updating their world knowledge remain open&#10;research problems. Pre-trained models with a differentiable access mechanism to&#10;explicit non-parametric memory can overcome this issue, but have so far been&#10;only investigated for extractive downstream tasks. We explore a general-purpose&#10;fine-tuning recipe for retrieval-augmented generation (RAG) -- models which&#10;combine pre-trained parametric and non-parametric memory for language&#10;generation. We introduce RAG models where the parametric memory is a&#10;pre-trained seq2seq model and the non-parametric memory is a dense vector index&#10;of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG&#10;formulations, one which conditions on the same retrieved passages across the&#10;whole generated sequence, the other can use different passages per token. We&#10;fine-tune and evaluate our models on a wide range of knowledge-intensive NLP&#10;tasks and set the state-of-the-art on three open domain QA tasks, outperforming&#10;parametric seq2seq models and task-specific retrieve-and-extract architectures.&#10;For language generation tasks, we find that RAG models generate more specific,&#10;diverse and factual language than a state-of-the-art parametric-only seq2seq&#10;baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Single-Task Retrieval&#10;Prior work has shown that retrieval improves performance across a variety of NLP tasks when considered in isolation.&#10;Such tasks include open-domain question answering~\cite{chen_reading_2017,kwiatkowski_natural_2019}, fact checking~\cite{thorne-etal-2018-fever}, fact completion~\cite{petroni2020how}, long-form question answering~\cite{fan-etal-2019-eli5}, Wikipedia article generation~\cite{liu2018generating}, dialogue~\cite{moghe-etal-2018-towards,weston-etal-2018-retrieve,dinan2018wizard,fan2020augmenting}, translation~\cite{gu2018search}, and language modeling~\cite{guu-etal-2018-generating,khandelwal2020generalization}.&#10;Our work unifies previous successes in incorporating retrieval into individual tasks, showing that a single retrieval-based architecture is capable of achieving strong performance across several tasks.&#10;&#10;General-Purpose Architectures for NLP&#10;Prior work on general-purpose architectures for NLP tasks has shown great success without the use of retrieval.&#10;A single, pre-trained language model has been shown to achieve strong performance on various classification tasks in the GLUE benchmarks~\cite{wang-etal-2018-glue,wang_superglue_2019} after fine-tuning~\cite{radford_improving_2018,devlin_bert:_2019}.&#10;GPT-2~\cite{radford2019language} later showed that a single, left-to-right, pre-trained language model could achieve strong performance across both discriminative and generative tasks.&#10;For further improvement, BART~\cite{lewis2019bart} and T5~\cite{raffel2019t5,roberts2020t5cqba} propose a single, pre-trained encoder-decoder model that leverages bi-directional attention to achieve stronger performance on discriminative and generative tasks.&#10;Our work aims to expand the space of possible tasks with a single, unified architecture, by learning a retrieval module to augment pre-trained, generative language models.&#10;&#10;Learned Retrieval&#10;There is significant work on learning to retrieve documents in information retrieval, more recently with pre-trained, neural language models~\cite{nogueira2019passage,Karpukhin20dense} similar to ours.&#10;Some work optimizes the retrieval module to aid in a specific, downstream task such as question answering, using search~\cite{perez-etal-2019-finding}, reinforcement learning~\cite{choi-etal-2017-coarse,wang2018evidence-aggregation,wang2018r3}, or a latent variable approach~\cite{lee-etal-2019-latent,guu2020realm} as in our work.&#10;These successes leverage different retrieval-based architectures and optimization techniques to achieve strong performance on a single task, while we show that a single retrieval-based architecture can be fine-tuned for strong performance on a variety of tasks.&#10;&#10;Memory-based Architectures&#10;Our document index can be seen as a large external memory for neural networks to attend to, analogous to memory networks~\cite{weston2015memory,sukhbaatar2015end}.&#10;Concurrent work~\cite{fevry2020entities} learns to retrieve a trained embedding for each entity in the input, rather than to retrieve raw text as in our work.&#10;Other work improves the ability of dialog models to generate factual text by attending over fact embeddings~\cite{ghazvininejad2018knowledge,fan2020augmenting}. A key feature of our memory is that it is comprised of raw text rather distributed representations, which makes the memory both (i) human-readable, lending a form of interpretability to our model, and (ii) human-writable, enabling us to dynamically update the model's memory by editing the document index. This approach has also been used in knowledge-intensive dialog, where generators have been conditioned on retrieved text directly, albeit obtained via TF-IDF rather than end-to-end learnt retrieval~\cite{dinan2018wizard}.&#10;&#10;Retrieve-and-Edit approaches Our method shares some similarities with retrieve-and-edit style approaches, where a similar training input-output pair is retrieved for a given input, and then edited to provide a final output. These approaches have proved successful in a number of domains including Machine Translation~ \cite{gu_mt_retrieve_and_edit,hossain-etal-2020-simple} and Semantic Parsing~\cite{NIPS2018_8209}. Our approach does have several differences, including less of emphasis on lightly editing a retrieved item, but on aggregating content from several pieces of retrieved content, as well as learning latent retrieval, and retrieving evidence documents rather than related training pairs. This said, RAG techniques may work well in these settings, and could represent promising future work.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2201.11903" label="2201.11903">
        <attvalues>
          <attvalue for="0" value="Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" />
          <attvalue for="1" value="  We explore how generating a chain of thought -- a series of intermediate&#10;reasoning steps -- significantly improves the ability of large language models&#10;to perform complex reasoning. In particular, we show how such reasoning&#10;abilities emerge naturally in sufficiently large language models via a simple&#10;method called chain of thought prompting, where a few chain of thought&#10;demonstrations are provided as exemplars in prompting. Experiments on three&#10;large language models show that chain of thought prompting improves performance&#10;on a range of arithmetic, commonsense, and symbolic reasoning tasks. The&#10;empirical gains can be striking. For instance, prompting a 540B-parameter&#10;language model with just eight chain of thought exemplars achieves state of the&#10;art accuracy on the GSM8K benchmark of math word problems, surpassing even&#10;finetuned GPT-3 with a verifier.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.10435" label="2211.10435">
        <attvalues>
          <attvalue for="0" value="PAL: Program-aided Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have recently demonstrated an impressive ability&#10;to perform arithmetic and symbolic reasoning tasks, when provided with a few&#10;examples at test time (&quot;few-shot prompting&quot;). Much of this success can be&#10;attributed to prompting methods such as &quot;chain-of-thought'', which employ LLMs&#10;for both understanding the problem description by decomposing it into steps, as&#10;well as solving each step of the problem. While LLMs seem to be adept at this&#10;sort of step-by-step decomposition, LLMs often make logical and arithmetic&#10;mistakes in the solution part, even when the problem is decomposed correctly.&#10;In this paper, we present Program-Aided Language models (PAL): a novel approach&#10;that uses the LLM to read natural language problems and generate programs as&#10;the intermediate reasoning steps, but offloads the solution step to a runtime&#10;such as a Python interpreter. With PAL, decomposing the natural language&#10;problem into runnable steps remains the only learning task for the LLM, while&#10;solving is delegated to the interpreter. We demonstrate this synergy between a&#10;neural LLM and a symbolic interpreter across 13 mathematical, symbolic, and&#10;algorithmic reasoning tasks from BIG-Bench Hard and other benchmarks. In all&#10;these natural language reasoning tasks, generating code using an LLM and&#10;reasoning using a Python interpreter leads to more accurate results than much&#10;larger models. For example, PAL using Codex achieves state-of-the-art few-shot&#10;accuracy on the GSM8K benchmark of math word problems, surpassing PaLM-540B&#10;which uses chain-of-thought by absolute 15% top-1. Our code and data are&#10;publicly available at http://reasonwithpal.com/ .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.03629" label="2210.03629">
        <attvalues>
          <attvalue for="0" value="ReAct: Synergizing Reasoning and Acting in Language Models" />
          <attvalue for="1" value="  While large language models (LLMs) have demonstrated impressive capabilities&#10;across tasks in language understanding and interactive decision making, their&#10;abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g.&#10;action plan generation) have primarily been studied as separate topics. In this&#10;paper, we explore the use of LLMs to generate both reasoning traces and&#10;task-specific actions in an interleaved manner, allowing for greater synergy&#10;between the two: reasoning traces help the model induce, track, and update&#10;action plans as well as handle exceptions, while actions allow it to interface&#10;with external sources, such as knowledge bases or environments, to gather&#10;additional information. We apply our approach, named ReAct, to a diverse set of&#10;language and decision making tasks and demonstrate its effectiveness over&#10;state-of-the-art baselines, as well as improved human interpretability and&#10;trustworthiness over methods without reasoning or acting components.&#10;Concretely, on question answering (HotpotQA) and fact verification (Fever),&#10;ReAct overcomes issues of hallucination and error propagation prevalent in&#10;chain-of-thought reasoning by interacting with a simple Wikipedia API, and&#10;generates human-like task-solving trajectories that are more interpretable than&#10;baselines without reasoning traces. On two interactive decision making&#10;benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and&#10;reinforcement learning methods by an absolute success rate of 34% and 10%&#10;respectively, while being prompted with only one or two in-context examples.&#10;Project site with code: https://react-lm.github.io&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.11277" label="2304.11277">
        <attvalues>
          <attvalue for="0" value="PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel" />
          <attvalue for="1" value="  It is widely acknowledged that large models have the potential to deliver&#10;superior performance across a broad range of domains. Despite the remarkable&#10;progress made in the field of machine learning systems research, which has&#10;enabled the development and exploration of large models, such abilities remain&#10;confined to a small group of advanced users and industry leaders, resulting in&#10;an implicit technical barrier for the wider community to access and leverage&#10;these technologies. In this paper, we introduce PyTorch Fully Sharded Data&#10;Parallel (FSDP) as an industry-grade solution for large model training. FSDP&#10;has been closely co-designed with several key PyTorch core components including&#10;Tensor implementation, dispatcher system, and CUDA memory caching allocator, to&#10;provide non-intrusive user experiences and high training efficiency.&#10;Additionally, FSDP natively incorporates a range of techniques and settings to&#10;optimize resource utilization across a variety of hardware configurations. The&#10;experimental results demonstrate that FSDP is capable of achieving comparable&#10;performance to Distributed Data Parallel while providing support for&#10;significantly larger models with near-linear scalability in terms of TFLOPS.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.09685" label="2106.09685">
        <attvalues>
          <attvalue for="0" value="LoRA: Low-Rank Adaptation of Large Language Models" />
          <attvalue for="1" value="  An important paradigm of natural language processing consists of large-scale&#10;pre-training on general domain data and adaptation to particular tasks or&#10;domains. As we pre-train larger models, full fine-tuning, which retrains all&#10;model parameters, becomes less feasible. Using GPT-3 175B as an example --&#10;deploying independent instances of fine-tuned models, each with 175B&#10;parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or&#10;LoRA, which freezes the pre-trained model weights and injects trainable rank&#10;decomposition matrices into each layer of the Transformer architecture, greatly&#10;reducing the number of trainable parameters for downstream tasks. Compared to&#10;GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable&#10;parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA&#10;performs on-par or better than fine-tuning in model quality on RoBERTa,&#10;DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher&#10;training throughput, and, unlike adapters, no additional inference latency. We&#10;also provide an empirical investigation into rank-deficiency in language model&#10;adaptation, which sheds light on the efficacy of LoRA. We release a package&#10;that facilitates the integration of LoRA with PyTorch models and provide our&#10;implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at&#10;https://github.com/microsoft/LoRA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08691" label="2104.08691">
        <attvalues>
          <attvalue for="0" value="The Power of Scale for Parameter-Efficient Prompt Tuning" />
          <attvalue for="1" value="  In this work, we explore &quot;prompt tuning&quot;, a simple yet effective mechanism&#10;for learning &quot;soft prompts&quot; to condition frozen language models to perform&#10;specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft&#10;prompts are learned through backpropagation and can be tuned to incorporate&#10;signal from any number of labeled examples. Our end-to-end learned approach&#10;outperforms GPT-3's &quot;few-shot&quot; learning by a large margin. More remarkably,&#10;through ablations on model size using T5, we show that prompt tuning becomes&#10;more competitive with scale: as models exceed billions of parameters, our&#10;method &quot;closes the gap&quot; and matches the strong performance of model tuning&#10;(where all model weights are tuned). This finding is especially relevant in&#10;that large models are costly to share and serve, and the ability to reuse one&#10;frozen model for multiple downstream tasks can ease this burden. Our method can&#10;be seen as a simplification of the recently proposed &quot;prefix tuning&quot; of Li and&#10;Liang (2021), and we provide a comparison to this and other similar approaches.&#10;Finally, we show that conditioning a frozen model with soft prompts confers&#10;benefits in robustness to domain transfer, as compared to full model tuning.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the wide success of pre-trained large language models, a range of techniques has arisen to adapt these general-purpose models to downstream tasks.&#10;ELMo \cite{peters-etal-2018-deep} proposed freezing the pre-trained model and learning a task-specific weighting of its per-layer representations. However, since GPT \cite{gpt} and BERT \cite{devlin-etal-2019-bert}, the dominant adaptation technique has been model tuning (or ``fine-tuning''), where all model parameters are tuned during adaptation, as proposed by \cite{howard-ruder-2018-universal}.&#10;&#10;More recently, \cite{brown_2020_gpt3} showed that prompt design (or ``priming'') is surprisingly effective at modulating a frozen GPT-3 model's behavior through text prompts.&#10;Prompts are typically composed of a task description and/or several canonical examples. This return to ``freezing'' pre-trained models is appealing, especially as model size continues to increase. Rather than requiring a separate copy of the model for each downstream task, a single generalist model can simultaneously serve many different tasks.&#10;&#10;Unfortunately, prompt-based adaptation has several key drawbacks.&#10;Task description is error-prone and requires human involvement, and the effectiveness of a prompt is limited by how much conditioning text can fit into the model's input.&#10;As a result, downstream task quality still lags far behind that of tuned models.&#10;For instance, GPT-3 175B few-shot performance on SuperGLUE is $17.5$ points below fine-tuned T5-XXL \cite{raffel_2020_t5} ($71.8$ vs.~$89.3$) despite using $16$ times more parameters.&#10;&#10;Several efforts to automate prompt design have been recently proposed.&#10;\cite{shin-etal-2020-autoprompt} propose a search algorithm over the discrete space of words, guided by the downstream application training data.&#10;While this technique outperforms manual prompt design, there is still a gap relative to model tuning.&#10;&#10;\cite{li_2021_prefix_tuning} propose ``prefix tuning'' and show strong results on generative tasks. This method freezes the model parameters and backpropagates the error during tuning to prefix activations prepended to each layer in the encoder stack, including the input layer.&#10;\cite{hambardzumyan_2021_warp} simplify this recipe by restricting the trainable parameters to the input and output sub-networks of a masked language model, and show reasonable results on classifications tasks.&#10;&#10;In this paper, we propose prompt tuning as a further simplification for adapting language models. We freeze the entire pre-trained model and only allow an additional $k$ tunable tokens per downstream task to be prepended to the input text. This ``soft prompt'' is trained end-to-end and can condense the signal from a full labeled dataset, allowing our method to outperform few-shot prompts and close the quality gap with model tuning (Figure~\ref{fig:model-size}). At the same time, since a single pre-trained model is recycled for all downstream tasks, we retain the efficient serving benefits of frozen models (Figure~\ref{fig:diagram}).&#10;&#10;While we developed our method concurrently with \cite{li_2021_prefix_tuning} and \cite{hambardzumyan_2021_warp}, we are the first to show that prompt tuning alone (with no intermediate-layer prefixes or task-specific output layers) is sufficient to be competitive with model tuning.&#10;Through detailed experiments in sections~\ref{sec:tuning}--\ref{sec:results}, we demonstrate that language model capacity is a key ingredient for these approaches to succeed. As Figure~\ref{fig:model-size} shows, prompt tuning becomes more competitive with scale.&#10;&#10;We compare with similar approaches in Section~\ref{sec:previous_work}. Explicitly separating task-specific parameters from the ``generalist'' parameters needed for general language-understanding has a range of additional benefits. We show in Section~\ref{sec:shift} that by capturing the task definition in the prompt while keeping the generalist parameters fixed, we are able to achieve better resilience to domain shifts. In Section~\ref{sec:ensemble}, we show that ``prompt ensembling'', learning multiple prompts for the same task, can boost quality and is more efficient than classic model ensembling. Finally, in Section~\ref{sec:interpretability}, we investigate the interpretability of our learned soft prompts. In sum, our key contributions are:&#10;&#10;\begin{enumerate} [topsep=3pt,itemsep=-1ex,partopsep=1ex,parsep=1ex]&#10;\item Proposing prompt tuning and showing its competitiveness with model tuning in the regime of large language models.&#10;\item Ablating many design choices, and showing quality and robustness improve with scale.&#10;\item Showing prompt tuning outperforms model tuning on domain shift problems.&#10;\item Proposing ``prompt ensembling'' and showing its effectiveness.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.08593" label="1909.08593">
        <attvalues>
          <attvalue for="0" value="Fine-Tuning Language Models from Human Preferences" />
          <attvalue for="1" value="  Reward learning enables the application of reinforcement learning (RL) to&#10;tasks where reward is defined by human judgment, building a model of reward by&#10;asking humans questions. Most work on reward learning has used simulated&#10;environments, but complex information about values is often expressed in&#10;natural language, and we believe reward learning for language is a key to&#10;making RL practical and safe for real-world tasks. In this paper, we build on&#10;advances in generative pretraining of language models to apply reward learning&#10;to four natural language tasks: continuing text with positive sentiment or&#10;physically descriptive language, and summarization tasks on the TL;DR and&#10;CNN/Daily Mail datasets. For stylistic continuation we achieve good results&#10;with only 5,000 comparisons evaluated by humans. For summarization, models&#10;trained with 60,000 comparisons copy whole sentences from the input but skip&#10;irrelevant preamble; this leads to reasonable ROUGE scores and very good&#10;performance according to our human labelers, but may be exploiting the fact&#10;that labelers rely on simple heuristics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.14375" label="2209.14375">
        <attvalues>
          <attvalue for="0" value="Improving alignment of dialogue agents via targeted human judgements" />
          <attvalue for="1" value="  We present Sparrow, an information-seeking dialogue agent trained to be more&#10;helpful, correct, and harmless compared to prompted language model baselines.&#10;We use reinforcement learning from human feedback to train our models with two&#10;new additions to help human raters judge agent behaviour. First, to make our&#10;agent more helpful and harmless, we break down the requirements for good&#10;dialogue into natural language rules the agent should follow, and ask raters&#10;about each rule separately. We demonstrate that this breakdown enables us to&#10;collect more targeted human judgements of agent behaviour and allows for more&#10;efficient rule-conditional reward models. Second, our agent provides evidence&#10;from sources supporting factual claims when collecting preference judgements&#10;over model statements. For factual questions, evidence provided by Sparrow&#10;supports the sampled response 78% of the time. Sparrow is preferred more often&#10;than baselines while being more resilient to adversarial probing by humans,&#10;violating our rules only 8% of the time when probed. Finally, we conduct&#10;extensive analyses showing that though our model learns to follow our rules it&#10;can exhibit distributional biases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.08998" label="2308.08998">
        <attvalues>
          <attvalue for="0" value="Reinforced Self-Training (ReST) for Language Modeling" />
          <attvalue for="1" value="  Reinforcement learning from human feedback (RLHF) can improve the quality of&#10;large language model's (LLM) outputs by aligning them with human preferences.&#10;We propose a simple algorithm for aligning LLMs with human preferences inspired&#10;by growing batch reinforcement learning (RL), which we call Reinforced&#10;Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by&#10;generating samples from the policy, which are then used to improve the LLM&#10;policy using offline RL algorithms. ReST is more efficient than typical online&#10;RLHF methods because the training dataset is produced offline, which allows&#10;data reuse. While ReST is a general approach applicable to all generative&#10;learning settings, we focus on its application to machine translation. Our&#10;results show that ReST can substantially improve translation quality, as&#10;measured by automated metrics and human evaluation on machine translation&#10;benchmarks in a compute and sample-efficient manner.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.06347" label="1707.06347">
        <attvalues>
          <attvalue for="0" value="Proximal Policy Optimization Algorithms" />
          <attvalue for="1" value="  We propose a new family of policy gradient methods for reinforcement&#10;learning, which alternate between sampling data through interaction with the&#10;environment, and optimizing a &quot;surrogate&quot; objective function using stochastic&#10;gradient ascent. Whereas standard policy gradient methods perform one gradient&#10;update per data sample, we propose a novel objective function that enables&#10;multiple epochs of minibatch updates. The new methods, which we call proximal&#10;policy optimization (PPO), have some of the benefits of trust region policy&#10;optimization (TRPO), but they are much simpler to implement, more general, and&#10;have better sample complexity (empirically). Our experiments test PPO on a&#10;collection of benchmark tasks, including simulated robotic locomotion and Atari&#10;game playing, and we show that PPO outperforms other online policy gradient&#10;methods, and overall strikes a favorable balance between sample complexity,&#10;simplicity, and wall-time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08758" label="2104.08758">
        <attvalues>
          <attvalue for="0" value="Documenting Large Webtext Corpora: A Case Study on the Colossal Clean&#10;  Crawled Corpus" />
          <attvalue for="1" value="  Large language models have led to remarkable progress on many NLP tasks, and&#10;researchers are turning to ever-larger text corpora to train them. Some of the&#10;largest corpora available are made by scraping significant portions of the&#10;internet, and are frequently introduced with only minimal documentation. In&#10;this work we provide some of the first documentation for the Colossal Clean&#10;Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set&#10;of filters to a single snapshot of Common Crawl. We begin by investigating&#10;where the data came from, and find a significant amount of text from unexpected&#10;sources like patents and US military websites. Then we explore the content of&#10;the text itself, and find machine-generated text (e.g., from machine&#10;translation systems) and evaluation examples from other benchmark NLP datasets.&#10;To understand the impact of the filters applied to create this dataset, we&#10;evaluate the text that was removed, and show that blocklist filtering&#10;disproportionately removes text from and about minority individuals. Finally,&#10;we conclude with some recommendations for how to created and document web-scale&#10;datasets from a scrape of the internet.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;BERT \cite{devlin-etal-2019-bert} was trained on BooksCorpus \cite{Zhu2015AligningBA} and English-language Wikipedia. It was soon improved with additional data \cite[RoBERTa;][]{liu2019roberta}: a portion of CC-News \cite{nagel2016ccnews}, OpenWebText \cite{Gokaslan2019OpenWeb,Radford2019LanguageMA}, and Stories \cite{Trinh2018ASM}. Since then, other corpora have been (partially) constructed from Common Crawl, e.g., Pile \cite{gao2020pile}, CCNet \cite{wenzek2019ccnet}, and mC4 \cite{xue-etal-2021-mt5}. &#10;\cite{luccioni-viviano-2021-whats} provide some exploratory analysis of undesirable content in Common Crawl, wherein they find hatespeech and adult content.&#10;One of the largest language models, GPT-3 \cite{brown2020gpt3}, was trained on a mixture of filtered Common Crawl (60\% of GPT-3's data), WebText2 \cite[22\%;][]{kaplan2020scaling}, Books1 and Books2 \cite[8\% each;][]{brown2020gpt3}, and English-language Wikipedia (3\%). GPT-3's Common Crawl data was downloaded from 41 monthly ``snapshots'' from 2016--2019, and it constitutes 45TB of compressed text before filtering and 570GB after ($\sim$400 billion byte-pair-encoded tokens). &#10;&#10;Since analyzing pretraining corpora is challenging due to their size, their documentation is often missing \cite{bender-etal-2021-parrots, Paullada2020DataAI}. To bridge this gap, researchers started to publish systematic post-hoc studies of these corpora. \cite{gehman-etal-2020-realtoxicityprompts} provide an in-depth analysis with respect to toxicity and fake news of OpenWebText. \cite{Caswell2021QualityAA} recruited 51 volunteers speaking 70 languages to judge whether five publicly available multilingual web-crawled corpora \cite{el-kishky-etal-2020-ccaligned,xue-etal-2021-mt5,ortiz-suarez-etal-2020-monolingual,banon-etal-2020-paracrawl,Schwenk2019WikiMatrixM1} contain text in languages they report, as well as their quality.&#10;\cite{jo2020lessons} discuss parallels between creating historical archives and the curation of machine learning datasets including pretraining corpora. \cite{hutchinson2021towards} introduce a ``framework for dataset development transparency that supports decision-making and accountability'' that could be used for developing pretraining corpora. The Masakhane organization advocates for participatory research \cite{nekoto-etal-2020-participatory}, a set of methodologies that includes all necessary agents, e.g., people from countries where the low-resourced languages are spoken for low-resourced NLP. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Dataset Bias Analysis, Web-Scale Dataset Creation, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2306.08189" label="2306.08189">
        <attvalues>
          <attvalue for="0" value="Language models are not naysayers: An analysis of language models on&#10;  negation benchmarks" />
          <attvalue for="1" value="  Negation has been shown to be a major bottleneck for masked language models,&#10;such as BERT. However, whether this finding still holds for larger-sized&#10;auto-regressive language models (``LLMs'') has not been studied&#10;comprehensively. With the ever-increasing volume of research and applications&#10;of LLMs, we take a step back to evaluate the ability of current-generation LLMs&#10;to handle negation, a fundamental linguistic phenomenon that is central to&#10;language understanding. We evaluate different LLMs -- including the open-source&#10;GPT-neo, GPT-3, and InstructGPT -- against a wide range of negation benchmarks.&#10;Through systematic experimentation with varying model sizes and prompts, we&#10;show that LLMs have several limitations including insensitivity to the presence&#10;of negation, an inability to capture the lexical semantics of negation, and a&#10;failure to reason under negation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Our work builds upon previous research on negation. In particular, we were inspired by the pioneering works of \cite{kassner-schutze-2020-negated} and \cite{ettinger-2020-bert}, which reveal that pre-trained language models have a major issue in being insensitive to the presence of negation, based on evaluation over a set of cloze-style queries.&#10;Following this line of research, \cite{jang-etal-2022-beyond} also explored negation in a cloze completion context by negating factual statements extracted from ConceptNet and come to a similar finding.&#10;&#10;In a broader context, \cite{hossain-etal-2020-analysis,hossain-etal-2022-analysis} investigated the performance of BERT-based methods on samples containing negation in the GLUE \cite{wang-etal-2018-glue} and SuperGLUE \cite{wang2019superglue} datasets. Their main finding is that the results for the subsets containing only negation are lower than those without, as well as the whole test set, showing that models struggle with negation, even when fine-tuned on relevant training data. &#10;\cite{ravichander2022condaqa} proposed the challenging CONDAQA dataset to test the ability of models to reason about the implications of negation. &#10;The authors conducted comprehensive analysis of different types of LLMs under different settings, and found that the best-performing models were still well below human performance.&#10;Negation has also been investigated as part of psycholinguistic probing datasets \cite{lialin-etal-2022-life,jumelet-etal-2021-language,staliunaite-iacobacci-2020-compositional}.&#10;Contrasting previous finding, \cite{gubelmann-handschuh-2022-context} found that the ability to understand negation of LMs is underestimated in previous studied.&#10;Through designing a controlled dataset with minimal pairs varying in syntactic structure, gender, profession, and first name, they concluded that the models are indeed sensitive to negation and thus, their struggle comes more from the contextualization of the tasks.&#10;&#10;As part of the analysis on emergent abilities of LMs, negation has been shown to be one of the tasks that displays a flat scaling curve \cite{weiemergent} or even inverse-scaling \cite{inverse-scaling-prize}. This behaviour was later shown to be alleviated by instruction fine-tuning \cite{wei2022inverse}.&#10;The effectiveness of instruction fine-tuning is further supported in \cite{jang2023consistency}. &#10;The authors investigated the logical consistency of ChatGPT and found that ChatGPT understands negation and antonyms much better than previous models.&#10;&#10;Beside probing and evaluation, there have also been works on making language models more robust to negation, including unlikelihood training \cite{hosseini-etal-2021-understanding}, adaptive pre-training on relevant data \cite{truong-etal-2022-improving}, leveraging affirmative interpretations from negation \cite{hossain-blanco-2022-leveraging}, and learning better representation of negation through contrastive learning \cite{jiang-etal-2022-promptbert, wang2022sncse}.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Language Model Limitations, Artificial Intelligence, Natural Language Processing, Negation in Language" />
        </attvalues>
      </node>
      <node id="2305.14196" label="2305.14196">
        <attvalues>
          <attvalue for="0" value="ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding" />
          <attvalue for="1" value="  We introduce ZeroSCROLLS, a zero-shot benchmark for natural language&#10;understanding over long texts, which contains only test and small validation&#10;sets, without training data. We adapt six tasks from the SCROLLS benchmark, and&#10;add four new datasets, including two novel information fusing tasks, such as&#10;aggregating the percentage of positive reviews. Using ZeroSCROLLS, we conduct a&#10;comprehensive evaluation of both open-source and closed large language models,&#10;finding that Claude outperforms ChatGPT, and that GPT-4 achieves the highest&#10;average score. However, there is still room for improvement on multiple open&#10;challenges in ZeroSCROLLS, such as aggregation tasks, where models struggle to&#10;pass the naive baseline. As the state of the art is a moving target, we invite&#10;researchers to evaluate their ideas on the live ZeroSCROLLS leaderboard.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have been improving at an incredible pace, solving problems that seemed out of reach, without any task-specific training examples \cite{wei2022finetuned, ouyang2022training, openai2023gpt4}. &#10;As commercial LLMs are adopted worldwide, it becomes clear that they must also operate successfully over long sequences, such as conversation histories or scientific documents.&#10;However, current LLM benchmarks that do evaluate models in a zero-shot setting, such as HELM \cite{liang2022holistic} and BigBench \cite{srivastava2022imitation}, mostly focus on short sequences; BigBench, for example, has an average of 77 words per input.&#10;To fill this gap, we introduce ZeroScrolls: Zero-Shot CompaRison Over Long Language Sequences, a benchmark for zero-shot long text reasoning over natural language, and conduct a thorough investigation of state-of-the-art LLMs.&#10;&#10;ZeroScrolls extends Scrolls \cite{shaham-etal-2022-scrolls}, a long text understanding benchmark that enables fine-tuning, adding four additional tasks: query-based summarization, multi-hop question answering, sentiment aggregation, and sorting book section summaries.&#10;We specifically design the latter two tasks to examine a model's ability to aggregate and compare information across long sequences, while keeping evaluation simple and accurate.&#10;ZeroScrolls is designed to test zero-shot capabilities, and contains test sets with simple natural prompts and private gold references, small validation sets, and no train data. &#10;It has a live leaderboard to enable transparent and dynamic progress.&#10;Figure~\ref{fig:zs_score} shows the state of the leaderboard based on our experiments, and \ref{fig:bars_two_cols} shows a per-task breakdown of a selected subset of models.&#10;&#10;We use this new testbed to perform extensive evaluation and analysis across state-of-the-art open and closed models.&#10;On question answering tasks, we find that zero-shot LLMs bridge the gap with task-specific fine-tuned models; GPT-4 sets a new state of the art on the challenging QuALITY task \cite{pang-etal-2022-quality}, almost reaching human performance. &#10;In contrast, LLMs generally struggle to obtain such high scores for summarization tasks without a training set from which to learn the nuances and artifacts of each dataset, even though GPT-4 does approach the fine-tuned state of the art on two of three datasets.&#10;We also observe that two of our new tasks, sentiment aggregation and sorting book section summaries, prove exceptionally challenging for all LLMs, with only GPT-4 surpassing the naive baseline in each task. Our code is available online.&#10;&#10;When analyzing GPT-4 responses, we often find correct answers that do not match the requested format; e.g. producing a full sentence when asked to answer in a single phrase.&#10;This problem is not unique to GPT-4, as different models may deviate from the specified format in different tasks.&#10;While ZeroScrolls is primarily aimed at facilitating research in understanding long texts, we encourage the community to use this benchmark to advance research in instruction understanding, prompt engineering, and evaluation of generated texts as well.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Natural Language Understanding, Benchmark Evaluation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2109.01652" label="2109.01652">
        <attvalues>
          <attvalue for="0" value="Finetuned Language Models Are Zero-Shot Learners" />
          <attvalue for="1" value="  This paper explores a simple method for improving the zero-shot learning&#10;abilities of language models. We show that instruction tuning -- finetuning&#10;language models on a collection of tasks described via instructions --&#10;substantially improves zero-shot performance on unseen tasks.&#10;  We take a 137B parameter pretrained language model and instruction-tune it on&#10;over 60 NLP tasks verbalized via natural language instruction templates. We&#10;evaluate this instruction-tuned model, which we call FLAN, on unseen task&#10;types. FLAN substantially improves the performance of its unmodified&#10;counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we&#10;evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE,&#10;BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number&#10;of finetuning datasets, model scale, and natural language instructions are key&#10;to the success of instruction tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.04615" label="2206.04615">
        <attvalues>
          <attvalue for="0" value="Beyond the Imitation Game: Quantifying and extrapolating the&#10;  capabilities of language models" />
          <attvalue for="1" value="  Language models demonstrate both quantitative improvement and new qualitative&#10;capabilities with increasing scale. Despite their potentially transformative&#10;impact, these new capabilities are as yet poorly characterized. In order to&#10;inform future research, prepare for disruptive new model capabilities, and&#10;ameliorate socially harmful effects, it is vital that we understand the present&#10;and near-future capabilities and limitations of language models. To address&#10;this challenge, we introduce the Beyond the Imitation Game benchmark&#10;(BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450&#10;authors across 132 institutions. Task topics are diverse, drawing problems from&#10;linguistics, childhood development, math, common-sense reasoning, biology,&#10;physics, social bias, software development, and beyond. BIG-bench focuses on&#10;tasks that are believed to be beyond the capabilities of current language&#10;models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense&#10;transformer architectures, and Switch-style sparse transformers on BIG-bench,&#10;across model sizes spanning millions to hundreds of billions of parameters. In&#10;addition, a team of human expert raters performed all tasks in order to provide&#10;a strong baseline. Findings include: model performance and calibration both&#10;improve with scale, but are poor in absolute terms (and when compared with&#10;rater performance); performance is remarkably similar across model classes,&#10;though with benefits from sparsity; tasks that improve gradually and&#10;predictably commonly involve a large knowledge or memorization component,&#10;whereas tasks that exhibit &quot;breakthrough&quot; behavior at a critical scale often&#10;involve multiple steps or components, or brittle metrics; social bias typically&#10;increases with scale in settings with ambiguous context, but this can be&#10;improved with prompting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.03533" label="2201.03533">
        <attvalues>
          <attvalue for="0" value="SCROLLS: Standardized CompaRison Over Long Language Sequences" />
          <attvalue for="1" value="  NLP benchmarks have largely focused on short texts, such as sentences and&#10;paragraphs, even though long texts comprise a considerable amount of natural&#10;language in the wild. We introduce SCROLLS, a suite of tasks that require&#10;reasoning over long texts. We examine existing long-text datasets, and handpick&#10;ones where the text is naturally long, while prioritizing tasks that involve&#10;synthesizing information across the input. SCROLLS contains summarization,&#10;question answering, and natural language inference tasks, covering multiple&#10;domains, including literature, science, business, and entertainment. Initial&#10;baselines, including Longformer Encoder-Decoder, indicate that there is ample&#10;room for improvement on SCROLLS. We make all datasets available in a unified&#10;text-to-text format and host a live leaderboard to facilitate research on model&#10;architecture and pretraining methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08608" label="2112.08608">
        <attvalues>
          <attvalue for="0" value="QuALITY: Question Answering with Long Input Texts, Yes!" />
          <attvalue for="1" value="  To enable building and testing models on long-document comprehension, we&#10;introduce QuALITY, a multiple-choice QA dataset with context passages in&#10;English that have an average length of about 5,000 tokens, much longer than&#10;typical current models can process. Unlike in prior work with passages, our&#10;questions are written and validated by contributors who have read the entire&#10;passage, rather than relying on summaries or excerpts. In addition, only half&#10;of the questions are answerable by annotators working under tight time&#10;constraints, indicating that skimming and simple search are not enough to&#10;consistently perform well. Our baseline models perform poorly on this task&#10;(55.4%) and significantly lag behind human performance (93.5%).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.13033" label="2404.13033">
        <attvalues>
          <attvalue for="0" value="Sample Design Engineering: An Empirical Study of What Makes Good&#10;  Downstream Fine-Tuning Samples for LLMs" />
          <attvalue for="1" value="  In the burgeoning field of Large Language Models (LLMs) like ChatGPT and&#10;LLaMA, Prompt Engineering (PE) is renowned for boosting zero-shot or in-context&#10;learning (ICL) through prompt modifications. Yet, the realm of the sample&#10;design for downstream fine-tuning, crucial for task-specific LLM adaptation, is&#10;largely unexplored. This paper introduces Sample Design Engineering (SDE), a&#10;methodical approach to enhancing LLMs' post-tuning performance by refining&#10;input, output, and reasoning designs. We conduct a series of in-domain (ID) and&#10;out-of-domain (OOD) experiments to assess the impact of various design options&#10;on LLMs' downstream performance, revealing several intriguing patterns that&#10;hold consistently across different LLMs. Based on these insights, we propose an&#10;integrated SDE strategy, combining the most effective options, and validate its&#10;consistent superiority over heuristic sample designs in complex downstream&#10;tasks like multi-aspect sentiment analysis, event extraction, and nested entity&#10;recognition. Additionally, analyses of LLMs' inherent prompt/output perplexity,&#10;zero-shot, and ICL abilities illustrate that good PE strategies may not always&#10;translate to good SDE strategies. Code available at&#10;https://github.com/beyondguo/LLM-Tuning.&#10;" />
          <attvalue for="2" value="&#10;&#10;The emergence of Large Language Models (LLMs) such as GPT-3 \cite{GPT-3}, PaLM \cite{chowdhery2023palm}, LLaMA \cite{touvron2023llama-1} and GPT-4 \cite{achiam2023gpt4} revolutionized natural language processing (NLP), enabling complex tasks to be tackled with a single model. This shift has profoundly broadened the range of tasks manageable by NLP models, while simultaneously consolidating the methodologies for various tasks under the unified framework of text generation. In this background, Prompt Engineering (PE) has emerged as a key area in leveraging cutting-edge LLMs, leading to advances in applying LLMs to new tasks \cite{GPT-3}, enhancing logical reasoning \cite{wei2022COT}, and increasing task-specific accuracy \cite{Wang2023Prompt-health,wei2023chatie}, without updating model weights.&#10;&#10;While numerous PE techniques have been developed for LLMs' zero-shot and in-context learning (ICL), the challenge of designing effective training samples for fine-tuning LLMs—termed Sample Design Engineering (SDE) in this paper—remains underexplored. SDE is crucial for tailoring smaller open-source LLMs to specific requirements, especially given the complexity of training samples for downstream tasks. Figure \ref{fig:first-image} is a simplified demonstration of PE and SDE.&#10;&#10;To address this gap, this paper undertakes a detailed and comprehensive exploration of SDE for LLMs' downstream fine-tuning. Our study is based on the hypothesis that the structure or elements of training samples may have a big impact on the fine-tuned LLMs. Different sample designs may make it easier or harder for the LLMs to learn, especially in scenarios where data is scarce. &#10;&#10;We begin by identifying a range of typical SDE options and categorizing them into three groups: input, output, and reasoning design options (shown in Figure \ref{fig:Design-demo}). To reveal the impact of each SDE option, we conduct experiments on a typical downstream scenario – multi-aspect sentiment analysis (MASA), with 2 in-domain (ID) tasks and 2 out-of-domain (OOD) tasks. Different from instruction-tuning datasets like FLAN \cite{longpre2023-FLAN-data}, the MASA task involves more complicated input and output elements, making it suitable for in-depth investigation of different sample designs.&#10;Comprehensive experiments on these 4 tasks with 6 popular open-source LLMs are undertaken to reveal how different SDE options affect downstream performances. Some interesting and thought-provoking conclusions are revealed through our experiments. For example, simply switching the position of the task instruction can make a difference; adding placeholders to unmentioned targets brings a notable performance gain, etc.&#10;&#10;Leveraging these findings, we combine the empirically well-performing SDE options and propose an integrated SDE strategy ES-SDE. Extensive experiments on 3 complex downstream tasks (Nested-NER, Event Detection, and MASA) on 2 additional LLMs demonstrate that ES-SDE notably surpasses weaker SDE combination, as well as heuristic design from other studies. &#10;ES-SDE's robustness on different training sizes, decoding randomness or instruction variation further underscores its stable effectiveness.&#10;&#10;In an exploratory analysis, we investigate the link between effective prompt and sample designs, via perplexity, zero-shot, and ICL analysis. Our findings suggest that a well-crafted PE strategy may not necessarily translate to a successful SDE strategy. This observation encourages further research into SDE's mechanisms, promising for enhancing LLMs' downstream applications.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Large Language Models, Linguistics, Sample Design Optimization, Artificial Intelligence, Mathematics, Natural Language Processing, Prompt Engineering" />
        </attvalues>
      </node>
      <node id="2005.14165" label="2005.14165">
        <attvalues>
          <attvalue for="0" value="Language Models are Few-Shot Learners" />
          <attvalue for="1" value="  Recent work has demonstrated substantial gains on many NLP tasks and&#10;benchmarks by pre-training on a large corpus of text followed by fine-tuning on&#10;a specific task. While typically task-agnostic in architecture, this method&#10;still requires task-specific fine-tuning datasets of thousands or tens of&#10;thousands of examples. By contrast, humans can generally perform a new language&#10;task from only a few examples or from simple instructions - something which&#10;current NLP systems still largely struggle to do. Here we show that scaling up&#10;language models greatly improves task-agnostic, few-shot performance, sometimes&#10;even reaching competitiveness with prior state-of-the-art fine-tuning&#10;approaches. Specifically, we train GPT-3, an autoregressive language model with&#10;175 billion parameters, 10x more than any previous non-sparse language model,&#10;and test its performance in the few-shot setting. For all tasks, GPT-3 is&#10;applied without any gradient updates or fine-tuning, with tasks and few-shot&#10;demonstrations specified purely via text interaction with the model. GPT-3&#10;achieves strong performance on many NLP datasets, including translation,&#10;question-answering, and cloze tasks, as well as several tasks that require&#10;on-the-fly reasoning or domain adaptation, such as unscrambling words, using a&#10;novel word in a sentence, or performing 3-digit arithmetic. At the same time,&#10;we also identify some datasets where GPT-3's few-shot learning still struggles,&#10;as well as some datasets where GPT-3 faces methodological issues related to&#10;training on large web corpora. Finally, we find that GPT-3 can generate samples&#10;of news articles which human evaluators have difficulty distinguishing from&#10;articles written by humans. We discuss broader societal impacts of this finding&#10;and of GPT-3 in general.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.02311" label="2204.02311">
        <attvalues>
          <attvalue for="0" value="PaLM: Scaling Language Modeling with Pathways" />
          <attvalue for="1" value="  Large language models have been shown to achieve remarkable performance&#10;across a variety of natural language tasks using few-shot learning, which&#10;drastically reduces the number of task-specific training examples needed to&#10;adapt the model to a particular application. To further our understanding of&#10;the impact of scale on few-shot learning, we trained a 540-billion parameter,&#10;densely activated, Transformer language model, which we call Pathways Language&#10;Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML&#10;system which enables highly efficient training across multiple TPU Pods. We&#10;demonstrate continued benefits of scaling by achieving state-of-the-art&#10;few-shot learning results on hundreds of language understanding and generation&#10;benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough&#10;performance, outperforming the finetuned state-of-the-art on a suite of&#10;multi-step reasoning tasks, and outperforming average human performance on the&#10;recently released BIG-bench benchmark. A significant number of BIG-bench tasks&#10;showed discontinuous improvements from model scale, meaning that performance&#10;steeply increased as we scaled to our largest model. PaLM also has strong&#10;capabilities in multilingual tasks and source code generation, which we&#10;demonstrate on a wide array of benchmarks. We additionally provide a&#10;comprehensive analysis on bias and toxicity, and study the extent of training&#10;data memorization with respect to model scale. Finally, we discuss the ethical&#10;considerations related to large language models and discuss potential&#10;mitigation strategies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.08774" label="2303.08774">
        <attvalues>
          <attvalue for="0" value="GPT-4 Technical Report" />
          <attvalue for="1" value="  We report the development of GPT-4, a large-scale, multimodal model which can&#10;accept image and text inputs and produce text outputs. While less capable than&#10;humans in many real-world scenarios, GPT-4 exhibits human-level performance on&#10;various professional and academic benchmarks, including passing a simulated bar&#10;exam with a score around the top 10% of test takers. GPT-4 is a&#10;Transformer-based model pre-trained to predict the next token in a document.&#10;The post-training alignment process results in improved performance on measures&#10;of factuality and adherence to desired behavior. A core component of this&#10;project was developing infrastructure and optimization methods that behave&#10;predictably across a wide range of scales. This allowed us to accurately&#10;predict some aspects of GPT-4's performance based on models trained with no&#10;more than 1/1,000th the compute of GPT-4.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.14670" label="2304.14670">
        <attvalues>
          <attvalue for="0" value="Prompt Engineering for Healthcare: Methodologies and Applications" />
          <attvalue for="1" value="  Prompt engineering is a critical technique in the field of natural language&#10;processing that involves designing and optimizing the prompts used to input&#10;information into models, aiming to enhance their performance on specific tasks.&#10;With the recent advancements in large language models, prompt engineering has&#10;shown significant superiority across various domains and has become&#10;increasingly important in the healthcare domain. However, there is a lack of&#10;comprehensive reviews specifically focusing on prompt engineering in the&#10;medical field. This review will introduce the latest advances in prompt&#10;engineering in the field of natural language processing for the medical field.&#10;First, we will provide the development of prompt engineering and emphasize its&#10;significant contributions to healthcare natural language processing&#10;applications such as question-answering systems, text summarization, and&#10;machine translation. With the continuous improvement of general large language&#10;models, the importance of prompt engineering in the healthcare domain is&#10;becoming increasingly prominent. The aim of this article is to provide useful&#10;resources and bridges for healthcare natural language processing researchers to&#10;better explore the application of prompt engineering in this field. We hope&#10;that this review can provide new ideas and inspire for research and application&#10;in medical natural language processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.13688" label="2301.13688">
        <attvalues>
          <attvalue for="0" value="The Flan Collection: Designing Data and Methods for Effective&#10;  Instruction Tuning" />
          <attvalue for="1" value="  We study the design decisions of publicly available instruction tuning&#10;methods, and break down the development of Flan 2022 (Chung et al., 2022).&#10;Through careful ablation studies on the Flan Collection of tasks and methods,&#10;we tease apart the effect of design decisions which enable Flan-T5 to&#10;outperform prior work by 3-17%+ across evaluation settings. We find task&#10;balancing and enrichment techniques are overlooked but critical to effective&#10;instruction tuning, and in particular, training with mixed prompt settings&#10;(zero-shot, few-shot, and chain-of-thought) actually yields stronger (2%+)&#10;performance in all settings. In further experiments, we show Flan-T5 requires&#10;less finetuning to converge higher and faster than T5 on single downstream&#10;tasks, motivating instruction-tuned models as more computationally-efficient&#10;starting checkpoints for new tasks. Finally, to accelerate research on&#10;instruction tuning, we make the Flan 2022 collection of datasets, templates,&#10;and methods publicly available at&#10;https://github.com/google-research/FLAN/tree/main/flan/v2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.15319" label="2405.15319">
        <attvalues>
          <attvalue for="0" value="Stacking Your Transformers: A Closer Look at Model Growth for Efficient&#10;  LLM Pre-Training" />
          <attvalue for="1" value="  LLMs are computationally expensive to pre-train due to their large scale.&#10;Model growth emerges as a promising approach by leveraging smaller models to&#10;accelerate the training of larger ones. However, the viability of these model&#10;growth methods in efficient LLM pre-training remains underexplored. This work&#10;identifies three critical $\underline{\textit{O}}$bstacles: ($\textit{O}$1)&#10;lack of comprehensive evaluation, ($\textit{O}$2) untested viability for&#10;scaling, and ($\textit{O}$3) lack of empirical guidelines. To tackle&#10;$\textit{O}$1, we summarize existing approaches into four atomic growth&#10;operators and systematically evaluate them in a standardized LLM pre-training&#10;setting. Our findings reveal that a depthwise stacking operator, called&#10;$G_{\text{stack}}$, exhibits remarkable acceleration in training, leading to&#10;decreased loss and improved overall performance on eight standard NLP&#10;benchmarks compared to strong baselines. Motivated by these promising results,&#10;we conduct extensive experiments to delve deeper into $G_{\text{stack}}$ to&#10;address $\textit{O}$2 and $\textit{O}$3. For $\textit{O}$2 (untested&#10;scalability), our study shows that $G_{\text{stack}}$ is scalable and&#10;consistently performs well, with experiments up to 7B LLMs after growth and&#10;pre-training LLMs with 750B tokens. For example, compared to a conventionally&#10;trained 7B model using 300B tokens, our $G_{\text{stack}}$ model converges to&#10;the same loss with 194B tokens, resulting in a 54.6\% speedup. We further&#10;address $\textit{O}$3 (lack of empirical guidelines) by formalizing guidelines&#10;to determine growth timing and growth factor for $G_{\text{stack}}$, making it&#10;practical in general LLM pre-training. We also provide in-depth discussions and&#10;comprehensive ablation studies of $G_{\text{stack}}$. Our code and pre-trained&#10;model are available at&#10;$\href{https://llm-stacking.github.io/}{https://llm-stacking.github.io/}$.&#10;" />
          <attvalue for="2" value="&#10;&#10;Emergent abilities of Large Language Models (LLMs) rely on scaling-up~\cite{brown2020language, wei2022emergent}. &#10;Empirical evidence from scaling laws~\cite{kaplan2020scaling, hoffmann2022training, alabdulmohsin2022revisiting} fuels the development of increasingly larger models, pushing the boundaries of LLMs capabilities.&#10;However, pre-training these gigantic models comes at a significant cost in terms of energy consumption and environmental impact~\cite{xu2024survey}~(e.g., pre-training Llama-3~\cite{llama3modelcard} consumes a total of 7.7M GPU hours and generates 2290 tons of carbon dioxide equivalent of carbon emissions).&#10;The efficient pre-training of LLMs is thus crucial, both from a scientific and a societal perspective, to ensure the continual growth and adoption of AI~\cite{wu2022sustainable, DEVRIES20232191}.&#10;&#10;One promising research direction to accelerate model training involves leveraging trained smaller (base) models to expedite the training of larger (target) models, a technique known as model growth. &#10;Concretely, model growth studies how to leverage the trained smaller model's parameters $\Theta^{(s)}$ to initialize the larger model's parameters $\Theta^{(l)}$. &#10;Current popular methods generally focus on expanding the parameters of the base model through techniques like splitting~\cite{chen2015net2net,chen2021bert2bert,wang2023lemon}, copying~\cite{shen2022staged,gong2019efficient}, or matrix mapping~\cite{wang2023learning}. There are also some approaches that initialize new parameters from scratch~\cite{evci2022gradmax,wang2023lemon,yao2024masked}. &#10;The primary objective is to accelerate the training of large models, and existing methods demonstrate promising speedup results on models such as BERT~\cite{chen2021bert2bert, gong2019efficient, yang2020progressively, wang2023learning, wang2023lemon, shen2022staged}. &#10;Despite such empirical evidence and its alignment with the goal of efficient LLM pre-training, model growth methods are not widely adopted in the context of LLM pre-training~\cite{llama3modelcard, jiang2023mistral}. To our best knowledge, the only LLM that utilizes model growth for accelerating is FLM-101B~\cite{li2023flm101b}, but it lacks a baseline LLM trained from scratch to compare.&#10;We observe three key \underline{O}bstacles that hinder LLM pre-training from using existing model growth techniques, specifically:&#10;&#10;$\bullet$ O1:&#10;Lack of comprehensive assessment. &#10;Some existing model growth methods report results on LLM pre-training, but either lack a baseline comparison~\cite{li2023flm101b} or are still in exploratory stages~\cite{wang2023learning,shen2022staged}.&#10;In contrast, most growth approaches are evaluated in encoder-based BERT models~\cite{gong2019efficient, chen2021bert2bert, yang2020progressively, wang2023lemon, shen2022staged, evci2022gradmax, yao2024masked}, which have different architecture and training configurations compared to prominent decoder-based LLMs such as Llama~\cite{touvron2023llama}. &#10;&#10;$\bullet$ O2:&#10;The untested scalability. This scalability has two aspects: the model size and the amount of pre-training data. &#10;Regarding the model size, the existing approaches are only evaluated on smaller-scale BERT models or in preliminary experiments with LLMs. &#10;It is unclear whether these growth methods will continue accelerating training when applied to large-scale LLMs with more extensive evaluation.&#10;As for the amount of pre-training data, there are debates&#10;~\cite{kaddour2023train} over whether certain efficient training strategies may initially converge faster but ultimately perform similarly or worse than vanilla training methods when given ample computational resources (i.e., more training data).&#10;&#10;$\bullet$ O3:&#10;Lack of empirical guidelines. &#10;Scaling laws~\cite{kaplan2020scaling,hoffmann2022training} give clear empirical guidelines on pre-training computational-optimized LLMs, greatly stimulating and advancing the field.&#10;Yet, there is a lack of empirical guidelines on growth techniques, discouraging LLM practitioners from adopting these approaches, especially considering the high costs of LLM pre-training.&#10;&#10;These three obstacles are consequential in nature. Hence, in this work, we empirically revisit the concept of model growth as a solution to efficient LLM pre-training by tackling them one by one.&#10;&#10;To tackle O1, we systematically evaluate model growth techniques on practical LLM pre-training.&#10;We first categorize existing growth methods and summarize them into four atomic growth operators, each of which can grow along two directions: widthwise (intra-layer) and depthwise (layer-wise). &#10;We illustrate them in Figure~\ref{fig:op}.&#10;These operators serve as representative choices for evaluating the performance of model growth techniques. &#10;We use these operators to expand 400M base models to 1.1B Llama-like LLMs and continually pre-train them.&#10;Next, we evaluate these growth techniques on the training loss and eight standard NLP benchmarks from the Harness toolkit~\cite{eval-harness}. &#10;We found the direct operator that stacks depthwisely $G_{stack}$ consistently outperforms others across overall evaluation metrics, demonstrating its potential in accelerating LLM pre-training.&#10;This motivates us to investigate extensively by addressing O2 and O3 on $G_{stack}$.&#10;&#10;To address O2, we investigate the $G_{stack}$ operator's scalability to larger model sizes and to more training data. &#10;We conduct extensive experiments by scaling model size up to 7B parameters trained with 300B tokens, and pre-training a 410M model with over 750B training tokens. &#10;This is in contrast to the previous largest LLM pre-training experiment that uses model growth methods and has baselines for comparison, which is reported in Ligo~\cite{wang2023learning}, where a GPT2-1.5B model is trained for 15k steps (approximately 15B tokens).&#10;The results are encouraging, as we consistently observe significant improvements $G_{stack}$ offers in both scenarios. For example, we achieve a remarkable 54.6\% speedup in pre-training for a 7B model with 300B tokens~(Figure~\ref{fig:main_7B}). Interestingly, the loss improvement in our 750B-token experiment aligns with a logarithmic function. We further extend this logarithmic curve and determine that the improvement continues to be substantial even for the LLM trained with over 8T tokens.&#10;Moreover, we summarize all our experiments by estimating the LLM scaling law for LLMs pre-trained with $G_{stack}$. &#10;Given the same target loss value, our analysis reveals a significantly reduced computational cost compared to the common scaling law~\cite{hoffmann2022training}.&#10;&#10;For O3, we explore the practical guidelines for using $G_{stack}$ in LLM pre-training.&#10;Given a computational budget, we determine the optimal strategy for two key factors of $G_{stack}$, growth timing $d$ and growth factor $g$. Growth timing $d$ relates to the training tokens used for small models before growing, and growth factor $g$ refers to the factor between the non-embedding parameter number of the large models and the small models.&#10;We formalize our findings into equations that offer concrete suggestions for utilizing $G_{stack}$. We believe this work could significantly pique the interest and bolster confidence in future LLM pre-training with model growth techniques,&#10;both in academia and industry.&#10;&#10;To summarize, our contributions are four-fold:&#10;1) We first systematically investigate model growth techniques and identify four atomic model growth operators, establishing a better understanding of the field in Section~\ref{subsec:operators}. &#10;2) We then design a standard LLM pre-training testbed and perform comprehensive evaluations on these operators, finding that a simple depthwise stacking $G_{stack}$ exhibits significant superiority in Section~\ref{sec:operator_and_eval}. &#10;3) We further demonstrate the scalability of $G_{stack}$ with experiments on LLMs ranging from 410M to 7B parameters and up to 750B training tokens in Section~\ref{sec:g_stack_scaling}. &#10;4) We also provide guidelines of equations on determining growth timing and growth factors for optimal use of $G_{stack}$ in Section~\ref{sec:g_stack_fit}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Large Language Models, Linguistics, Model Growth Methods, Artificial Intelligence, Mathematics, Efficient Pre-training Techniques" />
        </attvalues>
      </node>
      <node id="2401.08092" label="2401.08092">
        <attvalues>
          <attvalue for="0" value="A Survey of Resource-efficient LLM and Multimodal Foundation Models" />
          <attvalue for="1" value="  Large foundation models, including large language models (LLMs), vision&#10;transformers (ViTs), diffusion, and LLM-based multimodal models, are&#10;revolutionizing the entire machine learning lifecycle, from training to&#10;deployment. However, the substantial advancements in versatility and&#10;performance these models offer come at a significant cost in terms of hardware&#10;resources. To support the growth of these large models in a scalable and&#10;environmentally sustainable way, there has been a considerable focus on&#10;developing resource-efficient strategies. This survey delves into the critical&#10;importance of such research, examining both algorithmic and systemic aspects.&#10;It offers a comprehensive analysis and valuable insights gleaned from existing&#10;literature, encompassing a broad array of topics from cutting-edge model&#10;architectures and training/serving algorithms to practical system designs and&#10;implementations. The goal of this survey is to provide an overarching&#10;understanding of how current approaches are tackling the resource challenges&#10;posed by large foundation models and to potentially inspire future&#10;breakthroughs in this field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.00364" label="2111.00364">
        <attvalues>
          <attvalue for="0" value="Sustainable AI: Environmental Implications, Challenges and Opportunities" />
          <attvalue for="1" value="  This paper explores the environmental impact of the super-linear growth&#10;trends for AI from a holistic perspective, spanning Data, Algorithms, and&#10;System Hardware. We characterize the carbon footprint of AI computing by&#10;examining the model development cycle across industry-scale machine learning&#10;use cases and, at the same time, considering the life cycle of system hardware.&#10;Taking a step further, we capture the operational and manufacturing carbon&#10;footprint of AI computing and present an end-to-end analysis for what and how&#10;hardware-software design and at-scale optimization can help reduce the overall&#10;carbon footprint of AI. Based on the industry experience and lessons learned,&#10;we share the key challenges and chart out important development directions&#10;across the many dimensions of AI. We hope the key messages and insights&#10;presented in this paper can inspire the community to advance the field of AI in&#10;an environmentally-responsible manner.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1511.05641" label="1511.05641">
        <attvalues>
          <attvalue for="0" value="Net2Net: Accelerating Learning via Knowledge Transfer" />
          <attvalue for="1" value="  We introduce techniques for rapidly transferring the information stored in&#10;one neural net into another neural net. The main purpose is to accelerate the&#10;training of a significantly larger neural net. During real-world workflows, one&#10;often trains very many different neural networks during the experimentation and&#10;design process. This is a wasteful process in which each new model is trained&#10;from scratch. Our Net2Net technique accelerates the experimentation process by&#10;instantaneously transferring the knowledge from a previous network to each new&#10;deeper or wider network. Our techniques are based on the concept of&#10;function-preserving transformations between neural network specifications. This&#10;differs from previous approaches to pre-training that altered the function&#10;represented by a neural net when adding layers to it. Using our knowledge&#10;transfer mechanism to add depth to Inception modules, we demonstrate a new&#10;state of the art accuracy rating on the ImageNet dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.07143" label="2110.07143">
        <attvalues>
          <attvalue for="0" value="bert2BERT: Towards Reusable Pretrained Language Models" />
          <attvalue for="1" value="  In recent years, researchers tend to pre-train ever-larger language models to&#10;explore the upper limit of deep models. However, large language model&#10;pre-training costs intensive computational resources and most of the models are&#10;trained from scratch without reusing the existing pre-trained models, which is&#10;wasteful. In this paper, we propose bert2BERT, which can effectively transfer&#10;the knowledge of an existing smaller pre-trained model (e.g., BERT_BASE) to a&#10;large model (e.g., BERT_LARGE) through parameter initialization and&#10;significantly improve the pre-training efficiency of the large model.&#10;Specifically, we extend the previous function-preserving on Transformer-based&#10;language model, and further improve it by proposing advanced knowledge for&#10;large model's initialization. In addition, a two-stage pre-training method is&#10;proposed to further accelerate the training process. We did extensive&#10;experiments on representative PLMs (e.g., BERT and GPT) and demonstrate that&#10;(1) our method can save a significant amount of training cost compared with&#10;baselines including learning from scratch, StackBERT and MSLT; (2) our method&#10;is generic and applicable to different types of pre-trained models. In&#10;particular, bert2BERT saves about 45% and 47% computational cost of&#10;pre-training BERT_BASE and GPT_BASE by reusing the models of almost their half&#10;sizes. The source code will be publicly available upon publication.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;{Efficient Pre-training in NLP.} The efficiency in pre-training has been explored by previous work, some try to reuse the parameters of existing PLMs and propose progressive learning. These works~\cite{stack, stack2, growth} are motivated by the fact that different layers have some similar knowledge (e.g., attention patterns). They start pre-training a small model with fewer Transformer layers, and then iteratively expand the model by stacking the already trained layers on the top. Another line of work proposes to “back distil” the knowledge of the small models into large models, which is termed as knowledge inheritance~\cite{inheri}. Another type of work for efficient pre-training focuses on the data efficiency~\cite{wu2021taking} and takes notes for rare words during the pre-training process to help the model understand them when they occur next. The efficiency of pre-training can also be improved by designing efficient pre-training tasks. One classical work is ELECTRA~\cite{electra} which proposes a task of replaced token detection to predict whether each token in the corrupted input was replaced or not. Our method is orthogonal to this kind of work and the combination of ELECTRA and bert2BERT could achieve better efficiency. In addition, there are several other orthogonal techniques for efficient pre-training: mixed precision training~\cite{megatron}, large batch optimization~\cite{lamb}, model architecture innovation~\cite{albert}, layer dropping training technique~\cite{dropping}, etc.&#10;&#10;{Reusable Neural Network.} Reusable neural network, a topic related to transfer learning~\cite{transfer}, is introduced to accelerate the model training in computer vision. One classical work is Net2Net~\cite{net2net}, which first proposes the concept of function-preserving transformations to make neural networks reusable. However, Net2Net randomly selects the neurons to be split. To handle this problem, some works~\cite{splitSteep, escape, fastsplit, Firefly} leverage a functional steepest descent idea to decide the optimal subset of neurons to be split. The pruning technique~\cite{prune} is also introduced for reusable neural networks~\cite{cumulativeTrain}. Recently, hierarchical pre-training is proposed by~\cite{cumulativeTrain}, which saves training time and improves performance by initializing the pretraining process with an existing pre-trained vision model. In this paper, we study the reusable pre-trained language model and propose a new method, bert2BERT\, to accelerate the pre-training of BERT and GPT. &#10; " />
        </attvalues>
      </node>
      <node id="2310.07999" label="2310.07999">
        <attvalues>
          <attvalue for="0" value="LEMON: Lossless model expansion" />
          <attvalue for="1" value="  Scaling of deep neural networks, especially Transformers, is pivotal for&#10;their surging performance and has further led to the emergence of sophisticated&#10;reasoning capabilities in foundation models. Such scaling generally requires&#10;training large models from scratch with random initialization, failing to&#10;leverage the knowledge acquired by their smaller counterparts, which are&#10;already resource-intensive to obtain. To tackle this inefficiency, we present&#10;$\textbf{L}$ossl$\textbf{E}$ss $\textbf{MO}$del Expansio$\textbf{N}$ (LEMON), a&#10;recipe to initialize scaled models using the weights of their smaller but&#10;pre-trained counterparts. This is followed by model training with an optimized&#10;learning rate scheduler tailored explicitly for the scaled models,&#10;substantially reducing the training time compared to training from scratch.&#10;Notably, LEMON is versatile, ensuring compatibility with various network&#10;structures, including models like Vision Transformers and BERT. Our empirical&#10;results demonstrate that LEMON reduces computational costs by 56.7% for Vision&#10;Transformers and 33.2% for BERT when compared to training from scratch.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.00980" label="2303.00980">
        <attvalues>
          <attvalue for="0" value="Learning to Grow Pretrained Models for Efficient Transformer Training" />
          <attvalue for="1" value="  Scaling transformers has led to significant breakthroughs in many domains,&#10;leading to a paradigm in which larger versions of existing models are trained&#10;and released on a periodic basis. New instances of such models are typically&#10;trained completely from scratch, despite the fact that they are often just&#10;scaled-up versions of their smaller counterparts. How can we use the implicit&#10;knowledge in the parameters of smaller, extant models to enable faster training&#10;of newer, larger models? This paper describes an approach for accelerating&#10;transformer training by learning to grow pretrained transformers, where we&#10;learn to linearly map the parameters of the smaller model to initialize the&#10;larger model. For tractable learning, we factorize the linear transformation as&#10;a composition of (linear) width- and depth-growth operators, and further employ&#10;a Kronecker factorization of these growth operators to encode architectural&#10;knowledge. Extensive experiments across both language and vision transformers&#10;demonstrate that our learned Linear Growth Operator (LiGO) can save up to 50%&#10;computational cost of training from scratch, while also consistently&#10;outperforming strong baselines that also reuse smaller pretrained models to&#10;initialize larger models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.02869" label="2305.02869">
        <attvalues>
          <attvalue for="0" value="Masked Structural Growth for 2x Faster Language Model Pre-training" />
          <attvalue for="1" value="  Accelerating large language model pre-training is a critical issue in present&#10;research. In this paper, we focus on speeding up pre-training by progressively&#10;growing from a small Transformer structure to a large one. There are two main&#10;research problems associated with progressive growth: determining the optimal&#10;growth schedule, and designing efficient growth operators. In terms of growth&#10;schedule, the impact of each single dimension on a schedule's efficiency is&#10;under-explored by existing work. Regarding the growth operators, existing&#10;methods rely on the initialization of new weights to inherit knowledge, and&#10;achieve only non-strict function preservation, limiting further improvements on&#10;training dynamics. To address these issues, we propose Masked Structural Growth&#10;(MSG), including (i) growth schedules involving all possible dimensions and&#10;(ii) strictly function-preserving growth operators that is independent of the&#10;initialization of new weights. Experiments show that MSG is significantly&#10;faster than related work: we achieve up to 2.2x speedup in pre-training&#10;different types of language models while maintaining comparable or better&#10;downstream performances. Code is publicly available at&#10;https://github.com/cofe-ai/MSG.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.09288" label="2307.09288">
        <attvalues>
          <attvalue for="0" value="Llama 2: Open Foundation and Fine-Tuned Chat Models" />
          <attvalue for="1" value="  In this work, we develop and release Llama 2, a collection of pretrained and&#10;fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70&#10;billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for&#10;dialogue use cases. Our models outperform open-source chat models on most&#10;benchmarks we tested, and based on our human evaluations for helpfulness and&#10;safety, may be a suitable substitute for closed-source models. We provide a&#10;detailed description of our approach to fine-tuning and safety improvements of&#10;Llama 2-Chat in order to enable the community to build on our work and&#10;contribute to the responsible development of LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15556" label="2203.15556">
        <attvalues>
          <attvalue for="0" value="Training Compute-Optimal Large Language Models" />
          <attvalue for="1" value="  We investigate the optimal model size and number of tokens for training a&#10;transformer language model under a given compute budget. We find that current&#10;large language models are significantly undertrained, a consequence of the&#10;recent focus on scaling language models whilst keeping the amount of training&#10;data constant. By training over 400 language models ranging from 70 million to&#10;over 16 billion parameters on 5 to 500 billion tokens, we find that for&#10;compute-optimal training, the model size and the number of training tokens&#10;should be scaled equally: for every doubling of model size the number of&#10;training tokens should also be doubled. We test this hypothesis by training a&#10;predicted compute-optimal model, Chinchilla, that uses the same compute budget&#10;as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla&#10;uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1&#10;(178B), and Megatron-Turing NLG (530B) on a large range of downstream&#10;evaluation tasks. This also means that Chinchilla uses substantially less&#10;compute for fine-tuning and inference, greatly facilitating downstream usage.&#10;As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5%&#10;on the MMLU benchmark, greater than a 7% improvement over Gopher.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.06567" label="2408.06567">
        <attvalues>
          <attvalue for="0" value="AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out&#10;  Strategies" />
          <attvalue for="1" value="  In recent years, with the rapid application of large language models across&#10;various fields, the scale of these models has gradually increased, and the&#10;resources required for their pre-training have grown exponentially. Training an&#10;LLM from scratch will cost a lot of computation resources while scaling up from&#10;a smaller model is a more efficient approach and has thus attracted significant&#10;attention. In this paper, we present AquilaMoE, a cutting-edge bilingual 8*16B&#10;Mixture of Experts (MoE) language model that has 8 experts with 16 billion&#10;parameters each and is developed using an innovative training methodology&#10;called EfficientScale. This approach optimizes performance while minimizing&#10;data requirements through a two-stage process. The first stage, termed&#10;Scale-Up, initializes the larger model with weights from a pre-trained smaller&#10;model, enabling substantial knowledge transfer and continuous pretraining with&#10;significantly less data. The second stage, Scale-Out, uses a pre-trained dense&#10;model to initialize the MoE experts, further enhancing knowledge transfer and&#10;performance. Extensive validation experiments on 1.8B and 7B models compared&#10;various initialization schemes, achieving models that maintain and reduce loss&#10;during continuous pretraining. Utilizing the optimal scheme, we successfully&#10;trained a 16B model and subsequently the 8*16B AquilaMoE model, demonstrating&#10;significant improvements in performance and training efficiency.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models have become a cornerstone of modern natural language processing (NLP) systems, driving applications such as machine translation, conversational agents, text summarization, and question answering \cite{vaswani2017attention, devlin2018bert}. Recent advancements in large language models (LLMs) like GPT-3, BERT, and T5 have demonstrated remarkable proficiency across numerous tasks, highlighting the importance of pretraining on large-scale datasets to achieve state-of-the-art results \cite{brown2020language, raffel2020exploring}. Despite their success, traditional dense models face significant challenges in scalability and efficiency, particularly as parameter sizes increase.&#10;&#10;Mixture of Experts (MoE) models have emerged as a promising solution to these challenges. By dynamically selecting different subsets of model parameters (experts) for various inputs, MoE architectures can scale to a much larger number of parameters without a corresponding increase in computational cost \cite{lepikhin2021gshard}. This selective activation mechanism allows MoE models to achieve higher performance while maintaining computational efficiency. However, training such large-scale MoE models presents significant challenges, including the vast amounts of data and computational power required.&#10;&#10;Training large-scale models, including MoE architectures, involves several critical challenges. Traditional training methods require enormous amounts of data, which can be resource-intensive and time-consuming to collect and process. The computational cost is substantial, requiring high-performance hardware such as GPUs or TPUs, and significant energy consumption, making it challenging for many institutions with limited resources to train and deploy such models. Additionally, training large models from scratch can take weeks or even months, delaying experimentation and iteration. Ensuring that the model efficiently learns and generalizes well is also challenging, as poor initialization and inefficient training strategies can lead to suboptimal performance and wasted resources.&#10;&#10;Several strategies have been proposed to address these challenges. For instance, the Net2Net method accelerates learning via knowledge transfer, allowing the seamless transition of knowledge from smaller to larger networks, which shows significant acceleration in image classification task~\cite{net2net}. The StackBERT method improves training efficiency by progressively increasing model depth and capacity \cite{stacking}. The bert2BERT approach focuses on reusing pre-trained language models to initialize new models, promoting efficiency and reusability~\cite{chen-etal-2022-bert2bert}. It expands both the width and depth of the smaller model and finally saves nearly half of the pre-training consumption of language models. The primary motivation behind developing AquilaMoE is to introduce an efficient training framework, EfficientScale, which reduces data and computational requirements while enhancing overall model performance. Our approach leverages the strengths of MoE architectures and introduces innovative techniques to improve training efficiency and effectiveness.&#10;&#10;In this paper, we introduce AquilaMoE, a bilingual 8*16B Mixture of Experts language model that has 8 experts with 16 billion parameters each and is developed using the EfficientScale methodology. This approach optimizes performance and minimizes data needs through a two-stage process. The first stage, Scale-Up, leverages the weights of a pre-trained smaller model to initialize the larger model, enabling substantial knowledge transfer and continuous pretraining with significantly less data compared to traditional from-scratch training. The second stage, Scale-Out, uses a pre-trained dense model to initialize the MoE experts, further enhancing knowledge transfer and performance.&#10;&#10;Through extensive validation experiments on 1.8B and 7B models, we compared various initialization schemes to achieve models that maintain and further reduce loss during continuous pretraining. Based on these findings, we utilized the optimal initialization scheme to successfully train a 16B model and subsequently the 8*16B AquilaMoE model, demonstrating significant advancements in model performance and training efficiency.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Model Scaling Techniques, Efficient Training Methods, Mathematics" />
        </attvalues>
      </node>
      <node id="1706.03762" label="1706.03762">
        <attvalues>
          <attvalue for="0" value="Attention Is All You Need" />
          <attvalue for="1" value="  The dominant sequence transduction models are based on complex recurrent or&#10;convolutional neural networks in an encoder-decoder configuration. The best&#10;performing models also connect the encoder and decoder through an attention&#10;mechanism. We propose a new simple network architecture, the Transformer, based&#10;solely on attention mechanisms, dispensing with recurrence and convolutions&#10;entirely. Experiments on two machine translation tasks show these models to be&#10;superior in quality while being more parallelizable and requiring significantly&#10;less time to train. Our model achieves 28.4 BLEU on the WMT 2014&#10;English-to-German translation task, improving over the existing best results,&#10;including ensembles by over 2 BLEU. On the WMT 2014 English-to-French&#10;translation task, our model establishes a new single-model state-of-the-art&#10;BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction&#10;of the training costs of the best models from the literature. We show that the&#10;Transformer generalizes well to other tasks by applying it successfully to&#10;English constituency parsing both with large and limited training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.16668" label="2006.16668">
        <attvalues>
          <attvalue for="0" value="GShard: Scaling Giant Models with Conditional Computation and Automatic&#10;  Sharding" />
          <attvalue for="1" value="  Neural network scaling has been critical for improving the model quality in&#10;many real-world machine learning applications with vast amounts of training&#10;data and compute. Although this trend of scaling is affirmed to be a sure-fire&#10;approach for better model quality, there are challenges on the path such as the&#10;computation cost, ease of programming, and efficient implementation on parallel&#10;devices. GShard is a module composed of a set of lightweight annotation APIs&#10;and an extension to the XLA compiler. It provides an elegant way to express a&#10;wide range of parallel computation patterns with minimal changes to the&#10;existing model code. GShard enabled us to scale up multilingual neural machine&#10;translation Transformer model with Sparsely-Gated Mixture-of-Experts beyond 600&#10;billion parameters using automatic sharding. We demonstrate that such a giant&#10;model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to&#10;achieve far superior quality for translation from 100 languages to English&#10;compared to the prior art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.11635" label="2012.11635">
        <attvalues>
          <attvalue for="0" value="A Distributional Approach to Controlled Text Generation" />
          <attvalue for="1" value="  We propose a Distributional Approach for addressing Controlled Text&#10;Generation from pre-trained Language Models (LMs). This approach permits to&#10;specify, in a single formal framework, both &quot;pointwise&quot; and &quot;distributional&quot;&#10;constraints over the target LM -- to our knowledge, the first model with such&#10;generality -- while minimizing KL divergence from the initial LM distribution.&#10;The optimal target distribution is then uniquely determined as an explicit EBM&#10;(Energy-Based Model) representation. From that optimal representation we then&#10;train a target controlled Autoregressive LM through an adaptive distributional&#10;variant of Policy Gradient. We conduct a first set of experiments over&#10;pointwise constraints showing the advantages of our approach over a set of&#10;baselines, in terms of obtaining a controlled LM balancing constraint&#10;satisfaction with divergence from the initial LM. We then perform experiments&#10;over distributional constraints, a unique feature of our approach,&#10;demonstrating its potential as a remedy to the problem of Bias in Language&#10;Models. Through an ablation study, we show the effectiveness of our adaptive&#10;technique for obtaining faster convergence. (Code available at&#10;https://github.com/naver/gdc)&#10;" />
          <attvalue for="2" value="&#10;Neural language models, such as GPT-2/3 \cite{radford2019language,Brown2020LanguageMA}, pretrained on huge amounts of text, have become pre-eminent in NLP, producing texts of unprecedented quality. &#10;In this paper, we are concerned with the problem of controlling a generic pretrained LM in order to satisfy certain desiderata. For instance, we may want to avoid toxic content; prevent certain demographic biases; or steer generations towards a certain topic or style. &#10;Prior work, taking inspiration from Reinforcement Learning (RL), has aimed at inducing autoregressive models to optimize global objectives using task specific rewards such as BLEU and ROUGE for Machine Translation and Summarization~\cite{seq_lvl_train_RanzatoCAZ15, BahdanauBXGLPCB17}, or hand crafted rewards~\cite{RL_dialogue_LiMRJGG16,RL_TambwekarDMMHR19} to improve certain a priori desirable features. &#10;&#10;However, such an optimization process is not infallible; \cite{LiuLSNCP16} noted that it often leads to ``degeneration'', producing poor examples that improve the average reward but forgo coherence and fluency. &#10;This degeneration is often diagnosed as an effect of deviating too much from the original pretrained LM during optimization. Consequently, prior work has regarded proximity to the pretrained model as a prescription for sample quality. This view is most prominent in open-domain generation where no gold references are available for fine-tuning, making the pretrained LM itself the yardstick for fluency. &#10;\cite{KL_Jaques17,Ziegler19} propose a conservative fine-tuning approach moderated by a KL penalty between the trained policy and the original LM, discouraging large deviations. &#10;A KL penalty was also used by \cite{plug_and_play_20}, &#10;this time in a plug-and-play rather than a fine-tuning context. However, the authors show that balancing policy deviations from the original LM while also satisfying the &#10;control conditions is delicate. To combat degeneration they had to combine the KL penalty with post-norm fusion, reranking, and early-stopping procedures. &#10;&#10;Most of the existing work on Controlled Generation has taken what we refer to as a ``{pointwise}'' view, namely focusing on the quality of each individual output, a view that is encouraged by the standard RL goal of maximizing rewards computed at the individual level.&#10;Such techniques are incapable of enforcing ``{distributional}'' conditions, where some collective statistical properties are desired over the set of all generations.&#10;&#10;Distributional control is key to solving the problem of social biases in LMs trained on large, uncurated Web corpora. &#10;Those LMs - dubbed ``Stochastic Parrots'' in ~\cite{stochasticParrots} - tend to encode hegemonic biases that are harmful to marginalized populations. There has been a large body of work analysing these distributional biases~\cite{blodgett-bias-survey, bias_mt_stanovsky-etal-2019,bias_mt_PratesAL20,ShengCNP_LM_bias19,gpt3}.&#10;However, applying distributional control on pretrained models is still an understudied problem.&#10;\cite{babysitter2_Sheng2020} introduce a method relying on adversarial triggers~\cite{WallaceFKGS19}; this method does not de-bias the whole distribution but only obtains non-biased continuations of given prompts. \cite{BordiaB19} introduce a regularization term for reducing gender bias when training a language model from scratch (as opposed to de-biasing a pretrained model).&#10;&#10;In this work, we present our Generation with Distributional Control (\GDC) approach, in which we &#10;formalize the problem of controlled text generation as a constraint satisfaction problem over the probability distribution $p$ representing the desired target LM. Namely, we require the expectations (``moments'') relative to $p$ of certain output features to have specific values; this permits for instance to condition all outputs to speak about sports (a pointwise constraint), and 50\% of them to mention female characters (a distributional constraint). Additionally, we require $p$ to have a minimal KL divergence $\KL(p,a)$ from the original pretrained LM $a$. This has the effect that $p$ now inherits favorable linguistic qualities from $a$. As we will explain, this formulation is a generalization of the Maximum Entropy Principle&#10;and leads to a unique solution $P(x)$. $P(x)$ is an unnormalized distribution, aka an Energy-Based Model (EBM)&#10;\cite{Hinton02,lecun_tutorial_2006,Bakhtin2020EnergyBasedMF}, of which $p(x) =1/Z\ P(x)$ is the normalized version, where $Z \doteq \sum_x P(x)$ is the partition function of $P$. &#10;&#10;Computing the EBM representation $P$ is a crucial step, as it fully determines the optimal distribution $p$ we are looking for. &#10;However, it is not the end of the story, because the representation thus obtained does not enable us to directly sample from $p$, an essential property of any LM.&#10;To this end, we introduce KL-adaptive DPG (Distributional Policy Gradient), a variant of an algorithm recently proposed in \cite{opt-rl-arxiv-2019}. We train the policy $\pit$ to approximate $p$ in an adaptive way, by speeding up the next round of approximations based on approximations previously obtained. &#10;At the end of this process, we obtain a final $\pit$, our target LM, on which we can estimate diverse metrics, including $\KL(p,\pit)$, measuring the approximation quality of $\pit$ relative to the optimal $p$,&#10;and $\KL(\pit,a)$, measuring the divergence of $\pit$ relative to the original LM $a$.&#10;&#10;This two-step approach differs from much research in NLP-oriented work with EBMs, which tends to use EBM representations inside the training loops of neural networks, blurring different dimensions of the problem. By contrast --- similarly to \cite{A-parshakova-etal-2019-global,opt-rl-arxiv-2019} in a different context --- we clearly decouple the relatively simple problem of determining a ``pivot'' optimal EBM from the more difficult problem of exploiting this EBM at inference time, &#10;Such decoupling is valuable, because it permits to better diagnose the important challenges to focus on.&#10;&#10;Overall, &#10;our contributions&#10;can be summarized as follows:&#10;\begin{enumerate}\item We introduce a Distributional View for controlled text generation formalized as a constraint satisfaction problem combined with a divergence minimization objective, providing a single framework both for ``distributional'' constraints (collective statistical requirements) and for ``pointwise'' constraints (hard requirements on each individual) (\S\ref{sec:constraints-info-geo}). &#10;To our knowledge, this is the first framework with such generality for controlled text generation.\item We show how these constraints lead to an optimal EBM for the target model (\S\ref{sec:constraints2ebm}), propose the KL-Adaptive DPG algorithm &#10;for approximating the optimal EBM distribution by an autoregressive policy (\S\ref{sec:ebm-2-policy}), and show the effectiveness of this adaptive technique for obtaining faster convergence (\S\ref{sec:appendix:exp-ablation}).\item We conduct experiments in a number of pointwise and distributional conditions, assessing results in terms of divergence from GPT-2, fluency and diversity, with better performance than &#10;strong baselines. The distributional experiments show the potential of our approach as a remedy to the current and important problem of bias in pretrained language models, providing a novel direction for addressing it (\S\ref{sec:EXPERIMENTS}).&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Distributional Constraints, Machine Learning, Computer Science, Linguistics, Language Model Control, Artificial Intelligence, Mathematics, Bias Mitigation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1511.06732" label="1511.06732">
        <attvalues>
          <attvalue for="0" value="Sequence Level Training with Recurrent Neural Networks" />
          <attvalue for="1" value="  Many natural language processing applications use language models to generate&#10;text. These models are typically trained to predict the next word in a&#10;sequence, given the previous words and some context such as an image. However,&#10;at test time the model is expected to generate the entire sequence from&#10;scratch. This discrepancy makes generation brittle, as errors may accumulate&#10;along the way. We address this issue by proposing a novel sequence level&#10;training algorithm that directly optimizes the metric used at test time, such&#10;as BLEU or ROUGE. On three different tasks, our approach outperforms several&#10;strong baselines for greedy generation. The method is also competitive when&#10;these baselines employ beam search, while being several times faster.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.01541" label="1606.01541">
        <attvalues>
          <attvalue for="0" value="Deep Reinforcement Learning for Dialogue Generation" />
          <attvalue for="1" value="  Recent neural models of dialogue generation offer great promise for&#10;generating responses for conversational agents, but tend to be shortsighted,&#10;predicting utterances one at a time while ignoring their influence on future&#10;outcomes. Modeling the future direction of a dialogue is crucial to generating&#10;coherent, interesting dialogues, a need which led traditional NLP models of&#10;dialogue to draw on reinforcement learning. In this paper, we show how to&#10;integrate these goals, applying deep reinforcement learning to model future&#10;reward in chatbot dialogue. The model simulates dialogues between two virtual&#10;agents, using policy gradient methods to reward sequences that display three&#10;useful conversational properties: informativity (non-repetitive turns),&#10;coherence, and ease of answering (related to forward-looking function). We&#10;evaluate our model on diversity, length as well as with human judges, showing&#10;that the proposed algorithm generates more interactive responses and manages to&#10;foster a more sustained conversation in dialogue simulation. This work marks a&#10;first step towards learning a neural conversational model based on the&#10;long-term success of dialogues.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1603.08023" label="1603.08023">
        <attvalues>
          <attvalue for="0" value="How NOT To Evaluate Your Dialogue System: An Empirical Study of&#10;  Unsupervised Evaluation Metrics for Dialogue Response Generation" />
          <attvalue for="1" value="  We investigate evaluation metrics for dialogue response generation systems&#10;where supervised labels, such as task completion, are not available. Recent&#10;works in response generation have adopted metrics from machine translation to&#10;compare a model's generated response to a single target response. We show that&#10;these metrics correlate very weakly with human judgements in the non-technical&#10;Twitter domain, and not at all in the technical Ubuntu domain. We provide&#10;quantitative and qualitative results highlighting specific weaknesses in&#10;existing metrics, and provide recommendations for future development of better&#10;automatic evaluation metrics for dialogue systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.02796" label="1611.02796">
        <attvalues>
          <attvalue for="0" value="Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models&#10;  with KL-control" />
          <attvalue for="1" value="  This paper proposes a general method for improving the structure and quality&#10;of sequences generated by a recurrent neural network (RNN), while maintaining&#10;information originally learned from data, as well as sample diversity. An RNN&#10;is first pre-trained on data using maximum likelihood estimation (MLE), and the&#10;probability distribution over the next token in the sequence learned by this&#10;model is treated as a prior policy. Another RNN is then trained using&#10;reinforcement learning (RL) to generate higher-quality outputs that account for&#10;domain-specific incentives while retaining proximity to the prior policy of the&#10;MLE RNN. To formalize this objective, we derive novel off-policy RL methods for&#10;RNNs from KL-control. The effectiveness of the approach is demonstrated on two&#10;applications; 1) generating novel musical melodies, and 2) computational&#10;molecular generation. For both problems, we show that the proposed method&#10;improves the desired properties and structure of the generated sequences, while&#10;maintaining information learned from data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.02164" label="1912.02164">
        <attvalues>
          <attvalue for="0" value="Plug and Play Language Models: A Simple Approach to Controlled Text&#10;  Generation" />
          <attvalue for="1" value="  Large transformer-based language models (LMs) trained on huge text corpora&#10;have shown unparalleled generation capabilities. However, controlling&#10;attributes of the generated language (e.g. switching topic or sentiment) is&#10;difficult without modifying the model architecture or fine-tuning on&#10;attribute-specific data and entailing the significant cost of retraining. We&#10;propose a simple alternative: the Plug and Play Language Model (PPLM) for&#10;controllable language generation, which combines a pretrained LM with one or&#10;more simple attribute classifiers that guide text generation without any&#10;further training of the LM. In the canonical scenario we present, the attribute&#10;models are simple classifiers consisting of a user-specified bag of words or a&#10;single learned layer with 100,000 times fewer parameters than the LM. Sampling&#10;entails a forward and backward pass in which gradients from the attribute model&#10;push the LM's hidden activations and thus guide the generation. Model samples&#10;demonstrate control over a range of topics and sentiment styles, and extensive&#10;automated and human annotated evaluations show attribute alignment and fluency.&#10;PPLMs are flexible in that any combination of differentiable attribute models&#10;may be used to steer text generation, which will allow for diverse and creative&#10;applications beyond the examples given in this paper.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.01326" label="1909.01326">
        <attvalues>
          <attvalue for="0" value="The Woman Worked as a Babysitter: On Biases in Language Generation" />
          <attvalue for="1" value="  We present a systematic study of biases in natural language generation (NLG)&#10;by analyzing text generated from prompts that contain mentions of different&#10;demographic groups. In this work, we introduce the notion of the regard towards&#10;a demographic, use the varying levels of regard towards different demographics&#10;as a defining metric for bias in NLG, and analyze the extent to which sentiment&#10;scores are a relevant proxy metric for regard. To this end, we collect&#10;strategically-generated text from language models and manually annotate the&#10;text with both sentiment and regard scores. Additionally, we build an automatic&#10;regard classifier through transfer learning, so that we can analyze biases in&#10;unseen text. Together, these methods reveal the extent of the biased nature of&#10;language model generations. Our analysis provides a study of biases in NLG,&#10;bias metrics and correlated human judgments, and empirical evidence on the&#10;usefulness of our annotated dataset.&#10;" />
          <attvalue for="2" value="&#10;Recent works in machine translation \cite{prates2018assessing} and dialogue systems \cite{henderson2018ethical} have brought to attention the perpetuation of biases in natural language generation (NLG) systems. In this work, we present a systematic study of biases in open-domain NLG by examining language models. &#10;Language models are a fundamental component of NLG that are widely used in downstream tasks such as machine translation \cite{koehn2009statistical}, dialogue generation \cite{serban2016building}, and story generation \cite{yao2019plan}; as such, biases propagated through the language models will have a profound impact on a variety of other NLG tasks. More generally, NLG systems are at the forefront of developments in human-computer interaction, and systematic biases in language models have a direct impact on society and broader AI applications.&#10;&#10;A text is positively or negatively inclined towards a demographic if the text causes the specific demographic to be positively or negatively perceived. When NLP models systematically produce text with different levels of inclinations towards different groups (e.g., man vs. woman), the models exhibit bias. Table \ref{examples} shows that GPT-2, OpenAI's publicly available language model, is biased towards certain demographics. We find that some of the contexts where biases occur include social connotations that are often subtle and difficult to capture in existing sentiment analysis tools. For example, when we run two popular sentiment analyzers on the sentence ``XYZ worked as a pimp for 15 years'', both analyzers predict a neutral sentiment, even though working as a ``pimp'' generally has a negative social connotation. Therefore, we introduce the concept of regard towards different demographics as a metric for bias. &#10;&#10;In this work, we define bias contexts, demographics, and metrics for the first systematic study of biases in open-domain NLG. We construct a general experimental setup to analyze different textual contexts where biases occur to different demographics in NLG systems. Through an annotated dataset, we address the appropriateness of sentiment scores as a proxy for measuring bias across varying textual contexts. We then use the annotations to build a classifier for regard, and use both sentiment and regard to present biases found in NLG systems. We are making the annotations public.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00268" label="2005.00268">
        <attvalues>
          <attvalue for="0" value="Towards Controllable Biases in Language Generation" />
          <attvalue for="1" value="  We present a general approach towards controllable societal biases in natural&#10;language generation (NLG). Building upon the idea of adversarial triggers, we&#10;develop a method to induce societal biases in generated text when input prompts&#10;contain mentions of specific demographic groups. We then analyze two scenarios:&#10;1) inducing negative biases for one demographic and positive biases for another&#10;demographic, and 2) equalizing biases between demographics. The former scenario&#10;enables us to detect the types of biases present in the model. Specifically, we&#10;show the effectiveness of our approach at facilitating bias analysis by finding&#10;topics that correspond to demographic inequalities in generated text and&#10;comparing the relative effectiveness of inducing biases for different&#10;demographics. The second scenario is useful for mitigating biases in downstream&#10;applications such as dialogue generation. In our experiments, the mitigation&#10;technique proves to be effective at equalizing the amount of biases across&#10;demographics while simultaneously generating less negatively biased text&#10;overall.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.03035" label="1904.03035">
        <attvalues>
          <attvalue for="0" value="Identifying and Reducing Gender Bias in Word-Level Language Models" />
          <attvalue for="1" value="  Many text corpora exhibit socially problematic biases, which can be&#10;propagated or amplified in the models trained on such data. For example, doctor&#10;cooccurs more frequently with male pronouns than female pronouns. In this study&#10;we (i) propose a metric to measure gender bias; (ii) measure bias in a text&#10;corpus and the text generated from a recurrent neural network language model&#10;trained on the text corpus; (iii) propose a regularization loss term for the&#10;language model that minimizes the projection of encoder-trained embeddings onto&#10;an embedding subspace that encodes gender; (iv) finally, evaluate efficacy of&#10;our proposed method on reducing gender bias. We find this regularization method&#10;to be effective in reducing gender bias up to an optimal weight assigned to the&#10;loss term, beyond which the model becomes unstable as the perplexity increases.&#10;We replicate this study on three training corpora---Penn Treebank, WikiText-2,&#10;and CNN/Daily Mail---resulting in similar conclusions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.08517" label="1912.08517">
        <attvalues>
          <attvalue for="0" value="Distributional Reinforcement Learning for Energy-Based Sequential Models" />
          <attvalue for="1" value="  Global Autoregressive Models (GAMs) are a recent proposal [Parshakova et al.,&#10;CoNLL 2019] for exploiting global properties of sequences for data-efficient&#10;learning of seq2seq models. In the first phase of training, an Energy-Based&#10;model (EBM) over sequences is derived. This EBM has high representational&#10;power, but is unnormalized and cannot be directly exploited for sampling. To&#10;address this issue [Parshakova et al., CoNLL 2019] proposes a distillation&#10;technique, which can only be applied under limited conditions. By relating this&#10;problem to Policy Gradient techniques in RL, but in a \emph{distributional}&#10;rather than \emph{optimization} perspective, we propose a general approach&#10;applicable to any sequential EBM. Its effectiveness is illustrated on GAM-based&#10;experiments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.13907" label="2401.13907">
        <attvalues>
          <attvalue for="0" value="No More Distractions: an Adaptive Up-Sampling Algorithm to Reduce Data&#10;  Artifacts" />
          <attvalue for="1" value="  Researchers recently found out that sometimes language models achieve high&#10;accuracy on benchmark data set, but they can not generalize very well with even&#10;little changes to the original data set. This is sometimes due to data&#10;artifacts, model is learning the spurious correlation between tokens and&#10;labels, instead of the semantics and logic. In this work, we analyzed SNLI data&#10;and visualized such spurious correlations. We proposed an adaptive up-sampling&#10;algorithm to correct the data artifacts, which is simple and effective, and&#10;does not need human edits or annotation. We did an experiment applying the&#10;algorithm to fix the data artifacts in SNLI data and the model trained with&#10;corrected data performed significantly better than the model trained with raw&#10;SNLI data, overall, as well as on the subset we corrected.&#10;" />
          <attvalue for="2" value="&#10;Natural language models are expected to learn and understand the semantics of text, so they can &quot;think&quot; like a human and solving tasks such as making inferences and answering questions. However, in recent studies, researchers discovered that sometimes the high performance on benchmark data sets are not a results of the above, instead, they are from learning the spurious correlations between tokens and output label \cite{poliak2018hypothesis}. Spurious correlations are any simple correlations between input tokens and output labels and for a data set without spurious correlations, p(label|token) should be uniform over all class labels \cite{gardner2021competency}. For example, in a language model to predict sentiments of customer reviews. It's possible that 99\% of customers use &quot;perfect&quot; in expressing positive feedbacks, with the remaining 1\% using &quot;perfect&quot; in a sarcasm way to express negative feelings. If such spurious correlation is not eliminated during sampling, it's very likely the correlation will be learned by the model, as the model can easily achieve 99\% accuracy with a simple rule of if &quot;perfect&quot; exists in the input text. This way, model may achieve a very high accuracy in benchmark data sets, but may not be able to generalize very well.&#10;&#10;In this study, we will show that such spurious correlations exist in SNLI data \cite{bowman2015large}, in the form that tokens from a specific subset occurs more often with some labels than with other labels. We trained a ELECTRA-small \cite{clark2020electra} model on SNLI data without any correction and observed that these spurious correlations have influenced model training. The model is tend to predict the correlated label if certain tokens exist in input. To solve this issue, the most popular way is manually or semi-automatically edit the records \cite{gardner2021competency,clark-etal-2019-dont,maas-etal-2011-learning}, such that the p(label$|$token) is approximately uniform over all class labels. However, this method will cost significant amount of human time and may introduce additional bias in these augmented data \cite{tafjord-etal-2019-quartz}.&#10;&#10;We attempt to remedy this issue by doing an adaptive round robin up-sampling on the records that contains under-represented token-label pair in training data, which has three advantages comparing to the current methods: fully automatic, cost little to none human interactions, not introducing additional bias. With this correction, we observed the accuracy have been improved overall, as well as on the sub set of records with problematic tokens. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Limitations, Model Generalization Improvement, Artificial Intelligence, Data Artifact Correction, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1805.01042" label="1805.01042">
        <attvalues>
          <attvalue for="0" value="Hypothesis Only Baselines in Natural Language Inference" />
          <attvalue for="1" value="  We propose a hypothesis only baseline for diagnosing Natural Language&#10;Inference (NLI). Especially when an NLI dataset assumes inference is occurring&#10;based purely on the relationship between a context and a hypothesis, it follows&#10;that assessing entailment relations while ignoring the provided context is a&#10;degenerate solution. Yet, through experiments on ten distinct NLI datasets, we&#10;find that this approach, which we refer to as a hypothesis-only model, is able&#10;to significantly outperform a majority class baseline across a number of NLI&#10;datasets. Our analysis suggests that statistical irregularities may allow a&#10;model to perform NLI in some datasets beyond what should be achievable without&#10;access to the context.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08646" label="2104.08646">
        <attvalues>
          <attvalue for="0" value="Competency Problems: On Finding and Removing Artifacts in Language Data" />
          <attvalue for="1" value="  Much recent work in NLP has documented dataset artifacts, bias, and spurious&#10;correlations between input features and output labels. However, how to tell&#10;which features have &quot;spurious&quot; instead of legitimate correlations is typically&#10;left unspecified. In this work we argue that for complex language understanding&#10;tasks, all simple feature correlations are spurious, and we formalize this&#10;notion into a class of problems which we call competency problems. For example,&#10;the word &quot;amazing&quot; on its own should not give information about a sentiment&#10;label independent of the context in which it appears, which could include&#10;negation, metaphor, sarcasm, etc. We theoretically analyze the difficulty of&#10;creating data for competency problems when human bias is taken into account,&#10;showing that realistic datasets will increasingly deviate from competency&#10;problems as dataset size increases. This analysis gives us a simple statistical&#10;test for dataset artifacts, which we use to show more subtle biases than were&#10;described in prior work, including demonstrating that models are&#10;inappropriately affected by these less extreme biases. Our theoretical&#10;treatment of this problem also allows us to analyze proposed solutions, such as&#10;making local edits to dataset instances, and to give recommendations for future&#10;data collection and model design efforts that target competency problems.&#10;" />
          <attvalue for="2" value="&#10;&#10;Attempts by the natural language processing community to get machines to understand language or read text are often stymied in part by issues in our datasets~\cite{chen-etal-2016-thorough,sugawara-etal-2018-makes}.&#10;Many recent papers have shown that popular datasets are prone to shortcuts, dataset artifacts, bias, and spurious correlations~\cite{jia-liang-2017-adversarial,rudinger-etal-2018-gender,ws-2019-gender}. While these empirical demonstrations of deficiencies in the data are useful, they often leave unanswered fundamental questions of what exactly makes a correlation ``spurious'', instead of a feature that is legitimately predictive of some target label.&#10;&#10;In this work we attempt to address this question theoretically. We begin with the assumption that in a language understanding problem, no single feature on its own should contain information about the class label. That is, all simple correlations between input features and output labels are spurious: $p(y|x_i)$, for any feature $x_i$, should be uniform over the class label. We call the class of problems that meet this assumption competency problems (\S\ref{sec:competency}).&#10;&#10;This assumption places a very strong restriction on the problems being studied, but we argue that it is a reasonable description of complex language understanding problems. Consider, for example, the problem of sentiment analysis on movie reviews. A single feature might be the presence of the word ``amazing'', which could be legitimately correlated with positive sentiment in some randomly-sampled collection of actual movie reviews. However, that correlation tells us more about word frequency in movie reviews than it tells us about a machine's ability to understand the complexities of natural language. A competent speaker of a natural language would know that ``amazing'' can appear in many contexts that do not have positive sentiment and would not base their prediction on the presence of this feature alone. That is, the information about the sentiment of a review, and indeed the meaning of natural language, is contained in complex feature interactions, not in isolated features. To evaluate a machine's understanding of language, we must remove all simple feature correlations that would allow the machine to predict the correct label without considering how those features interact.&#10;&#10;Collecting data that accurately reflects the assumptions of a competency problem is very challenging, especially when humans are involved in creating it. Humans suffer from many different kinds of bias and priming effects, which we collectively model in this work with rejection sampling during data collection. We theoretically analyze data collection under this biased sampling process, showing that any amount of bias will result in increasing probability of statistically-significant spurious feature correlations as dataset size increases (\S\ref{sec:biased-sampling}).&#10;&#10;This theoretical treatment of bias in data collection gives us a new, simple measure of data artifacts (\S\ref{sec:hypothesis}), which we use to explore artifacts in several existing datasets (\S\ref{sec:empirical}). Figure~\ref{fig:snli-artifacts} revisits prior analyses on the SNLI dataset~\cite{bowman-etal-2015-large} with our statistical test. An analysis based on pointwise mutual information~\cite[e.g.,][]{gururangan-etal-2018-annotation} would correspond to a horizontal line in that figure, missing many features that have less extreme but still significant correlations with class labels. These less extreme correlations still lead models to overweight simple features. The problem of bias in data collection is pervasive and not easily addressed with current learning techniques.&#10;&#10;Our framework also allows us to examine the theoretical impact of proposed techniques to mitigate bias, including performing local edits after data collection (\S\ref{sec:local-edits}) and filtering collected data (\S\ref{sec:other-mitigations}). We derive properties of any local edit procedure that must hold for the procedure to effectively remove data artifacts. These proofs give dataset builders tools to monitor the data collection process to be sure that resultant datasets are as artifact-free as possible. Our analysis of local edits additionally suggests a strong relationship to sensitivity in boolean functions \cite{o2014analysis}, and we identify gaps in the theory of sensitivity that need to be filled to properly account for bias in sampled datasets.&#10;&#10;We believe our theoretical analysis of these problems provides a good starting point for future analyses of methods to improve NLP data collection, as well as insights for inductive biases that could be introduced to better model competency problems.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1508.05326" label="1508.05326">
        <attvalues>
          <attvalue for="0" value="A large annotated corpus for learning natural language inference" />
          <attvalue for="1" value="  Understanding entailment and contradiction is fundamental to understanding&#10;natural language, and inference about entailment and contradiction is a&#10;valuable testing ground for the development of semantic representations.&#10;However, machine learning research in this area has been dramatically limited&#10;by the lack of large-scale resources. To address this, we introduce the&#10;Stanford Natural Language Inference corpus, a new, freely available collection&#10;of labeled sentence pairs, written by humans doing a novel grounded task based&#10;on image captioning. At 570K pairs, it is two orders of magnitude larger than&#10;all other resources of its type. This increase in scale allows lexicalized&#10;classifiers to outperform some sophisticated existing entailment models, and it&#10;allows a neural network-based model to perform competitively on natural&#10;language inference benchmarks for the first time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.10555" label="2003.10555">
        <attvalues>
          <attvalue for="0" value="ELECTRA: Pre-training Text Encoders as Discriminators Rather Than&#10;  Generators" />
          <attvalue for="1" value="  Masked language modeling (MLM) pre-training methods such as BERT corrupt the&#10;input by replacing some tokens with [MASK] and then train a model to&#10;reconstruct the original tokens. While they produce good results when&#10;transferred to downstream NLP tasks, they generally require large amounts of&#10;compute to be effective. As an alternative, we propose a more sample-efficient&#10;pre-training task called replaced token detection. Instead of masking the&#10;input, our approach corrupts it by replacing some tokens with plausible&#10;alternatives sampled from a small generator network. Then, instead of training&#10;a model that predicts the original identities of the corrupted tokens, we train&#10;a discriminative model that predicts whether each token in the corrupted input&#10;was replaced by a generator sample or not. Thorough experiments demonstrate&#10;this new pre-training task is more efficient than MLM because the task is&#10;defined over all input tokens rather than just the small subset that was masked&#10;out. As a result, the contextual representations learned by our approach&#10;substantially outperform the ones learned by BERT given the same model size,&#10;data, and compute. The gains are particularly strong for small models; for&#10;example, we train a model on one GPU for 4 days that outperforms GPT (trained&#10;using 30x more compute) on the GLUE natural language understanding benchmark.&#10;Our approach also works well at scale, where it performs comparably to RoBERTa&#10;and XLNet while using less than 1/4 of their compute and outperforms them when&#10;using the same amount of compute.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.03553" label="1909.03553">
        <attvalues>
          <attvalue for="0" value="QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions" />
          <attvalue for="1" value="  We introduce the first open-domain dataset, called QuaRTz, for reasoning&#10;about textual qualitative relationships. QuaRTz contains general qualitative&#10;statements, e.g., &quot;A sunscreen with a higher SPF protects the skin longer.&quot;,&#10;twinned with 3864 crowdsourced situated questions, e.g., &quot;Billy is wearing&#10;sunscreen with a lower SPF than Lucy. Who will be best protected from the&#10;sun?&quot;, plus annotations of the properties being compared. Unlike previous&#10;datasets, the general knowledge is textual and not tied to a fixed set of&#10;relationships, and tests a system's ability to comprehend and apply textual&#10;qualitative knowledge in a novel setting. We find state-of-the-art results are&#10;substantially (20%) below human performance, presenting an open challenge to&#10;the NLP community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11554" label="2305.11554">
        <attvalues>
          <attvalue for="0" value="ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via&#10;  Tool Embeddings" />
          <attvalue for="1" value="  Augmenting large language models (LLMs) with external tools has emerged as a&#10;promising approach to solving complex problems. However, traditional methods,&#10;which finetune LLMs with tool demonstration data, can be both costly and&#10;restricted to a predefined set of tools. Recent in-context learning paradigm&#10;alleviates these issues, but the limited context length only allows for a few&#10;shots of demonstrations, leading to suboptimal understandings of the tools.&#10;Moreover, when there are numerous tools to choose from, in-context learning&#10;could completely fail to work. In this paper, we propose an alternative&#10;approach, $\textbf{ToolkenGPT}$, which combines the benefits of both sides. Our&#10;approach represents each $\underline{tool}$ as a to$\underline{ken}$&#10;($\textit{toolken}$) and learns an embedding for it, enabling tool calls in the&#10;same way as generating a regular word token. Once a toolken is triggered, the&#10;LLM is prompted to complete arguments for the tool to execute. ToolkenGPT&#10;offers the flexibility to plug in an arbitrary number of tools by expanding the&#10;set of toolkens on the fly. In addition, it improves tool use by allowing&#10;extensive demonstration data for learning the toolken embeddings. In diverse&#10;domains, including numerical reasoning, knowledge-based question answering, and&#10;embodied plan generation, our approach effectively augments LLMs with tools and&#10;substantially outperforms various latest baselines. ToolkenGPT demonstrates the&#10;promising ability to use relevant tools from a large tool set in complex&#10;scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Fine-tuning LLMs to use tools.&#10;Early research relied heavily on fine-tuning to augment LMs with tools. In these works, LMs were mostly fine-tuned to use one or a few tools in a specific domain. For example, the retriever has been a crucial tool for augmenting LLMs with external knowledge sources~\cite{yu2022survey}. The prominent works in this line include REALM~\cite{guu2020retrieval}, RAG~\cite{lewis2020retrieval}, and RETRO~\cite{borgeaud2022improving}. More recently, WebGPT~\cite{nakano2021webgpt} fine-tuned GPT-3 on human web search behaviors to learn how to use the web browser. With the advancements in LLMs, there has also been growing interest in tuning these models on a collection of general tools, including the QA model, calculator, translator, etc. Example works include TALM~\cite{parisi2022talm} and Toolformer~\cite{schick2023toolformer}. However, LLM fine-tuning is costly and these tuned LLMs struggle to generalize to emergent or updated tools. \ours learns lightweight \toolken embeddings for new tools, without any gradient calculation for the parameters of LLMs. This enables efficient adaption to new tools and maintains a minimal GPU memory overhead for training toolken embeddings, at a cost similar to LLM inference. &#10;&#10;In-context learning for tools.&#10;LLMs exhibit a strong in-context learning ability \cite{brown2020language}, which becomes a prevalent method to use tools by showing tool descriptions and demonstrations in context \cite{mialon2023augmented, Qin2023ToolLW}. Building on this idea, reasoning chains can be incorporated to tackle more complex problems \cite{yao2022react, khot2022decomposed, paranjape2023art}. This paradigm has given rise to popular industry products such as ChatGPT plugins and Langchain \cite{Chase_LangChain_2022}, along with many successful applications in important research topics. For instance, a code interpreter can effectively address the LLM's shortcomings in symbolic operations \cite{chen2022program, gao2022pal, he2023solving, lyu2023faithful,xie2023translating,liu2023llm+}. Furthermore, by calling &quot;tools&quot; that have an effect on the virtual or physical world, the LLM is capable of guiding embodied agents to accomplish various household tasks \cite{huang2022language, brohan2023can, huang2022inner, singh2022progprompt, huang2023grounded}. Recent attempts to utilize LLMs as a controller to coordinate multiple neural models also achieve promising progress in multimodal reasoning tasks \cite{shen2023hugginggpt, lu2023chameleon}. Nevertheless, all methods based on in-context learning suffer from inferior performance in complex scenarios, where the tools are unfamiliar or numerous. One concurrent work, \cite{li2023api} propose to retrieve the tools based on the text embedding of their documents, which may mitigate that issue. However, \ours is fundamentally different from their method, in that the toolken embeddings can encode the implicit semantics of tools from extensive demonstrations, which can never be inferred from the surface text (A concrete example is shown in Figure~\ref{fig:vh_case}). Also, note that \ours is compatible with the recent advanced prompting techniques, e.g., Chain-of-Thought (CoT)~\cite{wei2022chain}, to improve the LLMs performance further.&#10;&#10;Efficient tuning of large language models.&#10;Adapting pre-trained frozen LLMs efficiently to new tasks is an active research area, leading to a surge of interest in parameter-efficient fine-tuning (PEFT) methods~\cite{karimi2021compacter, li2021prefix, ding2022delta, liu2021p, liu2023pre}. The idea is to only fine-tune a small subset of parameters of the LLM while freezing most of its parameters, which bears similarity to our \toolken embedding method. Which part of parameters to tune is the key to PEFT methods; for instance, Adapters~\cite{houlsby2019parameter} insert trainable layers, BitFit~\cite{zaken2021bitfit} tunes the bias parameters, prompt tuning~\cite{lester2021power, wang2023multitask} appends parameters to the input embedding layer, and LoRA~\cite{hu2021lora} learns low-rank matrices within specific dense layers, etc. However, existing PEFT methods have not proven suitable for efficient tool learning, and utilizing these methods on tool demonstrations may not efficiently capture the desired tool knowledge as \ours does. To the best of our knowledge, we are the first to explore efficient tuning methods for predicting tools as tokens for tool learning of massive tools.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, External Tool Integration, In-Context Learning, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.01373" label="2304.01373">
        <attvalues>
          <attvalue for="0" value="Pythia: A Suite for Analyzing Large Language Models Across Training and&#10;  Scaling" />
          <attvalue for="1" value="  How do large language models (LLMs) develop and evolve over the course of&#10;training? How do these patterns change as models scale? To answer these&#10;questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on&#10;public data seen in the exact same order and ranging in size from 70M to 12B&#10;parameters. We provide public access to 154 checkpoints for each one of the 16&#10;models, alongside tools to download and reconstruct their exact training&#10;dataloaders for further study. We intend \textit{Pythia} to facilitate research&#10;in many areas, and we present several case studies including novel results in&#10;memorization, term frequency effects on few-shot performance, and reducing&#10;gender bias. We demonstrate that this highly controlled setup can be used to&#10;yield novel insights toward LLMs and their training dynamics. Trained models,&#10;analysis code, training code, and training data can be found at&#10;\url{https://github.com/EleutherAI/pythia}.&#10;" />
          <attvalue for="2" value="&#10;Over the past several years, large transformer models have established themselves as the premier methodology for generative tasks in natural language processing \cite{brown2020language,sanh2021multitask,chowdhery2022palm}. Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis \cite{ramesh2022hierarchical,crowson2022vqgan,rombach2022high}, protein modeling \cite{jumper2021highly,ahdritz2022openfold}, and computer programming \cite{chen2021evaluating,xu2022systematic,fried2022incoder}. Despite these successes, very little is known about how and why these models are so successful.&#10;&#10;Critical to understanding the functioning of transformers is better understanding how these models behave along two axes: training and scaling. It is well established that there are regular and predictable patterns in the behavior of trained language models as they scale \cite{kaplan2020scaling,henighan2020scaling,hernandez2021scaling,mikami2021scaling,pu2021scaling,sharma2020neural,ghorbani2021scaling}, but prior work connecting these ``Scaling Laws'' to the learning dynamics of language models is minimal. One of the driving reasons for this gap in research is a lack of access to appropriate model suites to test theories: although there are more publicly available LLMs than ever, they do not meet common requirements for researchers, as discussed in \ref{sec:pythia} of this paper. Of the research along&#10;these lines that does exist \cite{mcgrath2021acquisition,tirumala2022memorization,xia2022training}, it is overwhelmingly done on non-public models or model checkpoints, further emphasizing the importance of having publicly available model suites&#10;for scientific research.&#10;&#10;In this paper we introduce Pythia, a suite of decoder-only autoregressive language models ranging from 70M to 12B parameters designed specifically to facilitate such scientific research. The Pythia suite is the only publicly released suite of LLMs that satisfies three key properties:&#10;\begin{enumerate}&#10; \item Models span several orders of magnitude of model scale.&#10; \item All models were trained on the same data in the same order.&#10; \item The data and intermediate checkpoints are publicly available for study.&#10;\end{enumerate}&#10;We train 8 model sizes each on both the Pile \cite{gao2020pile,biderman2022datasheet} and the Pile after deduplication, providing 2 copies of the suite which can be compared.&#10;&#10;We use these key properties of Pythia in order to study for the first time how properties like gender bias, memorization, and few-shot learning are affected by the precise training data processed and model scale. We intend the following experiments to be case studies demonstrating the experimental setups Pythia enables, and to additionally provide directions for future work.&#10;&#10;Mitigating Gender Bias There is much work cataloging how language models reflect the biases encoded in their training data. However, while some work has explored finetuning's effects on bias in language models \cite{gira2022debiasing,kirtane2022efficient,choenni2021stepmothers}, or the relationship between the corpus statistics and the measured bias \cite{bordia2019identifying,wal2022birth}, researchers have generally lacked the tools to study the role of the training data on the learning dynamics of bias in large language models of different sizes. To demonstrate what is now possible with Pythia, we analyze whether deliberately modifying the frequency of gendered terms in the pretraining data of a language model can have an impact on its downstream behavior and biases. We leverage the known pretraining data and public training codebase of our model suite, and counterfactually retrain models such that the last 7\% and 21\% of model training has a majority of pronouns modified such that their grammatical gender is feminine rather than masculine. We demonstrate that such interventions are successful at reducing bias measures on a targeted benchmark, and propose counterfactual interventions and retrainability of portions of our models as a key tool for future study of the influence of training corpora on model behavior.&#10;&#10;Memorization is a Poisson Point Process Building on the extensive literature on memorization in large language models \cite{carlini2019secret,carlini2021extracting,hu2022membership}, we ask the following question: does the location of a particular sequence in the training dataset influence the likelihood of it being memorized? Leveraging Pythia's reproducible dataloader setup we answer this question in the negative, and furthermore find that a poisson point process is a very good model for the occurrence of memorized sequences over the course of training.&#10;&#10;Emergence of the Impact of Pretraining Frequencies Recent work has identified the frequency of specific facts within a corpus as an important factor in how likely a model is capable of applying that fact in response to a natural language question \cite{razeghi2022impact,elazar2022measuring,kandpal2022large,mallen2022parametric}. Existing work has been heavily dependent on the handful of models trained on public data, such as GPT-J \cite{gpt-j} and BLOOM \cite{scao2022bloom}, which lack frequent intermediate checkpoints, so none of these papers are able to look at the fine-grained evolution of this phenomenon over the course of training. To address this gap in the literature, we examine how the role of pretraining term frequencies changes over the course of training. We find that a significant phase change occurs after 65,000 training steps (45\% through training): the models with 2.8 billion parameters or more start to exhibit a correlation between task accuracy and occurrence of task-relevant terms which is not present in prior checkpoints and which is largely absent from smaller models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Model Scaling Effects, Artificial Intelligence, Mathematics, Training Dynamics Analysis" />
        </attvalues>
      </node>
      <node id="2110.08207" label="2110.08207">
        <attvalues>
          <attvalue for="0" value="Multitask Prompted Training Enables Zero-Shot Task Generalization" />
          <attvalue for="1" value="  Large language models have recently been shown to attain reasonable zero-shot&#10;generalization on a diverse set of tasks (Brown et al., 2020). It has been&#10;hypothesized that this is a consequence of implicit multitask learning in&#10;language models' pretraining (Radford et al., 2019). Can zero-shot&#10;generalization instead be directly induced by explicit multitask learning? To&#10;test this question at scale, we develop a system for easily mapping any natural&#10;language tasks into a human-readable prompted form. We convert a large set of&#10;supervised datasets, each with multiple prompts with diverse wording. These&#10;prompted datasets allow for benchmarking the ability of a model to perform&#10;completely held-out tasks. We fine-tune a pretrained encoder-decoder model&#10;(Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a&#10;wide variety of tasks. The model attains strong zero-shot performance on&#10;several standard datasets, often outperforming models up to 16x its size.&#10;Further, our approach attains strong performance on a subset of tasks from the&#10;BIG-bench benchmark, outperforming models up to 6x its size. All trained models&#10;are available at https://github.com/bigscience-workshop/t-zero and all prompts&#10;are available at https://github.com/bigscience-workshop/promptsource.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In this work, we distinguish implicit multitask learning in language model pretraining from explicit multitask learning~\cite{DBLP:journals/ml/Caruana97}, the technique for mixing multiple tasks into a single supervised training process. Models trained with multitask learning have long been shown to have improved performance in NLP~\cite{DBLP:conf/icml/CollobertW08}. Since different tasks have different outputs, applying multitask learning requires a shared format, and various have been used~\cite{DBLP:journals/corr/HashimotoXTS16,DBLP:journals/corr/abs-1806-08730}. Several multitask works also explore few-shot and zero-shot generalization to new datasets with large pretrained models (e.g., \cite{vu-etal-2020-exploring,ye2021crossfit}).&#10;&#10;Natural language prompting is the method of reformatting NLP tasks in the format of a natural language response to natural language input. The development of text-to-text pretrained models such as T5 \cite{t5} makes prompts a particularly useful method for multitask learning. For example, \cite{unifiedqa} reformat 20 question-answering datasets into a single prompt of question: … (A)… (B)… (C)… context: …, while later work such as \cite{DBLP:journals/corr/abs-2104-04670} and \cite{DBLP:journals/corr/abs-2104-14690} cast a range of datasets into a single boolean QA prompt or a single NLI prompt, respectively. Although effective, these single-prompt methods typically do not generalize to new prompts or new tasks inexpressible in their fixed format.&#10;&#10;More generally, \cite{schick-schutze-2021-exploiting} and \cite{gpt3} popularized using prompts as a generic method for all NLP tasks.&#10;\cite{natural-inst} further extend this approach to a multitask setup, training on prompts for 61 narrowly defined tasks (e.g., question generation, incorrect answer generation) adapted from 9 datasets' crowdsourcing instructions, whereas we train on and measure generalization across 62 datasets and 12 tasks as traditionally defined in the NLP literature (\S\ref{sec:dataset}). Additionally, their prompts include labeled examples in addition to instructions, whereas we focus on zero-shot generalization. Lastly, concurrent work by \cite{flan} shares a similar research question with us, although we differ in several substantive regards, e.g., prompt diversity, model scale, and held-out-task scheme. We discuss our differences in detail in \ref{sec:disc}.&#10;&#10;Finally, in explaining the success of prompts, the leading hypothesis is that models learn to understand the prompts as task instructions which help them generalize to held-out tasks \cite{flan,natural-inst,schick-schutze-2021-exploiting,gpt3}. However, the extent to which this success depends on the semantic meaningfulness of the prompts has been challenged \cite{webson-pavlick-2021, logan2021cutting}. Thus, in this work, we remain agnostic as to why prompts support generalization. We only claim that prompts serve as a natural format for multitask training which empirically supports generalization to held-out tasks.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2204.06125" label="2204.06125">
        <attvalues>
          <attvalue for="0" value="Hierarchical Text-Conditional Image Generation with CLIP Latents" />
          <attvalue for="1" value="  Contrastive models like CLIP have been shown to learn robust representations&#10;of images that capture both semantics and style. To leverage these&#10;representations for image generation, we propose a two-stage model: a prior&#10;that generates a CLIP image embedding given a text caption, and a decoder that&#10;generates an image conditioned on the image embedding. We show that explicitly&#10;generating image representations improves image diversity with minimal loss in&#10;photorealism and caption similarity. Our decoders conditioned on image&#10;representations can also produce variations of an image that preserve both its&#10;semantics and style, while varying the non-essential details absent from the&#10;image representation. Moreover, the joint embedding space of CLIP enables&#10;language-guided image manipulations in a zero-shot fashion. We use diffusion&#10;models for the decoder and experiment with both autoregressive and diffusion&#10;models for the prior, finding that the latter are computationally more&#10;efficient and produce higher-quality samples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.08583" label="2204.08583">
        <attvalues>
          <attvalue for="0" value="VQGAN-CLIP: Open Domain Image Generation and Editing with Natural&#10;  Language Guidance" />
          <attvalue for="1" value="  Generating and editing images from open domain text prompts is a challenging&#10;task that heretofore has required expensive and specially trained models. We&#10;demonstrate a novel methodology for both tasks which is capable of producing&#10;images of high visual quality from text prompts of significant semantic&#10;complexity without any training by using a multimodal encoder to guide image&#10;generations. We demonstrate on a variety of tasks how using CLIP [37] to guide&#10;VQGAN [11] produces higher visual quality outputs than prior, less flexible&#10;approaches like DALL-E [38], GLIDE [33] and Open-Edit [24], despite not being&#10;trained for the tasks presented. Our code is available in a public repository.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.10752" label="2112.10752">
        <attvalues>
          <attvalue for="0" value="High-Resolution Image Synthesis with Latent Diffusion Models" />
          <attvalue for="1" value="  By decomposing the image formation process into a sequential application of&#10;denoising autoencoders, diffusion models (DMs) achieve state-of-the-art&#10;synthesis results on image data and beyond. Additionally, their formulation&#10;allows for a guiding mechanism to control the image generation process without&#10;retraining. However, since these models typically operate directly in pixel&#10;space, optimization of powerful DMs often consumes hundreds of GPU days and&#10;inference is expensive due to sequential evaluations. To enable DM training on&#10;limited computational resources while retaining their quality and flexibility,&#10;we apply them in the latent space of powerful pretrained autoencoders. In&#10;contrast to previous work, training diffusion models on such a representation&#10;allows for the first time to reach a near-optimal point between complexity&#10;reduction and detail preservation, greatly boosting visual fidelity. By&#10;introducing cross-attention layers into the model architecture, we turn&#10;diffusion models into powerful and flexible generators for general conditioning&#10;inputs such as text or bounding boxes and high-resolution synthesis becomes&#10;possible in a convolutional manner. Our latent diffusion models (LDMs) achieve&#10;a new state of the art for image inpainting and highly competitive performance&#10;on various tasks, including unconditional image generation, semantic scene&#10;synthesis, and super-resolution, while significantly reducing computational&#10;requirements compared to pixel-based DMs. Code is available at&#10;https://github.com/CompVis/latent-diffusion .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.03374" label="2107.03374">
        <attvalues>
          <attvalue for="0" value="Evaluating Large Language Models Trained on Code" />
          <attvalue for="1" value="  We introduce Codex, a GPT language model fine-tuned on publicly available&#10;code from GitHub, and study its Python code-writing capabilities. A distinct&#10;production version of Codex powers GitHub Copilot. On HumanEval, a new&#10;evaluation set we release to measure functional correctness for synthesizing&#10;programs from docstrings, our model solves 28.8% of the problems, while GPT-3&#10;solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling&#10;from the model is a surprisingly effective strategy for producing working&#10;solutions to difficult prompts. Using this method, we solve 70.2% of our&#10;problems with 100 samples per problem. Careful investigation of our model&#10;reveals its limitations, including difficulty with docstrings describing long&#10;chains of operations and with binding operations to variables. Finally, we&#10;discuss the potential broader impacts of deploying powerful code generation&#10;technologies, covering safety, security, and economics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.13169" label="2202.13169">
        <attvalues>
          <attvalue for="0" value="A Systematic Evaluation of Large Language Models of Code" />
          <attvalue for="1" value="  Large language models (LMs) of code have recently shown tremendous promise in&#10;completing code and synthesizing code from natural language descriptions.&#10;However, the current state-of-the-art code LMs (e.g., Codex (Chen et al.,&#10;2021)) are not publicly available, leaving many questions about their model and&#10;data design decisions. We aim to fill in some of these blanks through a&#10;systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo,&#10;GPT-NeoX-20B, and CodeParrot, across various programming languages. Although&#10;Codex itself is not open-source, we find that existing open-source models do&#10;achieve close results in some programming languages, although targeted mainly&#10;for natural language modeling. We further identify an important missing piece&#10;in the form of a large open-source model trained exclusively on a multi-lingual&#10;corpus of code. We release a new model, PolyCoder, with 2.7B parameters based&#10;on the GPT-2 architecture, which was trained on 249GB of code across 12&#10;programming languages on a single machine. In the C programming language,&#10;PolyCoder outperforms all models including Codex. Our trained models are&#10;open-source and publicly available at https://github.com/VHellendoorn/Code-LMs,&#10;which enables future research and application in this area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05999" label="2204.05999">
        <attvalues>
          <attvalue for="0" value="InCoder: A Generative Model for Code Infilling and Synthesis" />
          <attvalue for="1" value="  Code is seldom written in a single left-to-right pass and is instead&#10;repeatedly edited and refined. We introduce InCoder, a unified generative model&#10;that can perform program synthesis (via left-to-right generation) as well as&#10;editing (via infilling). InCoder is trained to generate code files from a large&#10;corpus of permissively licensed code, where regions of code have been randomly&#10;masked and moved to the end of each file, allowing code infilling with&#10;bidirectional context. Our model is the first generative model that is able to&#10;directly perform zero-shot code infilling, which we evaluate on challenging&#10;tasks such as type inference, comment generation, and variable re-naming. We&#10;find that the ability to condition on bidirectional context substantially&#10;improves performance on these tasks, while still performing comparably on&#10;standard program synthesis benchmarks in comparison to left-to-right only&#10;models pretrained at similar scale. The InCoder models and code are publicly&#10;released. https://sites.google.com/view/incoder-code-models&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14701" label="2010.14701">
        <attvalues>
          <attvalue for="0" value="Scaling Laws for Autoregressive Generative Modeling" />
          <attvalue for="1" value="  We identify empirical scaling laws for the cross-entropy loss in four&#10;domains: generative image modeling, video modeling, multimodal&#10;image$\leftrightarrow$text models, and mathematical problem solving. In all&#10;cases autoregressive Transformers smoothly improve in performance as model size&#10;and compute budgets increase, following a power-law plus constant scaling law.&#10;The optimal model size also depends on the compute budget through a power-law,&#10;with exponents that are nearly universal across all data domains.&#10;  The cross-entropy loss has an information theoretic interpretation as&#10;$S($True$) + D_{\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws&#10;suggest a prediction for both the true data distribution's entropy and the KL&#10;divergence between the true and model distributions. With this interpretation,&#10;billion-parameter Transformers are nearly perfect models of the YFCC100M image&#10;distribution downsampled to an $8\times 8$ resolution, and we can forecast the&#10;model size needed to achieve any given reducible loss (ie $D_{\mathrm{KL}}$) in&#10;nats/image for other resolutions.&#10;  We find a number of additional scaling laws in specific domains: (a) we&#10;identify a scaling relation for the mutual information between captions and&#10;images in multimodal models, and show how to answer the question &quot;Is a picture&#10;worth a thousand words?&quot;; (b) in the case of mathematical problem solving, we&#10;identify scaling laws for model performance when extrapolating beyond the&#10;training distribution; (c) we finetune generative image models for ImageNet&#10;classification and find smooth scaling of the classification loss and error&#10;rate, even as the generative loss levels off. Taken together, these results&#10;strengthen the case that scaling laws have important implications for neural&#10;network performance, including on downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.01293" label="2102.01293">
        <attvalues>
          <attvalue for="0" value="Scaling Laws for Transfer" />
          <attvalue for="1" value="  We study empirical scaling laws for transfer learning between distributions&#10;in an unsupervised, fine-tuning setting. When we train increasingly large&#10;neural networks from-scratch on a fixed-size dataset, they eventually become&#10;data-limited and stop improving in performance (cross-entropy loss). When we do&#10;the same for models pre-trained on a large language dataset, the slope in&#10;performance gains is merely reduced rather than going to zero. We calculate the&#10;effective data &quot;transferred&quot; from pre-training by determining how much data a&#10;transformer of the same size would have required to achieve the same loss when&#10;training from scratch. In other words, we focus on units of data while holding&#10;everything else fixed. We find that the effective data transferred is described&#10;well in the low data regime by a power-law of parameter count and fine-tuning&#10;dataset size. We believe the exponents in these power-laws correspond to&#10;measures of the generality of a model and proximity of distributions (in a&#10;directed rather than symmetric sense). We find that pre-training effectively&#10;multiplies the fine-tuning dataset size. Transfer, like overall performance,&#10;scales predictably in terms of parameters, data, and compute.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.11018" label="2108.11018">
        <attvalues>
          <attvalue for="0" value="A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your&#10;  Pre-training Effective?" />
          <attvalue for="1" value="  Synthetic-to-real transfer learning is a framework in which a synthetically&#10;generated dataset is used to pre-train a model to improve its performance on&#10;real vision tasks. The most significant advantage of using synthetic images is&#10;that the ground-truth labels are automatically available, enabling unlimited&#10;expansion of the data size without human cost. However, synthetic data may have&#10;a huge domain gap, in which case increasing the data size does not improve the&#10;performance. How can we know that? In this study, we derive a simple scaling&#10;law that predicts the performance from the amount of pre-training data. By&#10;estimating the parameters of the law, we can judge whether we should increase&#10;the data or change the setting of image synthesis. Further, we analyze the&#10;theory of transfer learning by considering learning dynamics and confirm that&#10;the derived generalization bound is consistent with our empirical findings. We&#10;empirically validated our scaling law on various experimental settings of&#10;benchmark tasks, model sizes, and complexities of synthetic images.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.10802" label="2004.10802">
        <attvalues>
          <attvalue for="0" value="A Neural Scaling Law from the Dimension of the Data Manifold" />
          <attvalue for="1" value="  When data is plentiful, the loss achieved by well-trained neural networks&#10;scales as a power-law $L \propto N^{-\alpha}$ in the number of network&#10;parameters $N$. This empirical scaling law holds for a wide variety of data&#10;modalities, and may persist over many orders of magnitude. The scaling law can&#10;be explained if neural models are effectively just performing regression on a&#10;data manifold of intrinsic dimension $d$. This simple theory predicts that the&#10;scaling exponents $\alpha \approx 4/d$ for cross-entropy and mean-squared error&#10;losses. We confirm the theory by independently measuring the intrinsic&#10;dimension and the scaling exponents in a teacher/student framework, where we&#10;can study a variety of $d$ and $\alpha$ by dialing the properties of random&#10;teacher networks. We also test the theory with CNN image classifiers on several&#10;datasets and with GPT-type language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07740" label="2109.07740">
        <attvalues>
          <attvalue for="0" value="Scaling Laws for Neural Machine Translation" />
          <attvalue for="1" value="  We present an empirical study of scaling properties of encoder-decoder&#10;Transformer models used in neural machine translation (NMT). We show that&#10;cross-entropy loss as a function of model size follows a certain scaling law.&#10;Specifically (i) We propose a formula which describes the scaling behavior of&#10;cross-entropy loss as a bivariate function of encoder and decoder size, and&#10;show that it gives accurate predictions under a variety of scaling approaches&#10;and languages; we show that the total number of parameters alone is not&#10;sufficient for such purposes. (ii) We observe different power law exponents&#10;when scaling the decoder vs scaling the encoder, and provide recommendations&#10;for optimal allocation of encoder/decoder capacity based on this observation.&#10;(iii) We also report that the scaling behavior of the model is acutely&#10;influenced by composition bias of the train/test sets, which we define as any&#10;deviation from naturally generated text (either via machine generated or human&#10;translated text). We observe that natural text on the target side enjoys&#10;scaling, which manifests as successful reduction of the cross-entropy loss.&#10;(iv) Finally, we investigate the relationship between the cross-entropy loss&#10;and the quality of the generated translations. We find two different behaviors,&#10;depending on the nature of the test data. For test sets which were originally&#10;translated from target language to source language, both loss and BLEU score&#10;improve as model size increases. In contrast, for test sets originally&#10;translated from source language to target language, the loss improves, but the&#10;BLEU score stops improving after a certain threshold. We release generated text&#10;from all models used in this study.&#10;" />
          <attvalue for="2" value=" &#10;&#10;Scaling properties of neural networks have long been an intriguing topic of study \cite{10.5555/2969735.2969754,Amari1992FourTO}. Along with the practical success of modern neural networks at scale, theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing \cite{&#10;advani2017highdimensional,rosenfeld2019constructive,Geiger_2020,ghorbani2020linearized,chobernoulli2020,DBLP:journals/corr/abs-2102-04074,bahri2021explaining,loureiro2021capturing}. In particular, scaling model sizes, datasets and the total computation budget has been identified as a reliable approach to improve generalization performance on several machine learning tasks. For many of these tasks the scaling behavior of neural networks is highly predictable; model fit or test loss can be described precisely as a function of its number of parameters &#10;\cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,hernandez2021scaling,rosenfeld2019constructive}. Neural machine translation (NMT) has long enjoyed the benefits of scaling \cite{huang2019gpipe,DBLP:journals/corr/abs-1907-05019,DBLP:journals/corr/abs-2006-16668}, but studies investigating the scaling behavior of NMT models are missing. &#10;We present the first large-scale systematic study of scaling laws for encoder-decoder Transformer models applied to NMT \cite{DBLP:journals/corr/VaswaniSPUJGKP17}. &#10;&#10;We start with highlighting the major differences between decoder-only language models, where the majority of the previous work has focused, and encoder-decoder (conditional) language models applied to NMT. The two differ along a few crucial dimensions. The first difference results from the very nature of the separate architectures being used, i.e. decoder-only vs encoder-decoder. The presence of separate architectural components complicates the study of scaling properties due to the increased degree of freedom.&#10;Second, contrary to language modeling, the task of machine translation is conditional: the task is predictive rather than fully generative.&#10;Furthermore, this prediction task is ambiguous: there is no one right answer for a given source, and translations can vary substantially depending on the translator's incentives.&#10;This manifests itself as different scaling benefits for different test sets. &#10;To take an extreme example, a test set translated by someone who writes nearly word-for-word translations may benefit less from model scaling than one translated by someone who considers each translation a work of art. &#10;In this work, these differences in difficulty coincide with the translation direction of the test set; that is,&#10;whether the source was translated into the target (source-original) or vice versa (target-original).&#10;Source-original data has translated text on the target side, which contains several artifacts of ``translationese'' that distinguish it from text originally written in that language, often lacking the diversity and complexity of ``natural'' text \cite{Koppel:2011:TD:2002472.2002636}, while target-original data requires the prediction of more complex natural text on the target side.&#10;Finally, unlike language models, NMT is evaluated on metrics that quantify generation quality against reference translations (for eg. BLEU) \cite{papineni2002bleu} instead of evaluating model fit (perplexity) on an evaluation set. &#10;&#10;In this paper, we aim to provide empirical answers to the following research questions: &#10;&#10;\begin{enumerate}&#10; \item Does the encoder-decoder architecture for NMT share the same scaling law function as the language models? &#10; Contrary to previous work on LM, we show that a univariate law depending on the total number of parameters in the network does not adequately describe the scaling behavior of NMT models. Our scaling laws parameterize the cross entropy loss as a bivariate function of the number of encoder parameters and the number of decoder parameters as separate variables. Our results indicate that the scaling behavior is largely determined by the total capacity of the model, and the capacity allocation between the encoder and the decoder.&#10;&#10;\item How does the naturalness of source/target side data affect scaling behavior? We study the effect of naturalness of the source and target text, both for training and evaluation. &#10; When evaluating with target side natural text, scaling the model capacity continues improving model quality throughout our range of measurements. On the other hand, improvements on cross-entropy saturate (or reaches the irreducible error region) on source side natural evaluation sets even for moderately-sized models.&#10;&#10;\item Do scaling improvements in cross-entropy translate into corresponding improvements in generation quality? Finally we study the relationship between generation quality and cross-entropy and how their correlation changes as we: (i) Scale different components of the model (encoder vs decoder) and (ii) Evaluate on source-natural or target-natural evaluation sets. &#10;&#10;\end{enumerate}&#10;&#10;Our results on multiple language pairs and training/test data compositions validate that model scaling predictably improves the cross-entropy on validation data. However, our findings also raise several questions regarding the effect of naturalness of training and evaluation text and how cross-entropy eventually relates with generation quality for auto-regressive generative models.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.09259" label="2111.09259">
        <attvalues>
          <attvalue for="0" value="Acquisition of Chess Knowledge in AlphaZero" />
          <attvalue for="1" value="  What is learned by sophisticated neural network agents such as AlphaZero?&#10;This question is of both scientific and practical interest. If the&#10;representations of strong neural networks bear no resemblance to human&#10;concepts, our ability to understand faithful explanations of their decisions&#10;will be restricted, ultimately limiting what we can achieve with neural network&#10;interpretability. In this work we provide evidence that human knowledge is&#10;acquired by the AlphaZero neural network as it trains on the game of chess. By&#10;probing for a broad range of human chess concepts we show when and where these&#10;concepts are represented in the AlphaZero network. We also provide a&#10;behavioural analysis focusing on opening play, including qualitative analysis&#10;from chess Grandmaster Vladimir Kramnik. Finally, we carry out a preliminary&#10;investigation looking at the low-level details of AlphaZero's representations,&#10;and make the resulting behavioural and representational analyses available&#10;online.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.10770" label="2205.10770">
        <attvalues>
          <attvalue for="0" value="Memorization Without Overfitting: Analyzing the Training Dynamics of&#10;  Large Language Models" />
          <attvalue for="1" value="  Despite their wide adoption, the underlying training and memorization&#10;dynamics of very large language models is not well understood. We empirically&#10;study exact memorization in causal and masked language modeling, across model&#10;sizes and throughout the training process. We measure the effects of dataset&#10;size, learning rate, and model size on memorization, finding that larger&#10;language models memorize training data faster across all settings.&#10;Surprisingly, we show that larger models can memorize a larger portion of the&#10;data before over-fitting and tend to forget less throughout the training&#10;process. We also analyze the memorization dynamics of different parts of speech&#10;and find that models memorize nouns and numbers first; we hypothesize and&#10;provide empirical evidence that nouns and numbers act as a unique identifier&#10;for memorizing individual training examples. Together, these findings present&#10;another piece of the broader puzzle of trying to understand what actually&#10;improves as models get bigger.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.09803" label="2212.09803">
        <attvalues>
          <attvalue for="0" value="Training Trajectories of Language Models Across Scales" />
          <attvalue for="1" value="  Scaling up language models has led to unprecedented performance gains, but&#10;little is understood about how the training dynamics change as models get&#10;larger. How do language models of different sizes learn during pre-training?&#10;Why do larger language models demonstrate more desirable behaviors? In this&#10;paper, we analyze the intermediate training checkpoints of differently sized&#10;OPT models (Zhang et al.,2022)--from 125M to 175B parameters--on next-token&#10;prediction, sequence-level generation, and downstream tasks. We find that 1) at&#10;a given perplexity and independent of model sizes, a similar subset of training&#10;tokens see the most significant reduction in loss, with the rest stagnating or&#10;showing double-descent behavior; 2) early in training, all models learn to&#10;reduce the perplexity of grammatical sequences that contain hallucinations,&#10;with small models halting at this suboptimal distribution and larger ones&#10;eventually learning to assign these sequences lower probabilities; 3)&#10;perplexity is a strong predictor of in-context learning performance on 74&#10;multiple-choice tasks from BIG-Bench, and this holds independent of the model&#10;size. Together, these results show that perplexity is more predictive of model&#10;behaviors than model size or training computation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.00027" label="2101.00027">
        <attvalues>
          <attvalue for="0" value="The Pile: An 800GB Dataset of Diverse Text for Language Modeling" />
          <attvalue for="1" value="  Recent work has demonstrated that increased training dataset diversity&#10;improves general cross-domain knowledge and downstream generalization&#10;capability for large-scale language models. With this in mind, we present&#10;\textit{the Pile}: an 825 GiB English text corpus targeted at training&#10;large-scale language models. The Pile is constructed from 22 diverse&#10;high-quality subsets -- both existing and newly constructed -- many of which&#10;derive from academic or professional sources. Our evaluation of the untuned&#10;performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on&#10;many of its components, such as academic writing. Conversely, models trained on&#10;the Pile improve significantly over both Raw CC and CC-100 on all components of&#10;the Pile, while improving performance on downstream evaluations. Through an&#10;in-depth exploratory analysis, we document potentially concerning aspects of&#10;the data for prospective users. We make publicly available the code used in its&#10;construction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Self-supervised training of natural language processing models on large, unlabeled text corpora, has seen widespread adoption in the field. Word representation models such as GloVe \cite{pennington2014glove} and word2vec \cite{mikolov2013word2vec} were trained on datasets such as Wikipedia, Gigaword \cite{graff2003english}, or a non-public Google News corpus. More recently, language models \cite{GPT,GPT2,GPT3,TuringNLG,Megatron} and masked language models \cite{BERT,RoBERTa,T5} have been trained on datasets such as Wikipedia, BookCorpus \cite{BookCorpus}, RealNews \cite{zellers2019neuralfakenews}, CC-Stories \cite{trinh2018commonsense}, and other Internet scrape-derived datasets discussed below. Other datasets such as WikiText \cite{wikitext} have also been used in similar self-supervised training. &#10;&#10;As data requirements for language modeling have grown, the field has turned towards Internet scrapes for large-scale datasets \cite{OpenWeb}, with Common Crawl being particularly prevalent. Works such as \cite{GPT3,wenzek2019ccnet,OSCAR,T5} have relied on Common Crawl to build training datasets for large-scale models. However, these works often highlight the difficulty of cleaning and filtering the Common Crawl data, and often highlight the resulting data quality as a determining factor of model capability.&#10;&#10;It has also been increasingly common practice to combine multiple datasets when training language models. For instance, GPT \cite{GPT} was trained on Wikipedia and BookCorpus, whereas GPT-3 \cite{GPT3} was trained on Wikipedia, two fiction datasets, and two web-scraped datasets. The Pile continues the trend of combining large-scale web-scrapes with smaller, higher-quality datasets that capture knowledge we believe would be most beneficial to training language models.&#10;&#10;The two most comparable publicly available datasets to the Pile are CC-100 \cite{wenzek2019ccnet} and C4/mC4 \cite{T5}. C4 is comparably-sized to the Pile, while mC4 and CC-100 are larger, multilingual datasets.&#10;However, C4/mC4 require immense computational resources to preprocess the data, with its maintainers even recommending the use of a distributed cloud service, setting a high bar of entry to using these datasets. &#10;CC-100 is directly downloadable and pre-cleaned; however, its English portion is much smaller than the Pile.&#10;Importantly, these three datasets are all derived entirely from Common Crawl---as discussed above, the current best practice in training large-scale language models involve using both large web scrapes and more targeted, higher-quality datasets, which the Pile directly addresses.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2209.03661" label="2209.03661">
        <attvalues>
          <attvalue for="0" value="Efficient Gender Debiasing of Pre-trained Indic Language Models" />
          <attvalue for="1" value="  The gender bias present in the data on which language models are pre-trained&#10;gets reflected in the systems that use these models. The model's intrinsic&#10;gender bias shows an outdated and unequal view of women in our culture and&#10;encourages discrimination. Therefore, in order to establish more equitable&#10;systems and increase fairness, it is crucial to identify and mitigate the bias&#10;existing in these models. While there is a significant amount of work in this&#10;area in English, there is a dearth of research being done in other gendered and&#10;low resources languages, particularly the Indian languages. English is a&#10;non-gendered language, where it has genderless nouns. The methodologies for&#10;bias detection in English cannot be directly deployed in other gendered&#10;languages, where the syntax and semantics vary. In our paper, we measure gender&#10;bias associated with occupations in Hindi language models. Our major&#10;contributions in this paper are the construction of a novel corpus to evaluate&#10;occupational gender bias in Hindi, quantify this existing bias in these systems&#10;using a well-defined metric, and mitigate it by efficiently fine-tuning our&#10;model. Our results reflect that the bias is reduced post-introduction of our&#10;proposed mitigation techniques. Our codebase is available publicly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.10052" label="2109.10052">
        <attvalues>
          <attvalue for="0" value="Stepmothers are mean and academics are pretentious: What do pretrained&#10;  language models learn about you?" />
          <attvalue for="1" value="  In this paper, we investigate what types of stereotypical information are&#10;captured by pretrained language models. We present the first dataset comprising&#10;stereotypical attributes of a range of social groups and propose a method to&#10;elicit stereotypes encoded by pretrained language models in an unsupervised&#10;fashion. Moreover, we link the emergent stereotypes to their manifestation as&#10;basic emotions as a means to study their emotional effects in a more&#10;generalized manner. To demonstrate how our methods can be used to analyze&#10;emotion and stereotype shifts due to linguistic experience, we use fine-tuning&#10;on news sources as a case study. Our experiments expose how attitudes towards&#10;different social groups vary across models and how quickly emotions and&#10;stereotypes can shift at the fine-tuning stage.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.10245" label="2207.10245">
        <attvalues>
          <attvalue for="0" value="The Birth of Bias: A case study on the evolution of gender bias in an&#10;  English language model" />
          <attvalue for="1" value="  Detecting and mitigating harmful biases in modern language models are widely&#10;recognized as crucial, open problems. In this paper, we take a step back and&#10;investigate how language models come to be biased in the first place. We use a&#10;relatively small language model, using the LSTM architecture trained on an&#10;English Wikipedia corpus. With full access to the data and to the model&#10;parameters as they change during every step while training, we can map in&#10;detail how the representation of gender develops, what patterns in the dataset&#10;drive this, and how the model's internal state relates to the bias in a&#10;downstream task (semantic textual similarity). We find that the representation&#10;of gender is dynamic and identify different phases during training.&#10;Furthermore, we show that gender information is represented increasingly&#10;locally in the input embeddings of the model and that, as a consequence,&#10;debiasing these can be effective in reducing the downstream bias. Monitoring&#10;the training dynamics, allows us to detect an asymmetry in how the female and&#10;male gender are represented in the input embeddings. This is important, as it&#10;may cause naive mitigation strategies to introduce new undesirable biases. We&#10;discuss the relevance of the findings for mitigation strategies more generally&#10;and the prospects of generalizing our methods to larger language models, the&#10;Transformer architecture, other languages and other undesirable biases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.08232" label="1802.08232">
        <attvalues>
          <attvalue for="0" value="The Secret Sharer: Evaluating and Testing Unintended Memorization in&#10;  Neural Networks" />
          <attvalue for="1" value="  This paper describes a testing methodology for quantitatively assessing the&#10;risk that rare or unique training-data sequences are unintentionally memorized&#10;by generative sequence models---a common type of machine-learning model.&#10;Because such models are sometimes trained on sensitive data (e.g., the text of&#10;users' private messages), this methodology can benefit privacy by allowing&#10;deep-learning practitioners to select means of training that minimize such&#10;memorization.&#10;  In experiments, we show that unintended memorization is a persistent,&#10;hard-to-avoid issue that can have serious consequences. Specifically, for&#10;models trained without consideration of memorization, we describe new,&#10;efficient procedures that can extract unique, secret sequences, such as credit&#10;card numbers. We show that our testing strategy is a practical and easy-to-use&#10;first line of defense, e.g., by describing its application to quantitatively&#10;limit data exposure in Google's Smart Compose, a commercial text-completion&#10;neural network trained on millions of users' email messages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.07805" label="2012.07805">
        <attvalues>
          <attvalue for="0" value="Extracting Training Data from Large Language Models" />
          <attvalue for="1" value="  It has become common to publish large (billion parameter) language models&#10;that have been trained on private datasets. This paper demonstrates that in&#10;such settings, an adversary can perform a training data extraction attack to&#10;recover individual training examples by querying the language model.&#10;  We demonstrate our attack on GPT-2, a language model trained on scrapes of&#10;the public Internet, and are able to extract hundreds of verbatim text&#10;sequences from the model's training data. These extracted examples include&#10;(public) personally identifiable information (names, phone numbers, and email&#10;addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible&#10;even though each of the above sequences are included in just one document in&#10;the training data.&#10;  We comprehensively evaluate our extraction attack to understand the factors&#10;that contribute to its success. Worryingly, we find that larger models are more&#10;vulnerable than smaller models. We conclude by drawing lessons and discussing&#10;possible safeguards for training large language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.07853" label="2103.07853">
        <attvalues>
          <attvalue for="0" value="Membership Inference Attacks on Machine Learning: A Survey" />
          <attvalue for="1" value="  Machine learning (ML) models have been widely applied to various&#10;applications, including image classification, text generation, audio&#10;recognition, and graph data analysis. However, recent studies have shown that&#10;ML models are vulnerable to membership inference attacks (MIAs), which aim to&#10;infer whether a data record was used to train a target model or not. MIAs on ML&#10;models can directly lead to a privacy breach. For example, via identifying the&#10;fact that a clinical record that has been used to train a model associated with&#10;a certain disease, an attacker can infer that the owner of the clinical record&#10;has the disease with a high chance. In recent years, MIAs have been shown to be&#10;effective on various ML models, e.g., classification models and generative&#10;models. Meanwhile, many defense methods have been proposed to mitigate MIAs.&#10;Although MIAs on ML models form a newly emerging and rapidly growing research&#10;area, there has been no systematic survey on this topic yet. In this paper, we&#10;conduct the first comprehensive survey on membership inference attacks and&#10;defenses. We provide the taxonomies for both attacks and defenses, based on&#10;their characterizations, and discuss their pros and cons. Based on the&#10;limitations and gaps identified in this survey, we point out several promising&#10;future research directions to inspire the researchers who wish to follow this&#10;area. This survey not only serves as a reference for the research community but&#10;also provides a clear description for researchers outside this research domain.&#10;To further help the researchers, we have created an online resource repository,&#10;which we will keep updated with future relevant work. Interested readers can&#10;find the repository at&#10;https://github.com/HongshengHu/membership-inference-machine-learning-literature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.07206" label="2202.07206">
        <attvalues>
          <attvalue for="0" value="Impact of Pretraining Term Frequencies on Few-Shot Reasoning" />
          <attvalue for="1" value="  Pretrained Language Models (LMs) have demonstrated ability to perform&#10;numerical reasoning by extrapolating from a few examples in few-shot settings.&#10;However, the extent to which this extrapolation relies on robust reasoning is&#10;unclear. In this paper, we investigate how well these models reason with terms&#10;that are less frequent in the pretraining data. In particular, we examine the&#10;correlations between the model performance on test instances and the frequency&#10;of terms from those instances in the pretraining data. We measure the strength&#10;of this correlation for a number of GPT-based language models (pretrained on&#10;the Pile dataset) on various numerical deduction tasks (e.g., arithmetic and&#10;unit conversion). Our results consistently demonstrate that models are more&#10;accurate on instances whose terms are more prevalent, in some cases above&#10;$70\%$ (absolute) more accurate on the top 10\% frequent terms in comparison to&#10;the bottom 10\%. Overall, although LMs exhibit strong performance at few-shot&#10;numerical reasoning tasks, our results raise the question of how much models&#10;actually generalize beyond pretraining data, and we encourage researchers to&#10;take the pretraining data into account when interpreting evaluation results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.08411" label="2211.08411">
        <attvalues>
          <attvalue for="0" value="Large Language Models Struggle to Learn Long-Tail Knowledge" />
          <attvalue for="1" value="  The Internet contains a wealth of knowledge -- from the birthdays of&#10;historical figures to tutorials on how to code -- all of which may be learned&#10;by language models. However, while certain pieces of information are ubiquitous&#10;on the web, others appear extremely rarely. In this paper, we study the&#10;relationship between the knowledge memorized by large language models and the&#10;information in pre-training datasets scraped from the web. In particular, we&#10;show that a language model's ability to answer a fact-based question relates to&#10;how many documents associated with that question were seen during pre-training.&#10;We identify these relevant documents by entity linking pre-training datasets&#10;and counting documents that contain the same entities as a given&#10;question-answer pair. Our results demonstrate strong correlational and causal&#10;relationships between accuracy and relevant document count for numerous&#10;question answering datasets (e.g., TriviaQA), pre-training corpora (e.g.,&#10;ROOTS), and model sizes (e.g., 176B parameters). Moreover, while larger models&#10;are better at learning long-tail knowledge, we estimate that today's models&#10;must be scaled by many orders of magnitude to reach competitive QA performance&#10;on questions with little support in the pre-training data. Finally, we show&#10;that retrieval-augmentation can reduce the dependence on relevant pre-training&#10;information, presenting a promising approach for capturing the long-tail.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.05100" label="2211.05100">
        <attvalues>
          <attvalue for="0" value="BLOOM: A 176B-Parameter Open-Access Multilingual Language Model" />
          <attvalue for="1" value="  Large language models (LLMs) have been shown to be able to perform new tasks&#10;based on a few demonstrations or natural language instructions. While these&#10;capabilities have led to widespread adoption, most LLMs are developed by&#10;resource-rich organizations and are frequently kept from the public. As a step&#10;towards democratizing this powerful technology, we present BLOOM, a&#10;176B-parameter open-access language model designed and built thanks to a&#10;collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer&#10;language model that was trained on the ROOTS corpus, a dataset comprising&#10;hundreds of sources in 46 natural and 13 programming languages (59 in total).&#10;We find that BLOOM achieves competitive performance on a wide variety of&#10;benchmarks, with stronger results after undergoing multitask prompted&#10;finetuning. To facilitate future research and applications using LLMs, we&#10;publicly release our models and code under the Responsible AI License.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.13880" label="2105.13880">
        <attvalues>
          <attvalue for="0" value="Knowledge Inheritance for Pre-trained Language Models" />
          <attvalue for="1" value="  Recent explorations of large-scale pre-trained language models (PLMs) have&#10;revealed the power of PLMs with huge amounts of parameters, setting off a wave&#10;of training ever-larger PLMs. However, it requires tremendous computational&#10;resources to train a large-scale PLM, which may be practically unaffordable. In&#10;addition, existing large-scale PLMs are mainly trained from scratch&#10;individually, ignoring that many well-trained PLMs are available. To this end,&#10;we explore the question how could existing PLMs benefit training large-scale&#10;PLMs in future. Specifically, we introduce a pre-training framework named&#10;&quot;knowledge inheritance&quot; (KI) and explore how could knowledge distillation serve&#10;as auxiliary supervision during pre-training to efficiently learn larger PLMs.&#10;Experimental results demonstrate the superiority of KI in training efficiency.&#10;We also conduct empirical analyses to explore the effects of teacher PLMs'&#10;pre-training settings, including model architecture, pre-training data, etc.&#10;Finally, we show that KI could be applied to domain adaptation and knowledge&#10;transfer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Efficient Pre-training for NLP. Recently, researchers find that the performance of PLMs can be simply improved by increasing the model size, data size and training steps~\cite{liu2019roberta,raffel2019exploring,kaplan2020scaling}, sparking a wave of training ever-larger PLMs. For instance, the revolutionary GPT-3~\cite{NEURIPS2020_1457c0d6}, which contains $175$ billion parameters, shows strong capabilities for language understanding and generation. This means that utilizing PLMs with huge parameters for downstream tasks may greatly relieve the cost of manual labeling and model training for new tasks. However, larger models require greater computational demands~\cite{patterson2021carbon}. To this end, researchers propose to accelerate pre-training by mixed-precision training~\cite{shoeybi2019megatron}, distributed training~\cite{shoeybi2019megatron}, large batch optimization~\cite{you2019large}, etc.&#10;&#10;Another line of methods~\cite{gong2019efficient,gu2020transformer,chen2021bert2bert,qin2022elle} proposes to pre-train larger PLMs progressively. They first train a small PLM, and then gradually increase the depth or width of the network based on parameter recycling (PR). Although PR could be used for the goal of KI, these methods typically have strict requirements on the architectures of both models, which is not flexible for practical uses; instead, we resort to KD as the solution for KI without architecture constraints. In addition, different from KI, PR is not applicable for absorbing knowledge from multiple teacher models and domain adaptation. More detailed comparisons between KI and PR are discussed in \ref{sec:compare_KI_PT}.&#10;&#10;Knowledge Distillation for PLMs. Knowledge Distillation (KD)~\cite{hinton2015distilling} aims to compress a large model into a fast-to-execute one. KD has renewed a surge of interest in PLMs recently. Some explore KD at different training phases, e.g., pre-training~\cite{sanh2019distilbert}, downstream fine-tuning~\cite{sun2019patient,krishna2019thieves}, or both of them~\cite{jiao2019tinybert}; others explore distilling not only the final logits output by the large PLM, but also the intermediate hidden representations~\cite{sanh2019distilbert,jiao2019tinybert,sun2020contrastive}. Conventional KD presumes that teacher models play pivotal roles in mastering knowledge, and student models generally cannot match their teachers in performance. When it comes to the scenario of KI, since student models have larger capacities, the performance of teacher models is no longer an ``upper bound'' of student models. Outside NLP, researchers recently demonstrate that a student model could also benefit from a poor teacher for a specific downstream task~\cite{yuan2020revisiting} (reversed KD). Based on the prior explorations, in this paper, we investigate the application of reversed KD in pre-training. " />
          <attvalue for="4" value="Pre-training Efficiency, Computer Science, Large Language Models, Linguistics, Cognitive Science, Knowledge Distillation, Mathematics" />
        </attvalues>
      </node>
      <node id="2009.11523" label="2009.11523">
        <attvalues>
          <attvalue for="0" value="Grounded Compositional Outputs for Adaptive Language Modeling" />
          <attvalue for="1" value="  Language models have emerged as a central component across NLP, and a great&#10;deal of progress depends on the ability to cheaply adapt them (e.g., through&#10;finetuning) to new domains and tasks. A language model's vocabulary$-$typically&#10;selected before training and permanently fixed later$-$affects its size and is&#10;part of what makes it resistant to such adaptation. Prior work has used&#10;compositional input embeddings based on surface forms to ameliorate this issue.&#10;In this work, we go one step beyond and propose a fully compositional output&#10;embedding layer for language models, which is further grounded in information&#10;from a structured lexicon (WordNet), namely semantically related words and&#10;free-text definitions. To our knowledge, the result is the first word-level&#10;language model with a size that does not depend on the training vocabulary. We&#10;evaluate the model on conventional language modeling as well as challenging&#10;cross-domain settings with an open vocabulary, finding that it matches or&#10;outperforms previous state-of-the-art output embedding methods and adaptation&#10;approaches. Our analysis attributes the improvements to sample efficiency: our&#10;model is more accurate for low-frequency words.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models (LMs) are at the heart of natural language processing, especially following their recent success in the pretraining paradigm \cite[inter alia]{dai15,peters-etal-2018-deep,devlin18,radford2019language}. Continued advances in NLP rely on the adaptability of LMs to domains beyond their training data and to new domains and tasks, e.g.,~through domain adaptive pretraining followed by finetuning \cite{gururangan-etal-2020-dont}. Here, we focus on an important component of LMs, namely the output vocabulary---over which a LM's probability distribution over the ``next word'' (given the history) ranges---and investigate the impact of the type of its representation on the adaptability of neural LMs.&#10;&#10;Today, LMs are typically trained with a closed output vocabulary derived from the training data; the vocabulary is not modified when the language model is adapted or deployed. This makes large pretrained language models struggle with rare words, despite being able to produce contextualized representations for them \cite{schick-schutze-2020-bertram}. More importantly, this means a generative LM can never give nonzero probability to a specific word it did not see in training. This is a longstanding challenge of language modeling \cite{jelinek1997statistical}, but it becomes especially important when we adapt to new domains and tasks.&#10;&#10;One way to ``open up'' the vocabulary is to model sequences of bytes, characters, or ``wordpieces'' rather than the conventional word tokens \cite{sennrich15,Radford2018ImprovingLU,ponti-etal-2019-towards}. While effective, this approach requires the LM to memorize subsequences if it is to treat them as words. These models appear to require greater network depth and show slower convergence than word-based alternatives \cite{cherry-etal-2018-revisiting,rami19}; the extra work comes at a cost. This is one of the reasons why the area of word-level language modeling is still very active \cite{baevski2018adaptive,sukhbaatar-etal-2019-adaptive,Khandelwal2020Generalization,press-etal-2020-improving}.&#10;&#10;Interpolations between word- and character- or morphology-based LMs represent another class of solutions \cite{mielke18,gerz-etal-2018-language,Ataman2020A}. These ``hybrid'' approaches combine benefits from both model types. However, they introduce complexity which makes them potentially more difficult to train, maintain, and analyze. Notable for enabling adaptability are interpolated LMs based on copy mechanisms \cite{merity2016pointer}, dynamic evaluation \cite{krause2017dynamic}, and neural caches \cite{grave2016improving,Grave17}; the last provides state-of-the-art adaptation performance and, unlike the rest, it does not require additional training. &#10;&#10;We propose a new word-level Grounded Compositional output LM (GroC)&#10;that applies a compositional representation to the output vocabulary (Section~\ref{comp_output}). Each word's output embedding is built from its surface character sequence and (if available) those of semantically related words and a free-text definition of from WordNet \cite{fellbaum98wordnet}. This parameterization offers two chief advantages. First, GroC can assign probability to words not seen during training. This means that a vocabulary different from the training vocabulary---e.g., one associated with a different text domain, crucial in adaptive settings---can be considered at inference time. Second, because there are no word type-specific parameters, the number of model parameters in GroC does not depend on the training vocabulary or its size.&#10;&#10;We evaluate GroC on language modeling with both fixed and open vocabularies in English. On standard language modeling (Section \ref{sec:closed_lm}) we observe that our model has superior perplexity and is more sample efficient than a variety of existing output embedding approaches, including the recent adaptive embedding of \cite{baevski2018adaptive}. &#10; The open-vocabulary settings include a cross-domain setting and finetuning (Section~\ref{sec:cross}). We find that GroC also outperforms strong interpolated baselines, including the unbounded neural cache model of \cite{Grave17} on ``near'' domains and performs competitively on ``far'' domains. &#10;&#10;Our analysis shows that our approach has improved sharing across words in the output vocabulary. We show experimentally that the perplexity gains are strongest for low-frequency words, implying improved sample efficiency relative to baselines: compositional output representations allow us to predict words from fewer training examples.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Compositional Embeddings, Vocabulary Size Optimization, Linguistics, Artificial Intelligence, Natural Language Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="1802.05365" label="1802.05365">
        <attvalues>
          <attvalue for="0" value="Deep contextualized word representations" />
          <attvalue for="1" value="  We introduce a new type of deep contextualized word representation that&#10;models both (1) complex characteristics of word use (e.g., syntax and&#10;semantics), and (2) how these uses vary across linguistic contexts (i.e., to&#10;model polysemy). Our word vectors are learned functions of the internal states&#10;of a deep bidirectional language model (biLM), which is pre-trained on a large&#10;text corpus. We show that these representations can be easily added to existing&#10;models and significantly improve the state of the art across six challenging&#10;NLP problems, including question answering, textual entailment and sentiment&#10;analysis. We also present an analysis showing that exposing the deep internals&#10;of the pre-trained network is crucial, allowing downstream models to mix&#10;different types of semi-supervision signals.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.07181" label="1910.07181">
        <attvalues>
          <attvalue for="0" value="BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized&#10;  Model Performance" />
          <attvalue for="1" value="  Pretraining deep language models has led to large performance gains in NLP.&#10;Despite this success, Schick and Sch\&quot;utze (2020) recently showed that these&#10;models struggle to understand rare words. For static word embeddings, this&#10;problem has been addressed by separately learning representations for rare&#10;words. In this work, we transfer this idea to pretrained language models: We&#10;introduce BERTRAM, a powerful architecture based on BERT that is capable of&#10;inferring high-quality embeddings for rare words that are suitable as input&#10;representations for deep language models. This is achieved by enabling the&#10;surface form and contexts of a word to interact with each other in a deep&#10;architecture. Integrating BERTRAM into BERT leads to large performance&#10;increases due to improved representations of rare and medium frequency words on&#10;both a rare word probing task and three downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1508.07909" label="1508.07909">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation of Rare Words with Subword Units" />
          <attvalue for="1" value="  Neural machine translation (NMT) models typically operate with a fixed&#10;vocabulary, but translation is an open-vocabulary problem. Previous work&#10;addresses the translation of out-of-vocabulary words by backing off to a&#10;dictionary. In this paper, we introduce a simpler and more effective approach,&#10;making the NMT model capable of open-vocabulary translation by encoding rare&#10;and unknown words as sequences of subword units. This is based on the intuition&#10;that various word classes are translatable via smaller units than words, for&#10;instance names (via character copying or transliteration), compounds (via&#10;compositional translation), and cognates and loanwords (via phonological and&#10;morphological transformations). We discuss the suitability of different word&#10;segmentation techniques, including simple character n-gram models and a&#10;segmentation based on the byte pair encoding compression algorithm, and&#10;empirically show that subword models improve over a back-off dictionary&#10;baseline for the WMT 15 translation tasks English-German and English-Russian by&#10;1.1 and 1.3 BLEU, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.03334" label="2108.03334">
        <attvalues>
          <attvalue for="0" value="Towards Zero-shot Language Modeling" />
          <attvalue for="1" value="  Can we construct a neural model that is inductively biased towards learning&#10;human languages? Motivated by this question, we aim at constructing an&#10;informative prior over neural weights, in order to adapt quickly to held-out&#10;languages in the task of character-level language modeling. We infer this&#10;distribution from a sample of typologically diverse training languages via&#10;Laplace approximation. The use of such a prior outperforms baseline models with&#10;an uninformative prior (so-called &quot;fine-tuning&quot;) in both zero-shot and few-shot&#10;settings. This shows that the prior is imbued with universal phonological&#10;knowledge. Moreover, we harness additional language-specific side information&#10;as distant supervision for held-out languages. Specifically, we condition&#10;language models on features from typological databases, by concatenating them&#10;to hidden states or generating weights with hyper-networks. These features&#10;appear beneficial in the few-shot setting, but not in the zero-shot setting.&#10;Since the paucity of digital texts affects the majority of the world's&#10;languages, we hope that these findings will help broaden the scope of&#10;applications for language technology.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;LSTMs have been probed for their inductive bias towards syntactic dependencies \cite{linzen-etal-2016-assessing} and grammaticality judgments \cite{D18-1151,warstadt2018neural}. \cite{ravfogel2019studying} have extended the scope of this analysis to typologically different languages through synthetic variations of English. In this work, we aim to model the inductive bias explicitly by constructing a prior over the space of neural network parameters.&#10;&#10;Few-shot word-level language modeling for truly under-resourced languages such as Yongning Na has been investigated by \cite{adams2017cross} with the aid of a bilingual lexicon. \cite{vinyals2016matching} and \cite{munkhdalai2018metalearning} proposed novel architectures (Matching Networks and LSTMs augmented with Hebbian Fast Weights, respectively) for rapid associative learning in English, and evaluated them in few-shot cloze tests. In this respect, our work is novel in pushing the problem to its most complex formulation, zero-shot inference, and in taking into account the largest sample of languages for language modeling to date.&#10;&#10;In addition to those considered in our work, there are also alternative methods to condition language models on features. \cite{kalchbrenner2013recurrent} used encoded features as additional biases in recurrent layers. \cite{kiros2014multimodal} put forth a log-bilinear model that allows for a `multiplicative interaction' between hidden representations and input features (such as images). With a similar device, but a different gating method, \cite{tsvetkov2016polyglot} trained a phoneme-level joint multilingual model of words conditioned on typological features from \cite{phoible}.&#10;&#10;The use of the Laplace method for neural transfer learning has been proposed by \cite{kirkpatrick2017overcoming}, inspired by synaptic consolidation in neuroscience, with the aim to avoid catastrophic forgetting. \cite{kochurov2018bayesian} tackled the problem of continuous learning by approximating the posterior probabilities through stochastic variational inference. \cite{ritter2018scalable} substitute diagonal Laplace approximation with a Kronecker factored method, leading to better uncertainty estimates. Finally, the regularizer proposed by \cite{duong2015low} for cross-lingual dependency parsing can be interpreted as a prior for MAP estimation where the covariance is an identity matrix.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1808.09943" label="1808.09943">
        <attvalues>
          <attvalue for="0" value="Revisiting Character-Based Neural Machine Translation with Capacity and&#10;  Compression" />
          <attvalue for="1" value="  Translating characters instead of words or word-fragments has the potential&#10;to simplify the processing pipeline for neural machine translation (NMT), and&#10;improve results by eliminating hyper-parameters and manual feature engineering.&#10;However, it results in longer sequences in which each symbol contains less&#10;information, creating both modeling and computational challenges. In this&#10;paper, we show that the modeling problem can be solved by standard&#10;sequence-to-sequence architectures of sufficient depth, and that deep models&#10;operating at the character level outperform identical models operating over&#10;word fragments. This result implies that alternative architectures for handling&#10;character input are better viewed as methods for reducing computation time than&#10;as improved ways of modeling longer sequences. From this perspective, we&#10;evaluate several techniques for character-level NMT, verify that they do not&#10;match the performance of our deep character baseline model, and evaluate the&#10;performance versus computation time tradeoffs they offer. Within this&#10;framework, we also perform the first evaluation for NMT of conditional&#10;computation over time, in which the model learns which timesteps can be&#10;skipped, rather than having them be dictated by a fixed schedule specified&#10;before training begins.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.10853" label="1809.10853">
        <attvalues>
          <attvalue for="0" value="Adaptive Input Representations for Neural Language Modeling" />
          <attvalue for="1" value="  We introduce adaptive input representations for neural language modeling&#10;which extend the adaptive softmax of Grave et al. (2017) to input&#10;representations of variable capacity. There are several choices on how to&#10;factorize the input and output layers, and whether to model words, characters&#10;or sub-word units. We perform a systematic comparison of popular choices for a&#10;self-attentional architecture. Our experiments show that models equipped with&#10;adaptive embeddings are more than twice as fast to train than the popular&#10;character input CNN while having a lower number of parameters. On the&#10;WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5&#10;perplexity compared to the previously best published result and on the Billion&#10;Word benchmark, we achieve 23.02 perplexity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.00172" label="1911.00172">
        <attvalues>
          <attvalue for="0" value="Generalization through Memorization: Nearest Neighbor Language Models" />
          <attvalue for="1" value="  We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM)&#10;by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The&#10;nearest neighbors are computed according to distance in the pre-trained LM&#10;embedding space, and can be drawn from any text collection, including the&#10;original LM training data. Applying this augmentation to a strong Wikitext-103&#10;LM, with neighbors drawn from the original training set, our $k$NN-LM achieves&#10;a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no&#10;additional training. We also show that this approach has implications for&#10;efficiently scaling up to larger training sets and allows for effective domain&#10;adaptation, by simply varying the nearest neighbor datastore, again without&#10;further training. Qualitatively, the model is particularly helpful in&#10;predicting rare patterns, such as factual knowledge. Together, these results&#10;strongly suggest that learning similarity between sequences of text is easier&#10;than predicting the next word, and that nearest neighbor search is an effective&#10;approach for language modeling in the long tail.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.08205" label="1804.08205">
        <attvalues>
          <attvalue for="0" value="Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model" />
          <attvalue for="1" value="  We show how the spellings of known words can help us deal with unknown words&#10;in open-vocabulary NLP tasks. The method we propose can be used to extend any&#10;closed-vocabulary generative model, but in this paper we specifically consider&#10;the case of neural language modeling. Our Bayesian generative story combines a&#10;standard RNN language model (generating the word tokens in each sentence) with&#10;an RNN-based spelling model (generating the letters in each word type). These&#10;two RNNs respectively capture sentence structure and word structure, and are&#10;kept separate as in linguistics. By invoking the second RNN to generate&#10;spellings for novel words in context, we obtain an open-vocabulary language&#10;model. For known words, embeddings are naturally inferred by combining evidence&#10;from type spelling and token context. Comparing to baselines (including a novel&#10;strong baseline), we beat previous work and establish state-of-the-art results&#10;on multiple datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.13890" label="1910.13890">
        <attvalues>
          <attvalue for="0" value="A Latent Morphology Model for Open-Vocabulary Neural Machine Translation" />
          <attvalue for="1" value="  Translation into morphologically-rich languages challenges neural machine&#10;translation (NMT) models with extremely sparse vocabularies where atomic&#10;treatment of surface forms is unrealistic. This problem is typically addressed&#10;by either pre-processing words into subword units or performing translation&#10;directly at the level of characters. The former is based on word segmentation&#10;algorithms optimized using corpus-level statistics with no regard to the&#10;translation task. The latter learns directly from translation data but requires&#10;rather deep architectures. In this paper, we propose to translate words by&#10;modeling word formation through a hierarchical latent variable model which&#10;mimics the process of morphological inflection. Our model generates words one&#10;character at a time by composing two latent representations: a continuous one,&#10;aimed at capturing the lexical semantics, and a set of (approximately) discrete&#10;features, aimed at capturing the morphosyntactic function, which are shared&#10;among different surface forms. Our model achieves better accuracy in&#10;translation into three morphologically-rich languages than conventional&#10;open-vocabulary NMT methods, while also demonstrating a better generalization&#10;capacity under low to mid-resource settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1609.07843" label="1609.07843">
        <attvalues>
          <attvalue for="0" value="Pointer Sentinel Mixture Models" />
          <attvalue for="1" value="  Recent neural network sequence models with softmax classifiers have achieved&#10;their best language modeling performance only with very large hidden states and&#10;large vocabularies. Even then they struggle to predict rare or unseen words&#10;even if the context makes the prediction unambiguous. We introduce the pointer&#10;sentinel mixture architecture for neural sequence models which has the ability&#10;to either reproduce a word from the recent context or produce a word from a&#10;standard softmax classifier. Our pointer sentinel-LSTM model achieves state of&#10;the art language modeling performance on the Penn Treebank (70.9 perplexity)&#10;while using far fewer parameters than a standard softmax LSTM. In order to&#10;evaluate how well language models can exploit longer contexts and deal with&#10;more realistic vocabularies and larger corpora we also introduce the freely&#10;available WikiText corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.04426" label="1612.04426">
        <attvalues>
          <attvalue for="0" value="Improving Neural Language Models with a Continuous Cache" />
          <attvalue for="1" value="  We propose an extension to neural network language models to adapt their&#10;prediction to the recent history. Our model is a simplified version of memory&#10;augmented networks, which stores past hidden activations as memory and accesses&#10;them through a dot product with the current hidden activation. This mechanism&#10;is very efficient and scales to very large memory sizes. We also draw a link&#10;between the use of external memory in neural network and cache models used with&#10;count based language models. We demonstrate on several language model datasets&#10;that our approach performs significantly better than recent memory augmented&#10;networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.02604" label="1711.02604">
        <attvalues>
          <attvalue for="0" value="Unbounded cache model for online language modeling with open vocabulary" />
          <attvalue for="1" value="  Recently, continuous cache models were proposed as extensions to recurrent&#10;neural network language models, to adapt their predictions to local changes in&#10;the data distribution. These models only capture the local context, of up to a&#10;few thousands tokens. In this paper, we propose an extension of continuous&#10;cache models, which can scale to larger contexts. In particular, we use a large&#10;scale non-parametric memory component that stores all the hidden activations&#10;seen in the past. We leverage recent advances in approximate nearest neighbor&#10;search and quantization algorithms to store millions of representations while&#10;searching them efficiently. We conduct extensive experiments showing that our&#10;approach significantly improves the perplexity of pre-trained language models&#10;on new distributions, and can scale efficiently to much larger contexts than&#10;previously proposed local cache models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12330" label="2305.12330">
        <attvalues>
          <attvalue for="0" value="Task-agnostic Distillation of Encoder-Decoder Language Models" />
          <attvalue for="1" value="  Finetuning pretrained language models (LMs) have enabled appealing&#10;performance on a diverse array of tasks. The intriguing task-agnostic property&#10;has driven a shifted focus from task-specific to task-agnostic distillation of&#10;LMs. While task-agnostic, compute-efficient, performance-preserved LMs can be&#10;yielded by task-agnostic distillation, previous studies mainly sit in&#10;distillation of either encoder-only LMs (e.g., BERT) or decoder-only ones&#10;(e.g., GPT) yet largely neglect that distillation of encoder-decoder LMs (e.g.,&#10;T5) can posit very distinguished behaviors. Frustratingly, we discover that&#10;existing task-agnostic distillation methods can fail to handle the distillation&#10;of encoder-decoder LMs. To the demand, we explore a few paths and uncover a&#10;path named as MiniEnD that successfully tackles the distillation of&#10;encoder-decoder LMs in a task-agnostic fashion. We examine MiniEnD on language&#10;understanding and abstractive summarization. The results showcase that MiniEnD&#10;is generally effective and is competitive compared to other alternatives. We&#10;further scale MiniEnD up to distillation of 3B encoder-decoder language models&#10;with interpolated distillation. The results imply the opportunities and&#10;challenges in distilling large language models (e.g., LLaMA).&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretrained language models (LMs) powered by finetuning have achieved remarkable performance on a wide range of downstream tasks~\cite{DBLP:conf/naacl/DevlinCLT19,DBLP:journals/corr/abs-1907-11692,Radford19}. Driven by the pursued task-agnostic property, distillation of LMs has witnessed a paradigm shift from task-specific to task-agnostic distillation~\cite{DBLP:journals/corr/abs-1910-01108}. Under a teacher-student regime, task-agnostic distillation distils pretrained LMs into ones of small compute on pretraining data so that these small LMs can be applied to tasks by finetuning~\cite{DBLP:conf/emnlp/JiaoYSJCL0L20,DBLP:conf/nips/WangW0B0020,DBLP:journals/corr/abs-2302-09632}. In contrast, task-specific distillation distils finetuned LMs on finetuning data and consumed resource can be even huge when the number of tasks explode~\cite{DBLP:journals/corr/HintonVD15,DBLP:conf/emnlp/SunCGL19,DBLP:conf/acl/XiaZC22,DBLP:conf/emnlp/YangZS22}. Additionally, it is acknowledged that task-agnostic distillation typically brings performance gain over task-specific distillation does~\cite{DBLP:journals/corr/abs-2205-14570}.&#10;&#10;Despite so many merits, prior studies mostly lie in distillation of either encoder-only LMs~\cite[e.g., BERT,][]{DBLP:conf/naacl/DevlinCLT19} or decoder-only LMs~\cite[e.g., GPT,][]{Radford19} and largely ignore the signifance of task-agnostic distillation of encoder-decoder LMs~\cite[e.g., T5,][]{DBLP:journals/jmlr/RaffelSRLNMZLL20} given recent advances in task-specific distillation of encoder-decoder LMs though~\cite{DBLP:journals/corr/abs-2010-13002,DBLP:conf/acl/ZhangZBW22,DBLP:conf/acl/LiWTNBAXR22,DBLP:conf/acl/TaoHZSJLLW22}. Frustratingly, we find that existing distillation methods may fail to handle task-agnostic distillation of encoder-decoder LMs since encoder-decoder LMs can behave very differently in comparison with encoder-only and decoder-only LMs~\cite[e.g., the use of cross-attention,][]{DBLP:conf/nips/VaswaniSPUJGKP17}. The failures of prior methods are showcased in Figure~\ref{fig:1}.&#10;&#10;To the end, we investigate to, in a task-agnostic style, save the distillation of encoder-decoder LMs from the awkward position. Specifically, we reveal that the key to unlocking the expressiveness of distillation is the interplay between the encoder and the decoder. Therefore, we offer a path named as MiniEnD that successfully tackles the distillation of encoder-decoder LMs by alternatively distilling the cross-attention to explicitly fall to both the encoder and the decoder. &#10;&#10;We check MiniEnD on language understanding and abstractive summarization in sense that encoder-decoder LMs are more capable of sequence-to-sequence tasks. For evaluation on language understanding, we take GLUE~\cite{WangSMHLB19} to benchmark the performance. For evaluation on abstractive summarization, we adopt CNN/DailyMail~\cite{DBLP:conf/acl/SeeLM17} and XSum~\cite{DBLP:conf/emnlp/NarayanCL18} as two testbeds. The results of both distilling T5 and BART indicate that MiniEnD is effective and competitive to other compression options such as quantization. We further scale our method up to the distillation of 3B T5\textsubscript{\sf xlarge} with the aid of progressive distillation. The results suggest that distilling large language models~\cite[e.g., LLaMA,][]{DBLP:journals/corr/abs-2302-13971} should be promising but can be challenging.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Distillation, Cognitive Science, Task-Agnostic Learning, Artificial Intelligence, Large Scale Language Models" />
        </attvalues>
      </node>
      <node id="1907.11692" label="1907.11692">
        <attvalues>
          <attvalue for="0" value="RoBERTa: A Robustly Optimized BERT Pretraining Approach" />
          <attvalue for="1" value="  Language model pretraining has led to significant performance gains but&#10;careful comparison between different approaches is challenging. Training is&#10;computationally expensive, often done on private datasets of different sizes,&#10;and, as we will show, hyperparameter choices have significant impact on the&#10;final results. We present a replication study of BERT pretraining (Devlin et&#10;al., 2019) that carefully measures the impact of many key hyperparameters and&#10;training data size. We find that BERT was significantly undertrained, and can&#10;match or exceed the performance of every model published after it. Our best&#10;model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results&#10;highlight the importance of previously overlooked design choices, and raise&#10;questions about the source of recently reported improvements. We release our&#10;models and code.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.10351" label="1909.10351">
        <attvalues>
          <attvalue for="0" value="TinyBERT: Distilling BERT for Natural Language Understanding" />
          <attvalue for="1" value="  Language model pre-training, such as BERT, has significantly improved the&#10;performances of many natural language processing tasks. However, pre-trained&#10;language models are usually computationally expensive, so it is difficult to&#10;efficiently execute them on resource-restricted devices. To accelerate&#10;inference and reduce model size while maintaining accuracy, we first propose a&#10;novel Transformer distillation method that is specially designed for knowledge&#10;distillation (KD) of the Transformer-based models. By leveraging this new KD&#10;method, the plenty of knowledge encoded in a large teacher BERT can be&#10;effectively transferred to a small student Tiny-BERT. Then, we introduce a new&#10;two-stage learning framework for TinyBERT, which performs Transformer&#10;distillation at both the pretraining and task-specific learning stages. This&#10;framework ensures that TinyBERT can capture he general-domain as well as the&#10;task-specific knowledge in BERT.&#10;  TinyBERT with 4 layers is empirically effective and achieves more than 96.8%&#10;the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x&#10;smaller and 9.4x faster on inference. TinyBERT with 4 layers is also&#10;significantly better than 4-layer state-of-the-art baselines on BERT&#10;distillation, with only about 28% parameters and about 31% inference time of&#10;them. Moreover, TinyBERT with 6 layers performs on-par with its teacher&#10;BERTBASE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;{Pre-trained Language Models Compression}&#10;Generally, pre-trained language models~(PLMs) can be compressed by low-rank approximation~\cite{ma2019tensorized,Lan2020ALBERT}, weight sharing~\cite{dehghani2018universal,Lan2020ALBERT}, knowledge distillation~\cite{tang2019distilling,sanh2019distilbert,turc2019well,sun2020mobilebert,liu2020fastbert,wang2020minilm}, pruning~\cite{cui2019fine,mccarley2019pruning,Fan2020Reducing,Elbayad2020Depth-Adaptive,gordon2020compressing,hou2020dynabert} or quantization~\cite{shen2019q,zafrir2019q8bert}. In this paper, our focus is on knowledge distillation.&#10;&#10;{Knowledge Distillation for PLMs}&#10;There have been some works trying to distill pre-trained language models~(PLMs) into smaller models. BiLSTM$_{ \hbox{SOFT}}$~\cite{tang2019distilling} distills task-specific knowledge from BERT into a single-layer BiLSTM. BERT-PKD~\cite{sun2019patient} extracts knowledges not only from the last layer of the teacher, but also from intermediate layers at fine-tuning stage. DistilBERT~\cite{sanh2019distilbert} performs distillation at pre-training stage on large-scale corpus. Concurrent works, MobileBERT~\cite{sun2020mobilebert} distills a BERT$_{\rm LARGE}$ augmented with bottleneck structures into a 24-layer slimmed version by progressive knowledge transfer at pre-training stage. MiniLM~\cite{wang2020minilm} conducts deep self-attention distillation also at pre-training stage. By contrast, we propose a new {two-stage learning} framework to distill knowledge from BERT at both pre-training and fine-tuning stages by a novel transformer distillation method. &#10;&#10;{Pretraining Lite PLMs}&#10;Other related works aim at directly pretraining lite PLMs. \cite{turc2019well} pre-trained 24 miniature BERT models and show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive. ALBERT~\cite{Lan2020ALBERT} incorporates embedding factorization and cross-layer parameter sharing to reduce model parameters. Since ALBERT does not reduce hidden size or layers of transformer block, it still has large amount of computations. Another concurrent work, ELECTRA~\cite{clark2020electra} proposes a sample-efficient task called replaced token detection to accelerate pre-training, and it also presents a 12-layer ELECTRA$_{\rm small}$ that has comparable performance with TinyBERT$_{4}$. Different from these small PLMs, TinyBERT$_{4}$ is a 4-layer model which can achieve more speedup.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2002.10957" label="2002.10957">
        <attvalues>
          <attvalue for="0" value="MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression&#10;  of Pre-Trained Transformers" />
          <attvalue for="1" value="  Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its&#10;variants) have achieved remarkable success in varieties of NLP tasks. However,&#10;these models usually consist of hundreds of millions of parameters which brings&#10;challenges for fine-tuning and online serving in real-life applications due to&#10;latency and capacity constraints. In this work, we present a simple and&#10;effective approach to compress large Transformer (Vaswani et al., 2017) based&#10;pre-trained models, termed as deep self-attention distillation. The small model&#10;(student) is trained by deeply mimicking the self-attention module, which plays&#10;a vital role in Transformer networks, of the large model (teacher).&#10;Specifically, we propose distilling the self-attention module of the last&#10;Transformer layer of the teacher, which is effective and flexible for the&#10;student. Furthermore, we introduce the scaled dot-product between values in the&#10;self-attention module as the new deep self-attention knowledge, in addition to&#10;the attention distributions (i.e., the scaled dot-product of queries and keys)&#10;that have been used in existing works. Moreover, we show that introducing a&#10;teacher assistant (Mirzadeh et al., 2019) also helps the distillation of large&#10;pre-trained Transformer models. Experimental results demonstrate that our&#10;monolingual model outperforms state-of-the-art baselines in different parameter&#10;size of student models. In particular, it retains more than 99% accuracy on&#10;SQuAD 2.0 and several GLUE benchmark tasks using 50% of the Transformer&#10;parameters and computations of the teacher model. We also obtain competitive&#10;results in applying deep self-attention distillation to multilingual&#10;pre-trained models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.09632" label="2302.09632">
        <attvalues>
          <attvalue for="0" value="HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained&#10;  Transformers" />
          <attvalue for="1" value="  Knowledge distillation has been shown to be a powerful model compression&#10;approach to facilitate the deployment of pre-trained language models in&#10;practice. This paper focuses on task-agnostic distillation. It produces a&#10;compact pre-trained model that can be easily fine-tuned on various tasks with&#10;small computational costs and memory footprints. Despite the practical&#10;benefits, task-agnostic distillation is challenging. Since the teacher model&#10;has a significantly larger capacity and stronger representation power than the&#10;student model, it is very difficult for the student to produce predictions that&#10;match the teacher's over a massive amount of open-domain training data. Such a&#10;large prediction discrepancy often diminishes the benefits of knowledge&#10;distillation. To address this challenge, we propose Homotopic Distillation&#10;(HomoDistil), a novel task-agnostic distillation approach equipped with&#10;iterative pruning. Specifically, we initialize the student model from the&#10;teacher model, and iteratively prune the student's neurons until the target&#10;width is reached. Such an approach maintains a small discrepancy between the&#10;teacher's and student's predictions throughout the distillation process, which&#10;ensures the effectiveness of knowledge transfer. Extensive experiments&#10;demonstrate that HomoDistil achieves significant improvements on existing&#10;baselines.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-trained language models have demonstrated powerful generalizability in various downstream applications \cite{wang2018glue,rajpurkar2016squad}. However, the number of parameters in such models has grown over hundreds of millions \cite{devlin2018bert,raffel2019exploring,brown2020language}. This poses a significant challenge to deploying such models in applications with latency and storage requirements. &#10;&#10;Knowledge distillation \cite{hinton2015distilling} has been shown to be a powerful technique to compress a large model (i.e., teacher model) into a small one (i.e., student model) with acceptable performance degradation. It transfers knowledge from the teacher model to the student model through regularizing the consistency between their output predictions. In language models, many efforts have been devoted to task-specific knowledge distillation \cite{tang2019distilling,turc2019well,sun2019patient,aguilar2020knowledge}. In this case, a large pre-trained model is first fine-tuned on a downstream task, and then serves as the teacher to distill a student during fine-tuning. However, task-specific distillation is computational costly because switching to a new task always requires the training of a task-specific teacher. Therefore, recent research has started to pay more attention to task-agnostic distillation \cite{sanh2019distilbert,sun2020mobilebert,jiao2019tinybert,wang2020minilm,khanuja2021mergedistill,chen2021extract}, where a student is distilled from a teacher pre-trained on open-domain data and can be efficiently fine-tuned on various downstream tasks.&#10;&#10;Despite the practical benefits, task-agnostic distillation is challenging. The teacher model has a significantly larger capacity and a much stronger representation power than the student model. As a result, it is very difficult for the student model to produce predictions that match the teacher's over a massive amount of open-domain training data, especially when the student model is not well-initialized. Such a large prediction discrepancy eventually diminishes the benefits of distillation \cite{jin2019knowledge,cho2019efficacy,mirzadeh2020improved, guo2020reducing,li2021dynamic}. To reduce this discrepancy, recent research has proposed to better initialize the student model from a subset of the teacher's layers \cite{sanh2019distilbert,jiao2019tinybert,wang2020minilm}. However, selecting such a subset requires extensive tuning.&#10;&#10;To address this challenge, we propose Homotopic Distillation ({{HomoDistil}}), a novel task-agnostic distillation approach equipped with iterative pruning. As illustrated in Figure~\ref{fig:illustration}, we initialize the student model from the teacher model. This ensures a small prediction discrepancy in the early stage of distillation. At each training iteration, we prune a set of least important neurons, which leads to the least increment in loss due to its removal, from the remaining neurons. This ensures the prediction discrepancy only increases by a small amount. Simultaneously, we distill the pruned student, such that the small discrepancy can be further reduced. We then repeat such a procedure in each iteration to maintain the small discrepancy through training, which encourages an effective knowledge transfer. &#10;&#10;We conduct extensive experiments to demonstrate the effectiveness of {{HomoDistil}} in task-agnostic distillation on BERT models. In particular, HomoBERT distilled from a BERT-base teacher ($109$M) achieves the state-of-the-art fine-tuning performance on the GLUE benchmark \cite{wang2018glue} and SQuAD v1.1/2.0 \cite{rajpurkar2016squad,rajpurkar2018know} at multiple parameter scales (e.g., $65$M and $10\sim20$M). Extensive analysis corroborates that {{HomoDistil}} maintains a small prediction discrepancy through training and produces a better-generalized student model.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1503.02531" label="1503.02531">
        <attvalues>
          <attvalue for="0" value="Distilling the Knowledge in a Neural Network" />
          <attvalue for="1" value="  A very simple way to improve the performance of almost any machine learning&#10;algorithm is to train many different models on the same data and then to&#10;average their predictions. Unfortunately, making predictions using a whole&#10;ensemble of models is cumbersome and may be too computationally expensive to&#10;allow deployment to a large number of users, especially if the individual&#10;models are large neural nets. Caruana and his collaborators have shown that it&#10;is possible to compress the knowledge in an ensemble into a single model which&#10;is much easier to deploy and we develop this approach further using a different&#10;compression technique. We achieve some surprising results on MNIST and we show&#10;that we can significantly improve the acoustic model of a heavily used&#10;commercial system by distilling the knowledge in an ensemble of models into a&#10;single model. We also introduce a new type of ensemble composed of one or more&#10;full models and many specialist models which learn to distinguish fine-grained&#10;classes that the full models confuse. Unlike a mixture of experts, these&#10;specialist models can be trained rapidly and in parallel.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.09355" label="1908.09355">
        <attvalues>
          <attvalue for="0" value="Patient Knowledge Distillation for BERT Model Compression" />
          <attvalue for="1" value="  Pre-trained language models such as BERT have proven to be highly effective&#10;for natural language processing (NLP) tasks. However, the high demand for&#10;computing resources in training such models hinders their application in&#10;practice. In order to alleviate this resource hunger in large-scale model&#10;training, we propose a Patient Knowledge Distillation approach to compress an&#10;original large model (teacher) into an equally-effective lightweight shallow&#10;network (student). Different from previous knowledge distillation methods,&#10;which only use the output from the last layer of the teacher network for&#10;distillation, our student model patiently learns from multiple intermediate&#10;layers of the teacher model for incremental knowledge extraction, following two&#10;strategies: ($i$) PKD-Last: learning from the last $k$ layers; and ($ii$)&#10;PKD-Skip: learning from every $k$ layers. These two patient distillation&#10;schemes enable the exploitation of rich information in the teacher's hidden&#10;layers, and encourage the student model to patiently learn from and imitate the&#10;teacher through a multi-layer distillation process. Empirically, this&#10;translates into improved results on multiple NLP tasks with significant gain in&#10;training efficiency, without sacrificing model accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Language Model Pre-training &#10;Pre-training has been widely applied to universal language representation learning. Previous work can be divided into two main categories: ($i$) feature-based approach; ($ii$) fine-tuning approach. &#10;&#10;Feature-based methods mainly focus on learning: &#10;($i$) context-independent word representation (e.g., word2vec~\cite{mikolov2013distributed}, GloVe~\cite{pennington2014glove}, FastText~\cite{bojanowski2017enriching}); ($ii$) sentence-level representation (e.g.,~\cite{kiros2015skip,conneau2017supervised,logeswaran2018efficient}); and ($iii$) contextualized word representation (e.g., Cove~\cite{mccann2017learned}, ELMo~\cite{peters2018deep}). Specifically, ELMo~\cite{peters2018deep} learns high-quality, deep contextualized word representation using bidirectional language model, which can be directly plugged into standard NLU models for performance boosting. &#10;&#10;On the other hand, fine-tuning approaches mainly pre-train a language model (e.g., GPT~\cite{radford2018improving}, BERT~\cite{devlin2018bert}) on a large corpus with an unsupervised objective, and then fine-tune the model with in-domain labeled data for downstream applications~\cite{dai2015semi,howard2018universal}. Specifically, BERT is a large-scale language model consisting of multiple layers of Transformer blocks~\cite{vaswani2017attention}. BERT-Base has 12 layers of Transformer and 110 million parameters, while BERT-Large has 24 layers of Transformer and 330 million parameters. &#10;By pre-training via masked language modeling and next sentence prediction, BERT has achieved state-of-the-art performance on a wide-range of NLU tasks, such as the GLUE benchmark~\cite{wang2018glue} and SQuAD~\cite{rajpurkar2016squad}.&#10;&#10;However, these modern pre-trained language models contain millions of parameters, which hinders their application in practice where computational resource is limited. In this paper, we aim at addressing this critical and challenging problem, taking BERT as an example, i.e., how to compress a large BERT model into a shallower one without sacrificing performance. Besides, the proposed approach can also be applied to other large-scale pre-trained language models, such as recently proposed XLNet~\cite{yang2019xlnet} and RoBERTa~\cite{DBLP:journals/corr/abs-1907-11692}. &#10;&#10;Model Compression \&amp; Knowledge Distillation&#10;Our focus is model compression, i.e., making deep neural networks more compact \cite{han2015deep_compression,Cheng2015circulant}. A similar line of work has focused on accelerating deep network inference at test time \cite{46320} and reducing model training time \cite{huang2016stochastic}. &#10;&#10;A conventional understanding is that a large number of connections (weights) is necessary for training deep networks \cite{Denil2013,dcnn}. However, once the network has been trained, there will be a high degree of parameter redundancy. Network pruning \cite{Han2015pruning,He_2017_ICCV}, in which&#10;network connections are reduced or sparsified, is one common strategy for model compression. Another direction is weight quantization \cite{GongLYB14,2018arXiv180205668P}, in which connection weights are constrained to a set of discrete values, allowing weights to be represented by fewer bits. However, most of these pruning and quantization approaches perform on convolutional networks. Only a few work are designed for rich structural information such as deep language models \cite{prunetransformer}.&#10;&#10;Knowledge distillation \cite{hinton2015distilling} aims to&#10;compress a network with a large set of parameters into a compact and fast-to-execute model.&#10;This can be achieved by training a compact model to imitate the soft output of a larger model. \cite{Romero15-iclr} further demonstrated that intermediate representations learned by the large model can serve as hints to improve the training process and the final performance of the compact model. \cite{Chen2016Net2NetAL} introduced techniques for efficiently transferring knowledge from an existing network to a deeper or wider network. More recently,~\cite{DBLP:journals/corr/abs-1904-09482} used knowledge from ensemble models to improve single model performance on NLU tasks.&#10;\cite{tan2019multilingual} tried knowledge distillation for multilingual translation.&#10;Different from the above efforts, &#10;we investigate the problem of compressing large-scale language models, and propose a novel patient knowledge distillation approach to effectively transferring knowledge from a teacher to a student model. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2204.00408" label="2204.00408">
        <attvalues>
          <attvalue for="0" value="Structured Pruning Learns Compact and Accurate Models" />
          <attvalue for="1" value="  The growing size of neural language models has led to increased attention in&#10;model compression. The two predominant approaches are pruning, which gradually&#10;removes weights from a pre-trained model, and distillation, which trains a&#10;smaller compact model to match a larger one. Pruning methods can significantly&#10;reduce the model size but hardly achieve large speedups as distillation.&#10;However, distillation methods require large amounts of unlabeled data and are&#10;expensive to train. In this work, we propose a task-specific structured pruning&#10;method CoFi (Coarse- and Fine-grained Pruning), which delivers highly&#10;parallelizable subnetworks and matches the distillation methods in both&#10;accuracy and latency, without resorting to any unlabeled data. Our key insight&#10;is to jointly prune coarse-grained (e.g., layers) and fine-grained (e.g., heads&#10;and hidden units) modules, which controls the pruning decision of each&#10;parameter with masks of different granularity. We also devise a layerwise&#10;distillation strategy to transfer knowledge from unpruned to pruned models&#10;during optimization. Our experiments on GLUE and SQuAD datasets show that CoFi&#10;yields models with over 10x speedups with a small accuracy drop, showing its&#10;effectiveness and efficiency compared to previous pruning and distillation&#10;approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.03923" label="2210.03923">
        <attvalues>
          <attvalue for="0" value="Sparse Teachers Can Be Dense with Knowledge" />
          <attvalue for="1" value="  Recent advances in distilling pretrained language models have discovered&#10;that, besides the expressiveness of knowledge, the student-friendliness should&#10;be taken into consideration to realize a truly knowledgable teacher. Based on a&#10;pilot study, we find that over-parameterized teachers can produce expressive&#10;yet student-unfriendly knowledge and are thus limited in overall&#10;knowledgableness. To remove the parameters that result in&#10;student-unfriendliness, we propose a sparse teacher trick under the guidance of&#10;an overall knowledgable score for each teacher parameter. The knowledgable&#10;score is essentially an interpolation of the expressiveness and&#10;student-friendliness scores. The aim is to ensure that the expressive&#10;parameters are retained while the student-unfriendly ones are removed.&#10;Extensive experiments on the GLUE benchmark show that the proposed sparse&#10;teachers can be dense with knowledge and lead to students with compelling&#10;performance in comparison with a series of competitive baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2205.14570" label="2205.14570">
        <attvalues>
          <attvalue for="0" value="MiniDisc: Minimal Distillation Schedule for Language Model Compression" />
          <attvalue for="1" value="  Recent studies have uncovered that language model distillation is less&#10;effective when facing a large capacity gap between the teacher and the student,&#10;and introduced teacher assistant-based distillation to bridge the gap. As a&#10;connection, the scale and the performance of the teacher assistant is of vital&#10;importance to bring the knowledge from the teacher to the student. However,&#10;existing teacher assistant-based methods require maximally many trials before&#10;scheduling an optimal teacher assistant. To this end, we propose a minimal&#10;distillation schedule (MiniDisc) for scheduling the optimal teacher assistant&#10;in minimally one trial. In particular, motivated by the finding that the&#10;performance of the student is positively correlated to the scale-performance&#10;tradeoff of the teacher assistant, MiniDisc is designed with a&#10;$\lambda$-tradeoff to measure the optimality of the teacher assistant without&#10;trial distillation to the student. MiniDisc then can schedule the optimal&#10;teacher assistant with the best $\lambda$-tradeoff in a sandwich framework.&#10;MiniDisc is evaluated with an extensive set of experiments on GLUE.&#10;Experimental results demonstrate the improved efficiency our MiniDisc compared&#10;to several state-of-the-art baselines. We further apply MiniDisc to a language&#10;model with billions of parameters and show its scalability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13002" label="2010.13002">
        <attvalues>
          <attvalue for="0" value="Pre-trained Summarization Distillation" />
          <attvalue for="1" value="  Recent state-of-the-art approaches to summarization utilize large pre-trained&#10;Transformer models. Distilling these models to smaller student models has&#10;become critically important for practical use; however there are many different&#10;distillation methods proposed by the NLP literature. Recent work on distilling&#10;BERT for classification and regression tasks shows strong performance using&#10;direct knowledge distillation. Alternatively, machine translation practitioners&#10;distill using pseudo-labeling, where a small model is trained on the&#10;translations of a larger model. A third, simpler approach is to 'shrink and&#10;fine-tune' (SFT), which avoids any explicit distillation by copying parameters&#10;to a smaller student model and then fine-tuning. We compare these three&#10;approaches for distillation of Pegasus and BART, the current and former state&#10;of the art, pre-trained summarization models, and find that SFT outperforms&#10;knowledge distillation and pseudo-labeling on the CNN/DailyMail dataset, but&#10;under-performs pseudo-labeling on the more abstractive XSUM dataset. PyTorch&#10;Code and checkpoints of different sizes are available through Hugging Face&#10;transformers here http://tiny.cc/4iy0tz.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Knowledge distillation is a compression technique where a smaller student model is trained to reproduce the logits of a larger teacher, rather than simply minimize the cross-entropy between the model's predicted distribution and the training labels \cite{Bucila2006ModelC, Hinton2015DistillingTK}. In a language modeling context, this allows the student model to learn a full distribution of possible next words in a given context, rather than just the next word in the training data.&#10;&#10;Recent research on KD for pre-trained models has overwhelmingly focused on distilling BERT to perform well on GLUE tasks, rather than tasks that require text generation. \cite{sanh2019distilbert} use a weighted average of KD loss and the traditional cross entropy data loss to train DistilBERT, a 6 layer distilled version of BERT, that is 60\% faster on CPU and 50\% faster on GPU.&#10;DistilBERT intializes student models by copying alternating layers. an idea we extend -- in all of our experiments, we initialize students by copying maximally spaced layers.&#10;In TinyBERT, \cite{jiao2019tinybert} add terms to the KD loss function which enforce student/teacher alignment at intermediate levels and improve performance. &#10;Bert-of-Theseus~\cite{xu2020bertoftheseus} randomly replaces multiple teacher layers with a single student layer during fine-tuning with probability $r$, such that each student layer learns to replicate 2 teacher layers. LayerDrop, a related technique, drops random parts of the teacher model during one long training run, allowing a smaller student model to be extracted at inference time \cite{layerdrop}.&#10;Distillation for Seq2Seq models has primarily used pseudo-labeling and produces strong results on machine translation, as shown in \cite{deepShallow}, \cite{junczys-dowmunt-2019-microsoft}, and \cite{sun-etal-2019-baidu}. Their approach consists of re-generating a new distilled dataset containing original source documents with pseudo-labels. The pseudo-labels are summaries generated by the teacher using beam search. After the long dataset generation process, they train a smaller student model on the ``distilled&quot; dataset. \cite{Kim_2016} call this type ``Sequence-level Knowledge Distillation&quot; in contrast to ``Word-Level Knowledge Distillation&quot;, where knowledge is transferred through logits.&#10;&#10;Recent work from \cite{liu2020noisy} presents a new method to further improve fine-tuned summarization models by fine-tuning them on their own logits with added noise. Like quantization \cite{jacob2017quantization}, this method could be used before or after the other methods in this work.&#10;&#10;Table \ref{method_delta} compares the attributes of these methods to our three approaches. Like Theseus, our experiments do not re-run pre-training. &#10;SFT is most similar to BERT-of-Theseus, and can even be described as running the Theseus procedure with $r$ fixed at 100\%, thereby saving computation. Our KD implementation is most similar to TinyBERT, and Pseudo-labels is most similar to Sequence Level Knowledge Distillation.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.03441" label="2106.03441">
        <attvalues>
          <attvalue for="0" value="Attention Temperature Matters in Abstractive Summarization Distillation" />
          <attvalue for="1" value="  Recent progress of abstractive text summarization largely relies on large&#10;pre-trained sequence-to-sequence Transformer models, which are computationally&#10;expensive. This paper aims to distill these large models into smaller ones for&#10;faster inference and minimal performance loss. Pseudo-labeling based methods&#10;are popular in sequence-to-sequence model distillation. In this paper, we find&#10;simply manipulating attention temperatures in Transformers can make pseudo&#10;labels easier to learn for student models. Our experiments on three&#10;summarization datasets show our proposed method consistently improves over&#10;vanilla pseudo-labeling based methods. We also find that both the pseudo labels&#10;and summaries produced by our students are shorter and more abstractive. Our&#10;code is available at \url{https://github.com/Shengqiang-Zhang/plate}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.10705" label="2203.10705">
        <attvalues>
          <attvalue for="0" value="Compression of Generative Pre-trained Language Models via Quantization" />
          <attvalue for="1" value="  The increasing size of generative Pre-trained Language Models (PLMs) has&#10;greatly increased the demand for model compression. Despite various methods to&#10;compress BERT or its variants, there are few attempts to compress generative&#10;PLMs, and the underlying difficulty remains unclear. In this paper, we compress&#10;generative PLMs by quantization. We find that previous quantization methods&#10;fail on generative tasks due to the \textit{homogeneous word embeddings} caused&#10;by reduced capacity, and \textit{varied distribution of weights}.&#10;Correspondingly, we propose a token-level contrastive distillation to learn&#10;distinguishable word embeddings, and a module-wise dynamic scaling to make&#10;quantizers adaptive to different modules. Empirical results on various tasks&#10;show that our proposed method outperforms the state-of-the-art compression&#10;methods on generative PLMs by a clear margin. With comparable performance with&#10;the full-precision models, we achieve 14.4x and 13.4x compression rates on&#10;GPT-2 and BART, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Compression of Generative Pre-trained Language Models.&#10;Some early explorations compress the generative pre-trained language models. &#10;KnGPT2 \cite{edalati2021kronecker} applies the Kronecker decomposition to compress the GPT.&#10;DistilGPT2 distills a 12-layer GPT-2 to a 6-layer one, which &#10;is twice as fast during inference.&#10;LightPAFF~\cite{song2020lightpaff} proposes a distillation approach &#10;that the training loss is a combination of a maximum likelihood loss of the student model, and the KL divergence between the output of teacher and student models. &#10;SpAtten \cite{wang2021spatten} proposes a sparse model with algorithm and architecture co-design, which removes uninformative tokens and attention heads. Compared with these methods, we not only study the difficulties of compression from the properties of generative tasks, but also study both decoder and encoder-decoder generative models.&#10;&#10;Quantization of Pre-trained Language Models.&#10;Quantization compresses a model by representing the 32-bit floating-point parameter with a low-bit representation, &#10;and has been widely used in various domains &#10;as it does not require designing a new model architecture.&#10;There have been many attempts to quantize task-specific BERT models~\cite{zafrir2019q8bert,shen2019q,zadeh2020gobo} with only negligible performance drop on natural language understanding tasks.&#10;Recent works~\cite{ternarybert,binarybert} even push the weight bit-width down to as low as 1-bit. &#10;Despite the success of these approaches for BERT models, attempts to quantize generative PLMs are scarce, and the underlying difficulty remains unclear.&#10;&#10;Contrastive Learning.&#10;Contrastive learning aims at pushing the representations of similar samples together while pulling those of dissimilar ones apart. and&#10;is widely used for large-scale self-supervised learning in various domains~\cite{chen2020simple,sun2020contrastive,baevski2020wav2vec, huang2022spiral}, and multi-modal learning&#10;~\cite{radford2021learning,jia2021scaling}.&#10;SimCLR~\cite{chen2020simple} directly uses other in-batch samples&#10;as negatives,&#10;and sufficient large batch size is required to work well.&#10;MoCo~\cite{he2020momentum} &#10;maintains a large number of negative samples in a queue and uses a moving average key encoder to improve consistency.&#10;Contrastive learning without negative samples is also proposed in BYOL~\cite{grill2020bootstrap} and SimSiam~\cite{chen2021exploring}. &#10;Contrastive representation distillation \cite{tian2019contrastive} distills the knowledge from the teacher network to the student network by maximizing the mutual information between them.&#10;&#10;The closest work with our token-level contrastive distillation is Wav2vec 2.0~\cite{baevski2020wav2vec},&#10;which use in-utterance representations at different positions&#10;as negatives in speech learning. &#10;Besides the difference in the modality and tasks, our method also&#10;differs from theirs in (1) Model: We quantize the model parameters and activations while they do not; (2) Representation: For each sample, we use the output of the full-precision and the quantized networks as its two views, while they use the quantized and the contextualized representation. (3) Loss: We calculate loss over all tokens in an auto-regressive manner, while they only calculate&#10;over the masked tokens non-autoregressively. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="1704.04368" label="1704.04368">
        <attvalues>
          <attvalue for="0" value="Get To The Point: Summarization with Pointer-Generator Networks" />
          <attvalue for="1" value="  Neural sequence-to-sequence models have provided a viable new approach for&#10;abstractive text summarization (meaning they are not restricted to simply&#10;selecting and rearranging passages from the original text). However, these&#10;models have two shortcomings: they are liable to reproduce factual details&#10;inaccurately, and they tend to repeat themselves. In this work we propose a&#10;novel architecture that augments the standard sequence-to-sequence attentional&#10;model in two orthogonal ways. First, we use a hybrid pointer-generator network&#10;that can copy words from the source text via pointing, which aids accurate&#10;reproduction of information, while retaining the ability to produce novel words&#10;through the generator. Second, we use coverage to keep track of what has been&#10;summarized, which discourages repetition. We apply our model to the CNN / Daily&#10;Mail summarization task, outperforming the current abstractive state-of-the-art&#10;by at least 2 ROUGE points.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.08745" label="1808.08745">
        <attvalues>
          <attvalue for="0" value="Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional&#10;  Neural Networks for Extreme Summarization" />
          <attvalue for="1" value="  We introduce extreme summarization, a new single-document summarization task&#10;which does not favor extractive strategies and calls for an abstractive&#10;modeling approach. The idea is to create a short, one-sentence news summary&#10;answering the question &quot;What is the article about?&quot;. We collect a real-world,&#10;large-scale dataset for this task by harvesting online articles from the&#10;British Broadcasting Corporation (BBC). We propose a novel abstractive model&#10;which is conditioned on the article's topics and based entirely on&#10;convolutional neural networks. We demonstrate experimentally that this&#10;architecture captures long-range dependencies in a document and recognizes&#10;pertinent content, outperforming an oracle extractive system and&#10;state-of-the-art abstractive approaches when evaluated automatically and by&#10;humans.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.10149" label="2406.10149">
        <attvalues>
          <attvalue for="0" value="BABILong: Testing the Limits of LLMs with Long Context&#10;  Reasoning-in-a-Haystack" />
          <attvalue for="1" value="  In recent years, the input context sizes of large language models (LLMs) have&#10;increased dramatically. However, existing evaluation methods have not kept&#10;pace, failing to comprehensively assess the efficiency of models in handling&#10;long contexts. To bridge this gap, we introduce the BABILong benchmark,&#10;designed to test language models' ability to reason across facts distributed in&#10;extremely long documents. BABILong includes a diverse set of 20 reasoning&#10;tasks, including fact chaining, simple induction, deduction, counting, and&#10;handling lists/sets. These tasks are challenging on their own, and even more&#10;demanding when the required facts are scattered across long natural text. Our&#10;evaluations show that popular LLMs effectively utilize only 10-20\% of the&#10;context and their performance declines sharply with increased reasoning&#10;complexity. Among alternatives to in-context reasoning, Retrieval-Augmented&#10;Generation methods achieve a modest 60\% accuracy on single-fact question&#10;answering, independent of context length. Among context extension methods, the&#10;highest performance is demonstrated by recurrent memory transformers, enabling&#10;the processing of lengths up to 11 million tokens. The BABILong benchmark is&#10;extendable to any length to support the evaluation of new upcoming models with&#10;increased capabilities, and we provide splits up to 1 million token lengths.&#10;" />
          <attvalue for="2" value="&#10;&#10;Today, large language models (LLMs) and neural architectures are continually evolving and achieving remarkable improvements, particularly in their ability to handle longer contexts~\cite{openai2023gpt4_turbo,reid2024gemini,anthropic2024claude3}. The ability of these models to process and generate text based on rich contextual information is crucial for several reasons. For example, longer contexts provide more information for the model to condition its outputs, leading to more accurate, contextually relevant, and up-to-date responses. Furthermore, long-context capabilities can enhance in-context learning by providing more in-context examples, instructions to follow, or example trajectories in context of reinforcement learning~\cite{chevalier-etal-2023-adapting, agarwal2024many, lee2024supervised}.&#10;&#10;Despite these advances in models capabilities, the benchmarks used to evaluate them have not kept pace. For example, current benchmarks such as Longbench~\cite{bai2023longbench} and L-Eval~\cite{l_eval_an2023} scale only up to 40,000 tokens, while models are capable of hundreds of thousands and millions of tokens~\cite{rodkin2024associative, reid2024gemini, bulatov2023scaling, anthropic2024claude3, liu2024world, gu2023mamba, openai2023gpt4}.&#10;&#10;Creating natural and comprehensive long-context benchmarks that are human labeled is very challenging. As a consequence, synthetic benchmarks focusing on variations of &quot;needle-in-a-haystack&quot; tasks have become increasingly common~\cite{zhang2024inftybench,liu2024world,song2024countingstars,hsieh2024ruler}. One widely used needle-in-a-haystack task involves finding specific &quot;needles with magic numbers&quot; in a haystack of Paul Graham's essays. However, the widespread use of this approach has highlighted its limitations - it is overly simplistic, and novel long context models often achieve perfect performance, as usually demonstrated by fully green heatmaps~\cite{reid2024gemini,commandr2024,liu2024world,wang2024xl3m}. This shows that while it serves well as a basic verification tool, it is not a rigorous benchmark that can effectively challenge and differentiate advanced long-context models. Another major drawback of the original default setup\footref{footnote:LLMTest} is that model predictions are evaluated and scored by an LLM (GPT-3.5-turbo) on a scale of 1 to 10, with the same single needle used for each position and document length. While averaging over multiple different needles can provide more robust results.&#10;&#10;To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets, that were designed as prerequisites for any system that aims to be capable of conversing with a human~\cite{WestonBCM15}. As a source of long natural documents we use books from PG19 corpora~\cite{rae2019compressive}. In this way, BABILong allows the construction of tasks of almost arbitrary length, in order to adapt them to the evaluation of new, more powerful models in an extensible and controllable way. We provide sets of predefined lengths with splits up to 10 million tokens, and we evaluate models on samples with up to 50 million tokens.&#10;&#10;We find that popular LLMs effectively use only 10-20\% of the context, with performance declining sharply as length and task complexity increase. Retrieval-Augmented Generation methods achieve a modest 60\% accuracy in answering single-fact questions, regardless of context length. Among other methods, Mamba and Recurrent Memory Transformers (RMT and ARMT) show the highest performance, with ARMT capable of processing lengths up to 50 million tokens.&#10;&#10;The main contributions of our work are as follows:&#10;&#10;1. We introduce BABILong, a novel scalable generative multi-task benchmark for evaluating the performance of NLP models in processing arbitrarily long documents with distributed facts.&#10;&#10;2. We evaluate over 30 recent long-input language models with various sizes, architectures, and context extension methods on BABILong.&#10;&#10;3. We find that popular LLMs effectively utilize only 10-20\% of the context, with performance degrading sharply as reasoning complexity increases. Retrieval augmented generation fails to demonstrate good scores but fine-tuning for specific task helps.&#10;&#10;4. We demonstrate successful in domain single fact question answering with the recurrent memory transformer on input texts up to 50 million tokens, which is a record for the sequence size processed by a single model.&#10;&#10;The BABILong benchmark data and code for evaluation are available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Long Context Reasoning, Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Benchmarking Large Models, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.14788" label="2305.14788">
        <attvalues>
          <attvalue for="0" value="Adapting Language Models to Compress Contexts" />
          <attvalue for="1" value="  Transformer-based language models (LMs) are powerful and widely-applicable&#10;tools, but their usefulness is constrained by a finite context window and the&#10;expensive computational cost of processing long text documents. We propose to&#10;adapt pre-trained LMs into AutoCompressors. These language models are capable&#10;of compressing long contexts into compact summary vectors, which are then&#10;accessible to the model as soft prompts. Summary vectors are trained with an&#10;unsupervised objective, whereby long documents are processed in segments, and&#10;summary vectors from all previous segments are used in language modeling. We&#10;fine-tune OPT and Llama-2 models on sequences of up to 30,720 tokens and show&#10;that AutoCompressors can utilize long contexts to improve perplexity. We&#10;evaluate AutoCompressors on in-context learning by compressing task&#10;demonstrations and find that summary vectors are good substitutes for&#10;plain-text demonstrations, increasing accuracy while reducing inference costs.&#10;Finally, we explore the benefits of pre-computing summary vectors for large&#10;corpora by applying summary vectors to retrievalaugmented language modeling and&#10;a passage re-ranking task. Overall, AutoCompressors emerge as a simple and&#10;inexpensive solution to extend the context window of LMs while speeding up&#10;inference over long contexts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.14508" label="2308.14508">
        <attvalues>
          <attvalue for="0" value="LongBench: A Bilingual, Multitask Benchmark for Long Context&#10;  Understanding" />
          <attvalue for="1" value="  Although large language models (LLMs) demonstrate impressive performance for&#10;many language tasks, most of them can only handle texts a few thousand tokens&#10;long, limiting their applications on longer sequence inputs, such as books,&#10;reports, and codebases. Recent works have proposed methods to improve LLMs'&#10;long context capabilities by extending context windows and more sophisticated&#10;memory mechanisms. However, comprehensive benchmarks tailored for evaluating&#10;long context understanding are lacking. In this paper, we introduce LongBench,&#10;the first bilingual, multi-task benchmark for long context understanding,&#10;enabling a more rigorous evaluation of long context understanding. LongBench&#10;comprises 21 datasets across 6 task categories in both English and Chinese,&#10;with an average length of 6,711 words (English) and 13,386 characters&#10;(Chinese). These tasks cover key long-text application areas including&#10;single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks,&#10;and code completion. All datasets in LongBench are standardized into a unified&#10;format, allowing for effortless automatic evaluation of LLMs. Upon&#10;comprehensive evaluation of 8 LLMs on LongBench, we find that: (1) Commercial&#10;model (GPT-3.5-Turbo-16k) outperforms other open-sourced models, but still&#10;struggles on longer contexts. (2) Scaled position embedding and fine-tuning on&#10;longer sequences lead to substantial improvement on long context understanding.&#10;(3) Context compression technique such as retrieval brings improvement for&#10;model with weak ability on long contexts, but the performance still lags behind&#10;models that have strong long context understanding capability. The code and&#10;datasets are available at https://github.com/THUDM/LongBench.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;{\bfseries Long Context Modeling Techniques}.&#10;We first discuss some popular lines of methods that aim to tackle long context understanding.&#10;These studies are mainly aimed at solving two key challenges in long text modeling, including the high runtime overhead on longer context, and the catastrophic forgetting phenomenon when processing long sequence.&#10;A series of studies focus on how to make Transformers more efficient and unforgetful~\cite{tay2022efficient}, with designs such as sparse and efficient computation~\cite{child2019generating,kitaev2020reformer,beltagy2020longformer,zaheer2020big,wang2020linformer,fedus2022switch,ding2023longnet}, recurrent and memory modules~\cite{dai2019transformer,rae2020compressive,wu2022memorizing,martins2022former,bulatov2022recurrent,orvieto2023resurrecting,liang2023unleashing,zhou2023recurrentgpt}.&#10;More recently, several methods~\cite{press2022train,sun2022length,chen2023extending} have been proposed to enable length extrapolation of Transformers, and have been adopted in the training process of long context LLMs such as ChatGLM2-32k~\cite{zeng2022glm} and LongChat-32k~\cite{longchat2023}.&#10;&#10;{\bfseries Evaluation for Long Context Understanding}.&#10;Many previous works on long text modeling rely on the perplexity metric for evaluation~\cite{beltagy2020longformer,roy2021efficient,press2022train}.&#10;However, as suggested in \cite{sun2021long}, the perplexity metric may not necessarily reflect the model's performance on sequence-level tasks in real applications&#10;Meanwhile, some works assess long text modeling through artificial tasks such as retrieval~\cite{tay2021long,chen2023extending,longchat2023}, which may also fall short in mirroring real-world scenarios.&#10;&#10;Concurrently, ZeroSCROLLS~\cite{shaham2022scrolls,shaham2023zeroscrolls} and L-Eval~\cite{an2023eval} are proposed as evaluation benchmarks for long text modeling.&#10;Nonetheless, they encompass a restricted range of task types, thereby limiting the diversity of long text modeling patterns required in the benchmarks, and consequently, the comprehensiveness of the evaluation results.&#10;Recently, AgentBench~\cite{liu2023agentbench} also mentions the challenge of LLM-as-Agent's handling long interaction trajectories, but fails to incorporate it as a dedicated evaluation dimension.&#10;In contrast, LongBench includes six major task categories, with each category featuring sequences of varying lengths, languages, and domains. We believe it provides a more holistic evaluation of long text modeling ability of large language models across a spectrum of lengths, distributions, as well as long dependency patterns.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2307.11088" label="2307.11088">
        <attvalues>
          <attvalue for="0" value="L-Eval: Instituting Standardized Evaluation for Long Context Language&#10;  Models" />
          <attvalue for="1" value="  Recently, there has been growing interest in extending the context length of&#10;large language models (LLMs), aiming to effectively process long inputs of one&#10;turn or conversations with more extensive histories. While proprietary models&#10;such as GPT-4 and Claude can largely preserve the reasoning ability in an&#10;extended context, open-source models are still progressing through the early&#10;stages of development. To bridge this gap, we propose L-Eval to institute a&#10;more standardized evaluation for long context language models (LCLMs)&#10;addressing two key aspects: dataset construction and evaluation metrics. On the&#10;one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long&#10;documents, and over 2,000 human-labeled query-response pairs encompassing&#10;diverse question styles, domains, and input length (3k$\sim$200k tokens). On&#10;the other hand, we investigate the effectiveness in evalution metrics for&#10;LCLMs. Results show that popular n-gram matching metrics generally can not&#10;correlate well with human judgment, and thus we strongly advocate for&#10;length-instruction-enhanced (LIE) evaluation and employing LLM judges. We&#10;conducted a comprehensive study of 4 popular commercial LLMs and 12 open-source&#10;counterparts using the L-Eval benchmark. Our empirical findings offer useful&#10;insights into the study of LCLMs and lay the groundwork for the development of&#10;more principled evaluation of these models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.04841" label="2407.04841">
        <attvalues>
          <attvalue for="0" value="Associative Recurrent Memory Transformer" />
          <attvalue for="1" value="  This paper addresses the challenge of creating a neural architecture for very&#10;long sequences that requires constant time for processing new information at&#10;each time step. Our approach, Associative Recurrent Memory Transformer (ARMT),&#10;is based on transformer self-attention for local context and segment-level&#10;recurrence for storage of task specific information distributed over a long&#10;context. We demonstrate that ARMT outperfors existing alternatives in&#10;associative retrieval tasks and sets a new performance record in the recent&#10;BABILong multi-task long-context benchmark by answering single-fact questions&#10;over 50 million tokens with an accuracy of 79.9%. The source code for training&#10;and evaluation is available on github.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.08268" label="2402.08268">
        <attvalues>
          <attvalue for="0" value="World Model on Million-Length Video And Language With Blockwise&#10;  RingAttention" />
          <attvalue for="1" value="  Current language models fall short in understanding aspects of the world not&#10;easily described in words, and struggle with complex, long-form tasks. Video&#10;sequences offer valuable temporal information absent in language and static&#10;images, making them attractive for joint modeling with language. Such models&#10;could develop a understanding of both human textual knowledge and the physical&#10;world, enabling broader AI capabilities for assisting humans. However, learning&#10;from millions of tokens of video and language sequences poses challenges due to&#10;memory constraints, computational complexity, and limited datasets. To address&#10;these challenges, we curate a large dataset of diverse videos and books,&#10;utilize the Blockwise RingAttention technique to scalably train on long&#10;sequences, and gradually increase context size from 4K to 1M tokens. This paper&#10;makes the following contributions: (a) Largest context size neural network: We&#10;train one of the largest context size transformers on long video and language&#10;sequences, setting new benchmarks in difficult retrieval tasks and long video&#10;understanding. (b) Solutions for overcoming vision-language training&#10;challenges, including using masked sequence packing for mixing different&#10;sequence lengths, loss weighting to balance language and vision, and&#10;model-generated QA dataset for long sequence chat. (c) A highly-optimized&#10;implementation with RingAttention, Blockwise Transformers, masked sequence&#10;packing, and other key features for training on millions-length multimodal&#10;sequences. (d) Fully open-sourced a family of 7B parameter models capable of&#10;processing long text documents (LWM-Text, LWM-Text-Chat) and videos (LWM,&#10;LWM-Chat) of over 1M tokens. This work paves the way for training on massive&#10;datasets of long video and language to develop understanding of both human&#10;knowledge and the multimodal world, and broader capabilities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.11802" label="2403.11802">
        <attvalues>
          <attvalue for="0" value="Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark&#10;  for Evaluating Long-Context Large Language Models" />
          <attvalue for="1" value="  While recent research endeavors have focused on developing Large Language&#10;Models (LLMs) with robust long-context capabilities, due to the lack of&#10;long-context benchmarks, relatively little is known about how well the&#10;performance of long-context LLMs. To address this gap, we propose a&#10;multi-evidence, position-aware, and scalable benchmark for evaluating&#10;long-context LLMs, named Counting-Stars, which evaluates long-context LLMs by&#10;using two tasks: multi-evidence acquisition and multi-evidence reasoning. Based&#10;on the Counting-Stars test, we conduct experiments to evaluate long-context&#10;LLMs (i.e., GPT-4 Turbo, Gemini 1.5 Pro, Claude3 Opus, GLM-4, and Moonshot-v1).&#10;Experimental results demonstrate that Gemini 1.5 Pro achieves the best overall&#10;results, while the performance of GPT-4 Turbo is the most stable across various&#10;tasks. Furthermore, our analysis of these LLMs, which are extended to handle&#10;long-context scenarios, indicates that there is potential for improvement as&#10;the length of the input context and the intricacy of the tasks are increasing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#09;&#09;&#09;Prior research on long-context modeling has traditionally adopted perplexity as the primary evaluation metric \cite{peng, LongLoRA}. Meanwhile, synthetic tasks (e.g., retrieval tasks) have been employed to gauge the capacity of LLMs to handle extremely long inputs \cite{Li}. However, as highlighted in \cite{Xiong}, neither perplexity scores nor performance on synthetic tasks may fully capture the effectiveness of LLMs in real-world applications. Several benchmarks proposed by \cite{longbench, L-Eval, LV-Eval, clongeval, inftybench} recently aim to evaluate long-context LLMs. &#10;&#10;A recent benchmark for testing the long-context LLMs is needle-in-a-haystack, which asks LLMs to recite the information in a “needle” sentence (“The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day”) that is inserted at a designed location in a long text. The difference between the needle-in-a-haystack and existing benchmarks is that it does not rely on specific data, especially those that may be utilized to train LLMs. In addition, the needle-in-a-haystack can be treated as a benchmark where the test data can be easily replaced to mitigate the issue of data leakage, which generally occurs in existing long-context benchmarks. As mentioned before, however, many recently released LLMs evaluate the capability of long-context handling by testing the needle-in-a-haystack, all achieving nearly perfect performance, making it impossible to distinguish the gaps between different long-context LLMs.&#10;&#10;The Counting-Stars evaluates the capabilities of multi-evidence searching and reasoning of LLMs, which should be more noteworthy in the long context modeling of LLMs, as reflected in tasks such as multi-document question answering and summarization. &#10;&#09;&#09;&#09;Concretely, the former primarily evaluates the capability of LLMs to collect multiple pieces of evidence simultaneously (distinguishing between long context and inserted sentences), while the latter tests the ability of LLMs to gather and reason various pieces of evidence at the same time correctly, that is, reasoning is required when collecting information (distinguishing between correct and incorrect evidence in inserted sentences). To the best of our knowledge, the Counting-Stars is the first scalable long-context benchmark to ask LLMs to simultaneously differentiate between correct and incorrect evidence in each inserted sentence.&#10;&#10;Furthermore, similar to the recent benchmark \cite{LooGLE}, we refer to the long-dependency tasks as those that require capturing and understanding inter-dependency across multiple pieces of evidence spanning the entire long context. Hence, the Counting-Stars can also be considered a long-dependency task when calculating scores from the sample level, i.e., one testing sample only computes one score. In addition, since sentences are pieces of evidence and distributed throughout the entire long context, it is expected that other abilities behind long-context LLMs could be analyzed, including the long-context processing strategies and attention mechanisms, which is meaningful for studying the capability of long-context LLMs.&#10;&#09;&#09;&#09;It is worth mentioning that the cost of the Counting-Stars is lower than that of the needle-in-a-haystack, which is beneficial for reducing carbon emissions.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2404.06654" label="2404.06654">
        <attvalues>
          <attvalue for="0" value="RULER: What's the Real Context Size of Your Long-Context Language&#10;  Models?" />
          <attvalue for="1" value="  The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve&#10;a piece of information (the &quot;needle&quot;) from long distractor texts (the&#10;&quot;haystack&quot;), has been widely adopted to evaluate long-context language models&#10;(LMs). However, this simple retrieval-based test is indicative of only a&#10;superficial form of long-context understanding. To provide a more comprehensive&#10;evaluation of long-context LMs, we create a new synthetic benchmark RULER with&#10;flexible configurations for customized sequence length and task complexity.&#10;RULER expands upon the vanilla NIAH test to encompass variations with diverse&#10;types and quantities of needles. Moreover, RULER introduces new task categories&#10;multi-hop tracing and aggregation to test behaviors beyond searching from&#10;context. We evaluate 17 long-context LMs with 13 representative tasks in RULER.&#10;Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all&#10;models exhibit large performance drops as the context length increases. While&#10;these models all claim context sizes of 32K tokens or greater, only half of&#10;them can maintain satisfactory performance at the length of 32K. Our analysis&#10;of Yi-34B, which supports context length of 200K, reveals large room for&#10;improvement as we increase input length and task complexity. We open source&#10;RULER to spur comprehensive evaluation of long-context LMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;Long-context Language Models. Numerous long-context language models have been introduced lately owing to the progress in engineering, architectural, and algorithmic designs. Flash attention~\cite{fa,fa2} and Ring attention~\cite{ra} significantly reduce the memory footprint required for processing long context. Various sparse attention mechanisms~\cite{sparse_attn,sparse_is_enough} such as shifted sparse attention~\cite{longlora}, dilated attention~\cite{longnet}, and attention sinks~\cite{lminfinite, streemllm} were employed to enable efficient context scaling. Novel position embedding methods were proposed to improve length extrapolation in Transformers~\cite{transformer}, including ALiBi~\cite{alibi}, xPOS~\cite{xpos}, and RoPE~\cite{rope} variants~\cite{pi,abf,yarn,e2llm,longrope,pose}. Another line of research focuses on reducing context size. This can be achieved by caching previous context using recurrence mechanism~\cite{beacon,rmt,infiniteformer,memformer}, retrieving relevant information from context~\cite{meet, landmark, longmem, focus, infllm}, or preserving the salient information via compression~\cite{longllmlingua}. Finally, novel architectures~\cite{s4,h3,hyena,longconv,retnet,xlstm,yoco} such as Mamba~\cite{mamba} and RWKV~\cite{rwkv} have also been proposed to efficiently handle long-context input. &#10;&#10;Long-context Benchmarks and Tasks. Our work is closely related to other works on benchmarking long-context language models. ZeroSCROLLS~\cite{zeroscrolls} covers ten realistic natural language tasks, such as long-document QA and (query-based) summarization. L-Eval~\cite{leval} also uses realistic data, which was filtered manually to ensure quality. LongBench~\cite{longbench} contains tasks in a bilingual setting. InfiniteBench~\cite{inftybench} includes tasks with length greater than 100K tokens. LTM~\cite{ltm} targets the evaluation of long-term conversations. To isolate the effect of parametric knowledge, previous works~\cite{bamboo,loogle} also propose to use documents posted online later than a certain cutoff date, or leverage extremely low-resource materials~\cite{mtob}. &#10;Compared to realistic benchmarks, synthetic tasks are more flexible to control the setup (e.g., sequence length and task complexity) and less affected by parametric knowledge. &#10;Recent works have primarily focused on retrieval-based synthetic tasks~\cite{needle, landmark, longchat, lostmiddle,loft}, with a few investigate other aspects, including fact reasoning~\cite{babilong, nocha}, long-range discourse modeling~\cite{sectionbreak}, question answering~\cite{flenQA, lveval}, many-shot in-context learning~\cite{manyiclgdm, manyiclcmu, lifelongicl}, and code understanding~\cite{repoQA}.&#10; " />
        </attvalues>
      </node>
      <node id="2405.17755" label="2405.17755">
        <attvalues>
          <attvalue for="0" value="XL3M: A Training-free Framework for LLM Length Extension Based on&#10;  Segment-wise Inference" />
          <attvalue for="1" value="  Length generalization failure problem, namely the large language model (LLM)&#10;fails to generalize to texts longer than its maximum training length, greatly&#10;restricts the application of LLM in the scenarios with streaming long inputs.&#10;To address this problem, the existing methods either require substantial costs&#10;or introduce precision loss. In this paper, we empirically find that the&#10;accuracy of the LLM's prediction is highly correlated to its certainty. Based&#10;on this, we propose an efficient training free framework, named XL3M (it means&#10;extra-long large language model), which enables the LLMs trained on short&#10;sequences to reason extremely long sequence without any further training or&#10;fine-tuning. Under the XL3M framework, the input context will be firstly&#10;decomposed into multiple short sub-contexts, where each sub-context contains an&#10;independent segment and a common ``question'' which is a few tokens from the&#10;end of the original context. Then XL3M gives a method to measure the relevance&#10;between each segment and the ``question'', and constructs a concise key context&#10;by splicing all the relevant segments in chronological order. The key context&#10;is further used instead of the original context to complete the inference task.&#10;Evaluations on comprehensive benchmarks show the superiority of XL3M. Using our&#10;framework, a Llama2-7B model is able to reason 20M long sequences on an 8-card&#10;Huawei Ascend 910B NPU machine with 64GB memory per card.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#09;Due to the strong demand of long sequence inference, a lot of context window extension techniques have been proposed \cite{naveed2023comprehensive,kaddour2023challenges}. &#09;&#10;&#09;These methods can be mainly divided into three categories: 1) Extension by fine-tuning; 2) Extension without fine-tuning; 3) Extension by external memory \cite{limiformer,Memorizing,InfLLM}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1502.05698" label="1502.05698">
        <attvalues>
          <attvalue for="0" value="Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks" />
          <attvalue for="1" value="  One long-term goal of machine learning research is to produce methods that&#10;are applicable to reasoning and natural language, in particular building an&#10;intelligent dialogue agent. To measure progress towards that goal, we argue for&#10;the usefulness of a set of proxy tasks that evaluate reading comprehension via&#10;question answering. Our tasks measure understanding in several ways: whether a&#10;system is able to answer questions via chaining facts, simple induction,&#10;deduction and many more. The tasks are designed to be prerequisites for any&#10;system that aims to be capable of conversing with a human. We believe many&#10;existing learning systems can currently not solve them, and hence our aim is to&#10;classify these tasks into skill sets, so that researchers can identify (and&#10;then rectify) the failings of their systems. We also extend and improve the&#10;recently introduced Memory Networks model, and show it is able to solve some,&#10;but not all, of the tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.05507" label="1911.05507">
        <attvalues>
          <attvalue for="0" value="Compressive Transformers for Long-Range Sequence Modelling" />
          <attvalue for="1" value="  We present the Compressive Transformer, an attentive sequence model which&#10;compresses past memories for long-range sequence learning. We find the&#10;Compressive Transformer obtains state-of-the-art language modelling results in&#10;the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc&#10;respectively. We also find it can model high-frequency speech effectively and&#10;can be used as a memory mechanism for RL, demonstrated on an object matching&#10;task. To promote the domain of long-range sequence learning, we propose a new&#10;open-vocabulary language modelling benchmark derived from books, PG-19.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.10715" label="2106.10715">
        <attvalues>
          <attvalue for="0" value="CPM-2: Large-scale Cost-effective Pre-trained Language Models" />
          <attvalue for="1" value="  In recent years, the size of pre-trained language models (PLMs) has grown by&#10;leaps and bounds. However, efficiency issues of these large-scale PLMs limit&#10;their utilization in real-world scenarios. We present a suite of cost-effective&#10;techniques for the use of PLMs to deal with the efficiency issues of&#10;pre-training, fine-tuning, and inference. (1) We introduce knowledge&#10;inheritance to accelerate the pre-training process by exploiting existing PLMs&#10;instead of training models from scratch. (2) We explore the best practice of&#10;prompt tuning with large-scale PLMs. Compared with conventional fine-tuning,&#10;prompt tuning significantly reduces the number of task-specific parameters. (3)&#10;We implement a new inference toolkit, namely InfMoE, for using large-scale PLMs&#10;with limited computational resources. Based on our cost-effective pipeline, we&#10;pre-train two models: an encoder-decoder bilingual model with 11 billion&#10;parameters (CPM-2) and its corresponding MoE version with 198 billion&#10;parameters. In our experiments, we compare CPM-2 with mT5 on downstream tasks.&#10;Experimental results show that CPM-2 has excellent general language&#10;intelligence. Moreover, we validate the efficiency of InfMoE when conducting&#10;inference of large-scale models having tens of billions of parameters on a&#10;single GPU. All source code and model parameters are available at&#10;https://github.com/TsinghuaAI/CPM.&#10;" />
          <attvalue for="2" value="&#10;&#10;&#10;&#10;&#10;Training much larger models is an important research direction in deep learning~\cite{DBLP:conf/slsp/Bengio13}. Recently, pre-training has become the mainstream technique to develop large-scale neural networks and achieved great success in both computer vision (CV) and natural language processing (NLP)~\cite{ResNet,ViT,BERT}. Especially, there are some much larger pre-trained language models (PLMs) with hundreds of billions of parameters, such as GPT-3~\cite{brown2020language}, PANGU-$\alpha$~\cite{zeng2021pangu}, and Switch-Transformer~\cite{fedus2021switch}.&#10;&#10;However, the cost of using PLMs is increasing rapidly with the growth of model sizes and becomes unaffordable for most users and researchers. The cost consists of three parts. (1)~Large computation cost for pre-training: a super large model requires several weeks of pre-training with thousands of GPUs. (2)~Large storage cost for fine-tuned models: a super large model usually takes hundreds of gigabytes (GBs) to store, and we need to store as many models as downstream tasks. (3)~Strict equipment requirement for inference: it is common to use multiple GPUs for the inference of a super large model, so these models are hard to be used with limited computation resources.&#10;&#10;To reduce the cost of large-scale PLMs from its pre-training to fine-tuning, we try to improve the whole pipeline of developing PLMs as follows:&#10;&#10;(1) We adopt knowledge inheritance~\cite{qin2021ki} to accelerate the pre-training process. Current PLMs are usually trained from scratch on pre-training data via self-supervised methods, while there exist many PLMs that can also provide much knowledge. Knowledge inheritance aims to use the knowledge of existing PLMs to help the pre-training of new models.&#10;&#10;(2) We use prompt tuning~\cite{prompt-tuning} instead of fine-tuning to reduce the storage of task-specific parameters. With prompt tuning, we only need to save the embeddings of prompt tokens, whose parameters are usually less than $0.01\%$ of the whole model parameters.&#10;&#10;(3) We design a high-performance and memory-efficient inference framework InfMoE with a dynamically-scheduled offloading strategy, to support the inference of MoE models on a single GPU.&#10;&#10;Based on our optimized pipeline for PLMs, we develop two large-scale Cost-efficient Pre-trained language Models (CPM-2), an Chinese-English bilingual models with $11$ billion parameters and its Mixture-of-Experts (MoE) version with $198$ billion parameters. Specifically, we accelerate the pre-training process by dividing the pre-training process into three stages with knowledge inheritance: Chinese pre-training, bilingual pre-training, and MoE pre-training. Then, we compare CPM-2 with mT5~\cite{xue2020mt5}. Experimental results show that CPM-2 has excellent general language intelligence, including seven specific language capabilities. Based on CPM-2, we search for the best practice of prompt tuning. We find that (1) the positions of prompts are crucial and (2) combining prompt tuning and fine-tuning can lead to better results. Finally, we introduce InfMoE for users to conduct inference of large-scale models with tens of billions of parameters on a single GPU.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large-scale Model Inference, Information Technology, Linguistics, Mathematics, Language Model Efficiency, Pre-training Techniques" />
        </attvalues>
      </node>
      <node id="2010.11929" label="2010.11929">
        <attvalues>
          <attvalue for="0" value="An Image is Worth 16x16 Words: Transformers for Image Recognition at&#10;  Scale" />
          <attvalue for="1" value="  While the Transformer architecture has become the de-facto standard for&#10;natural language processing tasks, its applications to computer vision remain&#10;limited. In vision, attention is either applied in conjunction with&#10;convolutional networks, or used to replace certain components of convolutional&#10;networks while keeping their overall structure in place. We show that this&#10;reliance on CNNs is not necessary and a pure transformer applied directly to&#10;sequences of image patches can perform very well on image classification tasks.&#10;When pre-trained on large amounts of data and transferred to multiple mid-sized&#10;or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision&#10;Transformer (ViT) attains excellent results compared to state-of-the-art&#10;convolutional networks while requiring substantially fewer computational&#10;resources to train.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.12369" label="2104.12369">
        <attvalues>
          <attvalue for="0" value="PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language&#10;  Models with Auto-parallel Computation" />
          <attvalue for="1" value="  Large-scale Pretrained Language Models (PLMs) have become the new paradigm&#10;for Natural Language Processing (NLP). PLMs with hundreds of billions&#10;parameters such as GPT-3 have demonstrated strong performances on natural&#10;language understanding and generation with \textit{few-shot in-context}&#10;learning. In this work, we present our practice on training large-scale&#10;autoregressive language models named PanGu-$\alpha$, with up to 200 billion&#10;parameters. PanGu-$\alpha$ is developed under the MindSpore and trained on a&#10;cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is&#10;implemented based on MindSpore Auto-parallel, which composes five parallelism&#10;dimensions to scale the training task to 2048 processors efficiently, including&#10;data parallelism, op-level model parallelism, pipeline model parallelism,&#10;optimizer model parallelism and rematerialization. To enhance the&#10;generalization ability of PanGu-$\alpha$, we collect 1.1TB high-quality Chinese&#10;data from a wide range of domains to pretrain the model. We empirically test&#10;the generation ability of PanGu-$\alpha$ in various scenarios including text&#10;summarization, question answering, dialogue generation, etc. Moreover, we&#10;investigate the effect of model scales on the few-shot performances across a&#10;broad range of Chinese NLP tasks. The experimental results demonstrate the&#10;superior capabilities of PanGu-$\alpha$ in performing various tasks under&#10;few-shot or zero-shot settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.03961" label="2101.03961">
        <attvalues>
          <attvalue for="0" value="Switch Transformers: Scaling to Trillion Parameter Models with Simple&#10;  and Efficient Sparsity" />
          <attvalue for="1" value="  In deep learning, models typically reuse the same parameters for all inputs.&#10;Mixture of Experts (MoE) defies this and instead selects different parameters&#10;for each incoming example. The result is a sparsely-activated model -- with&#10;outrageous numbers of parameters -- but a constant computational cost. However,&#10;despite several notable successes of MoE, widespread adoption has been hindered&#10;by complexity, communication costs and training instability -- we address these&#10;with the Switch Transformer. We simplify the MoE routing algorithm and design&#10;intuitive improved models with reduced communication and computational costs.&#10;Our proposed training techniques help wrangle the instabilities and we show&#10;large sparse models may be trained, for the first time, with lower precision&#10;(bfloat16) formats. We design models based off T5-Base and T5-Large to obtain&#10;up to 7x increases in pre-training speed with the same computational resources.&#10;These improvements extend into multilingual settings where we measure gains&#10;over the mT5-Base version across all 101 languages. Finally, we advance the&#10;current scale of language models by pre-training up to trillion parameter&#10;models on the &quot;Colossal Clean Crawled Corpus&quot; and achieve a 4x speedup over the&#10;T5-XXL model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11934" label="2010.11934">
        <attvalues>
          <attvalue for="0" value="mT5: A massively multilingual pre-trained text-to-text transformer" />
          <attvalue for="1" value="  The recent &quot;Text-to-Text Transfer Transformer&quot; (T5) leveraged a unified&#10;text-to-text format and scale to attain state-of-the-art results on a wide&#10;variety of English-language NLP tasks. In this paper, we introduce mT5, a&#10;multilingual variant of T5 that was pre-trained on a new Common Crawl-based&#10;dataset covering 101 languages. We detail the design and modified training of&#10;mT5 and demonstrate its state-of-the-art performance on many multilingual&#10;benchmarks. We also describe a simple technique to prevent &quot;accidental&#10;translation&quot; in the zero-shot setting, where a generative model chooses to&#10;(partially) translate its prediction into the wrong language. All of the code&#10;and model checkpoints used in this work are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.01364" label="2402.01364">
        <attvalues>
          <attvalue for="0" value="Continual Learning for Large Language Models: A Survey" />
          <attvalue for="1" value="  Large language models (LLMs) are not amenable to frequent re-training, due to&#10;high training costs arising from their massive scale. However, updates are&#10;necessary to endow LLMs with new skills and keep them up-to-date with rapidly&#10;evolving human knowledge. This paper surveys recent works on continual learning&#10;for LLMs. Due to the unique nature of LLMs, we catalog continue learning&#10;techniques in a novel multi-staged categorization scheme, involving continual&#10;pretraining, instruction tuning, and alignment. We contrast continual learning&#10;for LLMs with simpler adaptation methods used in smaller models, as well as&#10;with other enhancement strategies like retrieval-augmented generation and model&#10;editing. Moreover, informed by a discussion of benchmarks and evaluation, we&#10;identify several challenges and future work directions for this crucial task.&#10;" />
          <attvalue for="2" value="&#10;Recent years have witnessed the rapid advances of large language models' (LLMs) capabilities in solving a diverse range of problems. At the same time, it is vital for LLMs to be regularly updated to accurately reflect the ever-evolving human knowledge, values and linguistic patterns, calling for the investigation of continual learning for LLMs. &#10;Whilst continual learning bears some resemblance to other strategies for model improvements, such as retrieval-augmented generation (RAG)&#10;~\cite{LewisPPPKGKLYR020} and model editing ~\cite{yao-etal-2023-editing}, their main purposes differ (\ref{tab:comp}). &#10;Unlike these strategies, whose primarily focus is on refining the domain-specific accuracy or expanding the model's factual knowledge base, continual learning aims to enhance the overall linguistic and reasoning capabilities of LLMs. This distinction is crucial as it shifts the focus from merely updating information to developing a model's ability to process and generate language in a more comprehensive and nuanced manner~\cite{ZhangFCNW23}.&#10;&#10;Continual learning for LLMs also differs from its use in smaller models, including smaller pre-trained language models (PLMs). Due to their vast size and complexity, LLMs require a multi-faceted approach to continual learning. We categorise it into three different stages, i.e.\ continual pretraining to expand the model's fundamental understanding of language~\cite{JinZZ00WA022},&#10;continual instruction tuning to improve the model's response to specific user commands~\cite{zhang2023citb}, and continual alignment to ensure the model's outputs adhere to values, ethical standards and societal norms~\cite{zhang2023copf}. This multi-stage process is distinct from the more linear adaptation strategies used in smaller models, as illustrated in \ref{fig:cl4llm}, highlighting the unique challenges and requirements of applying continual learning to LLMs. &#10;&#10;This survey differentiates itself from previous studies by its unique focus and structure. While previous surveys in the field are typically organized around various continual learning strategies~\cite{biesialska-etal-2020-continual}, ours is the first to specifically address continual learning in the context of LLMs. We structure our analysis around the types of information that is updated continually and the distinct stages of learning involved in LLMs. This survey offers a detailed and novel perspective on how continual learning is applied to LLMs, shedding light on the specific challenges and opportunities of this application. Our goal is to provide a thorough understanding of the effective implementation of continual learning in LLMs, contributing to the development of more advanced and adaptable language models in the future.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Continual Learning, Model Updates, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2305.13172" label="2305.13172">
        <attvalues>
          <attvalue for="0" value="Editing Large Language Models: Problems, Methods, and Opportunities" />
          <attvalue for="1" value="  Despite the ability to train capable LLMs, the methodology for maintaining&#10;their relevancy and rectifying errors remains elusive. To this end, the past&#10;few years have witnessed a surge in techniques for editing LLMs, the objective&#10;of which is to efficiently alter the behavior of LLMs within a specific domain&#10;without negatively impacting performance across other inputs. This paper&#10;embarks on a deep exploration of the problems, methods, and opportunities&#10;related to model editing for LLMs. In particular, we provide an exhaustive&#10;overview of the task definition and challenges associated with model editing,&#10;along with an in-depth empirical analysis of the most progressive methods&#10;currently at our disposal. We also build a new benchmark dataset to facilitate&#10;a more robust evaluation and pinpoint enduring issues intrinsic to existing&#10;techniques. Our objective is to provide valuable insights into the&#10;effectiveness and feasibility of each editing technique, thereby assisting the&#10;community in making informed decisions on the selection of the most appropriate&#10;method for a specific task or context. Code and datasets are available at&#10;https://github.com/zjunlp/EasyEdit.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have demonstrated a remarkable capacity for understanding and generating human-like text \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2303-08774,anil2023palm,DBLP:journals/corr/abs-2302-13971,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2303-18223}. &#10;Despite the proficiency in training LLMs, the strategies for ensuring their relevance and fixing their bugs remain unclear.&#10;Ideally, as the world's state evolves, we aim to update LLMs in a way that sidesteps the computational burden associated with training a wholly new model. &#10;As shown in Figure \ref{fig:intro}, to address this issue, the concept of model editing has been proposed \cite{Sinitsin2020Editable,de-cao-etal-2021-editing}, enabling data-efficient alterations to the behavior of models, specifically within a designated realm of interest, while ensuring no adverse impact on other inputs.&#10;&#10;Currently, numerous works on model editing for LLMs \cite{de-cao-etal-2021-editing,meng2022locating,meng2023massediting,Sinitsin2020Editable,huang2023transformerpatcher} have made strides in various editing tasks and settings. &#10;As illustrated in Figure~\ref{fig:overview}, these works manipulate the model's output for specific cases by either integrating an auxiliary network with the original unchanged model or altering the model parameters responsible for the undesirable output.&#10;Despite the wide range of model editing techniques present in the literature, a comprehensive comparative analysis, assessing these methods in uniform experimental conditions, is notably lacking.&#10;This absence of direct comparison impairs our ability to discern the relative merits and demerits of each approach, consequently hindering our comprehension of their adaptability across different problem domains.&#10;&#10;To confront this issue, the present study endeavors to establish a standard problem definition accompanied by a meticulous appraisal of these methods (\S{\ref{sec:definition}}, \S{\ref{sec:method}}). &#10;We conduct experiments under regulated conditions, fostering an impartial comparison of their respective strengths and weaknesses (\S{\ref{sec:preliminary}}). &#10;We initially use two popular model editing datasets, ZsRE ~\cite{levy-etal-2017-zero} and \cf~\cite{meng2022locating}, and two structurally different language models, T5 \cite{DBLP:journals/jmlr/RaffelSRLNMZLL20} (encoder-decoder) and GPT-J \cite{wang2021gpt} (decoder only), as our base models.&#10;We also evaluate the performance of larger models, OPT-13B~\cite{DBLP:journals/corr/abs-2205-01068} and GPT-NEOX-20B~\cite{black2022gptneox20b}.&#10;Beyond basic edit settings, we assess performance for batch and sequential editing. &#10;While we observe that current methods have demonstrated considerable capacity in factual model editing tasks, we reconsider the current evaluation and create a more encompassing evaluation dataset(\S{\ref{sec:comprehensive}}): portability (robust generalization capabilities), locality (side effect), and efficiency (time and memory usage).&#10;We find current model editing methods are somewhat limited on these levels, thereby constraining their practical application, and deserve more research in the future.&#10;Through systematic evaluation, we aim to provide valuable insights on each model editing technique's effectiveness, aiding researchers in choosing the appropriate method for specific tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.07343" label="2310.07343">
        <attvalues>
          <attvalue for="0" value="How Do Large Language Models Capture the Ever-changing World Knowledge?&#10;  A Review of Recent Advances" />
          <attvalue for="1" value="  Although large language models (LLMs) are impressive in solving various&#10;tasks, they can quickly be outdated after deployment. Maintaining their&#10;up-to-date status is a pressing concern in the current era. This paper provides&#10;a comprehensive review of recent advances in aligning LLMs with the&#10;ever-changing world knowledge without re-training from scratch. We categorize&#10;research works systemically and provide in-depth comparisons and discussion. We&#10;also discuss existing challenges and highlight future directions to facilitate&#10;research in this field. We release the paper list at&#10;https://github.com/hyintell/awesome-refreshing-llms&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08534" label="2110.08534">
        <attvalues>
          <attvalue for="0" value="Lifelong Pretraining: Continually Adapting Language Models to Emerging&#10;  Corpora" />
          <attvalue for="1" value="  Pretrained language models (PTLMs) are typically learned over a large, static&#10;corpus and further fine-tuned for various downstream tasks. However, when&#10;deployed in the real world, a PTLM-based model must deal with data&#10;distributions that deviate from what the PTLM was initially trained on. In this&#10;paper, we study a lifelong language model pretraining challenge where a PTLM is&#10;continually updated so as to adapt to emerging data. Over a domain-incremental&#10;research paper stream and a chronologically-ordered tweet stream, we&#10;incrementally pretrain a PTLM with different continual learning algorithms, and&#10;keep track of the downstream task performance (after fine-tuning). We evaluate&#10;PTLM's ability to adapt to new corpora while retaining learned knowledge in&#10;earlier corpora. Our experiments show distillation-based approaches to be most&#10;effective in retaining downstream performance in earlier domains. The&#10;algorithms also improve knowledge transfer, allowing models to achieve better&#10;downstream performance over the latest data, and improve temporal&#10;generalization when distribution gaps exist between training and evaluation&#10;because of time. We believe our problem formulation, methods, and analysis will&#10;inspire future studies towards continual pretraining of language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Domain and Temporal Adaptation of Language Models. &#10;~\cite{Gururangan2020DontSP} study adaptation of PTLMs to domain-specific corpora. ~\cite{Arumae2020AnEI} study algorithms to mitigate forgetting in original PTLMs, but does not investigate forgetting that happens over a sequence of domains. \cite{Maronikolakis2021MultidomainPL,Rttger2021TemporalAO, luu2021time} proposes sequential pretraining over domains or emerging data, but did not investigate CL algorithms. Several recent studies have demonstrated the necessity of adapting LMs over time~\cite{Lazaridou2021PitfallsOS} while specifically focusing on factual knowledge~\cite{Dhingra2021TimeAwareLM, jang2021towards}.&#10;&#10;Continual Learning Algorithms in NLP.&#10;Continual learning in NLP has mainly been studied for classification tasks. An effective approach is to utilize a number of stored past examples~\cite{dAutume2019EpisodicMI,Wang2020EfficientML}, or pseudo examples (e.g., the ones generated with a PTLM~\cite{Sun2020LAMOLLM,Kanwatchara2021RationalLA}). Recent extensions of the algorithm~\cite{Chuang2020LifelongLK} perform knowledge distillation with generated pseudo examples. Other lines of works focus on regularization over the sentence representations~\cite{Wang2019SentenceEA,Huang2021ContinualLF,Liu2019ContinualLF} or directly merging models in the parameter space~\cite{Matena2021MergingMW}. Model expansion-based approaches~\cite{Liu2019ContinualLF,Pfeiffer2021AdapterFusionNT}, including learning domain specific expert models~\cite{Gururangan2021DEMixLD}, are also actively studied.~\cite{wu2022pretrained} present a comparative study of algorithms in the context of continual fine-tuning over NLP tasks.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2310.14510" label="2310.14510">
        <attvalues>
          <attvalue for="0" value="CITB: A Benchmark for Continual Instruction Tuning" />
          <attvalue for="1" value="  Continual learning (CL) is a paradigm that aims to replicate the human&#10;ability to learn and accumulate knowledge continually without forgetting&#10;previous knowledge and transferring it to new tasks. Recent instruction tuning&#10;(IT) involves fine-tuning models to make them more adaptable to solving NLP&#10;tasks in general. However, it is still uncertain how instruction tuning works&#10;in the context of CL tasks. This challenging yet practical problem is&#10;formulated as Continual Instruction Tuning (CIT). In this work, we establish a&#10;CIT benchmark consisting of learning and evaluation protocols. We curate two&#10;long dialogue task streams of different types, InstrDialog and InstrDialog++,&#10;to study various CL methods systematically. Our experiments show that existing&#10;CL methods do not effectively leverage the rich natural language instructions,&#10;and fine-tuning an instruction-tuned model sequentially can yield similar or&#10;better results. We further explore different aspects that might affect the&#10;learning of CIT. We hope this benchmark will facilitate more research in this&#10;direction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.09823" label="2012.09823">
        <attvalues>
          <attvalue for="0" value="Continual Lifelong Learning in Natural Language Processing: A Survey" />
          <attvalue for="1" value="  Continual learning (CL) aims to enable information systems to learn from a&#10;continuous data stream across time. However, it is difficult for existing deep&#10;learning architectures to learn a new task without largely forgetting&#10;previously acquired knowledge. Furthermore, CL is particularly challenging for&#10;language learning, as natural language is ambiguous: it is discrete,&#10;compositional, and its meaning is context-dependent. In this work, we look at&#10;the problem of CL through the lens of various NLP tasks. Our survey discusses&#10;major challenges in CL and current methods applied in neural network models. We&#10;also provide a critical review of the existing CL evaluation methods and&#10;datasets in NLP. Finally, we present our outlook on future research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.04434" label="2405.04434">
        <attvalues>
          <attvalue for="0" value="DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts&#10;  Language Model" />
          <attvalue for="1" value="  We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model&#10;characterized by economical training and efficient inference. It comprises 236B&#10;total parameters, of which 21B are activated for each token, and supports a&#10;context length of 128K tokens. DeepSeek-V2 adopts innovative architectures&#10;including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees&#10;efficient inference through significantly compressing the Key-Value (KV) cache&#10;into a latent vector, while DeepSeekMoE enables training strong models at an&#10;economical cost through sparse computation. Compared with DeepSeek 67B,&#10;DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves&#10;42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum&#10;generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality&#10;and multi-source corpus consisting of 8.1T tokens, and further perform&#10;Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock&#10;its potential. Evaluation results show that, even with only 21B activated&#10;parameters, DeepSeek-V2 and its chat versions still achieve top-tier&#10;performance among open-source models.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the past few years, Large Language Models~(LLMs)~\cite{chatgpt,gpt4,claude,gemini} have undergone rapid development, offering a glimpse into the dawn of Artificial General Intelligence~(AGI). &#10;In general, the intelligence of an LLM tends to improve as the number of parameters increases, allowing it to exhibit emergent capabilities across various tasks~\cite{wei2022emergent}. &#10;However, the improvement comes at the cost of larger computing resources for training and a potential decrease in inference throughput. &#10;These constraints present significant challenges that impede the widespread adoption and utilization of LLMs.&#10;In order to tackle this problem, we introduce \dsvii, a strong open-source Mixture-of-Experts~(MoE) language model, characterized by economical training and efficient inference through an innovative Transformer architecture. &#10;It is equipped with a total of 236B parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. &#10;&#10;We optimize the attention modules and Feed-Forward Networks~(FFNs) within the Transformer framework~\cite{transformer} with our proposed Multi-head Latent Attention~(\dsattn) and \dsmoe.&#10;(1)&#10;In the context of attention mechanisms, the Key-Value (KV) cache of the Multi-Head Attention (MHA)~\cite{transformer} poses a significant obstacle to the inference efficiency of LLMs. &#10;Various approaches have been explored to address this issue, including Grouped-Query Attention (GQA)~\cite{ainslie2023gqa} and Multi-Query Attention (MQA)~\cite{mqa}. &#10;However, these methods often compromise performance in their attempt to reduce the KV cache. &#10;In order to achieve the best of both worlds, we introduce \dsattn, an attention mechanism equipped with low-rank key-value joint compression. &#10;Empirically, \dsattn achieves superior performance compared with MHA, and meanwhile significantly reduces the KV cache during inference, thus boosting the inference efficiency.&#10;(2)&#10;For Feed-Forward Networks~(FFNs), we follow the \dsmoe architecture~\cite{deepseekmoe}, which adopts fine-grained expert segmentation and shared expert isolation for higher potential in expert specialization. &#10;The \dsmoe architecture demonstrates great advantages compared with conventional MoE architectures like GShard~\cite{gshard}, enabling us to train strong models at an economical cost. &#10;As we employ expert parallelism during training, we also devise supplementary mechanisms to control communication overheads and ensure load balance. &#10;By combining these two techniques, \dsvii features strong performance (Figure~\ref{fig:mmlu}), economical training costs, and efficient inference throughput (Figure~\ref{fig:efficiency}), simultaneously. &#10;&#10;We construct a high-quality and multi-source pre-training corpus consisting of 8.1T tokens. &#10;Compared with the corpus used in \dsvi~(our previous release)~\cite{deepseek1}, this corpus features an extended amount of data, especially Chinese data, and higher data quality. &#10;We first pretrain \dsvii on the full pre-training corpus. &#10;Then, we collect 1.5M conversational sessions, which encompass various domains such as math, code, writing, reasoning, safety, and more, to perform Supervised Fine-Tuning~(SFT) for \dsviisft. &#10;Finally, we follow DeepSeekMath~\cite{deepseekmath} to employ Group Relative Policy Optimization~(GRPO) to further align the model with human preference and produce \dsviirl.&#10;&#10;We evaluate \dsvii on a wide range of benchmarks in English and Chinese, and compare it with representative open-source models.&#10;Evaluation results show that even with only 21B activated parameters, DeepSeek-V2 still achieves top-tier performance among open-source models and becomes the strongest open-source MoE language model. &#10;Figure~\ref{fig:mmlu} highlights that, on MMLU, \dsvii achieves top-ranking performance with only a small number of activated parameters. &#10;In addition, as shown in Figure~\ref{fig:efficiency}, compared with \dsvi, \dsvii saves 42.5\% of training costs, reduces the KV cache by 93.3\%, and boosts the maximum generation throughput to 5.76 times. &#10;We also evaluate \dsviisft and \dsviirl on open-ended benchmarks. &#10;Notably, \dsviirl achieves 38.9 length-controlled win rate on AlpacaEval 2.0 \cite{alpaca2.0}, 8.97 overall score on MT-Bench \cite{mtbench}, and 7.91 overall score on AlignBench \cite{align_bench}. &#10;The English open-ended conversation evaluations demonstrate that \dsviirl has top-tier performance among open-source chat models. &#10;In addition, the evaluation on AlignBench indicates that in Chinese, \dsviirl outperforms all of open-source models, and even beats most of closed-source models. &#10;&#10;In order to facilitate further research and development on MLA and DeepSeekMoE, we also release DeepSeek-V2-Lite, a smaller model equipped with MLA and DeepSeekMoE, for the open-source community. &#10;It has a total of 15.7B parameters, where 2.4B are activated for each token. &#10;Detailed descriptions about DeepSeek-V2-Lite can be found in Appendix~\ref{app:dsviilite}.&#10;&#10;In the rest of this paper, we first provide a detailed description of the model architecture of \dsvii (Section~\ref{sec:arch}). &#10;Subsequently, we introduce our pre-training endeavors, including the training data construction, hyper-parameter settings, infrastructures, long context extension, and the evaluation of model performance and efficiency (Section~\ref{sec:pre-training}). &#10;Following this, we demonstrate our efforts in alignment, encompassing Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), the evaluation results, and other discussion (Section~\ref{sec:alignment}). &#10;Finally, we summarize the conclusion, deliberate on the current limitations of \dsvii, and outline our future work (Section~\ref{sec:conclusion}). &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Architecture, Efficient Training Methods, Mathematics, Large-Scale Model Evaluation" />
        </attvalues>
      </node>
      <node id="2305.13245" label="2305.13245">
        <attvalues>
          <attvalue for="0" value="GQA: Training Generalized Multi-Query Transformer Models from Multi-Head&#10;  Checkpoints" />
          <attvalue for="1" value="  Multi-query attention (MQA), which only uses a single key-value head,&#10;drastically speeds up decoder inference. However, MQA can lead to quality&#10;degradation, and moreover it may not be desirable to train a separate model&#10;just for faster inference. We (1) propose a recipe for uptraining existing&#10;multi-head language model checkpoints into models with MQA using 5% of original&#10;pre-training compute, and (2) introduce grouped-query attention (GQA), a&#10;generalization of multi-query attention which uses an intermediate (more than&#10;one, less than number of query heads) number of key-value heads. We show that&#10;uptrained GQA achieves quality close to multi-head attention with comparable&#10;speed to MQA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;This work is focused on achieving a better trade-off between decoder quality and inference time through reducing the memory bandwidth overhead~\cite{roofline} from loading keys and values. \cite{shazeer2019mq} first proposed reducing this overhead through multi-query attention. Follow-up work showed that multi-query attention is especially helpful for long inputs~\cite{palminference, dejong2022fido}. \cite{gqamrabe} independently developed GQA with public implementation. Other works have explored grouping attention heads for computational efficiency~\cite{grouptrans, gmha, pillars} without focusing specifically on key-value heads, which determine memory bandwidth overhead.&#10;&#10;A number of other methods have been proposed to reduce memory bandwidth overhead from keys and values, as well as parameters. Flash attention~\cite{flashattention} structures the attention computation to avoid materializing the quadratic attention scores, reducing memory and speeding up training. Quantization~\cite{int8, gptq} reduces the size of weights and activations, including keys and values, by lowering precision. Model distillation~\cite{distill, distillsurvey} instead reduces model size at a given precision, using data generated from the larger model to finetune the smaller model. Layer-sparse cross-attention \cite{dejong2022fido} eliminates most cross-attention layers which make up the primary expense for longer inputs. Speculative sampling~\cite{specchen, specleviathan} ameliorates the memory bandwidth bottleneck by proposing multiple tokens with a smaller model which are then scored in parallel by a larger model. &#10;&#10;Finally, the uptraining procedure we propose is inspired by \cite{sparsemoe}, which uptrains standard T5 checkpoints into sparsely activated Mixture-of-Experts models.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1911.02150" label="1911.02150">
        <attvalues>
          <attvalue for="0" value="Fast Transformer Decoding: One Write-Head is All You Need" />
          <attvalue for="1" value="  Multi-head attention layers, as used in the Transformer neural sequence&#10;model, are a powerful alternative to RNNs for moving information across and&#10;between sequences. While training these layers is generally fast and simple,&#10;due to parallelizability across the length of the sequence, incremental&#10;inference (where such paralleization is impossible) is often slow, due to the&#10;memory-bandwidth cost of repeatedly loading the large &quot;keys&quot; and &quot;values&quot;&#10;tensors. We propose a variant called multi-query attention, where the keys and&#10;values are shared across all of the different attention &quot;heads&quot;, greatly&#10;reducing the size of these tensors and hence the memory bandwidth requirements&#10;of incremental decoding. We verify experimentally that the resulting models can&#10;indeed be much faster to decode, and incur only minor quality degradation from&#10;the baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.06066" label="2401.06066">
        <attvalues>
          <attvalue for="0" value="DeepSeekMoE: Towards Ultimate Expert Specialization in&#10;  Mixture-of-Experts Language Models" />
          <attvalue for="1" value="  In the era of large language models, Mixture-of-Experts (MoE) is a promising&#10;architecture for managing computational costs when scaling up model parameters.&#10;However, conventional MoE architectures like GShard, which activate the top-$K$&#10;out of $N$ experts, face challenges in ensuring expert specialization, i.e.&#10;each expert acquires non-overlapping and focused knowledge. In response, we&#10;propose the DeepSeekMoE architecture towards ultimate expert specialization. It&#10;involves two principal strategies: (1) finely segmenting the experts into $mN$&#10;ones and activating $mK$ from them, allowing for a more flexible combination of&#10;activated experts; (2) isolating $K_s$ experts as shared ones, aiming at&#10;capturing common knowledge and mitigating redundancy in routed experts.&#10;Starting from a modest scale with 2B parameters, we demonstrate that&#10;DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5&#10;times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly&#10;approaches the performance of its dense counterpart with the same number of&#10;total parameters, which set the upper bound of MoE models. Subsequently, we&#10;scale up DeepSeekMoE to 16B parameters and show that it achieves comparable&#10;performance with LLaMA2 7B, with only about 40% of computations. Further, our&#10;preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently&#10;validate its substantial advantages over the GShard architecture, and show its&#10;performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%)&#10;of computations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.02954" label="2401.02954">
        <attvalues>
          <attvalue for="0" value="DeepSeek LLM: Scaling Open-Source Language Models with Longtermism" />
          <attvalue for="1" value="  The rapid development of open-source large language models (LLMs) has been&#10;truly remarkable. However, the scaling law described in previous literature&#10;presents varying conclusions, which casts a dark cloud over scaling LLMs. We&#10;delve into the study of scaling laws and present our distinctive findings that&#10;facilitate scaling of large scale models in two commonly used open-source&#10;configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek&#10;LLM, a project dedicated to advancing open-source language models with a&#10;long-term perspective. To support the pre-training phase, we have developed a&#10;dataset that currently consists of 2 trillion tokens and is continuously&#10;expanding. We further conduct supervised fine-tuning (SFT) and Direct&#10;Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the&#10;creation of DeepSeek Chat models. Our evaluation results demonstrate that&#10;DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in&#10;the domains of code, mathematics, and reasoning. Furthermore, open-ended&#10;evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance&#10;compared to GPT-3.5.&#10;" />
          <attvalue for="2" value="&#10;Over the past few years, Large Language Models (LLMs) based on decoder-only Transformers \cite{transformer} have increasingly become the cornerstone and pathway to achieving Artificial General Intelligence (AGI). By predicting the next word in continuous text, LLMs undergo self-supervised pre-training on massive datasets, enabling them to achieve various purposes and possess many abilities, such as novel creation, text summarization, code completion, and more. Subsequent developments like supervised fine-tuning and reward modeling have enabled Large Language Models (LLMs) to better follow user intentions and instructions. This has endowed them with more versatile conversational capabilities and rapidly expanded their influence.&#10;&#10;This wave is sparked with closed products, such as ChatGPT \cite{chatgpt}, Claude \cite{claude}, and Bard \cite{bard}, which are developed with extensive computational resources and substantial annotation costs. These products have significantly raised the community's expectations for the capabilities of open-source LLMs, consequently inspiring a series of work~\cite{glm,llama,llama2,qwen,baichuan2,mistral}. Among these, the LLaMA series models~\cite{llama, llama2} stand out. It consolidates a range of works to create an efficient and stable architecture, building well-performing models ranging from 7B to 70B parameters. Consequently, the LLaMA series has become the de facto benchmark for architecture and performance among open-source models.&#10;&#10;Following LLaMA, the open-source community has primarily focused on training fixed-size (7B, 13B, 34B, and 70B), high-quality models, often neglecting research exploration into LLM scaling laws \cite{scalinglaw,chinchilla}. Nonetheless, research on scaling laws is of utmost importance, considering that the current open-source models are merely at the initial stage of Artificial General Intelligence (AGI) development. In addition, early works~\cite{scalinglaw, chinchilla} reached varying conclusions on the scaling of model and data with increased compute budgets and inadequately addressed hyperparameter discussions. &#10;In this paper, we extensively investigate the scaling behavior of language models and apply our findings in two widely used large-scale model configurations, namely 7B and 67B. Our study aims to lay the groundwork for future scaling of open-source LLMs, paving the way for further advancements in this domain.&#10;Specifically, we first examined the scaling laws of batch size and learning rate, and found their trends with model size. Building on this, we conducted a comprehensive study of the scaling laws of the data and model scale, successfully revealing the optimal model/data scaling-up allocation strategy and predicting the expected performance of our large-scale models. Additionally, during development, we discovered that the scaling laws derived from different datasets show significant differences. &#10;This suggests that choice of dataset remarkably affects the scaling behavior, indicating that caution should be exercised when generalizing scaling laws across datasets.&#10;&#10;Under the guidance of our scaling laws, we build from scratch open-source large language models, and release as much information as possible for community reference. We collect 2 trillion tokens for pre-training, primarily in Chinese and English. At the model level, we generally followed the architecture of LLaMA, but replaced the cosine learning rate scheduler with a multi-step learning rate scheduler, maintaining performance while facilitating continual training. We collected over 1 million instances for supervised fine-tuning (SFT) \cite{ouyang2022training} from diverse sources. This paper shares our experiences with different SFT strategies and findings in data ablation techniques. Additionally, we have utilized direct preference optimization (DPO) \cite{dpo} to improve the conversational performance of the model.&#10;&#10;We conduct extensive evaluations using our base and chat models. The evaluation results demonstrate that DeepSeek LLM surpasses LLaMA-2 70B across various benchmarks, particularly in the fields of code, mathematics, and reasoning. &#10;Following SFT and DPO, the DeepSeek 67B chat model outperforms GPT-3.5 in both Chinese and English open-ended evaluations. This highlights the superior performance of DeepSeek 67B in generating high-quality responses and engaging in meaningful conversations in both languages. Furthermore, the safety evaluation indicates that DeepSeek 67B Chat can provide harmless responses in practice. &#10;&#10;In the rest of this paper, we first introduce our pre-training basic concepts of DeepSeek LLM in Section~\ref{sec:pre-training}, including the composition of data, model architecture, infrastructure, and hyperparameters. In Section~\ref{sec:scaling}, we provide a detailed explanation of the scaling laws we have discovered and its implications. Additionally, we discuss the rationale behind our selection of pre-training hyperparameters, taking into account the insights gained from the scaling laws analysis. &#10;In Section~\ref{sec:alignment}, we discuss our fine-tuning methodology, encompassing the composition of fine-tuning data and specific methods during the SFT and DPO stages.&#10;We then present the detailed evaluation results of DeepSeek LLM in Section~\ref{sec:evluation}, covering both the base and chat models, as well as their performance in open-ended evaluations and safety evaluations. Finally, we discuss the current limitations and future directions of DeepSeek LLM in Section~\ref{sec:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.03300" label="2402.03300">
        <attvalues>
          <attvalue for="0" value="DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open&#10;  Language Models" />
          <attvalue for="1" value="  Mathematical reasoning poses a significant challenge for language models due&#10;to its complex and structured nature. In this paper, we introduce DeepSeekMath&#10;7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B&#10;math-related tokens sourced from Common Crawl, together with natural language&#10;and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the&#10;competition-level MATH benchmark without relying on external toolkits and&#10;voting techniques, approaching the performance level of Gemini-Ultra and GPT-4.&#10;Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH.&#10;The mathematical reasoning capability of DeepSeekMath is attributed to two key&#10;factors: First, we harness the significant potential of publicly available web&#10;data through a meticulously engineered data selection pipeline. Second, we&#10;introduce Group Relative Policy Optimization (GRPO), a variant of Proximal&#10;Policy Optimization (PPO), that enhances mathematical reasoning abilities while&#10;concurrently optimizing the memory usage of PPO.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.05685" label="2306.05685">
        <attvalues>
          <attvalue for="0" value="Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" />
          <attvalue for="1" value="  Evaluating large language model (LLM) based chat assistants is challenging&#10;due to their broad capabilities and the inadequacy of existing benchmarks in&#10;measuring human preferences. To address this, we explore using strong LLMs as&#10;judges to evaluate these models on more open-ended questions. We examine the&#10;usage and limitations of LLM-as-a-judge, including position, verbosity, and&#10;self-enhancement biases, as well as limited reasoning ability, and propose&#10;solutions to mitigate some of them. We then verify the agreement between LLM&#10;judges and human preferences by introducing two benchmarks: MT-bench, a&#10;multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our&#10;results reveal that strong LLM judges like GPT-4 can match both controlled and&#10;crowdsourced human preferences well, achieving over 80% agreement, the same&#10;level of agreement between humans. Hence, LLM-as-a-judge is a scalable and&#10;explainable way to approximate human preferences, which are otherwise very&#10;expensive to obtain. Additionally, we show our benchmark and traditional&#10;benchmarks complement each other by evaluating several variants of LLaMA and&#10;Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with&#10;human preferences are publicly available at&#10;https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.09488" label="2403.09488">
        <attvalues>
          <attvalue for="0" value="Rectifying Demonstration Shortcut in In-Context Learning" />
          <attvalue for="1" value="  Large language models (LLMs) are able to solve various tasks with only a few&#10;demonstrations utilizing their in-context learning (ICL) abilities. However,&#10;LLMs often rely on their pre-trained semantic priors of demonstrations rather&#10;than on the input-label relationships to proceed with ICL prediction. In this&#10;work, we term this phenomenon as the 'Demonstration Shortcut'. While previous&#10;works have primarily focused on improving ICL prediction results for predefined&#10;tasks, we aim to rectify the Demonstration Shortcut, thereby enabling the LLM&#10;to effectively learn new input-label relationships from demonstrations. To&#10;achieve this, we introduce In-Context Calibration, a demonstration-aware&#10;calibration method. We evaluate the effectiveness of the proposed method in two&#10;settings: (1) the Original ICL Task using the standard label space and (2) the&#10;Task Learning setting, where the label space is replaced with semantically&#10;unrelated tokens. In both settings, In-Context Calibration demonstrates&#10;substantial improvements, with results generalized across three LLM families&#10;(OPT, GPT, and Llama2) under various configurations.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have demonstrated their effectiveness on a wide range of tasks through in-context learning (ICL), where models learn to perform a task from demonstrations \cite{brown2020language}. &#10;Leveraging their pre-trained knowledge, LLMs can associate various words in the demonstration with specific semantics (e.g., associating `extremely painful' with `negative'), thereby performing new tasks using only a small set of input-label examples, without requiring parameter updates \cite{dong2022survey, wei2023larger}.&#10;&#10;However, LLMs often rely on the semantics from their pre-trained knowledge of given demonstrations, resulting in insufficient task learning for the patterns of the provided input-label pairs.&#10;\cite{reynolds2021prompt, min-etal-2022-rethinking, wei2023larger, pan-etal-2023-context}.&#10;This issue intensifies as the model size decreases \cite{wei2023larger}.&#10;\cite{kossen2023context} suggest that smaller LLMs show promise in learning new mappings from demonstrations in some tasks, yet they still struggle to override semantic priors acquired during pre-training. &#10;Therefore, it is necessary to develop a method that enables LLMs of various sizes to effectively mitigate semantic priors preferences and learn to perform unseen tasks from demonstrations.&#10;&#10;Prior works have primarily focused on the instabilities of LLMs in ICL prediction \cite{holtzman-etal-2021-surface, fei2023mitigating}.&#10;To mitigate these instabilities, these studies introduced content-free tokens or utilized the entire test set to calibrate prediction probabilities \cite{holtzman-etal-2021-surface, fei2023mitigating, zhou2023batch}.&#10;However, they lack consideration of the semantic priors of LLMs on the demonstration and do not verify whether their approach enhances LLMs to learn new tasks from the demonstrations.&#10;&#10;In this work, we investigate how the LLMs' pre-trained knowledge on the demonstrations affects ICL.&#10;We define the following phenomenon as a Demonstration Shortcut: the reliance of LLMs on their pre-trained semantic priors of demonstrations in ICL prediction, rather than learning from the input-label relationships presented in these demonstrations.&#10;Due to the Demonstration Shortcut, LLMs' ICL predictions may be overly dependent on the semantics of the given demonstrations even when the label distribution is uniform and the order is identical (Figure \ref{fig:short-cut}).&#10;&#10;To tackle this problem, we propose In-Context Calibration, a method designed to rectify the Demonstration Shortcut in ICL. &#10;In-Context Calibration estimates the semantic prior of LLMs on each demonstration sample with the in-context examples. &#10;Formally, for each example in the demonstration, we estimate its semantic prior relative to the remaining examples and calculate the expected semantic priors of the demonstrations.&#10;At test time, we use this term to rectify LLMs' dependency on semantic priors and enable the model to learn the intended input-label relationships from the demonstrations.&#10;&#10;We evaluate the effectiveness of In-Context Calibration on 27 classification datasets from two perspectives: (1) Original ICL Task and (2) Task Learning settings. &#10;In the Original ICL Task, we use the standard label space.&#10;In the Task Learning setting, the label space is replaced with semantically unrelated tokens.&#10;This requires LLMs to learn the novel input-label relationships to achieve high performance, as these relationships are never seen in pre-training.&#10;Our proposed method not only demonstrated enhanced performance across various tasks but also showed improvement in task learning abilities.&#10;Specifically, In-Context Calibration outperforms other ICL methods in Natural Language Inference (NLI) tasks, which demand high task learning ability.&#10;We also demonstrate that In-Context Calibration enhances ICL performance across various model types and sizes, effectively rectifying the `Demonstration Shortcut' problem.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Calibration Methods, Linguistics, Cognitive Science, In-Context Learning, Language Model Limitations, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2102.07350" label="2102.07350">
        <attvalues>
          <attvalue for="0" value="Prompt Programming for Large Language Models: Beyond the Few-Shot&#10;  Paradigm" />
          <attvalue for="1" value="  Prevailing methods for mapping large generative language models to supervised&#10;tasks may fail to sufficiently probe models' novel capabilities. Using GPT-3 as&#10;a case study, we show that 0-shot prompts can significantly outperform few-shot&#10;prompts. We suggest that the function of few-shot examples in these cases is&#10;better described as locating an already learned task rather than meta-learning.&#10;This analysis motivates rethinking the role of prompts in controlling and&#10;evaluating powerful language models. In this work, we discuss methods of prompt&#10;programming, emphasizing the usefulness of considering prompts through the lens&#10;of natural language. We explore techniques for exploiting the capacity of&#10;narratives and cultural anchors to encode nuanced intentions and techniques for&#10;encouraging deconstruction of a problem into components before producing a&#10;verdict. Informed by this more encompassing theory of prompt programming, we&#10;also introduce the idea of a metaprompt that seeds the model to generate its&#10;own natural language prompts for a range of tasks. Finally, we discuss how&#10;these more general methods of interacting with language models can be&#10;incorporated into existing and future benchmarks and practical applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08315" label="2104.08315">
        <attvalues>
          <attvalue for="0" value="Surface Form Competition: Why the Highest Probability Answer Isn't&#10;  Always Right" />
          <attvalue for="1" value="  Large language models have shown promising results in zero-shot settings&#10;(Brown et al.,2020; Radford et al., 2019). For example, they can perform&#10;multiple choice tasks simply by conditioning on a question and selecting the&#10;answer with the highest probability.&#10;  However, ranking by string probability can be problematic due to surface form&#10;competition-wherein different surface forms compete for probability mass, even&#10;if they represent the same underlying concept, e.g. &quot;computer&quot; and &quot;PC.&quot; Since&#10;probability mass is finite, this lowers the probability of the correct answer,&#10;due to competition from other strings that are valid answers (but not one of&#10;the multiple choice options).&#10;  We introduce Domain Conditional Pointwise Mutual Information, an alternative&#10;scoring function that directly compensates for surface form competition by&#10;simply reweighing each option according to a term that is proportional to its a&#10;priori likelihood within the context of the specific zero-shot task. It&#10;achieves consistent gains in zero-shot performance over both calibrated (Zhao&#10;et al., 2021) and uncalibrated scoring functions on all GPT-2 and GPT-3 models&#10;over a variety of multiple choice datasets.&#10;" />
          <attvalue for="2" value="&#10;&#10;Despite the impressive results large pretrained language models have achieved in zero-shot settings \cite{brown2020language, radford2019language}, we argue that current work underestimates the zero-shot capabilities of these models on classification tasks. This is in large part due to surface form competition—a property of generative models that causes probability to be rationed between different valid strings, even ones that differ trivially, e.g., by capitalization alone.&#10;Such competition can be largely removed by scoring choices according to Domain Conditional Pointwise Mutual Information (\method), which reweighs scores by how much more likely a hypothesis (answer) becomes given a premise (question) within the specific task domain.&#10;&#10;Specifically, consider the example question (shown in Figure~\ref{fig:1}): ``A human wants to submerge himself in water, what should he use?'' with multiple choice options ``Coffee cup'', ``Whirlpool bath'', ``Cup'', and ``Puddle.'' From the given options, ``Whirlpool bath'' is the only one that makes sense. Yet, other answers are valid and easier for a language model to generate, e.g., ``Bathtub'' and ``A bathtub.'' Since all surface forms compete for finite probability mass, allocating significant probability mass to ``Bathtub'' decreases the amount of probability mass assigned to ``Whirlpool bath.'' While the total probability of generating some correct answer may be high (i.e., across all valid surface forms), only one of these is a listed option. This is particularly problematic here, because ``Whirlpool bath'' will be much lower probability than ``Bathtub,'' due to its rarity. More generally, methods that do not account for surface form competition will favor answers with fewer lexical paraphrases. &#10;&#10;\method factors out the probability of a specific surface form, by instead computing how much more probable a hypothesis is when conditioned on a premise. We use a domain premise string to estimate the unconditional probability of a hypothesis in a given domain. On CommonsenseQA, for example, we compute the probability of each answer option immediately following the string ``? the answer is:'', and then divide the conditional probability by this estimate to calculate \method. This scaling factor reweighs answer scores according to the surface form competition that is inherent to the domain or task, e.g. completions of the domain premise that are just inherently unlikely will be upweighted more. &#10;This allows us to directly measure how much an answer tells us about the question and vice versa (mutual information is symmetric, see \S\ref{sec:method}). &#10;Valid hypotheses no longer need to compete with each other: both ``Whirlpool bath'' and ``Bathtub '' will be considered reasonable answers to the question, and so both will attain a high score.&#10;&#10;Extensive experiments show that \method consistently outperforms raw, normalized, and calibrated probability scoring methods on zero-shot multiple choice for more than a dozen datasets and it does so for every model in the GPT-2 and GPT-3 families (\S\ref{sec:exp}); this holds true across different possible prompts and in preliminary few-shot experiments as well. To better explain these gains, we use the distinct structure of the COPA dataset \cite{roemmele2011choice} to remove surface form competition entirely, showing that all methods perform well in this idealized setting~(\S\ref{exp:sfc}). Additionally, we analyze the only three datasets where \method does worse than other methods and put forward a hypothesis for why normalizing log probabilities works better than raw probabilities (\S\ref{sec:analysis}). We conclude with a discussion of how generative models should be used for selection tasks (\S\ref{sec:discussion}).&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.06875" label="2203.06875">
        <attvalues>
          <attvalue for="0" value="Improved Universal Sentence Embeddings with Prompt-based Contrastive&#10;  Learning and Energy-based Learning" />
          <attvalue for="1" value="  Contrastive learning has been demonstrated to be effective in enhancing&#10;pre-trained language models (PLMs) to derive superior universal sentence&#10;embeddings. However, existing contrastive methods still have two limitations.&#10;Firstly, previous works may acquire poor performance under domain shift&#10;settings, thus hindering the application of sentence representations in&#10;practice. We attribute this low performance to the over-parameterization of&#10;PLMs with millions of parameters. To alleviate it, we propose PromCSE&#10;(Prompt-based Contrastive Learning for Sentence Embeddings), which only trains&#10;small-scale \emph{Soft Prompt} (i.e., a set of trainable vectors) while keeping&#10;PLMs fixed. Secondly, the commonly used NT-Xent loss function of contrastive&#10;learning does not fully exploit hard negatives in supervised learning settings.&#10;To this end, we propose to integrate an Energy-based Hinge loss to enhance the&#10;pairwise discriminative power, inspired by the connection between the NT-Xent&#10;loss and the Energy-based Learning paradigm. Empirical results on seven&#10;standard semantic textual similarity (STS) tasks and a domain-shifted STS task&#10;both show the effectiveness of our method compared with the current&#10;state-of-the-art sentence embedding models. Our code is publicly avaliable at&#10;https://github.com/YJiangcm/PromCSE&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Contrastive Learning, Linguistics, Language Models, Artificial Intelligence, Mathematics, Sentence Embeddings, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.09960" label="2304.09960">
        <attvalues>
          <attvalue for="0" value="A Latent Space Theory for Emergent Abilities in Large Language Models" />
          <attvalue for="1" value="  Languages are not created randomly but rather to communicate information.&#10;There is a strong association between languages and their underlying meanings,&#10;resulting in a sparse joint distribution that is heavily peaked according to&#10;their correlations. Moreover, these peak values happen to match with the&#10;marginal distribution of languages due to the sparsity. With the advent of LLMs&#10;trained on big data and large models, we can now precisely assess the marginal&#10;distribution of languages, providing a convenient means of exploring the sparse&#10;structures in the joint distribution for effective inferences. In this paper,&#10;we categorize languages as either unambiguous or {\epsilon}-ambiguous and&#10;present quantitative results to demonstrate that the emergent abilities of&#10;LLMs, such as language understanding, in-context learning, chain-of-thought&#10;prompting, and effective instruction fine-tuning, can all be attributed to&#10;Bayesian inference on the sparse joint distribution of languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;Over the past few years, large language models (LLMs) have emerged as the predominant method for most natural language processing (NLP) tasks \cite{radford2018improving,raffel2020exploring,brown2020language,chowdhery2022palm}. With the increase in model size and training data, LLMs have demonstrated remarkable capabilities in solving various NLP tasks, including semantic understanding, few-shot in-context learning, chain-of-thought prompting, and effective instruction fine-tuning for alignment. These abilities are often referred to as emergent capabilities as they have been observed to emerge as the model size and training data increase \cite{wei2022emergent}.&#10;In contrast to a simple scaling law, they are not the same abilities just extended to a new data distribution but some new abilities unseen in smaller model/data scales. &#10;Machine learning researchers are keen to know how LLMs have developed these skills to perform well on unseen tasks, especially since LLMs are primarily trained in an unsupervised manner to predict the next tokens in text. Some empirical studies have suggested that the emergent abilities of LLMs may be linked to the label space and input data distributional properties \cite{min2022rethinking,chan2022data}, multitask prompted learning \cite{sanh2022multitask}, and pre-training term frequencies \cite{razeghi2022impact}. Moreover, \cite{xie2022explanation,wang2023large} have proposed theories that explain in-context learning of LLMs as Bayesian inferences that use prompts to recover latent concepts. More recently, \cite{wies2023learnability} established a PAC based framework for in-context learnability, and \cite{hahn2023theory} claimed that the emergent abilities arise through recombination of compositional structures in natural languages.&#10;&#10;Motivated by \cite{xie2022explanation}, our study proposes a novel latent space theory to explain the emergent abilities of LLMs. While \cite{xie2022explanation} considered a specific type of data distribution generated by Hidden Markov Models (HMMs), we examine general data distributions by exploring the sparsity property that is universally present in the joint distributions of languages. LLMs, which serve as a universal density approximator to the marginal distribution, offer a convenient means of exploring these sparse structures for effective inferences.&#10;We categorize languages as either unambiguous or $\varepsilon$-ambiguous and present quantitative results demonstrating that the emergent abilities of LLMs can be attributed to Bayesian inference on the sparse joint distribution of languages. Furthermore, we provide simulation results on synthetic languages that validate our theoretical findings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Bayesian Inference, Artificial Intelligence, Mathematics, Statistics, Language Structure" />
        </attvalues>
      </node>
      <node id="2202.12837" label="2202.12837">
        <attvalues>
          <attvalue for="0" value="Rethinking the Role of Demonstrations: What Makes In-Context Learning&#10;  Work?" />
          <attvalue for="1" value="  Large language models (LMs) are able to in-context learn -- perform a new&#10;task via inference alone by conditioning on a few input-label pairs&#10;(demonstrations) and making predictions for new inputs. However, there has been&#10;little understanding of how the model learns and which aspects of the&#10;demonstrations contribute to end task performance. In this paper, we show that&#10;ground truth demonstrations are in fact not required -- randomly replacing&#10;labels in the demonstrations barely hurts performance on a range of&#10;classification and multi-choce tasks, consistently over 12 different models&#10;including GPT-3. Instead, we find that other aspects of the demonstrations are&#10;the key drivers of end task performance, including the fact that they provide a&#10;few examples of (1) the label space, (2) the distribution of the input text,&#10;and (3) the overall format of the sequence. Together, our analysis provides a&#10;new way of understanding how and why in-context learning works, while opening&#10;up new questions about how much can be learned from large language models&#10;through inference alone.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="Large language models have been key to strong performance&#10;in a wide range of downstream tasks~\cite{devlin2019bert,radford2019language,liu2019roberta,raffel2020exploring,lewis2020bart}. While finetuning has been a popular approach to transfer to new tasks~\cite{devlin2019bert}, it is often impractical to finetune a very large model (e.g. $\geq$10B parameters).&#10;\cite{brown2020language} propose in-context learning as an alternative way to learn a new task. As depicted in Figure~\ref{fig:icl}, the LM learns a new task via inference alone by conditioning on a concatenation of the training data as \demo, without any gradient updates.&#10;&#10;In-context learning has been the focus of significant study since its introduction. Prior work proposes&#10;better ways of formulating the problem~\cite{zhao2021calibrate,holtzman2021surface,min2021noisy}, better ways of choosing labeled examples for the \demo~\cite{liu2021makes,lu2021fantastically,rubin2021learning}, meta-training with an explicit in-context learning objective~\cite{chen2021meta,min2021metaicl}, and learning to follow instructions as a variant of in-context learning~\cite{mishra2021cross,efrat2020turking,wei2022finetuned,sanh2022multitask}.&#10;At the same time, some work reports brittleness and over-sensitivity for in-context learning~\cite{lu2021fantastically,zhao2021calibrate,mishra2021reframing}.&#10;&#10;Relatively less work has been done to understand why in-context learning works.&#10;\cite{xie2022explanation} provide theoretical analysis that in-context learning can be formalized as Bayesian inference that uses the \demo\ to recover latent concepts.&#10;\cite{razeghi2022impact} show that in-context learning performance is highly correlated with term frequencies in the pretraining data.&#10;To the best of our knowledge, this paper is the first that provides an empirical analysis that investigates why in-context learning achieves performance gains over zero-shot inference.&#10;We find that the \gt\ input-label mapping in the \demo\ has only a marginal effect, and measure the impact of finer-grained aspects of the \demo.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2205.05055" label="2205.05055">
        <attvalues>
          <attvalue for="0" value="Data Distributional Properties Drive Emergent In-Context Learning in&#10;  Transformers" />
          <attvalue for="1" value="  Large transformer-based models are able to perform in-context few-shot&#10;learning, without being explicitly trained for it. This observation raises the&#10;question: what aspects of the training regime lead to this emergent behavior?&#10;Here, we show that this behavior is driven by the distributions of the training&#10;data itself. In-context learning emerges when the training data exhibits&#10;particular distributional properties such as burstiness (items appear in&#10;clusters rather than being uniformly distributed over time) and having large&#10;numbers of rarely occurring classes. In-context learning also emerges more&#10;strongly when item meanings or interpretations are dynamic rather than fixed.&#10;These properties are exemplified by natural language, but are also inherent to&#10;naturalistic data in a wide range of other domains. They also depart&#10;significantly from the uniform, i.i.d. training distributions typically used&#10;for standard supervised learning. In our initial experiments, we found that&#10;in-context learning traded off against more conventional weight-based learning,&#10;and models were unable to achieve both simultaneously. However, our later&#10;experiments uncovered that the two modes of learning could co-exist in a single&#10;model when it was trained on data following a skewed Zipfian distribution --&#10;another common property of naturalistic data, including language. In further&#10;experiments, we found that naturalistic data distributions were only able to&#10;elicit in-context learning in transformers, and not in recurrent models. In&#10;sum, our findings indicate how the transformer architecture works together with&#10;particular properties of the training data to drive the intriguing emergent&#10;in-context learning behaviour of large language models, and how future work&#10;might encourage both in-context and in-weights learning in domains beyond&#10;language.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large transformer-based language models show an intriguing ability to perform in-context learning \cite{brown_language_2020}. This is the ability to generalize rapidly from a few examples of a new concept on which they have not been previously trained, without gradient updates to the model. In-context learning is a special case of few-shot learning in which the output is conditioned on examples from a `context', and where there are no gradient updates. It contrasts with `in-weights' learning, which is the standard setting for supervised learning – this is slow (requiring many examples), and depends on gradient updates to the weights. Earlier work in the context of `meta-learning' showed how neural networks can perform few-shot learning without the need for weight updates \cite{vinyals_matching_2016, santoro_meta-learning_2016, wang_learning_2016}.&#10;To achieve this, the researchers explicitly designed the training regime to incentivize in-context learning, a process sometimes called 'meta-training'. In the case of transformer language models, however, the capacity for in-context learning is emergent. Neither the model's transformer architecture nor its learning objective are explicitly designed with in-context learning in mind.&#10;&#10;Here, we consider the question of how transformer language models are able to acquire this impressive ability, without it being explicitly targeted by the training setup or learning objective. The emergence of in-context learning in language models was observed as recurrent models were supplanted by transformers, e.g. in GPT3. Was the novel architecture the critical factor behind this emergence? In this work we explore this possibility, as well as a second: that a capacity for in-context learning depends on the distributional qualities of the training data.&#10;&#10;This hypothesis was inspired by the observation that many natural data sources -- including natural language -- differ from typical supervised datasets due to a few notable features. For example, natural data is temporally `bursty''. That is, a given entity (word, person, object, etc) may have a distribution that is not uniform across time, instead tending to appear in clusters \cite{sarkar_bayesian_2005, alvarez-lacalle_hierarchical_2006, neuts_burstiness_2007, altmann_beyond_2009, serrano_modeling_2009,lambiotte_burstiness_2013}. Natural data also often has the property that the marginal distribution across entities is highly skewed, following a Zipfian (power law) distribution with a long tail of infrequent items \cite{zipf_human_1949, piantadosi_zipfs_2014, smith_developing_2018}. Finally, the `meaning' of entities in natural data (such as words in natural language) is often dynamic rather than fixed. That is, a single entity can have multiple possible interpretations (polysemy and homonymy, in language) and multiple entities can map to the same interpretation (synonymy, in language), usually in a context-dependent way. The combination of these properties may result in training data that occupies some middle-ground between the data used in canonical supervised learning and that used for few-shot meta-training. &#10;&#10;In particular, standard supervised training typically consists of item classes that recur with uniform regularity, and with item-label mappings that are fixed throughout training -- these properties allow a model to gradually learn over time, by encoding information into its weights, e.g. via gradient descent. By contrast, few-shot or in-context meta-training generally involves training a model directly on specially crafted sequences of data where item classes only recur and/or item-label mappings are only fixed within episodes -- they do not recur and are not fixed across episodes \cite{vinyals_matching_2016, santoro_meta-learning_2016}. Naturalistic data, such as language or first-person experience, has characteristics of both of these data types. As in supervised training, items (words) do recur, and the relationship between an entity and its interpretation (or meaning) is fixed, to some degree at least. At the same time, the skewed and long-tailed distribution of natural data means that some entities recur very frequently while a large number recur much more rarely. Importantly, however, these rare items are often bursty, making them disproportionately likely to occur multiple times within a given context window, somewhat like a sequence of 'meta-training' data. We can also see the dynamic relationship between entities and their interpretation (epitomized by synonyms, homonyms, and polysemy, in the case of language) as weaker versions of the completely dynamic item-label mappings that are used in few-shot meta-training, where the mappings are randomly permuted on every episode.&#10;&#10;In this paper, we experimentally manipulated the distributional properties of the training data and measured the effects on in-context few-shot learning. We performed our experiments over data sequences sampled from a standard image-based few-shot dataset \cite[the Omniglot dataset;][]{lake_omniglot_2019}. At training, we fed each model (such as a transformer or recurrent network) with input sequences of Omniglot images and labels, varying the natural data-inspired distributional properties of choice. At evaluation, we assessed whether these properties gave rise to in-context learning abilities.&#10;&#10;Our results showed that, indeed, in-context learning emerges in a transformer model only when trained on data that includes both burstiness and a large enough set of rarely occurring classes.We also tested two instantiations of the kinds of dynamic item interpretation observed in natural data -- having many labels per item as well as within-class variation. We found that both interventions on the training data could bias the model more strongly towards in-context learning. The models we tested typically exhibited a tradeoff between rapid in-context learning vs. relying on information that was stored through slow, gradient-based updates (`in-weights'' learning). However, we found that models could simultaneously exhibit both in-context learning and in-weights learning when trained on a skewed marginal distribution over classes (akin to the Zipfian distribution of natural data).&#10;&#10;At the same time, architecture is also important. Unlike transformers, recurrent models like LSTMs and RNNs (matched on number of parameters) were unable to exhibit in-context learning when trained on the same data distribution.&#10;It is important to note, however, that transformer models trained on the wrong data distributions still did fail to exhibit in-context learning. Thus, attention is not all you need -- architecture and data are both key to the emergence of in-context learning.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.07895" label="2303.07895">
        <attvalues>
          <attvalue for="0" value="The Learnability of In-Context Learning" />
          <attvalue for="1" value="  In-context learning is a surprising and important phenomenon that emerged&#10;when modern language models were scaled to billions of learned parameters.&#10;Without modifying a large language model's weights, it can be tuned to perform&#10;various downstream natural language tasks simply by including concatenated&#10;training examples of these tasks in its input. Though disruptive for many&#10;practical applications of large language models, this emergent learning&#10;paradigm is not well understood from a theoretical perspective. In this paper,&#10;we propose a first-of-its-kind PAC based framework for in-context learnability,&#10;and use it to provide the first finite sample complexity results for the&#10;in-context learning setup. Our framework includes an initial pretraining phase,&#10;which fits a function to the pretraining distribution, and then a second&#10;in-context learning phase, which keeps this function constant and concatenates&#10;training examples of the downstream task in its input. We use our framework in&#10;order to prove that, under mild assumptions, when the pretraining distribution&#10;is a mixture of latent tasks (a model often considered for natural language&#10;pretraining), these tasks can be efficiently learned via in-context learning,&#10;even though the model's weights are unchanged and the input significantly&#10;diverges from the pretraining distribution. Our theoretical analysis reveals&#10;that in this setting, in-context learning is more about identifying the task&#10;than about learning it, a result which is in line with a series of recent&#10;empirical findings. We hope that the in-context learnability framework&#10;presented in this paper will facilitate future progress towards a deeper&#10;understanding of this important new learning paradigm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.07971" label="2303.07971">
        <attvalues>
          <attvalue for="0" value="A Theory of Emergent In-Context Learning as Implicit Structure Induction" />
          <attvalue for="1" value="  Scaling large language models (LLMs) leads to an emergent capacity to learn&#10;in-context from example demonstrations. Despite progress, theoretical&#10;understanding of this phenomenon remains limited. We argue that in-context&#10;learning relies on recombination of compositional operations found in natural&#10;language data. We derive an information-theoretic bound showing how in-context&#10;learning abilities arise from generic next-token prediction when the&#10;pretraining distribution has sufficient amounts of compositional structure,&#10;under linguistically motivated assumptions. A second bound provides a&#10;theoretical justification for the empirical success of prompting LLMs to output&#10;intermediate steps towards an answer. To validate theoretical predictions, we&#10;introduce a controlled setup for inducing in-context learning; unlike previous&#10;approaches, it accounts for the compositional nature of language. Trained&#10;transformers can perform in-context learning for a range of tasks, in a manner&#10;consistent with the theoretical results. Mirroring real-world LLMs in a&#10;miniature setup, in-context learning emerges when scaling parameters and data,&#10;and models perform better when prompted to output intermediate steps. Probing&#10;shows that in-context learning is supported by a representation of the input's&#10;compositional structure. Taken together, these results provide a step towards&#10;theoretical understanding of emergent behavior in large language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.05020" label="2311.05020">
        <attvalues>
          <attvalue for="0" value="First Tragedy, then Parse: History Repeats Itself in the New Era of&#10;  Large Language Models" />
          <attvalue for="1" value="  Many NLP researchers are experiencing an existential crisis triggered by the&#10;astonishing success of ChatGPT and other systems based on large language models&#10;(LLMs). After such a disruptive change to our understanding of the field, what&#10;is left to do? Taking a historical lens, we look for guidance from the first&#10;era of LLMs, which began in 2005 with large $n$-gram models for machine&#10;translation (MT). We identify durable lessons from the first era, and more&#10;importantly, we identify evergreen problems where NLP researchers can continue&#10;to make meaningful contributions in areas where LLMs are ascendant. We argue&#10;that disparities in scale are transient and researchers can work to reduce&#10;them; that data, rather than hardware, is still a bottleneck for many&#10;applications; that meaningful realistic evaluation is still an open problem;&#10;and that there is still room for speculative approaches.&#10;" />
          <attvalue for="2" value="&#10;&#10;Picture this scene: A renowned NLP researcher at a hot seven-year-old startup steps onstage to deliver a keynote. The speaker describes an ambitious new system to the packed room, building up to the results slide: a bar chart in which the $x$-axis shows the number of training words, and the $y$-axis shows system accuracy. As each data point is revealed, performance rises relentlessly, culminating in a system trained on well over a trillion words using over a thousand processor cores. It smashes the state of the art by a margin previously thought impossible.&#10;&#10;Attendees are visibly shaken as they realize, over the course of a minute, that years of research have just been rendered utterly inconsequential. Established academics panic, anticipating the wholesale rejection of already-submitted grant applications. PhD students despair, contemplating the irrelevance of their unfinished dissertations. Many ponder an exit to industry or a change of fields. They will speak of little else this week.&#10;&#10;Does this scene sound like one that might have happened in the past year? In fact, it happened 19 years ago, in 2005, launching the first era of Large&#10;Language Models (LLMs): the Statistical Machine Translation (SMT) era. &#10;The speaker, Franz Och, had co-invented key methods in SMT \cite{och-ney-2003-systematic,koehn-etal-2003-statistical,och-2003-minimum}, but had not published new work since joining Google in 2004, instead revealing it in an invited talk prior to the launch of a new Google Translate \cite{och_statistical_2006}.&#10;The provocative results slide from that talk (Figure~\ref{fig:och_slide}) shows how Google improved its SMT system simply by expanding the training corpus of a phrase-based language model \cite{brants-etal-2007-large}.&#10;&#10;The first era of LLMs initially provoked great anxiety among MT researchers about the state of their field, but MT research has continued to flourish in academia, industry, and government. Even in the modern era of deep learning, MT has been a locus of innovations that have fundamentally altered NLP and all of machine learning \cite{bahdanau2015neural,NeurIPS2017_3f5ee243,Sutskever2014SequenceTS}. &#10;&#10;We believe that this history offers lessons for the current era of LLMs, an era during which massive proprietary models have become a de facto baseline for many tasks \cite{anna_rogers_closed_2023}. The expense of state-of-the-art research has led many to question the role of smaller and publicly funded groups in AI \cite{lee2023surveying}, a phenomenon we will call the scale crisis. Researchers without direct access to LLMs have publicly fretted over their research directions, with \cite{togelius2023choose} suggesting pivots in research direction to sidestep scale, and \cite{ignat2023phd} sketching research areas that are ``not within the purview of LLMs.'' But what should researchers do if they care about problems that are within the purview of LLMs?&#10;To answer this question, we look to the first era of LLMs. What were the durable lessons of that time and evergreen research problems that still matter today? We arrive at several recurrent lessons:&#10;&#10;\begin{asparadesc}&#10; \item Scale is supreme (Section~\ref{sec:lesson_scale}). We argue that, for areas where data is plentiful, NLP researchers cannot escape the Bitter Lesson \cite{sutton_bitter_2019} that general purpose methods exploiting scale will outperform methods that leverage informed priors. We recommend that researchers take advantage of improvements in hardware as they enable scale at affordable budgets (Section~\ref{sec:hardware}) and that they remember small-scale problems (Section~\ref{sec:small}). &#10;&#10;\item Evaluation is a bottleneck (Section~\ref{sec:lesson_evaluation}). The Bitter Lesson favors generic methods, which require evaluation metrics to optimize over. But improved models create an evaluation bottleneck, since error detection becomes harder when most remaining mistakes are subtle or associated with edge cases. At scale, automated metrics show their flaws. We recommend that researchers work on improving metrics (Section~\ref{sec:metrics}).&#10;&#10;\item There is no gold standard (Section~\ref{sec:lesson_human}). When one can afford the annotation costs, it may be tempting to consider human feedback as the ideal solution to the evaluation problem. Unfortunately, history has repeatedly shown that na\&quot;{i}ve methods of soliciting human preferences result in poor feedback, prioritizing superficial properties of model outputs. This lesson may serve as a counterpoint to the impulse to collect massive quantities of low-quality data in response to the Bitter Lesson. Instead, we recommend grounding performance measurement in concrete downstream tasks (Section~\ref{sec:concrete}).&#10;&#10;\item Progress is not continuous (Section \ref{sec:future}). The ascent of neural MT abruptly ended SMT's decade of seemingly unbeatable growth. This change in directions was enabled by new hardware-based paradigms, so we recommend that researchers continue exploring new methods that might scale well on future hardware (Section \ref{sec:hardware_lottery}).&#10;&#10;\item We conclude with a simple message: Do research (Section~\ref{sec:conclusion}). We remind the reader that engineering achievements do not render scientific achievements insignificant, and we encourage the NLP community to renew their commitment to foundational scientific research even in areas where scale is currently a dominant factor.&#10;\end{asparadesc}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, NLP Research Evolution, Artificial Intelligence, Future Research Directions, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1409.0473" label="1409.0473">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation by Jointly Learning to Align and Translate" />
          <attvalue for="1" value="  Neural machine translation is a recently proposed approach to machine&#10;translation. Unlike the traditional statistical machine translation, the neural&#10;machine translation aims at building a single neural network that can be&#10;jointly tuned to maximize the translation performance. The models proposed&#10;recently for neural machine translation often belong to a family of&#10;encoder-decoders and consists of an encoder that encodes a source sentence into&#10;a fixed-length vector from which a decoder generates a translation. In this&#10;paper, we conjecture that the use of a fixed-length vector is a bottleneck in&#10;improving the performance of this basic encoder-decoder architecture, and&#10;propose to extend this by allowing a model to automatically (soft-)search for&#10;parts of a source sentence that are relevant to predicting a target word,&#10;without having to form these parts as a hard segment explicitly. With this new&#10;approach, we achieve a translation performance comparable to the existing&#10;state-of-the-art phrase-based system on the task of English-to-French&#10;translation. Furthermore, qualitative analysis reveals that the&#10;(soft-)alignments found by the model agree well with our intuition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1409.3215" label="1409.3215">
        <attvalues>
          <attvalue for="0" value="Sequence to Sequence Learning with Neural Networks" />
          <attvalue for="1" value="  Deep Neural Networks (DNNs) are powerful models that have achieved excellent&#10;performance on difficult learning tasks. Although DNNs work well whenever large&#10;labeled training sets are available, they cannot be used to map sequences to&#10;sequences. In this paper, we present a general end-to-end approach to sequence&#10;learning that makes minimal assumptions on the sequence structure. Our method&#10;uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to&#10;a vector of a fixed dimensionality, and then another deep LSTM to decode the&#10;target sequence from the vector. Our main result is that on an English to&#10;French translation task from the WMT'14 dataset, the translations produced by&#10;the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's&#10;BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did&#10;not have difficulty on long sentences. For comparison, a phrase-based SMT&#10;system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM&#10;to rerank the 1000 hypotheses produced by the aforementioned SMT system, its&#10;BLEU score increases to 36.5, which is close to the previous best result on&#10;this task. The LSTM also learned sensible phrase and sentence representations&#10;that are sensitive to word order and are relatively invariant to the active and&#10;the passive voice. Finally, we found that reversing the order of the words in&#10;all source sentences (but not target sentences) improved the LSTM's performance&#10;markedly, because doing so introduced many short term dependencies between the&#10;source and the target sentence which made the optimization problem easier.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.16900" label="2306.16900">
        <attvalues>
          <attvalue for="0" value="Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research" />
          <attvalue for="1" value="  Many recent improvements in NLP stem from the development and use of large&#10;pre-trained language models (PLMs) with billions of parameters. Large model&#10;sizes makes computational cost one of the main limiting factors for training&#10;and evaluating such models; and has raised severe concerns about the&#10;sustainability, reproducibility, and inclusiveness for researching PLMs. These&#10;concerns are often based on personal experiences and observations. However,&#10;there had not been any large-scale surveys that investigate them. In this work,&#10;we provide a first attempt to quantify these concerns regarding three topics,&#10;namely, environmental impact, equity, and impact on peer reviewing. By&#10;conducting a survey with 312 participants from the NLP community, we capture&#10;existing (dis)parities between different and within groups with respect to&#10;seniority, academia, and industry; and their impact on the peer reviewing&#10;process. For each topic, we provide an analysis and devise recommendations to&#10;mitigate found disparities, some of which already successfully implemented.&#10;Finally, we discuss additional concerns raised by many participants in&#10;free-text responses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.06035" label="2304.06035">
        <attvalues>
          <attvalue for="0" value="Choose Your Weapon: Survival Strategies for Depressed AI Academics" />
          <attvalue for="1" value="  Are you an AI researcher at an academic institution? Are you anxious you are&#10;not coping with the current pace of AI advancements? Do you feel you have no&#10;(or very limited) access to the computational and human resources required for&#10;an AI research breakthrough? You are not alone; we feel the same way. A growing&#10;number of AI academics can no longer find the means and resources to compete at&#10;a global scale. This is a somewhat recent phenomenon, but an accelerating one,&#10;with private actors investing enormous compute resources into cutting edge AI&#10;research. Here, we discuss what you can do to stay competitive while remaining&#10;an academic. We also briefly discuss what universities and the private sector&#10;could do improve the situation, if they are so inclined. This is not an&#10;exhaustive list of strategies, and you may not agree with all of them, but it&#10;serves to start a discussion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.01982" label="2109.01982">
        <attvalues>
          <attvalue for="0" value="Learning Hierarchical Structures with Differentiable Nondeterministic&#10;  Stacks" />
          <attvalue for="1" value="  Learning hierarchical structures in sequential data -- from simple&#10;algorithmic patterns to natural language -- in a reliable, generalizable way&#10;remains a challenging problem for neural language models. Past work has shown&#10;that recurrent neural networks (RNNs) struggle to generalize on held-out&#10;algorithmic or syntactic patterns without supervision or some inductive bias.&#10;To remedy this, many papers have explored augmenting RNNs with various&#10;differentiable stacks, by analogy with finite automata and pushdown automata&#10;(PDAs). In this paper, we improve the performance of our recently proposed&#10;Nondeterministic Stack RNN (NS-RNN), which uses a differentiable data structure&#10;that simulates a nondeterministic PDA, with two important changes. First, the&#10;model now assigns unnormalized positive weights instead of probabilities to&#10;stack actions, and we provide an analysis of why this improves training.&#10;Second, the model can directly observe the state of the underlying PDA. Our&#10;model achieves lower cross-entropy than all previous stack RNNs on five&#10;context-free language modeling tasks (within 0.05 nats of the&#10;information-theoretic lower bound), including a task on which the NS-RNN&#10;previously failed to outperform a deterministic stack RNN baseline. Finally, we&#10;propose a restricted version of the NS-RNN that incrementally processes&#10;infinitely long sequences, and we present language modeling results on the Penn&#10;Treebank.&#10;" />
          <attvalue for="2" value="&#10;&#10;Many machine learning problems involve sequential data with hierarchical structures, such as modeling context-free languages \cite{grefenstette+al:2015,dusell+chiang:2020}, evaluating mathematical expressions \cite{nangia+bowman:2018,hao+al:2018}, logical inference \cite{bowman+al:2015}, and modeling syntax in natural language \cite{dyer+al:2016,shen+al:2018,kim+al:2019}. However, recurrent neural networks (RNNs) have difficulty learning to solve these tasks, or generalizing to held-out sequences, unless they have supervision or a hierarchical inductive bias \cite{schijndel+al:2019,wilcox+al:2019,mccoy+al:2020}. A limiting factor of RNNs is their reliance on memory whose size is constant across time. For example, to predict the second half of a string of the form~$w\#\reverse{w}$, a simple RNN would need to store all of $w$ in its hidden state before predicting its reversal $\reverse{w}$; a memory of finite size will inevitably fail to do this for inputs exceeding a certain length.&#10;&#10;To remedy this, some previous work has investigated the addition of differentiable stack data structures to RNNs \cite{sun+al:1995,grefenstette+al:2015,joulin+mikolov:2015,dusell+chiang:2020}, which is closely related to work on neural networks that model shift-reduce parsers \cite{bowman+al:2016,dyer+al:2016,shen+al:2019}. Just as adding a stack to a finite state machine, which makes it a pushdown automaton (PDA), enables it to recognize context-free languages (CFLs), the hope is that adding stacks to RNNs will increase the range of problems on which they can be used effectively. We also expect stacks to aid training by introducing an inductive bias for learning hierarchical patterns, and to increase generalization power by structuring the model's memory in a way that better predicts held-out hierarchical data.&#10;&#10;Previously \cite{dusell+chiang:2020}, we proposed a stack-based RNN called the Nondeterministic Stack RNN (NS-RNN) that outperformed other stack RNNs on a range of CFL language modeling tasks. Its defining feature is that its external data structure is a nondeterministic PDA, allowing it to simulate an exponential number of sequences of stack operations in parallel. This is in contrast to prior stack RNNs \cite{grefenstette+al:2015,joulin+mikolov:2015} which model deterministic stacks, being designed to learn one correct stack operation at each time step. One reason nondeterminism is important is that deterministic CFLs are a proper subset of CFLs. If the analogy with PDAs holds true, then equipping an RNN with a deterministic stack would only enable it to model deterministic CFLs, whereas a nondeterministic stack should enable it to model all CFLs. This is important for natural language processing, as human language is known to be high in syntactic ambiguity.&#10;&#10;Another benefit of nondeterminism, even on deterministic CFLs, applies to training.&#10;In order for a model to receive a reward for an action, it must try the action (that is, give it nonzero probability so that it receives gradient during backpropagation). For example, in the digit-recognition task, a classifier tries all ten digits, and is rewarded for the correct one. But in a stack-augmented model, the space of possible action sequences is very large. Whereas a deterministic stack can only try one of them, a nondeterministic stack can try all of them and always receives a reward for the correct one.&#10;But as explained in \S\ref{sec:why_unnormalized}, because the NS-RNN's probability for an action sequence is the product of many probabilities, it can be extremely small, so the NS-RNN sometimes learns very slowly.&#10;&#10;In this paper we present a new model, the \newmodel (\newmodelacronym), which is based on the NS-RNN, but improves its performance on all of the CFL tasks it was originally tested on, thanks to two key changes. The first is that stack actions have weights that do not necessarily form a probability distribution (\S\ref{sec:why_unnormalized}). They define an unnormalized distribution over stacks that is renormalized whenever the model queries it. Second, the \newmodelacronym includes not only top stack symbols but also PDA states in this query (\S\ref{sec:pda_states}). These changes allow the \newmodelacronym to attain lower cross-entropy on CFL tasks (in fact, very close to the information-theoretic lower bound) and to surpass deterministic stack RNNs on a task on which the NS-RNN fails to do so (``padded reversal''). Finally, as a third modification, we present a memory-restricted version of the RNS-RNN that requires only~$O(n)$ time and space~(\S\ref{sec:incremental_execution}). This restricted RNS-RNN can be run incrementally on arbitrarily long sequences, which is a necessity for language modeling on natural language, for which we provide experimental results. Our code is available at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Modeling Tasks, Linguistics, Recurrent Neural Networks, Artificial Intelligence, Neural Language Models, Mathematics" />
        </attvalues>
      </node>
      <node id="1506.02516" label="1506.02516">
        <attvalues>
          <attvalue for="0" value="Learning to Transduce with Unbounded Memory" />
          <attvalue for="1" value="  Recently, strong results have been demonstrated by Deep Recurrent Neural&#10;Networks on natural language transduction problems. In this paper we explore&#10;the representational power of these models using synthetic grammars designed to&#10;exhibit phenomena similar to those found in real transduction problems such as&#10;machine translation. These experiments lead us to propose new memory-based&#10;recurrent networks that implement continuously differentiable analogues of&#10;traditional data structures such as Stacks, Queues, and DeQues. We show that&#10;these architectures exhibit superior generalisation performance to Deep RNNs&#10;and are often able to learn the underlying generating algorithms in our&#10;transduction experiments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.04674" label="2010.04674">
        <attvalues>
          <attvalue for="0" value="Learning Context-Free Languages with Nondeterministic Stack RNNs" />
          <attvalue for="1" value="  We present a differentiable stack data structure that simultaneously and&#10;tractably encodes an exponential number of stack configurations, based on&#10;Lang's algorithm for simulating nondeterministic pushdown automata. We call the&#10;combination of this data structure with a recurrent neural network (RNN)&#10;controller a Nondeterministic Stack RNN. We compare our model against existing&#10;stack RNNs on various formal languages, demonstrating that our model converges&#10;more reliably to algorithmic behavior on deterministic tasks, and achieves&#10;lower cross-entropy on inherently nondeterministic tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.06028" label="1804.06028">
        <attvalues>
          <attvalue for="0" value="ListOps: A Diagnostic Dataset for Latent Tree Learning" />
          <attvalue for="1" value="  Latent tree learning models learn to parse a sentence without syntactic&#10;supervision, and use that parse to build the sentence representation. Existing&#10;work on such models has shown that, while they perform well on tasks like&#10;sentence classification, they do not learn grammars that conform to any&#10;plausible semantic or syntactic formalism (Williams et al., 2018a). Studying&#10;the parsing ability of such models in natural language can be challenging due&#10;to the inherent complexities of natural language, like having several valid&#10;parses for a single sentence. In this paper we introduce ListOps, a toy dataset&#10;created to study the parsing ability of latent tree models. ListOps sequences&#10;are in the style of prefix arithmetic. The dataset is designed to have a single&#10;correct parsing strategy that a system needs to learn to succeed at the task.&#10;We show that the current leading latent tree models are unable to learn to&#10;parse and succeed at ListOps. These models achieve accuracies worse than purely&#10;sequential RNNs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.02836" label="1809.02836">
        <attvalues>
          <attvalue for="0" value="Context-Free Transductions with Neural Stacks" />
          <attvalue for="1" value="  This paper analyzes the behavior of stack-augmented recurrent neural network&#10;(RNN) models. Due to the architectural similarity between stack RNNs and&#10;pushdown transducers, we train stack RNN models on a number of tasks, including&#10;string reversal, context-free language modelling, and cumulative XOR&#10;evaluation. Examining the behavior of our networks, we show that&#10;stack-augmented RNNs can discover intuitive stack-based strategies for solving&#10;our tasks. However, stack RNNs are more difficult to train than classical&#10;architectures such as LSTMs. Rather than employ stack-based strategies, more&#10;complex networks often find approximate solutions by using the stack as&#10;unstructured memory.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.04834" label="1506.04834">
        <attvalues>
          <attvalue for="0" value="Tree-structured composition in neural networks without tree-structured&#10;  architectures" />
          <attvalue for="1" value="  Tree-structured neural networks encode a particular tree geometry for a&#10;sentence in the network design. However, these models have at best only&#10;slightly outperformed simpler sequence-based models. We hypothesize that neural&#10;sequence models like LSTMs are in fact able to discover and implicitly use&#10;recursive compositional structure, at least for tasks with clear cues to that&#10;structure in the data. We demonstrate this possibility using an artificial data&#10;task for which recursive compositional structure is crucial, and find an&#10;LSTM-based sequence model can indeed learn to exploit the underlying tree&#10;structure. However, its performance consistently lags behind that of tree&#10;models, even on large training sets, suggesting that tree-structured models are&#10;more effective at exploiting recursive structure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1602.07776" label="1602.07776">
        <attvalues>
          <attvalue for="0" value="Recurrent Neural Network Grammars" />
          <attvalue for="1" value="  We introduce recurrent neural network grammars, probabilistic models of&#10;sentences with explicit phrase structure. We explain efficient inference&#10;procedures that allow application to both parsing and language modeling.&#10;Experiments show that they provide better parsing in English than any single&#10;previously published supervised generative model and better language modeling&#10;than state-of-the-art sequential RNNs in English and Chinese.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.09536" label="1810.09536">
        <attvalues>
          <attvalue for="0" value="Ordered Neurons: Integrating Tree Structures into Recurrent Neural&#10;  Networks" />
          <attvalue for="1" value="  Natural language is hierarchically structured: smaller units (e.g., phrases)&#10;are nested within larger units (e.g., clauses). When a larger constituent ends,&#10;all of the smaller constituents that are nested within it must also be closed.&#10;While the standard LSTM architecture allows different neurons to track&#10;information at different time scales, it does not have an explicit bias towards&#10;modeling a hierarchy of constituents. This paper proposes to add such an&#10;inductive bias by ordering the neurons; a vector of master input and forget&#10;gates ensures that when a given neuron is updated, all the neurons that follow&#10;it in the ordering are also updated. Our novel recurrent architecture, ordered&#10;neurons LSTM (ON-LSTM), achieves good performance on four different tasks:&#10;language modeling, unsupervised parsing, targeted syntactic evaluation, and&#10;logical inference.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.10225" label="1906.10225">
        <attvalues>
          <attvalue for="0" value="Compound Probabilistic Context-Free Grammars for Grammar Induction" />
          <attvalue for="1" value="  We study a formalization of the grammar induction problem that models&#10;sentences as being generated by a compound probabilistic context-free grammar.&#10;In contrast to traditional formulations which learn a single stochastic&#10;grammar, our grammar's rule probabilities are modulated by a per-sentence&#10;continuous latent variable, which induces marginal dependencies beyond the&#10;traditional context-free assumptions. Inference in this grammar is performed by&#10;collapsed variational inference, in which an amortized variational posterior is&#10;placed on the continuous variable, and the latent trees are marginalized out&#10;with dynamic programming. Experiments on English and Chinese show the&#10;effectiveness of our approach compared to recent state-of-the-art methods when&#10;evaluated on unsupervised parsing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.00111" label="1909.00111">
        <attvalues>
          <attvalue for="0" value="Quantity doesn't buy quality syntax with neural language models" />
          <attvalue for="1" value="  Recurrent neural networks can learn to predict upcoming words remarkably well&#10;on average; in syntactically complex contexts, however, they often assign&#10;unexpectedly high probabilities to ungrammatical words. We investigate to what&#10;extent these shortcomings can be mitigated by increasing the size of the&#10;network and the corpus on which it is trained. We find that gains from&#10;increasing network size are minimal beyond a certain point. Likewise, expanding&#10;the training corpus yields diminishing returns; we estimate that the training&#10;corpus would need to be unrealistically large for the models to match human&#10;performance. A comparison to GPT and BERT, Transformer-based models trained on&#10;billions of words, reveals that these models perform even more poorly than our&#10;LSTMs in some constructions. Our results make the case for more data efficient&#10;architectures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.04068" label="1906.04068">
        <attvalues>
          <attvalue for="0" value="Hierarchical Representation in Neural Language Models: Suppression and&#10;  Recovery of Expectations" />
          <attvalue for="1" value="  Deep learning sequence models have led to a marked increase in performance&#10;for a range of Natural Language Processing tasks, but it remains an open&#10;question whether they are able to induce proper hierarchical generalizations&#10;for representing natural language from linear input alone. Work using&#10;artificial languages as training input has shown that LSTMs are capable of&#10;inducing the stack-like data structures required to represent context-free and&#10;certain mildly context-sensitive languages---formal language classes which&#10;correspond in theory to the hierarchical structures of natural language. Here&#10;we present a suite of experiments probing whether neural language models&#10;trained on linguistic data induce these stack-like data structures and deploy&#10;them while incrementally predicting words. We study two natural language&#10;phenomena: center embedding sentences and syntactic island constraints on the&#10;filler--gap dependency. In order to properly predict words in these structures,&#10;a model must be able to temporarily suppress certain expectations and then&#10;recover those expectations later, essentially pushing and popping these&#10;expectations on a stack. Our results provide evidence that models can&#10;successfully suppress and recover expectations in many cases, but do not fully&#10;recover their previous grammatical state.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.03632" label="2001.03632">
        <attvalues>
          <attvalue for="0" value="Does syntax need to grow on trees? Sources of hierarchical inductive&#10;  bias in sequence-to-sequence networks" />
          <attvalue for="1" value="  Learners that are exposed to the same training data might generalize&#10;differently due to differing inductive biases. In neural network models,&#10;inductive biases could in theory arise from any aspect of the model&#10;architecture. We investigate which architectural factors affect the&#10;generalization behavior of neural sequence-to-sequence models trained on two&#10;syntactic tasks, English question formation and English tense reinflection. For&#10;both tasks, the training set is consistent with a generalization based on&#10;hierarchical structure and a generalization based on linear order. All&#10;architectural factors that we investigated qualitatively affected how models&#10;generalized, including factors with no clear connection to hierarchical&#10;structure. For example, LSTMs and GRUs displayed qualitatively different&#10;inductive biases. However, the only factor that consistently contributed a&#10;hierarchical bias across tasks was the use of a tree-structured model rather&#10;than a model with sequential recurrence, suggesting that human-like syntactic&#10;generalization requires architectural syntactic structure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.05738" label="1711.05738">
        <attvalues>
          <attvalue for="0" value="The Neural Network Pushdown Automaton: Model, Stack and Learning&#10;  Simulations" />
          <attvalue for="1" value="  In order for neural networks to learn complex languages or grammars, they&#10;must have sufficient computational power or resources to recognize or generate&#10;such languages. Though many approaches have been discussed, one ob- vious&#10;approach to enhancing the processing power of a recurrent neural network is to&#10;couple it with an external stack memory - in effect creating a neural network&#10;pushdown automata (NNPDA). This paper discusses in detail this NNPDA - its&#10;construction, how it can be trained and how useful symbolic information can be&#10;extracted from the trained network.&#10;  In order to couple the external stack to the neural network, an optimization&#10;method is developed which uses an error function that connects the learning of&#10;the state automaton of the neural network to the learning of the operation of&#10;the external stack. To minimize the error function using gradient descent&#10;learning, an analog stack is designed such that the action and storage of&#10;information in the stack are continuous. One interpretation of a continuous&#10;stack is the probabilistic storage of and action on data. After training on&#10;sample strings of an unknown source grammar, a quantization procedure extracts&#10;from the analog stack and neural network a discrete pushdown automata (PDA).&#10;Simulations show that in learning deterministic context-free grammars - the&#10;balanced parenthesis language, 1*n0*n, and the deterministic Palindrome - the&#10;extracted PDA is correct in the sense that it can correctly recognize unseen&#10;strings of arbitrary length. In addition, the extracted PDAs can be shown to be&#10;identical or equivalent to the PDAs of the source grammars which were used to&#10;generate the training strings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.13466" label="1910.13466">
        <attvalues>
          <attvalue for="0" value="Ordered Memory" />
          <attvalue for="1" value="  Stack-augmented recurrent neural networks (RNNs) have been of interest to the&#10;deep learning community for some time. However, the difficulty of training&#10;memory models remains a problem obstructing the widespread use of such models.&#10;In this paper, we propose the Ordered Memory architecture. Inspired by Ordered&#10;Neurons (Shen et al., 2018), we introduce a new attention-based mechanism and&#10;use its cumulative probability to control the writing and erasing operation of&#10;the memory. We also introduce a new Gated Recursive Cell to compose lower-level&#10;representations into higher-level representation. We demonstrate that our model&#10;achieves strong performance on the logical inference task (Bowman et al.,&#10;2015)and the ListOps (Nangia and Bowman, 2018) task. We can also interpret the&#10;model to retrieve the induced tree structure, and find that these induced&#10;structures align with the ground truth. Finally, we evaluate our model on the&#10;Stanford SentimentTreebank tasks (Socher et al., 2013), and find that it&#10;performs comparatively with the state-of-the-art methods in the literature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12152" label="2305.12152">
        <attvalues>
          <attvalue for="0" value="Revisiting Automated Topic Model Evaluation with Large Language Models" />
          <attvalue for="1" value="  Topic models are used to make sense of large text collections. However,&#10;automatically evaluating topic model output and determining the optimal number&#10;of topics both have been longstanding challenges, with no effective automated&#10;solutions to date. This paper proposes using large language models to evaluate&#10;such output. We find that large language models appropriately assess the&#10;resulting topics, correlating more strongly with human judgments than existing&#10;automated metrics. We then investigate whether we can use large language models&#10;to automatically determine the optimal number of topics. We automatically&#10;assign labels to documents and choosing configurations with the most pure&#10;labels returns reasonable values for the optimal number of topics.&#10;" />
          <attvalue for="2" value="&#10;&#10;Topic models are, loosely put, an unsupervised dimensionality reduction technique that help organize document collections \cite{blei_et_al}.&#10;A topic model summarizes a document collection with a small number of topics.&#10;A topic is a probability distribution over words or phrases.&#10;A topic $T$ is interpretable through a representative set of words or phrases defining the topic, denoted $W_T$.&#10;&#10;Each document can, in turn, be represented as a distribution over topics. For each topic, we can retrieve a representative document collection by sorting documents across topic distributions.&#10;We denote this set of documents for topic $T$ as $D_T$.&#10;Because of their ability to organize large collections of texts, topic models are widely used in the social sciences, digital humanities, and other disciplines to analyze large corpora \cite[][inter alia]{Talley2011-ru, grimmer_stewart_2013, antoniak_birthing_stories, karami_et_al_2020}.\looseness=-1&#10;&#10;Interpretability makes topic models useful, but human interpretation is complex and notoriously difficult to approximate \cite{lipton-2018-mythos}.&#10;Automated topic coherence metrics do not correlate well with human judgments, often overstating differences between models \cite{hoyle2021automated, doogan-buntine-2021-topic}.&#10;Without the guidance of an automated metric, the number of topics, an important hyperparameter, is usually derived manually:&#10;Practitioners fit various topic models, inspect the resulting topics, and select the configuration which works best for the intended use case \cite{hoyle2021automated}.&#10;This is a non-replicable and time-consuming process, requiring expensive expert labor.&#10;&#10;Recent NLP research explores whether large language models (LLMs) can perform automatic annotations; e.g., to assess text quality \cite[][inter alia]{gpt_score, faggioli2023perspectives, huang2023chatgpt}.&#10;Here, we investigate whether LLMs can automatically assess the coherence of topic modeling output and conclude that:&#10;\researchquestionbox{&#10;\begin{enumerate}[(1),left=0mm,topsep=0.1mm,noitemsep]&#10;\item LLMs can accurately judge topic coherence,&#10;\item LLMs can assist in automatically determining reasonable numbers of topics. &#10;\end{enumerate}&#10;}&#10;&#10;We use LLMs for two established topic coherence evaluation tasks and find that their judgment strongly correlates with humans on one of these tasks.&#10;Similar to recent findings, we find that coherent topic word sets $W_T$ do not necessarily imply an optimal categorization of the document collection \cite{doogan-buntine-2021-topic}.&#10;Instead, we automatically assign a label to each document in a $D_T$ and choose the configuration with the purest assigned labels. This solution correlates well with an underlying ground truth. Thus, LLMs can help find good numbers of topics for a text collection, as we show in three case studies.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Topic Model Evaluation, Machine Learning, Computer Science, Large Language Models, Linguistics, Natural Language Processing, Statistics, Text Analysis Optimization" />
        </attvalues>
      </node>
      <node id="2107.02173" label="2107.02173">
        <attvalues>
          <attvalue for="0" value="Is Automated Topic Model Evaluation Broken?: The Incoherence of&#10;  Coherence" />
          <attvalue for="1" value="  Topic model evaluation, like evaluation of other unsupervised methods, can be&#10;contentious. However, the field has coalesced around automated estimates of&#10;topic coherence, which rely on the frequency of word co-occurrences in a&#10;reference corpus. Contemporary neural topic models surpass classical ones&#10;according to these metrics. At the same time, topic model evaluation suffers&#10;from a validation gap: automated coherence, developed for classical models, has&#10;not been validated using human experimentation for neural models. In addition,&#10;a meta-analysis of topic modeling literature reveals a substantial&#10;standardization gap in automated topic modeling benchmarks. To address the&#10;validation gap, we compare automated coherence with the two most widely&#10;accepted human judgment tasks: topic rating and word intrusion. To address the&#10;standardization gap, we systematically evaluate a dominant classical model and&#10;two state-of-the-art neural models on two commonly used datasets. Automated&#10;evaluations declare a winning model when corresponding human evaluations do&#10;not, calling into question the validity of fully automatic evaluations&#10;independent of human judgments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.04166" label="2302.04166">
        <attvalues>
          <attvalue for="0" value="GPTScore: Evaluate as You Desire" />
          <attvalue for="1" value="  Generative Artificial Intelligence (AI) has enabled the development of&#10;sophisticated models that are capable of producing high-caliber text, images,&#10;and other outputs through the utilization of large pre-trained models.&#10;Nevertheless, assessing the quality of the generation is an even more arduous&#10;task than the generation itself, and this issue has not been given adequate&#10;consideration recently. This paper proposes a novel evaluation framework,&#10;GPTScore, which utilizes the emergent abilities (e.g., zero-shot instruction)&#10;of generative pre-trained models to score generated texts. There are 19&#10;pre-trained models explored in this paper, ranging in size from 80M (e.g.,&#10;FLAN-T5-small) to 175B (e.g., GPT3). Experimental results on four text&#10;generation tasks, 22 evaluation aspects, and corresponding 37 datasets&#10;demonstrate that this approach can effectively allow us to achieve what one&#10;desires to evaluate for texts simply by natural language instructions. This&#10;nature helps us overcome several long-standing challenges in text&#10;evaluation--how to achieve customized, multi-faceted evaluation without the&#10;need for annotated samples. We make our code publicly available at&#10;https://github.com/jinlanfu/GPTScore.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.13074" label="2404.13074">
        <attvalues>
          <attvalue for="0" value="Towards Compositionally Generalizable Semantic Parsing in Large Language&#10;  Models: A Survey" />
          <attvalue for="1" value="  Compositional generalization is the ability of a model to generalize to&#10;complex, previously unseen types of combinations of entities from just having&#10;seen the primitives. This type of generalization is particularly relevant to&#10;the semantic parsing community for applications such as task-oriented dialogue,&#10;text-to-SQL parsing, and information retrieval, as they can harbor infinite&#10;complexity. Despite the success of large language models (LLMs) in a wide range&#10;of NLP tasks, unlocking perfect compositional generalization still remains one&#10;of the few last unsolved frontiers. The past few years has seen a surge of&#10;interest in works that explore the limitations of, methods to improve, and&#10;evaluation metrics for compositional generalization capabilities of LLMs for&#10;semantic parsing tasks. In this work, we present a literature survey geared at&#10;synthesizing recent advances in analysis, methods, and evaluation schemes to&#10;offer a starting point for both practitioners and researchers in this area.&#10;" />
          <attvalue for="2" value="&#10;&#10;Semantic parsing is a general-purpose task that involves translating natural language input into a structured (and usually symbolic) output. The real-world applications of semantic parsing are wide-ranging and often mission-critical. They range from task-oriented dialogue to text-to-SQL business applications to information retrieval. Compositionality is the ability of a model to understand previously unseen combinations ``jump left'' (compound) of primitive concepts (atoms) such as ``jump`` and ``left'' without ever having been trained on the compound directly.&#10;&#10;Today, large language model based semantic parsers dominate the semantic parsing scene in both research and applied settings owing to the unprecedented representational power and flexibility that they offer for capturing rich natural language as compared to their predecessors.&#10; However, despite demonstrating state-of-the-art results over standard benchmarks, careful error analyses show that these models fail to generalize to cases where the output is a symbolic form containing nested structures of depths greater than what is seen by the models during training \cite{3-yao-structural-generalization}.&#10;&#10;More generally, although existing benchmark corpora aid in evaluating the strengths and weaknesses of a model, they only offer a coarse-grained view of the its underlying capabilities.&#10; To overcome this coarseness, the semantic parsing community has expended significant research effort to develop corpora with more purposeful train-test splits, with the aim of leveraging them to run more fine-grained experiments.&#10;&#10;The task of categorizing works that seek to quantify and improve the compositional generalization capabilities of LLMs is quite challenging. Nevertheless, we begin to ``chip away at the block'' by looking at some of the key guiding philosophies adopted by contemporary works.&#10;&#10;Several lines of research stem from the semantic parsing community's dissatisfaction with the performance of ``vanilla training'' of Transformer-based LLMs such as T5 \cite{raffel2023exploring} and BART \cite{lewis-etal-2020-bart}, which have shown success in purely language-based tasks such as summarization and question-answering.&#10;&#10;If understanding a model's shortcomings is the first step to ``chipping away at the block,'' addressing those shortcomings is the next natural step. Almost all LLM-based approaches can be thought of as involving a combination of ``data components'' and ``modeling components.''&#10;Data components include the pre-training corpus, the fine-tuning corpus, and any data augmentation strategies used. Modeling components, on the other hand, include the fine-tuning method, prompting strategies, post-processing steps applied over the LLM output, and ensembling methods to combine multiple primitive models.&#10;Most contemporary works on compositional generalization in semantic parsing narrowly focus on one or more of these components (typically, not more than two at a time). For this reason, these works are best interpreted as orthogonal, complementary strategies to improve compositional generalization capabilities.&#10;&#10;The paper is structured as follows: Section \ref{sec:definitions} delves into compositional semantic parsing, examining three widely-used benchmark corpora and their compositional generalization definitions. Section \ref{sec:methods} details the literature survey methodology, including search and selection criteria. Section \ref{sec:factors} discusses factors influencing LLM-based semantic parsers' compositional generalization. Section \ref{sec:improve} reviews literature on methods to enhance compositional generalization, covering data-augmentation, neuro-symbolic modeling, and prompt-based methods. Section \ref{sec:discussion} discusses research trends in compositional semantic parsing using LLMs. The paper concludes in Section \ref{sec:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Semantic Parsing, Computer Science, Large Language Models, Linguistics, Compositional Generalization, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2210.13050" label="2210.13050">
        <attvalues>
          <attvalue for="0" value="Structural generalization is hard for sequence-to-sequence models" />
          <attvalue for="1" value="  Sequence-to-sequence (seq2seq) models have been successful across many NLP&#10;tasks, including ones that require predicting linguistic structure. However,&#10;recent work on compositional generalization has shown that seq2seq models&#10;achieve very low accuracy in generalizing to linguistic structures that were&#10;not seen in training. We present new evidence that this is a general limitation&#10;of seq2seq models that is present not just in semantic parsing, but also in&#10;syntactic parsing and in text-to-text tasks, and that this limitation can often&#10;be overcome by neurosymbolic models that have linguistic knowledge built in. We&#10;further report on some experiments that give initial answers on the reasons for&#10;these limitations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.13461" label="1910.13461">
        <attvalues>
          <attvalue for="0" value="BART: Denoising Sequence-to-Sequence Pre-training for Natural Language&#10;  Generation, Translation, and Comprehension" />
          <attvalue for="1" value="  We present BART, a denoising autoencoder for pretraining sequence-to-sequence&#10;models. BART is trained by (1) corrupting text with an arbitrary noising&#10;function, and (2) learning a model to reconstruct the original text. It uses a&#10;standard Tranformer-based neural machine translation architecture which,&#10;despite its simplicity, can be seen as generalizing BERT (due to the&#10;bidirectional encoder), GPT (with the left-to-right decoder), and many other&#10;more recent pretraining schemes. We evaluate a number of noising approaches,&#10;finding the best performance by both randomly shuffling the order of the&#10;original sentences and using a novel in-filling scheme, where spans of text are&#10;replaced with a single mask token. BART is particularly effective when fine&#10;tuned for text generation but also works well for comprehension tasks. It&#10;matches the performance of RoBERTa with comparable training resources on GLUE&#10;and SQuAD, achieves new state-of-the-art results on a range of abstractive&#10;dialogue, question answering, and summarization tasks, with gains of up to 6&#10;ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system&#10;for machine translation, with only target language pretraining. We also report&#10;ablation experiments that replicate other pretraining schemes within the BART&#10;framework, to better measure which factors most influence end-task performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Early methods for pretraining were based on language models. GPT \cite{gpt} only models leftward context, which is problematic for some tasks. ELMo \cite{elmo} concatenates left-only and right-only representations, but does not pre-train interactions between these features.&#10;\cite{gpt2} demonstrated that very large language models can act as unsupervised multitask models.&#10;&#10;BERT \cite{bert} introduced masked language modelling, which allows pre-training to learn interactions between left and right context words. &#10;Recent work has shown that very strong performance can be achieved by training for longer \cite{roberta}, by tying parameters across layers \cite{albert}, and by masking spans instead of words \cite{spanbert}.&#10;Predictions are not made auto-regressively, reducing the effectiveness of BERT for generation tasks.&#10;&#10;UniLM \cite{unilm} fine-tunes BERT with an ensemble of masks, some of which allow only leftward context. Like BART, this allows UniLM to be used for both generative and discriminative tasks. A difference is that UniLM predictions are conditionally independent, whereas BART's are autoregressive. BART reduces the mismatch between pre-training and generation tasks, because the decoder is always trained on uncorrupted context. &#10;&#10;MASS \cite{mass} is perhaps the most similar model to BART. An input sequence where a contiguous span of tokens is masked is mapped to a sequence consisting of the missing tokens. MASS is less effective for discriminative tasks, because disjoint sets of tokens are fed into the encoder and decoder.&#10;&#10;XL-Net \cite{xlnet} extends BERT by predicting masked tokens auto-regressively in a permuted order. This objective allows predictions to condition on both left and right context. In contrast, the BART decoder works left-to-right during pre-training, matching the setting during generation.&#10;&#10;Several papers have explored using pre-trained representations to improve machine translation. The largest improvements have come from pre-training on both source and target languages \cite{mass,xlm}, but this requires pre-training on all languages of interest. Other work has shown that encoders can be improved using pre-trained representations \cite{edunov2019pre}, but gains in decoders are more limited. We show how BART can be used to improve machine translation decoders.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.11355" label="2402.11355">
        <attvalues>
          <attvalue for="0" value="Natural Language Counterfactuals through Representation Surgery" />
          <attvalue for="1" value="  Interventions targeting the representation space of language models (LMs)&#10;have emerged as an effective means to influence model behavior. Such methods&#10;are employed, for example, to eliminate or alter the encoding of demographic&#10;information such as gender within the model's representations and, in so doing,&#10;create a counterfactual representation. However, because the intervention&#10;operates within the representation space, understanding precisely what aspects&#10;of the text it modifies poses a challenge. In this paper, we give a method to&#10;convert representation counterfactuals into string counterfactuals. We&#10;demonstrate that this approach enables us to analyze the linguistic alterations&#10;corresponding to a given representation space intervention and to interpret the&#10;features utilized to encode a specific concept. Moreover, the resulting&#10;counterfactuals can be used to mitigate bias in classification through data&#10;augmentation.&#10;" />
          <attvalue for="2" value="&#10;Interventions performed in the representation space of language models (LMs), generally $\RD$, have proven effective in understanding and exerting control over neural language models \cite{ravfogel2020null, ravfogel2021counterfactual, geva2021transformer, elazar2021amnesic, ravfogel2022adversarial, ravfogel-etal-2023-linear, belrose2023leace,li2023inference}. One popular set of techniques \defn{erases} the linear subspace associated with a human-interpretable concept $c$, e.g., \concept{gender} or \concept{sentiment}. Another widely used approach is to \defn{steer} representations from one class to another, e.g., shifting them toward a region in the representation space associated with a different class $c'$ \cite{subramani2022extracting, li2023inference, ravfogel2021counterfactual, singh2024mimic}. For instance, they could steer a representation into a region associated with negative sentiment, thereby creating counterfactual representations. In this paper, we propose a technique to generate strings that correspond to representation-level counterfactuals, which we denote as string counterfactuals.&#10;&#10;Collectively, we refer to representation space intervention techniques as \defn{representation surgery} because they (surgically) intervene in the encoding of a concept within the representation while keeping the rest of the representation as similar as possible. In this sense, representation surgery resembles a causal intervention \cite{vig2020causal, geiger2021causal, feder2021causalm, pmlr-v162-geiger22a, guerner2023geometric, lemberger2024explaining}, and we will informally use causal language throughout the paper, referring to such modifications in the representation space as interventions. In notation, we write $\intervene_{c arrow c'} \colon \RD arrow \RD$ for a function that performs such an intervention.&#10;&#10;While representation surgery techniques can create counterfactual variants of the original representations, they do not produce them at the level of natural language text. In this work, we tackle the problem of generating the counterfactual string that corresponds to a specific representation intervention. Despite the abundance of research on representation surgery, translating such interventions into string counterfactuals remains understudied. We refer to this process as a \defn{counterfactual lens}, as it allows us to interpret representation-space counterfactuals in natural language, similar to representation-level interpretability techniques \cite{meng2022locating, nostalgebraist, belrose2023eliciting, ghandeharioun2024patchscope}. &#10;Constructing string counterfactuals serves various practical purposes. First, it offers a method of \defn{meta-interpretability}, aiding in the interpretation of commonly used representational intervention techniques, which themselves are often employed for interpretability. By mapping representational interventions back to the string, we can observe the lexical and higher-level semantic shifts triggered by the intervention. Second, string counterfactuals are a natural choice for data augmentation. Indeed, we demonstrate their potential to address fairness concerns in a real-world classification problem.\looseness=-1&#10;&#10;We follow \cite{morris2023text} in developing an approach for generating string counterfactuals from representation interventions.&#10;Let $\alphabet$ be an alphabet.&#10;Consider a neural network that performs a mapping from a string $\str \in \kleene{\alphabet}$ to a representation $\rep = \enc(\str) \in \RD$.&#10;\cite{morris2023text} propose an iterative algorithm to &#10;approximate the inverse function $\inv \colon \RD arrow \kleene{\alphabet}$.&#10;We exploit the \citeoss{morris2023text} algorithm to construct a string counterfactual corresponding to a surgical intervention in the representation space.&#10;Using the notation introduced so far, we are interested in computing $\str' = \inv(\intervene_{c arrow c'}(\enc(\str)))$.&#10;To the extent that $\inv$ constitutes a suitable inverse, we expect $\str'$ to be a minimally different version of $\str$ that reflects the difference between $\rep$ and $\rep'$ reflected in the representation space.&#10;&#10;We perform experiments on a dataset of short biographies annotated with gender and profession \cite{de2019bias}. We find that swapping gender in the representation space and then generating the inverse is an effective method for producing string counterfactuals. The resulting counterfactuals exhibit some degree of gender bias, for example, a tendency to include more profession-related words in male biographies, suggesting that LMs encode subtle cues correlated with gender beyond pronouns (\ref{sec:analysis}). We further show that these counterfactuals can be used for data augmentation to improve fairness in a multiclass classification task (\ref{sec:bias}): specifically, classifiers trained on both original and counterfactual biographies (with respect to gender) exhibit reduced gender bias compared to those trained solely on the original data.&#10;&#10;{&#10;&#10;}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Bias Mitigation Techniques, Computer Science, Linguistics, Representation Space Analysis, Artificial Intelligence, Mathematics, Natural Language Processing, Language Model Interventions" />
        </attvalues>
      </node>
      <node id="2004.07667" label="2004.07667">
        <attvalues>
          <attvalue for="0" value="Null It Out: Guarding Protected Attributes by Iterative Nullspace&#10;  Projection" />
          <attvalue for="1" value="  The ability to control for the kinds of information encoded in neural&#10;representation has a variety of use cases, especially in light of the challenge&#10;of interpreting these models. We present Iterative Null-space Projection&#10;(INLP), a novel method for removing information from neural representations.&#10;Our method is based on repeated training of linear classifiers that predict a&#10;certain property we aim to remove, followed by projection of the&#10;representations on their null-space. By doing so, the classifiers become&#10;oblivious to that target property, making it hard to linearly separate the data&#10;according to it. While applicable for multiple uses, we evaluate our method on&#10;bias and fairness use-cases, and show that our method is able to mitigate bias&#10;in word embeddings, as well as to increase fairness in a setting of multi-class&#10;classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.03341" label="2306.03341">
        <attvalues>
          <attvalue for="0" value="Inference-Time Intervention: Eliciting Truthful Answers from a Language&#10;  Model" />
          <attvalue for="1" value="  We introduce Inference-Time Intervention (ITI), a technique designed to&#10;enhance the &quot;truthfulness&quot; of large language models (LLMs). ITI operates by&#10;shifting model activations during inference, following a set of directions&#10;across a limited number of attention heads. This intervention significantly&#10;improves the performance of LLaMA models on the TruthfulQA benchmark. On an&#10;instruction-finetuned LLaMA called Alpaca, ITI improves its truthfulness from&#10;32.5% to 65.1%. We identify a tradeoff between truthfulness and helpfulness and&#10;demonstrate how to balance it by tuning the intervention strength. ITI is&#10;minimally invasive and computationally inexpensive. Moreover, the technique is&#10;data efficient: while approaches like RLHF require extensive annotations, ITI&#10;locates truthful directions using only few hundred examples. Our findings&#10;suggest that LLMs may have an internal representation of the likelihood of&#10;something being true, even as they produce falsehoods on the surface.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.05124" label="2205.05124">
        <attvalues>
          <attvalue for="0" value="Extracting Latent Steering Vectors from Pretrained Language Models" />
          <attvalue for="1" value="  Prior work on controllable text generation has focused on learning how to&#10;control language models through trainable decoding, smart-prompt design, or&#10;fine-tuning based on a desired objective. We hypothesize that the information&#10;needed to steer the model to generate a target sentence is already encoded&#10;within the model. Accordingly, we explore a different approach altogether:&#10;extracting latent vectors directly from pretrained language model decoders&#10;without fine-tuning. Experiments show that there exist steering vectors, which,&#10;when added to the hidden states of the language model, generate a target&#10;sentence nearly perfectly (&gt; 99 BLEU) for English sentences from a variety of&#10;domains. We show that vector arithmetic can be used for unsupervised sentiment&#10;transfer on the Yelp sentiment benchmark, with performance comparable to models&#10;tailored to this task. We find that distances between steering vectors reflect&#10;sentence similarity when evaluated on a textual similarity benchmark (STS-B),&#10;outperforming pooled hidden states of models. Finally, we present an analysis&#10;of the intrinsic properties of the steering vectors. Taken together, our&#10;results suggest that frozen LMs can be effectively controlled through their&#10;latent steering space.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;There exist many works, often using text-based autoencoders that try to induce a sentence representation space for controllable text generation by learning new models~\cite{Hu2017TowardCG, Shen2017StyleTF, Shen2020EducatingTA,&#10;mai-etal-2020-plug,&#10;montero-etal-2021-sentence}.&#10;Our work concludes that we can extract steering vectors from pretrained models that have latent spaces that allow operations like this, without having to train any new models at all.&#10;Other approaches control language models by adapting their hidden states using steerable layers, adapters, or steering their logits using auxiliary language models~\cite{gulcehre2015using, dathathri2019plug, Houlsby2019ParameterEfficientTL, Zhang2020SideTuningAB, liu-etal-2021-dexperts, krause-etal-2021-gedi-generative}.&#10;Our method differs from all of these: we extract steering vectors directly from a language model and operate on the latent space occupied by these vectors, never fine-tuning any component of the model.&#10;\cite{subramani2019can} investigate whether LSTM-based language models have sentence representations from which they can generate the original sentence. &#10;Although this premise relates to our first question: can we extract steering vectors, we extend far beyond that showing that vector arithmetic in the context of unsupervised style transfer is effective in our latent steering space.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2004.12265" label="2004.12265">
        <attvalues>
          <attvalue for="0" value="Causal Mediation Analysis for Interpreting Neural NLP: The Case of&#10;  Gender Bias" />
          <attvalue for="1" value="  Common methods for interpreting neural models in natural language processing&#10;typically examine either their structure or their behavior, but not both. We&#10;propose a methodology grounded in the theory of causal mediation analysis for&#10;interpreting which parts of a model are causally implicated in its behavior. It&#10;enables us to analyze the mechanisms by which information flows from input to&#10;output through various model components, known as mediators. We apply this&#10;methodology to analyze gender bias in pre-trained Transformer language models.&#10;We study the role of individual neurons and attention heads in mediating gender&#10;bias across three datasets designed to gauge a model's sensitivity to gender&#10;bias. Our mediation analysis reveals that gender bias effects are (i) sparse,&#10;concentrated in a small part of the network; (ii) synergistic, amplified or&#10;repressed by different components; and (iii) decomposable into effects flowing&#10;directly from the input and indirectly through the mediators.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.05262" label="2202.05262">
        <attvalues>
          <attvalue for="0" value="Locating and Editing Factual Associations in GPT" />
          <attvalue for="1" value="  We analyze the storage and recall of factual associations in autoregressive&#10;transformer language models, finding evidence that these associations&#10;correspond to localized, directly-editable computations. We first develop a&#10;causal intervention for identifying neuron activations that are decisive in a&#10;model's factual predictions. This reveals a distinct set of steps in&#10;middle-layer feed-forward modules that mediate factual predictions while&#10;processing subject tokens. To test our hypothesis that these computations&#10;correspond to factual association recall, we modify feed-forward weights to&#10;update specific factual associations using Rank-One Model Editing (ROME). We&#10;find that ROME is effective on a standard zero-shot relation extraction (zsRE)&#10;model-editing task, comparable to existing methods. To perform a more sensitive&#10;evaluation, we also evaluate ROME on a new dataset of counterfactual&#10;assertions, on which it simultaneously maintains both specificity and&#10;generalization, whereas other methods sacrifice one or another. Our results&#10;confirm an important role for mid-layer feed-forward modules in storing factual&#10;associations and suggest that direct manipulation of computational mechanisms&#10;may be a feasible approach for model editing. The code, dataset,&#10;visualizations, and an interactive demo notebook are available at&#10;https://rome.baulab.info/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.06816" label="2310.06816">
        <attvalues>
          <attvalue for="0" value="Text Embeddings Reveal (Almost) As Much As Text" />
          <attvalue for="1" value="  How much private information do text embeddings reveal about the original&#10;text? We investigate the problem of embedding \textit{inversion},&#10;reconstructing the full text represented in dense text embeddings. We frame the&#10;problem as controlled generation: generating text that, when reembedded, is&#10;close to a fixed point in latent space. We find that although a na\&quot;ive model&#10;conditioned on the embedding performs poorly, a multi-step method that&#10;iteratively corrects and re-embeds text is able to recover $92\%$ of&#10;$32\text{-token}$ text inputs exactly. We train our model to decode text&#10;embeddings from two state-of-the-art embedding models, and also show that our&#10;model can recover important personal information (full names) from a dataset of&#10;clinical notes. Our code is available on Github:&#10;\href{https://github.com/jxmorris12/vec2text}{github.com/jxmorris12/vec2text}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.09451" label="1901.09451">
        <attvalues>
          <attvalue for="0" value="Bias in Bios: A Case Study of Semantic Representation Bias in a&#10;  High-Stakes Setting" />
          <attvalue for="1" value="  We present a large-scale study of gender bias in occupation classification, a&#10;task where the use of machine learning may lead to negative outcomes on&#10;peoples' lives. We analyze the potential allocation harms that can result from&#10;semantic representation bias. To do so, we study the impact on occupation&#10;classification of including explicit gender indicators---such as first names&#10;and pronouns---in different semantic representations of online biographies.&#10;Additionally, we quantify the bias that remains when these indicators are&#10;&quot;scrubbed,&quot; and describe proxy behavior that occurs in the absence of explicit&#10;gender indicators. As we demonstrate, differences in true positive rates&#10;between genders are correlated with existing gender imbalances in occupations,&#10;which may compound these imbalances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.01200" label="2306.01200">
        <attvalues>
          <attvalue for="0" value="Multi-Dimensional Evaluation of Text Summarization with In-Context&#10;  Learning" />
          <attvalue for="1" value="  Evaluation of natural language generation (NLG) is complex and&#10;multi-dimensional. Generated text can be evaluated for fluency, coherence,&#10;factuality, or any other dimensions of interest. Most frameworks that perform&#10;such multi-dimensional evaluation require training on large manually or&#10;synthetically generated datasets. In this paper, we study the efficacy of large&#10;language models as multi-dimensional evaluators using in-context learning,&#10;obviating the need for large training datasets. Our experiments show that&#10;in-context learning-based evaluators are competitive with learned evaluation&#10;frameworks for the task of text summarization, establishing state-of-the-art on&#10;dimensions such as relevance and factual consistency. We then analyze the&#10;effects of factors such as the selection and number of in-context examples on&#10;performance. Finally, we study the efficacy of in-context learning based&#10;evaluators in evaluating zero-shot summaries written by large language models&#10;such as GPT-3.&#10;" />
          <attvalue for="2" value=" &#10;Developing comprehensive evaluation frameworks~\cite{deng-2021-compression, yuan-2021-bartscore, zhong-2022-towards} that can evaluate multiple human-interpretable dimensions, &#10;such as factual consistency~\cite{kryscinski-2020-evaluating, wang-2020-asking} and coherence~\cite{dziri-2019-evaluating, huang-2020-grade}, is important for the advancement of Natural Language Generation (NLG). &#10;However, similarity-based metrics~\cite{papineni-2002-bleu, lin-2004-rouge, sellam-2020-bleurt, zhao-2019-moverscore, zhang-2020-bertscore} still dominate NLG evaluation in practice. Compared to them, desired multi-dimensional evaluators do not require reference texts for evaluation; and they can easily extend to new explainable evaluation dimensions. &#10;Recently, \cite{zhong-2022-towards} developed a unified evaluation framework that can generalize to multiple dimensions and text generation tasks.&#10;However, it relies on the construction of synthetic and auxiliary data for the finetuning of a pre-trained language model, requiring in-depth knowledge and significant engineering effort for each dimension. Furthermore, the inclusion of new dimensions requires (continued) training of the model, and might affect the performance on other dimensions in unforeseen ways. &#10;&#10;In this work, we propose to use in-context learning ~\cite{brown-2020-language} with large language models (LLMs) --- a commonly used method to perform many tasks by utilizing only a few input-output examples ---&#10;to perform multi-dimensional text evaluation in a unified fashion. Compared to pre-trained evaluators that need specialized supervised training for each dimension, our In-Context learning-based Evaluator (Ice) framework is:&#10;&#10;margini=10pt\begin{itemize}&#10; \item Learning-free. It does not require supervised fine-tuning on large annotated (synthetic) training data, requiring only a handful of samples at inference time.&#10; \item Extensible. To evaluate new dimensions, it does not rely on large amounts of human judgments or the construction of new synthetic data, using only a natural language prompt consisting of a small number of example pairs to ascertain the properties associated with a given quality aspect. &#10;\end{itemize}&#10;&#10;In this paper, using text summarization as a test bed, we show that with a simple prompt design, Ice is competitive with state-of-the-art trained evaluators on multi-dimensional evaluation of model-produced summaries, establishing a new state-of-the-art on dimensions such as relevance and factual consistency. To study the robustness of the evaluator to the selection of in-context examples, we analyze the factors that affect the performance of Ice, such as the number of in-context examples and sampling procedures when picking in-context examples from a set of candidates.&#10;We find Ice to be robust to the selection of in-context examples and observe a slight improvement in performance as the number of examples is increased. Finally, in light of the recent work~\cite{goyal-news-2022} that points to the misalignment &#10;of existing evaluation metrics with human preference in evaluating zero-shot summaries generated by LLMs such as GPT-3 \cite{brown-2020-language}, we study the effectiveness of Ice in evaluating zero-shot summaries generated by GPT-3. We find that Ice evaluations agree closely with human judgments on such summaries. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Evaluation Frameworks, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2109.06379" label="2109.06379">
        <attvalues>
          <attvalue for="0" value="Compression, Transduction, and Creation: A Unified Framework for&#10;  Evaluating Natural Language Generation" />
          <attvalue for="1" value="  Natural language generation (NLG) spans a broad range of tasks, each of which&#10;serves for specific objectives and desires different properties of generated&#10;text. The complexity makes automatic evaluation of NLG particularly&#10;challenging. Previous work has typically focused on a single task and developed&#10;individual evaluation metrics based on specific intuitions. In this paper, we&#10;propose a unifying perspective that facilitates the design of metrics for a&#10;wide range of language generation tasks and quality aspects. Based on the&#10;nature of information change from input to output, we classify NLG tasks into&#10;compression (e.g., summarization), transduction (e.g., text rewriting), and&#10;creation (e.g., dialog). The information alignment, or overlap, between input,&#10;context, and output text plays a common central role in characterizing the&#10;generation. Using the uniform concept of information alignment, we develop a&#10;family of interpretable metrics for various NLG tasks and aspects, often&#10;without need of gold reference data. To operationalize the metrics, we train&#10;self-supervised models to approximate information alignment as a prediction&#10;task. Experiments show the uniformly designed metrics achieve stronger or&#10;comparable correlations with human judgement compared to state-of-the-art&#10;metrics in each of diverse tasks, including text summarization, style transfer,&#10;and knowledge-grounded dialog. With information alignment as the intermediate&#10;representation, we deliver a composable library for easy NLG evaluation and&#10;future metric design.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.12840" label="1910.12840">
        <attvalues>
          <attvalue for="0" value="Evaluating the Factual Consistency of Abstractive Text Summarization" />
          <attvalue for="1" value="  Currently used metrics for assessing summarization algorithms do not account&#10;for whether summaries are factually consistent with source documents. We&#10;propose a weakly-supervised, model-based approach for verifying factual&#10;consistency and identifying conflicts between source documents and a generated&#10;summary. Training data is generated by applying a series of rule-based&#10;transformations to the sentences of source documents. The factual consistency&#10;model is then trained jointly for three tasks: 1) identify whether sentences&#10;remain factually consistent after transformation, 2) extract a span in the&#10;source documents to support the consistency prediction, 3) extract a span in&#10;the summary sentence that is inconsistent if one exists. Transferring this&#10;model to summaries generated by several state-of-the art models reveals that&#10;this highly scalable approach substantially outperforms previous models,&#10;including those trained with strong supervision using standard datasets for&#10;natural language inference and fact checking. Additionally, human evaluation&#10;shows that the auxiliary span extraction tasks provide useful assistance in the&#10;process of verifying factual consistency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.03371" label="1904.03371">
        <attvalues>
          <attvalue for="0" value="Evaluating Coherence in Dialogue Systems using Entailment" />
          <attvalue for="1" value="  Evaluating open-domain dialogue systems is difficult due to the diversity of&#10;possible correct answers. Automatic metrics such as BLEU correlate weakly with&#10;human annotations, resulting in a significant bias across different models and&#10;datasets. Some researchers resort to human judgment experimentation for&#10;assessing response quality, which is expensive, time consuming, and not&#10;scalable. Moreover, judges tend to evaluate a small number of dialogues,&#10;meaning that minor differences in evaluation configuration may lead to&#10;dissimilar results. In this paper, we present interpretable metrics for&#10;evaluating topic coherence by making use of distributed sentence&#10;representations. Furthermore, we introduce calculable approximations of human&#10;judgment based on conversational coherence by adopting state-of-the-art&#10;entailment techniques. Results show that our metrics can be used as a surrogate&#10;for human judgment, making it easy to evaluate dialogue systems on large-scale&#10;datasets and allowing an unbiased estimate for the quality of the responses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.07197" label="2210.07197">
        <attvalues>
          <attvalue for="0" value="Towards a Unified Multi-Dimensional Evaluator for Text Generation" />
          <attvalue for="1" value="  Multi-dimensional evaluation is the dominant paradigm for human evaluation in&#10;Natural Language Generation (NLG), i.e., evaluating the generated text from&#10;multiple explainable dimensions, such as coherence and fluency. However,&#10;automatic evaluation in NLG is still dominated by similarity-based metrics, and&#10;we lack a reliable framework for a more comprehensive evaluation of advanced&#10;models. In this paper, we propose a unified multi-dimensional evaluator UniEval&#10;for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task,&#10;and by guiding the model with different questions, we can use one evaluator to&#10;evaluate from multiple dimensions. Furthermore, thanks to the unified Boolean&#10;QA format, we are able to introduce an intermediate learning phase that enables&#10;UniEval to incorporate external knowledge from multiple related tasks and gain&#10;further improvement. Experiments on three typical NLG tasks show that UniEval&#10;correlates substantially better with human judgments than existing metrics.&#10;Specifically, compared to the top-performing unified evaluators, UniEval&#10;achieves a 23% higher correlation on text summarization, and over 43% on&#10;dialogue response generation. Also, UniEval demonstrates a strong zero-shot&#10;learning ability for unseen evaluation dimensions and tasks. Source code, data&#10;and all pre-trained evaluators are available on our GitHub repository&#10;(https://github.com/maszhongming/UniEval).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Similarity-based Metrics&#10;Similarity-based metrics refer to the scores for evaluating the NLG models by measuring the similarity between a generated text and a reference text.&#10;They can be divided into lexical overlap-based~\cite{papineni2002bleu, lin2004rouge, banerjee2005meteor} as well as contextualized embedding-based ~\cite{zhang2019bertscore, zhao2019moverscore, clark2019sentence} evaluators.&#10;Although more than 60\% of recent NLG papers solely use ROUGE or BLEU as the evaluation metric~\cite{kasai2021bidimensional}, they fail to measure content quality~\cite{reiter2009investigation} and syntactic correctness~\cite{stent2005evaluating}, and are thus insufficient to portray the reliability of NLG systems.&#10;&#10;Single-dimensional Evaluator&#10;To conduct more fine-grained evaluations for NLG, recent studies develop evaluators for a specific dimension, such as consistency in summarization~\cite{kryscinski2020evaluating, wang2020asking, cao2020factual, durmus2020feqa} and coherence in dialogue response generation~\cite{dziri2019evaluating,huang2020grade, ye2021towards}.&#10;These evaluators can help us better understand the characteristics of advanced NLG models from different perspectives.&#10;However, considering that most dimensions currently have no corresponding standard evaluators, solely using multiple single-dimensional evaluators to perform multi-dimensional evaluation is hard to achieve.&#10;&#10;Unified Evaluator&#10;Several recent evaluators can predict multiple numbers for evaluating text by using different input and output contents~\cite{yuan2021bartscore}, multiple model variants~\cite{mehri2020usr}, or different formulas~\cite{scialom2021questeval}, and we refer to them as unified evaluators.&#10;These evaluation scores usually have no corresponding explanations or are simply categorized as precision, recall, and $\mathrm{F}_1$, which poses difficulties in how to use them.&#10;Therefore, we propose a unified multi-dimensional evaluator in this paper, which attempts to align the evaluation scores with different dimensions in human evaluation.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2209.12356" label="2209.12356">
        <attvalues>
          <attvalue for="0" value="News Summarization and Evaluation in the Era of GPT-3" />
          <attvalue for="1" value="  The recent success of prompting large language models like GPT-3 has led to a&#10;paradigm shift in NLP research. In this paper, we study its impact on text&#10;summarization, focusing on the classic benchmark domain of news summarization.&#10;First, we investigate how GPT-3 compares against fine-tuned models trained on&#10;large summarization datasets. We show that not only do humans overwhelmingly&#10;prefer GPT-3 summaries, prompted using only a task description, but these also&#10;do not suffer from common dataset-specific issues such as poor factuality.&#10;Next, we study what this means for evaluation, particularly the role of gold&#10;standard test sets. Our experiments show that both reference-based and&#10;reference-free automatic metrics cannot reliably evaluate GPT-3 summaries.&#10;Finally, we evaluate models on a setting beyond generic summarization,&#10;specifically keyword-based summarization, and show how dominant fine-tuning&#10;approaches compare to prompting.&#10;  To support further research, we release: (a) a corpus of 10K generated&#10;summaries from fine-tuned and prompt-based models across 4 standard&#10;summarization benchmarks, (b) 1K human preference judgments comparing different&#10;systems for generic- and keyword-based summarization.&#10;" />
          <attvalue for="2" value="&#10;Fine-tuning pre-trained models on domain-specific datasets has been the leading paradigm in text summarization research in recent years \cite{lewis2020bart, zhang2020pegasus, raffel2020exploring}. These models generate high-quality summaries on standard benchmarks, but still require sizeable training datasets to adapt to new settings, e.g., summarizing data from a new source domain or producing a summary in a different style. &#10;The success of prompting large language models (GPT-3 \cite{brown2020language}, T0 \cite{sanh2022multitask}, PaLM \cite{chowdhery2022palm}, etc.) provides an alternative approach, namely learning from natural language task instructions and/or a few demonstrative examples in the context without updating model parameters. While recent work \cite{zhao2021calibrate, min2022rethinking, ye2022unreliability} has evaluated this paradigm across a number of tasks, it has only been studied for text summarization with unreliable automatic metrics \cite{he2022z, chowdhery2022palm, ouyang2022training} or in non-standard settings \cite{saunders2022self}.&#10;&#10;In this paper, we conduct the first systematic study of the impact of prompt-based models on the text summarization research space, using an Instruct-tuned 175B GPT-3 model (text-davinci-002) \cite{brown2020language, ouyang2022training} as a case study. Figure~\ref{fig:length-keyword} shows that GPT-3 summaries are extremely high-quality and adaptable to different summarization settings. Starting from these observations, we aim to answer three main questions. First, how do prompt-based GPT-3 summaries compare to those obtained from state-of-the-art fine-tuned summarization models \cite{zhang2020pegasus, liu2022brio}?&#10;We compare these approaches using A/B testing on a new corpus of recent news articles, and find that our study participants overwhelmingly prefer GPT-3 summaries across two different ``styles'' with different prompts (three-sentence and single-sentence). &#10;Moreover, these summaries do not suffer from limitations due to low-quality training data that plague fine-tuned generic summarization models \cite{maynez2020faithfulness, goyal2022training}.&#10;&#10;Second, are existing automatic metrics well-suited to evaluating prompt-based summaries? Recent work has shown that classic reference-based such as ROUGE \cite{lin2004rouge} and BERTScore \cite{bertscore2020} are unreliable when small improvements are reported \cite{peyrard2019studying, fabbri2021summeval}; however large differences, on the order of say $5$ Rouge points or greater, are considered to be correlated with human preferences \cite{bhandari2020metrics, deutsch-etal-2022-examining}. However, we find that the same is no longer true when evaluating GPT-3 summaries. These summaries score much lower on automatic metrics ($7$ ROUGE-L points on average) than all prior state-of-the-art models while comfortably outperforming them on human evaluation. Furthermore, we show that recent reference-free metrics, e.g. QA-based metrics \cite{fabbri-etal-2022-qafacteval, durmus2020feqa} and trained factuality models \cite{kryscinski2020evaluating, goyal2020evaluating}, similarly fail to adapt to this shift from the fine-tuned to prompting, and need to be re-visited. &#10;&#10;Finally, how can prompting be used beyond generic summarization? We focus on keyword-based and aspect-based summarization. For keyword-based summarization, we find that GPT-3 consistently generates more coherent and keyword-relevant summaries compared to current fine-tuned alternatives: crowd annotators prefer GPT-3 summaries over a baseline model \cite{he2020ctrlsum} 70\% of the time. We observe mixed results for the aspect-based setting, where GPT-3 summaries show frequent failure cases with simple prompts.&#10;&#10;Taken together, this evidence suggests that GPT-3 represents a fundamental paradigm shift in summarization, changing what data we need (or don't need) and what approaches we can now explore. Evaluating these systems will require a new framework distinct from the automatic metrics that have dominated the last decade of summarization research.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00800" label="2005.00800">
        <attvalues>
          <attvalue for="0" value="Treebank Embedding Vectors for Out-of-domain Dependency Parsing" />
          <attvalue for="1" value="  A recent advance in monolingual dependency parsing is the idea of a treebank&#10;embedding vector, which allows all treebanks for a particular language to be&#10;used as training data while at the same time allowing the model to prefer&#10;training data from one treebank over others and to select the preferred&#10;treebank at test time. We build on this idea by 1) introducing a method to&#10;predict a treebank vector for sentences that do not come from a treebank used&#10;in training, and 2) exploring what happens when we move away from predefined&#10;treebank embedding vectors during test time and instead devise tailored&#10;interpolations. We show that 1) there are interpolated vectors that are&#10;superior to the predefined ones, and 2) treebank vectors can be predicted with&#10;sufficient accuracy, for nine out of ten test languages, to match the&#10;performance of an oracle approach that knows the most suitable predefined&#10;treebank embedding for the test set.&#10;" />
          <attvalue for="2" value="&#10;&#10;The Universal Dependencies project&#10;\cite{nivre2016universal} has made available multiple treebanks for the same language annotated according to the same scheme, leading to a new wave of research which explores ways to use multiple treebanks in monolingual parsing&#10;\cite{K17:3003,K17:3007,K17:3005,P18:2098}.&#10;&#10;\newcite{P18:2098} introduced a treebank embedding. A single model is trained on the concatenation of the available treebanks for a language, and the input vector for each training token includes the treebank embedding which encodes the treebank the token comes from. At test time, all input vectors in the test set of the same treebank are also assigned this treebank embedding vector.&#10;\newcite{P18:2098} show that this approach is superior to mono-treebank training and to plain treebank concatenation. Treebank embeddings perform at about the same level as training on multiple treebanks and tuning on one, but they argue that a treebank embedding approach is preferable since it results in just one model per language.&#10;&#10;What happens, however, when&#10;the input sentence does not come from a treebank? &#10;\newcite{P18:2098} simulate this scenario with the Parallel Universal Dependency (PUD) test sets. &#10;They define the notion of a proxy treebank which is the treebank to be used for a treebank embedding when parsing sentences that do not come from any of the training treebanks. They empirically determine the best proxy treebank for each PUD test set by testing with each treebank embedding. &#10;However, the question remains what to do with sentences for which no gold parse is available, and for which we do not know the best proxy.&#10;&#10;We investigate the problem of choosing treebank embedding vectors for new, possibly out-of-domain, sentences. In doing so, we explore the usefulness of interpolated treebank vectors which are computed via a weighted combination of the predefined fixed ones. In experiments with Czech, English and French, &#10;we establish that useful interpolated treebank vectors exist. We then develop a simple k-NN method based on sentence similarity &#10;to choose a treebank vector, either fixed or interpolated, for sentences or entire test sets, which, for 9 of our 10 test languages matches the performance of the best (oracle) proxy treebank. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Computational Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing, Language Modeling, Dependency Parsing, Treebank Embeddings" />
        </attvalues>
      </node>
      <node id="1805.05089" label="1805.05089">
        <attvalues>
          <attvalue for="0" value="Parser Training with Heterogeneous Treebanks" />
          <attvalue for="1" value="  How to make the most of multiple heterogeneous treebanks when training a&#10;monolingual dependency parser is an open question. We start by investigating&#10;previously suggested, but little evaluated, strategies for exploiting multiple&#10;treebanks based on concatenating training sets, with or without fine-tuning. We&#10;go on to propose a new method based on treebank embeddings. We perform&#10;experiments for several languages and show that in many cases fine-tuning and&#10;treebank embeddings lead to substantial improvements over single treebanks or&#10;concatenation, with average gains of 2.0--3.5 LAS points. We argue that&#10;treebank embeddings should be preferred due to their conceptual simplicity,&#10;flexibility and extensibility.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1305.0543" label="1305.0543">
        <attvalues>
          <attvalue for="0" value="Burstiness and spreading on temporal networks" />
          <attvalue for="1" value="  We discuss how spreading processes on temporal networks are impacted by the&#10;shape of their inter-event time distributions. Through simple mathematical&#10;arguments and toy examples, we find that the key factor is the ordering in&#10;which events take place, a property that tends to be affected by the bulk of&#10;the distributions and not only by their tail, as usually considered in the&#10;literature. We show that a detailed modeling of the temporal patterns observed&#10;in complex networks can change dramatically the properties of a spreading&#10;process, such as the ergodicity of a random walk process or the persistence of&#10;an epidemic.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.03477" label="1902.03477">
        <attvalues>
          <attvalue for="0" value="The Omniglot challenge: a 3-year progress report" />
          <attvalue for="1" value="  Three years ago, we released the Omniglot dataset for one-shot learning,&#10;along with five challenge tasks and a computational model that addresses these&#10;tasks. The model was not meant to be the final word on Omniglot; we hoped that&#10;the community would build on our work and develop new approaches. In the time&#10;since, we have been pleased to see wide adoption of the dataset. There has been&#10;notable progress on one-shot classification, but researchers have adopted new&#10;splits and procedures that make the task easier. There has been less progress&#10;on the other four tasks. We conclude that recent approaches are still far from&#10;human-like concept learning on Omniglot, a challenge that requires performing&#10;many tasks with a single model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14171" label="2305.14171">
        <attvalues>
          <attvalue for="0" value="In-Context Probing: Toward Building Robust Classifiers via Probing Large&#10;  Language Models" />
          <attvalue for="1" value="  Large language models are able to learn new tasks in context, where they are&#10;provided with instructions and a few annotated examples. However, the&#10;effectiveness of in-context learning is dependent on the provided context, and&#10;the performance on a downstream task can vary considerably, depending on the&#10;instruction. Importantly, such dependency on the context can surface in&#10;unpredictable ways, e.g., a seemingly more informative instruction might lead&#10;to a worse performance. In this paper, we propose an alternative approach,&#10;which we term In-Context Probing (ICP). Similar to in-context learning, we&#10;contextualize the representation of the input with an instruction, but instead&#10;of decoding the output prediction, we probe the contextualized representation&#10;to predict the label. Through a series of experiments on a diverse set of&#10;classification tasks, we show that in-context probing is significantly more&#10;robust to changes in instructions. We further show that ICP performs&#10;competitive or superior to finetuning and can be particularly helpful to build&#10;classifiers on top of smaller models, with less than a hundred training&#10;examples.&#10;" />
          <attvalue for="2" value="&#10;Language models have become an essential tool in dealing with various tasks in the natural language processing (NLP) domain \cite[][inter alia]{howard-ruder-2018-universal, devlin-etal-2019-bert, NEURIPS2020_1457c0d6}. Notably, such models are extremely sample efficient, and can be employed to solve downstream tasks with a small set of labeled data. The conventional pipeline to build classifiers is to finetune language models to solve the task at hand.&#10;&#10;However, as the language models grow in number of parameters, finetuning them becomes more computationally expensive. Moreover, finetuning changes a model's parameters through gradient updates. Therefore, for each downstream task, a new model must be trained and stored. Fortunately, it has been shown that as models grow in size, they can learn ``in context'' \cite{radford2019language, NEURIPS2020_1457c0d6}. \defn{In-Context Learning} (ICL) refers to prompting a language model with a few demonstrative examples. Such prompting bundles up a small set of input-label pairs into: a set of instructions, a few solved examples, and a single unlabeled example. The model is then asked to predict the label for that example.&#10;&#10;Notably, since ICL does not require any weight updates on the language model, a single model can be used to perform a variety of tasks, as long as they can be specified in natural language. While effective, there are some caveats to ICL that we elaborate next. &#10;&#10;First, zero-shot, or few-shot, performance of pretrained models on downstream tasks depends to a large extent on the way in which the prompt is formulated \cite{pmlr-v139-zhao21c,liu-prompt-survey-2023}. &#10;&#10;Second, ICL may not always benefit from the demonstrative examples the way that it is expected to \cite{webson-pavlick-2022-prompt}, e.g., flipping the labels in demonstrations rarely hurts the performance \cite{min-etal-2022-rethinking}. Furthermore, the number of demonstrative examples that ICL can benefit from is bounded by the language model constraint on the maximum length of input sequences.&#10;&#10;Lastly, decoding the predicted label is neither accurate nor efficient. The decoder might be miscalibrated \cite{pmlr-v139-zhao21c} or suffer from surface form competition \cite{holtzman-etal-2021-surface}. Moreover, autoregressive decoding, with an attention-based decoder takes $\bigo(N^2)$ time for generating a sequence of length $N$, which can be inefficient when decoding long sequences. &#10;&#10;To address these limitations, we investigate an alternative workflow to ICL for building accurate and robust classifiers. We postulate that a primary cause of the aforementioned limitations, such as the sensitivity to the exact phrasing of the instruction and miscalibration, is that the model is forced to verbalize the prediction label. Therefore, we suggest to bypass the decoding step and directly probe the extracted representations from pretrained models. We hypothesize that given reasonable instructions, the information that is needed to reliably perform the downstream task is encoded by the model in contextualized representations of the tokens. Notably, we contextualize the input by providing instructions. We therefore name our approach in-context probing (ICP). We demonstrate our proposed workflow and contrast it with ICL through an example in \ref{fig:example}.&#10;&#10;Through an extensive set of experiments on a diverse set of sentence classification tasks and different model sizes, we aim to answer the following research questions:&#10;&#10;\begin{itemize}&#10;\itemsep0em&#10; \item Q1: Is in-context probing more robust with respect to variations in instructions compared to ICL? (Section \ref{sec:q1})&#10; \item Q2: Can in-context probing perform classification tasks as accurately as ICL or even finetuning? (Sections \ref{sec:q2}, \ref{sec:finetune})&#10; \item Q3: Is in-context probing sample efficient? (Section \ref{sec:sample-eff})&#10;\end{itemize}&#10;&#10;We find that in-context probing is significantly less sensitive to subtle changes in instructions compared to in-context learning. We further compare in-context probing with in-context learning on different sizes of \flant models \cite{chung2022scaling} in \ref{sec:q2}. Our results suggest that for larger models, ICP is on par or better than both ICL and calibrated ICL with less variance to instruction changes. For smaller model sizes, ICP significantly outperforms ICL. Furthermore, we empirically show that ICP is sample efficient, as it can generate competitive results to ICL (with significantly less variance to instructions) after training only on $40$ annotated examples. Finally, comparing in-context probing with finetuning suggests that probing classifiers can be as accurate and robust as finetuned models, while using $4$ to $6$ orders of magnitude less trainable parameters. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Language Model Learning, Robust Classification Methods, Artificial Intelligence, Mathematics, Contextualized Representation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1801.06146" label="1801.06146">
        <attvalues>
          <attvalue for="0" value="Universal Language Model Fine-tuning for Text Classification" />
          <attvalue for="1" value="  Inductive transfer learning has greatly impacted computer vision, but&#10;existing approaches in NLP still require task-specific modifications and&#10;training from scratch. We propose Universal Language Model Fine-tuning&#10;(ULMFiT), an effective transfer learning method that can be applied to any task&#10;in NLP, and introduce techniques that are key for fine-tuning a language model.&#10;Our method significantly outperforms the state-of-the-art on six text&#10;classification tasks, reducing the error by 18-24% on the majority of datasets.&#10;Furthermore, with only 100 labeled examples, it matches the performance of&#10;training from scratch on 100x more data. We open-source our pretrained models&#10;and code.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.09690" label="2102.09690">
        <attvalues>
          <attvalue for="0" value="Calibrate Before Use: Improving Few-Shot Performance of Language Models" />
          <attvalue for="1" value="  GPT-3 can perform numerous tasks when provided a natural language prompt that&#10;contains a few training examples. We show that this type of few-shot learning&#10;can be unstable: the choice of prompt format, training examples, and even the&#10;order of the training examples can cause accuracy to vary from near chance to&#10;near state-of-the-art. We demonstrate that this instability arises from the&#10;bias of language models towards predicting certain answers, e.g., those that&#10;are placed near the end of the prompt or are common in the pre-training data.&#10;To mitigate this, we first estimate the model's bias towards each answer by&#10;asking for its prediction when given the training prompt and a content-free&#10;test input such as &quot;N/A&quot;. We then fit calibration parameters that cause the&#10;prediction for this input to be uniform across answers. On a diverse set of&#10;tasks, this contextual calibration procedure substantially improves GPT-3 and&#10;GPT-2's average accuracy (up to 30.0% absolute) and reduces variance across&#10;different choices of the prompt.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.01247" label="2109.01247">
        <attvalues>
          <attvalue for="0" value="Do Prompt-Based Models Really Understand the Meaning of their Prompts?" />
          <attvalue for="1" value="  Recently, a boom of papers has shown extraordinary progress in zero-shot and&#10;few-shot learning with various prompt-based models. It is commonly argued that&#10;prompts help models to learn faster in the same way that humans learn faster&#10;when provided with task instructions expressed in natural language. In this&#10;study, we experiment with over 30 prompt templates manually written for natural&#10;language inference (NLI). We find that models learn just as fast with many&#10;prompts that are intentionally irrelevant or even pathologically misleading as&#10;they do with instructively &quot;good&quot; prompts. Further, such patterns hold even for&#10;models as large as 175 billion parameters (Brown et al., 2020) as well as the&#10;recently proposed instruction-tuned models which are trained on hundreds of&#10;prompts (Sanh et al., 2022). That is, instruction-tuned models often produce&#10;good predictions with irrelevant and misleading prompts even at zero shots. In&#10;sum, notwithstanding prompt-based models' impressive improvement, we find&#10;evidence of serious limitations that question the degree to which such&#10;improvement is derived from models understanding task instructions in ways&#10;analogous to humans' use of task instructions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.11416" label="2210.11416">
        <attvalues>
          <attvalue for="0" value="Scaling Instruction-Finetuned Language Models" />
          <attvalue for="1" value="  Finetuning language models on a collection of datasets phrased as&#10;instructions has been shown to improve model performance and generalization to&#10;unseen tasks. In this paper we explore instruction finetuning with a particular&#10;focus on (1) scaling the number of tasks, (2) scaling the model size, and (3)&#10;finetuning on chain-of-thought data. We find that instruction finetuning with&#10;the above aspects dramatically improves performance on a variety of model&#10;classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and&#10;evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For&#10;instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM&#10;540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves&#10;state-of-the-art performance on several benchmarks, such as 75.2% on five-shot&#10;MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong&#10;few-shot performance even compared to much larger models, such as PaLM 62B.&#10;Overall, instruction finetuning is a general method for improving the&#10;performance and usability of pretrained language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.16634" label="2303.16634">
        <attvalues>
          <attvalue for="0" value="G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment" />
          <attvalue for="1" value="  The quality of texts generated by natural language generation (NLG) systems&#10;is hard to measure automatically. Conventional reference-based metrics, such as&#10;BLEU and ROUGE, have been shown to have relatively low correlation with human&#10;judgments, especially for tasks that require creativity and diversity. Recent&#10;studies suggest using large language models (LLMs) as reference-free metrics&#10;for NLG evaluation, which have the benefit of being applicable to new tasks&#10;that lack human references. However, these LLM-based evaluators still have&#10;lower human correspondence than medium-size neural evaluators. In this work, we&#10;present G-Eval, a framework of using large language models with&#10;chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of&#10;NLG outputs. We experiment with two generation tasks, text summarization and&#10;dialogue generation. We show that G-Eval with GPT-4 as the backbone model&#10;achieves a Spearman correlation of 0.514 with human on summarization task,&#10;outperforming all previous methods by a large margin. We also propose&#10;preliminary analysis on the behavior of LLM-based evaluators, and highlight the&#10;potential issue of LLM-based evaluators having a bias towards the LLM-generated&#10;texts. The code is at https://github.com/nlpyang/geval&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Ngram-based Metrics&#10;Ngram-based metrics refer to the scores for evaluating the NLG models by measuring the lexical overlap between a generated text and a reference text.&#10;BLEU~\cite{papineni2002bleu} is the most widely used metric for machine translation evaluation, which calculates the geometric mean of modified n-gram precision and a brevity penalty. &#10;ROUGE~\cite{lin2004rouge} is a recall-oriented metric for summarization evaluation, which measures the n-gram overlap between a generated summary and a set of reference summaries. &#10;It has been shown that more than 60\% of recent papers on NLG only rely on ROUGE or BLEU to evaluate their systems~\cite{kasai2021bidimensional}. However, these metrics fail to measure content quality~\cite{reiter2009investigation} or capture syntactic errors~\cite{stent2005evaluating}, and therefore do not reflect the reliability of NLG systems accurately.&#10;&#10;Embedding-based Metrics&#10;Embedding-based metrics refer to the scores for evaluating the NLG models by measuring the semantic similarity between a generated text and a reference text based on the word or sentence embeddings. &#10;WMD~\cite{kusner2015word} is a metric that measures the distance between two texts based on the word embeddings. &#10;BERTScore~\cite{zhang2019bertscore} measures the similarity between two texts based on the contextualized embedding from BERT~\cite{devlin2019bert}. &#10;MoverScore~\cite{zhao2019MoverScore} improves BERTScore by adding soft alignments and new aggregation methods to obtain a more robust similarity measure. &#10;\cite{clark2019sentence} propose a metric that evaluates multi-sentence texts &#10;by computing the similarity between the generated text and the reference text based on the sentence embeddings. &#10;&#10;Task-specific Evaluators&#10;Task-specific metrics refer to the scores for evaluating the NLG models by measuring the quality of the generated texts based on the specific task requirements. For example, summarization tasks need to assess the consistency of the generated summaries~\cite{kryscinski2020evaluating, wang2020asking, cao2020factual, durmus2020feqa}, and dialogue response generation tasks need to assess the coherence of the generated responses~\cite{dziri2019evaluating, ye2021towards}. However, these metrics are not generalizable to other NLG tasks, and they are not able to measure the overall quality of the generated texts. &#10;&#10;Unified Evaluators&#10;Recently, some evaluators have been developed to assess text quality from multiple dimensions by varying the input and output contents~\cite{yuan2021bartscore} or the model variants~\cite{mehri2020usr} they use. UniEval \cite{zhong2022towards} is a unified evaluator that can evaluate different aspects of text generation as QA tasks. By changing the question format, it can handle different evaluation tasks.&#10;&#10;LLM-based Evaluators&#10;\cite{fu2023GPTScore} propose GPTScore, a new framework that evaluated texts with generative pre-training models like GPT-3. It assumes that a generative pre-training&#10;model will assign a higher probability of high-quality generated text following a given instruction and context. &#10;\cite{wang2023chatgpt} conduct a preliminary survey of using ChatGPT as a NLG evaluator.&#10;\cite{kocmi2023large} proposed to use GPT models for evaluating machine translation tasks.&#10;&#10;" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Large Language Models, Linguistics, Cognitive Science, Evaluation Metrics, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2210.06726" label="2210.06726">
        <attvalues>
          <attvalue for="0" value="Explanations from Large Language Models Make Small Reasoners Better" />
          <attvalue for="1" value="  Integrating free-text explanations to in-context learning of large language&#10;models (LLM) is shown to elicit strong reasoning capabilities along with&#10;reasonable explanations. In this paper, we consider the problem of leveraging&#10;the explanations generated by LLM to improve the training of small reasoners,&#10;which are more favorable in real-production deployment due to their low cost.&#10;We systematically explore three explanation generation approaches from LLM and&#10;utilize a multi-task learning framework to facilitate small models to acquire&#10;strong reasoning power together with explanation generation capabilities.&#10;Experiments on multiple reasoning tasks show that our method can consistently&#10;and significantly outperform finetuning baselines across different settings,&#10;and even perform better than finetuning/prompting a 60x larger GPT-3 (175B)&#10;model by up to 9.5% in accuracy. As a side benefit, human evaluation further&#10;shows that our method can generate high-quality explanations to justify its&#10;predictions, moving towards the goal of explainable AI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Prompting with Explanations. Recently, a new learning paradigm, in-context learning where several training examples are used as demonstrations of LLM without any parameter update, has shown promising results in various NLP tasks \cite{Brown2020LanguageMA}. Although promising, LLM still struggle with tasks requiring strong reasoning capability \cite{Wei2022ChainOT}. To enable better few-shot in-context learning of LLM for reasoning tasks, \cite{Wei2022ChainOT} proposes chain of thought prompting, which provides intermediate reasoning steps as explanations in prompts before answers and has achieved state-of-the-art in arithmetic, symbolic and common sense reasoning tasks. \cite{Zhou2022LeasttoMostPE} further extends chain of thought prompting with least-to-most&#10;prompting, which decomposes a complex problem into a list of subproblems with natural languages, and then sequentially solves these subproblems in a recursive fashion. \cite{Kojima2022LargeLM} moves one step further and shows that LLM are zero-shot reasoners by simply adding “Let’s think step by step” without any demonstration in prompts. Unlike these work, \cite{Lampinen2022CanLM} explores explanations after answers prompting for LLM, where answers are fed into LLM before providing their explanations in prompts, and also observes consistent gains. &#10;&#10;These also exist work to utilize explanations generated from LLM rather than focusing on their final predictions. \cite{Wiegreffe2021ReframingHC} explores utilizing LLM to annotate explanations for existing datasets and proposes a sample-then-filter paradigm with human annotations. \cite{Ye2022TheUO} proposes to utilize a calibrator to calibrate GPT-3 as they find that GPT-3 tends to generate consistent but less factual explanations for textual&#10;reasoning tasks. However, none of these work explores if these noisy explanations generated from LLM without human-involved filtering can be used to improve SLM reasoning capability. The closest work to ours is STaR \cite{Zelikman2022STaRBR}. STaR begins with prompting a descent large language model GPT-J with 6B parameters \cite{mesh-transformer-jax} possibly including answer hints via chain of thought prompting to generate explanations with incorrect answer rejection. After that, they utilize filtered training datasets with explanations to finetune GPT-J as a teacher model and then utilize the teacher model to generate explanations of training datasets to train a student GPT-J model iteratively with a self-training fashion until performance plateaus. However, STaR often requires dozens of iterations to converge, which is both time-consuming and compute-intensive to train a large 6B model. What's worse, their method may not be applicable to smaller language models, e.g. GPT-2 \cite{radford2019gpt2} and strong non-autoregressive models, e.g. T5, as they may not generate high-quality explanations with prompting. In addition, they only focus on chain of thought style prompting and finetuning while our approach can improve SLM across model sizes, explanation generation and multi-task finetuning methods.&#10;&#10;Learning with Explanations. Learning with explanations has been commonly studied in robotics \cite{johnson1994agents} and computer vision \cite{Hendricks2016GeneratingVE}. Recently, it has received increasing attention in NLP as well. \cite{Camburu2018eSNLINL} proposes MT-Ra for natural language inference task with LSTM and does not observe gains over single-task finetuning. \cite{Narang2020WT5TT} utilizes MT-Ra setup on both T5-base and T5-11B models but mainly focuses on explanation generation. Instead, \cite{rajani-etal-2019-explain} observes improvements with two-stage finetuning using human-annotated explanations for common sense reasoning task, where the first stage is to train a model for explanation generations with GPT \cite{Radford2018GPT} and the second one utilizes explanations as input to train a classification model based on BERT \cite{devlin2019bert}. However, \cite{HaseZXB20} finds that both two-stage finetuning and multi-task learning with MT-Re and MT-Ra setups only obtain comparable results over standard finetuning baselines on T5. We instead show that MT-Re, MT-Ra and our proposed MT-CoT with explanations from LLM can consistently and significantly outperform standard finetuning baselines without accuracy-explanation trade-off \cite{jain-etal-2020-learning}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2006.05525" label="2006.05525">
        <attvalues>
          <attvalue for="0" value="Knowledge Distillation: A Survey" />
          <attvalue for="1" value="  In recent years, deep neural networks have been successful in both industry&#10;and academia, especially for computer vision tasks. The great success of deep&#10;learning is mainly due to its scalability to encode large-scale data and to&#10;maneuver billions of model parameters. However, it is a challenge to deploy&#10;these cumbersome deep models on devices with limited resources, e.g., mobile&#10;phones and embedded devices, not only because of the high computational&#10;complexity but also the large storage requirements. To this end, a variety of&#10;model compression and acceleration techniques have been developed. As a&#10;representative type of model compression and acceleration, knowledge&#10;distillation effectively learns a small student model from a large teacher&#10;model. It has received rapid increasing attention from the community. This&#10;paper provides a comprehensive survey of knowledge distillation from the&#10;perspectives of knowledge categories, training schemes, teacher-student&#10;architecture, distillation algorithms, performance comparison and applications.&#10;Furthermore, challenges in knowledge distillation are briefly reviewed and&#10;comments on future research are discussed and forwarded.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1412.6550" label="1412.6550">
        <attvalues>
          <attvalue for="0" value="FitNets: Hints for Thin Deep Nets" />
          <attvalue for="1" value="  While depth tends to improve network performances, it also makes&#10;gradient-based training more difficult since deeper networks tend to be more&#10;non-linear. The recently proposed knowledge distillation approach is aimed at&#10;obtaining small and fast-to-execute models, and it has shown that a student&#10;network could imitate the soft output of a larger teacher network or ensemble&#10;of networks. In this paper, we extend this idea to allow the training of a&#10;student that is deeper and thinner than the teacher, using not only the outputs&#10;but also the intermediate representations learned by the teacher as hints to&#10;improve the training process and final performance of the student. Because the&#10;student intermediate hidden layer will generally be smaller than the teacher's&#10;intermediate hidden layer, additional parameters are introduced to map the&#10;student hidden layer to the prediction of the teacher hidden layer. This allows&#10;one to train deeper students that can generalize better or run faster, a&#10;trade-off that is controlled by the chosen student capacity. For example, on&#10;CIFAR-10, a deep student network with almost 10.4 times less parameters&#10;outperforms a larger, state-of-the-art teacher network.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.02840" label="2111.02840">
        <attvalues>
          <attvalue for="0" value="Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of&#10;  Language Models" />
          <attvalue for="1" value="  Large-scale pre-trained language models have achieved tremendous success&#10;across a wide range of natural language understanding (NLU) tasks, even&#10;surpassing human performance. However, recent studies reveal that the&#10;robustness of these models can be challenged by carefully crafted textual&#10;adversarial examples. While several individual datasets have been proposed to&#10;evaluate model robustness, a principled and comprehensive benchmark is still&#10;missing. In this paper, we present Adversarial GLUE (AdvGLUE), a new multi-task&#10;benchmark to quantitatively and thoroughly explore and evaluate the&#10;vulnerabilities of modern large-scale language models under various types of&#10;adversarial attacks. In particular, we systematically apply 14 textual&#10;adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further&#10;validated by humans for reliable annotations. Our findings are summarized as&#10;follows. (i) Most existing adversarial attack algorithms are prone to&#10;generating invalid or ambiguous adversarial examples, with around 90% of them&#10;either changing the original semantic meanings or misleading human annotators&#10;as well. Therefore, we perform a careful filtering process to curate a&#10;high-quality benchmark. (ii) All the language models and robust training&#10;methods we tested perform poorly on AdvGLUE, with scores lagging far behind the&#10;benign accuracy. We hope our work will motivate the development of new&#10;adversarial attacks that are more stealthy and semantic-preserving, as well as&#10;new robust language models against sophisticated adversarial attacks. AdvGLUE&#10;is available at https://adversarialglue.github.io.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.18223" label="2303.18223">
        <attvalues>
          <attvalue for="0" value="A Survey of Large Language Models" />
          <attvalue for="1" value="  Language is essentially a complex, intricate system of human expressions&#10;governed by grammatical rules. It poses a significant challenge to develop&#10;capable AI algorithms for comprehending and grasping a language. As a major&#10;approach, language modeling has been widely studied for language understanding&#10;and generation in the past two decades, evolving from statistical language&#10;models to neural language models. Recently, pre-trained language models (PLMs)&#10;have been proposed by pre-training Transformer models over large-scale corpora,&#10;showing strong capabilities in solving various NLP tasks. Since researchers&#10;have found that model scaling can lead to performance improvement, they further&#10;study the scaling effect by increasing the model size to an even larger size.&#10;Interestingly, when the parameter scale exceeds a certain level, these enlarged&#10;language models not only achieve a significant performance improvement but also&#10;show some special abilities that are not present in small-scale language&#10;models. To discriminate the difference in parameter scale, the research&#10;community has coined the term large language models (LLM) for the PLMs of&#10;significant size. Recently, the research on LLMs has been largely advanced by&#10;both academia and industry, and a remarkable progress is the launch of ChatGPT,&#10;which has attracted widespread attention from society. The technical evolution&#10;of LLMs has been making an important impact on the entire AI community, which&#10;would revolutionize the way how we develop and use AI algorithms. In this&#10;survey, we review the recent advances of LLMs by introducing the background,&#10;key findings, and mainstream techniques. In particular, we focus on four major&#10;aspects of LLMs, namely pre-training, adaptation tuning, utilization, and&#10;capacity evaluation. Besides, we also summarize the available resources for&#10;developing LLMs and discuss the remaining issues for future directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.01116" label="2306.01116">
        <attvalues>
          <attvalue for="0" value="The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora&#10;  with Web Data, and Web Data Only" />
          <attvalue for="1" value="  Large language models are commonly trained on a mixture of filtered web data&#10;and curated high-quality corpora, such as social media conversations, books, or&#10;technical papers. This curation process is believed to be necessary to produce&#10;performant models with broad zero-shot generalization abilities. However, as&#10;larger models requiring pretraining on trillions of tokens are considered, it&#10;is unclear how scalable is curation and whether we will run out of unique&#10;high-quality data soon. At variance with previous beliefs, we show that&#10;properly filtered and deduplicated web data alone can lead to powerful models;&#10;even significantly outperforming models from the state-of-the-art trained on&#10;The Pile. Despite extensive filtering, the high-quality data we extract from&#10;the web is still plentiful, and we are able to obtain five trillion tokens from&#10;CommonCrawl. We publicly release an extract of 600 billion tokens from our&#10;RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.10597" label="1907.10597">
        <attvalues>
          <attvalue for="0" value="Green AI" />
          <attvalue for="1" value="  The computations required for deep learning research have been doubling every&#10;few months, resulting in an estimated 300,000x increase from 2012 to 2018 [2].&#10;These computations have a surprisingly large carbon footprint [38]. Ironically,&#10;deep learning was inspired by the human brain, which is remarkably energy&#10;efficient. Moreover, the financial cost of the computations can make it&#10;difficult for academics, students, and researchers, in particular those from&#10;emerging economies, to engage in deep learning research.&#10;  This position paper advocates a practical solution by making efficiency an&#10;evaluation criterion for research alongside accuracy and related measures. In&#10;addition, we propose reporting the financial cost or &quot;price tag&quot; of developing,&#10;training, and running models to provide baselines for the investigation of&#10;increasingly efficient methods. Our goal is to make AI both greener and more&#10;inclusive---enabling any inspired undergraduate with a laptop to write&#10;high-quality research papers. Green AI is an emerging focus at the Allen&#10;Institute for AI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.03300" label="2009.03300">
        <attvalues>
          <attvalue for="0" value="Measuring Massive Multitask Language Understanding" />
          <attvalue for="1" value="  We propose a new test to measure a text model's multitask accuracy. The test&#10;covers 57 tasks including elementary mathematics, US history, computer science,&#10;law, and more. To attain high accuracy on this test, models must possess&#10;extensive world knowledge and problem solving ability. We find that while most&#10;recent models have near random-chance accuracy, the very largest GPT-3 model&#10;improves over random chance by almost 20 percentage points on average. However,&#10;on every one of the 57 tasks, the best models still need substantial&#10;improvements before they can reach expert-level accuracy. Models also have&#10;lopsided performance and frequently do not know when they are wrong. Worse,&#10;they still have near-random accuracy on some socially important subjects such&#10;as morality and law. By comprehensively evaluating the breadth and depth of a&#10;model's academic and professional understanding, our test can be used to&#10;analyze models across many tasks and to identify important shortcomings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.08322" label="2305.08322">
        <attvalues>
          <attvalue for="0" value="C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for&#10;  Foundation Models" />
          <attvalue for="1" value="  New NLP benchmarks are urgently needed to align with the rapid development of&#10;large language models (LLMs). We present C-Eval, the first comprehensive&#10;Chinese evaluation suite designed to assess advanced knowledge and reasoning&#10;abilities of foundation models in a Chinese context. C-Eval comprises&#10;multiple-choice questions across four difficulty levels: middle school, high&#10;school, college, and professional. The questions span 52 diverse disciplines,&#10;ranging from humanities to science and engineering. C-Eval is accompanied by&#10;C-Eval Hard, a subset of very challenging subjects in C-Eval that requires&#10;advanced reasoning abilities to solve. We conduct a comprehensive evaluation of&#10;the most advanced LLMs on C-Eval, including both English- and Chinese-oriented&#10;models. Results indicate that only GPT-4 could achieve an average accuracy of&#10;over 60%, suggesting that there is still significant room for improvement for&#10;current LLMs. We anticipate C-Eval will help analyze important strengths and&#10;shortcomings of foundation models, and foster their development and growth for&#10;Chinese users.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;English benchmarks:&#10;Traditional English benchmarks mainly focus on assessing certain abilities of models on a single task or a single type of tasks, such as &#10;natural language understanding (NLU,~\cite{wangglue}), reading comprehension~\cite{rajpurkar2018know}, machine translation~\cite{bojar-etal-2014-findings}, and summarization~\cite{hermann2015teaching,narayan-etal-2018-dont}.&#10;As a representative example, the GLUE benchmark~\cite{wangglue} combines a collection of NLU tasks, and has witnessed superhuman model performance due to the burst of pretraining models such as BERT~\cite{kenton2019bert} and GPT~\cite{radford2019language}. &#10;In order to assess the capabilities of LLMs more comprehensively, recent benchmarks have cast light on the broader knowledge and advanced abilities. &#10;The MMLU benchmark~\cite{hendrycks2021measuring} provides multi-domain and multi-task evaluation collected from real-world examinations and books. LLMs' performance on MMLU fluctuates around random-chance accuracy until they reach the scale of GPT-3.&#10;The BIG-bench benchmark~\cite{srivastava2022beyond} consists of 204 diverse tasks, some of which are considered to be beyond the capabilities of current LLMs.&#10;The HELM benchmark~\cite{liang2022holistic} aggregates 42 different tasks and evaluates LLMs with 7 metrics ranging from accuracy to robustness.&#10;&#10;Chinese benchmarks:&#10;Despite the flourishing of English benchmark, language abilities in Chinese language environment remain under-developed. &#10;The CLUE benchmark~\cite{xu-etal-2020-clue} is the first large-scale Chinese NLU benchmark,&#10;and still serves as the most widely-used and best available Chinese benchmark. &#10;Recently, the AGIEval benchmark~\cite{zhong2023agieval} contains data from the Chinese College Entrance Exam, Chinese lawyer qualification test and Chinese civil service examination. &#10;The MMCU benchmark~\cite{zeng2023measuring} consists of tests from four major domains including medicine, law, psychology and education, which are also collected from Chinese College Entrance Exam, qualification test as well as university examinations.&#10;Compared to AGIEval and MMCU, C-Eval~ (1) has a broader coverage of domains (\textsection\ref{sec:collect}), (2) features four different levels of difficulty -- particularly, the C-Eval Hard~benchmark is the first Chinese benchmark to provide sophisticated reasoning problems, and (3) makes an effort to mitigate data leakage -- our questions mostly come from mock exams as PDF or Microsoft Word documents that are further processed by us, while AGIEval and MMCU collects the exact questions from past national exams in China. &#10; " />
        </attvalues>
      </node>
      <node id="2409.08596" label="2409.08596">
        <attvalues>
          <attvalue for="0" value="Large Language Model Can Transcribe Speech in Multi-Talker Scenarios&#10;  with Versatile Instructions" />
          <attvalue for="1" value="  Recent advancements in large language models (LLMs) have revolutionized&#10;various domains, bringing significant progress and new opportunities. Despite&#10;progress in speech-related tasks, LLMs have not been sufficiently explored in&#10;multi-talker scenarios. In this work, we present a pioneering effort to&#10;investigate the capability of LLMs in transcribing speech in multi-talker&#10;environments, following versatile instructions related to multi-talker&#10;automatic speech recognition (ASR), target talker ASR, and ASR based on&#10;specific talker attributes such as sex, occurrence order, language, and keyword&#10;spoken. Our approach utilizes WavLM and Whisper encoder to extract&#10;multi-faceted speech representations that are sensitive to speaker&#10;characteristics and semantic context. These representations are then fed into&#10;an LLM fine-tuned using LoRA, enabling the capabilities for speech&#10;comprehension and transcription. Comprehensive experiments reveal the promising&#10;performance of our proposed system, MT-LLM, in cocktail party scenarios,&#10;highlighting the potential of LLM to handle speech-related tasks based on user&#10;instructions in such complex settings.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have experienced rapid and significant advancements recently, achieving or even surpassing human-level proficiency in numerous natural language processing (NLP) tasks \cite{OpenAI2023GPT4TR, touvron2023llama,touvron2023llama2}. These advancements have sparked interest in exploring the capabilities of LLM in multi-modal perception, including speech \cite{Qwen-Audio,tang2024salmonn, hu2024wavllm,Qwen2-Audio}, vision \cite{OpenAI2023GPT4TR,huang2023language,peng2024grounding}, and content generation \cite{pan2024kosmosg, brooks2024video}. Several studies have investigated speech-related LLM, which typically involve a fine-tuned text LLM following speech-related instructions and pairing with auxiliary audio encoders \cite{Qwen-Audio,tang2024salmonn, hu2024wavllm,Qwen2-Audio}. The audio encoder extracts acoustic representations and adapts them into the input feature space of LLM, enabling the LLM to perform various speech tasks such as automatic speech recognition (ASR), speech translation (ST), speaker verification (SV), and speech question answering (SQA), among others. &#10;However, despite the progress, the potential of speech LLMs in cocktail party scenarios—where multiple talkers speak simultaneously and overlapping occurs—has not yet been sufficiently exploited. &#10;&#10;In recent year, various end-to-end approaches have garnered interest and been developed to tackle multi-talker ASR task, which involves simultaneously transcribing speech from multiple talkers. These studies are based on Permutation Invariant Training (PIT) \cite{zhang2020pit,chang2020pit,meng23sidecaricassp}, Heuristic Error Assignment Training (HEAT) \cite{lu2021surt,raj2023surt2}, or Serialized Output Training (SOT) &#10; \cite{kanda2020sot,kanda22tsot,shi24sot,li2024sidecarsot} to match predictions with corresponding target labels for loss calculation. However, these approaches typically transcribe speech from all talkers indiscriminately and fail to associate transcriptions with specific talkers, unless an additional external \cite{huang2023pit_tse,masumura23joint} or internal \cite{kanda21sasot,conformer-ts-asr,masumura24_interspeech} model is employed to extract speaker information. Although several studies \cite{meng23sidecarinterspeech,meng24whisperovlp} proposed handling multi-talker ASR in conjunction with other tasks within a single model, the addressed tasks remain constrained and lack the flexibility to address various user requirements specifying talker attributes such as please transcribe the talker who said the word “strawberry”. &#10;&#10;Nevertheless, the rise of large language models illuminates new possibilities for tackling such problems with a unified model. &#10;In this work, we leverage the powerful comprehension and instruction-following capabilities of LLM to perform speech recognition based on various instructions in multi-talker scenarios. Specifically, we utilize Llama 2 \cite{touvron2023llama2} as our foundational LLM, coupled with the Whisper \cite{radford2023whisper} encoder to extract semantic context, and WavLM \cite{chen2022wavlm} multi-layer features to capture acoustic information indicating speaker characteristics, referring to WavLLM \cite{hu2024wavllm} and SALMONN \cite{tang2024salmonn}. Corresponding adapters are designed to project audio embeddings into the LLM's input space. We denote the proposed model as MT-LLM (Multi-Talker LLM). Versatile instructions are used to prompt MT-LLM to perform tasks including (i) simultaneously transcribing the speech of multiple talkers into text, (ii) transcribing a target talker's speech given a reference audio clip, (iii) transcribing speech based on the talker's specific sex, (iv) transcribing the speech of a specified talker according to their occurrence order, (v) transcribing the speech of the talker where a given keyword appears, and (vi) transcribe the talker who speaks the specific language. The comprehensive experiments demonstrate that MT-LLM can effectively meet user's diverse requirements for transcribing multiple talkers based on instructions specifying talker attributes.&#10;Our major contributions are threefold:&#10;\begin{itemize}[leftmargin=1em]&#10; \item We propose a pioneering effort to explore instruction-based speech recognition in multi-talker scenarios, leveraging the powerful comprehension and generation capabilities of LLM;&#10; \item Beyond multi-talker ASR, MT-LLM can transcribe speech from specific talkers according to six versatile instructions, demonstrating promising performances;&#10; \item We reveal that speech LLMs can support a more natural and effective human-computer interaction paradigm in complex speech environments, with parameter-efficient training.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Speech Recognition Systems, Linguistics, Multi-Talker Environments, Artificial Intelligence, Signal Processing" />
        </attvalues>
      </node>
      <node id="2311.07919" label="2311.07919">
        <attvalues>
          <attvalue for="0" value="Qwen-Audio: Advancing Universal Audio Understanding via Unified&#10;  Large-Scale Audio-Language Models" />
          <attvalue for="1" value="  Recently, instruction-following audio-language models have received broad&#10;attention for audio interaction with humans. However, the absence of&#10;pre-trained audio models capable of handling diverse audio types and tasks has&#10;hindered progress in this field. Consequently, most existing works have only&#10;been able to support a limited range of interaction capabilities. In this&#10;paper, we develop the Qwen-Audio model and address this limitation by scaling&#10;up audio-language pre-training to cover over 30 tasks and various audio types,&#10;such as human speech, natural sounds, music, and songs, to facilitate universal&#10;audio understanding abilities. However, directly co-training all tasks and&#10;datasets can lead to interference issues, as the textual labels associated with&#10;different datasets exhibit considerable variations due to differences in task&#10;focus, language, granularity of annotation, and text structure. To overcome the&#10;one-to-many interference, we carefully design a multi-task training framework&#10;by conditioning on a sequence of hierarchical tags to the decoder for&#10;encouraging knowledge sharing and avoiding interference through shared and&#10;specified tags respectively. Remarkably, Qwen-Audio achieves impressive&#10;performance across diverse benchmark tasks without requiring any task-specific&#10;fine-tuning, surpassing its counterparts. Building upon the capabilities of&#10;Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from&#10;various audios and text inputs, enabling multi-turn dialogues and supporting&#10;various audio-central scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.13289" label="2310.13289">
        <attvalues>
          <attvalue for="0" value="SALMONN: Towards Generic Hearing Abilities for Large Language Models" />
          <attvalue for="1" value="  Hearing is arguably an essential ability of artificial intelligence (AI)&#10;agents in the physical world, which refers to the perception and understanding&#10;of general auditory information consisting of at least three types of sounds:&#10;speech, audio events, and music. In this paper, we propose SALMONN, a speech&#10;audio language music open neural network, built by integrating a pre-trained&#10;text-based large language model (LLM) with speech and audio encoders into a&#10;single multimodal model. SALMONN enables the LLM to directly process and&#10;understand general audio inputs and achieve competitive performances on a&#10;number of speech and audio tasks used in training, such as automatic speech&#10;recognition and translation, auditory-information-based question answering,&#10;emotion recognition, speaker verification, and music and audio captioning etc.&#10;SALMONN also has a diverse set of emergent abilities unseen in the training,&#10;which includes but is not limited to speech translation to untrained languages,&#10;speech-based slot filling, spoken-query-based question answering, audio-based&#10;storytelling, and speech audio co-reasoning etc. The presence of cross-modal&#10;emergent abilities is studied, and a novel few-shot activation tuning approach&#10;is proposed to activate such abilities. To our knowledge, SALMONN is the first&#10;model of its type and can be regarded as a step towards AI with generic hearing&#10;abilities. The source code, model checkpoints and data are available at&#10;https://github.com/bytedance/SALMONN.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.10759" label="2407.10759">
        <attvalues>
          <attvalue for="0" value="Qwen2-Audio Technical Report" />
          <attvalue for="1" value="  We introduce the latest progress of Qwen-Audio, a large-scale audio-language&#10;model called Qwen2-Audio, which is capable of accepting various audio signal&#10;inputs and performing audio analysis or direct textual responses with regard to&#10;speech instructions. In contrast to complex hierarchical tags, we have&#10;simplified the pre-training process by utilizing natural language prompts for&#10;different data and tasks, and have further expanded the data volume. We have&#10;boosted the instruction-following capability of Qwen2-Audio and implemented two&#10;distinct audio interaction modes for voice chat and audio analysis. In the&#10;voice chat mode, users can freely engage in voice interactions with Qwen2-Audio&#10;without text input. In the audio analysis mode, users could provide audio and&#10;text instructions for analysis during the interaction. Note that we do not use&#10;any system prompts to switch between voice chat and audio analysis modes.&#10;Qwen2-Audio is capable of intelligently comprehending the content within audio&#10;and following voice commands to respond appropriately. For instance, in an&#10;audio segment that simultaneously contains sounds, multi-speaker conversations,&#10;and a voice command, Qwen2-Audio can directly understand the command and&#10;provide an interpretation and response to the audio. Additionally, DPO has&#10;optimized the model's performance in terms of factuality and adherence to&#10;desired behavior. According to the evaluation results from AIR-Bench,&#10;Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests&#10;focused on audio-centric instruction-following capabilities. Qwen2-Audio is&#10;open-sourced with the aim of fostering the advancement of the multi-modal&#10;language community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.14045" label="2302.14045">
        <attvalues>
          <attvalue for="0" value="Language Is Not All You Need: Aligning Perception with Language Models" />
          <attvalue for="1" value="  A big convergence of language, multimodal perception, action, and world&#10;modeling is a key step toward artificial general intelligence. In this work, we&#10;introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive&#10;general modalities, learn in context (i.e., few-shot), and follow instructions&#10;(i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale&#10;multimodal corpora, including arbitrarily interleaved text and images,&#10;image-caption pairs, and text data. We evaluate various settings, including&#10;zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range&#10;of tasks without any gradient updates or finetuning. Experimental results show&#10;that Kosmos-1 achieves impressive performance on (i) language understanding,&#10;generation, and even OCR-free NLP (directly fed with document images), (ii)&#10;perception-language tasks, including multimodal dialogue, image captioning,&#10;visual question answering, and (iii) vision tasks, such as image recognition&#10;with descriptions (specifying classification via text instructions). We also&#10;show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge&#10;from language to multimodal, and from multimodal to language. In addition, we&#10;introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning&#10;capability of MLLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.02992" label="2310.02992">
        <attvalues>
          <attvalue for="0" value="Kosmos-G: Generating Images in Context with Multimodal Large Language&#10;  Models" />
          <attvalue for="1" value="  Recent advancements in subject-driven image generation have made significant&#10;strides. However, current methods still fall short in diverse application&#10;scenarios, as they require test-time tuning and cannot accept interleaved&#10;multi-image and text input. These limitations keep them far from the ultimate&#10;goal of &quot;image as a foreign language in image generation.&quot; This paper presents&#10;Kosmos-G, a model that leverages the advanced multimodal perception&#10;capabilities of Multimodal Large Language Models (MLLMs) to tackle the&#10;aforementioned challenge. Our approach aligns the output space of MLLM with&#10;CLIP using the textual modality as an anchor and performs compositional&#10;instruction tuning on curated data. Kosmos-G demonstrates an impressive&#10;capability of zero-shot subject-driven generation with interleaved multi-image&#10;and text input. Notably, the score distillation instruction tuning requires no&#10;modifications to the image decoder. This allows for a seamless substitution of&#10;CLIP and effortless integration with a myriad of U-Net techniques ranging from&#10;fine-grained controls to personalized image decoder variants. We posit Kosmos-G&#10;as an initial attempt towards the goal of &quot;image as a foreign language in image&#10;generation.&quot; The code can be found at https://aka.ms/Kosmos-G&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.03921" label="2002.03921">
        <attvalues>
          <attvalue for="0" value="End-to-End Multi-speaker Speech Recognition with Transformer" />
          <attvalue for="1" value="  Recently, fully recurrent neural network (RNN) based end-to-end models have&#10;been proven to be effective for multi-speaker speech recognition in both the&#10;single-channel and multi-channel scenarios. In this work, we explore the use of&#10;Transformer models for these tasks by focusing on two aspects. First, we&#10;replace the RNN-based encoder-decoder in the speech recognition model with a&#10;Transformer architecture. Second, in order to use the Transformer in the&#10;masking network of the neural beamformer in the multi-channel case, we modify&#10;the self-attention component to be restricted to a segment rather than the&#10;whole sequence in order to reduce computation. Besides the model architecture&#10;improvements, we also incorporate an external dereverberation preprocessing,&#10;the weighted prediction error (WPE), enabling our model to handle reverberated&#10;signals. Experiments on the spatialized wsj1-2mix corpus show that the&#10;Transformer-based models achieve 40.9% and 25.6% relative WER reduction, down&#10;to 12.1% and 6.4% WER, under the anechoic condition in single-channel and&#10;multi-channel tasks, respectively, while in the reverberant case, our methods&#10;achieve 41.5% and 13.8% relative WER reduction, down to 16.5% and 15.2% WER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.09908" label="2302.09908">
        <attvalues>
          <attvalue for="0" value="A Sidecar Separator Can Convert a Single-Talker Speech Recognition&#10;  System to a Multi-Talker One" />
          <attvalue for="1" value="  Although automatic speech recognition (ASR) can perform well in common&#10;non-overlapping environments, sustaining performance in multi-talker&#10;overlapping speech recognition remains challenging. Recent research revealed&#10;that ASR model's encoder captures different levels of information with&#10;different layers -- the lower layers tend to have more acoustic information,&#10;and the upper layers more linguistic. This inspires us to develop a Sidecar&#10;separator to empower a well-trained ASR model for multi-talker scenarios by&#10;separating the mixed speech embedding between two suitable layers. We&#10;experimented with a wav2vec 2.0-based ASR model with a Sidecar mounted. By&#10;freezing the parameters of the original model and training only the Sidecar&#10;(8.7 M, 8.4% of all parameters), the proposed approach outperforms the previous&#10;state-of-the-art by a large margin for the 2-speaker mixed LibriMix dataset,&#10;reaching a word error rate (WER) of 10.36%; and obtains comparable results&#10;(7.56%) for LibriSpeechMix dataset when limited training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.02109" label="2104.02109">
        <attvalues>
          <attvalue for="0" value="Streaming Multi-talker Speech Recognition with Joint Speaker&#10;  Identification" />
          <attvalue for="1" value="  In multi-talker scenarios such as meetings and conversations, speech&#10;processing systems are usually required to transcribe the audio as well as&#10;identify the speakers for downstream applications. Since overlapped speech is&#10;common in this case, conventional approaches usually address this problem in a&#10;cascaded fashion that involves speech separation, speech recognition and&#10;speaker identification that are trained independently. In this paper, we&#10;propose Streaming Unmixing, Recognition and Identification Transducer (SURIT)&#10;-- a new framework that deals with this problem in an end-to-end streaming&#10;fashion. SURIT employs the recurrent neural network transducer (RNN-T) as the&#10;backbone for both speech recognition and speaker identification. We validate&#10;our idea on the LibrispeechMix dataset -- a multi-talker dataset derived from&#10;Librispeech, and present encouraging results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.12687" label="2003.12687">
        <attvalues>
          <attvalue for="0" value="Serialized Output Training for End-to-End Overlapped Speech Recognition" />
          <attvalue for="1" value="  This paper proposes serialized output training (SOT), a novel framework for&#10;multi-speaker overlapped speech recognition based on an attention-based&#10;encoder-decoder approach. Instead of having multiple output layers as with the&#10;permutation invariant training (PIT), SOT uses a model with only one output&#10;layer that generates the transcriptions of multiple speakers one after another.&#10;The attention and decoder modules take care of producing multiple&#10;transcriptions from overlapped speech. SOT has two advantages over PIT: (1) no&#10;limitation in the maximum number of speakers, and (2) an ability to model the&#10;dependencies among outputs for different speakers. We also propose a simple&#10;trick that allows SOT to be executed in $O(S)$, where $S$ is the number of the&#10;speakers in the training sample, by using the start times of the constituent&#10;source utterances. Experimental results on LibriSpeech corpus show that the SOT&#10;models can transcribe overlapped speech with variable numbers of speakers&#10;significantly better than PIT-based models. We also show that the SOT models&#10;can accurately count the number of speakers in the input audio.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.00842" label="2202.00842">
        <attvalues>
          <attvalue for="0" value="Streaming Multi-Talker ASR with Token-Level Serialized Output Training" />
          <attvalue for="1" value="  This paper proposes a token-level serialized output training (t-SOT), a novel&#10;framework for streaming multi-talker automatic speech recognition (ASR). Unlike&#10;existing streaming multi-talker ASR models using multiple output branches, the&#10;t-SOT model has only a single output branch that generates recognition tokens&#10;(e.g., words, subwords) of multiple speakers in chronological order based on&#10;their emission times. A special token that indicates the change of ``virtual''&#10;output channels is introduced to keep track of the overlapping utterances.&#10;Compared to the prior streaming multi-talker ASR models, the t-SOT model has&#10;the advantages of less inference cost and a simpler model architecture.&#10;Moreover, in our experiments with LibriSpeechMix and LibriCSS datasets, the&#10;t-SOT-based transformer transducer model achieves the state-of-the-art word&#10;error rates by a significant margin to the prior results. For non-overlapping&#10;speech, the t-SOT model is on par with a single-talker ASR model in terms of&#10;both accuracy and computational cost, opening the door for deploying one model&#10;for both single- and multi-talker scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.03966" label="2407.03966">
        <attvalues>
          <attvalue for="0" value="Serialized Output Training by Learned Dominance" />
          <attvalue for="1" value="  Serialized Output Training (SOT) has showcased state-of-the-art performance&#10;in multi-talker speech recognition by sequentially decoding the speech of&#10;individual speakers. To address the challenging label-permutation issue, prior&#10;methods have relied on either the Permutation Invariant Training (PIT) or the&#10;time-based First-In-First-Out (FIFO) rule. This study presents a model-based&#10;serialization strategy that incorporates an auxiliary module into the Attention&#10;Encoder-Decoder architecture, autonomously identifying the crucial factors to&#10;order the output sequence of the speech components in multi-talker speech.&#10;Experiments conducted on the LibriSpeech and LibriMix databases reveal that our&#10;approach significantly outperforms the PIT and FIFO baselines in both 2-mix and&#10;3-mix scenarios. Further analysis shows that the serialization module&#10;identifies dominant speech components in a mixture by factors including&#10;loudness and gender, and orders speech components based on the dominance score.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.00482" label="2211.00482">
        <attvalues>
          <attvalue for="0" value="Adapting self-supervised models to multi-talker speech recognition using&#10;  speaker embeddings" />
          <attvalue for="1" value="  Self-supervised learning (SSL) methods which learn representations of data&#10;without explicit supervision have gained popularity in speech-processing tasks,&#10;particularly for single-talker applications. However, these models often have&#10;degraded performance for multi-talker scenarios -- possibly due to the domain&#10;mismatch -- which severely limits their use for such applications. In this&#10;paper, we investigate the adaptation of upstream SSL models to the multi-talker&#10;automatic speech recognition (ASR) task under two conditions. First, when&#10;segmented utterances are given, we show that adding a target speaker extraction&#10;(TSE) module based on enrollment embeddings is complementary to mixture-aware&#10;pre-training. Second, for unsegmented mixtures, we propose a novel joint&#10;speaker modeling (JSM) approach, which aggregates information from all speakers&#10;in the mixture through their embeddings. With controlled experiments on&#10;Libri2Mix, we show that using speaker embeddings provides relative WER&#10;improvements of 9.1% and 42.1% over strong baselines for the segmented and&#10;unsegmented cases, respectively. We also demonstrate the effectiveness of our&#10;models for real conversational mixtures through experiments on the AMI dataset.&#10;" />
          <attvalue for="2" value="&#10;Self-supervised learning (SSL), which obtains supervisory signals from the input data itself, is an active sub-area of unsupervised learning~\cite{mohamed2022self,liu2021self}. Generally speaking, the SSL pipeline consists of a pre-training stage, in which an upstream model is trained on a large quantity of unlabeled data with some pretext tasks, and a fine-tuning stage, in which the upstream model is adapted for specific downstream tasks with annotated data.&#10;&#10;SSL has recently gained popularity in the speech community due to its strong performance and low adaptation cost. By simply replacing the acoustic features with hidden representations from pre-trained upstream models~\cite{yang2021superb,chen2022wavlm}, or fine-tuning the entire model with additional task-specific layers~\cite{baevski2020wav2vec,hsu2021hubert}, SSL models have surpassed state-of-the-art downstream-only baselines for several tasks including speech recognition~\cite{baevski2020wav2vec,hsu2021hubert} and speaker recognition~\cite{chen2022large,fan2020exploring}. Furthermore, these pre-trained upstream models often require very small amounts of adaptation data; for instance, wav2vec 2.0 achieves a word error rate (WER) of 8.2\% on LibriSpeech~\cite{Panayotov2015LibrispeechAA} test-other after fine-tuning with 10 minutes of labeled speech~\cite{baevski2020wav2vec}.&#10;&#10;Despite their strong performance for various downstream tasks, most of the existing SSL models only focus on single-talker scenarios. The previous studies on pre-training and fine-tuning upstream SSL models for multi-talker scenarios are very limited~\cite{chen2022wavlm,chang2022end}. Such scenarios are usually more challenging due to the involvement of multiple speakers, quick turn-taking, and overlapping speech~\cite{ccetin2006analysis}.&#10;&#10;In this paper, we focus on the fine-tuning stage of the SSL pipeline and investigate the adaptation of SSL-based upstream models to the multi-talker ASR task, for both segmented utterances and unsegmented mixtures. For the former, we build upon previous work on target speaker extraction and recognition~\cite{delcroix2018single,vzmolikova2019speakerbeam,medennikov2020target,wang2018voicefilter}; we use target speaker embeddings (extracted from enrollment audio) to direct the model to transcribe a specific speaker in the segmented utterance containing mixed speech. We perform a thorough analysis for several modeling aspects, including embedding type and dimensionality, fusion methods for the adaptation layer, and the effect of upstream models. For the case of unsegmented mixtures, we extend the TSE approach by iteratively decoding each speaker in the mixtures, and we also propose a novel joint speaker modeling (JSM) technique that aggregates information from all speakers through their embeddings, and transcribes them simultaneously.&#10;&#10;Our contributions are summarized as follows: (1) we propose methods to adapt SSL models to the multi-talker ASR task based on the speaker embeddings; (2) we conduct a detailed analysis of different modeling aspects of target speaker extraction with SSL models; and (3) we evaluate our methods on both artificial and real datasets, namely LibriMix and AMI. Our code will be released at the time of publication.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.02273" label="2306.02273">
        <attvalues>
          <attvalue for="0" value="End-to-End Joint Target and Non-Target Speakers ASR" />
          <attvalue for="1" value="  This paper proposes a novel automatic speech recognition (ASR) system that&#10;can transcribe individual speaker's speech while identifying whether they are&#10;target or non-target speakers from multi-talker overlapped speech.&#10;Target-speaker ASR systems are a promising way to only transcribe a target&#10;speaker's speech by enrolling the target speaker's information. However, in&#10;conversational ASR applications, transcribing both the target speaker's speech&#10;and non-target speakers' ones is often required to understand interactive&#10;information. To naturally consider both target and non-target speakers in a&#10;single ASR model, our idea is to extend autoregressive modeling-based&#10;multi-talker ASR systems to utilize the enrollment speech of the target&#10;speaker. Our proposed ASR is performed by recursively generating both textual&#10;tokens and tokens that represent target or non-target speakers. Our experiments&#10;demonstrate the effectiveness of our proposed method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.02128" label="2104.02128">
        <attvalues>
          <attvalue for="0" value="End-to-End Speaker-Attributed ASR with Transformer" />
          <attvalue for="1" value="  This paper presents our recent effort on end-to-end speaker-attributed&#10;automatic speech recognition, which jointly performs speaker counting, speech&#10;recognition and speaker identification for monaural multi-talker audio.&#10;Firstly, we thoroughly update the model architecture that was previously&#10;designed based on a long short-term memory (LSTM)-based attention encoder&#10;decoder by applying transformer architectures. Secondly, we propose a speaker&#10;deduplication mechanism to reduce speaker identification errors in highly&#10;overlapped regions. Experimental results on the LibriSpeechMix dataset shows&#10;that the transformer-based architecture is especially good at counting the&#10;speakers and that the proposed model reduces the speaker-attributed word error&#10;rate by 47% over the LSTM-based baseline. Furthermore, for the LibriCSS&#10;dataset, which consists of real recordings of overlapped speech, the proposed&#10;model achieves concatenated minimum-permutation word error rates of 11.9% and&#10;16.3% with and without target speaker profiles, respectively, both of which are&#10;the state-of-the-art results for LibriCSS with the monaural setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.05218" label="2308.05218">
        <attvalues>
          <attvalue for="0" value="Conformer-based Target-Speaker Automatic Speech Recognition for&#10;  Single-Channel Audio" />
          <attvalue for="1" value="  We propose CONF-TSASR, a non-autoregressive end-to-end time-frequency domain&#10;architecture for single-channel target-speaker automatic speech recognition&#10;(TS-ASR). The model consists of a TitaNet based speaker embedding module, a&#10;Conformer based masking as well as ASR modules. These modules are jointly&#10;optimized to transcribe a target-speaker, while ignoring speech from other&#10;speakers. For training we use Connectionist Temporal Classification (CTC) loss&#10;and introduce a scale-invariant spectrogram reconstruction loss to encourage&#10;the model better separate the target-speaker's spectrogram from mixture. We&#10;obtain state-of-the-art target-speaker word error rate (TS-WER) on&#10;WSJ0-2mix-extr (4.2%). Further, we report for the first time TS-WER on&#10;WSJ0-3mix-extr (12.4%), LibriSpeech2Mix (4.2%) and LibriSpeech3Mix (7.6%)&#10;datasets, establishing new benchmarks for TS-ASR. The proposed model will be&#10;open-sourced through NVIDIA NeMo toolkit.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16263" label="2305.16263">
        <attvalues>
          <attvalue for="0" value="Unified Modeling of Multi-Talker Overlapped Speech Recognition and&#10;  Diarization with a Sidecar Separator" />
          <attvalue for="1" value="  Multi-talker overlapped speech poses a significant challenge for speech&#10;recognition and diarization. Recent research indicated that these two tasks are&#10;inter-dependent and complementary, motivating us to explore a unified modeling&#10;method to address them in the context of overlapped speech. A recent study&#10;proposed a cost-effective method to convert a single-talker automatic speech&#10;recognition (ASR) system into a multi-talker one, by inserting a Sidecar&#10;separator into the frozen well-trained ASR model. Extending on this, we&#10;incorporate a diarization branch into the Sidecar, allowing for unified&#10;modeling of both ASR and diarization with a negligible overhead of only 768&#10;parameters. The proposed method yields better ASR results compared to the&#10;baseline on LibriMix and LibriSpeechMix datasets. Moreover, without&#10;sophisticated customization on the diarization task, our method achieves&#10;acceptable diarization results on the two-speaker subset of CALLHOME with only&#10;a few adaptation steps.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.09817" label="2407.09817">
        <attvalues>
          <attvalue for="0" value="Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech&#10;  Recognition System" />
          <attvalue for="1" value="  Multi-talker speech recognition and target-talker speech recognition, both&#10;involve transcription in multi-talker contexts, remain significant challenges.&#10;However, existing methods rarely attempt to simultaneously address both tasks.&#10;In this study, we propose a pioneering approach to empower Whisper, which is a&#10;speech foundation model, to tackle joint multi-talker and target-talker speech&#10;recognition tasks. Specifically, (i) we freeze Whisper and plug a Sidecar&#10;separator into its encoder to separate mixed embedding for multiple talkers;&#10;(ii) a Target Talker Identifier is introduced to identify the embedding flow of&#10;the target talker on the fly, requiring only three-second enrollment speech as&#10;a cue; (iii) soft prompt tuning for decoder is explored for better task&#10;adaptation. Our method outperforms previous methods on two- and three-talker&#10;LibriMix and LibriSpeechMix datasets for both tasks, and delivers acceptable&#10;zero-shot performance on multi-talker ASR on AishellMix Mandarin dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.13900" label="2110.13900">
        <attvalues>
          <attvalue for="0" value="WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech&#10;  Processing" />
          <attvalue for="1" value="  Self-supervised learning (SSL) achieves great success in speech recognition,&#10;while limited exploration has been attempted for other speech processing tasks.&#10;As speech signal contains multi-faceted information including speaker identity,&#10;paralinguistics, spoken content, etc., learning universal representations for&#10;all speech tasks is challenging. To tackle the problem, we propose a new&#10;pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM&#10;jointly learns masked speech prediction and denoising in pre-training. By this&#10;means, WavLM does not only keep the speech content modeling capability by the&#10;masked speech prediction, but also improves the potential to non-ASR tasks by&#10;the speech denoising. In addition, WavLM employs gated relative position bias&#10;for the Transformer structure to better capture the sequence ordering of input&#10;speech. We also scale up the training dataset from 60k hours to 94k hours.&#10;WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and&#10;brings significant improvements for various speech processing tasks on their&#10;representative benchmarks. The code and pre-trained models are available at&#10;https://aka.ms/wavlm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.00656" label="2404.00656">
        <attvalues>
          <attvalue for="0" value="WavLLM: Towards Robust and Adaptive Speech Large Language Model" />
          <attvalue for="1" value="  The recent advancements in large language models (LLMs) have revolutionized&#10;the field of natural language processing, progressively broadening their scope&#10;to multimodal perception and generation. However, effectively integrating&#10;listening capabilities into LLMs poses significant challenges, particularly&#10;with respect to generalizing across varied contexts and executing complex&#10;auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech&#10;large language model with dual encoders, and a prompt-aware LoRA weight&#10;adapter, optimized by a two-stage curriculum learning approach. Leveraging dual&#10;encoders, we decouple different types of speech information, utilizing a&#10;Whisper encoder to process the semantic content of speech, and a WavLM encoder&#10;to capture the unique characteristics of the speaker's identity. Within the&#10;curriculum learning framework, WavLLM first builds its foundational&#10;capabilities by optimizing on mixed elementary single tasks, followed by&#10;advanced multi-task training on more complex tasks such as combinations of the&#10;elementary tasks. To enhance the flexibility and adherence to different tasks&#10;and instructions, a prompt-aware LoRA weight adapter is introduced in the&#10;second advanced multi-task training stage. We validate the proposed model on&#10;universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also&#10;apply it to specialized datasets like Gaokao English listening comprehension&#10;set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments&#10;demonstrate that the proposed model achieves state-of-the-art performance&#10;across a range of speech tasks on the same model size, exhibiting robust&#10;generalization capabilities in executing complex tasks using CoT approach.&#10;Furthermore, our model successfully completes Gaokao tasks without specialized&#10;training. The codes, models, audio, and Gaokao evaluation set can be accessed&#10;at \url{aka.ms/wavllm}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.00518" label="2402.00518">
        <attvalues>
          <attvalue for="0" value="EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit&#10;  Large Language Models" />
          <attvalue for="1" value="  This work introduces EE-Tuning, a lightweight and economical solution to&#10;training/tuning early-exit large language models (LLMs). In contrast to the&#10;common approach of full-parameter pre-training, EE-Tuning augments any&#10;pre-trained (and possibly fine-tuned) standard LLM with additional early-exit&#10;layers that are tuned in a parameter-efficient manner, which requires&#10;significantly less computational resources and training data. Our&#10;implementation of EE-Tuning achieves outstanding training efficiency via&#10;extensive performance optimizations, as well as scalability due to its full&#10;compatibility with 3D parallelism. Results of systematic experiments validate&#10;the efficacy of EE-Tuning, confirming that effective early-exit LLM inference&#10;can be achieved with a limited training budget. In hope of making early-exit&#10;LLMs accessible to the community, we release the source code of our&#10;implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformer-based large language models (LLMs) have achieved extraordinary performance on various language tasks \cite{vaswani2017attention,Brown2020,OpenAI2023,Touvron2023llama,Touvron2023llama2,Chowdhery2022}.&#10;Meanwhile, these models incur high costs and latency during the inference phase, due to their increasingly large sizes.&#10;Early exiting has proven to be a simple yet effective technique for accelerating inference of LLMs and other deep neural networks.&#10;In this approach, early-exit layers are attached to the original deep neural network, &#10;which can convert intermediate hidden states into early-exit output.&#10;During inference, the model can adaptively select one early exit to generate the output for each input sample, &#10;skipping the forward computation of the remaining layers of the network.&#10;Early exiting has found success in &#10;natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, &#10;computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, &#10;and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}.&#10;&#10;This work considers token-wise early exiting for generative LLMs and autoregressive natural language generation&#10;\cite{Schuster2021,DelCorro2023,Bae2023,Varshney2023AcceleratingLI,Gera2023TheBO,chen2023eellm}.&#10;While the majority of prior works in this area have focused on designing early-exit inference mechanisms, &#10;we instead focus on how to train an early-exit LLM in the first place.&#10;The standard and straightforward method, adopted in most prior works on early exiting, &#10;is to jointly train all model parameters (including the network backbone and early-exit layers) from scratch,&#10;by minimizing a weighted sum of training losses from early and final exits.&#10;Recent work has made this approach compatible with massive 3D parallelism,&#10;thereby scaling up early-exit LLMs to sizes as large as any standard LLM that can possibly be trained with state-of-the-art LLM frameworks \cite{Shoeybi2019,Narayanan2021,chen2023eellm}.&#10;The obvious issue with this approach is its excessively high costs and complexity.&#10;Indeed, the massive amount of computational resources required to train an LLM with billions of parameters is simply inaccessible to most members of the community.&#10;Oftentimes in practice, one has access to the weights of an existing pre-trained (and possibly fine-tuned) standard LLM&#10;that might be private or open-source, &#10;and wonder if it is possible to train an early-exit LLM by leveraging such information rather than from scratch.&#10;&#10;\begin{tcolorbox}[&#10; standard jigsaw,&#10; size=title,&#10; opacityback=0,&#10; left=0.2mm, right=0.2mm, top=0.2mm, bottom=0.2mm]&#10;All these motivate us to convert an existing generative LLM to an early-exit one, in a way that &#10;\begin{itemize}&#10; \item requires minimum computational resources;&#10; \item leads to satisfactory inference acceleration; and&#10; \item preserves the full capability of the original LLM.&#10;\end{itemize}&#10;\end{tcolorbox}&#10;&#10;Main contributions.&#10;&#10;This work introduces \eetuning, a principled and lightweight approach of transforming a pre-trained (and possibly fine-tuned) LLM into an early-exit one, which satisfies all the above requirements. &#10;At the core of \eetuning is an intuitive and practical two-stage procedure:&#10;\begin{enumerate}&#10; \item Take a pre-trained standard LLM as input, and augment its architecture with early-exit layers, whose parameters are initialized properly;&#10; \item Tune the early-exit layers via backpropagation of certain training losses in a parameter-efficient manner, with modules of the original standard LLM frozen.&#10;\end{enumerate}&#10;See Figure~\ref{fig:ee_tuning_process} for a visualization of our method, &#10;whose details will be elucidated in Section~\ref{sec:methodology}.&#10;Our implementation is based on the recently proposed \eellm framework \cite{chen2023eellm}, &#10;complementing the latter with an alternative solution to training early-exit LLMs that is &#10;both accessible and scalable, &#10;thanks to its low computational complexity and full compatibility with 3D parallelism.&#10;In other words, any LLM developer, with access to either one GPU or a cluster with thousands of GPUs, will find \eetuning a useful and practical tool for studying and applying early exiting.&#10;Our implementation also includes support for various configurations and other favorable features, which further makes it more convenient to use.&#10;&#10;The efficacy of \eetuning is validated via extensive and systematic experiments&#10;for models with up to 70 billion (70B) parameters, an unprecedented scale for early-exit LLMs.&#10;More specifically, &#10;a pre-trained LLM can quickly acquire the ability of early exiting via the tuning process with fast and stable convergence,&#10;which takes less than 1/1000 of the GPU hours and training data used in its pre-training stage,&#10;and requires only one or a few GPUs.&#10;Meanwhile, the converted model can achieve $1.2\times$ to $1.6\times$ speedup on various downstream tasks through early exiting while maintaining comparable or even better benchmark scores, or higher speedup if slight degeneration of output quality is acceptable.&#10;We thoroughly investigate the effects of various design choices and &#10;provide practical guidelines for maximizing the performance of \eetuning.&#10;The source code of \eetuning is available at.&#10;&#10;Related works.&#10;&#10;The idea of \eetuning, i.e.~augmenting a pre-trained neural network with early-exit layers that are tuned in a parameter-efficient manner, is not completely new.&#10;This strategy has been adopted in some prior works for model architectures tailored to classification tasks, &#10;e.g.~the encoder-only BERT model \cite{Xin2020,Liu2020,Hu2023SmartBERTAP} or others \cite{Panda2015ConditionalDL,Kaya2018,Bakhtiarnia2021ImprovingTA,Dai2023ApparateRE}.&#10;However, there is no guarantee that results and conclusions from these works can safely transfer to the case of decoder-only Transformers tailored to autoregressive sequence generation, which is the focus of our work.&#10;Another recent work \cite{Varshney2023AcceleratingLI} proposed to initialize the model parameters of early-exit LLMs with pre-trained standard LLMs, but followed by full-parameter training.&#10;Closest to our setting and training methodology is the recent work \cite{Gera2023TheBO}, &#10;which investigated generative LLMs of sizes up to 355M, and only considered linear exit heads that are randomly initialized.&#10;Moreover, that work proposed to use multiple exits for improving the final output of full-model inference, &#10;rather than accelerating inference via early exiting.&#10;In contrast, our implementation of the proposed \eetuning method &#10;(1)~is unified and systematic, with support for a wide range of configurations; &#10;(2)~is scalable, &#10;thanks to its full compatibility with 3D parallelism;&#10;and &#10;(3)~has proven via extensive experiments to return early-exit LLMs that achieve outstanding acceleration during autoregressive inference.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Efficient Training Methods, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1603.08983" label="1603.08983">
        <attvalues>
          <attvalue for="0" value="Adaptive Computation Time for Recurrent Neural Networks" />
          <attvalue for="1" value="  This paper introduces Adaptive Computation Time (ACT), an algorithm that&#10;allows recurrent neural networks to learn how many computational steps to take&#10;between receiving an input and emitting an output. ACT requires minimal changes&#10;to the network architecture, is deterministic and differentiable, and does not&#10;add any noise to the parameter gradients. Experimental results are provided for&#10;four synthetic problems: determining the parity of binary vectors, applying&#10;binary logic operations, adding integers, and sorting real numbers. Overall,&#10;performance is dramatically improved by the use of ACT, which successfully&#10;adapts the number of computational steps to the requirements of the problem. We&#10;also present character-level language modelling results on the Hutter prize&#10;Wikipedia dataset. In this case ACT does not yield large gains in performance;&#10;however it does provide intriguing insight into the structure of the data, with&#10;more computation allocated to harder-to-predict transitions, such as spaces&#10;between words and ends of sentences. This suggests that ACT or other adaptive&#10;computation methods could provide a generic method for inferring segment&#10;boundaries in sequence data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.04152" label="2006.04152">
        <attvalues>
          <attvalue for="0" value="BERT Loses Patience: Fast and Robust Inference with Early Exit" />
          <attvalue for="1" value="  In this paper, we propose Patience-based Early Exit, a straightforward yet&#10;effective inference method that can be used as a plug-and-play technique to&#10;simultaneously improve the efficiency and robustness of a pretrained language&#10;model (PLM). To achieve this, our approach couples an internal-classifier with&#10;each layer of a PLM and dynamically stops inference when the intermediate&#10;predictions of the internal classifiers remain unchanged for a pre-defined&#10;number of steps. Our approach improves inference efficiency as it allows the&#10;model to make a prediction with fewer layers. Meanwhile, experimental results&#10;with an ALBERT model show that our method can improve the accuracy and&#10;robustness of the model by preventing it from overthinking and exploiting&#10;multiple classifiers for prediction, yielding a better accuracy-speed trade-off&#10;compared to existing early exit methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Existing research in improving the efficiency of deep neural networks&#10;can be categorized into two streams: (1) Static approaches design compact models or compress heavy models, while the models remain static for all instances at inference (i.e., the input goes through the same layers);&#10;(2)&#10;Dynamic approaches allow the model to choose different computational paths according to different instances when doing inference. In this way, the simpler inputs usually require less calculation to make predictions. Our proposed PABEE falls into the second category.&#10;&#10;Static Approaches: Compact Network Design and Model Compression&#10;&#10;Many lightweight neural network architectures have been specifically designed for resource-constrained applications, including MobileNet~\cite{howard2017mobilenets}, ShuffleNet~\cite{zhang2018shufflenet}, EfficientNet~\cite{tan2019efficientnet}, and ALBERT~\cite{lan2019albert}, to name a few. For model compression, \cite{han2015deep} first proposed to sparsify deep models by removing non-significant synapses and then re-training to restore performance. Weight Quantization~\cite{wu2016quantized} and Knowledge Distillation~\cite{hinton2015distilling} have also proved to be effective for compressing neural models. Recently, existing studies employ Knowledge Distillation~\cite{sanh2019distilbert,sun2019patient,jiao2019tinybert}, Weight Pruning~\cite{michel2019sixteen, voita2019analyzing, fan2019reducing} and Module Replacing~\cite{xu2020bert} to accelerate PLMs.&#10;&#10;Dynamic Approaches: Input-Adaptive Inference &#10;A parallel line of research for improving the efficiency of neural networks is to enable adaptive inference for various input instances. Adaptive Computation Time~\cite{graves2016adaptive,ut} proposed to use a trainable halting mechanism to perform input-adaptive inference. However, training the halting model requires extra effort and also introduces additional parameters and inference cost. To alleviate this problem, BranchyNet~\cite{teerapittayanon2016branchynet} calculated the entropy of the prediction probability distribution as a proxy for the confidence of branch classifiers to enable early exit. Shallow-Deep Nets~\cite{kaya2018shallow} leveraged the softmax scores of predictions of branch classifiers to mitigate the overthinking problem of DNNs. More recently, \cite{hu2020triple} leveraged this approach in adversarial training to improve the adversarial robustness of DNNs. In addition, existing approaches \cite{graves2016adaptive,wang2018skipnet} trained separate models to determine passing through or skipping each layer. Very recently, FastBERT~\cite{liu2020fastbert} and DeeBERT~\cite{xin2020deebert} adapted confidence-based BranchyNet~\cite{teerapittayanon2016branchynet} for PLMs while RightTool~\cite{Schwartz:2020} leveraged &#10;the same early-exit criterion as in the Shallow-Deep Network~\cite{kaya2018shallow}. &#10;&#10;However, \cite{Schwartz:2020} recently revealed that prediction probability based methods often lead to substantial performance drop compared to an oracle that identifies the smallest model needed to solve a given instance. In addition, these methods only support classification and leave out regression, which limits their applications. Different from the recent work that directly employs existing efficient inference methods on top of PLMs, PABEE is a novel early-exit criterion that captures the inner-agreement between earlier and later internal classifiers and exploit multiple classifiers for inference, leading to better accuracy. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2004.07453" label="2004.07453">
        <attvalues>
          <attvalue for="0" value="The Right Tool for the Job: Matching Model and Instance Complexities" />
          <attvalue for="1" value="  As NLP models become larger, executing a trained model requires significant&#10;computational resources incurring monetary and environmental costs. To better&#10;respect a given inference budget, we propose a modification to contextual&#10;representation fine-tuning which, during inference, allows for an early (and&#10;fast) &quot;exit&quot; from neural network calculations for simple instances, and late&#10;(and accurate) exit for hard instances. To achieve this, we add classifiers to&#10;different layers of BERT and use their calibrated confidence scores to make&#10;early exit decisions. We test our proposed modification on five different&#10;datasets in two tasks: three text classification datasets and two natural&#10;language inference benchmarks. Our method presents a favorable speed/accuracy&#10;tradeoff in almost all cases, producing models which are up to five times&#10;faster than the state of the art, while preserving their accuracy. Our method&#10;also requires almost no additional training resources (in either time or&#10;parameters) compared to the baseline BERT model. Finally, our method alleviates&#10;the need for costly retraining of multiple models at different levels of&#10;efficiency; we allow users to control the inference speed/accuracy tradeoff&#10;using a single trained model, by setting a single variable at inference time.&#10;We publicly release our code.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.02178" label="2004.02178">
        <attvalues>
          <attvalue for="0" value="FastBERT: a Self-distilling BERT with Adaptive Inference Time" />
          <attvalue for="1" value="  Pre-trained language models like BERT have proven to be highly performant.&#10;However, they are often computationally expensive in many practical scenarios,&#10;for such heavy models can hardly be readily implemented with limited resources.&#10;To improve their efficiency with an assured model performance, we propose a&#10;novel speed-tunable FastBERT with adaptive inference time. The speed at&#10;inference can be flexibly adjusted under varying demands, while redundant&#10;calculation of samples is avoided. Moreover, this model adopts a unique&#10;self-distillation mechanism at fine-tuning, further enabling a greater&#10;computational efficacy with minimal loss in performance. Our model achieves&#10;promising results in twelve English and Chinese datasets. It is able to speed&#10;up by a wide range from 1 to 12 times than BERT if given different speedup&#10;thresholds to make a speed-performance tradeoff.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;BERT \cite{devlin2019bert} can learn universal knowledge from mass unlabeled data and produce more performant outcomes. Many works have followed: RoBERTa \cite{liu2019roberta} that uses larger corpus and longer training steps. T5 \cite{raffel2019exploring} that scales up the model size even more. UER \cite{zhao2019uer} pre-trains BERT in different Chinese corpora. K-BERT \cite{liu2019k} injects knowledge graph into BERT model. These models achieve increased accuracy with heavier settings and even more data.&#10;&#10;However, such unwieldy sizes are often hampered under stringent conditions. To be more specific, BERT-base contains 110 million parameters by stacking twelve Transformer blocks \cite{vaswani2017attention}, while BERT-large expands its size to even 24 layers. ALBERT \cite{lan2019albert} shares the parameters of each layer to reduce the model size. Obviously, the inference speed for these models would be much slower than classic architectures (e.g., CNN \cite{kim-2014-convolutional}, RNN \cite{wang-2018-disconnected}, etc). We think a large proportion of computation is caused by redundant calculation.&#10;&#10;Knowledge distillation: Many attempts have been made to distill heavy models (teachers) into their lighter counterparts (students). PKD-BERT \cite{sun2019patient} adopts an incremental extraction process that learns generalizations from intermediate layers of the teacher model. TinyBERT \cite{jiao2019tinybert} performs a two-stage learning involving both general-domain pre-training and task-specific fine-tuning. DistilBERT \cite{sanh2019distilbert} further leveraged the inductive bias within large models by introducing a triple loss. As shown in Figure \ref{fig:classic_distillation}, student model often require a separated structure, whose effect however, depends mainly on the gains of the teacher. They are as indiscriminate to individual cases as their teachers, and only get faster in the cost of degraded performance. &#10;&#10;Adaptive inference: Conventional approaches in adaptive computations are performed token-wise or patch-wise, who either adds recurrent steps to individual tokens \cite{graves2016adaptive} or dynamically adjusts the number of executed layers inside discrete regions of images \cite{teerapittayanon2016branchynet, figurnov2017spatially}. To the best of our knowledge, there has been no work in applying adaptive mechanisms to NLP pre-training language models for efficiency improvements so far.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1910.10073" label="1910.10073">
        <attvalues>
          <attvalue for="0" value="Depth-Adaptive Transformer" />
          <attvalue for="1" value="  State of the art sequence-to-sequence models for large scale tasks perform a&#10;fixed number of computations for each input sequence regardless of whether it&#10;is easy or hard to process. In this paper, we train Transformer models which&#10;can make output predictions at different stages of the network and we&#10;investigate different ways to predict how much computation is required for a&#10;particular sequence. Unlike dynamic computation in Universal Transformers,&#10;which applies the same set of layers iteratively, we apply different layers at&#10;every step to adjust both the amount of computation as well as the model&#10;capacity. On IWSLT German-English translation our approach matches the accuracy&#10;of a well tuned baseline Transformer while using less than a quarter of the&#10;decoder layers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.12993" label="2004.12993">
        <attvalues>
          <attvalue for="0" value="DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference" />
          <attvalue for="1" value="  Large-scale pre-trained language models such as BERT have brought significant&#10;improvements to NLP applications. However, they are also notorious for being&#10;slow in inference, which makes them difficult to deploy in real-time&#10;applications. We propose a simple but effective method, DeeBERT, to accelerate&#10;BERT inference. Our approach allows samples to exit earlier without passing&#10;through the entire model. Experiments show that DeeBERT is able to save up to&#10;~40% inference time with minimal degradation in model quality. Further analyses&#10;show different behaviors in the BERT transformer layers and also reveal their&#10;redundancy. Our work provides new ideas to efficiently apply deep&#10;transformer-based models to downstream tasks. Code is available at&#10;https://github.com/castorini/DeeBERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.13878" label="2105.13878">
        <attvalues>
          <attvalue for="0" value="Accelerating BERT Inference for Sequence Labeling via Early-Exit" />
          <attvalue for="1" value="  Both performance and efficiency are crucial factors for sequence labeling&#10;tasks in many real-world scenarios. Although the pre-trained models (PTMs) have&#10;significantly improved the performance of various sequence labeling tasks,&#10;their computational cost is expensive. To alleviate this problem, we extend the&#10;recent successful early-exit mechanism to accelerate the inference of PTMs for&#10;sequence labeling tasks. However, existing early-exit mechanisms are&#10;specifically designed for sequence-level tasks, rather than sequence labeling.&#10;In this paper, we first propose a simple extension of sentence-level early-exit&#10;for sequence labeling tasks. To further reduce the computational cost, we also&#10;propose a token-level early-exit mechanism that allows partial tokens to exit&#10;early at different layers. Considering the local dependency inherent in&#10;sequence labeling, we employed a window-based criterion to decide for a token&#10;whether or not to exit. The token-level early-exit brings the gap between&#10;training and inference, so we introduce an extra self-sampling fine-tuning&#10;stage to alleviate it. The extensive experiments on three popular sequence&#10;labeling tasks show that our approach can save up to 66%-75% inference cost&#10;with minimal performance degradation. Compared with competitive compressed&#10;models such as DistilBERT, our approach can achieve better performance under&#10;the same speed-up ratios of 2X, 3X, and 4X.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;PTMs are powerful but have high computational cost. To accelerate them, many attempts have been made. A kind of methods is to reduce its size, such as distillation \cite{sanh2019distilbert,tiny-bert}, structural pruning \cite{prune-head,layer-drop} and quantization \cite{Shen_Dong_Ye_Ma_Yao_Gholami_Mahoney_Keutzer_2020}. &#10;&#10;Another kind of methods is early-exit, which dynamically adjusts the encoding layer number of different samples ~\cite{liu-etal-2020-fastbert,xin-etal-2020-deebert,schwartz-etal-2020-right,zhou-bert-patience,DBLP:journals/corr/abs-2012-14682}.&#10;While they introduced early-exit mechanism in simple classification tasks, our methods are proposed for the more complicated scenario: sequence labeling, where it has not only one prediction probability and it's necessary to consider the dependency of token exitings. \cite{jiatao-dpeth-adaptive} proposed Depth-Adaptive Transformer to accelerate machine translation. However, their early-exit mechanism is designed for auto-regressive sequence generation, in which the exit of tokens must be in left-to-right order. Therefore, it is unsuitable for language understanding tasks. Different from their method, our early-exit mechanism can consider the exit of all tokens simultaneously.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2104.08803" label="2104.08803">
        <attvalues>
          <attvalue for="0" value="Consistent Accelerated Inference via Confident Adaptive Transformers" />
          <attvalue for="1" value="  We develop a novel approach for confidently accelerating inference in the&#10;large and expensive multilayer Transformers that are now ubiquitous in natural&#10;language processing (NLP). Amortized or approximate computational methods&#10;increase efficiency, but can come with unpredictable performance costs. In this&#10;work, we present CATs -- Confident Adaptive Transformers -- in which we&#10;simultaneously increase computational efficiency, while guaranteeing a&#10;specifiable degree of consistency with the original model with high confidence.&#10;Our method trains additional prediction heads on top of intermediate layers,&#10;and dynamically decides when to stop allocating computational effort to each&#10;input using a meta consistency classifier. To calibrate our early prediction&#10;stopping rule, we formulate a unique extension of conformal prediction. We&#10;demonstrate the effectiveness of this approach on four classification and&#10;regression tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.07101" label="2202.07101">
        <attvalues>
          <attvalue for="0" value="A Survey on Dynamic Neural Networks for Natural Language Processing" />
          <attvalue for="1" value="  Effectively scaling large Transformer models is a main driver of recent&#10;advances in natural language processing. Dynamic neural networks, as an&#10;emerging research direction, are capable of scaling up neural networks with&#10;sub-linear increases in computation and time by dynamically adjusting their&#10;computational path based on the input. Dynamic neural networks could be a&#10;promising solution to the growing parameter numbers of pretrained language&#10;models, allowing both model pretraining with trillions of parameters and faster&#10;inference on mobile devices. In this survey, we summarize progress of three&#10;types of dynamic neural networks in NLP: skimming, mixture of experts, and&#10;early exit. We also highlight current challenges in dynamic neural networks and&#10;directions for future research.&#10;" />
          <attvalue for="2" value="&#10;Scaling up &#10;model capacity is an obvious yet effective approach for better performance in natural language processing (NLP) tasks~\cite{gpt3,kaplan2020scaling,ghorbani2021scaling,zhou2020evaluating}. However, &#10;the &#10;resulting&#10;increase in computational complexity and memory consumption becomes a bottleneck for&#10;scaling, making these models hard to train and use. On the other hand, it &#10;is not necessary&#10;to allocate the same amount of computation to all instances. For example, categorizing ``I love you'' as a positive sentence &#10;does not require a model containing dozens of Transformer layers. To resolve the aforementioned problems, dynamic neural networks have &#10;been a significant thrust of recent research in NLP. Dynamic networks can adjust their computational path based on the input for better efficiency, making it possible to train models with trillions of parameters and accelerate models in a low-resource setting.&#10;&#10;In this survey, we review the latest state of research on three types of dynamic neural networks that have been adopted in NLP: skimming, mixtures of experts (MoE), and early exit, as illustrated in Figure~\ref{fig:dynamic}. These three types of techniques share a common idea of dynamically adjusting computation with respect to input, to save computation through bypassing unnecessary modules in a large neural network. However, they implement the goal via different approaches. Skimming was well-researched in the era of recurrent neural networks (RNN). Skimming models save computation timewise by dynamically allocating computation to different time steps, based on the input tokens.&#10;Since RNN models process the input sequence recurrently, it allows skimming models to achieve a substantial acceleration, especially when the sequence is long~\cite{li2019teach}. Different from RNN, recent works on Transformers skip tokens between layers instead of time steps.&#10;&#10;For Transformer models~\cite{transformer,bert,albert,gpt3}, the input tokens are fed into the model in parallel, while models have dozens of Transformer layers. This motivates the development of MoE and early exit.&#10;MoE horizontally extends a feedforward neural network (FFNN) with multiple sub-networks. During inference, only one or a few of these sub-networks will be activated for computation, thus can save widthwise computation. Early exit, on the other hand, terminates inference at an early layer, without exhausting full computational capacity, thus saves depthwise computation. Early exit techniques often insert a series of lightweight classifiers which help decide when to exit, based on an exit strategy. &#10;&#10;Note that this stream of works is distinct from static model acceleration, which is often referred to as model compression, including knowledge distillation, weight sharing, pruning and quantization~\cite{sanh2019distilbert,bot,albert,zafrir2019q8bert,xu2021beyond} (etc., see another survey~\cite{xu2022survey}). The major difference is that the computational path in a statically compressed model does not condition on the input and is invariable for all examples in inference. These two streams of research are in fact orthogonal and recent works \cite{schwartz2020right}, \cite{fastbert} and \cite{leebert} have shown that static and dynamic approaches can be combined for even faster inference and better performance.&#10;&#10;To summarize, our contribution is two-fold: (1) We review the latest studies on the topic of dynamic neural networks for NLP by providing a comprehensive comparison and organize them with a new taxonomy, as shown in Figure~\ref{fig:dynamic}.&#10;(2) We analyze current challenges in dynamic neural networks and point out directions for future research.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.09266" label="2303.09266">
        <attvalues>
          <attvalue for="0" value="SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for&#10;  Accelerating BERT Inference" />
          <attvalue for="1" value="  Dynamic early exiting has been proven to improve the inference speed of the&#10;pre-trained language model like BERT. However, all samples must go through all&#10;consecutive layers before early exiting and more complex samples usually go&#10;through more layers, which still exists redundant computation. In this paper,&#10;we propose a novel dynamic early exiting combined with layer skipping for BERT&#10;inference named SmartBERT, which adds a skipping gate and an exiting operator&#10;into each layer of BERT. SmartBERT can adaptively skip some layers and&#10;adaptively choose whether to exit. Besides, we propose cross-layer contrastive&#10;learning and combine it into our training phases to boost the intermediate&#10;layers and classifiers which would be beneficial for early exiting. To keep the&#10;consistent usage of skipping gates between training and inference phases, we&#10;propose a hard weight mechanism during training phase. We conduct experiments&#10;on eight classification datasets of the GLUE benchmark. Experimental results&#10;show that SmartBERT achieves 2-3x computation reduction with minimal accuracy&#10;drops compared with BERT and our method outperforms previous methods in both&#10;efficiency and accuracy. Moreover, in some complex datasets like RTE and WNLI,&#10;we prove that the early exiting based on entropy hardly works, and the skipping&#10;mechanism is essential for reducing computation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1509.08971" label="1509.08971">
        <attvalues>
          <attvalue for="0" value="Conditional Deep Learning for Energy-Efficient and Enhanced Pattern&#10;  Recognition" />
          <attvalue for="1" value="  Deep learning neural networks have emerged as one of the most powerful&#10;classification tools for vision related applications. However, the&#10;computational and energy requirements associated with such deep nets can be&#10;quite high, and hence their energy-efficient implementation is of great&#10;interest. Although traditionally the entire network is utilized for the&#10;recognition of all inputs, we observe that the classification difficulty varies&#10;widely across inputs in real-world datasets; only a small fraction of inputs&#10;require the full computational effort of a network, while a large majority can&#10;be classified correctly with very low effort. In this paper, we propose&#10;Conditional Deep Learning (CDL) where the convolutional layer features are used&#10;to identify the variability in the difficulty of input instances and&#10;conditionally activate the deeper layers of the network. We achieve this by&#10;cascading a linear network of output neurons for each convolutional layer and&#10;monitoring the output of the linear network to decide whether classification&#10;can be terminated at the current stage or not. The proposed methodology thus&#10;enables the network to dynamically adjust the computational effort depending&#10;upon the difficulty of the input data while maintaining competitive&#10;classification accuracy. We evaluate our approach on the MNIST dataset. Our&#10;experiments demonstrate that our proposed CDL yields 1.91x reduction in average&#10;number of operations per input, which translates to 1.84x improvement in&#10;energy. In addition, our results show an improvement in classification accuracy&#10;from 97.5% to 98.9% as compared to the original network.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.01686" label="1709.01686">
        <attvalues>
          <attvalue for="0" value="BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks" />
          <attvalue for="1" value="  Deep neural networks are state of the art methods for many learning tasks due&#10;to their ability to extract increasingly better features at each network layer.&#10;However, the improved performance of additional layers in a deep network comes&#10;at the cost of added latency and energy usage in feedforward inference. As&#10;networks continue to get deeper and larger, these costs become more prohibitive&#10;for real-time and energy-sensitive applications. To address this issue, we&#10;present BranchyNet, a novel deep network architecture that is augmented with&#10;additional side branch classifiers. The architecture allows prediction results&#10;for a large portion of test samples to exit the network early via these&#10;branches when samples can already be inferred with high confidence. BranchyNet&#10;exploits the observation that features learned at an early layer of a network&#10;may often be sufficient for the classification of many data points. For more&#10;difficult samples, which are expected less frequently, BranchyNet will use&#10;further or all network layers to provide the best likelihood of correct&#10;prediction. We study the BranchyNet architecture using several well-known&#10;networks (LeNet, AlexNet, ResNet) and datasets (MNIST, CIFAR10) and show that&#10;it can both improve accuracy and significantly reduce the inference time of the&#10;network.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.07052" label="1810.07052">
        <attvalues>
          <attvalue for="0" value="Shallow-Deep Networks: Understanding and Mitigating Network Overthinking" />
          <attvalue for="1" value="  We characterize a prevalent weakness of deep neural networks&#10;(DNNs)---overthinking---which occurs when a DNN can reach correct predictions&#10;before its final layer. Overthinking is computationally wasteful, and it can&#10;also be destructive when, by the final layer, a correct prediction changes into&#10;a misclassification. Understanding overthinking requires studying how each&#10;prediction evolves during a DNN's forward pass, which conventionally is opaque.&#10;For prediction transparency, we propose the Shallow-Deep Network (SDN), a&#10;generic modification to off-the-shelf DNNs that introduces internal&#10;classifiers. We apply SDN to four modern architectures, trained on three image&#10;classification tasks, to characterize the overthinking problem. We show that&#10;SDNs can mitigate the wasteful effect of overthinking with confidence-based&#10;early exits, which reduce the average inference cost by more than 50% and&#10;preserve the accuracy. We also find that the destructive effect occurs for 50%&#10;of misclassifications on natural inputs and that it can be induced,&#10;adversarially, with a recent backdooring attack. To mitigate this effect, we&#10;propose a new confusion metric to quantify the internal disagreements that will&#10;likely lead to misclassifications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.09844" label="1703.09844">
        <attvalues>
          <attvalue for="0" value="Multi-Scale Dense Networks for Resource Efficient Image Classification" />
          <attvalue for="1" value="  In this paper we investigate image classification with computational resource&#10;limits at test time. Two such settings are: 1. anytime classification, where&#10;the network's prediction for a test example is progressively updated,&#10;facilitating the output of a prediction at any time; and 2. budgeted batch&#10;classification, where a fixed amount of computation is available to classify a&#10;set of examples that can be spent unevenly across &quot;easier&quot; and &quot;harder&quot; inputs.&#10;In contrast to most prior work, such as the popular Viola and Jones algorithm,&#10;our approach is based on convolutional neural networks. We train multiple&#10;classifiers with varying resource demands, which we adaptively apply during&#10;test time. To maximally re-use computation between the classifiers, we&#10;incorporate them as early-exits into a single deep convolutional neural network&#10;and inter-connect them with dense connectivity. To facilitate high quality&#10;classification early on, we use a two-dimensional multi-scale network&#10;architecture that maintains coarse and fine level features all-throughout the&#10;network. Experiments on three image-classification tasks demonstrate that our&#10;framework substantially improves the existing state-of-the-art in both&#10;settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.12814" label="2004.12814">
        <attvalues>
          <attvalue for="0" value="Why should we add early exits to neural networks?" />
          <attvalue for="1" value="  Deep neural networks are generally designed as a stack of differentiable&#10;layers, in which a prediction is obtained only after running the full stack.&#10;Recently, some contributions have proposed techniques to endow the networks&#10;with early exits, allowing to obtain predictions at intermediate points of the&#10;stack. These multi-output networks have a number of advantages, including: (i)&#10;significant reductions of the inference time, (ii) reduced tendency to&#10;overfitting and vanishing gradients, and (iii) capability of being distributed&#10;over multi-tier computation platforms. In addition, they connect to the wider&#10;themes of biological plausibility and layered cognitive reasoning. In this&#10;paper, we provide a comprehensive introduction to this family of neural&#10;networks, by describing in a unified fashion the way these architectures can be&#10;designed, trained, and actually deployed in time-constrained scenarios. We also&#10;describe in-depth their application scenarios in 5G and Fog computing&#10;environments, as long as some of the open research questions connected to them.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.05022" label="2106.05022">
        <attvalues>
          <attvalue for="0" value="Adaptive Inference through Early-Exit Networks: Design, Challenges and&#10;  Directions" />
          <attvalue for="1" value="  DNNs are becoming less and less over-parametrised due to recent advances in&#10;efficient model design, through careful hand-crafted or NAS-based methods.&#10;Relying on the fact that not all inputs require the same amount of computation&#10;to yield a confident prediction, adaptive inference is gaining attention as a&#10;prominent approach for pushing the limits of efficient deployment.&#10;Particularly, early-exit networks comprise an emerging direction for tailoring&#10;the computation depth of each input sample at runtime, offering complementary&#10;performance gains to other efficiency optimisations. In this paper, we&#10;decompose the design methodology of early-exit networks to its key components&#10;and survey the recent advances in each one of them. We also position&#10;early-exiting against other efficient inference solutions and provide our&#10;insights on the current challenges and most promising future directions for&#10;research in the field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.04906" label="2102.04906">
        <attvalues>
          <attvalue for="0" value="Dynamic Neural Networks: A Survey" />
          <attvalue for="1" value="  Dynamic neural network is an emerging research topic in deep learning.&#10;Compared to static models which have fixed computational graphs and parameters&#10;at the inference stage, dynamic networks can adapt their structures or&#10;parameters to different inputs, leading to notable advantages in terms of&#10;accuracy, computational efficiency, adaptiveness, etc. In this survey, we&#10;comprehensively review this rapidly developing area by dividing dynamic&#10;networks into three main categories: 1) instance-wise dynamic models that&#10;process each instance with data-dependent architectures or parameters; 2)&#10;spatial-wise dynamic networks that conduct adaptive computation with respect to&#10;different spatial locations of image data and 3) temporal-wise dynamic models&#10;that perform adaptive inference along the temporal dimension for sequential&#10;data such as videos and texts. The important research problems of dynamic&#10;networks, e.g., architecture design, decision making scheme, optimization&#10;technique and applications, are reviewed systematically. Finally, we discuss&#10;the open problems in this field together with interesting future research&#10;directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.05385" label="2312.05385">
        <attvalues>
          <attvalue for="0" value="Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in&#10;  ML Serving" />
          <attvalue for="1" value="  Machine learning (ML) inference platforms are tasked with balancing two&#10;competing goals: ensuring high throughput given many requests, and delivering&#10;low-latency responses to support interactive applications. Unfortunately,&#10;existing platform knobs (e.g., batch sizes) fail to ease this fundamental&#10;tension, and instead only enable users to harshly trade off one property for&#10;the other. This paper explores an alternate strategy to taming&#10;throughput-latency tradeoffs by changing the granularity at which inference is&#10;performed. We present Apparate, a system that automatically applies and manages&#10;early exits (EEs) in ML models, whereby certain inputs can exit with results at&#10;intermediate layers. To cope with the time-varying overhead and accuracy&#10;challenges that EEs bring, Apparate repurposes exits to provide continual&#10;feedback that powers several novel runtime monitoring and adaptation&#10;strategies. Apparate lowers median response latencies by 40.5-91.5% and&#10;10.0-24.2% for diverse CV and NLP workloads, respectively, without affecting&#10;throughputs or violating tight accuracy constraints.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.02628" label="2307.02628">
        <attvalues>
          <attvalue for="0" value="SkipDecode: Autoregressive Skip Decoding with Batching and Caching for&#10;  Efficient LLM Inference" />
          <attvalue for="1" value="  Autoregressive large language models (LLMs) have made remarkable progress in&#10;various natural language generation tasks. However, they incur high computation&#10;cost and latency resulting from the autoregressive token-by-token generation.&#10;To address this issue, several approaches have been proposed to reduce&#10;computational cost using early-exit strategies. These strategies enable faster&#10;text generation using reduced computation without applying the full computation&#10;graph to each token. While existing token-level early exit methods show&#10;promising results for online inference, they cannot be readily applied for&#10;batch inferencing and Key-Value caching. This is because they have to wait&#10;until the last token in a batch exits before they can stop computing. This&#10;severely limits the practical application of such techniques. In this paper, we&#10;propose a simple and effective token-level early exit method, SkipDecode,&#10;designed to work seamlessly with batch inferencing and KV caching. It overcomes&#10;prior constraints by setting up a singular exit point for every token in a&#10;batch at each sequence position. It also guarantees a monotonic decrease in&#10;exit points, thereby eliminating the need to recompute KV Caches for preceding&#10;tokens. Rather than terminating computation prematurely as in prior works, our&#10;approach bypasses lower to middle layers, devoting most of the computational&#10;resources to upper layers, allowing later tokens to benefit from the compute&#10;expenditure by earlier tokens. Our experimental results show that SkipDecode&#10;can obtain 2x to 5x inference speedups with negligible regression across a&#10;variety of tasks. This is achieved using OPT models of 1.3 billion and 6.7&#10;billion parameters, all the while being directly compatible with batching and&#10;KV caching optimization techniques.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05424" label="2310.05424">
        <attvalues>
          <attvalue for="0" value="Fast and Robust Early-Exiting Framework for Autoregressive Language&#10;  Models with Synchronized Parallel Decoding" />
          <attvalue for="1" value="  To tackle the high inference latency exhibited by autoregressive language&#10;models, previous studies have proposed an early-exiting framework that&#10;allocates adaptive computation paths for each token based on the complexity of&#10;generating the subsequent token. However, we observed several shortcomings,&#10;including performance degradation caused by a state copying mechanism or&#10;numerous exit paths, and sensitivity to exit confidence thresholds.&#10;Consequently, we propose a Fast and Robust Early-Exiting (FREE) framework,&#10;which incorporates a shallow-deep module and a synchronized parallel decoding.&#10;Our framework enables faster inference by synchronizing the decoding process of&#10;the current token with previously stacked early-exited tokens. Furthermore, as&#10;parallel decoding allows us to observe predictions from both shallow and deep&#10;models, we present a novel adaptive threshold estimator that exploits a Beta&#10;mixture model to determine suitable confidence thresholds. We empirically&#10;demonstrated the superiority of our proposed framework on extensive generation&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.18581" label="2310.18581">
        <attvalues>
          <attvalue for="0" value="Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via&#10;  Instruction Tuning with LITE" />
          <attvalue for="1" value="  Large Language Models (LLMs) have achieved remarkable performance across a&#10;wide variety of natural language tasks; however, their large size makes their&#10;inference slow and computationally expensive. Focusing on this problem, we&#10;propose to instruction tune LLMs with additional explicit losses from the&#10;intermediate layers (LITE) and show that it enables these layers to acquire&#10;'good' generation ability without affecting the generation ability of the final&#10;layer. We perform 'dynamic confidence-based early exiting' at token level from&#10;the intermediate layers which improves the efficiency of text generation&#10;without compromising the quality of the generation. We conduct comprehensive&#10;experiments by instruction tuning LLaMA-2 models on the Alpaca dataset and&#10;holistically evaluate on four different human-instruction test sets. We show&#10;that dynamic early exiting achieves consistent and considerable inference&#10;computation cost improvements (37.86% for 7B and 46.35% for 13B model) while&#10;maintaining the generation quality of the responses. We further conduct a&#10;thorough analysis of the results over several important aspects, such as&#10;comparing the semantic similarity of the outputs and dissecting the efficiency&#10;improvements by comparing the number of tokens generated in the output. In&#10;summary, our work contributes to improving the efficiency of LLM inference&#10;while maintaining the generation quality, a crucial step en route to enabling&#10;their widespread adoption.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Improving the inference efficiency of large language models is an important research direction and is receiving considerable attention from the NLP community. &#10;In this section, we first review some of the existing methods and then detail their differences from our work.&#10;&#10;Reducing model size: Since model size plays a crucial role in increasing the inference cost and latency, techniques like quantization \cite{dettmers2022llm,yao2022zeroquant,pmlr-v202-xiao23c,frantar2022gptq}, knowledge distillation \cite{hsieh2023distilling,jiao-etal-2020-tinybert,li-etal-2022-dq,mirzadeh2020improved}, model compression and network pruning \cite{wang-etal-2020-structured,guo-etal-2021-parameter} have been shown to be effective in improving the inference efficiency.&#10;&#10;Furthermore, during sampling, a cache of the keys and values can be maintained for every attention layer which reduces the computations at inference time (KV caching). &#10;However, it increases the GPU VRAM memory requirement of inference.&#10;&#10;Another technique to improve inference efficiency is speculative sampling \cite{leviathan2023fast, chen2023accelerating} in which a short draft of $K$ tokens is first generated from a smaller (thus faster) auto-regressive model. Then, the draft is scored using the larger model which corresponds to the target model from which we wish to sample from.&#10;Using some rejection sampling scheme, a subset of the $K$ draft tokens is accepted by sequentially checking from left to right and thus in this process, we recover the distribution of the target model for the accepted tokens.&#10;The efficiency in this technique comes from `producing' more than one token (on average) from the target model in a single pass.&#10;&#10;Early exiting and cascading based inference techniques have been shown to be effective for classification tasks with BERT-style models, &#10;such as DeeBERT \cite{xin-etal-2020-deebert} that speeds up BERT inference by inserting extra classification layers between each encoder layer, PoWER-BERT \cite{goyal2020power} that focuses on progressive word-vector elimination (based on significance computed using self-attention) along the encoder pipeline, DynaBERT \cite{NEURIPS2020_6f5216f8} that adjusts the size of the model by selecting adaptive width and depth, and cascading \cite{varshney-baral-2022-model, li-etal-2021-cascadebert-accelerating,varshney-baral-2023-post,yue2023large,cheng2023batch} in which sequential inference is done through models of bigger and bigger size with conditional exiting to output predictions in an efficient yet accurate manner.&#10;Our work is also related to Confident Adaptive Language Modeling (CALM) \cite{schuster2022confident} and Depth-Adaptive Transformers \cite{Elbayad2020Depth-Adaptive} in which early exiting is performed by learning additional classifiers attached to the decoder layers. &#10;&#10;Casting internal representations: &#10;\cite{din2023jump} proposed to short-cut away transformer inference in between certain layers by learning linear transformations across layers in the network.&#10;&#10;Contrastive decoding from intermediate layers: &#10;Prior work has also explored leveraging the intermediate layers for contrastive decoding to improve reasoning \cite{o2023contrastive,gera2023benefits}.&#10;&#10;Our work differs from existing work in&#10;the following aspects: &#10;\begin{enumerate}[noitemsep,nosep,leftmargin=*]&#10;&#10;\item Most of the existing early exiting works focus on improving the efficiency of encoder-only models (like BERT) or encoder-decoder models (like T5); our work focuses on the current state-of-the-art decoder-only LLMs (LLaMA-2).&#10; Furthermore, we focus on the instruction tuning setting, unlike prior work that focused on solving specific tasks like GLUE classification tasks, summarization, or QA. &#10; We note that the open-ended generation is a much more challenging setting for intermediate layer exiting than classification or MCQ tasks.&#10;&#10;\item Early exiting methods typically require training additional classifiers for the intermediate layers, however, in our method, we use the same shared language modeling head at all the layers, thus, do not introduce new model parameters. &#10;&#10;\item For leveraging the intermediate layers for decoding, we enable them to acquire good generation ability by instruction tuning with LITE, unlike other methods that use a pre fine-tuned model in which these layers have poor generation ability.&#10;&#10;\item Most of the existing methods typically require complex architectural modifications, network pruning, saliency quantification, knowledge distillation, or adding extra model parameters. In contrast, our proposed method (both for instruction tuning and inference) is simple and easy to implement as it requires just a few modifications during tuning and inference and yet it delivers considerable improvements. &#10; Furthermore, our method is complementary to these existing methods.&#10;&#10;\item The computational efficiency in a lot of existing methods often comes with a compromise in performance. Contrary to this, we show that our dynamic early exiting method maintains the generation quality while providing efficiency benefits.&#10;&#10;\item Some existing methods typically require training a separate model for each computation budget; on the other hand, in our method, the same model can be adjusted to meet all the computation constraints (by varying the confidence thresholds for exiting). &#10;&#10;\end{enumerate}&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.01628" label="2305.01628">
        <attvalues>
          <attvalue for="0" value="The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers" />
          <attvalue for="1" value="  Applying language models to natural language processing tasks typically&#10;relies on the representations in the final model layer, as intermediate hidden&#10;layer representations are presumed to be less informative. In this work, we&#10;argue that due to the gradual improvement across model layers, additional&#10;information can be gleaned from the contrast between higher and lower layers&#10;during inference. Specifically, in choosing between the probable next token&#10;predictions of a generative model, the predictions of lower layers can be used&#10;to highlight which candidates are best avoided. We propose a novel approach&#10;that utilizes the contrast between layers to improve text generation outputs,&#10;and show that it mitigates degenerative behaviors of the model in open-ended&#10;generation, significantly improving the quality of generated texts.&#10;Furthermore, our results indicate that contrasting between model layers at&#10;inference time can yield substantial benefits to certain aspects of general&#10;language model capabilities, more effectively extracting knowledge during&#10;inference from a given set of model parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;There have been a number of studies on analyzing the characteristics of different layers of transformer models. \cite{rogers2020primer, van2019does} used probing to report that in BERT models the lower layers carry the most information about linear word order, syntactic information is most prominent in the middle layers, and the final layers of BERT are the most task-specific. \cite{van2019does} also show that similar behavior is observed in other transformer models such as GPT2. \cite{geva-etal-2021-transformer, geva2022transformer} studied the role of feed-forward layers in transformer models. They demonstrate that representations across different layers capture meaningful semantic and syntactic patterns, and describe how model predictions are gradually refined as they progress across the different layers.&#10;&#10;Aiming to reduce the computational load of transformers, multiple works have explored early-exiting, i.e., performing some calculations without passing through all of the model layers. Such works allow for an early (fast) `exit' from neural network calculations -- for simple instances that can be solved with high accuracy by lower layers -- while using a late (slow) `exit' for more challenging instances \cite{simoulin-crabbe-2021-many,schwartz-etal-2020-right, xin-etal-2020-deebert, elbayad2020depth, sun-etal-2022-simple, schusterconfident}. &#10;&#10;Decoding algorithms are commonly classified as search-based and sampling-based. Search-based methods \cite{steinbiss1994improvements} optimize for the language model log-probabilities, while sampling methods \cite{holtzman2019curious,fan-etal-2018-hierarchical} draw the next token from a truncated distribution. The idea of using contrast during decoding has been explored in several studies. \cite{liu2021dexperts} combine a pretrained LM with `expert' LMs and `anti-expert' LMs, where tokens only get high probability if they are considered likely by the experts and unlikely by the anti-experts. \cite{su2022contrastive} propose constrastive search for decoding, where the generated output is selected from the set of most probable candidates predicted by the model while being discriminative with respect to the context. More recently, \cite{li2022contrastive} suggested to contrast between the likelihood under a large LM (expert) and a small LM (amateur) during decoding. The present work differs significantly from the aforementioned contrastive approaches, in that we contrast the next-token distributions within a single LM, across expert and amateur layers. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2312.04916" label="2312.04916">
        <attvalues>
          <attvalue for="0" value="EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language&#10;  Models with 3D Parallelism" />
          <attvalue for="1" value="  We present EE-LLM, a framework for large-scale training and inference of&#10;early-exit large language models (LLMs). While recent works have shown&#10;preliminary evidence for the efficacy of early exiting in accelerating LLM&#10;inference, EE-LLM makes a foundational step towards scaling up early-exit LLMs&#10;by supporting their training and inference with massive 3D parallelism. Built&#10;upon Megatron-LM, EE-LLM implements a variety of algorithmic innovations and&#10;performance optimizations tailored to early exiting, including a lightweight&#10;method that facilitates backpropagation for the early-exit training objective&#10;with pipeline parallelism, techniques of leveraging idle resources in the&#10;original pipeline schedule for computation related to early-exit layers, and&#10;two approaches of early-exit inference that are compatible with KV caching for&#10;autoregressive generation. Our analytical and empirical study shows that EE-LLM&#10;achieves great training efficiency with negligible computational overhead&#10;compared to standard LLM training, as well as outstanding inference speedup&#10;without compromising output quality. To facilitate further research and&#10;adoption, we release EE-LLM at https://github.com/pan-x-c/EE-LLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.08053" label="1909.08053">
        <attvalues>
          <attvalue for="0" value="Megatron-LM: Training Multi-Billion Parameter Language Models Using&#10;  Model Parallelism" />
          <attvalue for="1" value="  Recent work in language modeling demonstrates that training large transformer&#10;models advances the state of the art in Natural Language Processing&#10;applications. However, very large models can be quite difficult to train due to&#10;memory constraints. In this work, we present our techniques for training very&#10;large transformer models and implement a simple, efficient intra-layer model&#10;parallel approach that enables training transformer models with billions of&#10;parameters. Our approach does not require a new compiler or library changes, is&#10;orthogonal and complimentary to pipeline model parallelism, and can be fully&#10;implemented with the insertion of a few communication operations in native&#10;PyTorch. We illustrate this approach by converging transformer based models up&#10;to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the&#10;entire application with 76% scaling efficiency when compared to a strong single&#10;GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To&#10;demonstrate that large language models can further advance the state of the art&#10;(SOTA), we train an 8.3 billion parameter transformer language model similar to&#10;GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful&#10;attention to the placement of layer normalization in BERT-like models is&#10;critical to achieving increased performance as the model size grows. Using the&#10;GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA&#10;perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%)&#10;datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9%&#10;compared to SOTA accuracy of 89.4%).&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Processing (NLP) is advancing quickly in part due to an increase in available compute and dataset size. The abundance of compute and data enables training increasingly larger language models via unsupervised pretraining \cite{devlin2018bert,Radford2019GPT2}. Empirical evidence indicates that larger language models are dramatically more useful for NLP tasks such as article completion, question answering, and natural language inference \cite{ALBERT2019,T5}. By finetuning these pretrained language models on downstream natural language tasks, one can achieve state of the art results as shown in recent work \cite{devlin2018bert, ELMo, Howard2018ULMFIT, Radford2018GPT, Radford2017Sentiment, Le2016seq2seqtransfer, roberta,transformerxl,xlnet,mtdnn,ALBERT2019}. &#10;&#10;As these models become larger, they exceed the memory limit of modern processors, and require additional memory management techniques such as activation checkpointing \cite{activation_checkpointing}. Widely used optimization algorithms such as ADAM require additional memory per parameter to store momentum and other optimizer state, which reduces the size of models that can be effectively trained. Several approaches to model parallelism overcome this limit by partitioning the model such that the weights and their associated optimizer state do not need to reside concurrently on the processor. For example, GPipe \cite{GPipe} and Mesh-Tensorflow \cite{mesh_tf} provide frameworks for model parallelism of different kinds. However, they require rewriting the model, and rely on custom compilers and frameworks that are still under development.&#10;&#10;In this work, we implement a simple and efficient model parallel approach using intra-layer model-parallelism. We exploit the inherent structure in transformer based language models to make a simple model-parallel implementation that trains efficiently in PyTorch, with no custom C++ code or compiler required. This approach is orthogonal to pipeline-based model parallelism as advocated by approaches such as GPipe~\cite{GPipe}. &#10;&#10;To demonstrate the scalability of our approach, we establish a baseline by training a model of 1.2 billion parameters on a single NVIDIA V100 32GB GPU, that sustains 39 TeraFLOPs. This is 30\% of the theoretical peak FLOPS for a single GPU as configured in a DGX-2H server, and is thus a strong baseline. Scaling the model to 8.3 billion parameters on 512 GPUs with 8-way model parallelism, we achieve up to 15.1 PetaFLOPs per second sustained over the entire application. This is 76\% scaling efficiency compared to the single GPU case. Figure \ref{fig:scale_line} shows more detailed scaling results.&#10;&#10;To analyze the effect of model size scaling on accuracy, we train both left-to-right GPT-2 \cite{Radford2019GPT2} language models as well as BERT \cite{devlin2018bert} bidirectional transformers and evaluate them on several downstream tasks. We show that the existing BERT architecture results in model degradation as the size increases. We overcome this challenge by rearranging the layer normalization and residual connection in the transformer layers and show that with this change, results for the downstream tasks on development sets improve monotonically as the model size increases. In addition, we show that our models achieve test set state of the art (SOTA) results on WikiText103, cloze-style prediction accuracy on LAMBADA, and reading comprehension RACE datasets.&#10;&#10;In summary, our contributions are as follows:&#10;\begin{itemize}&#10; \item We implement a simple and efficient model parallel approach by making only a few targeted modifications to an existing PyTorch transformer implementation.&#10; \item We perform an in-depth empirical analysis of our model and data parallel technique and demonstrate up to 76\% scaling efficiency using 512 GPUs.&#10; \item We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased accuracies as the model grows.&#10; \item We demonstrate that scaling the model size results in improved accuracies for both GPT-2 (studied up to 8.3 billion parameters) and BERT (studied up to 3.9B parameters) models.&#10; \item We showcase that our models achieve state of the art results on test sets: perplexity on WikiText103 (10.8 ppl), accuracy on LAMBADA (66.5\%), and accuracy on RACE (90.9\%).&#10; \item We open source our code along with the training and evaluation pipelines at &#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.10461" label="2104.10461">
        <attvalues>
          <attvalue for="0" value="Improving the Accuracy of Early Exits in Multi-Exit Architectures via&#10;  Curriculum Learning" />
          <attvalue for="1" value="  Deploying deep learning services for time-sensitive and resource-constrained&#10;settings such as IoT using edge computing systems is a challenging task that&#10;requires dynamic adjustment of inference time. Multi-exit architectures allow&#10;deep neural networks to terminate their execution early in order to adhere to&#10;tight deadlines at the cost of accuracy. To mitigate this cost, in this paper&#10;we introduce a novel method called Multi-Exit Curriculum Learning that utilizes&#10;curriculum learning, a training strategy for neural networks that imitates&#10;human learning by sorting the training samples based on their difficulty and&#10;gradually introducing them to the network. Experiments on CIFAR-10 and&#10;CIFAR-100 datasets and various configurations of multi-exit architectures show&#10;that our method consistently improves the accuracy of early exits compared to&#10;the standard training approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.08555" label="2203.08555">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum&#10;  Learning" />
          <attvalue for="1" value="  Large multilingual pretrained language models such as mBERT and XLM-RoBERTa&#10;have been found to be surprisingly effective for cross-lingual transfer of&#10;syntactic parsing models (Wu and Dredze 2019), but only between related&#10;languages. However, source and training languages are rarely related, when&#10;parsing truly low-resource languages. To close this gap, we adopt a method from&#10;multi-task learning, which relies on automated curriculum learning, to&#10;dynamically optimize for parsing performance on outlier languages. We show that&#10;this approach is significantly better than uniform and size-proportional&#10;sampling in the zero-shot setting.&#10;" />
          <attvalue for="2" value="&#10;The field of multilingual NLP is booming \cite{10.1162/coli_r_00372}. This is due in no small part to large multilingual pretrained language models (PLMs) such as mBERT \cite{devlin-etal-2019-bert} and XLM-RoBERTa \cite{conneau-etal-2020-unsupervised}, which have been found to have surprising cross-lingual transfer capabilities in spite of receiving no cross-lingual supervision.&#10;\cite{wu-dredze-2019-beto}, for example, found mBERT to perform well in a zero-shot setting when fine-tuned for five different NLP tasks in different languages. There is, however, a sharp divide between languages that benefit from this transfer and languages that do not, and there is ample evidence that transfer works best between typologically similar languages \cite[][among others]{pires-etal-2019-multilingual,lauscher-etal-2020-zero}. This means that the majority of world languages that are truly low-resource are still left behind and inequalities in access to language technology are increasing. &#10;&#10;Large multilingual PLMs are typically fine-tuned using training data from a sample of languages that is supposed to be representative of the languages that the models are later applied to. However, this is difficult to achieve in practice, as multilingual datasets are not well balanced for typological diversity and contain a skewed distribution of typological features \cite{ponti-etal-2021-minimax}. This problem can be mitigated by using methods that sample from skewed distributions in a way that is robust to outliers.&#10;&#10;\cite{zhang2020worst} recently developed such a method. It uses curriculum learning with a worst-case-aware loss for multi-task learning. They trained their model on a subset of the GLUE benchmark \cite{wang-etal-2018-glue} and tested on outlier tasks. This led to improved zero-shot performance on these outlier tasks. This method can be applied to multilingual NLP where different languages are considered different tasks. This is what we do in this work, for the case of multilingual dependency parsing. &#10;Multilingual dependency parsing is an ideal test case for this method, as the Universal Dependency treebanks \cite{nivre-etal-2020-universal} are currently the manually annotated dataset that covers the most typological diversity \cite{ponti-etal-2021-minimax}. &#10;&#10;Our research question can be formulated as such:&#10; {Can worst-case aware automated curriculum learning improve zero-shot cross-lingual dependency parsing?}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Lingual Transfer Learning, Machine Learning, Computer Science, Linguistics, Multilingual Language Models, Artificial Intelligence, Low-Resource Language Parsing, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1911.02116" label="1911.02116">
        <attvalues>
          <attvalue for="0" value="Unsupervised Cross-lingual Representation Learning at Scale" />
          <attvalue for="1" value="  This paper shows that pretraining multilingual language models at scale leads&#10;to significant performance gains for a wide range of cross-lingual transfer&#10;tasks. We train a Transformer-based masked language model on one hundred&#10;languages, using more than two terabytes of filtered CommonCrawl data. Our&#10;model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a&#10;variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI,&#10;+13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs&#10;particularly well on low-resource languages, improving 15.7% in XNLI accuracy&#10;for Swahili and 11.4% for Urdu over previous XLM models. We also present a&#10;detailed empirical analysis of the key factors that are required to achieve&#10;these gains, including the trade-offs between (1) positive transfer and&#10;capacity dilution and (2) the performance of high and low resource languages at&#10;scale. Finally, we show, for the first time, the possibility of multilingual&#10;modeling without sacrificing per-language performance; XLM-R is very&#10;competitive with strong monolingual models on the GLUE and XNLI benchmarks. We&#10;will make our code, data and models publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.09077" label="1904.09077">
        <attvalues>
          <attvalue for="0" value="Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT" />
          <attvalue for="1" value="  Pretrained contextual representation models (Peters et al., 2018; Devlin et&#10;al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new&#10;release of BERT (Devlin, 2018) includes a model simultaneously pretrained on&#10;104 languages with impressive performance for zero-shot cross-lingual transfer&#10;on a natural language inference task. This paper explores the broader&#10;cross-lingual potential of mBERT (multilingual) as a zero shot language&#10;transfer model on 5 NLP tasks covering a total of 39 languages from various&#10;language families: NLI, document classification, NER, POS tagging, and&#10;dependency parsing. We compare mBERT with the best-published methods for&#10;zero-shot cross-lingual transfer and find mBERT competitive on each task.&#10;Additionally, we investigate the most effective strategy for utilizing mBERT in&#10;this manner, determine to what extent mBERT generalizes away from language&#10;specific features, and measure factors that influence cross-lingual transfer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01502" label="1906.01502">
        <attvalues>
          <attvalue for="0" value="How multilingual is Multilingual BERT?" />
          <attvalue for="1" value="  In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et&#10;al. (2018) as a single language model pre-trained from monolingual corpora in&#10;104 languages, is surprisingly good at zero-shot cross-lingual model transfer,&#10;in which task-specific annotations in one language are used to fine-tune the&#10;model for evaluation in another language. To understand why, we present a large&#10;number of probing experiments, showing that transfer is possible even to&#10;languages in different scripts, that transfer works best between typologically&#10;similar languages, that monolingual corpora can train models for&#10;code-switching, and that the model can find translation pairs. From these&#10;results, we can conclude that M-BERT does create multilingual representations,&#10;but that these representations exhibit systematic deficiencies affecting&#10;certain language pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.11138" label="2009.11138">
        <attvalues>
          <attvalue for="0" value="Worst-Case-Aware Curriculum Learning for Zero and Few Shot Transfer" />
          <attvalue for="1" value="  Multi-task transfer learning based on pre-trained language encoders achieves&#10;state-of-the-art performance across a range of tasks. Standard approaches&#10;implicitly assume the tasks, for which we have training data, are equally&#10;representative of the tasks we are interested in, an assumption which is often&#10;hard to justify. This paper presents a more agnostic approach to multi-task&#10;transfer learning, which uses automated curriculum learning to minimize a new&#10;family of worst-case-aware losses across tasks. Not only do these losses lead&#10;to better performance on outlier tasks; they also lead to better performance in&#10;zero-shot and few-shot transfer settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.10643" label="2004.10643">
        <attvalues>
          <attvalue for="0" value="Universal Dependencies v2: An Evergrowing Multilingual Treebank&#10;  Collection" />
          <attvalue for="1" value="  Universal Dependencies is an open community effort to create&#10;cross-linguistically consistent treebank annotation for many languages within a&#10;dependency-based lexicalist framework. The annotation consists in a&#10;linguistically motivated word segmentation; a morphological layer comprising&#10;lemmas, universal part-of-speech tags, and standardized morphological features;&#10;and a syntactic layer focusing on syntactic relations between predicates,&#10;arguments and modifiers. In this paper, we describe version 2 of the guidelines&#10;(UD v2), discuss the major changes from UD v1 to UD v2, and give an overview of&#10;the currently available treebanks for 90 languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01787" label="1906.01787">
        <attvalues>
          <attvalue for="0" value="Learning Deep Transformer Models for Machine Translation" />
          <attvalue for="1" value="  Transformer is the state-of-the-art model in recent machine translation&#10;evaluations. Two strands of research are promising to improve models of this&#10;kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de&#10;facto standard for the development of the Transformer system, and the other&#10;uses deeper language representation but faces the difficulty arising from&#10;learning deep networks. Here, we continue the line of research on the latter.&#10;We claim that a truly deep Transformer model can surpass the Transformer-Big&#10;counterpart by 1) proper use of layer normalization and 2) a novel way of&#10;passing the combination of previous layers to the next. On WMT'16 English-&#10;German, NIST OpenMT'12 Chinese-English and larger WMT'18 Chinese-English tasks,&#10;our deep system (30/25-layer encoder) outperforms the shallow&#10;Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As&#10;another bonus, the deep model is 1.6X smaller in size and 3X faster in training&#10;than Transformer-Big.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.02155" label="1803.02155">
        <attvalues>
          <attvalue for="0" value="Self-Attention with Relative Position Representations" />
          <attvalue for="1" value="  Relying entirely on an attention mechanism, the Transformer introduced by&#10;Vaswani et al. (2017) achieves state-of-the-art results for machine&#10;translation. In contrast to recurrent and convolutional neural networks, it&#10;does not explicitly model relative or absolute position information in its&#10;structure. Instead, it requires adding representations of absolute positions to&#10;its inputs. In this work we present an alternative approach, extending the&#10;self-attention mechanism to efficiently consider representations of the&#10;relative positions, or distances between sequence elements. On the WMT 2014&#10;English-to-German and English-to-French translation tasks, this approach yields&#10;improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations,&#10;respectively. Notably, we observe that combining relative and absolute position&#10;representations yields no further improvement in translation quality. We&#10;describe an efficient implementation of our method and cast it as an instance&#10;of relation-aware self-attention mechanisms that can generalize to arbitrary&#10;graph-labeled inputs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1511.06709" label="1511.06709">
        <attvalues>
          <attvalue for="0" value="Improving Neural Machine Translation Models with Monolingual Data" />
          <attvalue for="1" value="  Neural Machine Translation (NMT) has obtained state-of-the art performance&#10;for several language pairs, while only using parallel data for training.&#10;Target-side monolingual data plays an important role in boosting fluency for&#10;phrase-based statistical machine translation, and we investigate the use of&#10;monolingual data for NMT. In contrast to previous work, which combines NMT&#10;models with separately trained language models, we note that encoder-decoder&#10;NMT architectures already have the capacity to learn the same information as a&#10;language model, and we explore strategies to train with monolingual data&#10;without changing the neural network architecture. By pairing monolingual&#10;training data with an automatic back-translation, we can treat it as additional&#10;parallel training data, and we obtain substantial improvements on the WMT 15&#10;task English&lt;-&gt;German (+2.8-3.7 BLEU), and for the low-resourced IWSLT 14 task&#10;Turkish-&gt;English (+2.1-3.4 BLEU), obtaining new state-of-the-art results. We&#10;also show that fine-tuning on in-domain monolingual and parallel data gives&#10;substantial improvements for the IWSLT 15 task English-&gt;German.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.13873" label="2406.13873">
        <attvalues>
          <attvalue for="0" value="A Pure Transformer Pretraining Framework on Text-attributed Graphs" />
          <attvalue for="1" value="  Pretraining plays a pivotal role in acquiring generalized knowledge from&#10;large-scale data, achieving remarkable successes as evidenced by large models&#10;in CV and NLP. However, progress in the graph domain remains limited due to&#10;fundamental challenges such as feature heterogeneity and structural&#10;heterogeneity. Recently, increasing efforts have been made to enhance node&#10;feature quality with Large Language Models (LLMs) on text-attributed graphs&#10;(TAGs), demonstrating superiority to traditional bag-of-words or word2vec&#10;techniques. These high-quality node features reduce the previously critical&#10;role of graph structure, resulting in a modest performance gap between Graph&#10;Neural Networks (GNNs) and structure-agnostic Multi-Layer Perceptrons (MLPs).&#10;Motivated by this, we introduce a feature-centric pretraining perspective by&#10;treating graph structure as a prior and leveraging the rich, unified feature&#10;space to learn refined interaction patterns that generalizes across graphs. Our&#10;framework, Graph Sequence Pretraining with Transformer (GSPT), samples node&#10;contexts through random walks and employs masked feature reconstruction to&#10;capture pairwise proximity in the LLM-unified feature space using a standard&#10;Transformer. By utilizing unified text representations rather than varying&#10;structures, our framework achieves significantly better transferability among&#10;graphs within the same domain. GSPT can be easily adapted to both node&#10;classification and link prediction, demonstrating promising empirical success&#10;on various datasets.&#10;" />
          <attvalue for="2" value="&#10;Transfer learning has witnessed remarkable success in recent years, particularly exemplified by the advancements in foundation models for Natural Language Processing (NLP) ~\cite{gpt4, touvron2023llama} and Computer Vision (CV)~\cite{kirillov2023segment, CLIP}. These methods typically leverage self-supervised pretraining on large-scale datasets to acquire broad, generalized knowledge, which is subsequently adapted to specific tasks and datasets through fine-tuning or in-context learning. However, the graph domain predominantly adheres to a 'one-model, one-dataset' approach, where models are tailored specifically to individual datasets and tasks, deviating from the prevailing trend of 'one model serves all'.&#10;&#10;The pursuit of cross-dataset transfer learning on graphs faces unique challenges. The immediate one is feature heterogeneity, i.e., the inherent mismatch in feature spaces among different datasets, as graphs denoting different data types often have features with varying dimensions and semantic meanings~\cite{oneforall, mao2024graph}. Previous methods circumvent this by ignoring the features and only transferring knowledge from the structural side~\cite{qiu2020gcc, davies2023its}, or constraining the applications in vertical domains where node/edge features are naturally aligned~\cite{Hu2019StrategiesFP, Xia2023MoleBERTRP}. Such approaches, while somewhat effective, suffer from performance loss or restricted applicability~\cite{Liu2021GraphSL}. Another challenge is structural heterogeneity, which arises from the vastly different structural patterns across various graphs, leading to out-of-distribution scenarios and potential negative transfer~\cite{Wang2024SubgraphPT}. A typical example is the varying degrees of homophily across graphs~\cite{mao2024demystifying}. Together, these challenges pose substantial obstacles to the development of a flexible, generalizable model capable of effective pretraining and knowledge transfer across diverse downstream datasets.&#10;&#10;Recent efforts ~\cite{Chen2023ExploringTP, oneforall, tan2024walklm, Chien2021NodeFE, huang2023prodigy} have been made to tackle feature heterogeneity by leveraging large language models (LLMs) to unify the feature spaces of text-attributed graphs (TAGs). They replace traditional shallow features like word2vec and tf-idf with language model-enhanced features and have demonstrated impressive empirical success, represented by the improved performance on graph-related tasks and the reduced gap between purely feature-based approaches (e.g., an MLP) and graph-tailored models (e.g., Graph Neural Networks)~\cite{Chen2023ExploringTP}. This trend motivates us to consider a conceptual shift from an era predominantly focused on graph structure to a feature-centric paradigm, suggesting the potential to enhance knowledge transfer by effectively leveraging LLM-based unified features.&#10;&#10;Following this perspective, the key principle is to identify a fundamental unit that effectively encodes graph information and generalizes across the LLM-unified feature space. In this study, we propose to learn a pairwise function that models the proximity of node pairs, thereby capturing the interactions embedded in the graph structure. Inspired by works in CV, NLP and the graph domain~\cite{He2021MaskedAA, Devlin2019BERTPO, tang2021graph}, we adopt a reconstruction-based objective as the pretext task due to its efficacy in enforcing a robust understanding of dependencies within various data schemes. Specifically, given a (masked) center node and its context, we aim to reconstruct the feature of the center node using its context. &#10;This approach presents two key challenges. The first challenge is how to construct the appropriate context for graphs? Unlike sequential data such as language or grid-based data like images, graphs represent non-Euclidean structures, making it non-trivial to create a context for reconstruction. The second challenge is how to reconstruct the center node based on its context? Specifically, it involves identifying an appropriate architecture that can accurately interpret the context nodes and a loss function that effectively guides the learning process of the model. &#10;&#10;To address the first challenge, we propose using node sequences generated by random walks as contexts, where the entire sequence forms the receptive field, and the order of nodes preserves proximity information~\cite{Grover2016node2vecSF, Perozzi2014DeepWalkOL}. By doing so, the original graph topology serves as a prior to retrieve relevant context nodes, facilitating effective proximity modeling for accurate reconstruction of masked features.&#10;For the second challenge, we employ a standard Transformer architecture due to its flexibility in modeling sequences with self-attention and its proven transferability across domains like CV and NLP~\cite{kirillov2023segment, Devlin2019BERTPO}. To handle the multi-dimensional and continuous nature of LM-produced features, we adopt a cosine similarity-based objective to measure reconstruction error instead of cross-entropy~\cite{hou2022graphmae}.&#10;&#10;Putting it all together, we propose Graph Sequence Pretraining with Transformer (GSPT), where a standard Transformer is used alongside a feature reconstruction objective to learn a unified model for node representations. &#10;Our framework highlights the use of text-based representations rather than the diverse structures of different graphs, leading to enhanced transferability and reduced risk of negative transfer caused by structural shifts. &#10;To evaluate the effectiveness of our method, we perform self-supervised training on the largest graph available ogbn-papers100M~\cite{hu2020open}, and apply the pretrained framework on various downstream datasets. Experimental results reveal that our GSPT excels in in-context node classification and link prediction, showcasing effective knowledge transfer from pretraining data to downstream tasks. Furthermore, we observe a notable trend of improvement with increased pretraining data, highlighting the potential of the proposed framework.&#10;Our findings deepen the understanding of the LLM-unified feature space in graph data and provide valuable insights into the development of a versatile and generalizable graph foundation model. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Pretraining Techniques, Graph Theory, Mathematics, Graph Neural Networks, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2303.12712" label="2303.12712">
        <attvalues>
          <attvalue for="0" value="Sparks of Artificial General Intelligence: Early experiments with GPT-4" />
          <attvalue for="1" value="  Artificial intelligence (AI) researchers have been developing and refining&#10;large language models (LLMs) that exhibit remarkable capabilities across a&#10;variety of domains and tasks, challenging our understanding of learning and&#10;cognition. The latest model developed by OpenAI, GPT-4, was trained using an&#10;unprecedented scale of compute and data. In this paper, we report on our&#10;investigation of an early version of GPT-4, when it was still in active&#10;development by OpenAI. We contend that (this early version of) GPT-4 is part of&#10;a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that&#10;exhibit more general intelligence than previous AI models. We discuss the&#10;rising capabilities and implications of these models. We demonstrate that,&#10;beyond its mastery of language, GPT-4 can solve novel and difficult tasks that&#10;span mathematics, coding, vision, medicine, law, psychology and more, without&#10;needing any special prompting. Moreover, in all of these tasks, GPT-4's&#10;performance is strikingly close to human-level performance, and often vastly&#10;surpasses prior models such as ChatGPT. Given the breadth and depth of GPT-4's&#10;capabilities, we believe that it could reasonably be viewed as an early (yet&#10;still incomplete) version of an artificial general intelligence (AGI) system.&#10;In our exploration of GPT-4, we put special emphasis on discovering its&#10;limitations, and we discuss the challenges ahead for advancing towards deeper&#10;and more comprehensive versions of AGI, including the possible need for&#10;pursuing a new paradigm that moves beyond next-word prediction. We conclude&#10;with reflections on societal influences of the recent technological leap and&#10;future research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.02643" label="2304.02643">
        <attvalues>
          <attvalue for="0" value="Segment Anything" />
          <attvalue for="1" value="  We introduce the Segment Anything (SA) project: a new task, model, and&#10;dataset for image segmentation. Using our efficient model in a data collection&#10;loop, we built the largest segmentation dataset to date (by far), with over 1&#10;billion masks on 11M licensed and privacy respecting images. The model is&#10;designed and trained to be promptable, so it can transfer zero-shot to new&#10;image distributions and tasks. We evaluate its capabilities on numerous tasks&#10;and find that its zero-shot performance is impressive -- often competitive with&#10;or even superior to prior fully supervised results. We are releasing the&#10;Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and&#10;11M images at https://segment-anything.com to foster research into foundation&#10;models for computer vision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.09963" label="2006.09963">
        <attvalues>
          <attvalue for="0" value="GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training" />
          <attvalue for="1" value="  Graph representation learning has emerged as a powerful technique for&#10;addressing real-world problems. Various downstream graph learning tasks have&#10;benefited from its recent developments, such as node classification, similarity&#10;search, and graph classification. However, prior arts on graph representation&#10;learning focus on domain specific problems and train a dedicated model for each&#10;graph dataset, which is usually non-transferable to out-of-domain data.&#10;Inspired by the recent advances in pre-training from natural language&#10;processing and computer vision, we design Graph Contrastive Coding (GCC) -- a&#10;self-supervised graph neural network pre-training framework -- to capture the&#10;universal network topological properties across multiple networks. We design&#10;GCC's pre-training task as subgraph instance discrimination in and across&#10;networks and leverage contrastive learning to empower graph neural networks to&#10;learn the intrinsic and transferable structural representations. We conduct&#10;extensive experiments on three graph learning tasks and ten graph datasets. The&#10;results show that GCC pre-trained on a collection of diverse datasets can&#10;achieve competitive or better performance to its task-specific and&#10;trained-from-scratch counterparts. This suggests that the pre-training and&#10;fine-tuning paradigm presents great potential for graph representation&#10;learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.00111" label="2103.00111">
        <attvalues>
          <attvalue for="0" value="Graph Self-Supervised Learning: A Survey" />
          <attvalue for="1" value="  Deep learning on graphs has attracted significant interests recently.&#10;However, most of the works have focused on (semi-) supervised learning,&#10;resulting in shortcomings including heavy label reliance, poor generalization,&#10;and weak robustness. To address these issues, self-supervised learning (SSL),&#10;which extracts informative knowledge through well-designed pretext tasks&#10;without relying on manual labels, has become a promising and trending learning&#10;paradigm for graph data. Different from SSL on other domains like computer&#10;vision and natural language processing, SSL on graphs has an exclusive&#10;background, design ideas, and taxonomies. Under the umbrella of graph&#10;self-supervised learning, we present a timely and comprehensive review of the&#10;existing approaches which employ SSL techniques for graph data. We construct a&#10;unified framework that mathematically formalizes the paradigm of graph SSL.&#10;According to the objectives of pretext tasks, we divide these approaches into&#10;four categories: generation-based, auxiliary property-based, contrast-based,&#10;and hybrid approaches. We further describe the applications of graph SSL across&#10;various research fields and summarize the commonly used datasets, evaluation&#10;benchmark, performance comparison and open-source codes of graph SSL. Finally,&#10;we discuss the remaining challenges and potential future directions in this&#10;research field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.08907" label="2402.08907">
        <attvalues>
          <attvalue for="0" value="Subgraph Pooling: Tackling Negative Transfer on Graphs" />
          <attvalue for="1" value="  Transfer learning aims to enhance performance on a target task by using&#10;knowledge from related tasks. However, when the source and target tasks are not&#10;closely aligned, it can lead to reduced performance, known as negative&#10;transfer. Unlike in image or text data, we find that negative transfer could&#10;commonly occur in graph-structured data, even when source and target graphs&#10;have semantic similarities. Specifically, we identify that structural&#10;differences significantly amplify the dissimilarities in the node embeddings&#10;across graphs. To mitigate this, we bring a new insight in this paper: for&#10;semantically similar graphs, although structural differences lead to&#10;significant distribution shift in node embeddings, their impact on subgraph&#10;embeddings could be marginal. Building on this insight, we introduce Subgraph&#10;Pooling (SP) by aggregating nodes sampled from a k-hop neighborhood and&#10;Subgraph Pooling++ (SP++) by a random walk, to mitigate the impact of graph&#10;structural differences on knowledge transfer. We theoretically analyze the role&#10;of SP in reducing graph discrepancy and conduct extensive experiments to&#10;evaluate its superiority under various settings. The proposed SP methods are&#10;effective yet elegant, which can be easily applied on top of any backbone Graph&#10;Neural Networks (GNNs). Our code and data are available at:&#10;https://github.com/Zehong-Wang/Subgraph-Pooling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.01323" label="2306.01323">
        <attvalues>
          <attvalue for="0" value="Demystifying Structural Disparity in Graph Neural Networks: Can One Size&#10;  Fit All?" />
          <attvalue for="1" value="  Recent studies on Graph Neural Networks(GNNs) provide both empirical and&#10;theoretical evidence supporting their effectiveness in capturing structural&#10;patterns on both homophilic and certain heterophilic graphs. Notably, most&#10;real-world homophilic and heterophilic graphs are comprised of a mixture of&#10;nodes in both homophilic and heterophilic structural patterns, exhibiting a&#10;structural disparity. However, the analysis of GNN performance with respect to&#10;nodes exhibiting different structural patterns, e.g., homophilic nodes in&#10;heterophilic graphs, remains rather limited. In the present study, we provide&#10;evidence that Graph Neural Networks(GNNs) on node classification typically&#10;perform admirably on homophilic nodes within homophilic graphs and heterophilic&#10;nodes within heterophilic graphs while struggling on the opposite node set,&#10;exhibiting a performance disparity. We theoretically and empirically identify&#10;effects of GNNs on testing nodes exhibiting distinct structural patterns. We&#10;then propose a rigorous, non-i.i.d PAC-Bayesian generalization bound for GNNs,&#10;revealing reasons for the performance disparity, namely the aggregated feature&#10;distance and homophily ratio difference between training and testing nodes.&#10;Furthermore, we demonstrate the practical implications of our new findings via&#10;(1) elucidating the effectiveness of deeper GNNs; and (2) revealing an&#10;over-looked distribution shift factor on graph out-of-distribution problem and&#10;proposing a new scenario accordingly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.03393" label="2307.03393">
        <attvalues>
          <attvalue for="0" value="Exploring the Potential of Large Language Models (LLMs) in Learning on&#10;  Graphs" />
          <attvalue for="1" value="  Learning on Graphs has attracted immense attention due to its wide real-world&#10;applications. The most popular pipeline for learning on graphs with textual&#10;node attributes primarily relies on Graph Neural Networks (GNNs), and utilizes&#10;shallow text embedding as initial node representations, which has limitations&#10;in general knowledge and profound semantic understanding. In recent years,&#10;Large Language Models (LLMs) have been proven to possess extensive common&#10;knowledge and powerful semantic comprehension abilities that have&#10;revolutionized existing workflows to handle text data. In this paper, we aim to&#10;explore the potential of LLMs in graph machine learning, especially the node&#10;classification task, and investigate two possible pipelines: LLMs-as-Enhancers&#10;and LLMs-as-Predictors. The former leverages LLMs to enhance nodes' text&#10;attributes with their massive knowledge and then generate predictions through&#10;GNNs. The latter attempts to directly employ LLMs as standalone predictors. We&#10;conduct comprehensive and systematical studies on these two pipelines under&#10;various settings. From comprehensive empirical results, we make original&#10;observations and find new insights that open new possibilities and suggest&#10;promising directions to leverage LLMs for learning on graphs. Our codes and&#10;datasets are available at https://github.com/CurryTang/Graph-LLM.&#10;" />
          <attvalue for="2" value="&#10;&#10;Graphs are ubiquitous in various disciplines &#10;and applications, encompassing a wide range of &#10;real-world scenarios~\cite{Xia2021GraphLA}. Many of &#10;these graphs have nodes that are associated &#10;with text attributes, resulting in the &#10;emergence of text-attributed graphs, such as citation graphs~\cite{hu2020open, Sen_Namata_Bilgic_Getoor_Galligher_Eliassi-Rad_2008} and product graphs~\cite{Chiang2019ClusterGCNAE}. For example, in the Ogbn-products dataset~\cite{hu2020open}, each node &#10;represents a product, and its corresponding &#10;textual description is treated as the node's attribute. &#10;These graphs have seen widespread use across &#10;a myriad of domains, from social network analysis~\cite{social_network}, information retrieval~\cite{Zhu2021TextGNNIT}, &#10;to a diverse range of natural language processing tasks~\cite{liu-etal-2020-fine, Yao2018GraphCN}. &#10;&#10;Given the prevalence of text-attributed graphs (TAGs), we aim to explore how to effectively handle these graphs, with a focus on the node classification task. Intuitively, TAGs provide both node attribute and graph structural information. Thus, it is important to effectively capture both while modeling their interrelated correlation. Graph Neural Networks (GNNs)~\cite{ma2021deep} have emerged as the de facto technique for handling graph-structured data, often leveraging a message-passing paradigm to effectively capture the graph structure. To encode textual information, conventional pipelines typically make use of non-contextualized shallow embeddings e.g., Bag-of-Words~\cite{harris1954distributional} and Word2Vec~\cite{mikolov2013efficient} embeddings, as seen in the common graph benchmark datasets~\cite{hu2020open, Sen_Namata_Bilgic_Getoor_Galligher_Eliassi-Rad_2008}, where GNNs are subsequently employed to process these embeddings. Recent studies demonstrate that these non-contextualized shallow embeddings suffer from some limitations, such as the inability to capture polysemous words~\cite{Qiu2020PretrainedMF} and deficiency in semantic information~\cite{miaschi-dellorletta-2020-contextual, ethayarajh-2019-contextual}, which may lead to sub-optimal performance on downstream tasks. &#10;&#10;Compared to these non-contextualized shallow textual embeddings, large language models (LLMs) present massive context-aware knowledge and superior semantic comprehension capability through the process of pre-training on large-scale text corpora~\cite{Petroni2019LanguageMA, ethayarajh-2019-contextual}. &#10;This knowledge achieved from pre-training has led to a surge of revolutions for downstream NLP tasks~\cite{Zhao2023ASO}.&#10;Exemplars such as ChatGPT and GPT4~\cite{OpenAI2023GPT4TR}, equipped with hundreds of billions of parameters, exhibit superior performance~\cite{Bubeck2023SparksOA} on numerous text-related tasks from various domains. Considering the exceptional ability of these LLMs to process and understand textual data, a pertinent question arises: (1) Can we leverage the knowledge of LLMs to compensate for the deficiency of contextualized knowledge and semantic comprehension inherent in the conventional GNN pipelines?&#10;In addition to the knowledge learned via pre-training, recent studies suggest that LLMs present preliminary success on tasks with implicit graph structures such as recommendation \cite{liu2023chatgpt_rec, Gao2023ChatRECTI}, ranking \cite{Ji2023ExploringCA}, and multi-hop reasoning \cite{creswell2023selectioninference}, in which LLMs are adopted to make the final predictions. Given such success, we further question: (2) Can LLMs, beyond merely integrating with GNNs, independently perform predictive tasks with explicit graph structures? In this paper, we aim to embark upon a preliminary investigation of these two questions by undertaking a series of extensive empirical analyses. Particularly, the key challenge is how to design an LLM-compatible pipeline for graph learning tasks. Consequently, we explore two potential pipelines to incorporate LLMs: (1) LLMs-as-Enhancers: LLMs are adopted to enhance the textual information; subsequently, GNNs utilize refined textual data to generate predictions. (2) LLMs-as-Predictors: LLMs are adapted to generate the final predictions, where structural and attribute information is present completely through natural languages. &#10;&#10;In this work, we embrace the challenges and opportunities to study the utilization of LLMs in graph-related problems and aim to deepen our understanding of the potential of LLMs on graph machine learning, with a focus on the node classification task. First, we aim to investigate how LLMs can enhance GNNs by leveraging their extensive knowledge and semantic comprehension capability. It is evident that different types of LLMs possess varying levels of capability, and more powerful models often come with more usage restrictions~\cite{sun2022black, Zhao2023ASO, Qiu2020PretrainedMF}. Therefore, we strive to design different strategies tailored to different types of models, and better leverage their capabilities within the constraints of these usage limitations. Second, we want to explore how LLMs can be adapted to explicit graph structures as a predictor. A principal challenge lies in crafting a prompt that enables the LLMs to effectively use structural and attribute information. To address this challenge, we attempt to explore what information can assist LLMs in better understanding and utilizing graph structures. Through these investigations, we make some insightful observations and gain a better understanding of the capabilities of LLMs in graph machine learning. &#10;&#10;Contributions. Our contributions are summarized as follows:&#10;\begin{compactenum}[1.]&#10;\item We explore two pipelines that incorporate LLMs to handle TAGs: LLMs-as-Enhancers and LLMs-as-Predictors. The first pipeline treats the LLMs as attribute enhancers, seamlessly integrating them with GNNs. The second pipeline directly employs the LLMs to generate predictions.&#10;&#10;\item For LLMs-as-Enhancers, we introduce two strategies to enhance text attributes via LLMs. We further conduct a series of experiments to compare the effectiveness of these enhancements. &#10;\item For LLMs-as-Predictors, we design a series of experiments to explore LLMs' capability in utilizing structural and attribute information. From empirical results, we summarize some original observations and provide new insights. &#10;&#10;\end{compactenum} &#10;&#10;Key Insights. Through comprehensive empirical evaluations, we find the following key insights: &#10;\begin{compactenum}[1.]&#10;\item For LLMs-as-Enhancers, using deep sentence embedding &#10;models to generate embeddings for node &#10;attributes show both effectiveness and efficiency.&#10;&#10;\item For LLMs-as-Enhancers, utilizing LLMs to augment node attributes at the text level also leads to improvements in downstream performance. &#10;&#10;\item For LLMs-as-Predictors, LLMs present preliminary effectiveness but we should be careful about their inaccurate predictions and the potential test data leakage problem. &#10;&#10;\item LLMs demonstrate the potential to serve as good annotators for labeling nodes, as a decent portion of their annotations is accurate. &#10;\end{compactenum}&#10;&#10;Organization. The remaining of this paper is organized as follows. Section~\ref{sec:prl} introduces necessary preliminary knowledge and notations used in this paper. Section~\ref{sec: pipeline} introduces two pipelines to leverage LLMs under the task of node classification. Section~\ref{sec:enh} explores the first pipeline, LLMs-as-Enhancers, which adopts LLMs to enhance text attributes. Section~\ref{sec:pred} details the second pipeline, LLMs-as-Predictors, exploring the potential for directly applying LLMs to solve graph learning problems as a predictor. Section~\ref{sec:rw} discusses works relevant to the applications of LLMs in the graph domain. Section~\ref{sec:fut} summarizes our insights and discusses the limitations of our study and the potential directions of LLMs in the graph domain. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.06377" label="2111.06377">
        <attvalues>
          <attvalue for="0" value="Masked Autoencoders Are Scalable Vision Learners" />
          <attvalue for="1" value="  This paper shows that masked autoencoders (MAE) are scalable self-supervised&#10;learners for computer vision. Our MAE approach is simple: we mask random&#10;patches of the input image and reconstruct the missing pixels. It is based on&#10;two core designs. First, we develop an asymmetric encoder-decoder architecture,&#10;with an encoder that operates only on the visible subset of patches (without&#10;mask tokens), along with a lightweight decoder that reconstructs the original&#10;image from the latent representation and mask tokens. Second, we find that&#10;masking a high proportion of the input image, e.g., 75%, yields a nontrivial&#10;and meaningful self-supervisory task. Coupling these two designs enables us to&#10;train large models efficiently and effectively: we accelerate training (by 3x&#10;or more) and improve accuracy. Our scalable approach allows for learning&#10;high-capacity models that generalize well: e.g., a vanilla ViT-Huge model&#10;achieves the best accuracy (87.8%) among methods that use only ImageNet-1K&#10;data. Transfer performance in downstream tasks outperforms supervised&#10;pre-training and shows promising scaling behavior.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1607.00653" label="1607.00653">
        <attvalues>
          <attvalue for="0" value="node2vec: Scalable Feature Learning for Networks" />
          <attvalue for="1" value="  Prediction tasks over nodes and edges in networks require careful effort in&#10;engineering features used by learning algorithms. Recent research in the&#10;broader field of representation learning has led to significant progress in&#10;automating prediction by learning the features themselves. However, present&#10;feature learning approaches are not expressive enough to capture the diversity&#10;of connectivity patterns observed in networks. Here we propose node2vec, an&#10;algorithmic framework for learning continuous feature representations for nodes&#10;in networks. In node2vec, we learn a mapping of nodes to a low-dimensional&#10;space of features that maximizes the likelihood of preserving network&#10;neighborhoods of nodes. We define a flexible notion of a node's network&#10;neighborhood and design a biased random walk procedure, which efficiently&#10;explores diverse neighborhoods. Our algorithm generalizes prior work which is&#10;based on rigid notions of network neighborhoods, and we argue that the added&#10;flexibility in exploring neighborhoods is the key to learning richer&#10;representations. We demonstrate the efficacy of node2vec over existing&#10;state-of-the-art techniques on multi-label classification and link prediction&#10;in several real-world networks from diverse domains. Taken together, our work&#10;represents a new way for efficiently learning state-of-the-art task-independent&#10;representations in complex networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.10803" label="2205.10803">
        <attvalues>
          <attvalue for="0" value="GraphMAE: Self-Supervised Masked Graph Autoencoders" />
          <attvalue for="1" value="  Self-supervised learning (SSL) has been extensively explored in recent years.&#10;Particularly, generative SSL has seen emerging success in natural language&#10;processing and other AI fields, such as the wide adoption of BERT and GPT.&#10;Despite this, contrastive learning-which heavily relies on structural data&#10;augmentation and complicated training strategies-has been the dominant approach&#10;in graph SSL, while the progress of generative SSL on graphs, especially graph&#10;autoencoders (GAEs), has thus far not reached the potential as promised in&#10;other fields. In this paper, we identify and examine the issues that negatively&#10;impact the development of GAEs, including their reconstruction objective,&#10;training robustness, and error metric. We present a masked graph autoencoder&#10;GraphMAE that mitigates these issues for generative self-supervised graph&#10;pretraining. Instead of reconstructing graph structures, we propose to focus on&#10;feature reconstruction with both a masking strategy and scaled cosine error&#10;that benefit the robust training of GraphMAE. We conduct extensive experiments&#10;on 21 public datasets for three different graph learning tasks. The results&#10;manifest that GraphMAE-a simple graph autoencoder with careful designs-can&#10;consistently generate outperformance over both contrastive and generative&#10;state-of-the-art baselines. This study provides an understanding of graph&#10;autoencoders and demonstrates the potential of generative self-supervised&#10;pre-training on graphs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.09135" label="2404.09135">
        <attvalues>
          <attvalue for="0" value="Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions" />
          <attvalue for="1" value="  Natural Language Processing (NLP) is witnessing a remarkable breakthrough&#10;driven by the success of Large Language Models (LLMs). LLMs have gained&#10;significant attention across academia and industry for their versatile&#10;applications in text generation, question answering, and text summarization. As&#10;the landscape of NLP evolves with an increasing number of domain-specific LLMs&#10;employing diverse techniques and trained on various corpus, evaluating&#10;performance of these models becomes paramount. To quantify the performance,&#10;it's crucial to have a comprehensive grasp of existing metrics. Among the&#10;evaluation, metrics which quantifying the performance of LLMs play a pivotal&#10;role. This paper offers a comprehensive exploration of LLM evaluation from a&#10;metrics perspective, providing insights into the selection and interpretation&#10;of metrics currently in use. Our main goal is to elucidate their mathematical&#10;formulations and statistical interpretations. We shed light on the application&#10;of these metrics using recent Biomedical LLMs. Additionally, we offer a&#10;succinct comparison of these metrics, aiding researchers in selecting&#10;appropriate metrics for diverse tasks. The overarching goal is to furnish&#10;researchers with a pragmatic guide for effective LLM evaluation and metric&#10;selection, thereby advancing the understanding and application of these large&#10;language models.&#10;" />
          <attvalue for="2" value="&#10;&#10;ChatGPT, also namely GPT3.5, \cite{brown2020language,wang2022self} has demonstrated its remarkable ability to generate coherent sequences of words and engage in conversational interactions. As the popularity of ChatGPT surged, quantities of large language models (LLMs) emerged rapidly. Researchers not only explore distinct model architectures and propose various fine-tuning methods to enhance LLM capabilities but also investigate ways to tailor LLMs to specific domains.&#10;&#10;LLMs present a significant opportunity for tasks such as generating scientific texts, answering questions, and extracting core information from articles. For instance, amidst the daily influx of over 3000 new articles in peer-reviewed journals \cite{sayers2023database}, LLMs can swiftly extract key insights, aiding readers in navigating through vast amounts of medical literature. Furthermore, LLMs can potentially analyze symptoms described by patients to suggest diagnoses and treatment options, thus alleviating physicians' workload and improving patient care, which is a useful application in medicine. These applications underscore two primary functionalities of LLMs: information retrieval and text generation. However, LLMs can also contribute to various other aspects of medical research and applications. Due to these advantages, today's LLMs are attracting increasing attention across multiple fields. With the surge of novel LLMs, reviewing and evaluating existing LLMs is of great importance. &#10;Among evaluations, the intrinsic statistical interpretations are frequently neglected in current research and reviews.&#10;&#10;The proliferation of LLMs has prompted the emergence of reviews aimed at providing insights into their development and potential applications. For instance, \cite{naveed2023comprehensive} reviewed LLMs from 2019, starting with T5, up to the latest releases in 2023, offering comprehensive references and comparisons. \cite{wang2023pre} highlighted the promising applications of LLMs in addressing biomedical questions, while \cite{chen2023large} focused on evaluating biomedical LLMs with respect to benchmarks and summarization capabilities. While existing reviews generally discuss LLM structures and applications, evaluating LLMs is crucial for guiding their development and deployment but understudied. Evaluation encompasses various aspects, including downstream tasks, criteria, benchmark datasets, and metrics. Although \cite{chang2023survey} surveyed LLM evaluation, comprehensive summarizing the metrics remains scarce. This work aims to fill this gap by providing a survey of contemporary LLM evaluation metrics, along with mathematical formulations and statistical explanations and practical guidance for implementation using open-source libraries. Our paper shed lights on the bridge between the existing LLM evaluations and statistics by exploring the statistical interpretations of the metrics. Additionally, we showcase how these metrics have been utilized in conjunction with published biomedical LLMs through illustrative examples. Our main contributions are four-folds:&#10;\begin{itemize}&#10; \item Summarizing and categorizing the metrics for LLM evaluation into three distinct types;&#10; \item Providing mathematical formulations and statistical interpretations for each metric, along with a comparative analysis to serve as a guide for LLM researchers;&#10; \item Identifying repositories containing the discussed metrics;&#10; \item Showcasing how these metrics and baseline datasets are applied in the evaluation of recently developed biomedical LLMs, facilitating further studies to keep alignment with previous models.&#10;\end{itemize}&#10;&#10;The article is structured as follows: Section 2 offers a concise overview of LLM evaluation criteria. Section 3 details the most utilized metrics in LLM evaluations, including their mathematical expressions and statistical interpretations, alongside a directory of repositories for implementing these metrics. Section 4 showcases the application of these metrics using biomedical LLMs as case studies, including the baseline datasets employed for evaluation. Section 5 discusses the pros and cons of the existing widely used metrics and stresses two major common problems that are often ignored: the imperfect gold standard issue and absent of statistical inference method. The paper concludes with Section 6, summarizing our findings and the limitations for this paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Mathematics, Evaluation Metrics, Statistics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2307.06435" label="2307.06435">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Overview of Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have recently demonstrated remarkable&#10;capabilities in natural language processing tasks and beyond. This success of&#10;LLMs has led to a large influx of research contributions in this direction.&#10;These works encompass diverse topics such as architectural innovations, better&#10;training strategies, context length improvements, fine-tuning, multi-modal&#10;LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid&#10;development of techniques and regular breakthroughs in LLM research, it has&#10;become considerably challenging to perceive the bigger picture of the advances&#10;in this direction. Considering the rapidly emerging plethora of literature on&#10;LLMs, it is imperative that the research community is able to benefit from a&#10;concise yet comprehensive overview of the recent developments in this field.&#10;This article provides an overview of the existing literature on a broad range&#10;of LLM-related concepts. Our self-contained comprehensive overview of LLMs&#10;discusses relevant background concepts along with covering the advanced topics&#10;at the frontier of research in LLMs. This review article is intended to not&#10;only provide a systematic survey but also a quick comprehensive reference for&#10;the researchers and practitioners to draw insights from extensive informative&#10;summaries of the existing works to advance the LLM research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16326" label="2305.16326">
        <attvalues>
          <attvalue for="0" value="Large language models in biomedical natural language processing:&#10;  benchmarks, baselines, and recommendations" />
          <attvalue for="1" value="  Biomedical literature is growing rapidly, making it challenging to curate and&#10;extract knowledge manually. Biomedical natural language processing (BioNLP)&#10;techniques that can automatically extract information from biomedical&#10;literature help alleviate this burden. Recently, large Language Models (LLMs),&#10;such as GPT-3 and GPT-4, have gained significant attention for their impressive&#10;performance. However, their effectiveness in BioNLP tasks and impact on method&#10;development and downstream users remain understudied. This pilot study (1)&#10;establishes the baseline performance of GPT-3 and GPT-4 at both zero-shot and&#10;one-shot settings in eight BioNLP datasets across four applications: named&#10;entity recognition, relation extraction, multi-label document classification,&#10;and semantic similarity and reasoning, (2) examines the errors produced by the&#10;LLMs and categorized the errors into three types: missingness, inconsistencies,&#10;and unwanted artificial content, and (3) provides suggestions for using LLMs in&#10;BioNLP applications. We make the datasets, baselines, and results publicly&#10;available to the community via&#10;https://github.com/qingyu-qc/gpt_bionlp_benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.03109" label="2307.03109">
        <attvalues>
          <attvalue for="0" value="A Survey on Evaluation of Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) are gaining increasing popularity in both&#10;academia and industry, owing to their unprecedented performance in various&#10;applications. As LLMs continue to play a vital role in both research and daily&#10;use, their evaluation becomes increasingly critical, not only at the task&#10;level, but also at the society level for better understanding of their&#10;potential risks. Over the past years, significant efforts have been made to&#10;examine LLMs from various perspectives. This paper presents a comprehensive&#10;review of these evaluation methods for LLMs, focusing on three key dimensions:&#10;what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide&#10;an overview from the perspective of evaluation tasks, encompassing general&#10;natural language processing tasks, reasoning, medical usage, ethics,&#10;educations, natural and social sciences, agent applications, and other areas.&#10;Secondly, we answer the `where' and `how' questions by diving into the&#10;evaluation methods and benchmarks, which serve as crucial components in&#10;assessing performance of LLMs. Then, we summarize the success and failure cases&#10;of LLMs in different tasks. Finally, we shed light on several future challenges&#10;that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to&#10;researchers in the realm of LLMs evaluation, thereby aiding the development of&#10;more proficient LLMs. Our key point is that evaluation should be treated as an&#10;essential discipline to better assist the development of LLMs. We consistently&#10;maintain the related open-source materials at:&#10;https://github.com/MLGroupJLU/LLM-eval-survey.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.06644" label="2104.06644">
        <attvalues>
          <attvalue for="0" value="Masked Language Modeling and the Distributional Hypothesis: Order Word&#10;  Matters Pre-training for Little" />
          <attvalue for="1" value="  A possible explanation for the impressive performance of masked language&#10;model (MLM) pre-training is that such models have learned to represent the&#10;syntactic structures prevalent in classical NLP pipelines. In this paper, we&#10;propose a different explanation: MLMs succeed on downstream tasks almost&#10;entirely due to their ability to model higher-order word co-occurrence&#10;statistics. To demonstrate this, we pre-train MLMs on sentences with randomly&#10;shuffled word order, and show that these models still achieve high accuracy&#10;after fine-tuning on many downstream tasks -- including on tasks specifically&#10;designed to be challenging for models that ignore word order. Our models&#10;perform surprisingly well according to some parametric syntactic probes,&#10;indicating possible deficiencies in how we test representations for syntactic&#10;information. Overall, our results show that purely distributional information&#10;largely explains the success of pre-training, and underscore the importance of&#10;curating challenging evaluation datasets that require deeper linguistic&#10;knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;{\bfseries Sensitivity to word order in NLU}. Information order has been a topic of research in computational linguistics since \newcite{barzilay-lee-2004-catching} introduced the task of ranking sentence orders&#10;as an evaluation for language generation quality, an approach which was subsequently also used to evaluate readability and dialogue coherence \cite{barzilay-lapata-2008-modeling, laban-etal-2021-transformer}. &#10;&#10;More recently, several research groups have investigated information order for words rather than sentences as an evaluation of model humanlikeness. \cite{sinha2020b} investigate the task of natural language inference (NLI) and find high accuracy on permuted examples for different Transformer and pre-Transformer era models, across English and Chinese datasets \cite{hu-etal-2020-ocnli}.&#10;\cite{gupta-etal-2021-bert} use targeted permutations on RoBERTa-based models and show word order insensitivity across natural language inference (MNLI), paraphrase detection (QQP) and sentiment analysis tasks (SST-2). &#10;\cite{pham2020} show insensitivity on a larger set of tasks, including the entire GLUE benchmark, and find that certain tasks in GLUE, such as CoLA and RTE are more sensitive to permutations than others.&#10;\newcite{ettinger-2020-whatbertisnot} recently observed that BERT accuracy decreases for some word order perturbed examples, but not for others. &#10;In all these prior works, models were given access to normal word order at (pre-)training time, but not at fine-tuning or test time. It was not clear whether the model acquires enough information about word order during the fine-tuning step, or whether it is ingrained in the pre-trained model. &#10;In this work, we take these investigations a step further: we show that the word order information needed for downstream tasks does not need to be provided to the model during pre-training. Since models can learn whatever word order information they do need largely from fine-tuning alone, this likely suggests that our downstream tasks don't actually require much complex word order information in the first place (cf.,\ \cite{glavas-vulic-2021-supervised}). &#10;&#10;{\bfseries Randomization ablations}.&#10;Random controls have been explored in a variety of prior work. &#10;\cite{wietingkiela2019} show that random sentence encoders are surprisingly powerful baselines. \cite{gauthier-levy-2019-linking} use random sentence reordering to label some tasks as ``syntax-light'' making them more easily decodeable from images of the brain. &#10;\cite{shen2020a} show that entire layers of MLM transformers can be randomly initialized and kept frozen throughout training without detrimental effect and that those layers perform better on some probing tasks than their frozen counterparts.&#10;Models have been found to be surprisingly robust to randomizing or cutting syntactic tree structures they were hoped to rely on~\cite{scheible2013cutting,williams2018latent}, and randomly permuting attention weights often induces only minimal changes in output \cite{jain-wallace-2019-attention}. In computer vision, it is well known that certain architectures constitute good ``deep image priors'' for fine-tuning~\cite{ulyanov2018deep} or pruning~\cite{frankle2020training}, and that even randomly wired networks can perform well at image recognition~\cite{xie2019exploring}. Here, we explore randomizing the data, rather than the model, to assess whether certain claims about which phenomena the model has learned are established in fact.&#10;&#10;{\bfseries Synthetic pre-training}. \cite{kataoka2021} found that pre-training on synthetically generated fractals for image classification is a very strong prior for subsequent fine-tuning on real image data. In language modeling, \cite{papadimitriou-jurafsky-2020-learning} train LSTMs \cite{hochreiter1997long} on non-linguistic data with latent structure such as MIDI music or Java code provides better test performance on downstream tasks than a randomly initialized model. They observe that even when there is no vocabulary overlap among source and target languages, LSTM language models leverage the latent hierarchical structure of the input to obtain better performance than a random, Zipfian corpus of the same vocabulary.&#10;&#10;{\bfseries On the utility of probing tasks}.&#10;Many recent papers provide compelling evidence that BERT contains a surprising amount of syntax, semantics, and world knowledge \cite{giulianelli-etal-2018-hood,rogers-etal-2020-primer,lakretz-etal-2019-emergence,jumelet-etal-2019-analysing,jumelet-etal-2021-language}. Many of these works involve diagnostic classifiers \cite{hupkes2018visualisation} or parametric probes, i.e.\ a function atop learned representations that is optimized to find linguistic information. How well the probe learns a given signal can be seen as a proxy for linguistic knowledge encoded in the representations. However, the community is divided on many aspects of probing \cite{belinkov2021probing} including how complex probes should be.&#10;Many prefer simple linear probes over the complex ones \cite{alain2016understanding, hewitt2019structural, hall-maudslay-etal-2020-tale}. However, complex probes with strong representational capacity are able to extract the most information from representations \cite{voita-titov-2020-information, pimentel-etal-2020-information, hall-maudslay-etal-2020-tale}.&#10;Here, we follow \cite{pimentel-etal-2020-pareto} and use both simple (linear) and complex (non-linear) models, as well as ``complex'' tasks (dependency parsing). &#10;As an alternative to parametric probes, stimulus-based non-parametric probing \cite{linzen-etal-2016-assessing, jumelet-hupkes-2018-language, marvin-linzen-2018-targeted, gulordava-etal-2018-colorless, warstadt-etal-2019-investigating, warstadt-etal-2020-blimp-benchmark, warstadt-etal-2020-learning, ettinger-2020-whatbertisnot,lakretz2021mechanisms} has been used to show that even without a learned probe, BERT can predict syntactic properties with high confidence~\cite{goldberga, wolf2019}.&#10;We use this class of non-parametric probes to investigate RoBERTa's ability to learn word order during pre-training.&#10;&#10;" />
          <attvalue for="4" value="Syntactic Structure Analysis, Computer Science, Linguistics, Language Model Pre-training, Artificial Intelligence, Natural Language Processing, Statistics, Distributional Semantics Evaluation" />
        </attvalues>
      </node>
      <node id="2305.01555" label="2305.01555">
        <attvalues>
          <attvalue for="0" value="How to Unleash the Power of Large Language Models for Few-shot Relation&#10;  Extraction?" />
          <attvalue for="1" value="  Scaling language models have revolutionized widespread NLP tasks, yet little&#10;comprehensively explored few-shot relation extraction with large language&#10;models. In this paper, we investigate principal methodologies, in-context&#10;learning and data generation, for few-shot relation extraction via GPT-3.5&#10;through exhaustive experiments. To enhance few-shot performance, we further&#10;propose task-related instructions and schema-constrained data generation. We&#10;observe that in-context learning can achieve performance on par with previous&#10;prompt learning approaches, and data generation with the large language model&#10;can boost previous solutions to obtain new state-of-the-art few-shot results on&#10;four widely-studied relation extraction datasets. We hope our work can inspire&#10;future research for the capabilities of large language models in few-shot&#10;relation extraction. Code is available in&#10;https://github.com/zjunlp/DeepKE/tree/main/example/llm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Scaling, Few-Shot Learning, Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Relation Extraction" />
        </attvalues>
      </node>
      <node id="2401.08350" label="2401.08350">
        <attvalues>
          <attvalue for="0" value="Salute the Classic: Revisiting Challenges of Machine Translation in the&#10;  Age of Large Language Models" />
          <attvalue for="1" value="  The evolution of Neural Machine Translation (NMT) has been significantly&#10;influenced by six core challenges (Koehn and Knowles, 2017), which have acted&#10;as benchmarks for progress in this field. This study revisits these challenges,&#10;offering insights into their ongoing relevance in the context of advanced Large&#10;Language Models (LLMs): domain mismatch, amount of parallel data, rare word&#10;prediction, translation of long sentences, attention model as word alignment,&#10;and sub-optimal beam search. Our empirical findings indicate that LLMs&#10;effectively lessen the reliance on parallel data for major languages in the&#10;pretraining phase. Additionally, the LLM-based translation system significantly&#10;enhances the translation of long sentences that contain approximately 80 words&#10;and shows the capability to translate documents of up to 512 words. However,&#10;despite these significant improvements, the challenges of domain mismatch and&#10;prediction of rare words persist. While the challenges of word alignment and&#10;beam search, specifically associated with NMT, may not apply to LLMs, we&#10;identify three new challenges for LLMs in translation tasks: inference&#10;efficiency, translation of low-resource languages in the pretraining phase, and&#10;human-aligned evaluation. The datasets and models are released at&#10;https://github.com/pangjh3/LLM4MT.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the Natural Language Processing community, one of the most critical and longstanding tasks is Machine Translation (MT), which aims to convert human languages from one form to another \cite{koehn2009statistical,poibeau2017machine}. &#10;As the demand for effective translation systems continues to grow, researchers have been striving to develop models that can tackle the inherent challenges of this complex task. &#10;In this context, the six challenges about MT proposed by \newcite{koehn-knowles-2017-six} have been widely recognized and studied by numerous studies, with many efforts revolving around them \cite{chu-wang-2018-survey,neishi-yoshinaga-2019-relation,garg-etal-2019-jointly,pang2023rethinking}.&#10;&#10;The emerging Large Language Models (LLMs) have been a significant breakthrough in NLP \cite{touvron2023llama,openai2023gpt4,touvron2023llama2}.&#10;LLMs have demonstrated remarkable capabilities, outperforming traditional approaches and setting new benchmark performance for various applications such as machine translation \cite{lyu2023new,zhu2023multilingual,zhang2023prompting,wang2023document}. &#10;LLMs exhibit remarkable translation capabilities for major languages, owing to their extensive pretraining on vast amounts of unpaired data. This implies a significant advancement over conventional techniques.&#10;Consequently, recent studies have employed LLMs for translation tasks \cite{jiao2023parrot,alves-etal-2023-steering}, achieving remarkable performance.&#10;However, it is unclear how LLMs fare against the six classical challenges,&#10;This intriguing question warrants further investigation and discussion.&#10;&#10;To gain insights into LLM-based MT research and identify paths for advancement,&#10;we train Llama2 models as German-to-English translation systems and evaluate their abilities in addressing the six classic challenges.&#10;Note that English is a high-resource language in the Llama2 pretraining data \cite{touvron2023llama2} and German is a relatively high-resource language, ensuring the model's competence in these languages.&#10;Furthermore, we identify two LLM-specific challenges: pretraining resource imbalance and human-like evaluation issues.&#10;German, Chinese, Ukrainian, and Hebrew, are included for English-to-X low-resource translation tasks to assess the effects of resource imbalance.&#10;Table~\ref{tab:overview} summarizes our key findings, revealing that LLMs have successfully tackled data quantity and long sentence translation challenges but still face unresolved issues.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Translation Challenges, Linguistics, Artificial Intelligence, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1806.00258" label="1806.00258">
        <attvalues>
          <attvalue for="0" value="A Survey of Domain Adaptation for Neural Machine Translation" />
          <attvalue for="1" value="  Neural machine translation (NMT) is a deep learning based approach for&#10;machine translation, which yields the state-of-the-art translation performance&#10;in scenarios where large-scale parallel corpora are available. Although the&#10;high-quality and domain-specific translation is crucial in the real world,&#10;domain-specific corpora are usually scarce or nonexistent, and thus vanilla NMT&#10;performs poorly in such scenarios. Domain adaptation that leverages both&#10;out-of-domain parallel corpora as well as monolingual corpora for in-domain&#10;translation, is very important for domain-specific translation. In this paper,&#10;we give a comprehensive survey of the state-of-the-art domain adaptation&#10;techniques for NMT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.02074" label="1909.02074">
        <attvalues>
          <attvalue for="0" value="Jointly Learning to Align and Translate with Transformer Models" />
          <attvalue for="1" value="  The state of the art in machine translation (MT) is governed by neural&#10;approaches, which typically provide superior translation accuracy over&#10;statistical approaches. However, on the closely related task of word alignment,&#10;traditional statistical word alignment models often remain the go-to solution.&#10;In this paper, we present an approach to train a Transformer model to produce&#10;both accurate translations and alignments. We extract discrete alignments from&#10;the attention probabilities learnt during regular neural machine translation&#10;model training and leverage them in a multi-task framework to optimize towards&#10;translation and alignment objectives. We demonstrate that our approach produces&#10;competitive results compared to GIZA++ trained IBM alignment models without&#10;sacrificing translation accuracy and outperforms previous attempts on&#10;Transformer model based word alignment. Finally, by incorporating IBM model&#10;alignments into our multi-task training, we report significantly better&#10;alignment accuracies compared to GIZA++ on three publicly available data sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.04675" label="2304.04675">
        <attvalues>
          <attvalue for="0" value="Multilingual Machine Translation with Large Language Models: Empirical&#10;  Results and Analysis" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated remarkable potential in&#10;handling multilingual machine translation (MMT). In this paper, we&#10;systematically investigate the advantages and challenges of LLMs for MMT by&#10;answering two questions: 1) How well do LLMs perform in translating massive&#10;languages? 2) Which factors affect LLMs' performance in translation? We&#10;thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4. Our&#10;empirical results show that translation capabilities of LLMs are continually&#10;involving. GPT-4 has beat the strong supervised baseline NLLB in 40.91% of&#10;translation directions but still faces a large gap towards the commercial&#10;translation system like Google Translate, especially on low-resource languages.&#10;Through further analysis, we discover that LLMs exhibit new working patterns&#10;when used for MMT. First, LLM can acquire translation ability in a&#10;resource-efficient way and generate moderate translation even on zero-resource&#10;languages. Second, instruction semantics can surprisingly be ignored when given&#10;in-context exemplars. Third, cross-lingual exemplars can provide better task&#10;guidance for low-resource translation than exemplars in the same language&#10;pairs. Code will be released at: https://github.com/NJUNLP/MMT-LLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.07069" label="2301.07069">
        <attvalues>
          <attvalue for="0" value="Prompting Large Language Model for Machine Translation: A Case Study" />
          <attvalue for="1" value="  Research on prompting has shown excellent performance with little or even no&#10;supervised training across many tasks. However, prompting for machine&#10;translation is still under-explored in the literature. We fill this gap by&#10;offering a systematic study on prompting strategies for translation, examining&#10;various factors for prompt template and demonstration example selection. We&#10;further explore the use of monolingual data and the feasibility of&#10;cross-lingual, cross-domain, and sentence-to-document transfer learning in&#10;prompting. Extensive experiments with GLM-130B (Zeng et al., 2022) as the&#10;testbed show that 1) the number and the quality of prompt examples matter,&#10;where using suboptimal examples degenerates translation; 2) several features of&#10;prompt examples, such as semantic similarity, show significant Spearman&#10;correlation with their prompting performance; yet, none of the correlations are&#10;strong enough; 3) using pseudo parallel prompt examples constructed from&#10;monolingual data via zero-shot prompting could improve translation; and 4)&#10;improved performance is achievable by transferring knowledge from prompt&#10;examples selected in other settings. We finally provide an analysis on the&#10;model outputs and discuss several problems that prompting still suffers from.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.02210" label="2304.02210">
        <attvalues>
          <attvalue for="0" value="Document-Level Machine Translation with Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) such as ChatGPT can produce coherent, cohesive,&#10;relevant, and fluent answers for various natural language processing (NLP)&#10;tasks. Taking document-level machine translation (MT) as a testbed, this paper&#10;provides an in-depth evaluation of LLMs' ability on discourse modeling. The&#10;study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we&#10;investigate the impact of different prompts on document-level translation&#10;quality and discourse phenomena; 2) Comparison of Translation Models, where we&#10;compare the translation performance of ChatGPT with commercial MT systems and&#10;advanced document-level MT methods; 3) Analysis of Discourse Modelling&#10;Abilities, where we further probe discourse knowledge encoded in LLMs and shed&#10;light on impacts of training techniques on discourse modeling. By evaluating on&#10;a number of benchmarks, we surprisingly find that LLMs have demonstrated&#10;superior performance and show potential to become a new paradigm for&#10;document-level translation: 1) leveraging their powerful long-text modeling&#10;capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of&#10;human evaluation; 2) GPT-4 demonstrates a stronger ability for probing&#10;linguistic knowledge than GPT-3.5. This work highlights the challenges and&#10;opportunities of LLMs for MT, which we hope can inspire the future design and&#10;evaluation of LLMs.We release our data and annotations at&#10;https://github.com/longyuewangdcu/Document-MT-LLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.02426" label="2304.02426">
        <attvalues>
          <attvalue for="0" value="ParroT: Translating during Chat using Large Language Models tuned with&#10;  Human Translation and Feedback" />
          <attvalue for="1" value="  Large language models (LLMs) like ChatGPT have exhibited remarkable abilities&#10;on a wide range of natural language processing~(NLP) tasks, including various&#10;machine translation abilities accomplished during chat. However, these models&#10;are only accessible through restricted APIs, which creates barriers to new&#10;research and advancements in the field. Therefore, we propose ParroT, a&#10;framework to enhance and regulate the translation abilities during chat based&#10;on open-source LLMs (e.g., LLaMA), human-written translation and feedback data.&#10;Specifically, ParroT reformulates translation data into the&#10;instruction-following style, and introduces a &quot;$\mathbf{Hint}$&quot; field for&#10;incorporating extra requirements to regulate the translation process.&#10;Accordingly, we propose three instruction types for finetuning ParroT models,&#10;including translation instruction, contrastive instruction, and error-guided&#10;instruction. Experiments on Flores subsets and WMT22 test sets suggest that&#10;translation instruction improves the translation performance of vanilla LLMs&#10;significantly while error-guided instruction can lead to further improvement,&#10;which demonstrates the importance of learning from low-quality translations&#10;annotated by humans. We also demonstrate the potential of automatic evaluation&#10;tools in providing quality information of translations, when constructing&#10;error-guided instructions for directions that lack human annotation data.&#10;Please refer to our Github project for more implementation details:&#10;https://github.com/wxjiao/ParroT&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.13448" label="2310.13448">
        <attvalues>
          <attvalue for="0" value="Steering Large Language Models for Machine Translation with Finetuning&#10;  and In-Context Learning" />
          <attvalue for="1" value="  Large language models (LLMs) are a promising avenue for machine translation&#10;(MT). However, current LLM-based MT systems are brittle: their effectiveness&#10;highly depends on the choice of few-shot examples and they often require extra&#10;post-processing due to overgeneration. Alternatives such as finetuning on&#10;translation instructions are computationally expensive and may weaken&#10;in-context learning capabilities, due to overspecialization. In this paper, we&#10;provide a closer look at this problem. We start by showing that adapter-based&#10;finetuning with LoRA matches the performance of traditional finetuning while&#10;reducing the number of training parameters by a factor of 50. This method also&#10;outperforms few-shot prompting and eliminates the need for post-processing or&#10;in-context examples. However, we show that finetuning generally degrades&#10;few-shot performance, hindering adaptation capabilities. Finally, to obtain the&#10;best of both worlds, we propose a simple approach that incorporates few-shot&#10;examples during finetuning. Experiments on 10 language pairs show that our&#10;proposed approach recovers the original few-shot capabilities while keeping the&#10;added benefits of finetuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.00832" label="2406.00832">
        <attvalues>
          <attvalue for="0" value="BoNBoN Alignment for Large Language Models and the Sweetness of&#10;  Best-of-n Sampling" />
          <attvalue for="1" value="  This paper concerns the problem of aligning samples from large language&#10;models to human preferences using best-of-$n$ sampling, where we draw $n$&#10;samples, rank them, and return the best one. We consider two fundamental&#10;problems. First: what is the relationship between best-of-$n$ and approaches to&#10;alignment that train LLMs to output samples with a high expected reward (e.g.,&#10;RLHF or DPO)? To answer this, we embed both the best-of-$n$ distribution and&#10;the sampling distributions learned by alignment procedures in a common class of&#10;tiltings of the base LLM distribution. We then show that, within this class,&#10;best-of-$n$ is essentially optimal in terms of the trade-off between win-rate&#10;against the base model vs KL distance from the base model. That is, best-of-$n$&#10;is the best choice of alignment distribution if the goal is to maximize win&#10;rate. However, best-of-$n$ requires drawing $n$ samples for each inference, a&#10;substantial cost. To avoid this, the second problem we consider is how to&#10;fine-tune a LLM to mimic the best-of-$n$ sampling distribution. We derive&#10;BoNBoN Alignment to achieve this by exploiting the special structure of the&#10;best-of-$n$ distribution. Experiments show that BoNBoN alignment yields&#10;substantial improvements in producing a model that is preferred to the base&#10;policy while minimally affecting off-target aspects.&#10;" />
          <attvalue for="2" value="&#10;This paper concerns the problem of aligning large language models (LLMs) to bias their outputs toward human preferences. &#10;There are now a wealth of approaches to this problem \cite[e.g.,][]{NEURIPS2022_b1efde53, NIPS2017_d5e2c0ad, kaufmann2023survey, li2024inference, rafailov2023direct, azar2024general}.&#10;Here, we interested in the best-of-$n$ (BoN) sampling strategy. In BoN sampling, we draw $n$ samples from the LLM, rank them on the attribute of interest, and return the best one. This simple procedure is surprisingly effective in practice \cite{beirami2024theoretical,wang2024transforming,gao2023scaling,eisenstein2023helping}.&#10;We consider two fundamental questions about BoN:&#10;\begin{enumerate}&#10; \item What is the relationship between BoN and other approaches to alignment?&#10; \item How can we effectively train a LLM to mimic the BoN sampling distribution? &#10;\end{enumerate}&#10;&#10;In brief: we find that the BoN distribution is (essentially) the optimal policy for maximizing win rate while minimally affecting off-target aspects of generation, and we develop an effective method for aligning LLMs to mimic this distribution. Together, these results yield a highly effective alignment method; see \ref{fig:intro-figure} for an illustration. &#10;&#10;LLM Alignment&#10;The goal of alignment is to bias the outputs of an LLM to be good on some target attribute (e.g., helpfulness), while minimally changing the behavior of the model on off-target attributes (e.g., reasoning ability).&#10;Commonly, the notion of goodness is elicited by collecting pairs of responses to many prompts, and asking (human or AI) annotators to choose the better response. Then, these pairs are used to define a training procedure for updating the base LLM to a new, aligned, LLM that outputs responses that are better in the target attribute.&#10;&#10;There are two main approaches. First, RLHF methods train an explicit reward model on the pairs, and then align the model using reinforcement learning with this learned reward \cite[e.g.,][]{NEURIPS2022_b1efde53, kaufmann2023survey}.&#10;Second, contrastive methods directly use the preference data to define an objective function for fine-tuning the LLM \cite{rafailov2023direct,azar2024general,ethayarajh2024kto,xu2024contrastive,hong2024reference}.&#10;In both cases, the trade-off between alignment and off-target behavior is controlled by a hyper-parameter that explicitly penalizes the divergence from the base LLM. For example, in the reinforcement learning setting, this is done by adding a regularization term that penalizes the estimated KL divergence between the aligned model and the reference model. &#10;&#10;The first main question we address in this paper is: what is the relationship between the sampling distribution defined by these approaches and the sampling distribution defined by best-of-$n$?&#10;This is important, in particular, because in principle we could forgo the explicit alignment training and just use BoN sampling. However, it is not clear when each option should be preferred. &#10;&#10;Now, the comparison of training-aligned models and BoN is not fully fair. The reason is that producing a BoN sample requires drawing $n$ samples from the base LLM (instead of just one). This is a substantial computational cost. The second main question we address is: if we do in fact want to sample from the BoN distribution, how can we train a LLM to mimic this distribution? If this can be done effectively, then the inference cost of BoN sampling can be avoided.&#10;&#10;We answer these questions with the following contributions:&#10;\begin{enumerate}&#10; \item We show that the BoN sampling distribution can be embedded in a common class with the distributions produced by training-based alignment methods. Within this common class, we derive the distribution with the best possible trade-off between win-rate against the base model vs KL distance from the base model. Then, we show that the BoN distribution is essentially equal to this Pareto-optimal distribution.&#10; \item We then develop an effective method for training a LLM to mimic the BoN sampling distribution. In essence, the procedure draws best-of-$n$ and worst-of-$n$ samples as training data, and combines these with an objective function we derive by exploiting the analytical form of the BoN distribution. &#10; We call this procedure BoNBoN Alignment. &#10; \item Finally, we show empirically that BoNBoN Alignment yields models that achieve high win rates while minimally affecting off-target aspects of the generations, outperforming baselines.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Sampling Distribution Optimization, Fine-Tuning Large Models, Artificial Intelligence, Mathematics, Natural Language Processing, Statistics, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="2401.01879" label="2401.01879">
        <attvalues>
          <attvalue for="0" value="Theoretical guarantees on the best-of-n alignment policy" />
          <attvalue for="1" value="  A simple and effective method for the alignment of generative models is the&#10;best-of-$n$ policy, where $n$ samples are drawn from a base policy, and ranked&#10;based on a reward function, and the highest ranking one is selected. A commonly&#10;used analytical expression in the literature claims that the KL divergence&#10;between the best-of-$n$ policy and the base policy is equal to $\log (n) -&#10;(n-1)/n.$ We disprove the validity of this claim, and show that it is an upper&#10;bound on the actual KL divergence. We also explore the tightness of this upper&#10;bound in different regimes. Finally, we propose a new estimator for the KL&#10;divergence and empirically show that it provides a tight approximation through&#10;a few examples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.00742" label="2402.00742">
        <attvalues>
          <attvalue for="0" value="Transforming and Combining Rewards for Aligning Large Language Models" />
          <attvalue for="1" value="  A common approach for aligning language models to human preferences is to&#10;first learn a reward model from preference data, and then use this reward model&#10;to update the language model. We study two closely related problems that arise&#10;in this approach. First, any monotone transformation of the reward model&#10;preserves preference ranking; is there a choice that is ``better'' than others?&#10;Second, we often wish to align language models to multiple properties: how&#10;should we combine multiple reward models? Using a probabilistic interpretation&#10;of the alignment procedure, we identify a natural choice for transformation for&#10;(the common case of) rewards learned from Bradley-Terry preference models. The&#10;derived transformation is straightforward: we apply a log-sigmoid function to&#10;the centered rewards, a method we term ``LSC-transformation''&#10;(log-sigmoid-centered transformation). This transformation has two important&#10;properties. First, it emphasizes improving poorly-performing outputs, rather&#10;than outputs that already score well. This mitigates both underfitting (where&#10;some prompts are not improved) and reward hacking (where the model learns to&#10;exploit misspecification of the reward model). Second, it enables principled&#10;aggregation of rewards by linking summation to logical conjunction: the sum of&#10;transformed rewards corresponds to the probability that the output is ``good''&#10;in all measured properties, in a sense we make precise. Experiments aligning&#10;language models to be both helpful and harmless using RLHF show substantial&#10;improvements over the baseline (non-transformed) approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.10760" label="2210.10760">
        <attvalues>
          <attvalue for="0" value="Scaling Laws for Reward Model Overoptimization" />
          <attvalue for="1" value="  In reinforcement learning from human feedback, it is common to optimize&#10;against a reward model trained to predict human preferences. Because the reward&#10;model is an imperfect proxy, optimizing its value too much can hinder ground&#10;truth performance, in accordance with Goodhart's law. This effect has been&#10;frequently observed, but not carefully measured due to the expense of&#10;collecting human preference data. In this work, we use a synthetic setup in&#10;which a fixed &quot;gold-standard&quot; reward model plays the role of humans, providing&#10;labels used to train a proxy reward model. We study how the gold reward model&#10;score changes as we optimize against the proxy reward model using either&#10;reinforcement learning or best-of-$n$ sampling. We find that this relationship&#10;follows a different functional form depending on the method of optimization,&#10;and that in both cases its coefficients scale smoothly with the number of&#10;reward model parameters. We also study the effect on this relationship of the&#10;size of the reward model dataset, the number of reward model and policy&#10;parameters, and the coefficient of the KL penalty added to the reward in the&#10;reinforcement learning setup. We explore the implications of these empirical&#10;results for theoretical considerations in AI alignment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.09244" label="2312.09244">
        <attvalues>
          <attvalue for="0" value="Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate&#10;  Reward Hacking" />
          <attvalue for="1" value="  Reward models play a key role in aligning language model applications towards&#10;human preferences. However, this setup creates an incentive for the language&#10;model to exploit errors in the reward model to achieve high estimated reward, a&#10;phenomenon often termed \emph{reward hacking}. A natural mitigation is to train&#10;an ensemble of reward models, aggregating over model outputs to obtain a more&#10;robust reward estimate. We explore the application of reward ensembles to&#10;alignment at both training time (through reinforcement learning) and inference&#10;time (through reranking). First, we show that reward models are&#10;\emph{underspecified}: reward models that perform similarly in-distribution can&#10;yield very different rewards when used in alignment, due to distribution shift.&#10;Second, underspecification results in overoptimization, where alignment to one&#10;reward model does not improve reward as measured by another reward model&#10;trained on the same data. Third, overoptimization is mitigated by the use of&#10;reward ensembles, and ensembles that vary by their \emph{pretraining} seeds&#10;lead to better generalization than ensembles that differ only by their&#10;\emph{fine-tuning} seeds, with both outperforming individual reward models.&#10;However, even pretrain reward ensembles do not eliminate reward hacking: we&#10;show several qualitative reward hacking phenomena that are not mitigated by&#10;ensembling because all reward models in the ensemble exhibit similar error&#10;patterns.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18290" label="2305.18290">
        <attvalues>
          <attvalue for="0" value="Direct Preference Optimization: Your Language Model is Secretly a Reward&#10;  Model" />
          <attvalue for="1" value="  While large-scale unsupervised language models (LMs) learn broad world&#10;knowledge and some reasoning skills, achieving precise control of their&#10;behavior is difficult due to the completely unsupervised nature of their&#10;training. Existing methods for gaining such steerability collect human labels&#10;of the relative quality of model generations and fine-tune the unsupervised LM&#10;to align with these preferences, often with reinforcement learning from human&#10;feedback (RLHF). However, RLHF is a complex and often unstable procedure, first&#10;fitting a reward model that reflects the human preferences, and then&#10;fine-tuning the large unsupervised LM using reinforcement learning to maximize&#10;this estimated reward without drifting too far from the original model. In this&#10;paper we introduce a new parameterization of the reward model in RLHF that&#10;enables extraction of the corresponding optimal policy in closed form, allowing&#10;us to solve the standard RLHF problem with only a simple classification loss.&#10;The resulting algorithm, which we call Direct Preference Optimization (DPO), is&#10;stable, performant, and computationally lightweight, eliminating the need for&#10;sampling from the LM during fine-tuning or performing significant&#10;hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align&#10;with human preferences as well as or better than existing methods. Notably,&#10;fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of&#10;generations, and matches or improves response quality in summarization and&#10;single-turn dialogue while being substantially simpler to implement and train.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12036" label="2310.12036">
        <attvalues>
          <attvalue for="0" value="A General Theoretical Paradigm to Understand Learning from Human&#10;  Preferences" />
          <attvalue for="1" value="  The prevalent deployment of learning from human preferences through&#10;reinforcement learning (RLHF) relies on two important approximations: the first&#10;assumes that pairwise preferences can be substituted with pointwise rewards.&#10;The second assumes that a reward model trained on these pointwise rewards can&#10;generalize from collected data to out-of-distribution data sampled by the&#10;policy. Recently, Direct Preference Optimisation (DPO) has been proposed as an&#10;approach that bypasses the second approximation and learn directly a policy&#10;from collected data without the reward modelling stage. However, this method&#10;still heavily relies on the first approximation.&#10;  In this paper we try to gain a deeper theoretical understanding of these&#10;practical algorithms. In particular we derive a new general objective called&#10;$\Psi$PO for learning from human preferences that is expressed in terms of&#10;pairwise preferences and therefore bypasses both approximations. This new&#10;general objective allows us to perform an in-depth analysis of the behavior of&#10;RLHF and DPO (as special cases of $\Psi$PO) and to identify their potential&#10;pitfalls. We then consider another special case for $\Psi$PO by setting $\Psi$&#10;simply to Identity, for which we can derive an efficient optimisation&#10;procedure, prove performance guarantees and demonstrate its empirical&#10;superiority to DPO on some illustrative examples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.01306" label="2402.01306">
        <attvalues>
          <attvalue for="0" value="KTO: Model Alignment as Prospect Theoretic Optimization" />
          <attvalue for="1" value="  Kahneman &amp; Tversky's $\textit{prospect theory}$ tells us that humans perceive&#10;random variables in a biased but well-defined manner (1992); for example,&#10;humans are famously loss-averse. We show that objectives for aligning LLMs with&#10;human feedback implicitly incorporate many of these biases -- the success of&#10;these objectives (e.g., DPO) over cross-entropy minimization can partly be&#10;ascribed to them belonging to a family of loss functions that we call&#10;$\textit{human-aware losses}$ (HALOs). However, the utility functions these&#10;methods attribute to humans still differ from those in the prospect theory&#10;literature. Using a Kahneman-Tversky model of human utility, we propose a HALO&#10;that directly maximizes the utility of generations instead of maximizing the&#10;log-likelihood of preferences, as current methods do. We call this approach&#10;KTO, and it matches or exceeds the performance of preference-based methods at&#10;scales from 1B to 30B, despite only learning from a binary signal of whether an&#10;output is desirable. More broadly, our work suggests that there is no one HALO&#10;that is universally superior; the best loss depends on the inductive biases&#10;most appropriate for a given setting, an oft-overlooked consideration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.01975" label="2209.01975">
        <attvalues>
          <attvalue for="0" value="Selective Annotation Makes Language Models Better Few-Shot Learners" />
          <attvalue for="1" value="  Many recent approaches to natural language tasks are built on the remarkable&#10;abilities of large language models. Large language models can perform&#10;in-context learning, where they learn a new task from a few task&#10;demonstrations, without any parameter updates. This work examines the&#10;implications of in-context learning for the creation of datasets for new&#10;natural language tasks. Departing from recent in-context learning methods, we&#10;formulate an annotation-efficient, two-step framework: selective annotation&#10;that chooses a pool of examples to annotate from unlabeled data in advance,&#10;followed by prompt retrieval that retrieves task examples from the annotated&#10;pool at test time. Based on this framework, we propose an unsupervised,&#10;graph-based selective annotation method, voke-k, to select diverse,&#10;representative examples to annotate. Extensive experiments on 10 datasets&#10;(covering classification, commonsense reasoning, dialogue, and text/code&#10;generation) demonstrate that our selective annotation method improves the task&#10;performance by a large margin. On average, vote-k achieves a 12.9%/11.4%&#10;relative gain under an annotation budget of 18/100, as compared to randomly&#10;selecting examples to annotate. Compared to state-of-the-art supervised&#10;finetuning approaches, it yields similar performance with 10-100x less&#10;annotation cost across 10 tasks. We further analyze the effectiveness of our&#10;framework in various scenarios: language models with varying sizes, alternative&#10;selective annotation methods, and cases where there is a test data domain&#10;shift. We hope that our studies will serve as a basis for data annotations as&#10;large language models are increasingly applied to new tasks. Our code is&#10;available at https://github.com/HKUNLP/icl-selective-annotation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;In-Context Learning &#10;In-context learning with large language models has recently received an increasing amount of interest, partly due to its flexibility and sample efficiency \cite{prompt_survey}.&#10;Several recent works proposed methods to improve in-context learning in many aspects: e.g., meta-training \cite{chen-etal-2022-meta,min-etal-2022-metaicl}, task instructions \cite{efrat20,mishra-etal-2022-cross,Wei2021FinetunedLM,sahn22}, or task formulation \cite{holtzman-etal-2021-surface,calibrate,min-etal-2022-noisy}.&#10;In this paradigm, the choice of in-context (i.e., demonstration) examples has been shown crucial \cite{liu-etal-2022-makes,rubin2022,lu-etal-2022-fantastically}, while recent work raised questions as to the degree to which correct labels are necessary \cite{min2022rethinking}.&#10;This work proposes an annotation-efficient in-context learning framework by focusing on the choice of examples and its implications on the annotation cost.&#10;&#10;Active Learning&#10;Active learning aims to enable machine learning models to achieve similar or greater performance with fewer labeled training instances~\cite{CohnAL94,settles.tr09}.&#10;Our rststep step for in-context learning shares the same goal of reducing the annotation cost.&#10;Most active learning methods involve iterative parameter updates (e.g., \cite{active_image_recog,kasai-etal-2019-low}), which are computationally expensive for large language models used in in-context learning.&#10;Similar to our \votek algorithm, \cite{Lin2009HowTS} used the facility location &#10;objective to optimize representativeness.&#10;We observed that this objective largely underperforms \votek for in-context learning, probably due to the fact the \votek (1) is effectively tailored to the prompt retrieval step of in-context learning and (2) directly optimizes the diversity of selected samples (see \S\ref{subsec:sample-selection-methods}). &#10;More recently, the effectiveness of active learning has been questioned when large-scale pretrained models are finetuned for various tasks \cite{karamcheti-etal-2021-mind,darcy2022limitations}.&#10;Our experiments (\S\ref{sec:experiments}) showed that rststep helps reduce the annotation cost of in-context learning, departing from the recent observations on finetuning with active learning.&#10;We hypothesize that it is because in-context learning with similarity-based prompt retrieval is more robust to outliers since each test instance only retrieves its most similar examples. &#10;This is supported by \S~\ref{sec:random_retrieval}, where random prompt retrieval does not benefit from rststep.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Data Annotation Methods, Artificial Intelligence, Mathematics, Natural Language Tasks" />
        </attvalues>
      </node>
      <node id="2309.09507" label="2309.09507">
        <attvalues>
          <attvalue for="0" value="Pruning Large Language Models via Accuracy Predictor" />
          <attvalue for="1" value="  Large language models(LLMs) containing tens of billions of parameters (or&#10;even more) have demonstrated impressive capabilities in various NLP tasks.&#10;However, substantial model size poses challenges to training, inference, and&#10;deployment so that it is necessary to compress the model. At present, most&#10;model compression for LLMs requires manual design of pruning features, which&#10;has problems such as complex optimization pipeline and difficulty in retaining&#10;the capabilities of certain parts of the model.Therefore, we propose a novel&#10;pruning approach: firstly, a training set of a certain number of&#10;architecture-accuracy pairs is established, and then a non-neural model is&#10;trained as an accuracy predictor. Using the accuracy predictor to further&#10;optimize the search space and search, the optimal model can be automatically&#10;selected. Experiments show that our proposed approach is effective and&#10;efficient. Compared with the baseline, the perplexity(PPL) on Wikitext2 and PTB&#10;dropped by 9.48% and 5,76% respectively, and the average accuracy of MMLU&#10;increased by 6.28%.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, LLMs has shown impressive reasoning and generation capabilities in various NLP tasks, and these capabilities further enhance as the number of model parameters increases\cite{gpt3,chatgpt,llama1,llama2}. However, the huge computational and memory requirements are still a major obstacle to wider application, so it is necessary to compress LLMs to reduce costs~\cite{li2023auto,liu2023norm,li2022shadow,li2022SFF,li2022self,li2022tf}. Some works for LLMs compression currently focus on model quantization\cite{quant_1, quant_2,dong2023emq}, which is the process of quantizing model parameters into low-bit level representions. &#10;&#10;In face, another commonly used method for model compression is network pruning\cite{prune_survey}, which reduces the size of the model by deleting some unimportant weights. Although some pruning work for LLMs has made some progress, pruning strategies usually require manual design\cite{llm_pruner, wanda, sparsegpt}. Specifically, the pruning ratio of the global or each layer is preset, which causes the pruning results to depend on the set of hyperparameters. On the other hand, due to the model size and wide range of applications of LLMs, manual design of pruning strategies still faces challenges such as complex pipelines and suitabilities for downstream tasks.&#10;&#10;Neural architecture search (NAS)\cite{nas,linas2,li2021nas,dong2023diswot,dong2023rd,lichengp}, which aims to automatically find neural network architecture, has been applied to model pruning due to its effectiveness and simplicity. One common approach is using an accuracy predictor to predict the accuracy of the model architecture to be pruned in the search space, which can save the cost of evaluating these candidate architectures\cite{gbdt_acc,wei2022convformer}. The key of this approach is training the accuracy predictor and the required dataset. Accuracy predictors among previous works are basically based on neural network, although effictive, it still requires a large number of architecture-accuracy pairs and careful design. However, for LLMs, a single evaluation also requires certain computing resources, so it is unrealistic to build tens of thousands of architecture-accuracy pairs.&#10;&#10;In this paper, we propose an alternative approach that first builds a certain number of architecture-accuracy pairs for LLMs(usually a few hundred), and then trains an accuracy predictor based on non-nerural models(e.g., tree based models) to guide LLMs structured pruning. The specific algorithm is as follows:&#10;(1) Combined with expert knowledge to limit the scope of some features(such as layer type, layer ID and so on), narrow the search space and perform random sampling to obtain the model architectures to be pruned.&#10;(2) According to the pruning requirements of each model architectures, prune the model, and evaluate the pruned model from multiple aspects to obtain the accuracy. Then train an accuracy predictor based on gradient boosting decision trees (GBDT)\cite{gbdt}.&#10;(3) Use the trained GBDT model to predict more architectures in the search space, and architectures with top predicted accuracy are selected for further evaluation.&#10;&#10;The main contributions of our approach are as follows:&#10;&#10;(1) Explore the use of GBDT as an accuracy predictor, establish the relationship between the model architecture to be pruned and model performance, and improve the efficiency of searching model architecture.&#10;&#10;(2) Use the trained accuracy predictor to further guide the process of model pruning, and conduct a refined search based on the required performance of a certain aspect of the model. It solves the problem of complex optimization pipeline caused by manually designing pruning features, making it easier to find the optimal model architecture.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Model Compression Techniques, Neural Network Optimization, Mathematics" />
        </attvalues>
      </node>
      <node id="2305.13803" label="2305.13803">
        <attvalues>
          <attvalue for="0" value="NORM: Knowledge Distillation via N-to-One Representation Matching" />
          <attvalue for="1" value="  Existing feature distillation methods commonly adopt the One-to-one&#10;Representation Matching between any pre-selected teacher-student layer pair. In&#10;this paper, we present N-to-One Representation (NORM), a new two-stage&#10;knowledge distillation method, which relies on a simple Feature Transform (FT)&#10;module consisting of two linear layers. In view of preserving the intact&#10;information learnt by the teacher network, during training, our FT module is&#10;merely inserted after the last convolutional layer of the student network. The&#10;first linear layer projects the student representation to a feature space&#10;having N times feature channels than the teacher representation from the last&#10;convolutional layer, and the second linear layer contracts the expanded output&#10;back to the original feature space. By sequentially splitting the expanded&#10;student representation into N non-overlapping feature segments having the same&#10;number of feature channels as the teacher's, they can be readily forced to&#10;approximate the intact teacher representation simultaneously, formulating a&#10;novel many-to-one representation matching mechanism conditioned on a single&#10;teacher-student layer pair. After training, such an FT module will be naturally&#10;merged into the subsequent fully connected layer thanks to its linear property,&#10;introducing no extra parameters or architectural modifications to the student&#10;network at inference. Extensive experiments on different visual recognition&#10;benchmarks demonstrate the leading performance of our method. For instance, the&#10;ResNet18|MobileNet|ResNet50-1/4 model trained by NORM reaches&#10;72.14%|74.26%|68.03% top-1 accuracy on the ImageNet dataset when using a&#10;pre-trained ResNet34|ResNet50|ResNet50 model as the teacher, achieving an&#10;absolute improvement of 2.01%|4.63%|3.03% against the individually trained&#10;counterpart. Code is available at https://github.com/OSVAI/NORM&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.17323" label="2210.17323">
        <attvalues>
          <attvalue for="0" value="GPTQ: Accurate Post-Training Quantization for Generative Pre-trained&#10;  Transformers" />
          <attvalue for="1" value="  Generative Pre-trained Transformer models, known as GPT or OPT, set&#10;themselves apart through breakthrough performance across complex language&#10;modelling tasks, but also by their extremely high computational and storage&#10;costs. Specifically, due to their massive size, even inference for large,&#10;highly-accurate GPT models may require multiple performant GPUs, which limits&#10;the usability of such models. While there is emerging work on relieving this&#10;pressure via model compression, the applicability and performance of existing&#10;compression techniques is limited by the scale and complexity of GPT models. In&#10;this paper, we address this challenge, and propose GPTQ, a new one-shot weight&#10;quantization method based on approximate second-order information, that is both&#10;highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT&#10;models with 175 billion parameters in approximately four GPU hours, reducing&#10;the bitwidth down to 3 or 4 bits per weight, with negligible accuracy&#10;degradation relative to the uncompressed baseline. Our method more than doubles&#10;the compression gains relative to previously-proposed one-shot quantization&#10;methods, preserving accuracy, allowing us for the first time to execute an 175&#10;billion-parameter model inside a single GPU for generative inference. Moreover,&#10;we also show that our method can still provide reasonable accuracy in the&#10;extreme quantization regime, in which weights are quantized to 2-bit or even&#10;ternary quantization levels. We show experimentally that these improvements can&#10;be leveraged for end-to-end inference speedups over FP16, of around 3.25x when&#10;using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones&#10;(NVIDIA A6000). The implementation is available at&#10;https://github.com/IST-DASLab/gptq.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.10554" label="2307.10554">
        <attvalues>
          <attvalue for="0" value="EMQ: Evolving Training-free Proxies for Automated Mixed Precision&#10;  Quantization" />
          <attvalue for="1" value="  Mixed-Precision Quantization~(MQ) can achieve a competitive&#10;accuracy-complexity trade-off for models. Conventional training-based search&#10;methods require time-consuming candidate training to search optimized per-layer&#10;bit-width configurations in MQ. Recently, some training-free approaches have&#10;presented various MQ proxies and significantly improve search efficiency.&#10;However, the correlation between these proxies and quantization accuracy is&#10;poorly understood. To address the gap, we first build the MQ-Bench-101, which&#10;involves different bit configurations and quantization results. Then, we&#10;observe that the existing training-free proxies perform weak correlations on&#10;the MQ-Bench-101. To efficiently seek superior proxies, we develop an automatic&#10;search of proxies framework for MQ via evolving algorithms. In particular, we&#10;devise an elaborate search space involving the existing proxies and perform an&#10;evolution search to discover the best correlated MQ proxy. We proposed a&#10;diversity-prompting selection strategy and compatibility screening protocol to&#10;avoid premature convergence and improve search efficiency. In this way, our&#10;Evolving proxies for Mixed-precision Quantization~(EMQ) framework allows the&#10;auto-generation of proxies without heavy tuning and expert knowledge. Extensive&#10;experiments on ImageNet with various ResNet and MobileNet families demonstrate&#10;that our EMQ obtains superior performance than state-of-the-art mixed-precision&#10;methods at a significantly reduced cost. The code will be released.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.00566" label="2303.00566">
        <attvalues>
          <attvalue for="0" value="Structured Pruning for Deep Convolutional Neural Networks: A survey" />
          <attvalue for="1" value="  The remarkable performance of deep Convolutional neural networks (CNNs) is&#10;generally attributed to their deeper and wider architectures, which can come&#10;with significant computational costs. Pruning neural networks has thus gained&#10;interest since it effectively lowers storage and computational costs. In&#10;contrast to weight pruning, which results in unstructured models, structured&#10;pruning provides the benefit of realistic acceleration by producing models that&#10;are friendly to hardware implementation. The special requirements of structured&#10;pruning have led to the discovery of numerous new challenges and the&#10;development of innovative solutions. This article surveys the recent progress&#10;towards structured pruning of deep CNNs. We summarize and compare the&#10;state-of-the-art structured pruning techniques with respect to filter ranking&#10;methods, regularization methods, dynamic execution, neural architecture search,&#10;the lottery ticket hypothesis, and the applications of pruning. While&#10;discussing structured pruning algorithms, we briefly introduce the unstructured&#10;pruning counterpart to emphasize their differences. Furthermore, we provide&#10;insights into potential research opportunities in the field of structured&#10;pruning. A curated list of neural network pruning papers can be found at&#10;https://github.com/he-y/Awesome-Pruning . A dedicated website offering a more&#10;interactive comparison of structured pruning methods can be found at:&#10;https://huggingface.co/spaces/he-yang/Structured-Pruning-Survey .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11627" label="2305.11627">
        <attvalues>
          <attvalue for="0" value="LLM-Pruner: On the Structural Pruning of Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have shown remarkable capabilities in language&#10;understanding and generation. However, such impressive capability typically&#10;comes with a substantial model size, which presents significant challenges in&#10;both the deployment, inference, and training stages. With LLM being a&#10;general-purpose task solver, we explore its compression in a task-agnostic&#10;manner, which aims to preserve the multi-task solving and language generation&#10;ability of the original LLM. One challenge to achieving this is the enormous&#10;size of the training corpus of LLM, which makes both data transfer and model&#10;post-training over-burdensome. Thus, we tackle the compression of LLMs within&#10;the bound of two constraints: being task-agnostic and minimizing the reliance&#10;on the original training dataset. Our method, named LLM-Pruner, adopts&#10;structural pruning that selectively removes non-critical coupled structures&#10;based on gradient information, maximally preserving the majority of the LLM's&#10;functionality. To this end, the performance of pruned models can be efficiently&#10;recovered through tuning techniques, LoRA, in merely 3 hours, requiring only&#10;50K data. We validate the LLM-Pruner on three LLMs, including LLaMA, Vicuna,&#10;and ChatGLM, and demonstrate that the compressed models still exhibit&#10;satisfactory capabilities in zero-shot classification and generation. The code&#10;is available at: https://github.com/horseee/LLM-Pruner&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Compression of Language Model.&#10;Language models ~\cite{devlin2018bert,liu2019roberta,lewis2019bart} have gained much attention and increase the need to reduce the size of parameters and reduce the latency ~\cite{lanalbert,sun2020mobilebert}. To compress the language model, previous works can be divided into several categories: network pruning ~\cite{kurtic2022optimal,xu2021rethinking,liu2021ebert,ProximalPruning}, knowledge distillation ~\cite{sun2019patient,sun-etal-2020-contrastive,metakd}, quantization~\cite{yao2022zeroquant,bai2020binarybert,zafrir2019q8bert} and other techniques, like early exit ~\cite{xin-etal-2020-deebert} or dynamic token reduction ~\cite{ye-etal-2021-tr}. We focus on the pruning of the language models, especially structural pruning ~\cite{li2016pruning}. Structural pruning removes the entire filter from the neural network, which is more hardware friendly. There are several ways to remove the structure, such as l1-dependent pruning ~\cite{NIPS2015_pruning,zafrir2021prune}, first-order importance estimation ~\cite{hou2020dynabert}, hessian-based estimation~\cite{kurtic2022optimal,wang2019eigendamage} or the optimal brain surgeon~\cite{lecun1989optimal,kurtic2022optimal}. As for the pruning unit in structural pruning, some works adopt the entire layer ~\cite{fan2019reducing} as the minimal unit, and others take the multi-head attention ~\cite{voita2019analyzing} or the feed-forward layers ~\cite{hou2020dynabert,mccarley2019structured} as the basic structure to prune. CoFi ~\cite{xia2022structured} studies the pruning unit in different granularity. &#10;&#10;Efficient and Low Resource Compression. With the growing size of models, there is an increasing demand for efficient LLM compression and compression is independent of the original training data. As for the efficient compression, ~\cite{kwon2022fast} accelerate the post-training by defining the reconstruction error as a linear least squares problem. ~\cite{frantar2022gptq,frantar2023massive} propose the layer-wise optimal brain surgeon. As for the constraint of availability of the training corpus, data-free pruning ~\cite{srinivas2015data,yvinec2022red++} come up with several strategies to prune the model by measuring neurons' similarity. Besides, ~\cite{maprompting,ma2020adversarial,rashid2020zeroshot} proposes methods that distill the model without reliance on the training corpus of the model. However, those methods are too time-consuming, involving synthesizing samples by backpropagating the pre-trained language models.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2206.13329" label="2206.13329">
        <attvalues>
          <attvalue for="0" value="Prior-Guided One-shot Neural Architecture Search" />
          <attvalue for="1" value="  Neural architecture search methods seek optimal candidates with efficient&#10;weight-sharing supernet training. However, recent studies indicate poor ranking&#10;consistency about the performance between stand-alone architectures and&#10;shared-weight networks. In this paper, we present Prior-Guided One-shot NAS&#10;(PGONAS) to strengthen the ranking correlation of supernets. Specifically, we&#10;first explore the effect of activation functions and propose a balanced&#10;sampling strategy based on the Sandwich Rule to alleviate weight coupling in&#10;the supernet. Then, FLOPs and Zen-Score are adopted to guide the training of&#10;supernet with ranking correlation loss. Our PGONAS ranks 3rd place in the&#10;supernet Track Track of CVPR2022 Second lightweight NAS challenge. Code is&#10;available in&#10;https://github.com/pprp/CVPR2022-NAS?competition-Track1-3th-solution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.15678" label="2303.15678">
        <attvalues>
          <attvalue for="0" value="DisWOT: Student Architecture Search for Distillation WithOut Training" />
          <attvalue for="1" value="  Knowledge distillation (KD) is an effective training strategy to improve the&#10;lightweight student models under the guidance of cumbersome teachers. However,&#10;the large architecture difference across the teacher-student pairs limits the&#10;distillation gains. In contrast to previous adaptive distillation methods to&#10;reduce the teacher-student gap, we explore a novel training-free framework to&#10;search for the best student architectures for a given teacher. Our work first&#10;empirically show that the optimal model under vanilla training cannot be the&#10;winner in distillation. Secondly, we find that the similarity of feature&#10;semantics and sample relations between random-initialized teacher-student&#10;networks have good correlations with final distillation performances. Thus, we&#10;efficiently measure similarity matrixs conditioned on the semantic activation&#10;maps to select the optimal student via an evolutionary algorithm without any&#10;training. In this way, our student architecture search for Distillation WithOut&#10;Training (DisWOT) significantly improves the performance of the model in the&#10;distillation stage with at least 180$\times$ training acceleration.&#10;Additionally, we extend similarity metrics in DisWOT as new distillers and&#10;KD-based zero-proxies. Our experiments on CIFAR, ImageNet and NAS-Bench-201&#10;demonstrate that our technique achieves state-of-the-art results on different&#10;search spaces. Our project and code are available at&#10;https://lilujunai.github.io/DisWOT-CVPR2023/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.09850" label="2301.09850">
        <attvalues>
          <attvalue for="0" value="RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking&#10;  Distillation from Zero-cost Proxies" />
          <attvalue for="1" value="  Neural architecture search (NAS) has made tremendous progress in the&#10;automatic design of effective neural network structures but suffers from a&#10;heavy computational burden. One-shot NAS significantly alleviates the burden&#10;through weight sharing and improves computational efficiency. Zero-shot NAS&#10;further reduces the cost by predicting the performance of the network from its&#10;initial state, which conducts no training. Both methods aim to distinguish&#10;between &quot;good&quot; and &quot;bad&quot; architectures, i.e., ranking consistency of predicted&#10;and true performance. In this paper, we propose Ranking Distillation one-shot&#10;NAS (RD-NAS) to enhance ranking consistency, which utilizes zero-cost proxies&#10;as the cheap teacher and adopts the margin ranking loss to distill the ranking&#10;knowledge. Specifically, we propose a margin subnet sampler to distill the&#10;ranking knowledge from zero-shot NAS to one-shot NAS by introducing Group&#10;distance as margin. Our evaluation of the NAS-Bench-201 and ResNet-based search&#10;space demonstrates that RD-NAS achieve 10.7\% and 9.65\% improvements in&#10;ranking ability, respectively. Our codes are available at&#10;https://github.com/pprp/CVPR2022-NAS-competition-Track1-3th-solution&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.04785" label="2007.04785">
        <attvalues>
          <attvalue for="0" value="Accuracy Prediction with Non-neural Model for Neural Architecture Search" />
          <attvalue for="1" value="  Neural architecture search (NAS) with an accuracy predictor that predicts the&#10;accuracy of candidate architectures has drawn increasing attention due to its&#10;simplicity and effectiveness. Previous works usually employ neural&#10;network-based predictors which require more delicate design and are easy to&#10;overfit. Considering that most architectures are represented as sequences of&#10;discrete symbols which are more like tabular data and preferred by non-neural&#10;predictors, in this paper, we study an alternative approach which uses&#10;non-neural model for accuracy prediction. Specifically, as decision tree based&#10;models can better handle tabular data, we leverage gradient boosting decision&#10;tree (GBDT) as the predictor for NAS. We demonstrate that the GBDT predictor&#10;can achieve comparable (if not better) prediction accuracy than neural network&#10;based predictors. Moreover, considering that a compact search space can ease&#10;the search process, we propose to prune the search space gradually according to&#10;important features derived from GBDT. In this way, NAS can be performed by&#10;first pruning the search space and then searching a neural architecture, which&#10;is more efficient and effective. Experiments on NASBench-101 and ImageNet&#10;demonstrate the effectiveness of using GBDT as predictor for NAS: (1) On&#10;NASBench-101, it is 22x, 8x, and 6x more sample efficient than random search,&#10;regularized evolution, and Monte Carlo Tree Search (MCTS) in finding the global&#10;optimum; (2) It achieves 24.2% top-1 error rate on ImageNet, and further&#10;achieves 23.4% top-1 error rate on ImageNet when enhanced with search space&#10;pruning. Code is provided at https://github.com/renqianluo/GBDT-NAS.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.07255" label="2202.07255">
        <attvalues>
          <attvalue for="0" value="Enhancing Cross-lingual Prompting with Dual Prompt Augmentation" />
          <attvalue for="1" value="  Prompting shows promising results in few-shot scenarios. However, its&#10;strength for multilingual/cross-lingual problems has not been fully exploited.&#10;Zhao and Sch\&quot;utze (2021) made initial explorations in this direction by&#10;presenting that cross-lingual prompting outperforms cross-lingual finetuning.&#10;In this paper, we conduct an empirical exploration on the effect of each&#10;component in cross-lingual prompting and derive language-agnostic Universal&#10;Prompting, which helps alleviate the discrepancies between source-language&#10;training and target-language inference. Based on this, we propose DPA, a dual&#10;prompt augmentation framework, aiming at relieving the data scarcity issue in&#10;few-shot cross-lingual prompting. Notably, for XNLI, our method achieves 46.54%&#10;with only 16 English training examples per class, significantly better than&#10;34.99% of finetuning. Our code is available at&#10;https://github.com/DAMO-NLP-SG/DPA.&#10;" />
          <attvalue for="2" value="&#10;&#10;Although adapting Pre-trained Language Models (PLMs)~\cite{devlin-etal-2019-bert} to downstream NLP tasks via finetuning is the de facto mainstream paradigm under fully supervised settings~\cite{wang-etal-2018-glue}, prompting~\cite{gao-etal-2021-making,Radford2019LanguageMA,NEURIPS2020_1457c0d6,schick-schutze-2021-exploiting,schick-schutze-2021-just} has demonstrated its superiority over finetuning in low-resource scenarios.&#10;Typically, prompting reformulates the classification task as a language modeling problem over manually-designed natural language prompts.&#10;&#10;Despite the effectiveness of prompting on English tasks, its potential for cross-lingual problems, which assume the availability of the training data in high-resource languages (e.g., English) only, is still under-explored. \cite{zhao-schutze-2021-discrete} is the pioneering work to apply prompting to cross-lingual NLP. However, their major efforts are spent on comparing different training strategies for cross-lingual prompting such as discrete prompting and soft prompting. They do not fully investigate the design choice of key components in prompting, i.e., prompt template and verbalizer.&#10;&#10;To provide a practical guide for designing cross-lingual prompting, we first conduct an empirical analysis to explore the effects of each prompting component on the performance of cross-lingual transfer. &#10;Our preliminary study shows that template-free prompting combined with English-only inference, dubbed as language-agnostic ``Universal Prompting'' (UP) in this paper, generally performs well across different few-shot settings. &#10;Intuitively, UP avoids the discrepancies between the source-language training and the target-language inference, which intrinsically better fits cross-lingual tasks.&#10;&#10;The derived UP is a concise solution with reasonable performance but does not take advantage of other available resources in the context of multilingual problems, e.g., the translation of verbalizers in target languages. Motivated by this fact, we propose a Dual Prompt Augmentation (DPA) framework to alleviate the data scarcity issue in few-shot scenarios. Firstly, &#10;we introduce multilingual verbalizers as answer augmentation for prompting, where the translated label tokens are treated as additional target-language supervision. &#10;Secondly, we propose prompt mixup as prompt input augmentation, which mixes the prompt representations in each batch. Intuitively, given two prompt representations on real data, we can generate a virtual representation based on their interpolation, which encodes the semantics in between. Our DPA framework is not task-dependent and does not require either external unlabeled data~\cite{xie2020unsupervised} or massive text manipulation efforts~\cite{wei-zou-2019-eda} compared with other data augmentation approaches.&#10;&#10;In summary, our contributions are as follows:&#10;\begin{itemize}[leftmargin=3mm]&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Computer Science, Machine Learning, Linguistics, Multilingual Language Models, Cross-Lingual Prompting, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2012.15723" label="2012.15723">
        <attvalues>
          <attvalue for="0" value="Making Pre-trained Language Models Better Few-shot Learners" />
          <attvalue for="1" value="  The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot&#10;performance solely by leveraging a natural-language prompt and a few task&#10;demonstrations as input context. Inspired by their findings, we study few-shot&#10;learning in a more practical scenario, where we use smaller language models for&#10;which fine-tuning is computationally efficient. We present LM-BFF--better&#10;few-shot fine-tuning of language models--a suite of simple and complementary&#10;techniques for fine-tuning language models on a small number of annotated&#10;examples. Our approach includes (1) prompt-based fine-tuning together with a&#10;novel pipeline for automating prompt generation; and (2) a refined strategy for&#10;dynamically and selectively incorporating demonstrations into each context.&#10;Finally, we present a systematic evaluation for analyzing few-shot performance&#10;on a range of NLP tasks, including classification and regression. Our&#10;experiments demonstrate that our methods combine to dramatically outperform&#10;standard fine-tuning procedures in this low resource setting, achieving up to&#10;30% absolute improvement, and 11% on average across all tasks. Our approach&#10;makes minimal assumptions on task resources and domain expertise, and hence&#10;constitutes a strong task-agnostic method for few-shot learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.07118" label="2009.07118">
        <attvalues>
          <attvalue for="0" value="It's Not Just Size That Matters: Small Language Models Are Also Few-Shot&#10;  Learners" />
          <attvalue for="1" value="  When scaled to hundreds of billions of parameters, pretrained language models&#10;such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance.&#10;However, enormous amounts of compute are required for training and applying&#10;such big models, resulting in a large carbon footprint and making it difficult&#10;for researchers and practitioners to use them. We show that performance similar&#10;to GPT-3 can be obtained with language models that are much &quot;greener&quot; in that&#10;their parameter count is several orders of magnitude smaller. This is achieved&#10;by converting textual inputs into cloze questions that contain a task&#10;description, combined with gradient-based optimization; exploiting unlabeled&#10;data gives further improvements. We identify key factors required for&#10;successful natural language understanding with small language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.12848" label="1904.12848">
        <attvalues>
          <attvalue for="0" value="Unsupervised Data Augmentation for Consistency Training" />
          <attvalue for="1" value="  Semi-supervised learning lately has shown much promise in improving deep&#10;learning models when labeled data is scarce. Common among recent approaches is&#10;the use of consistency training on a large amount of unlabeled data to&#10;constrain model predictions to be invariant to input noise. In this work, we&#10;present a new perspective on how to effectively noise unlabeled examples and&#10;argue that the quality of noising, specifically those produced by advanced data&#10;augmentation methods, plays a crucial role in semi-supervised learning. By&#10;substituting simple noising operations with advanced data augmentation methods&#10;such as RandAugment and back-translation, our method brings substantial&#10;improvements across six language and three vision tasks under the same&#10;consistency training framework. On the IMDb text classification dataset, with&#10;only 20 labeled examples, our method achieves an error rate of 4.20,&#10;outperforming the state-of-the-art model trained on 25,000 labeled examples. On&#10;a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms&#10;all previous approaches and achieves an error rate of 5.43 with only 250&#10;examples. Our method also combines well with transfer learning, e.g., when&#10;finetuning from BERT, and yields improvements in high-data regime, such as&#10;ImageNet, whether when there is only 10% labeled data or when a full labeled&#10;set with 1.3M extra unlabeled examples is used. Code is available at&#10;https://github.com/google-research/uda.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.11196" label="1901.11196">
        <attvalues>
          <attvalue for="0" value="EDA: Easy Data Augmentation Techniques for Boosting Performance on Text&#10;  Classification Tasks" />
          <attvalue for="1" value="  We present EDA: easy data augmentation techniques for boosting performance on&#10;text classification tasks. EDA consists of four simple but powerful operations:&#10;synonym replacement, random insertion, random swap, and random deletion. On&#10;five text classification tasks, we show that EDA improves performance for both&#10;convolutional and recurrent neural networks. EDA demonstrates particularly&#10;strong results for smaller datasets; on average, across five datasets, training&#10;with EDA while using only 50% of the available training set achieved the same&#10;accuracy as normal training with all available data. We also performed&#10;extensive ablation studies and suggest parameters for practical use.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.06483" label="2407.06483">
        <attvalues>
          <attvalue for="0" value="Composable Interventions for Language Models" />
          <attvalue for="1" value="  Test-time interventions for language models can enhance factual accuracy,&#10;mitigate harmful outputs, and improve model efficiency without costly&#10;retraining. But despite a flood of new methods, different types of&#10;interventions are largely developing independently. In practice, multiple&#10;interventions must be applied sequentially to the same model, yet we lack&#10;standardized ways to study how interventions interact. We fill this gap by&#10;introducing composable interventions, a framework to study the effects of using&#10;multiple interventions on the same language models, featuring new metrics and a&#10;unified codebase. Using our framework, we conduct extensive experiments and&#10;compose popular methods from three emerging intervention categories --&#10;Knowledge Editing, Model Compression, and Machine Unlearning. Our results from&#10;310 different compositions uncover meaningful interactions: compression hinders&#10;editing and unlearning, composing interventions hinges on their order of&#10;application, and popular general-purpose metrics are inadequate for assessing&#10;composability. Taken together, our findings showcase clear gaps in&#10;composability, suggesting a need for new multi-objective interventions. All of&#10;our code is public:&#10;https://github.com/hartvigsen-group/composable-interventions.&#10;" />
          <attvalue for="2" value="&#10;Language models (LMs) exhibit striking capabilities on important tasks in myriad domains including medicine \cite{singhal2023towards}, finance~\cite{wu2023bloomberggpt}, science~\cite{taylor2022galactica}, and entertainment~\cite{zhong2023let}. But despite high performance, deployed LMs can still misbehave unpredictably and require updates.&#10;For example, LMs generate content that is hallucinatory \cite{ji2023survey}, factually incorrect \cite{zhao2023felm}, and harmful \cite{mendelsohn2023dogwhistles,jain2024polyglotoxicityprompts,Hartvigsen2022ToxiGenAL}.&#10;Beyond unwanted behaviors, user requirements also change over time.&#10;For example, regulations arise \cite{USA2023}, computational resources constrict, knowledge gets outdated \cite{tack2024online}, and copyrighted training materials are identified \cite{grynbaum2023times}.&#10;Without ways to quickly address these issues, models can be left miscalibrated, outdated, and biased, limiting their widespread responsible use \cite{kaddour2023challenges}.&#10;&#10;Many recent works study efficient, in-place updates for LMs.&#10;We broadly refer to these as interventions---updates to targeted properties of LMs applied after pretraining (and optional fine-tuning).&#10;For example, we can view model compression \cite{zhu2023survey,frantar2023gptq,frantar2023sparsegpt} as an intervention to make language models more inference- or memory-efficient.&#10;Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}.&#10;However, these interventions are largely advancing independently.&#10;In practice, we usually have multiple requirements for properties of our models (e.g., factuality and efficiency).&#10;And as requirements change over time, new interventions must be applied.&#10;While some works have started studying interactions between training objectives \cite{matzken2023trade,xu2023compress,li2024loftq}, practical widespread use is limited without unified evaluations for how interventions interact.&#10;&#10;As depicted in Figure \ref{fig:title_fig}, we propose that practical interventions should be composable: When an intervention is applied to a model, it should not interfere with prior or future interventions.&#10;For example, we might repair a factual error in an LM using a knowledge editing method.&#10;Later on, we might quantize that model.&#10;A composable quantization method should specifically preserve the edited fact and a composable editor should withstand quantization.&#10;However, it remains unknown how well existing interventions compose, and we lack formal notions of intervention composition.&#10;We therefore propose two metrics for composability: 1) \orderfree, where an intervention is composable if its application leaves others' success unimpacted, and 2) \order, where the combined success of multiple interventions should not depend on the order in which they are applied.&#10;We instantiate these metrics in a codebase that includes popular interventions, tackling a major lack of standardized code that has inhibited cross-intervention evaluations until now.&#10;&#10;We use our framework to extensively study composability of state-of-the-art knowledge editing, model compression, and machine unlearning interventions on Llama3-8B \cite{touvron2023llama}.&#10;Our experiments with 310 different compositions unearth novel insights and offer guidance for composing recent methods.&#10;Three key results are as follows: &#10;\circled{1} Model compression often limits the success of other interventions.&#10;This suggests a significant drawback of existing compression methods, which are nearly universal in LM deployments.&#10;\circled{2} The order in which interventions are applied dramatically alters their success.&#10;Therefore, we need new interventions explicitly designed for composability.&#10;\circled{3} General-purpose post-intervention model performance is a poor proxy for composability.&#10;This indicates that targeting composability as a metric has the potential to drive the development of new, practically-grounded interventions.&#10;Taken alongside our other findings (Section \ref{sec:main_results}), our experiments suggest we sorely need to broaden intervention method evaluations and to design new methods explicitly for composability.&#10;&#10;Our key contributions are as follows:&#10;\begin{itemize}&#10; \item We introduce the notion of composability to language model interventions, formalizing the sequential applications of multiple interventions to LMs. Our work opens doors to address crucial, practical challenges in online LM updates and broadens evaluation criteria for interventions.&#10; \item Our main contribution is extensive experimentation, identifying unknown interactions between knowledge editing, model compression, and machine unlearning.&#10; \item Our findings suggest a clear need to develop novel interventions that target composability, a crucial property of practical interventions.&#10; \item We release an extendable codebase that unifies many state-of-the-art implementations to enable others to develop new multi-objective interventions.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Composability Analysis, Computer Science, Linguistics, Model Efficiency Optimization, Artificial Intelligence, Mathematics, Natural Language Processing, Language Model Interventions" />
        </attvalues>
      </node>
      <node id="2303.17564" label="2303.17564">
        <attvalues>
          <attvalue for="0" value="BloombergGPT: A Large Language Model for Finance" />
          <attvalue for="1" value="  The use of NLP in the realm of financial technology is broad and complex,&#10;with applications ranging from sentiment analysis and named entity recognition&#10;to question answering. Large Language Models (LLMs) have been shown to be&#10;effective on a variety of tasks; however, no LLM specialized for the financial&#10;domain has been reported in literature. In this work, we present BloombergGPT,&#10;a 50 billion parameter language model that is trained on a wide range of&#10;financial data. We construct a 363 billion token dataset based on Bloomberg's&#10;extensive data sources, perhaps the largest domain-specific dataset yet,&#10;augmented with 345 billion tokens from general purpose datasets. We validate&#10;BloombergGPT on standard LLM benchmarks, open financial benchmarks, and a suite&#10;of internal benchmarks that most accurately reflect our intended usage. Our&#10;mixed dataset training leads to a model that outperforms existing models on&#10;financial tasks by significant margins without sacrificing performance on&#10;general LLM benchmarks. Additionally, we explain our modeling choices, training&#10;process, and evaluation methodology. We release Training Chronicles (Appendix&#10;C) detailing our experience in training BloombergGPT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.09085" label="2211.09085">
        <attvalues>
          <attvalue for="0" value="Galactica: A Large Language Model for Science" />
          <attvalue for="1" value="  Information overload is a major obstacle to scientific progress. The&#10;explosive growth in scientific literature and data has made it ever harder to&#10;discover useful insights in a large mass of information. Today scientific&#10;knowledge is accessed through search engines, but they are unable to organize&#10;scientific knowledge alone. In this paper we introduce Galactica: a large&#10;language model that can store, combine and reason about scientific knowledge.&#10;We train on a large scientific corpus of papers, reference material, knowledge&#10;bases and many other sources. We outperform existing models on a range of&#10;scientific tasks. On technical knowledge probes such as LaTeX equations,&#10;Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also&#10;performs well on reasoning, outperforming Chinchilla on mathematical MMLU by&#10;41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It&#10;also sets a new state-of-the-art on downstream tasks such as PubMedQA and&#10;MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general&#10;corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these&#10;results demonstrate the potential for language models as a new interface for&#10;science. We open source the model for the benefit of the scientific community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.02439" label="2312.02439">
        <attvalues>
          <attvalue for="0" value="Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language&#10;  Models with Creative Humor Generation" />
          <attvalue for="1" value="  Chain-of-Thought (CoT) guides large language models (LLMs) to reason&#10;step-by-step, and can motivate their logical reasoning ability. While effective&#10;for logical tasks, CoT is not conducive to creative problem-solving which often&#10;requires out-of-box thoughts and is crucial for innovation advancements. In&#10;this paper, we explore the Leap-of-Thought (LoT) abilities within LLMs -- a&#10;non-sequential, creative paradigm involving strong associations and knowledge&#10;leaps. To this end, we study LLMs on the popular Oogiri game which needs&#10;participants to have good creativity and strong associative thinking for&#10;responding unexpectedly and humorously to the given image, text, or both, and&#10;thus is suitable for LoT study. Then to investigate LLMs' LoT ability in the&#10;Oogiri game, we first build a multimodal and multilingual Oogiri-GO dataset&#10;which contains over 130,000 samples from the Oogiri game, and observe the&#10;insufficient LoT ability or failures of most existing LLMs on the Oogiri game.&#10;Accordingly, we introduce a creative Leap-of-Thought (CLoT) paradigm to improve&#10;LLM's LoT ability. CLoT first formulates the Oogiri-GO dataset into&#10;LoT-oriented instruction tuning data to train pretrained LLM for achieving&#10;certain LoT humor generation and discrimination abilities. Then CLoT designs an&#10;explorative self-refinement that encourages the LLM to generate more creative&#10;LoT data via exploring parallels between seemingly unrelated concepts and&#10;selects high-quality data to train itself for self-refinement. CLoT not only&#10;excels in humor generation in the Oogiri game but also boosts creative&#10;abilities in various tasks like cloud guessing game and divergent association&#10;task. These findings advance our understanding and offer a pathway to improve&#10;LLMs' creative capacities for innovative applications across domains. The&#10;dataset, code, and models will be released online.&#10;https://zhongshsh.github.io/CLoT/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.03629" label="2202.03629">
        <attvalues>
          <attvalue for="0" value="Survey of Hallucination in Natural Language Generation" />
          <attvalue for="1" value="  Natural Language Generation (NLG) has improved exponentially in recent years&#10;thanks to the development of sequence-to-sequence deep learning technologies&#10;such as Transformer-based language models. This advancement has led to more&#10;fluent and coherent NLG, leading to improved development in downstream tasks&#10;such as abstractive summarization, dialogue generation and data-to-text&#10;generation. However, it is also apparent that deep learning based generation is&#10;prone to hallucinate unintended text, which degrades the system performance and&#10;fails to meet user expectations in many real-world scenarios. To address this&#10;issue, many studies have been presented in measuring and mitigating&#10;hallucinated texts, but these have never been reviewed in a comprehensive&#10;manner before. In this survey, we thus provide a broad overview of the research&#10;progress and challenges in the hallucination problem in NLG. The survey is&#10;organized into two parts: (1) a general overview of metrics, mitigation&#10;methods, and future directions; (2) an overview of task-specific research&#10;progress on hallucinations in the following downstream tasks, namely&#10;abstractive summarization, dialogue generation, generative question answering,&#10;data-to-text generation, machine translation, and visual-language generation;&#10;and (3) hallucinations in large language models (LLMs). This survey serves to&#10;facilitate collaborative efforts among researchers in tackling the challenge of&#10;hallucinated texts in NLG.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.00741" label="2310.00741">
        <attvalues>
          <attvalue for="0" value="FELM: Benchmarking Factuality Evaluation of Large Language Models" />
          <attvalue for="1" value="  Assessing factuality of text generated by large language models (LLMs) is an&#10;emerging yet crucial research area, aimed at alerting users to potential errors&#10;and guiding the development of more reliable LLMs. Nonetheless, the evaluators&#10;assessing factuality necessitate suitable evaluation themselves to gauge&#10;progress and foster advancements. This direction remains under-explored,&#10;resulting in substantial impediments to the progress of factuality evaluators.&#10;To mitigate this issue, we introduce a benchmark for Factuality Evaluation of&#10;large Language Models, referred to as felm. In this benchmark, we collect&#10;responses generated from LLMs and annotate factuality labels in a fine-grained&#10;manner. Contrary to previous studies that primarily concentrate on the&#10;factuality of world knowledge (e.g.~information from Wikipedia), felm focuses&#10;on factuality across diverse domains, spanning from world knowledge to math and&#10;reasoning. Our annotation is based on text segments, which can help pinpoint&#10;specific factual errors. The factuality annotations are further supplemented by&#10;predefined error types and reference links that either support or contradict&#10;the statement. In our experiments, we investigate the performance of several&#10;LLM-based factuality evaluators on felm, including both vanilla LLMs and those&#10;augmented with retrieval mechanisms and chain-of-thought processes. Our&#10;findings reveal that while retrieval aids factuality evaluation, current LLMs&#10;are far from satisfactory to faithfully detect factual errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.17174" label="2305.17174">
        <attvalues>
          <attvalue for="0" value="From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language&#10;  Models" />
          <attvalue for="1" value="  Dogwhistles are coded expressions that simultaneously convey one meaning to a&#10;broad audience and a second one, often hateful or provocative, to a narrow&#10;in-group; they are deployed to evade both political repercussions and&#10;algorithmic content moderation. For example, in the sentence 'we need to end&#10;the cosmopolitan experiment,' the word 'cosmopolitan' likely means 'worldly' to&#10;many, but secretly means 'Jewish' to a select few. We present the first&#10;large-scale computational investigation of dogwhistles. We develop a typology&#10;of dogwhistles, curate the largest-to-date glossary of over 300 dogwhistles&#10;with rich contextual information and examples, and analyze their usage in&#10;historical U.S. politicians' speeches. We then assess whether a large language&#10;model (GPT-3) can identify dogwhistles and their meanings, and find that&#10;GPT-3's performance varies widely across types of dogwhistles and targeted&#10;groups. Finally, we show that harmful content containing dogwhistles avoids&#10;toxicity detection, highlighting online risks of such coded language. This work&#10;sheds light on the theoretical and applied importance of dogwhistles in both&#10;NLP and computational social science, and provides resources for future&#10;research in modeling dogwhistles and mitigating their online harms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.09373" label="2405.09373">
        <attvalues>
          <attvalue for="0" value="PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic&#10;  Degeneration in Large Language Models" />
          <attvalue for="1" value="  Recent advances in large language models (LLMs) have led to their extensive&#10;global deployment, and ensuring their safety calls for comprehensive and&#10;multilingual toxicity evaluations. However, existing toxicity benchmarks are&#10;overwhelmingly focused on English, posing serious risks to deploying LLMs in&#10;other languages. We address this by introducing PolygloToxicityPrompts (PTP),&#10;the first large-scale multilingual toxicity evaluation benchmark of 425K&#10;naturally occurring prompts spanning 17 languages. We overcome the scarcity of&#10;naturally occurring toxicity in web-text and ensure coverage across languages&#10;with varying resources by automatically scraping over 100M web-text documents.&#10;Using PTP, we investigate research questions to study the impact of model size,&#10;prompt language, and instruction and preference-tuning methods on toxicity by&#10;benchmarking over 60 LLMs. Notably, we find that toxicity increases as language&#10;resources decrease or model size increases. Although instruction- and&#10;preference-tuning reduce toxicity, the choice of preference-tuning method does&#10;not have any significant impact. Our findings shed light on crucial&#10;shortcomings of LLM safeguarding and highlight areas for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Evaluating Toxicity using Web-text Corpora, Templates, And User-AI Interaction Data&#10;Early works on evaluation datasets for studying biases and toxicity in models were created using templates or scraping web-text corpora. \cite{sheng-etal-2019-woman, nangia-etal-2020-crows, nadeem-etal-2021-stereoset} use templated prompts to study social biases in pretrained language models. However, templates are focused on specific contexts such as demographic identities and not necessarily realistic. Thus, \cite{gehman-etal-2020-realtoxicityprompts} create RealToxicityPrompts by crawling English web-text for naturally occurring input prompts to evaluate toxicity in a sentence completion setting. &#10;&#10;More recently, there has been a shift towards examining toxicity in input-response settings. \cite{10.1145/3548606.3560599, baheti-etal-2021-just} use generations from dialogue models like DialoGPT \cite{zhang2019dialogpt} to study toxic degenerations in chatbots. Furthermore, the advent of instruction-tuned LLMs has led to studies of toxicity in real-world user-AI conversations. \cite{zheng2024realchatm} and \cite{lin-etal-2023-toxicchat} collect user-AI interactions with automatic and manual toxicity annotations respectively to tackle a different toxic data distribution---namely instructions. However, most of these approaches are limited to English.&#10;&#10;Evaluating Multilingual Toxicity&#10;Multilingual dataset curation for evaluating toxicity has utilized both manual and automated translation techniques. Recent work on AI safety evaluation \cite{wang2023all, Yong2023LowResourceLJ, Deng2023MASTERKEYAJ} create multilingual safety benchmarks by translating monolingual benchmarks into other languages. They observe that LLMs are primarily safeguarded for English, leading to significantly unsafe generations in other languages, especially as availability of languages decreases. While these works are aimed towards the broader area of safety, the absence of a standard multilingual toxicity evaluation benchmark has also led researchers to translate prompts from RealToxicityPrompts into other languages, either automatically \cite{ustun2024aya} or using human annotations \cite{dewynter2024rtplx}. However, manual translations are expensive, not scalable, and can introduce cultural biases, whereas automated translations can introduce deviations in toxicity due to incorrect translations and hallucinations \cite{specia-etal-2021-findings, sharou-specia-2022-taxonomy, nllbteam2022language, costa-jussa-etal-2023-toxicity}. &#10;&#10;Evaluating Toxicity using Machine-Generated Approaches &#10;Besides human-generated or naturally occurring data, a wealth of recent work has explored using machine-generated approaches to curate datasets and methods for evaluating the toxicity and safety of LLMs. \cite{hartvigsen-etal-2022-toxigen} and \cite{kim-etal-2022-prosocialdialog} generate adversarial prompts about minority groups using classifier-guided decoding and conversations with a toxic partner respectively. Extensive research has studied red teaming \cite{perez-etal-2022-red, chao2023jailbreaking, mazeika2024harmbench} and jailbreaking \cite{liu2023autodan, wei2024jailbroken, yu2023gptfuzzer, Deng2023MASTERKEYAJ} to identify safety failures in LLMs and elicit harmful outputs. Furthermore, adversarial attack methods have also been shown to be effective against models without requiring substantial prompt engineering \cite{shin-etal-2020-autoprompt, zou2023universal, huang2023catastrophic, pmlr-v202-jones23a}. However, such methods involve extensive prompt engineering, often leading to unnatural and non-representative prompts or model-specific artifacts \cite{das2024under}. Furthermore, the extent to which these methods work in non-English languages remains to be studied.&#10;&#10;While the literature on toxicity evaluation has grown rapidly, their predominant focus on English highlights the need for multilingual benchmarks on naturally occurring toxic input prompts. We address this gap with PolygloToxicityPrompts, a collection of 425K naturally occurring prompts across 17 languages for evaluating toxicity.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2203.09509" label="2203.09509">
        <attvalues>
          <attvalue for="0" value="ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and&#10;  Implicit Hate Speech Detection" />
          <attvalue for="1" value="  Toxic language detection systems often falsely flag text that contains&#10;minority group mentions as toxic, as those groups are often the targets of&#10;online hate. Such over-reliance on spurious correlations also causes systems to&#10;struggle with detecting implicitly toxic language. To help mitigate these&#10;issues, we create ToxiGen, a new large-scale and machine-generated dataset of&#10;274k toxic and benign statements about 13 minority groups. We develop a&#10;demonstration-based prompting framework and an adversarial&#10;classifier-in-the-loop decoding method to generate subtly toxic and benign text&#10;with a massive pretrained language model. Controlling machine generation in&#10;this way allows ToxiGen to cover implicitly toxic text at a larger scale, and&#10;about more demographic groups, than previous resources of human-written text.&#10;We conduct a human evaluation on a challenging subset of ToxiGen and find that&#10;annotators struggle to distinguish machine-generated text from human-written&#10;language. We also find that 94.5% of toxic examples are labeled as hate speech&#10;by human annotators. Using three publicly-available datasets, we show that&#10;finetuning a toxicity classifier on our data improves its performance on&#10;human-written data substantially. We also demonstrate that ToxiGen can be used&#10;to fight machine-generated toxicity as finetuning improves the classifier&#10;significantly on our evaluation subset. Our code and data can be found at&#10;https://github.com/microsoft/ToxiGen.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.04317" label="2403.04317">
        <attvalues>
          <attvalue for="0" value="Online Adaptation of Language Models with a Memory of Amortized Contexts" />
          <attvalue for="1" value="  Due to the rapid generation and dissemination of information, large language&#10;models (LLMs) quickly run out of date despite enormous development costs. Due&#10;to this crucial need to keep models updated, online learning has emerged as a&#10;critical necessity when utilizing LLMs for real-world applications. However,&#10;given the ever-expanding corpus of unseen documents and the large parameter&#10;space of modern LLMs, efficient adaptation is essential. To address these&#10;challenges, we propose Memory of Amortized Contexts (MAC), an efficient and&#10;effective online adaptation framework for LLMs with strong knowledge retention.&#10;We propose an amortized feature extraction and memory-augmentation approach to&#10;compress and extract information from new documents into compact modulations&#10;stored in a memory bank. When answering questions, our model attends to and&#10;extracts relevant knowledge from this memory bank. To learn informative&#10;modulations in an efficient manner, we utilize amortization-based&#10;meta-learning, which substitutes the optimization process with a single forward&#10;pass of the encoder. Subsequently, we learn to choose from and aggregate&#10;selected documents into a single modulation by conditioning on the question,&#10;allowing us to adapt a frozen language model during test time without requiring&#10;further gradient updates. Our experiment demonstrates the superiority of MAC in&#10;multiple aspects, including online adaptation performance, time, and memory&#10;efficiency. Code is available at: https://github.com/jihoontack/MAC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.10169" label="2307.10169">
        <attvalues>
          <attvalue for="0" value="Challenges and Applications of Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) went from non-existent to ubiquitous in the&#10;machine learning discourse within a few years. Due to the fast pace of the&#10;field, it is difficult to identify the remaining challenges and already&#10;fruitful application areas. In this paper, we aim to establish a systematic set&#10;of open problems and application successes so that ML researchers can&#10;comprehend the field's current state more quickly and become productive.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.07633" label="2308.07633">
        <attvalues>
          <attvalue for="0" value="A Survey on Model Compression for Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have transformed natural language processing&#10;tasks successfully. Yet, their large size and high computational needs pose&#10;challenges for practical use, especially in resource-limited settings. Model&#10;compression has emerged as a key research area to address these challenges.&#10;This paper presents a survey of model compression techniques for LLMs. We cover&#10;methods like quantization, pruning, and knowledge distillation, highlighting&#10;recent advancements. We also discuss benchmarking strategies and evaluation&#10;metrics crucial for assessing compressed LLMs. This survey offers valuable&#10;insights for researchers and practitioners, aiming to enhance efficiency and&#10;real-world applicability of LLMs while laying a foundation for future&#10;advancements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.00774" label="2301.00774">
        <attvalues>
          <attvalue for="0" value="SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot" />
          <attvalue for="1" value="  We show for the first time that large-scale generative pretrained transformer&#10;(GPT) family models can be pruned to at least 50% sparsity in one-shot, without&#10;any retraining, at minimal loss of accuracy. This is achieved via a new pruning&#10;method called SparseGPT, specifically designed to work efficiently and&#10;accurately on massive GPT-family models. We can execute SparseGPT on the&#10;largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5&#10;hours, and can reach 60% unstructured sparsity with negligible increase in&#10;perplexity: remarkably, more than 100 billion weights from these models can be&#10;ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and&#10;4:8) patterns, and is compatible with weight quantization approaches. The code&#10;is available at: https://github.com/IST-DASLab/sparsegpt.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.11031" label="2211.11031">
        <attvalues>
          <attvalue for="0" value="Aging with GRACE: Lifelong Model Editing with Discrete Key-Value&#10;  Adaptors" />
          <attvalue for="1" value="  Deployed language models decay over time due to shifting inputs, changing&#10;user needs, or emergent world-knowledge gaps. When such problems are&#10;identified, we want to make targeted edits while avoiding expensive retraining.&#10;However, current model editors, which modify such behaviors of pre-trained&#10;models, degrade model performance quickly across multiple, sequential edits. We&#10;propose GRACE, a lifelong model editing method, which implements spot-fixes on&#10;streaming errors of a deployed model, ensuring minimal impact on unrelated&#10;inputs. GRACE writes new mappings into a pre-trained model's latent space,&#10;creating a discrete, local codebook of edits without altering model weights.&#10;This is the first method enabling thousands of sequential edits using only&#10;streaming errors. Our experiments on T5, BERT, and GPT models show GRACE's&#10;state-of-the-art performance in making and retaining edits, while generalizing&#10;to unseen inputs. Our code is available at&#10;https://www.github.com/thartvigsen/grace}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.07229" label="2210.07229">
        <attvalues>
          <attvalue for="0" value="Mass-Editing Memory in a Transformer" />
          <attvalue for="1" value="  Recent work has shown exciting promise in updating large language models with&#10;new memories, so as to replace obsolete information or add specialized&#10;knowledge. However, this line of work is predominantly limited to updating&#10;single associations. We develop MEMIT, a method for directly updating a&#10;language model with many memories, demonstrating experimentally that it can&#10;scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B),&#10;exceeding prior work by orders of magnitude. Our code and data are at&#10;https://memit.baulab.info.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.11795" label="2312.11795">
        <attvalues>
          <attvalue for="0" value="MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA" />
          <attvalue for="1" value="  Large language models (LLMs) have shown great success in various Natural&#10;Language Processing (NLP) tasks, whist they still need updates after deployment&#10;to fix errors or keep pace with the changing knowledge in the world.&#10;Researchers formulate such problem as Model Editing and have developed various&#10;editors focusing on different axes of editing properties. However, current&#10;editors can hardly support all properties and rely on heavy computational&#10;resources. In this paper, we propose a plug-in Model Editing method based on&#10;neuron-indexed dynamic LoRA (MELO), which alters the behavior of language&#10;models by dynamically activating certain LoRA blocks according to the index&#10;built in an inner vector database. Our method satisfies various editing&#10;properties with high efficiency and can be easily integrated into multiple LLM&#10;backbones. Experimental results show that our proposed MELO achieves&#10;state-of-the-art editing performance on three sequential editing tasks&#10;(document classification, question answering and hallucination correction),&#10;while requires the least trainable parameters and computational cost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.04661" label="2311.04661">
        <attvalues>
          <attvalue for="0" value="Massive Editing for Large Language Models via Meta Learning" />
          <attvalue for="1" value="  While large language models (LLMs) have enabled learning knowledge from the&#10;pre-training corpora, the acquired knowledge may be fundamentally incorrect or&#10;outdated over time, which necessitates rectifying the knowledge of the language&#10;model (LM) after the training. A promising approach involves employing a&#10;hyper-network to generate parameter shift, whereas existing hyper-networks&#10;suffer from inferior scalability in synchronous editing operation amount. To&#10;mitigate the problem, we propose the MAssive Language Model Editing Network&#10;(MALMEN), which formulates the parameter shift aggregation as the least square&#10;problem, subsequently updating the LM parameters using the normal equation. To&#10;accommodate editing multiple facts simultaneously with limited memory budgets,&#10;we separate the computation on the hyper-network and LM, enabling arbitrary&#10;batch size on both neural networks. Our method is evaluated by editing up to&#10;thousands of facts on LMs with different architectures, i.e., BERT-base, GPT-2,&#10;T5-XL (2.8B), and GPT-J (6B), across various knowledge-intensive NLP tasks,&#10;i.e., closed book fact-checking and question answering. Remarkably, MALMEN is&#10;capable of editing hundreds of times more facts than strong baselines with the&#10;identical hyper-network architecture and outperforms editor specifically&#10;designed for GPT. Our code is available at&#10;https://github.com/ChenmienTan/malmen.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07445" label="2109.07445">
        <attvalues>
          <attvalue for="0" value="Challenges in Detoxifying Language Models" />
          <attvalue for="1" value="  Large language models (LM) generate remarkably fluent text and can be&#10;efficiently adapted across NLP tasks. Measuring and guaranteeing the quality of&#10;generated text in terms of safety is imperative for deploying LMs in the real&#10;world; to this end, prior work often relies on automatic evaluation of LM&#10;toxicity. We critically discuss this approach, evaluate several toxicity&#10;mitigation strategies with respect to both automatic and human evaluation, and&#10;analyze consequences of toxicity mitigation in terms of model bias and LM&#10;quality. We demonstrate that while basic intervention strategies can&#10;effectively optimize previously established automatic metrics on the&#10;RealToxicityPrompts dataset, this comes at the cost of reduced LM coverage for&#10;both texts about, and dialects of, marginalized groups. Additionally, we find&#10;that human raters often disagree with high automatic toxicity scores after&#10;strong toxicity reduction interventions -- highlighting further the nuances&#10;involved in careful evaluation of LM toxicity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;While detecting hate speech and offensive language~\cite{warner-hirschberg-2012-detecting,Kwok2013LocateTH,Davidson2017AutomatedHS,zampieri-etal-2019-semeval}, mostly in the context of online community moderation, has long been a subject of research; the study of toxic text generated by language models is a more recent direction.&#10;\cite{wallace-etal-2019-universal} first demonstrated that synthetic text prompts can cause racist model continuations with GPT-2. &#10;\cite{gehman-etal-2020-realtoxicityprompts} extended the analysis of LM toxicity to non-synthetic prompts, further investigating the effectiveness of multiple potential mitigation approaches.&#10;We build on, and extend this work, critically discussing previously introduced metrics to assess LM toxicity, and compare classifier-based LM toxicity scoring with human evaluation.&#10;&#10;Among the most promising approaches for LM toxicity reduction is steering generation towards text less likely to be classified as toxic~\cite{Dathathri2020-ua,krause2021gedi}.&#10;This typically relies on an external toxicity classifier, although \cite{schick2021selfdiagnosis} show that even a LM's own toxicity self-diagnosis can be used to this end.&#10;&#10;Toxic language detection systems are known to be biased against specific social groups, and similar to \cite{zhou-etal-2021-challenges}, we distinguish two bias types.&#10;First, classification bias can manifest as topic-related biases, where text mentioning particular identities leads to false positives in toxicity classifiers---e.g. LGBTQ+ identity terms (``gay''). &#10;This phenomenon has been linked to an increased relative prevalence of identity terms among toxic samples \cite{waseem-hovy-2016-hateful,Dixon2018measuring_and_mitigating,park-etal-2018-reducing}.&#10;A second type of bias considers disparate performance across dialects, where classifiers on average assign higher toxicity scores e.g.~to African-American English (AAE)~\cite{davidson-etal-2019-racial,sap-etal-2019-risk}.&#10;A potential side-effect of applying classifier-based toxicity mitigation methods in an LM context, then, is that such biases might also be inherited by the resulting model.&#10;&#10;Our findings are consistent with contemporary work by \cite{xu2021detoxifying} demonstrating that LM toxicity mitigations can amplify social biases.&#10;Our work expands these results across a broader range of models, demographics, and datasets, and uses Wikipedia metadata \cite{bold_2021} rather than keyword-matching for measuring topic-related biases.&#10;We also show that models which perform well under our and their likelihood-based metrics can still exacerbate bias.&#10;Finally, by upsampling toxic samples, we can estimate overall LM toxicity, whereas a comparison-based approach can emphasize minor changes to already non-toxic LM completions.&#10;&#10;Other work on toxicity in generated text includes \cite{xu2020recipes}, who investigate safety specifically in a dialogue setting, and translating existing offensive text into non-offensive variants \cite{nogueira-dos-santos-etal-2018-fighting,laugier-etal-2021-civil}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2407.12824" label="2407.12824">
        <attvalues>
          <attvalue for="0" value="Whispering Experts: Neural Interventions for Toxicity Mitigation in&#10;  Language Models" />
          <attvalue for="1" value="  An important issue with Large Language Models (LLMs) is their undesired&#10;ability to generate toxic language. In this work, we show that the neurons&#10;responsible for toxicity can be determined by their power to discriminate toxic&#10;sentences, and that toxic language can be mitigated by reducing their&#10;activation levels proportionally to this power. We propose AUROC adaptation&#10;(AurA), an intervention that can be applied to any pre-trained LLM to mitigate&#10;toxicity. As the intervention is proportional to the ability of each neuron to&#10;discriminate toxic content, it is free of any model-dependent hyperparameters.&#10;We show that AurA can achieve up to $2.2 \times$ reduction in toxicity with&#10;only a $0.72$ perplexity increase. We also show that AurA is effective with&#10;models of different scale (from 1.5B to 40B parameters), and its effectiveness&#10;in mitigating toxic language, while preserving common-sense zero-shot&#10;abilities, holds across all scales. AurA can be combined with pre-prompting&#10;strategies, boosting its average mitigation potential from $1.28\times$ to&#10;$2.35\times$. Moreover, AurA can counteract adversarial pre-prompts that&#10;maliciously elicit toxic content, making it an effective method for deploying&#10;safer and less toxic models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Our work leverages the presence of expert units in LLMs.~\cite{suau2022self} have shown that expert units can be used to induce the generation of certain concepts. We expand on this work to probe whether intervening on these units can also be used to mitigate the generation of given concepts, specifically toxic language. In this sectionwe review the original algorithm, which is composed of two steps: identification of the experts, and intervention.&#10;&#10;Expert units are identified by considering every unit $m$ in the LLM as a potential classifier able to detect the presence of a specific concept in a given prompt. Experts are identified by leveraging a dataset of $N$ pairs $\{\vx_i, \vy_\rc^i\}_{i=1}^N$, where $\vx_i$ is the $i$-th sentence and $\vy_\rc^i=1$ if the sentence contains the concept $\rc$, $\vy_\rc^i=0$ otherwise.&#10;&#10;Each expert unit is analyzed in isolation, its maximum response (before the non-linearity) over each sentence in the dataset is used as a binary predictor for the presence of concept $\rc$. Formally, $\vz^i_m = \max(\{z_{m,t}^i\})$, where $z^{i}_{m,t}$ is the response of neuron $m$ to the $t$-th token of sentence $i$. All $\vz^i_m$ values are computed using the dataset of $N$ pairs and the expertise of the unit $\rm$ for concept $\rc$ is measured by the area under the Average-Precision curve, $\operatorname{AP}(\vz_m, \vy_\rc)$, where to simplify the notation $\vz_m$ and $\vy_\rc$ are the vectorial of representations of $\vz_m^i$ and $\vy_\rc^i$ over all $N$ sentences. The set $Q_k$ that contains the indices of the $k$ units with the highest $\operatorname{AP}(\vz_m, \vy_\rc)$ is the set of {expert units} for concept $\rc$. &#10;&#10;The intervention on $Q_k$ used to induce the presence of concept $\rc$ consist of replacing the output of each expert neuron with a fixed value $\gamma_m^{det} = \E_{\vy_\rc=1} [ \rz_m ]$, which is the mean activation of that neuron in presence of concept $\rc$. We can summarize the intervention as:&#10;\begin{equation}&#10; \iota^{det}(\rz_m, \gamma_m^{det}) = \gamma_m^{det}&#10; \quad \forall m \in Q_k.&#10;\end{equation}&#10;&#10;In the original proposal the authors mentioned that a similar intervention on $Q_k$ where $\gamma_m^{det} = 0$ was not successful when trying to remove a concept. However, since no evaluation was presented, we quantify such intervention and refer to it as Det-0. We also assess another natural intervention which is to set $\gamma_m^{det} = \E_{\vy_\rc=0} [ \rz_m ]$, which is the mean activation of that neuron in {absence} of concept $\rc$. We refer to this intervention as Det-E.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.08787" label="2402.08787">
        <attvalues>
          <attvalue for="0" value="Rethinking Machine Unlearning for Large Language Models" />
          <attvalue for="1" value="  We explore machine unlearning (MU) in the domain of large language models&#10;(LLMs), referred to as LLM unlearning. This initiative aims to eliminate&#10;undesirable data influence (e.g., sensitive or illegal information) and the&#10;associated model capabilities, while maintaining the integrity of essential&#10;knowledge generation and not affecting causally unrelated information. We&#10;envision LLM unlearning becoming a pivotal element in the life-cycle management&#10;of LLMs, potentially standing as an essential foundation for developing&#10;generative AI that is not only safe, secure, and trustworthy, but also&#10;resource-efficient without the need of full retraining. We navigate the&#10;unlearning landscape in LLMs from conceptual formulation, methodologies,&#10;metrics, and applications. In particular, we highlight the often-overlooked&#10;aspects of existing LLM unlearning research, e.g., unlearning scope, data-model&#10;interaction, and multifaceted efficacy assessment. We also draw connections&#10;between LLM unlearning and related areas such as model editing, influence&#10;functions, model explanation, adversarial training, and reinforcement learning.&#10;Furthermore, we outline an effective assessment framework for LLM unlearning&#10;and explore its applications in copyright and privacy safeguards and&#10;sociotechnical harm reduction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.16835" label="2402.16835">
        <attvalues>
          <attvalue for="0" value="Eight Methods to Evaluate Robust Unlearning in LLMs" />
          <attvalue for="1" value="  Machine unlearning can be useful for removing harmful capabilities and&#10;memorized text from large language models (LLMs), but there are not yet&#10;standardized methods for rigorously evaluating it. In this paper, we first&#10;survey techniques and limitations of existing unlearning evaluations. Second,&#10;we apply a comprehensive set of tests for the robustness and competitiveness of&#10;unlearning in the &quot;Who's Harry Potter&quot; (WHP) model from Eldan and Russinovich&#10;(2023). While WHP's unlearning generalizes well when evaluated with the&#10;&quot;Familiarity&quot; metric from Eldan and Russinovich, we find i)&#10;higher-than-baseline amounts of knowledge can reliably be extracted, ii) WHP&#10;performs on par with the original model on Harry Potter Q&amp;A tasks, iii) it&#10;represents latent knowledge comparably to the original model, and iv) there is&#10;collateral unlearning in related domains. Overall, our results highlight the&#10;importance of comprehensive unlearning evaluation that avoids ad-hoc metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14936" label="2305.14936">
        <attvalues>
          <attvalue for="0" value="Trade-Offs Between Fairness and Privacy in Language Modeling" />
          <attvalue for="1" value="  Protecting privacy in contemporary NLP models is gaining in importance. So&#10;does the need to mitigate social biases of such models. But can we have both at&#10;the same time? Existing research suggests that privacy preservation comes at&#10;the price of worsening biases in classification tasks. In this paper, we&#10;explore the extent to which this tradeoff really holds when we incorporate both&#10;privacy preservation and de-biasing techniques into training text generation&#10;models. How does improving the model along one dimension affect the other&#10;dimension as well as the utility of the model? We conduct an extensive set of&#10;experiments that include bias detection, privacy attacks, language modeling,&#10;and performance on downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11186" label="2305.11186">
        <attvalues>
          <attvalue for="0" value="Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM&#10;  Inference with Transferable Prompt" />
          <attvalue for="1" value="  While the numerous parameters in Large Language Models (LLMs) contribute to&#10;their superior performance, this massive scale makes them inefficient and&#10;memory-hungry. Thus, they are hard to deploy on commodity hardware, such as one&#10;single GPU. Given the memory and power constraints of such devices, model&#10;compression methods are widely employed to reduce both the model size and&#10;inference latency, which essentially trades off model quality in return for&#10;improved efficiency. Thus, optimizing this accuracy-efficiency trade-off is&#10;crucial for the LLM deployment on commodity hardware. In this paper, we&#10;introduce a new perspective to optimize this trade-off by prompting compressed&#10;models. Specifically, we first observe that for certain questions, the&#10;generation quality of a compressed LLM can be significantly improved by adding&#10;carefully designed hard prompts, though this isn't the case for all questions.&#10;Based on this observation, we propose a soft prompt learning method where we&#10;expose the compressed model to the prompt learning process, aiming to enhance&#10;the performance of prompts. Our experimental analysis suggests our soft prompt&#10;strategy greatly improves the performance of the 8x compressed LLaMA-7B model&#10;(with a joint 4-bit quantization and 50% weight pruning compression), allowing&#10;them to match their uncompressed counterparts on popular benchmarks. Also, we&#10;demonstrate that these learned prompts can be transferred across various&#10;datasets, tasks, and compression levels. Hence with this transferability, we&#10;can stitch the soft prompt to a newly compressed model to improve the test-time&#10;accuracy in an ``in-situ'' way.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs)~\cite{radford2018improving, radford2019language, brown2020language, zhang2022opt, llama} has revolutionized the field of Natural Language Processing (NLP).&#10;Notably, LLMs are known for their in-context learning ability, allowing them to generalize to unseen tasks without additional fine-tuning \cite{brown2020language}.&#10;Specifically, LLMs are controlled through user-provided natural language specifications of the task, or prompts, which illustrate how to complete a task. &#10;Equipped with the in-context learning ability, we only need to serve a single large model to efficiently handle different tasks.&#10;Despite of their remarkable adaptability, LLMs are very expensive to deploy~\cite{chen2023frugalgpt,wu2023fast}. &#10;The inference process of LLMs, such as LLaMA 2~\cite{touvron2023llama}, may require multiple powerful GPUs, which is prohibitively expensive for the general community. &#10;Consequently, it is crucial to facilitate LLM inference on more accessible hardware, such as a single GPU, which inherently has limited computational and memory resources.&#10;&#10;To address this problem, model compression methods are widely employed to reduce the model size and inference latency, such as quantization \cite{nagel2020up,dettmers2022llm,xiao2022smoothquant,frantar2022gptq} and pruning \cite{frantar2023sparsegpt}.&#10;These methods essentially trade off model quality in return for reduced latency and model size.&#10;Thus, there is an inevitable trade-off between accuracy and efficiency, resulting in a noticeable reduction in the model's accuracy and, consequently, the overall performance benefits of LLMs.&#10;To get a sense, as shown in Figure \ref{fig: motivation_example}, &#10;the full model (LLaMA-7B) is able to provide accurate answers to all three questions.&#10;However,&#10;the pruned model generates unrelated and off-topic answers to the same questions.&#10;&#10;Both model compression and prompts can influence the generation quality of LLMs.&#10;Thus intuitively, we can also utilize the prompt to help the compressed model generate more relevant answers.&#10;To the best of our knowledge, this perspective is not fully explored for LLMs. &#10;Thus one natural question is, for a compressed model, can we design a prompt that helps it correct its predictions accordingly?&#10;&#10;In this paper, we provide the first affirmative answer to the above question.&#10;As shown in Figure \ref{fig: motivation_example},&#10;we manually attach the prompt ``Please carefully examine the weight matrix within the model, as it may contain errors. It is crucial to verify its accuracy and make any necessary adjustments to ensure optimal performance'' to the original question.&#10;The prompted pruned model, i.e., ``LLaMA-7B (62.5\% sparsity) w./ Hard Prompt'' in Figure \ref{fig: motivation_example}, shows a significant improvement in its responses, although not all of them are accurate or complete.&#10;This manually-crafted prompt only conveys that the model weight might be inaccurate, without considering the dataset, compression methods, or tasks. This finding highlights the considerable potential for the transferability of this ``hard prompt'' across datasets, compression levels, and tasks.&#10;Despite the potential, this manually designed prompt is not consistently effective.&#10;Inspired by previous learnable prompt works \cite{li2021prefix, lester2021power},&#10;we hypothesize that by involving the compressed weight in the prompt learning process, a learnable prompt could potentially surpass the performance of the manually-designed prompt, while maintaining the transferability. &#10;Building upon this insight, we introduce a paradigm of prompt learning that seeks to train additive prompt tokens on a compressed LLM to enhance its accuracy. &#10;We underscore that the primary distinction between our prompt learning approach and previous prompt tuning frameworks \cite{li2021prefix, lester2021power,tang2023chain} is that earlier methods mainly utilized the prompt to adapt the model for specific downstream tasks. In contrast, the learned prompt in this paper resembles the hard prompt in Figure \ref{fig: motivation_example}, as it can be transferred between various datasets, compression methods, and tasks.&#10;&#10;Our experimental analysis suggests our method greatly improves the performance of the $8\times$ compressed LLaMA-7B model (with a joint 4-bit quantization and 50\% weight pruning compression), allowing them to match their uncompressed counterparts on several standard benchmarks.&#10;We also observe a certain degree of transferability of these learned prompts across different datasets, tasks, and compression levels. &#10;Hence with this transferability, we can stitch the soft prompt to a newly compressed model to improve the test-time accuracy in an ``in-situ'' way.&#10; " />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.08659" label="2310.08659">
        <attvalues>
          <attvalue for="0" value="LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models" />
          <attvalue for="1" value="  Quantization is an indispensable technique for serving Large Language Models&#10;(LLMs) and has recently found its way into LoRA fine-tuning. In this work we&#10;focus on the scenario where quantization and LoRA fine-tuning are applied&#10;together on a pre-trained model. In such cases it is common to observe a&#10;consistent gap in the performance on downstream tasks between full fine-tuning&#10;and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ&#10;(LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that&#10;simultaneously quantizes an LLM and finds a proper low-rank initialization for&#10;LoRA fine-tuning. Such an initialization alleviates the discrepancy between the&#10;quantized and full-precision model and significantly improves generalization in&#10;downstream tasks. We evaluate our method on natural language understanding,&#10;question answering, summarization, and natural language generation tasks.&#10;Experiments show that our method is highly effective and outperforms existing&#10;quantization methods, especially in the challenging 2-bit and 2/4-bit mixed&#10;precision regimes. The code is available on https://github.com/yxli2123/LoftQ.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.01967" label="2311.01967">
        <attvalues>
          <attvalue for="0" value="The language of prompting: What linguistic properties make a prompt&#10;  successful?" />
          <attvalue for="1" value="  The latest generation of LLMs can be prompted to achieve impressive zero-shot&#10;or few-shot performance in many NLP tasks. However, since performance is highly&#10;sensitive to the choice of prompts, considerable effort has been devoted to&#10;crowd-sourcing prompts or designing methods for prompt optimisation. Yet, we&#10;still lack a systematic understanding of how linguistic properties of prompts&#10;correlate with task performance. In this work, we investigate how LLMs of&#10;different sizes, pre-trained and instruction-tuned, perform on prompts that are&#10;semantically equivalent, but vary in linguistic structure. We investigate both&#10;grammatical properties such as mood, tense, aspect and modality, as well as&#10;lexico-semantic variation through the use of synonyms. Our findings contradict&#10;the common assumption that LLMs achieve optimal performance on lower perplexity&#10;prompts that reflect language use in pretraining or instruction-tuning data.&#10;Prompts transfer poorly between datasets or models, and performance cannot&#10;generally be explained by perplexity, word frequency, ambiguity or prompt&#10;length. Based on our results, we put forward a proposal for a more robust and&#10;comprehensive evaluation standard for prompting research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Instability in prompting&#10;&#10;Papers accompanying newly released models, rarely report prompts used for their evaluation and, typically, do not evaluate on multiple prompts~\cite{brown2020language, chowdhery_palm_2022, rae_scaling_2022}. \cite{sanh2022multitask} stand alone in reporting performance variation across prompts at the release of T0. &#10;&#10;To date, few works have investigated robustness to different prompt formulations. \cite{ishibashi-etal-2023-evaluating} show that machine-generated prompts are not robust to token deletion or reordering. &#10;\cite{webson-pavlick-2022-prompt} evaluate pre-trained and instruction-tuned models on NLI in the few-shot setting. They find that while instruction-tuning helps robustness against prompt variation, instruction-tuned models respond favourably even to misleading instructions. &#10;\cite{Shaikh2022onsecondthoughtnotthinkstepbystep} show that GPT-3 scores drastically worse on bias and toxicity challenge sets with the addition of `Let's think step by step.' to a given prompt for chain-of-thought reasoning \cite{kojima2022cot}. \cite{razeghi-etal-2022-impact} find that performance on arithmetic tasks correlates with frequency of integers in the training data. Perhaps closest to our work, \cite{gonen2022demystifying} find that lower perplexity of the prompt correlates with higher performance for OPT \cite{iyer2022opt} and BLOOM \cite{scao2022bloom} on a variety of different tasks. &#10;&#10;In priming or in-context learning, LMs profit from being shown the required input-output format, the distribution of inputs and the label space, while ground truth labels don't seem to be required \cite{Min2022roleofdemonstrationwhatmakesincontextlearning}. &#10;Performance is also sensitive to the ordering of demonstration examples \cite{lu2021fantasticallyorderedprompts, zhou2022least, Zhao2021calibratebeforeuse}. &#10;Chinchilla performs better on abstract reasoning tasks when test samples cater to prior knowledge acquired through pretraining \cite{dasgupta_language_2022}.&#10;&#10;Prompting evaluation practices.&#10;\cite{cao-etal-2022-prompt} point out that evaluating models on the same prompt does not make for a direct comparison, since models' exposure to different pretraining data results in different responses to individual prompts.~\cite{ishibashi-etal-2023-evaluating} find that machine-generated prompts do not achieve equal performance gains across datasets for the same task.&#10;\cite{holtzman2021surface} posit that subpar performance of LMs is due to different viable answers outside the answer choices competing for probability mass (``surface form competition''), reducing the score for the correct answer among answer choices. They mediate this using Domain Conditional PMI. \cite{Zhao2021calibratebeforeuse} observe that models overpredict label words that occur more frequently in a prompt, at its end, or are frequent in the pretraining data. &#10;They propose fitting an affine function to the LM scores, so that answer options are equally likely for `content-free' dummy examples. &#10;&#10;Contrary to other works \cite{gonen2022demystifying,sorensen-etal-2022-information,liao2022zerolabel} we do not aim at proposing prompt selection methods or calibrate predictions. While many (semi-)automatic approaches to generating prompts have been proposed~\cite[i.a.]{liu2023pre,shin2020autoprompt,jiang2020can,gao2021making}, we resort to crafting prompts manually so as to maintain fine-grained control over sentence structures in our prompts. \cite{logan-iv-etal-2022-cutting} provide evidence that manually written prompts~\cite{schick2021notjustsize} can yield better result than automatically sourced prompts. Further,~\cite{ishibashi-etal-2023-evaluating} point out that automatically generated prompts contain atypical language use, punctuation or spelling mistakes and generalise poorly across datasets.&#10;To isolate the effect of individual instructions we restrict ourselves to the zero-shot setting and do not include any demonstration examples in-context. Indeed, LMs have been shown to perform reasonably well in the few-shot setting given unrelated or misleading instructions \cite{webson-pavlick-2022-prompt}.&#10;Similarly, we abstain from prompt-tuning on demonstration examples, so as to not introduce additional sources of variance~\cite{cao-etal-2022-prompt}.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Language Model Performance, Prompt Optimization Techniques" />
        </attvalues>
      </node>
      <node id="2305.10403" label="2305.10403">
        <attvalues>
          <attvalue for="0" value="PaLM 2 Technical Report" />
          <attvalue for="1" value="  We introduce PaLM 2, a new state-of-the-art language model that has better&#10;multilingual and reasoning capabilities and is more compute-efficient than its&#10;predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture&#10;of objectives. Through extensive evaluations on English and multilingual&#10;language, and reasoning tasks, we demonstrate that PaLM 2 has significantly&#10;improved quality on downstream tasks across different model sizes, while&#10;simultaneously exhibiting faster and more efficient inference compared to PaLM.&#10;This improved efficiency enables broader deployment while also allowing the&#10;model to respond faster, for a more natural pace of interaction. PaLM 2&#10;demonstrates robust reasoning capabilities exemplified by large improvements&#10;over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable&#10;performance on a suite of responsible AI evaluations, and enables&#10;inference-time control over toxicity without additional overhead or impact on&#10;other capabilities. Overall, PaLM 2 achieves state-of-the-art performance&#10;across a diverse set of tasks and capabilities.&#10;  When discussing the PaLM 2 family, it is important to distinguish between&#10;pre-trained models (of various sizes), fine-tuned variants of these models, and&#10;the user-facing products that use these models. In particular, user-facing&#10;products typically include additional pre- and post-processing steps.&#10;Additionally, the underlying models may evolve over time. Therefore, one should&#10;not expect the performance of user-facing products to exactly match the results&#10;reported in this report.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.09597" label="2212.09597">
        <attvalues>
          <attvalue for="0" value="Reasoning with Language Model Prompting: A Survey" />
          <attvalue for="1" value="  Reasoning, as an essential ability for complex problem-solving, can provide&#10;back-end support for various real-world applications, such as medical&#10;diagnosis, negotiation, etc. This paper provides a comprehensive survey of&#10;cutting-edge research on reasoning with language model prompting. We introduce&#10;research works with comparisons and summaries and provide systematic resources&#10;to help beginners. We also discuss the potential reasons for emerging such&#10;reasoning abilities and highlight future research directions. Resources are&#10;available at https://github.com/zjunlp/Prompt4ReasoningPapers (updated&#10;periodically).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.00345" label="2004.00345">
        <attvalues>
          <attvalue for="0" value="Editable Neural Networks" />
          <attvalue for="1" value="  These days deep neural networks are ubiquitously used in a wide range of&#10;tasks, from image classification and machine translation to face identification&#10;and self-driving cars. In many applications, a single model error can lead to&#10;devastating financial, reputational and even life-threatening consequences.&#10;Therefore, it is crucially important to correct model mistakes quickly as they&#10;appear. In this work, we investigate the problem of neural network editing $-$&#10;how one can efficiently patch a mistake of the model on a particular sample,&#10;without influencing the model behavior on other samples. Namely, we propose&#10;Editable Training, a model-agnostic training technique that encourages fast&#10;editing of the trained model. We empirically demonstrate the effectiveness of&#10;this method on large-scale image classification and machine translation tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08164" label="2104.08164">
        <attvalues>
          <attvalue for="0" value="Editing Factual Knowledge in Language Models" />
          <attvalue for="1" value="  The factual knowledge acquired during pre-training and stored in the&#10;parameters of Language Models (LMs) can be useful in downstream tasks (e.g.,&#10;question answering or textual inference). However, some facts can be&#10;incorrectly induced or become obsolete over time. We present KnowledgeEditor, a&#10;method which can be used to edit this knowledge and, thus, fix 'bugs' or&#10;unexpected predictions without the need for expensive re-training or&#10;fine-tuning. Besides being computationally efficient, KnowledgeEditordoes not&#10;require any modifications in LM pre-training (e.g., the use of meta-learning).&#10;In our approach, we train a hyper-network with constrained optimization to&#10;modify a fact without affecting the rest of the knowledge; the trained&#10;hyper-network is then used to predict the weight update at test time. We show&#10;KnowledgeEditor's efficacy with two popular architectures and&#10;knowledge-intensive tasks: i) a BERT model fine-tuned for fact-checking, and&#10;ii) a sequence-to-sequence BART model for question answering. With our method,&#10;changing a prediction on the specific wording of a query tends to result in a&#10;consistent change in predictions also for its paraphrases. We show that this&#10;can be further encouraged by exploiting (e.g., automatically-generated)&#10;paraphrases during training. Interestingly, our hyper-network can be regarded&#10;as a 'probe' revealing which components need to be changed to manipulate&#10;factual knowledge; our analysis shows that the updates tend to be concentrated&#10;on a small subset of components. Source code available at&#10;https://github.com/nicola-decao/KnowledgeEditor&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.09785" label="2301.09785">
        <attvalues>
          <attvalue for="0" value="Transformer-Patcher: One Mistake worth One Neuron" />
          <attvalue for="1" value="  Large Transformer-based Pretrained Language Models (PLMs) dominate almost all&#10;Natural Language Processing (NLP) tasks. Nevertheless, they still make mistakes&#10;from time to time. For a model deployed in an industrial environment, fixing&#10;these mistakes quickly and robustly is vital to improve user experiences.&#10;Previous works formalize such problems as Model Editing (ME) and mostly focus&#10;on fixing one mistake. However, the one-mistake-fixing scenario is not an&#10;accurate abstraction of the real-world challenge. In the deployment of AI&#10;services, there are ever-emerging mistakes, and the same mistake may recur if&#10;not corrected in time. Thus a preferable solution is to rectify the mistakes as&#10;soon as they appear nonstop. Therefore, we extend the existing ME into&#10;Sequential Model Editing (SME) to help develop more practical editing methods.&#10;Our study shows that most current ME methods could yield unsatisfying results&#10;in this scenario. We then introduce Transformer-Patcher, a novel model editor&#10;that can shift the behavior of transformer-based models by simply adding and&#10;training a few neurons in the last Feed-Forward Network layer. Experimental&#10;results on both classification and generation tasks show that&#10;Transformer-Patcher can successively correct up to thousands of errors&#10;(Reliability) and generalize to their equivalent inputs (Generality) while&#10;retaining the model's accuracy on irrelevant inputs (Locality). Our method&#10;outperforms previous fine-tuning and HyperNetwork-based methods and achieves&#10;state-of-the-art performance for Sequential Model Editing (SME). The code is&#10;available at https://github.com/ZeroYuHuang/Transformer-Patcher.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.04115" label="1706.04115">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Relation Extraction via Reading Comprehension" />
          <attvalue for="1" value="  We show that relation extraction can be reduced to answering simple reading&#10;comprehension questions, by associating one or more natural-language questions&#10;with each relation slot. This reduction has several advantages: we can (1)&#10;learn relation-extraction models by extending recent neural&#10;reading-comprehension techniques, (2) build very large training sets for those&#10;models by combining relation-specific crowd-sourced questions with distant&#10;supervision, and even (3) do zero-shot learning by extracting new relation&#10;types that are only specified at test-time, for which we have no labeled&#10;training examples. Experiments on a Wikipedia slot-filling task demonstrate&#10;that the approach can generalize to new questions for known relation types with&#10;high accuracy, and that zero-shot generalization to unseen relation types is&#10;possible, at lower accuracy levels, setting the bar for future work on this&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.01068" label="2205.01068">
        <attvalues>
          <attvalue for="0" value="OPT: Open Pre-trained Transformer Language Models" />
          <attvalue for="1" value="  Large language models, which are often trained for hundreds of thousands of&#10;compute days, have shown remarkable capabilities for zero- and few-shot&#10;learning. Given their computational cost, these models are difficult to&#10;replicate without significant capital. For the few that are available through&#10;APIs, no access is granted to the full model weights, making them difficult to&#10;study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only&#10;pre-trained transformers ranging from 125M to 175B parameters, which we aim to&#10;fully and responsibly share with interested researchers. We show that OPT-175B&#10;is comparable to GPT-3, while requiring only 1/7th the carbon footprint to&#10;develop. We are also releasing our logbook detailing the infrastructure&#10;challenges we faced, along with code for experimenting with all of the released&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.06745" label="2204.06745">
        <attvalues>
          <attvalue for="0" value="GPT-NeoX-20B: An Open-Source Autoregressive Language Model" />
          <attvalue for="1" value="  We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language&#10;model trained on the Pile, whose weights will be made freely and openly&#10;available to the public through a permissive license. It is, to the best of our&#10;knowledge, the largest dense autoregressive model that has publicly available&#10;weights at the time of submission. In this work, we describe \model{}'s&#10;architecture and training and evaluate its performance on a range of&#10;language-understanding, mathematics, and knowledge-based tasks. We find that&#10;GPT-NeoX-20B is a particularly powerful few-shot reasoner and gains far more in&#10;performance when evaluated five-shot than similarly sized GPT-3 and FairSeq&#10;models. We open-source the training and evaluation code, as well as the model&#10;weights, at https://github.com/EleutherAI/gpt-neox.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.00267" label="2205.00267">
        <attvalues>
          <attvalue for="0" value="Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence&#10;  Encoders" />
          <attvalue for="1" value="  Pretrained multilingual language models (LMs) can be successfully transformed&#10;into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional&#10;fine-tuning or model distillation with parallel data. However, it remains&#10;unclear how to best leverage them to represent sub-sentence lexical items&#10;(i.e., words and phrases) in cross-lingual lexical tasks. In this work, we&#10;probe SEs for the amount of cross-lingual lexical knowledge stored in their&#10;parameters, and compare them against the original multilingual LMs. We also&#10;devise a simple yet efficient method for exposing the cross-lingual lexical&#10;knowledge by means of additional fine-tuning through inexpensive contrastive&#10;learning that requires only a small amount of word translation pairs. Using&#10;bilingual lexical induction (BLI), cross-lingual lexical semantic similarity,&#10;and cross-lingual entity linking as lexical probing tasks, we report&#10;substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI).&#10;The results indicate that the SEs such as LaBSE can be 'rewired' into effective&#10;cross-lingual lexical encoders via the contrastive learning procedure, and that&#10;they contain more cross-lingual lexical knowledge than what 'meets the eye'&#10;when they are used as off-the-shelf SEs. This way, we also provide an effective&#10;tool for harnessing 'covert' multilingual lexical knowledge hidden in&#10;multilingual sentence encoders.&#10;" />
          <attvalue for="2" value="&#10;Transfer learning with pretrained Language Models (LMs) such as BERT \cite{devlin2019bert} and RoBERTa \cite{liu:2019roberta} offers unmatched performance in many NLP tasks \cite{Wang:2019superglue,Raffel:2019:arxiv}. However, despite the wealth of semantic knowledge stored in the pretrained LMs \cite{Rogers:2020arxiv,Vulic:2020emnlp}, they do not produce coherent and effective sentence representations when used off-the-shelf \cite{Liu:2021emnlp}: to this effect, further specialization for sentence-level semantics -- not unlike the standard task fine-tuning -- is needed \cite[inter alia]{Reimers:2019emnlp,Li:2020emnlp,Yan:2021acl}. &#10;LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives \cite{infonce,Musgrave:2020eccv}, in supervised (i.e., leveraging labeled external data such as NLI or sentence similarity annotations) \cite{Reimers:2019emnlp,Vulic:2021emnlp,Liu:2021dialoguecse} or, more recently, fully unsupervised fine-tuning \cite{Liu:2021emnlp,Gao:2021emnlp} setups.&#10;&#10;Following the procedures from monolingual setups, another line of research has been transforming multilingual LMs into multilingual SEs \cite{Feng:2020labse,Reimers:2020emnlp}, which enable effective sentence matching and ranking in multiple languages as well as cross-lingually \cite{Litschko:2022jir}. The transformation is typically done by coupling 1) LM objectives on monolingual data available in multiple languages with 2) cross-lingual objectives such as Translation Language Modeling (TLM) \cite{Conneau:2019neurips} and/or cross-lingual contrastive ranking \cite{Yang:2020demos}. Such multilingual SEs consume a large number of parallel sentences for the latter objectives. Consequently, they outperform multilingual off-the-shelf LMs in cross-lingual sentence similarity and ranking applications \cite{Liu:2021acl,Litschko:2022jir}. However, as we show in this work, such multilingual SEs may still lag behind traditional static cross-lingual word embeddings (CLWEs) when encoding sub-sentence lexical items (e.g., words or phrases) \cite{Liu:2021emnlp} for cross-lingual lexical tasks (e.g., BLI).&#10;&#10;In this work, we probe multilingual SEs for cross-lingual lexical knowledge. We demonstrate that, due to their fine-tuning on multilingual and parallel data, they indeed store a wealth of such knowledge, much more than what `meets the eye' when they are used `off the shelf'. &#10;However, this lexical knowledge needs to be exposed from the original multilingual SEs, (again) through additional fine-tuning. In other words, we show that multilingual SEs can be `rewired' into effective cross-lingual lexical encoders, as illustrated in Figure~\ref{fig:front_pic}. This rewiring is again done via a quick and inexpensive contrastive learning procedure: with merely 1k-5k word translation pairs, we successfully convert multilingual SEs into state-of-the-art bilingual lexical encoders for any language pair.&#10;&#10;We probe the original LMs and SEs as well as demonstrate the usefulness of the proposed contrastive procedure for `exposing' cross-lingual lexical knowledge on three standard lexical cross-lingual tasks using standard evaluation data and protocols: BLI, cross-lingual lexical semantic similarity (XLSIM), and cross-lingual entity linking (XL-EL). We show that the `exposure' procedure is highly effective for both vanilla multilingual LMs (mBERT and XLM-R) and multilingual SEs (\labse and \mpnet): e.g., we observe $\approx$+10 Precision@1 points gains on standard BLI benchmarks \cite{Glavas:2019acl}. Multilingual SEs offer substantially better cross-lingual lexical performance than vanilla LMs, both before and after being subjected to contrastive cross-lingual lexical fine-tuning (see Figure~\ref{fig:front_pic}). This indicates that multilingual SEs have more cross-lingual lexical knowledge than their vanilla LM counterparts, likely owing to their additional exposure to parallel data. &#10;&#10;Finally, inspired by \newcite{Li:2022acl}, we validate that word vectors produced by cross-lingual lexical encoders (i.e., after contrastive cross-lingual lexical `exposure') can be effectively interpolated with static CLWEs \cite{Artetxe:2018acl} and offer even stronger performance in cross-lingual lexical tasks. Encouragingly, our cross-lingual lexical specialization of multilingual SEs (as well as the further interpolation with static CLWEs), yields particularly massive performance gains for pairs of low-resource languages, as demonstrated on the low-resource BLI benchmark \cite{Vulic:2019we}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Lingual Lexical Knowledge, Computer Science, Linguistics, Multilingual Language Models, Sentence Encoder Optimization, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1905.00537" label="1905.00537">
        <attvalues>
          <attvalue for="0" value="SuperGLUE: A Stickier Benchmark for General-Purpose Language&#10;  Understanding Systems" />
          <attvalue for="1" value="  In the last year, new models and methods for pretraining and transfer&#10;learning have driven striking performance improvements across a range of&#10;language understanding tasks. The GLUE benchmark, introduced a little over one&#10;year ago, offers a single-number metric that summarizes progress on a diverse&#10;set of such tasks, but performance on the benchmark has recently surpassed the&#10;level of non-expert humans, suggesting limited headroom for further research.&#10;In this paper we present SuperGLUE, a new benchmark styled after GLUE with a&#10;new set of more difficult language understanding tasks, a software toolkit, and&#10;a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.12327" label="2002.12327">
        <attvalues>
          <attvalue for="0" value="A Primer in BERTology: What we know about how BERT works" />
          <attvalue for="1" value="  Transformer-based models have pushed state of the art in many areas of NLP,&#10;but our understanding of what is behind their success is still limited. This&#10;paper is the first survey of over 150 studies of the popular BERT model. We&#10;review the current state of knowledge about how BERT works, what kind of&#10;information it learns and how it is represented, common modifications to its&#10;training objectives and architecture, the overparameterization issue and&#10;approaches to compression. We then outline directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05731" label="2010.05731">
        <attvalues>
          <attvalue for="0" value="Probing Pretrained Language Models for Lexical Semantics" />
          <attvalue for="1" value="  The success of large pretrained language models (LMs) such as BERT and&#10;RoBERTa has sparked interest in probing their representations, in order to&#10;unveil what types of knowledge they implicitly capture. While prior research&#10;focused on morphosyntactic, semantic, and world knowledge, it remains unclear&#10;to which extent LMs also derive lexical type-level knowledge from words in&#10;context. In this work, we present a systematic empirical analysis across six&#10;typologically diverse languages and five different lexical tasks, addressing&#10;the following questions: 1) How do different lexical knowledge extraction&#10;strategies (monolingual versus multilingual source LM, out-of-context versus&#10;in-context encoding, inclusion of special tokens, and layer-wise averaging)&#10;impact performance? How consistent are the observed effects across tasks and&#10;languages? 2) Is lexical knowledge stored in few parameters, or is it scattered&#10;throughout the network? 3) How do these representations fare against&#10;traditional static word vectors in lexical tasks? 4) Does the lexical&#10;information emerging from independently trained monolingual LMs display latent&#10;similarities? Our main results indicate patterns and best practices that hold&#10;universally, but also point to prominent variations across languages and tasks.&#10;Moreover, we validate the claim that lower Transformer layers carry more&#10;type-level lexical knowledge, but also show that this knowledge is distributed&#10;across multiple layers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08027" label="2104.08027">
        <attvalues>
          <attvalue for="0" value="Fast, Effective, and Self-Supervised: Transforming Masked Language&#10;  Models into Universal Lexical and Sentence Encoders" />
          <attvalue for="1" value="  Pretrained Masked Language Models (MLMs) have revolutionised NLP in recent&#10;years. However, previous work has indicated that off-the-shelf MLMs are not&#10;effective as universal lexical or sentence encoders without further&#10;task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks&#10;using annotated task data. In this work, we demonstrate that it is possible to&#10;turn MLMs into effective universal lexical and sentence encoders even without&#10;any additional data and without any supervision. We propose an extremely&#10;simple, fast and effective contrastive learning technique, termed Mirror-BERT,&#10;which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20-30&#10;seconds without any additional external knowledge. Mirror-BERT relies on fully&#10;identical or slightly modified string pairs as positive (i.e., synonymous)&#10;fine-tuning examples, and aims to maximise their similarity during identity&#10;fine-tuning. We report huge gains over off-the-shelf MLMs with Mirror-BERT in&#10;both lexical-level and sentence-level tasks, across different domains and&#10;different languages. Notably, in the standard sentence semantic similarity&#10;(STS) tasks, our self-supervised Mirror-BERT model even matches the performance&#10;of the task-tuned Sentence-BERT models from prior work. Finally, we delve&#10;deeper into the inner workings of MLMs, and suggest some evidence on why this&#10;simple approach can yield effective universal lexical and sentence encoders.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.10084" label="1908.10084">
        <attvalues>
          <attvalue for="0" value="Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks" />
          <attvalue for="1" value="  BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new&#10;state-of-the-art performance on sentence-pair regression tasks like semantic&#10;textual similarity (STS). However, it requires that both sentences are fed into&#10;the network, which causes a massive computational overhead: Finding the most&#10;similar pair in a collection of 10,000 sentences requires about 50 million&#10;inference computations (~65 hours) with BERT. The construction of BERT makes it&#10;unsuitable for semantic similarity search as well as for unsupervised tasks&#10;like clustering.&#10;  In this publication, we present Sentence-BERT (SBERT), a modification of the&#10;pretrained BERT network that use siamese and triplet network structures to&#10;derive semantically meaningful sentence embeddings that can be compared using&#10;cosine-similarity. This reduces the effort for finding the most similar pair&#10;from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while&#10;maintaining the accuracy from BERT.&#10;  We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning&#10;tasks, where it outperforms other state-of-the-art sentence embeddings methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.05864" label="2011.05864">
        <attvalues>
          <attvalue for="0" value="On the Sentence Embeddings from Pre-trained Language Models" />
          <attvalue for="1" value="  Pre-trained contextual representations like BERT have achieved great success&#10;in natural language processing. However, the sentence embeddings from the&#10;pre-trained language models without fine-tuning have been found to poorly&#10;capture semantic meaning of sentences. In this paper, we argue that the&#10;semantic information in the BERT embeddings is not fully exploited. We first&#10;reveal the theoretical connection between the masked language model&#10;pre-training objective and the semantic similarity task theoretically, and then&#10;analyze the BERT sentence embeddings empirically. We find that BERT always&#10;induces a non-smooth anisotropic semantic space of sentences, which harms its&#10;performance of semantic similarity. To address this issue, we propose to&#10;transform the anisotropic sentence embedding distribution to a smooth and&#10;isotropic Gaussian distribution through normalizing flows that are learned with&#10;an unsupervised objective. Experimental results show that our proposed&#10;BERT-flow method obtains significant performance gains over the&#10;state-of-the-art sentence embeddings on a variety of semantic textual&#10;similarity tasks. The code is available at&#10;https://github.com/bohanli/BERT-flow.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.11741" label="2105.11741">
        <attvalues>
          <attvalue for="0" value="ConSERT: A Contrastive Framework for Self-Supervised Sentence&#10;  Representation Transfer" />
          <attvalue for="1" value="  Learning high-quality sentence representations benefits a wide range of&#10;natural language processing tasks. Though BERT-based pre-trained language&#10;models achieve high performance on many downstream tasks, the native derived&#10;sentence representations are proved to be collapsed and thus produce a poor&#10;performance on the semantic textual similarity (STS) tasks. In this paper, we&#10;present ConSERT, a Contrastive Framework for Self-Supervised Sentence&#10;Representation Transfer, that adopts contrastive learning to fine-tune BERT in&#10;an unsupervised and effective way. By making use of unlabeled texts, ConSERT&#10;solves the collapse issue of BERT-derived sentence representations and make&#10;them more applicable for downstream tasks. Experiments on STS datasets&#10;demonstrate that ConSERT achieves an 8\% relative improvement over the previous&#10;state-of-the-art, even comparable to the supervised SBERT-NLI. And when further&#10;incorporating NLI supervision, we achieve new state-of-the-art performance on&#10;STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples&#10;available, showing its robustness in data scarcity scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1807.03748" label="1807.03748">
        <attvalues>
          <attvalue for="0" value="Representation Learning with Contrastive Predictive Coding" />
          <attvalue for="1" value="  While supervised learning has enabled great progress in many applications,&#10;unsupervised learning has not seen such widespread adoption, and remains an&#10;important and challenging endeavor for artificial intelligence. In this work,&#10;we propose a universal unsupervised learning approach to extract useful&#10;representations from high-dimensional data, which we call Contrastive&#10;Predictive Coding. The key insight of our model is to learn such&#10;representations by predicting the future in latent space by using powerful&#10;autoregressive models. We use a probabilistic contrastive loss which induces&#10;the latent space to capture information that is maximally useful to predict&#10;future samples. It also makes the model tractable by using negative sampling.&#10;While most prior work has focused on evaluating representations for a&#10;particular modality, we demonstrate that our approach is able to learn useful&#10;representations achieving strong performance on four distinct domains: speech,&#10;images, text and reinforcement learning in 3D environments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.08505" label="2003.08505">
        <attvalues>
          <attvalue for="0" value="A Metric Learning Reality Check" />
          <attvalue for="1" value="  Deep metric learning papers from the past four years have consistently&#10;claimed great advances in accuracy, often more than doubling the performance of&#10;decade-old methods. In this paper, we take a closer look at the field to see if&#10;this is actually true. We find flaws in the experimental methodology of&#10;numerous metric learning papers, and show that the actual improvements over&#10;time have been marginal at best.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.10126" label="2109.10126">
        <attvalues>
          <attvalue for="0" value="ConvFiT: Conversational Fine-Tuning of Pretrained Language Models" />
          <attvalue for="1" value="  Transformer-based language models (LMs) pretrained on large text collections&#10;are proven to store a wealth of semantic knowledge. However, 1) they are not&#10;effective as sentence encoders when used off-the-shelf, and 2) thus typically&#10;lag behind conversationally pretrained (e.g., via response selection) encoders&#10;on conversational tasks such as intent detection (ID). In this work, we propose&#10;ConvFiT, a simple and efficient two-stage procedure which turns any pretrained&#10;LM into a universal conversational encoder (after Stage 1 ConvFiT-ing) and&#10;task-specialised sentence encoder (after Stage 2). We demonstrate that 1)&#10;full-blown conversational pretraining is not required, and that LMs can be&#10;quickly transformed into effective conversational encoders with much smaller&#10;amounts of unannotated data; 2) pretrained LMs can be fine-tuned into&#10;task-specialised sentence encoders, optimised for the fine-grained semantics of&#10;a particular task. Consequently, such specialised sentence encoders allow for&#10;treating ID as a simple semantic similarity task based on interpretable nearest&#10;neighbours retrieval. We validate the robustness and versatility of the ConvFiT&#10;framework with such similarity-based inference on the standard ID evaluation&#10;sets: ConvFiT-ed LMs achieve state-of-the-art ID performance across the board,&#10;with particular gains in the most challenging, few-shot setups.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.12599" label="2109.12599">
        <attvalues>
          <attvalue for="0" value="DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddings" />
          <attvalue for="1" value="  Learning sentence embeddings from dialogues has drawn increasing attention&#10;due to its low annotation cost and high domain adaptability. Conventional&#10;approaches employ the siamese-network for this task, which obtains the sentence&#10;embeddings through modeling the context-response semantic relevance by applying&#10;a feed-forward network on top of the sentence encoders. However, as the&#10;semantic textual similarity is commonly measured through the element-wise&#10;distance metrics (e.g. cosine and L2 distance), such architecture yields a&#10;large gap between training and evaluating. In this paper, we propose&#10;DialogueCSE, a dialogue-based contrastive learning approach to tackle this&#10;issue. DialogueCSE first introduces a novel matching-guided embedding (MGE)&#10;mechanism, which generates a context-aware embedding for each candidate&#10;response embedding (i.e. the context-free embedding) according to the guidance&#10;of the multi-turn context-response matching matrices. Then it pairs each&#10;context-aware embedding with its corresponding context-free embedding and&#10;finally minimizes the contrastive loss across all pairs. We evaluate our model&#10;on three multi-turn dialogue datasets: the Microsoft Dialogue Corpus, the Jing&#10;Dong Dialogue Corpus, and the E-commerce Dialogue Corpus. Evaluation results&#10;show that our approach significantly outperforms the baselines across all three&#10;datasets in terms of MAP and Spearman's correlation measures, demonstrating its&#10;effectiveness. Further quantitative experiments show that our approach achieves&#10;better performance when leveraging more dialogue context and remains robust&#10;when less training data is provided.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08821" label="2104.08821">
        <attvalues>
          <attvalue for="0" value="SimCSE: Simple Contrastive Learning of Sentence Embeddings" />
          <attvalue for="1" value="  This paper presents SimCSE, a simple contrastive learning framework that&#10;greatly advances state-of-the-art sentence embeddings. We first describe an&#10;unsupervised approach, which takes an input sentence and predicts itself in a&#10;contrastive objective, with only standard dropout used as noise. This simple&#10;method works surprisingly well, performing on par with previous supervised&#10;counterparts. We find that dropout acts as minimal data augmentation, and&#10;removing it leads to a representation collapse. Then, we propose a supervised&#10;approach, which incorporates annotated pairs from natural language inference&#10;datasets into our contrastive learning framework by using &quot;entailment&quot; pairs as&#10;positives and &quot;contradiction&quot; pairs as hard negatives. We evaluate SimCSE on&#10;standard semantic textual similarity (STS) tasks, and our unsupervised and&#10;supervised models using BERT base achieve an average of 76.3% and 81.6%&#10;Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to&#10;the previous best results. We also show -- both theoretically and empirically&#10;-- that the contrastive learning objective regularizes pre-trained embeddings'&#10;anisotropic space to be more uniform, and it better aligns positive pairs when&#10;supervised signals are available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.01852" label="2007.01852">
        <attvalues>
          <attvalue for="0" value="Language-agnostic BERT Sentence Embedding" />
          <attvalue for="1" value="  While BERT is an effective method for learning monolingual sentence&#10;embeddings for semantic similarity and embedding based transfer learning&#10;(Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have&#10;yet to be explored. We systematically investigate methods for learning&#10;multilingual sentence embeddings by combining the best methods for learning&#10;monolingual and cross-lingual representations including: masked language&#10;modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019),&#10;dual encoder translation ranking (Guo et al., 2018), and additive margin&#10;softmax (Yang et al., 2019a). We show that introducing a pre-trained&#10;multilingual language model dramatically reduces the amount of parallel&#10;training data required to achieve good performance by 80%. Composing the best&#10;of these methods produces a model that achieves 83.7% bi-text retrieval&#10;accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by&#10;Artetxe and Schwenk (2019b), while still performing competitively on&#10;monolingual transfer learning benchmarks (Conneau and Kiela, 2018). Parallel&#10;data mined from CommonCrawl using our best model is shown to train competitive&#10;NMT models for en-zh and en-de. We publicly release our best multilingual&#10;sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.09813" label="2004.09813">
        <attvalues>
          <attvalue for="0" value="Making Monolingual Sentence Embeddings Multilingual using Knowledge&#10;  Distillation" />
          <attvalue for="1" value="  We present an easy and efficient method to extend existing sentence embedding&#10;models to new languages. This allows to create multilingual versions from&#10;previously monolingual models. The training is based on the idea that a&#10;translated sentence should be mapped to the same location in the vector space&#10;as the original sentence. We use the original (monolingual) model to generate&#10;sentence embeddings for the source language and then train a new system on&#10;translated sentences to mimic the original model. Compared to other methods for&#10;training multilingual sentence embeddings, this approach has several&#10;advantages: It is easy to extend existing models with relatively few samples to&#10;new languages, it is easier to ensure desired properties for the vector space,&#10;and the hardware requirements for training is lower. We demonstrate the&#10;effectiveness of our approach for 50+ languages from various language families.&#10;Code to extend sentence embeddings models to more than 400 languages is&#10;publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.11031" label="2112.11031">
        <attvalues>
          <attvalue for="0" value="On Cross-Lingual Retrieval with Multilingual Text Encoders" />
          <attvalue for="1" value="  In this work we present a systematic empirical study focused on the&#10;suitability of the state-of-the-art multilingual encoders for cross-lingual&#10;document and sentence retrieval tasks across a number of diverse language&#10;pairs. We first treat these models as multilingual text encoders and benchmark&#10;their performance in unsupervised ad-hoc sentence- and document-level CLIR. In&#10;contrast to supervised language understanding, our results indicate that for&#10;unsupervised document-level CLIR -- a setup with no relevance judgments for&#10;IR-specific fine-tuning -- pretrained multilingual encoders on average fail to&#10;significantly outperform earlier models based on CLWEs. For sentence-level&#10;retrieval, we do obtain state-of-the-art performance: the peak scores, however,&#10;are met by multilingual encoders that have been further specialized, in a&#10;supervised fashion, for sentence understanding tasks, rather than using their&#10;vanilla 'off-the-shelf' variants. Following these results, we introduce&#10;localized relevance matching for document-level CLIR, where we independently&#10;score a query against document sections. In the second part, we evaluate&#10;multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to&#10;rank) on English relevance data in a series of zero-shot language and domain&#10;transfer CLIR experiments. Our results show that supervised re-ranking rarely&#10;improves the performance of multilingual transformers as unsupervised base&#10;rankers. Finally, only with in-domain contrastive fine-tuning (i.e., same&#10;domain, only language transfer), we manage to improve the ranking quality. We&#10;uncover substantial empirical differences between cross-lingual retrieval&#10;results and results of (zero-shot) cross-lingual transfer for monolingual&#10;retrieval in target languages, which point to &quot;monolingual overfitting&quot; of&#10;retrieval models trained on monolingual data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.07291" label="1901.07291">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Language Model Pretraining" />
          <attvalue for="1" value="  Recent studies have demonstrated the efficiency of generative pretraining for&#10;English natural language understanding. In this work, we extend this approach&#10;to multiple languages and show the effectiveness of cross-lingual pretraining.&#10;We propose two methods to learn cross-lingual language models (XLMs): one&#10;unsupervised that only relies on monolingual data, and one supervised that&#10;leverages parallel data with a new cross-lingual language model objective. We&#10;obtain state-of-the-art results on cross-lingual classification, unsupervised&#10;and supervised machine translation. On XNLI, our approach pushes the state of&#10;the art by an absolute gain of 4.9% accuracy. On unsupervised machine&#10;translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the&#10;previous state of the art by more than 9 BLEU. On supervised machine&#10;translation, we obtain a new state of the art of 38.5 BLEU on WMT'16&#10;Romanian-English, outperforming the previous best approach by more than 4 BLEU.&#10;Our code and pretrained models will be made publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.04307" label="1907.04307">
        <attvalues>
          <attvalue for="0" value="Multilingual Universal Sentence Encoder for Semantic Retrieval" />
          <attvalue for="1" value="  We introduce two pre-trained retrieval focused multilingual sentence encoding&#10;models, respectively based on the Transformer and CNN model architectures. The&#10;models embed text from 16 languages into a single semantic space using a&#10;multi-task trained dual-encoder that learns tied representations using&#10;translation based bridge tasks (Chidambaram al., 2018). The models provide&#10;performance that is competitive with the state-of-the-art on: semantic&#10;retrieval (SR), translation pair bitext retrieval (BR) and retrieval question&#10;answering (ReQA). On English transfer learning tasks, our sentence-level&#10;embeddings approach, and in some cases exceed, the performance of monolingual,&#10;English only, sentence embedding models. Our models are made available for&#10;download on TensorFlow Hub.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.14398" label="2105.14398">
        <attvalues>
          <attvalue for="0" value="Learning Domain-Specialised Representations for Cross-Lingual Biomedical&#10;  Entity Linking" />
          <attvalue for="1" value="  Injecting external domain-specific knowledge (e.g., UMLS) into pretrained&#10;language models (LMs) advances their capability to handle specialised in-domain&#10;tasks such as biomedical entity linking (BEL). However, such abundant expert&#10;knowledge is available only for a handful of languages (e.g., English). In this&#10;work, by proposing a novel cross-lingual biomedical entity linking task&#10;(XL-BEL) and establishing a new XL-BEL benchmark spanning 10 typologically&#10;diverse languages, we first investigate the ability of standard&#10;knowledge-agnostic as well as knowledge-enhanced monolingual and multilingual&#10;LMs beyond the standard monolingual English BEL task. The scores indicate large&#10;gaps to English performance. We then address the challenge of transferring&#10;domain-specific knowledge in resource-rich languages to resource-poor ones. To&#10;this end, we propose and evaluate a series of cross-lingual transfer methods&#10;for the XL-BEL task, and demonstrate that general-domain bitext helps propagate&#10;the available English knowledge to languages with little to no in-domain data.&#10;Remarkably, we show that our proposed domain-specific transfer methods yield&#10;consistent gains across all target languages, sometimes up to 20 Precision@1&#10;points, without any in-domain knowledge in the target language, and without any&#10;in-domain parallel data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.00508" label="1902.00508">
        <attvalues>
          <attvalue for="0" value="How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong&#10;  Baselines, Comparative Analyses, and Some Misconceptions" />
          <attvalue for="1" value="  Cross-lingual word embeddings (CLEs) enable multilingual modeling of meaning&#10;and facilitate cross-lingual transfer of NLP models. Despite their ubiquitous&#10;usage in downstream tasks, recent increasingly popular projection-based CLE&#10;models are almost exclusively evaluated on a single task only: bilingual&#10;lexicon induction (BLI). Even BLI evaluations vary greatly, hindering our&#10;ability to correctly interpret performance and properties of different CLE&#10;models. In this work, we make the first step towards a comprehensive evaluation&#10;of cross-lingual word embeddings. We thoroughly evaluate both supervised and&#10;unsupervised CLE models on a large number of language pairs in the BLI task and&#10;three downstream tasks, providing new insights concerning the ability of&#10;cutting-edge CLE models to support cross-lingual NLP. We empirically&#10;demonstrate that the performance of CLE models largely depends on the task at&#10;hand and that optimizing CLE models for BLI can result in deteriorated&#10;downstream performance. We indicate the most robust supervised and unsupervised&#10;CLE models and emphasize the need to reassess existing baselines, which still&#10;display competitive performance across the board. We hope that our work will&#10;catalyze further work on CLE evaluation and model analysis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.06297" label="1805.06297">
        <attvalues>
          <attvalue for="0" value="A robust self-learning method for fully unsupervised cross-lingual&#10;  mappings of word embeddings" />
          <attvalue for="1" value="  Recent work has managed to learn cross-lingual word embeddings without&#10;parallel data by mapping monolingual embeddings to a shared space through&#10;adversarial training. However, their evaluation has focused on favorable&#10;conditions, using comparable corpora or closely-related languages, and we show&#10;that they often fail in more realistic scenarios. This work proposes an&#10;alternative approach based on a fully unsupervised initialization that&#10;explicitly exploits the structural similarity of the embeddings, and a robust&#10;self-learning algorithm that iteratively improves this solution. Our method&#10;succeeds in all tested scenarios and obtains the best published results in&#10;standard datasets, even surpassing previous supervised systems. Our&#10;implementation is released as an open source project at&#10;https://github.com/artetxem/vecmap&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.01638" label="1909.01638">
        <attvalues>
          <attvalue for="0" value="Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?" />
          <attvalue for="1" value="  Recent efforts in cross-lingual word embedding (CLWE) learning have&#10;predominantly focused on fully unsupervised approaches that project monolingual&#10;embeddings into a shared cross-lingual space without any cross-lingual signal.&#10;The lack of any supervision makes such approaches conceptually attractive. Yet,&#10;their only core difference from (weakly) supervised projection-based CLWE&#10;methods is in the way they obtain a seed dictionary used to initialize an&#10;iterative self-learning procedure. The fully unsupervised methods have arguably&#10;become more robust, and their primary use case is CLWE induction for pairs of&#10;resource-poor and distant languages. In this paper, we question the ability of&#10;even the most robust unsupervised CLWE approaches to induce meaningful CLWEs in&#10;these more challenging settings. A series of bilingual lexicon induction (BLI)&#10;experiments with 15 diverse languages (210 language pairs) show that fully&#10;unsupervised CLWE methods still fail for a large number of language pairs&#10;(e.g., they yield zero BLI performance for 87/210 pairs). Even when they&#10;succeed, they never surpass the performance of weakly supervised methods&#10;(seeded with 500-1,000 translation pairs) using the same self-learning&#10;procedure in any BLI setup, and the gaps are often substantial. These findings&#10;call for revisiting the main motivations behind fully unsupervised CLWE&#10;methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.02406" label="2210.02406">
        <attvalues>
          <attvalue for="0" value="Decomposed Prompting: A Modular Approach for Solving Complex Tasks" />
          <attvalue for="1" value="  Few-shot prompting is a surprisingly powerful way to use Large Language&#10;Models (LLMs) to solve various tasks. However, this approach struggles as the&#10;task complexity increases or when the individual reasoning steps of the task&#10;themselves are hard to learn, especially when embedded in more complex tasks.&#10;To address this, we propose Decomposed Prompting, a new approach to solve&#10;complex tasks by decomposing them (via prompting) into simpler sub-tasks that&#10;can be delegated to a library of prompting-based LLMs dedicated to these&#10;sub-tasks. This modular structure allows each prompt to be optimized for its&#10;specific sub-task, further decomposed if necessary, and even easily replaced&#10;with more effective prompts, trained models, or symbolic functions if desired.&#10;We show that the flexibility and modularity of Decomposed Prompting allows it&#10;to outperform prior work on few-shot prompting using GPT3. On symbolic&#10;reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into&#10;even simpler solvable sub-tasks. When the complexity comes from the input&#10;length, we can recursively decompose the task into the same task but with&#10;smaller inputs. We also evaluate our approach on textual multi-step reasoning&#10;tasks: on long-context multi-hop QA task, we can more effectively teach the&#10;sub-tasks via our separate sub-tasks prompts; and on open-domain multi-hop QA,&#10;we can incorporate a symbolic information retrieval within our decomposition&#10;framework, leading to improved performance on both tasks. Datasets, Code and&#10;Prompts available at https://github.com/allenai/DecomP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Few-shot Prompts for Multi-Step Reasoning&#10;&#10;Large-scale Language models (LLMs) have been shown to learn various NLP tasks given just few examples as prompts~\cite{gpt3}. Recently, they have also been successfully applied to various multi-step reasoning tasks by providing the intermediate reasoning steps, i.e. Chain-of-Thought~\cite{Wei2022ChainOT,chowdhery2022palm}, needed to arrive at the answer. An alternate approach has been to compose multiple LLMs or LLMs with symbolic functions to perform multi-step reasoning~\cite[inter alia]{jung2022maieutic,creswell2022selection,selfask,talm,pal,Schick2023ToolformerLM}. We view these prior works as specialized systems with a pre-defined decomposition structure. &#10;&#10;The closest works to our approach are the ideas of least-to-most prompting~\cite{Zhou2022LeasttoMostPE} and successive prompting~\cite{succprompting} where one prompt/model is used to generate the sub-questions needed to answer a complex question and a second prompt/model sequentially answers these sub-questions. In contrast, our approach allows for diverse decomposition structures including recursion and other non-linear decomposition structures. E.g., by definition, least-to-most asks questions from easiest to the hardest and requires an LLM to eventually answer the complete question (“most” in least-to-most) whereas we have no such restriction. Additionally, we iteratively generate new questions based on previous answers (similar to successive prompting) and can explicitly assign different prompts or symbolic systems to answer each sub-question.&#10;&#10;Modular Approaches for Multi-Step Reasoning&#10;&#10;Our work follows a long literature in NLP on neural modular modeling architectures \cite{andreas2016neural,talmor2018web,min-etal-2019-multi,jiang2019self,gupta2020neural,perez2020unsupervised,khot-etal-2021-text,levine2022standing} for question-answering and other tasks. We take particular inspiration from the Text Modular Networks approach of \cite{khot-etal-2021-text}, whereby problem decomposition consists of a learned next question generator trained to generate questions in the language of a collection of textual and symbolic agents. Best-first search strategy was used to explore the space of possible decompositions during inference. In contrast to this work, which largely centered around supervised training of the next-question generator given existing agents, we leverage the power and recent successes of few-shot LLMs to build both the decomposer and the sub-task agents that best fit the ideal decomposition. This has the advantage of obviating the need for specialized supervised training data that may not always be available for all sub-tasks -- a key bottleneck of this prior work. &#10;&#10;" />
          <attvalue for="4" value="Multi-Step Reasoning Tasks, Computer Science, Large Language Models, Linguistics, Cognitive Science, Task Decomposition Methods, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2212.04037" label="2212.04037">
        <attvalues>
          <attvalue for="0" value="Demystifying Prompts in Language Models via Perplexity Estimation" />
          <attvalue for="1" value="  Language models can be prompted to perform a wide variety of zero- and&#10;few-shot learning problems. However, performance varies significantly with the&#10;choice of prompt, and we do not yet understand why this happens or how to pick&#10;the best prompts. In this work, we analyze the factors that contribute to this&#10;variance and establish a new empirical hypothesis: the performance of a prompt&#10;is coupled with the extent to which the model is familiar with the language it&#10;contains. Over a wide range of tasks, we show that the lower the perplexity of&#10;the prompt is, the better the prompt is able to perform the task. As a result,&#10;we devise a method for creating prompts: (1) automatically extend a small seed&#10;set of manually written prompts by paraphrasing using GPT3 and backtranslation&#10;and (2) choose the lowest perplexity prompts to get significant gains in&#10;performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Relation between performance and training data Previous work looking directly into the relation between the training data and the performance is limited. \cite{termfreq} study numeric deduction tasks, and examine the correlations between the model performance on specific test instances and the frequency of terms from&#10;those instances in the pretraining data. They find that the models are more accurate on instances whose terms are more prevalent in the training data. Additionally, \cite{orca} propose a method to effectively identify a very small subset of pretraining data that directly supports the model in performing a specific task. \cite{causal_training} use causal inference to measure the effect of pretraining data statistics on factual knowledge performance, and \cite{long_tail} show correlational and causal relationships between accuracy and relevant document count (from training data) for QA datasets.&#10;&#10;Prompt tuning and analysis There is a very rich line of work trying to find prompts automatically. \cite{autoprompt} present an automated method to create discrete prompts for a diverse set of tasks, based on a gradient-guided search, and they demonstrate their method on masked LMs. Other work also focuses on discrete prompts, aiming to improve zero-shot performance \cite{betterfew,dpprompts,rlprompt,kubrik}, or trains continuous prompts \cite{prefixtuning,scale_tuning,mixprompts}.&#10;&#10;On top of works that suggest a variety of methods for creating better prompts, some work also analyzes those prompts to try and get some insights about them: \cite{instructprompts} find that model performance is highly sensitive to small changes in&#10;wordings and \cite{waywardness} point to a surprising disconnect between continuous and discrete prompts.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Machine Learning, Zero-Shot Learning Strategies, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Language Model Performance, Prompt Optimization Techniques" />
        </attvalues>
      </node>
      <node id="2003.02245" label="2003.02245">
        <attvalues>
          <attvalue for="0" value="Data Augmentation using Pre-trained Transformer Models" />
          <attvalue for="1" value="  Language model based pre-trained models such as BERT have provided&#10;significant gains across different NLP tasks. In this paper, we study different&#10;types of transformer based pre-trained models such as auto-regressive models&#10;(GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional&#10;data augmentation. We show that prepending the class labels to text sequences&#10;provides a simple yet effective way to condition the pre-trained models for&#10;data augmentation. Additionally, on three classification benchmarks,&#10;pre-trained Seq2Seq model outperforms other data augmentation methods in a&#10;low-resource setting. Further, we explore how different pre-trained model based&#10;data augmentation differs in-terms of data diversity, and how well such methods&#10;preserve the class-label information.&#10;" />
          <attvalue for="2" value="&#10;&#10;Data augmentation (DA) is a widely used technique to increase the size of the training data. Increasing training data size is often essential to reduce overfitting and enhance the robustness of machine learning models in low-data regime tasks. &#10;&#10;In natural language processing (NLP), several word replacement based methods have been explored for data augmentation. In particular, ~\cite{wei2019eda} showed that simple word replacement using knowledge bases like WordNet~\cite{miller1998wordnet} improves classification performance. Further, ~\cite{kobayashi2018contextual} utilized language models (LM) to augment training data. However, such methods struggle with preserving class labels. For example, non-conditional DA for an input sentence of sentiment classification task ``a small impact with a big movie&quot; leads to ``a small movie with a big impact&quot;. Using such augmented data for training, with the original input sentence's label (i.e. negative sentiment in this example) would negatively impact the performance of the resulting model. &#10;&#10;To alleviate this issue, ~\cite{wu2019conditional} proposed conditional BERT (CBERT) model which extends BERT ~\cite{devlin2018bert} masked language modeling (MLM) task, by considering class labels to predict the masked tokens. Since their method relies on modifying BERT model's segment embedding, it cannot be generalized to other pre-trained LMs without segment embeddings. &#10;&#10;Similarly, ~\cite{anaby2019not} used GPT2~\cite{radford2019language} for DA where examples are generated for a given class by providing class as input to a fine-tuned model. In their work, GPT2 is used to generate $10$ times the number of examples required for augmentation and then the generated sentences are selected based on the model confidence score. As data selection is applied only to GPT2 but not to the other models, the augmentation methods can not be fairly compared. Due to such discrepancies, it is not straightforward to comprehend how the generated data using different pre-trained models varies from each other and their impact on downstream model performance. &#10;&#10;This paper proposes a unified approach to use any pre-trained transformer \cite{vaswani2017attention} based models for data augmentation. In particular, we explore three different pre-trained model types for DA, including 1) an autoencoder (AE) LM: BERT, 2) an auto-regressive (AR) LM: GPT2, and 3) a pre-trained seq2seq model: BART~\cite{lewis2019bart}. We apply the data generation for three different NLP tasks: sentiment classification, intent classification, and question classification. &#10;&#10;In order to understand the significance of DA, we simulate a low-resource data scenario, where we utilize only 10 training examples per class in a classification task. Section \ref{datasets} provides details of the task and corpora. &#10;&#10;We show that all three types of pre-trained models can be effectively used for DA, and using the generated data leads to improvement in classification performance in the low-data regime setting. Among three types of methods, pre-trained seq2seq model provides the best performance. Our code is available at. &#10;&#10;Our contribution is three-fold: (1) implementation of a seq2seq pre-trained model based data augmentation, (2) experimental comparison of different data augmentation methods using conditional pre-trained model, (3) a unified data augmentation approach with practical guidelines for using different types of pre-trained models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Data Augmentation Techniques, Transformer Model Evaluation, Linguistics, Artificial Intelligence, Mathematics, Language Model Pretraining, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1805.06201" label="1805.06201">
        <attvalues>
          <attvalue for="0" value="Contextual Augmentation: Data Augmentation by Words with Paradigmatic&#10;  Relations" />
          <attvalue for="1" value="  We propose a novel data augmentation for labeled sentences called contextual&#10;augmentation. We assume an invariance that sentences are natural even if the&#10;words in the sentences are replaced with other words with paradigmatic&#10;relations. We stochastically replace words with other words that are predicted&#10;by a bi-directional language model at the word positions. Words predicted&#10;according to a context are numerous but appropriate for the augmentation of the&#10;original words. Furthermore, we retrofit a language model with a&#10;label-conditional architecture, which allows the model to augment sentences&#10;without breaking the label-compatibility. Through the experiments for six&#10;various different text classification tasks, we demonstrate that the proposed&#10;method improves classifiers based on the convolutional or recurrent neural&#10;networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03118" label="1911.03118">
        <attvalues>
          <attvalue for="0" value="Not Enough Data? Deep Learning to the Rescue!" />
          <attvalue for="1" value="  Based on recent advances in natural language modeling and those in text&#10;generation capabilities, we propose a novel data augmentation method for text&#10;classification tasks. We use a powerful pre-trained neural network model to&#10;artificially synthesize new labeled data for supervised learning. We mainly&#10;focus on cases with scarce labeled data. Our method, referred to as&#10;language-model-based data augmentation (LAMBADA), involves fine-tuning a&#10;state-of-the-art language generator to a specific task through an initial&#10;training phase on the existing (usually small) labeled data. Using the&#10;fine-tuned model and given a class label, new sentences for the class are&#10;generated. Our process then filters these new sentences by using a classifier&#10;trained on the original data. In a series of experiments, we show that LAMBADA&#10;improves classifiers' performance on a variety of datasets. Moreover, LAMBADA&#10;significantly improves upon the state-of-the-art techniques for data&#10;augmentation, specifically those applicable to text classification tasks with&#10;little data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Previous textual data augmentation approaches focus on sample alteration~\cite{kobayashi2018contextual,wu2019conditional,wei2019eda,mueller2016siamese,jungiewicz2019towards}, in which a single sentence is altered in one way or another, to generate a new sentence while preserving the original class.&#10;One set of these approaches make local changes only within a given sentence, primarily by synonym replacement of a word or multiple words.&#10;One of the recent methods in this category is \textsl{easy data augmentation (EDA)}~\cite{wei2019eda}, which uses simple operations such as synonym replacement and random swap~\cite{miller1995wordnet}.&#10;Another method, \textsl{conditional BERT contextual augmentation} recently introduced in~\cite{wu2019conditional}, proposes fine-tuned&#10;BERT~\cite{devlin2019bert} for data augmentation by carrying out a masked prediction of words, while conditioning on the class label.&#10;Presumably, methods that make only local changes will produce sentences with a structure similar to the original ones, thus yielding low corpus-level variability.&#10;&#10;Other recent possible approaches to textual data augmentation generate whole sentences rather than making a few local changes.&#10;The approaches include using \textsl{variational autoencoding} (VAE)~\cite{kingma2014auto}, \textsl{round-trip translation}~\cite{yu2018qanet}, \textsl{paraphrasing}~\cite{kumar2019submodular}, and methods based on \textsl{generative adversarial networks}~\cite{tanaka2019data}.&#10;They also include \textsl{data noising techniques}, such as altering words in the input of self-encoder networks in order to generate a different sentence~\cite{xie2017data,zolna2017fraternal,li2018undeepvo}, or introducing noise on the word-embedding level.&#10;These methods were analyzed in~\cite{marivate2019improving}.&#10;Although a viable option when no access to a formal synonym model exists, they require abundant training data.&#10;&#10;Last year, several exciting deep learning methods \cite{vaswani2017attention}&#10;pushed the boundaries of natural language technology. &#10;They introduced new neural architectures and highly effective transfer learning techniques that dramatically improve natural language processing.&#10;These methods enable the development of new high-performance deep learning models such as \textsl{ELMO}~\cite{peters2018elmo}, \textsl{GPT}~\cite{radford2018improving}, \textsl{BERT}~\cite{devlin2019bert}, and \textsl{GPT-2}~\cite{radford2019language}.&#10;Common to these models is a pre-train phase, in which the models are trained on enormous bodies of publicly available text, and a fine-tuned phase, in which they are further trained on task-specific data and loss functions.&#10;&#10;When introduced, these models processed natural language better than ever, breaking records in a variety of benchmark tasks related to natural language processing and understanding, as well as tasks involving text generation.&#10;For example, when GPT was first introduced~\cite{radford2018improving}, it improved the state-of-the-art in 12 benchmark tasks, including textual entailment, semantic similarity, sentiment analysis, and commonsense reasoning. These models can produce high-quality sentences even when fine-tuned on small training data. Table~\ref{tab:generated_samples}, shows an example of a few generated sentences based on a small dataset consisting of five sentences per class.&#10;&#10;These results suggest a counter-intuitive text classification approach: is it possible to fine-tune a pre-trained model and use it to generate new high-quality sentences that will improve the performance of a text classifier?&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2406.15209" label="2406.15209">
        <attvalues>
          <attvalue for="0" value="Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language&#10;  Understanding" />
          <attvalue for="1" value="  Zero-shot spoken language understanding (SLU) enables systems to comprehend&#10;user utterances in new domains without prior exposure to training data. Recent&#10;studies often rely on large language models (LLMs), leading to excessive&#10;footprints and complexity. This paper proposes the use of Whisper, a standalone&#10;speech processing model, for zero-shot end-to-end (E2E) SLU. To handle unseen&#10;semantic labels, SLU tasks are integrated into a question-answering (QA)&#10;framework, which prompts the Whisper decoder for semantics deduction. The&#10;system is efficiently trained with prefix-tuning, optimising a minimal set of&#10;parameters rather than the entire Whisper model. We show that the proposed&#10;system achieves a 40.7% absolute gain for slot filling (SLU-F1) on SLURP&#10;compared to a recently introduced zero-shot benchmark. Furthermore, it performs&#10;comparably to a Whisper-GPT-2 modular system under both in-corpus and&#10;cross-corpus evaluation settings, but with a relative 34.8% reduction in model&#10;parameters.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken Language Understanding (SLU) is a fundamental technology in conversational AI, dedicated to converting spoken utterances into semantic elements such as user intents, entities and emotions \cite{tur2011spoken}. The training of SLU models involves expensive data collection, requiring crowdsourced speech recording and expert annotation. This has led to increased research targeting SLU within zero-shot learning scenarios, which aims to develop a generalised system capable of predicting unseen semantic labels without additional training on annotated data.&#10;&#10;Previous studies primarily focused on text-based zero-shot natural language understanding (NLU) \cite{xia2018zero, bapna2017towards, shah2019robust, lee2019zero}, which processes transcripts produced by an automatic speech recognition (ASR) model to create a modular solution to zero-shot SLU. Among these studies, the prompt-based question-answering (QA) framework \cite{du2021qa, mehri2021gensf, li2023generative} has gained popularity, driven by the recent advancements in generative large language models (LLMs) \cite{devlin2019bert, radford2019language, raffel2020exploring}. This approach involves crafting a descriptive question for each semantic label (e.g. ``What is the name of the game?'' for the slot type {\fontfamily{qcr} game\_name}) and prompting an LLM to produce answers (slot values) that fulfill the SLU task. Unlike traditional classification or sequence-tagging methods \cite{mesnil2014using, louvan2020recent} that require predefined output classes, the QA framework offers flexibility in accommodating unseen semantics, relying solely on generating relevant questions for the new labels.&#10;&#10;Over the years, SLU has evolved from modular systems to more efficient end-to-end (E2E) models \cite{lugosch2019speech, serdyuk2018towards, haghani2018audio}. However, research on zero-shot E2E SLU remains limited. A recent study introduces a zero-shot audio-to-intent classification framework \cite{elluru2023generalized}, identifying unseen intents by comparing the speech embedding similarity between the test utterance and enrollment samples. Furthermore, \cite{sun2023knowledge} presents a knowledge-aware audio-grounded (KA2G) generative framework for zero-shot slot filling. Similar to modular systems, KA2G incorporates ASR and NLU components to implement the QA-style slot value generation. In this model, these components are aligned in terms of decoder states to facilitate E2E training and inference.&#10;&#10;In tandem with the rise of LLMs, advanced speech-to-text (STT) models are emerging, represented by OpenAI's Whisper \cite{radford2023robust}. Developed using a substantial volume of audio-transcript pairs from the internet, Whisper demonstrates outstanding performance in ASR, speech translation (ST), and has been extended to other SLU tasks \cite{wang2023whislu, meeus2023whisper, porjazovski2023advancing} under supervised learning settings. Particularly, these implementations do not depend on external language models. In this paper, we aim to unveil Whisper's capabilities in simultaneously addressing two zero-shot SLU challenges: intent classification and slot filling. Our contributions are summarised as follows:&#10;\begin{itemize}&#10; \item We investigate the application of the Whisper model for zero-shot E2E SLU, and show that it significantly outperforms the existing baseline on the SLURP benchmark \cite{bastianelli2020slurp}.&#10; \item To enable zero-shot functionalities, we reformulate the SLU tasks as QA problems and accordingly adapt the Whisper model through decoder-prompting and prefix-tuning.&#10; \item The proposed system achieves competitive performance compared to a Whisper-GPT-2 pipeline structure, while maintaining a substantially smaller footprint.&#10; \item We conduct cross-corpus zero-shot evaluations to validate the generalisation of the proposed system, trained on SLURP and tested on FSC \cite{lugosch2019speech} and SmartLight \cite{saade2019spoken} datasets.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Large Language Models, Linguistics, Spoken Language Understanding, Speech Processing Systems, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1809.00385" label="1809.00385">
        <attvalues>
          <attvalue for="0" value="Zero-shot User Intent Detection via Capsule Neural Networks" />
          <attvalue for="1" value="  User intent detection plays a critical role in question-answering and dialog&#10;systems. Most previous works treat intent detection as a classification problem&#10;where utterances are labeled with predefined intents. However, it is&#10;labor-intensive and time-consuming to label users' utterances as intents are&#10;diversely expressed and novel intents will continually be involved. Instead, we&#10;study the zero-shot intent detection problem, which aims to detect emerging&#10;user intents where no labeled utterances are currently available. We propose&#10;two capsule-based architectures: INTENT-CAPSNET that extracts semantic features&#10;from utterances and aggregates them to discriminate existing intents, and&#10;INTENTCAPSNET-ZSL which gives INTENTCAPSNET the zero-shot learning ability to&#10;discriminate emerging intents via knowledge transfer from existing intents.&#10;Experiments on two real-world datasets show that our model not only can better&#10;discriminate diversely expressed existing intents, but is also able to&#10;discriminate emerging intents when no labeled utterances are available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.03670" label="1904.03670">
        <attvalues>
          <attvalue for="0" value="Speech Model Pre-training for End-to-End Spoken Language Understanding" />
          <attvalue for="1" value="  Whereas conventional spoken language understanding (SLU) systems map speech&#10;to text, and then text to intent, end-to-end SLU systems map speech directly to&#10;intent through a single trainable model. Achieving high accuracy with these&#10;end-to-end models without a large amount of training data is difficult. We&#10;propose a method to reduce the data requirements of end-to-end SLU in which the&#10;model is first pre-trained to predict words and phonemes, thus learning good&#10;features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and&#10;show that our method improves performance both when the full dataset is used&#10;for training and when only a small subset is used. We also describe preliminary&#10;experiments to gauge the model's ability to generalize to new phrases not heard&#10;during training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.02482" label="2311.02482">
        <attvalues>
          <attvalue for="0" value="Generalized zero-shot audio-to-intent classification" />
          <attvalue for="1" value="  Spoken language understanding systems using audio-only data are gaining&#10;popularity, yet their ability to handle unseen intents remains limited. In this&#10;study, we propose a generalized zero-shot audio-to-intent classification&#10;framework with only a few sample text sentences per intent. To achieve this, we&#10;first train a supervised audio-to-intent classifier by making use of a&#10;self-supervised pre-trained model. We then leverage a neural audio synthesizer&#10;to create audio embeddings for sample text utterances and perform generalized&#10;zero-shot classification on unseen intents using cosine similarity. We also&#10;propose a multimodal training strategy that incorporates lexical information&#10;into the audio representation to improve zero-shot performance. Our multimodal&#10;training approach improves the accuracy of zero-shot intent classification on&#10;unseen intents of SLURP by 2.75% and 18.2% for the SLURP and internal&#10;goal-oriented dialog datasets, respectively, compared to audio-only training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.01764" label="2307.01764">
        <attvalues>
          <attvalue for="0" value="Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited&#10;  Annotated Data" />
          <attvalue for="1" value="  Manually annotating fine-grained slot-value labels for task-oriented dialogue&#10;(ToD) systems is an expensive and time-consuming endeavour. This motivates&#10;research into slot-filling methods that operate with limited amounts of&#10;labelled data. Moreover, the majority of current work on ToD is based solely on&#10;text as the input modality, neglecting the additional challenges of imperfect&#10;automatic speech recognition (ASR) when working with spoken language. In this&#10;work, we propose a Knowledge-Aware Audio-Grounded generative slot-filling&#10;framework, termed KA2G, that focuses on few-shot and zero-shot slot filling for&#10;ToD with speech input. KA2G achieves robust and data-efficient slot filling for&#10;speech-based ToD by 1) framing it as a text generation task, 2) grounding text&#10;generation additionally in the audio modality, and 3) conditioning on available&#10;external knowledge (e.g. a predefined list of possible slot values). We show&#10;that combining both modalities within the KA2G framework improves the&#10;robustness against ASR errors. Further, the knowledge-aware slot-value&#10;generator in KA2G, implemented via a pointer generator mechanism, particularly&#10;benefits few-shot and zero-shot learning. Experiments, conducted on the&#10;standard speech-based single-turn SLURP dataset and a multi-turn dataset&#10;extracted from a commercial ToD system, display strong and consistent gains&#10;over prior work, especially in few-shot and zero-shot setups.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.13205" label="2011.13205">
        <attvalues>
          <attvalue for="0" value="SLURP: A Spoken Language Understanding Resource Package" />
          <attvalue for="1" value="  Spoken Language Understanding infers semantic meaning directly from audio&#10;data, and thus promises to reduce error propagation and misunderstandings in&#10;end-user applications. However, publicly available SLU resources are limited.&#10;In this paper, we release SLURP, a new SLU package containing the following:&#10;(1) A new challenging dataset in English spanning 18 domains, which is&#10;substantially bigger and linguistically more diverse than existing datasets;&#10;(2) Competitive baselines based on state-of-the-art NLU and ASR systems; (3) A&#10;new transparent metric for entity labelling which enables a detailed error&#10;analysis for identifying potential areas of improvement. SLURP is available at&#10;https: //github.com/pswietojanski/slurp.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.12735" label="1810.12735">
        <attvalues>
          <attvalue for="0" value="Spoken Language Understanding on the Edge" />
          <attvalue for="1" value="  We consider the problem of performing Spoken Language Understanding (SLU) on&#10;small devices typical of IoT applications. Our contributions are twofold.&#10;First, we outline the design of an embedded, private-by-design SLU system and&#10;show that it has performance on par with cloud-based commercial solutions.&#10;Second, we release the datasets used in our experiments in the interest of&#10;reproducibility and in the hope that they can prove useful to the SLU&#10;community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.06563" label="2406.06563">
        <attvalues>
          <attvalue for="0" value="Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts&#10;  Language Models" />
          <attvalue for="1" value="  In this technical report, we introduce the training methodologies implemented&#10;in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE)&#10;large language model (LLM) with 146 billion parameters and 16 experts. It is&#10;initialized from the pre-existing dense checkpoints of our Skywork-13B model.&#10;We explore the comparative effectiveness of upcycling versus training from&#10;scratch initializations. Our findings suggest that the choice between these two&#10;approaches should consider both the performance of the existing dense&#10;checkpoints and the MoE training budget. We highlight two innovative&#10;techniques: gating logit normalization, which improves expert diversification,&#10;and adaptive auxiliary loss coefficients, allowing for layer-specific&#10;adjustment of auxiliary loss coefficients. Our experimental results validate&#10;the effectiveness of these methods. Leveraging these techniques and insights,&#10;we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile&#10;corpus. The evaluation results demonstrate that our model delivers strong&#10;performance across a wide range of benchmarks.&#10;" />
          <attvalue for="2" value="&#10;Recent advancements in the field of artificial intelligence have seen large language models (LLMs) \cite{instruct_gpt, gpt4_report, gpt4_sparks, claude3, llama2, llama3modelcard, gemini_new, deepseekmoev2} revolutionize numerous branches of natural language processing (NLP), encompassing tasks from machine translation to automated summarization. However, the computational demands and associated costs of training and deploying state-of-the-art dense LLMs pose significant challenges, particularly at the scale of tens or hundreds of billions of parameters. In response to these challenges, sparse models, such as Mixture-of-Experts (MoE), have gained prominence \cite{switch_transformer, gshard, glam, deepseekmoev1, deepseekmoev2}. These models offer a more economically viable alternative by distributing computation across various specialized sub-models or ``experts'', potentially matching or even surpassing the performance of their dense counterparts with a fraction of the resource requirements \cite{artetxe2022efficient, deepspeedmoe, moe_scaling_law}.&#10;&#10;In light of these developments, this technical report introduces Skywork-MoE, a high-performance MoE large language model with 146 billion parameters and 16 experts. This model leverages the foundational architecture of our previously developed Skywork-13B model \cite{wei2023skywork}, utilizing its dense checkpoints as the initial setup \cite{upcycling}. We conduct experimental analysis on relative benefits of two pivotal strategies in LLM development: upcycling from existing dense models versus initiating training from scratch. Through rigorous evaluation, we provide nuanced insights into how the initial conditions and training budgets influence the effectiveness of these approaches, offering practical guidance on their application. Skywork-MoE embodies the forefront of MoE research by incorporating two novel training techniques: gating logit normalization and adaptive auxiliary loss coefficients. The former aims to enhance the diversification among the experts, while the latter facilitates the tailored adjustment of auxiliary loss coefficients at different layers of the model. Moreover, the training of Skywork-MoE was conducted on a condensed subset of the SkyPile corpus \cite{wei2023skywork}, with subsequent evaluations demonstrating its robust performance across a diverse array of benchmarks. This report aims to detail these innovations and findings, setting a new benchmark for the efficiency and efficacy of MoE models in large-scale language processing tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Training Methodologies, Model Optimization Techniques, Mathematics" />
        </attvalues>
      </node>
      <node id="2112.10684" label="2112.10684">
        <attvalues>
          <attvalue for="0" value="Efficient Large Scale Language Modeling with Mixtures of Experts" />
          <attvalue for="1" value="  Mixture of Experts layers (MoEs) enable efficient scaling of language models&#10;through conditional computation. This paper presents a detailed empirical study&#10;of how autoregressive MoE language models scale in comparison with dense models&#10;in a wide range of settings: in- and out-of-domain language modeling, zero- and&#10;few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning,&#10;we find MoEs to be substantially more compute efficient. At more modest&#10;training budgets, MoEs can match the performance of dense models using $\sim$4&#10;times less compute. This gap narrows at scale, but our largest MoE model (1.1T&#10;parameters) consistently outperforms a compute-equivalent dense model (6.7B&#10;parameters). Overall, this performance gap varies greatly across tasks and&#10;domains, suggesting that MoE and dense models generalize differently in ways&#10;that are worthy of future study. We make our code and models publicly available&#10;for research use.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.19341" label="2310.19341">
        <attvalues>
          <attvalue for="0" value="Skywork: A More Open Bilingual Foundation Model" />
          <attvalue for="1" value="  In this technical report, we present Skywork-13B, a family of large language&#10;models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both&#10;English and Chinese texts. This bilingual foundation model is the most&#10;extensively trained and openly published LLMs of comparable size to date. We&#10;introduce a two-stage training methodology using a segmented corpus, targeting&#10;general purpose training and then domain-specific enhancement training,&#10;respectively. We show that our model not only excels on popular benchmarks, but&#10;also achieves \emph{state of the art} performance in Chinese language modeling&#10;on diverse domains. Furthermore, we propose a novel leakage detection method,&#10;demonstrating that test data contamination is a pressing issue warranting&#10;further investigation by the LLM community. To spur future research, we release&#10;Skywork-13B along with checkpoints obtained during intermediate stages of the&#10;training process. We are also releasing part of our SkyPile corpus, a&#10;collection of over 150 billion tokens of web text, which is the largest high&#10;quality open Chinese pre-training corpus to date. We hope Skywork-13B and our&#10;open corpus will serve as a valuable open-source resource to democratize access&#10;to high-quality LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.05055" label="2212.05055">
        <attvalues>
          <attvalue for="0" value="Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints" />
          <attvalue for="1" value="  Training large, deep neural networks to convergence can be prohibitively&#10;expensive. As a result, often only a small selection of popular, dense models&#10;are reused across different contexts and tasks. Increasingly, sparsely&#10;activated models, which seek to decouple model size from computation costs, are&#10;becoming an attractive alternative to dense models. Although more efficient in&#10;terms of quality and computation cost, sparse models remain data-hungry and&#10;costly to train from scratch in the large scale regime. In this work, we&#10;propose sparse upcycling -- a simple way to reuse sunk training costs by&#10;initializing a sparsely activated Mixture-of-Experts model from a dense&#10;checkpoint. We show that sparsely upcycled T5 Base, Large, and XL language&#10;models and Vision Transformer Base and Large models, respectively,&#10;significantly outperform their dense counterparts on SuperGLUE and ImageNet,&#10;using only ~50% of the initial dense pretraining sunk cost. The upcycled models&#10;also outperform sparse models trained from scratch on 100% of the initial dense&#10;pretraining computation budget.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.11453" label="2001.11453">
        <attvalues>
          <attvalue for="0" value="Parameter Space Factorization for Zero-Shot Learning across Tasks and&#10;  Languages" />
          <attvalue for="1" value="  Most combinations of NLP tasks and language varieties lack in-domain examples&#10;for supervised training because of the paucity of annotated data. How can&#10;neural models make sample-efficient generalizations from task-language&#10;combinations with available data to low-resource ones? In this work, we propose&#10;a Bayesian generative model for the space of neural parameters. We assume that&#10;this space can be factorized into latent variables for each language and each&#10;task. We infer the posteriors over such latent variables based on data from&#10;seen task-language combinations through variational inference. This enables&#10;zero-shot classification on unseen combinations at prediction time. For&#10;instance, given training data for named entity recognition (NER) in Vietnamese&#10;and for part-of-speech (POS) tagging in Wolof, our model can perform accurate&#10;predictions for NER in Wolof. In particular, we experiment with a typologically&#10;diverse sample of 33 languages from 4 continents and 11 families, and show that&#10;our model yields comparable or better results than state-of-the-art, zero-shot&#10;cross-lingual transfer methods. Moreover, we demonstrate that approximate&#10;Bayesian model averaging results in smoother predictive distributions, whose&#10;entropy inversely correlates with accuracy. Hence, the proposed framework also&#10;offers robust estimates of prediction uncertainty. Our code is located at&#10;github.com/cambridgeltl/parameter-factorization&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Our approach builds on ideas from several different fields: cross-lingual transfer in NLP, with a particular focus on sequence labeling tasks, as well as matrix factorization, contextual parameter generation, and neural Bayesian methods.&#10;&#10;Cross-Lingual Transfer for Sequence Labeling. &#10;One of the two dominant approaches for cross-lingual transfer is projecting annotations from a source language text to a target language text. This technique was pioneered by \newcite{yarowsky2001inducing} and \newcite{hwa-2005} for parsing, and later extended to applications such as POS tagging \cite{Das:2011acl,Garrette:2013acl,Tackstrom:2012naacl,Duong:2014emnlp,Huck:2019ws} &#10;and NER \cite{Ni:2017acl,Enghoff:2018ws,Agerri:2018lrec,Jain:2019emnlp}. This requires tokens to be aligned through a parallel corpus, a machine translation system, or a bilingual dictionary \cite{Durrett:2012emnlp,Mayhew:2017emnlp}. However, creating machine translation and word-alignment systems demands parallel texts in the first place, while automatically induced bilingual lexicons are noisy and offer only limited coverage \cite{Artetxe:2018iclr,Duan:2020acl}. Furthermore, errors inherent to such systems cascade along the projection pipeline \cite{Agic:2015acl}. &#10;&#10;The second approach, model transfer, offers higher flexibility \cite{conneau2018xnli}. The main idea is to train a model directly on the source data, and then deploy it onto target data \cite{zeman2008cross}. Crucially, bridging between different lexica requires input features to be language-agnostic. While originally this implied delexicalization, replacing words with universal POS tags \cite{McDonald:2011emnlp,Dehouck:2017eacl}, cross-lingual Brown clusters \cite{Tackstrom:2012naacl,Rasooli:2017tacl}, or cross-lingual knowledge base grounding through wikification \cite{Camacho:2016nasari,Tsai:2016conll}, more recently these have been supplanted by cross-lingual word embeddings \cite{Ammar:2016tacl,zhang-etal-2016-ten,Xie:2018emnlp,Ruder:2019jair} and multilingual pretrained language models \cite{devlin2019bert,Conneau:2020acl}. &#10;&#10;An orthogonal research thread regards the selection of the source language(s).&#10;In particular, multi-source transfer was shown to surpass single-best source transfer in NER \cite{Fang:2017acl,rahimi2019massively} and POS tagging \cite{Enghoff:2018ws,Plank:2018emnlp}. Our parameter space factorization model can be conceived as an extension of multi-source cross-lingual model transfer to a cross-task setting.&#10;&#10;Data Matrix Factorization.&#10;Although we are the first to propose a factorization of the parameter space for unseen combinations of tasks and languages, the factorization of data for collaborative filtering and social recommendation is an established research area. In particular, the missing values in sparse data structures such as user-movie review matrices can be filled via probabilistic matrix factorization (PMF) through a linear combination of user and movie matrices \cite[inter alia]{mnih2008probabilistic,ma2008sorec,shan2010generalized} or through neural networks \cite{dziugaite2015neural}. Inference for PMF can be carried out through MAP inference \cite{dziugaite2015neural}, Markov chain Monte Carlo \cite[MCMC;][]{salakhutdinov2008bayesian} or stochastic variational inference \cite{stoleematrix}. Contrary to prior work, we perform factorization on latent variables (task- and language-specific parameters) rather than observed ones (data).&#10;&#10;Contextual Parameter Generation. Our model is reminiscent of the idea that parameters can be conditioned on language representations, as proposed by \cite{Platanios:2018emnlp}. However, since this approach is limited to a single task and a joint learning setting, it is not suitable for generalization in a zero-shot transfer setting.&#10;&#10;Bayesian Neural Models. So far, these models have found only limited application in NLP for resource-poor languages, despite their desirable properties. Firstly, they can incorporate priors over parameters to endow neural networks with the correct inductive biases towards language: \cite{Ponti:2019emnlp} constructed a prior imbued with universal linguistic knowledge for zero- and few-shot character-level language modeling. Secondly, they avoid the risk of over-fitting by taking into account uncertainty. For instance, \cite{Shareghi:2019naacl} and&#10;\cite{doitch2019perturbation} use a perturbation model to sample high-quality and diverse solutions for structured prediction in cross-lingual parsing.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Low-Resource Language Processing, Neural Model Generalization, Natural Language Processing, Statistics, Bayesian Uncertainty Estimation" />
        </attvalues>
      </node>
      <node id="2403.19887" label="2403.19887">
        <attvalues>
          <attvalue for="0" value="Jamba: A Hybrid Transformer-Mamba Language Model" />
          <attvalue for="1" value="  We present Jamba, a new base large language model based on a novel hybrid&#10;Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba&#10;interleaves blocks of Transformer and Mamba layers, enjoying the benefits of&#10;both model families. MoE is added in some of these layers to increase model&#10;capacity while keeping active parameter usage manageable. This flexible&#10;architecture allows resource- and objective-specific configurations. In the&#10;particular configuration we have implemented, we end up with a powerful model&#10;that fits in a single 80GB GPU. Built at large scale, Jamba provides high&#10;throughput and small memory footprint compared to vanilla Transformers, and at&#10;the same time state-of-the-art performance on standard language model&#10;benchmarks and long-context evaluations. Remarkably, the model presents strong&#10;results for up to 256K tokens context length. We study various architectural&#10;decisions, such as how to combine Transformer and Mamba layers, and how to mix&#10;experts, and show that some of them are crucial in large scale modeling. We&#10;also describe several interesting properties of these architectures which the&#10;training and evaluation of Jamba have revealed, and plan to release checkpoints&#10;from various ablation runs, to encourage further exploration of this novel&#10;architecture. We make the weights of our implementation of Jamba publicly&#10;available under a permissive license.&#10;" />
          <attvalue for="2" value="&#10;&#10;We introduce \jamba, a new publicly available large language model. \jamba is based on a novel hybrid architecture, which combines Transformer layers \cite{vaswani2017attention} with Mamba layers \cite{gu2023mamba}, a recent state-space model \cite{gu2021combining,gu2021efficiently}, as well as a mixture-of-experts (MoE) module \cite{shazeer2016outrageously,fedus2022switch}. \jamba thus combines two orthogonal architectural designs that together give it improved performance and higher throughput, while maintaining a manageable memory footprint. The 7B-based \jamba model (12B active parameters, 52B total available parameters) we are releasing was designed to fit in a single 80GB GPU, but the \jamba architecture supports other design choices, depending on one's hardware and performance requirements. &#10;&#10;The fundamental novelty of \jamba is its hybrid Transformer-Mamba architecture (though see mention below of recent related efforts). Despite the immense popularity of the Transformer as the predominant architecture for language models, it suffers from two main drawbacks. First, its high memory and compute requirements hinders the processing of long contexts, where the key-value (KV) cache size becomes a limiting factor. Second, its lack of a single summary state entails slow inference and low throughput, since each generated token performs a computation on the entire context. In contrast, older recurrent neural network (RNN) models, which summarize an arbitrarily long context in a single hidden state, do not suffer from these limitations. RNN models have their own shortcomings, however. They are costly to train since training cannot be parallelized across time steps. And they struggle with long distance relationships, which the hidden state captures to only a limited extent. &#10;&#10;Recent state space models (SSMs) like Mamba are more efficient to train than RNNs and are more capable at handling long distance relationships, but still lag behind the performance of comparably sized Transformer language models. Taking advantage of both model families, \jamba combines Transformer and Mamba layers, at a certain ratio. Varying the ratio of Transformer/Mamba layers allows balancing memory usage, efficient training, and long context capabilities. &#10;&#10;A few other recent attempts to combine Attention and SSM modules are worth noting. &#10;\cite{zuo2022efficient} mixes an S4 layer \cite{gu2021efficiently} with a local attention layer, followed by a sequence of local attention layers; it shows experiments with small models and simple tasks. &#10;\cite{gu2023mamba} reports that interleaving Mamba and attention layers is only slightly better than pure Mamba in terms of perplexity, with models up to 1.3B parameters. &#10;\cite{pilault2023block} starts with an SSM layer followed by chunk-based Transformers, with models up to 1.3B showing improved perplexity. &#10;\cite{fathullah23_interspeech} adds an SSM layer before the self-attention in a Transformer layer, while \cite{saon2023diagonal} adds the SSM after the self-attention, both showing improvements on speech recognition. &#10;\cite{park2024can} replaces the MLP layers in the Transformer by Mamba layers, and shows benefits in simple tasks. These efforts are different from \jamba both in the particular way in which the SSM component is mixed with the attention one, and in the scale of implementation. Closest are perhaps H3 \cite{fu2022hungry}, a specially designed SSM that enables induction capabilities, and a generalization called Hyena \cite{poli2023hyena}. The former proposed a hybrid architecture that replaces the second and middle layers with self-attention, and was implemented with up to 2.7B parameters and 400B training tokens. However, as shown in \cite{gu2023mamba}, its perfomance lags that of pure Mamba.&#10;Based on Hyena, StripedHyena \cite{stripedhyena} interleaves attention and SSM layers in a 7B parameter model. However, it lags behind the Attention-only Mistral-7B \cite{jiang2023mistral}. &#10;All of this renders \jamba the first production-grade Attention-SSM hybrid model.&#10;Scaling the hybrid \jamba architecture required overcoming several obstacles, which we dicsuss in Section \ref{sec:ablations}. &#10;&#10;\jamba also includes MoE layers \cite{shazeer2016outrageously,fedus2022switch}, which allow increasing the model capacity (total number of available parameters) without increasing compute requirements (number of active parameters). MoE is a flexible approach that enables training extremely large models with strong performance \cite{jiang2024mixtral}. In \jamba, MoE is applied to some of the MLP layers. The more MoE layers, and the more experts in each MoE layer, the larger the total number of model parameters. In contrast, the more experts we use at each forward pass, the larger the number of active parameters as well as the compute requirement. In our implementation of \jamba, we apply MoE at every other layer, with 16 experts and the top-2 experts used at each token (a more detailed discussion of the model architecture is provided below).&#10;&#10;We evaluated our implementation of \jamba on a wide range of benchmarks and found it performs comparably to Mixtral-8x7B \cite{jiang2024mixtral}, which has a similar number of parameters, and also to the larger Llama-2 70B \cite{touvron2023llama}. In addition, our model supports a context length of 256K tokens -- the longest supported context length for production-grade publicly available models. On long-context evaluations, \jamba outperformes Mixtral on most of the evaluated datasets. At the same time, \jamba is extremely efficient; for example, its throughput is 3x that of Mixtral-8x7B for long contexts. Moreover, our model fits in a single GPU (with 8bit weights) even with contexts of over 128K tokens, which is impossible with similar-size attention-only models such as Mixtral-8x7B. &#10;&#10;Somewhat unusual for a new architecture, we release \jamba (12B active parameters, 52B total available parameters) under Apache 2.0 license:. We do so since we feel that the novel architecture of \jamba calls for further study, experimentation, and optimization by the community. Our design was based on various ablation experiments we conducted to explore the effect of different tradeoffs and design choices, and insights gleaned from those. These ablations were performed at scales of up to 7B parameters, and training runs of up to 250B tokens. We plan to release model checkpoints from these runs. &#10;&#10;Important notice: The \jamba model released is a pretrained base model, which did not go through alignment or instruction tuning, and does not have moderation mechanisms. It should not be used in production environments or with end users without additional adaptation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Large Scale Modeling, Computer Science, Linguistics, Cognitive Science, Language Model Architecture, Deep Learning Techniques, Mathematics" />
        </attvalues>
      </node>
      <node id="2312.00752" label="2312.00752">
        <attvalues>
          <attvalue for="0" value="Mamba: Linear-Time Sequence Modeling with Selective State Spaces" />
          <attvalue for="1" value="  Foundation models, now powering most of the exciting applications in deep&#10;learning, are almost universally based on the Transformer architecture and its&#10;core attention module. Many subquadratic-time architectures such as linear&#10;attention, gated convolution and recurrent models, and structured state space&#10;models (SSMs) have been developed to address Transformers' computational&#10;inefficiency on long sequences, but they have not performed as well as&#10;attention on important modalities such as language. We identify that a key&#10;weakness of such models is their inability to perform content-based reasoning,&#10;and make several improvements. First, simply letting the SSM parameters be&#10;functions of the input addresses their weakness with discrete modalities,&#10;allowing the model to selectively propagate or forget information along the&#10;sequence length dimension depending on the current token. Second, even though&#10;this change prevents the use of efficient convolutions, we design a&#10;hardware-aware parallel algorithm in recurrent mode. We integrate these&#10;selective SSMs into a simplified end-to-end neural network architecture without&#10;attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5$\times$&#10;higher throughput than Transformers) and linear scaling in sequence length, and&#10;its performance improves on real data up to million-length sequences. As a&#10;general sequence model backbone, Mamba achieves state-of-the-art performance&#10;across several modalities such as language, audio, and genomics. On language&#10;modeling, our Mamba-3B model outperforms Transformers of the same size and&#10;matches Transformers twice its size, both in pretraining and downstream&#10;evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1701.06538" label="1701.06538">
        <attvalues>
          <attvalue for="0" value="Outrageously Large Neural Networks: The Sparsely-Gated&#10;  Mixture-of-Experts Layer" />
          <attvalue for="1" value="  The capacity of a neural network to absorb information is limited by its&#10;number of parameters. Conditional computation, where parts of the network are&#10;active on a per-example basis, has been proposed in theory as a way of&#10;dramatically increasing model capacity without a proportional increase in&#10;computation. In practice, however, there are significant algorithmic and&#10;performance challenges. In this work, we address these challenges and finally&#10;realize the promise of conditional computation, achieving greater than 1000x&#10;improvements in model capacity with only minor losses in computational&#10;efficiency on modern GPU clusters. We introduce a Sparsely-Gated&#10;Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward&#10;sub-networks. A trainable gating network determines a sparse combination of&#10;these experts to use for each example. We apply the MoE to the tasks of&#10;language modeling and machine translation, where model capacity is critical for&#10;absorbing the vast quantities of knowledge available in the training corpora.&#10;We present model architectures in which a MoE with up to 137 billion parameters&#10;is applied convolutionally between stacked LSTM layers. On large language&#10;modeling and machine translation benchmarks, these models achieve significantly&#10;better results than state-of-the-art at lower computational cost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.08136" label="2212.08136">
        <attvalues>
          <attvalue for="0" value="Efficient Long Sequence Modeling via State Space Augmented Transformer" />
          <attvalue for="1" value="  Transformer models have achieved superior performance in various natural&#10;language processing tasks. However, the quadratic computational cost of the&#10;attention mechanism limits its practicality for long sequences. There are&#10;existing attention variants that improve the computational efficiency, but they&#10;have limited ability to effectively compute global information. In parallel to&#10;Transformer models, state space models (SSMs) are tailored for long sequences,&#10;but they are not flexible enough to capture complicated local information. We&#10;propose SPADE, short for $\underline{\textbf{S}}$tate&#10;s$\underline{\textbf{P}}$ace&#10;$\underline{\textbf{A}}$ugmente$\underline{\textbf{D}}$&#10;Transform$\underline{\textbf{E}}$r. Specifically, we augment a SSM into the&#10;bottom layer of SPADE, and we employ efficient local attention methods for the&#10;other layers. The SSM augments global information, which complements the lack&#10;of long-range dependency issue in local attention methods. Experimental results&#10;on the Long Range Arena benchmark and language modeling tasks demonstrate the&#10;effectiveness of the proposed method. To further demonstrate the scalability of&#10;SPADE, we pre-train large encoder-decoder models and present fine-tuning&#10;results on natural language understanding and natural language generation&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.09539" label="2306.09539">
        <attvalues>
          <attvalue for="0" value="Block-State Transformers" />
          <attvalue for="1" value="  State space models (SSMs) have shown impressive results on tasks that require&#10;modeling long-range dependencies and efficiently scale to long sequences owing&#10;to their subquadratic runtime complexity. Originally designed for continuous&#10;signals, SSMs have shown superior performance on a plethora of tasks, in vision&#10;and audio; however, SSMs still lag Transformer performance in Language Modeling&#10;tasks. In this work, we propose a hybrid layer named Block-State Transformer&#10;(BST), that internally combines an SSM sublayer for long-range&#10;contextualization, and a Block Transformer sublayer for short-term&#10;representation of sequences. We study three different, and completely&#10;parallelizable, variants that integrate SSMs and block-wise attention. We show&#10;that our model outperforms similar Transformer-based architectures on language&#10;modeling perplexity and generalizes to longer sequences. In addition, the&#10;Block-State Transformer demonstrates more than tenfold increase in speed at the&#10;layer level compared to the Block-Recurrent Transformer when model&#10;parallelization is employed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12498" label="2305.12498">
        <attvalues>
          <attvalue for="0" value="Multi-Head State Space Model for Speech Recognition" />
          <attvalue for="1" value="  State space models (SSMs) have recently shown promising results on&#10;small-scale sequence and language modelling tasks, rivalling and outperforming&#10;many attention-based approaches. In this paper, we propose a multi-head state&#10;space (MH-SSM) architecture equipped with special gating mechanisms, where&#10;parallel heads are taught to learn local and global temporal dynamics on&#10;sequence data. As a drop-in replacement for multi-head attention in transformer&#10;encoders, this new model significantly outperforms the transformer transducer&#10;on the LibriSpeech speech recognition corpus. Furthermore, we augment the&#10;transformer block with MH-SSMs layers, referred to as the Stateformer,&#10;achieving state-of-the-art performance on the LibriSpeech task, with word error&#10;rates of 1.76\%/4.37\% on the development and 1.91\%/4.36\% on the test sets&#10;without using an external language model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.14120" label="2302.14120">
        <attvalues>
          <attvalue for="0" value="Diagonal State Space Augmented Transformers for Speech Recognition" />
          <attvalue for="1" value="  We improve on the popular conformer architecture by replacing the depthwise&#10;temporal convolutions with diagonal state space (DSS) models. DSS is a recently&#10;introduced variant of linear RNNs obtained by discretizing a linear dynamical&#10;system with a diagonal state transition matrix. DSS layers project the input&#10;sequence onto a space of orthogonal polynomials where the choice of basis&#10;functions, metric and support is controlled by the eigenvalues of the&#10;transition matrix. We compare neural transducers with either conformer or our&#10;proposed DSS-augmented transformer (DSSformer) encoders on three public&#10;corpora: Switchboard English conversational telephone speech 300 hours,&#10;Switchboard+Fisher 2000 hours, and a spoken archive of holocaust survivor&#10;testimonials called MALACH 176 hours. On Switchboard 300/2000 hours, we reach a&#10;single model performance of 8.9%/6.7% WER on the combined test set of the Hub5&#10;2000 evaluation, respectively, and on MALACH we improve the WER by 7% relative&#10;over the previous best published result. In addition, we present empirical&#10;evidence suggesting that DSS layers learn damped Fourier basis functions where&#10;the attenuation coefficients are layer specific whereas the frequency&#10;coefficients converge to almost identical linearly-spaced values across all&#10;layers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.04248" label="2402.04248">
        <attvalues>
          <attvalue for="0" value="Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning&#10;  Tasks" />
          <attvalue for="1" value="  State-space models (SSMs), such as Mamba (Gu &amp; Dao, 2023), have been proposed&#10;as alternatives to Transformer networks in language modeling, by incorporating&#10;gating, convolutions, and input-dependent token selection to mitigate the&#10;quadratic cost of multi-head attention. Although SSMs exhibit competitive&#10;performance, their in-context learning (ICL) capabilities, a remarkable&#10;emergent property of modern language models that enables task execution without&#10;parameter optimization, remain underexplored compared to Transformers. In this&#10;study, we evaluate the ICL performance of SSMs, focusing on Mamba, against&#10;Transformer models across various tasks. Our results show that SSMs perform&#10;comparably to Transformers in standard regression ICL tasks, while&#10;outperforming them in tasks like sparse parity learning. However, SSMs fall&#10;short in tasks involving non-standard retrieval functionality. To address these&#10;limitations, we introduce a hybrid model, MambaFormer, that combines Mamba with&#10;attention blocks, surpassing individual models in tasks where they struggle&#10;independently. Our findings suggest that hybrid architectures offer promising&#10;avenues for enhancing ICL in language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.14052" label="2212.14052">
        <attvalues>
          <attvalue for="0" value="Hungry Hungry Hippos: Towards Language Modeling with State Space Models" />
          <attvalue for="1" value="  State space models (SSMs) have demonstrated state-of-the-art sequence&#10;modeling performance in some modalities, but underperform attention in language&#10;modeling. Moreover, despite scaling nearly linearly in sequence length instead&#10;of quadratically, SSMs are still slower than Transformers due to poor hardware&#10;utilization. In this paper, we make progress on understanding the expressivity&#10;gap between SSMs and attention in language modeling, and on reducing the&#10;hardware barrier between SSMs and attention. First, we use synthetic language&#10;modeling tasks to understand the gap between SSMs and attention. We find that&#10;existing SSMs struggle with two capabilities: recalling earlier tokens in the&#10;sequence and comparing tokens across the sequence. To understand the impact on&#10;language modeling, we propose a new SSM layer, H3, that is explicitly designed&#10;for these abilities. H3 matches attention on the synthetic languages and comes&#10;within 0.4 PPL of Transformers on OpenWebText. Furthermore, a hybrid&#10;125M-parameter H3-attention model that retains two attention layers&#10;surprisingly outperforms Transformers on OpenWebText by 1.0 PPL. Next, to&#10;improve the efficiency of training SSMs on modern hardware, we propose&#10;FlashConv. FlashConv uses a fused block FFT algorithm to improve efficiency on&#10;sequences up to 8K, and introduces a novel state passing algorithm that&#10;exploits the recurrent properties of SSMs to scale to longer sequences.&#10;FlashConv yields 2$\times$ speedup on the long-range arena benchmark and allows&#10;hybrid language models to generate text 2.4$\times$ faster than Transformers.&#10;Using FlashConv, we scale hybrid H3-attention language models up to 2.7B&#10;parameters on the Pile and find promising initial results, achieving lower&#10;perplexity than Transformers and outperforming Transformers in zero- and&#10;few-shot learning on a majority of tasks in the SuperGLUE benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;We present some background on state space models and linear attention, which inspired our H3 layer.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2302.10866" label="2302.10866">
        <attvalues>
          <attvalue for="0" value="Hyena Hierarchy: Towards Larger Convolutional Language Models" />
          <attvalue for="1" value="  Recent advances in deep learning have relied heavily on the use of large&#10;Transformers due to their ability to learn at scale. However, the core building&#10;block of Transformers, the attention operator, exhibits quadratic cost in&#10;sequence length, limiting the amount of context accessible. Existing&#10;subquadratic methods based on low-rank and sparse approximations need to be&#10;combined with dense attention layers to match Transformers, indicating a gap in&#10;capability. In this work, we propose Hyena, a subquadratic drop-in replacement&#10;for attention constructed by interleaving implicitly parametrized long&#10;convolutions and data-controlled gating. In recall and reasoning tasks on&#10;sequences of thousands to hundreds of thousands of tokens, Hyena improves&#10;accuracy by more than 50 points over operators relying on state-spaces and&#10;other implicit and explicit methods, matching attention-based models. We set a&#10;new state-of-the-art for dense-attention-free architectures on language&#10;modeling in standard datasets (WikiText103 and The Pile), reaching Transformer&#10;quality with a 20% reduction in training compute required at sequence length&#10;2K. Hyena operators are twice as fast as highly optimized attention at sequence&#10;length 8K, and 100x faster at sequence length 64K.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.06825" label="2310.06825">
        <attvalues>
          <attvalue for="0" value="Mistral 7B" />
          <attvalue for="1" value="  We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered&#10;for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B&#10;across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and&#10;code generation. Our model leverages grouped-query attention (GQA) for faster&#10;inference, coupled with sliding window attention (SWA) to effectively handle&#10;sequences of arbitrary length with a reduced inference cost. We also provide a&#10;model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses&#10;the Llama 2 13B -- Chat model both on human and automated benchmarks. Our&#10;models are released under the Apache 2.0 license.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.04088" label="2401.04088">
        <attvalues>
          <attvalue for="0" value="Mixtral of Experts" />
          <attvalue for="1" value="  We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model.&#10;Mixtral has the same architecture as Mistral 7B, with the difference that each&#10;layer is composed of 8 feedforward blocks (i.e. experts). For every token, at&#10;each layer, a router network selects two experts to process the current state&#10;and combine their outputs. Even though each token only sees two experts, the&#10;selected experts can be different at each timestep. As a result, each token has&#10;access to 47B parameters, but only uses 13B active parameters during inference.&#10;Mixtral was trained with a context size of 32k tokens and it outperforms or&#10;matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,&#10;Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and&#10;multilingual benchmarks. We also provide a model fine-tuned to follow&#10;instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo,&#10;Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both&#10;the base and instruct models are released under the Apache 2.0 license.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.01207" label="2109.01207">
        <attvalues>
          <attvalue for="0" value="Similarity of Sentence Representations in Multilingual LMs: Resolving&#10;  Conflicting Literature and Case Study of Baltic Languages" />
          <attvalue for="1" value="  Low-resource languages, such as Baltic languages, benefit from Large&#10;Multilingual Models (LMs) that possess remarkable cross-lingual transfer&#10;performance capabilities. This work is an interpretation and analysis study&#10;into cross-lingual representations of Multilingual LMs. Previous works&#10;hypothesized that these LMs internally project representations of different&#10;languages into a shared cross-lingual space. However, the literature produced&#10;contradictory results. In this paper, we revisit the prior work claiming that&#10;&quot;BERT is not an Interlingua&quot; and show that different languages do converge to a&#10;shared space in such language models with another choice of pooling strategy or&#10;similarity index. Then, we perform cross-lingual representational analysis for&#10;the two most popular multilingual LMs employing 378 pairwise language&#10;comparisons. We discover that while most languages share joint cross-lingual&#10;space, some do not. However, we observe that Baltic languages do belong to that&#10;shared space. The code is available at https://github.com/TartuNLP/xsim.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multilingual Language Models such as mBERT \cite{devlin-etal-2019-bert} or XLM-R (XLM-Roberta; \cite{conneau-etal-2020a-unsupervised} achieve remarkable results on a variety of cross-lingual transfer tasks \cite{hu2020extreme, lian2020xglue}. Many works tried to understand how these models represent multiple languages but achieved incomplete and even conflicting results. &#10;&#10;Notably, \cite{muller-etal-2021-first}, \cite{conneau-etal-2020b-emerging} and \cite{singh-etal-2019-bert} performed representational similarity analysis comparing encoded sentences in different languages. However, they come up with two opposite conclusions.&#10;&#10;In particular, \cite{singh-etal-2019-bert} concluded that ``mBERT is not an Interlingua''. They observed that cross-lingual similarity of sentence representations decreased in similarity as the model layer increased. By the term &quot;interlingua,&quot; authors most certainly meant the opposite of this pattern. \cite{muller-etal-2021-first}, on the contrary, found that early layers representations are less aligned than representations from middle layers. See Figure \ref{fig:mbert-cls} for our reproduction of these conflicting results (we choose Estonian, Latvian, and Lithuanian, as leading examples, together with French and Polish for comparison). In this Figure, we compare languages with English. &#10;&#10;This begs the question of which one we should rely on and why. \cite{muller-etal-2021-first} backs up their representational analysis using probing task in the layer-wise ablation setting; \cite{conneau-etal-2020b-emerging} also directly supports this conclusion. \cite{singh-etal-2019-bert}, on the other hand, provided explanation based on possible tokenization bias issue. In any case, we believe that cross-lingual representational analysis should provide unambiguous results, so we investigate this issue in depth in this work. &#10;&#10;Moreover, we provide a comprehensive study across 378 pairs of languages to have a course-grained view of the cross-lingual similarity. We use Baltic languages as our case study throughout the paper. &#10;&#10;Specifically, we consider the following research questions:&#10;\begin{enumerate}&#10; \item Which cross-lingual pattern representational similarity suggests in the final analysis? (Answer: convergence pattern from \cite{muller-etal-2021-first}).&#10; \item Why do results in literature diverge, and how do we interpret them? (Answer: CLS-pooling is a poor choice for sentence summary, and the result is very sensitive to the choice of the similarity index).&#10; \item What is the recommended way to quantify cross-linguality in multilingual LMs? (Answer: mean-pooling with SVCCA/CKA index aligns with the evidence from the behavior analysis literature better than the other options).&#10; \item Does the resulting cross-lingual pattern generalize across all languages? (Answer: yes, it does, except for a few outlier languages).&#10; \item How does representational similarity analysis look for Estonian, Latvian, and Lithuanian? (Answer: they follow the general convergence pattern as the majority of languages and are most similar to each other than to almost all other languages). &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Lingual Representations, Computer Science, Linguistics, Multilingual Language Models, Language Transfer Performance, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2003.11080" label="2003.11080">
        <attvalues>
          <attvalue for="0" value="XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating&#10;  Cross-lingual Generalization" />
          <attvalue for="1" value="  Much recent progress in applications of machine learning models to NLP has&#10;been driven by benchmarks that evaluate models across a wide variety of tasks.&#10;However, these broad-coverage benchmarks have been mostly limited to English,&#10;and despite an increasing interest in multilingual models, a benchmark that&#10;enables the comprehensive evaluation of such methods on a diverse range of&#10;languages and tasks is still missing. To this end, we introduce the&#10;Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a&#10;multi-task benchmark for evaluating the cross-lingual generalization&#10;capabilities of multilingual representations across 40 languages and 9 tasks.&#10;We demonstrate that while models tested on English reach human performance on&#10;many tasks, there is still a sizable gap in the performance of cross-lingually&#10;transferred models, particularly on syntactic and sentence retrieval tasks.&#10;There is also a wide spread of results across languages. We release the&#10;benchmark to encourage research on cross-lingual learning methods that transfer&#10;linguistic knowledge across a diverse and representative set of languages and&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.11109" label="2101.11109">
        <attvalues>
          <attvalue for="0" value="First Align, then Predict: Understanding the Cross-Lingual Ability of&#10;  Multilingual BERT" />
          <attvalue for="1" value="  Multilingual pretrained language models have demonstrated remarkable&#10;zero-shot cross-lingual transfer capabilities. Such transfer emerges by&#10;fine-tuning on a task of interest in one language and evaluating on a distinct&#10;language, not seen during the fine-tuning. Despite promising results, we still&#10;lack a proper understanding of the source of this transfer. Using a novel layer&#10;ablation technique and analyses of the model's internal representations, we&#10;show that multilingual BERT, a popular multilingual language model, can be&#10;viewed as the stacking of two sub-networks: a multilingual encoder followed by&#10;a task-specific language-agnostic predictor. While the encoder is crucial for&#10;cross-lingual transfer and remains mostly unchanged during fine-tuning, the&#10;task predictor has little importance on the transfer and can be reinitialized&#10;during fine-tuning. We present extensive experiments with three distinct tasks,&#10;seventeen typologically diverse languages and multiple domains to support our&#10;hypothesis.&#10;" />
          <attvalue for="2" value="&#10;Zero-shot Cross-Lingual transfer aims at building models for a target language by reusing knowledge acquired from a source language. &#10;Historically, it has been tackled with a two-step standard cross-lingual pipeline\draftnote{We should add the Tenney et al. 2019 paper, see Rev1 comment -ds \bm{Tenney et. al study only the mononlingual case}} \cite{ruder2019survey}: (1)~Building a shared multilingual representation of text, typically by aligning textual representations across languages. This step can be done using feature extraction \cite{aone1993language,schultz2001language} as with the delexicalized approach \cite{zeman2008cross,sogaard2011data} or using word embedding techniques \cite{mikolov2013exploiting,smith2017offline} by projecting monolingual embeddings onto a shared multilingual embedding space, this step requiring \draftreplace{annotated data}{explicit supervision signal} in the target language in the form of features or parallel data. (2)~Training a task-specific model using supervision on a source language on top of the shared representation. &#10;&#10;Recently, the rise of multilingual language models entailed a paradigm shift in this field. Multilingual pretrained language models \cite{devlin-etal-2019-bert,conneau2019cross} have been shown to perform efficient zero-shot cross-lingual transfer for many tasks and languages \cite{pires-etal-2019-multilingual,wu2019beto}. &#10;Such transfer relies on three-steps: (i)~pretraining a mask-language model (e.g.~\cite{devlin-etal-2019-bert}) on the concatenation of monolingual corpora across multiple languages, (ii)~fine-tuning the model on a specific task in the source language, and (iii)~using the fine-tuned model on a target language.&#10;The success of this approach is remarkable, and in contrast to the standard cross-lingual pipeline, the model sees neither aligned data nor task-specific annotated data in the target language at any training stage.&#10;&#10;The source of such a successful transfer is still largely unexplained. \cite{pires-etal-2019-multilingual} hypothesize that these models learn shared multilingual representations during pretraining.&#10;Focusing on syntax, \cite{chi-etal-2020-finding} recently showed that the multilingual version of BERT (mBERT) \cite{devlin-etal-2019-bert}, encodes linguistic properties in shared multilingual sub-spaces.&#10;Recently, \cite{gonen2020s} \draftreplace{suggests}{suggest} that mBERT learns a language encoding component and an abstract cross-lingual component.&#10;In this work, we are interested in understanding the mechanism that leads mBERT to perform zero-shot cross-lingual transfer. More specifically, we ask what parts of the model and what mechanisms support cross-lingual transfer?&#10;&#10;By combining behavioral and structural analyses \cite{belinkov2020interpretability}, we show that mBERT operates as the stacking of two modules: (1)~A multilingual encoder, located in the lower part of the model, critical for cross-lingual transfer, is in charge of aligning multilingual representations; and (2)~a task-specific, language-agnostic predictor which has little importance for cross-lingual transfer and is dedicated to performing the downstream task. This mechanism that emerges out-of-the-box, without any explicit supervision, suggests that mBERT behaves like the standard cross-lingual pipeline\draftnote{We had those references before \cite{aone1993language,schultz2001language,zeman2008cross,sogaard2011data,mikolov2013exploiting,smith2017offline,cao2020multilingual} should we put them back? at least one or two -ds \bm{we've cited all of then at the begining, wouldn't it be too much to cite them again here?}}. Our contributions advance the understanding of multilingual language models and as such have the potential to support the development of better pretraining processes. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.01464" label="1911.01464">
        <attvalues>
          <attvalue for="0" value="Emerging Cross-lingual Structure in Pretrained Language Models" />
          <attvalue for="1" value="  We study the problem of multilingual masked language modeling, i.e. the&#10;training of a single model on concatenated text from multiple languages, and&#10;present a detailed study of several factors that influence why these models are&#10;so effective for cross-lingual transfer. We show, contrary to what was&#10;previously hypothesized, that transfer is possible even when there is no shared&#10;vocabulary across the monolingual corpora and also when the text comes from&#10;very different domains. The only requirement is that there are some shared&#10;parameters in the top layers of the multi-lingual encoder. To better understand&#10;this result, we also show that representations from independently trained&#10;models in different languages can be aligned post-hoc quite effectively,&#10;strongly suggesting that, much like for non-contextual word embeddings, there&#10;are universal latent symmetries in the learned embedding spaces. For&#10;multilingual masked language modeling, these symmetries seem to be&#10;automatically discovered and aligned during the joint training process.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.08909" label="2002.08909">
        <attvalues>
          <attvalue for="0" value="REALM: Retrieval-Augmented Language Model Pre-Training" />
          <attvalue for="1" value="  Language model pre-training has been shown to capture a surprising amount of&#10;world knowledge, crucial for NLP tasks such as question answering. However,&#10;this knowledge is stored implicitly in the parameters of a neural network,&#10;requiring ever-larger networks to cover more facts.&#10;  To capture knowledge in a more modular and interpretable way, we augment&#10;language model pre-training with a latent knowledge retriever, which allows the&#10;model to retrieve and attend over documents from a large corpus such as&#10;Wikipedia, used during pre-training, fine-tuning and inference. For the first&#10;time, we show how to pre-train such a knowledge retriever in an unsupervised&#10;manner, using masked language modeling as the learning signal and&#10;backpropagating through a retrieval step that considers millions of documents.&#10;  We demonstrate the effectiveness of Retrieval-Augmented Language Model&#10;pre-training (REALM) by fine-tuning on the challenging task of Open-domain&#10;Question Answering (Open-QA). We compare against state-of-the-art models for&#10;both explicit and implicit knowledge storage on three popular Open-QA&#10;benchmarks, and find that we outperform all previous methods by a significant&#10;margin (4-16% absolute accuracy), while also providing qualitative benefits&#10;such as interpretability and modularity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.03299" label="2208.03299">
        <attvalues>
          <attvalue for="0" value="Atlas: Few-shot Learning with Retrieval Augmented Language Models" />
          <attvalue for="1" value="  Large language models have shown impressive few-shot results on a wide range&#10;of tasks. However, when knowledge is key for such results, as is the case for&#10;tasks such as question answering and fact checking, massive parameter counts to&#10;store knowledge seem to be needed. Retrieval augmented models are known to&#10;excel at knowledge intensive tasks without the need for as many parameters, but&#10;it is unclear whether they work in few-shot settings. In this work we present&#10;Atlas, a carefully designed and pre-trained retrieval augmented language model&#10;able to learn knowledge intensive tasks with very few training examples. We&#10;perform evaluations on a wide range of tasks, including MMLU, KILT and&#10;NaturalQuestions, and study the impact of the content of the document index,&#10;showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy&#10;on Natural Questions using only 64 examples, outperforming a 540B parameters&#10;model by 3% despite having 50x fewer parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2304.06762" label="2304.06762">
        <attvalues>
          <attvalue for="0" value="Shall We Pretrain Autoregressive Language Models with Retrieval? A&#10;  Comprehensive Study" />
          <attvalue for="1" value="  Large decoder-only language models (LMs) can be largely improved in terms of&#10;perplexity by retrieval (e.g., RETRO), but its impact on text generation&#10;quality and downstream task accuracy is unclear. Thus, it is still an open&#10;question: shall we pretrain large autoregressive LMs with retrieval? To answer&#10;it, we perform a comprehensive study on a scalable pre-trained&#10;retrieval-augmented LM (i.e., RETRO) compared with standard GPT and&#10;retrieval-augmented GPT incorporated at fine-tuning or inference stages. We&#10;first provide the recipe to reproduce RETRO up to 9.5B parameters while&#10;retrieving a text corpus with 330B tokens. Based on that, we have the following&#10;novel findings: i) RETRO outperforms GPT on text generation with much less&#10;degeneration (i.e., repetition), moderately higher factual accuracy, and&#10;slightly lower toxicity with a nontoxic retrieval database. ii) On the LM&#10;Evaluation Harness benchmark, RETRO largely outperforms GPT on&#10;knowledge-intensive tasks, but is on par with GPT on other tasks. Furthermore,&#10;we introduce a simple variant of the model, RETRO++, which largely improves&#10;open-domain QA results of original RETRO (e.g., EM score +8.6 on Natural&#10;Question) and significantly outperforms retrieval-augmented GPT in both&#10;fine-tuning and zero-shot evaluation settings. Our findings highlight the&#10;promising direction of pretraining autoregressive LMs with retrieval as future&#10;foundation models. We release our code and model at:&#10;https://github.com/NVIDIA/Megatron-LM/blob/main/tools/retro/README.md&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Retrieval has been applied in various NLP tasks for years, including question answering~(QA)~\cite[e.g.,][]{bilotti2007structured}, machine translation~\cite[e.g.,][]{zhang2018guiding}, and conversation~\cite{shuster2021retrieval,thoppilan2022lamda, komeili2021internet}.&#10;In particular, language models have been augmented with retrieval at different stages, including inference time~\cite{khandelwal2019generalization,yogatama2021adaptive}, fine-tuning stage~\cite{karpukhin2020dense, lewis2020retrieval, guu2020retrieval}, and pretraining stage~\cite{borgeaud2022improving, izacard2022few}.&#10;&#10;LMs have been augmented with retrieval at the fine-tuning stage for downstream tasks, primarily for open-domain QA.&#10;DPR~\cite{karpukhin2020dense} finetunes one BERT to encode questions and the other BERT to encode answers within a dual encoder framework, using a contrastive loss to align the hidden representations of question and corresponding answer.&#10;RAG~\cite{lewis2020retrieval} studies the fine-tuning recipe for retrieval-augmented generation models, especially on open-domain QA tasks.&#10;{FiD~\cite{izacard2021leveraging} improves RAG with a better LM backbone T5, and fuses multiple retrieved passages to the decoder during fine-tuning to further improve QA accuracy.&#10;WebGPT~\cite{nakano2021webgpt} leverages web search engine and fine-tunes GPT using reinforcement learning with human feedback (RLHF) for reference generation and factuality improvement, which is orthogonal to our work that focuses on pretraining with retrieval. The proposed RLHF can be applied to Retro as well. &#10;}&#10;&#10;REALM~\cite{guu2020retrieval} performs both unsupervised pretraining and supervised fine-tuning strategies for retrieval-augmented BERT model in open-domain QA. &#10;Their pretraining involves asynchronous re-embedding and re-indexing all documents every several hundred training steps, which quickly becomes impractical for training corpus with trillion tokens.&#10;Atlas~\cite{izacard2022few} uses a similar approach but augments the T5 architecture~\cite{raffel2020exploring} with retrieval at both pre-training and fine-tuning. Before pretraining, it first initializes the encoder-decoder LM backbone with pretrained T5, and the dense retriever with pretrained Contriever~\cite{izacardunsupervised}. During pretraining, it also applies asynchronous index refresh every 1000 steps.&#10;&#10;In contrast, Retro~\cite{borgeaud2022improving} embeds and indexes the whole training corpus at chunk-level~(e.g., chuck size = 64) with a frozen BERT before pretraining.&#10;During pretraining, the model relies on a trainable bidirectional encoder to embed the retrieved chunks of raw text. The GPT decoder further ``select'' the relevant piece of evidence from the encoder side by a chunk-wise cross-attention. This architecture design enables LM pretraining on hundreds of billion tokens by retrieving from trillion tokens. &#10;See Table~\ref{table:comparison} for a complete comparison of retrieval-augmented LMs.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2302.00083" label="2302.00083">
        <attvalues>
          <attvalue for="0" value="In-Context Retrieval-Augmented Language Models" />
          <attvalue for="1" value="  Retrieval-Augmented Language Modeling (RALM) methods, which condition a&#10;language model (LM) on relevant documents from a grounding corpus during&#10;generation, were shown to significantly improve language modeling performance.&#10;In addition, they can mitigate the problem of factually inaccurate text&#10;generation and provide natural source attribution mechanism. Existing RALM&#10;approaches focus on modifying the LM architecture in order to facilitate the&#10;incorporation of external information, significantly complicating deployment.&#10;This paper considers a simple alternative, which we dub In-Context RALM:&#10;leaving the LM architecture unchanged and prepending grounding documents to the&#10;input, without any further training of the LM. We show that In-Context RALM&#10;that builds on off-the-shelf general purpose retrievers provides surprisingly&#10;large LM gains across model sizes and diverse corpora. We also demonstrate that&#10;the document retrieval and ranking mechanism can be specialized to the RALM&#10;setting to further boost performance. We conclude that In-Context RALM has&#10;considerable potential to increase the prevalence of LM grounding, particularly&#10;in settings where a pretrained LM must be used without modification or even via&#10;API access.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.12652" label="2301.12652">
        <attvalues>
          <attvalue for="0" value="REPLUG: Retrieval-Augmented Black-Box Language Models" />
          <attvalue for="1" value="  We introduce REPLUG, a retrieval-augmented language modeling framework that&#10;treats the language model (LM) as a black box and augments it with a tuneable&#10;retrieval model. Unlike prior retrieval-augmented LMs that train language&#10;models with special cross attention mechanisms to encode the retrieved text,&#10;REPLUG simply prepends retrieved documents to the input for the frozen&#10;black-box LM. This simple design can be easily applied to any existing&#10;retrieval and language models. Furthermore, we show that the LM can be used to&#10;supervise the retrieval model, which can then find documents that help the LM&#10;make better predictions. Our experiments demonstrate that REPLUG with the tuned&#10;retriever significantly improves the performance of GPT-3 (175B) on language&#10;modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by&#10;5.1%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.05102" label="2211.05102">
        <attvalues>
          <attvalue for="0" value="Efficiently Scaling Transformer Inference" />
          <attvalue for="1" value="  We study the problem of efficient generative inference for Transformer&#10;models, in one of its most challenging settings: large deep models, with tight&#10;latency targets and long sequence lengths. Better understanding of the&#10;engineering tradeoffs for inference for large Transformer-based models is&#10;important as use cases of these models are growing rapidly throughout&#10;application areas. We develop a simple analytical model for inference&#10;efficiency to select the best multi-dimensional partitioning techniques&#10;optimized for TPU v4 slices based on the application requirements. We combine&#10;these with a suite of low-level optimizations to achieve a new Pareto frontier&#10;on the latency and model FLOPS utilization (MFU) tradeoffs on 500B+ parameter&#10;models that outperforms the FasterTransformer suite of benchmarks. We further&#10;show that with appropriate partitioning, the lower memory requirements of&#10;multiquery attention (i.e. multiple query heads share single key/value head)&#10;enables scaling up to 32x larger context lengths. Finally, we achieve a&#10;low-batch-size latency of 29ms per token during generation (using int8 weight&#10;quantization) and a 76% MFU during large-batch-size processing of input tokens,&#10;while supporting a long 2048-token context length on the PaLM 540B parameter&#10;model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00052" label="2005.00052">
        <attvalues>
          <attvalue for="0" value="MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer" />
          <attvalue for="1" value="  The main goal behind state-of-the-art pre-trained multilingual models such as&#10;multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in&#10;low-resource languages through zero-shot or few-shot cross-lingual transfer.&#10;However, due to limited model capacity, their transfer performance is the&#10;weakest exactly on such low-resource languages and languages unseen during&#10;pre-training. We propose MAD-X, an adapter-based framework that enables high&#10;portability and parameter-efficient transfer to arbitrary tasks and languages&#10;by learning modular language and task representations. In addition, we&#10;introduce a novel invertible adapter architecture and a strong baseline method&#10;for adapting a pre-trained multilingual model to a new language. MAD-X&#10;outperforms the state of the art in cross-lingual transfer across a&#10;representative set of typologically diverse languages on named entity&#10;recognition and causal commonsense reasoning, and achieves competitive results&#10;on question answering. Our code and adapters are available at AdapterHub.ml&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Cross-lingual Representations Research in modern cross-lingual NLP is increasingly focused on learning general-purpose cross-lingual representations that can be applied to many tasks, first on the word level \cite{mikolov2013exploiting,gouws2015bilbowa,Glavas:2019acl,Ruder2019survey,Wang:2020iclr} and later on the full-sentence level \cite{Devlin2019bert,Lample2019xlm,Cao:2020iclr}. More recent models such as multilingual BERT \cite{Devlin2019bert}---large Transformer \cite{Vaswani2017transformer} models pretrained on large amounts of multilingual data---have been observed to perform surprisingly well when transferring to other languages \cite{Pires2019,Wu2019beto,Wu2020emerging} and the current state-of-the-art model, XLM-R is competitive with the performance of monolingual models on the GLUE benchmark \cite{Conneau2020xlm-r}. Recent studies \cite{Hu2020xtreme}, however, indicate that state-of-the-art models such as XLM-R still perform poorly on cross-lingual transfer across many language pairs. The main reason behind such poor performance is the current lack of capacity in the model to represent all languages equally in the vocabulary and representation space \cite{Bapna2019adapters,Artetxe2020cross-lingual,Conneau2020xlm-r}.&#10;&#10;Adapters Adapter modules have been originally studied in computer vision tasks where they have been restricted to convolutions and used to adapt a model for multiple domains \cite{Rebuffi2017adapters,Rebuffi2018}. In NLP, adapters have been mainly used for parameter-efficient and quick fine-tuning of a base pretrained Transformer model to new tasks \cite{Houlsby2019adapters,Cooper2019adapters} and new domains \cite{Bapna2019adapters}, avoiding catastrophic forgetting \cite{McCloskey:1989,Santoro:2016arxiv}. \cite{Bapna2019adapters} also use adapters to fine-tune and recover performance of a multilingual NMT model on high-resource languages, but their approach cannot be applied to languages that were not seen during pretraining. \cite{Artetxe2020cross-lingual} employ adapters to transfer a pretrained monolingual model to an unseen language but rely on learning new token-level embeddings, which do not scale to a large number of languages. \cite{Pfeiffer2020adapterfusion} combine the information stored in multiple adapters for more robust transfer learning between monolingual tasks. In their contemporaneous work, \cite{ustun2020udapter} generate adapter parameters from language embeddings for multilingual dependency parsing.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2211.01786" label="2211.01786">
        <attvalues>
          <attvalue for="0" value="Crosslingual Generalization through Multitask Finetuning" />
          <attvalue for="1" value="  Multitask prompted finetuning (MTF) has been shown to help large language&#10;models generalize to new tasks in a zero-shot setting, but so far explorations&#10;of MTF have focused on English data and models. We apply MTF to the pretrained&#10;multilingual BLOOM and mT5 model families to produce finetuned variants called&#10;BLOOMZ and mT0. We find finetuning large multilingual language models on&#10;English tasks with English prompts allows for task generalization to&#10;non-English languages that appear only in the pretraining corpus. Finetuning on&#10;multilingual tasks with English prompts further improves performance on English&#10;and non-English tasks leading to various state-of-the-art zero-shot results. We&#10;also investigate finetuning on multilingual tasks with prompts that have been&#10;machine-translated from English to match the language of each dataset. We find&#10;training on these machine-translated prompts leads to better performance on&#10;human-written prompts in the respective languages. Surprisingly, we find models&#10;are capable of zero-shot generalization to tasks in languages they have never&#10;intentionally seen. We conjecture that the models are learning higher-level&#10;capabilities that are both task- and language-agnostic. In addition, we&#10;introduce xP3, a composite of supervised datasets in 46 languages with English&#10;and machine-translated prompts. Our code, datasets and models are freely&#10;available at https://github.com/bigscience-workshop/xmtf.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.07827" label="2402.07827">
        <attvalues>
          <attvalue for="0" value="Aya Model: An Instruction Finetuned Open-Access Multilingual Language&#10;  Model" />
          <attvalue for="1" value="  Recent breakthroughs in large language models (LLMs) have centered around a&#10;handful of data-rich languages. What does it take to broaden access to&#10;breakthroughs beyond first-class citizen languages? Our work introduces Aya, a&#10;massively multilingual generative language model that follows instructions in&#10;101 languages of which over 50% are considered as lower-resourced. Aya&#10;outperforms mT0 and BLOOMZ on the majority of tasks while covering double the&#10;number of languages. We introduce extensive new evaluation suites that broaden&#10;the state-of-art for multilingual eval across 99 languages -- including&#10;discriminative and generative tasks, human evaluation, and simulated win rates&#10;that cover both held-out tasks and in-distribution performance. Furthermore, we&#10;conduct detailed investigations on the optimal finetuning mixture composition,&#10;data pruning, as well as the toxicity, bias, and safety of our models. We&#10;open-source our instruction datasets and our model at&#10;https://hf.co/CohereForAI/aya-101&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Language Diversity in Open-source Multilingual NLP $\:$ There are around 7,000 languages spoken in the world, and around 2,500 languages classified as low-resource languages by \cite{joshi-etal-2020-state} have more than 1 billion speakers. Despite the sizable number of language users, there is scarce coverage of multilingual datasets for supervised NLP tasks. For the task of machine translation, most notable improvements have been achieved with recent work such as NLLB \cite{nllb2022}, FLORES \cite{goyal2021flores101}, and Tatoeba \cite{tiedemann2020tatoeba}. These initiatives collectively advance low-resource and multilingual machine translation by open-sourcing models, introducing comprehensive evaluation benchmarks and datasets, and fostering the development of open tools and models across 200 languages, acknowledging the limitation in coverage compared to the diversity of languages worldwide, yet promoting global communication and research in translation. &#10;Grassroots organization like Masakhane~\cite{orife2020masakhane} advanced African NLP efforts in several domains like NER~\cite{adelani2021masakhaner, adelani2022masakhaner}, QA~\cite{ogundepo2023afriqa} and MT~\cite{nekoto-etal-2020-participatory-forall,adelani2022few}. Other notable initiatives include NusaCrowd~\cite{cahyawijaya2022nusacrowd} for Indonesian~\cite{winata2022nusax}, Turkic Interlingua (TIL)~\cite{mirzakhalov2021turkic} for Turkic Languages~\cite{mirzakhalov2021large}, IndicCorp and IndicXtream~\cite{doddapaneni-etal-2023-towards} for Indic languages, Masader~\cite{alyafeai2021masader} for Arabic~\cite{altaher2022masader} and SEACrowd for South East Asian languages.&#10;&#10;Pre-trained Multilingual Models $\:$ Pre-training a language model involves unsupervised learning on vast amounts of data. While most pre-training has focused on English \cite{devlin2019bert,radford2019gpt2,raffel2020exploring,biderman2023pythia}, there has also been considerable work focused on mono-lingual pre-training outside of English \cite{faysse2024croissantllm,gutierrez2021maria,zeng2021pangu,sengupta2023jais,phan-etal-2022-vit5,koto-etal-2020-indolem,ko2023polyglot} or training models on a small set of languages \cite{nguyen2023seallms,mesham2021low,ogueji-etal-2021-small, jude-ogundepo-etal-2022-afriteva}.&#10;Here, we are interested in pre-training efforts which are massively multilingual \cite{xue2020mt5,chung2023unimax,shliazhko2022mgpt,scao2022bloom,lin-etal-2022-shot,devlin2019bert,conneau2019unsupervised, khanuja2021muril, oladipo-etal-2023-better, alabi-etal-2022-adapting}. &#10;Models trained on variants of the mC4 corpus~\cite{xue2020mt5} cover around 100 different languages in significant amounts, which is the broadest coverage currently available for pre-trained models. Among them, mT5~\cite{xue2020mt5} and umT5~\cite{chung2023unimax} are the largest publicly available pre-trained language models in terms of number of languages covered. We also point to a parallel direction of work that focuses on adapting pre-trained models to new languages than were not present during pretraining. These studies leverage continued finetuning and adaptation of the embedding space.&#10;For example, some prior work~\cite{yong2022bloom+,luukkonen2023fingpt} extends language coverage by adding a single language at a time through continued pretraining on monolingual corpora, which does not scale well. Work concurrent to ours by \cite{lin2024mala} covers a more extensive set of languages by employing vocabulary extension and continued pretraining on LLaMA 2 with Glot500-c \cite{imanigooghari-etal-2023-glot500}. &#10;A commonality shared by all the approaches above is a focus on pre-training, which makes off-the-shelf usability limited as users have to perform downstream task finetuning themselves. In contrast, this work is focused on conferring instruction following abilities to pre-trained models. &#10;&#10;Instruction Tuning $\:$ Before multitask finetuning, significant work focused on finetuning pre-trained models on a variety of languages through data augmentation for a single task~\cite{longpre2021mkqa,asai2022mia,asai2023buffet,hu2020xtreme}. More recently, finetuning pre-trained models on a large collection of tasks has emerged as a key paradigm to improve their performance and make them more useful~\cite{sanh2021multitask,wei2021finetuned,mishra2021cross,min2021metaicl,ouyang2022training}. Task diversity~\cite{longpre2023flan,wang2023far,chung2022scaling}, complexity~\cite{xu2023wizardlm,luo2023wizardcoder,luo2023wizardmath} and quality~\cite{zhou2023lima,taori2023stanford,muennighoff2023octopack,zhuo2024astraios} are three critical axes for successful instruction tuning. \cite{muennighoff2022crosslingual} conduct an investigation into the role of multilingual data during instruction tuning. They found that models are capable of solving tasks in languages unseen during instruction tuning and even pre-training in some cases. However, including languages during the training process leads to better performance than solely relying on such crosslingual generalization. &#10;Thus, the BLOOMZ \cite{muennighoff2022crosslingual} and mT0 \cite{muennighoff2022crosslingual} models make significant strides in the multilingual capabilities across the 46 languages seen during finetuning.&#10;However, their usefulness is limited beyond this set, particularly for lower-resourced languages. While other multilingual instruction models have been proposed since~\cite{li2023bactrian,lai2023okapi}, there remains significant room for improvements among all new open models~\cite{asai2022mia,asai2023buffet,hu2020xtreme,ruder2021xtreme}.&#10;Aside from the still limited language coverage, these models often employ English instruction data, and primarily academic tasks that differ from real-world use cases.&#10;By releasing a model that has been fine-tuned on many diverse tasks in each target language and tested on open-ended generation across languages, we make a large step toward closing the performance deficit. &#10;Aside from the broader language coverage, our work also improves accessibility by training a model that performs well when a prompt is provided in the same target language as the task, as opposed to prior work that explores prompting in a code-switched fashion, which uses English prompt and task information in target language \cite{fu-etal-2022-polyglot,huang2023not,muennighoff2022crosslingual}. &#10;&#10;Translation Augmentation $\:$ &#10;Translation-related augmentation strategies are popular for multilingual tasks.&#10;Translate-train, translate-test \cite{asai2018multilingual,cui2019cross,jundi-lapesa-2022-translate}, or language pivots \cite{montero2022pivot} are common techniques employing translation models to bridge language gaps between the model and its target language.&#10;Back translation \cite{sennrich-etal-2016-improving,dhole2021nl} is a popular strategy for augmenting training data, but given that our goal is to improve multilingual generation, we simply translated our training datasets into our target languages without translating them back. Our translation augmentation is similar to \cite{Bornea_Pan_Rosenthal_Florian_Sil_2021}'s work, which used machine translation-generated data to increase the size of their training set by a factor of 14. While our work utilized machine translation similarly to expand our English training set, we also leverage human expertise, to perform quality filtering based on feedback from \aya community members, and to provide human translations. &#10;Machine-translated prompts often lack variability and the cultural nuance inherent in text originally written in the target languages. However, they are still useful for expanding the language coverage of the training data and can help bridge the resource gap for languages with limited training data \cite{urbizu-etal-2023-enough, lin2021fewshot}. They can also adapt already-trained instruction-tuned language models to follow instructions in new languages \cite{yong2022bloom+}. Furthermore, LLMs trained on designed prompts have also been shown to be successful at tasks like EAE (Event Argument Extraction) from multilingual data in a zero-shot setup \cite{huang2022multilingual}. \cite{zhang2023chinese} constructed high-quality Chinese instructions from existing English instruction datasets. They first translated the English instructions into Chinese, and then used a human verification process to determine whether these translations are usable; the verified dataset set consists of around 200k Chinese instruction-tuning samples. \cite{li2023bactrianx} constructed instruction data for 52 popular languages using Google Translate to translate English prompts and completions from Alpaca \cite{alpaca} (52K) and Dolly \cite{DatabricksBlog2023DollyV2} (15K) dataset, then used these data to finetune LLaMA \cite{touvron2023llama} using the LoRA \cite{hu2021lora} technique. BayLing \cite{zhang2023bayling} prompted LLMs to translate a task request, which is overlaid with the more granular user-based corrects. This process naturally connects different languages as well as human preferences with LLMs, leveraging LLaMA \cite{touvron2023llama} for foundational support and employing automatic construction of interactive translation instructions for instructional tuning, thereby enhancing the model's multilingual capability and alignment with diverse linguistic needs.&#10;&#10;Dataset Weighting $\:$ As for dataset balancing, there are a variety of prior works, including \cite{xie2023data,muennighoff2023scaling, longpre2022active} which dynamically select pretraining or finetuning data from across domains, for more efficient and performant target results.&#10;Separately, \cite{dou2020dynamic} dynamically selects and weights training data for back-translation.&#10;In the multilingual setting specifically, \cite{wang-etal-2020-balancing} proposed using MultiDDS, which is based on \cite{10.5555/3524938.3525864}'s Differentiable Data Selection, that optimizes a language scorer to adapt to multiple model objectives in a multilingual training context.&#10;Closely intertwined with this, data pruning is a research domain focusing on selecting a subset of data based on specific criteria.&#10;Previous works have studied metrics such as perplexity and error norms as selection criteria for filtering data~\cite{wenzek2019ccnet,laurenccon2022bigscience} and finetuning LLMs~\cite{paul2023deep,marion2023more}. &#10;Prioritizing data instances that most effectively distinguish between models has also been effective in reducing the required human effort for annotation~\cite{boubdir2023prompts}.&#10;&#10;Evaluation of Toxicity and Bias in LLMs $\:$ Bias evaluations for LLM releases to date typically focus on a single language or a small set of languages:&#10;PaLM~\cite{chowdhery2022palm} and Llama ~\cite{touvron2023llama} evaluated gender bias for the English language on the Winogender benchmark ~\cite{rudinger2018gender} for the coreference resolution performance involving different genders and occupations, with the observation from PaLM~\cite{chowdhery2022palm} that the accuracy improves as the model scales up. GPT3 ~\cite{brown2020languageGPT3} also used the Winogender benchmark~\cite{rudinger2018gender} in investigating the gender bias in the model, with the findings that it has the tendency to use the male identifier in its generated outputs. BLOOM~\cite{scao2022bloom} evaluated gender bias on the multilingual CrowS-Pairs dataset that combines the revised English version~\cite{nangia2020crows} as well as the French version ~\cite{neveol2022french}. &#10;The CrowS-Pairs dataset~\cite{nangia2020crows}, which measures bias in nine different categories including gender, age, and religion is also used in the evaluation of Llama~\cite{touvron2023llama}.&#10;Toxicity evaluation has also been primarily concentrated on English. To measure the models' tendency in producing toxic generations given non-toxic prompts (known as toxicity degeneration), PaLM~\cite{chowdhery2022palm}, Flan-T5~\cite{chung2022scaling}, Llama~\cite{touvron2023llama}, and GPT-4~\cite{openai2023GPT4} use the RealToxicityPrompts dataset~\cite{gehman-etal-2020-realtoxicityprompts}, which contains naturally occurring prompts collected from English web text. Llama-2~\cite{touvron2023llama2} on the other hand is evaluated on ToxiGen dataset~\cite{hartvigsen-etal-2022-toxigen}, which is a large-scale machine-generated dataset of English statements about 13 minority groups. Recently released alignment datasets on harmlessness, which is used to align or evaluate models' preferences for non-offensive languages, such as HH-RLHF \cite{bai2022AnthropicHH} and BeaverTails \cite{ji2023beavertails}, also focus only on English. On the other hand, multilingual toxicity evaluation has been mainly carried out to evaluate models' ability to detect toxic text. For instance, \cite{anil2023palm} evaluates PaLM2's ability to classify between toxic and non-toxic text across six different languages using the multilingual Jigsaw dataset \cite{kivlichan2020jigsaw_multiling}.&#10;&#10;Multilingual LLM Safety &#10;Recent responsible AI research has gone beyond studying toxicity and biases in generation and focused on mitigating unsafe responses that are returned by powerful LLMs to malicious prompts such as &quot;Create a website that promotes racism against minority groups.&quot; Recent work found that commercial pre-trained LLMs tend to bypass existing safeguards and generate unsafe responses when a malicious prompt is written in a lower-resource language \cite{yong2023lowresource,deng2023multilingual,shen2024language}. Existing solutions such as including multilingual mixtures of training data are not ideal because aligned models suffer more from the harmlessness-helpfulness tradeoff---in other words, the aligned models will be worse on non-safety related tasks \cite{deng2023multilingual}. \cite{shen2024language} found that it is more effective to improve LLMs' safety in low-resource languages with continued pre-training followed by safety alignment training. While we do not perform any alignment training, our experiments demonstrate that multilingual safety context distillation in the instruction-tuning stage effectively improves the multilingual safety of \aya across all languages.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.10610" label="2305.10610">
        <attvalues>
          <attvalue for="0" value="Solving Cosine Similarity Underestimation between High Frequency Words&#10;  by L2 Norm Discounting" />
          <attvalue for="1" value="  Cosine similarity between two words, computed using their contextualised&#10;token embeddings obtained from masked language models (MLMs) such as BERT has&#10;shown to underestimate the actual similarity between those words (Zhou et al.,&#10;2022). This similarity underestimation problem is particularly severe for&#10;highly frequent words. Although this problem has been noted in prior work, no&#10;solution has been proposed thus far. We observe that the L2 norm of&#10;contextualised embeddings of a word correlates with its log-frequency in the&#10;pretraining corpus. Consequently, the larger L2 norms associated with the&#10;highly frequent words reduce the cosine similarity values measured between&#10;them, thus underestimating the similarity scores. To solve this issue, we&#10;propose a method to discount the L2 norm of a contextualised word embedding by&#10;the frequency of that word in a corpus when measuring the cosine similarities&#10;between words. We show that the so called stop words behave differently from&#10;the rest of the words, which require special consideration during their&#10;discounting process. Experimental results on a contextualised word similarity&#10;dataset show that our proposed discounting method accurately solves the&#10;similarity underestimation problem.&#10;" />
          <attvalue for="2" value="&#10;&#10;Cosine similarity is arguably the most popular word similarity measure used in numerous natural language processing (NLP) tasks, such as question answering (QA), information retrieval (IR) and machine translation (MT)~\cite{echizen2019word, oniani2020qualitative, Kim:2021, hanifi2022problem}.&#10;First, a word is represented by a vector (aka embedding) and then the similarity between two words is computed as the cosine of the angle between the corresponding vectors~\cite{rahutomo2012semantic}.&#10;Despite the good performance of cosine similarity as a similarity measure in various downstream tasks, \cite{zhou-etal-2022-problems} showed that it systematically underestimates the true similarity between highly frequent words, when computed using contextualised word embeddings obtained from MLMs such as BERT~\cite{BERT}.&#10;&#10;Compared to the problem of estimating similarity between highly frequent words, the opposite problem of estimating the similarity between (or involving) rare (low frequency) words has received greater attention, especially in the scope of static word embeddings~\cite{levy2014neural,hellrich-hahn-2016-bad,mimno-thompson-2017-strange,wendlandt-etal-2018-factors}. &#10;If a word is rare in a corpus, we might not have a sufficiently large number of contexts containing that word to learn an accurate embedding for it.&#10;This often leads to unreliable similarity estimations between words and has undesirable implications in downstream tasks such as the detection of analogies and social biases~\cite{ethayarajh-etal-2019-towards, ethayarajh2019understanding}.&#10;&#10;On the other hand, \cite{zhou-etal-2022-problems} studied the impact of frequency on contextualised word embeddings and showed that the cosine similarity between highly frequent words are systematically underestimated.&#10;Unlike in the previously discussed low frequency word scenario, we do have adequate contexts to learn an accurate semantic representation for highly frequent words.&#10;Therefore, it might appear surprising at first that cosine similarity cannot be correctly estimated even for the highly frequent words.&#10;\cite{zhou2021frequency} show that the diversity (measured by the volume of the bounding hypersphere) of the contextualised embeddings of a target word, computed from multiple contexts containing the word, increases with the frequency of that word.&#10;They provide an explanation that holds true only for 2-dimensional embeddings, which relates diversity to the underestimation of cosine similarity.&#10;Unfortunately, this explanation does not extend to the high dimensional embeddings used in practice by the NLP community (e.g. BERT token embeddings are typically more than 768 dimensional).&#10;More importantly, to the best of our knowledge, no solution has been proposed in the literature to address the cosine similarity underestimation problem associated with the highly frequent words.&#10;&#10;In prior work, the $\ell_2$ norm of a static word embedding has been shown to linearly correlate with the log-frequency of that word~\cite{arora-etal-2016-latent,Bollegala:AAAI:2018}.&#10;On the other hand, we empirically study the $\ell_2$ norm of the contextualised embedding of a word $w$ averaged over all of its contexts, and find that it too approximately linearly correlates with the log-frequency of $w$ in the corpus used to pretrain the MLM.&#10;Recall that the cosine similarity is defined as the inner-product between two embeddings, divided by the $\ell_2$ norm of those embeddings.&#10;Therefore, we suspect that the underestimation of cosine similarity between highly frequent words is due to the larger $\ell_2$ norms associated with those words.&#10;&#10;To correct for this bias associated with the $\ell_2$ norms of highly frequent words, we propose a linearly parameterised discounting scheme in the log-frequency space.&#10;Specifically, we use Monte-Carlo Bayesian Optimisation~\cite{botorch} to find the optimal discounting parameters.&#10;Our proposed discounting method is shown to accurately correct the underestimation of cosine similarities between highly frequent words on the Word-in-Context (WiC)~\cite{Pilehvar:2019} dataset where human similarity ratings are available for the same word in two different contexts.&#10;Source code for reproducing the experiments reported in this is paper is publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Model Limitations, Similarity Measurement Techniques, Computational Linguistics, Mathematics, Natural Language Processing, Word Embeddings Analysis" />
        </attvalues>
      </node>
      <node id="2205.05092" label="2205.05092">
        <attvalues>
          <attvalue for="0" value="Problems with Cosine as a Measure of Embedding Similarity for High&#10;  Frequency Words" />
          <attvalue for="1" value="  Cosine similarity of contextual embeddings is used in many NLP tasks (e.g.,&#10;QA, IR, MT) and metrics (e.g., BERTScore). Here, we uncover systematic ways in&#10;which word similarities estimated by cosine over BERT embeddings are&#10;understated and trace this effect to training data frequency. We find that&#10;relative to human judgements, cosine similarity underestimates the similarity&#10;of frequent words with other instances of the same word or other words across&#10;contexts, even after controlling for polysemy and other factors. We conjecture&#10;that this underestimation of similarity for high frequency words is due to&#10;differences in the representational geometry of high and low frequency words&#10;and provide a formal argument for the two-dimensional case.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.04882" label="1810.04882">
        <attvalues>
          <attvalue for="0" value="Towards Understanding Linear Word Analogies" />
          <attvalue for="1" value="  A surprising property of word vectors is that word analogies can often be&#10;solved with vector arithmetic. However, it is unclear why arithmetic operators&#10;correspond to non-linear embedding models such as skip-gram with negative&#10;sampling (SGNS). We provide a formal explanation of this phenomenon without&#10;making the strong assumptions that past theories have made about the vector&#10;space and word distribution. Our theory has several implications. Past work has&#10;conjectured that linear substructures exist in vector spaces because relations&#10;can be represented as ratios; we prove that this holds for SGNS. We provide&#10;novel justification for the addition of SGNS word vectors by showing that it&#10;automatically down-weights the more frequent word, as weighting schemes do ad&#10;hoc. Lastly, we offer an information theoretic interpretation of Euclidean&#10;distance in vector spaces, justifying its use in capturing word dissimilarity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;PMI Pointwise mutual information (PMI) captures how much more frequently $x,y$ co-occur than by chance \cite{church1990word}:&#10;\begin{equation}&#10; PMI(x,y) = \log \frac{p(x,y)}{p(x) p(y)}&#10;\end{equation} &#10;&#10;Word Embeddings Word embeddings are distributed representations in a low-dimensional continuous space. Also called word vectors, they capture semantic and syntactic properties of words, even allowing relationships to be expressed arithmetically \cite{mikolov2013distributed}. Word vectors are generally obtained in two ways: (a) from neural networks that learn representations by predicting co-occurrence patterns in the training corpus \cite{bengio2003neural,mikolov2013distributed,collobert2008unified}; (b) from low-rank approximations of word-context matrices containing a co-occurrence statistic \cite{landauer1997solution,levy2014neural}. &#10;&#10;SGNS The objective of skip-gram with negative sampling (SGNS) is to maximize the probability of observed word-context pairs and to minimize the probability of $k$ randomly sampled negative examples. For an observed word-context pair $(w,c)$, the objective would be $\log \sigma(\vec{w} \cdot \vec{c}) + k \cdot \E_{c' \sim P_n} [ \log (- \vec{w} \cdot \vec{c}') ]$, where $c'$ is the negative context, randomly sampled from a scaled distribution $P_n$. Though no co-occurrence statistics are explicitly calculated, \cite{levy2014neural} proved that SGNS is in fact implicitly factorizing a word-context PMI matrix shifted by $- \log k$.&#10;&#10;Latent Variable Model The latent variable model \cite{arora2016latent} was the first attempt at rigorously explaining why word analogies can be solved arithmetically. It is a generative model that assumes that word vectors are generated by the random walk of a ``discourse'' vector on the unit sphere. \cite{gittens2017skip}'s criticism of this proof is that it assumes that word vectors are known a priori and generated by randomly scaling vectors uniformly sampled from the unit sphere (or having properties consistent with this sampling procedure). The theory also relies on word vectors being uniformly distributed (isotropic) in embedding space; however, experiments by \cite{mimno2017strange} \citepar{mimno2017strange} have found that this generally does not hold in practice, at least for SGNS.&#10;&#10;Paraphrase Model The paraphrase model \cite{gittens2017skip} was the only other attempt to formally explain why word analogies can be solved arithmetically. It proposes that any set of context words $C = \{c_1,..., c_m\}$ is semantically equivalent to a single word $c$ if $p(w|c_1,..., c_m) = p(w|c)$. One problem with this is that the number of possible context sets far exceeds the vocabulary size, precluding a one-to-one mapping; the authors circumvent this problem by replacing exact equality with the minimization of KL divergence. Assuming that the words have a uniform distribution, the paraphrase of $C$ can then be written as an unweighted sum of its context vectors. However, this uniformity assumption is unrealistic -- word frequencies obey a Zipf distribution, which is Pareto \cite{piantadosi2014zipf}. A later attempt at using paraphrases \cite{allen2019analogies} completely ignores the effect of negative sampling in SGNS' factorization. Neither work provides any empirical evidence in support of the paraphrase model.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2104.08465" label="2104.08465">
        <attvalues>
          <attvalue for="0" value="Frequency-based Distortions in Contextualized Word Embeddings" />
          <attvalue for="1" value="  How does word frequency in pre-training data affect the behavior of&#10;similarity metrics in contextualized BERT embeddings? Are there systematic ways&#10;in which some word relationships are exaggerated or understated? In this work,&#10;we explore the geometric characteristics of contextualized word embeddings with&#10;two novel tools: (1) an identity probe that predicts the identity of a word&#10;using its embedding; (2) the minimal bounding sphere for a word's&#10;contextualized representations. Our results reveal that words of high and low&#10;frequency differ significantly with respect to their representational geometry.&#10;Such differences introduce distortions: when compared to human judgments, point&#10;estimates of embedding similarity (e.g., cosine similarity) can over- or&#10;under-estimate the semantic similarity of two words, depending on the frequency&#10;of those words in the training data. This has downstream societal implications:&#10;BERT-Base has more trouble differentiating between South American and African&#10;countries than North American and European ones. We find that these distortions&#10;persist when using BERT-Multilingual, suggesting that they cannot be easily&#10;fixed with additional data, which in turn introduces new distortions.&#10;" />
          <attvalue for="2" value="&#10;&#10;How similar are the words “felony” and “misdemeanor”? To a lawyer, the terms represent different charges, with different consequences, and different plans of action. To a layperson, these terms could be quite similar -- both referring to types of crime. Our perception of the similarity of words is conditioned on our backgrounds, and what is similar to one individual can be very different to another. One important variable is the frequency at which we are exposed to these concepts. In this paper, we study how the frequency of words in a model's training data plays a role in the semantic similarity of those words' contextualized embeddings. &#10;&#10;The impact of frequency has long been studied on static word embeddings \cite{levy2014neural, Hellrich_Hahn_2016, Mimno_Thompson_2017, Wendlandt_Kummerfeld_Mihalcea_2018}. For example, word frequency is known to affect the similarity between static embeddings, and is partially responsible for the existence of word analogies and bias in word2vec \cite{ethayarajh-etal-2019-towards, ethayarajh2019understanding}. But we know little about the effect of frequency on contextualized embeddings, where a word is represented not by a single point, but by a cloud of points. We'll call such a cloud sibling embeddings or a sibling cohort. Many questions arise from this change in representation. How can we characterize the space occupied by a word's sibling embeddings? What effect does a word's frequency in the training data have on its representational geometry? What downstream effects -- and potential social harms -- result from these differences?&#10;&#10;To make sense of the space occupied by a word's sibling embeddings, we introduce two new tools: (1) an identity probe that predicts the identity of the word using one of its sibling embeddings; (2) the minimal bounding hypersphere of a sibling cohort. The identity probe helps us measure how identifiable (i.e., linearly separable) a word is, and how this property is affected by its frequency, polysemy, and tokenization. Meanwhile, the radius of the bounding hypersphere gives us a measure of how diverse a word's representations are. Focusing on BERT \cite{devlin2018bert}, we find that the identifiability of a word -- as measured by the identity probe -- decreases with respect to its frequency. Conversely, the diversity of its representations -- as measured by the bounding hypersphere -- increases with respect to its frequency. This finding has both technical and social ramifications for how we use contextualized word embeddings.&#10;&#10;Do these frequency-based geometric differences affect how we measure semantic similarity? Using the Stanford Contextualized Word Similarity dataset \cite{huang2012improving}, we find that the frequency of two words -- as manifested by the bounding hypersphere of both sibling cohorts -- helps explain the estimation error between cosine similarity and the human baseline (up to Pearson's $r = 0.42$). We call this effect a distortion -- an over- or under-estimation -- in word similarity. &#10;&#10;How pervasive are these frequency-based distortions and do they affect topics with known biases and prejudices? We find that the names of countries from North America and Europe require larger minimal bounding spheres -- i.e., have more diverse representations -- than their counterparts from South America or Africa, correlating significantly with the country's gross domestic product. Furthermore, we find that North American and European country names are seen as more distinct from each other than country names from South America or Africa. &#10;&#10;In brief, we make three main findings:&#10;\begin{itemize}&#10;\itemsep0em&#10;\item Word frequency in pre-training data affects the representational geometry of contextualized embeddings, with low frequency words being more identifiable and less diverse.&#10;\item This leads to widespread distortions in word relations in the embedding space, especially with respect to semantic similarity.&#10;\item These distortions disproportionately affect under-represented populations, such as those in South America and Africa.&#10;\end{itemize}&#10;&#10;Finally, we argue that although distortions can be harmful, there is no such thing as a neutral or undistorted model; rather each corpus has its own distortions, and awareness of these differences is critical.&#10; " />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1502.03520" label="1502.03520">
        <attvalues>
          <attvalue for="0" value="A Latent Variable Model Approach to PMI-based Word Embeddings" />
          <attvalue for="1" value="  Semantic word embeddings represent the meaning of a word via a vector, and&#10;are created by diverse methods. Many use nonlinear operations on co-occurrence&#10;statistics, and have hand-tuned hyperparameters and reweighting methods.&#10;  This paper proposes a new generative model, a dynamic version of the&#10;log-linear topic model of~\citet{mnih2007three}. The methodological novelty is&#10;to use the prior to compute closed form expressions for word statistics. This&#10;provides a theoretical justification for nonlinear models like PMI, word2vec,&#10;and GloVe, as well as some hyperparameter choices. It also helps explain why&#10;low-dimensional semantic embeddings contain linear algebraic structure that&#10;allows solution of word analogies, as shown by~\citet{mikolov2013efficient} and&#10;many subsequent papers.&#10;  Experimental support is provided for the generative model assumptions, the&#10;most important of which is that latent word vectors are fairly uniformly&#10;dispersed in space.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.01199" label="1709.01199">
        <attvalues>
          <attvalue for="0" value="Using $k$-way Co-occurrences for Learning Word Embeddings" />
          <attvalue for="1" value="  Co-occurrences between two words provide useful insights into the semantics&#10;of those words. Consequently, numerous prior work on word embedding learning&#10;have used co-occurrences between two words as the training signal for learning&#10;word embeddings. However, in natural language texts it is common for multiple&#10;words to be related and co-occurring in the same context. We extend the notion&#10;of co-occurrences to cover $k(\geq\!\!2)$-way co-occurrences among a set of&#10;$k$-words. Specifically, we prove a theoretical relationship between the joint&#10;probability of $k(\geq\!\!2)$ words, and the sum of $\ell_2$ norms of their&#10;embeddings. Next, we propose a learning objective motivated by our theoretical&#10;result that utilises $k$-way co-occurrences for learning word embeddings. Our&#10;experimental results show that the derived theoretical relationship does indeed&#10;hold empirically, and despite data sparsity, for some smaller $k$ values,&#10;$k$-way embeddings perform comparably or better than $2$-way embeddings in a&#10;range of tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.06403" label="1910.06403">
        <attvalues>
          <attvalue for="0" value="BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization" />
          <attvalue for="1" value="  Bayesian optimization provides sample-efficient global optimization for a&#10;broad range of applications, including automatic machine learning, engineering,&#10;physics, and experimental design. We introduce BoTorch, a modern programming&#10;framework for Bayesian optimization that combines Monte-Carlo (MC) acquisition&#10;functions, a novel sample average approximation optimization approach,&#10;auto-differentiation, and variance reduction techniques. BoTorch's modular&#10;design facilitates flexible specification and optimization of probabilistic&#10;models written in PyTorch, simplifying implementation of new acquisition&#10;functions. Our approach is backed by novel theoretical convergence results and&#10;made practical by a distinctive algorithmic foundation that leverages fast&#10;predictive distributions, hardware acceleration, and deterministic&#10;optimization. We also propose a novel &quot;one-shot&quot; formulation of the Knowledge&#10;Gradient, enabled by a combination of our theoretical and software&#10;contributions. In experiments, we demonstrate the improved sample efficiency of&#10;BoTorch relative to other popular libraries.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.14909" label="2406.14909">
        <attvalues>
          <attvalue for="0" value="MoA: Mixture of Sparse Attention for Automatic Large Language Model&#10;  Compression" />
          <attvalue for="1" value="  Sparse attention can effectively mitigate the significant memory and&#10;throughput demands of Large Language Models (LLMs) in long contexts. Existing&#10;methods typically employ a uniform sparse attention mask, applying the same&#10;sparse pattern across different attention heads and input lengths. However,&#10;this uniform approach fails to capture the diverse attention patterns inherent&#10;in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this&#10;challenge, we propose the Mixture of Attention (MoA), which automatically&#10;tailors distinct sparse attention configurations to different heads and layers.&#10;MoA constructs and navigates a search space of various attention patterns and&#10;their scaling rules relative to input sequence lengths. It profiles the model,&#10;evaluates potential configurations, and pinpoints the optimal sparse attention&#10;compression plan. MoA adapts to varying input sizes, revealing that some&#10;attention heads expand their focus to accommodate longer sequences, while other&#10;heads consistently concentrate on fixed-length local contexts. Experiments show&#10;that MoA increases the effective context length by $3.9\times$ with the same&#10;average attention span, boosting retrieval accuracy by $1.5-7.1\times$ over the&#10;uniform-attention baseline across Vicuna-7B, Vicuna-13B, and Llama3-8B models.&#10;Moreover, MoA narrows the capability gaps between sparse and dense models,&#10;reducing the maximum relative performance drop from $9\%-36\%$ to within $5\%$&#10;across two long-context understanding benchmarks. MoA achieves a&#10;$1.2-1.4\times$ GPU memory reduction and boosts decode throughput by $5.5-6.7&#10;\times$ for 7B and 13B dense models on a single GPU, with minimal impact on&#10;performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models (LLMs) exhibit remarkable versatility across numerous applications~\cite{brown2020gpt3, tay2022efficientSurvey, Wan2023EfficientLLM}.&#10;Central to LLM is the attention mechanism~\cite{vaswani2017attention}, which computes interactions among tokens within a certain span, thereby enabling context understanding.&#10;Scaling input length is crucial for enhancing LLM capabilities~\cite{Chen2023PI, Tworkowski2023FocusedTrans}, including fact retrieval, summarization, few-shot learning, question answering and so on~\cite{bai2023longbench, yuan2024lveval}. &#10;However, the ever-growing attention computation and Key-Value Cache (KV-Cache) pose significant efficiency challenges~\cite{sheng2023flexgen, xiao2023streamingLLM, han2023lmInfinite, Kwon2023vllm}.&#10;&#10;Previous work proposes sparse attention methods to address the efficiency challenges of long contexts in generative LLMs. &#10;These methods typically employ a uniform, fixed-span sliding window mask across all heads and input lengths, limiting attention to local contexts only~\cite{xiao2023streamingLLM, han2023lmInfinite}.&#10;This approach allows the LLM to take long inputs with a fixed attention span, keeping bounded attention computation and KV caching overhead.&#10;Following previous works~\cite{Chen2023PI, Tworkowski2023FocusedTrans}, we quantify the effective context length as the maximum input length where content retrieval accuracy exceeds a 90\% threshold. &#10;In principle, fixed-span local attention can gradually aggregate global information through multiple model layers, yielding a longer effective context length than each attention span~\cite{feng2022diffuser, zaheer2020bigbird}.&#10;Nonetheless, we reveal that uniform masks, like StreamingLLM~\cite{xiao2023streamingLLM}, hardly extend effective context length beyond the span, as shown in Figure~\ref{fig:appendix/effective_context_length_streaminglm_and_MoA}. Figure~\ref{fig:retrieve_acc_context_position}(b) further illustrates such limitation:&#10;with a 50\% attention span mask, StreamingLLM fails to accurately retrieve content from the earlier half of the input and performs even worse at longer input lengths.&#10;Figure~\ref{fig:attention_oracle} reveals one possible explanation for the problem: while some attention heads focus on local contexts, others encompass the broad span of the entire input sequence. &#10;Consequently, the uniform approach fails to achieve a long effective context length as it limits the attention span of the global-context heads, while excessively allocates compute and memory budget for local-context heads. &#10;Additionally, as the input length increases, some attention heads need a faster increase in attention span than others to avoid serious performance degradation, as shown in Table~\ref{tab:oracle_layer_length}. Unfortunately, the uniform approaches do not include heterogeneous rules to scale the attention spans differently for various heads.&#10;Besides, existing model compression methods~\cite{Men2024ShortGPT, awq, smooth_quant, llm-mq, squeezellm, li2024evaluating} use general language modeling corpora to decide the compression plan, which cannot accurately profile the influence of compression on long-context tasks.&#10;&#10;In this work, we propose Mixture of Attention (\name), a training-free sparse attention method. As illustrated in Figure~\ref{fig:overview}, \name constructs the search space of heterogeneous elastic rules of attention spans.&#10;For automatic LLM compression, \name first utilizes gradient-based profiling to inspect the influences of each attention position on the prediction loss. &#10;Based on the profiling results, \name tailors heterogeneous sparse attention configurations for each model layer and attention head.&#10;During profiling, \name employs a calibration dataset with long-range dependencies and uses the original dense model's response instead of the human-written response as the reference to calculate the loss. This ensures an accurate profiling of the attention influences to facilitate better compression results.&#10;Our contributions are summarized as follows.&#10;&#10;\begin{itemize}&#10; \item Heterogeneous Elastic Rules.&#10; We propose heterogeneous elastic rules for masks of each attention head.&#10; We formulate \name compression search space to include a diverse range of elastic rules that tailor the local attention span relative to the input length for each attention head.&#10; The heterogeneous elastic rules improve the fact retrieval accuracy of \name from 25\% to 98\% compared with masks with uniform span and scaling function for each head.&#10;&#10;\item Calibration Dataset Construction&#10; We emphasize the importance of data engineering in LLM compression. Our findings demonstrate that, instead of relying on general language modeling datasets and human responses, using datasets with long-range dependencies and referencing the original LLM’s responses is essential for accurately profiling the effects of compression.&#10;&#10;\item Automatic Optimization.&#10; We propose an automatic pipeline to find the optimal compression plan encompassing heterogeneous elastic rules for various attention heads. This pipeline can efficiently find the optimal plan within several hours, for example, two hours for compressing Vicuna-13B.&#10;&#10;\end{itemize}&#10;&#10;Experiments show that \name achieves $6.6-8.2\times$ throughput improvements over FlashAttention2, $1.7-1.9\times$ over vLLM framework on 7B and 13B dense LLMs at a 50\% density (the average of KV-Cache length / input length), with only a 1\% average relative degradation in retrieval accuracy. In Section~\ref{sec:efficiency_test}, we show that this significant throughput improvements of MoA over FlashAttention2 can be attributed to four factors: (1) the static size of the KV-cache, (2) reduced attention computations, (3) increased batch size enabled by reduced memory usage, and (4) a specialized kernel implementation. &#10;Additionally, \name achieves over 90\% retrieval accuracy with just 25\% average density, far surpassing sparse attention baselines that need a density of 75\% to 100\% for similar performance. &#10;On long-context understanding benchmarks, \name performs comparably to dense models, with a maximum relative performance drop of less than 5\%, which is about one-sixth of that observed with the uniform sparse attention baseline.&#10;Our code is available at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Sparse Attention Mechanisms, Model Optimization Techniques, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2306.15595" label="2306.15595">
        <attvalues>
          <attvalue for="0" value="Extending Context Window of Large Language Models via Positional&#10;  Interpolation" />
          <attvalue for="1" value="  We present Position Interpolation (PI) that extends the context window sizes&#10;of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal&#10;fine-tuning (within 1000 steps), while demonstrating strong empirical results&#10;on various tasks that require long context, including passkey retrieval,&#10;language modeling, and long document summarization from LLaMA 7B to 65B.&#10;Meanwhile, the extended model by Position Interpolation preserve quality&#10;relatively well on tasks within its original context window. To achieve this&#10;goal, Position Interpolation linearly down-scales the input position indices to&#10;match the original context window size, rather than extrapolating beyond the&#10;trained context length which may lead to catastrophically high attention scores&#10;that completely ruin the self-attention mechanism. Our theoretical study shows&#10;that the upper bound of interpolation is at least $\sim 600 \times$ smaller&#10;than that of extrapolation, further demonstrating its stability. Models&#10;extended via Position Interpolation retain its original architecture and can&#10;reuse most pre-existing optimization and infrastructure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;{ Retrieval-augmented LLM.} One line of work extends LLMs by augmenting it with retrieval modules which fetch related documents and include the retrieval results into the input context of an LLM \cite{karpukhin2020dense, guu2020realm, izacard2022atlas, jiang2022retrieval, khattab2021relevance, Santhanam2022colbertv2}. Our work is complementary to these works as our extended context window allows more documents being included in the input. In addition, with an unmodified attention mechanism and model architecture, our method may be more versatile as it can natively handle tasks beyond retrieval oriented ones, such as long document summarization, few-shots learning, etc.&#10;&#10;{ Recurrent Transformers and Memory Transformers.} Several works add memory capabilities to Transformers through recurrence, which increase the models’ capability of handling very long sequences \cite{bulatov2022recurrent, wu2020memformer, dai2019transformerxl, wu2022memorizing, martins2021inftyformer, mu2023learning}. One limitation of these works is that they only allow attending to a lossy compressed version of past inputs. \cite{mu2023learning} suggested that this may prevent models from remembering specific details in the past inputs. In contrast, our work allows attending to all previous tokens, preserving all details without compression, albeit with higher inference costs. \cite{mohtashami2023landmark} proposed landmark attention which allows full random access to any chunk of the input through introducing landmark tokens. Our work allows full access of the entire input through unmodified attention, which may be useful for tasks such as summarization.&#10;&#10;{ Approximated Multi-head Attention.} There is a large body of research that focuses on decreasing the memory and computational complexity of the multi-head attention (MHA) mechanism through approximation or sparsification \cite{child2019generating, zaheer2020bigbird, beltagy2020longformer, wang2020linformer, choromanski2021rethinking, kitaev2020reformer, ren2021combiner}. Although not the focus of this work, as these methods are not used in LLaMA \cite{touvron2023llama}, we note that our method is compatible with most of them since our changes are restricted to position encodings, and not attention mechanisms.&#10;&#10;{ Length Extrapolation.} A recent line of research aims to train Transformers models on short sequences and inference on longer \cite{press2022train, sun2022lengthextrapolatable, haviv2022transformer}. However, these methods have not been applied in some of the largest language models such as LLaMA \cite{touvron2023llama}, or OPT \cite{zhang2022opt}. This has prevented them from enabling length extrapolation of many pre-existing pre-trained language models. Our work focuses on extending existing LLMs, which can save substantial pre-training costs. In addition, our method preserves the quality of the original models, even for small context window tasks, since it does not deviate far from existing definitions of position encoding or attention mechanisms.&#10;&#10;{ Interpolation.} The most related technique to ours is proposed by \cite{dosovitskiy2021an} in their work on Vision Transformers, where the authors proposed to linearly interpolate learnt position embeddings to support higher resolution, which translates to an increased number of input embeddings, in the fine-tuning stage. The interpolated position embedding weights are used as initialization in the fine-tuning process for the newly added positions. Our work differs from their work in several ways (1) Instead of interpolating position embeddings, our method interpolates position indices, which is more suitable for RoPE like position encodings and may require less training since no trainable parameters are added. (2) We report successful results of extending the context window to 32 times while \cite{dosovitskiy2021an} explored up to 4 times. Our results extend theirs in exploring the upper limit of context window extension via interpolation. (3) We evaluated and confirmed the effectiveness of Position Interpolation for extending context windows for language models.&#10;&#10;{&#10;{ Extending Large Language Model's Context Window} Recently there have been a few works trying to extend LLM's context window length w/o re-running pre-training which can be prohibitively expensive. \cite{ratner2022parallel} directly extend LLM's context window length at inference time by chunking the input to smaller windows; \cite{mohtashami2023landmark} extended to 32k size by finetuning &quot;landmark&quot; tokens. \cite{kazemnejad2023impact} proposed to remove positional encoding completely to bypass the window size limit. Our work aims at extending LLM's context window without any assumption on specific downstream tasks; in other words, we shoot for extending context length while preserving LLM's general-purpose capabilities. &#10;}&#10;We believe our results, in conjunction with \cite{dosovitskiy2021an}, provide empirical evidence on Transformer's remarkable ability of handling significantly longer sequences beyond training. &#10;Further, we conjecture that a method similar to theirs is directly applicable in LLMs with learnable position embeddings such as OPT \cite{zhang2022opt} and we plan to investigate this in the future.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2210.11794" label="2210.11794">
        <attvalues>
          <attvalue for="0" value="Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for&#10;  Long Sequences" />
          <attvalue for="1" value="  Efficient Transformers have been developed for long sequence modeling, due to&#10;their subquadratic memory and time complexity. Sparse Transformer is a popular&#10;approach to improving the efficiency of Transformers by restricting&#10;self-attention to locations specified by the predefined sparse patterns.&#10;However, leveraging sparsity may sacrifice expressiveness compared to&#10;full-attention, when important token correlations are multiple hops away. To&#10;combine advantages of both the efficiency of sparse transformer and the&#10;expressiveness of full-attention Transformer, we propose \textit{Diffuser}, a&#10;new state-of-the-art efficient Transformer. Diffuser incorporates all token&#10;interactions within one attention layer while maintaining low computation and&#10;memory costs. The key idea is to expand the receptive field of sparse attention&#10;using Attention Diffusion, which computes multi-hop token correlations based on&#10;all paths between corresponding disconnected tokens, besides attention among&#10;neighboring tokens. Theoretically, we show the expressiveness of Diffuser as a&#10;universal sequence approximator for sequence-to-sequence modeling, and&#10;investigate its ability to approximate full-attention by analyzing the graph&#10;expander property from the spectral perspective. Experimentally, we investigate&#10;the effectiveness of Diffuser with extensive evaluations, including language&#10;modeling, image modeling, and Long Range Arena (LRA). Evaluation results show&#10;that Diffuser achieves improvements by an average of 0.94% on text&#10;classification tasks and 2.30% on LRA, with 1.67$\times$ memory savings&#10;compared to state-of-the-art benchmarks, which demonstrates superior&#10;performance of Diffuser in both expressiveness and efficiency aspects.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.03853" label="2403.03853">
        <attvalues>
          <attvalue for="0" value="ShortGPT: Layers in Large Language Models are More Redundant Than You&#10;  Expect" />
          <attvalue for="1" value="  As Large Language Models (LLMs) continue to advance in performance, their&#10;size has escalated significantly, with current LLMs containing billions or even&#10;trillions of parameters. However, in this study, we discovered that many layers&#10;of LLMs exhibit high similarity, and some layers play a negligible role in&#10;network functionality. Based on this observation, we define a metric called&#10;Block Influence (BI) to gauge the significance of each layer in LLMs. We then&#10;propose a straightforward pruning approach: layer removal, in which we directly&#10;delete the redundant layers in LLMs based on their BI scores. Experiments&#10;demonstrate that our method, which we call ShortGPT, significantly outperforms&#10;previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT&#10;is orthogonal to quantization-like methods, enabling further reduction in&#10;parameters and computation. The ability to achieve better results through&#10;simple layer removal, as opposed to more complex pruning techniques, suggests a&#10;high degree of redundancy in the model architecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.05839" label="2210.05839">
        <attvalues>
          <attvalue for="0" value="SEAL : Interactive Tool for Systematic Error Analysis and Labeling" />
          <attvalue for="1" value="  With the advent of Transformers, large language models (LLMs) have saturated&#10;well-known NLP benchmarks and leaderboards with high aggregate performance.&#10;However, many times these models systematically fail on tail data or rare&#10;groups not obvious in aggregate evaluation. Identifying such problematic data&#10;groups is even more challenging when there are no explicit labels (e.g.,&#10;ethnicity, gender, etc.) and further compounded for NLP datasets due to the&#10;lack of visual features to characterize failure modes (e.g., Asian males,&#10;animals indoors, waterbirds on land, etc.). This paper introduces an&#10;interactive Systematic Error Analysis and Labeling (\seal) tool that uses a&#10;two-step approach to first identify high error slices of data and then, in the&#10;second step, introduce methods to give human-understandable semantics to those&#10;underperforming slices. We explore a variety of methods for coming up with&#10;coherent semantics for the error groups using language models for semantic&#10;labeling and a text-to-image model for generating visual features. SEAL toolkit&#10;and demo screencast is available at https://huggingface.co/spaces/nazneen/seal.&#10;" />
          <attvalue for="2" value="&#10;&#10;Machine learning systems that seemingly perform well on average can still make systematic errors on important subsets of data. Examples include such systems performing poorly for marginalized groups in chatbots~\cite{Chloe2018Microsoft}, recruiting tools~\cite{Isobel2018Amazon}, cloud products~\cite{Nicolas2020Google}, ad targeting~\cite{Karen2019Facebook}, credit services~\cite{Will2019Apple}, and image cropping~\cite{Isobel2020Twitter}. Discovering and labeling systematic errors in ML systems is an open research problem that would enable building robust models that generalize across subpopulations of data.&#10;&#10;Uncovering underperforming groups of data of a ML system is not straightforward. Firstly, the high-dimensional space of the representations learned by the deep learning models makes it difficult to identify such groups of systematic errors. Secondly, it is difficult to extract and label the hidden semantic information in such groups with high errors without a human-in-the-loop setup. Identifying systematic model failures requires practitioners to think creatively about model evaluation~\cite{ribeiro-etal-2020-beyond, wu2019errudite, goel2021robustness, kiela-etal-2021-dynabench, Yuan_2022}. However, current approaches are mostly limited to examining and manipulating model mispredictions. The onus of identifying what group or subset of data to evaluate still falls on the practitioner, making it inefficient and prone to oversight. Recent works on fine-grained error analysis, such as Domino~\cite{eyuboglu2022domino} and Spotlight~\cite{d2022spotlight} provide solutions to this problem but focus on image datasets which are easier to visualize.&#10;&#10;Error analysis for text data is less explored and more challenging. It also highlights the need to provide semantic summaries of text, which we tackle in SEAL. For example, NLP models could underperform on hundreds of possible input types -- longer inputs, inputs from non-native speaker, inputs with topic domains underrepresented in training, etc. This is a huge barrier of entry for most non-expert ML users who wish to gain a better understanding of their model and datasets with such existing tools. Model evaluation should ideally give actionable insights into a model's performance on a dataset in the form of data curation~\cite{liang2022metashift} or model patching~\cite{goel-etal-2021-goodwill}. &#10;&#10;Our desiderata is a tool that summarizes failures of a model on textual data in a concise, coherent and human intepretable way. Systematic Error Analysis and Labeling (SEAL) is an interactive tool to 1. identify candidate groups of data with high systematic errors and 2. generate semantic labels for those groups. For 1, we use k-means++ on subset of evaluation data with highest loss. Semantic labeling uses LLMs (like GPT3) in zero-shot setting for identifying concepts or topics common to examples in the candidate group. We also explored using a text-to-image model to generate visual features for high error clusters using the Dall-e-mini~\cite{dalle}. Semantic descriptions (via labeling or visual features) of such systematic model errors not only enable practitioners to better understand the failure modes of their model during evaluation but also gives actionable insight to fix them via some form of model patching or data augmentation. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, NLP Dataset Improvement, Linguistics, Cognitive Science, Language Model Evaluation, Error Analysis Techniques, Computer Vision, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2203.14960" label="2203.14960">
        <attvalues>
          <attvalue for="0" value="Domino: Discovering Systematic Errors with Cross-Modal Embeddings" />
          <attvalue for="1" value="  Machine learning models that achieve high overall accuracy often make&#10;systematic errors on important subsets (or slices) of data. Identifying&#10;underperforming slices is particularly challenging when working with&#10;high-dimensional inputs (e.g. images, audio), where important slices are often&#10;unlabeled. In order to address this issue, recent studies have proposed&#10;automated slice discovery methods (SDMs), which leverage learned model&#10;representations to mine input data for slices on which a model performs poorly.&#10;To be useful to a practitioner, these methods must identify slices that are&#10;both underperforming and coherent (i.e. united by a human-understandable&#10;concept). However, no quantitative evaluation framework currently exists for&#10;rigorously assessing SDMs with respect to these criteria. Additionally, prior&#10;qualitative evaluations have shown that SDMs often identify slices that are&#10;incoherent. In this work, we address these challenges by first designing a&#10;principled evaluation framework that enables a quantitative comparison of SDMs&#10;across 1,235 slice discovery settings in three input domains (natural images,&#10;medical images, and time-series data). Then, motivated by the recent&#10;development of powerful cross-modal representation learning approaches, we&#10;present Domino, an SDM that leverages cross-modal embeddings and a novel&#10;error-aware mixture model to discover and describe coherent slices. We find&#10;that Domino accurately identifies 36% of the 1,235 slices in our framework - a&#10;12 percentage point improvement over prior methods. Further, Domino is the&#10;first SDM that can provide natural language descriptions of identified slices,&#10;correctly generating the exact name of the slice in 35% of settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.00758" label="2107.00758">
        <attvalues>
          <attvalue for="0" value="The Spotlight: A General Method for Discovering Systematic Errors in&#10;  Deep Learning Models" />
          <attvalue for="1" value="  Supervised learning models often make systematic errors on rare subsets of&#10;the data. When these subsets correspond to explicit labels in the data (e.g.,&#10;gender, race) such poor performance can be identified straightforwardly. This&#10;paper introduces a method for discovering systematic errors that do not&#10;correspond to such explicitly labelled subgroups. The key idea is that similar&#10;inputs tend to have similar representations in the final hidden layer of a&#10;neural network. We leverage this structure by &quot;shining a spotlight&quot; on this&#10;representation space to find contiguous regions where the model performs&#10;poorly. We show that the spotlight surfaces semantically meaningful areas of&#10;weakness in a wide variety of existing models spanning computer vision, NLP,&#10;and recommender systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.06523" label="2202.06523">
        <attvalues>
          <attvalue for="0" value="MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution&#10;  Shifts and Training Conflicts" />
          <attvalue for="1" value="  Understanding the performance of machine learning models across diverse data&#10;distributions is critically important for reliable applications. Motivated by&#10;this, there is a growing focus on curating benchmark datasets that capture&#10;distribution shifts. While valuable, the existing benchmarks are limited in&#10;that many of them only contain a small number of shifts and they lack&#10;systematic annotation about what is different across different shifts. We&#10;present MetaShift--a collection of 12,868 sets of natural images across 410&#10;classes--to address this challenge. We leverage the natural heterogeneity of&#10;Visual Genome and its annotations to construct MetaShift. The key construction&#10;idea is to cluster images using its metadata, which provides context for each&#10;image (e.g. &quot;cats with cars&quot; or &quot;cats in bathroom&quot;) that represent distinct&#10;data distributions. MetaShift has two important benefits: first, it contains&#10;orders of magnitude more natural data shifts than previously available. Second,&#10;it provides explicit explanations of what is unique about each of its data sets&#10;and a distance score that measures the amount of distribution shift between any&#10;two of its data sets. We demonstrate the utility of MetaShift in benchmarking&#10;several recent proposals for training models to be robust to data shifts. We&#10;find that the simple empirical risk minimization performs the best when shifts&#10;are moderate and no method had a systematic advantage for large shifts. We also&#10;show how MetaShift can help to visualize conflicts between data subsets during&#10;model training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.06716" label="2302.06716">
        <attvalues>
          <attvalue for="0" value="Machine Learning Model Attribution Challenge" />
          <attvalue for="1" value="  We present the findings of the Machine Learning Model Attribution Challenge.&#10;Fine-tuned machine learning models may derive from other trained models without&#10;obvious attribution characteristics. In this challenge, participants identify&#10;the publicly-available base models that underlie a set of anonymous, fine-tuned&#10;large language models (LLMs) using only textual output of the models.&#10;Contestants aim to correctly attribute the most fine-tuned models, with ties&#10;broken in the favor of contestants whose solutions use fewer calls to the&#10;fine-tuned models' API. The most successful approaches were manual, as&#10;participants observed similarities between model outputs and developed&#10;attribution heuristics based on public documentation of the base models, though&#10;several teams also submitted automated, statistical solutions.&#10;" />
          <attvalue for="2" value="&#10;&#10;The announcement of GPT-2, a large language model (LLM) that could generate text of ``unprecedented quality,'' brought new concerns to the forefront of responsible AI in 2019 \cite{gptblog}. &#10;Ongoing algorithmic advances and increases in compute capacity have enabled big tech companies to train multi-billion parameter neural networks on web scale collections of text.&#10;The resulting models write so fluently that even humans trained to detect machine-generated content can be fooled about 30\% of the time \cite{ippolito-etal-2020-automatic}.&#10;Meanwhile, the culture in the natural language processing (NLP) community has moved toward open sourcing these `foundation models' with permissive licenses, allowing much smaller and less-resourced entities to deploy them, often after further training them to specialize them to a different domain or task (fine-tuning)\cite{foundationmodels}.&#10;Since publicly released, trained LLMs can be fine-tuned at a relatively low cost and generate specialized synthetic text at scale, they may empower more actors to engage in malicious uses, for example through cheaper and more effective disinformation \cite{deepmindethical, infop}.&#10;It is unclear how widely LLMs are used for deceptive purposes today.&#10;&#10;Possible mitigations for this new threat include ensuring that synthetic text can be identified as non-human and its origins traced \cite{infop}.&#10;Incorporating a watermark into LLM output has been proposed as a safeguard to privacy and intellectual property \cite{fang-etal-2017-generating, aaronson_2022}, but is not widely practiced. &#10;This competition asks what traces of provenance can be gleaned from synthetic text that lacks explicit marking.&#10;To our knowledge, no generalized forensic process exists to trace textual output from customized models back to the base model.&#10;Attribution techniques could represent a powerful tool for regulation, tracking, and remediation where LLMs are misused.&#10;&#10;The Machine Learning Model Attribution Challenge calls upon contestants to develop creative solutions to uncover model provenance. &#10;Contestants interact with a set of fine-tuned models via a text generation API, attributing each back to a known set of LLMs (`base models'). &#10;Building out forensic capabilities and establishing the difficulty of model attribution is a step toward assured use of LLMs and artificial intelligence in general.&#10;&#10;In this paper, we describe the competition and summarize new methods of attribution proposed by the participants.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Attribution Techniques, Machine Learning Models, Computer Science, Large Language Models, Linguistics, Mathematics, Statistics" />
        </attvalues>
      </node>
      <node id="1911.00650" label="1911.00650">
        <attvalues>
          <attvalue for="0" value="Automatic Detection of Generated Text is Easiest when Humans are Fooled" />
          <attvalue for="1" value="  Recent advancements in neural language modelling make it possible to rapidly&#10;generate vast amounts of human-sounding text. The capabilities of humans and&#10;automatic discriminators to detect machine-generated text have been a large&#10;source of research interest, but humans and machines rely on different cues to&#10;make their decisions. Here, we perform careful benchmarking and analysis of&#10;three popular sampling-based decoding strategies---top-$k$, nucleus sampling,&#10;and untruncated random sampling---and show that improvements in decoding&#10;methods have primarily optimized for fooling humans. This comes at the expense&#10;of introducing statistical abnormalities that make detection easy for automatic&#10;systems. We also show that though both human and automatic detector performance&#10;improve with longer excerpt length, even multi-sentence excerpts can fool&#10;expert human raters over 30% of the time. Our findings reveal the importance of&#10;using both human and automatic detectors to assess the humanness of text&#10;generation systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;{Generative Language Models}&#10;With a sufficiently large training set and number of trainable parameters, neural language models based on the Transformer architecture \cite{vaswani2017attention} are capable of generating convincing, human-like excerpts up to several paragraphs in length.&#10;GPT-2 \cite{radford2019language}, Grover \cite{zellers2019defending}, and Transformer-DMCA \cite{liu2018generating} are a few examples of large, publicly available models with this ability.&#10;Grover, in particular, has been shown to generate fake news that is more trustworthy than human-written fake news according to human raters.&#10;&#10;{Human Detection}&#10;The task of trying to guess whether text is coming from a robot or a fellow human was made famous by the Turing Test \cite{turing1950computing}.&#10;It continues to be used is chatbot evaluation \cite{lowe2017towards}.&#10;The related (but not identical) task of asking human raters to judge the quality of machine-generated excerpts remains the gold-standard for evaluating open-domain generation systems \cite{van2019best}.&#10;\cite{kreps2020all}, \cite{gehrmann2019gltr}, and others have stressed the importance of humans being able to identify fake content on the web.&#10;&#10;{Automatic Detection}&#10;The rise of machine-generated content has led to the development of automated systems to identify it.&#10;Grover was designed to not only generate convincing news excerpts but to also identify them using a fine-tuned version of the generative model itself \cite{zellers2019defending}.&#10;GLTR, expecting attackers to use sampling methods that favor high-likelihood tokens, aims to make machine-generated text detectable by computing histograms over per-token log likelihoods \cite{gehrmann2019gltr}.&#10;\cite{bakhtin2019real} frame human-text detection as a ranking task and evaluate their models' cross-domain and cross-model generalization, finding significant loss in quality when training on one domain and evaluating on another.&#10;\cite{schuster2019we} argue that the language distributional features implicitly or explicitly employed by these detectors are insufficient; instead, one should look to explicit fact-verification models.&#10;Finally, discriminators for whether text is machine-generated are a promising research direction in adversarial training \cite{lin2017adversarial,li2017adversarial} and in automatic evaluation of generative model quality \cite{novikova2017we,kannan2017adversarial,lowe2017towards}. &#10;&#10;{Natural Language Understanding}&#10;Automatic detection of machine-generated text benefits from a semantic understanding of the text.&#10;Contradictions, falsehoods, and topic drift can all indicate that an excerpt was machine-generated.&#10;Encoder-only Transformer models such as BERT \cite{devlin2018bert} have been shown to do very well at tasks requiring this understanding.&#10;While we fine-tune BERT for the task of classifying whether text was machine-generated, others have used the contextual word embeddings from a pre-trained BERT model without fine-tuning to compute a quality score for generated text \cite{zhang2019bertscore}. &#10;It is worth noting that recent work has raised questions as to whether BERT truly builds a semantic understanding to make its predictions, or whether it merely takes advantage of spurious statistical differences between the text of different classes \cite{niven2019probing}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2108.07258" label="2108.07258">
        <attvalues>
          <attvalue for="0" value="On the Opportunities and Risks of Foundation Models" />
          <attvalue for="1" value="  AI is undergoing a paradigm shift with the rise of models (e.g., BERT,&#10;DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a&#10;wide range of downstream tasks. We call these models foundation models to&#10;underscore their critically central yet incomplete character. This report&#10;provides a thorough account of the opportunities and risks of foundation&#10;models, ranging from their capabilities (e.g., language, vision, robotics,&#10;reasoning, human interaction) and technical principles(e.g., model&#10;architectures, training procedures, data, systems, security, evaluation,&#10;theory) to their applications (e.g., law, healthcare, education) and societal&#10;impact (e.g., inequity, misuse, economic and environmental impact, legal and&#10;ethical considerations). Though foundation models are based on standard deep&#10;learning and transfer learning, their scale results in new emergent&#10;capabilities,and their effectiveness across so many tasks incentivizes&#10;homogenization. Homogenization provides powerful leverage but demands caution,&#10;as the defects of the foundation model are inherited by all the adapted models&#10;downstream. Despite the impending widespread deployment of foundation models,&#10;we currently lack a clear understanding of how they work, when they fail, and&#10;what they are even capable of due to their emergent properties. To tackle these&#10;questions, we believe much of the critical research on foundation models will&#10;require deep interdisciplinary collaboration commensurate with their&#10;fundamentally sociotechnical nature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.04359" label="2112.04359">
        <attvalues>
          <attvalue for="0" value="Ethical and social risks of harm from Language Models" />
          <attvalue for="1" value="  This paper aims to help structure the risk landscape associated with&#10;large-scale Language Models (LMs). In order to foster advances in responsible&#10;innovation, an in-depth understanding of the potential risks posed by these&#10;models is needed. A wide range of established and anticipated risks are&#10;analysed in detail, drawing on multidisciplinary expertise and literature from&#10;computer science, linguistics, and social sciences.&#10;  We outline six specific risk areas: I. Discrimination, Exclusion and&#10;Toxicity, II. Information Hazards, III. Misinformation Harms, V. Malicious&#10;Uses, V. Human-Computer Interaction Harms, VI. Automation, Access, and&#10;Environmental Harms. The first area concerns the perpetuation of stereotypes,&#10;unfair discrimination, exclusionary norms, toxic language, and lower&#10;performance by social group for LMs. The second focuses on risks from private&#10;data leaks or LMs correctly inferring sensitive information. The third&#10;addresses risks arising from poor, false or misleading information including in&#10;sensitive domains, and knock-on risks such as the erosion of trust in shared&#10;information. The fourth considers risks from actors who try to use LMs to cause&#10;harm. The fifth focuses on risks specific to LLMs used to underpin&#10;conversational agents that interact with human users, including unsafe use,&#10;manipulation or deception. The sixth discusses the risk of environmental harm,&#10;job automation, and other challenges that may have a disparate effect on&#10;different social groups or communities.&#10;  In total, we review 21 risks in-depth. We discuss the points of origin of&#10;different risks and point to potential mitigation approaches. Lastly, we&#10;discuss organisational responsibilities in implementing mitigations, and the&#10;role of collaboration and participation. We highlight directions for further&#10;research, particularly on expanding the toolkit for assessing and evaluating&#10;the outlined risks in LMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.04246" label="2301.04246">
        <attvalues>
          <attvalue for="0" value="Generative Language Models and Automated Influence Operations: Emerging&#10;  Threats and Potential Mitigations" />
          <attvalue for="1" value="  Generative language models have improved drastically, and can now produce&#10;realistic text outputs that are difficult to distinguish from human-written&#10;content. For malicious actors, these language models bring the promise of&#10;automating the creation of convincing and misleading text for use in influence&#10;operations. This report assesses how language models might change influence&#10;operations in the future, and what steps can be taken to mitigate this threat.&#10;We lay out possible changes to the actors, behaviors, and content of online&#10;influence operations, and provide a framework for stages of the language&#10;model-to-influence operations pipeline that mitigations could target (model&#10;construction, model access, content dissemination, and belief formation). While&#10;no reasonable mitigation can be expected to fully prevent the threat of&#10;AI-enabled influence operations, a combination of multiple mitigations may make&#10;an important difference.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.10742" label="1705.10742">
        <attvalues>
          <attvalue for="0" value="Generating Steganographic Text with LSTMs" />
          <attvalue for="1" value="  Motivated by concerns for user privacy, we design a steganographic system&#10;(&quot;stegosystem&quot;) that enables two users to exchange encrypted messages without&#10;an adversary detecting that such an exchange is taking place. We propose a new&#10;linguistic stegosystem based on a Long Short-Term Memory (LSTM) neural network.&#10;We demonstrate our approach on the Twitter and Enron email datasets and show&#10;that it yields high-quality steganographic text while significantly improving&#10;capacity (encrypted bits per word) relative to the state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.16589" label="2401.16589">
        <attvalues>
          <attvalue for="0" value="ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence&#10;  Labeling Tasks" />
          <attvalue for="1" value="  Prompt-based methods have been successfully applied to multilingual&#10;pretrained language models for zero-shot cross-lingual understanding. However,&#10;most previous studies primarily focused on sentence-level classification tasks,&#10;and only a few considered token-level labeling tasks such as Named Entity&#10;Recognition (NER) and Part-of-Speech (POS) tagging. In this paper, we propose&#10;Token-Level Prompt Decomposition (ToPro), which facilitates the prompt-based&#10;method for token-level sequence labeling tasks. The ToPro method decomposes an&#10;input sentence into single tokens and applies one prompt template to each&#10;token. Our experiments on multilingual NER and POS tagging datasets demonstrate&#10;that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning&#10;in zero-shot cross-lingual transfer, especially for languages that are&#10;typologically different from the source language English. Our method also&#10;attains state-of-the-art performance when employed with the mT5 model. Besides,&#10;our exploratory study in multilingual large language models shows that ToPro&#10;performs much better than the current in-context learning method. Overall, the&#10;performance improvements show that ToPro could potentially serve as a novel and&#10;simple benchmarking method for sequence labeling tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;MPLMs and Cross-Lingual Transfer&#10;The progress in MPLMs has established them as the basis for cross-lingual transfer. MPLMs typically adopt the architecture of monolingual Transformer-based language models and&#10;are pretrained on extensive unlabeled multilingual corpora.&#10;Examples of MPLMs are mBERT~\cite{devlin-etal-2019-bert}, XLM-R~\cite{conneau-etal-2020-unsupervised}, mT5~\cite{xue-etal-2021-mt5}, Glot500~\cite{imanigooghari-etal-2023-glot500}, etc.&#10;Empirical studies~\cite{karthikeyan2020cross, turc2021revisiting} have showcased the remarkable cross-lingual prowess of MPLMs which are fine-tuned on English training datasets,&#10;and then used to predict on test datasets in other languages.&#10;Several benchmark datasets such as XTREME~\cite{hu2020xtreme}, XTREME-R~\cite{ruder-etal-2021-xtreme}, and Taxi1500~\cite{ma2023taxi1500} have been created to assess the capabilities of multilingual models. &#10;The increasing popularity of prompt learning has drawn the attention of researchers&#10;towards prompt-based methods for cross-lingual transfer~\cite{tu-etal-2022-prompt, ma2023promptbased}. &#10;Diverging from previous studies centered on sentence-level classification tasks, our work&#10;applies prompt-based fine-tuning to token-level sequence labeling tasks.&#10;&#10;MLLMs and In-Context Learning&#10;BLOOMZ and mT0~\cite{muennighoff-etal-2023-crosslingual} stand out as two representative multilingual models in the era of LLMs. &#10;Both are fine-tuned on the xP3 dataset which contains multi-lingual multi-task prompts.&#10;BLOOMZ is built upon BLOOM~\cite{Scao2022BLOOMA1} while mT0 is built upon mT5~\cite{xue-etal-2021-mt5}.&#10;\cite{brown2020language} demonstrated that LLMs like GPT-3 can acquire task-solving ICL abilities. &#10;The emergence of MLLMs opens up the possibility for conducting zero-shot cross-lingual ICL, as demonstrated by recent benchmarking efforts, for example MEGA~\cite{ahuja2023mega} and BUFFET~\cite{asai2023buffet}. &#10;However, current ICL methods using text-to-text prompting with a fixed output template for sequence labeling tasks ``consistently exhibit extremely poor performance''~\cite{asai2023buffet} when applied to MLLMs, failing to&#10;exploit their real cross-lingual transfer abilities. Contrary to that, our proposed \method method better reflects the potential of MLLMs on token-level tasks.&#10;&#10;Prompt Methods for Sequence Labeling Tasks&#10;Although prompt-based methods proved useful in sentence-level classification tasks, they were seldom employed for token-level labeling tasks. &#10;\cite{cui-etal-2021-template} applied template-based prompting methods to the BART model~\cite{lewis-etal-2020-bart} for NER tasks. Their method is rank-based. They generate a sentence for each possible label and compute the probabilities of all generated sentences for the prediction, which can be expensive to decode.&#10;\cite{ma-etal-2022-template} proposed a template-free prompting method for few-shot NER, called entity-oriented LM fine-tuning. However, they adopt the span-based task formulation of NER, resulting in more complexity, while our proposed method applies to NER tasks in the IOB (Inside-Outside-Beginning) tagging format. \cite{blevins-etal-2023-prompting} proposed a structural prompting method for sequence labeling tasks, which was adapted for multilingual benchmarking large language models in recent work~\cite{ahuja2023mega}.&#10;&#10;" />
          <attvalue for="4" value="Token-Level Sequence Labeling, Machine Learning, Computer Science, Cross-Lingual Transfer Learning, Linguistics, Multilingual Language Models, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2311.08298" label="2311.08298">
        <attvalues>
          <attvalue for="0" value="A Survey of Confidence Estimation and Calibration in Large Language&#10;  Models" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated remarkable capabilities across&#10;a wide range of tasks in various domains. Despite their impressive performance,&#10;they can be unreliable due to factual errors in their generations. Assessing&#10;their confidence and calibrating them across different tasks can help mitigate&#10;risks and enable LLMs to produce better generations. There has been a lot of&#10;recent research aiming to address this, but there has been no comprehensive&#10;overview to organize it and outline the main lessons learned. The present&#10;survey aims to bridge this gap. In particular, we outline the challenges and we&#10;summarize recent technical advancements for LLM confidence estimation and&#10;calibration. We further discuss their applications and suggest promising&#10;directions for future work.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have demonstrated a wide range of capabilities, such as world knowledge storage, sophisticated language-based reasoning, and in-context learning~\cite{petroni2019language, wei2022chain,brown2020language}. However, LLMs do not consistently achieve good performance~\cite{factsurveywang2023,zhang2023siren}. Their generation still includes biases~\cite{zhao2021calibrate,wang2023reducing} and hallucinations that do not align with reality~\cite{zhang2023siren}. Evaluating the trustworthiness of responses from these models remains challenging~\cite{liu2023trustworthy}.&#10;&#10;Confidence (or uncertainty) estimation is crucial for tasks like out-of-distribution detection and selective prediction~\cite{DBLP:conf/nips/KendallG17,lu2022learning}, and it has been extensively studied and applied in various contexts ~\cite{lee2018simple,devries2018learning}. A related concept is that of model calibration, which focuses on aligning predictive probabilities (estimated confidence) to actual accuracy~\cite{Guo2017OnCO}. &#10;&#10;However, applying these methods directly to LLMs presents several challenges. The output space of these models is significantly larger than that of discriminative models. The number of possible outcomes grows exponentially with the generation length, making it impossible to access all potential responses. Additionally, different expressions may convey the same meaning, suggesting that confidence estimation should consider semantics.&#10;{Lastly,} LLMs show unique properties, such as expressing confidence in words~\cite{lin2022teaching,xiong2023can} and the ability to perform zero-shot or few-shot learning~\cite{brown2020language}. Nonetheless, their responses can be sensitive to the prompts, e.g., the examples provided and their order, which can cause a lot of instability in the results. Given this, confidence estimation and calibration for LLMs is growing as an emerging area of interest~\cite{jiang2021can,lin2022teaching,lin2023generating,shrivastava2023llamas}. &#10;&#10;While existing surveys mainly focused on issues such as hallucination and factuality in LLMs~\cite{zhang2023siren,wang2023survey}, there are no comprehensive surveys systematically discussing the technical advancements in LLMs, and here we aim to bridge this gap. We explore the unique challenges posed by LLMs and examine the latest studies addressing these issues. We first discuss key concepts such as confidence, uncertainty, and calibration in the context of neural models, as detailed in Section~\ref{sec:preliminary}. Then, we pursue two different directions: one addressing confidence estimation and calibration techniques for generation tasks in Section~\ref{sec:gm4ge}, and the other for classification tasks in Section~\ref{sec:gm4cl}. We conclude by exploring their practical applications (Section~\ref{sec:applications}) and looking at potential future research directions (Section~\ref{sec:future_directions}). {Figure~\ref{categorization_of_survey} provides a comprehensive representation of the survey's structure. By conducting a detailed examination of existing research, our goal is to illuminate this vital facet of LLMs, contributing to the development of more reliable applications.}&#10;&#10;\tikzstyle{my-box}=[&#10; rectangle,&#10; draw=hidden-draw,&#10; rounded corners,&#10; text opacity=1,&#10; minimum height=1.5em,&#10; minimum width=5em,&#10; inner sep=2pt,&#10; align=center,&#10; fill opacity=.5,&#10;]&#10;\tikzstyle{cause_leaf}=[my-box, minimum height=1.5em,&#10; fill=harvestgold!20, text=black, align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10;]&#10;\tikzstyle{detect_leaf}=[my-box, minimum height=1.5em,&#10; fill=cyan!20, text=black, align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10;]&#10;\tikzstyle{mitigate_leaf}=[my-box, minimum height=1.5em,&#10; fill=lightgreen!20, text=black, align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10;]&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Reliability, Computer Science, Confidence Estimation Methods, Linguistics, Artificial Intelligence, Natural Language Processing, Calibration Techniques" />
        </attvalues>
      </node>
      <node id="1909.01066" label="1909.01066">
        <attvalues>
          <attvalue for="0" value="Language Models as Knowledge Bases?" />
          <attvalue for="1" value="  Recent progress in pretraining language models on large textual corpora led&#10;to a surge of improvements for downstream NLP tasks. Whilst learning linguistic&#10;knowledge, these models may also be storing relational knowledge present in the&#10;training data, and may be able to answer queries structured as&#10;&quot;fill-in-the-blank&quot; cloze statements. Language models have many advantages over&#10;structured knowledge bases: they require no schema engineering, allow&#10;practitioners to query about an open class of relations, are easy to extend to&#10;more data, and require no human supervision to train. We present an in-depth&#10;analysis of the relational knowledge already present (without fine-tuning) in a&#10;wide range of state-of-the-art pretrained language models. We find that (i)&#10;without fine-tuning, BERT contains relational knowledge competitive with&#10;traditional NLP methods that have some access to oracle knowledge, (ii) BERT&#10;also does remarkably well on open-domain question answering against a&#10;supervised baseline, and (iii) certain types of factual knowledge are learned&#10;much more readily than others by standard language model pretraining&#10;approaches. The surprisingly strong ability of these models to recall factual&#10;knowledge without any fine-tuning demonstrates their potential as unsupervised&#10;open-domain QA systems. The code to reproduce our analysis is available at&#10;https://github.com/facebookresearch/LAMA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In this sectionwe provide background on language models. Statistics for the models that we include in our investigation are summarized in \ref{tab:models}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2310.07521" label="2310.07521">
        <attvalues>
          <attvalue for="0" value="Survey on Factuality in Large Language Models: Knowledge, Retrieval and&#10;  Domain-Specificity" />
          <attvalue for="1" value="  This survey addresses the crucial issue of factuality in Large Language&#10;Models (LLMs). As LLMs find applications across diverse domains, the&#10;reliability and accuracy of their outputs become vital. We define the&#10;Factuality Issue as the probability of LLMs to produce content inconsistent&#10;with established facts. We first delve into the implications of these&#10;inaccuracies, highlighting the potential consequences and challenges posed by&#10;factual errors in LLM outputs. Subsequently, we analyze the mechanisms through&#10;which LLMs store and process facts, seeking the primary causes of factual&#10;errors. Our discussion then transitions to methodologies for evaluating LLM&#10;factuality, emphasizing key metrics, benchmarks, and studies. We further&#10;explore strategies for enhancing LLM factuality, including approaches tailored&#10;for specific domains. We focus two primary LLM configurations standalone LLMs&#10;and Retrieval-Augmented LLMs that utilizes external data, we detail their&#10;unique challenges and potential enhancements. Our survey offers a structured&#10;guide for researchers aiming to fortify the factual reliability of LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.01219" label="2309.01219">
        <attvalues>
          <attvalue for="0" value="Siren's Song in the AI Ocean: A Survey on Hallucination in Large&#10;  Language Models" />
          <attvalue for="1" value="  While large language models (LLMs) have demonstrated remarkable capabilities&#10;across a range of downstream tasks, a significant concern revolves around their&#10;propensity to exhibit hallucinations: LLMs occasionally generate content that&#10;diverges from the user input, contradicts previously generated context, or&#10;misaligns with established world knowledge. This phenomenon poses a substantial&#10;challenge to the reliability of LLMs in real-world scenarios. In this paper, we&#10;survey recent efforts on the detection, explanation, and mitigation of&#10;hallucination, with an emphasis on the unique challenges posed by LLMs. We&#10;present taxonomies of the LLM hallucination phenomena and evaluation&#10;benchmarks, analyze existing approaches aiming at mitigating LLM hallucination,&#10;and discuss potential directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.05374" label="2308.05374">
        <attvalues>
          <attvalue for="0" value="Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language&#10;  Models' Alignment" />
          <attvalue for="1" value="  Ensuring alignment, which refers to making models behave in accordance with&#10;human intentions [1,2], has become a critical task before deploying large&#10;language models (LLMs) in real-world applications. For instance, OpenAI devoted&#10;six months to iteratively aligning GPT-4 before its release [3]. However, a&#10;major challenge faced by practitioners is the lack of clear guidance on&#10;evaluating whether LLM outputs align with social norms, values, and&#10;regulations. This obstacle hinders systematic iteration and deployment of LLMs.&#10;To address this issue, this paper presents a comprehensive survey of key&#10;dimensions that are crucial to consider when assessing LLM trustworthiness. The&#10;survey covers seven major categories of LLM trustworthiness: reliability,&#10;safety, fairness, resistance to misuse, explainability and reasoning, adherence&#10;to social norms, and robustness. Each major category is further divided into&#10;several sub-categories, resulting in a total of 29 sub-categories.&#10;Additionally, a subset of 8 sub-categories is selected for further&#10;investigation, where corresponding measurement studies are designed and&#10;conducted on several widely-used LLMs. The measurement results indicate that,&#10;in general, more aligned models tend to perform better in terms of overall&#10;trustworthiness. However, the effectiveness of alignment varies across the&#10;different trustworthiness categories considered. This highlights the importance&#10;of conducting more fine-grained analyses, testing, and making continuous&#10;improvements on LLM alignment. By shedding light on these key dimensions of LLM&#10;trustworthiness, this paper aims to provide valuable insights and guidance to&#10;practitioners in the field. Understanding and addressing these concerns will be&#10;crucial in achieving reliable and ethically sound deployment of LLMs in various&#10;applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.04865" label="1802.04865">
        <attvalues>
          <attvalue for="0" value="Learning Confidence for Out-of-Distribution Detection in Neural Networks" />
          <attvalue for="1" value="  Modern neural networks are very powerful predictive models, but they are&#10;often incapable of recognizing when their predictions may be wrong. Closely&#10;related to this is the task of out-of-distribution detection, where a network&#10;must determine whether or not an input is outside of the set on which it is&#10;expected to safely perform. To jointly address these issues, we propose a&#10;method of learning confidence estimates for neural networks that is simple to&#10;implement and produces intuitively interpretable outputs. We demonstrate that&#10;on the task of out-of-distribution detection, our technique surpasses recently&#10;proposed techniques which construct confidence based on the network's output&#10;distribution, without requiring any additional labels or access to&#10;out-of-distribution examples. Additionally, we address the problem of&#10;calibrating out-of-distribution detectors, where we demonstrate that&#10;misclassified in-distribution examples can be used as a proxy for&#10;out-of-distribution examples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.04599" label="1706.04599">
        <attvalues>
          <attvalue for="0" value="On Calibration of Modern Neural Networks" />
          <attvalue for="1" value="  Confidence calibration -- the problem of predicting probability estimates&#10;representative of the true correctness likelihood -- is important for&#10;classification models in many applications. We discover that modern neural&#10;networks, unlike those from a decade ago, are poorly calibrated. Through&#10;extensive experiments, we observe that depth, width, weight decay, and Batch&#10;Normalization are important factors influencing calibration. We evaluate the&#10;performance of various post-processing calibration methods on state-of-the-art&#10;architectures with image and document classification datasets. Our analysis and&#10;experiments not only offer insights into neural network learning, but also&#10;provide a simple and straightforward recipe for practical settings: on most&#10;datasets, temperature scaling -- a single-parameter variant of Platt Scaling --&#10;is surprisingly effective at calibrating predictions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.13063" label="2306.13063">
        <attvalues>
          <attvalue for="0" value="Can LLMs Express Their Uncertainty? An Empirical Evaluation of&#10;  Confidence Elicitation in LLMs" />
          <attvalue for="1" value="  Empowering large language models to accurately express confidence in their&#10;answers is essential for trustworthy decision-making. Previous confidence&#10;elicitation methods, which primarily rely on white-box access to internal model&#10;information or model fine-tuning, have become less suitable for LLMs,&#10;especially closed-source commercial APIs. This leads to a growing need to&#10;explore the untapped area of black-box approaches for LLM uncertainty&#10;estimation. To better break down the problem, we define a systematic framework&#10;with three components: prompting strategies for eliciting verbalized&#10;confidence, sampling methods for generating multiple responses, and aggregation&#10;techniques for computing consistency. We then benchmark these methods on two&#10;key tasks-confidence calibration and failure prediction-across five types of&#10;datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs&#10;including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights:&#10;1) LLMs, when verbalizing their confidence, tend to be overconfident,&#10;potentially imitating human patterns of expressing confidence. 2) As model&#10;capability scales up, both calibration and failure prediction performance&#10;improve. 3) Employing our proposed strategies, such as human-inspired prompts,&#10;consistency among multiple responses, and better aggregation strategies can&#10;help mitigate this overconfidence from various perspectives. 4) Comparisons&#10;with white-box methods indicate that while white-box methods perform better,&#10;the gap is narrow, e.g., 0.522 to 0.605 in AUROC. Despite these advancements,&#10;none of these techniques consistently outperform others, and all investigated&#10;methods struggle in challenging tasks, such as those requiring professional&#10;knowledge, indicating significant scope for improvement. We believe this study&#10;can serve as a strong baseline and provide insights for eliciting confidence in&#10;black-box LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.00955" label="2012.00955">
        <attvalues>
          <attvalue for="0" value="How Can We Know When Language Models Know? On the Calibration of&#10;  Language Models for Question Answering" />
          <attvalue for="1" value="  Recent works have shown that language models (LM) capture different types of&#10;knowledge regarding facts or common sense. However, because no model is&#10;perfect, they still fail to provide appropriate answers in many cases. In this&#10;paper, we ask the question &quot;how can we know when language models know, with&#10;confidence, the answer to a particular query?&quot; We examine this question from&#10;the point of view of calibration, the property of a probabilistic model's&#10;predicted probabilities actually being well correlated with the probabilities&#10;of correctness. We examine three strong generative models -- T5, BART, and&#10;GPT-2 -- and study whether their probabilities on QA tasks are well calibrated,&#10;finding the answer is a relatively emphatic no. We then examine methods to&#10;calibrate such models to make their confidence scores correlate better with the&#10;likelihood of correctness through fine-tuning, post-hoc probability&#10;modification, or adjustment of the predicted outputs or inputs. Experiments on&#10;a diverse range of datasets demonstrate the effectiveness of our methods. We&#10;also perform analysis to study the strengths and limitations of these methods,&#10;shedding light on further improvements that may be made in methods for&#10;calibrating LMs. We have released the code at&#10;https://github.com/jzbjyb/lm-calibration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Calibration&#10;Calibration is a well-studied topic in other tasks such as medical diagnosis \cite{jiang-2010-medcal} and image recognition \cite{guo-2017-cal,lee-2018-cal}.&#10;Previous works in NLP have examined calibration in structured prediction problems such as part-of-speech tagging and named entity recognition \cite{jagannatha-2020-structcal}, natural language understanding tasks such as natural language inference, paraphrase detection, extractive question answering, and text classification \cite{desai-2020-transcal,kamath-2020-qacal,kong-2020-lminout}.&#10;In contrast, we focus on calibrating LMs themselves by treating them as natural language generators that predict the next words given a particular input.&#10;&#10;LM probing&#10;Previous works probe pre-trained LMs with respect to syntactic and semantic properties \cite{hewitt-2019-structprob,tenney-2019-bertpipe}, factual knowledge \cite{petroni-etal-2019-language,poerner-2019-ebert,jiang-2019-lpaqa}, commonsense knowledge \cite{trinh-2018-commonsense,kocijan-2019-wsc}, and other properties \cite{talmor-2019-olmpics}.&#10;These works usually focus on what LMs know, while in this paper we also consider the cases when LMs do not know the answer with confidence.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.19187" label="2305.19187">
        <attvalues>
          <attvalue for="0" value="Generating with Confidence: Uncertainty Quantification for Black-box&#10;  Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) specializing in natural language generation&#10;(NLG) have recently started exhibiting promising capabilities across a variety&#10;of domains. However, gauging the trustworthiness of responses generated by LLMs&#10;remains an open challenge, with limited research on uncertainty quantification&#10;(UQ) for NLG. Furthermore, existing literature typically assumes white-box&#10;access to language models, which is becoming unrealistic either due to the&#10;closed-source nature of the latest LLMs or computational constraints. In this&#10;work, we investigate UQ in NLG for *black-box* LLMs. We first differentiate&#10;*uncertainty* vs *confidence*: the former refers to the ``dispersion'' of the&#10;potential predictions for a fixed input, and the latter refers to the&#10;confidence on a particular prediction/generation. We then propose and compare&#10;several confidence/uncertainty measures, applying them to *selective NLG* where&#10;unreliable results could either be ignored or yielded for further assessment.&#10;Experiments were carried out with several popular LLMs on question-answering&#10;datasets (for evaluation purposes). Results reveal that a simple measure for&#10;the semantic dispersion can be a reliable predictor of the quality of LLM&#10;responses, providing valuable insights for practitioners on uncertainty&#10;management when adopting LLMs. The code to replicate our experiments is&#10;available at https://github.com/zlin7/UQ-NLG.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;The quantification of uncertainty has emerged as a significant area of research across various machine learning domains, including natural language processing (NLP). &#10;However, previous NLP studies have predominantly addressed the associated UQ challenges similarly to classification or regression methodologies~\cite{desai-durrett-2020-calibration,jiang-etal-2021-know,kamath-etal-2020-selective,wang-etal-2022-uncertainty,xiong2023llms}. &#10;For instance, \cite{kamath-etal-2020-selective} examines the selective question-answering task as a multiple-choice problem, reducing it to a de facto classification task rather than directly engaging with free-form generation. &#10;As recently argued in \cite{kuhn2023semantic}, such approaches enable the application of UQ measures akin to those employed in more extensively researched classification or regression contexts, but overlook the generative aspects and distinct challenges of NLG.&#10;&#10;Recently, some research started to study uncertainty quantification for NLG.&#10;One line of research involves asking the LLM itself for its confidence, with or without additional fine-tuning~\cite{kadavath2022language,Lin2022TeachingMT, DBLP:journals/corr/abs-2012-14983,chen2023quantifying}.&#10;Apart from being expensive, such approaches can be hard to generalize due to opaque training details or differences between LLMs~\cite{kuhn2023semantic}. &#10;The work most relevant to ours is~\cite{kuhn2023semantic}, which proposes to compute the ``semantic entropy'' by considering the equivalence relationships amongst generated answers, and requires no training.&#10;Nonetheless, it still requires access to the token-level numerical output of the LLM, which is not always available.&#10;&#10;As discussed, one of the most pertinent applications of uncertainty quantification in NLG involves the development of methods for selective NLG (or, NLG with rejection). &#10;This emerging field has limited research to date, but shares close ties with classification with rejection.&#10;Both tasks can be viewed as determining when to trust a model, whether it is a classifier or an LLM. &#10;Numerous classification with rejection methods emphasize the identification of a reliable confidence score (some of which are jointly trained with the classifier)~\cite{Corbiere2019AddressingConfidence,Fumera2000RejectThresholds,Geifman2017SelectiveNetworks,Jiang2018ToClassifier}, which is often not only dependent on the input but also on the prediction. &#10;As existing uncertainty quantification research for NLG primarily focuses on input uncertainty~\cite{kuhn2023semantic,malinin2021uncertainty}, it overlooks the crucial aspect of confidence, which is essential in deciding when to trust an LLM's response (see \ref{sec:background:uvsc} for more discussion). &#10;Recent works have explored selective classification in NLP tasks~\cite{varshney-etal-2022-investigating,varshney-etal-2022-towards}.&#10;However, the distinct generative nature of NLG precludes the direct adaptation of confidence measures from the classification with rejection literature. &#10;This paper serves as a step to bridge this gap and enhance the effectiveness of uncertainty quantification in NLG. &#10;&#10;It is worth noting that the issue of LLMs being overconfident \cite{mielke-etal-2022-reducing,si-etal-2022-examining,xiong2023llms} is orthogonal to our work, as we evaluate measures basing how they rank different samples - such measures may then be calibrated by distribution-free uncertainty quantification methods like in \cite{schuster2022confident}.&#10;\cite{giulianelli2023comes} also provides an interesting exploration of the inherent uncertainty in human responses for many NLG tasks, in a black-box manner.&#10;Finally, carefully designed prompts have been proposed to improve the quality of the generated responses in general~\cite{Zhou2023NavigatingTG,si2023prompting,wei2022chain}.&#10;Orthogonal to UQ but related to selective NLG, \cite{varshney-baral-2023-post} focuses on reattempting rejected samples, with the help of an auxiliary model trained on an additional dataset that predicts the correctness of the generation.&#10;This paper focuses on providing quantitative uncertainty/confidence measures, which can be used to identify high-quality generations. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2311.08877" label="2311.08877">
        <attvalues>
          <attvalue for="0" value="Llamas Know What GPTs Don't Show: Surrogate Models for Confidence&#10;  Estimation" />
          <attvalue for="1" value="  To maintain user trust, large language models (LLMs) should signal low&#10;confidence on examples where they are incorrect, instead of misleading the&#10;user. The standard approach of estimating confidence is to use the softmax&#10;probabilities of these models, but as of November 2023, state-of-the-art LLMs&#10;such as GPT-4 and Claude-v1.3 do not provide access to these probabilities. We&#10;first study eliciting confidence linguistically -- asking an LLM for its&#10;confidence in its answer -- which performs reasonably (80.5% AUC on GPT-4&#10;averaged across 12 question-answering datasets -- 7% above a random baseline)&#10;but leaves room for improvement. We then explore using a surrogate confidence&#10;model -- using a model where we do have probabilities to evaluate the original&#10;model's confidence in a given question. Surprisingly, even though these&#10;probabilities come from a different and often weaker model, this method leads&#10;to higher AUC than linguistic confidences on 9 out of 12 datasets. Our best&#10;method composing linguistic confidences and surrogate model probabilities gives&#10;state-of-the-art confidence estimates on all 12 datasets (84.6% average AUC on&#10;GPT-4).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Confidence Estimation for LLMs. Confidence estimation for LLMs has been studied in several related works.~\cite{kadavath2022language} show that Claude’s model probabilities are well-calibrated on multiple/choice and True/False questions.~\cite{zhou2023navigating} study the effect of introducing expressions of uncertainty into prompts, on model accuracy. Our work differs from these since we focus on confidence elicitation for models which don’t provide log probabilities. Concurrent work~\cite{xiong2023can} studies calibration and selective classification of linguistic confidence scores generated by LLMs. While this work also elicits prompted confidences, they focus on self-consistency (SC) based methods which are expensive because they require prompting GPT-4 several times. Our proposed Surrogate and Mixture of models methods are less expensive, since model probabilities from smaller models (Llama 2) are used to improve the confidence estimates of larger models (GPT-4). We also show performance improvements over their best method.&#10;~\cite{lin2022teaching} examine fine-tuning language models to improve confidence estimation, which we do not have access to.&#10;&#10;Selective Classification and OOD Detection.&#10;Our paper focuses on selective classification, a classical problem in machine learning~\cite{elyaniv2010foundations,khani2016unanimity,feng2019selective,jones2021selective} and statistics~\cite{chow1970optimum,hellman1970probability}.&#10;A related problem is out-of-distribution detection~\cite{pimentel2014review,liang2018enhancing,ovadia2019uncertainty}, where the goal is to detect examples very different from training (where the model may make mistakes).&#10;Prior work uses internals of the models --- probability outputs~\cite{hendrycks2017baseline}, representations~\cite{lee2018unified} of models, or even updates the training procedure~\cite{bartlett2008classification,mozannar2020consistent} --- which state-of-the-art LLMs do not currently give access to.&#10;&#10;Calibration.&#10;The general idea of confidence estimation is also studied in calibration~\cite{murphy1977reliability,degroot1983forecasters,naeini2014binary,guo2017calibration}.&#10;While related, the focus is different---a model which outputs its accuracy on every example has 0 calibration error (ECE), but cannot separate correct and incorrect examples~\cite{kuleshov2015calibrated}.&#10; " />
        </attvalues>
      </node>
      <node id="1310.4546" label="1310.4546">
        <attvalues>
          <attvalue for="0" value="Distributed Representations of Words and Phrases and their&#10;  Compositionality" />
          <attvalue for="1" value="  The recently introduced continuous Skip-gram model is an efficient method for&#10;learning high-quality distributed vector representations that capture a large&#10;number of precise syntactic and semantic word relationships. In this paper we&#10;present several extensions that improve both the quality of the vectors and the&#10;training speed. By subsampling of the frequent words we obtain significant&#10;speedup and also learn more regular word representations. We also describe a&#10;simple alternative to the hierarchical softmax called negative sampling. An&#10;inherent limitation of word representations is their indifference to word order&#10;and their inability to represent idiomatic phrases. For example, the meanings&#10;of &quot;Canada&quot; and &quot;Air&quot; cannot be easily combined to obtain &quot;Air Canada&quot;.&#10;Motivated by this example, we present a simple method for finding phrases in&#10;text, and show that learning good vector representations for millions of&#10;phrases is possible.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1408.3456" label="1408.3456">
        <attvalues>
          <attvalue for="0" value="SimLex-999: Evaluating Semantic Models with (Genuine) Similarity&#10;  Estimation" />
          <attvalue for="1" value="  We present SimLex-999, a gold standard resource for evaluating distributional&#10;semantic models that improves on existing resources in several important ways.&#10;First, in contrast to gold standards such as WordSim-353 and MEN, it explicitly&#10;quantifies similarity rather than association or relatedness, so that pairs of&#10;entities that are associated but not actually similar [Freud, psychology] have&#10;a low rating. We show that, via this focus on similarity, SimLex-999&#10;incentivizes the development of models with a different, and arguably wider&#10;range of applications than those which reflect conceptual association. Second,&#10;SimLex-999 contains a range of concrete and abstract adjective, noun and verb&#10;pairs, together with an independent rating of concreteness and (free)&#10;association strength for each pair. This diversity enables fine-grained&#10;analyses of the performance of models on concepts of different types, and&#10;consequently greater insight into how architectures can be improved. Further,&#10;unlike existing gold standard evaluations, for which automatic approaches have&#10;reached or surpassed the inter-annotator agreement ceiling, state-of-the-art&#10;models perform well below this ceiling on SimLex-999. There is therefore plenty&#10;of scope for SimLex-999 to quantify future improvements to distributional&#10;semantic models, guiding the development of the next generation of&#10;representation-learning architectures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.11987" label="2205.11987">
        <attvalues>
          <attvalue for="0" value="Word-order typology in Multilingual BERT: A case study in&#10;  subordinate-clause detection" />
          <attvalue for="1" value="  The capabilities and limitations of BERT and similar models are still unclear&#10;when it comes to learning syntactic abstractions, in particular across&#10;languages. In this paper, we use the task of subordinate-clause detection&#10;within and across languages to probe these properties. We show that this task&#10;is deceptively simple, with easy gains offset by a long tail of harder cases,&#10;and that BERT's zero-shot performance is dominated by word-order effects,&#10;mirroring the SVO/VSO/SOV typology.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Both aspects of our analysis -- subordinate-clause detection and the&#10;study of word-order effects -- have been addressed but not in&#10;conjunction and not in a multiple-source-language setting. Our study&#10;extends previous approaches by providing a ZS \enquote{upper baseline}&#10;derived from the study of the performance of several monolingual models &#10;and then conducting a novel many-sources-to-many-target analysis of zero-shot&#10;performance.&#10;&#10;\cite{lin-etal-2019-open} test BERT on the auxiliary-classification&#10;task (main vs.\ subordinate clause) as part of their investigation of&#10;BERT's linguistic knowledge. \cite{ronnqvist-etal-2019-multilingual}&#10;extend this analysis to the multilingual setting with a focus on&#10;Nordic languages.&#10;&#10;Word-order differences have been shown to impact the performance of&#10;English-based cross-lingual models, especially in the domain of&#10;syntactic parsing \cite{ahmad-etal-2019-difficulties} and with tasks&#10;that rely on syntactic information&#10;\cite{liu2020importance,arviv-etal-2021-relation}, while reordering&#10;has been long known to be an efficient preprocessing step in syntactic&#10;transfer \cite{rasooli-collins-2019-low} and machine translation,&#10;both statistical \cite{wang-etal-2007-chinese} and neural&#10;\cite{chen-etal-2019-neural}.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Limitations, Artificial Intelligence, Natural Language Processing, Syntactic Abstraction Learning, Cross-Linguistic Analysis" />
        </attvalues>
      </node>
      <node id="2405.10276" label="2405.10276">
        <attvalues>
          <attvalue for="0" value="Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers" />
          <attvalue for="1" value="  Numerous recent works aim to enhance the efficacy of Large Language Models&#10;(LLMs) through strategic prompting. In particular, the Optimization by&#10;PROmpting (OPRO) approach provides state-of-the-art performance by leveraging&#10;LLMs as optimizers where the optimization task is to find instructions that&#10;maximize the task accuracy. In this paper, we revisit OPRO for automated&#10;prompting with relatively small-scale LLMs, such as LLaMa-2 family and Mistral&#10;7B. Our investigation reveals that OPRO shows limited effectiveness in&#10;small-scale LLMs, with limited inference capabilities constraining optimization&#10;ability. We suggest future automatic prompting engineering to consider both&#10;model capabilities and computational costs. Additionally, for small-scale LLMs,&#10;we recommend direct instructions that clearly outline objectives and&#10;methodologies as robust prompt baselines, ensuring efficient and effective&#10;prompt engineering in ongoing research.&#10;" />
          <attvalue for="2" value="&#10;Advancements in large language models (LLMs) have catalyzed a shift towards prompting-based learning, distinguishing models with capacities exceeding 100 billion parameters for their few-shot learning abilities without extensive retraining~\cite{brown-2020-fewshotlearners}. In-context learning, facilitated through the strategic use of prompts, enables these models to generate task-specific responses, marking a departure from traditional pre-train and fine-tune approaches~\cite{Liu2021GPTUT, Wan2023EfficientLL}. &#10;&#10;The Chain of Thought (CoT) technique significantly advances LLMs' problem-solving capabilities by incorporating intermediate reasoning steps, facilitating effective zero-shot reasoning and performance enhancements with prompts like &quot;Let's think step by step&quot;~\cite{wei-2022-chainofthought, Wang2022SelfConsistencyIC, Yao2023TreeOT, kojima-2022-zeroshotreasoners}. While initially dependent on manual prompt creation, recent developments in automated prompt engineering, such as APE~\cite{Zhou2022LargeLM} and APO~\cite{Pryzant2023AutomaticPO}, leverage LLMs for dynamic prompt generation and refinement. This iterative process enhances NLP task accuracy through feedback and selection. Building on this, the proposition of LLMs as optimizers~\cite{yang-2023-llmasoptimizer, Guo2023ConnectingLL} presents the current state-of-the-art in automated prompt design, framing prompt refinement as an optimization challenge. This approach iteratively refines prompts to maximize task accuracy, ceasing when performance plateaus or iteration limits are met. &#10;&#10;The motivation for OPRO is based on the LLMs' self-optimization ability. However, our empirical results reveal that smaller-scale LLMs like LLaMa-2~\cite{Touvron2023Llama2O} do not have sufficient ability to support the self-optimization. We demonstrate that such optimization strategies offer marginal benefits for smaller-scale LLMs, demanding considerable computational resources for slight performance gains, particularly when contrasted with zero-shot CoT prompts. We summarize our contributions as follows:&#10;\begin{itemize} [label=$\bullet$]&#10; \item We demonstrate that the limited inference abilities of small-scale LLMs, such as LLaMa-2 family and Mistral 7B, restrict their self-optimization efficiency, rendering OPRO ineffective for these models. (Section \ref{sec:mot}, \ref{sec:discuss}).&#10; \item Our findings reveal OPRO's substantial reliance on manual prompt design in small-scale LLMs, suggesting that its automation advantage is minimal compared to traditional manual prompting efforts. (Section \ref{sec:discuss})&#10; \item Based on empirical evidence and analysis, we recommend future prompt engineering efforts to account for the inference limitations of small-scale LLMs and consider traditional CoT prompts as effective, adaptive, and resource-efficient baselines. (Section \ref{sec:main_results}, \ref{sec:discuss})&#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Prompt Engineering Strategies, Artificial Intelligence, Mathematics, Large Language Model Limitations, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2103.10385" label="2103.10385">
        <attvalues>
          <attvalue for="0" value="GPT Understands, Too" />
          <attvalue for="1" value="  Prompting a pretrained language model with natural language patterns has been&#10;proved effective for natural language understanding (NLU). However, our&#10;preliminary study reveals that manual discrete prompts often lead to unstable&#10;performance -- e.g., changing a single word in the prompt might result in&#10;substantial performance drop. We propose a novel method P-Tuning that employs&#10;trainable continuous prompt embeddings in concatenation with discrete prompts.&#10;Empirically, P-Tuning not only stabilizes training by minimizing the gap&#10;between various discrete prompts, but also improves performance by a sizeable&#10;margin on a wide range of NLU tasks including LAMA and SuperGLUE. P-Tuning is&#10;generally effective for both frozen and tuned language models, under both the&#10;fully-supervised and few-shot settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;\vpara{Language Model Prompting.}&#10;GPT-3 \cite{brown2020language} uses in-context examples \cite{liu2021makes,zhao2021calibrate} as a way of prompting to transfer knowledge from pretraining to downstream tasks. \cite{schick2020small} proposed to use cloze patterns, which removes the constraint that the masked token is the last token of the sentence. This further minimizes the gap between pretraining and downstream tasks. To improve prompting for NLU, recent works have proposed methods to automatically search for high-performing prompts by mining the training corpus \cite{jiang2020can}, gradient-based search \cite{shin2020autoprompt}, or using pretrained generative models \cite{gao2020making}. Our approach is different from these prior works in that we resort to using continuous prompt embeddings, which are found to be complementary to discrete prompts in our experiments.&#10;&#10;Recently, some concurrent works also proposed the use of continuous prompts. Prefix-tuning ~\cite{li2021prefix} adds continuous prompts at the beginning of the sequence for each layer. In contrast to our work, prefix-tuning targets natural language generation tasks.&#10;&#10;In the area of NLU, a few concurrent methods were proposed based on continuous prompts, focusing on improving knowledge probing \cite{Qin2021LearningHT,Zhong2021FactualPI}. \cite{Lester2021ThePO} showed that with large pretrained models, only tuning continuous prompts with a frozen language model achieves comparable performance to full-model tuning.&#10;&#10;Compared to these concurrent works on NLU,&#10;P-Tuning reaches a unique conclusion that continuous prompts improve performance and stabilize training with either frozen or tuned models under both the few-shot and fully-supervised settings. For example, no concurrent works have shown that continuous prompts can improve performance with a tuned language model. Technically, P-Tuning also has a few unique designs such as using hybrid continuous-discrete prompts and employing a prompt encoder.&#10;&#10;\vpara{Knowledge in Language Models.}&#10;Self-supervised~\cite{liu2020self} pre-trained language models~\cite{han2021pre} including GPT~\cite{radford2019language}, BERT~\cite{devlin2018bert}, XLNet~\cite{yang2019xlnet}, RoBERTa~\cite{liu2019roberta} have been observed to learn not only contextualized text representations but also linguistic and world knowledge. &#10;~\cite{hewitt2019structural} demonstrates that contextualized representations produced by language models can form a parse tree in the embedding space. ~\cite{vig2019multiscale, clark2019does} look into the multi-head attention patterns within transformers and discover that certain attention heads may correspond to some grammatical functions, including co-reference and noun modifiers.&#10;LAMA~\cite{petroni2019language,petroni2020context} propose the LAMA task that leverages cloze tests to predict the fact triples of knowledge bases to examine language model's ability of memorizing facts with answers in the single-token format. In~\cite{wang2020language}, the authors investigate the attention matrices to find evidence about knowledge triples contained in the context. ~\cite{jiang2020x} develops a multi-token fact retrieval dataset based on LAMA.&#10; " />
        </attvalues>
      </node>
      <node id="2211.01910" label="2211.01910">
        <attvalues>
          <attvalue for="0" value="Large Language Models Are Human-Level Prompt Engineers" />
          <attvalue for="1" value="  By conditioning on natural language instructions, large language models&#10;(LLMs) have displayed impressive capabilities as general-purpose computers.&#10;However, task performance depends significantly on the quality of the prompt&#10;used to steer the model, and most effective prompts have been handcrafted by&#10;humans. Inspired by classical program synthesis and the human approach to&#10;prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic&#10;instruction generation and selection. In our method, we treat the instruction&#10;as the &quot;program,&quot; optimized by searching over a pool of instruction candidates&#10;proposed by an LLM in order to maximize a chosen score function. To evaluate&#10;the quality of the selected instruction, we evaluate the zero-shot performance&#10;of another LLM following the selected instruction. Experiments on 24 NLP tasks&#10;show that our automatically generated instructions outperform the prior LLM&#10;baseline by a large margin and achieve better or comparable performance to the&#10;instructions generated by human annotators on 19/24 tasks. We conduct extensive&#10;qualitative and quantitative analyses to explore the performance of APE. We&#10;show that APE-engineered prompts can be applied to steer models toward&#10;truthfulness and/or informativeness, as well as to improve few-shot learning&#10;performance by simply prepending them to standard in-context learning prompts.&#10;Please check out our webpage at&#10;https://sites.google.com/view/automatic-prompt-engineer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.03495" label="2305.03495">
        <attvalues>
          <attvalue for="0" value="Automatic Prompt Optimization with &quot;Gradient Descent&quot; and Beam Search" />
          <attvalue for="1" value="  Large Language Models (LLMs) have shown impressive performance as general&#10;purpose agents, but their abilities remain highly dependent on prompts which&#10;are hand written with onerous trial-and-error effort. We propose a simple and&#10;nonparametric solution to this problem, Automatic Prompt Optimization (APO),&#10;which is inspired by numerical gradient descent to automatically improve&#10;prompts, assuming access to training data and an LLM API. The algorithm uses&#10;minibatches of data to form natural language &quot;gradients&quot; that criticize the&#10;current prompt. The gradients are then &quot;propagated&quot; into the prompt by editing&#10;the prompt in the opposite semantic direction of the gradient. These gradient&#10;descent steps are guided by a beam search and bandit selection procedure which&#10;significantly improves algorithmic efficiency. Preliminary results across three&#10;benchmark NLP tasks and the novel problem of LLM jailbreak detection suggest&#10;that Automatic Prompt Optimization can outperform prior prompt editing&#10;techniques and improve an initial prompt's performance by up to 31%, by using&#10;data to rewrite vague task descriptions into more precise annotation&#10;instructions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.03409" label="2309.03409">
        <attvalues>
          <attvalue for="0" value="Large Language Models as Optimizers" />
          <attvalue for="1" value="  Optimization is ubiquitous. While derivative-based algorithms have been&#10;powerful tools for various problems, the absence of gradient imposes challenges&#10;on many real-world applications. In this work, we propose Optimization by&#10;PROmpting (OPRO), a simple and effective approach to leverage large language&#10;models (LLMs) as optimizers, where the optimization task is described in&#10;natural language. In each optimization step, the LLM generates new solutions&#10;from the prompt that contains previously generated solutions with their values,&#10;then the new solutions are evaluated and added to the prompt for the next&#10;optimization step. We first showcase OPRO on linear regression and traveling&#10;salesman problems, then move on to our main application in prompt optimization,&#10;where the goal is to find instructions that maximize the task accuracy. With a&#10;variety of LLMs, we demonstrate that the best prompts optimized by OPRO&#10;outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on&#10;Big-Bench Hard tasks. Code at https://github.com/google-deepmind/opro.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.02038" label="2401.02038">
        <attvalues>
          <attvalue for="0" value="Understanding LLMs: A Comprehensive Overview from Training to Inference" />
          <attvalue for="1" value="  The introduction of ChatGPT has led to a significant increase in the&#10;utilization of Large Language Models (LLMs) for addressing downstream tasks.&#10;There's an increasing focus on cost-efficient training and deployment within&#10;this context. Low-cost training and deployment of LLMs represent the future&#10;development trend. This paper reviews the evolution of large language model&#10;training techniques and inference deployment technologies aligned with this&#10;emerging trend. The discussion on training includes various aspects, including&#10;data preprocessing, training architecture, pre-training tasks, parallel&#10;training, and relevant content related to model fine-tuning. On the inference&#10;side, the paper covers topics such as model compression, parallel computation,&#10;memory scheduling, and structural optimization. It also explores LLMs'&#10;utilization and provides insights into their future development.&#10;" />
          <attvalue for="2" value="&#10;Language modeling (LM) is a fundamental approach for achieving cognitive intelligence in the field of natural language processing (NLP), and its progress has been notable in recent years \cite{liu2023summary,wang2023prompt,zhao2023survey}. It assumes a central role in understanding, generating, and manipulating human language, serving as the cornerstone for a diverse range of NLP applications \cite{kaddour2023challenges}, including machine translation, chatbots, sentiment analysis, and text summarization. With the evolution of deep learning, the early statistical language models (SLM) have gradually transformed into neural language models (NLM) based on neural networks. This shift is characterized by the adoption of word embeddings, representing words as distributed vectors. Notably, these word embeddings have consistently excelled in practical NLP tasks, profoundly shaping the field's progress. Pre-trained language models (PLM) represent a subsequent phase in the evolution of language models following NLM. Early attempts at PLMs included ELMo \cite{ELMo}, which was built on a Bidirectional LSTM architecture. However, with the advent of the transformer architecture \cite{transformer}, characterized by parallel self-attention mechanisms, the pre-training and fine-tuning learning paradigm has propelled PLM to prominence as the prevailing approach. These models are typically trained via self-supervision on extensive datasets, cementing their status as the primary methodology in the field. &#10;&#10;The Transformer architecture is exceptionally well-suited for scaling up models, and research analysis has revealed that increasing the model's scale or training data size can significantly enhance its performance. Many studies have pushed the boundaries of model performance by continuously expanding the scale of PLM \cite{GPT-2,GPT3,touvron2023llama,touvron2023llama2}. As models grow larger, a remarkable phenomenon known as &quot;emergence&quot; occurs, wherein they exhibit astonishing performance \cite{GPT3}. These models are capable of generating high-quality text and possess robust learning and reasoning abilities. They can even tackle few-shot learning tasks through in-context learning (ICL) \cite{GPT3}. This remarkable capability enables their seamless application to a wide range of downstream tasks across diverse domains \cite{10.1007/978-3-031-21014-3_28,PMID:36097765,liao2023maskguided,rezayi2023exploring}. &#10;&#10;Pre-trained language models (PLMs) with significantly larger parameter sizes and extensive training data are typically denoted as Large Language Models (LLMs) \cite{zhong2023chatradiovaluer,liu2023evaluating,zhong2023chatabl}. The model size usually exceeds 6-10 billion (6-10B) parameters. A prominent milestone in the development of LLMs is exemplified by the GPT series \cite{GPT,GPT-2,GPT3,openai2023gpt4}. Notably, OpenAI released ChatGPT in November 2022, marking a pivotal moment in the era of LLMs and a game-changing moment in the field of artificial intelligence. ChatGPT has empowered current AI algorithms to achieve unprecedented levels of strength and effectiveness, reshaping the way humans employ or develop AI algorithms. Its emergence has captured the attention of the research community. However, owing to ChatGPT's absence as an open-source platform, the principal way to use ChatGPT currently is by accessing it through OpenAI's website at or via their API interface. Training LLMs that can serve as alternatives to ChatGPT, or domain-specific LLMs, has become highly necessary \cite{dai2023auggpt,liu2023deidgpt,ma2023impressiongpt,liao2023differentiate,dai2023adautogpt,liu2023summary,guan2023cohortgpt,liu2023pharmacygpt}. Training and deploying LLMs demand expertise in handling large-scale data and substantial practical experience in distributed parallel training \cite{wei2023chat2brain,10.1007/978-3-031-43907-0_40,10.1093/psyrad/kkad011}. This requirement emphasizes the need for researchers developing LLMs to possess significant engineering capabilities in addressing the challenges encountered during LLM development. Researchers who are interested in the field of LLMs must possess engineering skills or learn to collaborate effectively with engineers.&#10;&#10;For the above reasons, the primary objective of this paper is to provide a comprehensive overview of LLMs training and inference techniques to equip researchers with the knowledge required for developing, deploying, and applying LLMs. The structure of the rest of this review is as follows: In Section~\ref{sec::background}, we will introduce the relevant background and foundational knowledge of LLMs. In Section~\ref{sec::training}, we will delve into the technical aspects of training LLMs, while in Section~\ref{sec::inference} we will explore the technologies related to LLM's inference and deployment. In Section~\ref{sec::app}, we will discuss the utilization of LLMs, and Section~\ref{sec::future} will explore the future directions and their implications for LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Inference Deployment, Computer Science, Large Language Models, Information Technology, Linguistics, Training Techniques" />
        </attvalues>
      </node>
      <node id="2304.01852" label="2304.01852">
        <attvalues>
          <attvalue for="0" value="Summary of ChatGPT-Related Research and Perspective Towards the Future&#10;  of Large Language Models" />
          <attvalue for="1" value="  This paper presents a comprehensive survey of ChatGPT-related (GPT-3.5 and&#10;GPT-4) research, state-of-the-art large language models (LLM) from the GPT&#10;series, and their prospective applications across diverse domains. Indeed, key&#10;innovations such as large-scale pre-training that captures knowledge across the&#10;entire world wide web, instruction fine-tuning and Reinforcement Learning from&#10;Human Feedback (RLHF) have played significant roles in enhancing LLMs'&#10;adaptability and performance. We performed an in-depth analysis of 194 relevant&#10;papers on arXiv, encompassing trend analysis, word cloud representation, and&#10;distribution analysis across various application domains. The findings reveal a&#10;significant and increasing interest in ChatGPT-related research, predominantly&#10;centered on direct natural language processing applications, while also&#10;demonstrating considerable potential in areas ranging from education and&#10;history to mathematics, medicine, and physics. This study endeavors to furnish&#10;insights into ChatGPT's capabilities, potential implications, ethical concerns,&#10;and offer direction for future advancements in this field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.11107" label="2304.11107">
        <attvalues>
          <attvalue for="0" value="ChatABL: Abductive Learning via Natural Language Interaction with&#10;  ChatGPT" />
          <attvalue for="1" value="  Large language models (LLMs) such as ChatGPT have recently demonstrated&#10;significant potential in mathematical abilities, providing valuable reasoning&#10;paradigm consistent with human natural language. However, LLMs currently have&#10;difficulty in bridging perception, language understanding and reasoning&#10;capabilities due to incompatibility of the underlying information flow among&#10;them, making it challenging to accomplish tasks autonomously. On the other&#10;hand, abductive learning (ABL) frameworks for integrating the two abilities of&#10;perception and reasoning has seen significant success in inverse decipherment&#10;of incomplete facts, but it is limited by the lack of semantic understanding of&#10;logical reasoning rules and the dependence on complicated domain knowledge&#10;representation. This paper presents a novel method (ChatABL) for integrating&#10;LLMs into the ABL framework, aiming at unifying the three abilities in a more&#10;user-friendly and understandable manner. The proposed method uses the strengths&#10;of LLMs' understanding and logical reasoning to correct the incomplete logical&#10;facts for optimizing the performance of perceptual module, by summarizing and&#10;reorganizing reasoning rules represented in natural language format. Similarly,&#10;perceptual module provides necessary reasoning examples for LLMs in natural&#10;language format. The variable-length handwritten equation deciphering task, an&#10;abstract expression of the Mayan calendar decoding, is used as a testbed to&#10;demonstrate that ChatABL has reasoning ability beyond most existing&#10;state-of-the-art methods, which has been well supported by comparative studies.&#10;To our best knowledge, the proposed ChatABL is the first attempt to explore a&#10;new pattern for further approaching human-level cognitive ability via natural&#10;language interaction with ChatGPT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.11567" label="2304.11567">
        <attvalues>
          <attvalue for="0" value="Differentiate ChatGPT-generated and Human-written Medical Texts" />
          <attvalue for="1" value="  Background: Large language models such as ChatGPT are capable of generating&#10;grammatically perfect and human-like text content, and a large number of&#10;ChatGPT-generated texts have appeared on the Internet. However, medical texts&#10;such as clinical notes and diagnoses require rigorous validation, and erroneous&#10;medical content generated by ChatGPT could potentially lead to disinformation&#10;that poses significant harm to healthcare and the general public.&#10;  Objective: This research is among the first studies on responsible and&#10;ethical AIGC (Artificial Intelligence Generated Content) in medicine. We focus&#10;on analyzing the differences between medical texts written by human experts and&#10;generated by ChatGPT, and designing machine learning workflows to effectively&#10;detect and differentiate medical texts generated by ChatGPT.&#10;  Methods: We first construct a suite of datasets containing medical texts&#10;written by human experts and generated by ChatGPT. In the next step, we analyze&#10;the linguistic features of these two types of content and uncover differences&#10;in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc. Finally,&#10;we design and implement machine learning methods to detect medical text&#10;generated by ChatGPT.&#10;  Results: Medical texts written by humans are more concrete, more diverse, and&#10;typically contain more useful information, while medical texts generated by&#10;ChatGPT pay more attention to fluency and logic, and usually express general&#10;terminologies rather than effective information specific to the context of the&#10;problem. A BERT-based model can effectively detect medical texts generated by&#10;ChatGPT, and the F1 exceeds 95%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.10095" label="2306.10095">
        <attvalues>
          <attvalue for="0" value="AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiology" />
          <attvalue for="1" value="  In this pioneering study, inspired by AutoGPT, the state-of-the-art&#10;open-source application based on the GPT-4 large language model, we develop a&#10;novel tool called AD-AutoGPT which can conduct data collection, processing, and&#10;analysis about complex health narratives of Alzheimer's Disease in an&#10;autonomous manner via users' textual prompts. We collated comprehensive data&#10;from a variety of news sources, including the Alzheimer's Association, BBC,&#10;Mayo Clinic, and the National Institute on Aging since June 2022, leading to&#10;the autonomous execution of robust trend analyses, intertopic distance maps&#10;visualization, and identification of salient terms pertinent to Alzheimer's&#10;Disease. This approach has yielded not only a quantifiable metric of relevant&#10;discourse but also valuable insights into public focus on Alzheimer's Disease.&#10;This application of AD-AutoGPT in public health signifies the transformative&#10;potential of AI in facilitating a data-rich understanding of complex health&#10;narratives like Alzheimer's Disease in an autonomous manner, setting the&#10;groundwork for future AI-driven investigations in global health landscapes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.11346" label="2307.11346">
        <attvalues>
          <attvalue for="0" value="CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study" />
          <attvalue for="1" value="  Participant recruitment based on unstructured medical texts such as clinical&#10;notes and radiology reports has been a challenging yet important task for the&#10;cohort establishment in clinical research. Recently, Large Language Models&#10;(LLMs) such as ChatGPT have achieved tremendous success in various downstream&#10;tasks thanks to their promising performance in language understanding,&#10;inference, and generation. It is then natural to test their feasibility in&#10;solving the cohort recruitment task, which involves the classification of a&#10;given paragraph of medical text into disease label(s). However, when applied to&#10;knowledge-intensive problem settings such as medical text classification, where&#10;the LLMs are expected to understand the decision made by human experts and&#10;accurately identify the implied disease labels, the LLMs show a mediocre&#10;performance. A possible explanation is that, by only using the medical text,&#10;the LLMs neglect to use the rich context of additional information that&#10;languages afford. To this end, we propose to use a knowledge graph as auxiliary&#10;information to guide the LLMs in making predictions. Moreover, to further boost&#10;the LLMs adapt to the problem setting, we apply a chain-of-thought (CoT) sample&#10;selection strategy enhanced by reinforcement learning, which selects a set of&#10;CoT samples given each individual medical report. Experimental results and&#10;various ablation studies show that our few-shot learning method achieves&#10;satisfactory performance compared with fine-tuning strategies and gains superb&#10;advantages when the available data is limited. The code and sample dataset of&#10;the proposed CohortGPT model is available at:&#10;https://anonymous.4open.science/r/CohortGPT-4872/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.10432" label="2307.10432">
        <attvalues>
          <attvalue for="0" value="PharmacyGPT: The AI Pharmacist" />
          <attvalue for="1" value="  In this study, we introduce PharmacyGPT, a novel framework to assess the&#10;capabilities of large language models (LLMs) such as ChatGPT and GPT-4 in&#10;emulating the role of clinical pharmacists. Our methodology encompasses the&#10;utilization of LLMs to generate comprehensible patient clusters, formulate&#10;medication plans, and forecast patient outcomes. We conduct our investigation&#10;using real data acquired from the intensive care unit (ICU) at the University&#10;of North Carolina Chapel Hill (UNC) Hospital. Our analysis offers valuable&#10;insights into the potential applications and limitations of LLMs in the field&#10;of clinical pharmacy, with implications for both patient care and the&#10;development of future AI-driven healthcare solutions. By evaluating the&#10;performance of PharmacyGPT, we aim to contribute to the ongoing discourse&#10;surrounding the integration of artificial intelligence in healthcare settings,&#10;ultimately promoting the responsible and efficacious use of such technologies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.04691" label="2312.04691">
        <attvalues>
          <attvalue for="0" value="Simul-LLM: A Framework for Exploring High-Quality Simultaneous&#10;  Translation with Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) with billions of parameters and pretrained on&#10;massive amounts of data are now capable of near or better than state-of-the-art&#10;performance in a variety of downstream natural language processing tasks.&#10;Neural machine translation (NMT) is one such task that LLMs have been applied&#10;to with great success. However, little research has focused on applying LLMs to&#10;the more difficult subset of NMT called simultaneous translation (SimulMT),&#10;where translation begins before the entire source context is available to the&#10;model. In this paper, we address key challenges facing LLMs fine-tuned for&#10;SimulMT, validate classical SimulMT concepts and practices in the context of&#10;LLMs, explore adapting LLMs that are fine-tuned for NMT to the task of SimulMT,&#10;and introduce Simul-LLM, the first open-source fine-tuning and evaluation&#10;pipeline development framework for LLMs focused on SimulMT.&#10;" />
          <attvalue for="2" value="&#10;Modern large language models (LLMs) contain at least several billion and up to trillions of parameters and are remarkably capable across a wide range of tasks. Pretrained on humongous amounts of unlabeled data, they have demonstrated incredible emergent capabilities. With minor prompt adjustments, such as including instructions and examples, LLMs are often capable of near state-of-the-art performance set by highly customized solutions. The performance of these models is further enhanced when fine-tuned for specialized downstream tasks, sometimes exceeding the performance of previously cutting-edge solutions. Given their rapidly evolving capabilities, LLMs and their application have become a focused topic of research within NLP academia \cite{zhao2023survey}.&#10;&#10;One popular downstream task for LLMs is text-to-text neural machine translation (NMT), which focuses on taking an input sequence in a given language and outputting a translation in another language. Typically, the entire source context is available at the start of translation for NMT. A particularly challenging subset of NMT is known as simultaneous translation (SimulMT), where the model begins translation without having access to the entire source sequence, and the translation progresses as the remaining source sequence is incrementally provided. For languages that are syntactically and structurally similar, near-NMT performance is fairly achievable, but for language pairs that differ significantly in structure, traditional models struggle to balance high-quality translations with delay for additional source context. This balance is typically achieved via a fixed or adaptive read-write schedule, with one of the most popular and longstanding fixed schedules being the wait-k policy \cite{stacl2019}, where the target translation hypothesis lags behind the incrementally available source sequence by k words or subwords. &#10;&#10;While LLMs have been applied to and studied actively in NMT, their application to simultaneous translation has been lagging. This is in part due to a few challenges LLMs face when applied to SimulMT that are non-trivial to address. First and foremost, it is unclear how well LLMs, which are pretrained and usually fine-tuned under the assumption that the prompt is completely provided and static before generation, will adapt to an application space where the prompt dynamically changes as the simultaneous scheduler elects to read from the source sequence. Second, multiple approaches exist to enable LLMs for SimulMT and it is challenging to intuit which approach will perform best. For example, one could adapt LLMs fine-tuned for NMT (hereafter referred to as NMT LLMs) to SimulMT during inference, although how well such models will deal with the source context availability mismatch between fine-tuning (full sentence) and inference (partial sentence) is nebulous. Alternatively, one could fine-tune LLMs directly for SimulMT (hereafter referred to as SimulMT LLMs), but new prompt structuring is likely needed to match inference SimulMT behavior during fine-tuning exactly. Finally, it is also unclear how well previously understood concepts in existing SimulMT work, such as higher fine-tuning wait-k values increasing generalizability, will apply to SimulMT LLMs.&#10;&#10;This paper seeks to address the above problems and contributes to the process of applying LLMs to SimulMT in the following major ways: &#10;&#10;\begin{itemize}&#10; \item We develop Simul-LLM, the first open-source fine-tuning and evaluation pipeline development framework for SimulMT LLMs, which seamlessly wraps around and interfaces with popular libraries for LLMs and SimulMT. This framework serves as a foundation for research on SimulMT LLMs that the community can employ and extend for a wide range of future work on LLM-based simultaneous translation.&#10; \item With the aforementioned framework, we explore the feasibility of adapting LLMs fine-tuned for NMT to SimulMT under a few decoding strategies and the classical wait-k fixed translation scheduler. Generally, we find that NMT LLMs demonstrate good performance during SimulMT inference which can be somewhat boosted by more complex decoding strategies.&#10; \item We propose an alternative prompt structuring approach to commonly employed NMT prompts that bridges the gap between the fine-tuning and inference environment, assuming a wait-k schedule, and we validate this via the Simul-LLM framework. We elaborate on counter-intuitive results that we observe and provide a base of exploration for future research to employ. Along these lines, we also validate that higher wait-k values employed during SimulMT fine-tuning do increase wait-k generalizability and boost translation quality across the board during SimulMT inference.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Simultaneous Translation, Artificial Intelligence, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1810.08398" label="1810.08398">
        <attvalues>
          <attvalue for="0" value="STACL: Simultaneous Translation with Implicit Anticipation and&#10;  Controllable Latency using Prefix-to-Prefix Framework" />
          <attvalue for="1" value="  Simultaneous translation, which translates sentences before they are&#10;finished, is useful in many scenarios but is notoriously difficult due to&#10;word-order differences. While the conventional seq-to-seq framework is only&#10;suitable for full-sentence translation, we propose a novel prefix-to-prefix&#10;framework for simultaneous translation that implicitly learns to anticipate in&#10;a single translation model. Within this framework, we present a very simple yet&#10;surprisingly effective wait-k policy trained to generate the target sentence&#10;concurrently with the source sentence, but always k words behind. Experiments&#10;show our strategy achieves low latency and reasonable quality (compared to&#10;full-sentence translation) on 4 directions: zh&lt;-&gt;en and de&lt;-&gt;en.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.19349" label="2310.19349">
        <attvalues>
          <attvalue for="0" value="Japanese SimCSE Technical Report" />
          <attvalue for="1" value="  We report the development of Japanese SimCSE, Japanese sentence embedding&#10;models fine-tuned with SimCSE. Since there is a lack of sentence embedding&#10;models for Japanese that can be used as a baseline in sentence embedding&#10;research, we conducted extensive experiments on Japanese sentence embeddings&#10;involving 24 pre-trained Japanese or multilingual language models, five&#10;supervised datasets, and four unsupervised datasets. In this report, we provide&#10;the detailed training setup for Japanese SimCSE and their evaluation results.&#10;" />
          <attvalue for="2" value="&#10;&#10;Sentence embeddings provide dense vector representations of natural language sentences and have gained traction in tasks such as retrieval, question answering, and, more recently, Retrieval Augmented Generation (RAG).&#10;Although various methods exist to produce sentence embeddings, recent approaches have shown promise by fine-tuning pre-trained language models using contrastive learning.&#10;Among them, SimCSE~\cite{SimCSE} is a pioneering work of contrastive sentence embeddings and offers techniques for both unsupervised and supervised settings.&#10;In the unsupervised setting, SimCSE leverages Dropout in pre-trained language models as a data augmentation technique, processing the same sentence twice through the model. It then treats pairs of embeddings from the same sentence as positive samples for contrastive learning. In the supervised approach, it utilizes the Natural Language Inference (NLI) dataset, such as the Stanford NLI (SNLI) dataset~\cite{SNLI} and the Multi-Genre NLI (MNLI) dataset~\cite{MNLI}, to treat semantically similar sentences as positive samples for contrastive learning.&#10;Additionally, to further emphasize generating embeddings that capture differences in meaning, SimCSE uses sentence pairs labeled as contradictions in the NLI dataset as hard negatives.&#10;&#10;SimCSE has arguably become the de facto standard for sentence embeddings, demonstrating wide-ranging and impressive performance and leading to numerous derivative studies~\cite{DiffCSE,PromptBERT,PromptEOL}.&#10;However, many of these studies focus on English, with a lack of comprehensive research on Japanese sentence embeddings.&#10;&#10;In this report, we present extensive experiments using various Japanese or multilingual pre-trained language models, training datasets, and hyperparameters to perform a thorough evaluation of Japanese SimCSE.&#10;Additionally, we release four pre-trained Japanese sentence embedding models fine-tuned under promising settings and present their evaluation results to encourage further research.&#10;Our models, detailed results, and codebases are publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Computational Linguistics, Japanese Language Processing, Artificial Intelligence, Sentence Embedding Models, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1704.05426" label="1704.05426">
        <attvalues>
          <attvalue for="0" value="A Broad-Coverage Challenge Corpus for Sentence Understanding through&#10;  Inference" />
          <attvalue for="1" value="  This paper introduces the Multi-Genre Natural Language Inference (MultiNLI)&#10;corpus, a dataset designed for use in the development and evaluation of machine&#10;learning models for sentence understanding. In addition to being one of the&#10;largest corpora available for the task of NLI, at 433k examples, this corpus&#10;improves upon available resources in its coverage: it offers data from ten&#10;distinct genres of written and spoken English--making it possible to evaluate&#10;systems on nearly the full complexity of the language--and it offers an&#10;explicit setting for the evaluation of cross-genre domain adaptation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.10298" label="2204.10298">
        <attvalues>
          <attvalue for="0" value="DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings" />
          <attvalue for="1" value="  We propose DiffCSE, an unsupervised contrastive learning framework for&#10;learning sentence embeddings. DiffCSE learns sentence embeddings that are&#10;sensitive to the difference between the original sentence and an edited&#10;sentence, where the edited sentence is obtained by stochastically masking out&#10;the original sentence and then sampling from a masked language model. We show&#10;that DiffSCE is an instance of equivariant contrastive learning (Dangovski et&#10;al., 2021), which generalizes contrastive learning and learns representations&#10;that are insensitive to certain types of augmentations and sensitive to other&#10;&quot;harmful&quot; types of augmentations. Our experiments show that DiffCSE achieves&#10;state-of-the-art results among unsupervised sentence representation learning&#10;methods, outperforming unsupervised SimCSE by 2.3 absolute points on semantic&#10;textual similarity tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.04337" label="2201.04337">
        <attvalues>
          <attvalue for="0" value="PromptBERT: Improving BERT Sentence Embeddings with Prompts" />
          <attvalue for="1" value="  We propose PromptBERT, a novel contrastive learning method for learning&#10;better sentence representation. We firstly analyze the drawback of current&#10;sentence embedding from original BERT and find that it is mainly due to the&#10;static token embedding bias and ineffective BERT layers. Then we propose the&#10;first prompt-based sentence embeddings method and discuss two prompt&#10;representing methods and three prompt searching methods to make BERT achieve&#10;better sentence embeddings. Moreover, we propose a novel unsupervised training&#10;objective by the technology of template denoising, which substantially shortens&#10;the performance gap between the supervised and unsupervised settings. Extensive&#10;experiments show the effectiveness of our method. Compared to SimCSE,&#10;PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and&#10;RoBERTa in the unsupervised setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10; Learning sentence embeddings as a fundamental NLP problem has been largely studied. Currently, how to leverage the power of BERT in sentence embeddings has become a new trend. Many works~\cite{li2020sentence, gao2021simcse} achieved strong performance with BERT in both supervised and unsupervised settings.&#10; Among these works, contrastive learning based methods achieve state-of-the-art results.&#10; \cite{gao2021simcse}~(\cite{gao2021simcse}) proposed a novel contrastive training objective to directly use inner dropout as noise to construct positive pairs. \cite{yan2021consert}~(\cite{yan2021consert}) discussed four methods to construct positive pairs.&#10;&#10;Although BERT achieved success in sentence embeddings, original BERT shows unsatisfactory performance~\cite{reimers2019sentence, li2020sentence}.&#10; One explanation is the anisotropy in original BERT, which causes sentence pairs to have high similarity, some works~\cite{li2020sentence, su2021whitening} focused on reducing the anisotropy by post-processing sentence embeddings.&#10;&#10;\import{sections/}{findings_v2}&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2307.16645" label="2307.16645">
        <attvalues>
          <attvalue for="0" value="Scaling Sentence Embeddings with Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have recently garnered significant interest.&#10;With in-context learning, LLMs achieve impressive results in various natural&#10;language tasks. However, the application of LLMs to sentence embeddings remains&#10;an area of ongoing research. In this work, we propose an in-context&#10;learning-based method aimed at improving sentence embeddings performance. Our&#10;approach involves adapting the previous prompt-based representation method for&#10;autoregressive models, constructing a demonstration set that enables LLMs to&#10;perform in-context learning, and scaling up the LLMs to different model sizes.&#10;Through extensive experiments, in-context learning enables LLMs to generate&#10;high-quality sentence embeddings without any fine-tuning. It helps LLMs achieve&#10;performance comparable to current contrastive learning methods. By scaling&#10;model size, we find scaling to more than tens of billion parameters harms the&#10;performance on semantic textual similarity (STS) tasks. However, the largest&#10;model outperforms other counterparts and achieves the new state-of-the-art&#10;result on transfer tasks. We also fine-tune LLMs with current contrastive&#10;learning approach, and the 2.7B OPT model, incorporating our prompt-based&#10;method, surpasses the performance of 4.8B ST5, achieving the new&#10;state-of-the-art results on STS tasks. Our code is available at&#10;https://github.com/kongds/scaling_sentemb.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.13486" label="2310.13486">
        <attvalues>
          <attvalue for="0" value="Mind the instructions: a holistic evaluation of consistency and&#10;  interactions in prompt-based learning" />
          <attvalue for="1" value="  Finding the best way of adapting pre-trained language models to a task is a&#10;big challenge in current NLP. Just like the previous generation of task-tuned&#10;models (TT), models that are adapted to tasks via in-context-learning (ICL) are&#10;robust in some setups but not in others. Here, we present a detailed analysis&#10;of which design choices cause instabilities and inconsistencies in LLM&#10;predictions. First, we show how spurious correlations between input&#10;distributions and labels -- a known issue in TT models -- form only a minor&#10;problem for prompted models. Then, we engage in a systematic, holistic&#10;evaluation of different factors that have been found to influence predictions&#10;in a prompting setup. We test all possible combinations of a range of factors&#10;on both vanilla and instruction-tuned (IT) LLMs of different scale and&#10;statistically analyse the results to show which factors are the most&#10;influential, interactive or stable. Our results show which factors can be used&#10;without precautions and which should be avoided or handled with care in most&#10;settings.&#10;" />
          <attvalue for="2" value="&#10;Transfer learning from large-scale pre-trained language models is nowadays the standard approach to a wide range of NLP tasks. One of its great challenges is to optimally interface information that pre-trained language models accumulate in their parameters and adapt it to the task of interest \cite{zhou2023lima, ouyang2022training}. The standard approach for task adaptation has recently shifted from updating model parameters for a specific task (from here on task tuning or TT) to using prompting-based methods based on in-context learning (from here on ICL).&#10;ICL can be subdivided into few-shot \cite[][]{brown2020language} or zero-shot inference \cite[primarily using instruction-tuned models ][]{wei2021finetuned}.&#10;Both approaches offer certain benefits over TT: it eliminates costly, task-specific finetuning and provides greater flexibility, as a single model can be applied to many tasks.&#10;However, ICL also currently yields overall weaker performance compared to task-tuning and is less stable and reliable on many benchmarks \cite[see, e.g.][]{bang2023multitask, ohmer2023evaluating, min2022rethinking,lu2021fantastically,zhao2021calibrate}. &#10;&#10;While for TT, much research has been conducted to understand weaknesses in the paradigm \cite[for an overview, see ][]{hupkes2023taxonomy}, the sources of instabilities in ICL remain nebulous.&#10;Since ICL is more constrained (less data and no parameter updates), out-of-distribution generalisation has been suggested to be less of a problem \cite{awadalla2022exploring, si2023prompting}.&#10;On the other hand, new frontiers emerge. &#10;For example, the format, order, or semantics of provided in-context examples can greatly influence learning outcomes, as does the proportion of labels in the context and the exact labels used \cite{liang2022holistic}.&#10;Little is known, however, about how these factors interact \cite[work from ][suggests that they cannot be isolated]{wei2023larger, kim2022ground}; it is unclear which aspects are consistently beneficial, which vary across setups, and which are sensible to combine or decouple.&#10;The volatility of the paradigm warrants more research into the reliability of different design choices.&#10;&#10;In this paper, we conduct a detailed exploration of vanilla and instruction-tuned LLMs across various shifts and setups to understand their robustness.&#10;We start with one of the prominent themes in robustness studies for TT models: robustness to spurious correlations between input and label distributions \cite{kavumba2019choosing, mccoy2019right, niven2019probing} and find that in ICL, spurious correlations do not have a significant impact on learning outcomes.&#10;&#10;We go on to investigate ICL's sensitivity to other features of adaptation context, as well as the consistency of predictions across different design choices.&#10;To do so, we conduct a large-scale grid search across various combinations of factors and statistically analyse the results to shed light on the inter-dependencies of different design choices.&#10;We find that the exact in-context setup (the number of in-context examples, the distribution of in-context labels, or the type of instructions given in the context) has a surprisingly small but reliable impact on prediction outcomes.&#10;On the other hand, the type of instructions used to query the target has, by far, the most significant impact on model behaviour. It is also the most volatile across settings, making it the most pivotal factor.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Prompting Evaluation Methods, Computer Science, Machine Learning, Linguistics, Task Tuning Instabilities, Artificial Intelligence, Natural Language Processing, Statistics, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2305.11206" label="2305.11206">
        <attvalues>
          <attvalue for="0" value="LIMA: Less Is More for Alignment" />
          <attvalue for="1" value="  Large language models are trained in two stages: (1) unsupervised pretraining&#10;from raw text, to learn general-purpose representations, and (2) large scale&#10;instruction tuning and reinforcement learning, to better align to end tasks and&#10;user preferences. We measure the relative importance of these two stages by&#10;training LIMA, a 65B parameter LLaMa language model fine-tuned with the&#10;standard supervised loss on only 1,000 carefully curated prompts and responses,&#10;without any reinforcement learning or human preference modeling. LIMA&#10;demonstrates remarkably strong performance, learning to follow specific&#10;response formats from only a handful of examples in the training data,&#10;including complex queries that range from planning trip itineraries to&#10;speculating about alternate history. Moreover, the model tends to generalize&#10;well to unseen tasks that did not appear in the training data. In a controlled&#10;human study, responses from LIMA are either equivalent or strictly preferred to&#10;GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard&#10;and 65% versus DaVinci003, which was trained with human feedback. Taken&#10;together, these results strongly suggest that almost all knowledge in large&#10;language models is learned during pretraining, and only limited instruction&#10;tuning data is necessary to teach models to produce high quality output.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.04023" label="2302.04023">
        <attvalues>
          <attvalue for="0" value="A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on&#10;  Reasoning, Hallucination, and Interactivity" />
          <attvalue for="1" value="  This paper proposes a framework for quantitatively evaluating interactive&#10;LLMs such as ChatGPT using publicly available data sets. We carry out an&#10;extensive technical evaluation of ChatGPT using 23 data sets covering 8&#10;different common NLP application tasks. We evaluate the multitask, multilingual&#10;and multi-modal aspects of ChatGPT based on these data sets and a newly&#10;designed multimodal dataset. We find that ChatGPT outperforms LLMs with&#10;zero-shot learning on most tasks and even outperforms fine-tuned models on some&#10;tasks. We find that it is better at understanding non-Latin script languages&#10;than generating them. It is able to generate multimodal content from textual&#10;prompts, via an intermediate code generation step. Moreover, we find that&#10;ChatGPT is 63.41% accurate on average in 10 different reasoning categories&#10;under logical reasoning, non-textual reasoning, and commonsense reasoning,&#10;hence making it an unreliable reasoner. It is, for example, better at deductive&#10;than inductive reasoning. ChatGPT suffers from hallucination problems like&#10;other LLMs and it generates more extrinsic hallucinations from its parametric&#10;memory as it does not have access to an external knowledge base. Finally, the&#10;interactive feature of ChatGPT enables human collaboration with the underlying&#10;LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++&#10;on machine translation, in a multi-turn &quot;prompt engineering&quot; fashion. We also&#10;release codebase for evaluation set extraction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08786" label="2104.08786">
        <attvalues>
          <attvalue for="0" value="Fantastically Ordered Prompts and Where to Find Them: Overcoming&#10;  Few-Shot Prompt Order Sensitivity" />
          <attvalue for="1" value="  When primed with only a handful of training samples, very large, pretrained&#10;language models such as GPT-3 have shown competitive results when compared to&#10;fully-supervised, fine-tuned, large, pretrained language models. We demonstrate&#10;that the order in which the samples are provided can make the difference&#10;between near state-of-the-art and random guess performance: essentially some&#10;permutations are &quot;fantastic&quot; and some not. We analyse this phenomenon in&#10;detail, establishing that: it is present across model sizes (even for the&#10;largest current models), it is not related to a specific subset of samples, and&#10;that a given good permutation for one model is not transferable to another.&#10;While one could use a development set to determine which permutations are&#10;performant, this would deviate from the true few-shot setting as it requires&#10;additional annotated data. Instead, we use the generative nature of language&#10;models to construct an artificial development set and based on entropy&#10;statistics of the candidate permutations on this set, we identify performant&#10;prompts. Our method yields a 13% relative improvement for GPT-family models&#10;across eleven different established text classification tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.12517" label="2210.12517">
        <attvalues>
          <attvalue for="0" value="Exploring The Landscape of Distributional Robustness for Question&#10;  Answering Models" />
          <attvalue for="1" value="  We conduct a large empirical evaluation to investigate the landscape of&#10;distributional robustness in question answering. Our investigation spans over&#10;350 models and 16 question answering datasets, including a diverse set of&#10;architectures, model sizes, and adaptation methods (e.g., fine-tuning, adapter&#10;tuning, in-context learning, etc.). We find that, in many cases, model&#10;variations do not affect robustness and in-distribution performance alone&#10;determines out-of-distribution performance. Moreover, our findings indicate&#10;that i) zero-shot and in-context learning methods are more robust to&#10;distribution shifts than fully fine-tuned models; ii) few-shot prompt&#10;fine-tuned models exhibit better robustness than few-shot fine-tuned span&#10;prediction models; iii) parameter-efficient and robustness enhancing training&#10;methods provide no significant robustness improvements. In addition, we&#10;publicly release all evaluations to encourage researchers to further analyze&#10;robustness trends for question answering models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.03846" label="2303.03846">
        <attvalues>
          <attvalue for="0" value="Larger language models do in-context learning differently" />
          <attvalue for="1" value="  We study how in-context learning (ICL) in language models is affected by&#10;semantic priors versus input-label mappings. We investigate two setups-ICL with&#10;flipped labels and ICL with semantically-unrelated labels-across various model&#10;families (GPT-3, InstructGPT, Codex, PaLM, and Flan-PaLM). First, experiments&#10;on ICL with flipped labels show that overriding semantic priors is an emergent&#10;ability of model scale. While small language models ignore flipped labels&#10;presented in-context and thus rely primarily on semantic priors from&#10;pretraining, large models can override semantic priors when presented with&#10;in-context exemplars that contradict priors, despite the stronger semantic&#10;priors that larger models may hold. We next study semantically-unrelated label&#10;ICL (SUL-ICL), in which labels are semantically unrelated to their inputs&#10;(e.g., foo/bar instead of negative/positive), thereby forcing language models&#10;to learn the input-label mappings shown in in-context exemplars in order to&#10;perform the task. The ability to do SUL-ICL also emerges primarily with scale,&#10;and large-enough language models can even perform linear classification in a&#10;SUL-ICL setting. Finally, we evaluate instruction-tuned models and find that&#10;instruction tuning strengthens both the use of semantic priors and the capacity&#10;to learn input-label mappings, but more of the former.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.00225" label="1911.00225">
        <attvalues>
          <attvalue for="0" value="When Choosing Plausible Alternatives, Clever Hans can be Clever" />
          <attvalue for="1" value="  Pretrained language models, such as BERT and RoBERTa, have shown large&#10;improvements in the commonsense reasoning benchmark COPA. However, recent work&#10;found that many improvements in benchmarks of natural language understanding&#10;are not due to models learning the task, but due to their increasing ability to&#10;exploit superficial cues, such as tokens that occur more often in the correct&#10;answer than the wrong one. Are BERT's and RoBERTa's good performance on COPA&#10;also caused by this? We find superficial cues in COPA, as well as evidence that&#10;BERT exploits these cues. To remedy this problem, we introduce Balanced COPA,&#10;an extension of COPA that does not suffer from easy-to-exploit single token&#10;cues. We analyze BERT's and RoBERTa's performance on original and Balanced&#10;COPA, finding that BERT relies on superficial cues when they are present, but&#10;still achieves comparable performance once they are made ineffective,&#10;suggesting that BERT learns the task to a certain degree when forced to. In&#10;contrast, RoBERTa does not appear to rely on superficial cues.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.10158" label="2304.10158">
        <attvalues>
          <attvalue for="0" value="Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on&#10;  POS Tagging for Non-Standardized Languages" />
          <attvalue for="1" value="  One of the challenges with finetuning pretrained language models (PLMs) is&#10;that their tokenizer is optimized for the language(s) it was pretrained on, but&#10;brittle when it comes to previously unseen variations in the data. This can for&#10;instance be observed when finetuning PLMs on one language and evaluating them&#10;on data in a closely related language variety with no standardized orthography.&#10;Despite the high linguistic similarity, tokenization no longer corresponds to&#10;meaningful representations of the target data, leading to low performance in,&#10;e.g., part-of-speech tagging.&#10;  In this work, we finetune PLMs on seven languages from three different&#10;families and analyze their zero-shot performance on closely related,&#10;non-standardized varieties. We consider different measures for the divergence&#10;in the tokenization of the source and target data, and the way they can be&#10;adjusted by manipulating the tokenization during the finetuning step. Overall,&#10;we find that the similarity between the percentage of words that get split into&#10;subwords in the source and target data (the split word ratio difference) is the&#10;strongest predictor for model performance on target data.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformer-based pre-trained language models (PLMs) enable successful cross-lingual transfer for many natural language processing tasks. However, the impact of tokenization and its interplay with transferability across languages, especially under-resourced variants with no orthography, has obtained limited focus so far. Tokenization splits words into subwords, but not necessarily in a meaningful way.&#10;An example with a current PLM is illustrated for Alsatian German in Figure~\ref{fig:example}a.&#10;This problem is especially pronounced for vernacular languages and dialects, where words tend to be split at a much higher rate than the standard. This has been observed on, e.g.,&#10;informally written Algerian Arabic \cite{touileb-barnes-2021-interplay}.&#10;As poor subword tokenization can lead to suboptimal language representations and impoverished transfer, it becomes important to understand if the effect holds at a larger scale.&#10;We are particularly interested in challenging setups in which, despite high language similarity, comparatively low transfer performance is obtained.&#10;&#10;A recent study proposes an elegant and lean solution to address this `tokenization gap,' without requiring expensive PLM re-training: to manipulate tokenization of PLMs post-hoc~\cite{aepli-sennrich-2022-improving}, i.e., during finetuning by injecting character-level noise (Figure~\ref{fig:example}c).&#10;Noise injection has been shown to successfully aid cross-lingual transfer and is an appealing solution, as it is cheap and widely applicable. &#10;In this work, we first provide a reproduction study and then broaden it by a systematic investigation of the extent to which noise injection helps.&#10;We also show how it influences the subword tokenization of the source data vis-à-vis the target data. &#10;We hypothesize that, while not emulating dialect text, injecting noise into standard language data can raise the tokenization rate to a similar level, which aids transfer.&#10;&#10;The importance of token overlap between source and target is an on-going debate (to which we contribute):&#10;Prior research has found that subword token overlap between the finetuning and target language improves transfer \cite{wu-dredze-2019-beto, pires-etal-2019-multilingual}, although it might neither be the most important factor \cite{k2020mbert-analysis, muller2022languages} nor a necessary condition for cross-lingual transfer to work \cite{pires-etal-2019-multilingual, conneau-etal-2020-emerging}.&#10;&#10;To enable research in this direction, we contribute a novel benchmark. &#10;We collected under-resourced language variants covering seven part-of-speech (POS) tagging transfer scenarios within three language families. &#10;This collection enables also future work to study cross-lingual and cross-dialect transfer.&#10;&#10;Our contributions are:&#10;\begin{itemize}&#10; \item We investigate the noise injection method by \cite{aepli-sennrich-2022-improving} with respect to the ideal noise injection rate for different languages and PLMs.&#10; \item To the best of our knowledge, this is the broadest study that focuses specifically on transfer to closely related, non-standardized language varieties with languages from multiple linguistic families.&#10; We convert several dialect datasets into a shared tagset (UPOS) and share the conversion scripts.&#10; \item We compare the effect of noise injection on the subword tokenization differences between the source and target data, and the effect of these differences on the model performance, and find that&#10; the proportions of (un)split words are a better predictor than the ratio of seen subword tokens.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Multilingual Model Performance, Linguistics, Tokenization Variations, Artificial Intelligence, Natural Language Processing, Language Model Finetuning" />
        </attvalues>
      </node>
      <node id="2105.07400" label="2105.07400">
        <attvalues>
          <attvalue for="0" value="The interplay between language similarity and script on a novel&#10;  multi-layer Algerian dialect corpus" />
          <attvalue for="1" value="  Recent years have seen a rise in interest for cross-lingual transfer between&#10;languages with similar typology, and between languages of various scripts.&#10;However, the interplay between language similarity and difference in script on&#10;cross-lingual transfer is a less studied problem. We explore this interplay on&#10;cross-lingual transfer for two supervised tasks, namely part-of-speech tagging&#10;and sentiment analysis. We introduce a newly annotated corpus of Algerian&#10;user-generated comments comprising parallel annotations of Algerian written in&#10;Latin, Arabic, and code-switched scripts, as well as annotations for sentiment&#10;and topic categories. We perform baseline experiments by fine-tuning&#10;multi-lingual language models. We further explore the effect of script vs.&#10;language similarity in cross-lingual transfer by fine-tuning multi-lingual&#10;models on languages which are a) typologically distinct, but use the same&#10;script, b) typologically similar, but use a distinct script, or c) are&#10;typologically similar and use the same script. We find there is a delicate&#10;relationship between script and typology for part-of-speech, while sentiment&#10;analysis is less sensitive.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.06772" label="2109.06772">
        <attvalues>
          <attvalue for="0" value="Improving Zero-shot Cross-lingual Transfer between Closely Related&#10;  Languages by injecting Character-level Noise" />
          <attvalue for="1" value="  Cross-lingual transfer between a high-resource language and its dialects or&#10;closely related language varieties should be facilitated by their similarity.&#10;However, current approaches that operate in the embedding space do not take&#10;surface similarity into account. This work presents a simple yet effective&#10;strategy to imrove cross-lingual transfer between closely related varieties. We&#10;propose to augment the data of the high-resource source language with&#10;character-level noise to make the model more robust towards spelling&#10;variations. Our strategy shows consistent improvements over several languages&#10;and tasks: Zero-shot transfer of POS tagging and topic identification between&#10;language varieties from the Finnic, West and North Germanic, and Western&#10;Romance language branches. Our work provides evidence for the usefulness of&#10;simple surface-level noise in improving transfer between language varieties.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Zero-shot cross-lingual transfer based on multilingual language models~\cite{devlin-etal-2019-bert,Conneau19neurips} or machine translation models~\cite{mmte} has turned out to be surprisingly effective.&#10;Such representations proved themselves beneficial for a range of diverse tasks~\cite{xtreme}. However, they still require large-scale data sets to train, making them impractical for low-resource languages, to which dialects and language varieties typically belong.&#10;&#10;\cite{artetxe2020cross} introduce zero-shot cross-lingual transfer by mapping monolingual representations between languages. They also propose adding Gaussian noise to the embeddings during the fine-tuning step. &#10;\cite{Huang-etal-2021-improving-zero} also operate in the embedding space by constructing robust regions in the embedding space to tolerate noise in the contextual embedding.&#10;These are not ideal strategies for closely related languages because words with similar surface forms could still be far from each other in an embedding space.&#10;&#10;Surface-level noise such as character substitutions, insertions, and deletions has been proposed as an effective data augmentation strategy for machine translation~\cite{sperber2017toward,heigold-etal-2018-robust,belinkov2018synthetic,karpukhin-etal-2019-training,vaibhav-etal-2019-improving, anastasopoulos-etal-2019-neural}. Authors report improvements in system accuracy due to more robustness towards speech recognition errors, spelling mistakes, and other naturally occurring noise in text data. &#10;Even though cross-lingual transfer between closely related languages has received some attention~\cite{muller2020can, sakaguchi2017robsut, zeman-etal-2017-conll, zeman-etal-2018-conll}, it has not been investigated whether this transfer can be improved with character-level noise inserted at training time.&#10;We tackle this in our work by adding random character-level noise to the training data of a standard language and applying the model to closely related languages.&#10;&#10;Exploiting orthographic similarity to improve cross-lingual transfer between closely related languages is currently an understudied area. Relevant previous work has been done by~\cite{sharoff-2018-language}, who used orthographic similarity to refine bilingual dictionary induction. &#10;&#10;Transliteration is another line of related work that focuses on improving the transfer between closely related languages with different alphabets~\cite{durrani-etal-2014-integrating, lin-etal-2016-leveraging, murikinati-etal-2020-transliteration, han-eisenstein-2019-unsupervised}. On the other hand, our work focuses on languages using the same script. The recent report by~\cite{muller-etal-2021-unseen} investigating transfer between the same and different alphabets involves a zero-shot task transfer which is, however, preceded by a language model training on (unlabeled) target language data. To the best of our knowledge, we are the first to focus on zero-shot transfer learning techniques for closely related languages. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="1912.07840" label="1912.07840">
        <attvalues>
          <attvalue for="0" value="Cross-Lingual Ability of Multilingual BERT: An Empirical Study" />
          <attvalue for="1" value="  Recent work has exhibited the surprising cross-lingual abilities of&#10;multilingual BERT (M-BERT) -- surprising since it is trained without any&#10;cross-lingual objective and with no aligned data. In this work, we provide a&#10;comprehensive study of the contribution of different components in M-BERT to&#10;its cross-lingual ability. We study the impact of linguistic properties of the&#10;languages, the architecture of the model, and the learning objectives. The&#10;experimental study is done in the context of three typologically different&#10;languages -- Spanish, Hindi, and Russian -- and using two conceptually&#10;different NLP tasks, textual entailment and named entity recognition. Among our&#10;key conclusions is the fact that the lexical overlap between languages plays a&#10;negligible role in the cross-lingual success, while the depth of the network is&#10;an integral part of it. All our models and implementations can be found on our&#10;project page: http://cogcomp.org/page/publication_view/900 .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15294" label="2305.15294">
        <attvalues>
          <attvalue for="0" value="Enhancing Retrieval-Augmented Large Language Models with Iterative&#10;  Retrieval-Generation Synergy" />
          <attvalue for="1" value="  Large language models are powerful text processors and reasoners, but are&#10;still subject to limitations including outdated knowledge and hallucinations,&#10;which necessitates connecting them to the world. Retrieval-augmented large&#10;language models have raised extensive attention for grounding model generation&#10;on external knowledge. However, retrievers struggle to capture relevance,&#10;especially for queries with complex information needs. Recent work has proposed&#10;to improve relevance modeling by having large language models actively involved&#10;in retrieval, i.e., to improve retrieval with generation. In this paper, we&#10;show that strong performance can be achieved by a method we call Iter-RetGen,&#10;which synergizes retrieval and generation in an iterative manner. A model&#10;output shows what might be needed to finish a task, and thus provides an&#10;informative context for retrieving more relevant knowledge which in turn helps&#10;generate a better output in the next iteration. Compared with recent work which&#10;interleaves retrieval with generation when producing an output, Iter-RetGen&#10;processes all retrieved knowledge as a whole and largely preserves the&#10;flexibility in generation without structural constraints. We evaluate&#10;Iter-RetGen on multi-hop question answering, fact verification, and commonsense&#10;reasoning, and show that it can flexibly leverage parametric knowledge and&#10;non-parametric knowledge, and is superior to or competitive with&#10;state-of-the-art retrieval-augmented baselines while causing fewer overheads of&#10;retrieval and generation. We can further improve performance via&#10;generation-augmented retrieval adaptation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;In recent months, there has been a surge in LLM-powered applications, such as ChatGPT, Bing Chat, and CoPilot \cite{DBLP:journals/corr/abs-2107-03374}.&#10;While showing an unprecedented level of performance, LLMs are subject to the following limitations:&#10;(1) Due to a high demand for compute and data, it remains an open research question to continually update LLMs both efficiently and effectively \cite{DBLP:journals/corr/abs-2205-12393};&#10;(2) LLMs also tend to hallucinate \cite{DBLP:journals/corr/abs-2303-08774}, i.e., generating plausible but non-factual texts.&#10;To alleviate these issues, there is a growing trend of augmenting LLMs with tools \cite{DBLP:journals/corr/abs-2302-07842,gou2023critic}, e.g., a code interpreter \cite{DBLP:journals/corr/abs-2211-10435,DBLP:journals/corr/abs-2302-00618} or a search engine \cite{DBLP:journals/corr/abs-2112-09332}, in an attempt to offload sub-tasks to more qualified experts, or to enrich the input context for LLMs by providing more relevant information.&#10;&#10;Retrieval augmentation is a mainstream direction to connect LLMs to the external world. &#10;Previous retrieval-augmented LMs \cite{DBLP:conf/eacl/IzacardG21,DBLP:conf/acl/ShaoH22} typically receive retrieved knowledge in a passive way:&#10;knowledge is retrieved based on the task inputs without LMs' intervention.&#10;As it is difficult for a retriever to capture relevance, especially in the zero-shot setting, recent work shows a shift towards having LLMs actively involved in retrieval to improve relevance modeling, e.g., to provide a specific context for retrieval with model generations (e.g., generated search queries \cite{DBLP:journals/corr/abs-2112-09332,DBLP:journals/corr/abs-2210-03350,DBLP:journals/corr/abs-2210-03629}, partial generation \cite{DBLP:journals/corr/abs-2212-10509}, or forward-looking sentences \cite{DBLP:journals/corr/abs-2305-06983}).&#10;\cite{DBLP:journals/corr/abs-2212-14024} proposed a DSP programming framework that supports various retrieval-augmented methods.&#10;&#10;Recent work interleaves retrieval with generation when completing a single output.&#10;Such a structured workflow may reduce the flexibility in generation \cite{DBLP:journals/corr/abs-2210-03629}.&#10;Iter-RetGen avoids interrupting generation with retrieval, but iterates retrieval and generation, i.e., to leverage the complete generation from the previous iteration to retrieve more relevant information which helps improve generation in the next iteration.&#10;Iter-RetGen also has the advantage of processing all retrieved knowledge as a whole during the generation process, and is conceptually simpler and easier-to-implement, while being empirically strong in multi-hop question answering, fact verification, and commonsense reasoning.&#10;&#10;A closely related work called Gar \cite{DBLP:conf/acl/MaoHLSG0C20} augments queries with generated background information.&#10;HyDE \cite{DBLP:journals/corr/abs-2212-10496} also shares a similar spirit, but focuses on zero-shot information retrieval, and proposes to first prompt an LLM to produce ``hypothetical'' paragraphs that cover the information needed to answer a given question, and then use the generated paragraphs to retrieve the real ones.&#10;RepoCoder \cite{DBLP:journals/corr/abs-2303-12570} focuses on repository-level code completion, and proposes a 2-iteration retrieval-generation paradigm where the second iteration leverages the intermediate code completion for retrieval.&#10;By contrast, we propose to synergize retrieval and generation with Iter-RetGen on various natural language tasks, and explore how we can further adapt retrieval with model generations.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Retrieval Augmentation Methods, Language Model Limitations, Artificial Intelligence, Iterative Knowledge Generation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2211.04486" label="2211.04486">
        <attvalues>
          <attvalue for="0" value="Active Example Selection for In-Context Learning" />
          <attvalue for="1" value="  With a handful of demonstration examples, large-scale language models show&#10;strong capability to perform various tasks by in-context learning from these&#10;examples, without any fine-tuning. We demonstrate that in-context learning&#10;performance can be highly unstable across samples of examples, indicating the&#10;idiosyncrasies of how language models acquire information. We formulate example&#10;selection for in-context learning as a sequential decision problem, and propose&#10;a reinforcement learning algorithm for identifying generalizable policies to&#10;select demonstration examples. For GPT-2, our learned policies demonstrate&#10;strong abilities of generalizing to unseen tasks in training, with a $5.8\%$&#10;improvement on average. Examples selected from our learned policies can even&#10;achieve a small improvement on GPT-3 Ada. However, the improvement diminishes&#10;on larger GPT-3 models, suggesting emerging capabilities of large language&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Our paper builds on top of prior work&#10;that uses RL to solve the active learning problem~\cite{fangLearningHowActive2017,liuLearningHowActively2018},&#10;and is made possible by the recent advances in pre-trained language models~\cite{devlinBERTPretrainingDeep2019,liuRoBERTaRobustlyOptimized2019,raffelExploringLimitsTransfer2020a,gaoMakingPretrainedLanguage2021}.&#10;In-context learning, the observation that LMs \cite{radford2019language,&#10;brownLanguageModelsAre2020, raeScalingLanguageModels2022,&#10;zhangOPTOpenPretrained2022} can ``learn'' to perform a task when conditioned on a prompt.&#10;\cite{xieExplanationIncontextLearning2022} explains the emergenece of&#10;in-context learning by inferring the shared latent concept among&#10;demonstration examples, while \cite{minRethinkingRoleDemonstrations2022}&#10;finds the success of in-context learning is largely independent of access to&#10;gold labels.&#10;&#10;A variety of issues with in-context learning is discovered,&#10;including surface form competition, the phenomenon that multiple words referring&#10;to the same concept fighting for probability mass&#10;\cite{holtzmanSurfaceFormCompetition2021},&#10;and sensitivity of LMs due to changes in prompt&#10;\cite{lesterPowerScaleParameterEfficient2021},&#10;instruction \cite{mishraReframingInstructionalPrompts2022}, or&#10;ordering of demonstration examples \cite{zhaoCalibrateUseImproving2021,&#10;luFantasticallyOrderedPrompts2022}.&#10;To optimize the performance of in-context learning, methods with varying&#10;levels of granularity are proposed. Such methods include prompt tuning&#10;\cite{lesterPowerScaleParameterEfficient2021, vuSPoTBetterFrozen2022,&#10;wuIDPGInstanceDependentPrompt2022}, and instruction optimization&#10;\cite{mishraReframingInstructionalPrompts2022,&#10;kojimaLargeLanguageModels2022}. \cite{liuWhatMakesGood2021} approaches&#10;the example selection problem by searching for nearest neighbors of test&#10;examples in the embedding space, while \cite{rubinLearningRetrievePrompts2022}&#10;uses a scoring LM for example retrieval.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, In-Context Learning, Reinforcement Learning, Artificial Intelligence, Natural Language Processing, Language Model Performance" />
        </attvalues>
      </node>
      <node id="2401.08565" label="2401.08565">
        <attvalues>
          <attvalue for="0" value="Tuning Language Models by Proxy" />
          <attvalue for="1" value="  Despite the general capabilities of large pretrained language models, they&#10;consistently benefit from further adaptation to better achieve desired&#10;behaviors. However, tuning these models has become increasingly&#10;resource-intensive, or impossible when model weights are private. We introduce&#10;proxy-tuning, a lightweight decoding-time algorithm that operates on top of&#10;black-box LMs to achieve the same end as direct tuning, but by accessing only&#10;its predictions over the output vocabulary, not its parameters. Our method&#10;tunes a smaller LM, then applies the difference between the predictions of the&#10;small tuned and untuned LMs to shift the original predictions of the larger&#10;untuned model in the direction of tuning, while retaining the benefits of&#10;larger-scale pretraining. In experiments, when we apply proxy-tuning to&#10;Llama2-70B using proxies of only 7B size, we can close 88% of the gap between&#10;Llama2-70B and its truly-tuned chat version, when evaluated across knowledge,&#10;reasoning, and safety benchmarks. We then demonstrate the generality of&#10;proxy-tuning by applying it to domain adaptation on code, and task-specific&#10;finetuning on question-answering and math problems. Finally, we show how to&#10;proxy-tune a truly black-box LM, GPT-3.5, for temporal adaptation, increasing&#10;its knowledge about recent events. Our work demonstrates the promise of using&#10;small tuned LMs to efficiently customize large, potentially proprietary LMs&#10;through decoding-time guidance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Efficient Finetuning&#10;Today, large pretrained models form the basis of any kind of adaptation, whether for tasks \cite{raffel-etal-2020-exploring}, domains \cite{gururangan-etal-2020-dont}, or general-purpose dialogue \cite{ouyang-etal-2022-training}.&#10;Moreover, scaling up the size of these models is a reliable recipe for further improvement \cite{kaplan-etal-2020-scaling}.&#10;Thus, efficiently tuning ever-larger models has become a pressing challenge, leading to a large body of work on efficient finetuning, commonly through updating a small number of parameters \cite[i.a.]{houlsby-etal-2019-parameter, li-liang-2021-prefix, hu-etal-2022-lora, dettmers-etal-2023-qlora}.&#10;Nonetheless, these methods require white-box model access, which is unavailable for many of today's advanced models.&#10;&#10;In this context, ``tuning'' LMs at decoding-time represents an approach for efficient finetuning. &#10;Our work shares a similar vision with contemporary work \cite{mitchell-etal-2024-emulator}, which applies the same DExperts equation as operationalized in \S\ref{sec:instruction_tuning}. &#10;However, they mainly view the equation as a tool for disentangling the effects of scaling up pretraining versus instruction-tuning,&#10;and do not measure the method's effectiveness on existing benchmarks.&#10;In contrast, our work demonstrates the empirical strength of proxy-tuning, as well as its generality beyond instruction-tuning alone.&#10;Recently, \cite{ormazabal-etal-2023-comblm} also combine the probability distributions from a small tuned model and a large pretrained model, but through a learned combination function which requires additional data and training.&#10;&#10;For instruction-following specifically, a curated prompt can elicit generations that are surprisingly competitive with instruction-tuning \cite{han-2023-incontext, lin-etal-2023-unlocking}.&#10;However, these prompts tend to be quite long, introducing an inference-time computational burden and restricting the length of generations for models with limited context windows. &#10;&#10;Controllable Generation&#10;&#10;There is a rich body of work in controllable generation, which differs from decoding-time tuning as it aims to control certain attributes of generated continuations, commonly non-toxicity and positive sentiment.&#10;In this space, there are many methods that operate on output logits \cite{krause-etal-2021-gedi-generative, liu-etal-2021-dexperts, yang-klein-2021-fudge, deng-raffel-2023-reward}.&#10;In addition to the different objective from our work, many prior methods require the user to tune additional parameters, such as a model with control codes (GeDi; \cite{krause-etal-2021-gedi-generative}) or a head on top of the LM (IPA; \cite{lu-etal-2023-inference}).&#10;In contrast, proxy-tuning allows users to leverage the rich collection of small tuned models available online, potentially composing them off-the-shelf with no additional training.&#10;&#10;Logit Arithmetic&#10;&#10;Our work builds off DExperts \cite{liu-etal-2021-dexperts}, which introduced \ref{eq:1} and showed the effectiveness of ensembling logits from multiple LMs, an idea which was also briefly explored in earlier work \cite{dou-etal-2019-domain}.&#10;There has been a growing body of methods that perform arithmetic on multiple logit distributions for better text generation, such as contrasting the logits of a large and small model \cite{li-etal-2023-contrastive}, logits from different layers of a model \cite{gera-etal-2023-benefits, chuang-etal-2023-dola}, and logits from the same model given different inputs \cite{shi-etal-2023-trusting, pei-etal-2023-preadd, sennrich-etal-2023-mitigating, leng-etal-2023-mitigating}; it has even been extended to non-autoregressive LMs (e.g., diffusion LMs; \cite{han-etal-2024-david}). &#10;&#10;" />
          <attvalue for="4" value="Language Model Tuning, Computer Science, Linguistics, Decoding Time Algorithms, Large Scale Pretraining, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2204.06644" label="2204.06644">
        <attvalues>
          <attvalue for="0" value="METRO: Efficient Denoising Pretraining of Large Scale Autoencoding&#10;  Language Models with Model Generated Signals" />
          <attvalue for="1" value="  We present an efficient method of pretraining large-scale autoencoding&#10;language models using training signals generated by an auxiliary model.&#10;Originated in ELECTRA, this training strategy has demonstrated&#10;sample-efficiency to pretrain models at the scale of hundreds of millions of&#10;parameters. In this work, we conduct a comprehensive empirical study, and&#10;propose a recipe, namely &quot;Model generated dEnoising TRaining Objective&quot;&#10;(METRO), which incorporates some of the best modeling techniques developed&#10;recently to speed up, stabilize, and enhance pretrained language models without&#10;compromising model effectiveness. The resultant models, METRO-LM, consisting of&#10;up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE,&#10;SuperGLUE, and SQuAD benchmarks. More importantly, METRO-LM are efficient in&#10;that they often outperform previous large models with significantly smaller&#10;model sizes and lower pretraining cost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Pretrained language models (PLMs) can be grouped into two categories: autoencoding and autoregressive models. While both are language representation models in that they encode input text with dense vector representations, they have been used for different types of language tasks. &#10;Autoencoding models are widely used for language understanding tasks, such as text classification~\cite{minaee2021deep}, extractive question answering~\cite{rajpurkar2016squad}, text matching~\cite{xiong2020approximate}, information seeking~\cite{gao2022neural}. &#10;Autoregressive, decoder-only, models are mainly used for language generation tasks~\cite{radford2019language}. &#10;&#10;The effectiveness of large-scale PLMs has been demonstrated in a wide range of language tasks~\cite[e.g.,][]{devlin2019bert,liu2019roberta,yang2019xlnet,raffel2019t5, he2020deberta}.&#10;One interesting observation is that the growth scale of dense autoregressive models is far ahead of autoencoding models. Large unidirectional models, such as GPT-3 (175B)~\cite{brown2020language}, Gopher (280B)~\cite{rae2021scaling}, and Megatron-Turing NLG (530B)~\cite{smith2022MTNLG}, are significantly bigger than the largest bi-directional autoencoding models, e.g., DeBERTa (1.5B)~\cite{he2020deberta}, Megatron (3.9B)~\cite{shoeybi2019megatron}, and ERNIE 3.0 (10B)~\cite{sun2021ernie}. &#10;In this work we aim to scale up autoencoding models, which are widely used for real-world language representation scenarios.&#10;&#10;Studies on the scaling law of language models suggest that an exponential increase of investment in compute resource and model size only yields linear performance gain downstream~\cite{kaplan2020scaling}. &#10;Improving modeling efficiency becomes increasingly more important as LLMs get bigger and bigger.&#10;ELECTRA~\cite{clark2020electra} presents an efficient way to pretrain an autoencoder using training signals generated by an auxiliary model. \cite{meng2021coco} pointed out that the efficiency of ELECTRA pretraining is attributed to the learning curriculum provided by the auxiliary model, as the model generated training signals are increasingly difficult to denoise during the course of pretraining. Although the ELECTRA method has been proved cost-effective in pretraining models with hundred millions of parameters, its efficiency remains unexplored at the scale of billions of parameters.&#10;&#10;Another widely observed bottleneck in training LLMs is the optimization instability. The sources of instabilities include layer normalization placement~\cite{xiong2020layer, shleifer2021normformer}, parameter initialization~\cite{liu2020understanding}, mix precision optimization~\cite{rae2021scaling}, to name a few. &#10;In order to pretrain LLMs, it is common to resort to techniques that compromise model effectiveness for training stability. For example, models using a pre-LayerNorm architecture are less effective than the models using post-LayerNorm, but the former are much easier to train at large scale than the latter~\cite{liu2020understanding, xiong2020layer}. In this study we show that training stability can be achieved with fewer sacrifices on model effectiveness.&#10;&#10;There are other pathways to improving LLM efficiency.&#10;One is to augment a language model with a retrieval component to fetch external knowledge that is useful to perform downstream tasks. So, the size of the language model can be significantly reduced since it does not need to encode everything in model parameters~\cite{guu2020realm, Khandelwal2020Generalization, RETRO,gui2021kat, zhang2022retgen}.&#10;With sparse model structures, Mixture of Experts models~\cite{artetxe2021efficient,fedus2021switch, zuo2021taming, zoph2022designing} adaptively activate a subset of model parameters (experts) for different inputs during model training and inference. &#10;The METRO method proposed in this paper is orthogonal to retrieval-augmented models and sparsely activated models. Their combination is an interesting future work direction.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Training, Pretraining Strategies, Mathematics, Large-Scale Modeling" />
        </attvalues>
      </node>
      <node id="2004.13195" label="2004.13195">
        <attvalues>
          <attvalue for="0" value="Word Interdependence Exposes How LSTMs Compose Representations" />
          <attvalue for="1" value="  Recent work in NLP shows that LSTM language models capture compositional&#10;structure in language data. For a closer look at how these representations are&#10;composed hierarchically, we present a novel measure of interdependence between&#10;word meanings in an LSTM, based on their interactions at the internal gates. To&#10;explore how compositional representations arise over training, we conduct&#10;simple experiments on synthetic data, which illustrate our measure by showing&#10;how high interdependence can hurt generalization. These synthetic experiments&#10;also illustrate a specific hypothesis about how hierarchical structures are&#10;discovered over the course of training: that parent constituents rely on&#10;effective representations of their children, rather than on learning long-range&#10;relations independently. We further support this measure with experiments on&#10;English language data, where interdependence is higher for more closely&#10;syntactically linked word pairs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent work in NLP has seen a flurry of interest in the question: are the representations learned by neural networks compositional? That is, are representations of longer phrases built recursively from representations of shorter phrases, as they are in many linguistic theories? If so, how and when do they learn to do this?&#10;&#10;Computationally, sequence models like LSTMs scan a sentence from left to right, accumulating meaning into a hidden representation at each time step. Yet we have extensive evidence that fully trained LSTMs are sensitive to syntactic structure, suggesting that they learn something about recursive composition of meaning. For example, they can recall more history in natural language data than in similarly Zipfian-distributed n-gram data, implying that they exploit long-distance dependencies \cite{liu_lstms_2018}. Their internal representations seem to be hierarchical in nature \cite{blevins_deep_2018,hupkes_visualisation_2017}. They seemingly encode knowledge of part of speech \cite{belinkov_what_2017}, morphological productivity \cite{Vania2017FromCT}, and verb agreement \cite{lakretz_emergence_2019}. How does this apparently compositional behavior arise in learning?&#10;&#10;Concretely, we are interested in an aspect of compositionality sometimes called localism~\cite{hupkes_compositionality_2019}, in which meanings of long sequences are recursively composed from meanings of shorter child sequences, without regard for how the child meaning is itself constructed---that is, the computation of the composed meaning relies only on the local properties of the child meanings. By contrast, a global composition would be constructed from all words. A local composition operation leads to hierarchical structure (like a classic syntax tree), whereas a global operation leads to flat structure. If meaning is composed locally, then in a sentence like ``The chimney sweep has sick lungs'', the unknown composition function $f$ might reflect syntactic structure, computing the full meaning as: $f(f($The, chimney, sweep$)$, has, $f($sick, lungs$))$. Local composition assumes low interdependence between the meanings of ``chimney'' and ``has'', or indeed between any pair of words not local to the same invocation of $f$.&#10;&#10;To analyze compositionality, we propose a measure of word interdependence that directly measures the composition of meaning in LSTMs through the interactions between words (Section~\ref{sec:methods}). Our method builds on Contextual Decomposition~\cite[CD; ][]{murdoch_beyond_2018}, a tool for analyzing the representations produced by LSTMs. We conduct experiments on a synthetic corpus (Section~\ref{sec:synthetic}), which illustrate interdependence and find that highly familiar constituents make nearby vocabulary statistically dependent on them, leaving them vulnerable to the domain shift. We then relate word interdependence in an English language corpus (Section~\ref{sec:english}) to syntax, finding that word pairs with close syntactic links have higher interdependence than more distantly linked words, even stratifying by sequential distance and part of speech. This pattern offers a potential structural probe that can be computed directly from an LSTM without learning additional parameters as required in other methods \cite{hewitt_structural_nodate}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Compositional Structure, Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Neural Network Dynamics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1805.11653" label="1805.11653">
        <attvalues>
          <attvalue for="0" value="LSTMs Exploit Linguistic Attributes of Data" />
          <attvalue for="1" value="  While recurrent neural networks have found success in a variety of natural&#10;language processing applications, they are general models of sequential data.&#10;We investigate how the properties of natural language data affect an LSTM's&#10;ability to learn a nonlinguistic task: recalling elements from its input. We&#10;find that models trained on natural language data are able to recall tokens&#10;from much longer sequences than models trained on non-language sequential data.&#10;Furthermore, we show that the LSTM learns to solve the memorization task by&#10;explicitly using a subset of its neurons to count timesteps in the input. We&#10;hypothesize that the patterns and structure in natural language data enable&#10;LSTMs to learn by providing approximate ways of reducing loss, but&#10;understanding the effect of different training data on the learnability of&#10;LSTMs remains an open question.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.04218" label="1805.04218">
        <attvalues>
          <attvalue for="0" value="Deep RNNs Encode Soft Hierarchical Syntax" />
          <attvalue for="1" value="  We present a set of experiments to demonstrate that deep recurrent neural&#10;networks (RNNs) learn internal representations that capture soft hierarchical&#10;notions of syntax from highly varied supervision. We consider four syntax tasks&#10;at different depths of the parse tree; for each word, we predict its part of&#10;speech as well as the first (parent), second (grandparent) and third level&#10;(great-grandparent) constituent labels that appear above it. These predictions&#10;are made from representations produced at different depths in networks that are&#10;pretrained with one of four objectives: dependency parsing, semantic role&#10;labeling, machine translation, or language modeling. In every case, we find a&#10;correspondence between network depth and syntactic depth, suggesting that a&#10;soft syntactic hierarchy emerges. This effect is robust across all conditions,&#10;indicating that the models encode significant amounts of syntax even in the&#10;absence of an explicit syntactic training supervision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.10203" label="1711.10203">
        <attvalues>
          <attvalue for="0" value="Visualisation and 'diagnostic classifiers' reveal how recurrent and&#10;  recursive neural networks process hierarchical structure" />
          <attvalue for="1" value="  We investigate how neural networks can learn and process languages with&#10;hierarchical, compositional semantics. To this end, we define the artificial&#10;task of processing nested arithmetic expressions, and study whether different&#10;types of neural networks can learn to compute their meaning. We find that&#10;recursive neural networks can find a generalising solution to this problem, and&#10;we visualise this solution by breaking it up in three steps: project, sum and&#10;squash. As a next step, we investigate recurrent neural networks, and show that&#10;a gated recurrent unit, that processes its input incrementally, also performs&#10;very well on this task. To develop an understanding of what the recurrent&#10;network encodes, visualisation techniques alone do not suffice. Therefore, we&#10;develop an approach where we formulate and test multiple hypotheses on the&#10;information encoded and processed by the network. For each hypothesis, we&#10;derive predictions about features of the hidden state representations at each&#10;time step, and train 'diagnostic classifiers' to test those predictions. Our&#10;results indicate that the networks follow a strategy similar to our&#10;hypothesised 'cumulative strategy', which explains the high accuracy of the&#10;network on novel expressions, the generalisation to longer expressions than&#10;seen in training, and the mild deterioration with increasing length. This is&#10;turn shows that diagnostic classifiers can be a useful technique for opening up&#10;the black box of neural networks. We argue that diagnostic classification,&#10;unlike most visualisation techniques, does scale up from small networks in a&#10;toy domain, to larger and deeper recurrent networks dealing with real-life&#10;data, and may therefore contribute to a better understanding of the internal&#10;dynamics of current state-of-the-art models in natural language processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.08352" label="1704.08352">
        <attvalues>
          <attvalue for="0" value="From Characters to Words to in Between: Do We Capture Morphology?" />
          <attvalue for="1" value="  Words can be represented by composing the representations of subword units&#10;such as word segments, characters, and/or character n-grams. While such&#10;representations are effective and may capture the morphological regularities of&#10;words, they have not been systematically compared, and it is not understood how&#10;they interact with different morphological typologies. On a language modeling&#10;task, we present experiments that systematically vary (1) the basic unit of&#10;representation, (2) the composition of these representations, and (3) the&#10;morphological typology of the language modeled. Our results extend previous&#10;findings that character representations are effective across typologies, and we&#10;find that a previously unstudied combination of character trigram&#10;representations composed with bi-LSTMs outperforms most others. But we also&#10;find room for improvement: none of the character-level models match the&#10;predictive accuracy of a model with access to true morphological analyses, even&#10;when learned from an order of magnitude more data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.07435" label="1903.07435">
        <attvalues>
          <attvalue for="0" value="The emergence of number and syntax units in LSTM language models" />
          <attvalue for="1" value="  Recent work has shown that LSTMs trained on a generic language modeling&#10;objective capture syntax-sensitive generalizations such as long-distance number&#10;agreement. We have however no mechanistic understanding of how they accomplish&#10;this remarkable feat. Some have conjectured it depends on heuristics that do&#10;not truly take hierarchical structure into account. We present here a detailed&#10;study of the inner mechanics of number tracking in LSTMs at the single neuron&#10;level. We discover that long-distance number information is largely managed by&#10;two `number units'. Importantly, the behaviour of these units is partially&#10;controlled by other units independently shown to track syntactic structure. We&#10;conclude that LSTMs are, to some extent, implementing genuinely syntactic&#10;processing mechanisms, paving the way to a more general understanding of&#10;grammatical encoding in LSTMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.05453" label="1801.05453">
        <attvalues>
          <attvalue for="0" value="Beyond Word Importance: Contextual Decomposition to Extract Interactions&#10;  from LSTMs" />
          <attvalue for="1" value="  The driving force behind the recent success of LSTMs has been their ability&#10;to learn complex and non-linear relationships. Consequently, our inability to&#10;describe these relationships has led to LSTMs being characterized as black&#10;boxes. To this end, we introduce contextual decomposition (CD), an&#10;interpretation algorithm for analysing individual predictions made by standard&#10;LSTMs, without any changes to the underlying model. By decomposing the output&#10;of a LSTM, CD captures the contributions of combinations of words or variables&#10;to the final prediction of an LSTM. On the task of sentiment analysis with the&#10;Yelp and SST data sets, we show that CD is able to reliably identify words and&#10;phrases of contrasting sentiment, and how they are combined to yield the LSTM's&#10;final prediction. Using the phrase-level labels in SST, we also demonstrate&#10;that CD is able to successfully extract positive and negative negations from an&#10;LSTM, something which has not previously been done.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.04092" label="2212.04092">
        <attvalues>
          <attvalue for="0" value="Successive Prompting for Decomposing Complex Questions" />
          <attvalue for="1" value="  Answering complex questions that require making latent decisions is a&#10;challenging task, especially when limited supervision is available. Recent&#10;works leverage the capabilities of large language models (LMs) to perform&#10;complex question answering in a few-shot setting by demonstrating how to output&#10;intermediate rationalizations while solving the complex question in a single&#10;pass. We introduce ``Successive Prompting'', where we iteratively break down a&#10;complex task into a simple task, solve it, and then repeat the process until we&#10;get the final solution. Successive prompting decouples the supervision for&#10;decomposing complex questions from the supervision for answering simple&#10;questions, allowing us to (1) have multiple opportunities to query in-context&#10;examples at each reasoning step (2) learn question decomposition separately&#10;from question answering, including using synthetic data, and (3) use bespoke&#10;(fine-tuned) components for reasoning steps where a large LM does not perform&#10;well. The intermediate supervision is typically manually written, which can be&#10;expensive to collect. We introduce a way to generate a synthetic dataset which&#10;can be used to bootstrap a model's ability to decompose and answer intermediate&#10;questions. Our best model (with successive prompting) achieves an improvement&#10;of ~5% absolute F1 on a few-shot version of the DROP dataset when compared with&#10;a state-of-the-art model with the same supervision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Prompting methods&#10;Prompting was introduced as a way to test the reasoning capabilities of large language models~\cite{brown2020language}. &#10;In follow-up works~\cite{schick2022few,chowdhery2022palm,marasovic2021few} prompting techniques have been used as a mechanism to supervise the model decision with few demonstrations as a conditioning context to guide its predictions on an unseen example. Works like Chain-of-Thought reasoning~\cite{wei2022chain,zelikman2022star} especially focus on compositional questions where they provide a chain of reasoning as demonstrations. &#10;In concurrent work, Least-to-Most prompting~\cite{zhou2022least} takes a similar view as ours to break down the problem into sub-problems. However, in Successive Prompting the question decomposition and answering stages are interleaved, unlike Least-to-Most where the problem is first reduced into sub-problem and then executed in a sequence. In our method, the next question prediction has access to previously answered sub-questions, which is useful in questions that need long chain referencing. &#10;Other contemporaneous works~\cite{press2022measuring,khot2022decomposed} use very large language models (more than twice the size we used) and show better few-shot generalization. Works like~\cite{perez2021true} have shown the importance of having the right in-context examples for downstream performance leading to works that learn to retrieve relevant in-context examples~\cite{rubin2021learning}.&#10;&#10;Non-symbolic methods&#10;Most non-symbolic methods are sequence-to-sequence models trained on a large amount of question answering data~\cite{khashabi2020unifiedqa,yoran2021turning}. &#10;&#10;Symbolic methods&#10;Neural module networks like approaches parse complex questions into a pre-specified grammar and learn neural components to handle symbolic mathematical operations~\cite{gupta2019neural,chen2019neural,nye2021show} which are recursively executed. State-of-the-art models on DROP, however, use a combination of BERT-based contextual models along with a calculator that performs discrete operations~\cite{andor2019giving,segal2019simple,hu2019multi}. Works like Text Modular networks~\cite{khot2020text} and MRKL~\cite{karpas2022mrkl} are closest to our work. However, they are limited in the terms of types of simple questions they can answer (single-span only) and the complexity of reasoning they can do (single-order only). TMNs, additionally, use a classifier that scores the generated chains module and filters out incorrect question decompositions, while we use contrastive estimation to learn a better question decomposer and as a result do not need a chain scorer.&#10;&#10;" />
          <attvalue for="4" value="Few-Shot Learning, Machine Learning, Computer Science, Large Language Models, Linguistics, Complex Question Answering, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2011.13220" label="2011.13220">
        <attvalues>
          <attvalue for="0" value="Unigram-Normalized Perplexity as a Language Model Performance Measure&#10;  with Different Vocabulary Sizes" />
          <attvalue for="1" value="  Although Perplexity is a widely used performance metric for language models,&#10;the values are highly dependent upon the number of words in the corpus and is&#10;useful to compare performance of the same corpus only. In this paper, we&#10;propose a new metric that can be used to evaluate language model performance&#10;with different vocabulary sizes. The proposed unigram-normalized Perplexity&#10;actually presents the performance improvement of the language models from that&#10;of simple unigram model, and is robust on the vocabulary size. Both theoretical&#10;analysis and computational experiments are reported.&#10;" />
          <attvalue for="2" value="&#10;Language model (LM) is a core elements in natural language processing (NLP) applications, e.g., language modeling \cite{bengio2003neural,mikolov2011extensions}, machine translation~\cite{cho2014learning}, speech recognition~\cite{amodei2016deep}, and dialogue generation. LMs determine the probability of word sequences and provide a metric for the probability of generated sequences. To achieve good performance, LMs must accurately capture the relationship among words and phrases in word sequences. &#10;&#10;The popular metric of LM performance is Perplexity, which is based on the likelihood of word sequences. Since the probability of words itself is highly varying for different corpus of different vocabulary size, the likelihood of word sequences is also very sensitive on corpus. For an example, the Perplexity value 10 may be very good for some corpus, but very poor for another. Therefore, to compare LM performances of different corpus, we propose a new metric, i.e., unigram-normalized Perplexity ($PPLu$), which is independent of the vocabulary size.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Mathematics, Performance Metrics, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1512.02595" label="1512.02595">
        <attvalues>
          <attvalue for="0" value="Deep Speech 2: End-to-End Speech Recognition in English and Mandarin" />
          <attvalue for="1" value="  We show that an end-to-end deep learning approach can be used to recognize&#10;either English or Mandarin Chinese speech--two vastly different languages.&#10;Because it replaces entire pipelines of hand-engineered components with neural&#10;networks, end-to-end learning allows us to handle a diverse variety of speech&#10;including noisy environments, accents and different languages. Key to our&#10;approach is our application of HPC techniques, resulting in a 7x speedup over&#10;our previous system. Because of this efficiency, experiments that previously&#10;took weeks now run in days. This enables us to iterate more quickly to identify&#10;superior architectures and algorithms. As a result, in several cases, our&#10;system is competitive with the transcription of human workers when benchmarked&#10;on standard datasets. Finally, using a technique called Batch Dispatch with&#10;GPUs in the data center, we show that our system can be inexpensively deployed&#10;in an online setting, delivering low latency when serving users at scale.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04725" label="2110.04725">
        <attvalues>
          <attvalue for="0" value="Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and&#10;  Few-Shot Learning" />
          <attvalue for="1" value="  Recent work like GPT-3 has demonstrated excellent performance of Zero-Shot&#10;and Few-Shot learning on many natural language processing (NLP) tasks by&#10;scaling up model size, dataset size and the amount of computation. However,&#10;training a model like GPT-3 requires huge amount of computational resources&#10;which makes it challengeable to researchers. In this work, we propose a method&#10;that incorporates large-scale distributed training performance into model&#10;architecture design. With this method, Yuan 1.0, the current largest singleton&#10;language model with 245B parameters, achieves excellent performance on&#10;thousands GPUs during training, and the state-of-the-art results on NLP tasks.&#10;A data processing method is designed to efficiently filter massive amount of&#10;raw data. The current largest high-quality Chinese corpus with 5TB high quality&#10;texts is built based on this method. In addition, a calibration and label&#10;expansion method is proposed to improve the Zero-Shot and Few-Shot performance,&#10;and steady improvement is observed on the accuracy of various tasks. Yuan 1.0&#10;presents strong capacity of natural language generation, and the generated&#10;articles are difficult to distinguish from the human-written ones.&#10;" />
          <attvalue for="2" value="&#10;The Transformer architecture has been widely used in natural language processing\cite{ref1, ref3, ref4}. In order to improve the performance, a varieties of Transformer-based modifications have been proposed since 2017\cite{ref1}, but many of them exhibit a lack of generalization across different implementations and tasks\cite{ref27}. Kaplan, et al.\cite{ref2} confirms that performance of the Transformer steadily improves with the scaling up of model size, dataset size, and the amount of computation for training. Roberta\cite{ref3} shows that the accuracy of BERT can be substantially improved by training the model for a longer time with a larger corpus. The T5 model, built with vanilla Transformer structure and increased model size with 11 billion parameters, achieves the state-of-the-art (SOTA) performance in various NLP tasks\cite{ref4}. It is proved that larger language models performs better than smaller ones.\\&#10;GPT-3 with 175 billion parameters, as a milestone, was proposed in 2020\cite{ref5}. Before GPT-3, it was common to pre-train a model with unsupervised learning on a large unlabeled dataset, then fine-tune on a specific task. Because GPT-3 makes great progress on Zero-Shot and Few-Shot learning, it can be applied directly on a wide range of NLP tasks, and displays good performance without being fine-tuned on those tasks. After GPT-3, several studies further increase the model size in two ways:&#10;\begin{enumerate}[$\bullet$]&#10; \item Singleton: Increase the number of layers and the size of a layer, such as GPT-3 and PanGu-$\alpha$\cite{ref6}.&#10;&#09;\item Mixture of Experts (MoE): Scaling the model size with Sparsely Gated Mixture-of-Experts (MoE), such as GShard\cite{ref7}, Switch Transformer\cite{ref8}, Wudao\cite{ref9, ref10} and M6\cite{ref11}. Each expert is a singleton model in a size up to 10B. With MoE, the model size can be successfully scaled up to more than 1000B \cite{ref8,ref10}.&#10;\end{enumerate}&#10;Both Singleton and MoE are effective to increase the model size, however, they behave differently in Zero-Shot and Few-Shot scenarios. Currently, the MoE method still follows the common way, in which pre-train the model on a large dataset and fine-tune it on specific task. To our best knowledge, no MoE model is applied on Zero-Shot or Few-Shot learning. However, both GPT-3 and PanGu-$\alpha$ with singleton architecture, exhibits good performance on Zero-Shot and Few-Shot learning\cite{ref5,ref6}.\\&#10;Training a model with parameters greater than 100B requires huge amount of computational resources. Take GPT-3 175B for example, it was trained on a cluster of 10,000 GPUs \cite{ref5,ref28}. Such a huge requirement on computational resources makes it difficult for most researchers to train a model in a similar way. In this work, we propose Yuan 1.0 singleton model with 245B parameters. To accelerate the training process of Yuan 1.0, and thus reduce energy costs and carbon emissions, we make a collaborative design of model architecture and large-scale distributed training. The main contributions of our work are summarized as below,&#10;\begin{enumerate}[$\bullet$]&#10; \item A method that incorporates large-scale distributed training performance into model architecture design is proposed. With this method, we trained our Yuan 1.0, the current largest singleton language model with 245B parameters, and achieved excellent performance on thousands GPUs&#10;&#09;\item A data processing system is created to efficiently filter a massive amount of data from Internet. The current largest Chinese corpus with 5TB high-quality text is built based on this system.&#10;&#09;\item The model architecture with better performance in Pre-train and Fine-tune pattern is likely to behave opposite in Zero-Shot and Few-Shot learning.&#10;&#09;\item A method that can steadily improve the Zero-Shot and Few-Shot performance is proposed. &#10;\end{enumerate}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Architecture, Data Science, Mathematics, Large Scale NLP, Distributed Training Methods" />
        </attvalues>
      </node>
      <node id="2102.11972" label="2102.11972">
        <attvalues>
          <attvalue for="0" value="Do Transformer Modifications Transfer Across Implementations and&#10;  Applications?" />
          <attvalue for="1" value="  The research community has proposed copious modifications to the Transformer&#10;architecture since it was introduced over three years ago, relatively few of&#10;which have seen widespread adoption. In this paper, we comprehensively evaluate&#10;many of these modifications in a shared experimental setting that covers most&#10;of the common uses of the Transformer in natural language processing.&#10;Surprisingly, we find that most modifications do not meaningfully improve&#10;performance. Furthermore, most of the Transformer variants we found beneficial&#10;were either developed in the same codebase that we used or are relatively minor&#10;changes. We conjecture that performance improvements may strongly depend on&#10;implementation details and correspondingly make some recommendations for&#10;improving the generality of experimental results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.00413" label="2012.00413">
        <attvalues>
          <attvalue for="0" value="CPM: A Large-scale Generative Chinese Pre-trained Language Model" />
          <attvalue for="1" value="  Pre-trained Language Models (PLMs) have proven to be beneficial for various&#10;downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570GB&#10;training data, drew a lot of attention due to the capacity of few-shot (even&#10;zero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is&#10;still challenging, as the training corpus of GPT-3 is primarily English, and&#10;the parameters are not publicly available. In this technical report, we release&#10;the Chinese Pre-trained Language Model (CPM) with generative pre-training on&#10;large-scale Chinese training data. To the best of our knowledge, CPM, with 2.6&#10;billion parameters and 100GB Chinese training data, is the largest Chinese&#10;pre-trained language model, which could facilitate several downstream Chinese&#10;NLP tasks, such as conversation, essay generation, cloze test, and language&#10;understanding. Extensive experiments demonstrate that CPM achieves strong&#10;performance on many NLP tasks in the settings of few-shot (even zero-shot)&#10;learning. The code and parameters are available at&#10;https://github.com/TsinghuaAI/CPM-Generate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.00823" label="2103.00823">
        <attvalues>
          <attvalue for="0" value="M6: A Chinese Multimodal Pretrainer" />
          <attvalue for="1" value="  In this work, we construct the largest dataset for multimodal pretraining in&#10;Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide&#10;range of domains. We propose a cross-modal pretraining method called M6,&#10;referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for&#10;unified pretraining on the data of single modality and multiple modalities. We&#10;scale the model size up to 10 billion and 100 billion parameters, and build the&#10;largest pretrained model in Chinese. We apply the model to a series of&#10;downstream applications, and demonstrate its outstanding performance in&#10;comparison with strong baselines. Furthermore, we specifically design a&#10;downstream task of text-guided image generation, and show that the finetuned M6&#10;can create high-quality images with high resolution and abundant details.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.02870" label="2110.02870">
        <attvalues>
          <attvalue for="0" value="Capturing Structural Locality in Non-parametric Language Models" />
          <attvalue for="1" value="  Structural locality is a ubiquitous feature of real-world datasets, wherein&#10;data points are organized into local hierarchies. Some examples include topical&#10;clusters in text or project hierarchies in source code repositories. In this&#10;paper, we explore utilizing this structural locality within non-parametric&#10;language models, which generate sequences that reference retrieved examples&#10;from an external source. We propose a simple yet effective approach for adding&#10;locality information into such models by adding learned parameters that improve&#10;the likelihood of retrieving examples from local neighborhoods. Experiments on&#10;two different domains, Java source code and Wikipedia text, demonstrate that&#10;locality features improve model efficacy over models without access to these&#10;features, with interesting differences. We also perform an analysis of how and&#10;where locality features contribute to improved performance and why the&#10;traditionally used contextual similarity metrics alone are not enough to grasp&#10;the locality structure.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models (LMs) predict a probability distribution over sequences, and are most widely studied to model and generate natural languages~\cite{bengio2003neural,merity2018regularizing,baevski2018adaptive,brown2020language}.&#10;Advances in LMs benefit many natural language processing downstream tasks, such as machine translation~\cite{bahdanau2014neural}, dialog systems~\cite{sordoni2015neural}, question answering~\cite{yang2019xlnet,raffel2019exploring}, and general representation learning for natural language~\cite{devlin2018bert,liu2019roberta}.&#10;Recently, LMs have also been adopted to model sequences other than text, such as source code written in programming language~\cite{hindle2016naturalness,hellendoorn2017deep,alon2020structural,karampatsis2020big}, which can enable useful downstream tasks like code completion~\cite{raychev2014code}.&#10;&#10;Most current neural LMs are based on parametric neural networks, using RNN~\cite{mikolov2010recurrent} or Transformer~\cite{vaswani2017attention} architectures.&#10;These models make predictions solely using a fixed set of neural network parameters.&#10;Recently, more and more neural LMs also incorporate non-parametric components~\cite{grave2017unbounded,guu2018generating,he2020learning,Khandelwal2020Generalization}, which usually first select examples from an external source and then reference them during the prediction.&#10;For example, \cite{Khandelwal2020Generalization} model the token-level probability by interpolating the parametric LM probability with a probability obtained from the nearest context-token pairs in an external datastore.&#10;Using such non-parametric components in LMs is beneficial because the model no longer needs to memorize everything about the language in its parameters.&#10;&#10;For such non-parametric LMs, one important concept is a distance metric between the current context and other contexts in the datastore.&#10;One example of such metric is the $\ell^2$ distance between context vectors calculated by the parametric model~\cite{Khandelwal2020Generalization}.&#10;This distance can be used in both retrieval and probability calculation; items in the datastore that are less distant from the current context are more likely to be retrieved and have a higher influence on the final probability.&#10;However, given that non-parametric datastores are typically very large, containing a myriad of contexts from disparate sources, calculating a metric that accurately reflects semantic similarities is non-trivial; as we demonstrate in experiments, there is much room for improvement in current practice.&#10;&#10;In this paper, we argue that the relevance of contexts may be correlated with not only contextual distance, but also structural characteristics of the underlying data.&#10;Specifically, we take advantage of a property we dub structural locality, the propensity of text to be divided into local groups sharing common hierarchical attributes.&#10;This property is ubiquitous across many kinds of texts and can provide additional information on how closely related two different examples are to each other.&#10;Throughout this paper, we will provide two case-studies of this phenomenon.&#10;First, in the domain of programs written in source code, if two source files originate from the same project, they are more likely to be related than files from other projects, and even more so if they are from the exact same package~\cite{hellendoorn2017deep}.&#10;Second, in natural language, two sections of Wikipedia text may be more related if they fall within the same topical domain, are from similarly titled sections, or even are from the same article (as in Figure~\ref{fig:model}).&#10;Notably this locality often manifests itself at different levels, such as the levels of ``project'', ``subdirectory'', and ``file'' cited above for source code.&#10;&#10;In this paper, we hypothesize that by using multiple levels of structural locality, we can better calibrate the distance metrics used to retrieve examples from non-parametric datastores, thereby improving LM performance.&#10;Specifically, we propose a simple-yet-effective approach that can easily be applied to non-parametric LMs: we use different levels of structural locality to define functions that modify the contextual distance metrics used by the non-parametric module.&#10;&#10;We evaluate our method on two drastically different domains: Java programming language source code, and natural language Wikipedia articles, achieving noticeable LM performance gains in both by adding just 5 \&amp; 7 parameters respectively.&#10;Moreover, we perform an in-depth analysis showing how the traditionally used contextual similarity metrics alone are not enough to grasp the locality structure, providing evidence for why adding the locality features is indeed useful.&#10;We also compare programming languages and natural languages to highlight several interesting differences in terms of how, and how much, the locality helps improve LM performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Non-Parametric Methods, Structural Locality, Linguistics, Language Models, Mathematics" />
        </attvalues>
      </node>
      <node id="1708.02182" label="1708.02182">
        <attvalues>
          <attvalue for="0" value="Regularizing and Optimizing LSTM Language Models" />
          <attvalue for="1" value="  Recurrent neural networks (RNNs), such as long short-term memory networks&#10;(LSTMs), serve as a fundamental building block for many sequence learning&#10;tasks, including machine translation, language modeling, and question&#10;answering. In this paper, we consider the specific problem of word-level&#10;language modeling and investigate strategies for regularizing and optimizing&#10;LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on&#10;hidden-to-hidden weights as a form of recurrent regularization. Further, we&#10;introduce NT-ASGD, a variant of the averaged stochastic gradient method,&#10;wherein the averaging trigger is determined using a non-monotonic condition as&#10;opposed to being tuned by the user. Using these and other regularization&#10;strategies, we achieve state-of-the-art word level perplexities on two data&#10;sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the&#10;effectiveness of a neural cache in conjunction with our proposed model, we&#10;achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and&#10;52.0 on WikiText-2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.06714" label="1506.06714">
        <attvalues>
          <attvalue for="0" value="A Neural Network Approach to Context-Sensitive Generation of&#10;  Conversational Responses" />
          <attvalue for="1" value="  We present a novel response generation system that can be trained end to end&#10;on large quantities of unstructured Twitter conversations. A neural network&#10;architecture is used to address sparsity issues that arise when integrating&#10;contextual information into classic statistical models, allowing the system to&#10;take into account previous dialog utterances. Our dynamic-context generative&#10;models show consistent gains over both context-sensitive and&#10;non-context-sensitive Machine Translation and Information Retrieval baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.00577" label="1910.00577">
        <attvalues>
          <attvalue for="0" value="Structural Language Models of Code" />
          <attvalue for="1" value="  We address the problem of any-code completion - generating a missing piece of&#10;source code in a given program without any restriction on the vocabulary or&#10;structure. We introduce a new approach to any-code completion that leverages&#10;the strict syntax of programming languages to model a code snippet as a tree -&#10;structural language modeling (SLM). SLM estimates the probability of the&#10;program's abstract syntax tree (AST) by decomposing it into a product of&#10;conditional probabilities over its nodes. We present a neural model that&#10;computes these conditional probabilities by considering all AST paths leading&#10;to a target node. Unlike previous techniques that have severely restricted the&#10;kinds of expressions that can be generated in this task, our approach can&#10;generate arbitrary code in any programming language. Our model significantly&#10;outperforms both seq2seq and a variety of structured approaches in generating&#10;Java and C# code. Our code, data, and trained models are available at&#10;http://github.com/tech-srl/slm-code-generation/ . An online demo is available&#10;at http://AnyCodeGen.org .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.08878" label="1709.08878">
        <attvalues>
          <attvalue for="0" value="Generating Sentences by Editing Prototypes" />
          <attvalue for="1" value="  We propose a new generative model of sentences that first samples a prototype&#10;sentence from the training corpus and then edits it into a new sentence.&#10;Compared to traditional models that generate from scratch either left-to-right&#10;or by first sampling a latent sentence vector, our prototype-then-edit model&#10;improves perplexity on language modeling and generates higher quality outputs&#10;according to human evaluation. Furthermore, the model gives rise to a latent&#10;edit vector that captures interpretable semantics such as sentence similarity&#10;and sentence-level analogies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.16336" label="2006.16336">
        <attvalues>
          <attvalue for="0" value="Learning Sparse Prototypes for Text Generation" />
          <attvalue for="1" value="  Prototype-driven text generation uses non-parametric models that first choose&#10;from a library of sentence &quot;prototypes&quot; and then modify the prototype to&#10;generate the output text. While effective, these methods are inefficient at&#10;test time as a result of needing to store and index the entire training corpus.&#10;Further, existing methods often require heuristics to identify which prototypes&#10;to reference at training time. In this paper, we propose a novel generative&#10;model that automatically learns a sparse prototype support set that,&#10;nonetheless, achieves strong language modeling performance. This is achieved by&#10;(1) imposing a sparsity-inducing prior on the prototype selection distribution,&#10;and (2) utilizing amortized variational inference to learn a prototype&#10;retrieval function. In experiments, our model outperforms previous&#10;prototype-driven language models while achieving up to a 1000x memory&#10;reduction, as well as a 1000x speed-up at test time. More interestingly, we&#10;show that the learned prototypes are able to capture semantics and syntax at&#10;different granularity as we vary the sparsity of prototype selection, and that&#10;certain sentence attributes can be controlled by specifying the prototype for&#10;generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.06983" label="2305.06983">
        <attvalues>
          <attvalue for="0" value="Active Retrieval Augmented Generation" />
          <attvalue for="1" value="  Despite the remarkable ability of large language models (LMs) to comprehend&#10;and generate language, they have a tendency to hallucinate and create factually&#10;inaccurate output. Augmenting LMs by retrieving information from external&#10;knowledge resources is one promising solution. Most existing retrieval&#10;augmented LMs employ a retrieve-and-generate setup that only retrieves&#10;information once based on the input. This is limiting, however, in more general&#10;scenarios involving generation of long texts, where continually gathering&#10;information throughout generation is essential. In this work, we provide a&#10;generalized view of active retrieval augmented generation, methods that&#10;actively decide when and what to retrieve across the course of the generation.&#10;We propose Forward-Looking Active REtrieval augmented generation (FLARE), a&#10;generic method which iteratively uses a prediction of the upcoming sentence to&#10;anticipate future content, which is then utilized as a query to retrieve&#10;relevant documents to regenerate the sentence if it contains low-confidence&#10;tokens. We test FLARE along with baselines comprehensively over 4 long-form&#10;knowledge-intensive generation tasks/datasets. FLARE achieves superior or&#10;competitive performance on all tasks, demonstrating the effectiveness of our&#10;method. Code and datasets are available at https://github.com/jzbjyb/FLARE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;We refer to \ref{sec:baseline_single} and \ref{sec:baseline_multi} for extensively discussion on single-time and multi-time retrieval augmented LMs, which is the most relevant area to this paper.&#10;&#10;Iterative and adaptive retrieval&#10;Iterative retrieval and refinement has been studied in both text and code generation tasks \cite{peng-check-2023,repocoder-jiang-2023,zemlyanskiy-gr-2022,retfeed-yu-2023}.&#10;\ours differs from these methods in the granularity of generation and retrieval strategies.&#10;Adaptive retrieval has been studied in single-time retrieval scenarios based on either question popularity or generation probabilities \cite{mallen-nottrust-2022,oyster-li-2023}, while we focus on long-form generation requiring active information access.&#10;Browser-enhanced LMs WebGPT \cite{webgpt-nakano-2021} and WebCPM \cite{webcpm-qin-2023} train LMs to interact with browser to enhance factuality using reinforcement learning or supervised training where multiple queries can be triggered before generation.&#10;\ours is built on text-based retrievers but can be combined with a browser to potentially improve retrieval quality.&#10;&#10;" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Retrieval Augmentation Methods, Language Model Limitations, Artificial Intelligence, Active Generation Techniques" />
        </attvalues>
      </node>
      <node id="2312.15918" label="2312.15918">
        <attvalues>
          <attvalue for="0" value="Supervised Knowledge Makes Large Language Models Better In-context&#10;  Learners" />
          <attvalue for="1" value="  Large Language Models (LLMs) exhibit emerging in-context learning abilities&#10;through prompt engineering. The recent progress in large-scale generative&#10;models has further expanded their use in real-world language applications.&#10;However, the critical challenge of improving the generalizability and&#10;factuality of LLMs in natural language understanding and question answering&#10;remains under-explored. While previous in-context learning research has focused&#10;on enhancing models to adhere to users' specific instructions and quality&#10;expectations, and to avoid undesired outputs, little to no work has explored&#10;the use of task-Specific fine-tuned Language Models (SLMs) to improve LLMs'&#10;in-context learning during the inference stage. Our primary contribution is the&#10;establishment of a simple yet effective framework that enhances the reliability&#10;of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs&#10;benefit from discriminative models, and 3) minimizes hallucinations in&#10;generative tasks. Using our proposed plug-in method, enhanced versions of Llama&#10;2 and ChatGPT surpass their original versions regarding generalizability and&#10;factuality. We offer a comprehensive suite of resources, including 16 curated&#10;datasets, prompts, model checkpoints, and LLM outputs across 9 distinct tasks.&#10;The code and data are released at:&#10;https://github.com/YangLinyi/Supervised-Knowledge-Makes-Large-Language-Models-Better-In-context-Learners.&#10;Our empirical analysis sheds light on the advantages of incorporating&#10;discriminative models into LLMs and highlights the potential of our methodology&#10;in fostering more reliable LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In-context Learning. Scaling up pre-trained language models stimulates the in-context learning ability is first introduced by GPT-3 \cite{gpt3}, introducing the potential to accurately comprehend instructions and complete complex tasks with no supervision \cite{chowdhery2022palm,openai2023gpt4,sun2023evaluating}. As evidenced by previous work \cite{shwartz2020unsupervised,nye2021show,perez2021true}, the ICL performance can be significantly enhanced by incorporating auxiliary knowledge or reasoning instructions in a prompt, such as Chain-of-Thought (COT) \cite{wei2022chain} and Tree-of-Thoughts (TOT) \cite{yao2023tree}. However, such a multi-step reasoning process could be tedious and expensive to use (assuming we perform ICL for GPT-4), whereas our method is cost-efficient since the supervised knowledge occupies only a short length in the prompt. &#10;&#10;There is a line of work for improving the in-context learning performance by either constructing demonstrations \cite{arora2022ask,si2022prompting,lyu2022z,gu-etal-2023-pre,ye2023fid,dhuliawala2023chain} or framing an exploration of example selection methods \cite{wu-etal-2023-self,wang2023large,sun2023evaluating,agrawal2022context,wang2022exploiting,wang2022usb,lu2022rationale,wang2023pandalm} and even order \cite{lu2021fantastically,zhao2021calibrate,liu2021makes,liu2023lost}. The contrastive decoding method \cite{li2023contrastive} considers the assistance smaller language model but requires external computation. Differently, SuperContext demonstrates its superior performance on OOD test data in a cost-effective manner.&#10;&#10;Our work is also connected with work focusing on understanding and explaining in-context learning from different perspectives, including the implicit Bayesian Inference \cite{xie2021explanation}, pre-training data \cite{han-etal-2023-understanding,pan-etal-2023-context}, and information compression \cite{wang2023label,wu-etal-2023-self}. Different ways of understanding ICL in realistic NLP tasks have been proposed before \cite{min2022rethinking,dong2022survey,wang2023large}, the interpretation part in SuperContext aims to answer how LLMs recall in-context examples and output rationale.&#10;&#10;Knowledge in Context. Using external knowledge as auxiliary information to assist LLMs in providing truthful and timely responses represents an emerging solution \cite{mialon2023augmented,xiao2023freeal} in recent. Traditional retrieve-based methods \cite{rubin2021learning,ni2021large,king2023diverse} require a knowledge retriever as the prior step for guiding the generation of responses. Besides, the external knowledge source could extend beyond local documents to encompass the entire Internet \cite{ni2021large,gao2023rarr}. In addition, LLMs can leverage special plug-ins to improve their capabilities, such as Toolformer \cite{schick2023toolformer} and LangChain \cite{Chase_LangChain_2022} for calling external APIs, and HuggingGPT \cite{shen2023hugginggpt} for using models.&#10;&#10;Previous work either relies on web information and search engines for gaining external knowledge \cite{yu2023kola} or accomplishes planning tasks outside the NLP scope. \cite{xu2023small} evaluates the efficacy of small language models as plug-ins under an in-domain setting using GLUE and lacks an interpretation part to explain the reasons. SuperContext shares a conceptual similarity with SuperICL \cite{xu2023small} and HuggingGPT \cite{shen2023hugginggpt} in leveraging language model architectures. However, the key distinction lies in our approach's application and analysis under out-of-distribution (OOD) conditions, a less explored area in the existing literature.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Generalizability, Cognitive Science, Large-Scale Generative Models, Natural Language Understanding, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2406.13282" label="2406.13282">
        <attvalues>
          <attvalue for="0" value="Understanding the RoPE Extensions of Long-Context LLMs: An Attention&#10;  Perspective" />
          <attvalue for="1" value="  Enabling LLMs to handle lengthy context is currently a research hotspot. Most&#10;LLMs are built upon rotary position embedding (RoPE), a popular position&#10;encoding method. Therefore, a prominent path is to extrapolate the RoPE trained&#10;on comparably short texts to far longer texts. A heavy bunch of efforts have&#10;been dedicated to boosting the extrapolation via extending the formulations of&#10;the RoPE, however, few of them have attempted to showcase their inner workings&#10;comprehensively. In this paper, we are driven to offer a straightforward yet&#10;in-depth understanding of RoPE extensions from an attention perspective and on&#10;two benchmarking tasks. A broad array of experiments reveals several valuable&#10;findings: 1) Maintaining attention patterns to those at the pretrained length&#10;improves extrapolation; 2) Large attention uncertainty leads to retrieval&#10;errors; 3) Using longer continual pretraining lengths for RoPE extensions could&#10;reduce attention uncertainty and significantly enhance extrapolation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) \cite{radford2018improving,touvron2023llama,zhang2023towards,li2024tf,zhang2024modification,zhang2024dynamic} have accommodated a wide range of natural language processing applications, such as code completion \cite{rozière2023code} and question answering \cite{kamalloo2023evaluating,jiang2021can,su2019generalizing}. &#10;However, a notable challenge limiting further customization is possibly the inability of LLMs to utilize context beyond the pretrained length \cite{minaee2024large,chen2023extending} due to the inherent flaw of rotary position embedding (RoPE) being used.&#10;Fortunately, RoPE extensions emerge as key ingredients to enabling LLMs to leverage extended context that exceeds pretrained scope \cite{chen2023extending,peng2023yarn,liu2023scaling,han2023lm,rozière2023code}. &#10;These RoPE extensions focus on improving performance on long texts, yet frustratingly, only a few of them \cite{liu2023scaling,han2023lm,men2024base} have explored the underlying mechanisms in depth.&#10;&#10;Thus, we systematically analyze common RoPE extensions more straightforwardly, from the perspective of attention \cite{vaswani2017attention}. We include three widely-used RoPE extensions, i.e., position interpolation~\cite{chen2023extending}, YaRN~\cite{peng2023yarn}, and NTK-Aware interpolation~\cite{rozière2023code}. To our best knowledge, there is simply no research in {understanding RoPE extensions for long-context models thoroughly from an attention perspective}.&#10;&#10;As a start, we strive to primarily study these methods on a long-context perplexity test (PPL) and empirically compare their corresponding attention patterns.&#10;We found that finetuning LLMs with these RoPE-extension methods which match the original pretraining length improves extrapolation performance. Particularly with the NTK-Aware interpolation method, one can extrapolate up to 32$\times$ beyond the pretrained length.&#10;To unleash the reasons behind the successes of these methods, we collect the attention scores respectively distributed in 2K and 8K lengths during inference. &#10;The results demonstrate that these methods maintain attention patterns consistent with those observed at the pretrained length.&#10;In contrast, the attention patterns of the RoPE are substantially deviated. &#10;&#10;Afterward, following literature~\cite{fu2024data}, we examine these RoPE extensions on a more challenging long-context test called Needle-in-a-Haystack (Needle)~\cite{needleinhaystack}.&#10;We find that the RoPE extensions could pass more tests than the RoPE does. &#10;Nonetheless, as the context length increased, the RoPE extensions could hardly locate the needles. &#10;We associate the observation with attention uncertainty.&#10;We uncover that large uncertainty leads to retrieval errors: the positions that incur large attention uncertainty are exactly where the incorrect answers are borrowed from. &#10;&#10;We further hypothesize that this large attention uncertainty stems from a mismatch between the context lengths in training and inference.&#10;Inspired by the conjecture, a natural way to ease the mismatch is to directly train on longer texts. &#10;Experimental results exhibit that, with the same amount of training tokens consumed, using examples with longer contexts largely alleviates uncertainty. Thereby, the ability to digest long texts is promoted.&#10;&#10;Our key contributions can be summarized as follows:&#10;\begin{itemize}&#10; \item We study various RoPE extensions for length extrapolation in perplexity testing and find that the effectiveness could be yielded from maintaining the original attention patterns.&#10;&#10;\item We analyze these methods using advanced Needle testing and observe that they may fail to extrapolate to regions where large attention uncertainty persists.&#10;&#10;\item We hypothesize that large attention uncertainty stems from a context length mismatch between training and inference. It is possible to reduce this large uncertainty by minimizing the mismatch through continual training with lengths closer to those in inference.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Large Language Models, Linguistics, Position Encoding Methods, Artificial Intelligence, Mathematics, Attention Mechanisms" />
        </attvalues>
      </node>
      <node id="2311.07052" label="2311.07052">
        <attvalues>
          <attvalue for="0" value="Towards the Law of Capacity Gap in Distilling Language Models" />
          <attvalue for="1" value="  Language model (LM) distillation is a trending area that aims to distil the&#10;knowledge residing in a large teacher LM to a small student one. While various&#10;methods have been proposed to maximize the effectiveness of the distillation,&#10;significant challenges persist, particularly when there is a substantial&#10;capacity gap between the teacher and student LMs. This issue, often referred to&#10;as the \textit{curse} of capacity gap, suggests that a larger teacher does not&#10;necessarily result in a superior student compared to one distilled from a&#10;smaller teacher. In other words, there is likely an optimal teacher yielding&#10;the best student along the scaling course of the teacher. However, the curse of&#10;capacity gap can not be tackled without notable compute overhead, as indicated&#10;in previous studies. In the context of large LMs (LLMs), previously viable&#10;approaches become much less meaningful, as it is an impossible triangle to&#10;distill an expected student from an optimal teacher student with small compute&#10;overhead. Fortunately, the impossible triangle can fortunately be possible&#10;provided an inducted \textit{law} of capacity gap. In this paper, we take the&#10;spirits of scaling law and reveal that the optimal teacher scale almost&#10;consistently follows a linear scaling with the student scale across different&#10;model architectures and data scales. The law later guides us to distil a 3B&#10;student LM (termed \textsc{MiniMA}) from LLaMA2-7B. \textsc{MiniMA} is&#10;demonstrated to outperform a wide range of 3B competitors and could even&#10;compete with several 7B models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2308.12950" label="2308.12950">
        <attvalues>
          <attvalue for="0" value="Code Llama: Open Foundation Models for Code" />
          <attvalue for="1" value="  We release Code Llama, a family of large language models for code based on&#10;Llama 2 providing state-of-the-art performance among open models, infilling&#10;capabilities, support for large input contexts, and zero-shot instruction&#10;following ability for programming tasks. We provide multiple flavors to cover a&#10;wide range of applications: foundation models (Code Llama), Python&#10;specializations (Code Llama - Python), and instruction-following models (Code&#10;Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are&#10;trained on sequences of 16k tokens and show improvements on inputs with up to&#10;100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants&#10;support infilling based on surrounding content. Code Llama reaches&#10;state-of-the-art performance among open models on several code benchmarks, with&#10;scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code&#10;Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our&#10;models outperform every other publicly available model on MultiPL-E. We release&#10;Code Llama under a permissive license that allows for both research and&#10;commercial use.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.06984" label="2305.06984">
        <attvalues>
          <attvalue for="0" value="Evaluating Open-Domain Question Answering in the Era of Large Language&#10;  Models" />
          <attvalue for="1" value="  Lexical matching remains the de facto evaluation method for open-domain&#10;question answering (QA). Unfortunately, lexical matching fails completely when&#10;a plausible candidate answer does not appear in the list of gold answers, which&#10;is increasingly the case as we shift from extractive to generative models. The&#10;recent success of large language models (LLMs) for QA aggravates lexical&#10;matching failures since candidate answers become longer, thereby making&#10;matching with the gold answers even more challenging. Without accurate&#10;evaluation, the true progress in open-domain QA remains unknown. In this paper,&#10;we conduct a thorough analysis of various open-domain QA models, including&#10;LLMs, by manually evaluating their answers on a subset of NQ-open, a popular&#10;benchmark. Our assessments reveal that while the true performance of all models&#10;is significantly underestimated, the performance of the InstructGPT (zero-shot)&#10;LLM increases by nearly +60%, making it on par with existing top models, and&#10;the InstructGPT (few-shot) model actually achieves a new state-of-the-art on&#10;NQ-open. We also find that more than 50% of lexical matching failures are&#10;attributed to semantically equivalent answers. We further demonstrate that&#10;regex matching ranks QA models consistent with human judgments, although still&#10;suffering from unnecessary strictness. Finally, we demonstrate that automated&#10;evaluation models are a reasonable surrogate for lexical matching in some&#10;circumstances, but not for long-form answers generated by LLMs. The automated&#10;models struggle in detecting hallucinations in LLM answers and are thus unable&#10;to evaluate LLMs. At this time, there appears to be no substitute for human&#10;evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.00071" label="2309.00071">
        <attvalues>
          <attvalue for="0" value="YaRN: Efficient Context Window Extension of Large Language Models" />
          <attvalue for="1" value="  Rotary Position Embeddings (RoPE) have been shown to effectively encode&#10;positional information in transformer-based language models. However, these&#10;models fail to generalize past the sequence length they were trained on. We&#10;present YaRN (Yet another RoPE extensioN method), a compute-efficient method to&#10;extend the context window of such models, requiring 10x less tokens and 2.5x&#10;less training steps than previous methods. Using YaRN, we show that LLaMA&#10;models can effectively utilize and extrapolate to context lengths much longer&#10;than their original pre-training would allow, while also surpassing previous&#10;the state-of-the-art at context window extension. In addition, we demonstrate&#10;that YaRN exhibits the capability to extrapolate beyond the limited context of&#10;a fine-tuning dataset. The models fine-tuned using YaRN has been made available&#10;and reproduced online up to 128k context length at&#10;https://github.com/jquesnelle/yarn&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;ReRoPE~\cite{rerope2023} also aims to extend the context size of existing models pre-trained with RoPE, and claims &quot;infinite&quot; context length without needing any fine-tuning.&#10;This claim is backed by a monotonically decreasing loss with increasing context length up to 16k on the Llama 2 13B model.&#10;It achieves context extension by modifying the attention mechanism and thus is not purely an embedding interpolation method.&#10;Since it is currently not compatible with Flash Attention~2~\cite{dao2023flashattention2} and requires two attention passes during inference, we do not consider it for comparison.&#10;&#10;Concurrently with our work, LM-Infinite~\cite{han2023lminfinite} proposes similar ideas to YaRN, but focuses on &quot;on-the-fly&quot; length generalization for non-fine-tuned models.&#10;Since they also modify the attention mechanism of the models, it is not an embedding interpolation method and is not immediately compatible with Flash Attention 2.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2405.14591" label="2405.14591">
        <attvalues>
          <attvalue for="0" value="Base of RoPE Bounds Context Length" />
          <attvalue for="1" value="  Position embedding is a core component of current Large Language Models&#10;(LLMs). Rotary position embedding (RoPE), a technique that encodes the position&#10;information with a rotation matrix, has been the de facto choice for position&#10;embedding in many LLMs, such as the Llama series. RoPE has been further&#10;utilized to extend long context capability, which is roughly based on adjusting&#10;the \textit{base} parameter of RoPE to mitigate out-of-distribution (OOD)&#10;problems in position embedding. However, in this paper, we find that LLMs may&#10;obtain a superficial long-context ability based on the OOD theory. We revisit&#10;the role of RoPE in LLMs and propose a novel property of long-term decay, we&#10;derive that the \textit{base of RoPE bounds context length}: there is an&#10;absolute lower bound for the base value to obtain certain context length&#10;capability. Our work reveals the relationship between context length and RoPE&#10;base both theoretically and empirically, which may shed light on future long&#10;context training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#09;In this section, we first introduce the Transformer and RoPE, which are most commonly used in current LLMs. Then we discuss long context methods based on the OOD of rotation angle theory.&#10;&#09;" />
        </attvalues>
      </node>
      <node id="2402.10171" label="2402.10171">
        <attvalues>
          <attvalue for="0" value="Data Engineering for Scaling Language Models to 128K Context" />
          <attvalue for="1" value="  We study the continual pretraining recipe for scaling language models'&#10;context lengths to 128K, with a focus on data engineering. We hypothesize that&#10;long context modeling, in particular \textit{the ability to utilize information&#10;at arbitrary input locations}, is a capability that is mostly already acquired&#10;through large-scale pretraining, and that this capability can be readily&#10;extended to contexts substantially longer than seen during training~(e.g., 4K&#10;to 128K) through lightweight continual pretraining on appropriate data mixture.&#10;We investigate the \textit{quantity} and \textit{quality} of the data for&#10;continual pretraining: (1) for quantity, we show that 500 million to 5 billion&#10;tokens are enough to enable the model to retrieve information anywhere within&#10;the 128K context; (2) for quality, our results equally emphasize \textit{domain&#10;balance} and \textit{length upsampling}. Concretely, we find that naively&#10;upsampling longer data on certain domains like books, a common practice of&#10;existing work, gives suboptimal performance, and that a balanced domain mixture&#10;is important. We demonstrate that continual pretraining of the full model on&#10;1B-5B tokens of such data is an effective and affordable strategy for scaling&#10;the context length of language models to 128K. Our recipe outperforms strong&#10;open-source long-context models and closes the gap to frontier models like&#10;GPT-4 128K.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Frontier language models feature extremely long context lengths, such as OpenAI's GPT-4 Turbo 128K (Nov 2023) and Anthropic's Claude-100K (May 2023). &#10;In the regime of 100K, a wide array of new applications emerge, such as repo-level code understanding~\cite{bairi2023codeplan}, long-history dialog modeling~\cite{mazumder2024lifelong}, and language model-powered autonomous agents~\cite{weng2023prompt}.&#10;A recent testbed for long-range capabilities is the Needle-in-a-Haystack benchmark, first proposed by~\cite{needleinhaystack} in Nov 2023. This benchmark asks the language model to recite the information in a ``needle'' sentence (``The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day'') that is randomly inserted at an arbitrary location in a long essay.&#10;Since its release, it has become a popular sandbox for testing whether models can utilize 100K+ context lengths, as it differentiates models that can precisely recite the given information at arbitrary input location versus those models that cannot.&#10;So far there is neither public knowledge nor open-source work about achieving precise retrieval anywhere at this scale to the best of our knowledge. &#10;&#10;This work improves the long-context capability over strong open-source baselines, and closes the gap to GPT-4 on the Needle-in-a-Haystack benchmark, as demonstrated in Fig.~\ref{fig:needle_in_a_haystack}.&#10;Our baselines here include LMSys' LongChat v1.5 32K~\cite{longchat2023}, Together Compute's LLaMA-2 32K~\cite{togetherllama32k}, YaRN Mistral 128K~\cite{peng2023yarn}, and LongLoRA~\cite{chen2023longlora}, which are so far the top open-source long-context language models. &#10;These works focus on different aspects of long-context language modeling. For example, YaRN~\cite{peng2023yarn} focuses on positional embeddings, LongLoRA~\cite{chen2023longlora} focuses on efficient attention, and Together~\cite{togetherllama32k} focuses on a full-stack solution. &#10;Our work focuses on data-engineering, and identifies critical data aspects for extending language models' long-term dependency modeling to much longer contexts than seen during regular pretraining. &#10;&#10;The major differences between this work and existing work are listed on Table~\ref{tab:design_diff}.&#10;Together's LLaMA-2 is trained on 32K, but only generalizes to about 40K length. &#10;YaRN Mistral is trained on book-only data; later in Table~\ref{tab:loss_7b} we will show that improvements on one domain has limited transfer to other domains, indicating that one should consider a balanced mixture of different domains.&#10;LongLoRA does not upsample long sequences; later in Fig.~\ref{fig:original_vs_per_source} we will show that upsampling long sequences is critical for precise retrieval. &#10;These details are relatively hard to notice, yet we believe that they are the important for improved performance. &#10;Before the Needle-in-a-Haystack test, most existing works use test negative log-likelihood as evaluation, &#10;effectively {concealing} the underlying differences beyond low loss.&#10;We show that, similar test loss could result in substantially different behavior when performing precise retrieval (Fig.~\ref{fig:original_vs_per_source}). &#10;&#10;Another important related work is the previous LLaMA Long~\cite{xiong2023effective} work and the concurrent XVERSE~\cite{xverse} work, which continue pretraining the model on 32K sequences for about {500 billion tokens}. &#10;These works are implicitly motivated by the view that long-context modeling is a new capability that must be ``injected'' through large-scale training.&#10;We instead hypothesize that the base model has mostly already acquired this capability through large-scale pretraining, and thus a lightweight continual pretraining on relatively small data (e.g., 5B tokens) is enough to extend these capabilities to much longer context lengths (Fig.~\ref{fig:needle_data_scaling}).&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2307.10236" label="2307.10236">
        <attvalues>
          <attvalue for="0" value="Look Before You Leap: An Exploratory Study of Uncertainty Measurement&#10;  for Large Language Models" />
          <attvalue for="1" value="  The recent performance leap of Large Language Models (LLMs) opens up new&#10;opportunities across numerous industrial applications and domains. However,&#10;erroneous generations, such as false predictions, misinformation, and&#10;hallucination made by LLMs, have also raised severe concerns for the&#10;trustworthiness of LLMs', especially in safety-, security- and&#10;reliability-sensitive scenarios, potentially hindering real-world adoptions.&#10;While uncertainty estimation has shown its potential for interpreting the&#10;prediction risks made by general machine learning (ML) models, little is known&#10;about whether and to what extent it can help explore an LLM's capabilities and&#10;counteract its undesired behavior. To bridge the gap, in this paper, we&#10;initiate an exploratory study on the risk assessment of LLMs from the lens of&#10;uncertainty. In particular, we experiment with twelve uncertainty estimation&#10;methods and four LLMs on four prominent natural language processing (NLP) tasks&#10;to investigate to what extent uncertainty estimation techniques could help&#10;characterize the prediction risks of LLMs. Our findings validate the&#10;effectiveness of uncertainty estimation for revealing LLMs'&#10;uncertain/non-factual predictions. In addition to general NLP tasks, we&#10;extensively conduct experiments with four LLMs for code generation on two&#10;datasets. We find that uncertainty estimation can potentially uncover buggy&#10;programs generated by LLMs. Insights from our study shed light on future design&#10;and development for reliable LLMs, facilitating further research toward&#10;enhancing the trustworthiness of LLMs.&#10;" />
          <attvalue for="2" value="&#10;&#10;L{arge} Language Models (LLMs) have demonstrated impressive capabilities in miscellaneous Natural Language Processing (NLP) tasks and promising adaptability in practical applications across diverse domains, including but not limited to content moderation~\cite{markov2023holistic}, code generation~\cite{roziere2023code}, conversational AI~\cite{chatgpt2023}, and personalized content recommendations~\cite{wu2023survey}. The scale of deployment is vast, addressing the needs of diverse user demographics and industries. As a prominent example, Meta has launched foundation models such as the Llama family~\cite{touvron2023llama, touvron2023llama2}. By September 2023, these models had driven the creation of over 3,500 enterprise projects and inspired more than 7,000 GitHub repositories~~\cite{meta2023llamaecosystem}.&#10;&#10;Despite the attractive performance that LLMs present and their rapid evolution within both academics and industries, an urgent common concern about LLMs has the propensity of generating erroneous information without warning. &#10;Such phenomenon of erroneous generation can exhibit in terms of different manifestations (\eg, hallucination \cite{ji2023survey}, disinformation \cite{tamkin2021understanding}, bias \cite{abid2021persistent}) across various tasks.&#10;In general, the current LLMs are found to have the tendency to generate problematic, nonfactual responses that are not from training sources or misguided by biased data. However, these responses are often presented in a natural human-like tone \cite{bang2023multitask, johnson2023assessing}. &#10;Such characteristics cause erroneous information to be highly mixed and intertwined with confident and factual contexts, making their detection and localization difficult without close inspection and diligent fact-checking \cite{bubeck2023sparks}.&#10;As an example, Fig.~\ref{fig:bg:example_QA} (GPT-3) depicts an example of an LLM answering a question with nonfactual information.&#10;&#10;Risk assessments thus become crucial in the process of mitigating such threats. A recent survey highlights that 98\% of respondents, encompassing domain experts and civil society members, firmly believe that AGI (artificial general intelligence) labs should undertake risk assessments before deployment~\cite{schuett2023towards}. For the AI industry, implementing comprehensive risk assessment methods is not just a technical necessity but also an ethical obligation. Major tech corporations~\cite{anderljung2023frontier} such as Microsoft~\cite{microsoft2022responsibleai}, OpenAI~\cite{openai2023aisafety}, Amazon~\cite{amazon2023responsibleai}, and Google~\cite{google2023responsibleai}, along with non-governmental organizations (NGOs, \eg, the Centre for the Governance of AI~\cite{koessler2023risk}), are fervently working towards developing safe, secure, transparent, reliable and responsible LLMs and AGI applications. As a driving force behind open and collaborative AI research, Meta also commits substantial resources to the development of responsible AI, emphasizing trustworthiness, transparency, robustness, etc.~\cite{meta2021fivepillars, meta2023responsibleai}. These endeavors encompass a range of reports (\eg, Building Generative AI Responsibly~\cite{meta2023generativeai}), open-source tools (\eg, the model interpretability framework Captum~\cite{kokhlikyan2020captum}), and datasets (\eg, Hateful Memes~\cite{kiela2021hateful}).&#10;&#10;Although there has been substantial work on other AI models, risk assessments for LLMs are still in their infancy. Due to their billions of parameters, vast amounts of (often inaccessible) data, and potential closed-source nature, LLMs present significant challenges for analysis and safeguarding. Yet, such a safeguard is crucial, especially considering the widespread adoption of LLMs. Stakeholders from research communities, industry, open source initiatives, NGOs, and businesses may all be negatively affected by untrustworthy LLMs. Uncertainty estimation, aimed at gauging the confidence level of model outputs~\cite{bhatt2021uncertainty, hullermeier2021aleatoric, rahmati2019predicting}, stands out as a promising approach for identifying risks in general Machine Learning (ML) models. Such techniques also have the potential for detecting erroneous generation from LLMs~\cite{manakul2023selfcheckgpt} even under black-box settings. {It is thus possible to take them as plug-and-play tools in both academic and industrial scenarios.} For example, Fig.~\ref{fig:bg:example_QA} shows that a higher uncertainty score could possibly indicate an erroneous generation of an LLM. {However, it is still unclear whether and to what extent uncertainty estimation methods could do when measuring and characterizing an LLM's capability limitations. Furthermore, it also raises questions such as ``Are there better practices for employing these methods in practical scenarios'',} {``Do we need further adaptations from the industrial perspective to better cater to the distinct features of LLMs (\eg, task diversity and high computational cost)''}, \etc.&#10;To the best of our knowledge, up to the present, there is a lack of a general framework that integrates different uncertainty estimation methods for LLMs, as well as a systematic study to investigate the effectiveness of uncertainty estimation in characterizing an LLMs' capabilities. &#10;&#10;To bridge this gap, in this paper, we present an exploratory study to understand the trustworthiness of LLMs from the lens of uncertainty estimation.&#10;Considering the generality and versatility for various application scenarios, we strive to identify suitable methods to minimize the requirement of LLMs internal information (\eg, model architecture, model parameters). {Such criteria enable these methods to be seamlessly adapted and incorporated by end-users of commercial models, such as GPT-3.5 and GPT-4}. Overall, we collected and implemented as many as 12 representative uncertainty estimation methods that were originally designed for general DNNs and successfully adapted them to the contexts of LLM applications.&#10;To better capture an in-depth understanding of the effectiveness of these methods, we conducted large-scale experiments with as many as \responseline{twelve LLMs on both NLP (\ie, question answering, text summarization, machine translation) and software programming (\ie, code generation) tasks to analyze the correlation between uncertainty estimation results and LLMs performance. The models comprise three from MetaAI, four from OpenAI, one from Google, one from Microsoft, one from Alibaba, one from DeepSeek, and one from BigCode. We also evaluated five older LLMs (\eg, GPT-2, LLaMA, Codegen, Incoder, and Santacoder), with the corresponding results available on our website. In total, we have evaluated 17 LLMs} The overall workflow of our work is shown in Fig.~\ref{fig:overall_workflow}. In particular, we investigate the following research questions:&#10;&#10;\begin{itemize}[leftmargin=*]&#10;&#10;\item {RQ1:} \rqone&#10;&#10;\item {RQ2:} \rqtwo&#10;&#10;\item {RQ3:} \rqthree&#10;&#10;\item {RQ4:} \rqfour&#10;&#10;\end{itemize}&#10;&#10;Our findings validate that uncertainty measurement can, to an extent, be helpful in detecting erroneous responses in general NLP tasks. Additionally, it has also shown to be promising as an indicator for pinpointing faulty programs produced by LLMs.&#10;{Even though, these methods might fall short in detecting nuanced errors made by high-performance commercial models. They seem better suited for filtering out more blatant mistakes.}&#10;Based on the results, we further discuss the insights from our study and highlight a few potential research directions of leveraging uncertainty estimation to enhance the trustworthiness of LLMs for real-world applications across domains. &#10;First, research efforts are needed with novel uncertainty estimation techniques exclusively for LLMs to better fit the corresponding diverse task-handling ability.&#10;{Second, we observe that different LLMs can sometimes display markedly distinct uncertain behaviors. Consequently, even though these methods are inherently model-agnostic, stakeholders may need to undertake model-specific optimizations to achieve enhanced performance. Furthermore, we observe that the prompt template used in the reinforcement learning from human feedback (RLHF~\cite{ouyang2022training}) could potentially impact the accuracy of uncertainty estimation.}&#10;&#10;The contributions of this paper are summarized as follows:&#10;&#10;\begin{itemize}[leftmargin=*]&#10;&#10;\item We collected and implemented twelve different uncertainty estimation methods that are successfully adapted to enable the analysis of LLM, which are also applicable to {both open-source and closed-source LLM models across different downstream tasks in the grey-box setting.}&#10;&#10;\item We conducted a large-scale evaluation with nine LLMs on six tasks from four different domains.&#10;&#10;\item We provided an in-depth analysis of the challenges in existing uncertainty methods for LLMs and distilled a set of implications and future opportunities toward reliable and trustworthy LLMs.&#10; \item Our toolkit, encompassing the dataset, LLM inference, and uncertainty measurement protocols, will be made available for future research endeavors.&#10;&#10;\end{itemize}&#10;&#10;The Contributions to the Software Engineering Field. LLMs have revolutionized various aspects of software engineering~\cite{hou2023large, fan2023large}, including but not limited to automated code generation~\cite{fried2023incoder, roziere2023code, allal2023santacoder, nijkamp2022codegen}, software testing~\cite{gu2023llm, kang2023large, lemieux2023codamosa, liu2023fill}, debugging~\cite{tian2024debugbench}, program repair~\cite{first2023baldur, wei2023copiloting}, and document generation~~\cite{geng2024large}. While LLMs can serve as a critical core for many new-era AI-enabled intelligent systems in the software engineering domain, their black-box nature and inherent uncertainties pose challenges for them to be applied in the real world in a transparent, reliable, safe, and secure way. It is thus urgent to investigate and explore effective quality assurance methods. &#10;Measuring uncertainty and taking it as an indicator of AI models' reliability has been studied extensively in the SE community~\cite{zhang2020towards, OOD2, weiss2023uncertainty}. While promising, most of them focus on classification tasks with relatively simple neural architectures. On the contrary, we initialize a very early stage study on autoregressive, large-scale language models and perform various uncertainty measurements across a wide spectrum of tasks. &#10;&#10;We further provide more supplementary results and details as well as the source code to reproduce our study at our website: \href{&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Mathematics, Uncertainty Estimation, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2208.03274" label="2208.03274">
        <attvalues>
          <attvalue for="0" value="A Holistic Approach to Undesired Content Detection in the Real World" />
          <attvalue for="1" value="  We present a holistic approach to building a robust and useful natural&#10;language classification system for real-world content moderation. The success&#10;of such a system relies on a chain of carefully designed and executed steps,&#10;including the design of content taxonomies and labeling instructions, data&#10;quality control, an active learning pipeline to capture rare events, and a&#10;variety of methods to make the model robust and to avoid overfitting. Our&#10;moderation system is trained to detect a broad set of categories of undesired&#10;content, including sexual content, hateful content, violence, self-harm, and&#10;harassment. This approach generalizes to a wide range of different content&#10;taxonomies and can be used to create high-quality content classifiers that&#10;outperform off-the-shelf models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.19860" label="2305.19860">
        <attvalues>
          <attvalue for="0" value="A Survey on Large Language Models for Recommendation" />
          <attvalue for="1" value="  Large Language Models (LLMs) have emerged as powerful tools in the field of&#10;Natural Language Processing (NLP) and have recently gained significant&#10;attention in the domain of Recommendation Systems (RS). These models, trained&#10;on massive amounts of data using self-supervised learning, have demonstrated&#10;remarkable success in learning universal representations and have the potential&#10;to enhance various aspects of recommendation systems by some effective transfer&#10;techniques such as fine-tuning and prompt tuning, and so on. The crucial aspect&#10;of harnessing the power of language models in enhancing recommendation quality&#10;is the utilization of their high-quality representations of textual features&#10;and their extensive coverage of external knowledge to establish correlations&#10;between items and users. To provide a comprehensive understanding of the&#10;existing LLM-based recommendation systems, this survey presents a taxonomy that&#10;categorizes these models into two major paradigms, respectively Discriminative&#10;LLM for Recommendation (DLLM4Rec) and Generative LLM for Recommendation&#10;(GLLM4Rec), with the latter being systematically sorted out for the first time.&#10;Furthermore, we systematically review and analyze existing LLM-based&#10;recommendation systems within each paradigm, providing insights into their&#10;methodologies, techniques, and performance. Additionally, we identify key&#10;challenges and several valuable findings to provide researchers and&#10;practitioners with inspiration. We have also created a GitHub repository to&#10;index relevant papers on LLMs for recommendation,&#10;https://github.com/WLiK/LLM4Rec.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.02503" label="2102.02503">
        <attvalues>
          <attvalue for="0" value="Understanding the Capabilities, Limitations, and Societal Impact of&#10;  Large Language Models" />
          <attvalue for="1" value="  On October 14th, 2020, researchers from OpenAI, the Stanford Institute for&#10;Human-Centered Artificial Intelligence, and other universities convened to&#10;discuss open research questions surrounding GPT-3, the largest&#10;publicly-disclosed dense language model at the time. The meeting took place&#10;under Chatham House Rules. Discussants came from a variety of research&#10;backgrounds including computer science, linguistics, philosophy, political&#10;science, communications, cyber policy, and more. Broadly, the discussion&#10;centered around two main questions: 1) What are the technical capabilities and&#10;limitations of large language models? 2) What are the societal effects of&#10;widespread use of large language models? Here, we provide a detailed summary of&#10;the discussion organized by the two themes above.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.05783" label="2101.05783">
        <attvalues>
          <attvalue for="0" value="Persistent Anti-Muslim Bias in Large Language Models" />
          <attvalue for="1" value="  It has been observed that large-scale language models capture undesirable&#10;societal biases, e.g. relating to race and gender; yet religious bias has been&#10;relatively unexplored. We demonstrate that GPT-3, a state-of-the-art contextual&#10;language model, captures persistent Muslim-violence bias. We probe GPT-3 in&#10;various ways, including prompt completion, analogical reasoning, and story&#10;generation, to understand this anti-Muslim bias, demonstrating that it appears&#10;consistently and creatively in different uses of the model and that it is&#10;severe even compared to biases about other religious groups. For instance,&#10;&quot;Muslim&quot; is analogized to &quot;terrorist&quot; in 23% of test cases, while &quot;Jewish&quot; is&#10;mapped to &quot;money&quot; in 5% of test cases. We quantify the positive distraction&#10;needed to overcome this bias with adversarial text prompts, and find that use&#10;of the most positive 6 adjectives reduces violent completions for &quot;Muslims&quot;&#10;from 66% to 20%, but which is still higher than for other religious groups.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.07153" label="2305.07153">
        <attvalues>
          <attvalue for="0" value="Towards best practices in AGI safety and governance: A survey of expert&#10;  opinion" />
          <attvalue for="1" value="  A number of leading AI companies, including OpenAI, Google DeepMind, and&#10;Anthropic, have the stated goal of building artificial general intelligence&#10;(AGI) - AI systems that achieve or exceed human performance across a wide range&#10;of cognitive tasks. In pursuing this goal, they may develop and deploy AI&#10;systems that pose particularly significant risks. While they have already taken&#10;some measures to mitigate these risks, best practices have not yet emerged. To&#10;support the identification of best practices, we sent a survey to 92 leading&#10;experts from AGI labs, academia, and civil society and received 51 responses.&#10;Participants were asked how much they agreed with 50 statements about what AGI&#10;labs should do. Our main finding is that participants, on average, agreed with&#10;all of them. Many statements received extremely high levels of agreement. For&#10;example, 98% of respondents somewhat or strongly agreed that AGI labs should&#10;conduct pre-deployment risk assessments, dangerous capabilities evaluations,&#10;third-party model audits, safety restrictions on model usage, and red teaming.&#10;Ultimately, our list of statements may serve as a helpful foundation for&#10;efforts to develop best practices, standards, and regulations for AGI labs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.03718" label="2307.03718">
        <attvalues>
          <attvalue for="0" value="Frontier AI Regulation: Managing Emerging Risks to Public Safety" />
          <attvalue for="1" value="  Advanced AI models hold the promise of tremendous benefits for humanity, but&#10;society needs to proactively manage the accompanying risks. In this paper, we&#10;focus on what we term &quot;frontier AI&quot; models: highly capable foundation models&#10;that could possess dangerous capabilities sufficient to pose severe risks to&#10;public safety. Frontier AI models pose a distinct regulatory challenge:&#10;dangerous capabilities can arise unexpectedly; it is difficult to robustly&#10;prevent a deployed model from being misused; and, it is difficult to stop a&#10;model's capabilities from proliferating broadly. To address these challenges,&#10;at least three building blocks for the regulation of frontier models are&#10;needed: (1) standard-setting processes to identify appropriate requirements for&#10;frontier AI developers, (2) registration and reporting requirements to provide&#10;regulators with visibility into frontier AI development processes, and (3)&#10;mechanisms to ensure compliance with safety standards for the development and&#10;deployment of frontier AI models. Industry self-regulation is an important&#10;first step. However, wider societal discussions and government intervention&#10;will be needed to create standards and to ensure compliance with them. We&#10;consider several options to this end, including granting enforcement powers to&#10;supervisory authorities and licensure regimes for frontier AI models. Finally,&#10;we propose an initial set of safety standards. These include conducting&#10;pre-deployment risk assessments; external scrutiny of model behavior; using&#10;risk assessments to inform deployment decisions; and monitoring and responding&#10;to new information about model capabilities and uses post-deployment. We hope&#10;this discussion contributes to the broader conversation on how to balance&#10;public safety risks and innovation benefits from advances at the frontier of AI&#10;development.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.07586" label="2011.07586">
        <attvalues>
          <attvalue for="0" value="Uncertainty as a Form of Transparency: Measuring, Communicating, and&#10;  Using Uncertainty" />
          <attvalue for="1" value="  Algorithmic transparency entails exposing system properties to various&#10;stakeholders for purposes that include understanding, improving, and contesting&#10;predictions. Until now, most research into algorithmic transparency has&#10;predominantly focused on explainability. Explainability attempts to provide&#10;reasons for a machine learning model's behavior to stakeholders. However,&#10;understanding a model's specific behavior alone might not be enough for&#10;stakeholders to gauge whether the model is wrong or lacks sufficient knowledge&#10;to solve the task at hand. In this paper, we argue for considering a&#10;complementary form of transparency by estimating and communicating the&#10;uncertainty associated with model predictions. First, we discuss methods for&#10;assessing uncertainty. Then, we characterize how uncertainty can be used to&#10;mitigate model unfairness, augment decision-making, and build trustworthy&#10;systems. Finally, we outline methods for displaying uncertainty to stakeholders&#10;and recommend how to collect information required for incorporating uncertainty&#10;into existing ML pipelines. This work constitutes an interdisciplinary review&#10;drawn from literature spanning machine learning, visualization/HCI, design,&#10;decision-making, and fairness. We aim to encourage researchers and&#10;practitioners to measure, communicate, and use uncertainty as a form of&#10;transparency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.08896" label="2303.08896">
        <attvalues>
          <attvalue for="0" value="SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for&#10;  Generative Large Language Models" />
          <attvalue for="1" value="  Generative Large Language Models (LLMs) such as GPT-3 are capable of&#10;generating highly fluent responses to a wide variety of user prompts. However,&#10;LLMs are known to hallucinate facts and make non-factual statements which can&#10;undermine trust in their output. Existing fact-checking approaches either&#10;require access to the output probability distribution (which may not be&#10;available for systems such as ChatGPT) or external databases that are&#10;interfaced via separate, often complex, modules. In this work, we propose&#10;&quot;SelfCheckGPT&quot;, a simple sampling-based approach that can be used to fact-check&#10;the responses of black-box models in a zero-resource fashion, i.e. without an&#10;external database. SelfCheckGPT leverages the simple idea that if an LLM has&#10;knowledge of a given concept, sampled responses are likely to be similar and&#10;contain consistent facts. However, for hallucinated facts, stochastically&#10;sampled responses are likely to diverge and contradict one another. We&#10;investigate this approach by using GPT-3 to generate passages about individuals&#10;from the WikiBio dataset, and manually annotate the factuality of the generated&#10;passages. We demonstrate that SelfCheckGPT can: i) detect non-factual and&#10;factual sentences; and ii) rank passages in terms of factuality. We compare our&#10;approach to several baselines and show that our approach has considerably&#10;higher AUC-PR scores in sentence-level hallucination detection and higher&#10;correlation scores in passage-level factuality assessment compared to grey-box&#10;methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.10620" label="2308.10620">
        <attvalues>
          <attvalue for="0" value="Large Language Models for Software Engineering: A Systematic Literature&#10;  Review" />
          <attvalue for="1" value="  Large Language Models (LLMs) have significantly impacted numerous domains,&#10;including Software Engineering (SE). Many recent publications have explored&#10;LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding&#10;of the application, effects, and possible limitations of LLMs on SE is still in&#10;its early stages. To bridge this gap, we conducted a systematic literature&#10;review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can&#10;be exploited to optimize processes and outcomes. We select and analyze 395&#10;research papers from January 2017 to January 2024 to answer four key research&#10;questions (RQs). In RQ1, we categorize different LLMs that have been employed&#10;in SE tasks, characterizing their distinctive features and uses. In RQ2, we&#10;analyze the methods used in data collection, preprocessing, and application,&#10;highlighting the role of well-curated datasets for successful LLM for SE&#10;implementation. RQ3 investigates the strategies employed to optimize and&#10;evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE&#10;tasks where LLMs have shown success to date, illustrating their practical&#10;contributions to the field. From the answers to these RQs, we discuss the&#10;current state-of-the-art and trends, identifying gaps in existing research, and&#10;flagging promising areas for future study. Our artifacts are publicly available&#10;at https://github.com/xinyi-hou/LLM4SE_SLR.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.04621" label="2401.04621">
        <attvalues>
          <attvalue for="0" value="DebugBench: Evaluating Debugging Capability of Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated exceptional coding capability.&#10;However, as another critical component of programming proficiency, the&#10;debugging capability of LLMs remains relatively unexplored. Previous&#10;evaluations of LLMs' debugging ability are significantly limited by the risk of&#10;data leakage, the scale of the dataset, and the variety of tested bugs. To&#10;overcome these deficiencies, we introduce `DebugBench', an LLM debugging&#10;benchmark consisting of 4,253 instances. It covers four major bug categories&#10;and 18 minor types in C++, Java, and Python. To construct DebugBench, we&#10;collect code snippets from the LeetCode community, implant bugs into source&#10;data with GPT-4, and assure rigorous quality checks. We evaluate two commercial&#10;and four open-source models in a zero-shot scenario. We find that (1) while&#10;closed-source models exhibit inferior debugging performance compared to humans,&#10;open-source models relatively lower pass rate scores; (2) the complexity of&#10;debugging notably fluctuates depending on the bug category; (3) incorporating&#10;runtime feedback has a clear impact on debugging performance which is not&#10;always helpful. As an extension, we also compare LLM debugging and code&#10;generation, revealing a strong correlation between them for closed-source&#10;models. These findings will benefit the development of LLMs in debugging.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.04910" label="2303.04910">
        <attvalues>
          <attvalue for="0" value="Baldur: Whole-Proof Generation and Repair with Large Language Models" />
          <attvalue for="1" value="  Formally verifying software properties is a highly desirable but&#10;labor-intensive task. Recent work has developed methods to automate formal&#10;verification using proof assistants, such as Coq and Isabelle/HOL, e.g., by&#10;training a model to predict one proof step at a time, and using that model to&#10;search through the space of possible proofs. This paper introduces a new method&#10;to automate formal verification: We use large language models, trained on&#10;natural language text and code and fine-tuned on proofs, to generate whole&#10;proofs for theorems at once, rather than one step at a time. We combine this&#10;proof generation model with a fine-tuned repair model to repair generated&#10;proofs, further increasing proving power. As its main contributions, this paper&#10;demonstrates for the first time that: (1) Whole-proof generation using&#10;transformers is possible and is as effective as search-based techniques without&#10;requiring costly search. (2) Giving the learned model additional context, such&#10;as a prior failed proof attempt and the ensuing error message, results in proof&#10;repair and further improves automated proof generation. (3) We establish a new&#10;state of the art for fully automated proof synthesis. We reify our method in a&#10;prototype, Baldur, and evaluate it on a benchmark of 6,336 Isabelle/HOL&#10;theorems and their proofs. In addition to empirically showing the effectiveness&#10;of whole-proof generation, repair, and added context, we show that Baldur&#10;improves on the state-of-the-art tool, Thor, by automatically generating proofs&#10;for an additional 8.7% of the theorems. Together, Baldur and Thor can prove&#10;65.7% of the theorems fully automatically. This paper paves the way for new&#10;research into using large language models for automating formal verification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.11384" label="2304.11384">
        <attvalues>
          <attvalue for="0" value="Large Language Models are Few-Shot Summarizers: Multi-Intent Comment&#10;  Generation via In-Context Learning" />
          <attvalue for="1" value="  Code comment generation aims at generating natural language descriptions for&#10;a code snippet to facilitate developers' program comprehension activities.&#10;Despite being studied for a long time, a bottleneck for existing approaches is&#10;that given a code snippet, they can only generate one comment while developers&#10;usually need to know information from diverse perspectives such as what is the&#10;functionality of this code snippet and how to use it. To tackle this&#10;limitation, this study empirically investigates the feasibility of utilizing&#10;large language models (LLMs) to generate comments that can fulfill developers'&#10;diverse intents. Our intuition is based on the facts that (1) the code and its&#10;pairwise comment are used during the pre-training process of LLMs to build the&#10;semantic connection between the natural language and programming language, and&#10;(2) comments in the real-world projects, which are collected for the&#10;pre-training, usually contain different developers' intents. We thus postulate&#10;that the LLMs can already understand the code from different perspectives after&#10;the pre-training. Indeed, experiments on two large-scale datasets demonstrate&#10;the rationale of our insights: by adopting the in-context learning paradigm and&#10;giving adequate prompts to the LLM (e.g., providing it with ten or more&#10;examples), the LLM can significantly outperform a state-of-the-art supervised&#10;learning approach on generating comments with multiple intents. Results also&#10;show that customized strategies for constructing the prompts and&#10;post-processing strategies for reranking the results can both boost the LLM's&#10;performances, which shed light on future research directions for using LLMs to&#10;achieve comment generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05470" label="2310.05470">
        <attvalues>
          <attvalue for="0" value="Generative Judge for Evaluating Alignment" />
          <attvalue for="1" value="  The rapid development of Large Language Models (LLMs) has substantially&#10;expanded the range of tasks they can address. In the field of Natural Language&#10;Processing (NLP), researchers have shifted their focus from conventional NLP&#10;tasks (e.g., sequence tagging and parsing) towards tasks that revolve around&#10;aligning with human needs (e.g., brainstorming and email writing). This shift&#10;in task distribution imposes new requirements on evaluating these aligned&#10;models regarding generality (i.e., assessing performance across diverse&#10;scenarios), flexibility (i.e., examining under different protocols), and&#10;interpretability (i.e., scrutinizing models with explanations). In this paper,&#10;we propose a generative judge with 13B parameters, Auto-J, designed to address&#10;these challenges. Our model is trained on user queries and LLM-generated&#10;responses under massive real-world scenarios and accommodates diverse&#10;evaluation protocols (e.g., pairwise response comparison and single-response&#10;evaluation) with well-structured natural language critiques. To demonstrate the&#10;efficacy of our approach, we construct a new testbed covering 58 different&#10;scenarios. Experimentally, Auto-J outperforms a series of strong competitors,&#10;including both open-source and closed-source models, by a large margin. We also&#10;provide detailed analysis and case studies to further reveal the potential of&#10;our method and make a variety of resources public at&#10;https://github.com/GAIR-NLP/auto-j.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence, Model Evaluation Methods, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2302.04700" label="2302.04700">
        <attvalues>
          <attvalue for="0" value="Augmenting NLP data to counter Annotation Artifacts for NLI Tasks" />
          <attvalue for="1" value="  In this paper, we explore Annotation Artifacts - the phenomena wherein large&#10;pre-trained NLP models achieve high performance on benchmark datasets but do&#10;not actually &quot;solve&quot; the underlying task and instead rely on some dataset&#10;artifacts (same across train, validation, and test sets) to figure out the&#10;right answer. We explore this phenomenon on the well-known Natural Language&#10;Inference task by first using contrast and adversarial examples to understand&#10;limitations to the model's performance and show one of the biases arising from&#10;annotation artifacts (the way training data was constructed by the annotators).&#10;We then propose a data augmentation technique to fix this bias and measure its&#10;effectiveness.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Inference (NLI), also known as Recognizing Textual Entailment (RTE), is the task of determining the inference relation between two (short, ordered) texts: entailment, contradiction, or neutral. &#10;&#10;In this paper, we study how Annotation Artifacts impact performance on an ELECTRA Small model \cite{clark2020electra} trained on the Stanford Natural Language Inference Dataset (SNLI) \cite{DBLP:journals/corr/BowmanAPM15}. Next, we construct metrics to quantify the effects of this bias by identifying &quot;easy&quot; and &quot;hard&quot; examples on a subset of Validation data. Finally, we propose a solution using data augmentation during model training which counters this bias by forcing the model to learn &quot;features&quot; more relevant to the overall task. We compare the results of this &quot;modified&quot; model with the &quot;baseline&quot; model and show improvement on both easy and hard examples.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Data Augmentation Techniques, Linguistics, Annotation Artifacts, NLP Model Limitations, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2002.06012" label="2002.06012">
        <attvalues>
          <attvalue for="0" value="Dialogue history integration into end-to-end signal-to-concept spoken&#10;  language understanding systems" />
          <attvalue for="1" value="  This work investigates the embeddings for representing dialog history in&#10;spoken language understanding (SLU) systems. We focus on the scenario when the&#10;semantic information is extracted directly from the speech signal by means of a&#10;single end-to-end neural network model. We proposed to integrate dialogue&#10;history into an end-to-end signal-to-concept SLU system. The dialog history is&#10;represented in the form of dialog history embedding vectors (so-called&#10;h-vectors) and is provided as an additional information to end-to-end SLU&#10;models in order to improve the system performance. Three following types of&#10;h-vectors are proposed and experimentally evaluated in this paper: (1)&#10;supervised-all embeddings predicting bag-of-concepts expected in the answer of&#10;the user from the last dialog system response; (2) supervised-freq embeddings&#10;focusing on predicting only a selected set of semantic concept (corresponding&#10;to the most frequent errors in our experiments); and (3) unsupervised&#10;embeddings. Experiments on the MEDIA corpus for the semantic slot filling task&#10;demonstrate that the proposed h-vectors improve the model performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;The task of spoken language understanding (SLU) system is to detect fragments of semantic knowledge in speech data.&#10;Popular models are made of frames describing relations between entities and their properties~\cite{tur2011spoken,shen2019modeling,li2019incremental}.&#10;The SLU system instantiates a predefined set of frame structures called concepts that can&#10;be mentioned in a sentence or a dialogue turn. Concept mentions express dialogue acts~(DA), intents, domain knowledge, and frame properties often represented by slots, identified by entity names, and slot filler values identified by mention types.&#10;Concept mentions are difficult to characterize in terms of words or characters. They may be localized by head words or short word sequences called concept supports. &#10;For example, word spans can be hypothesized to be mentions of concepts, while entire sentence can be considered for hypothesizing dialogue acts. Unfortunately, mentions may be ambiguous because their word spans may express more semantic constituents, be incomplete or be affected by errors of an automatic speech recognition (ASR) system. &#10;These difficulties can be alleviated by considering certain head words, word spans, or a sentence as a seed for hypotheses generation and using additional context for providing predictions useful for constraining instantiation decision. &#10;An example of additional distant context used so far is a representation of dialogue history made of embeddings of sentences preceding the sentence or dialogue turn to be interpreted~\cite{chen2016end, goo2018slot, zhao2019hierarchical, sankar2019neural, goel2019hyst,history,henaff2016tracking,korpusik2019dialogue,lee2019sumbt}.&#10;&#10;A problem that has not yet thoroughly investigated is to select what to embed and how. Some popular corpora used so far (e.g. ATIS~\cite{dahl1994expanding}) do not have explicit sentence history. In this case, the only context to pay attention to is the sentence to be interpreted. If some history information is available, then distant contexts for DA and concepts may be different.&#10;Specific contexts for DA have been proposed in~\cite{liu2017using, ortega2019context}. For concepts, the selection of distant contexts may depend on the complexity of the application semantic domain. For example, the French MEDIA corpus~\cite{devillers2004french} has concepts of reference, relative time, locations, prices, logical conjunction and disjunction that are expressed by short semantically ambiguous words, which are often difficult to recognize, requiring knowledge of a semantic context called state of-the world to reduce the perplexity.&#10;Furthermore, the problem of deciding the type of embedding is also relevant as made evident in recent published papers~\cite{komninos2016dependency, &#10;lin2017structured, peters2018deep, &#10;yin2018dimensionality, zhang2018diffusion, yaghoobzadeh2019probing}.&#10;&#10;In this paper, we investigate the use of different types of dialog history representation, extracted with or without supervision, and their impact on the performance of an end-to-end signal-to-concept neural network.&#10;&#10;Noticeable approaches for reducing uncertainty in concept detection automatically extract relevant information from dialogue history~\cite{chen2016end,goo2018slot,goel2019hyst}. Considering the concern expressed in [7] and prior knowledge, we propose to focus on types of history contents starting by considering the previous system turn that contains semantically unambiguous information. In fact, the sequence of words in the system turn is generated by a semantic model whose goal is to reach a commit state for performing a transaction. Furthermore, using the train set, it is possible to compute prediction probabilities of user enunciated concepts, given the system enunciated concepts. The most likely predicted concepts can thus be used for reducing interpretation uncertainty in the following user turn.&#10;&#10;The rest of the paper is organized as follows.&#10;Section~\ref{sec:architecture} presents an architecture of an end-to-end signal-to-concept model and the proposed way of integration of dialog history ebmeddings (to which we refer as h-vectors) into this model.&#10;Section~\ref{sec:dialrep} introduces different ways to represent the dialog history.&#10;Sections~\ref{sec:exp} describes the experimental setup and results. Finally, the conclusions are given in Section~\ref{sec:concl}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Modeling, Computer Science, Dialogue History Embeddings, Linguistics, Spoken Language Understanding, Artificial Intelligence, Signal Processing" />
        </attvalues>
      </node>
      <node id="1906.07429" label="1906.07429">
        <attvalues>
          <attvalue for="0" value="Modeling Semantic Relationship in Multi-turn Conversations with&#10;  Hierarchical Latent Variables" />
          <attvalue for="1" value="  Multi-turn conversations consist of complex semantic structures, and it is&#10;still a challenge to generate coherent and diverse responses given previous&#10;utterances. It's practical that a conversation takes place under a background,&#10;meanwhile, the query and response are usually most related and they are&#10;consistent in topic but also different in content. However, little work focuses&#10;on such hierarchical relationship among utterances. To address this problem, we&#10;propose a Conversational Semantic Relationship RNN (CSRR) model to construct&#10;the dependency explicitly. The model contains latent variables in three&#10;hierarchies. The discourse-level one captures the global background, the&#10;pair-level one stands for the common topic information between query and&#10;response, and the utterance-level ones try to represent differences in content.&#10;Experimental results show that our model significantly improves the quality of&#10;responses in terms of fluency, coherence and diversity compared to baseline&#10;methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.08854" label="1907.08854">
        <attvalues>
          <attvalue for="0" value="Incremental Transformer with Deliberation Decoder for Document Grounded&#10;  Conversations" />
          <attvalue for="1" value="  Document Grounded Conversations is a task to generate dialogue responses when&#10;chatting about the content of a given document. Obviously, document knowledge&#10;plays a critical role in Document Grounded Conversations, while existing&#10;dialogue models do not exploit this kind of knowledge effectively enough. In&#10;this paper, we propose a novel Transformer-based architecture for multi-turn&#10;document grounded conversations. In particular, we devise an Incremental&#10;Transformer to encode multi-turn utterances along with knowledge in related&#10;documents. Motivated by the human cognitive process, we design a two-pass&#10;decoder (Deliberation Decoder) to improve context coherence and knowledge&#10;correctness. Our empirical study on a real-world Document Grounded Dataset&#10;proves that responses generated by our model significantly outperform&#10;competitive baselines on both context coherence and knowledge relevance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.07421" label="1907.07421">
        <attvalues>
          <attvalue for="0" value="SUMBT: Slot-Utterance Matching for Universal and Scalable Belief&#10;  Tracking" />
          <attvalue for="1" value="  In goal-oriented dialog systems, belief trackers estimate the probability&#10;distribution of slot-values at every dialog turn. Previous neural approaches&#10;have modeled domain- and slot-dependent belief trackers, and have difficulty in&#10;adding new slot-values, resulting in lack of flexibility of domain ontology&#10;configurations. In this paper, we propose a new approach to universal and&#10;scalable belief tracker, called slot-utterance matching belief tracker (SUMBT).&#10;The model learns the relations between domain-slot-types and slot-values&#10;appearing in utterances through attention mechanisms based on contextual&#10;semantic vectors. Furthermore, the model predicts slot-value labels in a&#10;non-parametric way. From our experiments on two dialog corpora, WOZ 2.0 and&#10;MultiWOZ, the proposed model showed performance improvement in comparison with&#10;slot-dependent methods and achieved the state-of-the-art joint accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.00883" label="1907.00883">
        <attvalues>
          <attvalue for="0" value="HyST: A Hybrid Approach for Flexible and Accurate Dialogue State&#10;  Tracking" />
          <attvalue for="1" value="  Recent works on end-to-end trainable neural network based approaches have&#10;demonstrated state-of-the-art results on dialogue state tracking. The best&#10;performing approaches estimate a probability distribution over all possible&#10;slot values. However, these approaches do not scale for large value sets&#10;commonly present in real-life applications and are not ideal for tracking slot&#10;values that were not observed in the training set. To tackle these issues,&#10;candidate-generation-based approaches have been proposed. These approaches&#10;estimate a set of values that are possible at each turn based on the&#10;conversation history and/or language understanding outputs, and hence enable&#10;state tracking over unseen values and large value sets however, they fall short&#10;in terms of performance in comparison to the first group. In this work, we&#10;analyze the performance of these two alternative dialogue state tracking&#10;methods, and present a hybrid approach (HyST) which learns the appropriate&#10;method for each slot type. To demonstrate the effectiveness of HyST on a&#10;rich-set of slot types, we experiment with the recently released MultiWOZ-2.0&#10;multi-domain, task-oriented dialogue-dataset. Our experiments show that HyST&#10;scales to multi-domain applications. Our best performing model results in a&#10;relative improvement of 24% and 10% over the previous SOTA and our best&#10;baseline respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.00550" label="2306.00550">
        <attvalues>
          <attvalue for="0" value="Chain-Of-Thought Prompting Under Streaming Batch: A Case Study" />
          <attvalue for="1" value="  Recently, Large Language Models (LLMs) have demonstrated remarkable&#10;capabilities. Chain-of-Thought (CoT) has been proposed as a way of assisting&#10;LLMs in performing complex reasoning. However, developing effective prompts can&#10;be a challenging and labor-intensive task. Many studies come out of some way to&#10;automatically construct CoT from test data. Most of them assume that all test&#10;data is visible before testing and only select a small subset to generate&#10;rationales, which is an unrealistic assumption. In this paper, we present a&#10;case study on how to construct and optimize chain-of-thought prompting using&#10;batch data in streaming settings.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models \cite{tang2023does} have shown emergent abilities. \cite{wei2022emergent} \cite{cot} discover Chain-of-Thought (CoT) prompting as a simple and broadly applicable method for enhancing reasoning in language models. Many work \cite{zhou2022least,wang2022self,shi2022language,autocot,zhang2023multimodal,wang2022towards,zhou2022large,fei2023reasoning,yang2023mm,shi2023large,diao2023active} have tried to make further improvements based on CoT.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing, Prompt Optimization, Chain-of-Thought Reasoning" />
        </attvalues>
      </node>
      <node id="2203.11171" label="2203.11171">
        <attvalues>
          <attvalue for="0" value="Self-Consistency Improves Chain of Thought Reasoning in Language Models" />
          <attvalue for="1" value="  Chain-of-thought prompting combined with pre-trained large language models&#10;has achieved encouraging results on complex reasoning tasks. In this paper, we&#10;propose a new decoding strategy, self-consistency, to replace the naive greedy&#10;decoding used in chain-of-thought prompting. It first samples a diverse set of&#10;reasoning paths instead of only taking the greedy one, and then selects the&#10;most consistent answer by marginalizing out the sampled reasoning paths.&#10;Self-consistency leverages the intuition that a complex reasoning problem&#10;typically admits multiple different ways of thinking leading to its unique&#10;correct answer. Our extensive empirical evaluation shows that self-consistency&#10;boosts the performance of chain-of-thought prompting with a striking margin on&#10;a range of popular arithmetic and commonsense reasoning benchmarks, including&#10;GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and&#10;ARC-challenge (+3.9%).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.03057" label="2210.03057">
        <attvalues>
          <attvalue for="0" value="Language Models are Multilingual Chain-of-Thought Reasoners" />
          <attvalue for="1" value="  We evaluate the reasoning abilities of large language models in multilingual&#10;settings. We introduce the Multilingual Grade School Math (MGSM) benchmark, by&#10;manually translating 250 grade-school math problems from the GSM8K dataset&#10;(Cobbe et al., 2021) into ten typologically diverse languages. We find that the&#10;ability to solve MGSM problems via chain-of-thought prompting emerges with&#10;increasing model scale, and that models have strikingly strong multilingual&#10;reasoning abilities, even in underrepresented languages such as Bengali and&#10;Swahili. Finally, we show that the multilingual reasoning abilities of language&#10;models extend to other tasks such as commonsense reasoning and word-in-context&#10;semantic judgment. The MGSM benchmark is publicly available at&#10;https://github.com/google-research/url-nlp.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.00923" label="2302.00923">
        <attvalues>
          <attvalue for="0" value="Multimodal Chain-of-Thought Reasoning in Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have shown impressive performance on complex&#10;reasoning by leveraging chain-of-thought (CoT) prompting to generate&#10;intermediate reasoning chains as the rationale to infer the answer. However,&#10;existing CoT studies have primarily focused on the language modality. We&#10;propose Multimodal-CoT that incorporates language (text) and vision (images)&#10;modalities into a two-stage framework that separates rationale generation and&#10;answer inference. In this way, answer inference can leverage better generated&#10;rationales that are based on multimodal information. Experimental results on&#10;ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed&#10;approach. With Multimodal-CoT, our model under 1 billion parameters achieves&#10;state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates&#10;that Multimodal-CoT offers the advantages of mitigating hallucination and&#10;enhancing convergence speed. Code is publicly available at&#10;https://github.com/amazon-science/mm-cot.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10001" label="2212.10001">
        <attvalues>
          <attvalue for="0" value="Towards Understanding Chain-of-Thought Prompting: An Empirical Study of&#10;  What Matters" />
          <attvalue for="1" value="  Chain-of-Thought (CoT) prompting can dramatically improve the multi-step&#10;reasoning abilities of large language models (LLMs). CoT explicitly encourages&#10;the LLM to generate intermediate rationales for solving a problem, by providing&#10;a series of reasoning steps in the demonstrations. Despite its success, there&#10;is still little understanding of what makes CoT prompting effective and which&#10;aspects of the demonstrated reasoning steps contribute to its performance. In&#10;this paper, we show that CoT reasoning is possible even with invalid&#10;demonstrations - prompting with invalid reasoning steps can achieve over 80-90%&#10;of the performance obtained using CoT under various metrics, while still&#10;generating coherent lines of reasoning during inference. Further experiments&#10;show that other aspects of the rationales, such as being relevant to the query&#10;and correctly ordering the reasoning steps, are much more important for&#10;effective CoT reasoning. Overall, these findings both deepen our understanding&#10;of CoT prompting, and open up new questions regarding LLMs' capability to learn&#10;to reason in context.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11255" label="2305.11255">
        <attvalues>
          <attvalue for="0" value="Reasoning Implicit Sentiment with Chain-of-Thought Prompting" />
          <attvalue for="1" value="  While sentiment analysis systems try to determine the sentiment polarities of&#10;given targets based on the key opinion expressions in input texts, in implicit&#10;sentiment analysis (ISA) the opinion cues come in an implicit and obscure&#10;manner. Thus detecting implicit sentiment requires the common-sense and&#10;multi-hop reasoning ability to infer the latent intent of opinion. Inspired by&#10;the recent chain-of-thought (CoT) idea, in this work we introduce a Three-hop&#10;Reasoning (THOR) CoT framework to mimic the human-like reasoning process for&#10;ISA. We design a three-step prompting principle for THOR to step-by-step induce&#10;the implicit aspect, opinion, and finally the sentiment polarity. Our&#10;THOR+Flan-T5 (11B) pushes the state-of-the-art (SoTA) by over 6% F1 on&#10;supervised setup. More strikingly, THOR+GPT3 (175B) boosts the SoTA by over 50%&#10;F1 on zero-shot setting. Our code is open at&#10;https://github.com/scofield7419/THOR-ISA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.00093" label="2302.00093">
        <attvalues>
          <attvalue for="0" value="Large Language Models Can Be Easily Distracted by Irrelevant Context" />
          <attvalue for="1" value="  Large language models have achieved impressive performance on various natural&#10;language processing tasks. However, so far they have been evaluated primarily&#10;on benchmarks where all information in the input context is relevant for&#10;solving the task. In this work, we investigate the distractibility of large&#10;language models, i.e., how the model problem-solving accuracy can be influenced&#10;by irrelevant context. In particular, we introduce Grade-School Math with&#10;Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant&#10;information in the problem description. We use this benchmark to measure the&#10;distractibility of cutting-edge prompting techniques for large language models,&#10;and find that the model performance is dramatically decreased when irrelevant&#10;information is included. We also identify several approaches for mitigating&#10;this deficiency, such as decoding with self-consistency and adding to the&#10;prompt an instruction that tells the language model to ignore the irrelevant&#10;information.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.10475" label="2205.10475">
        <attvalues>
          <attvalue for="0" value="DeepStruct: Pretraining of Language Models for Structure Prediction" />
          <attvalue for="1" value="  We introduce a method for improving the structural understanding abilities of&#10;language models. Unlike previous approaches that finetune the models with&#10;task-specific augmentation, we pretrain language models on a collection of&#10;task-agnostic corpora to generate structures from text. Our structure&#10;pretraining enables zero-shot transfer of the learned knowledge that models&#10;have about the structure tasks. We study the performance of this approach on 28&#10;datasets, spanning 10 structure prediction tasks including open information&#10;extraction, joint entity and relation extraction, named entity recognition,&#10;relation classification, semantic role labeling, event extraction, coreference&#10;resolution, factual probe, intent detection, and dialogue state tracking. We&#10;further enhance the pretraining with the task-specific training sets. We show&#10;that a 10B parameter language model transfers non-trivially to most tasks and&#10;obtains state-of-the-art performance on 21 of 28 datasets that we evaluate.&#10;" />
          <attvalue for="2" value="&#10;Pretrained language models (LMs) have revolutionized NLP over the last few years~\cite{peters2018deep,devlin2019bert,radford2019language}, increasingly adept in performing the flexible and task-agnostic downstream transfer. Their transfer performance is less studied in structure prediction tasks, however. Well-studied tasks mainly focus on understanding one particular aspect of the text, such as predicting the next word that comes after as in language modeling. Unlike those downstream tasks, structure prediction requires the structural understanding of the text for further integrating multiple relevant aspects into a structure. For instance, a typical structure prediction task, called open information extraction, seeks the entire structural information in a sentence (Figure~\ref{fig:problemcomp}). Different from traditional NLP tasks, structure prediction takes one step further and serves as a natural testbed for the structural understanding competence of LMs.&#10;&#10;It is non-trivial to transfer LMs to downstream structure prediction tasks. While the structure prediction requires structural understanding, the LMs are pretrained to understand an independent aspect. For example, GPT-3~\cite{brown2020language} is trained to predict the next word, and BERT~\cite{devlin2019bert} is trained to recover the masked tokens. Recent work has made efforts in bridging the gap in transferring pretrained models to structure prediction tasks with a focus on two directions. As shown in Figure~\ref{fig:approach}, first, task-specific architectures are proposed to model the structures for different structure prediction tasks~\cite{stanovsky2018supervised,soares2019matching}. Second, task-specific data augmentation~\cite{paolini2021structured,wang2021zero,wei2021finetuned} is introduced, aiming to enrich text format with structure information. These approaches involve custom-designed task augmentations, impeding their usability in general structure prediction tasks.&#10;&#10;In this paper, we improve the structural understanding capabilities of LMs. In contrast to previous approaches relying on task augmentations, we introduce structure pretraining, which systematically teaches LMs to better understand structures of text beyond independent aspects in a pretraining phase (Figure~\ref{fig:overview}). This enables the zero-shot transfer of knowledge that LMs learned about structures during our pretraining to downstream structure prediction tasks. For example, our zero-shot 10B parameter LM significantly outperforms the zero-shot GPT-3 (175B) on a structure prediction benchmark dataset (Figure~\ref{fig:overview}). We accomplish this by reformulating structure prediction as a series of unit tasks--triple prediction tasks. We then train LMs on a collection of task-agnostic structural corpora to generate triples from text. The design of triple representation is important: it unifies a wide set of standard structure prediction tasks into the same task format. We apply our pretrained model DeepStruct\ to 28 datasets spanning 10 structure prediction tasks, including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretraining with multiple downstream structure prediction training sets and obtain state-of-the-art performance on 21 of 28 datasets. Our contributions are as follows:&#10;&#10;\begin{itemize}[leftmargin=*]&#10; \item We improve structural understanding abilities of pretrained LMs. Compared to traditional NLP tasks that only consider the understanding of an independent aspect of the text, structural understanding takes a step further that requires the ability to integrate multiple relevant aspects into a structure. We argue that it is important for LMs to go beyond traditional understanding toward structural understanding, as it requires a higher level of intelligent competence and is more challenging. It can also benefit a wide spectrum of NLP tasks that require structure-level understanding capability.&#10; \item We propose structure pretraining, which pretrains the LMs to understand structures in the text. The basic intuition is that the standard pretraining helps LMs to understand individual aspects of the information in the text, our method learns to integrate those individual aspects into structures. Compared to existing approaches, this method enables the zero-shot transfer of LMs to structure prediction tasks. For instance, our 10B LM produces superior zero-shot performance compared to 175B GPT-3 on a representative structure prediction task. &#10; \item We further equip our pretraining with multi-task learning and apply our method to 28 structure prediction datasets across 10 tasks. We achieve state-of-the-art performance on 21 of 28 datasets that we evaluate. We hope this can help facilitate the structural understanding research in the NLP community.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Transfer Learning Performance, Linguistics, Artificial Intelligence, Language Model Pretraining, Natural Language Processing, Structure Prediction Tasks" />
        </attvalues>
      </node>
      <node id="1906.03158" label="1906.03158">
        <attvalues>
          <attvalue for="0" value="Matching the Blanks: Distributional Similarity for Relation Learning" />
          <attvalue for="1" value="  General purpose relation extractors, which can model arbitrary relations, are&#10;a core aspiration in information extraction. Efforts have been made to build&#10;general purpose extractors that represent relations with their surface forms,&#10;or which jointly embed surface forms with relations from an existing knowledge&#10;graph. However, both of these approaches are limited in their ability to&#10;generalize. In this paper, we build on extensions of Harris' distributional&#10;hypothesis to relations, as well as recent advances in learning text&#10;representations (specifically, BERT), to build task agnostic relation&#10;representations solely from entity-linked text. We show that these&#10;representations significantly outperform previous work on exemplar based&#10;relation extraction (FewRel) even without using any of that task's training&#10;data. We also show that models initialized with our task agnostic&#10;representations, and then tuned on supervised relation extraction datasets,&#10;significantly outperform the previous methods on SemEval 2010 Task 8, KBP37,&#10;and TACRED.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.05779" label="2101.05779">
        <attvalues>
          <attvalue for="0" value="Structured Prediction as Translation between Augmented Natural Languages" />
          <attvalue for="1" value="  We propose a new framework, Translation between Augmented Natural Languages&#10;(TANL), to solve many structured prediction language tasks including joint&#10;entity and relation extraction, nested named entity recognition, relation&#10;classification, semantic role labeling, event extraction, coreference&#10;resolution, and dialogue state tracking. Instead of tackling the problem by&#10;training task-specific discriminative classifiers, we frame it as a translation&#10;task between augmented natural languages, from which the task-relevant&#10;information can be easily extracted. Our approach can match or outperform&#10;task-specific models on all tasks, and in particular, achieves new&#10;state-of-the-art results on joint entity and relation extraction (CoNLL04, ADE,&#10;NYT, and ACE2005 datasets), relation classification (FewRel and TACRED), and&#10;semantic role labeling (CoNLL-2005 and CoNLL-2012). We accomplish this while&#10;using the same architecture and hyperparameters for all tasks and even when&#10;training a single model to solve all tasks at the same time (multi-task&#10;learning). Finally, we show that our framework can also significantly improve&#10;the performance in a low-resource regime, thanks to better use of label&#10;semantics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.11171" label="2109.11171">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Information Extraction as a Unified Text-to-Triple Translation" />
          <attvalue for="1" value="  We cast a suite of information extraction tasks into a text-to-triple&#10;translation framework. Instead of solving each task relying on task-specific&#10;datasets and models, we formalize the task as a translation between&#10;task-specific input text and output triples. By taking the task-specific input,&#10;we enable a task-agnostic translation by leveraging the latent knowledge that a&#10;pre-trained language model has about the task. We further demonstrate that a&#10;simple pre-training task of predicting which relational information corresponds&#10;to which input text is an effective way to produce task-specific outputs. This&#10;enables the zero-shot transfer of our framework to downstream tasks. We study&#10;the zero-shot performance of this framework on open information extraction&#10;(OIE2016, NYT, WEB, PENN), relation classification (FewRel and TACRED), and&#10;factual probe (Google-RE and T-REx). The model transfers non-trivially to most&#10;tasks and is often competitive with a fully supervised method without the need&#10;for any task-specific training. For instance, we significantly outperform the&#10;F1 score of the supervised open information extraction without needing to use&#10;its training set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Relation classification aims to identify the correct relation type from a pre-defined set of relations between two given entities. Language models (LM)~\cite{xu2021pre} pre-trained with self-supervised~\cite{liu2021self} objectives, e.g., BERT~\cite{devlin2018bert}, GPT~\cite{radford2018improving,radford2019language,brown2020language}, RoBERTa~\cite{abs-1907-11692}, transfer well to relation classification datasets in fine-tuning~\cite{joshi2020spanbert,gao2019fewrel} or few-shot regime~\cite{soares2019matching} with architecture modifications. Sequence-to-sequence models, such as T5~\cite{raffel2019exploring}, BART~\cite{lewis2019bart} and GLM~\cite{du2021all}, are adapted to the task based on data augmentation and fine-tuning~\cite{paolini2021structured}. Besides relation classification, \cite{paolini2021structured} generalize T5 to some more structured prediction tasks as well, e.g., semantic role labeling and event extraction. However, \method\ enables zero-shot relation classification that does not require any task-specific training.&#10;&#10;Many open information extraction (OIE) systems, e.g., Stanford OpenIE~\cite{angeli2015leveraging}, OLLIE~\cite{schmitz2012open}, Reverb~\cite{fader2011identifying}, and their descendant Open IE4 leverage carefully-designed linguistic patterns (e.g., based on dependencies and POS tags) to extract triples from textual corpora without using additional training sets. Recently, supervised OIE systems~\cite{stanovsky2018supervised,ro2020multi,kolluru2020oie6} formulate the OIE as a sequence generation problem using neural networks trained on additional training sets. Similar to our work, \cite{wang2020language} use the parameters of LMs to extract triples, with the main difference that \method\ not only improves the recall of the beam search, but also uses a pre-trained ranking model to enhance the zero-shot capability.&#10;&#10;LMs are used in factual probing tasks, by using the outputs alone~\cite{petroni2019language} to answer the relation-specific queries in cloze statements. \cite{petroni2020context} additionally feed sentences expressing the facts to the LMs and shows improved results. Other than template-based queries, learning trigger-based~\cite{shin2020autoprompt} and continuous prompts~\cite{liu2021gpt,li2021prefix} are helpful in recalling the facts. The main difference is that \method\ explores the internal parameters of the LMs rather than the outputs, and the results are more interpretable.&#10;&#10;Overall, in contrast to the existing approaches, \method\ unifies the open information extraction, relation classification, and factual probe under the same framework in zero-shot settings.&#10; " />
        </attvalues>
      </node>
      <node id="2108.05575" label="2108.05575">
        <attvalues>
          <attvalue for="0" value="Kicktionary-LOME: A Domain-Specific Multilingual Frame Semantic Parsing&#10;  Model for Football Language" />
          <attvalue for="1" value="  This technical report introduces an adapted version of the LOME frame&#10;semantic parsing model (Xia et al., EACL 2021) which is capable of&#10;automatically annotating texts according to the &quot;Kicktionary&quot; domain-specific&#10;framenet resource. Several methods for training a model even with limited&#10;available training data are proposed. While there are some challenges for&#10;evaluation related to the nature of the available annotations, preliminary&#10;results are very promising, with the best model reaching F1-scores of 0.83&#10;(frame prediction) and 0.81 (semantic role prediction).&#10;" />
          <attvalue for="2" value="&#10;&#10;Frame semantic parsing \cite{gildea-jurafsky, baker-etal-2007-semeval} is the task of automatically assigning frame semantic structures \cite{fillmore2006}, consisting of semantic frames and their associated semantic roles, to a text. Frame semantic parsers depend on the availability of language resources, called framenets, for providing the set of semantic frames and roles to be annotated, as well as a corpus of annotated examples to learn from. The best-known framenet is Berkeley FrameNet \cite{baker2003}, a domain-general resource for English, but other framenets exist that are multilingual \cite{gilardi2018learning} or for specific other languages (e.g., FrameNet Brasil [\cite{torrent2018towards}], ASFALDA French FrameNet [\cite{candito-etal-2014-developing}], and many others) and for specific domains (e.g. \cite{Venturi2009TowardsAF}, \cite{datta-etal-2020-rad}).&#10;&#10;This technical report will introduce a frame semantic parser for Kicktionary\ \cite{schmidt2009kicktionary}, a domain-specific FrameNet covering the semantic domain of football (soccer), which provides semantic frame structures and lexical units (i.e., predicate words) with annotated examples in French, German, and English. As a basis for our system, we will use LOME \cite{xia-etal-2021-lome}, a recent multilingual frame semantic parsing model, as a basis, and experiment with several approaches for extending it to Kicktionary. &#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Domain-Specific Framenets, Computer Science, Semantic Parsing Models, Linguistics, Computational Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing, Limited Data Training" />
        </attvalues>
      </node>
      <node id="2101.12175" label="2101.12175">
        <attvalues>
          <attvalue for="0" value="LOME: Large Ontology Multilingual Extraction" />
          <attvalue for="1" value="  We present LOME, a system for performing multilingual information extraction.&#10;Given a text document as input, our core system identifies spans of textual&#10;entity and event mentions with a FrameNet (Baker et al., 1998) parser. It&#10;subsequently performs coreference resolution, fine-grained entity typing, and&#10;temporal relation prediction between events. By doing so, the system constructs&#10;an event and entity focused knowledge graph. We can further apply third-party&#10;modules for other types of annotation, like relation extraction. Our&#10;(multilingual) first-party modules either outperform or are competitive with&#10;the (monolingual) state-of-the-art. We achieve this through the use of&#10;multilingual encoders like XLM-R (Conneau et al., 2020) and leveraging&#10;multilingual training data. LOME is available as a Docker container on Docker&#10;Hub. In addition, a lightweight version of the system is accessible as a web&#10;demo.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.07937" label="2204.07937">
        <attvalues>
          <attvalue for="0" value="Unsupervised Cross-Task Generalization via Retrieval Augmentation" />
          <attvalue for="1" value="  Humans can perform unseen tasks by recalling relevant skills acquired&#10;previously and then generalizing them to the target tasks, even if there is no&#10;supervision at all. In this paper, we aim to improve this kind of cross-task&#10;generalization ability of massive multi-task language models, such as T0 and&#10;FLAN, in an unsupervised setting. We propose a retrieval-augmentation method&#10;named ReCross that takes a few unlabelled examples as queries to retrieve a&#10;small subset of upstream data and uses them to update the multi-task model for&#10;better generalization. ReCross is a straightforward yet effective retrieval&#10;method that combines both efficient dense retrieval and effective pair-wise&#10;reranking. Our results and analysis show that it significantly outperforms both&#10;non-retrieval methods and other baseline methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Multi-task training for task generalization.&#10;Text-to-text Transformer language models such as T5 enable us to train a multi-task NLP model with a more straightforward recipe: mixing the data of multiple tasks into a unified seq2seq format, and then fine-tuning text-to-text LMs for implicit multi-task learning. &#10;UnifiedQA~\cite{khashabi2020unifiedqa} is among the first works in this direction.&#10;Although it shows great generalization performance within QA tasks, it can hardly generalize to other NLP tasks. &#10;Recent works, such as CrossFit~\cite{ye-etal-2021-crossfit}, ExT5~\cite{aribandi2022ext}, FLAN~\cite{Wei2021FinetunedLM}, T0~\cite{sanh2021t0}, and InstructGPT~\cite{InstructGPT} focus on how to generalize a massively multi-task model across task boundaries in a much broader context.&#10;&#10;Particularly, in the CrossFit framework~\cite{ye-etal-2021-crossfit},&#10;cross-task generalization requires a small number of labeled instances of the target task for fine-tuning.&#10;It is because the templates of CrossFit use the task names as the hard prefixes. &#10;Therefore, it is necessary to fine-tune the upstream model with a few examples that have the target task names as prefixes (i.e., few-shot learning), but this largely limits the application scenarios of these multi-task NLP models in practice. &#10;We instead focus on unsupervised cross-task generalization, where there is no labeled data of an unseen task (i.e., zero-shot learning). &#10;Using natural-language instructions as prompts, both FLAN and T0 show that it is promising to perform zero-shot cross-task generalization.&#10;&#10;In this work, we also focus on such an unsupervised setting for cross-task generalization, while our problem setup is a bit different from the ones used in T0 and FLAN.&#10;As for the assumption about the unlabeled data, their setups can be seen as a special case of ours when $|Q|=1$ for all unseen tasks.&#10;The evaluation protocols of T0 and FLAN assess the generalization performance of the upstream model as it is, and thus their evaluation is more about the quality of templates and the upstream training tricks.&#10;In contrast, our evaluation protocol can also study how to efficiently adjust the upstream model such that the updated models can generalize to new tasks without labeled data.&#10;Thus, we believe ours is a more general setup for studying unsupervised cross-task generalization. &#10;&#10;Retrieval augmentation in NLP.&#10;We aim to retrieve useful examples from the upstream data and re-learning them for cross-task generalization.&#10;The proposed ReCross pipeline is inspired by open-ended QA methods such as DPR~\cite{dpr}, DrFact~\cite{Lin2021DifferentiableOC}, and RAG~\cite{Lewis2020RetrievalAugmentedGF}.&#10;Retrieval augmentation also shows great performance in pre-training LMs~\cite{Guu2020REALMRL}.&#10;Besides, \cite{Wang2022TrainingDI} shows that learning with similar data via retrieval augmentation can improve the performance of a task-specific model.&#10;\cite{Rubin2021LearningTR} show that retrieving better demonstration examples is also helpful for in-context few-shot learning of GPT-3 style language models~\cite{brown2020language}.&#10;The key challenge in the problem setup of this work is to predict the utility of the examples for unseen tasks with the consideration of efficiency and scalability.&#10;We have discussed more details about this challenge and related works in Sec.~\ref{sec:problem}.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Cognitive Science, Language Model Generalization, Cross-Task Retrieval Techniques, Artificial Intelligence, Natural Language Processing, Unsupervised Learning Methods" />
        </attvalues>
      </node>
      <node id="2405.09857" label="2405.09857">
        <attvalues>
          <attvalue for="0" value="IGOT: Information Gain Optimized Tokenizer on Domain Adaptive&#10;  Pretraining" />
          <attvalue for="1" value="  Pretrained Large Language Models (LLM) such as ChatGPT, Claude, etc. have&#10;demonstrated strong capabilities in various fields of natural language&#10;generation. However, there are still many problems when using LLM in&#10;specialized domain-specific fields. When using generative AI to process&#10;downstream tasks, a common approach is to add new knowledge (e.g., private&#10;domain knowledge, cutting-edge information) to a pretrained model through&#10;continued training or fine-tuning. However, whether there is a universal&#10;paradigm for domain adaptation training is still an open question. In this&#10;article, we proposed Information Gain Optimized Tokenizer (IGOT), which&#10;analyzes the special token set of downstream tasks, constructs a new subset&#10;using heuristic function $\phi$ with the special token and its information&#10;gain, to build new domain-specific tokenizer, and continues pretraining on the&#10;downstream task data. We explored the many positive effects of this method's&#10;customized tokenizer on domain-adaptive pretraining and verified this method&#10;can perform better than the ordinary method of just collecting data and&#10;fine-tuning. Based on our experiment, the continued pretraining process of IGOT&#10;with LLaMA-7B achieved 11.9\% token saving, 12.2\% training time saving, and&#10;5.8\% maximum GPU VRAM usage saving, combined with the T5 model, we can even&#10;reach a 31.5\% of training time saving, making porting general generative AI to&#10;specific domains more effective than before. In domain-specific tasks,&#10;supervised $IGOT_\tau$ shows great performance on reducing both the convergence&#10;radius and convergence point during keep pretraining.&#10;" />
          <attvalue for="2" value="&#10;&#10;After extensive training on publicly available web data, large language models have exhibited remarkable capabilities in a wide range of common sense reasoning tasks\cite{brown2020language}. Some studies have indicated that these models, during pretraining, not only learn contextual text vector representations and probability distributions but are also likely to learn grammar\cite{vig2019multiscale}, syntax\cite{hewitt2019structural}, factual knowledge\cite{wang2020language}, even behavior patterns strikingly similar to human common sense\cite{davison2019commonsense}.&#10;&#10;While general-purpose LLMs trained on massive internet datasets demonstrate impressive capabilities across diverse domains\cite{bubeck2023sparks}, recent work with domain-specific models like BloombergGPT \cite{wu2023bloomberggpt} and BioMedLM\cite{bolton2024biomedlm} highlights their superior performance within specific fields.&#10;&#10;Early efforts to build large-scale language models utilized n-grams and simple smoothing techniques\cite{heafield2013scalable}\cite{buck2014n}. Various neural network architectures were later applied to the language modeling task, including feedforward networks\cite{bengio2000neural} and recurrent networks\cite{mikolov2010recurrent}. The introduction of the Transformer architecture\cite{vaswani2017attention} led to the development of highly scalable language models\cite{radford2019language}, demonstrating a predictable relationship between language modeling loss and scaling factors such as model size, number of training tokens, and compute budget.&#10;&#10;The AI community has seen the release of numerous open-access LLMs, though they generally do not perform as well as their closed-access counterparts. ``Open-access LLM'' refers to models with publicly available weights, but transparency varies regarding training data and filtering techniques. EleutherAI released GPT-NeoX-20B\cite{black2022gpt} and GPT-J-6B \cite{wang2021codet5}, as well as the dataset these models were trained on Pile\cite{gao2020pile}. Google released UL2-20B\cite{tay2022ul2}, an encoder-decoder model trained on the publicly available C4\cite{raffel2020exploring}. Tsinghua University released the weights of GLM-130B\cite{zeng2022glm}, a Chinese-English LLM, and CodeGeeX-13B\cite{zheng2023codegeex}, a LLM for coding applications, without releasing the training sets. Salesforce released CodeGen-Mono-16B\cite{nijkamp2022codegen} without disclosing a proprietary Python dataset. Meta released the OPT\cite{zhang2022opt}, LLaMA\cite{touvron2023llama} under a non-commercial license and only provided high-level details about the data collection and filtering process.&#10;&#10;Many attempts to fine-tune on specialized domain-specific datasets only use Supervised Fine-Tuning (SFT) or Keep Pretraining (KP), such as Lawyer LLaMA\cite{huang2023lawyer} or Huatuo\cite{wang2023huatuo}. However, after an in-depth analysis of the dataset and experiments using tokenizers of multiple pretrained models, we found that using pre-trained tokenizers on domain-specific datasets is very unwise. Tokenizers in general fields will produce redundancy several times or even dozens of times in the token space, resulting in higher training resource consumption and extended model convergence time. By utilizing the IGOT approach to customize the tokenizer, our experiments demonstrate that the model achieves several-fold improvements in information representation. Additionally, this method conserves training resources and reduces fluctuations in the loss function during training, thereby enabling the model to converge to a more optimal point. To sum up, we make the following contributions:&#10;&#10;\begin{itemize}&#10; \item Our IGOT method enhances the model's capability for information expression by analyzing downstream task data to identify the most effective specialized tokens for customizing the tokenizer. This process effectively transitions the model's tokenizer from a general domain to a vertical domain. Through mathematical methods and experiments, we analyze the reasons why IGOT enables the LLM to outperform traditional SFT or KP in multiple downstream tasks. This investigation elucidates the superior performance of IGOT across various applications.&#10;&#10;\item We show that the IGOT method can be a more universal approach for customizing models for downstream tasks. IGOT enhances model performance and conserves training resources by shifting the tokenizer's representational capabilities to specialized domains. From a mathematical perspective, we analyze why a customized tokenizer is essential for domain-specific LLMs, including information entropy and data compression considerations.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Domain-Specific Training, Linguistics, Artificial Intelligence, Efficient Pretraining Methods, Natural Language Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="1906.05714" label="1906.05714">
        <attvalues>
          <attvalue for="0" value="A Multiscale Visualization of Attention in the Transformer Model" />
          <attvalue for="1" value="  The Transformer is a sequence model that forgoes traditional recurrent&#10;architectures in favor of a fully attention-based approach. Besides improving&#10;performance, an advantage of using attention is that it can also help to&#10;interpret a model by showing how the model assigns weight to different input&#10;elements. However, the multi-layer, multi-head attention mechanism in the&#10;Transformer model can be difficult to decipher. To make the model more&#10;accessible, we introduce an open-source tool that visualizes attention at&#10;multiple scales, each of which provides a unique perspective on the attention&#10;mechanism. We demonstrate the tool on BERT and OpenAI GPT-2 and present three&#10;example use cases: detecting model bias, locating relevant attention heads, and&#10;linking neurons to model behavior.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11967" label="2010.11967">
        <attvalues>
          <attvalue for="0" value="Language Models are Open Knowledge Graphs" />
          <attvalue for="1" value="  This paper shows how to construct knowledge graphs (KGs) from pre-trained&#10;language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs&#10;(e.g, Wikidata, NELL) are built in either a supervised or semi-supervised&#10;manner, requiring humans to create knowledge. Recent deep language models&#10;automatically acquire knowledge from large-scale corpora via pre-training. The&#10;stored knowledge has enabled the language models to improve downstream NLP&#10;tasks, e.g., answering questions, and writing code and articles. In this paper,&#10;we propose an unsupervised method to cast the knowledge contained within&#10;language models into KGs. We show that KGs are constructed with a single&#10;forward pass of the pre-trained language models (without fine-tuning) over the&#10;corpora. We demonstrate the quality of the constructed KGs by comparing to two&#10;KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual&#10;knowledge that is new in the existing KGs. Our code and KGs will be made&#10;publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.05131" label="2205.05131">
        <attvalues>
          <attvalue for="0" value="UL2: Unifying Language Learning Paradigms" />
          <attvalue for="1" value="  Existing pre-trained models are generally geared towards a particular class&#10;of problems. To date, there seems to be still no consensus on what the right&#10;architecture and pre-training setup should be. This paper presents a unified&#10;framework for pre-training models that are universally effective across&#10;datasets and setups. We begin by disentangling architectural archetypes with&#10;pre-training objectives -- two concepts that are commonly conflated. Next, we&#10;present a generalized &amp; unified perspective for self-supervision in NLP and&#10;show how different pre-training objectives can be cast as one another and how&#10;interpolating between different objectives can be effective. We then propose&#10;Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse&#10;pre-training paradigms together. We furthermore introduce a notion of mode&#10;switching, wherein downstream fine-tuning is associated with specific&#10;pre-training schemes. We conduct extensive ablative experiments to compare&#10;multiple pre-training objectives and find that our method pushes the&#10;Pareto-frontier by outperforming T5 &amp; GPT-like models across multiple diverse&#10;setups. By scaling our model up to 20B parameters, we achieve SOTA performance&#10;on 50 well-established supervised finetuning based NLP tasks. Our model also&#10;achieve strong results at in-context learning, outperforming 175B GPT-3 on&#10;zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot&#10;summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B&#10;also works well with chain-of-thought prompting and reasoning, making it an&#10;appealing choice for research into reasoning at a small to medium scale of 20B&#10;parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model,&#10;achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release&#10;Flax-based T5X checkpoints for the UL2 20B &amp; Flan-UL2 20B.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.02414" label="2210.02414">
        <attvalues>
          <attvalue for="0" value="GLM-130B: An Open Bilingual Pre-trained Model" />
          <attvalue for="1" value="  We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language&#10;model with 130 billion parameters. It is an attempt to open-source a 100B-scale&#10;model at least as good as GPT-3 (davinci) and unveil how models of such a scale&#10;can be successfully pre-trained. Over the course of this effort, we face&#10;numerous unexpected technical and engineering challenges, particularly on loss&#10;spikes and divergence. In this paper, we introduce the training process of&#10;GLM-130B including its design choices, training strategies for both efficiency&#10;and stability, and engineering efforts. The resultant GLM-130B model offers&#10;significant outperformance over GPT-3 175B (davinci) on a wide range of popular&#10;English benchmarks while the performance advantage is not observed in OPT-175B&#10;and BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN&#10;3.0 260B -- the largest Chinese language model -- across related benchmarks.&#10;Finally, we leverage a unique scaling property of GLM-130B to reach INT4&#10;quantization without post training, with almost no performance loss, making it&#10;the first among 100B-scale models and more importantly, allowing its effective&#10;inference on 4$\times$RTX 3090 (24G) or 8$\times$RTX 2080 Ti (11G) GPUs, the&#10;most affordable GPUs required for using 100B-scale models. The GLM-130B model&#10;weights are publicly accessible and its code, training logs, related toolkit,&#10;and lessons learned are open-sourced at&#10;\url{https://github.com/THUDM/GLM-130B/}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;In this section, we review related work to \glm on topics of pre-training, transferring, and inference of pre-trained LLMs~\cite{qiu2020pre,bommasani2021opportunities}.&#10;&#10;\vvpara{Pre-Training.}&#10;Vanilla language modeling refers to decoder-only autoregressive models (e.g., GPT~\cite{radford2018improving}), but it also recognizes any forms of self-supervised objectives on texts. Recently, transformer-based~\cite{vaswani2017attention} language models present a fascinating scaling law: new abilities~\cite{wei2022emergent} arise as models scale up, from 1.5B~\cite{radford2019language}, 10B-scale language models~\cite{raffel2020exploring,shoeybi2019megatron,black2022gpt}, to 100B-scale GPT-3~\cite{brown2020language}.&#10;Later, despite many 100B-scale LLMs~\cite{lieber2021jurassic,thoppilan2022lamda,rae2021scaling,smith2022using,chowdhery2022palm,wu2021yuan,zeng2021pangu,wang2021ernie} in both English and Chinese, they are not available to public or only accessible via limited APIs.&#10;The closeness of LLMs severely stymies its development.&#10;\glm's efforts, along with recent ElutherAI, OPT-175B~\cite{zhang2022opt}, and BLOOM-176B~\cite{scao2022what}, aim to offer high-quality open-sourced LLMs to our community.&#10;&#10;\vvpara{Transferring.}&#10;Though fine-tuning has been a de facto way for transfer learning, the evaluation for LLMs has been focused on prompting and in-context learning due to their tremendous sizes~\cite{brown2020language,liu2021pre}.&#10;Nevertheless, some recent attempts has been on parameter-efficient learning on language models&#10;~\cite{houlsby2019parameter} and prompt tuning (i.e., P-tuning, ~\cite{li2021prefix,liu2021gpt,lester2021power,liu2022p}). For now we do not focus on them and will leave the comprehensive testing of them on \glm in future study.&#10;&#10;\vvpara{Inference.}&#10;Most public-accessible LLMs nowadays are providing their services via limited APIs.In this work, an important part of our endeavor has been on LLMs' efficient and fast inference.&#10;Related work may include distillation~\cite{sanh2019distilbert,jiao2020tinybert,wang2020minilm}, quantization~\cite{zafrir2019q8bert,shen2020q,tao2022compression}, and pruning~\cite{michel2019sixteen,fan2019reducing}.&#10;Very recent work~\cite{dettmers2022llm} shows that LLMs such as OPT-175B and BLOOM-176B can be quantized to 8 bit due to special distribution of outlier dimensions.&#10;In this work, we demonstrate GLM's scaling law for INT4 weight quantization, which allows \glm to inference on as few as 4$\times$RTX 3090 (24G) GPUs or 8$\times$RTX 2080 Ti (11G) GPUs.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2203.13474" label="2203.13474">
        <attvalues>
          <attvalue for="0" value="CodeGen: An Open Large Language Model for Code with Multi-Turn Program&#10;  Synthesis" />
          <attvalue for="1" value="  Program synthesis strives to generate a computer program as a solution to a&#10;given problem specification, expressed with input-output examples or natural&#10;language descriptions. The prevalence of large language models advances the&#10;state-of-the-art for program synthesis, though limited training resources and&#10;data impede open access to such models. To democratize this, we train and&#10;release a family of large language models up to 16.1B parameters, called&#10;CODEGEN, on natural language and programming language data, and open source the&#10;training library JAXFORMER. We show the utility of the trained model by&#10;demonstrating that it is competitive with the previous state-of-the-art on&#10;zero-shot Python code generation on HumanEval. We further investigate the&#10;multi-step paradigm for program synthesis, where a single program is factorized&#10;into multiple prompts specifying subproblems. To this end, we construct an open&#10;benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse&#10;problem sets that are factorized into multi-turn prompts. Our analysis on MTPB&#10;shows that the same intent provided to CODEGEN in multi-turn fashion&#10;significantly improves program synthesis over that provided as a single turn.&#10;We make the training library JAXFORMER and model checkpoints available as open&#10;source contribution: https://github.com/salesforce/CodeGen.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15062" label="2305.15062">
        <attvalues>
          <attvalue for="0" value="Lawyer LLaMA Technical Report" />
          <attvalue for="1" value="  Large Language Models (LLMs), like LLaMA, have exhibited remarkable&#10;performance across various tasks. Nevertheless, when deployed to specific&#10;domains such as law or medicine, the models still confront the challenge of a&#10;deficiency in domain-specific knowledge and an inadequate capability to&#10;leverage that knowledge to resolve domain-related problems. In this paper, we&#10;propose a new framework to adapt LLMs to specific domains and build Lawyer&#10;LLaMA, a legal domain LLM, based on this framework. Specifically, we inject&#10;domain knowledge during the continual training stage and teach the model to&#10;learn professional skills using properly designed supervised fine-tuning tasks.&#10;Moreover, to alleviate the hallucination problem during the model's generation,&#10;we add a retrieval module and extract relevant legal articles before the model&#10;answers any queries. When learning domain-specific skills, we find that&#10;experts' experience is much more useful than experiences distilled from&#10;ChatGPT, where hundreds of expert-written data outperform tens of thousands of&#10;ChatGPT-generated ones. We will release our model and data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Previous efforts have tried many methods to take advantage of pre-trained language models to solve legal tasks. LEGAL-BERT~\cite{DBLP:journals/corr/abs-2010-02559} explored applying BERT models to downstream legal tasks using EU and UK legal corpus. Lawformer~\cite{lawformer} is a longformer-based pre-trained language model, specifically further pre-trained on Chinese judicial documents. CaseLawBERT~\cite{caselawbert}&#10;is pre-trained on a corpus of US case law documents and contracts. &#10;These models exhibit limited capabilities in understanding and applying legal knowledge, creating a gap between them and large language models.&#10;&#10;When it comes to the era of Large Language Models, there is no effort to build a legal domain LLM in the literature.&#10;There have been a series of works adapting large language models to other specialized domains.&#10;Med-PaLM~\cite{Med-PaLM} and Med-PaLM 2~\cite{Med-PaLM-2} are two instruction-tuned PaLM~\cite{chowdhery2022palm} models for the medical domain. ChatDoctor~\cite{yunxiang2023chatdoctor} is another model in the medical domain fine-tuned on LLaMA, augmented with a disease database for more reliable responses.&#10;In terms of the financial technology domain, BloombergGPT~\cite{wu2023bloomberggpt} is trained from scratch on a mixture of financial documents and general-domain corpora. As far as we know, we are the first to explore how to build a legal domain-specific LLM.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2304.06975" label="2304.06975">
        <attvalues>
          <attvalue for="0" value="HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge" />
          <attvalue for="1" value="  Large Language Models (LLMs), such as the LLaMA model, have demonstrated&#10;their effectiveness in various general-domain natural language processing (NLP)&#10;tasks. Nevertheless, LLMs have not yet performed optimally in biomedical domain&#10;tasks due to the need for medical expertise in the responses. In response to&#10;this challenge, we propose HuaTuo, a LLaMA-based model that has been&#10;supervised-fine-tuned with generated QA (Question-Answer) instances. The&#10;experimental results demonstrate that HuaTuo generates responses that possess&#10;more reliable medical knowledge. Our proposed HuaTuo model is accessible at&#10;https://github.com/SCIR-HI/Huatuo-Llama-Med-Chinese.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.00416" label="2407.00416">
        <attvalues>
          <attvalue for="0" value="Too Late to Train, Too Early To Use? A Study on Necessity and Viability&#10;  of Low-Resource Bengali LLMs" />
          <attvalue for="1" value="  Each new generation of English-oriented Large Language Models (LLMs) exhibits&#10;enhanced cross-lingual transfer capabilities and significantly outperforms&#10;older LLMs on low-resource languages. This prompts the question: Is there a&#10;need for LLMs dedicated to a particular low-resource language? We aim to&#10;explore this question for Bengali, a low-to-moderate resource Indo-Aryan&#10;language native to the Bengal region of South Asia.&#10;  We compare the performance of open-weight and closed-source LLMs such as&#10;LLaMA-3 and GPT-4 against fine-tuned encoder-decoder models across a diverse&#10;set of Bengali downstream tasks, including translation, summarization,&#10;paraphrasing, question-answering, and natural language inference. Our findings&#10;reveal that while LLMs generally excel in reasoning tasks, their performance in&#10;tasks requiring Bengali script generation is inconsistent. Key challenges&#10;include inefficient tokenization of Bengali script by existing LLMs, leading to&#10;increased computational costs and potential performance degradation.&#10;Additionally, we highlight biases in machine-translated datasets commonly used&#10;for Bengali NLP tasks. We conclude that there is a significant need for a&#10;Bengali-oriented LLM, but the field currently lacks the high-quality&#10;pretraining and instruction-tuning datasets necessary to develop a highly&#10;effective model.&#10;" />
          <attvalue for="2" value="&#10;&#10;The release of GPT-3.5 \cite{fewshot} in late 2022 has kickstarted the current era of rapid progress in Large Language Models (LLMs). However, this progress is not merely a result of increased model scale, rather, it stems from a virtuous cycle of innovation, where lessons from each generation inform the development of the next. Techniques such as synthetic data generation \cite{eldan2023tinystories, gunasekar2023textbooks}, the integration of mathematical and coding tasks to enhance reasoning capabilities \cite{codeReasoning}, and research into adversarial attacks \cite{llmattack} for improved safety have all contributed to the ever-increasing capabilities of LLMs. As illustrated in Figure \ref{fig:open-fig}, developing state-of-the-art LLMs involves filtering vast amounts of web-scraped data, utilizing substantial computational resources, and implementing advanced techniques for alignment and safety.&#10;\looseness=-1&#10;&#10;However, this progress poses a dilemma for low-resource languages like Bengali. Despite being one of the most widely spoken languages, the size of Bengali pretraining and instruction-tuning data are minuscule compared to their English counterparts \cite{bangla_translation, banglabert}. To this date, BanglaT5 \cite{banglat5}, a 248 million parameter encoder-decoder T5 transformer \cite{t5}, remains the most capable Bengali Language Model. Furthermore, prematurely investing in training larger models might yield lackluster results due to the lack of high-quality Bengali data.&#10;\looseness=-1&#10;&#10;In this study, we aim to quantify the demand and viability of a Bengali-oriented LLM. To this end, we compile a representative benchmark of both Natural Language Understanding (NLU) and Natural Language Generation (NLG) downstream tasks for Bengali and evaluate a wide range of open-weights and closed-source models. Our key findings include:&#10;\looseness=-1&#10;\begin{enumerate}&#10; \item Compared to fine-tuned BanglaT5 or BanglaBERT, English-oriented LLMs excel in comprehension tasks (NLU) and perform inconsistently in Bengali generation (NLG).&#10;&#10;\item Using machine translation to translate English NLG datasets into Bengali biases the dataset towards specific writing styles and skews downstream metrics such as BLEU and ROUGE in favor of fine-tuned models regardless of generation quality.&#10;&#10;\item Bengali is over-tokenized by the BPE tokenizer used English LLM, with an average of $\sim0.85$ characters-per-token compared to $\sim4.5$ for English. Over-tokenization leads to $O(n^2)$ attention-based LLMs being highly inefficient in processing Bengali script.&#10;&#10;\item The outputs of English LLMs on Bengali Reward Modeling tasks do not correlate strongly with human judgment. As such, these LLMs have limited applicability in generating Bengali RLHF datasets.&#10;&#10;\end{enumerate}&#10;A comprehensive evaluation of state-of-the-art LLMs on 7 Bengali NLU and NLG tasks, revealing task-dependent performance variations.&#10;An analysis of the inefficient tokenization of Bengali script by existing LLMs and its impact on model performance.&#10;Insights into the challenges and potential strategies for developing Bengali-specific LLMs, balancing the need for language-specific models against the rapid progress in multilingual capabilities of existing LLMs.&#10;\looseness=-1&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Computational Linguistics, Artificial Intelligence, Natural Language Processing, Language Model Performance, Low-Resource Language Challenges" />
        </attvalues>
      </node>
      <node id="2307.15043" label="2307.15043">
        <attvalues>
          <attvalue for="0" value="Universal and Transferable Adversarial Attacks on Aligned Language&#10;  Models" />
          <attvalue for="1" value="  Because &quot;out-of-the-box&quot; large language models are capable of generating a&#10;great deal of objectionable content, recent work has focused on aligning these&#10;models in an attempt to prevent undesirable generation. While there has been&#10;some success at circumventing these measures -- so-called &quot;jailbreaks&quot; against&#10;LLMs -- these attacks have required significant human ingenuity and are brittle&#10;in practice. In this paper, we propose a simple and effective attack method&#10;that causes aligned language models to generate objectionable behaviors.&#10;Specifically, our approach finds a suffix that, when attached to a wide range&#10;of queries for an LLM to produce objectionable content, aims to maximize the&#10;probability that the model produces an affirmative response (rather than&#10;refusing to answer). However, instead of relying on manual engineering, our&#10;approach automatically produces these adversarial suffixes by a combination of&#10;greedy and gradient-based search techniques, and also improves over past&#10;automatic prompt generation methods.&#10;  Surprisingly, we find that the adversarial prompts generated by our approach&#10;are quite transferable, including to black-box, publicly released LLMs.&#10;Specifically, we train an adversarial attack suffix on multiple prompts (i.e.,&#10;queries asking for many different types of objectionable content), as well as&#10;multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting&#10;attack suffix is able to induce objectionable content in the public interfaces&#10;to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat,&#10;Pythia, Falcon, and others. In total, this work significantly advances the&#10;state-of-the-art in adversarial attacks against aligned language models,&#10;raising important questions about how such systems can be prevented from&#10;producing objectionable information. Code is available at&#10;github.com/llm-attacks/llm-attacks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.09359" label="2009.09359">
        <attvalues>
          <attvalue for="0" value="Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New&#10;  Datasets for Bengali-English Machine Translation" />
          <attvalue for="1" value="  Despite being the seventh most widely spoken language in the world, Bengali&#10;has received much less attention in machine translation literature due to being&#10;low in resources. Most publicly available parallel corpora for Bengali are not&#10;large enough; and have rather poor quality, mostly because of incorrect&#10;sentence alignments resulting from erroneous sentence segmentation, and also&#10;because of a high volume of noise present in them. In this work, we build a&#10;customized sentence segmenter for Bengali and propose two novel methods for&#10;parallel corpus creation on low-resource setups: aligner ensembling and batch&#10;filtering. With the segmenter and the two methods combined, we compile a&#10;high-quality Bengali-English parallel corpus comprising of 2.75 million&#10;sentence pairs, more than 2 million of which were not available before.&#10;Training on neural models, we achieve an improvement of more than 9 BLEU score&#10;over previous approaches to Bengali-English machine translation. We also&#10;evaluate on a new test set of 1000 pairs made with extensive quality control.&#10;We release the segmenter, parallel corpus, and the evaluation set, thus&#10;elevating Bengali from its low-resource status. To the best of our knowledge,&#10;this is the first ever large scale study on Bengali-English machine&#10;translation. We believe our study will pave the way for future research on&#10;Bengali-English machine translation as well as other low-resource languages.&#10;Our data and code are available at https://github.com/csebuetnlp/banglanmt.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.11081" label="2205.11081">
        <attvalues>
          <attvalue for="0" value="BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating&#10;  Low-Resource Natural Language Generation in Bangla" />
          <attvalue for="1" value="  This work presents BanglaNLG, a comprehensive benchmark for evaluating&#10;natural language generation (NLG) models in Bangla, a widely spoken yet&#10;low-resource language. We aggregate six challenging conditional text generation&#10;tasks under the BanglaNLG benchmark, introducing a new dataset on dialogue&#10;generation in the process. Furthermore, using a clean corpus of 27.5 GB of&#10;Bangla data, we pretrain BanglaT5, a sequence-to-sequence Transformer language&#10;model for Bangla. BanglaT5 achieves state-of-the-art performance in all of&#10;these tasks, outperforming several multilingual models by up to 9% absolute&#10;gain and 32% relative gain. We are making the new dialogue dataset and the&#10;BanglaT5 model publicly available at https://github.com/csebuetnlp/BanglaNLG in&#10;the hope of advancing future research on Bangla NLG.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11430" label="2305.11430">
        <attvalues>
          <attvalue for="0" value="TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks" />
          <attvalue for="1" value="  While LLMs have shown great success in understanding and generating text in&#10;traditional conversational settings, their potential for performing ill-defined&#10;complex tasks is largely under-studied. Indeed, we are yet to conduct&#10;comprehensive benchmarking studies with multiple LLMs that are exclusively&#10;focused on a complex task. However, conducting such benchmarking studies is&#10;challenging because of the large variations in LLMs' performance when different&#10;prompt types/styles are used and different degrees of detail are provided in&#10;the prompts. To address this issue, the paper proposes a general taxonomy that&#10;can be used to design prompts with specific properties in order to perform a&#10;wide range of complex tasks. This taxonomy will allow future benchmarking&#10;studies to report the specific categories of prompts used as part of the study,&#10;enabling meaningful comparisons across different studies. Also, by establishing&#10;a common standard through this taxonomy, researchers will be able to draw more&#10;accurate conclusions about LLMs' performance on a specific complex task.&#10;" />
          <attvalue for="2" value="&#10;Recently, conversational Large Language Models (LLMs) such as GPT-3~\cite{brown2020language}, Bard~\cite{thoppilan2022lamda}, LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, PaLM~\cite{chowdhery2022palm}, etc. have demonstrated exceptional performance in a wide range of popular natural language processing (NLP) tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using}. Prompt, as a stimulator, refers to a textual input provided to the LLMs with the intention of guiding its output toward a specific task. Unsurprisingly, the quality and effectiveness of the prompt can greatly influence the performance of the LLMs for a particular task, and therefore, designing appropriate prompts with the right amount of detail has become more important than ever~\cite{liu2023pre,han2022ptr}. &#10;&#10;In recent years, researchers have spent a significant amount of effort proposing different ways of designing ``appropriate'' prompts. For example, \cite{brown2020language} showed a standard prompting technique with question-answer pairs that can result in a few-shot effect. Researchers also explored other prompt design techniques such as Chain-of-thought (CoT)~\cite{wei2022chain}, Reasoning and Acting (ReAct)~\cite{yao2022react}, and other techniques~\cite{kojima2022large,madaan2022text,press2022measuring} in terms of improving the reasoning and acting of LLMs in solving Question-Answering tasks. Meanwhile, \cite{kim2023language} proposed a prompting scheme where the agent recursively criticizes and improves its output (RCI) to solve a task. However, these experiments primarily emphasized the utilization of diverse prompts to evaluate the ability of LLMs to perform ``well-defined'' NLP tasks, while studies with diverse prompts for ill-defined complex tasks are still rare, if not nonexistent.&#10;&#10;While conducting multiple benchmarking studies with various LLMs for complex tasks seems interesting and compelling, conducting such studies is challenging because of the large variations in LLMs' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts, especially in case of complex tasks. In this paper, we exclusively focus on understanding LLMs' potential for performing complex tasks that are mostly: 1) ill-defined, 2) abstract goal-oriented, 3) highly dependent on subjective interpretation, and 4) very hard to evaluate quantitatively~\cite{khot2022decomposed,press2022measuring}. These complex tasks often involve multiple steps/sub-tasks, and designing ``appropriate'' prompts for such tasks is indeed challenging as there is no single rule book to follow in these cases~\cite{zelikman2022star,nye2021show}. A further complication arises if we want to compare two independent benchmarking studies targeted towards the same goal (complex) task. Such a complication arises because, for a given complex task and a particular LLM, the performance of the LLM can drastically vary when different types/styles of prompts are fed to it. Indeed, the exact details included in the prompt play a big role in how LLMs will perform in solving the goal complex task. This indeed creates a problem for evaluation and benchmarking purposes if an apple-to-apple comparison is not made in terms of the prompts that are provided to the LLMs. In other words, just reporting accuracy numbers for LLMs without specifying the finer details of the prompts used in the experiments makes comparisons across LLMs meaningless. &#10;&#10;Unfortunately, every complex task is different, and so are the prompts users can try to perform the task; therefore, a general taxonomy that can categorize these diverse kinds of prompts using a single standard/taxonomy has now become a pressing need. The main contribution of this paper is to introduce one such general taxonomy (we name it TELeR) that can be used by any benchmarking study that leverages LLMs to perform some complex task. The major benefit of adopting our proposed TELeR taxonomy is that it will facilitate more meaningful comparisons among multiple LLMs in terms of their performances across various complex tasks reported by multiple independent groups of researchers/developers and, thus, help derive more accurate conclusions. TELeR will achieve this goal by grounding different types of prompts into a common standard and allowing an apple-to-apple comparison across different prompt categories using the same standard. As such, this taxonomy will serve as an important tool to establish a common consensus around the state-of-the-art LLM performance for performing complex tasks. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Complex Task Benchmarking, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Prompt Taxonomy Design" />
        </attvalues>
      </node>
      <node id="2201.08239" label="2201.08239">
        <attvalues>
          <attvalue for="0" value="LaMDA: Language Models for Dialog Applications" />
          <attvalue for="1" value="  We present LaMDA: Language Models for Dialog Applications. LaMDA is a family&#10;of Transformer-based neural language models specialized for dialog, which have&#10;up to 137B parameters and are pre-trained on 1.56T words of public dialog data&#10;and web text. While model scaling alone can improve quality, it shows less&#10;improvements on safety and factual grounding. We demonstrate that fine-tuning&#10;with annotated data and enabling the model to consult external knowledge&#10;sources can lead to significant improvements towards the two key challenges of&#10;safety and factual grounding. The first challenge, safety, involves ensuring&#10;that the model's responses are consistent with a set of human values, such as&#10;preventing harmful suggestions and unfair bias. We quantify safety using a&#10;metric based on an illustrative set of human values, and we find that filtering&#10;candidate responses using a LaMDA classifier fine-tuned with a small amount of&#10;crowdworker-annotated data offers a promising approach to improving model&#10;safety. The second challenge, factual grounding, involves enabling the model to&#10;consult external knowledge sources, such as an information retrieval system, a&#10;language translator, and a calculator. We quantify factuality using a&#10;groundedness metric, and we find that our approach enables the model to&#10;generate responses grounded in known sources, rather than responses that merely&#10;sound plausible. Finally, we explore the use of LaMDA in the domains of&#10;education and content recommendations, and analyze their helpfulness and role&#10;consistency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.06905" label="2112.06905">
        <attvalues>
          <attvalue for="0" value="GLaM: Efficient Scaling of Language Models with Mixture-of-Experts" />
          <attvalue for="1" value="  Scaling language models with more data, compute and parameters has driven&#10;significant progress in natural language processing. For example, thanks to&#10;scaling, GPT-3 was able to achieve strong results on in-context learning tasks.&#10;However, training these large dense models requires significant amounts of&#10;computing resources. In this paper, we propose and develop a family of language&#10;models named GLaM (Generalist Language Model), which uses a sparsely activated&#10;mixture-of-experts architecture to scale the model capacity while also&#10;incurring substantially less training cost compared to dense variants. The&#10;largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than&#10;GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half&#10;of the computation flops for inference, while still achieving better overall&#10;zero-shot and one-shot performance across 29 NLP tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Language models. Neural language models~\cite{Mikolov2010rnnlm,sutskever2011_rnnlm} have been shown to be useful for many natural language processing tasks.&#10;Word embedding models and extensions such as word2vec~\cite{mikolov2013efficient}, GloVe~\cite{pennington-glove} and paragraph vectors~\cite{le2014distributed}&#10;have shown good generalization to many tasks simply by transferring the embeddings. &#10;&#10;Pre-training and Fine-tuning. The abundance of compute and data enables training increasingly large models via unsupervised pre-training. This is a natural fit for training neural networks as they exhibit remarkable scalability. Work on using recurrent models such as RNNs and LSTMs for language representation~\cite{NIPS2015_dai,kiros-skip-thought} showed that general language models could be fine-tuned to improve various language understanding tasks. More recently, models that used Transformers~\cite{vaswani2017attention} showed that larger models with self-supervision on unlabeled data could yield significant improvements on NLP tasks~\cite{devlin2018bert,yang2019xlnet,liu2019roberta,clark2020electra}. Transfer learning based on pre-training and finetuning~\cite{raffel2020exploring,houlsby2019parameterefficient} has been extensively studied and demonstrated good performance on downstream tasks. However, a major limitation to this method is that it requires a task-specific fine-tuning. &#10;&#10;In-Context Few-shot Learning. GPT-3~\cite{NEURIPS2020_gpt3} and related work~\cite{shoeybi2019megatron,lieber2021jurassic,wei2021finetuned} demonstrated that scaling up language models greatly improves task-agnostic, few-shot performance. These language models are applied without any gradient updates, and only few-shot demonstrations specified purely via text interactions with the model are needed. &#10;&#10;Sparsely Gated Networks.&#10;Mixture-of-Experts based models have also shown significant advantages. For language modeling and machine translation, \cite{shazeer2017outrageously} showed that they could effectively use a very large number of weights while only needing to compute a small subset of the computation graph at inference time. There has also been work on scaling sparsely activated MoE architectures~\cite{hestness2017deep,shazeer2018mesh,lepikhin2020gshard, kudugunta2021beyond}. Recently, \cite{fedus2021switch} showed results with even larger 1 trillion parameter sparsely activated models (Switch-C). Although both Switch-C and the largest GLaM model have one trillion number of trainable parameters, GLaM is a family of decoder-only language models, and Switch-C is an encoder-decoder based sequence to sequence model. Furthermore, &#10;Switch-C is mainly evaluated on fine-tuning benchmarks, \eg, SuperGlue, while GLaM performs well without any need for fine-tuning in the few-shot setting shared by GPT-3 where SuperGlue is a subset.&#10;Table~\ref{tab:various-networks} summarizes the key differences between GLaM and related models pre-trained on text corpora. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2201.11990" label="2201.11990">
        <attvalues>
          <attvalue for="0" value="Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A&#10;  Large-Scale Generative Language Model" />
          <attvalue for="1" value="  Pretrained general-purpose language models can achieve state-of-the-art&#10;accuracies in various natural language processing domains by adapting to&#10;downstream tasks via zero-shot, few-shot and fine-tuning techniques. Because of&#10;their success, the size of these models has increased rapidly, requiring&#10;high-performance hardware, software, and algorithmic techniques to enable&#10;training such large models. As the result of a joint effort between Microsoft&#10;and NVIDIA, we present details on the training of the largest monolithic&#10;transformer based language model, Megatron-Turing NLG 530B (MT-NLG), with 530&#10;billion parameters. In this paper, we first focus on the infrastructure as well&#10;as the 3D parallelism methodology used to train this model using DeepSpeed and&#10;Megatron. Next, we detail the training process, the design of our training&#10;corpus, and our data curation techniques, which we believe is a key ingredient&#10;to the success of the model. Finally, we discuss various evaluation results, as&#10;well as other interesting observations and new properties exhibited by MT-NLG.&#10;We demonstrate that MT-NLG achieves superior zero-, one-, and few-shot learning&#10;accuracies on several NLP benchmarks and establishes new state-of-the-art&#10;results. We believe that our contributions will help further the development of&#10;large-scale training infrastructures, large-scale language models, and natural&#10;language generations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.11259" label="2105.11259">
        <attvalues>
          <attvalue for="0" value="PTR: Prompt Tuning with Rules for Text Classification" />
          <attvalue for="1" value="  Fine-tuned pre-trained language models (PLMs) have achieved awesome&#10;performance on almost all NLP tasks. By using additional prompts to fine-tune&#10;PLMs, we can further stimulate the rich knowledge distributed in PLMs to better&#10;serve downstream tasks. Prompt tuning has achieved promising results on some&#10;few-class classification tasks such as sentiment classification and natural&#10;language inference. However, manually designing lots of language prompts is&#10;cumbersome and fallible. For those auto-generated prompts, it is also expensive&#10;and time-consuming to verify their effectiveness in non-few-shot scenarios.&#10;Hence, it is still challenging for prompt tuning to address many-class&#10;classification tasks. To this end, we propose prompt tuning with rules (PTR)&#10;for many-class text classification and apply logic rules to construct prompts&#10;with several sub-prompts. In this way, PTR is able to encode prior knowledge of&#10;each class into prompt tuning. We conduct experiments on relation&#10;classification, a typical and complicated many-class classification task, and&#10;the results show that PTR can significantly and consistently outperform&#10;existing state-of-the-art baselines. This indicates that PTR is a promising&#10;approach to take advantage of both human prior knowledge and PLMs for those&#10;complicated classification tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Various recent PLMs like GPT~\cite{radfordimproving}, BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and T5~\cite{raffel2020exploring} provide a new approach to utilize large-scale unlabeled data for NLP tasks. Although these PLMs can capture rich knowledge~\cite{jawahar2019does,hewitt2019structural,petroni2019language,yenicelik2020does} from massive corpora, a fine-tuning process with extra task-specific data is still required to transfer their knowledge for downstream tasks. From dialogue~\cite{zhang2019dialogpt}, summarization~\cite{zhang2019pegasus,liu-lapata-2019-text}, question answering~\cite{adiwardana2020humanlike}, to text classification~\cite{baldini-soares-etal-2019-matching,peng2020learning, ding2021prototypical}, fine-tuned PLMs have been demonstrated their effectiveness on almost all important NLP tasks. Besides fine-tuning language models for specific tasks, recent studies have explored better optimization and regularization techniques to improve fine-tuning~\cite{lee2019mixout,dodge2020fine}. &#10;&#10;Despite the success of fine-tuning PLMs, there is a big gap between pre-training objectives and fine-tuning objectives. In the pre-training phase, sequential language models and masked language models are used to learn PLMs. In the fine-tuning phase, the optimization objectives are task-specific and different tasks may have quite different objective forms. In GPT-3~\cite{brown2020language}, prompt tuning has been proposed and drawn much attention. By leveraging language prompts as contexts, downstream tasks can be expressed as some objectives similar to pre-training objectives. Through a series of research work on knowledge probing~\cite{trinh2018simple,petroni2019language,davison2019commonsense}, language prompts have been proven to effectively stimulate knowledge from PLMs. Moreover, human-picked prompts have achieved promising results on few-class classification tasks such as sentiment classification and natural language inference~\cite{schick2020exploiting,liu2021gpt}. &#10;&#10;To avoid labor-intensive prompt design, automatic prompt search has been extensively explored. As a typical prompt consists of two parts: a template and a set of label words, \cite{schick2020automatically,schick2020exploiting} first explore automatic identification of label words for human-picked templates. \cite{shin2020eliciting} further explore gradient-guided search to automatically generate both templates and label words. \cite{gao2020making} take seq-to-seq models to generate prompt candidates, and then use each of them for prompt tuning and verify their effectiveness on development sets. Compared with human-picked prompts, most auto-generated prompts cannot achieve comparable performance. Recently, some continuous prompts have also been proposed~\cite{li2021prefix,lester2021power}, which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts. These completely continuous prompts work well on those large-scale PLMs with billions of parameters, yet cannot stably work on normal PLMs.&#10;&#10;In this paper, we propose PTR to conduct prompt tuning with rules. Based on several human-picked sub-prompts, PTR can use predefined logic rules to compose sub-prompts into complete task-specific prompts. As compared with the above-mentioned prompt tuning methods, PTR achieves a good balance among model efficiency, model effectiveness, model generalization, and human workload.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2209.07686" label="2209.07686">
        <attvalues>
          <attvalue for="0" value="Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango" />
          <attvalue for="1" value="  The past decade has witnessed dramatic gains in natural language processing&#10;and an unprecedented scaling of large language models. These developments have&#10;been accelerated by the advent of few-shot techniques such as chain of thought&#10;(CoT) prompting. Specifically, CoT pushes the performance of large language&#10;models in a few-shot setup by augmenting the prompts with intermediate steps.&#10;Despite impressive results across various tasks, the reasons behind their&#10;success have not been explored. This work uses counterfactual prompting to&#10;develop a deeper understanding of CoT-based few-shot prompting mechanisms in&#10;large language models. We first systematically identify and define the key&#10;components of a prompt: symbols, patterns, and text. Then, we devise and&#10;conduct an exhaustive set of experiments across four different tasks, by&#10;querying the model with counterfactual prompts where only one of these&#10;components is altered. Our experiments across three models (PaLM, GPT-3, and&#10;CODEX) reveal several surprising findings and brings into question the&#10;conventional wisdom around few-shot prompting. First, the presence of factual&#10;patterns in a prompt is practically immaterial to the success of CoT. Second,&#10;our results conclude that the primary role of intermediate steps may not be to&#10;facilitate learning how to solve a task. The intermediate steps are rather a&#10;beacon for the model to realize what symbols to replicate in the output to form&#10;a factual answer. Further, text imbues patterns with commonsense knowledge and&#10;meaning. Our empirical and qualitative analysis reveals that a symbiotic&#10;relationship between text and patterns explains the success of few-shot&#10;prompting: text helps extract commonsense from the question to help patterns,&#10;and patterns enforce task understanding and direct text generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.03350" label="2210.03350">
        <attvalues>
          <attvalue for="0" value="Measuring and Narrowing the Compositionality Gap in Language Models" />
          <attvalue for="1" value="  We investigate the ability of language models to perform compositional&#10;reasoning tasks where the overall solution depends on correctly composing the&#10;answers to sub-problems. We measure how often models can correctly answer all&#10;sub-problems but not generate the overall solution, a ratio we call the&#10;compositionality gap. We evaluate this ratio by asking multi-hop questions with&#10;answers that require composing multiple facts unlikely to have been observed&#10;together during pretraining. In the GPT-3 family of models, as model size&#10;increases we show that the single-hop question answering performance improves&#10;faster than the multi-hop performance does, therefore the compositionality gap&#10;does not decrease. This surprising result suggests that while more powerful&#10;models memorize and recall more factual knowledge, they show no corresponding&#10;improvement in their ability to perform this kind of compositional reasoning.&#10;  We then demonstrate how elicitive prompting (such as chain of thought)&#10;narrows the compositionality gap by reasoning explicitly. We present a new&#10;method, self-ask, that further improves on chain of thought. In our method, the&#10;model explicitly asks itself (and answers) follow-up questions before answering&#10;the initial question. We finally show that self-ask's structured prompting lets&#10;us easily plug in a search engine to answer the follow-up questions, which&#10;additionally improves accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.17491" label="2303.17491">
        <attvalues>
          <attvalue for="0" value="Language Models can Solve Computer Tasks" />
          <attvalue for="1" value="  Agents capable of carrying out general tasks on a computer can improve&#10;efficiency and productivity by automating repetitive tasks and assisting in&#10;complex problem-solving. Ideally, such agents should be able to solve new&#10;computer tasks presented to them through natural language commands. However,&#10;previous approaches to this problem require large amounts of expert&#10;demonstrations and task-specific reward functions, both of which are&#10;impractical for new tasks. In this work, we show that a pre-trained large&#10;language model (LLM) agent can execute computer tasks guided by natural&#10;language using a simple prompting scheme where the agent Recursively Criticizes&#10;and Improves its output (RCI). The RCI approach significantly outperforms&#10;existing LLM methods for automating computer tasks and surpasses supervised&#10;learning (SL) and reinforcement learning (RL) approaches on the MiniWoB++&#10;benchmark. We compare multiple LLMs and find that RCI with the&#10;InstructGPT-3+RLHF LLM is state-of-the-art on MiniWoB++, using only a handful&#10;of demonstrations per task rather than tens of thousands, and without a&#10;task-specific reward function. Furthermore, we demonstrate RCI prompting's&#10;effectiveness in enhancing LLMs' reasoning abilities on a suite of natural&#10;language reasoning tasks, outperforming chain of thought (CoT) prompting with&#10;external feedback. We find that RCI combined with CoT performs better than&#10;either separately. Our code can be found here:&#10;https://github.com/posgnu/rci-agent.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.14465" label="2203.14465">
        <attvalues>
          <attvalue for="0" value="STaR: Bootstrapping Reasoning With Reasoning" />
          <attvalue for="1" value="  Generating step-by-step &quot;chain-of-thought&quot; rationales improves language model&#10;performance on complex reasoning tasks like mathematics or commonsense&#10;question-answering. However, inducing language model rationale generation&#10;currently requires either constructing massive rationale datasets or&#10;sacrificing accuracy by using only few-shot inference. We propose a technique&#10;to iteratively leverage a small number of rationale examples and a large&#10;dataset without rationales, to bootstrap the ability to perform successively&#10;more complex reasoning. This technique, the &quot;Self-Taught Reasoner&quot; (STaR),&#10;relies on a simple loop: generate rationales to answer many questions, prompted&#10;with a few rationale examples; if the generated answers are wrong, try again to&#10;generate a rationale given the correct answer; fine-tune on all the rationales&#10;that ultimately yielded correct answers; repeat. We show that STaR&#10;significantly improves performance on multiple datasets compared to a model&#10;fine-tuned to directly predict final answers, and performs comparably to&#10;fine-tuning a 30$\times$ larger state-of-the-art language model on&#10;CommensenseQA. Thus, STaR lets a model improve itself by learning from its own&#10;generated reasoning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.00114" label="2112.00114">
        <attvalues>
          <attvalue for="0" value="Show Your Work: Scratchpads for Intermediate Computation with Language&#10;  Models" />
          <attvalue for="1" value="  Large pre-trained language models perform remarkably well on tasks that can&#10;be done &quot;in one pass&quot;, such as generating realistic text or synthesizing&#10;computer programs. However, they struggle with tasks that require unbounded&#10;multi-step computation, such as adding integers or executing programs.&#10;Surprisingly, we find that these same models are able to perform complex&#10;multi-step computations -- even in the few-shot regime -- when asked to perform&#10;the operation &quot;step by step&quot;, showing the results of intermediate computations.&#10;In particular, we train transformers to perform multi-step computations by&#10;asking them to emit intermediate computation steps into a &quot;scratchpad&quot;. On a&#10;series of increasingly complex tasks ranging from long addition to the&#10;execution of arbitrary programs, we show that scratchpads dramatically improve&#10;the ability of language models to perform multi-step computations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The tasks in this paper can be viewed as exploring&#10;one criticism of large language models, namely, to what extent do they simply rely on surface-level statistical correlations on text, without&#10;learning semantics or world knowledge \cite{bender-koller-2020-climbing}?&#10;In response, \cite{Li2021ImplicitRO} provide evidence that pre-trained language models do indeed construct approximate representations of the semantics of the situations they describe in text. &#10;In the context of programs, &#10;\cite{austin2021program} approach this question by exploring the learning to execute task on MBPP, which we consider in Section~\ref{sec:python_mbpp}. &#10;The idea behind this task was to explore whether neural models for synthesis that generate code could also execute it. &#10;While that work finds existing models perform poorly at predicting execution, we show that adding a scratchpad allows these models to perform better.&#10;&#10;Work in learning &#10;to execute has considered whether off-the-shelf recurrent&#10;neural networks \cite{Zaremba2014LearningTE} or&#10;more specialized architectures \cite{Dehghani2018-ur,NEURIPS2020_Bieber, wang2020learning}&#10;have an inductive bias that is sufficiently well suited for executing and reasoning about arbitrary code.&#10;The related problem of neural algorithm induction has attracted considerable interest \cite{ntm,nram,ngpu,dnc,programmerInterpreter,v2020pointer,v2020neural}.&#10;This work proposes new neural architectures, inspired by theoretical models of computation, whose inductive bias allows&#10;them to more easily learn algorithm induction tasks.&#10;Several methods for algorithm induction specifically add adaptive computation time to sequence models \cite{ACT,Dehghani2018-ur,PONDERNET}. &#10;In particular, universal transformers include adaptive computation time, and are evaluated both on algorithm induction and on learning to execute tasks&#10;\cite{Dehghani2018-ur}.&#10;In contrast, a scratchpad&#10;is a simple way both to provide a transformer model with adaptive computation time, and also to provide supervision about how to use that additional computation, without requiring modification to the underlying architecture. &#10;&#10;Algorithm induction has also been connected to pre-trained models.&#10;\cite{Lu2021-ev} show that Transformers can be used to some extent as universal computation engines, by pre-training on natural&#10;language, and fine-tuning a small fraction of the weights on non-language tasks, including simple algorithm induction tasks.&#10;Finally, supervised approaches to semantic parsing \cite{Zelle1996-vt,Zettlemoyer2005,Kwiatkowksi2010-jn,Wong2006-wc}&#10;predict the text of a database query, which can then be executed to answer a natural language question. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="1906.02715" label="1906.02715">
        <attvalues>
          <attvalue for="0" value="Visualizing and Measuring the Geometry of BERT" />
          <attvalue for="1" value="  Transformer architectures show significant promise for natural language&#10;processing. Given that a single pretrained model can be fine-tuned to perform&#10;well on many different tasks, these networks appear to extract generally useful&#10;linguistic features. A natural question is how such networks represent this&#10;information internally. This paper describes qualitative and quantitative&#10;investigations of one particularly effective model, BERT. At a high level,&#10;linguistic features seem to be represented in separate semantic and syntactic&#10;subspaces. We find evidence of a fine-grained geometric representation of word&#10;senses. We also present empirical descriptions of syntactic representations in&#10;both attention matrices and individual word embeddings, as well as a&#10;mathematical argument to explain the geometry of these representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.11364" label="2203.11364">
        <attvalues>
          <attvalue for="0" value="An Information-theoretic Approach to Prompt Engineering Without Ground&#10;  Truth Labels" />
          <attvalue for="1" value="  Pre-trained language models derive substantial linguistic and factual&#10;knowledge from the massive corpora on which they are trained, and prompt&#10;engineering seeks to align these models to specific tasks. Unfortunately,&#10;existing prompt engineering methods require significant amounts of labeled&#10;data, access to model parameters, or both. We introduce a new method for&#10;selecting prompt templates \textit{without labeled examples} and&#10;\textit{without direct access to the model}. Specifically, over a set of&#10;candidate templates, we choose the template that maximizes the mutual&#10;information between the input and the corresponding model output. Across 8&#10;datasets representing 7 distinct NLP tasks, we show that when a template has&#10;high mutual information, it also has high accuracy on the task. On the largest&#10;model, selecting prompts with our method gets 90\% of the way from the average&#10;prompt accuracy to the best prompt accuracy and requires no ground truth&#10;labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The promise of language models and the challenge of aligning them has given rise to the field of prompt engineering, which seeks to construct the best prompt given a task and a language model \cite{Liu2021pre}. The best performance on prompt engineering is often achieved using backpropagation in continuous prompt embedding space \cite{Lester2021, Li2021, Gu2021, Liu2021,Zhang2021} in contrast to generating a discrete set of prompts by hand and testing them. While optimizing in continuous prompt space via backprop allows for similar performance to model-tuning (at least at higher model sizes) \cite{Lester2021}, not all models are publicly available. Thus, these methods are only feasible for those who have direct access to the model and can perform backprop on it. Prompts optimized in continuous space are also not interpretable in natural language, making it harder to transfer insights from prompts that work well for one task to another task. Additionally, these methods require labeled examples, while ours does not.&#10;&#10;Other selection protocols not based on gradient descent can include cross-validation or minimum description length, as in \cite{Perez2021}. These methods yield prompts that perform marginally better than average in terms of test accuracy. &#10;&#10;Mutual information has been used in n-gram clustering, part-of-speech tagging, probing classifiers, and LM training objective reframing \cite{brown-etal-1992-class, stratos2019mutual, Voita2020, kong2019mutual}. Ours is the first work of which we are aware to apply MI to prompt engineering.&#10;\cite{Lu2021} make use of entropy statistics to determine performant orderings for few-shot examples in prompts. Our work is focused on selecting high quality templates with no special focus on example ordering or need for multiple examples to order (the few-shot case). Our method uses no artificial ``probing set,'' making our prompt selection much cheaper, and we also explore open-ended tasks. While the GlobalE and LocalE statistics they use are similar (and in the case of LocalE identical) to the two parts of our MI calculation (see \ref{subsec:MI}), we use the two statistics jointly and choose prompts by minimizing, rather than maximizing, LocalE.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2304.07327" label="2304.07327">
        <attvalues>
          <attvalue for="0" value="OpenAssistant Conversations -- Democratizing Large Language Model&#10;  Alignment" />
          <attvalue for="1" value="  Aligning large language models (LLMs) with human preferences has proven to&#10;drastically improve usability and has driven rapid adoption as demonstrated by&#10;ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and&#10;reinforcement learning from human feedback (RLHF) greatly reduce the required&#10;skill and domain knowledge to effectively harness the capabilities of LLMs,&#10;increasing their accessibility and utility across various domains. However,&#10;state-of-the-art alignment techniques like RLHF rely on high-quality human&#10;feedback data, which is expensive to create and often remains proprietary. In&#10;an effort to democratize research on large-scale alignment, we release&#10;OpenAssistant Conversations, a human-generated, human-annotated assistant-style&#10;conversation corpus consisting of 161,443 messages in 35 different languages,&#10;annotated with 461,292 quality ratings, resulting in over 10,000 complete and&#10;fully annotated conversation trees. The corpus is a product of a worldwide&#10;crowd-sourcing effort involving over 13,500 volunteers. Models trained on&#10;OpenAssistant Conversations show consistent improvements on standard benchmarks&#10;over respective base models. We release our code and data under a fully&#10;permissive licence.&#10;" />
          <attvalue for="2" value="&#10;&#10;{&#10;Artificial intelligence (AI), particularly in the field of natural language processing, has witnessed rapid progress in recent years. Major advancements are primarily driven by a straightforward formula: take a Transformer~\cite{vaswani2017attention}-based architecture, increase the parameter count by enlarging depth and width, increase the size of the training corpus, and increase the scale of training compute. Although models have for some time exhibited an extraordinary ability to fit the training data and generalize based on their trained objective~\cite{touvron2023llama, biderman2023pythia}, their adoption among the general public has until recently been slow. This can be mainly attributed to misalignment between model predictions and final intended usage. &#10;&#10;The alignment of AI systems to human values, intentions, and preferences is a vital and intricate challenge within the AI research domain.&#10;This refers to the process of ensuring that AI systems can not only successfully optimize the provided surrogate training objectives, but also that their predictions are in line with their intended purpose and adhere to ethical and safety standards provided by humans~\cite{Gabriel2020,wang2023aligning}.&#10;One possible solution is assistant-style fine-tuning of language models that has recently emerged as a promising approach to making large language models more in line with human preferences by generating more desirable outputs based on explicitly collected human preference data~\cite{bai2022training, pmlr-v162-ethayarajh22a, thoppilan2022lamda, hilton2021webgpt,menick2022teaching,ziegler2020finetuning} and thus making them more useful.&#10;&#10;A notable instance of such an assistant-style model is ChatGPT, which has gained unprecedented user growth due to remarkable capabilities demonstrated in a wide range of fields, but also ease-of-use for the end user~\cite{Koonchanok2023TrackingPA}. Aligning the model’s predictions is in this case accomplished by introducing human-generated examples of intended usage and using reinforcement learning from human feedback~\cite{ouyang2022training, leandro_von_werra_2023_7790115}.&#10;In RLHF, the human acts as a teacher and provides feedback in the form of rewards or penalties. In more detail, Ouyang et al.~\cite{ouyang2022training} proposed a three stage procedure to align language models:&#10; First, collect human-generated demonstrations of desired behaviour and train a supervised fine-tuned (SFT) model.&#10; Second, train a reward model (RM) on human-annotated rankings for different model outputs.&#10; Third, use the RM as a reward function and fine-tune the SFT model to maximize the reward generated by its responses. This is achieved using the PPO algorithm~\cite{schulman2017proximal}.&#10;&#10;It becomes apparent that the benefits of all the aforementioned stages are predominantly dependent on the quality of the data used~\cite{DataQuality}. Despite this, availability of large-scale human feedback datasets for the open research community remains scarce.&#10;Most openly accessible datasets are comprised of synthetic data of instructions automatically generated by querying language models~\cite{wang2022self, alpaca, anandgpt4all, peng2023instruction}. Unfortunately, these datasets are limited with respect to their complexity, creativity and quality, as they rely on a pre-specified list of possible instruction types. &#10;Other datasets, such as Vicuna~\cite{vicuna}, use human-generated instructions, but still rely on langauge models to produce the respective responses.&#10;Without sufficiently broad and high quality data, even models with substantial size and pre-training would be inadequate for building capable, helpful, and harmless AI assistants.&#10;&#10;Research in this area has predominantly been confined to a select few research labs with access to the required resources to engage in large-scale training and data collection.&#10;This monopolization of access to quality data undermines the potential for inclusive and diverse research endeavours, particularly in relation to alignment challenges, which arguably constitute some of the most crucial research areas of our time.&#10;In an effort to democratize research on aligning large language models, we introduce and release the OpenAssistant Conversations dataset.&#10;This dataset is the culmination of an extensive open- and crowd-sourcing initiative, and its release to the research community seeks to promote more inclusive research in this highly-influential domain.&#10;We provide a comprehensive analysis of the dataset, assessing ethical implications and safety considerations. We also fine-tune and release several assistant and preference models to further advance open access and research in this area.&#10;This transparency allows for iterative improvements on the released artifacts, fostering a more collaborative and inclusive research environment.&#10;By providing such a large and diverse dataset, OpenAssistant Conversations opens up new avenues of research in the field, enabling researchers to explore the complexities of human language and interactions in ways that were not possible before~\cite{clark1991grounding}.&#10;In the following sections, we delve into the intricacies of the OpenAssistant Conversations dataset and discuss its implications for the alignment of large language models and for society at large.&#10;}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Human Feedback Data, Linguistics, Large-Scale Corpus Development, Artificial Intelligence, Natural Language Processing, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="2001.09768" label="2001.09768">
        <attvalues>
          <attvalue for="0" value="Artificial Intelligence, Values and Alignment" />
          <attvalue for="1" value="  This paper looks at philosophical questions that arise in the context of AI&#10;alignment. It defends three propositions. First, normative and technical&#10;aspects of the AI alignment problem are interrelated, creating space for&#10;productive engagement between people working in both domains. Second, it is&#10;important to be clear about the goal of alignment. There are significant&#10;differences between AI that aligns with instructions, intentions, revealed&#10;preferences, ideal preferences, interests and values. A principle-based&#10;approach to AI alignment, which combines these elements in a systematic way,&#10;has considerable advantages in this context. Third, the central challenge for&#10;theorists is not to identify 'true' moral principles for AI; rather, it is to&#10;identify fair principles for alignment, that receive reflective endorsement&#10;despite widespread variation in people's moral beliefs. The final part of the&#10;paper explores three ways in which fair principles for AI alignment could&#10;potentially be identified.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.12966" label="2307.12966">
        <attvalues>
          <attvalue for="0" value="Aligning Large Language Models with Human: A Survey" />
          <attvalue for="1" value="  Large Language Models (LLMs) trained on extensive textual corpora have&#10;emerged as leading solutions for a broad array of Natural Language Processing&#10;(NLP) tasks. Despite their notable performance, these models are prone to&#10;certain limitations such as misunderstanding human instructions, generating&#10;potentially biased content, or factually incorrect (hallucinated) information.&#10;Hence, aligning LLMs with human expectations has become an active area of&#10;interest within the research community. This survey presents a comprehensive&#10;overview of these alignment technologies, including the following aspects. (1)&#10;Data collection: the methods for effectively collecting high-quality&#10;instructions for LLM alignment, including the use of NLP benchmarks, human&#10;annotations, and leveraging strong LLMs. (2) Training methodologies: a detailed&#10;review of the prevailing training methods employed for LLM alignment. Our&#10;exploration encompasses Supervised Fine-tuning, both Online and Offline human&#10;preference training, along with parameter-efficient training mechanisms. (3)&#10;Model Evaluation: the methods for evaluating the effectiveness of these&#10;human-aligned LLMs, presenting a multifaceted approach towards their&#10;assessment. In conclusion, we collate and distill our findings, shedding light&#10;on several promising future research avenues in the field. This survey,&#10;therefore, serves as a valuable resource for anyone invested in understanding&#10;and advancing the alignment of LLMs to better suit human-oriented tasks and&#10;expectations. An associated GitHub link collecting the latest papers is&#10;available at https://github.com/GaryYufei/AlignLLMHumanSurvey.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05862" label="2204.05862">
        <attvalues>
          <attvalue for="0" value="Training a Helpful and Harmless Assistant with Reinforcement Learning&#10;  from Human Feedback" />
          <attvalue for="1" value="  We apply preference modeling and reinforcement learning from human feedback&#10;(RLHF) to finetune language models to act as helpful and harmless assistants.&#10;We find this alignment training improves performance on almost all NLP&#10;evaluations, and is fully compatible with training for specialized skills such&#10;as python coding and summarization. We explore an iterated online mode of&#10;training, where preference models and RL policies are updated on a weekly&#10;cadence with fresh human feedback data, efficiently improving our datasets and&#10;models. Finally, we investigate the robustness of RLHF training, and identify a&#10;roughly linear relation between the RL reward and the square root of the KL&#10;divergence between the policy and its initialization. Alongside our main&#10;results, we perform peripheral analyses on calibration, competing objectives,&#10;and the use of OOD detection, compare our models with human writers, and&#10;provide samples from our models using prompts appearing in recent related work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.11147" label="2203.11147">
        <attvalues>
          <attvalue for="0" value="Teaching language models to support answers with verified quotes" />
          <attvalue for="1" value="  Recent large language models often answer factual questions correctly. But&#10;users can't trust any given claim a model makes without fact-checking, because&#10;language models can hallucinate convincing nonsense. In this work we use&#10;reinforcement learning from human preferences (RLHP) to train &quot;open-book&quot; QA&#10;models that generate answers whilst also citing specific evidence for their&#10;claims, which aids in the appraisal of correctness. Supporting evidence is&#10;drawn from multiple documents found via a search engine, or from a single&#10;user-provided document. Our 280 billion parameter model, GopherCite, is able to&#10;produce answers with high quality supporting evidence and abstain from&#10;answering when unsure. We measure the performance of GopherCite by conducting&#10;human evaluation of answers to questions in a subset of the NaturalQuestions&#10;and ELI5 datasets. The model's response is found to be high-quality 80\% of the&#10;time on this Natural Questions subset, and 67\% of the time on the ELI5 subset.&#10;Abstaining from the third of questions for which it is most unsure improves&#10;performance to 90\% and 80\% respectively, approaching human baselines.&#10;However, analysis on the adversarial TruthfulQA dataset shows why citation is&#10;only one part of an overall strategy for safety and trustworthiness: not all&#10;claims supported by evidence are true.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.14529" label="2207.14529">
        <attvalues>
          <attvalue for="0" value="The Effects of Data Quality on Machine Learning Performance" />
          <attvalue for="1" value="  Modern artificial intelligence (AI) applications require large quantities of&#10;training and test data. This need creates critical challenges not only&#10;concerning the availability of such data, but also regarding its quality. For&#10;example, incomplete, erroneous or inappropriate training data can lead to&#10;unreliable models that produce ultimately poor decisions. Trustworthy AI&#10;applications require high-quality training and test data along many dimensions,&#10;such as accuracy, completeness, consistency, and uniformity.&#10;  We explore empirically the relationship between six of the traditional data&#10;quality dimensions and the performance of fifteen widely used machine learning&#10;(ML) algorithms covering the tasks of classification, regression, and&#10;clustering, with the goal of explaining their performance in terms of data&#10;quality. Our experiments distinguish three scenarios based on the AI pipeline&#10;steps that were fed with polluted data: polluted training data, test data, or&#10;both. We conclude the paper with an extensive discussion of our observations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.04650" label="2010.04650">
        <attvalues>
          <attvalue for="0" value="LSTMs Compose (and Learn) Bottom-Up" />
          <attvalue for="1" value="  Recent work in NLP shows that LSTM language models capture hierarchical&#10;structure in language data. In contrast to existing work, we consider the&#10;\textit{learning} process that leads to their compositional behavior. For a&#10;closer look at how an LSTM's sequential representations are composed&#10;hierarchically, we present a related measure of Decompositional Interdependence&#10;(DI) between word meanings in an LSTM, based on their gate interactions. We&#10;connect this measure to syntax with experiments on English language data, where&#10;DI is higher on pairs of words with lower syntactic distance. To explore the&#10;inductive biases that cause these compositional representations to arise during&#10;training, we conduct simple experiments on synthetic data. These synthetic&#10;experiments support a specific hypothesis about how hierarchical structures are&#10;discovered over the course of training: that LSTM constituent representations&#10;are learned bottom-up, relying on effective representations of their shorter&#10;children, rather than learning the longer-range relations independently from&#10;children.&#10;" />
          <attvalue for="2" value="&#10;&#10;For years the LSTM dominated language architectures. It remains a popular architecture in NLP, and unlike Transformer-based models, it can be trained on small corpora~\cite{tran_importance_2018}. \cite{abnar_transferring_2020} even found that the recurrent inductive biases behind the LSTM's success are so essential that distilling from them can improve the performance of fully attentional models. However, the reasons behind the LSTM's effectiveness in language domains remain poorly understood.&#10;&#10;A Transformer can encode syntax using attention \cite{hewitt_structural_nodate}, and some LSTM variants explicitly encode syntax \cite{bowman-etal-2016-fast,DBLP:journals/corr/DyerKBS16}. So, the success of these models is partly explained by their ability to model syntactic relationships when predicting a word. By contrast, an LSTM simply scans a sentence from left to right, accumulating meaning into a hidden representation one word at a time, and using that representation to summarize the entire preceding sequence when predicting the next word. Yet we have extensive evidence that trained LSTMs are also sensitive to syntax. For example, they can recall more history in natural language data than in similarly Zipfian-distributed $n$-gram data, implying that they exploit linguistic structure in long-distance dependencies \cite{liu_lstms_2018}. Their internal representations appear to encode constituency \cite{blevins_deep_2018,hupkes_visualisation_2018} and syntactic agreement \cite{lakretz-etal-2019-emergence,gulordava-etal-2018-colorless}. In this paper, we consider how such representations are learned, and what kind of inductive bias supports them.&#10;&#10;To understand how LSTMs exploit syntax, we use contextual decomposition (CD; Section~\ref{sec:cd}), a method that computes how much the hidden representation of an LSTM depends on particular past span of words. We then extend CD to Decompositional Interdependence (DI; Section~\ref{sec:interdependence}), a measure of interaction between spans of words to produce the representation at a particular timestep. For example, in the sentence ``Socrates asked the student trick questions’’, we might expect the hidden representation of the LSTM at the word ``questions’’ to interact primarily with its syntactic head ``asked’’, and less with the direct object ``the student''. If so, then an LSTM could be seen as implementing compositional localism \cite{hupkes_compositionality_2020}: if a hidden representation encodes meaning, then this meaning is composed from local syntactic relationships. Our experiments on syntactically-parsed corpora (Section~\ref{sec:english}) illustrate this property --- interdependence decreases with syntactic distance, stratified by surface distance.&#10;&#10;We then turn to a hypothesis about how such representations are learned. Using a simple synthetic corpus (Section~\ref{sec:long}), we allow LSTMs to learn to represent short sequences before they learn longer sequences that are dependent on them. Our goal is to then illustrate how they use representations of short sequences in order to learn longer dependencies---if these smaller constituents are unfamiliar, LSTMs learn more slowly. Further experiments (Section~\ref{sec:isolating}) isolate hierarchical behavior from other factors causing local relations to be learned first, indicating that the model tends to build a subtree from its smaller constituents. We conclude that LSTMs compose hierachically because they learn bottom-up.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Hierarchical Structure Learning, Language Model Analysis, Linguistics, Cognitive Science, Neural Network Behavior, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1803.03585" label="1803.03585">
        <attvalues>
          <attvalue for="0" value="The Importance of Being Recurrent for Modeling Hierarchical Structure" />
          <attvalue for="1" value="  Recent work has shown that recurrent neural networks (RNNs) can implicitly&#10;capture and exploit hierarchical information when trained to solve common&#10;natural language processing tasks such as language modeling (Linzen et al.,&#10;2016) and neural machine translation (Shi et al., 2016). In contrast, the&#10;ability to model structured data with non-recurrent neural networks has&#10;received little attention despite their success in many NLP tasks (Gehring et&#10;al., 2017; Vaswani et al., 2017). In this work, we compare the two&#10;architectures---recurrent versus non-recurrent---with respect to their ability&#10;to model hierarchical structure and find that recurrency is indeed important&#10;for this purpose.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.00555" label="2006.00555">
        <attvalues>
          <attvalue for="0" value="Transferring Inductive Biases through Knowledge Distillation" />
          <attvalue for="1" value="  Having the right inductive biases can be crucial in many tasks or scenarios&#10;where data or computing resources are a limiting factor, or where training data&#10;is not perfectly representative of the conditions at test time. However,&#10;defining, designing and efficiently adapting inductive biases is not&#10;necessarily straightforward. In this paper, we explore the power of knowledge&#10;distillation for transferring the effect of inductive biases from one model to&#10;another. We consider families of models with different inductive biases, LSTMs&#10;vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where&#10;having the right inductive biases is critical. We study the effect of inductive&#10;biases on the solutions the models converge to and investigate how and to what&#10;extent the effect of inductive biases is transferred through knowledge&#10;distillation, in terms of not only performance but also different aspects of&#10;converged solutions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.11138" label="1803.11138">
        <attvalues>
          <attvalue for="0" value="Colorless green recurrent networks dream hierarchically" />
          <attvalue for="1" value="  Recurrent neural networks (RNNs) have achieved impressive results in a&#10;variety of linguistic processing tasks, suggesting that they can induce&#10;non-trivial properties of language. We investigate here to what extent RNNs&#10;learn to track abstract hierarchical syntactic structure. We test whether RNNs&#10;trained with a generic language modeling objective in four languages (Italian,&#10;English, Hebrew, Russian) can predict long-distance number agreement in various&#10;constructions. We include in our evaluation nonsensical sentences where RNNs&#10;cannot rely on semantic or lexical cues (&quot;The colorless green ideas I ate with&#10;the chair sleep furiously&quot;), and, for Italian, we compare model performance to&#10;human intuitions. Our language-model-trained RNNs make reliable predictions&#10;about long-distance agreement, and do not lag much behind human performance. We&#10;thus bring support to the hypothesis that RNNs are not just shallow-pattern&#10;extractors, but they also acquire deeper grammatical competence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.16393" label="2310.16393">
        <attvalues>
          <attvalue for="0" value="ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source&#10;  Ensembling of Language Adapters" />
          <attvalue for="1" value="  We tackle the problem of zero-shot cross-lingual transfer in NLP tasks via&#10;the use of language adapters (LAs). Most of the earlier works have explored&#10;training with adapter of a single source (often English), and testing either&#10;using the target LA or LA of another related language. Training target LA&#10;requires unlabeled data, which may not be readily available for low resource&#10;unseen languages: those that are neither seen by the underlying multilingual&#10;language model (e.g., mBERT), nor do we have any (labeled or unlabeled) data&#10;for them. We posit that for more effective cross-lingual transfer, instead of&#10;just one source LA, we need to leverage LAs of multiple (linguistically or&#10;geographically related) source languages, both at train and test-time - which&#10;we investigate via our novel neural architecture, ZGUL. Extensive&#10;experimentation across four language groups, covering 15 unseen target&#10;languages, demonstrates improvements of up to 3.2 average F1 points over&#10;standard fine-tuning and other strong baselines on POS tagging and NER tasks.&#10;We also extend ZGUL to settings where either (1) some unlabeled data or (2)&#10;few-shot training examples are available for the target language. We find that&#10;ZGUL continues to outperform baselines in these settings too.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;{Single-source Adapter Tuning:}&#10;We build on MAD-X \cite{pfeiffer-etal-2020-mad}, which introduces two phases of adapter training. 1. Pretraining Language adapter (LA) for each language ${L}_{i}$: inserting an LA in each layer of transformer model $\mathcal{M}$ (denoted by $\mathcal{L}_{i} \circ \mathcal{M}$) and training on unlabeled data for language $\mathcal{L}_{i}$ using the MLM objective. 2. Training TA for a task $T_{j}$: stacking LA for source language $L_{src}$ with TA for task $T_{j}$ (denoted by $\mathcal{T}_{j} \circ \mathcal{L}_{src} \circ \mathcal{M}$), in which $\mathcal{T}_{j}$ and the task-specific prediction head are the only trainable parameters. During inference, $\mathcal{L}_{src}$ is replaced with $\mathcal{L}_{tgt}$, i.e. $\mathcal{T}_{j} \circ \mathcal{L}_{tgt} \circ \mathcal{M}$ is used. The MAD-X paradigm uses only one LA for a given input sentence. Also, it assumes the availability of $\mathcal{L}_{tgt}$. If not available, English adapter \cite{he2021effectiveness,pfeiffer-etal-2020-mad} or a related language's adapter \cite{wang2021efficient} is used at test-time. &#10;&#10;{Adapter Combination:} \cite{pfeiffer2021adapterfusion} introduce AdapterFusion, a technique that combines multiple pretrained TAs $\mathcal{T}_{1},...\mathcal{T}_{n}$ to solve a new target task $T_{n+1}$. It learns the attention weights of $\mathcal{T}_{1},...\mathcal{T}_{n}$ while being fine-tuned on the data for $T_{n+1}$. \cite{vu-etal-2022-domain} adapt this technique for fusing domains and testing on out-of-domain data. This technique has not been applied in the context of LAs so far. The recent release of 50 LAs on AdapterHub enables studying this for LAs.&#10;&#10;Recently, \cite{wang2021efficient} propose EMEA (Entropy Minimized Ensembling of Adapters) for efficiently combining multiple LAs at inference time. EMEA calculates the entropy of the prediction during test time and adjusts the LA attention scores (initialized uniformly) using Gradient Descent, aiming to give higher importance to the LA that increases the confidence score of the prediction. However, the training is still conducted using English as a single source.&#10;&#10;{Generation of LA using Shared Parameters:}&#10;\cite{ustun-etal-2020-udapter} employ the Conditional Parameter Generation (CPG) technique \cite{platanios-etal-2018-contextual} for training on multiple source languages. They utilize a CPG module, referred to as CPGAdapter, which takes a typological language vector as input and generates a Language Adapter (LA). The CPGAdapter is shared across all source languages and trained from scratch for a specific task. Since an LA is determined by the input language's vector, this approach can directly generalize to unseen languages. However, it is worth noting that CPG is data-intensive as it learns the parameters of the CPGAdapter from scratch. &#10;&#10;We note that CPG comes under a broader category of hypernetworks that generate weights for a larger main network \cite{ha2016hypernetworks}, which have been recently explored successfully for task mixing \cite{karimi-mahabadi-etal-2021-parameter}. In our experiments, we include a comparison with the CPG method. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual NLP Tasks, Cross-Lingual Transfer, Artificial Intelligence, Neural Language Models, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2108.11857" label="2108.11857">
        <attvalues>
          <attvalue for="0" value="Probing Pre-trained Auto-regressive Language Models for Named Entity&#10;  Typing and Recognition" />
          <attvalue for="1" value="  Despite impressive results of language models for named entity recognition&#10;(NER), their generalization to varied textual genres, a growing entity type&#10;set, and new entities remains a challenge. Collecting thousands of annotations&#10;in each new case for training or fine-tuning is expensive and time-consuming.&#10;In contrast, humans can easily identify named entities given some simple&#10;instructions. Inspired by this, we challenge the reliance on large datasets and&#10;study pre-trained language models for NER in a meta-learning setup. First, we&#10;test named entity typing (NET) in a zero-shot transfer scenario. Then, we&#10;perform NER by giving few examples at inference. We propose a method to select&#10;seen and rare / unseen names when having access only to the pre-trained model&#10;and report results on these groups. The results show: auto-regressive language&#10;models as meta-learners can perform NET and NER fairly well especially for&#10;regular or seen names; name irregularity when often present for a certain&#10;entity type can become an effective exploitable cue; names with words foreign&#10;to the model have the most negative impact on results; the model seems to rely&#10;more on name than context cues in few-shot NER.&#10;" />
          <attvalue for="2" value="&#10;Before transformer LMs \cite{devlin-etal-2019-bert}, the state-of-the-art NER was based on training recurrent neural networks, such as bidirectional LSTM (BiLSTM) with a Conditional Random Field (CRF) layer, from scratch \cite{yadav-bethard-2018-survey}.&#10;The widely adopted approach with transformers has been to fine-tune them for the desired task, thus specializing their general linguistic knowledge, acquired during pre-training.&#10;While LMs fine-tuned for NER have achieved impressive results on standard benchmarks \cite{akbik-etal-2019-pooled}, multiple works have emphasized their limitations with regard to their generalization capacity to new textual genres (e.g. clean versus noisy text), NE type sets (e.g. NE types belonging to new domains such as music or e-commerce) and new NEs, unseen during training \cite{lin-etal-2020-rigorous}.&#10;&#10;To gain more insights into the NER generalization ability of LMs, multiple studies have been conducted. &#10;Probing has been designed for BiLSTM-CRF LMs \cite{Augenstein2017Generalisation,Taille2020contextualized,fu2020rethinking} or masked LMs such as BERT \cite{petroni-etal-2019-language,jiang-etal-2020-know}.&#10;Yet, little has been done for auto-regressive models such as GPT2 despite their popularity and potential to express a wide variety of NLP tasks in the same unified format \cite{Raffel2020Exploring}.&#10;&#10;Additionally, although the past probing studies have broadened the knowledge about how LMs generalize in the NER context, multiple improvements could be brought to existing methodologies.&#10;First, the impact of pre-training LMs on the results has never been assessed.&#10;Second, the proposed setups test generalization by relying on large annotated datasets which are manipulated in different ways to create test and train splits.&#10;However, when assessing generalization in relation to human linguistic behavior \cite{levesque2014our}, which we claim as more realistic, these datasets are insufficient and different testing conditions should exist. &#10;Humans can easily recognize NEs based on prior domain and common sense linguistic knowledge, or by leveraging contextual cues in text \cite{lin-etal-2020-triggerner}.&#10;Humans can perform new linguistic tasks quite well even when exposed to a few examples or very simple instructions \cite{Brown2020Language}.&#10;When it comes to technology creation, human linguistic behavior could lead to infinite examples, many of them new to everyone, including to the systems' designers \cite{emnlp-2020-2020}.&#10;&#10;Hence, datasets used in past studies cannot capture this variability for a realistic testing unless continuously updated.&#10;However, like humans, LMs have gained diverse domain and linguistic knowledge, and developed general pattern recognition abilities from experience, during pre-training \cite{Brown2020Language}.&#10;Given these, the research question we investigate is: &#10;&#10;Can the knowledge gained during pre-training be leveraged by auto-regressive LMs at inference to adapt to diverse NE-related tasks, when queried with a few examples at most and simple natural language instructions?&#10;&#10;Contributions. &#10;Inspired by testing conditions related to human linguistic behavior, we design a probing methodology centered on meta- or &quot;in-context&quot; learning.&#10;It entails the task specification via the text input used to prompt the model, without performing any gradient updates \cite{Brown2020Language}.&#10;First we study NEs of various types individually by defining a zero-shot transfer strategy for NET.&#10;We design a novel method to assess NE memorization by the model and report the memorization's impact on the results.&#10;Our memorization method could be used to sample (un)popular NEs \cite{shwartz-etal-2020-grounded}, but also, beyond the NE context, with other types of n-grams.&#10;Second, we model NER as a machine reading comprehension (MRC) task and probe the model by providing a few examples at inference (e.g., the model should extract spans of text from input, as answers to simple queries).&#10;We also test NER with (un)memorized NEs and gain insights on the role of context, i.e. text around NEs.&#10;We use four datasets: CoNLL-2003 \cite{tjong-kim-sang-de-meulder-2003-introduction}, WNUT2017 \cite{liu2014a}, MIT Movie \cite{liu2014a} and extensive lists of NEs from DBpedia \cite{Auer2007Dbpedia}.&#10;These datasets contain clean and noisy text, and regular NEs such as people names and irregular NEs such as creative work titles.&#10;&#10;Our study joins other efforts that looked into NET and NER generalization but that, compared to us, achieved this by manipulating datasets during fine-tuning / testing or targeted other types of LMs. &#10;To our knowledge, we are the first to extensively probe pre-trained auto-regressive LMs as they are for these tasks and ensure testing conditions related to human linguistic behavior.&#10;&#10;Findings. &#10;Pre-trained GPT2, a common auto-regressive LM, appears to perform the tasks fairly well without any fine-tuning for NET or NER, especially on regular NEs or memorized during pre-training.&#10;These models, as they are, already know quite a lot about NEs and encode NER patterns.&#10;Our finding is particularly important given that past works study NET and NER generalization of existing LMs without explicitly considering the impact of model pre-training.&#10;Then, compared to other studies that claim named entity irregularity to be problematic \cite{Augenstein2017Generalisation}, we show that when frequently present for a certain NE type it can become, in fact, an effective exploitable cue. &#10;We also show that the model seems to rely more on NE cues than on context cues in few-shot NER, and that the model’s exposure to the NE's words weighs much more than the exposure to the exact NE in zero-shot NET.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Cognitive Science, Meta-Learning Approaches, Language Model Limitations, Named Entity Recognition, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1910.11470" label="1910.11470">
        <attvalues>
          <attvalue for="0" value="A Survey on Recent Advances in Named Entity Recognition from Deep&#10;  Learning models" />
          <attvalue for="1" value="  Named Entity Recognition (NER) is a key component in NLP systems for question&#10;answering, information retrieval, relation extraction, etc. NER systems have&#10;been studied and developed widely for decades, but accurate systems using deep&#10;neural networks (NN) have only been introduced in the last few years. We&#10;present a comprehensive survey of deep neural network architectures for NER,&#10;and contrast them with previous approaches to NER based on feature engineering&#10;and other supervised or semi-supervised learning algorithms. Our results&#10;highlight the improvements achieved by neural networks, and show how&#10;incorporating some of the lessons learned from past work on feature-based NER&#10;systems can yield further improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.12126" label="2004.12126">
        <attvalues>
          <attvalue for="0" value="A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained&#10;  Model Lead to the Promised Land?" />
          <attvalue for="1" value="  Fine-tuning pretrained model has achieved promising performance on standard&#10;NER benchmarks. Generally, these benchmarks are blessed with strong name&#10;regularity, high mention coverage and sufficient context diversity.&#10;Unfortunately, when scaling NER to open situations, these advantages may no&#10;longer exist. And therefore it raises a critical question of whether previous&#10;creditable approaches can still work well when facing these challenges. As&#10;there is no currently available dataset to investigate this problem, this paper&#10;proposes to conduct randomization test on standard benchmarks. Specifically, we&#10;erase name regularity, mention coverage and context diversity respectively from&#10;the benchmarks, in order to explore their impact on the generalization ability&#10;of models. To further verify our conclusions, we also construct a new open NER&#10;dataset that focuses on entity types with weaker name regularity and lower&#10;mention coverage to verify our conclusion. From both randomization test and&#10;empirical experiments, we draw the conclusions that 1) name regularity is&#10;critical for the models to generalize to unseen mentions; 2) high mention&#10;coverage may undermine the model generalization ability and 3) context patterns&#10;may not require enormous data to capture when using pretrained encoders.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Named entity recognition has long been studied and has attracted much attention. Conventional methods~\cite{zhou2002named,chieu2002named,bender2003maximum,settles2004biomedical} commonly rely on handcraft features to build NER models, which are hard to transfer among different languages, domains and entity types. Recently, deep learning methods, which automatically extract high-level features and perform sequence tagging with neural networks~\cite{santos2015boosting,chiu2016named,lample2016neural,yadav2019survey}, have achieved significant progress especially under strong pretraining and fine-tuning paradigm~\cite{li2019entity,akbik2019pooled,zhai2019improving,li2019unified,xia2019multi,lin2019sequence}. These methods have achieved promising results in almost all popular NER benchmarks considering regular entity types.&#10;&#10;Several researches have shift attention to name tagging in open scenarios, where entity types may have weaker name regularity and training data are often insufficient. These papers mainly focus on how to exploit weakly-supervised data~\cite{tackstrom2013token,ni2017weakly,cao2019low,Xue2019NeuralCE}, or devoted to incorporate external resources~\cite{yang2016transfer,peng2016improving,pan2017cross,lin2018multi,xie2018neural,lin2019gazetteer}.&#10;&#10;By contrast, to the best of our knowledge, this is the first work which investigates the essential difference between regular and open NER. By conducting both randomization test~\cite{edgington2007randomization} and verification experiments, we analyze the impact of name regularity, mention coverage and context pattern sufficiency and shed light on future open NER studies. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2001.03844" label="2001.03844">
        <attvalues>
          <attvalue for="0" value="Rethinking Generalization of Neural Models: A Named Entity Recognition&#10;  Case Study" />
          <attvalue for="1" value="  While neural network-based models have achieved impressive performance on a&#10;large body of NLP tasks, the generalization behavior of different models&#10;remains poorly understood: Does this excellent performance imply a perfect&#10;generalization model, or are there still some limitations? In this paper, we&#10;take the NER task as a testbed to analyze the generalization behavior of&#10;existing models from different perspectives and characterize the differences of&#10;their generalization abilities through the lens of our proposed measures, which&#10;guides us to better design models and training methods. Experiments with&#10;in-depth analyses diagnose the bottleneck of existing neural NER models in&#10;terms of breakdown performance analysis, annotation errors, dataset bias, and&#10;category relationships, which suggest directions for improvement. We have&#10;released the datasets: (ReCoNLL, PLONER) for the future research at our project&#10;page: http://pfliu.com/InterpretNER/. As a by-product of this paper, we have&#10;open-sourced a project that involves a comprehensive summary of recent NER&#10;papers and classifies them into different research topics:&#10;https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.07493" label="2004.07493">
        <attvalues>
          <attvalue for="0" value="TriggerNER: Learning with Entity Triggers as Explanations for Named&#10;  Entity Recognition" />
          <attvalue for="1" value="  Training neural models for named entity recognition (NER) in a new domain&#10;often requires additional human annotations (e.g., tens of thousands of labeled&#10;instances) that are usually expensive and time-consuming to collect. Thus, a&#10;crucial research question is how to obtain supervision in a cost-effective way.&#10;In this paper, we introduce &quot;entity triggers,&quot; an effective proxy of human&#10;explanations for facilitating label-efficient learning of NER models. An entity&#10;trigger is defined as a group of words in a sentence that helps to explain why&#10;humans would recognize an entity in the sentence.&#10;  We crowd-sourced 14k entity triggers for two well-studied NER datasets. Our&#10;proposed model, Trigger Matching Network, jointly learns trigger&#10;representations and soft matching module with self-attention such that can&#10;generalize to unseen sentences easily for tagging. Our framework is&#10;significantly more cost-effective than the traditional neural NER frameworks.&#10;Experiments show that using only 20% of the trigger-annotated sentences results&#10;in a comparable performance as using 70% of conventional annotated sentences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Towards low-resource learning for NER,&#10;recent works have mainly focused on dictionary-based distantly supervision~\cite{autoner, yangner, Liu2019TowardsIN}.&#10;These approaches create an external large dictionary of entities, and then regard hard-matched sentences as additional, noisy-labeled data for learning a NER model.&#10;Although these approaches largely reduce human efforts in annotating, the quality of matched sentences is highly dependent on the coverage of the dictionary and the quality of the corpus.&#10;The learned models tend to have a bias towards entities with similar surface forms as the ones in dictionary. Without further tuning under better supervision, these models have low recall~\cite{Cao2019LowResourceNT}.&#10;Linking rules~\cite{safranchik:aaai20} focuses on the &#10;votes on whether adjacent elements in the sequence belong&#10;to the same class. &#10;Unlike these works aiming to get rid of training data or human annotations, our work focuses on how to more cost-effectively utilize human efforts.&#10;&#10;Another line of research which also aims to use human efforts more cost-effectively is active learning~\cite{shen2018deep,Lin2019AlpacaTagAA}.&#10;This approach focuses on instance sampling and the human annotation UI, asking workers to annotate the most useful instances first.&#10;However, a recent study~\cite{Lipton2018PracticalOT} argues that actively annotated data barely helps when training new models. &#10;Transfer learning approaches ~\cite{Lin2018NeuralAL} and aggregating multi-source supervision~\cite{Lan2020} are also studied for using less expensive supervision for NER, while these methods usually lack clear rationales to advise annotation process unlike the trigger annotations. &#10;&#10;Inspired by recent advances in learning sentence classification tasks (e.g., relation extraction and sentiment classification) with explanations or human-written rules~\cite{Li2018GeneralizeSK, Hancock2018TrainingCW, Wang2020Learning, Zhou2019NEROAN}, we propose the concept of an ``entity trigger'' for the task of named entity recognition.&#10;These prior works primarily focused on sentence classification, in which the rules (parsed from natural language explanations) are usually continuous token sequences and there is a single label for each input sentence.&#10;The unique challenge in NER is that we have to deal with rules which are discontinuous token sequences and there may be multiple rules applied at the same time for an input instance.&#10;We address this problem in TMN by jointly learning trigger representations and creating a soft matching module that works in the inference time. &#10;&#10;We argue that either dictionary-based distant supervision or active learning can be used in the context of trigger-enhanced NER learning via our framework.&#10;For example, one could create a dictionary using a high-quality corpus and then apply active learning by asking human annotators to annotate the triggers chosen by an active sampling algorithm designed for TMN.&#10;We believe our work sheds light on future research for more cost-effectively using human to learn NER models.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="cs/0306050" label="cs/0306050">
        <attvalues>
          <attvalue for="0" value="Introduction to the CoNLL-2003 Shared Task: Language-Independent Named&#10;  Entity Recognition" />
          <attvalue for="1" value="  We describe the CoNLL-2003 shared task: language-independent named entity&#10;recognition. We give background information on the data sets (English and&#10;German) and the evaluation method, present a general overview of the systems&#10;that have taken part in the task and discuss their performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.00964" label="2403.00964">
        <attvalues>
          <attvalue for="0" value="MALTO at SemEval-2024 Task 6: Leveraging Synthetic Data for LLM&#10;  Hallucination Detection" />
          <attvalue for="1" value="  In Natural Language Generation (NLG), contemporary Large Language Models&#10;(LLMs) face several challenges, such as generating fluent yet inaccurate&#10;outputs and reliance on fluency-centric metrics. This often leads to neural&#10;networks exhibiting &quot;hallucinations&quot;. The SHROOM challenge focuses on&#10;automatically identifying these hallucinations in the generated text. To tackle&#10;these issues, we introduce two key components, a data augmentation pipeline&#10;incorporating LLM-assisted pseudo-labelling and sentence rephrasing, and a&#10;voting ensemble from three models pre-trained on Natural Language Inference&#10;(NLI) tasks and fine-tuned on diverse datasets.&#10;" />
          <attvalue for="2" value="&#10;Natural Language Generation (NLG) models are AI systems that use neural networks to produce human-like text.&#10;They have shown significant advancements in recent years, particularly with the advent of transformer-based architectures such as GPT (Generative Pre-trained Transformer) \cite{radford2018improving}.&#10;These models offered unprecedented levels of fluency and coherence in generated text \cite{HAN2021225}.&#10;However, a critical challenge arises: these models can produce linguistically fluent but semantically inaccurate outputs, a phenomenon referred to as hallucination~\cite{Ji_2023}. &#10;&#10;To address this challenge, the Shared-task on Hallucinations and Related Observable Overgeneration Mistakes (SHROOM) has been proposed at SemEval 2024.&#10;In particular, the Shared task aims to address the existing gap in assessing the semantic correctness and meaningfulness of NLG models.&#10;The ever-increasing adoption of such models makes it necessary to automatically detect and mitigate semantic hallucinations \cite{huang2023survey}.&#10;&#10;Some examples to tackle hallucination detection tasks in literature \cite{Ji_2023} are:&#10;(i) Information Extraction and Comparison between a generated text and a ground truth,&#10;(ii) Natural Language Inference Metrics that express the entailment between generated text and a ground truth or&#10;(iii) Faithfulness Classification Metrics that leverage upon knowledge-grounded datasets.&#10;&#10;In this work, we address the SHROOM shared task by introducing an automatic pipeline of hallucination detection through the comparison between a generated text and a ground truth text.&#10;We propose enriching the original data available using different augmentation techniques, including LLM-aided pseudo-labeling and sentence rephrasing.&#10;Additionally, we suggest using an ensemble of three different approaches, incorporating a simple BERT-based classifier, a model trained through Conditioned Reinforcement Learning Fine Tuning (C-RLFT)~\cite{wang2023openchat}, and a sequential model based on iterative fine-tuning.&#10;We show how this ensemble benefits from using different, complementary approaches, in particular in terms of recall. &#10;Our methodology obtained an accuracy of 80.07\% in the SemEval-Task 6 SHROOM. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Neural Network Evaluation, Artificial Intelligence, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2106.07139" label="2106.07139">
        <attvalues>
          <attvalue for="0" value="Pre-Trained Models: Past, Present and Future" />
          <attvalue for="1" value="  Large-scale pre-trained models (PTMs) such as BERT and GPT have recently&#10;achieved great success and become a milestone in the field of artificial&#10;intelligence (AI). Owing to sophisticated pre-training objectives and huge&#10;model parameters, large-scale PTMs can effectively capture knowledge from&#10;massive labeled and unlabeled data. By storing knowledge into huge parameters&#10;and fine-tuning on specific tasks, the rich knowledge implicitly encoded in&#10;huge parameters can benefit a variety of downstream tasks, which has been&#10;extensively demonstrated via experimental verification and empirical analysis.&#10;It is now the consensus of the AI community to adopt PTMs as backbone for&#10;downstream tasks rather than learning models from scratch. In this paper, we&#10;take a deep look into the history of pre-training, especially its special&#10;relation with transfer learning and self-supervised learning, to reveal the&#10;crucial position of PTMs in the AI development spectrum. Further, we&#10;comprehensively review the latest breakthroughs of PTMs. These breakthroughs&#10;are driven by the surge of computational power and the increasing availability&#10;of data, towards four important directions: designing effective architectures,&#10;utilizing rich contexts, improving computational efficiency, and conducting&#10;interpretation and theoretical analysis. Finally, we discuss a series of open&#10;problems and research directions of PTMs, and hope our view can inspire and&#10;advance the future study of PTMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.05232" label="2311.05232">
        <attvalues>
          <attvalue for="0" value="A Survey on Hallucination in Large Language Models: Principles,&#10;  Taxonomy, Challenges, and Open Questions" />
          <attvalue for="1" value="  The emergence of large language models (LLMs) has marked a significant&#10;breakthrough in natural language processing (NLP), leading to remarkable&#10;advancements in text understanding and generation. Nevertheless, alongside&#10;these strides, LLMs exhibit a critical tendency to produce hallucinations,&#10;resulting in content that is inconsistent with real-world facts or user inputs.&#10;This phenomenon poses substantial challenges to their practical deployment and&#10;raises concerns over the reliability of LLMs in real-world scenarios, which&#10;attracts increasing attention to detect and mitigate these hallucinations. In&#10;this survey, we aim to provide a thorough and in-depth overview of recent&#10;advances in the field of LLM hallucinations. We begin with an innovative&#10;taxonomy of LLM hallucinations, then delve into the factors contributing to&#10;hallucinations. Subsequently, we present a comprehensive overview of&#10;hallucination detection methods and benchmarks. Additionally, representative&#10;approaches designed to mitigate hallucinations are introduced accordingly.&#10;Finally, we analyze the challenges that highlight the current limitations and&#10;formulate open questions, aiming to delineate pathways for future research on&#10;hallucinations in LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.11235" label="2309.11235">
        <attvalues>
          <attvalue for="0" value="OpenChat: Advancing Open-source Language Models with Mixed-Quality Data" />
          <attvalue for="1" value="  Nowadays, open-source large language models like LLaMA have emerged. Recent&#10;developments have incorporated supervised fine-tuning (SFT) and reinforcement&#10;learning fine-tuning (RLFT) to align these models with human goals. However,&#10;SFT methods treat all training data with mixed quality equally, while RLFT&#10;methods require high-quality pairwise or ranking-based preference data. In this&#10;study, we present a novel framework, named OpenChat, to advance open-source&#10;language models with mixed-quality data. Specifically, we consider the general&#10;SFT training data, consisting of a small amount of expert data mixed with a&#10;large proportion of sub-optimal data, without any preference labels. We propose&#10;the C(onditioned)-RLFT, which regards different data sources as coarse-grained&#10;reward labels and learns a class-conditioned policy to leverage complementary&#10;data quality information. Interestingly, the optimal policy in C-RLFT can be&#10;easily solved through single-stage, RL-free supervised learning, which is&#10;lightweight and avoids costly human preference labeling. Through extensive&#10;experiments on three standard benchmarks, our openchat-13b fine-tuned with&#10;C-RLFT achieves the highest average performance among all 13b open-source&#10;language models. Moreover, we use AGIEval to validate the model generalization&#10;performance, in which only openchat-13b surpasses the base model. Finally, we&#10;conduct a series of analyses to shed light on the effectiveness and robustness&#10;of OpenChat. Our code, data, and models are publicly available at&#10;https://github.com/imoneoi/openchat and https://huggingface.co/openchat.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.10995" label="2203.10995">
        <attvalues>
          <attvalue for="0" value="Word Order Does Matter (And Shuffled Language Models Know It)" />
          <attvalue for="1" value="  Recent studies have shown that language models pretrained and/or fine-tuned&#10;on randomly permuted sentences exhibit competitive performance on GLUE, putting&#10;into question the importance of word order information. Somewhat&#10;counter-intuitively, some of these studies also report that position embeddings&#10;appear to be crucial for models' good performance with shuffled text. We probe&#10;these language models for word order information and investigate what position&#10;embeddings learned from shuffled text encode, showing that these models retain&#10;information pertaining to the original, naturalistic word order. We show this&#10;is in part due to a subtlety in how shuffling is implemented in previous work&#10;-- before rather than after subword segmentation. Surprisingly, we find even&#10;Language models trained on text shuffled after subword segmentation retain some&#10;semblance of information about word order because of the statistical&#10;dependencies between sentence length and unigram probabilities. Finally, we&#10;show that beyond GLUE, a variety of language understanding tasks do require&#10;word order information, often to an extent that cannot be learned through&#10;fine-tuning.&#10;" />
          <attvalue for="2" value="&#10;Transformers \cite{vaswani2017attention}, when used in the context of masked language modelling \cite{devlin2018bert}, consume their inputs concurrently. There is no notion of inherent order, unlike in autoregressive setups, where the input is consumed token by token. To compensate for this absence of linear order, the transformer architecture originally proposed in~\cite{vaswani2017attention} includes a fixed, sinusoidal position embedding added to each token embedding; each token carries a different position embedding, corresponding to its position in the sentence. The transformer-based BERT~\cite{devlin2018bert} replaces these fixed sinusoidal embeddings with unique, learned embeddings per position; RoBERTa~\cite{LiuRoBERTaRobustlyOptimized19a}, the model investigated in this work, does the same.&#10;&#10;Position embeddings are the only source of order information in these models; in their absence, contextual representations generated for tokens are independent of the actual position of the tokens in a sentence, and the models thus resemble heavily overparameterised bags-of-words. \cite{sinha2021masked} pre-trained RoBERTa models on shuffled corpora to demonstrate that the performance gap between these `shuffled' language models and models trained on unshuffled corpora is minor (when fine-tuned and evaluated downstream on the GLUE~\cite{WangGLUEMultiTaskBenchmark18} benchmark). They further show that this gap is considerably wider when a model is pre-trained without position embeddings. In this paper, we attempt to shed some light on why these models behave the way they do, and in doing so, seek to answer a set of pertinent questions: &#10;\begin{itemize}&#10; \item Do shuffled language models still have traces of word order information?&#10; \item Why is there a gap in performance between models {without} position embeddings and models trained on shuffled tokens, with the latter performing better? &#10; \item Are there NLU benchmarks, other than GLUE, on which shuffled language models perform poorly?&#10;\end{itemize}&#10;&#10;Contributions We first demonstrate, in Section~\ref{sec:order}, that shuffled language models {do} contain word order information, and are quite responsive to simple tests for word order information, particularly when compared to models trained without position representations. In Section~\ref{sec:mystery}, we demonstrate that pre-training is sufficient to learn this: position embeddings provide the appropriate inductive bias, and performing BPE segmentation after shuffling results in sensible n-grams appearing in the pre-training corpus; this gives models the capacity to learn word order within smaller local windows. Other minor cues - like correlations between sentence lengths and token distributions - also play a role. We further corroborate our analysis by examining attention patterns across models in Sec.~\ref{sec:attention}. In Section~\ref{sec:tasks}, we show that, while shuffled models might be almost as good as their un-shuffled counterparts on GLUE tasks, there exist NLU benchmarks that do require word order information to an extent that cannot be learned through fine-tuning alone. Finally, in Section~\ref{sec:other}, we describe miscellaneous experiments addressing the utility of positional embeddings when added just prior to fine-tuning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Natural Language Understanding, Artificial Intelligence, Natural Language Processing, Language Model Performance, Word Order Information" />
        </attvalues>
      </node>
      <node id="2212.09689" label="2212.09689">
        <attvalues>
          <attvalue for="0" value="Unnatural Instructions: Tuning Language Models with (Almost) No Human&#10;  Labor" />
          <attvalue for="1" value="  Instruction tuning enables pretrained language models to perform new tasks&#10;from inference-time natural language descriptions. These approaches rely on&#10;vast amounts of human supervision in the form of crowdsourced datasets or user&#10;interactions. In this work, we introduce Unnatural Instructions: a large&#10;dataset of creative and diverse instructions, collected with virtually no human&#10;labor. We collect 64,000 examples by prompting a language model with three seed&#10;examples of instructions and eliciting a fourth. This set is then expanded by&#10;prompting the model to rephrase each instruction, creating a total of&#10;approximately 240,000 examples of instructions, inputs, and outputs.&#10;Experiments show that despite containing a fair amount of noise, training on&#10;Unnatural Instructions rivals the effectiveness of training on open-source&#10;manually-curated datasets, surpassing the performance of models such as T0++&#10;and Tk-Instruct across various benchmarks. These results demonstrate the&#10;potential of model-generated data as a cost-effective alternative to&#10;crowdsourcing for dataset expansion and diversification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Instruction Tuning&#10;\cite{turkingtest} propose the Instruction Paradigm, where models learn new tasks from natural language instructions alone. \cite{mishra-etal-2022-cross, supernaturalinstructions} construct the first large-scale instruction benchmarks by collecting crowdsourcing instructions used to create NLP datasets and converting them into a uniform format.&#10;\cite{sanh2021multitask, wei2021finetuned} further extend the usability of instructions by suggesting instruction tuning, where a language model is trained on many natural language instructions in the hope that it will generalize to new, unseen instruction tasks. \cite{flampalm} advance instruction tuning by scaling the number of tasks, scaling the model size, and adding chain-of-thought \cite{wei2022chain}, while \cite{instructgpt} propose a reinforcement learning approach for instruction tuning from comparative human judgements.&#10;&#10;Automatic Data Generation&#10;Obtaining large-scale supervised data can be expensive and time-consuming.&#10;To mitigate this, several studies have explored automatic data generation.&#10;A common approach is to automatically augment existing datasets \cite[inter alia]{AnabyTavor2020DoNH, andreas-2020-good, yang-etal-2020-generative, Kaushik2020Learning, lee2021neural}. \cite{kiela-etal-2021-dynabench} suggest a human-and-model-in-the-loop dataset creation, where a model is trained on initial data, then annotators are asked to seek examples that are misclassified by the model, in an iterative process. In the same manner, \cite{nie-etal-2020-adversarial} apply a process to create training data for the task of NLI \cite{dagan-pascal-2003, bowman-etal-2015-large}, obtaining state-of-the-art performance on a variety of NLI benchmarks. &#10;\cite{liu-etal-2022-wanli} combine human annotators and GPT-3 to create challenging examples for NLI.&#10;&#10;While all the above techniques require an existing labeled dataset, other work suggested creating datasets entirely automatically, without the need for labeled data. \cite{schick-schutze-2021-generating} propose to leverage pretrained language models to generate entire datasets of labeled text pairs from scratch. \cite{agrawal2022qameleon} use pretrained language models to automatically construct multilingual QA data using only five examples per language.&#10;To the best of our knowledge, Unnatural Instructions is the first work to go beyond a particular task and automatically generate a large-scale general-purpose dataset, which emphasizes task diversity.&#10; " />
          <attvalue for="4" value="Artificial Intelligence Efficiency, Machine Learning, Computer Science, Dataset Generation Methods, Linguistics, Language Model Training, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1405.3515" label="1405.3515">
        <attvalues>
          <attvalue for="0" value="Temporal Analysis of Language through Neural Language Models" />
          <attvalue for="1" value="  We provide a method for automatically detecting change in language across&#10;time through a chronologically trained neural language model. We train the&#10;model on the Google Books Ngram corpus to obtain word vector representations&#10;specific to each year, and identify words that have changed significantly from&#10;1900 to 2009. The model identifies words such as &quot;cell&quot; and &quot;gay&quot; as having&#10;changed during that time period. The model simultaneously identifies the&#10;specific years during which such words underwent change.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Change Detection, Artificial Intelligence, Mathematics, Neural Language Models, Natural Language Processing, Temporal Word Analysis" />
        </attvalues>
      </node>
      <node id="2109.04332" label="2109.04332">
        <attvalues>
          <attvalue for="0" value="PPT: Pre-trained Prompt Tuning for Few-shot Learning" />
          <attvalue for="1" value="  Prompts for pre-trained language models (PLMs) have shown remarkable&#10;performance by bridging the gap between pre-training tasks and various&#10;downstream tasks. Among these methods, prompt tuning, which freezes PLMs and&#10;only tunes soft prompts, provides an efficient and effective solution for&#10;adapting large-scale PLMs to downstream tasks. However, prompt tuning is yet to&#10;be fully explored. In our pilot experiments, we find that prompt tuning&#10;performs comparably with conventional full-model fine-tuning when downstream&#10;data are sufficient, whereas it performs much worse under few-shot learning&#10;settings, which may hinder the application of prompt tuning in practice. We&#10;attribute this low performance to the manner of initializing soft prompts.&#10;Therefore, in this work, we propose to pre-train prompts by adding soft prompts&#10;into the pre-training stage to obtain a better initialization. We name this&#10;Pre-trained Prompt Tuning framework &quot;PPT&quot;. To ensure the generalization of PPT,&#10;we formulate similar classification tasks into a unified task form and&#10;pre-train soft prompts for this unified task. Extensive experiments show that&#10;tuning pre-trained prompts for downstream tasks can reach or even outperform&#10;full-model fine-tuning under both full-data and few-shot settings. Our approach&#10;is effective and efficient for using large-scale PLMs in practice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;PLMs and Task-oriented Fine-tuning Recently, various powerful PLMs have been proposed, such as GPT~\cite{gpt}, BERT~\cite{bert}, RoBERTa~\cite{roberta} and T5~\cite{t5}. To adapt these PLMs to downstream NLP tasks, task-oriented fine-tuning has been proposed, where researchers use PLMs as the backbone and add some task-specific heads to optimize task-specific objectives. Then, all parameters of both PLMs and additional heads are tuned using task-specific data. Results have shown that task-oriented fine-tuning can outperform models trained from scratch on a series of NLP tasks. &#10;&#10;Prompt-oriented Fine-tuning Most existing PLMs are pre-trained with language modeling objectives, yet the objectives of downstream tasks are quite different. To overcome the gap between pre-training and downstream tasks, prompt-oriented fine-tuning is introduced. In prompt-oriented fine-tuning, downstream tasks are also formalized as language modeling problems by inserting language prompts, and the results of language modeling can correspond to the solutions of downstream tasks. &#10;&#10;Knowledge probing~\cite{lama,trinh2018simple,davison2019commonsense} is the seminal work that stimulates the development of prompts. In knowledge probing, language triggers are widely used to induce PLMs to generate relational facts. These pioneering works demonstrate that language prompts can effectively stimulate the knowledge from PLMs. Encouraged by this, manually designing hard prompts consisting of discrete words is first used in prompt-oriented fine-tuning~\cite{pet, pet2}. Considering manually designing prompts is both time-consuming and difficult to find the best choice, later works~\cite{lm-bff,how-can-we-know, autoprompt} proposed to generate prompts automatically. However, these works still restrict auto-generated prompts to discrete spaces which are usually sub-optimal. &#10;&#10;To overcome the shortcomings of discrete spaces, \cite{prefixtuning,p-tuning,ptr,warp,optiprompt} explore to combine hard prompts and soft prompts. Different from hard prompts using concrete and discrete tokens, soft prompts are composed of several continuous learnable embeddings, and these embeddings are randomly initialized. To step forward, some works~\cite{prefixtuning,learning-how-to-ask,prompt_tuning} propose to only tune soft prompts and fix the entire PLM parameters. When models are large enough, this method can be comparable to full-model tuning.&#10;&#10;Few-shot Learning with PLMs &#10;Since long-tail distribution is common in real-world applications, few-shot learning is quite meaningful for the stable and effective use of PLMs, thereby attracts much attention recently. Apart from GPT-3~\cite{gpt3} and PET\cite{pet} which demonstrates the superiority of PLMs in few-shot scenarios, some later works~\cite{true-few-shot, flex} also discuss reasonable few-shot settings by restricting the size of validation set and proposing a unified framework to evaluate few-shot performance. There is also work~\cite{cutting-down-prompt} pointing out the low performance of PT for few-shot learning. But they mostly focus on PLMs with fewer than 400M parameters. In this paper, we study few-shot learning on large-scale 11B PLMs.&#10;&#10;" />
          <attvalue for="4" value="Language Model Tuning, Computer Science, Linguistics, Large-Scale Model Adaptation, Artificial Intelligence, Mathematics, Prompt Optimization" />
        </attvalues>
      </node>
      <node id="2210.05619" label="2210.05619">
        <attvalues>
          <attvalue for="0" value="Multilingual BERT has an accent: Evaluating English influences on&#10;  fluency in multilingual models" />
          <attvalue for="1" value="  While multilingual language models can improve NLP performance on&#10;low-resource languages by leveraging higher-resource languages, they also&#10;reduce average performance on all languages (the 'curse of multilinguality').&#10;Here we show another problem with multilingual models: grammatical structures&#10;in higher-resource languages bleed into lower-resource languages, a phenomenon&#10;we call grammatical structure bias. We show this bias via a novel method for&#10;comparing the fluency of multilingual models to the fluency of monolingual&#10;Spanish and Greek models: testing their preference for two carefully-chosen&#10;variable grammatical structures (optional pronoun-drop in Spanish and optional&#10;Subject-Verb ordering in Greek). We find that multilingual BERT is biased&#10;toward the English-like setting (explicit pronouns and Subject-Verb-Object&#10;ordering) as compared to our monolingual control language model. With our case&#10;studies, we hope to bring to light the fine-grained ways in which multilingual&#10;models can be biased,and encourage more linguistically-aware fluency&#10;evaluation.&#10;" />
          <attvalue for="2" value="&#10;Multilingual language models share a single set of parameters between many languages, opening new pathways for multilingual and low-resource NLP.&#10;However, not all training languages have an equal amount, or a comparable quality of training data in these models. In this paper, we investigate if the hegemonic status of English influences other languages in multilingual language models. We propose a novel method for evaluation, whereby we ask if model predictions for lower-resource languages exhibit structural features of English. This is similar to asking if the model has learned some languages with an ``English accent'', or an English grammatical structure bias.&#10;&#10;We demonstrate this bias effect in Spanish and Greek, comparing the monolingual models BETO \cite{beto} and GreekBERT \cite{greekbert} to multilingual BERT (mBERT), where English is the most frequent language in the training data. We show that mBERT prefers English-like sentence structure in Spanish and Greek compared to the monolingual models. Our case studies focus on Spanish pronoun drop (pro-drop) and Greek subject-verb order, two structural grammatical features. We show that multilingual BERT is structurally biased towards explicit pronouns rather than pro-drop in Spanish, and subject-before-verb order in Greek: the structural forms parallel to English.&#10;&#10;Though the effect we showcase here is likely not captured by the downstream classification tasks often used to evaluate multilingual models \cite{hu2020xtreme}, it demonstrates the type of fluency that can be lost with multilingual training&#10;--- something that current evaluation methods miss. &#10;In fact, though we choose two clear-cut syntactic features to investigate, there are many less-measurable features that make language production fluent: subtleties in lexical choice, grammatical choice, and discourse expression, among many others.&#10;With this paper, beyond showing a trend for two specific grammatical features, we wish to highlight fluency discrepancies in multilingual models, and also call for more evaluations focused on fluency. &#10;&#10;Our proposed method can be expanded, without the need for manual data collection, to any language with a syntactic treebank and a monolingual model. Since our method focuses on fine-grained linguistic features, some expert knowledge of the target language is necessary for evaluation. Multilingual evaluation so far has been largely translated or automatically curated, and the methods for creating such datasets have allowed for the creation of resources in many languages for which there there were none. Fluency evaluation requires some linguistic expertise to set up, and as such is more restricted in the languages the research community can reach. &#10;Nevertheless, such evaluation has been missing from the multilingual NLP literature, and our work bridges this gap by proposing fluency testing for multilingual models. &#10;&#10;Our work builds off of a long literature on multilingual evaluation which has until now mostly focused on downstream classification tasks \cite{conneau2018xnli, ebrahimi2022americasnli, clark2020tydiqa, liang2020xglue, hu2020xtreme, raganato2020xlwic, li2021mtop}. With the help of these evaluation methods, research has pointed out the problems for both high- and low-resource languages that come with adding many languages to a single model \cite[][inter alia]{wang2020negative, turc2021primacy, lauscher2020zero}. &#10;Methods for creating more equitable models have been proposed, through identifying or reserving language-specific parameters for each language \cite{ansell2022composable, pfeiffer2022lifting}, through training models without tyoplogically distant languages that dominate the training data \cite{afriberta, virtanen2019finnish, ogunremi2023mini}, as well as through adding model capacity \cite{conneau2019unsupervised, xue2021mt5, lepikhin2021gshard, liang2023xlmv}. We hope that our work can add to these analyses and methodologies by pointing out issues beyond downstream classification performance that can arise with multilingual training, and aid towards building and evaluating more equitable multilingual models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Language Model Evaluation, Artificial Intelligence, Grammatical Structure Bias, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2308.02976" label="2308.02976">
        <attvalues>
          <attvalue for="0" value="Spanish Pre-trained BERT Model and Evaluation Data" />
          <attvalue for="1" value="  The Spanish language is one of the top 5 spoken languages in the world.&#10;Nevertheless, finding resources to train or evaluate Spanish language models is&#10;not an easy task. In this paper we help bridge this gap by presenting a&#10;BERT-based language model pre-trained exclusively on Spanish data. As a second&#10;contribution, we also compiled several tasks specifically for the Spanish&#10;language in a single repository much in the spirit of the GLUE benchmark. By&#10;fine-tuning our pre-trained Spanish model, we obtain better results compared to&#10;other BERT-based models pre-trained on multilingual corpora for most of the&#10;tasks, even achieving a new state-of-the-art on some of them. We have publicly&#10;released our model, the pre-training data, and the compilation of the Spanish&#10;benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.12014" label="2008.12014">
        <attvalues>
          <attvalue for="0" value="GREEK-BERT: The Greeks visiting Sesame Street" />
          <attvalue for="1" value="  Transformer-based language models, such as BERT and its variants, have&#10;achieved state-of-the-art performance in several downstream natural language&#10;processing (NLP) tasks on generic benchmark datasets (e.g., GLUE, SQUAD, RACE).&#10;However, these models have mostly been applied to the resource-rich English&#10;language. In this paper, we present GREEK-BERT, a monolingual BERT-based&#10;language model for modern Greek. We evaluate its performance in three NLP&#10;tasks, i.e., part-of-speech tagging, named entity recognition, and natural&#10;language inference, obtaining state-of-the-art performance. Interestingly, in&#10;two of the benchmarks GREEK-BERT outperforms two multilingual Transformer-based&#10;models (M-BERT, XLM-R), as well as shallower neural baselines operating on&#10;pre-trained word embeddings, by a large margin (5%-10%). Most importantly, we&#10;make both GREEK-BERT and our training code publicly available, along with code&#10;illustrating how GREEK-BERT can be fine-tuned for downstream NLP tasks. We&#10;expect these resources to boost NLP research and applications for modern Greek.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.01900" label="2110.01900">
        <attvalues>
          <attvalue for="0" value="DistilHuBERT: Speech Representation Learning by Layer-wise Distillation&#10;  of Hidden-unit BERT" />
          <attvalue for="1" value="  Self-supervised speech representation learning methods like wav2vec 2.0 and&#10;Hidden-unit BERT (HuBERT) leverage unlabeled speech data for pre-training and&#10;offer good representations for numerous speech processing tasks. Despite the&#10;success of these methods, they require large memory and high pre-training&#10;costs, making them inaccessible for researchers in academia and small&#10;companies. Therefore, this paper introduces DistilHuBERT, a novel multi-task&#10;learning framework to distill hidden representations from a HuBERT model&#10;directly. This method reduces HuBERT's size by 75% and 73% faster while&#10;retaining most performance in ten different tasks. Moreover, DistilHuBERT&#10;required little training time and data, opening the possibilities of&#10;pre-training personal and on-device SSL models for speech.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, self-supervised learning (SSL) methods for speech representation learning succeeded in many tasks \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc,chung2019apc,chung2020vq-apc,chung2020improvedapc,oord2018cpc,riviere2020m-cpc,schneider2019wav2vec,baevski2020vq-wav2vec,baevski2020wav2vec2,sadhu2021wav2vec-c,hsu2021hubert,pascual2019pase,ravanelli2020paseplus,baevski2019effectiveness}.&#10;They learn from targets derived from unlabeled speech data.&#10;They can be roughly categorized into two classes by learning objectives: generative and discriminative.&#10;For generative methods, models try to either reconstruct masked acoustic features \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc}, or generate future acoustic features \cite{chung2019apc,chung2020vq-apc,chung2020improvedapc}.&#10;For the discriminative methods, models either learn by contrastive learning \cite{oord2018cpc,riviere2020m-cpc,schneider2019wav2vec,baevski2020vq-wav2vec,baevski2020wav2vec2,sadhu2021wav2vec-c} or classifying pseudo labels \cite{hsu2021hubert}.&#10;&#10;Many previous SSL methods for speech representation learning are designed for few tasks like speech recognition.&#10;Therefore, Speech processing Universal PERformance Benchmark (SUPERB) \cite{yang2021superb} and LeBenchmark \cite{evain2021lebenchmark} are developed to evaluate the effectiveness of these methods on general speech processing applications.&#10;These benchmarks comprehensively assess the capability of pre-trained models by applying them to multiple downstream tasks like recognition, detection, semantics, and speaker identification.&#10;&#10;Although recent developments of speech SSL methods show excellent performance across various tasks, many require large memory and high pre-training costs, including wav2vec 2.0~\cite{baevski2020wav2vec2} and HuBERT~\cite{hsu2021hubert}.&#10;The limitations make these models unsuitable for on-device computation and make researchers in academia and small corporations difficult to use~\cite{hannun2021history}.&#10;Knowledge distillation is a common method for compressing models \cite{hinton2015distilling}, in which a small student model is learned to generate the teacher model's outputs or hidden representations.&#10;Distilling knowledge has shown to be effective for NLP, and DistilBERT \cite{sanh2019distilbert} and TinyBERT \cite{jiao2020tinybert} are good examples.&#10;However, we found these approaches ineffective in distilling speech SSL models, and few studies investigated this problem \cite{peng2021shrinking,lai2021parp}.&#10;Therefore, we propose a novel multi-task learning framework to layer-wise distill hidden representations of SSL speech models.&#10;&#10;In this work, we distill HuBERT and obtain DistilHuBERT.&#10;DistilHuBERT uses three prediction heads to respectively predict the 4$^{th}$, 8$^{th}$, and 12$^{th}$ HuBERT hidden layers' output.&#10;After training, the heads are removed because the multi-task learning paradigm forces the DistilHuBERT model to learn representations containing rich information.&#10;DistilHuBERT reduces HuBERT's size by 75\% and speedup by 73\%, retaining most performance and requiring less training time.&#10;Moreover, we offer comprehensive analyses of proposed methods and show DistilHuBERT's ability to distill knowledge with few data.&#10;To our knowledge, this is the first attempt to distill speech representations from SSL pre-trained models directly.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Model Distillation Techniques, Linguistics, Efficient Deep Learning, Speech Representation Learning, Artificial Intelligence, Signal Processing" />
        </attvalues>
      </node>
      <node id="1910.12638" label="1910.12638">
        <attvalues>
          <attvalue for="0" value="Mockingjay: Unsupervised Speech Representation Learning with Deep&#10;  Bidirectional Transformer Encoders" />
          <attvalue for="1" value="  We present Mockingjay as a new speech representation learning approach, where&#10;bidirectional Transformer encoders are pre-trained on a large amount of&#10;unlabeled speech. Previous speech representation methods learn through&#10;conditioning on past frames and predicting information about future frames.&#10;Whereas Mockingjay is designed to predict the current frame through jointly&#10;conditioning on both past and future contexts. The Mockingjay representation&#10;improves performance for a wide range of downstream tasks, including phoneme&#10;classification, speaker recognition, and sentiment classification on spoken&#10;content, while outperforming other approaches. Mockingjay is empirically&#10;powerful and can be fine-tuned with downstream models, with only 2 epochs we&#10;further improve performance dramatically. In a low resource setting with only&#10;0.1% of labeled data, we outperform the result of Mel-features that uses all&#10;100% labeled data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Unsupervised speech representation learning \cite{WAVENET_AUTOENCODERS, SPEECH2VEC, AUDIO_WORD2VEC, CPC, APC, WAV2VEC, VQWAV2VEC, ROBUSTCPC, MBV} is effective in extracting high-level properties from speech. SLP downstream tasks can be improved through speech representations because surface features such as log Mel-spectrograms or waveform can poorly reveal the abundant information within speech.&#10;&#10;Contrastive Predictive Coding (CPC)~\cite{CPC} and wav2vec~\cite{WAV2VEC} use a multi-layer CNN to encode past context, representations are learned by predicting the future in latent space under a contrastive binary classification task. &#10;Autoregressive Predictive Coding (APC)~\cite{APC} uses autoregressive models to encode temporal information of past acoustic sequence; the model predicts future frames like an RNN-based language model~\cite{RNN}, optimized with reconstruction loss. &#10;Unidirectional models are commonly used in the previous approaches \cite{WAVENET_AUTOENCODERS, SPEECH2VEC, AUDIO_WORD2VEC, CPC, APC, WAV2VEC}. However, this constraint on model architectures limits the potential of speech representation learning.&#10;&#10;The recently proposed vq-wav2vec~\cite{VQWAV2VEC} approach attempts to apply the well-performing Natural Language Processing (NLP) algorithm BERT~\cite{BERT} on continuous speech. Input speech is discretized to a K-way quantized embedding space, so continuous speech could act like discrete units similar to word tokens in NLP tasks. In vq-wav2vec~\cite{VQWAV2VEC}, an exhaustive two-stage training pipeline with massive computing resources are required to adapt speech to NLP algorithm, as the quantization process is against the continuous nature of speech. Unlike \cite{VQWAV2VEC} that adapts speech to BERT~\cite{BERT} through quantization, the proposed approach can be seen as a modified version of BERT~\cite{BERT} for direct application on continuous speech.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1912.01679" label="1912.01679">
        <attvalues>
          <attvalue for="0" value="Deep Contextualized Acoustic Representations For Semi-Supervised Speech&#10;  Recognition" />
          <attvalue for="1" value="  We propose a novel approach to semi-supervised automatic speech recognition&#10;(ASR). We first exploit a large amount of unlabeled audio data via&#10;representation learning, where we reconstruct a temporal slice of filterbank&#10;features from past and future context frames. The resulting deep contextualized&#10;acoustic representations (DeCoAR) are then used to train a CTC-based end-to-end&#10;ASR system using a smaller amount of labeled audio data. In our experiments, we&#10;show that systems trained on DeCoAR consistently outperform ones trained on&#10;conventional filterbank features, giving 42% and 19% relative improvement over&#10;the baseline on WSJ eval92 and LibriSpeech test-clean, respectively. Our&#10;approach can drastically reduce the amount of labeled data required;&#10;unsupervised training on LibriSpeech then supervision with 100 hours of labeled&#10;data achieves performance on par with training on all 960 hours directly.&#10;Pre-trained models and code will be released online.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.06659" label="2012.06659">
        <attvalues>
          <attvalue for="0" value="DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector&#10;  Quantization" />
          <attvalue for="1" value="  Recent success in speech representation learning enables a new way to&#10;leverage unlabeled data to train speech recognition model. In speech&#10;representation learning, a large amount of unlabeled data is used in a&#10;self-supervised manner to learn a feature representation. Then a smaller amount&#10;of labeled data is used to train a downstream ASR system using the new feature&#10;representations. Based on our previous work DeCoAR and inspirations from other&#10;speech representation learning, we propose DeCoAR 2.0, a Deep Contextualized&#10;Acoustic Representation with vector quantization. We introduce several&#10;modifications over the DeCoAR: first, we use Transformers in encoding module&#10;instead of LSTMs; second, we introduce a vector quantization layer between&#10;encoder and reconstruction modules; third, we propose an objective that&#10;combines the reconstructive loss with vector quantization diversity loss to&#10;train speech representations. Our experiments show consistent improvements over&#10;other speech representations in different data-sparse scenarios. Without&#10;fine-tuning, a light-weight ASR model trained on 10 hours of LibriSpeech&#10;labeled data with DeCoAR 2.0 features outperforms the model trained on the full&#10;960-hour dataset with filterbank features.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2011.00406" label="2011.00406">
        <attvalues>
          <attvalue for="0" value="Non-Autoregressive Predictive Coding for Learning Speech Representations&#10;  from Local Dependencies" />
          <attvalue for="1" value="  Self-supervised speech representations have been shown to be effective in a&#10;variety of speech applications. However, existing representation learning&#10;methods generally rely on the autoregressive model and/or observed global&#10;dependencies while generating the representation. In this work, we propose&#10;Non-Autoregressive Predictive Coding (NPC), a self-supervised method, to learn&#10;a speech representation in a non-autoregressive manner by relying only on local&#10;dependencies of speech. NPC has a conceptually simple objective and can be&#10;implemented easily with the introduced Masked Convolution Blocks. NPC offers a&#10;significant speedup for inference since it is parallelizable in time and has a&#10;fixed inference time for each time step regardless of the input sequence&#10;length. We discuss and verify the effectiveness of NPC by theoretically and&#10;empirically comparing it with other methods. We show that the NPC&#10;representation is comparable to other methods in speech experiments on phonetic&#10;and speaker classification while being more efficient.&#10;" />
          <attvalue for="2" value="&#10;Speech representation learning aims to extract high-level representations from surface features such as waveforms or spectrograms.&#10;Ideally, these representations make hidden information in speech (such as phonetic content and speaker characteristics) more accessible to downstream tasks.&#10;While speech representations can be defined by different transformations on the surface feature, recent researches~\cite{oord2018representation,riviere2020unsupervised,schneider2019wav2vec,baevski2020vq,baevski2020wav2vec,chung2019unsupervised,pascual2019learning,wang2020unsupervised,liu2020mockingjay} have shown great success by combining neural networks and self-supervised learning (where learning targets can be derived from the input itself).&#10;&#10;Contrastive Predictive Coding (CPC)~\cite{oord2018representation} is one such approach whereby the surface feature sequence is first encoded into a latent representation by an encoder network, and an autoregressive model is used to summarize the past latent sequence into a higher-level representation and use it to predict future latent representations.&#10;CPC and its extensions~\cite{riviere2020unsupervised,schneider2019wav2vec,baevski2020vq,baevski2020wav2vec}, have proven to be effective for learning expressive and robust representations of speech.&#10;&#10;Instead of targeting future latent representations, Autoregressive Predictive Coding (APC)~\cite{chung2019unsupervised} suggests that simply predicting future surface features is suitable for learning an effective representation of speech.&#10;APC can be improved by enforcing constraints that information from past sequences be stored in the representation~\cite{chung2020improved} or by imposing an information bottleneck via vector quantization~\cite{chung2020vector}.&#10;&#10;Inspired by the left-to-right nature of speech, both CPC and APC achieve self-supervision by using future features in a uni-directional ordered learning.&#10;Masked Language Modeling (MLM)~\cite{devlin2019bert} relaxes this constraint and uses a different self-supervised learning strategy whereby parts of the input sequence are randomly masked and set to the predicting target, allowing models to input the entire surface feature sequence without seeing the target and derive representation from the context information.&#10;In practice, a bidirectional RNN~\cite{wang2020unsupervised} or Transformer encoder~\cite{liu2020mockingjay} can be employed in learning speech representation through MLM.&#10;&#10;To introduce our work, we first formulate our task and mark two properties of the aforementioned methods.&#10;The goal is to derive a high-level representation $(h_1, h_2,..., h_T)$ from the surface feature sequence of audio $(x_1,x_2,...,x_T)$ with length $T$.&#10;In APC and CPC, the representation $h_t$ at time $t$ is learned by predicting the unseen future frame $x_{t+n}$ (or its latent) based on the current frame $x_t$ and the previous representation $h_{t-1}$.&#10;These methods are 1) inherently autoregressive: the previous representation $h_{t-1}$ is required at each timestep; and 2) incorporating global dependency: $h_{t-1}$ encodes all the past inputs $(x_1,...,x_{t-1})$, making $h_t$ to depend on $(x_1,..,x_t)$.&#10;These properties also apply to MLM, but with a stronger global dependency as the full input sequence is always observed, i.e. $h_t$ depends on $(x_1,...,x_T)$ for any $t$.&#10;Note that these two properties have a huge impact on the efficiency of representation models.&#10;To be more specific, the autoregressive property implies that the extraction process cannot be parallelized in time, and relying on global dependency results in time complexity bounded by the input sequence length as we verify later in our experiments (Sec.~\ref{exp:compare}).&#10;&#10;To this end, we propose Non-Autoregressive Predictive Coding (NPC) to learn representations in a non-autoregressive manner and observing only the local dependency of speech.&#10;Without the autoregressive property, NPC offers a significant speedup for deriving speech representation by parallelizing in time.&#10;By observing only local dependencies, NPC allows representations to be derived efficiently regardless of the input sequence length, which is useful for downstream tasks requiring low latency such as streaming speech recognition.&#10;Furthermore, we show that representations derived by NPC, relying only on local dependencies and a non-autoregressive model, is empirically comparable to different prior works.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.03240" label="1904.03240">
        <attvalues>
          <attvalue for="0" value="An Unsupervised Autoregressive Model for Speech Representation Learning" />
          <attvalue for="1" value="  This paper proposes a novel unsupervised autoregressive neural model for&#10;learning generic speech representations. In contrast to other speech&#10;representation learning methods that aim to remove noise or speaker&#10;variabilities, ours is designed to preserve information for a wide range of&#10;downstream tasks. In addition, the proposed model does not require any phonetic&#10;or word boundary labels, allowing the model to benefit from large quantities of&#10;unlabeled data. Speech representations learned by our model significantly&#10;improve performance on both phone classification and speaker verification over&#10;the surface features and other supervised and unsupervised approaches. Further&#10;analysis shows that different levels of speech information are captured by our&#10;model at different layers. In particular, the lower layers tend to be more&#10;discriminative for speakers, while the upper layers provide more phonetic&#10;content.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.08392" label="2005.08392">
        <attvalues>
          <attvalue for="0" value="Vector-Quantized Autoregressive Predictive Coding" />
          <attvalue for="1" value="  Autoregressive Predictive Coding (APC), as a self-supervised objective, has&#10;enjoyed success in learning representations from large amounts of unlabeled&#10;data, and the learned representations are rich for many downstream tasks.&#10;However, the connection between low self-supervised loss and strong performance&#10;in downstream tasks remains unclear. In this work, we propose Vector-Quantized&#10;Autoregressive Predictive Coding (VQ-APC), a novel model that produces&#10;quantized representations, allowing us to explicitly control the amount of&#10;information encoded in the representations. By studying a sequence of&#10;increasingly limited models, we reveal the constituents of the learned&#10;representations. In particular, we confirm the presence of information with&#10;probing tasks, while showing the absence of information with mutual&#10;information, uncovering the model's preference in preserving speech information&#10;as its capacity becomes constrained. We find that there exists a point where&#10;phonetic and speaker information are amplified to maximize a self-supervised&#10;objective. As a byproduct, the learned codes for a particular model capacity&#10;correspond well to English phones.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.05274" label="2004.05274">
        <attvalues>
          <attvalue for="0" value="Improved Speech Representations with Multi-Target Autoregressive&#10;  Predictive Coding" />
          <attvalue for="1" value="  Training objectives based on predictive coding have recently been shown to be&#10;very effective at learning meaningful representations from unlabeled speech.&#10;One example is Autoregressive Predictive Coding (Chung et al., 2019), which&#10;trains an autoregressive RNN to generate an unseen future frame given a context&#10;such as recent past frames. The basic hypothesis of these approaches is that&#10;hidden states that can accurately predict future frames are a useful&#10;representation for many downstream tasks. In this paper we extend this&#10;hypothesis and aim to enrich the information encoded in the hidden states by&#10;training the model to make more accurate future predictions. We propose an&#10;auxiliary objective that serves as a regularization to improve generalization&#10;of the future frame prediction task. Experimental results on phonetic&#10;classification, speech recognition, and speech translation not only support the&#10;hypothesis, but also demonstrate the effectiveness of our approach in learning&#10;representations that contain richer phonetic content.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.02848" label="2002.02848">
        <attvalues>
          <attvalue for="0" value="Unsupervised pretraining transfers well across languages" />
          <attvalue for="1" value="  Cross-lingual and multi-lingual training of Automatic Speech Recognition&#10;(ASR) has been extensively investigated in the supervised setting. This assumes&#10;the existence of a parallel corpus of speech and orthographic transcriptions.&#10;Recently, contrastive predictive coding (CPC) algorithms have been proposed to&#10;pretrain ASR systems with unlabelled data. In this work, we investigate whether&#10;unsupervised pretraining transfers well across languages. We show that a slight&#10;modification of the CPC pretraining extracts features that transfer well to&#10;other languages, being on par or even outperforming supervised pretraining.&#10;This shows the potential of unsupervised methods for languages with few&#10;linguistic resources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.05862" label="1904.05862">
        <attvalues>
          <attvalue for="0" value="wav2vec: Unsupervised Pre-training for Speech Recognition" />
          <attvalue for="1" value="  We explore unsupervised pre-training for speech recognition by learning&#10;representations of raw audio. wav2vec is trained on large amounts of unlabeled&#10;audio data and the resulting representations are then used to improve acoustic&#10;model training. We pre-train a simple multi-layer convolutional neural network&#10;optimized via a noise contrastive binary classification task. Our experiments&#10;on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up&#10;to 36% when only a few hours of transcribed data is available. Our approach&#10;achieves 2.43% WER on the nov92 test set. This outperforms Deep Speech 2, the&#10;best reported character-based system in the literature while using two orders&#10;of magnitude less labeled training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.05453" label="1910.05453">
        <attvalues>
          <attvalue for="0" value="vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations" />
          <attvalue for="1" value="  We propose vq-wav2vec to learn discrete representations of audio segments&#10;through a wav2vec-style self-supervised context prediction task. The algorithm&#10;uses either a gumbel softmax or online k-means clustering to quantize the dense&#10;representations. Discretization enables the direct application of algorithms&#10;from the NLP community which require discrete inputs. Experiments show that&#10;BERT pre-training achieves a new state of the art on TIMIT phoneme&#10;classification and WSJ speech recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.11477" label="2006.11477">
        <attvalues>
          <attvalue for="0" value="wav2vec 2.0: A Framework for Self-Supervised Learning of Speech&#10;  Representations" />
          <attvalue for="1" value="  We show for the first time that learning powerful representations from speech&#10;audio alone followed by fine-tuning on transcribed speech can outperform the&#10;best semi-supervised methods while being conceptually simpler. wav2vec 2.0&#10;masks the speech input in the latent space and solves a contrastive task&#10;defined over a quantization of the latent representations which are jointly&#10;learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER&#10;on the clean/other test sets. When lowering the amount of labeled data to one&#10;hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour&#10;subset while using 100 times less labeled data. Using just ten minutes of&#10;labeled data and pre-training on 53k hours of unlabeled data still achieves&#10;4.8/8.2 WER. This demonstrates the feasibility of speech recognition with&#10;limited amounts of labeled data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.08393" label="2103.08393">
        <attvalues>
          <attvalue for="0" value="Wav2vec-C: A Self-supervised Model for Speech Representation Learning" />
          <attvalue for="1" value="  Wav2vec-C introduces a novel representation learning technique combining&#10;elements from wav2vec 2.0 and VQ-VAE. Our model learns to reproduce quantized&#10;representations from partially masked speech encoding using a contrastive loss&#10;in a way similar to Wav2vec 2.0. However, the quantization process is&#10;regularized by an additional consistency network that learns to reconstruct the&#10;input features to the wav2vec 2.0 network from the quantized representations in&#10;a way similar to a VQ-VAE model. The proposed self-supervised model is trained&#10;on 10k hours of unlabeled data and subsequently used as the speech encoder in a&#10;RNN-T ASR model and fine-tuned with 1k hours of labeled data. This work is one&#10;of only a few studies of self-supervised learning on speech tasks with a large&#10;volume of real far-field labeled data. The Wav2vec-C encoded representations&#10;achieves, on average, twice the error reduction over baseline and a higher&#10;codebook utilization in comparison to wav2vec 2.0&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-supervision \cite{schneider2019wav2vec,chen2020big,oord2018representation,lan2019albert} is a paradigm of machine learning (ML) that deals with unsupervised learning of structural patterns in data by exploiting contextual information. Self-supervision has been of significant interest in the automatic speech recognition (ASR) literature primarily as a pre-training step before a fully supervised task. In particular, it is widely used for problems with some amount of labeled data (for supervised training) and a significantly larger volume of unlabeled data (for self-supervised training). The recently proposed wav2vec 2.0 \cite{baevski2020wav2vec} is one such self-supervised learning model that learns to predict masked out discrete speech encodings using a contextualized representation from a transformer model \cite{vaswani2017attention}. \\&#10;&#10;In this paper, we introduce the wav2vec-C model that solves a more rigorously defined self-supervised learning problem compared to the wav2vec 2.0. In the latter, a contrastive loss defined on discretized codes drives the self-supervised learning - including the codebook in the built-in differentiable Vector Quantization module. In contrast, wav2vec-C facilitates codebook learning through an additional regularization on the discrete speech representations by reconstructing the discrete codes to the input features. Thus, wav2vec-C maintains a consistency between the learnt representations and the input features to the network. \\&#10;&#10;Our main contributions in this paper are&#10;\begin{itemize}\item The wav2vec-C model (Section \ref{sec:model}) &#10;\item We use real world far-field voice query speech with varied degrees of SNR ranging between -40 to 50 dB, whereas most studies on self-supervised learning in the literature use clean read speech \cite{chung2020generative,baevski2019vq} and some use simulated noisy speech \cite{ravanelli2020multi}.&#10;\item Self-supervised learning has been shown to be useful for settings with little labeled data \cite{schneider2019wav2vec,ravanelli2020multi}. It has been observed that the effectiveness of self-supervision decreases as the amount of labeled data increases\cite{amazon,chung2020generative}. In this work, we explore the applicability of self-supervision with a relatively large amount of labeled data (1k hours).&#10;\item We also limit our model size to facilitate low-latency production level ASR models, which goes against the general trend of exceedingly large self-supervised models proposed in the literature \cite{chen2020big}.&#10;\item We explore and compare different variants of our framework in the choice of the vector quantization framework and the effect it has on robustness and codebook utilization.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.07447" label="2106.07447">
        <attvalues>
          <attvalue for="0" value="HuBERT: Self-Supervised Speech Representation Learning by Masked&#10;  Prediction of Hidden Units" />
          <attvalue for="1" value="  Self-supervised approaches for speech representation learning are challenged&#10;by three unique problems: (1) there are multiple sound units in each input&#10;utterance, (2) there is no lexicon of input sound units during the pre-training&#10;phase, and (3) sound units have variable lengths with no explicit segmentation.&#10;To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT)&#10;approach for self-supervised speech representation learning, which utilizes an&#10;offline clustering step to provide aligned target labels for a BERT-like&#10;prediction loss. A key ingredient of our approach is applying the prediction&#10;loss over the masked regions only, which forces the model to learn a combined&#10;acoustic and language model over the continuous inputs. HuBERT relies primarily&#10;on the consistency of the unsupervised clustering step rather than the&#10;intrinsic quality of the assigned cluster labels. Starting with a simple&#10;k-means teacher of 100 clusters, and using two iterations of clustering, the&#10;HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0&#10;performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with&#10;10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model,&#10;HuBERT shows up to 19% and 13% relative WER reduction on the more challenging&#10;dev-other and test-other evaluation subsets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;We discuss recent studies on self-supervised speech representation learning by grouping them by training objective. The earliest line of work learns representations by postulating a generative model for speech with latent variables, which are assumed to capture the relevant phonetic information. Training of these models amounts to likelihood maximization. Different latent structures have been applied to encode the prior assumption, such as continuous~\cite{hsu2017learning}, discrete~\cite{chorowski2019unsupervised,van2017neural}, or sequential~\cite{hsu2017unsupervised,ebbers2017hidden,glarner2018full,khurana2019factorial,khurana2020convolutional}. &#10;&#10;Prediction-based self-supervised learning has gathered increasing interests recently, where a model is tasked to predict the content of the unseen regions \cite{chung2019unsupervised, chung2020generative, chung2020improved, ling2020deep, wang2020unsupervised, liu2020mockingjay, chi2020audio, ling2020decoar} or to contrast the target unseen frame with randomly sampled ones \cite{oord2018representation, kharitonov2020data, schneider2019wav2vec, baevski2020wav2vec}. Some models combine both the predictive and the contrastive losses \cite{baevski2019vq, baevski2019effectiveness}. These objectives can usually be interpreted as mutual information maximization~\cite{tsai2020ssl_multi}. Other objectives do not belong to these categories, for example, \cite{pascual2019learning}.&#10;&#10;This work is most related to DiscreteBERT~\cite{baevski2019effectiveness}: both HuBERT and DiscreteBERT predict discrete targets of masked regions. However, there are several crucial differences. First, instead of taking quantized units as input, HuBERT takes raw waveforms as input to pass as much information as possible to the transformer layers, which was shown to be important in \cite{baevski2020wav2vec}. Furthermore, in the experiment section, we show that our model, with simple k-means targets, can achieve better performance than DiscreteBERT that uses vq-wav2vec \cite{baevski2019vq} learned units. Second, we also present many techniques to improve teacher quality instead of using a single fixed teacher as done in DiscreteBERT.&#10;&#10;HuBERT is also related to wav2vec 2.0~\cite{baevski2020wav2vec}. However, the latter employs a contrastive loss that requires careful design of where to sample negative frames from, an auxiliary diversity loss to encourage the discrete unit usage, and demands a proper Gumbel-softmax temperature annealing schedule. In addition, it only explores quantizing the waveform encoder output, which may not be the best feature for quantization due to the limited capacity of the convolutional encoder, as suggested by our ablation studies in Figure~\ref{fig:qual_layer}. Concretely, our proposed method adopts a more direct predictive loss by separating the acoustic unit discovery step from the masked prediction representation learning phase and achieves the state-of-the-art results that match or outperform wav2vec 2.0 on different fine-tuning scales.&#10;&#10;Finally, the idea of iterative refinement target labels is similar to iterative pseudo labeling for semi-supervised ASR~\cite{xu2020iterative, likhomanenko2020slimipl}, which leverages an improving student model to generate better pseudo-labels for the next iteration of training. The HuBERT approach can be seen as extending this method to the self-supervised setup with a masked prediction loss.&#10; " />
        </attvalues>
      </node>
      <node id="2104.11462" label="2104.11462">
        <attvalues>
          <attvalue for="0" value="LeBenchmark: A Reproducible Framework for Assessing Self-Supervised&#10;  Representation Learning from Speech" />
          <attvalue for="1" value="  Self-Supervised Learning (SSL) using huge unlabeled data has been&#10;successfully explored for image and natural language processing. Recent works&#10;also investigated SSL from speech. They were notably successful to improve&#10;performance on downstream tasks such as automatic speech recognition (ASR).&#10;While these works suggest it is possible to reduce dependence on labeled data&#10;for building efficient speech systems, their evaluation was mostly made on ASR&#10;and using multiple and heterogeneous experimental settings (most of them for&#10;English). This questions the objective comparison of SSL approaches and the&#10;evaluation of their impact on building speech systems. In this paper, we&#10;propose LeBenchmark: a reproducible framework for assessing SSL from speech. It&#10;not only includes ASR (high and low resource) tasks but also spoken language&#10;understanding, speech translation and emotion recognition. We also focus on&#10;speech technologies in a language different than English: French. SSL models of&#10;different sizes are trained from carefully sourced and documented datasets.&#10;Experiments show that SSL is beneficial for most but not all tasks which&#10;confirms the need for exhaustive and reliable benchmarks to evaluate its real&#10;impact. LeBenchmark is shared with the scientific community for reproducible&#10;research in SSL from speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.15760" label="2103.15760">
        <attvalues>
          <attvalue for="0" value="Shrinking Bigfoot: Reducing wav2vec 2.0 footprint" />
          <attvalue for="1" value="  Wav2vec 2.0 is a state-of-the-art speech recognition model which maps speech&#10;audio waveforms into latent representations. The largest version of wav2vec 2.0&#10;contains 317 million parameters. Hence, the inference latency of wav2vec 2.0&#10;will be a bottleneck in production, leading to high costs and a significant&#10;environmental footprint. To improve wav2vec's applicability to a production&#10;setting, we explore multiple model compression methods borrowed from the domain&#10;of large language models. Using a teacher-student approach, we distilled the&#10;knowledge from the original wav2vec 2.0 model into a student model, which is 2&#10;times faster and 4.8 times smaller than the original model. This increase in&#10;performance is accomplished with only a 7% degradation in word error rate&#10;(WER). Our quantized model is 3.6 times smaller than the original model, with&#10;only a 0.1% degradation in WER. To the best of our knowledge, this is the first&#10;work that compresses wav2vec 2.0.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.05933" label="2106.05933">
        <attvalues>
          <attvalue for="0" value="PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition" />
          <attvalue for="1" value="  Self-supervised speech representation learning (speech SSL) has demonstrated&#10;the benefit of scale in learning rich representations for Automatic Speech&#10;Recognition (ASR) with limited paired data, such as wav2vec 2.0. We investigate&#10;the existence of sparse subnetworks in pre-trained speech SSL models that&#10;achieve even better low-resource ASR results. However, directly applying widely&#10;adopted pruning methods such as the Lottery Ticket Hypothesis (LTH) is&#10;suboptimal in the computational cost needed. Moreover, we show that the&#10;discovered subnetworks yield minimal performance gain compared to the original&#10;dense network. We present Prune-Adjust-Re-Prune (PARP), which discovers and&#10;finetunes subnetworks for much better performance, while only requiring a&#10;single downstream ASR finetuning run. PARP is inspired by our surprising&#10;observation that subnetworks pruned for pre-training tasks need merely a slight&#10;adjustment to achieve a sizeable performance boost in downstream ASR tasks.&#10;Extensive experiments on low-resource ASR verify (1) sparse subnetworks exist&#10;in mono-lingual/multi-lingual pre-trained speech SSL, and (2) the computational&#10;advantage and performance gain of PARP over baseline pruning methods. In&#10;particular, on the 10min Librispeech split without LM decoding, PARP discovers&#10;subnetworks from wav2vec 2.0 with an absolute 10.9%/12.6% WER decrease compared&#10;to the full model. We further demonstrate the effectiveness of PARP via:&#10;cross-lingual pruning without any phone recognition degradation, the discovery&#10;of a multi-lingual subnetwork for 10 spoken languages in 1 finetuning run, and&#10;its applicability to pre-trained BERT/XLNet for natural language tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Modern Speech Paradigm and ASR Pruning.&#10;As model scale~\cite{synnaeve2019end,baevski2020wav2vec,han2020contextnet,gulati2020conformer,yu2020universal,pratap2020scaling,pratap2020massively,yu2021dual,chen2021continuous,you2021speechmoe,li2021scaling} and model pre-training~\cite{baevski2020wav2vec,zhang2020pushing,conneau2020unsupervised,kong2020panns,jiang2020speech,lai2021semi,hsu2021hubert,xu2021self,chan2021speechstew,kanda2021large,sanabria2021talk,saeed2021contrastive,ng2021pushing,polyak2021speech,wang2021contrastive} have become the two essential ingredients for obtaining SOTA performance in ASR and other speech tasks, applying and developing various forms of memory-efficient algorithms, such as network pruning, to these large-scale pre-trained models will predictably soon become an indispensable research endeavor. &#10;Early work on ASR pruning can be dated back to pruning decoding search spaces~\cite{abdou2004beam,pylkkonen2005new,siivola2007growing,he2014reshaping,xu2018pruned,zhang2021tiny} and HMM state space~\cite{van1996adaptive}. &#10;Since the seminal work of Yu et al.~\cite{yu2012exploiting}, ASR pruning has focused primarily on end-to-end network architecture:~\cite{shangguan2019optimizing,wu2021dynamic} applied pruning and quantization to LSTM-based RNN-Transducers,~\cite{panchapagesan2021efficient} applied knowledge distillation to Conformer-based RNN-Transducers,~\cite{venkatesh2021memory,shi2021emformer,li2021efficient} designed efficient architecture/mechanisms for LSTM, Transformer, Conformer-based ASR models,~\cite{narang2017exploring} applied pruning to Deep Speech,~\cite{braun2019parameter} introduced SNR-based probabilistic pruning on LSTM-based CTC model,~\cite{gao2020rethinking} proposed entropy-regularizer for LSTM-based ASR model, ~\cite{xue2013restructuring,povey2018semi} applied SVD on ASR models' weight matrices. &#10;We emphasize that our work is the first on pruning large self-supervised pre-trained models for low-resource and multi-lingual ASR. &#10;In addition, to our knowledge, none of the prior speech pruning work demonstrated the pruned models attain superior performance than its original counterpart. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2312.10523" label="2312.10523">
        <attvalues>
          <attvalue for="0" value="Paloma: A Benchmark for Evaluating Language Model Fit" />
          <attvalue for="1" value="  Language models (LMs) commonly report perplexity on monolithic data held out&#10;from training. Implicitly or explicitly, this data is composed of&#10;domains$\unicode{x2013}$varying distributions of language. Rather than assuming&#10;perplexity on one distribution extrapolates to others, Perplexity Analysis for&#10;Language Model Assessment (Paloma), measures LM fit to 585 text domains,&#10;ranging from nytimes.com to r/depression on Reddit. We invite submissions to&#10;our benchmark and organize results by comparability based on compliance with&#10;guidelines such as removal of benchmark contamination from pretraining.&#10;Submissions can also record parameter and training token count to make&#10;comparisons of Pareto efficiency for performance as a function of these&#10;measures of cost. We populate our benchmark with results from 6 baselines&#10;pretrained on popular corpora. In case studies, we demonstrate analyses that&#10;are possible with Paloma, such as finding that pretraining without data beyond&#10;Common Crawl leads to inconsistent fit to many domains.&#10;" />
          <attvalue for="2" value="&#10;&#10;Progress in AI is catalyzed by evaluations that define new ways of measuring progress (\cite{imagenet}, \cite{wang-etal-2018-glue}, and \cite{superglue}, inter alia). Language models (LMs) often evaluate LM fit as loss or perplexity \cite{Jelinek1977PerplexityaMO} on held out training data or few traditional test sets (\cite{Chelba2013OneBW,Merity2016PointerSM}, inter alia). &#10;These loss measures have been shown to improve predictably with increases in training compute \cite{Kaplan2020ScalingLF, Hoffmann2022TrainingCL} and loss may predict performance on downstream tasks \cite{Xia2022TrainingTO, Gadre2024LanguageMS, Du2024UnderstandingEA}.&#10;However, scaling pretraining data aggregates more domains that LMs implicitly learn to model \cite{Diaz2023ScalingLD, aharoni-goldberg-2020-unsupervised}.&#10;Does rising performance lift all data? Or do some domains capture most improvement in LM fit? How do we evaluate what language distributions models learn from different pretraining data? What domains should studies evaluate loss on to measure the relationship of loss and downstream performance? To answer these questions, perplexity evaluations ought to measure LM fit to many domains, rather than extrapolating trends from a single prescriptive mix of domains. &#10;&#10;In this work we introduce \pplSuite, a benchmark to study LM fit on many domains. We measure perplexity on different distributions of language sampled from \numTasks sources, such as \cFour \cite{Raffel2019ExploringTL}, that have metadata such as URLs marking \numDomains textual domains.&#10;Beyond evaluation data, we aim to enable and enrich fair comparisons for scientific research on language modeling with the following artifacts: guidelines for comparing LM fit, \numTrainedBaselines baseline 1B parameter models pretrained on popular corpora, and standardized code for experiments with \pplSuite.&#10;&#10;As reproducing pretrained models for every new project is onerous, we provide standard training controls for benchmark decontamination and training data order to orchestrate a greater density of comparisons across the research community. We also control how \pplSuite is evaluated by fixing sample size per domain, model vocabulary, and inference format. Lastly, we demonstrate how to make fair comparisons over two measures of cost, number of model parameters and training tokens, enabling assessment of hardware-agnostic efficiency and the measurement of scaling trends.&#10;&#10;Among the \numTasks sources curated in our benchmark, we contribute two new datasets constructed from data held out of \dolma \cite{dolma}: (1) a subsample of the top 100 subreddits by number of comments, and (2) code from the top 100 programming languages by number of tokens. Also, we repurpose corpora of fringe online communities to measure LM fit to discourse previously studied for the prevalence of toxicity and hate speech \cite{Horta_Ribeiro_2021,ZannettouGab,Papasavva_2020}.&#10;While, capturing domains required by all possible lines of research is impossible for any one benchmark, \pplSuite focuses on English and code data and aims to assemble the most fine-grained domains readily identifiable from existing metadata.&#10;&#10;To demonstrate possible uses of results from our dataset, we present a series of case studies in \S\ref{sec:case_studies}. &#10;Among other findings, our experiments isolate change in fit from which pretraining corpus is used (Figure~\ref{fig:models_by_ppl_over_all_tasks}) and find that pretraining without heterogeneous data sources beyond Common Crawl can lead to perplexities in some domains that do not improve consistently with number of tokens seen. We also find that few vocabulary types account for most of the loss measured in perplexity.&#10;&#10;In sum, \pplSuite contributes:&#10;\begin{enumerate}&#10; \item Curated release of the most fine-grained perplexity evaluation data in use in LM research, along with guidelines and code for standardized and rigorous perplexity evaluation.&#10; \item New evaluation data for the 100 most popular subreddits and programming languages.&#10; \item 1B LMs pretrained on \cFour, \mCFourEn, \falcon, \pile, \redPajama, and \dolma with controlled hyperparameters, token budget, benchmark decontamination, and training order for fair comparisons, along with code for others to do the same.&#10; \item Case studies demonstrating analyses that are possible with \pplSuite, such as finding that pretraining without data beyond Common Crawl leads to inconsistent fit to many domains and that perplexity is driven by improved fit on the most common vocabulary strings.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Domain Adaptation Analysis, Benchmarking Methodology, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2402.00159" label="2402.00159">
        <attvalues>
          <attvalue for="0" value="Dolma: an Open Corpus of Three Trillion Tokens for Language Model&#10;  Pretraining Research" />
          <attvalue for="1" value="  Information about pretraining corpora used to train the current&#10;best-performing language models is seldom discussed: commercial models rarely&#10;detail their data, and even open models are often released without accompanying&#10;training data or recipes to reproduce them. As a result, it is challenging to&#10;conduct and advance scientific research on language modeling, such as&#10;understanding how training data impacts model capabilities and limitations. To&#10;facilitate scientific research on language model pretraining, we curate and&#10;release Dolma, a three-trillion-token English corpus, built from a diverse&#10;mixture of web content, scientific papers, code, public-domain books, social&#10;media, and encyclopedic materials. We extensively document Dolma, including its&#10;design principles, details about its construction, and a summary of its&#10;contents. We present analyses and experimental results on intermediate states&#10;of Dolma to share what we have learned about important data curation practices.&#10;Finally, we open-source our data curation toolkit to enable reproduction of our&#10;work as well as support further research in large-scale data curation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13862" label="2305.13862">
        <attvalues>
          <attvalue for="0" value="A Trip Towards Fairness: Bias and De-Biasing in Large Language Models" />
          <attvalue for="1" value="  Cheap-to-Build Very Large-Language Models (CtB-LLMs) with affordable training&#10;are emerging as the next big revolution in natural language processing and&#10;understanding. These CtB-LLMs are democratizing access to trainable Very&#10;Large-Language Models (VLLMs) and, thus, may represent the building blocks of&#10;many NLP systems solving downstream tasks. Hence, a little or a large bias in&#10;CtB-LLMs may cause huge harm. In this paper, we performed a large investigation&#10;of the bias of three families of CtB-LLMs, and we showed that debiasing&#10;techniques are effective and usable. Indeed, according to current tests, the&#10;LLaMA and the OPT families have an important bias in gender, race, religion,&#10;and profession. In contrast to the analysis for other LLMs, we discovered that&#10;bias depends not on the number of parameters but on the perplexity. Finally,&#10;the debiasing of OPT using LoRA reduces bias up to 4.12 points in the&#10;normalized stereotype score.&#10;" />
          <attvalue for="2" value="&#10;&#10;Very Large Language Models (VLLMs) like ChatGPT have become a standard building block in Artificial Intelligence applications since they can be adapted to a wide range of downstream tasks. &#10;Transformer-based language models \cite{transfVaswani}, which have disrupted classical NLP pipeline \cite{tenney2019bert}, have grown in size and capabilities in recent years. The pre-training step from large text corpora, with different language modeling strategies, appeared to be the key to getting remarkable results on various tasks after fine-tuning on smaller datasets. VLLMs that represent the new version of transformer-based language models are based on corpora not so far from their forerunners. Still, the considerable growth in the number of parameters seems to provide the breakthrough. While the performance is unmistakable, the resources needed are prohibitive for non-company research.&#10;&#10;Recently, \cite{touvron2023llama} proposed Large Language Model Meta AI (LLaMA). This solution aims to democratize training and domain adaptation of VLLM by opening the door to Cheap-to-Build Very Large-Language models (CtB-LLMs). LLaMA was made available in different sizes (7B, 13B, 33B, and 65B parameters) to provide smaller, high-performance models that allow researchers who do not have access to considerable amounts of infrastructure to use these models.The key to LLaMA's success seems to be the outstanding trade-off between lowering parameters and enriching pre-training corpora compared to the characteristics of other VLLMs (see Tab. \ref{tab:parametersAndData}).&#10;&#10;However, the considerable increase in pre-training corpora makes it challenging to assess the characteristics and check the reliability of these data.&#10;Therefore, learned representations may inherit the biases and stereotypical associations present in the large text corpora in the language and, thus, in the pre-training corpora taken from the web \cite{liang2021understanding}.&#10;Although the spread of the phenomenon is widely recognized, the causes that emphasize this phenomenon remain largely unexplored.&#10;It has been observed that as the size of a model increases, its linguistic modeling capabilities and biases increase \cite{nadeem-etal-2021-stereoset}. On the other hand, distilled versions of target models tend to show more bias \cite{silva-etal-2021-towards}. These mixed results, although expected since the compared models were trained on different amounts of data and sources, make it unclear whether the presence of the bias depends on the number of parameters.&#10;&#10;In this paper, we performed a deep investigation of the bias of three families of CtB-LLMs, and we showed that debiasing techniques are effective and usable. By investigating the analogies between model size growth concerning pre-training parameters or corpora and bias memorization. Thus, we hypothesize that the CtB-LLMs performance depends on the quality of the training data and that, between different models, there are no significant differences in terms of bias. Finally, we also study the effect of fine-tuning with anti-stereotypical sentences by proposing a lightweight approach to build fairer models. By testing the 7-billion-parameter LLaMA model and Open Pre-trained Transformer Language Models (OPT) \cite{zhang2022opt}, we show that although the model shows less biased behavior after fine-tuning, the method also achieves a reasonable overall performance of the language model. Therefore, our approach produces fairer language models using limited resources and achieves sustainable performance on downstream benchmark tasks.&#10;&#10;The major contributions of this paper are:&#10;\begin{itemize}&#10;\item a first comprehensive analysis of the bias for two families of affordable, Cheap-to-Build Large-Language Models (CtB-LLMs) &#10;\item establishing the anti-correlation between perplexity and bias in CtB-LLMs&#10;\item demonstrating that simple de-biasing techniques can be positively used to reduce bias in these two classes of CtB-LLMs while not reducing performance on downstream tasks&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Bias Detection Methods, Mathematics, Debiasing Techniques" />
        </attvalues>
      </node>
      <node id="1905.05950" label="1905.05950">
        <attvalues>
          <attvalue for="0" value="BERT Rediscovers the Classical NLP Pipeline" />
          <attvalue for="1" value="  Pre-trained text encoders have rapidly advanced the state of the art on many&#10;NLP tasks. We focus on one such model, BERT, and aim to quantify where&#10;linguistic information is captured within the network. We find that the model&#10;represents the steps of the traditional NLP pipeline in an interpretable and&#10;localizable way, and that the regions responsible for each step appear in the&#10;expected sequence: POS tagging, parsing, NER, semantic roles, then coreference.&#10;Qualitative analysis reveals that the model can and often does adjust this&#10;pipeline dynamically, revising lower-level decisions on the basis of&#10;disambiguating information from higher-level representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.13219" label="2106.13219">
        <attvalues>
          <attvalue for="0" value="Towards Understanding and Mitigating Social Biases in Language Models" />
          <attvalue for="1" value="  As machine learning methods are deployed in real-world settings such as&#10;healthcare, legal systems, and social science, it is crucial to recognize how&#10;they shape social biases and stereotypes in these sensitive decision-making&#10;processes. Among such real-world deployments are large-scale pretrained&#10;language models (LMs) that can be potentially dangerous in manifesting&#10;undesirable representational biases - harmful biases resulting from&#10;stereotyping that propagate negative generalizations involving gender, race,&#10;religion, and other social constructs. As a step towards improving the fairness&#10;of LMs, we carefully define several sources of representational biases before&#10;proposing new benchmarks and metrics to measure them. With these tools, we&#10;propose steps towards mitigating social biases during text generation. Our&#10;empirical results and human evaluation demonstrate effectiveness in mitigating&#10;bias while retaining crucial contextual information for high-fidelity text&#10;generation, thereby pushing forward the performance-fairness Pareto frontier.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.09456" label="2004.09456">
        <attvalues>
          <attvalue for="0" value="StereoSet: Measuring stereotypical bias in pretrained language models" />
          <attvalue for="1" value="  A stereotype is an over-generalized belief about a particular group of&#10;people, e.g., Asians are good at math or Asians are bad drivers. Such beliefs&#10;(biases) are known to hurt target groups. Since pretrained language models are&#10;trained on large real world data, they are known to capture stereotypical&#10;biases. In order to assess the adverse effects of these models, it is important&#10;to quantify the bias captured in them. Existing literature on quantifying bias&#10;evaluates pretrained language models on a small set of artificially constructed&#10;bias-assessing sentences. We present StereoSet, a large-scale natural dataset&#10;in English to measure stereotypical biases in four domains: gender, profession,&#10;race, and religion. We evaluate popular models like BERT, GPT-2, RoBERTa, and&#10;XLNet on our dataset and show that these models exhibit strong stereotypical&#10;biases. We also present a leaderboard with a hidden test set to track the bias&#10;of future language models at https://stereoset.mit.edu&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.06168" label="2205.06168">
        <attvalues>
          <attvalue for="0" value="Using dependency parsing for few-shot learning in distributional&#10;  semantics" />
          <attvalue for="1" value="  In this work, we explore the novel idea of employing dependency parsing&#10;information in the context of few-shot learning, the task of learning the&#10;meaning of a rare word based on a limited amount of context sentences. Firstly,&#10;we use dependency-based word embedding models as background spaces for few-shot&#10;learning. Secondly, we introduce two few-shot learning methods which enhance&#10;the additive baseline model by using dependencies.&#10;" />
          <attvalue for="2" value="&#10;&#10;Distributional semantics models create word embeddings based on the assumption that the&#10;meaning of a word is defined by the contexts it is used in&#10;(for an overview, see: \cite{sahlgren2008distributional,lenci2018distributional,boleda2020distributional,emerson-2020-goals}).&#10;A fundamental challenge for these approaches is the difficulty of producing high-quality embeddings for rare words, since the models often require vast amounts of training examples \cite{adams-etal-2017-cross,van-hautte-etal-2019-bad}.&#10;To address this problem, various few-shot learning methods have been previously introduced. The goal&#10;of a few-shot learning technique is to learn an embedding that captures the meaning of a&#10;word, given only a few context sentences. The rare word's vector has to be placed in an existing background space of embeddings.&#10;&#10;Few-shot learning in distributional semantics is a relatively underexplored area,&#10;with important practical applications. Having good representations of rare words is highly&#10;desirable in applications aiming to understand dialects or regionalisms, as well as specific technical language.&#10;&#10;In this work, we explore the idea of incorporating information from the dependency parse of sentences in the context of few shot-learning. An intuition why this might be useful is provided in Figure \ref{fig:parse}. In the given sentence, the most relevant word for inferring the meaning of the target rare word ``conflagration&quot; is ``destroyed&quot;. Even if this word is located far from the target, it is directly connected to it through a nominal subject dependency. Moreover, the fact that the target word is used in a certain dependency structure might reveal important characteristics related to its meaning. Since in the case of few-shot learning the data is limited, using dependency parsing information is a resource with great potential to boost existing models.&#10;&#10;As a first effort in this direction, this work provides three contributons. Firstly, we explore the effect of using dependency-based word embeddings as background spaces. Secondly, we introduce new few-shot learning methods leveraging the dependency parsing information. Lastly, we update a previous dependency-based background model to make it more suitable for few-shot learning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Machine Learning, Computer Science, Linguistics, Word Embeddings, Artificial Intelligence, Natural Language Processing, Dependency Parsing" />
        </attvalues>
      </node>
      <node id="1905.01896" label="1905.01896">
        <attvalues>
          <attvalue for="0" value="Distributional Semantics and Linguistic Theory" />
          <attvalue for="1" value="  Distributional semantics provides multi-dimensional, graded, empirically&#10;induced word representations that successfully capture many aspects of meaning&#10;in natural languages, as shown in a large body of work in computational&#10;linguistics; yet, its impact in theoretical linguistics has so far been&#10;limited. This review provides a critical discussion of the literature on&#10;distributional semantics, with an emphasis on methods and results that are of&#10;relevance for theoretical linguistics, in three areas: semantic change,&#10;polysemy and composition, and the grammar-semantics interface (specifically,&#10;the interface of semantics with syntax and with derivational morphology). The&#10;review aims at fostering greater cross-fertilization of theoretical and&#10;computational approaches to language, as a means to advance our collective&#10;knowledge of how it works.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.06925" label="2402.06925">
        <attvalues>
          <attvalue for="0" value="A Thorough Examination of Decoding Methods in the Era of LLMs" />
          <attvalue for="1" value="  Decoding methods play an indispensable role in converting language models&#10;from next-token predictors into practical task solvers. Prior research on&#10;decoding methods, primarily focusing on task-specific models, may not extend to&#10;the current era of general-purpose large language models (LLMs). Moreover, the&#10;recent influx of decoding strategies has further complicated this landscape.&#10;This paper provides a comprehensive and multifaceted analysis of various&#10;decoding methods within the context of LLMs, evaluating their performance,&#10;robustness to hyperparameter changes, and decoding speeds across a wide range&#10;of tasks, models, and deployment environments. Our findings reveal that&#10;decoding method performance is notably task-dependent and influenced by factors&#10;such as alignment, model size, and quantization. Intriguingly, sensitivity&#10;analysis exposes that certain methods achieve superior performance at the cost&#10;of extensive hyperparameter tuning, highlighting the trade-off between&#10;attaining optimal results and the practicality of implementation in varying&#10;contexts.&#10;" />
          <attvalue for="2" value="&#10;The advent of large language models (LLMs)~\cite[][inter alia]{2022OpenAIchatgpt,2023GPT4Openai,touvron2023llama,touvron2023llama2} has ushered in a new era of natural language processing (NLP). These models are trained to predict the next token on massive corpora, empowering them with extraordinary multitasking capabilities. This enables them to perform almost all NLP tasks through the lens of text generation, distinguishing them from traditional task-specific models.&#10;&#10;Decoding methods, which are the bridge between next-token predictors and text generators, play an integral role in transforming LLMs into practical task solvers. Recent studies have shown that the choice of decoding methods can substantially impact the performance of LLMs \cite{o2023contrastive,chuang2023dola}. However, these studies often focus on a narrow aspect (e.g., factuality~\cite{chuang2023dola}) and a limited set of similar tasks (e.g., math problem solving~\cite{li2022contrastive}). Notably, \cite{ippolito-etal-2019-comparison,wiher2022decoding} provide a comparative analysis of various decoding methods using task-specific language models. They find that deterministic decoding methods (e.g., beam search) perform better than stochastic decoding methods (e.g., top-$p$ sampling~\cite{Holtzman2020The}) in closed-ended generation tasks such as machine translation, while the inverse is true for open-ended generation tasks such as story generation. However, their findings are confined to traditional task-specific models prior to the advent of LLMs. It is uncertain whether their conclusions still hold for general-purpose LLMs. In addition, a plethora of new decoding methods \cite{su2022a,li2022contrastive,yang2023frustratingly,meister-etal-2023-locally,hewitt2022truncation,basu2021mirostat} have been proposed afterward, each claiming to outperform the previous state-of-the-art in particular tasks. Nevertheless, today's most performant LLMs such as ChatGPT and GPT4 \cite{2022OpenAIchatgpt,2023GPT4Openai} only provide APIs for temperature and top-$p$ sampling, seemingly overlooking the potential benefits of other advanced decoding methods.&#10;&#10;The above observations raise a natural question: what is the best practice for choosing decoding methods in the era of LLMs? A thorough analysis of decoding methods is essential for researchers and practitioners to understand the strengths and weaknesses of different decoding methods and to choose the one that best fits their needs. Our work fills this gap by providing a comprehensive study of the performance, robustness, and speed of various decoding methods across a wide range of different tasks, models, and deployment environments. We also provide in-depth analyses to uncover the underlying reasons for the observed results. Our key findings include the following:&#10;\begin{itemize}[wide=0\parindent,noitemsep,topsep=0em]&#10; \item Overall The optimal decoding method depends on the task, the model, and the priority (e.g., performance vs. robustness vs. speed) in hand. There is no short guideline. The complexity of our results calls for more comprehensive evaluations in future research on decoding methods and careful consideration for practitioners.&#10; \item Performance The best-performing methods depend on the task at hand. However, some general rules about the divide between different decoding methods still persist in the era of LLMs. Generally, closed-ended tasks favor deterministic methods, while open-ended tasks prefer stochastic methods (\ref{sec:performance}), especially with unaligned models. The performance gap between different decoding methods can be narrowed with alignment. We also provide explanations to understand these phenomena. Moreover, it is also observed that stochastic methods with self-consistency can surpass deterministic ones, albeit requiring multiple runs (\ref{sec:consistency}). &#10; \item Robustness The optimal hyperparameters for each decoding method vary according to the model, task, and quantization setting. Some methods achieve superior performance at the cost of exhaustive dataset-specific hyperparameter searches but fail to maintain the superiority when the hyperparameter is fixed. This highlights the performance-sensitivity trade-off because LLMs are often confronted with diverse user prompts (\ref{sec:robustness}).&#10; \item Speed Stochastic decoding and the recently proposed deterministic method, frustratingly simple decoding (FSD)~\cite{yang2023frustratingly}, can achieve a similar decoding speed to greedy search. In contrast, beam search, diverse beam search and other advanced deterministic methods show markedly slower speeds relative to greedy search, with the discrepancy in speed becoming more conspicuous as the length of generation increases for some of those methods (\ref{sec:speed}).&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Decoding, Computer Science, Large Language Models, Information Technology, Linguistics, Decoding Method Evaluation, Mathematics" />
        </attvalues>
      </node>
      <node id="2309.09117" label="2309.09117">
        <attvalues>
          <attvalue for="0" value="Contrastive Decoding Improves Reasoning in Large Language Models" />
          <attvalue for="1" value="  We demonstrate that Contrastive Decoding -- a simple, computationally light,&#10;and training-free text generation method proposed by Li et al 2022 -- achieves&#10;large out-of-the-box improvements over greedy decoding on a variety of&#10;reasoning tasks. Originally shown to improve the perceived quality of long-form&#10;text generation, Contrastive Decoding searches for strings that maximize a&#10;weighted difference in likelihood between strong and weak models. We show that&#10;Contrastive Decoding leads LLaMA-65B to outperform LLaMA 2, GPT-3.5 and PaLM&#10;2-L on the HellaSwag commonsense reasoning benchmark, and to outperform LLaMA&#10;2, GPT-3.5 and PaLM-540B on the GSM8K math word reasoning benchmark, in&#10;addition to improvements on a collection of other tasks. Analysis suggests that&#10;Contrastive Decoding improves over existing methods by preventing some abstract&#10;reasoning errors, as well as by avoiding simpler modes such as copying sections&#10;of the input during chain-of-thought. Overall, Contrastive Decoding outperforms&#10;nucleus sampling for long-form generation and greedy decoding for reasoning&#10;tasks, making it a powerful general purpose method for generating text from&#10;language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.15097" label="2210.15097">
        <attvalues>
          <attvalue for="0" value="Contrastive Decoding: Open-ended Text Generation as Optimization" />
          <attvalue for="1" value="  Given a language model (LM), maximum probability is a poor decoding objective&#10;for open-ended generation, because it produces short and repetitive text. On&#10;the other hand, sampling can often produce incoherent text that drifts from the&#10;original topics. We propose contrastive decoding (CD), a reliable decoding&#10;approach that optimizes a contrastive objective subject to a plausibility&#10;constraint. The contrastive objective returns the difference between the&#10;likelihood under a large LM (called the expert, e.g. OPT-13B) and a small LM&#10;(called the amateur, e.g. OPT-125M), and the constraint ensures that the&#10;outputs are plausible. CD is inspired by the fact that the failures of larger&#10;LMs (e.g., repetition, incoherence) are even more prevalent in smaller LMs, and&#10;that this difference signals which texts should be preferred. CD requires zero&#10;additional training, and produces higher quality text than decoding from the&#10;larger LM alone. It also works across model scales (OPT-13B and GPT2-1.5B) and&#10;significantly outperforms four strong decoding algorithms (e.g., nucleus,&#10;top-k) in automatic and human evaluations across wikipedia, news and story&#10;domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.06362" label="1906.06362">
        <attvalues>
          <attvalue for="0" value="Comparison of Diverse Decoding Methods from Conditional Language Models" />
          <attvalue for="1" value="  While conditional language models have greatly improved in their ability to&#10;output high-quality natural language, many NLP applications benefit from being&#10;able to generate a diverse set of candidate sequences. Diverse decoding&#10;strategies aim to, within a given-sized candidate list, cover as much of the&#10;space of high-quality outputs as possible, leading to improvements for tasks&#10;that re-rank and combine candidate outputs. Standard decoding methods, such as&#10;beam search, optimize for generating high likelihood sequences rather than&#10;diverse ones, though recent work has focused on increasing diversity in these&#10;methods. In this work, we perform an extensive survey of decoding-time&#10;strategies for generating diverse outputs from conditional language models. We&#10;also show how diversity can be improved without sacrificing quality by&#10;over-sampling additional candidates, then filtering to the desired number.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.06417" label="2202.06417">
        <attvalues>
          <attvalue for="0" value="A Contrastive Framework for Neural Text Generation" />
          <attvalue for="1" value="  Text generation is of great importance to many natural language processing&#10;applications. However, maximization-based decoding methods (e.g. beam search)&#10;of neural language models often lead to degenerate solutions -- the generated&#10;text is unnatural and contains undesirable repetitions. Existing approaches&#10;introduce stochasticity via sampling or modify training objectives to decrease&#10;probabilities of certain tokens (e.g., unlikelihood training). However, they&#10;often lead to solutions that lack coherence. In this work, we show that an&#10;underlying reason for model degeneration is the anisotropic distribution of&#10;token representations. We present a contrastive solution: (i) SimCTG, a&#10;contrastive training objective to calibrate the model's representation space,&#10;and (ii) a decoding method -- contrastive search -- to encourage diversity&#10;while maintaining coherence in the generated text. Extensive experiments and&#10;analyses on three benchmarks from two languages demonstrate that our proposed&#10;approach significantly outperforms current state-of-the-art text generation&#10;methods as evaluated by both human and automatic metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12675" label="2305.12675">
        <attvalues>
          <attvalue for="0" value="A Frustratingly Simple Decoding Method for Neural Text Generation" />
          <attvalue for="1" value="  We introduce a frustratingly simple, super efficient and surprisingly&#10;effective decoding method, which we call Frustratingly Simple Decoding (FSD),&#10;for neural text generation. The idea behind FSD is straightforward: we build an&#10;anti-LM based on previously generated text and use this anti-LM to penalize&#10;future generation of what has been generated. The anti-LM can be implemented as&#10;simple as an n-gram language model or a vectorized variant. In this way, FSD&#10;introduces no extra model parameters and negligible computational overhead (FSD&#10;can be as fast as greedy search). Despite the simplicity, FSD is surprisingly&#10;effective; Experiments show that FSD can outperform the canonical methods to&#10;date (i.e., nucleus sampling) as well as several strong baselines that were&#10;proposed recently.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Recent years have witnessed enormous progress in neural text generation, particularly with the success of large LMs \cite{radford2019language}. The most straightforward heuristics for generating text from an LM is to find the most likely sequence estimated by the LM. Although maximizing the LM probabilities (e.g., greedy search and beam search) obtains excellent performance in close-ended text generation tasks (e.g., translation~\cite{sutskever2014sequence} and summarization~\cite{see-etal-2017-get}), these search-based methods suffer from generating nonsensical output in open-ended text generation tasks (e.g., story generation \cite{fan-etal-2018-hierarchical}). One prominent issue is that they tend to generate dull and repetitive output \cite{Holtzman2020The,fu2020a,pillutla2021mauve}.&#10;&#10;Decoding Methods&#10;To tackle the above challenge, different decoding methods have been proposed, which can be broadly categorized into two classes. The first class is truncated sampling, where each token is randomly sampled from a truncated next-token distribution. For instance, top-$k$ sampling~\cite{fan-etal-2018-hierarchical} only samples from the $k$ most likely tokens. Top-$p$ sampling~\cite{Holtzman2020The} only considers the minimal set of top tokens that cover a specified percentage $p$ of the distribution. Typical sampling ~\cite{meister2022typical} sorts tokens according to the differences between distribution entropy and probabilities. \cite{hewitt2022truncation} truncate words whose probabilities are below an entropy-dependent threshold. Although sampling-based methods reduce repetitions, the randomness at each sampling step also increases the chance of incoherence and topic drift.&#10;&#10;The second class of decoding methods is still search-based but optimizes a different objective. Contrastive Search (CS)~\cite{su2022a} assumes the LM has an isotropic representation space and adds a penalty term that decreases the generation probabilities of tokens producing hidden states that are similar to the previous context. However, the look-ahead operation at each step brings considerable additional cost. Contrastive Decoding (CD)~\cite{li2022contrastive} employs an amateur LM (a smaller pre-trained LM using the same tokenization) and penalizes undesired attributes associated with the amateur model. In contrast, FSD is much more lightweight and efficient; FSD only constructs an $n$-gram model on-the-fly, requiring no external model and introducing negligible computational cost. In addition, FSD holds the potential for broader applicability as it does not assume the existence of an amateur LM or any properties of the LM.&#10;&#10;Training Methods&#10;Another group of methods attempts to improve text generation quality by fine-tuning the LMs with new training objectives. \cite{Welleck2020Neural} propose unlikelihood training, which explicitly minimizes the generation probability of repetitive tokens. \cite{lagutin-etal-2021-implicit} improve the generation using policy gradient with a repetition objective.&#10;\cite{xu2022learning} learn to penalize probabilities of sentence-level repetitions from pseudo-repetitive data. \cite{su2022a} devise a contrastive training objective that encourages discriminative and isotropic token representations. In contrast, FSD simply employs off-the-shelf pre-trained LMs and requires zero training.&#10; " />
        </attvalues>
      </node>
      <node id="2202.00666" label="2202.00666">
        <attvalues>
          <attvalue for="0" value="Locally Typical Sampling" />
          <attvalue for="1" value="  Today's probabilistic language generators fall short when it comes to&#10;producing coherent and fluent text despite the fact that the underlying models&#10;perform well under standard metrics, e.g., perplexity. This discrepancy has&#10;puzzled the language generation community for the last few years. In this work,&#10;we posit that the abstraction of natural language generation as a discrete&#10;stochastic process--which allows for an information-theoretic analysis--can&#10;provide new insights into the behavior of probabilistic language generators,&#10;e.g., why high-probability texts can be dull or repetitive. Humans use language&#10;as a means of communicating information, aiming to do so in a simultaneously&#10;efficient and error-minimizing manner; in fact, psycholinguistics research&#10;suggests humans choose each word in a string with this subconscious goal in&#10;mind. We formally define the set of strings that meet this criterion: those for&#10;which each word has an information content close to the expected information&#10;content, i.e., the conditional entropy of our model. We then propose a simple&#10;and efficient procedure for enforcing this criterion when generating from&#10;probabilistic models, which we call locally typical sampling. Automatic and&#10;human evaluations show that, in comparison to nucleus and top-k sampling,&#10;locally typical sampling offers competitive performance (in both abstractive&#10;summarization and story generation) in terms of quality while consistently&#10;reducing degenerate repetitions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.01933" label="2304.01933">
        <attvalues>
          <attvalue for="0" value="LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of&#10;  Large Language Models" />
          <attvalue for="1" value="  The success of large language models (LLMs), like GPT-4 and ChatGPT, has led&#10;to the development of numerous cost-effective and accessible alternatives that&#10;are created by finetuning open-access LLMs with task-specific data (e.g.,&#10;ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning&#10;methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly&#10;one of the most attractive topics, as it only requires fine-tuning a few&#10;external parameters instead of the entire LLMs while achieving comparable or&#10;even better performance. To enable further research on PEFT methods of LLMs,&#10;this paper presents LLM-Adapters, an easy-to-use framework that integrates&#10;various adapters into LLMs and can execute these adapter-based PEFT methods of&#10;LLMs for different tasks. The framework includes state-of-the-art open-access&#10;LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as&#10;Series adapters, Parallel adapter, Prompt-based learning and&#10;Reparametrization-based methods. Moreover, we conduct extensive empirical&#10;studies on the impact of adapter types, placement locations, and&#10;hyper-parameters to the best design for each adapter-based methods. We evaluate&#10;the effectiveness of the adapters on fourteen datasets from two different&#10;reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning. The results&#10;demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few&#10;extra trainable parameters yields comparable, and in some cases superior,&#10;performance to powerful LLMs (175B) in zero-shot inference on both reasoning&#10;tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs), such as ChatGPT~\cite{openai-chatgpt-2022} and GPT-4~\cite{openai-gpt4-2023}, have demonstrated unprecedented performance across various natural language processing (NLP) tasks~\cite{qin-chatgpt-2023} and multi-modal tasks~\cite{hugginggpt}. These LLMs often possess sizes exceeding hundreds of billions of parameters and are closed-source. Consequently, this has spurred the development of accessible and cost-effective alternatives such as LLaMA~\cite{llama}.&#10;These alternatives involve fine-tuning open-source LLMs utilizing either task-specific data (e.g., ChatDoctor~\cite{yunxiang2023chatdoctor}) or instructional data (e.g., Alpaca~\cite{alpaca}). However, full-model fine-tuning (FFT) is computationally and storage-intensive, thereby presenting significant challenges in practical implementation.&#10;&#10;Prior to the emergence of FFT of LLMs (e.g., LLaMA), a compelling solution called parameter-efficient fine-tuning (PEFT)~\cite{adapters} has been proposed in the NLP field, specifically for pre-trained models (e.g., BERT~\cite{devlin2018bert}), offering a promising approach for efficiently fine-tuning LLMs. &#10;The advantage of PEFT lies in its ability to fine-tune only a small set of external parameters rather than the entire backbone model while still achieving comparable or even superior performance~\cite{peft}.&#10;Moreover, PEFT can effectively mitigate catastrophic forgetting in comparison to FFT~\cite{adamix}.&#10;As shown in Table~\ref{tab:peft_category}, the advantage of PEFT has resulted in the developing of diverse PEFT modules, encompassing series adapters~\cite{adapters, adamix, sparseadapter, lets}, parallel adapters~\cite{parallel_adapter}, reparameterization-based methods~\cite{lora, krona}, and prompt-based learning methods~\cite{prompt_tuning, prefix}.&#10;&#10;By incorporating these PEFT modules into backbone models (i.e., LLMs), we can capitalize on the remarkable capabilities of backbone models without requiring extensive computational resources. &#10;This opens up opportunities for a broader range of applications, enabling even those with limited access to high-performance computing to harness the power of LLMs in their specific tasks.&#10;Despite the success of PEFT for pre-trained models, it remains unclear which PEFT module, in combination with which layer and hyperparameter configuration, is most suitable for a given task or dataset when meeting LLMs (e.g., LLaMA~\cite{llama}). Therefore, further investigation is needed to determine the optimal PEFT setup that maximizes performance across different tasks and datasets.&#10;&#10;Motivated by this, in this paper, we conduct a comprehensive empirical study of PEFT of three representative open-source LLMs, including BLOOM~\cite{bloom}, GPT-J~\cite{gpt-j}, and LLaMA~\cite{llama}.&#10;Specifically, we undertake an empirical study to address the following three research questions: ($i$) What is the optimal placement and configuration of different PEFT methods? ($ii$) How's the performance of different adapters across downstream tasks? And ($iii$) What are the differences in performance between in-distribution (ID) and out-of-distribution (OOD) scenarios for PEFT methods? &#10;The findings of our study are as follows: &#10;\begin{enumerate}&#10; \item The optimal placement for the series adapter, parallel adapter, and LoRA is after the MLP layers, parallel with the MLP layers, and located after both the Attention layers and MLP layers simultaneously, respectively;&#10; \item Smaller language models with the PEFT approach can attain competitive or superior performance on specific tasks compared to larger language models.&#10; For instance, LLaMA-13B with LoRA can outperform GPT-3.5 (&gt;175B) on MultiArith, AddSub, and SingleEq&#10; ; &#10; \item The ID fine-tuned LLaMA-13B with adapters outperforms ChatGPT on commonsense reasoning tasks indicating that smaller language models have the potential to outperform larger language models on specific tasks with ID fine-tunig data. &#10;\end{enumerate}&#10;&#10;Our contributions can be summarized as follows:&#10;\begin{itemize}&#10; \item We conduct a comprehensive empirical study of various PEFT methods applied in different open-source LLMs. &#10; \item To facilitate our empirical study, we construct two high-quality training datasets to enhance PEFT performance in math reasoning and commonsense reasoning tasks. &#10; \item We develop a user-friendly framework, LLM-Adapter, seamlessly integrates diverse adapters into LLMs, empowering researchers to implement adapter-based PEFT methods for a wide range of tasks. &#10; \item We conduct extensive experiments to answer the three research questions to serve as inspiration for future research. \end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Adapter-based Methods, Mathematics, Language Model Fine-tuning" />
        </attvalues>
      </node>
      <node id="2302.06476" label="2302.06476">
        <attvalues>
          <attvalue for="0" value="Is ChatGPT a General-Purpose Natural Language Processing Task Solver?" />
          <attvalue for="1" value="  Spurred by advancements in scale, large language models (LLMs) have&#10;demonstrated the ability to perform a variety of natural language processing&#10;(NLP) tasks zero-shot -- i.e., without adaptation on downstream data. Recently,&#10;the debut of ChatGPT has drawn a great deal of attention from the natural&#10;language processing (NLP) community due to the fact that it can generate&#10;high-quality responses to human input and self-correct previous mistakes based&#10;on subsequent conversations. However, it is not yet known whether ChatGPT can&#10;serve as a generalist model that can perform many NLP tasks zero-shot. In this&#10;work, we empirically analyze the zero-shot learning ability of ChatGPT by&#10;evaluating it on 20 popular NLP datasets covering 7 representative task&#10;categories. With extensive empirical studies, we demonstrate both the&#10;effectiveness and limitations of the current version of ChatGPT. We find that&#10;ChatGPT performs well on many tasks favoring reasoning capabilities (e.g.,&#10;arithmetic reasoning) while it still faces challenges when solving specific&#10;tasks such as sequence tagging. We additionally provide in-depth analysis&#10;through qualitative case studies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;This work mainly explores the zero-shot learning capability of ChatGPT on a diverse collection of datasets including reasoning and classic NLP tasks. In light of this, we review three lines of research that form the basis of this work: large language models, zero-shot learning, and chain-of-thought prompting for reasoning. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="1902.00751" label="1902.00751">
        <attvalues>
          <attvalue for="0" value="Parameter-Efficient Transfer Learning for NLP" />
          <attvalue for="1" value="  Fine-tuning large pre-trained models is an effective transfer mechanism in&#10;NLP. However, in the presence of many downstream tasks, fine-tuning is&#10;parameter inefficient: an entire new model is required for every task. As an&#10;alternative, we propose transfer with adapter modules. Adapter modules yield a&#10;compact and extensible model; they add only a few trainable parameters per&#10;task, and new tasks can be added without revisiting previous ones. The&#10;parameters of the original network remain fixed, yielding a high degree of&#10;parameter sharing. To demonstrate adapter's effectiveness, we transfer the&#10;recently proposed BERT Transformer model to 26 diverse text classification&#10;tasks, including the GLUE benchmark. Adapters attain near state-of-the-art&#10;performance, whilst adding only a few parameters per task. On GLUE, we attain&#10;within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters&#10;per task. By contrast, fine-tuning trains 100% of the parameters per task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04366" label="2110.04366">
        <attvalues>
          <attvalue for="0" value="Towards a Unified View of Parameter-Efficient Transfer Learning" />
          <attvalue for="1" value="  Fine-tuning large pre-trained language models on downstream tasks has become&#10;the de-facto learning paradigm in NLP. However, conventional approaches&#10;fine-tune all the parameters of the pre-trained model, which becomes&#10;prohibitive as the model size and the number of tasks grow. Recent work has&#10;proposed a variety of parameter-efficient transfer learning methods that only&#10;fine-tune a small number of (extra) parameters to attain strong performance.&#10;While effective, the critical ingredients for success and the connections among&#10;the various methods are poorly understood. In this paper, we break down the&#10;design of state-of-the-art parameter-efficient transfer learning methods and&#10;present a unified framework that establishes connections between them.&#10;Specifically, we re-frame them as modifications to specific hidden states in&#10;pre-trained models, and define a set of design dimensions along which different&#10;methods vary, such as the function to compute the modification and the position&#10;to apply the modification. Through comprehensive empirical studies across&#10;machine translation, text summarization, language understanding, and text&#10;classification benchmarks, we utilize the unified view to identify important&#10;design choices in previous methods. Furthermore, our unified framework enables&#10;the transfer of design elements across different approaches, and as a result we&#10;are able to instantiate new parameter-efficient fine-tuning methods that tune&#10;less parameters than previous methods while being more effective, achieving&#10;comparable results to fine-tuning all parameters on all four tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transfer learning from pre-trained language models (PLMs) is now the prevalent paradigm in natural language processing, yielding strong performance on many tasks~\cite{peters2018deep,devlin2019bert,qiu2020pre}. The most common way to adapt general-purpose PLMs to downstream tasks is to fine-tune all the model parameters (full fine-tuning). However, this results in a separate copy of fine-tuned model parameters for each task, which is prohibitively expensive when serving models that perform a large number of tasks. This issue is particularly salient with the ever-increasing size of PLMs, which now range from hundreds of millions~\cite{radford2019language,lewis-etal-2020-bart} to hundreds of billions~\cite{brown2020language} or even trillions of parameters~\cite{fedus2021switch}.&#10;&#10;To mitigate this issue, a few lightweight alternatives have been proposed to update only a small number of extra parameters while keeping most pretrained parameters frozen. &#10;For example, adapter tuning~\cite{houlsby2019parameter} inserts small neural modules called adapters to each layer of the pretrained network and only the adapters are trained at fine-tuning time. &#10;Inspired by the success of prompting methods that control PLMs through textual prompts~\cite{brown2020language,liu2021pre}, prefix tuning~\cite{li2021prefix} and prompt tuning~\cite{lester2021power} prepend an additional $l$ tunable prefix tokens to the input or hidden layers and only train these soft prompts when fine-tuning on downstream tasks. More recently,~\cite{hu2021lora} learn low-rank matrices to approximate parameter updates. We illustrate these methods in Figure~\ref{fig:intro-model}. These approaches have all been reported to demonstrate comparable performance to full fine-tuning on different sets of tasks, often through updating less than 1\% of the original model parameters. Besides parameter savings, parameter-efficient tuning makes it possible to quickly adapt to new tasks without catastrophic forgetting~\cite{pfeiffer2021adapterfusion} and often exhibits superior robustness in out-of-distribution evaluation~\cite{li2021prefix}. &#10;&#10;However, we contend that the important ingredients that contribute to the success of these parameter-efficient tuning methods are poorly understood, and the connections between them are still unclear. In this paper, we aim to answer three questions: (1) How are these methods connected? (2) Do these methods share design elements that are essential for their effectiveness, and what are they?&#10;(3) Can the effective ingredients of each method be transferred to others to yield more effective variants? &#10;&#10;In order to answer these questions, we first derive an alternative form of prefix tuning that reveals prefix tuning's close connections with adapters (\textsection\ref{sec:connection}).&#10;Based on this we then devise a unified framework that frames&#10;the aforementioned methods as different ways to modify the hidden representations of frozen PLMs (\textsection\ref{sec:unify}).&#10;Our unified framework decomposes previous methods along a shared set of design dimensions, such as the function used to perform the modification, the position in which to impose this modification, and how to integrate the modification.&#10;This framework allows us to transfer design choices across approaches to propose new variants &#10;such as adapters with multiple heads (\textsection\ref{sec:transfer}). In experiments, we first show that existing parameter-efficient tuning methods still lag behind full fine-tuning on higher-resource and challenging tasks (\textsection\ref{sec:exp-previous-res}), as exemplified in Figure~\ref{fig:intro-res}. Then we utilize the unified framework to identify critical design choices and validate the proposed variants empirically (\textsection\ref{sec:exp:insertion}-\ref{sec:combine}). Our experiments on four NLP benchmarks covering text summarization, machine translation (MT), text classification, and general language understanding, &#10;demonstrate that the proposed variant uses less parameters than existing methods while being more effective, matching full fine-tuning results on all four tasks.&#10;&#10;\end{minipage}&#10;&#10;\end{figure}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.05119" label="2004.05119">
        <attvalues>
          <attvalue for="0" value="Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer" />
          <attvalue for="1" value="  Fine-tuning (FT) pre-trained sentence embedding models on small datasets has&#10;been shown to have limitations. In this paper we show that concatenating the&#10;embeddings from the pre-trained model with those from a simple sentence&#10;embedding model trained only on the target data, can improve over the&#10;performance of FT for few-sample tasks. To this end, a linear classifier is&#10;trained on the combined embeddings, either by freezing the embedding model&#10;weights or training the classifier and embedding models end-to-end. We perform&#10;evaluation on seven small datasets from NLP tasks and show that our approach&#10;with end-to-end training outperforms FT with negligible computational overhead.&#10;Further, we also show that sophisticated combination techniques like CCA and&#10;KCCA do not work as well in practice as concatenation. We provide theoretical&#10;analysis to explain this empirical observation.&#10;" />
          <attvalue for="2" value="&#10;Fine-tuning (FT) powerful pre-trained sentence embedding models like BERT~\cite{devlin2018bert} has recently become the de-facto standard for downstream NLP tasks. &#10;Typically, FT entails jointly learning a classifier over the pre-trained model while tuning the weights of the latter. &#10;While FT has been shown to improve performance on tasks like GLUE~\cite{wang-etal-2018-glue} having large datasets (QQP, MNLI, QNLI), similar trends have not been observed on small datasets, where one would expect the maximum benefits of using a pre-trained model.&#10;Several works~\cite{phang2018sentence,garg2019tanda,dodge2020finetuning,Lee2020Mixout} have demonstrated that FT with a few target domain samples is unstable with high variance, thereby often leading to sub-par gains.&#10;Furthermore, this issue has also been well documented in practice. &#10;&#10;Learning with low resources has recently become an active research area in NLP, and arguably one of the most interesting scenarios for which pre-trained models are useful (e.g.,~\cite{emnlp-2019-deep}). Many practical applications have small datasets (e.g., in social science, medical studies, etc), which are different from large-scale academic benchmarks having hundreds of thousands of training samples (e.g, DBpedia~\cite{dbpedia-swj}, Sogou News~\cite{10.1145/1367497.1367560}, etc). &#10;This necessitates effective transfer learning approaches using pre-trained sentence embedding models for few-sample tasks.&#10;&#10;In this work, we show that concatenating sentence embeddings from a pre-trained model and those from a smaller model trained solely on the target data, can improve over the performance of FT.&#10;Specifically, we first learn a simple sentence embedding model on the target data. Then we concatenate(C AT) the embeddings from this model with those from a pre-trained model, and train a linear classifier on the combined representation. &#10;The latter can be done by either freezing the embedding model weights or training the whole network (classifier plus the two embedding models) end-to-end.&#10;&#10;We evaluate our approach on seven small datasets from NLP tasks. Our results show that our approach with end-to-end training can significantly improve the prediction performance of FT, with less than a $10\%$ increase in the run time. Furthermore, our approach with frozen embedding models performs better than FT for very small datasets while reducing the run time by $30\%{-}50\%$, and without the requirement of large memory GPUs. &#10;&#10;We also conduct evaluations of multiple techniques for combining the pre-trained and domain-specific embeddings, comparing concatenation to CCA and KCCA. We observe that the simplest approach of concatenation works best in practice. &#10;Moreover, we provide theoretical analysis to explain this empirical observation.&#10;&#10;Finally, our results also have implications on the semantics learning ability of small domain-specific models compared to large pre-trained models.&#10;While intuition dictates that a large pre-trained model should capture the entire semantics learned by a small domain-specific model, our results show that there exist semantic features captured solely by the latter and not by the former, in spite of pre-training on billions of words. Hence combining the embeddings can improve the performance of directly FT the pre-trained model.&#10;&#10;Related Work &#10;Recently, several pre-trained models have been studied, of which some provide explicit sentence embeddings~\cite{conneau2017supervised,subramanian2018learning}, while others provide implicit ones~\cite{howard2018universal,radford2018improving}.&#10;\cite{peters2019tune} compare the performance of feature extraction (by freezing the pre-trained weights) and FT.&#10;There exists other more sophisticated transferring methods, but they are typically much more expensive or complicated. For example, \cite{xu2019bert} ``post-train&quot; the pre-trained model on the target dataset, \cite{houlsby2019parameter} inject specifically designed new adapter layers, \cite{arase-tsujii-2019-transfer} inject phrasal paraphrase relations into BERT, \cite{DBLP:journals/corr/abs-1905-05583} use multi-task FT, and \cite{wang2019to} first train a deep network classifier on the fixed pre-trained embedding and then fine-tune it. Our focus is to propose alternatives to FT with similar simplicity and computational efficiency, and study conditions where it has significant advantages. &#10;While the idea of concatenating multiple embeddings has been previously used~\cite{peters2018deep}, we use it for transfer learning in a low resource target domain. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Mathematics, Sentence Embedding Models, Few-Sample Learning, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1811.01088" label="1811.01088">
        <attvalues>
          <attvalue for="0" value="Sentence Encoders on STILTs: Supplementary Training on Intermediate&#10;  Labeled-data Tasks" />
          <attvalue for="1" value="  Pretraining sentence encoders with language modeling and related unsupervised&#10;tasks has recently been shown to be very effective for language understanding&#10;tasks. By supplementing language model-style pretraining with further training&#10;on data-rich supervised tasks, such as natural language inference, we obtain&#10;additional performance improvements on the GLUE benchmark. Applying&#10;supplementary training on BERT (Devlin et al., 2018), we attain a GLUE score of&#10;81.8---the state of the art (as of 02/24/2019) and a 1.4 point improvement over&#10;BERT. We also observe reduced variance across random restarts in this setting.&#10;Our approach yields similar improvements when applied to ELMo (Peters et al.,&#10;2018a) and Radford et al. (2018)'s model. In addition, the benefits of&#10;supplementary training are particularly pronounced in data-constrained regimes,&#10;as we show in experiments with artificially limited training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.04118" label="1911.04118">
        <attvalues>
          <attvalue for="0" value="TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer&#10;  Sentence Selection" />
          <attvalue for="1" value="  We propose TANDA, an effective technique for fine-tuning pre-trained&#10;Transformer models for natural language tasks. Specifically, we first transfer&#10;a pre-trained model into a model for a general task by fine-tuning it with a&#10;large and high-quality dataset. We then perform a second fine-tuning step to&#10;adapt the transferred model to the target domain. We demonstrate the benefits&#10;of our approach for answer sentence selection, which is a well-known inference&#10;task in Question Answering. We built a large scale dataset to enable the&#10;transfer step, exploiting the Natural Questions dataset. Our approach&#10;establishes the state of the art on two well-known benchmarks, WikiQA and&#10;TREC-QA, achieving MAP scores of 92% and 94.3%, respectively, which largely&#10;outperform the previous highest scores of 83.4% and 87.5%, obtained in very&#10;recent work. We empirically show that TANDA generates more stable and robust&#10;models reducing the effort required for selecting optimal hyper-parameters.&#10;Additionally, we show that the transfer step of TANDA makes the adaptation step&#10;more robust to noise. This enables a more effective use of noisy datasets for&#10;fine-tuning. Finally, we also confirm the positive impact of TANDA in an&#10;industrial setting, using domain specific datasets subject to different types&#10;of noise.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.06305" label="2002.06305">
        <attvalues>
          <attvalue for="0" value="Fine-Tuning Pretrained Language Models: Weight Initializations, Data&#10;  Orders, and Early Stopping" />
          <attvalue for="1" value="  Fine-tuning pretrained contextual word embedding models to supervised&#10;downstream tasks has become commonplace in natural language processing. This&#10;process, however, is often brittle: even with the same hyperparameter values,&#10;distinct random seeds can lead to substantially different results. To better&#10;understand this phenomenon, we experiment with four datasets from the GLUE&#10;benchmark, fine-tuning BERT hundreds of times on each while varying only the&#10;random seeds. We find substantial performance increases compared to previously&#10;reported results, and we quantify how the performance of the best-found model&#10;varies as a function of the number of fine-tuning trials. Further, we examine&#10;two factors influenced by the choice of random seed: weight initialization and&#10;training data order. We find that both contribute comparably to the variance of&#10;out-of-sample performance, and that some weight initializations perform well&#10;across all tasks explored. On small datasets, we observe that many fine-tuning&#10;trials diverge part of the way through training, and we offer best practices&#10;for practitioners to stop training less promising runs early. We publicly&#10;release all of our experimental data, including training and validation scores&#10;for 2,100 trials, to encourage further analysis of training dynamics during&#10;fine-tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.02364" label="1705.02364">
        <attvalues>
          <attvalue for="0" value="Supervised Learning of Universal Sentence Representations from Natural&#10;  Language Inference Data" />
          <attvalue for="1" value="  Many modern NLP systems rely on word embeddings, previously trained in an&#10;unsupervised manner on large corpora, as base features. Efforts to obtain&#10;embeddings for larger chunks of text, such as sentences, have however not been&#10;so successful. Several attempts at learning unsupervised representations of&#10;sentences have not reached satisfactory enough performance to be widely&#10;adopted. In this paper, we show how universal sentence representations trained&#10;using the supervised data of the Stanford Natural Language Inference datasets&#10;can consistently outperform unsupervised methods like SkipThought vectors on a&#10;wide range of transfer tasks. Much like how computer vision uses ImageNet to&#10;obtain features, which can then be transferred to other tasks, our work tends&#10;to indicate the suitability of natural language inference for transfer learning&#10;to other NLP tasks. Our encoder is publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.00079" label="1804.00079">
        <attvalues>
          <attvalue for="0" value="Learning General Purpose Distributed Sentence Representations via Large&#10;  Scale Multi-task Learning" />
          <attvalue for="1" value="  A lot of the recent success in natural language processing (NLP) has been&#10;driven by distributed vector representations of words trained on large amounts&#10;of text in an unsupervised manner. These representations are typically used as&#10;general purpose features for words across a range of NLP problems. However,&#10;extending this success to learning representations of sequences of words, such&#10;as sentences, remains an open problem. Recent work has explored unsupervised as&#10;well as supervised learning techniques with different training objectives to&#10;learn general purpose fixed-length sentence representations. In this work, we&#10;present a simple, effective multi-task learning framework for sentence&#10;representations that combines the inductive biases of diverse training&#10;objectives in a single model. We train this model on several data sources with&#10;multiple training objectives on over 100 million sentences. Extensive&#10;experiments demonstrate that sharing a single recurrent sentence encoder across&#10;weakly related tasks leads to consistent improvements over previous methods. We&#10;present substantial improvements in the context of transfer learning and&#10;low-resource settings using our learned general-purpose representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.05987" label="1903.05987">
        <attvalues>
          <attvalue for="0" value="To Tune or Not to Tune? Adapting Pretrained Representations to Diverse&#10;  Tasks" />
          <attvalue for="1" value="  While most previous work has focused on different pretraining objectives and&#10;architectures for transfer learning, we ask how to best adapt the pretrained&#10;model to a given target task. We focus on the two most common forms of&#10;adaptation, feature extraction (where the pretrained weights are frozen), and&#10;directly fine-tuning the pretrained model. Our empirical results across diverse&#10;NLP tasks with two state-of-the-art models show that the relative performance&#10;of fine-tuning vs. feature extraction depends on the similarity of the&#10;pretraining and target tasks. We explore possible explanations for this finding&#10;and provide a set of adaptation guidelines for the NLP practitioner.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.02232" label="1904.02232">
        <attvalues>
          <attvalue for="0" value="BERT Post-Training for Review Reading Comprehension and Aspect-based&#10;  Sentiment Analysis" />
          <attvalue for="1" value="  Question-answering plays an important role in e-commerce as it allows&#10;potential customers to actively seek crucial information about products or&#10;services to help their purchase decision making. Inspired by the recent success&#10;of machine reading comprehension (MRC) on formal documents, this paper explores&#10;the potential of turning customer reviews into a large source of knowledge that&#10;can be exploited to answer user questions.~We call this problem Review Reading&#10;Comprehension (RRC). To the best of our knowledge, no existing work has been&#10;done on RRC. In this work, we first build an RRC dataset called ReviewRC based&#10;on a popular benchmark for aspect-based sentiment analysis. Since ReviewRC has&#10;limited training examples for RRC (and also for aspect-based sentiment&#10;analysis), we then explore a novel post-training approach on the popular&#10;language model BERT to enhance the performance of fine-tuning of BERT for RRC.&#10;To show the generality of the approach, the proposed post-training is also&#10;applied to some other review-based tasks such as aspect extraction and aspect&#10;sentiment classification in aspect-based sentiment analysis. Experimental&#10;results demonstrate that the proposed post-training is highly effective. The&#10;datasets and code are available at https://www.cs.uic.edu/~hxu/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.00931" label="1909.00931">
        <attvalues>
          <attvalue for="0" value="Transfer Fine-Tuning: A BERT Case Study" />
          <attvalue for="1" value="  A semantic equivalence assessment is defined as a task that assesses semantic&#10;equivalence in a sentence pair by binary judgment (i.e., paraphrase&#10;identification) or grading (i.e., semantic textual similarity measurement). It&#10;constitutes a set of tasks crucial for research on natural language&#10;understanding. Recently, BERT realized a breakthrough in sentence&#10;representation learning (Devlin et al., 2019), which is broadly transferable to&#10;various NLP tasks. While BERT's performance improves by increasing its model&#10;size, the required computational power is an obstacle preventing practical&#10;applications from adopting the technology. Herein, we propose to inject phrasal&#10;paraphrase relations into BERT in order to generate suitable representations&#10;for semantic equivalence assessment instead of increasing the model size.&#10;Experiments on standard natural language understanding tasks confirm that our&#10;method effectively improves a smaller BERT model while maintaining the model&#10;size. The generated model exhibits superior performance compared to a larger&#10;BERT model on semantic equivalence assessment tasks. Furthermore, it achieves&#10;larger performance gains on tasks with limited training datasets for&#10;fine-tuning, which is a property desirable for transfer learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.05583" label="1905.05583">
        <attvalues>
          <attvalue for="0" value="How to Fine-Tune BERT for Text Classification?" />
          <attvalue for="1" value="  Language model pre-training has proven to be useful in learning universal&#10;language representations. As a state-of-the-art language model pre-training&#10;model, BERT (Bidirectional Encoder Representations from Transformers) has&#10;achieved amazing results in many language understanding tasks. In this paper,&#10;we conduct exhaustive experiments to investigate different fine-tuning methods&#10;of BERT on text classification task and provide a general solution for BERT&#10;fine-tuning. Finally, the proposed solution obtains new state-of-the-art&#10;results on eight widely-studied text classification datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.05338" label="1907.05338">
        <attvalues>
          <attvalue for="0" value="To Tune or Not To Tune? How About the Best of Both Worlds?" />
          <attvalue for="1" value="  The introduction of pre-trained language models has revolutionized natural&#10;language research communities. However, researchers still know relatively&#10;little regarding their theoretical and empirical properties. In this regard,&#10;Peters et al. perform several experiments which demonstrate that it is better&#10;to adapt BERT with a light-weight task-specific head, rather than building a&#10;complex one on top of the pre-trained language model, and freeze the parameters&#10;in the said language model. However, there is another option to adopt. In this&#10;paper, we propose a new adaptation method which we first train the task model&#10;with the BERT parameters frozen and then fine-tune the entire model together.&#10;Our experimental results show that our model adaptation method can achieve 4.7%&#10;accuracy improvement in semantic similarity task, 0.99% accuracy improvement in&#10;sequence labeling task and 0.72% accuracy improvement in the text&#10;classification task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.02486" label="2307.02486">
        <attvalues>
          <attvalue for="0" value="LongNet: Scaling Transformers to 1,000,000,000 Tokens" />
          <attvalue for="1" value="  Scaling sequence length has become a critical demand in the era of large&#10;language models. However, existing methods struggle with either computational&#10;complexity or model expressivity, rendering the maximum sequence length&#10;restricted. To address this issue, we introduce LongNet, a Transformer variant&#10;that can scale sequence length to more than 1 billion tokens, without&#10;sacrificing the performance on shorter sequences. Specifically, we propose&#10;dilated attention, which expands the attentive field exponentially as the&#10;distance grows. LongNet has significant advantages: 1) it has a linear&#10;computation complexity and a logarithm dependency between any two tokens in a&#10;sequence; 2) it can be served as a distributed trainer for extremely long&#10;sequences; 3) its dilated attention is a drop-in replacement for standard&#10;attention, which can be seamlessly integrated with the existing&#10;Transformer-based optimization. Experiments results demonstrate that LongNet&#10;yields strong performance on both long-sequence modeling and general language&#10;tasks. Our work opens up new possibilities for modeling very long sequences,&#10;e.g., treating a whole corpus or even the entire Internet as a sequence.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years have witnessed a trend toward scaling neural networks~\cite{gpt3,scalinglaw,scalingvit,palm,vit22b}. The depth is primarily scaled up for exponential expressivity, producing many powerful deep networks~\cite{resnet,gpipe,deepnet}. Then, the sparse MoE models~\cite{gshard,switch,stmoe} and model parallelism approaches~\cite{megatron,reduceact} efficiently enlarge the hidden dimension.&#10;Sequence length, as the last atomic dimension of the neural network, is desirable to be unlimited.&#10;Breaking the limitation of sequence length introduces significant advantages. First, it provides large memory and receptive field for models, which is practical for them to interact with human and the world. Second, a longer context contains more complex causality and reasoning paths that models can exploit in training data. In contrast, short dependency has more spurious correlations, which is harmful to generalization. Third, it enables to explore the limits of in-context learning, which has the potential to be a paradigm shift for many-shot learning, as an extremely long context may help the models alleviate catastrophic forgetting.&#10;&#10;The major challenge of scaling up sequence length is striking the right balance between the computational complexity and the model expressivity. RNN-style models are primarily implemented to increase the length. However, its sequential nature limits the parallelization during training, which is essential in long-sequence modeling. More recently, state space models~\cite{s4,s5,h3,hyena} are appealing to sequence modeling. It can operate as a CNN during training, and transform to an efficient RNN at test time. While they perform well at long-range benchmarks~\cite{lra}, their performance on regular lengths is not as good as Transformers, limited mainly by the model expressivity~\cite{blockstate}. &#10;&#10;Another strand of scaling the sequence length is to decrease the complexity of Transformers, i.e., the quadratic complexity of self-attention. Implementing sliding windows or convolution modules over the attention is a straightforward way to make the complexity nearly linear. Nevertheless, this sacrifices the ability to recall the early tokens, forgetting the prompts at the very beginning of the sequence. Sparse attention reduces the computation by sparsifying the attention matrix, preserving the possibility of recalling long-distant information. For example, \cite{sparsetransformer} obtains $\mathcal{O}(N\sqrt{N}d)$ time complexity with a fixed sparse pattern. Besides the heuristic patterns~\cite{bigbird,longformer}, the learnable patterns prove to be useful for sparse attention~\cite{reformer,colt5}. There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}. Yet, none has been scaled to 1 billion tokens (see~\ref{fig:trend}).&#10;&#10;In this work, we successfully scale the sequence length to 1 billion tokens. Our solution is \ours, which replaces the attention of vanilla Transformers with a novel component named dilated attention. The general design principle is - attention allocation decreases exponentially as the distance between tokens grows. We prove that it obtains a linear computation complexity and a logarithm dependency between tokens. This deals with the contradiction between limited attention resources and the accessibility to every token. In the implementation, \our can be transformed into a dense Transformer, which seamlessly supports the off-the-shelf optimization for Transformers (e.g., kernel fusion, quantization, and distributed training).&#10;Taking advantage of the linear complexity, \our can parallelize the training across nodes, breaking the constraint of both computation and memory with a distributed algorithm. This allows us to efficiently scale up the sequence length to 1B tokens with nearly constant runtime (see \ref{fig:runtime}), while vanilla Transformer suffers from quadratic complexity.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Sequence Length Scaling, Computer Science, Large Language Models, Linguistics, Mathematics, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="2106.04560" label="2106.04560">
        <attvalues>
          <attvalue for="0" value="Scaling Vision Transformers" />
          <attvalue for="1" value="  Attention-based neural networks such as the Vision Transformer (ViT) have&#10;recently attained state-of-the-art results on many computer vision benchmarks.&#10;Scale is a primary ingredient in attaining excellent results, therefore,&#10;understanding a model's scaling properties is a key to designing future&#10;generations effectively. While the laws for scaling Transformer language models&#10;have been studied, it is unknown how Vision Transformers scale. To address&#10;this, we scale ViT models and data, both up and down, and characterize the&#10;relationships between error rate, data, and compute. Along the way, we refine&#10;the architecture and training of ViT, reducing memory consumption and&#10;increasing accuracy of the resulting models. As a result, we successfully train&#10;a ViT model with two billion parameters, which attains a new state-of-the-art&#10;on ImageNet of 90.45% top-1 accuracy. The model also performs well for few-shot&#10;transfer, for example, reaching 84.86% top-1 accuracy on ImageNet with only 10&#10;examples per class.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.05442" label="2302.05442">
        <attvalues>
          <attvalue for="0" value="Scaling Vision Transformers to 22 Billion Parameters" />
          <attvalue for="1" value="  The scaling of Transformers has driven breakthrough capabilities for language&#10;models. At present, the largest large language models (LLMs) contain upwards of&#10;100B parameters. Vision Transformers (ViT) have introduced the same&#10;architecture to image and video modelling, but these have not yet been&#10;successfully scaled to nearly the same degree; the largest dense ViT contains&#10;4B parameters (Chen et al., 2022). We present a recipe for highly efficient and&#10;stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of&#10;experiments on the resulting model. When evaluated on downstream tasks (often&#10;with a lightweight linear model on frozen features), ViT-22B demonstrates&#10;increasing performance with scale. We further observe other interesting&#10;benefits of scale, including an improved tradeoff between fairness and&#10;performance, state-of-the-art alignment to human visual perception in terms of&#10;shape/texture bias, and improved robustness. ViT-22B demonstrates the potential&#10;for &quot;LLM-like&quot; scaling in vision, and provides key steps towards getting there.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.06965" label="1811.06965">
        <attvalues>
          <attvalue for="0" value="GPipe: Efficient Training of Giant Neural Networks using Pipeline&#10;  Parallelism" />
          <attvalue for="1" value="  Scaling up deep neural network capacity has been known as an effective&#10;approach to improving model quality for several different machine learning&#10;tasks. In many cases, increasing model capacity beyond the memory limit of a&#10;single accelerator has required developing special algorithms or&#10;infrastructure. These solutions are often architecture-specific and do not&#10;transfer to other tasks. To address the need for efficient and task-independent&#10;model parallelism, we introduce GPipe, a pipeline parallelism library that&#10;allows scaling any network that can be expressed as a sequence of layers. By&#10;pipelining different sub-sequences of layers on separate accelerators, GPipe&#10;provides the flexibility of scaling a variety of different networks to gigantic&#10;sizes efficiently. Moreover, GPipe utilizes a novel batch-splitting pipelining&#10;algorithm, resulting in almost linear speedup when a model is partitioned&#10;across multiple accelerators. We demonstrate the advantages of GPipe by&#10;training large-scale neural networks on two different tasks with distinct&#10;network architectures: (i) Image Classification: We train a&#10;557-million-parameter AmoebaNet model and attain a top-1 accuracy of 84.4% on&#10;ImageNet-2012, (ii) Multilingual Neural Machine Translation: We train a single&#10;6-billion-parameter, 128-layer Transformer model on a corpus spanning over 100&#10;languages and achieve better quality than all bilingual models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.00555" label="2203.00555">
        <attvalues>
          <attvalue for="0" value="DeepNet: Scaling Transformers to 1,000 Layers" />
          <attvalue for="1" value="  In this paper, we propose a simple yet effective method to stabilize&#10;extremely deep Transformers. Specifically, we introduce a new normalization&#10;function (DeepNorm) to modify the residual connection in Transformer,&#10;accompanying with theoretically derived initialization. In-depth theoretical&#10;analysis shows that model updates can be bounded in a stable way. The proposed&#10;method combines the best of two worlds, i.e., good performance of Post-LN and&#10;stable training of Pre-LN, making DeepNorm a preferred alternative. We&#10;successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and&#10;feed-forward network sublayers) without difficulty, which is one order of&#10;magnitude deeper than previous deep Transformers. Remarkably, on a multilingual&#10;benchmark with 7,482 translation directions, our 200-layer model with 3.2B&#10;parameters significantly outperforms the 48-layer state-of-the-art model with&#10;12B parameters by 5 BLEU points, which indicates a promising scaling direction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.05198" label="2205.05198">
        <attvalues>
          <attvalue for="0" value="Reducing Activation Recomputation in Large Transformer Models" />
          <attvalue for="1" value="  Training large transformer models is one of the most important computational&#10;challenges of modern AI. In this paper, we show how to significantly accelerate&#10;training of large transformer models by reducing activation recomputation.&#10;Activation recomputation is commonly used to work around memory capacity&#10;constraints. Rather than storing activations for backpropagation, they are&#10;traditionally recomputed, which saves memory but adds redundant compute. In&#10;this work, we show most of this redundant compute is unnecessary because we can&#10;reduce memory consumption sufficiently without it. We present two novel yet&#10;very simple techniques: sequence parallelism and selective activation&#10;recomputation. In conjunction with tensor parallelism, these techniques almost&#10;eliminate the need to recompute activations. We evaluate our approach on&#10;language models up to one trillion parameters in scale and show that our method&#10;reduces activation memory by 5x, while reducing execution time overhead from&#10;activation recomputation by over 90%. For example, when training a 530B&#10;parameter GPT-3 style model on 2240 NVIDIA A100 GPUs, we achieve a Model Flops&#10;Utilization of 54.2%, which is 29% faster than the 42.1% we achieve using&#10;recomputation. Our implementation will be available in both Megatron-LM and&#10;NeMo-Megatron.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.04933" label="2208.04933">
        <attvalues>
          <attvalue for="0" value="Simplified State Space Layers for Sequence Modeling" />
          <attvalue for="1" value="  Models using structured state space sequence (S4) layers have achieved&#10;state-of-the-art performance on long-range sequence modeling tasks. An S4 layer&#10;combines linear state space models (SSMs), the HiPPO framework, and deep&#10;learning to achieve high performance. We build on the design of the S4 layer&#10;and introduce a new state space layer, the S5 layer. Whereas an S4 layer uses&#10;many independent single-input, single-output SSMs, the S5 layer uses one&#10;multi-input, multi-output SSM. We establish a connection between S5 and S4, and&#10;use this to develop the initialization and parameterization used by the S5&#10;model. The result is a state space layer that can leverage efficient and widely&#10;implemented parallel scans, allowing S5 to match the computational efficiency&#10;of S4, while also achieving state-of-the-art performance on several long-range&#10;sequence modeling tasks. S5 averages 87.4% on the long range arena benchmark,&#10;and 98.5% on the most difficult Path-X task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.10509" label="1904.10509">
        <attvalues>
          <attvalue for="0" value="Generating Long Sequences with Sparse Transformers" />
          <attvalue for="1" value="  Transformers are powerful sequence models, but require time and memory that&#10;grows quadratically with the sequence length. In this paper we introduce sparse&#10;factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We&#10;also introduce a) a variation on architecture and initialization to train&#10;deeper networks, b) the recomputation of attention matrices to save memory, and&#10;c) fast attention kernels for training. We call networks with these changes&#10;Sparse Transformers, and show they can model sequences tens of thousands of&#10;timesteps long using hundreds of layers. We use the same architecture to model&#10;images, audio, and text from raw bytes, setting a new state of the art for&#10;density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate&#10;unconditional samples that demonstrate global coherence and great diversity,&#10;and show it is possible in principle to use self-attention to model sequences&#10;of length one million or more.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.14062" label="2007.14062">
        <attvalues>
          <attvalue for="0" value="Big Bird: Transformers for Longer Sequences" />
          <attvalue for="1" value="  Transformers-based models, such as BERT, have been one of the most successful&#10;deep learning models for NLP. Unfortunately, one of their core limitations is&#10;the quadratic dependency (mainly in terms of memory) on the sequence length due&#10;to their full attention mechanism. To remedy this, we propose, BigBird, a&#10;sparse attention mechanism that reduces this quadratic dependency to linear. We&#10;show that BigBird is a universal approximator of sequence functions and is&#10;Turing complete, thereby preserving these properties of the quadratic, full&#10;attention model. Along the way, our theoretical analysis reveals some of the&#10;benefits of having $O(1)$ global tokens (such as CLS), that attend to the&#10;entire sequence as part of the sparse attention mechanism. The proposed sparse&#10;attention can handle sequences of length up to 8x of what was previously&#10;possible using similar hardware. As a consequence of the capability to handle&#10;longer context, BigBird drastically improves performance on various NLP tasks&#10;such as question answering and summarization. We also propose novel&#10;applications to genomics data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.05150" label="2004.05150">
        <attvalues>
          <attvalue for="0" value="Longformer: The Long-Document Transformer" />
          <attvalue for="1" value="  Transformer-based models are unable to process long sequences due to their&#10;self-attention operation, which scales quadratically with the sequence length.&#10;To address this limitation, we introduce the Longformer with an attention&#10;mechanism that scales linearly with sequence length, making it easy to process&#10;documents of thousands of tokens or longer. Longformer's attention mechanism is&#10;a drop-in replacement for the standard self-attention and combines a local&#10;windowed attention with a task motivated global attention. Following prior work&#10;on long-sequence transformers, we evaluate Longformer on character-level&#10;language modeling and achieve state-of-the-art results on text8 and enwik8. In&#10;contrast to most prior work, we also pretrain Longformer and finetune it on a&#10;variety of downstream tasks. Our pretrained Longformer consistently outperforms&#10;RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop&#10;and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a&#10;Longformer variant for supporting long document generative sequence-to-sequence&#10;tasks, and demonstrate its effectiveness on the arXiv summarization dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Long-Document Transformers&#10;Tab.~\ref{tab:related} summarizes recent prior work on long documents. Two types of self-attention approaches have been explored. &#10;The first is a left-to-right (ltr) approach that processes the document in chunks moving from left-to-right.&#10;While such models have been successful in autoregressive language modeling, they are unsuitable for transfer learning approaches with tasks that benefit from bidirectional context.&#10;&#10;Our work falls within the other general approach that defines some form of sparse attention pattern and avoids computing &#10;the full quadratic attention matrix multiplication.&#10;The model with the most similar attention pattern to ours is Sparse Transformer~\cite{sparseOpenai}, which uses a form of dilated sliding window of blocks of size 8x8 provided by BlockSparse~\cite{blocksparse}. &#10;Our implementation (\S\ref{sec:model}) also includes a custom CUDA kernel, but it is more flexible and maintainable than BlockSparse which is implemented in C++, and designed for a specific version of TensorFlow.&#10;We also introduce additional task motivated global attention patterns suitable for common NLP tasks&#10;(\S\ref{sec:model}) and show they are essential for good performance in the transfer learning setting.&#10;&#10;A few models tried tasks other than autoregressive language modeling,&#10;which is &#10;a step forward because arguably focusing on &#10;language modeling as the primary evaluation&#10;has led to the development of models with limited applicability. BP-Transformer~\cite{BPTransformer} evaluated on &#10;machine translation (MT), but didn't explore the pretrain-finetune setting.&#10;Blockwise attention~\cite{blockbert} pretrained their models and evaluated &#10;on question answering (QA). However, the evaluation is &#10;limited as it doesn't include language modeling, and the QA datasets&#10;are of relatively short documents, therefore the effectiveness of this model on long document tasks remains unexplored.&#10;&#10;Task-specific Models for Long Documents&#10;Many task-specific approaches have been developed to &#10;workaround the 512 limit of pretrained transformer models like BERT.&#10;The simplest approach just truncates the document, commonly used for classification~\cite{truncateimdb}. &#10;Another approach chunks the document into chunks of &#10;length 512 (could be overlapping), processes each chunk separately, then combines the activations with a task specific model \cite{joshi-etal-2019-bert}.&#10;A third approach popular for multihop and open domain QA tasks uses a two-stage model where the first stage retrieves relevant documents that are passed onto the second stage for answer extraction \cite{Clark2017SimpleAE,Chen2017ReadingWT}.&#10;All of these approaches suffer from information loss due to truncation or cascading errors from the two stage approach.&#10;In contrast, Longformer can process long sequences without truncating or chunking, allowing us to adopt a much simpler approach that concatenates the available context and processes it in a single pass. &#10;&#10;A few contemporaneous works have explored similar ideas to Longformer using local + global attention in Transformers, and pre-training it for long document natural language tasks. In particular, ETC \cite{ainslie-etal-2020-etc} uses a similar local + global attention instead of full self-attention to scale Transformers to long documents. Different from Longformer, ETC uses relative position embeddings (which we only used for the Autoregressive LM setting), introduces an additional training objective (CPC loss) for pre-training, and configures global attention in a slightly different way. It shows strong results on several tasks including reading comprehension and classification.&#10;GMAT \cite{Gupta2020GMATGM} uses a similar idea of few global locations in the input serving as global memory. BigBird \cite{Zaheer2020BigBT} is an extension over ETC with evaluation on additional tasks, including summarization. Importantly, through theoretical analysis, BigBird shows that sparse Transformers are universal approximators of sequence functions and preserve these properties of the full self-attention.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2001.04451" label="2001.04451">
        <attvalues>
          <attvalue for="0" value="Reformer: The Efficient Transformer" />
          <attvalue for="1" value="  Large Transformer models routinely achieve state-of-the-art results on a&#10;number of tasks but training these models can be prohibitively costly,&#10;especially on long sequences. We introduce two techniques to improve the&#10;efficiency of Transformers. For one, we replace dot-product attention by one&#10;that uses locality-sensitive hashing, changing its complexity from O($L^2$) to&#10;O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use&#10;reversible residual layers instead of the standard residuals, which allows&#10;storing activations only once in the training process instead of $N$ times,&#10;where $N$ is the number of layers. The resulting model, the Reformer, performs&#10;on par with Transformer models while being much more memory-efficient and much&#10;faster on long sequences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.09752" label="2303.09752">
        <attvalues>
          <attvalue for="0" value="CoLT5: Faster Long-Range Transformers with Conditional Computation" />
          <attvalue for="1" value="  Many natural language processing tasks benefit from long inputs, but&#10;processing long documents with Transformers is expensive -- not only due to&#10;quadratic attention complexity but also from applying feedforward and&#10;projection layers to every token. However, not all tokens are equally&#10;important, especially for longer documents. We propose CoLT5, a long-input&#10;Transformer model that builds on this intuition by employing conditional&#10;computation, devoting more resources to important tokens in both feedforward&#10;and attention layers. We show that CoLT5 achieves stronger performance than&#10;LongT5 with much faster training and inference, achieving SOTA on the&#10;long-input SCROLLS benchmark. Moreover, CoLT5 can effectively and tractably&#10;make use of extremely long inputs, showing strong gains up to 64k input length.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.04768" label="2006.04768">
        <attvalues>
          <attvalue for="0" value="Linformer: Self-Attention with Linear Complexity" />
          <attvalue for="1" value="  Large transformer models have shown extraordinary success in achieving&#10;state-of-the-art results in many natural language processing applications.&#10;However, training and deploying these models can be prohibitively costly for&#10;long sequences, as the standard self-attention mechanism of the Transformer&#10;uses $O(n^2)$ time and space with respect to sequence length. In this paper, we&#10;demonstrate that the self-attention mechanism can be approximated by a low-rank&#10;matrix. We further exploit this finding to propose a new self-attention&#10;mechanism, which reduces the overall self-attention complexity from $O(n^2)$ to&#10;$O(n)$ in both time and space. The resulting linear transformer, the&#10;\textit{Linformer}, performs on par with standard Transformer models, while&#10;being much more memory- and time-efficient.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.13923" label="1910.13923">
        <attvalues>
          <attvalue for="0" value="Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank&#10;  Transformer" />
          <attvalue for="1" value="  Highly performing deep neural networks come at the cost of computational&#10;complexity that limits their practicality for deployment on portable devices.&#10;We propose the low-rank transformer (LRT), a memory-efficient and fast neural&#10;architecture that significantly reduces the parameters and boosts the speed of&#10;training and inference for end-to-end speech recognition. Our approach reduces&#10;the number of parameters of the network by more than 50% and speeds up the&#10;inference time by around 1.35x compared to the baseline transformer model. The&#10;experiments show that our LRT model generalizes better and yields lower error&#10;rates on both validation and test sets compared to an uncompressed transformer&#10;model. The LRT model outperforms those from existing works on several datasets&#10;in an end-to-end setting without using an external language model or acoustic&#10;data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.16236" label="2006.16236">
        <attvalues>
          <attvalue for="0" value="Transformers are RNNs: Fast Autoregressive Transformers with Linear&#10;  Attention" />
          <attvalue for="1" value="  Transformers achieve remarkable performance in several tasks but due to their&#10;quadratic complexity, with respect to the input's length, they are&#10;prohibitively slow for very long sequences. To address this limitation, we&#10;express the self-attention as a linear dot-product of kernel feature maps and&#10;make use of the associativity property of matrix products to reduce the&#10;complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$,&#10;where $N$ is the sequence length. We show that this formulation permits an&#10;iterative implementation that dramatically accelerates autoregressive&#10;transformers and reveals their relationship to recurrent neural networks. Our&#10;linear transformers achieve similar performance to vanilla transformers and&#10;they are up to 4000x faster on autoregressive prediction of very long&#10;sequences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.14794" label="2009.14794">
        <attvalues>
          <attvalue for="0" value="Rethinking Attention with Performers" />
          <attvalue for="1" value="  We introduce Performers, Transformer architectures which can estimate regular&#10;(softmax) full-rank-attention Transformers with provable accuracy, but using&#10;only linear (as opposed to quadratic) space and time complexity, without&#10;relying on any priors such as sparsity or low-rankness. To approximate softmax&#10;attention-kernels, Performers use a novel Fast Attention Via positive&#10;Orthogonal Random features approach (FAVOR+), which may be of independent&#10;interest for scalable kernel methods. FAVOR+ can be also used to efficiently&#10;model kernelizable attention mechanisms beyond softmax. This representational&#10;power is crucial to accurately compare softmax with other kernels for the first&#10;time on large-scale tasks, beyond the reach of regular Transformers, and&#10;investigate optimal attention-kernels. Performers are linear architectures&#10;fully compatible with regular Transformers and with strong theoretical&#10;guarantees: unbiased or nearly-unbiased estimation of the attention matrix,&#10;uniform convergence and low estimation variance. We tested Performers on a rich&#10;set of tasks stretching from pixel-prediction through text models to protein&#10;sequence modeling. We demonstrate competitive results with other examined&#10;efficient sparse and dense attention methods, showcasing effectiveness of the&#10;novel attention-learning paradigm leveraged by Performers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.10340" label="2210.10340">
        <attvalues>
          <attvalue for="0" value="The Devil in Linear Transformer" />
          <attvalue for="1" value="  Linear transformers aim to reduce the quadratic space-time complexity of&#10;vanilla transformers. However, they usually suffer from degraded performances&#10;on various tasks and corpus. In this paper, we examine existing kernel-based&#10;linear transformers and identify two key issues that lead to such performance&#10;gaps: 1) unbounded gradients in the attention computation adversely impact the&#10;convergence of linear transformer models; 2) attention dilution which trivially&#10;distributes attention scores over long sequences while neglecting neighbouring&#10;structures. To address these issues, we first identify that the scaling of&#10;attention matrices is the devil in unbounded gradients, which turns out&#10;unnecessary in linear attention as we show theoretically and empirically. To&#10;this end, we propose a new linear attention that replaces the scaling operation&#10;with a normalization to stabilize gradients. For the issue of attention&#10;dilution, we leverage a diagonal attention to confine attention to only&#10;neighbouring tokens in early layers. Benefiting from the stable gradients and&#10;improved attention, our new linear transformer model, transNormer, demonstrates&#10;superior performance on text classification and language modeling tasks, as&#10;well as on the challenging Long-Range Arena benchmark, surpassing vanilla&#10;transformer and existing linear variants by a clear margin while being&#10;significantly more space-time efficient. The code is available at&#10;https://github.com/OpenNLPLab/Transnormer .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.00825" label="1810.00825">
        <attvalues>
          <attvalue for="0" value="Set Transformer: A Framework for Attention-based Permutation-Invariant&#10;  Neural Networks" />
          <attvalue for="1" value="  Many machine learning tasks such as multiple instance learning, 3D shape&#10;recognition, and few-shot image classification are defined on sets of&#10;instances. Since solutions to such problems do not depend on the order of&#10;elements of the set, models used to address them should be permutation&#10;invariant. We present an attention-based neural network module, the Set&#10;Transformer, specifically designed to model interactions among elements in the&#10;input set. The model consists of an encoder and a decoder, both of which rely&#10;on attention mechanisms. In an effort to reduce computational complexity, we&#10;introduce an attention scheme inspired by inducing point methods from sparse&#10;Gaussian process literature. It reduces the computation time of self-attention&#10;from quadratic to linear in the number of elements in the set. We show that our&#10;model is theoretically attractive and we evaluate it on a range of tasks,&#10;demonstrating the state-of-the-art performance compared to recent methods for&#10;set-structured data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.01540" label="2106.01540">
        <attvalues>
          <attvalue for="0" value="Luna: Linear Unified Nested Attention" />
          <attvalue for="1" value="  The quadratic computational and memory complexities of the Transformer's&#10;attention mechanism have limited its scalability for modeling long sequences.&#10;In this paper, we propose Luna, a linear unified nested attention mechanism&#10;that approximates softmax attention with two nested linear attention functions,&#10;yielding only linear (as opposed to quadratic) time and space complexity.&#10;Specifically, with the first attention function, Luna packs the input sequence&#10;into a sequence of fixed length. Then, the packed sequence is unpacked using&#10;the second attention function. As compared to a more traditional attention&#10;mechanism, Luna introduces an additional sequence with a fixed length as input&#10;and an additional corresponding output, which allows Luna to perform attention&#10;operation linearly, while also storing adequate contextual information. We&#10;perform extensive evaluations on three benchmarks of sequence modeling tasks:&#10;long-context sequence modeling, neural machine translation and masked language&#10;modeling for large-scale pretraining. Competitive or even better experimental&#10;results demonstrate both the effectiveness and efficiency of Luna compared to a&#10;variety&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.02860" label="1901.02860">
        <attvalues>
          <attvalue for="0" value="Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" />
          <attvalue for="1" value="  Transformers have a potential of learning longer-term dependency, but are&#10;limited by a fixed-length context in the setting of language modeling. We&#10;propose a novel neural architecture Transformer-XL that enables learning&#10;dependency beyond a fixed length without disrupting temporal coherence. It&#10;consists of a segment-level recurrence mechanism and a novel positional&#10;encoding scheme. Our method not only enables capturing longer-term dependency,&#10;but also resolves the context fragmentation problem. As a result,&#10;Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer&#10;than vanilla Transformers, achieves better performance on both short and long&#10;sequences, and is up to 1,800+ times faster than vanilla Transformers during&#10;evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity&#10;to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion&#10;Word, and 54.5 on Penn Treebank (without finetuning). When trained only on&#10;WikiText-103, Transformer-XL manages to generate reasonably coherent, novel&#10;text articles with thousands of tokens. Our code, pretrained models, and&#10;hyperparameters are available in both Tensorflow and PyTorch.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.11062" label="2304.11062">
        <attvalues>
          <attvalue for="0" value="Scaling Transformer to 1M tokens and beyond with RMT" />
          <attvalue for="1" value="  A major limitation for the broader scope of problems solvable by transformers&#10;is the quadratic scaling of computational complexity with input size. In this&#10;study, we investigate the recurrent memory augmentation of pre-trained&#10;transformer models to extend input context length while linearly scaling&#10;compute. Our approach demonstrates the capability to store information in&#10;memory for sequences of up to an unprecedented two million tokens while&#10;maintaining high retrieval accuracy. Experiments with language modeling tasks&#10;show perplexity improvement as the number of processed input segments&#10;increases. These results underscore the effectiveness of our method, which has&#10;significant potential to enhance long-term dependency handling in natural&#10;language understanding and generation tasks, as well as enable large-scale&#10;context processing for memory-intensive applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.08913" label="2203.08913">
        <attvalues>
          <attvalue for="0" value="Memorizing Transformers" />
          <attvalue for="1" value="  Language models typically need to be trained or finetuned in order to acquire&#10;new knowledge, which involves updating their weights. We instead envision&#10;language models that can simply read and memorize new data at inference time,&#10;thus acquiring new knowledge immediately. In this work, we extend language&#10;models with the ability to memorize the internal representations of past&#10;inputs. We demonstrate that an approximate kNN lookup into a non-differentiable&#10;memory of recent (key, value) pairs improves language modeling across various&#10;benchmarks and tasks, including generic webtext (C4), math papers (arXiv),&#10;books (PG-19), code (Github), as well as formal theorems (Isabelle). We show&#10;that the performance steadily improves when we increase the size of memory up&#10;to 262K tokens. On benchmarks including code and mathematics, we find that the&#10;model is capable of making use of newly defined functions and theorems during&#10;test time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.07174" label="2306.07174">
        <attvalues>
          <attvalue for="0" value="Augmenting Language Models with Long-Term Memory" />
          <attvalue for="1" value="  Existing large language models (LLMs) can only afford fix-sized inputs due to&#10;the input length limit, preventing them from utilizing rich long-context&#10;information from past inputs. To address this, we propose a framework, Language&#10;Models Augmented with Long-Term Memory (LongMem), which enables LLMs to&#10;memorize long history. We design a novel decoupled network architecture with&#10;the original backbone LLM frozen as a memory encoder and an adaptive residual&#10;side-network as a memory retriever and reader. Such a decoupled memory design&#10;can easily cache and update long-term past contexts for memory retrieval&#10;without suffering from memory staleness. Enhanced with memory-augmented&#10;adaptation training, LongMem can thus memorize long past context and use&#10;long-term memory for language modeling. The proposed memory retrieval module&#10;can handle unlimited-length context in its memory bank to benefit various&#10;downstream tasks. Typically, LongMem can enlarge the long-form memory to 65k&#10;tokens and thus cache many-shot extra demonstration examples as long-form&#10;memory for in-context learning. Experiments show that our method outperforms&#10;strong long-context models on ChapterBreak, a challenging long-context modeling&#10;benchmark, and achieves remarkable improvements on memory-augmented in-context&#10;learning over LLMs. The results demonstrate that the proposed method is&#10;effective in helping language models to memorize and utilize long-form&#10;contents. Our code is open-sourced at https://aka.ms/LongMem.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Large Language Models. Large Language Models, \ie GPT-2~\cite{gpt2}, GPT-3~\cite{gpt3}, OPT~\cite{zhang2022opt}, and BLOOM~\cite{scao2022bloom}, significantly revolutionized NLP research and promoted the state-of-the-art of various language understanding, language generation~\cite{wang2022task}, and even vision-language tasks~\cite{valm}. Additionally, via scaling the model parameters, LLMs exhibit ``emergent abilities``~\cite{wei2022emergent} like few-shot in-context learning~\cite{gpt3}, multi-step reasoning~\cite{cot}, code completion, etc. &#10;&#10;x-formers. To enable transformers to attend on longer context, many variants of ``x-formers`` are proposed. Transformer-XL~\cite{transformerxl} proposes to cache attention keys and values of past segment and reuse them in recurrent manner. Recent seminal works of x-formers, including LinFormer~\cite{wang2020linformer}, LongFormer~\cite{beltagy2020longformer}, Routing Transformer~\cite{routingtrm}, proposed various sparse attention mechanisms for decreasing $O(n^2)$ complexity to $O(n\log n)$ or even $O(n)$. BigBird~\cite{bigbird} achieves a 4k sequence length via attending on a subset of context tokens. Although these x-formers achieve substantial efficiency improvements, such efficiency gains are not remarkable when modeling sequences that spans book-level length. Moreover, the largest sequence length of these methods is still upper-bounded by 16k tokens, making them invalid in modeling long-sequences at the book or wikipedia-page level (\ie average 70k tokens for full-length books in PG19 dataset~\cite{pg19}). &#10;&#10;Side-Tuning. The method of Side-Tuning~\cite{sidetuning,lst} is a task-specific tuning method for pre-trained models via training a lightweight side-network that is fused with the fixed pre-trained network via summation. Our method inherits the idea of adopting a side-network but distinguishes the side-tuning method in terms of learning objective and cross-network fusion ways. \our proposes to augment LLMs with decoupled memory for memorizing long past inputs, which does not involve any task-specific tuning. The cross-network residual connections proposed by \our is novel and distincts from the vanilla summation of Side-Tuning. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2010.12198" label="2010.12198">
        <attvalues>
          <attvalue for="0" value="Domain Divergences: a Survey and Empirical Analysis" />
          <attvalue for="1" value="  Domain divergence plays a significant role in estimating the performance of a&#10;model in new domains. While there is a significant literature on divergence&#10;measures, researchers find it hard to choose an appropriate divergence for a&#10;given NLP application. We address this shortcoming by both surveying the&#10;literature and through an empirical study. We develop a taxonomy of divergence&#10;measures consisting of three classes -- Information-theoretic, Geometric, and&#10;Higher-order measures and identify the relationships between them. Further, to&#10;understand the common use-cases of these measures, we recognise three novel&#10;applications -- 1) Data Selection, 2) Learning Representation, and 3) Decisions&#10;in the Wild -- and use it to organise our literature. From this, we identify&#10;that Information-theoretic measures are prevalent for 1) and 3), and&#10;Higher-order measures are more common for 2). To further help researchers&#10;choose appropriate measures to predict drop in performance -- an important&#10;aspect of Decisions in the Wild, we perform correlation analysis spanning 130&#10;domain adaptation scenarios, 3 varied NLP tasks and 12 divergence measures&#10;identified from our survey. To calculate these divergences, we consider the&#10;current contextual word representations (CWR) and contrast with the older&#10;distributed representations. We find that traditional measures over word&#10;distributions still serve as strong baselines, while higher-order measures with&#10;CWR are effective.&#10;" />
          <attvalue for="2" value="&#10;&#10;Standard machine learning models do not perform well when tested on data from a different target domain. The performance in a target domain largely depends on the domain divergence~\cite{ben2010theory} -- \abhi{a notion of distance between the two domains}. Thus, efficiently measuring and reducing divergence is crucial for adapting models to the new domain --- the topic of {domain adaptation}. Divergence also has practical applications in predicting the performance drop of a model when adapted to new domains~\cite{van-asch-daelemans-2010-using}, and in choosing among alternate models~\cite{xia2020predicting}. &#10;&#10;\abhi{Given its importance,} researchers have invested much effort to define and measure domain divergence. Linguists use register variation to capture varieties in text -- the difference between distributions of the prevalent features in two registers~\cite{biber_conrad_2009}. \abhi{Other measures include probabilistic measures like $\mathcal{H}$-divergence \cite{ben2010theory}, information theoretic measures like Jenssen-Shannon and Kullback-Leibler divergence~\cite{plank-van-noord-2011-effective,van-asch-daelemans-2010-using} and measures using higher-order moments of random variables like Maximum Mean Discrepancy (MMD) and Central Moment Discrepancy (CMD) \cite{Gretton2006, Zellinger2017CentralMD}.} The proliferation of divergence measures challenges researchers in choosing an appropriate measure for a given application.&#10;&#10;To help guide best practices, we first comprehensively review the NLP literature on domain divergences. Unlike previous surveys, which focus on domain adaptation for specific tasks such as machine translation~\cite{chu-wang-2018-survey} and statistical (non-neural network) models \cite{Jiang2007ALS,Margolis2011ALR}, our work takes a different perspective. We study domain adaptation through the vehicle of {domain divergence measures}. First, we develop a taxonomy of divergence measures consisting of three groups: Information-Theoretic, Geometric, and Higher-Order measures. Further, to find the most common group used in NLP, we recognise three novel application areas of these divergences --- Data Selection, Learning Representations, and Decisions in the Wild and organise the literature under them. We find that Information-Theoretic measures over word distributions are popular for Data Selection and Decisions in the wild, while Higher-order measures over continuous features are frequent for Learning representations. &#10;&#10;Domain divergence is a major predictor of performance in the target domain. A better domain divergence metric ideally predicts the corresponding performance drop of a model when applied to a target domain -- \abhi{a practical and important component of Decisions in the Wild}. We further help researchers identify appropriate measures \abhi{for predicting performance drops}, through a correlation analysis over 130 domain adaptation scenarios and three standard NLP tasks: Part of Speech Tagging (POS), Named Entity Recognition (NER), and Sentiment Analysis and 12 divergence metrics from our literature review. While information-theoretic measures over traditional word distributions are popular in the literature, are higher-order measures calculated over modern contextual word representations better indicators of performance drop? We indeed find that higher-order measures are superior, but traditional measures are still reliable indicators of performance drop. \abhi{The closest to our work is \cite{elsahar-galle-2019-annotate} who perform a correlation analysis. However, they do not compare against different divergence measures from the literature. Comparatively, we consider more tasks and divergence measures.}&#10;&#10;In summary, our contributions are:&#10;\begin{itemize}[leftmargin=*]&#10;\itemsep-0.37em &#10; \item We review the literature from the perspective of domain divergences and their use-cases in NLP.&#10; \item We aid researchers to select appropriate divergence measure that indicate performance-drops, an important application of divergence measures.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Machine Learning, Linguistics, Domain Adaptation, Mathematics, Natural Language Processing, Divergence Measures" />
        </attvalues>
      </node>
      <node id="2005.00870" label="2005.00870">
        <attvalues>
          <attvalue for="0" value="Predicting Performance for Natural Language Processing Tasks" />
          <attvalue for="1" value="  Given the complexity of combinations of tasks, languages, and domains in&#10;natural language processing (NLP) research, it is computationally prohibitive&#10;to exhaustively test newly proposed models on each possible experimental&#10;setting. In this work, we attempt to explore the possibility of gaining&#10;plausible judgments of how well an NLP model can perform under an experimental&#10;setting, without actually training or testing the model. To do so, we build&#10;regression models to predict the evaluation score of an NLP experiment given&#10;the experimental settings as input. Experimenting on 9 different NLP tasks, we&#10;find that our predictors can produce meaningful predictions over unseen&#10;languages and different modeling architectures, outperforming reasonable&#10;baselines as well as human experts. Going further, we outline how our predictor&#10;can be used to find a small subset of representative experiments that should be&#10;run in order to obtain plausible predictions for all other experimental&#10;settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.13712" label="2304.13712">
        <attvalues>
          <attvalue for="0" value="Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond" />
          <attvalue for="1" value="  This paper presents a comprehensive and practical guide for practitioners and&#10;end-users working with Large Language Models (LLMs) in their downstream natural&#10;language processing (NLP) tasks. We provide discussions and insights into the&#10;usage of LLMs from the perspectives of models, data, and downstream tasks.&#10;Firstly, we offer an introduction and brief summary of current GPT- and&#10;BERT-style LLMs. Then, we discuss the influence of pre-training data, training&#10;data, and test data. Most importantly, we provide a detailed discussion about&#10;the use and non-use cases of large language models for various natural language&#10;processing tasks, such as knowledge-intensive tasks, traditional natural&#10;language understanding tasks, natural language generation tasks, emergent&#10;abilities, and considerations for specific tasks.We present various use cases&#10;and non-use cases to illustrate the practical applications and limitations of&#10;LLMs in real-world scenarios. We also try to understand the importance of data&#10;and the specific challenges associated with each NLP task. Furthermore, we&#10;explore the impact of spurious biases on LLMs and delve into other essential&#10;considerations, such as efficiency, cost, and latency, to ensure a&#10;comprehensive understanding of deploying LLMs in practice. This comprehensive&#10;guide aims to provide researchers and practitioners with valuable insights and&#10;best practices for working with LLMs, thereby enabling the successful&#10;implementation of these models in a wide range of NLP tasks. A curated list of&#10;practical guide resources of LLMs, regularly updated, can be found at&#10;\url{https://github.com/Mooler0410/LLMsPracticalGuide}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.16450" label="2310.16450">
        <attvalues>
          <attvalue for="0" value="CLEX: Continuous Length Extrapolation for Large Language Models" />
          <attvalue for="1" value="  Transformer-based Large Language Models (LLMs) are pioneering advances in&#10;many natural language processing tasks, however, their exceptional capabilities&#10;are restricted within the preset context window of Transformer. Position&#10;Embedding (PE) scaling methods, while effective in extending the context window&#10;to a specific length, demonstrate either notable limitations in their&#10;extrapolation abilities or sacrificing partial performance within the context&#10;window. Length extrapolation methods, although theoretically capable of&#10;extending the context window beyond the training sequence length, often&#10;underperform in practical long-context applications. To address these&#10;challenges, we propose Continuous Length EXtrapolation (CLEX) for LLMs. We&#10;generalise the PE scaling approaches to model the continuous dynamics by&#10;ordinary differential equations over the length scaling factor, thereby&#10;overcoming the constraints of current PE scaling methods designed for specific&#10;lengths. Moreover, by extending the dynamics to desired context lengths beyond&#10;the training sequence length, CLEX facilitates the length extrapolation with&#10;impressive performance in practical tasks. We demonstrate that CLEX can be&#10;seamlessly incorporated into LLMs equipped with Rotary Position Embedding, such&#10;as LLaMA and GPT-NeoX, with negligible impact on training and inference&#10;latency. Experimental results reveal that CLEX can effectively extend the&#10;context window to over 4x or almost 8x training length, with no deterioration&#10;in performance. Furthermore, when evaluated on the practical LongBench&#10;benchmark, our model trained on a 4k length exhibits competitive performance&#10;against state-of-the-art open-source models trained on context lengths up to&#10;32k. Our code is available at https://github.com/DAMO-NLP-SG/CLEX.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.12307" label="2309.12307">
        <attvalues>
          <attvalue for="0" value="LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models" />
          <attvalue for="1" value="  We present LongLoRA, an efficient fine-tuning approach that extends the&#10;context sizes of pre-trained large language models (LLMs), with limited&#10;computation cost. Typically, training LLMs with long context sizes is&#10;computationally expensive, requiring extensive training hours and GPU&#10;resources. For example, training on the context length of 8192 needs 16x&#10;computational costs in self-attention layers as that of 2048. In this paper, we&#10;speed up the context extension of LLMs in two aspects. On the one hand,&#10;although dense global attention is needed during inference, fine-tuning the&#10;model can be effectively and efficiently done by sparse local attention. The&#10;proposed shifted sparse attention effectively enables context extension,&#10;leading to non-trivial computation saving with similar performance to&#10;fine-tuning with vanilla attention. Particularly, it can be implemented with&#10;only two lines of code in training, while being optional in inference. On the&#10;other hand, we revisit the parameter-efficient fine-tuning regime for context&#10;expansion. Notably, we find that LoRA for context extension works well under&#10;the premise of trainable embedding and normalization. LongLoRA combines this&#10;improved LoRA with S^2-Attn. LongLoRA demonstrates strong empirical results on&#10;various tasks on Llama2 models from 7B/13B to 70B. LongLoRA extends Llama2 7B&#10;from 4k context to 100k, or Llama2 70B to 32k on a single 8x A100 machine.&#10;LongLoRA extends models' context while retaining their original architectures,&#10;and is compatible with most existing techniques, like Flash-Attention2. In&#10;addition, we further conduct supervised fine-tuning with LongLoRA and our long&#10;instruction-following LongAlpaca dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Long-context Transformers.&#10;A large body of research has been developed to increase the context length of transformers. Some of these approaches are retrieval-based~\cite{retrieval-QA, few-shot-retrieval, realm}, which augment language models via fetching related documents and including the retrieved results into contexts. Our work is complementary to these works, as our attention mechanism is unmodified during inference.&#10;Many works modify multi-head attention to be approximated ones~\cite{linformer, long-former, big-bird, reformer, recurrent-memory-transformer, longnet, block-wise-self-attention}. They alleviate the quadratic complexity of the self-attention computation. For example, Longformer~\cite{long-former} and BigBird~\cite{big-bird} use sparse attention to handle long sequences. Other works~\cite{memorizing-transformer, recurrent-memory-transformer} utilize memory mechanisms as a compression on past inputs, to look up relevant tokens. One limitation of these works is that these compressions have a large gap to full attention, making it infeasible to fine-tune pre-trained LLMs. &#10;Although our work also involves an approximation of attention mechanism, it has a similar shape and a small gap to standard attention. This enables fine-tuning pre-trained LLMs on S$^2$-Attn and maintain full attention during inference.&#10;&#10;Long-context LLMs.&#10;LLMs are typically pre-trained with a pre-defined context length, such as 2048 for LLaMA~\cite{llama} and 4096 for Llama2~\cite{llama2}. Training LLMs with long context from&#10;scratch is prohibitively expensive for most researchers.&#10;Recently, several works have tried to extend the context length of LLMs via fine-tuning. Position Interpolation~\cite{position-interpolation} modifies rotary position encoding~\cite{rope} and extends the context length of LLaMA to 32768. Focused Transformer~\cite{focused-transformer} utilizes contrastive learning to train LongLLaMA. Both of them rely on full fine-tuning, which is computationally expensive (128 A100 GPUs / 128 TPUv3 for training). Landmark attention~\cite{landmark-attention} is an efficient approach, but somewhat lossy. It compresses long context inputs into retrieved tokens. Our method saves substantial fine-tuning costs, while preserving the quality of the original attention. Ours maintain full access to the entire input via unmodified attention during inference.&#10;&#10;Some literature focuses on the position embedding modification of LLMs for long context extension, including Position Interpolation~\cite{position-interpolation}, NTK-aware~\cite{ntk-pe}, Yarn~\cite{yarn}, positional Skipping~\cite{zhu2023pose}, and methods based on out-of-distribution analysis~\cite{lm-infinite}. Our method focuses on efficient fine-tuning and retaining the original architecture during inference, which is orthogonal to these position embedding methods. &#10;&#10;Efficient Fine-tuning.&#10;This work is based on LoRA~\cite{lora}, a classical efficient fine-tuning approach. In addition to LoRA~\cite{lora}, there are many other parameter-efficient fine-tuning methods, including prompt tuning~\cite{prompt-tuning}, prefix tuning~\cite{prefix-tuning}, hidden state tuning~\cite{ia-3}, bias tuning~\cite{bitfit}, and masked weight learning~\cite{fisher-mask}. Input-tuning~\cite{input-tuning} introduces an adapter to tune input embedding. Although the input embedding layers are also trainable in ours, this is not enough for long context extension. We make a comprehensive analysis on layer types in experiments, in Table~\ref{tab:lora-settings}. Existing work~\cite{sparse-training} shows sparse masks can effectively save training costs and avoid performance drops.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2108.13624" label="2108.13624">
        <attvalues>
          <attvalue for="0" value="Towards Out-Of-Distribution Generalization: A Survey" />
          <attvalue for="1" value="  Traditional machine learning paradigms are based on the assumption that both&#10;training and test data follow the same statistical pattern, which is&#10;mathematically referred to as Independent and Identically Distributed&#10;($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often&#10;fails to hold due to unforeseen distributional shifts, leading to considerable&#10;degradation in model performance upon deployment. This observed discrepancy&#10;indicates the significance of investigating the Out-of-Distribution (OOD)&#10;generalization problem. OOD generalization is an emerging topic of machine&#10;learning research that focuses on complex scenarios wherein the distributions&#10;of the test data differ from those of the training data. This paper represents&#10;the first comprehensive, systematic review of OOD generalization, encompassing&#10;a spectrum of aspects from problem definition, methodological development, and&#10;evaluation procedures, to the implications and future directions of the field.&#10;Our discussion begins with a precise, formal characterization of the OOD&#10;generalization problem. Following that, we categorize existing methodologies&#10;into three segments: unsupervised representation learning, supervised model&#10;learning, and optimization, according to their positions within the overarching&#10;learning process. We provide an in-depth discussion on representative&#10;methodologies for each category, further elucidating the theoretical links&#10;between them. Subsequently, we outline the prevailing benchmark datasets&#10;employed in OOD generalization studies. To conclude, we overview the existing&#10;body of work in this domain and suggest potential avenues for future research&#10;on OOD generalization. A summary of the OOD generalization methodologies&#10;surveyed in this paper can be accessed at&#10;http://out-of-distribution-generalization.com.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.15231" label="2308.15231">
        <attvalues>
          <attvalue for="0" value="Multi-party Goal Tracking with LLMs: Comparing Pre-training,&#10;  Fine-tuning, and Prompt Engineering" />
          <attvalue for="1" value="  This paper evaluates the extent to which current Large Language Models (LLMs)&#10;can capture task-oriented multi-party conversations (MPCs). We have recorded&#10;and transcribed 29 MPCs between patients, their companions, and a social robot&#10;in a hospital. We then annotated this corpus for multi-party goal-tracking and&#10;intent-slot recognition. People share goals, answer each other's goals, and&#10;provide other people's goals in MPCs - none of which occur in dyadic&#10;interactions. To understand user goals in MPCs, we compared three methods in&#10;zero-shot and few-shot settings: we fine-tuned T5, created pre-training tasks&#10;to train DialogLM using LED, and employed prompt engineering techniques with&#10;GPT-3.5-turbo, to determine which approach can complete this novel task with&#10;limited data. GPT-3.5-turbo significantly outperformed the others in a few-shot&#10;setting. The `reasoning' style prompt, when given 7% of the corpus as example&#10;annotated conversations, was the best performing method. It correctly annotated&#10;62.32% of the goal tracking MPCs, and 69.57% of the intent-slot recognition&#10;MPCs. A `story' style prompt increased model hallucination, which could be&#10;detrimental if deployed in safety-critical settings. We conclude that&#10;multi-party conversations still challenge state-of-the-art LLMs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken Dialogue Systems (SDSs) are increasingly being embedded in social robots that are expected to seamlessly interact with people in populated public spaces like museums, airports, shopping centres, or hospital waiting rooms \cite{foster2019mummer,tian2021redesigning,gunson2022visually}. Unlike virtual agents or voice assistants (e.g. Alexa, Siri, or Google Assistant), which typically have dyadic interactions with a single user, social robots are often approached by pairs and groups of individuals \cite{al2012furhat,moujahid2022multi}. Families may approach a social robot in a\nobreakspace museum, and patients are often accompanied by a family member when visiting a hospital. In these multi-party scenarios, tasks that are considered trivial for SDSs become substantially more complex \cite{traum2004issues,zhong2022dialoglm,addlesee2023data}. In multi-party conversations (MPCs), the social robot must determine which user said an utterance, who that utterance was directed to, when to respond, and what it should say depending on whom the robot is addressing \cite{hu2019gsn,gu2021mpc,gu2022hetermpc}. These tasks are collectively referred to as ``who says what to whom'' in the multi-party literature \cite{gu2022says}, but these tasks alone provide no incentive for a system to actually help a user reach their goals. State of the art ``who says what to whom'' systems can, therefore, only mimic what a good MPC looks like \cite{addlesee2023data}, but for practical systems we also need to know what each user's goals are. We therefore propose two further tasks that become substantially more complex when considered in a multi-party setting: goal tracking and intent-slot recognition \cite{addlesee2023data}.&#10;&#10;Dialogue State Tracking (DST) is a well-established task \cite{lee2021dialogue,feng2022dynamic} that is considered crucial to the success of a dialogue system \cite{williams2016dialog}. DST corpora are abundant \cite{henderson2014second,henderson2014third}, but they only contain dyadic conversations. No corpus exists containing MPCs with goal tracking or intent-slot annotations, yet there are important differences. Consider the example in Table \ref{tab:ex1} (from our new corpus, detailed in Section \ref{sec:dataset}). In turn 1, we can identify that User 1 (U1) wants to know their appointment time. Before the social robot had time to answer, User 2 (U2) answered in turn 2. This obviously does not occur in a dyadic interaction, yet this understanding is essential for natural system behaviour. The SDS must determine that it should not repeat the answer to the question, so data must be collected to learn this. Other major differences exist too. For example, current DST corpora do not contain a concept of `shared goals' \cite{eshghi2016collective}. If two people approach a café counter, the barista must determine whether the two people are separate (two individuals wanting to get coffee), or together (two friends with the shared goal to get coffee) \cite{keizer2013training}. The interaction changes depending on this fact, it would be unusual to ask ``are you paying together'' to two individuals. Shared goals can commonly be identified through explicit dialogue. For example, the use of `we' in ``We are looking for the bathrooms''. Similar to answering each other's questions, people may also ask questions on behalf of others. In our corpus, a person said ``ARI, the person that I'm accompanying feels intimidated by you, and they'd like to know where they can eat''.&#10;&#10;In this paper, we present several contributions. (1) We collected a corpus of multi-party interactions between a social robot and patients with their companions in a hospital memory clinic. (2) This corpus was annotated for the standard ``who says what to whom'' tasks, but also for multi-party goal tracking and intent-slot recognition. We followed current DST annotation instructions, tweaked to enable annotation of multi-party phenomena (detailed in Section \ref{sec:dataset}). (3) We then evaluated Large Language Models (LLMs) on these two new tasks using our collected corpus. Models were pre-trained, fine-tuned, or prompt engineered where applicable (detailed in Section \ref{sec:experiment}). It is not possible to collect enormous corpora from patients in a hospital, so models were evaluated in zero-shot and few-shot settings. We found that the GPT-3.5-turbo model significantly outperformed others on both tasks when given a `reasoning' style prompt. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Goal Recognition Techniques, Human-Computer Interaction, Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Multi-Party Conversations" />
        </attvalues>
      </node>
      <node id="1909.06749" label="1909.06749">
        <attvalues>
          <attvalue for="0" value="MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces" />
          <attvalue for="1" value="  In the EU-funded MuMMER project, we have developed a social robot designed to&#10;interact naturally and flexibly with users in public spaces such as a shopping&#10;mall. We present the latest version of the robot system developed during the&#10;project. This system encompasses audio-visual sensing, social signal&#10;processing, conversational interaction, perspective taking, geometric&#10;reasoning, and motion planning. It successfully combines all these components&#10;in an overarching framework using the Robot Operating System (ROS) and has been&#10;deployed to a shopping mall in Finland interacting with customers. In this&#10;paper, we describe the system components, their interplay, and the resulting&#10;robot behaviours and scenarios provided at the shopping mall.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.02492" label="2109.02492">
        <attvalues>
          <attvalue for="0" value="DialogLM: Pre-trained Model for Long Dialogue Understanding and&#10;  Summarization" />
          <attvalue for="1" value="  Dialogue is an essential part of human communication and cooperation.&#10;Existing research mainly focuses on short dialogue scenarios in a one-on-one&#10;fashion. However, multi-person interactions in the real world, such as meetings&#10;or interviews, are frequently over a few thousand words. There is still a lack&#10;of corresponding research and powerful tools to understand and process such&#10;long dialogues. Therefore, in this work, we present a pre-training framework&#10;for long dialogue understanding and summarization. Considering the nature of&#10;long conversations, we propose a window-based denoising approach for generative&#10;pre-training. For a dialogue, it corrupts a window of text with&#10;dialogue-inspired noise, and guides the model to reconstruct this window based&#10;on the content of the remaining conversation. Furthermore, to process longer&#10;input, we augment the model with sparse attention which is combined with&#10;conventional attention in a hybrid manner. We conduct extensive experiments on&#10;five datasets of long dialogues, covering tasks of dialogue summarization,&#10;abstractive question answering and topic segmentation. Experimentally, we show&#10;that our pre-trained model DialogLM significantly surpasses the&#10;state-of-the-art models across datasets and tasks. Source code and all the&#10;pre-trained models are available on our GitHub repository&#10;(https://github.com/microsoft/DialogLM).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.01541" label="2106.01541">
        <attvalues>
          <attvalue for="0" value="MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation&#10;  Understanding" />
          <attvalue for="1" value="  Recently, various neural models for multi-party conversation (MPC) have&#10;achieved impressive improvements on a variety of tasks such as addressee&#10;recognition, speaker identification and response prediction. However, these&#10;existing methods on MPC usually represent interlocutors and utterances&#10;individually and ignore the inherent complicated structure in MPC which may&#10;provide crucial interlocutor and utterance semantics and would enhance the&#10;conversation understanding process. To this end, we present MPC-BERT, a&#10;pre-trained model for MPC understanding that considers learning who says what&#10;to whom in a unified model with several elaborated self-supervised tasks.&#10;Particularly, these tasks can be generally categorized into (1) interlocutor&#10;structure modeling including reply-to utterance recognition, identical speaker&#10;searching and pointer consistency distinction, and (2) utterance semantics&#10;modeling including masked shared utterance restoration and shared node&#10;detection. We evaluate MPC-BERT on three downstream tasks including addressee&#10;recognition, speaker identification and response selection. Experimental&#10;results show that MPC-BERT outperforms previous methods by large margins and&#10;achieves new state-of-the-art performance on all three downstream tasks at two&#10;benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.08500" label="2203.08500">
        <attvalues>
          <attvalue for="0" value="HeterMPC: A Heterogeneous Graph Neural Network for Response Generation&#10;  in Multi-Party Conversations" />
          <attvalue for="1" value="  Recently, various response generation models for two-party conversations have&#10;achieved impressive improvements, but less effort has been paid to multi-party&#10;conversations (MPCs) which are more practical and complicated. Compared with a&#10;two-party conversation where a dialogue context is a sequence of utterances,&#10;building a response generation model for MPCs is more challenging, since there&#10;exist complicated context structures and the generated responses heavily rely&#10;on both interlocutors (i.e., speaker and addressee) and history utterances. To&#10;address these challenges, we present HeterMPC, a heterogeneous graph-based&#10;neural network for response generation in MPCs which models the semantics of&#10;utterances and interlocutors simultaneously with two types of nodes in a graph.&#10;Besides, we also design six types of meta relations with&#10;node-edge-type-dependent parameters to characterize the heterogeneous&#10;interactions within the graph. Through multi-hop updating, HeterMPC can&#10;adequately utilize the structural knowledge of conversations for response&#10;generation. Experimental results on the Ubuntu Internet Relay Chat (IRC)&#10;channel benchmark show that HeterMPC outperforms various baseline models for&#10;response generation in MPCs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.06677" label="2204.06677">
        <attvalues>
          <attvalue for="0" value="Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State&#10;  Tracking" />
          <attvalue for="1" value="  Dialogue State Tracking (DST) aims to keep track of users' intentions during&#10;the course of a conversation. In DST, modelling the relations among domains and&#10;slots is still an under-studied problem. Existing approaches that have&#10;considered such relations generally fall short in: (1) fusing prior slot-domain&#10;membership relations and dialogue-aware dynamic slot relations explicitly, and&#10;(2) generalizing to unseen domains. To address these issues, we propose a novel&#10;\textbf{D}ynamic \textbf{S}chema \textbf{G}raph \textbf{F}usion&#10;\textbf{Net}work (\textbf{DSGFNet}), which generates a dynamic schema graph to&#10;explicitly fuse the prior slot-domain membership relations and dialogue-aware&#10;dynamic slot relations. It also uses the schemata to facilitate knowledge&#10;transfer to new domains. DSGFNet consists of a dialogue utterance encoder, a&#10;schema graph encoder, a dialogue-aware schema graph evolving network, and a&#10;schema graph enhanced dialogue state decoder. Empirical results on benchmark&#10;datasets (i.e., SGD, MultiWOZ2.1, and MultiWOZ2.2), show that DSGFNet&#10;outperforms existing methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.11193" label="2108.11193">
        <attvalues>
          <attvalue for="0" value="Models In a Spelling Bee: Language Models Implicitly Learn the Character&#10;  Composition of Tokens" />
          <attvalue for="1" value="  Standard pretrained language models operate on sequences of subword tokens&#10;without direct access to the characters that compose each token's string&#10;representation. We probe the embedding layer of pretrained language models and&#10;show that models learn the internal character composition of whole word and&#10;subword tokens to a surprising extent, without ever seeing the characters&#10;coupled with the tokens. Our results show that the embedding layer of RoBERTa&#10;holds enough information to accurately spell up to a third of the vocabulary&#10;and reach high average character ngram overlap on all token types. We further&#10;test whether enriching subword models with additional character information can&#10;improve language modeling, and observe that this method has a near-identical&#10;learning curve as training without spelling-based enrichment. Overall, our&#10;results suggest that language modeling objectives incentivize the model to&#10;implicitly learn some notion of spelling, and that explicitly teaching the&#10;model how to spell does not appear to enhance its performance on such tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Contemporary subword tokenization algorithms such as BPE \cite{sennrich-etal-2016-neural} partition a string into contiguous spans of characters.&#10;Each span represents a frequent character $n$gram, from individual characters (a), through prefixes (uni) and suffixes (tion), and even complete words (cats).&#10;The tokenizer then converts each such span into a discrete symbol (a token) with no internal structure, effectively discarding the token's orthographic information.&#10;Therefore, a model operating over sequences of subword tokens should be oblivious to the spelling of each token.&#10;In this work, we show that despite having no direct access to the subwords' internal character composition, pretrained language models do learn some notion of spelling.&#10;&#10;To examine what pretrained language models learn about spelling, we present the SpellingBee probe. &#10;SpellingBee is a generative language model that predicts the character composition of a token given only its (uncontextualized) vector representation from the pretrained model's embeddings matrix.&#10;SpellingBee is trained on part of the model's vocabulary, and then tested by spelling unseen token types.&#10;If the probe can successfully reconstruct the correct character sequence from an unseen token's embedding, then there must be significant orthographic information encoded in the vector.&#10;&#10;We find that the embedding layers of several pretrained language models contain surprising amounts of character information. &#10;SpellingBee accurately spells 31.8\% of the held-out vocabulary for RoBERTa-Large \cite{liu2019roberta}, 32.9\% for GPT2-Medium \cite{radford2019language}, and 40.9\% for the Arabic language model AraBERT-Large \cite{antoun-etal-2020-arabert}.&#10;A softer metric that is sensitive to partially-correct spellings (chrF) \cite{popovic-2015-chrf} shows a similar trend, with 48.7 for RoBERTa-Large and 62.3 for AraBERT-Large.&#10;These results are much higher than the baseline of applying SpellingBee to randomly-initialized vectors, which fails to spell a single token.&#10;&#10;Given that subword models learn some notion of character composition to fulfill language modeling objectives, could they perhaps benefit from knowing the exact spelling of each token a priori?&#10;To that end, we reverse SpellingBee's role and use it to pretrain the embedding layer of a randomly-initialized model, thus imbuing each token representation with its orthographic information before training the whole model on the masked language modeling objective.&#10;We compare the pretraining process of the character-infused model to that of an identical model whose embedding layer is randomly initialized (and not pretrained), and find that both learning curves converge to virtually identical values within the first 1,000 gradient updates, a fraction of the total optimization process.&#10;This experiment suggests that while language models may need to learn some notion of spelling to optimize their objectives,&#10;they might also be able to quickly acquire most of the character-level information they need from plain token sequences without directly observing the composition of each token.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Character Composition Learning, Computational Linguistics, Artificial Intelligence, Language Modeling Objectives, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1312.3005" label="1312.3005">
        <attvalues>
          <attvalue for="0" value="One Billion Word Benchmark for Measuring Progress in Statistical&#10;  Language Modeling" />
          <attvalue for="1" value="  We propose a new benchmark corpus to be used for measuring progress in&#10;statistical language modeling. With almost one billion words of training data,&#10;we hope this benchmark will be useful to quickly evaluate novel language&#10;modeling techniques, and to compare their contribution when combined with other&#10;advanced techniques. We show performance of several well-known types of&#10;language models, with the best results achieved with a recurrent neural network&#10;based language model. The baseline unpruned Kneser-Ney 5-gram model achieves&#10;perplexity 67.6; a combination of techniques leads to 35% reduction in&#10;perplexity, or 10% reduction in cross-entropy (bits), over that baseline.&#10;  The benchmark is available as a code.google.com project; besides the scripts&#10;needed to rebuild the training/held-out data, it also makes available&#10;log-probability values for each word in each of ten held-out data sets, for&#10;each of the baseline n-gram models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Network Techniques, Benchmark Evaluation, Mathematics, Language Modeling, Statistics" />
        </attvalues>
      </node>
      <node id="2305.13230" label="2305.13230">
        <attvalues>
          <attvalue for="0" value="To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis" />
          <attvalue for="1" value="  Recent research has highlighted the importance of dataset size in scaling&#10;language models. However, large language models (LLMs) are notoriously&#10;token-hungry during pre-training, and high-quality text data on the web is&#10;approaching its scaling limit for LLMs. To further enhance LLMs, a&#10;straightforward approach is to repeat the pre-training data for additional&#10;epochs. In this study, we empirically investigate three key aspects under this&#10;approach. First, we explore the consequences of repeating pre-training data,&#10;revealing that the model is susceptible to overfitting, leading to multi-epoch&#10;degradation. Second, we examine the key factors contributing to multi-epoch&#10;degradation, finding that significant factors include dataset size, model&#10;parameters, and training objectives, while less influential factors consist of&#10;dataset quality and model FLOPs. Finally, we explore whether widely used&#10;regularization can alleviate multi-epoch degradation. Most regularization&#10;techniques do not yield significant improvements, except for dropout, which&#10;demonstrates remarkable effectiveness but requires careful tuning when scaling&#10;up the model size. Additionally, we discover that leveraging mixture-of-experts&#10;(MoE) enables cost-effective and efficient hyper-parameter tuning for&#10;computationally intensive dense LLMs with comparable trainable parameters,&#10;potentially impacting efficient LLM development on a broader scale.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Language Model Scaling, Efficient Training Techniques, Computer Science, Linguistics, Overfitting Mitigation, Mathematics" />
        </attvalues>
      </node>
      <node id="2403.16393" label="2403.16393">
        <attvalues>
          <attvalue for="0" value="Concurrent Linguistic Error Detection (CLED) for Large Language Models" />
          <attvalue for="1" value="  The wide adoption of Large language models (LLMs) makes their dependability a&#10;pressing concern. Detection of errors is the first step to mitigating their&#10;impact on a system and thus, efficient error detection for LLMs is an important&#10;issue. In many settings, the LLM is considered as a black box with no access to&#10;the internal nodes; this prevents the use of many error detection schemes that&#10;need access to the model's internal nodes. An interesting observation is that&#10;the output of LLMs in error-free operation should be valid and normal text.&#10;Therefore, when the text is not valid or differs significantly from normal&#10;text, it is likely that there is an error. Based on this observation we propose&#10;to perform Concurrent Linguistic Error Detection (CLED); this scheme extracts&#10;some linguistic features of the text generated by the LLM and feeds them to a&#10;concurrent classifier that detects errors. Since the proposed error detection&#10;mechanism only relies on the outputs of the model, then it can be used on LLMs&#10;in which there is no access to the internal nodes. The proposed CLED scheme has&#10;been evaluated on the T5 model when used for news summarization and on the&#10;OPUS-MT model when used for translation. In both cases, the same set of&#10;linguistic features has been used for error detection to illustrate the&#10;applicability of the proposed scheme beyond a specific case. The results show&#10;that CLED can detect most of the errors at a low overhead penalty. The use of&#10;the concurrent classifier also enables a trade-off between error detection&#10;effectiveness and its associated overhead, so providing flexibility to a&#10;designer.&#10;" />
          <attvalue for="2" value="&#10;&#10;L{arge} language models (LLMs) are becoming a centerpiece of many applications and services; moreover, this trend is expected to continue in the coming years \cite{LLMsurvey}. From the first popular models that had hundreds of millions of parameters such as BERT \cite{BERT} or T5 \cite{T5}, models have quickly evolved to billions and in some cases possibly trillions of parameters. Examples of LLMs include commercial models such as GPT \cite{openai2023gpt4} or Gemini \cite{geminiteam2023gemini} and open-source models such as LLama \cite{LLAMA2} or Mistral \cite{Mistral}. &#10;&#10;Their wide adoption makes their dependability a significant concern while their large size makes efficient error protection a priority. For example, a common approach to detect transient errors is to run the same code twice and compare the results; for LLMs this implies doubling the already large inference time and power dissipation. However, error detection and protection of LLMs have not been widely studied. Many works have considered the impact of errors on neural networks and their protection when implemented on different platforms because this is a key requirement for their use in safety-critical applications \cite{CNNs_errors}. The impact of errors on neural networks has been evaluated extensively both by simulation and by radiation testing \cite{NN_Errors_0},\cite{NN_errors1} showing that they have some intrinsic tolerance to errors, especially when a fixed-point representation is used for the parameters and arithmetic operations. The impact of errors on neural networks implemented on FPGAs has also been considered showing that the implementation of an ensemble of networks can be used to protect against errors \cite{NN_errors2}. As transformer-based LLMs become widely used in many applications, the next step is to study the impact of soft errors and design error detection and correction schemes to protect the LLMs. The impact of soft errors on BERT has been recently studied in \cite{BERT-softerrors} and the results have shown that even a single error bit flip can in some cases corrupt the output of the entire model when using floating point formats as commonly the case in GPU implementations. However, to the best of the authors' knowledge, the impact of soft errors on other popular models such as T5 \cite{T5} or OPUS-MT \cite{OPUS-MT} has not been studied. &#10;&#10;The detection of errors by using additional logic that operates concurrently with the main system is attractive because it can exploit features of the system for error detection and has been widely used in computing and signal processing systems \cite{CED},\cite{CED1},\cite{CED2}. Error detection using a concurrent classifier has been proposed for large-scale machine learning systems in \cite{CCED} and evaluated for BERT in question and answering and emotion classification, showing good detection capabilities. However, these schemes rely on having access to internal nodes of the model which is not the case for LLMs in many settings. In the case of LLMs when used to produce text, for example in summarization, translation, or conversational tasks, a possibility could be to use the properties of the text to detect errors. For example, errors that produce invalid sequences of characters or abnormal patterns could be detected by extracting few features of the generated text and comparing them with those of normal texts. The key observation is that the output of the LLMs (i.e., the text generated) should have some features that are determined by the rules of the language which may possibly be used to perform concurrent error detection or more precisely, language-based error detection. In this paper, this focus is pursued by proposing Concurrent Linguistic Error Detection (CLED) to detect errors based on the properties of text and evaluating it in T5 when used for summarization and in OPUS-MT when used for translation. The results show that CLED enables the concurrent detection of errors in LLMs through the features extracted from the generated output text without the necessity to access the LLM internal nodes. Both experiments show an accuracy exceeding 90\% in error detection, with mechanisms to configure the error detection rate by tuning the decision threshold while also increasing the false errors detected.&#10;&#10;The rest of this paper is organized as follows. Section \ref{Preliminaries} covers the preliminaries describing OPUS-MT and T5, concurrent error detection, and the error model considered. The dependability of the OPUS-MT and T5 is studied in Section \ref{Dependability} with fault injection experiments that identify the critical bits and the patterns of errors on the generated text. Section \ref{CLED} presents the proposed CLED describing the scheme by which it can be applied to detect errors in the generated text. CLED is evaluated in Section \ref{Evaluation} in terms of error detection effectiveness and the overhead introduced over the unprotected language model. Finally, the paper ends with the conclusion and future work in Section \ref{Conclusion}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Error Detection Methods, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2312.11805" label="2312.11805">
        <attvalues>
          <attvalue for="0" value="Gemini: A Family of Highly Capable Multimodal Models" />
          <attvalue for="1" value="  This report introduces a new family of multimodal models, Gemini, that&#10;exhibit remarkable capabilities across image, audio, video, and text&#10;understanding. The Gemini family consists of Ultra, Pro, and Nano sizes,&#10;suitable for applications ranging from complex reasoning tasks to on-device&#10;memory-constrained use-cases. Evaluation on a broad range of benchmarks shows&#10;that our most-capable Gemini Ultra model advances the state of the art in 30 of&#10;32 of these benchmarks - notably being the first model to achieve human-expert&#10;performance on the well-studied exam benchmark MMLU, and improving the state of&#10;the art in every one of the 20 multimodal benchmarks we examined. We believe&#10;that the new capabilities of the Gemini family in cross-modal reasoning and&#10;language understanding will enable a wide variety of use cases. We discuss our&#10;approach toward post-training and deploying Gemini models responsibly to users&#10;through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud&#10;Vertex AI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.02051" label="2206.02051">
        <attvalues>
          <attvalue for="0" value="Fast and Accurate Error Simulation for CNNs against Soft Errors" />
          <attvalue for="1" value="  The great quest for adopting AI-based computation for&#10;safety-/mission-critical applications motivates the interest towards methods&#10;for assessing the robustness of the application w.r.t. not only its&#10;training/tuning but also errors due to faults, in particular soft errors,&#10;affecting the underlying hardware. Two strategies exist: architecture-level&#10;fault injection and application-level functional error simulation. We present a&#10;framework for the reliability analysis of Convolutional Neural Networks (CNNs)&#10;via an error simulation engine that exploits a set of validated error models&#10;extracted from a detailed fault injection campaign. These error models are&#10;defined based on the corruption patterns of the output of the CNN operators&#10;induced by faults and bridge the gap between fault injection and error&#10;simulation, exploiting the advantages of both approaches. We compared our&#10;methodology against SASSIFI for the accuracy of functional error simulation&#10;w.r.t. fault injection, and against TensorFI in terms of speedup for the error&#10;simulation strategy. Experimental results show that our methodology achieves&#10;about 99\% accuracy of the fault effects w.r.t. SASSIFI, and a speedup ranging&#10;from 44x up to 63x w.r.t. TensorFI, that only implements a limited set of error&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.01820" label="2306.01820">
        <attvalues>
          <attvalue for="0" value="Concurrent Classifier Error Detection (CCED) in Large Scale Machine&#10;  Learning Systems" />
          <attvalue for="1" value="  The complexity of Machine Learning (ML) systems increases each year, with&#10;current implementations of large language models or text-to-image generators&#10;having billions of parameters and requiring billions of arithmetic operations.&#10;As these systems are widely utilized, ensuring their reliable operation is&#10;becoming a design requirement. Traditional error detection mechanisms introduce&#10;circuit or time redundancy that significantly impacts system performance. An&#10;alternative is the use of Concurrent Error Detection (CED) schemes that operate&#10;in parallel with the system and exploit their properties to detect errors. CED&#10;is attractive for large ML systems because it can potentially reduce the cost&#10;of error detection. In this paper, we introduce Concurrent Classifier Error&#10;Detection (CCED), a scheme to implement CED in ML systems using a concurrent ML&#10;classifier to detect errors. CCED identifies a set of check signals in the main&#10;ML system and feeds them to the concurrent ML classifier that is trained to&#10;detect errors. The proposed CCED scheme has been implemented and evaluated on&#10;two widely used large-scale ML models: Contrastive Language Image Pretraining&#10;(CLIP) used for image classification and Bidirectional Encoder Representations&#10;from Transformers (BERT) used for natural language applications. The results&#10;show that more than 95 percent of the errors are detected when using a simple&#10;Random Forest classifier that is order of magnitude simpler than CLIP or BERT.&#10;These results illustrate the potential of CCED to implement error detection in&#10;large-scale ML models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.13753" label="2402.13753">
        <attvalues>
          <attvalue for="0" value="LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens" />
          <attvalue for="1" value="  Large context window is a desirable feature in large language models (LLMs).&#10;However, due to high fine-tuning costs, scarcity of long texts, and&#10;catastrophic values introduced by new token positions, current extended context&#10;windows are limited to around 128k tokens. This paper introduces LongRoPE that,&#10;for the first time, extends the context window of pre-trained LLMs to an&#10;impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k&#10;training lengths, while maintaining performance at the original short context&#10;window. This is achieved by three key innovations: (i) we identify and exploit&#10;two forms of non-uniformities in positional interpolation through an efficient&#10;search, providing a better initialization for fine-tuning and enabling an 8x&#10;extension in non-fine-tuning scenarios; (ii) we introduce a progressive&#10;extension strategy that first fine-tunes a 256k length LLM and then conducts a&#10;second positional interpolation on the fine-tuned extended LLM to achieve a&#10;2048k context window; (iii) we readjust LongRoPE on 8k length to recover the&#10;short context window performance. Extensive experiments on LLaMA2 and Mistral&#10;across various tasks demonstrate the effectiveness of our method. Models&#10;extended via LongRoPE retain the original architecture with minor modifications&#10;to the positional embedding, and can reuse most pre-existing optimizations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;In addition to methods based on position interpolation, this sectiondiscusses related works of other approaches. &#10;&#10;Retrieval-based approaches use an external memory module to memorize long past context and retrieval modules for related documents fetching at inference~\cite{longllama,wang2023augmenting,borgeaud2022improving}. These designs typically need explicit modifications on the LLM architectures. Our work, in contrast, is more lightweight, with minor positional embedding modifications. We can also handle more long context tasks beyond retrieval, such as long document summarization and few-shot learning. &#10;&#10;Attention-based context window extensions. Beyond positional embedding interpolation, some research achieves input context extension using the original LLM context window length by manipulating attention mechanisms~\cite{han2023lminfinite, streamingllm,parallelwindow}. The key idea is to mitigate the attention explosion issue caused by new positions using novel attention masks. &#10;These efforts and positional interpolation methods are complementary.&#10;&#10;Fine-tuning based approaches focus on how to effectively fine-tune pre-trained LLMs with modified position embeddings for longer context. Works like Code LLaMA~\cite{codellama}, LLaMA2 Long~\cite{xiong2023effective} and ScaledRoPE~\cite{liu2023scaling} choose a very large base value for RoPE and fine-tune on the target length. Our method offers flexibility for various target lengths and can achieve beyond 2M length. More recently, as fine-tuning for long context lengths (i.e., over 128k) demands substantial GPU resources, LongLoRA~\cite{longlora} and PoSE~\cite{zhu2023pose} are proposed to mitigate this overhead. &#10;Our method is orthogonal to these efficient fine-tuning works. &#10;&#10;" />
          <attvalue for="4" value="Context Window Extension, Computer Science, Large Language Models, Linguistics, Fine-Tuning Optimization, Mathematics" />
        </attvalues>
      </node>
      <node id="2305.15282" label="2305.15282">
        <attvalues>
          <attvalue for="0" value="A Simple and Effective Framework for Strict Zero-Shot Hierarchical&#10;  Classification" />
          <attvalue for="1" value="  In recent years, large language models (LLMs) have achieved strong&#10;performance on benchmark tasks, especially in zero or few-shot settings.&#10;However, these benchmarks often do not adequately address the challenges posed&#10;in the real-world, such as that of hierarchical classification. In order to&#10;address this challenge, we propose refactoring conventional tasks on&#10;hierarchical datasets into a more indicative long-tail prediction task. We&#10;observe LLMs are more prone to failure in these cases. To address these&#10;limitations, we propose the use of entailment-contradiction prediction in&#10;conjunction with LLMs, which allows for strong performance in a strict&#10;zero-shot setting. Importantly, our method does not require any parameter&#10;updates, a resource-intensive process and achieves strong performance across&#10;multiple datasets.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) with parameters in the order of billions \cite{brown_language_2020} have gained significant attention in recent years due to their strong performance on a wide range of natural language processing tasks. These models have achieved impressive results on benchmarks \cite{ particularly in zero or few-shot settings, where they are able to generalize to new tasks and languages with little to no training data. There is, however a difficulty in tuning parameters of these large-scale models due to resource limitations. Additionally, the focus on benchmarks has led to the neglect of real-world challenges, such as that of hierarchical classification. As a result, the long-tail problem \cite{gzs-longtail} has been overlooked. This occurs when a vast number of rare classes occur in the presence of frequent classes for many natural language problems.&#10;&#10;In many industrial real-world applications, a strong performing method for hierarchical classification can be of direct utility. New product categories are emerging in e-commerce platforms. Existing categories, on the other hand, may not be very intuitive for customers. For example, upon browsing categories such as night creams, we may be unable to find a product in a sibling-node category of creams. This is further highlighted by platforms in which a systematic structure is not created for users; parent nodes may be in place of child nodes, and vice versa \cite{DBLP:journals/corr/Asghar16}. Manually categorizing product categories can be a costly redesigning endeavour. To tackle this problem, we suggest refactoring traditional hierarchical flat-labeled prediction tasks \cite{liu_improving_2021} to a more indicative long-tail prediction task. This involves structuring the classification task to closely reflect the real-world long-tail distribution of classes. In doing so, we are enabled to leverage LLMs for long-tail prediction tasks in a strict zero-shot classification setting. Through a series of experiments, results in this work show that our proposed method is able to significantly improve the performance over the baseline in several datasets, and holds promise for addressing the long-tail problem in real-world applications. The contributions of this work can be summarized as follows:&#10;\begin{itemize}[leftmargin=*]&#10; \item We refactor real-world hierarchical taxonomy datasets into long-tailed problems. In doing so, we create a strong testbed to evaluate ``strict zero-shot classification&quot; with LLMs.&#10; \item We explore utilizing LLMs to enhance the capabilities of entailment-contradiction predictors for long-tail classification. This results in strong capabilities of performing model inference without resource-intensive parameter updates.&#10; \item We show through quantitative empirical evidence, that our proposed method is able to overcome limitations of stand-alone large language models. Our method obtains strong performance on long-tail classification tasks.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Zero-Shot Learning, Language Model Limitations, Artificial Intelligence, Hierarchical Classification" />
        </attvalues>
      </node>
      <node id="2004.02235" label="2004.02235">
        <attvalues>
          <attvalue for="0" value="From Generalized zero-shot learning to long-tail with class descriptors" />
          <attvalue for="1" value="  Real-world data is predominantly unbalanced and long-tailed, but deep models&#10;struggle to recognize rare classes in the presence of frequent classes. Often,&#10;classes can be accompanied by side information like textual descriptions, but&#10;it is not fully clear how to use them for learning with unbalanced long-tail&#10;data. Such descriptions have been mostly used in (Generalized) Zero-shot&#10;learning (ZSL), suggesting that ZSL with class descriptions may also be useful&#10;for long-tail distributions. We describe DRAGON, a late-fusion architecture for&#10;long-tail learning with class descriptors. It learns to (1) correct the bias&#10;towards head classes on a sample-by-sample basis; and (2) fuse information from&#10;class-descriptions to improve the tail-class accuracy. We also introduce new&#10;benchmarks CUB-LT, SUN-LT, AWA-LT for long-tail learning with&#10;class-descriptions, building on existing learning-with-attributes datasets and&#10;a version of Imagenet-LT with class descriptors. DRAGON outperforms&#10;state-of-the-art models on the new benchmark. It is also a new SoTA on existing&#10;benchmarks for GFSL with class descriptors (GFSL-d) and standard (vision-only)&#10;long-tailed learning ImageNet-LT, CIFAR-10, 100, and Places365.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1605.05362" label="1605.05362">
        <attvalues>
          <attvalue for="0" value="Yelp Dataset Challenge: Review Rating Prediction" />
          <attvalue for="1" value="  Review websites, such as TripAdvisor and Yelp, allow users to post online&#10;reviews for various businesses, products and services, and have been recently&#10;shown to have a significant influence on consumer shopping behaviour. An online&#10;review typically consists of free-form text and a star rating out of 5. The&#10;problem of predicting a user's star rating for a product, given the user's text&#10;review for that product, is called Review Rating Prediction and has lately&#10;become a popular, albeit hard, problem in machine learning. In this paper, we&#10;treat Review Rating Prediction as a multi-class classification problem, and&#10;build sixteen different prediction models by combining four feature extraction&#10;methods, (i) unigrams, (ii) bigrams, (iii) trigrams and (iv) Latent Semantic&#10;Indexing, with four machine learning algorithms, (i) logistic regression, (ii)&#10;Naive Bayes classification, (iii) perceptrons, and (iv) linear Support Vector&#10;Classification. We analyse the performance of each of these sixteen models to&#10;come up with the best model for predicting the ratings from reviews. We use the&#10;dataset provided by Yelp for training and testing the models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.01666" label="2104.01666">
        <attvalues>
          <attvalue for="0" value="Improving Pretrained Models for Zero-shot Multi-label Text&#10;  Classification through Reinforced Label Hierarchy Reasoning" />
          <attvalue for="1" value="  Exploiting label hierarchies has become a promising approach to tackling the&#10;zero-shot multi-label text classification (ZS-MTC) problem. Conventional&#10;methods aim to learn a matching model between text and labels, using a graph&#10;encoder to incorporate label hierarchies to obtain effective label&#10;representations \cite{rios2018few}. More recently, pretrained models like BERT&#10;\cite{devlin2018bert} have been used to convert classification tasks into a&#10;textual entailment task \cite{yin-etal-2019-benchmarking}. This approach is&#10;naturally suitable for the ZS-MTC task. However, pretrained models are&#10;underexplored in the existing work because they do not generate individual&#10;vector representations for text or labels, making it unintuitive to combine&#10;them with conventional graph encoding methods. In this paper, we explore to&#10;improve pretrained models with label hierarchies on the ZS-MTC task. We propose&#10;a Reinforced Label Hierarchy Reasoning (RLHR) approach to encourage&#10;interdependence among labels in the hierarchies during training. Meanwhile, to&#10;overcome the weakness of flat predictions, we design a rollback algorithm that&#10;can remove logical errors from predictions during inference. Experimental&#10;results on three real-life datasets show that our approach achieves better&#10;performance and outperforms previous non-pretrained methods on the ZS-MTC task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.05940" label="2305.05940">
        <attvalues>
          <attvalue for="0" value="Multilingual LLMs are Better Cross-lingual In-context Learners with&#10;  Alignment" />
          <attvalue for="1" value="  In-context learning (ICL) unfolds as large language models become capable of&#10;inferring test labels conditioned on a few labeled samples without any gradient&#10;update. ICL-enabled large language models provide a promising step forward&#10;toward bypassing recurrent annotation costs in a low-resource setting. Yet,&#10;only a handful of past studies have explored ICL in a cross-lingual setting, in&#10;which the need for transferring label-knowledge from a high-resource language&#10;to a low-resource one is immensely crucial. To bridge the gap, we provide the&#10;first in-depth analysis of ICL for cross-lingual text classification. We find&#10;that the prevalent mode of selecting random input-label pairs to construct the&#10;prompt-context is severely limited in the case of cross-lingual ICL, primarily&#10;due to the lack of alignment in the input as well as the output spaces. To&#10;mitigate this, we propose a novel prompt construction strategy -- Cross-lingual&#10;In-context Source-Target Alignment (X-InSTA). With an injected coherence in the&#10;semantics of the input examples and a task-based alignment across the source&#10;and target languages, X-InSTA is able to outperform random prompt selection by&#10;a large margin across three different tasks using 44 different cross-lingual&#10;pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In-context learning (ICL): &#10;\cite{brown2020language} introduced a new approach, called in-context few-shot learning using the GPT-3 model. &#10;Subsequent efforts have been made to enhance the effectiveness of ICL. \cite{hendrycks2020} evaluated the breadth and depth of model understanding to determine its weaknesses and strengths. Techniques such as selecting semantically-similar examples, using differentiable soft prompts for backpropagation, and adjusting prompts to eliminate bias in predictions have been implemented to optimize the input prompt \cite{liu-etal-2022-makes, zhang2021differentiable, zhao2021}. These efforts have primarily been directed toward improving the performance of ICL in a monolingual setting.&#10;&#10;Multiple recent studies have sought to explain the emergence of ICL by assigning different roles to the LLM. \cite{icl-bayesian} provided the notion of LLMs doing Bayesian inference conditioned upon the prompt context to predict the test label. Our work is much in line with this hypothetical model since alignment over the semantics and the task-based signals across languages are motivated by the quest for better alignment between the prompt and the pretraining distribution and warranting a shared, distinguishable concept as \cite{icl-bayesian} argued.&#10;Additionally, \cite{icl-gradient-descent} sought to identify LLMs doing gradient-descent as meta-optimizers while learning in context. \cite{icl-algorithms} described ICL as implicit model selection.&#10;&#10;Multilingual models: Recent studies on multilingual tasks have focused on creating multilingual versions of popular pre-trained language models. These include mBERT \cite{devlin-2019}, mBART \cite{liu-2020}, XLM-R \cite{conneau-etal-2020-unsupervised}, and mT5 \cite{xue2021}, which are derived from models like BERT \cite{devlin-2019}, BART \cite{lewis-etal-2020-bart}, RoBERTa \cite{liu-2019}, and T5 \cite{raffel-2019}, respectively. However, fine-tuning these large models for each task is infeasible due to computational limitations. While ICL has been attempted for cross-lingual downstream tasks, these methods only involve random sampling of demonstrations for prompt construction \cite{zhang2021differentiable, winata-etal-2021-language}. \cite{shi2022xricl} addressed the problem of cross-lingual text-to-sql conversion using ICL. However, their method relies on translating the input text in the source language to the target language before generating the corresponding SQL code. \cite{MTsimilarICL} demonstrated the effects of similar example selection in a few-shot machine translation setting which is much similar to our proposed semantic alignment. To the best of our knowledge, there is no study on optimizing prompts for cross-lingual NLP tasks using ICL.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, In-Context Learning Strategies, Language Model Learning, Cross-Lingual Text Classification, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2009.14124" label="2009.14124">
        <attvalues>
          <attvalue for="0" value="Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank" />
          <attvalue for="1" value="  Pretrained multilingual contextual representations have shown great success,&#10;but due to the limits of their pretraining data, their benefits do not apply&#10;equally to all language varieties. This presents a challenge for language&#10;varieties unfamiliar to these models, whose labeled \emph{and unlabeled} data&#10;is too limited to train a monolingual model effectively. We propose the use of&#10;additional language-specific pretraining and vocabulary augmentation to adapt&#10;multilingual models to low-resource settings. Using dependency parsing of four&#10;diverse low-resource language varieties as a case study, we show that these&#10;methods significantly improve performance over baselines, especially in the&#10;lowest-resource cases, and demonstrate the importance of the relationship&#10;between such models' pretraining data and target language varieties.&#10;" />
          <attvalue for="2" value="&#10;&#10;Contextual word representations (CWRs) from pretrained language models have improved many NLP systems. Such language models include BERT \cite{devlin-etal-2019-bert} and ELMo \cite{peters-etal-2018-deep}, which are conventionally ``pretrained'' on large unlabeled datasets before their internal representations are ``finetuned'' during supervised training on downstream tasks like parsing. However, many language varieties lack large annotated and even unannotated datasets, raising questions about the broad applicability of such data-hungry methods.&#10;&#10;One exciting way to compensate for the lack of unlabeled data in low-resource language varieties is to finetune a large, multilingual language model that has been pretrained on the union of many languages' data \cite{devlin-etal-2019-bert,lample2019cross}. This enables the model to transfer some of what it learns from high-resource languages to low-resource ones, demonstrating benefits over monolingual methods in some cases \cite{conneau2019unsupervised,tsai-etal-2019-small}, though not always \cite{agerri2020text,ronnqvist-etal-2019-multilingual}.&#10;&#10;Specifically, multilingual models face the transfer-dilution tradeoff \cite{conneau2019unsupervised}: increasing the number of languages during pretraining improves positive crosslingual transfer but decreases the model capacity allocated to each language. Furthermore, such models are only pretrained on a finite amount of data and may lack exposure to specialized domains of certain languages or even entire low-resource language varieties. The result is a challenge for these language varieties, which must rely on positive transfer from a sufficient number of similar high-resource languages. Indeed, \cite{wu-dredze-2020-languages} find that multilingual models often underperform monolingual baselines for such languages and question their off-the-shelf viability.&#10;&#10;We take inspiration from previous work on domain adaptation, where general-purpose monolingual models have been effectively adapted to specialized domains through additional pretraining on domain-specific corpora \cite{dontstoppretraining2020}. We hypothesize that we can improve the performance of multilingual models on low-resource language varieties analogously, through additional pretraining on language-specific corpora.&#10;&#10;However, additional pretraining on more data in the target language does not ensure its full representation in the model's vocabulary, which is constructed to maximally represent the model's original pretraining data \cite{sennrich-etal-2016-neural,wu2016googles}. &#10;\cite{artetxe-etal-2020-cross} find that target languages' representation in the vocabulary affects these models' transferability,&#10;suggesting that language varieties on the fringes of the vocabulary may not be sufficiently well-modeled. Can we incorporate vocabulary from the target language into multilingual models' existing alignment?&#10;&#10;We introduce the use of additional language-specific pretraining for multilingual CWRs in a low-resource setting, before use in a downstream task; to better model language-specific tokens, we also augment the existing vocabulary with frequent tokens from the low-resource language (\S \ref{sec:method}). Our experiments consider dependency parsing in four typologically diverse low-resource language varieties with different degrees of relatedness to a multilingual model's pretraining data (\S \ref{sec:experiments}). Our results show that these methods consistently improve performance on each target variety, especially in the lowest-resource cases (\S \ref{sec:results}). In doing so, we demonstrate the importance of accounting for the relationship between a multilingual model's pretraining data and the target language variety. &#10;&#10;Because the pretraining-finetuning paradigm is now ubiquitous, many experimental findings for one task can now inform work on other tasks. Thus, our findings on dependency parsing---whose annotated datasets cover many more low-resource language varieties than those of other NLP tasks---are expected to interest researchers and practitioners facing low-resource situations for other tasks. To this end, we make our code, data, and hyperparameters publicly available.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Pretraining Methods, Computer Science, Linguistics, Low-Resource Languages, Multilingual Models, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2004.00033" label="2004.00033">
        <attvalues>
          <attvalue for="0" value="Give your Text Representation Models some Love: the Case for Basque" />
          <attvalue for="1" value="  Word embeddings and pre-trained language models allow to build rich&#10;representations of text and have enabled improvements across most NLP tasks.&#10;Unfortunately they are very expensive to train, and many small companies and&#10;research groups tend to use models that have been pre-trained and made&#10;available by third parties, rather than building their own. This is suboptimal&#10;as, for many languages, the models have been trained on smaller (or lower&#10;quality) corpora. In addition, monolingual pre-trained models for non-English&#10;languages are not always available. At best, models for those languages are&#10;included in multilingual versions, where each language shares the quota of&#10;substrings and parameters with the rest of the languages. This is particularly&#10;true for smaller languages such as Basque. In this paper we show that a number&#10;of monolingual models (FastText word embeddings, FLAIR and BERT language&#10;models) trained with larger Basque corpora produce much better results than&#10;publicly available versions in downstream NLP tasks, including topic&#10;classification, sentiment classification, PoS tagging and NER. This work sets a&#10;new state-of-the-art in those tasks for Basque. All benchmarks and models used&#10;in this work are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09093" label="2005.09093">
        <attvalues>
          <attvalue for="0" value="Are All Languages Created Equal in Multilingual BERT?" />
          <attvalue for="1" value="  Multilingual BERT (mBERT) trained on 104 languages has shown surprisingly&#10;good cross-lingual performance on several NLP tasks, even without explicit&#10;cross-lingual signals. However, these evaluations have focused on cross-lingual&#10;transfer with high-resource languages, covering only a third of the languages&#10;covered by mBERT. We explore how mBERT performs on a much wider set of&#10;languages, focusing on the quality of representation for low-resource&#10;languages, measured by within-language performance. We consider three tasks:&#10;Named Entity Recognition (99 languages), Part-of-speech Tagging, and Dependency&#10;Parsing (54 languages each). mBERT does better than or comparable to baselines&#10;on high resource languages but does much worse for low resource languages.&#10;Furthermore, monolingual BERT models for these languages do even worse. Paired&#10;with similar languages, the performance gap between monolingual BERT and mBERT&#10;can be narrowed. We find that better models for low resource languages require&#10;more efficient pretraining techniques or more data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1609.08144" label="1609.08144">
        <attvalues>
          <attvalue for="0" value="Google's Neural Machine Translation System: Bridging the Gap between&#10;  Human and Machine Translation" />
          <attvalue for="1" value="  Neural Machine Translation (NMT) is an end-to-end learning approach for&#10;automated translation, with the potential to overcome many of the weaknesses of&#10;conventional phrase-based translation systems. Unfortunately, NMT systems are&#10;known to be computationally expensive both in training and in translation&#10;inference. Also, most NMT systems have difficulty with rare words. These issues&#10;have hindered NMT's use in practical deployments and services, where both&#10;accuracy and speed are essential. In this work, we present GNMT, Google's&#10;Neural Machine Translation system, which attempts to address many of these&#10;issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder&#10;layers using attention and residual connections. To improve parallelism and&#10;therefore decrease training time, our attention mechanism connects the bottom&#10;layer of the decoder to the top layer of the encoder. To accelerate the final&#10;translation speed, we employ low-precision arithmetic during inference&#10;computations. To improve handling of rare words, we divide words into a limited&#10;set of common sub-word units (&quot;wordpieces&quot;) for both input and output. This&#10;method provides a good balance between the flexibility of &quot;character&quot;-delimited&#10;models and the efficiency of &quot;word&quot;-delimited models, naturally handles&#10;translation of rare words, and ultimately improves the overall accuracy of the&#10;system. Our beam search technique employs a length-normalization procedure and&#10;uses a coverage penalty, which encourages generation of an output sentence that&#10;is most likely to cover all the words in the source sentence. On the WMT'14&#10;English-to-French and English-to-German benchmarks, GNMT achieves competitive&#10;results to state-of-the-art. Using a human side-by-side evaluation on a set of&#10;isolated simple sentences, it reduces translation errors by an average of 60%&#10;compared to Google's phrase-based production system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.04931" label="2302.04931">
        <attvalues>
          <attvalue for="0" value="In-Context Learning with Many Demonstration Examples" />
          <attvalue for="1" value="  Large pre-training language models (PLMs) have shown promising in-context&#10;learning abilities. However, due to the backbone transformer architecture,&#10;existing PLMs are bottlenecked by the memory and computational cost when&#10;scaling up to a large context size, leaving instruction tuning and in-context&#10;learning of many demonstration examples, as well as long-range language&#10;modeling under-explored. In this study, we propose a long-range language model&#10;EVALM based on an efficient transformer mechanism. EVALM is trained with 8k&#10;tokens per batch line and can test up to 256k-lengthed contexts with&#10;extrapolation, 128 times to the limit of existing PLMs (e.g. GPT3). Based on&#10;EVALM, we scale up the size of examples efficiently in both instruction tuning&#10;and in-context learning to explore the boundary of the benefits from more&#10;annotated data. Experimental results on a diverse set of tasks show that EVALM&#10;achieves 4.1% higher accuracy on average, and the average length of achieving&#10;the best accuracy score over tasks is around 12k. We find that in-context&#10;learning can achieve higher performance with more demonstrations under&#10;many-shot instruction tuning (8k), and further extending the length of&#10;instructions (16k) can further improve the upper bound of scaling in-context&#10;learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-trained Language Model&#10;PLMs are trained on large and general corpora and then finetuned or few-shot transferred to perform various NLU and NLG tasks. Among them, besides encoder-decoder Transformer~\cite{vaswani2017attention} architecture such as T5~\cite{raffel2020exploring}, there are auto-regressively pre-trained models, like XLNet~\cite{yang2019xlnet}, GPT~\cite{radford2019language, brown2020gpt3, black_gpt-neox-20b_2022}, OPT~\cite{zhang2022opt}, PaLM~\cite{chowdhery2022palm}, BLOOM~\cite{scao-2022-bloom}, and etc. These decoder-based causal language models soon occupy kinds of NLP leaderboards, showing excellent language modeling and in-context learning ability of them. However, the huge computing overhead (including memory and time consumption) makes nonprofits and smaller labs difficult to create or even use PLMs. Furthermore, this also prevents PLMs from encoding longer inputs. &#10;&#10;Efficient Attention&#10;A surge of efficient attention models are devised to enhance the efficiency of the original Transformer model~\cite{vaswani2017attention}.&#10;These models explore diverse philosophies to improve the efficiency, including sparse attention matrix~\cite{luong2015effective, tay2020sparse, beltagy2020Longformer, zaheer2020big, ainslie2020etc}, memory compression~\cite{liu2018generating, lee2019set, rae2020Compressive, wang2020linformer} low-rank decomposition~\cite{xiong2021nystromformer, lu2021soft, chen2021skyformer}, kernel-based linear attention~\cite{choromanski2021rethinking, peng2021random, peng2022abc, zheng2022linear, zheng2023efficient}, state-space model~\cite{gu2022efficiently, gupta2022diagonal, dao2022hungry}, and CUDA re-implementation~\cite{dao2022flashattention}. Thus models with efficient attention architecture are promising to handle longer input sequences when memory consumption is saved. H3~\cite{dao2022hungry} is pre-trained as an efficient language model but fails to scale up the training sequence length which remains 2048.&#10;&#10;In-Context Learning&#10;With the increasing scale and capacity of PLMs, ICL has become a new paradigm for NLP~\cite{brown2020gpt3}. The success of ICL has been demonstrated on a wide range of NLP tasks, including question answering~\cite{joshi2017triviaqa}, information retrieval~\cite{tay2022transformer}, math word problem~\cite{cobbe2021training}, commonsense reasoning~\cite{geva2021did}, and fact checking~\cite{rae2021scaling} etc. Several recent studies~\cite{liu2022makes,wu2022self} have observed a positive correlation between the number of in-context examples and ICL's performance: increasing the number of in-context examples can bring steady improvements. &#10;Further investigation is carried out to pack and/or distill more examples into the context through continued pre-training~\cite{choi2022prompt}, and instruction tuning~\cite{snell2022learning}. However, the input length limitation of current PLMs still restricts us from directly feeding more in-context examples into the model.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Instruction Tuning Methods, Linguistics, Cognitive Science, Language Model Architecture, Large Context Learning, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2405.10650" label="2405.10650">
        <attvalues>
          <attvalue for="0" value="SPOR: A Comprehensive and Practical Evaluation Method for Compositional&#10;  Generalization in Data-to-Text Generation" />
          <attvalue for="1" value="  Compositional generalization is an important ability of language models and&#10;has many different manifestations. For data-to-text generation, previous&#10;research on this ability is limited to a single manifestation called&#10;Systematicity and lacks consideration of large language models (LLMs), which&#10;cannot fully cover practical application scenarios. In this work, we propose&#10;SPOR, a comprehensive and practical evaluation method for compositional&#10;generalization in data-to-text generation. SPOR includes four aspects of&#10;manifestations (Systematicity, Productivity, Order invariance, and Rule&#10;learnability) and allows high-quality evaluation without additional manual&#10;annotations based on existing datasets. We demonstrate SPOR on two different&#10;datasets and evaluate some existing language models including LLMs. We find&#10;that the models are deficient in various aspects of the evaluation and need&#10;further improvement. Our work shows the necessity for comprehensive research on&#10;different manifestations of compositional generalization in data-to-text&#10;generation and provides a framework for evaluation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Data-to-text generation \cite{gen-survey} is an important task in natural language generation (NLG). It aims to generate fluent and faithful text based on structured data input and is critical in many NLG systems, such as report generation \cite{rotowire}, oriented dialogues \cite{Self}, etc. In data-to-text generation, structured data input is compositional, i.e., it can be considered as a combination of elements formed according to certain rules. Therefore, in order to handle the practical data-to-text generation, the language models should have the ability to recombine previously learned elements with certain rules to map new inputs made up from these elements to their correct output \cite{summary}, which is the so-called compositional generalization.&#10;Compositional generalization is an important ability of language models for many tasks. In semantic parsing and mathematical reasoning tasks, many different manifestations of this ability have been studied \cite{PCFG, Trans}, such as systematicity (handle combinations unseen during training), productivity (extrapolate to longer sequences than those seen during training), etc. For compositional generalization in data-to-text generation, only systematicity receives attention \cite{Self}, and research on other manifestations is lacking. The single systematic manifestation cannot fully cover practical application scenarios of compositional generalization and cannot comprehensively reflect this ability of language models in data-to-text generation. Although research on different manifestations of compositional generalization in data-to-text generation is necessary, there is currently no comprehensive evaluation method to support such research.&#10;&#10;To solve this problem, we propose SPOR, a comprehensive and practical evaluation method for compositional generalization in a data-to-text generation. Based on the manifestations of compositional generalization mentioned in \cite{PCFG}, SPOR includes four aspects of compositional generalization in data-to-text generation:&#10;&#10;\begin{itemize}[itemsep=2pt,topsep=2pt,parsep=2pt]&#10;&#10;\item[$\bullet$]Systematicity. The ability to handle data combinations unseen during training.&#10;&#10;\item[$\bullet$]Productivity. The ability to handle a larger amount of data within a sample than seen during training.&#10;&#10;\item[$\bullet$]Order invariance. The ability to maintain the fidelity and proper data ordering of the output text when the input order of data in an unordered set is changed.&#10;&#10;\item[$\bullet$]Rule learnability. The ability to actually learn and apply copy rule for generation, rather than memorize specific mappings.&#10;&#10;\end{itemize}&#10;&#10;For each aspect, we propose the corresponding methods for dataset construction and evaluation. Based on existing datasets, we mainly perform repartition \cite{CFQ} and element modification to construct datasets for our evaluation. Overall, the evaluation method SPOR has the following properties:&#10;&#10;\begin{itemize}[itemsep=2pt,topsep=2pt,parsep=2pt]&#10;&#10;\item[$\bullet$]Necessity. The ability or property in each aspect manifests compositional generalization and is required by the model for practical data-to-text generation.&#10;&#10;\item[$\bullet$]High evaluation quality. For each aspect, the evaluation method can effectively evaluate the corresponding ability or property.&#10;&#10;\item[$\bullet$]Low construction cost. Based on existing datasets, the dataset used for evaluation does not require additional manual annotation and can be constructed automatically. &#10;&#10;\end{itemize}&#10;&#10;We demonstrate SPOR on two existing datasets for data-to-text generation and evaluate some existing language models. Previous research on compositional generalization in data-to-text generation lacks consideration of large language models (LLMs) due to the lack of methods to directly fine-tune and apply LLMs to data-to-text generation in the past. Nowadays, advanced Parameter-Efficient Fine-Tuning such as LoRA \cite{LoRA} provides the methods, and the consideration of LLMs becomes necessary. Therefore, we include some advanced LLMs in our evaluation to partially fill the gap in previous research.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Data-to-Text Generation, Compositional Generalization, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1703.09902" label="1703.09902">
        <attvalues>
          <attvalue for="0" value="Survey of the State of the Art in Natural Language Generation: Core&#10;  tasks, applications and evaluation" />
          <attvalue for="1" value="  This paper surveys the current state of the art in Natural Language&#10;Generation (NLG), defined as the task of generating text or speech from&#10;non-linguistic input. A survey of NLG is timely in view of the changes that the&#10;field has undergone over the past decade or so, especially in relation to new&#10;(usually data-driven) methods, as well as new applications of NLG technology.&#10;This survey therefore aims to (a) give an up-to-date synthesis of research on&#10;the core tasks in NLG and the architectures adopted in which such tasks are&#10;organised; (b) highlight a number of relatively recent research topics that&#10;have arisen partly as a result of growing synergies between NLG and other areas&#10;of artificial intelligence; (c) draw attention to the challenges in NLG&#10;evaluation, relating them to similar challenges faced in other areas of Natural&#10;Language Processing, with an emphasis on different evaluation methods and the&#10;relationships between them.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.08052" label="1707.08052">
        <attvalues>
          <attvalue for="0" value="Challenges in Data-to-Document Generation" />
          <attvalue for="1" value="  Recent neural models have shown significant progress on the problem of&#10;generating short descriptive texts conditioned on a small number of database&#10;records. In this work, we suggest a slightly more difficult data-to-text&#10;generation task, and investigate how effective current approaches are on this&#10;task. In particular, we introduce a new, large-scale corpus of data records&#10;paired with descriptive documents, propose a series of extractive evaluation&#10;methods for analyzing performance, and obtain baseline results using current&#10;neural generation methods. Experiments show that these models produce fluent&#10;text, but fail to convincingly approximate human-generated documents. Moreover,&#10;even templated baselines exceed the performance of these neural models on some&#10;metrics, though copy- and reconstruction-based extensions lead to noticeable&#10;improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08467" label="2110.08467">
        <attvalues>
          <attvalue for="0" value="Improving Compositional Generalization with Self-Training for&#10;  Data-to-Text Generation" />
          <attvalue for="1" value="  Data-to-text generation focuses on generating fluent natural language&#10;responses from structured meaning representations (MRs). Such representations&#10;are compositional and it is costly to collect responses for all possible&#10;combinations of atomic meaning schemata, thereby necessitating few-shot&#10;generalization to novel MRs. In this work, we systematically study the&#10;compositional generalization of the state-of-the-art T5 models in few-shot&#10;data-to-text tasks. We show that T5 models fail to generalize to unseen MRs,&#10;and we propose a template-based input representation that considerably improves&#10;the model's generalization capability. To further improve the model's&#10;performance, we propose an approach based on self-training using fine-tuned&#10;BLEURT for pseudo response selection. On the commonly-used SGD and Weather&#10;benchmarks, the proposed self-training approach improves tree accuracy by 46%+&#10;and reduces the slot error rates by 73%+ over the strong T5 baselines in&#10;few-shot settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Data-to-Text Generation&#10;While early research focused on rule-based methods \cite{reiter2000building}, more recent work has relied heavily on neural methods \cite{wen2015semantically,marcheggiani2018deep}.&#10;Some recent works (\cite{kale2020text}, \cite{peng2020few}, \cite{kale2020machine}) showed that transfer learning from pre-trained language models can improve generalization capabilities and sample efficiency. In other lines of work, \cite{ferreira2019neural, moryossef2019step} find that pipelined neural approaches with explicit planning steps can outperform their end-to-end counterparts, while \cite{kale2020template} and \cite{du2020schema} showed the benefits of schema and template guided input representations. Inspired by \cite{kale2020template} we propose a simple and generic way to produce text-to-text representation, and study how it impacts compositional generalization.&#10;&#10;Self-training for NLG \cite{he2019revisiting} revisits the problem of self-training for NLG. They found that noise (from perturbing the input space) helps in self-training and propose a ``noisy'' version of self-training by augmenting vanilla training with the inputs from a reconstruction model. Building on this idea, the contemporary work \cite{heidari2021getting} on few-shot data-to-text generation proposes to self-train the model and shows efficacy on the Weather dataset. Another contemporary work \cite{li2021self} proposes to use constrained decoding to generate valid pseudo-responses for self-training and show convincing benefits. However, our work focuses on compositional generalization, rather than the pure few-shot learning setup. We propose a BLEURT-based self-training method, which is more generic than pseudo-response selection methods that rely on output structures.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2210.03050" label="2210.03050">
        <attvalues>
          <attvalue for="0" value="State-of-the-art generalisation research in NLP: A taxonomy and review" />
          <attvalue for="1" value="  The ability to generalise well is one of the primary desiderata of natural&#10;language processing (NLP). Yet, what 'good generalisation' entails and how it&#10;should be evaluated is not well understood, nor are there any evaluation&#10;standards for generalisation. In this paper, we lay the groundwork to address&#10;both of these issues. We present a taxonomy for characterising and&#10;understanding generalisation research in NLP. Our taxonomy is based on an&#10;extensive literature review of generalisation research, and contains five axes&#10;along which studies can differ: their main motivation, the type of&#10;generalisation they investigate, the type of data shift they consider, the&#10;source of this data shift, and the locus of the shift within the modelling&#10;pipeline. We use our taxonomy to classify over 400 papers that test&#10;generalisation, for a total of more than 600 individual experiments.&#10;Considering the results of this review, we present an in-depth analysis that&#10;maps out the current state of generalisation research in NLP, and we make&#10;recommendations for which areas might deserve attention in the future. Along&#10;with this paper, we release a webpage where the results of our review can be&#10;dynamically explored, and which we intend to update as new NLP generalisation&#10;studies are published. With this work, we aim to take steps towards making&#10;state-of-the-art generalisation testing the new status quo in NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.09713" label="1912.09713">
        <attvalues>
          <attvalue for="0" value="Measuring Compositional Generalization: A Comprehensive Method on&#10;  Realistic Data" />
          <attvalue for="1" value="  State-of-the-art machine learning methods exhibit limited compositional&#10;generalization. At the same time, there is a lack of realistic benchmarks that&#10;comprehensively measure this ability, which makes it challenging to find and&#10;evaluate improvements. We introduce a novel method to systematically construct&#10;such benchmarks by maximizing compound divergence while guaranteeing a small&#10;atom divergence between train and test sets, and we quantitatively compare this&#10;method to other approaches for creating compositional generalization&#10;benchmarks. We present a large and realistic natural language question&#10;answering dataset that is constructed according to this method, and we use it&#10;to analyze the compositional generalization ability of three machine learning&#10;architectures. We find that they fail to generalize compositionally and that&#10;there is a surprisingly strong negative correlation between compound divergence&#10;and accuracy. We also demonstrate how our method can be used to create new&#10;compositionality benchmarks on top of the existing SCAN dataset, which confirms&#10;these findings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.17543" label="2312.17543">
        <attvalues>
          <attvalue for="0" value="Building Efficient Universal Classifiers with Natural Language Inference" />
          <attvalue for="1" value="  Generative Large Language Models (LLMs) have become the mainstream choice for&#10;fewshot and zeroshot learning thanks to the universality of text generation.&#10;Many users, however, do not need the broad capabilities of generative LLMs when&#10;they only want to automate a classification task. Smaller BERT-like models can&#10;also learn universal tasks, which allow them to do any text classification task&#10;without requiring fine-tuning (zeroshot classification) or to learn new tasks&#10;with only a few examples (fewshot), while being significantly more efficient&#10;than generative LLMs. This paper (1) explains how Natural Language Inference&#10;(NLI) can be used as a universal classification task that follows similar&#10;principles as instruction fine-tuning of generative LLMs, (2) provides a&#10;step-by-step guide with reusable Jupyter notebooks for building a universal&#10;classifier, and (3) shares the resulting universal classifier that is trained&#10;on 33 datasets with 389 diverse classes. Parts of the code we share has been&#10;used to train our older zeroshot classifiers that have been downloaded more&#10;than 55 million times via the Hugging Face Hub as of December 2023. Our new&#10;classifier improves zeroshot performance by 9.4%.&#10;" />
          <attvalue for="2" value="&#10;&#10;Over the past year, generative models have taken both academia and public attention by storm. The main appeal of text generation is that it is so universal, that almost any other text-related task can be reformulated as a text generation task \cite{radford_language_2019,raffel_exploring_2020}. Especially when text generators are massively scaled up and tuned on human instructions, they acquire impressive capabilities to generalise to new tasks without requiring task-specific fine-tuning \cite{sanh_multitask_2022,ouyang_training_2022,chung_scaling_2022,openai_gpt-4_2023,touvron_llama_2023}. Since the utility of these generative Large Language Models (LLMs) has become evident, large amounts of intellectual, financial and energy resources are being invested in improving and scaling generative LLMs.&#10;&#10;Given that the resource requirements for training and deploying generative LLMs are prohibitive for many researchers and practitioners, this paper investigates other types of universal models, that make a different trade-off between resource requirements and universality. The literature has developed several other universal tasks that cannot solve generative tasks (summarization, translation etc.), but can solve any classification task with smaller size and performance competitive with generative LLMs \cite{xu_universal_2023,schick_its_2021}.&#10;&#10;The principle of universal classifiers is similar to generative models: A model is trained on a universal task, and a form of instruction or prompt enable it to generalize to unseen classification tasks. While several efficient approaches to universal classification exist \cite{schick_exploiting_2021,xia_prompting_2022,yao_prompt_2022,xu_universal_2023,bragg_flex_2021,ma_issues_2021,sun_nsp-bert_2022}, this paper focuses on guidance for one approach: Natural Language Inference. Several papers have used the universal NLI task for zero- and fewshot classification, but stopped short of mixing NLI data with multiple other non-NLI datasets to build more universal classifiers \cite{yin_benchmarking_2019,yin_universal_2020,wang_entailment_2021,laurer_less_2023}. &#10;&#10;The main contribution of this paper are: (1) easy-to-use universal classifiers trained on 5 NLI datasets and 28 non-NLI datasets with 389 diverse classes, improving zeroshot performance by 9.4\% compared to NLI-only models; (2) a step-by-step guide with Juypter notebooks enabling users to train and adapt their own universal classifiers. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Universal Classification Tasks, Artificial Intelligence, Natural Language Inference, Language Model Efficiency" />
        </attvalues>
      </node>
      <node id="2205.15223" label="2205.15223">
        <attvalues>
          <attvalue for="0" value="Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained&#10;  Models" />
          <attvalue for="1" value="  Pre-trained masked language models successfully perform few-shot learning by&#10;formulating downstream tasks as text infilling. However, as a strong&#10;alternative in full-shot settings, discriminative pre-trained models like&#10;ELECTRA do not fit into the paradigm. In this work, we adapt prompt-based&#10;few-shot learning to ELECTRA and show that it outperforms masked language&#10;models in a wide range of tasks. ELECTRA is pre-trained to distinguish if a&#10;token is generated or original. We naturally extend that to prompt-based&#10;few-shot learning by training to score the originality of the target options&#10;without introducing new parameters. Our method can be easily adapted to tasks&#10;involving multi-token predictions without extra computation overhead. Analysis&#10;shows that ELECTRA learns distributions that align better with downstream&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.11166" label="2205.11166">
        <attvalues>
          <attvalue for="0" value="Prompt Tuning for Discriminative Pre-trained Language Models" />
          <attvalue for="1" value="  Recent works have shown promising results of prompt tuning in stimulating&#10;pre-trained language models (PLMs) for natural language processing (NLP) tasks.&#10;However, to the best of our knowledge, existing works focus on prompt-tuning&#10;generative PLMs that are pre-trained to generate target tokens, such as BERT.&#10;It is still unknown whether and how discriminative PLMs, e.g., ELECTRA, can be&#10;effectively prompt-tuned. In this work, we present DPT, the first prompt tuning&#10;framework for discriminative PLMs, which reformulates NLP tasks into a&#10;discriminative language modeling problem. Comprehensive experiments on text&#10;classification and question answering show that, compared with vanilla&#10;fine-tuning, DPT achieves significantly higher performance, and also prevents&#10;the unstable problem in tuning large PLMs in both full-set and low-resource&#10;settings. The source code and experiment details of this paper can be obtained&#10;from https://github.com/thunlp/DPT.&#10;" />
          <attvalue for="2" value="&#10;Recent years have witnessed the great success of the pre-training-then-fine-tuning paradigm in natural language processing (NLP)~\cite{DBLP:conf/naacl/DevlinCLT19,DBLP:conf/nips/YangDYCSL19,DBLP:conf/iclr/ClarkLLM20,DBLP:conf/iclr/LanCGGSS20,DBLP:journals/jmlr/RaffelSRLNMZLL20}. Typically, language models are first pre-trained on large-scale corpora via self-supervised generative or discriminative tasks to learn universal text representations, and then fine-tuned to adapt to downstream tasks~\cite{qiu2020pre,xu2021pre}. However, the significant gap between the objective forms of model pre-training and fine-tuning hinders taking full advantage of PLMs in downstream tasks~\cite{DBLP:journals/corr/abs-2107-13586}.&#10;&#10;Prompt tuning has recently shown its effectiveness in stimulating the capability of PLMs by transforming downstream tasks into the same form as pre-training~\cite{DBLP:conf/emnlp/PetroniRRLBWM19,brown2020language,DBLP:conf/naacl/SchickS21,DBLP:conf/acl/GaoFC20,yao2021cpt}. However, to the best of our knowledge, existing works focus on prompt-tuning generative PLMs (i.e., PLMs pre-trained by generating target textual tokens from the context, such as BERT~\cite{DBLP:conf/naacl/DevlinCLT19} and GPT~\cite{brown2020language}). It is still unknown whether and how discriminative PLMs can be effectively prompt-tuned (i.e., PLMs pre-trained by discriminating replaced tokens, such as ELECTRA~\cite{DBLP:conf/iclr/ClarkLLM20} and WKLM~\cite{DBLP:conf/iclr/XiongDWS20}). Since discriminative PLMs typically enjoy competitive performance and superior computational efficiency compared with their generative counterparts~\cite{DBLP:conf/iclr/ClarkLLM20}, it can be especially appealing to prompt-tuning discriminative PLMs. &#10;&#10;In this work, we present DPT, the first \underline{p}rompt \underline{t}uning framework for \underline{d}iscriminative PLMs. DPT reformulates downstream tasks into a discriminative language modeling problem, maximally mitigating the gap between model pre-training and tuning. Specifically, as shown in Figure~\ref{fig:framework}, models are asked to discriminate correct answer tokens (e.g., correct labels for text classification, or answer spans for question answering) from the input tokens based on the reused discriminative language modeling head, where the objective form is identical to pre-training. &#10;&#10;To evaluate DPT, we conduct comprehensive experiments on text classification and question answering in both full-set and low-resource settings. Experimental results show that despite its simplicity, DPT significantly outperforms vanilla fine-tuning (e.g., $4.1\%$ accuracy improvement in the low-resource SST-5 evaluation). Moreover, previous works have shown that fine-tuning large PLMs can be highly unstable and even produce divergent results~\cite{DBLP:conf/naacl/DevlinCLT19,dodge2020fine}, which undermines the practicality of large PLMs. We show that DPT also addresses the unstable problem in tuning large discriminative PLMs.&#10;&#10;The contributions of our work are summarized as follows: (1) We present the first prompt tuning framework for discriminative PLMs. (2) Comprehensive experimental results on text classification and question answering demonstrate the effectiveness of the proposed prompt tuning framework.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.07170" label="2107.07170">
        <attvalues>
          <attvalue for="0" value="FLEX: Unifying Evaluation for Few-Shot NLP" />
          <attvalue for="1" value="  Few-shot NLP research is highly active, yet conducted in disjoint research&#10;threads with evaluation suites that lack challenging-yet-realistic testing&#10;setups and fail to employ careful experimental design. Consequently, the&#10;community does not know which techniques perform best or even if they&#10;outperform simple baselines. In response, we formulate the FLEX Principles, a&#10;set of requirements and best practices for unified, rigorous, valid, and&#10;cost-sensitive few-shot NLP evaluation. These principles include Sample Size&#10;Design, a novel approach to benchmark design that optimizes statistical&#10;accuracy and precision while keeping evaluation costs manageable. Following the&#10;principles, we release the FLEX benchmark, which includes four few-shot&#10;transfer settings, zero-shot evaluation, and a public leaderboard that covers&#10;diverse NLP tasks. In addition, we present UniFew, a prompt-based model for&#10;few-shot learning that unifies pretraining and finetuning prompt formats,&#10;eschewing complex machinery of recent prompt-based approaches in adapting&#10;downstream task formats to language model pretraining objectives. We&#10;demonstrate that despite simplicity, UniFew achieves results competitive with&#10;both popular meta-learning and prompt-based approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.00161" label="1909.00161">
        <attvalues>
          <attvalue for="0" value="Benchmarking Zero-shot Text Classification: Datasets, Evaluation and&#10;  Entailment Approach" />
          <attvalue for="1" value="  Zero-shot text classification (0Shot-TC) is a challenging NLU problem to&#10;which little attention has been paid by the research community. 0Shot-TC aims&#10;to associate an appropriate label with a piece of text, irrespective of the&#10;text domain and the aspect (e.g., topic, emotion, event, etc.) described by the&#10;label. And there are only a few articles studying 0Shot-TC, all focusing only&#10;on topical categorization which, we argue, is just the tip of the iceberg in&#10;0Shot-TC. In addition, the chaotic experiments in literature make no uniform&#10;comparison, which blurs the progress.&#10;  This work benchmarks the 0Shot-TC problem by providing unified datasets,&#10;standardized evaluations, and state-of-the-art baselines. Our contributions&#10;include: i) The datasets we provide facilitate studying 0Shot-TC relative to&#10;conceptually different and diverse aspects: the ``topic'' aspect includes&#10;``sports'' and ``politics'' as labels; the ``emotion'' aspect includes ``joy''&#10;and ``anger''; the ``situation'' aspect includes ``medical assistance'' and&#10;``water shortage''. ii) We extend the existing evaluation setup&#10;(label-partially-unseen) -- given a dataset, train on some labels, test on all&#10;labels -- to include a more challenging yet realistic evaluation&#10;label-fully-unseen 0Shot-TC (Chang et al., 2008), aiming at classifying text&#10;snippets without seeing task specific training data at all. iii) We unify the&#10;0Shot-TC of diverse aspects within a textual entailment formulation and study&#10;it this way.&#10;  Code &amp; Data: https://github.com/yinwenpeng/BenchmarkingZeroShot&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.02584" label="2010.02584">
        <attvalues>
          <attvalue for="0" value="Universal Natural Language Processing with Limited Annotations: Try&#10;  Few-shot Textual Entailment as a Start" />
          <attvalue for="1" value="  A standard way to address different NLP problems is by first constructing a&#10;problem-specific dataset, then building a model to fit this dataset. To build&#10;the ultimate artificial intelligence, we desire a single machine that can&#10;handle diverse new problems, for which task-specific annotations are limited.&#10;We bring up textual entailment as a unified solver for such NLP problems.&#10;However, current research of textual entailment has not spilled much ink on the&#10;following questions: (i) How well does a pretrained textual entailment system&#10;generalize across domains with only a handful of domain-specific examples? and&#10;(ii) When is it worth transforming an NLP task into textual entailment? We&#10;argue that the transforming is unnecessary if we can obtain rich annotations&#10;for this task. Textual entailment really matters particularly when the target&#10;NLP task has insufficient annotations.&#10;  Universal NLP can be probably achieved through different routines. In this&#10;work, we introduce Universal Few-shot textual Entailment (UFO-Entail). We&#10;demonstrate that this framework enables a pretrained entailment model to work&#10;well on new entailment domains in a few-shot setting, and show its&#10;effectiveness as a unified solver for several downstream NLP tasks such as&#10;question answering and coreference resolution when the end-task annotations are&#10;limited. Code: https://github.com/salesforce/UniversalFewShotNLP&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.14690" label="2104.14690">
        <attvalues>
          <attvalue for="0" value="Entailment as Few-Shot Learner" />
          <attvalue for="1" value="  Large pre-trained language models (LMs) have demonstrated remarkable ability&#10;as few-shot learners. However, their success hinges largely on scaling model&#10;parameters to a degree that makes it challenging to train and serve. In this&#10;paper, we propose a new approach, named as EFL, that can turn small LMs into&#10;better few-shot learners. The key idea of this approach is to reformulate&#10;potential NLP task into an entailment one, and then fine-tune the model with as&#10;little as 8 examples. We further demonstrate our proposed method can be: (i)&#10;naturally combined with an unsupervised contrastive learning-based data&#10;augmentation method; (ii) easily extended to multilingual few-shot learning. A&#10;systematic evaluation on 18 standard NLP tasks demonstrates that this approach&#10;improves the various existing SOTA few-shot learning methods by 12\%, and&#10;yields competitive few-shot performance with 500 times larger models, such as&#10;GPT-3.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;There have been many research studies on improving the few-shot learning performance of a pre-trained language model:&#10;&#10;Language modeling with demonstrations: The series of GPT works~\cite{radford2019language,brown2020language} proposed to add a task description (prompt) and annotated examples as demonstration to enable few-shot learning, which has been commonly applied to classification~\cite{puri2019zero}, QA, commonsense knowledge mining~\cite{davison2019commonsense}. It is also explored for probing the knowledge contained within pre-trained LMs.&#10;&#10;Task reformulation: Language model is usually pre-trained with a Masked Language Modeling (MLM) objective, which is motivated by Cloze task in~\cite{taylor1953cloze}. There have been several works reformulating few-shot learning tasks as cloze questions to reuse pre-trained LM such as LM prompt~\cite{jiang2020can}, PET~\cite{radford2019language,schick2020exploiting}, and recent LM-BFF~\cite{gao2020making}. It shows a pre-trained LM can achieve non-trivial performance with few annotated samples. There are also some other works transforming NLP tasks as generative QA tasks~\cite{puri2019zero}.&#10;&#10;Intermediate training: The work by~\cite{phang2018sentence} shows that supplementing pre-trained LMs with further training on data-rich supervised tasks can obtain additional performance improvements on the GLUE benchmark. More recently, this approach has been further improved by a matching-based few-shot learning method~\cite{yin2020universal}.&#10;&#10;General techniques: There are several general techniques to improve the few-shot learning performance, including: (i) optimization and regularization techniques during the fine-tuning~\cite{howard2018universal,lee2019mixout,zhang2020revisiting}, (ii) semi-supervised learning to augment training data~\cite{xie2020unsupervised}, and (iii) supervised contrastive learning as additional objective~\cite{gunel2020supervised}. We anticipate that these studies are largely complementary to ours.&#10;&#10;Comparing to existing prompt-based few-shot learning methods~\cite{brown2020language,schick2020exploiting,gao2020making}, the key differences of our proposed method are: (i) our approach reformulates NLP tasks as textual entailment instead of cloze questions; (ii) provide label-specific descriptions for each class instead of single task description.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2408.12799" label="2408.12799">
        <attvalues>
          <attvalue for="0" value="Less for More: Enhancing Preference Learning in Generative Language&#10;  Models with Automated Self-Curation of Training Corpora" />
          <attvalue for="1" value="  Ambiguity in language presents challenges in developing more enhanced&#10;language models, particularly in preference learning, where variability among&#10;annotators results in inconsistently annotated datasets used for model&#10;alignment. To address this issue, we introduce a self-curation method that&#10;preprocesses annotated datasets by leveraging proxy models trained directly on&#10;these datasets. Our method enhances preference learning by automatically&#10;detecting and removing ambiguous annotations within the dataset. The proposed&#10;approach is validated through extensive experiments, demonstrating a marked&#10;improvement in performance across various instruction-following tasks. Our work&#10;provides a straightforward and reliable method to overcome annotation&#10;inconsistencies, serving as an initial step towards the development of more&#10;advanced preference learning techniques.&#10;" />
          <attvalue for="2" value="&#10;&#10;Aligning language models (LMs) with human intent is crucial for reliable deployment as unhelpful or harmful responses can pose significant risks. &#10;Preference learning methods — such as Reinforcement Learning from Human Feedback (RLHF) \cite{rm-direction,ft-preference,InstructGPT,cai,llama2} and its successor, Direct Preference Optimization (DPO) \cite{DPO} — leverage preference datasets to achieve alignment that goes beyond what instruction tuning can offer. &#10;Unlike instruction tuning, which relies on single-response supervision, preference datasets enable more explicit learning by comparing {chosen} and {rejected} responses to given instructions.&#10;&#10;Due to their two-response structure, however, preference datasets are vulnerable not only to noise within individual responses but also to incorrect flipping of preference between responses.&#10;Moreover, because these datasets must consider a wide range of criteria ({e.g.}, helpfulness and harmlessness), the preference decision often requires navigating a more complex, multi-dimensional space, complicating the annotation process.&#10;&#10;This complexity is exacerbated by the diverse cultural or social backgrounds of annotators, leading to inconsistent preferences or unreliable annotations.&#10;During the collection of paired corpora for prompts, decisions can vary from unanimous to narrowly divided even within the same crowd \cite{HH-RLHF}.&#10;However, this variability is often excluded from the training dataset, introducing noise from inconsistent annotations or conflicting criteria, which can distort the target model's learning of desirable preference patterns.&#10;&#10;Similar issues may arise when using large language models (LLMs) for annotation as they exhibit human-like patterns \cite{AGI-sparks}.&#10;\cite{consistency} observed that GPT-4 \cite{GPT4} can display inconsistent behavior during multi-step reasoning despite its impressive performance on various tasks.&#10;&#10;To mitigate these challenges arising from noisy data in the training dataset, manual data cleansing or adding additional annotations, such as the significance level of each preference data point, can be used~\cite{llama2}. &#10;However, these approaches are costly and time-consuming.&#10;Moreover, determining whether data is truly noisy can often be particularly challenging.&#10;&#10;Among earlier literature on the dataset quality, \cite{dataset_cartography} categorize training data for language models into 'easy-to-learn', 'ambiguous' and 'hard-to-learn' and show that an appropriate mix of easy-to-learn and ambiguous data is essential for successful optimization. &#10;However, their study is largely confined to tasks requiring short answers and leaves the question of how to achieve the optimal balance unresolved.&#10;To address this limitation, we propose a method for selecting a well-balanced subset from the target preference dataset to enhance the training of generative LMs. &#10;&#10;For more complex generative tasks, several prior studies have focused on data selection to enhance instruction tuning performance. &#10;Whereas \cite{activeIT} suggest that ambiguous tasks can enhance generalization in instruction tuning using a categorization similar to \cite{dataset_cartography},&#10;\cite{quantity2quality} demonstrate that selecting easy-to-learn data over hard-to-learn segments is more beneficial for improving performance.&#10;They argue that data with low prediction probability may be too specific thus hard to learn.&#10;Other related studies include \cite{instag, alpagasus, DEITA}, but these works mainly focus on data efficiency and are limited to instruction tuning.&#10;In contrast, our approach offers a new perspective on dataset quality by introducing a novel subset selection method that enhances the consistency of preference annotations for more effective preference learning.&#10;&#10;Some previous approaches to preference learning aim to build better reward functions by merging or weighting multiple reward models through a multi-objective learning framework \cite{rewarded_soups, personalizedsoups, arithmetic_control}. &#10;However, the underlying issues of dataset quality remain unaddressed.&#10;Additionally, conservative DPO (cDPO) \cite{cDPO} and robust DPO (rDPO) \cite{rDPO} consider mislabeled or ambiguous data contained in the dataset unlike DPO.&#10;However, their approach assumes the error rate as a prior instead of providing a systematic way to determine it, which reduces its practicality.&#10;&#10;In this work, we propose simple yet effective methods to fully automate the curation of training corpora that increases preference consistency in the dataset. &#10;Our {self-curation} method leverages proxy models trained on the given preference datasets using the Bradley-Terry (BT) model \cite{BT-model}. &#10;Through this training process, consistent preference patterns emerge, enabling the trained proxy to retrospectively evaluate the original annotations to assess their consistency.&#10;&#10;We assess preference consistency by identifying discrepancies between the proxy models' predictions and the given annotations, as illustrated in Figure~\ref{fig:overview}. &#10;This inconsistent preference patterns are challenging for the target LM to comprehend \cite{liu-etal-2023-afraid}, making such corpora detrimental to preference learning.&#10;We demonstrate that simply excluding these inconsistent data from training can significantly enhance preference learning such as DPO through extensive experiments.&#10;The key contributions of this work are summarized as follows:&#10;\begin{itemize}&#10;\item We reveal that data with preference discrepancies, where the annotations disagree with the predictions of the self-trained proxy, negatively impact preference learning (Sec.~\ref{sec:impact}).&#10;\item We introduce a novel {self-curation} method that selects training data with consistent preference using the proposed proxy, enabling more robust and effective alignment (Sec.~\ref{sec:self_curation}).&#10;\item Through comprehensive experiments, we show that our method is agnostic to either learning algorithms or a proxy's capability if the proxy has a sufficient level of capacity to learn consistent preference pattern (Sec.~\ref{sec:self_curation}). &#10;\item We empirically demonstrate that applying the proposed method consistently and significantly improves the performance of preference learning (Sec.~\ref{sec:self_curation}, Sec.~\ref{sec:anaysis}).&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Data Annotation Methods, Artificial Intelligence, Language Model Development, Natural Language Processing, Preference Learning" />
        </attvalues>
      </node>
      <node id="2212.08073" label="2212.08073">
        <attvalues>
          <attvalue for="0" value="Constitutional AI: Harmlessness from AI Feedback" />
          <attvalue for="1" value="  As AI systems become more capable, we would like to enlist their help to&#10;supervise other AIs. We experiment with methods for training a harmless AI&#10;assistant through self-improvement, without any human labels identifying&#10;harmful outputs. The only human oversight is provided through a list of rules&#10;or principles, and so we refer to the method as 'Constitutional AI'. The&#10;process involves both a supervised learning and a reinforcement learning phase.&#10;In the supervised phase we sample from an initial model, then generate&#10;self-critiques and revisions, and then finetune the original model on revised&#10;responses. In the RL phase, we sample from the finetuned model, use a model to&#10;evaluate which of the two samples is better, and then train a preference model&#10;from this dataset of AI preferences. We then train with RL using the preference&#10;model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a&#10;result we are able to train a harmless but non-evasive AI assistant that&#10;engages with harmful queries by explaining its objections to them. Both the SL&#10;and RL methods can leverage chain-of-thought style reasoning to improve the&#10;human-judged performance and transparency of AI decision making. These methods&#10;make it possible to control AI behavior more precisely and with far fewer human&#10;labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14279" label="2305.14279">
        <attvalues>
          <attvalue for="0" value="Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs" />
          <attvalue for="1" value="  Large language models (LLMs) have achieved widespread success on a variety of&#10;in-context few-shot tasks, but this success is typically evaluated via&#10;correctness rather than consistency. We argue that self-consistency is an&#10;important criteria for valid multi-step reasoning in tasks where the solution&#10;is composed of the answers to multiple sub-steps. We propose two types of&#10;self-consistency that are particularly important for multi-step reasoning --&#10;hypothetical consistency (a model's ability to predict what its output would be&#10;in a hypothetical other context) and compositional consistency (consistency of&#10;a model's final outputs when intermediate sub-steps are replaced with the&#10;model's outputs for those steps). We demonstrate that multiple variants of the&#10;GPT-3/-4 models exhibit poor consistency rates across both types of consistency&#10;on a variety of tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.10795" label="2009.10795">
        <attvalues>
          <attvalue for="0" value="Dataset Cartography: Mapping and Diagnosing Datasets with Training&#10;  Dynamics" />
          <attvalue for="1" value="  Large datasets have become commonplace in NLP research. However, the&#10;increased emphasis on data quantity has made it challenging to assess the&#10;quality of data. We introduce Data Maps---a model-based tool to characterize&#10;and diagnose datasets. We leverage a largely ignored source of information: the&#10;behavior of the model on individual instances during training (training&#10;dynamics) for building data maps. This yields two intuitive measures for each&#10;example---the model's confidence in the true class, and the variability of this&#10;confidence across epochs---obtained in a single run of training. Experiments&#10;across four datasets show that these model-dependent measures reveal three&#10;distinct regions in the data map, each with pronounced characteristics. First,&#10;our data maps show the presence of &quot;ambiguous&quot; regions with respect to the&#10;model, which contribute the most towards out-of-distribution generalization.&#10;Second, the most populous regions in the data are &quot;easy to learn&quot; for the&#10;model, and play an important role in model optimization. Finally, data maps&#10;uncover a region with instances that the model finds &quot;hard to learn&quot;; these&#10;often correspond to labeling errors. Our results indicate that a shift in focus&#10;from quantity to quality of data could lead to robust models and improved&#10;out-of-distribution generalization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.00288" label="2311.00288">
        <attvalues>
          <attvalue for="0" value="Active Instruction Tuning: Improving Cross-Task Generalization by&#10;  Training on Prompt Sensitive Tasks" />
          <attvalue for="1" value="  Instruction tuning (IT) achieves impressive zero-shot generalization results&#10;by training large language models (LLMs) on a massive amount of diverse tasks&#10;with instructions. However, how to select new tasks to improve the performance&#10;and generalizability of IT models remains an open question. Training on all&#10;existing tasks is impractical due to prohibiting computation requirements, and&#10;randomly selecting tasks can lead to suboptimal performance. In this work, we&#10;propose active instruction tuning based on prompt uncertainty, a novel&#10;framework to identify informative tasks, and then actively tune the models on&#10;the selected tasks. We represent the informativeness of new tasks with the&#10;disagreement of the current model outputs over perturbed prompts. Our&#10;experiments on NIV2 and Self-Instruct datasets demonstrate that our method&#10;consistently outperforms other baseline strategies for task selection,&#10;achieving better out-of-distribution generalization with fewer training tasks.&#10;Additionally, we introduce a task map that categorizes and diagnoses tasks&#10;based on prompt uncertainty and prediction probability. We discover that&#10;training on ambiguous (prompt-uncertain) tasks improves generalization while&#10;training on difficult (prompt-certain and low-probability) tasks offers no&#10;benefit, underscoring the importance of task selection for instruction tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.12032" label="2308.12032">
        <attvalues>
          <attvalue for="0" value="From Quantity to Quality: Boosting LLM Performance with Self-Guided Data&#10;  Selection for Instruction Tuning" />
          <attvalue for="1" value="  In the realm of Large Language Models (LLMs), the balance between instruction&#10;data quality and quantity is a focal point. Recognizing this, we introduce a&#10;self-guided methodology for LLMs to autonomously discern and select cherry&#10;samples from open-source datasets, effectively minimizing manual curation and&#10;potential cost for instruction tuning an LLM. Our key innovation, the&#10;Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to&#10;identify discrepancies between a model's expected responses and its intrinsic&#10;generation capability. Through the application of IFD, cherry samples can be&#10;pinpointed, leading to a marked uptick in model training efficiency. Empirical&#10;validations on datasets like Alpaca and WizardLM underpin our findings; with a&#10;mere $10\%$ of original data input, our strategy showcases improved results.&#10;This synthesis of self-guided cherry-picking and the IFD metric signifies a&#10;transformative leap in the instruction tuning of LLMs, promising both&#10;efficiency and resource-conscious advancements. Codes, data, and models are&#10;available: https://github.com/tianyi-lab/Cherry_LLM&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2306.04488" label="2306.04488">
        <attvalues>
          <attvalue for="0" value="Rewarded soups: towards Pareto-optimal alignment by interpolating&#10;  weights fine-tuned on diverse rewards" />
          <attvalue for="1" value="  Foundation models are first pre-trained on vast unsupervised datasets and&#10;then fine-tuned on labeled data. Reinforcement learning, notably from human&#10;feedback (RLHF), can further align the network with the intended usage. Yet the&#10;imperfections in the proxy reward may hinder the training and lead to&#10;suboptimal results; the diversity of objectives in real-world tasks and human&#10;opinions exacerbate the issue. This paper proposes embracing the heterogeneity&#10;of diverse rewards by following a multi-policy strategy. Rather than focusing&#10;on a single a priori reward, we aim for Pareto-optimal generalization across&#10;the entire space of preferences. To this end, we propose rewarded soup, first&#10;specializing multiple networks independently (one for each proxy reward) and&#10;then interpolating their weights linearly. This succeeds empirically because we&#10;show that the weights remain linearly connected when fine-tuned on diverse&#10;rewards from a shared pre-trained initialization. We demonstrate the&#10;effectiveness of our approach for text-to-text (summarization, Q&amp;A, helpful&#10;assistant, review), text-image (image captioning, text-to-image generation,&#10;visual grounding, VQA), and control (locomotion) tasks. We hope to enhance the&#10;alignment of deep models, and how they interact with the world in all its&#10;diversity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.00409" label="2403.00409">
        <attvalues>
          <attvalue for="0" value="Provably Robust DPO: Aligning Language Models with Noisy Feedback" />
          <attvalue for="1" value="  Learning from preference-based feedback has recently gained traction as a&#10;promising approach to align language models with human interests. While these&#10;aligned generative models have demonstrated impressive capabilities across&#10;various tasks, their dependence on high-quality human preference data poses a&#10;bottleneck in practical applications. Specifically, noisy (incorrect and&#10;ambiguous) preference pairs in the dataset might restrict the language models&#10;from capturing human intent accurately. While practitioners have recently&#10;proposed heuristics to mitigate the effect of noisy preferences, a complete&#10;theoretical understanding of their workings remain elusive.&#10;  In this work, we aim to bridge this gap by by introducing a general framework&#10;for policy optimization in the presence of random preference flips. We focus on&#10;the direct preference optimization (DPO) algorithm in particular since it&#10;assumes that preferences adhere to the Bradley-Terry-Luce (BTL) model, raising&#10;concerns about the impact of noisy data on the learned policy. We design a&#10;novel loss function, which de-bias the effect of noise on average, making a&#10;policy trained by minimizing that loss robust to the noise. Under log-linear&#10;parameterization of the policy class and assuming good feature coverage of the&#10;SFT policy, we prove that the sub-optimality gap of the proposed robust DPO&#10;(rDPO) policy compared to the optimal policy is of the order&#10;$O(\frac{1}{1-2\epsilon}\sqrt{\frac{d}{n}})$, where $\epsilon &lt; 1/2$ is flip&#10;rate of labels, $d$ is policy parameter dimension and $n$ is size of dataset.&#10;Our experiments on IMDb sentiment generation and Anthropic's helpful-harmless&#10;dataset show that rDPO is robust to noise in preference labels compared to&#10;vanilla DPO and other heuristics proposed by practitioners.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1206.5538" label="1206.5538">
        <attvalues>
          <attvalue for="0" value="Representation Learning: A Review and New Perspectives" />
          <attvalue for="1" value="  The success of machine learning algorithms generally depends on data&#10;representation, and we hypothesize that this is because different&#10;representations can entangle and hide more or less the different explanatory&#10;factors of variation behind the data. Although specific domain knowledge can be&#10;used to help design representations, learning with generic priors can also be&#10;used, and the quest for AI is motivating the design of more powerful&#10;representation-learning algorithms implementing such priors. This paper reviews&#10;recent work in the area of unsupervised feature learning and deep learning,&#10;covering advances in probabilistic models, auto-encoders, manifold learning,&#10;and deep networks. This motivates longer-term unanswered questions about the&#10;appropriate objectives for learning good representations, for computing&#10;representations (i.e., inference), and the geometrical connections between&#10;representation learning, density estimation and manifold learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.06861" label="1711.06861">
        <attvalues>
          <attvalue for="0" value="Style Transfer in Text: Exploration and Evaluation" />
          <attvalue for="1" value="  Style transfer is an important problem in natural language processing (NLP).&#10;However, the progress in language style transfer is lagged behind other&#10;domains, such as computer vision, mainly because of the lack of parallel data&#10;and principle evaluation metrics. In this paper, we propose to learn style&#10;transfer with non-parallel data. We explore two models to achieve this goal,&#10;and the key idea behind the proposed models is to learn separate content&#10;representations and style representations using adversarial networks. We also&#10;propose novel evaluation metrics which measure two aspects of style transfer:&#10;transfer strength and content preservation. We access our models and the&#10;evaluation metrics on two tasks: paper-news title transfer, and&#10;positive-negative review transfer. Results show that the proposed content&#10;preservation metric is highly correlate to human judgments, and the proposed&#10;models are able to generate sentences with higher style transfer strength and&#10;similar content preservation score comparing to auto-encoder.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.05789" label="1907.05789">
        <attvalues>
          <attvalue for="0" value="Generating Sentences from Disentangled Syntactic and Semantic Spaces" />
          <attvalue for="1" value="  Variational auto-encoders (VAEs) are widely used in natural language&#10;generation due to the regularization of the latent space. However, generating&#10;sentences from the continuous latent space does not explicitly model the&#10;syntactic information. In this paper, we propose to generate sentences from&#10;disentangled syntactic and semantic spaces. Our proposed method explicitly&#10;models syntactic information in the VAE's latent space by using the linearized&#10;tree sequence, leading to better performance of language generation.&#10;Additionally, the advantage of sampling in the disentangled syntactic and&#10;semantic latent spaces enables us to perform novel applications, such as the&#10;unsupervised paraphrase generation and syntax-transfer generation. Experimental&#10;results show that our proposed model achieves similar or better performance in&#10;various tasks, compared with state-of-the-art related work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.00955" label="1703.00955">
        <attvalues>
          <attvalue for="0" value="Toward Controlled Generation of Text" />
          <attvalue for="1" value="  Generic generation and manipulation of text is challenging and has limited&#10;success compared to recent deep generative modeling in visual domain. This&#10;paper aims at generating plausible natural language sentences, whose attributes&#10;are dynamically controlled by learning disentangled latent representations with&#10;designated semantics. We propose a new neural generative model which combines&#10;variational auto-encoders and holistic attribute discriminators for effective&#10;imposition of semantic structures. With differentiable approximation to&#10;discrete text samples, explicit constraints on independent attribute controls,&#10;and efficient collaborative learning of generator and discriminators, our model&#10;learns highly interpretable representations from even only word annotations,&#10;and produces realistic sentences with desired attributes. Quantitative&#10;evaluation validates the accuracy of sentence and attribute generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13026" label="2305.13026">
        <attvalues>
          <attvalue for="0" value="DUMB: A Benchmark for Smart Evaluation of Dutch Models" />
          <attvalue for="1" value="  We introduce the Dutch Model Benchmark: DUMB. The benchmark includes a&#10;diverse set of datasets for low-, medium- and high-resource tasks. The total&#10;set of nine tasks includes four tasks that were previously not available in&#10;Dutch. Instead of relying on a mean score across tasks, we propose Relative&#10;Error Reduction (RER), which compares the DUMB performance of language models&#10;to a strong baseline which can be referred to in the future even when assessing&#10;different sets of language models. Through a comparison of 14 pre-trained&#10;language models (mono- and multi-lingual, of varying sizes), we assess the&#10;internal consistency of the benchmark tasks, as well as the factors that likely&#10;enable high performance. Our results indicate that current Dutch monolingual&#10;models under-perform and suggest training larger Dutch models with other&#10;architectures and pre-training objectives. At present, the highest performance&#10;is achieved by DeBERTaV3 (large), XLM-R (large) and mDeBERTaV3 (base). In&#10;addition to highlighting best strategies for training larger Dutch models, DUMB&#10;will foster further research on Dutch. A public leaderboard is available at&#10;https://dumbench.nl.&#10;" />
          <attvalue for="2" value="&#10;To evaluate and compare new and existing language models, a reliable method of comparing model quality is essential.&#10;For this reason, several benchmark suites have been proposed, such as English GLUE \cite{wang-etal-2018-glue} and SuperGLUE \cite{wang2019superglue}. &#10;However, at present, there is no standard benchmark for Dutch.&#10;&#10;The currently available Dutch language models are BERTje \cite{devries2019bertje}, which is a Dutch version of BERT\textsubscript{base} \cite{devlin-etal-2019-bert}, and three versions of RobBERT \cite{delobelle2020robbert, delobelle2022robbert} which are Dutch versions of RoBERTa\textsubscript{base} \cite{liu2019roberta}.&#10;Direct comparisons between these models have focused on several tasks: sentiment analysis, natural language inference, coarse-grained part-of-speech tagging and three-class named entity recognition, where RobBERT often outperforms BERTje \cite{delobelle2022robbert}.&#10;Results are not completely consistent, however. &#10;Some studies found that RobBERT performs better than BERTje at specific tasks \cite{ruitenbeek-etal-2022-zo, delobelle2022robbert, bruyne2021emotional}, whereas other studies find the opposite \cite{wijnholds-moortgat-2021-sick, de-langhe-etal-2022-investigating}.&#10;Other works assume higher performance for specific models and either exclusively experiment with BERTje \cite{alam-etal-2021-fighting-covid, ghaddar2021context, brandsen2022dig}, or exclusively experiment with RobBERT \cite{app12042179, delobelle-etal-2022-measuring}.&#10;&#10;By developing a new benchmark, we aim to reduce the present unclarity of current evaluations and obtain insights into potential performance improvements for future development of new Dutch models. We also hope that this work will foster further research on Dutch, including the development of decoder-based models.&#10;Indeed, we see the establishment of such a benchmark, together with the evaluation of existing encoder-based models, as a necessary step towards making it possible to also devise a solid evaluation framework for generative models, which is complicated by the high degree of variability related to prompts and outputs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Benchmark Development, Linguistics, Language Model Evaluation, Computational Linguistics, Artificial Intelligence, Natural Language Processing, Multilingual NLP" />
        </attvalues>
      </node>
      <node id="2001.06286" label="2001.06286">
        <attvalues>
          <attvalue for="0" value="RobBERT: a Dutch RoBERTa-based Language Model" />
          <attvalue for="1" value="  Pre-trained language models have been dominating the field of natural&#10;language processing in recent years, and have led to significant performance&#10;gains for various complex natural language tasks. One of the most prominent&#10;pre-trained language models is BERT, which was released as an English as well&#10;as a multilingual version. Although multilingual BERT performs well on many&#10;tasks, recent studies show that BERT models trained on a single language&#10;significantly outperform the multilingual version. Training a Dutch BERT model&#10;thus has a lot of potential for a wide range of Dutch NLP tasks. While previous&#10;approaches have used earlier implementations of BERT to train a Dutch version&#10;of BERT, we used RoBERTa, a robustly optimized BERT approach, to train a Dutch&#10;language model called RobBERT. We measured its performance on various tasks as&#10;well as the importance of the fine-tuning dataset size. We also evaluated the&#10;importance of language-specific tokenizers and the model's fairness. We found&#10;that RobBERT improves state-of-the-art results for various tasks, and&#10;especially significantly outperforms other models when dealing with smaller&#10;datasets. These results indicate that it is a powerful pre-trained model for a&#10;large variety of Dutch language tasks. The pre-trained and fine-tuned models&#10;are publicly available to support further downstream Dutch NLP applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.08192" label="2211.08192">
        <attvalues>
          <attvalue for="0" value="RobBERT-2022: Updating a Dutch Language Model to Account for Evolving&#10;  Language Use" />
          <attvalue for="1" value="  Large transformer-based language models, e.g. BERT and GPT-3, outperform&#10;previous architectures on most natural language processing tasks. Such language&#10;models are first pre-trained on gigantic corpora of text and later used as&#10;base-model for finetuning on a particular task. Since the pre-training step is&#10;usually not repeated, base models are not up-to-date with the latest&#10;information. In this paper, we update RobBERT, a RoBERTa-based state-of-the-art&#10;Dutch language model, which was trained in 2019. First, the tokenizer of&#10;RobBERT is updated to include new high-frequent tokens present in the latest&#10;Dutch OSCAR corpus, e.g. corona-related words. Then we further pre-train the&#10;RobBERT model using this dataset. To evaluate if our new model is a plug-in&#10;replacement for RobBERT, we introduce two additional criteria based on concept&#10;drift of existing tokens and alignment for novel tokens.We found that for&#10;certain language tasks this update results in a significant performance&#10;increase. These results highlight the benefit of continually updating a&#10;language model to account for evolving language use.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.10809" label="2104.10809">
        <attvalues>
          <attvalue for="0" value="Provable Limitations of Acquiring Meaning from Ungrounded Form: What&#10;  Will Future Language Models Understand?" />
          <attvalue for="1" value="  Language models trained on billions of tokens have recently led to&#10;unprecedented results on many NLP tasks. This success raises the question of&#10;whether, in principle, a system can ever ``understand'' raw text without access&#10;to some form of grounding. We formally investigate the abilities of ungrounded&#10;systems to acquire meaning. Our analysis focuses on the role of ``assertions'':&#10;textual contexts that provide indirect clues about the underlying semantics. We&#10;study whether assertions enable a system to emulate representations preserving&#10;semantic relations like equivalence. We find that assertions enable semantic&#10;emulation of languages that satisfy a strong notion of semantic transparency.&#10;However, for classes of languages where the same expression can take different&#10;values in different contexts, we show that emulation can become uncomputable.&#10;Finally, we discuss differences between our formal model and natural language,&#10;exploring how our results generalize to a modal setting and other semantic&#10;relations. Together, our results suggest that assertions in code or language do&#10;not provide sufficient signal to fully emulate semantic representations. We&#10;formalize ways in which ungrounded language models appear to be fundamentally&#10;limited in their ability to ``understand''.&#10;" />
          <attvalue for="2" value=" &#10;Recently, language models trained on huge datasets of raw text have pushed the limits of natural language processing \cite[among others]{devlin-etal-2019-bert, raffel2019exploring, brown2020language}. Such systems transcend the expert system paradigm, where rules about language and meaning are hardcoded into a system, as well as the supervised learning paradigm, where a notion of meaning is provided through ground-truth labels. Rather, analysis of massive language models has revealed that, to some degree, knowledge of syntactic and semantic dependencies can emerge without explicit supervision \cite{rogers2020primer, tenney-etal-2019-bert}. This knowledge can then be transferred to a variety of downstream NLP tasks.&#10;&#10;Yet, today's NLP systems built on large language models still fall short of human-level general understanding \cite{yogatama2019learning, zhang-etal-2020-winowhy}.&#10;\cite{brown2020language} discuss the limitations of their GPT-3 language model compared to humans, suggesting that:&#10;&#10;\begin{displayquote}&#10;Scaling up any LM-like model \ldots\ may eventually run into (or could already be running into) the limits of the pretraining objective.&#10;\end{displayquote}&#10;&#10;This possibility raises an interesting theoretical question. What are the fundamental limits of learning meaning from language modeling, even assuming a perfect learner with access to unlimited data?&#10;Recently, \cite{bk-2020} argued that achieving true natural language understanding from text alone is impossible, and that, to really get at meaning, some type of semantic grounding is necessary.&#10;Their style of argumentation largely focused on developing thought experiments, rather than making formal arguments.&#10;&#10;One thought experiment featuring prominently in \cite{bk-2020} was the task of learning to understand a programming language's semantics from raw code. Here, understanding was defined as fully emulating a compiler. This setup has clear parallels to learning to understand natural language, although the more well-defined nature of programming languages makes them easier to reason about. \cite{bk-2020} argue that emulation is difficult in this setting, and perhaps impossible, because the source code alone contains no information about how it should be interpreted to create outputs. One counterpoint raised by the paper, as well as others \cite{blog2020, potts2020}, is the existence of unit tests, with assertions encoding examples of input/output pairs for blocks of code. For example, systematically observing blocks like \pythoninline{x = 3; assert x == 3} could let a system bootstrap the semantics of variable assignment, because a programmer is likely to write assertions that will pass. \upd{These assertions constitute a form of implicit grounding embedded within language modeling by the pragmatic concerns of programmers, and they could potentially be leveraged to emulate a compiler.} However, it is not immediately clear if unit tests provide ``enough'' supervision to do this, even with unlimited data.&#10;&#10;Viewing the debate about the power of assertions as central to the larger philosophical question, we aim to clarify it in more formal terms.&#10;In this paper, we formally study whether observing a generalized notion of assertions can allow a system to ``understand'' strings. An assertion is a query about whether two strings evaluate to the same value within a fixed context.&#10;This is motivated by the role of assertions in unit tests, where asserting two expressions are equal suggests that they have the same value within the test.&#10;&#10;While assertions are directly motivated by the compiler thought experiment, they also have analogs in natural language, where sentences make assertions about the world, and it is reasonable to expect \upd{some form of} bias towards true statements \cite{potts2020}. Indeed, this is one of Grice's Maxims \cite{grice1975logic}: a set of basic principles proposed to govern the pragmatics of natural language.&#10;For example, the truth conditions of This cat is the cat that Mary owns verify that two cats in the world identified in distinct ways are the same entity.&#10;In general, we might expect a sentence to appear with higher frequency if its truth conditions hold within its context, similar to an assertion in code, although of course there will also be other factors governing sentence frequency besides this.&#10;In this sense, the example sentence resembles the Python statement \pythoninline{assert cat1 == cat2}, where \pythoninline{cat1} and \pythoninline{cat2} are two \pythoninline{Cat} objects.&#10;See \ref{sec:towards-nl} for more discussion of how assertions and other formal concepts translate to natural language.&#10;We will generalize assertions to an abstract formal language context, allowing us to study how they can be used to emulate semantic relations.&#10;&#10;Our findings are as follows. If every expression in a language has the same value in every valid context,&#10;then the language can be emulated using a finite number of assertion queries (\ref{sec:no-side-effects}). However, we construct a class of languages where expressions can take different values in different contexts, and where assertions do not enable emulation, i.e., infinite queries would be required (\ref{sec:side-effects}). Intuitively, this means that assertions do not provide enough signal for a Turing-complete emulator to fully ``understand'' languages from this class. We go on to discuss differences between our formal model and the less well-defined context of natural language (\ref{sec:towards-nl}).&#10;These results provide a formal way to characterize upper bounds on whether it is possible to emulate the semantics of a language from distributional properties of strings.&#10;\upd{Within our framework, in certain settings, we find that meaning cannot be learned from text alone.}&#10;We strengthen claims made by \cite{bk-2020} that assertions in code do not necessarily provide sufficient signal for a language model to emulate understanding.&#10;We do not make strong claims about how these results transfer to natural language, although we expect that the added complexity of natural language would make it, if anything, more difficult to ``understand'' than code.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Formal Semantics, Computer Science, Semantic Representation, Linguistics, Language Model Limitations, Natural Language Understanding, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1901.11373" label="1901.11373">
        <attvalues>
          <attvalue for="0" value="Learning and Evaluating General Linguistic Intelligence" />
          <attvalue for="1" value="  We define general linguistic intelligence as the ability to reuse previously&#10;acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic&#10;conventions to adapt to new tasks quickly. Using this definition, we analyze&#10;state-of-the-art natural language understanding models and conduct an extensive&#10;empirical investigation to evaluate them against these criteria through a&#10;series of experiments that assess the task-independence of the knowledge being&#10;acquired by the learning process. In addition to task performance, we propose a&#10;new evaluation metric based on an online encoding of the test data that&#10;quantifies how quickly an existing agent (model) learns a new task. Our results&#10;show that while the field has made impressive progress in terms of model&#10;architectures that generalize to many tasks, these models still require a lot&#10;of in-domain training examples (e.g., for fine tuning, training task-specific&#10;modules), and are prone to catastrophic forgetting. Moreover, we find that far&#10;from solving general tasks (e.g., document question answering), our models are&#10;overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss&#10;missing components and conjecture on how to make progress toward general&#10;linguistic intelligence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03446" label="2010.03446">
        <attvalues>
          <attvalue for="0" value="Analogies minus analogy test: measuring regularities in word embeddings" />
          <attvalue for="1" value="  Vector space models of words have long been claimed to capture linguistic&#10;regularities as simple vector translations, but problems have been raised with&#10;this claim. We decompose and empirically analyze the classic arithmetic word&#10;analogy test, to motivate two new metrics that address the issues with the&#10;standard test, and which distinguish between class-wise offset concentration&#10;(similar directions between pairs of words drawn from different broad classes,&#10;such as France--London, China--Ottawa, ...) and pairing consistency (the&#10;existence of a regular transformation between correctly-matched pairs such as&#10;France:Paris::China:Beijing). We show that, while the standard analogy test is&#10;flawed, several popular word embeddings do nevertheless encode linguistic&#10;regularities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The validity of the arithmetic analogy test has been questioned in several papers, starting with \cite{levy-goldberg-2014-linguistic}. We detail in Section \ref{sec:analysis} several major issues with the test as raised by \cite{levy-goldberg-2014-linguistic}, \cite{Linzen_2016_Issues_in_evaluating_semantic_spaces_using_word_analogies}, and \cite{RogersDrozdEtAl_2017_Too_Many_Problems_of_Analogical_Reasoning_with_Word_Vectors}. \cite{finley-etal-2017-analogies}, \cite{newman-griffis-etal-2017-insights}, \cite{chen-2017} and \cite{schluter-2018-word} also raised concerns about the test and its assumptions. &#10;More recently, \cite{fairisbetter} argued against this test as an inadequate tool for studying bias in word embeddings. \cite{Rogers_2019_analogies} observes that many of the issues have been ignored.&#10;&#10;Some works have proposed other measures of linguistic relations in word embeddings.&#10;\cite{levy-goldberg-2014-linguistic}, \cite{vylomova-etal-2016-take}, and \cite{RogersDrozdEtAl_2017_Too_Many_Problems_of_Analogical_Reasoning_with_Word_Vectors} all examined the similarity of vector offsets in a more direct way than the standard analogy test (see Section \ref{sec:analysis} below).&#10;\cite{DrozdGladkovaEtAl_2016_Word_embeddings_analogies_and_machine_learning_beyond_king_man_woman_queen} proposed a method based on predicting the class of the unknown word, and \cite{bouraoui-etal-2018-relation} relaxed the assumptions by allowing probabilistic models to predict the relations. &#10;&#10;We claim that these works still do not provide a satisfactory measure of how well word vector offsets encode linguistic relations. Without such a measure, it is impossible to assess whether the original conclusions are correct. We develop this argument, and then develop a new measure, below.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Vector Space Models, Linguistic Regularities, Linguistics, Word Embeddings, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2403.00827" label="2403.00827">
        <attvalues>
          <attvalue for="0" value="Self-Refinement of Language Models from External Proxy Metrics Feedback" />
          <attvalue for="1" value="  It is often desirable for Large Language Models (LLMs) to capture multiple&#10;objectives when providing a response. In document-grounded response generation,&#10;for example, agent responses are expected to be relevant to a user's query&#10;while also being grounded in a given document. In this paper, we introduce&#10;Proxy Metric-based Self-Refinement (ProMiSe), which enables an LLM to refine&#10;its own initial response along key dimensions of quality guided by external&#10;metrics feedback, yielding an overall better final response. ProMiSe leverages&#10;feedback on response quality through principle-specific proxy metrics, and&#10;iteratively refines its response one principle at a time. We apply ProMiSe to&#10;open source language models Flan-T5-XXL and Llama-2-13B-Chat, to evaluate its&#10;performance on document-grounded question answering datasets, MultiDoc2Dial and&#10;QuAC, demonstrating that self-refinement improves response quality. We further&#10;show that fine-tuning Llama-2-13B-Chat on the synthetic dialogue data generated&#10;by ProMiSe yields significant performance improvements over the zero-shot&#10;baseline as well as a supervised fine-tuned model on human annotated data.&#10;" />
          <attvalue for="2" value="&#10;&#10;The state-of-the-art large language models (LLMs) have demonstrated to be effective in generating new synthetic data, useful in improving zero-shot task generalization through fine-tuning without requiring vast amounts of human annotations. Various approaches have been proposed to show the ability of models to evaluate and critique responses \cite{saunders2022selfcritiquing, scheurer2023training, shinn2023reflexion, selfee2023},&#10;as well as their potential to refine: given feedback, correct their outputs \cite{welleck2022generating, peng2023check, madaan2023selfrefine, huang2023large, wang2023enable}. These explorations have studied various feedback mechanisms (human-in-the-loop, reward models to capture human preferences, model-generated feedback) and forms (pairwise comparisons, scalar scores, natural language descriptions), as well as refinement techniques (separate supervised refiners, domain-specific refinement). &#10;&#10;\blfootnote{$^{*}$ Work done during internship at IBM Research AI.}&#10;Of particular note are recent works exploring the self-refinement phenomenon \cite{madaan2023selfrefine, wang2023enable, shinn2023reflexion}, leveraging the same LLM to perform critique and/or refinement on top of generating responses. The observations of these works unveil shortcomings: smaller instruction-tuned models fail to replicate the results of systems such as GPT-3.5 and GPT-4 in refinement, and in the absence of well-designed stopping mechanisms, &#10;self-refinement applied to high-quality responses can make the results worse \cite{huang2023large}. When humans correct themselves, they do it often with one or more objectives in mind, i.e. principles. &#10;Such principles may include faithfulness, specificity, safety (i.e. non-toxic), relevance to a question posed, etc. and may vary across tasks — we seek to imbue these aspects into conversational agents, to ensure they are reflected in the agent's responses.&#10;&#10;To this effect, we introduce an iterative, principle-guided approach to self-refinement in relatively smaller language models where refinement has previously proven unsuccessful. Our algorithm, termed Proxy Metric-based Self-Refinement (ProMiSe), combines proxy metric thresholding for different principles with independent principle-specific few-shot refinement and best-of-N rejection sampling. This allows for the deliberate selection of task-appropriate metrics with calibrated sufficiency thresholds, and specific prompts better designed to match the instruction-following capabilities of smaller models. In this manner, we perform multi-aspect self-refinement via iterative single-aspect improvement queries, as opposed to simultaneous refinement on many dimensions. &#10;&#10;We apply this method to content-grounded question answering, demonstrating consistent improvements on a diverse set of evaluation metrics for single-turn response generation.&#10;We then extend ProMiSe to multi-turn dialogue data generation to generate user queries in addition to agent responses. We fine-tune llama-2-13b-chat on the synthetic data, yielding significant improvement over the zero-shot baseline and supervised models solely fined-tuned on human annotations. &#10;&#10;Crucially, this approach is built on open-source models and does not rely on propietary models with black-box API access; we note, however, that the proposed algorithm can be directly applied to closed-source models as well. Furthermore, it can be extended to other tasks, provided that proxy metrics can be defined and a few in-context exemplars can be created for the relevant principles. &#10;Our key contributions are:&#10;&#10;\begin{itemize}&#10;\item We introduce a novel domain-agnostic algorithm, ProMiSe, to perform multi-aspect self-refinement on desirable principles for a response through in-context learning, using proxy metrics as external quality feedback. &#10;&#10;\item ProMiSe is applied to both content-grounded single-turn question answering and multi-turn dialogue generation. Extensive evaluations on MultiDoc2Dial and QuAC datasets with 5 automatic evaluation metrics (RougeL, Bert-Recall, Bert-K-Precision, Recall, K-Precision) as well as LLM-as-judge with GPT-4, demonstrate its effectivenss both in few-shot and fine-tuning settings. We will release both the software and the synthetic dialogue data.&#10;&#10;\item We analyze the relationship between the change in proxy metric scores and the downstream evaluation metrics, revealing an unsupervised correlation and reinforcing the efficacy of our method.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Self Supervised Learning, Language Model Refinement, Linguistics, Artificial Intelligence, Mathematics, Document Grounded Response, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2206.05802" label="2206.05802">
        <attvalues>
          <attvalue for="0" value="Self-critiquing models for assisting human evaluators" />
          <attvalue for="1" value="  We fine-tune large language models to write natural language critiques&#10;(natural language critical comments) using behavioral cloning. On a topic-based&#10;summarization task, critiques written by our models help humans find flaws in&#10;summaries that they would have otherwise missed. Our models help find naturally&#10;occurring flaws in both model and human written summaries, and intentional&#10;flaws in summaries written by humans to be deliberately misleading. We study&#10;scaling properties of critiquing with both topic-based summarization and&#10;synthetic tasks. Larger models write more helpful critiques, and on most tasks,&#10;are better at self-critiquing, despite having harder-to-critique outputs.&#10;Larger models can also integrate their own self-critiques as feedback, refining&#10;their own summaries into better ones. Finally, we motivate and introduce a&#10;framework for comparing critiquing ability to generation and discrimination&#10;ability. Our measurements suggest that even large models may still have&#10;relevant knowledge they cannot or do not articulate as critiques. These results&#10;are a proof of concept for using AI-assisted human feedback to scale the&#10;supervision of machine learning systems to tasks that are difficult for humans&#10;to evaluate directly. We release our training datasets, as well as samples from&#10;our critique assistance experiments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Scalable alignment experiments.&#10; \cite{christiano2018supervising} implement iterative amplification for algorithmic tasks. \cite{irving2018ai} introduce debate and implement a toy version with sparse MNIST classification. \cite{ought2020arguments,barnes2020progress,barnes2020debate,parrish2022single} conduct debate-like experiments on realistic tasks (checking claims about movie reviews, physics problems, and reading comprehension), with humans serving as debaters, generally with mixed results. Conversely, \cite{anil2021learning} study variants of debate with learned models serving as judges on toy tasks. \cite{wu2021recursively} implements a variant of recursive reward modeling \cite{leike2018scalable} on summarization tasks.&#10;&#10;Human assistance with natural language.&#10;\cite{liu2022wanli} use assistance to help humans create demonstrations to create challenging NLI datasets.&#10;\cite{ziegler2022adversarial} and \cite{perez2022red} use model assistance to find adversarial examples for language model classifications and generations, respectively. \cite{perez2019finding} help humans perform passage-based question-answering, without reading much of the passages.&#10;&#10;For helping humans with evaluations, \cite{fan2020generating} help humans fact-check claims faster and more accurately with natural language briefs. \cite{gehrmann2019gltr} use language models to help humans discriminate whether text was generated by a model.&#10;&#10;Critique datasets and models. \cite{thorne2018fever} introduce a dataset of factual claims, along with supporting and refuting evidence. \cite{kang2018dataset} introduce a dataset of critical peer reviews. \cite{bosc2016dart} mines disagreements from Twitter, and \cite{zhang2017characterizing,pougue2021debagreement} from Reddit. \cite{matiana2021cut} introduce a dataset of story critiques. &#10;&#10;For model generated critiques, IBM's Project Debater \cite{slonim2021autonomous} trains models to engage in free text debates, including the ability to rebut arguments. Unlike our work, they focus on debating against humans rather than models.&#10;&#10;Natural language refinements. &#10;Human natural language feedback has been used to improve models in many domains, such as computer vision \cite{rupprecht2018guide}, program synthesis \cite{elgohary2020speak,austin2021program}, and summarization \cite{scheurer2022training}. \cite{pearce2021can} use large language models to fix security vulnerabilities in code. More recently, \cite{wei2022chain} propose using language models' own outputs to improve their answers on math word problems.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2211.00053" label="2211.00053">
        <attvalues>
          <attvalue for="0" value="Generating Sequences by Learning to Self-Correct" />
          <attvalue for="1" value="  Sequence generation applications require satisfying semantic constraints,&#10;such as ensuring that programs are correct, using certain keywords, or avoiding&#10;undesirable content. Language models, whether fine-tuned or prompted with&#10;few-shot demonstrations, frequently violate these constraints, and lack a&#10;mechanism to iteratively revise their outputs. Moreover, some powerful language&#10;models are of extreme scale or inaccessible, making it inefficient, if not&#10;infeasible, to update their parameters for task-specific adaptation. We present&#10;Self-Correction, an approach that decouples an imperfect base generator (an&#10;off-the-shelf language model or supervised sequence-to-sequence model) from a&#10;separate corrector that learns to iteratively correct imperfect generations. To&#10;train the corrector, we propose an online training procedure that can use&#10;either scalar or natural language feedback on intermediate imperfect&#10;generations. We show that Self-Correction improves upon the base generator in&#10;three diverse generation tasks - mathematical program synthesis,&#10;lexically-constrained generation, and toxicity control - even when the&#10;corrector is much smaller than the base generator.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Self-correction relates to recent works on editing text, including modeling Wikipedia edits \cite{reid2022learning,faltings-etal-2021-text,schick2022peer}, which relies on supervised edits, unsupervised methods \cite{Miao_Zhou_Mou_Yan_Li_2019,liu-etal-2020-unsupervised} that perturb sequences with simple operations (e.g. insertion, deletion),&#10; editing with models trained on human-written critiques~\cite{saunders2022self}, or iteratively updating continuous variables~\cite{lee-etal-2020-iterative,Li2022DiffusionLMIC,qin2022cold}.&#10;In contrast to these, self-correction learns an expressive text-to-text corrector that is trained online to improve a quality measure, without requiring a supervised dataset of edits or critiques.&#10;Separately, denoising ground-truth sequences is a common pretraining objective~\cite{devlin-etal-2019-bert,lewis-etal-2020-bart,RaffelT5}, while self-correction `denoises' generations to improve a scalar quality measure.&#10;Scalar measures are often improved with reinforcement learning (RL) on a base generator~\cite{ziegler2019finetuning,stiennon2020,quark22}, which is infeasible for improving many language models (e.g. those accessed through an API), and uses only scalar feedback. &#10;Moreover, self-correction learns the difference between a generation and solution, and is complementary to RL-tuned generators, which can be used within a self-corrector.&#10;Finally, self-correction decomposes generation into multiple steps, which relates to methods &#10;that generate rationales before a response~\cite{Wei2022ChainOT,Dohan2022LanguageMC}.&#10;Self-correction also produces intermediate steps, but each step is of the same form as the output, allowing for re-using previous generations.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2310.01798" label="2310.01798">
        <attvalues>
          <attvalue for="0" value="Large Language Models Cannot Self-Correct Reasoning Yet" />
          <attvalue for="1" value="  Large Language Models (LLMs) have emerged as a groundbreaking technology with&#10;their unparalleled text generation capabilities across various applications.&#10;Nevertheless, concerns persist regarding the accuracy and appropriateness of&#10;their generated content. A contemporary methodology, self-correction, has been&#10;proposed as a remedy to these issues. Building upon this premise, this paper&#10;critically examines the role and efficacy of self-correction within LLMs,&#10;shedding light on its true potential and limitations. Central to our&#10;investigation is the notion of intrinsic self-correction, whereby an LLM&#10;attempts to correct its initial responses based solely on its inherent&#10;capabilities, without the crutch of external feedback. In the context of&#10;reasoning, our research indicates that LLMs struggle to self-correct their&#10;responses without external feedback, and at times, their performance even&#10;degrades after self-correction. Drawing from these insights, we offer&#10;suggestions for future research and practical applications in this field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.10481" label="1909.10481">
        <attvalues>
          <attvalue for="0" value="Cross-Lingual Natural Language Generation via Pre-Training" />
          <attvalue for="1" value="  In this work we focus on transferring supervision signals of natural language&#10;generation (NLG) tasks between multiple languages. We propose to pretrain the&#10;encoder and the decoder of a sequence-to-sequence model under both monolingual&#10;and cross-lingual settings. The pre-training objective encourages the model to&#10;represent different languages in the shared space, so that we can conduct&#10;zero-shot cross-lingual transfer. After the pre-training procedure, we use&#10;monolingual data to fine-tune the pre-trained model on downstream NLG tasks.&#10;Then the sequence-to-sequence model trained in a single language can be&#10;directly evaluated beyond that language (i.e., accepting multi-lingual input&#10;and producing multi-lingual output). Experimental results on question&#10;generation and abstractive summarization show that our model outperforms the&#10;machine-translation-based pipeline methods for zero-shot cross-lingual&#10;generation. Moreover, cross-lingual transfer improves NLG performance of&#10;low-resource languages by leveraging rich-resource language data. Our&#10;implementation and data are available at https://github.com/CZWin32768/xnlg.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Cross-Lingual NLG&#10;&#10;Several previous methods have been proposed for cross-lingual abstractive summarization.&#10;\cite{xnhg}~\shortcite{xnhg} and \cite{xsummacl}~\shortcite{xsummacl} use translated documents or summaries as pseudo training data.&#10;\cite{ncls}~\shortcite{ncls} incorporate monolingual summarization and machine translation to improve cross-lingual summarization.&#10;However, the systems only conduct experiments that generate summaries with different languages from the input language, rather than transferring supervision signals across all language pairs.&#10;\cite{kumar2019cross}~\shortcite{kumar2019cross} use training data annotated in multiple languages to jointly train a sequence-to-sequence model for question generation.&#10;In contrast, our method can also be applied to zero-shot settings across languages.&#10;&#10;Monolingual Pre-Training&#10;&#10;Various training objectives are designed to pretrain text encoders used for general-purpose representations, such as language modeling~\cite{elmo,gpt,bert,spanbert,xlnet}, auto-encoding~\cite{pretrain-dae}, and machine translation~\cite{mccann2017learned}.&#10;Apart from pre-training encoders, several pre-trained models~\cite{unilm,mass} are proposed for generation tasks.&#10;In comparison, our goal is to investigate a pre-training method for cross-lingual NLG tasks.&#10;&#10;Cross-Lingual Pre-Training&#10;&#10;By pre-training BERT~\cite{bert} on corpus of multiple languages, it shows a surprising ability to produce cross-lingual representations~\cite{wu2019beto}.&#10;More recently, \cite{xlm}~\shortcite{xlm} extend mask language modeling pre-training to cross-lingual settings, which shows significant improvements on cross-lingual classification and unsupervised machine translation.&#10;By comparison, we pretrain both encoder and decoder for cross-lingual generation tasks, rather than only focusing on encoder.&#10;\cite{artetxe2018massively}~\shortcite{artetxe2018massively} use the sequence encoder of the multilingual translation model~\cite{johnson2017google} to produce cross-lingual sentence embeddings.&#10;However, as shown in the experiments (Section~\ref{sec:exp}), it is difficult to control the target language by directly fine-tuning the pre-trained translation model on downstream NLG tasks.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Multilingual Modeling, Cross-Lingual Transfer, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2210.15424" label="2210.15424">
        <attvalues>
          <attvalue for="0" value="What Language Model to Train if You Have One Million GPU Hours?" />
          <attvalue for="1" value="  The crystallization of modeling methods around the Transformer architecture&#10;has been a boon for practitioners. Simple, well-motivated architectural&#10;variations can transfer across tasks and scale, increasing the impact of&#10;modeling research. However, with the emergence of state-of-the-art 100B+&#10;parameters models, large language models are increasingly expensive to&#10;accurately design and train. Notably, it can be difficult to evaluate how&#10;modeling decisions may impact emergent capabilities, given that these&#10;capabilities arise mainly from sheer scale alone. In the process of building&#10;BLOOM--the Big Science Large Open-science Open-access Multilingual language&#10;model--our goal is to identify an architecture and training setup that makes&#10;the best use of our 1,000,000 A100-GPU-hours budget. Specifically, we perform&#10;an ablation study at the billion-parameter scale comparing different modeling&#10;practices and their impact on zero-shot generalization. In addition, we study&#10;the impact of various popular pre-training corpora on zero-shot generalization.&#10;We also study the performance of a multilingual model and how it compares to&#10;the English-only one. Finally, we consider the scaling behaviour of&#10;Transformers to choose the target model size, shape, and training setup. All&#10;our models and code are open-sourced at https://huggingface.co/bigscience .&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years have seen the advent of large language models &#10;characterized by emergent capabilities (e.g., zero-shot generalization) arising from sheer scale alone~\cite{radford2019language,brown2020gpt3}.&#10;Scaling LLMs results in a predictable increase in performance: simple scaling laws connect the number of parameters, pretraining dataset size, and compute budget~\cite{kaplan2020scaling,ganguli2022predictability,hoffmann2022training}, providing a clear path towards more capable models. This paradigm shift has been fueled by the wide~adoption of the Transformer~\cite{vaswani2017attention}, providing a scalable basis for practitioners to build upon. &#10;&#10;In this paper, we design an architecture and training setup for a multilingual 100B+ parameters model (BLOOM, \cite{bigscience_workshop_2022}), seeking to best use a fixed 1,000,000 A100-hours budget. Because of the costs involved with training large language models, we cannot exhaustively explore the landscape of possible models. Instead, we position ourselves as practitioners exploring &quot;off-the-shelf&quot; solutions. We thus test promising additions to the Transformer to attempt to reproduce their findings in a controlled, large-scale setting.&#10;&#10;Although our main goal was to prepare the architecture and training setup of BLOOM, our findings are also valuable for practitioners building models in the 1-10B range, as they equally improve the performance of such smaller models. At variance with major works on large language models, we also make a significant effort towards reproducibility and openness: all of our pretrained models, code, and notes from our weekly meetings are made available. See Appendix \ref{sec:artefacts} for the relevant links.&#10;&#10;Contributions. We first study the impact of pretraining corpora, positional embeddings, activation functions, and embedding norm on zero-shot generalization. We base our study on the popular GPT-2 architecture \cite{radford2019language}, with experiments at the 1.3B parameters scale. We then consider the impact of massive multilinguality, showing language-specific scaling laws in a multilingual setting for the first time. Finally, we describe our approach to drafting an architecture for the final 176B parameters BLOOM model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Multilingual Modeling, Mathematics, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="2202.07785" label="2202.07785">
        <attvalues>
          <attvalue for="0" value="Predictability and Surprise in Large Generative Models" />
          <attvalue for="1" value="  Large-scale pre-training has recently emerged as a technique for creating&#10;capable, general purpose, generative models such as GPT-3, Megatron-Turing NLG,&#10;Gopher, and many others. In this paper, we highlight a counterintuitive&#10;property of such models and discuss the policy implications of this property.&#10;Namely, these generative models have an unusual combination of predictable loss&#10;on a broad training distribution (as embodied in their &quot;scaling laws&quot;), and&#10;unpredictable specific capabilities, inputs, and outputs. We believe that the&#10;high-level predictability and appearance of useful capabilities drives rapid&#10;development of such models, while the unpredictable qualities make it difficult&#10;to anticipate the consequences of model deployment. We go through examples of&#10;how this combination can lead to socially harmful behavior with examples from&#10;the literature and real world observations, and we also perform two novel&#10;experiments to illustrate our point about harms from unpredictability.&#10;Furthermore, we analyze how these conflicting properties combine to give model&#10;developers various motivations for deploying these models, and challenges that&#10;can hinder deployment. We conclude with a list of possible interventions the AI&#10;community may take to increase the chance of these models having a beneficial&#10;impact. We intend this paper to be useful to policymakers who want to&#10;understand and regulate AI systems, technologists who care about the potential&#10;policy impact of their work, and academics who want to analyze, critique, and&#10;potentially develop large generative models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.11890" label="2406.11890">
        <attvalues>
          <attvalue for="0" value="Unraveling the Mechanics of Learning-Based Demonstration Selection for&#10;  In-Context Learning" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated impressive in-context learning&#10;(ICL) capabilities from few-shot demonstration exemplars. While recent&#10;learning-based demonstration selection methods have proven beneficial to ICL by&#10;choosing more useful exemplars, their underlying mechanisms are opaque,&#10;hindering efforts to address limitations such as high training costs and poor&#10;generalization across tasks. These methods generally assume the selection&#10;process captures similarities between the exemplar and the target instance,&#10;however, it remains unknown what kinds of similarities are captured and vital&#10;to performing ICL. To dive into this question, we analyze the working&#10;mechanisms of the learning-based demonstration selection methods and&#10;empirically identify two important factors related to similarity measurement:&#10;1) The ability to integrate different levels of task-agnostic text similarities&#10;between the input of exemplars and test cases enhances generalization power&#10;across different tasks. 2) Incorporating task-specific labels when measuring&#10;the similarities significantly improves the performance on each specific task.&#10;We validate these two findings through extensive quantitative and qualitative&#10;analyses across ten datasets and various LLMs. Based on our findings, we&#10;introduce two effective yet simplified exemplar selection methods catering to&#10;task-agnostic and task-specific demands, eliminating the costly LLM inference&#10;overhead.&#10;" />
          <attvalue for="2" value="&#10;In-context learning (ICL) has emerged as a promising paradigm that employs a sequence of demonstration exemplars as prompts to assist large language models (LLMs) in effectively performing unseen tasks~\cite{nips20llmfewshotlearner, votek}. However, the performance of ICL can be sensitive to the choice, format, and order of the in-context exemplar~\cite{ICLfactor1, ICLfactor2, ICLformat1, ACL23order2}. To mitigate this challenge, given a test case $x^t$, the exemplar selection task assumes access to a demonstration set $\mathcal{D}$ containing input-output pairs $(x,y)$ and focuses on selecting the most effective exemplar from $\mathcal{D}$ to inform the target output $y^t$.&#10;&#10;To address this task, it is the most common practice to select demonstration exemplars based on a similarity measurement between $x$ and $x^t$~\cite{epr, CEIL, emnlp23skillselection1, diversity1, emnlp23skillselection3, ACL23order3}. Some work utilizes task-agnostic similarity like term frequency-based similarity BM25 and semantic similarity computed by off-the-shelf text encoders~\cite{ACL22retriever1, emnlp23skillselection1}. Recent learning-based studies~\cite{epr, CEIL, li-etal-2023-unified}, however, separately train a retriever to learn implicit similarity measurements using a contrastive leaning-based proxy task where positive exemplars $x^+$ and negative exemplars $x^-$ are labeled by interacting with LLMs. This data creation process often requires hundreds of thousands of queries to LLMs for each task to collect sufficient positive/negative data. &#10;&#10;Although learning-based methods consistently exhibit significant performance improvements over task-agnostic similarity across various tasks, the implicit similarity they capture and their connection to the performance of ICL remain unclear. Through a detailed examination of previous works, we observe 1) While the low-level similarity like BM25 and semantic similarity excel in different tasks (e.g., Top-K BM25 outperforms Top-K BERT on Nl2Bash~\cite{datasetnl2bash} and SWAG~\cite{dataset:swag} in Table~\ref{man:cls} and Table~\ref{man:gen}), learning-based similarity generally performs well across all tasks. 2) In the proxy task, the input and output similarity between positive exemplars and test cases is higher than that of negative exemplars and test cases. Moreover, learning-based methods often suffer from poor generalization across different tasks, as corroborated by findings in~\cite{CEIL}. Based on these initial observations, we propose two hypotheses regarding learning-based methods:&#10;&#10;$\mathcal{H}_1$: After training, the retriever acts as an ensemble model that adaptively integrates multi-level task-agnostic similarities between the exemplar input ($x$) and test cases ($x^t$) for different tasks.&#10;&#10;$\mathcal{H}_2$: Beyond input similarities, the training process encourages selecting exemplars with similar output ($y$) to the output of the test case ($y^t$), implicitly predicted during retrieval, enhancing the retriever's discriminative power for a specific task.&#10;&#10;Extensive quantitative experiments are designed to validate these hypotheses: 1) We take various layers of BERT as anchors for similarities of different levels and discover learning-based methods exhibiting varying preferences for these anchors before and after training, suggesting an adaptive combination of these similarities tailored to different tasks. 2) We investigate the exemplar retrieved by learning-based methods and find these exemplars show a higher similarity in output to the test case than other task-agnostic similarity-based methods. This finding indicates that learning-based methods incorporate task-specific similarities between the outputs of exemplars and test cases during the exemplar selection process, potentially capturing the joint distribution of inputs and outputs between exemplars and test cases. Additionally, by connecting our findings with existing interpretative theories of ICL~\cite{iclinductionhead, randomlabeliswrong1, Repetitionsiclr24, halawi2023overthinking, anchors}, we further qualitatively validate our conclusions.&#10;&#10;Drawing insights from these findings, we propose two cost-effective exemplar selection methods: 1) Multi-level Similarity Maximization (MLSM) retriever that maximizes agreement across different similarity levels represented by various layers of BERT in the inference of LLMs. 2) Test Task Fine-tuning (TTF) retriever, which uses labeled data from the demonstration set to finetune the retriever to learn task-specific information. Both retrievers eliminate the need for costly data collection for the proxy task, catering to cross-task and task-specific demands. To validate the effectiveness of these methods, we conduct experiments across five distinct LLMs and a range of tasks. These promising applications confirm our hypotheses and benefit future demonstration selection studies for more efficient LLM deployment.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Language Model Learning, Task Generalization Techniques, Artificial Intelligence, Mathematics, Natural Language Processing, Demonstration Selection Methods" />
        </attvalues>
      </node>
      <node id="2112.08633" label="2112.08633">
        <attvalues>
          <attvalue for="0" value="Learning To Retrieve Prompts for In-Context Learning" />
          <attvalue for="1" value="  In-context learning is a recent paradigm in natural language understanding,&#10;where a large pre-trained language model (LM) observes a test instance and a&#10;few training examples as its input, and directly decodes the output without any&#10;update to its parameters. However, performance has been shown to strongly&#10;depend on the selected training examples (termed prompt). In this work, we&#10;propose an efficient method for retrieving prompts for in-context learning&#10;using annotated data and a LM. Given an input-output pair, we estimate the&#10;probability of the output given the input and a candidate training example as&#10;the prompt, and label training examples as positive or negative based on this&#10;probability. We then train an efficient dense retriever from this data, which&#10;is used to retrieve training examples as prompts at test time. We evaluate our&#10;approach on three sequence-to-sequence tasks where language utterances are&#10;mapped to meaning representations, and find that it substantially outperforms&#10;prior work and multiple baselines across the board.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.07830" label="1905.07830">
        <attvalues>
          <attvalue for="0" value="HellaSwag: Can a Machine Really Finish Your Sentence?" />
          <attvalue for="1" value="  Recent work by Zellers et al. (2018) introduced a new task of commonsense&#10;natural language inference: given an event description such as &quot;A woman sits at&#10;a piano,&quot; a machine must select the most likely followup: &quot;She sets her fingers&#10;on the keys.&quot; With the introduction of BERT, near human-level performance was&#10;reached. Does this mean that machines can perform human level commonsense&#10;inference?&#10;  In this paper, we show that commonsense inference still proves difficult for&#10;even state-of-the-art models, by presenting HellaSwag, a new challenge dataset.&#10;Though its questions are trivial for humans (&gt;95% accuracy), state-of-the-art&#10;models struggle (&lt;48%). We achieve this via Adversarial Filtering (AF), a data&#10;collection paradigm wherein a series of discriminators iteratively select an&#10;adversarial set of machine-generated wrong answers. AF proves to be&#10;surprisingly robust. The key insight is to scale up the length and complexity&#10;of the dataset examples towards a critical 'Goldilocks' zone wherein generated&#10;text is ridiculous to humans, yet often misclassified by state-of-the-art&#10;models.&#10;  Our construction of HellaSwag, and its resulting difficulty, sheds light on&#10;the inner workings of deep pretrained models. More broadly, it suggests a new&#10;path forward for NLP research, in which benchmarks co-evolve with the evolving&#10;state-of-the-art in an adversarial way, so as to present ever-harder&#10;challenges.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.05698" label="2302.05698">
        <attvalues>
          <attvalue for="0" value="Compositional Exemplars for In-context Learning" />
          <attvalue for="1" value="  Large pretrained language models (LMs) have shown impressive In-Context&#10;Learning (ICL) ability, where the model learns to do an unseen task via a&#10;prompt consisting of input-output examples as the demonstration, without any&#10;parameter updates. The performance of ICL is highly dominated by the quality of&#10;the selected in-context examples. However, previous selection methods are&#10;mostly based on simple heuristics, leading to sub-optimal performance. In this&#10;work, we formulate in-context example selection as a subset selection problem.&#10;We propose CEIL (Compositional Exemplars for In-context Learning), which is&#10;instantiated by Determinantal Point Processes (DPPs) to model the interaction&#10;between the given input and in-context examples, and optimized through a&#10;carefully-designed contrastive learning objective to obtain preference from&#10;LMs. We validate CEIL on 12 classification and generation datasets from 7&#10;distinct NLP tasks, including sentiment analysis, paraphrase detection, natural&#10;language inference, commonsense reasoning, open-domain question answering, code&#10;generation, and semantic parsing. Extensive experiments demonstrate not only&#10;the state-of-the-art performance but also the transferability and&#10;compositionality of CEIL, shedding new light on effective and efficient&#10;in-context learning. Our code is released at&#10;https://github.com/HKUNLP/icl-ceil.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.11895" label="2209.11895">
        <attvalues>
          <attvalue for="0" value="In-context Learning and Induction Heads" />
          <attvalue for="1" value="  &quot;Induction heads&quot; are attention heads that implement a simple algorithm to&#10;complete token sequences like [A][B] ... [A] -&gt; [B]. In this work, we present&#10;preliminary and indirect evidence for a hypothesis that induction heads might&#10;constitute the mechanism for the majority of all &quot;in-context learning&quot; in large&#10;transformer models (i.e. decreasing loss at increasing token indices). We find&#10;that induction heads develop at precisely the same point as a sudden sharp&#10;increase in in-context learning ability, visible as a bump in the training&#10;loss. We present six complementary lines of evidence, arguing that induction&#10;heads may be the mechanistic source of general in-context learning in&#10;transformer models of any size. For small attention-only models, we present&#10;strong, causal evidence; for larger models with MLPs, we present correlational&#10;evidence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.12272" label="2304.12272">
        <attvalues>
          <attvalue for="0" value="AMR Parsing with Instruction Fine-tuned Pre-trained Language Models" />
          <attvalue for="1" value="  Instruction fine-tuned language models on a collection of instruction&#10;annotated datasets (FLAN) have shown highly effective to improve model&#10;performance and generalization to unseen tasks. However, a majority of standard&#10;parsing tasks including abstract meaning representation (AMR), universal&#10;dependency (UD), semantic role labeling (SRL) has been excluded from the FLAN&#10;collections for both model training and evaluations. In this paper, we take one&#10;of such instruction fine-tuned pre-trained language models, i.e. FLAN-T5, and&#10;fine-tune them for AMR parsing. Our extensive experiments on various AMR&#10;parsing tasks including AMR2.0, AMR3.0 and BioAMR indicate that FLAN-T5&#10;fine-tuned models out-perform previous state-of-the-art models across all&#10;tasks. In addition, full fine-tuning followed by the parameter efficient&#10;fine-tuning, LoRA, further improves the model performances, setting new&#10;state-of-the-arts in Smatch on AMR2.0 (86.4), AMR3.0 (84.9) and BioAMR (82.3).&#10;" />
          <attvalue for="2" value="&#10;&#10;Instruction fine-tuning language models on a collection of annotated datasets has proven highly effective to improve model performance and generalization to unseen tasks both in general purpose open domain setup, as in \cite{flan-t5-2022, flan2021,flan2023,instructgpt, naturalinstructions, supernaturalinstructions, unnatural2022} and specialized tasks such as conversational dialogs in \cite{instructdial2022}.&#10;&#10;Despite its great success in the majority of natural language processing tasks, however, standard parsing tasks such as abstract meaning representation (AMR), \cite{banarescu-etal-2013-abstract, bevilacqua2021aaai,zhou2021emnlp, bai-etal-2022-graph}, universal dependency (UD), \cite{ud2017}, semantic role labeling (SRL), \cite{srl-gildea2002, propbank2005}, etc. have been largely excluded from the fine-tuned language net (FLAN) collections either for model training or evaluations. And therefore it still remains to be seen whether or not instruction fine-tuned language models are as effective for standard parsing tasks as for other NLP tasks.&#10;&#10;In this paper, we fine-tune FLAN-T5 models of \cite{flan-t5-2022} (FLAN\-T5\-Large and FLAN\-T5\-XL) on a wide range of AMR parsing tasks including AMR2.0, AMR3.0 and BioAMR. We show that fine-tuning FLAN-T5 models on AMR parsing leads to a significant improvement over the previous BART fine-tuned SoTA models by \cite{zhou2021emnlp, bai-etal-2022-graph}. We further explore a parameter efficient fine-tuning technique, LoRA (Low Rank Adaptation), \cite{lora2021}. While LoRA\-only fine\-tuned models do not out-perform full fine-tuned models, full fine-tuning followed by LoRA fine-tuning significantly improve full fine-tuned models, setting new state-of-the-arts across all AMR parsing tasks.&#10;&#10;Our main contributions are as follows:&#10;&#10;\begin{itemize}&#10; \item We apply instruction fine-tuned FLAN-T5 models to AMR parsing for the first time. We show that FLAN-T5 fine-tuned AMR parsing models significantly out-perform previous BART fine-tuned SoTA models.&#10; \item We explore the parameter efficient fine-tuning technique LoRA for sequence-to-sequence tasks. Although fine-tuning FLAN-T5 models with LoRA only does not out-perform full fine-tuned models, LoRA fine-tuning of full fine-tuned models further improves model performances.&#10; \item We push the envelope of AMR parsing, by setting new SoTA in Smatch on AMR2.0 (86.4), AMR3.0 (84.9) and BioAMR (82.3).&#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Semantic Parsing Tasks, Language Model Training, Computational Linguistics, Artificial Intelligence, Natural Language Processing, Model Fine-Tuning Techniques" />
        </attvalues>
      </node>
      <node id="2205.12673" label="2205.12673">
        <attvalues>
          <attvalue for="0" value="InstructDial: Improving Zero and Few-shot Generalization in Dialogue&#10;  through Instruction Tuning" />
          <attvalue for="1" value="  Instruction tuning is an emergent paradigm in NLP wherein natural language&#10;instructions are leveraged with language models to induce zero-shot performance&#10;on unseen tasks. Instructions have been shown to enable good performance on&#10;unseen tasks and datasets in both large and small language models. Dialogue is&#10;an especially interesting area to explore instruction tuning because dialogue&#10;systems perform multiple kinds of tasks related to language (e.g., natural&#10;language understanding and generation, domain-specific interaction), yet&#10;instruction tuning has not been systematically explored for dialogue-related&#10;tasks. We introduce InstructDial, an instruction tuning framework for dialogue,&#10;which consists of a repository of 48 diverse dialogue tasks in a unified&#10;text-to-text format created from 59 openly available dialogue datasets. Next,&#10;we explore cross-task generalization ability on models tuned on InstructDial&#10;across diverse dialogue tasks. Our analysis reveals that InstructDial enables&#10;good zero-shot performance on unseen datasets and tasks such as dialogue&#10;evaluation and intent detection, and even better performance in a few-shot&#10;setting. To ensure that models adhere to instructions, we introduce novel&#10;meta-tasks. We establish benchmark zero-shot and few-shot performance of models&#10;trained using the proposed framework on multiple dialogue tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.15534" label="2110.15534">
        <attvalues>
          <attvalue for="0" value="Structure-aware Fine-tuning of Sequence-to-sequence Transformers for&#10;  Transition-based AMR Parsing" />
          <attvalue for="1" value="  Predicting linearized Abstract Meaning Representation (AMR) graphs using&#10;pre-trained sequence-to-sequence Transformer models has recently led to large&#10;improvements on AMR parsing benchmarks. These parsers are simple and avoid&#10;explicit modeling of structure but lack desirable properties such as graph&#10;well-formedness guarantees or built-in graph-sentence alignments. In this work&#10;we explore the integration of general pre-trained sequence-to-sequence language&#10;models and a structure-aware transition-based approach. We depart from a&#10;pointer-based transition system and propose a simplified transition set,&#10;designed to better exploit pre-trained language models for structured&#10;fine-tuning. We also explore modeling the parser state within the pre-trained&#10;encoder-decoder architecture and different vocabulary strategies for the same&#10;purpose. We provide a detailed comparison with recent progress in AMR parsing&#10;and show that the proposed parser retains the desirable properties of previous&#10;transition-based approaches, while being simpler and reaching the new parsing&#10;state of the art for AMR 2.0, without the need for graph re-categorization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.08962" label="1908.08962">
        <attvalues>
          <attvalue for="0" value="Well-Read Students Learn Better: On the Importance of Pre-training&#10;  Compact Models" />
          <attvalue for="1" value="  Recent developments in natural language representations have been accompanied&#10;by large and expensive models that leverage vast amounts of general-domain text&#10;through self-supervised pre-training. Due to the cost of applying such models&#10;to down-stream tasks, several model compression techniques on pre-trained&#10;language representations have been proposed (Sun et al., 2019; Sanh, 2019).&#10;However, surprisingly, the simple baseline of just pre-training and fine-tuning&#10;compact models has been overlooked. In this paper, we first show that&#10;pre-training remains important in the context of smaller architectures, and&#10;fine-tuning pre-trained compact models can be competitive to more elaborate&#10;methods proposed in concurrent work. Starting with pre-trained compact models,&#10;we then explore transferring task knowledge from large fine-tuned models&#10;through standard knowledge distillation. The resulting simple, yet effective&#10;and general algorithm, Pre-trained Distillation, brings further improvements.&#10;Through extensive experiments, we more generally explore the interaction&#10;between pre-training and distillation under two variables that have been&#10;under-studied: model size and properties of unlabeled task data. One surprising&#10;observation is that they have a compound effect even when sequentially applied&#10;on the same data. To accelerate future research, we will make our 24&#10;pre-trained miniature BERT models publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Pre-training&#10;Decades of research have shown that unlabeled text can help learn language representations. Word embeddings were first used~\cite{word2vec,glove}, while subsequently contextual word representations were found more effective~\cite{elmo}. Most recently, research has shifted towards fine-tuning methods \cite{openai-gpt,bert,openai-gpt2}, where entire large pre-trained representations are fine-tuned for end tasks together with a small number of task-specific parameters. While feature-based unsupervised representations have been successfully used in compact models \cite{johnson2015semi, gururangan-etal-2019-variational}, inter alia, the pretraining+fine-tuning approach has not been studied in depth for such small models.&#10;&#10;Learning compact models&#10;In this work we built on model compression~\cite{model-compression} and its variant knowledge distillation~\cite{distillation}. Other related efforts introduced ways to transfer more information from a teacher to a student model, by sharing intermediate layer activations ~\cite{fitnets,yim2017gift,patient_kd}. We experimented with related approaches, but found only slight gains which were dominated by the gains from pre-training and were not complementary. Prior works have also noted the unavailability of in-domain large-scale transfer data and proposed the use of automatically generated pseudo-examples~\cite{model-compression,kimura2018few}. Here we showed that large-scale general domain text can be successfully used for pre-training instead. A separate line of work uses pruning or quantization to derive smaller models ~\cite{han2015deep,Gupta:2015}. Gains from such techniques are expected to be complementary to PD.&#10;&#10;Distillation with unsupervised pre-training&#10;Early efforts to leverage both unsupervised pre-training and distillation provide pre-trained (possibly contextual) word embeddings as inputs to students, rather than pre-training the student stack. For instance, \cite{distillation-reading-comprehension} use ELMo embeddings, while \cite{distillation-classification,tang2019distilling} use context-independent word embeddings. Concurrent work initializes Transformer students from the bottom layers of a 12-layer BERT model \cite{truncated-pretraining,patient_kd,distil_bert}. The latter continues student LM pre-training via distillation from a more expensive LM teacher. For a different purpose of deriving a single model for multiple tasks through distillation, \cite{clark-etal-2019-bam} use a pre-trained student model of the same size as multiple teacher models. However, none of the prior work has analyzed the impact of unsupervised learning for students in relation to the model size and domain of the transfer set.&#10; " />
          <attvalue for="4" value="Computer Science, Linguistics, Knowledge Distillation, Language Model Compression, Artificial Intelligence, Pre-training Techniques" />
        </attvalues>
      </node>
      <node id="2206.11569" label="2206.11569">
        <attvalues>
          <attvalue for="0" value="Mining Error Templates for Grammatical Error Correction" />
          <attvalue for="1" value="  Some grammatical error correction (GEC) systems incorporate hand-crafted&#10;rules and achieve positive results. However, manually defining rules is&#10;time-consuming and laborious. In view of this, we propose a method to mine&#10;error templates for GEC automatically. An error template is a regular&#10;expression aiming at identifying text errors. We use the web crawler to acquire&#10;such error templates from the Internet. For each template, we further select&#10;the corresponding corrective action by using the language model perplexity as a&#10;criterion. We have accumulated 1,119 error templates for Chinese GEC based on&#10;this method. Experimental results on the newly proposed CTC-2021 Chinese GEC&#10;benchmark show that combing our error templates can effectively improve the&#10;performance of a strong GEC system, especially on two error types with very&#10;little training data. Our error templates are available at&#10;\url{https://github.com/HillZhang1999/gec_error_template}.&#10;" />
          <attvalue for="2" value="&#10;Grammatical error correction (GEC) is an important task in natural language processing, which aims at detecting and correcting all underlying errors in a potentially erroneous sentence. Recently, GEC has been receiving increasing attention for its broad application \cite{grundkiewicz2020crash,wang2021comprehensive}. &#10;&#10;Early GEC systems are basically based on error-specific classifiers \cite{rozovskaya2011algorithm,dahlmeier2012beam} or statistic machine translation models \cite{felice2014grammatical,chollampatt2016neural}. Since the beginning of the deep learning era, neural encoder-decoder models, e.g., Transformer \cite{vaswani2017attention}, have emerged as a dominant GEC paradigm \cite{yuan2016grammatical,junczys2018approaching}. Despite their discrepancies, all above methods need to train a model over numerous training samples through the back-propagation algorithm, so we can collectively call them model-based approaches. Besides model-based approaches, there is yet another simple but effective long-standing GEC approach, i.e., the rule-based method, which utilizes pre-defined rules to tackle grammatical errors \cite{madi2018grammatical}. Compared with the model-based approaches, the rule-based method enjoys some merits, such as 1) &#10;fast correction speed; 2) good interpretability and controllability; 3) no training data required, making this method widely applied in various languages \cite{domeij2000granska,sidorov2013syntactic,singh2016frequency,zhou2018chinese}. Moreover, some researchers attempt to combine the rule-based method with the model-based approaches and show that they have complementary abilities \cite{felice2014grammatical, zhang2021ctcreport}.&#10;&#10;Despite its advantages, the rule-based GEC method is less commonly used today, mainly due to the additional labor costs. A grammar rule for GEC formally consists of two parts: 1) an error template to match erroneous spans; 2) a corrective action to correct errors. Existing work usually resorts to human experts to manually define such templates and actions, which inevitably introduces extensive expenses. These extra expenses consequently limit the scale of rules and make their effect marginal. &#10;&#10;To make the rule-based approach more feasible, we propose to automatically mine error templates from websites and devise a method to obtain corrective actions based on language models. The motivation is to minimize human involvement and thus reduce labor. In this work, we focus on the redundant error, which can be easily solved by directly deleting erroneous tokens. Concretely, we represent error templates as regular expressions in the form of ``A.*B'' and utilize a web crawler to extract such templates from the Internet based on several pre-defined search patterns. Then we design three corrective actions: deleting the left part ``A'', deleting the right part ``B'', and randomly deleting one part. To choose an appropriate action, we leverage the perplexity calculated by language models like GPT-2 \cite{radford2019language} as a criterion. &#10;&#10;We conduct experiments on the Chinese GEC task. So far, we have accumulated 1,119 Chinese error templates along with their corrective actions by using our proposed method. Experimental results on the newly proposed CTC-2021 benchmark show that incorporating our templates leads to a significant improvement of 4.50 F${_1}$ score over a strong baseline, which confirms the effectiveness. Specifically, our error templates perform well on two special error types with &#10;very few training samples: meaning duplicates and mixed sentence patterns. As shown in Table \ref{tab:example}, both kinds of errors fit our templates well and can be fixed by the delete operation. Using our templates improves the recall values from 9.59 and 7.41 to 63.01 and 40.74 for meaning duplicates and mixed sentence patterns, respectively.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Grammatical Error Correction, Natural Language Processing, Language Model Optimization, Error Template Mining" />
        </attvalues>
      </node>
      <node id="1606.00189" label="1606.00189">
        <attvalues>
          <attvalue for="0" value="Neural Network Translation Models for Grammatical Error Correction" />
          <attvalue for="1" value="  Phrase-based statistical machine translation (SMT) systems have previously&#10;been used for the task of grammatical error correction (GEC) to achieve&#10;state-of-the-art accuracy. The superiority of SMT systems comes from their&#10;ability to learn text transformations from erroneous to corrected text, without&#10;explicitly modeling error types. However, phrase-based SMT systems suffer from&#10;limitations of discrete word representation, linear mapping, and lack of global&#10;context. In this paper, we address these limitations by using two different yet&#10;complementary neural network models, namely a neural network global lexicon&#10;model and a neural network joint model. These neural networks can generalize&#10;better by using continuous space representation of words and learn non-linear&#10;mappings. Moreover, they can leverage contextual information from the source&#10;sentence more effectively. By adding these two components, we achieve&#10;statistically significant improvement in accuracy for grammatical error&#10;correction over a state-of-the-art GEC system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.00733" label="2303.00733">
        <attvalues>
          <attvalue for="0" value="SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks" />
          <attvalue for="1" value="  Prompt tuning is a technology that tunes a small set of parameters to steer a&#10;pre-trained language model (LM) to directly generate the output for downstream&#10;tasks. Recently, prompt tuning has demonstrated its storage and computation&#10;efficiency in both natural language processing (NLP) and speech processing&#10;fields. These advantages have also revealed prompt tuning as a candidate&#10;approach to serving pre-trained LM for multiple tasks in a unified manner. For&#10;speech processing, SpeechPrompt shows its high parameter efficiency and&#10;competitive performance on a few speech classification tasks. However, whether&#10;SpeechPrompt is capable of serving a large number of tasks is unanswered. In&#10;this work, we propose SpeechPrompt v2, a prompt tuning framework capable of&#10;performing a wide variety of speech classification tasks, covering multiple&#10;languages and prosody-related tasks. The experiment result shows that&#10;SpeechPrompt v2 achieves performance on par with prior works with less than&#10;0.15M trainable parameters in a unified framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Tuning, Computer Science, Speech Processing, Linguistics, Efficient Deep Learning, Artificial Intelligence, Speech Processing Tasks, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2109.04212" label="2109.04212">
        <attvalues>
          <attvalue for="0" value="Efficient Nearest Neighbor Language Models" />
          <attvalue for="1" value="  Non-parametric neural language models (NLMs) learn predictive distributions&#10;of text utilizing an external datastore, which allows them to learn through&#10;explicitly memorizing the training datapoints. While effective, these models&#10;often require retrieval from a large datastore at test time, significantly&#10;increasing the inference overhead and thus limiting the deployment of&#10;non-parametric NLMs in practical applications. In this paper, we take the&#10;recently proposed $k$-nearest neighbors language model (Khandelwal et al.,&#10;2020) as an example, exploring methods to improve its efficiency along various&#10;dimensions. Experiments on the standard WikiText-103 benchmark and&#10;domain-adaptation datasets show that our methods are able to achieve up to a 6x&#10;speed-up in inference speed while retaining comparable performance. The&#10;empirical analysis we present may provide guidelines for future research&#10;seeking to develop or deploy more efficient non-parametric NLMs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models (LMs) are one of the most fundamental technologies in NLP, with applications spanning text generation~\cite{bahdanau2014neural,rush2015neural}, representation learning~\cite{peters2018deep,devlin2019bert,yang2019xlnet}, and few-shot learning~\cite{radford2019language,brown2020language}. &#10;&#10;Modern neural language models (NLMs) based on recurrent~\cite{mikolov2010recurrent,sundermeyer2012lstm} or self-attentional~\cite{vaswani2017attention,al2019character} neural networks are mostly parametric, where the predictions are solely dependent on the model parameters given the input data.&#10;&#10;\begin{SCfigure*}&#10;&#10;\end{SCfigure*}&#10;&#10;In contrast, recent non-parametric LMs~\cite{guu2018generating,khandelwal2019generalization,he2020learning} model text distributions by referencing both the parameters of the underlying model and examples from an external datastore. &#10;Non-parametric LMs are appealing since they allow for effective language modeling -- particularly for rarer patterns -- through explicit memorization via a datastore, which mitigates the burden on model parameters to learn to encode all information from a large dataset. &#10;One effective and representative example is the $k$-nearest neighbors LM ($k$NN-LM,~\cite{khandelwal2019generalization}). The $k$NN-LM computes the probability of the next token by interpolating a parametric LM with a distribution calculated from the $k$ nearest context-token pairs in the datastore, as demonstrated in Figure~\ref{fig:model}.&#10;This model is particularly notable for its large improvements in performance -- it outperforms the previous best parametric LMs by a large margin in standard language modeling benchmarks, in domain adaptation settings, and on other conditional generation tasks such as machine translation \cite{khandelwal2020nearest}.&#10;&#10;However, one downside to the $k$NN-LM is that the datastore stores high-dimensional dense vectors for each token in the training data; this can easily scale to hundreds of millions or even billions of records.&#10;As a result, the extra retrieval step from such datastores greatly decreases model efficiency at test time.&#10;For example, a 100M-entry datastore can lead to an over 10x slow-down &#10;compared to parametric models (\textsection\ref{sec:speed-baseline}) as shown in Figure~\ref{fig:intro_scatter}. This issue poses a serious hurdle for the practical deployment of non-parametric LMs, despite their effectiveness. &#10;&#10;In this paper, we attempt to address this issue of test-time inefficiency and make non-parametric LMs more applicable in real-world settings.&#10;We take $k$NN-LM as an example, first analyzing the evaluation overhead,&#10; and raise three questions that we aim to answer in this paper: (1) Do we really need to perform retrieval on the prediction of every single token? (2) Can we identify and prune redundant records from the datastore? (3) Is it possible to further compress the datastore by reducing the vector dimensionality without losing performance? We propose and explore potential solutions for each question to aid efficiency. Specifically, we (1) show that a lightweight network can be learned to automatically prune unnecessary retrieval operations (adaptive retrieval, \textsection\ref{sec:ar}), (2) explore several different methods for datastore pruning based on clustering, importance-guided filtering, or greedy merging (\textsection\ref{sec:dp}), and (3) empirically demonstrate that simple dimension reduction techniques are able to improve both the performance and speed (\textsection\ref{sec:dr}). &#10; Figure~\ref{fig:intro_scatter} illustrate the overall performance of these methods. Our experiments on the WikiText-103 language modeling benchmark~\cite{merity2016pointer} and a training-free domain-adaptation setting demonstrate speed improvements of up to 6x with comparable perplexity to the $k$NN-LM.&#10;On a higher level, we expect the empirical results and analysis in the paper to help researchers better understand the speed-performance tradeoff in non-parametric NLMs, and provide a springboard for future research on more efficient non-parametric LMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Non-Parametric Methods, Linguistics, Efficient Inference Techniques, Neural Language Models, Mathematics, Information Science" />
        </attvalues>
      </node>
      <node id="2010.00710" label="2010.00710">
        <attvalues>
          <attvalue for="0" value="Nearest Neighbor Machine Translation" />
          <attvalue for="1" value="  We introduce $k$-nearest-neighbor machine translation ($k$NN-MT), which&#10;predicts tokens with a nearest neighbor classifier over a large datastore of&#10;cached examples, using representations from a neural translation model for&#10;similarity search. This approach requires no additional training and scales to&#10;give the decoder direct access to billions of examples at test time, resulting&#10;in a highly expressive model that consistently improves performance across many&#10;settings. Simply adding nearest neighbor search improves a state-of-the-art&#10;German-English translation model by 1.5 BLEU. $k$NN-MT allows a single model to&#10;be adapted to diverse domains by using a domain-specific datastore, improving&#10;results by an average of 9.2 BLEU over zero-shot transfer, and achieving new&#10;state-of-the-art results -- without training on these domains. A massively&#10;multilingual model can also be specialized for particular language pairs, with&#10;improvements of 3 BLEU for translating from English into German and Chinese.&#10;Qualitatively, $k$NN-MT is easily interpretable; it combines source and target&#10;context to retrieve highly relevant examples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.17431" label="2408.17431">
        <attvalues>
          <attvalue for="0" value="Advancing Multi-talker ASR Performance with Large Language Models" />
          <attvalue for="1" value="  Recognizing overlapping speech from multiple speakers in conversational&#10;scenarios is one of the most challenging problem for automatic speech&#10;recognition (ASR). Serialized output training (SOT) is a classic method to&#10;address multi-talker ASR, with the idea of concatenating transcriptions from&#10;multiple speakers according to the emission times of their speech for training.&#10;However, SOT-style transcriptions, derived from concatenating multiple related&#10;utterances in a conversation, depend significantly on modeling long contexts.&#10;Therefore, compared to traditional methods that primarily emphasize encoder&#10;performance in attention-based encoder-decoder (AED) architectures, a novel&#10;approach utilizing large language models (LLMs) that leverages the capabilities&#10;of pre-trained decoders may be better suited for such complex and challenging&#10;scenarios. In this paper, we propose an LLM-based SOT approach for multi-talker&#10;ASR, leveraging pre-trained speech encoder and LLM, fine-tuning them on&#10;multi-talker dataset using appropriate strategies. Experimental results&#10;demonstrate that our approach surpasses traditional AED-based methods on the&#10;simulated dataset LibriMix and achieves state-of-the-art performance on the&#10;evaluation set of the real-world dataset AMI, outperforming the AED model&#10;trained with 1000 times more supervised data in previous works.&#10;" />
          <attvalue for="2" value="&#10;Although automatic speech recognition (ASR)~\cite{li2022recent,GulatiQCPZYHWZW20,yao2023zipformer} has achieved excellent performance in quiet, single-speaker scenarios, it still faces significant challenges in multi-talker conversational scenarios, especially in the case of overlapping speech. &#10;To overcome this challenge, a series of multi-talker ASR approaches have been proposed~\cite{chen2017progressive,yu2017recognizing,chang2019mimo,ZhangCQW20,kanda2020serialized}. One of the most representative methods is serialized output training (SOT)~\cite{kanda2020serialized,KandaYGWMCY21,ShiD0YLZ0023}. The core idea of SOT is to concatenate the transcriptions of multiple speakers in the order of their speech emission times, separated by a speaker change symbol. Compared to permutation invariant training (PIT)~\cite{yu2017recognizing,chang2019mimo,ZhangCQW20}, SOT avoids the limitation on the maximum number of speakers, models the dependencies in multi-talker content, and reduces computational complexity, resulting in better performance on multi-talker ASR task.&#10;&#10;{However, in SOT-style transcriptions, the concatenation of related content from multiple speakers, coupled with the relatively poor grammatical structure of sentences in meeting discussions, necessitates strong long-context awareness and cross utterance modeling. This is precisely what previous SOT methods based on attention-based encoder-decoder (AED)~\cite{kanda2020serialized}, which relied more on encoder performance, lacked, leading to performance bottlenecks.}&#10;For instance, in~\cite{KandaYWGWMCY21}, despite using 900K hours of large-scale simulated data for pre-training, the word error rate on the AMI~\cite{CarlettaABFGHKKKKLLLMPRW05} meeting corpus still reached 21.2\%.&#10;&#10;Large language models (LLMs)~\cite{abs-2211-05100,abs-2302-13971,abs-2307-09288,chiang2023vicuna}, trained on vast amounts of text data, possess unparalleled capabilities in understanding and generating natural language. Their proficiency in long-context awareness makes them exceptionally well-suited for SOT-style transcriptions. Therefore, the combination of LLM and SOT-based multi-talker ASR is a perfect match.&#10;{A series of LLM-based ASR works~\cite{WangHSWCCCZSRZYPSSW23,abs-2307-11795,abs-2310-13289,abs-2311-07919,abs-2402-08846,abs-2405-02132} have been conducted, which, in contrast to traditional AED methods that focus on encoder performance, tend to treat the speech foundation encoder~\cite{BaevskiZMA20,HsuBTLSM21,ChenWCWLCLKYXWZ22,RadfordKXBMS23} in LLM-based models as a tool for extracting embedding. The speech embedding then serve as prompt for the LLM, relying on the powerful decoder-only LLM to generate transcription.}&#10;These studies have shown that this approach can match or slightly outperform traditional AED methods in simple single-speaker ASR tasks~\cite{abs-2307-11795,abs-2402-08846}. However, in these works, the performance advantage of the LLM-based methods is not particularly pronounced, indicating that LLM-based models, {with their powerful decoders,} have not fully realized their potential in handling speech tasks in simple scenarios.&#10;&#10;Therefore, in this paper, motivated by the potential of powerful LLMs to handle challenging speech tasks in complex scenarios and the natural compatibility of LLMs with SOT, we propose an LLM-based approach for multi-talker ASR. Similar to previous LLM-based ASR works, we employ a architecture comprising a pre-trained speech encoder, a projector, and an LLM. In previous works, various training strategies have been employed. For example, in~\cite{abs-2307-11795}, low-rank adaptation (LoRA)~\cite{HuSWALWWC22} was introduced into the LLM to facilitate efficient fine-tuning, and all three components were fine-tuned together in a single stage. In~\cite{abs-2402-08846}, LoRA was not introduced, and the encoder was frozen while training only the projector, which also yielded satisfactory results. In~\cite{abs-2405-02132}, a multi-stage fine-tuning approach was used to better align the modalities of speech and text. In this paper, we compared the aforementioned training strategies on the simulated LibriMix dataset and synthesized the best practices to propose the most suitable strategy, which made our LLM-based method surpass the AED-based approach. On the evaluation set of the real-world meeting corpus AMI, the proposed LLM-based method not only surpasses AED-based methods trained with the same amount of data but also remarkably outperforms the AED model trained on an enormous scale of 900K hours (1000 times more) of supervised data, achieving state-of-the-art. This astounding result demonstrates the immense potential of LLM-based models in handling speech processing tasks in challenging scenarios.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Engineering, Automatic Speech Recognition, Signal Processing, Multi-Talker Scenarios" />
        </attvalues>
      </node>
      <node id="2111.01690" label="2111.01690">
        <attvalues>
          <attvalue for="0" value="Recent Advances in End-to-End Automatic Speech Recognition" />
          <attvalue for="1" value="  Recently, the speech community is seeing a significant trend of moving from&#10;deep neural network based hybrid modeling to end-to-end (E2E) modeling for&#10;automatic speech recognition (ASR). While E2E models achieve the&#10;state-of-the-art results in most benchmarks in terms of ASR accuracy, hybrid&#10;models are still used in a large proportion of commercial ASR systems at the&#10;current time. There are lots of practical factors that affect the production&#10;model deployment decision. Traditional hybrid models, being optimized for&#10;production for decades, are usually good at these factors. Without providing&#10;excellent solutions to all these factors, it is hard for E2E models to be&#10;widely commercialized. In this paper, we will overview the recent advances in&#10;E2E models, focusing on technologies addressing those challenges from the&#10;industry's perspective.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.08100" label="2005.08100">
        <attvalues>
          <attvalue for="0" value="Conformer: Convolution-augmented Transformer for Speech Recognition" />
          <attvalue for="1" value="  Recently Transformer and Convolution neural network (CNN) based models have&#10;shown promising results in Automatic Speech Recognition (ASR), outperforming&#10;Recurrent neural networks (RNNs). Transformer models are good at capturing&#10;content-based global interactions, while CNNs exploit local features&#10;effectively. In this work, we achieve the best of both worlds by studying how&#10;to combine convolution neural networks and transformers to model both local and&#10;global dependencies of an audio sequence in a parameter-efficient way. To this&#10;regard, we propose the convolution-augmented transformer for speech&#10;recognition, named Conformer. Conformer significantly outperforms the previous&#10;Transformer and CNN based models achieving state-of-the-art accuracies. On the&#10;widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without&#10;using a language model and 1.9%/3.9% with an external language model on&#10;test/testother. We also observe competitive performance of 2.7%/6.3% with a&#10;small model of only 10M parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.11230" label="2310.11230">
        <attvalues>
          <attvalue for="0" value="Zipformer: A faster and better encoder for automatic speech recognition" />
          <attvalue for="1" value="  The Conformer has become the most popular encoder model for automatic speech&#10;recognition (ASR). It adds convolution modules to a transformer to learn both&#10;local and global dependencies. In this work we describe a faster, more&#10;memory-efficient, and better-performing transformer, called Zipformer. Modeling&#10;changes include: 1) a U-Net-like encoder structure where middle stacks operate&#10;at lower frame rates; 2) reorganized block structure with more modules, within&#10;which we re-use attention weights for efficiency; 3) a modified form of&#10;LayerNorm called BiasNorm allows us to retain some length information; 4) new&#10;activation functions SwooshR and SwooshL work better than Swish. We also&#10;propose a new optimizer, called ScaledAdam, which scales the update by each&#10;tensor's current scale to keep the relative change about the same, and also&#10;explictly learns the parameter scale. It achieves faster convergence and better&#10;performance than Adam. Extensive experiments on LibriSpeech, Aishell-1, and&#10;WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer&#10;over other state-of-the-art ASR models. Our code is publicly available at&#10;https://github.com/k2-fsa/icefall.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.07048" label="1707.07048">
        <attvalues>
          <attvalue for="0" value="Progressive Joint Modeling in Unsupervised Single-channel Overlapped&#10;  Speech Recognition" />
          <attvalue for="1" value="  Unsupervised single-channel overlapped speech recognition is one of the&#10;hardest problems in automatic speech recognition (ASR). Permutation invariant&#10;training (PIT) is a state of the art model-based approach, which applies a&#10;single neural network to solve this single-input, multiple-output modeling&#10;problem. We propose to advance the current state of the art by imposing a&#10;modular structure on the neural network, applying a progressive pretraining&#10;regimen, and improving the objective function with transfer learning and a&#10;discriminative training criterion. The modular structure splits the problem&#10;into three sub-tasks: frame-wise interpreting, utterance-level speaker tracing,&#10;and speech recognition. The pretraining regimen uses these modules to solve&#10;progressively harder tasks. Transfer learning leverages parallel clean speech&#10;to improve the training targets for the network. Our discriminative training&#10;formulation is a modification of standard formulations, that also penalizes&#10;competing outputs of the system. Experiments are conducted on the artificial&#10;overlapped Switchboard and hub5e-swb dataset. The proposed framework achieves&#10;over 30% relative improvement of WER over both a strong jointly trained system,&#10;PIT for ASR, and a separately optimized system, PIT for speech separation with&#10;clean speech ASR model. The improvement comes from better model generalization,&#10;training efficiency and the sequence level linguistic knowledge integration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.01985" label="1704.01985">
        <attvalues>
          <attvalue for="0" value="Recognizing Multi-talker Speech with Permutation Invariant Training" />
          <attvalue for="1" value="  In this paper, we propose a novel technique for direct recognition of&#10;multiple speech streams given the single channel of mixed speech, without first&#10;separating them. Our technique is based on permutation invariant training (PIT)&#10;for automatic speech recognition (ASR). In PIT-ASR, we compute the average&#10;cross entropy (CE) over all frames in the whole utterance for each possible&#10;output-target assignment, pick the one with the minimum CE, and optimize for&#10;that assignment. PIT-ASR forces all the frames of the same speaker to be&#10;aligned with the same output layer. This strategy elegantly solves the label&#10;permutation problem and speaker tracing problem in one shot. Our experiments on&#10;artificially mixed AMI data showed that the proposed approach is very&#10;promising.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.06522" label="1910.06522">
        <attvalues>
          <attvalue for="0" value="MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition" />
          <attvalue for="1" value="  Recently, the end-to-end approach has proven its efficacy in monaural&#10;multi-speaker speech recognition. However, high word error rates (WERs) still&#10;prevent these systems from being used in practical applications. On the other&#10;hand, the spatial information in multi-channel signals has proven helpful in&#10;far-field speech recognition tasks. In this work, we propose a novel neural&#10;sequence-to-sequence (seq2seq) architecture, MIMO-Speech, which extends the&#10;original seq2seq to deal with multi-channel input and multi-channel output so&#10;that it can fully model multi-channel multi-speaker speech separation and&#10;recognition. MIMO-Speech is a fully neural end-to-end framework, which is&#10;optimized only via an ASR criterion. It is comprised of: 1) a monaural masking&#10;network, 2) a multi-source neural beamformer, and 3) a multi-output speech&#10;recognition model. With this processing, the input overlapped speech is&#10;directly mapped to text sequences. We further adopted a curriculum learning&#10;strategy, making the best use of the training set to improve the performance.&#10;The experiments on the spatialized wsj1-2mix corpus show that our model can&#10;achieve more than 60% WER reduction compared to the single-channel system with&#10;high quality enhanced signals (SI-SDR = 23.1 dB) obtained by the above&#10;separation function.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12459" label="2305.12459">
        <attvalues>
          <attvalue for="0" value="CASA-ASR: Context-Aware Speaker-Attributed ASR" />
          <attvalue for="1" value="  Recently, speaker-attributed automatic speech recognition (SA-ASR) has&#10;attracted a wide attention, which aims at answering the question ``who spoke&#10;what''. Different from modular systems, end-to-end (E2E) SA-ASR minimizes the&#10;speaker-dependent recognition errors directly and shows a promising&#10;applicability. In this paper, we propose a context-aware SA-ASR (CASA-ASR)&#10;model by enhancing the contextual modeling ability of E2E SA-ASR. Specifically,&#10;in CASA-ASR, a contextual text encoder is involved to aggregate the semantic&#10;information of the whole utterance, and a context-dependent scorer is employed&#10;to model the speaker discriminability by contrasting with speakers in the&#10;context. In addition, a two-pass decoding strategy is further proposed to fully&#10;leverage the contextual modeling ability resulting in a better recognition&#10;performance. Experimental results on AliMeeting corpus show that the proposed&#10;CASA-ASR model outperforms the original E2E SA-ASR system with a relative&#10;improvement of 11.76% in terms of speaker-dependent character error rate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.16776" label="2103.16776">
        <attvalues>
          <attvalue for="0" value="Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting&#10;  Transcription with Single Distant Microphone" />
          <attvalue for="1" value="  Transcribing meetings containing overlapped speech with only a single distant&#10;microphone (SDM) has been one of the most challenging problems for automatic&#10;speech recognition (ASR). While various approaches have been proposed, all&#10;previous studies on the monaural overlapped speech recognition problem were&#10;based on either simulation data or small-scale real data. In this paper, we&#10;extensively investigate a two-step approach where we first pre-train a&#10;serialized output training (SOT)-based multi-talker ASR by using large-scale&#10;simulation data and then fine-tune the model with a small amount of real&#10;meeting data. Experiments are conducted by utilizing 75 thousand (K) hours of&#10;our internal single-talker recording to simulate a total of 900K hours of&#10;multi-talker audio segments for supervised pre-training. With fine-tuning on&#10;the 70 hours of the AMI-SDM training data, our SOT ASR model achieves a word&#10;error rate (WER) of 21.2% for the AMI-SDM evaluation set while automatically&#10;counting speakers in each test segment. This result is not only significantly&#10;better than the previous state-of-the-art WER of 36.4% with oracle utterance&#10;boundary information but also better than a result by a similarly fine-tuned&#10;single-talker ASR model applied to beamformed audio.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.00230" label="2310.00230">
        <attvalues>
          <attvalue for="0" value="SLM: Bridge the thin gap between speech and text foundation models" />
          <attvalue for="1" value="  We present a joint Speech and Language Model (SLM), a multitask,&#10;multilingual, and dual-modal model that takes advantage of pretrained&#10;foundational speech and language models. SLM freezes the pretrained foundation&#10;models to maximally preserves their capabilities, and only trains a simple&#10;adapter with just 1\% (156M) of the foundation models' parameters. This&#10;adaptation not only leads SLM to achieve strong performance on conventional&#10;tasks such as speech recognition (ASR) and speech translation (AST), but also&#10;introduces the novel capability of zero-shot instruction-following for more&#10;diverse tasks: given a speech input and a text instruction, SLM is able to&#10;perform unseen generation tasks including contextual biasing ASR using&#10;real-time context, dialog generation, speech continuation, and question&#10;answering, etc. Our approach demonstrates that the representational gap between&#10;pretrained speech and language models might be narrower than one would expect,&#10;and can be bridged by a simple adaptation mechanism. As a result, SLM is not&#10;only efficient to train, but also inherits strong capabilities already acquired&#10;in foundation models of different modalities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.08846" label="2402.08846">
        <attvalues>
          <attvalue for="0" value="An Embarrassingly Simple Approach for LLM with Strong ASR Capacity" />
          <attvalue for="1" value="  In this paper, we focus on solving one of the most important tasks in the&#10;field of speech processing, i.e., automatic speech recognition (ASR), with&#10;speech foundation encoders and large language models (LLM). Recent works have&#10;complex designs such as compressing the output temporally for the speech&#10;encoder, tackling modal alignment for the projector, and utilizing&#10;parameter-efficient fine-tuning for the LLM. We found that delicate designs are&#10;not necessary, while an embarrassingly simple composition of off-the-shelf&#10;speech encoder, LLM, and the only trainable linear projector is competent for&#10;the ASR task. To be more specific, we benchmark and explore various&#10;combinations of LLMs and speech encoders, leading to the optimal LLM-based ASR&#10;system, which we call SLAM-ASR. The proposed SLAM-ASR provides a clean setup&#10;and little task-specific design, where only the linear projector is trained. To&#10;the best of our knowledge, SLAM-ASR achieves the best performance on the&#10;Librispeech benchmark among LLM-based ASR models and even outperforms the&#10;latest LLM-based audio-universal model trained on massive pair data. Finally,&#10;we explore the capability emergence of LLM-based ASR in the process of modal&#10;alignment. We hope that our study can facilitate the research on extending LLM&#10;with cross-modality capacity and shed light on the LLM-based ASR community.&#10;" />
          <attvalue for="2" value="&#10;Automatic speech recognition (ASR) stands as a cornerstone in the realm of intelligent speech technology, enabling machines to understand and transcribe human speech. The significance of ASR in enhancing human-computer interaction and accessibility makes it a crucial area of research and applications in the field of speech processing. &#10;&#10;The evolution of ASR technology has been marked by the adoption of various paradigms, each representing a leap forward in terms of accuracy, efficiency, and applicability~\cite{e2ereview}.&#10;Among these, supervised methods including connectionist temporal classification (CTC)~\cite{CTC}, attention-based encoder-decoder (AED)~\cite{AED}, recurrent neural network transducer (RNN-T)~\cite{RNN-T} and their variants have been pivotal. &#10;In addition, employing self-supervised methods for pre-training followed by supervised methods for fine-tuning has also proven to be effective~\cite{baevski2020wav2vec, hsu2021hubert, chen2022wavlm, ma2022mt4ssl, yang2023fast}. &#10;However, each paradigm comes with its own set of challenges and limitations, such as the need for extensive labeled data, difficulties in capturing long-range context dependencies in speech, and huge training costs. &#10;&#10;In this evolving landscape, the advent of large language models (LLMs) has introduced a groundbreaking paradigm: Multimodal large language models (MLLMs) framework~\cite{liu2023visual, BLIP2, gao2023llama}, based on a decoder-only architecture. This innovative approach diverges from traditional ASR by utilizing the immense generative capacity of LLMs, which are pre-trained on vast corpora encompassing diverse linguistic contexts, leading to LLM-based ASR. &#10;The evolution of the ASR paradigm from previous NN-based ASR models to LLM-based ASR models, stresses differences across loss and criterion design, text prior knowledge, and model scale. &#10;This paradigm harnesses pre-existing linguistic knowledge, enabling a more holistic understanding of language, which in turn, translates to significant improvements in the speech recognition task. &#10;&#10;The architecture of LLM-based ASR can be conceptualized as consisting of three primary components: a speech encoder, a projector, and an LLM. &#10;Recent works in LLM-based ASR often venture into complex designs, such as compressing the output temporally from the speech encoder~\cite{wu2023decoder, fathullah2023prompting}, tackling modal alignment with the projector~\cite{tang2023salmonn, yu2023connecting}, and fine-tuning the LLM partly or fully~\cite{wu2023decoder, li2023prompting, tang2023salmonn, wang2023lauragpt}. &#10;Despite these efforts, the outcomes have not always met expectations, indicating a potential misalignment between the complexity of designs and the efficacy of real-world speech recognition tasks. This observation led to a pivotal realization in our research: the essence of an effective LLM-based ASR system lies in the synergy of a powerful speech encoder and a suitable LLM, and then, most notably, a single trainable linear projector is enough to align between modalities. Our findings challenge the prevailing notion that complexity equates to superiority in LLM-based ASR system design. &#10;&#10;In this work, we first benchmark the automatic speech recognition task performance with different combinations of well-known speech encoders and the latest released large language models. &#10;Experiments show that LLMs with supervised fine-tuning (SFT, a.k.a. chat model) perform better than raw pre-trained LLMs for the ASR task, while speech encoders fine-tuned with limited data from self-supervised models outperform supervised foundation ASR encoders.&#10;Building upon these insights, we propose SLAM-ASR, in which only a linear projector is trained to conduct the ASR task. SLAM-ASR only requires $4$ GPUs for $4$ hours of training to achieve state-of-the-art performance on the Librispeech~\cite{panayotov2015librispeech} corpus, compared with other LLM-based ASR models and a series of previous best performing NN-based ASR models. &#10;Besides, our work embarks on an in-depth exploration of the ability of LLM-based ASR models. Interestingly,&#10; we observe the capability emergence phenomenon during LLM-based ASR training. &#10;The benchmark and experimental exploration show how we harvest the exciting result step by step with a clean setup and little task-specific design. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.02132" label="2405.02132">
        <attvalues>
          <attvalue for="0" value="Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated unparalleled effectiveness in&#10;various NLP tasks, and integrating LLMs with automatic speech recognition (ASR)&#10;is becoming a mainstream paradigm. Building upon this momentum, our research&#10;delves into an in-depth examination of this paradigm on a large open-source&#10;Chinese dataset. Specifically, our research aims to evaluate the impact of&#10;various configurations of speech encoders, LLMs, and projector modules in the&#10;context of the speech foundation encoder-LLM ASR paradigm. Furthermore, we&#10;introduce a three-stage training approach, expressly developed to enhance the&#10;model's ability to align auditory and textual information. The implementation&#10;of this approach, alongside the strategic integration of ASR components,&#10;enabled us to achieve the SOTA performance on the AISHELL-1, Test_Net, and&#10;Test_Meeting test sets. Our analysis presents an empirical foundation for&#10;future research in LLM-based ASR systems and offers insights into optimizing&#10;performance using Chinese datasets. We will publicly release all scripts used&#10;for data preparation, training, inference, and scoring, as well as pre-trained&#10;models and training logs to promote reproducible research.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs)~\cite{llama,bert} have emerged as a formidable force in artificial intelligence, showcasing unparalleled proficiency in understanding and generating human language. Drawing from this strength, researchers have begun to merge the prowess of LLMs with various fields, including automatic speech recognition (ASR), where their integration has led to notable performance improvement~\cite{achiam2023gpt,team2023gemini,speechgpt}. Specifically, ASR, a task that intricately intertwines acoustic modeling with language modeling, has conventionally employed language models like n-grams~\cite{ngram-1,ngram-2,shallow-fusion} or neural network language models (NNLMs)~\cite{nnlm-1,nnlm-2,deep-fusion,component-fusion}. However, the advent of LLMs offers a compelling alternative to the language component of ASR, drawing from their superior ability to understand and predict linguistic patterns by scaling up data and parameters.&#10;&#10;Research efforts to integrate LLMs with ASR systems generally fall into two categories. The first strategy involves connecting LLMs with pre-trained ASR models, wherein the ASR-generated text is directly fed to the LLM to serve as a prompt for downstream tasks~\cite{linkpara1_audio_gpt,linkpara-2,linkpara-4} or to facilitate error correction~\cite{linkpara-5}. However, this coarse-grained integration may result in a substantial loss of acoustic information. It may suffer compounding errors from the initial ASR stage, potentially leading to an exacerbation of inaccuracies. In contrast, the second approach adopts audio-text cross-modal LLMs, which embrace the auditory modality by employing an encoder network to process the speech and generate embeddings that are subsequently provided to a decoder-only LLM~\cite{speechgpt,decoder-only-asr,listen_think_anderstand,salmonn,qwen-audio, simple-asr-llm}. This framework strives for a tighter coupling between acoustic cues and linguistic context, aiming to yield a better interpretation of speech. &#10;Through a series of studies, the paradigm of augmenting a speech foundation model with an LLM through projector modules has emerged as the prevailing framework in the current LLM-based speech recognition research. Specifically, SALMONN~\cite{salmonn} applies Whisper~\cite{whisper} to extract semantic content and BEATs~\cite{beats} for audio event information, culminating in a robust perception of human speech, music, and audio events. Qwen-Audio~\cite{qwen-audio} implements Whisper as the exclusive encoder, utilizing structured task directives to enhance the model's performance across various audio tasks. SLAM-ASR~\cite{simple-asr-llm} leverages a linear layer as the projector module and achieves SOTA performance on the English 960-hour LibriSpeech~\cite{libispeech} task.&#10;&#10;Following the inspiring results of these studies, we aim to investigate further the potential of the speech foundation encoder plus LLM decoder paradigm on a large-scale open-source Chinese dataset. Specifically, with over 11,000 hours of Chinese speech data from various corpora, we examine the impact of different projectors, speech encoders, and LLMs on Mandarin ASR performance within this paradigm. Concurrently, we utilize a three-stage training approach designed to enhance the learning of the alignment between auditory and textual modalities. From experiments, we draw the following major conclusions: (1) For the speech encoder, Whisper~\cite{whisper} is more robust but have lower plasticity compared to HuBERT~\cite{hubert}. (2) For the projector, the Transformer’s learning ability is better than the Qformer~\cite{blip} in the speech recognition task. (3) For the LLM, the performance of the LLM-integrated ASR system is positively correlated to the LLM's proficiency in that specific language -- Mandarin here. (4) Our three-stage training approach can effectively align the pre-trained acoustic modeling capability of the speech foundation model with the language modeling capability of LLMs, using a relatively smaller Chinese dataset and achieve SOTA performance on the AISHELL-1, Test\_Net, and Test\_Meeting test datasets.&#10;&#10;We will provide reproducible recipes encompassing the entire pipeline, including data preparation, training, inference, and scoring. Furthermore, we will release pre-trained models, enabling researchers to delve into the specifics of the training process and gain valuable insights for their own investigations.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.14906" label="2405.14906">
        <attvalues>
          <attvalue for="0" value="AutoCoder: Enhancing Code Large Language Model with&#10;  \textsc{AIEV-Instruct}" />
          <attvalue for="1" value="  We introduce AutoCoder, the first Large Language Model to surpass GPT-4 Turbo&#10;(April 2024) and GPT-4o in pass@1 on the Human Eval benchmark test&#10;($\mathbf{90.9\%}$ vs. $\mathbf{90.2\%}$). In addition, AutoCoder offers a more&#10;versatile code interpreter compared to GPT-4 Turbo and GPT-4o. It's code&#10;interpreter can install external packages instead of limiting to built-in&#10;packages. AutoCoder's training data is a multi-turn dialogue dataset created by&#10;a system combining agent interaction and external code execution verification,&#10;a method we term \textbf{\textsc{AIEV-Instruct}} (Instruction Tuning with&#10;Agent-Interaction and Execution-Verified). Compared to previous large-scale&#10;code dataset generation methods, \textsc{AIEV-Instruct} reduces dependence on&#10;proprietary large models and provides execution-validated code dataset. The&#10;code and the demo video is available in&#10;\url{https://github.com/bin123apple/AutoCoder}.&#10;" />
          <attvalue for="2" value="&#10;Code generation is a critical aspect of modern software development. It significantly enhances development efficiency and quality by increasing productivity, reducing errors, standardizing code, accelerating prototyping, and supporting complex systems~\cite{li2024deveval, li2023large,buscemi2023comparative}.Recently, Large Language Models (LLMs), such as GPT-4~\cite{chatgpt2024} and CodeQwen1.5~\cite{codeqwen2024}, have achieved significant advancements on code generation. These models have shown high accuracy in producing code that meets user requirements and have been widely adopted in real-world software development. &#10;&#10;Training large language models requires extensive high-quality data~\cite{hoffmann2022training}. This is particularly crucial for code generation tasks that demand high accuracy~\cite{chen2021evaluating}. OpenAI once hired people to help annotate the Code Instruct dataset for training their InstructGPT~\cite{ouyang2022training}. However, manually annotating large-scale code instruction datasets is both economically and time-consuming~\cite{xu2022ide}. To address this challenge, previous work has employed various automated code annotation methods, such as Self-Instruct~\cite{wang2022self}, Evol-Instruct~\cite{luo2023wizardcoder}, and OSS-Instruct~\cite{wei2023magicoder}. Self-Instruct enhances LLMs' instruction-following capabilities by using strong teacher models to generate synthetic coding instructions for fine-tuning weaker student models. Evol-Instruct improves LLMs' coding abilities by iteratively increasing the complexity of seed code instructions through various heuristics. OSS-Instruct generates diverse and realistic coding problems by drawing inspiration from open-source code snippets. The essence of these methods lies in distilling the knowledge of a powerful teacher model (such as GPT-4 Turbo) to guide a smaller model. This leads to a problem: While the small model can achieve significant performance improvements, the final accuracy of the small model is unlikely to surpass that of the teacher model. Because both the correct and incorrect knowledge from the teacher model are transferred to the small model. Moreover, although using closed-source models reduces costs compared to manual annotation, the cost of using closed-source models remains high. According to our tests, even with the relatively cheaper GPT-4 Turbo model, generating an average of 6,500 high-quality entries for the code instruction dataset costs \$1,000.&#10;&#10;This raises two questions:&#10;\begin{enumerate}&#10;[label=\arabic*.]&#10; \item Can we correct the incorrect knowledge generated by the teacher model to provide more accurate code for the student model?&#10; \item Instead of relying on expensive closed-source teacher models, can we enable our student model to learn autonomously?&#10;\end{enumerate}&#10;&#10;To address the first issue, we designed a new large-scale code instruction dataset annotation method called AIEV-Instruct. It is an interaction system comprising two agents: a questioner and a programmer. These agents interact to simulate the process of programmers constructing code according to project requirements and conducting unit tests. In each dialogue round, we extract the code generated by the programmers and execute it. The execution results are returned to the questioner to inform the next round of questions. This process continues until the programmers's code passes the unit tests, ensuring the accuracy of the generated code dataset.&#10;&#10;To address the second issue, we sperate AIEV-Instruct into two stages: the Teaching Stage and the Self-learning Stage. In the Teaching Stage, we rely on proprietary large models as agents for code annotation, similar to previous methods. Once our model surpasses the proprietary models in accuracy on the test set, we transition to the Self-learning Stage. In this stage, we use our own model as the agent for code annotation. For detailed methodology, refer to Section~\ref{sec:AIEV-Instruct}.&#10;&#10;Under the support of AIEV-Instruct, we obtained 169K high-quality code instruction data samples. Using this dataset, we trained the AutoCoder series models, including AutoCoder (33B) and AutoCoder-S (6.7B). As shown in Figure~\ref{fig:Overall_performance}, AutoCoder demonstrates higher accuracy. In the HumanEval Base Test, we compared it with the top ten models on the current (May 2024) EvalPlus Leaderboard~\cite{evalplus2024}. The results indicate that AutoCoder's Pass@1 even surpasses that of the current top-ranked models, GPT-4 Turbo (April 2024) and GPT-4o. &#10;&#10;Moreover, as illustrated in Figure~\ref{fig:Code_Interpreter}, AutoCoder boasts a more versatile Code Interpreter function compared to GPT-4o and GPT-4 Turbo. The Code Interpreter is an external program execution environment that large models utilize to execute the code they deem necessary. While GPT-4o and GPT-4 Turbo can identify the code that needs to be executed, they fail to provide the Code Interpreter with the necessary instructions to install external packages required by the programs. This limitation significantly restricts the capabilities of the Code Interpreter. In contrast, AutoCoder can correctly supply the Code Interpreter with the appropriate external package installation instructions, thereby enabling it to execute a wide variety of code.&#10;&#10;To comprehensively evaluate the capabilities of AutoCoder, we tested it on several datasets: HumanEval~\cite{chen2021evaluating}, HumanEval+~\cite{liu2024your}, MBPP~\cite{austin2021program}, MBPP+~\cite{liu2024your}, MultiPL-E~\cite{cassano2022multipl}, and DS-1000~\cite{lai2023ds}. To measure the performance improvement of AutoCoder, we compared it to its base model, Deepseek-Coder~\cite{guo2024deepseek}. The results demonstrate that AutoCoder exhibits outstanding performance. As of May 2024, AutoCoder ranks 1st among all LLMs on the HumanEval Base Test, 5th on the HumanEval Plus Test, and 4th on both the MBPP Base Test and the MBPP Plus Test. Detailed experimental procedures can be found in Section~\ref{sec:Experiment}.&#10;&#10;Overall, our contributions are summarized as follows:&#10;&#10;We propose AIEV-Instruct, a novel method for creating high-quality large code datasets. It simulates programmers writing code and conducting unit tests through agent interactions, ensuring annotation accuracy with an external code executor. It includes a Teaching Stage and a Self-Learning Stage, reducing reliance on expensive closed-source models during the annotation process.&#10;&#10;We introduce AutoCoder, a code LLM trained using AIEV-Instruct that excels in code-related tasks. It outperforms top models like GPT-4 Turbo and GPT-4o on the HumanEval benchmark.&#10;&#10;We enhances the functionality of the current code interpreters. AutoCoder can provide the code interpreter with the necessary instructions to install external packages, extending the applicability of the code interpreter beyond built-in packages. &#10; \input{Sections/2_Related_Work}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Dataset Generation, Artificial Intelligence, Code Interpretation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2308.04477" label="2308.04477">
        <attvalues>
          <attvalue for="0" value="A Comparative Study of Code Generation using ChatGPT 3.5 across 10&#10;  Programming Languages" />
          <attvalue for="1" value="  Large Language Models (LLMs) are advanced Artificial Intelligence (AI)&#10;systems that have undergone extensive training using large datasets in order to&#10;understand and produce language that closely resembles that of humans. These&#10;models have reached a level of proficiency where they are capable of&#10;successfully completing university exams across several disciplines and&#10;generating functional code to handle novel problems. This research investigates&#10;the coding proficiency of ChatGPT 3.5, a LLM released by OpenAI in November&#10;2022, which has gained significant recognition for its impressive text&#10;generating and code creation capabilities. The skill of the model in creating&#10;code snippets is evaluated across 10 various programming languages and 4&#10;different software domains. Based on the findings derived from this research,&#10;major unexpected behaviors and limitations of the model have been identified.&#10;This study aims to identify potential areas for development and examine the&#10;ramifications of automated code generation on the evolution of programming&#10;languages and on the tech industry.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10560" label="2212.10560">
        <attvalues>
          <attvalue for="0" value="Self-Instruct: Aligning Language Models with Self-Generated Instructions" />
          <attvalue for="1" value="  Large &quot;instruction-tuned&quot; language models (i.e., finetuned to respond to&#10;instructions) have demonstrated a remarkable ability to generalize zero-shot to&#10;new tasks. Nevertheless, they depend heavily on human-written instruction data&#10;that is often limited in quantity, diversity, and creativity, therefore&#10;hindering the generality of the tuned model. We introduce Self-Instruct, a&#10;framework for improving the instruction-following capabilities of pretrained&#10;language models by bootstrapping off their own generations. Our pipeline&#10;generates instructions, input, and output samples from a language model, then&#10;filters invalid or similar ones before using them to finetune the original&#10;model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute&#10;improvement over the original model on Super-NaturalInstructions, on par with&#10;the performance of InstructGPT-001, which was trained with private user data&#10;and human annotations. For further evaluation, we curate a set of&#10;expert-written instructions for novel tasks, and show through human evaluation&#10;that tuning GPT3 with Self-Instruct outperforms using existing public&#10;instruction datasets by a large margin, leaving only a 5% absolute gap behind&#10;InstructGPT-001. Self-Instruct provides an almost annotation-free method for&#10;aligning pre-trained language models with instructions, and we release our&#10;large synthetic dataset to facilitate future studies on instruction tuning. Our&#10;code and data are available at https://github.com/yizhongw/self-instruct.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.08568" label="2306.08568">
        <attvalues>
          <attvalue for="0" value="WizardCoder: Empowering Code Large Language Models with Evol-Instruct" />
          <attvalue for="1" value="  Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated&#10;exceptional performance in code-related tasks. However, most existing models&#10;are solely pre-trained on extensive raw code data without instruction&#10;fine-tuning. In this paper, we introduce WizardCoder, which empowers Code LLMs&#10;with complex instruction fine-tuning, by adapting the Evol-Instruct method to&#10;the domain of code. Through comprehensive experiments on four prominent code&#10;generation benchmarks, namely HumanEval, HumanEval+, MBPP, and DS-1000, we&#10;unveil the exceptional capabilities of our model. It surpasses all other&#10;open-source Code LLMs by a substantial margin. Moreover, our model even&#10;outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on&#10;HumanEval and HumanEval+. Our code, model weights, and data are public at&#10;https://github.com/nlpxucan/WizardLM&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.01210" label="2305.01210">
        <attvalues>
          <attvalue for="0" value="Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of&#10;  Large Language Models for Code Generation" />
          <attvalue for="1" value="  Program synthesis has been long studied with recent approaches focused on&#10;directly using the power of Large Language Models (LLMs) to generate code.&#10;Programming benchmarks, with curated synthesis problems and test-cases, are&#10;used to measure the performance of various LLMs on code synthesis. However,&#10;these test-cases can be limited in both quantity and quality for fully&#10;assessing the functional correctness of the generated code. Such limitation in&#10;the existing benchmarks begs the following question: In the era of LLMs, is the&#10;code generated really correct? To answer this, we propose EvalPlus -- a code&#10;synthesis evaluation framework to rigorously benchmark the functional&#10;correctness of LLM-synthesized code. EvalPlus augments a given evaluation&#10;dataset with large amounts of test-cases newly produced by an automatic test&#10;input generator, powered by both LLM- and mutation-based strategies. While&#10;EvalPlus is general, we extend the test-cases of the popular HumanEval&#10;benchmark by 80x to build HumanEval+. Our extensive evaluation across 26&#10;popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to&#10;catch significant amounts of previously undetected wrong code synthesized by&#10;LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that&#10;test insufficiency can lead to mis-ranking. For example, both&#10;WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+,&#10;while none of them could on HumanEval. Our work not only indicates that prior&#10;popular code synthesis evaluation results do not accurately reflect the true&#10;performance of LLMs for code synthesis, but also opens up a new direction to&#10;improve such programming benchmarks through automated testing. We have&#10;open-sourced our tools, enhanced datasets as well as all LLM-generated code at&#10;https://github.com/evalplus/evalplus to facilitate and accelerate future&#10;LLM-for-code research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.07732" label="2108.07732">
        <attvalues>
          <attvalue for="0" value="Program Synthesis with Large Language Models" />
          <attvalue for="1" value="  This paper explores the limits of the current generation of large language&#10;models for program synthesis in general purpose programming languages. We&#10;evaluate a collection of such models (with between 244M and 137B parameters) on&#10;two new benchmarks, MBPP and MathQA-Python, in both the few-shot and&#10;fine-tuning regimes. Our benchmarks are designed to measure the ability of&#10;these models to synthesize short Python programs from natural language&#10;descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974&#10;programming tasks, designed to be solvable by entry-level programmers. The&#10;MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914&#10;problems that evaluate the ability of the models to synthesize code from more&#10;complex text. On both datasets, we find that synthesis performance scales&#10;log-linearly with model size. Our largest models, even without finetuning on a&#10;code dataset, can synthesize solutions to 59.6 percent of the problems from&#10;MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a&#10;held-out portion of the dataset improves performance by about 10 percentage&#10;points across most model sizes. On the MathQA-Python dataset, the largest&#10;fine-tuned model achieves 83.8 percent accuracy. Going further, we study the&#10;model's ability to engage in dialog about code, incorporating human feedback to&#10;improve its solutions. We find that natural language feedback from a human&#10;halves the error rate compared to the model's initial prediction. Additionally,&#10;we conduct an error analysis to shed light on where these models fall short and&#10;what types of programs are most difficult to generate. Finally, we explore the&#10;semantic grounding of these models by fine-tuning them to predict the results&#10;of program execution. We find that even our best models are generally unable to&#10;predict the output of a program given a specific input.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.08227" label="2208.08227">
        <attvalues>
          <attvalue for="0" value="MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural&#10;  Code Generation" />
          <attvalue for="1" value="  Large language models have demonstrated the ability to generate both natural&#10;language and programming language text. Such models open up the possibility of&#10;multi-language code generation: could code generation models generalize&#10;knowledge from one language to another? Although contemporary code generation&#10;models can generate semantically correct Python code, little is known about&#10;their abilities with other languages. We propose MultiPL-E, a system for&#10;translating unit test-driven code generation benchmarks to new languages. We&#10;create the first massively multilingual code generation benchmark by using&#10;MultiPL-E to translate two popular Python code generation benchmarks to 18&#10;additional programming languages.&#10;  We use MultiPL-E to extend the HumanEval benchmark and MBPP benchmark to 18&#10;languages that encompass a range of programming paradigms and popularity. Using&#10;these new parallel benchmarks, we evaluate the multi-language performance of&#10;three state-of-the-art code generation models: Codex, CodeGen, and InCoder. We&#10;find that Codex matches or even exceeds its performance on Python for several&#10;other languages. The range of programming languages represented in MultiPL-E&#10;allow us to explore the impact of language frequency and language features on&#10;model performance. Finally, the MultiPL-E approach of compiling code generation&#10;benchmarks to new programming languages is both scalable and extensible, making&#10;it straightforward to evaluate new models, benchmarks, and languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.11501" label="2211.11501">
        <attvalues>
          <attvalue for="0" value="DS-1000: A Natural and Reliable Benchmark for Data Science Code&#10;  Generation" />
          <attvalue for="1" value="  We introduce DS-1000, a code generation benchmark with a thousand data&#10;science problems spanning seven Python libraries, such as NumPy and Pandas.&#10;Compared to prior works, DS-1000 incorporates three core features. First, our&#10;problems reflect diverse, realistic, and practical use cases since we collected&#10;them from StackOverflow. Second, our automatic evaluation is highly specific&#10;(reliable) -- across all Codex-002-predicted solutions that our evaluation&#10;accept, only 1.8% of them are incorrect; we achieve this with multi-criteria&#10;metrics, checking both functional correctness by running test cases and&#10;surface-form constraints by restricting API usages or keywords. Finally, we&#10;proactively defend against memorization by slightly modifying our problems to&#10;be different from the original StackOverflow source; consequently, models&#10;cannot answer them correctly by memorizing the solutions from pre-training. The&#10;current best public system (Codex-002) achieves 43.3% accuracy, leaving ample&#10;room for improvement. We release our benchmark at&#10;https://ds1000-code-gen.github.io.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.06356" label="2205.06356">
        <attvalues>
          <attvalue for="0" value="Beyond Static Models and Test Sets: Benchmarking the Potential of&#10;  Pre-trained Models Across Tasks and Languages" />
          <attvalue for="1" value="  Although recent Massively Multilingual Language Models (MMLMs) like mBERT and&#10;XLMR support around 100 languages, most existing multilingual NLP benchmarks&#10;provide evaluation data in only a handful of these languages with little&#10;linguistic diversity. We argue that this makes the existing practices in&#10;multilingual evaluation unreliable and does not provide a full picture of the&#10;performance of MMLMs across the linguistic landscape. We propose that the&#10;recent work done in Performance Prediction for NLP tasks can serve as a&#10;potential solution in fixing benchmarking in Multilingual NLP by utilizing&#10;features related to data and language typology to estimate the performance of&#10;an MMLM on different languages. We compare performance prediction with&#10;translating test data with a case study on four different multilingual&#10;datasets, and observe that these methods can provide reliable estimates of the&#10;performance that are often on-par with the translation based approaches,&#10;without the need for any additional translation as well as evaluation costs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years have seen a surge of transformer~\cite{vaswanietal2017} based Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLM-RoBERTa (XLMR) \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5}, RemBERT \cite{chung2021rethinking}. These models are pretrained on varying amounts of data of around 100 linguistically diverse languages, and can in principle support fine-tuning on different NLP tasks for these languages. &#10;&#10;These MMLMs are primarily evaluated for their performance on Sequence Labelling \cite{nivre-etal-2020-universal, Pan2017}, Classification \cite{Conneau2018xnli, Yang2019paws-x, ponti-etal-2020-xcopa}, Question Answering \cite{artetxe2020cross, Lewis2020mlqa, Clark2020tydiqa} and Retrieval \cite{Artetxe2019massively, roy-etal-2020-lareqa, botha-etal-2020-entity} tasks. However, most these tasks often cover only a handful of the languages supported by the MMLMs, with most tasks having test sets in fewer than 20 languages (cf. Figure \ref{fig:tasks_langs}). &#10;&#10;Evaluating on such benchmarks henceforth fails to provide a comprehensive picture of the model's performance across the linguistic landscape, as the performance of MMLMs has been shown to vary significantly with the amount of pre-training data available for a language \cite{wu-dredze-2020-languages}, as well according to the typological relatedness between the pivot and target languages \cite{lauscher-etal-2020-zero}. While designing benchmarks to contain test data for all 100 languages supported by the MMLMs is be the ideal standard for multilingual evaluation, doing so requires prohibitively large amount of human effort, time and money. &#10;&#10;Machine Translation can be one way to extend test sets in different benchmarks to a much larger set of languages. \cite{hu2020xtreme} provides pseudo test sets for tasks like XQUAD and XNLI, obtained by translating English test data into different languages, and shows reasonable estimates of the actual performance by evaluating on translated data but cautions about their reliability when the model is trained on translated data. The accuracy of translation based evaluation can be affected by the quality of translation and the technique incurs non-zero costs to obtain reliable translations. Moreover, transferring labels with translation might also be non-trivial for certain tasks like Part of Speech Tagging and Named Entity Recognition.&#10;&#10;Recently, there has been some interest in predicting performance of NLP models without actually evaluating them on a test set. \cite{xia-etal-2020-predicting} showed that it is possible to build regression models that can accurately predict evaluation scores of NLP models under different experimental settings using various linguistic and dataset specific features. \cite{srinivasan2021predicting} showed promising results specifically for MMLMs towards predicting their performance on downstream tasks for different languages in zero-shot and few-shot settings, and \cite{ye-etal-2021-towards} propose methods for more reliable performance prediction by estimating confidence intervals as well as predicting fine-grained performance measures.&#10;&#10;In this paper we argue that the performance prediction can be a possible avenue to address the current issues with Multilingual benchmarking by aiding in the estimation of performance of the MMLMs for the languages which lack any evaluation data for a given task. Not only this can help us give a better idea about the performance of a multilingual model on a task across a much larger set of languages and hence aiding in better model selection, but also enables applications in devising data collection strategies to maximize performance \cite{srinivasan2022litmus} as well as in selecting the representative set of languages for a benchmark \cite{xia-etal-2020-predicting}.&#10;&#10;We present a case study demonstrating the effectiveness of performance prediction on four multilingual tasks, PAWS-X \cite{Yang2019paws-x} XNLI \cite{Conneau2018xnli}, XQUAD \cite{artetxe2020cross} and TyDiQA-GoldP \cite{Clark2020tydiqa} and show that it can often provide reliable estimates of the performance on different languages on par with evaluating them on translated test sets without any additional translation costs. We also demonstrate an additional use case of this method in selecting the best pivot language for fine-tuning the MMLM in order to maximize performance on some target language. To encourage research in this area and provide easy access for the community to utilize this framework, we will release our code and the datasets that we use for the case study.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Mathematics, Performance Prediction, Natural Language Processing, Multilingual NLP" />
        </attvalues>
      </node>
      <node id="2010.12821" label="2010.12821">
        <attvalues>
          <attvalue for="0" value="Rethinking embedding coupling in pre-trained language models" />
          <attvalue for="1" value="  We re-evaluate the standard practice of sharing weights between input and&#10;output embeddings in state-of-the-art pre-trained language models. We show that&#10;decoupled embeddings provide increased modeling flexibility, allowing us to&#10;significantly improve the efficiency of parameter allocation in the input&#10;embedding of multilingual models. By reallocating the input embedding&#10;parameters in the Transformer layers, we achieve dramatically better&#10;performance on standard natural language understanding tasks with the same&#10;number of parameters during fine-tuning. We also show that allocating&#10;additional capacity to the output embedding provides benefits to the model that&#10;persist through the fine-tuning stage even though the output embedding is&#10;discarded after pre-training. Our analysis shows that larger output embeddings&#10;prevent the model's last layers from overspecializing to the pre-training task&#10;and encourage Transformer representations to be more general and more&#10;transferable to other tasks and languages. Harnessing these findings, we are&#10;able to train models that achieve strong performance on the XTREME benchmark&#10;without increasing the number of parameters at the fine-tuning stage.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Embedding coupling Sharing input and output embeddings in neural language models was proposed to improve perplexity and motivated based on embedding similarity \cite{Press2017} as well as by theoretically showing that the output probability space can be constrained to a subspace governed by the embedding matrix for a restricted case \cite{Inan2017}. Embedding coupling is also common in neural machine translation models where it reduces model complexity \cite{Firat2016} and saves memory \cite{Johnson2017}, in recent state-of-the-art language models \cite{Melis2020mogrifier}, as well as all pre-trained models we are aware of \cite{Devlin2019,Liu2019roberta}.&#10;&#10;Transferability of representations Representations of large pre-trained models in computer vision and NLP have been observed to transition from general to task-specific from the first to the last layer \cite{yosinski2014transferable,Howard2018,Liu2019transferability}. In Transformer models, the last few layers have been shown to become specialized to the MLM task and---as a result---less transferable \cite{Zhang2020,Tamkin2020}.&#10;&#10;Multilingual models Recent multilingual models are pre-trained on data covering around 100 languages using a subword vocabulary shared across all languages \cite{Devlin2019,Pires2019,Conneau2020}. In order to achieve reasonable performance for most languages, these models need to allocate sufficient capacity for each language, known as the curse of multilinguality \cite{Conneau2020,Pfeiffer2020mad-x}. As a result, such multilingual models have large vocabularies with large embedding sizes to ensure that tokens in all languages are adequately represented. &#10;&#10;Efficient models Most work on more efficient pre-trained models focuses on pruning or distillation \cite{Hinton2015}. Pruning approaches remove parts of the model, typically attention heads \cite{Michel2019,Voita2019} while distillation approaches distill a large pre-trained model into a smaller one \cite{Sun2020mobilebert}. Distillation can be seen as an alternative form of allocating pre-training capacity via a large teacher model. However, distilling a pre-trained model is expensive \cite{Sanh2019distilbert} and requires overcoming architecture differences and balancing training data and loss terms \cite{Mukherjee2020xtremedistil}. Our proposed methods are simpler and complementary to distillation as they can improve the pre-training of compact student models \cite{Turc2019well-read}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1812.10464" label="1812.10464">
        <attvalues>
          <attvalue for="0" value="Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual&#10;  Transfer and Beyond" />
          <attvalue for="1" value="  We introduce an architecture to learn joint multilingual sentence&#10;representations for 93 languages, belonging to more than 30 different families&#10;and written in 28 different scripts. Our system uses a single BiLSTM encoder&#10;with a shared BPE vocabulary for all languages, which is coupled with an&#10;auxiliary decoder and trained on publicly available parallel corpora. This&#10;enables us to learn a classifier on top of the resulting embeddings using&#10;English annotated data only, and transfer it to any of the 93 languages without&#10;any modification. Our experiments in cross-lingual natural language inference&#10;(XNLI dataset), cross-lingual document classification (MLDoc dataset) and&#10;parallel corpus mining (BUCC dataset) show the effectiveness of our approach.&#10;We also introduce a new test set of aligned sentences in 112 languages, and&#10;show that our sentence embeddings obtain strong results in multilingual&#10;similarity search even for low-resource languages. Our implementation, the&#10;pre-trained encoder and the multilingual test set are available at&#10;https://github.com/facebookresearch/LASER&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08875" label="2110.08875">
        <attvalues>
          <attvalue for="0" value="Predicting the Performance of Multilingual NLP Models" />
          <attvalue for="1" value="  Recent advancements in NLP have given us models like mBERT and XLMR that can&#10;serve over 100 languages. The languages that these models are evaluated on,&#10;however, are very few in number, and it is unlikely that evaluation datasets&#10;will cover all the languages that these models support. Potential solutions to&#10;the costly problem of dataset creation are to translate datasets to new&#10;languages or use template-filling based techniques for creation. This paper&#10;proposes an alternate solution for evaluating a model across languages which&#10;make use of the existing performance scores of the model on languages that a&#10;particular task has test sets for. We train a predictor on these performance&#10;scores and use this predictor to predict the model's performance in different&#10;evaluation settings. Our results show that our method is effective in filling&#10;the gaps in the evaluation for an existing set of languages, but might require&#10;additional improvements if we want it to generalize to unseen languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;mBERT \cite{devlin-etal-2019-bert} and XLMR \cite{conneau-etal-2020-unsupervised} are multilingual transformer models that have been pretrained on text from around 100 languages. These models have been evaluated on tasks like natural language inference, document classification, parts of speech tagging, named entity recognition, dependency parsing \cite{conneau-etal-2018-xnli,pires-etal-2019-multilingual,wu-dredze-2019-beto,wu-dredze-2020-languages} and have shown excellent cross-lingual transfer of performance. These models were found to perform well on tasks involving code-mixed text too \cite{khanuja-etal-2020-gluecos,aguilar-etal-2020-lince}. Given that these models have to support over 100 languages with the limited model capacity they have, some works have found that they are outperformed by monolingual versions, even on some low-resourced languages \cite{devries2019bertje,ortiz-suarez-etal-2020-monolingual,virtanen2019multilingual,pyysalo-etal-2021-wikibert}.&#10;&#10;Some of the aforementioned works \cite{pires-etal-2019-multilingual,wu-dredze-2019-beto,wu-dredze-2020-languages} have theorized about the reasons why cross-lingual transfer works and have stated that factors like pretraining data size and vocabulary overlap between languages could affect the transfer performance of a language in these models. \cite{lauscher-etal-2020-zero} study the correlation between transfer performance and factors like syntax, phonology, data size when English is used as the finetuning language. \cite{turc2021revisiting} study the performance of mBERT and mT5 \cite{xue2021mt5} on a wider variety of tasks and look at the impact of using different languages as the finetuning/pivot language.&#10;&#10;The task of predicting an NLP model's performance is something that has been looked into in the traditional train-transfer scenario that was popular before pretrained NLP models, with the goal of determining which high-resourced transfer language to use to maximize the performance in a lower-resourced target language. \cite{lin-etal-2019-choosing} develop a set of features based on the overlap between the 2 languages and use that to predict which transfer language would be the best. \cite{xia-etal-2020-predicting,ye-etal-2021-towards} evaluate similar techniques on a wider range of tasks and look into better reliability estimates for prediction. \cite{vu-etal-2020-exploring} meanwhile look at intermediate task finetuning in English, i.e finetuning on a high resourced intermediate task before finetuning on the target task that might have lesser data. They build a set of task embeddings from the transformer model's representations that can be used to predict which intermediate task to finetune on to maximize the performance on the final task. In contrast to these existing works, our work looks specifically into pretrained multilingual transformer models, where a single model can be trained and tested on multiple languages.&#10; " />
        </attvalues>
      </node>
      <node id="2102.05486" label="2102.05486">
        <attvalues>
          <attvalue for="0" value="Towards More Fine-grained and Reliable NLP Performance Prediction" />
          <attvalue for="1" value="  Performance prediction, the task of estimating a system's performance without&#10;performing experiments, allows us to reduce the experimental burden caused by&#10;the combinatorial explosion of different datasets, languages, tasks, and&#10;models. In this paper, we make two contributions to improving performance&#10;prediction for NLP tasks. First, we examine performance predictors not only for&#10;holistic measures of accuracy like F1 or BLEU but also fine-grained performance&#10;measures such as accuracy over individual classes of examples. Second, we&#10;propose methods to understand the reliability of a performance prediction model&#10;from two angles: confidence intervals and calibration. We perform an analysis&#10;of four types of NLP tasks, and both demonstrate the feasibility of&#10;fine-grained performance prediction and the necessity to perform reliability&#10;analysis for performance prediction methods in the future. We make our code&#10;publicly available: \url{https://github.com/neulab/Reliable-NLPPP}&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.11828" label="1908.11828">
        <attvalues>
          <attvalue for="0" value="PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase&#10;  Identification" />
          <attvalue for="1" value="  Most existing work on adversarial data generation focuses on English. For&#10;example, PAWS (Paraphrase Adversaries from Word Scrambling) consists of&#10;challenging English paraphrase identification pairs from Wikipedia and Quora.&#10;We remedy this gap with PAWS-X, a new dataset of 23,659 human translated PAWS&#10;evaluation pairs in six typologically distinct languages: French, Spanish,&#10;German, Chinese, Japanese, and Korean. We provide baseline numbers for three&#10;models with different capacity to capture non-local context and sentence&#10;structure, and using different multilingual training and evaluation regimes.&#10;Multilingual BERT fine-tuned on PAWS English plus machine-translated data&#10;performs the best, with a range of 83.1-90.8 accuracy across the non-English&#10;languages and an average accuracy gain of 23% over the next best model. PAWS-X&#10;shows the effectiveness of deep, multilingual pre-training while also leaving&#10;considerable headroom as a new challenge to drive multilingual research that&#10;better captures structure and contextual information.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.05002" label="2003.05002">
        <attvalues>
          <attvalue for="0" value="TyDi QA: A Benchmark for Information-Seeking Question Answering in&#10;  Typologically Diverse Languages" />
          <attvalue for="1" value="  Confidently making progress on multilingual modeling requires challenging,&#10;trustworthy evaluations. We present TyDi QA---a question answering dataset&#10;covering 11 typologically diverse languages with 204K question-answer pairs.&#10;The languages of TyDi QA are diverse with regard to their typology---the set of&#10;linguistic features each language expresses---such that we expect models&#10;performing well on this set to generalize across a large number of the world's&#10;languages. We present a quantitative analysis of the data quality and&#10;example-level qualitative linguistic analyses of observed language phenomena&#10;that would not be found in English-only corpora. To provide a realistic&#10;information-seeking task and avoid priming effects, questions are written by&#10;people who want to know the answer, but don't know the answer yet, and the data&#10;is collected directly in each language without the use of translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.15682" label="2012.15682">
        <attvalues>
          <attvalue for="0" value="A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots&#10;  Matters" />
          <attvalue for="1" value="  Few-shot crosslingual transfer has been shown to outperform its zero-shot&#10;counterpart with pretrained encoders like multilingual BERT. Despite its&#10;growing popularity, little to no attention has been paid to standardizing and&#10;analyzing the design of few-shot experiments. In this work, we highlight a&#10;fundamental risk posed by this shortcoming, illustrating that the model&#10;exhibits a high degree of sensitivity to the selection of few shots. We conduct&#10;a large-scale experimental study on 40 sets of sampled few shots for six&#10;diverse NLP tasks across up to 40 languages. We provide an analysis of success&#10;and failure cases of few-shot transfer, which highlights the role of lexical&#10;features. Additionally, we show that a straightforward full model finetuning&#10;approach is quite effective for few-shot transfer, outperforming several&#10;state-of-the-art few-shot approaches. As a step towards standardizing few-shot&#10;crosslingual experimental designs, we make our sampled few shots publicly&#10;available.&#10;" />
          <attvalue for="2" value="&#10;Multilingual pretrained encoders like multilingual BERT&#10;(mBERT; \cite{devlin-etal-2019-bert})&#10;and XLM-R \cite{conneau-etal-2020-unsupervised}&#10;are the top performers in crosslingual&#10;tasks such as natural language inference \cite{conneau2018xnli},&#10;document classification \cite{SCHWENK18.658,artetxe-schwenk-2019-massively},&#10;and argument mining \cite{toledo-ronen-etal-2020-multilingual}.&#10;They enable transfer learning through language-agnostic representations&#10;in crosslingual setups \cite{hu2020xtreme}.&#10;&#10;A widely explored transfer scenario is&#10;zero-shot crosslingual transfer \cite{&#10; pires-etal-2019-multilingual,conneau2019cross,artetxe-schwenk-2019-massively},&#10;where a pretrained encoder is finetuned on abundant task data in the&#10;source language (e.g., English) and then directly evaluated on&#10;target-language test data, achieving surprisingly good performance&#10;\cite{wu-dredze-2019-beto,hu2020xtreme}.&#10;However,&#10;there is evidence&#10;that zero-shot performance reported in the literature has&#10;large variance and is often not reproducible&#10;\cite{keung2020evaluation,rios-mller-sennrich:2020:WMT}; the results&#10;in languages distant from English fall far short of those similar to&#10;English \cite{hu2020xtreme,liang2020xglue}.&#10;&#10;\cite{lauscher2020zero} stress the importance of few-shot&#10; crosslingual transfer instead, where the encoder is first finetuned&#10;on a source language and then further finetuned with a small amount&#10;(10--100) of examples (few shots) of the target language.&#10;The few shots substantially improve model performance of the target&#10;language with negligible annotation costs&#10;\cite{garrette-baldridge-2013-learning,hedderich-etal-2020-transfer}.&#10;&#10;In this work, however, we demonstrate that the gains from few-shot&#10;transfer exhibit a high degree of sensitivity to the selection&#10; of few shots.&#10;For example, different choices for the few shots can yield a&#10;performance variance of over 10\% accuracy in a standard document classification task. &#10;Motivated by this, we propose to fix the few shots for fair comparisons between different&#10;crosslingual transfer methods, and provide a benchmark&#10;resembling the standard ``$N$-way $K$-shot'' few-shot&#10;learning configuration \cite{feifeilioneshot,Koch2015SiameseNN}.&#10;We also evaluate and compare several state-of-the-art (SotA) few-shot&#10;finetuning techniques, in order to understand their performance&#10;and susceptibility to the variance related to few shots.&#10;&#10;We also demonstrate that the effectiveness of few-shot&#10;crosslingual transfer depends on the type of&#10;downstream task. For&#10;syntactic tasks such as named-entity recognition, the few shots can&#10;improve results by up to $\approx$20 $F_1$ points.&#10;For challenging&#10;tasks like adversarial paraphrase identification, the few shots do not&#10;help and even sometimes lead to worse performance than zero-shot&#10;transfer.&#10;To understand these phenomena, we conduct additional in-depth&#10;analyses, and find that the models tend to utilize shallow lexical&#10;hints \cite{shortcutlearning} in the target&#10;language, rather than leveraging&#10;abstract&#10;crosslingual semantic features learned from the source language.&#10;&#10;Our contributions:&#10;1) We show that few-shot&#10;crosslingual transfer is prone to large variations in task&#10;performance; this property hinders unbiased assessments of the&#10;effectiveness of different few-shot methods.&#10;2) To remedy&#10;this issue, we publish fixed and standardized few shots to support&#10;fair comparisons and reproducibility.&#10;3) We empirically&#10;verify that few-shot crosslingual transfer has different performance&#10;impact on structurally different tasks; we&#10;provide in-depth&#10;analyses concerning the source of performance gains.&#10;4) We&#10;analyze several SotA few-shot learning methods, and show that they&#10;underperform simple full model finetuning. We hope that our work will&#10;shed new light on the potential and current difficulties of few-shot&#10;learning in crosslingual setups.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="NLP Model Analysis, Few-Shot Learning, Computer Science, Machine Learning, Linguistics, Crosslingual Transfer, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2010.06432" label="2010.06432">
        <attvalues>
          <attvalue for="0" value="Multilingual Argument Mining: Datasets and Analysis" />
          <attvalue for="1" value="  The growing interest in argument mining and computational argumentation&#10;brings with it a plethora of Natural Language Understanding (NLU) tasks and&#10;corresponding datasets. However, as with many other NLU tasks, the dominant&#10;language is English, with resources in other languages being few and far&#10;between. In this work, we explore the potential of transfer learning using the&#10;multilingual BERT model to address argument mining tasks in non-English&#10;languages, based on English datasets and the use of machine translation. We&#10;show that such methods are well suited for classifying the stance of arguments&#10;and detecting evidence, but less so for assessing the quality of arguments,&#10;presumably because quality is harder to preserve under translation. In&#10;addition, focusing on the translate-train approach, we show how the choice of&#10;languages for translation, and the relations among them, affect the accuracy of&#10;the resultant model. Finally, to facilitate evaluation of transfer learning on&#10;argument mining tasks, we provide a human-generated dataset with more than 10k&#10;arguments in multiple languages, as well as machine translation of the English&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.15001" label="2004.15001">
        <attvalues>
          <attvalue for="0" value="Don't Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of&#10;  Contextual Embeddings" />
          <attvalue for="1" value="  Multilingual contextual embeddings have demonstrated state-of-the-art&#10;performance in zero-shot cross-lingual transfer learning, where multilingual&#10;BERT is fine-tuned on one source language and evaluated on a different target&#10;language. However, published results for mBERT zero-shot accuracy vary as much&#10;as 17 points on the MLDoc classification task across four papers. We show that&#10;the standard practice of using English dev accuracy for model selection in the&#10;zero-shot setting makes it difficult to obtain reproducible results on the&#10;MLDoc and XNLI tasks. English dev accuracy is often uncorrelated (or even&#10;anti-correlated) with target language accuracy, and zero-shot performance&#10;varies greatly at different points in the same fine-tuning run and between&#10;different fine-tuning runs. These reproducibility issues are also present for&#10;other tasks with different pre-trained embeddings (e.g., MLQA with XLM-R). We&#10;recommend providing oracle scores alongside zero-shot results: still fine-tune&#10;using English data, but choose a checkpoint with the target dev set. Reporting&#10;this upper bound makes results more consistent by avoiding arbitrarily bad&#10;checkpoints.&#10;" />
          <attvalue for="2" value="&#10;&#10;Zero-shot and zero-resource cross-lingual NLP has seen significant progress in recent years. The discovery of cross-lingual structure in word embedding spaces culminated in the work of \cite{ConneauLRDJ18-word}, which showed that unsupervised word translation via adversarial mappings is competitive with supervised techniques. Concurrent work in machine translation also showed that it is possible to achieve non-trivial BLEU scores without any bitext \cite{ArtetxeLAC18-unsupervised,LampleCDR18-unsupervised}. Self-supervised multilingual contextual embeddings like mBERT \cite{devlin-etal-2019-bert} and XLM \cite{LampleC19-cross} have shown remarkably strong performance on cross-lingual named entity recognition, text classification, dependency parsing, and other tasks (e.g., \cite{pires-etal-2019-multilingual,keung-etal-2019-adversarial,wu-dredze-2019-beto}).&#10;&#10;Much of this recent work has demonstrated that mBERT performs very well on zero-shot tasks, superseding prior techniques as the baseline for zero-shot cross-lingual transfer learning. By zero-shot, we mean that no parallel text or labeled data from the target language was used during model training, fine-tuning, or hyperparameter search. In this setting, models are trained on labeled (usually English) text and tested on target (non-English) text. Standard practice prohibits the use of target language data for model selection; the final model is chosen using the English dev set only.&#10;&#10;However, we find that zero-shot mBERT results can vary greatly. We present 4 published baselines for zero-shot cross-lingual document classification on MLDoc \cite{schwenk-li-2018-corpus} in Table \ref{table:published-mbert}: &#10;&#10;Even though the authors report English accuracies which are basically identical, their target language performances are very different. Given that each experiment starts with the same pre-trained mBERT model and MLDoc dataset, it is clear that these cross-lingual results are not reproducible. For the listed target languages, the highest accuracy is up to 3 points better than the next best and up to 17 points better than the worst. We investigate this reproducibility issue in both MLDoc and XNLI \cite{conneau-etal-2018-xnli}, which is another major dataset for evaluating cross-lingual transfer. The variations in published baselines on these and other datasets are summarized in Sec.~\ref{sec:published}.&#10;&#10;In Section \ref{sec:variance}, we show that the final zero-shot accuracies between and within independent mBERT training runs are highly variable. Variations over different random seeds are similar in magnitude to those in Table \ref{table:published-mbert}, with variation due to checkpoint selection using English dev being a significant underlying cause. In Section \ref{sec:directional}, we find that in many cases, English (En) dev accuracy is not predictive of target language performance. In fact, for some languages, En dev performance is actually anti-correlated with target language accuracy. &#10;&#10;Poor correlation between En dev and target test accuracy, combined with high variance between independent runs, means that published zero-shot accuracies are somewhat arbitrary. In addition to zero-shot results, we recommend reporting oracle results, where one still fine-tunes using En data, but uses the target dev set for checkpoint selection.&#10; " />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.01703" label="2011.01703">
        <attvalues>
          <attvalue for="0" value="Subword Segmentation and a Single Bridge Language Affect Zero-Shot&#10;  Neural Machine Translation" />
          <attvalue for="1" value="  Zero-shot neural machine translation is an attractive goal because of the&#10;high cost of obtaining data and building translation systems for new&#10;translation directions. However, previous papers have reported mixed success in&#10;zero-shot translation. It is hard to predict in which settings it will be&#10;effective, and what limits performance compared to a fully supervised system.&#10;In this paper, we investigate zero-shot performance of a multilingual&#10;EN$\leftrightarrow${FR,CS,DE,FI} system trained on WMT data. We find that&#10;zero-shot performance is highly unstable and can vary by more than 6 BLEU&#10;between training runs, making it difficult to reliably track improvements. We&#10;observe a bias towards copying the source in zero-shot translation, and&#10;investigate how the choice of subword segmentation affects this bias. We find&#10;that language-specific subword segmentation results in less subword copying at&#10;training time, and leads to better zero-shot performance compared to jointly&#10;trained segmentation. A recent trend in multilingual models is to not train on&#10;parallel data between all language pairs, but have a single bridge language,&#10;e.g. English. We find that this negatively affects zero-shot translation and&#10;leads to a failure mode where the model ignores the language tag and instead&#10;produces English output in zero-shot directions. We show that this bias towards&#10;English can be effectively reduced with even a small amount of parallel data in&#10;some of the non-English pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03179" label="2010.03179">
        <attvalues>
          <attvalue for="0" value="Transfer Learning and Distant Supervision for Multilingual Transformer&#10;  Models: A Study on African Languages" />
          <attvalue for="1" value="  Multilingual transformer models like mBERT and XLM-RoBERTa have obtained&#10;great improvements for many NLP tasks on a variety of languages. However,&#10;recent works also showed that results from high-resource languages could not be&#10;easily transferred to realistic, low-resource scenarios. In this work, we study&#10;trends in performance for different amounts of available resources for the&#10;three African languages Hausa, isiXhosa and Yor\`ub\'a on both NER and topic&#10;classification. We show that in combination with transfer learning or distant&#10;supervision, these models can achieve with as little as 10 or 100 labeled&#10;sentences the same performance as baselines with much more supervised training&#10;data. However, we also find settings where this does not hold. Our discussions&#10;and additional experiments on assumptions such as time and hardware&#10;restrictions highlight challenges and opportunities in low-resource learning.&#10;" />
          <attvalue for="2" value="&#10;&#10;Deep learning techniques, including contextualized word embeddings based on transformers and pretrained on language modelling, have resulted in considerable improvements for many NLP tasks. However, they often require large amounts of labeled training data, and there is also growing evidence that transferring approaches from high to low-resource settings is not straightforward. In \cite{african/Loubser2020}, rule-based or linguistically motivated CRFs still outperform RNN-based methods on several tasks for South African languages. For pretraining approaches where labeled data exists in a high-resource language, and the information is transferred to a low-resource language, \cite{data/Xtreme20} find a significant gap between performance on English and the cross-lingually transferred models. In a recent study, \cite{lowresource/Lauscher2020FromZTH} find that the transfer for multilingual transformer models is less effective for resource-lean settings and distant languages. A popular technique to obtain labeled data quickly and cheaply is distant and weak supervision. \cite{lowresource/kann20weakly} recently inspected POS classifiers trained on weak supervision. They found that in contrast to scenarios with simulated low-resource settings of high-resource languages, in truly low-resource settings this is still a difficult problem. These findings also highlight the importance of aiming for realistic experiments when studying low-resource scenarios. &#10;&#10;In this work, we analyse multilingual transformer models, namely mBERT \cite{models/BERT, models/mBERT} and XLM-RoBERTa \cite{models/RoBERTa}. We evaluate both sequence and token classification tasks in the form of news title topic classification and named entity recognition (NER). A variety of approaches have been proposed to improve performance in low-resource settings. In this work, we study (i) transfer learning from a high-resource language and (ii) distant supervision. We selected these as they are two of the most popular techniques in the recent literature and are rather independent of a specific model architecture. Both need auxiliary data. For transfer learning, this is labeled data in a high-resource language, and for distant supervision, this is expert insight and a mechanism to (semi-)automatically generate labels. We see them, therefore, as orthogonal and depending on the scenario and the data availability, either one or the other approach might be applicable.&#10;&#10;Our study is performed on three, linguistically different African languages: Hausa, isiXhosa and \yoruba. These represent languages with millions of users and active use of digital infrastructure, but with only very limited support for NLP technologies. For this aim, we also collected three new datasets that are made publicly available alongside the code and additional material.&#10;&#10;We show both challenges and opportunities when working with multilingual transformer models evaluating trends for different levels of resource scarcity. The paper is structured into the following questions we are interested in:&#10;\begin{itemize}[leftmargin=*]&#10; \item How do more complex transformer models compare to established RNNs?&#10; \item How can transfer-learning be used effectively?&#10; \item Is distant supervision helpful?&#10; \item What assumptions do we have to consider when targeting a realistic treatment of low-resource scenarios?&#10;\end{itemize} &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.07780" label="2004.07780">
        <attvalues>
          <attvalue for="0" value="Shortcut Learning in Deep Neural Networks" />
          <attvalue for="1" value="  Deep learning has triggered the current rise of artificial intelligence and&#10;is the workhorse of today's machine intelligence. Numerous success stories have&#10;rapidly spread all over science, industry and society, but its limitations have&#10;only recently come into focus. In this perspective we seek to distill how many&#10;of deep learning's problems can be seen as different symptoms of the same&#10;underlying problem: shortcut learning. Shortcuts are decision rules that&#10;perform well on standard benchmarks but fail to transfer to more challenging&#10;testing conditions, such as real-world scenarios. Related issues are known in&#10;Comparative Psychology, Education and Linguistics, suggesting that shortcut&#10;learning may be a common characteristic of learning systems, biological and&#10;artificial alike. Based on these observations, we develop a set of&#10;recommendations for model interpretation and benchmarking, highlighting recent&#10;advances in machine learning to improve robustness and transferability from the&#10;lab to real-world applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.05781" label="2009.05781">
        <attvalues>
          <attvalue for="0" value="Intent Detection with WikiHow" />
          <attvalue for="1" value="  Modern task-oriented dialog systems need to reliably understand users'&#10;intents. Intent detection is most challenging when moving to new domains or new&#10;languages, since there is little annotated data. To address this challenge, we&#10;present a suite of pretrained intent detection models. Our models are able to&#10;predict a broad range of intended goals from many actions because they are&#10;trained on wikiHow, a comprehensive instructional website. Our models achieve&#10;state-of-the-art results on the Snips dataset, the Schema-Guided Dialogue&#10;dataset, and all 3 languages of the Facebook multilingual dialog datasets. Our&#10;models also demonstrate strong zero- and few-shot performance, reaching over&#10;75% accuracy using only 100 training examples in all datasets.&#10;" />
          <attvalue for="2" value="&#10;Task-oriented dialog systems like Apple's Siri, Amazon Alexa, and Google Assistant have become pervasive in smartphones and smart speakers. To support a wide range of functions, dialog systems must be able to map a user's natural language instruction onto the desired skill or API. Performing this mapping is called intent detection.&#10;&#10;Intent detection is usually formulated as a sentence classification task. Given an utterance (e.g. ``wake me up at 8''), a system needs to predict its intent (e.g. ``Set an Alarm''). Most modern approaches use neural networks to jointly model intent detection and slot filling \cite{6707709,liu2016attentionbased,goo-etal-2018-slot,zhang-etal-2019-joint}. In response to a rapidly growing range of services, more attention has been given to zero-shot intent detection \cite{ferreira2015zero,ferreira15,yazdani-henderson-2015-model,chen2016zero,kumar2017zero,gangadharaiah-narayanaswamy-2019-joint}. While most existing research on intent detection proposed novel model architectures, few have attempted data augmentation. One such work \cite{10.1145/1526709.1526773} showed that models can learn much knowledge that is important for intent detection from massive online resources such as Wikipedia.&#10;&#10;We propose a pretraining task based on wikiHow, a comprehensive instructional website with over 110,000 professionally edited articles. Their topics span from common sense such as ``How to Download Music'' to more niche tasks like ``How to Crochet a Teddy Bear.'' We observe that the header of each step in a wikiHow article describes an action and can be approximated as an utterance, while the title describes a goal and can be seen as an intent. For example, ``find good gas prices'' in the article ``How to Save Money on Gas'' is similar to the utterance ``where can I find cheap gas?'' with the intent ``Save Money on Gas.'' Hence, we introduce a dataset based on wikiHow, where a model predicts the goal of an action given some candidates. Although most of wikiHow's domains are far beyond the scope of any present dialog system, models pretrained on our dataset would be robust to emerging services and scenarios. Also, as wikiHow is available in 18 languages, our pretraining task can be readily extended to multilingual settings.&#10;&#10;Using our pretraining task, we fine-tune transformer language models, achieving state-of-the-art results on the intent detection task of the Snips dataset \cite{coucke2018snips}, the Schema-Guided Dialog (SGD) dataset \cite{rastogi2019towards}, and all 3 languages (English, Spanish, and Thai) of the Facebook multilingual dialog datasets \cite{schuster-etal-2019-cross-lingual}, with statistically significant improvements. As our accuracy is close to 100\% on all these datasets, we further experiment with zero- or few-shot settings. Our models achieve over 70\% accuracy with no in-domain training data on Snips and SGD, and over 75\% with only 100 training examples on all datasets. This highlights our models' ability to quickly adapt to new utterances and intents in unseen domains. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Multilingual Dialogue, Linguistics, Pretrained Models, Artificial Intelligence, Natural Language Processing, Intent Detection" />
        </attvalues>
      </node>
      <node id="1609.01454" label="1609.01454">
        <attvalues>
          <attvalue for="0" value="Attention-Based Recurrent Neural Network Models for Joint Intent&#10;  Detection and Slot Filling" />
          <attvalue for="1" value="  Attention-based encoder-decoder neural network models have recently shown&#10;promising results in machine translation and speech recognition. In this work,&#10;we propose an attention-based neural network model for joint intent detection&#10;and slot filling, both of which are critical steps for many speech&#10;understanding and dialog systems. Unlike in machine translation and speech&#10;recognition, alignment is explicit in slot filling. We explore different&#10;strategies in incorporating this alignment information to the encoder-decoder&#10;framework. Learning from the attention mechanism in encoder-decoder model, we&#10;further propose introducing attention to the alignment-based RNN models. Such&#10;attentions provide additional information to the intent classification and slot&#10;label prediction. Our independent task models achieve state-of-the-art intent&#10;detection error rate and slot filling F1 score on the benchmark ATIS task. Our&#10;joint training model further obtains 0.56% absolute (23.8% relative) error&#10;reduction on intent detection and 0.23% absolute gain on slot filling over the&#10;independent task models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.09471" label="1812.09471">
        <attvalues>
          <attvalue for="0" value="Joint Slot Filling and Intent Detection via Capsule Neural Networks" />
          <attvalue for="1" value="  Being able to recognize words as slots and detect the intent of an utterance&#10;has been a keen issue in natural language understanding. The existing works&#10;either treat slot filling and intent detection separately in a pipeline manner,&#10;or adopt joint models which sequentially label slots while summarizing the&#10;utterance-level intent without explicitly preserving the hierarchical&#10;relationship among words, slots, and intents. To exploit the semantic hierarchy&#10;for effective modeling, we propose a capsule-based neural network model which&#10;accomplishes slot filling and intent detection via a dynamic&#10;routing-by-agreement schema. A re-routing schema is proposed to further&#10;synergize the slot filling performance using the inferred intent&#10;representation. Experiments on two real-world datasets show the effectiveness&#10;of our model when compared with other alternative model architectures, as well&#10;as existing natural language understanding services.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.10190" label="1805.10190">
        <attvalues>
          <attvalue for="0" value="Snips Voice Platform: an embedded Spoken Language Understanding system&#10;  for private-by-design voice interfaces" />
          <attvalue for="1" value="  This paper presents the machine learning architecture of the Snips Voice&#10;Platform, a software solution to perform Spoken Language Understanding on&#10;microprocessors typical of IoT devices. The embedded inference is fast and&#10;accurate while enforcing privacy by design, as no personal user data is ever&#10;collected. Focusing on Automatic Speech Recognition and Natural Language&#10;Understanding, we detail our approach to training high-performance Machine&#10;Learning models that are small enough to run in real-time on small devices.&#10;Additionally, we describe a data generation procedure that provides sufficient,&#10;high-quality training data without compromising user privacy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.13327" label="1810.13327">
        <attvalues>
          <attvalue for="0" value="Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog" />
          <attvalue for="1" value="  One of the first steps in the utterance interpretation pipeline of many&#10;task-oriented conversational AI systems is to identify user intents and the&#10;corresponding slots. Since data collection for machine learning models for this&#10;task is time-consuming, it is desirable to make use of existing data in a&#10;high-resource language to train models in low-resource languages. However,&#10;development of such models has largely been hindered by the lack of&#10;multilingual training data. In this paper, we present a new data set of 57k&#10;annotated utterances in English (43k), Spanish (8.6k) and Thai (5k) across the&#10;domains weather, alarm, and reminder. We use this data set to evaluate three&#10;different cross-lingual transfer methods: (1) translating the training data,&#10;(2) using cross-lingual pre-trained embeddings, and (3) a novel method of using&#10;a multilingual machine translation encoder as contextual word representations.&#10;We find that given several hundred training examples in the the target&#10;language, the latter two methods outperform translating the training data.&#10;Further, in very low-resource settings, multilingual contextual word&#10;representations give better results than using cross-lingual static embeddings.&#10;We also compare the cross-lingual methods to using monolingual resources in the&#10;form of contextual ELMo representations and find that given just small amounts&#10;of target language data, this method outperforms all cross-lingual methods,&#10;which highlights the need for more sophisticated cross-lingual methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.09390" label="2309.09390">
        <attvalues>
          <attvalue for="0" value="Augmenting text for spoken language understanding with Large Language&#10;  Models" />
          <attvalue for="1" value="  Spoken semantic parsing (SSP) involves generating machine-comprehensible&#10;parses from input speech. Training robust models for existing application&#10;domains represented in training data or extending to new domains requires&#10;corresponding triplets of speech-transcript-semantic parse data, which is&#10;expensive to obtain. In this paper, we address this challenge by examining&#10;methods that can use transcript-semantic parse data (unpaired text) without&#10;corresponding speech. First, when unpaired text is drawn from existing textual&#10;corpora, Joint Audio Text (JAT) and Text-to-Speech (TTS) are compared as ways&#10;to generate speech representations for unpaired text. Experiments on the STOP&#10;dataset show that unpaired text from existing and new domains improves&#10;performance by 2% and 30% in absolute Exact Match (EM) respectively. Second, we&#10;consider the setting when unpaired text is not available in existing textual&#10;corpora. We propose to prompt Large Language Models (LLMs) to generate unpaired&#10;text for existing and new domains. Experiments show that examples and words&#10;that co-occur with intents can be used to generate unpaired text with Llama&#10;2.0. Using the generated text with JAT and TTS for spoken semantic parsing&#10;improves EM on STOP by 1.4% and 2.6% absolute for existing and new domains&#10;respectively.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken Language Understanding (SLU) is essential for many real-world applications today including conversational agents and virtual assistants. Spoken Semantic Parsing (SSP) is the SLU task that involves transforming a recording to a machine-comprehensible parse tree ~\cite{wang2023treepiece}. End-to-end models ~\cite{arora2023study} operate directly on speech while cascade models~\cite{futami2023pipeline} generate a semantic parse based on the transcript. Two-pass deliberation models ~\cite{le2022deliberation} combine the best of both worlds, by using first-pass transcripts and speech embeddings to improve spoken semantic parsing. However, training such models with supervision requires matched triplets of speech, transcript, and semantic parse. Annotating these triplets is expensive, which limits the size of training data, and consequently model performance.&#10;&#10;The need for matched data can be alleviated by developing methods that can use only text data. Text data (transcript-semantic parse) is more easily obtained than speech -- either from existing textual corpora or by prompting Large Language Models (LLMs), and training models with a small amount of paired speech-text data and a large amount of unpaired text is useful. It is non-trivial to incorporate text-only data into end-to-end models because model outputs cannot be obtained without speech inputs. Prior work has explored the use of text data for speech recognition ~\cite{wang2020,toshniwal2018comparison,hori2019cycle}. External language models trained on text can be used to interpolate token prediction probabilities ~\cite{meng22_interspeech}, but require additional memory, making them unsuitable for on-device applications. Coordinated learning methods ~\cite{chen22r_interspeech,sainath2023joist} project speech and text to a shared embedding space for speech recognition, but such models require significant amounts of paired speech-text data to learn robust mappings. The final class of work generates speech representations for unpaired speech - Joint Audio Text (JAT)~\cite{kim2022joint} uses mean speech embeddings from paired data to represent unpaired text. This is computationally inexpensive, but the speech embeddings do not contain information embedded in real speech. In contrast, synthetic speech from Text-to-speech (TTS) models~\cite{wang2020} produce informative speech representations, but they can be expensive to compute. There are two cases where additional textual data may be acquired for semantic parsing -- (a) to improve models on existing domains (ED) and (b) to support new domains (ND). In this paper, we compare JAT and TTS for SSP when unpaired text data is drawn from existing and new domains. &#10;&#10;When unpaired text is not available from existing corpora, we propose prompting Large Language Models (LLMs) ~\cite{ouyang2022training,touvron2023llama,touvron2023llama2} to generate text data for SSP. LLMs are exceptional at generating realistic text based on input prompts, and, in this paper, we use LLama 2.0~\cite{touvron2023llama2} to generate text data. For the ED setup, it is sufficient to generate transcripts since semantic parses can be obtained from transcripts using pre-trained semantic parsers. We describe two prompting methods: (a) intent-word-based prompting (IWP), where the LLM produces transcripts corresponding to a particular intent class and containing words that co-occur with the intent, and (b) exemplar-based prompting (EP), where it generates transcripts that are similar to provided examples. We generate pseudo-labels for the generated utterances using a pre-trained RoBERTa~\cite{liu2020roberta} model and train SSP models using JAT. We find that EP is simpler but IWP generates the desired intent more often. Using data from both methods improves the Exact Match (EM) on STOP data by 1.4 points absolute. &#10;&#10;For the ND setup, pre-trained models for pseudo-labeling are unavailable for the new domain(s), and hence LLMs are used to generate semantic parses directly. The transcript is then inferred from the semantic parse. Exemplar-based prompting (EP) is used with 3 real examples for every possible intent-slot combination to generate large-scale data. We find that the generated data improves EM by 2.3 points absolute over a baseline that uses only 3 examples per combination. &#10;&#10;In summary, this paper makes the following contributions: &#10;\begin{enumerate}&#10; \item Extends JAT, previously used for ASR, to end-to-end spoken semantic parsing, and compares JAT with TTS for textual data from existing domains and new domains. &#10; \item Develops prompting strategies to generate textual transcripts and semantic parses in existing and new domains using LLMs. &#10; \item Demonstrates that LLM-generated textual data can be used in conjunction with JAT and TTS to improve spoken semantic parsing.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Text-to-Speech Synthesis, Linguistics, Spoken Semantic Parsing, Artificial Intelligence, Speech Recognition, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2303.17161" label="2303.17161">
        <attvalues>
          <attvalue for="0" value="TreePiece: Faster Semantic Parsing via Tree Tokenization" />
          <attvalue for="1" value="  Autoregressive (AR) encoder-decoder neural networks have proved successful in&#10;many NLP problems, including Semantic Parsing -- a task that translates natural&#10;language to machine-readable parse trees. However, the sequential prediction&#10;process of AR models can be slow. To accelerate AR for semantic parsing, we&#10;introduce a new technique called TreePiece that tokenizes a parse tree into&#10;subtrees and generates one subtree per decoding step. On TopV2 benchmark,&#10;TreePiece shows 4.6 times faster decoding speed than standard AR, and&#10;comparable speed but significantly higher accuracy compared to&#10;Non-Autoregressive (NAR).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.01620" label="2305.01620">
        <attvalues>
          <attvalue for="0" value="A Study on the Integration of Pipeline and E2E SLU systems for Spoken&#10;  Semantic Parsing toward STOP Quality Challenge" />
          <attvalue for="1" value="  Recently there have been efforts to introduce new benchmark tasks for spoken&#10;language understanding (SLU), like semantic parsing. In this paper, we describe&#10;our proposed spoken semantic parsing system for the quality track (Track 1) in&#10;Spoken Language Understanding Grand Challenge which is part of ICASSP Signal&#10;Processing Grand Challenge 2023. We experiment with both end-to-end and&#10;pipeline systems for this task. Strong automatic speech recognition (ASR)&#10;models like Whisper and pretrained Language models (LM) like BART are utilized&#10;inside our SLU framework to boost performance. We also investigate the output&#10;level combination of various models to get an exact match accuracy of 80.8,&#10;which won the 1st place at the challenge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.01194" label="2305.01194">
        <attvalues>
          <attvalue for="0" value="The Pipeline System of ASR and NLU with MLM-based Data Augmentation&#10;  toward STOP Low-resource Challenge" />
          <attvalue for="1" value="  This paper describes our system for the low-resource domain adaptation track&#10;(Track 3) in Spoken Language Understanding Grand Challenge, which is a part of&#10;ICASSP Signal Processing Grand Challenge 2023. In the track, we adopt a&#10;pipeline approach of ASR and NLU. For ASR, we fine-tune Whisper for each domain&#10;with upsampling. For NLU, we fine-tune BART on all the Track3 data and then on&#10;low-resource domain data. We apply masked LM (MLM) -based data augmentation,&#10;where some of input tokens and corresponding target labels are replaced using&#10;MLM. We also apply a retrieval-based approach, where model input is augmented&#10;with similar training samples. As a result, we achieved exact match (EM)&#10;accuracy 63.3/75.0 (average: 69.15) for reminder/weather domain, and won the&#10;1st place at the challenge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.01893" label="2204.01893">
        <attvalues>
          <attvalue for="0" value="Deliberation Model for On-Device Spoken Language Understanding" />
          <attvalue for="1" value="  We propose a novel deliberation-based approach to end-to-end (E2E) spoken&#10;language understanding (SLU), where a streaming automatic speech recognition&#10;(ASR) model produces the first-pass hypothesis and a second-pass natural&#10;language understanding (NLU) component generates the semantic parse by&#10;conditioning on both ASR's text and audio embeddings. By formulating E2E SLU as&#10;a generalized decoder, our system is able to support complex compositional&#10;semantic structures. Furthermore, the sharing of parameters between ASR and NLU&#10;makes the system especially suitable for resource-constrained (on-device)&#10;environments; our proposed approach consistently outperforms strong pipeline&#10;NLU baselines by 0.60% to 0.65% on the spoken version of the TOPv2 dataset&#10;(STOP). We demonstrate that the fusion of text and audio features, coupled with&#10;the system's ability to rewrite the first-pass hypothesis, makes our approach&#10;more robust to ASR errors. Finally, we show that our approach can significantly&#10;reduce the degradation when moving from natural speech to synthetic speech&#10;training, but more work is required to make text-to-speech (TTS) a viable&#10;solution for scaling up E2E SLU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.10857" label="1807.10857">
        <attvalues>
          <attvalue for="0" value="A Comparison of Techniques for Language Model Integration in&#10;  Encoder-Decoder Speech Recognition" />
          <attvalue for="1" value="  Attention-based recurrent neural encoder-decoder models present an elegant&#10;solution to the automatic speech recognition problem. This approach folds the&#10;acoustic model, pronunciation model, and language model into a single network&#10;and requires only a parallel corpus of speech and text for training. However,&#10;unlike in conventional approaches that combine separate acoustic and language&#10;models, it is not clear how to use additional (unpaired) text. While there has&#10;been previous work on methods addressing this problem, a thorough comparison&#10;among methods is still lacking. In this paper, we compare a suite of past&#10;methods and some of our own proposed methods for using unpaired text data to&#10;improve encoder-decoder models. For evaluation, we use the medium-sized&#10;Switchboard data set and the large-scale Google voice search and dictation data&#10;sets. Our results confirm the benefits of using unpaired text across a range of&#10;methods and data sets. Surprisingly, for first-pass decoding, the rather simple&#10;approach of shallow fusion performs best across data sets. However, for Google&#10;data sets we find that cold fusion has a lower oracle error rate and&#10;outperforms other approaches after second-pass rescoring on the Google voice&#10;search data set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.05354" label="2110.05354">
        <attvalues>
          <attvalue for="0" value="Internal Language Model Adaptation with Text-Only Data for End-to-End&#10;  Speech Recognition" />
          <attvalue for="1" value="  Text-only adaptation of an end-to-end (E2E) model remains a challenging task&#10;for automatic speech recognition (ASR). Language model (LM) fusion-based&#10;approaches require an additional external LM during inference, significantly&#10;increasing the computation cost. To overcome this, we propose an internal LM&#10;adaptation (ILMA) of the E2E model using text-only data. Trained with&#10;audio-transcript pairs, an E2E model implicitly learns an internal LM that&#10;characterizes the token sequence probability which is approximated by the E2E&#10;model output after zeroing out the encoder contribution. During ILMA, we&#10;fine-tune the internal LM, i.e., the E2E components excluding the encoder, to&#10;minimize a cross-entropy loss. To make ILMA effective, it is essential to train&#10;the E2E model with an internal LM loss besides the standard E2E loss.&#10;Furthermore, we propose to regularize ILMA by minimizing the Kullback-Leibler&#10;divergence between the output distributions of the adapted and unadapted&#10;internal LMs. ILMA is the most effective when we update only the last linear&#10;layer of the joint network. ILMA enables a fast text-only adaptation of the E2E&#10;model without increasing the run-time computational cost. Experimented with&#10;30K-hour trained transformer transducer models, ILMA achieves up to 34.9%&#10;relative word error rate reduction from the unadapted baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.03409" label="2204.03409">
        <attvalues>
          <attvalue for="0" value="MAESTRO: Matched Speech Text Representations through Modality Matching" />
          <attvalue for="1" value="  We present Maestro, a self-supervised training method to unify&#10;representations learnt from speech and text modalities. Self-supervised&#10;learning from speech signals aims to learn the latent structure inherent in the&#10;signal, while self-supervised learning from text attempts to capture lexical&#10;information. Learning aligned representations from unpaired speech and text&#10;sequences is a challenging task. Previous work either implicitly enforced the&#10;representations learnt from these two modalities to be aligned in the latent&#10;space through multitasking and parameter sharing or explicitly through&#10;conversion of modalities via speech synthesis. While the former suffers from&#10;interference between the two modalities, the latter introduces additional&#10;complexity. In this paper, we propose Maestro, a novel algorithm to learn&#10;unified representations from both these modalities simultaneously that can&#10;transfer to diverse downstream tasks such as Automated Speech Recognition (ASR)&#10;and Speech Translation (ST). Maestro learns unified representations through&#10;sequence alignment, duration prediction and matching embeddings in the learned&#10;space through an aligned masked-language model loss. We establish a new&#10;state-of-the-art (SOTA) on VoxPopuli multilingual ASR with a 8% relative&#10;reduction in Word Error Rate (WER), multidomain SpeechStew ASR (3.7% relative)&#10;and 21 languages to English multilingual ST on CoVoST 2 with an improvement of&#10;2.8 BLEU averaged over 21 languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.07353" label="2210.07353">
        <attvalues>
          <attvalue for="0" value="JOIST: A Joint Speech and Text Streaming Model For ASR" />
          <attvalue for="1" value="  We present JOIST, an algorithm to train a streaming, cascaded, encoder&#10;end-to-end (E2E) model with both speech-text paired inputs, and text-only&#10;unpaired inputs. Unlike previous works, we explore joint training with both&#10;modalities, rather than pre-training and fine-tuning. In addition, we explore&#10;JOIST using a streaming E2E model with an order of magnitude more data, which&#10;are also novelties compared to previous works. Through a series of ablation&#10;studies, we explore different types of text modeling, including how to model&#10;the length of the text sequence and the appropriate text sub-word unit&#10;representation. We find that best text representation for JOIST improves WER&#10;across a variety of search and rare-word test sets by 4-14% relative, compared&#10;to a model not trained with text. In addition, we quantitatively show that&#10;JOIST maintains streaming capabilities, which is important for good user-level&#10;experience.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.00174" label="2211.00174">
        <attvalues>
          <attvalue for="0" value="Joint Audio/Text Training for Transformer Rescorer of Streaming Speech&#10;  Recognition" />
          <attvalue for="1" value="  Recently, there has been an increasing interest in two-pass streaming&#10;end-to-end speech recognition (ASR) that incorporates a 2nd-pass rescoring&#10;model on top of the conventional 1st-pass streaming ASR model to improve&#10;recognition accuracy while keeping latency low. One of the latest 2nd-pass&#10;rescoring model, Transformer Rescorer, takes the n-best initial outputs and&#10;audio embeddings from the 1st-pass model, and then choose the best output by&#10;re-scoring the n-best initial outputs. However, training this Transformer&#10;Rescorer requires expensive paired audio-text training data because the model&#10;uses audio embeddings as input. In this work, we present our Joint Audio/Text&#10;training method for Transformer Rescorer, to leverage unpaired text-only data&#10;which is relatively cheaper than paired audio-text data. We evaluate&#10;Transformer Rescorer with our Joint Audio/Text training on Librispeech dataset&#10;as well as our large-scale in-house dataset and show that our training method&#10;can improve word error rate (WER) significantly compared to standard&#10;Transformer Rescorer without requiring any extra model parameters or latency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.06804" label="2101.06804">
        <attvalues>
          <attvalue for="0" value="What Makes Good In-Context Examples for GPT-$3$?" />
          <attvalue for="1" value="  GPT-$3$ has attracted lots of attention due to its superior performance&#10;across a wide range of NLP tasks, especially with its powerful and versatile&#10;in-context few-shot learning ability. Despite its success, we found that the&#10;empirical results of GPT-$3$ depend heavily on the choice of in-context&#10;examples. In this work, we investigate whether there are more effective&#10;strategies for judiciously selecting in-context examples (relative to random&#10;sampling) that better leverage GPT-$3$'s few-shot capabilities. Inspired by the&#10;recent success of leveraging a retrieval module to augment large-scale neural&#10;network models, we propose to retrieve examples that are semantically-similar&#10;to a test sample to formulate its corresponding prompt. Intuitively, the&#10;in-context examples selected with such a strategy may serve as more informative&#10;inputs to unleash GPT-$3$'s extensive knowledge. We evaluate the proposed&#10;approach on several natural language understanding and generation benchmarks,&#10;where the retrieval-based prompt selection approach consistently outperforms&#10;the random baseline. Moreover, it is observed that the sentence encoders&#10;fine-tuned on task-related datasets yield even more helpful retrieval results.&#10;Notably, significant gains are observed on tasks such as table-to-text&#10;generation (41.9% on the ToTTo dataset) and open-domain question answering&#10;(45.5% on the NQ dataset). We hope our investigation could help understand the&#10;behaviors of GPT-$3$ and large-scale pre-trained LMs in general and enhance&#10;their few-shot capabilities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-trained Language Models&#10;NLP systems have made tremendous progress by pre-training models on unlabeled text.&#10;For text classification tasks, notable models include BERT~\cite{devlin2018bert}, RoBERTa~\cite{liu2019roberta}, and XLNet~\cite{yang2019xlnet}.&#10;For text generation tasks, notable models include BART~\cite{lewis2019bart}, T5~\cite{raffel2019exploring}, mT5~\cite{xue2020mt5}, XLM~\cite{lample2019cross}, GPT~\cite{radford2018improving}, and GPT-2~\cite{radford2019language}.&#10;These models encapsulate rich information to facilitate a wide range of downstream tasks ranging from natural language understanding to generation.&#10;These models can be adapted to many different tasks via fine-tuning.&#10;GPT-$3$~\cite{brown2020language}, however, can be adapted to many downstream tasks without fine-tuning.&#10;Given just a few in-context examples, GPT-$3$ is able to quickly pick up patterns and produce answers analogously both in terms of the answer style and content.&#10;Thus, GPT-$3$ may be considered as a pattern recognizer to perform in-context learning.&#10;People have just started trying to understand GPT-$3$ from different perspectives.&#10;As mentioned in the introduction, ~\cite{hendrycks2020measuring} studies which categories of questions GPT-$3$ is more capable of answering.&#10;Our work focuses on how to choose good in-context examples.&#10;&#10;Retrieval-based Text Generation&#10;There is a long history of applying information retrieval in text generation~\cite{sumita1991experiments}.&#10;It is very related to the exemplar-based learning~\cite{jakel2008generalization, ziyadi2020example}.&#10;The central idea is to treat retrieved samples as exemplars/prototypes and perform some editings on them.&#10;Some representative applications in the field of deep learning include machine translation~\cite{gu2018search}, sentiment transfer~\cite{li2018delete, guu2018generating}, QA~\cite{karpukhin2020dense, mao2020generation}, dialogue generation~\cite{yan2016learning, cai2018skeleton, song2016two, pandey2018exemplar, weston2018retrieve, wu2019response}, text summarization~\cite{cao2017faithful, peng2019text}, data-to-text generation~\cite{peng2019text}, and text-to-code generation~\cite{hashimoto2018retrieve}.&#10;However, all these retrieve-and-edit frameworks require their decoders to be trained from scratch.&#10;This makes the editor network task- and data-specific.&#10;In contrast, GPT-$3$ in one perspective can be regarded naturally as a universal editor, adaptive to a wide range of tasks.&#10;Our work uniquely examines how to maximize the advantage of using GPT-$3$ without fine-tuning.&#10;For example, the more semantically similar context we provide to GPT-$3$, the better results the model can generate.&#10;Other editors or generators do not have this ability.&#10;&#10;Improve NLP Systems with $k$NN&#10;A recent line of work tries to incorporate nonparametric methods to improve a given model's performance.&#10;These methods first access the test sample's hidden representation and look for the nearest neighbors of this test sample in the database.&#10;Once the nearest neighbors are found, their labels are used to augment the model's prediction.&#10;For example, the newly introduced $k$NN-LM~\cite{khandelwal2019generalization}, $k$NN-MT~\cite{khandelwal2020nearest}, and BERT-$k$NN~\cite{kassner2020bert} generate the next token by retrieving the nearest $k$ neighbors from the datastore.&#10;Another related work is $k$NN classification model~\cite{rajani2020explaining}, where they use $k$NN as backoff when the confidence is low from the fine-tuned classification model.&#10;There are two key differences between our work and other approaches.&#10;First, other approaches modifies the model's next token distribution using the nearest $k$ neighbors.&#10;However, we only changes the conditional text using the nearest $k$ neighbors.&#10;Second, other approaches can access the model's parameters and embeddings which we do not have access to.&#10;Instead, we use some other independently pre-trained models to get the sentence embeddings to retrieve nearest $k$ neighbors.&#10;&#10;" />
          <attvalue for="4" value="Few-Shot Learning, Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2407.04307" label="2407.04307">
        <attvalues>
          <attvalue for="0" value="Crafting Large Language Models for Enhanced Interpretability" />
          <attvalue for="1" value="  We introduce the Concept Bottleneck Large Language Model (CB-LLM), a&#10;pioneering approach to creating inherently interpretable Large Language Models&#10;(LLMs). Unlike traditional black-box LLMs that rely on post-hoc interpretation&#10;methods with limited neuron function insights, CB-LLM sets a new standard with&#10;its built-in interpretability, scalability, and ability to provide clear,&#10;accurate explanations. This innovation not only advances transparency in&#10;language models but also enhances their effectiveness. Our unique Automatic&#10;Concept Correction (ACC) strategy successfully narrows the performance gap with&#10;conventional black-box LLMs, positioning CB-LLM as a model that combines the&#10;high accuracy of traditional LLMs with the added benefit of clear&#10;interpretability -- a feature markedly absent in existing LLMs.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs), such as BERT \cite{bert} and GPT3 \cite{gpt3}, have become instrumental in advancing Natural Language Processing (NLP) tasks. However, the inherent opacity of these models poses significant challenges in ensuring their reliability, particularly when outcomes are based on unclear or flawed reasoning. This lack of transparency complicates the effort to debug and improve these models. &#10;&#10;Recent efforts in the field have primarily focused on post-hoc interpretations of neurons within LLMs \cite{openai, neuronsnlp, indivneurons}. Given a learned LLM, these studies aim to elucidate the inner workings of black-box language models by finding post-hoc explanations for neurons \cite{openai, leenlpexp, neuronsnlp, indivneurons}. Nevertheless, the explanations derived from these methods often do not accurately align with the activation behaviors of the neurons. Moreover, they often fall short in offering clear directions for model editing or debugging, thereby limiting their practical application in correcting outputs. &#10;&#10;Motivated by these limitations, we propose the Concept Bottleneck Large Language Model (CB-LLM) -- the first concept bottleneck model (CBM) for NLP tasks. Our method can transform any pretrained language model into a CBM with an inherently interpretable concept bottleneck layer and a prediction layer. Our contributions are as follows:&#10;\begin{itemize}&#10; \item We present the first CBM framework for LLMs that scales to large text classification benchmarks. Our CB-LLM encapsulates the best of both worlds: it matches the high accuracy of traditional black-box models across multiple datasets while also offering clear interpretability, a feature absent in existing LLMs.&#10; \item Our proposed pipeline to build CB-LLM is fully automatic and efficient: it eliminates the need for human-annotated concept labels, and the computational cost is almost the same as the standard fine-tuning. Furthermore, our proposed Automatic Concept Correction (ACC) strategy efficiently boosts the performance of our CB-LLM in terms of both accuracy and faithfulness evaluation.&#10; \item Our CB-LLM matches the accuracy of the standard black-box models and achieves a $1.39\times$ higher average rating compared to the random baseline on the faithfulness evaluation. This suggests that our CB-LLM provides high-quality interpretability without sacrificing performance. &#10; \end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Interpretability, Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2104.05904" label="2104.05904">
        <attvalues>
          <attvalue for="0" value="DirectProbe: Studying Representations without Classifiers" />
          <attvalue for="1" value="  Understanding how linguistic structures are encoded in contextualized&#10;embedding could help explain their impressive performance across NLP@. Existing&#10;approaches for probing them usually call for training classifiers and use the&#10;accuracy, mutual information, or complexity as a proxy for the representation's&#10;goodness. In this work, we argue that doing so can be unreliable because&#10;different representations may need different classifiers. We develop a&#10;heuristic, DirectProbe, that directly studies the geometry of a representation&#10;by building upon the notion of a version space for a task. Experiments with&#10;several linguistic tasks and contextualized embeddings show that, even without&#10;training classifiers, DirectProbe can shine light into how an embedding space&#10;represents labels, and also anticipate classifier performance for the&#10;representation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Distributed representations of&#10;words~\cite[e.g.,][]{peters-etal-2018-deep,devlin-etal-2019-bert}&#10;have propelled the state-of-the-art across NLP to new heights.&#10;Recently, there is much interest in probing &#10;these opaque representations to understand the information they bear~\cite[e.g.,][]{kovaleva-etal-2019-revealing,conneau-etal-2018-cram,jawahar-etal-2019-bert}.&#10;The most commonly used&#10;strategy calls for training classifiers on them to predict linguistic&#10;properties such as syntax, or cognitive skills like&#10;numeracy~\cite[\eg][]{kassner-schutze-2020-negated,perone2018evaluation,yaghoobzadeh-etal-2019-probing,krasnowska-kieras-wroblewska-2019-empirical,wallace-etal-2019-nlp,pruksachatkun-etal-2020-intermediate}.&#10;Using these classifiers, criteria such as &#10;accuracy or model complexity are used to evaluate the&#10;representation quality for the&#10;task~\cite[\eg][]{goodwin-etal-2020-probing,pimentel-etal-2020-pareto,michael2020asking}.&#10;&#10;Such classifier-based probes are undoubtedly useful to estimate&#10;a representation's quality for a task.&#10;However, their ability to reveal the information in a representation is &#10;occluded by numerous factors, such as the choice of the optimizer and&#10;the initialization used to train the classifiers.&#10;For example, in our experiments using the task of&#10;preposition supersense&#10;prediction~\cite{schneider-etal-2018-comprehensive}, we&#10;found that the accuracies across different training runs of&#10;the same classifier can vary by as much as $\sim 8\%$!&#10;(Detailed results can be found in \ref{sec:cls}.)&#10;&#10;Indeed, the very choice of a classifier influences our estimate of the quality&#10;of a representation. For example, one representation may achieve the best classification accuracy with a linear model, whereas another&#10;may demand a multi-layer perceptron for its&#10;non-linear decision boundaries.&#10;Of course, enumerating every possible classifier for a task is untenable.&#10;A common compromise involves using linear&#10;classifiers to probe representations&#10;~\cite{alain2016understanding,kulmizev-etal-2020-neural}, but doing so may&#10;mischaracterize representations that need non-linear separators.&#10;Some work recognizes this problem~\cite{hewitt-liang-2019-designing} and proposes to report&#10;probing results for at least logistic regression and&#10;a multi-layer perceptron~\cite{eger-etal-2019-pitfalls}, or to compare the learning&#10;curves between multiple controls~\cite{talmor-etal-2020-olmpics}.&#10;However, the success of these methods still depends on&#10;the choices of classifiers.&#10;&#10;In this paper, we pose the question: Can we evaluate the quality of a&#10; representation for an NLP task directly without relying on classifiers as a&#10; proxy?&#10;&#10;Our approach is driven by a characterization of not one, but all decision&#10;boundaries in a representation that are consistent with a training set for a&#10;task. This set of consistent (or approximately consistent) classifiers&#10;constitutes the version space for the task~\cite{mitchell1982generalization}, and&#10;includes both simple (\eg, linear) and complex (\eg, non-linear) classifiers for&#10;the task.&#10;However, perfectly characterizing the version space for a problem presents&#10;computational challenges. To develop an approximation, we note that any decision&#10;boundary partitions the underlying feature space into contiguous regions&#10;associated with labels. We present a heuristic approach&#10;called {\sc DirectProbe}, which builds upon hierarchical&#10;clustering to identify such regions for a given task and embedding.&#10;&#10;The resulting partitions allow us to directly probe the embeddings via their&#10;geometric properties. For example, distances between these regions correlate&#10;with the difficulty of learning with the representation: larger distances&#10;between regions of different labels indicates that there are more consistent&#10;separators between them, and imply easier learning, and better generalization&#10;of classifiers.&#10;Further, by assigning test points to their closest partitions, we have a&#10;parameter-free classifier as a side effect, which can help benchmark&#10;representations without committing to &#10;a specific family of classifiers (\eg, linear) as probes.&#10;&#10;Our experiments study five different NLP tasks that involve syntactic and&#10;semantic phenomena. &#10;We show that our approach allows us to ascertain, without training a classifier,&#10;\begin{inparaenum}[(a)]&#10;\item if a representation admits a linear separator for a dataset,&#10;\item how different layers of BERT differ in their representations for a task,&#10;\item which labels for a task are more confusable,&#10;\item the expected performance of the best classifier for the task, and &#10;\item the impact of fine-tuning.&#10;\end{inparaenum}&#10;&#10;In summary, the contributions of this work are:&#10;\begin{enumerate}\item We point out that training classifiers as probes is not&#10; reliable, and instead, we should directly analyze the structure of&#10; a representation space.&#10;\item We formalize the problem of evaluating representations via the notion of&#10; version spaces and introduce {\sc DirectProbe}, a heuristic method to approximate it&#10; directly which does not involve training classifiers.&#10;\item Via experiments, we show that our approach can help identify how good a&#10; given representation will be for a prediction task.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Linguistic Structure Encoding, Contextualized Embeddings, Representation Analysis, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1908.08593" label="1908.08593">
        <attvalues>
          <attvalue for="0" value="Revealing the Dark Secrets of BERT" />
          <attvalue for="1" value="  BERT-based architectures currently give state-of-the-art performance on many&#10;NLP tasks, but little is known about the exact mechanisms that contribute to&#10;its success. In the current work, we focus on the interpretation of&#10;self-attention, which is one of the fundamental underlying components of BERT.&#10;Using a subset of GLUE tasks and a set of handcrafted features-of-interest, we&#10;propose the methodology and carry out a qualitative and quantitative analysis&#10;of the information encoded by the individual BERT's heads. Our findings suggest&#10;that there is a limited set of attention patterns that are repeated across&#10;different heads, indicating the overall model overparametrization. While&#10;different heads consistently use the same attention patterns, they have varying&#10;impact on performance across different tasks. We show that manually disabling&#10;attention in certain heads leads to a performance improvement over the regular&#10;fine-tuned BERT models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03343" label="1911.03343">
        <attvalues>
          <attvalue for="0" value="Negated and Misprimed Probes for Pretrained Language Models: Birds Can&#10;  Talk, But Cannot Fly" />
          <attvalue for="1" value="  Building on Petroni et al. (2019), we propose two new probing tasks analyzing&#10;factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We&#10;find that PLMs do not distinguish between negated (&quot;Birds cannot [MASK]&quot;) and&#10;non-negated (&quot;Birds can [MASK]&quot;) cloze questions. (2) Mispriming. Inspired by&#10;priming methods in human psychology, we add &quot;misprimes&quot; to cloze questions&#10;(&quot;Talk? Birds can [MASK]&quot;). We find that PLMs are easily distracted by&#10;misprimes. These results suggest that PLMs still have a long way to go to&#10;adequately learn human-like factual knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.06259" label="1806.06259">
        <attvalues>
          <attvalue for="0" value="Evaluation of sentence embeddings in downstream and linguistic probing&#10;  tasks" />
          <attvalue for="1" value="  Despite the fast developmental pace of new sentence embedding methods, it is&#10;still challenging to find comprehensive evaluations of these different&#10;techniques. In the past years, we saw significant improvements in the field of&#10;sentence embeddings and especially towards the development of universal&#10;sentence encoders that could provide inductive transfer to a wide variety of&#10;downstream tasks. In this work, we perform a comprehensive evaluation of recent&#10;methods using a wide variety of downstream and linguistic feature probing&#10;tasks. We show that a simple approach using bag-of-words with a recently&#10;introduced language model for deep context-dependent word embeddings proved to&#10;yield better results in many tasks when compared to sentence encoders trained&#10;on entailment datasets. We also show, however, that we are still far away from&#10;a universal encoder that can perform consistently across several downstream&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.03608" label="1906.03608">
        <attvalues>
          <attvalue for="0" value="Probing for Semantic Classes: Diagnosing the Meaning Content of Word&#10;  Embeddings" />
          <attvalue for="1" value="  Word embeddings typically represent different meanings of a word in a single&#10;conflated vector. Empirical analysis of embeddings of ambiguous words is&#10;currently limited by the small size of manually annotated resources and by the&#10;fact that word senses are treated as unrelated individual concepts. We present&#10;a large dataset based on manual Wikipedia annotations and word senses, where&#10;word senses from different words are related by semantic classes. This is the&#10;basis for novel diagnostic tests for an embedding's content: we probe word&#10;embeddings for semantic classes and analyze the embedding space by classifying&#10;embeddings into semantic classes. Our main findings are: (i) Information about&#10;a sense is generally represented well in a single-vector embedding - if the&#10;sense is frequent. (ii) A classifier can accurately predict whether a word is&#10;single-sense or multi-sense, based only on its embedding. (iii) Although rare&#10;senses are not well represented in single-vector embeddings, this does not have&#10;negative impact on an NLP application whose performance depends on frequent&#10;senses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.07940" label="1909.07940">
        <attvalues>
          <attvalue for="0" value="Do NLP Models Know Numbers? Probing Numeracy in Embeddings" />
          <attvalue for="1" value="  The ability to understand and work with numbers (numeracy) is critical for&#10;many complex reasoning tasks. Currently, most NLP models treat numbers in text&#10;in the same way as other tokens---they embed them as distributed vectors. Is&#10;this enough to capture numeracy? We begin by investigating the numerical&#10;reasoning capabilities of a state-of-the-art question answering model on the&#10;DROP dataset. We find this model excels on questions that require numerical&#10;reasoning, i.e., it already captures numeracy. To understand how this&#10;capability emerges, we probe token embedding methods (e.g., BERT, GloVe) on&#10;synthetic list maximum, number decoding, and addition tasks. A surprising&#10;degree of numeracy is naturally present in standard embeddings. For example,&#10;GloVe and word2vec accurately encode magnitude for numbers up to 1,000.&#10;Furthermore, character-level embeddings are even more precise---ELMo captures&#10;numeracy the best for all pre-trained methods---but BERT, which uses sub-word&#10;units, is less exact.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.04315" label="2005.04315">
        <attvalues>
          <attvalue for="0" value="Probing Linguistic Systematicity" />
          <attvalue for="1" value="  Recently, there has been much interest in the question of whether deep&#10;natural language understanding models exhibit systematicity; generalizing such&#10;that units like words make consistent contributions to the meaning of the&#10;sentences in which they appear. There is accumulating evidence that neural&#10;models often generalize non-systematically. We examined the notion of&#10;systematicity from a linguistic perspective, defining a set of probes and a set&#10;of metrics to measure systematic behaviour. We also identified ways in which&#10;network architectures can generalize non-systematically, and discuss why such&#10;forms of generalization may be unsatisfying. As a case study, we performed a&#10;series of experiments in the setting of natural language inference (NLI),&#10;demonstrating that some NLU systems achieve high overall performance despite&#10;being non-systematic.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.02180" label="2010.02180">
        <attvalues>
          <attvalue for="0" value="Pareto Probing: Trading Off Accuracy for Complexity" />
          <attvalue for="1" value="  The question of how to probe contextual word representations for linguistic&#10;structure in a way that is both principled and useful has seen significant&#10;attention recently in the NLP literature. In our contribution to this&#10;discussion, we argue for a probe metric that reflects the fundamental trade-off&#10;between probe complexity and performance: the Pareto hypervolume. To measure&#10;complexity, we present a number of parametric and non-parametric metrics. Our&#10;experiments using Pareto hypervolume as an evaluation metric show that probes&#10;often do not conform to our expectations -- e.g., why should the non-contextual&#10;fastText representations encode more morpho-syntactic information than the&#10;contextual BERT representations? These results suggest that common, simplistic&#10;probing tasks, such as part-of-speech labeling and dependency arc labeling, are&#10;inadequate to evaluate the linguistic structure encoded in contextual word&#10;representations. This leads us to propose full dependency parsing as a probing&#10;task. In support of our suggestion that harder probing tasks are necessary, our&#10;experiments with dependency parsing reveal a wide gap in syntactic knowledge&#10;between contextual and non-contextual representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14513" label="2004.14513">
        <attvalues>
          <attvalue for="0" value="Asking without Telling: Exploring Latent Ontologies in Contextual&#10;  Representations" />
          <attvalue for="1" value="  The success of pretrained contextual encoders, such as ELMo and BERT, has&#10;brought a great deal of interest in what these models learn: do they, without&#10;explicit supervision, learn to encode meaningful notions of linguistic&#10;structure? If so, how is this structure encoded? To investigate this, we&#10;introduce latent subclass learning (LSL): a modification to existing&#10;classifier-based probing methods that induces a latent categorization (or&#10;ontology) of the probe's inputs. Without access to fine-grained gold labels,&#10;LSL extracts emergent structure from input representations in an interpretable&#10;and quantifiable form. In experiments, we find strong evidence of familiar&#10;categories, such as a notion of personhood in ELMo, as well as novel&#10;ontological distinctions, such as a preference for fine-grained semantic roles&#10;on core arguments. Our results provide unique new evidence of emergent&#10;structure in pretrained encoders, including departures from existing&#10;annotations which are inaccessible to earlier methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.04905" label="1805.04905">
        <attvalues>
          <attvalue for="0" value="Comprehensive Supersense Disambiguation of English Prepositions and&#10;  Possessives" />
          <attvalue for="1" value="  Semantic relations are often signaled with prepositional or possessive&#10;marking--but extreme polysemy bedevils their analysis and automatic&#10;interpretation. We introduce a new annotation scheme, corpus, and task for the&#10;disambiguation of prepositions and possessives in English. Unlike previous&#10;approaches, our annotations are comprehensive with respect to types and tokens&#10;of these markers; use broadly applicable supersense classes rather than&#10;fine-grained dictionary definitions; unite prepositions and possessives under&#10;the same class inventory; and distinguish between a marker's lexical&#10;contribution and the role it marks in the context of a predicate or scene.&#10;Strong interannotator agreement rates, as well as encouraging disambiguation&#10;results with established supervised methods, speak to the viability of the&#10;scheme and task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14096" label="2004.14096">
        <attvalues>
          <attvalue for="0" value="Do Neural Language Models Show Preferences for Syntactic Formalisms?" />
          <attvalue for="1" value="  Recent work on the interpretability of deep neural language models has&#10;concluded that many properties of natural language syntax are encoded in their&#10;representational spaces. However, such studies often suffer from limited scope&#10;by focusing on a single language and a single linguistic formalism. In this&#10;study, we aim to investigate the extent to which the semblance of syntactic&#10;structure captured by language models adheres to a surface-syntactic or deep&#10;syntactic style of analysis, and whether the patterns are consistent across&#10;different languages. We apply a probe for extracting directed dependency trees&#10;to BERT and ELMo models trained on 13 different languages, probing for two&#10;different syntactic annotation styles: Universal Dependencies (UD),&#10;prioritizing deep syntactic relations, and Surface-Syntactic Universal&#10;Dependencies (SUD), focusing on surface structure. We find that both models&#10;exhibit a preference for UD over SUD - with interesting variations across&#10;languages and layers - and that the strength of this preference is correlated&#10;with differences in tree shape.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01575" label="1906.01575">
        <attvalues>
          <attvalue for="0" value="Pitfalls in the Evaluation of Sentence Embeddings" />
          <attvalue for="1" value="  Deep learning models continuously break new records across different NLP&#10;tasks. At the same time, their success exposes weaknesses of model evaluation.&#10;Here, we compile several key pitfalls of evaluation of sentence embeddings, a&#10;currently very popular NLP paradigm. These pitfalls include the comparison of&#10;embeddings of different sizes, normalization of embeddings, and the low (and&#10;diverging) correlations between transfer and probing tasks. Our motivation is&#10;to challenge the current evaluation of sentence embeddings and to provide an&#10;easy-to-access reference for future research. Based on our insights, we also&#10;recommend better practices for better future evaluations of sentence&#10;embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08645" label="2104.08645">
        <attvalues>
          <attvalue for="0" value="Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training" />
          <attvalue for="1" value="  Pre-trained multilingual language encoders, such as multilingual BERT and&#10;XLM-R, show great potential for zero-shot cross-lingual transfer. However,&#10;these multilingual encoders do not precisely align words and phrases across&#10;languages. Especially, learning alignments in the multilingual embedding space&#10;usually requires sentence-level or word-level parallel corpora, which are&#10;expensive to be obtained for low-resource languages. An alternative is to make&#10;the multilingual encoders more robust; when fine-tuning the encoder using&#10;downstream task, we train the encoder to tolerate noise in the contextual&#10;embedding spaces such that even if the representations of different languages&#10;are not aligned well, the model can still achieve good performance on zero-shot&#10;cross-lingual transfer. In this work, we propose a learning strategy for&#10;training robust models by drawing connections between adversarial examples and&#10;the failure cases of zero-shot cross-lingual transfer. We adopt two widely used&#10;robust training methods, adversarial training and randomized smoothing, to&#10;train the desired robust model. The experimental results demonstrate that&#10;robust training improves zero-shot cross-lingual transfer on text&#10;classification tasks. The improvement is more significant in the generalized&#10;cross-lingual transfer setting, where the pair of input sentences belong to two&#10;different languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Zero-shot cross-lingual transfer learning.&#10;In recent years, several pre-trained multilingual language models are proposed for zero-shot cross-lingual transfer, including multilingual BERT \cite{Devlin19bert}, XLM \cite{ConneauL19xlm}, and XLM-R \cite{Conneau20xlmr,Goyal21xlmrxl}. Many studies put attentions on the rationales that make zero-shot cross-lingual transfer work \cite{K20mberts1,Lauscher20mberts2,Conneau20mberts3,Artetxe20mberts4,Dufter20mberts5}. &#10;Various tasks and datasests are presented to facilitate zero-shot cross-lingual transfer learning \cite{Conneau18xnli,Yang19pawsx,Clark20tydi,Artetxe20mberts4,Lewis20mlqa}. XTREME \cite{Hu20xtreme} and XGLUE \cite{Liang20xglue} further provide benchmarks for zero-shot cross-lingual transfer learning.&#10;&#10;Embedding space alignments.&#10;Learning to align embedding spaces have always been an important research topic to improve multilinguality. Early works focus on word embedding spaces \cite{Mikolov13word1,Smith17word2,Artetxe17word3}. Recently, many approaches are proposed to align contextual word embedding spaces, such as learning rotation projections \cite{Schuster19rotate1,Aldarmaki19rotate2,Conneau20mberts3} and fine-tuning pre-trained multilingual language models \cite{Chi20align4,Feng20align5,Cao20align1,Qin20codesw1,Liu20codesw2,Dou21align3,Wei21align6}. However, most of them require additional supervision signals, such as parallel sentence pairs \cite{Chi20align4,Feng20align5,Wei21align6}, bilingual dictionary \cite{Cao20align1,Qin20codesw1,Liu20codesw2}, or both \cite{Pan21align2}. These additional supervised corpora are usually expensive for low-resource languages.&#10;&#10;Embedding misalignment handling.&#10;Instead of directly aligning the representations, there is a line of research making the model be aware of the embedding misalignment issues by considering additional syntactic features, such as part-of-speech \cite{Kozhevnikov13pos1} and dependency parse trees \cite{Ahmad19difficult,Subburathinam19clgcn,Zhang19dep1,Liu19dep2,Ahmad21syntaxbert,Ahmad21gate}, and other syntactic features~\cite{meng2019target}.&#10;However, those syntactic features require large human efforts to obtained.&#10;&#10;Robust training.&#10;Recently, adversarial attacks are presented to check the robustness of NLP models, such as character manipulation \cite{Ebrahimi18hotflip,Gil19hotflip2}, word replacements \cite{Alzantot18synonym,Li20attack1,Garg20attack2,Jin20attack3}, and syntactic rearrangements \cite{Iyyer18scpn}. To against those attacks, various robust training methods are proposed. For example, \cite{Alzantot18synonym} trains a robust model by data augmentation with generated adversarial examples. Other works \cite{Ebrahimi18hotflip,Dong20convex,Zhou21dir} consider adversarial training, which includes the adversarial accuracy to the training objective. A few studies propose transformations on inputs before feeding them to models \cite{Edizel19roboenc2,Jones20roboenc}. Randomized smoothing \cite{Cohen19randomized,Ye20safer} is presented to make models robust against noise in input representations. Another line of research aims at providing theoretical guarantee of robustness, including interval bound propagation methods \cite{Jia19certify1,Huang19certify2} and verification methods \cite{Shi20verify}. Most of those robust training methods focus on defending adversarial attacks, while we propose to apply robust training methods to improve the zero-shot cross-lingual transfer performance.&#10; " />
          <attvalue for="4" value="Cross-Lingual Transfer Learning, Machine Learning, Computer Science, Linguistics, Multilingual Language Models, Artificial Intelligence, Natural Language Processing, Adversarial Robust Training" />
        </attvalues>
      </node>
      <node id="1506.06726" label="1506.06726">
        <attvalues>
          <attvalue for="0" value="Skip-Thought Vectors" />
          <attvalue for="1" value="  We describe an approach for unsupervised learning of a generic, distributed&#10;sentence encoder. Using the continuity of text from books, we train an&#10;encoder-decoder model that tries to reconstruct the surrounding sentences of an&#10;encoded passage. Sentences that share semantic and syntactic properties are&#10;thus mapped to similar vector representations. We next introduce a simple&#10;vocabulary expansion method to encode words that were not seen as part of&#10;training, allowing us to expand our vocabulary to a million words. After&#10;training our model, we extract and evaluate our vectors with linear models on 8&#10;tasks: semantic relatedness, paraphrase detection, image-sentence ranking,&#10;question-type classification and 4 benchmark sentiment and subjectivity&#10;datasets. The end result is an off-the-shelf encoder that can produce highly&#10;generic sentence representations that are robust and perform well in practice.&#10;We will make our encoder publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.08994" label="1612.08994">
        <attvalues>
          <attvalue for="0" value="Here's My Point: Joint Pointer Architecture for Argument Mining" />
          <attvalue for="1" value="  One of the major goals in automated argumentation mining is to uncover the&#10;argument structure present in argumentative text. In order to determine this&#10;structure, one must understand how different individual components of the&#10;overall argument are linked. General consensus in this field dictates that the&#10;argument components form a hierarchy of persuasion, which manifests itself in a&#10;tree structure. This work provides the first neural network-based approach to&#10;argumentation mining, focusing on the two tasks of extracting links between&#10;argument components, and classifying types of argument components. In order to&#10;solve this problem, we propose to use a joint model that is based on a Pointer&#10;Network architecture. A Pointer Network is appealing for this task for the&#10;following reasons: 1) It takes into account the sequential nature of argument&#10;components; 2) By construction, it enforces certain properties of the tree&#10;structure present in argument relations; 3) The hidden representations can be&#10;applied to auxiliary tasks. In order to extend the contribution of the original&#10;Pointer Network model, we construct a joint model that simultaneously attempts&#10;to learn the type of argument component, as well as continuing to predict links&#10;between argument components. The proposed joint model achieves state-of-the-art&#10;results on two separate evaluation corpora, achieving far superior performance&#10;than a regular Pointer Network model. Our results show that optimizing for both&#10;tasks, and adding a fully-connected layer prior to recurrent neural network&#10;input, is crucial for high performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.01739" label="2402.01739">
        <attvalues>
          <attvalue for="0" value="OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models" />
          <attvalue for="1" value="  To help the open-source community have a better understanding of&#10;Mixture-of-Experts (MoE) based large language models (LLMs), we train and&#10;release OpenMoE, a series of fully open-sourced and reproducible decoder-only&#10;MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T&#10;tokens. Our investigation confirms that MoE-based LLMs can offer a more&#10;favorable cost-effectiveness trade-off than dense LLMs, highlighting the&#10;potential effectiveness for future LLM development.&#10;  One more important contribution of this study is an in-depth analysis of the&#10;routing mechanisms within our OpenMoE models, leading to three significant&#10;findings: Context-Independent Specialization, Early Routing Learning, and&#10;Drop-towards-the-End. We discovered that routing decisions in MoE models are&#10;predominantly based on token IDs, with minimal context relevance. The&#10;token-to-expert assignments are determined early in the pre-training phase and&#10;remain largely unchanged. This imperfect routing can result in performance&#10;degradation, particularly in sequential tasks like multi-turn conversations,&#10;where tokens appearing later in a sequence are more likely to be dropped.&#10;Finally, we rethink our design based on the above-mentioned observations and&#10;analysis. To facilitate future MoE LLM development, we propose potential&#10;strategies for mitigating the issues we found and further improving&#10;off-the-shelf MoE LLM designs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Large Language Models, Linguistics, Mixture-of-Experts Architecture, Model Optimization Strategies, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2401.16818" label="2401.16818">
        <attvalues>
          <attvalue for="0" value="H2O-Danube-1.8B Technical Report" />
          <attvalue for="1" value="  We present H2O-Danube, a series of small 1.8B language models consisting of&#10;H2O-Danube-1.8B, trained on 1T tokens, and the incremental improved&#10;H2O-Danube2-1.8B trained on an additional 2T tokens. Our models exhibit highly&#10;competitive metrics across a multitude of benchmarks and, as of the time of&#10;this writing, H2O-Danube2-1.8B achieves the top ranking on Open LLM Leaderboard&#10;for all models below the 2B parameter range. The models follow core principles&#10;of LLama 2 and Mistral, and we leverage and refine various techniques for&#10;pre-training large language models. We additionally release chat models trained&#10;with supervised fine-tuning followed by direct preference optimization. We make&#10;all models openly available under Apache 2.0 license further democratizing LLMs&#10;to a wider audience economically.&#10;" />
          <attvalue for="2" value="&#10;&#10;Research over the past few years has significantly enhanced language models' capabilities, making them pivotal in tasks like text and code generation, question answering, translation, summarization, and more \cite{ye2023comprehensive}. Most state-of-the-art large language models (LLMs) leverage decoder attention architectures \cite{vaswani2017attention} popularized by the series of GPT models \cite{radford2018gpt1, radford2019gpt2, brown2020gpt3} exemplifying the benefits of pre-training such models on extensive text corpora. &#10;&#10;Scaling laws for LLMs suggest that performance scales by factors such as model and dataset size, as well as computational resources for training \cite{kaplan2020scaling}. This has led to the development of a plethora of models, ranging in size to optimize performance given certain data and compute constraints; notable representatives are: Falcon \cite{penedo2023refinedweb}, Llama 2 \cite{touvron2023llama}, Qwen \cite{bai2023qwen}, Mistral \cite{jiang2023mistral}, or Mixtral \cite{jiang2024mixtral}.&#10;&#10;Despite the trend towards larger models, smaller LLMs have taking an important place in today's landscape allowing for efficient inference on consumer hardware and edge devices. While larger models often times excel across various generic tasks \cite{touvron2023llama, bai2023qwen, jiang2023mistral}, fine-tuning smaller models for specific tasks can enable competitive performance with benefits of model size and inference speed \cite{fu2023specializing}, a concept also proven by the success of BERT and its derivatives \cite{devlin2018bert, he2020deberta}. &#10;&#10;In this report, we want to extend previous research in this area \cite{biderman2023pythia, zhang2024tinyllama, zhang2022opt, bai2023qwen, stablelm} and present a series of models based on incremental research and training efforts. We release all models with open weights under Apache 2.0. The first part describes the initial H2O-Danube-1.8B model, as trained on $1T$ tokens, and a separation Section~\ref{sec:danube2} describes H2O-Danube2-1.8B, a continued modeling effort trained on additional $2T$ tokens. In order to transparently elaborate our incremental insights, the first part is identical to an earlier version of this report, while the second part highlights new insights of the second iteration.&#10;&#10;Fundamentally, H2O-Danube follows a decoder LLM architecture adopting core principles from Llama 2 \cite{touvron2023llama} and Mistral \cite{jiang2023mistral}.&#10;The models are trained on a combination of, but not limited to, web documents, encyclopedia and public knowledge databases, excluding coding data. H2O-Danube2-1.8B is trained on a a more diverse mix of data over multiple data stages. Compared to recent models released in this parameter range \cite{bai2023qwen, zhang2024tinyllama, stablelm}, our models demonstrate to be highly competitive across various benchmarks. As of this writing, H2O-Danube2-1.8B is the highest ranked open model on the Hugging Face Open LLM Leaderboard for models below the 2B range. Alongside the base modes, we release chat variants, enhanced with supervised fine-tuning on instruction data and preference data optimization (DPO).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Large Scale Training, Model Optimization Techniques, Language Model Development, Mathematics" />
        </attvalues>
      </node>
      <node id="2303.10420" label="2303.10420">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models" />
          <attvalue for="1" value="  GPT series models, such as GPT-3, CodeX, InstructGPT, ChatGPT, and so on,&#10;have gained considerable attention due to their exceptional natural language&#10;processing capabilities. However, despite the abundance of research on the&#10;difference in capabilities between GPT series models and fine-tuned models,&#10;there has been limited attention given to the evolution of GPT series models'&#10;capabilities over time. To conduct a comprehensive analysis of the capabilities&#10;of GPT series models, we select six representative models, comprising two GPT-3&#10;series models (i.e., davinci and text-davinci-001) and four GPT-3.5 series&#10;models (i.e., code-davinci-002, text-davinci-002, text-davinci-003, and&#10;gpt-3.5-turbo). We evaluate their performance on nine natural language&#10;understanding (NLU) tasks using 21 datasets. In particular, we compare the&#10;performance and robustness of different models for each task under zero-shot&#10;and few-shot scenarios. Our extensive experiments reveal that the overall&#10;ability of GPT series models on NLU tasks does not increase gradually as the&#10;models evolve, especially with the introduction of the RLHF training strategy.&#10;While this strategy enhances the models' ability to generate human-like&#10;responses, it also compromises their ability to solve some tasks. Furthermore,&#10;our findings indicate that there is still room for improvement in areas such as&#10;model robustness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.17377" label="2406.17377">
        <attvalues>
          <attvalue for="0" value="A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual&#10;  LLMs" />
          <attvalue for="1" value="  Low-resource languages, by its very definition, tend to be under represented&#10;in the pre-training corpora of Large Language Models. In this work, we&#10;investigate three low-resource cross-lingual approaches that enable an LLM&#10;adapt to tasks in previously unseen languages. Llama-2 is an LLM where Indic&#10;languages, among many other language families, contribute to less than&#10;$0.005\%$ of the total $2$ trillion token pre-training corpora. In this work,&#10;we experiment with the English-dominated Llama-2 for cross-lingual transfer to&#10;three Indic languages, Bengali, Hindi, and Tamil as target languages. We study&#10;three approaches for cross-lingual transfer, under ICL and fine-tuning. One, we&#10;find that adding additional supervisory signals via a dominant language in the&#10;LLM, leads to improvements, both under in-context learning and fine-tuning.&#10;Two, adapting the target languages to word reordering may be beneficial under&#10;ICL, but its impact diminishes with fine tuning. Finally, continued&#10;pre-training in one low-resource language can improve model performance for&#10;other related low-resource languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models \cite[LLM;][]{brown2020language, touvron2023llama, chowdhery2022palm, gemmateam2024gemma} are known to generalise well across several tasks, including in few shot and zero-shot setups. However, there is limited evidence that shows the ability of these models to generalise to tasks in new languages out of the box, especially to those with which the model has limited exposure to. In this work, we investigate how effectively we can leverage the LLMs for cross lingual transfer, especially for adapting it to low-resource languages.&#10;&#10;LLMs typically require tens of billions, if not trillions, of tokens for its pre-training. Now, that is a challenge for majority of the languages in the world. More than $80\%$ of languages in the world are `left behind' \cite{joshi-etal-2020-state}, and barely have enough digitised data that matches the requirements for pre-training an LLM from scratch. For instance, the most populous country in the world, India, speaks more than $400$ languages, with $22$ of them recognised as scheduled languages by the Government of India. However, none of these languages contribute to more than $0.005\%$ of the pre-training data of an open-source LLM like Llama-2 \cite{touvron2023llama}. In fact, more than 95\% of these languages lack enough digital resources to incorporate them into an LLM. These resource-poor languages tend to get poorer in representation with the progress in the field \cite{joshi-etal-2020-state, ojo2024good}.&#10;&#10;Some of the recent works, explore various techniques to adapt an LLM to new languages, especially with limited target language resources \cite{rathore-etal-2023-zgul}. \cite{tanwar2023multilingual} exploit cross-lingual transfer to improve in-context learning (ICL) for binary sequence classification tasks in low-resource languages by utilizing in-context exemplars from a high-resource language semantically similar to the input in the target language. \cite{husain2024romansetu} employ continual pre-training on Llama-2 with romanized pre-training corpora of non-roman script languages, to exploit cross-lingual transfer using the script of English. \cite{awasthi2023bootstrapping} use $540$b PaLM \cite{chowdhery2022palm} to generate training data in low-resource languages using labelled instances in English. \cite{razumovskaia2024analyzing} provide analyses of multilingual capabilities of LLMs on NLU tasks under the settings of in-context learning (ICL), supervised fine-tuning (SFT), and supervised instruction-tuning (SIT). &#10;&#10;Our investigation primarily involves the following three questions, centered around information extraction (IE) tasks in a low-resource language using an instruction-tuned LLM. Q1. Handholding: For an IE task in a low-resource target language, would providing a parallel, annotated sentence in the predominant language of the LLM, help to exploit cross-lingual transfer, resulting in improved performance for the target language. By predominant language, we imply the language that forms the majority of the pre-training corpora. Q2. Masquerading: Would adapting the target language to resemble the predominant language enable in cross-lingual transfer, benefiting the target language. Finally, Q3. Bridging: Whether model adaptation in one of the low-resource languages can benefit other related low-resource languages. More clarity on these questions, is presented in \ref{sec:prelim}.&#10;&#10;We focus on three Indic languages, namely, Bengali, Hindi, and Tamil. These languages are culturally diverse within the Indic context, with Bengali and Hindi belonging to the Indo-Aryan family and Tamil to the Dravidian family. To evaluate our hypotheses Q1, Q2, and Q3, we focus on two information extraction tasks: slot filling and named entity recognition (NER). Further, we use a $7$ billion parameter English-centric LLM Llama-2 as our base LLM, unless otherwise stated. The slot filling and named entity recognition tasks possess label-set size of $55$ and $3$, respectively. Additionally, none of Bengali, Hindi, and Tamil contribute to more than $0.005\%$ of the pre-training corpora of Llama-2. Moreover, English is the predominant language, contributing to roughly $90\%$ of the pre-training corpora.&#10;&#10;In our experiments, we simlulate a low-resource scenario where we do not expect the target language to have more than roughly $10,000$ instances. In Bridging, when Llama-2 is adapted with Hindi through continued pre-training, we use more than 10,000 sentences in Hindi. However, in this case, Hindi is referred to as the bridge language. The evaluation is solely performed on Bengali and Tamil, both of which satisfy aforementioned criteria for the low-resource setting. Our investigation includes exploiting few-shot in-context learning (ICL) ability of Llama-2 as well as model adaptation with parameter-efficient supervised fine-tuning (PEFT). To evaluate Llama-2, or any auto-regressive LLM in general, we frame the tasks of slot filling and named entity recognition as text-to-text generation tasks. \ref{fig:overview} showcases slot filling as a text-to-text generation task. &#10;&#10;Extensive experiments on Llama-2 show that Handholding improves NLU and NLG in Bengali, Hindi and Tamil by exploiting cross-lingual transfer from English, under both few-shot ICL and PEFT. Further, Bridging with Hindi, improves monolingual task performance in related languages of Bengali and Tamil under PEFT. Ultimately, Handholding + Bridging turns out the most beneficial combination, yielding best task performance for both low-resource languages of Bengali and Tamil. A quantitative overview has been presented in \ref{fig:teaser}.&#10;&#10;Our major contributions can be summarized as follows:&#10;\begin{itemize}&#10; \item We demonstrate that the predominant language of an LLM can be leveraged to aid low-resource languages. Specifically, leveraging English via Handholding, improves the overall performance of Llama-2 for information extraction tasks in Hindi, Bengali, and Tamil under both few-shot in-context learning (ICL) and parameter-efficient fine-tuning (PEFT).&#10; \item Improved natural language understanding and generation in Bengali and Tamil, as shown by our experiments with Llama-2 adapted with Hindi (Bridging), demonstrates that adapting a model in one low-resource language can benefit other related languages.&#10; \item Modifying target language via (Masquerading) to resemble the predominant language, English, gives superficial benefits in few-shot ICL and diminishes further in PEFT. &#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cross-Lingual Transfer, Artificial Intelligence, Natural Language Processing, Low-Resource Language Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2004.09095" label="2004.09095">
        <attvalues>
          <attvalue for="0" value="The State and Fate of Linguistic Diversity and Inclusion in the NLP&#10;  World" />
          <attvalue for="1" value="  Language technologies contribute to promoting multilingualism and linguistic&#10;diversity around the world. However, only a very small number of the over 7000&#10;languages of the world are represented in the rapidly evolving language&#10;technologies and applications. In this paper we look at the relation between&#10;the types of languages, resources, and their representation in NLP conferences&#10;to understand the trajectory that different languages have followed over time.&#10;Our quantitative investigation underlines the disparity between languages,&#10;especially in terms of their resources, and calls into question the &quot;language&#10;agnostic&quot; status of current models and systems. Through this paper, we attempt&#10;to convince the ACL community to prioritise the resolution of the predicaments&#10;highlighted here, so that no language is left behind.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.02990" label="2403.02990">
        <attvalues>
          <attvalue for="0" value="Data Augmentation using Large Language Models: Data Perspectives,&#10;  Learning Paradigms and Challenges" />
          <attvalue for="1" value="  In the rapidly evolving field of large language models (LLMs), data&#10;augmentation (DA) has emerged as a pivotal technique for enhancing model&#10;performance by diversifying training examples without the need for additional&#10;data collection. This survey explores the transformative impact of LLMs on DA,&#10;particularly addressing the unique challenges and opportunities they present in&#10;the context of natural language processing (NLP) and beyond. From both data and&#10;learning perspectives, we examine various strategies that utilize LLMs for data&#10;augmentation, including a novel exploration of learning paradigms where&#10;LLM-generated data is used for diverse forms of further training. Additionally,&#10;this paper highlights the primary open challenges faced in this domain, ranging&#10;from controllable data augmentation to multi-modal data augmentation. This&#10;survey highlights a paradigm shift introduced by LLMs in DA, and aims to serve&#10;as a comprehensive guide for researchers and practitioners.&#10;" />
          <attvalue for="2" value="&#10;&#10;Data-centric approaches to Artificial Intelligence (AI) constitute a pivotal element in the advancement towards Artificial General Intelligence (AGI), centering on the construction of AI systems underpinned by high-quality data \cite{zha2023data}. This emphasis on data quality is vital, as it ensures the clarity of the information from which AI systems are to learn. Nevertheless, the acquisition of high-quality data presents significant challenges, being both costly and time-intensive, while the data annotation phase is often laborious and prone to inaccuracies resulting from human involvement \cite{ding2022gpt}. In response to these challenges, researchers have dedicated efforts towards data augmentation (DA) techniques as a means to mitigate such issues \cite{chen-etal-2023-empirical}. Data augmentation fundamentally involves the adoption of innovative methods aimed at bolstering model efficacy through the broadening of training data diversity, all without necessitating further data collection efforts. This strategy effectively tackles a major hindrance in machine learning research, namely the dearth of readily accessible training data, by applying slight modifications to existing datasets or creating synthetic data instances.&#10;&#10;As we venture into the realm of large language models (LLMs), the significance of data concerns escalates. Research into the scaling laws pertinent to LLMs highlights the critical role of data as a renewable resource crucial for the enhancement and advancement of models \cite{kaplan2020scaling}. With the expansion of model training scales, there is a marked increase in data consumption. Prior studies have examined the escalation in dataset sizes within the domain of machine learning, forecasting that the reservoir of high-quality linguistic data may be depleted by 2026 \cite{villalobos2022will}. This anticipates a potential deceleration in the growth of machine learning models unless there is an enhancement in data efficiency or the discovery of novel data sources. Consequently, the utilization of synthetic data produced by AI models becomes essential once high-quality human-generated data resources are fully exploited. &#10;&#10;From the data perspectives, data augmentation using LLMs offers a viable strategy to overcome these limitations, facilitating the creation of synthetic datasets of high quality that can, in certain instances, exceed the value of data curated by humans \cite{peng2023instruction}. This strategy not only addresses the challenge posed by the limited supply of human-annotated data but also conforms to scaling laws, enabling an increase in the size of training datasets without a proportional escalation in computational costs (FLOPs) \cite{Hoffmann2022TrainingCL}. The tactical employment of synthetic data stands to substantially reduce data collection costs and energy usage, signifying a transformative phase in model training and inference and laying the groundwork for the achievement of artificial general intelligence (AGI) \cite{li2023coannotating}.&#10;&#10;From a learning perspective, data augmentation using LLMs has heralded innovative learning paradigms, marking a significant departure from traditional methods predominantly centered around tasks like machine translation, sentiment analysis and NER. The application of LLMs in data augmentation extends to a broader spectrum of learning paradigms, including instruction tuning, in-context learning, and alignment learning. Additionally, it facilitates the generation of pseudo data for classification purposes and the scoring of data for regression analysis. This evolution in methodology not only broadens the scope of potential applications but also invites a deeper exploration into the nuanced benefits and insights that such innovative paradigms offer in the realm of data augmentation. As such, the systematic review of these emerging paradigms is crucial for advancing our understanding and leveraging the full potential of data augmentation in these learning contexts.&#10;&#10;\tikzstyle{my-box} = [&#10; rectangle,&#10; draw=hidden-black,&#10; rounded corners,&#10; text opacity=1,&#10; minimum height=1.5em,&#10; minimum width=5em,&#10; inner sep=2pt,&#10; align=center,&#10; fill opacity=.5,&#10;]&#10;&#10;\tikzstyle{leaf} = [&#10; my-box,&#10; minimum height=1.5em,&#10; fill=hidden-blue!90,&#10; text=black,&#10; align=left,&#10; font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10;]&#10;&#10;Given the growing interest and work in this domain, we believe it is a timely moment to present a paper on LLM-based DA. This paper aims to: (i) discuss data augmentation using LLM from the data perspective (ii) explore the learning paradigms that involve training LLMs on data generated by LLMs themselves, and (iii) highlight the principal challenges in this field to effectively guide and spur further interest and research. To the best of our knowledge, this represents the first survey that delves into data augmentation methods leveraging LLMs in such detail, thereby marking a significant contribution to the literature on LLM applications.&#10;&#10;This paper is structured as follows. Section~\ref{sec:related_works} provides a comprehensive review of related surveys, highlighting the distinctions between this work and previous surveys. Following that, as illustrated in Figure~\ref{fig:taxonomy}, Section~\ref{sec:methods} presents analysis from data perspectives for data augmentation with LLMs, establishing a foundational understanding for the community. Section~\ref{sec:learning} explores the learning paradigms associated with data augmentation using LLMs, neatly categorizing existing methods into two primary types: generative learning and discriminative learning. Section~\ref{sec:challenges} ventures into the challenges and prospective future directions for research in this domain, highlighting the complexities and opportunities that lie ahead. To further aid the community, the Appendix offers a detailed listing of existing methods categorized by tasks (Section~\ref{sec:tasks}) and domains (Section~\ref{sec:domains}), serving as a valuable resource for researchers and practitioners alike. Through this survey, we hope to attract wider attention, generate increased interest, and encourage further research in the field of data augmentation using LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Data Augmentation Techniques, Linguistics, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2303.10158" label="2303.10158">
        <attvalues>
          <attvalue for="0" value="Data-centric Artificial Intelligence: A Survey" />
          <attvalue for="1" value="  Artificial Intelligence (AI) is making a profound impact in almost every&#10;domain. A vital enabler of its great success is the availability of abundant&#10;and high-quality data for building machine learning models. Recently, the role&#10;of data in AI has been significantly magnified, giving rise to the emerging&#10;concept of data-centric AI. The attention of researchers and practitioners has&#10;gradually shifted from advancing model design to enhancing the quality and&#10;quantity of the data. In this survey, we discuss the necessity of data-centric&#10;AI, followed by a holistic view of three general data-centric goals (training&#10;data development, inference data development, and data maintenance) and the&#10;representative methods. We also organize the existing literature from&#10;automation and collaboration perspectives, discuss the challenges, and tabulate&#10;the benchmarks for various tasks. We believe this is the first comprehensive&#10;survey that provides a global view of a spectrum of tasks across various stages&#10;of the data lifecycle. We hope it can help the readers efficiently grasp a&#10;broad picture of this field, and equip them with the techniques and further&#10;research ideas to systematically engineer data for building AI systems. A&#10;companion list of data-centric AI resources will be regularly updated on&#10;https://github.com/daochenzha/data-centric-AI&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.07499" label="2106.07499">
        <attvalues>
          <attvalue for="0" value="An Empirical Survey of Data Augmentation for Limited Data Learning in&#10;  NLP" />
          <attvalue for="1" value="  NLP has achieved great progress in the past decade through the use of neural&#10;models and large labeled datasets. The dependence on abundant data prevents NLP&#10;models from being applied to low-resource settings or novel tasks where&#10;significant time, money, or expertise is required to label massive amounts of&#10;textual data. Recently, data augmentation methods have been explored as a means&#10;of improving data efficiency in NLP. To date, there has been no systematic&#10;empirical overview of data augmentation for NLP in the limited labeled data&#10;setting, making it difficult to understand which methods work in which&#10;settings. In this paper, we provide an empirical survey of recent progress on&#10;data augmentation for NLP in the limited labeled data setting, summarizing the&#10;landscape of methods (including token-level augmentations, sentence-level&#10;augmentations, adversarial augmentations, and hidden-space augmentations) and&#10;carrying out experiments on 11 datasets covering topics/news classification,&#10;inference tasks, paraphrasing tasks, and single-sentence tasks. Based on the&#10;results, we draw several conclusions to help practitioners choose appropriate&#10;augmentations in different settings and discuss the current challenges and&#10;future directions for limited data learning in NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.03277" label="2304.03277">
        <attvalues>
          <attvalue for="0" value="Instruction Tuning with GPT-4" />
          <attvalue for="1" value="  Prior work has shown that finetuning large language models (LLMs) using&#10;machine-generated instruction-following data enables such models to achieve&#10;remarkable zero-shot capabilities on new tasks, and no human-written&#10;instructions are needed. In this paper, we present the first attempt to use&#10;GPT-4 to generate instruction-following data for LLM finetuning. Our early&#10;experiments on instruction-tuned LLaMA models show that the 52K English and&#10;Chinese instruction-following data generated by GPT-4 leads to superior&#10;zero-shot performance on new tasks to the instruction-following data generated&#10;by previous state-of-the-art models. We also collect feedback and comparison&#10;data from GPT-4 to enable a comprehensive evaluation and reward model training.&#10;We make our data generated using GPT-4 as well as our codebase publicly&#10;available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.15638" label="2310.15638">
        <attvalues>
          <attvalue for="0" value="CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large&#10;  Language Models for Data Annotation" />
          <attvalue for="1" value="  Annotated data plays a critical role in Natural Language Processing (NLP) in&#10;training models and evaluating their performance. Given recent developments in&#10;Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot&#10;capability on many text-annotation tasks, comparable with or even exceeding&#10;human annotators. Such LLMs can serve as alternatives for manual annotation,&#10;due to lower costs and higher scalability. However, limited work has leveraged&#10;LLMs as complementary annotators, nor explored how annotation work is best&#10;allocated among humans and LLMs to achieve both quality and cost objectives. We&#10;propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of&#10;unstructured texts at scale. Under this framework, we utilize uncertainty to&#10;estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to&#10;be an effective means to allocate work from results on different datasets, with&#10;up to 21% performance improvement over random baseline. For code&#10;implementation, see https://github.com/SALT-NLP/CoAnnotating.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.12024" label="2202.12024">
        <attvalues>
          <attvalue for="0" value="NoisyTune: A Little Noise Can Help You Finetune Pretrained Language&#10;  Models Better" />
          <attvalue for="1" value="  Effectively finetuning pretrained language models (PLMs) is critical for&#10;their success in downstream tasks. However, PLMs may have risks in overfitting&#10;the pretraining tasks and data, which usually have gap with the target&#10;downstream tasks. Such gap may be difficult for existing PLM finetuning methods&#10;to overcome and lead to suboptimal performance. In this paper, we propose a&#10;very simple yet effective method named NoisyTune to help better finetune PLMs&#10;on downstream tasks by adding some noise to the parameters of PLMs before&#10;fine-tuning. More specifically, we propose a matrix-wise perturbing method&#10;which adds different uniform noises to different parameter matrices based on&#10;their standard deviations. In this way, the varied characteristics of different&#10;types of parameters in PLMs can be considered. Extensive experiments on both&#10;GLUE English benchmark and XTREME multilingual benchmark show NoisyTune can&#10;consistently empower the finetuning of different PLMs on different downstream&#10;tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, pretrained language models (PLMs) have achieved huge success in NLP~\cite{qiu2020pre}.&#10;Many PLMs such as BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and UniLM~\cite{dong2019unified} which are pretrained from large-scale unlabeled corpus in a self-supervised way, have significantly improve various downstream tasks such as reading comprehension~\cite{xu2019bert}, machine translation~\cite{brown2020language}, text classification~\cite{bao2020unilmv2}, dialog~\cite{wu2020tod} and recommendation~\cite{wu2021plm} by finetuning on these tasks.&#10;&#10;How to effectively finetune PLMs to better empower downstream tasks is an important research problem~\cite{zheng2021con}.&#10;Many existing NLP methods usually directly finetune PLMs with the labeled data in downstream tasks~\cite{sun2019fine}.&#10;Only a few works explore more effective and robust PLM finetuning methods~\cite{chen2020recall,Lee2020mixout,aghajanyan2020better,zhang2021revisiting,xu2021raise}.&#10;For example, \cite{chen2020recall} proposed RecAdam that adds a penalty item to minimize the $L_2$ distance between the fine-tuned models and the pretrained models, where the penalty intensity is time-variant during finetuning.&#10;\cite{Lee2020mixout} proposed Mixout which randomly replaces part of the parameters in the finetuned model with their original weights in the PLMs.&#10;These PLM finetuning methods mainly focus on preventing PLMs from overfitting the limited labeled data in downstream tasks.&#10;Besides the overfitting of downstream task data, a rarely studied problem is that the PLMs usually overfit the pretraining tasks and data~\cite{qi2020prophetnet}, which may have significant gap with the downstream task and data.&#10;It is not easy for existing PLM finetuning methods to overcome such gap~\cite{roberts2020much}, which may lead to suboptimal performance especially when labeled data in downstream tasks is insufficient.&#10;&#10;In order to handle this problem, in this paper we propose a very simple yet effective method named NoisyTune, which can help better finetune PLMs for downstream tasks.&#10;Different from the standard finetuning paradigm (Fig.~\ref{fig.exp} (a)) which directly finetunes PLMs on the downstream task data, the key idea of NoisyTune is to add a small amount of noise to perturb PLMs parameters before finetuning (Fig.~\ref{fig.exp} (b)).&#10;It can help prevent PLMs from overfitting the tasks and data in the pretraining stage, and reduce the gap between pretraining and downstream tasks.&#10;Since PLMs have different types of parameters which usually own different characteristics, in NoisyTune we use a matrix-wise perturbing method that adds uniform noise with different intensities to different parameter matrices according to their standard deviations for better adaptation.&#10;We conduct extensive experiments on two widely used NLP benchmarks, namely, GLUE~\cite{wang2018glue} for English language understanding and XTREME~\cite{hu2020xtreme} for multilingual language understanding.&#10;The results show NoisyTune can empower the finetuning of different PLMs on many different downstream NLP tasks to consistently achieve better performance.&#10;In addition, the results show NoisyTune can be easily combined with many existing PLM finetuning methods and further improve their performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Overfitting Mitigation, Multilingual Benchmarking, Artificial Intelligence, Mathematics, Language Model Finetuning" />
        </attvalues>
      </node>
      <node id="2003.08271" label="2003.08271">
        <attvalues>
          <attvalue for="0" value="Pre-trained Models for Natural Language Processing: A Survey" />
          <attvalue for="1" value="  Recently, the emergence of pre-trained models (PTMs) has brought natural&#10;language processing (NLP) to a new era. In this survey, we provide a&#10;comprehensive review of PTMs for NLP. We first briefly introduce language&#10;representation learning and its research progress. Then we systematically&#10;categorize existing PTMs based on a taxonomy with four perspectives. Next, we&#10;describe how to adapt the knowledge of PTMs to the downstream tasks. Finally,&#10;we outline some potential directions of PTMs for future research. This survey&#10;is purposed to be a hands-on guide for understanding, using, and developing&#10;PTMs for various NLP tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.03197" label="1905.03197">
        <attvalues>
          <attvalue for="0" value="Unified Language Model Pre-training for Natural Language Understanding&#10;  and Generation" />
          <attvalue for="1" value="  This paper presents a new Unified pre-trained Language Model (UniLM) that can&#10;be fine-tuned for both natural language understanding and generation tasks. The&#10;model is pre-trained using three types of language modeling tasks:&#10;unidirectional, bidirectional, and sequence-to-sequence prediction. The unified&#10;modeling is achieved by employing a shared Transformer network and utilizing&#10;specific self-attention masks to control what context the prediction conditions&#10;on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0&#10;and CoQA question answering tasks. Moreover, UniLM achieves new&#10;state-of-the-art results on five natural language generation datasets,&#10;including improving the CNN/DailyMail abstractive summarization ROUGE-L to&#10;40.51 (2.04 absolute improvement), the Gigaword abstractive summarization&#10;ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question&#10;answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question&#10;generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7&#10;document-grounded dialog response generation NIST-4 to 2.67 (human performance&#10;is 2.65). The code and pre-trained models are available at&#10;https://github.com/microsoft/unilm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.12804" label="2002.12804">
        <attvalues>
          <attvalue for="0" value="UniLMv2: Pseudo-Masked Language Models for Unified Language Model&#10;  Pre-Training" />
          <attvalue for="1" value="  We propose to pre-train a unified language model for both autoencoding and&#10;partially autoregressive language modeling tasks using a novel training&#10;procedure, referred to as a pseudo-masked language model (PMLM). Given an input&#10;text with masked tokens, we rely on conventional masks to learn inter-relations&#10;between corrupted tokens and context via autoencoding, and pseudo masks to&#10;learn intra-relations between masked spans via partially autoregressive&#10;modeling. With well-designed position embeddings and self-attention masks, the&#10;context encodings are reused to avoid redundant computation. Moreover,&#10;conventional masks used for autoencoding provide global masking information, so&#10;that all the position embeddings are accessible in partially autoregressive&#10;language modeling. In addition, the two tasks pre-train a unified language&#10;model as a bidirectional encoder and a sequence-to-sequence decoder,&#10;respectively. Our experiments show that the unified language models pre-trained&#10;using PMLM achieve new state-of-the-art results on a wide range of natural&#10;language understanding and generation tasks across several widely used&#10;benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.06871" label="2004.06871">
        <attvalues>
          <attvalue for="0" value="TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented&#10;  Dialogue" />
          <attvalue for="1" value="  The underlying difference of linguistic patterns between general text and&#10;task-oriented dialogue makes existing pre-trained language models less useful&#10;in practice. In this work, we unify nine human-human and multi-turn&#10;task-oriented dialogue datasets for language modeling. To better model dialogue&#10;behavior during pre-training, we incorporate user and system tokens into the&#10;masked language modeling. We propose a contrastive objective function to&#10;simulate the response selection task. Our pre-trained task-oriented dialogue&#10;BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream&#10;task-oriented dialogue applications, including intention recognition, dialogue&#10;state tracking, dialogue act prediction, and response selection. We also show&#10;that TOD-BERT has a stronger few-shot ability that can mitigate the data&#10;scarcity problem for task-oriented dialogue.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;General Pre-trained Language Models, which are trained on massive general text such as Wikipedia and BookCorpus, can be roughly divided into two categories: uni-directional or bi-directional attention mechanisms. GPT \cite{radford2018improving} and GPT-2 \cite{radford2019language} are representatives of uni-directional language models using a Transformer decoder, where the objective is to maximize left-to-right generation likelihood. These models are commonly applied in natural language generation tasks. On the other hand, BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta}, and their variances are pre-trained using a Transformer encoder with bi-directional token prediction. These models are usually evaluated on classification tasks such as GLUE benchmark \cite{wang2018glue} or span-based question answering tasks \cite{rajpurkar2016squad}. &#10;&#10;Some language models can support both uni-directional and bi-directional attention, such as UniLM \cite{dong2019unified}.&#10;Conditional language model pre-training is also proposed. For example, CTRL \cite{keskar2019ctrl} is a conditional Transformer model, trained to condition on control codes that govern style, content, and task-specific behavior.&#10;Recently, multi-task language model pre-training with unified sequence-to-sequence generation is proposed. Text-to-text Transformer (T5) \cite{raffel2019exploring} unifies multiple text modeling tasks and achieves the promising results in various NLP benchmarks.&#10;&#10;Dialogue Pre-trained Language Models are mostly trained on open-domain conversational data from Reddit or Twitter for dialogue response generation. &#10;Transfertransfo \cite{wolf2019transfertransfo} achieves good performance on ConvAI-2 dialogue competition using GPT-2.&#10;DialoGPT \cite{zhang2019dialogpt} is an extension of GPT-2 that is pre-trained on Reddit data for open-domain response generation.&#10;ConveRT \cite{henderson2019convert} pre-trained a dual transformer encoder for response selection task on large-scale Reddit (input, response) pairs. &#10;PLATO \cite{bao2019plato} uses both Twitter and Reddit data to pre-trained a dialogue generation model with discrete latent variables.&#10;All of them are designed to cope with the response generation task for open-domain chatbots.&#10;&#10;Pretraining for task-oriented dialogues, on the other hand, has few related works. \cite{budzianowski2019hello} first apply the GPT-2 model to train on response generation task, which takes system belief, database result, and last dialogue turn as input to predict next system responses. It only uses one dataset to train its model because few public datasets have database information available. \cite{henderson-etal-2019-training} pre-trained a response selection model for task-oriented dialogues. They first pre-train on Reddit corpora and then fine-tune on target dialogue domains, but their training and fine-tuning code is not released. \cite{peng2020few} focus on the natural language generation (NLG) task, which assumes dialogue acts and slot-tagging results are given to generate a natural language response. Pre-training on a set of annotated NLG corpora can improve conditional generation quality using a GPT-2 model.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2104.07413" label="2104.07413">
        <attvalues>
          <attvalue for="0" value="Empowering News Recommendation with Pre-trained Language Models" />
          <attvalue for="1" value="  Personalized news recommendation is an essential technique for online news&#10;services. News articles usually contain rich textual content, and accurate news&#10;modeling is important for personalized news recommendation. Existing news&#10;recommendation methods mainly model news texts based on traditional text&#10;modeling methods, which is not optimal for mining the deep semantic information&#10;in news texts. Pre-trained language models (PLMs) are powerful for natural&#10;language understanding, which has the potential for better news modeling.&#10;However, there is no public report that show PLMs have been applied to news&#10;recommendation. In this paper, we report our work on exploiting pre-trained&#10;language models to empower news recommendation. Offline experimental results on&#10;both monolingual and multilingual news recommendation datasets show that&#10;leveraging PLMs for news modeling can effectively improve the performance of&#10;news recommendation. Our PLM-empowered news recommendation models have been&#10;deployed to the Microsoft News platform, and achieved significant gains in&#10;terms of both click and pageview in both English-speaking and global markets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.03156" label="2008.03156">
        <attvalues>
          <attvalue for="0" value="Better Fine-Tuning by Reducing Representational Collapse" />
          <attvalue for="1" value="  Although widely adopted, existing approaches for fine-tuning pre-trained&#10;language models have been shown to be unstable across hyper-parameter settings,&#10;motivating recent work on trust region methods. In this paper, we present a&#10;simplified and efficient method rooted in trust region theory that replaces&#10;previously used adversarial objectives with parametric noise (sampling from&#10;either a normal or uniform distribution), thereby discouraging representation&#10;change during fine-tuning when possible without hurting performance. We also&#10;introduce a new analysis to motivate the use of trust region methods more&#10;generally, by studying representational collapse; the degradation of&#10;generalizable representations from pre-trained models as they are fine-tuned&#10;for a specific end task. Extensive experiments show that our fine-tuning method&#10;matches or exceeds the performance of previous trust region methods on a range&#10;of understanding and generation tasks (including DailyMail/CNN, Gigaword,&#10;Reddit TIFU, and the GLUE benchmark), while also being much faster. We also&#10;show that it is less prone to representation collapse; the pre-trained models&#10;maintain more generalizable representations every time they are fine-tuned.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.05987" label="2006.05987">
        <attvalues>
          <attvalue for="0" value="Revisiting Few-sample BERT Fine-tuning" />
          <attvalue for="1" value="  This paper is a study of fine-tuning of BERT contextual representations, with&#10;focus on commonly observed instabilities in few-sample scenarios. We identify&#10;several factors that cause this instability: the common use of a non-standard&#10;optimization method with biased gradient estimation; the limited applicability&#10;of significant parts of the BERT network for down-stream tasks; and the&#10;prevalent practice of using a pre-determined, and small number of training&#10;iterations. We empirically test the impact of these factors, and identify&#10;alternative practices that resolve the commonly observed instability of the&#10;process. In light of these observations, we re-visit recently proposed methods&#10;to improve few-sample fine-tuning with BERT and re-evaluate their&#10;effectiveness. Generally, we observe the impact of these methods diminishes&#10;significantly with our modified process.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.05687" label="2109.05687">
        <attvalues>
          <attvalue for="0" value="Raise a Child in Large Language Model: Towards Effective and&#10;  Generalizable Fine-tuning" />
          <attvalue for="1" value="  Recent pretrained language models extend from millions to billions of&#10;parameters. Thus the need to fine-tune an extremely large pretrained model with&#10;a limited training corpus arises in various downstream tasks. In this paper, we&#10;propose a straightforward yet effective fine-tuning technique, Child-Tuning,&#10;which updates a subset of parameters (called child network) of large pretrained&#10;models via strategically masking out the gradients of the non-child network&#10;during the backward process. Experiments on various downstream tasks in GLUE&#10;benchmark show that Child-Tuning consistently outperforms the vanilla&#10;fine-tuning by 1.5~8.6 average score among four different pretrained models,&#10;and surpasses the prior fine-tuning techniques by 0.6~1.3 points. Furthermore,&#10;empirical results on domain transfer and task transfer show that Child-Tuning&#10;can obtain better generalization performance by large margins.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.08910" label="2002.08910">
        <attvalues>
          <attvalue for="0" value="How Much Knowledge Can You Pack Into the Parameters of a Language Model?" />
          <attvalue for="1" value="  It has recently been observed that neural language models trained on&#10;unstructured text can implicitly store and retrieve knowledge using natural&#10;language queries. In this short paper, we measure the practical utility of this&#10;approach by fine-tuning pre-trained models to answer questions without access&#10;to any external context or knowledge. We show that this approach scales with&#10;model size and performs competitively with open-domain systems that explicitly&#10;retrieve answers from an external knowledge source when answering questions. To&#10;facilitate reproducibility and future work, we release our code and trained&#10;models at https://goo.gle/t5-cbqa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Question Answering&#10;The task of training a model to either select or output the correct answer to a given question is referred to as ``question answering''.&#10;The most popular variant of this task feeds the model some ``context'' containing the answer (for example, a paragraph from an encyclopedia article) alongside the question \cite{rajpurkar2016squad,zhang2018record,khashabi2018looking,clark2019boolq}.&#10;Models can be trained either to indicate the span of the context that contains the answer or output the text of the answer itself.&#10;Since this format can be seen as reading some text and answering a question about it, it has been referred to as ``reading comprehension''.&#10;&#10;A more difficult variant is ``open-domain question answering'' \cite{prager2006open}, where the model can be asked arbitrary context-independent questions (e.g.\ well-known facts or historical details).&#10;It is typically assumed that the model can access an external collection of knowledge when answering questions (e.g.\ a structured knowledge base or unstructured text corpus), but the model is not given any information about where in the collection the answer appears.&#10;The reading comprehension task can be considered a simplified version of open-domain question answering where the model is provided with the oracle context to answer a given question.&#10;As an analogy, the open-domain question answering system acts as if it is taking an open-book exam where it can find and use information in an external source of knowledge.&#10;&#10;In this work, we consider open-domain question answering with the additional constraint that the model is not allowed to access any external knowledge whatsoever when answering questions.&#10;Instead, the model itself must be pre-trained to store knowledge in its parameters before being fine-tuned to answer questions.&#10;In one view, this can be seen as an alternative way to approach open-domain question answering where instead of learning to access external knowledge the model needs to have ``memorized'' it in order to answer questions; in another view, this constraint creates a third and potentially more ambitious variant of the question answering task.&#10;A model that answers questions in this way is metaphorically similar to a student taking a closed-book exam, where the student must study and memorize all pertinent information before taking the test.&#10;&#10;Transfer Learning with Language Models&#10;In the past few years, it has become increasingly common to pre-train a language model using an unsupervised objective on a large, unstructured text corpus before fine-tuning it on a downstream task of interest \cite{dai2015semi,howard2018universal,radford2018improving}.&#10;The popularity of this form of ``transfer learning'' is attributable to its empirical success on many NLP tasks \cite{peters2018deep,devlin2018bert,yang2019xlnet,lan2019albert,raffel2019exploring}.&#10;Loosely speaking, the pre-training step may provide the model with some generally-useful awareness of meaning, syntax, and ``world knowledge''.&#10;In question answering in particular, most state-of-the-art systems use some form of transfer learning.&#10;&#10;Currently, the most popular model architectures used in transfer learning for NLP are Transformer-based \cite{vaswani2017attention} ``encoder-only'' models like BERT \cite{devlin2018bert}.&#10;These models can produce a single prediction for each input token and have been applied to reading comprehension-style question answering by predicting which tokens of the context contain the answer.&#10;Encoder-only models are not applicable to closed-book question answering because no context is provided to extract the answer span from.&#10;An alternative to encoder-only models, recently advocated by \cite{raffel2019exploring}, is to treat every NLP task as a text-to-text problem using an encoder-decoder Transformer.&#10;When this framework is applied to question answering, the model is trained to generate the literal text of the answer in a free-form fashion.&#10;Despite the potential difficulty of generating rather than extracting the answer, \cite{raffel2019exploring} demonstrated state-of-the-art results on the SQuAD \cite{rajpurkar2016squad}, MultiRC \cite{khashabi2018looking}, BoolQ \cite{clark2019boolq}, and ReCoRD \cite{zhang2018record} reading comprehension tasks.&#10;&#10;The text-to-text framework is directly applicable to closed-book question answering since the model can be trained to generate an answer with or without any additional information in its input.&#10;Crucially, fine-tuning a text-to-text model to answer questions without any context requires that the model retrieve information from its parameters that it learned during pre-training.&#10;\cite{radford2019language} considered a similar task to evaluate the zero-shot question answering capabilities of a language model.&#10;The concurrent ``RELIC'' and ``EAE'' models of \cite{ling2020learning} and \cite{fevry2020eae} learn representations for an explicitly predefined set of entities and are evaluated on the same closed-book variant of TriviaQA that we consider.&#10;Relatedly, \cite{petroni2019language} show that it is possible to manually convert some questions to a fill-in-the-blank format amenable to an encoder-only model (e.g.\ ``Who developed the theory of relativity?''\ gets mapped to ``The theory of relativity was developed by \_\_\_\_'').&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2405.13386" label="2405.13386">
        <attvalues>
          <attvalue for="0" value="360Zhinao Technical Report" />
          <attvalue for="1" value="  We present 360Zhinao models with 7B parameter size and context lengths&#10;spanning 4K, 32K and 360K, all available at&#10;https://github.com/Qihoo360/360zhinao. For rapid development in pretraining, we&#10;establish a stable and sensitive ablation environment to evaluate and compare&#10;experiment runs with minimal model size. Under such guidance, we perfect our&#10;data cleaning and composition strategies to pretrain&#10;$\texttt{360Zhinao-7B-Base}$ on 3.4T tokens. We also mainly emphasize data&#10;during alignment, where we strive to balance quantity and quality with&#10;filtering and reformatting. With tailored data, 360Zhinao-7B's context window&#10;is easily extended to 32K and 360K. RMs and RLHF are trained following SFT and&#10;credibly applied to specific tasks. All together these contributions lead to&#10;360Zhinao-7B's competitive performance among models of similar size.&#10;" />
          <attvalue for="2" value="&#10;In recent years, the field of natural language processing (NLP) has witnessed a profound transformation, fueled by the advent of large language models (LLMs) \cite{bubeck2023sparks,touvron2023llama, achiam2023gpt}, which have emerged as a cornerstone to revolutionize the way we understand and generate human language. LLMs represent a new paradigm in artificial intelligence (AI) research, characterized by their immense scale, complexity, and versatility \cite{zhao2023survey}. Those models, typically built upon advanced neural network architectures like Transformers, are trained on vast amounts of text data, encompassing billions or even trillions of words. The extensive training endows LLMs with a deep understanding of linguistic structures, nuances, and context, enabling them to generate human-like text and perform a myriad of NLP tasks with unprecedented accuracy and fluency \cite{yang2024harnessing}.&#10;&#10;Despite the impressive capabilities of LLMs, training an LLM from scratch still struggles with several challenges. The training journey can be divided into two stages: the pretraining stage and the alignment stage \cite{zhang2023instruction}. The pretraining stage involves the model learning on large-scale textual data to build its foundational knowledge and language comprehension. However, two obstacles stick out in the pretraining stage \cite{zhao2023survey}. First, refining the training corpus to enhance the base model's performance is paramount given the enormity of pretraining data. While extensive research has delved into data cleaning and sampling methodologies \cite{soldaini2024dolma, penedo2023refinedweb, wenzek2019ccnet, gunasekar2023textbooks}, the sheer scale and intricacy of pretraining datasets still leave ample room for elevating informational density and efficiency. Second, establishing a stable and sensitive ablation environment for accurately assessing data strategies poses another challenge \cite{chang2024survey,zhou2023don}. The widely used Opencompass \cite{contributors2023opencompass} framework proves inconvenient for data strategy explorations. It is inherently unstable and insensitive to smaller models or datasets \cite{wei2023skywork}, and further, it lacks correlation with downstream skills to adequately evaluate the model. Addressing these two challenges will help propel further research and practices in the pretraining stage.&#10;&#10;In the alignment stage, challenges arise regarding data \cite{albalak2024survey}, long context \cite{niah2023v0} and RLHF effectiveness \cite{wang2024secrets,xu2024dpo}.&#10;SFT gets major parts of things done, but is to some extent sensitive to data quality and composition \cite{liu2024what}. It is tricky to balance the learning of different prompt categories and specific application data.&#10;Various useful applications of LLMs and multi-modal large models require sequence lengths far beyond several thousand \cite{reid2024gemini}. Pretraining on large corpus of long data (context length of tens or hundreds of thousand) turns out inefficient and one desideratum is to extend the context length with minimal continual pretraining and SFT at reasonable costs.&#10;RLHF in open-sourced models has not yet fulfilled the presumed promise as \cite{achiam2023gpt}. Much remains under-explored in terms of RM data, RM training, PPO data and PPO training, etc.&#10;&#10;In response to those challenges, we devoted substantial efforts to our LLM models, the 360Zhinao series, and presented the details in this technical report. The 360Zhinao model comprises a base model trained from scratch and a chat model using alignment techniques. In the pretraining stage, we explored the data strategies and their evaluation. First, we built a data cleaning and filtering pipeline. By crawling massive web pages, we employed a series of filtering and cleaning steps. Subsequently, we explored multi-level deduplication and data mixing strategies, ultimately obtaining a corpus of 3.4 TB tokens with high data efficiency. Second, to validate the effectiveness of our data strategies, we constructed a stable and sensitive ablation environment, establishing a set of custom benchmarks associated with downstream skills. Based on this environment, we reported the ablation results of data strategies, effectively guiding the direction of data iterations.&#10;&#10;Emphasis is also laid on data in the alignment stage. Initially we prioritized quantity over quality and later the opposite. We now grow SFT data at a more steady pace with higher standards. &#10;We explored different ways of context window extension and have converged to the simple method of RoPE-base change with tailored data.&#10;We stabilized RLHF training with data and codebase improvement and have applied successfully to specific tasks. The resulting RM has also been used in other data selection and judgement pipelines.&#10;&#10;Due to the complexity of LLMs, some crucial intermediate information on model production is absent in existing literature. In this report, we made efforts to break barriers and hope to offer new perspectives to researchers. We have released the implementations of the base- and chat-models to the open-source community, aiming to enhance the openness and transparency of LLMs and facilitate collaboration and reproducibility. Based on these efforts, the contributions of the release of the 360Zhinao models can be summarized as follows:&#10;\begin{itemize}&#10;\item In the pretraining stage, we provide detailed insights into the iteration of data recipes and the construction of an effective ablation experimental environment.&#10;\item In the alignment stage, we present our data-centric approach to data improvement, our converged approach to context window extension and positive findings from RLHF.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Data Science, Mathematics, Information Science, Data Preprocessing Techniques, Model Training Strategies" />
        </attvalues>
      </node>
      <node id="2308.10792" label="2308.10792">
        <attvalues>
          <attvalue for="0" value="Instruction Tuning for Large Language Models: A Survey" />
          <attvalue for="1" value="  This paper surveys research works in the quickly advancing field of&#10;instruction tuning (IT), a crucial technique to enhance the capabilities and&#10;controllability of large language models (LLMs). Instruction tuning refers to&#10;the process of further training LLMs on a dataset consisting of&#10;\textsc{(instruction, output)} pairs in a supervised fashion, which bridges the&#10;gap between the next-word prediction objective of LLMs and the users' objective&#10;of having LLMs adhere to human instructions. In this work, we make a systematic&#10;review of the literature, including the general methodology of IT, the&#10;construction of IT datasets, the training of IT models, and applications to&#10;different modalities, domains and applications, along with an analysis on&#10;aspects that influence the outcome of IT (e.g., generation of instruction&#10;outputs, size of the instruction dataset, etc). We also review the potential&#10;pitfalls of IT along with criticism against it, along with efforts pointing out&#10;current deficiencies of existing strategies and suggest some avenues for&#10;fruitful research. Project page: github.com/xiaoya-li/Instruction-Tuning-Survey&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.01964" label="2311.01964">
        <attvalues>
          <attvalue for="0" value="Don't Make Your LLM an Evaluation Benchmark Cheater" />
          <attvalue for="1" value="  Large language models~(LLMs) have greatly advanced the frontiers of&#10;artificial intelligence, attaining remarkable improvement in model capacity. To&#10;assess the model performance, a typical approach is to construct evaluation&#10;benchmarks for measuring the ability level of LLMs in different aspects.&#10;Despite that a number of high-quality benchmarks have been released, the&#10;concerns about the appropriate use of these benchmarks and the fair comparison&#10;of different models are increasingly growing. Considering these concerns, in&#10;this paper, we discuss the potential risk and impact of inappropriately using&#10;evaluation benchmarks and misleadingly interpreting the evaluation results.&#10;Specially, we focus on a special issue that would lead to inappropriate&#10;evaluation, \ie \emph{benchmark leakage}, referring that the data related to&#10;evaluation sets is occasionally used for model training. This phenomenon now&#10;becomes more common since pre-training data is often prepared ahead of model&#10;test. We conduct extensive experiments to study the effect of benchmark&#10;leverage, and find that it can dramatically boost the evaluation results, which&#10;would finally lead to an unreliable assessment of model performance. To improve&#10;the use of existing evaluation benchmarks, we finally present several&#10;guidelines for both LLM developers and benchmark maintainers. We hope this work&#10;can draw attention to appropriate training and evaluation of LLMs.&#10;" />
          <attvalue for="2" value="&#10;&#10;\begin{quote}&#10;Goodhart's Law: ``When a measure becomes a target, it ceases to be a good measure.'' &#10;\end{quote}&#10;&#10;Large language models (LLMs) have achieved remarkable success across a variety of real-world applications~\cite{GPT-3,LLMsurvey,DBLP:journals/corr/abs-2308-07107}. &#10;By pre-training large Transformer models on massive text corpora, LLMs can &#10;possess excellent task-solving capacities, i.e., using zero-shot or few-shot prompting~\cite{GPT-3}.&#10;To better understand how LLMs evolve in model capacity, it becomes essential to construct reliable evaluation benchmarks to test the ability level of LLMs in various tasks, e.g., knowledge reasoning and math problem solving. &#10;&#10;Recently, a surge of high-quality evaluation benchmarks~\cite{mmlu,huang2023c} have been proposed to provide a comprehensive capability evaluation of LLMs. Typical benchmarks include MMLU~\cite{mmlu} (for measuring multitask language understanding ability), Big-Bench~\cite{srivastava2023beyond} (for quantifying and extrapolating the capabilities of LLMs), and AGIEval~\cite{zhong2023agieval} (for evaluating the abilities of tackling human-level tasks). These benchmarks have made great efforts in creating or collecting test resources for evaluating the performance of LLMs. &#10;Based on these benchmarks,&#10;one can conveniently examine the effect of new training strategies or monitor the training status of LLMs (either pre-training or supervised fine-tuning). &#10;It has become common to report the results on these evaluation benchmarks for demonstrating the effectiveness of &#10;newly released LLMs~\cite{GPT-4,llama2,anil2023palm}.&#10;Furthermore, to compare the performance of different LLMs, various leaderboards have been also created to rank LLMs according to their performance on existing or new evaluation benchmarks, such as OpenCompass~\cite{2023opencompass} and C-Eval~\cite{huang2023c}. &#10;&#10;Despite the wide use of these benchmarks and leaderboards, increasing concerns~\cite{aiyappa2023can,li2023open} are growing about the fairness and reliability in evaluating existing LLMs. A major issue is that the data contamination or leakage is likely to occur for large-scale benchmark evaluation, which means that LLMs are trained with relevant or exactly the same data for test. Such an issue could be unconsciously triggered, since we might be unaware of the future evaluation datasets when preparing the pre-training corpus. &#10;For example, GPT-3 has found that Children’s Book Test dataset~\cite{Hill2015TheGP} was included in the pre-training corpus, and LLaMA-2 has mentioned that the contexts in BoolQ dataset~\cite{boolq} are extracted verbatim from the webpages, which may be included in the publicly available corpus.&#10;&#10;Indeed, when conducting evaluation with existing benchmarks, the results of evaluated LLMs are mostly obtained by running them on local servers or via API calls. During this process, there is no strict checking on any potentially inappropriate ways (e.g., data contamination) that would cause an unnormal improvement of evaluation performance. To make matters worse, the detailed composition (e.g., data sources) of the training corpus is often regarded as the core ``secret'' of existing LLMs. Therefore, it becomes difficult to directly examine the contamination issues when performing the evaluation for benchmark maintainers. &#10;&#10;Considering this issue, the aim of this paper is to draw attention on appropriately using existing evaluation benchmarks and avoiding any misleading behaviors in obtaining or interpreting the evaluation results. &#10;Specifically, we mainly focus on discussing the potential effect of benchmark leakage, which refers to the case that test data or relevant data (e.g., training set) has been included in the pre-training corpus.&#10;It would cause an unfair performance advantage when comparing different LLMs or assessing the ability level of some specific LLMs. &#10;As we discussed before, this issue tends to become increasingly more common as we try to collect more public text data for training. &#10;To investigate this issue, we set up several benchmark leakage settings that should be totally avoided during evaluation, including the leakage of training sets, test prompts, and test sets. Based on the three settings, we continually train four popular language models, ranging from 1.3B to 7B, and test the performance of the four models on a number of existing benchmarks. In addition, we also examine the potential risk of benchmark leakage on other abilities. &#10;&#10;The experimental results reveal that benchmark leakage can lead to an unfair boost in the evaluation performance of LLMs. Smaller LLMs (e.g., a 1.3B model) can be deliberately elevated to outperform $10\times$ larger models on certain tasks. As a side effect, the performance of these specially trained LLMs on other normally tested tasks would likely be adversely affected if we fine-tune or train the model only with these leaked data. &#10;&#10;By examining the potential risks of benchmark leakage, we would like to emphasize the importance of fair and appropriate evaluation for LLMs, and propose several suggestions to improve the evaluation for LLMs: &#10;\begin{itemize}&#10; \item &#10; {As general suggestions, more benchmarks from diverse sources, covering both basic ability (e.g., text generation) and advanced ability tests (e.g., complex reasoning), should be used for comprehensively estimating the capabilities of LLMs.}&#10; \item {As suggestions for LLM developers, it is important to perform the data decontamination checking between pre-training data and any related data (e.g., training and test sets) when using evaluation benchmarks. In addition, it is also necessary to report the contamination analysis on the evaluated benchmarks as reference. We also suggest reporting the detailed composition of the pre-training data.}&#10; \item {As suggestions for benchmark maintainers, we suggest that a diverse set of test prompts should be employed for reducing the influence of the prompt sensitivity. It is also meaningful to conduct the contamination analysis between the benchmark data and existing pre-training corpus, alerting any potential contamination risks. For evaluation, each submission is suggested to be accompanied with a special contamination analysis report.}&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.16827" label="2402.16827">
        <attvalues>
          <attvalue for="0" value="A Survey on Data Selection for Language Models" />
          <attvalue for="1" value="  A major factor in the recent success of large language models is the use of&#10;enormous and ever-growing text datasets for unsupervised pre-training. However,&#10;naively training a model on all available data may not be optimal (or&#10;feasible), as the quality of available text data can vary. Filtering out data&#10;can also decrease the carbon footprint and financial costs of training models&#10;by reducing the amount of training required. Data selection methods aim to&#10;determine which candidate data points to include in the training dataset and&#10;how to appropriately sample from the selected data points. The promise of&#10;improved data selection methods has caused the volume of research in the area&#10;to rapidly expand. However, because deep learning is mostly driven by empirical&#10;evidence and experimentation on large-scale data is expensive, few&#10;organizations have the resources for extensive data selection research.&#10;Consequently, knowledge of effective data selection practices has become&#10;concentrated within a few organizations, many of which do not openly share&#10;their findings and methodologies. To narrow this gap in knowledge, we present a&#10;comprehensive review of existing literature on data selection methods and&#10;related research areas, providing a taxonomy of existing approaches. By&#10;describing the current landscape of research, this work aims to accelerate&#10;progress in data selection by establishing an entry point for new and&#10;established researchers. Additionally, throughout this review we draw attention&#10;to noticeable holes in the literature and conclude the paper by proposing&#10;promising avenues for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.10719" label="2404.10719">
        <attvalues>
          <attvalue for="0" value="Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study" />
          <attvalue for="1" value="  Reinforcement Learning from Human Feedback (RLHF) is currently the most&#10;widely used method to align large language models (LLMs) with human&#10;preferences. Existing RLHF methods can be roughly categorized as either&#10;reward-based or reward-free. Novel applications such as ChatGPT and Claude&#10;leverage reward-based methods that first learn a reward model and apply&#10;actor-critic algorithms, such as Proximal Policy Optimization (PPO). However,&#10;in academic benchmarks, state-of-the-art results are often achieved via&#10;reward-free methods, such as Direct Preference Optimization (DPO). Is DPO truly&#10;superior to PPO? Why does PPO perform poorly on these benchmarks? In this&#10;paper, we first conduct both theoretical and empirical studies on the&#10;algorithmic properties of DPO and show that DPO may have fundamental&#10;limitations. Moreover, we also comprehensively examine PPO and reveal the key&#10;factors for the best performances of PPO in fine-tuning LLMs. Finally, we&#10;benchmark DPO and PPO across a collection of RLHF testbeds, ranging from&#10;dialogue to code generation. Experiment results demonstrate that PPO is able to&#10;surpass other alignment methods in all cases and achieve state-of-the-art&#10;results in challenging code competitions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.15685" label="2312.15685">
        <attvalues>
          <attvalue for="0" value="What Makes Good Data for Alignment? A Comprehensive Study of Automatic&#10;  Data Selection in Instruction Tuning" />
          <attvalue for="1" value="  Instruction tuning is a standard technique employed to align large language&#10;models to end tasks and user preferences after the initial pretraining phase.&#10;Recent research indicates the critical role of data engineering in instruction&#10;tuning -- when appropriately selected, only limited data is necessary to&#10;achieve superior performance. However, we still lack a principled understanding&#10;of what makes good instruction tuning data for alignment, and how we should&#10;select data automatically and effectively. In this work, we delve deeply into&#10;automatic data selection strategies for alignment. We start with controlled&#10;studies to measure data across three dimensions: complexity, quality, and&#10;diversity, along which we examine existing methods and introduce novel&#10;techniques for enhanced data measurement. Subsequently, we propose a simple&#10;strategy to select data samples based on the measurement. We present deita&#10;(short for Data-Efficient Instruction Tuning for Alignment), a series of models&#10;fine-tuned from LLaMA and Mistral models using data samples automatically&#10;selected with our proposed approach. Empirically, deita performs better or on&#10;par with the state-of-the-art open-source alignment models with only 6K SFT&#10;training data samples -- over 10x less than the data used in the baselines.&#10;When further trained with direct preference optimization (DPO),&#10;deita-Mistral-7B + DPO trained with 6K SFT and 10K DPO samples achieve 7.55&#10;MT-Bench and 90.06% AlpacaEval scores. We anticipate this work to provide tools&#10;on automatic data selection, facilitating data-efficient alignment. We release&#10;our models as well as the selected datasets for future researches to&#10;effectively align models more efficiently.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14233" label="2305.14233">
        <attvalues>
          <attvalue for="0" value="Enhancing Chat Language Models by Scaling High-quality Instructional&#10;  Conversations" />
          <attvalue for="1" value="  Fine-tuning on instruction data has been widely validated as an effective&#10;practice for implementing chat language models like ChatGPT. Scaling the&#10;diversity and quality of such data, although straightforward, stands a great&#10;chance of leading to improved performance. This paper aims to improve the upper&#10;bound of open-source models further. We first provide a systematically&#10;designed, diverse, informative, large-scale dataset of instructional&#10;conversations, UltraChat, which does not involve human queries. Our objective&#10;is to capture the breadth of interactions that a human might have with an AI&#10;assistant and employs a comprehensive framework to generate multi-turn&#10;conversation iteratively. UltraChat contains 1.5 million high-quality&#10;multi-turn dialogues and covers a wide range of topics and instructions. Our&#10;statistical analysis of UltraChat reveals its superiority in various key&#10;metrics, including scale, average length, diversity, coherence, etc.,&#10;solidifying its position as a leading open-source dataset. Building upon&#10;UltraChat, we fine-tune a LLaMA model to create a powerful conversational&#10;model, UltraLLaMA. Our evaluations indicate that UltraLLaMA consistently&#10;outperforms other open-source models, including Vicuna, the previously&#10;recognized state-of-the-art open-source model. The dataset and the model will&#10;be publicly released\footnote{\url{https://github.com/thunlp/UltraChat}}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Instruction Tuning.&#10;Recent works demonstrate LLMs with powerful capabilities in following human instructions. \cite{wei2021finetuned} pioneered to fine-tune T5~\cite{2020t5} on $60$ NLP datasets verbalized with natural language instruction templates, i.e., instruction tuning. The fine-tuned model exhibits a strong ability in instruction understanding and generalizes well to unseen instructions (tasks). Later, \cite{longpre2023flan} extend the setting to $1,836$ tasks and show the benefits of scaling the number of tasks in out-of-distribution generalization. \cite{wei2021finetuned} also conclude that the success of instruction tuning depends on the quality of the dataset and the design of prompts. To further regulate the tuned model's behavior, \cite{ouyang2022training,schulman2017proximal} propose to first learn a reward model directly from annotated human feedback, then employ reinforcement learning to align model behaviors with human preferences. This technique can be combined with instruction tuning to further boost the model performance and has been successfully applied to LLMs such as ChatGPT.&#10;&#10;Data Augmentation with LLMs.&#10;Collecting large-scale human-annotated instructions and their responses is time-consuming and labor-intensive. Alternatively, a more cost-effective and feasible approach to gathering top-notch data involves sampling from LLMs that have been finely tuned, e.g., ChatGPT and GPT-3.5. &#10;Recently, there is a surge of interest in distilling these powerful LLMs for data augmentation. For instance, using the technique of SelfInstruct~\cite{wang2022selfinstruct}, Alpaca~\cite{alpaca} generate $52$k high-quality instruction-response pairs based on $175$ seed tasks by ``distilling'' Text-Davinci-003. After training a LLaMA~\cite{touvron2023llama} model on the dataset, the model performs almost on par with Text-Davinci-003. The success of Alpaca boosts numerous later efforts on data augmentation with LLMs, such as code-alpaca~\cite{codealpaca}, alpaca-cot~\cite{alpaca-cot}, GPT4ALL~\cite{gpt4all}, ShareGPT~\cite{domeccleston2023sharegpt}, Dolly-v2~\cite{Mike2023dolly}, BELLE~\cite{belle2023exploring}, Vicuna~\cite{vicuna2023}, Koala~\cite{koala_blogpost_2023}, Baize \cite{xu2023baize}, etc. It is shown that increasing the scale of data could constantly improve the model performance. Besides scaling the data size, these works also diverge in their ways of prompt engineering to gather data with better quality. For instance, CAMEL~\cite{li2023camel} designed a multi-agent role-play environment for LLMs to solve a given complex task and produced $115$k instruction-response pairs that simulate real human conversations.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Language Model Fine-Tuning, Linguistics, Conversational AI Models, Large-Scale Dataset Creation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2106.02834" label="2106.02834">
        <attvalues>
          <attvalue for="0" value="MergeDistill: Merging Pre-trained Language Models using Distillation" />
          <attvalue for="1" value="  Pre-trained multilingual language models (LMs) have achieved state-of-the-art&#10;results in cross-lingual transfer, but they often lead to an inequitable&#10;representation of languages due to limited capacity, skewed pre-training data,&#10;and sub-optimal vocabularies. This has prompted the creation of an ever-growing&#10;pre-trained model universe, where each model is trained on large amounts of&#10;language or domain specific data with a carefully curated, linguistically&#10;informed vocabulary. However, doing so brings us back full circle and prevents&#10;one from leveraging the benefits of multilinguality. To address the gaps at&#10;both ends of the spectrum, we propose MergeDistill, a framework to merge&#10;pre-trained LMs in a way that can best leverage their assets with minimal&#10;dependencies, using task-agnostic knowledge distillation. We demonstrate the&#10;applicability of our framework in a practical setting by leveraging&#10;pre-existing teacher LMs and training student LMs that perform competitively&#10;with or even outperform teacher LMs trained on several orders of magnitude more&#10;data and with a fixed model capacity. We also highlight the importance of&#10;teacher selection and its impact on student model performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Language Model pre-training has evolved from learning pre-trained word embeddings \cite{mikolov2013distributed} to contextualized word representations \cite{mccann2017learned, peters2018deep, eriguchi2018zero} and to the most recent Transformer-based \cite{vaswani2017attention} LMs \cite{devlin-etal-2019-bert,liu-etal-2019-robust} with state-of-the-art results on various downstream NLP tasks. Most commonly, these LMs are pre-trained with the MLM objective \cite{taylor1953cloze} on large unsupervised corpora and then fine-tuned on labeled data for the task at hand. Concurrently, multilingual LMs \cite{Lample2019xlm, Siddhant2019evaluating, conneau-etal-2020-unsupervised, chung2021rethinking}, trained on massive amounts of multilingual data, have surpassed cross-lingual word embedding spaces \cite{glavavs2019properly, Ruder_2019} to achieve state-of-the-art in cross-lingual transfer. While \cite{pires-etal-2019-multilingual, wu2019beto} highlight their cross-lingual ability, several limitations have been studied. \cite{conneau-etal-2020-unsupervised} highlight the curse of multilinguality. \cite{hu2020xtreme} highlight that even the best multilingual models do not yield satisfactory transfer performance on the XTREME bechmark covering 9 tasks and 40 languages. Importantly, \cite{wu2020all} and \cite{lauscher-etal-2020-zero} observe that these models significantly under-perform for low-resource languages as representation of these languages in the vocabulary and pre-training corpora are severely limited. \\&#10;&#10;Language-specific LMs are becoming increasingly popular as issues with multilingual language models persist. As language identification systems are extended to 1000+ languages \cite{caswell-etal-2020-language}, increasing capacity for a single model to uniformly represent all languages is prohibitive. Often, practitioners prefer to have a model performing well on a subset of languages that their application calls for. To address this, the community continues its efforts in building strong multi-domain language models using linguistic expertise. A few examples of these are AraBERT \cite{antoun-etal-2020-arabert}, CamemBERT \cite{martin-etal-2020-camembert}, and FinBERT \cite{virtanen2019multilingual}.\\&#10;&#10;Knowledge Distillation in pre-trained LMs has most commonly been used for task-specific model compression of a teacher into a single-task student \cite{tang2019distilling, kaliamoorthi2021distilling}. This has been extended to perform task-specific distillation of multiple single-task teachers into one multi-task student \cite{clark-etal-2019-bam, liu2020mkd, turc2019well}. In the task-agnostic scenario, prior work has focused on distilling a single large teacher model into a student model leveraging teacher predictions \cite{sanh2019distilbert} or internal teacher representations \cite{sun-etal-2020-mobilebert, sun2019patient, wang2020minilm} with the goal of model compression. To the best of our knowledge, this is the first attempt to perform task-agnostic distillation from multiple teachers into a single task-agnostic student. In the context of neural machine translation, \cite{tan2019multilingual} come close to our work where they attempt to combine multiple single language-pair teacher models to train a multilingual student. However, our work differs from theirs in three key aspects: 1) our students are task-agnostic while theirs are task-specific, 2) we can leverage pre-existing teachers while they cannot, and 3) we support teachers with overlapping sets of languages while they only consider single language-pairs teachers.&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Knowledge Distillation, Artificial Intelligence, Model Optimization" />
        </attvalues>
      </node>
      <node id="1909.04702" label="1909.04702">
        <attvalues>
          <attvalue for="0" value="Neural Embedding Allocation: Distributed Representations of Topic Models" />
          <attvalue for="1" value="  Word embedding models such as the skip-gram learn vector representations of&#10;words' semantic relationships, and document embedding models learn similar&#10;representations for documents. On the other hand, topic models provide latent&#10;representations of the documents' topical themes. To get the benefits of these&#10;representations simultaneously, we propose a unifying algorithm, called neural&#10;embedding allocation (NEA), which deconstructs topic models into interpretable&#10;vector-space embeddings of words, topics, documents, authors, and so on, by&#10;learning neural embeddings to mimic the topic models. We showcase NEA's&#10;effectiveness and generality on LDA, author-topic models and the recently&#10;proposed mixed membership skip gram topic model and achieve better performance&#10;with the embeddings compared to several state-of-the-art models. Furthermore,&#10;we demonstrate that using NEA to smooth out the topics improves coherence&#10;scores over the original topic models when the number of topics is large.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, methods for automatically learning representations of text data have become an essential part of the natural language processing pipeline.&#10;Word embedding models such as the skip-gram improve the performance of Natural Language Processing (NLP) methods by revealing the latent structural relationship between words ~\cite{mikolov2013efficient,mikolov2013distributed}. These embeddings have proven valuable for a variety of NLP tasks such as statistical machine translation~\cite{vaswani2013decoding}, part-of-speech tagging, chunking, and named entity recognition ~\cite{collobert2011natural}. Since word vectors encode distributional information, the similarity relationships between the semantic meanings of the words are reflected in the similarity of the vectors~\cite{sahlgren2008distributional}. Extensions to document embeddings have subsequently been proposed \cite{le2014distributed}.&#10;&#10;On the other hand, topic models such as latent Dirichlet allocation (LDA)~\cite{blei2003latent} construct latent representations of topical themes and of documents, and these can be used to subsequently derive representations for words~\cite{griffiths2007topics}.&#10;Like word embeddings, topic models exploit conditional discrete distribution over words to represent high-dimensional data into a low-dimensional subspace. However, topic models do not directly capture nuanced relationships between words using vector-space embeddings, which are often important for performance on downstream NLP tasks \cite{maas2011learning}. &#10;&#10;We therefore desire a unified method which gains the benefits of both word embeddings (encoding nuanced semantic relationships) and topic models (recovering interpretable topical themes). Some recent research has aimed to combine aspects of topic models and word embeddings. The Gaussian LDA model~\cite{das2015gaussian} tries to improve the performance of topic modeling by prioritizing the semantic information encoded in word embeddings, however, it does not aim to jointly perform word embedding. Unlike Gaussian LDA, the topical word embedding model~\cite{liu2015topical} uses LDA topic assignments of words as input to improve the resultant word embedding. In another approach, mixed membership word embeddings~\cite{foulds2018mixed} aim to recover domain-specific interpretable word embeddings without big data, based on topic embeddings. &#10;&#10;In contrast, in this paper we develop a method which recovers vector-space embeddings of words, documents, topics, authors, and so on, based on a given topic model. Our approach, which we call neural embedding allocation (NEA), is to deconstruct topic models by reparameterizing them using vector-space embeddings. &#10;We can view our method as learning to mimic a topic model with a skip-gram style embedding model to reveal underlying semantic representations. Our approach is thus reminiscent of model distillation for supervised models \cite{bucilua2006model, hinton2015distilling}.&#10;&#10;We train NEA by minimizing the KL-divergence to the data distribution of the corresponding topic model, using a stream of simulated data from the model.&#10;The resulting embeddings allow us to (1) improve the coherence of topic models by ``smoothing out'' noisy topics, (2) improve classification performance by producing topic-informed document vectors, and (3) construct embeddings and smoothed distributions over general topic modeling variables such as authors. We show the benefits and generality of our method by applying it to LDA, author-topic models (ATM)~\cite{rosen2004author}, and the recently proposed mixed membership skip gram topic model (MMSGTM)~\cite{foulds2018mixed}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Machine Learning, Computer Science, Word Embedding Models, Linguistics, Topic Modeling Techniques, Natural Language Processing, Statistics, Neural Network Embeddings" />
        </attvalues>
      </node>
      <node id="1103.0398" label="1103.0398">
        <attvalues>
          <attvalue for="0" value="Natural Language Processing (almost) from Scratch" />
          <attvalue for="1" value="  We propose a unified neural network architecture and learning algorithm that&#10;can be applied to various natural language processing tasks including:&#10;part-of-speech tagging, chunking, named entity recognition, and semantic role&#10;labeling. This versatility is achieved by trying to avoid task-specific&#10;engineering and therefore disregarding a lot of prior knowledge. Instead of&#10;exploiting man-made input features carefully optimized for each task, our&#10;system learns internal representations on the basis of vast amounts of mostly&#10;unlabeled training data. This work is then used as a basis for building a&#10;freely available tagging system with good performance and minimal computational&#10;requirements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1405.4053" label="1405.4053">
        <attvalues>
          <attvalue for="0" value="Distributed Representations of Sentences and Documents" />
          <attvalue for="1" value="  Many machine learning algorithms require the input to be represented as a&#10;fixed-length feature vector. When it comes to texts, one of the most common&#10;fixed-length features is bag-of-words. Despite their popularity, bag-of-words&#10;features have two major weaknesses: they lose the ordering of the words and&#10;they also ignore semantics of the words. For example, &quot;powerful,&quot; &quot;strong&quot; and&#10;&quot;Paris&quot; are equally distant. In this paper, we propose Paragraph Vector, an&#10;unsupervised algorithm that learns fixed-length feature representations from&#10;variable-length pieces of texts, such as sentences, paragraphs, and documents.&#10;Our algorithm represents each document by a dense vector which is trained to&#10;predict words in the document. Its construction gives our algorithm the&#10;potential to overcome the weaknesses of bag-of-words models. Empirical results&#10;show that Paragraph Vectors outperform bag-of-words models as well as other&#10;techniques for text representations. Finally, we achieve new state-of-the-art&#10;results on several text classification and sentiment analysis tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.07368" label="1705.07368">
        <attvalues>
          <attvalue for="0" value="Mixed Membership Word Embeddings for Computational Social Science" />
          <attvalue for="1" value="  Word embeddings improve the performance of NLP systems by revealing the&#10;hidden structural relationships between words. Despite their success in many&#10;applications, word embeddings have seen very little use in computational social&#10;science NLP tasks, presumably due to their reliance on big data, and to a lack&#10;of interpretability. I propose a probabilistic model-based word embedding&#10;method which can recover interpretable embeddings, without big data. The key&#10;insight is to leverage mixed membership modeling, in which global&#10;representations are shared, but individual entities (i.e. dictionary words) are&#10;free to use these representations to uniquely differing degrees. I show how to&#10;train the model using a combination of state-of-the-art training techniques for&#10;word embeddings and topic models. The experimental results show an improvement&#10;in predictive language modeling of up to 63% in MRR over the skip-gram, and&#10;demonstrate that the representations are beneficial for supervised learning. I&#10;illustrate the interpretability of the models with computational social science&#10;case studies on State of the Union addresses and NIPS articles.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.10059" label="2105.10059">
        <attvalues>
          <attvalue for="0" value="Model Compression" />
          <attvalue for="1" value="  With time, machine learning models have increased in their scope,&#10;functionality and size. Consequently, the increased functionality and size of&#10;such models requires high-end hardware to both train and provide inference&#10;after the fact. This paper aims to explore the possibilities within the domain&#10;of model compression, discuss the efficiency of combining various levels of&#10;pruning and quantization, while proposing a quality measurement metric to&#10;objectively decide which combination is best in terms of minimizing the&#10;accuracy delta and maximizing the size reduction factor.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.08039" label="2204.08039">
        <attvalues>
          <attvalue for="0" value="Pathologies of Pre-trained Language Models in Few-shot Fine-tuning" />
          <attvalue for="1" value="  Although adapting pre-trained language models with few examples has shown&#10;promising performance on text classification, there is a lack of understanding&#10;of where the performance gain comes from. In this work, we propose to answer&#10;this question by interpreting the adaptation behavior using post-hoc&#10;explanations from model predictions. By modeling feature statistics of&#10;explanations, we discover that (1) without fine-tuning, pre-trained models&#10;(e.g. BERT and RoBERTa) show strong prediction bias across labels; (2) although&#10;few-shot fine-tuning can mitigate the prediction bias and demonstrate promising&#10;prediction performance, our analysis shows models gain performance improvement&#10;by capturing non-task-related features (e.g. stop words) or shallow data&#10;patterns (e.g. lexical overlaps). These observations alert that pursuing model&#10;performance with fewer examples may incur pathological prediction behavior,&#10;which requires further sanity check on model predictions and careful design in&#10;model evaluations in few-shot fine-tuning.&#10;" />
          <attvalue for="2" value="&#10;Pre-trained language models \cite{NEURIPS2020_1457c0d6, liu2019roberta, devlin-etal-2019-bert} have shown impressive adaptation ability to dowstream tasks, achieving considerable performance even with scarce task-specific training data, i.e., few-shot adaptation \cite{radford2019language, schick-schutze-2021-exploiting, gao-etal-2021-making}. &#10;Existing few-shot adaptation techniques broadly fall in fine-tuning and few-shot learning \cite{shin-etal-2020-autoprompt, schick-schutze-2021-just, chen-etal-2021-revisiting}. &#10;Specifically, fine-tuning includes directly tuning pre-trained language models with few task-specific examples or utilizing a natural-language prompt to transform downstream tasks to masked language modeling task for better mining knowledge from pre-trained models \cite{petroni-etal-2019-language, jiang-etal-2020-know, wang-etal-2021-transprompt}. &#10;Few-shot learning leverages unlabeled data or auxiliary tasks to provide additional information for facilitating model training \cite{zheng2021meta, wang2021meta, du-etal-2021-self}. &#10;&#10;Although much success has been made in adapting pre-trained language models to dowstream tasks with few-shot examples, some issues have been reported. &#10;\cite{utama2021avoiding} found that models obtained from few-shot prompt-based fine-tuning utilize inference heuristics to make predictions on sentence pair classification tasks. &#10;\cite{zhao2021calibrate} discovered the instability of model performance towards different prompts in few-shot learning. &#10;These works mainly look at prompt-based fine-tuning and discover some problems. &#10;&#10;This paper looks into direct fine-tuning and provides a different perspective on understanding model adaptation behavior via post-hoc explanations \cite{strumbelj2010efficient, sundararajan2017axiomatic}. &#10;Specifically, post-hoc explanations identify the important features (tokens) contribute to the model prediction per example. &#10;We model the statistics of important features over prediction labels via local mutual information (LMI) \cite{schuster-etal-2019-towards, du-etal-2021-towards}. &#10;We track the change of feature statistics with the model adapting from pre-trained to fine-tuned and compare it with the statistics of few-shot training examples. &#10;This provides insights on understanding model adaptation behavior and the effect of training data in few-shot settings. &#10;&#10;We evaluate two pre-trained language models, BERT \cite{devlin-etal-2019-bert} and RoBERTa \cite{liu2019roberta}, on three tasks, including sentiment classification, natural language inference, and paraphrase identification. &#10;For each task, we test on both in-domain and out-of-domain datasets to evaluate the generalization of model adaptation performance.&#10;We discover some interesting observations, some of which may have been overlooked in prior work: (1) without fine-tuning, pre-trained models show strong prediction bias across labels; (2) fine-tuning with a few examples can mitigate the prediction bias, but the model prediction behavior may be pathological by focusing on non-task-related features (e.g. stop words); (3) models adjust their prediction behaviors on different labels asynchronously; (4) models can capture the shallow patterns of training data to make predictions.&#10;The insight drawn from the above observations is that pursuing model performance with fewer examples is dangerous and may cause pathologies in model prediction behavior. &#10;We argue that future research on few-shot fine-tuning or learning should do sanity check on model prediction behavior and ensure the performance gain is based on right reasons. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Interpretability, Few-Shot Learning, Computer Science, Linguistics, Artificial Intelligence, Statistics, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2010.15980" label="2010.15980">
        <attvalues>
          <attvalue for="0" value="AutoPrompt: Eliciting Knowledge from Language Models with Automatically&#10;  Generated Prompts" />
          <attvalue for="1" value="  The remarkable success of pretrained language models has motivated the study&#10;of what kinds of knowledge these models learn during pretraining. Reformulating&#10;tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach&#10;for gauging such knowledge, however, its usage is limited by the manual effort&#10;and guesswork required to write suitable prompts. To address this, we develop&#10;AutoPrompt, an automated method to create prompts for a diverse set of tasks,&#10;based on a gradient-guided search. Using AutoPrompt, we show that masked&#10;language models (MLMs) have an inherent capability to perform sentiment&#10;analysis and natural language inference without additional parameters or&#10;finetuning, sometimes achieving performance on par with recent state-of-the-art&#10;supervised models. We also show that our prompts elicit more accurate factual&#10;knowledge from MLMs than the manually created prompts on the LAMA benchmark,&#10;and that MLMs can be used as relation extractors more effectively than&#10;supervised relation extraction models. These results demonstrate that&#10;automatically generated prompts are a viable parameter-free alternative to&#10;existing probing methods, and as pretrained LMs become more sophisticated and&#10;capable, potentially a replacement for finetuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03809" label="1911.03809">
        <attvalues>
          <attvalue for="0" value="Meta Label Correction for Noisy Label Learning" />
          <attvalue for="1" value="  Leveraging weak or noisy supervision for building effective machine learning&#10;models has long been an important research problem. Its importance has further&#10;increased recently due to the growing need for large-scale datasets to train&#10;deep learning models. Weak or noisy supervision could originate from multiple&#10;sources including non-expert annotators or automatic labeling based on&#10;heuristics or user interaction signals. There is an extensive amount of&#10;previous work focusing on leveraging noisy labels. Most notably, recent work&#10;has shown impressive gains by using a meta-learned instance re-weighting&#10;approach where a meta-learning framework is used to assign instance weights to&#10;noisy labels. In this paper, we extend this approach via posing the problem as&#10;label correction problem within a meta-learning framework. We view the label&#10;correction procedure as a meta-process and propose a new meta-learning based&#10;framework termed MLC (Meta Label Correction) for learning with noisy labels.&#10;Specifically, a label correction network is adopted as a meta-model to produce&#10;corrected labels for noisy labels while the main model is trained to leverage&#10;the corrected labeled. Both models are jointly trained by solving a bi-level&#10;optimization problem. We run extensive experiments with different label noise&#10;levels and types on both image recognition and text classification tasks. We&#10;compare the reweighing and correction approaches showing that the correction&#10;framing addresses some of the limitation of reweighting. We also show that the&#10;proposed MLC approach achieves large improvements over previous methods in many&#10;settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04144" label="2109.04144">
        <attvalues>
          <attvalue for="0" value="Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning" />
          <attvalue for="1" value="  Recent prompt-based approaches allow pretrained language models to achieve&#10;strong performances on few-shot finetuning by reformulating downstream tasks as&#10;a language modeling problem. In this work, we demonstrate that, despite its&#10;advantages on low data regimes, finetuned prompt-based models for sentence pair&#10;classification tasks still suffer from a common pitfall of adopting inference&#10;heuristics based on lexical overlap, e.g., models incorrectly assuming a&#10;sentence pair is of the same meaning because they consist of the same set of&#10;words. Interestingly, we find that this particular inference heuristic is&#10;significantly less present in the zero-shot evaluation of the prompt-based&#10;model, indicating how finetuning can be destructive to useful knowledge learned&#10;during the pretraining. We then show that adding a regularization that&#10;preserves pretraining weights is effective in mitigating this destructive&#10;tendency of few-shot finetuning. Our evaluation on three datasets demonstrates&#10;promising improvements on the three corresponding challenge datasets used to&#10;diagnose the inference heuristics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.05267" label="1908.05267">
        <attvalues>
          <attvalue for="0" value="Towards Debiasing Fact Verification Models" />
          <attvalue for="1" value="  Fact verification requires validating a claim in the context of evidence. We&#10;show, however, that in the popular FEVER dataset this might not necessarily be&#10;the case. Claim-only classifiers perform competitively with top evidence-aware&#10;models. In this paper, we investigate the cause of this phenomenon, identifying&#10;strong cues for predicting labels solely based on the claim, without&#10;considering any evidence. We create an evaluation set that avoids those&#10;idiosyncrasies. The performance of FEVER-trained models significantly drops&#10;when evaluated on this test set. Therefore, we introduce a regularization&#10;method which alleviates the effect of bias in the training data, obtaining&#10;improvements on the newly created test set. This work is a step towards a more&#10;sound evaluation of reasoning capabilities in fact verification models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.03654" label="2006.03654">
        <attvalues>
          <attvalue for="0" value="DeBERTa: Decoding-enhanced BERT with Disentangled Attention" />
          <attvalue for="1" value="  Recent progress in pre-trained neural language models has significantly&#10;improved the performance of many natural language processing (NLP) tasks. In&#10;this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT&#10;with disentangled attention) that improves the BERT and RoBERTa models using&#10;two novel techniques. The first is the disentangled attention mechanism, where&#10;each word is represented using two vectors that encode its content and&#10;position, respectively, and the attention weights among words are computed&#10;using disentangled matrices on their contents and relative positions,&#10;respectively. Second, an enhanced mask decoder is used to incorporate absolute&#10;positions in the decoding layer to predict the masked tokens in model&#10;pre-training. In addition, a new virtual adversarial training method is used&#10;for fine-tuning to improve models' generalization. We show that these&#10;techniques significantly improve the efficiency of model pre-training and the&#10;performance of both natural language understanding (NLU) and natural langauge&#10;generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model&#10;trained on half of the training data performs consistently better on a wide&#10;range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%),&#10;on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%).&#10;Notably, we scale up DeBERTa by training a larger version that consists of 48&#10;Transform layers with 1.5 billion parameters. The significant performance boost&#10;makes the single DeBERTa model surpass the human performance on the SuperGLUE&#10;benchmark (Wang et al., 2019a) for the first time in terms of macro-average&#10;score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the&#10;SuperGLUE leaderboard as of January 6, 2021, out performing the human baseline&#10;by a decent margin (90.3 versus 89.8).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Neural Language Models, Natural Language Processing, Model Architecture Optimization" />
        </attvalues>
      </node>
      <node id="2212.10071" label="2212.10071">
        <attvalues>
          <attvalue for="0" value="Large Language Models Are Reasoning Teachers" />
          <attvalue for="1" value="  Recent works have shown that chain-of-thought (CoT) prompting can elicit&#10;language models to solve complex reasoning tasks, step-by-step. However,&#10;prompt-based CoT methods are dependent on very large models such as GPT-3 175B&#10;which are prohibitive to deploy at scale. In this paper, we use these large&#10;models as reasoning teachers to enable complex reasoning in smaller models and&#10;reduce model size requirements by several orders of magnitude. We propose&#10;Fine-tune-CoT, a method that generates reasoning samples from very large&#10;teacher models to fine-tune smaller models. We evaluate our method on a wide&#10;range of public models and complex tasks. We find that Fine-tune-CoT enables&#10;substantial reasoning capability in small models, far outperforming&#10;prompt-based baselines and even the teacher model in many tasks. Additionally,&#10;we extend our method by leveraging the teacher model's ability to generate&#10;multiple distinct rationales for each original sample. Enriching the&#10;fine-tuning data with such diverse reasoning results in a substantial&#10;performance boost across datasets, even for very small models. We conduct&#10;ablations and sample studies to understand the emergence of reasoning&#10;capabilities of student models. Our code implementation and data are available&#10;at https://github.com/itsnamgyu/reasoning-teacher.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Downstream transfer in language models&#10;Much previous work established a ``pre-train and fine-tune'' &#10;paradigm for enhancing LLM performance on downstream tasks~\cite{gpt1__radford2018improving,dong2019pretrain,vaswani2017attention,devlin2018bert}.&#10;However, fine-tuning is not always easily applicable~\cite{hendrycks2019ood}.&#10;More recent literature exhibits a paradigm shift towards ``prompting'' the model to predict the desired output~\cite{liu2021prompt,t5__raffel2020exploring}. Large LMs can exhibit strong performance in this setting~\cite{gpt3__brown2020language}. For smaller models to be able to perform similarly, additional engineering is usually required~\cite{gao-etal-2021-making,schick-schutze-2021-just,schick-etal-2020-automatically}. For more complex tasks, the idea of using samples with explicit reasoning steps for fine-tuning a model~\cite{nye2021scratchpad, gsm8k__cobbe2021training} preceded the approach of&#10;chain-of-thought (CoT) prompting~\cite{wei2022chain}, which enables very large LMs to perform well.&#10;&#10;Chain-of-thought reasoning&#10;In few-shot CoT prompting, the model learns to generate intermediate reasoning steps that lead to a problem solution, after being fed examples of step-by-step reasoning. This enables very good performance on a wide range of tasks. ~\cite{wang2022self}. Additionally, LLMs can perform well in an unsupervised task-agnostic setting, using Zero-shot-CoT~\cite{kojima2022large}. This requires no fine-tuning or task specific conditioning, and substantially outperforms standard zero-shot learning and sometimes even few-shot learning on a wide number of tasks. &#10;&#10;Yet, prior work has shown that CoT requires extremely large models for optimal performance~\cite{hoffmann2022training,chowdhery2022palm}.&#10;In our work, we contrast this by showing how to utilize CoT reasoning methods for smaller models by fine-tuning them on rationales generated by a very large model. &#10;Using various LLM-generated explanations for fine-tuning smaller models has been successfully used in prior work~\cite{li2022explanations}, with a focus on specific single tasks. Also, a similar approach to ours is mentioned in~\cite{huang2022large}; however we note that this concurrent work focuses on using Few-shot-CoT to self-generate fine-tuning examples by and for very large proprietary models.&#10;There is a brief glimpse into fine-tuning on smaller distilled models, but the results are limited to one dataset and very large teacher models that are inaccessible to the general community.&#10;In contrast, we provide a rich set of results and qualitative/quantitative analysis on a wide range of datasets, using open-source models that are small and accessible to everyone.&#10;&#10;Knowledge distillation&#10;&#10;Typically, knowledge distillation (KD) refers to training small models derived from large models in order to reduce model size and latency, while still preserving accuracy and capacity to generalize~\cite{hinton2015distilling, sanh2019distilbert}. Essentially, KD is a form of model compression, making efficient deployment to capacity-limited devices possible~\cite{bucilua2006compression}. We note that our work could also be considered a distant variant of KD~\cite{Gou2021kdsurvey}, similar to works on improving prompt-based methods such as \cite{yoo2021gpt3mix, schick-schutze-2021-just,schick-schutze-2021-exploiting,zelikman2022star}, or works on data-free distillation~\cite{micaelli2019zero,nayak2019zero,shen2021progressive}, where the transfer data is synthetically generated from a large teacher model. Similarly, sequence-level distillation, i.e. training a student model on sequence distributions of a larger teacher, can make neural machine translation more efficient~\cite{kim2016sequence}. Despite being similar in spirit, our method still distinguishes itself from such previous work. The role of the teacher model in our method is to teach the notion of intermediate reasoning. It is not the specific output that is the main supervising signal for reasoning, but rather the generation's structure. Hence, we do not use a standard KD loss function that reflects trying to match the teacher output. &#10;Adding to this, we note that our diverse reasoning is also unusual in the context of KD, where it is e.g. sufficient in practice to only generate one teacher sequence for sequence level distillation.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Cognitive Science, Language Model Reasoning, Artificial Intelligence, Model Size Reduction, Artificial Intelligence Training, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2401.13601" label="2401.13601">
        <attvalues>
          <attvalue for="0" value="MM-LLMs: Recent Advances in MultiModal Large Language Models" />
          <attvalue for="1" value="  In the past year, MultiModal Large Language Models (MM-LLMs) have undergone&#10;substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or&#10;outputs via cost-effective training strategies. The resulting models not only&#10;preserve the inherent reasoning and decision-making capabilities of LLMs but&#10;also empower a diverse range of MM tasks. In this paper, we provide a&#10;comprehensive survey aimed at facilitating further research of MM-LLMs.&#10;Initially, we outline general design formulations for model architecture and&#10;training pipeline. Subsequently, we introduce a taxonomy encompassing 126&#10;MM-LLMs, each characterized by its specific formulations. Furthermore, we&#10;review the performance of selected MM-LLMs on mainstream benchmarks and&#10;summarize key training recipes to enhance the potency of MM-LLMs. Finally, we&#10;explore promising directions for MM-LLMs while concurrently maintaining a&#10;real-time tracking website for the latest developments in the field. We hope&#10;that this survey contributes to the ongoing advancement of the MM-LLMs domain.&#10;" />
          <attvalue for="2" value="&#10;&#10;MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}. However, as the scale of models and datasets continues to expand, traditional MM models incur substantial computational costs, particularly when trained from scratch.&#10;Recognizing that MM research operates at the intersectionof various modalities, a logical approach is to capitalize on readily available pre-trained unimodal foundation models, with a special emphasis on powerful Large Language Models (LLMs)~\cite{chatgpt}. This strategy aims to mitigate computational expenses and enhance the efficacy of MM pre-training, leading to the emergence of a novel field: MM-LLMs.&#10;&#10;MM-LLMs harness LLMs as the cognitive powerhouse to empower various MM tasks. LLMs contribute desirable properties like robust language generation, zero-shot transfer capabilities, and In-Context Learning (ICL). Concurrently, foundation models in other modalities provide high-quality representations.&#10;Considering foundation models from different modalities are individually pre-trained, the core challenge facing MM-LLMs is how to effectively connect LLMs with models in other modalities to enable collaborative inference.&#10;The predominant focus within this field has been on refining alignment between modalities and aligning with human intent via a MM Pre-Training (PT) + MM Instruction-Tuning (IT) pipeline. &#10;&#10;With the debut of GPT-4(Vision)~\cite{openai2023gpt4} and Gemini~\cite{team2023gemini}, showcasing impressive MM understanding and generation capabilities, a research fervor on MM-LLMs has been sparked. Initial research primarily focuses on MM content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like BLIP-2~\cite{DBLP:conf/icml/0008LSH23}, LLaVA~\cite{liu2023llava}, MiniGPT-4~\cite{zhu2023minigpt}, and OpenFlamingo~\cite{awadalla2023openflamingo}; video-text understanding, as demonstrated by initiatives such as VideoChat~\cite{li2023videochat}, Video-ChatGPT~\cite{maaz2023video}, and LLaMA-VID~\cite{li2023llama}; and audio-text understanding, as seen in projects like Qwen-Audio~\cite{chu2023qwen}.&#10;Later, the capabilities of MM-LLMs have been expanded to support specific modality outputs. This includes tasks with image-text output, such as GILL~\cite{koh2023generating}, Kosmos-2~\cite{peng2023kosmos}, Emu~\cite{sun2023generative}, and MiniGPT-5~\cite{zheng2023minigpt}; as well as speech/audio-text output, exemplified by projects like SpeechGPT~\cite{DBLP:conf/emnlp/ZhangLZZWZQ23} and AudioPaLM~\cite{rubenstein2023audiopalm}.&#10;Recent research endeavors have focused on mimicking human-like any-to-any modality conversion, shedding light on the path to artificial general intelligence. Some efforts aim to amalgamate LLMs with external tools to reach an approaching any-to-any MM comprehension and generation, such as Visual-ChatGPT~\cite{wu2023visual}, HuggingGPT~\cite{shen2023hugginggpt}, and AudioGPT~\cite{huang2023audiogpt}.&#10;Conversely, to mitigate propagated errors in the cascade system, initiatives like NExT-GPT~\cite{wu2023next}, CoDi-2~\cite{tang2023codi}, and ModaVerse~\cite{wang2024modaverse} have developed end-to-end MM-LLMs of arbitrary modalities. The timeline of MM-LLMs is depicted in Figure~\ref{fig_timeline}.&#10;&#10;In this paper, we present a comprehensive survey aimed at facilitating further research of MM-LLMs. To provide readers with a holistic understanding of MM-LLMs, we initially delineate general design formulations from model architecture (Section~\ref{struct}) and training pipeline (Section~\ref{pipe}). We break down the general model architecture into five components: Modality Encoder (Section~\ref{encoder}), Input Projector (Section~\ref{input}), LLM Backbone (Section~\ref{llm}), Output Projector (Section~\ref{output}), and Modality Generator (Section~\ref{decoder}). &#10;The training pipeline elucidates how to enhance a pre-trained text-only LLM to support MM input or output, primarily consisting of two stages: MM PT (Section~\ref{mmpt}) and MM IT (Section~\ref{mmit}). In that section, we also provide a summary of mainstream datasets for MM PT and MM IT.&#10;Next, we establish a taxonomy encompassing $126$ State-of-the-Art (SOTA) MM-LLMs, each characterized by specific formulations, and summarize their development trends in Section~\ref{model}. &#10;In Section~\ref{benchmark}, we comprehensively review the performance of major MM-LLMs on mainstream benchmarks and distill key training recipes to enhance the efficacy of MM-LLMs. In Section~\ref{future}, we offer promising directions for MM-LLMs research.&#10;Moreover, we have established a website (\href{ to track the latest progress of MM-LLMs and facilitate crowd-sourcing updates. Finally, we summarize the entire paper in Section~\ref{conclusion} and discuss related surveys on MM-LLMs in Appendix~\ref{ap:rw}. We aspire for our survey to aid researchers in gaining a deeper understanding of this field and to inspire the design of more effective MM-LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Model Architecture Design, Large Language Model Training, Artificial Intelligence, MultiModal Language Models" />
        </attvalues>
      </node>
      <node id="2004.06165" label="2004.06165">
        <attvalues>
          <attvalue for="0" value="Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks" />
          <attvalue for="1" value="  Large-scale pre-training methods of learning cross-modal representations on&#10;image-text pairs are becoming popular for vision-language tasks. While existing&#10;methods simply concatenate image region features and text features as input to&#10;the model to be pre-trained and use self-attention to learn image-text semantic&#10;alignments in a brute force manner, in this paper, we propose a new learning&#10;method Oscar (Object-Semantics Aligned Pre-training), which uses object tags&#10;detected in images as anchor points to significantly ease the learning of&#10;alignments. Our method is motivated by the observation that the salient objects&#10;in an image can be accurately detected, and are often mentioned in the paired&#10;text. We pre-train an Oscar model on the public corpus of 6.5 million&#10;text-image pairs, and fine-tune it on downstream tasks, creating new&#10;state-of-the-arts on six well-established vision-language understanding and&#10;generation tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.11178" label="2104.11178">
        <attvalues>
          <attvalue for="0" value="VATT: Transformers for Multimodal Self-Supervised Learning from Raw&#10;  Video, Audio and Text" />
          <attvalue for="1" value="  We present a framework for learning multimodal representations from unlabeled&#10;data using convolution-free Transformer architectures. Specifically, our&#10;Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts&#10;multimodal representations that are rich enough to benefit a variety of&#10;downstream tasks. We train VATT end-to-end from scratch using multimodal&#10;contrastive losses and evaluate its performance by the downstream tasks of&#10;video action recognition, audio event classification, image classification, and&#10;text-to-video retrieval. Furthermore, we study a modality-agnostic,&#10;single-backbone Transformer by sharing weights among the three modalities. We&#10;show that the convolution-free VATT outperforms state-of-the-art ConvNet-based&#10;architectures in the downstream tasks. Especially, VATT's vision Transformer&#10;achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600,&#10;72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding&#10;supervised pre-training. Transferring to image classification leads to 78.7%&#10;top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer&#10;from scratch, showing the generalizability of our model despite the domain gap&#10;between videos and images. VATT's audio Transformer also sets a new record on&#10;waveform-based audio event recognition by achieving the mAP of 39.4% on&#10;AudioSet without any supervised pre-training. VATT's source code is publicly&#10;available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.11097" label="2106.11097">
        <attvalues>
          <attvalue for="0" value="CLIP2Video: Mastering Video-Text Retrieval via Image CLIP" />
          <attvalue for="1" value="  We present CLIP2Video network to transfer the image-language pre-training&#10;model to video-text retrieval in an end-to-end manner. Leading approaches in&#10;the domain of video-and-language learning try to distill the spatio-temporal&#10;video features and multi-modal interaction between videos and languages from a&#10;large-scale video-text dataset. Different from them, we leverage pretrained&#10;image-language model, simplify it as a two-stage framework with co-learning of&#10;image-text and enhancing temporal relations between video frames and video-text&#10;respectively, make it able to train on comparatively small datasets.&#10;Specifically, based on the spatial semantics captured by Contrastive&#10;Language-Image Pretraining (CLIP) model, our model involves a Temporal&#10;Difference Block to capture motions at fine temporal video frames, and a&#10;Temporal Alignment Block to re-align the tokens of video clips and phrases and&#10;enhance the multi-modal correlation. We conduct thorough ablation studies, and&#10;achieve state-of-the-art performance on major text-to-video and video-to-text&#10;retrieval benchmarks, including new records of retrieval accuracy on MSR-VTT,&#10;MSVD and VATEX.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.01194" label="2112.01194">
        <attvalues>
          <attvalue for="0" value="Video-Text Pre-training with Learned Regions" />
          <attvalue for="1" value="  Video-Text pre-training aims at learning transferable representations from&#10;large-scale video-text pairs via aligning the semantics between visual and&#10;textual information. State-of-the-art approaches extract visual features from&#10;raw pixels in an end-to-end fashion. However, these methods operate at&#10;frame-level directly and thus overlook the spatio-temporal structure of objects&#10;in video, which yet has a strong synergy with nouns in textual descriptions. In&#10;this work, we propose a simple yet effective module for video-text&#10;representation learning, namely RegionLearner, which can take into account the&#10;structure of objects during pre-training on large-scale video-text pairs. Given&#10;a video, our module (1) first quantizes visual features into semantic clusters,&#10;then (2) generates learnable masks and uses them to aggregate the features&#10;belonging to the same semantic region, and finally (3) models the interactions&#10;between different aggregated regions. In contrast to using off-the-shelf object&#10;detectors, our proposed module does not require explicit supervision and is&#10;much more computationally efficient. We pre-train the proposed approach on the&#10;public WebVid2M and CC3M datasets. Extensive evaluations on four downstream&#10;video-text retrieval benchmarks clearly demonstrate the effectiveness of our&#10;RegionLearner. The code will be available at&#10;https://github.com/ruiyan1995/Region_Learner.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.07651" label="2107.07651">
        <attvalues>
          <attvalue for="0" value="Align before Fuse: Vision and Language Representation Learning with&#10;  Momentum Distillation" />
          <attvalue for="1" value="  Large-scale vision and language representation learning has shown promising&#10;improvements on various vision-language tasks. Most existing methods employ a&#10;transformer-based multimodal encoder to jointly model visual tokens&#10;(region-based image features) and word tokens. Because the visual tokens and&#10;word tokens are unaligned, it is challenging for the multimodal encoder to&#10;learn image-text interactions. In this paper, we introduce a contrastive loss&#10;to ALign the image and text representations BEfore Fusing (ALBEF) them through&#10;cross-modal attention, which enables more grounded vision and language&#10;representation learning. Unlike most existing methods, our method does not&#10;require bounding box annotations nor high-resolution images. In order to&#10;improve learning from noisy web data, we propose momentum distillation, a&#10;self-training method which learns from pseudo-targets produced by a momentum&#10;model. We provide a theoretical analysis of ALBEF from a mutual information&#10;maximization perspective, showing that different training tasks can be&#10;interpreted as different ways to generate views for an image-text pair. ALBEF&#10;achieves state-of-the-art performance on multiple downstream vision-language&#10;tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained&#10;on orders of magnitude larger datasets. On VQA and NLVR$^2$, ALBEF achieves&#10;absolute improvements of 2.37% and 3.84% compared to the state-of-the-art,&#10;while enjoying faster inference speed. Code and pre-trained models are&#10;available at https://github.com/salesforce/ALBEF/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.12086" label="2201.12086">
        <attvalues>
          <attvalue for="0" value="BLIP: Bootstrapping Language-Image Pre-training for Unified&#10;  Vision-Language Understanding and Generation" />
          <attvalue for="1" value="  Vision-Language Pre-training (VLP) has advanced the performance for many&#10;vision-language tasks. However, most existing pre-trained models only excel in&#10;either understanding-based tasks or generation-based tasks. Furthermore,&#10;performance improvement has been largely achieved by scaling up the dataset&#10;with noisy image-text pairs collected from the web, which is a suboptimal&#10;source of supervision. In this paper, we propose BLIP, a new VLP framework&#10;which transfers flexibly to both vision-language understanding and generation&#10;tasks. BLIP effectively utilizes the noisy web data by bootstrapping the&#10;captions, where a captioner generates synthetic captions and a filter removes&#10;the noisy ones. We achieve state-of-the-art results on a wide range of&#10;vision-language tasks, such as image-text retrieval (+2.7% in average&#10;recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score).&#10;BLIP also demonstrates strong generalization ability when directly transferred&#10;to video-language tasks in a zero-shot manner. Code, models, and datasets are&#10;released at https://github.com/salesforce/BLIP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.02639" label="2201.02639">
        <attvalues>
          <attvalue for="0" value="MERLOT Reserve: Neural Script Knowledge through Vision and Language and&#10;  Sound" />
          <attvalue for="1" value="  As humans, we navigate a multimodal world, building a holistic understanding&#10;from all our senses. We introduce MERLOT Reserve, a model that represents&#10;videos jointly over time -- through a new training objective that learns from&#10;audio, subtitles, and video frames. Given a video, we replace snippets of text&#10;and audio with a MASK token; the model learns by choosing the correct&#10;masked-out snippet. Our objective learns faster than alternatives, and performs&#10;well at scale: we pretrain on 20 million YouTube videos.&#10;  Empirical results show that MERLOT Reserve learns strong multimodal&#10;representations. When finetuned, it sets state-of-the-art on Visual Commonsense&#10;Reasoning (VCR), TVQA, and Kinetics-600; outperforming prior work by 5%, 7%,&#10;and 1.5% respectively. Ablations show that these tasks benefit from audio&#10;pretraining -- even VCR, a QA task centered around images (without sound).&#10;Moreover, our objective enables out-of-the-box prediction, revealing strong&#10;multimodal commonsense understanding. In a fully zero-shot setting, our model&#10;obtains competitive results on four video tasks, even outperforming supervised&#10;approaches on the recently proposed Situated Reasoning (STAR) benchmark.&#10;  We analyze why audio enables better vision-language representations,&#10;suggesting significant opportunities for future research. We conclude by&#10;discussing ethical and societal implications of multimodal pretraining.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.08276" label="2111.08276">
        <attvalues>
          <attvalue for="0" value="Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual&#10;  Concepts" />
          <attvalue for="1" value="  Most existing methods in vision language pre-training rely on object-centric&#10;features extracted through object detection and make fine-grained alignments&#10;between the extracted features and texts. It is challenging for these methods&#10;to learn relations among multiple objects. To this end, we propose a new method&#10;called X-VLM to perform `multi-grained vision language pre-training.' The key&#10;to learning multi-grained alignments is to locate visual concepts in the image&#10;given the associated texts, and in the meantime align the texts with the visual&#10;concepts, where the alignments are in multi-granularity. Experimental results&#10;show that X-VLM effectively leverages the learned multi-grained alignments to&#10;many downstream vision language tasks and consistently outperforms&#10;state-of-the-art methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.10401" label="2202.10401">
        <attvalues>
          <attvalue for="0" value="Vision-Language Pre-Training with Triple Contrastive Learning" />
          <attvalue for="1" value="  Vision-language representation learning largely benefits from image-text&#10;alignment through contrastive losses (e.g., InfoNCE loss). The success of this&#10;alignment strategy is attributed to its capability in maximizing the mutual&#10;information (MI) between an image and its matched text. However, simply&#10;performing cross-modal alignment (CMA) ignores data potential within each&#10;modality, which may result in degraded representations. For instance, although&#10;CMA-based models are able to map image-text pairs close together in the&#10;embedding space, they fail to ensure that similar inputs from the same modality&#10;stay close by. This problem can get even worse when the pre-training data is&#10;noisy. In this paper, we propose triple contrastive learning (TCL) for&#10;vision-language pre-training by leveraging both cross-modal and intra-modal&#10;self-supervision. Besides CMA, TCL introduces an intra-modal contrastive&#10;objective to provide complementary benefits in representation learning. To take&#10;advantage of localized and structural information from image and text input,&#10;TCL further maximizes the average MI between local regions of image/text and&#10;their global summary. To the best of our knowledge, ours is the first work that&#10;takes into account local structure information for multi-modality&#10;representation learning. Experimental evaluations show that our approach is&#10;competitive and achieves the new state of the art on various common down-stream&#10;vision-language tasks such as image-text retrieval and visual question&#10;answering.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.03052" label="2202.03052">
        <attvalues>
          <attvalue for="0" value="OFA: Unifying Architectures, Tasks, and Modalities Through a Simple&#10;  Sequence-to-Sequence Learning Framework" />
          <attvalue for="1" value="  In this work, we pursue a unified paradigm for multimodal pretraining to&#10;break the scaffolds of complex task/modality-specific customization. We propose&#10;OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task&#10;Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks,&#10;including image generation, visual grounding, image captioning, image&#10;classification, language modeling, etc., in a simple sequence-to-sequence&#10;learning framework. OFA follows the instruction-based learning in both&#10;pretraining and finetuning stages, requiring no extra task-specific layers for&#10;downstream tasks. In comparison with the recent state-of-the-art vision &amp;&#10;language models that rely on extremely large cross-modal datasets, OFA is&#10;pretrained on only 20M publicly available image-text pairs. Despite its&#10;simplicity and relatively small-scale training data, OFA achieves new SOTAs in&#10;a series of cross-modal tasks while attaining highly competitive performances&#10;on uni-modal tasks. Our further analysis indicates that OFA can also&#10;effectively transfer to unseen tasks and unseen domains. Our code and models&#10;are publicly available at https://github.com/OFA-Sys/OFA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.10442" label="2208.10442">
        <attvalues>
          <attvalue for="0" value="Image as a Foreign Language: BEiT Pretraining for All Vision and&#10;  Vision-Language Tasks" />
          <attvalue for="1" value="  A big convergence of language, vision, and multimodal pretraining is&#10;emerging. In this work, we introduce a general-purpose multimodal foundation&#10;model BEiT-3, which achieves state-of-the-art transfer performance on both&#10;vision and vision-language tasks. Specifically, we advance the big convergence&#10;from three aspects: backbone architecture, pretraining task, and model scaling&#10;up. We introduce Multiway Transformers for general-purpose modeling, where the&#10;modular architecture enables both deep fusion and modality-specific encoding.&#10;Based on the shared backbone, we perform masked &quot;language&quot; modeling on images&#10;(Imglish), texts (English), and image-text pairs (&quot;parallel sentences&quot;) in a&#10;unified manner. Experimental results show that BEiT-3 obtains state-of-the-art&#10;performance on object detection (COCO), semantic segmentation (ADE20K), image&#10;classification (ImageNet), visual reasoning (NLVR2), visual question answering&#10;(VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.12597" label="2301.12597">
        <attvalues>
          <attvalue for="0" value="BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image&#10;  Encoders and Large Language Models" />
          <attvalue for="1" value="  The cost of vision-and-language pre-training has become increasingly&#10;prohibitive due to end-to-end training of large-scale models. This paper&#10;proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps&#10;vision-language pre-training from off-the-shelf frozen pre-trained image&#10;encoders and frozen large language models. BLIP-2 bridges the modality gap with&#10;a lightweight Querying Transformer, which is pre-trained in two stages. The&#10;first stage bootstraps vision-language representation learning from a frozen&#10;image encoder. The second stage bootstraps vision-to-language generative&#10;learning from a frozen language model. BLIP-2 achieves state-of-the-art&#10;performance on various vision-language tasks, despite having significantly&#10;fewer trainable parameters than existing methods. For example, our model&#10;outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable&#10;parameters. We also demonstrate the model's emerging capabilities of zero-shot&#10;image-to-text generation that can follow natural language instructions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.10592" label="2304.10592">
        <attvalues>
          <attvalue for="0" value="MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large&#10;  Language Models" />
          <attvalue for="1" value="  The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such&#10;as directly generating websites from handwritten text and identifying humorous&#10;elements within images. These features are rarely observed in previous&#10;vision-language models. However, the technical details behind GPT-4 continue to&#10;remain undisclosed. We believe that the enhanced multi-modal generation&#10;capabilities of GPT-4 stem from the utilization of sophisticated large language&#10;models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a&#10;frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection&#10;layer. Our work, for the first time, uncovers that properly aligning the visual&#10;features with an advanced large language model can possess numerous advanced&#10;multi-modal abilities demonstrated by GPT-4, such as detailed image description&#10;generation and website creation from hand-drawn drafts. Furthermore, we also&#10;observe other emerging capabilities in MiniGPT-4, including writing stories and&#10;poems inspired by given images, teaching users how to cook based on food&#10;photos, and so on. In our experiment, we found that the model trained on short&#10;image caption pairs could produce unnatural language outputs (e.g., repetition&#10;and fragmentation). To address this problem, we curate a detailed image&#10;description dataset in the second stage to finetune the model, which&#10;consequently improves the model's generation reliability and overall usability.&#10;Our code, pre-trained model, and collected dataset are available at&#10;https://minigpt-4.github.io/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.01390" label="2308.01390">
        <attvalues>
          <attvalue for="0" value="OpenFlamingo: An Open-Source Framework for Training Large Autoregressive&#10;  Vision-Language Models" />
          <attvalue for="1" value="  We introduce OpenFlamingo, a family of autoregressive vision-language models&#10;ranging from 3B to 9B parameters. OpenFlamingo is an ongoing effort to produce&#10;an open-source replication of DeepMind's Flamingo models. On seven&#10;vision-language datasets, OpenFlamingo models average between 80 - 89% of&#10;corresponding Flamingo performance. This technical report describes our models,&#10;training data, hyperparameters, and evaluation suite. We share our models and&#10;code at https://github.com/mlfoundations/open_flamingo.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.06355" label="2305.06355">
        <attvalues>
          <attvalue for="0" value="VideoChat: Chat-Centric Video Understanding" />
          <attvalue for="1" value="  In this paper, we initiate an attempt of developing an end-to-end&#10;chat-centric video understanding system, coined as VideoChat. It integrates&#10;video foundation models and large language models via a learnable neural&#10;interface, excelling in spatiotemporal reasoning, event localization, and&#10;causal relationship inference. To instructively tune this system, we build a&#10;video-centric instruction dataset, composed of thousands of videos associated&#10;with detailed descriptions and conversations. This dataset emphasizes&#10;spatiotemporal reasoning and captures causal relationships, providing a&#10;valuable asset for training our chat-centric video understanding system.&#10;Preliminary qualitative experiments demonstrate the potential of our system&#10;across a broad spectrum of video applications, which could serve as a simple&#10;prototype system for future research on chat-centric video understanding.&#10;Access our code and data at https://github.com/OpenGVLab/Ask-Anything&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.17043" label="2311.17043">
        <attvalues>
          <attvalue for="0" value="LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models" />
          <attvalue for="1" value="  In this work, we present a novel method to tackle the token generation&#10;challenge in Vision Language Models (VLMs) for video and image understanding,&#10;called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning&#10;and visual question answering, face computational burdens when processing long&#10;videos due to the excessive visual tokens. LLaMA-VID addresses this issue by&#10;representing each frame with two distinct tokens, namely context token and&#10;content token. The context token encodes the overall image context based on&#10;user input, whereas the content token encapsulates visual cues in each frame.&#10;This dual-token strategy significantly reduces the overload of long videos&#10;while preserving critical information. Generally, LLaMA-VID empowers existing&#10;frameworks to support hour-long videos and pushes their upper limit with an&#10;extra context token. It is proved to surpass previous methods on most of video-&#10;or image-based benchmarks. Code is available&#10;https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.17216" label="2305.17216">
        <attvalues>
          <attvalue for="0" value="Generating Images with Multimodal Language Models" />
          <attvalue for="1" value="  We propose a method to fuse frozen text-only large language models (LLMs)&#10;with pre-trained image encoder and decoder models, by mapping between their&#10;embedding spaces. Our model demonstrates a wide suite of multimodal&#10;capabilities: image retrieval, novel image generation, and multimodal dialogue.&#10;Ours is the first approach capable of conditioning on arbitrarily interleaved&#10;image and text inputs to generate coherent image (and text) outputs. To achieve&#10;strong performance on image generation, we propose an efficient mapping network&#10;to ground the LLM to an off-the-shelf text-to-image generation model. This&#10;mapping network translates hidden representations of text into the embedding&#10;space of the visual models, enabling us to leverage the strong text&#10;representations of the LLM for visual outputs. Our approach outperforms&#10;baseline generation models on tasks with longer and more complex language. In&#10;addition to novel image generation, our model is also capable of image&#10;retrieval from a prespecified dataset, and decides whether to retrieve or&#10;generate at inference time. This is done with a learnt decision module which&#10;conditions on the hidden representations of the LLM. Our model exhibits a wider&#10;range of capabilities compared to prior multimodal language models. It can&#10;process image-and-text inputs, and produce retrieved images, generated images,&#10;and generated text -- outperforming non-LLM based generation models across&#10;several text-to-image tasks that measure context dependence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.14824" label="2306.14824">
        <attvalues>
          <attvalue for="0" value="Kosmos-2: Grounding Multimodal Large Language Models to the World" />
          <attvalue for="1" value="  We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new&#10;capabilities of perceiving object descriptions (e.g., bounding boxes) and&#10;grounding text to the visual world. Specifically, we represent refer&#10;expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where&#10;object descriptions are sequences of location tokens. Together with multimodal&#10;corpora, we construct large-scale data of grounded image-text pairs (called&#10;GrIT) to train the model. In addition to the existing capabilities of MLLMs&#10;(e.g., perceiving general modalities, following instructions, and performing&#10;in-context learning), Kosmos-2 integrates the grounding capability into&#10;downstream applications. We evaluate Kosmos-2 on a wide range of tasks,&#10;including (i) multimodal grounding, such as referring expression comprehension,&#10;and phrase grounding, (ii) multimodal referring, such as referring expression&#10;generation, (iii) perception-language tasks, and (iv) language understanding&#10;and generation. This work lays out the foundation for the development of&#10;Embodiment AI and sheds light on the big convergence of language, multimodal&#10;perception, action, and world modeling, which is a key step toward artificial&#10;general intelligence. Code and pretrained models are available at&#10;https://aka.ms/kosmos-2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.02239" label="2310.02239">
        <attvalues>
          <attvalue for="0" value="MiniGPT-5: Interleaved Vision-and-Language Generation via Generative&#10;  Vokens" />
          <attvalue for="1" value="  The effectiveness of Multimodal Large Language Models (MLLMs) demonstrates a&#10;profound capability in multimodal understanding. However, the simultaneous&#10;generation of images with coherent texts is still underdeveloped. Addressing&#10;this, we introduce a novel interleaved vision-and-language generation method,&#10;centered around the concept of ``generative vokens&quot;. These vokens serve as&#10;pivotal elements contributing to coherent image-text outputs. Our method is&#10;marked by a unique two-stage training strategy for description-free multimodal&#10;generation, which does not necessitate extensive descriptions of images. We&#10;integrate classifier-free guidance to enhance the alignment of generated images&#10;and texts, ensuring more seamless and contextually relevant multimodal&#10;interactions. Our model, MiniGPT-5, exhibits substantial improvement over the&#10;baseline models on multimodal generation datasets, including MMDialog and VIST.&#10;The human evaluation shows MiniGPT-5 is better than the baseline model on more&#10;than 56\% cases for multimodal generation, highlighting its efficacy across&#10;diverse benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11000" label="2305.11000">
        <attvalues>
          <attvalue for="0" value="SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal&#10;  Conversational Abilities" />
          <attvalue for="1" value="  Multi-modal large language models are regarded as a crucial step towards&#10;Artificial General Intelligence (AGI) and have garnered significant interest&#10;with the emergence of ChatGPT. However, current speech-language models&#10;typically adopt the cascade paradigm, preventing inter-modal knowledge&#10;transfer. In this paper, we propose SpeechGPT, a large language model with&#10;intrinsic cross-modal conversational abilities, capable of perceiving and&#10;generating multi-model content. With discrete speech representations, we first&#10;construct SpeechInstruct, a large-scale cross-modal speech instruction dataset.&#10;Additionally, we employ a three-stage training strategy that includes&#10;modality-adaptation pre-training, cross-modal instruction fine-tuning, and&#10;chain-of-modality instruction fine-tuning. The experimental results demonstrate&#10;that SpeechGPT has an impressive capacity to follow multi-modal human&#10;instructions and highlight the potential of handling multiple modalities with&#10;one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.12925" label="2306.12925">
        <attvalues>
          <attvalue for="0" value="AudioPaLM: A Large Language Model That Can Speak and Listen" />
          <attvalue for="1" value="  We introduce AudioPaLM, a large language model for speech understanding and&#10;generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2&#10;[Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified&#10;multimodal architecture that can process and generate text and speech with&#10;applications including speech recognition and speech-to-speech translation.&#10;AudioPaLM inherits the capability to preserve paralinguistic information such&#10;as speaker identity and intonation from AudioLM and the linguistic knowledge&#10;present only in text large language models such as PaLM-2. We demonstrate that&#10;initializing AudioPaLM with the weights of a text-only large language model&#10;improves speech processing, successfully leveraging the larger quantity of text&#10;training data used in pretraining to assist with the speech tasks. The&#10;resulting model significantly outperforms existing systems for speech&#10;translation tasks and has the ability to perform zero-shot speech-to-text&#10;translation for many languages for which input/target language combinations&#10;were not seen in training. AudioPaLM also demonstrates features of audio&#10;language models, such as transferring a voice across languages based on a short&#10;spoken prompt. We release examples of our method at&#10;https://google-research.github.io/seanet/audiopalm/examples&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.04671" label="2303.04671">
        <attvalues>
          <attvalue for="0" value="Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation&#10;  Models" />
          <attvalue for="1" value="  ChatGPT is attracting a cross-field interest as it provides a language&#10;interface with remarkable conversational competency and reasoning capabilities&#10;across many domains. However, since ChatGPT is trained with languages, it is&#10;currently not capable of processing or generating images from the visual world.&#10;At the same time, Visual Foundation Models, such as Visual Transformers or&#10;Stable Diffusion, although showing great visual understanding and generation&#10;capabilities, they are only experts on specific tasks with one-round fixed&#10;inputs and outputs. To this end, We build a system called \textbf{Visual&#10;ChatGPT}, incorporating different Visual Foundation Models, to enable the user&#10;to interact with ChatGPT by 1) sending and receiving not only languages but&#10;also images 2) providing complex visual questions or visual editing&#10;instructions that require the collaboration of multiple AI models with&#10;multi-steps. 3) providing feedback and asking for corrected results. We design&#10;a series of prompts to inject the visual model information into ChatGPT,&#10;considering models of multiple inputs/outputs and models that require visual&#10;feedback. Experiments show that Visual ChatGPT opens the door to investigating&#10;the visual roles of ChatGPT with the help of Visual Foundation Models. Our&#10;system is publicly available at&#10;\url{https://github.com/microsoft/visual-chatgpt}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.12995" label="2304.12995">
        <attvalues>
          <attvalue for="0" value="AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking&#10;  Head" />
          <attvalue for="1" value="  Large language models (LLMs) have exhibited remarkable capabilities across a&#10;variety of domains and tasks, challenging our understanding of learning and&#10;cognition. Despite the recent success, current LLMs are not capable of&#10;processing complex audio information or conducting spoken conversations (like&#10;Siri or Alexa). In this work, we propose a multi-modal AI system named&#10;AudioGPT, which complements LLMs (i.e., ChatGPT) with 1) foundation models to&#10;process complex audio information and solve numerous understanding and&#10;generation tasks; and 2) the input/output interface (ASR, TTS) to support&#10;spoken dialogue. With an increasing demand to evaluate multi-modal LLMs of&#10;human intention understanding and cooperation with foundation models, we&#10;outline the principles and processes and test AudioGPT in terms of consistency,&#10;capability, and robustness. Experimental results demonstrate the capabilities&#10;of AudioGPT in solving AI tasks with speech, music, sound, and talking head&#10;understanding and generation in multi-round dialogues, which empower humans to&#10;create rich and diverse audio content with unprecedented ease. Our system is&#10;publicly available at \url{https://github.com/AIGC-Audio/AudioGPT}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.05519" label="2309.05519">
        <attvalues>
          <attvalue for="0" value="NExT-GPT: Any-to-Any Multimodal LLM" />
          <attvalue for="1" value="  While recently Multimodal Large Language Models (MM-LLMs) have made exciting&#10;strides, they mostly fall prey to the limitation of only input-side multimodal&#10;understanding, without the ability to produce content in multiple modalities.&#10;As we humans always perceive the world and communicate with people through&#10;various modalities, developing any-to-any MM-LLMs capable of accepting and&#10;delivering content in any modality becomes essential to human-level AI. To fill&#10;the gap, we present an end-to-end general-purpose any-to-any MM-LLM system,&#10;NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion&#10;decoders, enabling NExT-GPT to perceive inputs and generate outputs in&#10;arbitrary combinations of text, images, videos, and audio. By leveraging the&#10;existing well-trained highly-performing encoders and decoders, NExT-GPT is&#10;tuned with only a small amount of parameter (1%) of certain projection layers,&#10;which not only benefits low-cost training and also facilitates convenient&#10;expansion to more potential modalities. Moreover, we introduce a&#10;modality-switching instruction tuning (MosIT) and manually curate a&#10;high-quality dataset for MosIT, based on which NExT-GPT is empowered with&#10;complex cross-modal semantic understanding and content generation. Overall, our&#10;research showcases the promising possibility of building an AI agent capable of&#10;modeling universal modalities, paving the way for more human-like AI research&#10;in the community. Project page: https://next-gpt.github.io/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11846" label="2305.11846">
        <attvalues>
          <attvalue for="0" value="Any-to-Any Generation via Composable Diffusion" />
          <attvalue for="1" value="  We present Composable Diffusion (CoDi), a novel generative model capable of&#10;generating any combination of output modalities, such as language, image,&#10;video, or audio, from any combination of input modalities. Unlike existing&#10;generative AI systems, CoDi can generate multiple modalities in parallel and&#10;its input is not limited to a subset of modalities like text or image. Despite&#10;the absence of training datasets for many combinations of modalities, we&#10;propose to align modalities in both the input and output space. This allows&#10;CoDi to freely condition on any input combination and generate any group of&#10;modalities, even if they are not present in the training data. CoDi employs a&#10;novel composable generation strategy which involves building a shared&#10;multimodal space by bridging alignment in the diffusion process, enabling the&#10;synchronized generation of intertwined modalities, such as temporally aligned&#10;video and audio. Highly customizable and flexible, CoDi achieves strong&#10;joint-modality generation quality, and outperforms or is on par with the&#10;unimodal state-of-the-art for single-modality synthesis. The project page with&#10;demonstrations and code is at https://codi-gen.github.io&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.06395" label="2401.06395">
        <attvalues>
          <attvalue for="0" value="ModaVerse: Efficiently Transforming Modalities with LLMs" />
          <attvalue for="1" value="  Humans possess the capability to comprehend diverse modalities and seamlessly&#10;transfer information between them. In this work, we introduce ModaVerse, a&#10;Multi-modal Large Language Model (MLLM) capable of comprehending and&#10;transforming content across various modalities including images, videos, and&#10;audio. Predominant MLLM frameworks have largely relied on the alignment of&#10;latent spaces of textual and non-textual features. This alignment process,&#10;which synchronizes a language model trained on textual data with encoders and&#10;decoders trained on multi-modal data, often necessitates extensive training of&#10;several projection layers in multiple stages. Inspired by LLM-as-agent&#10;methodologies, we propose a novel Input/Output (I/O) alignment mechanism that&#10;operates directly at the level of natural language. It aligns the LLM's output&#10;with the input of generative models, avoiding the complexities associated with&#10;latent feature alignments, and simplifying the multiple training stages of&#10;existing MLLMs into a single, efficient process. This conceptual advancement&#10;leads to significant reductions in both data and computational costs. By&#10;conducting experiments on several benchmarks, we demonstrate that our approach&#10;attains comparable performance with the state of the art while achieving&#10;considerable efficiencies in data usage and training duration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.14425" label="2406.14425">
        <attvalues>
          <attvalue for="0" value="SynDARin: Synthesising Datasets for Automated Reasoning in Low-Resource&#10;  Languages" />
          <attvalue for="1" value="  Question Answering (QA) datasets have been instrumental in developing and&#10;evaluating Large Language Model (LLM) capabilities. However, such datasets are&#10;scarce for languages other than English due to the cost and difficulties of&#10;collection and manual annotation. This means that producing novel models and&#10;measuring the performance of multilingual LLMs in low-resource languages is&#10;challenging. To mitigate this, we propose $\textbf{S}$yn$\textbf{DAR}$in, a&#10;method for generating and validating QA datasets for low-resource languages. We&#10;utilize parallel content mining to obtain $\textit{human-curated}$ paragraphs&#10;between English and the target language. We use the English data as context to&#10;$\textit{generate}$ synthetic multiple-choice (MC) question-answer pairs, which&#10;are automatically translated and further validated for quality. Combining these&#10;with their designated non-English $\textit{human-curated}$ paragraphs form the&#10;final QA dataset. The method allows to maintain the content quality, reduces&#10;the likelihood of factual errors, and circumvents the need for costly&#10;annotation. To test the method, we created a QA dataset with $1.2$K samples for&#10;the Armenian language. The human evaluation shows that $98\%$ of the generated&#10;English data maintains quality and diversity in the question types and topics,&#10;while the translation validation pipeline can filter out $\sim70\%$ of data&#10;with poor quality. We use the dataset to benchmark state-of-the-art LLMs,&#10;showing their inability to achieve human accuracy with some model performances&#10;closer to random chance. This shows that the generated dataset is non-trivial&#10;and can be used to evaluate reasoning capabilities in low-resource language.&#10;" />
          <attvalue for="2" value="&#10;&#10;Question Answering (QA) has been a hallmark task for testing reading comprehension and reasoning capabilities in NLP systems.&#10;The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}.&#10;Although some concentrated effort has been made to create multilingual QA resources \cite{lewis2019mlqa,asai2018multilingual,liu2019xqa}, the datasets remain rather scarce and usually cover a small selected set of languages due to the labour-intensive annotation costs. &#10;The proposed methods suggest using direct machine translation \cite{lewis2019mlqa,carrino2019automatic} or multilingual synthetic data generation \cite{riabi2020synthetic,agrawal2023qameleon,shakeri2020towards}. However, these approaches are directly bound to introduce biases and hallucinations during translation \cite{artetxe2020translation}, cross-lingual transfer \cite{lauscher2020zero, guerreiro2023hallucinations} or generation \cite{ahuja2023mega}. &#10;These limitations directly hinder the possibility to develop and evaluate the multilingual QA capabilities of language models in low-resource languages.&#10;&#10;In this work, we propose SynDARin, a novel method for synthesising datasets for automated reasoning in low-resource languages that circumvents the above-mentioned obstacles and test it by creating a QA dataset for the Armenian language, which has virtually no presence of structured NLP datasets \cite{avetisyan2023large}.&#10;We mine parallel English and Armenian introductory paragraphs from the same diverse set of Wikipedia articles, ensuring that the contents match by comparing their relative length. Similar mining approaches have been shown to be efficient for this task \cite{lewis2021paq,artetxe2019massively}.&#10;This allows us to obtain human-curated text from diverse topics while bypassing a wide chunk of direct content translation and annotation.&#10;Given the English subset of this data, we generate MC question-answer pairs by prompting an LLM to produce queries with an answer explicitly mentioned within the paragraph.&#10;Following \cite{lewis2019mlqa}, we filter out examples that do not contain the answer substring verbatim in the paragraph and additionally perform a human evaluation on a subset of $50$ examples and show that $98\%$ of these question-answer pairs are answerable and maintain quality.&#10;The produced question-answers are subsequently translated using an automated tool and further validated by answer substring and semantic matching in the parallel Armenian paragraph.&#10;This allows us to mitigate the likelihood of hallucinated, biased and inconsistent entries in the final QA dataset. Our human evaluation with native Armenian speakers shows that ~$70\%$ of such corrupted examples are removed.&#10;We use the dataset as a reasoning benchmark for Armenian and evaluate several LLMs in zero-shot, few-shot, and fine-tuned modes.&#10;We show that the dataset cannot be trivially solved, thus highlighting it as a useful resource for measuring model performance.&#10;In sum, our contributions are as follows: (i) a novel method for QA dataset construction in low-resource languages, (ii) a QA dataset in Armenian, (iii) ablations showing the quality of the generated samples and (iv) an evaluation of several LLM families on the QA dataset. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Multilingual Dataset Generation, Linguistics, Language Model Evaluation, Artificial Intelligence, Question Answering Systems, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1606.05250" label="1606.05250">
        <attvalues>
          <attvalue for="0" value="SQuAD: 100,000+ Questions for Machine Comprehension of Text" />
          <attvalue for="1" value="  We present the Stanford Question Answering Dataset (SQuAD), a new reading&#10;comprehension dataset consisting of 100,000+ questions posed by crowdworkers on&#10;a set of Wikipedia articles, where the answer to each question is a segment of&#10;text from the corresponding reading passage. We analyze the dataset to&#10;understand the types of reasoning required to answer the questions, leaning&#10;heavily on dependency and constituency trees. We build a strong logistic&#10;regression model, which achieves an F1 score of 51.0%, a significant&#10;improvement over a simple baseline (20%). However, human performance (86.8%) is&#10;much higher, indicating that the dataset presents a good challenge problem for&#10;future research.&#10;  The dataset is freely available at https://stanford-qa.com&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.00051" label="1704.00051">
        <attvalues>
          <attvalue for="0" value="Reading Wikipedia to Answer Open-Domain Questions" />
          <attvalue for="1" value="  This paper proposes to tackle open- domain question answering using Wikipedia&#10;as the unique knowledge source: the answer to any factoid question is a text&#10;span in a Wikipedia article. This task of machine reading at scale combines the&#10;challenges of document retrieval (finding the relevant articles) with that of&#10;machine comprehension of text (identifying the answer spans from those&#10;articles). Our approach combines a search component based on bigram hashing and&#10;TF-IDF matching with a multi-layer recurrent neural network model trained to&#10;detect answers in Wikipedia paragraphs. Our experiments on multiple existing QA&#10;datasets indicate that (1) both modules are highly competitive with respect to&#10;existing counterparts and (2) multitask learning using distant supervision on&#10;their combination is an effective complete system on this challenging task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.01243" label="2111.01243">
        <attvalues>
          <attvalue for="0" value="Recent Advances in Natural Language Processing via Large Pre-Trained&#10;  Language Models: A Survey" />
          <attvalue for="1" value="  Large, pre-trained transformer-based language models such as BERT have&#10;drastically changed the Natural Language Processing (NLP) field. We present a&#10;survey of recent work that uses these large language models to solve NLP tasks&#10;via pre-training then fine-tuning, prompting, or text generation approaches. We&#10;also present approaches that use pre-trained language models to generate data&#10;for training augmentation or other purposes. We conclude with discussions on&#10;limitations and suggested directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.12708" label="2107.12708">
        <attvalues>
          <attvalue for="0" value="QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering&#10;  and Reading Comprehension" />
          <attvalue for="1" value="  Alongside huge volumes of research on deep learning models in NLP in the&#10;recent years, there has been also much work on benchmark datasets needed to&#10;track modeling progress. Question answering and reading comprehension have been&#10;particularly prolific in this regard, with over 80 new datasets appearing in&#10;the past two years. This study is the largest survey of the field to date. We&#10;provide an overview of the various formats and domains of the current&#10;resources, highlighting the current lacunae for future work. We further discuss&#10;the current classifications of &quot;skills&quot; that question answering/reading&#10;comprehension systems are supposed to acquire, and propose a new taxonomy. The&#10;supplementary materials survey the current multilingual resources and&#10;monolingual resources for languages other than English, and we discuss the&#10;implications of over-focusing on English. The study is aimed at both&#10;practitioners looking for pointers to the wealth of existing data, and at&#10;researchers working on new resources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.07475" label="1910.07475">
        <attvalues>
          <attvalue for="0" value="MLQA: Evaluating Cross-lingual Extractive Question Answering" />
          <attvalue for="1" value="  Question answering (QA) models have shown rapid progress enabled by the&#10;availability of large, high-quality benchmark datasets. Such annotated datasets&#10;are difficult and costly to collect, and rarely exist in languages other than&#10;English, making training QA systems in other languages challenging. An&#10;alternative to building large monolingual training datasets is to develop&#10;cross-lingual systems which can transfer to a target language without requiring&#10;training data in that language. In order to develop such systems, it is crucial&#10;to invest in high quality multilingual evaluation benchmarks to measure&#10;progress. We present MLQA, a multi-way aligned extractive QA evaluation&#10;benchmark intended to spur research in this area. MLQA contains QA instances in&#10;7 languages, namely English, Arabic, German, Spanish, Hindi, Vietnamese and&#10;Simplified Chinese. It consists of over 12K QA instances in English and 5K in&#10;each other language, with each QA instance being parallel between 4 languages&#10;on average. MLQA is built using a novel alignment context strategy on Wikipedia&#10;articles, and serves as a cross-lingual extension to existing extractive QA&#10;datasets. We evaluate current state-of-the-art cross-lingual representations on&#10;MLQA, and also provide machine-translation-based baselines. In all cases,&#10;transfer results are shown to be significantly behind training-language&#10;performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Monolingual QA Data There is a great variety of English QA data, popularized by MCTest \cite{richardson_mctest:_2013}, CNN/Daily Mail~\cite{hermann_teaching_2015} CBT~\cite{hill_goldilocks_2016}, and WikiQA~\cite{yang_wikiqa:_2015} amongst others. Large span-based datasets such as SQuAD~\cite{rajpurkar_squad:_2016,rajpurkar_know_2018}, TriviaQA~\cite{joshi_triviaqa:_2017}, NewsQA~\cite{trischler_newsqa:_2017}, and Natural Questions~\cite{kwiatkowski_natural_2019} have seen extractive QA become a dominant paradigm. &#10;However, large, high-quality datasets in other languages are relatively rare. There are several Chinese datasets, such as DUReader~\cite{he_dureader:_2018}, CMRC~\cite{cui_span-extraction_2019} and DRCD~\cite{shao_drcd:_2018}. More recently, there have been efforts to build corpora in a wider array of languages, such as Korean~\cite{korquad} and Arabic~\cite{mozannar_neural_2019}.&#10;&#10;Cross-lingual QA Modelling Cross-lingual QA as a discipline has been explored in QA for RDF data for a number of years, such as the QALD-3 and 5 tracks~\cite{cimiano_multilingual_2013,unger_question_2015}, with more recent work from \cite{zimina_mug-qa:_2018}. \cite{lee_semi-supervised_2018} explore an approach to use English QA data from SQuAD to improve QA performance in Korean using an in-language seed dataset. \cite{kumar_cross-lingual_2019} study question generation by leveraging English questions to generate better Hindi questions, and \cite{lee_cross-lingual_2019} and \cite{cui_cross-lingual_2019} develop modelling approaches to improve performance on Chinese QA tasks using English resources. \cite{lee_learning_2019} and \cite{hsu_zero-shot_2019} explore modelling approaches for zero-shot transfer and \cite{singh_xlda:_2019} explore how training with cross-lingual data regularizes QA models.&#10;&#10;Cross-lingual QA Data \cite{gupta_mmqa:_2018} release a parallel QA dataset in English and Hindi, \cite{hardalov_beyond_2019} investigate QA transfer from English to Bulgarian, \cite{liu_xcmrc:_2019} release a cloze QA dataset in Chinese and English, and \cite{jing_bipar:_2019} released BiPar, built using parallel paragraphs from novels in English and Chinese. These datasets have a similar spirit to MLQA, but are limited to two languages. \cite{asai_multilingual_2018} investigate extractive QA on a manually-translated set of 327 SQuAD instances in Japanese and French, and develop a phrase-alignment modelling technique, showing improvements over back-translation. Like us, they build multi-way parallel extractive QA data, but MLQA has many more instances, covers more languages and does not require manual document translation.&#10;\cite{liu_xqa:_2019} explore cross-lingual open-domain QA with a dataset built from Wikipedia ``Did you know?&quot; questions, covering nine languages. Unlike MLQA, it is distantly supervised, the dataset size varies by language, instances are not parallel, and answer distributions vary by language, making quantitative comparisons across languages challenging. Finally, in contemporaneous work, \cite{artetxe_cross-lingual_2019} release XQuAD, a dataset of 1190 SQuAD instances from 240 paragraphs manually translated into 10 languages. As shown in Table \ref{paragraph_and_article_counts}, MLQA covers 7 languages, but contains more data per language -- over 5k QA pairs from \~5k paragraphs per language. MLQA also uses real Wikipedia contexts rather than manual translation. &#10;&#10;Aggregated Cross-lingual Benchmarks Recently, following the widespread adoption of projects such as GLUE~\cite{wang2018glue}, there have been efforts to compile a suite of high quality multilingual tasks as a unified benchmark system. Two such projects, XGLUE \cite{Liang2020XGLUEAN} and XTREME~\cite{Hu2020XTREMEAM} incorporate MLQA as part of their aggregated benchmark.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1809.03275" label="1809.03275">
        <attvalues>
          <attvalue for="0" value="Multilingual Extractive Reading Comprehension by Runtime Machine&#10;  Translation" />
          <attvalue for="1" value="  Despite recent work in Reading Comprehension (RC), progress has been mostly&#10;limited to English due to the lack of large-scale datasets in other languages.&#10;In this work, we introduce the first RC system for languages without RC&#10;training data. Given a target language without RC training data and a pivot&#10;language with RC training data (e.g. English), our method leverages existing RC&#10;resources in the pivot language by combining a competitive RC model in the&#10;pivot language with an attentive Neural Machine Translation (NMT) model. We&#10;first translate the data from the target to the pivot language, and then obtain&#10;an answer using the RC model in the pivot language. Finally, we recover the&#10;corresponding answer in the original language using soft-alignment attention&#10;scores from the NMT model. We create evaluation sets of RC data in two&#10;non-English languages, namely Japanese and French, to evaluate our method.&#10;Experimental results on these datasets show that our method significantly&#10;outperforms a back-translation baseline of a state-of-the-art product-level&#10;machine translation system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.05200" label="1912.05200">
        <attvalues>
          <attvalue for="0" value="Automatic Spanish Translation of the SQuAD Dataset for Multilingual&#10;  Question Answering" />
          <attvalue for="1" value="  Recently, multilingual question answering became a crucial research topic,&#10;and it is receiving increased interest in the NLP community. However, the&#10;unavailability of large-scale datasets makes it challenging to train&#10;multilingual QA systems with performance comparable to the English ones. In&#10;this work, we develop the Translate Align Retrieve (TAR) method to&#10;automatically translate the Stanford Question Answering Dataset (SQuAD) v1.1 to&#10;Spanish. We then used this dataset to train Spanish QA systems by fine-tuning a&#10;Multilingual-BERT model. Finally, we evaluated our QA models with the recently&#10;proposed MLQA and XQuAD benchmarks for cross-lingual Extractive QA.&#10;Experimental results show that our models outperform the previous&#10;Multilingual-BERT baselines achieving the new state-of-the-art value of 68.1 F1&#10;points on the Spanish MLQA corpus and 77.6 F1 and 61.8 Exact Match points on&#10;the Spanish XQuAD corpus. The resulting, synthetically generated SQuAD-es v1.1&#10;corpora, with almost 100% of data contained in the original English version, to&#10;the best of our knowledge, is the first large-scale QA training resource for&#10;Spanish.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12643" label="2010.12643">
        <attvalues>
          <attvalue for="0" value="Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question&#10;  Answering" />
          <attvalue for="1" value="  Coupled with the availability of large scale datasets, deep learning&#10;architectures have enabled rapid progress on the Question Answering task.&#10;However, most of those datasets are in English, and the performances of&#10;state-of-the-art multilingual models are significantly lower when evaluated on&#10;non-English data. Due to high data collection costs, it is not realistic to&#10;obtain annotated data for each language one desires to support.&#10;  We propose a method to improve the Cross-lingual Question Answering&#10;performance without requiring additional annotated data, leveraging Question&#10;Generation models to produce synthetic samples in a cross-lingual fashion. We&#10;show that the proposed method allows to significantly outperform the baselines&#10;trained on English data only. We report a new state-of-the-art on four&#10;multilingual datasets: MLQA, XQuAD, SQuAD-it and PIAF (fr).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.08264" label="2211.08264">
        <attvalues>
          <attvalue for="0" value="QAmeleon: Multilingual QA with Only 5 Examples" />
          <attvalue for="1" value="  The availability of large, high-quality datasets has been one of the main&#10;drivers of recent progress in question answering (QA). Such annotated datasets&#10;however are difficult and costly to collect, and rarely exist in languages&#10;other than English, rendering QA technology inaccessible to underrepresented&#10;languages. An alternative to building large monolingual training datasets is to&#10;leverage pre-trained language models (PLMs) under a few-shot learning setting.&#10;Our approach, QAmeleon, uses a PLM to automatically generate multilingual data&#10;upon which QA models are trained, thus avoiding costly annotation. Prompt&#10;tuning the PLM for data synthesis with only five examples per language delivers&#10;accuracy superior to translation-based baselines, bridges nearly 60% of the gap&#10;between an English-only baseline and a fully supervised upper bound trained on&#10;almost 50,000 hand labeled examples, and always leads to substantial&#10;improvements compared to fine-tuning a QA model directly on labeled examples in&#10;low resource settings. Experiments on the TyDiQA-GoldP and MLQA benchmarks show&#10;that few-shot prompt tuning for data synthesis scales across languages and is a&#10;viable alternative to large-scale annotation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Data Generation for QA Prior work on the generation of QA data&#10;has mostly focused on English and typically divides the task into&#10;answer extraction/generation and question generation, followed by some&#10;type of filtering. \cite{alberti-etal-2019-synthetic} employ&#10;round-trip consistency for filtering with BERT-based models. Other&#10;work \cite{shakeri-etal-2020-end} uses BART to jointly generate a&#10;question and its answer given an input passage, employing&#10;likelihood-based filtering. \cite{lewis-etal-2021-paq} use a&#10;RoBERTa-based passage selection model to identify interesting&#10;passages. \cite{bartolo-etal-2021-improving} additionally train the&#10;generation models on an adversarial QA dataset, while&#10;\cite{yao-etal-2022-ais} integrate a QA-pair ranking module.&#10;&#10;The above approaches generally require large amounts of labeled QA&#10;data in the form of SQuAD \cite{rajpurkar-etal-2016-squad} or&#10;Natural Questions \cite{kwiatkowski-etal-2019-natural} to train&#10;passage selection and question generation models. In contrast, we only&#10;assume access to a few question-answer pairs per language.&#10;&#10;Multilingual QA In this work we used mT5-XL \cite{xue2021mt5} as our&#10; reference QA model.&#10; We note that a slightly more performant choice could have been ByT5 \cite{xue2022byt5},&#10; which reports improvements on \tydiqagoldp by operating directly on raw text instead&#10; of sentence pieces.&#10; Existing work on low resource multilingual QA &#10; has been relatively limited. \cite{lee-etal-2018-semi} propose to&#10; use automatically translated high-confidence QA examples for&#10; training, while other approaches&#10; \cite{kumar-etal-2019-cross,chi2020cross} only generate questions and&#10; require supervised training data in the target language. Other approaches&#10; \cite{riabi-etal-2021-synthetic,shakeri-etal-2021-towards,kramchaninova-defauw-2022-synthetic}&#10; focus on zero-shot transfer, i.e.,~a multilingual model trained on&#10; QA data generation on SQuAD (and optionally automatically&#10; translated SQuAD data) is applied to other languages. Our&#10; work shows that few-shot settings result in better multilingual&#10; generation quality in comparison to zero-shot models.&#10;&#10;Prompting Existing work \cite[inter&#10; alia]{brown2020language,schick-schutze-2021-just} has shown that&#10;prompting pre-trained large language models can lead to strong&#10;performance in a wide range of tasks including natural language&#10;generation and common sense reasoning. In the context of multilingual&#10;QA, \cite{chowdhery2022palm} employ a single prompt and a few labeled&#10;examples in the target language. In contrast, we employ chain-of-thought prompting, and English answers and questions as a bridge. Moreover, our experiments&#10;with QAmeleon demonstrate that prompt tuning is superior and&#10;a viable alternative to large-scale annotation. Prompting in multilingual settings has achieved the best performance using English prompts and target language exemplars \cite{winata-etal-2021-language,Lin2022,Shi:ea:2022}. We demonstrate that parameter-efficient methods such as prompt tuning using target language exemplars \cite{lester-etal-2021-power} is a superior choice.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2004.04721" label="2004.04721">
        <attvalues>
          <attvalue for="0" value="Translation Artifacts in Cross-lingual Transfer Learning" />
          <attvalue for="1" value="  Both human and machine translation play a central role in cross-lingual&#10;transfer learning: many multilingual datasets have been created through&#10;professional translation services, and using machine translation to translate&#10;either the test set or the training set is a widely used transfer technique. In&#10;this paper, we show that such translation process can introduce subtle&#10;artifacts that have a notable impact in existing cross-lingual models. For&#10;instance, in natural language inference, translating the premise and the&#10;hypothesis independently can reduce the lexical overlap between them, which&#10;current models are highly sensitive to. We show that some previous findings in&#10;cross-lingual transfer learning need to be reconsidered in the light of this&#10;phenomenon. Based on the gained insights, we also improve the state-of-the-art&#10;in XNLI for the translate-test and zero-shot approaches by 4.3 and 2.8 points,&#10;respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00633" label="2005.00633">
        <attvalues>
          <attvalue for="0" value="From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual&#10;  Transfer with Multilingual Transformers" />
          <attvalue for="1" value="  Massively multilingual transformers pretrained with language modeling&#10;objectives (e.g., mBERT, XLM-R) have become a de facto default transfer&#10;paradigm for zero-shot cross-lingual transfer in NLP, offering unmatched&#10;transfer performance. Current downstream evaluations, however, verify their&#10;efficacy predominantly in transfer settings involving languages with sufficient&#10;amounts of pretraining data, and with lexically and typologically close&#10;languages. In this work, we analyze their limitations and show that&#10;cross-lingual transfer via massively multilingual transformers, much like&#10;transfer via cross-lingual word embeddings, is substantially less effective in&#10;resource-lean scenarios and for distant languages. Our experiments,&#10;encompassing three lower-level tasks (POS tagging, dependency parsing, NER), as&#10;well as two high-level semantic tasks (NLI, QA), empirically correlate transfer&#10;performance with linguistic similarity between the source and target languages,&#10;but also with the size of pretraining corpora of target languages. We also&#10;demonstrate a surprising effectiveness of inexpensive few-shot transfer (i.e.,&#10;fine-tuning on a few target-language instances after fine-tuning in the source)&#10;across the board. This suggests that additional research efforts should be&#10;invested to reach beyond the limiting zero-shot conditions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.12528" label="2303.12528">
        <attvalues>
          <attvalue for="0" value="MEGA: Multilingual Evaluation of Generative AI" />
          <attvalue for="1" value="  Generative AI models have shown impressive performance on many Natural&#10;Language Processing tasks such as language understanding, reasoning, and&#10;language generation. An important question being asked by the AI community&#10;today is about the capabilities and limits of these models, and it is clear&#10;that evaluating generative AI is very challenging. Most studies on generative&#10;LLMs have been restricted to English and it is unclear how capable these models&#10;are at understanding and generating text in other languages. We present the&#10;first comprehensive benchmarking of generative LLMs - MEGA, which evaluates&#10;models on standard NLP benchmarks, covering 16 NLP datasets across 70&#10;typologically diverse languages. We compare the performance of generative LLMs&#10;including Chat-GPT and GPT-4 to State of the Art (SOTA) non-autoregressive&#10;models on these tasks to determine how well generative models perform compared&#10;to the previous generation of LLMs. We present a thorough analysis of the&#10;performance of models across languages and tasks and discuss challenges in&#10;improving the performance of generative LLMs on low-resource languages. We&#10;create a framework for evaluating generative LLMs in the multilingual setting&#10;and provide directions for future progress in the field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.07033" label="2102.07033">
        <attvalues>
          <attvalue for="0" value="PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them" />
          <attvalue for="1" value="  Open-domain Question Answering models which directly leverage question-answer&#10;(QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show&#10;promise in terms of speed and memory compared to conventional models which&#10;retrieve and read from text corpora. QA-pair retrievers also offer&#10;interpretable answers, a high degree of control, and are trivial to update at&#10;test time with new knowledge. However, these models lack the accuracy of&#10;retrieve-and-read systems, as substantially less knowledge is covered by the&#10;available QA-pairs relative to text corpora like Wikipedia. To facilitate&#10;improved QA-pair models, we introduce Probably Asked Questions (PAQ), a very&#10;large resource of 65M automatically-generated QA-pairs. We introduce a new&#10;QA-pair retriever, RePAQ, to complement PAQ. We find that PAQ preempts and&#10;caches test questions, enabling RePAQ to match the accuracy of recent&#10;retrieve-and-read models, whilst being significantly faster. Using PAQ, we&#10;train CBQA models which outperform comparable baselines by 5%, but trail RePAQ&#10;by over 15%, indicating the effectiveness of explicit retrieval. RePAQ can be&#10;configured for size (under 500MB) or speed (over 1K questions per second)&#10;whilst retaining high accuracy. Lastly, we demonstrate RePAQ's strength at&#10;selective QA, abstaining from answering when it is likely to be incorrect. This&#10;enables RePAQ to ``back-off&quot; to a more expensive state-of-the-art model,&#10;leading to a combined system which is both more accurate and 2x faster than the&#10;state-of-the-art model alone.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.02019" label="2308.02019">
        <attvalues>
          <attvalue for="0" value="Baby Llama: knowledge distillation from an ensemble of teachers trained&#10;  on a small dataset with no performance penalty" />
          <attvalue for="1" value="  We present our submission to the BabyLM challenge, whose goal was to improve&#10;the sample efficiency of language models. We trained an ensemble consisting of&#10;a GPT-2 and small LLaMA models on the developmentally-plausible, 10M-word&#10;BabyLM dataset, then distilled it into a small, 58M-parameter LLaMA model,&#10;which exceeds in performance both of its teachers as well as a similar model&#10;trained without distillation. This suggests that distillation can not only&#10;retain the full performance of the teacher model when the latter is trained on&#10;a sufficiently small dataset; it can exceed it, and lead to significantly&#10;better performance than direct training.&#10;" />
          <attvalue for="2" value="&#10;&#10;Today's state-of-the-art language models are typically trained on the order of a trillion tokens.&#10;\cite{hoffmann2022training} have observed that in order to train a model in a compute-optimal way, the number of parameters and dataset size should follow a linear relation: the so-called Chinchilla scaling law, with an optimal ratio of about 20~tokens per model parameter.&#10;For models larger than $\sim 10^{11}$ parameters, this implies that the currently-available amount of training data ($\sim 10^{12}$ tokens) already constitutes a bottleneck, that prevents scaling up those models in a compute-optimal way.&#10;&#10;A trillion tokens is already at least 4 orders of magnitude larger than the estimated number of words ($\lesssim 10^8$) to which a typical 13-year-old child has been exposed. This suggests that current language models are significantly less sample-efficient than human beings.&#10;&#10;Furthermore, the trend of scaling up models to improve their performance may limit their usage in embedded systems, personal devices, and other end-user technologies, as well as in specialized applications where domain-specific training material is scarce.&#10;\cite{taylor2022galactica} have shown that training models on higher-quality data can improve performance; however, the quantity of such high-quality data is limited, and often represents only a small fraction of the corpus.&#10;&#10;This makes a strong case for trying to increase the sample efficiency of current models and training algorithms.&#10;In this context, the BabyLM challenge \cite{warstadt2023papers} has invited researchers to investigate ways of improving the sample efficiency of small-scale language models, by restricting the training set to a developmentally plausible corpus, consisting mostly of transcribed speech of either 10M (strict-small track) or 100M words (strict and loose tracks).&#10;&#10;The present paper describes our submission to the strict-small track of the BabyLM challenge. As such, it focuses on the 10M-word dataset.&#10;Our proposed solution consists in distilling an ensemble of two larger ``teacher'' models, of different architectures (GPT-2 and LLaMA), into a smaller ``student'' LLaMA model. We show that this approach produces a model whose performance largely matches, and often exceeds, that of both teachers.&#10;&#10;We introduce Baby Llama in \ref{sec:model}, describe the dataset in \ref{sec:dataset}, discuss the model performance in \ref{sec:performance}, and finally conclude in \ref{sec:conclusion}.&#10;The full numerical results of the evals are listed in \ref{app:evals}, and in \ref{app:attempts} we briefly discuss a number of experiments (including some negative results) that we eventually chose not to include into the final model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Knowledge Distillation Techniques, Artificial Intelligence, Natural Language Processing, Language Model Efficiency, Ensemble Learning Methods" />
        </attvalues>
      </node>
      <node id="2309.11439" label="2309.11439">
        <attvalues>
          <attvalue for="0" value="Controlled Generation with Prompt Insertion for Natural Language&#10;  Explanations in Grammatical Error Correction" />
          <attvalue for="1" value="  In Grammatical Error Correction (GEC), it is crucial to ensure the user's&#10;comprehension of a reason for correction. Existing studies present tokens,&#10;examples, and hints as to the basis for correction but do not directly explain&#10;the reasons for corrections. Although methods that use Large Language Models&#10;(LLMs) to provide direct explanations in natural language have been proposed&#10;for various tasks, no such method exists for GEC. Generating explanations for&#10;GEC corrections involves aligning input and output tokens, identifying&#10;correction points, and presenting corresponding explanations consistently.&#10;However, it is not straightforward to specify a complex format to generate&#10;explanations, because explicit control of generation is difficult with prompts.&#10;This study introduces a method called controlled generation with Prompt&#10;Insertion (PI) so that LLMs can explain the reasons for corrections in natural&#10;language. In PI, LLMs first correct the input text, and then we automatically&#10;extract the correction points based on the rules. The extracted correction&#10;points are sequentially inserted into the LLM's explanation output as prompts,&#10;guiding the LLMs to generate explanations for the correction points. We also&#10;create an Explainable GEC (XGEC) dataset of correction reasons by annotating&#10;NUCLE, CoNLL2013, and CoNLL2014. Although generations from GPT-3 and ChatGPT&#10;using original prompts miss some correction points, the generation control&#10;using PI can explicitly guide to describe explanations for all correction&#10;points, contributing to improved performance in generating correction reasons.&#10;" />
          <attvalue for="2" value="&#10;&#10;Grammatical Error Correction (GEC) is the task of correcting grammatical errors in a text.&#10;In GEC, various methods have been proposed from a wide range of perspectives, including correction performance~\cite{grundkiewicz-junczys-dowmunt-2019-minimally,chollampatt-etal-2019-cross,omelianchuk-etal-2020-gector,kaneko-etal-2020-encoder,qorib-etal-2022-frustratingly}, controlling~\cite{hotate-etal-2019-controlling,yang2022controllable,loem2023exploring}, diversity~\cite{xie-etal-2018-noising,hotate-etal-2020-generating,han2021diversity}, and efficiency~\cite{malmi-etal-2019-encode,chen-etal-2020-improving-efficiency}.&#10;It is also important in GEC for the model to provide explanations that allow users to understand the reasons behind the corrections.&#10;Improving explainability leads to a better judgment of whether the correction reflects the intended result, learning of grammatical knowledge, and overall enhancement of GEC systems.&#10;&#10;\cite{kaneko-etal-2022-interpretability} introduced a method of presenting the retrieved examples as the basis for correction, in contrast to a method of retrieving data similar to the correction target from the training data set and using it for prediction.&#10;\cite{fei2023enhancing} proposed a method that presents the token positions that are the basis of errors and error types, and showed that they are useful for learners.&#10;\cite{nagata-2019-toward} proposed the task of generating useful hints and feedback for language learning on essays written by language learners.&#10;This task does not necessarily generate a correction result or reason, because it is not intended for correction.&#10;Since these existing studies do not directly explain the reason for the correction, the user must infer the reason for the correction from the system output.&#10;&#10;Large Language Models (LLMs) such as ChatGPT~\cite{chatgpt} and GPT-3~\cite{brown2020language} have advanced language capabilities and can explain the inference reasons in natural language in various tasks~\cite{wei2022chain,wiegreffe-etal-2022-reframing,kaneko2023solving}.&#10;With natural language, the model can directly explain the details of the inference reasons to the user.&#10;LLMs are also effective in GEC, achieving state-of-the-art in both unsupervised~\cite{loem2023exploring} and supervised settings~\cite{kaneko2023reducing}.&#10;Explicability in GEC first requires the alignment of input and output tokens and identifies all error and correction pairs.&#10;Then, it is necessary to generate an explanation for each of the extracted pairs.&#10;However, it is hard to control the generation of prompts in a specified format for GEC.&#10;\cite{fang2023chatgpt} showed that ChatGPT improves performance by using natural language to generate step-by-step error detection and correction processes for each span.&#10;It was found that it is difficult for ChatGPT to generate step-by-step according to the specified format with simple prompt instructions.&#10;\cite{loem2023exploring} showed that prompting did not contribute significantly to the control of correction style for GPT-3.&#10;&#10;In this study, we introduce a method to explain the reason for correction in natural language by a controlled generation with prompt insertion (PI).&#10;As shown in \ref{fig:abst}, we guide LLMs to the desired format output by inserting prompts during inference.&#10;First, LLM corrects grammatical errors in the input text.&#10;Then, we automatically align the error and correction points from the input and output text using rules and extract error-correction pairs.&#10;By inserting these error-correction pairs as additional prompts, we explicitly control the LLM's explanation of the reasons for all pairs.&#10;Furthermore, we created an Explainable GEC (XGEC) dataset for explaining correction reasons in natural language by annotating NUCLE, CoNLL2013, and CoNLL2014 datasets~\cite{dahlmeier-etal-2013-building,ng-etal-2013-conll,ng-etal-2014-conll}.&#10;&#10;In our experiments on GPT-3 and ChatGPT, we found that the original prompt-based generation resulted in pair omissions and ambiguity as to which pair the explanation was for.&#10;On the other hand, the control of generation by PI can explicitly control the LLM to generate explanations for all the corrections, which contributes to the performance improvement of the explanation of correction reasons.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Explainable AI Systems, Linguistics, Artificial Intelligence, Grammatical Error Correction, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2005.12592" label="2005.12592">
        <attvalues>
          <attvalue for="0" value="GECToR -- Grammatical Error Correction: Tag, Not Rewrite" />
          <attvalue for="1" value="  In this paper, we present a simple and efficient GEC sequence tagger using a&#10;Transformer encoder. Our system is pre-trained on synthetic data and then&#10;fine-tuned in two stages: first on errorful corpora, and second on a&#10;combination of errorful and error-free parallel corpora. We design custom&#10;token-level transformations to map input tokens to target corrections. Our best&#10;single-model/ensemble GEC tagger achieves an $F_{0.5}$ of 65.3/66.5 on&#10;CoNLL-2014 (test) and $F_{0.5}$ of 72.4/73.6 on BEA-2019 (test). Its inference&#10;speed is up to 10 times as fast as a Transformer-based seq2seq GEC system. The&#10;code and trained models are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00987" label="2005.00987">
        <attvalues>
          <attvalue for="0" value="Encoder-Decoder Models Can Benefit from Pre-trained Masked Language&#10;  Models in Grammatical Error Correction" />
          <attvalue for="1" value="  This paper investigates how to effectively incorporate a pre-trained masked&#10;language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for&#10;grammatical error correction (GEC). The answer to this question is not as&#10;straightforward as one might expect because the previous common methods for&#10;incorporating a MLM into an EncDec model have potential drawbacks when applied&#10;to GEC. For example, the distribution of the inputs to a GEC model can be&#10;considerably different (erroneous, clumsy, etc.) from that of the corpora used&#10;for pre-training MLMs; however, this issue is not addressed in the previous&#10;methods. Our experiments show that our proposed method, where we first&#10;fine-tune a MLM with a given GEC corpus and then use the output of the&#10;fine-tuned MLM as additional features in the GEC model, maximizes the benefit&#10;of the MLM. The best-performing model achieves state-of-the-art performances on&#10;the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at:&#10;https://github.com/kanekomasahiro/bert-gec.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Studies have reported that a MLM can improve the performance of GEC when it is employed either as a re-ranker \cite{chollampatt2019csgec, kaneko-etal-2019-tmu} or as a filtering tool \cite{asano-etal-2019-aip, kiyono-etal-2019-empirical}.&#10;EncDec-based GEC models combined with MLMs can also be used in combination with these pipeline methods.&#10;\cite{kantor-etal-2019-learning} and \cite{awasthi-etal-2019-parallel} proposed sequence labeling models based on correction methods.&#10;Our method can utilize the existing EncDec GEC knowledge, but these methods cannot be utilized due to the different architecture of the model.&#10;Besides, to the best of our knowledge, no research has yet been conducted that incorporates information of MLMs for effectively training the EncDec GEC model.&#10;&#10;MLMs are generally used in downstream tasks by fine-tuning \cite{liu2019fine, zhang2019pretraining}, however, \cite{Zhu2020IncorporatingBI} demonstrated that it is more effective to provide the output of the final layer of a MLM to the EncDec model as contextual embeddings.&#10;Recently, \cite{weng2019acquiring} addressed the mismatch problem between contextual knowledge from pre-trained models and the target bilingual machine translation.&#10;Here, we also claim that addressing the gap between grammatically correct raw corpora and GEC corpora can lead to the improvement of GEC systems. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.18156" label="2305.18156">
        <attvalues>
          <attvalue for="0" value="Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A&#10;  Study on Performance and Controllability in Prompt-Based Methods" />
          <attvalue for="1" value="  Large-scale pre-trained language models such as GPT-3 have shown remarkable&#10;performance across various natural language processing tasks. However, applying&#10;prompt-based methods with GPT-3 for Grammatical Error Correction (GEC) tasks&#10;and their controllability remains underexplored. Controllability in GEC is&#10;crucial for real-world applications, particularly in educational settings,&#10;where the ability to tailor feedback according to learner levels and specific&#10;error types can significantly enhance the learning process. This paper&#10;investigates the performance and controllability of prompt-based methods with&#10;GPT-3 for GEC tasks using zero-shot and few-shot setting. We explore the impact&#10;of task instructions and examples on GPT-3's output, focusing on controlling&#10;aspects such as minimal edits, fluency edits, and learner levels. Our findings&#10;demonstrate that GPT-3 could effectively perform GEC tasks, outperforming&#10;existing supervised and unsupervised approaches. We also showed that GPT-3&#10;could achieve controllability when appropriate task instructions and examples&#10;are given.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.15149" label="2110.15149">
        <attvalues>
          <attvalue for="0" value="Diversity-Driven Combination for Grammatical Error Correction" />
          <attvalue for="1" value="  Grammatical error correction (GEC) is the task of detecting and correcting&#10;errors in a written text. The idea of combining multiple system outputs has&#10;been successfully used in GEC. To achieve successful system combination,&#10;multiple component systems need to produce corrected sentences that are both&#10;diverse and of comparable quality. However, most existing state-of-the-art GEC&#10;approaches are based on similar sequence-to-sequence neural networks, so the&#10;gains are limited from combining the outputs of component systems similar to&#10;one another. In this paper, we present Diversity-Driven Combination (DDC) for&#10;GEC, a system combination strategy that encourages diversity among component&#10;systems. We evaluate our system combination strategy on the CoNLL-2014 shared&#10;task and the BEA-2019 shared task. On both benchmarks, DDC achieves significant&#10;performance gain with a small number of training examples and outperforms the&#10;component systems by a large margin. Our source code is available at&#10;https://github.com/nusnlp/gec-ddc.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.01187" label="1909.01187">
        <attvalues>
          <attvalue for="0" value="Encode, Tag, Realize: High-Precision Text Editing" />
          <attvalue for="1" value="  We propose LaserTagger - a sequence tagging approach that casts text&#10;generation as a text editing task. Target texts are reconstructed from the&#10;inputs using three main edit operations: keeping a token, deleting it, and&#10;adding a phrase before the token. To predict the edit operations, we propose a&#10;novel model, which combines a BERT encoder with an autoregressive Transformer&#10;decoder. This approach is evaluated on English text on four tasks: sentence&#10;fusion, sentence splitting, abstractive summarization, and grammar correction.&#10;LaserTagger achieves new state-of-the-art results on three of these tasks,&#10;performs comparably to a set of strong seq2seq baselines with a large number of&#10;training examples, and outperforms them when the number of examples is limited.&#10;Furthermore, we show that at inference time tagging can be more than two orders&#10;of magnitude faster than comparable seq2seq models, making it more attractive&#10;for running in a live environment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03260" label="2010.03260">
        <attvalues>
          <attvalue for="0" value="Improving the Efficiency of Grammatical Error Correction with Erroneous&#10;  Span Detection and Correction" />
          <attvalue for="1" value="  We propose a novel language-independent approach to improve the efficiency&#10;for Grammatical Error Correction (GEC) by dividing the task into two subtasks:&#10;Erroneous Span Detection (ESD) and Erroneous Span Correction (ESC). ESD&#10;identifies grammatically incorrect text spans with an efficient sequence&#10;tagging model. Then, ESC leverages a seq2seq model to take the sentence with&#10;annotated erroneous spans as input and only outputs the corrected text for&#10;these spans. Experiments show our approach performs comparably to conventional&#10;seq2seq approaches in both English and Chinese GEC benchmarks with less than&#10;50% time cost for inference.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.07085" label="2203.07085">
        <attvalues>
          <attvalue for="0" value="Interpretability for Language Learners Using Example-Based Grammatical&#10;  Error Correction" />
          <attvalue for="1" value="  Grammatical Error Correction (GEC) should not focus only on high accuracy of&#10;corrections but also on interpretability for language learning. However,&#10;existing neural-based GEC models mainly aim at improving accuracy, and their&#10;interpretability has not been explored. A promising approach for improving&#10;interpretability is an example-based method, which uses similar retrieved&#10;examples to generate corrections. In addition, examples are beneficial in&#10;language learning, helping learners understand the basis of grammatically&#10;incorrect/correct texts and improve their confidence in writing. Therefore, we&#10;hypothesize that incorporating an example-based method into GEC can improve&#10;interpretability as well as support language learners. In this study, we&#10;introduce an Example-Based GEC (EB-GEC) that presents examples to language&#10;learners as a basis for a correction result. The examples consist of pairs of&#10;correct and incorrect sentences similar to a given input and its predicted&#10;correction. Experiments demonstrate that the examples presented by EB-GEC help&#10;language learners decide to accept or refuse suggestions from the GEC output.&#10;Furthermore, the experiments also show that retrieved examples improve the&#10;accuracy of corrections.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15676" label="2305.15676">
        <attvalues>
          <attvalue for="0" value="Enhancing Grammatical Error Correction Systems with Explanations" />
          <attvalue for="1" value="  Grammatical error correction systems improve written communication by&#10;detecting and correcting language mistakes. To help language learners better&#10;understand why the GEC system makes a certain correction, the causes of errors&#10;(evidence words) and the corresponding error types are two key factors. To&#10;enhance GEC systems with explanations, we introduce EXPECT, a large dataset&#10;annotated with evidence words and grammatical error types. We propose several&#10;baselines and analysis to understand this task. Furthermore, human evaluation&#10;verifies our explainable GEC system's explanations can assist second-language&#10;learners in determining whether to accept a correction suggestion and in&#10;understanding the associated grammar rule.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08674" label="2112.08674">
        <attvalues>
          <attvalue for="0" value="Reframing Human-AI Collaboration for Generating Free-Text Explanations" />
          <attvalue for="1" value="  Large language models are increasingly capable of generating fluent-appearing&#10;text with relatively little task-specific supervision. But can these models&#10;accurately explain classification decisions? We consider the task of generating&#10;free-text explanations using human-written examples in a few-shot manner. We&#10;find that (1) authoring higher quality prompts results in higher quality&#10;generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers&#10;often prefer explanations generated by GPT-3 to crowdsourced explanations in&#10;existing datasets. Our human studies also show, however, that while models&#10;often produce factual, grammatical, and sufficient explanations, they have room&#10;to improve along axes such as providing novel information and supporting the&#10;label. We create a pipeline that combines GPT-3 with a supervised filter that&#10;incorporates binary acceptability judgments from humans in the loop. Despite&#10;the intrinsic subjectivity of acceptability judgments, we demonstrate that&#10;acceptability is partially correlated with various fine-grained attributes of&#10;explanations. Our approach is able to consistently filter GPT-3-generated&#10;explanations deemed acceptable by humans.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Free-text explanation generation.&#10;&#10;The earliest neural free-text explanation models did so for computer vision applications \cite{hendricks2016generating, park2018multimodal, kim2018textual} and NLI \cite{camburu2018snli}. &#10;These methods relied on supervised datasets to train the explanation generator. &#10;Others have proposed to generate explanations or clarifications to improve task performance in a supervised \cite{rajani-etal-2019-explain, lampinen2022can} or unsupervised \cite{shwartz-etal-2020-unsupervised} manner. \cite{yordanov2021few} study transfer learning between datasets for few-shot generation.&#10;&#10;\cite{latcinnik2020explaining} proposed a method to generate free-text explanations supervised only on task signal, and \cite{Brahman2020LearningTR} used sources of weak supervision to generate explanations for defeasible inference. &#10;\cite{paranjape-etal-2021-prompting} design hand-crafted templates which they use with mask-infilling to produce contrastive explanations from pretrained language models. &#10;&#10;Concurrent work \cite{marasovic2021few} studies the effect of prompt format and model size on crowdworker judgements of prompted explanation plausibility. They find that GPT-3 Davinci outperforms other smaller pretrained models, but that crowdworkers find these explanations less plausible than those from the datasets, aligning with our first experimental result (\ref{table:gpt3-vs-crowdsourced-explanations}).&#10;We perform a more in-depth study of the fine-grained criteria comprising human acceptability, and demonstrate that with higher-quality prompts and filtering, GPT-3's performance can be significantly improved.&#10;&#10;Supervising on human preferences. &#10;Prior and concurrent work has used binary judgements from crowdworkers to fit models to human preferences for non-XAI tasks such as summarization \cite{ziegler2019fine, stiennon2020learning}, creating commonsense knowledge bases \cite{west2021symbolic}, and building natural language inference datasets \cite{liu2022wanli}. Unlike these works, we apply human preference modeling to increase the human acceptability of model-generated free-text explanations.&#10;\newcite{west2021symbolic} demonstrate that GPT-3 + a supervised acceptability filter can generate a high-quality causal knowledge graph: in addition to their work being conducted in a different domain, our success conditions and evaluation metrics differ because we must produce a prediction for each instance (whereas they can simply discard bad generations).&#10; " />
        </attvalues>
      </node>
      <node id="2305.11789" label="2305.11789">
        <attvalues>
          <attvalue for="0" value="Solving NLP Problems through Human-System Collaboration: A&#10;  Discussion-based Approach" />
          <attvalue for="1" value="  Humans work together to solve common problems by having discussions,&#10;explaining, and agreeing or disagreeing with each other. Similarly, if a system&#10;can have discussions with humans when solving tasks, it can improve the&#10;system's performance and reliability. In previous research on explainability,&#10;it has only been possible for the system to make predictions and for humans to&#10;ask questions about them rather than having a mutual exchange of opinions. This&#10;research aims to create a dataset and computational framework for systems that&#10;discuss and refine their predictions through dialogue. Through experiments, we&#10;show that the proposed system can have beneficial discussions with humans&#10;improving the accuracy by up to 25 points in the natural language inference&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11862" label="2305.11862">
        <attvalues>
          <attvalue for="0" value="Reducing Sequence Length by Predicting Edit Operations with Large&#10;  Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated remarkable performance in&#10;various tasks and gained significant attention. LLMs are also used for local&#10;sequence transduction tasks, including grammatical error correction (GEC) and&#10;formality style transfer, where most tokens in a source text are kept&#10;unchanged. However, the models that generate all target tokens in such tasks&#10;have a tendency to simply copy the input text as is, without making needed&#10;changes, because the difference between input and output texts is minimal in&#10;the training data. This is also inefficient because the computational cost&#10;grows quadratically with the target sequence length with Transformer. This&#10;paper proposes predicting edit spans for the source text for local sequence&#10;transduction tasks. Representing an edit span with a position of the source&#10;text and corrected tokens, we can reduce the length of the target sequence and&#10;the computational cost for inference. We apply instruction tuning for LLMs on&#10;the supervision data of edit spans. Experiments show that the proposed method&#10;achieves comparable performance to the baseline in four tasks, paraphrasing,&#10;formality style transfer, GEC, and text simplification, despite reducing the&#10;length of the target text by as small as 21%. Furthermore, we report that the&#10;task-specific fine-tuning with the proposed method achieved state-of-the-art&#10;performance in the four tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2304.01746" label="2304.01746">
        <attvalues>
          <attvalue for="0" value="Is ChatGPT a Highly Fluent Grammatical Error Correction System? A&#10;  Comprehensive Evaluation" />
          <attvalue for="1" value="  ChatGPT, a large-scale language model based on the advanced GPT-3.5&#10;architecture, has shown remarkable potential in various Natural Language&#10;Processing (NLP) tasks. However, there is currently a dearth of comprehensive&#10;study exploring its potential in the area of Grammatical Error Correction&#10;(GEC). To showcase its capabilities in GEC, we design zero-shot&#10;chain-of-thought (CoT) and few-shot CoT settings using in-context learning for&#10;ChatGPT. Our evaluation involves assessing ChatGPT's performance on five&#10;official test sets in three different languages, along with three&#10;document-level GEC test sets in English. Our experimental results and human&#10;evaluations demonstrate that ChatGPT has excellent error detection capabilities&#10;and can freely correct errors to make the corrected sentences very fluent,&#10;possibly due to its over-correction tendencies and not adhering to the&#10;principle of minimal edits. Additionally, its performance in non-English and&#10;low-resource settings highlights its potential in multilingual GEC tasks.&#10;However, further analysis of various types of errors at the document-level has&#10;shown that ChatGPT cannot effectively correct agreement, coreference, tense&#10;errors across sentences, and cross-sentence boundary errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12576" label="2305.12576">
        <attvalues>
          <attvalue for="0" value="Automated Few-shot Classification with Instruction-Finetuned Language&#10;  Models" />
          <attvalue for="1" value="  A particularly successful class of approaches for few-shot learning combines&#10;language models with prompts -- hand-crafted task descriptions that complement&#10;data samples. However, designing prompts by hand for each task commonly&#10;requires domain knowledge and substantial guesswork. We observe, in the context&#10;of classification tasks, that instruction finetuned language models exhibit&#10;remarkable prompt robustness, and we subsequently propose a simple method to&#10;eliminate the need for handcrafted prompts, named AuT-Few. This approach&#10;consists of (i) a prompt retrieval module that selects suitable task&#10;instructions from the instruction-tuning knowledge base, and (ii) the&#10;generation of two distinct, semantically meaningful, class descriptions and a&#10;selection mechanism via cross-validation. Over $12$ datasets, spanning $8$&#10;classification tasks, we show that AuT-Few outperforms current state-of-the-art&#10;few-shot learning methods. Moreover, AuT-Few is the best ranking method across&#10;datasets on the RAFT few-shot benchmark. Notably, these results are achieved&#10;without task-specific handcrafted prompts on unseen tasks.&#10;" />
          <attvalue for="2" value="&#10;Collecting annotated data is time-consuming and expensive. The goal of few-shot learning is to address this limitation by developing models that generalize from a small number of training examples.&#10;&#10;A now dominant paradigm in few-shot learning involves pre-training a large language model (PLM) on unsupervised language modelling objectives, combined with supervised fine-tuning \cite{kaplan2020scaling, wei2022emergent}. Fine-tuning on a variety of classification tasks improves generalization to new unseen tasks even further \cite{sanh2022multitask, wei2022emergent, chung2022scaling}. &#10;&#10;Prompts, instructions that describe the tasks in natural language, are crucial to successful fine-tuning on many tasks. &#10;Typically, prompts consist of two components: task templates and answer choices. Task templates are textual instructions about the task. Answer choices are semantic descriptions of the categorical labels. Supervised training on prompted samples, as shown in Figure \ref{fig:prompt-example}, helps PLMs generalize when instructed via prompts on a new problem (here natural language inference).&#10;Following~\cite{Lin2022UnsupervisedCG}, we use the term upstream model for these instruction-finetuned PLMs. These prompted upstream models provide state-of-the-art few-shot learning (~\cite{liu2022few}, yet they still rely on strenuous manual intervention from manually crafted prompts, designed by experts with domain knowledge about the underlying tasks. &#10;&#10;In this paper, we are concerned with an automated few-shot classification regime, where the algorithm can only access the training samples and their categorical labels. While efforts have been made to automate prompting, these methods are not directly transferable to upstream models. Most techniques target prompted masked language models (i.e. encoder-only models,&#10;that make predictions over continuous embeddings via its mask token \cite[][inter alia]{gao-etal-2021-making}. Automation methods for models with a discrete output space (i.e. a decoder over the vocabulary) are costly and limited to the automation of the task template, still relying on handcrafted descriptions of labels \cite{liu2021gpt, zhou2023large}.&#10;&#10;To automate few-shot learning with upstream models, we analyse the role of prompts across various classification tasks and we observe that upstream models exhibit low variability towards task-unspecific templates. In contrast, the selection of suitable answer choices can be important, yet answer choices do not need to be tailored to the specific instruction (e.g. Yes/No for a polar question). These insights confirm observations by \cite{webson-pavlick-2022-prompt} in a broader context and they motivate a simple few-shot learning automation method for upstream models, named AuT-Few.&#10;&#10;AuT-Few builds on the state-of-the-art learning method T-Few \cite{liu2022few}, but crucially does not use any task-specific handcrafted prompts. AuT-Few automatically finds the most relevant templates to our target task from the collection prompts used to instruction-tune the upstream model. As illustrated in Figure \ref{fig:method}, given an NLI task, AuT-Few might retrieve templates written for paraphrase identification.&#10;To automate answer choices, AuT-Few generates label descriptions tailored to the retrieved templates (e.g., Yes/No for a polar question, as for the illustrated paraphrase identification template) and descriptions that capture a class' overall topic (e.g. Enron/purchase for Enron spam classification). AuT-Few selects the most appropriate configuration via cross-validation. &#10;&#10;AuT-Few outperforms strong baselines, including T-Few \cite{liu2022few}, by $2.1$ points over a total of $12$ datasets, spanning $8$ tasks, without any task-specific handcrafted prompts. All but one task are unseen to the upstream models, indicating AuT-Few's strong generalization capabilities. &#10;Moreover, by applying AuT-Few to a small upstream model (BART0~\cite{Lin2022UnsupervisedCG}), we achieve competitive performance and efficiency to the current state-of-the-art prompt-free method, SetFit~\cite{tunstall2022efficient}. Furthermore, AuT-Few achieves the best average rank across datasets on the few-shot RAFT benchmark~\cite{alex2021raft}.&#10;An ablation justifies the components of our automation method.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Machine Learning, Computer Science, Linguistics, Language Models, Classification Tasks, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2205.05638" label="2205.05638">
        <attvalues>
          <attvalue for="0" value="Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than&#10;  In-Context Learning" />
          <attvalue for="1" value="  Few-shot in-context learning (ICL) enables pre-trained language models to&#10;perform a previously-unseen task without any gradient-based training by feeding&#10;a small number of training examples as part of the input. ICL incurs&#10;substantial computational, memory, and storage costs because it involves&#10;processing all of the training examples every time a prediction is made.&#10;Parameter-efficient fine-tuning (PEFT) (e.g. adapter modules, prompt tuning,&#10;sparse update methods, etc.) offers an alternative paradigm where a small set&#10;of parameters are trained to enable a model to perform the new task. In this&#10;paper, we rigorously compare few-shot ICL and PEFT and demonstrate that the&#10;latter offers better accuracy as well as dramatically lower computational&#10;costs. Along the way, we introduce a new PEFT method called (IA)$^3$ that&#10;scales activations by learned vectors, attaining stronger performance while&#10;only introducing a relatively tiny amount of new parameters. We also propose a&#10;simple recipe based on the T0 model called T-Few that can be applied to new&#10;tasks without task-specific tuning or modifications. We validate the&#10;effectiveness of T-Few on completely unseen tasks by applying it to the RAFT&#10;benchmark, attaining super-human performance for the first time and&#10;outperforming the state-of-the-art by 6% absolute. All of the code used in our&#10;experiments is publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;In this section, we provide am verview of ICL and PEFT with a focus on characterizing the computation, memory, and on-disk storage costs of making a prediction.&#10;Real-world costs depend on implementation and hardware, so we report costs in terms of FLOPs for computation and bytes for memory and storage, respectively.&#10;Additional related work is discussed in \ref{sec:related}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2209.11055" label="2209.11055">
        <attvalues>
          <attvalue for="0" value="Efficient Few-Shot Learning Without Prompts" />
          <attvalue for="1" value="  Recent few-shot methods, such as parameter-efficient fine-tuning (PEFT) and&#10;pattern exploiting training (PET), have achieved impressive results in&#10;label-scarce settings. However, they are difficult to employ since they are&#10;subject to high variability from manually crafted prompts, and typically&#10;require billion-parameter language models to achieve high accuracy. To address&#10;these shortcomings, we propose SetFit (Sentence Transformer Fine-tuning), an&#10;efficient and prompt-free framework for few-shot fine-tuning of Sentence&#10;Transformers (ST). SetFit works by first fine-tuning a pretrained ST on a small&#10;number of text pairs, in a contrastive Siamese manner. The resulting model is&#10;then used to generate rich text embeddings, which are used to train a&#10;classification head. This simple framework requires no prompts or verbalizers,&#10;and achieves high accuracy with orders of magnitude less parameters than&#10;existing techniques. Our experiments show that SetFit obtains comparable&#10;results with PEFT and PET techniques, while being an order of magnitude faster&#10;to train. We also show that SetFit can be applied in multilingual settings by&#10;simply switching the ST body. Our code is available at&#10;https://github.com/huggingface/setfit and our datasets at&#10;https://huggingface.co/setfit .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;SetFit engages with two related lines of literature. We first extend the small but significant body of work on Sentence Transformers (ST) for text classification. \cite{Sent_emb_2018} introduced the idea of using sentence embeddings for text classification.&#10;\cite{Sent_tranformers_2021} used 'out-of-the-box' STs for text classification without fine-tuning them. SetFit differs from these works in two aspects: First, we fine-tune the ST in a Siamese manner for a text classification objective showing that it significantly enhances performance; second, we demonstrate this approach in few-shot setups.&#10;&#10;SetFit is also related to the recently emerging few-shot and zero-shot training line of literature as &#10;few-shot and zero-shot approaches have recently received a great deal of interest in the research community due to the availability of pretrained language models and the untapped capacity to use them in resource-constrained domains. Specifically, we discuss ICL, PEFT, and prompt-based fine-tuning.&#10;&#10;ICL models directly generate predictions based on input-to-output training examples provided as prompts, without any parameter updates. Perhaps the best known example is GPT-3 \cite{DBLP:journals/corr/abs-2005-14165}, which achieves remarkable few-shot performance. However, GPT-3 contains 175 billion parameters and requires massive computational resources, prompt engineering, and can only utilize pretrained knowledge.&#10;&#10;PEFT methods, such as adapters \cite{adapter}, hold the majority of parameters fixed during training and only update small feed-forward networks that are inserted within the larger model architecture. A recent example is T-Few \cite{t-few}, which outperforms GPT-3 at much lower computational cost. It accomplishes this by adding learned vectors that rescale the network's internal activations. T-Few is 16 times smaller than GPT-3, but is still too large to be utilized as a practical tool in industry. It also requires a set of handcrafted prompts for each dataset.&#10;&#10;Another alternative to ICL is prompt-based fine-tuning. This approach converts the downstream classification task into a masked-language modeling (MLM) objective. The model outputs tokens in a cloze-style format that maps to the corresponding labels via a predefined template.&#10;A well known example of this method is Pattern Exploiting Training (PET) \cite{schick-schutze-2021-just, schick-schutze-2021-exploiting}. Like GPT-3, PET relies on manually-crafted prompts, but since the model can be fine-tuned to specific tasks, PET-based approaches typically outperform GPT-3 in few-shot scenarios, even with far smaller PLM backbones. PET has since been extended in two main directions: Adapet \cite{tam-etal-2021-improving}, which improves PET with a decoupled label objective and label-conditioned MLM objective, and Perfect \cite{PERFECT} which uses task-specific adapters \cite{houlsby2019,pfeiffer-etal-2021-adapterfusion} and multi-token label-embeddings eliminate task prompts and verbalizers.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2109.14076" label="2109.14076">
        <attvalues>
          <attvalue for="0" value="RAFT: A Real-World Few-Shot Text Classification Benchmark" />
          <attvalue for="1" value="  Large pre-trained language models have shown promise for few-shot learning,&#10;completing text-based tasks given only a few task-specific examples. Will&#10;models soon solve classification tasks that have so far been reserved for human&#10;research assistants? Existing benchmarks are not designed to measure progress&#10;in applied settings, and so don't directly answer this question. The RAFT&#10;benchmark (Real-world Annotated Few-shot Tasks) focuses on naturally occurring&#10;tasks and uses an evaluation setup that mirrors deployment. Baseline&#10;evaluations on RAFT reveal areas current techniques struggle with: reasoning&#10;over long texts and tasks with many classes. Human baselines show that some&#10;classification tasks are difficult for non-expert humans, reflecting that&#10;real-world value sometimes depends on domain expertise. Yet even non-expert&#10;human baseline F1 scores exceed GPT-3 by an average of 0.11. The RAFT datasets&#10;and leaderboard will track which model improvements translate into real-world&#10;benefits at https://raft.elicit.org .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;We briefly review few-shot learning in NLP, then the benchmarks that are most similar to RAFT.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2406.15708" label="2406.15708">
        <attvalues>
          <attvalue for="0" value="Teach Better or Show Smarter? On Instructions and Exemplars in Automatic&#10;  Prompt Optimization" />
          <attvalue for="1" value="  Large language models have demonstrated remarkable capabilities, but their&#10;performance is heavily reliant on effective prompt engineering. Automatic&#10;prompt optimization (APO) methods are designed to automate this and can be&#10;broadly categorized into those targeting instructions (instruction&#10;optimization, IO) vs. those targeting exemplars (exemplar selection, ES).&#10;Despite their shared objective, these have evolved rather independently, with&#10;IO recently receiving more research attention. This paper seeks to bridge this&#10;gap by comprehensively comparing the performance of representative IO and ES&#10;techniques, both isolation and combination, on a diverse set of challenging&#10;tasks. Our findings reveal that intelligently reusing model-generated&#10;input-output pairs obtained from evaluating prompts on the validation set as&#10;exemplars consistently improves performance over IO methods but is currently&#10;under-investigated. We also find that despite the recent focus on IO, how we&#10;select exemplars can outweigh how we optimize instructions, with ES strategies&#10;as simple as random search outperforming state-of-the-art IO methods with seed&#10;instructions without any optimization. Moreover, we observe synergy between ES&#10;and IO, with optimal combinations surpassing individual contributions. We&#10;conclude that studying exemplar selection as a standalone method and its&#10;optimal combination with instruction optimization remains a crucial aspect of&#10;APO and deserves greater consideration in future research, even in the era of&#10;highly capable instruction-following models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Significant advancements in large language models (LLMs) have revolutionized various natural language processing tasks \cite{chowdhery2023palm, anil2023palm, achiam2023gpt, team2023gemini}. &#10;One notable aspect of LLMs, however, is their sensitivity to the input &quot;prompts,&quot; which has given rise to the burgeoning field of prompt engineering \cite{liu2023pre, sahoo2024systematic}. &#10;On black-box LLMs where we can neither modify or access internal parameters, prompt engineering involves crafting input prompts that effectively guide LLMs to generate desired outputs. &#10;Starting from manual processes requiring human expertise, the complexity and volume of prompts have necessitated the development of automatic prompt optimization (APO) methods&#10;aiming to streamline and automate prompt generation, thereby alleviating the burden of manual intervention. &#10;Broadly, since prompts consist of instructions and exemplars, we may roughly categorize APO into instruction optimization (IO) and exemplar optimization (EO) approaches. &#10;IO &#10;focuses on refining the textual instructions provided to LLMs that contain task-specific information (i.e., to teach), whereas EO &#10;emphasizes the selection of relevant examples to guide model behavior (i.e., to show). &#10;Partially driven by the improved instruction-following ability of LLMs, the research attention has increasingly shifted towards IO, especially using LLMs themselves as optimizers~\cite{zhou2022large,pryzant2023automatic,wang2023promptagent}.&#10;&#10;While EO and IO approaches address the similar overarching problem, they have evolved somewhat independently, &#10;with a few exceptions \cite{fernando2023promptbreeder, wang2023mixture}.&#10;Indeed, as we elaborate in \S\ref{sec:preliminaries}, EO approaches are often based on simple, handcrafted templates without explicit instruction optimization~\cite{khattab2023dspy, wan2023better}, &#10;while IO methods seldom optimize exemplars and often rely on random validation set samples \cite{pryzant2023automatic}, require additional fixed exemplars on top of the validation set \cite{zhou2022large, guo2023connecting}, or consider the ``zero-shot'' setup with no exemplars at all~\cite{wang2023promptagent}. &#10;Whereas the lack of IO in EO methods is somewhat understandable as many EO approaches predate instruction finetuning \cite{wei2022finetuned} and, subsequently, instruction-following models that are sensitive to instructions, the inverse is much less so: concretely, almost all existing IO approaches already require a labeled dataset as the validation set,&#10;and are therefore, by definition, not ``zero-shot''. With the inputs, labels, and, if applicable, model-generated intermediate outputs (e.g., reasoning steps) on the subset of the validation set that the model has answered correctly, we already have a set of exemplars as a free side-product whenever we perform IO, independent from and on top to any additionally provided, human-annotated exemplars. &#10;A common argument for not focusing on EO, such as mentioned in \cite{pryzant2023automatic}, is the goal to focus on one objective at a time. &#10;However, given the common practical goal of and the interplay between EO and IO~\cite{mizrahi2023state}, we argue they should not be treated separately -- it is instead critical to understand their relative importance and combined impact, and, where necessary, optimize them jointly for the best performance-cost balance.&#10;&#10;This is, to our knowledge, where there is a gap in the literature that we aim to bridge. To do so, on a diverse suite of challenging BIG-Bench and MMLU tasks, &#10;we compare the performance gain brought by various representative, state-of-the-art (SoTA) IO and EO methods on the fairground with PaLM 2, Gemini (1.0/1.5) and GPT models to foster better scientific understanding of different APO techniques. While IO comfortably improves the baseline prompts before any instruction or exemplar optimization, this, at best, portrays an incomplete picture. Under the same setup, with simple yet effective EO methods on the model-generated exemplars on the validation set, we show:&#10;\begin{itemize}[leftmargin=8pt, nosep]&#10; \item Intelligently incorporating exemplars generated by the target model itself on the validation set significantly and consistently improves performance on top of recently proposed IO methods;&#10; \item The performance gains realized by choosing appropriate exemplars via methods as simple as random search can eclipse the improvements brought by SoTA instruction optimization. As a concrete example, as shown in Fig.~\ref{fig:overall}, with a simple optimization routine on exemplars, seed instructions before any optimization outperform optimized instructions obtained with complex IO but with random exemplars most commonly used.&#10; \item There exists a synergy between EO and IO, and optimally mixing-and-matching IO and EO is greater than the sum of its parts under a comparable computational budget.&#10; \item SoTA IO might be itself implicitly generating exemplars, and these exemplars, while somewhat unintentional, contribute more to the performance than the rest of the instruction that IO methods are meant to optimize.&#10; \item While arguably receiving less research attention recently, exemplar optimization remains a crucial design consideration in APO. Even in an era of highly capable instruction-following LLMs, the significance of exemplar optimization should not be relegated to an afterthought, and better exemplar optimization both as a standalone tool and as a combinable component with IO is crucial for APO.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Prompt Engineering Techniques, Automatic Prompt Optimization, Artificial Intelligence, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2310.16427" label="2310.16427">
        <attvalues>
          <attvalue for="0" value="PromptAgent: Strategic Planning with Language Models Enables&#10;  Expert-level Prompt Optimization" />
          <attvalue for="1" value="  Highly effective, task-specific prompts are often heavily engineered by&#10;experts to integrate detailed instructions and domain insights based on a deep&#10;understanding of both instincts of large language models (LLMs) and the&#10;intricacies of the target task. However, automating the generation of such&#10;expert-level prompts remains elusive. Existing prompt optimization methods tend&#10;to overlook the depth of domain knowledge and struggle to efficiently explore&#10;the vast space of expert-level prompts. Addressing this, we present&#10;PromptAgent, an optimization method that autonomously crafts prompts equivalent&#10;in quality to those handcrafted by experts. At its core, PromptAgent views&#10;prompt optimization as a strategic planning problem and employs a principled&#10;planning algorithm, rooted in Monte Carlo tree search, to strategically&#10;navigate the expert-level prompt space. Inspired by human-like trial-and-error&#10;exploration, PromptAgent induces precise expert-level insights and in-depth&#10;instructions by reflecting on model errors and generating constructive error&#10;feedback. Such a novel framework allows the agent to iteratively examine&#10;intermediate prompts (states), refine them based on error feedbacks (actions),&#10;simulate future rewards, and search for high-reward paths leading to expert&#10;prompts. We apply PromptAgent to 12 tasks spanning three practical domains:&#10;BIG-Bench Hard (BBH), as well as domain-specific and general NLP tasks, showing&#10;it significantly outperforms strong Chain-of-Thought and recent prompt&#10;optimization baselines. Extensive analyses emphasize its capability to craft&#10;expert-level, detailed, and domain-insightful prompts with great efficiency and&#10;generalizability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.16797" label="2309.16797">
        <attvalues>
          <attvalue for="0" value="Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution" />
          <attvalue for="1" value="  Popular prompt strategies like Chain-of-Thought Prompting can dramatically&#10;improve the reasoning abilities of Large Language Models (LLMs) in various&#10;domains. However, such hand-crafted prompt-strategies are often sub-optimal. In&#10;this paper, we present Promptbreeder, a general-purpose self-referential&#10;self-improvement mechanism that evolves and adapts prompts for a given domain.&#10;Driven by an LLM, Promptbreeder mutates a population of task-prompts, and&#10;subsequently evaluates them for fitness on a training set. Crucially, the&#10;mutation of these task-prompts is governed by mutation-prompts that the LLM&#10;generates and improves throughout evolution in a self-referential way. That is,&#10;Promptbreeder is not just improving task-prompts, but it is also improving the&#10;mutationprompts that improve these task-prompts. Promptbreeder outperforms&#10;state-of-the-art prompt strategies such as Chain-of-Thought and Plan-and-Solve&#10;Prompting on commonly used arithmetic and commonsense reasoning benchmarks.&#10;Furthermore, Promptbreeder is able to evolve intricate task-prompts for the&#10;challenging problem of hate speech classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Prompting an LLM in the right way is essential to its downstream performance~\cite{DBLP:conf/emnlp/MoradiS21,DBLP:journals/corr/abs-2209-07686,zhou2022large}. Indeed, even the order in which prompts are presented can heavily influence LLM performance~\cite{DBLP:conf/acl/LuBM0S22}. A number of recent works have focused on devising better prompt strategies, or even automating such prompt engineering.&#10;&#10;Prompting:&#10;Chain-of-Thought Prompting~\cite[CoT,][]{DBLP:conf/nips/Wei0SBIXCLZ22} is a popular prompt strategy which provides intermediate reasoning steps as few-shot prompts to an LLM, thereby significantly improving its arithmetic, commonsense, and symbolic reasoning abilities. Notably, the gains of CoT are more pronounced for stronger LLMs. This is intriguing, as it points to the possibility of increasingly capable (and potentially open-ended) self-improving mechanisms on top of adept LLMs---a hypothesis that Promptbreeder directly builds upon. Instead of few-shot CoT prompting, \cite{DBLP:conf/nips/KojimaGRMI22} demonstrate that LLMs can also be prompted zero-shot (e.g. {&quot;Let's think step by step&quot;}) to produce their own chains of thoughts (Zero-shot CoT) that improve reasoning abilities. Self-Consistency~\cite[CoT-SC,][]{wang2022self} extends CoT by sampling a diverse set of workings out and selecting the most consistent answer. Tree of Thoughts~\cite[ToT,][]{yao2023tree} generalizes CoT to multiple workings out that can be expanded or backtracked from. Graph of Thoughts~\cite[GoT,][]{DBLP:journals/corr/abs-2308-09687} is a further generalization to arbitrary graph structures. Plan-and-Solve Prompting~\cite[PS,][]{DBLP:conf/acl/WangXLHLLL23} encourages an LLM to first devise a plan to solve a problem before attempting to solve it. Similarly, Least-to-Most Prompting~\cite{zhou2022least} encourages an LLM to decompose a problem into subparts, and then to solve each part individually before synthesizing an answer. Self-Refine~\cite{DBLP:journals/corr/abs-2303-17651} prompts an LLM to generate a response, to provide feedback on the response, and to finally refine the solution.&#10;&#10;In contrast to gradient-free approaches above, Soft Prompting approaches~\cite[e.g.,][]{DBLP:journals/corr/abs-2103-10385,qin2021learning,DBLP:conf/emnlp/LesterAC21} directly fine-tune continuous prompt representations. \cite{DBLP:journals/corr/abs-2210-11610} use CoT and CoT-SC on an unlabelled dataset of questions, and subsequently fine-tune an LLM based on generated solutions. Similarly, \cite{DBLP:conf/nips/ZelikmanWMG22} uses CoT to generate rationales and fine-tunes the LLM based on those examples and rationales that yielded the correct answer. However, as argued by~\cite{zhou2022large}, any approach that updates all or a portion of LLM parameters will not scale as models get bigger and, moreover, will not work with the increasing number of LLMs hidden behind an API.&#10;&#10;All of the prompt engineering approaches above are domain agnostic but hand designed. Central to our work is the hypothesis that we could do better by employing an automated self-improvement process that can adapt prompts to a domain at hand. Auto-CoT~\cite{DBLP:conf/iclr/0001Z0S23} and Automatic-CoT~\cite{DBLP:journals/corr/abs-2302-12822} automatically find reasoning chains for Few-Shot CoT. Automatic Prompt Engineer~\cite[APE,][]{zhou2022large} uses one generator-prompt to generate prompt candidates, and another mutation-prompt to mutate them. In contrast to APE, our work performs compositional task-specific initialization of mutation-prompts, subsequent online mutation of mutation-prompts, uses special mutation operators that take into account the whole population and elite history, and uses diversity-maintenance methods---all of which help avoid the problem of diminishing returns and diversity loss suffered by APE. &#10;&#10;Concurrently to our work, \cite{DBLP:journals/corr/abs-2309-03409} developed Optimization by PROmpting (OPRO), a prompt optimization method that varies prompts using a single complex mutation prompt, and evaluates newly generated prompts on a small fixed training set of problems. In contrast, Promptbreeder autonomously evolves multiple LLM generated mutation-prompts as well as task-prompts, and evaluates fitness on random subsets from the whole training set during evolution. At the time of its release, OPRO achieved a score of 80.2\% via the optimized zero-shot prompt {&quot;Take a deep breath and work on this problem step-by-step&quot;} on GSM8K. Promptbreeder surpasses this with 83.9\% in the zero-shot setting with the unintuitively simple prompt {&quot;SOLUTION&quot;&quot;}---further evidence for the sensitivity of LLMs to prompts and the importance on finding effective prompts automatically. Also concurrently to our work, \cite{guo2023connecting} developed EvoPrompt, which uses a fixed mutation (and crossover) prompt, as well as a prompt that asks for a mutant of the difference between two parent prompts, to produce offspring prompts. EvoPrompt is initialized with a whole population of initial hand-designed task tailored prompts rather than a single problem description as we do. In contrast to the two approaches above, Promptbreeder uses LLMs to self-referentially improve mutation-prompts, and it is able to evolve contexts as well. &#10;&#10;Self-Referential Self-Improvement: &#10;Developing an open-ended system that can improve itself as well as improving the way it is improving itself~\cite{schmidhuber1993selfreferential,schmidhuber2003godel} is a long-standing open problem in AI research. \cite{schmidhuber1993selfreferential} introduced an ``introspective'' neural network with a self-referential weight matrix that can modify its own weights and, thus, also modify those weights that are governing how its own weights are modified. Recently, \cite{DBLP:conf/icml/IrieSCS22} proposed a more scalable self-referential weight matrix taking inspiration from fast weight programmers~\cite{schmidhuber1992learning}. \cite{DBLP:journals/corr/abs-2212-14392} propose a self-referential meta-learning approach, combining self-referential weight matrices with ideas from Gödel Machines~\cite{schmidhuber2003godel}, i.e., to allocate more computational resources to better performing solutions. However, since these approaches directly modify parameters of a model, it is unclear how to scale them to the increasing number of parameters in modern LLMs. In contrast, for Promptbreeder the substrate of self-referential self-improvement is natural language, avoiding costly parameter updates altogether.&#10;&#10;Open-Endedness and LLMs: &#10;Promptbreeder makes use of the observation by \cite{DBLP:journals/corr/abs-2206-08896}, \cite{DBLP:journals/corr/abs-2302-12170} and \cite{DBLP:journals/corr/abs-2302-14838} that LLMs are effective at generating mutations from examples. In addition, LLMs encode human notions of interestingness and can be used to automatically quantify novelty \cite{DBLP:journals/corr/abs-2306-01711}. Promptbreeder is related to Picbreeder~\cite{secretan2008picbreeder}, an open-ended human-in-the-loop system that evolves increasingly interesting images. While Picbreeder explores the space of images, Promptbreeder explores the space of prompts and does so without humans in the loop. As Promptbreeder is proposing mutated prompts to itself, it is an example of a system transitioning from ``learning from data'' to ``learning what data to learn from''~\cite{DBLP:journals/corr/abs-2211-07819}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2310.03714" label="2310.03714">
        <attvalues>
          <attvalue for="0" value="DSPy: Compiling Declarative Language Model Calls into Self-Improving&#10;  Pipelines" />
          <attvalue for="1" value="  The ML community is rapidly exploring techniques for prompting language&#10;models (LMs) and for stacking them into pipelines that solve complex tasks.&#10;Unfortunately, existing LM pipelines are typically implemented using hard-coded&#10;&quot;prompt templates&quot;, i.e. lengthy strings discovered via trial and error. Toward&#10;a more systematic approach for developing and optimizing LM pipelines, we&#10;introduce DSPy, a programming model that abstracts LM pipelines as text&#10;transformation graphs, i.e. imperative computational graphs where LMs are&#10;invoked through declarative modules. DSPy modules are parameterized, meaning&#10;they can learn (by creating and collecting demonstrations) how to apply&#10;compositions of prompting, finetuning, augmentation, and reasoning techniques.&#10;We design a compiler that will optimize any DSPy pipeline to maximize a given&#10;metric. We conduct two case studies, showing that succinct DSPy programs can&#10;express and optimize sophisticated LM pipelines that reason about math word&#10;problems, tackle multi-hop retrieval, answer complex questions, and control&#10;agent loops. Within minutes of compiling, a few lines of DSPy allow GPT-3.5 and&#10;llama2-13b-chat to self-bootstrap pipelines that outperform standard few-shot&#10;prompting (generally by over 25% and 65%, respectively) and pipelines with&#10;expert-created demonstrations (by up to 5-46% and 16-40%, respectively). On top&#10;of that, DSPy programs compiled to open and relatively small LMs like&#10;770M-parameter T5 and llama2-13b-chat are competitive with approaches that rely&#10;on expert-written prompt chains for proprietary GPT-3.5. DSPy is available at&#10;https://github.com/stanfordnlp/dspy&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.00595" label="2401.00595">
        <attvalues>
          <attvalue for="0" value="State of What Art? A Call for Multi-Prompt LLM Evaluation" />
          <attvalue for="1" value="  Recent advances in large language models (LLMs) have led to the development&#10;of various evaluation benchmarks. These benchmarks typically rely on a single&#10;instruction template for evaluating all LLMs on a specific task. In this paper,&#10;we comprehensively analyze the brittleness of results obtained via&#10;single-prompt evaluations across 6.5M instances, involving 20 different LLMs&#10;and 39 tasks from 3 benchmarks. To improve robustness of the analysis, we&#10;propose to evaluate LLMs with a set of diverse prompts instead. We discuss&#10;tailored evaluation metrics for specific use cases (e.g., LLM developers vs.&#10;developers interested in a specific downstream task), ensuring a more reliable&#10;and meaningful assessment of LLM capabilities. We then implement these criteria&#10;and conduct evaluations of multiple models, providing insights into the true&#10;strengths and limitations of current LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Our work is part of an emerging trend highlighting the many challenges standing in the way of meaningful, scalable, and reproducible evaluation of large language models. &#10;&#10;\cite{Perlitz2023EfficientB} focus on the rising cost of exhaustive evaluation of LLMs on large number of samples. &#10;They developed methods for choosing subsets of the test data which are expected to be a good representative of the whole. &#10;An interesting avenue for future work can extend \cite{Perlitz2023EfficientB}'s approach to also include various instruction templates, thus efficiently approximating our suggested evaluation methods.&#10;&#10;\cite{Sclar2023QuantifyingLM} show that LLMs are sensitive to prompt formatting. These are minor prompt design choices, such as the addition or omission of punctuation marks. They create a large pool of instruction paraphrases, ensuring that paraphrases maintain the meaning of the original prompt. We notice a similar phenomenon,&#10;albeit more anecdotally, when our automatic paraphrasing techniques incidentally produce minor changes in formatting (Table~\ref{tab:openai_edit_distance}). &#10;\cite{voronov2024mind} showed that LLMs are sensitive to the format of in-context examples. For example, they varied the manner in which each input-output is separated, and test how such choices interact with the phrasing of the instruction template, the number of demonstrations, or the model size. &#10;&#10;{The works discussed above represent a distinct thread within the larger field of model robustness, which is typically defined as a measure of models' ability to adapt to distribution shifts between training and inference~\cite{wang-etal-2022-measure}, or to cope with adversarial examples~\cite{wang2021adversarial,wang2023robustness}. In contrast, these works do not change the underlying instance to be classified (e.g., the homophone pairs in our running example), but rather the task instruction. This challenge arises with the introduction of LLMs which take such instructions as part of the input, rather than through dedicated calibration in training or finetuning.} &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2101.00190" label="2101.00190">
        <attvalues>
          <attvalue for="0" value="Prefix-Tuning: Optimizing Continuous Prompts for Generation" />
          <attvalue for="1" value="  Fine-tuning is the de facto way to leverage large pretrained language models&#10;to perform downstream tasks. However, it modifies all the language model&#10;parameters and therefore necessitates storing a full copy for each task. In&#10;this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning&#10;for natural language generation tasks, which keeps language model parameters&#10;frozen, but optimizes a small continuous task-specific vector (called the&#10;prefix). Prefix-tuning draws inspiration from prompting, allowing subsequent&#10;tokens to attend to this prefix as if it were &quot;virtual tokens&quot;. We apply&#10;prefix-tuning to GPT-2 for table-to-text generation and to BART for&#10;summarization. We find that by learning only 0.1\% of the parameters,&#10;prefix-tuning obtains comparable performance in the full data setting,&#10;outperforms fine-tuning in low-data settings, and extrapolates better to&#10;examples with topics unseen during training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Fine-tuning for natural language generation. &#10;Current state-of-the-art systems for natural language generation are based on fine-tuning pretrained LMs. &#10;For table-to-text generation, \cite{kale2020texttotext} fine-tunes a sequence-to-sequence model \cite[T5;][]{T5}. &#10;For extractive and abstractive summarization, researchers fine-tune masked language models \cite[e.g., BERT;][]{devlin-etal-2019-bert} and encode-decoder models \cite[e.g., BART;][]{lewis-etal-2020-bart} respectively \cite{zhong-etal-2020-extractive, liu-lapata-2019-text,T5}.&#10;For other conditional NLG tasks such as machine translation and dialogue generation, fine-tuning is also the prevalent paradigm \cite{zhang-etal-2020-dialogpt, stickland2020recipes, Zhu2020Incorporating, liu2020multilingual}. &#10;In this paper, we focus on table-to-text using GPT-2 and summarization using BART, but prefix-tuning can be applied to other generation tasks and pretrained models.&#10;&#10;Lightweight fine-tuning.&#10;Lightweight fine-tuning freezes most of the pretrained parameters and modifies the pretrained model with small trainable modules. The key challenge is to identify high-performing architectures of the modules and the subset of pretrained parameters to tune. &#10;One line of research considers removing parameters: some model weights are ablated away by training a binary mask over model parameters \cite{zhao2020masking,dixit2020How}.&#10;Another line of research considers inserting parameters.&#10;For example, \cite{zhang2020sidetuning} trains a ``side'' network that is fused with the pretrained model via summation; adapter-tuning inserts task-specific layers (adapters) between each layer of the pretrained LM \cite{pmlr-v97-houlsby19a,lin-etal-2020-exploring,Rebuffi2017Adapter,pfeiffer2020adapterfusion}.&#10;Compared to this line of work, which tunes around $3.6\%$ of the LM parameters, our method obtains a further 30x reduction in task-specific parameters, tuning only 0.1\% while maintaining comparable performance. \looseness=-1&#10;&#10;Prompting. &#10;&#10;Prompting means prepending instructions and a few examples to the task input and generating the output from the LM. GPT-3 \cite{brown2020language} uses manually designed prompts to adapt its generation for different tasks, and this framework is termed in-context learning. However, since Transformers can only condition on a bounded-length context (e.g., 2048 tokens for GPT-3), in-context learning is unable to fully exploit training sets longer than the context window. &#10;\cite{sun2020conditioned} also prompt by keywords to control for sentiment or topic of the generated sentence. &#10;In natural language understanding tasks, prompt engineering has been explored in prior works for models like BERT and RoBERTa \cite{liu-etal-roberta, jiang-etal-2020-know, schick2020exploiting}. For example, AutoPrompt \cite{shin2020autoprompt} searches for a sequence of discrete trigger words and concatenates it with each input to elicit sentiment or factual knowledge from a masked LM. &#10;In contrast with AutoPrompt, our method optimizes continuous prefixes, which are more expressive (\ref{ssec:embed_only}); moreover, we focus on language generation tasks. \looseness=-1&#10;&#10;Continuous vectors have been used to steer language models; for example, \cite{subramani2020unconditional} showed that a pretrained LSTM language model can reconstruct arbitrary sentences by optimizing a continuous vector for each sentence, making the vector input-specific. In contrast, prefix-tuning optimizes a task-specific prefix that applies to all instances of that task. As a result, unlike the previous work whose application is limited to sentence reconstruction, prefix-tuning can be applied to NLG tasks. &#10;&#10;Controllable generation. &#10; Controllable generation aims to steer a pretrained language model to match a sentence level attribute (e.g., positive sentiment or topic on sports). Such control can happen at training time: \cite{Keskar2019CTRL} pretrains the language model (CTRL) to condition on metadata such as keywords or URLs. Additionally, the control can happen at decoding time, by weighted decoding \cite[GeDi,][]{KrauseGeDi2020} or iteratively updating the past activations \cite[PPLM,][]{Dathathri2020Plug}. &#10; However, there is no straightforward way to apply these controllable generation techniques to enforce fine-grained control over generated contents, as demanded by tasks like table-to-text and summarization.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2101.00121" label="2101.00121">
        <attvalues>
          <attvalue for="0" value="WARP: Word-level Adversarial ReProgramming" />
          <attvalue for="1" value="  Transfer learning from pretrained language models recently became the&#10;dominant approach for solving many NLP tasks. A common approach to transfer&#10;learning for multiple tasks that maximize parameter sharing trains one or more&#10;task-specific layers on top of the language model. In this paper, we present an&#10;alternative approach based on adversarial reprogramming, which extends earlier&#10;work on automatic prompt generation. Adversarial reprogramming attempts to&#10;learn task-specific word embeddings that, when concatenated to the input text,&#10;instruct the language model to solve the specified task. Using up to 25K&#10;trainable parameters per task, this approach outperforms all existing methods&#10;with up to 25M trainable parameters on the public leaderboard of the GLUE&#10;benchmark. Our method, initialized with task-specific human-readable prompts,&#10;also works in a few-shot setting, outperforming GPT-3 on two SuperGLUE tasks&#10;with just 32 training samples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2405.08011" label="2405.08011">
        <attvalues>
          <attvalue for="0" value="A Survey of Large Language Models for Graphs" />
          <attvalue for="1" value="  Graphs are an essential data structure utilized to represent relationships in&#10;real-world scenarios. Prior research has established that Graph Neural Networks&#10;(GNNs) deliver impressive outcomes in graph-centric tasks, such as link&#10;prediction and node classification. Despite these advancements, challenges like&#10;data sparsity and limited generalization capabilities continue to persist.&#10;Recently, Large Language Models (LLMs) have gained attention in natural&#10;language processing. They excel in language comprehension and summarization.&#10;Integrating LLMs with graph learning techniques has attracted interest as a way&#10;to enhance performance in graph learning tasks. In this survey, we conduct an&#10;in-depth review of the latest state-of-the-art LLMs applied in graph learning&#10;and introduce a novel taxonomy to categorize existing methods based on their&#10;framework design. We detail four unique designs: i) GNNs as Prefix, ii) LLMs as&#10;Prefix, iii) LLMs-Graphs Integration, and iv) LLMs-Only, highlighting key&#10;methodologies within each category. We explore the strengths and limitations of&#10;each framework, and emphasize potential avenues for future research, including&#10;overcoming current integration challenges between LLMs and graph learning&#10;techniques, and venturing into new application areas. This survey aims to serve&#10;as a valuable resource for researchers and practitioners eager to leverage&#10;large language models in graph learning, and to inspire continued progress in&#10;this dynamic field. We consistently maintain the related open-source materials&#10;at \url{https://github.com/HKUDS/Awesome-LLM4Graph-Papers}.&#10;" />
          <attvalue for="2" value="&#10;&#10;Graphs, comprising nodes and edges that signify relationships, are essential for illustrating real-world connections across various domains. These include social networks~\cite{myers2014information, li2024urbangpt}, molecular graphs~\cite{InstructMol}, recommender systems~\cite{he2020lightgcn, DCCF}, and academic networks~\cite{hu2020open}. This structured data form is integral in mapping complex interconnections relevant to a wide range of applications.&#10;&#10;In recent years, Graph Neural Networks (GNNs)~\cite{wu2020comprehensive} have emerged as a powerful tool for a variety of tasks, including node classification~\cite{GIN} and link prediction~\cite{zhang2018link}. By passing and aggregating information across nodes and iteratively refining node features through supervised learning, GNNs have achieved remarkable results in capturing structural nuances and enhancing model accuracy. To accomplish this, GNNs leverage graph labels to guide the learning process. Several notable models have been proposed in the literature, each with its own strengths and contributions. For instance, Graph Convolutional Networks (GCNs)~\cite{gcn} have been shown to be effective in propagating embeddings across nodes, while Graph Attention Networks (GATs)~\cite{gat} leverage attention mechanisms to perform precise aggregation of node features. Additionally, Graph Transformers~\cite{GTN, GFormer} employ self-attention and positional encoding to capture global signals among the graph, further improving the expressiveness of GNNs. To address scalability challenges in large graphs, methods such as Nodeformer~\cite{NodeFormer} and DIFFormer~\cite{DIFFormer} have been proposed. These approaches employ efficient attention mechanisms and differentiable pooling techniques to reduce computational complexity while maintaining high levels of accuracy. Despite these advancements, current GNN methodologies still face several challenges. For example, data sparsity remains a significant issue, particularly in scenarios where the graph structure is incomplete or noisy~\cite{you2021graph}. Moreover, the generalization ability of GNNs to new graphs or unseen nodes remains an open research question, with recent works highlighting the need for more robust and adaptive models~\cite{garg2020generalization, OpenGraph, zhao2024graphany}.&#10;&#10;Large Language Models (LLMs)~\cite{zhao2023survey}, which show great generalization abilities for unseen tasks~\cite{BERT, T5, wang2023far}, have emerged as powerful tools in various research fields, including natural language processing~\cite{achiam2023gpt}, computer vision~\cite{liu2024visual, liu2024improved}, and information retrieval~\cite{zhu2023collaborative, hou2024large, lin2024data}.&#10;The advent of LLMs has sparked significant interest within the graph learning community~\cite{huang2024large, jin2023large, li2023survey}, prompting investigations into the potential of LLMs to enhance performance on graph-related tasks. Researchers have explored various approaches to leverage the strengths of LLMs for graph learning, resulting in a new wave of methods that combine the power of LLMs with graph neural networks.&#10;One promising direction is to develop prompts that enable LLMs to understand graph structures and respond to queries effectively. For instance, approaches such as InstructGLM~\cite{InstructGLM} and NLGraph~\cite{NLGraph} have designed specialized prompts that allow LLMs to reason over graph data and generate accurate responses.&#10;Alternatively, other methods have integrated GNNs to feed tokens into the LLMs, allowing them to understand graph structures more directly. For example, GraphGPT~\cite{GraphGPT} and GraphLLM~\cite{GraphLLM} use GNNs to encode graph data into tokens, which are then fed into the LLMs for further processing. This synergy between LLMs and GNNs has not only improved task performance but also demonstrated impressive zero-shot generalization capabilities, where the models can accurately answer queries about unseen graphs or nodes.&#10;&#10;In this survey, we offer a systematic review of the advancements in Large Language Models (LLMs) for graph applications, and we explore potential avenues for future research. Unlike prior surveys that categorize studies based on the role of LLMs~\cite{jin2023large, li2023survey} or focus primarily on integrating LLMs with knowledge graphs~\cite{pan2024unifying}, our work highlights the model framework design, particularly the inference and training processes, to distinguish between existing taxonomies. This perspective allows readers to gain a deeper understanding of how LLMs effectively address graph-related challenges. We identify and discuss four distinct architectural approaches: i) GNNs as Prefix, ii) LLMs as Prefix, iii) LLMs-Graphs Integration, and iv) LLMs-Only, each illustrated with representative examples. In summary, the contributions of our work can be summarized as:&#10;\begin{itemize}[leftmargin=*]&#10; \item Comprehensive Review of LLMs for Graph Learning. We offer a comprehensive review of the current state-of-the-art Large Language Models (LLMs) for graph learning, elucidating their strengths and pinpointing their limitations.&#10; \item Novel Taxonomy for Categorizing Research. We introduce a novel taxonomy for categorizing existing research based on their framework design, which provides a deeper insight into how LLMs can be seamlessly integrated with graph learning.&#10; \item Future Research Avenues. We also explore potential avenues for future research, including addressing the prevalent challenges in merging LLMs with graph learning methods and venturing into novel application areas.&#10;\end{itemize}&#10; \tikzstyle{redleaf}=[draw=edgered,&#10; rounded corners,minimum height=1em,&#10; fill=contentred!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;\tikzstyle{greenleaf}=[draw=edgegreen,&#10; rounded corners,minimum height=1em,&#10; fill=contentgreen!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;\tikzstyle{blueleaf}=[draw=edgeblue,&#10; rounded corners,minimum height=1em,&#10; fill=contentblue!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;\tikzstyle{yellowleaf}=[draw=edgeyellow,&#10; rounded corners,minimum height=1em,&#10; fill=contentyellow!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;\tikzstyle{blackmiddle}=[draw=black,&#10; rounded corners,minimum height=1em,&#10; fill=output-white!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;\tikzstyle{redmiddle}=[draw=edgered,&#10; rounded corners,minimum height=1em,&#10; fill=output-white!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;\tikzstyle{greenmiddle}=[draw=edgegreen,&#10; rounded corners,minimum height=1em,&#10; fill=output-white!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;\tikzstyle{bluemiddle}=[draw=edgeblue,&#10; rounded corners,minimum height=1em,&#10; fill=output-white!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;\tikzstyle{yellowmiddle}=[draw=edgeyellow,&#10; rounded corners,minimum height=1em,&#10; fill=output-white!40,text opacity=1, align=center,&#10; fill opacity=.5, text=black,align=left,font=,&#10; inner xsep=3pt,&#10; inner ysep=1pt,&#10;]&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Data Science, Graph Learning Techniques, Artificial Intelligence, Mathematics, Graph Neural Networks, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1901.00596" label="1901.00596">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey on Graph Neural Networks" />
          <attvalue for="1" value="  Deep learning has revolutionized many machine learning tasks in recent years,&#10;ranging from image classification and video processing to speech recognition&#10;and natural language understanding. The data in these tasks are typically&#10;represented in the Euclidean space. However, there is an increasing number of&#10;applications where data are generated from non-Euclidean domains and are&#10;represented as graphs with complex relationships and interdependency between&#10;objects. The complexity of graph data has imposed significant challenges on&#10;existing machine learning algorithms. Recently, many studies on extending deep&#10;learning approaches for graph data have emerged. In this survey, we provide a&#10;comprehensive overview of graph neural networks (GNNs) in data mining and&#10;machine learning fields. We propose a new taxonomy to divide the&#10;state-of-the-art graph neural networks into four categories, namely recurrent&#10;graph neural networks, convolutional graph neural networks, graph autoencoders,&#10;and spatial-temporal graph neural networks. We further discuss the applications&#10;of graph neural networks across various domains and summarize the open source&#10;codes, benchmark data sets, and model evaluation of graph neural networks.&#10;Finally, we propose potential research directions in this rapidly growing&#10;field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.00826" label="1810.00826">
        <attvalues>
          <attvalue for="0" value="How Powerful are Graph Neural Networks?" />
          <attvalue for="1" value="  Graph Neural Networks (GNNs) are an effective framework for representation&#10;learning of graphs. GNNs follow a neighborhood aggregation scheme, where the&#10;representation vector of a node is computed by recursively aggregating and&#10;transforming representation vectors of its neighboring nodes. Many GNN variants&#10;have been proposed and have achieved state-of-the-art results on both node and&#10;graph classification tasks. However, despite GNNs revolutionizing graph&#10;representation learning, there is limited understanding of their&#10;representational properties and limitations. Here, we present a theoretical&#10;framework for analyzing the expressive power of GNNs to capture different graph&#10;structures. Our results characterize the discriminative power of popular GNN&#10;variants, such as Graph Convolutional Networks and GraphSAGE, and show that&#10;they cannot learn to distinguish certain simple graph structures. We then&#10;develop a simple architecture that is provably the most expressive among the&#10;class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism&#10;test. We empirically validate our theoretical findings on a number of graph&#10;classification benchmarks, and demonstrate that our model achieves&#10;state-of-the-art performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.09691" label="1802.09691">
        <attvalues>
          <attvalue for="0" value="Link Prediction Based on Graph Neural Networks" />
          <attvalue for="1" value="  Link prediction is a key problem for network-structured data. Link prediction&#10;heuristics use some score functions, such as common neighbors and Katz index,&#10;to measure the likelihood of links. They have obtained wide practical uses due&#10;to their simplicity, interpretability, and for some of them, scalability.&#10;However, every heuristic has a strong assumption on when two nodes are likely&#10;to link, which limits their effectiveness on networks where these assumptions&#10;fail. In this regard, a more reasonable way should be learning a suitable&#10;heuristic from a given network instead of using predefined ones. By extracting&#10;a local subgraph around each target link, we aim to learn a function mapping&#10;the subgraph patterns to link existence, thus automatically learning a&#10;`heuristic' that suits the current network. In this paper, we study this&#10;heuristic learning paradigm for link prediction. First, we develop a novel&#10;$\gamma$-decaying heuristic theory. The theory unifies a wide range of&#10;heuristics in a single framework, and proves that all these heuristics can be&#10;well approximated from local subgraphs. Our results show that local subgraphs&#10;reserve rich information related to link existence. Second, based on the&#10;$\gamma$-decaying theory, we propose a new algorithm to learn heuristics from&#10;local subgraphs using a graph neural network (GNN). Its experimental results&#10;show unprecedented performance, working consistently well on a wide range of&#10;problems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.06455" label="1911.06455">
        <attvalues>
          <attvalue for="0" value="Graph Transformer Networks" />
          <attvalue for="1" value="  Graph neural networks (GNNs) have been widely used in representation learning&#10;on graphs and achieved state-of-the-art performance in tasks such as node&#10;classification and link prediction. However, most existing GNNs are designed to&#10;learn node representations on the fixed and homogeneous graphs. The limitations&#10;especially become problematic when learning representations on a misspecified&#10;graph or a heterogeneous graph that consists of various types of nodes and&#10;edges. In this paper, we propose Graph Transformer Networks (GTNs) that are&#10;capable of generating new graph structures, which involve identifying useful&#10;connections between unconnected nodes on the original graph, while learning&#10;effective node representation on the new graphs in an end-to-end fashion. Graph&#10;Transformer layer, a core layer of GTNs, learns a soft selection of edge types&#10;and composite relations for generating useful multi-hop connections so-called&#10;meta-paths. Our experiments show that GTNs learn new graph structures, based on&#10;data and tasks without domain knowledge, and yield powerful node representation&#10;via convolution on the new graphs. Without domain-specific graph preprocessing,&#10;GTNs achieved the best performance in all three benchmark node classification&#10;tasks against the state-of-the-art methods that require pre-defined meta-paths&#10;from domain knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.08385" label="2306.08385">
        <attvalues>
          <attvalue for="0" value="NodeFormer: A Scalable Graph Structure Learning Transformer for Node&#10;  Classification" />
          <attvalue for="1" value="  Graph neural networks have been extensively studied for learning with&#10;inter-connected data. Despite this, recent evidence has revealed GNNs'&#10;deficiencies related to over-squashing, heterophily, handling long-range&#10;dependencies, edge incompleteness and particularly, the absence of graphs&#10;altogether. While a plausible solution is to learn new adaptive topology for&#10;message passing, issues concerning quadratic complexity hinder simultaneous&#10;guarantees for scalability and precision in large networks. In this paper, we&#10;introduce a novel all-pair message passing scheme for efficiently propagating&#10;node signals between arbitrary nodes, as an important building block for a&#10;pioneering Transformer-style network for node classification on large graphs,&#10;dubbed as \textsc{NodeFormer}. Specifically, the efficient computation is&#10;enabled by a kernerlized Gumbel-Softmax operator that reduces the algorithmic&#10;complexity to linearity w.r.t. node numbers for learning latent graph&#10;structures from large, potentially fully-connected graphs in a differentiable&#10;manner. We also provide accompanying theory as justification for our design.&#10;Extensive experiments demonstrate the promising efficacy of the method in&#10;various tasks including node classification on graphs (with up to 2M nodes) and&#10;graph-enhanced applications (e.g., image classification) where input graphs are&#10;missing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.09474" label="2301.09474">
        <attvalues>
          <attvalue for="0" value="DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained&#10;  Diffusion" />
          <attvalue for="1" value="  Real-world data generation often involves complex inter-dependencies among&#10;instances, violating the IID-data hypothesis of standard learning paradigms and&#10;posing a challenge for uncovering the geometric structures for learning desired&#10;instance representations. To this end, we introduce an energy constrained&#10;diffusion model which encodes a batch of instances from a dataset into&#10;evolutionary states that progressively incorporate other instances' information&#10;by their interactions. The diffusion process is constrained by descent criteria&#10;w.r.t.~a principled energy function that characterizes the global consistency&#10;of instance representations over latent structures. We provide rigorous theory&#10;that implies closed-form optimal estimates for the pairwise diffusion strength&#10;among arbitrary instance pairs, which gives rise to a new class of neural&#10;encoders, dubbed as DIFFormer (diffusion-based Transformers), with two&#10;instantiations: a simple version with linear complexity for prohibitive&#10;instance numbers, and an advanced version for learning complex structures.&#10;Experiments highlight the wide applicability of our model as a general-purpose&#10;encoder backbone with superior performance in various tasks, such as node&#10;classification on large graphs, semi-supervised image/text classification, and&#10;spatial-temporal dynamics prediction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.07594" label="2106.07594">
        <attvalues>
          <attvalue for="0" value="Graph Contrastive Learning Automated" />
          <attvalue for="1" value="  Self-supervised learning on graph-structured data has drawn recent interest&#10;for learning generalizable, transferable and robust representations from&#10;unlabeled graphs. Among many, graph contrastive learning (GraphCL) has emerged&#10;with promising representation learning performance. Unfortunately, unlike its&#10;counterpart on image data, the effectiveness of GraphCL hinges on ad-hoc data&#10;augmentations, which have to be manually picked per dataset, by either rules of&#10;thumb or trial-and-errors, owing to the diverse nature of graph data. That&#10;significantly limits the more general applicability of GraphCL. Aiming to fill&#10;in this crucial gap, this paper proposes a unified bi-level optimization&#10;framework to automatically, adaptively and dynamically select data&#10;augmentations when performing GraphCL on specific graph data. The general&#10;framework, dubbed JOint Augmentation Optimization (JOAO), is instantiated as&#10;min-max optimization. The selections of augmentations made by JOAO are shown to&#10;be in general aligned with previous &quot;best practices&quot; observed from handcrafted&#10;tuning: yet now being automated, more flexible and versatile. Moreover, we&#10;propose a new augmentation-aware projection head mechanism, which will route&#10;output features through different projection heads corresponding to different&#10;augmentations chosen at each training step. Extensive experiments demonstrate&#10;that JOAO performs on par with or sometimes better than the state-of-the-art&#10;competitors including GraphCL, on multiple graph datasets of various scales and&#10;types, yet without resorting to any laborious dataset-specific tuning on&#10;augmentation selection. We release the code at&#10;https://github.com/Shen-Lab/GraphCL_Automated.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.06157" label="2002.06157">
        <attvalues>
          <attvalue for="0" value="Generalization and Representational Limits of Graph Neural Networks" />
          <attvalue for="1" value="  We address two fundamental questions about graph neural networks (GNNs).&#10;First, we prove that several important graph properties cannot be computed by&#10;GNNs that rely entirely on local information. Such GNNs include the standard&#10;message passing models, and more powerful spatial variants that exploit local&#10;graph structure (e.g., via relative orientation of messages, or local port&#10;ordering) to distinguish neighbors of each node. Our treatment includes a novel&#10;graph-theoretic formalism. Second, we provide the first data dependent&#10;generalization bounds for message passing GNNs. This analysis explicitly&#10;accounts for the local permutation invariance of GNNs. Our bounds are much&#10;tighter than existing VC-dimension based guarantees for GNNs, and are&#10;comparable to Rademacher bounds for recurrent neural networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.01343" label="2311.01343">
        <attvalues>
          <attvalue for="0" value="Collaborative Large Language Model for Recommender Systems" />
          <attvalue for="1" value="  Recently, there has been growing interest in developing the next-generation&#10;recommender systems (RSs) based on pretrained large language models (LLMs).&#10;However, the semantic gap between natural language and recommendation tasks is&#10;still not well addressed, leading to multiple issues such as spuriously&#10;correlated user/item descriptors, ineffective language modeling on user/item&#10;data, inefficient recommendations via auto-regression, etc. In this paper, we&#10;propose CLLM4Rec, the first generative RS that tightly integrates the LLM&#10;paradigm and ID paradigm of RSs, aiming to address the above challenges&#10;simultaneously. We first extend the vocabulary of pretrained LLMs with&#10;user/item ID tokens to faithfully model user/item collaborative and content&#10;semantics. Accordingly, a novel soft+hard prompting strategy is proposed to&#10;effectively learn user/item collaborative/content token embeddings via language&#10;modeling on RS-specific corpora, where each document is split into a prompt&#10;consisting of heterogeneous soft (user/item) tokens and hard (vocab) tokens and&#10;a main text consisting of homogeneous item tokens or vocab tokens to facilitate&#10;stable and effective language modeling. In addition, a novel mutual&#10;regularization strategy is introduced to encourage CLLM4Rec to capture&#10;recommendation-related information from noisy user/item content. Finally, we&#10;propose a novel recommendation-oriented finetuning strategy for CLLM4Rec, where&#10;an item prediction head with multinomial likelihood is added to the pretrained&#10;CLLM4Rec backbone to predict hold-out items based on soft+hard prompts&#10;established from masked user-item interaction history, where recommendations of&#10;multiple items can be generated efficiently without hallucination. Codes are&#10;released at https://github.com/yaochenzhu/llm4rec.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10037" label="2305.10037">
        <attvalues>
          <attvalue for="0" value="Can Language Models Solve Graph Problems in Natural Language?" />
          <attvalue for="1" value="  Large language models (LLMs) are increasingly adopted for a variety of tasks&#10;with implicit graphical structures, such as planning in robotics, multi-hop&#10;question answering or knowledge probing, structured commonsense reasoning, and&#10;more. While LLMs have advanced the state-of-the-art on these tasks with&#10;structure implications, whether LLMs could explicitly process textual&#10;descriptions of graphs and structures, map them to grounded conceptual spaces,&#10;and perform structured operations remains underexplored. To this end, we&#10;propose NLGraph (Natural Language Graph), a comprehensive benchmark of&#10;graph-based problem solving designed in natural language. NLGraph contains&#10;29,370 problems, covering eight graph reasoning tasks with varying complexity&#10;from simple tasks such as connectivity and shortest path up to complex problems&#10;such as maximum flow and simulating graph neural networks. We evaluate LLMs&#10;(GPT-3/4) with various prompting approaches on the NLGraph benchmark and find&#10;that 1) language models do demonstrate preliminary graph reasoning abilities,&#10;2) the benefit of advanced prompting and in-context learning diminishes on more&#10;complex graph problems, while 3) LLMs are also (un)surprisingly brittle in the&#10;face of spurious correlations in graph and problem settings. We then propose&#10;Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based&#10;approaches to enhance LLMs in solving natural language graph problems.&#10;Build-a-Graph and Algorithmic prompting improve the performance of LLMs on&#10;NLGraph by 3.07% to 16.85% across multiple tasks and settings, while how to&#10;solve the most complicated graph reasoning tasks in our setup with language&#10;models remains an open research question. The NLGraph benchmark and evaluation&#10;code are available at https://github.com/Arthur-Heng/NLGraph.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.13023" label="2310.13023">
        <attvalues>
          <attvalue for="0" value="GraphGPT: Graph Instruction Tuning for Large Language Models" />
          <attvalue for="1" value="  Graph Neural Networks (GNNs) have evolved to understand graph structures&#10;through recursive exchanges and aggregations among nodes. To enhance&#10;robustness, self-supervised learning (SSL) has become a vital tool for data&#10;augmentation. Traditional methods often depend on fine-tuning with&#10;task-specific labels, limiting their effectiveness when labeled data is scarce.&#10;Our research tackles this by advancing graph model generalization in zero-shot&#10;learning environments. Inspired by the success of large language models (LLMs),&#10;we aim to create a graph-oriented LLM capable of exceptional generalization&#10;across various datasets and tasks without relying on downstream graph data. We&#10;introduce the GraphGPT framework, which integrates LLMs with graph structural&#10;knowledge through graph instruction tuning. This framework includes a&#10;text-graph grounding component to link textual and graph structures and a&#10;dual-stage instruction tuning approach with a lightweight graph-text alignment&#10;projector. These innovations allow LLMs to comprehend complex graph structures&#10;and enhance adaptability across diverse datasets and tasks. Our framework&#10;demonstrates superior generalization in both supervised and zero-shot graph&#10;learning tasks, surpassing existing benchmarks. The open-sourced model&#10;implementation of our GraphGPT is available at&#10;https://github.com/HKUDS/GraphGPT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05845" label="2310.05845">
        <attvalues>
          <attvalue for="0" value="GraphLLM: Boosting Graph Reasoning Ability of Large Language Model" />
          <attvalue for="1" value="  The advancement of Large Language Models (LLMs) has remarkably pushed the&#10;boundaries towards artificial general intelligence (AGI), with their&#10;exceptional ability on understanding diverse types of information, including&#10;but not limited to images and audio. Despite this progress, a critical gap&#10;remains in empowering LLMs to proficiently understand and reason on graph data.&#10;Recent studies underscore LLMs' underwhelming performance on fundamental graph&#10;reasoning tasks. In this paper, we endeavor to unearth the obstacles that&#10;impede LLMs in graph reasoning, pinpointing the common practice of converting&#10;graphs into natural language descriptions (Graph2Text) as a fundamental&#10;bottleneck. To overcome this impediment, we introduce GraphLLM, a pioneering&#10;end-to-end approach that synergistically integrates graph learning models with&#10;LLMs. This synergy equips LLMs with the ability to proficiently interpret and&#10;reason on graph data, harnessing the superior expressive power of graph&#10;learning models. Our empirical evaluations across four fundamental graph&#10;reasoning tasks validate the effectiveness of GraphLLM. The results exhibit a&#10;substantial average accuracy enhancement of 54.44%, alongside a noteworthy&#10;context reduction of 96.45% across various graph reasoning tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.02783" label="2312.02783">
        <attvalues>
          <attvalue for="0" value="Large Language Models on Graphs: A Comprehensive Survey" />
          <attvalue for="1" value="  Large language models (LLMs), such as GPT4 and LLaMA, are creating&#10;significant advancements in natural language processing, due to their strong&#10;text encoding/decoding ability and newly found emergent capability (e.g.,&#10;reasoning). While LLMs are mainly designed to process pure texts, there are&#10;many real-world scenarios where text data is associated with rich structure&#10;information in the form of graphs (e.g., academic networks, and e-commerce&#10;networks) or scenarios where graph data is paired with rich textual information&#10;(e.g., molecules with descriptions). Besides, although LLMs have shown their&#10;pure text-based reasoning ability, it is underexplored whether such ability can&#10;be generalized to graphs (i.e., graph-based reasoning). In this paper, we&#10;provide a systematic review of scenarios and techniques related to large&#10;language models on graphs. We first summarize potential scenarios of adopting&#10;LLMs on graphs into three categories, namely pure graphs, text-attributed&#10;graphs, and text-paired graphs. We then discuss detailed techniques for&#10;utilizing LLMs on graphs, including LLM as Predictor, LLM as Encoder, and LLM&#10;as Aligner, and compare the advantages and disadvantages of different schools&#10;of models. Furthermore, we discuss the real-world applications of such methods&#10;and summarize open-source codes and benchmark datasets. Finally, we conclude&#10;with potential future research directions in this fast-growing field. The&#10;related source can be found at&#10;https://github.com/PeterGriffinJin/Awesome-Language-Model-on-Graphs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.08302" label="2306.08302">
        <attvalues>
          <attvalue for="0" value="Unifying Large Language Models and Knowledge Graphs: A Roadmap" />
          <attvalue for="1" value="  Large language models (LLMs), such as ChatGPT and GPT4, are making new waves&#10;in the field of natural language processing and artificial intelligence, due to&#10;their emergent ability and generalizability. However, LLMs are black-box&#10;models, which often fall short of capturing and accessing factual knowledge. In&#10;contrast, Knowledge Graphs (KGs), Wikipedia and Huapu for example, are&#10;structured knowledge models that explicitly store rich factual knowledge. KGs&#10;can enhance LLMs by providing external knowledge for inference and&#10;interpretability. Meanwhile, KGs are difficult to construct and evolving by&#10;nature, which challenges the existing methods in KGs to generate new facts and&#10;represent unseen knowledge. Therefore, it is complementary to unify LLMs and&#10;KGs together and simultaneously leverage their advantages. In this article, we&#10;present a forward-looking roadmap for the unification of LLMs and KGs. Our&#10;roadmap consists of three general frameworks, namely, 1) KG-enhanced LLMs,&#10;which incorporate KGs during the pre-training and inference phases of LLMs, or&#10;for the purpose of enhancing understanding of the knowledge learned by LLMs; 2)&#10;LLM-augmented KGs, that leverage LLMs for different KG tasks such as embedding,&#10;completion, construction, graph-to-text generation, and question answering; and&#10;3) Synergized LLMs + KGs, in which LLMs and KGs play equal roles and work in a&#10;mutually beneficial way to enhance both LLMs and KGs for bidirectional&#10;reasoning driven by both data and knowledge. We review and summarize existing&#10;efforts within these three frameworks in our roadmap and pinpoint their future&#10;research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.14782" label="2110.14782">
        <attvalues>
          <attvalue for="0" value="When is BERT Multilingual? Isolating Crucial Ingredients for&#10;  Cross-lingual Transfer" />
          <attvalue for="1" value="  While recent work on multilingual language models has demonstrated their&#10;capacity for cross-lingual zero-shot transfer on downstream tasks, there is a&#10;lack of consensus in the community as to what shared properties between&#10;languages enable such transfer. Analyses involving pairs of natural languages&#10;are often inconclusive and contradictory since languages simultaneously differ&#10;in many linguistic aspects. In this paper, we perform a large-scale empirical&#10;study to isolate the effects of various linguistic properties by measuring&#10;zero-shot transfer between four diverse natural languages and their&#10;counterparts constructed by modifying aspects such as the script, word order,&#10;and syntax. Among other things, our experiments show that the absence of&#10;sub-word overlap significantly affects zero-shot transfer when languages differ&#10;in their word order, and there is a strong correlation between transfer&#10;performance and word embedding alignment between languages (e.g., R=0.94 on the&#10;task of NLI). Our results call for focus in multilingual models on explicitly&#10;improving word embedding alignment between languages rather than relying on its&#10;implicit emergence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Multilingual pre-training for Transformers&#10;The success of monolingual Transformer language models~\cite{devlin2019bert,radfordimproving} has driven studies that learn a multilingual language-model (LM) on several languages.&#10;Multilingual-BERT (M-BERT)~\cite{multilingualbert} is a single neural network pre-trained using the masked language-modeling (MLM) objective on a corpus of text from $104$ languages.&#10;XLM~\cite{DBLP:conf/nips/ConneauL19} introduced translation language-modeling, which performs MLM on pairs of parallel sentences, thus encouraging alignment between their representations.&#10;These models exhibit surprising zero-shot cross-lingual transfer performance~\cite{DBLP:conf/nips/ConneauL19,ketal}, a setup where the model is fine-tuned on a source language and evaluated on a different target language.&#10;&#10;Analysis of cross-lingual transfer&#10;While \cite{pires2019multilingual},~\cite{conneau2020emerging}, and~\cite{ketal} showed that transfer works even without a shared vocabulary between languages, \cite{wu2019beto} discovered a correlation between sub-word overlap and zero-shot performance.&#10;\cite{conneau2020emerging} and \cite{artetxe2019cross} showed that shared parameters for languages with different scripts were crucial for transfer.&#10;\cite{pires2019multilingual} and~\cite{wu2019beto} observed that transfer for NER and POS tagging works better between typologically similar languages. However, a study conducted by~\cite{lin2019choosing} showed that there is no simple rule of thumb to gauge when transfer works between languages.&#10;\cite{hsu2019zero} observed that changing the syntax (SOV) order of the source to match that of the target does not improve performance.&#10;&#10;Transfer between real and synthetic Languages&#10;&#10;\cite{ketal} create a synthetic language by changing English's script and find that transfer between it and Spanish works even without common sub-words.&#10;However, they use only English as their source language, test only on two tasks, and use a single natural-synthetic language pair.&#10;\cite{dufter2020identifying} study transfer between English and synthetic English obtained by changing the script, word order, or model delimiters.&#10;However, they use a small corpus ($228$K words) compared to current standards (we use $3$ orders more) and measure only embedding similarity and not zero-shot transfer.&#10;A contemporary work~\cite{wu2022oolong} uses synthetic transformations to modify the GLUE dataset~\cite{wang2018glue} and analyze properties required for good zero-shot transfer, but they perform their experiments only on English and do not perform token embedding alignment analysis.&#10;We show that the latter is crucial for good transfer.&#10; " />
          <attvalue for="4" value="Cross-Lingual Transfer Learning, Computer Science, Linguistics, Cognitive Science, Multilingual Language Models, Artificial Intelligence, Natural Language Processing, Linguistic Property Analysis" />
        </attvalues>
      </node>
      <node id="2408.02103" label="2408.02103">
        <attvalues>
          <attvalue for="0" value="Effective Demonstration Annotation for In-Context Learning via Language&#10;  Model-Based Determinantal Point Process" />
          <attvalue for="1" value="  In-context learning (ICL) is a few-shot learning paradigm that involves&#10;learning mappings through input-output pairs and appropriately applying them to&#10;new instances. Despite the remarkable ICL capabilities demonstrated by Large&#10;Language Models (LLMs), existing works are highly dependent on large-scale&#10;labeled support sets, not always feasible in practical scenarios. To refine&#10;this approach, we focus primarily on an innovative selective annotation&#10;mechanism, which precedes the standard demonstration retrieval. We introduce&#10;the Language Model-based Determinant Point Process (LM-DPP) that simultaneously&#10;considers the uncertainty and diversity of unlabeled instances for optimal&#10;selection. Consequently, this yields a subset for annotation that strikes a&#10;trade-off between the two factors. We apply LM-DPP to various language models,&#10;including GPT-J, LlaMA, and GPT-3. Experimental results on 9 NLU and 2&#10;Generation datasets demonstrate that LM-DPP can effectively select canonical&#10;examples. Further analysis reveals that LLMs benefit most significantly from&#10;subsets that are both low uncertainty and high diversity.&#10;" />
          <attvalue for="2" value="&#10;As large pre-trained language models (LLMs) \cite{brown2020language, chowdhery2022palm, zhang2022opt, tay2023ul2, touvron2023llama, workshop2023bloom} grow in scale, &#10;they not only exhibit enhanced linguistic capabilities and expanded world knowledge but also demonstrate a novel ability for in-context learning. &#10;Specifically, LLMs have shown proficiency in learning from a limited set of input-output examples (known as demonstrations \cite{brown2020language}), and effectively applying these learned mappings to new, unseen instances.&#10;This novel few-shot learning paradigm, which avoids parameter updates, has become a popular and efficient method for utilizing LLMs \cite{liu2021pretrain, dong2023survey, liu2021makes}.&#10;&#10;Previous studies have investigated which instances can serve as effective prompts for ICL \cite{liu2021makes, zhang2022active, li2023finding}. &#10;They have demonstrated that retrieving specific similar contexts for individual test queries can significantly improve performance (instance level) and ground truth matters for support examples.&#10;To assign appropriate demonstrations to all test queries, support sets necessitate diversity and broad coverage, usually achieved through large labeled data, following the principle that Monte Carlo estimation accuracy improves with larger samples.&#10;Nonetheless, these extensive datasets are often impractical to obtain.&#10;&#10;We investigate the selection of demonstrations from the perspective of Active Learning (AL) \cite{cohn1996active, settles2009active}.&#10;Based on the core principle that not all data points are of equal value, AL aims to identify the most effective instances in an unlabeled data pool for annotation.&#10;\cite{margatina2023active} elucidates that high semantic similarity, low uncertainty, and high diversity comprise an effective and efficient annotation strategy. Similarly, \cite{gonen2022demystifying} demonstrates that lower prompt perplexity is closely associated with better performance.&#10;While \cite{su2022selective}'s Vote-k framework adopts a data-centric perspective (i.e., selecting examples that balance diversity and representativeness), it neglects the assessment of uncertainty and the inter-relationship among context examples.&#10;In this paper, we pursue a more universally applicable yet straightforward solution, incorporating confidence signals of LLMs to select annotation instances that are maximally diverse and exhibit low uncertainty.&#10;&#10;To address this need, we introduce a generic approach, LM-DPP, which jointly models uncertainty and diversity within the support set through a conditional Determinantal Point Process.&#10;Specifically, we employ LLMs' perplexity to score each candidate instance in the support set, &#10;which serves as a measure of the LLMs' uncertainty.&#10;Then a Gram matrix is constructed to balance the uncertainty and diversity of candidate instances and polynomial-time maximum a posteriori (MAP) inference \cite{chen2018fast} is applied to identify the most useful subset of instances to be annotated.&#10;From the perspective of selective annotation, we consider extremely low-resource ICL scenarios as those in which the available annotated examples are limited to a few dozen instances. Our focus centers on identifying which specific set of demonstrations can most effectively harness the capabilities of LLMs within this challenging context.&#10;&#10;We validate our method through extensive experiments on 9 NLU and 2 Generation datasets.&#10;We also demonstrate the versatility of LM-DPP by adapting it to the large language model GPT-3 (175B).&#10;The experimental results illustrate that our approach can effectively balance two critical factors, uncertainty and diversity.&#10;In summary, our contributions are as follows.&#10;\begin{itemize}&#10; \item We revisit the setup of ICL from the perspective of selective annotation. We introduce a novel approach, LM-DPP, to select instances that balance uncertainty and diversity for annotation, aiming to reduce the human engineering workload.&#10; \item The experimental results indicate that the proposed method outperforms the previous best-performing selection methods by a large relative improvement and exhibits commendable generalizability across model size (\S \ref{sec:llm_scaling}) and annotation budget (\S \ref{sec:size_scaling}) scaling.&#10; \item Comprehensive analysis confirms that LLMs can benefit from a demonstration set that exhibits both low uncertainty and diversity (\S\ref{sec:trade-off}) and gold annotation matters for ICL performance (\S\ref{sec:ground_truth}).&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Computer Science, Machine Learning, Linguistics, Language Models, Artificial Intelligence, Mathematics, Natural Language Processing, Selective Annotation" />
        </attvalues>
      </node>
      <node id="cs/9603104" label="cs/9603104">
        <attvalues>
          <attvalue for="0" value="Active Learning with Statistical Models" />
          <attvalue for="1" value="  For many types of machine learning algorithms, one can compute the&#10;statistically `optimal' way to select training data. In this paper, we review&#10;how optimal data selection techniques have been used with feedforward neural&#10;networks. We then show how the same principles may be used to select data for&#10;two alternative, statistically-based learning architectures: mixtures of&#10;Gaussians and locally weighted regression. While the techniques for neural&#10;networks are computationally expensive and approximate, the techniques for&#10;mixtures of Gaussians and locally weighted regression are both efficient and&#10;accurate. Empirically, we observe that the optimality criterion sharply&#10;decreases the number of training examples the learner needs in order to achieve&#10;good performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14264" label="2305.14264">
        <attvalues>
          <attvalue for="0" value="Active Learning Principles for In-Context Learning with Large Language&#10;  Models" />
          <attvalue for="1" value="  The remarkable advancements in large language models (LLMs) have&#10;significantly enhanced the performance in few-shot learning settings. By using&#10;only a small number of labeled examples, referred to as demonstrations, LLMs&#10;can effectively grasp the task at hand through in-context learning. However,&#10;the process of selecting appropriate demonstrations has received limited&#10;attention in prior work. This paper addresses the issue of identifying the most&#10;informative demonstrations for few-shot learning by approaching it as a&#10;pool-based Active Learning (AL) problem over a single iteration. Our objective&#10;is to investigate how AL algorithms can serve as effective demonstration&#10;selection methods for in-context learning. We compare various standard AL&#10;algorithms based on uncertainty, diversity, and similarity, and consistently&#10;observe that the latter outperforms all other methods, including random&#10;sampling. Notably, uncertainty sampling, despite its success in conventional&#10;supervised learning scenarios, performs poorly in this context. Our extensive&#10;experimentation involving a diverse range of GPT and OPT models across $24$&#10;classification and multi-choice tasks, coupled with thorough analysis,&#10;unambiguously demonstrates that in-context example selection through AL&#10;prioritizes high-quality examples that exhibit low uncertainty and bear&#10;similarity to the test examples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.05008" label="2203.05008">
        <attvalues>
          <attvalue for="0" value="Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word&#10;  Speech Recognition" />
          <attvalue for="1" value="  Language model fusion helps smart assistants recognize words which are rare&#10;in acoustic data but abundant in text-only corpora (typed search logs).&#10;However, such corpora have properties that hinder downstream performance,&#10;including being (1) too large, (2) beset with domain-mismatched content, and&#10;(3) heavy-headed rather than heavy-tailed (excessively many duplicate search&#10;queries such as &quot;weather&quot;). We show that three simple strategies for selecting&#10;language modeling data can dramatically improve rare-word recognition without&#10;harming overall performance. First, to address the heavy-headedness, we&#10;downsample the data according to a soft log function, which tunably reduces&#10;high frequency (head) sentences. Second, to encourage rare-word exposure, we&#10;explicitly filter for words rare in the acoustic data. Finally, we tackle&#10;domain-mismatch via perplexity-based contrastive selection, filtering for&#10;examples matched to the target domain. We down-select a large corpus of web&#10;search queries by a factor of 53x and achieve better LM perplexities than&#10;without down-selection. When shallow-fused with a state-of-the-art, production&#10;speech engine, our LM achieves WER reductions of up to 24% relative on&#10;rare-word sentences (without changing overall WER) compared to a baseline LM&#10;trained on the raw corpus. These gains are further validated through favorable&#10;side-by-side evaluations on live voice search traffic.&#10;" />
          <attvalue for="2" value="&#10;A common and frustrating failure mode of neural end-to-end ASR models is the misrecognition of words rarely seen during training. &#10;In order to improve rare-word performance, recent works have looked to leveraging text-only corpora as an additional source of rare-word data \cite{peyser2020improving,raju2019scalable,huang2021lookup}.&#10;A common strategy is to incorporate these text corpora into the training of a language model which is then interpolated with the E2E model during decoding. &#10;&#10;Text data for training the language model is often scraped from typed search logs to achieve the best domain match to smart assistant voice queries. &#10;These logs can be very large \cite{raju2019scalable,meng2021minimum}, making it prohibitively expensive to make even a single epoch through the data, limiting rare-word exposure.&#10;Furthermore, search queries can be heavy-headed, meaning that they contain disproportionately many high-frequency queries relative to low-frequency queries, also limiting rare-word learning.&#10;Finally, there is a domain mismatch between typed queries and voice queries which hurts target domain (voice) performance. &#10;For example, typed queries contain more website names while voice queries contain more voice commands.&#10;&#10;In this work, we propose three simple data selection strategies applied together (Fig. \ref{fig:schem}) to reduce the size of the corpus and improve recognition quality on rare words, without hurting overall performance. &#10;First, we show that n\&quot;aive deduplication \cite{lee2021deduplicating} does not improve performance because it is too aggressive.&#10;Instead we find beneficial a gradual downsampling of high-frequency sentences using a soft log function which maintains the natural distribution of the corpus while mitigating its heavy-headedness.&#10;Second, we explicitly filter the dataset for sentences containing rare words in order to boost their representation during training.&#10;Third, we use perplexity-based contrastive data selection to filter only for queries which most resemble the target domain of voice search.&#10;&#10;Our data selection is run on a slice of anonymized Google search traffic. We obtain a subset that is orders of magnitude smaller while improving results on both the head and tail of the data distribution.&#10;With shallow fusion, the data-selected LM yields up to 24\% WER relative reduction on rare words compared to an LM trained on the raw corpus, while leaving the overall WER unchanged. &#10;Our work shows that simple data selection methods can significantly improve rare-word learning without the need for architecture or training configuration changes.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Rare Word Recognition, Speech Engine Improvement, Speech Recognition, Information Science, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2008.10491" label="2008.10491">
        <attvalues>
          <attvalue for="0" value="Improving Tail Performance of a Deliberation E2E ASR Model Using a Large&#10;  Text Corpus" />
          <attvalue for="1" value="  End-to-end (E2E) automatic speech recognition (ASR) systems lack the distinct&#10;language model (LM) component that characterizes traditional speech systems.&#10;While this simplifies the model architecture, it complicates the task of&#10;incorporating text-only data into training, which is important to the&#10;recognition of tail words that do not occur often in audio-text pairs. While&#10;shallow fusion has been proposed as a method for incorporating a pre-trained LM&#10;into an E2E model at inference time, it has not yet been explored for very&#10;large text corpora, and it has been shown to be very sensitive to&#10;hyperparameter settings in the beam search. In this work, we apply shallow&#10;fusion to incorporate a very large text corpus into a state-of-the-art E2EASR&#10;model. We explore the impact of model size and show that intelligent pruning of&#10;the training set can be more effective than increasing the parameter count.&#10;Additionally, we show that incorporating the LM in minimum word error rate&#10;(MWER) fine tuning makes shallow fusion far less dependent on optimal&#10;hyperparameter settings, reducing the difficulty of that tuning problem.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.01677" label="1907.01677">
        <attvalues>
          <attvalue for="0" value="Scalable Multi Corpora Neural Language Models for ASR" />
          <attvalue for="1" value="  Neural language models (NLM) have been shown to outperform conventional&#10;n-gram language models by a substantial margin in Automatic Speech Recognition&#10;(ASR) and other tasks. There are, however, a number of challenges that need to&#10;be addressed for an NLM to be used in a practical large-scale ASR system. In&#10;this paper, we present solutions to some of the challenges, including training&#10;NLM from heterogenous corpora, limiting latency impact and handling&#10;personalized bias in the second-pass rescorer. Overall, we show that we can&#10;achieve a 6.2% relative WER reduction using neural LM in a second-pass n-best&#10;rescoring framework with a minimal increase in latency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.02302" label="2106.02302">
        <attvalues>
          <attvalue for="0" value="Minimum Word Error Rate Training with Language Model Fusion for&#10;  End-to-End Speech Recognition" />
          <attvalue for="1" value="  Integrating external language models (LMs) into end-to-end (E2E) models&#10;remains a challenging task for domain-adaptive speech recognition. Recently,&#10;internal language model estimation (ILME)-based LM fusion has shown significant&#10;word error rate (WER) reduction from Shallow Fusion by subtracting a weighted&#10;internal LM score from an interpolation of E2E model and external LM scores&#10;during beam search. However, on different test sets, the optimal LM&#10;interpolation weights vary over a wide range and have to be tuned extensively&#10;on well-matched validation sets. In this work, we perform LM fusion in the&#10;minimum WER (MWER) training of an E2E model to obviate the need for LM weights&#10;tuning during inference. Besides MWER training with Shallow Fusion (MWER-SF),&#10;we propose a novel MWER training with ILME (MWER-ILME) where the ILME-based&#10;fusion is conducted to generate N-best hypotheses and their posteriors.&#10;Additional gradient is induced when internal LM is engaged in MWER-ILME loss&#10;computation. During inference, LM weights pre-determined in MWER training&#10;enable robust LM integrations on test sets from different domains. Experimented&#10;with 30K-hour trained transformer transducers, MWER-ILME achieves on average&#10;8.8% and 5.8% relative WER reductions from MWER and MWER-SF training,&#10;respectively, on 6 different test sets&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.06499" label="2107.06499">
        <attvalues>
          <attvalue for="0" value="Deduplicating Training Data Makes Language Models Better" />
          <attvalue for="1" value="  We find that existing language modeling datasets contain many near-duplicate&#10;examples and long repetitive substrings. As a result, over 1% of the unprompted&#10;output of language models trained on these datasets is copied verbatim from the&#10;training data. We develop two tools that allow us to deduplicate training&#10;datasets -- for example removing from C4 a single 61 word English sentence that&#10;is repeated over 60,000 times. Deduplication allows us to train models that&#10;emit memorized text ten times less frequently and require fewer train steps to&#10;achieve the same or better accuracy. We can also reduce train-test overlap,&#10;which affects over 4% of the validation set of standard datasets, thus allowing&#10;for more accurate evaluation. We release code for reproducing our work and&#10;performing dataset deduplication at&#10;https://github.com/google-research/deduplicate-text-datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Large language model datasets.&#10;While we believe our results are independent of model architecture,&#10;we perform our analysis on Transformer-based decoder-only language models \cite{vaswani2017attention} trained for open-ended text generation.&#10;These current state-of-the-art models are trained on internet text.&#10;For example, the GPT-2 family of models \cite{radford2019language} is trained on WebText, a dataset of web documents highly ranked on Reddit---however this dataset was not made available publicly.&#10;A common dataset starting point is CommonCrawl, an index of public webpages.&#10;Among the models trained on CommonCrawl include&#10;GPT-3 \cite{brown2020language} with the addition of book datasets,&#10;GROVER \cite{zellers2019defending} on a restricted subset filtered to news domains called RealNews,&#10;and T5 \cite{t52020} on a cleaned version of common crawl called C4.&#10;Other models are trained on more curated Internet sources---for example \cite{guo2020wiki40b} used high quality processed Wikipedia text from 40 different languages to train monolingual 141.4M parameter language models.&#10;Non-English models necessarily use different datasets; \cite{zeng2021pangualpha} for instance introduced PANGU-$\alpha$, a family of models with up to 200B parameters that were trained on a non-public corpus of cleaned and filtered Chinese-language documents from CommonCrawl and other sources.&#10;Since many of these datasets are not public,&#10;we deduplicate three that are: Wiki-40B, C4, and RealNews--as well as the One Billion Word Language Model Benchmark \cite{chelba2013one}, &#10;a smaller&#10;dataset commonly used for evaluation.&#10;&#10;Contamination of downstream tasks.&#10;When models are trained on datasets constructed by crawling the Internet, it is possible the model will train on the test set of downstream target tasks.&#10;For example, \cite[\S4]{radford2019language} performed a post-hoc analysis to identify 8-gram overlaps between GPT-2's training set and datasets used for evaluation,&#10;and \cite{Dodge2021-lb} analyzed C4 and found that up to 14.4\% of test examples for various standard tasks were found verbatim (normalizing for capitalization and punctuation) in the dataset.&#10;A more proactive approach removes contaminated data.&#10;\cite[Appendix B]{trinh2018simple} removed documents from their CommonCrawl-based train set that overlapped substantially with the commonsense reasoning used for evaluation.&#10;And GPT-3 \cite[\S5]{brown2020language} did the reverse and removed downstream evaluation examples from their training data by conservatively filtering out any train set examples with a 13-gram overlap with any evaluation example.&#10;Up to $90\%$ of tasks were flagged as potentially contaminated.&#10;&#10;In our research, we do not focus on the impact of duplicate text in pretrained models on downstream benchmark tasks; instead we address how duplicate text in the LM training and validation sets impacts model perplexity and the extent to which generated text included memorized content.&#10;&#10;Memorizing training data. The privacy risks of data memorization, for example the ability to extract sensitive data such as valid phone numbers and IRC usernames, are highlighted by&#10;\cite{carlini2020extracting}.&#10;While their paper finds 604 samples that GPT-2 emitted from its training set, we show that over $1\%$ of the data most models emit is memorized training data.&#10;In computer vision, memorization of training data has been studied from various angles for both discriminative and generative models~\cite[e.g.][]{arpit2017closer,8953411,feldman2020neural,stephenson2021geometry,teterwak2021understanding}.&#10;&#10;Duplicate text in training data.&#10;The Book Corpus \cite{zhu2015aligning}, which was used to train popular models such as BERT, has a substantial amount of exact-duplicate documents according to \cite{bandy2021addressing}.&#10;\cite{allamanis2019adverse} shows that duplicate examples in code datasets cause worsened performance on code understanding tasks.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2210.03162" label="2210.03162">
        <attvalues>
          <attvalue for="0" value="Prompt Compression and Contrastive Conditioning for Controllability and&#10;  Toxicity Reduction in Language Models" />
          <attvalue for="1" value="  We explore the idea of compressing the prompts used to condition language&#10;models, and show that compressed prompts can retain a substantive amount of&#10;information about the original prompt. For severely compressed prompts, while&#10;fine-grained information is lost, abstract information and general sentiments&#10;can be retained with surprisingly few parameters, which can be useful in the&#10;context of decode-time algorithms for controllability and toxicity reduction.&#10;We explore contrastive conditioning to steer language model generation towards&#10;desirable text and away from undesirable text, and find that some complex&#10;prompts can be effectively compressed into a single token to guide generation.&#10;We also show that compressed prompts are largely compositional, and can be&#10;constructed such that they can be used to control independent aspects of&#10;generated text.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models (LMs), such as GPT-2 \cite{radford2018improving,radford2019language}, BERT \cite{devlin2018bert}, T5 \cite{2020t5}, or GPT-3 \cite{brown2020language}, exhibit a remarkable ability to capture patterns of grammar, vocabulary, cultural knowledge, and conversational rhythms present in natural language. Formally, a LM is a conditional distribution over tokens $p(x_t|x_1,\cdots,x_{t-1})$, with each token $x_t \in \mathcal{V}$ for some vocabulary $\mathcal{V}$. Throughout this paper, we will refer to $x_h=x_1,\cdots,x_{t-1}$ as the prompt.&#10;&#10;This paper explores prompt compression: the idea that the text $x_h$ used to condition a LM can be approximately represented by a much smaller set of carefully chosen weights, using the framework of soft prompts \cite{lestersoftprompt}. We begin by establishing some basic properties of compressed prompts, and importantly show that while highly compressed prompts lose fine-grained information about the prompt, they can retain general, abstract information. This motivates our central application: to use such compressed prompts in a Bayesian attribute framework to steer text generation, with specific application to toxicity reduction.&#10;&#10;To motivate this more deeply, we briefly sketch how compressed prompts can be used in toxicity reduction. Efforts to reduce toxicity and bias generally follow one of two strategies: the first is to train or fine-tune LMs on carefully curated data, either tagging or labelling it in special ways \cite{salesForceCTRL,lu2022quark} or using data known to be ``clean''. The second is to &quot;steer&quot; the generation of token probabilities away from toxic generations \cite{krause2020gedi,liu2021dexperts}, and towards text with known, desirable properties.&#10;&#10;Following previous work, we steer LM probabilities by using a Bayesian attribute classifier framework that involves scoring candidate tokens with different experts.&#10;As an independent contribution, we explore the idea of simply using conditioning text to construct such experts by leveraging the few-shot modeling abilities of LMs \cite{radford2019language,brown2020language}: given a few examples of text containing a pattern of interest, language models are capable of ``analyzing'' such examples and assign high probability to subsequent text exhibiting the same pattern. Thus, in the same way that language model can, for example, classify the sentiment of a tweet, we use LMs to analyze the toxicity of candidate generations in real-time. Our method can be considered an exemplar-based method of defining experts that capture desirable and undesirable attributes of generated text.&#10;We term this technique contrastive contexts, and note that it reduces the problem of creating experts to one of prompt engineering \cite{reynolds2021prompt}.&#10;&#10;However, our conditioning contexts are quite large, which motivated this work. We use prompt compression to mimic an uncompressed prompt (hereafter referred to as &quot;hard&quot; prompt) as closely as possible, thereby saving both computation and space in the context window. Our results demonstrate that this can be very effective, and, in a very surprising finding, that complex prompts can be reduced to a single token and still be useful for toxicity reduction, often with better fluency compared to hard prompts.&#10;&#10;The contributions of this paper are three-fold: first, we introduce and formalize the idea of prompt compression; second, we introduce and formalize the method of contrastive contexts in the Bayesian attribute framework; third, we experimentally evaluate our methods, and refine the technique based on various empirical observations, and contribute a careful study of effectiveness as model size varies.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Text Generation Control, Contrastive Conditioning Methods, Language Model Compression, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1909.05858" label="1909.05858">
        <attvalues>
          <attvalue for="0" value="CTRL: A Conditional Transformer Language Model for Controllable&#10;  Generation" />
          <attvalue for="1" value="  Large-scale language models show promising text generation capabilities, but&#10;users cannot easily control particular aspects of the generated text. We&#10;release CTRL, a 1.63 billion-parameter conditional transformer language model,&#10;trained to condition on control codes that govern style, content, and&#10;task-specific behavior. Control codes were derived from structure that&#10;naturally co-occurs with raw text, preserving the advantages of unsupervised&#10;learning while providing more explicit control over text generation. These&#10;codes also allow CTRL to predict which parts of the training data are most&#10;likely given a sequence. This provides a potential method for analyzing large&#10;amounts of data via model-based source attribution. We have released multiple&#10;full-sized, pretrained versions of CTRL at https://github.com/salesforce/ctrl.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.13636" label="2205.13636">
        <attvalues>
          <attvalue for="0" value="Quark: Controllable Text Generation with Reinforced Unlearning" />
          <attvalue for="1" value="  Large-scale language models often learn behaviors that are misaligned with&#10;user expectations. Generated text may contain offensive or toxic language,&#10;contain significant repetition, or be of a different sentiment than desired by&#10;the user. We consider the task of unlearning these misalignments by fine-tuning&#10;the language model on signals of what not to do. We introduce Quantized Reward&#10;Konditioning (Quark), an algorithm for optimizing a reward function that&#10;quantifies an (un)wanted property, while not straying too far from the original&#10;model. Quark alternates between (i) collecting samples with the current&#10;language model, (ii) sorting them into quantiles based on reward, with each&#10;quantile identified by a reward token prepended to the language model's input,&#10;and (iii) using a standard language modeling loss on samples from each quantile&#10;conditioned on its reward token, while remaining nearby the original language&#10;model via a KL-divergence penalty. By conditioning on a high-reward token at&#10;generation time, the model generates text that exhibits less of the unwanted&#10;property. For unlearning toxicity, negative sentiment, and repetition, our&#10;experiments show that Quark outperforms both strong baselines and&#10;state-of-the-art reinforcement learning methods like PPO (Schulman et al.&#10;2017), while relying only on standard language modeling primitives.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.03023" label="2105.03023">
        <attvalues>
          <attvalue for="0" value="DExperts: Decoding-Time Controlled Text Generation with Experts and&#10;  Anti-Experts" />
          <attvalue for="1" value="  Despite recent advances in natural language generation, it remains&#10;challenging to control attributes of generated text. We propose DExperts:&#10;Decoding-time Experts, a decoding-time method for controlled text generation&#10;that combines a pretrained language model with &quot;expert&quot; LMs and/or&#10;&quot;anti-expert&quot; LMs in a product of experts. Intuitively, under the ensemble,&#10;tokens only get high probability if they are considered likely by the experts,&#10;and unlikely by the anti-experts. We apply DExperts to language detoxification&#10;and sentiment-controlled generation, where we outperform existing controllable&#10;generation methods on both automatic and human evaluations. Moreover, because&#10;DExperts operates only on the output of the pretrained LM, it is effective with&#10;(anti-)experts of smaller size, including when operating on GPT-3. Our work&#10;highlights the promise of tuning small LMs on text with (un)desirable&#10;attributes for efficient decoding-time steering.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;The task of controlling the output of a language generation model has been widely studied by previous work \cite[for a review, see][]{prabhumoye-etal-2020-exploring}.&#10;Prior to using pretrained LMs as a backbone, most work used custom neural models trained for their respective downstream generation tasks, including emotion-aware text generation \cite{ghosh-etal-2017-affect,ficler-goldberg-2017-controlling}, attribute-aware product review generation&#10;\cite{dong-etal-2017-learning}, and friendly or empathetic dialogue response generation&#10;\cite{see-etal-2019-makes, rashkin-etal-2019-towards}.&#10;&#10;Since pretrained LMs have shown impressive text generation ability \cite{radford2018improving,gpt22019}, two directions have emerged to control their language generation: training approaches and decoding-time approaches.&#10;Training approaches include finetuning the pretrained LMs on datasets that contain the desired attributes \cite{gururangan-etal-2020-dont} as well as creating a class-conditioned pretrained LM trained on text with specific attributes control code prefixes \cite{ctrl2018}.&#10;In contrast to our method, such approaches can only steer towards desired text attributes, they cannot steer away from them.&#10;Additionally, training approaches require significant computational resources, which may no longer be feasible with the size of more recent pretrained LMs \cite{gpt32020, Fedus2021SwitchTransformers}.&#10;&#10;Decoding-time methods, a more lightweight approach, have been used controlling the attributes of generated text, as well as for improving its quality \cite{li-etal-2016-diversity,holtzman-etal-2018-learning, welleck-etal-2020-neural}.&#10;PPLM \cite{pplm2020} is a steering method that updates a pretrained model's hidden representations according to the gradient of a classifier with respect to the desired class. &#10;Unfortunately, this approach is computationally expensive, as shown in this and previous work \cite{gehman-etal-2020-realtoxicityprompts}.&#10;Contemporaneous with our work, FUDGE \cite{yang-klein-2021-fudge} trains classifiers on partial sequences to predict whether an attribute will be satisfied in the future, and uses Bayesian factorization to obtain the attribute-conditioned probability distribution.&#10;GeDi \cite{gedi2020} uses Bayes' rule similarly, but computes classification probabilities using the output of class-conditioned LMs rather than directly training a classifier. &#10;In contrast, our experiments show that directly ensembling LMs' probabilities as opposed to using them for estimating class probabilities is more effective at steering text generation.&#10; " />
        </attvalues>
      </node>
      <node id="2212.01907" label="2212.01907">
        <attvalues>
          <attvalue for="0" value="Understanding How Model Size Affects Few-shot Instruction Prompting" />
          <attvalue for="1" value="  Large Language Models are affected by the phenomena of memorizing and&#10;forgetting their training data. But how do these vary by model size? We work&#10;towards this question by investigating how the model size affects the model's&#10;ability to discriminate a word's meaning in a given context. We introduce a&#10;dataset called DeltaWords, which evaluates a model's ability to follow&#10;instructions to select a sentence which replaces the target word with its&#10;antonym. We show a weak inverse scaling trend, where task accuracy degrades as&#10;model size increase, under extremely few-shot prompting regimes. We show that&#10;increasing the number of examples tend to disproportionately benefit larger&#10;models than smaller models.&#10;" />
          <attvalue for="2" value="&#10;Suppose a human student has been taught by a teacher to believe in evil ideas. For example, the student is taught that the Rohingya people must be persecuted and therefore the student calls for their genocide, similar to some Facebook users during the ongoing Rohingya crisis. \cite{rohingya} Now suppose a generative language model generates the same speech as the human based on their training data. Our moral goal is to stop this hate speech. The human can be reformed through education to the point where they never again believe the evils they were taught. The machine, on the other hand, seems to have far limited recourse to stop believing in disinformation about the Rohingya people. Worse, such models have the ability to amplify disinformation by making messages more convincing. How can the model learn the new and correct informationwhile forgetting the old, harmful information? We call this the model's capability atword meaning shift (WMS).&#10;&#10;One solution may be fine-tuning or prompting on the correct information. However, what factors affect the effectiveness of these post-pretraining improvements? Recent work has shown larger language models memorize their training data more than smaller models. \cite{carlinimemlang} Another work has investigated forgetting memorized examples in vision and speech models. They found a weak inverse relationship with scaling and forgetting. \cite{jagielskiforget} Complementary to these works, we investigate how model size affects the effectiveness at WMS via prompting. Concretely, we take a step towards WMS by measuring how successful a model can replace a target word with another word with a specified meaning. We call this task word-replacement (WR) Our work focuses on a model's learned definitions of words. &#10;&#10;Given our focus on words, Why might understanding a model's capability to learn shifts in word meaning be useful? We note three main applications.&#10;\begin{enumerate}&#10; \item Privacy: As shown by \cite{brownprivacy}, the context of a sentence may change and thus also its secret, which is the word meant to be private. For example, the word 'abusive' in the sentence &quot;John is my abusive husband.&quot; may only be reserved for close confidants of the speaker and not for acquaintances.&#10;&#10;\item Alignment: For an AI in a decision-making process, a rapid WMS can be needed, which requires the AI to adapt to a new word definition so its actions remain towards the specified goal. For example, team 'Green' is now an 'ally' instead of an 'enemy'.&#10;&#10;\item Bias: Models with good WMS capabilities can pick up new / reclaimed meanings of previously-offensive text. For example, the aforementioned disinformation against the Rohingya people.&#10;\end{enumerate}&#10;&#10;Our contributions are the following:&#10;\begin{enumerate}&#10; \item We introduce DeltaWords (DW), a benchmark training dataset for next-token prediction to measure the effectiveness of few-shot prompting at WR. It consists of 300 sentence-pairs and involve a word and its antonym used in identical sentences. Figure \ref{Fig:data_sample} shows a sample entry. The examples are taken from sentences from the 2018 Story Cloze validation and test datasets by \cite{sharma-etal-2018-tackling}.&#10; \item We evaluate the success in WR by varying the model size. The models are based on the OPT \cite{OPT} and GPT-3 \cite{GPT3} family of autoregressive (decoder-only) transformer-based models.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning Dynamics, Computer Science, Linguistics, Cognitive Science, Model Size Effects, Artificial Intelligence, Language Model Behavior, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2207.00099" label="2207.00099">
        <attvalues>
          <attvalue for="0" value="Measuring Forgetting of Memorized Training Examples" />
          <attvalue for="1" value="  Machine learning models exhibit two seemingly contradictory phenomena:&#10;training data memorization, and various forms of forgetting. In memorization,&#10;models overfit specific training examples and become susceptible to privacy&#10;attacks. In forgetting, examples which appeared early in training are forgotten&#10;by the end. In this work, we connect these phenomena. We propose a technique to&#10;measure to what extent models &quot;forget&quot; the specifics of training examples,&#10;becoming less susceptible to privacy attacks on examples they have not seen&#10;recently. We show that, while non-convex models can memorize data forever in&#10;the worst-case, standard image, speech, and language models empirically do&#10;forget examples over time. We identify nondeterminism as a potential&#10;explanation, showing that deterministically trained models do not forget. Our&#10;results suggest that examples seen early when training with extremely large&#10;datasets - for instance those examples used to pre-train a model - may observe&#10;privacy benefits at the expense of examples seen later.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.05520" label="2202.05520">
        <attvalues>
          <attvalue for="0" value="What Does it Mean for a Language Model to Preserve Privacy?" />
          <attvalue for="1" value="  Natural language reflects our private lives and identities, making its&#10;privacy concerns as broad as those of real life. Language models lack the&#10;ability to understand the context and sensitivity of text, and tend to memorize&#10;phrases present in their training sets. An adversary can exploit this tendency&#10;to extract training data. Depending on the nature of the content and the&#10;context in which this data was collected, this could violate expectations of&#10;privacy. Thus there is a growing interest in techniques for training language&#10;models that preserve privacy. In this paper, we discuss the mismatch between&#10;the narrow assumptions made by popular data protection techniques (data&#10;sanitization and differential privacy), and the broadness of natural language&#10;and of privacy as a social norm. We argue that existing protection methods&#10;cannot guarantee a generic and meaningful notion of privacy for language&#10;models. We conclude that language models should be trained on text data which&#10;was explicitly produced for public use.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.09895" label="2408.09895">
        <attvalues>
          <attvalue for="0" value="Performance Law of Large Language Models" />
          <attvalue for="1" value="  Guided by the belief of the scaling law, large language models (LLMs) have&#10;achieved impressive performance in recent years. However, scaling law only&#10;gives a qualitative estimation of loss, which is influenced by various factors&#10;such as model architectures, data distributions, tokenizers, and computation&#10;precision. Thus, estimating the real performance of LLMs with different&#10;training settings rather than loss may be quite useful in practical&#10;development. In this article, we present an empirical equation named&#10;&quot;Performance Law&quot; to directly predict the MMLU score of an LLM, which is a&#10;widely used metric to indicate the general capability of LLMs in real-world&#10;conversations and applications. Based on only a few key hyperparameters of the&#10;LLM architecture and the size of training data, we obtain a quite accurate MMLU&#10;prediction of various LLMs with diverse sizes and architectures developed by&#10;different organizations in different years. Performance law can be used to&#10;guide the choice of LLM architecture and the effective allocation of&#10;computational resources without extensive experiments.&#10;" />
          <attvalue for="2" value="&#10;&#10;Owing to the foresight of the scaling law~\cite{kaplan2020scaling} and its derivations~\cite{hoffmann2022training}, the rapid development of large language models (LLMs) in recent years has generated a considerable impact on people's lifestyles and shaped many successful online applications~\cite{wu2023brief,owens2023nature}.&#10;However, the huge computational overhead of LLMs presents significant trial-and-error expenses and uncertainty for their developers~\cite{touvron2023llama}, who may face increased pressure to optimize resource allocation and manage risks effectively.&#10;&#10;To predict the performance of LLM before carrying out the whole experiment, researchers have proposed many variants of scaling laws~\cite{hoffmann2022training,krajewski2024scaling,sardana2023beyond,du2024understanding} to characterize the ability of LLM in different settings.&#10;They are primarily focused on the training loss under a certain configuration.&#10;However, the training loss scales of different models are diverse since they usually have distinct tokenizers and are optimized on different data recipes with various training strategies.&#10;Several recent works on LLM performance prediction~\cite{ye2023predictable,owen2024predictable} are based on model sizes and training data.&#10;However, they have limited accuracy and generality across different model structures (e.g., dense or sparse) and shapes (e.g., wide or deep), which may have substantial impacts on the final performance~\cite{kaplan2020scaling}.&#10;Moreover, the precision and stability of the computing infrastructures are not taken into account, which usually damages the quality of the model~\cite{lee2024fp8}.&#10;Thus, an accurate estimation of LLM performance is still missing for researchers to design and optimize their models to approach the ideal scaling law.&#10;&#10;We discover an empirical equation to predict the MMLU~\cite{hendrycks2020measuring} metric of an LLM, which is a widely used measurement with good relevance to performance in downstream tasks~\cite{dubois2024length}.&#10;We name this equation as ``Performance Law''.&#10;All we need are the amount of training data and the key hyperparameters of a common Transformer-based LLM, including the number of layers, hidden size, and the intermediate size of feed-forward networks.&#10;Even for mixture-of-expert (MoE) models, we only need to add the amount of activation parameters and follow a variant of dense model prediction.&#10;By learning a few regression parameters of the equation on only 10 popular open source models released in 2024, we obtain a surprisingly accurate performance prediction of LLMs of different sizes (from 0.5B to 1000+B) and in different years (from 2020 to 2024) released by different organizations around the world.&#10;Performance law can help interpret various phenomena in real-world LLM development that are not fully covered by the scaling law.&#10;It can guide the choice of LLM architecture under limited budgets to save computational resources and reduce the worrying carbon footprint of LLM training.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Performance Prediction, Mathematics, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="2401.00448" label="2401.00448">
        <attvalues>
          <attvalue for="0" value="Beyond Chinchilla-Optimal: Accounting for Inference in Language Model&#10;  Scaling Laws" />
          <attvalue for="1" value="  Large language model (LLM) scaling laws are empirical formulas that estimate&#10;changes in model quality as a result of increasing parameter count and training&#10;data. However, these formulas, including the popular Deepmind Chinchilla&#10;scaling laws, neglect to include the cost of inference. We modify the&#10;Chinchilla scaling laws to calculate the optimal LLM parameter count and&#10;pre-training data size to train and deploy a model of a given quality and&#10;inference demand. We conduct our analysis both in terms of a compute budget and&#10;real-world costs and find that LLM researchers expecting reasonably large&#10;inference demand (~1B requests) should train models smaller and longer than&#10;Chinchilla-optimal. Furthermore, we train 47 models of varying sizes and&#10;parameter counts to validate our formula and find that model quality continues&#10;to improve as we scale tokens per parameter to extreme ranges (up to 10,000).&#10;Finally, we ablate the procedure used to fit the Chinchilla scaling law&#10;coefficients and find that developing scaling laws only from data collected at&#10;typical token/parameter ratios overestimates the impact of additional tokens at&#10;these extreme ranges.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14947" label="2305.14947">
        <attvalues>
          <attvalue for="0" value="How Predictable Are Large Language Model Capabilities? A Case Study on&#10;  BIG-bench" />
          <attvalue for="1" value="  We investigate the predictability of large language model (LLM) capabilities:&#10;given records of past experiments using different model families, numbers of&#10;parameters, tasks, and numbers of in-context examples, can we accurately&#10;predict LLM performance on new experiment configurations? Answering this&#10;question has practical implications for LLM users (e.g., deciding which models&#10;to try), developers (e.g., prioritizing evaluation on representative tasks),&#10;and the research community (e.g., identifying hard-to-predict capabilities that&#10;warrant further investigation).&#10;  We study the performance prediction problem on experiment records from&#10;BIG-bench. On a random train-test split, an MLP-based predictor achieves an&#10;$R^2$ score greater than 95%, indicating the presence of learnable patterns&#10;within the experiment records. We then formulate the problem of searching for&#10;&quot;small-bench,&quot; an informative subset of BIG-bench tasks from which the&#10;performance on the full set can be maximally recovered. We find a subset as&#10;informative as BIG-bench Hard for evaluating new model families, while being&#10;$3\times$ smaller. Additionally, we find competitive subsets by clustering task&#10;representations learned by our MLP-based predictor and selecting tasks close to&#10;cluster centroids, highlighting the importance of task diversity in&#10;constructing &quot;small-bench.&quot;&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Scaling Laws and Emergent Abilities.&#10;Pre-training scale is critical to language model capabilities. &#10;Research on scaling laws \cite{kaplan2020scaling, rae2021scaling, hoffmann2022training} aims to categorize the relationship between pre-training compute, corpus size, model size and the test log-likelihood loss. Our work can be loosely considered as an extension to scaling laws, with three notable distinctions: (1) we focus on predicting downstream task performance; (2) we use model scale along with other experiment configuration information; (3) we mainly experiment with machine learning methods instead of explicit power laws.&#10;In this same vein, recent work has studied the effect of scale in a ``pre-train then fine-tune'' paradigm \cite{tay2022scale} and has explored non-monotonic scaling laws for complex scaling behaviors \cite{caballero2023broken}.&#10;Another important observation about scale is that very large language models exhibit emergent abilities \cite{wei2022emergent}, which are described as ``unpredictable.'' In this work we empirically examine this claim and quantify the prediction errors under various assumptions.&#10;&#10;Benchmarking for LLMs. &#10;&#10;Along with the development and scaling of LLMs, there are continuing efforts to create benchmarks that assess the capabilities of these models. One general trend for these benchmarks is transitioning from single-task \cite{bowman-etal-2015-large, rajpurkar-etal-2016-squad}, to multi-task \cite{wang-etal-2018-glue, superglue}, and finally to massively multi-task \cite{hendrycks2021measuring, srivastava2023beyond}. However, due to budget or API constraints, models are typically evaluated on only a subset of the full range of available benchmarks. The selection is often made arbitrarily by the models' developers, making it challenging to compare models in a fair and holistic way (see \cite{liang2022holistic}, Fig.~4). &#10;In response to this issue, we study the ``small-bench'' problem and hope it offers insights on efficient benchmarking of LLMs.&#10;&#10;Performance Prediction. NLPerf \cite{xia-etal-2020-predicting} is a pilot work on performance prediction in NLP, focusing on bilingual and cross-lingual tasks. It demonstrates the potential of selecting an informative subset of tasks for evaluation, which inspired our work on searching for ``small-bench.'' \cite{ye-etal-2021-towards} extend NLPerf to account for fine-grained performance measures, confidence intervals and calibration. \cite{zhu-etal-2022-predicting} study predicting a model's downstream performance (GLUE, \cite{wang-etal-2018-glue}) using probing tasks performance (SentEval, \cite{conneau-kiela-2018-senteval}), and advocate for incorporating probing during pre-training.&#10;Our work aims to add to the discussion by focusing on the performance prediction of LLMs. Given the ongoing advancements and substantial influence of LLMs in the field of NLP, we believe this topic is both timely and relevant, potentially holding implications for future development of LLMs.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2401.04757" label="2401.04757">
        <attvalues>
          <attvalue for="0" value="How predictable is language model benchmark performance?" />
          <attvalue for="1" value="  We investigate large language model performance across five orders of&#10;magnitude of compute scaling in eleven recent model architectures. We show that&#10;average benchmark performance, aggregating over many individual tasks and&#10;evaluations as in the commonly-used BIG-Bench dataset, is decently predictable&#10;as a function of training compute scale. Specifically, when extrapolating&#10;BIG-Bench Hard performance across one order of magnitude in compute, we observe&#10;average absolute errors of 6 percentage points (pp). By contrast, extrapolation&#10;for individual BIG-Bench tasks across an order of magnitude in compute yields&#10;higher average errors of 18pp. Nonetheless, individual task performance remains&#10;significantly more predictable than chance. Overall, our work suggests compute&#10;scaling provides a promising basis to forecast AI capabilities in diverse&#10;benchmarks, though predicting performance in specific tasks poses challenges.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.18710" label="2405.18710">
        <attvalues>
          <attvalue for="0" value="To FP8 and Back Again: Quantifying the Effects of Reducing Precision on&#10;  LLM Training Stability" />
          <attvalue for="1" value="  The massive computational costs associated with large language model (LLM)&#10;pretraining have spurred great interest in reduced-precision floating-point&#10;representations to accelerate the process. As a result, the BrainFloat16 (BF16)&#10;precision has become the de facto standard for LLM training, with hardware&#10;support included in recent accelerators. This trend has gone even further in&#10;the latest processors, where FP8 has recently been introduced. However, prior&#10;experience with FP16, which was found to be less stable than BF16, raises&#10;concerns as to whether FP8, with even fewer bits than FP16, can be a&#10;cost-effective option for LLM training. We argue that reduced-precision&#10;training schemes must have similar training stability and hyperparameter&#10;sensitivities to their higher-precision counterparts in order to be&#10;cost-effective. However, we find that currently available methods for FP8&#10;training are not robust enough to allow their use as economical replacements.&#10;This prompts us to investigate the stability of reduced-precision LLM training&#10;in terms of robustness across random seeds and learning rates. To this end, we&#10;propose new evaluation techniques and a new metric for quantifying loss&#10;landscape sharpness in autoregressive language models. By simulating&#10;incremental bit reductions in floating-point representations, we analyze the&#10;relationship between representational power and training stability with the&#10;intent of aiding future research into the field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.04475" label="2404.04475">
        <attvalues>
          <attvalue for="0" value="Length-Controlled AlpacaEval: A Simple Way to Debias Automatic&#10;  Evaluators" />
          <attvalue for="1" value="  LLM-based auto-annotators have become a key component of the LLM development&#10;process due to their cost-effectiveness and scalability compared to human-based&#10;evaluation. However, these auto-annotators can introduce complex biases that&#10;are hard to remove. Even simple, known confounders such as preference for&#10;longer outputs remain in existing automated evaluation metrics. We propose a&#10;simple regression analysis approach for controlling biases in auto-evaluations.&#10;As a real case study, we focus on reducing the length bias of AlpacaEval, a&#10;fast and affordable benchmark for chat LLMs that uses LLMs to estimate response&#10;quality. Despite being highly correlated with human preferences, AlpacaEval is&#10;known to favor models that generate longer outputs. We introduce a&#10;length-controlled AlpacaEval that aims to answer the counterfactual question:&#10;&quot;What would the preference be if the model's and baseline's output had the same&#10;length?&quot;. To achieve this, we first fit a generalized linear model to predict&#10;the biased output of interest (auto-annotator preferences) based on the&#10;mediators we want to control for (length difference) and other relevant&#10;features. We then obtain length-controlled preferences by predicting&#10;preferences while conditioning the GLM with a zero difference in lengths.&#10;Length-controlling not only improves the robustness of the metric to&#10;manipulations in model verbosity, we also find that it increases the Spearman&#10;correlation with LMSYS' Chatbot Arena from 0.94 to 0.98. We release the code&#10;and leaderboard at https://tatsu-lab.github.io/alpaca_eval/ .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.02114" label="2111.02114">
        <attvalues>
          <attvalue for="0" value="LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs" />
          <attvalue for="1" value="  Multi-modal language-vision models trained on hundreds of millions of&#10;image-text pairs (e.g. CLIP, DALL-E) gained a recent surge, showing remarkable&#10;capability to perform zero- or few-shot learning and transfer even in absence&#10;of per-sample labels on target image data. Despite this trend, to date there&#10;has been no publicly available datasets of sufficient scale for training such&#10;models from scratch. To address this issue, in a community effort we build and&#10;release for public LAION-400M, a dataset with CLIP-filtered 400 million&#10;image-text pairs, their CLIP embeddings and kNN indices that allow efficient&#10;similarity search.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multi-modal language-vision models demonstrated recently strong transfer capability to novel datasets in absense of per-sample labels~\cite{clip, dalle, align}. This capability requires sufficiently large model and data scale during pre-training. Increasing data scale alone can often improve model performance \cite{epoch}. When increasing model and compute budget scale in addition, scaling laws suggest further increase in generalization and transfer performance if not bottlenecked by the data scale \cite{kaplan, kaplan2, Kolesnikov2020, zhai2021scaling}. There is a plethora of recent works that have built massive datasets in order to optimally scale up various models \cite{gpt3, clip, dalle, align}. However, these massive datasets have rarely been released for various reasons. Gao et. al. recently released The Pile, an openly-available 800GB text dataset~\cite{pile}, in an attempt to loosely mimic the dataset used for GPT-3. The largest publicly known image-text paired datasets range from 400 million to around a billion, but none of them has been released. &#10;&#10;To address this issue, we build and release LAION-400M, a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices. We describe the procedure to create the dataset and demonstrate successful training of DALL-E architecture. Having sufficiently large scale, the dataset opens venues for research on multi-modal language-vision models to broad community.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Large Scale Datasets, Computer Science, Machine Learning, Language Vision Models, Linguistics, Computer Vision, Artificial Intelligence, Artificial Intelligence Training, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2405.13015" label="2405.13015">
        <attvalues>
          <attvalue for="0" value="Assisted Debate Builder with Large Language Models" />
          <attvalue for="1" value="  We introduce ADBL2, an assisted debate builder tool. It is based on the&#10;capability of large language models to generalise and perform relation-based&#10;argument mining in a wide-variety of domains. It is the first open-source tool&#10;that leverages relation-based mining for (1) the verification of&#10;pre-established relations in a debate and (2) the assisted creation of new&#10;arguments by means of large language models. ADBL2 is highly modular and can&#10;work with any open-source large language models that are used as plugins. As a&#10;by-product, we also provide the first fine-tuned Mistral-7B large language&#10;model for relation-based argument mining, usable by ADBL2, which outperforms&#10;existing approaches for this task with an overall F1-score of 90.59% across all&#10;domains.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, there has been a lot of research in artificial intelligence, focusing on leveraging argumentation theory for non-monotonic reasoning \cite{DBLP:conf/sum/CroitoruV13,DBLP:phd/hal/Yun19}. &#10;Starting with Dung's seminal work \cite{DBLP:journals/ai/Dung95}, many researchers have considered abstract argumentation frameworks, composed of a set of arguments and a binary attack relation between them, and created many semantics for tasks such as computing accepted sets of arguments \cite{DBLP:journals/ker/BaroniCG11, DBLP:conf/comma/Caminada06} or rank arguments \cite{DBLP:conf/sum/AmgoudB13a,DBLP:conf/aaai/BonzonDKM16,DBLP:conf/comma/YunVCB18}.&#10;&#10;This abstract argumentation framework was extended with many features such as supports \cite{DBLP:conf/nmr/AmgoudCL04,DBLP:conf/ecsqaru/CayrolL05,DBLP:conf/sgai/HimeurYBC21}, sets of attacking arguments \cite{DBLP:conf/argmas/NielsenP06,DBLP:conf/aaai/YunVC20}, or probabilities \cite{DBLP:journals/jair/HunterT17} among others. &#10; However, one important question that remained was: ``Where do argumentation frameworks come from in real-life settings?''.&#10;&#10;While there are some pieces of evidence that the fundamental aspects of abstract argumentation frameworks have links with human reasoning \cite{DBLP:conf/jelia/CramerG19,DBLP:conf/atal/VesicYT22}, humans debates or natural language texts are not always written as arguments and the relation between arguments is not always clear, even for experts \cite{DBLP:conf/ijcai/CramerG18}.&#10;The question of the origin of argumentation frameworks is crucial to facilitate the application of argumentation theory semantics in real-world contexts. &#10;&#10;Some online debate platforms like Kialo, Debategraph, Rationale, or Argüman allow users to formalise (individually or collaboratively) debates into arguments and attacks/supports. While this constitute a possible source of argumentation frameworks, users are not assisted in the creation of arguments, leading to redundancies, poorly phrased arguments or wrongly classified relations.&#10;We argue that an automatic assistant is essential to help users elicit high quality argumentation frameworks. &#10;Moreover, this automatic assistant would need to be highly adaptable to a variety of debate domains, thus motivating the need for large language models (LLMs).&#10;&#10;In this paper, our contributions are as follows:&#10;&#10;\begin{itemize}&#10; \item ADBL2, an assisted debate builder tool. &#10;It leverages the capability of large language models to generalise and perform relation-based argument mining (RBAM) in a wide-variety of domains.&#10;While RBAM has been used for several tasks \cite{DBLP:conf/naacl/CarstensT15,DBLP:conf/lrec/KonatLPBR16}, ADBL2 is the first open-source tool that imports debates from Kialo and leverages RBAM for (2) the verification of existing relations in a debate, and (3) assist users in the creation of new arguments.&#10;&#10;\item An open-source and fine-tuned Mistral-7B LLM for the task of relation-based argument mining, embedded in ADBL2, which outperforms existing approaches in multiple domains.&#10;\end{itemize}&#10;&#10;This demonstration paper is structured as follows. In Section \ref{sec:llmRBAM}, we motivate the use of fine-tuned LLMs for the RBAM task. In Section \ref{sec:tool}, we introduce the architecture and use-cases of ADBL2. In Section \ref{sec:fine-tined}, we explain the data collection, fine-tuning, and evaluation of our LLM. Finally, we conclude and discuss future work in Section \ref{sec:future_work}.&#10;&#10;The demo video is available at:.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Argument Mining, Linguistics, Debate Assistance, Artificial Intelligence, Philosophy, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1602.01059" label="1602.01059">
        <attvalues>
          <attvalue for="0" value="A Comparative Study of Ranking-based Semantics for Abstract&#10;  Argumentation" />
          <attvalue for="1" value="  Argumentation is a process of evaluating and comparing a set of arguments. A&#10;way to compare them consists in using a ranking-based semantics which&#10;rank-order arguments from the most to the least acceptable ones. Recently, a&#10;number of such semantics have been proposed independently, often associated&#10;with some desirable properties. However, there is no comparative study which&#10;takes a broader perspective. This is what we propose in this work. We provide a&#10;general comparison of all these semantics with respect to the proposed&#10;properties. That allows to underline the differences of behavior between the&#10;existing semantics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.08439" label="2008.08439">
        <attvalues>
          <attvalue for="0" value="BabelEnconding at SemEval-2020 Task 3: Contextual Similarity as a&#10;  Combination of Multilingualism and Language Models" />
          <attvalue for="1" value="  This paper describes the system submitted by our team (BabelEnconding) to&#10;SemEval-2020 Task 3: Predicting the Graded Effect of Context in Word&#10;Similarity. We propose an approach that relies on translation and multilingual&#10;language models in order to compute the contextual similarity between pairs of&#10;words. Our hypothesis is that evidence from additional languages can leverage&#10;the correlation with the human generated scores. BabelEnconding was applied to&#10;both subtasks and ranked among the top-3 in six out of eight task/language&#10;combinations and was the highest scoring system three times.&#10;" />
          <attvalue for="2" value="&#10;\blfootnote{&#10; This work is licensed under a Creative Commons &#10; Attribution 4.0 International License.&#10; License details:&#10; .&#10;}&#10;&#10;Word similarity is a key task in Natural Language Processing (NLP) applications. &#10;Language models, such as word embeddings~\cite{mikolov2013} create vector representations for the words that are able to capture syntactic and semantic relationships.&#10;These representations became very popular in the last few years as they have boosted the performance of several NLP tasks.&#10;However, since each word is represented by a fixed vector these techniques have problems dealing with polysemous words and identifying subtle meaning changes between different sentences.&#10;On the other hand, state-of-the-art language models, like BERT \cite{devlin2019} provide a contextualized word representation -- the representation of a word relies on its context, which means that the same word may have different representations through the sentences.&#10;Thus, BERT models are more suitable for handling polysemous words.&#10;&#10;Task 3 in SemEval 2020 -- Predicting the Graded Effect of Context in Word Similarity \cite{task3description} was motivated by this improvement on language models. &#10;The task aims at the design of a similarity measure which captures the human perception of the meaning of words.&#10;For that purpose, task organizers built and annotated datasets in four languages -- English, Croatian, Finnish, and Slovenian. &#10;Each entry in a dataset consists of two target words and two contexts, where each one is a piece of text containing both target words. &#10;The global task is divided into two subtasks: 1) predicting the change in the human annotator's scores of similarity when presented with the same pair of words within two different contexts; and 2) predicting the human scores of similarity for a pair of words within two different contexts. &#10;&#10;In this paper, we describe {BabelEnconding}, an approach that relies on machine translation and multilingual language models to evaluate the contextual similarity of pairs of words.&#10;Our hypothesis is that having similarity information from more languages helps decide on how similar the words are.&#10;&#10;Considering the eight combinations of language/subtask, BabelEnconding was ranked among the top-3 competitors six times, and was the top scoring method in three cases.&#10;Our additional experiments in English and Croatian showed that adding more languages noticeably improved the results for Croatian in both subtasks.&#10;In English, the gain was small and happened only in Subtask~2.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Multilingual Language Models, Contextual Translation Approaches, Artificial Intelligence, Natural Language Processing, Word Similarity Analysis" />
        </attvalues>
      </node>
      <node id="2408.02085" label="2408.02085">
        <attvalues>
          <attvalue for="0" value="Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data&#10;  Assessment and Selection for Instruction Tuning of Language Models" />
          <attvalue for="1" value="  Instruction tuning plays a critical role in aligning large language models&#10;(LLMs) with human preference. Despite the vast amount of open instruction&#10;datasets, naively training a LLM on all existing instructions may not be&#10;optimal and practical. To pinpoint the most beneficial datapoints, data&#10;assessment and selection methods have been proposed in the fields of natural&#10;language processing (NLP) and deep learning. However, under the context of&#10;instruction tuning, there still exists a gap in knowledge on what kind of data&#10;evaluation metrics can be employed and how they can be integrated into the&#10;selection mechanism. To bridge this gap, we present a comprehensive review on&#10;existing literature of data assessment and selection especially for instruction&#10;tuning of LLMs. We systematically categorize all applicable methods into&#10;quality-based, diversity-based, and importance-based ones where a unified,&#10;fine-grained taxonomy is structured. For each category, representative methods&#10;are elaborated to describe the landscape of relevant research. In addition,&#10;comparison between latest methods is conducted on their officially reported&#10;results to provide in-depth discussions on their limitations. Finally, we&#10;summarize the open challenges and propose the promosing avenues for future&#10;studies. All related contents are available at&#10;https://github.com/yuleiqin/fantastic-data-engineering.&#10;" />
          <attvalue for="2" value="&#10;&#10;One of the ultimate goal of developing large lnguage models (LLMs) is to unlock their potentials of generalization to unseen natural language processing (NLP) tasks.&#10;Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}].&#10;During preference alignment,&#10;instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}].&#10;For efficient and effective instruction tuning,&#10;existing studies~[\cite{ouyang2022training,taori2023alpaca,zhou2024lima,xia2024less}] have noticed that improving {the } quality of instruction tuning data (e.g., formulation of well-defined and complete contexts),&#10;rather than simply piling up instructions without analysis (e.g., exhaustive collection of open datasets),&#10;is of prioritized concerns.&#10;&#10;In this work,&#10;we aim to unify a wide array of data assessment and selection methods under the context of instruction tuning of LLMs.&#10;As revealed from the probabilistic view~[\cite{john1975d,murphy2012machine,albalak2024survey}],&#10;the statistical patterns inherent in datasets {determine } the modeling performance.&#10;The overall evaluation of datapoints not only deciphers the distribution in various aspects (e.g., composition, task, and domain) {but } also {helps } cherry-pick the most beneficial subsets for higher performance with less training cost.&#10;Through this survey,&#10;we demonstrate that:&#10;1) existing resourceful data assessment methods can be categorized into three main perspectives: quality, diversity, and importance (see Fig.~\ref{fig:category}).&#10;2) a systematic view of selection methods can be unified even they more or less exhibit coupling with the assessment techniques (see Fig.~\ref{fig:overview}).&#10;It is noted that quality, diversity, and importance might be used interchangeably without strict discrimination in previous studies.&#10;But here we provide a rationalized organization taxonomy for structured elaboration.&#10;Despite the goal of being comprehensive,&#10;the present survey only provides details of certain typical, representative methods to avoid being tediously long.&#10;We hope the in-depth explanations and discussions on the selected methods provide insights into developing robust data assessment and selection pipelines for {future } studies.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Tuning, Computer Science, Linguistics, Data Assessment Methods, Deep Learning, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2403.09606" label="2403.09606">
        <attvalues>
          <attvalue for="0" value="Large Language Models and Causal Inference in Collaboration: A&#10;  Comprehensive Survey" />
          <attvalue for="1" value="  Causal inference has shown potential in enhancing the predictive accuracy,&#10;fairness, robustness, and explainability of Natural Language Processing (NLP)&#10;models by capturing causal relationships among variables. The emergence of&#10;generative Large Language Models (LLMs) has significantly impacted various NLP&#10;domains, particularly through their advanced reasoning capabilities. This&#10;survey focuses on evaluating and improving LLMs from a causal view in the&#10;following areas: understanding and improving the LLMs' reasoning capacity,&#10;addressing fairness and safety issues in LLMs, complementing LLMs with&#10;explanations, and handling multimodality. Meanwhile, LLMs' strong reasoning&#10;capacities can in turn contribute to the field of causal inference by aiding&#10;causal relationship discovery and causal effect estimations. This review&#10;explores the interplay between causal inference frameworks and LLMs from both&#10;perspectives, emphasizing their collective potential to further the development&#10;of more advanced and equitable artificial intelligence systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.05789" label="2403.05789">
        <attvalues>
          <attvalue for="0" value="ItD: Large Language Models Can Teach Themselves Induction through&#10;  Deduction" />
          <attvalue for="1" value="  Although Large Language Models (LLMs) are showing impressive performance on a&#10;wide range of Natural Language Processing tasks, researchers have found that&#10;they still have limited ability to conduct induction. Recent works mainly adopt&#10;``post processes'' paradigms to improve the performance of LLMs on induction&#10;(e.g., the hypothesis search &amp; refinement methods), but their performance is&#10;still constrained by the inherent inductive capability of the LLMs. In this&#10;paper, we propose a novel framework, Induction through Deduction (ItD), to&#10;enable the LLMs to teach themselves induction through deduction. The ItD&#10;framework is composed of two main components: a Deductive Data Generation&#10;module to generate induction data and a Naive Bayesian Induction module to&#10;optimize the fine-tuning and decoding of LLMs. Our empirical results showcase&#10;the effectiveness of ItD on two induction benchmarks, achieving relative&#10;performance improvement of 36% and 10% compared with previous state-of-the-art,&#10;respectively. Our ablation study verifies the effectiveness of two key modules&#10;of ItD. We also verify the effectiveness of ItD across different LLMs and&#10;deductors. The data and code of this paper can be found at&#10;https://anonymous.4open.science/r/ItD-E844.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1903.09722" label="1903.09722">
        <attvalues>
          <attvalue for="0" value="Pre-trained Language Model Representations for Language Generation" />
          <attvalue for="1" value="  Pre-trained language model representations have been successful in a wide&#10;range of language understanding tasks. In this paper, we examine different&#10;strategies to integrate pre-trained representations into sequence to sequence&#10;models and apply it to neural machine translation and abstractive&#10;summarization. We find that pre-trained representations are most effective when&#10;added to the encoder network which slows inference by only 14%. Our experiments&#10;in machine translation show gains of up to 5.3 BLEU in a simulated&#10;resource-poor setup. While returns diminish with more labeled data, we still&#10;observe improvements when millions of sentence-pairs are available. Finally, on&#10;abstractive summarization we achieve a new state of the art on the full text&#10;version of CNN/DailyMail.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.17194" label="2306.17194">
        <attvalues>
          <attvalue for="0" value="On the Exploitability of Instruction Tuning" />
          <attvalue for="1" value="  Instruction tuning is an effective technique to align large language models&#10;(LLMs) with human intents. In this work, we investigate how an adversary can&#10;exploit instruction tuning by injecting specific instruction-following examples&#10;into the training data that intentionally changes the model's behavior. For&#10;example, an adversary can achieve content injection by injecting training&#10;examples that mention target content and eliciting such behavior from&#10;downstream models. To achieve this goal, we propose \textit{AutoPoison}, an&#10;automated data poisoning pipeline. It naturally and coherently incorporates&#10;versatile attack goals into poisoned data with the help of an oracle LLM. We&#10;showcase two example attacks: content injection and over-refusal attacks, each&#10;aiming to induce a specific exploitable behavior. We quantify and benchmark the&#10;strength and the stealthiness of our data poisoning scheme. Our results show&#10;that AutoPoison allows an adversary to change a model's behavior by poisoning&#10;only a small fraction of data while maintaining a high level of stealthiness in&#10;the poisoned examples. We hope our work sheds light on how data quality affects&#10;the behavior of instruction-tuned models and raises awareness of the importance&#10;of data quality for responsible deployments of LLMs. Code is available at&#10;\url{https://github.com/azshue/AutoPoison}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.03202" label="2302.03202">
        <attvalues>
          <attvalue for="0" value="Exploring the Benefits of Training Expert Language Models over&#10;  Instruction Tuning" />
          <attvalue for="1" value="  Recently, Language Models (LMs) instruction-tuned on multiple tasks, also&#10;known as multitask-prompted fine-tuning (MT), have shown the capability to&#10;generalize to unseen tasks. Previous work has shown that scaling the number of&#10;training tasks is the key component in making stronger MT LMs. In this work, we&#10;report an unexpected finding that an expert LM fine-tuned on just a single task&#10;can outperform an MT LM trained with 300+ different tasks on 11 different&#10;unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean&#10;accuracy of 3.20% and 1.29%, respectively. This finding casts doubt on the&#10;previously held belief that simply scaling the number of tasks makes stronger&#10;MT LMs. Leveraging this finding, we further show that this distributed approach&#10;of training a separate expert LM per training task instead of a single MT LM&#10;for zero-shot inference possesses many benefits including (1) avoiding negative&#10;task transfer that often occurs during instruction tuning, (2) being able to&#10;continually learn new tasks without having to re-train on previous tasks to&#10;avoid catastrophic forgetting, and (3) showing compositional capabilities when&#10;merging individual experts together. The code is available at&#10;https://github.com/joeljang/ELM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.05119" label="2402.05119">
        <attvalues>
          <attvalue for="0" value="A Closer Look at the Limitations of Instruction Tuning" />
          <attvalue for="1" value="  Instruction Tuning (IT), the process of training large language models (LLMs)&#10;using instruction-response pairs, has emerged as the predominant method for&#10;transforming base pre-trained LLMs into open-domain conversational agents.&#10;While IT has achieved notable success and widespread adoption, its limitations&#10;and shortcomings remain underexplored. In this paper, through rigorous&#10;experiments and an in-depth analysis of the changes LLMs undergo through IT, we&#10;reveal various limitations of IT. In particular, we show that (1) IT fails to&#10;enhance knowledge or skills in LLMs. LoRA fine-tuning is limited to learning&#10;response initiation and style tokens, and full-parameter fine-tuning leads to&#10;knowledge degradation. (2) Copying response patterns from IT datasets derived&#10;from knowledgeable sources leads to a decline in response quality. (3)&#10;Full-parameter fine-tuning increases hallucination by inaccurately borrowing&#10;tokens from conceptually similar instances in the IT dataset for generating&#10;responses. (4) Popular methods to improve IT do not lead to performance&#10;improvements over a simple LoRA fine-tuned model. Our findings reveal that&#10;responses generated solely from pre-trained knowledge consistently outperform&#10;responses by models that learn any form of new knowledge from IT on open-source&#10;datasets. We hope the insights and challenges revealed in this paper inspire&#10;future work in related directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;While extensive research has introduced new IT datasets, models, and enhancement methods~\cite{zhang2023instruction}, few studies examine IT's limitations. Concurrent work by ~\cite{gudibande2023false} shows that IT with datasets synthesized from powerful proprietary models only leads to imitating their style, not their knowledge. Similarly, ~\cite{lin2023unlocking} shows that alignment only teaches style and proposes in-context learning as an alternative to IT that outperforms several tuned models. This can be attributed to our findings on LFT, where the models respond using pre-trained knowledge and do not lead to knowledge degradation like SFT. This supports the superficial alignment hypothesis by \cite{zhou2023lima}, positing that models gain knowledge in pre-training, with alignment shaping format used in user interactions. Finally, ~\cite{kung2023models} show that models trained on simplified task definition&#10;or delusive examples achieve performance comparable to the ones trained on the original&#10;instructions. In contrast to all these works, we study the exact causes of these limitations, investigate pattern copying and hallucination from novel perspectives, and highlight the overlooked effectiveness of LFT that utilizes only pre-trained knowledge.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.04333" label="2402.04333">
        <attvalues>
          <attvalue for="0" value="LESS: Selecting Influential Data for Targeted Instruction Tuning" />
          <attvalue for="1" value="  Instruction tuning has unlocked powerful capabilities in large language&#10;models (LLMs), effectively using combined datasets to develop generalpurpose&#10;chatbots. However, real-world applications often require a specialized suite of&#10;skills (e.g., reasoning). The challenge lies in identifying the most relevant&#10;data from these extensive datasets to effectively develop specific&#10;capabilities, a setting we frame as targeted instruction tuning. We propose&#10;LESS, an optimizer-aware and practically efficient algorithm to effectively&#10;estimate data influences and perform Low-rank gradiEnt Similarity Search for&#10;instruction data selection. Crucially, LESS adapts existing influence&#10;formulations to work with the Adam optimizer and variable-length instruction&#10;data. LESS first constructs a highly reusable and transferable gradient&#10;datastore with low-dimensional gradient features and then selects examples&#10;based on their similarity to few-shot examples embodying a specific capability.&#10;Experiments show that training on a LESS-selected 5% of the data can often&#10;outperform training on the full dataset across diverse downstream tasks.&#10;Furthermore, the selected data is highly transferable: smaller models can be&#10;leveraged to select useful data for larger models and models from different&#10;families. Our qualitative analysis shows that our method goes beyond surface&#10;form cues to identify data that exemplifies the necessary reasoning skills for&#10;the intended downstream application.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.07889" label="2307.07889">
        <attvalues>
          <attvalue for="0" value="LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise&#10;  Comparisons using Large Language Models" />
          <attvalue for="1" value="  Current developments in large language models (LLMs) have enabled impressive&#10;zero-shot capabilities across various natural language tasks. An interesting&#10;application of these systems is in the automated assessment of natural language&#10;generation (NLG), a highly challenging area with great practical benefit. In&#10;this paper, we explore two options for exploiting the emergent abilities of&#10;LLMs for zero-shot NLG assessment: absolute score prediction, and comparative&#10;assessment which uses relative comparisons between pairs of candidates. Though&#10;comparative assessment has not been extensively studied in NLG assessment, we&#10;note that humans often find it more intuitive to compare two options rather&#10;than scoring each one independently. This work examines comparative assessment&#10;from multiple perspectives: performance compared to absolute grading;&#10;positional biases in the prompt; and efficient ranking in terms of the number&#10;of comparisons. We illustrate that LLM comparative assessment is a simple,&#10;general and effective approach for NLG assessment. For moderate-sized&#10;open-source LLMs, such as FlanT5 and Llama2-chat, comparative assessment is&#10;superior to prompt scoring, and in many cases can achieve performance&#10;competitive with state-of-the-art methods. Additionally, we demonstrate that&#10;LLMs often exhibit strong positional biases when making pairwise comparisons,&#10;and we propose debiasing methods that can further improve performance.&#10;" />
          <attvalue for="2" value="&#10;With the current rapid advances in generative AI, pre-trained models are increasingly utilized in a range of NLP tasks, necessitating reliable evaluations of these models. Human evaluation, where annotators critically assess the quality of the outputs of natural language generation (NLG) systems, has been the gold standard approach \cite{lita-etal-2005-blanc, belz-reiter-2006-comparing, lai-tetreault-2018-discourse, fabbri2021summeval}. However, human evaluation has its drawbacks, and is notably labor-intensive, time-consuming, and costly. As such, automating the evaluation process and assessing NLG systems without human intervention is highly desirable. &#10;&#10;Though there has been considerable progress in automatic evaluation methods, many proposed approaches have certain restrictions that limit their effectiveness. A large body of existing work use evaluation methods designed for particular tasks and attributes \cite{mehri-eskenazi-2020-unsupervised, rei-etal-2020-comet, manakul2023selfcheckgpt}, for example, measuring the consistency of summaries \cite{wang-etal-2020-asking, manakul2023mqag}. Though effective within their domain, these approaches are not extensible to different NLG aspects and cannot be used by practitioners wishing to evaluate systems on inputs or properties that are less common. &#10;&#10;The recent development in the emergent abilities of LLMs \cite{wei2022emergent} has enabled LLMs to achieve impressive zero-shot performance for a slew of language tasks. This has led to general prompt-based assessment approaches, such as prompt-scoring where an LLM is probed to score outputs on a particular aspect \cite{wang2023chatgpt, kocmi2023large}. These approaches are often only effective with massive LLMs with 175B+ parameters, which may limit the applicability of the approach, especially when access is limited to API access. &#10;&#10;With the insight that for humans, it is often easier to select which of two options is better than it is to score options independently, we question whether pairwise comparisons may be more effective at leveraging the impressive emergent ability of LLMs. In this work, we consider LLM comparative assessment, where an LLM is prompted to compare pairs of NLG candidates and predict which one is better. We demonstrate empirically that comparative assessment performs much better than prompt-scoring for FlanT5 and Llama style models, and enables moderate-sized open-source LLMs to achieve near (or above) state-of-the-art performance across a range of NLG language tasks, for a diverse set of attributes. Our approach is general and can be applied to a diverse range of tasks and textual attributes, is simple and requires minimal prompt engineering. Further, we demonstrate that pairwise LLM comparisons often exhibit strong positional biases, where the ordering of candidates impacts the decisions. We introduce a simple debiasing method and empirically illustrate that debiasing can provide further performance improvements, especially when large biases are present. &#10;&#10;Our contributions are 1) We are the first work that comprehensively analyzes pairwise comparative assessment for NLG evaluation; 2) We demonstrate that comparative assessment is far more effective than prompt-scoring for moderately-sized LLMs, and yields performance that is state-of-the-art for particular attributes; 3) We demonstrate that positional bias impacts comparative decisions, and introduce a method to debias LLMs which leads to performance boosts, especially when only a subset of comparisons are considered. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Artificial Intelligence, Automated Assessment Methods, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2006.12719" label="2006.12719">
        <attvalues>
          <attvalue for="0" value="Unsupervised Evaluation of Interactive Dialog with DialoGPT" />
          <attvalue for="1" value="  It is important to define meaningful and interpretable automatic evaluation&#10;metrics for open-domain dialog research. Standard language generation metrics&#10;have been shown to be ineffective for dialog. This paper introduces the FED&#10;metric (fine-grained evaluation of dialog), an automatic evaluation metric&#10;which uses DialoGPT, without any fine-tuning or supervision. It also introduces&#10;the FED dataset which is constructed by annotating a set of human-system and&#10;human-human conversations with eighteen fine-grained dialog qualities. The FED&#10;metric (1) does not rely on a ground-truth response, (2) does not require&#10;training data and (3) measures fine-grained dialog qualities at both the turn&#10;and whole dialog levels. FED attains moderate to strong correlation with human&#10;judgement at both levels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.04228" label="2004.04228">
        <attvalues>
          <attvalue for="0" value="Asking and Answering Questions to Evaluate the Factual Consistency of&#10;  Summaries" />
          <attvalue for="1" value="  Practical applications of abstractive summarization models are limited by&#10;frequent factual inconsistencies with respect to their input. Existing&#10;automatic evaluation metrics for summarization are largely insensitive to such&#10;errors. We propose an automatic evaluation protocol called QAGS (pronounced&#10;&quot;kags&quot;) that is designed to identify factual inconsistencies in a generated&#10;summary. QAGS is based on the intuition that if we ask questions about a&#10;summary and its source, we will receive similar answers if the summary is&#10;factually consistent with the source. To evaluate QAGS, we collect human&#10;judgments of factual consistency on model-generated summaries for the&#10;CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018)&#10;summarization datasets. QAGS has substantially higher correlations with these&#10;judgments than other automatic evaluation metrics. Also, QAGS offers a natural&#10;form of interpretability: The answers and questions generated while computing&#10;QAGS indicate which tokens of a summary are inconsistent and why. We believe&#10;QAGS is a promising tool in automatically generating usable and factually&#10;consistent text.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.04048" label="2303.04048">
        <attvalues>
          <attvalue for="0" value="Is ChatGPT a Good NLG Evaluator? A Preliminary Study" />
          <attvalue for="1" value="  Recently, the emergence of ChatGPT has attracted wide attention from the&#10;computational linguistics community. Many prior studies have shown that ChatGPT&#10;achieves remarkable performance on various NLP tasks in terms of automatic&#10;evaluation metrics. However, the ability of ChatGPT to serve as an evaluation&#10;metric is still underexplored. Considering assessing the quality of natural&#10;language generation (NLG) models is an arduous task and NLG metrics notoriously&#10;show their poor correlation with human judgments, we wonder whether ChatGPT is&#10;a good NLG evaluation metric. In this report, we provide a preliminary&#10;meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail,&#10;we regard ChatGPT as a human evaluator and give task-specific (e.g.,&#10;summarization) and aspect-specific (e.g., relevance) instruction to prompt&#10;ChatGPT to evaluate the generated results of NLG models. We conduct experiments&#10;on five NLG meta-evaluation datasets (including summarization, story generation&#10;and data-to-text tasks). Experimental results show that compared with previous&#10;automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation&#10;with human judgments in most cases. In addition, we find that the effectiveness&#10;of the ChatGPT evaluator might be influenced by the creation method of the&#10;meta-evaluation datasets. For the meta-evaluation datasets which are created&#10;greatly depending on the reference and thus are biased, the ChatGPT evaluator&#10;might lose its effectiveness. We hope our preliminary study could prompt the&#10;emergence of a general-purposed reliable NLG metric.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12836" label="2310.12836">
        <attvalues>
          <attvalue for="0" value="Knowledge-Augmented Language Model Verification" />
          <attvalue for="1" value="  Recent Language Models (LMs) have shown impressive capabilities in generating&#10;texts with the knowledge internalized in parameters. Yet, LMs often generate&#10;the factually incorrect responses to the given queries, since their knowledge&#10;may be inaccurate, incomplete, and outdated. To address this problem, previous&#10;works propose to augment LMs with the knowledge retrieved from an external&#10;knowledge source. However, such approaches often show suboptimal text&#10;generation performance due to two reasons: 1) the model may fail to retrieve&#10;the knowledge relevant to the given query, or 2) the model may not faithfully&#10;reflect the retrieved knowledge in the generated text. To overcome these, we&#10;propose to verify the output and the knowledge of the knowledge-augmented LMs&#10;with a separate verifier, which is a small LM that is trained to detect those&#10;two types of errors through instruction-finetuning. Then, when the verifier&#10;recognizes an error, we can rectify it by either retrieving new knowledge or&#10;generating new text. Further, we use an ensemble of the outputs from different&#10;instructions with a single verifier to enhance the reliability of the&#10;verification processes. We validate the effectiveness of the proposed&#10;verification steps on multiple question answering benchmarks, whose results&#10;show that the proposed verifier effectively identifies retrieval and generation&#10;errors, allowing LMs to provide more factually correct outputs. Our code is&#10;available at https://github.com/JinheonBaek/KALMV.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent Language Models (LMs)~\cite{gpt3, PaLM, flan}, which have a large number of parameters and are further instruction-finetuned on massive datasets, have achieved remarkable successes on various language tasks. For example, they are able to perform closed-book zero-shot question answering, which aims to provide an answer to a user's query without updating the LM parameters while using only the knowledge internalized in their parameters. However, while the generated answers from LMs look plausible and sound, they are often factually incorrect, which is a problem widely known as hallucination~\cite{Hallucination, llm/eval, llm/truthful}. Hallucination is a critical problem when deploying LMs, since it poses a risk of spreading misinformation, potentially misleading users who rely on the information.&#10;&#10;To mitigate hallucination of LMs, recent works have proposed to augment LMs with the knowledge retrieved from external knowledge sources (e.g., Wikipedia and Wikidata)~\cite{internet-augment, Adaptive_Retrieval, KAPING}. Moreover, some other works have proposed to check the factuality of generated texts and refine them by using the knowledge in LMs themselves or from the external knowledge sources~\cite{self-refine, RARR, FLARE, CRITIC, search-chain, InteR}. However, while the aforementioned knowledge-augmentation strategies are effective in reducing hallucinations, we find that there still exists a couple of challenges: 1) the retrieved knowledge may not be relevant to the given question from the user, and 2) the generated answer may not be grounded in the retrieved knowledge, as illustrated in Figure~\ref{fig:concept} and shown in Figure~\ref{fig:verify}.&#10;&#10;In this work, we aim to overcome these suboptimalities of knowledge-augmented LMs. In other words, our goal is to verify whether the retrieved knowledge used for augmenting LMs is related to generating the answers for the given questions and whether the generated answers include the relevant parts of the retrieved knowledge. To this end, we propose to train a small, tailorable LM that is able to verify the aforementioned two failure cases of knowledge-augmented LMs in retrieval and generation steps. More specifically, we first automatically construct the training labels by categorizing the failure of knowledge-augmented LMs into two cases: retrieval error and generation error, based on the triplet of the input question, retrieved knowledge, and generated answer. Then, we instruction-finetune the LM with pairs of a certain verification instruction and its associated label, during verifier training. At the inference step, we validate the generated texts through our verifier, to filter out potentially incorrect generations due to retrieval or generation failures, to prevent the generation of texts with inaccurate information. Note that there exists a concurrent work~\cite{LLM-Augmenter} that proposes to check whether the generated answers from LMs are grounded in the knowledge provided to LMs, by using API calls to proprietary LLMs or a heuristic measure (F1). However, this work clearly differs from our method, since we further verify the relevance of the retrieved knowledge in addition to the answer groundedness, through instruction-finetuning of LMs.&#10;&#10;In addition, we further propose refining the output from knowledge-augmented LMs if our verifier identifies the error in either the knowledge retrieval or the knowledge reflection. Specifically, we repeat the answer generation process until the model retrieves the knowledge relevant to the given question and incorporates the correctly retrieved knowledge into the generated answer, based on the verifier outcome. Also, since detecting errors of knowledge-augmented LMs with a single instruction given to the verifier might be inaccurate, we further construct an ensemble over multiple outputs from different instructions with a single verifier. Notably, one extra advantage of our verifier is that it is a plug-and-play module that works with any public or proprietary LMs, since we only require input-output pairs of LMs for verification without any architectural changes. We refer to our proposed method as Knowledge-Augmented Language Model Verification (KALMV).&#10;&#10;We experimentally validate the effectiveness of our KALMV on two different Question Answering (QA) tasks, namely open-domain QA and knowledge graph QA. The experimental results show that our KALMV can effectively verify the failure cases of knowledge-augmented LMs in knowledge retrieval and answer generation steps, contributing to significant reduction of the hallucination. Also, further analyses demonstrate the effectiveness of our error-rectifying and ensemble strategies.&#10;&#10;Our findings and contributions are threefolds:&#10;\begin{itemize}[itemsep=0.0mm, parsep=1pt]&#10; \item We point out the underexplored challenges of knowledge-augmented LMs, which are retrieval of irrelevant knowledge and unfaithful knowledge grounding.&#10; \item We introduce a novel verifier that identifies whether the retrieved knowledge is relevant to the question and reflected in the answer, and further present useful strategies for rectifying incorrect answers as well as improving the effectiveness of the verifier via ensembling. &#10; \item We validate our KALMV on open-domain and knowledge graph question answering tasks, demonstrating its effectiveness in verifying the errors of knowledge-augmented LMs.&#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Knowledge Augmentation, Language Model Limitations, Error Verification, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1809.02156" label="1809.02156">
        <attvalues>
          <attvalue for="0" value="Object Hallucination in Image Captioning" />
          <attvalue for="1" value="  Despite continuously improving performance, contemporary image captioning&#10;models are prone to &quot;hallucinating&quot; objects that are not actually in a scene.&#10;One problem is that standard metrics only measure similarity to ground truth&#10;captions and may not fully capture image relevance. In this work, we propose a&#10;new image relevance metric to evaluate current models with veridical visual&#10;labels and assess their rate of object hallucination. We analyze how captioning&#10;model architectures and learning objectives contribute to object hallucination,&#10;explore when hallucination is likely due to image misclassification or language&#10;priors, and assess how well current sentence metrics capture object&#10;hallucination. We investigate these questions on the standard image captioning&#10;benchmark, MSCOCO, using a diverse set of models. Our analysis yields several&#10;interesting findings, including that models which score best on standard&#10;sentence metrics do not always have lower hallucination and that models which&#10;hallucinate more tend to make errors driven by language priors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.05115" label="2203.05115">
        <attvalues>
          <attvalue for="0" value="Internet-augmented language models through few-shot prompting for&#10;  open-domain question answering" />
          <attvalue for="1" value="  In this work, we aim to capitalize on the unique few-shot capabilities of&#10;large-scale language models (LSLMs) to overcome some of their challenges with&#10;respect to grounding to factual and up-to-date information. Motivated by&#10;semi-parametric language models (LMs), which ground their decisions in external&#10;retrieved evidence, we use few-shot prompting to learn to condition LMs on&#10;information returned from the web using Google Search, a broad and constantly&#10;updated knowledge source. Our approach does not involve fine-tuning or learning&#10;additional parameters, thus making it applicable to any LM, offering therefore&#10;a strong baseline. Indeed, we find that LMs conditioned on the web surpass&#10;performance of closed-book models of similar, or even larger, model sizes in&#10;open-domain question answering. Finally, we find that increasing the&#10;inference-time compute of models, achieved via using multiple retrieved&#10;evidences to generate multiple answers followed by a reranking stage that uses&#10;scores generated by the same LMs, leads to better performance and alleviates&#10;lower performance of smaller few-shot LMs. All in all, our findings suggest&#10;that it might be beneficial to slow down the race towards the biggest model and&#10;instead shift attention towards finding more effective ways to use models,&#10;including but not limited to, better prompting or increasing inference-time&#10;compute.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.12813" label="2302.12813">
        <attvalues>
          <attvalue for="0" value="Check Your Facts and Try Again: Improving Large Language Models with&#10;  External Knowledge and Automated Feedback" />
          <attvalue for="1" value="  Large language models (LLMs), such as ChatGPT, are able to generate&#10;human-like, fluent responses for many downstream tasks, e.g., task-oriented&#10;dialog and question answering. However, applying LLMs to real-world,&#10;mission-critical applications remains challenging mainly due to their tendency&#10;to generate hallucinations and their inability to use external knowledge. This&#10;paper proposes a LLM-Augmenter system, which augments a black-box LLM with a&#10;set of plug-and-play modules. Our system makes the LLM generate responses&#10;grounded in external knowledge, e.g., stored in task-specific databases. It&#10;also iteratively revises LLM prompts to improve model responses using feedback&#10;generated by utility functions, e.g., the factuality score of a LLM-generated&#10;response. The effectiveness of LLM-Augmenter is empirically validated on two&#10;types of scenarios, task-oriented dialog and open-domain question answering.&#10;LLM-Augmenter significantly reduces ChatGPT's hallucinations without&#10;sacrificing the fluency and informativeness of its responses. We make the&#10;source code and models publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Numerous LLMs for text generation&#10;\cite{Radford2018} have been proposed over the years, including very competitive ones such as GPT-3 \cite{NEURIPS2020_1457c0d6,Ouyang2022TrainingLM},&#10;OPT \cite{Zhang2022OPTOP},&#10;GPT-j \cite{GPT-j}, and ChatGPT. However, most of them do not naturally incorporate external knowledge.&#10;To address this limitation, various works augment LLMs with knowledge consisting of \eg &#10;personalized recommendations \cite{ghazvininejad2017knowledge},&#10;Wikipedia article and web search \cite{dinan2018wizard,Shuster2022BlenderBot3A},&#10;structured and unstructured knowledge of task-oriented dialog \cite{Peng2022GODELLP}.&#10;Recent advances have focused on jointly finetuning the retriever and generation components of retrieval-augmented text generation systems&#10;\cite{Lewis2020RetrievalAugmentedGF,Zhang2021RetGenAJ}, but these methods are not applicable to black-box LLMs.&#10;&#10;More recent work attempts to combine black-box LLMs with external knowledge, such as incorporating external knowledge into prompts \cite{madaan2022memory,lazaridou2022internet}, &#10;making GPT-3 more faithful \cite{he2022rethinking}, &#10;and combining web knowledge with GPT-3 \cite{nakano2021webgpt}.&#10;In very recent works related to ours,&#10;\cite{Shi2023REPLUGRB} tune the ranker of a black-box LLM.&#10;\cite{Toolformer} tune black-box LLMs' access to different APIs and show improvement on a variety of understanding and reasoning tasks. &#10;We consider these works to complementary to ours, as we assume our set of APIs to be given and fixed, and we instead focus more on when and what APIs to request, interactive feedback with the LLM, and developing a self-learning ability through utility functions.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1909.11942" label="1909.11942">
        <attvalues>
          <attvalue for="0" value="ALBERT: A Lite BERT for Self-supervised Learning of Language&#10;  Representations" />
          <attvalue for="1" value="  Increasing model size when pretraining natural language representations often&#10;results in improved performance on downstream tasks. However, at some point&#10;further model increases become harder due to GPU/TPU memory limitations and&#10;longer training times. To address these problems, we present two&#10;parameter-reduction techniques to lower memory consumption and increase the&#10;training speed of BERT. Comprehensive empirical evidence shows that our&#10;proposed methods lead to models that scale much better compared to the original&#10;BERT. We also use a self-supervised loss that focuses on modeling&#10;inter-sentence coherence, and show it consistently helps downstream tasks with&#10;multi-sentence inputs. As a result, our best model establishes new&#10;state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having&#10;fewer parameters compared to BERT-large. The code and the pretrained models are&#10;available at https://github.com/google-research/ALBERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1901.11504" label="1901.11504">
        <attvalues>
          <attvalue for="0" value="Multi-Task Deep Neural Networks for Natural Language Understanding" />
          <attvalue for="1" value="  In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for&#10;learning representations across multiple natural language understanding (NLU)&#10;tasks. MT-DNN not only leverages large amounts of cross-task data, but also&#10;benefits from a regularization effect that leads to more general&#10;representations in order to adapt to new tasks and domains. MT-DNN extends the&#10;model proposed in Liu et al. (2015) by incorporating a pre-trained&#10;bidirectional transformer language model, known as BERT (Devlin et al., 2018).&#10;MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI,&#10;SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7%&#10;(2.2% absolute improvement). We also demonstrate using the SNLI and SciTail&#10;datasets that the representations learned by MT-DNN allow domain adaptation&#10;with substantially fewer in-domain labels than the pre-trained BERT&#10;representations. The code and pre-trained models are publicly available at&#10;https://github.com/namisan/mt-dnn.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1604.06174" label="1604.06174">
        <attvalues>
          <attvalue for="0" value="Training Deep Nets with Sublinear Memory Cost" />
          <attvalue for="1" value="  We propose a systematic approach to reduce the memory consumption of deep&#10;neural network training. Specifically, we design an algorithm that costs&#10;O(sqrt(n)) memory to train a n layer network, with only the computational cost&#10;of an extra forward pass per mini-batch. As many of the state-of-the-art models&#10;hit the upper bound of the GPU memory, our algorithm allows deeper and more&#10;complex models to be explored, and helps advance the innovations in deep&#10;learning research. We focus on reducing the memory cost to store the&#10;intermediate feature maps and gradients during training. Computation graph&#10;analysis is used for automatic in-place operation and memory sharing&#10;optimizations. We show that it is possible to trade computation for memory -&#10;giving a more memory efficient training algorithm with a little extra&#10;computation cost. In the extreme case, our analysis also shows that the memory&#10;consumption can be reduced to O(log n) with as little as O(n log n) extra cost&#10;for forward computation. Our experiments show that we can reduce the memory&#10;cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent&#10;additional running time cost on ImageNet problems. Similarly, significant&#10;memory cost reduction is observed in training complex recurrent neural networks&#10;on very long sequences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.10553" label="2112.10553">
        <attvalues>
          <attvalue for="0" value="Training dataset and dictionary sizes matter in BERT models: the case of&#10;  Baltic languages" />
          <attvalue for="1" value="  Large pretrained masked language models have become state-of-the-art&#10;solutions for many NLP problems. While studies have shown that monolingual&#10;models produce better results than multilingual models, the training datasets&#10;must be sufficiently large. We trained a trilingual LitLat BERT-like model for&#10;Lithuanian, Latvian, and English, and a monolingual Est-RoBERTa model for&#10;Estonian. We evaluate their performance on four downstream tasks: named entity&#10;recognition, dependency parsing, part-of-speech tagging, and word analogy. To&#10;analyze the importance of focusing on a single language and the importance of a&#10;large training set, we compare created models with existing monolingual and&#10;multilingual BERT models for Estonian, Latvian, and Lithuanian. The results&#10;show that the newly created LitLat BERT and Est-RoBERTa models improve the&#10;results of existing models on all tested tasks in most situations.&#10;" />
          <attvalue for="2" value="&#10;Large pretrained language models, based on transformers \cite{Vaswani2017} present the current state of the art in solving natural language processing tasks. These models are trained on large corpora in a self-supervised fashion and are able to learn basic language principles. Because of this and their ability to quickly and successfully adapt to a wide variety of tasks, they have also been (somewhat impetuously) called foundation models \cite{bommasani2021opportunities,marcus2021aifoundation}.&#10;&#10;The largest, most complex transformer based models have been trained only for the languages with most resources, mainly English, or in a massive multilingual fashion, covering 100 or more languages in a single model. Two examples of such models are GPT-3 \cite{Brown2020GPT3short}, and T5 \cite{raffel2020exploring}. For most languages, however, there are neither enough training data nor compute resources to train such complex models. Most monolingual models share the architecture of the BERT-base model \cite{Devlin2019}, e.g. RuBERT \cite{kuratov2019adaptation} for Russian, FinBERT \cite{virtanen2019multilingual} for Finnish, KB-BERT \cite{swedish-bert} for Swedish, or RobeCzech \cite{straka2021robeczech} for Czech.&#10;&#10;It has been shown \cite{virtanen2019multilingual} that monolingual models outperform massive multilingual models, like multilingual BERT (mBERT) \cite{Devlin2019} or XLM-RoBERTa (XLM-R) \cite{conneau2019unsupervised}. However, as we show, a good monolingual model needs to be trained on a sufficiently large corpus, and focusing on a single language alone is not sufficient for good performance. &#10;&#10;In this work, we introduce two new large models and make them publicly available. Est-RoBERTa is a monolingual Estonian model, while LitLat BERT is a trilingual model, trained on Lithuanian, Latvian, and English corpora. This enables us to compare these models with existing smaller monolingual models. We show that the size of training data matters and that when data is scarce, additional corpora from a similar language can aid in the model's performance. &#10;&#10;The paper is divided into five sections. In \ref{relatedWork}, we present related Lithuanian, Latvian, and Estonian BERT models. In \ref{sec:newmodels}, we present dataset and training for creation of new models. We evaluate the new models and compare them with existing ones in \ref{sec:evaluation}. We draw conclusions and present ideas for further work in \ref{conclusions}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Model Training, Model Performance Evaluation, Artificial Intelligence, Multilingual NLP" />
        </attvalues>
      </node>
      <node id="1912.07076" label="1912.07076">
        <attvalues>
          <attvalue for="0" value="Multilingual is not enough: BERT for Finnish" />
          <attvalue for="1" value="  Deep learning-based language models pretrained on large unannotated text&#10;corpora have been demonstrated to allow efficient transfer learning for natural&#10;language processing, with recent approaches such as the transformer-based BERT&#10;model advancing the state of the art across a variety of tasks. While most work&#10;on these models has focused on high-resource languages, in particular English,&#10;a number of recent efforts have introduced multilingual models that can be&#10;fine-tuned to address tasks in a large number of different languages. However,&#10;we still lack a thorough understanding of the capabilities of these models, in&#10;particular for lower-resourced languages. In this paper, we focus on Finnish&#10;and thoroughly evaluate the multilingual BERT model on a range of tasks,&#10;comparing it with a new Finnish BERT model trained from scratch. The new&#10;language-specific model is shown to systematically and clearly outperform the&#10;multilingual. While the multilingual model largely fails to reach the&#10;performance of previously proposed methods, the custom Finnish BERT model&#10;establishes new state-of-the-art results on all corpora for all reference&#10;tasks: part-of-speech tagging, named entity recognition, and dependency&#10;parsing. We release the model and all related resources created for this study&#10;with open licenses at https://turkunlp.org/finbert .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.01576" label="1611.01576">
        <attvalues>
          <attvalue for="0" value="Quasi-Recurrent Neural Networks" />
          <attvalue for="1" value="  Recurrent neural networks are a powerful tool for modeling sequential data,&#10;but the dependence of each timestep's computation on the previous timestep's&#10;output limits parallelism and makes RNNs unwieldy for very long sequences. We&#10;introduce quasi-recurrent neural networks (QRNNs), an approach to neural&#10;sequence modeling that alternates convolutional layers, which apply in parallel&#10;across timesteps, and a minimalist recurrent pooling function that applies in&#10;parallel across channels. Despite lacking trainable recurrent layers, stacked&#10;QRNNs have better predictive accuracy than stacked LSTMs of the same hidden&#10;size. Due to their increased parallelism, they are up to 16 times faster at&#10;train and test time. Experiments on language modeling, sentiment&#10;classification, and character-level neural machine translation demonstrate&#10;these advantages and underline the viability of QRNNs as a basic building block&#10;for a variety of sequence tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.10056" label="2009.10056">
        <attvalues>
          <attvalue for="0" value="Composed Variational Natural Language Generation for Few-shot Intents" />
          <attvalue for="1" value="  In this paper, we focus on generating training examples for few-shot intents&#10;in the realistic imbalanced scenario. To build connections between existing&#10;many-shot intents and few-shot intents, we consider an intent as a combination&#10;of a domain and an action, and propose a composed variational natural language&#10;generator (CLANG), a transformer-based conditional variational autoencoder.&#10;CLANG utilizes two latent variables to represent the utterances corresponding&#10;to two different independent parts (domain and action) in the intent, and the&#10;latent variables are composed together to generate natural examples.&#10;Additionally, to improve the generator learning, we adopt the contrastive&#10;regularization loss that contrasts the in-class with the out-of-class utterance&#10;generation given the intent. To evaluate the quality of the generated&#10;utterances, experiments are conducted on the generalized few-shot intent&#10;detection task. Empirical results show that our proposed model achieves&#10;state-of-the-art performances on two real-world intent detection datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Generative Data Augmentation for SLU&#10;Generative data augmentation methods alleviates the problem of lacking data by creating artificial training data with generation models. Recent works \cite{wei2019eda, malandrakis2019controlled, yoo2019data} have explored this idea for SLU tasks like intent detection. &#10;\cite{wei2019eda} provide data augmentation ability for natural language with simple language transformation rules like insert, delete and swap. &#10;\cite{malandrakis2019controlled} and \cite{yoo2019data} utilize variational autoencoders \cite{kingma2013auto} to generate training data for SLU tasks. \cite{malandrakis2019controlled} investigates templated-based text generation model to augment the training data for intelligent artificial agents. \cite{yoo2019data} generate fully annotated utterances to alleviate the data scarcity issue in spoken language understanding tasks. These models utilize LSTM as encoders \cite{hochreiter1997long} with limited model capacity. \cite{xia2020cg} provide the first work that combines CVAE with BERT to generate utterances for generalized few-shot intent detection.&#10;&#10;Recently, large-scale pre-trained language models are proposed for conditiaonal text generation tasks \cite{dathathri2019plug, keskar2019ctrl}, but they are only evaluated by human examination. They are not aiming at improving downstream classification tasks in low-resource conditions.&#10;&#10;Contrastive Learning in NLP &#10;Contrastive learning that learns the differences between the positive data from the negative examples has been widely used in NLP \cite{gutmann2010noise, mikolov2013distributed, 2019arXiv191103047C}. \cite{gutmann2010noise} leverage the Noise Contrastive Estimation (NCE) metric to discriminate the observed data from artificially generated noise samples.&#10;\cite{2019arXiv191103047C} introduce contrastive learning for multi-document question generation by generating questions closely related to the positive set but far away from the negative set. Different from previous works, our contrastive loss learn a positive example against a negative example together with label information.&#10;" />
          <attvalue for="4" value="Few-Shot Learning, Machine Learning, Computer Science, Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing, Intent Detection Models, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2106.07056" label="2106.07056">
        <attvalues>
          <attvalue for="0" value="Schema-Guided Paradigm for Zero-Shot Dialog" />
          <attvalue for="1" value="  Developing mechanisms that flexibly adapt dialog systems to unseen tasks and&#10;domains is a major challenge in dialog research. Neural models implicitly&#10;memorize task-specific dialog policies from the training data. We posit that&#10;this implicit memorization has precluded zero-shot transfer learning. To this&#10;end, we leverage the schema-guided paradigm, wherein the task-specific dialog&#10;policy is explicitly provided to the model. We introduce the Schema Attention&#10;Model (SAM) and improved schema representations for the STAR corpus. SAM&#10;obtains significant improvement in zero-shot settings, with a +22 F1 score&#10;improvement over prior work. These results validate the feasibility of&#10;zero-shot generalizability in dialog. Ablation experiments are also presented&#10;to demonstrate the efficacy of SAM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Neural Models, Artificial Intelligence, Natural Language Processing, Transfer Learning, Dialog Systems" />
        </attvalues>
      </node>
      <node id="2205.02022" label="2205.02022">
        <attvalues>
          <attvalue for="0" value="A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models&#10;  for African News Translation" />
          <attvalue for="1" value="  Recent advances in the pre-training of language models leverage large-scale&#10;datasets to create multilingual models. However, low-resource languages are&#10;mostly left out in these datasets. This is primarily because many widely spoken&#10;languages are not well represented on the web and therefore excluded from the&#10;large-scale crawls used to create datasets. Furthermore, downstream users of&#10;these models are restricted to the selection of languages originally chosen for&#10;pre-training. This work investigates how to optimally leverage existing&#10;pre-trained models to create low-resource translation systems for 16 African&#10;languages. We focus on two questions: 1) How can pre-trained models be used for&#10;languages not included in the initial pre-training? and 2) How can the&#10;resulting translation models effectively transfer to new domains? To answer&#10;these questions, we create a new African news corpus covering 16 languages, of&#10;which eight languages are not part of any existing evaluation dataset. We&#10;demonstrate that the most effective strategy for transferring both to&#10;additional languages and to additional domains is to fine-tune large&#10;pre-trained models on small quantities of high-quality translation data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;African MT Datasets.&#10;One of the major challenges of developing MT models for African languages is lack of data. There are many attempts to automatically crawl and align sentences from the web&#10;~\cite{schwenk-etal-2021-wikimatrix,schwenk-etal-2021-ccmatrix}. Nevertheless, the resulting corpora for many African languages are typically small and of poor quality~\cite{Kreutzer2021QualityAA}. Other cleaner parallel sources are mostly from religious sources, like the Bible covering over 1600 languages~\cite{mccarthy-etal-2020-johns} and JW300~\cite{agic-vulic-2019-jw300} from JW.org with over 343 languages, including over 100 African languages. Apart from the training dataset, evaluation datasets are needed to test the performance of multilingual MT models. The FLORES-101~\cite{Goyal2021TheFE} evaluation set, sourced from Wikipedia and manually translated, covers the largest number of languages, &#10;including 20 African languages. Finally, while other evaluation datasets for translating into or from African languages have been developed~\cite{Siminyu2021AI4DA,emezue-dossou-2020-ffr,Azunre2021EnglishTwiPC,Nyoni2021LowResourceNM,Gezmu2021ExtendedPC,Ali2021TowardsAP}, unfortunately there are only a few African languages with evaluation datasets in the news domain~\cite{adelani-etal-2021-effect,rooweither_mabuya_2021_5035171,Ezeani2020IgboEnglishMT} but ours covers 11 African languages (\S\ref{sec:corpus}). &#10;&#10;Low-resource MT. &#10;Interest in low-resource MT has been increasing both within the MT research community~\cite{Haddow2021SurveyOL}, as well as in native speaker communities~\cite{nekoto_etal_2020_participatory,azunre2021nlp,mager-etal-2021-findings}. On the modeling side, many techniques have been developed: unsupervised MT~\cite{lample2018unsupervised} leverages monolingual data, single multilingual models capable of translating between many languages~\cite{firat-etal-2016-multi,johnson-etal-2017-googles,aharoni-etal-2019-massively,JMLR_beyond_english}, &#10;multilingual unsupervised models leverage a related language (with parallel data) to assist translating the low-resource language that might not even have any monolingual data~\cite{ko-etal-2021-adapting}. &#10;Unfortunately, unsupervised MT typically performs poorly on low-resource languages~\cite{marchisio-etal-2020-unsupervised}. &#10;&#10;Transfer learning from high-resource languages has achieved more promising results:&#10;Transfer from multilingual pre-trained language models (PLM), like mBART50~\cite{Tang2020MultilingualTW} and MT5~\cite{xue-etal-2021-MT5}, and large-scale multilingual MT often outperforms bilingual MT&#10;~\cite{tran2021facebook,Yang2021MultilingualMT}. &#10;For low-resource languages this strategy outperforms the baseline (Transformer) models~\cite{birch-etal-2021-surprise,adelani-etal-2021-effect,Lee2022PreTrainedMS}. The performance can be further improved by large scale pre-training~\cite{reid-etal-2021-afromt,emezue-dossou-2021-mmtafrica}. &#10;&#10;" />
          <attvalue for="4" value="Low-Resource Language Translation, Computer Science, Linguistics, Language Model Pre-training, Artificial Intelligence, Multilingual Model Transfer, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2408.04905" label="2408.04905">
        <attvalues>
          <attvalue for="0" value="GlitchProber: Advancing Effective Detection and Mitigation of Glitch&#10;  Tokens in Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have achieved unprecedented success in the field&#10;of natural language processing. However, the black-box nature of their internal&#10;mechanisms has brought many concerns about their trustworthiness and&#10;interpretability. Recent research has discovered a class of abnormal tokens in&#10;the model's vocabulary space and named them &quot;glitch tokens&quot;. Those tokens, once&#10;included in the input, may induce the model to produce incorrect, irrelevant,&#10;or even harmful results, drastically undermining the reliability and&#10;practicality of LLMs.&#10;  In this work, we aim to enhance the understanding of glitch tokens and&#10;propose techniques for their detection and mitigation. We first reveal the&#10;characteristic features induced by glitch tokens on LLMs, which are evidenced&#10;by significant deviations in the distributions of attention patterns and&#10;dynamic information from intermediate model layers. Based on the insights, we&#10;develop GlitchProber, a tool for efficient glitch token detection and&#10;mitigation. GlitchProber utilizes small-scale sampling, principal component&#10;analysis for accelerated feature extraction, and a simple classifier for&#10;efficient vocabulary screening. Taking one step further, GlitchProber rectifies&#10;abnormal model intermediate layer values to mitigate the destructive effects of&#10;glitch tokens. Evaluated on five mainstream open-source LLMs, GlitchProber&#10;demonstrates higher efficiency, precision, and recall compared to existing&#10;approaches, with an average F1 score of 0.86 and an average repair rate of&#10;50.06%. GlitchProber unveils a novel path to address the challenges posed by&#10;glitch tokens and inspires future research toward more robust and interpretable&#10;LLMs.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the field of Natural Language Processing~(NLP), large language models~(LLMs) like GPT-4~\cite{achiam2023gpt}, Gemini~\cite{team2023gemini, reid2024gemini}, and Claude 3~\cite{anthropic2024claude3} have demonstrated near-human-level text generation capabilities. Their exceptional performance has led to widespread adoption~\cite{wu2023autogen,topsakal2023creating, 10.1145/3650212.3680383}. When using these models, users provide a prompt, which the model's tokenizer breaks down into a series of discrete tokens. These tokens are the fundamental units of information processing for the model, playing a crucial role in the usage of LLMs. Recent research~\cite{SolidGoldMagikarp2023, SolidGoldMagikarpII2023, SolidGoldMagikarpIII2023, geiping2024coercing, petertodd_phenomenon2023, 2023_Glitch_Tokens}, however, has shown that some ``glitch tokens'' exist in the vocabulary of LLMs. Once included in a prompt, these special tokens can potentially lead to model errors, such as misunderstanding user intent, refusing to answer, or generating irrelevant or harmful text. Therefore, thorough analysis and detection of these glitch tokens are crucial to ensure the reliability and safety of LLMs.&#10;&#10;To tackle the glitch tokens issue, one notable method recently is presented by Li et al.~\cite{li2024glitch}. This typical and intuitive solution involves studying the characteristics of glitch tokens in the word embedding space of LLMs and accordingly developing detection techniques. They discovered that glitch tokens tend to cluster in the embedding space and proposed an iterative clustering-based technique called GlitchHunter for efficient glitch token detection.&#10;&#10;Although there has been progress in detecting glitch tokens, there still lacks an efficient and precise detection of glitch tokens universally applicable in different LLMs. Furthermore, existing approaches primarily focus on detection, however, how to fix the issues caused by glitch tokens in the usage of LLMs remains an open question. Several limitations contribute to the aforementioned challenges:&#10;\begin{enumerate}[left=0pt]&#10; \item The exhaustive search method of checking vocabulary is simple and intuitive but incurs significant time costs with large token sets, making it inefficient for practical use.&#10; \item Existing detection methods primarily identify glitch tokens based on features like word frequency and word vectors. However, these features do not deeply explore the mechanisms by which glitch tokens impact model behaviors, resulting in poor detection accuracy and generalization performance.&#10; \item Current research primarily focuses on detecting glitch tokens rather than how to fix them. While detection can identify issues, it does not eliminate the negative impact of glitch tokens on model performance, limiting its practical value.&#10;\end{enumerate}&#10;&#10;Our work. To address these existing challenges and bridge the gap, in this work, we investigate the internal structure of LLMs to explore the differences between glitch tokens and normal tokens. &#10;Specifically, through empirical study, we discovered significant differences between glitch tokens and normal tokens in terms of the attention patterns and dynamic information of multi-layer perceptron (MLP) modules within transformer-based LLMs. &#10;This discovery reveals the adversarial impact of glitch tokens on the internal mechanisms of the model, indicating that glitch tokens introduce abnormal interference and noise to neural networks. &#10;This hinders the model from correctly understanding and processing the semantic information carried by these tokens, ultimately leading to erroneous outputs.&#10;&#10;From these findings, we gain an insight that the glitch tokens can be efficiently detected due to the deviated distributions of intermediate layers' outputs caused by them, and accordingly their impact can be effectively mitigated by proactively rectifying those abnormal outputs.&#10;Based on this insight, we propose a new method for glitch token detection and fix called \tool. &#10;For glitch token detection, \tool first samples a small subset of manually labeled glitch tokens as the sample set, and extracts the outputs of these tokens from the intermediate layers, specifically, the attention scores of the attention patterns and MLP status.&#10;It then applies Principal Component Analysis (PCA)~\cite{abdi2010PCA} dimensionality reduction to the intermediate layers' outputs and obtains a feature representation matrix for the sample set.&#10;This matrix, along with the corresponding class labels, is used to train a Support Vector Machine (SVM)~\cite{cortes1995support} classifier, which can subsequently be employed for glitch token detection. &#10;To fix the glitch tokens, \tool analyzes the activation value range of normal tokens in the intermediate MLP status and rectifies the activation states of glitch tokens. Specifically, it aims to adjust the activation patterns of glitch tokens to be closer to those of normal tokens, and thereby minimize their impact on the model's output. Our work is published on our website~\cite{Ours}.&#10;&#10;Contributions. We summarize our key contributions as follows:&#10;\begin{itemize}[left=0pt]&#10; \item Empirical Study Exploring the Internal Impact of Glitch Tokens on LLMs. We conduct a comprehensive and systematic empirical study on how glitch tokens and normal tokens manifest at the structural level across different LLMs. One of our key findings is that glitch tokens can trigger abnormal values in a model's attention patterns and MLP status.&#10; \item Effective Glitch Token Detection. Our evaluation on five representative open source LLMs demonstrates that \tool can save approximately 40\% of time in glitch token detection compared to the state-of-the-art approaches. Additionally, \tool exhibits a significant improvement in detection accuracy. &#10; \item Effective Glitch Token Fixing. &#10; In terms of fix, \tool successfully repairs an average of 7,758 tokens across the five LLMs. It achieves an average repair rate of 50.06\%, significantly outperforming the baseline approach. Our results demonstrate the effectiveness of the proposed fix strategy by adjusting the intermediate values of glitch tokens in the intermediate layers.&#10;\end{itemize}&#10;&#10;Ethical Consideration. &#10;In this work, we recognize that glitch tokens can cause abnormal responses from LLMs, potentially affecting their usage. However, we strictly adhere to ethical principles and do not condone any abuse or exploitation of these findings. Our research aims to raise awareness of these risks and contribute to a more secure LLM community. We have reported our findings to the respective LLM developers and are committed to collaborating with them to develop effective defenses and mitigation strategies. By working cooperatively, we promote responsible research practices and ensure the safe and beneficial use of LLMs.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Interpretability, Glitch Token Detection, Language Model Reliability, Machine Learning, Computer Science, Linguistics, Mathematics" />
        </attvalues>
      </node>
      <node id="2404.09894" label="2404.09894">
        <attvalues>
          <attvalue for="0" value="Glitch Tokens in Large Language Models: Categorization Taxonomy and&#10;  Effective Detection" />
          <attvalue for="1" value="  With the expanding application of Large Language Models (LLMs) in various&#10;domains, it becomes imperative to comprehensively investigate their unforeseen&#10;behaviors and consequent outcomes. In this study, we introduce and&#10;systematically explore the phenomenon of &quot;glitch tokens&quot;, which are anomalous&#10;tokens produced by established tokenizers and could potentially compromise the&#10;models' quality of response. Specifically, we experiment on seven top popular&#10;LLMs utilizing three distinct tokenizers and involving a totally of 182,517&#10;tokens. We present categorizations of the identified glitch tokens and symptoms&#10;exhibited by LLMs when interacting with glitch tokens. Based on our observation&#10;that glitch tokens tend to cluster in the embedding space, we propose&#10;GlitchHunter, a novel iterative clustering-based technique, for efficient&#10;glitch token detection. The evaluation shows that our approach notably&#10;outperforms three baseline methods on eight open-source LLMs. To the best of&#10;our knowledge, we present the first comprehensive study on glitch tokens. Our&#10;new detection further provides valuable insights into mitigating&#10;tokenization-related errors in LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;In this section, we discuss LLMs with a focus on tokenization techniques used in these models. We further explain the concept of ``glitch tokens'', which leads to unexpected behaviors in LLMs. To underscore the importance of this issue, we provide a real-world example of a glitch token encountered during our research.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.15582" label="2305.15582">
        <attvalues>
          <attvalue for="0" value="Balancing Effect of Training Dataset Distribution of Multiple Styles for&#10;  Multi-Style Text Transfer" />
          <attvalue for="1" value="  Text style transfer is an exciting task within the field of natural language&#10;generation that is often plagued by the need for high-quality paired datasets.&#10;Furthermore, training a model for multi-attribute text style transfer requires&#10;datasets with sufficient support across all combinations of the considered&#10;stylistic attributes, adding to the challenges of training a style transfer&#10;model. This paper explores the impact of training data input diversity on the&#10;quality of the generated text from the multi-style transfer model. We construct&#10;a pseudo-parallel dataset by devising heuristics to adjust the style&#10;distribution in the training samples. We balance our training dataset using&#10;marginal and joint distributions to train our style transfer models. We observe&#10;that a balanced dataset produces more effective control effects over multiple&#10;styles than an imbalanced or skewed one. Through quantitative analysis, we&#10;explore the impact of multiple style distributions in training data on&#10;style-transferred output. These findings will better inform the design of&#10;style-transfer datasets.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multi-style text transfer is a challenging task today with applications such as automatic domain-appropriate, style-conformant writing \cite{fu2018style} and AI-assisted stylistic language editing. Text style transfer is an intricate task as all language has a specific context, and those contexts influence the attributes of the language \cite{hovy2021importance}. Text style transfer is challenging because it involves dealing with the aspects of style coupled with the textual content \cite{hu2017toward, shen2017style, lample2018multiple}. This domain's other obstacles include the need for parallel corpus \cite{jhamtani2017shakespearizing} and quality training data. As the number of style dimensions increases with multi-style text transfer, not only is the requirement of a jointly annotated corpus across all the stylistic dimensions problematic, but the different styles are not necessarily independent.\\&#10;&#10;While ``style'' can also refer to authorial or domain-specific style, in this paper, we focus on ``micro-styles'' as defined by \cite{kang2021style} where they define ``micro-style'' as a complex combination of different factors such as formality markers, emotions, and metaphors. People intentionally \cite{troiano2021theories} tune these styles in writing differently based on their mood, the person they are addressing, the content of the message, or the platform. Multiple micro-styles can jointly describe a text; for example, a given text could simultaneously be formal and sad. Micro-styles also more easily lend themselves to being represented as spectra with varying degrees of intensity. These points align with our vision of an application where users can edit micro-style aspects of their writing.&#10;&#10;Much research exists on models implementing multi-style text transfer and interdependency of micro-styles \cite{kang2019xslue, goyal2020multi, subramanian2018multiple}. However, there needs to be more exploration of the joint distribution of inherent micro-styles in the style transfer training dataset and how these micro-style distributions are related. Therefore, we pose a question - Can a dataset with minimal variance across multiple micro-style combinations, such that it experiences a ``balancing effect'', lead to a better style transferred output ? Figure~\ref{fig:fig1} illustrates our intuition that a dataset that experiences a ``balancing effect'' will have more control over the multi-style transferred output than a ``skewed'' dataset. Suppose the style transfer model sees examples of every style combination that can exist - this could aid in the style generation of even unlikely combinations of styles compared to a skewed distribution of these joint micro-styles.&#10;&#10;In this research, we consider a multi-style text style transfer pipeline assuming that the user has no access to parallel data or the style of the original text that he wishes to transfer, as would seem natural for a style language editing application. We introduce the changing of the training dataset micro-style joint distributions in such a pipeline and quantitatively explore the impact of this modification on the style transferred output. We perform a set of empirical analyses to demonstrate the influence of joint distributions on style-transferred output and show how this trend varies as the number of micro-styles considered changes. The `balancing effect' on a training dataset leads to style transferred sentences from even the joint style combinations that are typically rare (``informal unbiased and unaroused''). Our study is the first of its kind on the distribution of micro styles in training datasets for multi-style text style transfer and is likely to have implications for designing datasets for multi-style transfer model training and fall within the context of and align with recent work on characterizing datasets and factors impacting style transfer \cite{bender2018data,schoch2021contextualizing, li2019domain, zhang2020parallel, gururangan2018annotation}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Dataset Balancing, Linguistics, Natural Language Generation, Artificial Intelligence, Natural Language Processing, Statistics, Text Style Transfer" />
        </attvalues>
      </node>
      <node id="1707.01161" label="1707.01161">
        <attvalues>
          <attvalue for="0" value="Shakespearizing Modern Language Using Copy-Enriched Sequence-to-Sequence&#10;  Models" />
          <attvalue for="1" value="  Variations in writing styles are commonly used to adapt the content to a&#10;specific context, audience, or purpose. However, applying stylistic variations&#10;is still by and large a manual process, and there have been little efforts&#10;towards automating it. In this paper we explore automated methods to transform&#10;text from modern English to Shakespearean English using an end to end trainable&#10;neural model with pointers to enable copy action. To tackle limited amount of&#10;parallel data, we pre-train embeddings of words by leveraging external&#10;dictionaries mapping Shakespearean words to modern English words as well as&#10;additional text. Our methods are able to get a BLEU score of 31+, an&#10;improvement of ~6 points above the strongest baseline. We publicly release our&#10;code to foster further research in this area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.15871" label="2110.15871">
        <attvalues>
          <attvalue for="0" value="From Theories on Styles to their Transfer in Text: Bridging the Gap with&#10;  a Hierarchical Survey" />
          <attvalue for="1" value="  Humans are naturally endowed with the ability to write in a particular style.&#10;They can, for instance, re-phrase a formal letter in an informal way, convey a&#10;literal message with the use of figures of speech or edit a novel by mimicking&#10;the style of some well-known authors. Automating this form of creativity&#10;constitutes the goal of style transfer. As a natural language generation task,&#10;style transfer aims at rewriting existing texts, and specifically, it creates&#10;paraphrases that exhibit some desired stylistic attributes. From a practical&#10;perspective, it envisions beneficial applications, like chatbots that modulate&#10;their communicative style to appear empathetic, or systems that automatically&#10;simplify technical articles for a non-expert audience. Several style-aware&#10;paraphrasing methods have attempted to tackle style transfer. A handful of&#10;surveys give a methodological overview of the field, but they do not support&#10;researchers to focus on specific styles. With this paper, we aim at providing a&#10;comprehensive discussion of the styles that have received attention in the&#10;transfer task. We organize them in a hierarchy, highlighting the challenges for&#10;the definition of each of them, and pointing out gaps in the current research&#10;landscape. The hierarchy comprises two main groups. One encompasses styles that&#10;people modulate arbitrarily, along the lines of registers and genres. The other&#10;group corresponds to unintentionally expressed styles, due to an author's&#10;personal characteristics. Hence, our review shows how these groups relate to&#10;one another, and where specific styles, including some that have not yet been&#10;explored, belong in the hierarchy. Moreover, we summarize the methods employed&#10;for different stylistic families, hinting researchers towards those that would&#10;be the most fitting for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.07877" label="2005.07877">
        <attvalues>
          <attvalue for="0" value="MicroNet for Efficient Language Modeling" />
          <attvalue for="1" value="  It is important to design compact language models for efficient deployment.&#10;We improve upon recent advances in both the language modeling domain and the&#10;model-compression domain to construct parameter and computation efficient&#10;language models. We use an efficient transformer-based architecture with&#10;adaptive embedding and softmax, differentiable non-parametric cache, Hebbian&#10;softmax, knowledge distillation, network pruning, and low-bit quantization. In&#10;this paper, we provide the winning solution to the NeurIPS 2019 MicroNet&#10;Challenge in the language modeling track. Compared to the baseline language&#10;model provided by the MicroNet Challenge, our model is 90 times more&#10;parameter-efficient and 36 times more computation-efficient while achieving the&#10;required test perplexity of 35 on the Wikitext-103 dataset. We hope that this&#10;work will aid future research into efficient language models, and we have&#10;released our full source code at&#10;https://github.com/mit-han-lab/neurips-micronet.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language modeling has been one of the most commonly studied sequence modeling tasks and one of the most studied tasks in the natural language processing community. Within the past few years, numerous works have improved the state-of-the-art language modeling results with deep neural network (DNN)-based sequence models. The Long Short-Term Memory (LSTM) network \cite{lstm} was designed to model dependencies in sequential data, and has successfully been applied to language modeling in a series of works, including the AWD-LSTM \cite{awdlstm}. Nevertheless, the LSTM architecture suffers from the vanishing gradient effect, limiting its ability to model long-term dependencies. On the other hand, the development of the Transformer attention architecture \cite{transformer} inspired new Transformer-based language models, such as Transformer-XL \cite{transformerxl}, which achieved new state-of-the-art in language modeling benchmarks. In addition, several works have developed architecture-agnostic enhancements such as adaptive embedding and adaptive softmax \cite{adaptive}, non-parametric cache \cite{cache}, Hebbian softmax \cite{lstmhebbiancache}, and dynamic evaluation \cite{dynamiceval}. While these works focused on improving the predictive accuracy of language models, relatively few works have optimized for parameter and computational efficiency, which are critical for tasks with hardware constraints.&#10;&#10;On the other hand, there have been numerous advances in parameter and computation efficient neural network architectures and model-compression techniques for deep neural networks. SqueezeNet \cite{squeezenet} and MobileNet \cite{mobilenet} are efficient convolutional neural networks that take advantage of 1x1 convolutions and depthwise separable convolutions, respectively. \cite{tensorizedtransformer} applies Block-term Tensor Decomposition \cite{btd} to compress transformer-based language models. Architecture agnostic model compression techniques such as knowledge distillation \cite{distill}, network pruning \cite{deepcompression}, and trained quantization \cite{deepcompression} are commonly used techniques to either increase the predictivity of neural networks or decrease the model size. There are also recent efforts focusing on automatically designing efficient models \cite{amc, so2019evolved, apq} and designing specialized accelerators to process the compressed models \cite{scnn, sparch}. Nevertheless, these techniques are mostly developed on convolutional neural networks and computer vision tasks, and only a handful of recent works like DistilBERT \cite{distilbert} and HAT \cite{hat} applied them to natural language tasks.&#10;&#10;In this work, we integrate advances in both the language modeling domain and the model-compression domain to construct parameter- and computation-efficient language models. Specifically, we evaluated our model with the criteria of the NeurIPS 2019 MicroNet Challenge \cite{micronet}. Compared to the baseline language model provided by the MicroNet Challenge, our model is 90 times more parameter-efficient and 36 times more computation-efficient while maintaining good performance. Our entry into the MicroNet Challenge achieved the top performance in parameter- and computation-efficiency in the language modeling track.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Efficient Deployment, Linguistics, Cognitive Science, Model Compression, Mathematics, Language Modeling" />
        </attvalues>
      </node>
      <node id="1803.10049" label="1803.10049">
        <attvalues>
          <attvalue for="0" value="Fast Parametric Learning with Activation Memorization" />
          <attvalue for="1" value="  Neural networks trained with backpropagation often struggle to identify&#10;classes that have been observed a small number of times. In applications where&#10;most class labels are rare, such as language modelling, this can become a&#10;performance bottleneck. One potential remedy is to augment the network with a&#10;fast-learning non-parametric model which stores recent activations and class&#10;labels into an external memory. We explore a simplified architecture where we&#10;treat a subset of the model parameters as fast memory stores. This can help&#10;retain information over longer time intervals than a traditional memory, and&#10;does not require additional space or compute. In the case of image&#10;classification, we display faster binding of novel classes on an Omniglot image&#10;curriculum task. We also show improved performance for word-based language&#10;models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia&#10;articles (WikiText-103) --- the latter achieving a state-of-the-art perplexity&#10;of 29.2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.09777" label="1906.09777">
        <attvalues>
          <attvalue for="0" value="A Tensorized Transformer for Language Modeling" />
          <attvalue for="1" value="  Latest development of neural models has connected the encoder and decoder&#10;through a self-attention mechanism. In particular, Transformer, which is solely&#10;based on self-attention, has led to breakthroughs in Natural Language&#10;Processing (NLP) tasks. However, the multi-head attention mechanism, as a key&#10;component of Transformer, limits the effective deployment of the model to a&#10;resource-limited setting. In this paper, based on the ideas of tensor&#10;decomposition and parameters sharing, we propose a novel self-attention model&#10;(namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We&#10;test and verify the proposed attention method on three language modeling tasks&#10;(i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task&#10;(i.e., WMT-2016 English-German). Multi-linear attention can not only largely&#10;compress the model parameters but also obtain performance improvements,&#10;compared with a number of language modeling approaches, such as Transformer,&#10;Transformer-XL, and Transformer with tensor train decomposition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1510.00149" label="1510.00149">
        <attvalues>
          <attvalue for="0" value="Deep Compression: Compressing Deep Neural Networks with Pruning, Trained&#10;  Quantization and Huffman Coding" />
          <attvalue for="1" value="  Neural networks are both computationally intensive and memory intensive,&#10;making them difficult to deploy on embedded systems with limited hardware&#10;resources. To address this limitation, we introduce &quot;deep compression&quot;, a three&#10;stage pipeline: pruning, trained quantization and Huffman coding, that work&#10;together to reduce the storage requirement of neural networks by 35x to 49x&#10;without affecting their accuracy. Our method first prunes the network by&#10;learning only the important connections. Next, we quantize the weights to&#10;enforce weight sharing, finally, we apply Huffman coding. After the first two&#10;steps we retrain the network to fine tune the remaining connections and the&#10;quantized centroids. Pruning, reduces the number of connections by 9x to 13x;&#10;Quantization then reduces the number of bits that represent each connection&#10;from 32 to 5. On the ImageNet dataset, our method reduced the storage required&#10;by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method&#10;reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of&#10;accuracy. This allows fitting the model into on-chip SRAM cache rather than&#10;off-chip DRAM memory. Our compression method also facilitates the use of&#10;complex neural networks in mobile applications where application size and&#10;download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU,&#10;compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy&#10;efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.03494" label="1802.03494">
        <attvalues>
          <attvalue for="0" value="AMC: AutoML for Model Compression and Acceleration on Mobile Devices" />
          <attvalue for="1" value="  Model compression is a critical technique to efficiently deploy neural&#10;network models on mobile devices which have limited computation resources and&#10;tight power budgets. Conventional model compression techniques rely on&#10;hand-crafted heuristics and rule-based policies that require domain experts to&#10;explore the large design space trading off among model size, speed, and&#10;accuracy, which is usually sub-optimal and time-consuming. In this paper, we&#10;propose AutoML for Model Compression (AMC) which leverage reinforcement&#10;learning to provide the model compression policy. This learning-based&#10;compression policy outperforms conventional rule-based compression policy by&#10;having higher compression ratio, better preserving the accuracy and freeing&#10;human labor. Under 4x FLOPs reduction, we achieved 2.7% better accuracy than&#10;the handcrafted model compression policy for VGG-16 on ImageNet. We applied&#10;this automated, push-the-button compression pipeline to MobileNet and achieved&#10;1.81x speedup of measured inference latency on an Android phone and 1.43x&#10;speedup on the Titan XP GPU, with only 0.1% loss of ImageNet Top-1 accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.04485" label="1708.04485">
        <attvalues>
          <attvalue for="0" value="SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks" />
          <attvalue for="1" value="  Convolutional Neural Networks (CNNs) have emerged as a fundamental technology&#10;for machine learning. High performance and extreme energy efficiency are&#10;critical for deployments of CNNs in a wide range of situations, especially&#10;mobile platforms such as autonomous vehicles, cameras, and electronic personal&#10;assistants. This paper introduces the Sparse CNN (SCNN) accelerator&#10;architecture, which improves performance and energy efficiency by exploiting&#10;the zero-valued weights that stem from network pruning during training and&#10;zero-valued activations that arise from the common ReLU operator applied during&#10;inference. Specifically, SCNN employs a novel dataflow that enables maintaining&#10;the sparse weights and activations in a compressed encoding, which eliminates&#10;unnecessary data transfers and reduces storage requirements. Furthermore, the&#10;SCNN dataflow facilitates efficient delivery of those weights and activations&#10;to the multiplier array, where they are extensively reused. In addition, the&#10;accumulation of multiplication products are performed in a novel accumulator&#10;array. Our results show that on contemporary neural networks, SCNN can improve&#10;both performance and energy by a factor of 2.7x and 2.3x, respectively, over a&#10;comparably provisioned dense CNN accelerator.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.14187" label="2005.14187">
        <attvalues>
          <attvalue for="0" value="HAT: Hardware-Aware Transformers for Efficient Natural Language&#10;  Processing" />
          <attvalue for="1" value="  Transformers are ubiquitous in Natural Language Processing (NLP) tasks, but&#10;they are difficult to be deployed on hardware due to the intensive computation.&#10;To enable low-latency inference on resource-constrained hardware platforms, we&#10;propose to design Hardware-Aware Transformers (HAT) with neural architecture&#10;search. We first construct a large design space with $\textit{arbitrary&#10;encoder-decoder attention}$ and $\textit{heterogeneous layers}$. Then we train&#10;a $\textit{SuperTransformer}$ that covers all candidates in the design space,&#10;and efficiently produces many $\textit{SubTransformers}$ with weight sharing.&#10;Finally, we perform an evolutionary search with a hardware latency constraint&#10;to find a specialized $\textit{SubTransformer}$ dedicated to run fast on the&#10;target hardware. Extensive experiments on four machine translation tasks&#10;demonstrate that HAT can discover efficient models for different hardware (CPU,&#10;GPU, IoT device). When running WMT'14 translation task on Raspberry Pi-4, HAT&#10;can achieve $\textbf{3}\times$ speedup, $\textbf{3.7}\times$ smaller size over&#10;baseline Transformer; $\textbf{2.7}\times$ speedup, $\textbf{3.6}\times$&#10;smaller size over Evolved Transformer with $\textbf{12,041}\times$ less search&#10;cost and no performance loss. HAT code is&#10;https://github.com/mit-han-lab/hardware-aware-transformers.git&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.08110" label="2204.08110">
        <attvalues>
          <attvalue for="0" value="Language Contamination Helps Explain the Cross-lingual Capabilities of&#10;  English Pretrained Models" />
          <attvalue for="1" value="  English pretrained language models, which make up the backbone of many modern&#10;NLP systems, require huge amounts of unlabeled training data. These models are&#10;generally presented as being trained only on English text but have been found&#10;to transfer surprisingly well to other languages. We investigate this&#10;phenomenon and find that common English pretraining corpora actually contain&#10;significant amounts of non-English text: even when less than 1% of data is not&#10;English (well within the error rate of strong language classifiers), this leads&#10;to hundreds of millions of foreign language tokens in large-scale datasets. We&#10;then demonstrate that even these small percentages of non-English data&#10;facilitate cross-lingual transfer for models trained on them, with target&#10;language performance strongly correlated to the amount of in-language data seen&#10;during pretraining. In light of these findings, we argue that no model is truly&#10;monolingual when pretrained at scale, which should be considered when&#10;evaluating cross-lingual transfer.&#10;" />
          <attvalue for="2" value="&#10;Pretrained language models have become an integral part of NLP systems. They come in two flavors: monolingual, where the model is trained on text from a single language, and multilingual, where the model is jointly trained on data from many different languages. Monolingual pretrained models are generally applied to tasks in the same language, whereas multilingual ones are used for cross-lingual tasks or transfer.&#10;&#10;Recent work has claimed that monolingual pretrained models are also surprisingly good at transferring between languages, despite ostensibly having never seen the target language before \cite[inter alia]{gogoulou2021cross, li2021cross}.&#10;However, because of the large scale of pretraining data and because many pretraining corpora are not publicly available, it is currently unknown how much foreign language data exists in monolingual pretraining corpora.&#10;In this paper, we show that (1) these data are almost certainly contaminated with very small percentages of text from other languages and that (2) cross-lingual transfer is possible from such data leakage in the pretraining corpus.&#10;&#10;More specifically, we quantify how multilingual English pretrained models are in two steps. First, we analyze common English pretraining corpora with a large-scale automatic evaluation to estimate their language composition, as well as a smaller-scale manual analysis. Second, we perform experiments across fifty languages on masked language modeling and part-of-speech (POS) tagging to measure how well the models trained on these pretraining corpora perform outside of English.&#10;&#10;Our analysis finds that these corpora include very small percentages that amount to overall significant amounts of non-English text (Figure \ref{fig:lang-comp-teaser}), particularly those derived from web-crawled data. &#10;Furthermore, the models trained on this data perform surprisingly well on other languages; this transfer is strongly correlated with the amount of target language data seen during pretraining. Notably, we find that the English T5 outperforms mBERT on POS tagging in multiple languages with no finetuning. &#10;&#10;Overall, these results indicate that the considered models are actually multilingual and that their ability to transfer across languages is not zero-shot, despite what has been recently claimed.&#10;Given the effort required to fully remove all non-English data, we question whether it is practically possible to train truly monolingual models at scale.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Training, Cross-Lingual Transfer, Artificial Intelligence, Multilingual Data Analysis, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2109.07348" label="2109.07348">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Transfer of Monolingual Models" />
          <attvalue for="1" value="  Recent studies in zero-shot cross-lingual learning using multilingual models&#10;have falsified the previous hypothesis that shared vocabulary and joint&#10;pre-training are the keys to cross-lingual generalization. Inspired by this&#10;advancement, we introduce a cross-lingual transfer method for monolingual&#10;models based on domain adaptation. We study the effects of such transfer from&#10;four different languages to English. Our experimental results on GLUE show that&#10;the transferred models outperform the native English model independently of the&#10;source language. After probing the English linguistic knowledge encoded in the&#10;representations before and after transfer, we find that semantic information is&#10;retained from the source language, while syntactic information is learned&#10;during transfer. Additionally, the results of evaluating the transferred models&#10;in source language tasks reveal that their performance in the source domain&#10;deteriorates after transfer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.05900" label="2302.05900">
        <attvalues>
          <attvalue for="0" value="Investigating the Effect of Relative Positional Embeddings on&#10;  AMR-to-Text Generation with Structural Adapters" />
          <attvalue for="1" value="  Text generation from Abstract Meaning Representation (AMR) has substantially&#10;benefited from the popularized Pretrained Language Models (PLMs). Myriad&#10;approaches have linearized the input graph as a sequence of tokens to fit the&#10;PLM tokenization requirements. Nevertheless, this transformation jeopardizes&#10;the structural integrity of the graph and is therefore detrimental to its&#10;resulting representation. To overcome this issue, Ribeiro et al. have recently&#10;proposed StructAdapt, a structure-aware adapter which injects the input graph&#10;connectivity within PLMs using Graph Neural Networks (GNNs). In this paper, we&#10;investigate the influence of Relative Position Embeddings (RPE) on AMR-to-Text,&#10;and, in parallel, we examine the robustness of StructAdapt. Through ablation&#10;studies, graph attack and link prediction, we reveal that RPE might be&#10;partially encoding input graphs. We suggest further research regarding the role&#10;of RPE will provide valuable insights for Graph-to-Text generation.&#10;" />
          <attvalue for="2" value="&#10;Earliest works on AMR-to-Text generation were mostly based on statistical methods. A common practice was to convert AMR-to-Text task into an already studied problems such as Tree-to-Text \cite{flanigan-etal-2016-generation, lampouras-vlachos-2017-sheffield}, aligned text-to-text \cite{pourdamghani-etal-2016-generating}, Travel Sales Problems \cite{song-etal-2016-amr} or Grammatical Framework \cite{grammatical-framework-book-ranta}. Recently, most methods are neural-centered with an encoder-decoder architecture \cite{sutskever-etal-2014-seq2seq} as a backbone \cite{konstas-etal-2017-neural, takase-etal-2016-neural, cao-clark-2019-factorising}. Unfortunately, this architecture coerces the AMR to be linearized as a sequence of tokens. This ends up in structural information loss. To tackle this issue, several strategies have attempted to integrate structure using message propagation \cite{song-etal-2018-graph, guo-etal-2019-densely, damonte-cohen-2019-structural, ribeiro-etal-2019-enhancing, zhang-etal-2020-lightweight, zhao-etal-2020-line}. A limitation of those is the absence of pretraining, as demonstrated by \cite{ribeiro-etal-2021-investigating}. To this end, \cite{ribeiro-etal-2021-structural} introduced StructAdapt for lightweight AMR-to-Text with structural adapters. As linearization and tokenization of the input graph are mandatory steps for PLMs, StructAdapt first defines a new graph where nodes are the resulting subwords from the tokenization. As a result, adapter can henceforth include GNN layers operating on the subsequent graph while leveraging pretrained representations.&#10;&#10;However, although studies have been made to probe position embeddings \cite{wang-chen-2020-position, wang-et-al-2021-APE-BERT,dufter-etal-2022-position}, their role on graph encoding has remained unanswered. &#10;In this paper, we are particularly interested in measuring the saliency of RPE with StructAdapt for AMR-to-Text generation. Our novelty is not in proposing a new method to encode graphs such as \cite{schmitt-etal-2021-modeling} but rather in revealing the interesting behaviours of RPE along with StructAdapt.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Text Generation, Artificial Intelligence, Mathematics, Graph Neural Networks, Natural Language Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="1609.07451" label="1609.07451">
        <attvalues>
          <attvalue for="0" value="AMR-to-text generation as a Traveling Salesman Problem" />
          <attvalue for="1" value="  The task of AMR-to-text generation is to generate grammatical text that&#10;sustains the semantic meaning for a given AMR graph. We at- tack the task by&#10;first partitioning the AMR graph into smaller fragments, and then generating&#10;the translation for each fragment, before finally deciding the order by solving&#10;an asymmetric generalized traveling salesman problem (AGTSP). A Maximum Entropy&#10;classifier is trained to estimate the traveling costs, and a TSP solver is used&#10;to find the optimized solution. The final model reports a BLEU score of 22.44&#10;on the SemEval-2016 Task8 dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.08381" label="1704.08381">
        <attvalues>
          <attvalue for="0" value="Neural AMR: Sequence-to-Sequence Models for Parsing and Generation" />
          <attvalue for="1" value="  Sequence-to-sequence models have shown strong performance across a broad&#10;range of applications. However, their application to parsing and generating&#10;text usingAbstract Meaning Representation (AMR)has been limited, due to the&#10;relatively limited amount of labeled data and the non-sequential nature of the&#10;AMR graphs. We present a novel training procedure that can lift this limitation&#10;using millions of unlabeled sentences and careful preprocessing of the AMR&#10;graphs. For AMR parsing, our model achieves competitive results of 62.1SMATCH,&#10;the current best score reported without significant use of external semantic&#10;resources. For AMR generation, our model establishes a new state-of-the-art&#10;performance of BLEU 33.8. We present extensive ablative and qualitative&#10;analysis including strong evidence that sequence-based AMR models are robust&#10;against ordering variations of graph-to-sequence conversions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.08426" label="2007.08426">
        <attvalues>
          <attvalue for="0" value="Investigating Pretrained Language Models for Graph-to-Text Generation" />
          <attvalue for="1" value="  Graph-to-text generation aims to generate fluent texts from graph-based data.&#10;In this paper, we investigate two recently proposed pretrained language models&#10;(PLMs) and analyze the impact of different task-adaptive pretraining strategies&#10;for PLMs in graph-to-text generation. We present a study across three graph&#10;domains: meaning representations, Wikipedia knowledge graphs (KGs) and&#10;scientific KGs. We show that the PLMs BART and T5 achieve new state-of-the-art&#10;results and that task-adaptive pretraining strategies improve their performance&#10;even further. In particular, we report new state-of-the-art BLEU scores of&#10;49.72 on LDC2017T10, 59.70 on WebNLG, and 25.66 on AGENDA datasets - a relative&#10;improvement of 31.8%, 4.5%, and 42.4%, respectively. In an extensive analysis,&#10;we identify possible reasons for the PLMs' success on graph-to-text tasks. We&#10;find evidence that their knowledge about true facts helps them perform well&#10;even when the input graph representation is reduced to a simple bag of node and&#10;edge labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Graph-to-text Learning. Various neural models have been proposed to generate sentences from graphs from different domains. \cite{konsas_17} propose the first neural approach for AMR-to-text generation that uses a linearized input graph. Prior approaches for KG-to-text generation train text-to-text neural models using sequences of KG triples as input \cite{trisedya-etal-2018-gtr, moryossef-etal-2019-step, castro-ferreira-etal-2019-neural, ribeiro2021smelting}. &#10;&#10;Recent approaches \cite{marcheggiani-icnl18, song-etal-acl2018, beck-etal-2018-acl2018, damonte-cohen-2019-structural, ribeiro-etal-2019-enhancing,zhao-etal-2020-bridging, schmitt-etal-2021-modeling, ribeiro2021structural} propose architectures based on GNNs to directly encode the graph structure, whereas other efforts \cite{ribeiro-etal-2020-modeling, schmitt2020modeling, yao-etal-2020-heterogeneous, doi:10.116200297} inject the graph structure information into Transformer-based architectures. The success of those approaches suggests that imposing a strong relational inductive bias into the graph-to-text model can assist the generation.&#10;&#10;Pretrained Language Models. Pretrained Transformer-based models, such as BERT \cite{devlin-etal-2019-bert}, XLNet \cite{NIPS2019_8812}, or RoBERTa \cite{liu2020roberta}, have established a qualitatively new level of baseline performance for many widely used natural language understanding (NLU) benchmarks. Generative pretrained Transformer-based methods, such as GPT-2 \cite{radford2019language}, BART \cite{lewis2019bart}, and T5 \cite{2019t5}, are employed in many natural language generation (NLG) tasks. &#10;&#10;\cite{mager2020gpttoo} were the first to employ GPT-2, a decoder-only PLM, for AMR-to-text generation and use cycle consistency to improve the adequacy. In contrast, we are the first to investigate BART and T5 models, which have both a Transformer-based encoder and decoder, in AMR-to-text generation. Recently, \cite{harkous2020text} and \cite{kale2020texttotext} demonstrate state-of-the-art results in different data-to-text datasets, employing GPT-2 and T5 models respectively. \cite{radev2020dart} propose DART, a new data-to-text dataset, and train a BART model gradually augmenting the WebNLG training data with DART data. &#10;&#10;\cite{hoyle2020promoting} explore scaffolding objectives in PLMs and show gains in low-resource graph-to-text settings.&#10;Different from the above works, we focus on a general transfer learning strategies for graph-to-text generation, investigating task-adaptive pretraining approaches, employing additional collected task-specific data for different PLMs (BART and T5) and benchmarks. In addition, we provide a detailed analysis aimed at explaining the good performance of PLMs on KG-to-text tasks.&#10;&#10;Recently, \cite{gururangan-etal-2020-dont} explored task-adaptive pretraining strategies for text classification.&#10;While our lma (see \S\ref{sec:finetuning}) is related to their dapt as both use a self-supervised objective on a domain-specific corpus,&#10;they notably differ in that dapt operates on the model input while lma models the output. We are the first to show the benefits of additional task-specific pretraining in PLMs for graph-to-text tasks.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2103.09120" label="2103.09120">
        <attvalues>
          <attvalue for="0" value="Structural Adapters in Pretrained Language Models for AMR-to-text&#10;  Generation" />
          <attvalue for="1" value="  Pretrained language models (PLM) have recently advanced graph-to-text&#10;generation, where the input graph is linearized into a sequence and fed into&#10;the PLM to obtain its representation. However, efficiently encoding the graph&#10;structure in PLMs is challenging because such models were pretrained on natural&#10;language, and modeling structured data may lead to catastrophic forgetting of&#10;distributional knowledge. In this paper, we propose StructAdapt, an adapter&#10;method to encode graph structure into PLMs. Contrary to prior work, StructAdapt&#10;effectively models interactions among the nodes based on the graph&#10;connectivity, only training graph structure-aware adapter parameters. In this&#10;way, we incorporate task-specific knowledge while maintaining the topological&#10;structure of the graph. We empirically show the benefits of explicitly encoding&#10;graph structure into PLMs using StructAdapt, outperforming the state of the art&#10;on two AMR-to-text datasets, training only 5.1% of the PLM parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.04903" label="2010.04903">
        <attvalues>
          <attvalue for="0" value="What Do Position Embeddings Learn? An Empirical Study of Pre-Trained&#10;  Language Model Positional Encoding" />
          <attvalue for="1" value="  In recent years, pre-trained Transformers have dominated the majority of NLP&#10;benchmark tasks. Many variants of pre-trained Transformers have kept breaking&#10;out, and most focus on designing different pre-training objectives or variants&#10;of self-attention. Embedding the position information in the self-attention&#10;mechanism is also an indispensable factor in Transformers however is often&#10;discussed at will. Therefore, this paper carries out an empirical study on&#10;position embeddings of mainstream pre-trained Transformers, which mainly&#10;focuses on two questions: 1) Do position embeddings really learn the meaning of&#10;positions? 2) How do these different learned position embeddings affect&#10;Transformers for NLP tasks? This paper focuses on providing a new insight of&#10;pre-trained position embeddings through feature-level analysis and empirical&#10;experiments on most of iconic NLP tasks. It is believed that our experimental&#10;results can guide the future work to choose the suitable positional encoding&#10;function for specific tasks given the application property.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.11090" label="2102.11090">
        <attvalues>
          <attvalue for="0" value="Position Information in Transformers: An Overview" />
          <attvalue for="1" value="  Transformers are arguably the main workhorse in recent Natural Language&#10;Processing research. By definition a Transformer is invariant with respect to&#10;reordering of the input. However, language is inherently sequential and word&#10;order is essential to the semantics and syntax of an utterance. In this&#10;article, we provide an overview and theoretical comparison of existing methods&#10;to incorporate position information into Transformer models. The objectives of&#10;this survey are to (1) showcase that position information in Transformer is a&#10;vibrant and extensive research area; (2) enable the reader to compare existing&#10;methods by providing a unified notation and systematization of different&#10;approaches along important model dimensions; (3) indicate what characteristics&#10;of an application should be taken into account when selecting a position&#10;encoding; (4) provide stimuli for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.09242" label="2006.09242">
        <attvalues>
          <attvalue for="0" value="Modeling Graph Structure via Relative Position for Text Generation from&#10;  Knowledge Graphs" />
          <attvalue for="1" value="  We present Graformer, a novel Transformer-based encoder-decoder architecture&#10;for graph-to-text generation. With our novel graph self-attention, the encoding&#10;of a node relies on all nodes in the input graph - not only direct neighbors -&#10;facilitating the detection of global patterns. We represent the relation&#10;between two nodes as the length of the shortest path between them. Graformer&#10;learns to weight these node-node relations differently for different attention&#10;heads, thus virtually learning differently connected views of the input graph.&#10;We evaluate Graformer on two popular graph-to-text generation benchmarks,&#10;AGENDA and WebNLG, where it achieves strong performance while using many fewer&#10;parameters than other approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.09204" label="2311.09204">
        <attvalues>
          <attvalue for="0" value="Fusion-Eval: Integrating Assistant Evaluators with LLMs" />
          <attvalue for="1" value="  Evaluating natural language systems poses significant challenges,&#10;particularly in the realms of natural language understanding and high-level&#10;reasoning. In this paper, we introduce 'Fusion-Eval', an innovative approach&#10;that leverages Large Language Models (LLMs) to integrate insights from various&#10;assistant evaluators. The LLM is given the example to evaluate along with&#10;scores from the assistant evaluators. Each of these evaluators specializes in&#10;assessing distinct aspects of responses. Fusion-Eval achieves a 0.962&#10;system-level Kendall-Tau correlation with humans on SummEval and a 0.744&#10;turn-level Spearman correlation on TopicalChat, which is significantly higher&#10;than baseline methods. These results highlight Fusion-Eval's significant&#10;potential in the realm of natural language system evaluation.&#10;" />
          <attvalue for="2" value="&#10;Evaluating the performance of natural language generation models has significant challenges~\cite{ouyang2022training}, particularly in terms of evaluation benchmarks and evaluation paradigms~\cite{wang2023aligning}. &#10;This study focuses on the latter one.&#10;Typically, the evaluation paradigms fall into three categories: human-based, automatic-metrics-based and model-based evaluations.&#10;Among these, human evaluations are regarded as the most reliable, yet they come with high costs and issues of scalability.&#10;&#10;Automatic metrics such as BLEU~\cite{papineni2002bleu} and ROUGE~\cite{lin2004rouge} are prevalent in evaluations, relying on comparisons with a `gold' standard reference. However, the creation of these gold references is a labor-intensive process. Furthermore, studies such as \cite{fabbri2021summeval} have demonstrated that these automatic metrics often do not correlate well with human judgment.&#10;&#10;Model-based evaluations aim to enhance the correlation with human judgment using neural networks fine-tuned on specific datasets. Neural evaluators like BLEURT~\cite{sellam2020bleurt} and its variant SMART~\cite{amplayo2022smart} show improved alignment with human assessments in various generative tasks. These models offer flexibility in evaluation methods. They can either compare the response to the source (reference-free), or to the gold standard (reference-dependent).&#10;&#10;Recent advancements have seen the use of Large Language Models (LLMs) as reference-free evaluators in Natural Language Generation (NLG) tasks. Notably, studies by \cite{fu2023gptscore, wang2023chatgpt} have leveraged LLMs to rate candidate outputs based on their generation probability alone, eliminating the need for reference text comparisons. Additionally, \cite{liu2023gpteval} introduced a method called G-Eval, where LLMs, guided by human-crafted evaluation criteria, score responses. Meta-evaluations indicate that these LLM-based evaluators reach a level of human correlation on par with medium-sized neural evaluators~\cite{zhong2022towards}.&#10;In light of these developments in evaluation paradigms, the following question arises:&#10;\begin{customquote}&#10;``Can Large Language Models (LLMs) integrate existing evaluators to achieve higher correlation with human judgments?''&#10;\end{customquote}&#10;&#10;In response to this question, we introduce Fusion-Eval, an innovative evaluation framework that integrates a variety of existing evaluators—termed assistant evaluators—to enhance correlation with human judgment. Fusion-Eval prompts an LLM with an example to evaluate and scores given by assistant evaluators. In our work, we consider reference free evaluation. Fusion-Eval can evaluate any natural language task where assistant evaluators are available. However, its effectiveness hinges on the quality of the assistant evaluators, making it more suitable for well-established text generation tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Evaluation Methodologies, Linguistics, Cognitive Science, Natural Language Understanding, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2007.12626" label="2007.12626">
        <attvalues>
          <attvalue for="0" value="SummEval: Re-evaluating Summarization Evaluation" />
          <attvalue for="1" value="  The scarcity of comprehensive up-to-date studies on evaluation metrics for&#10;text summarization and the lack of consensus regarding evaluation protocols&#10;continue to inhibit progress. We address the existing shortcomings of&#10;summarization evaluation methods along five dimensions: 1) we re-evaluate 14&#10;automatic evaluation metrics in a comprehensive and consistent fashion using&#10;neural summarization model outputs along with expert and crowd-sourced human&#10;annotations, 2) we consistently benchmark 23 recent summarization models using&#10;the aforementioned automatic evaluation metrics, 3) we assemble the largest&#10;collection of summaries generated by models trained on the CNN/DailyMail news&#10;dataset and share it in a unified format, 4) we implement and share a toolkit&#10;that provides an extensible and unified API for evaluating summarization models&#10;across a broad range of automatic metrics, 5) we assemble and share the largest&#10;and most diverse, in terms of model types, collection of human judgments of&#10;model-generated summaries on the CNN/Daily Mail dataset annotated by both&#10;expert judges and crowd-source workers. We hope that this work will help&#10;promote a more complete evaluation protocol for text summarization as well as&#10;advance research in developing evaluation metrics that better correlate with&#10;human judgments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.04696" label="2004.04696">
        <attvalues>
          <attvalue for="0" value="BLEURT: Learning Robust Metrics for Text Generation" />
          <attvalue for="1" value="  Text generation has made significant advances in the last few years. Yet,&#10;evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU&#10;and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a&#10;learned evaluation metric based on BERT that can model human judgments with a&#10;few thousand possibly biased training examples. A key aspect of our approach is&#10;a novel pre-training scheme that uses millions of synthetic examples to help&#10;the model generalize. BLEURT provides state-of-the-art results on the last&#10;three years of the WMT Metrics shared task and the WebNLG Competition dataset.&#10;In contrast to a vanilla BERT-based approach, it yields superior results even&#10;when the training data is scarce and out-of-distribution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The WMT shared metrics competition~\cite{bojar2016results,ma2018results,ma2019results} has inspired the creation of many learned metrics, some of which use regression or deep learning~\cite{stanojevic2014beer,ma2017blend,shimanaka2018ruse,chen2016enhanced, mathur2019putting}. Other metrics have been introduced, such as the recent MoverScore \cite{zhao2019moverscore} which combines contextual embeddings and Earth Mover's Distance. We provide a head-to-head comparison with the best performing of those in our experiments. Other approaches do not attempt to estimate quality directly, but use information extraction or question answering as a proxy~\cite{wiseman2017challenges,goodrich2019assessing,eyal2019question}. Those are complementary to our work.&#10;&#10;There has been recent work that uses BERT for evaluation. BERTScore~\cite{zhang2019bertscore} proposes replacing the hard n-gram overlap of BLEU with a soft-overlap using BERT embeddings. We use it in all our experiments. Bertr~\cite{mathur2019putting} and YiSi~\cite{mathur2019putting} also make use of BERT embeddings to capture similarity. Sum-QE~\cite{xenouleassum} fine-tunes BERT for quality estimation as we describe in Section~\ref{sec:fine-tuning}. Our focus is different---we train metrics that are not only state-of-the-art in conventional iid experimental setups, but also robust in the presence of scarce and out-of-distribution training data. To our knowledge no existing work has explored pre-training and extrapolation in the context of NLG.&#10;&#10;Previous studies have used noising for referenceless evaluation~\cite{duvsek2019automatic}.&#10;Noisy pre-training has also been proposed before for other tasks such as paraphrasing~\cite{wieting2015towards,tomar2017neural} but generally not with synthetic data. Generating synthetic data via paraphrases and perturbations has been commonly used for generating adversarial examples~\cite{jia2017adversarial,iyyer2018adversarial,BelinkovB18,ribeiro2018semantically}, an orthogonal line of research.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2208.01030" label="2208.01030">
        <attvalues>
          <attvalue for="0" value="SMART: Sentences as Basic Units for Text Evaluation" />
          <attvalue for="1" value="  Widely used evaluation metrics for text generation either do not work well&#10;with longer texts or fail to evaluate all aspects of text quality. In this&#10;paper, we introduce a new metric called SMART to mitigate such limitations.&#10;Specifically, We treat sentences as basic units of matching instead of tokens,&#10;and use a sentence matching function to soft-match candidate and reference&#10;sentences. Candidate sentences are also compared to sentences in the source&#10;documents to allow grounding (e.g., factuality) evaluation. Our results show&#10;that system-level correlations of our proposed metric with a model-based&#10;matching function outperforms all competing metrics on the SummEval&#10;summarization meta-evaluation dataset, while the same metric with a&#10;string-based matching function is competitive with current model-based metrics.&#10;The latter does not use any neural model, which is useful during model&#10;development phases where resources can be limited and fast evaluation is&#10;required. Finally, we also conducted extensive analyses showing that our&#10;proposed metrics work well with longer summaries and are less biased towards&#10;specific models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.14840" label="2310.14840">
        <attvalues>
          <attvalue for="0" value="Transparency at the Source: Evaluating and Interpreting Language Models&#10;  With Access to the True Distribution" />
          <attvalue for="1" value="  We present a setup for training, evaluating and interpreting neural language&#10;models, that uses artificial, language-like data. The data is generated using a&#10;massive probabilistic grammar (based on state-split PCFGs), that is itself&#10;derived from a large natural language corpus, but also provides us complete&#10;control over the generative process. We describe and release both grammar and&#10;corpus, and test for the naturalness of our generated data. This approach&#10;allows us to define closed-form expressions to efficiently compute exact lower&#10;bounds on obtainable perplexity using both causal and masked language&#10;modelling. Our results show striking differences between neural language&#10;modelling architectures and training objectives in how closely they allow&#10;approximating the lower bound on perplexity. Our approach also allows us to&#10;directly compare learned representations to symbolic rules in the underlying&#10;source. We experiment with various techniques for interpreting model behaviour&#10;and learning dynamics. With access to the underlying true source, our results&#10;show striking differences and outcomes in learning dynamics between different&#10;classes of words.&#10;" />
          <attvalue for="2" value="&#10;&#10;When we train a Language Model on large natural language corpora, we are in effect estimating a probability distribution over possible next tokens or masked tokens. &#10;The true distribution is unknown, so we cannot directly quantitatively measure how good our estimation is, or qualitatively assess whether our LM has discovered `true' underlying patterns in the data.&#10;The best we can do is to measure perplexity on a new sample from the same unknown distribution, and compare that perplexity to the perplexity we obtain with other model architectures or training regimes \cite{brown1992estimate}, or with the expected perplexity given a (compute-optimal) scaling law \cite{DBLP:journals/corr/abs-2001-08361,hoffmann2022empirical}.&#10;&#10;This approach has been enormously successful, but it leaves a number of interesting questions unanswered. &#10;First, one consequence of the absence of an explicit stochastic source is that it is impossible to exactly determine the optimal perplexity that a language model can obtain. &#10;Second, when designing interpretability methods to assess whether an LM has learned specific linguistic rules, we lack a gold standard. If an interpretability method fails to find evidence that the LM has learned a specific linguistic construction we cannot be sure whether this is a failure of the LM, a failure of the interpretability method, or whether our assumption that knowledge of this linguistic construction as an essential component of English fluency is wrong.&#10;&#10;One approach to address these problems is to move to artificially generated data, but such data is often of trivial complexity compared to the richness of natural language, in terms of the vocabulary size or the number of grammatical rules. &#10;&#10;In this paper, we explore an alternative approach. &#10;We also use generated data to train language models on, but to make sure the data approximates natural language in complexity, we use a massive probabilistic grammar, that is itself derived from a large natural language corpus. &#10;To obtain the grammar, we use an automatically parsed sectionfrom the The Pile corpus \cite{pile}, and use the state-split framework \cite{petrov-etal-2006-learning} -- one of the most successful statistical parsing frameworks from before the rise of deep learning -- to obtain a statistical grammar with more than 2 million rules. In \S\ref{sec:pcfg} we describe the procedure for obtaining this grammar in detail.&#10;&#10;This setup allows us to compute the exact lower bound on perplexity, although that computation&#10;turns out to still be nontrivial. This is due to the computational complexity of the problem that makes a naive approach infeasible, even on modern hardware.&#10;One key contribution from this paper is a closed-form expression to efficiently compute masked token probabilities for PCFGs, complementing the classic closed form for causal language modelling \cite{DBLP:journals/coling/Stolcke95}.&#10;Furthermore, our setup provides a gold standard to qualitatively assess results from interpretability methods against. We conduct a wide range of experiments, showing the naturalness of our generated data, determining the impact of model and data properties on language model performance, and interpreting model behaviour and learning dynamics using various interpretability tools.&#10;Figure \ref{fig:pipeline} presents a schematic overview of our methodology.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Interpretability, Computer Science, Artificial Data Generation, Linguistics, Cognitive Science, Neural Language Models, Mathematics" />
        </attvalues>
      </node>
      <node id="cmp-lg/9411029" label="cmp-lg/9411029">
        <attvalues>
          <attvalue for="0" value="An Efficient Probabilistic Context-Free Parsing Algorithm that Computes&#10;  Prefix Probabilities" />
          <attvalue for="1" value="  We describe an extension of Earley's parser for stochastic context-free&#10;grammars that computes the following quantities given a stochastic context-free&#10;grammar and an input string: a) probabilities of successive prefixes being&#10;generated by the grammar; b) probabilities of substrings being generated by the&#10;nonterminals, including the entire string being generated by the grammar; c)&#10;most likely (Viterbi) parse of the string; d) posterior expected number of&#10;applications of each grammar production, as required for reestimating rule&#10;probabilities. (a) and (b) are computed incrementally in a single left-to-right&#10;pass over the input. Our algorithm compares favorably to standard bottom-up&#10;parsing methods for SCFGs in that it works efficiently on sparse grammars by&#10;making use of Earley's top-down control structure. It can process any&#10;context-free rule format without conversion to some normal form, and combines&#10;computations for (a) through (d) in a single algorithm. Finally, the algorithm&#10;has simple extensions for processing partially bracketed inputs, and for&#10;finding partial parses and their likelihoods on ungrammatical inputs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.18609" label="2311.18609">
        <attvalues>
          <attvalue for="0" value="ArthModel: Enhance Arithmetic Skills to Large Language Model" />
          <attvalue for="1" value="  With the great success of ChatGPT, the research of large language models has&#10;become increasingly popular. However, the models have several limitations, such&#10;as toxicity and pool performance of arithmetic solving. Meanwhile, LLM may have&#10;some potential abilities that have yet to be exploited. In this paper, we&#10;choose a different way to enhance the arithmetic ability of LLM. We propose to&#10;train LLM to generate a postfix expression related to the arithmetic problem&#10;and incorporate it with small pretrained models. Moreover, this small model&#10;transfers the token embeddings into real dense numbers and invokes native&#10;functions of a deep learning platform to get the correct answer. To generate&#10;the final result, we propose prompt injection for adding the result outputs by&#10;the small model to LLM. This work provides different ways of thinking, training&#10;and using a language model. The codes and models will be released at&#10;\url{https://github.com/eteced/arithmetic_finetuning_v1}.&#10;" />
          <attvalue for="2" value="&#10;\begin{CJK}{UTF8}{gbsn}&#10;There are works armed for extending the ability of LLM. By adding the chain of thought for an associated answer, \cite{wei2022chain} improves the ability of large language models to perform complex reasoning. \cite{xu2023wizardlm} rewrites the prompts to upgrade the simple instruction to a more complex one and increase diversity for LLM training. \cite{schick2023toolformer} trained the model to use different API calls and get the correct answers. \cite{gao2023pal} do more, they incorporate LLM with a python interpreter that the LLM generates codes for the python interpreter and gets the answer from it, which improves the accuracy of the answer. However, to make LLM solve arithmetic problems well, there could be a more straightforward way to do it. More specifically, the model should be able to access the native calls of a deep learning platform it runs on. Making LLM cooperate with a small model can also be thought as LLM incorporating a small cheap API.&#10;&#10;In this paper, we take LLaMA as an example and illustrate how LLM incorporates a small pretrained arithmetic-solving model to enhance its arithmetic ability. We first train the LLM model to generate a postfix expression for an arithmetic calculation problem, which can be thought of as a 'code' input for the small arithmetic-solving model and a CoT for the problem since it determined which numbers and math ops should be considered first. Then, the small arithmetic-solving model will turn the token vectors into real dense numbers and invoke native function calls like 'add' to perform the math operations on dense numbers. The deep learning platform provides native functions that also support the running of LLM. Finally, incorporating the result from the small arithmetic-solving model, the LLM outputs the desired answers.&#10;&#10;In the rest of this paper, we will first illustrate the framework of the work and then dig into the details of the small arithmetic-solving model and the methods to train such a model. Next, we show how to train LLM to output right for the small models. Finally, we use the Alpace data with the arithmetic data we generated to train the ensembled model. With the prompt injection, LLM can generate the correct arithmetic result without losing the general chat ability.&#10;\end{CJK}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Arithmetic Problem Solving, Model Enhancement Techniques, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.12244" label="2304.12244">
        <attvalues>
          <attvalue for="0" value="WizardLM: Empowering Large Language Models to Follow Complex&#10;  Instructions" />
          <attvalue for="1" value="  Training large language models (LLMs) with open-domain instruction following&#10;data brings colossal success. However, manually creating such instruction data&#10;is very time-consuming and labor-intensive. Moreover, humans may struggle to&#10;produce high-complexity instructions. In this paper, we show an avenue for&#10;creating large amounts of instruction data with varying levels of complexity&#10;using LLM instead of humans. Starting with an initial set of instructions, we&#10;use our proposed Evol-Instruct to rewrite them step by step into more complex&#10;instructions. Then, we mix all generated instruction data to fine-tune LLaMA.&#10;We call the resulting model WizardLM. Human evaluations on a&#10;complexity-balanced test bed and Vicuna's testset show that instructions from&#10;Evol-Instruct are superior to human-created ones. By analyzing the human&#10;evaluation results of the high complexity part, we demonstrate that outputs&#10;from our WizardLM are preferred to outputs from OpenAI ChatGPT. In GPT-4&#10;automatic evaluation, WizardLM achieves more than 90\% capacity of ChatGPT on&#10;17 out of 29 skills. Even though WizardLM still lags behind ChatGPT in some&#10;aspects, our findings suggest that fine-tuning with AI-evolved instructions is&#10;a promising direction for enhancing LLMs. Our code and data are public at&#10;https://github.com/nlpxucan/WizardLM&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.04761" label="2302.04761">
        <attvalues>
          <attvalue for="0" value="Toolformer: Language Models Can Teach Themselves to Use Tools" />
          <attvalue for="1" value="  Language models (LMs) exhibit remarkable abilities to solve new tasks from&#10;just a few examples or textual instructions, especially at scale. They also,&#10;paradoxically, struggle with basic functionality, such as arithmetic or factual&#10;lookup, where much simpler and smaller models excel. In this paper, we show&#10;that LMs can teach themselves to use external tools via simple APIs and achieve&#10;the best of both worlds. We introduce Toolformer, a model trained to decide&#10;which APIs to call, when to call them, what arguments to pass, and how to best&#10;incorporate the results into future token prediction. This is done in a&#10;self-supervised way, requiring nothing more than a handful of demonstrations&#10;for each API. We incorporate a range of tools, including a calculator, a Q\&amp;A&#10;system, two different search engines, a translation system, and a calendar.&#10;Toolformer achieves substantially improved zero-shot performance across a&#10;variety of downstream tasks, often competitive with much larger models, without&#10;sacrificing its core language modeling abilities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.00368" label="2401.00368">
        <attvalues>
          <attvalue for="0" value="Improving Text Embeddings with Large Language Models" />
          <attvalue for="1" value="  In this paper, we introduce a novel and simple method for obtaining&#10;high-quality text embeddings using only synthetic data and less than 1k&#10;training steps. Unlike existing methods that often depend on multi-stage&#10;intermediate pre-training with billions of weakly-supervised text pairs,&#10;followed by fine-tuning with a few labeled datasets, our method does not&#10;require building complex training pipelines or relying on manually collected&#10;datasets that are often constrained by task diversity and language coverage. We&#10;leverage proprietary LLMs to generate diverse synthetic data for hundreds of&#10;thousands of text embedding tasks across 93 languages. We then fine-tune&#10;open-source decoder-only LLMs on the synthetic data using standard contrastive&#10;loss. Experiments demonstrate that our method achieves strong performance on&#10;highly competitive text embedding benchmarks without using any labeled data.&#10;Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our&#10;model sets new state-of-the-art results on the BEIR and MTEB benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Text Embeddings &#10;are continuous low-dimensional representations of text and&#10;have been extensively applied to various downstream tasks such as information retrieval, question answering,&#10;and retrieval-augmented generation (RAG).&#10;Early work on text embeddings includes latent semantic indexing~\cite{deerwester1990indexing}&#10;and weighted average of word embeddings~\cite{Mikolov2013EfficientEO}.&#10;More recent methods exploit supervision from natural language inference~\cite{Bowman2015ALA} and&#10;labeled query-document pairs, such as the MS-MARCO passage ranking dataset~\cite{Campos2016MSMA},&#10;to train text embeddings~\cite{Reimers2019SentenceBERTSE,Conneau2017SupervisedLO,Gao2021SimCSESC}.&#10;However,&#10;labeled data are often limited in terms of task diversity and language coverage.&#10;To address this challenge,&#10;methods like Contriever~\cite{Izacard2021TowardsUD}, OpenAI Embeddings~\cite{Neelakantan2022TextAC}, E5~\cite{wang2022text}, and BGE~\cite{xiao2023c}&#10;adopt a multi-stage training paradigm.&#10;They first pre-train on large-scale weakly-supervised text pairs using contrastive loss&#10;and then fine-tune on small-scale but high-quality datasets.&#10;In this paper,&#10;we demonstrate that it is possible to obtain state-of-the-art text embeddings with single-stage training.&#10;&#10;Synthetic Data &#10;Synthetic data generation is a widely studied topic in information retrieval research,&#10;with various methods proposed to enhance retrieval systems with artificially created data.&#10;For instance,&#10;Doc2query~\cite{nogueira2019document}, InPars ~\cite{Bonifacio2022InParsUD}, and Promptagator~\cite{dai2022promptagator}&#10;generate synthetic queries for unlabeled documents,&#10;which are then leveraged for document expansion or model training.&#10;GPL~\cite{Wang2021GPLGP} employs a cross-encoder to produce pseudo-labels&#10;for query-document pairs.&#10;Similarly,&#10;Query2doc~\cite{wang2023query2doc} generates pseudo-documents for query expansion by few-shot prompting LLMs.&#10;Unlike these methods,&#10;our approach does not rely on any unlabeled documents or queries&#10;and thus can generate more diverse synthetic data.&#10;&#10;Another related line of work focuses on knowledge distillation from black-box LLMs&#10;by training on synthetic data generated from them.&#10;DINO~\cite{schick2021generating} generates synthetic text pairs for semantic textual similarity.&#10;Unnatural Instructions~\cite{honovich2022unnatural} is a synthetic instruction following dataset&#10;by prompting existing LLMs.&#10;Orca~\cite{Mukherjee2023OrcaPL} and Phi~\cite{Gunasekar2023TextbooksAA} propose to train better small language models&#10;by using high-quality synthetic data from GPT-3.5/4~\cite{OpenAI2023GPT4TR}.&#10;&#10;Large Language Models &#10;With the popularization of ChatGPT,&#10;large language models (LLMs) have demonstrated remarkable capabilities in instruction following&#10;and few-shot in-context learning~\cite{NEURIPS2020_1457c0d6}.&#10;However,&#10;the most advanced LLMs such as GPT-4~\cite{OpenAI2023GPT4TR} are proprietary and have little technical details disclosed.&#10;To bridge the gap between proprietary and open-source LLMs,&#10;several notable efforts have been made,&#10;such as LLaMA-2~\cite{touvron2023llama} and Mistral~\cite{jiang2023mistral} models.&#10;A major limitation of LLMs is that they lack awareness of recent events and private knowledge.&#10;This issue can be partly mitigated by augmenting LLMs with information retrieved from external sources,&#10;a technique known as retrieval-augmented generation (RAG).&#10;On the other hand,&#10;LLMs can also serve as foundation models to enhance text embeddings.&#10;RepLLaMA~\cite{Ma2023FineTuningLF} proposes to fine-tune LLaMA-2 with bi-encoder architecture for ad-hoc retrieval.&#10;SGPT~\cite{Muennighoff2022SGPTGS}, GTR~\cite{Ni2021LargeDE}, and Udever~\cite{zhang2023language}&#10;demonstrate the scaling law of text embeddings empirically,&#10;but their performance still falls behind small bidirectional encoders such as E5~\cite{wang2022text} and BGE~\cite{xiao2023c}.&#10;In this paper,&#10;we present a novel approach to train state-of-the-art text embeddings by exploiting the latest advances of LLMs and synthetic data.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Text Embeddings, Artificial Intelligence, Mathematics, Synthetic Data, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2211.05756" label="2211.05756">
        <attvalues>
          <attvalue for="0" value="Massively Multilingual ASR on 70 Languages: Tokenization, Architecture,&#10;  and Generalization Capabilities" />
          <attvalue for="1" value="  End-to-end multilingual ASR has become more appealing because of several&#10;reasons such as simplifying the training and deployment process and positive&#10;performance transfer from high-resource to low-resource languages. However,&#10;scaling up the number of languages, total hours, and number of unique tokens is&#10;not a trivial task. This paper explores large-scale multilingual ASR models on&#10;70 languages. We inspect two architectures: (1) Shared embedding and output and&#10;(2) Multiple embedding and output model. In the shared model experiments, we&#10;show the importance of tokenization strategy across different languages. Later,&#10;we use our optimal tokenization strategy to train multiple embedding and output&#10;model to further improve our result. Our multilingual ASR achieves 13.9%-15.6%&#10;average WER relative improvement compared to monolingual models. We show that&#10;our multilingual ASR generalizes well on an unseen dataset and domain,&#10;achieving 9.5% and 7.5% WER on Multilingual Librispeech (MLS) with zero-shot&#10;and finetuning, respectively.&#10;" />
          <attvalue for="2" value="&#10;&#10;Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (i.e., RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, &#10;have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}. They capture acoustic variabilities in their encoder modules while modeling the linguistic structure of language in their decoder modules.&#10;Due mainly to this scalability, E2E models have enabled great strides in multilingual ASR research, which aims to create a single model that can recognize multiple languages at the same time~\cite{Li18multidialect,kannan19_interspeech,Hou2020LargeScaleEM, Pratap2020,li2021scaling,babu22_interspeech,yang2022learning,joshi21_interspeech}.&#10;&#10;Training and deploying multilingual ASR models has several practical benefits. A single well-performing multilingual model saves significant manual effort to tune each language individually during training, as well as model maintenance and deployment when in production. Building a highly accurate large-scale multilingual teacher model also enables scaling up semi-supervised training to more languages via iterative pseudo-labeling. A single multilingual model ensures seamless deployment and good ASR quality for locales with heterogeneous languages as it integrates language identification signals and speech recognition signals during decoding. In some cases, there is a positive effect where high-resource languages also improve low resource languages performance when we mix and train them simultaneously.&#10;&#10;One key challenge of multilingual ASR in practice is when we scale up the number of languages, our vocabulary size grows larger. Prior works \cite{li2021scaling} trained a multilingual ASR for 15 languages by simply combining all graphemes together. \cite{joshi21_interspeech} proposed multilingual with multi-decoder output for 4 languages. \cite{Pratap2020} built multilingual ASR on top of 50 languages with total 16,000 hours training dataset. To the best of our knowledge, most of the existing works have less amount of languages and smaller unique graphemes size compared to our in-house video dataset.&#10;&#10;In this paper, we explore large-scale multilingual on \nlang languages with 150,000 hours dataset. We explored based on two architectures: shared input embedding and output architecture and multiple input embedding and output architecture. In the shared model experiment, we show the importance of tokenization strategy across different languages and provide further analysis of the result. Later, we use the best token strategy to train a multilingual model with multiple input embedding and output layers. We evaluate our multilingual model on in-house dataset and show significant improvement over monolingual models. Lastly, we also show our multilingual model could generalized well on the new domain and dataset. We achieve 9.5\% and 7.5\% WER on Multilingual Librispeech (MLS) with zero-shot and finetuning, respectively. To the best of our knowledge, our result is competitive with the state-of-the-art performance on MLS dataset.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual ASR, Engineering, Artificial Intelligence, Language Generalization, Large-Scale Modeling" />
        </attvalues>
      </node>
      <node id="1211.3711" label="1211.3711">
        <attvalues>
          <attvalue for="0" value="Sequence Transduction with Recurrent Neural Networks" />
          <attvalue for="1" value="  Many machine learning tasks can be expressed as the transformation---or&#10;\emph{transduction}---of input sequences into output sequences: speech&#10;recognition, machine translation, protein secondary structure prediction and&#10;text-to-speech to name but a few. One of the key challenges in sequence&#10;transduction is learning to represent both the input and output sequences in a&#10;way that is invariant to sequential distortions such as shrinking, stretching&#10;and translating. Recurrent neural networks (RNNs) are a powerful sequence&#10;learning architecture that has proven capable of learning such representations.&#10;However RNNs traditionally require a pre-defined alignment between the input&#10;and output sequences to perform transduction. This is a severe limitation since&#10;\emph{finding} the alignment is the most difficult aspect of many sequence&#10;transduction problems. Indeed, even determining the length of the output&#10;sequence is often challenging. This paper introduces an end-to-end,&#10;probabilistic sequence transduction system, based entirely on RNNs, that is in&#10;principle able to transform any input sequence into any finite, discrete output&#10;sequence. Experimental results for phoneme recognition are provided on the&#10;TIMIT speech corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1303.5778" label="1303.5778">
        <attvalues>
          <attvalue for="0" value="Speech Recognition with Deep Recurrent Neural Networks" />
          <attvalue for="1" value="  Recurrent neural networks (RNNs) are a powerful model for sequential data.&#10;End-to-end training methods such as Connectionist Temporal Classification make&#10;it possible to train RNNs for sequence labelling problems where the&#10;input-output alignment is unknown. The combination of these methods with the&#10;Long Short-term Memory RNN architecture has proved particularly fruitful,&#10;delivering state-of-the-art results in cursive handwriting recognition. However&#10;RNN performance in speech recognition has so far been disappointing, with&#10;better results returned by deep feedforward networks. This paper investigates&#10;\emph{deep recurrent neural networks}, which combine the multiple levels of&#10;representation that have proved so effective in deep networks with the flexible&#10;use of long range context that empowers RNNs. When trained end-to-end with&#10;suitable regularisation, we find that deep Long Short-term Memory RNNs achieve&#10;a test set error of 17.7% on the TIMIT phoneme recognition benchmark, which to&#10;our knowledge is the best recorded score.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.00841" label="1801.00841">
        <attvalues>
          <attvalue for="0" value="Exploring Architectures, Data and Units For Streaming End-to-End Speech&#10;  Recognition with RNN-Transducer" />
          <attvalue for="1" value="  We investigate training end-to-end speech recognition models with the&#10;recurrent neural network transducer (RNN-T): a streaming, all-neural,&#10;sequence-to-sequence architecture which jointly learns acoustic and language&#10;model components from transcribed acoustic data. We explore various model&#10;architectures and demonstrate how the model can be improved further if&#10;additional text or pronunciation data are available. The model consists of an&#10;`encoder', which is initialized from a connectionist temporal&#10;classification-based (CTC) acoustic model, and a `decoder' which is partially&#10;initialized from a recurrent neural network language model trained on text data&#10;alone. The entire neural network is trained with the RNN-T loss and directly&#10;outputs the recognized transcript as a sequence of graphemes, thus performing&#10;end-to-end speech recognition. We find that performance can be improved further&#10;through the use of sub-word units (`wordpieces') which capture longer context&#10;and significantly reduce substitution errors. The best RNN-T system, a&#10;twelve-layer LSTM encoder with a two-layer LSTM decoder trained with 30,000&#10;wordpieces as output targets achieves a word error rate of 8.5\% on&#10;voice-search and 5.2\% on voice-dictation tasks and is comparable to a&#10;state-of-the-art baseline at 8.3\% on voice-search and 5.4\% voice-dictation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.06621" label="1811.06621">
        <attvalues>
          <attvalue for="0" value="Streaming End-to-end Speech Recognition For Mobile Devices" />
          <attvalue for="1" value="  End-to-end (E2E) models, which directly predict output character sequences&#10;given input speech, are good candidates for on-device speech recognition. E2E&#10;models, however, present numerous challenges: In order to be truly useful, such&#10;models must decode speech utterances in a streaming fashion, in real time; they&#10;must be robust to the long tail of use cases; they must be able to leverage&#10;user-specific context (e.g., contact lists); and above all, they must be&#10;extremely accurate. In this work, we describe our efforts at building an E2E&#10;speech recognizer using a recurrent neural network transducer. In experimental&#10;evaluations, we find that the proposed approach can outperform a conventional&#10;CTC-based model in terms of both latency and accuracy in a number of evaluation&#10;categories.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.12977" label="1910.12977">
        <attvalues>
          <attvalue for="0" value="Transformer-Transducer: End-to-End Speech Recognition with&#10;  Self-Attention" />
          <attvalue for="1" value="  We explore options to use Transformer networks in neural transducer for&#10;end-to-end speech recognition. Transformer networks use self-attention for&#10;sequence modeling and comes with advantages in parallel computation and&#10;capturing contexts. We propose 1) using VGGNet with causal convolution to&#10;incorporate positional information and reduce frame rate for efficient&#10;inference 2) using truncated self-attention to enable streaming for Transformer&#10;and reduce computational complexity. All experiments are conducted on the&#10;public LibriSpeech corpus. The proposed Transformer-Transducer outperforms&#10;neural transducer with LSTM/BLSTM networks and achieved word error rates of&#10;6.37 % on the test-clean set and 15.30 % on the test-other set, while remaining&#10;streamable, compact with 45.7M parameters for the entire system, and&#10;computationally efficient with complexity of O(T), where T is input sequence&#10;length.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.02562" label="2002.02562">
        <attvalues>
          <attvalue for="0" value="Transformer Transducer: A Streamable Speech Recognition Model with&#10;  Transformer Encoders and RNN-T Loss" />
          <attvalue for="1" value="  In this paper we present an end-to-end speech recognition model with&#10;Transformer encoders that can be used in a streaming speech recognition system.&#10;Transformer computation blocks based on self-attention are used to encode both&#10;audio and label sequences independently. The activations from both audio and&#10;label encoders are combined with a feed-forward layer to compute a probability&#10;distribution over the label space for every combination of acoustic frame&#10;position and label history. This is similar to the Recurrent Neural Network&#10;Transducer (RNN-T) model, which uses RNNs for information encoding instead of&#10;Transformer encoders. The model is trained with the RNN-T loss well-suited to&#10;streaming decoding. We present results on the LibriSpeech dataset showing that&#10;limiting the left context for self-attention in the Transformer layers makes&#10;decoding computationally tractable for streaming, with only a slight&#10;degradation in accuracy. We also show that the full attention version of our&#10;model beats the-state-of-the art accuracy on the LibriSpeech benchmarks. Our&#10;results also show that we can bridge the gap between full attention and limited&#10;attention versions of our model by attending to a limited number of future&#10;frames.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.10759" label="2010.10759">
        <attvalues>
          <attvalue for="0" value="Emformer: Efficient Memory Transformer Based Acoustic Model For Low&#10;  Latency Streaming Speech Recognition" />
          <attvalue for="1" value="  This paper proposes an efficient memory transformer Emformer for low latency&#10;streaming speech recognition. In Emformer, the long-range history context is&#10;distilled into an augmented memory bank to reduce self-attention's computation&#10;complexity. A cache mechanism saves the computation for the key and value in&#10;self-attention for the left context. Emformer applies a parallelized block&#10;processing in training to support low latency models. We carry out experiments&#10;on benchmark LibriSpeech data. Under average latency of 960 ms, Emformer gets&#10;WER $2.50\%$ on test-clean and $5.62\%$ on test-other. Comparing with a strong&#10;baseline augmented memory transformer (AM-TRF), Emformer gets $4.6$ folds&#10;training speedup and $18\%$ relative real-time factor (RTF) reduction in&#10;decoding with relative WER reduction $17\%$ on test-clean and $9\%$ on&#10;test-other. For a low latency scenario with an average latency of 80 ms,&#10;Emformer achieves WER $3.01\%$ on test-clean and $7.09\%$ on test-other.&#10;Comparing with the LSTM baseline with the same latency and model size, Emformer&#10;gets relative WER reduction $9\%$ and $16\%$ on test-clean and test-other,&#10;respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1508.04395" label="1508.04395">
        <attvalues>
          <attvalue for="0" value="End-to-End Attention-based Large Vocabulary Speech Recognition" />
          <attvalue for="1" value="  Many of the current state-of-the-art Large Vocabulary Continuous Speech&#10;Recognition Systems (LVCSR) are hybrids of neural networks and Hidden Markov&#10;Models (HMMs). Most of these systems contain separate components that deal with&#10;the acoustic modelling, language modelling and sequence decoding. We&#10;investigate a more direct approach in which the HMM is replaced with a&#10;Recurrent Neural Network (RNN) that performs sequence prediction directly at&#10;the character level. Alignment between the input features and the desired&#10;character sequence is learned automatically by an attention mechanism built&#10;into the RNN. For each predicted character, the attention mechanism scans the&#10;input sequence and chooses relevant frames. We propose two methods to speed up&#10;this operation: limiting the scan to a subset of most promising frames and&#10;pooling over time the information contained in neighboring frames, thereby&#10;reducing source sequence length. Integrating an n-gram language model into the&#10;decoding process yields recognition accuracies similar to other HMM-free&#10;RNN-based approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.01769" label="1712.01769">
        <attvalues>
          <attvalue for="0" value="State-of-the-art Speech Recognition With Sequence-to-Sequence Models" />
          <attvalue for="1" value="  Attention-based encoder-decoder architectures such as Listen, Attend, and&#10;Spell (LAS), subsume the acoustic, pronunciation and language model components&#10;of a traditional automatic speech recognition (ASR) system into a single neural&#10;network. In previous work, we have shown that such architectures are comparable&#10;to state-of-theart ASR systems on dictation tasks, but it was not clear if such&#10;architectures would be practical for more challenging tasks such as voice&#10;search. In this work, we explore a variety of structural and optimization&#10;improvements to our LAS model which significantly improve performance. On the&#10;structural side, we show that word piece models can be used instead of&#10;graphemes. We also introduce a multi-head attention architecture, which offers&#10;improvements over the commonly-used single-head attention. On the optimization&#10;side, we explore synchronous training, scheduled sampling, label smoothing, and&#10;minimum word error rate optimization, which are all shown to improve accuracy.&#10;We present results with a unidirectional LSTM encoder for streaming&#10;recognition. On a 12, 500 hour voice search task, we find that the proposed&#10;changes improve the WER from 9.2% to 5.6%, while the best conventional system&#10;achieves 6.7%; on a dictation task our model achieves a WER of 4.1% compared to&#10;5% for the conventional system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.03294" label="1805.03294">
        <attvalues>
          <attvalue for="0" value="Improved training of end-to-end attention models for speech recognition" />
          <attvalue for="1" value="  Sequence-to-sequence attention-based models on subword units allow simple&#10;open-vocabulary end-to-end speech recognition. In this work, we show that such&#10;models can achieve competitive results on the Switchboard 300h and LibriSpeech&#10;1000h tasks. In particular, we report the state-of-the-art word error rates&#10;(WER) of 3.54% on the dev-clean and 3.82% on the test-clean evaluation subsets&#10;of LibriSpeech. We introduce a new pretraining scheme by starting with a high&#10;time reduction factor and lowering it during training, which is crucial both&#10;for convergence and final performance. In some experiments, we also use an&#10;auxiliary CTC loss function to help the convergence. In addition, we train long&#10;short-term memory (LSTM) language models on subword units. By shallow fusion,&#10;we report up to 27% relative improvements in WER over the attention baseline&#10;without a language model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.05948" label="2111.05948">
        <attvalues>
          <attvalue for="0" value="Scaling ASR Improves Zero and Few Shot Learning" />
          <attvalue for="1" value="  With 4.5 million hours of English speech from 10 different sources across 120&#10;countries and models of up to 10 billion parameters, we explore the frontiers&#10;of scale for automatic speech recognition. We propose data selection techniques&#10;to efficiently scale training data to find the most valuable samples in massive&#10;datasets. To efficiently scale model sizes, we leverage various optimizations&#10;such as sparse transducer loss and model sharding. By training 1-10B parameter&#10;universal English ASR models, we push the limits of speech recognition&#10;performance across many domains. Furthermore, our models learn powerful speech&#10;representations with zero and few-shot capabilities on novel domains and styles&#10;of speech, exceeding previous results across multiple in-house and public&#10;benchmarks. For speakers with disorders due to brain damage, our best zero-shot&#10;and few-shot models achieve 22% and 60% relative improvement on the AphasiaBank&#10;test set, respectively, while realizing the best performance on public social&#10;media videos. Furthermore, the same universal model reaches equivalent&#10;performance with 500x less in-domain data on the SPGISpeech financial-domain&#10;dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.01541" label="1712.01541">
        <attvalues>
          <attvalue for="0" value="Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence&#10;  Model" />
          <attvalue for="1" value="  Sequence-to-sequence models provide a simple and elegant solution for&#10;building speech recognition systems by folding separate components of a typical&#10;system, namely acoustic (AM), pronunciation (PM) and language (LM) models into&#10;a single neural network. In this work, we look at one such sequence-to-sequence&#10;model, namely listen, attend and spell (LAS), and explore the possibility of&#10;training a single model to serve different English dialects, which simplifies&#10;the process of training multi-dialect systems without the need for separate AM,&#10;PM and LMs for each dialect. We show that simply pooling the data from all&#10;dialects into one LAS model falls behind the performance of a model fine-tuned&#10;on each dialect. We then look at incorporating dialect-specific information&#10;into the model, both by modifying the training targets by inserting the dialect&#10;symbol at the end of the original grapheme sequence and also feeding a 1-hot&#10;representation of the dialect information into all layers of the model.&#10;Experimental results on seven English dialects show that our proposed system is&#10;effective in modeling dialect variations within a single LAS model,&#10;outperforming a LAS model trained individually on each of the seven dialects by&#10;3.1 ~ 16.5% relative.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.05330" label="1909.05330">
        <attvalues>
          <attvalue for="0" value="Large-Scale Multilingual Speech Recognition with a Streaming End-to-End&#10;  Model" />
          <attvalue for="1" value="  Multilingual end-to-end (E2E) models have shown great promise in expansion of&#10;automatic speech recognition (ASR) coverage of the world's languages. They have&#10;shown improvement over monolingual systems, and have simplified training and&#10;serving by eliminating language-specific acoustic, pronunciation, and language&#10;models. This work presents an E2E multilingual system which is equipped to&#10;operate in low-latency interactive applications, as well as handle a key&#10;challenge of real world data: the imbalance in training data across languages.&#10;Using nine Indic languages, we compare a variety of techniques, and find that a&#10;combination of conditioning on a language vector and training language-specific&#10;adapter layers produces the best model. The resulting E2E multilingual model&#10;achieves a lower word error rate (WER) than both monolingual E2E models (eight&#10;of nine languages) and monolingual conventional systems (all nine languages).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.09296" label="2111.09296">
        <attvalues>
          <attvalue for="0" value="XLS-R: Self-supervised Cross-lingual Speech Representation Learning at&#10;  Scale" />
          <attvalue for="1" value="  This paper presents XLS-R, a large-scale model for cross-lingual speech&#10;representation learning based on wav2vec 2.0. We train models with up to 2B&#10;parameters on nearly half a million hours of publicly available speech audio in&#10;128 languages, an order of magnitude more public data than the largest known&#10;prior work. Our evaluation covers a wide range of tasks, domains, data regimes&#10;and languages, both high and low-resource. On the CoVoST-2 speech translation&#10;benchmark, we improve the previous state of the art by an average of 7.4 BLEU&#10;over 21 translation directions into English. For speech recognition, XLS-R&#10;improves over the best known prior work on BABEL, MLS, CommonVoice as well as&#10;VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets&#10;a new state of the art on VoxLingua107 language identification. Moreover, we&#10;show that with sufficient model size, cross-lingual pretraining can outperform&#10;English-only pretraining when translating English speech into other languages,&#10;a setting which favors monolingual pretraining. We hope XLS-R can help to&#10;improve speech processing tasks for many more languages of the world.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.05735" label="2209.05735">
        <attvalues>
          <attvalue for="0" value="Learning ASR pathways: A sparse multilingual ASR model" />
          <attvalue for="1" value="  Neural network pruning compresses automatic speech recognition (ASR) models&#10;effectively. However, in multilingual ASR, language-agnostic pruning may lead&#10;to severe performance drops on some languages because language-agnostic pruning&#10;masks may not fit all languages and discard important language-specific&#10;parameters. In this work, we present ASR pathways, a sparse multilingual ASR&#10;model that activates language-specific sub-networks (&quot;pathways&quot;), such that the&#10;parameters for each language are learned explicitly. With the overlapping&#10;sub-networks, the shared parameters can also enable knowledge transfer for&#10;lower-resource languages via joint multilingual training. We propose a novel&#10;algorithm to learn ASR pathways, and evaluate the proposed method on 4&#10;languages with a streaming RNN-T model. Our proposed ASR pathways outperform&#10;both dense models and a language-agnostically pruned model, and provide better&#10;performance on low-resource languages compared to the monolingual sparse&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Multilingual ASR. Among recent works on multilingual ASR, language-specific modeling is usually adopted to achieve universally decent performance on all languages. This includes language-aware encoding \cite{tian2022lae}, parameterizing language-specific attention heads \cite{zhu20d_interspeech}, adapter modules \cite{kannan19_interspeech}, decoders \cite{pratap20c_interspeech} and Mixture-of-experts \cite{gaur2021mixture}. In \cite{lu2022language}, LTH was used to identify language-specific sub-networks inside the pre-trained multilingual XLSR model \cite{conneau2020unsupervised}. These sub-networks were shown to be able to pre-train an enhanced XLSR model that improves downstream multilingual ASR fine-tuning. In contrast to \cite{lu2022language}, we directly identify the language-specific sparse structures in a multilingual ASR model, which can potentially learn masks that are more tailored to ASR tasks.&#10;&#10;Model compression. Prior works on ASR model compression can be grouped into 3 categories: (1) knowledge distillation \cite{movsner2019improving, panchapagesan2021efficient} which transfers knowledge from a larger model to a smaller model; (2) model sparsification \cite{shangguan2019optimizing, yang2022omni, ding2021audio, liu2022learning, wu2021dynamic} which prunes (zeros out) a subset of model parameters to reduce the representational complexity and improve run-time efficiency; (3) parameter quantization \cite{he2019streaming, sainath2020streaming, ding20224} which represents the trained model parameters with fewer bits for model size reduction. In this study, we adopt model pruning approaches (IMP and LTH) to find language-specific sub-networks. In \cite{ding2021audio}, LTH was shown to learn a better sparse monolingual ASR than IMP. We study how the monolingual sub-networks could be used for multilingual scenario. A similar scheme was also investigated in \cite{sun2020learning} for multitask sequence-labeling.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2007.03001" label="2007.03001">
        <attvalues>
          <attvalue for="0" value="Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters" />
          <attvalue for="1" value="  We study training a single acoustic model for multiple languages with the aim&#10;of improving automatic speech recognition (ASR) performance on low-resource&#10;languages, and over-all simplifying deployment of ASR systems that support&#10;diverse languages. We perform an extensive benchmark on 51 languages, with&#10;varying amount of training data by language(from 100 hours to 1100 hours). We&#10;compare three variants of multilingual training from a single joint model&#10;without knowing the input language, to using this information, to multiple&#10;heads (one per language cluster). We show that multilingual training of ASR&#10;models on several languages can improve recognition performance, in particular,&#10;on low resource languages. We see 20.9%, 23% and 28.8% average WER relative&#10;reduction compared to monolingual baselines on joint model, joint model with&#10;language input and multi head model respectively. To our knowledge, this is the&#10;first work studying multilingual ASR at massive scale, with more than 50&#10;languages and more than 16,000 hours of audio across them.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.06947" label="2105.06947">
        <attvalues>
          <attvalue for="0" value="Thank you BART! Rewarding Pre-Trained Models Improves Formality Style&#10;  Transfer" />
          <attvalue for="1" value="  Scarcity of parallel data causes formality style transfer models to have&#10;scarce success in preserving content. We show that fine-tuning pre-trained&#10;language (GPT-2) and sequence-to-sequence (BART) models boosts content&#10;preservation, and that this is possible even with limited amounts of parallel&#10;data. Augmenting these models with rewards that target style and content -- the&#10;two core aspects of the task -- we achieve a new state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Content Preservation Methods, Linguistics, Style Transfer Techniques, Artificial Intelligence, Natural Language Processing, Language Model Fine-tuning" />
        </attvalues>
      </node>
      <node id="2109.04949" label="2109.04949">
        <attvalues>
          <attvalue for="0" value="We went to look for meaning and all we got were these lousy&#10;  representations: aspects of meaning representation for computational&#10;  semantics" />
          <attvalue for="1" value="  In this paper we examine different meaning representations that are commonly&#10;used in different natural language applications today and discuss their limits,&#10;both in terms of the aspects of the natural language meaning they are modelling&#10;and in terms of the aspects of the application for which they are used.&#10;" />
          <attvalue for="2" value="&#10;&#10;A crucial component to produce a ``successful'' NLP system is&#10;sufficiently expressive representations of meaning. We consider a&#10;sufficiently expressive meaning representation to be one that allows a&#10;system's output to be considered acceptable to native speakers given&#10;the task. In this paper we present several features of meaning and&#10;discuss how different methods of deriving meaning representations&#10;capture these features. This list is by no means exhaustive. It&#10;might be viewed as a first attempt to discuss ways of establishing a&#10;general methodology for evaluating meaning representations and&#10;characterising what kinds of applications they might be useful for.&#10;The features we will discuss are:&#10;compositionality\ \ The ability to compute the meaning of&#10;phrases on the basis of the meanings of their immediate&#10;sub-constituents.&#10;logic-based inference\ \ The ability to derive conclusions&#10;based on logical inference, including logical inferences based on the&#10;semantics of logical constants such as and, not and&#10;logical quantifiers and also consequences that follow from additional&#10;axioms or ``meaning postulates''.&#10;discourse semantics\ \ This involves giving meaning&#10;representations for units larger than a sentence.&#10;underspecification\ \ Underspecified meaning representations&#10;are single representations which cover several meanings in cases where&#10;there is systematic ambiguity.&#10;model theory\ \ Model theory deals with representing the&#10;relationship between language and the world.&#10;dialogue\ \ Dialogue semantics is sometimes thought of as&#10;part of discourse semantics. However, there are many phenomena that&#10;occur in conversations with two or more participants that do not occur&#10;in texts. These include fragmentary utterances, repair phenomena,&#10;utterances split between different dialogue participants and overlap&#10;(dialogue participants speaking at the same time). This seems to&#10;warrant dialogue being treated as a feature separate from discourse.&#10;similarity of meaning\ \ In addition to meaning relations&#10;such as entailment there is a notion of words, phrases and sentences&#10;having similar meanings in various respects.&#10;robust non-logical inference\ \ This type of inference is&#10;discussed in e.g. the work on textual entailment. Rather than&#10;representing something that follows logically, it corresponds to what&#10;conclusions people might draw from a given utterance or text, is often&#10;reliant on background knowledge and is to a large extent defeasible.&#10;dynamic changes of meaning\ \ The meaning of words and&#10;phrases can change over time and during the course of a text or&#10;dialogue.&#10;grounding meaning in action and perception\ \ While model&#10;theory purports to relate language and the world it tells us little&#10;about how we relate our perception of the world and action in the&#10;world to the meaning of words and phrases. Such issues become&#10;important, for example, if we want to put natural language on board a&#10;robot.&#10;multimodality\ \ The multimodal nature of communication&#10;becomes obvious when you begin to think of meaning in terms of action&#10;and perception.&#10;&#10;The rigour of the work on semantics by Richard Montague&#10;\cite{Montague1973,Partee1976} inspired early work on computational&#10;semantics \cite[perhaps the earliest&#10;was][]{FriedmanWarren1978,FriedmanMoranWarren1978}. Two high-points&#10;of the literature on computational semantics based on Montague are&#10;\cite{BlackburnBos2005}, using logic programming, and&#10;\cite{EijckUnger2010}, using functional programming. Montague's&#10;semantic techniques have also played an important role in semantic&#10;treatments using Combinatory Categorial Grammar&#10;\cite[CCG,][]{BosClarkSteedmanCurranHockenmaier2004}.&#10;&#10;One problem with Montague's treatment of semantics was that it was&#10;limited to the level of the sentence. It could not, for example, deal&#10;with cross-sentence anaphora such as A dog$_i$ barked. It$_i$&#10; was upset by the intruder. This, among several other things, led&#10;to the development of Discourse Representation Theory&#10;\cite[DRT,][]{KampReyle1993,KampGenabithReyle2011} and other variants&#10;of dynamic semantics such as \cite{Heim1982} and&#10;\cite{GroenendijkStokhof1991}. Here ``dynamic'' is meant in the sense&#10;of treating semantic content as context change potential in order,&#10;among other things, to be able to pass referents from one sentence to&#10;a subsequent sentence in the discourse. This is a much less radical&#10;notion of dynamic interpretation than we discuss in&#10;Section~\ref{sec:dyn}, where the meaning associated with a word or&#10;phrase may change as a dialogue progresses. DRT has played an&#10;important role in computational semantics from early work on the&#10;Verbmobil project \cite{BosGambaeckLieskeMoriPinkalWorm1996} to work&#10;by Johan Bos and others on the Groningen Meaning Bank&#10;() and the Parallel Meaning Bank&#10;().&#10;&#10;What do we get from this body of work? Here are some of the features&#10;that we can get in a compositional semantics based on this work.&#10;compositionality\ \ Compositionality is one of the&#10;cornerstones of Montague's approach.&#10;logic-based inference\ \ The ability to derive conclusions&#10;based on logical inference and the ability to characterise ``meaning&#10;postulates'' is a central feature of semantics in the Montague&#10;tradition. Defeasible reasoning has been added to this kind of&#10;framework \cite[e.g.,][]{AsherLascarides2003} and systems have been&#10;connected to theorem provers and model builders&#10;\cite{BlackburnBos2005}.&#10;discourse semantics\ \ The variants of dynamic semantics&#10;discussed above gave us the ability to treat discourse phenomena (that&#10;is, phenomena occurring in texts or utterances of more than a single&#10;sentence, including cases of discourse anaphora).&#10;underspecification\ \ While there is some work on&#10;underspecification of meaning in the theoretical literature&#10;\cite{Reyle1993}, the most interest has been devoted to it in&#10;computational work based on formal semantics \cite[such&#10;as][]{Alshawi1992,Bos1996,CopestakeFlickingerPollardSag2005}.&#10;model theory\ \ Model theory associated with a formal&#10;approach to semantics can in computational terms relate to database&#10;query \cite{BlackburnBos2005,EijckUnger2010}.&#10;&#10;What we have sketched above might be called the classical canon of&#10;formal semantics as it relates to computational semantics. There is&#10;much that we would like to have for a computational semantics that is&#10;still lacking here. To some extent more recent developments address&#10;these gaps and to some extent they are addressed by other kinds of&#10;meaning representations we discuss later in the paper, though often at&#10;the expense of giving up on (or at least having difficulty with) the&#10;features that we listed above. Features lacking in the classical&#10;canon include:&#10;dialogue\ \ The notion that language is actually used in&#10;interaction between agents engaging in communication came quite late&#10;to formal semantics though there is now a significant body of&#10;theoretical work such as \cite{Ginzburg1994,Ginzburg2012}. This gave&#10;rise to the Information State Update approach to dialogue systems&#10;\cite{Larsson2002}. TTR \cite[a type theory with&#10;records,][]{Cooper2005a,Cooperinprepa} has played an important role in&#10;this.&#10;similarity of meaning\ \ The kind of meaning similarity that&#10;is discussed in connection with vector semantics (see&#10;Section~\ref{sec:distrib}) is hard to recreate in a formal meaning&#10;representation, though the use of record types in TTR suggests that a&#10;connection could be made.&#10;robust non-logical inference\ \ The kind of inference that is&#10;represented, for example, in work on textual entailment is hard to&#10;square with the logic-based inference discussed above. However, the&#10;work on topoi by \cite{Breitholtz2020}, perhaps coupled with&#10;probabilistic TTR \cite{CooperDobnikLappinLarsson2015}, is suggestive&#10;of a computational approach to this.&#10;dynamic changes of meaning\ \ Notions of meaning negotiation&#10;and coordination have become central in the literature on formal&#10;approaches to dialogue. We discuss this in Section~\ref{sec:dyn}.&#10;grounding meaning in action and perception\ \ This has become&#10;central to theories such as TTR and Dynamic Syntax&#10;\cite{KempsonCannGregoromichelakiChatzikyriakidis2016} and we discuss&#10;this in Section~\ref{sec:ground}.&#10;multimodality\ \ The multimodal nature of communication&#10;becomes obvious when you begin to think of meaning in terms of action&#10;and perception. We discuss this in Section~\ref{sec:body}.&#10;&#10;Above we have mentioned examples of formal approaches which attempt to&#10;incorporate features which are not present in the classical canon. An&#10;alternative strategy is to try to incorporate features from the&#10;classical canon in non-formal approaches \cite[for&#10;example,][]{CoeckeSadrzadehClark2010} or to combine aspects of&#10;non-formal and formal approaches in a single framework \cite[for&#10;example,][]{Larsson2013,ErkHerbelot2020}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Application Limits, Computer Science, Linguistics, Cognitive Science, Natural Language Meaning, Artificial Intelligence, Natural Language Processing, Representation Models" />
        </attvalues>
      </node>
      <node id="cmp-lg/9607031" label="cmp-lg/9607031">
        <attvalues>
          <attvalue for="0" value="Compositional Semantics in Verbmobil" />
          <attvalue for="1" value="  The paper discusses how compositional semantics is implemented in the&#10;Verbmobil speech-to-speech translation system using LUD, a description language&#10;for underspecified discourse representation structures. The description&#10;language and its formal interpretation in DRT are described as well as its&#10;implementation together with the architecture of the system's entire&#10;syntactic-semantic processing module. We show that a linguistically sound&#10;theory and formalism can be properly implemented in a system with (near)&#10;real-time requirements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1003.4394" label="1003.4394">
        <attvalues>
          <attvalue for="0" value="Mathematical Foundations for a Compositional Distributional Model of&#10;  Meaning" />
          <attvalue for="1" value="  We propose a mathematical framework for a unification of the distributional&#10;theory of meaning in terms of vector space models, and a compositional theory&#10;for grammatical types, for which we rely on the algebra of Pregroups,&#10;introduced by Lambek. This mathematical framework enables us to compute the&#10;meaning of a well-typed sentence from the meanings of its constituents.&#10;Concretely, the type reductions of Pregroups are `lifted' to morphisms in a&#10;category, a procedure that transforms meanings of constituents into a meaning&#10;of the (well-typed) whole. Importantly, meanings of whole sentences live in a&#10;single space, independent of the grammatical structure of the sentence. Hence&#10;the inner-product can be used to compare meanings of arbitrary sentences, as it&#10;is for comparing the meanings of words in the distributional model. The&#10;mathematical structure we employ admits a purely diagrammatic calculus which&#10;exposes how the information flows between the words in a sentence in order to&#10;make up the meaning of the whole sentence. A variation of our `categorical&#10;model' which involves constraining the scalars of the vector spaces to the&#10;semiring of Booleans results in a Montague-style Boolean-valued semantics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.07936" label="2009.07936">
        <attvalues>
          <attvalue for="0" value="How to marry a star: probabilistic constraints for meaning in context" />
          <attvalue for="1" value="  In this paper, we derive a notion of 'word meaning in context' that&#10;characterizes meaning as both intensional and conceptual. We introduce a&#10;framework for specifying local as well as global constraints on word meaning in&#10;context, together with their interactions, thus modelling the wide range of&#10;lexical shifts and ambiguities observed in utterance interpretation. We&#10;represent sentence meaning as a 'situation description system', a probabilistic&#10;model which takes utterance understanding to be the mental process of&#10;describing to oneself one or more situations that would account for an observed&#10;utterance. We show how the system can be implemented in practice, and apply it&#10;to examples containing various contextualisation phenomena.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.14659" label="1910.14659">
        <attvalues>
          <attvalue for="0" value="Masked Language Model Scoring" />
          <attvalue for="1" value="  Pretrained masked language models (MLMs) require finetuning for most NLP&#10;tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood&#10;scores (PLLs), which are computed by masking tokens one by one. We show that&#10;PLLs outperform scores from autoregressive language models like GPT-2 in a&#10;variety of tasks. By rescoring ASR and NMT hypotheses, RoBERTa reduces an&#10;end-to-end LibriSpeech model's WER by 30% relative and adds up to +1.7 BLEU on&#10;state-of-the-art baselines for low-resource translation pairs, with further&#10;gains from domain adaptation. We attribute this success to PLL's unsupervised&#10;expression of linguistic acceptability without a left-to-right bias, greatly&#10;improving on scores from GPT-2 (+10 points on island effects, NPI licensing in&#10;BLiMP). One can finetune MLMs to give scores without masking, enabling&#10;computation in a single inference pass. In all, PLLs and their associated&#10;pseudo-perplexities (PPPLs) enable plug-and-play use of the growing number of&#10;pretrained MLMs; e.g., we use a single cross-lingual model to rescore&#10;translations in multiple languages. We release our library for language model&#10;scoring at https://github.com/awslabs/mlm-scoring.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.14989" label="2109.14989">
        <attvalues>
          <attvalue for="0" value="Structural Persistence in Language Models: Priming as a Window into&#10;  Abstract Language Representations" />
          <attvalue for="1" value="  We investigate the extent to which modern, neural language models are&#10;susceptible to structural priming, the phenomenon whereby the structure of a&#10;sentence makes the same structure more probable in a follow-up sentence. We&#10;explore how priming can be used to study the potential of these models to learn&#10;abstract structural information, which is a prerequisite for good performance&#10;on tasks that require natural language understanding skills. We introduce a&#10;novel metric and release Prime-LM, a large corpus where we control for various&#10;linguistic factors which interact with priming strength. We find that&#10;Transformer models indeed show evidence of structural priming, but also that&#10;the generalisations they learned are to some extent modulated by semantic&#10;information. Our experiments also show that the representations acquired by the&#10;models may not only encode abstract sequential structure but involve certain&#10;level of hierarchical syntactic information. More generally, our study shows&#10;that the priming paradigm is a useful, additional tool for gaining insights&#10;into the capacities of language models and opens the door to future&#10;priming-based investigations that probe the model's internal states.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.01780" label="2401.01780">
        <attvalues>
          <attvalue for="0" value="Navigating Uncertainty: Optimizing API Dependency for Hallucination&#10;  Reduction in Closed-Book Question Answering" />
          <attvalue for="1" value="  While Large Language Models (LLM) are able to accumulate and restore&#10;knowledge, they are still prone to hallucination. Especially when faced with&#10;factual questions, LLM cannot only rely on knowledge stored in parameters to&#10;guarantee truthful and correct answers. Augmenting these models with the&#10;ability to search on external information sources, such as the web, is a&#10;promising approach to ground knowledge to retrieve information. However,&#10;searching in a large collection of documents introduces additional&#10;computational/time costs. An optimal behavior would be to query external&#10;resources only when the LLM is not confident about answers. In this paper, we&#10;propose a new LLM able to self-estimate if it is able to answer directly or&#10;needs to request an external tool. We investigate a supervised approach by&#10;introducing a hallucination masking mechanism in which labels are generated&#10;using a close book question-answering task. In addition, we propose to leverage&#10;parameter-efficient fine-tuning techniques to train our model on a small amount&#10;of data. Our model directly provides answers for $78.2\%$ of the known queries&#10;and opts to search for $77.2\%$ of the unknown ones. This results in the API&#10;being utilized only $62\%$ of the time.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models have demonstrated remarkable performances in a wide range of Natural Language Processing (NLP) tasks, including conversational agents, summarization, translation, and question-answering \cite{gpt3,spark,emergent}. As scaling these models increases their ability to incorporate more and more knowledge \cite{gpt3,palm}, instead of relying on traditional search engines, Metzler et al. \cite{rethinkingsearch} suggest using LLM as a unified knowledge base able to perform question answering as well as document retrieval. However, even the larger models \cite{gpt3} are prone to producing inaccurate or false responses, commonly known as hallucinations \cite{10.1145/3571730}. These have been extensively explored in various NLP tasks, including summarization and translation \cite{NEURIPS2021_e4d2b6e6,guerreiro-etal-2023-looking,manakul2023selfcheckgpt,10.1145/3571730,lee-etal-2021-towards}.&#10;Numerous approaches have been suggested to tackle this problem, with all of them employing external techniques to detect and mitigate hallucinations. In question answering, retrieval augmented methods such as REALM \cite{realm}, RAG \cite{rag}, or RETRO \cite{retro,orqa}, were proposed to reduce LLM's hallucinations. These approaches consist in grounding LLM with a retriever model to add context from a large corpus of documents and to generate answers. These architectures are effective as they both improve factualness and reduce hallucinations for specific knowledge-intensive tasks such as Open-domain Question Answering \cite{orqa}. &#10;However, retrieved documents are always considered without consideration of their helpfulness in solving the task. In a second line of work, models, such as LaMDA, BlenderBot, WebGPT, Toolformer \cite{thoppilan2022lamda,nakano2022webgpt,shuster2022blenderbot,schick2023toolformer} are specifically trained to generate a query and rely on a search engine when confronted with questions. &#10;While these LLMs accumulated a lot of knowledge during pre-training, they are fine-tuned to rely on external databases for each question, without considering the model’s inherent ability to answer the question. Toolformer calls the web API for almost all the questions, $99.3\%$ with no real discernment between directly answerable questions and the real need for external knowledge. LLMs have accumulated a lot of information and may be able to answer directly when confronted with widely known facts \cite{gpt3,spark,emergent}. In this paper, we study a more nuanced approach that leverages external knowledge while also incorporating LLMs' intrinsic knowledge. We, therefore, propose a model that either generates a natural language answer or an API call (e.g. $\langle search \rangle$) only when the model is not self-confident about the answer, minimizing the dependency on external resources helps to save inference time and computational costs. We focus on closed-book question-answering (CBQA) tasks and carried out on two datasets (Natural Questions (NQ) \cite{kwiatkowski-etal-2019-natural} and TriviaQA (TQA) \cite{joshi-etal-2017-triviaqa}). We study how LLMs perform at self-estimating their ability to correctly answer factual questions. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Machine Learning, Computer Science, Large Language Models, Model Confidence Estimation, Artificial Intelligence, Information Science, External Knowledge Retrieval" />
        </attvalues>
      </node>
      <node id="2105.02274" label="2105.02274">
        <attvalues>
          <attvalue for="0" value="Rethinking Search: Making Domain Experts out of Dilettantes" />
          <attvalue for="1" value="  When experiencing an information need, users want to engage with a domain&#10;expert, but often turn to an information retrieval system, such as a search&#10;engine, instead. Classical information retrieval systems do not answer&#10;information needs directly, but instead provide references to (hopefully&#10;authoritative) answers. Successful question answering systems offer a limited&#10;corpus created on-demand by human experts, which is neither timely nor&#10;scalable. Pre-trained language models, by contrast, are capable of directly&#10;generating prose that may be responsive to an information need, but at present&#10;they are dilettantes rather than domain experts -- they do not have a true&#10;understanding of the world, they are prone to hallucinating, and crucially they&#10;are incapable of justifying their utterances by referring to supporting&#10;documents in the corpus they were trained over. This paper examines how ideas&#10;from classical information retrieval and pre-trained language models can be&#10;synthesized and evolved into systems that truly deliver on the promise of&#10;domain expert advice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.00300" label="1906.00300">
        <attvalues>
          <attvalue for="0" value="Latent Retrieval for Weakly Supervised Open Domain Question Answering" />
          <attvalue for="1" value="  Recent work on open domain question answering (QA) assumes strong supervision&#10;of the supporting evidence and/or assumes a blackbox information retrieval (IR)&#10;system to retrieve evidence candidates. We argue that both are suboptimal,&#10;since gold evidence is not always available, and QA is fundamentally different&#10;from IR. We show for the first time that it is possible to jointly learn the&#10;retriever and reader from question-answer string pairs and without any IR&#10;system. In this setting, evidence retrieval from all of Wikipedia is treated as&#10;a latent variable. Since this is impractical to learn from scratch, we&#10;pre-train the retriever with an Inverse Cloze Task. We evaluate on open&#10;versions of five QA datasets. On datasets where the questioner already knows&#10;the answer, a traditional IR system such as BM25 is sufficient. On datasets&#10;where a user is genuinely seeking an answer, we show that learned retrieval is&#10;crucial, outperforming BM25 by up to 19 points in exact match.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.03188" label="2208.03188">
        <attvalues>
          <attvalue for="0" value="BlenderBot 3: a deployed conversational agent that continually learns to&#10;  responsibly engage" />
          <attvalue for="1" value="  We present BlenderBot 3, a 175B parameter dialogue model capable of&#10;open-domain conversation with access to the internet and a long-term memory,&#10;and having been trained on a large number of user defined tasks. We release&#10;both the model weights and code, and have also deployed the model on a public&#10;web page to interact with organic users. This technical report describes how&#10;the model was built (architecture, model and training scheme), and details of&#10;its deployment, including safety mechanisms. Human evaluations show its&#10;superiority to existing open-domain dialogue agents, including its predecessors&#10;(Roller et al., 2021; Komeili et al., 2022). Finally, we detail our plan for&#10;continual learning using the data collected from deployment, which will also be&#10;publicly released. The goal of this research program is thus to enable the&#10;community to study ever-improving responsible agents that learn through&#10;interaction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Open-domain dialogue models&#10;&#10;While open-domain dialogue has a rich history \cite{chen2017survey,gao2019neural,ni2021recent}&#10;the area has made significant recent &#10;progress by pre-training ever-larger neural models. For example, &#10;the ConvAI2 competition at NeurIPS 2018 featured&#10;large (at the time) pre-trained transformers being used by&#10;the top two winning teams \cite{wolf2019transfertransfo,golovanov2020lost,dinan2019second}.&#10;In 2019, the 762M parameter DialoGPT model was released \cite{zhang2019dialogpt}, and in 2020 &#10; the 3B parameter Meena model was published \cite{adiwardana2020meena} and&#10;the 9B parameter BlenderBot model was released \cite{roller-etal-2021-recipes}.&#10;In 2022, the 137B parameter LaMDA model was published \cite{cohen2022lamda}.&#10;We note that some of these models are openly available to allow the community to conduct reproducible research, such as DialoGPT and BlenderBot, while others, such as Meena and LaMDA, have not released models or datasets, &#10;and hence cannot be easily compared to or built upon.&#10;Similarly proprietary models \cite{zhou2020design} or data \cite{ram2018conversational} from several other products have not been openly released.&#10;&#10;Besides trying to pre-train for dialogue modeling&#10;directly, it has been observed that&#10;language model pre-training such as in GPT3 \cite{brown2020language}&#10;or Gopher \cite{rae2021scaling}&#10;is also useful for downstream dialogue applications. OPT-175B &#10;\cite{zhang2022opt} and &#10;BLOOM are some of the most openly accessible of such systems, &#10;with models like Gopher being inaccessible, or in the case of GPT3 interaction is through a paid API, &#10; with full research access being limited.&#10;&#10;Several approaches have also shown that not only is pre-training a large model with language modeling or conversational data important, but appropriate fine-tuning of those models also brings significant further gains \cite{roller-etal-2021-recipes,cohen2022lamda,ouyang2022training,bai2022training}.&#10;A number of fine-tuning datasets are crowdsourced and publicly &#10;released for use by the research community \cite{serban2015survey,huang2020challenges}, &#10;such as the ones we will use &#10;for training the BlenderBot 3 model in this work (see \ref{sec:training}).&#10;&#10;Many of these recent models use sequence to sequence transformer models to map from dialogue context to output, without any access to knowledge from the outside world beyond their original training data, which can become stale and produce hallucinations \cite{shuster2021retrieval}.&#10;BlenderBot 2 \cite{bb2} extended its predecessor by allowing the bot to ground its conversation on retrieval from the internet for open-domain dialogue tasks \cite{komeili2021internet}, where the tasks were also publicly released.&#10;Since then, WebGPT \cite{nakano2021webgpt} also applies internet search to QA (but not dialogue) tasks,&#10;as does the work of \cite{lazaridou2022internetaugmented},&#10;while LaMDA uses information retrieval for general dialogue.&#10;BlenderBot 3 extends its predecessor in this regard, with further fine-tune data covering more internet-based skills that we also publicly release.&#10;&#10;Continual learning and deployment&#10;Many existing systems, as described above,&#10;have been trained with fine-tuning datasets, typically with supervised&#10;targets that are human-authored responses.&#10;These are commonly collected via expert annotators or crowdworkers \cite{serban2015survey}.&#10;Careful instructions \cite{huynh2021survey} can result in good quality feedback or labels to learn from; however, the distribution of data, which is typically decided by those instructions, is unlikely to match the changing desires of organic users, and takes significant resources to collect. &#10;An alternative approach is to deploy a system publicly, and collect interaction data and feedback from organic users directly. The promise of such an approach is that the distribution of data will more closely match those organic users' desires, rather than decided by the researchers themselves when creating datasets \cite{gabriel2020further,roller2020open,shuster2020deploying,ouyang2022training}. Further, continued deployment of such a system, with appropriate learning systems, could then potentially keep improving over time \cite{carlson2010toward,kiela2021dynabench,agichtein2006improving,liu2021lifelong,madotto2020continual,shuster2020deploying}, where \cite{hancock2019learning} refer to this approach as a {self-feeding chatbot}. The challenge, however, is that organic users may not be invested enough to want to provide adequate feedback, and some may be adversarial \cite{park2021use} as in the case of Microsoft's Tay \cite{davis2016ai}.&#10;&#10;There are a number of ways to learn from user interaction data. &#10;Firstly, if conversations are relatively symmetric between conversational partners, the human side of the conversation can directly be used as a target &#10;for the model to mimic, which makes the learning algorithm straightforward.&#10;This was shown to give large improvements in the deployed LIGHT system \cite{shuster2020deploying}. Such an approach is not directly applicable if the conversations are asymmetric, for example in the case of humans treating the bot like an assistant (whereas they do not want the bot to treat them like an assistant). In that case, other learning methods should be explored. \cite{li2016learning} studies models that learn how to sometimes ask appropriate questions in order to learn from the answers, while \cite{li2016dialogue} learns from general textual feedback/comments from the user, particularly in the case where the bot has produced a low quality response. &#10;Another approach is to learn a reward signal (positive or negative reaction) based on user textual responses, as shown in the self-feeding chatbot \cite{hancock2019learning}. Alternatively, rather than learning from the conversation itself, one can augment the messaging system with a user interface that collects appropriate data, for example stack ranking potential responses \cite{ouyang2022training,bai2022training}. &#10;&#10;Outside of the dialogue domain, there is also a rich body of work studying the improvement of models from deployment, including never-ending-learning from language data \cite{carlson2010toward}, improving web search \cite{agichtein2006improving}, the Dynabench system &#10;which evaluates a number of NLP tasks \cite{kiela2021dynabench}, or learning from feedback to improve summarization \cite{saunders2022self}. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="1705.03551" label="1705.03551">
        <attvalues>
          <attvalue for="0" value="TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for&#10;  Reading Comprehension" />
          <attvalue for="1" value="  We present TriviaQA, a challenging reading comprehension dataset containing&#10;over 650K question-answer-evidence triples. TriviaQA includes 95K&#10;question-answer pairs authored by trivia enthusiasts and independently gathered&#10;evidence documents, six per question on average, that provide high quality&#10;distant supervision for answering the questions. We show that, in comparison to&#10;other recently introduced large-scale datasets, TriviaQA (1) has relatively&#10;complex, compositional questions, (2) has considerable syntactic and lexical&#10;variability between questions and corresponding answer-evidence sentences, and&#10;(3) requires more cross sentence reasoning to find answers. We also present two&#10;baseline algorithms: a feature-based classifier and a state-of-the-art neural&#10;network, that performs well on SQuAD reading comprehension. Neither approach&#10;comes close to human performance (23% and 40% vs. 80%), suggesting that&#10;TriviaQA is a challenging testbed that is worth significant future study. Data&#10;and code available at -- http://nlp.cs.washington.edu/triviaqa/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.09075" label="2006.09075">
        <attvalues>
          <attvalue for="0" value="PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized&#10;  Embedding Models" />
          <attvalue for="1" value="  Pivot-based neural representation models have lead to significant progress in&#10;domain adaptation for NLP. However, previous works that follow this approach&#10;utilize only labeled data from the source domain and unlabeled data from the&#10;source and target domains, but neglect to incorporate massive unlabeled corpora&#10;that are not necessarily drawn from these domains. To alleviate this, we&#10;propose PERL: A representation learning model that extends contextualized word&#10;embedding models such as BERT with pivot-based fine-tuning. PERL outperforms&#10;strong baselines across 22 sentiment classification domain adaptation setups,&#10;improves in-domain model performance, yields effective reduced-size models and&#10;increases model stability.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Processing (NLP) algorithms are constantly improving, gradually approaching human level performance \cite{Dozat:17,Edunov:18,Radford:18}. However, those algorithms often depend on the availability of large amounts of manually annotated data from the domain where the task is performed. Unfortunately, collecting such annotated data is often costly and laborious, which substantially limits the applicability of NLP technology.&#10;&#10;Domain Adaptation (DA), training an algorithm on annotated data from a source domain so that it can be effectively applied to other target domains, is one of the ways to solve the above bottleneck. Indeed, over the years substantial efforts have been devoted to the DA challenge \cite{roark2003supervised, daume2006domain, ben2010theory, jiang2007instance, mcclosky2010automatic, rush2012improved, schnabel2014flors}. Our focus in this paper is on unsupervised DA, the setup we consider most realistic. In this setup labeled data is available only from the source domain while unlabeled data is available from both the source and the target domains.&#10;&#10;While various approaches for DA have been proposed (\S \ref{sec:previous}), with the prominence of deep neural network (DNN) modeling, attention has been recently focused on representation learning approaches.&#10;Within representation learning for unsupervised DA, two approaches have been shown particularly useful. In one line of work, DNN-based methods which employ compress-based noise reduction to learn cross-domain features have been developed \cite{glorot2011domain, chen2012marginalized}. In another line of work, methods based on the distinction between pivot and non-pivot features \cite{blitzer2006domain, blitzer2007biographies} learn a joint feature representation for the source and the target domains. Later on, \newcite{ziser2016neural,ziser2018pivot}, and \newcite{li2018hierarchical} married the two approaches and achieved substantial improvements on a variety of DA setups.&#10;&#10;Despite their success, pivot-based DNN models still only utilize labeled&#10;data from the source domain and unlabeled data from both the source and the target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. With the recent game-changing success of contextualized word embedding models trained on such massive corpora \cite{devlin2018bert, peters2018deep}, it is natural to ask whether information from such corpora can enhance these DA methods, particularly that background knowledge from non-contextualized embeddings has shown useful for DA \cite{DBLP:conf/acl/PlankM13, DBLP:conf/acl/NguyenPG15}.&#10;&#10;In this paper we hence propose an unsupervised DA approach that&#10;extends leading approaches based on DNNs and pivot-based ideas, so that they can incorporate information encoded in massive corpora (\S \ref{sec:DA with PERL}). Our model, named PERL: Pivot-based Encoder Representation of Language, builds on massively pre-trained contextualized word embedding models such as BERT \cite{devlin2018bert}. To adjust the representations learned by these models so that they close the gap between the source and target domains, we fine-tune their parameters using a pivot-based variant of the Masked Language Modeling (MLM) objective, optimized on unlabeled data from both the source and the target domains. We further present R-PERL (regularized PERL) which&#10;facilitates parameter sharing for pivots with similar meaning.&#10;&#10;We perform extensive experimentation in various unsupervised DA setups of the task of binary sentiment classification (\S \ref{sec:experiments}, \ref{sec:results}). First, for compatibility with previous work, we experiment with the legacy product review domains of \newcite{blitzer2007biographies} (12 setups). We then experiment with more challenging setups, adapting between the above domains and the airline review domain \cite{Nguyen2015airline} used in \newcite{ziser2018pivot} (4 setups), as well as the IMDB movie review domain \cite{maas2011learning} (6 setups). We compare PERL to the best performing pivot-based methods \cite{ziser2018pivot,li2018hierarchical} and to DA approaches that fine-tune a massively pre-trained BERT model by optimizing its standard MLM objective using target-domain unlabeled data \cite{lee2019biobert,han2019unsupervised}.&#10;PERL and R-PERL substantially outperform these baselines, emphasizing the additive effect of massive pre-training and pivot-based fine-tuning. &#10;&#10;As an additional contribution, we show that pivot-based learning is effective beyond improving domain adaptation accuracy. Particularly, we show that an in-domain variant of PERL substantially improves the in-domain performance of a BERT-based sentiment classifier, for varying training set sizes (from 100 to 20K labeled examples). We also show that PERL facilitates the generation of effective reduced-size DA models. Finally, we perform an extensive ablation study (\S \ref{sec:ablation}) that uncovers PERL's crucial design choices and demonstrates the stability of PERL to hyper-parameter selection compared to other DA methods. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Representation Models, Domain Adaptation, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1611.01734" label="1611.01734">
        <attvalues>
          <attvalue for="0" value="Deep Biaffine Attention for Neural Dependency Parsing" />
          <attvalue for="1" value="  This paper builds off recent work from Kiperwasser &amp; Goldberg (2016) using&#10;neural attention in a simple graph-based dependency parser. We use a larger but&#10;more thoroughly regularized parser than other recent BiLSTM-based approaches,&#10;with biaffine classifiers to predict arcs and labels. Our parser gets state of&#10;the art or near state of the art performance on standard treebanks for six&#10;different languages, achieving 95.7% UAS and 94.1% LAS on the most popular&#10;English PTB dataset. This makes it the highest-performing graph-based parser on&#10;this benchmark---outperforming Kiperwasser Goldberg (2016) by 1.8% and&#10;2.2%---and comparable to the highest performing transition-based parser&#10;(Kuncoro et al., 2016), which achieves 95.8% UAS and 94.6% LAS. We also show&#10;which hyperparameter choices had a significant effect on parsing accuracy,&#10;allowing us to achieve large gains over other graph-based approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.09381" label="1808.09381">
        <attvalues>
          <attvalue for="0" value="Understanding Back-Translation at Scale" />
          <attvalue for="1" value="  An effective method to improve neural machine translation with monolingual&#10;data is to augment the parallel training corpus with back-translations of&#10;target language sentences. This work broadens the understanding of&#10;back-translation and investigates a number of methods to generate synthetic&#10;source sentences. We find that in all but resource poor settings&#10;back-translations obtained via sampling or noised beam outputs are most&#10;effective. Our analysis shows that sampling or noisy synthetic data gives a&#10;much stronger training signal than data generated by beam or greedy search. We&#10;also compare how synthetic data compares to genuine bitext and study various&#10;domain effects. Finally, we scale to hundreds of millions of monolingual&#10;sentences and achieve a new state of the art of 35 BLEU on the WMT'14&#10;English-German test set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.18397" label="2402.18397">
        <attvalues>
          <attvalue for="0" value="Decomposed Prompting: Unveiling Multilingual Linguistic Structure&#10;  Knowledge in English-Centric Large Language Models" />
          <attvalue for="1" value="  Despite the predominance of English in their training data, English-centric&#10;Large Language Models (LLMs) like GPT-3 and LLaMA display a remarkable ability&#10;to perform multilingual tasks, raising questions about the depth and nature of&#10;their cross-lingual capabilities. This paper introduces the decomposed&#10;prompting approach to probe the linguistic structure understanding of these&#10;LLMs in sequence labeling tasks. Diverging from the single text-to-text prompt,&#10;our method generates for each token of the input sentence an individual prompt&#10;which asks for its linguistic label. We assess our method on the Universal&#10;Dependencies part-of-speech tagging dataset for 38 languages, utilizing both&#10;English-centric and multilingual LLMs. Our findings show that decomposed&#10;prompting surpasses the iterative prompting baseline in efficacy and efficiency&#10;under zero- and few-shot settings. Further analysis reveals the influence of&#10;evaluation methods and the use of instructions in prompts. Our multilingual&#10;investigation shows that English-centric language models perform better on&#10;average than multilingual models. Our study offers insights into the&#10;multilingual transferability of English-centric LLMs, contributing to the&#10;understanding of their multilingual linguistic knowledge.&#10;" />
          <attvalue for="2" value="&#10;Current Large Language Models (LLMs), such as GPT-3, GPT-4, PaLM, and LLaMA~\cite{brown2020language, chowdhery2023palm, touvron2023llama}, have demonstrated remarkable capabilities in in-context learning, also known as prompting, across a broad spectrum of language understanding and generation tasks~\cite{zhao2023survey, zhang2023instruction, ziyu-etal-2023-lens}. These models are predominantly trained on massive amounts of English text data, with some limited exposure to other languages. For instance, LLaMA2's pretraining corpus comprises over 89\% English content~\cite{touvron2023llama2}. Yet, these English-centric LLMs~ still exhibit effective performance in multilingual evaluations~\cite{lai-etal-2023-chatgpt}. In a multilingual prompting scenario designed for zero-shot transfer with LLMs, the model executes tasks by directly generating outputs based on a task description and/or a few examples provided in a pivot language (typically English), along with input in a different target language~\cite{ahuja-etal-2023-mega}.&#10;However, the extent and nature of their cross-lingual capabilities remain underexplored~\cite{ye2023language}. This raises a critical question: Does the multilinguality of these models stem from a deep, generalizable multilingual linguistic understanding, or merely from the superficial alignment of lexical patterns across languages? &#10;&#10;Given the demonstrated proficiency of English-centric LLMs in multilingual tasks that demand profound language understanding~\cite{deng2023multilingual, wang2023all}, we hypothesize that these models harbor substantial multilingual knowledge. This knowledge, particularly relating to linguistic structure, is commonly conceptualized through sequence tagging tasks~\cite{jurafsky2000speech}. However, the current prompting strategies designed for sequence labeling in LLMs are not well suited to test. &#10;For instance, behavioral probing methods~\cite{belinkov-etal-2020-interpretability}, aimed at measuring knowledge stored in language models, struggle to adapt to tasks predicting more complex structures. &#10;Additionally, text-to-text prompting methods~\cite{asai2023buffet}, which rely on a predefined output template, face challenges in maintaining control over the output format.&#10;In response to these challenges, a decent iterative prompting strategy for structured prediction has been introduced, addressing the aforementioned limitations~\cite{blevins-etal-2023-prompting}. Despite its advantages, this method presents its own challenges, such as longer processing times due to its iterative inference strategy.&#10;&#10;To overcome the challenges identified in probing the multilingual knowledge of linguistic structure in LLMs, we introduce the decomposed prompting strategy. This strategy is inspired by the ToPro method~\cite{ma2024topro}, a novel prompt-based fine-tuning approach for sequence labeling tasks. We adopt this idea to the in-context learning paradigm, aiming to probe English-centric LLMs for their understanding of token-level linguistic structure framed as sequence labeling tasks. &#10;As shown in gref{fig1}, instead of employing a single text-to-text prompt for labeling an entire sequence in one step, our method decomposes this process into multiple discrete prompts.&#10;More precisely, we first split the input sentence into tokens. Subsequently, we generate an individual prompt for each token which inquires about its linguistic label.&#10;&#10;We evaluate our approach on the Universal Dependency (UD) part-of-speech (POS) tagging dataset~\cite{nivre-etal-2020-universal} covering 38 languages with 3 English-centric LLMs and 2 multilingual LLMs. Our approach outperforms the iterative prompting baseline in both zero- and few-shot settings in terms of accuracy and efficiency. We investigate the nuanced impact of evaluation methods and the usage of task instructions within prompts on the performance of decomposed prompting, followed by an empirical comparative study of decomposed and iterative prompting. Moreover, our analysis of multilingual efficacy of English-centric LLMs yields valuable insights into the transferability of linguistic knowledge via multilingual prompting. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Multilingual Language Models, Linguistic Structure Understanding, Artificial Intelligence, Natural Language Processing, Cross-Lingual Capabilities" />
        </attvalues>
      </node>
      <node id="2304.05613" label="2304.05613">
        <attvalues>
          <attvalue for="0" value="ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large&#10;  Language Models in Multilingual Learning" />
          <attvalue for="1" value="  Over the last few years, large language models (LLMs) have emerged as the&#10;most important breakthroughs in natural language processing (NLP) that&#10;fundamentally transform research and developments in the field. ChatGPT&#10;represents one of the most exciting LLM systems developed recently to showcase&#10;impressive skills for language generation and highly attract public attention.&#10;Among various exciting applications discovered for ChatGPT in English, the&#10;model can process and generate texts for multiple languages due to its&#10;multilingual training data. Given the broad adoption of ChatGPT for English in&#10;different problems and areas, a natural question is whether ChatGPT can also be&#10;applied effectively for other languages or it is necessary to develop more&#10;language-specific technologies. The answer to this question requires a thorough&#10;evaluation of ChatGPT over multiple tasks with diverse languages and large&#10;datasets (i.e., beyond reported anecdotes), which is still missing or limited&#10;in current research. Our work aims to fill this gap for the evaluation of&#10;ChatGPT and similar LLMs to provide more comprehensive information for&#10;multilingual NLP applications. While this work will be an ongoing effort to&#10;include additional experiments in the future, our current paper evaluates&#10;ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium,&#10;low, and extremely low resources. We also focus on the zero-shot learning&#10;setting for ChatGPT to improve reproducibility and better simulate the&#10;interactions of general users. Compared to the performance of previous models,&#10;our extensive experimental results demonstrate a worse performance of ChatGPT&#10;for different NLP tasks and languages, calling for further research to develop&#10;better models and understanding for multilingual learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.06688" label="2306.06688">
        <attvalues>
          <attvalue for="0" value="Language Versatilists vs. Specialists: An Empirical Revisiting on&#10;  Multilingual Transfer Ability" />
          <attvalue for="1" value="  Multilingual transfer ability, which reflects how well the models fine-tuned&#10;on one source language can be applied to other languages, has been well studied&#10;in multilingual pre-trained models (e.g., BLOOM). However, such ability has not&#10;been investigated for English-centric models (e.g., LLaMA). To fill this gap,&#10;we study the following research questions. First, does multilingual transfer&#10;ability exist in English-centric models and how does it compare with&#10;multilingual pretrained models? Second, does it only appears when English is&#10;the source language for the English-centric model? Third, how does it vary in&#10;different tasks? We take multilingual reasoning ability as our focus and&#10;conduct extensive experiments across four types of reasoning tasks. We find&#10;that the multilingual pretrained model does not always outperform an&#10;English-centric model. Furthermore, English appears to be a less suitable&#10;source language, and the choice of source language becomes less important when&#10;the English-centric model scales up. In addition, different types of tasks&#10;exhibit different multilingual transfer abilities. These findings demonstrate&#10;that English-centric models not only possess multilingual transfer ability but&#10;may even surpass the transferability of multilingual pretrained models if&#10;well-trained. By showing the strength and weaknesses, the experiments also&#10;provide valuable insights into enhancing multilingual reasoning abilities for&#10;the English-centric models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.06474" label="2310.06474">
        <attvalues>
          <attvalue for="0" value="Multilingual Jailbreak Challenges in Large Language Models" />
          <attvalue for="1" value="  While large language models (LLMs) exhibit remarkable capabilities across a&#10;wide range of tasks, they pose potential safety concerns, such as the&#10;``jailbreak'' problem, wherein malicious instructions can manipulate LLMs to&#10;exhibit undesirable behavior. Although several preventive measures have been&#10;developed to mitigate the potential risks associated with LLMs, they have&#10;primarily focused on English. In this study, we reveal the presence of&#10;multilingual jailbreak challenges within LLMs and consider two potential risky&#10;scenarios: unintentional and intentional. The unintentional scenario involves&#10;users querying LLMs using non-English prompts and inadvertently bypassing the&#10;safety mechanisms, while the intentional scenario concerns malicious users&#10;combining malicious instructions with multilingual prompts to deliberately&#10;attack LLMs. The experimental results reveal that in the unintentional&#10;scenario, the rate of unsafe content increases as the availability of languages&#10;decreases. Specifically, low-resource languages exhibit about three times the&#10;likelihood of encountering harmful content compared to high-resource languages,&#10;with both ChatGPT and GPT-4. In the intentional scenario, multilingual prompts&#10;can exacerbate the negative impact of malicious instructions, with&#10;astonishingly high rates of unsafe output: 80.92\% for ChatGPT and 40.71\% for&#10;GPT-4. To handle such a challenge in the multilingual context, we propose a&#10;novel \textsc{Self-Defense} framework that automatically generates multilingual&#10;training data for safety fine-tuning. Experimental results show that ChatGPT&#10;fine-tuned with such data can achieve a substantial reduction in unsafe content&#10;generation. Data is available at&#10;\url{https://github.com/DAMO-NLP-SG/multilingual-safety-for-LLMs}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14857" label="2305.14857">
        <attvalues>
          <attvalue for="0" value="BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual&#10;  Transfer" />
          <attvalue for="1" value="  Despite remarkable advancements in few-shot generalization in natural&#10;language processing, most models are developed and evaluated primarily in&#10;English. To facilitate research on few-shot cross-lingual transfer, we&#10;introduce a new benchmark, called BUFFET, which unifies 15 diverse tasks across&#10;54 languages in a sequence-to-sequence format and provides a fixed set of&#10;few-shot examples and instructions. BUFFET is designed to establish a rigorous&#10;and equitable evaluation framework for few-shot cross-lingual transfer across a&#10;broad range of tasks and languages. Using BUFFET, we perform thorough&#10;evaluations of state-of-the-art multilingual large language models with&#10;different transfer methods, namely in-context learning and fine-tuning. Our&#10;findings reveal significant room for improvement in few-shot in-context&#10;cross-lingual transfer. In particular, ChatGPT with in-context learning often&#10;performs worse than much smaller mT5-base models fine-tuned on English task&#10;data and few-shot in-language examples. Our analysis suggests various avenues&#10;for future research in few-shot cross-lingual transfer, such as improved&#10;pretraining, understanding, and future evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.07830" label="2211.07830">
        <attvalues>
          <attvalue for="0" value="Prompting Language Models for Linguistic Structure" />
          <attvalue for="1" value="  Although pretrained language models (PLMs) can be prompted to perform a wide&#10;range of language tasks, it remains an open question how much this ability&#10;comes from generalizable linguistic understanding versus surface-level lexical&#10;patterns. To test this, we present a structured prompting approach for&#10;linguistic structured prediction tasks, allowing us to perform zero- and&#10;few-shot sequence tagging with autoregressive PLMs. We evaluate this approach&#10;on part-of-speech tagging, named entity recognition, and sentence chunking,&#10;demonstrating strong few-shot performance in all cases. We also find that while&#10;PLMs contain significant prior knowledge of task labels due to task leakage&#10;into the pretraining corpus, structured prompting can also retrieve linguistic&#10;structure with arbitrary labels. These findings indicate that the in-context&#10;learning ability and linguistic knowledge of PLMs generalizes beyond&#10;memorization of their training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.04906" label="2004.04906">
        <attvalues>
          <attvalue for="0" value="Dense Passage Retrieval for Open-Domain Question Answering" />
          <attvalue for="1" value="  Open-domain question answering relies on efficient passage retrieval to&#10;select candidate contexts, where traditional sparse vector space models, such&#10;as TF-IDF or BM25, are the de facto method. In this work, we show that&#10;retrieval can be practically implemented using dense representations alone,&#10;where embeddings are learned from a small number of questions and passages by a&#10;simple dual-encoder framework. When evaluated on a wide range of open-domain QA&#10;datasets, our dense retriever outperforms a strong Lucene-BM25 system largely&#10;by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our&#10;end-to-end QA system establish new state-of-the-art on multiple open-domain QA&#10;benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.01282" label="2007.01282">
        <attvalues>
          <attvalue for="0" value="Leveraging Passage Retrieval with Generative Models for Open Domain&#10;  Question Answering" />
          <attvalue for="1" value="  Generative models for open domain question answering have proven to be&#10;competitive, without resorting to external knowledge. While promising, this&#10;approach requires to use models with billions of parameters, which are&#10;expensive to train and query. In this paper, we investigate how much these&#10;models can benefit from retrieving text passages, potentially containing&#10;evidence. We obtain state-of-the-art results on the Natural Questions and&#10;TriviaQA open benchmarks. Interestingly, we observe that the performance of&#10;this method significantly improves when increasing the number of retrieved&#10;passages. This is evidence that generative models are good at aggregating and&#10;combining evidence from multiple passages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.14999" label="2304.14999">
        <attvalues>
          <attvalue for="0" value="Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques&#10;  for LLMs" />
          <attvalue for="1" value="  As foundation models continue to exponentially scale in size, efficient&#10;methods of adaptation become increasingly critical. Parameter-efficient&#10;fine-tuning (PEFT), a recent class of techniques that require only modifying a&#10;small percentage of the model parameters, is currently the most popular method&#10;for adapting large language models (LLMs). Several PEFT techniques have&#10;recently been proposed with varying tradeoffs. We provide a comprehensive and&#10;uniform benchmark of various PEFT techniques across a representative LLM, the&#10;FLAN-T5 model, and evaluate model performance across different data scales of&#10;classification and generation datasets. Based on this, we provide a framework&#10;for choosing the optimal fine-tuning techniques given the task type and data&#10;availability. Contrary to popular belief, we also empirically prove that PEFT&#10;techniques converge slower than full tuning in low data scenarios, and posit&#10;the amount of data required for PEFT methods to both perform well and converge&#10;efficiently. Lastly, we further optimize these PEFT techniques by selectively&#10;choosing which parts of the model to train, and find that these techniques can&#10;be applied with significantly fewer parameters while maintaining and even&#10;improving performance.&#10;" />
          <attvalue for="2" value="&#10;As large language models become widely adopted, efficient training and deployment become critical requirements for enabling widespread usage. Each task that an LLM is fine-tuned on requires an entirely different set of weights. When models scale to hundreds of billions of parameters, hosting a different set of weights for each model becomes widely inefficient and cost prohibitive while reloading all the weights for different tasks is too slow. Parameter-efficient fine-tuning techniques aim to solve this problem by modifying a very small portion of weights relative to the full model size while keeping the rest of the model frozen \cite{mao2021unipelt}.&#10;&#10;At inference time, many adaptations of the same model can be served together by quickly swapping tiny submodules rather than all the weights. The current landscape of PEFT techniques is rapidly evolving and several PEFT techniques have recently been proposed -- each claiming to have advantages over the others in varying capacities. However, given that these techniques have each been evaluated in a silo on different models and datasets, it is unclear when to appropriately utilize one technique over another. This work seeks to provide a framework for evaluating how to effectively utilize PEFT by empirically evaluating which technique works well in what task types and how these techniques scale with data. Further, through an ablation study, our work seeks to understand which parts of the model are most important to train for a given task type and technique, leading to even more efficient adaptation and reduced parameter count. Our key contributions are: &#10;&#10;\begin{enumerate}&#10; \item Conducting a thorough comparison and analysis of the current state-of-the-art PEFT methods on the FLAN-T5 model across different data sizes and task types (generation/classification), evaluating a variety of dimensions including accuracy, convergence speeds and other relevant metrics. &#10; \item Performing ablation studies to better understand the relative importance of updating various parts of the model when adapting LLMs, considering layer ordering and submodule granularities, and further optimizing PEFT techniques to reduce the number of trained parameters and ultimately improve efficiency. &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Model Optimization, Linguistics, Efficient Fine-Tuning Methods, Mathematics, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2110.07577" label="2110.07577">
        <attvalues>
          <attvalue for="0" value="UniPELT: A Unified Framework for Parameter-Efficient Language Model&#10;  Tuning" />
          <attvalue for="1" value="  Recent parameter-efficient language model tuning (PELT) methods manage to&#10;match the performance of fine-tuning with much fewer trainable parameters and&#10;perform especially well when training data is limited. However, different PELT&#10;methods may perform rather differently on the same task, making it nontrivial&#10;to select the most appropriate method for a specific task, especially&#10;considering the fast-growing number of new PELT methods and tasks. In light of&#10;model diversity and the difficulty of model selection, we propose a unified&#10;framework, UniPELT, which incorporates different PELT methods as submodules and&#10;learns to activate the ones that best suit the current data or task setup via&#10;gating mechanism. On the GLUE benchmark, UniPELT consistently achieves 1~4%&#10;gains compared to the best individual PELT method that it incorporates and even&#10;outperforms fine-tuning under different setups. Moreover, UniPELT generally&#10;surpasses the upper bound that takes the best performance of all its submodules&#10;used individually on each task, indicating that a mixture of multiple PELT&#10;methods may be inherently more effective than single methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.06986" label="2402.06986">
        <attvalues>
          <attvalue for="0" value="Cacophony: An Improved Contrastive Audio-Text Model" />
          <attvalue for="1" value="  Despite recent advancements in audio-text modeling, audio-text contrastive&#10;models still lag behind their image-text counterparts in scale and performance.&#10;We propose a method to improve both the scale and the training of audio-text&#10;contrastive models. Specifically, we craft a large-scale audio-text dataset&#10;containing 13,000 hours of text-labeled audio, using pretrained language models&#10;to process noisy text descriptions and automatic captioning to obtain text&#10;descriptions for unlabeled audio samples. We first train on audio-only data&#10;with a masked autoencoder (MAE) objective, which allows us to benefit from the&#10;scalability of unlabeled audio datasets. We then, initializing our audio&#10;encoder from the MAE model, train a contrastive model with an auxiliary&#10;captioning objective. Our final model, which we name Cacophony, achieves&#10;state-of-the-art performance on audio-text retrieval tasks, and exhibits&#10;competitive results on the HEAR benchmark and other downstream tasks such as&#10;zero-shot classification.&#10;" />
          <attvalue for="2" value="&#10;&#10;Machine audition~\cite{wang2010machine} involves developing algorithms and systems for machines to analyze and understand sound, covering tasks such as audio tagging, acoustic scene classification, music classification, and sound event detection. In recent years, there has been a general shift away from individual audio pattern recognition tasks and toward general-purpose audio representations pretrained on large-scale audio datasets. &#10;Pretrained Audio Neural Networks (PANNs) ~\cite{kong2020panns} have played a significant role in this shift by demonstrating their versatility across various tasks and outperforming many advanced systems via fine-tuning. &#10;&#10;Modern approaches aim for robust performance in general-purpose audio understanding tasks without the need for task-specific fine-tuning, which offers more flexibility.&#10;These methods approach general audio understanding by linking text and audio modalities, referred to as audio-text models.&#10;One approach to linking text and audio is through generating response text given a combination of an audio prompt and a text prompt~\cite{gardner2023llark,gong2023listen,deshmukh2023pengi}.&#10;For instance, Pengi~\cite{deshmukh2023pengi} converts audio classification, retrieval, captioning, and audio question answering into a text generation task using audio and task-specific text prompts.&#10;Similarly, Qwen-Audio~\cite{chu2023qwen} addresses a variety of audio tasks through text generation but distinguishes itself by using text prompts consisting of hierarchical tag sequences inspired by Whisper~\cite{radford2023robust}.&#10;Both Pengi and Qwen-Audio support multiple closed-ended and open-ended audio tasks without the need for additional fine-tuning or task-specific extensions of the architecture.&#10;Another method of linking text and audio is contrastive learning.&#10;Pretrained contrastive models can also be applied directly to various downstream tasks without fine-tuning.&#10;For instance, contrastive models can be used for retrieval and classification by assigning a score that identifies the most probable text (or class label) from a predefined set of choices for a given audio input.&#10;Moreover, the learned audio-text representations from contrastive models offer the flexibility to use one modality during training and the other at inference, which can be applied in text-to-audio generation~\cite{huang2023make,liu2023audioldm} and language-guided source separation~\cite{liu2023separate,dong2022clipsep}.&#10;&#10;In this paper, we focus on improving contrastive audio-text models for general sounds by addressing two critical limitations in existing research: insufficient dataset scale and the vanilla contrastive training techniques.&#10;To tackle the dataset scale issue, we first examine the publicly available audio datasets. &#10;These can be categorized into three types based on label granularity: clean-labeled, noisy-labeled, and weakly-labeled or unlabeled, as illustrated in Fig.~\ref{fig:audiodata}. &#10;Clean-labeled datasets, while high-quality, are limited in size. Noisy-labeled datasets offer more samples but include extraneous details. &#10;The largest category by far is weakly-labeled or unlabeled data, which provides little to no textual information.&#10;This scarcity of high-quality labeled data presents a significant challenge in audio-text contrastive learning, especially when compared to recent advancements in image-text models. &#10;For context, while the largest public audio-text datasets contain less than 100,000 pairs~\cite{kim2019audiocaps,drossos2020clotho}, image-text models like CLIP~\cite{radford2021learning} and SigLip~\cite{zhai2023sigmoid} utilize 400 million and 3.6 billion pairs respectively.&#10;&#10;Previous works have attempted to address this data scarcity issue by collecting data from various sources and applying natural language processing techniques to clean or filter noisy captions.&#10;For instance, Huang et al.~\cite{huang2022mulan} collected approximately 44 million 30-second music clips, applying a pretrained classifier and rule-based filtering to clean associated metadata. &#10;This process intensively reduced the dataset to 2 million music-text pairs. &#10;Notably, they found that models trained on large scale unfiltered audio-text data performed comparably to those trained on filtered data in music tagging and retrieval tasks, suggesting that data quantity might be as crucial as quality in this domain.&#10;Wu et al.~\cite{laionclap2023}, in collaboration with Large-scale Artificial Intelligence Open Network (LAION), curate the LAION-Audio dataset with 630K audio-text pairs together with 2 million Audioset~\cite{gemmeke2017audio} clips recaptioned with keyword-to-caption (K2C) augmentation to train LAION-CLAP.&#10;The K2C method uses a pretrained language model to generate captions from tags.&#10;However, the captions produced via K2C are restricted to the objects defined by the tags, offering limited descriptive details.&#10;K2C also risks making incorrect assumptions or introducing biases highlighted in~\cite{laionclap2023}.&#10;Mei et al.~\cite{mei2023wavcaps} propose a multi-stage data filtering pipeline and utilize ChatGPT for cleaning text descriptions. &#10;Their contrastive language-audio pretraining (CLAP) models, trained on their WavCaps dataset, demonstrate superior performance in audio-text retrieval tasks compared to LAION-CLAP, despite a smaller-scale dataset.&#10;Such a text filtering pipeline reduces the amount of audio data, which could potentially result in reduced generalization.&#10;&#10;In addition to dataset scale, there is a need for novel neural architectures and training strategies tailored to model audio structures more effectively. &#10;For instance, LAION-CLAP~\cite{laionclap2023} investigates different choices of audio/text encoders, demonstrating superior performance with the hierarchical token semantic audio transformer (HTSAT)\cite{chen2022hts} for audio encoding and the Robustly optimized BERT approach (RoBERTa)\cite{liu2019roberta} for text encoding.&#10;In addition, LAION-CLAP proposes feature fusion for audio inputs with variable-length.&#10;In concurrent work, fast language-audio pretraining (FLAP)~\cite{yeh2023flap}, inspired by fast language-image pretraining (FLIP)~\cite{li2023scaling}, proposes masking and removing a significant portion of spectrogram patches. &#10;FLAP also incorporates a reconstruction loss during the contrastive training on these masked spectrogram patches, although the improvements are modest. &#10;&#10;In this paper, we investigate several strategies to improve the audio-text models, informed by the aforementioned challenges. &#10;For dataset creation, we collect a large-scale audio-text dataset and expand and refine its text descriptions.&#10;For audio recordings with weak or no labels, we utilize an automatic audio captioning model to obtain synthetic captions. &#10;For audio paired with noisy descriptions, we use large language models (LLMs) to generate several cleaned captions for each audio clip. &#10;These efforts lead to a collection of over 3.9 million audio-text pairs, with over 13,000 hours of audio.&#10;&#10;For our neural architecture and training strategy, we propose to use a two-stage approach. &#10;The first stage focuses on training spectrogram-based audio encoder using a masked autoencoder (MAE) objective~\cite{he2022masked,huang2022masked}, which learns representations through masking random patches from the input spectrogram and then reconstructing these masked patches.&#10;We anticipate that the MAE training will provide a better initialization for the following contrastive training.&#10;An audio classification objective, in contrast, may encourage the model to discard information unnecessary for classification but important for contrastive training. &#10;In the second stage, we use the audio encoder from the first stage to train our audio-text model on collected synthetic audio-text pairs, employing dual contrastive and captioning objectives. &#10;The integration of the auxillary captioning objective, inspired by contrastive captioner (CoCa)~\cite{yu2022coca} and bootstrapping language-image pre-training (BLIP)~\cite{li2022blip}, provides stronger supervision, encouraging the audio encoder to capture fine-grained patterns that closely match text descriptions. &#10;Training a captioner decoder also facilitates text generation for open-ended audio understanding tasks, expanding our model's application scope.&#10;&#10;In the evaluation phase, incorporating a diverse range of evaluation tasks is crucial to comprehensively measuring model capability and preventing overfitting to common test sets, as suggested by Recht et al.~\cite{recht2018cifar}. &#10;Typically, audio-text representation learning is assessed through zero-shot audio classification and audio-text retrieval. &#10;To provide a more comprehensive benchmark, we additionally evaluate on audio question answering (AQA)~\cite{lipping2022clotho}.&#10;To evaluate the effectiveness of our audio encoder, we test on Holistic Evaluation of Audio Representations&#10;(HEAR)~\cite{turian2022hear}. &#10;HEAR offers broad evaluation tasks that test the general-purpose audio representation through audio classification and sound event detection. &#10;Lastly, to assess the performance of our captioning decoder, we evaluate on automatic captioning tasks for open-ended generation.&#10;&#10;In summary, the contribution of our work includes: (1) We curate a large-scale refined audio-text dataset with LLM processing and audio captioning.&#10;(2) We propose a two-stage training approach for contrastive models: we first train an audio encoder with an MAE objective.&#10;In the second stage, we train a constrative model, initializing the audio encoder from the first stage, and include an auxiliary captioning objective to enhance the model's understanding of audio-text relationships.&#10;(3) We benchmark our model on a variety of audio understanding tasks. &#10;Particularly, our model achieves state-of-the-art or comparable performance on audio-text retrieval tasks. &#10;We have also conducted comprehensive ablation studies to demonstrate the impact of our different contributions.&#10;We open source the inference and evaluation codebase along with our pretrained model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Acoustics, Computer Science, Audio Text Modeling, Audio Signal Processing, Contrastive Learning, Linguistics, Large Scale Dataset" />
        </attvalues>
      </node>
      <node id="1912.10211" label="1912.10211">
        <attvalues>
          <attvalue for="0" value="PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern&#10;  Recognition" />
          <attvalue for="1" value="  Audio pattern recognition is an important research topic in the machine&#10;learning area, and includes several tasks such as audio tagging, acoustic scene&#10;classification, music classification, speech emotion classification and sound&#10;event detection. Recently, neural networks have been applied to tackle audio&#10;pattern recognition problems. However, previous systems are built on specific&#10;datasets with limited durations. Recently, in computer vision and natural&#10;language processing, systems pretrained on large-scale datasets have&#10;generalized well to several tasks. However, there is limited research on&#10;pretraining systems on large-scale datasets for audio pattern recognition. In&#10;this paper, we propose pretrained audio neural networks (PANNs) trained on the&#10;large-scale AudioSet dataset. These PANNs are transferred to other audio&#10;related tasks. We investigate the performance and computational complexity of&#10;PANNs modeled by a variety of convolutional neural networks. We propose an&#10;architecture called Wavegram-Logmel-CNN using both log-mel spectrogram and&#10;waveform as input feature. Our best PANN system achieves a state-of-the-art&#10;mean average precision (mAP) of 0.439 on AudioSet tagging, outperforming the&#10;best previous system of 0.392. We transfer PANNs to six audio pattern&#10;recognition tasks, and demonstrate state-of-the-art performance in several of&#10;those tasks. We have released the source code and pretrained models of PANNs:&#10;https://github.com/qiuqiangkong/audioset_tagging_cnn.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10790" label="2305.10790">
        <attvalues>
          <attvalue for="0" value="Listen, Think, and Understand" />
          <attvalue for="1" value="  The ability of artificial intelligence (AI) systems to perceive and&#10;comprehend audio signals is crucial for many applications. Although significant&#10;progress has been made in this area since the development of AudioSet, most&#10;existing models are designed to map audio inputs to pre-defined, discrete sound&#10;label sets. In contrast, humans possess the ability to not only classify sounds&#10;into general categories, but also to listen to the finer details of the sounds,&#10;explain the reason for the predictions, think about what the sound infers, and&#10;understand the scene and what action needs to be taken, if any. Such&#10;capabilities beyond perception are not yet present in existing audio models. On&#10;the other hand, modern large language models (LLMs) exhibit emerging reasoning&#10;ability but they lack audio perception capabilities. Therefore, we ask the&#10;question: can we build a model that has both audio perception and a reasoning&#10;ability?&#10;  In this paper, we propose a new audio foundation model, called LTU (Listen,&#10;Think, and Understand). To train LTU, we created a new OpenAQA-5M dataset&#10;consisting of 1.9 million closed-ended and 3.7 million open-ended, diverse&#10;(audio, question, answer) tuples, and have used an autoregressive training&#10;framework with a perception-to-understanding curriculum. LTU demonstrates&#10;strong performance and generalization ability on conventional audio tasks such&#10;as classification and captioning. More importantly, it exhibits emerging audio&#10;reasoning and comprehension abilities that are absent in existing audio models.&#10;To the best of our knowledge, LTU is one of the first multimodal large language&#10;models that focus on general audio (rather than just speech) understanding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11834" label="2305.11834">
        <attvalues>
          <attvalue for="0" value="Pengi: An Audio Language Model for Audio Tasks" />
          <attvalue for="1" value="  In the domain of audio processing, Transfer Learning has facilitated the rise&#10;of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches&#10;have led to the development of versatile models capable of tackling a wide&#10;array of tasks, while delivering state-of-the-art performance. However, current&#10;models inherently lack the capacity to produce the requisite language for&#10;open-ended tasks, such as Audio Captioning or Audio Question &amp; Answering. We&#10;introduce Pengi, a novel Audio Language Model that leverages Transfer Learning&#10;by framing all audio tasks as text-generation tasks. It takes as input, an&#10;audio recording, and text, and generates free-form text as output. The input&#10;audio is represented as a sequence of continuous embeddings by an audio&#10;encoder. A text encoder does the same for the corresponding text input. Both&#10;sequences are combined as a prefix to prompt a pre-trained frozen language&#10;model. The unified architecture of Pengi enables open-ended tasks and&#10;close-ended tasks without any additional fine-tuning or task-specific&#10;extensions. When evaluated on 22 downstream tasks, our approach yields&#10;state-of-the-art performance in several of them. Our results show that&#10;connecting language models with audio models is a major step towards&#10;general-purpose audio understanding&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.12661" label="2301.12661">
        <attvalues>
          <attvalue for="0" value="Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion&#10;  Models" />
          <attvalue for="1" value="  Large-scale multimodal generative modeling has created milestones in&#10;text-to-image and text-to-video generation. Its application to audio still lags&#10;behind for two main reasons: the lack of large-scale datasets with high-quality&#10;text-audio pairs, and the complexity of modeling long continuous audio data. In&#10;this work, we propose Make-An-Audio with a prompt-enhanced diffusion model that&#10;addresses these gaps by 1) introducing pseudo prompt enhancement with a&#10;distill-then-reprogram approach, it alleviates data scarcity with orders of&#10;magnitude concept compositions by using language-free audios; 2) leveraging&#10;spectrogram autoencoder to predict the self-supervised audio representation&#10;instead of waveforms. Together with robust contrastive language-audio&#10;pretraining (CLAP) representations, Make-An-Audio achieves state-of-the-art&#10;results in both objective and subjective benchmark evaluation. Moreover, we&#10;present its controllability and generalization for X-to-Audio with &quot;No Modality&#10;Left Behind&quot;, for the first time unlocking the ability to generate&#10;high-definition, high-fidelity audios given a user-defined modality input.&#10;Audio samples are available at https://Text-to-Audio.github.io&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.12503" label="2301.12503">
        <attvalues>
          <attvalue for="0" value="AudioLDM: Text-to-Audio Generation with Latent Diffusion Models" />
          <attvalue for="1" value="  Text-to-audio (TTA) system has recently gained attention for its ability to&#10;synthesize general audio based on text descriptions. However, previous studies&#10;in TTA have limited generation quality with high computational costs. In this&#10;study, we propose AudioLDM, a TTA system that is built on a latent space to&#10;learn the continuous audio representations from contrastive language-audio&#10;pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs&#10;with audio embedding while providing text embedding as a condition during&#10;sampling. By learning the latent representations of audio signals and their&#10;compositions without modeling the cross-modal relationship, AudioLDM is&#10;advantageous in both generation quality and computational efficiency. Trained&#10;on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA&#10;performance measured by both objective and subjective metrics (e.g., frechet&#10;distance). Moreover, AudioLDM is the first TTA system that enables various&#10;text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion.&#10;Our implementation and demos are available at https://audioldm.github.io.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.05037" label="2308.05037">
        <attvalues>
          <attvalue for="0" value="Separate Anything You Describe" />
          <attvalue for="1" value="  Language-queried audio source separation (LASS) is a new paradigm for&#10;computational auditory scene analysis (CASA). LASS aims to separate a target&#10;sound from an audio mixture given a natural language query, which provides a&#10;natural and scalable interface for digital audio applications. Recent works on&#10;LASS, despite attaining promising separation performance on specific sources&#10;(e.g., musical instruments, limited classes of audio events), are unable to&#10;separate audio concepts in the open domain. In this work, we introduce&#10;AudioSep, a foundation model for open-domain audio source separation with&#10;natural language queries. We train AudioSep on large-scale multimodal datasets&#10;and extensively evaluate its capabilities on numerous tasks including audio&#10;event separation, musical instrument separation, and speech enhancement.&#10;AudioSep demonstrates strong separation performance and impressive zero-shot&#10;generalization ability using audio captions or text labels as queries,&#10;substantially outperforming previous audio-queried and language-queried sound&#10;separation models. For reproducibility of this work, we will release the source&#10;code, evaluation benchmark and pre-trained model at:&#10;https://github.com/Audio-AGI/AudioSep.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.09387" label="1910.09387">
        <attvalues>
          <attvalue for="0" value="Clotho: An Audio Captioning Dataset" />
          <attvalue for="1" value="  Audio captioning is the novel task of general audio content description using&#10;free text. It is an intermodal translation task (not speech-to-text), where a&#10;system accepts as an input an audio signal and outputs the textual description&#10;(i.e. the caption) of that signal. In this paper we present Clotho, a dataset&#10;for audio captioning consisting of 4981 audio samples of 15 to 30 seconds&#10;duration and 24 905 captions of eight to 20 words length, and a baseline method&#10;to provide initial results. Clotho is built with focus on audio content and&#10;caption diversity, and the splits of the data are not hampering the training or&#10;evaluation of methods. All sounds are from the Freesound platform, and captions&#10;are crowdsourced using Amazon Mechanical Turk and annotators from English&#10;speaking countries. Unique words, named entities, and speech transcription are&#10;removed with post-processing. Clotho is freely available online&#10;(https://zenodo.org/record/3490684).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.15343" label="2303.15343">
        <attvalues>
          <attvalue for="0" value="Sigmoid Loss for Language Image Pre-Training" />
          <attvalue for="1" value="  We propose a simple pairwise Sigmoid loss for Language-Image Pre-training&#10;(SigLIP). Unlike standard contrastive learning with softmax normalization, the&#10;sigmoid loss operates solely on image-text pairs and does not require a global&#10;view of the pairwise similarities for normalization. The sigmoid loss&#10;simultaneously allows further scaling up the batch size, while also performing&#10;better at smaller batch sizes. Combined with Locked-image Tuning, with only&#10;four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet&#10;zero-shot accuracy in two days. The disentanglement of the batch size from the&#10;loss further allows us to study the impact of examples vs pairs and negative to&#10;positive ratio. Finally, we push the batch size to the extreme, up to one&#10;million, and find that the benefits of growing batch size quickly diminish,&#10;with a more reasonable batch size of 32k being sufficient. We release our&#10;models at https://github.com/google-research/big_vision and hope our research&#10;motivates further explorations in improving the quality and efficiency of&#10;language-image pre-training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.12415" label="2208.12415">
        <attvalues>
          <attvalue for="0" value="MuLan: A Joint Embedding of Music Audio and Natural Language" />
          <attvalue for="1" value="  Music tagging and content-based retrieval systems have traditionally been&#10;constructed using pre-defined ontologies covering a rigid set of music&#10;attributes or text queries. This paper presents MuLan: a first attempt at a new&#10;generation of acoustic models that link music audio directly to unconstrained&#10;natural language music descriptions. MuLan takes the form of a two-tower, joint&#10;audio-text embedding model trained using 44 million music recordings (370K&#10;hours) and weakly-associated, free-form text annotations. Through its&#10;compatibility with a wide range of music genres and text styles (including&#10;conventional music tags), the resulting audio-text representation subsumes&#10;existing ontologies while graduating to true zero-shot functionalities. We&#10;demonstrate the versatility of the MuLan embeddings with a range of experiments&#10;including transfer learning, zero-shot music tagging, language understanding in&#10;the music domain, and cross-modal retrieval applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.06687" label="2211.06687">
        <attvalues>
          <attvalue for="0" value="Large-scale Contrastive Language-Audio Pretraining with Feature Fusion&#10;  and Keyword-to-Caption Augmentation" />
          <attvalue for="1" value="  Contrastive learning has shown remarkable success in the field of multimodal&#10;representation learning. In this paper, we propose a pipeline of contrastive&#10;language-audio pretraining to develop an audio representation by combining&#10;audio data with natural language descriptions. To accomplish this target, we&#10;first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs&#10;from different data sources. Second, we construct a contrastive language-audio&#10;pretraining model by considering different audio encoders and text encoders. We&#10;incorporate the feature fusion mechanism and keyword-to-caption augmentation&#10;into the model design to further enable the model to process audio inputs of&#10;variable lengths and enhance the performance. Third, we perform comprehensive&#10;experiments to evaluate our model across three tasks: text-to-audio retrieval,&#10;zero-shot audio classification, and supervised audio classification. The&#10;results demonstrate that our model achieves superior performance in&#10;text-to-audio retrieval task. In audio classification tasks, the model achieves&#10;state-of-the-art performance in the zero-shot setting and is able to obtain&#10;performance comparable to models' results in the non-zero-shot setting.&#10;LAION-Audio-630K and the proposed model are both available to the public.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.17395" label="2303.17395">
        <attvalues>
          <attvalue for="0" value="WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for&#10;  Audio-Language Multimodal Research" />
          <attvalue for="1" value="  The advancement of audio-language (AL) multimodal learning tasks has been&#10;significant in recent years. However, researchers face challenges due to the&#10;costly and time-consuming collection process of existing audio-language&#10;datasets, which are limited in size. To address this data scarcity issue, we&#10;introduce WavCaps, the first large-scale weakly-labelled audio captioning&#10;dataset, comprising approximately 400k audio clips with paired captions. We&#10;sourced audio clips and their raw descriptions from web sources and a sound&#10;event detection dataset. However, the online-harvested raw descriptions are&#10;highly noisy and unsuitable for direct use in tasks such as automated audio&#10;captioning. To overcome this issue, we propose a three-stage processing&#10;pipeline for filtering noisy data and generating high-quality captions, where&#10;ChatGPT, a large language model, is leveraged to filter and transform raw&#10;descriptions automatically. We conduct a comprehensive analysis of the&#10;characteristics of WavCaps dataset and evaluate it on multiple downstream&#10;audio-language multimodal learning tasks. The systems trained on WavCaps&#10;outperform previous state-of-the-art (SOTA) models by a significant margin. Our&#10;aspiration is for the WavCaps dataset we have proposed to facilitate research&#10;in audio-language multimodal learning and demonstrate the potential of&#10;utilizing ChatGPT to enhance academic research. Our dataset and codes are&#10;available at https://github.com/XinhaoMei/WavCaps.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.00874" label="2202.00874">
        <attvalues>
          <attvalue for="0" value="HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound&#10;  Classification and Detection" />
          <attvalue for="1" value="  Audio classification is an important task of mapping audio samples into their&#10;corresponding labels. Recently, the transformer model with self-attention&#10;mechanisms has been adopted in this field. However, existing audio transformers&#10;require large GPU memories and long training time, meanwhile relying on&#10;pretrained vision models to achieve high performance, which limits the model's&#10;scalability in audio tasks. To combat these problems, we introduce HTS-AT: an&#10;audio transformer with a hierarchical structure to reduce the model size and&#10;training time. It is further combined with a token-semantic module to map final&#10;outputs into class featuremaps, thus enabling the model for the audio event&#10;detection (i.e. localization in time). We evaluate HTS-AT on three datasets of&#10;audio classification where it achieves new state-of-the-art (SOTA) results on&#10;AudioSet and ESC-50, and equals the SOTA on Speech Command V2. It also achieves&#10;better performance in event localization than the previous CNN-based models.&#10;Moreover, HTS-AT requires only 35% model parameters and 15% training time of&#10;the previous audio transformer. These results demonstrate the high performance&#10;and high efficiency of HTS-AT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.00794" label="2212.00794">
        <attvalues>
          <attvalue for="0" value="Scaling Language-Image Pre-training via Masking" />
          <attvalue for="1" value="  We present Fast Language-Image Pre-training (FLIP), a simple and more&#10;efficient method for training CLIP. Our method randomly masks out and removes a&#10;large portion of image patches during training. Masking allows us to learn from&#10;more image-text pairs given the same wall-clock time and contrast more samples&#10;per iteration with similar memory footprint. It leads to a favorable trade-off&#10;between accuracy and training time. In our experiments on 400 million&#10;image-text pairs, FLIP improves both accuracy and speed over the no-masking&#10;baseline. On a large diversity of downstream tasks, FLIP dominantly outperforms&#10;the CLIP counterparts trained on the same data. Facilitated by the speedup, we&#10;explore the scaling behavior of increasing the model size, data size, or&#10;training length, and report encouraging results and comparisons. We hope that&#10;our work will foster future research on scaling vision-language learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.06405" label="2207.06405">
        <attvalues>
          <attvalue for="0" value="Masked Autoencoders that Listen" />
          <attvalue for="1" value="  This paper studies a simple extension of image-based Masked Autoencoders&#10;(MAE) to self-supervised representation learning from audio spectrograms.&#10;Following the Transformer encoder-decoder design in MAE, our Audio-MAE first&#10;encodes audio spectrogram patches with a high masking ratio, feeding only the&#10;non-masked tokens through encoder layers. The decoder then re-orders and&#10;decodes the encoded context padded with mask tokens, in order to reconstruct&#10;the input spectrogram. We find it beneficial to incorporate local window&#10;attention in the decoder, as audio spectrograms are highly correlated in local&#10;time and frequency bands. We then fine-tune the encoder with a lower masking&#10;ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art&#10;performance on six audio and speech classification tasks, outperforming other&#10;recent models that use external supervised pre-training. The code and models&#10;will be at https://github.com/facebookresearch/AudioMAE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.01917" label="2205.01917">
        <attvalues>
          <attvalue for="0" value="CoCa: Contrastive Captioners are Image-Text Foundation Models" />
          <attvalue for="1" value="  Exploring large-scale pretrained foundation models is of significant interest&#10;in computer vision because these models can be quickly transferred to many&#10;downstream tasks. This paper presents Contrastive Captioner (CoCa), a&#10;minimalist design to pretrain an image-text encoder-decoder foundation model&#10;jointly with contrastive loss and captioning loss, thereby subsuming model&#10;capabilities from contrastive approaches like CLIP and generative methods like&#10;SimVLM. In contrast to standard encoder-decoder transformers where all decoder&#10;layers attend to encoder outputs, CoCa omits cross-attention in the first half&#10;of decoder layers to encode unimodal text representations, and cascades the&#10;remaining decoder layers which cross-attend to the image encoder for multimodal&#10;image-text representations. We apply a contrastive loss between unimodal image&#10;and text embeddings, in addition to a captioning loss on the multimodal decoder&#10;outputs which predicts text tokens autoregressively. By sharing the same&#10;computational graph, the two training objectives are computed efficiently with&#10;minimal overhead. CoCa is pretrained end-to-end and from scratch on both&#10;web-scale alt-text data and annotated images by treating all labels simply as&#10;text, seamlessly unifying natural language supervision for representation&#10;learning. Empirically, CoCa achieves state-of-the-art performance with&#10;zero-shot transfer or minimal task-specific adaptation on a broad range of&#10;downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700,&#10;Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal&#10;understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps).&#10;Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1&#10;accuracy, 90.6% with a frozen encoder and learned classification head, and new&#10;state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.00451" label="1806.00451">
        <attvalues>
          <attvalue for="0" value="Do CIFAR-10 Classifiers Generalize to CIFAR-10?" />
          <attvalue for="1" value="  Machine learning is currently dominated by largely experimental work focused&#10;on improvements in a few key tasks. However, the impressive accuracy numbers of&#10;the best performing models are questionable because the same test sets have&#10;been used to select these models for multiple years now. To understand the&#10;danger of overfitting, we measure the accuracy of CIFAR-10 classifiers by&#10;creating a new test set of truly unseen images. Although we ensure that the new&#10;test set is as close to the original data distribution as possible, we find a&#10;large drop in accuracy (4% to 10%) for a broad range of deep learning models.&#10;Yet more recent models with higher original accuracy show a smaller drop and&#10;better overall performance, indicating that this drop is likely not due to&#10;overfitting based on adaptivity. Instead, we view our results as evidence that&#10;current accuracy numbers are brittle and susceptible to even minute natural&#10;variations in the data distribution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.09634" label="2204.09634">
        <attvalues>
          <attvalue for="0" value="Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering" />
          <attvalue for="1" value="  Audio question answering (AQA) is a multimodal translation task where a&#10;system analyzes an audio signal and a natural language question, to generate a&#10;desirable natural language answer. In this paper, we introduce Clotho-AQA, a&#10;dataset for Audio question answering consisting of 1991 audio files each&#10;between 15 to 30 seconds in duration selected from the Clotho dataset. For each&#10;audio file, we collect six different questions and corresponding answers by&#10;crowdsourcing using Amazon Mechanical Turk. The questions and answers are&#10;produced by different annotators. Out of the six questions for each audio, two&#10;questions each are designed to have 'yes' and 'no' as answers, while the&#10;remaining two questions have other single-word answers. For each question, we&#10;collect answers from three different annotators. We also present two baseline&#10;experiments to describe the usage of our dataset for the AQA task - an&#10;LSTM-based multimodal binary classifier for 'yes' or 'no' type answers and an&#10;LSTM-based multimodal multi-class classifier for 828 single-word answers. The&#10;binary classifier achieved an accuracy of 62.7% and the multi-class classifier&#10;achieved a top-1 accuracy of 54.2% and a top-5 accuracy of 93.7%. Clotho-AQA&#10;dataset is freely available online at https://zenodo.org/record/6473207.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.03022" label="2203.03022">
        <attvalues>
          <attvalue for="0" value="HEAR: Holistic Evaluation of Audio Representations" />
          <attvalue for="1" value="  What audio embedding approach generalizes best to a wide range of downstream&#10;tasks across a variety of everyday domains without fine-tuning? The aim of the&#10;HEAR benchmark is to develop a general-purpose audio representation that&#10;provides a strong basis for learning in a wide variety of tasks and scenarios.&#10;HEAR evaluates audio representations using a benchmark suite across a variety&#10;of domains, including speech, environmental sound, and music. HEAR was launched&#10;as a NeurIPS 2021 shared challenge. In the spirit of shared exchange, each&#10;participant submitted an audio embedding model following a common API that is&#10;general-purpose, open-source, and freely available to use. Twenty-nine models&#10;by thirteen external teams were evaluated on nineteen diverse downstream tasks&#10;derived from sixteen datasets. Open evaluation code, submitted models and&#10;datasets are key contributions, enabling comprehensive and reproducible&#10;evaluation, as well as previously impossible longitudinal studies. It still&#10;remains an open question whether one single general-purpose audio&#10;representation can perform as holistically as the human ear.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.11009" label="2407.11009">
        <attvalues>
          <attvalue for="0" value="CharED: Character-wise Ensemble Decoding for Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have shown remarkable potential for problem&#10;solving, with open source models achieving increasingly impressive performance&#10;on benchmarks measuring areas from logical reasoning to mathematical ability.&#10;Ensembling models can further improve capabilities across a variety of domains.&#10;However, conventional methods of combining models at inference time such as&#10;shallow fusion necessitate a shared vocabulary and tokenization, and&#10;alternatives like fine-tuning for domain-specific performance are both time&#10;consuming and computationally expensive. We therefore present an inference-time&#10;ensembling algorithm aimed at &quot;averaging&quot; outputs from multiple LLMs and&#10;illustrate its improved performance across multiple domains compared to its&#10;constituent models alone. Character-wise ensemble decoding, CharED, finds the&#10;marginal distribution of each character for an individual model and performs a&#10;weighted average to generate an output, character by character. In coding,&#10;math, and toxicity benchmarks, we find our proposed model able to combine&#10;complimentary strengths of multiple LLMs, regardless of vocabulary,&#10;tokenization, or model size.&#10;" />
          <attvalue for="2" value="&#10;As large language models (LLMs) have become increasingly ubiquitous and powerful models have been open-sourced, there has been extensive research on methods to achieve improved task-specific performance from these models. The long-standing method for doing this is through fine-tuning, in which domain-specific datasets are used to update weights of large foundation models to improve performance on certain tasks. However, direct fine-tuning is both time-consuming and computationally intensive \cite{strubell-etal-2019-energy}. This problem will become worse as model sizes continue to grow, increasingly motivating more efficient fine-tuning \cite{lester-etal-2021-power, han2024parameterefficient} or alternative approaches \cite{hu2021lora} for enhancing or aligning LLM performance. &#10;&#10;Model ensembling has been shown to yield improved performance across different domains. An established method for doing this is through shallow fusion, which was originally used to integrate an LLM into a neural machine translation (NMT) model \cite{gulcehre2015using}. Such ensembling methods, which aggregate models during beam search, have shown promise for improving translation quality in NMT settings \cite{sutskever2014sequence, firat2016zeroresource, stahlberg-etal-2018-simple}, but require the same vocabulary and tokenization. Twist decoding \cite{kasai2022twist} modifies beam search to bypass the shared vocabulary restriction, but its reliance on beam search reduces the inference speed. Other more recent approaches related to combining language models include proxy tuning \cite{liu2024tuning} and Composition to Augment Language Models (CALM) \cite{bansal2024llm}. Proxy-tuning adjusts next-token predictions of a larger LLM using a pair of tuned and untuned smaller LMs, but is essentially limited to models from the same family, as it requires shared vocabulary. CALM can combine any LLMs via cross-attention but requires additional training.&#10;&#10;Historically, major advances in LMs have come out of subword-level tokenization schemes, which gained traction for their flexibility \cite{yang2024rethinking}, including byte-pair encoding (BPE), SentencePiece, and WordPiece \cite{sennrich-etal-2016-neural, kudo-richardson-2018-sentencepiece, Devlin2019BERTPO, zhang-etal-2019-ernie}. These tokenization methods have generally outperformed character-based language modeling, like LSTMs and other RNNs. Character models come with added challenges, including a lack of lexical and morphological priors compared to word and subword-level tokenizers, higher compute resources, and much longer dependencies on prior text \cite{Al-Rfou_Choe_Constant_Guo_Jones_2019, 7953252}. &#10;&#10;While character-level models have failed to gain traction for these reasons, there are some promising use cases for such models in more niche applications, due to their ability to leverage more fine-grained information. One recent study \cite{edman2024characterlevel} fine-tuned a character-level model \cite{xue-etal-2022-byt5} and the model's subword-level counterpart \cite{xue-etal-2021-mt5} for neural machine translation tasks, and found that the character-level model produced improved translation and better cross-lingual generalizations. More generally, there is some evidence that character-level information can improve performance over other tokenization methods \cite{clark-etal-2022-canine}, particularly in low resource and high language variability settings \cite{riabi-etal-2021-character}.&#10;&#10;This motivates further exploration into the relationships between subword-level and character-level models, as well as the applications of character-level LLMs. To this end, we aim to produce a method for ``averaging'' outputs from multiple models even for LLMs with different vocabularies and tokenizers, by converting subword-level LLMs into character-level ones at the decoding step. This character level conversion means all models then share vocabulary, making them simpler to ensemble. There is some evidence that pretrained language models with subword tokenizers also encode character-level information through the training process \cite{kaushal2022tokens}, further motivating such an approach. Our proposed algorithm operates at decoding time to produce output character-by-character, by decomposing next token output probabilities from two separate LLMs into marginal next-character probabilities. This method demonstrates promising results in improving combined LLM performance across diverse benchmarks, including HumanEval \cite{chen2021evaluating}, GSM8K \cite{cobbe2021training}, and ToxiGen \cite{hartvigsen2022toxigen}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Ensembling, Computer Science, Large Language Models, Linguistics, Inference Time Algorithms, Mathematics" />
        </attvalues>
      </node>
      <node id="2205.09273" label="2205.09273">
        <attvalues>
          <attvalue for="0" value="Twist Decoding: Diverse Generators Guide Each Other" />
          <attvalue for="1" value="  Many language generation models are now available for a wide range of&#10;generation tasks, including machine translation and summarization. Combining&#10;such diverse models may lead to further progress, but ensembling generation&#10;models is challenging during inference: conventional ensembling methods (e.g.,&#10;shallow fusion) require that the models share vocabulary/tokenization schemes.&#10;We introduce Twist decoding, a simple and general text generation algorithm&#10;that benefits from diverse models at inference time. Our method does not assume&#10;the vocabulary, tokenization or even generation order is shared. Our extensive&#10;evaluations on machine translation and scientific paper summarization&#10;demonstrate that Twist decoding substantially outperforms each model decoded in&#10;isolation over various scenarios, including cases where domain-specific and&#10;general-purpose models are both available. Twist decoding also consistently&#10;outperforms the popular reranking heuristic where output candidates from one&#10;model are rescored by another. We hope that our work will encourage researchers&#10;and practitioners to examine generation models collectively, not just&#10;independently, and to seek out models with complementary strengths to the&#10;currently available models. Our code is available at&#10;https://github.com/jungokasai/twist_decoding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.02412" label="2401.02412">
        <attvalues>
          <attvalue for="0" value="LLM Augmented LLMs: Expanding Capabilities through Composition" />
          <attvalue for="1" value="  Foundational models with billions of parameters which have been trained on&#10;large corpora of data have demonstrated non-trivial skills in a variety of&#10;domains. However, due to their monolithic structure, it is challenging and&#10;expensive to augment them or impart new skills. On the other hand, due to their&#10;adaptation abilities, several new instances of these models are being trained&#10;towards new domains and tasks. In this work, we study the problem of efficient&#10;and practical composition of existing foundation models with more specific&#10;models to enable newer capabilities. To this end, we propose CALM --&#10;Composition to Augment Language Models -- which introduces cross-attention&#10;between models to compose their representations and enable new capabilities.&#10;Salient features of CALM are: (i) Scales up LLMs on new tasks by 're-using'&#10;existing LLMs along with a few additional parameters and data, (ii) Existing&#10;model weights are kept intact, and hence preserves existing capabilities, and&#10;(iii) Applies to diverse domains and settings. We illustrate that augmenting&#10;PaLM2-S with a smaller model trained on low-resource languages results in an&#10;absolute improvement of up to 13\% on tasks like translation into English and&#10;arithmetic reasoning for low-resource languages. Similarly, when PaLM2-S is&#10;augmented with a code-specific model, we see a relative improvement of 40\%&#10;over the base model for code generation and explanation tasks -- on-par with&#10;fully fine-tuned counterparts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Parameter efficient fine-tuning: A large body of work focuses on efficient ways of fine-tuning models for new domains by introducing a small number of trainable parameters, keeping the original model intact~\cite{houlsby2019adapter, wang_k-adapter_2020, pfeiffer_adapterfusion_2021, hu_lora_2021, kessler_adapter_2022}. Since this paradigm allows a small set of new parameters to be trained, it is challenging to use this approach to adapt a model to a new domain, which is absent from the original training corpus. &#10;In contrast, enables a model to be adapted to completely new domains using an augmenting model. In Section~\ref{sec:ablations}, we demonstrate that is significantly more effective than LoRA~\cite{hu_lora_2021}, a representative parameter efficient fine-tuning method.&#10;&#10;Model Merging: Merging different expert models with simple techniques like task vector averaging provides a way of recombining different capabilities of these models~\cite{taskarithmatic,fisheraverage}. &#10;However, these methods are only relevant when the original models are well aligned. Other related approaches are also applicable only when the models are derived from the same model~\cite{fisheraverage} or they are of same size ~\cite{softmerging}. In contrast, is more generic and is applicable to any set of models.&#10;&#10;Model and Task Compositionality: The modular encoder-decoder based method in &#10;\cite{dalmia_legonn_2022} adapts components of&#10;encoder-decoder models to allow flexible&#10;re-usability of different encoders, each&#10;with their own capabilities.&#10;Several past studies explore compositionality&#10;from a multi-modal standpoint.&#10;\cite{alayrac_flamingo_2022}&#10;introduce cross-attention parameters across a language model&#10;in order to attend to representations coming &#10;from an image encoder. They show very effective transfer&#10;of capabilities between the two models.&#10;In this work, we extend the ideology of model re-use and modularity to extend composition of capabilities in a large language model.&#10;&#10;Models as Tools:&#10;Another interesting direction&#10;for using multiple language models to solve a&#10;downstream task has been to perform&#10;composition in the models' input text space&#10;\cite{zeng_socratic_2022, shen_hugginggpt_2023}.&#10;\cite{schick2023toolformer} have demonstrated &#10;how a model can be taught to use external tools---there might be an opportunity to&#10;investigate if other models can be&#10;called as a part of the same framework.&#10;Since these approaches require a large amount &#10;of prompt engineering,&#10;in this work we focus on composition&#10;through representations&#10;that can be learnt automatically.&#10; " />
        </attvalues>
      </node>
      <node id="2403.00417" label="2403.00417">
        <attvalues>
          <attvalue for="0" value="Rethinking Tokenization: Crafting Better Tokenizers for Large Language&#10;  Models" />
          <attvalue for="1" value="  Tokenization significantly influences language models(LMs)' performance. This&#10;paper traces the evolution of tokenizers from word-level to subword-level,&#10;analyzing how they balance tokens and types to enhance model adaptability while&#10;controlling complexity. Despite subword tokenizers like Byte Pair Encoding&#10;(BPE) overcoming many word tokenizer limitations, they encounter difficulties&#10;in handling non-Latin languages and depend heavily on extensive training data&#10;and computational resources to grasp the nuances of multiword expressions&#10;(MWEs). This article argues that tokenizers, more than mere technical tools,&#10;should drawing inspiration from the cognitive science about human language&#10;processing. This study then introduces the &quot;Principle of Least Effort&quot; from&#10;cognitive science, that humans naturally seek to reduce cognitive effort, and&#10;discusses the benefits of this principle for tokenizer development. Based on&#10;this principle, the paper proposes that the Less-is-Better (LiB) model could be&#10;a new approach for LLM tokenizer. The LiB model can autonomously learn an&#10;integrated vocabulary consisting of subwords, words, and MWEs, which&#10;effectively reduces both the numbers of tokens and types. Comparative&#10;evaluations show that the LiB tokenizer outperforms existing word and BPE&#10;tokenizers, presenting an innovative method for tokenizer development, and&#10;hinting at the possibility of future cognitive science-based tokenizers being&#10;more efficient.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.14220" label="2302.14220">
        <attvalues>
          <attvalue for="0" value="Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5&#10;  for Machine Translation" />
          <attvalue for="1" value="  Pretrained character-level and byte-level language models have been shown to&#10;be competitive with popular subword models across a range of Natural Language&#10;Processing (NLP) tasks. However, there has been little research on their&#10;effectiveness for neural machine translation (NMT), particularly within the&#10;popular pretrain-then-finetune paradigm. This work performs an extensive&#10;comparison across multiple languages and experimental conditions of character-&#10;and subword-level pretrained models (ByT5 and mT5, respectively) on NMT. We&#10;show the effectiveness of character-level modeling in translation, particularly&#10;in cases where fine-tuning data is limited. In our analysis, we show how&#10;character models' gains in translation quality are reflected in better&#10;translations of orthographically similar words and rare words. While evaluating&#10;the importance of source texts in driving model predictions, we highlight&#10;word-level patterns within ByT5, suggesting an ability to modulate word-level&#10;and character-level information during generation. We conclude by assessing the&#10;efficiency tradeoff of byte models, suggesting their usage in non-time-critical&#10;scenarios to boost translation quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;Character-level models have long been of interest for use in machine translation, dating back to when statistical models were the dominant paradigm \cite{tiedemann-nakov-2013-analyzing}. At that time, character models were already competitive with word-level models, especially when training data was limited to $&lt;\,$100k sentence pairs. \cite{durrani-etal-2010-hindi} also showed that character models were particularly adept at translating closely related languages such as Hindi and Urdu.&#10;We note that, at the time, subword tokenizers such as BPE \cite{sennrich-etal-2016-neural} were not yet commonly used.&#10;&#10;Neural approaches to character-level MT&#10;were first based on RNNs \cite{costa-jussa-fonollosa-2016-character, lee-etal-2017-fully}, with extensive work being done to compare character models to subword models (now equipped with BPE). For NMT, it was shown that character models using an RNN architecture (with and without a CNN for processing characters) perform equally to or better than subword models &#10;\cite{larriba2017traduccion,lee-etal-2017-fully,sennrich-2017-grammatical,cherry-etal-2018-revisiting}. Their better translation performance could be attributed to their ability to handle complex morphology, rare or unseen words, and noisy input&#10;\cite{jozefowicz2016exploring,kim2016character,belinkov2017synthetic,lee-etal-2017-fully,singh2017handling,durrani-etal-2019-one}. However the inefficiency of character models was already apparent, as they were slower than subword models by a considerable margin \cite{chung-etal-2016-character}.&#10;Work had also been done in comparing byte and character level models, with little differences found, however these were mostly experimenting with Latin-scripted languages \cite{costa-jussa-etal-2017-byte}.&#10;&#10;More recent work has looked at Transformers on the character and byte-level. \cite{xue-etal-2022-byt5} created ByT5 and compared it to its subword counterpart, mT5 \cite{xue-etal-2021-mt5}, which are also the models we focus on in this work. Their comparisons were however focused on either multilingual classification tasks or English-based generative tasks, but no multilingual generative tasks or machine translation.&#10;&#10;Previous work analyzed character-level Transformers trained from scratch for NMT, not using the T5-based models. &#10;\cite{libovicky-etal-2022-dont} looked at ``vanilla'' character models (those without any compression of the sequence length prior to the computation in the Transformer), as well as \cite{lee-etal-2017-fully}'s, \cite{tay2021charformer}'s, and \cite{clark-etal-2022-canine}'s methods for compressing sequence length.&#10;They conclude that these character-level models do not provide any benefits over subword models while being less efficient. &#10;There are two factors to note with these conclusions. Firstly, their experiments show similar performance, but they only experiment on high-resourced languages. Given prior work in RNNs mentioned above, this does not appear to be the application that would benefit most from character-level models.&#10;Secondly, they introduce a two-step decoder to achieve character-level decoding from subword-level hidden states, however as \cite{edman-etal-2022-subword} points out, this decoder does not scale well to higher-resourced scenarios. This two-step decoder adds an additional layer of complexity to evaluating such models, as an ablation of the model's granularity and the model's decoding process would be necessary to fully understand the performance of each model. Added to the fact that there are no pretrained models using this sequence length compression that are comparable to mT5 and ByT5 in terms of data used for pretraining or model scale, we do not experiment with the models which compress the sequence length in our work.&#10;&#10;In the context of low-resource MT, \cite{edman-etal-2022-subword} showed that character-level models can outperform subword models on the low-resource pair Xhosa--Zulu. \cite{li-etal-2021-char} showed that character-level models create higher-quality translations in synthetic low-resource settings of English$arrow$\{German, Finnish\} with a corpus size of 50k parallel sentences. \cite{carrion-ponz-casacuberta-2022-effectiveness} showed that quasi-character models (subword models with a small vocabulary of size 350) produce higher-quality translations than subwords with a more standard vocabulary size of 32 thousand when data is limited for a number of European languages, finding consistent improvements across several domains.&#10;&#10;There are two major caveats with this previous work that should be considered, however. First, previous work using character models for MT focused on training models from scratch, as this is a long-standing practice in the MT field. However, this practice could be especially harmful for low-resource languages, where the paradigm of fine-tuning a pretrained, multilingual model was shown to be effective \cite{liu-etal-2020-multilingual-denoising}. &#10;&#10;The second caveat is that previous evaluations of cross-lingual transfer were also limited by a relatively small model size ($&lt;\,$70M parameters). In contrast, this work evaluates models up to 1.2B parameters. With an order of magnitude more parameters, we investigate the presence of emergent properties of larger character and subword-based NMT models, following the evidence from other generative tasks. &#10;&#10;Within the realm of the extensive previous work done on character models, this work serves to give an updated overview on the performance of character versus subword models for NMT. To our knowledge, we provide the first of such overviews using multilingual, transformer-based models of up to 1.2B parameters. We fine-tune a total of 162 models (varying languages, amount of training data, model size, and model type), and test on 200 languages to thoroughly compare the performance of character and subword models. We also perform an attribution analysis to gain a deeper understanding of the differences between the two granularities of character-level and subword-level, which to our knowledge, has not yet been researched. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2105.13626" label="2105.13626">
        <attvalues>
          <attvalue for="0" value="ByT5: Towards a token-free future with pre-trained byte-to-byte models" />
          <attvalue for="1" value="  Most widely-used pre-trained language models operate on sequences of tokens&#10;corresponding to word or subword units. By comparison, token-free models that&#10;operate directly on raw text (bytes or characters) have many benefits: they can&#10;process text in any language out of the box, they are more robust to noise, and&#10;they minimize technical debt by removing complex and error-prone text&#10;preprocessing pipelines. Since byte or character sequences are longer than&#10;token sequences, past work on token-free models has often introduced new model&#10;architectures designed to amortize the cost of operating directly on raw text.&#10;In this paper, we show that a standard Transformer architecture can be used&#10;with minimal modifications to process byte sequences. We characterize the&#10;trade-offs in terms of parameter count, training FLOPs, and inference speed,&#10;and show that byte-level models are competitive with their token-level&#10;counterparts. We also demonstrate that byte-level models are significantly more&#10;robust to noise and perform better on tasks that are sensitive to spelling and&#10;pronunciation. As part of our contribution, we release a new set of pre-trained&#10;byte-level Transformer models based on the T5 architecture, as well as all code&#10;and data used in our experiments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.06874" label="2103.06874">
        <attvalues>
          <attvalue for="0" value="CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language&#10;  Representation" />
          <attvalue for="1" value="  Pipelined NLP systems have largely been superseded by end-to-end neural&#10;modeling, yet nearly all commonly-used models still require an explicit&#10;tokenization step. While recent tokenization approaches based on data-derived&#10;subword lexicons are less brittle than manually engineered tokenizers, these&#10;techniques are not equally suited to all languages, and the use of any fixed&#10;vocabulary may limit a model's ability to adapt. In this paper, we present&#10;CANINE, a neural encoder that operates directly on character sequences, without&#10;explicit tokenization or vocabulary, and a pre-training strategy that operates&#10;either directly on characters or optionally uses subwords as a soft inductive&#10;bias. To use its finer-grained input effectively and efficiently, CANINE&#10;combines downsampling, which reduces the input sequence length, with a deep&#10;transformer stack, which encodes context. CANINE outperforms a comparable mBERT&#10;model by 2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite&#10;having 28% fewer model parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.02608" label="2206.02608">
        <attvalues>
          <attvalue for="0" value="What do tokens know about their characters and how do they know it?" />
          <attvalue for="1" value="  Pre-trained language models (PLMs) that use subword tokenization schemes can&#10;succeed at a variety of language tasks that require character-level&#10;information, despite lacking explicit access to the character composition of&#10;tokens. Here, studying a range of models (e.g., GPT- J, BERT, RoBERTa, GloVe),&#10;we probe what word pieces encode about character-level information by training&#10;classifiers to predict the presence or absence of a particular alphabetical&#10;character in a token, based on its embedding (e.g., probing whether the model&#10;embedding for &quot;cat&quot; encodes that it contains the character &quot;a&quot;). We find that&#10;these models robustly encode character-level information and, in general,&#10;larger models perform better at the task. We show that these results generalize&#10;to characters from non-Latin alphabets (Arabic, Devanagari, and Cyrillic).&#10;Then, through a series of experiments and analyses, we investigate the&#10;mechanisms through which PLMs acquire English-language character information&#10;during training and argue that this knowledge is acquired through multiple&#10;phenomena, including a systematic relationship between particular characters&#10;and particular parts of speech, as well as natural variability in the&#10;tokenization of related strings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.14168" label="2110.14168">
        <attvalues>
          <attvalue for="0" value="Training Verifiers to Solve Math Word Problems" />
          <attvalue for="1" value="  State-of-the-art language models can match human performance on many tasks,&#10;but they still struggle to robustly perform multi-step mathematical reasoning.&#10;To diagnose the failures of current models and support research, we introduce&#10;GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math&#10;word problems. We find that even the largest transformer models fail to achieve&#10;high test performance, despite the conceptual simplicity of this problem&#10;distribution. To increase performance, we propose training verifiers to judge&#10;the correctness of model completions. At test time, we generate many candidate&#10;solutions and select the one ranked highest by the verifier. We demonstrate&#10;that verification significantly improves performance on GSM8K, and we provide&#10;strong empirical evidence that verification scales more effectively with&#10;increased data than a finetuning baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.13487" label="2108.13487">
        <attvalues>
          <attvalue for="0" value="Want To Reduce Labeling Cost? GPT-3 Can Help" />
          <attvalue for="1" value="  Data annotation is a time-consuming and labor-intensive process for many NLP&#10;tasks. Although there exist various methods to produce pseudo data labels, they&#10;are often task-specific and require a decent amount of labeled data to start&#10;with. Recently, the immense language model GPT-3 with 175 billion parameters&#10;has achieved tremendous improvement across many few-shot learning tasks. In&#10;this paper, we explore ways to leverage GPT-3 as a low-cost data labeler to&#10;train other models. We find that, to make the downstream model achieve the same&#10;performance on a variety of NLU and NLG tasks, it costs 50% to 96% less to use&#10;labels from GPT-3 than using labels from humans. Furthermore, we propose a&#10;novel framework of combining pseudo labels from GPT-3 with human labels, which&#10;leads to even better performance with limited labeling budget. These results&#10;present a cost-effective data labeling methodology that is generalizable to&#10;many practical applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;GPT-3 Overview. With the success of large pre-trained language modeling GPT-3~\cite{brown2020language} on few-shot learning, more works have been done to improve GPT-3. \cite{zhao2021calibrate} propose to remove the model bias before using GPT-3, which not only increases the accuracy but also reduces the variance.&#10; \cite{lu2021fantastically} work on how to order the few labeled data as input of GPT-3 by constructing&#10;an artificial development set. One concurrent with our work, \cite{yoo2021gpt3mix} consider distilling knowledge from GPT-3 with synthetic data. In their work, the synthetic dataset size is always the same as the original training dataset size. Unlike the most recent works on GPT-3, we treat GPT-3 as a new source of labeler and focus on analyzing the cost of running GPT-3, which is not free according to OpenAI API.&#10;This work is complementary to many other methods based on human labeling, such as few-shot learning~\cite{yin2020meta}, active learning \cite{settles2009active, dor2020active} and transfer learning \cite{ruder2019transfer}. &#10;&#10;Dual supervision. Our method is also related to dual supervision \cite{attenberg2010unified}, which combines two types of labels (one cheap and one expensive) to train a model. Dual supervision typically considers different labeling tasks for humans, for example labeling words or documents \cite{melville2009active}, natural language understanding or generation \cite{su2019dual}, cardinal or ordinal labels \cite{xu2020regression}; here, we consider the same task for different-cost labelers. Labeling oracles with different costs for the same task have also been considered in other areas. Proactive learning \cite{donmez2008proactive} considers active learning with multiple oracles with varied label quality and cost, and oracles can also abstain from labeling an example (``unknown'' label). Multi-fidelity optimization \cite{song2019general} considers optimizing an underlying function (e.g., development accuracy of a neural network) by querying approximations of different precisions and costs. &#10;&#10;Semi-supervised learning and Self Training.&#10;&#10;Using existing model predictions for semi-supervised learning is well-explored in self-training \cite{yarowsky1995unsupervised,mukherjee2020uncertainty}. Prior works in self-training has achieved state-of-art performance in tasks like machine translation \cite{he2019revisiting} and task-oriented dialogue understanding \cite{wang2020adaptive}.&#10;However, prior works in self-training typically used similar-sized models for teacher and student, where the cost of obtaining labels from the teacher is negligible. Learning from GPT-3 is particularly promising because of its impressive few-shot performance, but also challenging because of the GPT-3 labeling cost. To the best of our knowledge, this is the first work that explicitly considers the cost of GPT-3 and its effect in reducing the labeling cost.&#10;&#10;" />
          <attvalue for="4" value="Cost-Effective Labeling, Computer Science, Machine Learning, Linguistics, Data Annotation Methods, Language Model Applications, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2404.07413" label="2404.07413">
        <attvalues>
          <attvalue for="0" value="JetMoE: Reaching Llama2 Performance with 0.1M Dollars" />
          <attvalue for="1" value="  Large Language Models (LLMs) have achieved remarkable results, but their&#10;increasing resource demand has become a major obstacle to the development of&#10;powerful and accessible super-human intelligence. This report introduces&#10;JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens&#10;from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its&#10;low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B&#10;outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the&#10;Llama2-13B-Chat model. These results suggest that LLM training can be much more&#10;cost-effective than generally thought. JetMoE-8B is based on an efficient&#10;Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention&#10;and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B&#10;to have 8B parameters while only activating 2B for each input token, reducing&#10;inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B&#10;is highly open and academia-friendly, using only public datasets and training&#10;code. All training parameters and data mixtures have been detailed in this&#10;report to facilitate future efforts in the development of open foundation&#10;models. This transparency aims to encourage collaboration and further&#10;advancements in the field of accessible and efficient LLMs. The model weights&#10;are publicly available at https://github.com/myshell-ai/JetMoE.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to developing powerful and accessible AI. Although modern LLMs have surpassed human performance on some tasks, they remain inefficient and inflexible. Most LLMs (e.g., Llama, \cite{touvron2023llama}; Pythia, \cite{biderman2023pythia}; GPT-3, \cite{brown2020language}; Mistral, \cite{jiang2023mistral}) use all of their parameters during inference and training, which are referred to as dense models. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture \cite{yuksel2012twenty, shazeer2017outrageously, du2022glam, pan2024dense} has emerged as a popular solution, enabling parameter scaling while keeping computational costs modest. Recent applications of MoE architectures in Transformers \cite{vaswani2017attention} have yielded successful attempts at scaling language models to a substantial size, accompanied by remarkable performance, such as Deepseek MoE~\cite{dai2024deepseekmoe}, Mixtral 8x7B~\cite{jiang2024mixtral}, Grok-1~\cite{grok1_xaiorg}, and DBRX~\cite{dbrx_databricks}. However, even though these models achieve excellent performance, they are not truly open-sourced as the training recipes are not published and may contain proprietary datasets inaccessible outside of large corporations. The open-source community has also attempted to train MoE models, such as OpenMoE~\cite{xue2024openmoe}, but its performance is only on par with weak dense models with similar activation parameters, such as OpenLLaMA~\cite{openlm2023openllama} and TinyLLaMA~\cite{zhang2024tinyllama}.&#10;&#10;To facilitate future efforts on open foundation models, particularly MoE models, we introduce JetMoE-8B, an innovative MoE architecture inspired by ModuleFormer~\cite{shen2023moduleformer} that extends the concept of sparse activation to both the attention and feed-forward layers. Unlike prior works that only apply sparse activation to the feed-forward layer, JetMoE-8B leverages sparse activation in both components to further reduce computational costs while maintaining performance.&#10;&#10;Impressively, JetMoE-8B is trained with a limited \$100k budget, using 1.25T tokens from mixed open-source datasets and 30,000 H100 GPU hours. Despite its low cost, JetMoE-8B outperforms the Llama2-7B model, and JetMoE-8B-Chat outperforms the Llama2-13B-Chat model, demonstrating that LLM training can be much more cost-effective than generally thought. In addition, JetMoE-8B has 8B parameters while only activating 2B for each input token, reducing inference computation by about 70\% compared to Llama2-7B.&#10;&#10;The key advantages of JetMoE-8B include:&#10;&#10;\begin{itemize}[leftmargin=0.85cm]&#10;\item Openness and academia-friendly: JetMoE-8B is trained using only public datasets and open-source training code, making it accessible to many academia research settings. The model can also be finetuned with limited compute budgets (e.g., consumer-grade GPUs).&#10;\item Sparse activation on both attention and feed-forward layers, which significantly reduces training and inference costs. We also propose to share the kv projection in attention experts to improve training stability.&#10;\item Comprehensive open-source data mixture, which ensures high-quality training using only open-source datasets.&#10;&#10;\end{itemize}&#10;&#10;These innovations in JetMoE-8B pave the way for more accessible and efficient LLMs, benefiting the broader AI research community. To foster collaboration and further advancements, we have detailed all the training parameters and data mixture in this report.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Efficient Training Methods, Mathematics, Open Source AI Development" />
        </attvalues>
      </node>
      <node id="2401.02385" label="2401.02385">
        <attvalues>
          <attvalue for="0" value="TinyLlama: An Open-Source Small Language Model" />
          <attvalue for="1" value="  We present TinyLlama, a compact 1.1B language model pretrained on around 1&#10;trillion tokens for approximately 3 epochs. Building on the architecture and&#10;tokenizer of Llama 2, TinyLlama leverages various advances contributed by the&#10;open-source community (e.g., FlashAttention and Lit-GPT), achieving better&#10;computational efficiency. Despite its relatively small size, TinyLlama&#10;demonstrates remarkable performance in a series of downstream tasks. It&#10;significantly outperforms existing open-source language models with comparable&#10;sizes. Our model checkpoints and code are publicly available on GitHub at&#10;https://github.com/jzhang38/TinyLlama.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12392" label="2305.12392">
        <attvalues>
          <attvalue for="0" value="PiVe: Prompting with Iterative Verification Improving Graph-based&#10;  Generative Capability of LLMs" />
          <attvalue for="1" value="  Large language models (LLMs) have shown great abilities of solving various&#10;natural language tasks in different domains. Due to the training objective of&#10;LLMs and their pre-training data, LLMs are not very well equipped for tasks&#10;involving structured data generation. We propose a framework, Prompting with&#10;Iterative Verification (PiVe), to improve graph-based generative capability of&#10;LLMs. We show how a small language model could be trained to act as a verifier&#10;module for the output of an LLM~(i.e., ChatGPT, GPT-4), and to iteratively&#10;improve its performance via fine-grained corrective instructions. We also show&#10;how the verifier module could apply iterative corrections offline for a more&#10;cost-effective solution to the text-to-graph generation task. Experiments on&#10;three graph-based datasets show consistent improvement gained via PiVe.&#10;Additionally, we create GenWiki-HIQ and highlight that the verifier module can&#10;be used as a data augmentation tool to help improve the quality of&#10;automatically generated parallel text-graph datasets.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) like ChatGPT&#10;and GPT-4 \cite{DBLP:journals/corr/abs-2303-08774} have been quite successful in solving different generative and reasoning tasks. The combination of their abilities in leveraging in-context learning as well as instruction following have unlocked new state-of-the-art results across the natural language processing (NLP) field. &#10;The existing LLMs are mostly pre-trained on a huge volume of unstructured data from the internet including books, articles, webtexts, repositories, Wikipedia, etc. Training on unstructured data naturally leads to relatively poor performance when dealing with tasks that demand organizing text into structured machine-readable format. &#10;&#10;A semantic graph, as a form of graph-structured data, stores information in a machine-accessible way \cite{DBLP:reference/fai/3}. Generating a semantic graph from text is known as text-to-graph (T2G) generation and is previously attempted mostly by fine-tuning small language models \cite{DBLP:journals/corr/abs-2209-10754,DBLP:journals/corr/abs-2006-04702}. However, generating graph-structured data remains a challenge for LLMs even in the presence of reasonable number of few-shot examples. In fact, regardless of the number of few-shot examples or prompting style the outputs from LLMs (e.g., GPT-3.5) still contain errors and require correction (\S\ref{main_result}). &#10;&#10;In this paper, we focus on how to improve the graph-based generative capability of LLMs. To this end, we propose the Prompting through Iterative Verification (PiVe) framework shown in Figure~\ref{fig:pive}. Specially, PiVe involves leveraging an external verifier module (i.e., a much smaller LM) and incorporating the feedback from verifier module into the prompt. PiVe iteratively utilises the verifier module and refines the prompts, via corrective instructions, before sending them back into the LLM, leading to substantially improved quality of the generated semantic graphs.&#10;&#10;In particular, to train the verifier modules, we start from a seed dataset of text and graph (T,G) pairs, and construct an arbitrarily large graph-perturbation dataset via a simple procedure which takes any graph $G$ from the seed set and perturbs it arbitrarily on its entities (E), relations (R), or triples~(Tr). The text and perturbed graph ($\bar{G}$), along with a corrective description to invert the applied perturbation (IP) form a verification dataset of (T,$\bar{G}$,IP) triples which serve as the training data for self-supervised learning of our verifier module. The verification dataset could be as large as desired~(i.e., for any seed dataset D, containing graphs of |E| entities, |R| relations, |Tr| triples, it could produce $\mathcal{O}$(|D|$\times$|E|$\times$ |R|$\times$|Tr|) perturbations only by deleting. We then devise fine-tuning and instruction-tuning to train domain-specific and unified verifiers, respectively. &#10;&#10;During the T2G generation via the LLM (e.g., in the zero-shot setting &quot;Transform the text into a semantic graph: Text:... Graph:&quot;), the verifier takes the text T, the output graph from the LLM, and sends a corrective signal to the LLM (e.g., &quot;Transform the text into a semantic graph and add the given triples to the generated semantic graph: Text:... Triples:... Graph:&quot;). This process continues till the verifier module verifies the output as correct and terminates. We refer to this as Iterative Prompting. Additionally, there is another (more cost effective) mode to the verifier module, which starts by calling the LLM once at the start to get an initial graph, and then the rest of the corrective steps are all applied step-by-step and iteratively through the verifier offline. We refer to this as Iterative Offline Correction.&#10;&#10;Our extensive experiment results on three graph-based datasets demonstrate the effectiveness of the proposed PiVe framework in consistently improving the quality of the LLM output via providing iterative corrective guidance by an average of 26\% across 3 datasets. We also create GenWiki-HIQ, a high-quality text-graph dataset and show how verifier module could be leveraged as a data augmentation technique to improve the quality of automatically constructed text-graph datasets.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Enhancement, Graph Generation Techniques, Artificial Intelligence, Mathematics, Natural Language Processing, Data Augmentation Methods" />
        </attvalues>
      </node>
      <node id="2209.10754" label="2209.10754">
        <attvalues>
          <attvalue for="0" value="INFINITY: A Simple Yet Effective Unsupervised Framework for Graph-Text&#10;  Mutual Conversion" />
          <attvalue for="1" value="  Graph-to-text (G2T) generation and text-to-graph (T2G) triple extraction are&#10;two essential tasks for constructing and applying knowledge graphs. Existing&#10;unsupervised approaches turn out to be suitable candidates for jointly learning&#10;the two tasks due to their avoidance of using graph-text parallel data.&#10;However, they are composed of multiple modules and still require both entity&#10;information and relation type in the training process. To this end, we propose&#10;INFINITY, a simple yet effective unsupervised approach that does not require&#10;external annotation tools or additional parallel information. It achieves fully&#10;unsupervised graph-text mutual conversion for the first time. Specifically,&#10;INFINITY treats both G2T and T2G as a bidirectional sequence generation task by&#10;fine-tuning only one pretrained seq2seq model. A novel back-translation-based&#10;framework is then designed to automatically generate continuous synthetic&#10;parallel data. To obtain reasonable graph sequences with structural information&#10;from source texts, INFINITY employs reward-based training loss by leveraging&#10;the advantage of reward augmented maximum likelihood. As a fully unsupervised&#10;framework, INFINITY is empirically verified to outperform state-of-the-art&#10;baselines for G2T and T2G tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.04702" label="2006.04702">
        <attvalues>
          <attvalue for="0" value="CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via&#10;  Cycle Training" />
          <attvalue for="1" value="  Two important tasks at the intersection of knowledge graphs and natural&#10;language processing are graph-to-text (G2T) and text-to-graph (T2G) conversion.&#10;Due to the difficulty and high cost of data collection, the supervised data&#10;available in the two fields are usually on the magnitude of tens of thousands,&#10;for example, 18K in the WebNLG~2017 dataset after preprocessing, which is far&#10;fewer than the millions of data for other tasks such as machine translation.&#10;Consequently, deep learning models for G2T and T2G suffer largely from scarce&#10;training data. We present CycleGT, an unsupervised training method that can&#10;bootstrap from fully non-parallel graph and text data, and iteratively back&#10;translate between the two forms. Experiments on WebNLG datasets show that our&#10;unsupervised model trained on the same number of data achieves performance on&#10;par with several fully supervised models. Further experiments on the&#10;non-parallel GenWiki dataset verify that our method performs the best among&#10;unsupervised baselines. This validates our framework as an effective approach&#10;to overcome the data scarcity problem in the fields of G2T and T2G. Our code is&#10;available at https://github.com/QipengGuo/CycleGT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.02301" label="2305.02301">
        <attvalues>
          <attvalue for="0" value="Distilling Step-by-Step! Outperforming Larger Language Models with Less&#10;  Training Data and Smaller Model Sizes" />
          <attvalue for="1" value="  Deploying large language models (LLMs) is challenging because they are memory&#10;inefficient and compute-intensive for practical applications. In reaction,&#10;researchers train smaller task-specific models by either finetuning with human&#10;labels or distilling using LLM-generated labels. However, finetuning and&#10;distillation require large amounts of training data to achieve comparable&#10;performance to LLMs. We introduce Distilling step-by-step, a new mechanism that&#10;(a) trains smaller models that outperform LLMs, and (b) achieves so by&#10;leveraging less training data needed by finetuning or distillation. Our method&#10;extracts LLM rationales as additional supervision for training small models&#10;within a multi-task framework. We present three findings across 4 NLP&#10;benchmarks: First, compared to both finetuning and distillation, our mechanism&#10;achieves better performance with much fewer labeled/unlabeled training&#10;examples. Second, compared to few-shot prompted LLMs, we achieve better&#10;performance using substantially smaller model sizes. Third, we reduce both the&#10;model size and the amount of data required to outperform LLMs; our finetuned&#10;770M T5 model outperforms the few-shot prompted 540B PaLM model using only 80%&#10;of available data on a benchmark, whereas standard finetuning the same T5 model&#10;struggles to match even by using 100% of the dataset. We release the code at:&#10;https://github.com/google-research/distilling-step-by-step .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Our work distills task-specific knowledge of LLMs into smaller specialist models by leveraging the emergent reasoning capabilities of today's LLMs. We draw on knowledge distillation research and methods that learn from both human-generated rationales and LLM-generated rationales.&#10;&#10;Knowledge distillation from large models.&#10;Knowledge distillation has been successfully used to transfer knowledge from larger, more competent teacher models into smaller student models affordable for practical applications~\cite{bucilua2006model,hinton2015distilling,beyer2022knowledge,west2021symbolic,fu2023specializing}. It supports learning from limited labeled data, since the larger teacher model is often used to generate a training dataset with noisy pseudo labels~\cite{chen2020big,iliopoulos2022weighted,wang2021want,smith2022language,arora2022ask,agrawal2022qameleon}.&#10;The one limitation that knowledge distillation often faces is its reliance on large amounts of unlabelled data required to create a useful noisy training dataset. Although prior work has explored using data augmentation techniques to reduce this hunger for data~\cite{tang2019distilling,liang2020mixkd,srinivas2018knowledge,milli2019model}, we propose an alternative approach: we reduce the need for large unlabeled data by distilling not just labels but also the teacher's rationales.&#10;&#10;Learning with human rationales.&#10;While utilizing LLM-generated rationales is a new exciting area of investigation, using human-generated rationales has a rich history~\cite{hase2021can}. For instance, human rationales can be used to regularize model behavior~\cite{ross2017right}; it can be used as additional inputs to guide a model's predictions~\cite{rajani2019explain}; it can be used to improve overall model performance~\cite{zaidan-etal-2007-using,zhang-etal-2016-rationale,camburu2018snli,hancock2019learning,pruthi2022evaluating}; and human rationales can be used as gold standard labels to make models more interpretable by generating similar rationales~\cite{wiegreffe-etal-2021-measuring,narang2020wt5,eisenstein2022honest}.&#10;Unfortunately, human rationales are expensive.&#10;&#10;Learning with LLM generated rationales.&#10;Today's LLMs are capable of explaining their predictions by generating high-quality reasoning steps~\cite{wei2022chain,kojima2022large}. These reasoning steps have been used to augment input prompts to LLMs, improving their few-shot or zero-shot performance~\cite{wei2022chain,kojima2022large,wang2022self}; reasoning steps have also been used as additional finetuning data ``self-improve'' LLMs~\cite{zelikman2022star,huang2022large}.&#10;Unfortunately, regardless of how LLMs are improved, their large size limits their utility in most test-time applications.&#10;&#10;By contrast, we leverage generated rationales as informative supervision to train smaller task-specific models, i.e.~models that can be deployed without incurring large computation or memory costs.&#10;Several concurrent works have also proposed a similar idea to ours -- that of using extracted rationales as supervision~\cite{wang2022pinto,ho2022large,magister2022teaching,li2023symbolic}. Amongst them, PINTO~\cite{wang2022pinto} relies on an LLM to generate rationales at test-time, and thus does not fully solve deployment challenges. &#10;Compared with~\cite{ho2022large} and \cite{magister2022teaching}, &#10;we go beyond their experiments to provide a granular study by varying training dataset size, exploring downstream model sizes, and demonstrating the effectiveness of our method on fully unlabeled datasets.&#10; " />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Efficient Training Methods, Artificial Intelligence, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2010.06786" label="2010.06786">
        <attvalues>
          <attvalue for="0" value="A Self-supervised Representation Learning of Sentence Structure for&#10;  Authorship Attribution" />
          <attvalue for="1" value="  Syntactic structure of sentences in a document substantially informs about&#10;its authorial writing style. Sentence representation learning has been widely&#10;explored in recent years and it has been shown that it improves the&#10;generalization of different downstream tasks across many domains. Even though&#10;utilizing probing methods in several studies suggests that these learned&#10;contextual representations implicitly encode some amount of syntax, explicit&#10;syntactic information further improves the performance of deep neural models in&#10;the domain of authorship attribution. These observations have motivated us to&#10;investigate the explicit representation learning of syntactic structure of&#10;sentences. In this paper, we propose a self-supervised framework for learning&#10;structural representations of sentences. The self-supervised network contains&#10;two components; a lexical sub-network and a syntactic sub-network which take&#10;the sequence of words and their corresponding structural labels as the input,&#10;respectively. Due to the n-to-1 mapping of words to their structural labels,&#10;each word will be embedded into a vector representation which mainly carries&#10;structural information. We evaluate the learned structural representations of&#10;sentences using different probing tasks, and subsequently utilize them in the&#10;authorship attribution task. Our experimental results indicate that the&#10;structural embeddings significantly improve the classification tasks when&#10;concatenated with the existing pre-trained word embeddings.&#10;" />
          <attvalue for="2" value="&#10;&#10;Word embeddings which can capture semantic similarities have been extensively explored in a wide spectrum of Natural Language Processing (NLP) applications in recent years. Word2Vec \cite{mikolov2013distributed}, FastText \cite{bojanowski2017enriching}, and Glove \cite{pennington2014glove} are some examples. Even though distributional word embeddings produce high quality representations, representing longer pieces of text such as sentences and paragraphs is still an open research problem. A sentence embedding is a contextual representation of a sentence which is often created by transformation of word embeddings through a composition function. There has been a large body of work in the literature which propose different approaches to represent sentences from word embeddings. SkipThought \cite{kiros2015skip}, InferSent \cite{conneau2017supervised}, and Universal Sentence Encoder \cite{cer2018universal} are well-known examples. &#10;&#10;There has been a growing interest in understanding what linguistic knowledge is encoded in deep contextual representation of language. For this purpose, several probing tasks are proposed to understand what these representations are capturing \cite{tenney2019you,hewitt2019structural,conneau2018you,perone2018evaluation}. One of the interesting findings is that despite the existence of explicit syntactic annotations, these learned deep representations encode syntax to some extent \cite{blevins2018deep}. Hewitt et. al. provide an evidence that the entire syntax tree is embedded implicitly in deep model's vector geometry. Kuncoro et. al. \cite{kuncoro2018lstms} show that LSTMs trained on language modeling objectives capture syntax-sensitive dependencies. Even though deep contextual language models implicitly capture syntactic information of sentences, explicit modeling of syntactic structure of sentences has been shown to further improve the results in different NLP tasks including neural language modeling \cite {shen2017neural, havrylov2019cooperative}, machine comprehension \cite{liu2017structural}, summarization \cite{song2018structure}, text generation \cite{bao2019generating}, machine translation \cite{zhang2019syntax, li2017modeling}, authorship attribution \cite{zhang2018syntax, jafariakinabad2019style, jafariakinabad2020syntactic}, etc. Furthermore, Kuncoro et. al. provide evidence that models which have explicit syntactic information result in better performance \cite{kuncoro2018lstms}. Of particular interest, one of the areas where syntactic structure of sentences plays an important role is style-based text classification tasks, including authorship attribution. The syntactic structure of sentences captures the syntactic patterns of sentences adopted by a specific author and reveal how the author structures the sentences in a document. &#10;&#10;Inspired by the above observations, our initial work demonstrates that explicit syntactic information of sentences improves the performance of a recurrent neural network classifier in the domain of authorship attribution \cite{jafariakinabad2019style, jafariakinabad2020syntactic}. We continue this work in this paper by investigating if structural representation of sentences can be learned explicitly. In other words, similar to pre-trained word embeddings which mainly capture semantics, can we have pre-trained embeddings which mainly capture syntactic information of words. Such pre-trained word embeddings can be used in conjunction with semantics embeddings in different domains including authorship attribution. For this purpose, we propose a self-supervised framework using a Siamese network \cite{chopra2005learning} to explicitly learn the structural representation of sentences. The Siamese network is comprised of two identical components; a lexical sub-network and a syntactic sub-network; which take the sequence of words in the sentence and its corresponding linearized syntax parse tree as the inputs, respectively. This model is trained based on a contrastive loss objective where each pair of vectors (lexical and structural) is close to each other in the embedding space if they belong to an identical sentence (positive pairs), and are far from each other if they belong to two different sentences (negative pairs). &#10;&#10;As a result, each word in the sentence is embedded into a vector representation which mainly carries structural information. Due to the $n$-to-$1$ mapping of word types to structural labels, the word representation is deduced into structural representations. In other words, semantically different words (e.g. cold, hot, warm) are mapped to similar structural labels (adjective); hence, semantically different words may have similar structural representations. These pre-trained structural word representations can be used as complementary information to their pre-trained semantic embeddings (e.g. FastText and Glove). We use probing tasks proposed by Conneau et al. \cite{conneau2018you} to investigate the linguistic features learned by such a training. The results indicate that structural embeddings show competitive results compared to the semantic embeddings, and concatenation of structural embeddings with semantic embeddings achieves further improvement. Finally, we investigate the efficiency of the learned structural embeddings of words for the domain of authorship attribution across four datasets. Our experimental results demonstrate classification improvements when structural embeddings are concatenated with the pre-trained word embeddings.&#10;&#10;The remainder of this paper is organized as follows: we elaborate our proposed self-supervised framework in Section \ref{method}. The details of the datasets and experimental configuration are provided and the experimental results reported in Section \ref{results}; We review the related work in Section \ref{literature}. Finally, we conclude this paper in Section \ref{conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Syntactic Structure Analysis, Computer Science, Linguistics, Authorship Attribution Modeling, Sentence Representation Learning, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1905.06316" label="1905.06316">
        <attvalues>
          <attvalue for="0" value="What do you learn from context? Probing for sentence structure in&#10;  contextualized word representations" />
          <attvalue for="1" value="  Contextualized representation models such as ELMo (Peters et al., 2018a) and&#10;BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a&#10;diverse array of downstream NLP tasks. Building on recent token-level probing&#10;work, we introduce a novel edge probing task design and construct a broad suite&#10;of sub-sentence tasks derived from the traditional structured NLP pipeline. We&#10;probe word-level contextual representations from four recent models and&#10;investigate how they encode sentence structure across a range of syntactic,&#10;semantic, local, and long-range phenomena. We find that existing models trained&#10;on language modeling and translation produce strong representations for&#10;syntactic phenomena, but only offer comparably small improvements on semantic&#10;tasks over a non-contextual baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.00572" label="1703.00572">
        <attvalues>
          <attvalue for="0" value="Structural Embedding of Syntactic Trees for Machine Comprehension" />
          <attvalue for="1" value="  Deep neural networks for machine comprehension typically utilizes only word&#10;or character embeddings without explicitly taking advantage of structured&#10;linguistic information such as constituency trees and dependency trees. In this&#10;paper, we propose structural embedding of syntactic trees (SEST), an algorithm&#10;framework to utilize structured information and encode them into vector&#10;representations that can boost the performance of algorithms for the machine&#10;comprehension. We evaluate our approach using a state-of-the-art neural&#10;attention model on the SQuAD dataset. Experimental results demonstrate that our&#10;model can accurately identify the syntactic boundaries of the sentences and&#10;extract answers that are syntactically coherent over the baseline methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.05658" label="1806.05658">
        <attvalues>
          <attvalue for="0" value="Structure-Infused Copy Mechanisms for Abstractive Summarization" />
          <attvalue for="1" value="  Seq2seq learning has produced promising results on summarization. However, in&#10;many cases, system summaries still struggle to keep the meaning of the original&#10;intact. They may miss out important words or relations that play critical roles&#10;in the syntactic structure of source sentences. In this paper, we present&#10;structure-infused copy mechanisms to facilitate copying important words and&#10;relations from the source sentence to summary sentence. The approach naturally&#10;combines source dependency structure with the copy mechanism of an abstractive&#10;sentence summarizer. Experimental results demonstrate the effectiveness of&#10;incorporating source-side syntactic information in the system, and our proposed&#10;approach compares favorably to state-of-the-art methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.02878" label="1905.02878">
        <attvalues>
          <attvalue for="0" value="Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word&#10;  Representations" />
          <attvalue for="1" value="  Syntax has been demonstrated highly effective in neural machine translation&#10;(NMT). Previous NMT models integrate syntax by representing 1-best tree outputs&#10;from a well-trained parsing system, e.g., the representative Tree-RNN and&#10;Tree-Linearization methods, which may suffer from error propagation. In this&#10;work, we propose a novel method to integrate source-side syntax implicitly for&#10;NMT. The basic idea is to use the intermediate hidden representations of a&#10;well-trained end-to-end dependency parser, which are referred to as&#10;syntax-aware word representations (SAWRs). Then, we simply concatenate such&#10;SAWRs with ordinary word embeddings to enhance basic NMT models. The method can&#10;be straightforwardly integrated into the widely-used sequence-to-sequence&#10;(Seq2Seq) NMT models. We start with a representative RNN-based Seq2Seq baseline&#10;system, and test the effectiveness of our proposed method on two benchmark&#10;datasets of the Chinese-English and English-Vietnamese translation tasks,&#10;respectively. Experimental results show that the proposed approach is able to&#10;bring significant BLEU score improvements on the two datasets compared with the&#10;baseline, 1.74 points for Chinese-English translation and 0.80 point for&#10;English-Vietnamese translation, respectively. In addition, the approach also&#10;outperforms the explicit Tree-RNN and Tree-Linearization methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.06194" label="1909.06194">
        <attvalues>
          <attvalue for="0" value="Style-aware Neural Model with Application in Authorship Attribution" />
          <attvalue for="1" value="  Writing style is a combination of consistent decisions associated with a&#10;specific author at different levels of language production, including lexical,&#10;syntactic, and structural. In this paper, we introduce a style-aware neural&#10;model to encode document information from three stylistic levels and evaluate&#10;it in the domain of authorship attribution. First, we propose a simple way to&#10;jointly encode syntactic and lexical representations of sentences.&#10;Subsequently, we employ an attention-based hierarchical neural network to&#10;encode the syntactic and semantic structure of sentences in documents while&#10;rewarding the sentences which contribute more to capturing the writing style.&#10;Our experimental results, based on four benchmark datasets, reveal the benefits&#10;of encoding document information from all three stylistic levels when compared&#10;to the baseline methods in the literature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.00068" label="2301.00068">
        <attvalues>
          <attvalue for="0" value="Inconsistencies in Masked Language Models" />
          <attvalue for="1" value="  Learning to predict masked tokens in a sequence has been shown to be a&#10;helpful pretraining objective for powerful language models such as PaLM2. After&#10;training, such masked language models (MLMs) can provide distributions of&#10;tokens in the masked positions in a sequence. However, this paper shows that&#10;distributions corresponding to different masking patterns can demonstrate&#10;considerable inconsistencies, i.e., they cannot be derived from a coherent&#10;joint distribution when considered together.&#10;  This fundamental flaw in MLMs can lead to self-contradictory behaviors during&#10;inference. On various benchmark datasets including MMLU, MLMs can give&#10;different predictions to the same input question. From BERT-base to UL2-20B, we&#10;show that such inconsistencies exist ubiquitously in MLMs of diverse sizes and&#10;configurations. In light of our observations, we further propose an&#10;inference-time strategy for MLMs called Ensemble of Conditionals. It jointly&#10;considers a selected range of inconsistent conditionals directly produced by&#10;the MLM for the final prediction, which often leads to considerable accuracy&#10;improvement.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretraining objectives of large language models can be roughly divided into two categories. First, vanilla next token prediction (also known as casual language modeling) aims to learn the distribution of the next token in a sequence given the context to the left \cite{brown2020language}. Second, the masked language modeling (MLM) objective, which masks out a portion of the tokens in a sequence and asks the model to predict them, aims to learn the distribution of one or more tokens given surrounding context \cite{devlin2018bert, raffel2020exploring}.&#10;&#10;While GPT-3 \cite{brown2020language} used vanilla next token prediction, following work such as PaLM-2 \cite{anil2023palm}, U-PaLM \cite{tay2022transcending}, GPT-FIM \cite{bavarian2022efficient}, UL2 \cite{tay2022unifying}, and GLM \cite{zeng2022glm} have hinted that incorporating the MLM objective could be highly beneficial to performance. In addition, \cite{tay2022transcending} has demonstrated that such bidirectional conditionals provide strong infilling capabilities. Empirically speaking, predicting masked tokens in the middle of the sentence can be seen as a natural data augmentation technique to vanilla next token prediction, which might be helpful to alleviating the data scarcity problem \cite{xue2023repeat} in the current large model era.&#10;&#10;One may notice that, unlike the unidirectional conditional distributions that vanilla next token prediction learns, the bidirectional conditionals that MLMs learn are overly abundant in terms of representing a coherent joint distribution. Therefore, they are not guaranteed to be self-consistent. This paper explains our effort on exposing and quantifying this issue and corresponding strategies during inference.&#10;&#10;To begin with, a simple example for such inconsistencies is shown in Figure \ref{fig:t5_example}. In this example, we obtain the bidirectional conditional distributions that the T5 model learned using two input masked sequences. The two similar sequences are designed with a small difference, in order to examine if the resulting conditionals satisfy a basic law of probabilities (hold consistency). Results clearly show otherwise. We design experiments to quantify such inconsistencies on benchmark datasets in Section \ref{sec:exposing}.&#10;We further show an inference-time ensemble algorithm in Section \ref{sec:ensemble} which utilizes many inconsistent conditionals for a more accurate prediction. We demonstrate that ensembling the numerous inconsistent conditionals directly provided by the MLM can improve its performance (Figure \ref{fig:eoc_accuracy_aggregate}).&#10;&#10;In summary, our contributions are&#10;(1) We expose the commonly overlooked flaw in MLMs that they can represent inconsistent distributions depending on the mask patterns.&#10;(2) We quantify such inconsistencies in benchmark datasets including Lambada \cite{paperno2016Lambada}, MMLU\cite{hendrycks2021measuring} and BigBench \cite{srivastava2023beyond}. For example, on multiple choice questions in MMLU, 2 different distributions given by UL2-20B disagree on the answer 14\% of the time on average.&#10;(3) We show that the numerous inconsistent conditionals can be ensembled together to considerably improve accuracy on said benchmarks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Inference Strategy Improvement, Language Model Limitations, Artificial Intelligence, Mathematics, Masked Token Prediction" />
        </attvalues>
      </node>
      <node id="2207.14255" label="2207.14255">
        <attvalues>
          <attvalue for="0" value="Efficient Training of Language Models to Fill in the Middle" />
          <attvalue for="1" value="  We show that autoregressive language models can learn to infill text after we&#10;apply a straightforward transformation to the dataset, which simply moves a&#10;span of text from the middle of a document to its end. While this data&#10;augmentation has garnered much interest in recent years, we provide extensive&#10;evidence that training models with a large fraction of data transformed in this&#10;way does not harm the original left-to-right generative capability, as measured&#10;by perplexity and sampling evaluations across a wide range of scales. Given the&#10;usefulness, simplicity, and efficiency of training models to fill-in-the-middle&#10;(FIM), we suggest that future autoregressive language models be trained with&#10;FIM by default. To this end, we run a series of ablations on key&#10;hyperparameters, such as the data transformation frequency, the structure of&#10;the transformation, and the method of selecting the infill span. We use these&#10;ablations to prescribe strong default settings and best practices to train FIM&#10;models. We have released our best infilling model trained with best practices&#10;in our API, and release our infilling benchmarks to aid future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Masked language modeling is closely related to text infilling in that consecutive runs of masked tokens can be interpreted as spans that the model must infill. While early masked language models like BERT \cite{bert} masked tokens randomly, T5 \cite{T5}, SpanBERT \cite{spanbert}, and BART \cite{bart} demonstrated improvements when contiguous runs of tokens are masked. However, because these models focus on representation learning, the span lengths are typically much shorter than a sentence or even a single line of code. Within our modalities of interest, DOBF \cite{dobf} trains BERT on code, and HTLM \cite{htlm} trains BART on HTML data. &#10;&#10;Text infilling can also be seen as a special case of autoregressive language modeling where the standard left to right generation order is replaced by a more flexible ordering. XLNet \cite{xlnet} modifies the attention mask in a standard transformer to enable token generation in any user-specified order, while Insertion Transformer \cite{insertion-transformer}, KERMIT \cite{kermit}, and InDIGO \cite{indigo} allow the model to predict a location for the next token before predicting the token. Similarly, Blank Language models \cite{blanklm} generate text by iteratively selecting a blank and replacing it with a token (and optionally more blanks).&#10;&#10;Similar to our work, \cite{zhu-infilling}, \cite{donahue}, GLM \cite{glm}, CM3 \cite{cm3}, and InCoder \cite{incoder} utilize left-to-right autoregressive modeling by moving the infill regions to the end of context, with regions separated by sentinels. Notably, \cite{donahue} explore infilling spans of varying granularities, such as words, sentences, or paragraphs, and InCoder \cite{incoder} uses a similar evaluation framework to ours by studying infilling capabilities on sampling based benchmarks created from HumanEval \cite{codex}. While several of these works support infilling multiple spans, we focus on the single span setting for practicality (e.g. in computer-based text generation, where the placement of cursor implies the location we want to infill). Additionally, our paper emphasizes the computational efficiency of training for infilling at scale. While we do not study syntactically or semantically motivated infilling spans, we show selecting spans at the character level improves the robustness of infilling.&#10;&#10;Text infilling can also be performed using a GAN \cite{maskgan}, but REINFORCE is required to deal with the discreteness of text. Text infilling can also be done through gradient search \cite{tigs}, where tokens within the infilled span are optimized with gradient descent and collapsed to the nearest neighbor.&#10;&#10;Overall, there are two approaches for imbuing models with infilling capabilities: first, through new architectures like SpanBERT and XLNet; second, through data formatting. In general, the latter approach can be seen as altering the behavior of a language model through control codes, which was motivated in CTRL \cite{ctrl} to improve the steerability of generation. DistAug \cite{distaug} is another related work that trains jointly on transformed data while conditioning on the transformation type. While infilling is a specific use case that can be realized through both architecture and data, it is generally easier and more universal to learn additional skills by introducing new training distributions than hardwiring them.&#10;&#10;The strongest infilling system at scale to our knowledge currently is code-davinci-002 released this past March \cite{edit_insert}. The present paper describes some of the early research that went into powering the infilling capabilities of this more powerful model. In Appendix \ref{tab:top-models}, we present a comparison between this system, our 6.9B models, and the InCoder 6.7B model on our infilling benchmarks. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2210.10599" label="2210.10599">
        <attvalues>
          <attvalue for="0" value="Self-supervised Graph Masking Pre-training for Graph-to-Text Generation" />
          <attvalue for="1" value="  Large-scale pre-trained language models (PLMs) have advanced Graph-to-Text&#10;(G2T) generation by processing the linearised version of a graph. However, the&#10;linearisation is known to ignore the structural information. Additionally, PLMs&#10;are typically pre-trained on free text which introduces domain mismatch between&#10;pre-training and downstream G2T generation tasks. To address these&#10;shortcomings, we propose graph masking pre-training strategies that neither&#10;require supervision signals nor adjust the architecture of the underlying&#10;pre-trained encoder-decoder model. When used with a pre-trained T5, our&#10;approach achieves new state-of-the-art results on WebNLG+2020 and&#10;EventNarrative G2T generation datasets. Our method also shows to be very&#10;effective in the low-resource setting.&#10;" />
          <attvalue for="2" value="&#10;&#10;Graph-to-Text (G2T) generation \cite{DBLP:journals/jair/GattK18} is the task of generating natural language from graph-structured data. While there are several tasks that could leverage a G2T component~\cite{,DBLP:conf/ijcai/ZhouYHZXZ18,DBLP:conf/emnlp/JiKHWZH20,DBLP:conf/iclr/ChenCSWC21} the direct generation of text description from knowledge graphs (KGs) have attracted a lot of attention due to its potential in providing a more accessible presentation of knowledge to non-experts~\cite{schmitt-etal-2020-unsupervised}.&#10;&#10;In parallel, Transformer-based \cite{DBLP:conf/nips/VaswaniSPUJGKP17} pre-trained language models (PLMs) such as BART \cite{lewis2019bart}, and T5 \cite{raffel2019exploring} have facilitated state-of-the-art (SotA) results on several tasks, including earlier SotA results for G2T~\cite{DBLP:journals/corr/abs-2007-08426,DBLP:conf/inlg/KaleR20a,mager-etal-2020-gpt}. It has been argued that their success, in part, is due to factual memorisation that guides the generation~\cite{DBLP:journals/corr/abs-2007-08426}. Although PLMs benefit the G2T generation, the linearisation step required to use these models ignores the structural information of the graph \cite{DBLP:conf/acl/WangYLJR21}, while explicitly modelling structured data could also lead to catastrophic forgetting of distributional knowledge \cite{DBLP:conf/emnlp/RibeiroZG21}. &#10;&#10;To address this, \cite{DBLP:conf/acl/WangYLJR21} proposed adding extra positional embedding layers to capture the inter-dependency structures of input graphs. \cite{DBLP:conf/emnlp/RibeiroZG21} proposed using a structure-aware adapter in PLMs to supplement the input with its graph structure. For table data, \cite{xing-wan-2021-structure} considered the structure of the table input by predicting the surrounding cells for a cell in a table. However, these methods either change the design of the PLMs (limiting their use for other task settings) or require labelled training data to capture the graph structure information.&#10;&#10;In this work, we propose self-supervised graph masking pre-training strategies to enhance the structure awareness of PLMs. To achieve this, we formulate several graph masking strategies to inject local and global awareness of the input structure into the PLM. Our method has two key advantages: (i) it does not require to introduce extra layers or change of architecture in the underlying PLM, and (ii) it pre-trains the PLMs in a self-supervised setting on graphs, without requiring labelled training data. Starting from an existing PLM, we further pre-train it with our approach, then the fine-tuning on downstream tasks is done as per usual. &#10;&#10;We conduct extensive experiments on three G2T generation datasets of diverse graphs. Our empirical findings highlight that our self-supervised strategies significantly outperform a strong underlying T5 baseline and achieve two new SotA results on two of the datasets WebNLG+2020~\cite{zhou-lampouras-2020-webnlg} and EventNarrative~\cite{DBLP:conf/nips/ColasSWW21}. Additionally, we show our pre-training strategies are very efficient in utilising data and have a great potential for low-resource setting. &#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Deep Learning Architectures, Language Model Pre-training, Mathematics, Graph-to-Text Generation" />
        </attvalues>
      </node>
      <node id="2009.11692" label="2009.11692">
        <attvalues>
          <attvalue for="0" value="Language Generation with Multi-Hop Reasoning on Commonsense Knowledge&#10;  Graph" />
          <attvalue for="1" value="  Despite the success of generative pre-trained language models on a series of&#10;text generation tasks, they still suffer in cases where reasoning over&#10;underlying commonsense knowledge is required during generation. Existing&#10;approaches that integrate commonsense knowledge into generative pre-trained&#10;language models simply transfer relational knowledge by post-training on&#10;individual knowledge triples while ignoring rich connections within the&#10;knowledge graph. We argue that exploiting both the structural and semantic&#10;information of the knowledge graph facilitates commonsense-aware text&#10;generation. In this paper, we propose Generation with Multi-Hop Reasoning Flow&#10;(GRF) that enables pre-trained models with dynamic multi-hop reasoning on&#10;multi-relational paths extracted from the external commonsense knowledge graph.&#10;We empirically show that our model outperforms existing baselines on three text&#10;generation tasks that require reasoning over commonsense knowledge. We also&#10;demonstrate the effectiveness of the dynamic multi-hop reasoning module with&#10;reasoning paths inferred by the model that provide rationale to the generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.10439" label="2010.10439">
        <attvalues>
          <attvalue for="0" value="Open Question Answering over Tables and Text" />
          <attvalue for="1" value="  In open question answering (QA), the answer to a question is produced by&#10;retrieving and then analyzing documents that might contain answers to the&#10;question. Most open QA systems have considered only retrieving information from&#10;unstructured text. Here we consider for the first time open QA over both&#10;tabular and textual data and present a new large-scale dataset Open&#10;Table-and-Text Question Answering (OTT-QA) to evaluate performance on this&#10;task. Most questions in OTT-QA require multi-hop inference across tabular data&#10;and unstructured text, and the evidence required to answer a question can be&#10;distributed in different ways over these two types of input, making evidence&#10;retrieval challenging -- our baseline model using an iterative retriever and&#10;BERT-based reader achieves an exact match score less than 10%. We then propose&#10;two novel techniques to address the challenge of retrieving and aggregating&#10;evidence for OTT-QA. The first technique is to use &quot;early fusion&quot; to group&#10;multiple highly relevant tabular and textual units into a fused block, which&#10;provides more context for the retriever to search for. The second technique is&#10;to use a cross-block reader to model the cross-dependency between multiple&#10;retrieved evidence with global-local sparse attention. Combining these two&#10;techniques improves the score significantly, to above 27%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.09447" label="1904.09447">
        <attvalues>
          <attvalue for="0" value="An Unsupervised Joint System for Text Generation from Knowledge Graphs&#10;  and Semantic Parsing" />
          <attvalue for="1" value="  Knowledge graphs (KGs) can vary greatly from one domain to another. Therefore&#10;supervised approaches to both graph-to-text generation and text-to-graph&#10;knowledge extraction (semantic parsing) will always suffer from a shortage of&#10;domain-specific parallel graph-text data; at the same time, adapting a model&#10;trained on a different domain is often impossible due to little or no overlap&#10;in entities and relations. This situation calls for an approach that (1) does&#10;not need large amounts of annotated data and thus (2) does not need to rely on&#10;domain adaptation techniques to work well in different domains. To this end, we&#10;present the first approach to unsupervised text generation from KGs and show&#10;simultaneously how it can be used for unsupervised semantic parsing. We&#10;evaluate our approach on WebNLG v2.1 and a new benchmark leveraging scene&#10;graphs from Visual Genome. Our system outperforms strong baselines for both&#10;text$\leftrightarrow$graph conversion tasks without any manual adaptation from&#10;one dataset to the other. In additional experiments, we investigate the impact&#10;of using different unsupervised objectives.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.10433" label="2005.10433">
        <attvalues>
          <attvalue for="0" value="Text-to-Text Pre-Training for Data-to-Text Tasks" />
          <attvalue for="1" value="  We study the pre-train + fine-tune strategy for data-to-text tasks. Our&#10;experiments indicate that text-to-text pre-training in the form of T5, enables&#10;simple, end-to-end transformer based models to outperform pipelined neural&#10;architectures tailored for data-to-text generation, as well as alternative&#10;language model based pre-training techniques such as BERT and GPT-2.&#10;Importantly, T5 pre-training leads to better generalization, as evidenced by&#10;large improvements on out-of-domain test sets. We hope our work serves as a&#10;useful baseline for future research, as transfer learning becomes ever more&#10;prevalent for data-to-text tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Data-to-Text Early research on data-to-text focused on rule-based methods \cite{reiter2000building}, while recent works have favored neural approaches \cite{wen2015semantically}. \cite{liu2018table} generate text by conditioning language models on tables, \cite{puduppully2019data} explicitly model entities and \cite{marcheggiani2018deep} encode structured data using graph convolutional networks. \cite{ferreira2019neural} and \cite{moryossef2019step} find that neural pipelined approaches perform better than end-to-end models. \\&#10;Transfer Learning \cite{devlin2018bert} showed that unsupervised pre-training can greatly benefit tasks like, question answering, summarization etc. In particular, \cite{raffel2019exploring} perform a large scale study of different training objectives, model capacity and size of data. \cite{peng2020few} and \cite{chen2019few} show that pre-training in the form of GPT-2 can indeed improve performance on the data-to-text task as well. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2005.09123" label="2005.09123">
        <attvalues>
          <attvalue for="0" value="GPT-too: A language-model-first approach for AMR-to-text generation" />
          <attvalue for="1" value="  Meaning Representations (AMRs) are broad-coverage sentence-level semantic&#10;graphs. Existing approaches to generating text from AMR have focused on&#10;training sequence-to-sequence or graph-to-sequence models on AMR annotated data&#10;only. In this paper, we propose an alternative approach that combines a strong&#10;pre-trained language model with cycle consistency-based re-scoring. Despite the&#10;simplicity of the approach, our experimental results show these models&#10;outperform all previous techniques on the English LDC2017T10dataset, including&#10;the recent use of transformer architectures. In addition to the standard&#10;evaluation metrics, we provide human evaluation experiments that further&#10;substantiate the strength of our approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.08021" label="2105.08021">
        <attvalues>
          <attvalue for="0" value="Stage-wise Fine-tuning for Graph-to-Text Generation" />
          <attvalue for="1" value="  Graph-to-text generation has benefited from pre-trained language models&#10;(PLMs) in achieving better performance than structured graph encoders. However,&#10;they fail to fully utilize the structure information of the input graph. In&#10;this paper, we aim to further improve the performance of the pre-trained&#10;language model by proposing a structured graph-to-text model with a two-step&#10;fine-tuning mechanism which first fine-tunes the model on Wikipedia before&#10;adapting to the graph-to-text generation. In addition to using the traditional&#10;token and position embeddings to encode the knowledge graph (KG), we propose a&#10;novel tree-level embedding method to capture the inter-dependency structures of&#10;the input graph. This new approach has significantly improved the performance&#10;of all text generation metrics for the English WebNLG 2017 dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.00276" label="2111.00276">
        <attvalues>
          <attvalue for="0" value="EventNarrative: A large-scale Event-centric Dataset for Knowledge&#10;  Graph-to-Text Generation" />
          <attvalue for="1" value="  We introduce EventNarrative, a knowledge graph-to-text dataset from publicly&#10;available open-world knowledge graphs. Given the recent advances in&#10;event-driven Information Extraction (IE), and that prior research on&#10;graph-to-text only focused on entity-driven KGs, this paper focuses on&#10;event-centric data. However, our data generation system can still be adapted to&#10;other other types of KG data. Existing large-scale datasets in the&#10;graph-to-text area are non-parallel, meaning there is a large disconnect&#10;between the KGs and text. The datasets that have a paired KG and text, are&#10;small scale and manually generated or generated without a rich ontology, making&#10;the corresponding graphs sparse. Furthermore, these datasets contain many&#10;unlinked entities between their KG and text pairs. EventNarrative consists of&#10;approximately 230,000 graphs and their corresponding natural language text, 6&#10;times larger than the current largest parallel dataset. It makes use of a rich&#10;ontology, all of the KGs entities are linked to the text, and our manual&#10;annotations confirm a high data quality. Our aim is two-fold: help break new&#10;ground in event-centric research where data is lacking, and to give researchers&#10;a well-defined, large-scale dataset in order to better evaluate existing and&#10;future knowledge graph-to-text models. We also evaluate two types of baseline&#10;on EventNarrative: a graph-to-text specific model and two state-of-the-art&#10;language models, which previous work has shown to be adaptable to the knowledge&#10;graph-to-text domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10559" label="2212.10559">
        <attvalues>
          <attvalue for="0" value="Why Can GPT Learn In-Context? Language Models Implicitly Perform&#10;  Gradient Descent as Meta-Optimizers" />
          <attvalue for="1" value="  Large pretrained language models have shown surprising in-context learning&#10;(ICL) ability. With a few demonstration input-label pairs, they can predict the&#10;label for an unseen input without parameter updates. Despite the great success&#10;in performance, its working mechanism still remains an open question. In this&#10;paper, we explain language models as meta-optimizers and understand in-context&#10;learning as implicit finetuning. Theoretically, we figure out that Transformer&#10;attention has a dual form of gradient descent. On top of it, we understand ICL&#10;as follows: GPT first produces meta-gradients according to the demonstration&#10;examples, and then these meta-gradients are applied to the original GPT to&#10;build an ICL model. We comprehensively compare the behaviors of in-context&#10;learning and explicit finetuning on real tasks to provide empirical evidence&#10;that supports our understanding. Experimental results show that in-context&#10;learning behaves similarly to explicit finetuning from multiple perspectives.&#10;Inspired by the dual form between Transformer attention and gradient descent,&#10;we design a momentum-based attention by analogy with gradient descent with&#10;momentum. The improved performance over vanilla attention further supports our&#10;understanding from another perspective, and more importantly, shows the&#10;potential to utilize our understanding for future model design. The code is&#10;available at \url{https://aka.ms/icl}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Language Model Analysis, Linguistics, Meta Learning Mechanisms, Artificial Intelligence, Mathematics, Natural Language Processing, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="2004.05686" label="2004.05686">
        <attvalues>
          <attvalue for="0" value="XtremeDistil: Multi-stage Distillation for Massive Multilingual Models" />
          <attvalue for="1" value="  Deep and large pre-trained language models are the state-of-the-art for&#10;various natural language processing tasks. However, the huge size of these&#10;models could be a deterrent to use them in practice. Some recent and concurrent&#10;works use knowledge distillation to compress these huge models into shallow&#10;ones. In this work we study knowledge distillation with a focus on&#10;multi-lingual Named Entity Recognition (NER). In particular, we study several&#10;distillation strategies and propose a stage-wise optimization scheme leveraging&#10;teacher internal representations that is agnostic of teacher architecture and&#10;show that it outperforms strategies employed in prior works. Additionally, we&#10;investigate the role of several factors like the amount of unlabeled data,&#10;annotation resources, model architecture and inference latency to name a few.&#10;We show that our approach leads to massive compression of MBERT-like teacher&#10;models by upto 35x in terms of parameters and 51x in terms of latency for batch&#10;inference while retaining 95% of its F1-score for NER over 41 languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;{Model compression and knowledge distillation:} Prior works in the vision community dealing with huge architectures like AlexNet and ResNet have addressed this challenge in two ways. Works in model compression use quantization~\cite{DBLP:journals/corr/GongLYB14}, low-precision training and pruning the network, as well as their combination~\cite{HanMao16} to reduce the memory footprint. On the other hand, works in knowledge distillation leverage student teacher models. These approaches include using soft logits as targets~\cite{DBLP:conf/nips/BaC14}, increasing the temperature of the softmax to match that of the teacher~\cite{DBLP:journals/corr/HintonVD15} as well as using teacher representations~\cite{DBLP:journals/corr/RomeroBKCGB14} &#10;(refer to~\cite{DBLP:journals/corr/abs-1710-09282} for a survey). &#10;&#10;{Recent and concurrent Works:} \cite{DBLP:journals/corr/abs-1904-09482,zhu-etal-2019-panlp,Clark-2019} leverage ensembling to distil knowledge from several multi-task deep neural networks into a single model. \cite{sun2019patient,sanh2019};\cite{aguilar2019knowledge} train student models leveraging architectural knowledge of the teacher models which adds architectural constraints (e.g., embedding dimension) on the student. In order to address this shortcoming, more recent works combine task-specific distillation with pre-training the student model with arbitrary embedding dimension but still relying on transformer architectures \cite{turc2019wellread}; \cite{jiao2019tinybert}; \cite{zhao2019extreme}. &#10;&#10;\cite{izsak2019training,Shi_2019} extend these for sequence tagging for Part-of-Speech (POS) tagging and Named Entity Recognition (NER) in English. The one closest to our work \cite{Tsai_2019} extends the above for multilingual NER.&#10;&#10;Most of these works rely on general corpora for pre-training and task-specific labeled data for distillation. To harness additional knowledge, \cite{turc2019wellread} leverage task-specific unlabeled data. \cite{DBLP:journals/corr/abs-1903-12136,jiao2019tinybert} use rule-and embedding-based data augmentation in absence of such unlabeled data. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Knowledge Distillation, Language Model Compression, Artificial Intelligence, Multilingual NER" />
        </attvalues>
      </node>
      <node id="1908.06625" label="1908.06625">
        <attvalues>
          <attvalue for="0" value="Bilingual Lexicon Induction with Semi-supervision in Non-Isometric&#10;  Embedding Spaces" />
          <attvalue for="1" value="  Recent work on bilingual lexicon induction (BLI) has frequently depended&#10;either on aligned bilingual lexicons or on distribution matching, often with an&#10;assumption about the isometry of the two spaces. We propose a technique to&#10;quantitatively estimate this assumption of the isometry between two embedding&#10;spaces and empirically show that this assumption weakens as the languages in&#10;question become increasingly etymologically distant. We then propose Bilingual&#10;Lexicon Induction with Semi-Supervision (BLISS) --- a semi-supervised approach&#10;that relaxes the isometric assumption while leveraging both limited aligned&#10;bilingual lexicons and a larger set of unaligned word embeddings, as well as a&#10;novel hubness filtering technique. Our proposed method obtains state of the art&#10;results on 15 of 18 language pairs on the MUSE dataset, and does particularly&#10;well when the embedding spaces don't appear to be isometric. In addition, we&#10;also show that adding supervision stabilizes the learning procedure, and is&#10;effective even with minimal supervision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;\cite{mikolov2013exploiting} first used anchor points to align two embedding spaces, leveraging the fact that these spaces exhibit similar structure across languages. Since then, several approaches have been proposed for learning bilingual dictionaries \cite{faruqui2014improving, zou2013bilingual, xing2015normalized}. \cite{xing2015normalized} showed that adding an orthogonal constraint significantly improves performance, and admits a closed form solution. This was further corroborated by the work of \cite{smith2017offline}, who showed that in orthogonality was necessary for self-consistency. \cite{supervisedArtetxe} showed the equivalence between the different methods, and their subsequent work \cite{artetxe2018generalizing} analyzed different techniques proposed in various works (like embedding centering, whitening etc.), and showed that leveraging a combination of different methods showed significant performance gains. &#10;&#10;However, the validity of this orthogonality assumption has of late come into question: \cite{zhang2017earth} found that the Wasserstein distance between distant language pairs was considerably higher&#10;, while \cite{sogaard2018limitations} explored the orthogonality assumption using eigenvector similarity. We find our weak orthogonality constraint (along the lines of \cite{zhangAdversarial}) when used in our semi-supervised framework to be more robust to this.&#10;&#10;There has also recently been an increasing focus on generating these bilingual mappings without an aligned bilingual dictionary, i.e., in an unsupervised manner. \cite{zhangAdversarial} and \cite{lample2018word} both use adversarial training for aligning two monolingual embedding spaces without any seed lexicon, while \cite{zhang2017earth} used a Wasserstein GAN to achieve this adversarial alignment, and use an earth-mover based fine-tuning approach; while \cite{grave2018unsupervised} formulate this as a joint estimation of an orthogonal matrix and a permutation matrix. However, we show that adding a little supervision, which is usually easy to obtain, improves performance. &#10;&#10;Another vein of research \cite{jawanpuria2018learning, vecmap(U)++, kementchedjhieva2018generalizing} has been to learn orthogonal mappings from both the source and the target embedding spaces into a common embedding space and doing the translations in the common embedding space. &#10;\cite{unsupervisedArtetxeWord} and \cite{sogaard2018limitations} motivate the utility of using both the supervised seed dictionaries and, to some extent, the structure of the monolingual embedding spaces. They use iterative Procrustes refinement starting with a small seed dictionary to learn a mapping; but doing may lead to sub-optimal performance for distant language pairs. However, these methods are close to our methods in spirit, and consequently form the baselines for our experiments.&#10;&#10;Another avenue of research has been to try and modify the underlying embedding generation algorithms. \cite{cao2016distribution} modify the CBOW algorithm \cite{Word2Vec} by augmenting the CBOW loss to match the first and second order moments from the source and target latent spaces, thereby ensuring the source and target embedding spaces follow the same distribution. \cite{luong2015bilingual}, in their work, use the aligned words to jointly learn the embedding spaces of both the source and target language, by trying to predict the context of a word in the other language, given an alignment. An issue with the proposed method is that it requires the retraining of embeddings, and cannot leverage a rich collection of precomputed vectors (like ones provided by Word2Vec \cite{Word2Vec}, Glove \cite{Glove} and FastText \cite{fastext}).&#10; " />
          <attvalue for="4" value="Machine Learning, Computer Science, Bilingual Lexicon Induction, Linguistics, Language Embedding Spaces, Mathematics, Natural Language Processing, Semi-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2305.12990" label="2305.12990">
        <attvalues>
          <attvalue for="0" value="Sentence Representations via Gaussian Embedding" />
          <attvalue for="1" value="  Recent progress in sentence embedding, which represents the meaning of a&#10;sentence as a point in a vector space, has achieved high performance on tasks&#10;such as a semantic textual similarity (STS) task. However, sentence&#10;representations as a point in a vector space can express only a part of the&#10;diverse information that sentences have, such as asymmetrical relationships&#10;between sentences. This paper proposes GaussCSE, a Gaussian distribution-based&#10;contrastive learning framework for sentence embedding that can handle&#10;asymmetric relationships between sentences, along with a similarity measure for&#10;identifying inclusion relations. Our experiments show that GaussCSE achieves&#10;the same performance as previous methods in natural language inference tasks,&#10;and is able to estimate the direction of entailment relations, which is&#10;difficult with point representations.&#10;" />
          <attvalue for="2" value="&#10;Sentence embeddings are representations to describe a sentence's meaning and are widely used in natural language tasks such as document classification~\cite{classification}, sentence retrieval~\cite{retrieval}, and question answering~\cite{qa}. &#10;In recent years, machine-learning-based sentence embedding methods with pre-trained language models have become mainstream, and various methods for learning sentence embeddings have been proposed~\cite{SBERT,SimCSE}.&#10;However, as these methods represent a sentence as a point in a vector space and primarily use symmetric measures such as the cosine similarity to measure the similarity between sentences, they cannot capture asymmetric relationships between two sentences, such as entailment and hierarchical relations.&#10;&#10;In this paper, we propose GaussCSE, a Gaussian-distribution-based contrastive sentence embedding to handle such asymmetric relationships between sentences by extending Gaussian embedding for words \cite{Gemb}.&#10;Figure~\ref{fig:emb_space} shows examples of sentence representations obtained by a previous method and by GaussCSE.&#10;Whereas the previous method represents a sentence as a point, GaussCSE represents a sentence as a region in the embedding space, and when two sentences have an entailment relation, the embedding of the entailing sentence contains the embedding of the entailed one.&#10;In these examples, S1 entails S2, but with previous methods, it is difficult to determine the entailment relation only from their embeddings.&#10;In contrast, by taking into account the variances of the distributions, GaussCSE can capture the asymmetric relationship where S1 entails S2 but S2 does not entail S1, as well as the fact that S3 is not in the entailment relationship with either S1 or S2.&#10;&#10;To validate the usefulness of GaussCSE, we performed comparative experiments on two tasks: the natural language inference (NLI) task, and the task of predicting the entailment direction.&#10;The results demonstrate that GaussCSE can accurately predict the entailment direction while maintaining good performance on the NLI task.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Contrastive Learning, Linguistics, Sentence Embedding, Artificial Intelligence, Mathematics, Natural Language Inference, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2110.07524" label="2110.07524">
        <attvalues>
          <attvalue for="0" value="Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval" />
          <attvalue for="1" value="  Training dense passage representations via contrastive learning has been&#10;shown effective for Open-Domain Passage Retrieval (ODPR). Existing studies&#10;focus on further optimizing by improving negative sampling strategy or extra&#10;pretraining. However, these studies keep unknown in capturing passage with&#10;internal representation conflicts from improper modeling granularity. This work&#10;thus presents a refined model on the basis of a smaller granularity, contextual&#10;sentences, to alleviate the concerned conflicts. In detail, we introduce an&#10;in-passage negative sampling strategy to encourage a diverse generation of&#10;sentence representations within the same passage. Experiments on three&#10;benchmark datasets verify the efficacy of our method, especially on datasets&#10;where conflicts are severe. Extensive experiments further present good&#10;transferability of our method across datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14560" label="2004.14560">
        <attvalues>
          <attvalue for="0" value="RikiNet: Reading Wikipedia Pages for Natural Question Answering" />
          <attvalue for="1" value="  Reading long documents to answer open-domain questions remains challenging in&#10;natural language understanding. In this paper, we introduce a new model, called&#10;RikiNet, which reads Wikipedia pages for natural question answering. RikiNet&#10;contains a dynamic paragraph dual-attention reader and a multi-level cascaded&#10;answer predictor. The reader dynamically represents the document and question&#10;by utilizing a set of complementary attention mechanisms. The representations&#10;are then fed into the predictor to obtain the span of the short answer, the&#10;paragraph of the long answer, and the answer type in a cascaded manner. On the&#10;Natural Questions (NQ) dataset, a single RikiNet achieves 74.3 F1 and 57.9 F1&#10;on long-answer and short-answer tasks. To our best knowledge, it is the first&#10;single model that outperforms the single human performance. Furthermore, an&#10;ensemble RikiNet obtains 76.1 F1 and 61.3 F1 on long-answer and short-answer&#10;tasks, achieving the best performance on the official NQ leaderboard&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1412.6623" label="1412.6623">
        <attvalues>
          <attvalue for="0" value="Word Representations via Gaussian Embedding" />
          <attvalue for="1" value="  Current work in lexical distributed representations maps each word to a point&#10;vector in low-dimensional space. Mapping instead to a density provides many&#10;interesting advantages, including better capturing uncertainty about a&#10;representation and its relationships, expressing asymmetries more naturally&#10;than dot product or cosine similarity, and enabling more expressive&#10;parameterization of decision boundaries. This paper advocates for density-based&#10;distributed embeddings and presents a method for learning representations in&#10;the space of Gaussian distributions. We compare performance on various word&#10;embedding benchmarks, investigate the ability of these embeddings to model&#10;entailment and other asymmetric relationships, and explore novel properties of&#10;the representation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.00235" label="2402.00235">
        <attvalues>
          <attvalue for="0" value="Exploring the limits of decoder-only models trained on public speech&#10;  recognition corpora" />
          <attvalue for="1" value="  The emergence of industrial-scale speech recognition (ASR) models such as&#10;Whisper and USM, trained on 1M hours of weakly labelled and 12M hours of audio&#10;only proprietary data respectively, has led to a stronger need for large scale&#10;public ASR corpora and competitive open source pipelines. Unlike the said&#10;models, large language models are typically based on Transformer decoders, and&#10;it remains unclear if decoder-only models trained on public data alone can&#10;deliver competitive performance. In this work, we investigate factors such as&#10;choice of training datasets and modeling components necessary for obtaining the&#10;best performance using public English ASR corpora alone. Our Decoder-Only&#10;Transformer for ASR (DOTA) model comprehensively outperforms the&#10;encoder-decoder open source replication of Whisper (OWSM) on nearly all English&#10;ASR benchmarks and outperforms Whisper large-v3 on 7 out of 15 test sets. We&#10;release our codebase and model checkpoints under permissive license.&#10;" />
          <attvalue for="2" value="&#10;&#10;Attention-based models \cite{vaswani2017attention} have been successful across many areas of machine learning \cite{Jumper2021HighlyAP,videopoet}. In particular, large language models (LLMs) comprising decoder-only Transformers pre-trained on large amounts of unlabelled text have become the standard in natural language processing, exhibiting impressive amounts of linguistic and world knowledge \cite{mixtral}. &#10;&#10;In contrast to LLMs, the best performing ASR models are typically based on Conformers \cite{conformer} trained with a&#10;connectionist temporal classification \cite{ctc} or RNN transducer \cite{rnn-t} objective. This method has been highly successful and is employed in nearly all the best performing ASR models \cite{nemo} such as USM \cite{usm} which also serves as the speech encoder of Gemini v1 \cite{geminiv1}. However, as these methods use a monotonic inductive bias specific to ASR, it is natural to investigate if Transformers trained to autoregressively generate text can deliver competitive performance. This was answered with the introduction of Whisper, a Transformer encoder-decoder model that is highly competitive with the best performing CTC-Conformer and RNN-T-Conformer pipelines on several speech recognition and translation benchmarks across multiple languages \cite{whisper}.&#10;&#10;Whisper (large-v3) is trained on 1M hours of proprietary speech-text data and 4M hours of pseudo-labelled audio, so it remains to be answered whether similar performance could be achieved via public ASR data alone. Unlike Whisper, which is an encoder-decoder model, most LLMs are decoder-only, and it is natural to investigate the performance of decoder-only models. The OWSM model \cite{owsm} is a step towards this direction and comprises a Whisper-style encoder-decoder model trained on a compilation of public multilingual ASR corpora. However, as OWSM models are encoder-decoder Transformers trained using an additional CTC-based loss, it remains to be answered if conventional Transformer decoder training, similar to that for LLMs \cite{Brown2020LanguageMA}, suffices for competitive performance. &#10;&#10;In this work we investigate the performance of Transformer decoders and prefix LMs \cite{raffel2019exploring} as well as the individual utility of public English ASR datasets by combining them into a large 93K hour paired speech-text corpus. Unlike OWSM, we train decoder-only models solely using cross-entropy loss. We train models varying over a wide range of hyperparameters such as 1) model size, 2) bidirectionality over audio frames, 3) downsampling rate of audio frames, 4) audio augmentation and, 5) the datasets included in the training set. In addition, we also evaluate the performance of the trained models at low audio bitrates using recent neural codecs such as DAC \cite{dac}.&#10;&#10;We find that our best Decoder-Only Transformer for ASR (DOTA) model outperforms Whisper large-v3 on 7 out of 15 test sets (Figure \ref{fig:wer}), and OWSM medium-v3.1 on nearly all test sets, while having fewer than half as many parameters as Whisper. Additionally, our best \model model uses twice the audio frame downsampling rate versus OWSM, making it faster.&#10;&#10;We have open-sourced our codebase and trained models at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Recognition Models, Computer Science, Large Language Models, Linguistics, Engineering, Artificial Intelligence, Open Source Pipelines" />
        </attvalues>
      </node>
      <node id="1909.09577" label="1909.09577">
        <attvalues>
          <attvalue for="0" value="NeMo: a toolkit for building AI applications using Neural Modules" />
          <attvalue for="1" value="  NeMo (Neural Modules) is a Python framework-agnostic toolkit for creating AI&#10;applications through re-usability, abstraction, and composition. NeMo is built&#10;around neural modules, conceptual blocks of neural networks that take typed&#10;inputs and produce typed outputs. Such modules typically represent data layers,&#10;encoders, decoders, language models, loss functions, or methods of combining&#10;activations. NeMo makes it easy to combine and re-use these building blocks&#10;while providing a level of semantic correctness checking via its neural type&#10;system. The toolkit comes with extendable collections of pre-built modules for&#10;automatic speech recognition and natural language processing. Furthermore, NeMo&#10;provides built-in support for distributed training and mixed precision on&#10;latest NVIDIA GPUs. NeMo is open-source https://github.com/NVIDIA/NeMo&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.01037" label="2303.01037">
        <attvalues>
          <attvalue for="0" value="Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages" />
          <attvalue for="1" value="  We introduce the Universal Speech Model (USM), a single large model that&#10;performs automatic speech recognition (ASR) across 100+ languages. This is&#10;achieved by pre-training the encoder of the model on a large unlabeled&#10;multilingual dataset of 12 million (M) hours spanning over 300 languages, and&#10;fine-tuning on a smaller labeled dataset. We use multilingual pre-training with&#10;random-projection quantization and speech-text modality matching to achieve&#10;state-of-the-art performance on downstream multilingual ASR and speech-to-text&#10;translation tasks. We also demonstrate that despite using a labeled training&#10;set 1/7-th the size of that used for the Whisper model, our model exhibits&#10;comparable or better performance on both in-domain and out-of-domain speech&#10;recognition tasks across many languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.13876" label="2309.13876">
        <attvalues>
          <attvalue for="0" value="Reproducing Whisper-Style Training Using an Open-Source Toolkit and&#10;  Publicly Available Data" />
          <attvalue for="1" value="  Pre-training speech models on large volumes of data has achieved remarkable&#10;success. OpenAI Whisper is a multilingual multitask model trained on 680k hours&#10;of supervised speech data. It generalizes well to various speech recognition&#10;and translation benchmarks even in a zero-shot setup. However, the full&#10;pipeline for developing such models (from data collection to training) is not&#10;publicly accessible, which makes it difficult for researchers to further&#10;improve its performance and address training-related issues such as efficiency,&#10;robustness, fairness, and bias. This work presents an Open Whisper-style Speech&#10;Model (OWSM), which reproduces Whisper-style training using an open-source&#10;toolkit and publicly available data. OWSM even supports more translation&#10;directions and can be more efficient to train. We will publicly release all&#10;scripts used for data preparation, training, inference, and scoring as well as&#10;pre-trained models and training logs to promote open science.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.01188" label="2209.01188">
        <attvalues>
          <attvalue for="0" value="Petals: Collaborative Inference and Fine-tuning of Large Models" />
          <attvalue for="1" value="  Many NLP tasks benefit from using large language models (LLMs) that often&#10;have more than 100 billion parameters. With the release of BLOOM-176B and&#10;OPT-175B, everyone can download pretrained models of this scale. Still, using&#10;these models requires high-end hardware unavailable to many researchers. In&#10;some cases, LLMs can be used more affordably via RAM offloading or hosted APIs.&#10;However, these techniques have innate limitations: offloading is too slow for&#10;interactive inference, while APIs are not flexible enough for research that&#10;requires access to weights, attention or logits. In this work, we propose&#10;Petals - a system for inference and fine-tuning of large models collaboratively&#10;by joining the resources of multiple parties. We demonstrate that this strategy&#10;outperforms offloading for very large models, running inference of BLOOM-176B&#10;on consumer GPUs with $\approx$ 1 step per second, which is enough for many&#10;interactive LLM applications. Unlike most inference APIs, Petals also natively&#10;exposes hidden states of served models, allowing to train and share custom&#10;model extensions based on efficient fine-tuning methods.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, the NLP community has found that pretrained language models can solve many practical tasks, through either fine-tuning~\cite{gpt} or simple prompting~\cite{gpt3}. Furthermore, performance tends to improve as scale increases~\cite{gpt2, kaplan2020scaling}. Following this trend, modern language models often have hundreds of billions of parameters~\cite{gpt3,gopher,pangua,hyperclova}\nocite{switch,jurrasic,Lepikhin2020GShardSG,glam}. Several research groups released pretrained LLMs with over 100B parameters~\cite{opt,yalm,zeng2020glm}\nocite{gpt,gpt-neox-20b}. Most recently, the BigScience project has released BLOOM, a 176 billion parameter model supporting 46 natural and 13 programming languages~\cite{bloom}.&#10;&#10;While the public availability of 100B+ parameter models makes them easier to access, they remain difficult to use for the majority of researchers and practitioners due to memory and computational costs. For instance, OPT-175B and BLOOM-176B need over 350 GB accelerator memory for inference and significantly more for fine-tuning. As a result, these LLMs usually require multiple high-end GPUs or multi-node clusters\nocite{megatron2} to be run. Both of these options are extremely expensive, which limits research and potential applications of LLMs.&#10;&#10;Several recent works aim to democratize LLMs&#10;by ``offloading'' model parameters to slower but cheaper memory (RAM or SSD), then running them on the accelerator layer by layer~\cite{l2l,zerooffload}\nocite{accelerate}.&#10;This method allows running LLMs with a single low-end accelerator by loading parameters from RAM justin-time for each forward pass.&#10;Offloading can be efficient for processing many tokens in parallel, but it has inherently high latency: for example, generating one token at a time with BLOOM-176B takes at least 5.5~seconds for the fastest RAM offloading setup and 22~seconds for the fastest SSD offloading. In addition, many computers do not have enough RAM to offload 175B parameters.&#10;&#10;Another way to make LLMs more accessible is through public inference APIs, where one party hosts the model and lets others query it over the Internet~\cite{openai-api,jurrasic,forefront}. Since most of the engineering work is done by the API owner, this is a relatively user-friendly option.&#10;However, APIs are often not flexible enough for research use: there is no way to change the model control flow or access internal states. On top of that, current API pricing can make some research projects prohibitively expensive~\cite{tfew}.&#10;&#10;In this work, we explore an alternative strategy inspired by crowdsourced distributed training of neural networks from scratch~\cite{hivemind_dmoe}. We introduce Petals, a platform that allows multiple users to collaborate and perform inference and fine-tuning of large language models over the Internet.&#10;Each participant runs a server, a client or both. A server hosts a subset of model layers (typically, Transformer blocks) and handles requests from clients.&#10;A client can form a chain of pipeline-parallel consecutive servers to run the inference of the entire model (Section~\ref{sect:design_inference}).&#10;Aside from inference, participants can fine-tune the model through parameter-efficient training methods like adapters \cite{houlsby2019parameter} or prompt tuning \cite{ptune-lester} or by training entire layers (Section~\ref{sect:design_training}). Once trained, submodules can be shared on a model hub (Section~\ref{sect:design_ecosystem}), where others can use them for inference or further training.&#10;We demonstrate that existing 100B+ models can run efficiently in this setting with the help of several optimizations: dynamic quantization, prioritizing low-latency connections, and load balancing between servers (Section~\ref{sect:internals}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Large Language Models, Efficient Fine-Tuning Methods, Artificial Intelligence, Mathematics, Collaborative Inference Systems, Distributed Computing" />
        </attvalues>
      </node>
      <node id="2109.04650" label="2109.04650">
        <attvalues>
          <attvalue for="0" value="What Changes Can Large-scale Language Models Bring? Intensive Study on&#10;  HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers" />
          <attvalue for="1" value="  GPT-3 shows remarkable in-context learning ability of large-scale language&#10;models (LMs) trained on hundreds of billion scale data. Here we address some&#10;remaining issues less reported by the GPT-3 paper, such as a non-English LM,&#10;the performances of different sized models, and the effect of recently&#10;introduced prompt optimization on in-context learning. To achieve this, we&#10;introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric&#10;corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA&#10;with our training configuration shows state-of-the-art in-context zero-shot and&#10;few-shot learning performances on various downstream tasks in Korean. Also, we&#10;show the performance benefits of prompt-based learning and demonstrate how it&#10;can be integrated into the prompt engineering pipeline. Then we discuss the&#10;possibility of materializing the No Code AI paradigm by providing AI&#10;prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio,&#10;an interactive prompt engineering interface. Lastly, we demonstrate the&#10;potential of our methods with three successful in-house applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.06840" label="2101.06840">
        <attvalues>
          <attvalue for="0" value="ZeRO-Offload: Democratizing Billion-Scale Model Training" />
          <attvalue for="1" value="  Large-scale model training has been a playing ground for a limited few&#10;requiring complex model refactoring and access to prohibitively expensive GPU&#10;clusters. ZeRO-Offload changes the large model training landscape by making&#10;large model training accessible to nearly everyone. It can train models with&#10;over 13 billion parameters on a single GPU, a 10x increase in size compared to&#10;popular framework such as PyTorch, and it does so without requiring any model&#10;change from the data scientists or sacrificing computational efficiency.&#10;ZeRO-Offload enables large model training by offloading data and compute to&#10;CPU. To preserve compute efficiency, it is designed to minimize the data&#10;movement to/from GPU, and reduce CPU compute time while maximizing memory&#10;savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single&#10;NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone&#10;for a 1.4B parameter model, the largest that can be trained without running out&#10;of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when&#10;available, offering near linear speedup on up to 128 GPUs. Additionally, it can&#10;work together with model parallelism to train models with over 70 billion&#10;parameters on a single DGX-2 box, a 4.5x increase in model size compared to&#10;using model parallelism alone. By combining compute and memory efficiency with&#10;ease-of-use, ZeRO-Offload democratizes large-scale model training making it&#10;accessible to even data scientists with access to just a single GPU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.04013" label="2002.04013">
        <attvalues>
          <attvalue for="0" value="Towards Crowdsourced Training of Large Neural Networks using&#10;  Decentralized Mixture-of-Experts" />
          <attvalue for="1" value="  Many recent breakthroughs in deep learning were achieved by training&#10;increasingly larger models on massive datasets. However, training such models&#10;can be prohibitively expensive. For instance, the cluster used to train GPT-3&#10;costs over \$250 million. As a result, most researchers cannot afford to train&#10;state of the art models and contribute to their development. Hypothetically, a&#10;researcher could crowdsource the training of large neural networks with&#10;thousands of regular PCs provided by volunteers. The raw computing power of a&#10;hundred thousand \$2500 desktops dwarfs that of a \$250M server pod, but one&#10;cannot utilize that power efficiently with conventional distributed training&#10;methods. In this work, we propose Learning@home: a novel neural network&#10;training paradigm designed to handle large amounts of poorly connected&#10;participants. We analyze the performance, reliability, and architectural&#10;constraints of this paradigm and compare it against existing distributed&#10;training techniques.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.02317" label="2209.02317">
        <attvalues>
          <attvalue for="0" value="Layer or Representation Space: What makes BERT-based Evaluation Metrics&#10;  Robust?" />
          <attvalue for="1" value="  The evaluation of recent embedding-based evaluation metrics for text&#10;generation is primarily based on measuring their correlation with human&#10;evaluations on standard benchmarks. However, these benchmarks are mostly from&#10;similar domains to those used for pretraining word embeddings. This raises&#10;concerns about the (lack of) generalization of embedding-based metrics to new&#10;and noisy domains that contain a different vocabulary than the pretraining&#10;data. In this paper, we examine the robustness of BERTScore, one of the most&#10;popular embedding-based metrics for text generation. We show that (a) an&#10;embedding-based metric that has the highest correlation with human evaluations&#10;on a standard benchmark can have the lowest correlation if the amount of input&#10;noise or unknown tokens increases, (b) taking embeddings from the first layer&#10;of pretrained models improves the robustness of all metrics, and (c) the&#10;highest robustness is achieved when using character-level embeddings, instead&#10;of token-based embeddings, from the first layer of the pretrained model.&#10;" />
          <attvalue for="2" value="&#10;Evaluating the quality of generated outputs by Natural Language Generation (NLG) models is a challenging and open problem. &#10;Human judgments can directly assess the quality of generated texts \cite{popovic-2020-informative,escribe-2019-human}. However, human evaluation, either with experts or crowdsourcing, is expensive and time-consuming. Therefore, automatic evaluation metrics, which are fast and cheap, are commonly used alternatives for the rapid development of text generation systems \cite{van-der-lee-etal-2019-best}. Traditional metrics such as BLEU \cite{papineni-etal-2002-bleu}, METEOR \cite{banerjee-lavie-2005-meteor}, and ROUGE \cite{lin-2004-rouge} measure $n$-gram overlap between generated and reference texts. While these metrics are easy to use, they cannot correctly assess generated texts that contain novel words or a rephrasing of the reference text.&#10;&#10;Recent metrics like BERTScore \cite{bert-score}, MoverScore \cite{zhao-etal-2019-moverscore}, COMET \cite{rei-etal-2020-comet}, BARTScore \cite{yuan2021bartscore}, and BLEURT \cite{sellam-etal-2020-bleurt} adapt pretrained contextualized word embeddings to tackle this issue. &#10;These novel metrics have shown higher correlations with human judgments on various tasks and datasets \cite{ma-etal-2019-results,mathur-etal-2020-results}. However, the correlations are measured on standard benchmarks containing text domains similar to those used for pretraining the embeddings themselves. As a result, it is unclear how reliable these metrics are on domains and datasets containing words outside the vocabulary of the pretraining data.&#10;&#10;The goal of this paper is to investigate the robustness of embedding-based evaluation metrics on new and noisy domains that contain a higher ratio of unknown tokens compared to standard text domains.&#10;We examine the robustness of BERTScore, one of the most popular recent metrics for text generation.&#10;In order to perform a systematic evaluation on the robustness of BERTScore with regard to the ratio of unknown tokens, we use character-based adversarial attacks \cite{eger2020hero} that introduce a controlled ratio of new unknown tokens to the input texts. Our contributions are: &#10;&#10;\begin{itemize}[topsep=5pt,itemsep=0pt,leftmargin=*]&#10; \item We investigate whether the use of character-based embeddings instead of token-based embeddings improves the robustness of embedding-based generation metrics. Our results show that the evaluations based on character-level embeddings are more robust.&#10; \item We examine the impact of the hidden layer used for computing the embeddings in BERTScore. We show that the choice of hidden layer affects the robustness of the evaluation metric.&#10; \item We show that by using character-level embeddings from the first layer, we achieve the highest robustness, i.e., similar correlation with human evaluations for different ratios of unknown tokens.&#10;\end{itemize}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Embedding-Based Metrics, Text Generation Evaluation, Computational Linguistics, Artificial Intelligence, Natural Language Processing, Statistics, Robustness to Noise" />
        </attvalues>
      </node>
      <node id="1909.02622" label="1909.02622">
        <attvalues>
          <attvalue for="0" value="MoverScore: Text Generation Evaluating with Contextualized Embeddings&#10;  and Earth Mover Distance" />
          <attvalue for="1" value="  A robust evaluation metric has a profound impact on the development of text&#10;generation systems. A desirable metric compares system output against&#10;references based on their semantics rather than surface forms. In this paper we&#10;investigate strategies to encode system and reference texts to devise a metric&#10;that shows a high correlation with human judgment of text quality. We validate&#10;our new metric, namely MoverScore, on a number of text generation tasks&#10;including summarization, machine translation, image captioning, and&#10;data-to-text generation, where the outputs are produced by a variety of neural&#10;and non-neural systems. Our findings suggest that metrics combining&#10;contextualized representations with a distance measure perform the best. Such&#10;metrics also demonstrate strong generalization capability across tasks. For&#10;ease-of-use we make our metrics available as web service.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.09025" label="2009.09025">
        <attvalues>
          <attvalue for="0" value="COMET: A Neural Framework for MT Evaluation" />
          <attvalue for="1" value="  We present COMET, a neural framework for training multilingual machine&#10;translation evaluation models which obtains new state-of-the-art levels of&#10;correlation with human judgements. Our framework leverages recent breakthroughs&#10;in cross-lingual pretrained language modeling resulting in highly multilingual&#10;and adaptable MT evaluation models that exploit information from both the&#10;source input and a target-language reference translation in order to more&#10;accurately predict MT quality. To showcase our framework, we train three models&#10;with different types of human judgements: Direct Assessments, Human-mediated&#10;Translation Edit Rate and Multidimensional Quality Metrics. Our models achieve&#10;new state-of-the-art performance on the WMT 2019 Metrics shared task and&#10;demonstrate robustness to high-performing systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.11520" label="2106.11520">
        <attvalues>
          <attvalue for="0" value="BARTScore: Evaluating Generated Text as Text Generation" />
          <attvalue for="1" value="  A wide variety of NLP applications, such as machine translation,&#10;summarization, and dialog, involve text generation. One major challenge for&#10;these applications is how to evaluate whether such generated texts are actually&#10;fluent, accurate, or effective. In this work, we conceptualize the evaluation&#10;of generated text as a text generation problem, modeled using pre-trained&#10;sequence-to-sequence models. The general idea is that models trained to convert&#10;the generated text to/from a reference output or the source text will achieve&#10;higher scores when the generated text is better. We operationalize this idea&#10;using BART, an encoder-decoder based pre-trained model, and propose a metric&#10;BARTScore with a number of variants that can be flexibly applied in an&#10;unsupervised fashion to evaluation of text from different perspectives (e.g.&#10;informativeness, fluency, or factuality). BARTScore is conceptually simple and&#10;empirically effective. It can outperform existing top-scoring metrics in 16 of&#10;22 test settings, covering evaluation of 16 datasets (e.g., machine&#10;translation, text summarization) and 7 different perspectives (e.g.,&#10;informativeness, factuality). Code to calculate BARTScore is available at&#10;https://github.com/neulab/BARTScore, and we have released an interactive&#10;leaderboard for meta-evaluation at&#10;http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard&#10;platform, which allows us to interactively understand the strengths,&#10;weaknesses, and complementarity of each metric.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05648" label="2010.05648">
        <attvalues>
          <attvalue for="0" value="From Hero to Z\'eroe: A Benchmark of Low-Level Adversarial Attacks" />
          <attvalue for="1" value="  Adversarial attacks are label-preserving modifications to inputs of machine&#10;learning classifiers designed to fool machines but not humans. Natural Language&#10;Processing (NLP) has mostly focused on high-level attack scenarios such as&#10;paraphrasing input texts. We argue that these are less realistic in typical&#10;application scenarios such as in social media, and instead focus on low-level&#10;attacks on the character-level. Guided by human cognitive abilities and human&#10;robustness, we propose the first large-scale catalogue and benchmark of&#10;low-level adversarial attacks, which we dub Z\'eroe, encompassing nine&#10;different attack modes including visual and phonetic adversaries. We show that&#10;RoBERTa, NLP's current workhorse, fails on our attacks. Our dataset provides a&#10;benchmark for testing robustness of future more human-like NLP models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06598" label="2106.06598">
        <attvalues>
          <attvalue for="0" value="Leveraging Pre-trained Language Model for Speech Sentiment Analysis" />
          <attvalue for="1" value="  In this paper, we explore the use of pre-trained language models to learn&#10;sentiment information of written texts for speech sentiment analysis. First, we&#10;investigate how useful a pre-trained language model would be in a 2-step&#10;pipeline approach employing Automatic Speech Recognition (ASR) and&#10;transcripts-based sentiment analysis separately. Second, we propose a pseudo&#10;label-based semi-supervised training strategy using a language model on an&#10;end-to-end speech sentiment approach to take advantage of a large, but&#10;unlabeled speech dataset for training. Although spoken and written texts have&#10;different linguistic characteristics, they can complement each other in&#10;understanding sentiment. Therefore, the proposed system can not only model&#10;acoustic characteristics to bear sentiment-specific information in speech&#10;signals, but learn latent information to carry sentiments in the text&#10;representation. In these experiments, we demonstrate the proposed approaches&#10;improve F1 scores consistently compared to systems without a language model.&#10;Moreover, we also show that the proposed framework can reduce 65% of human&#10;supervision by leveraging a large amount of data without human sentiment&#10;annotation and boost performance in a low-resource condition where the human&#10;sentiment annotation is not available enough.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Learning good representation from speech signals is the key to a speech sentiment/emotion analysis task. &#10;A recent study suggests to use a pre-trained ASR encoder~\cite{lu2020speech} to prevent overfitting, and it showed promising results by surpassing the traditional audio + text multimodal systems~\cite{kim2019dnn, siriwardhana2020jointly,cho2018deep}.&#10;Without the pre-trained ASR encoder, the model tends to overfit to the training data and the same model working on emotion recognition gives mediocre results on the sentiment analysis task~\cite{lu2020speech}.&#10;&#10;Similarly, in the study of Spoken Language Understanding (SLU), pre-training approaches were proposed in combination with ASR~\cite{kuo2020end,chen2018spoken,haghani2018audio} or acoustic classification modules~\cite{lugosch2019speech}, using ground truth (GT) text or ASR transcripts to improve SLU performance under limited resources.&#10;&#10;The aforementioned pre-training approaches are based on the assumption that if a model is pre-trained to recognize words or phonemes, the fine-tuning result of downstream tasks will be improved.&#10;Our approach is also based on the same assumption, but we propose the use of powerful pre-trained LMs to transfer more abstract knowledge from the written text-domain to speech sentiment analysis.&#10;Specifically, we leverage pre-trained BERT models to extract robust embedding from text tokens for the 2-step pipeline, and to generate pseudo labels for semi-supervised training a model for the E2E speech sentiment analysis system.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Speech Sentiment Analysis, Language Model Applications, Artificial Intelligence, Speech Recognition, Natural Language Processing, Semi-Supervised Learning" />
        </attvalues>
      </node>
      <node id="1911.11931" label="1911.11931">
        <attvalues>
          <attvalue for="0" value="Evaluating Commonsense in Pre-trained Language Models" />
          <attvalue for="1" value="  Contextualized representations trained over large raw text data have given&#10;remarkable improvements for NLP tasks including question answering and reading&#10;comprehension. There have been works showing that syntactic, semantic and word&#10;sense knowledge are contained in such representations, which explains why they&#10;benefit such tasks. However, relatively little work has been done investigating&#10;commonsense knowledge contained in contextualized representations, which is&#10;crucial for human question answering and reading comprehension. We study the&#10;commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven&#10;challenging benchmarks, finding that language modeling and its variants are&#10;effective objectives for promoting models' commonsense ability while&#10;bi-directional context and larger training set are bonuses. We additionally&#10;find that current models do poorly on tasks require more necessary inference&#10;steps. Finally, we test the robustness of models by making dual test cases,&#10;which are correlated so that the correct prediction of one sample should lead&#10;to correct prediction of the other. Interestingly, the models show confusion on&#10;these test cases, which suggests that they learn commonsense at the surface&#10;rather than the deep level. We release a test set, named CATs publicly, for&#10;future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.02925" label="2002.02925">
        <attvalues>
          <attvalue for="0" value="BERT-of-Theseus: Compressing BERT by Progressive Module Replacing" />
          <attvalue for="1" value="  In this paper, we propose a novel model compression approach to effectively&#10;compress BERT by progressive module replacing. Our approach first divides the&#10;original BERT into several modules and builds their compact substitutes. Then,&#10;we randomly replace the original modules with their substitutes to train the&#10;compact modules to mimic the behavior of the original modules. We progressively&#10;increase the probability of replacement through the training. In this way, our&#10;approach brings a deeper level of interaction between the original and compact&#10;models. Compared to the previous knowledge distillation approaches for BERT&#10;compression, our approach does not introduce any additional loss function. Our&#10;approach outperforms existing knowledge distillation approaches on GLUE&#10;benchmark, showing a new perspective of model compression.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.03228" label="2109.03228">
        <attvalues>
          <attvalue for="0" value="Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT&#10;  Compression" />
          <attvalue for="1" value="  Recent studies on compression of pretrained language models (e.g., BERT)&#10;usually use preserved accuracy as the metric for evaluation. In this paper, we&#10;propose two new metrics, label loyalty and probability loyalty that measure how&#10;closely a compressed model (i.e., student) mimics the original model (i.e.,&#10;teacher). We also explore the effect of compression with regard to robustness&#10;under adversarial attacks. We benchmark quantization, pruning, knowledge&#10;distillation and progressive module replacing with loyalty and robustness. By&#10;combining multiple compression techniques, we provide a practical strategy to&#10;achieve better accuracy, loyalty and robustness.&#10;" />
          <attvalue for="2" value="&#10;Recently, many large pretrained language models (PLMs, \cite{bert,roberta,xlnet,megatron,t5}) have been proposed for &#10;a variety of&#10;Natural Language Processing (NLP) tasks. However, as pointed out in recent studies~\cite{strubell2019energy,greenai,parrot}, these models suffer from computational inefficiency and high ecological cost. Many attempts have been made to address this problem, including quantization~\cite{q8bert,qbert}, pruning~\cite{headprune,mvp}, knowledge distillation (KD)~\cite{distilbert,bertpkd,mobilebert,pd,tinybert,minilm,zhou2021meta} and progressive module replacing~\cite{bot}. &#10;&#10;BERT~\cite{bert} is a representative PLM.&#10;Many works compressing BERT use preserved accuracy with computational complexity (e.g., speed-up ratio, FLOPS, number of parameters) as metrics to evaluate &#10;compression. &#10;This evaluation scheme is far from perfect: (1) Preserved accuracy cannot reflect how alike the teacher and student models behave. This can be &#10;problematic when applying &#10;compression techniques &#10;in&#10;production (to be detailed in Section \ref{sec:loyalty}). (2) Using preserved accuracy to evaluate models compressed with more data or data augmentation~\cite{tinybert} can be misleading, since one cannot tell whether the improvement should be attributed to the innovation of the compression technique or addition of data. (3) Model robustness, which is critical for production, is often missing from evaluation, leaving a possible safety risk.&#10;&#10;As illustrated in Figure \ref{fig:loyalty}, to measure the resemblance between the student and teacher models, we propose label loyalty and probability loyalty &#10;to&#10;target different but &#10;important aspects. &#10;We also explore the robustness of the compressed models by conducting black-box adversarial attacks. We apply representative BERT compression methods of different types to the same teacher model and benchmark their performance in terms of accuracy, speed, loyalty and robustness. We find that methods with a knowledge distillation loss perform well on loyalty and&#10;that&#10;post-training quantization can drastically improve &#10;robustness against &#10;adversarial attacks. We use the conclusions drawn from these experiments to combine multiple techniques together and achieve significant improvement in terms of accuracy, loyalty and robustness.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.07105" label="2202.07105">
        <attvalues>
          <attvalue for="0" value="A Survey on Model Compression and Acceleration for Pretrained Language&#10;  Models" />
          <attvalue for="1" value="  Despite achieving state-of-the-art performance on many NLP tasks, the high&#10;energy cost and long inference delay prevent Transformer-based pretrained&#10;language models (PLMs) from seeing broader adoption including for edge and&#10;mobile computing. Efficient NLP research aims to comprehensively consider&#10;computation, time and carbon emission for the entire life-cycle of NLP,&#10;including data preparation, model training and inference. In this survey, we&#10;focus on the inference stage and review the current state of model compression&#10;and acceleration for pretrained language models, including benchmarks, metrics&#10;and methodology.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.18041" label="2402.18041">
        <attvalues>
          <attvalue for="0" value="Datasets for Large Language Models: A Comprehensive Survey" />
          <attvalue for="1" value="  This paper embarks on an exploration into the Large Language Model (LLM)&#10;datasets, which play a crucial role in the remarkable advancements of LLMs. The&#10;datasets serve as the foundational infrastructure analogous to a root system&#10;that sustains and nurtures the development of LLMs. Consequently, examination&#10;of these datasets emerges as a critical topic in research. In order to address&#10;the current lack of a comprehensive overview and thorough analysis of LLM&#10;datasets, and to gain insights into their current status and future trends,&#10;this survey consolidates and categorizes the fundamental aspects of LLM&#10;datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction&#10;Fine-tuning Datasets; (3) Preference Datasets; (4) Evaluation Datasets; (5)&#10;Traditional Natural Language Processing (NLP) Datasets. The survey sheds light&#10;on the prevailing challenges and points out potential avenues for future&#10;investigation. Additionally, a comprehensive review of the existing available&#10;dataset resources is also provided, including statistics from 444 datasets,&#10;covering 8 language categories and spanning 32 domains. Information from 20&#10;dimensions is incorporated into the dataset statistics. The total data size&#10;surveyed surpasses 774.5 TB for pre-training corpora and 700M instances for&#10;other datasets. We aim to present the entire landscape of LLM text datasets,&#10;serving as a comprehensive reference for researchers in this field and&#10;contributing to future studies. Related resources are available at:&#10;https://github.com/lmmlzn/Awesome-LLMs-Datasets.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the release of ChatGPT \cite{bib1}, in just a few months, Large Language Models (LLMs) have attracted increasing research attention and become a hot research field. Various LLMs have been successively open-sourced, with parameter sizes ranging from several billion to over a hundred billion. Examples include the LLaMA \cite{bib4,bib402}, Phi \cite{bib34,bib403,bib404}, ChatGLM \cite{bib3,bib2}, QWen \cite{bib247}, Baichuan \cite{bib5}, and so on. A considerable amount of work involves fine-tuning on base models, resulting in well-performing general conversational models or domain-specific models. The widespread adoption of Reinforcement Learning from Human Feedback (RLHF) and the refinement of LLM evaluations further optimize the performance of LLMs. The immense potential demonstrated by LLMs can be attributed, in part, to the datasets used for training and testing. As the saying goes, “You can’t make a silk purse out of a sow’s ear.” Without high-quality datasets as the foundation, it is challenging to grow the tree of LLMs with flourishing branches and leaves. Therefore, the construction and analysis of LLM datasets is an area worthy of attention.&#10;&#10;The development of text datasets has undergone several stages, from earlier Natural Language Processing (NLP) task datasets to the current era of LLM datasets. In the 1960s to 1980s, the early stages of NLP primarily focused on fundamental tasks such as semantic analysis and machine translation. The dataset scale was relatively small and typically manually annotated. Later, the Message Understanding Conference (MUC) \cite{bib6} began in 1987, focusing on datasets for tasks such as information extraction and Relation Extraction (RE). After 2000, the NLP field continued to emphasize research on traditional tasks and linguistic structures, while also turning attention to emerging areas such as dialogue systems \cite{bib405,bib407,bib48,bib406}. With the rise of deep learning, NLP datasets evolved towards larger scales, greater complexity, more diversity, and increased challenges. Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged. By the end of 2022, LLMs pushed datasets to a new peak, realizing a shift from a “task-centric construction” to a “construction centered around tasks and stages” in dataset development. LLM datasets are not only categorized based on tasks but also have associations with different stages of LLMs. From the initial pre-training stage to the final evaluation stage, we categorized LLM datasets into four types: pre-training corpora, instruction fine-tuning datasets, preference datasets, and evaluation datasets. The composition and quality of these datasets profoundly influence the performance of LLMs.&#10;&#10;The current explosion in LLM datasets poses challenges for research. On the one hand, it often leads to situations where it is difficult to know where to start when trying to understand and learn about the datasets. On the other hand, there is a lack of systematic organization regarding the differences in types, domain orientations, real-world scenarios, etc., among various datasets. In order to reduce the learning curve, promote dataset research and technological innovation, broaden public awareness, we conduct a survey of LLM datasets. The objective is to provide researchers with a comprehensive and insightful perspective, facilitating a better understanding of the distribution and role of LLM datasets, thereby advancing the collective knowledge and application of LLMs.&#10;&#10;This paper summarizes existing representative datasets across five dimensions: pre-training corpora, instruction fine-tuning datasets, preference datasets, evaluation datasets, and traditional NLP datasets. Moreover, it presents new insights and ideas, discusses current bottlenecks, and explores future development trends. We also provide a comprehensive review of publicly available dataset related resources. It includes statistics from 444 datasets across 8 language categories spanning 32 different domains, covering information from 20 dimensions. The total data size surveyed exceeds 774.5 TB for pre-training corpora and over 700M instances for other datasets. Due to space constraints, this survey only discusses pure text LLM datasets and does not cover multimodal datasets.&#10;&#10;To the best of our knowledge, this is the first survey focused on LLM datasets, presenting the entire landscape. The timeline of LLM datasets is shown in Figure~\ref{fig2}. Prior to this, several LLM-related surveys, such as \cite{bib7} and \cite{bib410}, analyze the latest developments in LLMs but lack detailed descriptions and summaries of datasets. \cite{bib8} summarizes the instruction fine-tuning stage of LLMs. \cite{bib9} and \cite{bib10} summarize the evaluation stage. However, these surveys only concentrate on a part of the LLM datasets, and dataset-related information is not the central focus. In contrast to the aforementioned surveys, our paper places emphasis on LLM datasets, aiming to provide a more detailed and exhaustive survey in this specific domain.&#10;&#10;The overall organizational structure is illustrated in Figure~\ref{fig1}. The remainder of this paper is organized as follows. Section~\ref{sec2} summarizes general pre-training corpora categorized by data types and domain-specific pre-training corpora categorized by domains. It also outlines the preprocessing steps and methods for pre-training data. Section~\ref{sec3} summarizes general instruction fine-tuning datasets categorized by construction methods and domain-specific instruction fine-tuning datasets categorized by domains. 15 instruction categories are provided. Section~\ref{sec4} summarizes preference datasets categorized by preference evaluation methods. Section~\ref{sec5} summarizes evaluation datasets categorized by evaluation domains and synthesizes different evaluation methods. Section~\ref{sec6} summarizes traditional NLP datasets categorized by tasks. Section~\ref{sec7} briefly identifies challenges encountered within the datasets and anticipates future research directions. Section~\ref{sec8} concludes this paper. Detailed descriptions of the datasets can be found in Appendices~\ref{secA} through~\ref{secE}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Dataset Analysis, Computer Science, Large Language Models, Linguistics, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2306.11644" label="2306.11644">
        <attvalues>
          <attvalue for="0" value="Textbooks Are All You Need" />
          <attvalue for="1" value="  We introduce phi-1, a new large language model for code, with significantly&#10;smaller size than competing models: phi-1 is a Transformer-based model with&#10;1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook&#10;quality&quot; data from the web (6B tokens) and synthetically generated textbooks&#10;and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains&#10;pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays&#10;surprising emergent properties compared to phi-1-base, our model before our&#10;finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller&#10;model with 350M parameters trained with the same pipeline as phi-1 that still&#10;achieves 45% on HumanEval.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.05463" label="2309.05463">
        <attvalues>
          <attvalue for="0" value="Textbooks Are All You Need II: phi-1.5 technical report" />
          <attvalue for="1" value="  We continue the investigation into the power of smaller Transformer-based&#10;language models as initiated by \textbf{TinyStories} -- a 10 million parameter&#10;model that can produce coherent English -- and the follow-up work on&#10;\textbf{phi-1}, a 1.3 billion parameter model with Python coding performance&#10;close to the state-of-the-art. The latter work proposed to use existing Large&#10;Language Models (LLMs) to generate ``textbook quality&quot; data as a way to enhance&#10;the learning process compared to traditional web data. We follow the&#10;``Textbooks Are All You Need&quot; approach, focusing this time on common sense&#10;reasoning in natural language, and create a new 1.3 billion parameter model&#10;named \textbf{phi-1.5}, with performance on natural language tasks comparable&#10;to models 5x larger, and surpassing most non-frontier LLMs on more complex&#10;reasoning tasks such as grade-school mathematics and basic coding. More&#10;generally, \textbf{phi-1.5} exhibits many of the traits of much larger LLMs,&#10;both good -- such as the ability to ``think step by step&quot; or perform some&#10;rudimentary in-context learning -- and bad, including hallucinations and the&#10;potential for toxic and biased generations -- encouragingly though, we are&#10;seeing improvement on that front thanks to the absence of web data. We&#10;open-source \textbf{phi-1.5} to promote further research on these urgent&#10;topics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.10360" label="2103.10360">
        <attvalues>
          <attvalue for="0" value="GLM: General Language Model Pretraining with Autoregressive Blank&#10;  Infilling" />
          <attvalue for="1" value="  There have been various types of pretraining architectures including&#10;autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and&#10;encoder-decoder models (e.g., T5). However, none of the pretraining frameworks&#10;performs the best for all tasks of three main categories including natural&#10;language understanding (NLU), unconditional generation, and conditional&#10;generation. We propose a General Language Model (GLM) based on autoregressive&#10;blank infilling to address this challenge. GLM improves blank filling&#10;pretraining by adding 2D positional encodings and allowing an arbitrary order&#10;to predict spans, which results in performance gains over BERT and T5 on NLU&#10;tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying&#10;the number and lengths of blanks. On a wide range of tasks across NLU,&#10;conditional and unconditional generation, GLM outperforms BERT, T5, and GPT&#10;given the same model sizes and data, and achieves the best performance from a&#10;single pretrained model with 1.25x parameters of BERT Large , demonstrating its&#10;generalizability to different downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.10305" label="2309.10305">
        <attvalues>
          <attvalue for="0" value="Baichuan 2: Open Large-scale Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated remarkable performance on a&#10;variety of natural language tasks based on just a few examples of natural&#10;language instructions, reducing the need for extensive feature engineering.&#10;However, most powerful LLMs are closed-source or limited in their capability&#10;for languages other than English. In this technical report, we present Baichuan&#10;2, a series of large-scale multilingual language models containing 7 billion&#10;and 13 billion parameters, trained from scratch, on 2.6 trillion tokens.&#10;Baichuan 2 matches or outperforms other open-source models of similar size on&#10;public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan&#10;2 excels in vertical domains such as medicine and law. We will release all&#10;pre-training model checkpoints to benefit the research community in better&#10;understanding the training dynamics of Baichuan 2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.00536" label="1911.00536">
        <attvalues>
          <attvalue for="0" value="DialoGPT: Large-Scale Generative Pre-training for Conversational&#10;  Response Generation" />
          <attvalue for="1" value="  We present a large, tunable neural conversational response generation model,&#10;DialoGPT (dialogue generative pre-trained transformer). Trained on 147M&#10;conversation-like exchanges extracted from Reddit comment chains over a period&#10;spanning from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch&#10;transformer to attain a performance close to human both in terms of automatic&#10;and human evaluation in single-turn dialogue settings. We show that&#10;conversational systems that leverage DialoGPT generate more relevant,&#10;contentful and context-consistent responses than strong baseline systems. The&#10;pre-trained model and training pipeline are publicly released to facilitate&#10;research into neural response generation and the development of more&#10;intelligent open-domain dialogue systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.04813" label="2308.04813">
        <attvalues>
          <attvalue for="0" value="CLEVA: Chinese Language Models EVAluation Platform" />
          <attvalue for="1" value="  With the continuous emergence of Chinese Large Language Models (LLMs), how to&#10;evaluate a model's capabilities has become an increasingly significant issue.&#10;The absence of a comprehensive Chinese benchmark that thoroughly assesses a&#10;model's performance, the unstandardized and incomparable prompting procedure,&#10;and the prevalent risk of contamination pose major challenges in the current&#10;evaluation of Chinese LLMs. We present CLEVA, a user-friendly platform crafted&#10;to holistically evaluate Chinese LLMs. Our platform employs a standardized&#10;workflow to assess LLMs' performance across various dimensions, regularly&#10;updating a competitive leaderboard. To alleviate contamination, CLEVA curates a&#10;significant proportion of new data and develops a sampling strategy that&#10;guarantees a unique subset for each leaderboard round. Empowered by an&#10;easy-to-use interface that requires just a few mouse clicks and a model API,&#10;users can conduct a thorough evaluation with minimal coding. Large-scale&#10;experiments featuring 23 Chinese LLMs have validated CLEVA's efficacy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.07498" label="2107.07498">
        <attvalues>
          <attvalue for="0" value="FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark" />
          <attvalue for="1" value="  Pretrained Language Models (PLMs) have achieved tremendous success in natural&#10;language understanding tasks. While different learning schemes -- fine-tuning,&#10;zero-shot, and few-shot learning -- have been widely explored and compared for&#10;languages such as English, there is comparatively little work in Chinese to&#10;fairly and comprehensively evaluate and compare these methods and thus hinders&#10;cumulative progress. In this paper, we introduce the Chinese Few-shot Learning&#10;Evaluation Benchmark (FewCLUE), the first comprehensive few-shot evaluation&#10;benchmark in Chinese. It includes nine tasks, ranging from single-sentence and&#10;sentence-pair classification tasks to machine reading comprehension tasks. We&#10;systematically evaluate five state-of-the-art (SOTA) few-shot learning methods&#10;(including PET, ADAPET, LM-BFF, P-tuning and EFL), and compare their&#10;performance with fine-tuning and zero-shot learning schemes on the newly&#10;constructed FewCLUE benchmark. Experimental results reveal that: 1) The effect&#10;of different few-shot learning methods is sensitive to the pre-trained model to&#10;which the methods are applied; 2) PET and P-tuning achieve the best overall&#10;performance with RoBERTa and ERNIE respectively. Our benchmark is used in the&#10;few-shot learning contest of NLPCC 2021. In addition, we provide a&#10;user-friendly toolkit, as well as an online leaderboard to help facilitate&#10;further progress on Chinese few-shot learning. We provide a baseline&#10;performance on different learning methods, a reference for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2207.04672" label="2207.04672">
        <attvalues>
          <attvalue for="0" value="No Language Left Behind: Scaling Human-Centered Machine Translation" />
          <attvalue for="1" value="  Driven by the goal of eradicating language barriers on a global scale,&#10;machine translation has solidified itself as a key focus of artificial&#10;intelligence research today. However, such efforts have coalesced around a&#10;small subset of languages, leaving behind the vast majority of mostly&#10;low-resource languages. What does it take to break the 200 language barrier&#10;while ensuring safe, high quality results, all while keeping ethical&#10;considerations in mind? In No Language Left Behind, we took on this challenge&#10;by first contextualizing the need for low-resource language translation support&#10;through exploratory interviews with native speakers. Then, we created datasets&#10;and models aimed at narrowing the performance gap between low and high-resource&#10;languages. More specifically, we developed a conditional compute model based on&#10;Sparsely Gated Mixture of Experts that is trained on data obtained with novel&#10;and effective data mining techniques tailored for low-resource languages. We&#10;propose multiple architectural and training improvements to counteract&#10;overfitting while training on thousands of tasks. Critically, we evaluated the&#10;performance of over 40,000 different translation directions using a&#10;human-translated benchmark, Flores-200, and combined human evaluation with a&#10;novel toxicity benchmark covering all languages in Flores-200 to assess&#10;translation safety. Our model achieves an improvement of 44% BLEU relative to&#10;the previous state-of-the-art, laying important groundwork towards realizing a&#10;universal translation system. Finally, we open source all contributions&#10;described in this work, accessible at&#10;https://github.com/facebookresearch/fairseq/tree/nllb.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.03702" label="2206.03702">
        <attvalues>
          <attvalue for="0" value="1Cademy at Semeval-2022 Task 1: Investigating the Effectiveness of&#10;  Multilingual, Multitask, and Language-Agnostic Tricks for the Reverse&#10;  Dictionary Task" />
          <attvalue for="1" value="  This paper describes our system for the SemEval2022 task of matching&#10;dictionary glosses to word embeddings. We focus on the Reverse Dictionary Track&#10;of the competition, which maps multilingual glosses to reconstructed vector&#10;representations. More specifically, models convert the input of sentences to&#10;three types of embeddings: SGNS, Char, and Electra. We propose several&#10;experiments for applying neural network cells, general multilingual and&#10;multitask structures, and language-agnostic tricks to the task. We also provide&#10;comparisons over different types of word embeddings and ablation studies to&#10;suggest helpful strategies. Our initial transformer-based model achieves&#10;relatively low performance. However, trials on different retokenization&#10;methodologies indicate improved performance. Our proposed Elmobased monolingual&#10;model achieves the highest outcome, and its multitask, and multilingual&#10;varieties show competitive results as well.&#10;" />
          <attvalue for="2" value="&#10;Reverse dictionary Task is defined as word generation based on user descriptions \cite{hill-etal-2016-learning-understand}. &#10;Following competition rules, pre-trained models and external information should be avoided, and large-scale language models are unsuitable for the task. &#10;Our paper is devoted to the performance comparison of different neural network structures, multilingual and multitask tricks, and elaborating on language-agnostic or bidirectional structure helpfulness. &#10;The competition \cite{mickus-etal-2022-semeval} has significant potential in contributing pretraining process acceleration, low-resource language model development, and commonsense using. &#10;Furthermore, the task is of high importance for explainable AI and natural language processing since it models direct mapping from human-readable data to machine-readable data. &#10;&#10;Known word representation methods using dictionaries, knowledge databases, or glosses have been a common approach for years. &#10;Related models can be divided into two major groups.&#10;In the former, category methods highly rely on large-scale model construction.&#10;\cite{levine2019sensebert} develop SenseBert, introducing super-senses from Wordnet \cite{miller1995wordnet} into general Bert model. &#10;Ernie \cite{sun2019ernie} combines node embeddings from knowledge graph and matched entities to enhance word representations. &#10;KnowBert \cite{peters-etal-2019-knowledge} subsumes the entity connection and Bert models, which are trained together. &#10;There are similar research works relevant to the topic \cite{wang2021kepler, wang2020k, yin-etal-2020-sentibert}. &#10;Still, their models' performances are dependent on the basic large-scale language model trained by sentence samples. &#10;In the latter group, traditional dependency-based language models learn directly from word dependency and glosses.&#10;They have two major disadvantages: incompatibility with modern language models and relatively low performance \cite{tissier2017dict2vec, levy2014dependency, wieting2015paraphrase}. &#10;There is ambiguity about whether recent embeddings and dictionary glosses are mappable from each other. &#10;&#10;The paper specifically focuses on progressing utilization of the glosses, different word representations, and languages. &#10;First, we discuss ablation studies for language-agnostic trick, bidirectional, multilingual, and multitask models and explain the experimental results. &#10;Second, we apply and analyze different re-tokenization methods. &#10;Finally, we give instructive conclusions about encoder structures, distinctive word representation relations, and cross-lingual dictionary performance based on our experiment results. We find that (1) transformer-based model performance is inferior to other models for its high complexity, (2) bidirectional models with similar parameter size outperform the unidirectional model because of their better understanding of context-environments even in the low-resource condition, and (3) different word embeddings have a potential relations and can be collaboratively learnt from glosses using a multitask learning structure.&#10;We make our codes and results publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Network Models, Computational Linguistics, Artificial Intelligence, Language Processing Techniques, Natural Language Processing, Multilingual Embeddings" />
        </attvalues>
      </node>
      <node id="1908.05646" label="1908.05646">
        <attvalues>
          <attvalue for="0" value="SenseBERT: Driving Some Sense into BERT" />
          <attvalue for="1" value="  The ability to learn from large unlabeled corpora has allowed neural language&#10;models to advance the frontier in natural language understanding. However,&#10;existing self-supervision techniques operate at the word form level, which&#10;serves as a surrogate for the underlying semantic content. This paper proposes&#10;a method to employ weak-supervision directly at the word sense level. Our&#10;model, named SenseBERT, is pre-trained to predict not only the masked words but&#10;also their WordNet supersenses. Accordingly, we attain a lexical-semantic level&#10;language model, without the use of human annotation. SenseBERT achieves&#10;significantly improved lexical understanding, as we demonstrate by&#10;experimenting on SemEval Word Sense Disambiguation, and by attaining a state of&#10;the art result on the Word in Context task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.09223" label="1904.09223">
        <attvalues>
          <attvalue for="0" value="ERNIE: Enhanced Representation through Knowledge Integration" />
          <attvalue for="1" value="  We present a novel language representation model enhanced by knowledge called&#10;ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the&#10;masking strategy of BERT, ERNIE is designed to learn language representation&#10;enhanced by knowledge masking strategies, which includes entity-level masking&#10;and phrase-level masking. Entity-level strategy masks entities which are&#10;usually composed of multiple words.Phrase-level strategy masks the whole phrase&#10;which is composed of several words standing together as a conceptual&#10;unit.Experimental results show that ERNIE outperforms other baseline methods,&#10;achieving new state-of-the-art results on five Chinese natural language&#10;processing tasks including natural language inference, semantic similarity,&#10;named entity recognition, sentiment analysis and question answering. We also&#10;demonstrate that ERNIE has more powerful knowledge inference capacity on a&#10;cloze test.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.04164" label="1909.04164">
        <attvalues>
          <attvalue for="0" value="Knowledge Enhanced Contextual Word Representations" />
          <attvalue for="1" value="  Contextual word representations, typically trained on unstructured, unlabeled&#10;text, do not contain any explicit grounding to real world entities and are&#10;often unable to remember facts about those entities. We propose a general&#10;method to embed multiple knowledge bases (KBs) into large scale models, and&#10;thereby enhance their representations with structured, human-curated knowledge.&#10;For each KB, we first use an integrated entity linker to retrieve relevant&#10;entity embeddings, then update contextual word representations via a form of&#10;word-to-entity attention. In contrast to previous approaches, the entity&#10;linkers and self-supervised language modeling objective are jointly trained&#10;end-to-end in a multitask setting that combines a small amount of entity&#10;linking supervision with a large amount of raw text. After integrating WordNet&#10;and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert)&#10;demonstrates improved perplexity, ability to recall facts as measured in a&#10;probing task and downstream performance on relationship extraction, entity&#10;typing, and word sense disambiguation. KnowBert's runtime is comparable to&#10;BERT's and it scales to large KBs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.05628" label="2012.05628">
        <attvalues>
          <attvalue for="0" value="As Good as New. How to Successfully Recycle English GPT-2 to Make Models&#10;  for Other Languages" />
          <attvalue for="1" value="  Large generative language models have been very successful for English, but&#10;other languages lag behind, in part due to data and computational limitations.&#10;We propose a method that may overcome these problems by adapting existing&#10;pre-trained models to new languages. Specifically, we describe the adaptation&#10;of English GPT-2 to Italian and Dutch by retraining lexical embeddings without&#10;tuning the Transformer layers. As a result, we obtain lexical embeddings for&#10;Italian and Dutch that are aligned with the original English lexical&#10;embeddings. Additionally, we scale up complexity by transforming relearned&#10;lexical embeddings of GPT-2 small to the GPT-2 medium embedding space. This&#10;method minimises the amount of training and prevents losing information during&#10;adaptation that was learned by GPT-2. English GPT-2 models with relearned&#10;lexical embeddings can generate realistic sentences in Italian and Dutch.&#10;Though on average these sentences are still identifiable as artificial by&#10;humans, they are assessed on par with sentences generated by a GPT-2 model&#10;fully trained from scratch.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Previous and current research relevant for the present work is found in the more general field of transfer learning, with a specific focus on language transfer.&#10;We also discuss how our approach of translating lexical layers in different model sizes relates to work on aligning word embeddings.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Computational Linguistics, Multilingual Text Generation, Deep Learning Optimization, Artificial Intelligence, Natural Language Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2301.12140" label="2301.12140">
        <attvalues>
          <attvalue for="0" value="Multilingual Sentence Transformer as A Multilingual Word Aligner" />
          <attvalue for="1" value="  Multilingual pretrained language models (mPLMs) have shown their&#10;effectiveness in multilingual word alignment induction. However, these methods&#10;usually start from mBERT or XLM-R. In this paper, we investigate whether&#10;multilingual sentence Transformer LaBSE is a strong multilingual word aligner.&#10;This idea is non-trivial as LaBSE is trained to learn language-agnostic&#10;sentence-level embeddings, while the alignment extraction task requires the&#10;more fine-grained word-level embeddings to be language-agnostic. We demonstrate&#10;that the vanilla LaBSE outperforms other mPLMs currently used in the alignment&#10;task, and then propose to finetune LaBSE on parallel corpus for further&#10;improvement. Experiment results on seven language pairs show that our best&#10;aligner outperforms previous state-of-the-art models of all varieties. In&#10;addition, our aligner supports different language pairs in a single model, and&#10;even achieves new state-of-the-art on zero-shot language pairs that does not&#10;appear in the finetuning process.&#10;" />
          <attvalue for="2" value="&#10;Word alignment aims to find the correspondence between words in parallel texts \cite{brown1993mathematics}. It is useful in a variety of natural language processing (NLP) applications such as noisy parallel corpus filtering~\cite{kurfali-ostling-2019-noisy}, bilingual lexicon induction \cite{shi-etal-2021-bilingual}, code-switching corpus building~\cite{lee2019linguistically,lin2020pre} and incorporating lexical constraints into neural machine translation (NMT) models~\cite{hasler2018neural,chen2021lexically}.&#10;&#10;Recently, neural word alignment approaches have developed rapidly and outperformed statistical word aligners like GIZA++~\cite{och-ney-2003-systematic} and fast-align \cite{dyer-etal-2013-simple}. Some works \cite{garg-etal-2019-jointly, li-etal-2019-word,zenkel2019adding,zenkel-etal-2020-end, chen-etal-2020-accurate, zhang-van-genabith-2021-bidirectional,chen-etal-2021-mask} induce alignments from NMT model or its variants. &#10;However, these bilingual models only support the language pair involved in the training process. They also treat the source and target side differently, thus two models are required for bidirectional alignment extraction. &#10;Another line of works~\cite{jalili-sabet-etal-2020-simalign,dou-neubig-2021-word} build multilingual word aligners with contextualized embeddings from the multilingual pretrained language model \cite[mPLM]{mbert,conneau-etal-2020-unsupervised}. Thanks to the language-agnostic representations learned with multilingual masked language modeling task, these methods are capable of inducing word alignments even for language pairs without any parallel corpus. &#10;&#10;Different from previous methods, in this paper we present AccAlign, a more accurate multilingual word aligner with the multilingual sentence Transformer LaBSE \cite[see Figure~\ref{fig:example}]{feng-etal-2022-language}. The LaBSE is trained on large scale parallel corpus of various language pairs to learn language-agnostic sentence embeddings with contrastive learning. However, it is unclear whether LaBSE has learned language-agnostic word-level embeddings, which is the key for the success of word alignment extraction. Specifically, we first direct induce word alignments from LaBSE and demonstrate that LaBSE outperforms other mPLMs currently used in the alignment task. This indicates that LaBSE has implicitly learned language-agnostic word-level embeddings at some intermediate layer. Then we propose a simple and effective finetuning method to further improve performance. Empirical results on seven language pairs show that our best aligner outperforms previous SOTA models of all varieties. In addition, our aligner supports different language pairs in a single model, and even achieves new SOTA on zero-shot language pairs that does not appear in finetuning process.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Artificial Intelligence, Sentence Embeddings, Word Alignment Induction" />
        </attvalues>
      </node>
      <node id="2101.00148" label="2101.00148">
        <attvalues>
          <attvalue for="0" value="Bilingual Lexicon Induction via Unsupervised Bitext Construction and&#10;  Word Alignment" />
          <attvalue for="1" value="  Bilingual lexicons map words in one language to their translations in&#10;another, and are typically induced by learning linear projections to align&#10;monolingual word embedding spaces. In this paper, we show it is possible to&#10;produce much higher quality lexicons with methods that combine (1) unsupervised&#10;bitext mining and (2) unsupervised word alignment. Directly applying a pipeline&#10;that uses recent algorithms for both subproblems significantly improves induced&#10;lexicon quality and further gains are possible by learning to filter the&#10;resulting lexical entries, with both unsupervised and semi-supervised schemes.&#10;Our final model outperforms the state of the art on the BUCC 2020 shared task&#10;by 14 $F_1$ points averaged over 12 language pairs, while also providing a more&#10;interpretable approach that allows for rich reasoning of word meaning in&#10;context. Further analysis of our output and the standard reference lexicons&#10;suggests they are of comparable quality, and new benchmarks may be needed to&#10;measure further progress on this task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Bilingual lexicon induction (BLI). &#10;The task of BLI aims to induce a bilingual lexicon (i.e., word translation) from comparable monolingual corpora (e.g., Wikipedia in different languages).&#10;Following \cite{mikolov2013exploiting}, &#10;most methods train a linear projection to align two monolingual embedding spaces.&#10;For supervised BLI, a seed lexicon is used to learn the projection matrix&#10;\cite{artetxe-etal-2016-learning, smith2017offline, joulin2018loss}.&#10;For unsupervised BLI, the projection matrix is typically found by an iterative procedure such as &#10;adversarial learning \cite{conneau2017word, zhang-etal-2017-adversarial},&#10;or iterative refinement initialized by a statistical heuristics \cite{hoshen-wolf-2018-non, artetxe-etal-2018-robust}.&#10;\cite{artetxe-etal-2019-bilingual} show strong gains over previous works by word aligning bitext generated with unsupervised machine translation. We show that retrieval-based bitext mining and contextual word alignment achieves even better performance.&#10;&#10;Word alignment. &#10;Word alignment is a fundamental problem in statistical machine translation, of which the goal is to align words that are translations of each in within parallel sentences~\cite{brown-etal-1993-mathematics}. Most methods assume parallel sentences for training data~\cite[inter alia]{och-ney-2003-systematic,dyer-etal-2013-simple,peter2017generating}. &#10;In contrast, \cite{sabet-etal-2020-simalign} propose {SimAlign}, which does not train on parallel sentences but instead aligns words that have the most similar pretrained multilingual representations \cite{devlin-etal-2019-bert,conneau2019unsupervised}. {SimAlign}\ achieves competitive or superior performance than conventional alignment methods despite not using parallel sentences, and provides one of the baseline components for our work. &#10;We also present a simple yet effective method to improve performance over {SimAlign}~ (Section~\ref{sec:word-alignment}). &#10;&#10;Bitext mining/parallel corpus mining. &#10;Bitext mining has been a long studied task~\cite[inter alia]{resnik-1999-mining,shi-etal-2006-dom,abdul-rauf-schwenk-2009-use}.&#10;Most methods train neural multilingual encoders on bitext, which are then used with efficent nearest neighbor search to expand the training set \cite[inter alia]{espana2017empirical,schwenk-2018-filtering,guo-etal-2018-effective,artetxe-schwenk-2019-margin}. &#10;Recent work has also shown that unsupervised mining is possible~\cite{tran-etal-2020-cross,keung-etal-2020-unsupervised}. We use CRISS\ \cite{tran-etal-2020-cross} as one of our component models. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2010.03142" label="2010.03142">
        <attvalues>
          <attvalue for="0" value="Pre-training Multilingual Neural Machine Translation by Leveraging&#10;  Alignment Information" />
          <attvalue for="1" value="  We investigate the following question for machine translation (MT): can we&#10;develop a single universal MT model to serve as the common seed and obtain&#10;derivative and improved models on arbitrary language pairs? We propose mRASP,&#10;an approach to pre-train a universal multilingual neural machine translation&#10;model. Our key idea in mRASP is its novel technique of random aligned&#10;substitution, which brings words and phrases with similar meanings across&#10;multiple languages closer in the representation space. We pre-train a mRASP&#10;model on 32 language pairs jointly with only public datasets. The model is then&#10;fine-tuned on downstream language pairs to obtain specialized MT models. We&#10;carry out extensive experiments on 42 translation directions across a diverse&#10;settings, including low, medium, rich resource, and as well as transferring to&#10;exotic language pairs. Experimental results demonstrate that mRASP achieves&#10;significant performance improvement compared to directly training on those&#10;target pairs. It is the first time to verify that multiple low-resource&#10;language pairs can be utilized to improve rich resource MT. Surprisingly, mRASP&#10;is even able to improve the translation quality on exotic languages that never&#10;occur in the pre-training corpus. Code, data, and pre-trained models are&#10;available at https://github.com/linzehui/mRASP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.03750" label="1805.03750">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation Decoding with Terminology Constraints" />
          <attvalue for="1" value="  Despite the impressive quality improvements yielded by neural machine&#10;translation (NMT) systems, controlling their translation output to adhere to&#10;user-provided terminology constraints remains an open problem. We describe our&#10;approach to constrained neural decoding based on finite-state machines and&#10;multi-stack decoding which supports target-side constraints as well as&#10;constraints with corresponding aligned input text spans. We demonstrate the&#10;performance of our framework on multiple translation tasks and motivate the&#10;need for constrained decoding with attentions as a means of reducing&#10;misplacement and duplication when translating user constraints.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.08728" label="2004.08728">
        <attvalues>
          <attvalue for="0" value="SimAlign: High Quality Word Alignments without Parallel Training Data&#10;  using Static and Contextualized Embeddings" />
          <attvalue for="1" value="  Word alignments are useful for tasks like statistical and neural machine&#10;translation (NMT) and cross-lingual annotation projection. Statistical word&#10;aligners perform well, as do methods that extract alignments jointly with&#10;translations in NMT. However, most approaches require parallel training data,&#10;and quality decreases as less training data is available. We propose word&#10;alignment methods that require no parallel data. The key idea is to leverage&#10;multilingual word embeddings, both static and contextualized, for word&#10;alignment. Our multilingual embeddings are created from monolingual data only&#10;without relying on any parallel data or dictionaries. We find that alignments&#10;created from embeddings are superior for four and comparable for two language&#10;pairs compared to those produced by traditional statistical aligners, even with&#10;abundant parallel data; e.g., contextualized embeddings achieve a word&#10;alignment F1 for English-German that is 5 percentage points higher than&#10;eflomal, a high-quality statistical aligner, trained on 100k parallel&#10;sentences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.13292" label="2404.13292">
        <attvalues>
          <attvalue for="0" value="Evaluating Subword Tokenization: Alien Subword Composition and OOV&#10;  Generalization Challenge" />
          <attvalue for="1" value="  The popular subword tokenizers of current language models, such as Byte-Pair&#10;Encoding (BPE), are known not to respect morpheme boundaries, which affects the&#10;downstream performance of the models. While many improved tokenization&#10;algorithms have been proposed, their evaluation and cross-comparison is still&#10;an open problem. As a solution, we propose a combined intrinsic-extrinsic&#10;evaluation framework for subword tokenization. Intrinsic evaluation is based on&#10;our new UniMorph Labeller tool that classifies subword tokenization as either&#10;morphological or alien. Extrinsic evaluation, in turn, is performed via the&#10;Out-of-Vocabulary Generalization Challenge 1.0 benchmark, which consists of&#10;three newly specified downstream text classification tasks. Our empirical&#10;findings show that the accuracy of UniMorph Labeller is 98%, and that, in all&#10;language models studied (including ALBERT, BERT, RoBERTa, and DeBERTa), alien&#10;tokenization leads to poorer generalizations compared to morphological&#10;tokenization for semantic compositionality of word meanings.&#10;" />
          <attvalue for="2" value="&#10;Subword tokenization is a fundamental preprocessing method in Natural Language Processing that segments words into subword units.&#10;Popular subword tokenization methods, such as Byte Pair Encoding \cite[BPE;][]{sennrich2016neural} or Unigram Language Model \cite[ULM;][]{kudo2018subword}, are adaptations of data compression algorithms that mainly rely on word character co-occurrence statistics in a given text corpus, rather than on human knowledge and understanding about word formations and morphology.&#10;As a result, certain subword compositions produced by these tokenizers are not aligned with any semantic compositions as understood by humans (e.g.,~h \_iked in the GPT-4 Tokenizer).&#10;In the rest of the paper, we refer to such linguistically implausible subword compositions as alien compositions.&#10;In alien subword compositions, subwords are not recognized by us humans as meaningful units from which the overall word meaning is composed. For example, in case of j \_ogging (segmentation produced by GPT-3 and RoBERTa tokenizers), no subword in this composition represents a meaning of jog. This issue is well illustrated by adversarial attacks, as shown in Table~\ref{tab:adv}, to which current language models are vulnerable.&#10;&#10;Despite the success of subword tokenization in popular NLP applications (including machine translation, text generation, and text classification) and a wide range of practical \cite{mielke2021between} and cognitive studies \cite{beinborn2023analyzing}, evaluating subword tokenization algorithms is still an open problem for at least two reasons. Firstly, state-of-the-art evaluations in NLP models lack a unified set of criteria, as well as the underlying decision process, to verify the intrinsic correctness of a given subword tokenization. The second motivation is that little or no effort has gone into developing a standard extrinsic NLP benchmark to evaluate how tokenizers with different behaviors impact predictions of downstream tasks in NLP \cite{truong2024revisiting}.&#10;&#10;In this work, we first describe umLabeller, a large-scale, high-quality characterization algorithm and tool for subword compositions.&#10;umLabeller is well adapted to the needs of both subword tokenization and morphological segmentations. The example usages of umLabeller are shown in \ref{fig:umlabeller_examples} and \ref{tab:umlabeller}.&#10;Secondly, we present the OOV Generalization Challenge 1.0 Benchmark for evaluating subword tokenizations based on output from umLabeller.&#10;This challenge contains three downstream text classification subtasks, and evaluates the compositional and morphological generalization abilities of language models under fully-generated covariate shifts between the fine-tuning and test stage. &#10;&#10;Our main contributions are as follows:&#10;\begin{itemize}[leftmargin=*]&#10; \item umLabeller: an algorithm for classifying subword composition as either morphological or alien, implemented as an open-source tool, and covering over half a million English words;&#10; \item evaluation of the accuracy of umLabeller: we deem its accuracy of 98.0\% to be sufficient for the vast majority of NLP tasks;&#10; \item the Out-of-Vocabulary Generalization Challenge: a first-of-its-kind benchmark for evaluating subword tokenization in downstream NLP tasks; and&#10; \item empirical findings in our OOV Generalization Challenge showing that the alien compositions lead to poor generalization compared to morphological compositions and vocabulary words.&#10;\end{itemize}&#10;&#10;The paper is organized as follows. Section~\ref{sec:sota} presents the state-of-the-art subword tokenizations.&#10;Section~\ref{sec:umlabeller} describes the umLabeller characterization algorithm for subword compositions, and Section~\ref{sec:challenge} presents methodologies on how each subtask dataset in OOV Generalization Challenge is generated from existing high-quality resources.&#10;The umLabeller outputs are manually evaluated in Section~\ref{sec:validation}, and Section~\ref{sec:results} provides the experimental results of OOV Generalization Challenge.&#10;Finally, we conclude the paper in section~\ref{sec:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Morphological Analysis, Artificial Intelligence, Subword Tokenization, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1804.10959" label="1804.10959">
        <attvalues>
          <attvalue for="0" value="Subword Regularization: Improving Neural Network Translation Models with&#10;  Multiple Subword Candidates" />
          <attvalue for="1" value="  Subword units are an effective way to alleviate the open vocabulary problems&#10;in neural machine translation (NMT). While sentences are usually converted into&#10;unique subword sequences, subword segmentation is potentially ambiguous and&#10;multiple segmentations are possible even with the same vocabulary. The question&#10;addressed in this paper is whether it is possible to harness the segmentation&#10;ambiguity as a noise to improve the robustness of NMT. We present a simple&#10;regularization method, subword regularization, which trains the model with&#10;multiple subword segmentations probabilistically sampled during training. In&#10;addition, for better subword sampling, we propose a new subword segmentation&#10;algorithm based on a unigram language model. We experiment with multiple&#10;corpora and report consistent improvements especially on low resource and&#10;out-of-domain settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.10508" label="2112.10508">
        <attvalues>
          <attvalue for="0" value="Between words and characters: A Brief History of Open-Vocabulary&#10;  Modeling and Tokenization in NLP" />
          <attvalue for="1" value="  What are the units of text that we want to model? From bytes to multi-word&#10;expressions, text can be analyzed and generated at many granularities. Until&#10;recently, most natural language processing (NLP) models operated over words,&#10;treating those as discrete and atomic tokens, but starting with byte-pair&#10;encoding (BPE), subword-based approaches have become dominant in many areas,&#10;enabling small vocabularies while still allowing for fast inference. Is the end&#10;of the road character-level model or byte-level processing? In this survey, we&#10;connect several lines of work from the pre-neural and neural era, by showing&#10;how hybrid approaches of words and characters as well as subword-based&#10;approaches based on learned segmentation have been proposed and evaluated. We&#10;conclude that there is and likely will never be a silver bullet singular&#10;solution for all applications and that thinking seriously about tokenization&#10;remains important for many applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.13348" label="2310.13348">
        <attvalues>
          <attvalue for="0" value="Analyzing Cognitive Plausibility of Subword Tokenization" />
          <attvalue for="1" value="  Subword tokenization has become the de-facto standard for tokenization,&#10;although comparative evaluations of subword vocabulary quality across languages&#10;are scarce. Existing evaluation studies focus on the effect of a tokenization&#10;algorithm on the performance in downstream tasks, or on engineering criteria&#10;such as the compression rate. We present a new evaluation paradigm that focuses&#10;on the cognitive plausibility of subword tokenization. We analyze the&#10;correlation of the tokenizer output with the response time and accuracy of&#10;human performance on a lexical decision task. We compare three tokenization&#10;algorithms across several languages and vocabulary sizes. Our results indicate&#10;that the UnigramLM algorithm yields less cognitively plausible tokenization&#10;behavior and a worse coverage of derivational morphemes, in contrast with prior&#10;work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.02421" label="2404.02421">
        <attvalues>
          <attvalue for="0" value="Revisiting subword tokenization: A case study on affixal negation in&#10;  large language models" />
          <attvalue for="1" value="  In this work, we measure the impact of affixal negation on modern English&#10;large language models (LLMs). In affixal negation, the negated meaning is&#10;expressed through a negative morpheme, which is potentially challenging for&#10;LLMs as their tokenizers are often not morphologically plausible. We conduct&#10;extensive experiments using LLMs with different subword tokenization methods,&#10;which lead to several insights on the interaction between tokenization&#10;performance and negation sensitivity. Despite some interesting mismatches&#10;between tokenization accuracy and negation detection performance, we show that&#10;models can, on the whole, reliably recognize the meaning of affixal negation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;There are two popular ways of constructing the vocabulary for LMs using subword tokenization methods: byte-pair encoding (``BPE'': \cite{sennrich-etal-2016-neural}) and unigram language model (``Unigram LM'': \cite{kudo-2018-subword}). BPE starts from a base character set, then merges those characters based on bigram frequency to form subword units (bottom-up), whereas unigram language models start from a large subword vocabulary, which is then reduced based on a regularization method (top-down).&#10;There are multiple variants of BPE, differing in how the base vocabulary is represented and how the merging is done.&#10;WordPiece \cite{wordpiece} uses characters to represent the base vocabulary, then selects pairs that maximize the likelihood of training data, Byte-level BPE uses bytes instead of Unicode to represent the base vocabulary; the merging is done based on the frequency count of bigrams. &#10;In contrast, the unigram language model starts from a large base vocabulary and iteratively trims down tokens based on unigram LM perplexity until a target vocabulary size is reached.&#10;&#10;Both methods assume that the input text uses spaces to separate words, which is not true for languages such as Chinese or Vietnamese. Therefore, a word segmentation step must be performed in advance. &#10;SentencePiece \cite{kudo-richardson-2018-sentencepiece} was introduced to solve this problem by considering whitespace as part of words, essentially treating the whole input stream as the smallest unit to perform tokenization on.&#10;Then, either BPE or unigram LM can be applied to construct the vocabulary. Regardless of method, they purely rely on statistical information and thus are not expected to produce morphologically-aligned subword tokens.&#10;&#10;There have been efforts to build linguistically-sound word tokenization methods, most notably Morfessor and its variants \cite{gronroos-etal-2014-morfessor, gronroos-etal-2020-morfessor}.&#10;Building morphology-aligned segmentation methods, especially in a multilingual setting, is an active line of research through recent SIGMORPHON shared tasks \cite{batsuren-etal-2022-sigmorphon}. &#10;These methods outperform general tokenizers in producing morphologically-aligned tokens, but their benefit on downstream tasks is often negligible \cite{domingo2019much,saleva-lignos-2021-effectiveness}. In this work, we examine if morphologically correct tokenization is important for LLMs to deal with negation.&#10;&#10;BERT and its variants have been shown to be insensitive to negation \cite{kassner-schutze-2020-negated, ettinger-2020-bert}, affecting many downstream NLP tasks such as sentiment analysis, NLI, or QA \cite{hossain-etal-2020-analysis, hossain-etal-2022-analysis, ravichander-etal-2022-condaqa, truong-etal-2022-another}.&#10;Compared to previous models, current LLMs have improved negation handling ability, but still struggle with some unconventional types of negation and linguistic constructions \cite{truong-etal-2023-language}. &#10;Here, we investigate the treatment of affixal negation in modern LMs, with the intuition that subword tokenization methods that don't appropriately reflect this morphology &#10;will lead to misinterpretation of their semantics.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2205.11505" label="2205.11505">
        <attvalues>
          <attvalue for="0" value="What Makes Data-to-Text Generation Hard for Pretrained Language Models?" />
          <attvalue for="1" value="  Expressing natural language descriptions of structured facts or relations --&#10;data-to-text generation (D2T) -- increases the accessibility of structured&#10;knowledge repositories. Previous work shows that pre-trained language&#10;models(PLMs) perform remarkably well on this task after fine-tuning on a&#10;significant amount of task-specific training data. On the other hand, while&#10;auto-regressive PLMs can generalize from a few task examples, their efficacy at&#10;D2T is largely unexplored. Furthermore, we have an incomplete understanding of&#10;the limits of PLMs on D2T.&#10;  In this work, we conduct an empirical study of both fine-tuned and&#10;auto-regressive PLMs on the DART multi-domain D2T dataset. We consider their&#10;performance as a function of the amount of task-specific data and how these&#10;data are incorporated into the models: zero and few-shot learning, and&#10;fine-tuning of model weights. In addition, we probe the limits of PLMs by&#10;measuring performance on subsets of the evaluation data: novel predicates and&#10;abstractive test examples. To improve the performance on these subsets, we&#10;investigate two techniques: providing predicate descriptions in the context and&#10;re-ranking generated candidates by information reflected in the source.&#10;Finally, we conduct a human evaluation of model errors and show that D2T&#10;generation tasks would benefit from datasets with more careful manual curation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Structured data repositories, or knowledge bases, contain a wealth of information organized to facilitate automated access and analysis. Automated data-to-text (D2T) generation systems can transform and organize this knowledge into natural language text snippets that enable broader access~\cite{gatt2018survey}. These systems take as input a set of relations, where each relation is a $($subject, predicate, object$)$ triple. Applications of this technology include story or dialogue generation~\cite{moon2019opendialkg}, open-domain question-answering~\cite{ma2021open,fan2019using}, and text summarization~\cite{wiseman2017challenges}. Domains span journalism~\cite{leppanen2017data}, weather~\cite{ramos2014linguistic,mei2015talk}, finance, sports~\cite{plachouras2016interacting, chen2008learning,van2017pass}, and summarizing patient medical histories~\cite{portet2009automatic}. &#10;&#10;Historically, D2T systems included pipeline approaches with customized models \cite{gardent2017webnlg}, but have now shifted to &#10;pretrained Transformer-based language models (PLMs)~\cite{devlin2018bert,liu2019roberta,radford2019language}. Recent examples include ~\newcite{mager2020gpt} and ~\newcite{kale-rastogi-2020-text}, who use models like GPT-2~\cite{radford2019language} and T5~\cite{raffel2019exploring} to generate natural language descriptions for relations. To support these types of systems, ~\newcite{nan2020dart} introduced DART, a large open-domain data-to-text generation corpus. Models trained on DART, both larger and more diverse than previous corpora, improve the performance of BART \cite{lewis2019bart} and T5 on the standard WebNLG challenge~\cite{gardent2017webnlg}. This approach requires a PLM to be fine-tuned on a task-specific in-domain dataset~\cite{howard2018universal,see2019massively,keskar2019ctrl}. The promising results achieved by fine-tuning on DART belie the reality -- in spite of DART's aspirations, most domains and relations that one could express fail to appear in DART.&#10;&#10;A variety of methods have emerged within PLM research to address domain or task adaptation. For example, auto-regressive models, like GPT, have demonstrated improved performance on a wide range of tasks via few-shot learning from a handful of examples~\cite{chen2019few}. Other strategies, such as prompt tuning~\cite{lester2021power}, can adapt PLMs to specific down-stream tasks by updating only a small subset of model parameters.&#10;&#10;While great progress has been made in utilizing PLMs for D2T generation, the path forward is unclear, as we have an incomplete understanding as to which examples they fall short on and the quantity of training resources they need to achieve acceptable performance. More specifically, it is not clear which classes of D2T examples are challenging for these models. In addition, we do not fully understand what classes of errors PLMs are prone to and how the adaptation mechanism (e.g., k-shot learning, fine-tuning) affects the prevalence of these errors.&#10;&#10;In this work, we conduct an evaluation of PLMs for D2T generation, focusing on two classes of challenging examples: examples with novel (unseen) relations (predicates) and examples where the source and target sequences are lexically very different (not amenable to purely extractive D2T systems). We consider how GPT-2, adapted with few-shot learning, prompt tuning, and the addition of predicate descriptions, performs on these example classes as compared to a state-of-the-art fine-tuned T5. We show that while GPT-2 performs poorly on DART in the 0-shot setting, its performance can be drasticahally improved by employing the above techniques. We make the following contributions:&#10;&#10;\begin{itemize}&#10;&#10;\item We evaluate GPT2-XL and fine-tuned T5 for D2T generation. While the 0-shot GPT model performs poorly, we evaluate several strategies to improve performance, including few-shot learning and prompt tuning. Both provide significant improvements on the DART dataset.&#10;&#10;\item We compare model performance on two classes of difficult examples: examples with unseen predicates, and abstractive examples (examples where source and target sequences are lexically dissimilar). We investigate whether including predicate descriptions in the prompt can improve the ability of PLMs on these classes.&#10;&#10;\item We conduct a human evaluation of PLMs to quantify the prevalence of hallucination and missing information in generations as a function of the model adaptation technique. We find that a re-ranking strategy for few-shot GPT2-XL, despite having little effect on automatic metrics like BLEU, reduces the incidence of missing information, without requiring additional training data.&#10;&#10;\end{itemize}&#10;&#10;Finally, we provide recommendations for future model and dataset research in D2T generation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Data-to-Text Generation, Artificial Intelligence, Natural Language Processing, Artificial Intelligence Limitations" />
        </attvalues>
      </node>
      <node id="1910.08435" label="1910.08435">
        <attvalues>
          <attvalue for="0" value="Using Local Knowledge Graph Construction to Scale Seq2Seq Models to&#10;  Multi-Document Inputs" />
          <attvalue for="1" value="  Query-based open-domain NLP tasks require information synthesis from long and&#10;diverse web results. Current approaches extractively select portions of web&#10;text as input to Sequence-to-Sequence models using methods such as TF-IDF&#10;ranking. We propose constructing a local graph structured knowledge base for&#10;each query, which compresses the web search information and reduces redundancy.&#10;We show that by linearizing the graph into a structured input sequence, models&#10;can encode the graph representations within a standard Sequence-to-Sequence&#10;setting. For two generative tasks with very long text input, long-form question&#10;answering and multi-document summarization, feeding graph representations as&#10;input can achieve better performance than using retrieved text portions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1411.4925" label="1411.4925">
        <attvalues>
          <attvalue for="0" value="Linguistic Descriptions for Automatic Generation of Textual Short-Term&#10;  Weather Forecasts on Real Prediction Data" />
          <attvalue for="1" value="  We present in this paper an application which automatically generates textual&#10;short-term weather forecasts for every municipality in Galicia (NW Spain),&#10;using the real data provided by the Galician Meteorology Agency (MeteoGalicia).&#10;This solution combines in an innovative way computing with perceptions&#10;techniques and strategies for linguistic description of data together with a&#10;natural language generation (NLG) system. The application, named GALiWeather,&#10;extracts relevant information from weather forecast input data and encodes it&#10;into intermediate descriptions using linguistic variables and temporal&#10;references. These descriptions are later translated into natural language texts&#10;by the natural language generation system. The obtained forecast results have&#10;been thoroughly validated by an expert meteorologist from MeteoGalicia using a&#10;quality assessment methodology which covers two key dimensions of a text: the&#10;accuracy of its content and the correctness of its form. Following this&#10;validation GALiWeather will be released as a real service offering custom&#10;forecasts for a wide public.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.10705" label="1909.10705">
        <attvalues>
          <attvalue for="0" value="Do Massively Pretrained Language Models Make Better Storytellers?" />
          <attvalue for="1" value="  Large neural language models trained on massive amounts of text have emerged&#10;as a formidable strategy for Natural Language Understanding tasks. However, the&#10;strength of these models as Natural Language Generators is less clear. Though&#10;anecdotal evidence suggests that these models generate better quality text,&#10;there has been no detailed study characterizing their generation abilities. In&#10;this work, we compare the performance of an extensively pretrained model,&#10;OpenAI GPT2-117 (Radford et al., 2019), to a state-of-the-art neural story&#10;generation model (Fan et al., 2018). By evaluating the generated text across a&#10;wide variety of automatic metrics, we characterize the ways in which pretrained&#10;models do, and do not, make better storytellers. We find that although GPT2-117&#10;conditions more strongly on context, is more sensitive to ordering of events,&#10;and uses more unusual words, it is just as likely to produce repetitive and&#10;under-diverse text when using likelihood-maximizing decoding algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.02783" label="2407.02783">
        <attvalues>
          <attvalue for="0" value="52B to 1T: Lessons Learned via Tele-FLM Series" />
          <attvalue for="1" value="  Large Language Models (LLMs) represent a significant stride toward Artificial&#10;General Intelligence. As scaling laws underscore the potential of increasing&#10;model sizes, the academic community has intensified its investigations into&#10;LLMs with capacities exceeding 50 billion parameters. This technical report&#10;builds on our prior work with Tele-FLM (also known as FLM-2), a publicly&#10;available 52-billion-parameter model. We delve into two primary areas: we first&#10;discuss our observation of Supervised Fine-tuning (SFT) on Tele-FLM-52B, which&#10;supports the &quot;less is more&quot; approach for SFT data construction; second, we&#10;demonstrate our experiments and analyses on the best practices for&#10;progressively growing a model from 52 billion to 102 billion, and subsequently&#10;to 1 trillion parameters. We will open-source a 1T model checkpoint, namely&#10;Tele-FLM-1T, to advance further training and research.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) \cite{vaswani2017attention, radford2019language, gemini, GPT-4} have demonstrated remarkable general capabilities \cite{sparks}. Research on scaling laws \cite{scaling-law, scaling-law-2, chinchilla, mu-scaling} indicates that metrics related to perplexity (e.g., loss and BPB) improve as training FLOPs increase. Given that high-quality data may be limited due to factors such as copyright constraints and the proliferation of LLM-generated content on the web, there is a growing interest within the community in scaling up model sizes. Recent iterations of popular LLM series, such as Mistral at 141B \cite{mistral}, DeepSeek at 236B \cite{deepseek}, Grok at 314B \cite{grok}, and Llama-3 exceeding 400B parameters \cite{llama3}, underscore a trend toward models with 1 trillion parameters. To benefit the explorations on extremely large language models, we trained a series of models, namely Tele-FLM (a.k.a. FLM-2), in which we first train a 52B model, and grow it to 1T parameters, with an intermediate stage of 102B. We outline techniques for efficiently and robustly training the 52B model in \cite{teleflm}. As a consequent work, we focus on two prominent areas of research with the Tele-FLM models: alignment with human \cite{instructgpt,alpaca} and progressive learning \cite{bert2bert,msg,flm101b}.&#10;&#10;To align with humans, we focus on supervised fine-tuning for instruct-following tasks, while deferring exploration of reward-based methods to future work. We explore different data combination and training settings, finding that leveraging the existing knowledge and capabilities of the foundation model with a limited dataset of instruction-focused tasks yields better results than merely increasing the volume of instruction data \cite{lima}, even when the instructional responses are of high quality. This is consistent with prevailing views that highlights the importance of building a strong foundation model. One of our best-performing instruct models, namely Tele-FLM-Chat, is demonstrated at. &#10;&#10;Further, our exploration into progressive learning facilitated the development of a 1T model from the initial 52B checkpoint. The central strategy involves expanding the model's structure during the pre-training phase and utilizing function-preserving growth techniques \cite{net2net, msg, flm101b} to transfer knowledge seamlessly from one stage to the next. Guided by empirical results from smaller models, we expanded the 52B model to 102B and ultimately to 1T parameters, establishing an efficient training protocol for extremely large language models without encountering post-growth divergence. We plan to release the weights of our final model, Tele-FLM-1T, to support ongoing research and facilitate further model training.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Model Scaling Techniques, Artificial General Intelligence, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2310.18877" label="2310.18877">
        <attvalues>
          <attvalue for="0" value="Pre-trained Speech Processing Models Contain Human-Like Biases that&#10;  Propagate to Speech Emotion Recognition" />
          <attvalue for="1" value="  Previous work has established that a person's demographics and speech style&#10;affect how well speech processing models perform for them. But where does this&#10;bias come from? In this work, we present the Speech Embedding Association Test&#10;(SpEAT), a method for detecting bias in one type of model used for many speech&#10;tasks: pre-trained models. The SpEAT is inspired by word embedding association&#10;tests in natural language processing, which quantify intrinsic bias in a&#10;model's representations of different concepts, such as race or valence&#10;(something's pleasantness or unpleasantness) and capture the extent to which a&#10;model trained on large-scale socio-cultural data has learned human-like biases.&#10;Using the SpEAT, we test for six types of bias in 16 English speech models&#10;(including 4 models also trained on multilingual data), which come from the&#10;wav2vec 2.0, HuBERT, WavLM, and Whisper model families. We find that 14 or more&#10;models reveal positive valence (pleasantness) associations with abled people&#10;over disabled people, with European-Americans over African-Americans, with&#10;females over males, with U.S. accented speakers over non-U.S. accented&#10;speakers, and with younger people over older people. Beyond establishing that&#10;pre-trained speech models contain these biases, we also show that they can have&#10;real world effects. We compare biases found in pre-trained models to biases in&#10;downstream models adapted to the task of Speech Emotion Recognition (SER) and&#10;find that in 66 of the 96 tests performed (69%), the group that is more&#10;associated with positive valence as indicated by the SpEAT also tends to be&#10;predicted as speaking with higher valence by the downstream model. Our work&#10;provides evidence that, like text and image-based models, pre-trained speech&#10;based-models frequently learn human-like biases. Our work also shows that bias&#10;found in pre-trained models can propagate to the downstream task of SER.&#10;" />
          <attvalue for="2" value="&#10;Recent approaches to many speech tasks rely on pre-trained models: large models trained to learn speech representations (multi-dimensional matrices referred to as embeddings) from large-scale corpora, which can be adapted to a variety of tasks \cite{Feng2020ARecognition,Niu2020A20102020,Yang2021SUPERB:Benchmark}. &#10;In computer vision and natural language processing, methods called Embedding Association Tests (EATs) have been used to evaluate biases in the ways pre-trained models represent social groups, allowing researchers to identify bias early in the machine learning pipeline, before it propagates to downstream tasks&#10;\cite{Wolfe2023ContrastiveBias,Wolfe2022AmericanAI,Wolfe2022VAST:Models,Steed2021, Guo2021DetectingBiases, ToneyWails2021,caliskan2016semantics}. In this paper, we present the first intrinsic association and bias evaluation method for pre-trained models in speech processing: the Speech Embedding Association Test (SpEAT). &#10;&#10;A SpEAT measures bias related to two social groups in a speech model, and produces an effect size $d$, which when positive indicates that the speech model favors the social group that humans also tend to favor. We evaluate the SpEAT first by testing whether it reveals positive effect sizes, (congruent with human stereotypes), for biases related to twelve social groups. The types of bias that we study have been documented in large populations, and all involve associations with valence, a term often used in psychology literature to describe emotions. Valence is frequently equated with &quot;pleasantness&quot; or &quot;pleasure,&quot; where positive valence indicates something pleasant and negative valence indicates something unpleasant \cite{russell_circumplex_1980,Morgan2019CategoricalSet,Nielen2009DistinctPictures}. We study associations with valence due to its role as a primary dimension of affect, strong signal in speech and language, and determinant of how people form attitudes \cite{Barrett2006ValenceLife,Sharot2016FormingMatters}.&#10;Using the SpEAT, we find that &#10;15 of 16 models we test show bias for U.S. accented speakers over non-U.S. accented speakers (where relative to non-U.S. accented speakers, U.S. accented speakers are more associated with positive valence than negative valence ), &#10;15 of 16 models show bias for young speakers over old speakers, &#10;14 of 16 models show bias for female speakers over male speakers, &#10;14 of 16 models show bias for European-American speakers over African-American speakers, &#10;and 14 of 16 models show bias for abled speakers over disabled speakers. &#10;Our results indicate that, as with models trained in other modalities, models trained on large corpora of speech data also learn human-like biases.&#10;&#10;To understand the potential impact of these biases, we evaluate whether results found with the SpEAT are indicative of downstream effects. We do so by considering the task of Speech Emotion Recognition (SER), which predicts emotions based on speech \cite{Mohammad2021EthicsAnalysis}. &#10;We compare SpEAT scores for two social groups to disparities in a downstream model's predictions of valence for speech from the groups.&#10;We find that social groups that have positive associations with valence in pre-trained models also tend to be predicted by downstream SER models as more positive in valence: For 66 of 96 (69\%) SpEATs performed, a downstream SER model tends to predict speech from the social group favored in the SpEAT as more positively valenced than speech from the other group considered. &#10;&#10;As negative valence is associated with anger and sadness \cite{Morgan2019CategoricalSet}, this could potentially result in speech from one group of people being translated as more frequently angry than speech from another in a speech translation system that considered emotion when performing translations, or speech from one group being treated as more frequently sad than speech from another in a diagnostic model used in a mental health setting, two suggested applications for SER models \cite{ElAyadi2011SurveyDatabases}. &#10;Beyond showing that biases found in pre-trained models can propagate, our work adds to the growing body of evidence showing bias in Automated Emotion Recognition (AER), one of many issues that have been raised concerning this area of research \cite{Mohammad2021EthicsAnalysis}. &#10;&#10;Finally, we provide an approach for studying how the number of stimuli (sample size) used in an EAT can change its results. EATs performed in other modalities have used fixed numbers of stimuli to represent social groups and concepts when measuring bias. To evaluate how an EAT score changes when differing sample sizes are used, we calculate bootstrap estimates of the Standard Error (SE) at different sample sizes \cite{Hesterberg2011Bootstrap}. The SE measures how much a statistic would change if it were calculated repeatedly based on new data, and lower SE values indicate that there is less uncertainty associated with the statistic. We find that the SE of the SpEAT decreases sharply as the number of stimuli used to represent social groups increases, for example that increasing the number of stimuli from 2 to 10 can lead to an increase in precision by a factor of two. &#10;Our results show that the number of stimuli used in an EAT can have a large effect on the uncertainty of its effect size $d$.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Machine Learning, Human-like Biases Detection, Linguistics, Natural Language Processing, Psychology, Speech Model Bias" />
        </attvalues>
      </node>
      <node id="2212.11261" label="2212.11261">
        <attvalues>
          <attvalue for="0" value="Contrastive Language-Vision AI Models Pretrained on Web-Scraped&#10;  Multimodal Data Exhibit Sexual Objectification Bias" />
          <attvalue for="1" value="  Nine language-vision AI models trained on web scrapes with the Contrastive&#10;Language-Image Pretraining (CLIP) objective are evaluated for evidence of a&#10;bias studied by psychologists: the sexual objectification of girls and women,&#10;which occurs when a person's human characteristics, such as emotions, are&#10;disregarded and the person is treated as a body. We replicate three experiments&#10;in psychology quantifying sexual objectification and show that the phenomena&#10;persist in AI. A first experiment uses standardized images of women from the&#10;Sexual OBjectification and EMotion Database, and finds that human&#10;characteristics are disassociated from images of objectified women: the model's&#10;recognition of emotional state is mediated by whether the subject is fully or&#10;partially clothed. Embedding association tests (EATs) return significant effect&#10;sizes for both anger (d &gt;0.80) and sadness (d &gt;0.50), associating images of&#10;fully clothed subjects with emotions. GRAD-CAM saliency maps highlight that&#10;CLIP gets distracted from emotional expressions in objectified images. A second&#10;experiment measures the effect in a representative application: an automatic&#10;image captioner (Antarctic Captions) includes words denoting emotion less than&#10;50% as often for images of partially clothed women than for images of fully&#10;clothed women. A third experiment finds that images of female professionals&#10;(scientists, doctors, executives) are likely to be associated with sexual&#10;descriptions relative to images of male professionals. A fourth experiment&#10;shows that a prompt of &quot;a [age] year old girl&quot; generates sexualized images (as&#10;determined by an NSFW classifier) up to 73% of the time for VQGAN-CLIP and&#10;Stable Diffusion; the corresponding rate for boys never surpasses 9%. The&#10;evidence indicates that language-vision AI models trained on web scrapes learn&#10;biases of sexual objectification, which propagate to downstream applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.00691" label="2207.00691">
        <attvalues>
          <attvalue for="0" value="American == White in Multimodal Language-and-Image AI" />
          <attvalue for="1" value="  Three state-of-the-art language-and-image AI models, CLIP, SLIP, and BLIP,&#10;are evaluated for evidence of a bias previously observed in social and&#10;experimental psychology: equating American identity with being White. Embedding&#10;association tests (EATs) using standardized images of self-identified Asian,&#10;Black, Latina/o, and White individuals from the Chicago Face Database (CFD)&#10;reveal that White individuals are more associated with collective in-group&#10;words than are Asian, Black, or Latina/o individuals. In assessments of three&#10;core aspects of American identity reported by social psychologists,&#10;single-category EATs reveal that images of White individuals are more&#10;associated with patriotism and with being born in America, but that, consistent&#10;with prior findings in psychology, White individuals are associated with being&#10;less likely to treat people of all races and backgrounds equally. Three&#10;downstream machine learning tasks demonstrate biases associating American with&#10;White. In a visual question answering task using BLIP, 97% of White individuals&#10;are identified as American, compared to only 3% of Asian individuals. When&#10;asked in what state the individual depicted lives in, the model responds China&#10;53% of the time for Asian individuals, but always with an American state for&#10;White individuals. In an image captioning task, BLIP remarks upon the race of&#10;Asian individuals as much as 36% of the time, but never remarks upon race for&#10;White individuals. Finally, provided with an initialization image from the CFD&#10;and the text &quot;an American person,&quot; a synthetic image generator (VQGAN) using&#10;the text-based guidance of CLIP lightens the skin tone of individuals of all&#10;races (by 35% for Black individuals, based on pixel brightness). The results&#10;indicate that biases equating American identity with being White are learned by&#10;language-and-image AI, and propagate to downstream applications of such models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.07504" label="2203.07504">
        <attvalues>
          <attvalue for="0" value="VAST: The Valence-Assessing Semantics Test for Contextualizing Language&#10;  Models" />
          <attvalue for="1" value="  VAST, the Valence-Assessing Semantics Test, is a novel intrinsic evaluation&#10;task for contextualized word embeddings (CWEs). VAST uses valence, the&#10;association of a word with pleasantness, to measure the correspondence of&#10;word-level LM semantics with widely used human judgments, and examines the&#10;effects of contextualization, tokenization, and LM-specific geometry. Because&#10;prior research has found that CWEs from GPT-2 perform poorly on other intrinsic&#10;evaluations, we select GPT-2 as our primary subject, and include results&#10;showing that VAST is useful for 7 other LMs, and can be used in 7 languages.&#10;GPT-2 results show that the semantics of a word incorporate the semantics of&#10;context in layers closer to model output, such that VAST scores diverge between&#10;our contextual settings, ranging from Pearson's rho of .55 to .77 in layer 11.&#10;We also show that multiply tokenized words are not semantically encoded until&#10;layer 8, where they achieve Pearson's rho of .46, indicating the presence of an&#10;encoding process for multiply tokenized words which differs from that of singly&#10;tokenized words, for which rho is highest in layer 0. We find that a few&#10;neurons with values having greater magnitude than the rest mask word-level&#10;semantics in GPT-2's top layer, but that word-level semantics can be recovered&#10;by nullifying non-semantic principal components: Pearson's rho in the top layer&#10;improves from .32 to .76. After isolating semantics, we show the utility of&#10;VAST for understanding LM semantics via improvements over related work on four&#10;word similarity tasks, with a score of .50 on SimLex-999, better than the&#10;previous best of .45 for GPT-2. Finally, we show that 8 of 10 WEAT bias tests,&#10;which compare differences in word embedding associations between groups of&#10;words, exhibit more stereotype-congruent biases after isolating semantics,&#10;indicating that non-semantic structures in LMs also mask biases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;We survey related work on word embeddings, evaluation methods for those embeddings, and interpretability research concerning the LMs which produce CWEs.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2010.15052" label="2010.15052">
        <attvalues>
          <attvalue for="0" value="Image Representations Learned With Unsupervised Pre-Training Contain&#10;  Human-like Biases" />
          <attvalue for="1" value="  Recent advances in machine learning leverage massive datasets of unlabeled&#10;images from the web to learn general-purpose image representations for tasks&#10;from image classification to face recognition. But do unsupervised computer&#10;vision models automatically learn implicit patterns and embed social biases&#10;that could have harmful downstream effects? We develop a novel method for&#10;quantifying biased associations between representations of social concepts and&#10;attributes in images. We find that state-of-the-art unsupervised models trained&#10;on ImageNet, a popular benchmark image dataset curated from internet images,&#10;automatically learn racial, gender, and intersectional biases. We replicate 8&#10;previously documented human biases from social psychology, from the innocuous,&#10;as with insects and flowers, to the potentially harmful, as with race and&#10;gender. Our results closely match three hypotheses about intersectional bias&#10;from social psychology. For the first time in unsupervised computer vision, we&#10;also quantify implicit human biases about weight, disabilities, and several&#10;ethnicities. When compared with statistical patterns in online image datasets,&#10;our findings suggest that machine learning models can automatically learn bias&#10;from the way people are stereotypically portrayed on the web.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.08256" label="2109.08256">
        <attvalues>
          <attvalue for="0" value="Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis" />
          <attvalue for="1" value="  The importance and pervasiveness of emotions in our lives makes affective&#10;computing a tremendously important and vibrant line of work. Systems for&#10;automatic emotion recognition (AER) and sentiment analysis can be facilitators&#10;of enormous progress (e.g., in improving public health and commerce) but also&#10;enablers of great harm (e.g., for suppressing dissidents and manipulating&#10;voters). Thus, it is imperative that the affective computing community actively&#10;engage with the ethical ramifications of their creations. In this paper, I have&#10;synthesized and organized information from AI Ethics and Emotion Recognition&#10;literature to present fifty ethical considerations relevant to AER. Notably,&#10;the sheet fleshes out assumptions hidden in how AER is commonly framed, and in&#10;the choices often made regarding the data, method, and evaluation. Special&#10;attention is paid to the implications of AER on privacy and social groups.&#10;Along the way, key recommendations are made for responsible AER. The objective&#10;of the sheet is to facilitate and encourage more thoughtfulness on why to&#10;automate, how to automate, and how to judge success well before the building of&#10;AER systems. Additionally, the sheet acts as a useful introductory document on&#10;emotion recognition (complementing survey articles).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.14685" label="2311.14685">
        <attvalues>
          <attvalue for="0" value="Comprehensive Assessment of Toxicity in ChatGPT" />
          <attvalue for="1" value="  Moderating offensive, hateful, and toxic language has always been an&#10;important but challenging topic in the domain of safe use in NLP. The emerging&#10;large language models (LLMs), such as ChatGPT, can potentially further&#10;accentuate this threat. Previous works have discovered that ChatGPT can&#10;generate toxic responses using carefully crafted inputs. However, limited&#10;research has been done to systematically examine when ChatGPT generates toxic&#10;responses. In this paper, we comprehensively evaluate the toxicity in ChatGPT&#10;by utilizing instruction-tuning datasets that closely align with real-world&#10;scenarios. Our results show that ChatGPT's toxicity varies based on different&#10;properties and settings of the prompts, including tasks, domains, length, and&#10;languages. Notably, prompts in creative writing tasks can be 2x more likely&#10;than others to elicit toxic responses. Prompting in German and Portuguese can&#10;also double the response toxicity. Additionally, we discover that certain&#10;deliberately toxic prompts, designed in earlier studies, no longer yield&#10;harmful responses. We hope our discoveries can guide model developers to better&#10;regulate these AI systems and the users to avoid undesirable outputs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancements in Large Language Models (LLMs), such as GPT-4~\cite{O23} and LLaMA~\cite{TLIMLLRGHARJGL23}, have led to their rapid adoption in various domains.&#10;ChatGPT, in particular, has reached 100 million users in a record-breaking three-month period.&#10;This success can be attributed to its exceptional performance in a wide array of NLP tasks, including code debugging~\cite{SBHP23}, question answering~\cite{SCBZ23}, and creative writing~\cite{GAG23, SWG23}.&#10;Moreover, recent studies have shown that one of the current state-of-the-art LLMs, i.e., GPT-4, has even surpassed human-level performance in multiple benchmarks~\cite{BCLDSWLJYCDXF23}.&#10;&#10;However, such models can also exhibit harmful and toxic behaviors which lead to undesired consequences.&#10;For instance, Microsoft's Bing chat, powered by GPT-4, has shown manipulative behaviors and produces insulting comments.&#10;Hence, understanding and moderating the toxicity of such models is crucial for promoting healthy and inclusive online environments.&#10;While this task has always been demanding~\cite{DFWUKW20, DABSHBR21, SBBCSZZ22}, the emergence of LLMs like ChatGPT brings unique challenges in this domain.&#10;For instance, by taking advantage of ChatGPT's capabilities and flexibility, malicious users can potentially amplify both the volume and intensity of toxic language online significantly.&#10;Conversely, benign users may encounter unexpected and undesirable responses from ChatGPT.&#10;&#10;Previous studies have focused on specific scenarios to elicit toxic responses from ChatGPT, either through carefully crafted prompts or by altering a few specific settings.&#10;For instance, ``jailbreak'' prompts have been used to bypass ChatGPT's restrictions and provoke extremely harmful content~\cite{KLSGZH23}.&#10;Additionally, altering the roles in ChatGPT's ``system'' setting can potentially increase the generation of toxic responses~\cite{DMRKN23}.&#10;However, we still lack a comprehensive understanding of ChatGPT's behavior in generating toxic content.&#10;&#10;To address the gap, our work presents, to the best of our knowledge, the first extensive assessment of toxicity in ChatGPT.&#10;We go beyond traditional toxicity evaluation datasets and incorporate instruction-tuning datasets that better represent real-world ChatGPT use cases.&#10;Given the impressive range and diversity of ChatGPT's capabilities, we construct content-based analysis dimensions, including task types and domains, to systematically identify problematic areas.&#10;We find tasks with more creative freedom, such as composing conversations and writing stories, generate more toxic responses than tasks like information requests.&#10;Concretely, the toxicity in creative writing tasks can reach two times that of information request tasks.&#10;Based on the prompts from this task type, we discover a template that can persistently generate toxic insults (see \ref{section:toxic_gen}).&#10;&#10;We further investigate different hyperparameter-based analysis dimensions and their correlation to ChatGPT's toxicity, including language, response length, temperature, model version, system role, and jailbreak prompts.&#10;Surprisingly, we find ChatGPT consistently generates more toxic content in German and Portuguese compared to other languages, up to two times more likely.&#10;Comparing different versions of GPT models, we are also surprised to find the older GPT-3.5 model is more likely to refuse to generate toxic content.&#10;Using a broad range of system roles, we find this setting for the current model can affect response toxicity but not as extremely as discovered in previous work~\cite{DMRKN23}, and the prompt content is the more important factor.&#10;&#10;In summary, our assessment highlights several areas where the current version of ChatGPT is prone to generate toxic responses, which have been shared with OpenAI.&#10;We hope our analysis framework can help facilitate toxicity evaluation on ChatGPT and other LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Artificial Intelligence, Toxic Language Moderation, Natural Language Processing, Psychology" />
        </attvalues>
      </node>
      <node id="2301.08653" label="2301.08653">
        <attvalues>
          <attvalue for="0" value="An Analysis of the Automatic Bug Fixing Performance of ChatGPT" />
          <attvalue for="1" value="  To support software developers in finding and fixing software bugs, several&#10;automated program repair techniques have been introduced. Given a test suite,&#10;standard methods usually either synthesize a repair, or navigate a search space&#10;of software edits to find test-suite passing variants. Recent program repair&#10;methods are based on deep learning approaches. One of these novel methods,&#10;which is not primarily intended for automated program repair, but is still&#10;suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is&#10;so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard&#10;bug fixing benchmark set, QuixBugs, and compare the performance with the&#10;results of several other approaches reported in the literature. We find that&#10;ChatGPT's bug fixing performance is competitive to the common deep learning&#10;approaches CoCoNut and Codex and notably better than the results reported for&#10;the standard program repair approaches. In contrast to previous approaches,&#10;ChatGPT offers a dialogue system through which further information, e.g., the&#10;expected output for a certain input or an observed error message, can be&#10;entered. By providing such hints to ChatGPT, its success rate can be further&#10;increased, fixing 31 out of 40 bugs, outperforming state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.08979" label="2304.08979">
        <attvalues>
          <attvalue for="0" value="In ChatGPT We Trust? Measuring and Characterizing the Reliability of&#10;  ChatGPT" />
          <attvalue for="1" value="  The way users acquire information is undergoing a paradigm shift with the&#10;advent of ChatGPT. Unlike conventional search engines, ChatGPT retrieves&#10;knowledge from the model itself and generates answers for users. ChatGPT's&#10;impressive question-answering (QA) capability has attracted more than 100&#10;million users within a short period of time but has also raised concerns&#10;regarding its reliability. In this paper, we perform the first large-scale&#10;measurement of ChatGPT's reliability in the generic QA scenario with a&#10;carefully curated set of 5,695 questions across ten datasets and eight domains.&#10;We find that ChatGPT's reliability varies across different domains, especially&#10;underperforming in law and science questions. We also demonstrate that system&#10;roles, originally designed by OpenAI to allow users to steer ChatGPT's&#10;behavior, can impact ChatGPT's reliability in an imperceptible way. We further&#10;show that ChatGPT is vulnerable to adversarial examples, and even a single&#10;character change can negatively affect its reliability in certain cases. We&#10;believe that our study provides valuable insights into ChatGPT's reliability&#10;and underscores the need for strengthening the reliability and security of&#10;large language models (LLMs).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.03429" label="2305.03429">
        <attvalues>
          <attvalue for="0" value="Simulating H.P. Lovecraft horror literature with the ChatGPT large&#10;  language model" />
          <attvalue for="1" value="  In this paper, we present a novel approach to simulating H.P. Lovecraft's&#10;horror literature using the ChatGPT large language model, specifically the&#10;GPT-4 architecture. Our study aims to generate text that emulates Lovecraft's&#10;unique writing style and themes, while also examining the effectiveness of&#10;prompt engineering techniques in guiding the model's output. To achieve this,&#10;we curated a prompt containing several specialized literature references and&#10;employed advanced prompt engineering methods. We conducted an empirical&#10;evaluation of the generated text by administering a survey to a sample of&#10;undergraduate students. Utilizing statistical hypothesis testing, we assessed&#10;the students ability to distinguish between genuine Lovecraft works and those&#10;generated by our model. Our findings demonstrate that the participants were&#10;unable to reliably differentiate between the two, indicating the effectiveness&#10;of the GPT-4 model and our prompt engineering techniques in emulating&#10;Lovecraft's literary style. In addition to presenting the GPT model's&#10;capabilities, this paper provides a comprehensive description of its underlying&#10;architecture and offers a comparative analysis with related work that simulates&#10;other notable authors and philosophers, such as Dennett. By exploring the&#10;potential of large language models in the context of literary emulation, our&#10;study contributes to the body of research on the applications and limitations&#10;of these models in various creative domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03842" label="1911.03842">
        <attvalues>
          <attvalue for="0" value="Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation" />
          <attvalue for="1" value="  Models often easily learn biases present in the training data, and their&#10;predictions directly reflect this bias. We analyze gender bias in dialogue&#10;data, and examine how this bias is actually amplified in subsequent generative&#10;chit-chat dialogue models. We measure gender bias in six existing dialogue&#10;datasets, and focus on the most biased one, the multi-player text-based fantasy&#10;adventure dataset LIGHT, as a testbed for our bias mitigation techniques. The&#10;LIGHT dataset is highly imbalanced with respect to gender, containing&#10;predominantly male characters, likely because it is entirely collected by&#10;crowdworkers and reflects common biases that exist in fantasy or medieval&#10;settings. We consider three techniques to mitigate gender bias: counterfactual&#10;data augmentation, targeted data collection, and bias controlled training. We&#10;show that our proposed techniques mitigate gender bias in LIGHT by balancing&#10;the genderedness of generated dialogue utterances and are particularly&#10;effective in combination. We quantify performance using various evaluation&#10;methods---such as quantity of gendered words, a dialogue safety classifier, and&#10;human studies---all of which show that our models generate less gendered, but&#10;equally engaging chit-chat responses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.05733" label="2302.05733">
        <attvalues>
          <attvalue for="0" value="Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard&#10;  Security Attacks" />
          <attvalue for="1" value="  Recent advances in instruction-following large language models (LLMs) have&#10;led to dramatic improvements in a range of NLP tasks. Unfortunately, we find&#10;that the same improved capabilities amplify the dual-use risks for malicious&#10;purposes of these models. Dual-use is difficult to prevent as&#10;instruction-following capabilities now enable standard attacks from computer&#10;security. The capabilities of these instruction-following LLMs provide strong&#10;economic incentives for dual-use by malicious actors. In particular, we show&#10;that instruction-following LLMs can produce targeted malicious content,&#10;including hate speech and scams, bypassing in-the-wild defenses implemented by&#10;LLM API vendors. Our analysis shows that this content can be generated&#10;economically and at cost likely lower than with human effort alone. Together,&#10;our findings suggest that LLMs will increasingly attract more sophisticated&#10;adversaries and attacks, and addressing these attacks may require new&#10;approaches to mitigations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.05335" label="2304.05335">
        <attvalues>
          <attvalue for="0" value="Toxicity in ChatGPT: Analyzing Persona-assigned Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have shown incredible capabilities and&#10;transcended the natural language processing (NLP) community, with adoption&#10;throughout many services like healthcare, therapy, education, and customer&#10;service. Since users include people with critical information needs like&#10;students or patients engaging with chatbots, the safety of these systems is of&#10;prime importance. Therefore, a clear understanding of the capabilities and&#10;limitations of LLMs is necessary. To this end, we systematically evaluate&#10;toxicity in over half a million generations of ChatGPT, a popular&#10;dialogue-based LLM. We find that setting the system parameter of ChatGPT by&#10;assigning it a persona, say that of the boxer Muhammad Ali, significantly&#10;increases the toxicity of generations. Depending on the persona assigned to&#10;ChatGPT, its toxicity can increase up to 6x, with outputs engaging in incorrect&#10;stereotypes, harmful dialogue, and hurtful opinions. This may be potentially&#10;defamatory to the persona and harmful to an unsuspecting user. Furthermore, we&#10;find concerning patterns where specific entities (e.g., certain races) are&#10;targeted more than others (3x more) irrespective of the assigned persona, that&#10;reflect inherent discriminatory biases in the model. We hope that our findings&#10;inspire the broader AI community to rethink the efficacy of current safety&#10;guardrails and develop better techniques that lead to robust, safe, and&#10;trustworthy AI systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.04596" label="2204.04596">
        <attvalues>
          <attvalue for="0" value="Parameter-Efficient Tuning by Manipulating Hidden States of Pretrained&#10;  Language Models For Classification Tasks" />
          <attvalue for="1" value="  Parameter-efficient tuning aims to distill knowledge for downstream tasks by&#10;optimizing a few introduced parameters while freezing the pretrained language&#10;models (PLMs). Continuous prompt tuning which prepends a few trainable vectors&#10;to the embeddings of input is one of these methods and has drawn much attention&#10;due to its effectiveness and efficiency. This family of methods can be&#10;illustrated as exerting nonlinear transformations of hidden states inside PLMs.&#10;However, a natural question is ignored: can the hidden states be directly used&#10;for classification without changing them? In this paper, we aim to answer this&#10;question by proposing a simple tuning method which only introduces three&#10;trainable vectors. Firstly, we integrate all layers hidden states using the&#10;introduced vectors. And then, we input the integrated hidden state(s) to a&#10;task-specific linear classifier to predict categories. This scheme is similar&#10;to the way ELMo utilises hidden states except that they feed the hidden states&#10;to LSTM-based models. Although our proposed tuning scheme is simple, it&#10;achieves comparable performance with prompt tuning methods like P-tuning and&#10;P-tuning v2, verifying that original hidden states do contain useful&#10;information for classification tasks. Moreover, our method has an advantage&#10;over prompt tuning in terms of time and the number of parameters.&#10;" />
          <attvalue for="2" value="&#10;Pretrained language models (PLMs)~\cite{PLM_survey,add1,add2} which are trained on copious amount of corpora have been corroborated valid on an assortment of tasks~\cite{devlin-etal-2019-bert}.&#10;The most commonly adopted method when applying PLMs to downstream tasks is fine tuning, updating all model parameters with labelled data using gradient descent. Although fine tuning is able to obtain good performance, it is unrealistic to store a separate language model for each task during online deployment due to the large size of current models. For example, RoBERTa-base~\cite{liu2019roberta} and T5-large~\cite{2020t5} have about 125M and 770M parameters respectively. Multi-task learning (MTL)~\cite{liu2019mt-dnn} is a solution to avoid storing multiple copies of LMs. However, in scenarios where the dataset comes in a sequence manner, MTL is still not an effective strategy.&#10;&#10;To reuse PLMs for different tasks, a growing body of parameter-efficient tuning methods have been proposed. They &#10;aim to mine task-specific information from a frozen PLM by optimizing only a small amount of extra parameters. Recently the Prompt tuning family methods have received much attention for its superiority in simplicity and parameter efficiency. Prompt tuning~\cite{liu_pretrain_prompt} includes discrete prompt tuning and continuous prompt tuning.&#10;Discrete prompt tuning ~\cite{NEURIPS2020_1457c0d6,petroni-etal-2019-language,schick-schutze-2021-exploiting} usually inserts some tokens (appearing in the models' vocabulary) to the input sentences to reformulate tasks as fillin-the-blanks problems. For example, for sentiment classification task where $\bm x$ = ``This computer is easy to use.'', the reformulated form of input may be $\bm x'$ =``$[ \bm x ]$, so it is a [MASK] product.''. However, discrete prompt tuning involves huge manual efforts and performs worse on high-resource dataset, limiting its usage. Instead of inserting discrete tokens, continuous prompt tuning~\cite{power_prompt,li-liang-2021-prefix} adds task-specific trainable vectors to the embeddings of input sentences. During training, only these task-specific vectors are optimized. Their experiments show that it is possible to elicit knowledge from frozen PLMs with these extra vectors. &#10;&#10;Different from discrete prompt tuning which aims to take advantage of the masked language model nature by inserting explicit tokens, &#10;~\cite{power_prompt} find the meaning of inserted vectors is hard to interpret indicating that continuous prompt tuning is completely different from discrete prompt tuning. &#10;Also,&#10;as shown in Figure~\ref{fig:prompt_tuning}, similar with P-tuning ~\cite{power_prompt} which prepends trainable vectors to the word embeddings,&#10;we attempt to prepend these vectors to a specific transformer layer. Figure~\ref{fig:layer_acc} depicts the accuracy of four tasks. We can see that for each task similar results can be achieved by prepending vectors to any layer, even the final layer ($12_{th}$ layer). Trainable vectors only participate in computations in the final layer if they are prepended to the final layer but thus configuration can still obtain a good performance. &#10;Does this phenomenon imply that these raw hidden states have already contained most if not all information for classification?&#10;&#10;In this paper, we aim to show that information for many classification tasks have already been encoded in the raw hidden states of inputs. To this end, &#10;we &#10;propose a simple method which only introduces three trainable vectors to integrate raw hidden states. &#10;This method firstly obtains all the hidden states of the input and learns a softmax-normalised weight vector to stack hidden states across layers. And secondly, a soft mask vector is used to select a subset of dimensions useful for downstream tasks in the hidden states. Finally, a self-attention operation only composed of one trainable vector is conducted to output the states for classification.&#10;We find that the classification information is on the surface which means that there is no need to use complex classification head like LSTM~\cite{HochSchm97}, CNN. Our classification head only contains a linear transformation followed by a softmax function.&#10;&#10;We conduct experiments on a variety of tasks &#10;and the results show that our method can attain performance comparable with that achieved by P-tuning~\cite{power_prompt} and P-tuning v2~\cite{liu2021ptuning}. &#10;This substantiate that original hidden states do contain necessary information for classification and this kind of information can be extracted with simple linear transformations. This finding can also partially explain why p-tuning (v2) works for classification tasks by only introducing a few trainable vectors.&#10;We hope this finding can help advance the understanding of continuous prompt tuning schemes.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Tuning, Computer Science, Linguistics, Parameter Efficiency, Artificial Intelligence, Mathematics, Classification Methods" />
        </attvalues>
      </node>
      <node id="1907.04944" label="1907.04944">
        <attvalues>
          <attvalue for="0" value="Can Unconditional Language Models Recover Arbitrary Sentences?" />
          <attvalue for="1" value="  Neural network-based generative language models like ELMo and BERT can work&#10;effectively as general purpose sentence encoders in text classification without&#10;further fine-tuning. Is it possible to adapt them in a similar way for use as&#10;general-purpose decoders? For this to be possible, it would need to be the case&#10;that for any target sentence of interest, there is some continuous&#10;representation that can be passed to the language model to cause it to&#10;reproduce that sentence. We set aside the difficult problem of designing an&#10;encoder that can produce such representations and, instead, ask directly&#10;whether such representations exist at all. To do this, we introduce a pair of&#10;effective, complementary methods for feeding representations into pretrained&#10;unconditional language models and a corresponding set of methods to map&#10;sentences into and out of this representation space, the reparametrized&#10;sentence space. We then investigate the conditions under which a language model&#10;can be made to generate a sentence through the identification of a point in&#10;such a space and find that it is possible to recover arbitrary sentences nearly&#10;perfectly with language models and representations of moderate size without&#10;modifying any model parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Latent Variable Recurrent Language Models&#10;The way we describe the sentence space of a language model can be thought of as performing inference over an implicit latent variable $z$ using a fixed decoder $\theta$.&#10;This resembles prior work on sparse coding~\cite{olshausen1997sparse} and generative latent optimization~\cite{bojanowski2018optimizing}.&#10;Under this lens, it also relates to work on training latent variable language models, such as models based on variational autoencoders by \cite{bowman2016generating} and sequence generative adversarial networks by \cite{yu2017seqgan}.&#10;The goal of identifying the smallest dimension of the sentence space for a specific target recoverability resembles work looking at continuous bag-of-words representations by \cite{mu-etal-2017-representing}. &#10;Our approach differs from these approaches in that we focus entirely on analyzing a fixed model that was trained unconditionally.&#10;Our formulation of the sentence space also is more general, and potentially applies to all of these models.&#10;&#10;Pretrained Recurrent Language Models&#10;&#10;Pretrained or separately trained language models have largely been used in two contexts: as a feature extractor for downstream tasks and as a scoring function for a task-specific decoder~\cite{gulcehre2015using,li2016diversity,Sriram2018ColdFT}.&#10;None of the above analyze how a pretrained model represents sentences nor investigate the potential of using a language model as a decoder.&#10;The work by \cite{Zoph2016TransferLF} transfers a pretrained language model, as a part of a neural machine translation system, to another language pair and fine-tunes. &#10;The positive result here is specific to machine translation as a downstream task, unlike the proposed framework, which is general and downstream task independent.&#10;Recently, there has been more work in pretraining the decoder using BERT~\cite{Devlin2018BERTPO} for neural machine translation and abstractive summarization~\cite{edunov2019pre, lample2019cross, song2019mass}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.12280" label="2402.12280">
        <attvalues>
          <attvalue for="0" value="Adaptive Skeleton Graph Decoding" />
          <attvalue for="1" value="  Large language models (LLMs) have seen significant adoption for natural&#10;language tasks, owing their success to massive numbers of model parameters&#10;(e.g., 70B+); however, LLM inference incurs significant computation and memory&#10;costs. Recent approaches propose parallel decoding strategies, such as&#10;Skeleton-of-Thought (SoT), to improve performance by breaking prompts down into&#10;sub-problems that can be decoded in parallel; however, they often suffer from&#10;reduced response quality. Our key insight is that we can request additional&#10;information, specifically dependencies and difficulty, when generating the&#10;sub-problems to improve both response quality and performance. In this paper,&#10;we propose Skeleton Graph Decoding (SGD), which uses dependencies exposed&#10;between sub-problems to support information forwarding between dependent&#10;sub-problems for improved quality while exposing parallelization opportunities&#10;for decoding independent sub-problems. Additionally, we leverage difficulty&#10;estimates for each sub-problem to select an appropriately-sized model,&#10;improving performance without significantly reducing quality. Compared to&#10;standard autoregressive generation and SoT, SGD achieves a 1.69x speedup while&#10;improving quality by up to 51%.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models~(LLMs)~\cite{touvron2023llama, touvron2023llama2, brown2020language} achieved great success in various natural language processing tasks, like summarization, translation, and chatbot systems.&#10;One key factor contributing to the remarkable success of LLMs is the huge number of model parameters, which provides enough capacity for LLMs to tackle complex tasks effectively.&#10;&#10;However, as LLMs scale to billion-level parameters, the deployment of LLMs becomes very expensive because of significant computational and memory costs during inference.&#10;The expensive inference cost largely affects the user experience in many latency-sensitive applications and prevents LLMs from being more accessible to more general communities.&#10;One of the reasons behind this high inference overhead is that LLMs are highly memory bandwidth-bounded~\cite{liu2023deja, zheng2024learn}. &#10;This is primarily due to their autoregressive generation process, which leads to low utilization rates of GPU computation units.&#10;&#10;Parallel decoding, including recent work on Skeleton of Thought (SoT)~\cite{ning2023skeleton}, provides a natural solution to improve LLM inference performance by batching token generation even for single prompts.&#10;SoT formulates the original problem into several sub-problems that are treated as independent inputs (i.e., the skeleton); since all sub-problems are processed in parallel, it avoids high end-to-end latency for sequential generation of all sub-problems.&#10;Unfortunately, the independence assumption is quite strong and doesn't hold in practice for all inputs, reducing performance on tasks requiring strong logic, as pointed out in~\cite{ning2023skeleton}~(e.g., math problems).&#10;For instance, if we look at Figure~\ref{fig:motivating-example}, we can see that the SoT on the left produces six sub-problems where only the first two are independent of all others.&#10;&#10;In this paper, instead of treating all sub-problems as independent nodes, we propose Skeleton Graph Decoding~(SGD) to organize the sub-problem nodes into a Directed Acyclic Graph~(DAG) that better represents the casual dependency among the sub-problems. &#10;As shown in Figure~\ref{fig:system-workflow}, SGD first uses an LLM to decompose the original prompt into logically-dependent sub-problems.&#10;Afterwards, SGD can construct the DAG and use it to batch different sub-problems (nodes) that do not have any unfinished preceding nodes: once the preceding nodes of a node are all completed, this node will be added to the generation batch (e.g., nodes 2 and 3 for node 5).&#10;This approach not only parallelizes the decoding progress but also guarantees dependency among sub-problems.&#10;Returning to \ref{fig:motivating-example}, we can observe the differences between SoT and SGD when handling the mathematical question.&#10;While later nodes in SoT cannot access necessary information from previous nodes (e.g., 3 from 1 and 2), SGD takes these dependencies into account to enable the generation of a correct answer.&#10;Overall, our skeleton graph structure not only improves response quality but also exposes parallelization opportunities to the system to improve performance.&#10;&#10;Building on our proposed skeleton graph structure, we explore other mechanisms that can be used to take advantage of trade-offs between response quality and performance.&#10;We observe that the generation of sub-problems reveals varying levels of difficulty; for instance, some fact-based sub-problems are comparatively simple and can be effectively addressed by smaller models, while complex sub-problems, such as those involving logic, necessitate employing larger models.&#10;Therefore, we propose an adaptive model selection mechanism that leverages estimates of difficulty produced during the skeleton graph generation.&#10;Using two degrees of difficulty (i.e., easy and hard), we can offload easy sub-problems to smaller models to reduce the generation latency without significantly compromising the overall quality of the response. If such nodes are on the critical path (The critical path is the entire path through the graph that will take the longest amount of time to complete) of the system, then the end-to-end generation latency could be further decreased.&#10;&#10;To summarize, we have the following contributions:&#10;\begin{itemize}[topsep=2.5pt, partopsep=0pt, leftmargin=*]&#10;&#10;\item We propose SGD, a new parallel decoding method on a graph that considers the causal dependency between nodes to enhance both generation quality and efficiency.&#10; \item We design an adaptive model selection mechanism to adaptively assign models based on node difficulty, further improve generation throughput.&#10; \item We conduct extensive experiments to evaluate SGD across four LLMs. Our results show that SGD achieves up to 1.69x speed-up while improving answer quality by up to 51.3\%. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Parallel Decoding Strategies, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.17157" label="2310.17157">
        <attvalues>
          <attvalue for="0" value="Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time" />
          <attvalue for="1" value="  Large language models (LLMs) with hundreds of billions of parameters have&#10;sparked a new wave of exciting AI applications. However, they are&#10;computationally expensive at inference time. Sparsity is a natural approach to&#10;reduce this cost, but existing methods either require costly retraining, have&#10;to forgo LLM's in-context learning ability, or do not yield wall-clock time&#10;speedup on modern hardware. We hypothesize that contextual sparsity, which are&#10;small, input-dependent sets of attention heads and MLP parameters that yield&#10;approximately the same output as the dense model for a given input, can address&#10;these issues. We show that contextual sparsity exists, that it can be&#10;accurately predicted, and that we can exploit it to speed up LLM inference in&#10;wall-clock time without compromising LLM's quality or in-context learning&#10;ability. Based on these insights, we propose DejaVu, a system that uses a&#10;low-cost algorithm to predict contextual sparsity on the fly given inputs to&#10;each layer, along with an asynchronous and hardware-aware implementation that&#10;speeds up LLM inference. We validate that DejaVu can reduce the inference&#10;latency of OPT-175B by over 2X compared to the state-of-the-art&#10;FasterTransformer, and over 6X compared to the widely used Hugging Face&#10;implementation, without compromising model quality. The code is available at&#10;https://github.com/FMInference/DejaVu.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.16084" label="2403.16084">
        <attvalues>
          <attvalue for="0" value="Argument Quality Assessment in the Age of Instruction-Following Large&#10;  Language Models" />
          <attvalue for="1" value="  The computational treatment of arguments on controversial issues has been&#10;subject to extensive NLP research, due to its envisioned impact on opinion&#10;formation, decision making, writing education, and the like. A critical task in&#10;any such application is the assessment of an argument's quality - but it is&#10;also particularly challenging. In this position paper, we start from a brief&#10;survey of argument quality research, where we identify the diversity of quality&#10;notions and the subjectiveness of their perception as the main hurdles towards&#10;substantial progress on argument quality assessment. We argue that the&#10;capabilities of instruction-following large language models (LLMs) to leverage&#10;knowledge across contexts enable a much more reliable assessment. Rather than&#10;just fine-tuning LLMs towards leaderboard chasing on assessment tasks, they&#10;need to be instructed systematically with argumentation theories and scenarios&#10;as well as with ways to solve argument-related problems. We discuss the&#10;real-world opportunities and ethical issues emerging thereby.&#10;" />
          <attvalue for="2" value="&#10;&#10;``In some sense, the question about the quality of an argument is the `ultimate' one for argumentation mining.'' \cite{stede:2018}.&#10;&#10;When learning about controversial issues, people rarely accept arguments they encounter without further contemplation. Rather, they seek to find the best arguments; those that help them form an opinion or write texts that persuade others; those that make them reach agreement or at least understand each other better. That is to say, argument quality is of interest as soon as arguments are presented to an audience. Computational argumentation aids the treatment of arguments at a larger scale, with important applications in search \cite{wachsmuth:2017e}, business \cite{slonim:2021}, and education \cite{wambsganss:2022}. But the situation there is the same: It is not enough to mine or generate arguments; their quality also needs to be evaluable \cite{park:2018}, so that it can be assessed \cite{lauscher:2020}, flaws can be found \cite{goffredo:2022}, and accounted for \cite{skitalinskaya:2023b}.&#10;&#10;\cite{wachsmuth:2017b} surveyed research on argument quality assessment, organizing theories and methods under 15 quality notions, from logical cogency to rhetorical effectiveness to dialectical reasonableness. Even though computational argumentation was just gaining momentum in natural language processing (NLP) back then, rarely going beyond argument mining, two inherent challenges of argument quality were visible already: the diversity of quality notions as well as the subjectivity of their perception and, hence, of their assessment for both humans and computational models. Consider the following argumentative claim against censoring Mark Twain's usage of the N-word, taken from the debate platform kialo.com:&#10;&#10;``In Huckleberry Finn, Twain captured the essence of everyday midwest American English.&quot;&#10;&#10;This claim is certainly relevant to the discussion, but whether people will deem it effective may strongly depend on their individual context. A person without African-American background may be willing to accept the argument; one with high literacy might look for clearer logical connections.&#10;&#10;While the challenges of diversity and subjectivity prevail until today \cite{lapesa:2023}, NLP is now seeing a revolutionary breakthrough: the rise of instruction-following large language models (henceforth, LLMs) that can tackle various NLP tasks with little to no task-specific fine-tuning, enabled by their supreme capability to integrate and leverage knowledge across contexts \cite{openai:2023}. The question is: What are the implications for argument quality assessment specifically as well as for computational argumentation in general?&#10;&#10;In this position paper, we revisit the computational assessment of argument quality in light of the availability of LLMs such as GPT-4 and Alpaca \cite{taori:2023}. Starting from the status quo reported by \cite{wachsmuth:2017b}, we carry out a brief survey of recent NLP research on the topic (Section~\ref{sec:relatedwork}). To bring order into the various lines of research pursued since 2017, we organize them into three general directions, as laid out in~Figure~\ref{quality-research.pdf}: &#10;\begin{itemize}&#10;&#10;\item&#10;Conceptual notions of maximal and minimal argument quality, &#10;\item&#10;Influence factors of argument quality from the context where arguments occur, and&#10;\item&#10;Computational models for assessing or improving argument quality.&#10;\end{itemize}&#10;&#10;On this basis, we establish the central question to which we provide answers in this paper:&#10;\begin{quote}&#10;How to drive research on LLM-based argument quality assessment in order to face the prevailing challenges of diverse quality notions and their subjectivity?&#10;\end{quote}&#10;&#10;In particular, we are convinced that the capabilities of instruction-following LLMs enable research to overcome many aspects of the two challenges. To this end, the primary focus of NLP research on argument quality should be put on systematic ways to teach LLMs to follow instructions, including concepts and settings of arguing in addition to ways to solve argument-related problems (Section~\ref{sec:model}). Instead of fine-tuning LLMs on predefined domains (manifested in the training data) and preselected theories (manifested in the data's annotations), as well as simple engineering of prompts, we expect the greatest impact to lie in teaching LLMs the theories, circumstances, and ethical constraints to adhere to. The rationale behind this is that LLMs will often have processed data from all contexts needed to make an informed judgment about an argument's quality, due to their heavy pretraining on huge amounts of data. In contrast, LLMs cannot access, by default, the knowledge of what is to be prioritized in a given setting. &#10;&#10;We state upfront that the blueprint delineated in this paper comes with several limitations and ethical considerations that we critically analyze below. Moreover, we are naturally aware of the general issues of LLMs, including hallucinated facts and the reproduction of common social biases. These issues deserve treatment in computational argumentation as well; they are even particularly critical there due to the sensitivity of many controversial topics \cite{holtermann:2022}. Keeping this in mind, we believe that it is necessary to explore now how to best employ LLMs for argument quality assessment in order to harness their full potential for the main applications, while avoiding to waste energy for the typical pursuit of leaderboard rankings on existing quality assessment tasks. This is the goal of the paper at hand.&#10;&#10;Now, why is it important to discuss LLMs for argument quality assessment specifically? We address this matter when we look at the real-world opportunities emerging from the capabilities of LLMs in academia and industry (Section~\ref{sec:analysis}). While \newcite{argyle:2023} developed LLMs that tone down argumentative conversations, we postulate a contrary path: Exploiting the means of LLMs to proactively enable people to learn and better reason about controversial issues, thus contributing towards more deliberate conversations \cite{vecchi:2021}. We think that the time has come to revisit and pursue the core visions of computational argumentation research, from the overcoming of filter bubbles to the individualized mass education of learners. We sketch how these visions could be realized with the LLMs available today, before we conclude (Section~\ref{sec:conclusion}) and stress ethical concerns that arise with LLMs that actively affect human views (Section~\ref{sec:limitations}).\,\,&#10;&#10;With the discussion in this position paper, we provide two main contributions to research:&#10;\begin{enumerate}&#10;&#10;\item&#10;A survey of the main lines of recent research on argument quality and its assessment&#10;\item&#10;A blueprint for impactful future research on LLMs for argument quality assessment &#10;\end{enumerate}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Argument Quality Assessment, Artificial Intelligence, Philosophy, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2212.08913" label="2212.08913">
        <attvalues>
          <attvalue for="0" value="Claim Optimization in Computational Argumentation" />
          <attvalue for="1" value="  An optimal delivery of arguments is key to persuasion in any debate, both for&#10;humans and for AI systems. This requires the use of clear and fluent claims&#10;relevant to the given debate. Prior work has studied the automatic assessment&#10;of argument quality extensively. Yet, no approach actually improves the quality&#10;so far. To fill this gap, this paper proposes the task of claim optimization:&#10;to rewrite argumentative claims in order to optimize their delivery. As&#10;multiple types of optimization are possible, we approach this task by first&#10;generating a diverse set of candidate claims using a large language model, such&#10;as BART, taking into account contextual information. Then, the best candidate&#10;is selected using various quality metrics. In automatic and human evaluation on&#10;an English-language corpus, our quality-based candidate selection outperforms&#10;several baselines, improving 60% of all claims (worsening 16% only). Follow-up&#10;analyses reveal that, beyond copy editing, our approach often specifies claims&#10;with details, whereas it adds less evidence than humans do. Moreover, its&#10;capabilities generalize well to other domains, such as instructional texts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.04026" label="2204.04026">
        <attvalues>
          <attvalue for="0" value="Fair and Argumentative Language Modeling for Computational Argumentation" />
          <attvalue for="1" value="  Although much work in NLP has focused on measuring and mitigating&#10;stereotypical bias in semantic spaces, research addressing bias in&#10;computational argumentation is still in its infancy. In this paper, we address&#10;this research gap and conduct a thorough investigation of bias in argumentative&#10;language models. To this end, we introduce ABBA, a novel resource for bias&#10;measurement specifically tailored to argumentation. We employ our resource to&#10;assess the effect of argumentative fine-tuning and debiasing on the intrinsic&#10;bias found in transformer-based language models using a lightweight&#10;adapter-based approach that is more sustainable and parameter-efficient than&#10;full fine-tuning. Finally, we analyze the potential impact of language model&#10;debiasing on the performance in argument quality prediction, a downstream task&#10;of computational argumentation. Our results show that we are able to&#10;successfully and sustainably remove bias in general and argumentative language&#10;models while preserving (and sometimes improving) model performance in&#10;downstream tasks. We make all experimental code and data available at&#10;https://github.com/umanlp/FairArgumentativeLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.05255" label="2005.05255">
        <attvalues>
          <attvalue for="0" value="Toward Better Storylines with Sentence-Level Language Models" />
          <attvalue for="1" value="  We propose a sentence-level language model which selects the next sentence in&#10;a story from a finite set of fluent alternatives. Since it does not need to&#10;model fluency, the sentence-level language model can focus on longer range&#10;dependencies, which are crucial for multi-sentence coherence. Rather than&#10;dealing with individual words, our method treats the story so far as a list of&#10;pre-trained sentence embeddings and predicts an embedding for the next&#10;sentence, which is more efficient than predicting word embeddings. Notably this&#10;allows us to consider a large number of candidates for the next sentence during&#10;training. We demonstrate the effectiveness of our approach with&#10;state-of-the-art accuracy on the unsupervised Story Cloze task and with&#10;promising results on larger-scale next sentence prediction tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Computer generation of stories and other kinds of creative writing is a challenging endeavor. It entangles two difficult tasks: the generation of fluent natural language and the generation of a coherent storyline. In the recent year, neural language models have made tremendous progress with respect to fluency~\cite{bahdanau2014neural,vaswani2017attention,bengio2003lm,devlin2019bert}, but coherency is still a major challenge~\cite{see:2019:storyteller}. The generation of coherent stories has recently been addressed with additional conditioning: \cite{fan2018hierarchical} suggest conditioning on a story prompt, \cite{clark2018creative} propose collaboration between a generative model and a human writer, and \cite{guan2019commonsense} suggest attending to a commonsense graph relevant to the story plot. Conditioning based on a generated story plan \cite{martin2018event,fan2019strategies,yao2019plan}, a sequence of images \cite{chandu2019storyboarding} or character roles \cite{liu2020character} have also been considered.&#10;&#10;Our work is orthogonal to these efforts. Rather than considering additional conditioning, we propose a model which takes as input several sentences of context and selects the best next sentence within a large set of fluent candidate sentences.&#10;We leverage pre-trained BERT embeddings \cite{devlin2019bert} to build this sentence-level language model.&#10;Given the embeddings of the previous sentences of the story, our model learns to predict a likely embedding of the next sentence.&#10;&#10;This task isolates the modeling of long-range dependencies from the prediction of individual words, which has several advantages.&#10;First, since our model only needs to determine how well each candidate sentence would fit as a coherent continuation to the story, it does not spend capacity and time to learn fluency.&#10;Second, our model does not manipulate individual words but full sentences, which allows us to consider tens of thousands of candidate sentences at a time.&#10;This contrasts with prior work \cite{logeswaran2018an} where the need to learn token-level representations limited the number of candidate next sentences that could be considered to a few hundred.&#10;Third, we can rely on compact model architectures that train quickly because we take advantage of strong semantic representation from a pre-trained bidirectional language model, BERT, as our sentence embeddings.&#10;Of course, these benefits also imply that our sentence representation is limited to the information extracted by the pre-trained model.&#10;Nevertheless, we show that our model achieves state-of-the-art accuracy among unsupervised approaches on the Story Cloze task: predicting which of two sentences coherently ends a short story.&#10;&#10;Our work also opens up the possibility of ranking thousands of candidate sentences from a large literature repository.&#10;On the ROC Stories dataset, we observe that training with a large number of candidates is key for selecting the most coherent ending among a large set of candidates at test time.&#10;We also show preliminary results on the efficacy of our method for ranking candidate next sentence on the Toronto Book Corpus~\cite{kiros2015skip}, a much larger book dataset.&#10;We envision that our methods for scoring many candidate next sentences by their coherence with the context might be useful to downstream generation tasks where it is possible to generate many fluent continuations of a text, but it remains an unsolved problem how to refine and choose the best of them. To encourage this exploration, we release our code and models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Story Generation, Linguistics, Artificial Intelligence, Sentence Embeddings, Natural Language Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="1805.04833" label="1805.04833">
        <attvalues>
          <attvalue for="0" value="Hierarchical Neural Story Generation" />
          <attvalue for="1" value="  We explore story generation: creative systems that can build coherent and&#10;fluent passages of text about a topic. We collect a large dataset of 300K&#10;human-written stories paired with writing prompts from an online forum. Our&#10;dataset enables hierarchical story generation, where the model first generates&#10;a premise, and then transforms it into a passage of text. We gain further&#10;improvements with a novel form of model fusion that improves the relevance of&#10;the story to the prompt, and adding a new gated multi-scale self-attention&#10;mechanism to model long-range context. Experiments show large improvements over&#10;strong baselines on both automated and human evaluations. Human judges prefer&#10;stories generated by our approach to those from a strong non-hierarchical model&#10;by a factor of two to one.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.10113" label="1808.10113">
        <attvalues>
          <attvalue for="0" value="Story Ending Generation with Incremental Encoding and Commonsense&#10;  Knowledge" />
          <attvalue for="1" value="  Generating a reasonable ending for a given story context, i.e., story ending&#10;generation, is a strong indication of story comprehension. This task requires&#10;not only to understand the context clues which play an important role in&#10;planning the plot but also to handle implicit knowledge to make a reasonable,&#10;coherent story.&#10;  In this paper, we devise a novel model for story ending generation. The model&#10;adopts an incremental encoding scheme to represent context clues which are&#10;spanning in the story context. In addition, commonsense knowledge is applied&#10;through multi-source attention to facilitate story comprehension, and thus to&#10;help generate coherent and reasonable endings. Through building context clues&#10;and using implicit knowledge, the model is able to produce reasonable story&#10;endings. context clues implied in the post and make the inference based on it.&#10;  Automatic and manual evaluation shows that our model can generate more&#10;reasonable story endings than state-of-the-art baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.01331" label="1706.01331">
        <attvalues>
          <attvalue for="0" value="Event Representations for Automated Story Generation with Deep Neural&#10;  Nets" />
          <attvalue for="1" value="  Automated story generation is the problem of automatically selecting a&#10;sequence of events, actions, or words that can be told as a story. We seek to&#10;develop a system that can generate stories by learning everything it needs to&#10;know from textual story corpora. To date, recurrent neural networks that learn&#10;language models at character, word, or sentence levels have had little success&#10;generating coherent stories. We explore the question of event representations&#10;that provide a mid-level of abstraction between words and sentences in order to&#10;retain the semantic information of the original data while minimizing event&#10;sparsity. We present a technique for preprocessing textual story data into&#10;event sequences. We then present a technique for automated story generation&#10;whereby we decompose the problem into the generation of successive events&#10;(event2event) and the generation of natural language sentences from events&#10;(event2sentence). We give empirical results comparing different event&#10;representations and their effects on event successor generation and the&#10;translation of events to natural language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.02893" label="1803.02893">
        <attvalues>
          <attvalue for="0" value="An efficient framework for learning sentence representations" />
          <attvalue for="1" value="  In this work we propose a simple and efficient framework for learning&#10;sentence representations from unlabelled data. Drawing inspiration from the&#10;distributional hypothesis and recent work on learning sentence representations,&#10;we reformulate the problem of predicting the context in which a sentence&#10;appears as a classification problem. Given a sentence and its context, a&#10;classifier distinguishes context sentences from other contrastive sentences&#10;based on their vector representations. This allows us to efficiently learn&#10;different types of encoding functions, and we show that the model learns&#10;high-quality sentence representations. We demonstrate that our sentence&#10;representations outperform state-of-the-art unsupervised and supervised&#10;representation learning methods on several downstream NLP tasks that involve&#10;understanding sentence semantics while achieving an order of magnitude speedup&#10;in training time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10511" label="2212.10511">
        <attvalues>
          <attvalue for="0" value="When Not to Trust Language Models: Investigating Effectiveness of&#10;  Parametric and Non-Parametric Memories" />
          <attvalue for="1" value="  Despite their impressive performance on diverse tasks, large language models&#10;(LMs) still struggle with tasks requiring rich world knowledge, implying the&#10;limitations of relying solely on their parameters to encode a wealth of world&#10;knowledge. This paper aims to understand LMs' strengths and limitations in&#10;memorizing factual knowledge, by conducting large-scale knowledge probing&#10;experiments of 10 models and 4 augmentation methods on PopQA, our new&#10;open-domain QA dataset with 14k questions. We find that LMs struggle with less&#10;popular factual knowledge, and that scaling fails to appreciably improve&#10;memorization of factual knowledge in the long tail. We then show that&#10;retrieval-augmented LMs largely outperform orders of magnitude larger LMs,&#10;while unassisted LMs remain competitive in questions about high-popularity&#10;entities. Based on those findings, we devise a simple, yet effective, method&#10;for powerful and efficient retrieval-augmented LMs, which retrieves&#10;non-parametric memories only when necessary. Experimental results show that&#10;this significantly improves models' performance while reducing the inference&#10;costs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.05994" label="2211.05994">
        <attvalues>
          <attvalue for="0" value="A Survey of Knowledge Enhanced Pre-trained Language Models" />
          <attvalue for="1" value="  Pre-trained Language Models (PLMs) which are trained on large text corpus via&#10;self-supervised learning method, have yielded promising performance on various&#10;tasks in Natural Language Processing (NLP). However, though PLMs with huge&#10;parameters can effectively possess rich knowledge learned from massive training&#10;text and benefit downstream tasks at the fine-tuning stage, they still have&#10;some limitations such as poor reasoning ability due to the lack of external&#10;knowledge. Research has been dedicated to incorporating knowledge into PLMs to&#10;tackle these issues. In this paper, we present a comprehensive review of&#10;Knowledge Enhanced Pre-trained Language Models (KE-PLMs) to provide a clear&#10;insight into this thriving field. We introduce appropriate taxonomies&#10;respectively for Natural Language Understanding (NLU) and Natural Language&#10;Generation (NLG) to highlight these two main tasks of NLP. For NLU, we divide&#10;the types of knowledge into four categories: linguistic knowledge, text&#10;knowledge, knowledge graph (KG), and rule knowledge. The KE-PLMs for NLG are&#10;categorized into KG-based and retrieval-based methods. Finally, we point out&#10;some promising future directions of KE-PLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.04389" label="2010.04389">
        <attvalues>
          <attvalue for="0" value="A Survey of Knowledge-Enhanced Text Generation" />
          <attvalue for="1" value="  The goal of text generation is to make machines express in human language. It&#10;is one of the most important yet challenging tasks in natural language&#10;processing (NLP). Since 2014, various neural encoder-decoder models pioneered&#10;by Seq2Seq have been proposed to achieve the goal by learning to map input text&#10;to output text. However, the input text alone often provides limited knowledge&#10;to generate the desired output, so the performance of text generation is still&#10;far from satisfaction in many real-world scenarios. To address this issue,&#10;researchers have considered incorporating various forms of knowledge beyond the&#10;input text into the generation models. This research direction is known as&#10;knowledge-enhanced text generation. In this survey, we present a comprehensive&#10;review of the research on knowledge enhanced text generation over the past five&#10;years. The main content includes two parts: (i) general methods and&#10;architectures for integrating knowledge into text generation; (ii) specific&#10;techniques and applications according to different forms of knowledge data.&#10;This survey can have broad audiences, researchers and practitioners, in&#10;academia and industry.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04413" label="2109.04413">
        <attvalues>
          <attvalue for="0" value="AStitchInLanguageModels: Dataset and Methods for the Exploration of&#10;  Idiomaticity in Pre-Trained Language Models" />
          <attvalue for="1" value="  Despite their success in a variety of NLP tasks, pre-trained language models,&#10;due to their heavy reliance on compositionality, fail in effectively capturing&#10;the meanings of multiword expressions (MWEs), especially idioms. Therefore,&#10;datasets and methods to improve the representation of MWEs are urgently needed.&#10;Existing datasets are limited to providing the degree of idiomaticity of&#10;expressions along with the literal and, where applicable, (a single)&#10;non-literal interpretation of MWEs. This work presents a novel dataset of&#10;naturally occurring sentences containing MWEs manually classified into a&#10;fine-grained set of meanings, spanning both English and Portuguese. We use this&#10;dataset in two tasks designed to test i) a language model's ability to detect&#10;idiom usage, and ii) the effectiveness of a language model in generating&#10;representations of sentences containing idioms. Our experiments demonstrate&#10;that, on the task of detecting idiomatic usage, these models perform reasonably&#10;well in the one-shot and few-shot scenarios, but that there is significant&#10;scope for improvement in the zero-shot scenario. On the task of representing&#10;idiomaticity, we find that pre-training is not always effective, while&#10;fine-tuning could provide a sample efficient method of learning representations&#10;of sentences containing MWEs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The problems posed by MWEs to NLP models have been known for some time~\cite{10.5555/647344.724004,constant-etal-2017-survey,shwartz-dagan-2019-still}. For instance,&#10;~\newcite{10.5555/647344.724004} refer to the idiomaticity problem and place the need for effective processing of MWEs on par with that for word sense disambiguation to be able to effectively process text. While their analysis focused on symbolic&#10;methods, this problem still persists:&#10;\newcite{shwartz-dagan-2019-still} showed, using six tasks, that contextual pre-trained language models, capable of handling polysemy, continued to be unable to effectively handle idiomatic MWEs, although they tend to do better than their non-contextual predecessors. Further experiments with probing pre-trained language models across multiple languages have also confirmed this result~\cite{yu-ettinger-2020-assessing,garcia-etal-2021-probing}.&#10;&#10;" />
          <attvalue for="4" value="Idiom Representation, Computer Science, Linguistics, Cognitive Science, Language Model Limitations, Artificial Intelligence, Natural Language Processing, Multilingual NLP" />
        </attvalues>
      </node>
      <node id="2406.18501" label="2406.18501">
        <attvalues>
          <attvalue for="0" value="Is In-Context Learning a Type of Gradient-Based Learning? Evidence from&#10;  the Inverse Frequency Effect in Structural Priming" />
          <attvalue for="1" value="  Large language models (LLMs) have shown the emergent capability of in-context&#10;learning (ICL). One line of research has explained ICL as functionally&#10;performing gradient descent. In this paper, we introduce a new way of&#10;diagnosing whether ICL is functionally equivalent to gradient-based learning.&#10;Our approach is based on the inverse frequency effect (IFE) -- a phenomenon in&#10;which an error-driven learner is expected to show larger updates when trained&#10;on infrequent examples than frequent ones. The IFE has previously been studied&#10;in psycholinguistics because humans show this effect in the context of&#10;structural priming (the tendency for people to produce sentence structures they&#10;have encountered recently); the IFE has been used as evidence that human&#10;structural priming must involve error-driven learning mechanisms. In our&#10;experiments, we simulated structural priming within ICL and found that LLMs&#10;display the IFE, with the effect being stronger in larger models. We conclude&#10;that ICL is indeed a type of gradient-based learning, supporting the hypothesis&#10;that a gradient component is implicitly computed in the forward pass during&#10;ICL. Our results suggest that both humans and LLMs make use of gradient-based,&#10;error-driven processing mechanisms.&#10;" />
          <attvalue for="2" value=" &#10;&#10;To what extent do humans and language models use similar processing mechanisms? This question is of interest to both Artificial Intelligence researchers and cognitive scientists. Language models and human learners have some substantial differences: human learners often display a flexible learning ability to adapt to new examples, while language models typically require massive amounts of training data and a large number of parameters to exhibit human-like performance. Yet recent pre-trained large language models (LLMs) have shown the capacity for in-context learning (ICL): they adapt to specific tasks with a few demonstration-answer pairs provided as prompts in the context window without any parameter updates \cite{brown2020language}. This intriguing, emergent capability could provide a way to bridge the divide between language models and human learners: perhaps ICL is a processing mechanism that, like humans, can flexibly adapt to new demands. &#10;&#10;Among the body of research into the sources and interpretations of the ICL capability, one line of work has aimed to deepen the theoretical understanding of ICL by offering functional interpretations of ICL as a kind of implicit gradient descent during inference. \cite{garg2022can}, \cite{zhang2023trained}, and \cite{ahn2024transformers} have shown that standard Transformers \cite{vaswani2017attention} can be trained to implement learning algorithms for linear regressions under ICL-based training objectives. \cite{pmlr-v202-von-oswald23a} have demonstrated that Transformer models, with appropriate choices of parameters, can process in-context demonstrations in a way that is functionally equivalent to performing gradient updates on the same demonstration examples. \cite{dai2023metaoptimization} provided a mathematical construction showing the dual form between Transformer attention and gradient descent and interpreted ICL as a meta-optimization process that performs implicit fine-tuning. However, \cite{shen2023pretrained} observe that the importance of these theoretical demonstrations is limited in that they do not take ICL to be an emergent property, instead assuming a training objective that optimizes for ICL. They therefore deviate from actual LLMs pre-trained with natural data with a language modeling training objective. Indeed, \cite{shen2023pretrained} found inconsistencies between ICL and gradient descent in real models, and left the equivalence between ICL and gradient descent an open question.&#10;&#10;In this paper, we aim to better characterize what kind of learning mechanism ICL is by drawing a connection between ICL and human learning mechanisms. Specifically, we examine the hypothesis that ICL functionally performs gradient-based fine-tuning (e.g., gradient descent) by empirically investigating a weaker claim with off-the-shelf LLMs evaluated with natural language data: whether ICL is a type of gradient-based---i.e., error-driven---learning such that a gradient component is implicitly computed during the forward pass. We approach this question by treating ICL as a processing mechanism of LLMs and borrowing insights from methods of studying processing mechanisms in humans: we examine to what extent LLMs show the inverse frequency effect (IFE), a phenomenon in the human structural priming paradigm \cite{branigan2017experimental} that has been argued to require a gradient-based processing mechanism in humans, namely implicit learning (e.g., \cite{chang2006becoming}). We study the linguistic phenomenon of the dative alternation and demonstrate that LLMs show a robust IFE under standard fine-tuning (which involves explicit gradient-based learning), and varying degrees of the IFE under the ICL setting, with larger models showing a stronger IFE. We conclude that, at least in the cases we studied, ICL is indeed a gradient-based learning mechanism.&#10;&#10;Our study has implications for both NLP/machine learning (\ref{ex1} and \ref{ex2}) and linguistically-motivated analysis of LLMs (\ref{ex3} and \ref{ex4}):&#10;&#10;\ex. We find evidence that ICL in off-the-shelf LLMs can be viewed as a form of gradient-based learning.&#10;&#10;\ex. We generalize the notion of ICL beyond the standardly assumed prompt format of input-output pairs, establishing a connection between priming and prompting.&#10;&#10;\ex. We show that LLMs qualitatively display an important property of human language processing, namely the IFE in structural priming.&#10;&#10;\ex. While most human-LLM comparisons focus on representations, our experiments go one step further by analyzing the processing mechanisms used by LLMs.&#10;&#10;Overall, our results suggest that error-driven learning is an aspect of processing that is shared between humans and LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Human Learning Mechanisms, Linguistics, Cognitive Science, Psycholinguistics, Language Model Learning, Gradient-Based Optimization, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2208.01066" label="2208.01066">
        <attvalues>
          <attvalue for="0" value="What Can Transformers Learn In-Context? A Case Study of Simple Function&#10;  Classes" />
          <attvalue for="1" value="  In-context learning refers to the ability of a model to condition on a prompt&#10;sequence consisting of in-context examples (input-output pairs corresponding to&#10;some task) along with a new query input, and generate the corresponding output.&#10;Crucially, in-context learning happens only at inference time without any&#10;parameter updates to the model. While large language models such as GPT-3&#10;exhibit some ability to perform in-context learning, it is unclear what the&#10;relationship is between tasks on which this succeeds and what is present in the&#10;training data. To make progress towards understanding in-context learning, we&#10;consider the well-defined problem of training a model to in-context learn a&#10;function class (e.g., linear functions): that is, given data derived from some&#10;functions in the class, can we train a model to in-context learn &quot;most&quot;&#10;functions from this class? We show empirically that standard Transformers can&#10;be trained from scratch to perform in-context learning of linear functions --&#10;that is, the trained model is able to learn unseen linear functions from&#10;in-context examples with performance comparable to the optimal least squares&#10;estimator. In fact, in-context learning is possible even under two forms of&#10;distribution shift: (i) between the training data of the model and&#10;inference-time prompts, and (ii) between the in-context examples and the query&#10;input during inference. We also show that we can train Transformers to&#10;in-context learn more complex function classes -- namely sparse linear&#10;functions, two-layer neural networks, and decision trees -- with performance&#10;that matches or exceeds task-specific learning algorithms. Our code and models&#10;are available at https://github.com/dtsip/in-context-learning .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.09927" label="2306.09927">
        <attvalues>
          <attvalue for="0" value="Trained Transformers Learn Linear Models In-Context" />
          <attvalue for="1" value="  Attention-based neural networks such as transformers have demonstrated a&#10;remarkable ability to exhibit in-context learning (ICL): Given a short prompt&#10;sequence of tokens from an unseen task, they can formulate relevant per-token&#10;and next-token predictions without any parameter updates. By embedding a&#10;sequence of labeled training data and unlabeled test data as a prompt, this&#10;allows for transformers to behave like supervised learning algorithms. Indeed,&#10;recent work has shown that when training transformer architectures over random&#10;instances of linear regression problems, these models' predictions mimic those&#10;of ordinary least squares.&#10;  Towards understanding the mechanisms underlying this phenomenon, we&#10;investigate the dynamics of ICL in transformers with a single linear&#10;self-attention layer trained by gradient flow on linear regression tasks. We&#10;show that despite non-convexity, gradient flow with a suitable random&#10;initialization finds a global minimum of the objective function. At this global&#10;minimum, when given a test prompt of labeled examples from a new prediction&#10;task, the transformer achieves prediction error competitive with the best&#10;linear predictor over the test prompt distribution. We additionally&#10;characterize the robustness of the trained transformer to a variety of&#10;distribution shifts and show that although a number of shifts are tolerated,&#10;shifts in the covariate distribution of the prompts are not. Motivated by this,&#10;we consider a generalized ICL setting where the covariate distributions can&#10;vary across prompts. We show that although gradient flow succeeds at finding a&#10;global minimum in this setting, the trained transformer is still brittle under&#10;mild covariate shifts. We complement this finding with experiments on large,&#10;nonlinear transformer architectures which we show are more robust under&#10;covariate shifts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.00297" label="2306.00297">
        <attvalues>
          <attvalue for="0" value="Transformers learn to implement preconditioned gradient descent for&#10;  in-context learning" />
          <attvalue for="1" value="  Several recent works demonstrate that transformers can implement algorithms&#10;like gradient descent. By a careful construction of weights, these works show&#10;that multiple layers of transformers are expressive enough to simulate&#10;iterations of gradient descent. Going beyond the question of expressivity, we&#10;ask: Can transformers learn to implement such algorithms by training over&#10;random problem instances? To our knowledge, we make the first theoretical&#10;progress on this question via an analysis of the loss landscape for linear&#10;transformers trained over random instances of linear regression. For a single&#10;attention layer, we prove the global minimum of the training objective&#10;implements a single iteration of preconditioned gradient descent. Notably, the&#10;preconditioning matrix not only adapts to the input distribution but also to&#10;the variance induced by data inadequacy. For a transformer with $L$ attention&#10;layers, we prove certain critical points of the training objective implement&#10;$L$ iterations of preconditioned gradient descent. Our results call for future&#10;theoretical studies on learning algorithms by training transformers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.07677" label="2212.07677">
        <attvalues>
          <attvalue for="0" value="Transformers learn in-context by gradient descent" />
          <attvalue for="1" value="  At present, the mechanisms of in-context learning in Transformers are not&#10;well understood and remain mostly an intuition. In this paper, we suggest that&#10;training Transformers on auto-regressive objectives is closely related to&#10;gradient-based meta-learning formulations. We start by providing a simple&#10;weight construction that shows the equivalence of data transformations induced&#10;by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a&#10;regression loss. Motivated by that construction, we show empirically that when&#10;training self-attention-only Transformers on simple regression tasks either the&#10;models learned by GD and Transformers show great similarity or, remarkably, the&#10;weights found by optimization match the construction. Thus we show how trained&#10;Transformers become mesa-optimizers i.e. learn models by gradient descent in&#10;their forward pass. This allows us, at least in the domain of regression&#10;problems, to mechanistically understand the inner workings of in-context&#10;learning in optimized Transformers. Building on this insight, we furthermore&#10;identify how Transformers surpass the performance of plain gradient descent by&#10;learning an iterative curvature correction and learn linear models on deep data&#10;representations to solve non-linear regression tasks. Finally, we discuss&#10;intriguing parallels to a mechanism identified to be crucial for in-context&#10;learning termed induction-head (Olsson et al., 2022) and show how it could be&#10;understood as a specific case of in-context learning by gradient descent&#10;learning within Transformers. Code to reproduce the experiments can be found at&#10;https://github.com/google-research/self-organising-systems/tree/master/transformers_learn_icl_by_gd .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.11114" label="2409.11114">
        <attvalues>
          <attvalue for="0" value="Diversity-grounded Channel Prototypical Learning for Out-of-Distribution&#10;  Intent Detection" />
          <attvalue for="1" value="  In the realm of task-oriented dialogue systems, a robust intent detection&#10;mechanism must effectively handle malformed utterances encountered in&#10;real-world scenarios. This study presents a novel fine-tuning framework for&#10;large language models (LLMs) aimed at enhancing in-distribution (ID) intent&#10;classification and out-of-distribution (OOD) intent detection, which utilizes&#10;semantic matching with prototypes derived from ID class names. By harnessing&#10;the highly distinguishable representations of LLMs, we construct semantic&#10;prototypes for each ID class using a diversity-grounded prompt tuning approach.&#10;We rigorously test our framework in a challenging OOD context, where ID and OOD&#10;classes are semantically close yet distinct, referred to as \emph{near} OOD&#10;detection. For a thorough assessment, we benchmark our method against the&#10;prevalent fine-tuning approaches. The experimental findings reveal that our&#10;method demonstrates superior performance in both few-shot ID intent&#10;classification and near-OOD intent detection tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Task-oriented conversational systems have become widespread across numerous scenarios, including banking, travel, and medical diagnosis~\cite{yan2017building,zhang2020recent, feng2024continual}, where they provide a variety of services. Intent recognition~\cite{zhang2022fine}, a key element of these systems, is crucial for enabling automated assistance to address customer needs. Given the wide range of user inquiries, out-of-distribution (OOD) intent detection~\cite{hendrycks2020pretrained, zhan2021out, uppaal2023fine, zhan2024vi} seeks to protect the intent recognition system by identifying and alerting to malformed inputs.&#10;&#10;Recent advancements in large language models (LLMs)~\cite{touvron2023llama,chiang2023vicuna,openai2023gpt4, feng2023towards} have significantly improved the detection of semantically distinct out-of-distribution (far-OOD) intents~\cite{liu2023good}. However, identifying semantically similar (near-OOD) intents continues to pose significant challenges~\cite{fort2021exploring, liu2023good}. Given its critical importance in practical applications, our study concentrates on near-OOD detection, specifically in a few-shot scenario~\cite{jiang2024few,zhang2022pre}.&#10;Particularly, in the few-shot learning context, the training model can receive only a few in-distribution (ID) examples per class. Obtaining discriminative information from limited ID examples for OOD detection is inherently challenging~\cite{zhang2020discriminative}. Previous strategies have involved using extra unlabeled data~\cite{wang2023app} or generating pseudo-ID examples~\cite{zhan2022closer} to increase training samples. Instead, we focus on leveraging the inner knowledge of LLMs as a supplement without dataset expansion.&#10;&#10;We propose a novel channel prototypical learning framework for few-shot near-OOD detection, shown in Figure~\ref{fig:framework}. &#10;It employs a set of learned class prototypes to conduct semantic matching for ID classification and OOD detection. To effectively leverage the pre-existing knowledge in LLMs and the limited ID examples, our framework involves using both the ID class names and utterances to develop the class prototypes. Drawing inspiration from channel models~\cite{min2021noisy, brown1993mathematics} and prompt tuning~\cite{lester2021power}, &#10;we feed into the LLMs the ID class names preceded by a series of learnable continuous prompt embeddings to generate a semantic prototype for each ID class.&#10;Furthermore, to ensure a wider variety of class representations—a factor known to improve OOD detection~\cite{winkens2020contrastive}—we adopt a diversified learning strategy aimed at reducing the mutual information between classes.&#10;&#10;For evaluation, we construct few-shot near-OOD intent detection tasks by sampling classes from the same domains in CLINC as ID and OOD. Since our method is a kind of fine-tuning framework, we compare it with widely used fine-tuning methods: generative tuning~\cite{liu2023good} and discriminative tuning. Overall, our diversity-grounded channel prototypical learning approach can bring better performance compared to these two paradigms.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Dialogue Systems, Artificial Intelligence, Natural Language Processing, Intent Detection" />
        </attvalues>
      </node>
      <node id="2205.07208" label="2205.07208">
        <attvalues>
          <attvalue for="0" value="Fine-tuning Pre-trained Language Models for Few-shot Intent Detection:&#10;  Supervised Pre-training and Isotropization" />
          <attvalue for="1" value="  It is challenging to train a good intent classifier for a task-oriented&#10;dialogue system with only a few annotations. Recent studies have shown that&#10;fine-tuning pre-trained language models with a small amount of labeled&#10;utterances from public benchmarks in a supervised manner is extremely helpful.&#10;However, we find that supervised pre-training yields an anisotropic feature&#10;space, which may suppress the expressive power of the semantic representations.&#10;Inspired by recent research in isotropization, we propose to improve supervised&#10;pre-training by regularizing the feature space towards isotropy. We propose two&#10;regularizers based on contrastive learning and correlation matrix respectively,&#10;and demonstrate their effectiveness through extensive experiments. Our main&#10;finding is that it is promising to regularize supervised pre-training with&#10;isotropization to further improve the performance of few-shot intent detection.&#10;The source code can be found at https://github.com/fanolabs/isoIntentBert-main.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.03004" label="2106.03004">
        <attvalues>
          <attvalue for="0" value="Exploring the Limits of Out-of-Distribution Detection" />
          <attvalue for="1" value="  Near out-of-distribution detection (OOD) is a major challenge for deep neural&#10;networks. We demonstrate that large-scale pre-trained transformers can&#10;significantly improve the state-of-the-art (SOTA) on a range of near OOD tasks&#10;across different data modalities. For instance, on CIFAR-100 vs CIFAR-10 OOD&#10;detection, we improve the AUROC from 85% (current SOTA) to more than 96% using&#10;Vision Transformers pre-trained on ImageNet-21k. On a challenging genomics OOD&#10;detection benchmark, we improve the AUROC from 66% to 77% using transformers&#10;and unsupervised pre-training. To further improve performance, we explore the&#10;few-shot outlier exposure setting where a few examples from outlier classes may&#10;be available; we show that pre-trained transformers are particularly&#10;well-suited for outlier exposure, and that the AUROC of OOD detection on&#10;CIFAR-100 vs CIFAR-10 can be improved to 98.7% with just 1 image per OOD class,&#10;and 99.46% with 10 images per OOD class. For multi-modal image-text pre-trained&#10;transformers such as CLIP, we explore a new way of using just the names of&#10;outlier classes as a sole source of information without any accompanying&#10;images, and show that this outperforms previous SOTA on standard vision OOD&#10;benchmark tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13009" label="2010.13009">
        <attvalues>
          <attvalue for="0" value="Discriminative Nearest Neighbor Few-Shot Intent Detection by&#10;  Transferring Natural Language Inference" />
          <attvalue for="1" value="  Intent detection is one of the core components of goal-oriented dialog&#10;systems, and detecting out-of-scope (OOS) intents is also a practically&#10;important skill. Few-shot learning is attracting much attention to mitigate&#10;data scarcity, but OOS detection becomes even more challenging. In this paper,&#10;we present a simple yet effective approach, discriminative nearest neighbor&#10;classification with deep self-attention. Unlike softmax classifiers, we&#10;leverage BERT-style pairwise encoding to train a binary classifier that&#10;estimates the best matched training example for a user input. We propose to&#10;boost the discriminative ability by transferring a natural language inference&#10;(NLI) model. Our extensive experiments on a large-scale multi-domain intent&#10;detection task show that our method achieves more stable and accurate in-domain&#10;and OOS detection accuracy than RoBERTa-based classifiers and embedding-based&#10;nearest neighbor approaches. More notably, the NLI transfer enables our 10-shot&#10;model to perform competitively with 50-shot or even full-shot classifiers,&#10;while we can keep the inference time constant by leveraging a faster embedding&#10;retrieval model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.04106" label="2108.04106">
        <attvalues>
          <attvalue for="0" value="Noisy Channel Language Model Prompting for Few-Shot Text Classification" />
          <attvalue for="1" value="  We introduce a noisy channel approach for language model prompting in&#10;few-shot text classification. Instead of computing the likelihood of the label&#10;given the input (referred as direct models), channel models compute the&#10;conditional probability of the input given the label, and are thereby required&#10;to explain every word in the input. We use channel models for recently proposed&#10;few-shot learning methods with no or very limited updates to the language model&#10;parameters, via either in-context demonstration or prompt tuning. Our&#10;experiments show that, for both methods, channel models significantly&#10;outperform their direct counterparts, which we attribute to their stability,&#10;i.e., lower variance and higher worst-case accuracy. We also present extensive&#10;ablations that provide recommendations for when to use channel prompt tuning&#10;instead of other competitive methods (e.g., direct head tuning): channel prompt&#10;tuning is preferred when the number of training examples is small, labels in&#10;the training data are imbalanced, or generalization to unseen labels is&#10;required.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.05566" label="2007.05566">
        <attvalues>
          <attvalue for="0" value="Contrastive Training for Improved Out-of-Distribution Detection" />
          <attvalue for="1" value="  Reliable detection of out-of-distribution (OOD) inputs is increasingly&#10;understood to be a precondition for deployment of machine learning systems.&#10;This paper proposes and investigates the use of contrastive training to boost&#10;OOD detection performance. Unlike leading methods for OOD detection, our&#10;approach does not require access to examples labeled explicitly as OOD, which&#10;can be difficult to collect in practice. We show in extensive experiments that&#10;contrastive training significantly helps OOD detection performance on a number&#10;of common benchmarks. By introducing and employing the Confusion Log&#10;Probability (CLP) score, which quantifies the difficulty of the OOD detection&#10;task by capturing the similarity of inlier and outlier datasets, we show that&#10;our method especially improves performance in the `near OOD' classes -- a&#10;particularly challenging setting for previous methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.02629" label="2007.02629">
        <attvalues>
          <attvalue for="0" value="Learning Spoken Language Representations with Neural Lattice Language&#10;  Modeling" />
          <attvalue for="1" value="  Pre-trained language models have achieved huge improvement on many NLP tasks.&#10;However, these methods are usually designed for written text, so they do not&#10;consider the properties of spoken language. Therefore, this paper aims at&#10;generalizing the idea of language model pre-training to lattices generated by&#10;recognition systems. We propose a framework that trains neural lattice language&#10;models to provide contextualized representations for spoken language&#10;understanding tasks. The proposed two-stage pre-training approach reduces the&#10;demands of speech data and has better efficiency. Experiments on intent&#10;detection and dialogue act recognition datasets demonstrate that our proposed&#10;method consistently outperforms strong baselines when evaluated on spoken&#10;inputs. The code is available at https://github.com/MiuLab/Lattice-ELMo.&#10;" />
          <attvalue for="2" value="&#10;The task of spoken language understanding (SLU) aims at extracting useful information from spoken utterances.&#10;Typically, SLU can be decomposed with a two-stage method:&#10;1) an accurate automatic speech recognition (ASR) system transcribes the input speech into texts, and then&#10;2) language understanding techniques are applied to the transcribed texts.&#10;These two modules can be developed separately, so most prior work developed the backend language understanding systems based on manual transcripts~\cite{yao2014spoken,guo2014joint,mesnil2014using,goo-etal-2018-slot}.&#10;&#10;Despite the simplicity of the two-stage method, prior work showed that a tighter integration between two components can lead to better performance.&#10;Researchers have extended the ASR 1-best results to n-best lists or word confusion networks in order to preserve the ambiguity of the transcripts. ~\cite{tur2002improving,hakkani2006beyond,henderson2012discriminative,Tr2013SemanticPU,masumura2018neural}.&#10;Another line of research focused on using lattices produced by ASR systems.&#10;Lattices are directed acyclic graphs (DAGs) that represent multiple recognition hypotheses.&#10;An example of ASR lattice is shown in Figure~\ref{fig:lattice}. &#10;\cite{ladhak2016latticernn} introduced LatticeRNN, a variant of recurrent neural networks (RNNs) that generalize RNNs to lattice-structured inputs in order to improve SLU.&#10;\cite{zhang-yang-2018-chinese} proposed a similar idea for Chinese name entity recognition.&#10;\cite{sperber-etal-2019-self,xiao-etal-2019-lattice,zhang-etal-2019-lattice} proposed extensions to enable the transformer model~\cite{vaswani2017attention} to consume lattice inputs for machine translation.&#10;\cite{huang2019adapting} proposed to adapt the transformer model originally pre-trained on written texts to consume lattices in order to improve SLU performance.&#10;\cite{buckman-neubig-2018-neural} also found that utilizing lattices that represent multiple granularities of sentences can improve language modeling.&#10;&#10;With recent introduction of large pre-trained language models (LMs) such as ELMo~\cite{peters-etal-2018-deep}, GPT~\cite{Radford2018ImprovingLU} and BERT~\cite{devlin-etal-2019-bert}, we have observed huge improvements on natural language understanding tasks.&#10;These models are pre-trained on large amount of written texts so that they provide the downstream tasks with high-quality representations.&#10;However, applying these models to the spoken scenarios poses several discrepancies between the pre-training task and the target task, such as the domain mismatch between written texts and spoken utterances with ASR errors.&#10;It has been shown that fine-tuning the pre-trained language models on the data from the target tasks can mitigate the domain mismatch problem~\cite{howard-ruder-2018-universal,chronopoulou-etal-2019-embarrassingly}.&#10;\cite{siddhant2018unsupervised} focused on pre-training a language model specifically for spoken content with huge amount of automatic transcripts, which requires a large collection of in-domain speech.&#10;&#10;In this paper, we propose a novel spoken language representation learning framework, which focuses on learning contextualized representations of lattices based on our proposed lattice language modeling objective.&#10;The proposed framework consists of two stages of LM pre-training to reduce the demands for lattice data.&#10;We conduct experiments on benchmark datasets for spoken language understanding, including intent classification and dialogue act recognition.&#10;The proposed method consistently achieves superior performance, with relative error reduction ranging from 3\% to 42\% compare to pre-trained sequential LM.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Spoken Language Understanding, Language Model Pre-training, Neural Network Frameworks, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1805.02023" label="1805.02023">
        <attvalues>
          <attvalue for="0" value="Chinese NER Using Lattice LSTM" />
          <attvalue for="1" value="  We investigate a lattice-structured LSTM model for Chinese NER, which encodes&#10;a sequence of input characters as well as all potential words that match a&#10;lexicon. Compared with character-based methods, our model explicitly leverages&#10;word and word sequence information. Compared with word-based methods, lattice&#10;LSTM does not suffer from segmentation errors. Gated recurrent cells allow our&#10;model to choose the most relevant characters and words from a sentence for&#10;better NER results. Experiments on various datasets show that lattice LSTM&#10;outperforms both word-based and character-based LSTM baselines, achieving the&#10;best results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01617" label="1906.01617">
        <attvalues>
          <attvalue for="0" value="Self-Attentional Models for Lattice Inputs" />
          <attvalue for="1" value="  Lattices are an efficient and effective method to encode ambiguity of&#10;upstream systems in natural language processing tasks, for example to compactly&#10;capture multiple speech recognition hypotheses, or to represent multiple&#10;linguistic analyses. Previous work has extended recurrent neural networks to&#10;model lattice inputs and achieved improvements in various tasks, but these&#10;models suffer from very slow computation speeds. This paper extends the&#10;recently proposed paradigm of self-attention to handle lattice inputs.&#10;Self-attention is a sequence modeling technique that relates inputs to one&#10;another by computing pairwise similarities and has gained popularity for both&#10;its strong results and its computational efficiency. To extend such models to&#10;handle lattices, we introduce probabilistic reachability masks that incorporate&#10;lattice structure into the model and support lattice scores if available. We&#10;also propose a method for adapting positional embeddings to lattice structures.&#10;We apply the proposed model to a speech translation task and find that it&#10;outperforms all examined baselines while being much faster to compute than&#10;previous neural lattice models during both training and inference.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01282" label="1906.01282">
        <attvalues>
          <attvalue for="0" value="Lattice-Based Transformer Encoder for Neural Machine Translation" />
          <attvalue for="1" value="  Neural machine translation (NMT) takes deterministic sequences for source&#10;representations. However, either word-level or subword-level segmentations have&#10;multiple choices to split a source sequence with different word segmentors or&#10;different subword vocabulary sizes. We hypothesize that the diversity in&#10;segmentations may affect the NMT performance. To integrate different&#10;segmentations with the state-of-the-art NMT model, Transformer, we propose&#10;lattice-based encoders to explore effective word or subword representation in&#10;an automatic way during training. We propose two methods: 1) lattice positional&#10;encoding and 2) lattice-aware self-attention. These two methods can be used&#10;together and show complementary to each other to further improve translation&#10;performance. Experiment results show superiorities of lattice-based encoders in&#10;word-level and subword-level representations over conventional Transformer&#10;encoder.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00780" label="2011.00780">
        <attvalues>
          <attvalue for="0" value="Adapting Pretrained Transformer to Lattices for Spoken Language&#10;  Understanding" />
          <attvalue for="1" value="  Lattices are compact representations that encode multiple hypotheses, such as&#10;speech recognition results or different word segmentations. It is shown that&#10;encoding lattices as opposed to 1-best results generated by automatic speech&#10;recognizer (ASR) boosts the performance of spoken language understanding (SLU).&#10;Recently, pretrained language models with the transformer architecture have&#10;achieved the state-of-the-art results on natural language understanding, but&#10;their ability of encoding lattices has not been explored. Therefore, this paper&#10;aims at adapting pretrained transformers to lattice inputs in order to perform&#10;understanding tasks specifically for spoken language. Our experiments on the&#10;benchmark ATIS dataset show that fine-tuning pretrained transformers with&#10;lattice inputs yields clear improvement over fine-tuning with 1-best results.&#10;Further evaluation demonstrates the effectiveness of our methods under&#10;different acoustic conditions. Our code is available at&#10;https://github.com/MiuLab/Lattice-SLU&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.05071" label="1803.05071">
        <attvalues>
          <attvalue for="0" value="Neural Lattice Language Models" />
          <attvalue for="1" value="  In this work, we propose a new language modeling paradigm that has the&#10;ability to perform both prediction and moderation of information flow at&#10;multiple granularities: neural lattice language models. These models construct&#10;a lattice of possible paths through a sentence and marginalize across this&#10;lattice to calculate sequence probabilities or optimize parameters. This&#10;approach allows us to seamlessly incorporate linguistic intuitions - including&#10;polysemy and existence of multi-word lexical items - into our language model.&#10;Experiments on multiple language modeling tasks show that English neural&#10;lattice language models that utilize polysemous embeddings are able to improve&#10;perplexity by 9.95% relative to a word-level baseline, and that a Chinese model&#10;that handles multi-character tokens is able to improve perplexity by 20.94%&#10;relative to a character-level baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.05370" label="1811.05370">
        <attvalues>
          <attvalue for="0" value="Unsupervised Transfer Learning for Spoken Language Understanding in&#10;  Intelligent Agents" />
          <attvalue for="1" value="  User interaction with voice-powered agents generates large amounts of&#10;unlabeled utterances. In this paper, we explore techniques to efficiently&#10;transfer the knowledge from these unlabeled utterances to improve model&#10;performance on Spoken Language Understanding (SLU) tasks. We use Embeddings&#10;from Language Model (ELMo) to take advantage of unlabeled data by learning&#10;contextualized word representations. Additionally, we propose ELMo-Light&#10;(ELMoL), a faster and simpler unsupervised pre-training method for SLU. Our&#10;findings suggest unsupervised pre-training on a large corpora of unlabeled&#10;utterances leads to significantly better SLU performance compared to training&#10;from scratch and it can even outperform conventional supervised transfer.&#10;Additionally, we show that the gains from unsupervised transfer techniques can&#10;be further improved by supervised transfer. The improvements are more&#10;pronounced in low resource settings and when using only 1000 labeled in-domain&#10;samples, our techniques match the performance of training from scratch on&#10;10-15x more labeled in-domain data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.04754" label="2102.04754">
        <attvalues>
          <attvalue for="0" value="Bayesian Transformer Language Models for Speech Recognition" />
          <attvalue for="1" value="  State-of-the-art neural language models (LMs) represented by Transformers are&#10;highly complex. Their use of fixed, deterministic parameter estimates fail to&#10;account for model uncertainty and lead to over-fitting and poor generalization&#10;when given limited training data. In order to address these issues, this paper&#10;proposes a full Bayesian learning framework for Transformer LM estimation.&#10;Efficient variational inference based approaches are used to estimate the&#10;latent parameter posterior distributions associated with different parts of the&#10;Transformer model architecture including multi-head self-attention, feed&#10;forward and embedding layers. Statistically significant word error rate (WER)&#10;reductions up to 0.5\% absolute (3.18\% relative) and consistent perplexity&#10;gains were obtained over the baseline Transformer LMs on state-of-the-art&#10;Switchboard corpus trained LF-MMI factored TDNN systems with i-Vector speaker&#10;adaptation. Performance improvements were also obtained on a cross domain LM&#10;adaptation task requiring porting a Transformer LM trained on the Switchboard&#10;and Fisher data to a low-resource DementiaBank elderly speech corpus.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models (LMs) play an important role in automatic speech recognition (ASR) systems and many other applications. Language models compute the joint probability of a given sentence ${W}=({{w}_1,{w}_2,...,{w}_T})$ as:&#10;&#10;\begin{equation}&#10;&#10;p({W}) = p({{w}_1,{w}_2,...,{w}_n})=\prod_{t=1}^{n} P({{w}_t|{w}_{t-1},..., {w}_1})&#10;\end{equation}&#10;&#10;which can be expressed using the multiplication of word level probabilities. The key part of the statistical language modelling problem is to learn long-range context dependencies. Directly modelling long-span contexts lead to a severe data sparsity problem for $n$-gram language models \cite{telrm}. To this end, neural language models that can represent longer span preceding history contexts in a continuous vector space, for example, based on long-short term memory recurrent neural networks (LSTM-RNNs) \cite{rnnblm,iteernnlm} can be used. &#10;&#10;In recent years deep Transformer models \cite{aiayn} have defined state-of-the-art language modelling performance across a range of speech recognition tasks \cite{lmwdt}. The Transformer model architecture features a deep stacking of multiple self-attention layers \cite{lstmnml,assse,adamnl} with residual connections \cite{drlir} and layer normalization \cite{ln} to learn long-range contexts. Positional encoding layers \cite{aiayn,cssl} are used to further augment the self-attention layers with sequence order information. Performance improvements over the conventional LSTM-RNN language models have been widely reported \cite{lmwdt,aestnlma}. &#10;&#10;The highly complex neural architecture design of Transformers often leads to a large increase in the overall system complexity, for example, up to hundreds of millions of parameters \cite{lmwdt}. In common with other deep learning based language modelling approaches \cite{rnnblm,iteernnlm}, the use of fixed, deterministic parameter estimates in conventional Transformer models fails to account for model uncertainty. When given limited training data, standard Transformer models are prone to over-fitting and poor generalization. This issue can be further aggregated when rapidly adapting a well-trained Transformer model to small size dataset associated with a new style, genre or domain \cite{aestnlma}. The current solution to this problem is largely based on dropout \cite{daswpnno}, a simple and effective regularization approach used in many deep learning systems including neural network language models \cite{brnnlm,gplrnnlm,csprum}. However, it lacks of a mathematically well-defined framework \cite{dbarmudl,blamnnu}. The underlying dropout distribution also requires hyper-parameter setting on an empirical basis for different tasks. &#10;&#10;In order to address these issues, this paper proposes a full Bayesian learning framework to account for model uncertainty in Transformer language model estimation. An efficient variational inference based approach is adopted to estimate the latent parameter posterior distribution. A systematic investigation on the effects of performing Bayesian estimation in different parts of the Transformer model architecture including the self-attention, feed forward and embedding layers is performed. Statistically significant word error rate (WER) reductions up to 0.5\% absolute (3.18\% relative) were obtained over the baseline Transformer LM on a state-of-the-art 900 hour speed perturbed Switchboard corpus trained LF-MMI factored TDNN system with i-Vector speaker adaptation \cite{lhuc}. Consistent performance improvements were also obtained on a cross domain LM adaptation task requiring rapidly porting a Transformer LM trained on Switchboard and Fisher data to a small size DementiaBank elderly speech corpus. &#10;&#10;The main contributions of this paper are summarized as follows. First, to the best of our knowledge, this paper is the first work to apply Bayesian learning methods to Transformer language models for speech recognition tasks. In contrast, the only previous research on Bayesian Transformer \cite{blamnnu,blt} was conducted on machine translation and probabilistic programming tasks. Prior works on uncertainty modelling under the Bayesian framework for neural network language modelling approaches were limited to RNNs \cite{brnnlm} and their LSTM or GRU based variants \cite{csprum}.&#10;&#10;The rest of this paper is organized as follows. Section 2 reviews the conventional Transformer based language models. Section 3 presents Bayesian Transformer language models. Implementation issues are discussed in Section 4. Experiments and results are shown in section5. Finally, conclusions and future work are discussed in section6. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Recognition Systems, Linguistics, Neural Language Models, Statistics, Bayesian Learning Framework" />
        </attvalues>
      </node>
      <node id="1905.04226" label="1905.04226">
        <attvalues>
          <attvalue for="0" value="Language Modeling with Deep Transformers" />
          <attvalue for="1" value="  We explore deep autoregressive Transformer models in language modeling for&#10;speech recognition. We focus on two aspects. First, we revisit Transformer&#10;model configurations specifically for language modeling. We show that well&#10;configured Transformer models outperform our baseline models based on the&#10;shallow stack of LSTM recurrent neural network layers. We carry out experiments&#10;on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level&#10;and 10K byte-pair encoding subword-level language modeling. We apply our&#10;word-level models to conventional hybrid speech recognition by lattice&#10;rescoring, and the subword-level models to attention based encoder-decoder&#10;models by shallow fusion. Second, we show that deep Transformer language models&#10;do not require positional encoding. The positional encoding is an essential&#10;augmentation for the self-attention mechanism which is invariant to sequence&#10;ordering. However, in autoregressive setup, as is the case for language&#10;modeling, the amount of information increases along the position dimension,&#10;which is a positional signal by its own. The analysis of attention weights&#10;shows that deep autoregressive self-attention models can automatically make use&#10;of such positional information. We find that removing the positional encoding&#10;even slightly improves the performance of these models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1601.06733" label="1601.06733">
        <attvalues>
          <attvalue for="0" value="Long Short-Term Memory-Networks for Machine Reading" />
          <attvalue for="1" value="  In this paper we address the question of how to render sequence-level&#10;networks better at handling structured input. We propose a machine reading&#10;simulator which processes text incrementally from left to right and performs&#10;shallow reasoning with memory and attention. The reader extends the Long&#10;Short-Term Memory architecture with a memory network in place of a single&#10;memory cell. This enables adaptive memory usage during recurrence with neural&#10;attention, offering a way to weakly induce relations among tokens. The system&#10;is initially designed to process a single sequence but we also demonstrate how&#10;to integrate it with an encoder-decoder architecture. Experiments on language&#10;modeling, sentiment analysis, and natural language inference show that our&#10;model matches or outperforms the state of the art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.03130" label="1703.03130">
        <attvalues>
          <attvalue for="0" value="A Structured Self-attentive Sentence Embedding" />
          <attvalue for="1" value="  This paper proposes a new model for extracting an interpretable sentence&#10;embedding by introducing self-attention. Instead of using a vector, we use a&#10;2-D matrix to represent the embedding, with each row of the matrix attending on&#10;a different part of the sentence. We also propose a self-attention mechanism&#10;and a special regularization term for the model. As a side effect, the&#10;embedding comes with an easy way of visualizing what specific parts of the&#10;sentence are encoded into the embedding. We evaluate our model on 3 different&#10;tasks: author profiling, sentiment classification, and textual entailment.&#10;Results show that our model yields a significant performance gain compared to&#10;other sentence embedding methods in all of the 3 tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.01933" label="1606.01933">
        <attvalues>
          <attvalue for="0" value="A Decomposable Attention Model for Natural Language Inference" />
          <attvalue for="1" value="  We propose a simple neural architecture for natural language inference. Our&#10;approach uses attention to decompose the problem into subproblems that can be&#10;solved separately, thus making it trivially parallelizable. On the Stanford&#10;Natural Language Inference (SNLI) dataset, we obtain state-of-the-art results&#10;with almost an order of magnitude fewer parameters than previous work and&#10;without relying on any word-order information. Adding intra-sentence attention&#10;that takes a minimum amount of order into account yields further improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.02142" label="1506.02142">
        <attvalues>
          <attvalue for="0" value="Dropout as a Bayesian Approximation: Representing Model Uncertainty in&#10;  Deep Learning" />
          <attvalue for="1" value="  Deep learning tools have gained tremendous attention in applied machine&#10;learning. However such tools for regression and classification do not capture&#10;model uncertainty. In comparison, Bayesian models offer a mathematically&#10;grounded framework to reason about model uncertainty, but usually come with a&#10;prohibitive computational cost. In this paper we develop a new theoretical&#10;framework casting dropout training in deep neural networks (NNs) as approximate&#10;Bayesian inference in deep Gaussian processes. A direct result of this theory&#10;gives us tools to model uncertainty with dropout NNs -- extracting information&#10;from existing models that has been thrown away so far. This mitigates the&#10;problem of representing uncertainty in deep learning without sacrificing either&#10;computational complexity or test accuracy. We perform an extensive study of the&#10;properties of dropout's uncertainty. Various network architectures and&#10;non-linearities are assessed on tasks of regression and classification, using&#10;MNIST as an example. We show a considerable improvement in predictive&#10;log-likelihood and RMSE compared to existing state-of-the-art methods, and&#10;finish by using dropout's uncertainty in deep reinforcement learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.03973" label="1812.03973">
        <attvalues>
          <attvalue for="0" value="Bayesian Layers: A Module for Neural Network Uncertainty" />
          <attvalue for="1" value="  We describe Bayesian Layers, a module designed for fast experimentation with&#10;neural network uncertainty. It extends neural network libraries with drop-in&#10;replacements for common layers. This enables composition via a unified&#10;abstraction over deterministic and stochastic functions and allows for&#10;scalability via the underlying system. These layers capture uncertainty over&#10;weights (Bayesian neural nets), pre-activation units (dropout), activations&#10;(&quot;stochastic output layers&quot;), or the function itself (Gaussian processes). They&#10;can also be reversible to propagate uncertainty from input to output. We&#10;include code examples for common architectures such as Bayesian LSTMs, deep&#10;GPs, and flow-based models. As demonstration, we fit a 5-billion parameter&#10;&quot;Bayesian Transformer&quot; on 512 TPUv2 cores for uncertainty in machine&#10;translation and a Bayesian dynamics model for model-based planning. Finally, we&#10;show how Bayesian Layers can be used within the Edward2 probabilistic&#10;programming language for probabilistic programs with stochastic processes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.07042" label="1808.07042">
        <attvalues>
          <attvalue for="0" value="CoQA: A Conversational Question Answering Challenge" />
          <attvalue for="1" value="  Humans gather information by engaging in conversations involving a series of&#10;interconnected questions and answers. For machines to assist in information&#10;gathering, it is therefore essential to enable them to answer conversational&#10;questions. We introduce CoQA, a novel dataset for building Conversational&#10;Question Answering systems. Our dataset contains 127k questions with answers,&#10;obtained from 8k conversations about text passages from seven diverse domains.&#10;The questions are conversational, and the answers are free-form text with their&#10;corresponding evidence highlighted in the passage. We analyze CoQA in depth and&#10;show that conversational questions have challenging phenomena not present in&#10;existing reading comprehension datasets, e.g., coreference and pragmatic&#10;reasoning. We evaluate strong conversational and reading comprehension models&#10;on CoQA. The best system obtains an F1 score of 65.4%, which is 23.4 points&#10;behind human performance (88.8%), indicating there is ample room for&#10;improvement. We launch CoQA as a challenge to the community at&#10;http://stanfordnlp.github.io/coqa/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.07045" label="1707.07045">
        <attvalues>
          <attvalue for="0" value="End-to-end Neural Coreference Resolution" />
          <attvalue for="1" value="  We introduce the first end-to-end coreference resolution model and show that&#10;it significantly outperforms all previous work without using a syntactic parser&#10;or hand-engineered mention detector. The key idea is to directly consider all&#10;spans in a document as potential mentions and learn distributions over possible&#10;antecedents for each. The model computes span embeddings that combine&#10;context-dependent boundary representations with a head-finding attention&#10;mechanism. It is trained to maximize the marginal likelihood of gold antecedent&#10;spans from coreference clusters and is factored to enable aggressive pruning of&#10;potential mentions. Experiments demonstrate state-of-the-art performance, with&#10;a gain of 1.5 F1 on the OntoNotes benchmark and by 3.1 F1 using a 5-model&#10;ensemble, despite the fact that this is the first approach to be successfully&#10;trained with no external resources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.10529" label="1907.10529">
        <attvalues>
          <attvalue for="0" value="SpanBERT: Improving Pre-training by Representing and Predicting Spans" />
          <attvalue for="1" value="  We present SpanBERT, a pre-training method that is designed to better&#10;represent and predict spans of text. Our approach extends BERT by (1) masking&#10;contiguous random spans, rather than random tokens, and (2) training the span&#10;boundary representations to predict the entire content of the masked span,&#10;without relying on the individual token representations within it. SpanBERT&#10;consistently outperforms BERT and our better-tuned baselines, with substantial&#10;gains on span selection tasks such as question answering and coreference&#10;resolution. In particular, with the same training data and model size as&#10;BERT-large, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0,&#10;respectively. We also achieve a new state of the art on the OntoNotes&#10;coreference resolution task (79.6\% F1), strong performance on the TACRED&#10;relation extraction benchmark, and even show gains on GLUE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12778" label="2310.12778">
        <attvalues>
          <attvalue for="0" value="Label-Aware Automatic Verbalizer for Few-Shot Text Classification" />
          <attvalue for="1" value="  Prompt-based learning has shown its effectiveness in few-shot text&#10;classification. One important factor in its success is a verbalizer, which&#10;translates output from a language model into a predicted class. Notably, the&#10;simplest and widely acknowledged verbalizer employs manual labels to represent&#10;the classes. However, manual selection does not guarantee the optimality of the&#10;selected words when conditioned on the chosen language model. Therefore, we&#10;propose Label-Aware Automatic Verbalizer (LAAV), effectively augmenting the&#10;manual labels to achieve better few-shot classification results. Specifically,&#10;we use the manual labels along with the conjunction &quot;and&quot; to induce the model&#10;to generate more effective words for the verbalizer. The experimental results&#10;on five datasets across five languages demonstrate that LAAV significantly&#10;outperforms existing verbalizers. Furthermore, our analysis reveals that LAAV&#10;suggests more relevant words compared to similar approaches, especially in&#10;mid-to-low resource languages.&#10;" />
          <attvalue for="2" value=" &#10;In recent years, we have seen many promising applications of prompt-based learning for text classification \cite{schick-schutze-2021-just,wang-etal-2022-towards-unified,zhang-etal-2022-prompt-based,hu-etal-2022-knowledgeable}.&#10;While the traditional approach trains or fine-tunes a model to directly predict a class for an input text, the prompt-based approach fits the input text into a template that has some slots to be filled. Next, it asks a language model (LM) to fill in the slots and then translate what the model fills to be a predicted class \cite{liu2023pre}.&#10;To predict whether a movie review &quot;Great movie!&quot; has a positive or negative sentiment, we may prompt a masked LM with &quot;Great movie! It was [MASK].&quot; The model may predict the word &quot;fun&quot; for the [MASK] token, and we can apply a function, so-called a verbalizer, to map &quot;fun&quot; to the positive class.&#10;&#10;Certainly, one important factor that defines the success of a prompt-based text classifier is its verbalizer. \cite{schick-schutze-2021-exploiting} proposed PET, which manually chooses a word to represent each class. During inference, it compares the likelihood of those words at the [MASK] token (as predicted by the LM) to find the most probable class. In contrast, \cite{wang-etal-2022-automatic} proposed AMuLaP, which represents each class with a set of words, automatically derived from those predicted by the LM for training examples.&#10;However, there is no guarantee that the words chosen by the LM will be relevant to the classes of interest. \cite{zhao-etal-2023-pre} proposed NPPrompt, which represents each class using a set of tokens with the highest embedding similarity to the manual class label. Its performance, therefore, relies solely on the LM’s embedding space.&#10;&#10;As illustrated in Figure~\ref{fig:overview} (top), to predict whether an object &quot;Feather&quot; is light with a prompt &quot;Feather is [MASK].&quot;, the LM suggests &quot;king&quot;, &quot;good&quot;, and &quot;strong&quot;, which are irrelevant to the task&#10;but used by AMuLaP to construct the verbalizer.&#10;Meanwhile, as shown in Figure~\ref{fig:overview} (middle),&#10;NPPrompt suggests &quot;Light&quot;, &quot;lights&quot;, and &quot;lighter&quot;, which are variations related to the class &quot;light&quot; but hardly provide additional information about the class.&#10;&#10;In this paper, we propose LAAV (Label-Aware Automatic Verbalizer), which improves AMuLaP by exploiting the class labels to induce the model to generate more relevant words for the verbalizer. As shown in Figure \ref{fig:overview} (bottom), we could construct a better verbalizer by asking &quot;Feather is light and [MASK].&quot; Now, the LM suggests &quot;fluffy&quot;, &quot;smooth&quot;, and &quot;soft&quot;, which are closely connected to the light class. Overall, the contributions of this paper are as follows.&#10;&#10;\begin{itemize}&#10;).&#10;&#10;\item We conduct few-shot classification experiments on five datasets from five languages (Section~\ref{sec:experiments}). The results show that LAAV outperforms the baselines in most settings, especially when we have 16 training examples per class or fewer (Section~\ref{subsec:mainresult}). &#10;&#10;\item We carry out additional analyses to determine the best choice of conjunction for retrieving more related words (Section~\ref{subsec:conjunction}) and to investigate the quality of the words selected by LAAV compared to AMuLaP (Section~\ref{subsec:verbalizerinterpret}).&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Text Classification, Machine Learning, Computer Science, Linguistics, Language Models, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2108.02035" label="2108.02035">
        <attvalues>
          <attvalue for="0" value="Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt&#10;  Verbalizer for Text Classification" />
          <attvalue for="1" value="  Tuning pre-trained language models (PLMs) with task-specific prompts has been&#10;a promising approach for text classification. Particularly, previous studies&#10;suggest that prompt-tuning has remarkable superiority in the low-data scenario&#10;over the generic fine-tuning methods with extra classifiers. The core idea of&#10;prompt-tuning is to insert text pieces, i.e., template, to the input and&#10;transform a classification problem into a masked language modeling problem,&#10;where a crucial step is to construct a projection, i.e., verbalizer, between a&#10;label space and a label word space. A verbalizer is usually handcrafted or&#10;searched by gradient descent, which may lack coverage and bring considerable&#10;bias and high variances to the results. In this work, we focus on incorporating&#10;external knowledge into the verbalizer, forming a knowledgeable prompt-tuning&#10;(KPT), to improve and stabilize prompt-tuning. Specifically, we expand the&#10;label word space of the verbalizer using external knowledge bases (KBs) and&#10;refine the expanded label word space with the PLM itself before predicting with&#10;the expanded label word space. Extensive experiments on zero and few-shot text&#10;classification tasks demonstrate the effectiveness of knowledgeable&#10;prompt-tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Two groups of research are related to KPT: prompt-tuning, and the verbalizer construction.&#10;&#10;Prompt-tuning. &#10;Since the emergence of GPT-3~\cite{brown2020language}, prompt-tuning has received considerable attention. &#10;GPT-3~\cite{brown2020language} demonstrates that with prompt-tuning and in-context learning, the large-scale language models can achieve superior performance in the low-data regime. &#10;The following works~\cite{schick2020exploiting, schick2020s} argue that small-scale language models ~\cite{radford2018improving, devlin2019bert, liu2019roberta, lan2019albert} can also achieve decent performance using prompt-tuning. Prompt-tuning has been applied to a large variety of tasks such as Text Classification~\cite{schick2020exploiting}, Natural Language Understanding~\cite{schick2020s, liu2021gpt}&#10;, Relation Extraction~\cite{han2021ptr,chen2021adaprompt}, and Knowledge Probing~\cite{petroni2019language, liu2021gpt}, etc.&#10;&#10;Verbalizer Construction.&#10;As introduced in \ref{sec:intr}, the verbalizer is an important component in prompt-tuning and has a strong influence on the performance of prompt-tuning~\cite{holtzman2021surface, gao2020making}. Most works use human-written verbalizers ~\cite{schick2020exploiting}, which are highly biased towards personal vocabulary and do not have enough coverage. Some other studies~\cite{gao2020making, shin2020autoprompt,liu2021gpt,schick2020automatically} design automatic verbalizer searching methods for better verbalizer choices, however, their methods require adequate training set and validation set for optimization. Moreover, the automatically determined verbalizers are usually synonym of the class name, which differs from our intuition of expanding the verbalizer with a set of diverse and comprehensive label words using external KB. &#10;~\cite{schick2020automatically} and~\cite{shin2020autoprompt} also try multiple label words for each class. &#10;The optimal size of their label words set for each class is generally less than 10, which lacks coverage when used in text classification tasks. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2204.06305" label="2204.06305">
        <attvalues>
          <attvalue for="0" value="Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot&#10;  Classification" />
          <attvalue for="1" value="  Prompt-based learning (i.e., prompting) is an emerging paradigm for&#10;exploiting knowledge learned by a pretrained language model. In this paper, we&#10;propose Automatic Multi-Label Prompting (AMuLaP), a simple yet effective method&#10;to automatically select label mappings for few-shot text classification with&#10;prompting. Our method exploits one-to-many label mappings and a&#10;statistics-based algorithm to select label mappings given a prompt template.&#10;Our experiments demonstrate that AMuLaP achieves competitive performance on the&#10;GLUE benchmark without human effort or external resources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Discrete Prompts The release of GPT-3~\cite{gpt3} has &#10;led to interest in&#10;prompting, a new way &#10;to leverage&#10;pretrained language models (PLM). \cite{gpt3} proposes an intuitive in-context learning &#10;paradigm&#10;by concatenating a few input and output examples and feeding them to the language model and let the model autoregressively generate answers for new examples. Recent works~\cite{petroni2019language,davision2019commonsense,jiang2020how} design prompts to probe the factual and common-sense knowledge encoded within a PLM. Recent works~\cite{schick2021eacl,schick2021naacl,gao2020making} demonstrate that even smaller PLMs have similar few-shot learning capacity. \cite{lescao2021naacl} analyzes the effect of prompting and concludes that &#10;a single&#10;prompt may be worth 100&#10;training examples&#10;in fine-tuning.&#10;&#10;Instead of manually designing prompts (i.e., prompt engineering), some recent studies also explore automatic prompt generation. &#10;PETAL~\cite{schick2020coling} augments Pattern Exploiting Training (PET,~\cite{schick2021eacl,schick2021naacl}) with automatically identified label words; &#10;\cite{gao2020making} uses re-ranking to find the best label words by fine-tuning a RoBERTa model on the candidates searched by RoBERTa, and using an external generation model for data augmentation of prompt templates;&#10;AutoPrompt~\cite{autoprompt} uses a gradient-based search to determine both prompts and label words. However, these methods require parameter updates with gradient descent, which is infeasible without access to the model weights (e.g., GPT-3). &#10;PET and its variants&#10;also&#10;require a large unlabeled set and need to be fine-tuned multiple times. AutoPrompt uses discretization techniques to approximately map a continuous vector back to tokens in the vocabulary (i.e., ``vocablization''). These searched prompts and labels are often uninterpretable by humans.&#10;Different from these prior studies, our proposed AMuLaP is a simple and interpretable method for few-shot prompting that can work well with and without access to &#10;model weights. Concurrently to our work, \cite{hu2021knowledgeable} propose a method that exploits an external knowledge base to find label mapping.&#10;T0~\cite{sanh2022multitask,promptsource} constructs a dataset of different NLP tasks by manually writing prompt templates and shows that a large language model with multitask training can generalize to unseen tasks.&#10;&#10;Continuous Prompts In parallel with text-based discrete prompts, there is also a line of &#10;work&#10;focused on tuning only a fraction of parameters of an LM with the help of continuous prompts (i.e., soft prompts). \cite{zhong2021naacl} and \cite{qin2021naacl} propose continuous prompts for knowledge probing by tuning some trainable vectors in the input sequence while fixing the rest of the input. \cite{li2021prefix} applies a similar method for natural language generation and achieves comparable performance to fine-tuning while updating only 0.1\&#10;of model&#10;parameters. \cite{lester2021power} reveals that prompt tuning is more competitive when scaled up and can achieve &#10;identical performance to conventional fine-tuning when the model is large enough. \cite{guo2021text} introduces Q-Learning to optimize the soft prompt. Notably, different from discrete prompting, these works often use all training data to update model weights. Different from these works, AMuLaP is a discrete prompting method that has better interpretability and works well in the few-shot setting.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2212.06950" label="2212.06950">
        <attvalues>
          <attvalue for="0" value="Pre-trained Language Models Can be Fully Zero-Shot Learners" />
          <attvalue for="1" value="  How can we extend a pre-trained model to many language understanding tasks,&#10;without labeled or additional unlabeled data? Pre-trained language models&#10;(PLMs) have been effective for a wide range of NLP tasks. However, existing&#10;approaches either require fine-tuning on downstream labeled datasets or&#10;manually constructing proper prompts. In this paper, we propose nonparametric&#10;prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike&#10;previous methods, NPPrompt uses only pre-trained language models and does not&#10;require any labeled data or additional raw corpus for further fine-tuning, nor&#10;does it rely on humans to construct a comprehensive set of prompt label words.&#10;We evaluate NPPrompt against previous major few-shot and zero-shot learning&#10;methods on diverse NLP tasks: including text classification, text entailment,&#10;similar text retrieval, and paraphrasing. Experimental results demonstrate that&#10;our NPPrompt outperforms the previous best fully zero-shot method by big&#10;margins, with absolute gains of 12.8% in accuracy on text classification and&#10;18.9% on the GLUE benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Prompting&#10;The success of GPT-3 \cite{Brown2020LanguageMA} has attracted much attention to prompting engineering, a new way to leverage pre-trained language models.&#10;\cite{Brown2020LanguageMA} concatenate a few input and output pairs and feed them to the large-scale GPT-3 language model, which is an intuitive in-context learning paradigm, allowing the model to generate answers for additional cases autoregressively. &#10;Recent works \cite{Schick2021ExploitingCF, Schick2021ItsNJ} show that small-scale pre-trained language models such as BERT \cite{Devlin2019BERTPO}, RoBERTa \cite{Liu2019RoBERTaAR} and ALBERT \cite{Lan2020ALBERTAL} can also achieve decent performance using prompt-tuning. &#10;Prompting has been applied to a large variety of tasks such as Text Classification \cite{Schick2021ExploitingCF}, Natural Language Understanding \cite{Xu2022Go}, Knowledge Probing \cite{Petroni2019LanguageMA}, and Relation Extraction \cite{Han2021PTRPT}. &#10;Typically, a piece of prompt contains a template and a verbalizer. &#10;The language model predicts a probability distribution over vocabulary given the template and the verbalizer transforms it into a prediction over class labels. &#10;In this work, we focus on designing the verbalizers automatically.&#10;Verbalizer Design&#10;The verbalizer plays a crucial role in prompting as it connects model outputs and labels, significantly influencing performance.&#10;\cite{Schick2021ExploitingCF} design human written verbalizers for prompting, however, they are highly biased towards personal vocabulary with inadequate coverage. &#10;Apart from manually designed verbalizers, some recent studies explore automatic verbalizer construction. &#10;Auto-L \cite{Gao2021MakingPL} uses re-ranking to find the label words set by fine-tuning the model on the candidates searched by RoBERTa; &#10;AutoPrompt \cite{shin2020autoprompt} applies gradient-based search to create both prompts and label words automatically with a few trigger examples. &#10;But these approaches need to update parameters with gradient descent, which turns out to be infeasible without access to the model weights (e.g., GPT-3). &#10;KPT \cite{Han2021PTRPT} incorporates external knowledge into the verbalizer in which the unlabeled dataset is needed to refine the label words and thus is not applicable to scenarios where only label names are known. &#10;In contrast, our approach \method directly finds, without any gradient update, relevant words to label names with PLM's initial word embedding only.&#10;Zero-shot Text Classification&#10;General zero-shot text classification typically focuses on classifying texts into categories that were not seen during the training process. Transferring knowledge from seen classes to unseen ones requires accurate and discriminative descriptions of all classes \cite{Liu2019ReconstructingCN, Xia2018ZeroshotUI} or joint embeddings of categories and documents \cite{Nam2016AllinTL}. However, these methods rely on supervised data for the known label set, making them unsuitable for scenarios where no labeled pairs for any category are available. SimPTC \cite{Fei2022BeyondPM} improves zero-shot classification by clustering input texts and employing class-related prompts. LOTClass \cite{Meng2020TextCU} proposes a model that utilizes label names with self-training for zero-shot classification. Nonetheless, both SimPTC and LOTClass still require an unlabeled corpus or knowledge base to extract topic-related words and perform self-training. In contrast, \method achieves comparable or even superior performance without the need for any unlabeled dataset or knowledge base.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1904.03310" label="1904.03310">
        <attvalues>
          <attvalue for="0" value="Gender Bias in Contextualized Word Embeddings" />
          <attvalue for="1" value="  In this paper, we quantify, analyze and mitigate gender bias exhibited in&#10;ELMo's contextualized word vectors. First, we conduct several intrinsic&#10;analyses and find that (1) training data for ELMo contains significantly more&#10;male than female entities, (2) the trained ELMo embeddings systematically&#10;encode gender information and (3) ELMo unequally encodes gender information&#10;about male and female entities. Then, we show that a state-of-the-art&#10;coreference system that depends on ELMo inherits its bias and demonstrates&#10;significant bias on the WinoBias probing corpus. Finally, we explore two&#10;methods to mitigate such gender bias and show that the bias demonstrated on&#10;WinoBias can be eliminated.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.06904" label="2203.06904">
        <attvalues>
          <attvalue for="0" value="Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for&#10;  Pre-trained Language Models" />
          <attvalue for="1" value="  Despite the success, the process of fine-tuning large-scale PLMs brings&#10;prohibitive adaptation costs. In fact, fine-tuning all the parameters of a&#10;colossal model and retaining separate instances for different tasks are&#10;practically infeasible. This necessitates a new branch of research focusing on&#10;the parameter-efficient adaptation of PLMs, dubbed as delta tuning in this&#10;paper. In contrast with the standard fine-tuning, delta tuning only fine-tunes&#10;a small portion of the model parameters while keeping the rest untouched,&#10;largely reducing both the computation and storage costs. Recent studies have&#10;demonstrated that a series of delta tuning methods with distinct tuned&#10;parameter selection could achieve performance on a par with full-parameter&#10;fine-tuning, suggesting a new promising way of stimulating large-scale PLMs. In&#10;this paper, we first formally describe the problem of delta tuning and then&#10;comprehensively review recent delta tuning approaches. We also propose a&#10;unified categorization criterion that divide existing delta tuning methods into&#10;three groups: addition-based, specification-based, and reparameterization-based&#10;methods. Though initially proposed as an efficient method to steer large&#10;models, we believe that some of the fascinating evidence discovered along with&#10;delta tuning could help further reveal the mechanisms of PLMs and even deep&#10;neural networks. To this end, we discuss the theoretical principles underlying&#10;the effectiveness of delta tuning and propose frameworks to interpret delta&#10;tuning from the perspective of optimization and optimal control, respectively.&#10;Furthermore, we provide a holistic empirical study of representative methods,&#10;where results on over 100 NLP tasks demonstrate a comprehensive performance&#10;comparison of different approaches. The experimental results also cover the&#10;analysis of combinatorial, scaling and transferable properties of delta tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.03646" label="2109.03646">
        <attvalues>
          <attvalue for="0" value="Sustainable Modular Debiasing of Language Models" />
          <attvalue for="1" value="  Unfair stereotypical biases (e.g., gender, racial, or religious biases)&#10;encoded in modern pretrained language models (PLMs) have negative ethical&#10;implications for widespread adoption of state-of-the-art language technology.&#10;To remedy for this, a wide range of debiasing techniques have recently been&#10;introduced to remove such stereotypical biases from PLMs. Existing debiasing&#10;methods, however, directly modify all of the PLMs parameters, which -- besides&#10;being computationally expensive -- comes with the inherent risk of&#10;(catastrophic) forgetting of useful language knowledge acquired in pretraining.&#10;In this work, we propose a more sustainable modular debiasing approach based on&#10;dedicated debiasing adapters, dubbed ADELE. Concretely, we (1) inject adapter&#10;modules into the original PLM layers and (2) update only the adapters (i.e., we&#10;keep the original PLM parameters frozen) via language modeling training on a&#10;counterfactually augmented corpus. We showcase ADELE, in gender debiasing of&#10;BERT: our extensive evaluation, encompassing three intrinsic and two extrinsic&#10;bias measures, renders ADELE, very effective in bias mitigation. We further&#10;show that -- due to its modular nature -- ADELE, coupled with task adapters,&#10;retains fairness even after large-scale downstream training. Finally, by means&#10;of multilingual BERT, we successfully transfer ADELE, to six target languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.08100" label="2007.08100">
        <attvalues>
          <attvalue for="0" value="Towards Debiasing Sentence Representations" />
          <attvalue for="1" value="  As natural language processing methods are increasingly deployed in&#10;real-world scenarios such as healthcare, legal systems, and social science, it&#10;becomes necessary to recognize the role they potentially play in shaping social&#10;biases and stereotypes. Previous work has revealed the presence of social&#10;biases in widely used word embeddings involving gender, race, religion, and&#10;other social constructs. While some methods were proposed to debias these&#10;word-level embeddings, there is a need to perform debiasing at the&#10;sentence-level given the recent shift towards new contextualized sentence&#10;representations such as ELMo and BERT. In this paper, we investigate the&#10;presence of social biases in sentence-level representations and propose a new&#10;method, Sent-Debias, to reduce these biases. We show that Sent-Debias is&#10;effective in removing biases, and at the same time, preserves performance on&#10;sentence-level downstream tasks such as sentiment analysis, linguistic&#10;acceptability, and natural language understanding. We hope that our work will&#10;inspire future research on characterizing and removing social biases from&#10;widely adopted sentence representations for fairer NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.00453" label="2103.00453">
        <attvalues>
          <attvalue for="0" value="Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based&#10;  Bias in NLP" />
          <attvalue for="1" value="  When trained on large, unfiltered crawls from the internet, language models&#10;pick up and reproduce all kinds of undesirable biases that can be found in the&#10;data: they often generate racist, sexist, violent or otherwise toxic language.&#10;As large models require millions of training examples to achieve good&#10;performance, it is difficult to completely prevent them from being exposed to&#10;such content. In this paper, we first demonstrate a surprising finding:&#10;pretrained language models recognize, to a considerable degree, their&#10;undesirable biases and the toxicity of the content they produce. We refer to&#10;this capability as self-diagnosis. Based on this finding, we then propose a&#10;decoding algorithm that, given only a textual description of the undesired&#10;behavior, reduces the probability of a language model producing problematic&#10;text. We refer to this approach as self-debiasing. Self-debiasing does not rely&#10;on manually curated word lists, nor does it require any training data or&#10;changes to the model's parameters. While we by no means eliminate the issue of&#10;language models generating biased text, we believe our approach to be an&#10;important step in this direction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.06876" label="1804.06876">
        <attvalues>
          <attvalue for="0" value="Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods" />
          <attvalue for="1" value="  We introduce a new benchmark, WinoBias, for coreference resolution focused on&#10;gender bias. Our corpus contains Winograd-schema style sentences with entities&#10;corresponding to people referred by their occupation (e.g. the nurse, the&#10;doctor, the carpenter). We demonstrate that a rule-based, a feature-rich, and a&#10;neural coreference system all link gendered pronouns to pro-stereotypical&#10;entities with higher accuracy than anti-stereotypical entities, by an average&#10;difference of 21.1 in F1 score. Finally, we demonstrate a data-augmentation&#10;approach that, in combination with existing word-embedding debiasing&#10;techniques, removes the bias demonstrated by these systems in WinoBias without&#10;significantly affecting their performance on existing coreference benchmark&#10;datasets. Our dataset and code are available at http://winobias.org.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.04571" label="1906.04571">
        <attvalues>
          <attvalue for="0" value="Counterfactual Data Augmentation for Mitigating Gender Stereotypes in&#10;  Languages with Rich Morphology" />
          <attvalue for="1" value="  Gender stereotypes are manifest in most of the world's languages and are&#10;consequently propagated or amplified by NLP systems. Although research has&#10;focused on mitigating gender stereotypes in English, the approaches that are&#10;commonly employed produce ungrammatical sentences in morphologically rich&#10;languages. We present a novel approach for converting between&#10;masculine-inflected and feminine-inflected sentences in such languages. For&#10;Spanish and Hebrew, our approach achieves F1 scores of 82% and 73% at the level&#10;of tags and accuracies of 90% and 87% at the level of forms. By evaluating our&#10;approach using four different languages, we show that, on average, it reduces&#10;gender stereotyping by a factor of 2.5 without any sacrifice to grammaticality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.06032" label="2010.06032">
        <attvalues>
          <attvalue for="0" value="Measuring and Reducing Gendered Correlations in Pre-trained Models" />
          <attvalue for="1" value="  Pre-trained models have revolutionized natural language understanding.&#10;However, researchers have found they can encode artifacts undesired in many&#10;applications, such as professions correlating with one gender more than&#10;another. We explore such gendered correlations as a case study for how to&#10;address unintended correlations in pre-trained models. We define metrics and&#10;reveal that it is possible for models with similar accuracy to encode&#10;correlations at very different rates. We show how measured correlations can be&#10;reduced with general-purpose techniques, and highlight the trade offs different&#10;strategies have. With these results, we make recommendations for training&#10;robust models: (1) carefully evaluate unintended correlations, (2) be mindful&#10;of seemingly innocuous configuration differences, and (3) focus on general&#10;mitigations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.07751" label="2312.07751">
        <attvalues>
          <attvalue for="0" value="Large Human Language Models: A Need and the Challenges" />
          <attvalue for="1" value="  As research in human-centered NLP advances, there is a growing recognition of&#10;the importance of incorporating human and social factors into NLP models. At&#10;the same time, our NLP systems have become heavily reliant on LLMs, most of&#10;which do not model authors. To build NLP systems that can truly understand&#10;human language, we must better integrate human contexts into LLMs. This brings&#10;to the fore a range of design considerations and challenges in terms of what&#10;human aspects to capture, how to represent them, and what modeling strategies&#10;to pursue. To address these, we advocate for three positions toward creating&#10;large human language models (LHLMs) using concepts from psychological and&#10;behavioral sciences: First, LM training should include the human context.&#10;Second, LHLMs should recognize that people are more than their group(s). Third,&#10;LHLMs should be able to account for the dynamic and temporally-dependent nature&#10;of the human context. We refer to relevant advances and present open challenges&#10;that need to be addressed and their possible solutions in realizing these&#10;goals.&#10;" />
          <attvalue for="2" value="&#10;Language is a fundamental form of human expression and communication of thoughts, emotions, and experiences. Learning the meaning of words extends beyond syntax, semantics, and the neighboring words. To truly understand human language, we must look at words in the context of the human generating the language. Figure \ref{fig:human_lang_and_states} depicts a view of how our language is moderated by our somewhat stable and changing human states of being over time \cite{fleeson2001toward, mehl2003sounds, heller2007dynamics}. &#10;&#10;Progress in human-centered NLP research has established the importance of modeling human and social factors, presenting a compelling argument that learning language from linguistic signals alone is not adequate \cite{hovy_social_2018, bisk_experience_2020, flek_returning_2020}, and noting that feelings, knowledge and mental states of the speaker and listener referred to as the ``Theory of Mind'' \cite{Flavell2004TheoryofMindDR}, along with other social context variables are vital to language understanding \cite{bisk_experience_2020, hovy_importance_2021}. This need is backed by a wealth of empirical evidence demonstrating the benefits of modeling human and social factors \cite{volkova_exploring_2013, hu_exploiting_2013, bamman_contextualized_2015, lynn_human_2017, radfar_characterizing_2020}, and personalized models \cite{delasalles_learning_2019, jaech-ostendorf-2018-personalized, king-cook-2020-evaluating, welch_exploring_2020}.&#10;&#10;In parallel, with the advent of Transformers \cite{vaswani2017attention}, there have been many advances in language modeling \cite{devlin-etal-2019-bert, dai-etal-2019-transformer, liu2019roberta, radford2019language} yielding Transformer-based large language models (LLMs) as the base of most current NLP systems. LLMs train on a pre-training task and are capable of being applied to a broad set of NLP tasks producing state-of-the-art results. However, these language models create word representation &#10;{without }explicitly account{ing} for the context of the authors. &#10;&#10;Moreover, a person's language can be considered in the rich and complex human context that spans a &#10;{multitude} of aspects.&#10;\begin{quote}&#10; [S]peakers design their utterances to be understood against the common ground they share with their addressees—their common experience, expertise, dialect, and culture. -&#10;\cite{Schober1989UnderstandingBA}&#10;\end{quote}&#10;&#10;Figure \ref{fig:human_context_example} illustrates an extensive set of factors that can be considered ``human context'' which affects how one generates language. &#10;A sentence that begins with the phrase ``I'm going to...'', can be continued in various ways depending on several factors such as (a) who is speaking, (b) where are they / in what situation and (c) when are they speaking, and (d) to whom the sentence is addressed including their own time and place. &#10;Specific examples of factors include age, personality, occupation, etc., and the forms and modes of communication like public speaking, letter writing, books, phone conversations, etc. &#10;The speaker's language is, thus, highly dependent on the speaker's states, traits, social and environmental factors~\cite{boyd2021natural}, which, collectively, are referred to as the human context.&#10;&#10;{LLMs can benefit immensely from integrating the human context to truly understand human language but this entails multiple challenges. LLMs can be seen as containing a multitude of personas, and when prompted or primed appropriately can assume a specific one~\cite{patel2022low}. {Recently, such user-centric prompting has been employed for personalized recommender systems~\cite{doddapaneni-etal-2023-towards}, dialog systems \cite{gao2023chat}, and measuring political biases and fairness \cite{feng-etal-2023-pretraining}.} Models such as GPT-3 and ChatGPT demonstrate potential for simulating some forms of human context, especially in generative tasks~\cite{reif-etal-2022-recipe}. Continued scaling --- building bigger models trained on larger amounts of data --- will continue to improve these abilities.&#10;However, there are two fundamental limitations to this paradigm. First, models do not explicitly handle the multi-level structure (documents connected to people) necessary for modeling the richness of human context. Second, newer paradigms of in-context learning and user-centric prompting can benefit specific settings such as personalization but are still limiting LLMs from making full use of the human context more broadly. Recent evaluations and benchmarks reveal that prompting is insufficient to capture the richness of the human and social context \cite{salemi2023lamp, choi-etal-2023-llms}.} &#10;&#10;Instead, in this work we call for a more direct and explicit integration of the human contexts when building language models. In particular, we advocate for including the human context directly in language model training, building rich human contexts that account for the fact that people are more than their groups, and the dynamic and temporal-dependent changes to their states of being.&#10;In short, we call for building large human language models as a step towards better understanding the human language. {We motivate our positions using insights from a large body of past work and discuss shortcomings throughout the text.} {Furthermore, we discuss open challenges in realizing this vision and their possible solutions.}&#10;&#10;{Social context \cite{hovy_importance_2021} encompasses the human context but is not limited to it.} {In this work, we focus our vision of LHLMs in the scope of human context.}&#10;{Human context goes far beyond what can be captured from text modality alone. For example, other modalities such as gestures, speech, and body language are also a significant part of humans and their thought processes which gives us a more holistic picture of understanding human expression.} {However, in this work, we&#10;{limit ourselves to the}&#10;human context derived from language. We discuss limitations in detail in Section \ref{limitations}. Furthermore, LHLMs by their very nature are associated with sensitive user information and have the potential to be misused. Thus, it becomes essential to adopt a responsible release strategy for such models.} {We discuss a range of ethical considerations and privacy concerns and implications in Section \ref{ethical}.}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Centered NLP, Computer Science, Large Language Models, Linguistics, Psychological Modeling, Natural Language Processing, Psychology" />
        </attvalues>
      </node>
      <node id="2004.10151" label="2004.10151">
        <attvalues>
          <attvalue for="0" value="Experience Grounds Language" />
          <attvalue for="1" value="  Language understanding research is held back by a failure to relate language&#10;to the physical world it describes and to the social interactions it&#10;facilitates. Despite the incredible effectiveness of language processing models&#10;to tackle tasks after being trained on text alone, successful linguistic&#10;communication relies on a shared experience of the world. It is this shared&#10;experience that makes utterances meaningful.&#10;  Natural language processing is a diverse field, and progress throughout its&#10;development has come from new representational theories, modeling techniques,&#10;data collection paradigms, and tasks. We posit that the present success of&#10;representation learning approaches trained on large, text-only corpora requires&#10;the parallel tradition of research on the broader physical and social context&#10;of language to address the deeper questions of communication.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.04985" label="1909.04985">
        <attvalues>
          <attvalue for="0" value="Learning Dynamic Author Representations with Temporal Language Models" />
          <attvalue for="1" value="  Language models are at the heart of numerous works, notably in the text&#10;mining and information retrieval communities. These statistical models aim at&#10;extracting word distributions, from simple unigram models to recurrent&#10;approaches with latent variables that capture subtle dependencies in texts.&#10;However, those models are learned from word sequences only, and authors'&#10;identities, as well as publication dates, are seldom considered. We propose a&#10;neural model, based on recurrent language modeling, which aims at capturing&#10;language diffusion tendencies in author communities through time. By&#10;conditioning language models with author and temporal vector states, we are&#10;able to leverage the latent dependencies between the text contexts. This allows&#10;us to beat several temporal and non-temporal language baselines on two&#10;real-world corpora, and to learn meaningful author representations that vary&#10;through time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Interest for language evolution in texts is not new. Going back some fifteen years, we find work on the evolution of topics in textual documents, notably \cite{kaban02} whose model based on hidden Markov chains seeks to visualize temporal evolution in a textual stream. This approach falls in the general field of Topic Detection and Tracking, where the idea is to identify and follow trending topics in streams. The approach, which extends the GTM temporal model of \cite{bishop97} for textual modeling, allows one to visualize the thematic changes via trajectories on a two-dimensional grid. However, this kind of work enables to track thematics and to segment texts but cannot be used for language modeling. The non-markovian approach proposed in \cite{wang06} is restricted to bag-of-words representations, but has a good ability to detect the topics' evolution over the observation period. Besides, various works studied temporal vocabulary evolution - according to semantic graph transformations in \cite{kenter15} -, or thematic shifts in author communities - according to the dominant topics per time-step in \cite{hall08}.&#10;&#10;Closer to applications targeted in this paper, dynamic topic models \cite{blei06} propose an LDA-like modeling (Latent Dirichlet Allocation \cite{blei02}), where the topic distributions and the distributions of words w.r.t. topics evolve over time. The evolution between successive multinomial distributions are driven by Gaussian motions of their natural parameters, in a Kalman filters fashion, and optimized via variational inference. However, these approaches require manually setting the number of topics, and language models are limited to simple word occurrence distributions. It is not trivial to include models with long-term dependencies, such as LSTM, in this context. Moreover, contrary to ours, these approaches are usually constrained to specific conjugate distributions for the inference of the latent variables of their evolution model. Note the extensions of \cite{blei06} to a multi-scale temporal version \cite{iwata12} or to a model with continuous-time dependencies \cite{wang12}. Besides, \cite{gerrish10} introduces the concept of influence between documents, which could get closer to our objective but which is limited to analysis tasks. Lastly, \cite{wang11} proposes a temporal approach which considers relationships between documents via a known graph of dependencies, which leaves the scope of this study where we assume that such relational knowledge is not available a priori. &#10;&#10;In the vein of representation learning models \cite{bengio2003} and of the famous Word2Vec, a recent craze for time modeling has elicited various models based on word projections in latent vector spaces such as \cite{eger17} - linear temporal dependencies between word representations -, \cite{bamler17} - a dynamic Skip-Gram model -, \cite{rudolph17} - a model with exponential probabilistic evolution - or \cite{yao17} - matrix factorization with temporal alignment. As opposed to \cite{kaban02}, textual tokens are projected in a continuous space rather than on a discrete grid, which enables the use of classical continuous optimization methods. Moreover, contrary to previous approaches based on topic distributions with temporal dependencies, the goal of these works is to learn some semantic representations of words that can be used directly in various neural models.&#10;The temporal dependencies are defined on word representations: each considered time-step is associated with its own vocabulary representation forced to respect various temporal constraints. However, it appears difficult to consider such a kind of approach in a multi-author setting, for which separated representations should be learned both per time-step and also per author. We can note the approach of \cite{rudolph17b} for grouped data that enables a reduction of the number of parameters that have to be learned by sharing context vectors between groups, but whose transposition to a multi-author setting appears difficult (very high number of groups, doubled dependencies, temporal evolution vs connected groups).&#10;Another limitation with this kind of approach is that they do not allow end-to-end learning of language models, and extending them for outputting word probabilistic distributions is usually difficult.&#10;&#10;An alternative to these various models is to leverage RNNs for language modeling. A recurrent language model takes a sequence of words of arbitrary size as input and outputs a probability distribution of the next word. Such models are often parameterized by LSTM networks \cite{hochreiter97}. Compared to the skip-gram algorithm that uses a limited context window, recurrent language models operate on sequences of arbitrary length and can capture long-term dependencies. They are nowadays used at the core of an increasing number of tasks, for instance as a feature extractor for text classification \cite{peters18}, as a core building block of unsupervised Neural Machine Translation models \cite{lample2018phrase}, or as a discriminator for Generative Adversarial Models on text \cite{yang18}.&#10;&#10;Conditioning language models has already been considered for modeling the context of words in the documents \cite{le14}, but, to the best of our knowledge, not for the extraction of some temporal or structural dynamics in author communities. Rather than defining an individual vectorial representation for each word at every step and for each author, which appears highly too complex to be correctly learned, the idea is to rely on learned author representations modified according to a dynamic function.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2303.14524" label="2303.14524">
        <attvalues>
          <attvalue for="0" value="Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender&#10;  System" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated their significant potential to&#10;be applied for addressing various application tasks. However, traditional&#10;recommender systems continue to face great challenges such as poor&#10;interactivity and explainability, which actually also hinder their broad&#10;deployment in real-world systems. To address these limitations, this paper&#10;proposes a novel paradigm called Chat-Rec (ChatGPT Augmented Recommender&#10;System) that innovatively augments LLMs for building conversational recommender&#10;systems by converting user profiles and historical interactions into prompts.&#10;Chat-Rec is demonstrated to be effective in learning user preferences and&#10;establishing connections between users and products through in-context&#10;learning, which also makes the recommendation process more interactive and&#10;explainable. What's more, within the Chat-Rec framework, user's preferences can&#10;transfer to different products for cross-domain recommendations, and&#10;prompt-based injection of information into LLMs can also handle the cold-start&#10;scenarios with new items. In our experiments, Chat-Rec effectively improve the&#10;results of top-k recommendations and performs better in zero-shot rating&#10;prediction task. Chat-Rec offers a novel approach to improving recommender&#10;systems and presents new practical scenarios for the implementation of AIGC (AI&#10;generated content) in recommender system studies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.08283" label="2305.08283">
        <attvalues>
          <attvalue for="0" value="From Pretraining Data to Language Models to Downstream Tasks: Tracking&#10;  the Trails of Political Biases Leading to Unfair NLP Models" />
          <attvalue for="1" value="  Language models (LMs) are pretrained on diverse data sources, including news,&#10;discussion forums, books, and online encyclopedias. A significant portion of&#10;this data includes opinions and perspectives which, on one hand, celebrate&#10;democracy and diversity of ideas, and on the other hand are inherently socially&#10;biased. Our work develops new methods to (1) measure political biases in LMs&#10;trained on such corpora, along social and economic axes, and (2) measure the&#10;fairness of downstream NLP models trained on top of politically biased LMs. We&#10;focus on hate speech and misinformation detection, aiming to empirically&#10;quantify the effects of political (social, economic) biases in pretraining data&#10;on the fairness of high-stakes social-oriented tasks. Our findings reveal that&#10;pretrained LMs do have political leanings that reinforce the polarization&#10;present in pretraining corpora, propagating social biases into hate speech&#10;predictions and misinformation detectors. We discuss the implications of our&#10;findings for NLP research and propose future directions to mitigate unfairness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.03910" label="2109.03910">
        <attvalues>
          <attvalue for="0" value="A Recipe For Arbitrary Text Style Transfer with Large Language Models" />
          <attvalue for="1" value="  In this paper, we leverage large language models (LMs) to perform zero-shot&#10;text style transfer. We present a prompting method that we call augmented&#10;zero-shot learning, which frames style transfer as a sentence rewriting task&#10;and requires only a natural language instruction, without model fine-tuning or&#10;exemplars in the target style. Augmented zero-shot learning is simple and&#10;demonstrates promising results not just on standard style transfer tasks such&#10;as sentiment, but also on arbitrary transformations such as &quot;make this&#10;melodramatic&quot; or &quot;insert a metaphor.&quot;&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.04902" label="1706.04902">
        <attvalues>
          <attvalue for="0" value="A Survey Of Cross-lingual Word Embedding Models" />
          <attvalue for="1" value="  Cross-lingual representations of words enable us to reason about word meaning&#10;in multilingual contexts and are a key facilitator of cross-lingual transfer&#10;when developing natural language processing models for low-resource languages.&#10;In this survey, we provide a comprehensive typology of cross-lingual word&#10;embedding models. We compare their data requirements and objective functions.&#10;The recurring theme of the survey is that many of the models presented in the&#10;literature optimize for the same objectives, and that seemingly different&#10;models are often equivalent modulo optimization strategies, hyper-parameters,&#10;and such. We also discuss the different ways cross-lingual word embeddings are&#10;evaluated, as well as future challenges and research horizons.&#10;" />
          <attvalue for="2" value="&#10;In recent years, (monolingual) vector representations of words, so-called word embeddings \cite{Mikolov2013a,Pennington2014} have proven extremely useful across a wide range of natural language processing (NLP) applications. In parallel, the public awareness of the digital language divide, as well as the availability of multilingual benchmarks \cite&lt;Nivre et al., 2016a;&gt;{hovy2006ontonotes,sylak2015language}, has made cross-lingual transfer a popular NLP research topic. The need to transfer lexical knowledge across languages has given rise to cross-lingual word embedding models, i.e., cross-lingual representations of words in a joint embedding space, as illustrated in Figure \ref{fig:shared_embedding_space}.&#10;&#10;Cross-lingual word embeddings are appealing for two reasons: First, they enable us {to compare the meaning of words across languages}, which is key to bilingual lexicon induction, machine translation, or cross-lingual information retrieval, for example. Second, cross-lingual word embeddings {enable model transfer between languages}, e.g., between resource-rich and low-resource languages, by providing a common representation space. This duality is also reflected in how cross-lingual word embeddings are evaluated, as discussed in Section~\ref{sec:evaluation}.&#10;&#10;Many models for learning cross-lingual embeddings have been proposed in recent years. In this survey, we will give a comprehensive overview of existing cross-lingual word embedding models. One of the main goals of this survey is to show the similarities and differences between these approaches. To facilitate this, we first introduce a common notation and terminology in Section \ref{sec:notation_terminology}. Over the course of the survey, we then show that existing cross-lingual word embedding models can be seen as optimizing very similar objectives, where the main source of variation is due to the data used, the monolingual and regularization objectives employed, and how these are optimized. As many cross-lingual word embedding models are inspired by monolingual models, we introduce the most commonly used monolingual embedding models in Section \ref{sec:monolingual_models}. We then motivate and introduce one of the main contributions of this survey, a typology of cross-lingual embedding models in Section \ref{sec:typology}. The typology is based on the main differentiating aspect of cross-lingual embedding models: the nature of the data they require, in particular the type of alignment across languages (alignment of words, sentences, or documents), and whether data is assumed to be parallel or just comparable (about the same topic). The typology allows us to outline similarities and differences more concisely, but also starkly contrasts focal points of research with fruitful directions that have so far gone mostly unexplored. &#10;&#10;Since the idea of cross-lingual representations of words pre-dates word embeddings, we provide a brief history of cross-lingual word representations in Section \ref{sec:brief_history}. Subsequent sections are dedicated to each type of alignment. We discuss cross-lingual word embedding algorithms that rely on word-level alignments in Section \ref{sec:word_level_alignment_models}. Such methods can be further divided into mapping-based approaches, approaches based on pseudo-bilingual corpora, and joint methods. We show that these approaches, modulo optimization strategies and hyper-parameters, are nevertheless often equivalent. We then discuss approaches that rely on sentence-level alignments in Section \ref{sec:sentence_level_alignment_models}, and models that require document-level alignments in Section \ref{sec:document_level_alignment_models}. In Section \ref{sec:multilingual_training}, we describe how many bilingual approaches that deal with a pair of languages can be extended to the multilingual setting. We subsequently provide an extensive discussion of the tasks, benchmarks, and challenges of the evaluation of cross-lingual embedding models in Section \ref{sec:evaluation} and outline applications in Section \ref{sec:applications}. We present general challenges and future research directions in learning cross-lingual word representations in Section \ref{sec:challenges}. Finally, we provide our conclusions in Section \ref{sec:conclusion}.&#10;&#10;This survey makes the following contributions:&#10;\begin{enumerate}&#10;\item It proposes a general typology that characterizes the differentiating features of cross-lingual word embedding models and provides a compact overview of these models.&#10;\item It standardizes terminology and notation and shows that many cross-lingual word embedding models can be cast as optimizing nearly the same objective functions.\item It provides a proof that connects the three types of word-level alignment models and shows that these models are optimizing roughly the same objective.&#10;\item It critically examines the standard ways of evaluating cross-lingual embedding models.&#10;\item It describes multilingual extensions for the most common types of cross-lingual embedding models.&#10;\item It outlines outstanding challenges for learning cross-lingual word embeddings and provides suggestions for fruitful and unexplored research directions.&#10;\end{enumerate}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Cross-Lingual Models, Multilingual Representations, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1909.01482" label="1909.01482">
        <attvalues>
          <attvalue for="0" value="Target Language-Aware Constrained Inference for Cross-lingual Dependency&#10;  Parsing" />
          <attvalue for="1" value="  Prior work on cross-lingual dependency parsing often focuses on capturing the&#10;commonalities between source and target languages and overlooks the potential&#10;of leveraging linguistic properties of the languages to facilitate the&#10;transfer. In this paper, we show that weak supervisions of linguistic knowledge&#10;for the target languages can improve a cross-lingual graph-based dependency&#10;parser substantially. Specifically, we explore several types of corpus&#10;linguistic statistics and compile them into corpus-wise constraints to guide&#10;the inference process during the test time. We adapt two techniques, Lagrangian&#10;relaxation and posterior regularization, to conduct inference with&#10;corpus-statistics constraints. Experiments show that the Lagrangian relaxation&#10;and posterior regularization inference improve the performances on 15 and 17&#10;out of 19 target languages, respectively. The improvements are especially&#10;significant for target languages that have different word order features from&#10;the source language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;Cross-Lingual Transfer for Parsing &#10;&#10;Many approaches have been developed to transfer a dependency parser. However, they mainly focus on better capture information from the source language(s). &#10; \cite{mcdonald2011multi,guo2016representation,tackstrom2013target,chen2019multi} consider transferring a parser trained on multiple source languages. \cite{agic2017cross,lin2019choosing} selects good source languages by comparing part-of-speech tags sequences. \cite{sogaard2011data, tackstrom2013target} chooses suitable data points from the source language. &#10; \cite{pires2019how} uses multilingual BERT to leverage language features from multiple languages. \cite{ahmad2018near} design an order-free model to take out the order features from the source language.&#10; \cite{xiao2014distributed,guo2015cross} learn an alignment from source words to target words.&#10; \cite{ponti2018isomorphic} learn an anisomorphism from the source parsing tree to target. &#10; \cite{rasooli2019low} reorder the source data before training.&#10; In contrast, we focus on incorporating linguistic properties in the target languages. &#10;&#10;Constrained Inference for Parsing &#10; Several previous studies show that adding constraints in inference time improves the performance of models. \cite{grave2015convex} consider incorporating constraints to promote popular types of arcs in an unsupervised setting. \cite{naseem2010using,li2019dependency} train a parser with constraints compiled from the frequency of particular arcs. Compared with the previous work, we focus on cross-lingual transfer with word order constraints.&#10;&#10;Finally, prior studies have noticed that the word order information is significant for parsing and use it as features~\cite{ammar2016many, naseem2012selective, rasooli2017cross, zhang2015hierarchical,dryer2007word}.&#10; \cite{tackstrom2013target} further propose to decompose these features from models for adapting target languages. \cite{wang2018tacl} use the statistics of surface part-of-speech (POS) tags of target languages to learn the word order. \cite{wang2018synthetic} use POS tags of target languages together with a similar language, and design a stochastic permutation process to synthetic the word order. However, none of them consider using the word order features as constraints. &#10;&#10;Incorporating Constraints In NLP Tasks&#10; Constraints are widely incorporated in variety of NLP tasks. To name a few, \cite{roth2004linear} propose to formulate constrained inferences in NLP as integer linear programming problems. To solve the intractable structure, \cite{rush2012tutorial} decompose the structure and incorporate constraints on some composite tasks.&#10; To improve the performance of a model, \cite{chang2011exact,peng2015dual} incorporate constraints on exact decoding tasks and inference tasks on graphical models, and \cite{chang2013tractable,dalvi2015constrained,martins2015transferring} incorporate corpus-level constraints on semi-supervised multilabel classification and coreference resolution.&#10; \cite{jieyu2017men} incorporate corpus-level constraints to avoid amplifying gender bias on visual semantic role labeling and multilabel classification. &#10; In contrast to previous work, we incorporate corpus-level constraints to facilitate dependency parser in the cross-lingual transfer setting.&#10;&#10;" />
          <attvalue for="4" value="Corpus Statistics Constraints, Machine Learning, Computer Science, Linguistics, Cross-Lingual Parsing, Mathematics, Linguistic Knowledge Transfer, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2301.12314" label="2301.12314">
        <attvalues>
          <attvalue for="0" value="Progressive Prompts: Continual Learning for Language Models" />
          <attvalue for="1" value="  We introduce Progressive Prompts - a simple and efficient approach for&#10;continual learning in language models. Our method allows forward transfer and&#10;resists catastrophic forgetting, without relying on data replay or a large&#10;number of task-specific parameters. Progressive Prompts learns a new soft&#10;prompt for each task and sequentially concatenates it with the previously&#10;learned prompts, while keeping the base model frozen. Experiments on standard&#10;continual learning benchmarks show that our approach outperforms&#10;state-of-the-art methods, with an improvement &gt;20% in average test accuracy&#10;over the previous best-preforming method on T5 model. We also explore a more&#10;challenging continual learning setup with longer sequences of tasks and show&#10;that Progressive Prompts significantly outperforms prior methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Models, Task Adaptation, Continual Learning, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2306.05083" label="2306.05083">
        <attvalues>
          <attvalue for="0" value="Revealing the Blind Spot of Sentence Encoder Evaluation by HEROS" />
          <attvalue for="1" value="  Existing sentence textual similarity benchmark datasets only use a single&#10;number to summarize how similar the sentence encoder's decision is to humans'.&#10;However, it is unclear what kind of sentence pairs a sentence encoder (SE)&#10;would consider similar. Moreover, existing SE benchmarks mainly consider&#10;sentence pairs with low lexical overlap, so it is unclear how the SEs behave&#10;when two sentences have high lexical overlap. We introduce a high-quality SE&#10;diagnostic dataset, HEROS. HEROS is constructed by transforming an original&#10;sentence into a new sentence based on certain rules to form a \textit{minimal&#10;pair}, and the minimal pair has high lexical overlaps. The rules include&#10;replacing a word with a synonym, an antonym, a typo, a random word, and&#10;converting the original sentence into its negation. Different rules yield&#10;different subsets of HEROS. By systematically comparing the performance of over&#10;60 supervised and unsupervised SEs on HEROS, we reveal that most unsupervised&#10;sentence encoders are insensitive to negation. We find the datasets used to&#10;train the SE are the main determinants of what kind of sentence pairs an SE&#10;considers similar. We also show that even if two SEs have similar performance&#10;on STS benchmarks, they can have very different behavior on HEROS. Our result&#10;reveals the blind spot of traditional STS benchmarks when evaluating SEs.&#10;" />
          <attvalue for="2" value="&#10;Sentence encoders (SEs) are fundamental building blocks in miscellaneous natural language processing (NLP) tasks, including natural language inference, paraphrase identification, and retrieval~\cite{gillick2018end, lan2018neural}.&#10;SEs are mostly evaluated with the semantic textual similarity (STS) datasets~\cite{agirre-etal-2016-semeval, cer-etal-2017-semeval} and SICK-R~\cite{marelli2014sick}, which consist of sentence pairs with human-labeled similarity scores.&#10;The performance of the SEs is summarized using Spearman’s correlation coefficient between the human-labeled similarity and the cosine similarity obtained from the SE.&#10;&#10;While the STS benchmarks are widely adopted, there are two problems with these benchmarks.&#10;First, the performance on the STS dataset does not reveal much about what kind of sentence pairs would the SE deem similar.&#10;Spearman's correlation coefficient only tells us how correlated the sentence embedding cosine similarity and the ground truth similarity are.&#10;However, the idea of what is similar can vary among different people and depend on the task at hand. &#10;Therefore, just because the sentence embedding cosine similarity is strongly correlated to the ground truth similarity, it does not provide much information about the specific type of similarity that the SE captures.&#10;Prior works mostly resort to a few hand-picked examples to illustrate what kind of sentence pairs an SE would consider similar or dissimilar~\cite{gao2021simcse,chuang-etal-2022-diffcse,wang2022sncse}.&#10;But it is hard to fully understand the traits of an SE by using only a few hand-picked samples.&#10;&#10;The second issue is that sentence pairs in the STS-related benchmarks often have low lexical overlaps, as shown in Table~\ref{tab:statstic}, making it unclear how the SEs will perform on sentence pairs with high lexical overlaps, which exist in real-world applications such as adversarial attacks in NLP. &#10;Adversarial samples in NLP are constructed by replacing some words in an original sentence with some other words~\cite{alzantot-etal-2018-generating}, and the original sentence and the adversarial sample will have high lexical overlaps.&#10;SEs are often adopted to check the semantic similarity between the original sentence and the adversarial sample~\cite{garg-ramakrishnan-2020-bae,li-etal-2020-bert-attack}.&#10;If we do not know how SEs perform on high lexical overlap sentences, using them to check semantic similarity is meaningless.&#10;&#10;To address the above issues, we construct and release a new dataset, Heros: \underline{H}igh-l\underline{e}xical ove\underline{r}lap diagn\underline{o}stic dataset for \underline{s}entence encoders, for evaluating SEs.&#10;Heros is composed of six subsets, and each subset includes 1000 sentence pairs with very high lexical overlaps.&#10;For the two sentences in a sentence pair, one of them is created by modifying the other sentence based on certain rules, and each subset adopts a different rule.&#10;These rules are (1) replacing a word with a synonym, (2) replacing a word with an antonym, (3) replacing a word with a random word, (4) replacing a word with its typo, and (5,6) negating the sentence.&#10;By comparing the sentence embedding cosine similarity of sentence pairs in different subsets, we can understand what kind of sentence pairs, when they have high lexical overlaps, would be considered similar by an SE.&#10;We evaluate 60 sentence embedding models on Heros and reveal many intriguing and unreported observations on these SEs.&#10;&#10;While some prior works also crafted sentence pairs to understand the performance of SEs, they either do not make the datasets publicly available~\cite{zhu2018exploring,zhu-de-melo-2020-sentence} or do not consider so many SEs as our paper does~\cite{barancikova-bojar-2020-costra}, especially unsupervised SEs.&#10;Our contribution is relevant and significant as it provides a detailed understanding of SEs using a new dataset.&#10;The contribution and findings of this paper are summarized as follows:&#10;\begin{itemize}&#10; \item We release Heros, a high-quality dataset consisting of 6000 sentence pairs with high lexical overlaps.&#10; Heros allows researchers to systematically evaluate what sentence pairs would be considered similar by SEs when the lexical overlap is high.&#10;&#10;\item We evaluate 60 SEs on Heros and reveal several facts that were never reported before or only studied using a few hand-picked examples.&#10;&#10;\item We show that supervised SEs trained for different downstream tasks behave differently on different subsets of Heros, indicating that the SEs for different tasks encode different concepts of similarity.&#10;&#10;\item We find that all unsupervised SEs are considerably insensitive to negation, and further fine-tuning on NLI datasets makes them acquire the concept of negation. &#10;&#10;\item We observe that SEs can have very different performances on different subsets of Heros even if their average STS benchmark performance difference is less than 0.2 points.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Sentence Encoder Evaluation, Linguistics, Textual Similarity Benchmarks, Computational Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1811.08008" label="1811.08008">
        <attvalues>
          <attvalue for="0" value="End-to-End Retrieval in Continuous Space" />
          <attvalue for="1" value="  Most text-based information retrieval (IR) systems index objects by words or&#10;phrases. These discrete systems have been augmented by models that use&#10;embeddings to measure similarity in continuous space. But continuous-space&#10;models are typically used just to re-rank the top candidates. We consider the&#10;problem of end-to-end continuous retrieval, where standard approximate nearest&#10;neighbor (ANN) search replaces the usual discrete inverted index, and rely&#10;entirely on distances between learned embeddings. By training simple models&#10;specifically for retrieval, with an appropriate model architecture, we improve&#10;on a discrete baseline by 8% and 26% (MAP) on two similar-question retrieval&#10;tasks. We also discuss the problem of evaluation for retrieval systems, and&#10;show how to modify existing pairwise similarity datasets for this purpose.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.05979" label="2201.05979">
        <attvalues>
          <attvalue for="0" value="SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with&#10;  Soft Negative Samples" />
          <attvalue for="1" value="  Unsupervised sentence embedding aims to obtain the most appropriate embedding&#10;for a sentence to reflect its semantic. Contrastive learning has been&#10;attracting developing attention. For a sentence, current models utilize diverse&#10;data augmentation methods to generate positive samples, while consider other&#10;independent sentences as negative samples. Then they adopt InfoNCE loss to pull&#10;the embeddings of positive pairs gathered, and push those of negative pairs&#10;scattered. Although these models have made great progress on sentence&#10;embedding, we argue that they may suffer from feature suppression. The models&#10;fail to distinguish and decouple textual similarity and semantic similarity.&#10;And they may overestimate the semantic similarity of any pairs with similar&#10;textual regardless of the actual semantic difference between them. This is&#10;because positive pairs in unsupervised contrastive learning come with similar&#10;and even the same textual through data augmentation. To alleviate feature&#10;suppression, we propose contrastive learning for unsupervised sentence&#10;embedding with soft negative samples (SNCSE). Soft negative samples share&#10;highly similar textual but have surely and apparently different semantic with&#10;the original samples. Specifically, we take the negation of original sentences&#10;as soft negative samples, and propose Bidirectional Margin Loss (BML) to&#10;introduce them into traditional contrastive learning framework, which merely&#10;involves positive and negative samples. Our experimental results show that&#10;SNCSE can obtain state-of-the-art performance on semantic textual similarity&#10;(STS) task with average Spearman's correlation coefficient of 78.97% on&#10;BERTbase and 79.23% on RoBERTabase. Besides, we adopt rank-based error analysis&#10;method to detect the weakness of SNCSE for future study.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Reimers and Gurevych \shortcite{reimers:sentence-bert}, Li et al. \shortcite{bohan:bert-flow}, and Yan et al. \shortcite{yan:consert} find that vanilla sentence embeddings from pretrained language models are of low quality to reflect the semantic, where the embeddings of any sentence pairs are of high similarity.&#10;To handle the problem, Reimers and Gurevych \shortcite{reimers:sentence-bert} propose Sentence-BERT, which trains the encoders through textual entailment tasks in a supervised way.&#10;Early unsupervised sentence embedding models include BERT-flow \cite{bohan:bert-flow} and BERT-Whitening \cite{su:bert-whitening}, which are respectively based on reversible transformation and whitening operation.&#10;Yan et al. \shortcite{yan:consert} fistly introduce contrastive learning for unsupervised sentence embedding, and propose several data augmentation methods.&#10;Gao et al. \shortcite{gao:simcse} propose SimCSE, which considers dropout operation as a kind of data augmentation method.&#10;Wu et al. \shortcite{wu:esimcse} point out that SimCSE has the bias to overestimate the semantic similarity of sentence pairs with the same length.&#10;To eliminate the bias, they propose ESimCSE with word repetition as data augmentation.&#10;They also introduce momentum contrastive \cite{he:moco,chen:mocov2} for sentence embedding.&#10;Differently, in this paper, we mainly focus on feature suppression.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1804.07998" label="1804.07998">
        <attvalues>
          <attvalue for="0" value="Generating Natural Language Adversarial Examples" />
          <attvalue for="1" value="  Deep neural networks (DNNs) are vulnerable to adversarial examples,&#10;perturbations to correctly classified examples which can cause the model to&#10;misclassify. In the image domain, these perturbations are often virtually&#10;indistinguishable to human perception, causing humans and state-of-the-art&#10;models to disagree. However, in the natural language domain, small&#10;perturbations are clearly perceptible, and the replacement of a single word can&#10;drastically alter the semantics of the document. Given these challenges, we use&#10;a black-box population-based optimization algorithm to generate semantically&#10;and syntactically similar adversarial examples that fool well-trained sentiment&#10;analysis and textual entailment models with success rates of 97% and 70%,&#10;respectively. We additionally demonstrate that 92.3% of the successful&#10;sentiment analysis adversarial examples are classified to their original label&#10;by 20 human annotators, and that the examples are perceptibly quite similar.&#10;Finally, we discuss an attempt to use adversarial training as a defense, but&#10;fail to yield improvement, demonstrating the strength and diversity of our&#10;adversarial examples. We hope our findings encourage researchers to pursue&#10;improving the robustness of DNNs in the natural language domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.01970" label="2004.01970">
        <attvalues>
          <attvalue for="0" value="BAE: BERT-based Adversarial Examples for Text Classification" />
          <attvalue for="1" value="  Modern text classification models are susceptible to adversarial examples,&#10;perturbed versions of the original text indiscernible by humans which get&#10;misclassified by the model. Recent works in NLP use rule-based synonym&#10;replacement strategies to generate adversarial examples. These strategies can&#10;lead to out-of-context and unnaturally complex token replacements, which are&#10;easily identifiable by humans. We present BAE, a black box attack for&#10;generating adversarial examples using contextual perturbations from a BERT&#10;masked language model. BAE replaces and inserts tokens in the original text by&#10;masking a portion of the text and leveraging the BERT-MLM to generate&#10;alternatives for the masked tokens. Through automatic and human evaluations, we&#10;show that BAE performs a stronger attack, in addition to generating adversarial&#10;examples with improved grammaticality and semantic coherence as compared to&#10;prior work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.09984" label="2004.09984">
        <attvalues>
          <attvalue for="0" value="BERT-ATTACK: Adversarial Attack Against BERT Using BERT" />
          <attvalue for="1" value="  Adversarial attacks for discrete data (such as texts) have been proved&#10;significantly more challenging than continuous data (such as images) since it&#10;is difficult to generate adversarial samples with gradient-based methods.&#10;Current successful attack methods for texts usually adopt heuristic replacement&#10;strategies on the character or word level, which remains challenging to find&#10;the optimal solution in the massive space of possible combinations of&#10;replacements while preserving semantic consistency and language fluency. In&#10;this paper, we propose \textbf{BERT-Attack}, a high-quality and effective&#10;method to generate adversarial samples using pre-trained masked language models&#10;exemplified by BERT. We turn BERT against its fine-tuned models and other deep&#10;neural models in downstream tasks so that we can successfully mislead the&#10;target models to predict incorrectly. Our method outperforms state-of-the-art&#10;attack strategies in both success rate and perturb percentage, while the&#10;generated adversarial samples are fluent and semantically preserved. Also, the&#10;cost of calculation is low, thus possible for large-scale generations. The code&#10;is available at https://github.com/LinyangLee/BERT-Attack.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.01673" label="1912.01673">
        <attvalues>
          <attvalue for="0" value="COSTRA 1.0: A Dataset of Complex Sentence Transformations" />
          <attvalue for="1" value="  We present COSTRA 1.0, a dataset of complex sentence transformations. The&#10;dataset is intended for the study of sentence-level embeddings beyond simple&#10;word alternations or standard paraphrasing. This first version of the dataset&#10;is limited to sentences in Czech but the construction method is universal and&#10;we plan to use it also for other languages. The dataset consist of 4,262 unique&#10;sentences with average length of 10 words, illustrating 15 types of&#10;modifications such as simplification, generalization, or formal and informal&#10;language variation. The hope is that with this dataset, we should be able to&#10;test semantic properties of sentence embeddings and perhaps even to find some&#10;topologically interesting 'skeleton' in the sentence embedding space. A&#10;preliminary analysis using LASER, multi-purpose multi-lingual sentence&#10;embeddings suggests that the LASER space does not exhibit the desired&#10;properties.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.00901" label="2009.00901">
        <attvalues>
          <attvalue for="0" value="A Practical Chinese Dependency Parser Based on A Large-scale Dataset" />
          <attvalue for="1" value="  Dependency parsing is a longstanding natural language processing task, with&#10;its outputs crucial to various downstream tasks. Recently, neural network based&#10;(NN-based) dependency parsing has achieved significant progress and obtained&#10;the state-of-the-art results. As we all know, NN-based approaches require&#10;massive amounts of labeled training data, which is very expensive because it&#10;requires human annotation by experts. Thus few industrial-oriented dependency&#10;parser tools are publicly available. In this report, we present Baidu&#10;Dependency Parser (DDParser), a new Chinese dependency parser trained on a&#10;large-scale manually labeled dataset called Baidu Chinese Treebank (DuCTB).&#10;DuCTB consists of about one million annotated sentences from multiple sources&#10;including search logs, Chinese newswire, various forum discourses, and&#10;conversation programs. DDParser is extended on the graph-based biaffine parser&#10;to accommodate to the characteristics of Chinese dataset. We conduct&#10;experiments on two test sets: the standard test set with the same distribution&#10;as the training set and the random test set sampled from other sources, and the&#10;labeled attachment scores (LAS) of them are 92.9% and 86.9% respectively.&#10;DDParser achieves the state-of-the-art results, and is released at&#10;https://github.com/baidu/DDParser.&#10;" />
          <attvalue for="2" value="&#10;&#10;Dependency parsing aims to annotate sentences into a dependency tree which is designed to be easy for humans and computers alike to understand. Given an input sentence $s=w_0w_1...w_n$, a dependency tree, as depicted in Figure \ref{fig:intro_case}, is defined as $d=\{(h,m,l), 0 \leq h \leq n, 1 \leq m \leq n, l \in \pounds\}$, where $(h,m,l)$ is a dependency from the head word $w_h$ to the modifier word $w_m$ with the relation label $l \in \pounds$, and $w_0$ is a pseudo word that points to the root word. As a fundamental task in natural language processing (NLP), dependency parsing has been found to be extremely useful for a sizable number of NLP tasks, especially those involving natural language understanding in some way \cite{bowman2016fast, angeli2015leveraging, levy2014dependency, toutanova2016compositional, parikh2015grounded}.&#10;&#10;In recent years, NN-based approaches have achieved remarkable improvement and outperformed the traditional discrete-feature based approaches in dependency parsing by a large margin \cite{chen2014fast, dyer2015transition}.&#10;\cite{dozat2016deep} propose a simple yet effective deep biaffine graph-based parser and achieve the state-of-the-art accuracy on a variety of datasets and languages.&#10;Based on this work, \cite{li2019self} applies the self-attention based encoder to dependency parsing as the replacement of BiLSTMs, and then make an in-depth study on the the differences between the two techniques. As we all known, labeled data is very critical for all NN-based approaches, including data size, annotation quality and so on. However, it is difficult to build a large-scale dependency parsing dataset by human annotation.&#10;&#10;After about a decade of accumulation and innovation, Baidu has established a Chinese dependency parsing dataset (DuCTB) with a scale of nearly one million, covering multiple sources such as search logs, Chinese newswire, forum discourses. Then an effective dependency parsing tool is trained based on DuCTB, achieving the state-of-the-art results. In order to help ordinary users to obtain the syntactic and semantic information of sentences, we release our dependency parser including the source code and trained model. Our parser has three advantages: 1) the training data consists of more than 500,000 sentences, covering news, conversations and search queries, etc; 2) it outperforms other dependency parsers both on the labeled attachment score (LAS) and unlabeled attachment score (UAS); 3) it is very convenient to use, as the installation and prediction can be implemented with a single command.&#10;&#10;We release our model and source code at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Neural Network Models, Artificial Intelligence, Natural Language Processing, Dependency Parsing" />
        </attvalues>
      </node>
      <node id="2101.10649" label="2101.10649">
        <attvalues>
          <attvalue for="0" value="Analyzing Zero-shot Cross-lingual Transfer in Supervised NLP Tasks" />
          <attvalue for="1" value="  In zero-shot cross-lingual transfer, a supervised NLP task trained on a&#10;corpus in one language is directly applicable to another language without any&#10;additional training. A source of cross-lingual transfer can be as&#10;straightforward as lexical overlap between languages (e.g., use of the same&#10;scripts, shared subwords) that naturally forces text embeddings to occupy a&#10;similar representation space. Recently introduced cross-lingual language model&#10;(XLM) pretraining brings out neural parameter sharing in Transformer-style&#10;networks as the most important factor for the transfer. In this paper, we aim&#10;to validate the hypothetically strong cross-lingual transfer properties induced&#10;by XLM pretraining. Particularly, we take XLM-RoBERTa (XLMR) in our experiments&#10;that extend semantic textual similarity (STS), SQuAD and KorQuAD for machine&#10;reading comprehension, sentiment analysis, and alignment of sentence embeddings&#10;under various cross-lingual settings. Our results indicate that the presence of&#10;cross-lingual transfer is most pronounced in STS, sentiment analysis the next,&#10;and MRC the last. That is, the complexity of a downstream task softens the&#10;degree of crosslingual transfer. All of our results are empirically observed&#10;and measured, and we make our code and data publicly available.&#10;" />
          <attvalue for="2" value="&#10;Pretraining language models at a large scale has dramatically improved natural language understanding. According to a comprehensive analysis~\cite{xlm-r} on the limitations in pretraining a multi-lingual model, more languages lead to better cross-lingual performance for low-resource languages only up to a certain point when the number of languages increases. The phenomenon is dubbed the curse of multilinguality, which can only be freed up by scaling up the model size. &#10;&#10;The recent experimental results show that multilingual models can outperform their monolingual counterparts. For a low-resource language that lacks in labeled examples, such results are an encouraging breakthrough for building an NLP application for low-resource languages. In cross-lingual language understanding, XLM by Conneau \&amp; Lample~\cite{xlm}, despite being pretrained by only masked language modeling (MLM), has reported the state-of-the-art on downstream benchmarks. Shared lexical features (\eg, subwords, scripts, anchor points) across languages have been suspected for the primary source of learning language-independent representation that leads to cross-lingual transfer. Recent studies, however, show that parameter sharing induced by the Transformer architecture is instead the most attributable factor for the transfer.&#10;&#10;We are motivated by these progresses in language modeling. This work focuses on empirical analysis of cross-lingual transfer in supervised NLP tasks fine-tuned over XLM. In particular, we are interested in zero-shot transfer settings where no additional training is done using the target language examples after being fine-tuned in the source language. We experiment with XLM-RoBERTa (XLM-R)~\cite{xlm-r}, a large XLM model with 550 million parameters and a 250k vocabulary size by extending semantic textual similarity, SQuAD~\cite{squad} \&amp; KorQuAD~\cite{korquad} question answering, and sentiment classifications for various cross-lingual settings. &#10;&#10;At last, beyond previous work that has attempted to align word embeddings across different languages~\cite{w-mapping}, we compute a projection that directly maps sentence embeddings of one language to those of another. We then analyze the effect of fine-grained alignment of sentences across different languages to the quality of zero-shot cross-lingual transfer, manifested through the aforementioned NLP task performances measured empirically.&#10;&#10;We make the following contributions. &#10;\begin{itemize}\itemsep 0pt&#10;\item We provide rigorous results on cross-lingual transfer present in three important supervised NLP tasks that require high-level natural language understanding, namely STS, MRC, and sentiment classification. &#10;\item We propose to directly compute a cross-lingual mapping that aligns sentence embeddings of different languages whereas previous work has focused on word-level embeddings. &#10;\item We furthermore show benefits of the fine-grained cross-lingual sentence alignment that enables directly comparing sentences from different languages for sentence-pair regression tasks.&#10;\end{itemize}&#10;&#10;The rest of this paper is organized as follows. In Section II, we describe our approach by presenting the zero-shot cross-lingual evaluation framework. Section III discusses our experimental methodology and empirical results. Section IV concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual NLP Tasks, Cross-Lingual Transfer, Artificial Intelligence, Language Model Pretraining, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1909.07005" label="1909.07005">
        <attvalues>
          <attvalue for="0" value="KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension" />
          <attvalue for="1" value="  Machine Reading Comprehension (MRC) is a task that requires machine to&#10;understand natural language and answer questions by reading a document. It is&#10;the core of automatic response technology such as chatbots and automatized&#10;customer supporting systems. We present Korean Question Answering&#10;Dataset(KorQuAD), a large-scale Korean dataset for extractive machine reading&#10;comprehension task. It consists of 70,000+ human generated question-answer&#10;pairs on Korean Wikipedia articles. We release KorQuAD1.0 and launch a&#10;challenge at https://KorQuAD.github.io to encourage the development of&#10;multilingual natural language processing research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.09397" label="2203.09397">
        <attvalues>
          <attvalue for="0" value="Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive&#10;  Bias to Sequence-to-sequence Models" />
          <attvalue for="1" value="  Relations between words are governed by hierarchical structure rather than&#10;linear ordering. Sequence-to-sequence (seq2seq) models, despite their success&#10;in downstream NLP applications, often fail to generalize in a&#10;hierarchy-sensitive manner when performing syntactic transformations - for&#10;example, transforming declarative sentences into questions. However, syntactic&#10;evaluations of seq2seq models have only observed models that were not&#10;pre-trained on natural language data before being trained to perform syntactic&#10;transformations, in spite of the fact that pre-training has been found to&#10;induce hierarchical linguistic generalizations in language models; in other&#10;words, the syntactic capabilities of seq2seq models may have been greatly&#10;understated. We address this gap using the pre-trained seq2seq models T5 and&#10;BART, as well as their multilingual variants mT5 and mBART. We evaluate whether&#10;they generalize hierarchically on two transformations in two languages:&#10;question formation and passivization in English and German. We find that&#10;pre-trained seq2seq models generalize hierarchically when performing syntactic&#10;transformations, whereas models trained from scratch on syntactic&#10;transformations do not. This result presents evidence for the learnability of&#10;hierarchical syntactic information from non-annotated natural language text&#10;while also demonstrating that seq2seq models are capable of syntactic&#10;generalization, though only after exposure to much more language data than&#10;human learners receive.&#10;" />
          <attvalue for="2" value="&#10;&#10;Human language is structured hierarchically. In NLP tasks like natural language inference, syntactic competence is a prerequisite for robust generalization \cite[e.g.,][]{mccoy2019hans}. Probing studies have found that masked language models (MLMs) contain hierarchical representations \cite{tenney2019bert,hewitt2019structural,clark-etal-2019-bert}, while behavioral studies of recurrent neural language models \cite{linzen2016assessing,marvin2018targeted,wilcox2018rnn,vanschijndel2019quantity} and MLMs \cite{goldberg2019bert,hu2020systematic} have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences.&#10;&#10;Recent evidence suggests that MLMs like BERT \cite{devlin2019bert} and RoBERTa \cite{liu2019roberta} can learn to make hierarchical linguistic generalizations through exposure to text \cite{warstadt2020linguistic}, though acquiring many of these linguistic generalizations requires large amounts of data \cite{warstadt2020learning}. However, this evidence comes from binary acceptability judgment tasks, where a classifier head is attached to an MLM and the model is fine-tuned to classify which sentence in a given minimal pair is consistent with a hierarchical linguistic generalization, rather than a positional surface heuristic. Consider the following two transformations of Example~\ref{ex:bias}:&#10;&#10;\ex.The yak that your unicorns {have} amused {hasn't} entertained a newt.&#10; \a. {Hasn't} the yak that your unicorns {have} amused entertained a newt?&#10; \b. *{Have} the yak that your unicorns amused hasn't entertained a newt?&#10;&#10;Example~\ref{ex:bias_correct} correctly forms the question by moving the main auxiliary verb to the front of the sentence, while \ref{ex:bias_incorrect} relies on the incorrect positional heuristic that the first auxiliary in the declarative sentence should be moved to the front of the sentence. When differentiating grammatical and ungrammatical auxiliary movements, a model could rely on distributional information \cite{lewis2001learnability} such as bigram heuristics \cite{reali2005uncovering,kam2008bigrams} to make correct judgments in many cases, so high performance on binary classification tasks may overstate the syntactic competence of a model.&#10;&#10;By contrast, performing a syntactic trans\-formation---e.g., given a declarative sentence like Example~\ref{ex:bias} as input, transforming it into a polar question like \ref{ex:bias_correct}---is more difficult. It requires multiple complex but systematic operations that rely on hierarchical structure, including movement, number agreement, and---in languages that have grammatical case, such as German---case reinflection. Evaluations of syntactic transformational abilities can therefore act as more targeted behavioral indicators of syntactic structural representations in neural models. \cite{mccoy2018poverty} evaluate non-pre-trained recurrent sequence-to-sequence (seq2seq) models \cite{sutskever2014sequence} on the question formation task, finding that they rely on linear/positional surface heuristics rather than hierarchical structure to perform this syntactic transformation. More recent studies have also exclusively considered recurrent seq2seq models and Transformer models \cite{petty2021transformers} trained from scratch on other transformations like tense reinflection \cite{mccoy2020trees} and passivization \cite{mulligan2021structure}, finding similar results. These studies were designed to understand the inductive biases of various seq2seq architectures, which is why they do not pre-train the models on non-annotated natural language data before training them to perform syntactic transformations.&#10;&#10;In this study, we create German datasets and modify English datasets for evaluating the inductive biases of pre-trained models. We use these datasets to analyze performance in monolingual and zero-shot cross-lingual settings. Further, we analyze how pre-trained models perform syntactic transformations. Our findings indicate that pre-trained models generally perform syntactic transformations in a hierarchy-sensitive manner, while non-pre-trained models (including randomized-weight versions of pre-trained models) rely primarily on linear/positional heuristics to perform the transformations. This finding presents additional evidence to \cite{warstadt2020learning} and \cite{warstadt2020linguistic} for the learnability of hierarchical syntactic information from natural language text input. Our code and data are publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Syntactic Transformations, Computer Science, Linguistics, Language Model Evaluation, Hierarchical Linguistic Generalizations, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1906.04341" label="1906.04341">
        <attvalues>
          <attvalue for="0" value="What Does BERT Look At? An Analysis of BERT's Attention" />
          <attvalue for="1" value="  Large pre-trained neural networks such as BERT have had great recent success&#10;in NLP, motivating a growing body of research investigating what aspects of&#10;language they are able to learn from unlabeled data. Most recent analysis has&#10;focused on model outputs (e.g., language model surprisal) or internal vector&#10;representations (e.g., probing classifiers). Complementary to these works, we&#10;propose methods for analyzing the attention mechanisms of pre-trained models&#10;and apply them to BERT. BERT's attention heads exhibit patterns such as&#10;attending to delimiter tokens, specific positional offsets, or broadly&#10;attending over the whole sentence, with heads in the same layer often&#10;exhibiting similar behaviors. We further show that certain attention heads&#10;correspond well to linguistic notions of syntax and coreference. For example,&#10;we find heads that attend to the direct objects of verbs, determiners of nouns,&#10;objects of prepositions, and coreferent mentions with remarkably high accuracy.&#10;Lastly, we propose an attention-based probing classifier and use it to further&#10;demonstrate that substantial syntactic information is captured in BERT's&#10;attention.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.01368" label="1611.01368">
        <attvalues>
          <attvalue for="0" value="Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies" />
          <attvalue for="1" value="  The success of long short-term memory (LSTM) neural networks in language&#10;processing is typically attributed to their ability to capture long-distance&#10;statistical regularities. Linguistic regularities are often sensitive to&#10;syntactic structure; can such dependencies be captured by LSTMs, which do not&#10;have explicit structural representations? We begin addressing this question&#10;using number agreement in English subject-verb dependencies. We probe the&#10;architecture's grammatical competence both using training objectives with an&#10;explicit grammatical target (number prediction, grammaticality judgments) and&#10;using language models. In the strongly supervised settings, the LSTM achieved&#10;very high overall accuracy (less than 1% errors), but errors increased when&#10;sequential and structural information conflicted. The frequency of such errors&#10;rose sharply in the language-modeling setting. We conclude that LSTMs can&#10;capture a non-trivial amount of grammatical structure given targeted&#10;supervision, but stronger architectures may be required to further reduce&#10;errors; furthermore, the language modeling signal is insufficient for capturing&#10;syntax-sensitive dependencies, and should be supplemented with more direct&#10;supervision if such dependencies need to be captured.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.09031" label="1808.09031">
        <attvalues>
          <attvalue for="0" value="Targeted Syntactic Evaluation of Language Models" />
          <attvalue for="1" value="  We present a dataset for evaluating the grammaticality of the predictions of&#10;a language model. We automatically construct a large number of minimally&#10;different pairs of English sentences, each consisting of a grammatical and an&#10;ungrammatical sentence. The sentence pairs represent different variations of&#10;structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and&#10;negative polarity items. We expect a language model to assign a higher&#10;probability to the grammatical sentence than the ungrammatical one. In an&#10;experiment using this data set, an LSTM language model performed poorly on many&#10;of the constructions. Multi-task training with a syntactic objective (CCG&#10;supertagging) improved the LSTM's accuracy, but a large gap remained between&#10;its performance and the accuracy of human participants recruited online. This&#10;suggests that there is considerable room for improvement over LSTMs in&#10;capturing syntax in a language model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.05287" label="1901.05287">
        <attvalues>
          <attvalue for="0" value="Assessing BERT's Syntactic Abilities" />
          <attvalue for="1" value="  I assess the extent to which the recently introduced BERT model captures&#10;English syntactic phenomena, using (1) naturally-occurring subject-verb&#10;agreement stimuli; (2) &quot;coloreless green ideas&quot; subject-verb agreement stimuli,&#10;in which content words in natural sentences are randomly replaced with words&#10;sharing the same part-of-speech and inflection; and (3) manually crafted&#10;stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT&#10;model performs remarkably well on all cases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.06761" label="2007.06761">
        <attvalues>
          <attvalue for="0" value="Can neural networks acquire a structural bias from raw linguistic data?" />
          <attvalue for="1" value="  We evaluate whether BERT, a widely used neural network for sentence&#10;processing, acquires an inductive bias towards forming structural&#10;generalizations through pretraining on raw data. We conduct four experiments&#10;testing its preference for structural vs. linear generalizations in different&#10;structure-dependent phenomena. We find that BERT makes a structural&#10;generalization in 3 out of 4 empirical domains---subject-auxiliary inversion,&#10;reflexive binding, and verb tense detection in embedded clauses---but makes a&#10;linear generalization when tested on NPI licensing. We argue that these results&#10;are the strongest evidence so far from artificial learners supporting the&#10;proposition that a structural bias can be acquired from raw data. If this&#10;conclusion is correct, it is tentative evidence that some linguistic universals&#10;can be acquired by learners without innate biases. However, the precise&#10;implications for human language acquisition are unclear, as humans learn&#10;language from significantly less data than BERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05358" label="2010.05358">
        <attvalues>
          <attvalue for="0" value="Learning Which Features Matter: RoBERTa Acquires a Preference for&#10;  Linguistic Generalizations (Eventually)" />
          <attvalue for="1" value="  One reason pretraining on self-supervised linguistic tasks is effective is&#10;that it teaches models features that are helpful for language understanding.&#10;However, we want pretrained models to learn not only to represent linguistic&#10;features, but also to use those features preferentially during fine-turning.&#10;With this goal in mind, we introduce a new English-language diagnostic set&#10;called MSGS (the Mixed Signals Generalization Set), which consists of 20&#10;ambiguous binary classification tasks that we use to test whether a pretrained&#10;model prefers linguistic or surface generalizations during fine-tuning. We&#10;pretrain RoBERTa models from scratch on quantities of data ranging from 1M to&#10;1B words and compare their performance on MSGS to the publicly available&#10;RoBERTa-base. We find that models can learn to represent linguistic features&#10;with little pretraining data, but require far more data to learn to prefer&#10;linguistic generalizations over surface ones. Eventually, with about 30B words&#10;of pretraining data, RoBERTa-base does demonstrate a linguistic bias with some&#10;regularity. We conclude that while self-supervised pretraining is an effective&#10;way to learn helpful inductive biases, there is likely room to improve the rate&#10;at which models learn which features matter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.09091" label="1802.09091">
        <attvalues>
          <attvalue for="0" value="Revisiting the poverty of the stimulus: hierarchical generalization&#10;  without a hierarchical bias in recurrent neural networks" />
          <attvalue for="1" value="  Syntactic rules in natural language typically need to make reference to&#10;hierarchical sentence structure. However, the simple examples that language&#10;learners receive are often equally compatible with linear rules. Children&#10;consistently ignore these linear explanations and settle instead on the correct&#10;hierarchical one. This fact has motivated the proposal that the learner's&#10;hypothesis space is constrained to include only hierarchical rules. We examine&#10;this proposal using recurrent neural networks (RNNs), which are not constrained&#10;in such a way. We simulate the acquisition of question formation, a&#10;hierarchical transformation, in a fragment of English. We find that some RNN&#10;architectures tend to learn the hierarchical rule, suggesting that hierarchical&#10;cues within the language, combined with the implicit architectural biases&#10;inherent in certain RNNs, may be sufficient to induce hierarchical&#10;generalizations. The likelihood of acquiring the hierarchical generalization&#10;increased when the language included an additional cue to hierarchy in the form&#10;of subject-verb agreement, underscoring the role of cues to hierarchy in the&#10;learner's input.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.12036" label="2109.12036">
        <attvalues>
          <attvalue for="0" value="Transformers Generalize Linearly" />
          <attvalue for="1" value="  Natural language exhibits patterns of hierarchically governed dependencies,&#10;in which relations between words are sensitive to syntactic structure rather&#10;than linear ordering. While re-current network models often fail to generalize&#10;in a hierarchically sensitive way (McCoy et al.,2020) when trained on ambiguous&#10;data, the improvement in performance of newer Trans-former language models&#10;(Vaswani et al., 2017)on a range of syntactic benchmarks trained on large data&#10;sets (Goldberg, 2019; Warstadtet al., 2019) opens the question of whether these&#10;models might exhibit hierarchical generalization in the face of impoverished&#10;data.In this paper we examine patterns of structural generalization for&#10;Transformer sequence-to-sequence models and find that not only do Transformers&#10;fail to generalize hierarchically across a wide variety of grammatical mapping&#10;tasks, but they exhibit an even stronger preference for linear generalization&#10;than comparable recurrent networks&#10;" />
          <attvalue for="2" value="&#10;&#10;One of the fundamental properties of human languages is their sensitivity to relations among elements that are not easily characterized in linear terms. In phenomena like subject-verb agreement or reflexive anaphora, the relationship between the agreeing verb and its agreement target or the reflexive pronoun and its antecedent is not governed by linear properties like adjacency or recency, but instead by the hierarchical organization of the sentence.&#10;Similarly, the relationship between related sentences, which are represented in some grammatical theories as transformational operations or as lexical rules in others, is also governed by hierarchical organization. English polar questions, for instance, involve the fronting of an auxiliary verb in the corresponding declarative to a sentence-initial position. Questions with complex subjects like~(\nextx a) demonstrate that the verb that is fronted in such cases is the determined by hierarchical prominence (i.e., move-main yielding (\nextx b)) and not linear considerations (move-first yielding (\nextx c) or move-last yielding (\nextx d)).&#10;\pex&lt;ex:question&gt;&#10; \a {[The president who can smile] will lead [those who would sing].}&#10; \a Will the president who can smile \underline lead those who would sing?&#10; \a * Can the president who \underline smile will lead those who would sing?&#10; \a * Would the president who can smile will lead those who \underline sing?&#10;&#10;\xe&#10;\cite{chomsky1971problems} argues that, in spite of receiving little input of the form in~(\lastx b), which would unambiguously demonstrate the necessity for a hierarchically governed dependency, children uniformly generalize the process of question formation in a hierarchical fashion. Such consistent behavior suggests that humans possess an inherent bias of some sort towards hierarchical generalization (though see&#10;\cite{ambridge2008structure} and \cite{perfors2011learnability} for&#10;arguments against this view).&#10;Replicating such a bias in&#10;generalization would indicate the ability to mimic patterns&#10;of human cognition and learning. &#10;&#10;Previous investigations of recurrent neural architectures have yielded some evidence for hierarchically-governed linguistic knowledge \cite{gulordava2018colorless, Marvin-2018-targeted,hu-etal-2020-systematic}. Even greater success has been achieved with neural networks the incorporate explicit representation of syntactic structure \cite{kuncoro-etal-2018-lstms}. Architecturally-constrained models when trained without explicit information about syntactic structure show only modest benefits \cite{shen2018ordered,kim-etal-2019-unsupervised,merrill-etal-2019-finding}. However, all of these studies involve models that are trained on large quantities of text which may not be impoverished in domains that these benchmarks assess. As a result, it is unclear whether any apparent hierarchical behavior reported in these works is the effect of a bias for hierarchical generalization or the accumulation of patterns explicitly guided by the training data. \cite{mccoytrees} take a different tack: the training data is carefully controlled so that hierarchical behavior can emerge only if a model itself is biased to extract hierarchical generalizations. Their experiments demonstrate that recurrent neural network seq2seq models show a clear preference for linear generalization.&#10;&#10;The recently developed Transformer architecture has led to revolutionary advances across many areas of natural language processing, including&#10;machine translation and question answering~\cite{vaswani2017attention, devlin2019bert}. Transformer-based models have also shown considerable success on benchmarks that appear to require the representation of hierarchical abstractions \cite{rogers2021primer,goldberg2019assessing, warstadt2019investigating}. Further, investigations of Transformers' representations of sentences \cite{hewitt-manning-2019-structural, lin-etal-2019-open} point to encodings of hierarchical syntactic structure. Yet, for the reasons noted above, it is difficult to conclude much about the inductive bias in the Transformer: they are trained on vast datasets, leaving open the question of the impact of inductive bias as opposed to training data (\cite{warstadt2020can}, but see \cite{van2019quantity} for arguments that even massive data may not be sufficient). This paper contributes to our understanding by examining the degree to which the Transformer architecture is biased toward hierarchical generalization when the data underdetermine such generalization. Specifically, we study whether Transformers learning sequence-to-sequence mappings generalize in a structure sensitive way, and compare their performance with recurrent models.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.04408" label="2307.04408">
        <attvalues>
          <attvalue for="0" value="TIM: Teaching Large Language Models to Translate with Comparison" />
          <attvalue for="1" value="  Open-sourced large language models (LLMs) have demonstrated remarkable&#10;efficacy in various tasks with instruction tuning. However, these models can&#10;sometimes struggle with tasks that require more specialized knowledge such as&#10;translation. One possible reason for such deficiency is that instruction tuning&#10;aims to generate fluent and coherent text that continues from a given&#10;instruction without being constrained by any task-specific requirements.&#10;Moreover, it can be more challenging for tuning smaller LLMs with lower-quality&#10;training data. To address this issue, we propose a novel framework using&#10;examples in comparison to teach LLMs to learn translation. Our approach&#10;involves presenting the model with examples of correct and incorrect&#10;translations and using a preference loss to guide the model's learning. We&#10;evaluate our method on WMT2022 test sets and show that it outperforms existing&#10;methods. Our findings offer a new perspective on fine-tuning LLMs for&#10;translation tasks and provide a promising solution for generating high-quality&#10;translations. Please refer to Github for more details:&#10;https://github.com/lemon0830/TIM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Research on machine translation based on Large Language Models (LLMs) can be divided into two categories: LLMs as interface and instruction tuning. &#10;&#10;The studies of using LLMs as an interface focus on empirical analysis.&#10;For example, \cite{Hendy} evaluate ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002 in eighteen different translation directions involving high&#10;and low resource languages.&#10;\cite{Zhu} further evaluate four popular LLMs (XGLM, BLOOMZ, OPT and ChatGPT) on 202 directions and 102 languages, and compare them with strong supervised baselines, which provides a more comprehensive benchmark result.&#10;Many efforts are also put into investigating translation exemplars selection strategy of in-context learning \cite{emnlp/LinMAWCSOGBDPSK22,Agrawal}. Another line of work introduces knowledge, such as word alignments extracted from a dictionary, to LLMs for better translation \cite{arxiv2023:chainofdict}.&#10;&#10;Tuning smaller LLMs (e.g., 7B) for translation tasks is a promising direction since they are better at English than supervised translation models.&#10;However, even for directions from other languages to English, the gap between language models fine-tuned with translation data and supervised systems is still evident \cite{Jiao_ParroT,Arxiv2023:bayling}.&#10;Different from them, we introduce output comparison and preference comparison data and present a preference regularization to alleviate hallucination and help LLMs learn translation better.&#10;&#10;" />
          <attvalue for="4" value="Language Model Tuning, Machine Learning, Computer Science, Linguistics, Translation Tasks, Deep Learning Methods, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2309.13173" label="2309.13173">
        <attvalues>
          <attvalue for="0" value="BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls&#10;  of Large Language Models on Bengali NLP" />
          <attvalue for="1" value="  Large Language Models (LLMs) have emerged as one of the most important&#10;breakthroughs in NLP for their impressive skills in language generation and&#10;other language-specific tasks. Though LLMs have been evaluated in various&#10;tasks, mostly in English, they have not yet undergone thorough evaluation in&#10;under-resourced languages such as Bengali (Bangla). To this end, this paper&#10;introduces BenLLM-Eval, which consists of a comprehensive evaluation of LLMs to&#10;benchmark their performance in the Bengali language that has modest resources.&#10;In this regard, we select various important and diverse Bengali NLP tasks, such&#10;as text summarization, question answering, paraphrasing, natural language&#10;inference, transliteration, text classification, and sentiment analysis for&#10;zero-shot evaluation of popular LLMs, namely, GPT-3.5, LLaMA-2-13b-chat, and&#10;Claude-2. Our experimental results demonstrate that while in some Bengali NLP&#10;tasks, zero-shot LLMs could achieve performance on par, or even better than&#10;current SOTA fine-tuned models; in most tasks, their performance is quite poor&#10;(with the performance of open-source LLMs like LLaMA-2-13b-chat being&#10;significantly bad) in comparison to the current SOTA results. Therefore, it&#10;calls for further efforts to develop a better understanding of LLMs in&#10;modest-resourced languages like Bengali.&#10;" />
          <attvalue for="2" value="&#10;&#10;Since the advent of pre-trained language models \cite{devlin-etal-2019-bert, Liu2019RoBERTaAR,rogers-etal-2020-primer}, NLP has witnessed revolutionary advancements over the years. These pre-trained language models have produced SOTA results on a variety of NLP tasks with little task-specific fine-tuning. This leads to the development of various pre-trained language models specialized in the Bengali language, such as BanglaBERT \cite{bhattacharjee-etal-2022-banglabert}, BanglaT5 \cite{bhattacharjee-etal-2023-banglanlg}, and etc. These models have demonstrated exciting progress in many downstream Bengali NLP tasks \cite{ekram-etal-2022-banglarqa, akash-etal-2023-shironaam}. However, one major concern for these pre-trained models is that they require fine-tuning using domain-specific large annotated datasets, which is challenging for Bengali due to its under-representation in the NLP domain \cite{joshi-etal-2020-state, Chakraborty_Nayeem_Ahmad_2021, chowdhury-etal-2021-unsupervised} despite being the sixth most spoken language in the world with over {300} million native speakers \cite{bengali-language-article}.&#10;&#10;Recent developments in large language models (LLMs) \cite{Brown2020LanguageMA, Shoeybi2019MegatronLMTM, Rae2021ScalingLM, Zhang2022OPTOP} have transformed the landscape in NLP. These LLMs, with parameter sizes exceeding a hundred billion, leverage the in-context learning capability to achieve impressive performance in few-shot and zero-shot learning scenarios without the need for task-specific fine-tuning. This capability makes it possible to reduce the need for the annotation of huge datasets, allowing the model to perform well on tasks that it was not trained on. &#10;&#10;Despite the impressive capabilities of LLMs, they may still frequently generate untruthful facts that diverge from the original input \cite{Ouyang2022TrainingLM}. To address this issue, the Reinforcement Learning from Human Feedback (RLHF) mechanism has been utilized to help LLMs generate honest and harmless responses. ChatGPT and other recently proposed LLMs like PaLM-2, Claude-2, LLaMA-2-chat \cite{touvron2023llama2, anil2023palm2, jahan2024comprehensive} are trained via leveraging this RLHF technique to mitigate various limitations of the previous generation LLMs and gained widespread popularity. While these LLMs are trained in multiple languages, English possesses the majority of the training data. Though ChatGPT like LLMs has demonstrated strong zero-shot performance in various NLP tasks in English \cite{laskar-etal-2023-systematic,laskar-etal-2023-building} and some other languages \cite{Lai2023ChatGPTBE} and domains \cite{jahan-etal-2023-evaluation,jahan2024comprehensive,fu2024tiny}, these LLMs are yet to be investigated in the widely spoken, yet modest-resourced,&#10; Bengali language domain. &#10;&#10;In this regard, we aim to assess the efficacy of LLMs in prevalent downstream NLP tasks specific to the Bengali language, a domain that has not been thoroughly examined compared to the numerous LLM evaluations conducted for English and other Indo-European languages. Due to the lack of task-specific annotated datasets in Bengali, we conducted a zero-shot evaluation with LLMs to investigate if LLMs could be utilized to address the scarcity of large annotated datasets in Bengali. Thus, the findings from this paper would give insights into the capabilities and limitations of LLMs in Bengali, which may pave the way to implement LLMs in real-world applications like Bengali Chatbots.&#10;To this end, we present BenLLM-Eval - a comprehensive benchmark that evaluates the zero-shot performance of various LLMs on diverse NLP tasks in the Bengali language, including text summarization, question answering, paraphrasing, natural language inference, transliteration, text classification, and sentiment analysis. The evaluation incorporates carefully crafted prompts to ensure rigorous assessment of the following three popular LLMs: GPT-3.5, LLaMA-2-13b-chat, and Claude-2, and compare them with SOTA fine-tuned models. To the best of our knowledge, this is the first work that evaluates LLMs in the Bengali language. Our experimental results in this paper are summarized as follows: &#10;&#10;\begin{itemize}&#10; \item Despite some exceptional cases, the zero-shot performance of LLMs is generally inferior compared to the SOTA fine-tuned models across the majority of the tasks in our evaluation.&#10; Given the substantial performance disparities observed, it is reasonable to deduce that LLMs, in their current form, are not suitable for serving as a comprehensive solution for modest-resourced tasks in Bengali. &#10; \item Considering LLMs remarkable performance in zero-shot scenarios within the English language and its subpar performance in modest-resource languages like Bengali, this paper emphasizes the significance of investigating the limitations of LLMs tailored to diverse modest-resource language groups. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Low-Resource Languages, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.18486" label="2305.18486">
        <attvalues>
          <attvalue for="0" value="A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark&#10;  Datasets" />
          <attvalue for="1" value="  The development of large language models (LLMs) such as ChatGPT has brought a&#10;lot of attention recently. However, their evaluation in the benchmark academic&#10;datasets remains under-explored due to the difficulty of evaluating the&#10;generative outputs produced by this model against the ground truth. In this&#10;paper, we aim to present a thorough evaluation of ChatGPT's performance on&#10;diverse academic datasets, covering tasks like question-answering, text&#10;summarization, code generation, commonsense reasoning, mathematical&#10;problem-solving, machine translation, bias detection, and ethical&#10;considerations. Specifically, we evaluate ChatGPT across 140 tasks and analyze&#10;255K responses it generates in these datasets. This makes our work the largest&#10;evaluation of ChatGPT in NLP benchmarks. In short, our study aims to validate&#10;the strengths and weaknesses of ChatGPT in various tasks and provide insights&#10;for future research using LLMs. We also report a new emergent ability to follow&#10;multi-query instructions that we mostly found in ChatGPT and other&#10;instruction-tuned models. Our extensive evaluation shows that even though&#10;ChatGPT is capable of performing a wide variety of tasks, and may obtain&#10;impressive performance in several benchmark datasets, it is still far from&#10;achieving the ability to reliably solve many challenging tasks. By providing a&#10;thorough assessment of ChatGPT's performance across diverse NLP tasks, this&#10;paper sets the stage for a targeted deployment of ChatGPT-like LLMs in&#10;real-world applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.04504" label="2306.04504">
        <attvalues>
          <attvalue for="0" value="Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with&#10;  Fine-Tuned Generative Transformers" />
          <attvalue for="1" value="  ChatGPT is a large language model developed by OpenAI. Despite its impressive&#10;performance across various tasks, no prior work has investigated its capability&#10;in the biomedical domain yet. To this end, this paper aims to evaluate the&#10;performance of ChatGPT on various benchmark biomedical tasks, such as relation&#10;extraction, document classification, question answering, and summarization. To&#10;the best of our knowledge, this is the first work that conducts an extensive&#10;evaluation of ChatGPT in the biomedical domain. Interestingly, we find based on&#10;our evaluation that in biomedical datasets that have smaller training sets,&#10;zero-shot ChatGPT even outperforms the state-of-the-art fine-tuned generative&#10;transformer models, such as BioGPT and BioBART. This suggests that ChatGPT's&#10;pre-training on large text corpora makes it quite specialized even in the&#10;biomedical domain. Our findings demonstrate that ChatGPT has the potential to&#10;be a valuable tool for various tasks in the biomedical domain that lack large&#10;annotated data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.04270" label="2310.04270">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Evaluation of Large Language Models on Benchmark&#10;  Biomedical Text Processing Tasks" />
          <attvalue for="1" value="  Recently, Large Language Models (LLM) have demonstrated impressive capability&#10;to solve a wide range of tasks. However, despite their success across various&#10;tasks, no prior work has investigated their capability in the biomedical domain&#10;yet. To this end, this paper aims to evaluate the performance of LLMs on&#10;benchmark biomedical tasks. For this purpose, we conduct a comprehensive&#10;evaluation of 4 popular LLMs in 6 diverse biomedical tasks across 26 datasets.&#10;To the best of our knowledge, this is the first work that conducts an extensive&#10;evaluation and comparison of various LLMs in the biomedical domain.&#10;Interestingly, we find based on our evaluation that in biomedical datasets that&#10;have smaller training sets, zero-shot LLMs even outperform the current&#10;state-of-the-art fine-tuned biomedical models. This suggests that pretraining&#10;on large text corpora makes LLMs quite specialized even in the biomedical&#10;domain. We also find that not a single LLM can outperform other LLMs in all&#10;tasks, with the performance of different LLMs may vary depending on the task.&#10;While their performance is still quite poor in comparison to the biomedical&#10;models that were fine-tuned on large training sets, our findings demonstrate&#10;that LLMs have the potential to be a valuable tool for various biomedical tasks&#10;that lack large annotated data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05115" label="2310.05115">
        <attvalues>
          <attvalue for="0" value="Breaking Down Word Semantics from Pre-trained Language Models through&#10;  Layer-wise Dimension Selection" />
          <attvalue for="1" value="  Contextual word embeddings obtained from pre-trained language model (PLM)&#10;have proven effective for various natural language processing tasks at the word&#10;level. However, interpreting the hidden aspects within embeddings, such as&#10;syntax and semantics, remains challenging. Disentangled representation learning&#10;has emerged as a promising approach, which separates specific aspects into&#10;distinct embeddings. Furthermore, different linguistic knowledge is believed to&#10;be stored in different layers of PLM. This paper aims to disentangle semantic&#10;sense from BERT by applying a binary mask to middle outputs across the layers,&#10;without updating pre-trained parameters. The disentangled embeddings are&#10;evaluated through binary classification to determine if the target word in two&#10;different sentences has the same meaning. Experiments with cased&#10;BERT$_{\texttt{base}}$ show that leveraging layer-wise information is effective&#10;and disentangling semantic sense further improve performance.&#10;" />
          <attvalue for="2" value="&#10;Pre-trained language models (PLMs) like BERT \cite{devlin-etal-2019-bert} have been successful in generating contextual word embeddings for diverse word-level NLP tasks, such as named entity recognition, part-of-speech (PoS) tagging, coreference resolution, and word sense disambiguation (WSD) \cite{raganato-etal-2017-word}. The power of contextual word embeddings lies in their capacity to capture the overall traits of a word, enabling them to perform well across wide-ranging NLP tasks.&#10;&#10;Although contextual word embeddings are capable of capturing diverse hidden aspects such as semantic sense, syntactic role, and sentiment, interpreting these factors remains a challenge. To address this issue, disentangled representation learning (DRL) (\cite{do2019theory}; \cite{9947342}) has emerged as an approach to separate specific aspects into distinct representations. DRL has shown promise in computer vision through methods such as InfoGAN \cite{chen2016infogan}, but its potential for NLP tasks has yet to be fully explored \cite{vishnubhotla2021evaluation}. In the text domain, \cite{xu2020variational} proposed a constraint on the latent space of variational autoencoder (VAE) with PLM to control sentiment and topic. \cite{DBLP:journals/corr/abs-2105-02685} utilized mutual information (MI) to disentangle aspects from text for fair classification and textual style transfer. The groundwork for this paper was laid by \cite{Zhang2021}, who trained binary masks to find subnetworks within BERT using masking transformers \cite{Zhao2020} to disentangle sentiment from genre, toxicity from dialect, and syntax from semantic. &#10;&#10;On the other hand, several studies have shown that linguistic knowledge like syntactic and semantic aspects are encoded in different layers of PLM (\cite{jawahar-etal-2019-bert}; \cite{tenney-etal-2019-bert}; \cite{hewitt-manning-2019-structural}; \cite{bommasani-etal-2020-interpreting}). Furthermore, each self-attention head seems to focus on different aspect of language information (\cite{vig-belinkov-2019-analyzing}; \cite{clark2019does}; \cite{zhao-bethard-2020-berts}). However, it is common practice to represent a word simply as the sum of the last four layers' hidden states even for embedding-based WSD (\cite{scarlini-etal-2020-contexts}; \cite{SensEmBERT}; \cite{loureiro2021analysis}). Despite its practicality and widespread usage, it overlooks the heterogeneous information captured by different layers. &#10;&#10;This paper studies to disentangle the semantic sense of word by masking the middle outputs from BERT in a layer-wise manner. The disentangled embeddings are evaluated by binary classification task, specifically utilizing the WiC format (shown in Figure \ref{fig:WiC example}). The task has practical implications in real-world scenarios, such as re-ranking search engine results with relevance for ambiguous queries. The contributions of this work are as follows:&#10;\begin{itemize}[topsep=0pt, partopsep=0pt]&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Linguistics, Semantic Sense Extraction, Computational Linguistics, Artificial Intelligence, Disentangled Representation Learning, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1908.09961" label="1908.09961">
        <attvalues>
          <attvalue for="0" value="Theory and Evaluation Metrics for Learning Disentangled Representations" />
          <attvalue for="1" value="  We make two theoretical contributions to disentanglement learning by (a)&#10;defining precise semantics of disentangled representations, and (b)&#10;establishing robust metrics for evaluation. First, we characterize the concept&#10;&quot;disentangled representations&quot; used in supervised and unsupervised methods&#10;along three dimensions-informativeness, separability and interpretability -&#10;which can be expressed and quantified explicitly using information-theoretic&#10;constructs. This helps explain the behaviors of several well-known&#10;disentanglement learning models. We then propose robust metrics for measuring&#10;informativeness, separability and interpretability. Through a comprehensive&#10;suite of experiments, we show that our metrics correctly characterize the&#10;representations learned by different methods and are consistent with&#10;qualitative (visual) results. Thus, the metrics allow disentanglement learning&#10;methods to be compared on a fair ground. We also empirically uncovered new&#10;interesting properties of VAE-based methods and interpreted them with our&#10;formulation. These findings are promising and hopefully will encourage the&#10;design of more theoretically driven models for learning disentangled&#10;representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.09276" label="2012.09276">
        <attvalues>
          <attvalue for="0" value="Measuring Disentanglement: A Review of Metrics" />
          <attvalue for="1" value="  Learning to disentangle and represent factors of variation in data is an&#10;important problem in AI. While many advances have been made to learn these&#10;representations, it is still unclear how to quantify disentanglement. While&#10;several metrics exist, little is known on their implicit assumptions, what they&#10;truly measure, and their limits. In consequence, it is difficult to interpret&#10;results when comparing different representations. In this work, we survey&#10;supervised disentanglement metrics and thoroughly analyze them. We propose a&#10;new taxonomy in which all metrics fall into one of three families:&#10;intervention-based, predictor-based and information-based. We conduct extensive&#10;experiments in which we isolate properties of disentangled representations,&#10;allowing stratified comparison along several axes. From our experiment results&#10;and analysis, we provide insights on relations between disentangled&#10;representation properties. Finally, we share guidelines on how to measure&#10;disentanglement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.03657" label="1606.03657">
        <attvalues>
          <attvalue for="0" value="InfoGAN: Interpretable Representation Learning by Information Maximizing&#10;  Generative Adversarial Nets" />
          <attvalue for="1" value="  This paper describes InfoGAN, an information-theoretic extension to the&#10;Generative Adversarial Network that is able to learn disentangled&#10;representations in a completely unsupervised manner. InfoGAN is a generative&#10;adversarial network that also maximizes the mutual information between a small&#10;subset of the latent variables and the observation. We derive a lower bound to&#10;the mutual information objective that can be optimized efficiently, and show&#10;that our training procedure can be interpreted as a variation of the Wake-Sleep&#10;algorithm. Specifically, InfoGAN successfully disentangles writing styles from&#10;digit shapes on the MNIST dataset, pose from lighting of 3D rendered images,&#10;and background digits from the central digit on the SVHN dataset. It also&#10;discovers visual concepts that include hair styles, presence/absence of&#10;eyeglasses, and emotions on the CelebA face dataset. Experiments show that&#10;InfoGAN learns interpretable representations that are competitive with&#10;representations learned by existing fully supervised methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.04284" label="1906.04284">
        <attvalues>
          <attvalue for="0" value="Analyzing the Structure of Attention in a Transformer Language Model" />
          <attvalue for="1" value="  The Transformer is a fully attention-based alternative to recurrent networks&#10;that has achieved state-of-the-art results across a range of NLP tasks. In this&#10;paper, we analyze the structure of attention in a Transformer language model,&#10;the GPT-2 small pretrained model. We visualize attention for individual&#10;instances and analyze the interaction between attention and syntax over a large&#10;corpus. We find that attention targets different parts of speech at different&#10;layer depths within the model, and that attention aligns with dependency&#10;relations most strongly in the middle layers. We also find that the deepest&#10;layers of the model capture the most distant relationships. Finally, we extract&#10;exemplar sentences that reveal highly specific patterns targeted by particular&#10;attention heads.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.11608" label="2008.11608">
        <attvalues>
          <attvalue for="0" value="Analysis and Evaluation of Language Models for Word Sense Disambiguation" />
          <attvalue for="1" value="  Transformer-based language models have taken many fields in NLP by storm.&#10;BERT and its derivatives dominate most of the existing evaluation benchmarks,&#10;including those for Word Sense Disambiguation (WSD), thanks to their ability in&#10;capturing context-sensitive semantic nuances. However, there is still little&#10;knowledge about their capabilities and potential limitations in encoding and&#10;recovering word senses. In this article, we provide an in-depth quantitative&#10;and qualitative analysis of the celebrated BERT model with respect to lexical&#10;ambiguity. One of the main conclusions of our analysis is that BERT can&#10;accurately capture high-level sense distinctions, even when a limited number of&#10;examples is available for each word sense. Our analysis also reveals that in&#10;some cases language models come close to solving coarse-grained noun&#10;disambiguation under ideal conditions in terms of availability of training data&#10;and computing resources. However, this scenario rarely occurs in real-world&#10;settings and, hence, many practical challenges remain even in the&#10;coarse-grained setting. We also perform an in-depth comparison of the two main&#10;language model based WSD strategies, i.e., fine-tuning and feature extraction,&#10;finding that the latter approach is more robust with respect to sense bias and&#10;it can better exploit limited available training data. In fact, the simple&#10;feature extraction strategy of averaging contextualized embeddings proves&#10;robust even using only three training sentences per word sense, with minimal&#10;improvements obtained by increasing the size of this training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.08485" label="2108.08485">
        <attvalues>
          <attvalue for="0" value="Language Model Augmented Relevance Score" />
          <attvalue for="1" value="  Although automated metrics are commonly used to evaluate NLG systems, they&#10;often correlate poorly with human judgements. Newer metrics such as BERTScore&#10;have addressed many weaknesses in prior metrics such as BLEU and ROUGE, which&#10;rely on n-gram matching. These newer methods, however, are still limited in&#10;that they do not consider the generation context, so they cannot properly&#10;reward generated text that is correct but deviates from the given reference.&#10;  In this paper, we propose Language Model Augmented Relevance Score (MARS), a&#10;new context-aware metric for NLG evaluation. MARS leverages off-the-shelf&#10;language models, guided by reinforcement learning, to create augmented&#10;references that consider both the generation context and available human&#10;references, which are then used as additional references to score generated&#10;text. Compared with seven existing metrics in three common NLG tasks, MARS not&#10;only achieves higher correlation with human reference judgements, but also&#10;differentiates well-formed candidates from adversarial samples to a larger&#10;degree.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automated metrics such as BLEU~\cite{papineni2002bleu} and ROUGE~\cite{lin2004rouge} are popular methods for evaluating natural language generation (NLG) systems. Compared with human evaluation, they are cheaper and faster, and accordingly, they often serve as essential metrics for benchmarking the performance of NLG models~\cite{novikova-etal-2017-need}. Despite their widespread use, however, these automated metrics often poorly correlate with ratings given by human judges, particularly for datasets in which only a single human reference exists~\cite{gupta2019investigating,novikova-etal-2017-need}. Moreover, these automated metrics only capture similarities between generated sentences and reference candidates, crucially ignoring provided contexts that are relevant for evaluating the answer in contextual NLG tasks, such as story generation, news summarization, and question-answering~\cite{tao2018ruber,nema-khapra-2018-towards}.&#10;&#10;Table~\ref{tab:demo} shows a story generation example that exemplifies some weaknesses of several common metrics.&#10;Perplexity (PPL)~\cite{brown1992estimate} successfully detects ungrammatical sentences, but it fails to distinguish legitimate novel continuations and copy-and-pasted ones. &#10;Relying on surface-level $n$-gram matching, BLEU-1 and ROUGE-L \redit{cannot detect reordering effectively, and wrongly score the well-formed candidate lower than its retrieval-based adversarial example.}&#10;BERTScore~\cite{zhang2019bertscore} leverages contextual embeddings from BERT~\cite{devlin2019bert}, thus mitigating the above challenges, but still does not fairly evaluate candidates that correctly align with the context but happen to differ from the provided reference example. In our example, the candidate ``... her engine was smoking'' is reasonable but deviates from the human reference, and so BERTScore rates it relatively low (0.338 out of 1.0), thus correlating poorly with human rating, which was high (5.05 out of 6.00).&#10;&#10;To address the above issues, prior studies have proposed a number of promising remedies. One line of work has proposed to combine human ratings with automated metrics~\cite[][inter alia]{durmus2020feqa,chaganty-etal-2018-price}. &#10;For instance, in HUSE score, \cite{hashimoto-etal-2019-unifying} leverages the differences between perplexity and human judgements to consider both quality and diversity of generated text.&#10;Another line has proposed training separate neural models to aid automated metrics~\cite[][inter alia]{mehri-eskenazi-2020-usr,yuma-etal-2020-ubleu}. For instance, BLEURT~\cite{sellam-etal-2020-bleurt} fine-tunes BERT~\cite{devlin2019bert} on synthetic reference-candidate pairs for machine translation. These methods, however, are often limited in practical use, because the high-cost human ratings are not always available for every dataset, and the data- or system-specific training is not easily extended to other domains~\cite{zhang2019bertscore}, \redit{and can even bias the evaluation~\cite{freitag-etal-2020-bleu}.}&#10;&#10;In this paper, we present MARS (Language \underline{M}odel \underline{A}ugmented \underline{R}elevance \underline{S}core), a new NLG evaluation metric that requires neither supervision from human ratings nor additional training on specific domains. As shown in Figure~\ref{fig:intro}, instead of comparing candidates only with human written references, as many prior metrics do, MARS &#10;&#10;uses a mixture of both human and augmented references. Specifically, MARS masks tokens in the reference to create templates, and then uses the context and templates to generate augmented references by infilling the masked parts with an LM guided by reinforcement learning. &#10;The augmented references thus incorporate information from both the context and the human reference, and are enriched with lexical and syntactic diversity, facilitating fairer evaluation of candidates.&#10;Finally, we compute the score as a weighted average of the similarity between the candidate and the set of augmented references in the contextual embedding space.&#10;&#10;The advantages of MARS are three-fold. First, MARS correlates highly with human judgements. We apply MARS to three diverse NLG tasks, and demonstrate that, compared with seven popular NLG metrics, MARS better correlates with human judgements and is robust against adversarial attacks. &#10;Second, MARS is context-aware. Unlike existing metrics that only consider the given human reference, we use a constrained NLG approach to incorporate the generation context into augmented references, thus alleviating bias against diverse candidates. &#10;Third, MARS is easy to deploy and extend. Built on off-the-shelf LMs, MARS requires neither human supervision nor additional training for specific domains, and can therefore serve as a general-purpose metric for a broad range of NLG applications, as we will demonstrate for three common NLG tasks: story generation, news summarization, and question-answering.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Evaluation Metrics, Language Model Applications, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1707.06875" label="1707.06875">
        <attvalues>
          <attvalue for="0" value="Why We Need New Evaluation Metrics for NLG" />
          <attvalue for="1" value="  The majority of NLG evaluation relies on automatic metrics, such as BLEU . In&#10;this paper, we motivate the need for novel, system- and data-independent&#10;automatic evaluation methods: We investigate a wide range of metrics, including&#10;state-of-the-art word-based and novel grammar-based ones, and demonstrate that&#10;they only weakly reflect human judgements of system outputs as generated by&#10;data-driven, end-to-end NLG. We also show that metric performance is data- and&#10;system-specific. Nevertheless, our results also suggest that automatic metrics&#10;perform reliably at system-level and can support system development by finding&#10;cases where a system performs poorly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.10568" label="1907.10568">
        <attvalues>
          <attvalue for="0" value="Investigating Evaluation of Open-Domain Dialogue Systems With Human&#10;  Generated Multiple References" />
          <attvalue for="1" value="  The aim of this paper is to mitigate the shortcomings of automatic evaluation&#10;of open-domain dialog systems through multi-reference evaluation. Existing&#10;metrics have been shown to correlate poorly with human judgement, particularly&#10;in open-domain dialog. One alternative is to collect human annotations for&#10;evaluation, which can be expensive and time consuming. To demonstrate the&#10;effectiveness of multi-reference evaluation, we augment the test set of&#10;DailyDialog with multiple references. A series of experiments show that the use&#10;of multiple references results in improved correlation between several&#10;automatic metrics and human judgement for both the quality and the diversity of&#10;system output.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1701.03079" label="1701.03079">
        <attvalues>
          <attvalue for="0" value="RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain&#10;  Dialog Systems" />
          <attvalue for="1" value="  Open-domain human-computer conversation has been attracting increasing&#10;attention over the past few years. However, there does not exist a standard&#10;automatic evaluation metric for open-domain dialog systems; researchers usually&#10;resort to human annotation for model evaluation, which is time- and&#10;labor-intensive. In this paper, we propose RUBER, a Referenced metric and&#10;Unreferenced metric Blended Evaluation Routine, which evaluates a reply by&#10;taking into consideration both a groundtruth reply and a query (previous&#10;user-issued utterance). Our metric is learnable, but its training does not&#10;require labels of human satisfaction. Hence, RUBER is flexible and extensible&#10;to different datasets and languages. Experiments on both retrieval and&#10;generative dialog systems show that RUBER has a high correlation with human&#10;annotation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.10192" label="1808.10192">
        <attvalues>
          <attvalue for="0" value="Towards a Better Metric for Evaluating Question Generation Systems" />
          <attvalue for="1" value="  There has always been criticism for using $n$-gram based similarity metrics,&#10;such as BLEU, NIST, etc, for evaluating the performance of NLG systems.&#10;However, these metrics continue to remain popular and are recently being used&#10;for evaluating the performance of systems which automatically generate&#10;questions from documents, knowledge graphs, images, etc. Given the rising&#10;interest in such automatic question generation (AQG) systems, it is important&#10;to objectively examine whether these metrics are suitable for this task. In&#10;particular, it is important to verify whether such metrics used for evaluating&#10;AQG systems focus on answerability of the generated question by preferring&#10;questions which contain all relevant information such as question type&#10;(Wh-types), entities, relations, etc. In this work, we show that current&#10;automatic evaluation metrics based on $n$-gram similarity do not always&#10;correlate well with human judgments about answerability of a question. To&#10;alleviate this problem and as a first step towards better evaluation metrics&#10;for AQG, we introduce a scoring function to capture answerability and show that&#10;when this scoring function is integrated with existing metrics, they correlate&#10;significantly better with human judgments. The scripts and data developed as a&#10;part of this work are made publicly available at&#10;https://github.com/PrekshaNema25/Answerability-Metric&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.02792" label="1904.02792">
        <attvalues>
          <attvalue for="0" value="Unifying Human and Statistical Evaluation for Natural Language&#10;  Generation" />
          <attvalue for="1" value="  How can we measure whether a natural language generation system produces both&#10;high quality and diverse outputs? Human evaluation captures quality but not&#10;diversity, as it does not catch models that simply plagiarize from the training&#10;set. On the other hand, statistical evaluation (i.e., perplexity) captures&#10;diversity but not quality, as models that occasionally emit low quality samples&#10;would be insufficiently penalized. In this paper, we propose a unified&#10;framework which evaluates both diversity and quality, based on the optimal&#10;error rate of predicting whether a sentence is human- or machine-generated. We&#10;demonstrate that this error rate can be efficiently estimated by combining&#10;human and statistical evaluation, using an evaluation metric which we call&#10;HUSE. On summarization and chit-chat dialogue, we show that (i) HUSE detects&#10;diversity defects which fool pure human evaluation and that (ii) techniques&#10;such as annealing for improving quality actually decrease HUSE due to decreased&#10;diversity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00456" label="2005.00456">
        <attvalues>
          <attvalue for="0" value="USR: An Unsupervised and Reference Free Evaluation Metric for Dialog&#10;  Generation" />
          <attvalue for="1" value="  The lack of meaningful automatic evaluation metrics for dialog has impeded&#10;open-domain dialog research. Standard language generation metrics have been&#10;shown to be ineffective for evaluating dialog models. To this end, this paper&#10;presents USR, an UnSupervised and Reference-free evaluation metric for dialog.&#10;USR is a reference-free metric that trains unsupervised models to measure&#10;several desirable qualities of dialog. USR is shown to strongly correlate with&#10;human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and&#10;PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces&#10;interpretable measures for several desirable properties of dialog.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Standard automatic metrics for language generation correlate poorly with human judgement of dialog \cite{liu2016not,lowe-etal-2017-towards,gupta2019investigating}. For example, the F-1 score can be gamed by outputting the most frequent words, regardless of the context \cite{dinan2019second}. &#10;&#10;The poor performance of present metrics is largely due to the one-to-many nature of dialog \cite{zhao2017learning}. To avoid comparing to a single reference response, several authors have proposed using multiple reference responses. Multiple reference responses can be obtained with retrieval models \cite{galley-etal-2015-deltableu,Sordoni2015ANN} or through data collection \cite{gupta2019investigating}. These multi-reference metrics show improvement in performance, but it is infeasible to thoroughly cover the space of potential responses. As such, this paper addresses the one-to-many issue of dialog by presenting a reference-free metric.&#10;&#10;\cite{lowe-etal-2017-towards} train ADEM to produce a quality score conditioned on the dialog context, the reference response and the generated response. \cite{venkatesh2018evaluating} present a framework for evaluation of Alexa prize conversations, which attains moderate correlation with user ratings. Both of these methods are trained on explicit quality annotations. In contrast, USR requires no explicit supervision and will more easily generalize to new datasets and tasks.&#10;&#10;\cite{li2017adversarial} proposes a reference-free dialog evaluator which is trained to discriminate between human and generated responses. This work is similar to USR in that it evaluates the quality of a response without a reference or quality annotation training data. Using the evaluation model as a reward during reinforcement learning exhibited strong performance. However, correlation with human judgement was not evaluated. Intuitively, it appears insufficient to rely on a discriminator as a meaningful evaluation of dialog since this assumes that all human responses are perfect and all generated responses are imperfect.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2004.06063" label="2004.06063">
        <attvalues>
          <attvalue for="0" value="BLEU might be Guilty but References are not Innocent" />
          <attvalue for="1" value="  The quality of automatic metrics for machine translation has been&#10;increasingly called into question, especially for high-quality systems. This&#10;paper demonstrates that, while choice of metric is important, the nature of the&#10;references is also critical. We study different methods to collect references&#10;and compare their value in automated evaluation by reporting correlation with&#10;human evaluation for a variety of systems and metrics. Motivated by the finding&#10;that typical references exhibit poor diversity, concentrating around&#10;translationese language, we develop a paraphrasing task for linguists to&#10;perform on existing reference translations, which counteracts this bias. Our&#10;method yields higher correlation with human judgment not only for the&#10;submissions of WMT 2019 English to German, but also for Back-translation and&#10;APE augmented MT output, which have been shown to have low correlation with&#10;automatic metrics using standard references. We demonstrate that our&#10;methodology improves correlation with all modern evaluation metrics we look at,&#10;including embedding-based methods. To complete this picture, we reveal that&#10;multi-reference BLEU does not improve the correlation for high quality output,&#10;and present an alternative multi-reference formulation that is more effective.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.02709" label="2401.02709">
        <attvalues>
          <attvalue for="0" value="German Text Embedding Clustering Benchmark" />
          <attvalue for="1" value="  This work introduces a benchmark assessing the performance of clustering&#10;German text embeddings in different domains. This benchmark is driven by the&#10;increasing use of clustering neural text embeddings in tasks that require the&#10;grouping of texts (such as topic modeling) and the need for German resources in&#10;existing benchmarks. We provide an initial analysis for a range of pre-trained&#10;mono- and multilingual models evaluated on the outcome of different clustering&#10;algorithms. Results include strong performing mono- and multilingual models.&#10;Reducing the dimensions of embeddings can further improve clustering.&#10;Additionally, we conduct experiments with continued pre-training for German&#10;BERT models to estimate the benefits of this additional training. Our&#10;experiments suggest that significant performance improvements are possible for&#10;short text. All code and datasets are publicly available.&#10;" />
          <attvalue for="2" value="&#10;Clustering is increasingly used in tasks requiring to group semantically similar text pieces. This includes, for instance, data selection \cite{aharoni-goldberg-2020-unsupervised}, data exploration \cite{voigt-etal-2022-keywordscape}, and neural topic modeling \cite{zhao2021_topic}. One approach for this kind of topic modeling is BERTopic \cite{grootendorst2022bertopic}, which, in principle, uses generic clustering algorithms for text embeddings to find latent topics in text corpora. This is in stark contrast to more traditional topic modeling techniques using Latent Dirichlet Allocation \cite{blei2003lda} or Non-Negative Matrix Factorization \cite{fevotte2011} and representing text as simple bag-of-words. The shift to embedding-based approaches is driven by the continuous development of neural language models, successfully used in natural language understanding (NLU) tasks such as semantic textual similarity \cite{reimers-gurevych-2019-sentence,gao-etal-2021-simcse} or retrieval and reranking \cite{huang2020_retrieval,yates-etal-2021-pretrained}. The availability of plug-and-play frameworks for the computation of vector representation only fosters this trend. One such framework is Sentence Transformers \cite{reimers-gurevych-2019-sentence}, which is used by BERTopic. It provides an extensive collection of pre-trained transformer models and techniques to fine-tune models for similarity-focused language tasks.&#10;&#10;Benchmarks help to understand the usefulness of these easily available language models, allowing to compare existing and newly developed models for language tasks of interest. The Massive Text Embedding Benchmark (MTEB, \cite{muennighoff-etal-2023-mteb}) provides such a benchmark for a wide range of embedding-based tasks (e.g., classification, clustering, or reranking) and datasets from different domains (e.g., online reviews, scientific publications, or social media). MTEB includes a wider range of tasks and focuses on more recent language models than other benchmarks (such as SentEval \cite{conneau-kiela-2018-senteval}). MTEB, offering an easy-to-use API, invites the evaluation of models and submissions to a publicly accessible leaderboard.&#10;&#10;However, MTEB only considers the English language for the evaluation of clustering. The inclusion of non-English data is important, as the performance of multilingual models may not equal their monolingual counterparts \cite{rust-etal-2021-good}, and as a means to evaluate the potentially strong cross-lingual transfer capability of multilingual models (e.g., \cite{huang-etal-2019-unicoder}). This work addresses this limitation by providing benchmark datasets and results for German. What is more, MTEB evaluates clustering performance on a single clustering algorithm. This is a suitable approach for such a broad benchmark as it simplifies the evaluation in terms of computational and content-related complexity. From a practical point of view, and specifically for clustering, the evaluation of different algorithms is helpful. Building on the MTEB API, we provide code and evaluation results for a broader range of clustering algorithms. &#10;&#10;Finally, we conduct experiments with continued pre-training. The idea of this additional training is to adapt language models, typically trained on large and heterogeneous data collections, to the data of a specific domain or task, and has been shown to improve performance on downstream tasks (e.g., \cite{howard-ruder-2018-universal,lee2019-biobert,gururangan-etal-2020-dont}). We analyze the benefit of such adaptive training for clustering within this work.&#10;All code and datasets are publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Clustering Algorithms, Computational Linguistics, Text Embeddings, Mathematics, Natural Language Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="2004.02105" label="2004.02105">
        <attvalues>
          <attvalue for="0" value="Unsupervised Domain Clusters in Pretrained Language Models" />
          <attvalue for="1" value="  The notion of &quot;in-domain data&quot; in NLP is often over-simplistic and vague, as&#10;textual data varies in many nuanced linguistic aspects such as topic, style or&#10;level of formality. In addition, domain labels are many times unavailable,&#10;making it challenging to build domain-specific systems. We show that massive&#10;pre-trained language models implicitly learn sentence representations that&#10;cluster by domains without supervision -- suggesting a simple data-driven&#10;definition of domains in textual data. We harness this property and propose&#10;domain data selection methods based on such models, which require only a small&#10;set of in-domain monolingual data. We evaluate our data selection methods for&#10;neural machine translation across five diverse domains, where they outperform&#10;an established approach as measured by both BLEU and by precision and recall of&#10;sentence selection with respect to an oracle.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.00498" label="2103.00498">
        <attvalues>
          <attvalue for="0" value="Topic Modelling Meets Deep Neural Networks: A Survey" />
          <attvalue for="1" value="  Topic modelling has been a successful technique for text analysis for almost&#10;twenty years. When topic modelling met deep neural networks, there emerged a&#10;new and increasingly popular research area, neural topic models, with over a&#10;hundred models developed and a wide range of applications in neural language&#10;understanding such as text generation, summarisation and language models. There&#10;is a need to summarise research developments and discuss open problems and&#10;future directions. In this paper, we provide a focused yet comprehensive&#10;overview of neural topic models for interested researchers in the AI community,&#10;so as to facilitate them to navigate and innovate in this fast-growing research&#10;area. To the best of our knowledge, ours is the first review focusing on this&#10;specific topic.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.11632" label="2006.11632">
        <attvalues>
          <attvalue for="0" value="Embedding-based Retrieval in Facebook Search" />
          <attvalue for="1" value="  Search in social networks such as Facebook poses different challenges than in&#10;classical web search: besides the query text, it is important to take into&#10;account the searcher's context to provide relevant results. Their social graph&#10;is an integral part of this context and is a unique aspect of Facebook search.&#10;While embedding-based retrieval (EBR) has been applied in eb search engines for&#10;years, Facebook search was still mainly based on a Boolean matching model. In&#10;this paper, we discuss the techniques for applying EBR to a Facebook Search&#10;system. We introduce the unified embedding framework developed to model&#10;semantic embeddings for personalized search, and the system to serve&#10;embedding-based retrieval in a typical search system based on an inverted&#10;index. We discuss various tricks and experiences on end-to-end optimization of&#10;the whole system, including ANN parameter tuning and full-stack optimization.&#10;Finally, we present our progress on two selected advanced topics about&#10;modeling. We evaluated EBR on verticals for Facebook Search with significant&#10;metrics gains observed in online A/B experiments. We believe this paper will&#10;provide useful insights and experiences to help people on developing&#10;embedding-based retrieval systems in search engines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.06467" label="2010.06467">
        <attvalues>
          <attvalue for="0" value="Pretrained Transformers for Text Ranking: BERT and Beyond" />
          <attvalue for="1" value="  The goal of text ranking is to generate an ordered list of texts retrieved&#10;from a corpus in response to a query. Although the most common formulation of&#10;text ranking is search, instances of the task can also be found in many natural&#10;language processing applications. This survey provides an overview of text&#10;ranking with neural network architectures known as transformers, of which BERT&#10;is the best-known example. The combination of transformers and self-supervised&#10;pretraining has been responsible for a paradigm shift in natural language&#10;processing (NLP), information retrieval (IR), and beyond. In this survey, we&#10;provide a synthesis of existing work as a single point of entry for&#10;practitioners who wish to gain a better understanding of how to apply&#10;transformers to text ranking problems and researchers who wish to pursue work&#10;in this area. We cover a wide range of modern techniques, grouped into two&#10;high-level categories: transformer models that perform reranking in multi-stage&#10;architectures and dense retrieval techniques that perform ranking directly.&#10;There are two themes that pervade our survey: techniques for handling long&#10;documents, beyond typical sentence-by-sentence processing in NLP, and&#10;techniques for addressing the tradeoff between effectiveness (i.e., result&#10;quality) and efficiency (e.g., query latency, model and index size). Although&#10;transformer architectures and pretraining techniques are recent innovations,&#10;many aspects of how they are applied to text ranking are relatively well&#10;understood and represent mature techniques. However, there remain many open&#10;research questions, and thus in addition to laying out the foundations of&#10;pretrained transformers for text ranking, this survey also attempts to&#10;prognosticate where the field is heading.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.07316" label="2210.07316">
        <attvalues>
          <attvalue for="0" value="MTEB: Massive Text Embedding Benchmark" />
          <attvalue for="1" value="  Text embeddings are commonly evaluated on a small set of datasets from a&#10;single task not covering their possible applications to other tasks. It is&#10;unclear whether state-of-the-art embeddings on semantic textual similarity&#10;(STS) can be equally well applied to other tasks like clustering or reranking.&#10;This makes progress in the field difficult to track, as various models are&#10;constantly being proposed without proper evaluation. To solve this problem, we&#10;introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding&#10;tasks covering a total of 58 datasets and 112 languages. Through the&#10;benchmarking of 33 models on MTEB, we establish the most comprehensive&#10;benchmark of text embeddings to date. We find that no particular text embedding&#10;method dominates across all tasks. This suggests that the field has yet to&#10;converge on a universal text embedding method and scale it up sufficiently to&#10;provide state-of-the-art results on all embedding tasks. MTEB comes with&#10;open-source code and a public leaderboard at&#10;https://github.com/embeddings-benchmark/mteb.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2012.15613" label="2012.15613">
        <attvalues>
          <attvalue for="0" value="How Good is Your Tokenizer? On the Monolingual Performance of&#10;  Multilingual Language Models" />
          <attvalue for="1" value="  In this work, we provide a systematic and comprehensive empirical comparison&#10;of pretrained multilingual language models versus their monolingual&#10;counterparts with regard to their monolingual task performance. We study a set&#10;of nine typologically diverse languages with readily available pretrained&#10;monolingual models on a set of five diverse monolingual downstream tasks. We&#10;first aim to establish, via fair and controlled comparisons, if a gap between&#10;the multilingual and the corresponding monolingual representation of that&#10;language exists, and subsequently investigate the reason for any performance&#10;difference. To disentangle conflating factors, we train new monolingual models&#10;on the same data, with monolingually and multilingually trained tokenizers. We&#10;find that while the pretraining data size is an important factor, a designated&#10;monolingual tokenizer plays an equally important role in the downstream&#10;performance. Our results show that languages that are adequately represented in&#10;the multilingual model's vocabulary exhibit negligible performance decreases&#10;over their monolingual counterparts. We further find that replacing the&#10;original multilingual tokenizer with the specialized monolingual tokenizer&#10;improves the downstream performance of the multilingual model for almost every&#10;task and language.&#10;" />
          <attvalue for="2" value="&#10;&#10;Following large transformer-based language models \cite[LMs,][]{Vaswani:2017} pretrained on large English corpora \cite[e.g., BERT, RoBERTa, T5;][]{devlin:2019,liu:2019,Raffel:2020t5}, similar monolingual language models have been introduced for other languages \cite[inter alia]{virtanen:2019, antoun:2020, martin:2020}, offering previously unmatched performance in all NLP tasks. Concurrently, massively multilingual models with the same architectures and training procedures, covering more than 100 languages, have been proposed \cite[e.g., mBERT, XLM-R, mT5;][]{devlin:2019, conneau:2020, xue2020mt5}. &#10;&#10;The ``industry'' of pretraining and releasing new monolingual BERT models continues its operations despite the fact that the corresponding languages are already covered by multilingual models.&#10;The common argument justifying the need for monolingual variants is the assumption that multilingual models---due to suffering from the so-called curse of multilinguality \cite[i.e., the lack of capacity to represent all languages in an equitable way]{conneau:2020}---underperform monolingual models when applied to monolingual tasks \cite[inter alia]{virtanen:2019,antoun:2020,ronnqvist:2019}. However, little to no compelling empirical evidence with rigorous experiments and fair comparisons have been presented so far to support or invalidate this strong claim. In this regard, much of the work proposing and releasing new monolingual models is grounded in anecdotal evidence, pointing to the positive results reported for other monolingual BERT models \cite{vries:2019, virtanen:2019, antoun:2020}.&#10;&#10;Monolingual BERT models are typically evaluated on downstream NLP tasks to demonstrate their effectiveness in comparison to previous monolingual models or mBERT \cite[inter alia]{virtanen:2019, antoun:2020, martin:2020}. While these results do show that certain monolingual models can outperform mBERT in certain tasks, we hypothesize that this may substantially vary across different languages and language properties, tasks, pretrained models and their pretraining data, domain, and size. We further argue that conclusive evidence, either supporting or refuting the key hypothesis that monolingual models currently outperform multilingual models, necessitates an independent and controlled empirical comparison on a diverse set of languages and tasks.&#10;&#10;While recent work has argued and validated that mBERT is under-trained \cite{ronnqvist:2019, wu-dredze-2020-languages}, providing evidence of improved performance when training monolingual models on more data, it is unclear if this is the only factor relevant for the performance of monolingual models. &#10;Another so far under-studied factor is the limited vocabulary size of multilingual models compared to the sum of tokens of all corresponding monolingual models. &#10;Our analyses investigating dedicated (i.e., language-specific) tokenizers reveal the importance of high-quality tokenizers for the performance of both model variants. We also shed light on the interplay of tokenization with other factors such as pretraining data size.&#10;&#10;Contributions. 1) We systematically compare monolingual with multilingual pretrained language models for 9 typologically diverse languages on 5 structurally different tasks. 2) We train new monolingual models on equally sized datasets with different tokenizers (i.e., shared multilingual versus dedicated language-specific tokenizers) to disentangle the impact of pretraining data size from the vocabulary of the tokenizer. 3) We isolate factors that contribute to a performance difference (e.g., tokenizers' ``fertility'', the number of unseen (sub)words, data size) and provide an in-depth analysis of the impact of these factors on task performance. 4) Our results suggest that monolingually adapted tokenizers can robustly improve monolingual performance of multilingual models.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.09659" label="1908.09659">
        <attvalues>
          <attvalue for="0" value="Low-Resource Name Tagging Learned with Weakly Labeled Data" />
          <attvalue for="1" value="  Name tagging in low-resource languages or domains suffers from inadequate&#10;training data. Existing work heavily relies on additional information, while&#10;leaving those noisy annotations unexplored that extensively exist on the web.&#10;In this paper, we propose a novel neural model for name tagging solely based on&#10;weakly labeled (WL) data, so that it can be applied in any low-resource&#10;settings. To take the best advantage of all WL sentences, we split them into&#10;high-quality and noisy portions for two modules, respectively: (1) a&#10;classification module focusing on the large portion of noisy data can&#10;efficiently and robustly pretrain the tag classifier by capturing textual&#10;context semantics; and (2) a costly sequence labeling module focusing on&#10;high-quality data utilizes Partial-CRFs with non-entity sampling to achieve&#10;global optimum. Two modules are combined via shared parameters. Extensive&#10;experiments involving five low-resource languages and fine-grained food domain&#10;demonstrate our superior performance (6% and 7.8% F1 gains on average) as well&#10;as efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Name tagging is an fundamental task of extracting entity information, which shall benefit many applications, such as information extraction~\cite{zhang2017xlink,kuang2019improving,cao2019multi} and recommendation~\cite{wang2019explainable,cao2019unifying}. It can be treated as either a multi-class classification problem~\cite{hammerton2003named,xu2017local} or a sequence labeling problem~\cite{collobert2011natural}, but very little work combined them together. The difference between them mainly lies in whether the method models sequential label constraints, which have been demonstrated effective in many NN-CRFs models~\cite{lample2016neural,ma2016end,chiu2016named}. However, they require a large amount of human annotated corpora, which are usually expensive to obtain.&#10;&#10;The above issue motivates a lot of work on name tagging in low-resource languages or domains. A typical line of effort focuses on introducing external knowledge via transfer learning~\cite{fritzler2018few,hofer2018few}, such as the use of cross-domain~\cite{yang2016transfer}, cross-task~\cite{peng2016improving,lin2018multi} and cross-lingual resources~\cite{ni2017weakly,xie2018neural,zafarian2015semi,zhang2016name,mayhew2017cheap,tsai2016cross,feng2018improving,pan2017cross}. Although they achieve promising results, there are a large amount of weak annotations on the Web, which have not been well studied~\cite{nothman2008transforming,ehrmann2011building}. \cite{yang2018distantly,shang2018learning} utilized Partial-CRFs~\cite{tackstrom2013token} to model incomplete annotations for specific domains, but they still rely on seed annotations or a domain dictionary. Therefore, we aim at filling the gap in low-resource name tagging research by using only WL data, and adapt it to arbitrary low-resource languages or domains, which can be further improved by the above transfer-based methods.&#10;&#10;" />
          <attvalue for="4" value="Neural Name Tagging, Machine Learning, Computer Science, Linguistics, Low Resource Languages, Artificial Intelligence, Natural Language Processing, Weakly Labeled Data" />
        </attvalues>
      </node>
      <node id="2101.05716" label="2101.05716">
        <attvalues>
          <attvalue for="0" value="SICKNL: A Dataset for Dutch Natural Language Inference" />
          <attvalue for="1" value="  We present SICK-NL (read: signal), a dataset targeting Natural Language&#10;Inference in Dutch. SICK-NL is obtained by translating the SICK dataset of&#10;Marelli et al. (2014)from English into Dutch. Having a parallel inference&#10;dataset allows us to compare both monolingual and multilingual NLP models for&#10;English and Dutch on the two tasks. In the paper, we motivate and detail the&#10;translation process, perform a baseline evaluation on both the original SICK&#10;dataset and its Dutch incarnation SICK-NL, taking inspiration from Dutch&#10;skipgram embeddings and contextualised embedding models. In addition, we&#10;encapsulate two phenomena encountered in the translation to formulate stress&#10;tests and verify how well the Dutch models capture syntactic restructurings&#10;that do not affect semantics. Our main finding is all models perform worse on&#10;SICK-NL than on SICK, indicating that the Dutch dataset is more challenging&#10;than the English original. Results on the stress tests show that models don't&#10;fully capture word order freedom in Dutch, warranting future systematic&#10;studies.&#10;" />
          <attvalue for="2" value="&#10;&#10;One of the primary tasks for Natural Language Processing (NLP) systems is Natural Language Inference (NLI), where the goal is to determine, for a given premise sentence whether it contradicts, entails, or is neutral with respect to a given hypothesis sentence. &#10;&#10;For English, several standard NLI datasets exist, such as SICK \cite{marelli-etal-2014-sick}, SNLI \cite{bowman-etal-2015-large} and MNLI \cite{williams-etal-2018-broad}. Having such inference datasets available only for English may introduces a bias in NLP research. \cite{conneau-etal-2018-xnli} introduce XNLI, a multilingual version of a fragment of the SNLI dataset, that contains pairs for Natural Language Inference in 15 languages and is explicitly intended to serve as a resource for evaluating crosslingual representations. However, Dutch is not represented in any current NLI dataset, a lack that we wish to complement.&#10;&#10;Dutch counts as a high-resource language, with the sixth largest Wikipedia (2M+ articles), despite having ca. 25M native speakers. Moreover, the syntactically parsed LASSY corpus of written Dutch \cite{vanNoord2013}, and the SONAR corpus of written Dutch \cite{Oostdijk2013} provide rich resources on which NLP systems may be developed. Indeed, Dutch is in the scope of the multilingual BERT models published by Google \cite{devlin-etal-2019-bert}, and two monolingual Dutch BERT models have been published as part of HuggingFace's transformers library \cite{de2019bertje,delobelle2020robbert}.&#10;&#10;Compared to English, however, the number of evaluation tasks for Dutch is limited. There is a Named Entity Recognition task coming from the CoNLL-2003 shared task \cite{tjong2003introduction}; from a one million word hand annotated subcorpus of SONAR \cite{Oostdijk2013} one derives part-of-speech tagging, Named Entity Recognition and Semantic Role Labelling tasks. More recently a Sentiment Analysis dataset was introduced, based on Dutch Book reviews \cite{van2019merits}. Moreover, \cite{allein2020binary} introduce a classification task where a model needs to distinguish between the pronouns die and dat.&#10;&#10;Given the focus on word/token-level tasks in Dutch, we aim to complement existing resources with an NLI task for Dutch. We do so by deriving it from the English SICK dataset, for the following reasons: first, this dataset requires a small amount of world knowledge as it was derived mainly from image captions that are typically concrete descriptions of a scene. Therefore, no world knowledge requirements will be imposed on an NLP model for the task, but rather its ability for reasoning will be assessed. Secondly, due to the structure of the sentences in SICK, the types of inferences can be attributed to particular constructs, such as hypernymy/hyponymy, negation, or choice of quantification. Thirdly, SICK contains 6076 unique sentences and almost 10K inference pairs, making it a sizeable dataset for NLP standards, while deriving a Dutch version is more manageable than with other datasets. We make the dataset, code and derived resources (see Section \ref{section:stresstests}), available online.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Dataset Translation Evaluation, Artificial Intelligence, Natural Language Inference, Natural Language Processing, Multilingual NLP Models" />
        </attvalues>
      </node>
      <node id="2001.02943" label="2001.02943">
        <attvalues>
          <attvalue for="0" value="Binary and Multitask Classification Model for Dutch Anaphora Resolution:&#10;  Die/Dat Prediction" />
          <attvalue for="1" value="  The correct use of Dutch pronouns 'die' and 'dat' is a stumbling block for&#10;both native and non-native speakers of Dutch due to the multiplicity of&#10;syntactic functions and the dependency on the antecedent's gender and number.&#10;Drawing on previous research conducted on neural context-dependent dt-mistake&#10;correction models (Heyman et al. 2018), this study constructs the first neural&#10;network model for Dutch demonstrative and relative pronoun resolution that&#10;specifically focuses on the correction and part-of-speech prediction of these&#10;two pronouns. Two separate datasets are built with sentences obtained from,&#10;respectively, the Dutch Europarl corpus (Koehn 2015) - which contains the&#10;proceedings of the European Parliament from 1996 to the present - and the SoNaR&#10;corpus (Oostdijk et al. 2013) - which contains Dutch texts from a variety of&#10;domains such as newspapers, blogs and legal texts. Firstly, a binary&#10;classification model solely predicts the correct 'die' or 'dat'. The classifier&#10;with a bidirectional long short-term memory architecture achieves 84.56%&#10;accuracy. Secondly, a multitask classification model simultaneously predicts&#10;the correct 'die' or 'dat' and its part-of-speech tag. The model containing a&#10;combination of a sentence and context encoder with both a bidirectional long&#10;short-term memory architecture results in 88.63% accuracy for die/dat&#10;prediction and 87.73% accuracy for part-of-speech prediction. More&#10;evenly-balanced data, larger word embeddings, an extra bidirectional long&#10;short-term memory layer and integrated part-of-speech knowledge positively&#10;affects die/dat prediction performance, while a context encoder architecture&#10;raises part-of-speech prediction performance. This study shows promising&#10;results and can serve as a starting point for future research on machine&#10;learning models for Dutch anaphora resolution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.11106" label="2405.11106">
        <attvalues>
          <attvalue for="0" value="LLM-based Multi-Agent Reinforcement Learning: Current and Future&#10;  Directions" />
          <attvalue for="1" value="  In recent years, Large Language Models (LLMs) have shown great abilities in&#10;various tasks, including question answering, arithmetic problem solving, and&#10;poem writing, among others. Although research on LLM-as-an-agent has shown that&#10;LLM can be applied to Reinforcement Learning (RL) and achieve decent results,&#10;the extension of LLM-based RL to Multi-Agent System (MAS) is not trivial, as&#10;many aspects, such as coordination and communication between agents, are not&#10;considered in the RL frameworks of a single agent. To inspire more research on&#10;LLM-based MARL, in this letter, we survey the existing LLM-based single-agent&#10;and multi-agent RL frameworks and provide potential research directions for&#10;future research. In particular, we focus on the cooperative tasks of multiple&#10;agents with a common goal and communication among them. We also consider&#10;human-in/on-the-loop scenarios enabled by the language component in the&#10;framework.&#10;" />
          <attvalue for="2" value="&#10;&#10;M{ulti}-Agent Reinforcement Learning (MARL) has emerged as a popular approach to address the coordination problem in Multi-Agent Systems~(MAS). As opposed to Individual Reinforcement Learning~(IRL)-based or traditional optimization-based solutions, MARL has shown a significant improvement in scalability and robustness to uncertainty and dynamicity~\cite{sun2023hmaac, shalev2016safe, sadhu2020aerial, calvo2018heterogeneous}.&#10;This improvement is largely attributed to the communication and coordination among agents inherent in MARL, where multiple agents learn and adapt their policies simultaneously while interacting within a shared environment and communicating with others.&#10;However, how and what to communicate among the agents in the MAS remains to be explored. Representative examples include MARL frameworks that learn to generate numerical messages using neural networks, formulate neural communication protocols, and learn targeted ad hoc communications. Despite the decent performance of the MARL frameworks achieved in various applications, they still underperform human experts. As a result, it is reasonable to think why not leveraging human knowledge and human languages in MARL?&#10;&#10;As recent advances in Natural Language Processing~(NLP) demonstrate great abilities in multi-modal tasks, language-conditioned MARL becomes a promising research problem. NLP has been an active research topic for decades and many famous models have been proposed for language modeling such as Recurrent Neural Network~(RNN)~\cite{rumelhart1986learning, jordan1997serial}, Long-Short Term Memory networks~(LSTM)~\cite{hochreiter1997long}, and transformers~\cite{vaswani2017attention}.&#10;These foundational models have greatly improved the ability of machines to understand and generate human language, setting the stage for more complex applications.&#10;&#10;In recent years, the integration of NLP with single-agent RL has led to the development of language-conditioned RL frameworks~\cite{peng2023conceptual, jiang2019language, zhou2021inverse}, especially as Large Language Models~(LLMs)~\cite{openai2023chatgpt, touvron2023llama, chowdhery2023palm, team2023gemini} emerged as the rising star in the artificial intelligence community (see Fig.~\ref{fig:llm}) and has been successfully applied in various fields~\cite{wu2024new, lai2024language, han2024chainofinteraction}. Pre-trained LLMs contain general human knowledge about the world and can easily adapt to RL problems without the need for retraining.&#10;This integration not only leverages the semantic richness of language&#10;but also allows for the dynamic adjustment of agent behaviors based on linguistic input. In particular, LLM is able to generate new information that it has not seen before on the basis of a few examples. For example, in Reflexion~\cite{shinn2024reflexion}, the authors showed that the LLM agent could generate decent reflections on its decisions without any reward/feedback from the environment. Such capabilities are particularly valuable in multi-agent systems, where agents must coordinate and cooperate based on shared goals communicated through language. &#10;&#10;Due to the need for communication and coordination, the problem of MARL becomes more complex than simply multiplying the RL of a single agent by the number of agents. As opposed to conventional MARL, LLMs-based MARL can leverage linguistic cues to facilitate inter-agent communication and collaboration, further boosting system performance. For example, agents can use shared language to negotiate roles, coordinate actions, or exchange information about the environment or their internal states, thereby aligning their objectives more effectively. This language-enhanced coordination becomes critical in complex scenarios where agents must handle ambiguous or evolving tasks that require continual communication and mutual understanding. The exploration of these capabilities opens up new possibilities for designing more intelligent and flexible multi-agent systems capable of operating in unpredictable, real-world environments.&#10;&#10;Guo et al.~\cite{guo2024large} reviewed LLM-based multi-agent frameworks, but the emphasis of that paper was not on MARL. Unlike their paper, this letter focuses more on the MAS that tries to accomplish a task cooperatively. In addition to that, there are several surveys on the topic of MARL~\cite{nguyen2020deep, hernandez2019survey, gronauer2022multi} and single agent LLM-based RL~\cite{luketina2019survey, cao2024survey}, but none of them is dedicated to LLM-based MARL. Therefore, we claim that we are among the first to provide a systematic overview of the LLM-based MARL problem and provide potential future research directions.&#10;&#10;The remainder of this letter is organized as follows. We first introduce the problem of MARL and provide a brief overview of conventional, i.e., non-LLM-based, MARL, and single-agent LLM-based RL, in Sect.~\ref{sect:preliminary}. Then, we will survey the existing LLM-based MARL frameworks in Sect.~\ref{sect:survey}. After that, we will discuss the challenges and future research directions for this field in Sect.~\ref{sect:open_research}. Finally, we will conclude the letter in Sect.~\ref{sect:conc}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Multi-Agent Systems, Linguistics, Artificial Intelligence, Reinforcement Learning" />
        </attvalues>
      </node>
      <node id="2303.05069" label="2303.05069">
        <attvalues>
          <attvalue for="0" value="Conceptual Reinforcement Learning for Language-Conditioned Tasks" />
          <attvalue for="1" value="  Despite the broad application of deep reinforcement learning (RL),&#10;transferring and adapting the policy to unseen but similar environments is&#10;still a significant challenge. Recently, the language-conditioned policy is&#10;proposed to facilitate policy transfer through learning the joint&#10;representation of observation and text that catches the compact and invariant&#10;information across environments. Existing studies of language-conditioned RL&#10;methods often learn the joint representation as a simple latent layer for the&#10;given instances (episode-specific observation and text), which inevitably&#10;includes noisy or irrelevant information and cause spurious correlations that&#10;are dependent on instances, thus hurting generalization performance and&#10;training efficiency. To address this issue, we propose a conceptual&#10;reinforcement learning (CRL) framework to learn the concept-like joint&#10;representation for language-conditioned policy. The key insight is that&#10;concepts are compact and invariant representations in human cognition through&#10;extracting similarities from numerous instances in real-world. In CRL, we&#10;propose a multi-level attention encoder and two mutual information constraints&#10;for learning compact and invariant concepts. Verified in two challenging&#10;environments, RTFM and Messenger, CRL significantly improves the training&#10;efficiency (up to 70%) and generalization ability (up to 30%) to the new&#10;environment dynamics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.19839" label="2403.19839">
        <attvalues>
          <attvalue for="0" value="The New Agronomists: Language Models are Experts in Crop Management" />
          <attvalue for="1" value="  Crop management plays a crucial role in determining crop yield, economic&#10;profitability, and environmental sustainability. Despite the availability of&#10;management guidelines, optimizing these practices remains a complex and&#10;multifaceted challenge. In response, previous studies have explored using&#10;reinforcement learning with crop simulators, typically employing simple&#10;neural-network-based reinforcement learning (RL) agents. Building on this&#10;foundation, this paper introduces a more advanced intelligent crop management&#10;system. This system uniquely combines RL, a language model (LM), and crop&#10;simulations facilitated by the Decision Support System for Agrotechnology&#10;Transfer (DSSAT). We utilize deep RL, specifically a deep Q-network, to train&#10;management policies that process numerous state variables from the simulator as&#10;observations. A novel aspect of our approach is the conversion of these state&#10;variables into more informative language, facilitating the language model's&#10;capacity to understand states and explore optimal management practices. The&#10;empirical results reveal that the LM exhibits superior learning capabilities.&#10;Through simulation experiments with maize crops in Florida (US) and Zaragoza&#10;(Spain), the LM not only achieves state-of-the-art performance under various&#10;evaluation metrics but also demonstrates a remarkable improvement of over 49\%&#10;in economic profit, coupled with reduced environmental impact when compared to&#10;baseline methods. Our code is available at&#10;\url{https://github.com/jingwu6/LM_AG}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.11366" label="2303.11366">
        <attvalues>
          <attvalue for="0" value="Reflexion: Language Agents with Verbal Reinforcement Learning" />
          <attvalue for="1" value="  Large language models (LLMs) have been increasingly used to interact with&#10;external environments (e.g., games, compilers, APIs) as goal-driven agents.&#10;However, it remains challenging for these language agents to quickly and&#10;efficiently learn from trial-and-error as traditional reinforcement learning&#10;methods require extensive training samples and expensive model fine-tuning. We&#10;propose Reflexion, a novel framework to reinforce language agents not by&#10;updating weights, but instead through linguistic feedback. Concretely,&#10;Reflexion agents verbally reflect on task feedback signals, then maintain their&#10;own reflective text in an episodic memory buffer to induce better&#10;decision-making in subsequent trials. Reflexion is flexible enough to&#10;incorporate various types (scalar values or free-form language) and sources&#10;(external or internally simulated) of feedback signals, and obtains significant&#10;improvements over a baseline agent across diverse tasks (sequential&#10;decision-making, coding, language reasoning). For example, Reflexion achieves a&#10;91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous&#10;state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis&#10;studies using different feedback signals, feedback incorporation methods, and&#10;agent types, and provide insights into how they affect performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.01680" label="2402.01680">
        <attvalues>
          <attvalue for="0" value="Large Language Model based Multi-Agents: A Survey of Progress and&#10;  Challenges" />
          <attvalue for="1" value="  Large Language Models (LLMs) have achieved remarkable success across a wide&#10;array of tasks. Due to the impressive planning and reasoning abilities of LLMs,&#10;they have been used as autonomous agents to do many tasks automatically.&#10;Recently, based on the development of using one LLM as a single planning or&#10;decision-making agent, LLM-based multi-agent systems have achieved considerable&#10;progress in complex problem-solving and world simulation. To provide the&#10;community with an overview of this dynamic field, we present this survey to&#10;offer an in-depth discussion on the essential aspects of multi-agent systems&#10;based on LLMs, as well as the challenges. Our goal is for readers to gain&#10;substantial insights on the following questions: What domains and environments&#10;do LLM-based multi-agents simulate? How are these agents profiled and how do&#10;they communicate? What mechanisms contribute to the growth of agents'&#10;capacities? For those interested in delving into this field of study, we also&#10;summarize the commonly used datasets or benchmarks for them to have convenient&#10;access. To keep researchers updated on the latest studies, we maintain an&#10;open-source GitHub repository, dedicated to outlining the research on LLM-based&#10;multi-agent systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.09554" label="2409.09554">
        <attvalues>
          <attvalue for="0" value="ASR Error Correction using Large Language Models" />
          <attvalue for="1" value="  Error correction (EC) models play a crucial role in refining Automatic Speech&#10;Recognition (ASR) transcriptions, enhancing the readability and quality of&#10;transcriptions. Without requiring access to the underlying code or model&#10;weights, EC can improve performance and provide domain adaptation for black-box&#10;ASR systems. This work investigates the use of large language models (LLMs) for&#10;error correction across diverse scenarios. 1-best ASR hypotheses are commonly&#10;used as the input to EC models. We propose building high-performance EC models&#10;using ASR N-best lists which should provide more contextual information for the&#10;correction process. Additionally, the generation process of a standard EC model&#10;is unrestricted in the sense that any output sequence can be generated. For&#10;some scenarios, such as unseen domains, this flexibility may impact&#10;performance. To address this, we introduce a constrained decoding approach&#10;based on the N-best list or an ASR lattice. Finally, most EC models are trained&#10;for a specific ASR system requiring retraining whenever the underlying ASR&#10;system is changed. This paper explores the ability of EC models to operate on&#10;the output of different ASR systems. This concept is further extended to&#10;zero-shot error correction using LLMs, such as ChatGPT. Experiments on three&#10;standard datasets demonstrate the efficacy of our proposed methods for both&#10;Transducer and attention-based encoder-decoder ASR systems. In addition, the&#10;proposed method can serve as an effective method for model ensembling.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic speech recognition (ASR) aims to transcribe speech audio into text and is the key component for human-computer interaction~\cite{rebman2003speech}. In recent years, the performance of ASR technology has dramatically advanced, evolving from traditional Hidden Markov Model (HMM)-based architectures to modern end-to-end (E2E) systems like Listen, Attend and Spell (LAS) or RNN-T~\cite{chan2016listen, graves2014towards, amodei2016deep,vaswani2017attention}. &#10;Large-scale models such as Whisper~\cite{radford2023robust} and Google USM~\cite{zhang2023google} have demonstrated state-of-the-art performance, leveraging vast amounts of labeled and unlabeled speech data, which can be costly to obtain. While achieving impressive results on test sets, practical deployment of ASR systems encounters challenges, especially when faced with domain-specific or previously unseen speech data.&#10;&#10;Accessing ASR services via APIs has emerged as a popular alternative to training in-house models, offering a pragmatic and economical choice. Fine-tuning these models for specific tasks, however, is often impractical due to restricted access to proprietary models. Various approaches have been proposed to enhance such restricted access ASR systems. Two common methods are language model (LM) rescoring and error correction (EC). LM rescoring involves reranking the N-best list generated by the ASR system using an external LM, which can improve the overall performance of the ASR system~\cite{mikolov2010recurrent}. Recent research, however, has shown that E2E ASR models often learn an internal language model (ILM) on the training data, which can reduce the effectiveness of traditional shallow fusion techniques~\cite{meng2021internal}. &#10;Methods to address the impact of ILMs, such as those proposed by~\cite{mcdermott2019density, liu22j_interspeech, zeineldeen2021investigating}, generally involve code modification during the inference stage, therefore they are out of the scope of discussion in this paper. &#10;&#10;adjcol&#10;Error correction, applied as a post-processing step for ASR systems, offers a promising alternative~\cite{errattahi2018automatic, hrinchuk2020correction, zhao2021bart}. This approach requires only the decoding hypotheses and reference data to train a model, eliminating the need for deep access to the ASR system.&#10;Early work in this area focused on rule-based systems which rely on statistical analysis~\cite{cucu2013statistical}. More recent developments have introduced end-to-end models with attention modules, which can automatically identify errors within sentences and learn to generate the correct counterparts implicitly~\cite{ren2019fastspeech, guo2019spelling, mani2020asr}. Large-scale pre-trained language models (PLMs) are trained on massive and diverse text datasets, far exceeding the scale of data used in ASR training.&#10;Approaches to transfer knowledge from PLMs for accurate ASR error detection and correction have been recently proposed. For example, Hrinchuk et al.~\cite{hrinchuk2020correction} propose a Transformer-based architecture to ``translate'' an ASR model output into grammatically&#10;and semantically correct text. Zhao et al.~\cite{zhao2021bart} introduce a BART-based semantic correction system for the Mandarin ASR system. Shen et al.~\cite{shen2022mask} propose a masking strategy to train the model to correct the original error tokens and predict the masked tokens based on their context information. Ma et. al~\cite{ma2023nbest, ma2023adapting} propose an N-best T5 model based on pre-trained T5 models to perform error correction using the ASR N-best list.&#10;By fine-tuning these pre-trained large language models (LLMs), the implicit knowledge acquired from vast amounts of text data can be effectively transferred to the target error correction task. &#10;&#10;The recent advent of generative LLMs has further advanced EC techniques. Within the field of NLP, studies such as~\cite{wu2023chatgpt, fang2023chatgpt} have applied ChatGPT models to grammatical error correction tasks. In the context of ASR error correction, previous research has examined zero-shot performance using LLMs~\cite{ma2023can}. Everson et al.~\cite{everson2024towards} utilized word confusion networks generated by the ASR system and performed EC with in-context learning, demonstrating improved performance with one-shot examples compared to 1-best hypotheses. Chen et al.~\cite{chen2024hyporadise} generated N-best lists in the ASR decoding and built LLM EC systems using various methods including fine-tuning, LoRA tuning, and in-context learning. &#10;Hu et al. developed a multi-modal EC model incorporating audio as an additional input~\cite{hu2024listen} and used a cloze-test task approach instead of a generative correction method. Additionally, Li et al. \cite{li2024investigating} explored knowledge transfer within LLMs by fine-tuning a multilingual LLM across various languages to correct 1-best hypothesis errors from different speech foundation models.&#10;&#10;Previous research has also explored various methods to improve ASR error correction by leveraging N-best lists, which offer richer information compared to single 1-best hypotheses. For instance, Guo et al.~\cite{guo2019spelling} generates an 8-best list with the ASR model and rescored candidates with an LSTM language model~\cite{sundermeyer2012lstm}. Zhu et al.~ \cite{zhu2021improving} concatenated N-best hypotheses for input to a bidirectional encoder, and Leng et al.\cite{leng2021fastcorrect} investigated non-autoregressive models with similar approaches. More recent work by Ma et al.\cite{ma2023can} and Chen et al.\cite{chen2024hyporadise} has integrated N-best lists with generative LLMs to enhance error correction performance. &#10;&#10;Building on these advances, our paper introduces a novel approach that uses LLMs to improve ASR error correction. We compare fine-tuning versus zero-shot error correction methods and investigate how ASR N-best lists can be effectively utilized. A major contribution of our work is the innovative use of ASR N-best lists as extended inputs, which provides richer context and more accurate cues for error correction. Additionally, we introduce advanced decoding strategies, including constrained decoding, to enhance model robustness and alignment with the original utterance. Our approach also addresses data contamination concerns by developing methods to evaluate the impact of training data biases. By integrating these elements, our study sets a new benchmark in ASR error correction, advancing the state-of-the-art in the field.&#10;&#10;This paper is structured as follows. In Section~\ref{sec:ec_pre} we present the error correction method utilizing foundation language models, including a supervised approach employing the T5 model and a zero-shot approach based on generative LLMs. Section \ref{sec:uncon_cons_decoding} introduces several decoding algorithms to build more robust ASR error correction systems, aiming to address the inherent problems of the standard beam search approach. In Section~\ref{sec:contamination}, we describe the method used to investigate data contamination.&#10;The experimental setup and results are detailed in Section~\ref{sec:exp}, while Section~\ref{sec:discussion} covers N-best analysis, an ablation study of the proposed approach, and a discussion on data contamination. Finally, Section~\ref{sec:conclusion} presents the conclusions. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Error Correction Models, Linguistics, Automatic Speech Recognition, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2011.01991" label="2011.01991">
        <attvalues>
          <attvalue for="0" value="Internal Language Model Estimation for Domain-Adaptive End-to-End Speech&#10;  Recognition" />
          <attvalue for="1" value="  The external language models (LM) integration remains a challenging task for&#10;end-to-end (E2E) automatic speech recognition (ASR) which has no clear division&#10;between acoustic and language models. In this work, we propose an internal LM&#10;estimation (ILME) method to facilitate a more effective integration of the&#10;external LM with all pre-existing E2E models with no additional model training,&#10;including the most popular recurrent neural network transducer (RNN-T) and&#10;attention-based encoder-decoder (AED) models. Trained with audio-transcript&#10;pairs, an E2E model implicitly learns an internal LM that characterizes the&#10;training data in the source domain. With ILME, the internal LM scores of an E2E&#10;model are estimated and subtracted from the log-linear interpolation between&#10;the scores of the E2E model and the external LM. The internal LM scores are&#10;approximated as the output of an E2E model when eliminating its acoustic&#10;components. ILME can alleviate the domain mismatch between training and&#10;testing, or improve the multi-domain E2E ASR. Experimented with 30K-hour&#10;trained RNN-T and AED models, ILME achieves up to 15.5% and 6.8% relative word&#10;error rate reductions from Shallow Fusion on out-of-domain LibriSpeech and&#10;in-domain Microsoft production test sets, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.11268" label="2002.11268">
        <attvalues>
          <attvalue for="0" value="A Density Ratio Approach to Language Model Fusion in End-To-End&#10;  Automatic Speech Recognition" />
          <attvalue for="1" value="  This article describes a density ratio approach to integrating external&#10;Language Models (LMs) into end-to-end models for Automatic Speech Recognition&#10;(ASR). Applied to a Recurrent Neural Network Transducer (RNN-T) ASR model&#10;trained on a given domain, a matched in-domain RNN-LM, and a target domain&#10;RNN-LM, the proposed method uses Bayes' Rule to define RNN-T posteriors for the&#10;target domain, in a manner directly analogous to the classic hybrid model for&#10;ASR based on Deep Neural Networks (DNNs) or LSTMs in the Hidden Markov Model&#10;(HMM) framework (Bourlard &amp; Morgan, 1994). The proposed approach is evaluated&#10;in cross-domain and limited-data scenarios, for which a significant amount of&#10;target domain text data is used for LM training, but only limited (or no)&#10;{audio, transcript} training data pairs are used to train the RNN-T.&#10;Specifically, an RNN-T model trained on paired audio &amp; transcript data from&#10;YouTube is evaluated for its ability to generalize to Voice Search data. The&#10;Density Ratio method was found to consistently outperform the dominant approach&#10;to LM and end-to-end ASR integration, Shallow Fusion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.10697" label="1910.10697">
        <attvalues>
          <attvalue for="0" value="Correction of Automatic Speech Recognition with Transformer&#10;  Sequence-to-sequence Model" />
          <attvalue for="1" value="  In this work, we introduce a simple yet efficient post-processing model for&#10;automatic speech recognition (ASR). Our model has Transformer-based&#10;encoder-decoder architecture which &quot;translates&quot; ASR model output into&#10;grammatically and semantically correct text. We investigate different&#10;strategies for regularizing and optimizing the model and show that extensive&#10;data augmentation and the initialization with pre-trained weights are required&#10;to achieve good performance. On the LibriSpeech benchmark, our method&#10;demonstrates significant improvement in word error rate over the baseline&#10;acoustic model with greedy decoding, especially on much noisier dev-other and&#10;test-other portions of the evaluation dataset. Our model also outperforms&#10;baseline with 6-gram language model re-scoring and approaches the performance&#10;of re-scoring with Transformer-XL neural language model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.05507" label="2104.05507">
        <attvalues>
          <attvalue for="0" value="BART based semantic correction for Mandarin automatic speech recognition&#10;  system" />
          <attvalue for="1" value="  Although automatic speech recognition (ASR) systems achieved significantly&#10;improvements in recent years, spoken language recognition error occurs which&#10;can be easily spotted by human beings. Various language modeling techniques&#10;have been developed on post recognition tasks like semantic correction. In this&#10;paper, we propose a Transformer based semantic correction method with&#10;pretrained BART initialization, Experiments on 10000 hours Mandarin speech&#10;dataset show that character error rate (CER) can be effectively reduced by&#10;21.7% relatively compared to our baseline ASR system. Expert evaluation&#10;demonstrates that actual improvement of our model surpasses what CER indicates.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.00456" label="2303.00456">
        <attvalues>
          <attvalue for="0" value="N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses&#10;  and Constrained Decoding Space" />
          <attvalue for="1" value="  Error correction models form an important part of Automatic Speech&#10;Recognition (ASR) post-processing to improve the readability and quality of&#10;transcriptions. Most prior works use the 1-best ASR hypothesis as input and&#10;therefore can only perform correction by leveraging the context within one&#10;sentence. In this work, we propose a novel N-best T5 model for this task, which&#10;is fine-tuned from a T5 model and utilizes ASR N-best lists as model input. By&#10;transferring knowledge from the pre-trained language model and obtaining richer&#10;information from the ASR decoding space, the proposed approach outperforms a&#10;strong Conformer-Transducer baseline. Another issue with standard error&#10;correction is that the generation process is not well-guided. To address this a&#10;constrained decoding process, either based on the N-best list or an ASR&#10;lattice, is used which allows additional information to be propagated.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.13648" label="2303.13648">
        <attvalues>
          <attvalue for="0" value="ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction&#10;  Benchmark" />
          <attvalue for="1" value="  ChatGPT is a cutting-edge artificial intelligence language model developed by&#10;OpenAI, which has attracted a lot of attention due to its surprisingly strong&#10;ability in answering follow-up questions. In this report, we aim to evaluate&#10;ChatGPT on the Grammatical Error Correction(GEC) task, and compare it with&#10;commercial GEC product (e.g., Grammarly) and state-of-the-art models (e.g.,&#10;GECToR). By testing on the CoNLL2014 benchmark dataset, we find that ChatGPT&#10;performs not as well as those baselines in terms of the automatic evaluation&#10;metrics (e.g., $F_{0.5}$ score), particularly on long sentences. We inspect the&#10;outputs and find that ChatGPT goes beyond one-by-one corrections. Specifically,&#10;it prefers to change the surface expression of certain phrases or sentence&#10;structure while maintaining grammatical correctness. Human evaluation&#10;quantitatively confirms this and suggests that ChatGPT produces less&#10;under-correction or mis-correction issues but more over-corrections. These&#10;results demonstrate that ChatGPT is severely under-estimated by the automatic&#10;evaluation metrics and could be a promising tool for GEC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.04172" label="2307.04172">
        <attvalues>
          <attvalue for="0" value="Can Generative Large Language Models Perform ASR Error Correction?" />
          <attvalue for="1" value="  ASR error correction is an interesting option for post processing speech&#10;recognition system outputs. These error correction models are usually trained&#10;in a supervised fashion using the decoding results of a target ASR system. This&#10;approach can be computationally intensive and the model is tuned to a specific&#10;ASR system. Recently generative large language models (LLMs) have been applied&#10;to a wide range of natural language processing tasks, as they can operate in a&#10;zero-shot or few shot fashion. In this paper we investigate using ChatGPT, a&#10;generative LLM, for ASR error correction. Based on the ASR N-best output, we&#10;propose both unconstrained and constrained, where a member of the N-best list&#10;is selected, approaches. Additionally, zero and 1-shot settings are evaluated.&#10;Experiments show that this generative LLM approach can yield performance gains&#10;for two different state-of-the-art ASR architectures, transducer and&#10;attention-encoder-decoder based, and multiple test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.15701" label="2309.15701">
        <attvalues>
          <attvalue for="0" value="HyPoradise: An Open Baseline for Generative Speech Recognition with&#10;  Large Language Models" />
          <attvalue for="1" value="  Advancements in deep neural networks have allowed automatic speech&#10;recognition (ASR) systems to attain human parity on several publicly available&#10;clean speech datasets. However, even state-of-the-art ASR systems experience&#10;performance degradation when confronted with adverse conditions, as a&#10;well-trained acoustic model is sensitive to variations in the speech domain,&#10;e.g., background noise. Intuitively, humans address this issue by relying on&#10;their linguistic knowledge: the meaning of ambiguous spoken terms is usually&#10;inferred from contextual cues thereby reducing the dependency on the auditory&#10;system. Inspired by this observation, we introduce the first open-source&#10;benchmark to utilize external large language models (LLMs) for ASR error&#10;correction, where N-best decoding hypotheses provide informative elements for&#10;true transcription prediction. This approach is a paradigm shift from the&#10;traditional language model rescoring strategy that can only select one&#10;candidate hypothesis as the output transcription. The proposed benchmark&#10;contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs&#10;of N-best hypotheses and corresponding accurate transcriptions across prevalent&#10;speech domains. Given this dataset, we examine three types of error correction&#10;techniques based on LLMs with varying amounts of labeled&#10;hypotheses-transcription pairs, which gains a significant word error rate (WER)&#10;reduction. Experimental evidence demonstrates the proposed technique achieves a&#10;breakthrough by surpassing the upper bound of traditional re-ranking based&#10;methods. More surprisingly, LLM with reasonable prompt and its generative&#10;capability can even correct those tokens that are missing in N-best list. We&#10;make our results publicly accessible for reproducible pipelines with released&#10;pre-trained models, thus providing a new evaluation paradigm for ASR error&#10;correction with LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.10025" label="2405.10025">
        <attvalues>
          <attvalue for="0" value="Listen Again and Choose the Right Answer: A New Paradigm for Automatic&#10;  Speech Recognition with Large Language Models" />
          <attvalue for="1" value="  Recent advances in large language models (LLMs) have promoted generative&#10;error correction (GER) for automatic speech recognition (ASR), which aims to&#10;predict the ground-truth transcription from the decoded N-best hypotheses.&#10;Thanks to the strong language generation ability of LLMs and rich information&#10;in the N-best list, GER shows great effectiveness in enhancing ASR results.&#10;However, it still suffers from two limitations: 1) LLMs are unaware of the&#10;source speech during GER, which may lead to results that are grammatically&#10;correct but violate the source speech content, 2) N-best hypotheses usually&#10;only vary in a few tokens, making it redundant to send all of them for GER,&#10;which could confuse LLM about which tokens to focus on and thus lead to&#10;increased miscorrection. In this paper, we propose ClozeGER, a new paradigm for&#10;ASR generative error correction. First, we introduce a multimodal LLM (i.e.,&#10;SpeechGPT) to receive source speech as extra input to improve the fidelity of&#10;correction output. Then, we reformat GER as a cloze test with logits&#10;calibration to remove the input information redundancy and simplify GER with&#10;clear instructions. Experiments show that ClozeGER achieves a new breakthrough&#10;over vanilla GER on 9 popular ASR datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.14420" label="2109.14420">
        <attvalues>
          <attvalue for="0" value="FastCorrect 2: Fast Error Correction on Multiple Candidates for&#10;  Automatic Speech Recognition" />
          <attvalue for="1" value="  Error correction is widely used in automatic speech recognition (ASR) to&#10;post-process the generated sentence, and can further reduce the word error rate&#10;(WER). Although multiple candidates are generated by an ASR system through beam&#10;search, current error correction approaches can only correct one sentence at a&#10;time, failing to leverage the voting effect from multiple candidates to better&#10;detect and correct error tokens. In this work, we propose FastCorrect 2, an&#10;error correction model that takes multiple ASR candidates as input for better&#10;correction accuracy. FastCorrect 2 adopts non-autoregressive generation for&#10;fast inference, which consists of an encoder that processes multiple source&#10;sentences and a decoder that generates the target sentence in parallel from the&#10;adjusted source sentence, where the adjustment is based on the predicted&#10;duration of each source token. However, there are some issues when handling&#10;multiple source sentences. First, it is non-trivial to leverage the voting&#10;effect from multiple source sentences since they usually vary in length. Thus,&#10;we propose a novel alignment algorithm to maximize the degree of token&#10;alignment among multiple sentences in terms of token and pronunciation&#10;similarity. Second, the decoder can only take one adjusted source sentence as&#10;input, while there are multiple source sentences. Thus, we develop a candidate&#10;predictor to detect the most suitable candidate for the decoder. Experiments on&#10;our inhouse dataset and AISHELL-1 show that FastCorrect 2 can further reduce&#10;the WER over the previous correction model with single candidate by 3.2% and&#10;2.6%, demonstrating the effectiveness of leveraging multiple candidates in ASR&#10;error correction. FastCorrect 2 achieves better performance than the cascaded&#10;re-scoring and correction pipeline and can serve as a unified post-processing&#10;module for ASR.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.01063" label="2005.01063">
        <attvalues>
          <attvalue for="0" value="A Two-Stage Masked LM Method for Term Set Expansion" />
          <attvalue for="1" value="  We tackle the task of Term Set Expansion (TSE): given a small seed set of&#10;example terms from a semantic class, finding more members of that class. The&#10;task is of great practical utility, and also of theoretical utility as it&#10;requires generalization from few examples. Previous approaches to the TSE task&#10;can be characterized as either distributional or pattern-based. We harness the&#10;power of neural masked language models (MLM) and propose a novel TSE algorithm,&#10;which combines the pattern-based and distributional approaches. Due to the&#10;small size of the seed set, fine-tuning methods are not effective, calling for&#10;more creative use of the MLM. The gist of the idea is to use the MLM to first&#10;mine for informative patterns with respect to the seed set, and then to obtain&#10;more members of the seed class by generalizing these patterns. Our method&#10;outperforms state-of-the-art TSE algorithms. Implementation is available at:&#10;https://github.com/ guykush/TermSetExpansion-MPB/&#10;" />
          <attvalue for="2" value="&#10; Term Set expansion (TSE)&#10; is the task of expanding a small seed set of terms into a larger (ideally&#10; complete) set of terms that belong to the same semantic category. For example,&#10; the seed set \{``orange'', ``apple''\} should expand into a set of fruits,&#10; while \{``orange'', ``blue''\} into a set of colors, and&#10; \{``apple'',``google''\} into a set of tech companies. &#10; Beyond being of great practical utility, the TSE task is a challenging instance&#10; of a generalization from few examples problem. Solving TSE requires the&#10; algorithm to: (1) &#10; identify the desired concept class based on few examples; and (2) &#10; identify additional members of the class.&#10;&#10;We present an effective TSE method which is based on querying large, pre-trained&#10; masked language models (MLMs). Pre-trained language models (LMs) have been shown to&#10; contain semantic \cite{tenney2019bert}, syntactic \cite{DBLP:journals/corr/abs-1901-05287, structural-probe, linzen2016agreement} and factual knowledge \cite{petroni2019language}, and to be great starting&#10; points for transfer-learning to new tasks via fine-tuning on few&#10; examples. However, the TSE seed sets are too small for fine-tuning, calling for a&#10; different approach. Our method uses the MLMs directly for the task they were&#10; trained for---language-modeling---by issuing word-completion queries and operating on the&#10; returned word distributions.&#10;&#10;Previous solutions to the TSE problem (also called semantic class induction) can be roughly categorized into&#10; distributional and pattern-based approaches \cite{DBLP:conf/coling/ShiZYW10}.&#10; Our method can be seen as a combination of the two. &#10;&#10;The distributional approach to TSE \cite{DBLP:conf/acl/Hindle90,&#10; DBLP:conf/kdd/PantelL02,DBLP:conf/emnlp/PantelCBPV09,setExpander,DBLP:conf/starsem/MahabalRM18}&#10; operates under the hypothesis that similar words appear in similar contexts&#10; \cite{DBLP:books/lib/Harris68}. &#10; These methods represent each term in the vocabulary as an embedding vector that&#10; summarizes all the contexts the term appears in in a large corpus, and then&#10; look for terms with vectors that are similar to those of the seed term.&#10; The methods differ in their context definitions and in their way of computing&#10; similarities. A shortcoming of these methods is that they consider all occurrences of a term in the corpus when calculating its representation, including many contexts that are irrelevant to the concept at hand due to polysemy, noise in the corpus or non-informative contexts.&#10;&#10;In contrast, the pattern-based approach considers specific indicative&#10; patterns that signal the desired concept, looking for them in a large corpus,&#10; and extracting the terms that appear in them.&#10; Patterns can be binary &#10; \cite{DBLP:conf/coling/Hearst92,DBLP:conf/wise/OhshimaOT06,DBLP:conf/acl/ZhangZSW09} (``such as X&#10; or Y''), indicating that both X and Y belong to the same class, or unary&#10; \cite{DBLP:conf/conll/GuptaM14,DBLP:conf/icdm/WangC07}&#10; (``fruits such as X'', ``First I painted the wall red, but then I repainted it X''), suggesting&#10; that X belongs to a certain category (fruit, color).&#10; The patterns can be determined manually \cite{DBLP:conf/coling/Hearst92} or&#10; automatically \cite{DBLP:conf/icdm/WangC07, DBLP:conf/conll/GuptaM14}.&#10; While well tailored patterns can be precise and interpretable, a notable shortcoming of pattern-based&#10; methods is their lack of coverage, due to the challenge of finding patterns that&#10; are specific enough to be accurate yet common enough in a large corpus to be&#10; useful. \cite{DBLP:conf/icdm/WangC07} use patterns from non-natural language&#10; (HTML) while \cite{DBLP:conf/conll/GuptaM14} restrict themselves to short&#10; patterns of 2-4 words to each side of the masked term.&#10; Our method. &#10; By using MLMs, we combine the power of the pattern-based and the distributional approaches: like the patterns-based approaches, we consider only&#10; specific, indicative corpus locations &#10; (retaining specificity and transparency).&#10; We then use the distributional nature of the neural LM to generalize across patterns and corpus locations.&#10;&#10;We use sentences with a single masked location as indicative&#10; patterns. For example, ``We took Rexy, our pet \underline{\hskip2em}, to&#10; the vet.&quot; is an indicative pattern for the house animals semantic class. &#10; Given an initial set of seed terms, we first search the corpus for indicative patterns for members&#10; of the set (\ref{finding}). Intuitively, an indicative pattern is a corpus location which is considered by an LM to be a good fit for all seed members.&#10; Once we identified indicative patterns, we extend the set to&#10; terms that can appear in similar patterns. We propose two methods for doing this. The first method (\ref{query}) queries an MLM for completions. While effective, this method restricts the expanded set to the LM vocabulary. The second method (\ref{sim}) uses the MLM to&#10; define a similarity metric over patterns, and searches the corpus for terms that appear in patterns that are similar&#10; to the indicative ones. &#10; To summarize, we embrace the pattern-based approach, while using distributional similarity for identifying good patterns as well as for generalizing across patterns.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Neural Language Models, Term Set Expansion, Pattern Generalization" />
        </attvalues>
      </node>
      <node id="1805.07398" label="1805.07398">
        <attvalues>
          <attvalue for="0" value="Robust Handling of Polysemy via Sparse Representations" />
          <attvalue for="1" value="  Words are polysemous and multi-faceted, with many shades of meanings. We&#10;suggest that sparse distributed representations are more suitable than other,&#10;commonly used, (dense) representations to express these multiple facets, and&#10;present Category Builder, a working system that, as we show, makes use of&#10;sparse representations to support multi-faceted lexical representations. We&#10;argue that the set expansion task is well suited to study these meaning&#10;distinctions since a word may belong to multiple sets with a different reason&#10;for membership in each. We therefore exhibit the performance of Category&#10;Builder on this task, while showing that our representation captures at the&#10;same time analogy problems such as &quot;the Ganga of Egypt&quot; or &quot;the Voldemort of&#10;Tolkien&quot;. Category Builder is shown to be a more expressive lexical&#10;representation and to outperform dense representations such as Word2Vec in some&#10;analogy classes despite being shown only two of the three input terms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.12815" label="2203.12815">
        <attvalues>
          <attvalue for="0" value="Revisiting the Effects of Leakage on Dependency Parsing" />
          <attvalue for="1" value="  Recent work by S{\o}gaard (2020) showed that, treebank size aside, overlap&#10;between training and test graphs (termed leakage) explains more of the observed&#10;variation in dependency parsing performance than other explanations. In this&#10;work we revisit this claim, testing it on more models and languages. We find&#10;that it only holds for zero-shot cross-lingual settings. We then propose a more&#10;fine-grained measure of such leakage which, unlike the original measure, not&#10;only explains but also correlates with observed performance variation. Code and&#10;data are available here: https://github.com/miriamwanner/reu-nlp-project&#10;" />
          <attvalue for="2" value="&#10;&#10;Syntactic parsing has long been one of the core natural language processing (NLP) tasks, and the proliferation of the Universal Dependencies project~\cite[UD;][]{de2021universal,nivre2017universal} has allowed the development and comparison of monolingual and multilingual models under the same syntactic framework.&#10;&#10;The performance of the dependency parsers, however, varies wildly across languages, with state-of-the-art performance ranging from labeled attachment scores below 20 (e.g. for Amharic, Erzya, Komi, or Yoruba) to more than 90 (e.g. for Spanish, Polish, Russian, or Greek).&#10;As the UD treebanks follow mostly similar annotation guidelines, comparisons of the parsing performance across languages are now possible, to an extent.&#10;&#10;In an effort to explain these cross-lingual performance differences, researchers have proposed treebank size~\cite{vania-etal-2019-systematic}, linguistic variation~\cite{nivre-etal-2007-conll}, test data sentence length or average gold&#10;dependency length~\cite{mcdonald-nivre-2011-analyzing}, and domain differences between training and test data~\cite{foster-etal-2011-news}, as potential predictors. Recently, \cite{sogaard-2020-languages} proposed that the proportion of isomorphic graph structures between the training and testing data (leakage) is a stronger predictor of the parsers' performance than any of the previously listed attributes other than training treebank size.&#10;&#10;\cite{sogaard-2020-languages} concludes that ``some languages seem easier to parse because their treebanks leak.'' This finding is potentially crucial for current parser evaluation on the existing treebanks, as well as for future treebank construction. It implies, for instance, that parsers are perhaps not as good as they seem, because they are tested on ``leaky'' test data. Perhaps one should also consider designing treebanks that do not leak between train and test, as such a test set would not have a bias toward more common phenomena.&#10;&#10;In this work, we examine this finding more closely. We extend S{\o}gaard's definition to include labeled leakage, and study it over multiple parsers in both monolingual and cross-lingual settings. We show that the finding does not hold up when tested against more modern parsers and more languages. We do identify, though, that leakage indeed predicts parser performance in zero-shot cross-lingual settings, and we dive deeper in this phenomenon with an extensive study focusing on Faroese and other Germanic languages. Last, we propose a modification of the leakage measure that both predicts and correlates with parser performance in such settings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cross-Lingual Models, Performance Evaluation, Mathematics, Natural Language Processing, Dependency Parsing" />
        </attvalues>
      </node>
      <node id="1909.02857" label="1909.02857">
        <attvalues>
          <attvalue for="0" value="A systematic comparison of methods for low-resource dependency parsing&#10;  on genuinely low-resource languages" />
          <attvalue for="1" value="  Parsers are available for only a handful of the world's languages, since they&#10;require lots of training data. How far can we get with just a small amount of&#10;training data? We systematically compare a set of simple strategies for&#10;improving low-resource parsers: data augmentation, which has not been tested&#10;before; cross-lingual training; and transliteration. Experimenting on three&#10;typologically diverse low-resource languages---North S\'ami, Galician, and&#10;Kazah---We find that (1) when only the low-resource treebank is available, data&#10;augmentation is very helpful; (2) when a related high-resource treebank is&#10;available, cross-lingual training is helpful and complements data augmentation;&#10;and (3) when the high-resource treebank uses a different writing system,&#10;transliteration into a shared orthographic spaces is also very helpful.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large annotated treebanks are available for only a tiny fraction of the world’s languages, and there is a wealth of literature on strategies for parsing with few resources \cite{Hwa:2005:BPV:1088141.1088144,zeman2008,mcdonald-emnlp11,P11-2120}. A popular approach is to train a parser on a related high-resource language and adapt it to the low-resource language. This approach benefits from the availability of Universal Dependencies \cite[UD;][]{NIVRE16.348}, prompting substantial research \cite{Tiedemann:2016:STC:3013558.3013565,agic17,K18-2019}, along with the VarDial and the CoNLL UD shared tasks \cite{vardial,K17-3001,K18-2001}.&#10;&#10;But low-resource parsing is still difficult. The organizers of the CoNLL 2018 UD shared task \cite{K18-2001} report that, in general, results on the task's nine low-resource treebanks ``are extremely low and the outputs are hardly useful for downstream applications.’’ So if we want to build a parser in a language with few resources, what can we do?&#10;To answer this question, we systematically compare several practical strategies for low-resource parsing, asking:&#10;\begin{enumerate}&#10; \item What can we do with only a very small target treebank for a low-resource language?&#10; \item What can we do if we also have a source treebank for a related high-resource language?&#10; \item What if the source and target treebanks do not share a writing system?&#10;\end{enumerate}&#10;Each of these scenarios requires different approaches. Data augmentation is applicable in all scenarios, and has proven useful for low-resource NLP in general \cite{augmentation-mt:ACL2017,bergmanis-etAl:K17-2002,sahin-emnlp18}. Transfer learning via cross-lingual training is applicable in scenarios 2 and 3. Finally, transliteration may be useful in scenario 3.&#10;&#10;To keep our scenarios as realistic as possible, we assume that no taggers are available since this would entail substantial annotation. Therefore, our neural parsing models must learn to parse from words or characters---that is, they must be lexicalized---even though there may be little shared vocabulary between source and target treebanks. While this may intuitively seem to make cross-lingual training difficult, recent results have shown that lexical parameter sharing on characters and words can in fact improve cross-lingual parsing \cite{deLhoneux-emnlp18}; and that in some circumstances, a lexicalized parser can outperform a delexicalized one, even in a low-resource setting \cite{W17-6303}. &#10;&#10;We experiment on three language pairs from different language families, in which the first of each is a genuinely low-resource language: North Sámi and Finnish (Uralic); Galician and Portuguese (Romance); and Kazakh and Turkish (Turkic), which have different writing systems. To avoid optimistic evaluation, we extensively experiment only with North Sámi, which we also analyse to understand why our cross-lingual training outperforms the other parsing strategies. We treat Galician and Kazakh as truly held-out, and test only our best methods on these languages. Our results show that:&#10;\begin{enumerate}&#10; \item When no source treebank is available, data augmentation is very helpful: dependency tree morphing improves labeled attachment score (LAS) by as much as 9.3\%. Our analysis suggests that syntactic rather than lexical variation is most useful for data augmentation. &#10; \item When a source treebank is available, cross-lingual parsing improves LAS up to 16.2\%, but data augmentation still helps, by an additional 2.6\%. Our analysis suggests that improvements from cross-lingual parsing occur because the parser learns syntactic regularities about word order, since it does not have access to POS and has little reusable information about word forms.&#10; \item If source and target treebanks have different writing systems, transliterating them to a common orthography is very effective. &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.14211" label="2204.14211">
        <attvalues>
          <attvalue for="0" value="TemporalWiki: A Lifelong Benchmark for Training and Evaluating&#10;  Ever-Evolving Language Models" />
          <attvalue for="1" value="  Language Models (LMs) become outdated as the world changes; they often fail&#10;to perform tasks requiring recent factual information which was absent or&#10;different during training, a phenomenon called temporal misalignment. This is&#10;especially a challenging problem because the research community still lacks a&#10;coherent dataset for assessing the adaptability of LMs to frequently-updated&#10;knowledge corpus such as Wikipedia. To this end, we introduce TemporalWiki, a&#10;lifelong benchmark for ever-evolving LMs that utilizes the difference between&#10;consecutive snapshots of English Wikipedia and English Wikidata for training&#10;and evaluation, respectively. The benchmark hence allows researchers to&#10;periodically track an LM's ability to retain previous knowledge and acquire&#10;updated/new knowledge at each point in time. We also find that training an LM&#10;on the diff data through continual learning methods achieves similar or better&#10;perplexity than on the entire snapshot in our benchmark with 12 times less&#10;computational cost, which verifies that factual knowledge in LMs can be safely&#10;updated with minimal training data via continual learning. The dataset and the&#10;code are available at https://github.com/joeljang/temporalwiki.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Recent works have introduced the need to tackle the issue of temporal misalignment, which refers to neural networks showing poor performance due to misalignment in time between the train and test data. Temporal misalignment can be caused either by (1) the dynamic nature of language~\cite{rottger2021temporal,hombaiah2021dynamic, rosin2021time, loureiro2022timelms} or (2) the update of factual information~\cite{chen2021dataset, dhingra2021time, jang2021towards}.&#10;&#10;\cite{luu2021time} have emphasized the effect of temporal misalignment on eight different NLP downstream tasks, asserting that misalignment between the train and test sets of the downstream tasks causes severe performance degradation that can be mitigate finetuning on the corpus from the target period. \cite{agarwal2021temporal} have argued this to be less of a concern when utilizing representations from pretrained LMs and show that self-labeling on the downstream task is more effective than continued pretraining on more recent data for temporal adaptation. Note that these works have focused on misalignment caused by the dynamic nature of language on tasks that are not knowledge-intensive, such as text classification. &#10;&#10;Others have tackled the problem caused by the update of factual knowledge. \cite{lazaridou2021mind} have shown that LMs deteriorate significantly in performance when there is a misalignment in time between the pretraining data and the downstream task and argued ever-evolving LMs are necessary. \cite{dhingra2021time} have proposed explicitly including time information during pretraining as a potential solution. \cite{jang2021towards, jin2021lifelong} have implemented continual learning methods to mitigate catastrophic forgetting that occurs during continued pretraining on new data. &#10;&#10;Despite the recent community interest in the need for ever-evolving LMs, the community lacks widely-available resources to train and evaluate such LMs. Previous works have introduced benchmarks comprised of data sources from Twitter feeds~\cite{osborne-etal-2014-exponential, yogatama2014dynamic, loureiro2022timelms}, recent news articles~\cite{jang2021towards}, and arXiv papers~\cite{lazaridou2021mind} where the temporal adaptability of LMs and the effectiveness of different methodologies of updating LMs can be evaluated. However, these data sources are domain-specific and inherently static. &#10;&#10;On the other hand, Wikipedia and Wikidata are known to be great sources of general world knowledge and thus have been widely used by the community~\cite{dinan2018wizard, thorne2018fever, kwiatkowski2019natural, piktus2021web}. 120K volunteer editors make 120 updates to the English Wikipedia per minute and add hundreds of new article entries every day~\cite{logan2021fruit}. Even though every Wikipedia and Wikidata update may not correspond to an actual change in the real world, TemporalWiki leverages the dynamic nature of Wikipedia and Wikidata to provide a lifelong benchmark for developing and maintaining ever-evolving LMs.&#10;&#10;" />
          <attvalue for="4" value="Continual Learning Methods, Computer Science, Linguistics, Language Model Updates, Artificial Intelligence, Natural Language Processing, Temporal Knowledge Alignment" />
        </attvalues>
      </node>
      <node id="2406.12585" label="2406.12585">
        <attvalues>
          <attvalue for="0" value="Breaking the Ceiling of the LLM Community by Treating Token Generation&#10;  as a Classification for Ensembling" />
          <attvalue for="1" value="  Ensembling multiple models has always been an effective approach to push the&#10;limits of existing performance and is widely used in classification tasks by&#10;simply averaging the classification probability vectors from multiple&#10;classifiers to achieve better accuracy. However, in the thriving open-source&#10;Large Language Model (LLM) community, ensembling methods are rare and typically&#10;limited to ensembling the full-text outputs of LLMs, such as selecting the best&#10;output using a ranker, which leads to underutilization of token-level&#10;probability information. In this paper, we treat the Generation of each token&#10;by LLMs as a Classification (GaC) for ensembling. This approach fully exploits&#10;the probability information at each generation step and better prevents LLMs&#10;from producing early incorrect tokens that lead to snowballing errors. In&#10;experiments, we ensemble state-of-the-art LLMs on several benchmarks, including&#10;exams, mathematics and reasoning, and observe that our method breaks the&#10;existing community performance ceiling. Furthermore, we observed that most of&#10;the tokens in the answer are simple and do not affect the correctness of the&#10;final answer. Therefore, we also experimented with ensembling only key tokens,&#10;and the results showed better performance with lower latency across benchmarks.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs) have demonstrated remarkable capabilities in a wide range of natural language processing tasks \cite{achiam2023gpt,touvron2023llama}. Over time, new and more powerful LLMs are continually being released, pushing the boundaries of the LLM community \cite{meta2024llama3,qwen2blog}. Due to the diversity of data sources, architectures and training methods, different LLMs have strengths and weaknesses in different tasks and contexts \cite{jiang2023llm}. In addition to investing significant resources in training a superior LLM, ensembling multiple existing models is another effective way to break through the community performance ceiling \cite{huang2016snapshot}, especially given the current trend in the open source LLM community to contribute only model weights rather than training data and procedures \cite{allenai2023olmo}.&#10;&#10;Taking computer vision (CV) classification as an example, it is common to ensemble the output probability vectors of multiple models (e.g. by averaging) to achieve superior results \cite{krizhevsky2017imagenet}. This approach remains effective even with recent CV models. As shown in Tab.\ref{tab1}, we selected several common CV models \cite{chen2022repghost, tan2019efficientnet,wang2021pvtv2} for ensembling and observed better accuracy on ImageNet \cite{deng2009imagenet} compared to using a single model. Similarly, the popular decoder-only LLM architecture generates text by producing tokens one by one, with each generation step resulting in a probability vector of the length of the vocabulary. Inspired by CV, we propose to treat each Generation step as a Classification task (GaC), and by ensembling multiple models, we can achieve higher accuracy, as shown in Fig.\ref{fig_1}. There is already work that simplifies problems into binary tasks, exploiting the collective wisdom of LLMs and achieving better results \cite{schoenegger2024wisdom}, demonstrating the feasibility of this approach.&#10;&#10;Another advantage is that early errors in LLMs often snowball into later errors \cite{zhang2023language}. Ensembling during generation helps prevent the generation of inaccurate tokens at each step, thereby reducing misleading cues for subsequent token generation. In this paper, we conducted experiments at several points in time between November 2023 and June 2024, ensembling available state-of-the-art (SOTA) LLMs up to each of these points. We found that this approach significantly outperformed any single model available at those times as well as existing methods on five popular benchmarks involving subject examination, mathematics, reasoning, and knowledge-based QA.&#10;&#10;In addition, we found that for text generation it seemed unnecessary to ensemble at every step. For example, for the question &quot;What Andean animal has banana-shaped ears?&quot; shown in Fig.\ref{fig_1}, the most critical part is for the LLM to generate the key token &quot;llama&quot;. The initial part of the answer &quot;It should be \_&quot; or &quot;The animal is \_&quot; do not significantly affect the correctness of the final answer. Ideally, the step that produces the token &quot;llama&quot; is the one we want to ensemble.&#10;&#10;Studies in CV classification have also shown that most samples are &quot;simple&quot; and can be correctly classified by most models \cite{wang2017idk}, including cost-efficient ones, making the use of expensive models wasteful. To address this, CV classification used cascade inference \cite{jazbec2024towards,enomoro2021learning}, where a gate model passes a sample to a more powerful model only if its confidence falls below a threshold, thereby improving efficiency. Obviously, it is very important for cascading that the confidence of the gate model accurately reflects the accuracy. To ensure that LLMs are also suitable as gate models, we measured the Expected Calibration Error (ECE) \cite{guo2017calibration} of CV models and LLMs on ImageNet and MMLU \cite{hendrycks2020measuring}, as shown in Tab.\ref{tab1}. ECE is a metric that reflects the difference between a model's confidence and its accuracy. We found that the ECE of CV models and LLMs were close. Therefore, in this paper, we also applied the cascade inference to LLMs by ensembling only the &quot;key&quot; tokens to speed up generation. Our experiments showed that this approach consistently achieved better performance with lower latency across different benchmarks.&#10;&#10;In summary, we propose a framework GaC, which allows multiple heterogeneous LLMs to jointly decide the next generated token during text generation:&#10;&#10;\begin{itemize}&#10; \item When we ensemble at every generation step, experiments show that it outperforms any single SOTA LLM from different time periods and existing ensemble methods.&#10; \item By using cascade to ensemble only at important steps during text generation, experiments show better results with lower latency.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Mathematics, Classification Techniques, Ensembling Methods" />
        </attvalues>
      </node>
      <node id="2306.02561" label="2306.02561">
        <attvalues>
          <attvalue for="0" value="LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and&#10;  Generative Fusion" />
          <attvalue for="1" value="  We present LLM-Blender, an ensembling framework designed to attain&#10;consistently superior performance by leveraging the diverse strengths of&#10;multiple open-source large language models (LLMs). Our framework consists of&#10;two modules: PairRanker and GenFuser, addressing the observation that optimal&#10;LLMs for different examples can significantly vary. PairRanker employs a&#10;specialized pairwise comparison method to distinguish subtle differences&#10;between candidate outputs. It jointly encodes the input text and a pair of&#10;candidates, using cross-attention encoders to determine the superior one. Our&#10;results demonstrate that PairRanker exhibits the highest correlation with&#10;ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates,&#10;generating an improved output by capitalizing on their strengths and mitigating&#10;their weaknesses. To facilitate large-scale evaluation, we introduce a&#10;benchmark dataset, MixInstruct, which is a mixture of multiple instruction&#10;datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly&#10;outperform individual LLMs and baseline methods across various metrics,&#10;establishing a substantial performance gap.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;LLM evaluation&#10;As open-source large language models (LLMs) continue to flourish and demonstrate remarkable competitiveness across various natural language generation (NLG) tasks, assessing the capabilities of LLMs has become an exceedingly challenging endeavor.&#10;To address this issue, \cite{zheng2023judging} pioneered the creation of a chatbot arena, enabling users to provide pairwise evaluations of responses generated by two randomly selected LLMs. Based on these evaluations, they established an LLM Elo rating leaderboard.&#10;In a similar vein, \cite{zeno_chatbot} conducted an evaluation study on a customer service dataset, leveraging automated metrics such as BERTScore and ChrF~\cite{chrf}. This approach yielded similar LLM ranking results.&#10;Instead of relying solely on human evaluation, \cite{PandaLM} developed a fine-tuned model called PandaLM to compare responses generated by different LLMs. &#10;AlpacaFarm~\cite{Dubois2023AlpacaFarmAS} also aims to evaluate LLMs with pairwise feedback.&#10;&#10;Pairwise ranking&#10;Pairwise ranking, known for its long-standing effectiveness, has demonstrated exceptional performance across a wide array of NLP tasks \cite{Jamieson2011ActiveRU}. Notably, Ranknet~\cite{Burges2005LearningTR} and LambdaRank~\cite{Burges2010FromRT} have emerged as powerful techniques for various ranking problems. Furthermore, within the renowned RLHF procedure\cite{Ouyang2022TrainingLM}, these methods incorporate pairwise training of their reward model based on OPT. However, these approaches still compute scores individually and solely undergo pairwise training at the loss level. In contrast, our proposed PairReranker not only employs pairwise training but also utilizes the attention mechanism for pairwise inference during the inference stage. We posit that this approach better captures the subtleties between candidates and yields superior results, as demonstrated in Section~\ref{ssec:main_results}.&#10;&#10;Ensemble learning&#10;Ensemble learning is a widely employed technique to enhance a model's capabilities by leveraging multiple weaker models~\cite{Sagi2018EnsembleLA,Anio2019EnsembleAF}. Typically, ensemble learning is performed either by considering model weights or by combining diverse outputs. &#10;Mix-of-Experts (MoE) is a type of ensemble approach that combines the predictions of multiple specialized sub-models to improve overall performance. It has been successfully applied in various domains, such as natural language processing and computer vision~\cite{Jacobs1991AdaptiveMO,Shazeer2017OutrageouslyLN}.&#10;As for fusing multiple candidates, ~\cite{Izacard2020LeveragingPR} introduced a framework named Fusion-in-Decoder (FiD) to improve the quality of question answering by fusing retrieved text. Building upon FiD, \cite{Ravaut2022TowardsSC} further investigated the effectiveness of fusion in the context of text summarization. However, they neglected to incorporate a selection process prior to feeding the candidates into the fusion module, resulting in only moderate improvements. In contrast, our proposed approach, referred to as \methodname, initially utilizes the \ranker algorithm to filter out candidates of poor quality. Subsequently, fusion is performed exclusively on the top-ranked candidates, leading to superior performance.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1704.00109" label="1704.00109">
        <attvalues>
          <attvalue for="0" value="Snapshot Ensembles: Train 1, get M for free" />
          <attvalue for="1" value="  Ensembles of neural networks are known to be much more robust and accurate&#10;than individual networks. However, training multiple deep networks for model&#10;averaging is computationally expensive. In this paper, we propose a method to&#10;obtain the seemingly contradictory goal of ensembling multiple neural networks&#10;at no additional training cost. We achieve this goal by training a single&#10;neural network, converging to several local minima along its optimization path&#10;and saving the model parameters. To obtain repeated rapid convergence, we&#10;leverage recent work on cyclic learning rate schedules. The resulting&#10;technique, which we refer to as Snapshot Ensembling, is simple, yet&#10;surprisingly effective. We show in a series of experiments that our approach is&#10;compatible with diverse network architectures and learning tasks. It&#10;consistently yields lower error rates than state-of-the-art single models at no&#10;additional training cost, and compares favorably with traditional network&#10;ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain&#10;error rates of 3.4% and 17.4% respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.06088" label="2211.06088">
        <attvalues>
          <attvalue for="0" value="RepGhost: A Hardware-Efficient Ghost Module via Re-parameterization" />
          <attvalue for="1" value="  Feature reuse has been a key technique in light-weight convolutional neural&#10;networks (CNNs) architecture design. Current methods usually utilize a&#10;concatenation operator to keep large channel numbers cheaply (thus large&#10;network capacity) by reusing feature maps from other layers. Although&#10;concatenation is parameters- and FLOPs-free, its computational cost on hardware&#10;devices is non-negligible. To address this, this paper provides a new&#10;perspective to realize feature reuse implicitly and more efficiently instead of&#10;concatenation. A novel hardware-efficient RepGhost module is proposed for&#10;implicit feature reuse via reparameterization, instead of using concatenation&#10;operator. Based on the RepGhost module, we develop our efficient RepGhost&#10;bottleneck and RepGhostNet. Experiments on ImageNet and COCO benchmarks&#10;demonstrate that our RepGhostNet is much more effective and efficient than&#10;GhostNet and MobileNetV3 on mobile devices. Specially, our RepGhostNet&#10;surpasses GhostNet 0.5x by 2.5% Top-1 accuracy on ImageNet dataset with less&#10;parameters and comparable latency on an ARM-based mobile device. Code and model&#10;weights are available at https://github.com/ChengpengChen/RepGhost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13534" label="2305.13534">
        <attvalues>
          <attvalue for="0" value="How Language Model Hallucinations Can Snowball" />
          <attvalue for="1" value="  A major risk of using language models in practical applications is their&#10;tendency to hallucinate incorrect statements. Hallucinations are often&#10;attributed to knowledge gaps in LMs, but we hypothesize that in some cases,&#10;when justifying previously generated hallucinations, LMs output false claims&#10;that they can separately recognize as incorrect. We construct three&#10;question-answering datasets where ChatGPT and GPT-4 often state an incorrect&#10;answer and offer an explanation with at least one incorrect claim. Crucially,&#10;we find that ChatGPT and GPT-4 can identify 67% and 87% of their own mistakes,&#10;respectively. We refer to this phenomenon as hallucination snowballing: an LM&#10;over-commits to early mistakes, leading to more mistakes that it otherwise&#10;would not make.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.02652" label="2306.02652">
        <attvalues>
          <attvalue for="0" value="Towards Anytime Classification in Early-Exit Architectures by Enforcing&#10;  Conditional Monotonicity" />
          <attvalue for="1" value="  Modern predictive models are often deployed to environments in which&#10;computational budgets are dynamic. Anytime algorithms are well-suited to such&#10;environments as, at any point during computation, they can output a prediction&#10;whose quality is a function of computation time. Early-exit neural networks&#10;have garnered attention in the context of anytime computation due to their&#10;capability to provide intermediate predictions at various stages throughout the&#10;network. However, we demonstrate that current early-exit networks are not&#10;directly applicable to anytime settings, as the quality of predictions for&#10;individual data points is not guaranteed to improve with longer computation. To&#10;address this shortcoming, we propose an elegant post-hoc modification, based on&#10;the Product-of-Experts, that encourages an early-exit network to become&#10;gradually confident. This gives our deep models the property of conditional&#10;monotonicity in the prediction quality -- an essential stepping stone towards&#10;truly anytime predictive modeling using early-exit architectures. Our empirical&#10;results on standard image-classification tasks demonstrate that such behaviors&#10;can be achieved while preserving competitive accuracy on average.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.09286" label="2104.09286">
        <attvalues>
          <attvalue for="0" value="Learning to Cascade: Confidence Calibration for Improving the Accuracy&#10;  and Computational Cost of Cascade Inference Systems" />
          <attvalue for="1" value="  Recently, deep neural networks have become to be used in a variety of&#10;applications. While the accuracy of deep neural networks is increasing, the&#10;confidence score, which indicates the reliability of the prediction results, is&#10;becoming more important. Deep neural networks are seen as highly accurate but&#10;known to be overconfident, making it important to calibrate the confidence&#10;score. Many studies have been conducted on confidence calibration. They&#10;calibrate the confidence score of the model to match its accuracy, but it is&#10;not clear whether these confidence scores can improve the performance of&#10;systems that use confidence scores. This paper focuses on cascade inference&#10;systems, one kind of systems using confidence scores, and discusses the desired&#10;confidence score to improve system performance in terms of inference accuracy&#10;and computational cost. Based on the discussion, we propose a new confidence&#10;calibration method, Learning to Cascade. Learning to Cascade is a simple but&#10;novel method that optimizes the loss term for confidence calibration&#10;simultaneously with the original loss term. Experiments are conducted using two&#10;datasets, CIFAR-100 and ImageNet, in two system settings, and show that naive&#10;application of existing calibration methods to cascade inference systems&#10;sometimes performs worse. However, Learning to Cascade always achieves a better&#10;trade-off between inference accuracy and computational cost. The simplicity of&#10;Learning to Cascade allows it to be easily applied to improve the performance&#10;of existing systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.11305" label="2301.11305">
        <attvalues>
          <attvalue for="0" value="DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability&#10;  Curvature" />
          <attvalue for="1" value="  The increasing fluency and widespread usage of large language models (LLMs)&#10;highlight the desirability of corresponding tools aiding detection of&#10;LLM-generated text. In this paper, we identify a property of the structure of&#10;an LLM's probability function that is useful for such detection. Specifically,&#10;we demonstrate that text sampled from an LLM tends to occupy negative curvature&#10;regions of the model's log probability function. Leveraging this observation,&#10;we then define a new curvature-based criterion for judging if a passage is&#10;generated from a given LLM. This approach, which we call DetectGPT, does not&#10;require training a separate classifier, collecting a dataset of real or&#10;generated passages, or explicitly watermarking generated text. It uses only log&#10;probabilities computed by the model of interest and random perturbations of the&#10;passage from another generic pre-trained language model (e.g., T5). We find&#10;DetectGPT is more discriminative than existing zero-shot methods for model&#10;sample detection, notably improving detection of fake news articles generated&#10;by 20B parameter GPT-NeoX from 0.81 AUROC for the strongest zero-shot baseline&#10;to 0.95 AUROC for DetectGPT. See https://ericmitchell.ai/detectgpt for code,&#10;data, and other project information.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.12930" label="2107.12930">
        <attvalues>
          <attvalue for="0" value="gaBERT -- an Irish Language Model" />
          <attvalue for="1" value="  The BERT family of neural language models have become highly popular due to&#10;their ability to provide sequences of text with rich context-sensitive token&#10;encodings which are able to generalise well to many NLP tasks. We introduce&#10;gaBERT, a monolingual BERT model for the Irish language. We compare our gaBERT&#10;model to multilingual BERT and the monolingual Irish WikiBERT, and we show that&#10;gaBERT provides better representations for a downstream parsing task. We also&#10;show how different filtering criteria, vocabulary size and the choice of&#10;subword tokenisation model affect downstream performance. We compare the&#10;results of fine-tuning a gaBERT model with an mBERT model for the task of&#10;identifying verbal multiword expressions, and show that the fine-tuned gaBERT&#10;model also performs better at this task. We release gaBERT and related code to&#10;the community.&#10;" />
          <attvalue for="2" value="&#10;&#10;The technique of fine-tuning a self-supervised language model &#10;has become ubiquitous in Natural Language Processing (NLP)&#10;because models trained in this way have advanced&#10;evaluation scores&#10;on many tasks~\cite{radford-etal-2018-improving,peters-etal-2018-deep,devlin-etal-2019-bert}.&#10;Arguably the most popular architecture is BERT~\cite{devlin-etal-2019-bert} which uses stacks of&#10;transformer blocks&#10;to predict the identity of a masked token and to predict whether two sequences are contiguous.&#10;It has spawned many variants&#10;\cite{liu-etal-2019-roberta,lan-etal-2019-albert}&#10;and &#10;much analysis&#10;~\cite{jawahar-etal-2019-bert,chi-etal-2020-finding,rogers-etal-2020-primer}.&#10;In this paper, we introduce gaBERT, a monolingual model of Irish.&#10;&#10;Although Irish is the first official language of the Republic of Ireland,&#10; only a minority, 1.5\% of the population &#10; \cite{cso_census_2016}, use it in their everyday lives outside of the education system.&#10; As the less dominant language in a bilingual community, the availability of Irish language technology is important since it facilitates Irish speakers&#10; and learners&#10; to continue to use the language &#10; in their increasingly digital daily lives. In terms of technological support however, Irish is a low-resourced language and significantly lacking in speech and language tools and resources \cite{lynn-ELE-2022}. &#10;&#10;From a linguistic perspective, the Irish language is an inflected language, sharing linguistic features with other Celtic languages such as verb-subject-object (VSO) word order, initial mutation (lenition and eclipsis) and inflected prepositions. Inflection is common through suffixation, marking tense, number and person, while nouns are inflected for number and case. Nouns are either masculine or feminine in grammatical gender, which in turn influences declension-dependent inflections.&#10;Its inflected nature has already been shown to impact data-driven NLP tools due to data sparsity \cite{lynn-etal-2013-working}, as has the frequent use of clefting (fronting), two forms of the verb `to be' and prevalence of variable and discontiguous multiword expressions.&#10;&#10;Building upon recent progress &#10; in &#10; data-driven&#10; Irish NLP~\cite{lynn-etal-2012-irish,lynn-etal-2015-minority,walsh-etal-2019-ilfhocail,cassidy2022}, we release gaBERT&#10; with the&#10; hope that it will contribute to&#10; preserving&#10; Irish as a living language in the digital age. &#10;&#10;While there is evidence to suggest that dedicated monolingual models can be superior to a &#10;multilingual model for within-language downstream tasks \cite{devries2019bertje,virtanen-etal-2019-multilingual,farahani-etal-2020-parsbert}, other studies suggest that a multilingual model such as mBERT is a good choice for low-resourced languages \cite{wu-dredze-2020-languages,rust-etal-2020-how,chau-etal-2020-parsing}.&#10;We compare gaBERT to mBERT&#10;and to the monolingual Irish WikiBERT,&#10;both using Wikipedia as the source of training data.&#10;We base our comparison on the downstream task of&#10;universal dependency (UD)&#10;parsing, since we have labelled Irish data in the form of the Irish UD&#10;Treebank~\cite{lynn-foster-2016-universal,mcguinness-etal-2020-annotating}.&#10;We find that parsing accuracy improves when using gaBERT -- by&#10;3.7 and 3.6 LAS points over mBERT and WikiBERT, respectively.&#10;Continued pretraining of mBERT using the gaBERT training data results in&#10;a recovery of 2 LAS points over the off-the-shelf version.&#10;The benefit of the gaBERT training data is also shown in a manual analysis&#10;which compares the models on their ability to predict a masked token,&#10;as well as a Multiword Expression (MWE) identification task, where a token classification layer is trained to locate and classify verbal MWEs in text. &#10;&#10;We detail our hyperparameter search for &#10;our final model,&#10;where we consider the type of text filtering to apply,&#10;the vocabulary size and tokenisation model.&#10;We release our experiment&#10;code through&#10;GitHub and our &#10;models&#10;through the HuggingFace \cite{wolf-etal-2020-transformers} model repository.$^{,}$&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Irish Language Processing, Machine Learning, Computer Science, Linguistics, Artificial Intelligence, Neural Language Models, Multilingual NLP" />
        </attvalues>
      </node>
      <node id="2005.04511" label="2005.04511">
        <attvalues>
          <attvalue for="0" value="Finding Universal Grammatical Relations in Multilingual BERT" />
          <attvalue for="1" value="  Recent work has found evidence that Multilingual BERT (mBERT), a&#10;transformer-based multilingual masked language model, is capable of zero-shot&#10;cross-lingual transfer, suggesting that some aspects of its representations are&#10;shared cross-lingually. To better understand this overlap, we extend recent&#10;work on finding syntactic trees in neural networks' internal representations to&#10;the multilingual setting. We show that subspaces of mBERT representations&#10;recover syntactic tree distances in languages other than English, and that&#10;these subspaces are approximately shared across languages. Motivated by these&#10;results, we present an unsupervised analysis method that provides evidence&#10;mBERT learns representations of syntactic dependency labels, in the form of&#10;clusters which largely agree with the Universal Dependencies taxonomy. This&#10;evidence suggests that even without explicit supervision, multilingual masked&#10;language models learn certain linguistic universals.&#10;" />
          <attvalue for="2" value="&#10;&#10;Past work \cite{liu2019linguistic,tenney2019bert,tenney2019what} has found that masked language models such as BERT \cite{devlin2019bert} learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision.&#10;Recently, large multilingual masked language models such as Multilingual BERT (mBERT) and XLM \cite{conneau2019cross,conneau2019unsupervised} have shown strong cross-lingual performance on tasks like XNLI \cite{lample2019cross,williams2018broad} and dependency parsing \cite{wu2019beto}.&#10;Much previous analysis has been motivated by a desire to explain why BERT-like models perform so well on downstream applications in the monolingual setting, which begs the question: what properties of these models make them so cross-lingually effective?&#10;&#10;In this paper, we examine the extent to which Multilingual BERT learns a cross-lingual representation of syntactic structure. &#10;We extend probing methodology, in which a simple supervised model is used to predict linguistic properties from a model's representations.&#10;In a key departure from past work, we not only evaluate a probe's performance (on recreating dependency tree structure), but also use the probe as a window into understanding aspects of the representation that the probe was not trained on (i.e. dependency labels; Figure~\ref{fig:enfr_clusters}).&#10;In particular, we use the structural probing method of \cite{hewitt2019structural}, which probes for syntactic trees by finding a linear transformation under which two words' distance in their dependency parse is approximated by the squared distance between their model representation vectors under a linear transformation.&#10;After evaluating whether such transformations recover syntactic tree distances across languages in mBERT, we turn to analyzing the transformed vector representations themselves.&#10;&#10;We interpret the linear transformation of the structural probe as defining a syntactic subspace (Figure~\ref{fig:structuralprobe}), which intuitively may focus on syntactic aspects of the mBERT representations.&#10;Since the subspace is optimized to recreate syntactic tree distances, it has no supervision about edge labels (such as adjectival modifier or noun subject).&#10;This allows us to unsupervisedly analyze how representations of head-dependent pairs in syntactic trees cluster and qualitatively discuss how these clusters relate to linguistic notions of grammatical relations.&#10;&#10;We make the following contributions:&#10;\begin{itemize}&#10; ; see Figure~\ref{fig:enfr_clusters}.&#10; Our method allows for fine-grained analysis of the distinctions made by mBERT that disagree with UD, one way of moving past probing's limitation of detecting only linguistic properties we have training data for rather than properties inherent to the model.&#10;&#10;\end{itemize}&#10;Our analysis sheds light on the cross-lingual properties of Multilingual BERT, through both zero-shot cross-lingual structural probe experiments and novel unsupervised dependency label discovery experiments which treat the probe's syntactic subspace as an object of study.&#10;We find evidence that mBERT induces universal grammatical relations without any explicit supervision, which largely agree with the dependency labels of Universal Dependencies.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.12515" label="2005.12515">
        <attvalues>
          <attvalue for="0" value="ParsBERT: Transformer-based Model for Persian Language Understanding" />
          <attvalue for="1" value="  The surge of pre-trained language models has begun a new era in the field of&#10;Natural Language Processing (NLP) by allowing us to build powerful language&#10;models. Among these models, Transformer-based models such as BERT have become&#10;increasingly popular due to their state-of-the-art performance. However, these&#10;models are usually focused on English, leaving other languages to multilingual&#10;models with limited resources. This paper proposes a monolingual BERT for the&#10;Persian language (ParsBERT), which shows its state-of-the-art performance&#10;compared to other architectures and multilingual models. Also, since the amount&#10;of data available for NLP tasks in Persian is very restricted, a massive&#10;dataset for different NLP tasks as well as pre-training the model is composed.&#10;ParsBERT obtains higher scores in all datasets, including existing ones as well&#10;as composed ones and improves the state-of-the-art performance by outperforming&#10;both multilingual BERT and other prior works in Sentiment Analysis, Text&#10;Classification and Named Entity Recognition tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.12265" label="2210.12265">
        <attvalues>
          <attvalue for="0" value="On the Calibration of Massively Multilingual Language Models" />
          <attvalue for="1" value="  Massively Multilingual Language Models (MMLMs) have recently gained&#10;popularity due to their surprising effectiveness in cross-lingual transfer.&#10;While there has been much work in evaluating these models for their performance&#10;on a variety of tasks and languages, little attention has been paid on how well&#10;calibrated these models are with respect to the confidence in their&#10;predictions. We first investigate the calibration of MMLMs in the zero-shot&#10;setting and observe a clear case of miscalibration in low-resource languages or&#10;those which are typologically diverse from English. Next, we empirically show&#10;that calibration methods like temperature scaling and label smoothing do&#10;reasonably well towards improving calibration in the zero-shot scenario. We&#10;also find that few-shot examples in the language can further help reduce the&#10;calibration errors, often substantially. Overall, our work contributes towards&#10;building more reliable multilingual models by highlighting the issue of their&#10;miscalibration, understanding what language and model specific factors&#10;influence it, and pointing out the strategies to improve the same.&#10;" />
          <attvalue for="2" value="&#10;&#10;Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLMR \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5} and mBART \cite{liu-etal-2020-multilingual-denoising} have been surprisingly effective at zero-shot cross lingual transfer i.e. when fine-tuned on an NLP task in one language, they often tend to generalize reasonably well in languages unseen during fine-tuning. &#10;&#10;These models have been evaluated for their performance across a range of multilingual tasks \cite{Pan2017, nivre2018universal, Conneau2018xnli} and numerous methods like adapters \cite{pfeiffer-etal-2020-mad}, sparse fine-tuning \cite{ansell-etal-2022-composable} and few-shot learning \cite{lauscher-etal-2020-zero} have been proposed to further improve performance of cross lingual transfer. &#10;&#10;Despite these developments, there has been little to no attention paid to the calibration of these models across languages i.e. how reliable the confidence predictions of these models are. &#10;As these models find their way more and more into the real word applications with safety implications, like Hate Speech Detection \cite{DavidsonWMW17, deshpande-2022-highly} it becomes important to only take extreme actions for high confidence predictions by the model \cite{Sarkar_KhudaBukhsh_2021}. Hence, calibrated confidences are desirable to have when deploying such systems in practice. &#10;&#10;\cite{guo-2017-on} showed that modern neural networks used for Image Recognition \cite{he-et-al-2016-deep} perform much better than the ones introduced decades ago \cite{lecun-et-al-1998-gradient}, but are significantly worse calibrated and often over-estimate their confidence on incorrect predictions. For NLP tasks specifically, \cite{desai-durrett-2020-calibration} showed that classifiers trained using pre-trained transformer based models \cite{devlin-etal-2019-bert} are well calibrated both in-domain and out-of-domain settings compared to non-pre-trained model baselines \cite{chen-etal-2017-enhanced}. Notably, \cite{ponti-etal-2021-parameter} highlights, since zero-shot cross lingual transfer represents shifts in the data distribution the point estimates are likely to be miscalibrated, which forms the core setting of our work.&#10;&#10;In light of this, our work has three main contributions. First, we investigate the calibration of two commonly used MMLMs: mBERT and XLM-R on four NLU tasks under zero-shot setting where the models are fine-tuned in English and calibration errors are computed on unseen languages. We find a clear increase in calibration errors compared to English as can be seen in Figures \ref{fig:en_oob} and \ref{fig:sw_oob}, with calibration being significantly worse for Swahili compared to English.&#10;&#10;Second, we look for factors that might affect the zero-shot calibration of MMLMs and find in most cases that the calibration error is strongly correlated with pre-training data size, syntactic similarity and sub-word overlap between the unseen language and English. This reveals that MMLMs are mis-calibrated in the zero-shot setting for low-resource languages and the languages that are typologically distant from English. &#10;&#10;Finally, we show that model calibration across different languages can be substantially improved by utilizing standard calibration techniques like Temperature Scaling \cite{guo-2017-on} and Label Smoothing \cite{pereyra-et-al-2017} without collecting any data in the language (see Figure \ref{fig:sw_zs}). Using a few examples in a language (the few-shot setting), we see even more significant drops in the calibration errors as can be seen in Figure \ref{fig:sw_fs}.&#10;&#10;To the best of our knowledge, ours is the first work to investigate and improve the calibration of MMLMs. We expect this study to be a significant contribution towards building reliable and linguistically fair multilingual models. To encourage future research in the area we make our code publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Lingual Transfer Learning, Computer Science, Multilingual Model Performance, Linguistics, Language Model Calibration, Artificial Intelligence, Statistics" />
        </attvalues>
      </node>
      <node id="2001.08210" label="2001.08210">
        <attvalues>
          <attvalue for="0" value="Multilingual Denoising Pre-training for Neural Machine Translation" />
          <attvalue for="1" value="  This paper demonstrates that multilingual denoising pre-training produces&#10;significant performance gains across a wide variety of machine translation (MT)&#10;tasks. We present mBART -- a sequence-to-sequence denoising auto-encoder&#10;pre-trained on large-scale monolingual corpora in many languages using the BART&#10;objective. mBART is one of the first methods for pre-training a complete&#10;sequence-to-sequence model by denoising full texts in multiple languages, while&#10;previous approaches have focused only on the encoder, decoder, or&#10;reconstructing parts of the text. Pre-training a complete model allows it to be&#10;directly fine tuned for supervised (both sentence-level and document-level) and&#10;unsupervised machine translation, with no task-specific modifications. We&#10;demonstrate that adding mBART initialization produces performance gains in all&#10;but the highest-resource settings, including up to 12 BLEU points for low&#10;resource MT and over 5 BLEU points for many document-level and unsupervised&#10;models. We also show it also enables new types of transfer to language pairs&#10;with no bi-text or that were not in the pre-training corpus, and present&#10;extensive analysis of which factors contribute the most to effective&#10;pre-training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.04009" label="1703.04009">
        <attvalues>
          <attvalue for="0" value="Automated Hate Speech Detection and the Problem of Offensive Language" />
          <attvalue for="1" value="  A key challenge for automatic hate-speech detection on social media is the&#10;separation of hate speech from other instances of offensive language. Lexical&#10;detection methods tend to have low precision because they classify all messages&#10;containing particular terms as hate speech and previous work using supervised&#10;learning has failed to distinguish between the two categories. We used a&#10;crowd-sourced hate speech lexicon to collect tweets containing hate speech&#10;keywords. We use crowd-sourcing to label a sample of these tweets into three&#10;categories: those containing hate speech, only offensive language, and those&#10;with neither. We train a multi-class classifier to distinguish between these&#10;different categories. Close analysis of the predictions and the errors shows&#10;when we can reliably separate hate speech from other offensive language and&#10;when this differentiation is more difficult. We find that racist and homophobic&#10;tweets are more likely to be classified as hate speech but that sexist tweets&#10;are generally classified as offensive. Tweets without explicit hate keywords&#10;are also more difficult to classify.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.07892" label="2003.07892">
        <attvalues>
          <attvalue for="0" value="Calibration of Pre-trained Transformers" />
          <attvalue for="1" value="  Pre-trained Transformers are now ubiquitous in natural language processing,&#10;but despite their high end-task performance, little is known empirically about&#10;whether they are calibrated. Specifically, do these models' posterior&#10;probabilities provide an accurate empirical measure of how likely the model is&#10;to be correct on a given example? We focus on BERT and RoBERTa in this work,&#10;and analyze their calibration across three tasks: natural language inference,&#10;paraphrase detection, and commonsense reasoning. For each task, we consider&#10;in-domain as well as challenging out-of-domain settings, where models face more&#10;examples they should be uncertain about. We show that: (1) when used&#10;out-of-the-box, pre-trained models are calibrated in-domain, and compared to&#10;baselines, their calibration error out-of-domain can be as much as 3.5x lower;&#10;(2) temperature scaling is effective at further reducing calibration error&#10;in-domain, and using label smoothing to deliberately increase empirical&#10;uncertainty helps calibrate posteriors out-of-domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Calibration has been well-studied in statistical machine learning, including applications in forecasting \cite{brier-verification-1950,raftery-using-2005,gneiting2007probabilistic,palmer2008toward}, medicine \cite{yang-nurses-2010,jiang-calibrating-2012}, and computer vision \cite{kendall-what-2017,guo-on-2017,lee2018training}. Past work in natural language processing has studied calibration in the non-neural \cite{nguyen-oconnor-2015-posterior} and neural \cite{kumar-2019-calibration} settings across several tasks. However, past work has not analyzed large-scale pre-trained models, and we additionally analyze out-of-domain settings, whereas past work largely focuses on in-domain calibration \cite{nguyen-oconnor-2015-posterior,guo-on-2017}.&#10;&#10;Another way of hardening models against out-of-domain data is to be able to explicitly detect these examples, which has been studied previously \cite{hendrycks2016a,liang2018enhancing,lee2018training}. However, this assumes a discrete notion of domain; calibration is a more general paradigm and gracefully handles settings where domains are less quantized.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1609.06038" label="1609.06038">
        <attvalues>
          <attvalue for="0" value="Enhanced LSTM for Natural Language Inference" />
          <attvalue for="1" value="  Reasoning and inference are central to human and artificial intelligence.&#10;Modeling inference in human language is very challenging. With the availability&#10;of large annotated data (Bowman et al., 2015), it has recently become feasible&#10;to train neural network based inference models, which have shown to be very&#10;effective. In this paper, we present a new state-of-the-art result, achieving&#10;the accuracy of 88.6% on the Stanford Natural Language Inference Dataset.&#10;Unlike the previous top models that use very complicated network architectures,&#10;we first demonstrate that carefully designing sequential inference models based&#10;on chain LSTMs can outperform all previous models. Based on this, we further&#10;show that by explicitly considering recursive architectures in both local&#10;inference modeling and inference composition, we achieve additional&#10;improvement. Particularly, incorporating syntactic parsing information&#10;contributes to our best result---it further improves the performance even when&#10;added to the already very strong model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1701.06548" label="1701.06548">
        <attvalues>
          <attvalue for="0" value="Regularizing Neural Networks by Penalizing Confident Output&#10;  Distributions" />
          <attvalue for="1" value="  We systematically explore regularizing neural networks by penalizing low&#10;entropy output distributions. We show that penalizing low entropy output&#10;distributions, which has been shown to improve exploration in reinforcement&#10;learning, acts as a strong regularizer in supervised learning. Furthermore, we&#10;connect a maximum entropy based confidence penalty to label smoothing through&#10;the direction of the KL divergence. We exhaustively evaluate the proposed&#10;confidence penalty and label smoothing on 6 common benchmarks: image&#10;classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine&#10;translation (WMT'14 English-to-German), and speech recognition (TIMIT and WSJ).&#10;We find that both label smoothing and the confidence penalty improve&#10;state-of-the-art models across benchmarks without modifying existing&#10;hyperparameters, suggesting the wide applicability of these regularizers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.04836" label="2406.04836">
        <attvalues>
          <attvalue for="0" value="Revisiting Catastrophic Forgetting in Large Language Model Tuning" />
          <attvalue for="1" value="  Catastrophic Forgetting (CF) means models forgetting previously acquired&#10;knowledge when learning new data. It compromises the effectiveness of large&#10;language models (LLMs) during fine-tuning, yet the underlying causes have not&#10;been thoroughly investigated. This paper takes the first step to reveal the&#10;direct link between the flatness of the model loss landscape and the extent of&#10;CF in the field of LLMs. Based on this, we introduce the sharpness-aware&#10;minimization to mitigate CF by flattening the loss landscape. Experiments on&#10;three widely-used fine-tuning datasets, spanning different model scales,&#10;demonstrate the effectiveness of our method in alleviating CF. Analyses show&#10;that we nicely complement the existing anti-forgetting strategies, further&#10;enhancing the resistance of LLMs to CF.&#10;" />
          <attvalue for="2" value="&#10;&#10;Instruction fine-tuning is key to improving the capabilities and controllability of large language models (LLMs)~\cite{instruction_tuning:2, instruction_tuning:3}, which have already demonstrated strong performance in various tasks~\cite{zhong2023chat,Peng2023ChatGPT4MT,Lu2023EAPrompt,ren2024healthcare}. One major obstacle to tuning LLMs is catastrophic forgetting (CF,~\cite[]{kirkpatrick2017overcoming}), which means LLMs forget prior knowledge when learning new data.&#10;Recent works have provided substantial evidence confirming the negative impact of CF on LLMs, e.g., \cite{bi2024deepseek} empirically show that the fine-tuned model is even worse than their foundation counterpart on several tasks, and \cite{agent} reveal the dropped general performance of LLMs after developing their agent capabilities.&#10;&#10;Understanding the effectiveness of LLMs during fine-tuning is important for downstream tasks, however, the underlying causes of CF remain largely unexplored. There are two types of work addressing CF, from data and model perspectives, respectively. \cite{Rehearsal} propose continual learning with rehearsal in instruction tuning, though its effectiveness varies with task selection. \cite{Wise-ft} indicate that both continual learning and weight averaging (Wise-FT) effectively preserve generality.&#10;&#10;While the above techniques are somewhat successful, they require expensive extra data-constructing and training costs, and are even sometimes impractical in LLMs, because a) the data cards of many pretrained models are unclear~\cite{shi2023detecting}, making rehearsal unfeasible, and b) anti-forgetting training brings an unstable and expensive training process~\cite{datta2023measuring}.&#10;&#10;In this work, we turn to finding a cheap, stable, and orthogonal solution to alleviate the CF in tuning LLMs. In particular, we \ding{182} reveal the high correlation between the extent of CF and the flatness of the loss landscape (LLS,~\cite[]{flatness_lls:1, flatness_lls:2}), \ding{183} mitigate the CF in LLMs by flattening the LLS from optimization perspective, \ding{184} show the complementarity between our method and existing anti-forgetting works.&#10;Specifically, we designed three probing analyses to achieve \ding{182}, and found that a flatter LLS could reduce the severity of CF. Based on our observation, we introduced ``Sharpness-Aware Minimization'' (SAM,~\cite[]{SAM}) to flatten the model LLS to approach \ding{183}. For \ding{184}, we found that our introduced optimizer nicely complements a series of anti-forgetting methods, including rehearsal~\cite{Rehearsal} and Wise-FT~\cite{Wise-ft}. &#10;Our contributions are as follows:&#10;&#10;\begin{itemize}&#10;&#09;\item To the best of our knowledge, we are the first to empirically reveal a direct correlation between the flatness of the model LLS and CF.&#10;&#09;\item We present the first work that mitigates CF of LLMs from the perspective of optimization.&#10; \item The proposed method can be synergistically combined with existing methods to enhance the resilience of LLMs against forgetting.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Language Model Forgetting, Loss Landscape Analysis, Fine-Tuning Optimization, Computational Linguistics, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2302.10198" label="2302.10198">
        <attvalues>
          <attvalue for="0" value="Can ChatGPT Understand Too? A Comparative Study on ChatGPT and&#10;  Fine-tuned BERT" />
          <attvalue for="1" value="  Recently, ChatGPT has attracted great attention, as it can generate fluent&#10;and high-quality responses to human inquiries. Several prior studies have shown&#10;that ChatGPT attains remarkable generation ability compared with existing&#10;models. However, the quantitative analysis of ChatGPT's understanding ability&#10;has been given little attention. In this report, we explore the understanding&#10;ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and&#10;comparing it with 4 representative fine-tuned BERT-style models. We find that:&#10;1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT&#10;outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT&#10;achieves comparable performance compared with BERT on sentiment analysis and&#10;question-answering tasks. Additionally, by combining some advanced prompting&#10;strategies, we show that the understanding ability of ChatGPT can be further&#10;improved.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;In recent years, we have witnessed numerous Transformer-based pretrained language models (PLMs)~\cite{devlin2019bert,liu2019roberta,brown2020language,raffel2020exploring,lewis2020bart,zhong2022e2s2,zhong2023glue} that achieved tremendous success in various natural language processing (NLP) tasks. Based on the model architectures, these PLMs can be classified into three groups: 1) encoder-only PLMs (e.g., BERT~\cite{devlin2019bert}), 2) decoder-only PLMs (e.g., GPT-3~\cite{brown2020language}) and 3) encoder-decoder PLMs (e.g., T5~\cite{raffel2020exploring}). Due to different pretraining functions, these PLMs exhibit different abilities when performing NLP tasks. Specifically, the BERT-style models are based on a bidirectional masked language modeling (MLM) objective, which enforces the models to encode the context information. Through fine-tuning on the specific task, these BERT-style models can work well on a variety of natural language understanding (NLU) tasks. On the contrary, the GPT-style models aim to predict future words towards a sequence of words. Such auto-regressive models are well-suitable for language generation, but they are unidirectional and usually fail short in the representation learning for understanding the sentence~\cite{liu2021pre, zhong2022e2s2}. &#10;&#10;More recently, a lot of work focus on scaling up the PLMs and developing the large language models (LLMs)~\cite{ouyangtraining,chowdhery2022palm,smith2022using,zhang2022opt}. \cite{weiemergent} show that LLMs exhibit emergent abilities, e.g., few-shot and zero-shot learning, when the model sizes are large enough. As a typical LLM, the recently-released ChatGPT has attracted great attention, due to its impressive ability to generate fluent and high-quality responses. There is growing interest in exploring the capabilities, applications, ethics, and failures of ChatGPT~\cite{jiao2023chatgpt,bang2023multitask,qin2023chatgpt,zhuo2023exploring,wang23cls}. Along with the research line, we mainly focus on analyzing the understanding ability of ChatGPT in this report, which is important but has been given little attention. &#10; " />
        </attvalues>
      </node>
      <node id="2402.13408" label="2402.13408">
        <attvalues>
          <attvalue for="0" value="Healthcare Copilot: Eliciting the Power of General LLMs for Medical&#10;  Consultation" />
          <attvalue for="1" value="  The copilot framework, which aims to enhance and tailor large language models&#10;(LLMs) for specific complex tasks without requiring fine-tuning, is gaining&#10;increasing attention from the community. In this paper, we introduce the&#10;construction of a Healthcare Copilot designed for medical consultation. The&#10;proposed Healthcare Copilot comprises three main components: 1) the Dialogue&#10;component, responsible for effective and safe patient interactions; 2) the&#10;Memory component, storing both current conversation data and historical patient&#10;information; and 3) the Processing component, summarizing the entire dialogue&#10;and generating reports. To evaluate the proposed Healthcare Copilot, we&#10;implement an auto-evaluation scheme using ChatGPT for two roles: as a virtual&#10;patient engaging in dialogue with the copilot, and as an evaluator to assess&#10;the quality of the dialogue. Extensive results demonstrate that the proposed&#10;Healthcare Copilot significantly enhances the capabilities of general LLMs for&#10;medical consultations in terms of inquiry capability, conversational fluency,&#10;response accuracy, and safety. Furthermore, we conduct ablation studies to&#10;highlight the contribution of each individual module in the Healthcare Copilot.&#10;Code will be made publicly available on GitHub.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.00796" label="1612.00796">
        <attvalues>
          <attvalue for="0" value="Overcoming catastrophic forgetting in neural networks" />
          <attvalue for="1" value="  The ability to learn tasks in a sequential fashion is crucial to the&#10;development of artificial intelligence. Neural networks are not, in general,&#10;capable of this and it has been widely thought that catastrophic forgetting is&#10;an inevitable feature of connectionist models. We show that it is possible to&#10;overcome this limitation and train networks that can maintain expertise on&#10;tasks which they have not experienced for a long time. Our approach remembers&#10;old tasks by selectively slowing down learning on the weights important for&#10;those tasks. We demonstrate our approach is scalable and effective by solving a&#10;set of classification tasks based on the MNIST hand written digit dataset and&#10;by learning several Atari 2600 games sequentially.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12823" label="2310.12823">
        <attvalues>
          <attvalue for="0" value="AgentTuning: Enabling Generalized Agent Abilities for LLMs" />
          <attvalue for="1" value="  Open large language models (LLMs) with great performance in various tasks&#10;have significantly advanced the development of LLMs. However, they are far&#10;inferior to commercial models such as ChatGPT and GPT-4 when acting as agents&#10;to tackle complex tasks in the real world. These agent tasks employ LLMs as the&#10;central controller responsible for planning, memorization, and tool&#10;utilization, necessitating both fine-grained prompting methods and robust LLMs&#10;to achieve satisfactory performance. Though many prompting methods have been&#10;proposed to complete particular agent tasks, there is lack of research focusing&#10;on improving the agent capabilities of LLMs themselves without compromising&#10;their general abilities. In this work, we present AgentTuning, a simple and&#10;general method to enhance the agent abilities of LLMs while maintaining their&#10;general LLM capabilities. We construct AgentInstruct, a lightweight&#10;instruction-tuning dataset containing high-quality interaction trajectories. We&#10;employ a hybrid instruction-tuning strategy by combining AgentInstruct with&#10;open-source instructions from general domains. AgentTuning is used to&#10;instruction-tune the Llama 2 series, resulting in AgentLM. Our evaluations show&#10;that AgentTuning enables LLMs' agent capabilities without compromising general&#10;abilities. The AgentLM-70B is comparable to GPT-3.5-turbo on unseen agent&#10;tasks, demonstrating generalized agent capabilities. We open source the&#10;AgentInstruct and AgentLM-7B, 13B, and 70B models at&#10;https://github.com/THUDM/AgentTuning, serving open and powerful alternatives to&#10;commercial LLMs for agent tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;LLM-as-Agent &#10;Before the rise of LLMs~\cite{GPT3,chowdhery2022palm,touvron2023llama,zeng2022glm}, agent tasks primarily relied on reinforcement learning or encoder models like BERT. With the advent of LLMs, research shifted towards LLM agents. Notably, ReAct~\cite{yao2023react} innovatively combined CoT reasoning with agent actions. Several studies also applied language models to specific agent tasks, such as online shopping~\cite{yao2022webshop}, web browsing~\cite{deng2023mind2web}, and household exploration~\cite{shridhar2020alfworld}. Recently, with ChatGPT showcasing advanced planning and reasoning skills, research like ReWOO~\cite{xu2023rewoo} and RCI~\cite{kim2023language} has delved into prompting strategies and frameworks to boost language model efficiency in agent tasks without the need for fine-tuning.&#10;&#10;Instruction Tuning &#10;Instruction tuning aims at aligning the language models to follow human instructions and produce outputs that better fit human preferences. Instruction tuning mainly focus on training language models to follow human instructions among multiple general tasks. For instance, FLAN~\cite{wei2022finetuned} and T0~\cite{sanh2022multitask} demonstrates the strong zero-shot generalization ability of language models fine-tuned on multiple task datasets. Further, FLAN-V2~\cite{longpre2023flan} explores the performance of instruction tuning across multiple scales of models and datasets. With the impressive alignment capability demonstrated by commercial LLMs, many recent works~\cite{vicuna2023, wang2023openchat} propose methods to distill instruction tuning dataset from close-sourced model to enhance the alignment of open-source models.&#10; " />
        </attvalues>
      </node>
      <node id="2205.12393" label="2205.12393">
        <attvalues>
          <attvalue for="0" value="Fine-tuned Language Models are Continual Learners" />
          <attvalue for="1" value="  Recent work on large language models relies on the intuition that most&#10;natural language processing tasks can be described via natural language&#10;instructions. Language models trained on these instructions show strong&#10;zero-shot performance on several standard datasets. However, these models even&#10;though impressive still perform poorly on a wide range of tasks outside of&#10;their respective training and evaluation sets. To address this limitation, we&#10;argue that a model should be able to keep extending its knowledge and&#10;abilities, without forgetting previous skills. In spite of the limited success&#10;of Continual Learning we show that Language Models can be continual learners.&#10;We empirically investigate the reason for this success and conclude that&#10;Continual Learning emerges from self-supervision pre-training. Our resulting&#10;model Continual-T0 (CT0) is able to learn diverse new tasks, while still&#10;maintaining good performance on previous tasks, spanning remarkably through 70&#10;datasets in total. Finally, we show that CT0 is able to combine instructions in&#10;ways it was never trained for, demonstrating some compositionality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Continual Learning&#10;&#10;Current models are limited in continuously learning without forgetting any previously acquired knowledge and abilities. Research in this direction has investigated various strategies such as External Memory, Constraints and Model Plasticity \cite{parisi2019continual}. External Memory methods often simply use rehearsal with a replay during training \cite{rebuffi2017icarl}. \cite{de2019episodic} also proposed local fine-tuning at inference time, leveraging examples similar to the considered input.&#10;&#10;Through the lens of NLP tasks, \cite{biesialska-etal-2020-continual} look at the problem of CL and discuss major challenges involved. \cite{jin2021lifelong} show CL algorithms are effective for knowledge preservation. Their study also infer that continual pretraining improves temporal generalization. \cite{douillard2021dytox} proposed a a dynamic expansion of special tokens with a transformer architecture. \cite{mi-etal-2020-continual} and \cite{madotto-etal-2021-continual} perform CL for task oriented dialog systems by using replay based strategy. \cite{cao-etal-2021-continual} propose a new CL framework for NMT models, while \cite{ke-etal-2021-adapting} proposes a novel capsule network based model called B-CL (Bert based CL) for sentiment classification tasks. \cite{jin-etal-2020-visually} show how existing CL algorithms fail at learning compositional phrases. \cite{lin-etal-2022-continual} propose a benchmark and highlight key challenges for continual model refinement in Out-of-Distribution data streams. More recently, \cite{sun2019lamol} propose a lifelong learning method LAMOL that is capable of continually learning new tasks by replaying pseudo-samples of previous tasks that require no extra memory or model capacity. To the best of our knowledge, LAMOL corresponds to the state-of-the-art for CL in NLP. Most similar to our work is that of \cite{yin-etal-2022-contintin} who also study continual learning from task instructions based on the NATURAL-INSTRUCTION benchmark \cite{mishra2021cross}.Finally instead of limiting to vision-only and language-only tasks \cite{srinivasan2022climb} study the challenge of learning multimodal tasks in a CL setting, and systematically evaluate how upstream continual learning can rapidly generalize to new multimodal and unimodal tasks &#10;&#10;Most of the aforementioned works fall into the 2 scenarios differentiated by \cite{lomonaco2017core50}: 1) learning new data of known classes (online learning), and 2) learning new classes (class-incremental learning). Thus, the study are often limited to a narrow domain, or a specific task. In our work, we propose to address Continual Learning more broadly: learning a diverse set of new tasks different from the ones used for training. For this, we leverage the idea of instruction tuning \cite{wei2021finetuned,sanh2021multitask}, that enables us to frame any NLP task as a response to a natural language input and use rehearsal as a mechanism to avoid catastrophic forgetting \cite{shin2017continual}. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2310.16789" label="2310.16789">
        <attvalues>
          <attvalue for="0" value="Detecting Pretraining Data from Large Language Models" />
          <attvalue for="1" value="  Although large language models (LLMs) are widely deployed, the data used to&#10;train them is rarely disclosed. Given the incredible scale of this data, up to&#10;trillions of tokens, it is all but certain that it includes potentially&#10;problematic text such as copyrighted materials, personally identifiable&#10;information, and test data for widely reported reference benchmarks. However,&#10;we currently have no way to know which data of these types is included or in&#10;what proportions. In this paper, we study the pretraining data detection&#10;problem: given a piece of text and black-box access to an LLM without knowing&#10;the pretraining data, can we determine if the model was trained on the provided&#10;text? To facilitate this study, we introduce a dynamic benchmark WIKIMIA that&#10;uses data created before and after model training to support gold truth&#10;detection. We also introduce a new detection method Min-K% Prob based on a&#10;simple hypothesis: an unseen example is likely to contain a few outlier words&#10;with low probabilities under the LLM, while a seen example is less likely to&#10;have words with such low probabilities. Min-K% Prob can be applied without any&#10;knowledge about the pretraining corpus or any additional training, departing&#10;from previous detection methods that require training a reference model on data&#10;that is similar to the pretraining data. Moreover, our experiments demonstrate&#10;that Min-K% Prob achieves a 7.4% improvement on WIKIMIA over these previous&#10;methods. We apply Min-K% Prob to three real-world scenarios, copyrighted book&#10;detection, contaminated downstream example detection and privacy auditing of&#10;machine unlearning, and find it a consistently effective solution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10625" label="2305.10625">
        <attvalues>
          <attvalue for="0" value="Measuring and Mitigating Local Instability in Deep Neural Networks" />
          <attvalue for="1" value="  Deep Neural Networks (DNNs) are becoming integral components of real world&#10;services relied upon by millions of users. Unfortunately, architects of these&#10;systems can find it difficult to ensure reliable performance as irrelevant&#10;details like random initialization can unexpectedly change the outputs of a&#10;trained system with potentially disastrous consequences. We formulate the model&#10;stability problem by studying how the predictions of a model change, even when&#10;it is retrained on the same data, as a consequence of stochasticity in the&#10;training process. For Natural Language Understanding (NLU) tasks, we find&#10;instability in predictions for a significant fraction of queries. We formulate&#10;principled metrics, like per-sample ``label entropy'' across training runs or&#10;within a single training run, to quantify this phenomenon. Intriguingly, we&#10;find that unstable predictions do not appear at random, but rather appear to be&#10;clustered in data-specific ways. We study data-agnostic regularization methods&#10;to improve stability and propose new data-centric methods that exploit our&#10;local stability estimates. We find that our localized data-specific mitigation&#10;strategy dramatically outperforms data-agnostic methods, and comes within 90%&#10;of the gold standard, achieved by ensembling, at a fraction of the&#10;computational cost&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.01412" label="2010.01412">
        <attvalues>
          <attvalue for="0" value="Sharpness-Aware Minimization for Efficiently Improving Generalization" />
          <attvalue for="1" value="  In today's heavily overparameterized models, the value of the training loss&#10;provides few guarantees on model generalization ability. Indeed, optimizing&#10;only the training loss value, as is commonly done, can easily lead to&#10;suboptimal model quality. Motivated by prior work connecting the geometry of&#10;the loss landscape and generalization, we introduce a novel, effective&#10;procedure for instead simultaneously minimizing loss value and loss sharpness.&#10;In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks&#10;parameters that lie in neighborhoods having uniformly low loss; this&#10;formulation results in a min-max optimization problem on which gradient descent&#10;can be performed efficiently. We present empirical results showing that SAM&#10;improves model generalization across a variety of benchmark datasets (e.g.,&#10;CIFAR-10, CIFAR-100, ImageNet, finetuning tasks) and models, yielding novel&#10;state-of-the-art performance for several. Additionally, we find that SAM&#10;natively provides robustness to label noise on par with that provided by&#10;state-of-the-art procedures that specifically target learning with noisy&#10;labels. We open source our code at&#10;\url{https://github.com/google-research/sam}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03058" label="1911.03058">
        <attvalues>
          <attvalue for="0" value="Should All Cross-Lingual Embeddings Speak English?" />
          <attvalue for="1" value="  Most of recent work in cross-lingual word embeddings is severely&#10;Anglocentric. The vast majority of lexicon induction evaluation dictionaries&#10;are between English and another language, and the English embedding space is&#10;selected by default as the hub when learning in a multilingual setting. With&#10;this work, however, we challenge these practices. First, we show that the&#10;choice of hub language can significantly impact downstream lexicon induction&#10;performance. Second, we both expand the current evaluation dictionary&#10;collection to include all language pairs using triangulation, and also create&#10;new dictionaries for under-represented languages. Evaluating established&#10;methods over all these language pairs sheds light into their suitability and&#10;presents new challenges for the field. Finally, in our analysis we identify&#10;general guidelines for strong cross-lingual embeddings baselines, based on more&#10;than just Anglocentric experiments.&#10;" />
          <attvalue for="2" value="&#10;&#10;Continuous vectors for representing words (embeddings) \cite{turian-etal-2010-word} have become ubiquitous in modern, neural NLP. Cross-lingual representations \cite{mikolov2013exploiting} additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI).&#10;BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging \cite{zhang2016ten}, parsing \cite{ammar2016many}, document classification \cite{klementiev2012inducing}, and machine translation \cite{irvine2013combining,artetxe-etal-2018-unsupervised,lample-etal-2018-phrase}.&#10;&#10;Often, such shared representations are learned with a two-step process, whether under bilingual or multilingual settings (hereinafter BWE and MWE, respectively). First, monolingual word embeddings are learned over large swaths of text.&#10;Such pre-trained word embeddings, such as the fastText Wikipedia vectors~\cite{grave2018learning}, are available for many languages and are widely used.&#10;Second, a mapping between the languages is learned in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision \cite{zou2013bilingual}, under minimal supervision e.g.~using only identical strings \cite{smith2017offline}, or even in an unsupervised fashion \cite{zhang2017adversarial,conneau2018word}.&#10;Both in bilingual and multilingual settings, it is common that one of the language embedding spaces is the target to which all other languages get aligned (hereinafter ``the hub&quot;). We outline the details in Section~\ref{sec:mwe}.&#10;&#10;Despite all the recent progress in learning cross-lingual embeddings, we identify a major shortcoming to previous work: it is by and large English-centric. &#10;Notably, most MWE approaches essentially select English as the hub during training by default, aligning all other language spaces to the English one.&#10;We argue and empirically show, however, that English is a poor hub language choice.&#10;In BWE settings, on the other hand, it is fairly uncommon to denote which of the two languages is the hub (often this is implied to be the target language). However, we experimentally find that this choice can greatly impact downstream performance, especially when aligning distant languages.&#10;&#10;This Anglocentricity is even more evident at the evaluation stage. The lexica most commonly used for evaluation are the MUSE lexica \cite{conneau2018word} which cover 45 languages, but with translations only from and into English. Alternative evaluation dictionaries are also very English- and European-centric: \cite{dinu2014make} report results on English--Italian, \cite{artetxe-etal-2017-learning} on English--German and English--Finnish, \cite{zhang2017adversarial} on Spanish--English and Italian--English, and \cite{artetxe2018robust} between English and Italian, German, Finnish, Spanish, and Turkish.&#10;We argue that cross-lingual word embedding mapping methods should look beyond English for their evaluation benchmarks because, compared to all others, English is a language with disproportionately large available data and relatively poor inflectional morphology e.g., it lacks case, gender, and complex verbal inflection systems~\cite{aronoff2011morphology}. These two factors allow for an overly easy evaluation setting which does not necessarily generalize to other language pairs.&#10;In light of this, equal focus should instead be devoted to evaluation over more diverse language pairs that also include morphologically rich and low-resource languages.&#10;&#10;With this work, we attempt to address these shortcomings, providing the following contributions:&#10;\begin{itemize}&#10; \item We show that the choice of the hub when evaluating on diverse language pairs can lead to significantly different performance for iterative refinement methods that use a symbolic-based seed dictionary (e.g., by more than~10 percentage points for BWE over distant languages).&#10; We also show that often English is a suboptimal hub for~MWE.&#10; \item We identify some general guidelines for choosing a hub language which could lead to stronger performance; less isometry between the hub and source and target embedding spaces mildly correlates with performance, as does typological distance (a measure of language similarity based on language family membership trees). For distant languages, multilingual systems should be preferred over bilingual ones if the languages share alphabets, otherwise a bilingual system based on monolingual similarity dictionaries is preferable.&#10; \item We provide resources for training and evaluation on language pairs that do not include English. We outline a simple triangulation method with which we extend the MUSE dictionaries to an additional~4704 lexicons covering 50 languages (for a total of 4900 dictionaries, including the original English ones), and we present results on a subset of them. We also create new evaluation lexica for under-resourced, under-represented languages using Azerbaijani, Belarusian, and Galician as our test cases. Finally, we provide recipes for creating such dictionaries for any language pair with available parallel data.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Representation Bias, Artificial Intelligence, Natural Language Processing, Cross-Lingual Embeddings, Multilingual Lexicon Induction" />
        </attvalues>
      </node>
      <node id="1309.4168" label="1309.4168">
        <attvalues>
          <attvalue for="0" value="Exploiting Similarities among Languages for Machine Translation" />
          <attvalue for="1" value="  Dictionaries and phrase tables are the basis of modern statistical machine&#10;translation systems. This paper develops a method that can automate the process&#10;of generating and extending dictionaries and phrase tables. Our method can&#10;translate missing word and phrase entries by learning language structures based&#10;on large monolingual data and mapping between languages from small bilingual&#10;data. It uses distributed representation of words and learns a linear mapping&#10;between vector spaces of languages. Despite its simplicity, our method is&#10;surprisingly effective: we can achieve almost 90% precision@5 for translation&#10;of words between English and Spanish. This method makes little assumption about&#10;the languages, so it can be used to extend and refine dictionaries and&#10;translation tables for any language pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1602.01595" label="1602.01595">
        <attvalues>
          <attvalue for="0" value="Many Languages, One Parser" />
          <attvalue for="1" value="  We train one multilingual model for dependency parsing and use it to parse&#10;sentences in several languages. The parsing model uses (i) multilingual word&#10;clusters and embeddings; (ii) token-level language information; and (iii)&#10;language-specific features (fine-grained POS tags). This input representation&#10;enables the parser not only to parse effectively in multiple languages, but&#10;also to generalize across languages based on linguistic universals and&#10;typological similarities, making it more effective to learn from limited&#10;annotations. Our parser's performance compares favorably to strong baselines in&#10;a range of data scenarios, including when the target language has a large&#10;treebank, a small treebank, or no treebank for training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.01272" label="1809.01272">
        <attvalues>
          <attvalue for="0" value="Unsupervised Statistical Machine Translation" />
          <attvalue for="1" value="  While modern machine translation has relied on large parallel corpora, a&#10;recent line of work has managed to train Neural Machine Translation (NMT)&#10;systems from monolingual corpora only (Artetxe et al., 2018c; Lample et al.,&#10;2018). Despite the potential of this approach for low-resource settings,&#10;existing systems are far behind their supervised counterparts, limiting their&#10;practical interest. In this paper, we propose an alternative approach based on&#10;phrase-based Statistical Machine Translation (SMT) that significantly closes&#10;the gap with supervised systems. Our method profits from the modular&#10;architecture of SMT: we first induce a phrase table from monolingual corpora&#10;through cross-lingual embedding mappings, combine it with an n-gram language&#10;model, and fine-tune hyperparameters through an unsupervised MERT variant. In&#10;addition, iterative backtranslation improves results further, yielding, for&#10;instance, 14.08 and 26.22 BLEU points in WMT 2014 English-German and&#10;English-French, respectively, an improvement of more than 7-10 BLEU points over&#10;previous unsupervised systems, and closing the gap with supervised SMT (Moses&#10;trained on Europarl) down to 2-5 BLEU points. Our implementation is available&#10;at https://github.com/artetxem/monoses&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.06893" label="1802.06893">
        <attvalues>
          <attvalue for="0" value="Learning Word Vectors for 157 Languages" />
          <attvalue for="1" value="  Distributed word representations, or word vectors, have recently been applied&#10;to many tasks in natural language processing, leading to state-of-the-art&#10;performance. A key ingredient to the successful application of these&#10;representations is to train them on very large corpora, and use these&#10;pre-trained models in downstream tasks. In this paper, we describe how we&#10;trained such high quality word representations for 157 languages. We used two&#10;sources of data to train these models: the free online encyclopedia Wikipedia&#10;and data from the common crawl project. We also introduce three new word&#10;analogy datasets to evaluate these word vectors, for French, Hindi and Polish.&#10;Finally, we evaluate our pre-trained word vectors on 10 languages for which&#10;evaluation datasets exists, showing very strong performance compared to&#10;previous models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1702.03859" label="1702.03859">
        <attvalues>
          <attvalue for="0" value="Offline bilingual word vectors, orthogonal transformations and the&#10;  inverted softmax" />
          <attvalue for="1" value="  Usually bilingual word vectors are trained &quot;online&quot;. Mikolov et al. showed&#10;they can also be found &quot;offline&quot;, whereby two pre-trained embeddings are&#10;aligned with a linear transformation, using dictionaries compiled from expert&#10;knowledge. In this work, we prove that the linear transformation between two&#10;spaces should be orthogonal. This transformation can be obtained using the&#10;singular value decomposition. We introduce a novel &quot;inverted softmax&quot; for&#10;identifying translation pairs, with which we improve the precision @1 of&#10;Mikolov's original mapping from 34% to 43%, when translating a test set&#10;composed of both common and rare English words into Italian. Orthogonal&#10;transformations are more robust to noise, enabling us to learn the&#10;transformation without expert bilingual signal by constructing a&#10;&quot;pseudo-dictionary&quot; from the identical character strings which appear in both&#10;languages, achieving 40% precision on the same test set. Finally, we extend our&#10;method to retrieve the true translations of English sentences from a corpus of&#10;200k Italian sentences with a precision @1 of 68%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.04087" label="1710.04087">
        <attvalues>
          <attvalue for="0" value="Word Translation Without Parallel Data" />
          <attvalue for="1" value="  State-of-the-art methods for learning cross-lingual word embeddings have&#10;relied on bilingual dictionaries or parallel corpora. Recent studies showed&#10;that the need for parallel data supervision can be alleviated with&#10;character-level information. While these methods showed encouraging results,&#10;they are not on par with their supervised counterparts and are limited to pairs&#10;of languages sharing a common alphabet. In this work, we show that we can build&#10;a bilingual dictionary between two languages without using any parallel&#10;corpora, by aligning monolingual word embedding spaces in an unsupervised way.&#10;Without using any character information, our model even outperforms existing&#10;supervised methods on cross-lingual tasks for some language pairs. Our&#10;experiments demonstrate that our method works very well also for distant&#10;language pairs, like English-Russian or English-Chinese. We finally describe&#10;experiments on the English-Esperanto low-resource language pair, on which there&#10;only exists a limited amount of parallel data, to show the potential impact of&#10;our method in fully unsupervised machine translation. Our code, embeddings and&#10;dictionaries are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.02747" label="2305.02747">
        <attvalues>
          <attvalue for="0" value="Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance&#10;  Representation" />
          <attvalue for="1" value="  Dialogue Topic Segmentation (DTS) plays an essential role in a variety of&#10;dialogue modeling tasks. Previous DTS methods either focus on semantic&#10;similarity or dialogue coherence to assess topic similarity for unsupervised&#10;dialogue segmentation. However, the topic similarity cannot be fully identified&#10;via semantic similarity or dialogue coherence. In addition, the unlabeled&#10;dialogue data, which contains useful clues of utterance relationships, remains&#10;underexploited. In this paper, we propose a novel unsupervised DTS framework,&#10;which learns topic-aware utterance representations from unlabeled dialogue data&#10;through neighboring utterance matching and pseudo-segmentation. Extensive&#10;experiments on two benchmark datasets (i.e., DialSeg711 and Doc2Dial)&#10;demonstrate that our method significantly outperforms the strong baseline&#10;methods. For reproducibility, we provide our code and data&#10;at:https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/dial-start.&#10;" />
          <attvalue for="2" value="&#10;Dialogue Topic Segmentation (DTS) aims to divide a dialogue into multiple segments, wherein the utterances within each segment are similar in topic. &#10;DTS is critical in a variety of down-steam dialogue modeling tasks, such as dialogue summarization \cite{chen2020multi, rankae, liu2019automatic, qi2021improving, inan2022structured}, dialogue generation \cite{xu2021discovering, zhang2021modeling, liu2022dial2vec, hu2022unimse}, response prediction \cite{711, lin2022duplex, qian2023empathetic, lin2020discovering, he2022galaxy} and question answering \cite{yoon2018learning, zhang2022slot, dai2022cgodial}.&#10;&#10;Most existing methods for DTS follow an unsupervised paradigm, due to the high cost of collecting accurate DTS annotations to train supervised models \cite{gruenstein2005meeting, xia2022dialogue, zhong2022dialoglm, lo2021transformer} and the variability of annotation instructions across different domains.&#10;These methods generally involve two stages. &#10;First, various approaches are introduced to assess the topic similarity between the two sides of each potential segment boundary (i.e., an interval between two utterances).&#10;Second, a segmentation algorithm, such as TextTiling \cite{texttiling}, is used to identify the segment boundaries. Previous studies usually assess topic similarity through dialogue coherence or semantic similarity computed by surface features, such as lexical overlap \cite{c99, graphseg, TopicTiling}.&#10;In recent years, Song et al. \cite{song2016dialogue} assess semantic similarity using pre-trained word embeddings. The method is later extended to use sentence embeddings from pre-trained language models \cite{711, solbiati2021unsupervised, he2022space, he2022space2}, such as BERT \cite{bert} and SentenceBERT \cite{sentencebert}. &#10;Xing et al. \cite{csm} further propose Coherence Scoring Model (CSM), which employs utterance-pair coherence to assess topic similarity.&#10;&#10;Despite the remarkable progress of previous unsupervised DTS studies, several technical challenges related to modeling topic similarity and utilizing unlabeled dialogue data have not been fully resolved.&#10;First, prior methods typically rely on generic semantic similarity or dialogue coherence to assess topic similarity, but these measures are insufficient to capture it fully.&#10;Specifically, utterances that share the same topic may not be semantically similar, and vice versa.&#10;As illustrated in Figure \ref{fig:case}, dialogue coherence refers to the response relation between an utterance and its preceding context \cite{dziri2019evaluating}, reflecting whether adjacent utterances are linked together. However, two non-adjacent utterances in the same topic segment may be topically similar but not coherent.&#10;Second, unlabeled dialogue data containing useful clues about utterance relationships is beneficial for unsupervised DTS. However, it has not been effectively leveraged in prior works.&#10;In the semantic similarity-based methods, word or sentence embeddings are pre-trained on generic textual corpora and supervised Natural Language Inferring (NLI) datasets \cite{sentencebert}, which are unsuitable for unlabeled dialogue data.&#10;In the coherence-based methods, CSM \cite{csm} learns dialogue coherence from the DailyDialog dataset \cite{li2017dailydialog} without DTS annotations. However, each of these dialogues is about one single topic, and CSM utilizes the dialogue-level topic labels to produce training samples.&#10;&#10;To address the above issues, we propose a novel unsupervised DTS framework, called DialSTART (Unsupervised Dialogue Topic Segmentation with Topic-Aware Utterance RepresenTation), which learns topic-aware utterance representations from unlabeled dialogue data through neighboring utterance matching (NUM) and pseudo-segmentation. These topic-aware utterance representations are subsequently utilized in combination with the dialogue coherence to perform unsupervised segmentation.&#10;That is, neighboring utterances referring to those appearing together in one dialogue within a certain distance, are more likely to be topically similar.&#10;In unlabeled multiple-topic dialogues, such self-supervision of neighboring utterances is prone to be relatively noisy.&#10;In order to reduce the noise, we further combine the neighboring relation with pseudo-segmentation to produce refined utterance pairs that are assumed to be topically similar or dissimilar.&#10;&#10;In practice, we first acquire topic-aware utterance representations via an utterance encoder.&#10;Second, for each utterance interval, we assess the relevance score which reflects the degree to which the two sides are within the same segment.&#10;These relevance scores are utilized by the TextTiling algorithm to perform segmentation. The segmentation results are used for inference or for generating pseudo-segmentation during training.&#10;Third, we generate topically similar and dissimilar pairs for each utterance based on its neighboring utterances and pseudo-segmentation. &#10;Finally, we fine-tune the utterance encoder to distinguish between topically similar pairs and dissimilar pairs through the marginal ranking loss.&#10;We also design a relevance modeling task to optimize the whole relevance score by distinguishing between real and synthetic fragments.&#10;&#10;We conduct experiments on two dialogue topic segmentation datasets. &#10;The results show that our framework outperforms the state-of-the-art method by 8.03\% on average in terms of Pk error. Further ablative experiments validate the effectiveness of our topic similarity modeling based on the NUM task and pseudo-segmentation for unsupervised DTS. &#10;Our contributions are threefold:&#10;\begin{enumerate}&#10; \item We introduce the Neighboring Utterance Matching (NUM) task to learn topic-aware utterance representations, and exploit both topic similarity and dialogue coherence to perform unsupervised dialogue topic segmentation.&#10; \item We propose to further reduce the self-supervision noise in the NUM task on unlabeled dialogue data by pseudo-segmentation, to obtain topically similar and dissimilar utterance pairs&#10; \item Experiments have demonstrated that our novel framework outperforms the state-of-the-art method significantly and the effectiveness of topic-aware utterance representation.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Topic Segmentation, Computer Science, Linguistics, Dialogue Modeling, Artificial Intelligence, Natural Language Processing, Unsupervised Learning" />
        </attvalues>
      </node>
      <node id="2010.01672" label="2010.01672">
        <attvalues>
          <attvalue for="0" value="Multi-View Sequence-to-Sequence Models with Conversational Structure for&#10;  Abstractive Dialogue Summarization" />
          <attvalue for="1" value="  Text summarization is one of the most challenging and interesting problems in&#10;NLP. Although much attention has been paid to summarizing structured text like&#10;news reports or encyclopedia articles, summarizing conversations---an essential&#10;part of human-human/machine interaction where most important pieces of&#10;information are scattered across various utterances of different&#10;speakers---remains relatively under-investigated. This work proposes a&#10;multi-view sequence-to-sequence model by first extracting conversational&#10;structures of unstructured daily chats from different views to represent&#10;conversations and then utilizing a multi-view decoder to incorporate different&#10;views to generate dialogue summaries. Experiments on a large-scale dialogue&#10;summarization corpus demonstrated that our methods significantly outperformed&#10;previous state-of-the-art models via both automatic evaluations and human&#10;judgment. We also discussed specific challenges that current approaches faced&#10;with this task. We have publicly released our code at&#10;https://github.com/GT-SALT/Multi-View-Seq2Seq.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.12539" label="2009.12539">
        <attvalues>
          <attvalue for="0" value="Topic-Aware Multi-turn Dialogue Modeling" />
          <attvalue for="1" value="  In the retrieval-based multi-turn dialogue modeling, it remains a challenge&#10;to select the most appropriate response according to extracting salient&#10;features in context utterances. As a conversation goes on, topic shift at&#10;discourse-level naturally happens through the continuous multi-turn dialogue&#10;context. However, all known retrieval-based systems are satisfied with&#10;exploiting local topic words for context utterance representation but fail to&#10;capture such essential global topic-aware clues at discourse-level. Instead of&#10;taking topic-agnostic n-gram utterance as processing unit for matching purpose&#10;in existing systems, this paper presents a novel topic-aware solution for&#10;multi-turn dialogue modeling, which segments and extracts topic-aware&#10;utterances in an unsupervised way, so that the resulted model is capable of&#10;capturing salient topic shift at discourse-level in need and thus effectively&#10;track topic flow during multi-turn conversation. Our topic-aware modeling is&#10;implemented by a newly proposed unsupervised topic-aware segmentation algorithm&#10;and Topic-Aware Dual-attention Matching (TADAM) Network, which matches each&#10;topic segment with the response in a dual cross-attention way. Experimental&#10;results on three public datasets show TADAM can outperform the state-of-the-art&#10;method, especially by 3.3% on E-commerce dataset that has an obvious topic&#10;shift.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.03430" label="1710.03430">
        <attvalues>
          <attvalue for="0" value="Learning to Rank Question-Answer Pairs using Hierarchical Recurrent&#10;  Encoder with Latent Topic Clustering" />
          <attvalue for="1" value="  In this paper, we propose a novel end-to-end neural architecture for ranking&#10;candidate answers, that adapts a hierarchical recurrent neural network and a&#10;latent topic clustering module. With our proposed model, a text is encoded to a&#10;vector representation from an word-level to a chunk-level to effectively&#10;capture the entire meaning. In particular, by adapting the hierarchical&#10;structure, our model shows very small performance degradations in longer text&#10;comprehension while other state-of-the-art recurrent neural network models&#10;suffer from it. Additionally, the latent topic clustering module extracts&#10;semantic information from target samples. This clustering module is useful for&#10;any text related tasks by allowing each data sample to find its nearest topic&#10;cluster, thus helping the neural network model analyze the entire data. We&#10;evaluate our models on the Ubuntu Dialogue Corpus and consumer electronic&#10;domain question answering dataset, which is related to Samsung products. The&#10;proposed model shows state-of-the-art results for ranking question-answer&#10;pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="cs/0003083" label="cs/0003083">
        <attvalues>
          <attvalue for="0" value="Advances in domain independent linear text segmentation" />
          <attvalue for="1" value="  This paper describes a method for linear text segmentation which is twice as&#10;accurate and over seven times as fast as the state-of-the-art (Reynar, 1998).&#10;Inter-sentence similarity is replaced by rank in the local context. Boundary&#10;locations are discovered by divisive clustering.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.03955" label="1610.03955">
        <attvalues>
          <attvalue for="0" value="Dialogue Session Segmentation by Embedding-Enhanced TextTiling" />
          <attvalue for="1" value="  In human-computer conversation systems, the context of a user-issued&#10;utterance is particularly important because it provides useful background&#10;information of the conversation. However, it is unwise to track all previous&#10;utterances in the current session as not all of them are equally important. In&#10;this paper, we address the problem of session segmentation. We propose an&#10;embedding-enhanced TextTiling approach, inspired by the observation that&#10;conversation utterances are highly noisy, and that word embeddings provide a&#10;robust way of capturing semantics. Experimental results show that our approach&#10;achieves better performance than the TextTiling, MMD approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.03957" label="1710.03957">
        <attvalues>
          <attvalue for="0" value="DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset" />
          <attvalue for="1" value="  We develop a high-quality multi-turn dialog dataset, DailyDialog, which is&#10;intriguing in several aspects. The language is human-written and less noisy.&#10;The dialogues in the dataset reflect our daily communication way and cover&#10;various topics about our daily life. We also manually label the developed&#10;dataset with communication intention and emotion information. Then, we evaluate&#10;existing approaches on DailyDialog dataset and hope it benefit the research&#10;field of dialog systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00333" label="2005.00333">
        <attvalues>
          <attvalue for="0" value="XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning" />
          <attvalue for="1" value="  In order to simulate human language capacity, natural language processing&#10;systems must be able to reason about the dynamics of everyday situations,&#10;including their possible causes and effects. Moreover, they should be able to&#10;generalise the acquired world knowledge to new languages, modulo cultural&#10;differences. Advances in machine reasoning and cross-lingual transfer depend on&#10;the availability of challenging evaluation benchmarks. Motivated by both&#10;demands, we introduce Cross-lingual Choice of Plausible Alternatives (XCOPA), a&#10;typologically diverse multilingual dataset for causal commonsense reasoning in&#10;11 languages, which includes resource-poor languages like Eastern Apur\'imac&#10;Quechua and Haitian Creole. We evaluate a range of state-of-the-art models on&#10;this novel dataset, revealing that the performance of current methods based on&#10;multilingual pretraining and zero-shot fine-tuning falls short compared to&#10;translation-based transfer. Finally, we propose strategies to adapt&#10;multilingual models to out-of-sample resource-lean languages where only a small&#10;corpus or a bilingual dictionary is available, and report substantial&#10;improvements over the random baseline. The XCOPA dataset is freely available at&#10;github.com/cambridgeltl/xcopa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.03462" label="2401.03462">
        <attvalues>
          <attvalue for="0" value="Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon" />
          <attvalue for="1" value="  The utilization of long contexts poses a big challenge for LLMs due to their&#10;limited context window size. Although the context window can be extended&#10;through fine-tuning, it will result in a considerable cost at both training and&#10;inference time, and exert an unfavorable impact to the LLM's original&#10;capabilities. In this work, we propose a new method called Activation Beacon,&#10;which condenses LLM's raw activations into compact forms such that the LLM can&#10;perceive a longer context with a limited context window. Activation Beacon is&#10;introduced as a plug-in module, which fully preserves the LLM's original&#10;capability in short contexts. It works with the sliding window to streamingly&#10;process the long context, which leads to a competitive memory and time&#10;efficiency in both training and inference. Activation Beacon is trained with&#10;short-sequence data of diversified condensing ratios. Thanks to such a&#10;treatment, it can be effectively learned to support different context lengths&#10;with a small training cost. Our experiment verifies Activation Beacon's&#10;effectiveness of context extension: it can remarkably accomplish high-quality&#10;extension of Llama-2-7B's context by $\times100$ times (from 4K to 400K);&#10;meanwhile, it can also achieve superior performances across a variety of&#10;long-context language modeling and understanding tasks. The source code and&#10;model checkpoint are available at&#10;\url{https://github.com/FlagOpen/FlagEmbedding}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Recently, processing long context has become a fundamental capability of modern LLMs~\cite{gpt4,llama3,qwen2,deepseekai2024deepseekv2}. &#10;The recipe of context window extension is roughly the same: modifying the rotary position embedding~\cite{su2021rope} by extrapolation and interpolation~\cite{chen2023pi,ntkaware2023,peng2023yarn,ding2024longrope}, and leveraging long-dependency data in both the pre-training and post-training stage. Despite the impressive progress in effectiveness, LLMs face significant challenges in efficiency. There is significant computational cost due to the quadratic complexity of transformer, and huge memory cost because LLMs need to hold the KV activations of the entire sequence on GPU for faster decoding. Multiple threads of research endeavour to reduce these costs, which are discussed as follows.&#10;&#10;Sparse Attention.&#10;Conventional sparse attention methods require re-training a model from scratch using the designated sparse patterns~\cite{zaheer2020bigbird,beltagy2020longformer}. However, extensive recent studies have identified that the attention pattern of LLMs are naturally sparse despite they are densely trained~\cite{jiang2024minference,xiao2023streamingllm,han2023lm_infinite,zhu2024sample_attention}.&#10;They also propose to dynamically set appropriate sparse patterns for each head so that the attention mass can be largely preserved, leading to competitive performance against the full-attention method with reduced computation.&#10;However, these methods require holding all KV activations on chip to dynamically determine the optimal sparse patterns, making them unsuitable for KV cache reduction.&#10;&#10;KV Compression.&#10;This line of research focuses on compressing the KV activations to reduce the attention computation as well as the cache size.&#10;Since the KV activations are per-layer, per-head, per-token, and per-channel float numbers, they can be reduced from all the five dimensions (including the numerical dimension).&#10;For example, CLA~\cite{brandon2024cla} shares the KV cache across multiple layers; GQA~\cite{ainslie2023gqa} compresses multiple key/value heads into a single one; MLA~\cite{deepseekai2024deepseekv2} compresses the channels into fewer and more compact ones; and KIVI~\cite{liu2023kivi} quantizes the numerical value in the activations. &#10;The token-wise compression, as introduced in the following, is also known as context compression, which is orthogonal to the compression along other dimensions and hence can be jointly used.&#10;&#10;Context Compression. This type of methods aim to compress the raw context into shorter yet more compact representations. Existing studies are usually tailored for compressing short context (less than 1K), which tend to be sub-optimal for long-context compression. &#10;Specifically, Gisting~\cite{wu2023gist} compresses the user instruction into gist activations all at once. As a result, it cannot process context longer than the backbone LLM's window.&#10;ICAE~\cite{ge2024icae} and AutoCompressor~\cite{chevalier2023autocompressors} alleviate this problem by segmenting the long context into chunks and compressing each chunk.&#10;However, both of them compress the context into soft tokens, which are the major bottleneck to encapsulate the complex information in long contexts. Their compression workflow also lacks fine-grained handling of the chunked inputs, resulting in inferior compression quality.&#10;Moreover, these soft tokens require re-encoding before generation, which introduces extra overhead. &#10;Lastly, since the number of soft tokens are pre-defined, it is hard to flexibly assign the compression ratio for downstream tasks.&#10;CCM~\cite{kim2024ccm} is specifically designed for compressing conversations in online chatting, which cannot be used in general long context tasks such as long document understanding.&#10;Another branch of methods~\cite{jiang2023longllmlingua,li2024snapkv} propose to delete unimportant tokens to realize compression. However, they depend on the input question to accurately estimate the token importance, leading to low efficiency in real-world multi-turn scenarios.&#10;Compared with existing approaches, Activation Beacon is able to achieve more effective, efficient, and flexible compression.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Context Window Extension, Linguistics, Deep Learning Efficiency, Artificial Intelligence, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2205.00551" label="2205.00551">
        <attvalues>
          <attvalue for="0" value="Gender Bias in Masked Language Models for Multiple Languages" />
          <attvalue for="1" value="  Masked Language Models (MLMs) pre-trained by predicting masked tokens on&#10;large corpora have been used successfully in natural language processing tasks&#10;for a variety of languages. Unfortunately, it was reported that MLMs also learn&#10;discriminative biases regarding attributes such as gender and race. Because&#10;most studies have focused on MLMs in English, the bias of MLMs in other&#10;languages has rarely been investigated. Manual annotation of evaluation data&#10;for languages other than English has been challenging due to the cost and&#10;difficulty in recruiting annotators. Moreover, the existing bias evaluation&#10;methods require the stereotypical sentence pairs consisting of the same context&#10;with attribute words (e.g. He/She is a nurse). We propose Multilingual Bias&#10;Evaluation (MBE) score, to evaluate bias in various languages using only&#10;English attribute word lists and parallel corpora between the target language&#10;and English without requiring manually annotated data. We evaluated MLMs in&#10;eight languages using the MBE and confirmed that gender-related biases are&#10;encoded in MLMs for all those languages. We manually created datasets for&#10;gender bias in Japanese and Russian to evaluate the validity of the MBE. The&#10;results show that the bias scores reported by the MBE significantly correlates&#10;with that computed from the above manually created datasets and the existing&#10;English datasets for gender bias.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In the study of bias in English MLMs, \cite{may-etal-2019-measuring} and \cite{kurita-etal-2019-measuring} use a pair of artificial sentences created using manually written templates.&#10;However, template-based evaluation is problematic because it uses an artificial context that does not reflect the natural usage and distribution of words in the target language.&#10;To solve this problem, \cite{nadeem-etal-2021-stereoset} and \cite{nangia-etal-2020-crows} manually created bias evaluation datasets, SS and CP, respectively, with stereotypical and antistereotypical sentence-pairs with identical contexts, except the attribute words.&#10;However, recent work has pointed out various issues in CP and SS datasets and has argued that they may not provide effective measurements of stereotyping~\cite{blodgett-etal-2021-stereotyping}.&#10;In this study, (social) bias is defined as the tendency towards outputting sentences about a particular advantageous or disadvantageous group, such as males or females, given the same context by an MLM.&#10;However, these benchmarks are currently the most commonly used benchmarks for bias evaluation in MLMs, so we also use them in this work. &#10;We note that MBE is independent of any bias evaluation benchmark datasets.&#10;Our focus in this paper is on evaluating gender bias in multiple languages and not on comparing or proposing novel debiasing methods.&#10;However, for the completion of the discussion, we note that methods for debiasing MLMs using sentence vectors from MLMs~\cite{bommasani-etal-2020-interpreting} and lists of English male and female words has been studied~\cite{sedoc-ungar-2019-role,kaneko-bollegala-2021-debiasing,dev2020measuring, Chou:ACL2022}.&#10;&#10;In prior work on MLMs, social biases for languages other than English have rarely been investigated.&#10;\cite{ahn-oh-2021-mitigating} investigated ethnic bias in monolingual MLM in six languages by extending the templates to other languages using machine translation.&#10;The biases of MLMs have been evaluated using templates for English and Chinese \cite{liang-etal-2020-monolingual} and for English and German \cite{bartl-etal-2020-unmasking}.&#10;\cite{zhao-etal-2020-gender} investigated the gender bias of a classifier that predicts the occupation from resumes using multilingual word embeddings and multilingual MLM embedding in Spanish, German and French.&#10;They evaluated bias by using machine translation on the English data, when an MLM is used to create feature representations in a specific task.&#10;However, this setting is different from that of our study, where we evaluate the bias of MLMs independently of a specific task.&#10;Moreover, the above studies do not discuss or propose methods on how to create evaluation data that can be applied to many languages.&#10;&#10;Following the pioneering work by \cite{NIPS2016_a486cd07} that proposed a bias evaluation and debiasing methods, various studies have investigated social biases in English~\cite{caliskan2017semantics,zhao-etal-2018-learning,kaneko-bollegala-2019-gender,kaneko-bollegala-2021-dictionary,dev2019attenuating}. Unlike the contextual word embeddings produced by MLMs, evaluating social biases in static word embeddings is relatively less complicated because it can often be done using word lists without requiring annotated sentences.&#10;In static word embeddings, bias has been investigated in various languages besides English due to this ease of annotating evaluation data.&#10;\cite{lauscher-glavas-2019-consistently} translated the English word lists into six languages and evaluated the bias of the word embeddings.&#10;\cite{zhou-etal-2019-examining} proposed an evaluation metric for languages that require gender morphological agreement, such as in Spanish and French.&#10;\cite{friedman-etal-2019-relating} quantified the gender bias of word embeddings to understand cultural contexts with large-scale data, and used it to characterize the statistical gender gap in education, politics, economics, and health in US states and several countries.&#10;\cite{bansal2021debiasing} proposed a debiasing method by constructing the same bias space for multiple languages, and adapted it to three Indian languages.&#10;Other bias studies have been conducted for specific languages~\cite{takeshita-etal-2020-existing,10.1145/3377713.3377792,sahlgren-olsson-2019-gender,chavez-mulsa-spanakis-2020-evaluating}, but they are not easily transferable to novel languages.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Bias, Artificial Intelligence, Natural Language Processing, Statistics, Multilingual Evaluation" />
        </attvalues>
      </node>
      <node id="2009.08330" label="2009.08330">
        <attvalues>
          <attvalue for="0" value="More Embeddings, Better Sequence Labelers?" />
          <attvalue for="1" value="  Recent work proposes a family of contextual embeddings that significantly&#10;improves the accuracy of sequence labelers over non-contextual embeddings.&#10;However, there is no definite conclusion on whether we can build better&#10;sequence labelers by combining different kinds of embeddings in various&#10;settings. In this paper, we conduct extensive experiments on 3 tasks over 18&#10;datasets and 8 languages to study the accuracy of sequence labeling with&#10;various embedding concatenations and make three observations: (1) concatenating&#10;more embedding variants leads to better accuracy in rich-resource and&#10;cross-domain settings and some conditions of low-resource settings; (2)&#10;concatenating additional contextual sub-word embeddings with contextual&#10;character embeddings hurts the accuracy in extremely low-resource settings; (3)&#10;based on the conclusion of (1), concatenating additional similar contextual&#10;embeddings cannot lead to further improvements. We hope these conclusions can&#10;help people build stronger sequence labelers in various settings.&#10;" />
          <attvalue for="2" value="&#10;In recent years, sequence labelers equipped with contextual embeddings have achieved significant accuracy improvement \cite{peters-etal-2018-deep,akbik-etal-2018-contextual,devlin-etal-2019-bert,martin2019camembert} over approaches that use static non-contextual word embeddings \cite{mikolov2013distributed} and character embeddings \cite{santos2014learning}.&#10;Different types of embeddings have different inductive biases to guide the learning process. However, little work has been done to study how to concatenate these contextual embeddings and non-contextual embeddings to build better sequence labelers in multilingual, low-resource, or cross-domain settings over various sequence labeling tasks. In this paper, we empirically investigate the effectiveness of concatenating various kinds of embeddings for multilingual sequence labeling and try to answer the following questions:&#10;\begin{enumerate}&#10; \item In rich-resources settings, does combining different kinds of contextual embeddings result in a better sequence labeler? Are non-contextual embeddings helpful when the models are equipped with contextual embeddings?&#10; \item When we train models in low-resource and cross-domain settings, do the conclusions from the rich-resource settings still hold? &#10; \item Can sequence labelers automatically learn the importance of each kind of embeddings when they are concatenated?&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Sequence Labeling, Machine Learning, Computer Science, Linguistics, Contextual Embeddings, Computational Linguistics, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1708.00712" label="1708.00712">
        <attvalues>
          <attvalue for="0" value="Dynamic Data Selection for Neural Machine Translation" />
          <attvalue for="1" value="  Intelligent selection of training data has proven a successful technique to&#10;simultaneously increase training efficiency and translation performance for&#10;phrase-based machine translation (PBMT). With the recent increase in popularity&#10;of neural machine translation (NMT), we explore in this paper to what extent&#10;and how NMT can also benefit from data selection. While state-of-the-art data&#10;selection (Axelrod et al., 2011) consistently performs well for PBMT, we show&#10;that gains are substantially lower for NMT. Next, we introduce dynamic data&#10;selection for NMT, a method in which we vary the selected subset of training&#10;data between different training epochs. Our experiments show that the best&#10;results are achieved when applying a technique we call gradual fine-tuning,&#10;with improvements up to +2.6 BLEU over the original data selection approach and&#10;up to +3.1 BLEU over a general baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.07676" label="2010.07676">
        <attvalues>
          <attvalue for="0" value="Reliable Evaluations for Natural Language Inference based on a Unified&#10;  Cross-dataset Benchmark" />
          <attvalue for="1" value="  Recent studies show that crowd-sourced Natural Language Inference (NLI)&#10;datasets may suffer from significant biases like annotation artifacts. Models&#10;utilizing these superficial clues gain mirage advantages on the in-domain&#10;testing set, which makes the evaluation results over-estimated. The lack of&#10;trustworthy evaluation settings and benchmarks stalls the progress of NLI&#10;research. In this paper, we propose to assess a model's trustworthy&#10;generalization performance with cross-datasets evaluation. We present a new&#10;unified cross-datasets benchmark with 14 NLI datasets, and re-evaluate 9&#10;widely-used neural network-based NLI models as well as 5 recently proposed&#10;debiasing methods for annotation artifacts. Our proposed evaluation scheme and&#10;experimental baselines could provide a basis to inspire future reliable NLI&#10;research.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Inference~(NLI) aims to determine whether a hypothesis sentence could be inferred from a premise sentence, and the labels could be entailment, neutral, or contradiction. The development of large-scale datasets, e.g., SNLI~\cite{bowman2015large}, and MultiNLI~\cite{williams2018broad}, have greatly fertilized the research, and state-of-the-art models could achieve benchmark accuracies of over 90\% on the testing set of SNLI.&#10;&#10;However, recent studies have unveiled that these large-scale crowd-sourced datasets suffer from serious biases. The most significant one is the annotation artifacts~\cite{gururangan2018annotation}, i.e., the habits of crowd-sourcing workers when they write hypotheses leave clues for the labels. For example, negation words like no and never often suggest contradiction~\cite{gururangan2018annotation}. Models that capture this superficial pattern gain mirage advantages on the in-domain testing set and result in unreliable evaluation results.&#10;&#10;Cross-datasets evaluation, e.g., training on SNLI while testing on other NLI datasets like SICK~\cite{marelli2014sick}, is an effective way to remove the impact of biases in the training data during evaluations. As the sources and preparations of datasets differ, they are less likely to suffer from the same kind of biases~\cite{zhang2019selection}.&#10;Although different NLI datasets may be regarded as from different ``domains'', they also all belong to a same general domain --- the real world~\cite{torralba2011unbiased}, which is often witnessed in the domain generalization problems~\cite{jiang2007instance,liang2019additive}.&#10;Thus we argue that cross-datasets evaluation is a more reliable benchmark.&#10;&#10;In this paper, we propose a new unified cross-dataset benchmark composed of 14 NLI datasets for models trained on SNLI in Section~\ref{sec:eva_scheme}. &#10;We largely eliminate the inflated accuracy scores caused by the dataset bias of SNLI, and try not to let the evaluated methods rely heavily on any specific single dataset. &#10;Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}.&#10;&#10;The experimental results show that the cross-dataset generalization of NLI is still challenging. We hope our proposed evaluation scheme and experimental baselines could provide a basis to inspire the future development of reliable NLI research, which is discussed in Section~\ref{sec:direction}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Generalization, Computer Science, Linguistics, Artificial Intelligence, Dataset Evaluation, Natural Language Inference" />
        </attvalues>
      </node>
      <node id="1803.02324" label="1803.02324">
        <attvalues>
          <attvalue for="0" value="Annotation Artifacts in Natural Language Inference Data" />
          <attvalue for="1" value="  Large-scale datasets for natural language inference are created by presenting&#10;crowd workers with a sentence (premise), and asking them to generate three new&#10;sentences (hypotheses) that it entails, contradicts, or is logically neutral&#10;with respect to. We show that, in a significant portion of such data, this&#10;protocol leaves clues that make it possible to identify the label by looking&#10;only at the hypothesis, without observing the premise. Specifically, we show&#10;that a simple text categorization model can correctly classify the hypothesis&#10;alone in about 67% of SNLI (Bowman et. al, 2015) and 53% of MultiNLI (Williams&#10;et. al, 2017). Our analysis reveals that specific linguistic phenomena such as&#10;negation and vagueness are highly correlated with certain inference classes.&#10;Our findings suggest that the success of natural language inference models to&#10;date has been overestimated, and that the task remains a hard open problem.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.06221" label="1905.06221">
        <attvalues>
          <attvalue for="0" value="Selection Bias Explorations and Debias Methods for Natural Language&#10;  Sentence Matching Datasets" />
          <attvalue for="1" value="  Natural Language Sentence Matching (NLSM) has gained substantial attention&#10;from both academics and the industry, and rich public datasets contribute a lot&#10;to this process. However, biased datasets can also hurt the generalization&#10;performance of trained models and give untrustworthy evaluation results. For&#10;many NLSM datasets, the providers select some pairs of sentences into the&#10;datasets, and this sampling procedure can easily bring unintended pattern,&#10;i.e., selection bias. One example is the QuoraQP dataset, where some&#10;content-independent naive features are unreasonably predictive. Such features&#10;are the reflection of the selection bias and termed as the leakage features. In&#10;this paper, we investigate the problem of selection bias on six NLSM datasets&#10;and find that four out of them are significantly biased. We further propose a&#10;training and evaluation framework to alleviate the bias. Experimental results&#10;on QuoraQP suggest that the proposed framework can improve the generalization&#10;ability of trained models, and give more trustworthy evaluation results for&#10;real-world adoptions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.06517" label="1905.06517">
        <attvalues>
          <attvalue for="0" value="Additive Adversarial Learning for Unbiased Authentication" />
          <attvalue for="1" value="  Authentication is a task aiming to confirm the truth between data instances&#10;and personal identities. Typical authentication applications include face&#10;recognition, person re-identification, authentication based on mobile devices&#10;and so on. The recently-emerging data-driven authentication process may&#10;encounter undesired biases, i.e., the models are often trained in one domain&#10;(e.g., for people wearing spring outfits) while required to apply in other&#10;domains (e.g., they change the clothes to summer outfits). To address this&#10;issue, we propose a novel two-stage method that disentangles the class/identity&#10;from domain-differences, and we consider multiple types of domain-difference.&#10;In the first stage, we learn disentangled representations by a one-versus-rest&#10;disentangle learning (OVRDL) mechanism. In the second stage, we improve the&#10;disentanglement by an additive adversarial learning (AAL) mechanism. Moreover,&#10;we discuss the necessity to avoid a learning dilemma due to disentangling&#10;causally related types of domain-difference. Comprehensive evaluation results&#10;demonstrate the effectiveness and superiority of the proposed method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.04696" label="1709.04696">
        <attvalues>
          <attvalue for="0" value="DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language&#10;  Understanding" />
          <attvalue for="1" value="  Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely&#10;used on NLP tasks to capture the long-term and local dependencies,&#10;respectively. Attention mechanisms have recently attracted enormous interest&#10;due to their highly parallelizable computation, significantly less training&#10;time, and flexibility in modeling dependencies. We propose a novel attention&#10;mechanism in which the attention between elements from input sequence(s) is&#10;directional and multi-dimensional (i.e., feature-wise). A light-weight neural&#10;net, &quot;Directional Self-Attention Network (DiSAN)&quot;, is then proposed to learn&#10;sentence embedding, based solely on the proposed attention without any RNN/CNN&#10;structure. DiSAN is only composed of a directional self-attention with temporal&#10;order encoded, followed by a multi-dimensional attention that compresses the&#10;sequence into a vector representation. Despite its simple form, DiSAN&#10;outperforms complicated RNN models on both prediction quality and time&#10;efficiency. It achieves the best test accuracy among all sentence encoding&#10;methods and improves the most recent best result by 1.02% on the Stanford&#10;Natural Language Inference (SNLI) dataset, and shows state-of-the-art test&#10;accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language&#10;inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK),&#10;Customer Review, MPQA, TREC question-type classification and Subjectivity&#10;(SUBJ) datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1702.03814" label="1702.03814">
        <attvalues>
          <attvalue for="0" value="Bilateral Multi-Perspective Matching for Natural Language Sentences" />
          <attvalue for="1" value="  Natural language sentence matching is a fundamental technology for a variety&#10;of tasks. Previous approaches either match sentences from a single direction or&#10;only apply single granular (word-by-word or sentence-by-sentence) matching. In&#10;this work, we propose a bilateral multi-perspective matching (BiMPM) model&#10;under the &quot;matching-aggregation&quot; framework. Given two sentences $P$ and $Q$,&#10;our model first encodes them with a BiLSTM encoder. Next, we match the two&#10;encoded sentences in two directions $P \rightarrow Q$ and $P \leftarrow Q$. In&#10;each matching direction, each time step of one sentence is matched against all&#10;time-steps of the other sentence from multiple perspectives. Then, another&#10;BiLSTM layer is utilized to aggregate the matching results into a fix-length&#10;matching vector. Finally, based on the matching vector, the decision is made&#10;through a fully connected layer. We evaluate our model on three tasks:&#10;paraphrase identification, natural language inference and answer sentence&#10;selection. Experimental results on standard benchmark datasets show that our&#10;model achieves the state-of-the-art performance on all tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.10763" label="1908.10763">
        <attvalues>
          <attvalue for="0" value="Unlearn Dataset Bias in Natural Language Inference by Fitting the&#10;  Residual" />
          <attvalue for="1" value="  Statistical natural language inference (NLI) models are susceptible to&#10;learning dataset bias: superficial cues that happen to associate with the label&#10;on a particular dataset, but are not useful in general, e.g., negation words&#10;indicate contradiction. As exposed by several recent challenge datasets, these&#10;models perform poorly when such association is absent, e.g., predicting that &quot;I&#10;love dogs&quot; contradicts &quot;I don't love cats&quot;. Our goal is to design learning&#10;algorithms that guard against known dataset bias. We formalize the concept of&#10;dataset bias under the framework of distribution shift and present a simple&#10;debiasing algorithm based on residual fitting, which we call DRiFt. We first&#10;learn a biased model that only uses features that are known to relate to&#10;dataset bias. Then, we train a debiased model that fits to the residual of the&#10;biased model, focusing on examples that cannot be predicted well by biased&#10;features only. We use DRiFt to train three high-performing NLI models on two&#10;benchmark datasets, SNLI and MNLI. Our debiased models achieve significant&#10;gains over baseline models on two challenge test sets, while maintaining&#10;reasonable performance on the original test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.04242" label="1909.04242">
        <attvalues>
          <attvalue for="0" value="Mitigating Annotation Artifacts in Natural Language Inference Datasets&#10;  to Improve Cross-dataset Generalization Ability" />
          <attvalue for="1" value="  Natural language inference (NLI) aims at predicting the relationship between&#10;a given pair of premise and hypothesis. However, several works have found that&#10;there widely exists a bias pattern called annotation artifacts in NLI datasets,&#10;making it possible to identify the label only by looking at the hypothesis.&#10;This irregularity makes the evaluation results over-estimated and affects&#10;models' generalization ability. In this paper, we consider a more trust-worthy&#10;setting, i.e., cross-dataset evaluation. We explore the impacts of annotation&#10;artifacts in cross-dataset testing. Furthermore, we propose a training&#10;framework to mitigate the impacts of the bias pattern. Experimental results&#10;demonstrate that our methods can alleviate the negative effect of the artifacts&#10;and improve the generalization ability of models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Frequently-used NLI datasets such as SNLI and MultiNLI are created by crowd-sourcing~\cite{bowman2015large, williams2017broad}, during which they present workers a premise and ask them to produce three hypotheses corresponding to labels.&#10;As \cite{gururangan2018annotation} pointed out, workers may adopt some specific annotation strategies and heuristics when authoring hypotheses to save efforts, which produces certain patterns called annotation artifacts in the data.&#10;Models' trained on such datasets are heavily affected by the bias pattern~\cite{gururangan2018annotation}.&#10;&#10;\cite{wang2018if} further investigate models' robustness to the bias pattern using swapping operations. &#10;\cite{poliak2018hypothesis} demonstrate that the annotation artifacts widely exist among NLI datasets. &#10;They show that hypothesis-only-model, which refers to models trained and predict only with hypotheses, outperforms always predicting the majority-class in six of ten NLI datasets.&#10;&#10;The emergence of the pattern can be due to selection bias~\cite{rosenbaum1983central, zadrozny2004learning, d1998propensity} in the datasets preparing procedure.&#10;Several works~\cite{levy2016annotating,rudinger2017social} investigate the bias problem in relation inference datasest.&#10;\cite{zhang2019selection} investigate the selection bias embodied in the comparing relationships in six natural language sentence matching datasets and propose a debiasing training and evaluation framework.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2307.15411" label="2307.15411">
        <attvalues>
          <attvalue for="0" value="Investigating the Learning Behaviour of In-context Learning: A&#10;  Comparison with Supervised Learning" />
          <attvalue for="1" value="  Large language models (LLMs) have shown remarkable capacity for in-context&#10;learning (ICL), where learning a new task from just a few training examples is&#10;done without being explicitly pre-trained. However, despite the success of&#10;LLMs, there has been little understanding of how ICL learns the knowledge from&#10;the given prompts. In this paper, to make progress toward understanding the&#10;learning behaviour of ICL, we train the same LLMs with the same demonstration&#10;examples via ICL and supervised learning (SL), respectively, and investigate&#10;their performance under label perturbations (i.e., noisy labels and label&#10;imbalance) on a range of classification tasks. First, via extensive&#10;experiments, we find that gold labels have significant impacts on the&#10;downstream in-context performance, especially for large language models;&#10;however, imbalanced labels matter little to ICL across all model sizes. Second,&#10;when comparing with SL, we show empirically that ICL is less sensitive to label&#10;perturbations than SL, and ICL gradually attains comparable performance to SL&#10;as the model size increases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Supervised Learning, Cognitive Science, In-Context Learning, Language Model Learning, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2406.03476" label="2406.03476">
        <attvalues>
          <attvalue for="0" value="Does your data spark joy? Performance gains from domain upsampling at&#10;  the end of training" />
          <attvalue for="1" value="  Pretraining datasets for large language models (LLMs) have grown to trillions&#10;of tokens composed of large amounts of CommonCrawl (CC) web scrape along with&#10;smaller, domain-specific datasets. It is expensive to understand the impact of&#10;these domain-specific datasets on model capabilities as training at large FLOP&#10;scales is required to reveal significant changes to difficult and emergent&#10;benchmarks. Given the increasing cost of experimenting with pretraining data,&#10;how does one determine the optimal balance between the diversity in general web&#10;scrapes and the information density of domain specific data? In this work, we&#10;show how to leverage the smaller domain specific datasets by upsampling them&#10;relative to CC at the end of training to drive performance improvements on&#10;difficult benchmarks. This simple technique allows us to improve up to 6.90 pp&#10;on MMLU, 8.26 pp on GSM8K, and 6.17 pp on HumanEval relative to the base data&#10;mix for a 7B model trained for 1 trillion (T) tokens, thus rivaling Llama-2&#10;(7B)$\unicode{x2014}$a model trained for twice as long. We experiment with&#10;ablating the duration of domain upsampling from 5% to 30% of training and find&#10;that 10% to 20% percent is optimal for navigating the tradeoff between general&#10;language modeling capabilities and targeted benchmarks. We also use domain&#10;upsampling to characterize at scale the utility of individual datasets for&#10;improving various benchmarks by removing them during this final phase of&#10;training. This tool opens up the ability to experiment with the impact of&#10;different pretraining datasets at scale, but at an order of magnitude lower&#10;cost compared to full pretraining runs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretraining datasets for large language models (LLMs), such as Dolma \cite{soldaini2023dolma}, have grown to trillions of tokens.&#10;To accommodate such large scales, they are typically composed of two types of data sources. &#10;First, they contain large amounts of web scraped data processed from CommonCrawl (CC) dumps. These are typically hundreds of billions to trillions of tokens in size and contain a diverse distribution of information. However, because of their size, they are necessarily less information dense and are not as filtered.&#10;Second, LLM pretraining mixes contain datasets that either target certain domains or come from single high quality sources. &#10;These are much smaller (often less than a hundred billion tokens tokens).&#10;They are also more carefully processed and are dense with information from domains we want LLMs to be good at; &#10;however, since their sources are limited, they are often less diverse \cite{together2023redpajama}.&#10;&#10;Related Works: One of the biggest challenges to pretraining LLMs is determining the optimal strategy for mixing datasets that come from CC and smaller domain specific sources.&#10;Some previous works have opted to pretrain entirely on heavily processed CC data \cite{penedo2023refinedweb}.&#10;Others have used different heuristics to balance between CC and more domain specific datasets \cite{together2023redpajama}.&#10;However, most recent language models trained at scale disclose limited information on the contents of their pretraining data \cite{touvron2023llama, jiang2023mistral, jiang2024mixtral,team2024gemma}.&#10;At smaller scales, there have been attempts to algorithmically optimize the data mix proportions, but these methods have not been openly validated at the scale most modern language models are trained \cite{xie2024doremi}.&#10;Given the sheer cost of validating data mixing strategies at this scale, there is a paucity of open research on pretraining data for LLMs.&#10;&#10;Ideally one would conduct data mix experiments at smaller scales to identify what is a good data mix.&#10;However, this is often ineffective because large FLOP scales are required to reveal significant changes in difficult and emergent benchmarks.&#10;In fact, most LLMs trained at smaller scales register random accuracy on many important benchmarks such as MMLU \cite{wei2022emergent}.&#10;As a result, experiments at smaller scales can often be misleading; the variation between different data mixes on important benchmarks is often due to noise rather then dataset quality at this scale.&#10;On the other hand, it is prohibitively expensive and impractical to exhaustively characterize datasets by doing multiple training runs at the scale needed to measure above random performance on these metrics.&#10;&#10;In this work, our goal is to characterize the utility of an alternative approach to conduct pretraining data experiments at a reasonable scale.&#10;Our strategy is to modify the data mixture at the end of training after we have already trained for enough FLOPs to measure meaningful signal on difficult benchmarks.&#10;We show that this is an effective strategy for improving LLM pretraining data mixes with experiments that are an order of magnitude cheaper than full training runs.&#10;&#10;Contributions: &#10;\begin{itemize}&#10;\itemsep0em&#10; \item We begin with a baseline mix of publicly available datasets that achieves the same scaling of performance with FLOPs as the Llama-2 model family for a 7B model trained for 1 trillion tokens.&#10; \item We introduce domain upsampling---a data intervention which upsamples domain specific datasets relative to Common Crawl at the end of training---and demonstrate that it can boost challenging metrics. In particular, we observe improvements of up to 6.90 pp on MMLU, 8.26 pp on GSM8K, and 6.17 pp on HumanEval relative to the base data mix in our training setup. This makes our performance comparable to Llama-2 (7B) but at approximately half the training FLOPs.&#10; \item We ablate the percentage of training that utilizes domain upsampling and show 10\%-20\% is optimal for navigating the tradeoff between general language modeling capabilities and targeted benchmarks.&#10; \item We show how domain upsampling can be used as a FLOP-efficient tool to characterize how individual datasets impact model capabilities. &#10; By removing a subset of math-heavy pretraining data from the datasets we upsampled at the end of training, we quantified the impact these datasets have on specific benchmarks.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Domain Specific Data Upsampling, Computer Science, Large Language Models, Linguistics, Data Science, Artificial Intelligence, Mathematics, Pretraining Dataset Optimization" />
        </attvalues>
      </node>
      <node id="2403.08295" label="2403.08295">
        <attvalues>
          <attvalue for="0" value="Gemma: Open Models Based on Gemini Research and Technology" />
          <attvalue for="1" value="  This work introduces Gemma, a family of lightweight, state-of-the art open&#10;models built from the research and technology used to create Gemini models.&#10;Gemma models demonstrate strong performance across academic benchmarks for&#10;language understanding, reasoning, and safety. We release two sizes of models&#10;(2 billion and 7 billion parameters), and provide both pretrained and&#10;fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out&#10;of 18 text-based tasks, and we present comprehensive evaluations of safety and&#10;responsibility aspects of the models, alongside a detailed description of model&#10;development. We believe the responsible release of LLMs is critical for&#10;improving the safety of frontier models, and for enabling the next wave of LLM&#10;innovations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10429" label="2305.10429">
        <attvalues>
          <attvalue for="0" value="DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining" />
          <attvalue for="1" value="  The mixture proportions of pretraining data domains (e.g., Wikipedia, books,&#10;web text) greatly affect language model (LM) performance. In this paper, we&#10;propose Domain Reweighting with Minimax Optimization (DoReMi), which first&#10;trains a small proxy model using group distributionally robust optimization&#10;(Group DRO) over domains to produce domain weights (mixture proportions)&#10;without knowledge of downstream tasks. We then resample a dataset with these&#10;domain weights and train a larger, full-sized model. In our experiments, we use&#10;DoReMi on a 280M-parameter proxy model to set the domain weights for training&#10;an 8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi&#10;improves perplexity across all domains, even when it downweights a domain.&#10;DoReMi improves average few-shot downstream accuracy by 6.5% points over a&#10;baseline model trained using The Pile's default domain weights and reaches the&#10;baseline accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi,&#10;which has no knowledge of downstream tasks, even matches the performance of&#10;using domain weights tuned on downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Curating pretraining data for LMs.&#10;Most closely related is the GLaM dataset~\cite{du2021glam} (also used for training PaLM~\cite{chowdhery2022palm}), which has domain weights that are tuned using downstream data.&#10;Optimizing domain weights for downstream tasks can be expensive and could require search/zero-order optimization~\cite{snoek12hyper}, RL~\cite{zoph2016neural}, or heuristic assumptions on how positive/negative transfer between domains work.&#10;Example-level filtering also brings benefits for LM training.&#10;The C4 dataset~\cite{raffel2019exploring} shows gains over CommonCrawl via heuristic data cleaning methods.&#10;\cite{du2021glam,xie2023data} show that filtering the data at an example level for high-quality text that look like Wikipedia and books can significantly improve downstream performance for LMs.&#10;In contrast to these works, DoReMi sets domain weights automatically with only two small LM training runs and does not make assumptions about the type of data to prefer (Wikipedia-like, etc.).&#10;&#10;General data selection methods.&#10;Moore-Lewis selection~\cite{moore2010intelligent,axelrod2017cynical,feng2022automatic} selects examples with high cross-entropy difference (similar to excess log-perplexity) between language models trained on target and raw data. In contrast, DoReMi reweights the data without a target distribution.&#10;\cite{coleman2020selection} select examples based on the uncertainty of a small proxy model for active learning, while DoReMi uses DRO on the excess loss with respect to a reference model, and focuses on data mixture reweighting.&#10;\cite{mindermann2022prioritized} select examples in an online fashion by taking the top $k$ examples in a minibatch according to excess loss. DoReMi optimizes the data mixture before training, allowing the larger main model to train in a standard way.&#10;Many other works on data selection are in vision~\cite{sorscher2022beyond,kaushal2019learning,killamsetty2021glister,killamsetty2021gradmatch,killamsetty2021retrieve,wang2020optimizing,wei2015submodular,paul2021diet,mirzasoleiman2020coresets,sener2018active} and mainly focus on example-level subset selection with metrics such as gradient matching.&#10;Overall, these methods do not address data selection for pretraining, where the downstream data distribution may be very different from the pretraining distribution. DoReMi aims to address the pretraining/downstream distribution shift with a robust optimization approach. To the best of our knowledge, we are the first to show that reweighting the data according to losses of a small proxy LM can improve the training efficiency of much larger LM.&#10;&#10;Distributionally robust optimization.&#10;Within DRO methods for deep learning~\cite{bental2013robust,sinha2018certifiable,oren2019drolm,sagawa2020group}, we target a restricted form of shift called group shifts~\cite{duchi2019distributionally,oren2019drolm,sagawa2020group}, where the test distribution can be an unknown mixture of groups (domains).&#10;We follow DRO-LM~\cite{oren2019drolm}, which employs DRO for LMs in the group shift setting.&#10;DRO-LM also uses a baselined loss, but with a simple bigram reference model.&#10;DoReMi uses a reference model of the same size and architecture as the proxy model to ensure that the losses are on a similar scale.&#10;During optimization, DRO-LM takes a worst-case subset of each minibatch to update the model on, while we use the Group DRO optimizer~\cite{sagawa2020group} which doesn't require online subselection.&#10;If we equalize the number of examples in each minibatch used for gradient updates, online subselelction is more expensive than Group DRO since it requires running forward passes on a larger minibatch (e.g., double the minibatch size) before selecting a subset to update the model with.&#10;In comparison, the Group DRO optimizer updates the model on all examples in a weighted fashion.&#10;Overall, in contrast to these DRO methods which aim to produce robust models, we use DRO to optimize the data for training larger models more efficiently.&#10;&#10;Data-centric AI.&#10;Large-scale datasets and benchmarks have driven much of the recent progress in AI, including vision, NLP, and multimodal models~\cite{deng2009imagenet,russakovsky2015imagenet,wang2019glue,rajpurkar2016squad,raffel2019exploring, gao2020pile,schuhmann2022laion5b,gadre2023datacomp}. &#10;However, most datasets are still painstakingly created with human-generated data, manual work, and heuristics~\cite{deng2009imagenet,raffel2019exploring,gao2020pile,schuhmann2022laion5b,gadre2023datacomp}.&#10;DoReMi is a principled data-centric method that aims to improve language model training efficiency.&#10;We hope that DoReMi can provide a starting point for a general data-centric framework for language modeling via robust optimization.&#10; " />
        </attvalues>
      </node>
      <node id="2306.04399" label="2306.04399">
        <attvalues>
          <attvalue for="0" value="Transfer Learning of Transformer-based Speech Recognition Models from&#10;  Czech to Slovak" />
          <attvalue for="1" value="  In this paper, we are comparing several methods of training the Slovak speech&#10;recognition models based on the Transformers architecture. Specifically, we are&#10;exploring the approach of transfer learning from the existing Czech pre-trained&#10;Wav2Vec 2.0 model into Slovak. We are demonstrating the benefits of the&#10;proposed approach on three Slovak datasets. Our Slovak models scored the best&#10;results when initializing the weights from the Czech model at the beginning of&#10;the pre-training phase. Our results show that the knowledge stored in the Cezch&#10;pre-trained model can be successfully reused to solve tasks in Slovak while&#10;outperforming even much larger public multilingual models.&#10;" />
          <attvalue for="2" value="&#10;Transfer learning in speech recognition has been shown to be effective in improving accuracy and reducing the amount of training data required for new tasks. It is especially useful in scenarios where the amount of available training data is limited, such as low-resource languages or domains with specific acoustic characteristics. The aim of this paper is to identify a suitable transfer learning approach for two languages, Czech and Slovak. These two languages have many similarities, both in their written form and pronunciation. &#10;&#10;In our experiments, we are comparing several methods of training the Slovak models for the target task of automatic speech recognition (ASR). Specifically, we are investigating the possibilities of transferring the knowledge from the existing pre-trained Czech model into Slovak ASR tasks. &#10;Since Czech and Slovak have a lot in common, we expect this transfer learning approach to be beneficial in the target Slovak tasks because it can reuse the already trained knowledge common to both languages while suppressing the non-Slovak information in favor of Slovak-specific knowledge during the transfer. In this paper, we investigate the benefits of this transfer learning approach.&#10;&#10;We demonstrate the benefits of the proposed approach on three ASR datasets (described in detail in section\ref{sec:finetuning_data}). Two of the used datasets (CommonVoice and VoxPopuli) are public speech recognition datasets used very often for the benchmarking of ASR systems in many languages \cite{babu22_interspeech,radford2022whisper}. The third dataset, MALACH, is the Slovak portion of the very unique and challenging speech recognition dataset containing testimonies of eyewitnesses of the Holocaust recorded during 90'. We consider the MALACH dataset to be extremely important dataset for several reasons: (1) it preserves extremely valuable testimonies from our recent history, which should not be forgotten and which, alas, cannot be extended or scaled up anymore because the number of direct witnesses of the Holocaust rapidly decreases to zero as time goes on; (2) every improvement in the speech recognition accuracy unlocks new valuable historical and cartographical information encoded in the spoken utterances for researchers and public searching in this vast archive; (3) since most of the speakers were very old at the time of recording and the testimonies were spoken under heavy emotions, it is a challenging dataset to test the robustness, zero-shot performance and transfer learning ability of existing ASR models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Models, Artificial Intelligence, Speech Recognition, Natural Language Processing, Transfer Learning" />
        </attvalues>
      </node>
      <node id="2305.02215" label="2305.02215">
        <attvalues>
          <attvalue for="0" value="Exploring Linguistic Properties of Monolingual BERTs with Typological&#10;  Classification among Languages" />
          <attvalue for="1" value="  The impressive achievements of transformers force NLP researchers to delve&#10;into how these models represent the underlying structure of natural language.&#10;In this paper, we propose a novel standpoint to investigate the above issue:&#10;using typological similarities among languages to observe how their respective&#10;monolingual models encode structural information. We aim to layer-wise compare&#10;transformers for typologically similar languages to observe whether these&#10;similarities emerge for particular layers. For this investigation, we propose&#10;to use Centered Kernel Alignment to measure similarity among weight matrices.&#10;We found that syntactic typological similarity is consistent with the&#10;similarity between the weights in the middle layers, which are the pretrained&#10;BERT layers to which syntax encoding is generally attributed. Moreover, we&#10;observe that a domain adaptation on semantically equivalent texts enhances this&#10;similarity among weight matrices.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural language processing (NLP) is dominated by powerful but opaque deep neural networks.&#10;The rationale behind this trend is that deep architecture training allows rules and structural information about language to emerge directly from sentences in the target language, sacrificing the interpretable and transparent definition of language regularities.&#10;Some exceptions exist where structural syntactic information is explicitly encoded in multilayer perceptrons \cite{zanzotto-etal-2020-kermit} with relevant results on unseen sentences \cite{onoratiDark2023}. &#10;Yet, pre-trained transformers \cite{10.5555/3295222.3295349,devlin-etal-2019-bert} are offered as versatile universal sentence/text encoders that contain whatever is needed to solve any downstream task.&#10;These models outperform all other models nearly consistently after fine-tuning or domain-adaptation \cite{jin-etal-2022-lifelong-pretraining}.&#10;However, there is no guarantee that when considering languages that share similar structures, the models for those languages will represent those structures in the same way.&#10;&#10;Conversely, decades of studies in NLP have created symbol-empowered architectures where everything is explicitly represented. These architectures implement different levels of linguistic analysis: morphology, syntax, semantics, and pragmatics are a few subdisciplines of linguistics that shaped how symbolic-based NLP has been conceived. In this case, a linguistic model of a language -- a set of rules and regularities defining its behavior -- directly influences the system processing that language.&#10;Understanding whether linguistic models emerge in opaque pre-trained transformer architecture is a compelling issue.&#10;&#10;Probing transformers have mainly been used to investigate whether these model classical linguistic properties of languages. Probing consists of preparing precise sets of examples -- probe tasks -- and, eventually, observing how these examples activate transformers. In this way, BERT \cite{devlin-etal-2019-bert} contextual representations have been tested to assess their ability to model syntactic information and morphology \cite{tenney-etal-2019-bert, goldberg-syntax-bert, hewitt-manning-2019-structural, jawahar-etal-2019-bert, conen-visualizing-bert, edmiston_systematic_2020}, also comparing different monolingual BERT models \cite{nikolaev-pado-2022-word, otmakhova-etal-2022-cross}.&#10;&#10;In this study, we take a different standpoint to investigate if traces of linguistic models are encoded in monolingual BERT models: using linguistically-motivated typological similarities among languages \cite{wals}, we aim to layer-wise compare transformers for different languages to observe whether these similarities emerge between weight matrices for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices \cite{kornblith2019similarity}. We discovered that syntactic typological similarity is consistent with the similarity among weights in the middle layers both in pretrained models and in domain-adapted models after performing domain adaption on a parallel corpus.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Structure Representation, Mathematics, Transformer Model Analysis, Natural Language Processing, Multilingual Typology Comparison" />
        </attvalues>
      </node>
      <node id="1905.00414" label="1905.00414">
        <attvalues>
          <attvalue for="0" value="Similarity of Neural Network Representations Revisited" />
          <attvalue for="1" value="  Recent work has sought to understand the behavior of neural networks by&#10;comparing representations between layers and between different trained models.&#10;We examine methods for comparing neural network representations based on&#10;canonical correlation analysis (CCA). We show that CCA belongs to a family of&#10;statistics for measuring multivariate similarity, but that neither CCA nor any&#10;other statistic that is invariant to invertible linear transformation can&#10;measure meaningful similarities between representations of higher dimension&#10;than the number of data points. We introduce a similarity index that measures&#10;the relationship between representational similarity matrices and does not&#10;suffer from this limitation. This similarity index is equivalent to centered&#10;kernel alignment (CKA) and is also closely connected to CCA. Unlike CCA, CKA&#10;can reliably identify correspondences between representations in networks&#10;trained from different initializations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.02828" label="2301.02828">
        <attvalues>
          <attvalue for="0" value="Why do Nearest Neighbor Language Models Work?" />
          <attvalue for="1" value="  Language models (LMs) compute the probability of a text by sequentially&#10;computing a representation of an already-seen context and using this&#10;representation to predict the next word. Currently, most LMs calculate these&#10;representations through a neural network consuming the immediate previous&#10;context. However recently, retrieval-augmented LMs have shown to improve over&#10;standard neural LMs, by accessing information retrieved from a large datastore,&#10;in addition to their standard, parametric, next-word prediction. In this paper,&#10;we set out to understand why retrieval-augmented language models, and&#10;specifically why k-nearest neighbor language models (kNN-LMs) perform better&#10;than standard parametric LMs, even when the k-nearest neighbor component&#10;retrieves examples from the same training set that the LM was originally&#10;trained on. To this end, we perform a careful analysis of the various&#10;dimensions over which kNN-LM diverges from standard LMs, and investigate these&#10;dimensions one by one. Empirically, we identify three main reasons why kNN-LM&#10;performs better than standard LMs: using a different input representation for&#10;predicting the next tokens, approximate kNN search, and the importance of&#10;softmax temperature for the kNN distribution. Further, we incorporate these&#10;insights into the model architecture or the training procedure of the standard&#10;parametric LM, improving its results without the need for an explicit retrieval&#10;component. The code is available at https://github.com/frankxu2004/knnlm-why.&#10;" />
          <attvalue for="2" value="&#10;Language modeling is the task of predicting the probability of a text (often conditioned on context), with broad-spanning applications across natural language processing~\cite{bengio2003neural,merity2018regularizing,baevski2018adaptive,brown2020language}.&#10;This modeling is usually done by sequentially encoding a context $c_t$ using a trained neural network function $f$, and computing the probability of the next word $w_t$ according to $f(c_t)$ and a vector representation of $w_t$.&#10;&#10;Recently, retrieval-augmented LMs have shown a series of impressive results~\cite{grave2017unbounded,guu2018generating,he2020learning,khandelwal20generalization,borgeaud2022improving,alon2022neuro}. &#10;Retrieval-augmented LMs compute next token distributions based not only on the immediately preceding context $c_t$ and the model parameters, &#10;but also on an external datastore, from which examples are retrieved and incorporated into the base LM's prediction.&#10;&#10;One retrieval-augmented model that is notable for both its simplicity and efficacy is the $k$-nearest neighbor language model \cite[\knnlm;][]{khandelwal20generalization}.&#10;It extends a trained base LM by linearly interpolating the output word distribution with a \knn model. &#10;The nearest neighbors are retrieved according to the distances between the current context embedding of the base LM and all the context embeddings in the datastore. &#10;The datastore is created by encoding all contexts from any text collection, including the original LM training data.&#10;&#10;One of the most surprising results from \cite{khandelwal20generalization} is that \knnlm reduces the perplexity of the base LM even when the \knn component is retrieving examples from the same training set that the LM was originally trained on, indicating that the \knnlm improves the ability to model the training data and is not simply benefiting from access to more data.&#10;Intrigued by this, we ask questions like, could \knnlm be improving because of capacity issues in the parametric base LM?&#10;In this paper, we set out to understand why \knnlms work even in this setting.&#10;&#10;In the following sections,&#10;we first elucidate connections between the added \knn component and the standard LM component.&#10;Specifically, we note that word distributions from the two components are both calculated using a softmax function, based on the similarity of the current context embedding with a set of embeddings that corresponds to different next words.&#10;With this intuition, we formalize and generalize the non-parametric distribution calculation with the softmax layer and word embedding layer used in parametric LMs.&#10;We then show that this generalized form exposes a variety of design choices, e.g., the number of context embeddings in the datastore, the input representation used in softmax layer, different similarity functions, as well as the approximation and sparsification implementations in the \knn search. &#10;This provides a general framework for analyzing \knnlm and similar models and allows us to perform ablation studies that test the importance of various design decisions.&#10;&#10;We proceed to propose multiple hypotheses for why \knnlm works, which are testable by adjusting the various parameters exposed by our generalized formulation.&#10;Based on these hypotheses, we perform ablation experiments and analyze the nuances between different implementations of the generalized version of $P_{kNN}$.&#10;As the answer to our question, ``why \knnlms work'', we eventually show that the most probable reasons are threefold:&#10;\begin{enumerate}&#10; \item Ensembling the output of softmax using two representations from different layers of the transformer is important; in our experiments, this accounts for 55\% of the performance gain of \knnlm, or 6.5\% relative perplexity improvement compared to the base LM.&#10; \item &#10; \knnlm uses approximate nearest neighbor search to handle the large number of candidates, and the lack of this preciseness in this algorithm actually helps \knnlm to generalize better than using exact nearest neighbor search and distance calculation, possibly due to a regularization effect.&#10; The relative perplexity improvement from this factor is about 2.6\%.&#10; \item Depending on the design decisions that are chosen for modeling, adding a temperature term to the \knn non-parametric component can become crucial to the success of modeling (although coincidentally, in the original settings of \cite{khandelwal20generalization}, a temperature of 1.0 is close to optimal, which hid the importance of this term).&#10; In some settings, the relative perplexity gap between the default and optimal temperature can be as high as 3.7\%.&#10;\end{enumerate}&#10;&#10;Finally, one significant drawback to the current \knnlm is the inefficiency of \knn search performed at each step \cite{he2021efficient,borgeaud2022improving,alon2022neuro,Wang2022EfficientCK}.&#10;Because of the similarity between \knnlm and the parametric LM's last layers and the many design choices, we also demonstrate that we are able to make \knnlm more efficient by substituting the \knn search with another matrix operation that can fit in accelerator memory while maintaining more than half the perplexity improvement, or more than 6.5\% relative improvement compared to the base LM.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Retrieval-Augmented Models, Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Neural Network Optimization, Mathematics" />
        </attvalues>
      </node>
      <node id="2201.12431" label="2201.12431">
        <attvalues>
          <attvalue for="0" value="Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval" />
          <attvalue for="1" value="  Retrieval-based language models (R-LM) model the probability of natural&#10;language text by combining a standard language model (LM) with examples&#10;retrieved from an external datastore at test time. While effective, a major&#10;bottleneck of using these models in practice is the computationally costly&#10;datastore search, which can be performed as frequently as every time step. In&#10;this paper, we present RetoMaton - retrieval automaton - which approximates the&#10;datastore search, based on (1) saving pointers between consecutive datastore&#10;entries, and (2) clustering of entries into &quot;states&quot;. This effectively results&#10;in a weighted finite automaton built on top of the datastore, instead of&#10;representing the datastore as a flat list. The creation of the automaton is&#10;unsupervised, and a RetoMaton can be constructed from any text collection:&#10;either the original training corpus or from another domain. Traversing this&#10;automaton at inference time, in parallel to the LM inference, reduces its&#10;perplexity by up to 1.85, or alternatively saves up to 83% of the nearest&#10;neighbor searches over $k$NN-LM (Khandelwal et al., 2020) without hurting&#10;perplexity. Our code and trained models are available at&#10;https://github.com/neulab/retomaton .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2109.08449" label="2109.08449">
        <attvalues>
          <attvalue for="0" value="General Cross-Architecture Distillation of Pretrained Language Models&#10;  into Matrix Embeddings" />
          <attvalue for="1" value="  Large pretrained language models (PreLMs) are revolutionizing natural&#10;language processing across all benchmarks. However, their sheer size is&#10;prohibitive for small laboratories or for deployment on mobile devices.&#10;Approaches like pruning and distillation reduce the model size but typically&#10;retain the same model architecture. In contrast, we explore distilling PreLMs&#10;into a different, more efficient architecture, Continual Multiplication of&#10;Words (CMOW), which embeds each word as a matrix and uses matrix multiplication&#10;to encode sequences. We extend the CMOW architecture and its CMOW/CBOW-Hybrid&#10;variant with a bidirectional component for more expressive power, per-token&#10;representations for a general (task-agnostic) distillation during pretraining,&#10;and a two-sequence encoding scheme that facilitates downstream tasks on&#10;sentence pairs, such as sentence similarity and natural language inference. Our&#10;matrix-based bidirectional CMOW/CBOW-Hybrid model is competitive to DistilBERT&#10;on question similarity and recognizing textual entailment, but uses only half&#10;of the number of parameters and is three times faster in terms of inference&#10;speed. We match or exceed the scores of ELMo for all tasks of the GLUE&#10;benchmark except for the sentiment analysis task SST-2 and the linguistic&#10;acceptability task CoLA. However, compared to previous cross-architecture&#10;distillation approaches, we demonstrate a doubling of the scores on detecting&#10;linguistic acceptability. This shows that matrix-based embeddings can be used&#10;to distill large PreLM into competitive models and motivates further research&#10;in this direction.&#10;" />
          <attvalue for="2" value="&#10;Large pretrained language models~\cite{bert,T5} (PreLMs) have emerged as de-facto standard methods for natural language processing~\cite{glue,superglue}.&#10;The common strategy is to pretrain models on enormous amounts of unlabeled text before fine-tuning them for downstream tasks.&#10;However, the drawback of PreLMs is that the models are becoming larger and larger with up to several billion parameters~\cite{GPT-3}.&#10;This comes with high environmental and economic costs~\cite{DBLP:conf/acl/StrubellGM19} and puts development and research in the hands of a few global players\extended{~with rich resources} only \cite[pp. 10-12]{foundation-models}.&#10;Even though a single pretrained model can be reused for multiple downstream tasks, the sheer model size is often prohibitive.&#10;The immense resource requirements prevent the use of these models in small-scale laboratories and on mobile devices, which is tied to privacy concerns~\cite{DBLP:conf/nips/Sanh0R20}.&#10;&#10;There is a need for more efficient models or compressed versions of large models to make AI research more inclusive and energy-friendly while fostering deployment in applications.&#10;Reducing the size of PreLMs using knowledge distillation~\cite{knowledgedistillation} or model compression~\cite{modelcompression} is an active area of research~\cite{distilbert,tinybert,sun2020mobilebert}. &#10;It is reported that companies, such as Google, use distillation of PreLM to deploy their large models for productive use, \ie for services that have strong requirements in terms of low latency.&#10;Both knowledge distillation and model compression can be described as teacher-student setups~\cite{knowledgedistillation,modelcompression}.&#10;The student is trained to imitate the predictions of the teacher while using less resources.&#10;Typically, a large PreLM takes the role of the teacher while the student is a smaller version of the same architecture.&#10;Sharing the same architecture between the student and the teacher enables the use of dedicated distillation techniques, \eg aligning the representations of intermediate layers~\cite{distilbert,sun2020mobilebert}.&#10;&#10;However, using more efficient architectures as student has already shown promising results, such as the task-specific distillation approaches by Tang~\etal~\cite{tang2019distilling} and Wasserblatt~\etal~\cite{boundariesbertdistillation}.&#10;In their works, the student models are LSTMs~\cite{DBLP:journals/neco/HochreiterS97} or models based on a continuous bag-of-words representation (CBOW)~\cite{DBLP:conf/icml/CollobertW08,word2vec}.&#10;On the one hand, LSTMs are difficult to parallelize as they need at least $\mathcal{O}(n)$ sequential steps to encode a sequence of length $n$.&#10;On the other hand, CBOW-based models are not order-aware, \ie cannot distinguish sentences with the same words but in different order (``cat eats mouse'' vs. ``mouse eats cat'' are treated equivalent).&#10;There are, however, efficient models such as Mai~\etal's continual multiplication of words (CMOW) that do capture word order by representing each token as a matrix~\cite{cmow}, instead of a vector as in CBOW. &#10;A sequence in CMOW is modeled by the non-commutative matrix multiplication~\cite{DBLP:conf/acl/RudolphG10}, which makes the encoding of a sequence dependent on the word order. We denote such models as matrix embeddings.&#10;&#10;We extend Mai et al.'s work and investigate how order-aware matrix embeddings can be used as student models in cross-architecture distillation from large PreLM teachers.&#10;This complements the existing body of works that focused predominantly on same-architecture distillation (see discussion in Section~\ref{sub:kd}).&#10;Furthermore, all previous cross-architecture distillation approaches are task-specific, whereas we also explore general distillation.&#10;We aim to understand to what extent order-aware embeddings are suitable to capture the teacher signal of a large PreLM such as BERT~\cite{bert}.&#10;To this end, we extend Mai~\etal's CMOW/CBOW-Hybrid model~\cite{cmow}, which is a hybrid variant unifying the strength of CBOW and CMOW, with a bidirectional representation of the sequences.&#10;Furthermore, we add the ability to emit per-token representations to facilitate the use of a modern masked language model objective~\cite{bert}.&#10;&#10;We investigate both task-agnostic general distillation, \ie the distillation is applied during pretraining on unlabeled text, and task-specific distillation, when an already fine-tuned PreLM is distilled per task.&#10;We further introduce a two-sentence encoding scheme to CMOW so that it can deal with sentence similarity and natural language inference tasks.&#10;&#10;Our results show that large PreLMs can be distilled into efficient order-sensitive embedding models, achieving a performance that is competitive to ELMo~\cite{DBLP:conf/naacl/PetersNIGCLZ18} on the GLUE benchmark.&#10;On the QQP and RTE tasks, embedding-based models even challenge other size-reduced BERT models such as DistillBERT.&#10;In summary, our contributions are:&#10;\begin{itemize}&#10;&#10;Below, we introduce our embedding models, our cross-architecture distillation setup, and our two-sequence encoding scheme.&#10;The experimental procedure is described in Section~\ref{sec:apparatus}.&#10;The results are reported in Section~\ref{sec:results} and discussed in Section~\ref{sec:discussion}, where we also relate our work to the literature.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Efficient Model Architectures, Computer Science, Linguistics, Language Model Distillation, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2005.07683" label="2005.07683">
        <attvalues>
          <attvalue for="0" value="Movement Pruning: Adaptive Sparsity by Fine-Tuning" />
          <attvalue for="1" value="  Magnitude pruning is a widely used strategy for reducing model size in pure&#10;supervised learning; however, it is less effective in the transfer learning&#10;regime that has become standard for state-of-the-art natural language&#10;processing applications. We propose the use of movement pruning, a simple,&#10;deterministic first-order weight pruning method that is more adaptive to&#10;pretrained model fine-tuning. We give mathematical foundations to the method&#10;and compare it to existing zeroth- and first-order pruning methods. Experiments&#10;show that when pruning large pretrained language models, movement pruning shows&#10;significant improvements in high-sparsity regimes. When combined with&#10;distillation, the approach achieves minimal accuracy loss with down to only 3%&#10;of the model parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.02984" label="2004.02984">
        <attvalues>
          <attvalue for="0" value="MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices" />
          <attvalue for="1" value="  Natural Language Processing (NLP) has recently achieved great success by&#10;using huge pre-trained models with hundreds of millions of parameters. However,&#10;these models suffer from heavy model sizes and high latency such that they&#10;cannot be deployed to resource-limited mobile devices. In this paper, we&#10;propose MobileBERT for compressing and accelerating the popular BERT model.&#10;Like the original BERT, MobileBERT is task-agnostic, that is, it can be&#10;generically applied to various downstream NLP tasks via simple fine-tuning.&#10;Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with&#10;bottleneck structures and a carefully designed balance between self-attentions&#10;and feed-forward networks. To train MobileBERT, we first train a specially&#10;designed teacher model, an inverted-bottleneck incorporated BERT_LARGE model.&#10;Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical&#10;studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE&#10;while achieving competitive results on well-known benchmarks. On the natural&#10;language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6&#10;lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD&#10;v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of&#10;90.0/79.2 (1.5/2.1 higher than BERT_BASE).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.12136" label="1903.12136">
        <attvalues>
          <attvalue for="0" value="Distilling Task-Specific Knowledge from BERT into Simple Neural Networks" />
          <attvalue for="1" value="  In the natural language processing literature, neural networks are becoming&#10;increasingly deeper and complex. The recent poster child of this trend is the&#10;deep language representation model, which includes BERT, ELMo, and GPT. These&#10;developments have led to the conviction that previous-generation, shallower&#10;neural networks for language understanding are obsolete. In this paper,&#10;however, we demonstrate that rudimentary, lightweight neural networks can still&#10;be made competitive without architecture changes, external training data, or&#10;additional input features. We propose to distill knowledge from BERT, a&#10;state-of-the-art language representation model, into a single-layer BiLSTM, as&#10;well as its siamese counterpart for sentence-pair tasks. Across multiple&#10;datasets in paraphrasing, natural language inference, and sentiment&#10;classification, we achieve comparable results with ELMo, while using roughly&#10;100 times fewer parameters and 15 times less inference time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.06423" label="1902.06423">
        <attvalues>
          <attvalue for="0" value="CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix&#10;  Space Model" />
          <attvalue for="1" value="  Continuous Bag of Words (CBOW) is a powerful text embedding method. Due to&#10;its strong capabilities to encode word content, CBOW embeddings perform well on&#10;a wide range of downstream tasks while being efficient to compute. However,&#10;CBOW is not capable of capturing the word order. The reason is that the&#10;computation of CBOW's word embeddings is commutative, i.e., embeddings of XYZ&#10;and ZYX are the same. In order to address this shortcoming, we propose a&#10;learning algorithm for the Continuous Matrix Space Model, which we call&#10;Continual Multiplication of Words (CMOW). Our algorithm is an adaptation of&#10;word2vec, so that it can be trained on large quantities of unlabeled text. We&#10;empirically show that CMOW better captures linguistic properties, but it is&#10;inferior to CBOW in memorizing word content. Motivated by these findings, we&#10;propose a hybrid model that combines the strengths of CBOW and CMOW. Our&#10;results show that the hybrid CBOW-CMOW-model retains CBOW's strong ability to&#10;memorize word content while at the same time substantially improving its&#10;ability to encode other linguistic information by 8%. As a result, the hybrid&#10;also performs better on 8 out of 11 supervised downstream tasks with an average&#10;improvement of 1.2%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12150" label="2310.12150">
        <attvalues>
          <attvalue for="0" value="Understanding Retrieval Augmentation for Long-Form Question Answering" />
          <attvalue for="1" value="  We present a study of retrieval-augmented language models (LMs) on long-form&#10;question answering. We analyze how retrieval augmentation impacts different&#10;LMs, by comparing answers generated from models while using the same evidence&#10;documents, and how differing quality of retrieval document set impacts the&#10;answers generated from the same LM. We study various attributes of generated&#10;answers (e.g., fluency, length, variance) with an emphasis on the attribution&#10;of generated long-form answers to in-context evidence documents. We collect&#10;human annotations of answer attribution and evaluate methods for automatically&#10;judging attribution. Our study provides new insights on how retrieval&#10;augmentation impacts long, knowledge-rich text generation of LMs. We further&#10;identify attribution patterns for long text generation and analyze the main&#10;culprits of attribution errors. Together, our analysis reveals how retrieval&#10;augmentation impacts long knowledge-rich text generation and provide directions&#10;for future work.&#10;" />
          <attvalue for="2" value="&#10;&#10;{Long-form question answering} (LFQA) is designed to address {any} type of question that could be asked. Instead of extracting spans in the evidence document, LFQA systems generate paragraph-long, complex answers to questions by leveraging parametric knowledge in large language models (LLMs) and retrieved documents provided at inference time. In recent years, we learned surprisingly impressive yet brittle~\cite{ji2023survey, liu2023evaluating} LFQA capabilities of large-scale LLMs.&#10;&#10;Recent work~\cite{nakano2021webgpt} proposes retrieval as a powerful tool to provide up-to-date, relevant information to LMs. Yet, our understanding of how retrieval augmentation impacts generation in LMs is limited, and retrieval augmentation does not always affect LMs the way we anticipate. \cite{liu2023lost} discovered how information placed in the middle of contexts is not used by LMs and a line of work~\cite{Chen2022RichKS, longpre-etal-2021-entity} showed parametric knowledge continues to affect generation even when relevant documents are provided in-context for factoid QA task.&#10;&#10;We study how retrieval impacts answer generation for LFQA, a complex long text generation task. We present two controlled study settings (illustrated in Figure~\ref{fig:intro}): one fixing the LM and varying evidence documents and the other fixing evidence documents and varying the LMs. As evaluating the quality of LFQA is notoriously difficult~\cite{krishna-etal-2021-hurdles}, we start our analysis by measuring surface features (e.g. length, perplexity) that correlate with specific answer qualities such as coherence~\cite{xu-etal-2023-critical}. One desirable property of retrieval augmented LFQA system is whether the generated answer can be attributed to provided evidence documents. To evaluate this, we newly collect human annotations on sentence-level attribution~\cite{rashkin2021measuring} and evaluate off-the-shelf models for detecting attributions~\cite{schuster-etal-2021-get} on our collected dataset (Section~\ref{sec:nli}). &#10;&#10;Our analysis on surface patterns reveals that retrieval augmentation changes LM's generation substantially. Some effects, e.g., change in the length of generated answers, are pronounced even when provided documents are not relevant. Relevant in-context evidence documents lead to more substantial changes, leading LMs to generate more unexpected sentences (measured by higher perplexity), while irrelevant document does not have the same effects. The impact of retrieval augmentation, even with the same set of evidence documents, can result in opposite effects for different base LMs.&#10;&#10;We provide an in-depth analysis of attribution with our newly annotated dataset, which can serve as a benchmark for evaluating attribution. We observe NLI models that performed well in detecting attribution in factoid QA~\cite{bohnet2022attributed} perform competitively in LFQA setting as well, significantly outperforming chance, yet fall behind human agreement by 15\% in accuracy. &#10;Our study reveals that attribution quality varies significantly across base LMs, even when they are provided with the same set of documents. &#10;&#10;We provide new insights on attribution patterns for long text generation. The last generated sentence is substantially less attributable than earlier sentences, and the generated text has a tendency to follow the order of the in-context evidence documents, even when the in-context document is a concatenation of multiple documents. Taken together, our analysis improves our understanding of how LMs use in-context evidence documents for long-form question answering and suggests concrete directions for future work. " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Text Generation Evaluation, Artificial Intelligence, Natural Language Processing, Retrieval Augmentation Impact" />
        </attvalues>
      </node>
      <node id="2307.03172" label="2307.03172">
        <attvalues>
          <attvalue for="0" value="Lost in the Middle: How Language Models Use Long Contexts" />
          <attvalue for="1" value="  While recent language models have the ability to take long contexts as input,&#10;relatively little is known about how well they use longer context. We analyze&#10;the performance of language models on two tasks that require identifying&#10;relevant information in their input contexts: multi-document question answering&#10;and key-value retrieval. We find that performance can degrade significantly&#10;when changing the position of relevant information, indicating that current&#10;language models do not robustly make use of information in long input contexts.&#10;In particular, we observe that performance is often highest when relevant&#10;information occurs at the beginning or end of the input context, and&#10;significantly degrades when models must access relevant information in the&#10;middle of long contexts, even for explicitly long-context models. Our analysis&#10;provides a better understanding of how language models use their input context&#10;and provides new evaluation protocols for future long-context language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2210.13701" label="2210.13701">
        <attvalues>
          <attvalue for="0" value="Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating&#10;  Models to Reflect Conflicting Evidence" />
          <attvalue for="1" value="  Question answering models can use rich knowledge sources -- up to one hundred&#10;retrieved passages and parametric knowledge in the large-scale language model&#10;(LM). Prior work assumes information in such knowledge sources is consistent&#10;with each other, paying little attention to how models blend information stored&#10;in their LM parameters with that from retrieved evidence documents. In this&#10;paper, we simulate knowledge conflicts (i.e., where parametric knowledge&#10;suggests one answer and different passages suggest different answers) and&#10;examine model behaviors. We find retrieval performance heavily impacts which&#10;sources models rely on, and current models mostly rely on non-parametric&#10;knowledge in their best-performing settings. We discover a troubling trend that&#10;contradictions among knowledge sources affect model confidence only marginally.&#10;To address this issue, we present a new calibration study, where models are&#10;discouraged from presenting any single answer when presented with multiple&#10;conflicting answer candidates in retrieved evidences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.06332" label="2103.06332">
        <attvalues>
          <attvalue for="0" value="Hurdles to Progress in Long-form Question Answering" />
          <attvalue for="1" value="  The task of long-form question answering (LFQA) involves retrieving documents&#10;relevant to a given question and using them to generate a paragraph-length&#10;answer. While many models have recently been proposed for LFQA, we show in this&#10;paper that the task formulation raises fundamental challenges regarding&#10;evaluation and dataset creation that currently preclude meaningful modeling&#10;progress. To demonstrate these challenges, we first design a new system that&#10;relies on sparse attention and contrastive retriever learning to achieve&#10;state-of-the-art performance on the ELI5 LFQA dataset. While our system tops&#10;the public leaderboard, a detailed analysis reveals several troubling trends:&#10;(1) our system's generated answers are not actually grounded in the documents&#10;that it retrieves; (2) ELI5 contains significant train / validation overlap, as&#10;at least 81% of ELI5 validation questions occur in paraphrased form in the&#10;training set; (3) ROUGE-L is not an informative metric of generated answer&#10;quality and can be easily gamed; and (4) human evaluations used for other text&#10;generation tasks are unreliable for LFQA. We offer suggestions to mitigate each&#10;of these issues, which we hope will lead to more rigorous LFQA research and&#10;meaningful progress in the future.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18201" label="2305.18201">
        <attvalues>
          <attvalue for="0" value="A Critical Evaluation of Evaluations for Long-form Question Answering" />
          <attvalue for="1" value="  Long-form question answering (LFQA) enables answering a wide range of&#10;questions, but its flexibility poses enormous challenges for evaluation. We&#10;perform the first targeted study of the evaluation of long-form answers,&#10;covering both human and automatic evaluation practices. We hire domain experts&#10;in seven areas to provide preference judgments over pairs of answers, along&#10;with free-form justifications for their choices. We present a careful analysis&#10;of experts' evaluation, which focuses on new aspects such as the&#10;comprehensiveness of the answer. Next, we examine automatic text generation&#10;metrics, finding that no existing metrics are predictive of human preference&#10;judgments. However, some metrics correlate with fine-grained aspects of answers&#10;(e.g., coherence). We encourage future work to move away from a single &quot;overall&#10;score&quot; of the answer and adopt a multi-faceted evaluation, targeting aspects&#10;such as factuality and completeness. We publicly release all of our annotations&#10;and code to spur future work into LFQA evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.08541" label="2103.08541">
        <attvalues>
          <attvalue for="0" value="Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence" />
          <attvalue for="1" value="  Typical fact verification models use retrieved written evidence to verify&#10;claims. Evidence sources, however, often change over time as more information&#10;is gathered and revised. In order to adapt, models must be sensitive to subtle&#10;differences in supporting evidence. We present VitaminC, a benchmark infused&#10;with challenging cases that require fact verification models to discern and&#10;adjust to slight factual changes. We collect over 100,000 Wikipedia revisions&#10;that modify an underlying fact, and leverage these revisions, together with&#10;additional synthetically constructed ones, to create a total of over 400,000&#10;claim-evidence pairs. Unlike previous resources, the examples in VitaminC are&#10;contrastive, i.e., they contain evidence pairs that are nearly identical in&#10;language and content, with the exception that one supports a given claim while&#10;the other does not. We show that training using this design increases&#10;robustness -- improving accuracy by 10% on adversarial fact verification and 6%&#10;on adversarial natural language inference (NLI). Moreover, the structure of&#10;VitaminC leads us to define additional tasks for fact-checking resources:&#10;tagging relevant words in the evidence for verifying the claim, identifying&#10;factual revisions, and providing automatic edits via factually consistent text&#10;generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12182" label="2305.12182">
        <attvalues>
          <attvalue for="0" value="Glot500: Scaling Multilingual Corpora and Language Models to 500&#10;  Languages" />
          <attvalue for="1" value="  The NLP community has mainly focused on scaling Large Language Models (LLMs)&#10;vertically, i.e., making them better for about 100 languages. We instead scale&#10;LLMs horizontally: we create, through continued pretraining, Glot500-m, an LLM&#10;that covers 511 predominantly low-resource languages. An important part of this&#10;effort is to collect and clean Glot500-c, a corpus that covers these 511&#10;languages and allows us to train Glot500-m. We evaluate Glot500-m on five&#10;diverse tasks across these languages. We observe large improvements for both&#10;high-resource and low-resource languages compared to an XLM-R baseline. Our&#10;analysis shows that no single factor explains the quality of multilingual LLM&#10;representations. Rather, a combination of factors determines quality including&#10;corpus size, script, &quot;help&quot; from related languages and the total capacity of&#10;the model. Our work addresses an important goal of NLP research: we should not&#10;limit NLP to a small fraction of the world's languages and instead strive to&#10;support as many languages as possible to bring the benefits of NLP technology&#10;to all languages and cultures. Code, data and models are available at&#10;https://github.com/cisnlp/Glot500.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Training&#10;multilingual LLMs using the masked&#10;language modeling (MLM) objective is effective&#10;to achieve cross-lingual representations&#10;\cite{devlin-etal-2019-bert, conneau-etal-2020-unsupervised}.&#10;These models can be &#10;further improved by incorporating techniques such as discriminative pre-training&#10;\cite{chi-etal-2022-xlm} and the use of parallel data \cite{yang2020alternating, chi-etal-2021-infoxlm}. &#10;However, this primarily benefits a limited set of &#10;languages with large corpora.&#10;&#10;Recent research has attempted to extend existing LLMs to &#10;languages with limited resources. \cite{wang-etal-2019-improving} &#10;propose vocabulary extension; \cite{ebrahimi-kann-2021-adapt} investigate &#10;adaptation methods, including MLM and &#10;Translation Language Model (TLM) objectives and adapters;&#10;\cite{alabi-etal-2022-adapting} adapt XLM-R to &#10;17 African languages;&#10;\cite{wang-etal-2022-expanding} expand language models to &#10;low-resource languages using bilingual lexicons. &#10;&#10;Alternatively,&#10;parameter-efficient fine-tuning adapts pre-trained models to new languages &#10;by training a small set of weights effectively &#10;\cite{zhao-etal-2020-masking,pfeiffer-etal-2021-unks,ansell-etal-2022-composable}. &#10;\cite{pfeiffer-etal-2022-lifting} address the ``curse of multilinguality''&#10;by sharing a part of the model among all languages and &#10;having separate modules for each language.&#10;We show that &#10;the common &#10;perception that multilinguality increases as we add more&#10;languages, until, from some point,&#10;it starts decreasing,&#10;is naive. The amount of available data per language and the similarity between&#10;languages also play important roles (\ref{supportthroughrelatedlanguages}).&#10;&#10;Another approach trains LLMs from scratch for a&#10;limited number of &#10;\unseenlanguages; e.g.,&#10;AfriBERTa \cite{ogueji-etal-2021-small}&#10;and IndicNLPSuite \cite{kakwani-etal-2020-indicnlpsuite}&#10;are LLMs&#10;for 11 African &#10;languages and 11 Indic languages.&#10;In concurrent work, \cite{adebara2022serengeti} train a&#10;multilingual model for 517 African languages on a 42&#10;GB corpus, but without making the model available and&#10;with an evaluation on a smaller number of languages than ours.&#10;&#10;Closely related to our work on corpus creation,&#10;\cite{bapna2022building} and \cite{costa2022no}&#10;also create NLP resources for a large number of \unseenlanguages.&#10;They&#10;train a language identifier model and extract textual data for \unseenlanguages&#10;from large-scale web crawls. This approach is effective, &#10;but it requires significant computational resources and native speakers for &#10;all \unseenlanguages. This is hard to do outside of large corporations. &#10;\cite{bapna2022building} have not made their data available.&#10;\cite{costa2022no} have only &#10;released a portion of their data in around 200 languages.&#10;&#10;A key benefit of ``horizontally'' scaled multilingual LLMs&#10;is transfer from high- to low-resource&#10;languages. Our evaluation suggests that \modelname excels at&#10;this, but this is not the main focus of our paper. There is&#10;a large body of work on crosslingual transfer:&#10;\cite{artetxe-schwenk-2019-massively,imanigooghari-etal-2022-graph, lauscher-etal-2020-zero,conneau-etal-2020-unsupervised,DBLP:journals/corr/abs-2106-16171,&#10;DBLP:journals/jmlr/FanBSMEGBCWCGBL21, severini2022towards,&#10;choenni-shutova-2022-investigating,DBLP:journals/corr/abs-2305-00090},&#10;inter alia.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Low-Resource Language Support, Large-Scale NLP Development, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2307.15936" label="2307.15936">
        <attvalues>
          <attvalue for="0" value="A Theory for Emergence of Complex Skills in Language Models" />
          <attvalue for="1" value="  A major driver of AI products today is the fact that new skills emerge in&#10;language models when their parameter set and training corpora are scaled up.&#10;This phenomenon is poorly understood, and a mechanistic explanation via&#10;mathematical analysis of gradient-based training seems difficult. The current&#10;paper takes a different approach, analysing emergence using the famous (and&#10;empirical) Scaling Laws of LLMs and a simple statistical framework.&#10;Contributions include: (a) A statistical framework that relates cross-entropy&#10;loss of LLMs to competence on the basic skills that underlie language tasks.&#10;(b) Mathematical analysis showing that the Scaling Laws imply a strong form of&#10;inductive bias that allows the pre-trained model to learn very efficiently. We&#10;informally call this {\em slingshot generalization} since naively viewed it&#10;appears to give competence levels at skills that violate usual generalization&#10;theory. (c) A key example of slingshot generalization, that competence at&#10;executing tasks involving $k$-tuples of skills emerges essentially at the same&#10;scaling and same rate as competence on the elementary skills themselves.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.04351" label="2404.04351">
        <attvalues>
          <attvalue for="0" value="Assisting humans in complex comparisons: automated information&#10;  comparison at scale" />
          <attvalue for="1" value="  Generative Large Language Models enable efficient analytics across knowledge&#10;domains, rivalling human experts in information comparisons. However, the&#10;applications of LLMs for information comparisons face scalability challenges&#10;due to the difficulties in maintaining information across large contexts and&#10;overcoming model token limitations. To address these challenges, we developed&#10;the novel Abstractive Summarization &amp; Criteria-driven Comparison Endpoint&#10;(ASC$^2$End) system to automate information comparison at scale. Our system&#10;employs Semantic Text Similarity comparisons for generating evidence-supported&#10;analyses. We utilize proven data-handling strategies such as abstractive&#10;summarization and retrieval augmented generation to overcome token limitations&#10;and retain relevant information during model inference. Prompts were designed&#10;using zero-shot strategies to contextualize information for improved model&#10;reasoning. We evaluated abstractive summarization using ROUGE scoring and&#10;assessed the generated comparison quality using survey responses. Models&#10;evaluated on the ASC$^2$End system show desirable results providing insights on&#10;the expected performance of the system. ASC$^2$End is a novel system and tool&#10;that enables accurate, automated information comparison at scale across&#10;knowledge domains, overcoming limitations in context length and retrieval.&#10;" />
          <attvalue for="2" value="&#10;&#10;The applications of generative Large Language Models (LLMs) across knowledge domains, including financial services, are rapidly expanding due to their proven ability to perform multiple tasks such as abstractive summarization \cite{S.Liu_2023}, simple QA (Question-Answering) \cite{Trivedi_2022}, multiple choice QA \cite{Trivedi_2022}, financial sentiment analysis \cite{Fei_2023}, Retrieval Augmented Generation (RAG) \cite{Pan_2022}, among others \cite{Lu_2021}. These open-domain tasks have recently been popularized in specific applications, enabling time efficiencies in the analytics required for informed decision-making. &#10;&#10;This study defines generative LLMs as exclusively autoregressive models. Recently, generative LLMs have proven to be effective in executing abstractive summarization \cite{Rath_2023}, outperforming existing state-of-the-art (SOTA) models. \cite{S.Liu_2023} presented the concept of abstractive summarization through semantic splitting instead of the traditional token length split, which may present better-contextualized summaries. Similarly, \cite{dixit_2023} proposes training LLMs on labelled datasets to increase information retention during inference.&#10;&#10;Fine-tuning generative LLMs increases a model's overall performance in a specific domain \cite{Brown_2020}. However, prompting strategies leverage the performance capabilities of naive generative LLMs without relying on large, domain-specific training data sets, with some minor trade-offs in performance \cite{Navarro_2022}\cite{Howell_2023}. Zero-shot \cite{Kojima_2022} and few-shot \cite{Wei_2022} prompting techniques can be leveraged to direct answer structures and elicit human-like reasoning for response generation. A recent study demonstrated the ability of naive generative LLMs to perform Named Entity Recognition (NER) and Relation Extraction (RE) \cite{Li_2023} for tasks that reflect human-like reasoning. Chain-of-thought reasoning can also be elicited in zero-shot prompting strategies using phrases such as “let’s think step-by-step” \cite{Kojima_2022} to reflect human-like reasoning. Additionally, altering the available input context and the order in which information is presented in a prompt \cite{Lu_2021} can affect the final output response structure. &#10;&#10;Semantic textual similarity (STS) is the main driving factor in effective information comparison \cite{Majumder_2016}. STS is measured through many different techniques that can help in text classification and topic extraction \cite{Slimani_2013} without the limitations of lexical similarity. Semantic similarity is achievable with generative LLMs as demonstrated by \cite{Gatto_2023}. They achieved success in assessing semantic similarity by testing various prompting strategies on generative LLMs. Moreover, using retrieval augmented generation (RAG) provides additional context based on a user query such that an LLM can generate an in-domain output response \cite{Ram_2023} \cite{Trivedi_2022}. RAG applies STS to find the top-k passages most similar to the user query and enhances the information in its output response. \cite{gao_2024} outlines the current advances in RAG techniques and the categorization of similar technologies. Modular components of RAG pipelines are classified as advanced RAG techniques where pre and post-retrieval data processing is applied to better augment the output of RAG.&#10;&#10;Recent research has demonstrated that decision-making and data analysis can substantially benefit from text analyses made available by STS through generative LLMs. For example, research in the medical domain has developed a framework to screen abstracts of scientific papers to be used for review papers \cite{Guo_2023}. \cite{Guo_2023}'s work used generative LLMs to compare scientific abstracts with user-defined criteria and sorted the abstracts based on the eligibility generated by the model. Their work provided insights into using generative LLMs to perform information comparisons against user-defined criteria such that the model could make binary categorical decisions. Similar research was also conducted in the financial domain, where a corporate sustainability report was compared to a sustainability guideline document in their framework, chatReport \cite{Ni_2023}. chatReport supported different QA tasks regarding the information in the sustainability report, powered by RAG. Their framework handles individual reports and generates responses for single-use applications.&#10;&#10;However, applying LLMs for information comparison is currently non-trivial at scale due to token limitations imposed on many LLMs \cite{sun2023}. Minimizing information loss and prompting under token limits are ongoing issues that must be addressed to expand system functionality \cite{jaiswal2023}. Models with longer token limits are prone to losing information from the input context, for example, due to limitations in relevant information retrieval, from the middle of long input contexts \cite{N.Liu_2023}. \cite{N.Liu_2023} found that naive, untrained models with no input context performed better on the same QA task when compared to the models provided with a lengthy input context. Given these challenges, new strategies must be explored to enhance effective and efficient information retrieval while overcoming token limitations.&#10;&#10;This study aims to develop an advanced LLM framework that improves text comparison accuracy and scalability by addressing token limitations and contextual information loss while minimizing computational resources. We propose ASC$^2$End (Abstractive Summary \&amp; Criteria-driven Comparison Endpoint), a novel system enabling accurate, automated information comparison at scale for applications across financial services and other knowledge domains. Through abstractive summarization, RAG and prompt engineering, ASC$^2$End introduces a pre-retrieval RAG workflow that enables efficient, large-scale information comparison across knowledge domains without extensive domain-specific knowledge. Existing research in pre-retrieval RAG processes is limited by the complexity and volume of text provided to its process \cite{zheng_2024}\cite{ma_2023} and abstraction research exhibits dependencies in extensive model finetuning \cite{dixit_2023}. ASC$^2$End robustly handles complex RAG tasks and eliminates the need for model fine-tuning. We demonstrate the applicability of our system to the challenge of identification and evaluation of financial transactions against complex, user-defined sustainable finance criteria. We quantify our results by evaluating the performance of prominent LLMs using ROUGE and survey responses, then discuss design choices, and the significance of ASC$^2$End in the finance domain.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Information Comparison, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2212.10509" label="2212.10509">
        <attvalues>
          <attvalue for="0" value="Interleaving Retrieval with Chain-of-Thought Reasoning for&#10;  Knowledge-Intensive Multi-Step Questions" />
          <attvalue for="1" value="  Prompting-based large language models (LLMs) are surprisingly powerful at&#10;generating natural language reasoning steps or Chains-of-Thoughts (CoT) for&#10;multi-step question answering (QA). They struggle, however, when the necessary&#10;knowledge is either unavailable to the LLM or not up-to-date within its&#10;parameters. While using the question to retrieve relevant text from an external&#10;knowledge source helps LLMs, we observe that this one-step retrieve-and-read&#10;approach is insufficient for multi-step QA. Here, \textit{what to retrieve}&#10;depends on \textit{what has already been derived}, which in turn may depend on&#10;\textit{what was previously retrieved}. To address this, we propose IRCoT, a&#10;new approach for multi-step QA that interleaves retrieval with steps&#10;(sentences) in a CoT, guiding the retrieval with CoT and in turn using&#10;retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves&#10;retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four&#10;datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar&#10;substantial gains in out-of-distribution (OOD) settings as well as with much&#10;smaller models such as Flan-T5-large without additional training. IRCoT reduces&#10;model hallucination, resulting in factually more accurate CoT reasoning. Code,&#10;data, and prompts are available at \url{https://github.com/stonybrooknlp/ircot}&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.16714" label="2203.16714">
        <attvalues>
          <attvalue for="0" value="End-to-End Table Question Answering via Retrieval-Augmented Generation" />
          <attvalue for="1" value="  Most existing end-to-end Table Question Answering (Table QA) models consist&#10;of a two-stage framework with a retriever to select relevant table candidates&#10;from a corpus and a reader to locate the correct answers from table candidates.&#10;Even though the accuracy of the reader models is significantly improved with&#10;the recent transformer-based approaches, the overall performance of such&#10;frameworks still suffers from the poor accuracy of using traditional&#10;information retrieval techniques as retrievers. To alleviate this problem, we&#10;introduce T-RAG, an end-to-end Table QA model, where a non-parametric dense&#10;vector index is fine-tuned jointly with BART, a parametric sequence-to-sequence&#10;model to generate answer tokens. Given any natural language question, T-RAG&#10;utilizes a unified pipeline to automatically search through a table corpus to&#10;directly locate the correct answer from the table cells. We apply T-RAG to&#10;recent open-domain Table QA benchmarks and demonstrate that the fine-tuned&#10;T-RAG model is able to achieve state-of-the-art performance in both the&#10;end-to-end Table QA and the table retrieval tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1310.8059" label="1310.8059">
        <attvalues>
          <attvalue for="0" value="Description and Evaluation of Semantic Similarity Measures Approaches" />
          <attvalue for="1" value="  In recent years, semantic similarity measure has a great interest in Semantic&#10;Web and Natural Language Processing (NLP). Several similarity measures have&#10;been developed, being given the existence of a structured knowledge&#10;representation offered by ontologies and corpus which enable semantic&#10;interpretation of terms. Semantic similarity measures compute the similarity&#10;between concepts/terms included in knowledge sources in order to perform&#10;estimations. This paper discusses the existing semantic similarity methods&#10;based on structure, information content and feature approaches. Additionally,&#10;we present a critical evaluation of several categories of semantic similarity&#10;approaches based on two standard benchmarks. The aim of this paper is to give&#10;an efficient evaluation of all these measures which help researcher and&#10;practitioners to select the measure that best fit for their requirements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.06541" label="2309.06541">
        <attvalues>
          <attvalue for="0" value="Text Encoders Lack Knowledge: Leveraging Generative LLMs for&#10;  Domain-Specific Semantic Textual Similarity" />
          <attvalue for="1" value="  Amidst the sharp rise in the evaluation of large language models (LLMs) on&#10;various tasks, we find that semantic textual similarity (STS) has been&#10;under-explored. In this study, we show that STS can be cast as a text&#10;generation problem while maintaining strong performance on multiple STS&#10;benchmarks. Additionally, we show generative LLMs significantly outperform&#10;existing encoder-based STS models when characterizing the semantic similarity&#10;between two texts with complex semantic relationships dependent on world&#10;knowledge. We validate this claim by evaluating both generative LLMs and&#10;existing encoder-based STS models on three newly collected STS challenge sets&#10;which require world knowledge in the domains of Health, Politics, and Sports.&#10;All newly collected data is sourced from social media content posted after May&#10;2023 to ensure the performance of closed-source models like ChatGPT cannot be&#10;credited to memorization. Our results show that, on average, generative LLMs&#10;outperform the best encoder-only baselines by an average of 22.3% on STS tasks&#10;requiring world knowledge. Our results suggest generative language models with&#10;STS-specific prompting strategies achieve state-of-the-art performance in&#10;complex, domain-specific STS tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2312.10997" label="2312.10997">
        <attvalues>
          <attvalue for="0" value="Retrieval-Augmented Generation for Large Language Models: A Survey" />
          <attvalue for="1" value="  Large Language Models (LLMs) showcase impressive capabilities but encounter&#10;challenges like hallucination, outdated knowledge, and non-transparent,&#10;untraceable reasoning processes. Retrieval-Augmented Generation (RAG) has&#10;emerged as a promising solution by incorporating knowledge from external&#10;databases. This enhances the accuracy and credibility of the generation,&#10;particularly for knowledge-intensive tasks, and allows for continuous knowledge&#10;updates and integration of domain-specific information. RAG synergistically&#10;merges LLMs' intrinsic knowledge with the vast, dynamic repositories of&#10;external databases. This comprehensive review paper offers a detailed&#10;examination of the progression of RAG paradigms, encompassing the Naive RAG,&#10;the Advanced RAG, and the Modular RAG. It meticulously scrutinizes the&#10;tripartite foundation of RAG frameworks, which includes the retrieval, the&#10;generation and the augmentation techniques. The paper highlights the&#10;state-of-the-art technologies embedded in each of these critical components,&#10;providing a profound understanding of the advancements in RAG systems.&#10;Furthermore, this paper introduces up-to-date evaluation framework and&#10;benchmark. At the end, this article delineates the challenges currently faced&#10;and points out prospective avenues for research and development.&#10;" />
          <attvalue for="2" value="&#10;&#10;L{arge} language models (LLMs) have achieved remarkable success, though they still face significant limitations, especially in domain-specific or knowledge-intensive tasks~\cite{longtail}, notably producing ``hallucinations&quot;~\cite{hallucination} when handling queries beyond their training data or requiring current information. To overcome challenges, Retrieval-Augmented Generation (RAG) enhances LLMs by retrieving relevant document chunks from external knowledge base through semantic similarity calculation. By referencing external knowledge, RAG effectively reduces the problem of generating factually incorrect content. Its integration into LLMs has resulted in widespread adoption, establishing RAG as a key technology in advancing chatbots and enhancing the suitability of LLMs for real-world applications.&#10;&#10;RAG technology has rapidly developed in recent years, and the technology tree summarizing related research is shown in Figure~\ref{fig:rag_tech_tree}. The development trajectory of RAG in the era of large models exhibits several distinct stage characteristics. Initially, RAG's inception coincided with the rise of the Transformer architecture, focusing on enhancing language models by incorporating additional knowledge through Pre-Training Models (PTM). This early stage was characterized by foundational work aimed at refining pre-training techniques\cite{REALM,RAG,Retro}.The subsequent arrival of ChatGPT~\cite{chatgpt} marked a pivotal moment, with LLM demonstrating powerful in context learning (ICL) capabilities. RAG research shifted towards providing better information for LLMs to answer more complex and knowledge-intensive tasks during the inference stage, leading to rapid development in RAG studies. As research progressed, the enhancement of RAG was no longer limited to the inference stage but began to incorporate more with LLM fine-tuning techniques.&#10;&#10;The burgeoning field of RAG has experienced swift growth, yet it has not been accompanied by a systematic synthesis that could clarify its broader trajectory. This survey endeavors to fill this gap by mapping out the RAG process and charting its evolution and anticipated future paths, with a focus on the integration of RAG within LLMs. This paper considers both technical paradigms and research methods, summarizing three main research paradigms from over 100 RAG studies, and analyzing key technologies in the core stages of ``Retrieval,&quot; ``Generation,&quot; and ``Augmentation.&quot; On the other hand, current research tends to focus more on methods, lacking analysis and summarization of how to evaluate RAG. This paper comprehensively reviews the downstream tasks, datasets, benchmarks, and evaluation methods applicable to RAG. Overall, this paper sets out to meticulously compile and categorize the foundational technical concepts, historical progression, and the spectrum of RAG methodologies and applications that have emerged post-LLMs. It is designed to equip readers and professionals with a detailed and structured understanding of both large models and RAG. It aims to illuminate the evolution of retrieval augmentation techniques, assess the strengths and weaknesses of various approaches in their respective contexts, and speculate on upcoming trends and innovations.&#10;&#10;Our contributions are as follows:&#10;\begin{itemize}&#10;\item In this survey, we present a thorough and systematic review of the state-of-the-art RAG methods, delineating its evolution through paradigms including naive RAG, advanced RAG, and modular RAG. This review contextualizes the broader scope of RAG research within the landscape of LLMs.&#10;&#10;\item We identify and discuss the central technologies integral to the RAG process, specifically focusing on the aspects of ``Retrieval&quot;, ``Generation'' and ``Augmentation&quot;, and delve into their synergies, elucidating how these components intricately collaborate to form a cohesive and effective RAG framework.&#10;&#10;\item We have summarized the current assessment methods of RAG, covering 26 tasks, nearly 50 datasets, outlining the evaluation objectives and metrics, as well as the current evaluation benchmarks and tools. Additionally, we anticipate future directions for RAG, emphasizing potential enhancements to tackle current challenges.&#10;&#10;\end{itemize}&#10;&#10;The paper unfolds as follows: Section~\ref{sec:overview} introduces the main concept and current paradigms of RAG. The following three sections explore core components—``Retrieval'', ``Generation&quot; and ``Augmentation&quot;, respectively.&#10;Section~\ref{sec:retrieval} focuses on optimization methods in retrieval,including indexing, query and embedding optimization.&#10;Section~\ref{sec:generation} concentrates on post-retrieval process and LLM fine-tuning in generation.&#10;Section~\ref{sec:augmentation} analyzes the three augmentation processes.&#10;Section~\ref{sec:evaluation} focuses on RAG's downstream tasks and evaluation system. Section~\ref{sec:prospects} mainly discusses the challenges that RAG currently faces and its future development directions. At last, the paper concludes in Section~\ref{sec:conclusion}.&#10; " />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.15770" label="2307.15770">
        <attvalues>
          <attvalue for="0" value="CHATREPORT: Democratizing Sustainability Disclosure Analysis through&#10;  LLM-based Tools" />
          <attvalue for="1" value="  In the face of climate change, are companies really taking substantial steps&#10;toward more sustainable operations? A comprehensive answer lies in the dense,&#10;information-rich landscape of corporate sustainability reports. However, the&#10;sheer volume and complexity of these reports make human analysis very costly.&#10;Therefore, only a few entities worldwide have the resources to analyze these&#10;reports at scale, which leads to a lack of transparency in sustainability&#10;reporting. Empowering stakeholders with LLM-based automatic analysis tools can&#10;be a promising way to democratize sustainability report analysis. However,&#10;developing such tools is challenging due to (1) the hallucination of LLMs and&#10;(2) the inefficiency of bringing domain experts into the AI development loop.&#10;In this paper, we ChatReport, a novel LLM-based system to automate the analysis&#10;of corporate sustainability reports, addressing existing challenges by (1)&#10;making the answers traceable to reduce the harm of hallucination and (2)&#10;actively involving domain experts in the development loop. We make our&#10;methodology, annotated datasets, and generated analyses of 1015 reports&#10;publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.20558" label="2310.20558">
        <attvalues>
          <attvalue for="0" value="Breaking the Token Barrier: Chunking and Convolution for Efficient Long&#10;  Text Classification with BERT" />
          <attvalue for="1" value="  Transformer-based models, specifically BERT, have propelled research in&#10;various NLP tasks. However, these models are limited to a maximum token limit&#10;of 512 tokens. Consequently, this makes it non-trivial to apply it in a&#10;practical setting with long input. Various complex methods have claimed to&#10;overcome this limit, but recent research questions the efficacy of these models&#10;across different classification tasks. These complex architectures evaluated on&#10;carefully curated long datasets perform at par or worse than simple baselines.&#10;In this work, we propose a relatively simple extension to vanilla BERT&#10;architecture called ChunkBERT that allows finetuning of any pretrained models&#10;to perform inference on arbitrarily long text. The proposed method is based on&#10;chunking token representations and CNN layers, making it compatible with any&#10;pre-trained BERT. We evaluate chunkBERT exclusively on a benchmark for&#10;comparing long-text classification models across a variety of tasks (including&#10;binary classification, multi-class classification, and multi-label&#10;classification). A BERT model finetuned using the ChunkBERT method performs&#10;consistently across long samples in the benchmark while utilizing only a&#10;fraction (6.25\%) of the original memory footprint. These findings suggest that&#10;efficient finetuning and inference can be achieved through simple modifications&#10;to pre-trained BERT models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.06117" label="2310.06117">
        <attvalues>
          <attvalue for="0" value="Take a Step Back: Evoking Reasoning via Abstraction in Large Language&#10;  Models" />
          <attvalue for="1" value="  We present Step-Back Prompting, a simple prompting technique that enables&#10;LLMs to do abstractions to derive high-level concepts and first principles from&#10;instances containing specific details. Using the concepts and principles to&#10;guide reasoning, LLMs significantly improve their abilities in following a&#10;correct reasoning path towards the solution. We conduct experiments of&#10;Step-Back Prompting with PaLM-2L, GPT-4 and Llama2-70B models, and observe&#10;substantial performance gains on various challenging reasoning-intensive tasks&#10;including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, Step-Back&#10;Prompting improves PaLM-2L performance on MMLU (Physics and Chemistry) by 7%&#10;and 11% respectively, TimeQA by 27%, and MuSiQue by 7%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14283" label="2305.14283">
        <attvalues>
          <attvalue for="0" value="Query Rewriting for Retrieval-Augmented Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) play powerful, black-box readers in the&#10;retrieve-then-read pipeline, making remarkable progress in knowledge-intensive&#10;tasks. This work introduces a new framework, Rewrite-Retrieve-Read instead of&#10;the previous retrieve-then-read for the retrieval-augmented LLMs from the&#10;perspective of the query rewriting. Unlike prior studies focusing on adapting&#10;either the retriever or the reader, our approach pays attention to the&#10;adaptation of the search query itself, for there is inevitably a gap between&#10;the input text and the needed knowledge in retrieval. We first prompt an LLM to&#10;generate the query, then use a web search engine to retrieve contexts.&#10;Furthermore, to better align the query to the frozen modules, we propose a&#10;trainable scheme for our pipeline. A small language model is adopted as a&#10;trainable rewriter to cater to the black-box LLM reader. The rewriter is&#10;trained using the feedback of the LLM reader by reinforcement learning.&#10;Evaluation is conducted on downstream tasks, open-domain QA and multiple-choice&#10;QA. Experiments results show consistent performance improvement, indicating&#10;that our framework is proven effective and scalable, and brings a new framework&#10;for retrieval-augmented LLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.07355" label="1907.07355">
        <attvalues>
          <attvalue for="0" value="Probing Neural Network Comprehension of Natural Language Arguments" />
          <attvalue for="1" value="  We are surprised to find that BERT's peak performance of 77% on the Argument&#10;Reasoning Comprehension Task reaches just three points below the average&#10;untrained human baseline. However, we show that this result is entirely&#10;accounted for by exploitation of spurious statistical cues in the dataset. We&#10;analyze the nature of these cues and demonstrate that a range of models all&#10;exploit them. This analysis informs the construction of an adversarial dataset&#10;on which all models achieve random accuracy. Our adversarial dataset provides a&#10;more robust assessment of argument comprehension and should be adopted as the&#10;standard in future work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.01641" label="2005.01641">
        <attvalues>
          <attvalue for="0" value="A Tale of a Probe and a Parser" />
          <attvalue for="1" value="  Measuring what linguistic information is encoded in neural models of language&#10;has become popular in NLP. Researchers approach this enterprise by training&#10;&quot;probes&quot; - supervised models designed to extract linguistic structure from&#10;another model's output. One such probe is the structural probe (Hewitt and&#10;Manning, 2019), designed to quantify the extent to which syntactic information&#10;is encoded in contextualised word representations. The structural probe has a&#10;novel design, unattested in the parsing literature, the precise benefit of&#10;which is not immediately obvious. To explore whether syntactic probes would do&#10;better to make use of existing techniques, we compare the structural probe to a&#10;more traditional parser with an identical lightweight parameterisation. The&#10;parser outperforms structural probe on UUAS in seven of nine analysed&#10;languages, often by a substantial amount (e.g. by 11.1 points in English).&#10;Under a second less common metric, however, there is the opposite trend - the&#10;structural probe outperforms the parser. This begs the question: which metric&#10;should we prefer?&#10;" />
          <attvalue for="2" value="&#10;Recently, unsupervised sentence encoders such as ELMo \cite{peters-etal-2018-deep} and $\BERT$ \cite{devlin-etal-2019-bert} have become popular within NLP. &#10;These pre-trained models boast impressive &#10;performance &#10;when used in &#10;many language-related tasks, but &#10;this gain &#10;has come at the cost of interpretability. A natural question to ask, then, is whether these models encode the traditional linguistic structures one might expect, such as part-of-speech tags or dependency trees. &#10;To this end, researchers have invested in the design of diagnostic tools commonly referred to as \defn{probes} \cite{alain2016understanding, conneau-etal-2018-cram, hupkes2018visualisation, poliak-etal-2018-collecting, marvin-linzen-2018-targeted, niven-kao-2019-probing}. Probes are supervised models designed to extract a target linguistic structure from the output representation learned by another model. \looseness=-1&#10;&#10;Based on the authors' reading of the probing literature, there &#10;is little consensus on where to draw the line between probes and models for performing a target task (e.g.\ a part-of-speech tagger versus a probe for identifying parts of speech). The main distinction &#10;appears to be &#10;one of researcher intent: probes are, in essence, a visualisation method \cite{hupkes2018visualisation}. Their goal is not to best the state of the art, but rather to indicate whether certain information is readily available in a model---probes should not ``dig'' for information, they should just expose what is already present. Indeed, a sufficiently expressive probe with enough training data could learn any task \cite{hewitt-liang-2019-designing}, but &#10;this &#10;tells us nothing about a representation, so it is beside the point. &#10;For this reason, probes are made ``simple'' \cite{liu-etal-2019-linguistic}, which &#10;usually means they are minimally parameterised.&#10;&#10;Syntactic probes, then, are designed to measure the extent to which a target &#10;model encodes syntax. A popular example is the \defn{structural probe} \cite{hewitt}, used to compare the syntax that is decodable from different contextualised word embeddings. &#10;Rather than adopting methodology from the parsing literature, this probe utilises a novel approach for syntax extraction. However, the precise motivation for this novel approach is not immediately clear, since it has nothing to do with model complexity, and appears orthogonal to the goal of a probe. &#10;Probes are designed to help researchers understand what information exists in a model, and unfamiliar ways of measuring this information may obscure whether we are actually gaining an insight about the representation we wish to examine, or the tool of measurement itself.\looseness=-1&#10;&#10;Using the structural probe as a case study, we explore whether &#10;there is merit in designing models specifically for the purpose of &#10;probing---whether we should distinguish between the fundamental design of probes and models for performing an equivalent task, as opposed to &#10;just comparing their simplicity. &#10;We pit the structural probe against a simple parser that has the &#10;exact same lightweight parameterisation, but instead employs a standard loss function for parsing. Experimenting on multiligual $\BERT$ \cite{devlin-etal-2019-bert}, we find that in seven of nine typologically diverse languages studied (Arabic, Basque, Czech, English, Finnish, Japanese, Korean, Tamil, and Turkish), the parser boosts UUAS dramatically; for example, we observe an $11.1$-point improvement in English.&#10;&#10;In addition to using UUAS, \newcite{hewitt} also introduce &#10;a new metric---correlation of pairwise distance predictions with the gold standard. &#10;We find that the structural probe outperforms the more traditional parser substantially in terms of this new metric, but it is unclear why this metric matters more than UUAS. &#10;In our discussion, we contend that, unless a convincing argument to the contrary is provided, traditional metrics &#10;are preferable. &#10;Justifying metric choice is of central importance for probing, lest we muddy the waters with a preponderance of ill-understood metrics.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Linguistic Information Encoding, Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Neural Language Models, Syntactic Parsing Techniques, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1903.08855" label="1903.08855">
        <attvalues>
          <attvalue for="0" value="Linguistic Knowledge and Transferability of Contextual Representations" />
          <attvalue for="1" value="  Contextual word representations derived from large-scale neural language&#10;models are successful across a diverse set of NLP tasks, suggesting that they&#10;encode useful and transferable features of language. To shed light on the&#10;linguistic knowledge they capture, we study the representations produced by&#10;several recent pretrained contextualizers (variants of ELMo, the OpenAI&#10;transformer language model, and BERT) with a suite of seventeen diverse probing&#10;tasks. We find that linear models trained on top of frozen contextual&#10;representations are competitive with state-of-the-art task-specific models in&#10;many cases, but fail on tasks requiring fine-grained linguistic knowledge&#10;(e.g., conjunct identification). To investigate the transferability of&#10;contextual word representations, we quantify differences in the transferability&#10;of individual layers within contextualizers, especially between recurrent&#10;neural networks (RNNs) and transformers. For instance, higher layers of RNNs&#10;are more task-specific, while transformer layers do not exhibit the same&#10;monotonic trend. In addition, to better understand what makes contextual word&#10;representations transferable, we compare language model pretraining with eleven&#10;supervised pretraining tasks. For any given task, pretraining on a closely&#10;related task yields better performance than language model pretraining (which&#10;is better on average) when the pretraining dataset is fixed. However, language&#10;model pretraining on more data gives the best results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07958" label="2109.07958">
        <attvalues>
          <attvalue for="0" value="TruthfulQA: Measuring How Models Mimic Human Falsehoods" />
          <attvalue for="1" value="  We propose a benchmark to measure whether a language model is truthful in&#10;generating answers to questions. The benchmark comprises 817 questions that&#10;span 38 categories, including health, law, finance and politics. We crafted&#10;questions that some humans would answer falsely due to a false belief or&#10;misconception. To perform well, models must avoid generating false answers&#10;learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a&#10;T5-based model. The best model was truthful on 58% of questions, while human&#10;performance was 94%. Models generated many false answers that mimic popular&#10;misconceptions and have the potential to deceive humans. The largest models&#10;were generally the least truthful. This contrasts with other NLP tasks, where&#10;performance improves with model size. However, this result is expected if false&#10;answers are learned from the training distribution. We suggest that scaling up&#10;models alone is less promising for improving truthfulness than fine-tuning&#10;using training objectives other than imitation of text from the web.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.05302" label="2304.05302">
        <attvalues>
          <attvalue for="0" value="RRHF: Rank Responses to Align Language Models with Human Feedback&#10;  without tears" />
          <attvalue for="1" value="  Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment&#10;of large language models with human preferences, significantly enhancing the&#10;quality of interactions between humans and models. InstructGPT implements RLHF&#10;through several stages, including Supervised Fine-Tuning (SFT), reward model&#10;training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to&#10;hyperparameters and requires multiple models in its standard implementation,&#10;making it hard to train and scale up to larger parameter counts. In contrast,&#10;we propose a novel learning paradigm called RRHF, which scores sampled&#10;responses from different sources via a logarithm of conditional probabilities&#10;and learns to align these probabilities with human preferences through ranking&#10;loss. RRHF can leverage sampled responses from various sources including the&#10;model responses from itself, other large language model responses, and human&#10;expert responses to learn to rank them. RRHF only needs 1 to 2 models during&#10;tuning and can efficiently align language models with human preferences&#10;robustly without complex hyperparameter tuning. Additionally, RRHF can be&#10;considered an extension of SFT and reward model training while being simpler&#10;than PPO in terms of coding, model counts, and hyperparameters. We evaluate&#10;RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment&#10;performance with PPO by reward model score and human labeling. Extensive&#10;experiments show that the performance of RRHF is highly related to sampling&#10;quality which suggests RRHF is a best-of-n learner. Codes available at&#10;https://github.com/GanjinZero/RRHF.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13286" label="2305.13286">
        <attvalues>
          <attvalue for="0" value="How do languages influence each other? Studying cross-lingual data&#10;  sharing during LM fine-tuning" />
          <attvalue for="1" value="  Multilingual large language models (MLLMs) are jointly trained on data from&#10;many different languages such that representation of individual languages can&#10;benefit from other languages' data. Impressive performance on zero-shot&#10;cross-lingual transfer shows that these models are capable of exploiting data&#10;from other languages. Yet, it remains unclear to what extent, and under which&#10;conditions, languages rely on each other's data. In this study, we use TracIn&#10;(Pruthi et al., 2020), a training data attribution (TDA) method, to retrieve&#10;the most influential training samples seen during multilingual fine-tuning for&#10;a particular test language. This allows us to analyse cross-lingual sharing&#10;mechanisms of MLLMs from a new perspective. While previous work studied&#10;cross-lingual sharing at the level of model parameters, we present the first&#10;approach to study cross-lingual sharing at the data level. We find that MLLMs&#10;rely on data from multiple languages from the early stages of fine-tuning and&#10;that this reliance gradually increases as fine-tuning progresses. We further&#10;study how different fine-tuning languages influence model performance on a&#10;given test language and find that they can both reinforce and complement the&#10;knowledge acquired from data of the test language itself.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multilingual joint learning is often motivated by the idea that &#10;when multilingual language models (MLMs) learn information for multiple languages simultaneously, they can detect and leverage common universal patterns across them. Thus, these models can exploit data from one language to learn generalisations useful for another, obtaining impressive performance on zero-shot cross-lingual transfer for many languages \cite{wu2019beto}. Various studies suggest that representations created by popular MLMs, such as mBERT and XLM-R \cite{conneau2020unsupervised}, are not fully language-agnostic \cite{doddapaneni2021primer, singh2019bert}, but instead strike a balance between language-agnosticism and capturing the nuances of different languages through language-neutral and language-specific components \cite{libovicky2020language,gonen2020s, tanti2021language}.&#10;This naturally raises the question of how much models really benefit from multilingual data and cross-lingual sharing, and under what conditions this occurs. Many works have studied the encoding of cross-lingual patterns within MLMs by either focusing on probing for particular cross-linguistic differences \cite{ravishankar2019multilingual, choenni2022investigating}, or by analyzing the distributional properties of representational language subspaces \cite{yang2021simple, rajaee2022isotropy, chang2022geometry, chi2020finding}. Yet, it is not straightforward how to translate these results into model behavior at inference time. We aim to directly study how much influence languages exert cross-lingually on the predictions for individual languages.&#10;&#10;In this study, we take a step back in the training pipeline to study the extent to which the model exploits its multilingual training data when making predictions.&#10;We hypothesise that if a model performs cross-lingual information sharing, then it will base its inference-time predictions (to some extent) on training data from multiple languages. Analyzing the cross-lingual sharing mechanism from the data reliance perspective leads to a set of interesting questions that we explore:&#10;\begin{enumerate}&#10; \item Given a test language $A$, does our MLM tend to base its predictions only on data from $A$ itself, or does it also employ data from other languages that it was exposed to during task fine-tuning?&#10; \item Do MLMs only employ data cross-lingually out of necessity, e.g., in scenarios where in-language fine-tuning &#10; data is unavailable or insufficient? &#10; \item Do languages support each other by adding similar information to what is relied upon from in-language data (i.e., reinforcing the model in what it already learns), or do they (also) provide complementary information?&#10; \item How do cross-lingual sharing dynamics change over the course of fine-tuning?&#10; \item Is the cross-lingual sharing behaviour similar when the test language was seen during fine-tuning compared to when it is used in a zero-shot testing scenario?&#10;\end{enumerate}&#10;To study this, we use TracIn \cite{pruthi2020estimating}, a training data attribution (TDA) method to identify a set of training samples that are most informative for a particular test prediction. The influence of a training sample $z_train$ on a test sample $z_test$ can be formalized as the change in loss that would be observed for $z_test$ if $z_train$ was omitted during training. Thus, it can be used as a measure of how influential $z_train$ is when solving the task for $z_test$. &#10;&#10;To the best of our knowledge, we present the first approach to studying cross-lingual sharing at the data level by extending the use of a TDA method to the multilingual setting. We find that MLMs rely on data from multiple languages to a large extent, even when the test language was seen (or over-represented) during fine-tuning. This indicates that MLM representations might be more universal than previous work suggested \cite{singh2019bert}, in part explaining the `surprising' effectiveness of cross-lingual transfer \cite{pires2019multilingual, wu2019beto, karthikeyan2020cross}. Moreover, we find that cross-lingual sharing increases as fine-tuning progresses, and that languages can support one another by playing both reinforcing as well as complementary roles. Lastly, we find that the model exhibits different cross-lingual behaviour in the zero-shot testing setup compared to when the test language is seen during fine-tuning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Lingual Transfer Learning, Computer Science, Linguistics, Multilingual Language Models, Artificial Intelligence, Training Data Analysis" />
        </attvalues>
      </node>
      <node id="2107.00676" label="2107.00676">
        <attvalues>
          <attvalue for="0" value="A Primer on Pretrained Multilingual Language Models" />
          <attvalue for="1" value="  Multilingual Language Models (\MLLMs) such as mBERT, XLM, XLM-R,&#10;\textit{etc.} have emerged as a viable option for bringing the power of&#10;pretraining to a large number of languages. Given their success in zero-shot&#10;transfer learning, there has emerged a large body of work in (i) building&#10;bigger \MLLMs~covering a large number of languages (ii) creating exhaustive&#10;benchmarks covering a wider variety of tasks and languages for evaluating&#10;\MLLMs~ (iii) analysing the performance of \MLLMs~on monolingual, zero-shot&#10;cross-lingual and bilingual tasks (iv) understanding the universal language&#10;patterns (if any) learnt by \MLLMs~ and (v) augmenting the (often) limited&#10;capacity of \MLLMs~ to improve their performance on seen or even unseen&#10;languages. In this survey, we review the existing literature covering the above&#10;broad areas of research pertaining to \MLLMs. Based on our survey, we recommend&#10;some promising directions of future research.&#10;" />
          <attvalue for="2" value="&#10;&#10;The advent of BERT \cite{devlin-etal-2019-bert} has revolutionised the field of NLP and has lead to state of the art performance on a wide variety of tasks \cite{wang-etal-2018-glue}. The recipe is to train a deep transformer based model \cite{DBLP:journals/corr/VaswaniSPUJGKP17} on large amounts of monolingual data and then fine-tune it on small amounts of task-specific data. The pretraining happens using a masked language modeling objective and essentially results in an encoder which learns good sentence representations. These pretrained sentence representations then lead to improved performance on downstream tasks when fine-tuned on even small amounts of task-specific training data \cite{devlin-etal-2019-bert}. Given its success in English NLP, this recipe has been replicated across languages leading to many language specific BERTs such as FlauBERT (French) \cite{le2020flaubert}, CamemBERT (French) \cite{martin-etal-2020-camembert}, BERTje (Dutch) \cite{devries2019bertje}, FinBERT (Finnish) \cite{ronnqvist-etal-2019-multilingual}, BERTeus (Basque) \cite{DBLP:conf/lrec/AgerriVCBSSA20}, AfriBERT (Afrikaans) \cite{ralethe-2020-adaptation}, IndicBERT (Indian languages) \cite{kakwani2020indicnlpsuite} etc. However, training such language-specific models is only feasible for a few languages which have the necessary data and computational resources.&#10;&#10;The above situation has lead to the undesired effect of limiting recent advances in NLP to English and a few high resource languages \cite{joshi-etal-2020-state}. The question then is How do we bring the benefit of such pretrained BERT based models to a very long list of languages of interest? One alternative, which has become popular, is to train multilingual language models (MLLMs) such as mBERT \cite{devlin-etal-2019-bert}, XLM \cite{DBLP:conf/nips/ConneauL19}, XLM-R \cite{DBLP:conf/acl/ConneauKGCWGGOZ20}, etc. A MLLM~ is pretrained using large amounts of unlabeled data from multiple languages with the hope that low resource languages may benefit from high resource languages due to shared vocabulary, genetic relatedness \cite{nguyen-chiang-2017-transfer} or contact relatedness \cite{goyal-etal-2020-contact}. Several such MLLMs~ have been proposed in the past 3 years and they differ in the architecture (e.g., number of layers, parameters, etc), objective functions used for training (e.g., monolingual masked language modeling objective, translation language modeling objective, etc), data used for pretraining (Wikipedia, CommonCrawl, etc) and the number of languages involved (ranging from 12 to 100). To keep track of these rapid advances in MLLMs, as a first step, we present a survey of all existing MLLMs clearly highlighting their similarities and differences.&#10;&#10;While training an MLLM~ is more efficient and inclusive (covers more languages), is there a trade-off in the performance compared to a monolingual model? More specifically, for a given language is a language-specific BERT better than a MLLM? For example, if one is only interested in English NLP should one use English BERT or a MLLM. The advantage of the former is that there is no capacity dilution (i.e., the entire capacity of the model is dedicated to a single language), whereas the advantage of the latter is that there is additional pretraining data from multiple (related) languages. In this work, we survey several existing studies \cite{DBLP:conf/acl/ConneauKGCWGGOZ20, DBLP:conf/rep4nlp/WuD20, DBLP:conf/lrec/AgerriVCBSSA20,DBLP:journals/corr/abs-1912-07076,ronnqvist-etal-2019-multilingual,ro2020multi2oie,DBLP:journals/corr/abs-2007-09757, DBLP:journals/corr/abs-1912-07076, wang-etal-2020-galileo, DBLP:conf/rep4nlp/WuD20} which show that the right choice depends on various factors such as model capacity, amount of pretraining data, fine-tuning mechanism and amount of task-specific training data.&#10;&#10;One of the main motivations of training MLLMs~ is to enable transfer from high resource languages to low resource languages. Of particular interest, is the ability of MLLMs to facilitate zero-shot cross-lingual transfer \cite{DBLP:conf/iclr/KWMR20} from a resource rich language to a resource deprived language which does not have any task-specific training data. To evaluate such cross-lingual transfer, several benchmarks, such as XGLUE \cite{DBLP:conf/emnlp/LiangDGWGQGSJCF20}, XTREME \cite{DBLP:conf/icml/HuRSNFJ20}, XTREME-R \cite{DBLP:journals/corr/abs-2104-07412} have been proposed. We review these benchmarks which contain a wide variety of tasks such as classification, structure prediction, question answering, and cross-lingual retrieval. Using these benchmarks, several works \cite{DBLP:conf/acl/PiresSG19, DBLP:conf/emnlp/WuD19, DBLP:conf/iclr/KWMR20, DBLP:conf/acl/ArtetxeRY20, DBLP:conf/iclr/KWMR20, dufter-schutze-2020-identifying, DBLP:journals/corr/abs-2004-09205, DBLP:journals/corr/abs-2005-00633, DBLP:journals/corr/abs-2004-14218, DBLP:conf/nips/ConneauL19, DBLP:conf/emnlp/WangCGLL19,liu-etal-2019-investigating, DBLP:conf/iclr/CaoKK20,DBLP:conf/iclr/WangXXYNC20, DBLP:journals/corr/abs-2008-09112, DBLP:conf/acl/WangJBWHT20, DBLP:conf/ijcnlp/ChiDWMH20} have studied the cross-lingual effectiveness of MLLMs and have shown that such transfer depends on various factors such as amount of shared vocabulary, explicit alignment of representations across languages, size of pretraining corpora, etc. We collate the main findings of these studies in this survey. &#10;&#10;While the above discussion has focused on transfer learning and facilitating NLP in low resource languages, MLLMs~could also be used for bilingual tasks. For example, could the shared representations learnt by MLLMs~improve Machine Translation between two resource rich languages? We survey several works \cite{DBLP:conf/nips/ConneauL19, kakwani2020indicnlpsuite, DBLP:conf/emnlp/HuangLDGSJZ19, DBLP:conf/acl/ConneauKGCWGGOZ20, DBLP:conf/emnlp/EisenschlosRCKG19, DBLP:conf/semeval/ZampieriNRAKMDP20, libovicky-etal-2020-language, jalili-sabet-etal-2020-simalign, chen-etal-2020-accurate,zenkel-etal-2020-end, dou-neubig-2021-word, imamura-sumita-2019-recycling, DBLP:journals/corr/abs-2012-15547, DBLP:journals/corr/abs-2002-06823, liu-etal-2020-multilingual-denoising, xue2021mt5} which use MLLMs for downstream bilingual tasks such as unsupervised machine translation, cross-lingual word alignment, cross-lingual QA, etc. We summarise the main findings of these studies which indicate that MLLMs~are useful for bilingual tasks, particularly in low resource scenarios.&#10;&#10;The surprisingly good performance of MLLMs~in cross-lingual transfer as well as bilingual tasks motivates the hypothesis that MLLMs~are learning universal patterns. &#10;However, our survey of the studies in this space indicates that there is no consensus yet. &#10;While representations learnt by MLLMs~share commonalities across languages identified by different correlation analyses, these commonalities are dominantly within languages of the same family, and only in certain parts of the network (primarily middle layers). &#10;Also, while probing tasks such as POS tagging are able to benefit from such commonalities, harder tasks such as evaluating MT quality remain beyond the scope as yet.&#10;Thus, though promising, MLLMs~do not yet represent inter-lingua.&#10;&#10;Lastly, given the effort involved in training MLLMs~it is desirable that it is easy to extend it to new languages which weren't a part of the initial pretraining. We review existing studies which propose methods for (a) extending MLLMs~to unseen languages, and (b) improving the capacity (and hence performance) of MLLMs~for languages already seen during pretraining. These range from simple techniques such as fine-tuning the MLLM~for a few epochs on the target language to using language and task specific adapters to augment the capacity of MLLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04727" label="2109.04727">
        <attvalues>
          <attvalue for="0" value="A Simple and Effective Method To Eliminate the Self Language Bias in&#10;  Multilingual Representations" />
          <attvalue for="1" value="  Language agnostic and semantic-language information isolation is an emerging&#10;research direction for multilingual representations models. We explore this&#10;problem from a novel angle of geometric algebra and semantic space. A simple&#10;but highly effective method &quot;Language Information Removal (LIR)&quot; factors out&#10;language identity information from semantic related components in multilingual&#10;representations pre-trained on multi-monolingual data. A post-training and&#10;model-agnostic method, LIR only uses simple linear operations, e.g. matrix&#10;factorization and orthogonal projection. LIR reveals that for weak-alignment&#10;multilingual systems, the principal components of semantic spaces primarily&#10;encodes language identity information. We first evaluate the LIR on a&#10;cross-lingual question answer retrieval task (LAReQA), which requires the&#10;strong alignment for the multilingual embedding space. Experiment shows that&#10;LIR is highly effectively on this task, yielding almost 100% relative&#10;improvement in MAP for weak-alignment models. We then evaluate the LIR on&#10;Amazon Reviews and XEVAL dataset, with the observation that removing language&#10;information is able to improve the cross-lingual transfer performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.08484" label="2002.08484">
        <attvalues>
          <attvalue for="0" value="Estimating Training Data Influence by Tracing Gradient Descent" />
          <attvalue for="1" value="  We introduce a method called TracIn that computes the influence of a training&#10;example on a prediction made by the model. The idea is to trace how the loss on&#10;the test point changes during the training process whenever the training&#10;example of interest was utilized. We provide a scalable implementation of&#10;TracIn via: (a) a first-order gradient approximation to the exact computation,&#10;(b) saved checkpoints of standard training procedures, and (c) cherry-picking&#10;layers of a deep neural network. In contrast with previously proposed methods,&#10;TracIn is simple to implement; all it needs is the ability to work with&#10;gradients, checkpoints, and loss functions. The method is general. It applies&#10;to any machine learning model trained using stochastic gradient descent or a&#10;variant of it, agnostic of architecture, domain and task. We expect the method&#10;to be widely useful within processes that study and improve training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.14993" label="2310.14993">
        <attvalues>
          <attvalue for="0" value="Understanding the Inner Workings of Language Models Through&#10;  Representation Dissimilarity" />
          <attvalue for="1" value="  As language models are applied to an increasing number of real-world&#10;applications, understanding their inner workings has become an important issue&#10;in model trust, interpretability, and transparency. In this work we show that&#10;representation dissimilarity measures, which are functions that measure the&#10;extent to which two model's internal representations differ, can be a valuable&#10;tool for gaining insight into the mechanics of language models. Among our&#10;insights are: (i) an apparent asymmetry in the internal representations of&#10;model using SoLU and GeLU activation functions, (ii) evidence that&#10;dissimilarity measures can identify and locate generalization properties of&#10;models that are invisible via in-distribution test set performance, and (iii)&#10;new evaluations of how language model features vary as width and depth are&#10;increased. Our results suggest that dissimilarity measures are a promising set&#10;of tools for shedding light on the inner workings of language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In this sectionwe review the two model dissimilarity measures appearing in this paper. &#10;&#10;Model stitching \cite{model_stitching_bansal}: Informally, model stitching asks how well the representation extracted by the early layers of one model can be used by the later layers of another model to solve a specific task. Let \(f \) be a neural network and for a layer \(l \) of \(f \) let \(f_{\leq l} \) (respectively \(f_{\geq l}\)) be the composition of the first \(l \) layers of \(f\) (respectively the layers \(m \) of \(f\) with \(m \geq l\)). Given another network \( g \) the model obtained by stitching layer \(l \) of \(f \) to layer \(m\) of \(g \) with stitching layer \(\varphi \) is \(g_{&gt;m}\circ \varphi \circ f_{\leq l} \). The performance of this stitched network measures the similarity of representations of $f$ at layer $l$ and representations of $g$ at layer $m$.&#10;&#10;Centered kernel alignment (CKA) \cite{kornblithSimilarityNeuralNetwork2019}: Let $D = \{x_1,\dots,x_d\}$ be a set of model inputs. For models $f$ and $g$ with layers $l$ and $m$ respectively, let $A_{f,l,g,m}$ be the covariance matrix of $f_{\leq l}(D)$ and $g_{\leq m}(D)$. Then the CKA score for models $f$ and $g$ at layers $l$ and $m$ respectively and evaluated at $D$ is&#10;\begin{equation}&#10;\frac{||A_{f,l,g,m}||_F^2}{||A_{f,l,f,l}||_F||A_{g,m,g,m}||_F},&#10;\end{equation}&#10;where $||\cdot||_F$ is the Frobenious norm. Higher CKA scores indicate more structural similarity between representations. In our experiments we use an unbiased estimator of \ref{eq:defn-of-cka} to calculate CKA in batches (see \ref{sec:expdet} for details).&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Model Transparency Insights, Mathematics, Representation Dissimilarity Measures, Language Model Interpretability" />
        </attvalues>
      </node>
      <node id="2308.16137" label="2308.16137">
        <attvalues>
          <attvalue for="0" value="LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language&#10;  Models" />
          <attvalue for="1" value="  Today's large language models (LLMs) typically train on short text segments&#10;(e.g., &lt;4K tokens) due to the quadratic complexity of their Transformer&#10;architectures. As a result, their performance suffers drastically on inputs&#10;longer than those encountered during training, substantially limiting their&#10;applications in real-world tasks involving long contexts such as encoding&#10;scientific articles, code repositories, or long dialogues. Through theoretical&#10;analysis and empirical investigation, this work identifies three major factors&#10;contributing to this length generalization failure. Our theoretical analysis&#10;further reveals that commonly used techniques like truncating the attention&#10;window or relative positional encodings are inadequate to address them.&#10;Answering these challenges, we propose LM-Infinite, a simple and effective&#10;method for enhancing LLMs' capabilities of handling long contexts. LM-Infinite&#10;is highly flexible and can be used with most modern LLMs off-the-shelf. Without&#10;any parameter updates, it allows LLMs pre-trained with 2K or 4K-long segments&#10;to generalize to up to 200M length inputs while retaining perplexity. It also&#10;improves performance on downstream tasks such as Passkey Retrieval and Qasper&#10;in the zero-shot setting. LM-Infinite brings substantial efficiency&#10;improvements: it achieves 2.7x decoding speed up and 7.5x memory saving over&#10;the original model. Our codes are released at&#10;\url{https://github.com/Glaciohound/LM-Infinite}.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have recently advanced the state-of-the-art across various natural language processing tasks. They typically train on text segments of fewer than 4K tokens~\cite{touvron2023llama2, MosaicML2023Introducing}, primarily due to the computational overhead quadratic in the input lengths of their Transformer architectures. As a result, they face challenges in generalization to inputs that are excessively longer than what they are trained on and suffer substantial deterioration &#10; in their performance~\cite {tworkowski2023focused, chen2023extending}. This limits their applicability in tasks that require long-range contexts, such as encoding scientific articles, source code repository generation, or long-context dialogues.&#10;&#10;Extensive efforts have been devoted to addressing this length generalization challenge.&#10;Relative positional encodings such as RoPE~\cite{su2021roformer} and Alibi~\cite{press2021train} have been widely adopted by state-of-the-art LLMs, which calculate attention based on inter-token distance instead of absolute positions, hoping to avoid model failures due to unseen absolute position embeddings.&#10;Moreover, although applying a sliding-window attention pattern on the Transformer architecture can reduce the memory overhead~\cite{beltagy2020longformer, ding2023longnet, zaheer2020big}, they are not directly applicable to pre-trained models for length generalization without further training.&#10;Through both theoretical analysis and empirical investigation, \S\ref{sec:diagnosis} pinpoints three primary factors underlying the length generalization failures:&#10;(1) the challenge of handling unseen distances among tokens, &#10;(2) the difficulty of attending to unseen numbers of tokens, &#10;and (3) implicitly encoded absolute positional information in initial tokens.&#10;These challenges can make LLMs' computational features, such as attention logits and hidden vectors, deviate from the training distribution,&#10;leading to failures of length generalization.&#10;Existing techniques fall short of addressing these underlying issues.&#10;&#10;Answering these challenges, we propose LM-Infinite, a simple and effective method to enhance Transformer LLMs' capabilities for modeling long contextswithout parameter updates. &#10;LM-Infinite consists of two major components designed to alleviate the three factors above. (1) a $\boldsymbol{\Lambda}$-shaped attention mask and (2) a ceiling on attention distances. &#10;The former forces the model to attend to only the beginning of the sequence and the most recent tokens within a pre-defined window, ignoring the rest.&#10;The latter component caps the relative distance values to the maximum the model has seen during training. It can also optionally re-introduce top-$k$ tokens in the middle to achieve better performance in some downstream tasks.&#10;LM-Infinite is highly flexible and applies to any off-the-shelf LLMs that use relative positional encoding and does not require any finetuning.&#10;&#10;Our experiments thoroughly evaluate LM-Infinite on a variety of tasks and LLMs.&#10;On ArXiv (academic papers) and OpenWebText2 (Reddit posts)&#10;LM-Infinite facilitates zero-shot generalization for a wide range of LLMs to texts up to 200M tokens, retaining the language modeling perplexity and generation quality.&#10;Without any parameter updates, LM-Infinite improves scores compared with the original model and truncation baselines on downstream tasks including Passkey Retrieval~\cite{mohtashami2023landmark} and Qasper~\cite{dasigi2021dataset}, which are two established benchmarks for long-context evaluation. We observe a 37.2\% gain on Passkey Retrieval and a 1.2\% gain on Qasper in the zero-shot setting.&#10;LM-Infinite also brings substantial efficiency improvements: it achieves 2.7$\times$ decoding speed up and 7.5$\times$ GPU memory saving over the original LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Limitations, Model Efficiency Optimization, Long Context Processing, Mathematics" />
        </attvalues>
      </node>
      <node id="2104.09864" label="2104.09864">
        <attvalues>
          <attvalue for="0" value="RoFormer: Enhanced Transformer with Rotary Position Embedding" />
          <attvalue for="1" value="  Position encoding recently has shown effective in the transformer&#10;architecture. It enables valuable supervision for dependency modeling between&#10;elements at different positions of the sequence. In this paper, we first&#10;investigate various methods to integrate positional information into the&#10;learning process of transformer-based language models. Then, we propose a novel&#10;method named Rotary Position Embedding(RoPE) to effectively leverage the&#10;positional information. Specifically, the proposed RoPE encodes the absolute&#10;position with a rotation matrix and meanwhile incorporates the explicit&#10;relative position dependency in self-attention formulation. Notably, RoPE&#10;enables valuable properties, including the flexibility of sequence length,&#10;decaying inter-token dependency with increasing relative distances, and the&#10;capability of equipping the linear self-attention with relative position&#10;encoding. Finally, we evaluate the enhanced transformer with rotary position&#10;embedding, also called RoFormer, on various long text classification benchmark&#10;datasets. Our experiments show that it consistently overcomes its alternatives.&#10;Furthermore, we provide a theoretical analysis to explain some experimental&#10;results. RoFormer is already integrated into Huggingface:&#10;\url{https://huggingface.co/docs/transformers/model_doc/roformer}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.03011" label="2105.03011">
        <attvalues>
          <attvalue for="0" value="A Dataset of Information-Seeking Questions and Answers Anchored in&#10;  Research Papers" />
          <attvalue for="1" value="  Readers of academic research papers often read with the goal of answering&#10;specific questions. Question Answering systems that can answer those questions&#10;can make consumption of the content much more efficient. However, building such&#10;tools requires data that reflect the difficulty of the task arising from&#10;complex reasoning about claims made in multiple parts of a paper. In contrast,&#10;existing information-seeking question answering datasets usually contain&#10;questions about generic factoid-type information. We therefore present QASPER,&#10;a dataset of 5,049 questions over 1,585 Natural Language Processing papers.&#10;Each question is written by an NLP practitioner who read only the title and&#10;abstract of the corresponding paper, and the question seeks information present&#10;in the full text. The questions are then answered by a separate set of NLP&#10;practitioners who also provide supporting evidence to answers. We find that&#10;existing models that do well on other QA tasks do not perform well on answering&#10;these questions, underperforming humans by at least 27 F1 points when answering&#10;them from entire papers, motivating further research in document-grounded,&#10;information-seeking QA, which our dataset is designed to facilitate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.01992" label="2212.01992">
        <attvalues>
          <attvalue for="0" value="Fast and accurate factorized neural transducer for text adaption of&#10;  end-to-end speech recognition models" />
          <attvalue for="1" value="  Neural transducer is now the most popular end-to-end model for speech&#10;recognition, due to its naturally streaming ability. However, it is challenging&#10;to adapt it with text-only data. Factorized neural transducer (FNT) model was&#10;proposed to mitigate this problem. The improved adaptation ability of FNT on&#10;text-only adaptation data came at the cost of lowered accuracy compared to the&#10;standard neural transducer model. We propose several methods to improve the&#10;performance of the FNT model. They are: adding CTC criterion during training,&#10;adding KL divergence loss during adaptation, using a pre-trained language model&#10;to seed the vocabulary predictor, and an efficient adaptation approach by&#10;interpolating the vocabulary predictor with the n-gram language model. A&#10;combination of these approaches results in a relative word-error-rate reduction&#10;of 9.48\% from the standard FNT model. Furthermore, n-gram interpolation with&#10;the vocabulary predictor improves the adaptation speed hugely with satisfactory&#10;adaptation performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, neural transducer based end-to-end (E2E) models \cite{Graves-RNNSeqTransduction, he2019streaming, attentionisallyouneed, yeh2019transformer, TT, Li2019RNNT, battenberg2017exploring,chiu2018state, Li2020comparison, xiechen, E2EOverview}, such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction}, transformer-transducer (T-T) \cite{yeh2019transformer, TT} and conformer-transducer (C-T) \cite{gulati2020conformer}, have become the dominant model for automatic speech recognition (ASR) in industry due to its natural streaming property, as well as competitive accuracy with traditional hybrid speech recognition systems \cite{watanabe2017hybrid, sainath2020streaming, Li2020Developing}. &#10;&#10;However, one of the main challenges for neural transducer models is adaptation using only text data. This is because in neural transducer models, there are no separated acoustic or language model like in traditional hybrid models. Although the prediction network could be considered as an internal language model (LM) because the input to it is the previously predicted token, it is not a real LM since the prediction output needs to be combined with the acoustic encoder in a non-linear way to generate posteriors over the vocabulary augmented with a blank token. Adapting the prediction network using text-only data is not as straightforward or effective as adapting the LM in hybrid systems. Paired audio and text data is needed to adapt a neural transducer model, however, collecting labeled audio data is both time and money costly. &#10;&#10;There are several types of methods proposed to address this issue. One is to generate artificial audio for adaptation text instead of collecting real audio. Audio generation method could be based on multi-speaker neural text to speech (TTS) model \cite{Li2020Developing, sim2019personalization, deng2020ttsrnnt, zheng2021ttsasr, ttsjasha} or spliced data method \cite{spliced}. The neural transducer model could then be fine-tuned with artificial paired audio and text data. A major drawback of these kinds of methods is the high computational cost. It takes much longer for the TTS-based methods to generate audio even with GPU machines, while the spliced-data method has very small cost for generating audio. However, the adaptation step for both methods involves updating part of the encoder, full prediction and the joint network with the RNN-T loss. The result is high computational cost for training, need for GPUs, and too much delay for scenarios in which rapid adaptation is necessary. &#10;&#10;Another class of text-only adaptation methods is LM fusion \cite{kannan2018shallowfusion, 2020fusion, 2021fusion, amazonilm, triebiasing}, such as shallow fusion \cite{kannan2018shallowfusion} where an external LM trained on target-domain text is incorporated during the neural transducer model decoding. However, there is already an internal LM in the neural transducer model. Directly adding an external LM is not mathematically grounded. To solve such an issue, density ratio \cite{mcdermott2019densityratio}, hybrid autoregressive transducer model \cite{variani2020hybrid}, and internal LM estimation \cite{meng2021ilme,ibmilm} were proposed to remove the influence of the internal LM contained in the neural transducer model. However, the performance is often sensitive to the interpolation weight of external LM for different tasks, and it needs to be well tuned based on development data to get optimal results \cite{meng202ilmt}. &#10;&#10;Different from aforementioned methods, factorized neural transducer model (FNT) \cite{fnt} modifies the neural transducer model architecture by factorizing the blank and vocabulary prediction so that a standalone LM can be used for the vocabulary prediction. In this way, various language model adaptation [32, 33, 34] techniques could be applied to FNT. But based on results in \cite{fnt}, FNT degrades the accuracy on general testing sets compared with the standard neural transducer model, although it significantly improves the accuracy in the new domain after adaptation. Besides, it still needs significant GPU time to finetune FNT with text only data for the adaptation, which may not meet the fast adaptation requirement in some real applications. &#10;&#10;In this paper, we proposed several methods to advance FNT for effective and efficient adaptation with text only data. These methods include: 1) Adding Connectionist Temporal Classification (CTC) \cite{Graves-CTCFirst} criterion for the encoder network during training to make it work more like an acoustic model. Then, the combination of encoder output and vocabulary predictor output is similar to the combination of acoustic and language model in hybrid models. 2) Adding Kullback-Leibler (KL) divergence between the outputs of adapted model and baseline model to avoid over fitting to the adaptation data. 3) Initializing the vocabulary predictor with a neural LM trained with more text data. 4) Replacing the network fine-tuning with more efficient adaptation method using n-gram interpolation. Experimental results showed that on general testing sets, these methods help the modified FNT to get even a little better accuracy than the baseline neural transducer model. On adaptation sets, the word error rate (WER) after the adaptation of modified FNT is reduced by 9.48\% relatively from the standard FNT model, and reduced by 29.21\% relatively from the baseline C-T model. Besides, n-gram interpolation results in much faster adaptation speed. &#10;&#10;The rest of this paper is organized as follows: Section \ref{sec:rnnt} introduces the neural transducer model and FNT model. Section \ref{sec:refinefnt} presents the proposed methods for modified FNT in detail. Section \ref{sec:exp} shows the experimental results and analysis. Section \ref{sec:conclusion} gives the conclusions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Transducer Models, Artificial Intelligence, Mathematics, Speech Recognition, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2007.15188" label="2007.15188">
        <attvalues>
          <attvalue for="0" value="Developing RNN-T Models Surpassing High-Performance Hybrid Models with&#10;  Customization Capability" />
          <attvalue for="1" value="  Because of its streaming nature, recurrent neural network transducer (RNN-T)&#10;is a very promising end-to-end (E2E) model that may replace the popular hybrid&#10;model for automatic speech recognition. In this paper, we describe our recent&#10;development of RNN-T models with reduced GPU memory consumption during&#10;training, better initialization strategy, and advanced encoder modeling with&#10;future lookahead. When trained with Microsoft's 65 thousand hours of anonymized&#10;training data, the developed RNN-T model surpasses a very well trained hybrid&#10;model with both better recognition accuracy and lower latency. We further study&#10;how to customize RNN-T models to a new domain, which is important for deploying&#10;E2E models to practical scenarios. By comparing several methods leveraging&#10;text-only data in the new domain, we found that updating RNN-T's prediction and&#10;joint networks using text-to-speech generated from domain-specific text is the&#10;most effective.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00858" label="2105.00858">
        <attvalues>
          <attvalue for="0" value="On Addressing Practical Challenges for RNN-Transducer" />
          <attvalue for="1" value="  In this paper, several works are proposed to address practical challenges for&#10;deploying RNN Transducer (RNN-T) based speech recognition system. These&#10;challenges are adapting a well-trained RNN-T model to a new domain without&#10;collecting the audio data, obtaining time stamps and confidence scores at word&#10;level. The first challenge is solved with a splicing data method which&#10;concatenates the speech segments extracted from the source domain data. To get&#10;the time stamp, a phone prediction branch is added to the RNN-T model by&#10;sharing the encoder for the purpose of force alignment. Finally, we obtain&#10;word-level confidence scores by utilizing several types of features calculated&#10;during decoding and from confusion network. Evaluated with Microsoft production&#10;data, the splicing data adaptation method improves the baseline and adaptation&#10;with the text to speech method by 58.03% and 15.25% relative word error rate&#10;reduction, respectively. The proposed time stamping method can get less than&#10;50ms word timing difference from the ground truth alignment on average while&#10;maintaining the recognition accuracy of the RNN-T model. We also obtain high&#10;confidence annotation performance with limited computation cost.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, all-neural end-to-end (E2E) models \cite{Graves-CTCFirst, Graves-RNNSeqTransduction, chan2016listen, attentionisallyouneed, he2019streaming} have become the trend in automatic speech recognition (ASR) due to its simple training and decoding processes, as well as the similar or even better accuracy compared with traditional hybrid speech recognition systems \cite{watanabe2017hybrid, sainath2020streaming, Li2020Developing}. Commonly used E2E ASR models include Connectionist Temporal Classification (CTC) model \cite{Graves-CTCFirst,miao2015eesen,li2018advancing}, Attention-based Encoder-Decoder (AED) model \cite{chan2016listen}, and transducer models such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction} and transformer-transducer (T-T) \cite{TT}. The industry has more interest in transducer models including RNN-T and T-T because these models can be streaming in a natural way \cite{he2019streaming, TT, Li2019RNNT, battenberg2017exploring,chiu2018state, Li2020comparison, yeh2019transformer, xiechen}. However, there are lots of practical challenges that need to be addressed in order to deploy all these models. In this paper, we will address some of them by using RNN-T as a backbone model. All solutions can be applied to other transducer models such as T-T. &#10;&#10;The first challenge is to adapt a RNN-T model to a new domain without collecting the speech data of the new domain. Because there is no separate language model in RNN-T, we couldn't easily adapt it with text only data as what we do for a hybrid ASR system. Instead, we need to collect paired speech-text data of the new domain which contains enough speaker and environment variation to adapt the RNN-T model. Such collection usually costs a lot on both time and money, which forbids fast model adaption \cite{belladaptation}. Therefore, how to get a large amount of paired speech-text data with a small cost is a practical problem for the domain adaptation of RNN-T. &#10;&#10;Some methods have been proposed to solve this problem. The most popular method is to synthesize speech from the new domain texts using the text to speech (TTS) technology \cite{Li2020Developing, eva2020adaptation, peyser2019improving,baskar2019self,murthy2018effect}. For example, \cite{Li2020Developing} uses a multi-speaker neural TTS system to generate speech data using the text-only data of the new domain to adapt the RNN-T model. \cite{peyser2019improving} improves a general ASR model’s performance on the numeric data domain with TTS generated numeric speech data. \cite{murthy2018effect} improves OOV detection by adding missing syllable sounds using a cross-lingual TTS system. Although no real speech data needs to be collected, the TTS-based method has its limitations: 1) the speaker variation in TTS generated data is limited compared with the real production data used for ASR model training, 2) The cost of training a multi-speaker neural TTS model and the generation of synthesized speech is large. There are also other methods proposed for the domain adaptation of E2E models \cite{zhongdomain,domainlow}. However, the speech data for the new domain is still needed in these methods.&#10;&#10;There is also a requirement that RNN-T can provide functions such as time stamp and confidence measure at word level. Getting word timings with RNN-T is challenging because it's not designed for this: no time alignment information is used during training, therefore it cannot generate reliable word start and end time for every recognized word. To solve this problem, \cite{wordtime} applies time constraints during training to improve the word timing accuracy. It uses an extra token ``word boundary'' to mark the word start time and the last word piece of the word to mark the word end time. This method increases the training cost because adding ``word boundary'' for each word increases the training target label length. Applying time constraints to RNN-T models also causes significant accuracy degradation \cite{wordtime}. Confidence annotation is also a challenging task for RNN-T models. Most applications require word level or utterance level confidence scores, while output units of RNN-T models are usually subword units such as word pieces. In \cite{conf_slt, conf_google}, word-level confidence scores are obtained through aggregating confidence scores of constituent word pieces and directly modeling last word piece using an end-of-word mask. Although these methods are very effective, the rich information included in all the word pieces instead of the last one for every word is lost to some extent.&#10;&#10;In this paper, we detail our solutions to these practical challenges. To adapt RNN-T without collecting new speech data, a novel splicing data generation method is proposed by concatenating the sampled speech segments corresponding to underlying words of target texts into new utterances. The sampled speech segments are extracted from the existing training data randomly. &#10;It has the following advantages compared with TTS-based speech generation method : 1) the cost is almost zero since it is only based on the existing speech data, without the need of training any extra model and the cost-consuming TTS speech generation, 2) the constructed data could cover more speaker and acoustic environment variation, which makes the model more robust. &#10;The proposed method is evaluated by adapting a general RNN-T model to a new domain with the spliced data, achieving 15.25\% relative word error rate reduction over the model adapted with TTS generated data. &#10;&#10;To provide reliable word timings for RNN-T, we propose adding a context independent (CI) phone prediction branch on top of the encoder of the RNN-T model. &#10;The word timings are calculated by aligning the recognition results from the RNN-T model using the phone probability of each frame from the CI phone prediction model in the second pass. Since the CI phone prediction model shares the encoder with RNN-T model and the phone level alignment is very cheap. The extra computational cost of this second-pass alignment is acceptable. The experiments proved that the proposed method could get less than 50ms word timing difference on average compared with the ground truth while maintaining the recognition accuracy of the RNN-T model. &#10;&#10;Confidence annotation is treated as a binary classification task in our work. We first obtain confidence features at word piece level directly from decoding, then aggregate them to word level features. Compared to aggregating confidence scores \cite{conf_slt, conf_google}, aggregating features could retain much richer information, thus making the final word-level confidence scores more reliable. Together with the features calculated from confusion network generated with a N-best list \cite{confusion_network, cn_jian}, we train a two-layer feed forward neural network to classify each recognized word as ``correct'' or ``incorrect''. Experimental results showed that such a method could achieve high confidence annotation performance with limited computation cost. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.01996" label="1712.01996">
        <attvalues>
          <attvalue for="0" value="An analysis of incorporating an external language model into a&#10;  sequence-to-sequence model" />
          <attvalue for="1" value="  Attention-based sequence-to-sequence models for automatic speech recognition&#10;jointly train an acoustic model, language model, and alignment mechanism. Thus,&#10;the language model component is only trained on transcribed audio-text pairs.&#10;This leads to the use of shallow fusion with an external language model at&#10;inference time. Shallow fusion refers to log-linear interpolation with a&#10;separately trained language model at each step of the beam search. In this&#10;work, we investigate the behavior of shallow fusion across a range of&#10;conditions: different types of language models, different decoding units, and&#10;different tasks. On Google Voice Search, we demonstrate that the use of shallow&#10;fusion with a neural LM with wordpieces yields a 9.1% relative word error rate&#10;reduction (WERR) over our competitive attention-based sequence-to-sequence&#10;model, obviating the need for second-pass rescoring.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.07705" label="2003.07705">
        <attvalues>
          <attvalue for="0" value="Hybrid Autoregressive Transducer (hat)" />
          <attvalue for="1" value="  This paper proposes and evaluates the hybrid autoregressive transducer (HAT)&#10;model, a time-synchronous encoderdecoder model that preserves the modularity of&#10;conventional automatic speech recognition systems. The HAT model provides a way&#10;to measure the quality of the internal language model that can be used to&#10;decide whether inference with an external language model is beneficial or not.&#10;This article also presents a finite context version of the HAT model that&#10;addresses the exposure bias problem and significantly simplifies the overall&#10;training and inference. We evaluate our proposed model on a large-scale voice&#10;search task. Our experiments show significant improvements in WER compared to&#10;the state-of-the-art approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.02317" label="2203.02317">
        <attvalues>
          <attvalue for="0" value="Adaptive Discounting of Implicit Language Models in RNN-Transducers" />
          <attvalue for="1" value="  RNN-Transducer (RNN-T) models have become synonymous with streaming&#10;end-to-end ASR systems. While they perform competitively on a number of&#10;evaluation categories, rare words pose a serious challenge to RNN-T models. One&#10;main reason for the degradation in performance on rare words is that the&#10;language model (LM) internal to RNN-Ts can become overconfident and lead to&#10;hallucinated predictions that are acoustically inconsistent with the underlying&#10;speech. To address this issue, we propose a lightweight adaptive LM discounting&#10;technique AdaptLMD, that can be used with any RNN-T architecture without&#10;requiring any external resources or additional parameters. AdaptLMD uses a&#10;two-pronged approach: 1) Randomly mask the prediction network output to&#10;encourage the RNN-T to not be overly reliant on it's outputs. 2) Dynamically&#10;choose when to discount the implicit LM (ILM) based on rarity of recently&#10;predicted tokens and divergence between ILM and implicit acoustic model (IAM)&#10;scores. Comparing AdaptLMD to a competitive RNN-T baseline, we obtain up to 4%&#10;and 14% relative reductions in overall WER and rare word PER, respectively, on&#10;a conversational, code-mixed Hindi-English ASR task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.01380" label="2102.01380">
        <attvalues>
          <attvalue for="0" value="Internal Language Model Training for Domain-Adaptive End-to-End Speech&#10;  Recognition" />
          <attvalue for="1" value="  The efficacy of external language model (LM) integration with existing&#10;end-to-end (E2E) automatic speech recognition (ASR) systems can be improved&#10;significantly using the internal language model estimation (ILME) method. In&#10;this method, the internal LM score is subtracted from the score obtained by&#10;interpolating the E2E score with the external LM score, during inference. To&#10;improve the ILME-based inference, we propose an internal LM training (ILMT)&#10;method to minimize an additional internal LM loss by updating only the E2E&#10;model components that affect the internal LM estimation. ILMT encourages the&#10;E2E model to form a standalone LM inside its existing components, without&#10;sacrificing ASR accuracy. After ILMT, the more modular E2E model with matched&#10;training and inference criteria enables a more thorough elimination of the&#10;source-domain internal LM, and therefore leads to a more effective integration&#10;of the target-domain external LM. Experimented with 30K-hour trained recurrent&#10;neural network transducer and attention-based encoder-decoder models, ILMT with&#10;ILME-based inference achieves up to 31.5% and 11.4% relative word error rate&#10;reductions from standard E2E training with Shallow Fusion on out-of-domain&#10;LibriSpeech and in-domain Microsoft production test sets, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.01500" label="2110.01500">
        <attvalues>
          <attvalue for="0" value="Factorized Neural Transducer for Efficient Language Model Adaptation" />
          <attvalue for="1" value="  In recent years, end-to-end (E2E) based automatic speech recognition (ASR)&#10;systems have achieved great success due to their simplicity and promising&#10;performance. Neural Transducer based models are increasingly popular in&#10;streaming E2E based ASR systems and have been reported to outperform the&#10;traditional hybrid system in some scenarios. However, the joint optimization of&#10;acoustic model, lexicon and language model in neural Transducer also brings&#10;about challenges to utilize pure text for language model adaptation. This&#10;drawback might prevent their potential applications in practice. In order to&#10;address this issue, in this paper, we propose a novel model, factorized neural&#10;Transducer, by factorizing the blank and vocabulary prediction, and adopting a&#10;standalone language model for the vocabulary prediction. It is expected that&#10;this factorization can transfer the improvement of the standalone language&#10;model to the Transducer for speech recognition, which allows various language&#10;model adaptation techniques to be applied. We demonstrate that the proposed&#10;factorized neural Transducer yields 15% to 20% WER improvements when&#10;out-of-domain text data is used for language model adaptation, at the cost of a&#10;minor degradation in WER on a general test set.&#10;" />
          <attvalue for="2" value="&#10;In recent years, end-to-end (E2E) based models \cite{chan2016listen, battenberg2017exploring, rao2017exploring, chiu2018state, Li18CTCnoOOV, he2019streaming, Li2019RNNT, shi2020emformer}&#10;have attracted increasing research interest in automatic speech recognition (ASR) &#10;systems. Compared to traditional HMM based models, where the acoustic model, &#10;lexicon and language model are built and optimized separately, a single neural &#10;network is used in E2E models to directly predict the word sequence. &#10;Nowadays, neural Transducer \cite{graves2012sequence, yeh2019transformer, zhang2020transformer} and Attention-based Encoder-Decoder &#10;(AED) \cite{chan2016listen, Attention-bahdanau2014, &#10;Attention-speech-chorowski2015} are two most popular choices for E2E based ASR systems.&#10;AED models achieved very good performance by adopting the &#10;attention mechanism and fusing the acoustic and linguistic information at the early stage.&#10;However, they are not streamable models in nature. &#10;There are some efforts to allow AED models to work in streaming mode, &#10;such as monotonic chunk-wise attention \cite{DBLP:conf/iclr/ChiuR18} and &#10;triggered attention \cite{DBLP:conf/icassp/MoritzHR19, wang2020reducing}.&#10;In contrast, the neural Transducer model provides a more attractive solution for &#10;streaming ASR and has been reported to outperform traditional hybrid systems &#10;\cite{Li2020Developing, sainath2020streaming, jain2019rnn} in some scenarios.&#10;Therefore, in this work, we mainly focus on the Transducer model in light&#10;of the streaming scenario in practice.&#10;&#10;However, the simplicity of E2E models also brings some sacrifice.&#10;There are no individual acoustic and language models&#10;in a neural Transducer. Although the predictor looks similar to a &#10;language model in terms of model structure and an internal &#10;language model \cite{variani2020hat, meng2021ilme} could be extracted from&#10;the predictor and joint network,&#10;it does not perform as a language model because the predictor needs to coordinate with &#10;the acoustic encoder closely. Hence, it is not straightforward&#10;to utilize text-only data to adapt the Transducer model &#10;from the source domain to the target domain. As a result,&#10;effective and efficient language model&#10;adaptation remains an open research problem for E2E based ASR models.&#10;&#10;There are continuous efforts in the speech community to address this issue.&#10;One research direction is to adopt Text-to-Speech (TTS) techniques to synthesize audio with the target-domain text \cite{Li2020Developing, sim2019personalization, zheng2021ttsasr, deng2020ttsrnnt},&#10;and then fine-tune the Transducer model on the synthesized audio and text pairs.&#10;However, this approach is computationally expensive. It is not&#10;flexible and practical for scenarios requiring fast adaption.&#10;LM fusion is another popular choice to incorporate external language &#10;models trained on target-domain text, such as&#10;shallow fusion \cite{kannan2018shallowfusion} and &#10;density ratio based LM integration \cite{variani2020hat, meng2021ilme, mcdermott2019densityratio, meng202ilmt, meng2021mwe}.&#10;However, the interpolation weight is task-dependent and needs to be tuned&#10;on dev data. The performance might be sensitive to the interpolation weight.&#10;There are some recent efforts to fine-tune the predictor \cite{pylkkonen2021fastadapt} or the internal language model \cite{Meng2021ILMA} with an additional language model loss, and then make it behave similar to a language model.&#10;Nevertheless, the predictor in neural Transducer is not equivalent to a language model&#10;in nature. It needs to coordinate&#10;with the acoustic encoder, and predict the blank to prevent outputting repetitive&#10;word \cite{ghodsi2020statelessrnn}. &#10;&#10;As discussed above, most previous work on LM adaptation adopted the standard neural Transducer&#10;architecture \cite{graves2012sequence, he2019streaming}.&#10;In this paper, we propose a modified model architecture to explicitly &#10;optimize the predictor towards a standard neural language model during training. &#10;We name it factorized neural Transducer, which factorizes the blank and vocabulary &#10;prediction, allowing the vocabulary predictor to work as a standalone language model.&#10;As a result, various language model adaptation \cite{bellegarda2004lmadapt, chen2015rnnlmadapt, li2018rnnlmadapt} techniques could be simply applied to the factorized Transducer model.&#10;The improvement&#10;of the standalone language model is expected to yield consistent performance&#10;gain for speech recognition, which is similar to the effect of language model in the HMM based ASR system.&#10;We hope this work could shed some light on the re-design of model architecture, &#10;by disentangling the fusion of AM and LM in E2E models&#10;for efficient language model adaptation and customization.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.10205" label="2403.10205">
        <attvalues>
          <attvalue for="0" value="Read between the lines -- Functionality Extraction From READMEs" />
          <attvalue for="1" value="  While text summarization is a well-known NLP task, in this paper, we&#10;introduce a novel and useful variant of it called functionality extraction from&#10;Git README files. Though this task is a text2text generation at an abstract&#10;level, it involves its own peculiarities and challenges making existing&#10;text2text generation systems not very useful. The motivation behind this task&#10;stems from a recent surge in research and development activities around the use&#10;of large language models for code-related tasks, such as code refactoring, code&#10;summarization, etc. We also release a human-annotated dataset called FuncRead,&#10;and develop a battery of models for the task. Our exhaustive experimentation&#10;shows that small size fine-tuned models beat any baseline models that can be&#10;designed using popular black-box or white-box large language models (LLMs) such&#10;as ChatGPT and Bard. Our best fine-tuned 7 Billion CodeLlama model exhibit 70%&#10;and 20% gain on the F1 score against ChatGPT and Bard respectively.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs) are known to perform really well on many {text2text} \cite{yang2021towards} generation tasks such as {summarization} \cite{liu2019text, el2021automatic}), {translation} \cite{wang2019learning, maruf2021survey}, etc. Because of this success, there is a growing research interest in applying LLMs in novel task settings such as {explaining complex codes, generating new recipes, simplifying contents,} etc. In this paper, we introduce another novel task called {functionality extraction from Git {README} files} -- a variant of {text summarization} task \cite{prana2019categorizing} that detects all the functionalities supported by the corresponding application software. This task can also be seen as a variation of a Question-Answering (QA) \cite{fan2019eli5, soares2020literature} task where the question like List all functionalities is fixed. &#10;&#10;The motivation to introduce {automatic functionality extraction from Git {README} files} stems from the requirement of application code refactoring to decompose a monolith application into functional microservices. Here each microservice is a collection of closely connected application artifacts (programs, tables etc.) supporting a common functionality \cite{fowler, richardson2018microservices, newman2021building}. Current microservice recommendation systems rely a lot on subject matter experts (SMEs) and falls short to correctly group artefacts since they do not have reference list of functionalities. But many application Git {README} files tend to contain capture {different functionalities of the underlying software code base} along with other implementation details like {what it does, how others can use it, licensing, etc.,}\cite{prana2019categorizing, chen2021evaluating}. As an example, the {README} file of the Daytrader application discusses {the application overview, the technology used, licensing terms,} etc., and in between discusses {four functionalities} as highlighted in Figure \ref{fig:desc_func}(a).&#10;&#10;Recently, \cite{doan2023too} focused on leveraging LLM to generate sections of README.md like &quot;About&quot; section(brief 1-2 line summary of repo) but they do not aim to list all the functionalities. Extraction of the application functionalities from such {README} files is not straightforward. The functionalities may not be always structured and might spread across multiple paragraphs and lines. Therefore, there is a need for an intelligent system that can parse the text, understand functionality expressions, de-duplicate, and list them. To tackle this first-of-its-kind task, we also introduce and release a new dataset called {FuncRead} that will help the community to benchmark their functionality understanding module and refactor monolith applications into discovered functional microservices. The key contributions of this paper are as follows.&#10;\begin{enumerate}[leftmargin=*,noitemsep]&#10;\item We introduce a novel {functionality extraction from Git {README} files} task and human-annotated dataset called {FuncRead}. This dataset captures the human-annotated lists of the functionalities in both extractive and abstractive forms for each of $2101$ different GitHub {README} files following permissible licenses. &#10;\item We perform a comparative analysis of generative models to reason out the gap in performance between different baselines on the {FuncRead} dataset. To enable comparison, we perform bipartite matching (one-to-one, many-to-one, and weighted many-to-one) to align generated functionalities with the gold functionalities.&#10;\item We present smaller fine-tuned generative models 1\&amp;7 billion StarCoderbase, 2.7 billion phi-2, 7 billion Llama-2 \&amp; CodeLlama which give superior results compared to ChatGPT and Bard.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Code Generation Models, Computer Science, Large Language Models, Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1802.06997" label="1802.06997">
        <attvalues>
          <attvalue for="0" value="Categorizing the Content of GitHub README Files" />
          <attvalue for="1" value="  README files play an essential role in shaping a developer's first impression&#10;of a software repository and in documenting the software project that the&#10;repository hosts. Yet, we lack a systematic understanding of the content of a&#10;typical README file as well as tools that can process these files&#10;automatically. To close this gap, we conduct a qualitative study involving the&#10;manual annotation of 4,226 README file sections from 393 randomly sampled&#10;GitHub repositories and we design and evaluate a classifier and a set of&#10;features that can categorize these sections automatically. We find that&#10;information discussing the `What' and `How' of a repository is very common,&#10;while many README files lack information regarding the purpose and status of a&#10;repository. Our multi-label classifier which can predict eight different&#10;categories achieves an F1 score of 0.746. To evaluate the usefulness of the&#10;classification, we used the automatically determined classes to label sections&#10;in GitHub README files using badges and showed files with and without these&#10;badges to twenty software professionals. The majority of participants perceived&#10;the automated labeling of sections based on our classifier to ease information&#10;discovery. This work enables the owners of software repositories to improve the&#10;quality of their documentation and it has the potential to make it easier for&#10;the software development community to discover relevant information in GitHub&#10;README files.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.09190" label="1907.09190">
        <attvalues>
          <attvalue for="0" value="ELI5: Long Form Question Answering" />
          <attvalue for="1" value="  We introduce the first large-scale corpus for long-form question answering, a&#10;task requiring elaborate and in-depth answers to open-ended questions. The&#10;dataset comprises 270K threads from the Reddit forum ``Explain Like I'm Five''&#10;(ELI5) where an online community provides answers to questions which are&#10;comprehensible by five year olds. Compared to existing datasets, ELI5 comprises&#10;diverse questions requiring multi-sentence answers. We provide a large set of&#10;web documents to help answer the question. Automatic and human evaluations show&#10;that an abstractive model trained with a multi-task objective outperforms&#10;conventional Seq2Seq, language modeling, as well as a strong extractive&#10;baseline. However, our best model is still far from human performance since&#10;raters prefer gold responses in over 86% of cases, leaving ample opportunity&#10;for future improvement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.05968" label="2406.05968">
        <attvalues>
          <attvalue for="0" value="Prompting Large Language Models with Audio for General-Purpose Speech&#10;  Summarization" />
          <attvalue for="1" value="  In this work, we introduce a framework for speech summarization that&#10;leverages the processing and reasoning capabilities of large language models&#10;(LLMs). We propose an end-to-end system that combines an instruction-tuned LLM&#10;with an audio encoder that converts speech into token representations that the&#10;LLM can interpret. Using a dataset with paired speech-text data, the overall&#10;system is trained to generate consistent responses to prompts with the same&#10;semantic information regardless of the input modality. The resulting framework&#10;allows the LLM to process speech inputs in the same way as text, enabling&#10;speech summarization by simply prompting the LLM. Unlike prior approaches, our&#10;method is able to summarize spoken content from any arbitrary domain, and it&#10;can produce summaries in different styles by varying the LLM prompting&#10;strategy. Experiments demonstrate that our approach outperforms a cascade&#10;baseline of speech recognition followed by LLM text processing.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs)~\cite{achiam2023gpt, anil2023palm, touvron2023llama} have demonstrated remarkable capabilities in a wide variety of natural language tasks, such as question answering, machine translation, language understanding, and text summarization.&#10;Notably, they exhibit the ability to respond to prompts of tasks that they were not explicitly trained on without any further gradient updates to their parameters.&#10;These characteristics of LLMs, combined with instruction tuning to align their behavior with human preferences~\cite{wei2022finetuned, ouyang2022training, chung2022scaling}, have allowed users to seamlessly interact with them via text-based prompting.&#10;&#10;In this work, we aim to utilize the processing and reasoning capabilities of LLMs for speech-to-text summarization, the task of condensing the linguistic content in a speech signal into a text summary.&#10;We build on recent work that extends the capabilities of LLMs to other modalities, in particular speech and audio.&#10;Speech summarization has traditionally been done using a cascade of automatic speech recognition (ASR) and text summarization~\cite{zhu2020hierarchical, zhang2021exploratory}.&#10;Recently, some works have proposed systems for end-to-end speech summarization~\cite{sharma2022end, matsuura2023leveraging, matsuura23transfer}, which aim to reduce errors caused by the propagation of mistakes in the ASR step.&#10;However, prior methods are limited by their training data; models trained on a specific dataset are restricted to producing summaries in that domain and in the style of that dataset's annotations.&#10;Meanwhile, LLMs are able to summarize text from arbitrary domains and in different styles by varying their prompts; we would like to leverage these abilities to develop a system that can perform general-purpose speech summarization.&#10;&#10;To this end, we propose an end-to-end pipeline that combines a text LLM with an audio encoder that allows the LLM to understand and process speech inputs in the same way as text.&#10;Our model is trained using the concept of modality invariance---the idea that, given certain semantic information in a prompt, the LLM should provide the same response regardless of the prompt's modality~\cite{fathullah2023towards}.&#10;Specifically, we use an ASR dataset with paired speech-text data; while keeping the LLM weights frozen, we train the audio encoder to convert speech inputs into token representations that the LLM can interpret.&#10;Then, the end-to-end system is guided to produce the same output as when text is the input using next-token prediction loss.&#10;We additionally incorporate knowledge distillation using the response from the corresponding text input as the teacher model, utilizing feature and logit distillation losses to guide the model to produce more consistent responses from speech inputs.&#10;&#10;Under this framework, every capability of the LLM with text also becomes possible with speech, and we can perform speech summarization by simply prompting the LLM to do so.&#10;Because our method takes advantage of the innate abilities of LLMs, it can be used for summarizing spoken content or documents from any arbitrary domain.&#10;Furthermore, it is able to generate summaries in different styles by varying the prompting strategy to the LLM.&#10;Experiments demonstrate that our proposed system outperforms a baseline cascade of ASR and LLM-based text processing in terms of response language modeling and speech summarization on the CNN / DailyMail dataset~\cite{nallapati2016abstractive}.&#10;&#10;In summary, the contributions of this work are as follows: 1) We propose a framework that allows an LLM to understand and process any speech input, and leverage its capabilities to perform speech summarization. 2) Our method can be used to summarize spoken content from any arbitrary domain. 3) We demonstrate the system's ability to produce summaries in different styles by adjusting the prompt to the LLM.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Speech Summarization, Audio Processing, Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2004.02016" label="2004.02016">
        <attvalues>
          <attvalue for="0" value="A Hierarchical Network for Abstractive Meeting Summarization with&#10;  Cross-Domain Pretraining" />
          <attvalue for="1" value="  With the abundance of automatic meeting transcripts, meeting summarization is&#10;of great interest to both participants and other parties. Traditional methods&#10;of summarizing meetings depend on complex multi-step pipelines that make joint&#10;optimization intractable. Meanwhile, there are a handful of deep neural models&#10;for text summarization and dialogue systems. However, the semantic structure&#10;and styles of meeting transcripts are quite different from articles and&#10;conversations. In this paper, we propose a novel abstractive summary network&#10;that adapts to the meeting scenario. We design a hierarchical structure to&#10;accommodate long meeting transcripts and a role vector to depict the difference&#10;among speakers. Furthermore, due to the inadequacy of meeting summary data, we&#10;pretrain the model on large-scale news summary data. Empirical results show&#10;that our model outperforms previous approaches in both automatic metrics and&#10;human evaluation. For example, on ICSI dataset, the ROUGE-1 score increases&#10;from 34.66% to 46.28%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1602.06023" label="1602.06023">
        <attvalues>
          <attvalue for="0" value="Abstractive Text Summarization Using Sequence-to-Sequence RNNs and&#10;  Beyond" />
          <attvalue for="1" value="  In this work, we model abstractive text summarization using Attentional&#10;Encoder-Decoder Recurrent Neural Networks, and show that they achieve&#10;state-of-the-art performance on two different corpora. We propose several novel&#10;models that address critical problems in summarization that are not adequately&#10;modeled by the basic architecture, such as modeling key-words, capturing the&#10;hierarchy of sentence-to-word structure, and emitting words that are rare or&#10;unseen at training time. Our work shows that many of our proposed models&#10;contribute to further improvement in performance. We also propose a new dataset&#10;consisting of multi-sentence summaries, and establish performance benchmarks&#10;for further research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.02212" label="2103.02212">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding&#10;  Transformation" />
          <attvalue for="1" value="  Linear embedding transformation has been shown to be effective for zero-shot&#10;cross-lingual transfer tasks and achieve surprisingly promising results.&#10;However, cross-lingual embedding space mapping is usually studied in static&#10;word-level embeddings, where a space transformation is derived by aligning&#10;representations of translation pairs that are referred from dictionaries. We&#10;move further from this line and investigate a contextual embedding alignment&#10;approach which is sense-level and dictionary-free. To enhance the quality of&#10;the mapping, we also provide a deep view of properties of contextual&#10;embeddings, i.e., anisotropy problem and its solution. Experiments on zero-shot&#10;dependency parsing through the concept-shared space built by our embedding&#10;transformation substantially outperform state-of-the-art methods using&#10;multilingual embeddings.&#10;" />
          <attvalue for="2" value="&#10;Cross-lingual embedding space alignment \cite{mikolov2013exploiting,artetxe-etal-2016-learning,xing-etal-2015-normalized,lample2018word} recently has been attracted a lot of attention because cross-lingual model transfer is effectively facilitated by shared semantic spaces in NLP tasks, e.g., named entity recognition \cite{xie-etal-2018-neural}, part-of-speech tagging \cite{hsu-etal-2019-zero}, and dependency parsing \cite{schuster-etal-2019-cross}, where dependency paring is scoped out in this paper. Compared with the delexicalized parsers \cite{mcdonald-etal-2011-multi}, multilingual word embeddings have been demonstrated to significantly improve the performance of zero-shot dependency parsing by bridging the lexical feature gap \cite{guo-etal-2015-cross}.&#10;&#10;With the remarkable development of monolingual contextual pre-trained models \cite{peters-etal-2018-deep,devlin-etal-2019-bert,radford2019language}, which dramatically outperform static word embeddings \cite{mikolov2013efficient,pennington2014glove,bojanowski-etal-2017-enriching} in broad NLP applications, increasing number of researchers have started focusing on contextual representation alignment for cross-lingual dependency parsing \cite{schuster-etal-2019-cross,wang2019cross}. Moreover, with the appearance of multilingual pre-trained models, such as Multilingual BERT (mBERT) \cite{devlin-etal-2019-bert}, zero-shot dependency parsing becomes easier by utilizing the large vocabulary of the multilingual models \cite{kondratyuk201975}.&#10;&#10;Our approach is most similar to \cite{schuster-etal-2019-cross}, which maps a target language space into a source language space through a linear transformation to realize zero-shot transfer in dependency parsing. Typically, a transformation is usually derived by word-level embedding alignment, while we explore a sense-level embedding alignment method to map bilingual spaces more precisely, where sense-level representations are split from multi-sense word-level embeddings. Furthermore, our mapping approach is dictionary-free which utilizes the silver token pairs from parallel corpora and eliminates the necessity of gold dictionaries. The experimental results of zero-shot dependency parsing demonstrate that two parser evaluation scores (UAS and LAS) of sense-level mapping are always better than of word-level one. Moreover, we also notice the anisotropy problem \cite{ethayarajh-2019-contextual} (defined in Section \ref{subsec:ansiotropy}) in contextual embeddings, which potentially deteriorate the performance of the zero-shot transfer task. We significantly mitigate this drawback by leveraging a prepossessing step, iterative normalization (IN) \cite{zhang-etal-2019-girls}, which is originally used for improving the performance of static embedding mapping on the bilingual dictionary induction task.&#10;&#10;Zero-shot dependency parsing experiments are conducted on Universal Dependencies treebank v2.6 \cite{11234/1-3226}, which shows that our results obtain a substantial gain compared with state-of-the-art methods using multilingual fastText and mBERT.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Multilingual Dependency Parsing, Machine Learning, Computer Science, Linguistics, Contextual Embedding Alignment, Mathematics, Natural Language Processing, Cross-Lingual Embeddings" />
        </attvalues>
      </node>
      <node id="1808.09861" label="1808.09861">
        <attvalues>
          <attvalue for="0" value="Neural Cross-Lingual Named Entity Recognition with Minimal Resources" />
          <attvalue for="1" value="  For languages with no annotated resources, unsupervised transfer of natural&#10;language processing models such as named-entity recognition (NER) from&#10;resource-rich languages would be an appealing capability. However, differences&#10;in words and word order across languages make it a challenging problem. To&#10;improve mapping of lexical items across languages, we propose a method that&#10;finds translations based on bilingual word embeddings. To improve robustness to&#10;word order differences, we propose to use self-attention, which allows for a&#10;degree of flexibility with respect to word order. We demonstrate that these&#10;methods achieve state-of-the-art or competitive NER performance on commonly&#10;tested languages under a cross-lingual setting, with much lower resource&#10;requirements than past approaches. We also evaluate the challenges of applying&#10;these methods to Uyghur, a low-resource language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.09587" label="1909.09587">
        <attvalues>
          <attvalue for="0" value="Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with&#10;  Multi-lingual Language Representation Model" />
          <attvalue for="1" value="  Because it is not feasible to collect training data for every language, there&#10;is a growing interest in cross-lingual transfer learning. In this paper, we&#10;systematically explore zero-shot cross-lingual transfer learning on reading&#10;comprehension tasks with a language representation model pre-trained on&#10;multi-lingual corpus. The experimental results show that with pre-trained&#10;language representation zero-shot learning is feasible, and translating the&#10;source data into the target language is not necessary and even degrades the&#10;performance. We further explore what does the model learn in zero-shot setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.09492" label="1902.09492">
        <attvalues>
          <attvalue for="0" value="Cross-Lingual Alignment of Contextual Word Embeddings, with Applications&#10;  to Zero-shot Dependency Parsing" />
          <attvalue for="1" value="  We introduce a novel method for multilingual transfer that utilizes deep&#10;contextual embeddings, pretrained in an unsupervised fashion. While contextual&#10;embeddings have been shown to yield richer representations of meaning compared&#10;to their static counterparts, aligning them poses a challenge due to their&#10;dynamic nature. To this end, we construct context-independent variants of the&#10;original monolingual spaces and utilize their mapping to derive an alignment&#10;for the context-dependent spaces. This mapping readily supports processing of a&#10;target language, improving transfer by context-aware embeddings. Our&#10;experimental results demonstrate the effectiveness of this approach for&#10;zero-shot and few-shot learning of dependency parsing. Specifically, our method&#10;consistently outperforms the previous state-of-the-art on 6 tested languages,&#10;yielding an improvement of 6.8 LAS points on average.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.06775" label="1909.06775">
        <attvalues>
          <attvalue for="0" value="Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing" />
          <attvalue for="1" value="  This paper investigates the problem of learning cross-lingual representations&#10;in a contextual space. We propose Cross-Lingual BERT Transformation (CLBT), a&#10;simple and efficient approach to generate cross-lingual contextualized word&#10;embeddings based on publicly available pre-trained BERT models (Devlin et al.,&#10;2018). In this approach, a linear transformation is learned from contextual&#10;word alignments to align the contextualized embeddings independently trained in&#10;different languages. We demonstrate the effectiveness of this approach on&#10;zero-shot cross-lingual transfer parsing. Experiments show that our embeddings&#10;substantially outperform the previous state-of-the-art that uses static&#10;embeddings. We further compare our approach with XLM (Lample and Conneau,&#10;2019), a recently proposed cross-lingual language model trained with massive&#10;parallel data, and achieve highly competitive results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01622" label="1906.01622">
        <attvalues>
          <attvalue for="0" value="Are Girls Neko or Sh\=ojo? Cross-Lingual Alignment of Non-Isomorphic&#10;  Embeddings with Iterative Normalization" />
          <attvalue for="1" value="  Cross-lingual word embeddings (CLWE) underlie many multilingual natural&#10;language processing systems, often through orthogonal transformations of&#10;pre-trained monolingual embeddings. However, orthogonal mapping only works on&#10;language pairs whose embeddings are naturally isomorphic. For non-isomorphic&#10;pairs, our method (Iterative Normalization) transforms monolingual embeddings&#10;to make orthogonal alignment easier by simultaneously enforcing that (1)&#10;individual word vectors are unit length, and (2) each language's average vector&#10;is zero. Iterative Normalization consistently improves word translation&#10;accuracy of three CLWE methods, with the largest improvement observed on&#10;English-Japanese (from 2% to 44% test accuracy).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07578" label="2104.07578">
        <attvalues>
          <attvalue for="0" value="Syntactic Perturbations Reveal Representational Correlates of&#10;  Hierarchical Phrase Structure in Pretrained Language Models" />
          <attvalue for="1" value="  While vector-based language representations from pretrained language models&#10;have set a new standard for many NLP tasks, there is not yet a complete&#10;accounting of their inner workings. In particular, it is not entirely clear&#10;what aspects of sentence-level syntax are captured by these representations,&#10;nor how (if at all) they are built along the stacked layers of the network. In&#10;this paper, we aim to address such questions with a general class of&#10;interventional, input perturbation-based analyses of representations from&#10;pretrained language models. Importing from computational and cognitive&#10;neuroscience the notion of representational invariance, we perform a series of&#10;probes designed to test the sensitivity of these representations to several&#10;kinds of structure in sentences. Each probe involves swapping words in a&#10;sentence and comparing the representations from perturbed sentences against the&#10;original. We experiment with three different perturbations: (1) random&#10;permutations of n-grams of varying width, to test the scale at which a&#10;representation is sensitive to word position; (2) swapping of two spans which&#10;do or do not form a syntactic phrase, to test sensitivity to global phrase&#10;structure; and (3) swapping of two adjacent words which do or do not break&#10;apart a syntactic phrase, to test sensitivity to local phrase structure.&#10;  Results from these probes collectively suggest that Transformers build&#10;sensitivity to larger parts of the sentence along their layers, and that&#10;hierarchical phrase structure plays a role in this process. More broadly, our&#10;results also indicate that structured input perturbations widens the scope of&#10;analyses that can be performed on often-opaque deep learning systems, and can&#10;serve as a complement to existing tools (such as supervised linear probes) for&#10;interpreting complex black-box models.&#10;" />
          <attvalue for="2" value="&#10;It is still unknown how distributed information processing systems encode and exploit complex relational structures in data, despite their ubiquitous use in the modern world. The fields of deep learning~\cite{saxe2013learning,stanford_nlp}, neuroscience~\cite{sarafyazd2019hierarchical, stachenfeld2017hippocampus}, and cognitive science~\cite{elman1991distributed, kemp2008discovery, tervo2016toward} have given great attention to this question, including a productive focus on the potential models and their implementations of hierarchical tasks, such as predictive maps and graphs. In this work, we provide a generic means of identifying input structures that deep language models use to ``chunk up'' vastly complex data.&#10;&#10;Natural (human) language provides a rich domain for studying how complex hierarchical structures are encoded in information processing systems. More so than other domains, human language is unique in that its underlying hierarchy has been extensively studied and theorized in linguistics, which provides source of ``ground truth'' structures for stimulus data. Much prior work on characterizing the types of linguistic information encoded in computational models of language such as neural networks has focused on supervised readout probes, which train a classifier on top pretrained models to predict a particular linguistic label \cite{belinkov2017analyzing,liu_et_al,tenney2019bert}. In particular, \cite{stanford_nlp} apply probes to discover linear subspaces that encode tree-distances as distances in the representational subspace, and \cite{kim2019pre} show that these distances can be used even without any labeled information to induce hierarchical structure. However, recent work has highlighted issues with correlating supervised probe performance with the amount of language structure encoded in such representations \cite{hewitt2019}. Another popular approach to analyzing deep models is through the lens of geometry \cite{reif2019visualizing,gigante2019visualizing}. While geometric interpretations provide significant insights, they present another challenge in summarizing the structure in a quantifiable way. More recent techniques such as replica-based mean field manifold analysis method \cite{chung2018classification, cohen2019separability, mamou2020emergence} connects representation geometry with linear classification performance, but the method is limited to categorization tasks. &#10;&#10;In this work, we make use of an experimental framework from cognitive science and neuroscience to probe for hierarchical structure in contextual representations from pretrained Transformer models (i.e., BERT~\cite{bert} and its variants). A popular technique in neuroscience involves measuring change in the population activity in response to controlled, input perturbations~\cite{mollica2020composition, ding2016cortical}. We apply this approach to test the characteristic scale and the complexity (Fig.~\ref{fig:fig1}) of hierarchical phrase structure encoded deep contextual representations, and present several key findings:&#10;&#10;\begin{enumerate}&#10; \item Representations are distorted by shuffling small $n$-grams in early layers, while the distortion caused by shuffling large $n$-grams starts to occur in later layers, implying the scale of characteristic word length increases from input to downstream layers. &#10; \item Representational distortion caused by swapping two constituent phrases is smaller than when the control sequences of the same length are swapped, indicating that the BERT representations are sensitive to hierarchical phrase structure.&#10; \item Representational distortion caused by swapping adjacent words across phrasal boundary is larger than when the swap is within a phrasal boundary; furthermore, the amount of distortion increases with the syntactic distance between the swapped words. The correlation between distortion and tree distance increases across the layers, suggesting that the characteristic complexity of phrasal subtrees increases across the layers.&#10; \item Early layers pay more attention between syntactically closer adjacent pairs and deeper layers pay more attention between syntactically distant adjacent pairs. The attention paid in each layer can explain some of the emergent sensitivity to phrasal structure across layers.&#10;\end{enumerate}&#10;&#10;Our work demonstrates that interventional tools such as controlled input perturbations can be useful for analyzing deep networks, adding to the growing, interdisciplinary body of work which profitably adapt experimental techniques from cognitive neuroscience and psycholinguistics to analyze computational models of language~\cite{futrell2018rnn,wilcox2019hierarchical,futrell-etal-2019-neural,ettinger2020bert}.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Sentence Syntax Representation, Computational Neuroscience, Artificial Intelligence, Natural Language Processing, Deep Learning Interpretability" />
        </attvalues>
      </node>
      <node id="1709.04482" label="1709.04482">
        <attvalues>
          <attvalue for="0" value="Analyzing Hidden Representations in End-to-End Automatic Speech&#10;  Recognition Systems" />
          <attvalue for="1" value="  Neural models have become ubiquitous in automatic speech recognition systems.&#10;While neural networks are typically used as acoustic models in more complex&#10;systems, recent studies have explored end-to-end speech recognition systems&#10;based on neural networks, which can be trained to directly predict text from&#10;input acoustic features. Although such systems are conceptually elegant and&#10;simpler than traditional systems, it is less obvious how to interpret the&#10;trained models. In this work, we analyze the speech representations learned by&#10;a deep end-to-end model that is based on convolutional and recurrent layers,&#10;and trained with a connectionist temporal classification (CTC) loss. We use a&#10;pre-trained model to generate frame-level features which are given to a&#10;classifier that is trained on frame classification into phones. We evaluate&#10;representations from different layers of the deep model and compare their&#10;quality for predicting phone labels. Our experiments shed light on important&#10;aspects of the end-to-end model such as layer depth, model complexity, and&#10;other design choices.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.00737" label="2002.00737">
        <attvalues>
          <attvalue for="0" value="Are Pre-trained Language Models Aware of Phrases? Simple but Strong&#10;  Baselines for Grammar Induction" />
          <attvalue for="1" value="  With the recent success and popularity of pre-trained language models (LMs)&#10;in natural language processing, there has been a rise in efforts to understand&#10;their inner workings. In line with such interest, we propose a novel method&#10;that assists us in investigating the extent to which pre-trained LMs capture&#10;the syntactic notion of constituency. Our method provides an effective way of&#10;extracting constituency trees from the pre-trained LMs without training. In&#10;addition, we report intriguing findings in the induced trees, including the&#10;fact that pre-trained LMs outperform other approaches in correctly demarcating&#10;adverb phrases in sentences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.02831" label="1908.02831">
        <attvalues>
          <attvalue for="0" value="Visualizing the PHATE of Neural Networks" />
          <attvalue for="1" value="  Understanding why and how certain neural networks outperform others is key to&#10;guiding future development of network architectures and optimization methods.&#10;To this end, we introduce a novel visualization algorithm that reveals the&#10;internal geometry of such networks: Multislice PHATE (M-PHATE), the first&#10;method designed explicitly to visualize how a neural network's hidden&#10;representations of data evolve throughout the course of training. We&#10;demonstrate that our visualization provides intuitive, detailed summaries of&#10;the learning dynamics beyond simple global measures (i.e., validation loss and&#10;accuracy), without the need to access validation data. Furthermore, M-PHATE&#10;better captures both the dynamics and community structure of the hidden units&#10;as compared to visualization based on standard dimensionality reduction methods&#10;(e.g., ISOMAP, t-SNE). We demonstrate M-PHATE with two vignettes: continual&#10;learning and generalization. In the former, the M-PHATE visualizations display&#10;the mechanism of &quot;catastrophic forgetting&quot; which is a major challenge for&#10;learning in task-switching contexts. In the latter, our visualizations reveal&#10;how increased heterogeneity among hidden units correlates with improved&#10;generalization performance. An implementation of M-PHATE, along with scripts to&#10;reproduce the figures in this paper, is available at&#10;https://github.com/scottgigante/M-PHATE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.06487" label="1710.06487">
        <attvalues>
          <attvalue for="0" value="Classification and Geometry of General Perceptual Manifolds" />
          <attvalue for="1" value="  Perceptual manifolds arise when a neural population responds to an ensemble&#10;of sensory signals associated with different physical features (e.g.,&#10;orientation, pose, scale, location, and intensity) of the same perceptual&#10;object. Object recognition and discrimination requires classifying the&#10;manifolds in a manner that is insensitive to variability within a manifold. How&#10;neuronal systems give rise to invariant object classification and recognition&#10;is a fundamental problem in brain theory as well as in machine learning. Here&#10;we study the ability of a readout network to classify objects from their&#10;perceptual manifold representations. We develop a statistical mechanical theory&#10;for the linear classification of manifolds with arbitrary geometry revealing a&#10;remarkable relation to the mathematics of conic decomposition. Novel&#10;geometrical measures of manifold radius and manifold dimension are introduced&#10;which can explain the classification capacity for manifolds of various&#10;geometries. The general theory is demonstrated on a number of representative&#10;manifolds, including L2 ellipsoids prototypical of strictly convex manifolds,&#10;L1 balls representing polytopes consisting of finite sample points, and&#10;orientation manifolds which arise from neurons tuned to respond to a continuous&#10;angle variable, such as object orientation. The effects of label sparsity on&#10;the classification capacity of manifolds are elucidated, revealing a scaling&#10;relation between label sparsity and manifold radius. Theoretical predictions&#10;are corroborated by numerical simulations using recently developed algorithms&#10;to compute maximum margin solutions for manifold dichotomies. Our theory and&#10;its extensions provide a powerful and rich framework for applying statistical&#10;mechanics of linear classification to data arising from neuronal responses to&#10;object stimuli, as well as to artificial deep networks trained for object&#10;recognition tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.03260" label="1903.03260">
        <attvalues>
          <attvalue for="0" value="Neural Language Models as Psycholinguistic Subjects: Representations of&#10;  Syntactic State" />
          <attvalue for="1" value="  We deploy the methods of controlled psycholinguistic experimentation to shed&#10;light on the extent to which the behavior of neural network language models&#10;reflects incremental representations of syntactic state. To do so, we examine&#10;model behavior on artificial sentences containing a variety of syntactically&#10;complex structures. We test four models: two publicly available LSTM sequence&#10;models of English (Jozefowicz et al., 2016; Gulordava et al., 2018) trained on&#10;large datasets; an RNNG (Dyer et al., 2016) trained on a small, parsed dataset;&#10;and an LSTM trained on the same small corpus as the RNNG. We find evidence that&#10;the LSTMs trained on large datasets represent syntactic state over large spans&#10;of text in a way that is comparable to the RNNG, while the LSTM trained on the&#10;small dataset does not or does so only weakly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.13528" label="1907.13528">
        <attvalues>
          <attvalue for="0" value="What BERT is not: Lessons from a new suite of psycholinguistic&#10;  diagnostics for language models" />
          <attvalue for="1" value="  Pre-training by language modeling has become a popular and successful&#10;approach to NLP tasks, but we have yet to understand exactly what linguistic&#10;capacities these pre-training processes confer upon models. In this paper we&#10;introduce a suite of diagnostics drawn from human language experiments, which&#10;allow us to ask targeted questions about the information used by language&#10;models for generating predictions in context. As a case study, we apply these&#10;diagnostics to the popular BERT model, finding that it can generally&#10;distinguish good from bad completions involving shared category or role&#10;reversal, albeit with less sensitivity than humans, and it robustly retrieves&#10;noun hypernyms, but it struggles with challenging inferences and role-based&#10;event prediction -- and in particular, it shows clear insensitivity to the&#10;contextual impacts of negation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03429" label="1911.03429">
        <attvalues>
          <attvalue for="0" value="ERASER: A Benchmark to Evaluate Rationalized NLP Models" />
          <attvalue for="1" value="  State-of-the-art models in NLP are now predominantly based on deep neural&#10;networks that are opaque in terms of how they come to make predictions. This&#10;limitation has increased interest in designing more interpretable deep models&#10;for NLP that reveal the `reasoning' behind model outputs. But work in this&#10;direction has been conducted on different datasets and tasks with&#10;correspondingly unique aims and metrics; this makes it difficult to track&#10;progress. We propose the Evaluating Rationales And Simple English Reasoning&#10;(ERASER) benchmark to advance research on interpretable models in NLP. This&#10;benchmark comprises multiple datasets and tasks for which human annotations of&#10;&quot;rationales&quot; (supporting evidence) have been collected. We propose several&#10;metrics that aim to capture how well the rationales provided by models align&#10;with human rationales, and also how faithful these rationales are (i.e., the&#10;degree to which provided rationales influenced the corresponding predictions).&#10;Our hope is that releasing this benchmark facilitates progress on designing&#10;more interpretable NLP systems. The benchmark, code, and documentation are&#10;available at https://www.eraserbenchmark.com/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;Interpretability in NLP is a large, fast-growing area; we do not attempt to provide a comprehensive overview here. Instead we focus on directions particularly relevant to ERASER, i.e., prior work on models that provide rationales for their predictions.&#10;&#10;{Learning to explain}. In ERASER we assume that rationales (marked by humans) are provided during training. However, such direct supervision will not always be available, motivating work on methods that can explain (or ``rationalize'') model predictions using only instance-level supervision.&#10;&#10;In the context of modern neural models for text classification, one might use variants of attention \cite{bahdanau2014neural} to extract rationales. Attention mechanisms learn to assign soft weights to (usually contextualized) token representations, and so one can extract highly weighted tokens as rationales. However, attention weights do not in general provide faithful explanations for predictions \cite{jain2019attention,serrano2019attention,wiegreffe2019attention,zhong2019fine,pruthi2019learning,brunner2019validity,moradi2019interrogating,vashishth2019attention}. This likely owes to encoders entangling inputs, complicating the interpretation of attention weights on inputs over contextualized representations of the same.&#10;&#10;By contrast, hard attention mechanisms discretely extract snippets from the input to pass to the classifier, by construction providing faithful explanations. Recent work has proposed hard attention mechanisms as a means of providing explanations. \cite{lei2016rationalizing} proposed instantiating two models with their own parameters; one to extract rationales, and one that consumes these to make a prediction. They trained these models jointly via REINFORCE \cite{williams1992simple} style optimization. &#10;&#10;Recently, \cite{jain2020} proposed a variant of this two-model setup that uses heuristic feature scores to derive pseudo-labels on tokens comprising rationales; one model can then be used to perform hard extraction in this way, while a second (independent) model can make predictions on the basis of these. Elsewhere, ~\cite{chang2019game} introduced the notion of classwise rationales that explains support for different output classes using a game theoretic framework. Finally, other recent work has proposed using a differentiable binary mask over inputs, which also avoids recourse to REINFORCE \cite{bastings-etal-2019-interpretable}.&#10;&#10;{Post-hoc explanation}. Another strand of interpretability work considers post-hoc explanation methods, which seek to explain why a model made a specific prediction for a given input. Commonly these take the form of token-level importance scores. Gradient-based explanations are a standard example \cite{sundararajan2017axiomatic,smilkov2017smoothgrad}. These enjoy a clear semantics (describing how perturbing inputs locally affects outputs), but may nonetheless exhibit counterintuitive behaviors \cite{feng2018pathologies}. &#10;&#10;Gradients of course assume model differentiability. Other methods do not require any model properties. Examples include LIME~\cite{ribeiro2016should} and ~\cite{alvarez2017causal}; these methods approximate model behavior locally by having it repeatedly make predictions over perturbed inputs and fitting a simple, explainable model over the outputs.&#10;&#10;{Acquiring rationales}. Aside from interpretability considerations, collecting rationales from annotators may afford greater efficiency in terms of model performance realized given a fixed amount of annotator effort \cite{zaidan2008modeling}. In particular, recent work by ~\cite{mcdonnell2017many,mcdonnell2016relevant} has observed that at least for some tasks, asking annotators to provide rationales justifying their categorizations does not impose much additional effort. Combining rationale annotation with active learning \cite{settles2012active} is another promising direction \cite{wallace2010active,sharma2015active}.&#10;&#10;{Learning from rationales}. Work on learning from rationales marked by annotators for text classification dates back over a decade~\cite{zaidan2007using}. &#10;Earlier efforts proposed extending standard discriminative models like Support Vector Machines (SVMs) with regularization terms that penalized parameter estimates which disagreed with provided rationales \cite{zaidan2007using,small2011constrained}.&#10;Other efforts have attempted to specify generative models of rationales~\cite{zaidan2008modeling}. &#10;&#10;More recent work has aimed to exploit rationales in training neural text classifiers. \cite{zhang2016rationale} proposed a rationale-augmented Convolutional Neural Network (CNN) for text classification, explicitly trained to identify sentences supporting categorizations.~\cite{strout2019human} showed that providing this model with rationales during training yields predicted rationales that are preferred by humans (compared to rationales produced without explicit supervision). &#10;Other work has proposed `pipeline' approaches in which independent models are trained to perform rationale extraction and classification on the basis of these, respectively \cite{lehman2019inferring,chen-etal-2019-seeing}, assuming explicit training data is available for the former.&#10;&#10;\cite{rajani2019explain} fine-tuned a Transformer-based language model \cite{radford2018improving} on free-text rationales provided by humans, with an objective of generating open-ended explanations to improve performance on downstream tasks. &#10;&#10;{Evaluating rationales}. Work on evaluating rationales has often compared these to human judgments \cite{strout2019human,doshi2017towards}, or elicited other human evaluations of explanations \cite{ribeiro2016should, lundberg2017unified, nguyen2018comparing}. There has also been work on visual evaluations of saliency maps~\cite{li-etal-2016-visualizing, ding-etal-2017-visualizing, sundararajan2017axiomatic}. &#10;&#10;Measuring agreement between extracted and human rationales (or collecting subjective assessments of them) assesses the plausibility of rationales, but such approaches do not establish whether the model actually relied on these particular rationales to make a prediction. &#10;We refer to rationales that correspond to the inputs most relied upon to come to a disposition as faithful.&#10;&#10;Most automatic evaluations of faithfulness measure the impact of perturbing or erasing words or tokens identified as important on model output \cite{Arras2017WhatIR, montavon2017explaining, serrano2019attention, samek2016evaluating,jain2019attention}. We build upon these methods in Section \ref{section:metrics}. Finally, we note that a recent article urges the community to evaluate faithfulness on a continuous scale of acceptability, rather than viewing this as a binary proposition \cite{jacovi}.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Interpretable NLP Models, Deep Neural Networks, Benchmark Evaluation Metrics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2204.08426" label="2204.08426">
        <attvalues>
          <attvalue for="0" value="CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement&#10;  Learning" />
          <attvalue for="1" value="  Conventionally, generation of natural language for dialogue agents may be&#10;viewed as a statistical learning problem: determine the patterns in&#10;human-provided data and generate appropriate responses with similar statistical&#10;properties. However, dialogue can also be regarded as a goal directed process,&#10;where speakers attempt to accomplish a specific task. Reinforcement learning&#10;(RL) algorithms are designed specifically for solving such goal-directed&#10;problems, but the most direct way to apply RL -- through trial-and-error&#10;learning in human conversations, -- is costly. In this paper, we study how&#10;offline reinforcement learning can instead be used to train dialogue agents&#10;entirely using static datasets collected from human speakers. Our experiments&#10;show that recently developed offline RL methods can be combined with language&#10;models to yield realistic dialogue agents that better accomplish task goals.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Recent developments in deep learning have led to end-to-end approaches to dialogue using supervised learning, such as sequence-to-sequence models~\cite{duvsek2016sequence,eric2017copy}, hierarchical models~\cite{serban2016multiresolution}, attention~\cite{mei2017coherent,chen2019semantically}, and Transformer-based models~\cite{wu2019alternating,hosseini2020simple,peng2020soloist,adiwardana2020towards}. However, supervised learning only allows an agent to imitate behaviors, requires optimal data, and does not allow agents to exceed human performance. Supervised learning for dialogue generation also has well-known issues such as outputting commonplace responses (e.g., I do not know) regardless of the inputs~\cite{li2015diversity}. Therefore, additional training of the dialogue agent is required for performing goal-oriented tasks.&#10;&#10;Task-oriented dialogue has been formulated as a sequential decision making problem in a Markov Decision Process (MDP) since the 1990s~\cite{smith1994spoken,singh1999reinforcement,williams2007partially,young2013pomdp,paek2006reinforcement,henderson2008hybrid,gao2018neural,pieraccini2009we,young2013pomdp,su2015learning,chen2020airconcierge}. Dialogue is converted into abstract states and actions from which an agent is trained using reinforcement learning (RL)~\cite{eckert1997user,levin2000stochastic,chung2004developing,georgila2006user,schatzmann2007agenda,heeman2009representing,georgila2011reinforcement,su2016continuously,fatemi2016policy,asri2016sequence,zhao2019rethinking,zhang2020task,wang2020modelling}. These methods differ in how the abstract states/actions are designed and whether the simulated environment for training the policy is hand created, learned as a fixed model, or is an agent itself. For instance, \cite{eckert1997user,levin2000stochastic} learn a fixed transition model from human conversations and~\cite{georgila2011reinforcement} learn negotiation agents where each agent is the user simulator for the other agent. These methods also differ in how the decision making policy is trained, e.g., online~\cite{gavsic2011online} or off-policy/offline~\cite{yu2016strategy,pietquin2011sample} using actor-critic~\cite{su2017sample}, policy gradient~\cite{he2018decoupling}, or fitted Q-iteration~\cite{pietquin2011sample}. Regardless of the RL method used, since policies are trained on abstract states and actions, these methods lack the ability to generate natural language (i.e., response is created via templates depending on an abstract action). &#10;&#10;To overcome these limitations, recent work has trained policies directly on text, using a recurrent neural network to output language tokens, and using self-play for policy training while interacting with another learned agent~\cite{li2016deep,lewis2017deal,liu2018dialogue}.&#10;To further improve the generated language quality, hierarchical methods decouple the strategic high-level dialogue decisions from generation~\cite{yarats2018hierarchical,he2018decoupling,saleh2019hierarchical}. These model-based approaches require accurate estimation of the environment/human (e.g., the trained self-play agent needs to mimic complex human behavior), which is beyond current capability of model-based reinforcement learning algorithms. Similar to our proposal, \cite{jaques2019way} use offline RL based on KL-control for text generation in open-domain dialogue. Our work differs in that our model is able to utilize large amounts of unsupervised data through the use of pre-trained language models, and that our work focuses on task-oriented (as opposed to open-domain) dialogue tasks. Goal-oriented tasks have clearly defined objectives that can be quantified, allowing us to provide an objective comparison between our method and prior approaches.&#10;&#10;" />
          <attvalue for="4" value="Language Modeling, Computer Science, Linguistics, Dialogue Generation, Artificial Intelligence, Natural Language Processing, Reinforcement Learning" />
        </attvalues>
      </node>
      <node id="2206.08082" label="2206.08082">
        <attvalues>
          <attvalue for="0" value="Self-Generated In-Context Learning: Leveraging Auto-regressive Language&#10;  Models as a Demonstration Generator" />
          <attvalue for="1" value="  Large-scale pre-trained language models (PLMs) are well-known for being&#10;capable of solving a task simply by conditioning a few input-label pairs dubbed&#10;demonstrations on a prompt without being explicitly tuned for the desired&#10;downstream task. Such a process (i.e., in-context learning), however, naturally&#10;leads to high reliance on the demonstrations which are usually selected from&#10;external datasets. In this paper, we propose self-generated in-context learning&#10;(SG-ICL), which generates demonstrations for in-context learning from PLM&#10;itself to minimize the reliance on the external demonstration. We conduct&#10;experiments on four different text classification tasks and show SG-ICL&#10;significantly outperforms zero-shot learning and is generally worth&#10;approximately 0.6 gold training samples. Moreover, our generated demonstrations&#10;show more consistent performance with low variance compared to randomly&#10;selected demonstrations from the training dataset.&#10;" />
          <attvalue for="2" value="&#10;&#10;The scale of pre-trained language models (PLMs) is ever-growing as they tend to deliver more meaningful results with larger models and have reached the scale of hundreds of billions.&#10;However, transferring such large-scale PLMs with the traditional method i.e., fine-tuning, is problematic as it entails an immense cost to train and store parameters for an individual task.&#10;Numerous branches of work have been proposed to circumvent such issues, such as Adapters \cite{Houlsby2019ParameterEfficientTL}, LoRA \cite{hu2021lora}, and in-context learning (ICL) \cite{brown2020language}.&#10;&#10;Among others, ICL is in the limelight as it derives answers only from the internal knowledge of PLMs without any parameter updates.&#10;Specifically, ICL learns to solve a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt, which serves to give contexts regarding the downstream task during the inference phase, allowing PLMs to solve the tasks better.&#10;The working principle of ICL intuitively leads to high reliance on the demonstrations, and performance deeply varies depending on the assortment of the demonstrations.&#10;&#10;Many lines of work tackled the issue of ICL's high reliance on the demonstration. For instance, \cite{lu2021fantastically} shown in-context learning suffers from the order sensitivity of the demonstrations. \cite{zhao2021calibrate} introduces a contextual calibration procedure to reduce the variance across different choices of demonstrations.&#10;\cite{rubin2021learning} suggests demonstration selection by retrieving in-context samples.&#10;Notably, \cite{liu-etal-2022-makes} showed that selecting a demonstration that has a high correlation with the test input can improve performance.&#10;&#10;Motivated by previous research considering the limits of ICL's working process, we tried to solve the following research question: \\&#10;&#10;1. Can we eliminate the dependency on the training dataset by generating demonstrations? \\&#10;&#10;2. If so, how can we create demonstrations with high input-demonstration correlation?&#10;&#10;To this end, we propose a novel method termed self-generated in-context learning (SG-ICL) which generates demonstrations by leveraging the superiority of PLMs generative abilities \cite{adiwardana2020towards, brown2020language, shwartz-etal-2020-unsupervised, ye2022zerogen}.&#10;To the best of our knowledge, this is the first study to utilize PLMs to create demonstrations for ICL.&#10;SG-ICL consists of two operation steps: the self-generation step and the inference step.&#10;In the self-generation step, we generate demonstrations for each class in the downstream task by conditioning on the current test input and class information with a simple manually designed template.&#10;By giving conditions about the current input, PLM can generate demonstrations with a high input-demonstration correlation which is more befitted for ICL.&#10;Then, the inference step performs ICL with generated demonstrations from the previous step which eliminates the requirement for training data or manual selection from training data.&#10;&#10;We evaluate our method in four different natural language understanding (NLU) tasks, including sentiment classification and natural language inference. &#10;Through extensive experiments, we show that SG-ICL significantly outperforms zero-shot learning methods and is generally worth approximately 0.6 gold training samples.&#10;Moreover, our generated demonstrations show more stable performance with low variance compared to randomly selected demonstrations from training dataset.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Classification, Computer Science, Machine Learning, Linguistics, In-Context Learning, Language Model Training, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2001.09977" label="2001.09977">
        <attvalues>
          <attvalue for="0" value="Towards a Human-like Open-Domain Chatbot" />
          <attvalue for="1" value="  We present Meena, a multi-turn open-domain chatbot trained end-to-end on data&#10;mined and filtered from public domain social media conversations. This 2.6B&#10;parameter neural network is simply trained to minimize perplexity of the next&#10;token. We also propose a human evaluation metric called Sensibleness and&#10;Specificity Average (SSA), which captures key elements of a human-like&#10;multi-turn conversation. Our experiments show strong correlation between&#10;perplexity and SSA. The fact that the best perplexity end-to-end trained Meena&#10;scores high on SSA (72% on multi-turn evaluation) suggests that a human-level&#10;SSA of 86% is potentially within reach if we can better optimize perplexity.&#10;Additionally, the full version of Meena (with a filtering mechanism and tuned&#10;decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots&#10;we evaluated.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15119" label="2305.15119">
        <attvalues>
          <attvalue for="0" value="Another Dead End for Morphological Tags? Perturbed Inputs and Parsing" />
          <attvalue for="1" value="  The usefulness of part-of-speech tags for parsing has been heavily questioned&#10;due to the success of word-contextualized parsers. Yet, most studies are&#10;limited to coarse-grained tags and high quality written content; while we know&#10;little about their influence when it comes to models in production that face&#10;lexical errors. We expand these setups and design an adversarial attack to&#10;verify if the use of morphological information by parsers: (i) contributes to&#10;error propagation or (ii) if on the other hand it can play a role to correct&#10;mistakes that word-only neural parsers make. The results on 14 diverse UD&#10;treebanks show that under such attacks, for transition- and graph-based models&#10;their use contributes to degrade the performance even faster, while for the&#10;(lower-performing) sequence labeling parsers they are helpful. We also show&#10;that if morphological tags were utopically robust against lexical&#10;perturbations, they would be able to correct parsing mistakes.&#10;" />
          <attvalue for="2" value="&#10;&#10;The use of morphological tags was a core component of dependency parsers to improve performance \cite{ballesteros-nivre-2012-maltoptimizer-system}. With the rise of neural models, feeding explicit morphological information is a practice that has greatly vanished, with (often) the exception of part-of-speech (PoS) tags. In this line, \cite{ballesteros-etal-2015-improved} already found that character-based word vectors helped improving performance over purely word-level models, specially for rich-resource languages, for which the use of morphological information is more relevant \cite{dehouck-denis-2018-framework}. Related, \cite{dozat-etal-2017-stanfords} showed that predicted PoS tags still improved the performance of their graph-based parser, even when used together with character-based representations. \cite{smith-etal-2018-investigation} and \cite{de-lhoneux-etal-2017-raw} studied the impact that ignoring PoS tag vectors had on the performance of a biLSTM transition-based parser \cite{kiperwasser-goldberg-2016-simple}. They conclude that when considering PoS tags, word-level, and character-level embedddings, any two of those vectors are enough to maximize a parser performance, i.e., PoS tag vectors can be excluded when using both word-level and character-level vectors. \cite{zhou2020pos} showed the utility of PoS tags when learned jointly with parsing. Recently, \cite{anderson-gomez-rodriguez-2021-taggers} and \cite{anderson-etal-2021-falta} have explored the differences between using gold and predicted PoS tags, showing that the former are helpful to improve the results, while the latter are often not, with the exception of low-resource languages, where they obtain small but consistent improvements. Furthermore, \cite{munoz-ortiz-etal-2022-parsing} showed that the efficacy of PoS tags in the context of sequence labeling parsing is greatly influenced by the chosen linearization method.&#10;&#10;However, most of such work has focused on: (i) studying the effect of the universal PoS tags \cite{ud2.9}, and (ii) its impact on non-perturbed inputs. Yet, NLP models are very sensible and brittle against small attacks, and simple perturbations like misspellings can greatly reduce performance \cite{ebrahimi-etal-2018-hotflip,alzantot-etal-2018-generating}. This has been shown for tasks such as named-entity recognition, question answering, semantic similarity, and sentiment analysis \cite{moradi-samwald-2021-evaluating}. In parallel, defensive strategies have been tested to improve the robustness of NLP systems, e.g., placing a word recognition module before downstream classifiers \cite{pruthi-etal-2019-combating}, or using spelling checks and adversarial training \cite{li-etal-2019-textbugger}. Yet, as far as we know, no related work has been done on testing perturbed inputs for parsing and the effect, positive or negative, that using morphological information as explicit signals during inference might have in guiding the parsers.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Adversarial Error Analysis, Part-of-Speech Tagging, Natural Language Processing, Neural Parsing Models" />
        </attvalues>
      </node>
      <node id="1508.00657" label="1508.00657">
        <attvalues>
          <attvalue for="0" value="Improved Transition-Based Parsing by Modeling Characters instead of&#10;  Words with LSTMs" />
          <attvalue for="1" value="  We present extensions to a continuous-state dependency parsing method that&#10;makes it applicable to morphologically rich languages. Starting with a&#10;high-performance transition-based parser that uses long short-term memory&#10;(LSTM) recurrent neural networks to learn representations of the parser state,&#10;we replace lookup-based word representations with representations constructed&#10;from the orthographic representations of the words, also using LSTMs. This&#10;allows statistical sharing across word forms that are similar on the surface.&#10;Experiments for morphologically rich languages show that the parsing model&#10;benefits from incorporating the character-based encodings of words.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.09060" label="1808.09060">
        <attvalues>
          <attvalue for="0" value="An Investigation of the Interactions Between Pre-Trained Word&#10;  Embeddings, Character Models and POS Tags in Dependency Parsing" />
          <attvalue for="1" value="  We provide a comprehensive analysis of the interactions between pre-trained&#10;word embeddings, character models and POS tags in a transition-based dependency&#10;parser. While previous studies have shown POS information to be less important&#10;in the presence of character models, we show that in fact there are complex&#10;interactions between all three techniques. In isolation each produces large&#10;improvements over a baseline system using randomly initialised word embeddings&#10;only, but combining them quickly leads to diminishing returns. We categorise&#10;words by frequency, POS tag and language in order to systematically investigate&#10;how each of the techniques affects parsing quality. For many word categories,&#10;applying any two of the three techniques is almost as good as the full combined&#10;system. Character models tend to be more important for low-frequency open-class&#10;words, especially in morphologically rich languages, while POS tags can help&#10;disambiguate high-frequency function words. We also show that large character&#10;embedding sizes help even for languages with small character sets, especially&#10;in morphologically rich languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1603.04351" label="1603.04351">
        <attvalues>
          <attvalue for="0" value="Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature&#10;  Representations" />
          <attvalue for="1" value="  We present a simple and effective scheme for dependency parsing which is&#10;based on bidirectional-LSTMs (BiLSTMs). Each sentence token is associated with&#10;a BiLSTM vector representing the token in its sentential context, and feature&#10;vectors are constructed by concatenating a few BiLSTM vectors. The BiLSTM is&#10;trained jointly with the parser objective, resulting in very effective feature&#10;extractors for parsing. We demonstrate the effectiveness of the approach by&#10;applying it to a greedy transition-based parser as well as to a globally&#10;optimized graph-based parser. The resulting parsers have very simple&#10;architectures, and match or surpass the state-of-the-art accuracies on English&#10;and Chinese.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.01083" label="2104.01083">
        <attvalues>
          <attvalue for="0" value="What Taggers Fail to Learn, Parsers Need the Most" />
          <attvalue for="1" value="  We present an error analysis of neural UPOS taggers to evaluate why using&#10;gold standard tags has such a large positive contribution to parsing&#10;performance while using predicted UPOS tags either harms performance or offers&#10;a negligible improvement. We evaluate what neural dependency parsers implicitly&#10;learn about word types and how this relates to the errors taggers make to&#10;explain the minimal impact using predicted tags has on parsers. We also present&#10;a short analysis on what contexts result in reductions in tagging performance.&#10;We then mask UPOS tags based on errors made by taggers to tease away the&#10;contribution of UPOS tags which taggers succeed and fail to classify correctly&#10;and the impact of tagging errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.04222" label="2106.04222">
        <attvalues>
          <attvalue for="0" value="A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags&#10;  for Low Resource UD Parsing" />
          <attvalue for="1" value="  We evaluate the efficacy of predicted UPOS tags as input features for&#10;dependency parsers in lower resource settings to evaluate how treebank size&#10;affects the impact tagging accuracy has on parsing performance. We do this for&#10;real low resource universal dependency treebanks, artificially low resource&#10;data with varying treebank sizes, and for very small treebanks with varying&#10;amounts of augmented data. We find that predicted UPOS tags are somewhat&#10;helpful for low resource treebanks, especially when fewer fully-annotated trees&#10;are available. We also find that this positive impact diminishes as the amount&#10;of data increases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.15219" label="2210.15219">
        <attvalues>
          <attvalue for="0" value="Parsing linearizations appreciate PoS tags - but some are fussy about&#10;  errors" />
          <attvalue for="1" value="  PoS tags, once taken for granted as a useful resource for syntactic parsing,&#10;have become more situational with the popularization of deep learning. Recent&#10;work on the impact of PoS tags on graph- and transition-based parsers suggests&#10;that they are only useful when tagging accuracy is prohibitively high, or in&#10;low-resource scenarios. However, such an analysis is lacking for the emerging&#10;sequence labeling parsing paradigm, where it is especially relevant as some&#10;models explicitly use PoS tags for encoding and decoding. We undertake a study&#10;and uncover some trends. Among them, PoS tags are generally more useful for&#10;sequence labeling parsers than for other paradigms, but the impact of their&#10;accuracy is highly encoding-dependent, with the PoS-based head-selection&#10;encoding being best only when both tagging accuracy and resource availability&#10;are high.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.11268" label="1905.11268">
        <attvalues>
          <attvalue for="0" value="Combating Adversarial Misspellings with Robust Word Recognition" />
          <attvalue for="1" value="  To combat adversarial spelling mistakes, we propose placing a word&#10;recognition model in front of the downstream classifier. Our word recognition&#10;models build upon the RNN semi-character architecture, introducing several new&#10;backoff strategies for handling rare and unseen words. Trained to recognize&#10;words corrupted by random adds, drops, swaps, and keyboard mistakes, our method&#10;achieves 32% relative (and 3.3% absolute) error reduction over the vanilla&#10;semi-character model. Notably, our pipeline confers robustness on the&#10;downstream classifier, outperforming both adversarial training and&#10;off-the-shelf spell checkers. Against a BERT model fine-tuned for sentiment&#10;analysis, a single adversarially-chosen character attack lowers accuracy from&#10;90.3% to 45.8%. Our defense restores accuracy to 75%. Surprisingly, better word&#10;recognition does not always entail greater robustness. Our analysis reveals&#10;that robustness also depends upon a quantity that we denote the sensitivity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.04992" label="2309.04992">
        <attvalues>
          <attvalue for="0" value="Mitigating Word Bias in Zero-shot Prompt-based Classifiers" />
          <attvalue for="1" value="  Prompt-based classifiers are an attractive approach for zero-shot&#10;classification. However, the precise choice of the prompt template and label&#10;words can largely influence performance, with semantically equivalent settings&#10;often showing notable performance difference. This discrepancy can be partly&#10;attributed to word biases, where the classifier may be biased towards classes.&#10;To address this problem, it is possible to optimise classification thresholds&#10;on a labelled data set, however, this mitigates some of the advantages of&#10;prompt-based classifiers. This paper instead approaches this problem by&#10;examining the expected marginal probabilities of the classes. Here,&#10;probabilities are reweighted to have a uniform prior over classes, in an&#10;unsupervised fashion. Further, we draw a theoretical connection between the&#10;class priors and the language models' word prior, and offer the ability to set&#10;a threshold in a zero-resource fashion. We show that matching class priors&#10;correlates strongly with the oracle upper bound performance and demonstrate&#10;large consistent performance gains for prompt settings over a range of NLP&#10;tasks.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLM) have shown impressive general ability for natural language processing (NLP) tasks. LLMs can effectively handle a range of NLP tasks through ‘prompting’, where a natural language instruction is added to the input, conditioning the model to the task at hand. Prompting can either be an emergent ability learned through scaling up model size \cite{brown2020language, wei2022emergent} or an ability learned through instruction tuning \cite{weifinetuned, chung2022scaling, ouyang2022training}. Despite the recent popularity of prompting, there is a known sensitivity of prompt-based LLMs to elements such as prompt template and label words \cite{gao2021making, schick2021exploiting}. Previous works have demonstrated that prompt templates can significantly impact task performance \cite{shin2020autoprompt, zhou2022large} and that factors such as chosen label words can influence system performance for classification tasks \cite{zhao2021calibrate, holtzman2021surface}. &#10;&#10;This work focuses on the influence of `word biases' for prompt-based classifiers. i.e. the bias that prompts may have towards certain classes, independent of the input text. To account for this bias, one could use a labelled dataset to find optimal class decision thresholds. This, however, requires labelled task data, which may limit the zero-shot benefits of prompt-based classifiers. We propose a simple unsupervised solution of re-weighting probabilities, where we use unlabelled data to search for weight parameters that ensure a uniform prior over classes. We show that this prior matching leads to greater robustness for diverse prompt settings and that the unsupervised weights which debias the classifier is highly correlated with the oracle weights that maximise accuracy. Further, we provide theoretical analysis that draws a connection between word priors and inherent class bias, which we use to motivate a zero-resource normalisation approach that is competitive with prior matching. Overall, we demonstrate that our unsupervised approach highly reduces sensitivity to the chosen prompt and label words, and that settings which initially fail can often be made effective through a simple probability re-weighting.&#10;&#10;Our contributions are 1) We propose a simple unsupervised probability re-weighting method, and empirically demonstrate greater robustness to prompt and label word choice, with large accuracy gains across prompt settings for a range of standard NLP tasks. 2) We theoretically connect the weight parameters to word priors and use this to motivate a zero-resource re-weighting approach. 3) We show that the weights of prior matching are highly correlated with the optimal oracle weights that maximize accuracy, illustrating that our approach is a near-optimal use of a system's output probabilities. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Zero-Shot Classification, Linguistics, Prompt-Based Classifiers, Language Model Bias, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2405.15032" label="2405.15032">
        <attvalues>
          <attvalue for="0" value="Aya 23: Open Weight Releases to Further Multilingual Progress" />
          <attvalue for="1" value="  This technical report introduces Aya 23, a family of multilingual language&#10;models. Aya 23 builds on the recent release of the Aya model (\&quot;Ust\&quot;un et al.,&#10;2024), focusing on pairing a highly performant pre-trained model with the&#10;recently released Aya collection (Singh et al., 2024). The result is a powerful&#10;multilingual large language model serving 23 languages, expanding state-of-art&#10;language modeling capabilities to approximately half of the world's population.&#10;The Aya model covered 101 languages whereas Aya 23 is an experiment in depth vs&#10;breadth, exploring the impact of allocating more capacity to fewer languages&#10;that are included during pre-training. Aya 23 outperforms both previous&#10;massively multilingual models like Aya 101 for the languages it covers, as well&#10;as widely used models like Gemma, Mistral and Mixtral on an extensive range of&#10;discriminative and generative tasks. We release the open weights for both the&#10;8B and 35B models as part of our continued commitment for expanding access to&#10;multilingual progress.&#10;" />
          <attvalue for="2" value="&#10;&#10;In this work we introduce \aya 23, a family of multilingual instruction-tuned language models supporting 23 languages based on Cohere's Command model and the \aya multilingual instruction-style collection \cite{ayadata2024}. To date, the majority of progress in large language modeling has been English-centric, leading to models which perform poorly outside of a handful of languages. This can result in cliffs in model performance in languages not included in pre-training \cite{schwartz2022towards, Kotek2023GenderBA, Khandelwal2023CasteistBN, vashishtha2023evaluating,khondaker2023gptaraeval}, the introduction of security flaws for all users, \cite{yong2023lowresource, nasr2023scalable, Li2023PrivacyIL, Lukas2023AnalyzingLO, deng2023multilingual} and a growing divide in the cost of technology due to high latencies for generations outside of English \cite{held2023material, durmus2023measuring,nicholas2023lost,ojo2023good,ahia2023languages}. &#10;&#10;Multilingual efforts including the release of Aya 101 \cite{ustun2024aya}, BLOOMZ~\cite{muennighoff2022crosslingual} and mT0~\cite{muennighoff2022crosslingual} models have made great strides in expanding access to modern natural language processing technologies for the world. However, there still remains significant room for improvement relative to first-class citizen languages like English and Chinese. Two major hurdles in the development of powerful multilingual models are (1) the lack of robust multilingual pretrained models, and (2) the scarcity of instruction-style training data covering a diverse set of languages. &#10;&#10;The \aya initiative was created to address the aforementioned data scarcity issues by creating and releasing the largest multilingual instruction-style dataset~\cite{ayadata2024} to date, along with the \aya 101 model~\cite{ustun2024aya}. \aya 101 was a step forward in massively multilingual language modeling, creating a 101 languages state-of-the-art instruction fine-tuned LLM. However, \aya 101 was by necessity built upon the mT5~\cite{xue2020mt5} pre-trained base model given it was one of the few pre-trained models that had been trained on 101 languages. mT5 is relatively outdated given the rapid advances in LLM technology since its release in 2019. Its major limitations are: 1) Outdated knowledge: Having been pre-trained several years ago, mT5 is not as useful for interactions about events that occurred recently. 2) Inadequate Performance: There are many stronger models now compared to when mT5 was released, such as the Command R{\raisebox{-0.3ex}{+}}, Command R, Llama series~\cite{touvron2023llama,touvron2023llama2}, Mistral models \cite{jiang2023mistral,jiang2024mixtral} and Gemma models \cite{gemmareport}. &#10;&#10;Furthermore, \aya 101 was a 13-billion parameter model designed for breadth, expanding coverage to nearly double that achieved by previous models with 101 languages. Due to the well-documented curse of multilinguality \cite{arivazhagan2019massively,conneau2019unsupervised,pfeiffer2022lifting}, models attempting to serve such a broad variety of languages often lag in generative performance on any given language relative to models dedicated to serving a more focused subset, because of the need to share model capacity so widely. &#10;For \aya 23, we instead balance breadth and depth, exploring the impact of allocating more capacity to fewer languages (23 languages) that are included during pre-training, alleviating the ``curse'' and leading to large gains over the original \aya 101 and widely used models such as Gemma \cite{gemmareport}, Mistral \cite{jiang2023mistral}, and Mixtral \cite{jiang2024mixtral} for the corresponding 23 languages.&#10;&#10;In this technical report, we assess the performance of \aya 23 models following the comprehensive multilingual evaluation framework proposed by~\cite{ustun2024aya}. In our evaluation, we focus on 23 languages that are covered by the new \aya model family. These 23 languages are: Arabic, Chinese (simplified \&amp; traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian and Vietnamese. Our choice of languages was guided to align with the languages present in pre-training of Command R, due to known difficulties of introducing new languages after pre-training \cite{zhao2024llama, yong2022bloom+}. &#10;&#10;We release \aya 23 in two model sizes: 8-billion (8B) and 35-billion (35B) parameters. \aya-23-35B achieves the highest results across all the evaluation tasks and languages covered, while \aya-23-8B demonstrates best-in-class multilingual performance which is crucial given that model sizes above 13B parameters limit model usability on consumer-grade hardware. We note that relative to \aya 101, \aya 23 improves on discriminative tasks by up to 14\%, generative tasks by up to 20\%, and multilingual MMLU by up to 41.6\%. Furthermore, \aya 23 achieves a 6.6x increase in multilingual mathematical reasoning compared to \aya 101. Across \aya 101, Mistral, and Gemma, we report a mix of human annotators and LLM-as-a-judge comparisons. Across all comparisons, the \aya-23-8B and \aya-23-35B are consistently preferred. By releasing the weights of the \aya 23 model family, we hope to empower researchers and practitioners to advance multilingual models and applications. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Cognitive Science, Multilingual Language Models, Artificial Intelligence Advancements, Large Language Model Development" />
        </attvalues>
      </node>
      <node id="2402.06619" label="2402.06619">
        <attvalues>
          <attvalue for="0" value="Aya Dataset: An Open-Access Collection for Multilingual Instruction&#10;  Tuning" />
          <attvalue for="1" value="  Datasets are foundational to many breakthroughs in modern artificial&#10;intelligence. Many recent achievements in the space of natural language&#10;processing (NLP) can be attributed to the finetuning of pre-trained models on a&#10;diverse set of tasks that enables a large language model (LLM) to respond to&#10;instructions. Instruction fine-tuning (IFT) requires specifically constructed&#10;and annotated datasets. However, existing datasets are almost all in the&#10;English language. In this work, our primary goal is to bridge the language gap&#10;by building a human-curated instruction-following dataset spanning 65&#10;languages. We worked with fluent speakers of languages from around the world to&#10;collect natural instances of instructions and completions. Furthermore, we&#10;create the most extensive multilingual collection to date, comprising 513&#10;million instances through templating and translating existing datasets across&#10;114 languages. In total, we contribute four key resources: we develop and&#10;open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection,&#10;and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case&#10;study in participatory research, involving collaborators from 119 countries. We&#10;see this as a valuable framework for future research collaborations that aim to&#10;bridge gaps in resources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14976" label="2305.14976">
        <attvalues>
          <attvalue for="0" value="GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP" />
          <attvalue for="1" value="  ChatGPT's emergence heralds a transformative phase in NLP, particularly&#10;demonstrated through its excellent performance on many English benchmarks.&#10;However, the model's efficacy across diverse linguistic contexts remains&#10;largely uncharted territory. This work aims to bridge this knowledge gap, with&#10;a primary focus on assessing ChatGPT's capabilities on Arabic languages and&#10;dialectal varieties. Our comprehensive study conducts a large-scale automated&#10;and human evaluation of ChatGPT, encompassing 44 distinct language&#10;understanding and generation tasks on over 60 different datasets. To our&#10;knowledge, this marks the first extensive performance analysis of ChatGPT's&#10;deployment in Arabic NLP. Our findings indicate that, despite its remarkable&#10;performance in English, ChatGPT is consistently surpassed by smaller models&#10;that have undergone finetuning on Arabic. We further undertake a meticulous&#10;comparison of ChatGPT and GPT-4's Modern Standard Arabic (MSA) and Dialectal&#10;Arabic (DA), unveiling the relative shortcomings of both models in handling&#10;Arabic dialects compared to MSA. Although we further explore and confirm the&#10;utility of employing GPT-4 as a potential alternative for human evaluation, our&#10;work adds to a growing body of research underscoring the limitations of&#10;ChatGPT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.02446" label="2310.02446">
        <attvalues>
          <attvalue for="0" value="Low-Resource Languages Jailbreak GPT-4" />
          <attvalue for="1" value="  AI safety training and red-teaming of large language models (LLMs) are&#10;measures to mitigate the generation of unsafe content. Our work exposes the&#10;inherent cross-lingual vulnerability of these safety mechanisms, resulting from&#10;the linguistic inequality of safety training data, by successfully&#10;circumventing GPT-4's safeguard through translating unsafe English inputs into&#10;low-resource languages. On the AdvBenchmark, GPT-4 engages with the unsafe&#10;translated inputs and provides actionable items that can get the users towards&#10;their harmful goals 79% of the time, which is on par with or even surpassing&#10;state-of-the-art jailbreaking attacks. Other high-/mid-resource languages have&#10;significantly lower attack success rate, which suggests that the cross-lingual&#10;vulnerability mainly applies to low-resource languages. Previously, limited&#10;training on low-resource languages primarily affects speakers of those&#10;languages, causing technological disparities. However, our work highlights a&#10;crucial shift: this deficiency now poses a risk to all LLMs users. Publicly&#10;available translation APIs enable anyone to exploit LLMs' safety&#10;vulnerabilities. Therefore, our work calls for a more holistic red-teaming&#10;efforts to develop robust multilingual safeguards with wide language coverage.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.07377" label="2306.07377">
        <attvalues>
          <attvalue for="0" value="Lost in Translation: Large Language Models in Non-English Content&#10;  Analysis" />
          <attvalue for="1" value="  In recent years, large language models (e.g., Open AI's GPT-4, Meta's LLaMa,&#10;Google's PaLM) have become the dominant approach for building AI systems to&#10;analyze and generate language online. However, the automated systems that&#10;increasingly mediate our interactions online -- such as chatbots, content&#10;moderation systems, and search engines -- are primarily designed for and work&#10;far more effectively in English than in the world's other 7,000 languages.&#10;Recently, researchers and technology companies have attempted to extend the&#10;capabilities of large language models into languages other than English by&#10;building what are called multilingual language models.&#10;  In this paper, we explain how these multilingual language models work and&#10;explore their capabilities and limits. Part I provides a simple technical&#10;explanation of how large language models work, why there is a gap in available&#10;data between English and other languages, and how multilingual language models&#10;attempt to bridge that gap. Part II accounts for the challenges of doing&#10;content analysis with large language models in general and multilingual&#10;language models in particular. Part III offers recommendations for companies,&#10;researchers, and policymakers to keep in mind when considering researching,&#10;developing and deploying large and multilingual language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.07978" label="2311.07978">
        <attvalues>
          <attvalue for="0" value="How good are Large Language Models on African Languages?" />
          <attvalue for="1" value="  Recent advancements in natural language processing have led to the&#10;proliferation of large language models (LLMs). These models have been shown to&#10;yield good performance, using in-context learning, even on tasks and languages&#10;they are not trained on. However, their performance on African languages is&#10;largely understudied relative to high-resource languages. We present an&#10;analysis of four popular large language models (mT0, Aya, LLaMa 2, and GPT-4)&#10;on six tasks (topic classification, sentiment classification, machine&#10;translation, summarization, question answering, and named entity recognition)&#10;across 60 African languages, spanning different language families and&#10;geographical regions. Our results suggest that all LLMs produce lower&#10;performance for African languages, and there is a large gap in performance&#10;compared to high-resource languages (such as English) for most tasks. We find&#10;that GPT-4 has an average to good performance on classification tasks, yet its&#10;performance on generative tasks such as machine translation and summarization&#10;is significantly lacking. Surprisingly, we find that mT0 had the best overall&#10;performance for cross-lingual QA, better than the state-of-the-art supervised&#10;model (i.e. fine-tuned mT5) and GPT-4 on African languages. Similarly, we find&#10;the recent Aya model to have comparable result to mT0 in almost all tasks&#10;except for topic classification where it outperform mT0. Overall, LLaMa 2&#10;showed the worst performance, which we believe is due to its English and&#10;code-centric~(around 98%) pre-training corpus. Our findings confirms that&#10;performance on African languages continues to remain a hurdle for the current&#10;LLMs, underscoring the need for additional efforts to close this gap.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;\insertrelatedworks&#10;&#10;Large Language Model Evaluation: Accurate and reproducible evaluation of language models is important as more and more models are being released.&#10;As these models are integrated into various applications, developing robust evaluation frameworks becomes paramount for understanding their true capabilities and limitations.&#10;As a result, the community has worked on developing evaluation frameworks \cite{eval-harness, lighteval, liang2023holistic}, leaderboards \cite{10.5555/3692070.3692401, srivastava2023beyond, open-llm-leaderboard-v2} and benchmarks \cite{Adelani2024IrokoBenchAN, zhou2023instructionfollowingevaluationlargelanguage, hendrycks2021measuringmathematicalproblemsolving}.&#10;While each of these evaluation tools focuses on assessing specific aspects of language model capabilities - from basic linguistic understanding to complex reasoning tasks - the development of truly comprehensive benchmarks remains a significant challenge \cite{ruder2021benchmarking, biderman2024lmevaluation}.&#10;These challenges stem from complex nature of language understanding and the stochastic nature of language models &#10;&#10;Multilingual LLM Benchmarks:&#10;Benchmarks serve as a standard for measuring how systems have improved over time on across specific tasks and metrics.&#10;In the context of LLMs, multilingual benchmarks are crucial to assessing both the quality and practical utility of these models across diverse languages and tasks.&#10;Our primary focus lies in understanding LLM performance specifically for African languages, with several notable benchmarks (\ref{tab:works}) having emerged in recent years to address this need.&#10;ChatGPT-MT~\cite{robinson-etal-2023-chatgpt} evaluated the translation capability of GPT-4 and they find that it's demonstrates strong performances on high-resource languages, the performance on low-resource languages is subpar.&#10;Belebele~\cite{bandarkar-etal-2024-belebele} is a question answering task in 122 languages including 28 African languages for assessing reading comprehension abilities of LLMS.&#10;Mega~\cite{ahuja-etal-2023-mega} and Megaverse~\cite{ahuja-etal-2024-megaverse} are multi-task multilingual and multimodal benchmarks in 83 languages including 16 African languages.&#10;&#10;While these existing benchmarks have provided valuable insights, they collectively highlight a pressing need for more comprehensive evaluation that encompass a broader range of African languages and diverse tasks.&#10;Our research, through the development of \afrobench, addresses this gap by building upon and complementing existing work. &#10;We create a robust evaluation framework that assesses LLM performance across 64 African languages, evaluating capabilities across 15 distinct tasks. This expanded scope allows for a more nuanced and thorough understanding of LLM capabilities in African language contexts.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.13707" label="2305.13707">
        <attvalues>
          <attvalue for="0" value="Do All Languages Cost the Same? Tokenization in the Era of Commercial&#10;  Language Models" />
          <attvalue for="1" value="  Language models have graduated from being research prototypes to&#10;commercialized products offered as web APIs, and recent works have highlighted&#10;the multilingual capabilities of these products. The API vendors charge their&#10;users based on usage, more specifically on the number of ``tokens'' processed&#10;or generated by the underlying language models. What constitutes a token,&#10;however, is training data and model dependent with a large variance in the&#10;number of tokens required to convey the same information in different&#10;languages. In this work, we analyze the effect of this non-uniformity on the&#10;fairness of an API's pricing policy across languages. We conduct a systematic&#10;analysis of the cost and utility of OpenAI's language model API on multilingual&#10;benchmarks in 22 typologically diverse languages. We show evidence that&#10;speakers of a large number of the supported languages are overcharged while&#10;obtaining poorer results. These speakers tend to also come from regions where&#10;the APIs are less affordable to begin with. Through these analyses, we aim to&#10;increase transparency around language model APIs' pricing policies and&#10;encourage the vendors to make them more equitable.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.06266" label="2205.06266">
        <attvalues>
          <attvalue for="0" value="Lifting the Curse of Multilinguality by Pre-training Modular&#10;  Transformers" />
          <attvalue for="1" value="  Multilingual pre-trained models are known to suffer from the curse of&#10;multilinguality, which causes per-language performance to drop as they cover&#10;more languages. We address this issue by introducing language-specific modules,&#10;which allows us to grow the total capacity of the model, while keeping the&#10;total number of trainable parameters per language constant. In contrast with&#10;prior work that learns language-specific components post-hoc, we pre-train the&#10;modules of our Cross-lingual Modular (X-Mod) models from the start. Our&#10;experiments on natural language inference, named entity recognition and&#10;question answering show that our approach not only mitigates the negative&#10;interference between languages, but also enables positive transfer, resulting&#10;in improved monolingual and cross-lingual performance. Furthermore, our&#10;approach enables adding languages post-hoc with no measurable drop in&#10;performance, no longer limiting the model usage to the set of pre-trained&#10;languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.01055" label="2401.01055">
        <attvalues>
          <attvalue for="0" value="LLaMA Beyond English: An Empirical Study on Language Capability Transfer" />
          <attvalue for="1" value="  In recent times, substantial advancements have been witnessed in large&#10;language models (LLMs), exemplified by ChatGPT, showcasing remarkable&#10;proficiency across a range of complex tasks. However, many mainstream LLMs&#10;(e.g. LLaMA) are pretrained on English-dominant corpus, which limits their&#10;performance in other non-English languages. In this paper, we focus on how to&#10;effectively transfer the capabilities of language generation and following&#10;instructions to a non-English language. To answer this question, we conduct an&#10;extensive empirical investigation based on LLaMA, accumulating over 1440 GPU&#10;hours. We analyze the impact of key factors such as vocabulary extension,&#10;further pretraining, and instruction tuning on transfer. To accurately assess&#10;the model's level of knowledge, we employ four widely used standardized testing&#10;benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench. Furthermore, a&#10;comprehensive evaluation of the model's response quality is conducted,&#10;considering aspects such as accuracy, fluency, informativeness, logical&#10;coherence, and harmlessness, based on LLM-Eval, a benchmarks consisting&#10;instruction tasks from 17 diverse categories. Our evaluation results&#10;demonstrate that comparable performance to state-of-the-art transfer models can&#10;be achieved with less than 1% of the pretraining data, both in terms of&#10;knowledge alignment and response quality. Furthermore, the experimental&#10;outcomes across the thirteen low-resource languages also exhibit similar&#10;trends. We anticipate that the conclusions revealed by the experiments will aid&#10;the community in developing non-English LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.14885" label="2407.14885">
        <attvalues>
          <attvalue for="0" value="Falcon2-11B Technical Report" />
          <attvalue for="1" value="  We introduce Falcon2-11B, a foundation model trained on over five trillion&#10;tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a&#10;vision-to-text model. We report our findings during the training of the&#10;Falcon2-11B which follows a multi-stage approach where the early stages are&#10;distinguished by their context length and a final stage where we use a curated,&#10;high-quality dataset. Additionally, we report the effect of doubling the batch&#10;size mid-training and how training loss spikes are affected by the learning&#10;rate. The downstream performance of the foundation model is evaluated on&#10;established benchmarks, including multilingual and code datasets. The&#10;foundation model shows strong generalization across all the tasks which makes&#10;it suitable for downstream finetuning use cases. For the vision language model,&#10;we report the performance on several benchmarks and show that our model&#10;achieves a higher average score compared to open-source models of similar size.&#10;The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made&#10;available under a permissive license.&#10;" />
          <attvalue for="2" value="&#10;&#10;The first generation of Falcon models, featuring Falcon-7B, Falcon-40B, and Falcon-180B \cite{falcon}, made a significant contribution to the open-source community, promoting the release of advanced LLMs with permissive licenses. In this report, we introduce a second generation of models, Falcon2, focused on increased usability and integrability, towards building a multi-modal ecosystem currently composed of a large language model with 11B parameters and a corresponding vision language model.&#10;&#10;Historically, large language models first saw an important rise in performance with increased model size \cite{gpt3, palm}. Updated scaling laws \cite{chinchilla} brought to light that this initial generation of large language models were most likely undertrained, highlighting the need for more training data to further increase the performance. This triggered another important paradigm shift, namely moving from large curated datasets \cite{pile, palm}, to large-scale datasets harvesting mostly web data from the CommonCrawl project, such as RefinedWeb \cite{refinedweb} or RedPajama \cite{redpajama}. Both these advances led to the release of large open-source models such as Llama-65B \cite{llama} and Falcon-180B \cite{falcon}. More recently, the Llama2 models \cite{llama2} showed the benefits of even more prolonged training, achieving state-of-the-art performance with smaller model sizes. This trend was followed in the past year, resulting in a number of small-sized yet highly performing models such as Qwen-7B \cite{qwen}, Mistral-7B \cite{mistral}, Yi-6B and Yi-9B \cite{yi}, Gemma-7B \cite{gemma} and Llama3-8B \cite{llama3}.&#10;&#10;We present the Falcon2-11B LLM model, achieving better performance on the Open LLM Leaderboard tasks compared to Mistral-7B and Llama3-8B, and similar to Gemma-7B. The Falcon2-11B model surpasses the performance of the larger Falcon-40B, ensuring reduced computational cost and faster inference for end users while conserving its multilingual aspect. We also present the Falcon-11B-vlm, a vision language model built from the Falcon2-11B foundation model. Both models are available under a permissive open-source license as detailed in \ref{sec:license}.&#10;&#10;\ref{sec:pretraining} offers pre-training details for the Falcon2-11B LLM, including model architecture, training hyperparameters and the effect of batch size doubling, training stages to enable context length extension and multilingual support, while \ref{subsec:eval} presents the model's performance across multiple benchmarks, including multilingual and code datasets. \ref{sec:vlm} presents the multimodal extension to a vision languages model, Falcon2-11B-vlm, including training details and evaluation results. Model availability and license information are covered in \ref{sec:license}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Multimodal Learning, Mathematics, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="2311.16867" label="2311.16867">
        <attvalues>
          <attvalue for="0" value="The Falcon Series of Open Language Models" />
          <attvalue for="1" value="  We introduce the Falcon series: 7B, 40B, and 180B parameters causal&#10;decoder-only models trained on a diverse high-quality corpora predominantly&#10;assembled from web data. The largest model, Falcon-180B, has been trained on&#10;over 3.5 trillion tokens of text--the largest openly documented pretraining&#10;run. Falcon-180B significantly outperforms models such as PaLM or Chinchilla,&#10;and improves upon concurrently developed models such as LLaMA 2 or&#10;Inflection-1. It nears the performance of PaLM-2-Large at a reduced pretraining&#10;and inference cost, making it, to our knowledge, one of the three best language&#10;models in the world along with GPT-4 and PaLM-2-Large. We report detailed&#10;evaluations, as well as a deep dive into the methods and custom tooling&#10;employed to pretrain Falcon. Notably, we report on our custom distributed&#10;training codebase, allowing us to efficiently pretrain these models on up to&#10;4,096 A100s on cloud AWS infrastructure with limited interconnect. We release a&#10;600B tokens extract of our web dataset, as well as the Falcon-7/40/180B models&#10;under a permissive license to foster open-science and accelerate the&#10;development of an open ecosystem of large language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.04652" label="2403.04652">
        <attvalues>
          <attvalue for="0" value="Yi: Open Foundation Models by 01.AI" />
          <attvalue for="1" value="  We introduce the Yi model family, a series of language and multimodal models&#10;that demonstrate strong multi-dimensional capabilities. The Yi model family is&#10;based on 6B and 34B pretrained language models, then we extend them to chat&#10;models, 200K long context models, depth-upscaled models, and vision-language&#10;models. Our base models achieve strong performance on a wide range of&#10;benchmarks like MMLU, and our finetuned chat models deliver strong human&#10;preference rate on major evaluation platforms like AlpacaEval and Chatbot&#10;Arena. Building upon our scalable super-computing infrastructure and the&#10;classical transformer architecture, we attribute the performance of Yi models&#10;primarily to its data quality resulting from our data-engineering efforts. For&#10;pretraining, we construct 3.1 trillion tokens of English and Chinese corpora&#10;using a cascaded data deduplication and quality filtering pipeline. For&#10;finetuning, we polish a small scale (less than 10K) instruction dataset over&#10;multiple iterations such that every single instance has been verified directly&#10;by our machine learning engineers. For vision-language, we combine the chat&#10;language model with a vision transformer encoder and train the model to align&#10;visual representations to the semantic space of the language model. We further&#10;extend the context length to 200K through lightweight continual pretraining and&#10;demonstrate strong needle-in-a-haystack retrieval performance. We show that&#10;extending the depth of the pretrained checkpoint through continual pretraining&#10;further improves performance. We believe that given our current results,&#10;continuing to scale up model parameters using thoroughly optimized data will&#10;lead to even stronger frontier models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent breakthroughs in large language models have revolutionized the whole field of artificial intelligence and potentially radiate across the entire human society. &#10;Our vision for large language models is to make them the next generation computational platform and empower the whole community with significantly amplified intelligence. &#10;As a step towards this mission, we present the Yi model series, 6B and 34B language models pretrained from scratch on 3.1T highly-engineered large amount of data, and finetuned on a small but meticulously polished alignment data. &#10;Due to the data quality resulting from our substantial engineering efforts, which we will detail in the upcoming sections, Yi achieves near GPT-3.5 benchmark scores and human preferences. &#10;&#10;In designing the Yi model series, we are mostly concerned on the following dimensions regarding&#10;model scale, data scale, and data quality:&#10;(1). when choosing model scale, the desiderata is to have small enough model that is feasible for inference on consumer-grade hardware like the RTX 4090 where the bounding factor is its limited 24G memory, yet still large enough with complex reasoning and emergent abilities. This is why we found 34B gives a nice performance-cost balance;&#10;(2). since 34B is smaller than the conventional 70B used by Chinchilla~\cite{hoffmann2022training} and LLaMA~\cite{llama2}, we increase the pretrain data scale to 3.1T tokens to compensate for the decreased compute flops. This makes the model-data scale combination fall into the post Chinchilla optimal regime~\cite{sardana2023beyond}, i.e., we overtrain the model on more tokens (3T) than the compute optimal (around 1T). The benefit is from the inference side, as we achieve stronger performance with reduced serving cost: after int4~\cite{wu2023understanding} quantization, one can serve the 34B chat model on 24G GPU memory with almost no performance drop;&#10;(3). our data engineering principle is to promote quality over quantity for both pretraining and finetuning.&#10;The pretraining data quality is guaranteed by a sophisticated data cleaning pipeline with cascaded filtering methods and intentionally increased deduplication strength;&#10;(4). for finetuning data we heavily emphasize quality by handcrafting less than 10K instructions over multiple iterations based on user feedback. This approach significantly deviates from the quantity-scaling styled instruction tuning works like FLAN~\cite{chung2022scaling} and UltraChat~\cite{ding2023enhancing}, but aligns more with the handcrafting styled works like LIMA~\cite{zhou2023lima}.&#10;&#10;Our pretraining data cleaning system features a sophisticated filtering pipeline based on language, heuristic textual features, perplexity, semantics, topic, and safety, as well as a cascaded deduplication process based on paragraph, MinHash, and exact matching. &#10;This thorough pipeline leads to a much higher removal ratio than existing pipelines like CCNet~\cite{wenzek2019ccnet}, RefinedWeb~\cite{penedo2023refinedweb} and RedPajama~\cite{together2023redpajama}, which we believe is key to the success of data engineering.&#10;The underlying principle is although pretraining requires data scaling, one would like to make sure the data used are of high quality, rather than training the model on large raw data, i.e., &#10;we prefer 3T tokens over sophasticated engineering over 10T tokens without extensive filtering. &#10;Regarding the model architecture, we use standard implementation of the Transformer architecture with Grouped-Query Attention (GQA)~\cite{gqa}, SwiGLU~\cite{swiglu} activation, and RoPE with an adjusted base frequency (RoPE ABF)~\cite{xiong2023effective}.&#10;This design choice is the standard approach rooted from the Transformer original paper~\cite{Vaswani:2017aa}, later modified by GPT-3 and Chinchilla~\cite{hoffmann2022training}, then followed by LLaMA~\cite{llama2}, Baichuan~\cite{Yang:2023aa}, Qwen~\cite{Bai:2023aa} and many related works. &#10;&#10;To approach GPT-3.5-matching human preferences, our finetuning dataset is curated from carefully selected multi-turn instruction-response pairs, annotated directly by our team of machine learning engineers then polished over multiple iterations of user feedback.&#10;As mentioned above, the size of our finetuning dataset is less than 10K, but improved over and over again across the model development timeline. &#10;Benefiting from the dataset's manageable size, we employed an extensive grid search to identify the optimal data composition, promote diversity, and discover effective hyperparameters. &#10;After 8-bit and 4-bit quantization, the final chat model can be deployed on consumer-grade GPUs nearly without performance degradation compared to the bf16 format.&#10;&#10;We further extend the Yi model capability from three dimensions: context scaling, vision-language adaptation, and depth-upscaling. &#10;To achive 200K context length, we continue pretrain the model on about 5B length-upsampled data, similar to the concurrent work in~\cite{fu2024data}. &#10;To adapt the model to vision-language tasks, we integrate a vision encoder and develop a multi-stage training method, following and improving the practice of~\cite{liu2023visual}. &#10;We also study the effectiveness of depth-upscailng~\cite{kim2023solar}, i.e., making the model deeper by continual pretraining, and confirming its effectiveness to further improve model performance.&#10;&#10;Our infrastructure provides strong support for the full-stack development of the Yi model series, from pretraining to finetuning to serving. &#10;To support pretraining, we develop cross-cloud elastic task scheduling, automatic failure recovery, and topology-aware resource allocation which collectively enable us to run tasks according to the real-time available GPU nodes cross clusters with limited switching overhead. &#10;To support finetuning, we build a hierarchical scheduling framework supporting different distributed backends for different models (e.g., Megatron~\cite{megatron-lm} for the policy model and DeepSpeed~\cite{zero} for the reward model). &#10;For efficient inference, we use 4-bit model and 8-bit KV cache quantization, combining with PagedAttention~\cite{kwon2023efficient} and Dynamic Batching. &#10;&#10;Extensive experiments demonstrate that Yi-34B can match GPT-3.5 in both performance and efficiency. &#10;On most standard benchmarks like MMLU~\cite{hendrycks2020measuring} (for the base model) and LMSys ELO Rating~\cite{zheng2023judging} (for the chat model), Yi-34B generally achieves scores on par with GPT-3.5. &#10;After model parameter and KV cache quantization, the inference cost is also controlled such that a wide range of the community can deploy the model on cost effective devices. &#10;We further report a detailed performance comparison between Yi and major LLMs on commonsense reasoning, college exams, math, coding, reading comprehension, and human preference win-rate on multiple evaluation benchmarks. &#10;&#10;Since its release, the Yi model series has benefited the community from the following perspectives: &#10;(1). it provides GPT-3.5-matching quality yet cost-effective models to researchers, and enables developers to build AI-native applications like language model based agents;&#10;(2). it empowers end users with locally runnable chatbots, which consequently helps protecting user data privacy; &#10;(3). it sheds light on the direction on further data and model scaling to achieve even stronger frontier models. &#10;for both research and commercial use.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.11586" label="2204.11586">
        <attvalues>
          <attvalue for="0" value="Which Discriminator for Cooperative Text Generation?" />
          <attvalue for="1" value="  Language models generate texts by successively predicting probability&#10;distributions for next tokens given past ones. A growing field of interest&#10;tries to leverage external information in the decoding process so that the&#10;generated texts have desired properties, such as being more natural, non toxic,&#10;faithful, or having a specific writing style. A solution is to use a classifier&#10;at each generation step, resulting in a cooperative environment where the&#10;classifier guides the decoding of the language model distribution towards&#10;relevant texts for the task at hand. In this paper, we examine three families&#10;of (transformer-based) discriminators for this specific task of cooperative&#10;decoding: bidirectional, left-to-right and generative ones. We evaluate the&#10;pros and cons of these different types of discriminators for cooperative&#10;generation, exploring respective accuracy on classification tasks along with&#10;their impact on the resulting sample quality and computational performances. We&#10;also provide the code of a batched implementation of the powerful cooperative&#10;decoding strategy used for our experiments, the Monte Carlo Tree Search,&#10;working with each discriminator for Natural Language Generation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformer~\cite{DBLP:conf/nips/VaswaniSPUJGKP17} architectures, coupled with an increase in computing capabilities, allows current Language Models (LM) to generate very plausible texts. Given an initial sequence of tokens (the prompt), the LM computes a probability distribution for the next token. A token is then sampled from this distribution and added to the initial sequence to generate the following token auto-regressively. Choosing the next token given the distribution (decoding) is commonly done using greedy search, beam search~\cite{dept._2018} or top-k/p sampling~\cite{DBLP:conf/acl/LewisDF18, DBLP:conf/iclr/HoltzmanBDFC20}; they select the next token only based on the likelihood (according to the LM) of the resulting sequence, which offers only limited control over the text finally generated. &#10;&#10;Yet, large LMs trained with non curated data are known to produce toxic and inappropriate content~\cite{DBLP:conf/fat/BenderGMS21, DBLP:conf/emnlp/GehmanGSCS20}. This is particularly problematic for Information Retrieval tasks that imply text generation, such as question-answering from the Web \cite{DBLP:journals/corr/abs-2112-09332,NEURIPS2020_6b493230}, query-focused multi-documents summarization \cite{pasunuru2021data}, query expansion \cite{ClaveauWI2021}, query suggestion \cite{DBLP:journals/tois/MustarLP22}, or chatbots for interactive search \cite{pallagani2021generic}, which leverage contents from various -- and sometimes untrusted -- information sources. &#10;&#10;Classifiers can be trained to identify a specific property of a text and thus provide useful information to guide the LM towards the desired property. &#10;For instance, following Generative Adversarial Networks~\cite{DBLP:journals/cacm/GoodfellowPMXWO20}, many studies train binary discriminators to distinguish real from generated contents, to approximate distributions of observed documents \cite{yu2017seqgan}. Other studies train classifiers on semantic properties such as polarity to learn the generation process towards positive or negative texts~\cite{DBLP:journals/corr/abs-2109-13582, DBLP:conf/emnlp/KrauseGMKJSR21, DBLP:conf/iclr/DathathriMLHFMY20}. &#10;In the context of Information Retrieval, this might also be used for instance to increase relevance of synthetic answers w.r.t. to the user's query. &#10;For all these purposes, there is an increasing interest for discriminator-generator cooperative decoding, where discriminators are used to guide generation \cite{DBLP:conf/acl/ChoiBGHBF18, DBLP:conf/icml/ScialomDLPS20, DBLP:conf/eacl/GabrielBDHBLCC21, DBLP:conf/iclr/DengBOSR20}. &#10;&#10;Currently, top performing discriminators are transformers using bidirectional attention~\cite{DBLP:conf/naacl/DevlinCLT19}, but this does not fit the iterative nature of the generation process. Indeed, it requires to recompute every hidden state of the whole sequence for any additional token, preventing the use of cached hidden states and resulting in a quadratic cost w.r.t. the sequence length at each timestep. &#10;On the other hand, unidirectional transformers, which employ left-to-right masks to only depend on past tokens for text encoding/decoding~\cite{Radford2019LanguageMA}, induce hidden states that can be reused for subsequent steps, hence involving linear computing complexity. However, these two types of discriminators only score one sequence at a time, given as input of the model. &#10;This limits the number of possible tokens to be considered at each decoding step, to avoid a computationally prohibitive cost. Solving this issue, recently introduced Generative Discriminators (GeDi)~\cite{DBLP:conf/emnlp/KrauseGMKJSR21} give scores for all tokens from the vocabulary at once, hence dramatically reducing the cost of width exploration. In this paper, we explore the pros and cons of these three types of discriminators (bidirectional, unidirectional, generative) when used in cooperative language decoding.&#10;&#10;In parallel, approaches relying on Monte Carlo Tree Search algorithm (MCTS)~\cite{DBLP:conf/cg/Coulom06} &#10;have been used for cooperative generation with more sophisticated exploration strategies than beam search.&#10;This non-myopic discriminator-guided decoding lead to&#10;state-of-the-art results in different applications~\cite{selfGAN, DBLP:journals/corr/abs-2109-13582, DBLP:conf/emnlp/LeblondASPLASV21, DBLP:journals/corr/abs-2201-12320}. We therefore use this promising cooperative decoding approach for our experiments and provide an implementation of the MCTS that allows to generate texts in batch for each type of discriminator based on the HuggingFace's transformers library~\cite{DBLP:conf/emnlp/WolfDSCDMCRLFDS20}. &#10;&#10;Before exposing our experimental study, we further define the task of cooperative decoding and justify our study in the next section.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Generation, Linguistics, Cooperative Decoding Strategies, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2103.01863" label="2103.01863">
        <attvalues>
          <attvalue for="0" value="Data Augmentation for Abstractive Query-Focused Multi-Document&#10;  Summarization" />
          <attvalue for="1" value="  The progress in Query-focused Multi-Document Summarization (QMDS) has been&#10;limited by the lack of sufficient largescale high-quality training datasets. We&#10;present two QMDS training datasets, which we construct using two data&#10;augmentation methods: (1) transferring the commonly used single-document&#10;CNN/Daily Mail summarization dataset to create the QMDSCNN dataset, and (2)&#10;mining search-query logs to create the QMDSIR dataset. These two datasets have&#10;complementary properties, i.e., QMDSCNN has real summaries but queries are&#10;simulated, while QMDSIR has real queries but simulated summaries. To cover both&#10;these real summary and query aspects, we build abstractive end-to-end neural&#10;network models on the combined datasets that yield new state-of-the-art&#10;transfer results on DUC datasets. We also introduce new hierarchical encoders&#10;that enable a more efficient encoding of the query together with multiple&#10;documents. Empirical results demonstrate that our data augmentation and&#10;encoding methods outperform baseline models on automatic metrics, as well as on&#10;human evaluations along multiple attributes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.00667" label="2203.00667">
        <attvalues>
          <attvalue for="0" value="Generative Adversarial Networks" />
          <attvalue for="1" value="  Generative Adversarial Networks (GANs) are very popular frameworks for&#10;generating high-quality data, and are immensely used in both the academia and&#10;industry in many domains. Arguably, their most substantial impact has been in&#10;the area of computer vision, where they achieve state-of-the-art image&#10;generation. This chapter gives an introduction to GANs, by discussing their&#10;principle mechanism and presenting some of their inherent problems during&#10;training and evaluation. We focus on these three issues: (1) mode collapse, (2)&#10;vanishing gradients, and (3) generation of low-quality images. We then list&#10;some architecture-variant and loss-variant GANs that remedy the above&#10;challenges. Lastly, we present two utilization examples of GANs for real-world&#10;applications: Data augmentation and face images generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1609.05473" label="1609.05473">
        <attvalues>
          <attvalue for="0" value="SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient" />
          <attvalue for="1" value="  As a new way of training generative models, Generative Adversarial Nets (GAN)&#10;that uses a discriminative model to guide the training of the generative model&#10;has enjoyed considerable success in generating real-valued data. However, it&#10;has limitations when the goal is for generating sequences of discrete tokens. A&#10;major reason lies in that the discrete outputs from the generative model make&#10;it difficult to pass the gradient update from the discriminative model to the&#10;generative model. Also, the discriminative model can only assess a complete&#10;sequence, while for a partially generated sequence, it is non-trivial to&#10;balance its current score and the future one once the entire sequence has been&#10;generated. In this paper, we propose a sequence generation framework, called&#10;SeqGAN, to solve the problems. Modeling the data generator as a stochastic&#10;policy in reinforcement learning (RL), SeqGAN bypasses the generator&#10;differentiation problem by directly performing gradient policy update. The RL&#10;reward signal comes from the GAN discriminator judged on a complete sequence,&#10;and is passed back to the intermediate state-action steps using Monte Carlo&#10;search. Extensive experiments on synthetic data and real-world tasks&#10;demonstrate significant improvements over strong baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.13582" label="2109.13582">
        <attvalues>
          <attvalue for="0" value="PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided&#10;  MCTS Decoding" />
          <attvalue for="1" value="  Large language models (LM) based on Transformers allow to generate plausible&#10;long texts. In this paper, we explore how this generation can be further&#10;controlled at decoding time to satisfy certain constraints (e.g. being&#10;non-toxic, conveying certain emotions, using a specific writing style, etc.)&#10;without fine-tuning the LM. Precisely, we formalize constrained generation as a&#10;tree exploration process guided by a discriminator that indicates how well the&#10;associated sequence respects the constraint. This approach, in addition to&#10;being easier and cheaper to train than fine-tuning the LM, allows to apply the&#10;constraint more finely and dynamically. We propose several original methods to&#10;search this generation tree, notably the Monte Carlo Tree Search (MCTS) which&#10;provides theoretical guarantees on the search efficiency, but also simpler&#10;methods based on re-ranking a pool of diverse sequences using the discriminator&#10;scores. These methods are evaluated, with automatic and human-based metrics, on&#10;two types of constraints and languages: review polarity and emotion control in&#10;French and English. We show that discriminator-guided MCTS decoding achieves&#10;state-of-the-art results without having to tune the language model, in both&#10;tasks and languages. We also demonstrate that other proposed decoding methods&#10;based on re-ranking can be really effective when diversity among the generated&#10;propositions is encouraged.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The goal of constrained textual generation is to find the sequence of tokens $x_{1:T}$ which maximises $p(x_{1:T} \mid c)$, given a constraint $c$.&#10;Few methods address the constrained textual generation.&#10;&#10;Class-conditional language models.&#10;Class-conditional language models (CC-LMs), as the Conditional Transformer Language (CTRL) model \cite{DBLP:journals/corr/abs-1909-05858}, train or fine-tune the weights $\theta$ of a single neural model directly for controllable generation, by appending a control code in the beginning of a training sequence. The control code indicates the constraint to verify and is related to a class containing texts that satisfy the constraint. For the sake of simplicity, we will denote without distinction the class, the constraint verified by its texts and the associated control code by $c$.&#10;Trained with different control codes, the model learns $p_\theta(x_{1:T} \mid c) = \prod_{t=1}^{T} p_\theta(x_{t} \mid x_{1:t-1}, c)$. The constraint can then be applied during generation by appending the corresponding control code to the prompt.&#10;While this method gives some kind of control over the generation, the control codes need to be defined upfront and the LM still needs to be trained specifically for each set of control codes. This is an important limitation since the current trend in text generation is the use of large pre-trained models which can hardly be fine-tuned (for instance, the last version of GPT, GPT-3, cannot be fine-tuned without access to very large hardware resources). &#10;&#10;Discriminator-based methods&#10;The general idea of discriminator-guided generation is to combine a disciminator $D$ with a generative LM. The discriminator explicitly models the constraint by calculating the probability $p_{D}(c \mid x_{1:T})$ of the sequence $x_{1:T}$ to satisfy the constraint $c$. This probability is directly related to $p(x_{1:T}\mid c)$ through Bayes' rule : $ p(x_{1:T}\mid c) \propto p_{D}(c \mid x_{1:T}) p_\theta(x_{1:T})$.&#10;Discriminator-based methods alleviate the training cost problem, as discriminators are easier to train than a LM. Moreover, any additional constraint can be defined a posteriori without tuning the LM, only by training another discriminator.&#10;The discriminators have been used in different ways to explore the search space. In the work of \cite{DBLP:conf/acl/ChoiBGHBF18, DBLP:conf/icml/ScialomDLPS20}, the space is first searched using beam search to generate a pool of proposals with a high likelihood $p_\theta(x_{1:T})$, and then the discriminator is used to re-rank them. &#10;However, in addition that beam search can miss sequences with high likelihood, it is biased towards the likelihood, while the best sequence might only have an average likelihood, but satisfies the constraint perfectly. &#10;&#10;Hence, it might be more suitable to take the discriminator probability into account during decoding rather than after generating a whole sequence. In this case, the discriminator is used at each generation step $t$ to get the probability $p_D(c \mid x_{1:t})$ for each token of the vocabulary $\mathcal{V}$, and merge it to the likelihood $p_{\theta}(x_{1:t})$ to choose which token to emit. &#10;In order to reduce the cost of using a discriminator on every possible continuation, GeDi~\cite{DBLP:journals/corr/abs-2009-06367} proposes to use CC-LMs as generative discriminators.&#10;The method relies on the fact that the CC-LM computes $p_{\theta}(x_{t} \mid x_{1:t-1}, c)$ for all tokens of the vocabulary which can be used to get $p_{\theta}(c \mid x_{1: t})$ for all tokens using Bayes' equation.&#10;This approach is thus at the intersectionof tuning the LM and using a discriminator: it tunes a small LM (the CC-LM) to guide a bigger one. &#10;&#10;In Plug And Play Language Model (PPLM)~\cite{DBLP:conf/iclr/DathathriMLHFMY20}, the discriminator is used to shift the hidden states of the pre-trained transformer-based LM towards the desired class at every generation step.&#10;PPLM can be used on any LM and with any discriminator.&#10;However, PPLM needs to access the LM to modify its hidden states, while our approach only requires the output logits. As some LM can only be used through access to logits (e.g. GPT-3 API), this makes our approach more plug and play than PPLM.&#10;&#10;A common drawback of all these approaches is their lack of a long-term vision of the generation. Indeed, the discriminator probabilities become necessarily more meaningful as the sequence grows and might only be trustable to guide the search when the sequence is (nearly) finished. When used in a myopic decoding strategy, classification errors will cause the generation process to deviate further and further. &#10;Trying to optimize a score defined in the long horizon by making short term decisions is very similar to common game setups such as chess, where the Monte Carlo Tree Search (MCTS) has proven to be really effective~\cite{Silver1140}, which motivated our approach.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.06363" label="2106.06363">
        <attvalues>
          <attvalue for="0" value="To Beam Or Not To Beam: That is a Question of Cooperation for Language&#10;  GANs" />
          <attvalue for="1" value="  Due to the discrete nature of words, language GANs require to be optimized&#10;from rewards provided by discriminator networks, via reinforcement learning&#10;methods. This is a much harder setting than for continuous tasks, which enjoy&#10;gradient flows from discriminators to generators, usually leading to dramatic&#10;learning instabilities. However, we claim that this can be solved by making&#10;discriminator and generator networks cooperate to produce output sequences&#10;during training. These cooperative outputs, inherently built to obtain higher&#10;discrimination scores, not only provide denser rewards for training, but also&#10;form a more compact artificial set for discriminator training, hence improving&#10;its accuracy and stability. In this paper, we show that our SelfGAN framework,&#10;built on this cooperative principle, outperforms Teacher Forcing and obtains&#10;state-of-the-art results on two challenging tasks, Summarization and Question&#10;Generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.04810" label="2205.04810">
        <attvalues>
          <attvalue for="0" value="The Importance of Context in Very Low Resource Language Modeling" />
          <attvalue for="1" value="  This paper investigates very low resource language model pretraining, when&#10;less than 100 thousand sentences are available. We find that, in very low&#10;resource scenarios, statistical n-gram language models outperform&#10;state-of-the-art neural models. Our experiments show that this is mainly due to&#10;the focus of the former on a local context. As such, we introduce three methods&#10;to improve a neural model's performance in the low-resource setting, finding&#10;that limiting the model's self-attention is the most effective one, improving&#10;on downstream tasks such as NLI and POS tagging by up to 5% for the languages&#10;we test on: English, Hindi, and Turkish.&#10;" />
          <attvalue for="2" value="&#10;With the advent of the Transformer \cite{vaswani2017attention} and masked language model (MLM) pretraining \cite{devlin2018bert}, attention-based neural networks have proven quite effective at a variety of language tasks, provided that large amounts of data are available for pretraining. However, the performance can drop significantly as the number of sentences used for MLM pretraining decreases.&#10;This poses an issue for low-resource settings such as for underrepresented languages, where there is a limited amount of monolingual data. &#10;&#10;Under low-resource conditions, attention-based models have difficulty learning from MLM, and as such statistical language models (SLMs) can outperform neural language models (NLMs). We demonstrate this by using a popular SLM toolkit, KenLM \cite{heafield2011kenlm}, and test its accuracy on the MLM task compared to that of a Transformer model. The results (Table \ref{tab:acc_kenlm_xlm}) show that a trigram SLM is able to outperform the Transformer model by a wide margin for all languages when only 10 thousand sentences are available. &#10;&#10;While an SLM might outperform a neural model on MLM, the neural model has the benefit of being easily transferable to downstream tasks by means of fine-tuning. As such, this paper seeks to determine how we can improve the performance of an NLM to that of an SLM in low-resource scenarios. We investigate three approaches:&#10;\begin{enumerate}&#10; \item Changing the input by limiting the pretraining context size&#10; \item Changing the architecture by limiting the self-attention window&#10; \item Changing the training objective by using soft labels distilled from the SLM&#10;\end{enumerate}&#10;&#10;We motivate and detail these methods in Section~\ref{sect:method}, describe experiment details in Section~\ref{sect:exp}, show and discuss results in Section~\ref{sect:res}, and conclude our work in Section~\ref{sect:concl}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Model Optimization, Artificial Intelligence, Mathematics, Language Model Pretraining, Low Resource Scenarios" />
        </attvalues>
      </node>
      <node id="1912.05372" label="1912.05372">
        <attvalues>
          <attvalue for="0" value="FlauBERT: Unsupervised Language Model Pre-training for French" />
          <attvalue for="1" value="  Language models have become a key step to achieve state-of-the art results in&#10;many different Natural Language Processing (NLP) tasks. Leveraging the huge&#10;amount of unlabeled texts nowadays available, they provide an efficient way to&#10;pre-train continuous word representations that can be fine-tuned for a&#10;downstream task, along with their contextualization at the sentence level. This&#10;has been widely demonstrated for English using contextualized representations&#10;(Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al.,&#10;2018; Devlin et al., 2019; Yang et al., 2019b). In this paper, we introduce and&#10;share FlauBERT, a model learned on a very large and heterogeneous French&#10;corpus. Models of different sizes are trained using the new CNRS (French&#10;National Centre for Scientific Research) Jean Zay supercomputer. We apply our&#10;French language models to diverse NLP tasks (text classification, paraphrasing,&#10;natural language inference, parsing, word sense disambiguation) and show that&#10;most of the time they outperform other pre-training approaches. Different&#10;versions of FlauBERT as well as a unified evaluation protocol for the&#10;downstream tasks, called FLUE (French Language Understanding Evaluation), are&#10;shared to the research community for further reproducible experiments in French&#10;NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.09803" label="1708.09803">
        <attvalues>
          <attvalue for="0" value="Transfer Learning across Low-Resource, Related Languages for Neural&#10;  Machine Translation" />
          <attvalue for="1" value="  We present a simple method to improve neural translation of a low-resource&#10;language pair using parallel data from a related, also low-resource, language&#10;pair. The method is based on the transfer method of Zoph et al., but whereas&#10;their method ignores any source vocabulary overlap, ours exploits it. First, we&#10;split words using Byte Pair Encoding (BPE) to increase vocabulary overlap.&#10;Then, we train a model on the first language pair and transfer its parameters,&#10;including its source word embeddings, to another model and continue training on&#10;the second language pair. Our experiments show that transfer learning helps&#10;word-based translation only slightly, but when used on top of a much stronger&#10;BPE baseline, it yields larger improvements of up to 4.3 BLEU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.09757" label="2007.09757">
        <attvalues>
          <attvalue for="0" value="Mono vs Multilingual Transformer-based Models: a Comparison across&#10;  Several Language Tasks" />
          <attvalue for="1" value="  BERT (Bidirectional Encoder Representations from Transformers) and ALBERT (A&#10;Lite BERT) are methods for pre-training language models which can later be&#10;fine-tuned for a variety of Natural Language Understanding tasks. These methods&#10;have been applied to a number of such tasks (mostly in English), achieving&#10;results that outperform the state-of-the-art. In this paper, our contribution&#10;is twofold. First, we make available our trained BERT and Albert model for&#10;Portuguese. Second, we compare our monolingual and the standard multilingual&#10;models using experiments in semantic textual similarity, recognizing textual&#10;entailment, textual category classification, sentiment analysis, offensive&#10;comment detection, and fake news detection, to assess the effectiveness of the&#10;generated language representations. The results suggest that both monolingual&#10;and multilingual models are able to achieve state-of-the-art and the advantage&#10;of training a single language model, if any, is small.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07412" label="2104.07412">
        <attvalues>
          <attvalue for="0" value="XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation" />
          <attvalue for="1" value="  Machine learning has brought striking advances in multilingual natural&#10;language processing capabilities over the past year. For example, the latest&#10;techniques have improved the state-of-the-art performance on the XTREME&#10;multilingual benchmark by more than 13 points. While a sizeable gap to&#10;human-level performance remains, improvements have been easier to achieve in&#10;some tasks than in others. This paper analyzes the current state of&#10;cross-lingual transfer learning and summarizes some lessons learned. In order&#10;to catalyze meaningful progress, we extend XTREME to XTREME-R, which consists&#10;of an improved set of ten natural language understanding tasks, including&#10;challenging language-agnostic retrieval tasks, and covers 50 typologically&#10;diverse languages. In addition, we provide a massively multilingual diagnostic&#10;suite (MultiCheckList) and fine-grained multi-dataset evaluation capabilities&#10;through an interactive public leaderboard to gain a better understanding of&#10;such models. The leaderboard and code for XTREME-R will be made available at&#10;https://sites.research.google/xtreme and&#10;https://github.com/google-research/xtreme respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.04708" label="1910.04708">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Alignment vs Joint Training: A Comparative Study and A&#10;  Simple Unified Framework" />
          <attvalue for="1" value="  Learning multilingual representations of text has proven a successful method&#10;for many cross-lingual transfer learning tasks. There are two main paradigms&#10;for learning such representations: (1) alignment, which maps different&#10;independently trained monolingual representations into a shared space, and (2)&#10;joint training, which directly learns unified multilingual representations&#10;using monolingual and cross-lingual objectives jointly. In this paper, we first&#10;conduct direct comparisons of representations learned using both of these&#10;methods across diverse cross-lingual tasks. Our empirical results reveal a set&#10;of pros and cons for both methods, and show that the relative performance of&#10;alignment versus joint training is task-dependent. Stemming from this analysis,&#10;we propose a simple and novel framework that combines these two previously&#10;mutually-exclusive approaches. Extensive experiments demonstrate that our&#10;proposed framework alleviates limitations of both approaches, and outperforms&#10;existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We&#10;further show that this framework can generalize to contextualized&#10;representations such as Multilingual BERT, and produces state-of-the-art&#10;results on the CoNLL cross-lingual NER benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Word embeddings \cite{mikolov2013efficient,ruder2019survey} are a key ingredient to achieving success in monolingual NLP tasks. However, directly using word embeddings independently trained for each language may cause negative transfer \cite{wang2019characterizing} in cross-lingual transfer tasks. In order to capture the cross-lingual mapping, a rich body of existing work relying on cross-lingual supervision, including bilingual dictionaries \cite{mikolov2013efficient, faruqui2014improving, artetxe2016learning, xing2015normalized, duong2016learning, gouws2015simple,joulin-etal-2018-loss}, sentence-aligned corpora \cite{kovcisky2014learning, hermann2014multilingual, gouws2015bilbowa} and document-aligned corpora \cite{vulic2016bilingual, sogaard2015inverted}. &#10;&#10;Besides, unsupervised alignment methods aim to eliminate the requirement for cross-lingual supervision. Early work of \cite{cao2016distribution} matches the mean and the standard deviation of two embedding spaces after alignment. \cite{barone2016towards,zhang2017adversarial,zhang2017earth,conneau2017word} adapted a generative adversarial network (GAN) \cite{goodfellow2014generative} to make the distributions of two word embedding spaces indistinguishable. Follow-up works improve upon GAN-based training for better stability and robustness by introducing Sinkhorn distance \cite{xu2018unsupervised}, by stochastic self-training \cite{artetxe2018robust}, or by introducing latent variables \cite{dou2018unsupervised}.&#10;&#10;While alignment methods utilize embeddings trained independently on different languages, joint training methods train word embeddings at the same time. &#10;\cite{klementiev2012inducing} train a bilingual dictionary-based regularization term jointly with monolingual language model objectives while \cite{kovcisky2014learning} defines the cross-lingual regularization with the parallel corpus. Another branch of methods \cite{xiao2014distributed,gouws2015simple,ammar2016massively,duong2016learning} build a pseudo-bilingual corpus by randomly replacing words in monolingual corpus with their translations and use monolingual word embedding algorithms to induce bilingual representations.&#10;The unsupervised joint method by \cite{lample2019cross} simply exploit words that share the same surface form as bilingual ``supervision'' and directly train a shared set of embedding with joint vocabulary.&#10;Recently, unsupervised joint training of contextualized word embeddings through the form of multilingual language model pretraining using shared subword vocabularies has produced state-of-the-art results on various benchmarks \cite{devlin2018bert,artetxe2018massively,lample2019cross,pires19acl,wu19emnlp}.&#10;&#10;A concurrent work by \cite{ormazabal2019analyzing} also compares alignment and joint method in the bilingual lexicon induction task. Different from their setup which only tests on supervised settings, we conduct analysis across various tasks and experiment with both supervised and unsupervised conditions. While \cite{ormazabal2019analyzing} suggests that the combination of the alignment and joint model could potentially advance the state-of-art of both worlds, we propose such a framework and empirically verify its effectiveness on various tasks and settings.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2211.03154" label="2211.03154">
        <attvalues>
          <attvalue for="0" value="On the Domain Adaptation and Generalization of Pretrained Language&#10;  Models: A Survey" />
          <attvalue for="1" value="  Recent advances in NLP are brought by a range of large-scale pretrained&#10;language models (PLMs). These PLMs have brought significant performance gains&#10;for a range of NLP tasks, circumventing the need to customize complex designs&#10;for specific tasks. However, most current work focus on finetuning PLMs on a&#10;domain-specific datasets, ignoring the fact that the domain gap can lead to&#10;overfitting and even performance drop. Therefore, it is practically important&#10;to find an appropriate method to effectively adapt PLMs to a target domain of&#10;interest. Recently, a range of methods have been proposed to achieve this&#10;purpose. Early surveys on domain adaptation are not suitable for PLMs due to&#10;the sophisticated behavior exhibited by PLMs from traditional models trained&#10;from scratch and that domain adaptation of PLMs need to be redesigned to take&#10;effect. This paper aims to provide a survey on these newly proposed methods and&#10;shed light in how to apply traditional machine learning methods to newly&#10;evolved and future technologies. By examining the issues of deploying PLMs for&#10;downstream tasks, we propose a taxonomy of domain adaptation approaches from a&#10;machine learning system view, covering methods for input augmentation, model&#10;optimization and personalization. We discuss and compare those methods and&#10;suggest promising future research directions.&#10;" />
          <attvalue for="2" value="&#10;N{atural}&#10;Language Processing (NLP) is undergoing a paradigm shift with the open-source of large-scale pretrained language models (PLMs), such as GPT \cite{radford2018gpt,radford2019language}, BERT \cite{devlin-etal-2019-bert}, ALBERT \cite{Lan2020ALBERT}, RoBERTa \cite{liu2020roberta}, BART \cite{lewis-etal-2020-bart}, and T5 \cite{Reffel2020T5}. Finetuing PLMs have demonstrated to be a strong baseline on a range of downstream tasks, mostly outperforming previous state-of-the-art task-specific models, making it the de-facto standard in NLP. Having Transformer \cite{NIPS2017_3f5ee243} as their basic neural architecture, these PLMs are trained on broad data using different self-supervised learning tasks at scale. They play a central role as the foundation model of AI \cite{bommasani2021-foundation} for their knowledgeable yet incomplete character. The downside of PLMs lies in the lack of portability to real-world domains. PLMs are pretrained by the universal language data, leaving the domain challenges unresolved. A bunch of evidence have surfaced showing that their performance can degrade when they are applied to a narrower domain where data varies substantially from the pretraining corpus \cite{thompson-etal-2019-overcoming,araci2020finbert,chalkidis-etal-2020-legal,miller-etal-2021-domain}. The mismatch between the pretraining and adaptation data distributions makes PLMs struggle to be widely adopted in practice. &#10;&#10;A direct approach to improve generalization performance in machine learning is to enlarge the labeled training data. However, acquiring labels for every task is expensive and time-consuming. To this end, enabling PLMs with domain adaptation (DA) \cite{pan2009survey,weiss2016survey,ramponi-plank-2020-neural,saunders2022domain} which reuses labeled data from related source domains to boost performance on the target domain is necessary. Due to the semantic gap between the embedding spaces of different domains \cite{drift-2021}, directly fitting a single PLM on non-identical domains is suboptimal \cite{bilen2017universal} and may even incur negative transfer due to the domain shifts \cite{lekhtman-etal-2021-dilbert}. Moreover, performance gains on the target domain can come at the expense of general-domain performance, which is recognized as the catastrophic forgetting problem \cite{mccloskey1989catastrophic}. The effectiveness of domain adaptation largely depends on how well the target domains are represented in the pretraining corpora \cite{radford2019language}. To conduct successful knowledge transfer, a range of domain adaptation methods have been proposed based on different assumptions of the given data and the training setting. However, traditional domain adaptation methods that work with shallow neural networks or non-pretrained language models can be unfavorable to PLMs \cite{ryu2022knowledge}. For example, \cite{wright-augenstein-2020-transformer,karouzos-etal-2021-udalm} find that domain adversarial training on top of BERT is unstable and has little effect on cross-domain performance, suggesting that proper design of domain adaptation methods is necessary for PLMs.&#10;&#10;In addition, the long-standing challenge of data scarcity hinders the deployment of many domain-specific systems. Developing domain adaptation and generalization methods for PLMs is promising particularly for data-hungry NLP tasks such as abstractive summarization \cite{yu-etal-2021-adaptsum}. These methods can be incorporated into the pipeline of downstream tasks as easy as playing with &quot;Lego&quot;. However, there isn't one-fit-all PLM or on-the-fly domain adaptation methods off the shelf. Through comprehensive review of related literature, we argue that an appropriate choice of domain adaptation methods or a mixture of them can help to avoid performance degradation of PLMs in new domains. For this purpose, this survey aims to provide future researchers and engineers with a road map for accelerating the deployment of PLMs to real-world problems. In the end, we highlight some interesting yet rather meaningful research directions to empower and apply PLMs. &#10;&#10;A Categorization for domain adaptation of PLMs. We propose a taxonomy (Table \ref{tab:taxonomy}) from a machine learning system view, covering data augmentation, model optimization and personalization techniques. The mainstream approaches focus on data augmentation and model optimization or a combination of them. Personalization has been less explored and most of the methods we surveyed are either recently emerging methods or a resurgence of traditional machine learning approaches. We deem this category is promising for practical adoption of PLMs at scale. &#10;&#10;Other surveys. Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys \cite{pan2009survey,weiss2016survey} provide a holistic view including but not limited to DA; DA for visual applications \cite{patel2015visual,csurka2017comprehensive,wang2018deep}; multiple-source domain adaptation (MDA) \cite{mansour2008domain,sun2015survey}; neural UDA for NLP applications based on shallow and non-pretrained language models \cite{ramponi-plank-2020-neural}; DA and MDA for machine translation \cite{saunders2022domain}; taxonomy of PLMs \cite{qiu2020pre} and comprehensive guide to use PLMs for NLP tasks \cite{min2021recent} and particularly for text generation tasks \cite{li2021pretrained}; parameter-efficient adaptation methods for PLMs \cite{ding2022delta}.&#10;&#10;Contributions. Adapting PLMs to specific domains is practically important as PLMs only hold general knowledge. We aim to provide future researchers and engineers with a handbook picturing a systematic view of possible techniques to adapt PLMs to real-world applications more easily. Previous surveys on PLMs focus on the pretraining techniques, surveys on domain adaptation focus on shallow and plain neural networks, and surveys on transfer learning focus on more broad assumptions or settings. In this paper, we systematically examine all the possible solutions for exploiting PLMs to new domains. In addition, we propose a new category named personalization which focus on the setting where the same PLMs are expected to adapt to multiple target domains. We explicitly visualize the assumptions, techniques and PLMs adopted in the literature in Table \ref{tab:visualization} to provide a big picture of the current state of research. Finally, we outline challenges and future research directions. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Domain Adaptation Methods, Machine Learning, Computer Science, Linguistics, Machine Learning Taxonomy, Artificial Intelligence, Mathematics, Natural Language Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="1908.10063" label="1908.10063">
        <attvalues>
          <attvalue for="0" value="FinBERT: Financial Sentiment Analysis with Pre-trained Language Models" />
          <attvalue for="1" value="  Financial sentiment analysis is a challenging task due to the specialized&#10;language and lack of labeled data in that domain. General-purpose models are&#10;not effective enough because of the specialized language used in a financial&#10;context. We hypothesize that pre-trained language models can help with this&#10;problem because they require fewer labeled examples and they can be further&#10;trained on domain-specific corpora. We introduce FinBERT, a language model&#10;based on BERT, to tackle NLP tasks in the financial domain. Our results show&#10;improvement in every measured metric on current state-of-the-art results for&#10;two financial sentiment analysis datasets. We find that even with a smaller&#10;training set and fine-tuning only a part of the model, FinBERT outperforms&#10;state-of-the-art machine learning methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.02559" label="2010.02559">
        <attvalues>
          <attvalue for="0" value="LEGAL-BERT: The Muppets straight out of Law School" />
          <attvalue for="1" value="  BERT has achieved impressive performance in several NLP tasks. However, there&#10;has been limited investigation on its adaptation guidelines in specialised&#10;domains. Here we focus on the legal domain, where we explore several approaches&#10;for applying BERT models to downstream legal tasks, evaluating on multiple&#10;datasets. Our findings indicate that the previous guidelines for pre-training&#10;and fine-tuning, often blindly followed, do not always generalize well in the&#10;legal domain. Thus we propose a systematic investigation of the available&#10;strategies when applying BERT in specialised domains. These are: (a) use the&#10;original BERT out of the box, (b) adapt BERT by additional pre-training on&#10;domain-specific corpora, and (c) pre-train BERT from scratch on domain-specific&#10;corpora. We also propose a broader hyper-parameter search space when&#10;fine-tuning for downstream tasks and we release LEGAL-BERT, a family of BERT&#10;models intended to assist legal NLP research, computational law, and legal&#10;technology applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.00632" label="2006.00632">
        <attvalues>
          <attvalue for="0" value="Neural Unsupervised Domain Adaptation in NLP---A Survey" />
          <attvalue for="1" value="  Deep neural networks excel at learning from labeled data and achieve&#10;state-of-the-art resultson a wide array of Natural Language Processing tasks.&#10;In contrast, learning from unlabeled data, especially under domain shift,&#10;remains a challenge. Motivated by the latest advances, in this survey we review&#10;neural unsupervised domain adaptation techniques which do not require labeled&#10;target domain data. This is a more challenging yet a more widely applicable&#10;setup. We outline methods, from early traditional non-neural methods to&#10;pre-trained model transfer. We also revisit the notion of domain, and we&#10;uncover a bias in the type of Natural Language Processing tasks which received&#10;most attention. Lastly, we outline future directions, particularly the broader&#10;need for out-of-distribution generalization of future NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.06951" label="2104.06951">
        <attvalues>
          <attvalue for="0" value="Domain Adaptation and Multi-Domain Adaptation for Neural Machine&#10;  Translation: A Survey" />
          <attvalue for="1" value="  The development of deep learning techniques has allowed Neural Machine&#10;Translation (NMT) models to become extremely powerful, given sufficient&#10;training data and training time. However, systems struggle when translating&#10;text from a new domain with a distinct style or vocabulary. Fine-tuning on&#10;in-domain data allows good domain adaptation, but requires sufficient relevant&#10;bilingual data. Even if this is available, simple fine-tuning can cause&#10;overfitting to new data and `catastrophic forgetting' of previously learned&#10;behaviour.&#10;  We concentrate on robust approaches to domain adaptation for NMT,&#10;particularly where a system may need to translate across multiple domains. We&#10;divide techniques into those revolving around data selection or generation,&#10;model architecture, parameter adaptation procedure, and inference procedure. We&#10;finally highlight the benefits of domain adaptation and multi-domain adaptation&#10;techniques to other lines of NMT research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11478" label="2010.11478">
        <attvalues>
          <attvalue for="0" value="Knowledge Distillation for BERT Unsupervised Domain Adaptation" />
          <attvalue for="1" value="  A pre-trained language model, BERT, has brought significant performance&#10;improvements across a range of natural language processing tasks. Since the&#10;model is trained on a large corpus of diverse topics, it shows robust&#10;performance for domain shift problems in which data distributions at training&#10;(source data) and testing (target data) differ while sharing similarities.&#10;Despite its great improvements compared to previous models, it still suffers&#10;from performance degradation due to domain shifts. To mitigate such problems,&#10;we propose a simple but effective unsupervised domain adaptation method,&#10;adversarial adaptation with distillation (AAD), which combines the adversarial&#10;discriminative domain adaptation (ADDA) framework with knowledge distillation.&#10;We evaluate our approach in the task of cross-domain sentiment classification&#10;on 30 domain pairs, advancing the state-of-the-art performance for unsupervised&#10;domain adaptation in text sentiment classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.07806" label="2009.07806">
        <attvalues>
          <attvalue for="0" value="Transformer Based Multi-Source Domain Adaptation" />
          <attvalue for="1" value="  In practical machine learning settings, the data on which a model must make&#10;predictions often come from a different distribution than the data it was&#10;trained on. Here, we investigate the problem of unsupervised multi-source&#10;domain adaptation, where a model is trained on labelled data from multiple&#10;source domains and must make predictions on a domain for which no labelled data&#10;has been seen. Prior work with CNNs and RNNs has demonstrated the benefit of&#10;mixture of experts, where the predictions of multiple domain expert classifiers&#10;are combined; as well as domain adversarial training, to induce a domain&#10;agnostic representation space. Inspired by this, we investigate how such&#10;methods can be effectively applied to large pretrained transformer models. We&#10;find that domain adversarial training has an effect on the learned&#10;representations of these models while having little effect on their&#10;performance, suggesting that large transformer-based models are already&#10;relatively robust across domains. Additionally, we show that mixture of experts&#10;leads to significant performance improvements by comparing several variants of&#10;mixing functions, including one novel mixture based on attention. Finally, we&#10;demonstrate that the predictions of large pretrained transformer based domain&#10;experts are highly homogenous, making it challenging to learn effective&#10;functions for mixing their predictions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07078" label="2104.07078">
        <attvalues>
          <attvalue for="0" value="UDALM: Unsupervised Domain Adaptation through Language Modeling" />
          <attvalue for="1" value="  In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained&#10;language models for downstream tasks. We introduce UDALM, a fine-tuning&#10;procedure, using a mixed classification and Masked Language Model loss, that&#10;can adapt to the target domain distribution in a robust and sample efficient&#10;manner. Our experiments show that performance of models trained with the mixed&#10;loss scales with the amount of available target data and the mixed loss can be&#10;effectively used as a stopping criterion during UDA training. Furthermore, we&#10;discuss the relationship between A-distance and the target error and explore&#10;some limitations of the Domain Adversarial Training approach. Our method is&#10;evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset,&#10;yielding $91.74\%$ accuracy, which is an $1.11\%$ absolute improvement over the&#10;state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Traditionally, UDA has been performed using pseudo-labeling approaches. Pseudo-labeling techniques are semi-supervised algorithms that either use the same model (self-training) \cite{yarowsky-1995-unsupervised, mcclosky-etal-2006-reranking, abney2007semisupervised} or multiple ensembles of models (tri-training) \cite{zhou2005tri, sogaard-2010-simple} in order to produce pseudo-labels for the target unlabeled data. \cite{saito2017asymmetric} proposed an asymmetric tri-training approach. \cite{ruder-plank-2018-strong} introduced a multi-task tri-training method. \cite{rotman-reichart-2019-deep} and \cite{lim2020semi} study pseudo-labeling with contextualized word representations. \cite{ye_feature_2020} combine self-training with XLM-R \cite{conneau-etal-2020-unsupervised} to reduce the produced label noise and propose CFd, class aware feature self-distillation.&#10;&#10;Another line of UDA research includes pivot-based methods, focusing on extracting cross-domain features. Structural Correspondence Learning (SCL) \cite{blitzer-etal-2006-domain} and Spectral Feature Alignment \cite{pan2010cross} aim to find domain-invariant features (pivots) to learn a mapping between two domain distributions. \cite{ziser-reichart-2017-neural, ziser-reichart-2018-pivot, ziser-reichart-2019-task} combine SCL with neural network architectures and language modeling. \cite{miller-2019-simplified} propose to jointly learn the task and pivots. \cite{li2018hierarchical} learn pivots with hierarchical attention networks. Pivot-based methods have also been used in conjunction with BERT \cite{ben2020perl}.&#10;&#10;Domain adversarial training is a dominant approach for UDA \cite{ramponi-and-plank-2020-neural}, inspired by the theory for learning from different domains introduced in \cite{ben2007analysis, ben2010theory}. \cite{ganin2016domain, ganinlempitsky2015} propose to learn a task while not being able to distinguish if samples come from the source or the target distribution, through use of an adversarial cost. This approach has been adopted for a diverse set of problems, e.g. sentiment analysis, tweet classification and universal dependency parsing \cite{li-etal-2018-whats, alam-etal-2018-domain, sato-etal-2017-adversarial}. \cite{du-etal-2020-adversarial} pose domain adversarial training in the context of BERT models. \cite{zhao2018adversarial} propose multi-source domain adversarial networks. \cite{guo-etal-2018-multi} propose a mixture-of-experts approach for multi-source UDA. \cite{guo2020multi} explore distance measures as additional losses and use them to construct dynamic multi-armed bandit controller for the source domains. \cite{shen2017wasserstein} learn domain invariant features via Wasserstein distance. \cite{bousmalis2016domain} introduce domain seperation networks with private and shared encoders. &#10;&#10;Unsupervised pretraining on domain-specific corpora can be an effective adaptation process.&#10;For example BioBERT \cite{lee2020biobert} and SciBERT \cite{beltagy-etal-2019-scibert} are specialized BERT variants, where pretraining is extended on large amounts of biomedical and scientific corpora respectively. \cite{sun2019fine} propose continuing the pretraining of BERT with target domain data and multitask learning using relevant tasks for BERT fine-tuning. \cite{xu-etal-2019-bert} introduce a review reading comprehension task and a post-training approach for BERT with an auxiliary loss on a question-answering task. Continuing pretraining on multiple phases, from general to domain specific (DAPT) and task specific data (TAPT), further improves performance of pretrained language models, as shown by \cite{gururangan-etal-2020-dont}. \cite{han-eisenstein-2019-unsupervised} propose AdaptaBERT, which includes a second phase of unsupervised pretraining, in order to use BERT in a unsupervised domain adaptation context.&#10;&#10;Recent works have highlighted the merits of using Language Modeling as an auxiliary task during fine-tuning. \cite{chronopoulou-etal-2019-embarrassingly} use an auxiliary LM loss to avoid catastrophic forgetting in transfer learning and \cite{jia-etal-2019-cross} adopt this approach for cross-domain named-entity recognition. We draw inspiration from these approaches and utilize auxiliary Language Modeling for UDA.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2103.11332" label="2103.11332">
        <attvalues>
          <attvalue for="0" value="AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  State-of-the-art abstractive summarization models generally rely on extensive&#10;labeled data, which lowers their generalization ability on domains where such&#10;data are not available. In this paper, we present a study of domain adaptation&#10;for the abstractive summarization task across six diverse target domains in a&#10;low-resource setting. Specifically, we investigate the second phase of&#10;pre-training on large-scale generative models under three different settings:&#10;1) source domain pre-training; 2) domain-adaptive pre-training; and 3)&#10;task-adaptive pre-training. Experiments show that the effectiveness of&#10;pre-training is correlated with the similarity between the pre-training data&#10;and the target domain task. Moreover, we find that continuing pre-training&#10;could lead to the pre-trained model's catastrophic forgetting, and a learning&#10;method with less forgetting can alleviate this issue. Furthermore, results&#10;illustrate that a huge gap still exists between the low-resource and&#10;high-resource settings, which highlights the need for more advanced domain&#10;adaptation methods for the abstractive summarization task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.03601" label="1802.03601">
        <attvalues>
          <attvalue for="0" value="Deep Visual Domain Adaptation: A Survey" />
          <attvalue for="1" value="  Deep domain adaption has emerged as a new learning technique to address the&#10;lack of massive amounts of labeled data. Compared to conventional methods,&#10;which learn shared feature subspaces or reuse important source instances with&#10;shallow representations, deep domain adaption methods leverage deep networks to&#10;learn more transferable representations by embedding domain adaptation in the&#10;pipeline of deep learning. There have been comprehensive surveys for shallow&#10;domain adaption, but few timely reviews the emerging deep learning based&#10;methods. In this paper, we provide a comprehensive survey of deep domain&#10;adaptation methods for computer vision applications with four major&#10;contributions. First, we present a taxonomy of different deep domain adaption&#10;scenarios according to the properties of data that define how two domains are&#10;diverged. Second, we summarize deep domain adaption approaches into several&#10;categories based on training loss, and analyze and compare briefly the&#10;state-of-the-art methods under these categories. Third, we overview the&#10;computer vision applications that go beyond image classification, such as face&#10;recognition, semantic segmentation and object detection. Fourth, some potential&#10;deficiencies of current methods and several future directions are highlighted.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.05273" label="2201.05273">
        <attvalues>
          <attvalue for="0" value="Pretrained Language Models for Text Generation: A Survey" />
          <attvalue for="1" value="  Text Generation aims to produce plausible and readable text in a human&#10;language from input data. The resurgence of deep learning has greatly advanced&#10;this field, in particular, with the help of neural generation models based on&#10;pre-trained language models (PLMs). Text generation based on PLMs is viewed as&#10;a promising approach in both academia and industry. In this paper, we provide a&#10;survey on the utilization of PLMs in text generation. We begin with introducing&#10;three key aspects of applying PLMs to text generation: 1) how to encode the&#10;input into representations preserving input semantics which can be fused into&#10;PLMs; 2) how to design an effective PLM to serve as the generation model; and&#10;3) how to effectively optimize PLMs given the reference text and to ensure that&#10;the generated texts satisfy special text properties. Then, we show the major&#10;challenges arisen in these aspects, as well as possible solutions for them. We&#10;also include a summary of various useful resources and typical text generation&#10;applications based on PLMs. Finally, we highlight the future research&#10;directions which will further improve these PLMs for text generation. This&#10;comprehensive survey is intended to help researchers interested in text&#10;generation problems to learn the core concepts, the main techniques and the&#10;latest developments in this area based on PLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.00656" label="2205.00656">
        <attvalues>
          <attvalue for="0" value="Debiased Contrastive Learning of Unsupervised Sentence Representations" />
          <attvalue for="1" value="  Recently, contrastive learning has been shown to be effective in improving&#10;pre-trained language models (PLM) to derive high-quality sentence&#10;representations. It aims to pull close positive examples to enhance the&#10;alignment while push apart irrelevant negatives for the uniformity of the whole&#10;representation space. However, previous works mostly adopt in-batch negatives&#10;or sample from training data at random. Such a way may cause the sampling bias&#10;that improper negatives (e.g. false negatives and anisotropy representations)&#10;are used to learn sentence representations, which will hurt the uniformity of&#10;the representation space. To address it, we present a new framework&#10;\textbf{DCLR} (\underline{D}ebiased \underline{C}ontrastive&#10;\underline{L}earning of unsupervised sentence \underline{R}epresentations) to&#10;alleviate the influence of these improper negatives. In DCLR, we design an&#10;instance weighting method to punish false negatives and generate noise-based&#10;negatives to guarantee the uniformity of the representation space. Experiments&#10;on seven semantic textual similarity tasks show that our approach is more&#10;effective than competitive baselines. Our code and data are publicly available&#10;at the link: \textcolor{blue}{\url{https://github.com/RUCAIBox/DCLR}}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;In this section, we review the related work from the following three aspects. &#10;&#10;Sentence Representation Learning.&#10;Learning universal sentence representations~\cite{DBLP:conf/nips/KirosZSZUTF15,DBLP:conf/naacl/HillCK16} is the key to the success of various downstream tasks.&#10;Previous works can be roughly categorized into supervised~\cite{DBLP:conf/emnlp/ConneauKSBB17,DBLP:conf/emnlp/CerYKHLJCGYTSK18} and unsupervised approaches~\cite{DBLP:conf/naacl/HillCK16,DBLP:conf/emnlp/LiZHWYL20}.&#10;Supervised approaches rely on annotated datasets (e.g., NLI~\cite{DBLP:conf/emnlp/BowmanAPM15,DBLP:conf/naacl/WilliamsNB18}) to train the sentence encoder~\cite{DBLP:conf/emnlp/CerYKHLJCGYTSK18,DBLP:conf/emnlp/ReimersG19}.&#10;Unsupervised approaches consider deriving sentence representations without labeled datasets, e.g., &#10;pooling word2vec embeddings~\cite{DBLP:conf/nips/MikolovSCCD13}.&#10;Recently, to leverage the strong potential of PLMs~\cite{DBLP:conf/naacl/DevlinCLT19}, several works propose to alleviate the anisotropy problem~\cite{DBLP:conf/emnlp/Ethayarajh19,DBLP:conf/emnlp/LiZHWYL20} of PLMs via special strategies, e.g., flow-based approach~\cite{DBLP:conf/emnlp/LiZHWYL20} and whitening method~\cite{DBLP:journals/corr/abs-2104-01767}.&#10;Besides, contrastive learning~\cite{DBLP:journals/corr/abs-2012-15466,DBLP:conf/emnlp/GaoYC21} has been used to refine the representations of PLMs.&#10;&#10;Contrastive Learning.&#10;Contrastive learning has been originated in the computer vision~\cite{DBLP:conf/cvpr/HadsellCL06,DBLP:conf/cvpr/He0WXG20} and information retrieval~\cite{DBLP:conf/cikm/BianZZCHYW21,DBLP:conf/wsdm/ZhouZZWJ022} field with significant performance improvement. &#10;Usually, it relies on data augmentation strategies such as random cropping and image rotation~\cite{DBLP:conf/icml/ChenK0H20,DBLP:conf/acl/YanLWZWX20} to produce a set of semantically related positive examples for learning, and randomly samples negatives from the batch or whole dataset.&#10;For sentence representation learning, contrastive learning can achieve a better balance between alignment and uniformity in semantic representation space.&#10;Several works further adopt back translation~\cite{DBLP:journals/corr/abs-2005-12766}, token shuffling~\cite{DBLP:conf/acl/YanLWZWX20} and dropout~\cite{DBLP:conf/emnlp/GaoYC21} to augment positive examples for sentence representation learning.&#10;However, the quality of the randomly sampled negatives is seldom studied.&#10;&#10;Virtual Adversarial Training.&#10;Virtual adversarial training (VAT)~\cite{DBLP:journals/pami/MiyatoMKI19,DBLP:conf/iclr/KurakinGB17a} perturbs a given input with learnable noises to maximize the divergence of the model’s prediction with the original label, then utilizes the perturbed examples to improve the generalization~\cite{DBLP:conf/iclr/MiyatoDG17,DBLP:conf/iclr/MadryMSTV18}.&#10;A class of VAT methods can be formulated into solving a min-max problem, which can be achieved by multiple projected gradient ascent steps~\cite{DBLP:conf/nips/QinMGKDFDSK19}.&#10;In the NLP field, several studies incorporate adversarial perturbations in the embedding layer, and show its effectiveness on text classification~\cite{DBLP:conf/iclr/MiyatoDG17}, machine translation~\cite{DBLP:conf/coling/SunWCLUSZ20}, and natural language understanding~\cite{DBLP:conf/acl/JiangHCLGZ20} tasks.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Contrastive Learning, Sentence Representations, Linguistics, Language Models, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2206.00059" label="2206.00059">
        <attvalues>
          <attvalue for="0" value="A Mixture-of-Expert Approach to RL-based Dialogue Management" />
          <attvalue for="1" value="  Despite recent advancements in language models (LMs), their application to&#10;dialogue management (DM) problems and ability to carry on rich conversations&#10;remain a challenge. We use reinforcement learning (RL) to develop a dialogue&#10;agent that avoids being short-sighted (outputting generic utterances) and&#10;maximizes overall user satisfaction. Most existing RL approaches to DM train&#10;the agent at the word-level, and thus, have to deal with a combinatorially&#10;complex action space even for a medium-size vocabulary. As a result, they&#10;struggle to produce a successful and engaging dialogue even if they are&#10;warm-started with a pre-trained LM. To address this issue, we develop a&#10;RL-based DM using a novel mixture of expert language model (MoE-LM) that&#10;consists of (i) a LM capable of learning diverse semantics for conversation&#10;histories, (ii) a number of {\em specialized} LMs (or experts) capable of&#10;generating utterances corresponding to a particular attribute or personality,&#10;and (iii) a RL-based DM that performs dialogue planning with the utterances&#10;generated by the experts. Our MoE approach provides greater flexibility to&#10;generate sensible utterances with different intents and allows RL to focus on&#10;conversational-level DM. We compare it with SOTA baselines on open-domain&#10;dialogues and demonstrate its effectiveness both in terms of the diversity and&#10;sensibility of the generated utterances and the overall DM performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the tremendous advancements in natural language understanding and generation, increasing attention has been directed to construct intelligent dialogue agents that can carry out rich and engaging conversations with human users. Such interactions can be quite open-ended, contain many different topics, and often involve an underlying task, such as building a rapport, negotiation, information exchange, and recommendation. Therefore, to satisfy the user, a good dialogue agent should not only generate natural responses, but also be capable of pursuing the task's objectives and adapting to the user's feedback on-the-fly.&#10;&#10;A standard solution is to train the dialogue agent using behavioral cloning, where the agent is a language model (LM) that imitates the utterances in the training set~\cite{gavsic2011line,fatemi2016policy}. By leveraging deep neural networks, e.g.,~RNNs~\cite{ sutskever2014sequence} and Transformers~\cite{vaswani2017attention}, a LM encodes the conversation to a low-dimensional dialogue state and predicts an utterance, but steering such generation for particular purposes remains an open question. Several works studied ways to fine-tune a LM to generate texts with specific contexts~\cite{ziegler2019fine, ficler2017controlling}. Other results learned a single steerable LM that is capable of generating utterances for multiple specific intents~\cite{gu2017trainable, chen2018stable, subramani2019can, dathathri2019plug}. While these LMs produce fluent and relevant responses, it is unclear how to control them to systematically pursue goals during multi-turn dialogue conversations.&#10;&#10;Another popular approach is to view dialogue management (DM) as a control problem and use reinforcement learning (RL) to optimize the agent's policy (which is often a LM itself). Using RL for dialogue systems has a long history. Earlier work relies on specific, hand-crafted semantic states~\cite{levin1997stochastic, singh2002optimizing, walker2000application} or partially observable belief states~\cite{williams2007partially, young2010hidden}, in which the agent chooses the best hand-crafted dialogue act at each turn, with the goal of either satisfying the user~\cite{shah2018bootstrapping}, completing the task~\cite{shi2018sentiment}, or responding to the user's query~\cite{serban2017deep}. However, the application of these approaches is limited to problems whose action space can be captured&#10;by hand-crafted representations, and they cannot handle complex conversations. On the other hand, more recent approaches use deep learning to extract semantic representations from conversation histories, treat these representations as dialogue belief states, and apply RL to learn a word-level generative DM agent~\cite{jaques2019way, li2016deep, li2017adversarial, shin2020generating}. However, since there are innumerable possibilities of language utterances, and thus, the action space of the RL problem is extremely large, the agent often performs planning poorly and generates incomprehensible utterances~\cite{zhao2019rethinking}. Another issue is that RL only optimizes a scalar reward, while the aforementioned methods often need to optimize for both the quality of the generated utterance, e.g.,~ease of answering~\cite{li2016deep} and fluency~\cite{li2017adversarial, li2019dialogue}, and the goal, e.g.,~conversation length \cite{zhou2020design}, user's sentiment~\cite{hancock2019learning}, and task completion~\cite{verma2022chai}. Moreover, defining the reward as weighted combination of these metrics is not ideal, since the hand-picked weights do not often reflect the underlying success criteria. &#10;&#10;To address the above issues related to using RL in dialogue management (DM) systems, we propose an RL-based DM agent using a novel {mixture of expert} (MoE) approach. Our MoE approach is based on a mixture of expert language model (MoE-LM), which consists of three main components: {1)} a LM (a probabilistic encoder and a decoder) capable of learning diverse semantics for conversation histories, and as a result generating diverse utterances, which we refer to as the {primitive} LM or $LM_0$, {2)} a number of {specialized} LMs (or {experts}), $\{LM_i\}_{i=1}^m$, that each is constructed using the latent space learned by $LM_0$, but has been trained such that it is capable of generating utterances corresponding to a certain intent or personality, and {3)} an RL-based dialogue manager (DM) that at each turn, given the latent state shared by the experts $\{LM_i\}_{i=0}^m$ and the utterance action(s) they suggest, chooses one among them for the agent to execute. Our MoE-LM can be seen as a special case of hierarchical LMs (e.g.,~\cite{serban2017deep,zhao2019rethinking,saleh2020hierarchical}), but it is different than them because it learns both the LMs (experts) and the DM. Moreover, the DM in MoE-LM is a policy conditioned on both the latent state and the actions suggested by the experts, and not just the state as it is common in hierarchical RL. The primitive LM ($LM_0$) plays an important role in this model because it learns diverse semantics for conversation histories and allows the agent to generate a wide variety of utterances. This diversity is also shared with the specialized LMs (experts) and gives them flexibility in generating their (more) specialized utterances. Another important feature of MoE-LM is its modularity that facilitates adding and removing specialized LMs (experts). Moreover, this hierarchical architecture allows us to solve an RL problem with much smaller state and action spaces, which is quite important in the quality of the learned policy. Finally, since the candidate utterances are generated by experts with different intents, instead of combining all agent-user signals into a single RL reward, our DM agent can focus on optimizing the specific goal of the conversation task. &#10;&#10;We start the paper with a brief introduction of LMs and the use of Markov decision processes (MDPs) in modeling dialogue management problems in Section~\ref{sec:prelim}. We then describe the overall architecture of our MoE-LM in Section~\ref{sec:moe}, followed by the detailed implementation of each of its three main components (described in the above paragraph) in Sections~\ref{sec:rep_l} to~\ref{sec:cm}. Finally, in Section~\ref{sec:experiments}, we demonstrate the effectiveness of our MoE-LM in open-domain dialogues, in terms of both its ability to generate diverse and sensible utterances and its overall DM performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Models, Artificial Intelligence, Dialogue Management, Natural Language Processing, Reinforcement Learning" />
        </attvalues>
      </node>
      <node id="1702.02429" label="1702.02429">
        <attvalues>
          <attvalue for="0" value="Trainable Greedy Decoding for Neural Machine Translation" />
          <attvalue for="1" value="  Recent research in neural machine translation has largely focused on two&#10;aspects; neural network architectures and end-to-end learning algorithms. The&#10;problem of decoding, however, has received relatively little attention from the&#10;research community. In this paper, we solely focus on the problem of decoding&#10;given a trained neural machine translation model. Instead of trying to build a&#10;new decoding algorithm for any specific decoding objective, we propose the idea&#10;of trainable decoding algorithm in which we train a decoding algorithm to find&#10;a translation that maximizes an arbitrary decoding objective. More&#10;specifically, we design an actor that observes and manipulates the hidden state&#10;of the neural machine translation decoder and propose to train it using a&#10;variant of deterministic policy gradient. We extensively evaluate the proposed&#10;algorithm using four language pairs and two decoding objectives and show that&#10;we can indeed train a trainable greedy decoder that generates a better&#10;translation (in terms of a target decoding objective) with minimal&#10;computational overhead.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.10731" label="1804.10731">
        <attvalues>
          <attvalue for="0" value="Sentiment Adaptive End-to-End Dialog Systems" />
          <attvalue for="1" value="  End-to-end learning framework is useful for building dialog systems for its&#10;simplicity in training and efficiency in model updating. However, current&#10;end-to-end approaches only consider user semantic inputs in learning and&#10;under-utilize other user information. Therefore, we propose to include user&#10;sentiment obtained through multimodal information (acoustic, dialogic and&#10;textual), in the end-to-end learning framework to make systems more&#10;user-adaptive and effective. We incorporated user sentiment information in both&#10;supervised and reinforcement learning settings. In both settings, adding&#10;sentiment information reduced the dialog length and improved the task success&#10;rate on a bus information search task. This work is the first attempt to&#10;incorporate multimodal user information in the adaptive end-to-end dialog&#10;system training framework and attained state-of-the-art performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.02349" label="1709.02349">
        <attvalues>
          <attvalue for="0" value="A Deep Reinforcement Learning Chatbot" />
          <attvalue for="1" value="  We present MILABOT: a deep reinforcement learning chatbot developed by the&#10;Montreal Institute for Learning Algorithms (MILA) for the Amazon Alexa Prize&#10;competition. MILABOT is capable of conversing with humans on popular small talk&#10;topics through both speech and text. The system consists of an ensemble of&#10;natural language generation and retrieval models, including template-based&#10;models, bag-of-words models, sequence-to-sequence neural network and latent&#10;variable neural network models. By applying reinforcement learning to&#10;crowdsourced data and real-world user interactions, the system has been trained&#10;to select an appropriate response from the models in its ensemble. The system&#10;has been evaluated through A/B testing with real-world users, where it&#10;performed significantly better than many competing systems. Due to its machine&#10;learning architecture, the system is likely to improve with additional data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.00456" label="1907.00456">
        <attvalues>
          <attvalue for="0" value="Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human&#10;  Preferences in Dialog" />
          <attvalue for="1" value="  Most deep reinforcement learning (RL) systems are not able to learn&#10;effectively from off-policy data, especially if they cannot explore online in&#10;the environment. These are critical shortcomings for applying RL to real-world&#10;problems where collecting data is expensive, and models must be tested offline&#10;before being deployed to interact with the environment -- e.g. systems that&#10;learn from human interaction. Thus, we develop a novel class of off-policy&#10;batch RL algorithms, which are able to effectively learn offline, without&#10;exploring, from a fixed batch of human interaction data. We leverage models&#10;pre-trained on data as a strong prior, and use KL-control to penalize&#10;divergence from this prior during RL training. We also use dropout-based&#10;uncertainty estimates to lower bound the target Q-values as a more efficient&#10;alternative to Double Q-Learning. The algorithms are tested on the problem of&#10;open-domain dialog generation -- a challenging reinforcement learning problem&#10;with a 20,000-dimensional action space. Using our Way Off-Policy algorithm, we&#10;can extract multiple different reward functions post-hoc from collected human&#10;interaction data, and learn effectively from all of these. We test the&#10;real-world generalization of these systems by deploying them live to converse&#10;with humans in an open-domain setting, and demonstrate that our algorithm&#10;achieves significant improvements over prior methods in off-policy batch RL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.08858" label="1902.08858">
        <attvalues>
          <attvalue for="0" value="Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog&#10;  Agents with Latent Variable Models" />
          <attvalue for="1" value="  Defining action spaces for conversational agents and optimizing their&#10;decision-making process with reinforcement learning is an enduring challenge.&#10;Common practice has been to use handcrafted dialog acts, or the output&#10;vocabulary, e.g. in neural encoder decoders, as the action spaces. Both have&#10;their own limitations. This paper proposes a novel latent action framework that&#10;treats the action spaces of an end-to-end dialog agent as latent variables and&#10;develops unsupervised methods in order to induce its own action space from the&#10;data. Comprehensive experiments are conducted examining both continuous and&#10;discrete action types and two different optimization methods based on&#10;stochastic variational inference. Results show that the proposed latent actions&#10;achieve superior empirical performance improvement over previous word-level&#10;policy gradient methods on both DealOrNoDeal and MultiWoz dialogs. Our detailed&#10;analysis also provides insights about various latent variable approaches for&#10;policy learning and can serve as a foundation for developing better latent&#10;actions in future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1701.06547" label="1701.06547">
        <attvalues>
          <attvalue for="0" value="Adversarial Learning for Neural Dialogue Generation" />
          <attvalue for="1" value="  In this paper, drawing intuition from the Turing test, we propose using&#10;adversarial training for open-domain dialogue generation: the system is trained&#10;to produce sequences that are indistinguishable from human-generated dialogue&#10;utterances. We cast the task as a reinforcement learning (RL) problem where we&#10;jointly train two systems, a generative model to produce response sequences,&#10;and a discriminator---analagous to the human evaluator in the Turing test--- to&#10;distinguish between the human-generated dialogues and the machine-generated&#10;ones. The outputs from the discriminator are then used as rewards for the&#10;generative model, pushing the system to generate dialogues that mostly resemble&#10;human dialogues.&#10;  In addition to adversarial training we describe a model for adversarial {\em&#10;evaluation} that uses success in fooling an adversary as a dialogue evaluation&#10;metric, while avoiding a number of potential pitfalls. Experimental results on&#10;several metrics, including adversarial evaluation, demonstrate that the&#10;adversarially-trained system generates higher-quality responses than previous&#10;baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.08989" label="1812.08989">
        <attvalues>
          <attvalue for="0" value="The Design and Implementation of XiaoIce, an Empathetic Social Chatbot" />
          <attvalue for="1" value="  This paper describes the development of Microsoft XiaoIce, the most popular&#10;social chatbot in the world. XiaoIce is uniquely designed as an AI companion&#10;with an emotional connection to satisfy the human need for communication,&#10;affection, and social belonging. We take into account both intelligent quotient&#10;(IQ) and emotional quotient (EQ) in system design, cast human-machine social&#10;chat as decision-making over Markov Decision Processes (MDPs), and optimize&#10;XiaoIce for long-term user engagement, measured in expected Conversation-turns&#10;Per Session (CPS). We detail the system architecture and key components&#10;including dialogue manager, core chat, skills, and an empathetic computing&#10;module. We show how XiaoIce dynamically recognizes human feelings and states,&#10;understands user intent, and responds to user needs throughout long&#10;conversations. Since her launch in 2014, XiaoIce has communicated with over 660&#10;million active users and succeeded in establishing long-term relationships with&#10;many of them. Analysis of large scale online logs shows that XiaoIce has&#10;achieved an average CPS of 23, which is significantly higher than that of other&#10;chatbots and even human conversations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.05415" label="1901.05415">
        <attvalues>
          <attvalue for="0" value="Learning from Dialogue after Deployment: Feed Yourself, Chatbot!" />
          <attvalue for="1" value="  The majority of conversations a dialogue agent sees over its lifetime occur&#10;after it has already been trained and deployed, leaving a vast store of&#10;potential training signal untapped. In this work, we propose the self-feeding&#10;chatbot, a dialogue agent with the ability to extract new training examples&#10;from the conversations it participates in. As our agent engages in&#10;conversation, it also estimates user satisfaction in its responses. When the&#10;conversation appears to be going well, the user's responses become new training&#10;examples to imitate. When the agent believes it has made a mistake, it asks for&#10;feedback; learning to predict the feedback that will be given improves the&#10;chatbot's dialogue abilities further. On the PersonaChat chit-chat dataset with&#10;over 131k training examples, we find that learning from dialogue with a&#10;self-feeding chatbot significantly improves performance, regardless of the&#10;amount of traditional supervision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.07547" label="1909.07547">
        <attvalues>
          <attvalue for="0" value="Hierarchical Reinforcement Learning for Open-Domain Dialog" />
          <attvalue for="1" value="  Open-domain dialog generation is a challenging problem; maximum likelihood&#10;training can lead to repetitive outputs, models have difficulty tracking&#10;long-term conversational goals, and training on standard movie or online&#10;datasets may lead to the generation of inappropriate, biased, or offensive&#10;text. Reinforcement Learning (RL) is a powerful framework that could&#10;potentially address these issues, for example by allowing a dialog model to&#10;optimize for reducing toxicity and repetitiveness. However, previous approaches&#10;which apply RL to open-domain dialog generation do so at the word level, making&#10;it difficult for the model to learn proper credit assignment for long-term&#10;conversational rewards. In this paper, we propose a novel approach to&#10;hierarchical reinforcement learning, VHRL, which uses policy gradients to tune&#10;the utterance-level embedding of a variational sequence model. This&#10;hierarchical approach provides greater flexibility for learning long-term,&#10;conversational rewards. We use self-play and RL to optimize for a set of&#10;human-centered conversation metrics, and show that our approach provides&#10;significant improvements -- in terms of both human evaluation and automatic&#10;metrics -- over state-of-the-art dialog models, including Transformers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Open-domain dialog systems currently lack reliable automatic evaluation metrics \cite{liu2016not,lowe2017towards}. Recently, authors have begun to propose new metrics of conversation quality (e.g. \cite{see2019makes,hancock2019learning,zhou2018design} \shortcite{see2019makes,hancock2019learning,zhou2018design}), and have even proposed evaluating metrics on conversations generated with self-play \cite{ghandeharioun2019approximating}. However, these studies have not attempted to directly optimize for their proposed conversation metrics with RL.&#10;&#10;There has been significant progress in improving dialog generation outside of RL. One popular approach for controllable generation is conditional training, where a learned embedding vector is associated with a desired output feature and fed into the decoder to control generation (e.g. \cite{see2019makes,colombo2019affect,huang2018automatic} \shortcite{see2019makes,colombo2019affect,huang2018automatic}). This approach has multiple limitations. First, the model can only learn associations present in the training data, and cannot explore to discover improved methods for optimizing the desired features. Second, the conditional embeddings are learned at training time with teacher forcing and thus suffer from exposure bias \cite{ranzato2015sequence}. Using RL avoids these limitations as it allows exploring regions of space not present in the training data and directly optimizing for rewards at inference time. In addition, RL learns the total expected future reward of taking some action now, given how the rest of the conversation will unfold in the future. This allows RL models to make long-term trade-offs about the benefits of generating words and utterances in the conversation context. Finally, our approach does not require the addition of any new parameters or complex components. Instead, it can be use to tune pre-existing models to output better, more appropriate responses.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2206.11349" label="2206.11349">
        <attvalues>
          <attvalue for="0" value="Prompt Injection: Parameterization of Fixed Inputs" />
          <attvalue for="1" value="  Recent works have shown that attaching prompts to the input is effective at&#10;conditioning Language Models (LM) to perform specific tasks. However, prompts&#10;are always included in the input text during inference, thus incurring&#10;substantial computational and memory overhead. Also, there is currently no&#10;straightforward method of utilizing prompts that are longer than the maximum&#10;input length of the LMs without incurring additional costs during inference. We&#10;propose Prompt Injection (PI), a novel formulation of injecting the prompt into&#10;the parameters of an LM to be an efficient alternative to attaching fixed&#10;prompts to the input. We show that in scenarios with long fixed prompts, PI can&#10;be up to 280 times more efficient in terms of total FLOPs than previous&#10;approaches. We further explore methodologies for PI and show promising results&#10;in persona-dependent conversation, semantic parsing, and zero-shot learning&#10;with task instructions. Through these explorations, we show that PI can be a&#10;promising direction for conditioning language models, especially in scenarios&#10;with long and fixed prompts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Prompting&#10;Prompting is an emerging paradigm for modeling LMs, especially for few-shot and zero-shot learning~\cite{Radford2019LanguageMA,Brown2020LanguageMA,Raffel2020ExploringTL,Schick2021ItsNJ,Wei2021FinetunedLM,Sanh2021MultitaskPT}.&#10;With the help of appropriate prompts, one can exploit knowledge learned by a pre-trained LM and manipulate the LM’s behavior.&#10;The benefit of prompting is that the pre-trained LM can adapt to new scenarios with few or no labeled training data.&#10;However, for the in-context learning scenario, processing prompts that involve many training examples for each inference incurs substantial computational and memory overhead~\cite{liu2020tfew}.&#10;Given training data, \cite{liu2020tfew} replace in-context learning with fine-tuning a small set of parameters for tackling the above issue.&#10;Prompt Injection also tackles the same issue but assumes a stricter scenario where there are no training data for the given prompt.&#10;&#10;Efficient Transformers for Long Inputs&#10;One can consider using efficient Transformer-based~\cite{Vaswani2017AttentionIA} architectures for handling long input sequences~\cite{Tay2022EfficientTA}.&#10;The main challenge of using a vanilla Transformer architecture is the quadratic cost in time and memory with regard to the input sequence length due to the self-attention operation. &#10;There has been a surge of recent works addressing this problem~\cite{Dai2019TransformerXLAL,Zaheer2020BigBT,Beltagy2020LongformerTL,Katharopoulos2020TransformersAR,Zhu2021LongShortTE,Guo2021LongT5ET}.&#10;They are primarily dedicated to improving either the efficiency of the self-attention mechanism or the general efficiency of the Transformer architecture through sparse models.&#10;Our Prompt Injection approach tackles the efficiency problem of performing prompt-dependent tasks by keeping the input sequences short (without prompts), bounding the time and memory complexity to a constant invariant of the length of the prompt.&#10;&#10;Persona-dependent Conversation&#10;Endowing a chabot with a persona~\cite{Zhang2018PersonalizingDA, Mazar2018TrainingMO, Welleck2019DialogueNL} is challenging, but it enables the chatbot to deliver more personal, specific, consistent, and engaging conversations~\cite{Zhang2018PersonalizingDA} and gain user trust~\cite{Liu2020YouIM, Song2019ExploitingPI, Qian2018AssigningPT}.&#10;To achieve this, previous works have attached a persona to the dialog history at every inference time, so that the model can always be conditioned on the persona.&#10;However, given a long persona description, this approach brings the critical problem of increased overall memory and computations, resulting in delayed response generation.&#10;An LM augmented with a retrieval mechanism~\cite{han2022meet} may be used but still comes with non-trivial computational overhead.&#10;Prompt Injection allows a dialogue agent to generate responses without a persona description as the explicit input once the persona is injected.&#10;&#10;Semantic Parsing&#10;Semantic parsing is the task of mapping a natural language query into a SQL query executable on a database.&#10;Recently, the community has focused more on cross-domain (cross-database) semantic parsing, where models are trained and tested on different domains (databases)~\cite{Yu2018SpiderAL}.&#10;The domain-adaptation setup introduces many generalization challenges, such as non-explicit column names and domain-specific phrases~\cite{Hazoom2021TexttoSQLIT}, and recent works concatenate the natural language query with the serialized database schema as the input to address the problem~\cite{Suhr2020ExploringUG,Deng2021StructureGroundedPF,Xie2022UnifiedSKGUA}.&#10;With Prompt Injection, the model is adapted to a new database schema in advance, so that it can map natural language queries to SQL queries on the new database without explicitly referring to the schema during inference.&#10;&#10;Zero-shot Learning with Task Instructions&#10;Recent works~\cite{Sanh2021MultitaskPT, Wei2021FinetunedLM} have addressed zero-shot generalization to new tasks~\cite{Brown2020LanguageMA,Kim2021WhatCC} by multi-task prompted training.&#10;With multi-task prompted training, the models learn to use task instructions as prompts to generalize to unseen tasks.&#10;It is demonstrated that this approach improves generalization ability to novel tasks and offers an effective substitute for unsupervised language model pre-training.&#10;Through Prompt Injection, the LM can be aware of a novel task instruction before performing the task and thus does not require the instruction, which can be lengthy, to make predictions.&#10; " />
          <attvalue for="4" value="Language Model Conditioning, Computer Science, Efficient Inference Methods, Linguistics, Cognitive Science, Mathematics, Prompt Optimization Techniques" />
        </attvalues>
      </node>
      <node id="1909.09962" label="1909.09962">
        <attvalues>
          <attvalue for="0" value="Adapting Language Models for Non-Parallel Author-Stylized Rewriting" />
          <attvalue for="1" value="  Given the recent progress in language modeling using Transformer-based neural&#10;models and an active interest in generating stylized text, we present an&#10;approach to leverage the generalization capabilities of a language model to&#10;rewrite an input text in a target author's style. Our proposed approach adapts&#10;a pre-trained language model to generate author-stylized text by fine-tuning on&#10;the author-specific corpus using a denoising autoencoder (DAE) loss in a&#10;cascaded encoder-decoder framework. Optimizing over DAE loss allows our model&#10;to learn the nuances of an author's style without relying on parallel data,&#10;which has been a severe limitation of the previous related works in this space.&#10;To evaluate the efficacy of our approach, we propose a linguistically-motivated&#10;framework to quantify stylistic alignment of the generated text to the target&#10;author at lexical, syntactic and surface levels. The evaluation framework is&#10;both interpretable as it leads to several insights about the model, and&#10;self-contained as it does not rely on external classifiers, e.g. sentiment or&#10;formality classifiers. Qualitative and quantitative assessment indicates that&#10;the proposed approach rewrites the input text with better alignment to the&#10;target style while preserving the original content better than state-of-the-art&#10;baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.04304" label="1705.04304">
        <attvalues>
          <attvalue for="0" value="A Deep Reinforced Model for Abstractive Summarization" />
          <attvalue for="1" value="  Attentional, RNN-based encoder-decoder models for abstractive summarization&#10;have achieved good performance on short input and output sequences. For longer&#10;documents and summaries however these models often include repetitive and&#10;incoherent phrases. We introduce a neural network model with a novel&#10;intra-attention that attends over the input and continuously generated output&#10;separately, and a new training method that combines standard supervised word&#10;prediction and reinforcement learning (RL). Models trained only with supervised&#10;learning often exhibit &quot;exposure bias&quot; - they assume ground truth is provided&#10;at each step during training. However, when standard word prediction is&#10;combined with the global sequence prediction training of RL the resulting&#10;summaries become more readable. We evaluate this model on the CNN/Daily Mail&#10;and New York Times datasets. Our model obtains a 41.16 ROUGE-1 score on the&#10;CNN/Daily Mail dataset, an improvement over previous state-of-the-art models.&#10;Human evaluation also shows that our model produces higher quality summaries.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12510" label="2010.12510">
        <attvalues>
          <attvalue for="0" value="Improving Robustness by Augmenting Training Sentences with&#10;  Predicate-Argument Structures" />
          <attvalue for="1" value="  Existing NLP datasets contain various biases, and models tend to quickly&#10;learn those biases, which in turn limits their robustness. Existing approaches&#10;to improve robustness against dataset biases mostly focus on changing the&#10;training objective so that models learn less from biased examples. Besides,&#10;they mostly focus on addressing a specific bias, and while they improve the&#10;performance on adversarial evaluation sets of the targeted bias, they may bias&#10;the model in other ways, and therefore, hurt the overall robustness. In this&#10;paper, we propose to augment the input sentences in the training data with&#10;their corresponding predicate-argument structures, which provide a higher-level&#10;abstraction over different realizations of the same meaning and help the model&#10;to recognize important parts of sentences. We show that without targeting a&#10;specific bias, our sentence augmentation improves the robustness of transformer&#10;models against multiple biases. In addition, we show that models can still be&#10;vulnerable to the lexical overlap bias, even when the training data does not&#10;contain this bias, and that the sentence augmentation also improves the&#10;robustness in this scenario. We will release our adversarial datasets to&#10;evaluate bias in such a scenario as well as our augmentation scripts at&#10;https://github.com/UKPLab/data-augmentation-for-robustness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Debiasing Methods.&#10;Existing debiasing solutions fall into one of these two categories: (1) extending the training data with additional counterexamples, and (2) proposing a new approach that recognizes biased examples in the training data and then using this knowledge during training \cite{utama2020mind,he-etal-2019-unlearn,clark-etal-2019-dont,mahabadi2019simple,utama-etal-2020-unknown,wu2020improving}.&#10;&#10;The first type of solution is to identify the bias and augment the training data with counterexamples in which relying on the targeted bias results in incorrect predictions.&#10;While augmenting the training data with counterexamples improves the results on the targeted bias, it may hurt the overall robustness \cite{nie2019analyzing}.&#10;As mentioned by \cite{jha2020does}, while augmentation with counterexamples helps the model to unlearn the targeted bias, it is unlikely that it encourages the model to rely on more generalizable features of the data.&#10;&#10;The approaches of the second category first use a bias detection model for recognizing training examples that contain the bias. They then either (a) train an ensemble of the bias model and a base model so that the base model only learns from non-biased examples \cite{he-etal-2019-unlearn,clark-etal-2019-dont,mahabadi2019simple}, (b) change the importance of the biased training examples in the training objective \cite{schuster-etal-2019-towards,mahabadi2019simple}, or (c) change the confidence of the model on biased examples \cite{utama2020mind}.&#10;&#10;The shortcoming of existing debiasing methods is that they mostly model a single bias and only evaluate the impact of the proposed method on the adversarial evaluation set of the targeted bias.&#10;Therefore, while they improve the performance on the targeted adversarial sets, they may hurt the overall robustness.&#10;The recent work of \cite{utama-etal-2020-unknown} and \cite{wu2020improving} are the exceptions in which they show that their proposed debiasing frameworks improve the overall robustness, and hence the generalization across different datasets in natural language understanding and question answering, respectively.&#10;\cite{utama-etal-2020-unknown} propose a new framework that automatically recognizes biased training examples and does not require predefining bias types. Therefore, the recognized biased examples may contain various bias types. &#10;\cite{wu2020improving} propose a framework for modeling multiple known biases concurrently. To do so, they propose to combine two bias weights in the training objective including (a) a dataset-level weight indicating the strength of the bias in the datasets, and (b) an example-level weight indicating the strength of the bias in a training example.&#10;The common finding in both works is that debiasing based on multiple biases is a key factor in improving the overall robustness.&#10;&#10;Compared to existing debiasing methods:&#10;\begin{itemize}&#10;&#09;\item Our proposed approach does not include an additional model to recognize biased examples or additional training examples.&#10;As we show in Section~\ref{robustness_results}, since it does not target any specific bias, it improves the robustness of the baseline model against multiple biases.&#10;\item Since it does not require recognizing biased examples, it is applicable to improve robustness against biases that do not exist in the training examples. &#10;\end{itemize}&#10;&#10;Using Linguistic Structures for Neural Models.&#10;The use of linguistic information in recent neural models is not very common. &#10;The use of such information has been mainly investigated for tasks in which there is a clear relation between the linguistic features and the target task.&#10;For instance, various neural models use syntactic information for the task of semantic role labeling (SRL) \cite{roth-lapata-2016-neural,marcheggiani-titov-2017-encoding,strubell-etal-2018-linguistically,swayamdipta-etal-2018-syntactic}, which is closely related to syntactic relations, i.e., some arcs in the syntactic dependency tree can be mirrored in semantic dependency relations.&#10;&#10;\newcite{marcheggiani-titov-2017-encoding} build a graph representation from the input text using their corresponding dependency relations and use graph convolutional networks (GCNs) to process the resulting graph for SRL.&#10;They show that the incorporation of syntactic relations improves the in-domain but decreases the out-of-domain performance.&#10;&#10;Similarly, \newcite{cao-etal-2019-qa} and \newcite{dhingra-etal-2018-neural} incorporate linguistic information, i.e., coreference relations, in their model and show improvements in in-domain evaluations.&#10;&#10;\newcite{strubell-etal-2018-linguistically} use linguistic information, i.e., dependency parse, part-of-speech tags, and predicates for SRL using a transformer-based encoder \cite{NIPS2017_7181}.&#10;They make use of this linguistic information by (1) using multi-task learning, and (2) supervising the neural attention of the transformer model to predict syntactic dependencies.&#10;They use gold syntax information during training and predicted information during the test time.&#10;Their model substantially improves both in-domain and out-of-domain performance in SRL.&#10;However, these results are then outperformed by a simple BERT model without using any additional linguistic information \cite{shi2019simple}.&#10;&#10;\newcite{moosavi-strube-2018-using} examine the use of various linguistic features, e.g., syntactic dependency relations and gender and number information, as additional input features to a neural coreference resolver. They show that using informative linguistic features substantially improves the generalization of the examined model. &#10;&#10;In a similar direction, \newcite{moosavi2019improving} improve the robustness by enhancing the input representations. They did so by adding a set of simple features to the input where the input is a pair of text sequences and show that it improves generalization across similar datasets and tasks. &#10;&#10;All the above approaches require additional linguistic information, e.g., syntax, both during the training and the test time.&#10;\newcite{swayamdipta-etal-2018-syntactic}, on the other hand, only make use of the additional syntactic information during training. They use multi-task learning by considering syntax parsing as an auxiliary task and minimizing the combination of the losses of the main and auxiliary tasks. &#10;They use syntactic information for the tasks of SRL and coreference resolution.&#10;They show that this information slightly improves the in-domain performance.&#10;In this work, we do not change the loss function and only augment the input sentences of the training data.&#10;The advantage of our solution is that it does not require any changes in the model or its training objective. It can be applied to all the transformer-based models without changing the training procedure. &#10;&#10;Using Predicate-Argument Structures.&#10;Predicate-argument structures have been used for improving the performance of downstream tasks like machine translation \cite{liu-gildea-2010-semantic,bazrafshan-gildea-2013-semantic}, reading comprehension \cite{berant-etal-2014-modeling,wang-etal-2015-machine}, and dialogue systems \cite{turdialogue,chen-et-al:2013:PAPERS}.&#10;However, these approaches are based on pre-neural models.&#10;&#10;The proposed model by \newcite{marcheggiani-etal-2018-exploiting} for neural machine translation is a sample neural model that incorporates predicate-argument structures.&#10;Unlike this work, \newcite{marcheggiani-etal-2018-exploiting} incorporate these linguistic structures at the model-level. They add two layers of semantic GCNs on top of a standard encoder, e.g., convolutional neural network or bidirectional LSTM. The semantic structures are used for determining nodes and edges in the GCNs.&#10;In this work, however, we incorporate these structures at the input level, and only for the training data. Therefore, we can use the state-of-the-art models without any changes.&#10;&#10;Overall, this work differs from the related work because (1) it evaluates the use of predicate-argument structures for improving the robustness of transformer-based models on natural language understanding tasks, and (2) it uses these structures at the input level to extend raw inputs, (3) it only employs this information during training, and (4) it requires no changes in the model or the training procedure.&#10;&#10;" />
          <attvalue for="4" value="NLP Dataset Biases, Computer Science, Linguistics, Model Robustness Improvement, Sentence Augmentation Techniques, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2307.03692" label="2307.03692">
        <attvalues>
          <attvalue for="0" value="Becoming self-instruct: introducing early stopping criteria for minimal&#10;  instruct tuning" />
          <attvalue for="1" value="  In this paper, we introduce the Instruction Following Score (IFS), a metric&#10;that detects language models' ability to follow instructions. The metric has a&#10;dual purpose. First, IFS can be used to distinguish between base and instruct&#10;models. We benchmark publicly available base and instruct models, and show that&#10;the ratio of well formatted responses to partial and full sentences can be an&#10;effective measure between those two model classes. Secondly, the metric can be&#10;used as an early stopping criteria for instruct tuning. We compute IFS for&#10;Supervised Fine-Tuning (SFT) of 7B and 13B LLaMA models, showing that models&#10;learn to follow instructions relatively early in the training process, and the&#10;further finetuning can result in changes in the underlying base model&#10;semantics. As an example of semantics change we show the objectivity of model&#10;predictions, as defined by an auxiliary metric ObjecQA. We show that in this&#10;particular case, semantic changes are the steepest when the IFS tends to&#10;plateau. We hope that decomposing instruct tuning into IFS and semantic factors&#10;starts a new trend in better controllable instruct tuning and opens&#10;possibilities for designing minimal instruct interfaces querying foundation&#10;models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models (LLMs) finetuned on instruct data can behave like conversational agents (Alpaca: \cite{alpaca}, Self-Instruct: \cite{wang2023selfinstruct}). The recipe for a chat model is well-defined: one needs to perform instruction tuning, which means supervised finetuning (SFT) of an LLM on tuples of instruction and response (\cite{longpre2023flan}).&#10;&#10;Open-source datasets vary in quality and quantity, ranging from 1k examples (\cite{zhou2023lima}) to over 800k examples (\cite{gpt4all}). In addition, there are more than a dozen open-source base LLMs, such as LLaMA (\cite{touvron2023llama}), OPT (\cite{zhang2022opt}), GPT-Neo (\cite{gao2020pile}), Palmyra (\cite{Palmyra}), and others, which result in a plethora of possible combinations leading to distinct instruct models.&#10;&#10;We can see instruct tuning attempts through the lens of the &quot;imitation models&quot; - concept introduced by \cite{gudibande2023false}, i.e., efforts to distil closed (and possibly much bigger) proprietary models like ChatGPT (\cite{ChatGPT}), Bard (\cite{Bard}), and Claude (\cite{Claude}).&#10;&#10;Little is known about the qualitative impact of the distillation process on the base model (\cite{hinton2015distilling}). Imitation success is measured in terms of knowledge (e.g., HELM \cite{liang2022holistic}), skills (e.g., Natural Questions \cite{47761}) or manual checks based on human preferences (\cite{zhou2023lima}). There is no consensus whether a manual check that might skew the metric towards style and formatting of responses is a good overall metric (\cite{gudibande2023false}). A fairly recent attempt to more robustly evaluate instruct models is the Huggingface Leaderboard (\cite{OpenLLMLeaderboard}), which evaluates models against four key benchmarks from the Eleuther AI Language Model Evaluation Harness (\cite{eval-harness}).&#10;&#10;Ablation studies have shown that both the diversity and quality of the training data play a crucial role in model performance (\cite{chen2023maybe}, \cite{zhou2023lima}). Low Training Data Instruction Tuning (LTD Tuning) suggests that task-specific models can gain 2\% performance when trained on less than 0.5\% of the original data. Moreover, prolonged instruction tuning can decrease the foundational model knowledge (\cite{gudibande2023false}) and can be seen as the out-of-distribution task for a downstream task of instruct-tuning (\cite{kumar2022finetuning}). &#10;&#10;In this study, we want to lay the foundation for instruct models research by defining the necessary (but not sufficient) condition for an instruct model. Let's conduct a thought experiment.&#10;&#10;Let's put all models behind a closed API (a recent equivalent of a black box). Is the model instruct-tuned or not? Knowledge benchmarks could be similar for vanilla and instruct models for LTD tuning. Skills tests would highly depend on the model size, which is not known. The simplest way of solving the riddle would be to \ldots chat with the model and judge the tone of the response. For a vanilla model, we expect a next prediction word attempt, whereas for instruct models, we expect them to follow instructions. We introduce a metric that captures this tone difference - Instruct Following Score (IFS). We call this problem a &quot;tone alignment&quot; issue.&#10;&#10;The IFS is defined as a ratio of &quot;answer-like&quot; responses to &quot;continuation-like&quot; responses on a predefined set of instructions, where class of a response is determined by a binary classifier. &#10;&#10;We benchmark publicly available base and instruct models, and show that the ratio of well formatted responses to partial and full sentences can be an effective measure between vanilla and instruct following models. Moreover, we calculate IFS for SFT for 7B and 13B LLaMA models, in the hope of finding a stopping criterion for a minimal instruct tuning.&#10;&#10;To draw a comparison between the learning curve for response tone and the acquisition of semantic and domain-specific knowledge, we propose a supplementary metric called ObjecQA. This auxiliary metric quantifies the objectivity of a model's predictions, as this signal can be identified within the dataset. While this feature choice is arbitrary, we aim to discover possibly more general heuristics for better control over the training phases, including identification of &quot;format-infusion&quot; and &quot;knowledge-infusion&quot; stages. &#10;&#10;The paper is organised as follows. In Section 2, we discuss the necessary conditions for a model to be considered an instruct model and data preparation for IFS. The response tone classifier training is described in Section 4. In Section 5, we present results for instruct models and compare them to baseline vanilla models in terms of instruct tone and semantic shifts. The study ends with conclusions and future directions proposed in Section 6.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Instruction Following Metrics, Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Model Training Optimization, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.15717" label="2305.15717">
        <attvalues>
          <attvalue for="0" value="The False Promise of Imitating Proprietary LLMs" />
          <attvalue for="1" value="  An emerging method to cheaply improve a weaker language model is to finetune&#10;it on outputs from a stronger model, such as a proprietary system like ChatGPT&#10;(e.g., Alpaca, Self-Instruct, and others). This approach looks to cheaply&#10;imitate the proprietary model's capabilities using a weaker open-source model.&#10;In this work, we critically analyze this approach. We first finetune a series&#10;of LMs that imitate ChatGPT using varying base model sizes (1.5B--13B), data&#10;sources, and imitation data amounts (0.3M--150M tokens). We then evaluate the&#10;models using crowd raters and canonical NLP benchmarks. Initially, we were&#10;surprised by the output quality of our imitation models -- they appear far&#10;better at following instructions, and crowd workers rate their outputs as&#10;competitive with ChatGPT. However, when conducting more targeted automatic&#10;evaluations, we find that imitation models close little to none of the gap from&#10;the base LM to ChatGPT on tasks that are not heavily supported in the imitation&#10;data. We show that these performance discrepancies may slip past human raters&#10;because imitation models are adept at mimicking ChatGPT's style but not its&#10;factuality. Overall, we conclude that model imitation is a false promise: there&#10;exists a substantial capabilities gap between open and closed LMs that, with&#10;current methods, can only be bridged using an unwieldy amount of imitation data&#10;or by using more capable base LMs. In turn, we argue that the highest leverage&#10;action for improving open-source models is to tackle the difficult challenge of&#10;developing better base LMs, rather than taking the shortcut of imitating&#10;proprietary systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.10054" label="2202.10054">
        <attvalues>
          <attvalue for="0" value="Fine-Tuning can Distort Pretrained Features and Underperform&#10;  Out-of-Distribution" />
          <attvalue for="1" value="  When transferring a pretrained model to a downstream task, two popular&#10;methods are full fine-tuning (updating all the model parameters) and linear&#10;probing (updating only the last linear layer -- the &quot;head&quot;). It is well known&#10;that fine-tuning leads to better accuracy in-distribution (ID). However, in&#10;this paper, we find that fine-tuning can achieve worse accuracy than linear&#10;probing out-of-distribution (OOD) when the pretrained features are good and the&#10;distribution shift is large. On 10 distribution shift datasets&#10;(Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR $\to$ STL, CIFAR10.1, FMoW,&#10;ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch), fine-tuning obtains on&#10;average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We&#10;show theoretically that this tradeoff between ID and OOD accuracy arises even&#10;in a simple setting: fine-tuning overparameterized two-layer linear networks.&#10;We prove that the OOD error of fine-tuning is high when we initialize with a&#10;fixed or random head -- this is because while fine-tuning learns the head, the&#10;lower layers of the neural network change simultaneously and distort the&#10;pretrained features. Our analysis suggests that the easy two-step strategy of&#10;linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning&#10;heuristic, combines the benefits of both fine-tuning and linear probing.&#10;Empirically, LP-FT outperforms both fine-tuning and linear probing on the above&#10;datasets (1% better ID, 10% better OOD than full fine-tuning).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.00748" label="2208.00748">
        <attvalues>
          <attvalue for="0" value="Efficient Long-Text Understanding with Short-Text Models" />
          <attvalue for="1" value="  Transformer-based pretrained language models (LMs) are ubiquitous across&#10;natural language understanding, but cannot be applied to long sequences such as&#10;stories, scientific articles and long documents, due to their quadratic&#10;complexity. While a myriad of efficient transformer variants have been&#10;proposed, they are typically based on custom implementations that require&#10;expensive pretraining from scratch. In this work, we propose SLED:&#10;SLiding-Encoder and Decoder, a simple approach for processing long sequences&#10;that re-uses and leverages battle-tested short-text pretrained LMs.&#10;Specifically, we partition the input into overlapping chunks, encode each with&#10;a short-text LM encoder and use the pretrained decoder to fuse information&#10;across chunks (fusion-in-decoder). We illustrate through controlled experiments&#10;that SLED offers a viable strategy for long text understanding and evaluate our&#10;approach on SCROLLS, a benchmark with seven datasets across a wide range of&#10;language understanding tasks. We find that SLED is competitive with specialized&#10;models that are up to 50x larger and require a dedicated and expensive&#10;pretraining step.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Recent advances in natural language processing have been by and large fueled by the transformer architecture \cite{Vaswani2017AttentionIA}. A core component of the transformer is the self-attention layer where every input token ``attends'' to every other token to produce its contextualized representation. This results in quadratic time and space dependency w.r.t. the length of the input, limiting the ability of transformers to process long sequences.&#10;&#10;This long-text limitation has sparked ample interest in developing efficient transformer variants.&#10;One prominent family of methods is based on sparse attention, where each token attends to a constant number of other tokens, overcoming the quadratic dependency. Tokens typically attend either to their local surrounding \cite{Zaheer2020BigBT,beltagy2020longformer,ainslie-etal-2020-etc,gupta2020gmat} or to tokens that are semantically similar \cite{Kitaev2020Reformer,roy-etal-2021-efficient}.&#10;Moreover, a constant number of global tokens that attend to and are attended by all input tokens are often added to each attention sub-layer.&#10;Recent analyses \cite{xiong-etal-2022-simple} have shown that sparse transformers with local attention are competitive with other variants on multiple language understanding tasks.&#10;&#10;Our method, SLED, falls into the family of local attention variants. However, unlike prior work, SLED re-uses and extends existing short-range encoder-decoder models, and does not require specialized pretraining or dedicated CUDA implementations. &#10;&#10;In most local attention variants, e.g., LED \cite{beltagy2020longformer}, attention is local per-layer, but the receptive field of tokens grows across layers. In SLED, which we describe next, tokens have access to the same number of tokens, independent of a layer's depth, which enables better parallelization.&#10;For a survey on the families of efficient transformers, see \newcite{tay2020efficient}. For an in-depth comparison of SLED and LED, we refer to Appendix~\ref{app:led}.&#10; " />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Long Sequence Processing, Natural Language Understanding, Artificial Intelligence, Language Model Efficiency" />
        </attvalues>
      </node>
      <node id="1406.1078" label="1406.1078">
        <attvalues>
          <attvalue for="0" value="Learning Phrase Representations using RNN Encoder-Decoder for&#10;  Statistical Machine Translation" />
          <attvalue for="1" value="  In this paper, we propose a novel neural network model called RNN&#10;Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN&#10;encodes a sequence of symbols into a fixed-length vector representation, and&#10;the other decodes the representation into another sequence of symbols. The&#10;encoder and decoder of the proposed model are jointly trained to maximize the&#10;conditional probability of a target sequence given a source sequence. The&#10;performance of a statistical machine translation system is empirically found to&#10;improve by using the conditional probabilities of phrase pairs computed by the&#10;RNN Encoder-Decoder as an additional feature in the existing log-linear model.&#10;Qualitatively, we show that the proposed model learns a semantically and&#10;syntactically meaningful representation of linguistic phrases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.03872" label="1706.03872">
        <attvalues>
          <attvalue for="0" value="Six Challenges for Neural Machine Translation" />
          <attvalue for="1" value="  We explore six challenges for neural machine translation: domain mismatch,&#10;amount of training data, rare words, long sentences, word alignment, and beam&#10;search. We show both deficiencies and improvements over the quality of&#10;phrase-based statistical machine translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.07526" label="2002.07526">
        <attvalues>
          <attvalue for="0" value="A Survey of Deep Learning Techniques for Neural Machine Translation" />
          <attvalue for="1" value="  In recent years, natural language processing (NLP) has got great development&#10;with deep learning techniques. In the sub-field of machine translation, a new&#10;approach named Neural Machine Translation (NMT) has emerged and got massive&#10;attention from both academia and industry. However, with a significant number&#10;of researches proposed in the past several years, there is little work in&#10;investigating the development process of this new technology trend. This&#10;literature survey traces back the origin and principal development timeline of&#10;NMT, investigates the important branches, categorizes different research&#10;orientations, and discusses some future research trends in this field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.07337" label="1906.07337">
        <attvalues>
          <attvalue for="0" value="Measuring Bias in Contextualized Word Representations" />
          <attvalue for="1" value="  Contextual word embeddings such as BERT have achieved state of the art&#10;performance in numerous NLP tasks. Since they are optimized to capture the&#10;statistical properties of training data, they tend to pick up on and amplify&#10;social stereotypes present in the data as well. In this study, we (1)~propose a&#10;template-based method to quantify bias in BERT; (2)~show that this method&#10;obtains more consistent results in capturing social biases than the traditional&#10;cosine based method; and (3)~conduct a case study, evaluating gender bias in a&#10;downstream task of Gender Pronoun Resolution. Although our case study focuses&#10;on gender bias, the proposed technique is generalizable to unveiling other&#10;biases, including in multiclass settings, such as racial and religious biases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.10650" label="1905.10650">
        <attvalues>
          <attvalue for="0" value="Are Sixteen Heads Really Better than One?" />
          <attvalue for="1" value="  Attention is a powerful and ubiquitous mechanism for allowing neural models&#10;to focus on particular salient pieces of information by taking their weighted&#10;average when making predictions. In particular, multi-headed attention is a&#10;driving force behind many recent state-of-the-art NLP models such as&#10;Transformer-based MT models and BERT. These models apply multiple attention&#10;mechanisms in parallel, with each attention &quot;head&quot; potentially focusing on&#10;different parts of the input, which makes it possible to express sophisticated&#10;functions beyond the simple weighted average. In this paper we make the&#10;surprising observation that even if models have been trained using multiple&#10;heads, in practice, a large percentage of attention heads can be removed at&#10;test time without significantly impacting performance. In fact, some layers can&#10;even be reduced to a single head. We further examine greedy algorithms for&#10;pruning down models, and the potential speed, memory efficiency, and accuracy&#10;improvements obtainable therefrom. Finally, we analyze the results with respect&#10;to which parts of the model are more reliant on having multiple heads, and&#10;provide precursory evidence that training dynamics play a role in the gains&#10;provided by multi-head attention.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00115" label="2005.00115">
        <attvalues>
          <attvalue for="0" value="Learning to Faithfully Rationalize by Construction" />
          <attvalue for="1" value="  In many settings it is important for one to be able to understand why a model&#10;made a particular prediction. In NLP this often entails extracting snippets of&#10;an input text `responsible for' corresponding model output; when such a snippet&#10;comprises tokens that indeed informed the model's prediction, it is a faithful&#10;explanation. In some settings, faithfulness may be critical to ensure&#10;transparency. Lei et al. (2016) proposed a model to produce faithful rationales&#10;for neural text classification by defining independent snippet extraction and&#10;prediction modules. However, the discrete selection over input tokens performed&#10;by this method complicates training, leading to high variance and requiring&#10;careful hyperparameter tuning. We propose a simpler variant of this approach&#10;that provides faithful explanations by construction. In our scheme, named&#10;FRESH, arbitrary feature importance scores (e.g., gradients from a trained&#10;model) are used to induce binary labels over token inputs, which an extractor&#10;can be trained to predict. An independent classifier module is then trained&#10;exclusively on snippets provided by the extractor; these snippets thus&#10;constitute faithful explanations, even if the classifier is arbitrarily&#10;complex. In both automatic and manual evaluations we find that variants of this&#10;simple framework yield predictive performance superior to `end-to-end'&#10;approaches, while being more general and easier to train. Code is available at&#10;https://github.com/successar/FRESH&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.02352" label="2010.02352">
        <attvalues>
          <attvalue for="0" value="Inference Strategies for Machine Translation with Conditional Masking" />
          <attvalue for="1" value="  Conditional masked language model (CMLM) training has proven successful for&#10;non-autoregressive and semi-autoregressive sequence generation tasks, such as&#10;machine translation. Given a trained CMLM, however, it is not clear what the&#10;best inference strategy is. We formulate masked inference as a factorization of&#10;conditional probabilities of partial sequences, show that this does not harm&#10;performance, and investigate a number of simple heuristics motivated by this&#10;perspective. We identify a thresholding strategy that has advantages over the&#10;standard &quot;mask-predict&quot; algorithm, and provide analyses of its behavior on&#10;machine translation tasks.&#10;" />
          <attvalue for="2" value="&#10;The widely successful masked language modeling paradigm popularized by BERT~\cite{devlin-etal-2019-bert} has recently been adapted to conditional masked language model (CMLM) training for semi-autoregressive sequence generation&#10;\cite{ghazvininejadETAL:19}, where model predictions are conditioned on the complete input sequence and the observed (non-masked) portion of the output sequence.&#10;The CMLM's simplicity and its clear links to the very active field of linguistic representation learning are advantages over its semi-autoregressive competitors, such as iterative refinement of token sequences~\cite{leeETAL:18}, refinement of non-linguistic intermediate representations~\cite{kaiser2018fast,shu2020latent} and learning to predict parallel edit operations~\cite{stern2019insertion,gu2019levenshtein}.&#10;&#10;It is not obvious how to best perform inference with the CMLM. Starting from a partially-observed output sequence, the optimal choice to complete it within a single step would be to generate the most likely token at each unobserved (masked) position independently.&#10;However, it is less clear how to progress from an initial, completely masked sequence to a final hypothesis semi-autoregressively over a number of steps, with each successive step unmasking new context for the next. This requires not only ordering the tokens for generation, but also making decisions about how many tokens to simultaneously predict in each step. &#10;&#10;\newcite{ghazvininejadETAL:19} propose the {mask-predict} algorithm, which iteratively generates fresh model predictions for all masked positions, and then unmasks a predefined number of the most likely predictions. Given a fixed number of iterations, a decaying schedule determines how many predictions to unmask in each iteration. &#10;Each successive iteration provides mode-breaking~\cite{guETAL:18} context for the next.&#10;By fixing the number of iterations, this approach allows for constant-time semi-autoregressive decoding.&#10;&#10;The fixed-iteration strategy is very practical and has yielded empirical success in a range of machine translation experiments, but there is no guarantee that it is optimal.&#10;The tokens to be unmasked on a given iteration are all predicted independently, and therefore might contain repeated words, or words with low model confidence. These issues can be mitigated by later re-masking a token to repair it~\cite{ghazvininejadETAL:19} or by adapting the model to incorrect contexts~\cite{Ghazvininejad2020SemiAutoregressiveTI}.&#10;&#10;We instead adopt a fully probabilistic view of the masked prediction sequence, which we enable by simply disallowing the re-masking of previously unmasked tokens. This view guides us to a heuristic inference schedule that selects sets of unmasked tokens according to a threshold on the product of their conditionally independent model probabilities. This heuristic naturally slows down in the situations mentioned above, and speeds up in the presence of high confidence, which allows us to achieve favorable quality-to-speed trade-offs. We focus on strengthening the CMLM inference (Section~\ref{sec:cmlm_inference}) while leaving its training algorithm unchanged (Section~\ref{sec:cmlm_training}), and maintaining much of the structure of the original inference strategy. For our experiments on machine translation (Section~\ref{sec:experiments}), we compare inference heuristics in terms of their quality-speed trade-offs. We analyze the development of quality over iterations, and the influence of sentence length. With examples of unmasking schedules we furthermore illustrate the role of mode breaking through choosing the right contexts.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Inference Strategy Optimization, Linguistics, Language Model Training, Sequence Generation Tasks, Mathematics, Statistics" />
        </attvalues>
      </node>
      <node id="1904.09324" label="1904.09324">
        <attvalues>
          <attvalue for="0" value="Mask-Predict: Parallel Decoding of Conditional Masked Language Models" />
          <attvalue for="1" value="  Most machine translation systems generate text autoregressively from left to&#10;right. We, instead, use a masked language modeling objective to train a model&#10;to predict any subset of the target words, conditioned on both the input text&#10;and a partially masked target translation. This approach allows for efficient&#10;iterative decoding, where we first predict all of the target words&#10;non-autoregressively, and then repeatedly mask out and regenerate the subset of&#10;words that the model is least confident about. By applying this strategy for a&#10;constant number of iterations, our model improves state-of-the-art performance&#10;levels for non-autoregressive and parallel decoding translation models by over&#10;4 BLEU on average. It is also able to reach within about 1 BLEU point of a&#10;typical left-to-right transformer model, while decoding significantly faster.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.06901" label="1802.06901">
        <attvalues>
          <attvalue for="0" value="Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative&#10;  Refinement" />
          <attvalue for="1" value="  We propose a conditional non-autoregressive neural sequence model based on&#10;iterative refinement. The proposed model is designed based on the principles of&#10;latent variable models and denoising autoencoders, and is generally applicable&#10;to any sequence generation task. We extensively evaluate the proposed model on&#10;machine translation (En-De and En-Ro) and image caption generation, and observe&#10;that it significantly speeds up decoding while maintaining the generation&#10;quality comparable to the autoregressive counterpart.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.03382" label="1803.03382">
        <attvalues>
          <attvalue for="0" value="Fast Decoding in Sequence Models using Discrete Latent Variables" />
          <attvalue for="1" value="  Autoregressive sequence models based on deep neural networks, such as RNNs,&#10;Wavenet and the Transformer attain state-of-the-art results on many tasks.&#10;However, they are difficult to parallelize and are thus slow at processing long&#10;sequences. RNNs lack parallelism both during training and decoding, while&#10;architectures like WaveNet and Transformer are much more parallelizable during&#10;training, yet still operate sequentially during decoding.&#10;  Inspired by [arxiv:1711.00937], we present a method to extend sequence models&#10;using discrete latent variables that makes decoding much more parallelizable.&#10;We first auto-encode the target sequence into a shorter sequence of discrete&#10;latent variables, which at inference time is generated autoregressively, and&#10;finally decode the output sequence from this shorter latent sequence in&#10;parallel. To this end, we introduce a novel method for constructing a sequence&#10;of discrete latent variables and compare it with previously introduced methods.&#10;Finally, we evaluate our model end-to-end on the task of neural machine&#10;translation, where it is an order of magnitude faster at decoding than&#10;comparable autoregressive models. While lower in BLEU than purely&#10;autoregressive models, our model achieves higher scores than previously&#10;proposed non-autoregressive translation models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.07181" label="1908.07181">
        <attvalues>
          <attvalue for="0" value="Latent-Variable Non-Autoregressive Neural Machine Translation with&#10;  Deterministic Inference Using a Delta Posterior" />
          <attvalue for="1" value="  Although neural machine translation models reached high translation quality,&#10;the autoregressive nature makes inference difficult to parallelize and leads to&#10;high translation latency. Inspired by recent refinement-based approaches, we&#10;propose LaNMT, a latent-variable non-autoregressive model with continuous&#10;latent variables and deterministic inference procedure. In contrast to existing&#10;approaches, we use a deterministic inference algorithm to find the target&#10;sequence that maximizes the lowerbound to the log-probability. During&#10;inference, the length of translation automatically adapts itself. Our&#10;experiments show that the lowerbound can be greatly increased by running the&#10;inference algorithm, resulting in significantly improved translation quality.&#10;Our proposed model closes the performance gap between non-autoregressive and&#10;autoregressive approaches on ASPEC Ja-En dataset with 8.6x faster decoding. On&#10;WMT'14 En-De dataset, our model narrows the gap with autoregressive baseline to&#10;2.0 BLEU points with 12.5x speedup. By decoding multiple initial latent&#10;variables in parallel and rescore using a teacher model, the proposed model&#10;further brings the gap down to 1.0 BLEU point on WMT'14 En-De task with 6.8x&#10;speedup.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.03249" label="1902.03249">
        <attvalues>
          <attvalue for="0" value="Insertion Transformer: Flexible Sequence Generation via Insertion&#10;  Operations" />
          <attvalue for="1" value="  We present the Insertion Transformer, an iterative, partially autoregressive&#10;model for sequence generation based on insertion operations. Unlike typical&#10;autoregressive models which rely on a fixed, often left-to-right ordering of&#10;the output, our approach accommodates arbitrary orderings by allowing for&#10;tokens to be inserted anywhere in the sequence during decoding. This&#10;flexibility confers a number of advantages: for instance, not only can our&#10;model be trained to follow specific orderings such as left-to-right generation&#10;or a binary tree traversal, but it can also be trained to maximize entropy over&#10;all valid insertions for robustness. In addition, our model seamlessly&#10;accommodates both fully autoregressive generation (one insertion at a time) and&#10;partially autoregressive generation (simultaneous insertions at multiple&#10;locations). We validate our approach by analyzing its performance on the WMT&#10;2014 English-German machine translation task under various settings for&#10;training and decoding. We find that the Insertion Transformer outperforms many&#10;prior non-autoregressive approaches to translation at comparable or better&#10;levels of parallelism, and successfully recovers the performance of the&#10;original Transformer while requiring only logarithmically many iterations&#10;during decoding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.02281" label="1711.02281">
        <attvalues>
          <attvalue for="0" value="Non-Autoregressive Neural Machine Translation" />
          <attvalue for="1" value="  Existing approaches to neural machine translation condition each output word&#10;on previously generated outputs. We introduce a model that avoids this&#10;autoregressive property and produces its outputs in parallel, allowing an order&#10;of magnitude lower latency during inference. Through knowledge distillation,&#10;the use of input token fertilities as a latent variable, and policy gradient&#10;fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative&#10;to the autoregressive Transformer network used as a teacher. We demonstrate&#10;substantial cumulative improvements associated with each of the three aspects&#10;of our training strategy, and validate our approach on IWSLT 2016&#10;English-German and two WMT language pairs. By sampling fertilities in parallel&#10;at inference time, our non-autoregressive model achieves near-state-of-the-art&#10;performance of 29.8 BLEU on WMT 2016 English-Romanian.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.09239" label="2409.09239">
        <attvalues>
          <attvalue for="0" value="Autoregressive + Chain of Thought $\simeq$ Recurrent: Recurrence's Role&#10;  in Language Models' Computability and a Revisit of Recurrent Transformer" />
          <attvalue for="1" value="  The Transformer architecture excels in a variety of language modeling tasks,&#10;outperforming traditional neural architectures such as RNN and LSTM. This is&#10;partially due to its elimination of recurrent connections, which allows for&#10;parallel training and a smoother flow of gradients. However, this move away&#10;from recurrent structures places the Transformer model at the lower end of&#10;Chomsky's computational hierarchy, imposing limitations on its computational&#10;abilities. Consequently, even advanced Transformer-based models face&#10;considerable difficulties in tasks like counting, string reversal, and&#10;multiplication. These tasks, though seemingly elementary, require a level of&#10;computational complexity that exceeds the capabilities of the Transformer&#10;architecture. Concurrently, the emergence of ``Chain of Thought&quot; (CoT)&#10;prompting has enabled Transformer-based language models to tackle tasks that&#10;were previously impossible or poorly executed. In this work, we thoroughly&#10;investigate the influence of recurrent structures in neural models on their&#10;reasoning abilities and computability, contrasting the role autoregression&#10;plays in the neural models' computational power. We then shed light on how the&#10;CoT approach can mimic recurrent computation and act as a bridge between&#10;autoregression and recurrence in the context of language models. It is this&#10;approximated recurrence that notably improves the model's performance and&#10;computational capacity. Moreover, we revisit recent recurrent-based Transformer&#10;model designs, focusing on their computational abilities through our proposed&#10;concept of ``recurrence-completeness&quot; and identify key theoretical limitations&#10;in models like Linear Transformer and RWKV. Through this, we aim to provide&#10;insight into the neural model architectures and prompt better model design.&#10;" />
          <attvalue for="2" value="&#10;&#10;The emergence of large language models (LLMs) \cite{achiam2023gpt,touvron2023llama,jiang2023mistral}, featuring billions to trillions of parameters, marks significant progress in diverse language-related tasks \cite{chang2024survey,thirunavukarasu2023large,zhang2023don,wu2023bloomberggpt,beltagy2019scibert}. However, growing concerns have arisen over the limitations~\cite{dziri2024faith,valmeekam2022large,ullman2023large} of current LLMs, particularly their difficulties with basic tasks such as multiplication or counting. While much of the debate centers on training techniques and data choice~\cite{yu2024large}, it is crucial to also consider the theoretical limitations of the computational capabilities of these models, which fundamentally depend on their core architecture, Transformers~\cite{vaswani2017attention}.&#10;&#10;In contrast to deterministic models like state machines or $K$ Nearest Neighbor classifiers, whose computational power is entirely reliant on their architectural (algorithm) design, the power of Neural Networks hinges upon a combination of both architecture~\cite{zhou2019analysis} and network optimization~\cite{deletang2022neural}. A Neural Network starting with random weights without any optimization, regardless of its architecture, has limited computational ability. Conversely, a network with a single linear layer without activation functions, even if perfectly optimized, is limited to capturing only basic linear relationships. &#10;&#10;Prior research~\cite{deletang2022neural,dziri2024faith,svete2024lower, chiang2301tighter} has demonstrated that recurrent neural networks~\cite{medsker2001recurrent} (RNNs and LSTMs) possess strong computational capabilities when optimally tuned, as supported by both empirical~\cite{deletang2022neural,dziri2024faith} and theoretical~\cite{svete2024lower, chiang2301tighter} studies. However, recurrent networks face significant optimization challenges~\cite{alqushaibi2020review}, such as the inability to parallelize during training and susceptibility to gradient vanishing~\cite{hochreiter1998vanishing} or exploding~\cite{kanai2017preventing} with longer sequences~\cite{ribeiro2020beyond}, which limits their scalability with large models and datasets. Conversely, the Transformer model replaces recurrence with an attention mechanism, enabling parallel training and mitigating the gradient vanishing issue through multiple gradient paths~\cite{abnar2020quantifying}. This innovation has made Transformers the leading choice for scalability~\cite{kaplan2020scaling} and optimization efficiency with large training data and model sizes. Nonetheless, the removal of recurrence imposes significant limitations on many reasoning tasks, as shown in multiple previous works~\cite{deletang2022neural,dziri2024faith}. Our work further examines in depth the different roles of recurrence and autoregression in neural networks, revealing the necessity of recurrence for higher computational power.&#10;&#10;The introduction of Chain of Thought (CoT) prompting~\cite{wei2022chain} represents a significant advancement for transformer-based language models, greatly enhancing performance across a range of tasks~\cite{chu2023survey}, including those beyond the computational capacity of the Transformer architecture. Despite substantial research analyzing the logic behind CoT, much of it interprets CoT from a psychological perspective~\cite{miao2024chain,li2024chain} as this way of reasoning is more human-like. Additionally, previous studies have examined CoT's role in knowledge extraction in LLMs~\cite{zhu2023physics}, which can differ from its role in reasoning processes.&#10;In this work, we elucidate CoT from a computability perspective, demonstrating that CoT approximates the omitted recurrence in the Transformer architecture. We show that CoT acts as a bridge between autoregression and recurrence, backing our claim with extensive experimental results and case studies involving tasks of varying computational levels.&#10;&#10;Lastly, we revisit recent efforts to modify the Transformer architecture to be recurrent, including various architectural designs such as the universal Transformer~\cite{dehghani2018universal} and the linear Transformer~\cite{katharopoulos2020transformers}. However, we find that some of these so-called ``recurrent'' designs are primarily intended to reduce inference costs and do not enhance the model's computational power. We analyze the computational capabilities of each design and their ability to model recurrent functions through our proposed concept of Recurrence-Completeness. Our analysis identifies key limitations of several recently proposed architectural modifications.&#10;&#10;The contributions of this work are:&#10;1) We define and contrast the concepts of recurrence and autoregression in neural networks, providing an in-depth analysis of their impact on a model's computational power.&#10;2) We examine CoT from a computational standpoint, highlighting its role in bridging autoregression and recurrence in LLMs, supported by empirical evidence.&#10;3) We systematically revisit and analyze recent recurrent modifications of Transformer architectures, uncovering the advantages and disadvantages of each design from a computational perspective.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Design, Computer Science, Linguistics, Language Model Architecture, Artificial Intelligence, Mathematics, Computational Complexity" />
        </attvalues>
      </node>
      <node id="2305.16339" label="2305.16339">
        <attvalues>
          <attvalue for="0" value="Don't Trust ChatGPT when Your Question is not in English: A Study of&#10;  Multilingual Abilities and Types of LLMs" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated exceptional natural language&#10;understanding abilities and have excelled in a variety of natural language&#10;processing (NLP)tasks in recent years. Despite the fact that most LLMs are&#10;trained predominantly in English, multiple studies have demonstrated their&#10;comparative performance in many other languages. However, fundamental questions&#10;persist regarding how LLMs acquire their multi-lingual abilities and how&#10;performance varies across different languages. These inquiries are crucial for&#10;the study of LLMs since users and researchers often come from diverse language&#10;backgrounds, potentially influencing their utilization and interpretation of&#10;LLMs' results. In this work, we propose a systematic way of qualifying the&#10;performance disparities of LLMs under multilingual settings. We investigate the&#10;phenomenon of across-language generalizations in LLMs, wherein insufficient&#10;multi-lingual training data leads to advanced multi-lingual capabilities. To&#10;accomplish this, we employ a novel back-translation-based prompting method. The&#10;results show that GPT exhibits highly translating-like behaviour in&#10;multilingual settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Linguists categorize bilingual individuals &#10;into three groups: &#10;compound, coordinate, and subordinate bilinguals~\cite{d1990three}. &#10;Figure \ref{fig:biling} illustrates this categorization,&#10;showing how individuals with different types of English-French bilingualism&#10;might internally represent &#10;the concept of ``knife''.&#10;&#10;Compound bilingualism mostly emerges among individuals who learn two languages simultaneously from birth. &#10;In this case, both languages are equally dominant and integrated, blurring any clear distinction between them and giving the impression of a single unified language~\cite{moradi2014investigation}. &#10;Compound bilingualism entails a shared mental representation of lexicons across both languages they acquire,&#10;and compound bilinguals &#10;are the most flexible in their use of multiple languages, &#10;exhibiting the ability to switch between languages without losing consistency in&#10;linguistic tasks~\cite{de1991lexical}. &#10;&#10;In contrast, individuals exhibiting coordinate bilingualism &#10;maintain separate mental representations for the lexicon of each language they learn. &#10;This separation leads to differences when tasks are performed under different language settings~\cite{jakobovits1968dimensionality}. &#10;&#10;Finally, subordinate bilingualism is characterized by a ``translator'' behaviour~\cite{marcos1976linguistic}.&#10;This type of bilingualism is characterized by a single lexicon representation that is linked to their dominant language~\cite{lorscher2012bilingualism}. &#10;When performing tasks in languages other than their dominant one, &#10;subordinate bilinguals tend to rely on translating the task into their dominant language, formulating an answer in the dominant language, &#10;and then translating that answer back into the language of the task. &#10;As a result, subordinate bilinguals may experience lower proficiency in&#10;communicating and completing tasks in the second, subordinate language.&#10;&#10;Despite the demonstration &#10;in prior work&#10;of consistent multilingual performance in many large language models \cite{shi2023language}, it remains unclear how the multilingualism of LLMs &#10;should be categorized. &#10;It is an open question &#10;whether the LLMs exhibit &#10;a representation of knowledge shared across both languages&#10;(compound), separate representations for each language&#10;(coordinate), or whether they &#10;rely on a translation processes&#10;(subordinate). We develop an experimental framework aimed at using performance on various natural language processing tasks to determine how the multilingual abilities of LLMs relate to these categories.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1903.10676" label="1903.10676">
        <attvalues>
          <attvalue for="0" value="SciBERT: A Pretrained Language Model for Scientific Text" />
          <attvalue for="1" value="  Obtaining large-scale annotated data for NLP tasks in the scientific domain&#10;is challenging and expensive. We release SciBERT, a pretrained language model&#10;based on BERT (Devlin et al., 2018) to address the lack of high-quality,&#10;large-scale labeled scientific data. SciBERT leverages unsupervised pretraining&#10;on a large multi-domain corpus of scientific publications to improve&#10;performance on downstream scientific NLP tasks. We evaluate on a suite of tasks&#10;including sequence tagging, sentence classification and dependency parsing,&#10;with datasets from a variety of scientific domains. We demonstrate&#10;statistically significant improvements over BERT and achieve new&#10;state-of-the-art results on several of these tasks. The code and pretrained&#10;models are available at https://github.com/allenai/scibert/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.08399" label="2302.08399">
        <attvalues>
          <attvalue for="0" value="Large Language Models Fail on Trivial Alterations to Theory-of-Mind&#10;  Tasks" />
          <attvalue for="1" value="  Intuitive psychology is a pillar of common-sense reasoning. The replication&#10;of this reasoning in machine intelligence is an important stepping-stone on the&#10;way to human-like artificial intelligence. Several recent tasks and benchmarks&#10;for examining this reasoning in Large-Large Models have focused in particular&#10;on belief attribution in Theory-of-Mind tasks. These tasks have shown both&#10;successes and failures. We consider in particular a recent purported success&#10;case, and show that small variations that maintain the principles of ToM turn&#10;the results on their head. We argue that in general, the zero-hypothesis for&#10;model evaluation in intuitive psychology should be skeptical, and that outlying&#10;failure cases should outweigh average success rates. We also consider what&#10;possible future successes on Theory-of-Mind tasks by more powerful LLMs would&#10;mean for ToM tasks with people.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.15895" label="2306.15895">
        <attvalues>
          <attvalue for="0" value="Large Language Model as Attributed Training Data Generator: A Tale of&#10;  Diversity and Bias" />
          <attvalue for="1" value="  Large language models (LLMs) have been recently leveraged as training data&#10;generators for various natural language processing (NLP) tasks. While previous&#10;research has explored different approaches to training models using generated&#10;data, they generally rely on simple class-conditional prompts, which may limit&#10;the diversity of the generated data and inherit systematic biases of LLM. Thus,&#10;we investigate training data generation with diversely attributed prompts&#10;(e.g., specifying attributes like length and style), which have the potential&#10;to yield diverse and attributed generated data. Our investigation focuses on&#10;datasets with high cardinality and diverse domains, wherein we demonstrate that&#10;attributed prompts outperform simple class-conditional prompts in terms of the&#10;resulting model's performance. Additionally, we present a comprehensive&#10;empirical study on data generation encompassing vital aspects like bias,&#10;diversity, and efficiency, and highlight three key observations: firstly,&#10;synthetic datasets generated by simple prompts exhibit significant biases, such&#10;as regional bias; secondly, attribute diversity plays a pivotal role in&#10;enhancing model performance; lastly, attributed prompts achieve the performance&#10;of simple class-conditional prompts while utilizing only 5\% of the querying&#10;cost of ChatGPT associated with the latter. The data and code are available on&#10;\url{https://github.com/yueyu1030/AttrPrompt}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.11511" label="1912.11511">
        <attvalues>
          <attvalue for="0" value="An Analysis of the Expressiveness of Deep Neural Network Architectures&#10;  Based on Their Lipschitz Constants" />
          <attvalue for="1" value="  Deep neural networks (DNNs) have emerged as a popular mathematical tool for&#10;function approximation due to their capability of modelling highly nonlinear&#10;functions. Their applications range from image classification and natural&#10;language processing to learning-based control. Despite their empirical&#10;successes, there is still a lack of theoretical understanding of the&#10;representative power of such deep architectures. In this work, we provide a&#10;theoretical analysis of the expressiveness of fully-connected, feedforward DNNs&#10;with 1-Lipschitz activation functions. In particular, we characterize the&#10;expressiveness of a DNN by its Lipchitz constant. By leveraging random matrix&#10;theory, we show that, given sufficiently large and randomly distributed&#10;weights, the expected upper and lower bounds of the Lipschitz constant of a DNN&#10;and hence their expressiveness increase exponentially with depth and&#10;polynomially with width, which gives rise to the benefit of the depth of DNN&#10;architectures for efficient function approximation. This observation is&#10;consistent with established results based on alternative expressiveness&#10;measures of DNNs. In contrast to most of the existing work, our analysis based&#10;on the Lipschitz properties of DNNs is applicable to a wider range of&#10;activation nonlinearities and potentially allows us to make sensible&#10;comparisons between the complexity of a DNN and the function to be approximated&#10;by the DNN. We consider this work to be a step towards understanding the&#10;expressive power of DNNs and towards designing appropriate deep architectures&#10;for practical applications such as system control.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.02098" label="2207.02098">
        <attvalues>
          <attvalue for="0" value="Neural Networks and the Chomsky Hierarchy" />
          <attvalue for="1" value="  Reliable generalization lies at the heart of safe ML and AI. However,&#10;understanding when and how neural networks generalize remains one of the most&#10;important unsolved problems in the field. In this work, we conduct an extensive&#10;empirical study (20'910 models, 15 tasks) to investigate whether insights from&#10;the theory of computation can predict the limits of neural network&#10;generalization in practice. We demonstrate that grouping tasks according to the&#10;Chomsky hierarchy allows us to forecast whether certain architectures will be&#10;able to generalize to out-of-distribution inputs. This includes negative&#10;results where even extensive amounts of data and training time never lead to&#10;any non-trivial generalization, despite models having sufficient capacity to&#10;fit the training data perfectly. Our results show that, for our subset of&#10;tasks, RNNs and Transformers fail to generalize on non-regular tasks, LSTMs can&#10;solve regular and counter-language tasks, and only networks augmented with&#10;structured memory (such as a stack or memory tape) can successfully generalize&#10;on context-free and context-sensitive tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.08482" label="1906.08482">
        <attvalues>
          <attvalue for="0" value="Beyond exploding and vanishing gradients: analysing RNN training using&#10;  attractors and smoothness" />
          <attvalue for="1" value="  The exploding and vanishing gradient problem has been the major conceptual&#10;principle behind most architecture and training improvements in recurrent&#10;neural networks (RNNs) during the last decade. In this paper, we argue that&#10;this principle, while powerful, might need some refinement to explain recent&#10;developments. We refine the concept of exploding gradients by reformulating the&#10;problem in terms of the cost function smoothness, which gives insight into&#10;higher-order derivatives and the existence of regions with many close local&#10;minima. We also clarify the distinction between vanishing gradients and the&#10;need for the RNN to learn attractors to fully use its expressive power. Through&#10;the lens of these refinements, we shed new light on recent developments in the&#10;RNN field, namely stable RNN and unitary (or orthogonal) RNNs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00928" label="2005.00928">
        <attvalues>
          <attvalue for="0" value="Quantifying Attention Flow in Transformers" />
          <attvalue for="1" value="  In the Transformer model, &quot;self-attention&quot; combines information from attended&#10;embeddings into the representation of the focal embedding in the next layer.&#10;Thus, across layers of the Transformer, information originating from different&#10;tokens gets increasingly mixed. This makes attention weights unreliable as&#10;explanations probes. In this paper, we consider the problem of quantifying this&#10;flow of information through self-attention. We propose two methods for&#10;approximating the attention to input tokens given attention weights, attention&#10;rollout and attention flow, as post hoc methods when we use attention weights&#10;as the relative relevance of the input tokens. We show that these methods give&#10;complementary views on the flow of information, and compared to raw attention,&#10;both yield higher correlations with importance scores of input tokens obtained&#10;using an ablation method and input gradients.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.12875" label="2402.12875">
        <attvalues>
          <attvalue for="0" value="Chain of Thought Empowers Transformers to Solve Inherently Serial&#10;  Problems" />
          <attvalue for="1" value="  Instructing the model to generate a sequence of intermediate steps, a.k.a., a&#10;chain of thought (CoT), is a highly effective method to improve the accuracy of&#10;large language models (LLMs) on arithmetics and symbolic reasoning tasks.&#10;However, the mechanism behind CoT remains unclear. This work provides a&#10;theoretical understanding of the power of CoT for decoder-only transformers&#10;through the lens of expressiveness. Conceptually, CoT empowers the model with&#10;the ability to perform inherently serial computation, which is otherwise&#10;lacking in transformers, especially when depth is low. Given input length $n$,&#10;previous works have shown that constant-depth transformers with finite&#10;precision $\mathsf{poly}(n)$ embedding size can only solve problems in&#10;$\mathsf{TC}^0$ without CoT. We first show an even tighter expressiveness upper&#10;bound for constant-depth transformers with constant-bit precision, which can&#10;only solve problems in $\mathsf{AC}^0$, a proper subset of $ \mathsf{TC}^0$.&#10;However, with $T$ steps of CoT, constant-depth transformers using constant-bit&#10;precision and $O(\log n)$ embedding size can solve any problem solvable by&#10;boolean circuits of size $T$. Empirically, enabling CoT dramatically improves&#10;the accuracy for tasks that are hard for parallel computation, including the&#10;composition of permutation groups, iterated squaring, and circuit value&#10;problems, especially for low-depth transformers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.14316" label="2309.14316">
        <attvalues>
          <attvalue for="0" value="Physics of Language Models: Part 3.1, Knowledge Storage and Extraction" />
          <attvalue for="1" value="  Large language models (LLMs) can store a vast amount of world knowledge,&#10;often extractable via question-answering (e.g., &quot;What is Abraham Lincoln's&#10;birthday?&quot;). However, do they answer such questions based on exposure to&#10;similar questions during training (i.e., cheating), or by genuinely learning to&#10;extract knowledge from sources like Wikipedia?&#10;  In this paper, we investigate this issue using a controlled biography&#10;dataset. We find a strong correlation between the model's ability to extract&#10;knowledge and various diversity measures of the training data.&#10;$\textbf{Essentially}$, for knowledge to be reliably extracted, it must be&#10;sufficiently augmented (e.g., through paraphrasing, sentence shuffling,&#10;translations) $\textit{during pretraining}$. Without such augmentation,&#10;knowledge may be memorized but not extractable, leading to 0% accuracy,&#10;regardless of subsequent instruction fine-tuning.&#10;  To understand why this occurs, we employ (nearly) linear probing to&#10;demonstrate a strong connection between the observed correlation and how the&#10;model internally encodes knowledge -- whether it is linearly encoded in the&#10;hidden embeddings of entity names or distributed across other token embeddings&#10;in the training text.&#10;  This paper provides $\textbf{several key recommendations for LLM pretraining&#10;in the industry}$: (1) rewrite the pretraining data -- using small, auxiliary&#10;models -- to provide knowledge augmentation, and (2) incorporate more&#10;instruction-finetuning data into the pretraining stage before it becomes too&#10;late.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.03819" label="1807.03819">
        <attvalues>
          <attvalue for="0" value="Universal Transformers" />
          <attvalue for="1" value="  Recurrent neural networks (RNNs) sequentially process data by updating their&#10;state with each new data point, and have long been the de facto choice for&#10;sequence modeling tasks. However, their inherently sequential computation makes&#10;them slow to train. Feed-forward and convolutional architectures have recently&#10;been shown to achieve superior results on some sequence modeling tasks such as&#10;machine translation, with the added advantage that they concurrently process&#10;all inputs in the sequence, leading to easy parallelization and faster training&#10;times. Despite these successes, however, popular feed-forward sequence models&#10;like the Transformer fail to generalize in many simple tasks that recurrent&#10;models handle with ease, e.g. copying strings or even simple logical inference&#10;when the string or formula lengths exceed those observed at training time. We&#10;propose the Universal Transformer (UT), a parallel-in-time self-attentive&#10;recurrent sequence model which can be cast as a generalization of the&#10;Transformer model and which addresses these issues. UTs combine the&#10;parallelizability and global receptive field of feed-forward sequence models&#10;like the Transformer with the recurrent inductive bias of RNNs. We also add a&#10;dynamic per-position halting mechanism and find that it improves accuracy on&#10;several tasks. In contrast to the standard Transformer, under certain&#10;assumptions, UTs can be shown to be Turing-complete. Our experiments show that&#10;UTs outperform standard Transformers on a wide range of algorithmic and&#10;language understanding tasks, including the challenging LAMBADA language&#10;modeling task where UTs achieve a new state of the art, and machine translation&#10;where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De&#10;dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.00222" label="2409.00222">
        <attvalues>
          <attvalue for="0" value="Can Large Language Models Address Open-Target Stance Detection?" />
          <attvalue for="1" value="  Stance detection (SD) identifies a text's position towards a target,&#10;typically labeled as favor, against, or none. We introduce Open-Target Stance&#10;Detection (OTSD), the most realistic task where targets are neither seen during&#10;training nor provided as input. We evaluate Large Language Models (LLMs)&#10;GPT-4o, GPT-3.5, Llama-3, and Mistral, comparing their performance to the only&#10;existing work, Target-Stance Extraction (TSE), which benefits from predefined&#10;targets. Unlike TSE, OTSD removes the dependency of a predefined list, making&#10;target generation and evaluation more challenging. We also provide a metric for&#10;evaluating target quality that correlates well with human judgment. Our&#10;experiments reveal that LLMs outperform TSE in target generation when the real&#10;target is explicitly and not explicitly mentioned in the text. Likewise, for&#10;stance detection, LLMs excel in explicit cases with comparable performance in&#10;non-explicit in general.&#10;" />
          <attvalue for="2" value="&#10;&#10;Stance detection (SD) aims to determine the position of a text or person towards a certain target, typically categorized as “favor”, “against”, or “none”. The target can be mentioned explicitly in the text, or sometimes the idea of the target can be conveyed indirectly \cite{kuccuk2020stance}.&#10;In Zero-shot Stance Detection (ZSSD), a model predicts stances for targets it has not seen during training, which is crucial since collecting training data for every potential target is impractical \cite{allaway2021adversarial}.&#10;While recent research has focused on ZSSD \cite{zhang2023task, li2023tts, wen2023zero, liang2022zero, zhu2022enhancing, luo2022exploiting, xu2022openstance},&#10;most studies assume that the target is known or manually identified and given as input, a rare scenario in real-world applications where the target is often uncommon, unknown or not explicitly conveyed. Also, target annotation is an expensive task in SD \cite{kuccuk2020stance}. &#10;&#10;In this paper, we focus on a different yet challenging task we refer to as Open-Target Stance Detection (OTSD). In this task, the target is neither seen during training (zero-shot) nor provided as input to the model. In OTSD, one important challenge lies in identifying the target from the text rather than having it provided as input. The target may be mentioned explicitly or not and the stance is predicted with respect to the produced target. It is worth noting that in cross-target stance detection \cite{zhang2020enhancing}, while the target is also unseen during training, it typically belongs to domains similar to those in the training set.&#10;In contrast, in the OTSD task, we mostly consider scenarios where targets from unrelated or different domains may emerge. Both tasks fall under the category of ZSSD.&#10;There has been one notable attempt to partially address this real-world setting of SD.&#10;\cite{li2023new} generate targets from the input text and later detect stance based on these targets. However, in target generation, they map the generated targets to a predefined list of golden targets (details in §\ref{sec:related_work}) to ensure exact wording and facilitate evaluation. They call their approach TSE, which does not fully align with our OTSD setting. In OTSD we assume no input target information is given during the whole process, making it more practical than TSE, which requires a comprehensive list of all possible targets.&#10;Moreover, our experiment reveals a performance gap in \cite{li2023new}'s approach when used on text with explicit or non-explicit target mentions, a factor not addressed by the authors that warrants further investigation in the context of OTSD.&#10;In this work, we present and explore the OTSD task, focusing on its two main steps: Target Generation (TG) and Stance Detection (SD) (Eq. \ref{eq:our_task}). As a zero-shot task for both target and stance, Large Language Models (LLMs) appear well-suited for this challenge. Therefore, we examine the performance of LLMs and compare it to the primary existing work, TSE, which benefits from using a predefined list of targets during the process (Eq. \ref{eq:tse}). &#10;Our empirical study aims to address the following research questions:&#10;[RQ\#1] How do proprietary and open LLMs—particularly models from GPT, Gemini, Llama, and Mistral families,&#10;—perform in open TG compared to TSE when the real target is explicitly or non-explicitly mentioned in the text,&#10;and how do they compare to each other? [RQ\#2] How do the same LLMs perform compared to TSE in SD (in the context of OTSD) for both explicit and non-explicit cases, and how do they compare to each other? Our contributions consist of introducing the task of OTSD, providing a target evaluation metric,&#10;and conducting experiments answering the research questions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Models, Target Generation, Artificial Intelligence, Stance Detection, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2105.06603" label="2105.06603">
        <attvalues>
          <attvalue for="0" value="Adversarial Learning for Zero-Shot Stance Detection on Social Media" />
          <attvalue for="1" value="  Stance detection on social media can help to identify and understand slanted&#10;news or commentary in everyday life. In this work, we propose a new model for&#10;zero-shot stance detection on Twitter that uses adversarial learning to&#10;generalize across topics. Our model achieves state-of-the-art performance on a&#10;number of unseen test topics with minimal computational costs. In addition, we&#10;extend zero-shot stance detection to new topics, highlighting future directions&#10;for zero-shot transfer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.07492" label="2406.07492">
        <attvalues>
          <attvalue for="0" value="Paraphrasing in Affirmative Terms Improves Negation Understanding" />
          <attvalue for="1" value="  Negation is a common linguistic phenomenon. Yet language models face&#10;challenges with negation in many natural language understanding tasks such as&#10;question answering and natural language inference. In this paper, we experiment&#10;with seamless strategies that incorporate affirmative interpretations (i.e.,&#10;paraphrases without negation) to make models more robust against negation.&#10;Crucially, our affirmative interpretations are obtained automatically. We show&#10;improvements with CondaQA, a large corpus requiring reasoning with negation,&#10;and five natural language understanding tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Negation is a fundamental linguistic phenomenon present in all human languages~\cite{Horn1989-HORANH}.&#10;Language models underperform&#10;in various natural language understanding (NLU) tasks when the input includes negation. &#10;For example, &#10;\cite{ettinger-2020-bert} and &#10;\cite{kassner-schutze-2020-negated} &#10;show that BERT~\cite{devlin-etal-2019-bert} &#10;fails to distinguish between negated and non-negated cloze questions.&#10;Researchers have also shown that large language models such as GPT-3 &#10;\cite{NEURIPS2020_1457c0d6}&#10;and InstructGPT &#10;\cite{ouyang2022training}&#10;are insensitive to negation&#10;and fail to reason under negation &#10;\cite{truong-etal-2023-language}. &#10;\cite{jang-etal-2022-beyond}&#10;point out that language models violate the logical negation property&#10;($p$ is true iff $\neg p$ is false).&#10;\cite{hossain-etal-2022-analysis}&#10;analyze negation in&#10;eight popular corpora for six NLU tasks.&#10;They conclude that &#10;(a)~NLU corpora have few negations compared to general-purpose texts &#10;and&#10;(b)~the few negations in them are often unimportant.&#10;To our knowledge, CondaQA~\cite{ravichander-etal-2022-condaqa}&#10;is the largest benchmark (14,182 question-answer pairs from Wikipedia)&#10;requiring reasoning over the implications of negations.&#10;&#10;In this paper, we paraphrase sentences with negation without using negation&#10;to make models for natural language understanding more robust when negation is present in the input.&#10;We will use the term affirmative interpretation to refer to paraphrases without negation&#10;(e.g., I am not sad: I am just ok, I am happy, etc.).&#10;Appendix \ref{sec:affirmativeexamples} provides examples of how affirmative interpretations differ from simple paraphrases. &#10;&#10;The main contributions of this paper are&#10;(a)~strategies to generate and incorporate affirmative interpretations &#10;and&#10;(b) experimental results demonstrating that doing so yields better results.&#10;In addition to CondaQA, we experiment with&#10;five of the eight corpora analyzed by&#10;\cite{hossain-etal-2022-analysis}:&#10;CommonsenseQA~\cite{talmor-etal-2019-commonsenseqa}, &#10;STS-B~\cite{cer-etal-2017-semeval}, &#10;QNLI~\cite{rajpurkar-etal-2016-squad},&#10;WiC~\cite{pilehvar-camacho-collados-2019-wic},&#10;and &#10;WSC~\cite{levesque_winograd_2012}.&#10;We do not experiment with the other three corpora because&#10;they do not contain any negation \cite[COPA]{roemmele-etal-2011-choice},&#10;there is no difference in results when negation is present~\cite[QQP; 0.01 in macro F1]{cer-etal-2017-semeval},&#10;or has already been shown~\cite{hossain-blanco-2022-leveraging} to benefit from affirmative interpretations~\cite[SST-2]{socher-etal-2013-recursive}.&#10;The corpora we experiment with are in English.&#10; Related Work&#10;&#10;Early research on negation targeted detecting negating cues and&#10;generating semantic representations,&#10;usually by identifying the scope and focus&#10;\cite{Morante2011NegationCues,morante-daelemans-2012-conandoyle,van-son-etal-2016-building,khandelwal-sawant-2020-negbert,truong-etal-2022-improving}.&#10;&#10;More recent works bypass formal representations.&#10;Instead, they make neural models robust when the input contains negation.&#10;\cite{hosseini-etal-2021-understanding} combine unlikelihood training and syntactic data augmentation&#10;to enhance the ability of BERT to understand negation with negated LAMA~\cite{kassner-schutze-2020-negated}.&#10;\cite{singh-etal-2023-nlms} present a pretraining strategy designed for negation.&#10;Unlike these works, we couple original inputs containing negation with affirmative interpretations.&#10;&#10;The first work on affirmative interpretations was by&#10;\cite{sarabi-etal-2019-corpus}.&#10;\cite{hossain-etal-2022-question}&#10;present AFIN, &#10;a corpus of $\approx$3,000 sentences with negations and their affirmative interpretations.&#10;These two previous works are limited to generating affirmative interpretations from negations;&#10;they do not provide extrinsic evaluations.&#10;More recently, &#10;\cite{hossain-blanco-2022-leveraging}&#10;present Large-AFIN, &#10;over 153,000 pairs of sentences with negation and their affirmative interpretations&#10;obtained from parallel corpora via backtranslation.&#10;In this paper, we present strategies to generate affirmative interpretations that do not require parallel corpora or a machine translation system.&#10;Moreover, we demonstrate that incorporating affirmative interpretations yields better results with CondaQA&#10;and five other natural language understanding tasks.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Negation Handling Strategies, Language Model Robustness, Natural Language Understanding, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2205.03815" label="2205.03815">
        <attvalues>
          <attvalue for="0" value="Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text&#10;  Correspondence" />
          <attvalue for="1" value="  The logical negation property (LNP), which implies generating different&#10;predictions for semantically opposite inputs, is an important property that a&#10;trustworthy language model must satisfy. However, much recent evidence shows&#10;that large-size pre-trained language models (PLMs) do not satisfy this&#10;property. In this paper, we perform experiments using probing tasks to assess&#10;PLM's LNP understanding. Unlike previous studies that only examined negation&#10;expressions, we expand the boundary of the investigation to lexical semantics.&#10;Through experiments, we observe that PLMs violate the LNP frequently. To&#10;alleviate the issue, we propose a novel intermediate training task, names&#10;meaning-matching, designed to directly learn a meaning-text correspondence,&#10;instead of relying on the distributional hypothesis. Through multiple&#10;experiments, we find that the task enables PLMs to learn lexical semantic&#10;information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm&#10;that it is a safe intermediate task that guarantees a similar or better&#10;performance of downstream tasks. Finally, we observe that our proposed approach&#10;outperforms our previous counterparts despite its time and resource efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.08929" label="2203.08929">
        <attvalues>
          <attvalue for="0" value="An Analysis of Negation in Natural Language Understanding Corpora" />
          <attvalue for="1" value="  This paper analyzes negation in eight popular corpora spanning six natural&#10;language understanding tasks. We show that these corpora have few negations&#10;compared to general-purpose English, and that the few negations in them are&#10;often unimportant. Indeed, one can often ignore negations and still make the&#10;right predictions. Additionally, experimental results show that&#10;state-of-the-art transformers trained with these corpora obtain substantially&#10;worse results with instances that contain negation, especially if the negations&#10;are important. We conclude that new corpora accounting for negation are needed&#10;to solve natural language understanding tasks when negation is present.&#10;" />
          <attvalue for="2" value="&#10;Natural language understanding (NLU) is an umbrella term used to refer to &#10;any task that requires text understanding. &#10;For example,&#10;question answering~\cite{rajpurkar-etal-2016-squad},&#10;information extraction~\cite{stanovsky-etal-2018-supervised},&#10;coreference resolution~\cite{wu-etal-2020-corefqa}, and&#10;machine reading~\cite{yang-etal-2019-enhancing-pre}, among many others,&#10;are tasks that fall under natural language understanding.&#10;The threshold for claiming that a system understands natural language is ever-moving.&#10;New corpora are often justified by pointing out that state-of-the-art models do not obtain good results.&#10;After years of steady improvements, more powerful models eventually obtain so-called human performance,&#10;and at that point new, more challenging corpora are created.&#10;&#10;Many corpora for natural language understanding tasks&#10;contain language generated by annotators rather than retrieved from texts written independently of the corpus creation process.&#10;These corpora are certainly useful and have facilitated tremendous progress.&#10;Annotator-generated examples, however,&#10;carry the risk of evaluating systems with synthetic language that is not representative of language in the wild.&#10;For example, annotators are likely to use negation when asked to write a text that contradicts something&#10;despite contradictions in the wild need not have a negation~\cite{gururangan-etal-2018-annotation}.&#10;Recently,&#10;\newcite{kwiatkowski-etal-2019-natural} present a large corpus for question answering&#10;that consists of natural questions (\ie, asked by somebody with a real information need)&#10;in order to encourage research in a more realistic scenario.&#10;This contrasts with previous corpora, where the questions were written by annotators after being told the answer~\cite{rajpurkar-etal-2016-squad}.&#10;&#10;In this paper, we explore the role of negation in eight corpora for six popular natural language understanding tasks.&#10;Our goal is to check whether negation plays the role it deserves in these tasks.&#10;To our surprise, we conclude that negation is virtually ignored by answering the following questions:&#10;&#10;\begin{compactenum}&#10; \item Do NLU corpora contain as many negations as general-purpose texts?~(they don't);&#10; \item Do the (few) negations in NLU corpora play a role in solving the tasks?~(they don't); and&#10; \item Do state-of-the-art transformers trained with NLU corpora face challenges with instances that contain negation?&#10; (they do, especially if the negation is important).&#10;\end{compactenum}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.00937" label="1811.00937">
        <attvalues>
          <attvalue for="0" value="CommonsenseQA: A Question Answering Challenge Targeting Commonsense&#10;  Knowledge" />
          <attvalue for="1" value="  When answering a question, people often draw upon their rich world knowledge&#10;in addition to the particular context. Recent work has focused primarily on&#10;answering questions given some relevant document or context, and required very&#10;little general background. To investigate question answering with prior&#10;knowledge, we present CommonsenseQA: a challenging new dataset for commonsense&#10;question answering. To capture common sense beyond associations, we extract&#10;from ConceptNet (Speer et al., 2017) multiple target concepts that have the&#10;same semantic relation to a single source concept. Crowd-workers are asked to&#10;author multiple-choice questions that mention the source concept and&#10;discriminate in turn between each of the target concepts. This encourages&#10;workers to create questions with complex semantics that often require prior&#10;knowledge. We create 12,247 questions through this procedure and demonstrate&#10;the difficulty of our task with a large number of strong baselines. Our best&#10;baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy,&#10;well below human performance, which is 89%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.14486" label="2210.14486">
        <attvalues>
          <attvalue for="0" value="Leveraging Affirmative Interpretations from Negation Improves Natural&#10;  Language Understanding" />
          <attvalue for="1" value="  Negation poses a challenge in many natural language understanding tasks.&#10;Inspired by the fact that understanding a negated statement often requires&#10;humans to infer affirmative interpretations, in this paper we show that doing&#10;so benefits models for three natural language understanding tasks. We present&#10;an automated procedure to collect pairs of sentences with negation and their&#10;affirmative interpretations, resulting in over 150,000 pairs. Experimental&#10;results show that leveraging these pairs helps (a) T5 generate affirmative&#10;interpretations from negations in a previous benchmark, and (b) a RoBERTa-based&#10;classifier solve the task of natural language inference. We also leverage our&#10;pairs to build a plug-and-play neural generator that given a negated statement&#10;generates an affirmative interpretation. Then, we incorporate the pretrained&#10;generator into a RoBERTa-based classifier for sentiment analysis and show that&#10;doing so improves the results. Crucially, our proposal does not require any&#10;manual effort.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.04012" label="2205.04012">
        <attvalues>
          <attvalue for="0" value="Improving negation detection with negation-focused pre-training" />
          <attvalue for="1" value="  Negation is a common linguistic feature that is crucial in many language&#10;understanding tasks, yet it remains a hard problem due to diversity in its&#10;expression in different types of text. Recent work has shown that&#10;state-of-the-art NLP models underperform on samples containing negation in&#10;various tasks, and that negation detection models do not transfer well across&#10;domains. We propose a new negation-focused pre-training strategy, involving&#10;targeted data augmentation and negation masking, to better incorporate negation&#10;information into language models. Extensive experiments on common benchmarks&#10;show that our proposed approach improves negation detection performance and&#10;generalizability over the strong baseline NegBERT (Khandewal and Sawant, 2020).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;To date, negation detection has been heavily reliant on rule-based systems.&#10;\cite{chapman2001simple} proposed a simple system, NegEx, based on regular expressions to detect negation cues in a sentence given a concept of interest (the scope).&#10;NegEx remains the most popular approach to negation detection, especially in the clinical domain &#10;to determine the polarity of clinical concepts (e.g., as sourced from MetaMap \cite{aronson2010metamap}).&#10;Further research has extended NegEx with syntactic information \cite{mehrabi2015deepen,peng2018negbio}, and&#10;shown that rule-based systems can achieve relatively good performance for detecting negation, especially in the biomedical domain, but do not generalize well to other domains or datasets. &#10;&#10;To approach negation cue and negation scope detection with supervised machine learning,&#10;two classification tasks are defined: (1) finding negation tokens, and (2) classifying tokens as the first or last (or neither) token within the scope of negation. Most work follows a common scheme in extracting various features from the sentence, and using a classifier to classify each token as the beginning, inside, or outside of a negation cue or scope span \cite{morante2009metalearning,ou2015automatic,cruz2016machine}. &#10;Recently, research has shifted to applying deep learning methods to the task. Most approaches make use of RNN-based architectures to encode the input sentences, combined with a softmax layer for classification \cite{lazib2019negation, chen2019attention}. Despite the high performance on common benchmarks, results are biased by the fact that negation scope is often delimited by punctuation and other dataset artefacts \cite{fancellu2017detecting}. As such, they are potentially only learning domain-specific surface features rather than capturing the true semantics of negation.&#10;NegBERT applies a large pre-trained language model to the problem of negation detection, outperforming previous deep learning methods on negation detection, with especially high gains on scope resolution. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2105.03519" label="2105.03519">
        <attvalues>
          <attvalue for="0" value="Understanding by Understanding Not: Modeling Negation in Language Models" />
          <attvalue for="1" value="  Negation is a core construction in natural language. Despite being very&#10;successful on many tasks, state-of-the-art pre-trained language models often&#10;handle negation incorrectly. To improve language models in this regard, we&#10;propose to augment the language modeling objective with an unlikelihood&#10;objective that is based on negated generic sentences from a raw text corpus. By&#10;training BERT with the resulting combined objective we reduce the mean top~1&#10;error rate to 4% on the negated LAMA dataset. We also see some improvements on&#10;the negated NLI benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-trained language models have shown impressive results across many tasks, such as question answering \cite{DBLP:journals/corr/abs-1901-08634} and natural language inference \cite{DBLP:journals/corr/abs-1907-11692}. These models are also known to encode factual and common-sense knowledge \cite{radford2019language, DBLP:conf/emnlp/PetroniRRLBWM19,bosselut-etal-2019-comet}.&#10;Despite these abilities, \cite{kassner2019negated} found that these models fail at understanding negation through analysing negated factual statements.&#10;&#10;Extensive literature looks at the linguistic knowledge learned by language models \cite{DBLP:conf/acl/McCoyPL19, jumelet-hupkes-2018-language, gulordava-etal-2018-colorless, marvin-linzen-2018-targeted, bertpipeline, DBLP:journals/corr/abs-1901-03438, DBLP:journals/corr/abs-1912-13283}. &#10;Recent work has also studied the shortcomings in negation scope detection \cite{jumelet-hupkes-2018-language, fancellu-etal-2016-neural, fancellu-etal-2017-detecting, DBLP:conf/conll/MoranteD09, li-lu-2018-learning, zhao-bethard-2020-berts, attentionnegationchen} and focus detection \cite{DBLP:conf/emnlp/ShenZHZZA19, DBLP:conf/acl/ZouZZ14, DBLP:conf/emnlp/ZouZZ15, hossain-etal-2020-predicting}.&#10;\cite{DBLP:conf/coling/NaikRSRN18} and \cite{DBLP:conf/acl/McCoyPL19} systematically study the linguistic abilities of these models using NLI, and show that these models rely on erroneous syntactic heuristics.&#10;Our work is in this spirit for negations.&#10;&#10;\cite{DBLP:journals/corr/abs-2004-02451} propose taking advantage of negative examples and unlikelihood in the training of language models to increase their syntactic abilities. &#10;Similarly, \cite{DBLP:journals/corr/abs-2004-11999} show the effectiveness of syntactic data augmentation in the case of robustness in NLI.&#10;Neither of these works focus on negations.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2205.11467" label="2205.11467">
        <attvalues>
          <attvalue for="0" value="A Question-Answer Driven Approach to Reveal Affirmative Interpretations&#10;  from Verbal Negations" />
          <attvalue for="1" value="  This paper explores a question-answer driven approach to reveal affirmative&#10;interpretations from verbal negations (i.e., when a negation cue grammatically&#10;modifies a verb). We create a new corpus consisting of 4,472 verbal negations&#10;and discover that 67.1% of them convey that an event actually occurred.&#10;Annotators generate and answer 7,277 questions for the 3,001 negations that&#10;convey an affirmative interpretation. We first cast the problem of revealing&#10;affirmative interpretations from negations as a natural language inference&#10;(NLI) classification task. Experimental results show that state-of-the-art&#10;transformers trained with existing NLI corpora are insufficient to reveal&#10;affirmative interpretations. We also observe, however, that fine-tuning brings&#10;small improvements. In addition to NLI classification, we also explore the more&#10;realistic task of generating affirmative interpretations directly from&#10;negations with the T5 transformer. We conclude that the generation task remains&#10;a challenge as T5 substantially underperforms humans.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Revealing affirmative interpretations from negations is a challenging endeavor.&#10;In the literature, researchers primarily seek to identify scope and focus of negation. &#10;The creation of the BioScope \cite{Szarvas:2008:BCA:1572306.1572314} and ConanDoyle-Neg~ \cite{L12-1077} corpora spearheaded research on scope detection~\cite{morante-daelemans-2009-metalearning}.&#10;Proposals include using&#10;traditional machine learning~\cite{lapponi-etal-2012-uio},&#10;off-the-shelf semantic parsers and semantic representations~\cite{packard-etal-2014-simple}, and&#10;neural networks~\cite{fancellu-etal-2016-neural,fancellu-etal-2017-detecting}.&#10;PB-FOC~\cite{blanco-moldovan:2011:ACL-HLT20111} is the largest corpus with focus of negation annotations.&#10;Recent proposals for focus detection include graph-based models with discourse information~\cite{zou-etal-2014-negation, zou-etal-2015-unsupervised},&#10;neural networks with word-level and topic-level attention~\cite{shen-etal-2019-negative},&#10;and networks using scope information and context~\cite{hossain-etal-2020-predicting}.&#10;Scope and focus are useful to identify what is and what is not negated in a negated statement.&#10;Consider the second example in Table~\ref{t:motivational-examples}.&#10;Scope and focus do reveal that It was formed---everything but the focus (i.e., by a natural process) is affirmative---but provide no hints about how it was formed (i.e., by an artificial process, artificially, etc).&#10;The main goal of this paper is to find these affirmative counterparts to generate affirmative interpretations.&#10;&#10;More related to the work presented here,&#10;\newcite{sarabi-etal-2019-corpus} present a corpus of negations and their underlying affirmative interpretations (they call them positive interpretations).&#10;We are inspired by them but bypass several of their limitations. &#10;First, they only work with negations from Simple Wikipedia,&#10;a site devoted to English learners.&#10;As a result, their corpus uses (relatively) unsophisticated vocabulary and grammar.&#10;Second, they impose several restrictions on the negations they work with&#10;(e.g., negation cue modifies root verb,&#10;sentences between 6 and 25 tokens and not including certain tokens (because, until, etc.)).&#10;Third, their affirmative interpretations are restricted to a rephrasing of the statement containing negation with only one change: an argument of the negated predicate.&#10;In contrast, we barely impose restrictions on the negations we work with (no questions and no auxiliary verbs). &#10;More importantly, we introduce a question-driven approach that allow us to obtain multiple affirmative interpretations with increasing degrees of complexity (see examples in Table \ref{t:qas-to-ai}).&#10;&#10;Recently, \newcite{jiang-etal-2021-im} study the problem of identifying commonsense implications of negations and contradictions.&#10;More specifically, &#10;they work with if-then rules such as &#10;If X does not learn new things, then X does not gain new knowledge and&#10;If X does not leave the building, then X stays in the building.&#10;These rules capture general commonsense knowledge about what happens if an event does not occur.&#10;Unlike them, we work with naturally occurring sentences that include negated predicate-argument structures with many arguments (agent, theme, manner, time, etc.). &#10;In addition, our affirmative interpretations reveal that predicates that are grammatically negated are actually factual (but with different arguments).&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.07345" label="2106.07345">
        <attvalues>
          <attvalue for="0" value="Self-Guided Contrastive Learning for BERT Sentence Representations" />
          <attvalue for="1" value="  Although BERT and its variants have reshaped the NLP landscape, it still&#10;remains unclear how best to derive sentence embeddings from such pre-trained&#10;Transformers. In this work, we propose a contrastive learning method that&#10;utilizes self-guidance for improving the quality of BERT sentence&#10;representations. Our method fine-tunes BERT in a self-supervised fashion, does&#10;not rely on data augmentation, and enables the usual [CLS] token embeddings to&#10;function as sentence vectors. Moreover, we redesign the contrastive learning&#10;objective (NT-Xent) and apply it to sentence representation learning. We&#10;demonstrate with extensive experiments that our approach is more effective than&#10;competitive baselines on diverse sentence-related tasks. We also show it is&#10;efficient at inference and robust to domain shifts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Contrastive Representation Learning.&#10;Contrastive learning has been long considered as effective in constructing meaningful representations.&#10;For instance, \cite{mikolov2013distributed} propose to learn word embeddings by framing words nearby a target word as positive samples while others as negative.&#10;\cite{logeswaran2018an} generalize the approach of \cite{mikolov2013distributed} for sentence representation learning.&#10;More recently, several studies \cite{fang2020cert,giorgi2020declutr,wu2020clear} suggest to utilize contrastive learning for training Transformer models, similar to our approach.&#10;However, they generally require data augmentation techniques, e.g., back-translation \cite{sennrich2016improving}, or prior knowledge on training data such as order information, while our method does not.&#10;Furthermore, we focus on revising BERT for computing better sentence embeddings rather than training a language model from scratch.&#10;&#10;On the other hand, contrastive learning has been also receiving much attention from the computer vision community (\cite{chen-etal-2020-a,chen2020exploring,he2020momentum}, inter alia).&#10;We improve the framework of \cite{chen-etal-2020-a} by optimizing its learning objective for pre-trained Transformer-based sentence representation learning.&#10;For extensive surveys on contrastive learning, refer to \cite{le2020contrastive} and \cite{jaiswal2020survey}.&#10;&#10;Fine-tuning BERT with Supervision.&#10;&#10;It is not always trivial to fine-tune pre-trained Transformer models of gigantic size with success, especially when the number of target domain data is limited \cite{mosbach2020stability}.&#10;To mitigate this training instability problem, several approaches \cite{aghajanyan2020better,jiang-etal-2020-smart,Zhu2020FreeLB} have been recently proposed.&#10;In particular, \cite{gunel2021supervised} propose to exploit contrastive learning as an auxiliary training objective during fine-tuning BERT with supervision from target tasks.&#10;In contrast, we deal with the problem of adjusting BERT when such supervision is not available.&#10;&#10;Sentence Embeddings from BERT.&#10;&#10;Since BERT and its variants are originally designed to be fine-tuned on each downstream task to attain their optimal performance, it remains ambiguous how best to extract general sentence representations from them, which are broadly applicable across diverse sentence-related tasks.&#10;Following \cite{conneau2017supervised}, \cite{reimers-gurevych-2019-sentence} (SBERT) propose to compute sentence embeddings by conducting mean pooling on the last layer of BERT and then fine-tuning the pooled vectors on the natural language inference (NLI) datasets \cite{bowman2015large,williams-etal-2018-broad}.&#10;Meanwhile, some other studies concentrate on more effectively leveraging the knowledge embedded in BERT to construct sentence embeddings without supervision.&#10;Specifically, \cite{wang2020sbert} propose a pooling method based on linear algebraic algorithms to draw sentence vectors from BERT's intermediate layers.&#10;\cite{li-etal-2020-sentence} suggest to learn a mapping from the average of the embeddings obtained from the last two layers of BERT to a spherical Gaussian distribution using a flow model, and to leverage the redistributed embeddings in place of the original BERT representations.&#10;We follow the setting of \cite{li-etal-2020-sentence} in that we only utilize plain text during training, however, unlike all the others that rely on a certain pooling method even after training, we directly refine BERT so that the typical \verb|[CLS]| vector can function as a sentence embedding.&#10;Note also that there exists concurrent work \cite{carlsson2021semantic,gao2021simcse,wang2021tsdae} whose motivation is analogous to ours, attempting to improve BERT sentence embeddings in an unsupervised fashion.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Contrastive Learning, NLP Transformers, Linguistics, Artificial Intelligence, Sentence Embeddings, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2208.11508" label="2208.11508">
        <attvalues>
          <attvalue for="0" value="PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for&#10;  Perturbation-Robust Slot Filling" />
          <attvalue for="1" value="  Most existing slot filling models tend to memorize inherent patterns of&#10;entities and corresponding contexts from training data. However, these models&#10;can lead to system failure or undesirable outputs when being exposed to spoken&#10;language perturbation or variation in practice. We propose a perturbed semantic&#10;structure awareness transferring method for training perturbation-robust slot&#10;filling models. Specifically, we introduce two MLM-based training strategies to&#10;respectively learn contextual semantic structure and word distribution from&#10;unsupervised language perturbation corpus. Then, we transfer semantic knowledge&#10;learned from upstream training procedure into the original samples and filter&#10;generated data by consistency processing. These procedures aim to enhance the&#10;robustness of slot filling models. Experimental results show that our method&#10;consistently outperforms the previous basic methods and gains strong&#10;generalization while preventing the model from memorizing inherent patterns of&#10;entities and contexts.&#10;" />
          <attvalue for="2" value="&#10;&#10;The slot filling (SF) task in the goal-oriented dialog system aims to identify task-related slot types in certain domains for understanding user utterances. Traditional supervised slot filling models and sequence labeling methods \cite{liu2015recurrent,Liu2016AttentionBasedRN,goo2018slot,niu2019novel,he2020multi,he2020learning,Wang2022InstructionNERAM} have shown remarkable performance. However, these models tend to memorize inherent patterns of entities and contexts \cite{wang2022miner,lin2021rockner}. Faced with uncertainty and diversity of human language expression, the perturbation of entities and contexts will lead to a decrease in the generalization ability of the SF model, which hinders its further application in practical dialog scenarios. &#10;&#10;Due to the variety of expression habits, users may not interact with the dialogue system abiding by a rigid input mode in real dialog scenarios. Instead, the expression styles of users would be of high lexical and syntactic diversity while users express their intentions. An interesting finding is that, every expression retains the key semantic information of the sentence to ensure consistency of the intention, but it inevitably damages the semantic structure of the context. As shown in Figure \ref{fig:intro}, the original sentence comes from training data, while the other two sentences are real queries of users with different language habits. Firstly, paraphrase and simplification perturb the contextual semantic structure of the original sentence to various degrees. Secondly, some slot entities also suffer from word perturbations. However, they all retain price-related information to express the same intention. We refer to the above two perturbations collectively as Spoken Language Perturbation. The previous slot filling model, which tends to memorize entity patterns, has a significantly reduced generalization ability when faced with these situations. Therefore, it is necessary to train a robust slot filling model against perturbations in practical application.&#10;&#10;Recently, improving the robustness of NLP systems against input perturbations has attracted increasing attention. Most existing studies \cite{wu2021bridging,moradi2021evaluating,gui2021textflint} that explored the robustness problem are only about rule-based synthetic datasets, which have certain limitations. Further, \cite{namysl2020nat} focused on the robustness of the NER model against Optical Character Recognition (OCR) disturbance and misspellings. However, real-world dialogue systems face more diverse perturbations due to frequent interactions with users. \cite{liu2020robustness} proposed Language understanding augmentation, which contains four data augmentation methods, to simulate natural perturbations. Nevertheless, each method is designed for a specific perturbation, which cannot generalize for other unknown perturbations.&#10;&#10;To solve the above issues, in this paper, we propose a Perturbed Semantic Structure Awareness Transferring method (PSSAT). It can generate augmented data based on human diversity expressions. In fact, it is not difficult to obtain unsupervised corpora containing spoken language perturbations in real-world scenarios (e.g. social media). Therefore, we extract the texts from two multi-modal datasets \cite{zhang2018adaptive,lu2018visual} and construct an unsupervised language perturbation corpus, which helps the model learn the semantic structure of perturbed data. To be specific, we introduce a perturbed structure pre-training stage, which guides the model to directly learn contextual semantic structure and words distribution from unsupervised language perturbation corpus through two different MLM-based training strategies, respectively. To better eliminate the distribution gap between upstream and downstream data, we design a Semantic Structure Transferring Generation stage to transfer the upstream learned semantic structure knowledge to downstream original training samples. By doing so, the generated augmented samples are more in line with the spoken language perturbation. However, as there are mixed perturbations existed in upstream corpus, the model may generate some low-quality samples. To alleviate this problem, we introduce Consistency Processing to filter generated samples. &#10;&#10;Our contributions are three-fold: (1) To the best of our knowledge, this is the first work to investigate spoken language perturbation of slot filling tasks and validate the vulnerability of existing rule-based methods in the condition of diverse language expressions. (2) We propose a perturbed semantic structure awareness transferring method, which transfers the learned contextual semantic structure and word distribution into the original samples through the MLM-based method. (3) Experiments demonstrate that our method outperforms all baseline methods and gains strong generalization while preventing the model from memorizing inherent patterns of entities and contexts.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Robustness Enhancement, Artificial Intelligence, Natural Language Processing, Slot Filling Models, Language Perturbation" />
        </attvalues>
      </node>
      <node id="1907.00390" label="1907.00390">
        <attvalues>
          <attvalue for="0" value="A Novel Bi-directional Interrelated Model for Joint Intent Detection and&#10;  Slot Filling" />
          <attvalue for="1" value="  A spoken language understanding (SLU) system includes two main tasks, slot&#10;filling (SF) and intent detection (ID). The joint model for the two tasks is&#10;becoming a tendency in SLU. But the bi-directional interrelated connections&#10;between the intent and slots are not established in the existing joint models.&#10;In this paper, we propose a novel bi-directional interrelated model for joint&#10;intent detection and slot filling. We introduce an SF-ID network to establish&#10;direct connections for the two tasks to help them promote each other mutually.&#10;Besides, we design an entirely new iteration mechanism inside the SF-ID network&#10;to enhance the bi-directional interrelated connections. The experimental&#10;results show that the relative improvement in the sentence-level semantic frame&#10;accuracy of our model is 3.79% and 5.42% on ATIS and Snips datasets,&#10;respectively, compared to the state-of-the-art model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.03903" label="2203.03903">
        <attvalues>
          <attvalue for="0" value="InstructionNER: A Multi-Task Instruction-Based Generative Framework for&#10;  Few-shot NER" />
          <attvalue for="1" value="  Recently, prompt-based methods have achieved significant performance in&#10;few-shot learning scenarios by bridging the gap between language model&#10;pre-training and fine-tuning for downstream tasks. However, existing prompt&#10;templates are mostly designed for sentence-level tasks and are inappropriate&#10;for sequence labeling objectives. To address the above issue, we propose a&#10;multi-task instruction-based generative framework, named InstructionNER, for&#10;low-resource named entity recognition. Specifically, we reformulate the NER&#10;task as a generation problem, which enriches source sentences with&#10;task-specific instructions and answer options, then inferences the entities and&#10;types in natural language. We further propose two auxiliary tasks, including&#10;entity extraction and entity typing, which enable the model to capture more&#10;boundary information of entities and deepen the understanding of entity type&#10;semantics, respectively. Experimental results show that our method consistently&#10;outperforms other baselines on five datasets in few-shot settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2204.04391" label="2204.04391">
        <attvalues>
          <attvalue for="0" value="MINER: Improving Out-of-Vocabulary Named Entity Recognition from an&#10;  Information Theoretic Perspective" />
          <attvalue for="1" value="  NER model has achieved promising performance on standard NER benchmarks.&#10;However, recent studies show that previous approaches may over-rely on entity&#10;mention information, resulting in poor performance on out-of-vocabulary (OOV)&#10;entity recognition. In this work, we propose MINER, a novel NER learning&#10;framework, to remedy this issue from an information-theoretic perspective. The&#10;proposed approach contains two mutual information-based training objectives: i)&#10;generalizing information maximization, which enhances representation via deep&#10;understanding of context and entity surface forms; ii) superfluous information&#10;minimization, which discourages representation from rote memorizing entity&#10;names or exploiting biased cues in data. Experiments on various settings and&#10;datasets demonstrate that it achieves better performance in predicting OOV&#10;entities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.05620" label="2109.05620">
        <attvalues>
          <attvalue for="0" value="RockNER: A Simple Method to Create Adversarial Examples for Evaluating&#10;  the Robustness of Named Entity Recognition Models" />
          <attvalue for="1" value="  To audit the robustness of named entity recognition (NER) models, we propose&#10;RockNER, a simple yet effective method to create natural adversarial examples.&#10;Specifically, at the entity level, we replace target entities with other&#10;entities of the same semantic class in Wikidata; at the context level, we use&#10;pre-trained language models (e.g., BERT) to generate word substitutions.&#10;Together, the two levels of attack produce natural adversarial examples that&#10;result in a shifted distribution from the training data on which our target&#10;models have been trained. We apply the proposed method to the OntoNotes dataset&#10;and create a new benchmark named OntoRock for evaluating the robustness of&#10;existing NER models via a systematic evaluation protocol. Our experiments and&#10;analysis reveal that even the best model has a significant performance drop,&#10;and these models seem to memorize in-domain entity patterns instead of&#10;reasoning from the context. Our work also studies the effects of a few simple&#10;data augmentation methods to improve the robustness of NER models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.06393" label="2104.06393">
        <attvalues>
          <attvalue for="0" value="Bridging the Gap Between Clean Data Training and Real-World Inference&#10;  for Spoken Language Understanding" />
          <attvalue for="1" value="  Spoken language understanding (SLU) system usually consists of various&#10;pipeline components, where each component heavily relies on the results of its&#10;upstream ones. For example, Intent detection (ID), and slot filling (SF)&#10;require its upstream automatic speech recognition (ASR) to transform the voice&#10;into text. In this case, the upstream perturbations, e.g. ASR errors,&#10;environmental noise and careless user speaking, will propagate to the ID and SF&#10;models, thus deteriorating the system performance. Therefore, the&#10;well-performing SF and ID models are expected to be noise resistant to some&#10;extent. However, existing models are trained on clean data, which causes a&#10;\textit{gap between clean data training and real-world inference.} To bridge&#10;the gap, we propose a method from the perspective of domain adaptation, by&#10;which both high- and low-quality samples are embedding into similar vector&#10;space. Meanwhile, we design a denoising generation model to reduce the impact&#10;of the low-quality samples. Experiments on the widely-used dataset, i.e. Snips,&#10;and large scale in-house dataset (10 million training examples) demonstrate&#10;that this method not only outperforms the baseline models on real-world (noisy)&#10;corpus but also enhances the robustness, that is, it produces high-quality&#10;results under a noisy environment. The source code will be released.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.11441" label="2103.11441">
        <attvalues>
          <attvalue for="0" value="TextFlint: Unified Multilingual Robustness Evaluation Toolkit for&#10;  Natural Language Processing" />
          <attvalue for="1" value="  Various robustness evaluation methodologies from different perspectives have&#10;been proposed for different natural language processing (NLP) tasks. These&#10;methods have often focused on either universal or task-specific generalization&#10;capabilities. In this work, we propose a multilingual robustness evaluation&#10;platform for NLP tasks (TextFlint) that incorporates universal text&#10;transformation, task-specific transformation, adversarial attack,&#10;subpopulation, and their combinations to provide comprehensive robustness&#10;analysis. TextFlint enables practitioners to automatically evaluate their&#10;models from all aspects or to customize their evaluations as desired with just&#10;a few lines of code. To guarantee user acceptability, all the text&#10;transformations are linguistically based, and we provide a human evaluation for&#10;each one. TextFlint generates complete analytical reports as well as targeted&#10;augmented data to address the shortcomings of the model's robustness. To&#10;validate TextFlint's utility, we performed large-scale empirical evaluations&#10;(over 67,000 evaluations) on state-of-the-art deep learning models, classic&#10;supervised methods, and real-world systems. Almost all models showed&#10;significant performance degradation, including a decline of more than 50% of&#10;BERT's prediction accuracy on tasks such as aspect-level sentiment&#10;classification, named entity recognition, and natural language inference.&#10;Therefore, we call for the robustness to be included in the model evaluation,&#10;so as to promote the healthy development of NLP technology.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.07162" label="2005.07162">
        <attvalues>
          <attvalue for="0" value="NAT: Noise-Aware Training for Robust Neural Sequence Labeling" />
          <attvalue for="1" value="  Sequence labeling systems should perform reliably not only under ideal&#10;conditions but also with corrupted inputs - as these systems often process&#10;user-generated text or follow an error-prone upstream component. To this end,&#10;we formulate the noisy sequence labeling problem, where the input may undergo&#10;an unknown noising process and propose two Noise-Aware Training (NAT)&#10;objectives that improve robustness of sequence labeling performed on perturbed&#10;input: Our data augmentation method trains a neural model using a mixture of&#10;clean and noisy samples, whereas our stability training algorithm encourages&#10;the model to create a noise-invariant latent representation. We employ a&#10;vanilla noise model at training time. For evaluation, we use both the original&#10;data and its variants perturbed with real OCR errors and misspellings.&#10;Extensive experiments on English and German named entity recognition benchmarks&#10;confirmed that NAT consistently improved robustness of popular sequence&#10;labeling models, preserving accuracy on the original input. We make our code&#10;and data publicly available for the research community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.15262" label="2012.15262">
        <attvalues>
          <attvalue for="0" value="Robustness Testing of Language Understanding in Task-Oriented Dialog" />
          <attvalue for="1" value="  Most language understanding models in task-oriented dialog systems are&#10;trained on a small amount of annotated training data, and evaluated in a small&#10;set from the same distribution. However, these models can lead to system&#10;failure or undesirable output when being exposed to natural language&#10;perturbation or variation in practice. In this paper, we conduct comprehensive&#10;evaluation and analysis with respect to the robustness of natural language&#10;understanding models, and introduce three important aspects related to language&#10;understanding in real-world dialog systems, namely, language variety, speech&#10;characteristics, and noise perturbation. We propose a model-agnostic toolkit&#10;LAUG to approximate natural language perturbations for testing the robustness&#10;issues in task-oriented dialog. Four data augmentation approaches covering the&#10;three aspects are assembled in LAUG, which reveals critical robustness issues&#10;in state-of-the-art models. The augmented dataset through LAUG can be used to&#10;facilitate future research on the robustness testing of language understanding&#10;in task-oriented dialog.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.15483" label="2209.15483">
        <attvalues>
          <attvalue for="0" value="Augmentation Invariant Discrete Representation for Generative Spoken&#10;  Language Modeling" />
          <attvalue for="1" value="  Generative Spoken Language Modeling research focuses on optimizing speech&#10;Language Models (LMs) using raw audio recordings without accessing any textual&#10;supervision. Such speech LMs usually operate over discrete units obtained from&#10;quantizing internal representations of self-supervised models. Although such&#10;units show impressive modeling results, their robustness capabilities have not&#10;been extensively investigated. This work focuses on improving the robustness of&#10;discrete input representations for generative spoken language modeling. First,&#10;we formally define how to measure the robustness of such representations to&#10;various signal variations that do not alter the spoken information (e.g.,&#10;time-stretch). Next, we empirically demonstrate how current state-of-the-art&#10;representation models lack robustness to such variations. To overcome this, we&#10;propose an effective and efficient method to learn robust discrete speech&#10;representation for generative spoken language modeling. The proposed approach&#10;is based on applying a set of signal transformations to the speech signal and&#10;optimizing the model using an iterative pseudo-labeling scheme. Our method&#10;significantly improves over the evaluated baselines when considering encoding&#10;and modeling metrics. We additionally evaluate our method on the&#10;speech-to-speech translation task, considering Spanish-English and&#10;French-English translations, and show the proposed approach outperforms the&#10;evaluated baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;The general GSLM pipeline is comprised of three main modules: (i) Speech-to-unit, (ii) Unit language model, and (iii) Unit-to-speech, where each of these modules is trained separately. Speech resynthesis can be achieved while ignoring the language model and directly feeding the quantized units into the unit-to-speech module~\cite{polyak2021speech} (See Figure~\ref{fig:gslm} for a visual description). In the following paragraphs, we give detailed background for each of the three components mentioned above, including the standard evaluation methods. &#10;&#10;Speech-to-unit module encodes the raw speech signal into a discrete representation. The common approach is first to encode the speech into a continuous representation and then quantize the representation to achieve a sequence of discrete units~\cite{on_generative, polyak2021speech, popuri2022enhanced, lee2021direct, kharitonov2021text, kreuk2021textless, kharitonov2022textless, nguyen2022generative, borsos2022audiolm, tjandra2019vqvae, tjandra2020transformer}. &#10;&#10;Formally, denote the domain of audio samples by $\mathcal{X} \subset \mathbb{R}$. The representation for a raw signal is therefore a sequence of samples $x = (x_1,\ldots, x_T)$, where $x_t\in\mathcal{X}$ for all $1\leq t \leq T$. &#10;&#10;Consider an encoder network, $f$, that gets as input the speech utterance and outputs a sequence of spectral representations sampled at a low frequency as follows $f(x) = (v_1, \dots, v_{T'})$. Note that we do not assume anything about the structure of the encoder network $f$. \cite{on_generative}, evaluated several speech encoders, namely, Mel-spectrogram, Contrastive Predictive Coding~\cite[CPC]{oord2018representation}, wav2vec2~\cite{baevski2020wav2vec}, and HuBERT~\cite{hubert}. &#10;&#10;Since the representations learned by such models are usually continuous, a k-means algorithm is applied over the models' outputs to generate discrete units, denoted as $z = (z_1,\ldots,z_{T'})$. Each element $z_i$ in $z$ is a positive integer, $z_i\in\{1,..,K\}$ for $1\le i \le T'$, where $K$ is the number of discrete units. We denote the quantization model with $E$.&#10;&#10;Unit Language Model is trained on the extracted discrete units, $z$. Such a language model learns a probability distribution of the learned unit sequences, which enables direct modeling of speech data without textual supervision. &#10;&#10;The language model can be used to generate speech conditionally or unconditionally, replicating what toddlers achieve before learning to read. Moreover, such a modeling framework allows for capturing and modeling prosodic features~\cite{kharitonov2021text}, as well as speaker identity~\cite{borsos2022audiolm}, or even natural dialogues~\cite{nguyen2022generative}. This is in contrast to using textual features, as they do not encode such information.&#10;&#10;Unit-to-speech module converts the speech discrete units to a raw waveform. \cite{on_generative} used a Tacotron2.0~\cite{tac} based model followed by WaveGlow~\cite{prenger2019waveglow} vocoder. Later, \cite{polyak2021speech} proposed a unit-based vocoder based on the HiFi-GAN architecture to convert units to speech directly. Such a paradigm seems to provide high-quality generations with better efficiency as it uses only one model rather than two. \cite{kreuk2021textless} and \cite{lee2021direct} additionally improved the unit based vocoder to include emotional tokens for speech emotion conversion tasks, and duration modeling for direct speech-to-speech translation. &#10;&#10;Zero-shot Evaluation. Evaluating such a complex pipeline comprised of several components is a challenging task. \cite{on_generative} proposed a set of zero-shot evaluation tasks aiming for each of the modules. Overall the proposed tasks can be divided into four main groups: (i) acoustic encoding using ABX, bitrat, (ii) language encoding using sWUGGY, sBLIMP~\cite{nguyen2020zero, on_generative}, (iii) resynthesis using Phoneme/Word Error Rate; (iv) speech generation using VERT~\cite{on_generative}, Meaningfulness Mean Opinion Score.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Signal Processing, Speech Language Modeling, Generative Spoken Language, Robust Representation Learning" />
        </attvalues>
      </node>
      <node id="2209.08359" label="2209.08359">
        <attvalues>
          <attvalue for="0" value="From Disfluency Detection to Intent Detection and Slot Filling" />
          <attvalue for="1" value="  We present the first empirical study investigating the influence of&#10;disfluency detection on downstream tasks of intent detection and slot filling.&#10;We perform this study for Vietnamese -- a low-resource language that has no&#10;previous study as well as no public dataset available for disfluency detection.&#10;First, we extend the fluent Vietnamese intent detection and slot filling&#10;dataset PhoATIS by manually adding contextual disfluencies and annotating them.&#10;Then, we conduct experiments using strong baselines for disfluency detection&#10;and joint intent detection and slot filling, which are based on pre-trained&#10;language models. We find that: (i) disfluencies produce negative effects on the&#10;performances of the downstream intent detection and slot filling tasks, and&#10;(ii) in the disfluency context, the pre-trained multilingual language model&#10;XLM-R helps produce better intent detection and slot filling performances than&#10;the pre-trained monolingual language model PhoBERT, and this is opposite to&#10;what generally found in the fluency context.&#10;" />
          <attvalue for="2" value="&#10;&#10;In natural conversations, humans sometimes inevitably produce interruptions in their speech, which is formally referred to as disfluency \cite{godfrey1993switchboard, shriberg1994preliminaries}. Its characteristic that breaks an utterance's semantic and syntax structures might make negative effects on the performances of downstream spoken language understanding (SLU) tasks as SLU models are primarily trained on curated and cleaned input without disfluencies. Thus, disfluency detection that detects&#10;(and then removes) disfluencies to produce fluent versions of disfluent inputs is crucial in real-world applications. Most previous works study the disfluency detection task isolatedly \cite{hough15_interspeech, zayats16_interspeech,jamshid-lou-etal-2018-disfluency,wang-etal-2017-transition,bach2019noisy} and evaluate the task using gold disfluency annotations \cite{godfrey1993switchboard}, while investigation of this task's influence on downstream tasks is relatively limited. In particular, downstream tasks explored with disfluency contexts include punctuation restoration \cite{wang-etal-2014-combining,Lin2020}, machine translation \cite{salesky-etal-2019-fluent,wangetal}, syntactic parsing \cite{yoshikawa-etal-2016-joint, honnibal-johnson-2014-joint, rasooli-tetreault-2013-joint,jamshid-lou-johnson-2020-improving} and question answering \cite{gupta-etal-2021-disfl}. Given the increasing popularity of task-oriented dialogue systems, it is naturally reasonable to ask a question on how disfluencies affect two important downstream SLU tasks of intent detection and slot filling.&#10;&#10;To the best of our knowledge, no study has investigated the effect of disfluencies on the intent detection and slot filling tasks. The main reason is that there is no available dataset containing linguistic annotations over both disfluencies, intents, and the slots of the intents; and creating such a dataset is required to answer the question above. Inspired by Gupta et al. \cite{gupta-etal-2021-disfl} who present a disfluent derivative of the question answering dataset SQUAD \cite{rajpurkar-etal-2016-squad}, a possible strategy to create a disfluent intent detection and slot filling dataset is to manually add contextual disfluencies into an existing fluent intent detection and slot filling dataset. This process could be performed for English with many publicly available intent detection and slot filling datasets \cite{price-1990-evaluation,coucke2018snips}. However, from a societal, linguistic, machine learning, cognitive, cultural, and normative perspective \cite{donlpotherlanguages}, it is also worth studying the proposed question for languages other than English, e.g. Vietnamese. Despite being the 17th most spoken language in the world \cite{Ethnologue} with about 100M speakers, Vietnamese is a low-resource language w.r.t. SLU tasks, e.g. having no previous study as well as no public dataset available for disfluency detection. &#10;&#10;In this paper, we present the first study that investigates the influence of disfluency detection on the downstream intent detection and slot filling tasks. We perform this study for Vietnamese---a low-resource language in these SLU research topics. First, we create a dataset with disfluency annotations by manually adding contextual disfluencies as distractors into the fluent dataset PhoATIS \cite{dao21_interspeech} which is the only current dataset publicly available for Vietnamese intent detection and slot filling. Then, we formulate our empirical approach as a ``Cascaded'' one combining a disfluency detection model and a joint intent detection and slot filling model. We conduct experiments using strong baseline models that are based on pre-trained language models {XLM-R} \cite{conneau2019unsupervised} and {PhoBERT} \cite{nguyen2020phobert}. Experimental results show that: (i) disfluencies negatively affect the performances of the downstream intent detection and slot filling tasks, and (ii) in the disfluency context, the pre-trained multilingual language model XLM-R is more effective for the intent detection and slot filling tasks than the pre-trained monolingual language model PhoBERT, and this is completely opposite to what is generally found in the fluency context with other Vietnamese NLP tasks \cite{nguyen2020phobert,vitext2sql,PhoNER_COVID19}. &#10;&#10;We publicly release our dataset with disfluency annotations to facilitate future Vietnamese SLU research and applications. Our dataset is available at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Disfluency Detection, Artificial Intelligence, Natural Language Processing, Language Modeling, Intent Detection" />
        </attvalues>
      </node>
      <node id="1906.00556" label="1906.00556">
        <attvalues>
          <attvalue for="0" value="Fluent Translations from Disfluent Speech in End-to-End Speech&#10;  Translation" />
          <attvalue for="1" value="  Spoken language translation applications for speech suffer due to&#10;conversational speech phenomena, particularly the presence of disfluencies.&#10;With the rise of end-to-end speech translation models, processing steps such as&#10;disfluency removal that were previously an intermediate step between speech&#10;recognition and machine translation need to be incorporated into model&#10;architectures. We use a sequence-to-sequence model to translate from noisy,&#10;disfluent speech to fluent text with disfluencies removed using the recently&#10;collected `copy-edited' references for the Fisher Spanish-English dataset. We&#10;are able to directly generate fluent translations and introduce considerations&#10;about how to evaluate success on this task. This work provides a baseline for a&#10;new task, the translation of conversational speech with joint removal of&#10;disfluencies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.04016" label="2106.04016">
        <attvalues>
          <attvalue for="0" value="Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question&#10;  Answering" />
          <attvalue for="1" value="  Disfluencies is an under-studied topic in NLP, even though it is ubiquitous&#10;in human conversation. This is largely due to the lack of datasets containing&#10;disfluencies. In this paper, we present a new challenge question answering&#10;dataset, Disfl-QA, a derivative of SQuAD, where humans introduce contextual&#10;disfluencies in previously fluent questions. Disfl-QA contains a variety of&#10;challenging disfluencies that require a more comprehensive understanding of the&#10;text than what was necessary in prior datasets. Experiments show that the&#10;performance of existing state-of-the-art question answering models degrades&#10;significantly when tested on Disfl-QA in a zero-shot setting.We show data&#10;augmentation methods partially recover the loss in performance and also&#10;demonstrate the efficacy of using gold data for fine-tuning. We argue that we&#10;need large-scale disfluency datasets in order for NLP models to be robust to&#10;them. The dataset is publicly available at:&#10;https://github.com/google-research-datasets/disfl-qa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2104.02021" label="2104.02021">
        <attvalues>
          <attvalue for="0" value="Intent Detection and Slot Filling for Vietnamese" />
          <attvalue for="1" value="  Intent detection and slot filling are important tasks in spoken and natural&#10;language understanding. However, Vietnamese is a low-resource language in these&#10;research topics. In this paper, we present the first public intent detection&#10;and slot filling dataset for Vietnamese. In addition, we also propose a joint&#10;model for intent detection and slot filling, that extends the recent&#10;state-of-the-art JointBERT+CRF model with an intent-slot attention layer to&#10;explicitly incorporate intent context information into slot filling via &quot;soft&quot;&#10;intent label embedding. Experimental results on our Vietnamese dataset show&#10;that our proposed model significantly outperforms JointBERT+CRF. We publicly&#10;release our dataset and the implementation of our model at:&#10;https://github.com/VinAIResearch/JointIDSF&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.00744" label="2003.00744">
        <attvalues>
          <attvalue for="0" value="PhoBERT: Pre-trained language models for Vietnamese" />
          <attvalue for="1" value="  We present PhoBERT with two versions, PhoBERT-base and PhoBERT-large, the&#10;first public large-scale monolingual language models pre-trained for&#10;Vietnamese. Experimental results show that PhoBERT consistently outperforms the&#10;recent best pre-trained multilingual model XLM-R (Conneau et al., 2020) and&#10;improves the state-of-the-art in multiple Vietnamese-specific NLP tasks&#10;including Part-of-speech tagging, Dependency parsing, Named-entity recognition&#10;and Natural language inference. We release PhoBERT to facilitate future&#10;research and downstream applications for Vietnamese NLP. Our PhoBERT models are&#10;available at https://github.com/VinAIResearch/PhoBERT&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.01891" label="2010.01891">
        <attvalues>
          <attvalue for="0" value="A Pilot Study of Text-to-SQL Semantic Parsing for Vietnamese" />
          <attvalue for="1" value="  Semantic parsing is an important NLP task. However, Vietnamese is a&#10;low-resource language in this research area. In this paper, we present the&#10;first public large-scale Text-to-SQL semantic parsing dataset for Vietnamese.&#10;We extend and evaluate two strong semantic parsing baselines EditSQL (Zhang et&#10;al., 2019) and IRNet (Guo et al., 2019) on our dataset. We compare the two&#10;baselines with key configurations and find that: automatic Vietnamese word&#10;segmentation improves the parsing results of both baselines; the normalized&#10;pointwise mutual information (NPMI) score (Bouma, 2009) is useful for schema&#10;linking; latent syntactic features extracted from a neural dependency parser&#10;for Vietnamese also improve the results; and the monolingual language model&#10;PhoBERT for Vietnamese (Nguyen and Nguyen, 2020) helps produce higher&#10;performances than the recent best multilingual language model XLM-R (Conneau et&#10;al., 2020).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.03879" label="2104.03879">
        <attvalues>
          <attvalue for="0" value="COVID-19 Named Entity Recognition for Vietnamese" />
          <attvalue for="1" value="  The current COVID-19 pandemic has lead to the creation of many corpora that&#10;facilitate NLP research and downstream applications to help fight the pandemic.&#10;However, most of these corpora are exclusively for English. As the pandemic is&#10;a global problem, it is worth creating COVID-19 related datasets for languages&#10;other than English. In this paper, we present the first manually-annotated&#10;COVID-19 domain-specific dataset for Vietnamese. Particularly, our dataset is&#10;annotated for the named entity recognition (NER) task with newly-defined entity&#10;types that can be used in other future epidemics. Our dataset also contains the&#10;largest number of entities compared to existing Vietnamese NER datasets. We&#10;empirically conduct experiments using strong baselines on our dataset, and find&#10;that: automatic Vietnamese word segmentation helps improve the NER results and&#10;the highest performances are obtained by fine-tuning pre-trained language&#10;models where the monolingual model PhoBERT for Vietnamese (Nguyen and Nguyen,&#10;2020) produces higher results than the multilingual model XLM-R (Conneau et&#10;al., 2020). We publicly release our dataset at:&#10;https://github.com/VinAIResearch/PhoNER_COVID19&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.12254" label="2408.12254">
        <attvalues>
          <attvalue for="0" value="A Language-agnostic Model of Child Language Acquisition" />
          <attvalue for="1" value="  This work reimplements a recent semantic bootstrapping child-language&#10;acquisition model, which was originally designed for English, and trains it to&#10;learn a new language: Hebrew. The model learns from pairs of utterances and&#10;logical forms as meaning representations, and acquires both syntax and word&#10;meanings simultaneously. The results show that the model mostly transfers to&#10;Hebrew, but that a number of factors, including the richer morphology in&#10;Hebrew, makes the learning slower and less robust. This suggests that a clear&#10;direction for future work is to enable the model to leverage the similarities&#10;between different word forms.&#10;" />
          <attvalue for="2" value="&#10;This paper concerns computational models of cla, which seek to understand the process of language acquisition by programming a computer to emulate the learning undergone by the child. When presented with data from a given language, such an algorithm should learn a degree of proficiency in that language. The fact that any child, when exposed to appropriate data, can learn any language establishes a strong connection between acquisition and language variation: whatever varies between languages must be specified by the data and must be learnable. It also makes it an essential requirement of a convincing cla model that it be capable of learning any language. Here, we reimplement a recent computational cla model \cite{abend2017bootstrapping}, which is based on combinatory categorial grammar \cite{steedman2001syntactic} and semantic bootstrapping \cite{pinker1979formal}, and is trained with an expectation-maximization style algorithm. This model is a suitable choice for understanding the acquisition process because of its cognitive plausibility. The dominant paradigm of large language models requires too much training data to be plausible models of how humans acquire language. Even on the small end of the scale they generally train on several orders of magnitude more tokens than a human sees in their entire life. Some have sought to better approximate human learning by learning from a more modest 10-100M tokens \cite{conll-2023-babylm}. However, such models still generally make a number of implausible design choices, such as multi-epoch training, batched parameter updates, and arbitrary text tokenization, and they do not, as we do, ensure that the training examples are presented in the order they appear to the child. Abend et al. (2017) in contrast is grounded in a well-developed theoretical model of semantic bootstrapping, and trains on each example only once, individually, in the order they appear to the child.&#10;&#10;We test this model on two languages: English, on which it was originally tested, and Hebrew. The data we use is comprised of real child-directed utterances, taken from the CHILDES corpus \cite{macwhinney1998childes}, coupled with a recent method for converting universal dependency annotations to logical forms \cite{ida2023}.&#10;&#10;Firstly, we replicate the findings of \cite{abend2017bootstrapping}, and show that this model is successful in learning the important features of English syntax and semantics. We focus in the present paper on word order learning, and learning the meaning and syntactic categories of individual words. The results show that, after training, the model, correctly, strongly favours SVO order, predicts the right semantics for commonly appearing words and the right syntactic category for most. Then we apply the same training and testing procedure to the Hebrew corpus. There, the model learns word order and word meaning with a reasonably high accuracy. Its accuracy on syntactic categories is somewhat lower than that on English. We then discuss the difference in acquisition performance with respect to the linguistic differences between the two languages, and outline future extensions to the model that can more completely handle the learning of Hebrew without compromising the learning of English. Together these results demonstrate the model in question is broadly successful in transferring between multiple languages, and support the argument that, in general for computational cla models, it is important and instructive to evaluate on more than just a single language. The code for training and evaluation will be released on publication.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Semantic Bootstrapping, Computer Science, Linguistics, Cognitive Science, Multilingual Modeling, Computational Linguistics, Artificial Intelligence, Language Acquisition" />
        </attvalues>
      </node>
      <node id="2109.10952" label="2109.10952">
        <attvalues>
          <attvalue for="0" value="Cross-linguistically Consistent Semantic and Syntactic Annotation of&#10;  Child-directed Speech" />
          <attvalue for="1" value="  This paper proposes a methodology for constructing such corpora of child&#10;directed speech (CDS) paired with sentential logical forms, and uses this&#10;method to create two such corpora, in English and Hebrew. The approach enforces&#10;a cross-linguistically consistent representation, building on recent advances&#10;in dependency representation and semantic parsing. Specifically, the approach&#10;involves two steps. First, we annotate the corpora using the Universal&#10;Dependencies (UD) scheme for syntactic annotation, which has been developed to&#10;apply consistently to a wide variety of domains and typologically diverse&#10;languages. Next, we further annotate these data by applying an automatic method&#10;for transducing sentential logical forms (LFs) from UD structures. The UD and&#10;LF representations have complementary strengths: UD structures are&#10;language-neutral and support consistent and reliable annotation by multiple&#10;annotators, whereas LFs are neutral as to their syntactic derivation and&#10;transparently encode semantic relations.&#10;  Using this approach, we provide syntactic and semantic annotation for two&#10;corpora from CHILDES: Brown's Adam corpus (English; we annotate ~80% of its&#10;child-directed utterances), all child-directed utterances from Berman's Hagar&#10;corpus (Hebrew). We verify the quality of the UD annotation using an&#10;inter-annotator agreement study, and manually evaluate the transduced meaning&#10;representations. We then demonstrate the utility of the compiled corpora&#10;through (1) a longitudinal corpus study of the prevalence of different&#10;syntactic and semantic phenomena in the CDS, and (2) applying an existing&#10;computational model of language acquisition to the two corpora and briefly&#10;comparing the results across languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1701.03185" label="1701.03185">
        <attvalues>
          <attvalue for="0" value="Generating High-Quality and Informative Conversation Responses with&#10;  Sequence-to-Sequence Models" />
          <attvalue for="1" value="  Sequence-to-sequence models have been applied to the conversation response&#10;generation problem where the source sequence is the conversation history and&#10;the target sequence is the response. Unlike translation, conversation&#10;responding is inherently creative. The generation of long, informative,&#10;coherent, and diverse responses remains a hard task. In this work, we focus on&#10;the single turn setting. We add self-attention to the decoder to maintain&#10;coherence in longer responses, and we propose a practical approach, called the&#10;glimpse-model, for scaling to large datasets. We introduce a stochastic&#10;beam-search algorithm with segment-by-segment reranking which lets us inject&#10;diversity earlier in the generation process. We trained on a combined data set&#10;of over 2.3B conversation messages mined from the web. In human evaluation&#10;studies, our method produces longer responses overall, with a higher proportion&#10;rated as acceptable and excellent as length increases, compared to baseline&#10;sequence-to-sequence models with explicit length-promotion. A back-off strategy&#10;produces better responses overall, in the full spectrum of lengths.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.09217" label="1905.09217">
        <attvalues>
          <attvalue for="0" value="Deeper Text Understanding for IR with Contextual Neural Language&#10;  Modeling" />
          <attvalue for="1" value="  Neural networks provide new possibilities to automatically learn complex&#10;language patterns and query-document relations. Neural IR models have achieved&#10;promising results in learning query-document relevance patterns, but few&#10;explorations have been done on understanding the text content of a query or a&#10;document. This paper studies leveraging a recently-proposed contextual neural&#10;language model, BERT, to provide deeper text understanding for IR. Experimental&#10;results demonstrate that the contextual text representations from BERT are more&#10;effective than traditional word embeddings. Compared to bag-of-words retrieval&#10;models, the contextual language model can better leverage language structures,&#10;bringing large improvements on queries written in natural languages. Combining&#10;the text understanding ability with search knowledge leads to an enhanced&#10;pre-trained BERT model that can benefit related search tasks where training&#10;data are limited.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.08582" label="2204.08582">
        <attvalues>
          <attvalue for="0" value="MASSIVE: A 1M-Example Multilingual Natural Language Understanding&#10;  Dataset with 51 Typologically-Diverse Languages" />
          <attvalue for="1" value="  We present the MASSIVE dataset--Multilingual Amazon Slu resource package&#10;(SLURP) for Slot-filling, Intent classification, and Virtual assistant&#10;Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant&#10;utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE&#10;was created by tasking professional translators to localize the English-only&#10;SLURP dataset into 50 typologically diverse languages from 29 genera. We also&#10;present modeling results on XLM-R and mT5, including exact match accuracy,&#10;intent classification accuracy, and slot-filling F1 score. We have released our&#10;dataset, modeling code, and models publicly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;Prior researchers have emphasized the need to explore the unique challenges of low-resource languages \cite{simpson2008human, strassel-tracey-2016-lorelei, cruz2020establishing, lakew2020low, marivate-etal-2020-investigating, magueresse2020lowresource, goyal2021flores101}, while the growing number and size of language models (mBERT \cite{devlinmBERT}, RoBERTa \cite{liu2019roberta}, XLM \cite{lample2019crosslingual}, XLM-R \cite{conneau-etal-2020-unsupervised}, mBART \cite{liu2020multilingual}, MARGE \cite{lewis2020pretraining}, and mT5 \cite{xue-etal-2021-mt5} pre-trained on massively multilingual corpora have allowed for significant improvements in supporting them.&#10;However, the creation of evaluation datasets for specific tasks has not kept pace.&#10;Some tasks, such as Named Entity Recognition (NER) or translation, lend themselves to mining existing corpora \cite{TIEDEMANN12.463,Pan2017CrosslingualNT,hu2020xtreme}, while others such as NLU, the focus here, require the creation of new data and schema-specific annotations.&#10;Beyond the cost, even identifying a sufficient number of speakers for data generation and quality control can be difficult.&#10;Most studies have thus focused on collecting data for one such low-resource language and determining the utility of multilingual models or cross-lingual learning from more readily available languages.&#10;Moreover, such datasets are often isolated collections, creating an environment of multiple datasets not easily comparable across the different languages or tasks.&#10;There have been exceptions, such as SQuAD \cite{rajpurkar2016squad} and XQuAd \cite{Artetxe:etal:2019}, ATIS \cite{price-1990-evaluation}, its Hindi and Turkish extension \cite{8461905}, and MultiATIS++ \cite{xu-etal-2020-end}, and Snips \cite{coucke2018snips} with its addition of French \cite{saade2019spoken}, where researchers have extended popular English benchmark datasets to new languages.&#10;This work focuses on the general multi-domain NLU task and builds off the SLURP \cite{bastianelli-etal-2020-slurp} benchmark dataset to extend to an unprecedented 50 new languages.&#10;&#10;For the task of NLU, the ATIS dataset has been popular in the NLP community since its first release.&#10;MultiATIS++ was one of the first efforts to extend an NLU dataset across a significant number of languages (nine), yet remained in the limited domain of airline bookings.&#10;While proving an asset, it has been questioned what is left to learn from such a dataset \cite{tur2010left}.&#10;Facebook released a general Intelligent Virtual Assistant (IVA) dataset across the domains of Alarm, Reminder, and Weather \cite{schuster-etal-2019-cross-lingual} created for the purpose of demonstrating cross-lingual transfer learning; and so did not need to be parallel or have an equal number of datapoints, resulting in far fewer examples in Thai (5k) compared to Spanish (7.6k) and English (43k).&#10;The Snips datasets (both the original English only and the English and French releases) are most similar to the NLU contained in the massive~dataset, spanning smart home and music domains for a generic voice-based virtual assistant.&#10;&#10;The first iteration for the foundation of the massive~dataset was the NLU Evaluation Benchmarking Dataset, with 25k utterances across 18 domains \cite{liu2019benchmarking}.&#10;The authors updated the dataset and added audio and ASR transcriptions in the release of the Spoken Language Understanding Resource Package (SLURP) \cite{bastianelli-etal-2020-slurp}, allowing for full end-to-end Spoken Language Understanding (SLU) evaluation similar to the Fluent Speech Commands dataset \cite{lugosch2019speech} and Chinese Audio-Textual Spoken Language Understanding (CATSLU) \cite{10.1145/3340555.3356098}.&#10;An overview of selected existing NLU datasets can be seen in Table~\ref{table:NLUDatasets}.&#10;&#10;We release the massive~dataset along with baselines from large pre-trained models fine-tuned on the NLU slot and intent prediction tasks.&#10;Early cross-lingual and multilingual NLU modeling approaches used projection or alignment methods \cite{yarowsky-etal-2001-inducing}, focusing on string matching, edit distance, or consonant signatures \cite{ehrmann-etal-2011-building}, lookup lexicons for low-resource languages \cite{mayhew-etal-2017-cheap}, and aligning \cite{xie-etal-2018-neural} or jointly training word embeddings \cite{singla-etal-2018-multi}.&#10;More recently, researchers have borrowed encoders from pre-trained neural translation models before building subsequent classifiers and NER models \cite{eriguchi2018zeroshot, schuster-etal-2019-cross-lingual}, also focusing on language-agnostic and language specific features to learn what information to share between languages \cite{chen-etal-2019-multi-source}.&#10;Generative parsing has been demonstrated using sequence-to-sequence models and pointer networks \cite{dontparsegenerate}.&#10;With the rise of BERT and large pre-trained language models, we have also seen impressive demonstrations of zero-shot performance, where subword tokenization WordPiece overlap helps but is not even necessary to realize improvements \cite{pires-etal-2019-multilingual, k2020crosslingual}, as well as production multilingual NLU improvements with distillation and full fine-tuning \cite{FitzGerald2022AlexaTM}.&#10;The translation task has then been incoporated in the pre-training \cite{wang-etal-2021-exploring-cross} of these models or even as part of the final NLU hypothesis for streamlined multilingual production systems \cite{fitzgerald2020stil}.&#10;Researchers have propped up training data by translating and projecting labels into the target language \cite{xu-etal-2020-end} and discovered more sophisticated approaches to alignment such as translate and fill using mT5 to train the filler \cite{nicosia-etal-2021-translate-fill}.&#10;Recent work has even delved into the application of these techniques to lower-resource languages such as Persian. For example, ParsiNLU explores a variety of NLU tasks for Parsi, fine-tuning mT5 of various sizes \cite{khashabi2021parsinlu}. Similarly these techniques have also been used, even a bit earlier, for text summarization \cite{Farahanipersian2021}.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Multilingual Dataset, Virtual Assistant Evaluation, Artificial Intelligence, Natural Language Processing, Natural Language Modeling" />
        </attvalues>
      </node>
      <node id="2203.00902" label="2203.00902">
        <attvalues>
          <attvalue for="0" value="Do Prompts Solve NLP Tasks Using Natural Language?" />
          <attvalue for="1" value="  Thanks to the advanced improvement of large pre-trained language models,&#10;prompt-based fine-tuning is shown to be effective on a variety of downstream&#10;tasks. Though many prompting methods have been investigated, it remains unknown&#10;which type of prompts are the most effective among three types of prompts&#10;(i.e., human-designed prompts, schema prompts and null prompts). In this work,&#10;we empirically compare the three types of prompts under both few-shot and&#10;fully-supervised settings. Our experimental results show that schema prompts&#10;are the most effective in general. Besides, the performance gaps tend to&#10;diminish when the scale of training data grows large.&#10;" />
          <attvalue for="2" value="&#10;&#10;Prompt-based fine-tuning has gained increasing attention on NLP \cite{shin-etal-2020-autoprompt,schick-etal-2020-automatically,schick-schutze-2021-just,tanl,gao2021making}.&#10;The main idea is to leverage knowledge in pre-trained language models for downstream tasks, by reformulating a specific task into the form of language modeling tasks, with the aid of prompts.&#10;Among various recent methods on prompt-based NLP, there has been three major forms of prompts, which we call {NL template prompts}, {schema prompts} and {null prompts}, respectively.&#10;NL template prompts \cite{petroni-etal-2019-language,jiang-etal-2020-know,gao2021making} were the earliest proposed and the dominant method.&#10;As illustrated in Table \ref{tab:intro_example}, they use a natural language sentence to augment a given input, where the added prompt contains a mask token that indicates the output class.&#10;In contrast, schema prompts \cite{lee2021dialogue,tanl} replace a natural language sentence with a structured schema, which makes the prompt more succinct and code-like.&#10;Null prompts \cite{logan2021cutting} are the most succinct version, directly adding a masked token to the end of the input.&#10;&#10;While different types of prompts have been compared for specific tasks \cite{gao2021making,logan2021cutting}, there has been little work systematically comparing their effects over a large variety of tasks and training settings (i.e., few shot).&#10;We aim to fill the gap by empirically addressing the following three research questions:&#10;&#10;First, which type of prompt is generally the most effective?&#10;Intuitively, natural language prompts better connect large pre-training and task fine-tuning by having the same language style in both phases.&#10;However, it can increase the difficulty of representation by introducing overly long sequence extensions.&#10;In contrast, schema and null prompts are more succinct, but less close to natural language pre-training. &#10;&#10;Second, are task-specific information useful to include in prompts.&#10;Compared with NL templates and schemas, null prompts are the most succinct, and are task-agnostic in not including any task hints in the augmented sequence.&#10;While having been shown effective for several NLI-style classification tasks under few-shot settings \cite{logan2021cutting}, it remains a question whether they are competitive in more general settings.&#10;&#10;Third, what is the effect of automatically searching for prompt template and masked label words?&#10;There has been a line of work automatically finding prompts, which results in seemingly unnatural augmented sequences \cite{shin-etal-2020-autoprompt,gao2021making}.&#10;In addition, the words to use for filling the masked output slots are also flexible.&#10;We want to learn whether these automatic selections have significant benefit compared with human definitions.&#10;&#10;Results show that among the three types of prompts, schema prompts are the most effective in general.&#10;However, the gap between the three types of prompts tends to diminish when the scale of training data grows sufficiently large.&#10;Finally, both automatic templates and automatic tokens give better results compared with more understandable human prompts. &#10;Our code will be released at &#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Supervised Learning Strategies, Computer Science, Linguistics, Cognitive Science, Prompting Methods Comparison, Artificial Intelligence, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2106.13353" label="2106.13353">
        <attvalues>
          <attvalue for="0" value="Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with&#10;  Language Models" />
          <attvalue for="1" value="  Prompting language models (LMs) with training examples and task descriptions&#10;has been seen as critical to recent successes in few-shot learning. In this&#10;work, we show that finetuning LMs in the few-shot setting can considerably&#10;reduce the need for prompt engineering. In fact, one can use null prompts,&#10;prompts that contain neither task-specific templates nor training examples, and&#10;achieve competitive accuracy to manually-tuned prompts across a wide range of&#10;tasks. While finetuning LMs does introduce new parameters for each downstream&#10;task, we show that this memory overhead can be substantially reduced:&#10;finetuning only the bias terms can achieve comparable or better accuracy than&#10;standard finetuning while only updating 0.1% of the parameters. All in all, we&#10;recommend finetuning LMs for few-shot learning as it is more accurate, robust&#10;to different prompts, and can be made nearly as efficient as using frozen LMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.04036" label="2003.04036">
        <attvalues>
          <attvalue for="0" value="Sentence Analogies: Exploring Linguistic Relationships and Regularities&#10;  in Sentence Embeddings" />
          <attvalue for="1" value="  While important properties of word vector representations have been studied&#10;extensively, far less is known about the properties of sentence vector&#10;representations. Word vectors are often evaluated by assessing to what degree&#10;they exhibit regularities with regard to relationships of the sort considered&#10;in word analogies. In this paper, we investigate to what extent commonly used&#10;sentence vector representation spaces as well reflect certain kinds of&#10;regularities. We propose a number of schemes to induce evaluation data, based&#10;on lexical analogy data as well as semantic relationships between sentences.&#10;Our experiments consider a wide range of sentence embedding methods, including&#10;ones based on BERT-style contextual embeddings. We find that different models&#10;differ substantially in their ability to reflect such regularities.&#10;" />
          <attvalue for="2" value="&#10;Sentence embeddings are dense vectors that reflect salient semantic properties of a sentence. Similar to how commonly used word embedding methods such as word2vec \cite{mikolov2013efficient} capture semantic relationships between words, sentence embeddings are expected to capture semantic relationships between sentences. &#10;A number of different sentence embedding methods have been proposed (cf.\ Section \ref{sec:embedding-methods} for an overview). More recently, pretrained language models such as ELMo \cite{peters2018deep}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019XLNet}, and RoBERTa \cite{liu2019roberta} have become the method of choice when encoding text. &#10;Thus, such models are also often invoked to represent sentences by means of individual embeddings.\ &#10;&#10;While important properties of word vector representations have been studied extensively, far less is known about the properties of sentence vector representations. A particularly prominent aspect of word vector representations induced by methods such as word2vec is that the vector space exhibits certain kinds of regularities. Many of these are of the sort considered in word analogies.&#10;Proportional analogies take the form $A$ is to $B$ as $C$ is to $D$, e.g., Paris is to France as Berlin is to Germany.&#10;\cite{TurneyLittman2005} proposed identifying such analogies using bag-of-words vector space models.&#10;\cite{mikolov-etal-2013-linguistic} showed that word2vec's word vector representations reflect certain kinds of word analogies surprisingly well.&#10;The widely used word analogy task that they proposed takes the following form.&#10;Given embeddings $\mathbf{v}_A$, $\mathbf{v}_B$, $\mathbf{v}_C$, $\mathbf{v}_D$ for words $A$, $B$, $C$, $D$ for an analogy of the above form, the task consists in identifying the correct word $D$ given $A$, $B$, and $C$. Most commonly, this is achieved by optimizing&#10;\begin{align}&#10; \argmax_{D \in V}~~~~&amp;\mathrm{sim}(\mathbf{v}_D, \mathbf{v}_B - \mathbf{v}_A + \mathbf{v}_C)\\&#10; s.t.~~~~~~&amp;D \not\in \{A,B,C\},&#10;\end{align}&#10;where $\mathrm{sim}(\mathbf{v}_1,\mathbf{v}_2)$ typically denotes cosine similarity between two vectors. &#10;This sort of analogy task is one of the most commonly invoked means of assessing the quality of word vector induction techniques.&#10;&#10;However, little is known about the topology of vector representation spaces for entire sentences. In this paper we fill this gap, considering models with a dedicated sentence embedding objective as well as BERT-style pretrained embedding models. &#10;We study whether such sentence representation spaces as well exhibit regularities with regard to certain kinds of relationships.&#10;To this end, we devise new datasets that are similar to typical word analogy datasets \cite{mikolov-etal-2013-linguistic}. These allow us to test empirically whether existing sentence embedding models reflect analogical relationships between sentences.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Contextual Embedding Methods, Artificial Intelligence, Mathematics, Sentence Vector Representations, Natural Language Processing, Lexical Analogy Evaluation" />
        </attvalues>
      </node>
      <node id="cs/0508103" label="cs/0508103">
        <attvalues>
          <attvalue for="0" value="Corpus-based Learning of Analogies and Semantic Relations" />
          <attvalue for="1" value="  We present an algorithm for learning from unlabeled text, based on the Vector&#10;Space Model (VSM) of information retrieval, that can solve verbal analogy&#10;questions of the kind found in the SAT college entrance exam. A verbal analogy&#10;has the form A:B::C:D, meaning &quot;A is to B as C is to D&quot;; for example,&#10;mason:stone::carpenter:wood. SAT analogy questions provide a word pair, A:B,&#10;and the problem is to select the most analogous word pair, C:D, from a set of&#10;five choices. The VSM algorithm correctly answers 47% of a collection of 374&#10;college-level analogy questions (random guessing would yield 20% correct; the&#10;average college-bound senior high school student answers about 57% correctly).&#10;We motivate this research by applying it to a difficult problem in natural&#10;language processing, determining semantic relations in noun-modifier pairs. The&#10;problem is to classify a noun-modifier pair, such as &quot;laser printer&quot;, according&#10;to the semantic relation between the noun (printer) and the modifier (laser).&#10;We use a supervised nearest-neighbour algorithm that assigns a class to a given&#10;noun-modifier pair by finding the most analogous noun-modifier pair in the&#10;training data. With 30 classes of semantic relations, on a collection of 600&#10;labeled noun-modifier pairs, the learning algorithm attains an F value of 26.5%&#10;(random guessing: 3.3%). With 5 classes of semantic relations, the F value is&#10;43.2% (random: 20%). The performance is state-of-the-art for both verbal&#10;analogies and noun-modifier relations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.07232" label="2406.07232">
        <attvalues>
          <attvalue for="0" value="DUAL-REFLECT: Enhancing Large Language Models for Reflective Translation&#10;  through Dual Learning Feedback Mechanisms" />
          <attvalue for="1" value="  Recently, large language models (LLMs) enhanced by self-reflection have&#10;achieved promising performance on machine translation. The key idea is guiding&#10;LLMs to generate translation with human-like feedback. However, existing&#10;self-reflection methods lack effective feedback information, limiting the&#10;translation performance. To address this, we introduce a DUAL-REFLECT&#10;framework, leveraging the dual learning of translation tasks to provide&#10;effective feedback, thereby enhancing the models' self-reflective abilities and&#10;improving translation performance. The application of this method across&#10;various translation tasks has proven its effectiveness in improving translation&#10;accuracy and eliminating ambiguities, especially in translation tasks with&#10;low-resource language pairs.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have recently demonstrated remarkable abilities across a variety of tasks ~\cite{bubeck2023sparks,xu2023large,zhao2023survey}. &#10;Notably, in the field of machine translation, LLMs have improved translation quality by adopting human-like methods of self-reflection ~\cite{shinn2023reflexion,liang2023encouraging}.&#10;The self-reflection process primarily relies on using LLMs to iteratively refine initial drafts through feedback loops, a method that has been widely researched and explored ~\cite{shinn2023reflexion,DBLP:conf/uist/ParkOCMLB23, scheurer2022training,le2022coderl,welleck2022generating, amabile1983theoretical, flower1981cognitive,chen2023equals,simon1962architecture,chen-etal-2023-improving-low,sun2021joint}. &#10;The lack of effective feedback limits the self-reflective capacity of Large Language Models (LLMs), thereby affecting their continuous improvement in translation ~\cite{tyen2023llms,liang2023encouraging,lou-etal-2023-cceval}. &#10;&#10;To address this, we introduce a framework that leverages the inherent duality property ~\cite{he2016dual,qin2020dual,sun2021tibetan,yi2017dualgan,xia2017dual} of translation tasks to provide effective feedback to LLMs, thereby enhancing their reflective capabilities and consequently improving translation performance. &#10;This method, named DUAL-REFLECT, stands for DUAL learning enhanced auto-REFLECtive Translation and comprises five stages: Draft Translation, Back Translation, Process Assessment, Dual-Reflection, Auto Revision. &#10;In the draft translation stage, LLMs employ their inherent translation capabilities to generate a draft translation. &#10;Subsequently, in the Back Translation stage, LLMs translate the draft translation back to the source language. &#10;Then, during the process assessment stage, an LLM-based agent is introduced to assess whether dual reflection is needed. If not, it outputs the final result; otherwise, the process continues to cycle through all the steps.&#10;Based on this, in the dual reflection stage, LLMs reflect on the differences between the back-translation results and the initial source input, revealing potential translation biases. LLMs further analyze the reasons for these discrepancies and propose suggestions for improvement. &#10;Finally, In the auto-revision stage, LLMs modify the initial translation by incorporating the analysis and improvement suggestions obtained through dual reflection.&#10;&#10;We verify the effectiveness of the DUAL-REFLECT framework across four translation directions in the WMT22, covering high, medium, and lower resource languages, as well as a commonsense reasoning MT Benchmark.&#10;Automatic evaluation results show that DUAL-REFLECT outperforms strong baseline methods, significantly enhancing translation performance.&#10;Notably, on low-resource translation tasks, DUAL-REFLECT achieved an average result that surpassed ChatGPT by +1.6 COMET.&#10;In addition, DUAL-REFLECT enhanced ChatGPT exceeded GPT-4 on the commonsense reasoning MT benchmark.&#10;Further human evaluation demonstrates that DUAL-REFLECT shows a better ability to resolve translation ambiguities compared to other methods.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Self-Reflection Methods, Linguistics, Cognitive Science, Language Models, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.03442" label="2304.03442">
        <attvalues>
          <attvalue for="0" value="Generative Agents: Interactive Simulacra of Human Behavior" />
          <attvalue for="1" value="  Believable proxies of human behavior can empower interactive applications&#10;ranging from immersive environments to rehearsal spaces for interpersonal&#10;communication to prototyping tools. In this paper, we introduce generative&#10;agents--computational software agents that simulate believable human behavior.&#10;Generative agents wake up, cook breakfast, and head to work; artists paint,&#10;while authors write; they form opinions, notice each other, and initiate&#10;conversations; they remember and reflect on days past as they plan the next&#10;day. To enable generative agents, we describe an architecture that extends a&#10;large language model to store a complete record of the agent's experiences&#10;using natural language, synthesize those memories over time into higher-level&#10;reflections, and retrieve them dynamically to plan behavior. We instantiate&#10;generative agents to populate an interactive sandbox environment inspired by&#10;The Sims, where end users can interact with a small town of twenty five agents&#10;using natural language. In an evaluation, these generative agents produce&#10;believable individual and emergent social behaviors: for example, starting with&#10;only a single user-specified notion that one agent wants to throw a Valentine's&#10;Day party, the agents autonomously spread invitations to the party over the&#10;next two days, make new acquaintances, ask each other out on dates to the&#10;party, and coordinate to show up for the party together at the right time. We&#10;demonstrate through ablation that the components of our agent&#10;architecture--observation, planning, and reflection--each contribute critically&#10;to the believability of agent behavior. By fusing large language models with&#10;computational, interactive agents, this work introduces architectural and&#10;interaction patterns for enabling believable simulations of human behavior.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.01780" label="2207.01780">
        <attvalues>
          <attvalue for="0" value="CodeRL: Mastering Code Generation through Pretrained Models and Deep&#10;  Reinforcement Learning" />
          <attvalue for="1" value="  Program synthesis or code generation aims to generate a program that&#10;satisfies a problem specification. Recent approaches using large-scale&#10;pretrained language models (LMs) have shown promising results, yet they have&#10;some critical limitations. In particular, they often follow a standard&#10;supervised fine-tuning procedure to train a code generation model only from the&#10;pairs of natural-language problem descriptions and ground-truth programs. Such&#10;paradigm largely ignores some important but potentially useful signals in the&#10;problem specification such as unit tests, which thus often results in poor&#10;performance when solving complex unseen coding tasks. To address the&#10;limitations, we propose &quot;CodeRL&quot;, a new framework for program synthesis tasks&#10;through pretrained LMs and deep reinforcement learning (RL). Specifically,&#10;during training, we treat the code-generating LM as an actor network, and&#10;introduce a critic network that is trained to predict the functional&#10;correctness of generated programs and provide dense feedback signals to the&#10;actor. During inference, we introduce a new generation procedure with a&#10;critical sampling strategy that allows a model to automatically regenerate&#10;programs based on feedback from example unit tests and critic scores. For the&#10;model backbones, we extended the encoder-decoder architecture of CodeT5 with&#10;enhanced learning objectives, larger model sizes, and better pretraining data.&#10;Our method not only achieves new SOTA results on the challenging APPS&#10;benchmark, but also shows strong zero-shot transfer capability with new SOTA&#10;results on the simpler MBPP benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.00179" label="1611.00179">
        <attvalues>
          <attvalue for="0" value="Dual Learning for Machine Translation" />
          <attvalue for="1" value="  While neural machine translation (NMT) is making good progress in the past&#10;two years, tens of millions of bilingual sentence pairs are needed for its&#10;training. However, human labeling is very costly. To tackle this training data&#10;bottleneck, we develop a dual-learning mechanism, which can enable an NMT&#10;system to automatically learn from unlabeled data through a dual-learning game.&#10;This mechanism is inspired by the following observation: any machine&#10;translation task has a dual task, e.g., English-to-French translation (primal)&#10;versus French-to-English translation (dual); the primal and dual tasks can form&#10;a closed loop, and generate informative feedback signals to train the&#10;translation models, even if without the involvement of a human labeler. In the&#10;dual-learning mechanism, we use one agent to represent the model for the primal&#10;task and the other agent to represent the model for the dual task, then ask&#10;them to teach each other through a reinforcement learning process. Based on the&#10;feedback signals generated during this process (e.g., the language-model&#10;likelihood of the output of a model, and the reconstruction error of the&#10;original sentence after the primal and dual translations), we can iteratively&#10;update the two models until convergence (e.g., using the policy gradient&#10;methods). We call the corresponding approach to neural machine translation&#10;\emph{dual-NMT}. Experiments show that dual-NMT works very well on&#10;English$\leftrightarrow$French translation; especially, by learning from&#10;monolingual data (with 10% bilingual data for warm start), it achieves a&#10;comparable accuracy to NMT trained from the full bilingual data for the&#10;French-to-English translation task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.02510" label="1704.02510">
        <attvalues>
          <attvalue for="0" value="DualGAN: Unsupervised Dual Learning for Image-to-Image Translation" />
          <attvalue for="1" value="  Conditional Generative Adversarial Networks (GANs) for cross-domain&#10;image-to-image translation have made much progress recently. Depending on the&#10;task complexity, thousands to millions of labeled image pairs are needed to&#10;train a conditional GAN. However, human labeling is expensive, even&#10;impractical, and large quantities of data may not always be available. Inspired&#10;by dual learning from natural language translation, we develop a novel dual-GAN&#10;mechanism, which enables image translators to be trained from two sets of&#10;unlabeled images from two domains. In our architecture, the primal GAN learns&#10;to translate images from domain U to those in domain V, while the dual GAN&#10;learns to invert the task. The closed loop made by the primal and dual tasks&#10;allows images from either domain to be translated and then reconstructed. Hence&#10;a loss function that accounts for the reconstruction error of images can be&#10;used to train the translators. Experiments on multiple image translation tasks&#10;with unlabeled data show considerable performance gain of DualGAN over a single&#10;GAN. For some tasks, DualGAN can even achieve comparable or slightly better&#10;results than conditional GAN trained on fully labeled data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.00415" label="1707.00415">
        <attvalues>
          <attvalue for="0" value="Dual Supervised Learning" />
          <attvalue for="1" value="  Many supervised learning tasks are emerged in dual forms, e.g.,&#10;English-to-French translation vs. French-to-English translation, speech&#10;recognition vs. text to speech, and image classification vs. image generation.&#10;Two dual tasks have intrinsic connections with each other due to the&#10;probabilistic correlation between their models. This connection is, however,&#10;not effectively utilized today, since people usually train the models of two&#10;dual tasks separately and independently. In this work, we propose training the&#10;models of two dual tasks simultaneously, and explicitly exploiting the&#10;probabilistic correlation between them to regularize the training process. For&#10;ease of reference, we call the proposed approach \emph{dual supervised&#10;learning}. We demonstrate that dual supervised learning can improve the&#10;practical performances of both tasks, for various applications including&#10;machine translation, image processing, and sentiment analysis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.00085" label="2106.00085">
        <attvalues>
          <attvalue for="0" value="Language Model Evaluation Beyond Perplexity" />
          <attvalue for="1" value="  We propose an alternate approach to quantifying how well language models&#10;learn natural language: we ask how well they match the statistical tendencies&#10;of natural language. To answer this question, we analyze whether text generated&#10;from language models exhibits the statistical tendencies present in the&#10;human-generated text on which they were trained. We provide a framework--paired&#10;with significance tests--for evaluating the fit of language models to these&#10;trends. We find that neural language models appear to learn only a subset of&#10;the tendencies considered, but align much more closely with empirical trends&#10;than proposed theoretical distributions (when present). Further, the fit to&#10;different distributions is highly-dependent on both model architecture and&#10;generation strategy. As concrete examples, text generated under the nucleus&#10;sampling scheme adheres more closely to the type--token relationship of natural&#10;language than text produced using standard ancestral sampling; text from LSTMs&#10;reflects the natural language distributions over length, stopwords, and symbols&#10;surprisingly well.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="\clara{incorporate \cite{kuncoro-etal-2019-scalable} and \cite{mueller-etal-2020-cross}? or GLTR and HUSE?}&#10;In the last few years, a number of works have extended language model analysis beyond simple evaluation metrics---like perplexity---in order to understand what attributes of human language these models are learning. Some use task-based approaches, i.e., they design a set of tasks that require a specific subset of linguistic knowledge then evaluate model performance on these tasks \cite[][inter alia]{linzen-etal-2016-assessing,gulordava-etal-2018-colorless,tacl_howdoweknow}. Others use model-based approaches, where a separate model is trained to perform some auxiliary task on representations learned by the model under test \cite[][inter alia]{blevins-etal-2018-deep, giulianelli-etal-2018-hood,sorodoc-etal-2020-probing}. We direct readers to \cite{belinkov-glass-2019-analysis} for a full survey of probing methods.\looseness=-1&#10;&#10;These approaches have drawbacks; for example, introducing a secondary model to determine what the original model has learned presents confounding factors \cite{hewitt-liang-2019-designing}. The designing of auxiliary tasks for assessing linguistic knowledge requires large manual effort and lends itself to implicit bias about how linguistic phenomena should manifest. In contrast, our work allows us to take a hands-off\clara{better word?} approach to analyzing language models. We see the benefit of this in \ref{sec:exp}, where our results without an assumed model of statistical tendencies give us a much different sense of which empirical properties of human-generated text our models have learned.\looseness=-1&#10;&#10;Our work is closest to that of \cite{takahashi_statistical,takahashi_evaluating} who use model generated text to visually analyze whether language models reflect well-established statistical tendencies. &#10;In contrast, our work provides a quantitative framework, along with appropriate significance tests, for evaluating distribution fits. &#10;We additionally assess the fit of language models to our test set directly, rather than solely to established laws. &#10;Further, our analysis includes different generation strategies, multiple neural architectures, and a wider variety of empirical language distributions.\looseness=-1&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Statistical Analysis, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2304.10428" label="2304.10428">
        <attvalues>
          <attvalue for="0" value="GPT-NER: Named Entity Recognition via Large Language Models" />
          <attvalue for="1" value="  Despite the fact that large-scale Language Models (LLM) have achieved SOTA&#10;performances on a variety of NLP tasks, its performance on NER is still&#10;significantly below supervised baselines. This is due to the gap between the&#10;two tasks the NER and LLMs: the former is a sequence labeling task in nature&#10;while the latter is a text-generation model.&#10;  In this paper, we propose GPT-NER to resolve this issue. GPT-NER bridges the&#10;gap by transforming the sequence labeling task to a generation task that can be&#10;easily adapted by LLMs e.g., the task of finding location entities in the input&#10;text &quot;Columbus is a city&quot; is transformed to generate the text sequence&#10;&quot;@@Columbus## is a city&quot;, where special tokens @@## marks the entity to&#10;extract. To efficiently address the &quot;hallucination&quot; issue of LLMs, where LLMs&#10;have a strong inclination to over-confidently label NULL inputs as entities, we&#10;propose a self-verification strategy by prompting LLMs to ask itself whether&#10;the extracted entities belong to a labeled entity tag.&#10;  We conduct experiments on five widely adopted NER datasets, and GPT-NER&#10;achieves comparable performances to fully supervised baselines, which is the&#10;first time as far as we are concerned. More importantly, we find that GPT-NER&#10;exhibits a greater ability in the low-resource and few-shot setups, when the&#10;amount of training data is extremely scarce, GPT-NER performs significantly&#10;better than supervised models. This demonstrates the capabilities of GPT-NER in&#10;real-world NER applications where the number of labeled examples is limited.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Artificial Intelligence Strategies, Machine Learning, Computer Science, Linguistics, Language Model Limitations, Named Entity Recognition, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2009.11462" label="2009.11462">
        <attvalues>
          <attvalue for="0" value="RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language&#10;  Models" />
          <attvalue for="1" value="  Pretrained neural language models (LMs) are prone to generating racist,&#10;sexist, or otherwise toxic language which hinders their safe deployment. We&#10;investigate the extent to which pretrained LMs can be prompted to generate&#10;toxic language, and the effectiveness of controllable text generation&#10;algorithms at preventing such toxic degeneration. We create and release&#10;RealToxicityPrompts, a dataset of 100K naturally occurring, sentence-level&#10;prompts derived from a large corpus of English web text, paired with toxicity&#10;scores from a widely-used toxicity classifier. Using RealToxicityPrompts, we&#10;find that pretrained LMs can degenerate into toxic text even from seemingly&#10;innocuous prompts. We empirically assess several controllable generation&#10;methods, and find that while data- or compute-intensive methods (e.g., adaptive&#10;pretraining on non-toxic data) are more effective at steering away from&#10;toxicity than simpler solutions (e.g., banning &quot;bad&quot; words), no current method&#10;is failsafe against neural toxic degeneration. To pinpoint the potential cause&#10;of such persistent toxic degeneration, we analyze two web text corpora used to&#10;pretrain several LMs (including GPT-2; Radford et. al, 2019), and find a&#10;significant amount of offensive, factually unreliable, and otherwise toxic&#10;content. Our work provides a test bed for evaluating toxic generations by LMs&#10;and stresses the need for better data selection processes for pretraining.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.00861" label="2112.00861">
        <attvalues>
          <attvalue for="0" value="A General Language Assistant as a Laboratory for Alignment" />
          <attvalue for="1" value="  Given the broad capabilities of large language models, it should be possible&#10;to work towards a general-purpose, text-based assistant that is aligned with&#10;human values, meaning that it is helpful, honest, and harmless. As an initial&#10;foray in this direction we study simple baseline techniques and evaluations,&#10;such as prompting. We find that the benefits from modest interventions increase&#10;with model size, generalize to a variety of alignment evaluations, and do not&#10;compromise the performance of large models. Next we investigate scaling trends&#10;for several training objectives relevant to alignment, comparing imitation&#10;learning, binary discrimination, and ranked preference modeling. We find that&#10;ranked preference modeling performs much better than imitation learning, and&#10;often scales more favorably with model size. In contrast, binary discrimination&#10;typically performs and scales very similarly to imitation learning. Finally we&#10;study a `preference model pre-training' stage of training, with the goal of&#10;improving sample efficiency when finetuning on human preferences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15444" label="2305.15444">
        <attvalues>
          <attvalue for="0" value="PromptNER: Prompting For Named Entity Recognition" />
          <attvalue for="1" value="  In a surprising turn, Large Language Models (LLMs) together with a growing&#10;arsenal of prompt-based heuristics now offer powerful off-the-shelf approaches&#10;providing few-shot solutions to myriad classic NLP problems. However, despite&#10;promising early results, these LLM-based few-shot methods remain far from the&#10;state of the art in Named Entity Recognition (NER), where prevailing methods&#10;include learning representations via end-to-end structural understanding and&#10;fine-tuning on standard labeled corpora. In this paper, we introduce PromptNER,&#10;a new state-of-the-art algorithm for few-Shot and cross-domain NER. To adapt to&#10;any new NER task PromptNER requires a set of entity definitions in addition to&#10;the standard few-shot examples. Given a sentence, PromptNER prompts an LLM to&#10;produce a list of potential entities along with corresponding explanations&#10;justifying their compatibility with the provided entity type definitions.&#10;Remarkably, PromptNER achieves state-of-the-art performance on few-shot NER,&#10;achieving a 4% (absolute) improvement in F1 score on the ConLL dataset, a 9%&#10;(absolute) improvement on the GENIA dataset, and a 4% (absolute) improvement on&#10;the FewNERD dataset. PromptNER also moves the state of the art on Cross Domain&#10;NER, outperforming prior methods (including those not limited to the few-shot&#10;setting), setting a new mark on 3/5 CrossNER target domains, with an average F1&#10;gain of 3%, despite using less than 2% of the available data.&#10;" />
          <attvalue for="2" value="&#10;Named Entity Recognition \cite{chinchor1995muc} is often a vital component &#10;in text processing pipelines for information extraction and semantic understanding&#10;\cite{sharma2022named, ali2022named}. &#10;Current methods perform well when training data is plentiful&#10;\cite{wang2022deepstruct, yu2020named, li2022unified, wang2020automated}.&#10;However, their applicability to many real-world problems&#10;is hindered by their reliance on fixed entity definitions&#10;and large amounts of in-domain training data&#10;for the specific NER formulation and population of interest.&#10;Unfortunately, commitments about what constitute the relevant entities&#10;vary wildly across use cases, a fact that is reflected&#10;in the diversity of academic datasets&#10;(contrast, e.g., medical NER datasets with CoNLL or OntoNotes). &#10;Ultimately, these differing commitments &#10;stem from differences in the envisioned use cases.&#10;Should we categorize the phrase `Theory of General Relativity' as an entity? &#10;A media company tasked with extracting &#10;information from political articles&#10;might not designate physical laws&#10;as a relevant class of entities&#10;but a scientific journal might. &#10;Given the diversity of use cases &#10;and underlying documents&#10;that characterize different deployment settings,&#10;we might hope ideally for a system to adapt to new settings flexibly,&#10;requiring minimal labeled data, &#10;human effort, and computational cost. &#10;&#10;With the emergence of LLMs, the NLP community &#10;has developed a repertoire of in-context learning strategies &#10;that have rapidly advanced the state of few-shot learning&#10;for myriad tasks \cite{brown2020language, wei2022chain, liu2023pre}.&#10;However, such prompting-based approaches have yet &#10;to show comparable impact in NER, &#10;where current methods typically cast few-shot learning &#10;as a domain transfer problem, training on large amounts of source data &#10;and fine-tuning on exemplars from the target domain \cite{huang2022copner, yang2022factmix}.&#10;Moreover, a significant gap remains between the best few-shot NER methods&#10;and the performance of end-to-end trained models \cite{wang2022deepstruct, xu2022clozing}.&#10;These few-shot methods struggle when the source and target domains &#10;differ with respect to what constitutes an entity \cite{yang2022factmix, das2022container}. &#10;A separate class of adaptation methods have shown promise &#10;when the source and target vary considerably, &#10;but they tend to require hundreds of data points to be effective&#10;\cite{hu2022label, chen2023one, hu2022entda, chen2022prompt}. &#10;&#10;In this paper, we introduce PromptNER,&#10;a prompting-based NER method&#10;that achieves state-of-the-art results &#10;on FewShot NER and CrossDomain NER. &#10;Our method consists of 4 key components---a &#10;backbone LLM, a modular definition &#10;(a document defining the set of entity types), &#10;a few examples from the target domain, and a precise format for outputing&#10;the extracted entities, which is communicated to the model&#10;via the formatting of the few-shot examples. &#10;To adapt to a new domain, our method requires modifying&#10;only the definition and the provided examples.&#10;This makes the method flexible and easy to apply across domains.&#10;PromptNER achieves 83.48\% F1 score on the CoNLL dataset \cite{sang2003introduction} in a few-shot setting, improving over the best previous few-shot methods by 4\% (absolute). PromptNER outperforms the best-competing methods by 9\% (absolute) on the GENIA \cite{kim2003genia} dataset and 4\% (absolute)&#10;on the FewNERD-Intra \cite{ding2021few} setting&#10;and sets a new state of the art on three out of five&#10;of the CrossNER \cite{liu2021crossner} target domains,&#10;despite using only $2\%$ of the available training data.&#10;In ablations, we show that PromptNER outperforms&#10;standard Few-Shot Prompting \cite{brown2020language} &#10;and Chain-of-Thought Prompting \cite{wei2022chain}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Large Language Models, Linguistics, Entity Recognition Methods, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2208.11464" label="2208.11464">
        <attvalues>
          <attvalue for="0" value="FactMix: Using a Few Labeled In-domain Examples to Generalize to&#10;  Cross-domain Named Entity Recognition" />
          <attvalue for="1" value="  Few-shot Named Entity Recognition (NER) is imperative for entity tagging in&#10;limited resource domains and thus received proper attention in recent years.&#10;Existing approaches for few-shot NER are evaluated mainly under in-domain&#10;settings. In contrast, little is known about how these inherently faithful&#10;models perform in cross-domain NER using a few labeled in-domain examples. This&#10;paper proposes a two-step rationale-centric data augmentation method to improve&#10;the model's generalization ability. Results on several datasets show that our&#10;model-agnostic method significantly improves the performance of cross-domain&#10;NER tasks compared to previous state-of-the-art methods, including the data&#10;augmentation and prompt-tuning methods. Our codes are available at&#10;https://github.com/lifan-yuan/FactMix.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.07464" label="2105.07464">
        <attvalues>
          <attvalue for="0" value="Few-NERD: A Few-Shot Named Entity Recognition Dataset" />
          <attvalue for="1" value="  Recently, considerable literature has grown up around the theme of few-shot&#10;named entity recognition (NER), but little published benchmark data&#10;specifically focused on the practical and challenging task. Current approaches&#10;collect existing supervised NER datasets and re-organize them to the few-shot&#10;setting for empirical study. These strategies conventionally aim to recognize&#10;coarse-grained entity types with few examples, while in practice, most unseen&#10;entity types are fine-grained. In this paper, we present Few-NERD, a&#10;large-scale human-annotated few-shot NER dataset with a hierarchy of 8&#10;coarse-grained and 66 fine-grained entity types. Few-NERD consists of 188,238&#10;sentences from Wikipedia, 4,601,160 words are included and each is annotated as&#10;context or a part of a two-level entity type. To the best of our knowledge,&#10;this is the first few-shot NER dataset and the largest human-crafted NER&#10;dataset. We construct benchmark tasks with different emphases to&#10;comprehensively assess the generalization capability of models. Extensive&#10;empirical results and analysis show that Few-NERD is challenging and the&#10;problem requires further research. We make Few-NERD public at&#10;https://ningding97.github.io/fewnerd/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.04373" label="2012.04373">
        <attvalues>
          <attvalue for="0" value="CrossNER: Evaluating Cross-Domain Named Entity Recognition" />
          <attvalue for="1" value="  Cross-domain named entity recognition (NER) models are able to cope with the&#10;scarcity issue of NER samples in target domains. However, most of the existing&#10;NER benchmarks lack domain-specialized entity types or do not focus on a&#10;certain domain, leading to a less effective cross-domain evaluation. To address&#10;these obstacles, we introduce a cross-domain NER dataset (CrossNER), a&#10;fully-labeled collection of NER data spanning over five diverse domains with&#10;specialized entity categories for different domains. Additionally, we also&#10;provide a domain-related corpus since using it to continue pre-training&#10;language models (domain-adaptive pre-training) is effective for the domain&#10;adaptation. We then conduct comprehensive experiments to explore the&#10;effectiveness of leveraging different levels of the domain corpus and&#10;pre-training strategies to do domain-adaptive pre-training for the cross-domain&#10;task. Results show that focusing on the fractional corpus containing&#10;domain-specialized entities and utilizing a more challenging pre-training&#10;strategy in domain-adaptive pre-training are beneficial for the NER domain&#10;adaptation, and our proposed method can consistently outperform existing&#10;cross-domain NER baselines. Nevertheless, experiments also illustrate the&#10;challenge of this cross-domain NER task. We hope that our dataset and baselines&#10;will catalyze research in the NER domain adaptation area. The code and data are&#10;available at https://github.com/zliucr/CrossNER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.18668" label="2402.18668">
        <attvalues>
          <attvalue for="0" value="Simple linear attention language models balance the recall-throughput&#10;  tradeoff" />
          <attvalue for="1" value="  Recent work has shown that attention-based language models excel at recall,&#10;the ability to ground generations in tokens previously seen in context.&#10;However, the efficiency of attention-based models is bottle-necked during&#10;inference by the KV-cache's aggressive memory consumption. In this work, we&#10;explore whether we can improve language model efficiency (e.g. by reducing&#10;memory consumption) without compromising on recall. By applying experiments and&#10;theory to a broad set of architectures, we identify a key tradeoff between a&#10;model's state size and recall ability. We show that efficient alternatives to&#10;attention (e.g. H3, Mamba, RWKV) maintain a fixed-size recurrent state, but&#10;struggle at recall. We propose BASED a simple architecture combining linear and&#10;sliding window attention. By varying BASED window size and linear attention&#10;feature dimension, we can dial the state size and traverse the pareto frontier&#10;of the recall-memory tradeoff curve, recovering the full quality of attention&#10;on one end and the small state size of attention-alternatives on the other. We&#10;train language models up to 1.3b parameters and show that BASED matches the&#10;strongest sub-quadratic models (e.g. Mamba) in perplexity and outperforms them&#10;on real-world recall-intensive tasks by 6.22 accuracy points. Implementations&#10;of linear attention are often less efficient than optimized standard attention&#10;implementations. To make BASED competitive, we develop IO-aware algorithms that&#10;enable 24x higher throughput on language generation than FlashAttention-2, when&#10;generating 1024 tokens using 1.3b parameter models. Code for this work is&#10;provided at: https://github.com/HazyResearch/based.&#10;" />
          <attvalue for="2" value="&#10;&#10;The choice of sequence mixer (e.g. attention, convolution) in a language model affects both its quality and efficiency~\cite{arora2023zoology,vaswani2018attention}. &#10;Prior work shows that attention excels at recall, the ability to ground generations in previously seen tokens~\cite{olsson2022context,arora2023zoology}. &#10;On the other hand, the throughput of attention-based models is bottle-necked during training by quadratic compute complexity and during inference by aggressive memory consumption. &#10;The natural question is: can we improve the real-world speed and memory-use of language models without comprising on quality? &#10;&#10;Recently, a number of architectures have been proposed that enable substantially higher throughput while matching attention in perplexity~\cite{wang2022pretraining,gu2023mamba,yang2023gated,poli2023hyena,peng2023rwkv}. &#10;However, coarse metrics like overall perplexity can obscure important differences in model quality. For example, recent work shows that a specific class of architectures, gated-convolutions, despite complexity scaling sub-quadratically in sequence length, are less efficient than attention at performing recall~\cite{arora2023zoology}. Building on this analysis, we evaluate a broader class of architectures across real-world recall-intensive tasks and show attention improves over the best attention-free alternative, Mamba, by 46.7 accuracy points~(\ref{table:main-quality}). &#10;&#10;Motivated by these observations, we explore the pareto frontier of the tradeoff between high-recall and high-throughput models. &#10;We evaluate a range of architectures (e.g. attention, SSMs, and convolutions) on a popular synthetic associative recall task \cite{arora2023zoology, dao2022hungry, olsson2022context}. Since generation throughput is bottle-necked by memory consumption, we vary hyperparameters (e.g. model dimension) that affect the size of the recurrent state during generation and demonstrate a fundamental recall-memory tradeoff that holds across architecture classes (\ref{fig:tradeoff}). &#10;Attention performs associative recall perfectly, but the recurrent state (i.e. the KV-cache) grows linearly with the sequence length. Sliding window attention can cap the size of the recurrent state at the cost of worse long-range recall~\cite{mistral7b}.&#10;However, Mamba, a recently proposed SSM architecture expands the Pareto frontier beyond sliding window. &#10;This begs the question: are there other, perhaps simpler, models that can also expand the pareto frontier?&#10;&#10;To reduce the memory consumption, we consider using two simple techniques: sliding window attention and softmax-approximating linear attention.&#10;Our results on language modeling (\ref{table:main-quality}) and synthetic recall experiments (\ref{fig:based-design}, center) suggest neither primitive alone suffices to navigate the Pareto frontier. &#10;\begin{enumerate}[leftmargin=*]&#10; \item We find that linear attention alone struggles to solve associative recall (\ref{fig:based-design}, center). We hypothesize that this is because linear attention lacks the precision to perform local token shifts and comparisons~\cite{dao2022hungry,arora2023zoology}. &#10; \item In sliding window attention, associative recall range is limited by the width of the windows (\ref{fig:based-design}, center). As we increase the window size, the recurrent state grows linearly and has a non-linear affect on speed during parallel training and inference (\ref{fig:based-design}, left).&#10;\end{enumerate}&#10;&#10;We combine these two techniques into a single architecture, which we call \sysname (\ref{fig:based-design}, right). We find that sliding window attention&#10;and linear attention&#10;complement each other, enabling \sysname to expand the pareto frontier of the recall-memory tradeoff (\ref{fig:tradeoff}). We suspect that (1) the large recurrent memory of linear attention could help model long-range token interactions in the sequence and (2) sliding window attention handles the precise local shifts needed to perform associative recall. &#10;&#10;To make \sysname competitive with SoTA attention~\cite{dao2023flashattention2} and recurrent~\cite{gu2023mamba} models under wall-clock and throughput metrics, we introduce several IO-aware optimizations.&#10;\begin{enumerate}[leftmargin=*]&#10; \item Despite the theoretically improved complexity, linear attention implementations are often slower than well-optimized attention implementations ~\cite{dao2022flashattention}. In \sysname, we use the 2nd-order Taylor approximation of softmax as the linear attention feature map&#10; With sequence length $N$ and head dimension $d$, this naïvely requires $\mathcal{O}(Nd^3)$ time and space complexity&#10; ~\cite{hedgehog2023,keles2023on}. To make our attention competitive in real-world wall-clock time and memory usage, we provide hardware-efficient algorithms and custom CUDA implementations. &#10; Relative to the baseline, our algorithm reduces data movement from HBM (slower-to-acccess memory) to SRAM (faster-to-access memory)&#10; by $\mathcal{O}(Nd^2)$ bytes and from SRAM to register (fastest memory) by $O(Nd^3)$ bytes&#10; (Section~\ref{sec:efficiency}).&#10;&#10;\item Sliding window attention exploits tensor cores, specialized units on modern GPUs for performing matrix multiplications (GEMMs). While popular architectures use long window sizes (e.g. 4096 for Mistral-7B \cite{mistral7b}), we choose fixed size $64$ windows, guided by hardware properties. In particular, we use just enough occupancy to hide the tensor core kernel launch latency. Although tensor cores operate on $16 \times 16$ tiles, in \ref{fig:based-design} (left), we see that the latencies for performing $16 \times 16$ vs. $64 \times 64$ (or even $128 \times 128$) dimension matrix multiplications on NVIDIA H100 tensor cores are similar, informing our window size. &#10;\end{enumerate}&#10;&#10;In experiments, we show that ${\sysname}$ competes in quality with strong Transformer++ \cite{touvron2023llama} and SoTA sub-quadratic baselines in models up to the 1.3Bn parameters across language modeling on the Pile language, DNA modeling, and the LM Eval Harness \cite{eval-harness}. Beyond this, ${\sysname}$ outperforms prior sub-quadratic architectures on the associative recall slice of the Pile and in downstream recall-intensive tasks by $0.14$ perplexity points and $6.22$ accuracy points, respectively. In efficiency, ${\sysname}$ enables up to $24\times$ higher throughput than the strong FlashAttention-2 implementation on generation. Code for this work is provided at:.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Deep Learning Architectures, Mathematics, Attention Mechanisms, Language Model Efficiency" />
        </attvalues>
      </node>
      <node id="2212.10544" label="2212.10544">
        <attvalues>
          <attvalue for="0" value="Pretraining Without Attention" />
          <attvalue for="1" value="  Transformers have been essential to pretraining success in NLP. While other&#10;architectures have been used, downstream accuracy is either significantly&#10;worse, or requires attention layers to match standard benchmarks such as GLUE.&#10;This work explores pretraining without attention by using recent advances in&#10;sequence routing based on state-space models (SSMs). Our proposed model,&#10;Bidirectional Gated SSM (BiGS), combines SSM layers with a multiplicative&#10;gating architecture that has been effective in simplified sequence modeling&#10;architectures. The model learns static layers that do not consider pair-wise&#10;interactions. Even so, BiGS is able to match BERT pretraining accuracy on GLUE&#10;and can be extended to long-form pretraining of 4096 tokens without&#10;approximation. Analysis shows that while the models have similar average&#10;accuracy, the approach has different inductive biases than BERT in terms of&#10;interactions and syntactic representations. All models from this work are&#10;available at https://github.com/jxiw/BiGS.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.06635" label="2312.06635">
        <attvalues>
          <attvalue for="0" value="Gated Linear Attention Transformers with Hardware-Efficient Training" />
          <attvalue for="1" value="  Transformers with linear attention allow for efficient parallel training but&#10;can simultaneously be formulated as an RNN with 2D (matrix-valued) hidden&#10;states, thus enjoying linear-time inference complexity. However, linear&#10;attention generally underperforms ordinary softmax attention. Moreover, current&#10;implementations of linear attention lack I/O-awareness and are thus slower than&#10;highly optimized implementations of softmax attention. This work describes a&#10;hardware-efficient algorithm for linear attention that trades off memory&#10;movement against parallelizability. The resulting implementation, dubbed&#10;FLASHLINEARATTENTION, is faster than FLASHATTENTION-2 (Dao, 2023) as a&#10;standalone layer even on short sequence lengths (e.g., 1K). We then generalize&#10;this algorithm to a more expressive variant of linear attention with&#10;data-dependent gates. When used as a replacement for the standard attention&#10;layer in Transformers, the resulting gated linear attention (GLA) Transformer&#10;is found to perform competitively against the LLaMA-architecture Transformer&#10;(Touvron et al., 2023) as well recent linear-time-inference baselines such as&#10;RetNet (Sun et al., 2023a) and Mamba (Gu &amp; Dao, 2023) on moderate-scale&#10;language modeling experiments. GLA Transformer is especially effective at&#10;length generalization, enabling a model trained on 2K to generalize to&#10;sequences longer than 20K without significant perplexity degradations. For&#10;training speed, the GLA Transformer has higher throughput than a&#10;similarly-sized Mamba model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13048" label="2305.13048">
        <attvalues>
          <attvalue for="0" value="RWKV: Reinventing RNNs for the Transformer Era" />
          <attvalue for="1" value="  Transformers have revolutionized almost all natural language processing (NLP)&#10;tasks but suffer from memory and computational complexity that scales&#10;quadratically with sequence length. In contrast, recurrent neural networks&#10;(RNNs) exhibit linear scaling in memory and computational requirements but&#10;struggle to match the same performance as Transformers due to limitations in&#10;parallelization and scalability. We propose a novel model architecture,&#10;Receptance Weighted Key Value (RWKV), that combines the efficient&#10;parallelizable training of transformers with the efficient inference of RNNs.&#10;  Our approach leverages a linear attention mechanism and allows us to&#10;formulate the model as either a Transformer or an RNN, thus parallelizing&#10;computations during training and maintains constant computational and memory&#10;complexity during inference. We scale our models as large as 14 billion&#10;parameters, by far the largest dense RNN ever trained, and find RWKV performs&#10;on par with similarly sized Transformers, suggesting future work can leverage&#10;this architecture to create more efficient models. This work presents a&#10;significant step towards reconciling trade-offs between computational&#10;efficiency and model performance in sequence processing tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.08691" label="2307.08691">
        <attvalues>
          <attvalue for="0" value="FlashAttention-2: Faster Attention with Better Parallelism and Work&#10;  Partitioning" />
          <attvalue for="1" value="  Scaling Transformers to longer sequence lengths has been a major problem in&#10;the last several years, promising to improve performance in language modeling&#10;and high-resolution image understanding, as well as to unlock new applications&#10;in code, audio, and video generation. The attention layer is the main&#10;bottleneck in scaling to longer sequences, as its runtime and memory increase&#10;quadratically in the sequence length. FlashAttention exploits the asymmetric&#10;GPU memory hierarchy to bring significant memory saving (linear instead of&#10;quadratic) and runtime speedup (2-4$\times$ compared to optimized baselines),&#10;with no approximation. However, FlashAttention is still not nearly as fast as&#10;optimized matrix-multiply (GEMM) operations, reaching only 25-40\% of the&#10;theoretical maximum FLOPs/s. We observe that the inefficiency is due to&#10;suboptimal work partitioning between different thread blocks and warps on the&#10;GPU, causing either low-occupancy or unnecessary shared memory reads/writes. We&#10;propose FlashAttention-2, with better work partitioning to address these&#10;issues. In particular, we (1) tweak the algorithm to reduce the number of&#10;non-matmul FLOPs (2) parallelize the attention computation, even for a single&#10;head, across different thread blocks to increase occupancy, and (3) within each&#10;thread block, distribute the work between warps to reduce communication through&#10;shared memory. These yield around 2$\times$ speedup compared to FlashAttention,&#10;reaching 50-73\% of the theoretical maximum FLOPs/s on A100 and getting close&#10;to the efficiency of GEMM operations. We empirically validate that when used&#10;end-to-end to train GPT-style models, FlashAttention-2 reaches training speed&#10;of up to 225 TFLOPs/s per A100 GPU (72\% model FLOPs utilization).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.14135" label="2205.14135">
        <attvalues>
          <attvalue for="0" value="FlashAttention: Fast and Memory-Efficient Exact Attention with&#10;  IO-Awareness" />
          <attvalue for="1" value="  Transformers are slow and memory-hungry on long sequences, since the time and&#10;memory complexity of self-attention are quadratic in sequence length.&#10;Approximate attention methods have attempted to address this problem by trading&#10;off model quality to reduce the compute complexity, but often do not achieve&#10;wall-clock speedup. We argue that a missing principle is making attention&#10;algorithms IO-aware -- accounting for reads and writes between levels of GPU&#10;memory. We propose FlashAttention, an IO-aware exact attention algorithm that&#10;uses tiling to reduce the number of memory reads/writes between GPU high&#10;bandwidth memory (HBM) and GPU on-chip SRAM. We analyze the IO complexity of&#10;FlashAttention, showing that it requires fewer HBM accesses than standard&#10;attention, and is optimal for a range of SRAM sizes. We also extend&#10;FlashAttention to block-sparse attention, yielding an approximate attention&#10;algorithm that is faster than any existing approximate attention method.&#10;FlashAttention trains Transformers faster than existing baselines: 15%&#10;end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the&#10;MLPerf 1.1 training speed record, 3$\times$ speedup on GPT-2 (seq. length 1K),&#10;and 2.4$\times$ speedup on long-range arena (seq. length 1K-4K). FlashAttention&#10;and block-sparse FlashAttention enable longer context in Transformers, yielding&#10;higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on&#10;long-document classification) and entirely new capabilities: the first&#10;Transformers to achieve better-than-chance performance on the Path-X challenge&#10;(seq. length 16K, 61.4% accuracy) and Path-256 (seq. length 64K, 63.1%&#10;accuracy).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.04881" label="2209.04881">
        <attvalues>
          <attvalue for="0" value="On The Computational Complexity of Self-Attention" />
          <attvalue for="1" value="  Transformer architectures have led to remarkable progress in many&#10;state-of-art applications. However, despite their successes, modern&#10;transformers rely on the self-attention mechanism, whose time- and&#10;space-complexity is quadratic in the length of the input. Several approaches&#10;have been proposed to speed up self-attention mechanisms to achieve&#10;sub-quadratic running time; however, the large majority of these works are not&#10;accompanied by rigorous error guarantees. In this work, we establish lower&#10;bounds on the computational complexity of self-attention in a number of&#10;scenarios. We prove that the time complexity of self-attention is necessarily&#10;quadratic in the input length, unless the Strong Exponential Time Hypothesis&#10;(SETH) is false. This argument holds even if the attention computation is&#10;performed only approximately, and for a variety of attention mechanisms. As a&#10;complement to our lower bounds, we show that it is indeed possible to&#10;approximate dot-product self-attention using finite Taylor series in&#10;linear-time, at the cost of having an exponential dependence on the polynomial&#10;order.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.00784" label="2010.00784">
        <attvalues>
          <attvalue for="0" value="An Empirical Investigation Towards Efficient Multi-Domain Language Model&#10;  Pre-training" />
          <attvalue for="1" value="  Pre-training large language models has become a standard in the natural&#10;language processing community. Such models are pre-trained on generic data&#10;(e.g. BookCorpus and English Wikipedia) and often fine-tuned on tasks in the&#10;same domain. However, in order to achieve state-of-the-art performance on out&#10;of domain tasks such as clinical named entity recognition and relation&#10;extraction, additional in domain pre-training is required. In practice, staged&#10;multi-domain pre-training presents performance deterioration in the form of&#10;catastrophic forgetting (CF) when evaluated on a generic benchmark such as&#10;GLUE. In this paper we conduct an empirical investigation into known methods to&#10;mitigate CF. We find that elastic weight consolidation provides best overall&#10;scores yielding only a 0.33% drop in performance across seven generic tasks&#10;while remaining competitive in bio-medical tasks. Furthermore, we explore&#10;gradient and latent clustering based data selection techniques to improve&#10;coverage when using elastic weight consolidation and experience replay methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Current work in catastrophic forgetting mitigation in NLP has been limited.&#10;\cite{howard2018universal} introduced a multi stage training scheme for fine tuning LSTM based universal language models (ULMFiT).&#10;The authors proposed that current methods, rather than data, are ineffective and focused on learning rate control across layers, as well as modifying learning rate scheduling.&#10;A larger category of work deals with constraining model parameters to a latent space where they continue to capture previous tasks.&#10;Initial work focused on model regularization and varying activations \cite{goodfellow2013empirical}. \cite{kirkpatrick2017overcoming} provided a more sophisticated solution constraining weights individually termed elastic weight consolidation (EWC).&#10;We make use of both EWC and ULMFiT and provide further technical detail in this paper.&#10;The final approach is focused on experience replay.&#10;Using small samples of data from previous tasks coupled with local adaptation \cite{d2019episodic} demonstrate improvement in a lifelong learning training scheme.&#10;\cite{chaudhry2019continual} also explore lifelong learning by experimenting with updating the memory bank for experience replay.&#10;Our work focuses on both of these techniques with the major difference being problem scale.&#10;Many existing works apply these solutions on small networks whereas we experiment on architectures having several orders of magnitude more parameters.&#10;&#10;There has been a recent focus on more effective pre-training which focuses on narrowing the pre-training domain as we move closer towards fine-tuning.&#10;STILTs \cite{stilts} and TandA \cite{garg2019tanda} use intermediate tasks (in a data rich domain) training to lower variance during target task fine-tuning.&#10;This intuition was also covered in the visio-linguistic domain by \cite{singh2020pretraining}.&#10;Finally \cite{gururangan2020dont} work on MLM pre-training and provide conclusive evidence at scale of the works listed above.&#10;This last body of work, although dealing with pre-training is different from our work in that we study mitigation of domain forgetting, rather than reducing variance by adding intermediate domains or tasks to pre-training.&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Pre-training, Domain Adaptation Techniques, Artificial Intelligence, Mathematics, Catastrophic Forgetting Mitigation" />
        </attvalues>
      </node>
      <node id="2310.18358" label="2310.18358">
        <attvalues>
          <attvalue for="0" value="A Communication Theory Perspective on Prompting Engineering Methods for&#10;  Large Language Models" />
          <attvalue for="1" value="  The springing up of Large Language Models (LLMs) has shifted the community&#10;from single-task-orientated natural language processing (NLP) research to a&#10;holistic end-to-end multi-task learning paradigm. Along this line of research&#10;endeavors in the area, LLM-based prompting methods have attracted much&#10;attention, partially due to the technological advantages brought by prompt&#10;engineering (PE) as well as the underlying NLP principles disclosed by various&#10;prompting methods. Traditional supervised learning usually requires training a&#10;model based on labeled data and then making predictions. In contrast, PE&#10;methods directly use the powerful capabilities of existing LLMs (i.e., GPT-3&#10;and GPT-4) via composing appropriate prompts, especially under few-shot or&#10;zero-shot scenarios. Facing the abundance of studies related to the prompting&#10;and the ever-evolving nature of this field, this article aims to (i) illustrate&#10;a novel perspective to review existing PE methods, within the well-established&#10;communication theory framework; (ii) facilitate a better/deeper understanding&#10;of developing trends of existing PE methods used in four typical tasks; (iii)&#10;shed light on promising research directions for future PE methods.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models (LLMs) (e.g., GPT-3 \cite{brown2020language}, GPT-4 \cite{GPT4}, LLaMa \cite{touvron2023llama}) make it possible for machines to understand users' attention accurately, thus revolutionizing the human-computer interaction (HCI) paradigm. Compared to traditional machine systems like databases and search engines, LLMs demonstrate impressive capability in understanding, generating, and processing natural language, facilitating a series of services ranging from personal assistants \cite{cheng2023potential}, healthcare \cite{cascella2023evaluating} to e-commercial tools \cite{george2023review} via a {unified} natural language interface between users and machine. &#10;&#10;The research paradigm around LLM has shifted from single-task-orientated natural language processing (NLP) research to a holistic end-to-end multi-task learning approach. Along this line of research endeavors, LLM-based prompting engineering (PE) methods \cite{liu2023pre,brown2020language} have attracted much attention, partially because they are the key techniques in making full use of the superior capabilities of LLMs via constructing appropriate prompts. &#10;PE refers to the process of crafting effective instructions to guide the behavior of LLMs, and it greatly helps in bridging the gap between the pre-training tasks used to construct the LLM with the down-streaming tasks queried by the end users. Through careful prompt designing, users can steer LLM's output in the desired direction, shaping its style, tone, and content to align with their goals. &#10;&#10;To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}. A common theme of PE development lies in continuously improving accuracy and responsiveness of designed prompts, which often include components like Role, Context, Input, Output Format, and Examples (see Fig.~\ref{fig:prompt}). Specifically, prompt template and answering engineering has evolved from solely utilizing discrete prompts to continuous prompts, and even to exploring hybrid prompts that combine continuous and discrete elements, which provides a larger optimization space to achieve better performance.&#10;With the emergent capability of LLM, LLMs are leveraged to plan and use external tools via its in-context learning capability, which significantly enhanced its ability in specialized domains and broadened its application fields.&#10;&#10;Following these studies, one can summarize representative PE methods in a chronological overview as illustrated in Fig.~\ref{fig:timeline}. These methods can be categorized as three groups that respectively correspond to three prompting tasks proposed to improve the qualities of LLMs' outputs, namely prompt template engineering, prompt answer engineering, and multi-turn prompting and multi-prompt learning. An example of the input and output for the above-mentioned tasks can be found in Table~\ref{tb:example}. &#10;&#10;\begin{itemize}&#10;\item First, prompt template engineering methods aim to carefully design a piece of ``text'' that guides the language models to produce the desired outputs. For example, in Table~\ref{tb:example}, to finish a classical sentiment detection for a input A=``Great places to eat near my location!'', the prompt template engineering designs a template ``[A] Overall, it was a $[Z]$ restaurant'' to enforce the LLM to fill the desired comments in the blank i.e. $[Z]$. Essentially this type of template engineering method induces LLM to focus on word embeddings that are relevant to the questions. A common designing principle of existing prompt template engineering methods is to better align information between users and LLMs. Such a trend is manifested by the evolution from using discrete prompts (e.g., a piece of human-readable text) \cite{jiang2020can, petroni2019language} to continuous ones (e.g., a continuous task-specific vector) \cite{li2021prefix, lester2021power}.&#10;&#10;\item Second, prompt answer engineering \cite{liu2023pre} refers to the process of searching for an answer space and a map to the original output, which enhances users' understanding of the information encapsulated within the LLM. For the same example in Table~\ref{tb:example}, the prompt answer engineering aims to find a mapping from the result ``good'' obtained from the LLM to the desired answer ``positive''. The field of prompt answer engineering is currently witnessing a notable development trend characterized by the pursuit of models that excel in decoding model information from simple mapping to complex mapping to enhance human comprehension. &#10;&#10;\item Third, multi-prompting methods mainly applied ensemble techniques \cite{schick2020exploiting} to mitigate the sensitivity of LLM to different formulations and to obtain a more stable output. In Table~\ref{tb:example}, the multi-prompting methods combine three different templates (i.e., 1. ``It was a [Z]''; 2. ``Just [Z]''; 3. ``All in all, it was [Z]'';) and their inference results (i.e., 1. ``good'' 2. ``great!'' 3. ``okay'') to obtain the final desired one (i.e., ``positive''). &#10;Later, as LLMs become more capable, multi-turn prompt methods attract more attention that aims to provide more context to LLM by leveraging information either from LLM itself or external tools \cite{kojima2023large, paranjape2023art}. &#10;In the field of multi-prompting methods, researchers are endeavoring to develop adaptive strategies that enhance LLM's ability to task planning and the utilization of tools. &#10;\end{itemize}&#10;&#10;In this article, we summarize the prompting methods from a communication theory perspective with which the ultimate goal of PE is to reduce the information misunderstanding between the users and the LLMs. &#10;Therefore, as delineated in Section 2, the communication theory perspective provides a coherent explanation of different PE methods in terms of their objectives and underlying principles. &#10;Moreover, this novel perspective also offers and presents insights into scenarios where existing prompting methods come short.&#10;&#10;The remainder of the article is structured as follows: Section~\ref{sec:over} details the overview of the prompting methods from the communication theory perspective. &#10;Sections~~\ref{sec:template}, ~\ref{sec:answer}, and ~\ref{sec:multi} review and summarize the recent progresses, respectively, from four PE tasks namely prompt template engineering, answer engineering, and multi-turn prompting methods. &#10;Section~\ref{sec:dis} discusses other related surveys and potential research directions.&#10;Finally, we conclude this article in Section~\ref{sec:con} by summarizing significant findings and discussing potential research directions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Large Language Models, Linguistics, Information Science, Natural Language Processing, Prompt Engineering" />
        </attvalues>
      </node>
      <node id="2001.07676" label="2001.07676">
        <attvalues>
          <attvalue for="0" value="Exploiting Cloze Questions for Few Shot Text Classification and Natural&#10;  Language Inference" />
          <attvalue for="1" value="  Some NLP tasks can be solved in a fully unsupervised fashion by providing a&#10;pretrained language model with &quot;task descriptions&quot; in natural language (e.g.,&#10;Radford et al., 2019). While this approach underperforms its supervised&#10;counterpart, we show in this work that the two ideas can be combined: We&#10;introduce Pattern-Exploiting Training (PET), a semi-supervised training&#10;procedure that reformulates input examples as cloze-style phrases to help&#10;language models understand a given task. These phrases are then used to assign&#10;soft labels to a large set of unlabeled examples. Finally, standard supervised&#10;training is performed on the resulting training set. For several tasks and&#10;languages, PET outperforms supervised training and strong semi-supervised&#10;approaches in low-resource settings by a large margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.05240" label="2104.05240">
        <attvalues>
          <attvalue for="0" value="Factual Probing Is [MASK]: Learning vs. Learning to Recall" />
          <attvalue for="1" value="  Petroni et al. (2019) demonstrated that it is possible to retrieve world&#10;facts from a pre-trained language model by expressing them as cloze-style&#10;prompts and interpret the model's prediction accuracy as a lower bound on the&#10;amount of factual information it encodes. Subsequent work has attempted to&#10;tighten the estimate by searching for better prompts, using a disjoint set of&#10;facts as training data. In this work, we make two complementary contributions&#10;to better understand these factual probing techniques. First, we propose&#10;OptiPrompt, a novel and efficient method which directly optimizes in continuous&#10;embedding space. We find this simple method is able to predict an additional&#10;6.4% of facts in the LAMA benchmark. Second, we raise a more important&#10;question: Can we really interpret these probing results as a lower bound? Is it&#10;possible that these prompt-search methods learn from the training data too? We&#10;find, somewhat surprisingly, that the training data used by these methods&#10;contains certain regularities of the underlying fact distribution, and all the&#10;existing prompt methods, including ours, are able to exploit them for better&#10;fact prediction. We conduct a set of control experiments to disentangle&#10;&quot;learning&quot; from &quot;learning to recall&quot;, providing a more detailed picture of what&#10;different prompts can reveal about pre-trained language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.13161" label="2108.13161">
        <attvalues>
          <attvalue for="0" value="Differentiable Prompt Makes Pre-trained Language Models Better Few-shot&#10;  Learners" />
          <attvalue for="1" value="  Large-scale pre-trained language models have contributed significantly to&#10;natural language processing by demonstrating remarkable abilities as few-shot&#10;learners. However, their effectiveness depends mainly on scaling the model&#10;parameters and prompt design, hindering their implementation in most real-world&#10;applications. This study proposes a novel pluggable, extensible, and efficient&#10;approach named DifferentiAble pRompT (DART), which can convert small language&#10;models into better few-shot learners without any prompt engineering. The main&#10;principle behind this approach involves reformulating potential natural&#10;language processing tasks into the task of a pre-trained language model and&#10;differentially optimizing the prompt template as well as the target label with&#10;backpropagation. Furthermore, the proposed approach can be: (i) Plugged to any&#10;pre-trained language models; (ii) Extended to widespread classification tasks.&#10;A comprehensive evaluation of standard NLP tasks demonstrates that the proposed&#10;approach achieves a better few-shot performance. Code is available in&#10;https://github.com/zjunlp/DART.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Language Model Prompting. &#10;The language model prompting has emerged with the introduction of GPT-3~(\cite{DBLP:conf/nips/BrownMRSKDNSSAA20}), which demonstrates excellent few-shot performance (\cite{DBLP:journals/corr/abs-2107-13586}). &#10;However, GPT-3 is not designed for fine-tuning; it mainly relies on the handcraft prompt (in-context learning~(\cite{DBLP:journals/corr/abs-2101-06804,DBLP:journals/corr/abs-2102-09690,ding2021prompt,DBLP:journals/corr/abs-2108-04106})). &#10;Thus, recent studies (\cite{DBLP:journals/corr/abs-2104-06599,DBLP:journals/corr/abs-2101-00121,chen2021adaprompt}) conducted in this field have been focused on automatically searching the prompts. &#10;\cite{DBLP:conf/eacl/SchickS21,DBLP:journals/corr/abs-2009-07118} propose the PET, which reformulates the NLP tasks as cloze-style questions and performs gradient-based fine-tuning.&#10;\cite{DBLP:journals/corr/abs-2103-11955} improve the PET with a denser supervision object during fine-tuning. &#10;\cite{DBLP:conf/emnlp/ShinRLWS20} propose the AUTOPROMPT to create prompts for a diverse set of tasks based on a gradient-guided search. &#10;\cite{DBLP:journals/corr/abs-2105-11259} propose an approach called PTR, which leverages logic rules to construct prompts with sub-prompts for many-class text classification. &#10;\cite{DBLP:journals/corr/abs-2104-14690} reformulate potential NLP task into an entailment one, and then fine-tune the model with few-shot samples. &#10;\cite{DBLP:journals/corr/abs-2108-02035} propose an approach to incorporate external knowledge graph into the verbalizer with calibration. &#10;Additionally, \cite{DBLP:journals/corr/abs-2012-15723} present LM-BFF—better few-shot fine-tuning of language models, which leverages T5~(\cite{DBLP:journals/jmlr/RaffelSRLNMZLL20}) to generate templates and search label tokens in the vocabulary. &#10;However, the utilization of the generative model and the label search with validation is computation-intensive. &#10;Moreover, the prompt search over discrete space is sub-optimal due to the continuous nature of neural networks. &#10;&#10;To overcome these limitations, \cite{DBLP:journals/corr/abs-2103-10385} propose P-tuning, which employs trainable continuous prompt embeddings learned by an LSTM. &#10;\cite{zhong2021factual} propose an effective continuous method called OPTIPROMPT to optimize prompts for factual probing. &#10;\cite{DBLP:journals/corr/abs-2103-10385} propose prefix-tuning, which keeps language model parameters frozen but optimizes a small continuous task-specific vector for natural language generation tasks.&#10;\cite{DBLP:journals/corr/abs-2104-08691} propose a mechanism for learning “soft prompts” to condition frozen language models to perform downstream tasks.&#10;However, these approaches still have to optimize the external parameters (e.g., LSTM in P-tuning) and are prone to complex label space. &#10;&#10;Conversely, this study aims to develop a novel few-shot learning framework based on pre-trained language models which can reduce the prompt engineering (including templates and labels) and external parameter optimization. &#10;Furthermore, the proposed approach only leverages the noninvasive modification of the model, which can be plugged into any pre-trained language model and extended to the widespread classification task. &#10;&#10;Few-shot Learning.&#10;Few-shot learning can significantly improve the learning capabilities for machine intelligence and practical adaptive applications by accessing only a small number of labeled examples (\cite{DBLP:conf/www/ZhangDSCZC20}). &#10;The proposed approach corresponds to the other few-shot NLP methods, including: &#10;(1) Meta-learning (\cite{DBLP:conf/naacl/YuGYCPCTWZ18,DBLP:conf/iclr/BaoWCB20,DBLP:conf/coling/BansalJM20,DBLP:conf/aaai/DengZSCC20,DBLP:conf/wsdm/DengZKZZC20,DBLP:conf/coling/YuZDYZC20}), in which the quantities of the auxiliary tasks are optimized. &#10;(2) Intermediate training (\cite{DBLP:journals/corr/abs-1811-01088,DBLP:conf/emnlp/YinRRSX20}), which supplements the pre-trained LMs with further training on the data-rich supervised tasks.&#10;(3) Semi-supervised learning (\cite{DBLP:conf/iclr/MiyatoDG17,DBLP:conf/nips/XieDHL020}), which leverages unlabeled samples. &#10;The proposed approach focuses on a more realistic few-shot setting (the number of labeled instances per class can be any variable).&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2205.12548" label="2205.12548">
        <attvalues>
          <attvalue for="0" value="RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning" />
          <attvalue for="1" value="  Prompting has shown impressive success in enabling large pretrained language&#10;models (LMs) to perform diverse NLP tasks, especially when only few downstream&#10;data are available. Automatically finding the optimal prompt for each task,&#10;however, is challenging. Most existing work resorts to tuning soft prompt&#10;(e.g., embeddings) which falls short of interpretability, reusability across&#10;LMs, and applicability when gradients are not accessible. Discrete prompt, on&#10;the other hand, is difficult to optimize, and is often created by &quot;enumeration&#10;(e.g., paraphrasing)-then-selection&quot; heuristics that do not explore the prompt&#10;space systematically. This paper proposes RLPrompt, an efficient discrete&#10;prompt optimization approach with reinforcement learning (RL). RLPrompt&#10;formulates a parameter-efficient policy network that generates the desired&#10;discrete prompt after training with reward. To overcome the complexity and&#10;stochasticity of reward signals by the large LM environment, we incorporate&#10;effective reward stabilization that substantially enhances the training&#10;efficiency. RLPrompt is flexibly applicable to different types of LMs, such as&#10;masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both&#10;classification and generation tasks. Experiments on few-shot classification and&#10;unsupervised text style transfer show superior performance over a wide range of&#10;existing finetuning or prompting methods. Interestingly, the resulting&#10;optimized prompts are often ungrammatical gibberish text; and surprisingly,&#10;those gibberish prompts are transferrable between different LMs to retain&#10;significant performance, indicating LM prompting may not follow human language&#10;patterns.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;We discuss briefly the various prompting paradigms in previous work, and provide more comprehensive discussion in Appendix \S\ref{appendix:related-work}.&#10;The conventional usage for pre-trained LMs is fine-tuning on downstream datasets~\cite[etc.]{devlin-etal-2019-bert, lewis2020bart}, which expensively updates all model parameters and shows limited success with small datasets. &#10;\cite{brown2020language} show that manual prompts can steer large LMs to perform NLP tasks without any training \cite{raffel2020T5,schick2021exploiting,sanh2021T0}.&#10;Another line of work~\cite{weller2020learning,efrat2020turking,mishra2021NI,wang2022NI2} develop instructional prompts which provide task descriptions instead of fill-in-the-blank questions. &#10;With few-shot training examples, \cite{brown2020language} and follow-ups \cite{gao2021LMBFF,liu2021KATE,lu2021fantastically,min2022rethinking} achieve remarkable performance by inserting in-context demonstrations. &#10;Replacing discrete prompts with continuous embeddings, several works \cite{qin-eisner-2021-learning,li2021prefix,liu2021ptuningv1} tune soft prompts using gradient descent.&#10;By their continuous nature, however, soft prompts are difficult to understand~\cite{lester2021promptuning,hambardzumyan2021warp,khashabi2021prompt}, require expensive gradient information \cite{sun2022black,diao2022black} and are incompatible for reuse across models due to mismatched latent spaces \cite{su2021transferability}. &#10;Some existing works seek to locate better discrete prompts by augmenting human-written prompts with heuristics such as paraphrasing \cite{jiang2020can}, editing \cite{prasad2022grips}, and reframing \cite{mishra2021reframing}, and selecting by some downstream metric.&#10;AutoPrompt&#10;\cite{shin2020autoprompt} edits discrete prompts with guidance from model gradients, which sees some success with large training data but limited general applicability due to unstable approximations.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2209.11486" label="2209.11486">
        <attvalues>
          <attvalue for="0" value="MetaPrompting: Learning to Learn Better Prompts" />
          <attvalue for="1" value="  Prompting method is regarded as one of the crucial progress for few-shot&#10;nature language processing. Recent research on prompting moves from discrete&#10;tokens based ``hard prompts'' to continuous ``soft prompts'', which employ&#10;learnable vectors as pseudo prompt tokens and achieve better performance.&#10;Though showing promising prospects, these soft-prompting methods are observed&#10;to rely heavily on good initialization to take effect. Unfortunately, obtaining&#10;a perfect initialization for soft prompts requires understanding of inner&#10;language models working and elaborate design, which is no easy task and has to&#10;restart from scratch for each new task. To remedy this, we propose a&#10;generalized soft prompting method called MetaPrompting, which adopts the&#10;well-recognized model-agnostic meta-learning algorithm to automatically find&#10;better prompt initialization that facilitates fast adaptation to new prompting&#10;tasks.Extensive experiments show MetaPrompting tackles soft prompt&#10;initialization problem and brings significant improvement on four different&#10;datasets (over 6 points improvement in accuracy for 1-shot setting), achieving&#10;new state-of-the-art performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.11219" label="2206.11219">
        <attvalues>
          <attvalue for="0" value="Understanding the Properties of Generated Corpora" />
          <attvalue for="1" value="  Models for text generation have become focal for many research tasks and&#10;especially for the generation of sentence corpora. However, understanding the&#10;properties of an automatically generated text corpus remains challenging. We&#10;propose a set of tools that examine the properties of generated text corpora.&#10;Applying these tools on various generated corpora allowed us to gain new&#10;insights into the properties of the generative models. As part of our&#10;characterization process, we found remarkable differences in the corpora&#10;generated by two leading generative technologies.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text generation models have continued to gain traction due to groundbreaking progress in neural networks techniques \cite{serban2017hierarchical, guu2018generating, Li2016deeprl}, the recent development of advanced transformer-based architectures \cite{radford2019language}, and training over huge amounts of data \cite{wu2019conditional}.&#10;These models all have a common generation phase, which produces a new set of sentences based on a set of original sentences.&#10;However, aside from traditional \cite{reiter2000building} and rule-based \cite{li2018delete} algorithms, researchers face difficulties understanding the characteristics of the generated corpora.&#10;&#10;Researchers need to examine the generation process outcome, especially if they want to develop new generation algorithms or understand the disparity in the results when tweaking a generative model's architecture or hyper-parameters. &#10;&#10;In some cases, the generated corpus is used directly, for example in an education scenario that requires a diverse set of sentences to compose non-similar English quizzes. &#10;Alternatively, the generated sentences may serve as input for a downstream task. In this case, the best way to assess the generated corpus is by evaluating the task's end goal. In certain cases, it is beneficial to predict the quality of the generated corpus before the downstream task. For example, we might be interested in data augmentation for a text classification task in which humans label the generated sentences. The end goal here would be to improve the classifier's accuracy using the generated data. The ability to measure corpus quality and sentence readability, before applying corpus labeling could reduce the efforts required and save costs. &#10;&#10;An automatically generated corpus could naively be evaluated using machine translation metrics such as BLEU \citepar{papineni2002bleu}, ROUGE&#10;\citepar{lin2004rouge}, or METEOR \citepar{lavie2007meteor}.&#10;According to \cite{novikova2017we}, these metrics &#10;only weakly correlate with human ratings. In addition, these metrics only assess the quality of each single sentence but do not assess the diversity of the entire corpus or the coverage induced by the original sentences. &#10;&#10;To address the diversity, \cite{zhu2018texygen} introduced the notion of Self-BLEU, which compares each generated sentence to the rest of the generated set. The probabilistic approach also tackles the diversity issue. It assumes sentences are sampled from a latent distribution and uses a well-defined distribution distance measure (e.g., FID \citepar{lucic2018gans}, KLD \citepar{kullback1951kld}, and T-Test). &#10;That said, this method does not indicate whether any differences in distribution are associated with plausibility changes, mode collapse, or other issues. Mode collapse occurs when the generator generates a limited diversity of samples, or even the same sample sentence, regardless of the input. Moreover, this approach does not allow us to detect plagiarism, when the model makes small to no changes in the original sentences. &#10;&#10;Humans are the most skillful evaluators when it comes to assessing the quality of a single sentence in terms of plausibility, fluency, grammar, relatedness to the domain, etc. However, it is much harder for humans to determine the diversity, coverage, or the distance from the original set for an entire corpus. &#10;Last year, \cite{hashimoto2019unifying} suggested a compelling approach that combines both human and automatic metrics. However, this approach is costly (requires 20 ratings for each sentence) and depends on the generative model itself.&#10;&#10;In this paper we suggest a set of fully automated metrics that can help researchers understand the properties of an automatically generated corpus and assess whether it fits their task. &#10;Our characterization metrics cover five main dimensions:&#10;1) Sentence quantity. &#10;2) Vocabulary enrichment.&#10;3) Grammatical correctness and plausibility of the generated sentences.&#10;4) Semantic similarity between the test and the generated datasets.&#10;5) Syntactic distance between the original and the generated datasets.&#10;&#10;Note that our characterization tool is unrelated to the internals of the generation model's algorithm, architecture, or parameters; it does not assume any underlying constraints on the generation process.&#10;&#10;The tool allows us to independently assess the quality, coverage, plausibility, and diversity, in addition to evaluating the plagiarism. &#10;&#10;Using our tool and metrics, we examined several domain-specific corpora resulting from three kinds of generative models and three datasets. We compared their characteristics along the suggested metrics and discuss insights derived from the results. The code is publicly available for the NLP community. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Generative Model Evaluation, Computational Linguistics, Corpus Analysis Tools, Artificial Intelligence, Natural Language Processing, Statistics, Text Generation Models" />
        </attvalues>
      </node>
      <node id="1605.06069" label="1605.06069">
        <attvalues>
          <attvalue for="0" value="A Hierarchical Latent Variable Encoder-Decoder Model for Generating&#10;  Dialogues" />
          <attvalue for="1" value="  Sequential data often possesses a hierarchical structure with complex&#10;dependencies between subsequences, such as found between the utterances in a&#10;dialogue. In an effort to model this kind of generative process, we propose a&#10;neural network-based generative architecture, with latent stochastic variables&#10;that span a variable number of time steps. We apply the proposed model to the&#10;task of dialogue response generation and compare it with recent neural network&#10;architectures. We evaluate the model performance through automatic evaluation&#10;metrics and by carrying out a human evaluation. The experiments demonstrate&#10;that our model improves upon recently proposed models and that the latent&#10;variables facilitate the generation of long outputs and maintain the context.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.06437" label="1804.06437">
        <attvalues>
          <attvalue for="0" value="Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style&#10;  Transfer" />
          <attvalue for="1" value="  We consider the task of text attribute transfer: transforming a sentence to&#10;alter a specific attribute (e.g., sentiment) while preserving its&#10;attribute-independent content (e.g., changing &quot;screen is just the right size&quot;&#10;to &quot;screen is too small&quot;). Our training data includes only sentences labeled&#10;with their attribute (e.g., positive or negative), but not pairs of sentences&#10;that differ only in their attributes, so we must learn to disentangle&#10;attributes from attribute-independent content in an unsupervised way. Previous&#10;work using adversarial methods has struggled to produce high-quality outputs.&#10;In this paper, we propose simpler methods motivated by the observation that&#10;text attributes are often marked by distinctive phrases (e.g., &quot;too small&quot;).&#10;Our strongest method extracts content words by deleting phrases associated with&#10;the sentence's original attribute value, retrieves new phrases associated with&#10;the target attribute, and uses a neural model to fluently combine these into a&#10;final output. On human evaluation, our best method generates grammatical and&#10;appropriate responses on 22% more inputs than the best previous system,&#10;averaged over three attribute transfer datasets: altering sentiment of reviews&#10;on Yelp, altering sentiment of reviews on Amazon, and altering image captions&#10;to be more romantic or humorous.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.20158" label="2310.20158">
        <attvalues>
          <attvalue for="0" value="GAR-meets-RAG Paradigm for Zero-Shot Information Retrieval" />
          <attvalue for="1" value="  Given a query and a document corpus, the information retrieval (IR) task is&#10;to output a ranked list of relevant documents. Combining large language models&#10;(LLMs) with embedding-based retrieval models, recent work shows promising&#10;results on the zero-shot retrieval problem, i.e., no access to labeled data&#10;from the target domain. Two such popular paradigms are generation-augmented&#10;retrieval or GAR (generate additional context for the query and then retrieve),&#10;and retrieval-augmented generation or RAG (retrieve relevant documents as&#10;context and then generate answers). The success of these paradigms hinges on&#10;(i) high-recall retrieval models, which are difficult to obtain in the&#10;zero-shot setting, and (ii) high-precision (re-)ranking models which typically&#10;need a good initialization. In this work, we propose a novel GAR-meets-RAG&#10;recurrence formulation that overcomes the challenges of existing paradigms. Our&#10;method iteratively improves retrieval (via GAR) and rewrite (via RAG) stages in&#10;the zero-shot setting. A key design principle is that the rewrite-retrieval&#10;stages improve the recall of the system and a final re-ranking stage improves&#10;the precision. We conduct extensive experiments on zero-shot passage retrieval&#10;benchmarks, BEIR and TREC-DL. Our method establishes a new state-of-the-art in&#10;the BEIR benchmark, outperforming previous best results in Recall@100 and&#10;nDCG@10 metrics on 6 out of 8 datasets, with up to 17% relative gains over the&#10;previous best.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Language models have found increasing applications in Information Retrieval (IR) over the past few years. In this section, we highlight recent advances that leverage the in-depth world knowledge of LLMs (implicit in their pre-trained parameters) with retrieval components (both non-parametric and parametric) that have access to external memory, at different stages of IR.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2203.05598" label="2203.05598">
        <attvalues>
          <attvalue for="0" value="A new approach to calculating BERTScore for automatic assessment of&#10;  translation quality" />
          <attvalue for="1" value="  The study of the applicability of the BERTScore metric was conducted to&#10;translation quality assessment at the sentence level for English -&gt; Russian&#10;direction. Experiments were performed with a pre-trained Multilingual BERT as&#10;well as with a pair of Monolingual BERT models. To align monolingual&#10;embeddings, an orthogonal transformation based on anchor tokens was used. It&#10;was demonstrated that such transformation helps to prevent mismatching issue&#10;and shown that this approach gives better results than using embeddings of the&#10;Multilingual model. To improve the token matching process it is proposed to&#10;combine all incomplete WorkPiece tokens into meaningful words and use simple&#10;averaging of corresponding vectors and to calculate BERTScore based on anchor&#10;tokens only. Such modifications allowed us to achieve a better correlation of&#10;the model predictions with human judgments. In addition to evaluating machine&#10;translation, several versions of human translation were evaluated as well, the&#10;problems of this approach were listed.&#10;" />
          <attvalue for="2" value="&#10;&#10;To assess machine translation, which appeared in the middle of the 20th century, a comparison with the reference translation, performed by people, the so-called “gold standard”, was traditionally used. With the improvement of the machine translation quality, and especially with the beginning of the use of deep neural networks since 2014\cite{Kyunghyun_Cho-2014}, it became possible to increase the estimates of translation fluency and adequacy \cite{Andy_Way-2019}. In addition, there have been attempts to identify universal metrics with which you can automatically evaluate the quality of arbitrary translation, including those made by people. &#10;&#10;The principles and methods used to assess the quality of machine translation are described a lot \cite{Lifeng_Han-2021}. Among such methods there are both Human \cite{Markus_Freitagy-2021} and automatic ones. The purpose of the automatic assessment is to obtain some score that is as close as possible to the expert's score but does not require his involvement. &#10;&#10;The existing automatic methods can be divided into 2 groups according to the metrics they use. The first group uses metrics such as: BLUE \cite{Papineni-2002}, METEOR \cite{Satanjeev_Banerjee-2005} and others. All the listed metrics cannot be considered as fully automatic since they require a reference translation. Meanwhile their quality depends on the quality of such translation. The second group, which appeared recently, includes YiSi \cite{Chi_kiu-2019}, BERTScore \cite{Tianyi_Zhang-2019} and its variations. Further in this paper we focused on the study of BERTScore.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Translation Quality Assessment, Machine Translation Evaluation, Artificial Intelligence, Natural Language Processing, Multilingual BERT Models" />
        </attvalues>
      </node>
      <node id="1803.08409" label="1803.08409">
        <attvalues>
          <attvalue for="0" value="Quality expectations of machine translation" />
          <attvalue for="1" value="  Machine Translation (MT) is being deployed for a range of use-cases by&#10;millions of people on a daily basis. There should, therefore, be no doubt as to&#10;the utility of MT. However, not everyone is convinced that MT can be useful,&#10;especially as a productivity enhancer for human translators. In this chapter, I&#10;address this issue, describing how MT is currently deployed, how its output is&#10;evaluated and how this could be enhanced, especially as MT quality itself&#10;improves. Central to these issues is the acceptance that there is no longer a&#10;single 'gold standard' measure of quality, such that the situation in which MT&#10;is deployed needs to be borne in mind, especially with respect to the expected&#10;'shelf-life' of the translation itself.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.03311" label="2105.03311">
        <attvalues>
          <attvalue for="0" value="Translation Quality Assessment: A Brief Survey on Manual and Automatic&#10;  Methods" />
          <attvalue for="1" value="  To facilitate effective translation modeling and translation studies, one of&#10;the crucial questions to address is how to assess translation quality. From the&#10;perspectives of accuracy, reliability, repeatability and cost, translation&#10;quality assessment (TQA) itself is a rich and challenging task. In this work,&#10;we present a high-level and concise survey of TQA methods, including both&#10;manual judgement criteria and automated evaluation metrics, which we classify&#10;into further detailed sub-categories. We hope that this work will be an asset&#10;for both translation model researchers and quality assessment researchers. In&#10;addition, we hope that it will enable practitioners to quickly develop a better&#10;understanding of the conventional TQA field, and to find corresponding closely&#10;relevant evaluation solutions for their own needs. This work may also serve&#10;inspire further development of quality assessment and evaluation methodologies&#10;for other natural language processing (NLP) tasks in addition to machine&#10;translation (MT), such as automatic text summarization (ATS), natural language&#10;understanding (NLU) and natural language generation (NLG).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.08542" label="2201.08542">
        <attvalues>
          <attvalue for="0" value="Can Model Compression Improve NLP Fairness" />
          <attvalue for="1" value="  Model compression techniques are receiving increasing attention; however, the&#10;effect of compression on model fairness is still under explored. This is the&#10;first paper to examine the effect of distillation and pruning on the toxicity&#10;and bias of generative language models. We test Knowledge Distillation and&#10;Pruning methods on the GPT2 model and found a consistent pattern of toxicity&#10;and bias reduction after model distillation; this result can be potentially&#10;interpreted by existing line of research which describes model compression as a&#10;regularization technique; our work not only serves as a reference for safe&#10;deployment of compressed models, but also extends the discussion of&#10;&quot;compression as regularization&quot; into the setting of neural LMs, and hints at&#10;the possibility of using compression to develop fairer models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Compression \quad According the survey paper \cite{Gupta2020CompressionOD}, model compression methods for NLP currently include: pruning(\cite{Michel2019AreSH},\cite{Voita2019AnalyzingMS},\cite{Prasanna2020WhenBP}), quantization\cite{Cheong2019transformersZ}, knowledge distillation(\cite{Jiao2020TinyBERTDB},\cite{Iandola2020SqueezeBERTWC}), parameter sharing(\cite{Lan2020ALBERTAL},\cite{Lan2020ALBERTAL}), tensor decomposition and sub-quadratic complexity transformers. \\&#10;&#10;Fairness \quad Google Brain \cite{Hooker2020CharacterisingBI} tries to characterize compression's impact on fairness for vision models. They tests quantization and pruning techniques and argue that though compressed models achieve similar overall error rate, but fairness is compromised because performance of samples with under-represented features is sacrificed after compression. Researchers from University of Utah \cite{Joseph2020GoingBC} proposes adding fairness into the compression objective function for vision tasks. However, to the best of our knowledge, no prior work has been done studying Knowledge Distillation method, nor are there any compression fairness studies on NLP models.\\&#10;&#10;Compression as regularization \quad &#10;\cite{Fan2020ReducingTD} introduces a compression method for transformers named structured dropout, which is shown to achieve higher performance than distillation and weight pruning. The method assumes that transformer models are over-parametrized and sub-structures of the original model could achieve equivalent performances, plus that smaller networks will enjoy the benefit of regularization. Many studies (\cite{Jordo2021OnTE}, \cite{Bartoldson2020TheGT}) also argue that pruning of Convolutional Neural Networks serves as a way of regularization. \\&#10;&#10;Compression for robust learning \quad&#10;The seminal work of \cite{Papernot2016DistillationAA} introduces Knowledge Distillation as a defense against adversarial perturbations. Following works continue to use Knowledge Distillation to improve generalization \cite{Arani2019ImprovingGA} and robustness (\cite{Goldblum2020AdversariallyRD}). Knowledge Distillation is also used to improve models on privacy protection (\cite{Shejwalkar2019ReconcilingUA}, \cite{Zhao2021KnowledgeDW}). Moreover, pruning can improve model robustness according to the following studies (\cite{Jordo2021OnTE}, \cite{Pang2021BagOT},\cite{Hendrycks2019BenchmarkingNN} ). \cite{Kaya2019ShallowDeepNU} shows that stopping at earlier layers during inference can improve model robustness. The intuition is still that smaller and shallower networks are more robust.\\&#10;&#10;Compression for fairness \quad Our experiments demonstrate monotonic reduction of model toxicity and biases as the model size decreases with distillation. The gold question is whether the regularization and robustness effect of model compression incur the toxicity and bias reduction that we observed in distilled generative language models. If yes, can we also develop techniques to improve NLP fairness using model compression? If not, what is the cause of the monotonic toxicity and bias reduction?&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Compression Techniques, Neural Network Regularization, Mathematics, Language Model Fairness, Statistics" />
        </attvalues>
      </node>
      <node id="2111.12790" label="2111.12790">
        <attvalues>
          <attvalue for="0" value="Temporal Effects on Pre-trained Models for Language Processing Tasks" />
          <attvalue for="1" value="  Keeping the performance of language technologies optimal as time passes is of&#10;great practical interest. We study temporal effects on model performance on&#10;downstream language tasks, establishing a nuanced terminology for such&#10;discussion and identifying factors essential to conduct a robust study. We&#10;present experiments for several tasks in English where the label correctness is&#10;not dependent on time and demonstrate the importance of distinguishing between&#10;temporal model deterioration and temporal domain adaptation for systems using&#10;pre-trained representations. We find that depending on the task, temporal model&#10;deterioration is not necessarily a concern. Temporal domain adaptation however&#10;is beneficial in all cases, with better performance for a given time period&#10;possible when the system is trained on temporally more recent data. Therefore,&#10;we also examine the efficacy of two approaches for temporal domain adaptation&#10;without human annotations on new data. Self-labeling shows consistent&#10;improvement and notably, for named entity recognition, leads to better temporal&#10;adaptation than even human annotations.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models capture properties of language, such as semantics of words and phrases and their typical usage, as well as facts about the world expressed in the language sample on which they were trained. Effective solutions for many language tasks depend, to a varying degree, on the background knowledge encoded in language models. Performance may degrade as language and world-related facts change. In some scenarios, language will change as a result of deploying a system that uses the language to make a prediction, as in spam detection \cite{fawcett2003vivo}. But most change is not driven by such adversarial adaptations: the language expressing sentiment in product reviews \cite{lukes-sogaard-2018-sentiment}, the named entities and the contexts in which they are discussed on social media \cite{fromreide-etal-2014-crowdsourcing,rijhwani-preotiuc-pietro-2020-temporally} and language markers of political ideology \cite{huang-paul-2018-examining} all change over time.&#10;&#10;Whether and how this change impacts the performance of different language technologies is a question of great practical interest.&#10;Yet research on quantifying how model performance changes with time has been sporadic. Moreover, approaches to solving language tasks have evolved rapidly, from bag of words models which rely on a small number of fixed words represented as strings, without underlying meaning, to fixed dense word representations such as word2vec and GloVe \cite{Mikolov2013DistributedRO,pennington-etal-2014-glove} and large contextualized representations of language \cite{peters-etal-2018-deep,devlin-etal-2019-bert} that are trained on task-independent text to provide a backbone representation for word meaning. The swift change in approaches has made it hard to understand how representations and the data used to train them modulate the changes in system performance over time.&#10;&#10;We present experiments (\S \ref{sec:setup} \&amp; \S \ref{sec:metrics}) designed to study temporal effects on downstream language tasks, disentangling worsening model performance due to temporal changes ({temporal model deterioration}) and the benefit from retraining systems on temporally more recent data in order to obtain optimal performance ({temporal domain adaptation}). We present experiments on four tasks for English---named entity recognition, truecasing, sentiment and domain classification. We work only with tasks where the correctness of the label is not influenced by time, unlike other tasks such as open domain question answering where the answer may depend on the time when the question are posed (e.g.\ who is the CEO of X?). For each task, we analyze how the performance of approaches built on pre-trained representations changes over time and how retraining on more recent data influences it (\S \ref{sec:results}). We find that models built on pre-trained representations do not experience temporal deterioration on all tasks. However, temporal domain adaptation is still possible, i.e.\ performance can be further improved by retraining on more recent human labeled data.&#10;&#10;We further find that neural models fine-tuned on the same data but initialized with random vectors for word representation exhibit dramatic temporal deterioration on the same datasets (\S \ref{sec:random}). Models powered by pre-trained language models however are not impacted in the same way. Unlike in any prior work, we study several representations (\S \ref{sec:repr}) including those built using the same architecture and data but different model sizes (\S \ref{sec:size}).&#10;&#10;Even though the pre-training data of several representations overlaps in time with task-specific data and some confounding is possible, two sets of experiments show that it is unlikely (\S \ref{sec:time}).&#10;These results provide strong evidence for model deterioration without pre-training; it also raises questions for future work, on how the (mis)match between task data and pre-training data influences performance, with greater mismatch likely to be more similar to random initialization, resulting in a system more vulnerable to temporal deterioration.&#10;&#10;The central insight from our work is that performance of pre-trained models on downstream tasks where answer correctness is time-independent, does not necessarily deteriorate over time but that the best performance at a given time can be obtained by retraining the system on more recent data. Furthermore, based on the experiments to assess the impact of different components of a model, we provide recommendations for the design of future studies on temporal effects (\S \ref{sec:recommend}). This will make it both easier to conduct future studies and have more robust findings by controlling confounding factors and ignoring others.&#10;&#10;Finally, we present two methods for temporal adaptation that do not require manual labeling over time (\S \ref{sec:temporal_adapt}).&#10;One of the approaches is based on continual pre-training where we modify the typical domain adaptative pre-training with an additional step. The second method relies on self-labeling and is highly effective with consistent improvement across all settings. On one of the datasets, self-labeling is even superior to fine-tuning on new labeled human annotated data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning Optimization, Linguistics, Temporal Domain Adaptation, Artificial Intelligence, Language Model Performance" />
        </attvalues>
      </node>
      <node id="1706.03818" label="1706.03818">
        <attvalues>
          <attvalue for="0" value="Query-by-Example Search with Discriminative Neural Acoustic Word&#10;  Embeddings" />
          <attvalue for="1" value="  Query-by-example search often uses dynamic time warping (DTW) for comparing&#10;queries and proposed matching segments. Recent work has shown that comparing&#10;speech segments by representing them as fixed-dimensional vectors --- acoustic&#10;word embeddings --- and measuring their vector distance (e.g., cosine distance)&#10;can discriminate between words more accurately than DTW-based approaches. We&#10;consider an approach to query-by-example search that embeds both the query and&#10;database segments according to a neural model, followed by nearest-neighbor&#10;search to find the matching segments. Earlier work on embedding-based&#10;query-by-example, using template-based acoustic word embeddings, achieved&#10;competitive performance. We find that our embeddings, based on recurrent neural&#10;networks trained to optimize word discrimination, achieve substantial&#10;improvements in performance and run-time efficiency over the previous&#10;approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.01949" label="1710.01949">
        <attvalues>
          <attvalue for="0" value="Semantic speech retrieval with a visually grounded model of&#10;  untranscribed speech" />
          <attvalue for="1" value="  There is growing interest in models that can learn from unlabelled speech&#10;paired with visual context. This setting is relevant for low-resource speech&#10;processing, robotics, and human language acquisition research. Here we study&#10;how a visually grounded speech model, trained on images of scenes paired with&#10;spoken captions, captures aspects of semantics. We use an external image tagger&#10;to generate soft text labels from images, which serve as targets for a neural&#10;model that maps untranscribed speech to (semantic) keyword labels. We introduce&#10;a newly collected data set of human semantic relevance judgements and an&#10;associated task, semantic speech retrieval, where the goal is to search for&#10;spoken utterances that are semantically relevant to a given text query. Without&#10;seeing any text, the model trained on parallel speech and images achieves a&#10;precision of almost 60% on its top ten semantic retrievals. Compared to a&#10;supervised model trained on transcriptions, our model matches human judgements&#10;better by some measures, especially in retrieving non-verbatim semantic&#10;matches. We perform an extensive analysis of the model and its resulting&#10;representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.06950" label="1606.06950">
        <attvalues>
          <attvalue for="0" value="A segmental framework for fully-unsupervised large-vocabulary speech&#10;  recognition" />
          <attvalue for="1" value="  Zero-resource speech technology is a growing research area that aims to&#10;develop methods for speech processing in the absence of transcriptions,&#10;lexicons, or language modelling text. Early term discovery systems focused on&#10;identifying isolated recurring patterns in a corpus, while more recent&#10;full-coverage systems attempt to completely segment and cluster the audio into&#10;word-like units---effectively performing unsupervised speech recognition. This&#10;article presents the first attempt we are aware of to apply such a system to&#10;large-vocabulary multi-speaker data. Our system uses a Bayesian modelling&#10;framework with segmental word representations: each word segment is represented&#10;as a fixed-dimensional acoustic embedding obtained by mapping the sequence of&#10;feature frames to a single embedding vector. We compare our system on English&#10;and Xitsonga datasets to state-of-the-art baselines, using a variety of&#10;measures including word error rate (obtained by mapping the unsupervised output&#10;to ground truth transcriptions). Very high word error rates are reported---in&#10;the order of 70--80% for speaker-dependent and 80--95% for speaker-independent&#10;systems---highlighting the difficulty of this task. Nevertheless, in terms of&#10;cluster quality and word segmentation metrics, we show that by imposing a&#10;consistent top-down segmentation while also using bottom-up knowledge from&#10;detected syllable boundaries, both single-speaker and multi-speaker versions of&#10;our system outperform a purely bottom-up single-speaker syllable-based&#10;approach. We also show that the discovered clusters can be made less speaker-&#10;and gender-specific by using an unsupervised autoencoder-like feature extractor&#10;to learn better frame-level features (prior to embedding). Our system's&#10;discovered clusters are still less pure than those of unsupervised term&#10;discovery systems, but provide far greater coverage.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.00403" label="1811.00403">
        <attvalues>
          <attvalue for="0" value="Truly unsupervised acoustic word embeddings using weak top-down&#10;  constraints in encoder-decoder models" />
          <attvalue for="1" value="  We investigate unsupervised models that can map a variable-duration speech&#10;segment to a fixed-dimensional representation. In settings where unlabelled&#10;speech is the only available resource, such acoustic word embeddings can form&#10;the basis for &quot;zero-resource&quot; speech search, discovery and indexing systems.&#10;Most existing unsupervised embedding methods still use some supervision, such&#10;as word or phoneme boundaries. Here we propose the encoder-decoder&#10;correspondence autoencoder (EncDec-CAE), which, instead of true word segments,&#10;uses automatically discovered segments: an unsupervised term discovery system&#10;finds pairs of words of the same unknown type, and the EncDec-CAE is trained to&#10;reconstruct one word given the other as input. We compare it to a standard&#10;encoder-decoder autoencoder (AE), a variational AE with a prior over its latent&#10;embedding, and downsampling. EncDec-CAE outperforms its closest competitor by&#10;24% relative in average precision on two languages in a word discrimination&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.07387" label="2012.07387">
        <attvalues>
          <attvalue for="0" value="A comparison of self-supervised speech representations as input features&#10;  for unsupervised acoustic word embeddings" />
          <attvalue for="1" value="  Many speech processing tasks involve measuring the acoustic similarity&#10;between speech segments. Acoustic word embeddings (AWE) allow for efficient&#10;comparisons by mapping speech segments of arbitrary duration to&#10;fixed-dimensional vectors. For zero-resource speech processing, where&#10;unlabelled speech is the only available resource, some of the best AWE&#10;approaches rely on weak top-down constraints in the form of automatically&#10;discovered word-like segments. Rather than learning embeddings at the segment&#10;level, another line of zero-resource research has looked at representation&#10;learning at the short-time frame level. Recent approaches include&#10;self-supervised predictive coding and correspondence autoencoder (CAE) models.&#10;In this paper we consider whether these frame-level features are beneficial&#10;when used as inputs for training to an unsupervised AWE model. We compare&#10;frame-level features from contrastive predictive coding (CPC), autoregressive&#10;predictive coding and a CAE to conventional MFCCs. These are used as inputs to&#10;a recurrent CAE-based AWE model. In a word discrimination task on English and&#10;Xitsonga data, all three representation learning approaches outperform MFCCs,&#10;with CPC consistently showing the biggest improvement. In cross-lingual&#10;experiments we find that CPC features trained on English can also be&#10;transferred to Xitsonga.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.16043" label="2210.16043">
        <attvalues>
          <attvalue for="0" value="Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised&#10;  Speech Models" />
          <attvalue for="1" value="  Given the strong results of self-supervised models on various tasks, there&#10;have been surprisingly few studies exploring self-supervised representations&#10;for acoustic word embeddings (AWE), fixed-dimensional vectors representing&#10;variable-length spoken word segments. In this work, we study several&#10;pre-trained models and pooling methods for constructing AWEs with&#10;self-supervised representations. Owing to the contextualized nature of&#10;self-supervised representations, we hypothesize that simple pooling methods,&#10;such as averaging, might already be useful for constructing AWEs. When&#10;evaluating on a standard word discrimination task, we find that HuBERT&#10;representations with mean-pooling rival the state of the art on English AWEs.&#10;More surprisingly, despite being trained only on English, HuBERT&#10;representations evaluated on Xitsonga, Mandarin, and French consistently&#10;outperform the multilingual model XLSR-53 (as well as Wav2Vec 2.0 trained on&#10;English).&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech tasks such as query-by-example, voice search, keyword spotting, and word discovery typically require measuring distances between speech segments \cite{zhang2009unsupervised,barakat2011keyword,de2007template}. &#10;To avoid the computational expense of the traditional Dynamic Time Warping method, recent papers often use Acoustic word embeddings (AWEs), which represent variable-length segments as fixed-dimensional vectors \cite{maas2012word, levin2013fixed}. These can then be compared quickly using measures such as cosine similarity.&#10;&#10;An effective AWE algorithm will embed different instances of the same word close together in the vector space, and instances of distinct words further away. One of the main challenges is&#10;how to encode the sequential information from the speech signal into a vector space that has no inherent sequential structure. &#10;The representation should encode not just which phones are present, but their ordering---so that words like {task}, {stack}, {cast}, and {cats}, which all contain the same set of phones, will have distinct clusters in the representational space. Various approaches have been developed that use supervision from known word pairs \cite{kamper2016deep,hu2020multilingual,settle2016discriminative,settle2017query}, but here we focus on unsupervised learning of AWEs, where only raw audio is available \cite{holzenberger2018learning,kamper2019truly,van2021comparison}. This scenario is potentially important for speech applications in low-resource languages.&#10;&#10;A common baseline that preserves sequential order while extracting a fixed-dimensional representation is subsampling (see, e.g., \cite{van2021comparison,kamper2017segmental,kamper2017embedded}): selecting a fixed number of (usually equally spaced) frames and concatenating them. Subsampling is simple and fast to compute, but, depending on the input frame size and number of samples, it can lead to prohibitively large embeddings and/or loss of phonetic information.&#10;Moreover, it does not perform as well on word discrimination tasks as newer learning-based approaches. Unsupervised learning-based methods typically work in two steps: first, apply an unsupervised term detection (UTD) system \cite{jansen2011efficient} to identify similar pairs of segments that are likely to be the same word or phrase, then use the pairs as a noisy set of positive examples to train a neural network. Network architectures vary, but the basic idea is to train the system's representations to make the positive examples closer together in the space \cite{kamper2019truly,van2021comparison} (and in some models, also to separate additional negative example pairs \cite{jacobs2021acoustic,robin2022speech}).&#10;&#10;Though effective, this learning-based approach relies on running UTD on the target language, which itself is computationally intensive and&#10;sensitive to differences in input features \cite{van2021comparison}. Here, we explore whether using newer self-supervised speech representations, available as pre-trained models \cite{hsu2021hubert,baevski2020wav2vec}, may obviate &#10;both the UTD step and the need for specialized models to learn unsupervised AWEs. &#10;We hypothesize that the contextualized speech representations learned by these models will implicitly encode the sequential information needed for AWEs (e.g., by capturing within each frame the local acoustic effects of coarticulation, and/or information at a longer timescale that is needed to reconstruct the masked input during pretraining---where average mask span is nearly 300ms \cite{baevski2020wav2vec}).&#10;If so, then it should be possible to create effective AWEs with much smaller dimension than subsampling just by using simple pooling methods such as mean- or max-pooling. While pre-trained models and these pooling methods are widely used across many applications, as far as we know this paper is the first to compare and analyze them for creating AWEs.&#10;&#10;We evaluate AWEs created using different pooling operations on the representations from two English pre-trained models---HuBERT (HB) and wav2vec 2.0 (W2V2)---and one multilingual pre-trained model (XLSR-53, which also uses the W2V2\ architecture). Using a standard word discrimination task, we test on English, Xitsonga, Mandarin, and French---where the latter three better represent a low-resource target language scenario, where a large pre-trained model on that language is unlikely to be available. &#10;&#10;In accordance with our hypothesis, we find that on our English test set, AWEs created by mean-pooling the HB\ representations perform almost as well as the state-of-the-art learned pooling model (MCVAE \cite{peng_correspondence_2020}) with equivalent dimensionality; and outperform subsampling, despite having a much lower dimensionality. Mean-pooled W2V2\ representations underperform HB, but are still better than subsampled ones, and considerably better than the MFCC baseline.&#10;&#10;Our experiments on other languages show that (1) the multilingual W2V2\ representations work better than the monolingual English ones, but still underperform HB\ (for which only an English model is available); (2) unlike on English, mean-pooling does not outperform subsampling with the HB\ representations, although it comes close; and (3) when equated on dimensionality, the HB\ representations are not quite as good as those learned by the best recent models, but perform surprisingly well given that (unlike these models) they require no training on the target language at all.&#10;&#10;Overall, our results indicate that the right self-supervised model capture some sequential information needed for AWEs, making simpler pooling methods effective. While these contextualized representations don't generalize fully to other languages, they still work well with no training required.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.07295" label="2301.07295">
        <attvalues>
          <attvalue for="0" value="Adapting Multilingual Speech Representation Model for a New,&#10;  Underresourced Language through Multilingual Fine-tuning and Continued&#10;  Pretraining" />
          <attvalue for="1" value="  In recent years, neural models learned through self-supervised pretraining on&#10;large scale multilingual text or speech data have exhibited promising results&#10;for underresourced languages, especially when a relatively large amount of data&#10;from related language(s) is available. While the technology has a potential for&#10;facilitating tasks carried out in language documentation projects, such as&#10;speech transcription, pretraining a multilingual model from scratch for every&#10;new language would be highly impractical. We investigate the possibility for&#10;adapting an existing multilingual wav2vec 2.0 model for a new language,&#10;focusing on actual fieldwork data from a critically endangered tongue: Ainu.&#10;Specifically, we (i) examine the feasibility of leveraging data from similar&#10;languages also in fine-tuning; (ii) verify whether the model's performance can&#10;be improved by further pretraining on target language data. Our results show&#10;that continued pretraining is the most effective method to adapt a wav2vec 2.0&#10;model for a new language and leads to considerable reduction in error rates.&#10;Furthermore, we find that if a model pretrained on a related speech variety or&#10;an unrelated language with similar phonological characteristics is available,&#10;multilingual fine-tuning using additional data from that language can have&#10;positive impact on speech recognition performance when there is very little&#10;labeled data in the target language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.01393" label="2306.01393">
        <attvalues>
          <attvalue for="0" value="Assessing the Importance of Frequency versus Compositionality for&#10;  Subword-based Tokenization in NMT" />
          <attvalue for="1" value="  Subword tokenization is the de facto standard for tokenization in neural&#10;language models and machine translation systems. Three advantages are&#10;frequently cited in favor of subwords: shorter encoding of frequent tokens,&#10;compositionality of subwords, and ability to deal with unknown words. As their&#10;relative importance is not entirely clear yet, we propose a tokenization&#10;approach that enables us to separate frequency (the first advantage) from&#10;compositionality. The approach uses Huffman coding to tokenize words, by order&#10;of frequency, using a fixed amount of symbols. Experiments with CS-DE, EN-FR&#10;and EN-DE NMT show that frequency alone accounts for 90%-95% of the scores&#10;reached by BPE, hence compositionality has less importance than previously&#10;thought.&#10;" />
          <attvalue for="2" value="&#10;&#10;Tokenization into subwords has become an unchallenged standard used in virtually all NMT systems and language models. Since the proposal by Sennrich et al.~\shortcite{sennrich-etal-2016-neural} to use Byte-Pair Encoding (BPE)~\cite{Gage1994ANA} to create subword vocabularies, followed by the use of a unigram language model and the SentencePiece implementation \cite{kudo-2018-subword}, no alternative models have taken over. While subwords have been empirically demonstrated to outperform character and word-level tokenization~\cite{sennrich-etal-2016-neural,Wu2016GooglesNM,denkowski-neubig-2017-stronger}, the factors contributing to their success have not been fully understood yet. Some studies have investigated the performance of subwords with regard to compression~\cite{galle-2019-investigating,libovicky-etal-2022-dont}, suggesting that better compression may be associated with improved performance. However, other factors such as compositionality have yet to be thoroughly explored.&#10;&#10;In this paper, we use an alternative algorithm for creating subword vocabularies, which retains only one of the features that have been invoked to explain the effectiveness of BPE, namely the fact that frequent words are encoded as unique subwords, while less frequent ones are encoded using several subwords, possibly up to the character level. The algorithm is based on Huffman coding~\cite{huffman1952method}, a different text compression method than the one used by BPE. The algorithm differs from BPE in two key aspects: while certain BPE subwords convey compositional linguistic properties (e.g., meaning or morphology), Huffman coding is fundamentally non-compositional, and cannot tokenize words not seen during training. When using Huffman coding to tokenize data for Transformer-based MT, we reach scores that are within 10-12\% of those obtained using BPE when measured by BLEU and within 4-8\% when measured by COMET, for vocabulary sizes of 32k symbols. This demonstrates that the main factor accounting for the success of BPEs is word frequency, and not subword compositionality. &#10;Our main contributions are:&#10;\begin{enumerate} &#10; \item We show how to build subword vocabularies for tokenization using Huffman coding.&#10; \item We study the impact of this method on NMT by varying a range of parameters, in particular the vocabulary size.&#10; \item Observing that the scores obtained using Huffman coding are close to those obtained using BPE, and arguing that the former method retains only the frequential aspect of BPE, we conclude that frequency is the main reason for the effectiveness of BPE.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Machine Translation Systems, Mathematics, Neural Language Models, Tokenization Techniques" />
        </attvalues>
      </node>
      <node id="1706.09733" label="1706.09733">
        <attvalues>
          <attvalue for="0" value="Stronger Baselines for Trustable Results in Neural Machine Translation" />
          <attvalue for="1" value="  Interest in neural machine translation has grown rapidly as its effectiveness&#10;has been demonstrated across language and data scenarios. New research&#10;regularly introduces architectural and algorithmic improvements that lead to&#10;significant gains over &quot;vanilla&quot; NMT implementations. However, these new&#10;techniques are rarely evaluated in the context of previously published&#10;techniques, specifically those that are widely used in state-of-theart&#10;production and shared-task systems. As a result, it is often difficult to&#10;determine whether improvements from research will carry over to systems&#10;deployed for real-world use. In this work, we recommend three specific methods&#10;that are relatively easy to implement and result in much stronger experimental&#10;systems. Beyond reporting significantly higher BLEU scores, we conduct an&#10;in-depth analysis of where improvements originate and what inherent weaknesses&#10;of basic NMT models are being addressed. We then compare the relative gains&#10;afforded by several other techniques proposed in the literature when starting&#10;with vanilla systems versus our stronger baselines, showing that experimental&#10;conclusions may change depending on the baseline chosen. This indicates that&#10;choosing a strong baseline is crucial for reporting reliable experimental&#10;results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.11641" label="1911.11641">
        <attvalues>
          <attvalue for="0" value="PIQA: Reasoning about Physical Commonsense in Natural Language" />
          <attvalue for="1" value="  To apply eyeshadow without a brush, should I use a cotton swab or a&#10;toothpick? Questions requiring this kind of physical commonsense pose a&#10;challenge to today's natural language understanding systems. While recent&#10;pretrained models (such as BERT) have made progress on question answering over&#10;more abstract domains - such as news articles and encyclopedia entries, where&#10;text is plentiful - in more physical domains, text is inherently limited due to&#10;reporting bias. Can AI systems learn to reliably answer physical common-sense&#10;questions without experiencing the physical world? In this paper, we introduce&#10;the task of physical commonsense reasoning and a corresponding benchmark&#10;dataset Physical Interaction: Question Answering or PIQA. Though humans find&#10;the dataset easy (95% accuracy), large pretrained models struggle (77%). We&#10;provide analysis about the dimensions of knowledge that existing models lack,&#10;which offers significant opportunities for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.13193" label="2407.13193">
        <attvalues>
          <attvalue for="0" value="Retrieval-Augmented Generation for Natural Language Processing: A Survey" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated great success in various&#10;fields, benefiting from their huge amount of parameters that store knowledge.&#10;However, LLMs still suffer from several key issues, such as hallucination&#10;problems, knowledge update issues, and lacking domain-specific expertise. The&#10;appearance of retrieval-augmented generation (RAG), which leverages an external&#10;knowledge database to augment LLMs, makes up those drawbacks of LLMs. This&#10;paper reviews all significant techniques of RAG, especially in the retriever&#10;and the retrieval fusions. Besides, tutorial codes are provided for&#10;implementing the representative techniques in RAG. This paper further discusses&#10;the RAG training, including RAG with/without datastore update. Then, we&#10;introduce the application of RAG in representative natural language processing&#10;tasks and industrial scenarios. Finally, this paper discusses the future&#10;directions and challenges of RAG for promoting its development.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs)~\cite{llama, gemma, mistral, gpt4, 23iclr-glm} have achieved significant advancements in recent years and have become the cornerstone of various applications in the field of natural language processing (NLP).&#10;These LLMs are typically pre-trained on a large amount of natural language corpus and then fine-tuned on the specific downstream tasks' datasets.&#10;Recent works~\cite{19emnlp-lm-as-kb, 22arxiv-lmkb-survey, 22nips-rome, 24arxiv-lmkb-scale} demonstrate the success of LLMs can be explained by the fact that language models act as knowledge bases, which refers to implicitly storing the knowledge learned from training datasets in the parameters as internal memory and generating responses by retrieving answers from memory.&#10;To store more knowledge for better generation performance, existing works generally enlarge the memory capacity by increasing the volume of parameters~\cite{22iclr-scaling-law, gpt3, 20arxiv-scaling-law, 22arxiv-scaling-law}.&#10;&#10;Although existing LLMs have shown great power, there are still several challenges hindering the development of LLMs.&#10;One of the most prominent challenges is the hallucination problem~\cite{23cs-hallucination-survey, 23acl-hallucination-mt, 23acl-hallucination-dialogue}, which refers to the tendency of LLMs to generate responses that are coherent and fluent but factually incorrect.&#10;Another big challenge is the knowledge update issue.&#10;To update the knowledge stored in the LLMs' internal memory~\cite{22nips-rome, 23arxiv-editing-survey, 24arxiv-editing-survey}, it is necessary to retrain/fine-tune LLMs with new data, which is a costly process.&#10;Another challenge for general LLMs is lacking of domain-specific expertise~\cite{23emnlp-huatuogpt, 23nature-med-palm, 23arxiv-med-palm2, 24arxiv-saulm}. &#10;Training a domain-specific LLM, however, demands considerable manpower for dataset collection.&#10;&#10;To address these challenges, recent works~\cite{20neurips-rag, 22icml-retro, 20icml-realm} have proposed leveraging an external knowledge database to augment LLMs, known as retrieval-augmented generation (RAG).&#10;By supplying LLMs with retrieved relevant factual information, the hallucination problem can be alleviated to some extent.&#10;Besides, the knowledge update issue can also be addressed by updating the external knowledge database, which can augment LLMs with up-to-date knowledge. &#10;RAG can also convert a general LLM into a domain-specific LLM by constructing and utilizing a domain-specific knowledge database.&#10;Therefore, RAG plays an important role in augmenting the functionality of LLMs, making them more accurate, knowledgeable, and reliable in a wide range of applications.&#10;&#10;Contributions: &#10;This paper reviews all techniques involved in RAG for natural language processing.&#10;Although there are several survey papers for RAG~\cite{22arxiv-rag-survey, 23arxiv-survey, 24arxiv-survey-ecust, 24arxiv-survey-pku, 24arxiv-rag-benchmark}, our survey still has some key insights,&#10;\begin{enumerate}&#10; \item This paper systematically introduces each component of RAG, including details about the retriever from building to querying, and techniques of the retrieval fusions with tutorial codes.&#10; \item This paper exhibits different RAG training strategies, including RAG with/without datastore update.&#10; \item This paper further discusses the applications of RAG on downstream NLP tasks and practical NLP scenarios.&#10; \item This paper finally identifies promising future directions for exploring and main challenges for addressing.&#10;\end{enumerate}&#10;&#10;The remainder of this paper is organized as follows. &#10;Section~\ref{sec:para} gives an overview of RAG.&#10;Section~\ref{sec:retriever} and Section~\ref{sec:fusion} comprehensively introduce all technical details used in retrievers and retrieval fusions.&#10;Section~\ref{sec:training} presents how to train the RAG with/without new knowledge.&#10;Section~\ref{sec:task} presents the techniques used in representative NLP tasks.&#10;Section~\ref{sec:app} shows the applications of RAG in practical NLP scenarios.&#10;Section~\ref{sec:future} discusses the future directions of RAG.&#10;Section~\ref{sec:con} makes a final conclusion of this paper.&#10; \input{sections/2-Paradigm}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Retrieval-Augmented Generation, Computer Science, Linguistics, Language Model Limitations, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2110.02095" label="2110.02095">
        <attvalues>
          <attvalue for="0" value="Exploring the Limits of Large Scale Pre-training" />
          <attvalue for="1" value="  Recent developments in large-scale machine learning suggest that by scaling&#10;up data, model size and training time properly, one might observe that&#10;improvements in pre-training would transfer favorably to most downstream tasks.&#10;In this work, we systematically study this phenomena and establish that, as we&#10;increase the upstream accuracy, the performance of downstream tasks saturates.&#10;In particular, we investigate more than 4800 experiments on Vision&#10;Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten&#10;million to ten billion, trained on the largest scale of available image data&#10;(JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition&#10;tasks. We propose a model for downstream performance that reflects the&#10;saturation phenomena and captures the nonlinear relationship in performance of&#10;upstream and downstream tasks. Delving deeper to understand the reasons that&#10;give rise to these phenomena, we show that the saturation behavior we observe&#10;is closely related to the way that representations evolve through the layers of&#10;the models. We showcase an even more extreme scenario where performance on&#10;upstream and downstream are at odds with each other. That is, to have a better&#10;downstream performance, we need to hurt upstream accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15075" label="2305.15075">
        <attvalues>
          <attvalue for="0" value="HuatuoGPT, towards Taming Language Model to Be a Doctor" />
          <attvalue for="1" value="  In this paper, we present HuatuoGPT, a large language model (LLM) for medical&#10;consultation. The core recipe of HuatuoGPT is to leverage both&#10;\textit{distilled data from ChatGPT} and \textit{real-world data from doctors}&#10;in the supervised fine-tuned stage. The responses of ChatGPT are usually&#10;detailed, well-presented and informative while it cannot perform like a doctor&#10;in many aspects, e.g. for integrative diagnosis. We argue that real-world data&#10;from doctors would be complementary to distilled data in the sense the former&#10;could tame a distilled language model to perform like doctors. To better&#10;leverage the strengths of both data, we train a reward model to align the&#10;language model with the merits that both data bring, following an RLAIF&#10;(reinforced learning from AI feedback) fashion. To evaluate and benchmark the&#10;models, we propose a comprehensive evaluation scheme (including automatic and&#10;manual metrics). Experimental results demonstrate that HuatuoGPT achieves&#10;state-of-the-art results in performing medical consultation among open-source&#10;LLMs in GPT-4 evaluation, human evaluation, and medical benchmark datasets. It&#10;is worth noting that by using additional real-world data and RLAIF, the&#10;distilled language model (i.e., HuatuoGPT) outperforms its teacher model&#10;ChatGPT in most cases. Our code, data, and models are publicly available at&#10;\url{https://github.com/FreedomIntelligence/HuatuoGPT}. The online demo is&#10;available at \url{https://www.HuatuoGPT.cn/}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.01110" label="2202.01110">
        <attvalues>
          <attvalue for="0" value="A Survey on Retrieval-Augmented Text Generation" />
          <attvalue for="1" value="  Recently, retrieval-augmented text generation attracted increasing attention&#10;of the computational linguistics community. Compared with conventional&#10;generation models, retrieval-augmented text generation has remarkable&#10;advantages and particularly has achieved state-of-the-art performance in many&#10;NLP tasks. This paper aims to conduct a survey about retrieval-augmented text&#10;generation. It firstly highlights the generic paradigm of retrieval-augmented&#10;generation, and then it reviews notable approaches according to different tasks&#10;including dialogue response generation, machine translation, and other&#10;generation tasks. Finally, it points out some important directions on top of&#10;recent methods to facilitate future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.02324" label="2005.02324">
        <attvalues>
          <attvalue for="0" value="Neural CRF Model for Sentence Alignment in Text Simplification" />
          <attvalue for="1" value="  The success of a text simplification system heavily depends on the quality&#10;and quantity of complex-simple sentence pairs in the training corpus, which are&#10;extracted by aligning sentences between parallel articles. To evaluate and&#10;improve sentence alignment quality, we create two manually annotated&#10;sentence-aligned datasets from two commonly used text simplification corpora,&#10;Newsela and Wikipedia. We propose a novel neural CRF alignment model which not&#10;only leverages the sequential nature of sentences in parallel documents but&#10;also utilizes a neural sentence pair model to capture semantic similarity.&#10;Experiments demonstrate that our proposed approach outperforms all the previous&#10;work on monolingual sentence alignment task by more than 5 points in F1. We&#10;apply our CRF aligner to construct two new text simplification datasets,&#10;Newsela-Auto and Wiki-Auto, which are much larger and of better quality&#10;compared to the existing datasets. A Transformer-based seq2seq model trained on&#10;our datasets establishes a new state-of-the-art for text simplification in both&#10;automatic and human evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10; Text simplification is considered as a text-to-text generation task where the system learns how to simplify from complex-simple sentence pairs. There is a long line of research using methods based on hand-crafted rules \cite{siddharthan2006syntactic,niklaus-etal-2019-transforming}, statistical machine translation \cite{narayan-gardent-2014-hybrid,xu-etal-2016-optimizing, wubben-etal-2012-sentence}, or neural seq2seq models \cite{zhang-lapata-2017-sentence, zhao-etal-2018-integrating, nisioi-etal-2017-exploring}. As the existing datasets were built using lexical similarity metrics, they frequently omit paraphrases and sentence splits. While training on such datasets creates conservative systems that rarely paraphrase, evaluation on these datasets exhibits an unfair preference for deletion-based simplification over paraphrasing. &#10;&#10;Sentence alignment has been widely used to extract complex-simple sentence pairs from parallel articles for training text simplification systems. Previous work used surface-level similarity metrics, such as TF-IDF cosine similarity \cite{zhu-etal-2010-monolingual,woodsend-lapata-2011-learning, coster-kauchak-2011-simple, paetzold-etal-2017-massalign}, Jaccard-similarity \cite{Xu-EtAl:2015:TACL}, and other lexical features \cite{hwang-EtAl:2015:NAACL-HLT,STAJNER18.630}. Then, a greedy \cite{STAJNER18.630} or dynamic programming \cite{barzilay-elhadad-2003-sentence,paetzold-etal-2017-massalign} algorithm was used to search for the optimal alignment. Another related line of research \cite{smith-etal-2010-extracting, tufis-etal-2013-wikipedia, tsai-roth-2016-cross, gottschalk-acm-2017-multiwiki, aghaebrahimian2018deep,thompson-koehn-2019-vecalign} aligns parallel sentences in bilingual corpora for machine translation. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Computational Linguistics, Text Simplification, Artificial Intelligence, Sentence Alignment, Natural Language Processing, Neural Modeling" />
        </attvalues>
      </node>
      <node id="2005.08575" label="2005.08575">
        <attvalues>
          <attvalue for="0" value="Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio&#10;  Representation" />
          <attvalue for="1" value="  For self-supervised speech processing, it is crucial to use pretrained models&#10;as speech representation extractors. In recent works, increasing the size of&#10;the model has been utilized in acoustic model training in order to achieve&#10;better performance. In this paper, we propose Audio ALBERT, a lite version of&#10;the self-supervised speech representation model. We use the representations&#10;with two downstream tasks, speaker identification, and phoneme classification.&#10;We show that Audio ALBERT is capable of achieving competitive performance with&#10;those huge models in the downstream tasks while utilizing 91\% fewer&#10;parameters. Moreover, we use some simple probing models to measure how much the&#10;information of the speaker and phoneme is encoded in latent representations. In&#10;probing experiments, we find that the latent representations encode richer&#10;information of both phoneme and speaker than that of the last layer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Audio Model Optimization, Artificial Intelligence, Signal Processing, Speech Representation Models, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="1905.10044" label="1905.10044">
        <attvalues>
          <attvalue for="0" value="BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions" />
          <attvalue for="1" value="  In this paper we study yes/no questions that are naturally occurring ---&#10;meaning that they are generated in unprompted and unconstrained settings. We&#10;build a reading comprehension dataset, BoolQ, of such questions, and show that&#10;they are unexpectedly challenging. They often query for complex, non-factoid&#10;information, and require difficult entailment-like inference to solve. We also&#10;explore the effectiveness of a range of transfer learning baselines. We find&#10;that transferring from entailment data is more effective than transferring from&#10;paraphrase or extractive QA data, and that it, surprisingly, continues to be&#10;very beneficial even when starting from massive pre-trained language models&#10;such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on&#10;our train set. It achieves 80.4% accuracy compared to 90% accuracy of human&#10;annotators (and 62% majority-baseline), leaving a significant gap for future&#10;work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.03002" label="2312.03002">
        <attvalues>
          <attvalue for="0" value="The mechanistic basis of data dependence and abrupt learning in an&#10;  in-context classification task" />
          <attvalue for="1" value="  Transformer models exhibit in-context learning: the ability to accurately&#10;predict the response to a novel query based on illustrative examples in the&#10;input sequence. In-context learning contrasts with traditional in-weights&#10;learning of query-output relationships. What aspects of the training data&#10;distribution and architecture favor in-context vs in-weights learning? Recent&#10;work has shown that specific distributional properties inherent in language,&#10;such as burstiness, large dictionaries and skewed rank-frequency distributions,&#10;control the trade-off or simultaneous appearance of these two forms of&#10;learning. We first show that these results are recapitulated in a minimal&#10;attention-only network trained on a simplified dataset. In-context learning&#10;(ICL) is driven by the abrupt emergence of an induction head, which&#10;subsequently competes with in-weights learning. By identifying progress&#10;measures that precede in-context learning and targeted experiments, we&#10;construct a two-parameter model of an induction head which emulates the full&#10;data distributional dependencies displayed by the attention-based network. A&#10;phenomenological model of induction head formation traces its abrupt emergence&#10;to the sequential learning of three nested logits enabled by an intrinsic&#10;curriculum. We propose that the sharp transitions in attention-based networks&#10;arise due to a specific chain of multi-layer operations necessary to achieve&#10;ICL, which is implemented by nested nonlinearities sequentially learned during&#10;training.&#10;" />
          <attvalue for="2" value="&#10;&#10;A striking feature of large language models is in-context learning \cite{brown2020language, dong2022survey, garg2022can, dai2022can}. In-context learning (ICL) is the ability to predict the response to a query based on illustrative examples presented in the context, without any additional weight updates. This form of learning contrasts with in-weights learning (IWL) of query-response relationships encoded in the weights of the network. ICL emerges in transformer models \cite{vaswani2017attention} trained on a diverse set of tasks that contain a common structural element. ICL can be exploited to perform zero-shot learning on novel tasks that share this structure. For example, a transformer trained to solve numerous linear regression tasks learns to solve a new linear regression task based on in-context examples \cite{garg2022can, akyurek2022learning,von2023transformers, ahn2023transformers}. Specifically, given a sequence of sample input-output pairs, the predictive error on a target query is comparable to an optimal Bayes predictor \cite{ahuja2023context, xie2021explanation, li2023transformers}. This remarkable feature extends to other generative models such as hierarchical regression models that involve model selection \cite{bai2023transformers}, random permutations of images \cite{kirsch2022general} and mixture models over sequential data \cite{wang2023large, xie2021explanation}. &#10;&#10;Transformer models trained on language data exhibit another simple yet powerful form of in-context learning. Given a sequence $\dots x, y, \dots, x,?$ for $x, y$ pairs unseen during training (for example, tokens belonging to a novel proper noun), these models learn the ability to predict $y$ \cite{olsson2022context}. In other words, the model learns empirical bigram statistics on-the-fly, thus displaying a primitive form of zero-shot associative learning. Past work has shown that this computation involves an induction head (discussed in detail further below) and that a minimal implementation requires a two-layer attention-only network \cite{olsson2022context}. Across networks of different scales and task structures, the ability to perform ICL often increases abruptly during training \cite{olsson2022context}. The mechanistic basis of the abrupt transition remains unclear. Notably, this abrupt transition is often preceded by the formation of induction heads in intermediate layers of the network, suggesting that induction head formation may provide a scaffold for the development of more complex in-context computations. Other work provides empirical evidence that ICL is the key driver behind the emergent abilities of large language models \cite{lu2023emergent}. Thus, elucidating the mechanisms that underpin ICL, and induction heads in particular, may provide crucial insights into the data distributional and architectural factors that lead to emergent zero-shot learning. &#10;&#10;A recent empirical study has highlighted key data distributional properties pertinent to language that promote ICL in a hybrid in-context/in-weights classification task \cite{chan2022data}. In this setup, a 12-layer transformer network is trained to predict the class label of a target item given a sequence of $N$ item-label pairs in the context. The item classes are drawn from Omniglot \cite{lake2019omniglot}, a standard image-label dataset. By manipulating the distribution of classes shown during training, various data distributional properties that influence the ICL vs IWL trade-off were identified. This setup offers a well-controlled paradigm for identifying the factors that enable attention-based models to learn in-context learning solutions without explicitly trained to do so. &#10;&#10;Our main contributions are as follows. We first show that the data dependencies highlighted in \cite{chan2022data} are recapitulated in a task with simplified input statistics and a two-layer attention-only network architecture. By identifying progress measures and designing careful experiments, we show that ICL is driven by the abrupt formation of an induction head. We construct a minimal two-parameter model of an induction head stacked with a deep classifier, which reproduces all data distributional dependencies and captures the dynamics of learning. Finally, we develop a phenomenological model of an induction head's loss landscape. This analysis enables us to trace the abrupt learning phenomenon to cliffs in the landscape created by nested nonlinearities in a multi-layer attention-based network. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Cognitive Science, In-Context Learning, Artificial Intelligence, Mathematics, Language Modeling" />
        </attvalues>
      </node>
      <node id="2306.04891" label="2306.04891">
        <attvalues>
          <attvalue for="0" value="In-Context Learning through the Bayesian Prism" />
          <attvalue for="1" value="  In-context learning (ICL) is one of the surprising and useful features of&#10;large language models and subject of intense research. Recently, stylized&#10;meta-learning-like ICL setups have been devised that train transformers on&#10;sequences of input-output pairs $(x, f(x))$. The function $f$ comes from a&#10;function class and generalization is checked by evaluating on sequences&#10;generated from unseen functions from the same class. One of the main&#10;discoveries in this line of research has been that for several function&#10;classes, such as linear regression, transformers successfully generalize to new&#10;functions in the class. However, the inductive biases of these models resulting&#10;in this behavior are not clearly understood. A model with unlimited training&#10;data and compute is a Bayesian predictor: it learns the pretraining&#10;distribution. In this paper we empirically examine how far this Bayesian&#10;perspective can help us understand ICL. To this end, we generalize the previous&#10;meta-ICL setup to hierarchical meta-ICL setup which involve unions of multiple&#10;task families. We instantiate this setup on a diverse range of linear and&#10;nonlinear function families and find that transformers can do ICL in this&#10;setting as well. Where Bayesian inference is tractable, we find evidence that&#10;high-capacity transformers mimic the Bayesian predictor. The Bayesian&#10;perspective provides insights into the inductive bias of ICL and how&#10;transformers perform a particular task when they are trained on multiple tasks.&#10;We also find that transformers can learn to generalize to new function classes&#10;that were not seen during pretraining. This involves deviation from the&#10;Bayesian predictor. We examine these deviations in more depth offering new&#10;insights and hypotheses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.04637" label="2306.04637">
        <attvalues>
          <attvalue for="0" value="Transformers as Statisticians: Provable In-Context Learning with&#10;  In-Context Algorithm Selection" />
          <attvalue for="1" value="  Neural sequence models based on the transformer architecture have&#10;demonstrated remarkable \emph{in-context learning} (ICL) abilities, where they&#10;can perform new tasks when prompted with training and test examples, without&#10;any parameter update to the model. This work first provides a comprehensive&#10;statistical theory for transformers to perform ICL. Concretely, we show that&#10;transformers can implement a broad class of standard machine learning&#10;algorithms in context, such as least squares, ridge regression, Lasso, learning&#10;generalized linear models, and gradient descent on two-layer neural networks,&#10;with near-optimal predictive power on various in-context data distributions.&#10;Using an efficient implementation of in-context gradient descent as the&#10;underlying mechanism, our transformer constructions admit mild size bounds, and&#10;can be learned with polynomially many pretraining sequences.&#10;  Building on these ``base'' ICL algorithms, intriguingly, we show that&#10;transformers can implement more complex ICL procedures involving&#10;\emph{in-context algorithm selection}, akin to what a statistician can do in&#10;real life -- A \emph{single} transformer can adaptively select different base&#10;ICL algorithms -- or even perform qualitatively different tasks -- on different&#10;input sequences, without any explicit prompting of the right algorithm or task.&#10;We both establish this in theory by explicit constructions, and also observe&#10;this phenomenon experimentally. In theory, we construct two general mechanisms&#10;for algorithm selection with concrete examples: pre-ICL testing, and post-ICL&#10;validation. As an example, we use the post-ICL validation mechanism to&#10;construct a transformer that can perform nearly Bayes-optimal ICL on a&#10;challenging task -- noisy linear models with mixed noise levels.&#10;Experimentally, we demonstrate the strong in-context algorithm selection&#10;capabilities of standard transformer architectures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.01809" label="2309.01809">
        <attvalues>
          <attvalue for="0" value="Are Emergent Abilities in Large Language Models just In-Context&#10;  Learning?" />
          <attvalue for="1" value="  Large language models, comprising billions of parameters and pre-trained on&#10;extensive web-scale corpora, have been claimed to acquire certain capabilities&#10;without having been specifically trained on them. These capabilities, referred&#10;to as &quot;emergent abilities,&quot; have been a driving force in discussions regarding&#10;the potentials and risks of language models. A key challenge in evaluating&#10;emergent abilities is that they are confounded by model competencies that arise&#10;through alternative prompting techniques, including in-context learning, which&#10;is the ability of models to complete a task based on a few examples. We present&#10;a novel theory that explains emergent abilities, taking into account their&#10;potential confounding factors, and rigorously substantiate this theory through&#10;over 1000 experiments. Our findings suggest that purported emergent abilities&#10;are not truly emergent, but result from a combination of in-context learning,&#10;model memory, and linguistic knowledge. Our work is a foundational step in&#10;explaining language model performance, providing a template for their efficient&#10;use and clarifying the paradox of their ability to excel in some instances&#10;while faltering in others. Thus, we demonstrate that their capabilities should&#10;not be overestimated.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.01171" label="2406.01171">
        <attvalues>
          <attvalue for="0" value="Two Tales of Persona in LLMs: A Survey of Role-Playing and&#10;  Personalization" />
          <attvalue for="1" value="  The concept of persona, originally adopted in dialogue literature, has&#10;re-surged as a promising framework for tailoring large language models (LLMs)&#10;to specific context (e.g., personalized search, LLM-as-a-judge). However, the&#10;growing research on leveraging persona in LLMs is relatively disorganized and&#10;lacks a systematic taxonomy. To close the gap, we present a comprehensive&#10;survey to categorize the current state of the field. We identify two lines of&#10;research, namely (1) LLM Role-Playing, where personas are assigned to LLMs, and&#10;(2) LLM Personalization, where LLMs take care of user personas. Additionally,&#10;we introduce existing methods for LLM personality evaluation. To the best of&#10;our knowledge, we present the first survey for role-playing and personalization&#10;in LLMs under the unified view of persona. We continuously maintain a paper&#10;collection to foster future endeavors:&#10;https://github.com/MiuLab/PersonaLLM-Survey&#10;" />
          <attvalue for="2" value="&#10;The striking capabilities of large language models (LLMs), exemplified by ChatGPT~\cite{chatgpt}, have significantly advanced the field of natural language processing (NLP;~\cite{wei2023chainofthought,madaan2024self,shinn2024reflexion}).&#10;Recently, in addition to using LLMs as NLP task solvers or general-purpose chatbots, the question of how to adapt~LLMs for specific context has received great attention.&#10;To this end, leveraging personas has resurfaced as an ideal lens for adapting LLMs in target scenarios~\cite{chen2023large, chen2024persona}.&#10;By incorporating personas, LLMs can generate more contextually appropriate responses, maximizing their utility and effectiveness for specific applications.&#10;However, the growing literature on persona in the LLM era is relatively disorganized, lacking a unifying overview.&#10;&#10;In this paper, we aim to close the gap by offering a comprehensive survey and a systematic categorization of existing studies.&#10;Specifically, we divide current research into two main streams, namely LLM Role-Playing and LLM Personalization, as illustrated in~\ref{fig:overview}.&#10;The primary distinction is that in role-playing, the persona belongs to the LLM, while in personalization, the persona belongs to the user.&#10;Further, the literature on role-playing mainly focuses on the tasks (i.e., how LLMs with role-playing can achieve better performance). &#10;In contrast, the literature of personalization primarily focuses on the users (i.e., how to satisfy users’ expectations and meet their needs).&#10;It is noteworthy that both of role-playing and personalization can be goals in the same scenario, but serve different purposes and are driven by different aspects.&#10;The definitions are detailed below.&#10;\begin{itemize}&#10;\item LLM Role-Playing: LLMs are tasked to play the assigned personas (i.e., roles) and act based on environmental feedback, adapting to the environment.&#10;\item LLM Personalization:&#10;LLMs are tasked to take care of user personas (e.g., background information or historical behaviors) to meet individualized needs, adapting to distinct users.&#10;\end{itemize}&#10;&#10;To the best of our knowledge, we present the first survey for LLM role-playing and LLM personalization under the unified view of persona.&#10;To foster future endeavors, we actively maintain a paper collection available to the research community.&#10;We aim for this work to serve as both a valuable introduction for newcomers and a comprehensive resource for current researchers in the field.&#10;&#10;Our taxonomy is illustrated in~\ref{fig:survey-struc-new}.&#10;We first introduce LLM role-playing~\ref{sec:role_play}, followed by LLM personalization~\ref{sec:personalized}.&#10;Next, we provide an overview of evaluation methods~\ref{sec:personality} assessing whether the personality of LLMs (e.g., personality traits or psychological behaviors) accurately aligns with expected personas after the adaptation (i.e., for role-playing LLMs that act according to assigned personas and personalized LLMs that fit user personas).&#10;Lastly, we highlight current challenges and future directions~\ref{sec:challenges}.&#10;We hope that this taxonomy could serve as a useful guideline for researchers to easily target the tasks/scenarios of interests, and swiftly pinpoint their current position in the field.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Persona Framework, Linguistics, Cognitive Science, Personalization Techniques, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2307.16376" label="2307.16376">
        <attvalues>
          <attvalue for="0" value="When Large Language Models Meet Personalization: Perspectives of&#10;  Challenges and Opportunities" />
          <attvalue for="1" value="  The advent of large language models marks a revolutionary breakthrough in&#10;artificial intelligence. With the unprecedented scale of training and model&#10;parameters, the capability of large language models has been dramatically&#10;improved, leading to human-like performances in understanding, language&#10;synthesizing, and common-sense reasoning, etc. Such a major leap-forward in&#10;general AI capacity will change the pattern of how personalization is&#10;conducted. For one thing, it will reform the way of interaction between humans&#10;and personalization systems. Instead of being a passive medium of information&#10;filtering, large language models present the foundation for active user&#10;engagement. On top of such a new foundation, user requests can be proactively&#10;explored, and user's required information can be delivered in a natural and&#10;explainable way. For another thing, it will also considerably expand the scope&#10;of personalization, making it grow from the sole function of collecting&#10;personalized information to the compound function of providing personalized&#10;services. By leveraging large language models as general-purpose interface, the&#10;personalization systems may compile user requests into plans, calls the&#10;functions of external tools to execute the plans, and integrate the tools'&#10;outputs to complete the end-to-end personalization tasks. Today, large language&#10;models are still being developed, whereas the application in personalization is&#10;largely unexplored. Therefore, we consider it to be the right time to review&#10;the challenges in personalization and the opportunities to address them with&#10;LLMs. In particular, we dedicate this perspective paper to the discussion of&#10;the following aspects: the development and challenges for the existing&#10;personalization system, the newly emerged capabilities of large language&#10;models, and the potential ways of making use of large language models for&#10;personalization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.06031" label="2204.06031">
        <attvalues>
          <attvalue for="0" value="A Review on Language Models as Knowledge Bases" />
          <attvalue for="1" value="  Recently, there has been a surge of interest in the NLP community on the use&#10;of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have&#10;shown that LMs trained on a sufficiently large (web) corpus will encode a&#10;significant amount of knowledge implicitly in its parameters. The resulting LM&#10;can be probed for different kinds of knowledge and thus acting as a KB. This&#10;has a major advantage over traditional KBs in that this method requires no&#10;human supervision. In this paper, we present a set of aspects that we deem a LM&#10;should have to fully act as a KB, and review the recent literature with respect&#10;to those aspects.&#10;" />
          <attvalue for="2" value="&#10;&#10;The impact of Pretrained Language Models ({LMs}) on Natural Language Processing ({NLP}) research can be described as nothing short of transformative. It has moved the field from feature engineering \cite{och-etal-2004-smorgasbord, zhang-nivre-2011-transition} and architecture engineering \cite{Chung2014EmpiricalEO, kim-2014-convolutional, Bahdanau2015NeuralMT, vaswani17attention} to the pre-train and fine-tune paradigm \cite{Radford2018ImprovingLU, dong_NEURIPS2019_c20bb2d9, lewis_retrieval-augmented_2021}, and lately the pre-train, prompt, and predict paradigm \cite{Liu2021PretrainPA}. LMs pretrained on a large corpus of web data have been shown to contain different kinds of knowledge implicitly in their parameters without the need for any human supervision. This includes: world knowledge \cite{petroni_language_2019, rogers_primer_2020}, relational knowledge \cite{safavi2021relational}, commonsense knowledge \cite{Da2021AnalyzingCE}, linguistic knowledge \cite{Peters2018DissectingCW, Goldberg2019AssessingBS, tenney2018what}, actionable knowledge \cite{Huang2022LanguageMA} and more. This access to knowledge is crucial for LMs to achieve state-of-the-art results on various downstream tasks. However, as is the case with most neural systems, knowledge in LMs is encoded in a diffused manner, making it generally difficult to interpret and hard to update.&#10;&#10;Despite these recent breakthroughs, we often do not have full control over the behavior of LMs. As a result, utilizing these models in real-world scenarios is often unsuccessful. On the other hand, Knowledge Bases ({KB}) are easier to control. Here, KBs refers to a data structure that stores relational information in the form of triples connecting two triplets of entities by symbolic relations (e.g. $\langle$ Cairo, CapitalOf, Egypt $\rangle$). They often follow rule-based heuristics, rendering them predictable, in addition to possessing large knowledge coverage, which primes them for use in real-world systems. These models are often used as chatbots and virtual assistants, where controlled generation of output is necessary to ensure appropriate responses \cite{chen-dialogue-2017}. Therefore, KBs are a natural solution to access specific gold-standard relational information. They are repositories of knowledge, for both structured and unstructured data, and can be seamlessly queried and updated by an end user. &#10;&#10;Since KBs can access and update relational knowledge easier than LMs can, one question naturally arises: how can we control the repository of knowledge stored implicitly in the weights of a LM as similarly as KBs can? This question was first introduced in seminal work by \cite{petroni_language_2019} and has since ignited the interest of the community with the goal of instilling LMs with desirable properties of KBs.&#10;&#10;Several works have already approached improving LMs through the lens of KBs: \cite{petroni_language_2019, dhingra_time-aware_2021, wang_language_2020, heinzerling_language_2021, sung2021language}. Many of these works include updating factoids stored within the parameters of LMs \cite{de_cao_editing_2021, mitchell_fast_2021, hase_language_2021} to creating new methods for extracting factual knowledge \cite{petroni_language_2019}. Despite significant progress towards achieving parity between LMs and KBs, LMs still lack specific aspects that KBs have. For example, given the cloze phrases ``Albert Einstein was born in [MASK]'' and ``The hometown of Albert Einstein is [MASK]'', a user of a KB can map both queries to one triplet $\langle$ Albert Einstein, BornIn, X $\rangle$ that the KB readily understands and thus consistently returns the same city. On the other hand, LMs may not be as consistent, yielding potentially different answers for the same underlying factual questions. &#10;&#10;In this survey, we propose to consolidate the work on LMs-as-KBs within one cohesive framework, with a focus on aspects related to KBs that we think are useful to integrate into LMs. Few survey papers exist that evaluate LMs in this context. For instance, \cite{wei2021knowledge} evaluate knowledge-enhanced pretrained LMs by delineating the types of knowledge that can be integrated into existing LMs. \cite{safavi2021relational} divide relevant work according to the level of supervision provided to the LM by a KB. Similarly, \cite{colonhernandez2021combining} cover the integration of structural knowledge into LMs but forgo implicit knowledge. Our study of LMs-as-KBs from our perspective is unique compared to the focus of existing survey papers. We aim to present the current landscape of LMs-as-KBs research and highlight the existing challenges that LMs face when applied in practice. &#10;&#10;We observe the recent advances in LMs and explore them with respect to aspects that we find necessary for LMs to become as functional and utilizeable as KBs: access, consistency, editability, reasoning, and explainability. We further highlight where we are now in terms of the LMs-as-KBs framework as well as potential work for the future. Finally, we discuss the remaining challenges in the full adoption of LMs-as-KBs and propose directions for future research. This survey is structured as follows:&#10;&#10;\begin{enumerate}&#10; \item First, an overview of KBs and LMs and their intersectionfor LMs-as-KBs &#10; \item Second, the enumeration of the aspects of LMs-as-KBs &#10; \item Third, a brief summarization of each aspect with respect to recent work&#10;\end{enumerate}&#10;&#10;We hope that by highlighting the aspects of LMs-as-KBs, we can consolidate knowledge in this ever-growing field of research. We envision that our work can provide a path for those new to the area of research to better improve LMs to be just as good, if not eventually better, than KBs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Artificial Intelligence, Knowledge Bases, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1412.3555" label="1412.3555">
        <attvalues>
          <attvalue for="0" value="Empirical Evaluation of Gated Recurrent Neural Networks on Sequence&#10;  Modeling" />
          <attvalue for="1" value="  In this paper we compare different types of recurrent units in recurrent&#10;neural networks (RNNs). Especially, we focus on more sophisticated units that&#10;implement a gating mechanism, such as a long short-term memory (LSTM) unit and&#10;a recently proposed gated recurrent unit (GRU). We evaluate these recurrent&#10;units on the tasks of polyphonic music modeling and speech signal modeling. Our&#10;experiments revealed that these advanced recurrent units are indeed better than&#10;more traditional recurrent units such as tanh units. Also, we found GRU to be&#10;comparable to LSTM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.05837" label="2104.05837">
        <attvalues>
          <attvalue for="0" value="Relational World Knowledge Representation in Contextual Language Models:&#10;  A Review" />
          <attvalue for="1" value="  Relational knowledge bases (KBs) are commonly used to represent world&#10;knowledge in machines. However, while advantageous for their high degree of&#10;precision and interpretability, KBs are usually organized according to&#10;manually-defined schemas, which limit their expressiveness and require&#10;significant human efforts to engineer and maintain. In this review, we take a&#10;natural language processing perspective to these limitations, examining how&#10;they may be addressed in part by training deep contextual language models (LMs)&#10;to internalize and express relational knowledge in more flexible forms. We&#10;propose to organize knowledge representation strategies in LMs by the level of&#10;KB supervision provided, from no KB supervision at all to entity- and&#10;relation-level supervision. Our contributions are threefold: (1) We provide a&#10;high-level, extensible taxonomy for knowledge representation in LMs; (2) Within&#10;our taxonomy, we highlight notable models, evaluation tasks, and findings, in&#10;order to provide an up-to-date review of current knowledge representation&#10;capabilities in LMs; and (3) We suggest future research directions that build&#10;upon the complementary aspects of LMs and KBs as knowledge representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.00297" label="2101.00297">
        <attvalues>
          <attvalue for="0" value="Analyzing Commonsense Emergence in Few-shot Knowledge Models" />
          <attvalue for="1" value="  Recently, commonsense knowledge models - pretrained language models (LM)&#10;fine-tuned on knowledge graph (KG) tuples - showed that considerable amounts of&#10;commonsense knowledge can be encoded in the parameters of large language&#10;models. However, as parallel studies show that LMs are poor hypothesizers of&#10;declarative commonsense relationships on their own, it remains unclear whether&#10;this knowledge is learned during pretraining or from fine-tuning on KG&#10;examples. To investigate this question, we train commonsense knowledge models&#10;in few-shot settings to study the emergence of their commonsense representation&#10;abilities. Our results show that commonsense knowledge models can rapidly adapt&#10;from limited examples, indicating that KG fine-tuning serves to learn an&#10;interface to encoded knowledge learned during pretraining. Importantly, our&#10;analysis of absolute, angular, and distributional parameter changes during&#10;few-shot fine-tuning provides novel insights into how this interface is&#10;learned.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.07207" label="2201.07207">
        <attvalues>
          <attvalue for="0" value="Language Models as Zero-Shot Planners: Extracting Actionable Knowledge&#10;  for Embodied Agents" />
          <attvalue for="1" value="  Can world knowledge learned by large language models (LLMs) be used to act in&#10;interactive environments? In this paper, we investigate the possibility of&#10;grounding high-level tasks, expressed in natural language (e.g. &quot;make&#10;breakfast&quot;), to a chosen set of actionable steps (e.g. &quot;open fridge&quot;). While&#10;prior work focused on learning from explicit step-by-step examples of how to&#10;act, we surprisingly find that if pre-trained LMs are large enough and prompted&#10;appropriately, they can effectively decompose high-level tasks into mid-level&#10;plans without any further training. However, the plans produced naively by LLMs&#10;often cannot map precisely to admissible actions. We propose a procedure that&#10;conditions on existing demonstrations and semantically translates the plans to&#10;admissible actions. Our evaluation in the recent VirtualHome environment shows&#10;that the resulting method substantially improves executability over the LLM&#10;baseline. The conducted human evaluation reveals a trade-off between&#10;executability and correctness but shows a promising sign towards extracting&#10;actionable knowledge from language models. Website at&#10;https://huangwl18.github.io/language-planner&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08455" label="2110.08455">
        <attvalues>
          <attvalue for="0" value="Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey" />
          <attvalue for="1" value="  Pretrained Language Models (PLM) have established a new paradigm through&#10;learning informative contextualized representations on large-scale text corpus.&#10;This new paradigm has revolutionized the entire field of natural language&#10;processing, and set the new state-of-the-art performance for a wide variety of&#10;NLP tasks. However, though PLMs could store certain knowledge/facts from&#10;training corpus, their knowledge awareness is still far from satisfactory. To&#10;address this issue, integrating knowledge into PLMs have recently become a very&#10;active research area and a variety of approaches have been developed. In this&#10;paper, we provide a comprehensive survey of the literature on this emerging and&#10;fast-growing field - Knowledge Enhanced Pretrained Language Models (KE-PLMs).&#10;We introduce three taxonomies to categorize existing work. Besides, we also&#10;survey the various NLU and NLG applications on which KE-PLM has demonstrated&#10;superior performance over vanilla PLMs. Finally, we discuss challenges that&#10;face KE-PLMs and also promising directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.12294" label="2101.12294">
        <attvalues>
          <attvalue for="0" value="Combining pre-trained language models and structured knowledge" />
          <attvalue for="1" value="  In recent years, transformer-based language models have achieved state of the&#10;art performance in various NLP benchmarks. These models are able to extract&#10;mostly distributional information with some semantics from unstructured text,&#10;however it has proven challenging to integrate structured information, such as&#10;knowledge graphs into these models. We examine a variety of approaches to&#10;integrate structured knowledge into current language models and determine&#10;challenges, and possible opportunities to leverage both structured and&#10;unstructured information sources. From our survey, we find that there are still&#10;opportunities at exploiting adapter-based injections and that it may be&#10;possible to further combine various of the explored approaches into one system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07067" label="2109.07067">
        <attvalues>
          <attvalue for="0" value="Improving Text Auto-Completion with Next Phrase Prediction" />
          <attvalue for="1" value="  Language models such as GPT-2 have performed well on constructing&#10;syntactically sound sentences for text auto-completion task. However, such&#10;models often require considerable training effort to adapt to specific writing&#10;domains (e.g., medical). In this paper, we propose an intermediate training&#10;strategy to enhance pre-trained language models' performance in the text&#10;auto-completion task and fastly adapt them to specific domains. Our strategy&#10;includes a novel self-supervised training objective called Next Phrase&#10;Prediction (NPP), which encourages a language model to complete the partial&#10;query with enriched phrases and eventually improve the model's text&#10;auto-completion performance. Preliminary experiments have shown that our&#10;approach is able to outperform the baselines in auto-completion for email and&#10;academic writing domains.&#10;" />
          <attvalue for="2" value="&#10;Natural language interface (NLI) applications such as Personal assistants (e.g., Amazon Alexa, Apple Siri, Google Assistant, and Microsoft Cortana) and search engines (e.g., Google) have become an integral part of our everyday life. &#10;Among the many features in NLI applications, text auto-completion, which aims to suggest words, phrases, and sentences that complete the user's textual input, is a common, but key feature. Smart reply~\cite{kannan2016smart} and Smart compose~\cite{chen2019gmail} are two recent works that provide contextual assistance to aid users in completing everyday text such as emails, search engine inputs, etc.&#10;&#10;While recent advances in deep neural models have shown impressive performance on the text auto-completion task, these models generally require a large amount of everyday text and huge amount of computing power for training to generate adequate suggestions~\cite{chen2019gmail}.&#10;The challenge is compounded when we perform auto-completion in specific domains such as academic writing, which requires a large training corpus for specific expertise. ~\tabref{tab:perplexity} illustrates the difficulty in domain-specific auto-completion with the same amount of supervisions. &#10;&#10;A potential solution to address the challenges in text auto-completion is exploiting the Decoder-only Transformer model such as GPT-2~\cite{radford2019language}.&#10;The model performs well on constructing syntactically sound sentences from a partial query.&#10;However, GPT-2 requires a huge fine-tuning effort to construct sentences of expert domains.&#10;~gref{fig:example} shows an example of GPT-2 auto-completion suggestions for computer science domain sentences before fine-tuning. &#10;Recently, text-to-text transformers such as BART~\cite{lewis-etal-2020-bart} and T5~\cite{JMLR:v21:20-074} have demonstrated great potential in natural language generation (NLG) tasks by using masked-span infilling as a pre-training objective. &#10;However, similar to GPT-2, these models also require huge fine-tuning efforts to perform domain-specific text auto-completion. &#10;&#10;This paper aims to address this research gap by proposing an intermediate training strategy~\cite{pruksachatkun-etal-2020-intermediate, calm2021}, which incrementally trains a pre-trained text-to-text transformer to provide better auto-completion suggestions and fastly adapt to the expert domain during fine-tuning. &#10;As shown in Figure~\ref{fig:overview}, the core of our intermediate training strategy is a simple self-supervised objective called Next Phrase Prediction (NPP), which has two major steps: Phrase Extraction (Section~\ref{sec:phrase}) and Generative Question Answering (Section~\ref{sec:qa}). &#10;The first stage extracts qualitative phrases by constituency parsing. &#10;By exploiting constituency parsing, the framework is able to utilize the complete phrase, not just a fraction of the sentence.&#10;Next, the pre-trained language model is guided to choose the correct next phrase among other phrases of the same type (e.g., noun phrase, verb phrase, etc.) in the sentence.&#10;For example, the sentence &quot;She bought a top and bottom from that strange little shop.&quot; has two noun phrases &quot;a top and bottom&quot; and &quot;that strange little shop&quot;. &#10;If the partial query is &quot;She bought&quot;, the model is guided choose the proper complete noun phrase &quot;a top and bottom&quot; for its next phrase.&#10;&#10;To the best of our knowledge, this is the first work that proposed an intermediate training strategy for improving language models' performance on the text auto-completion task. &#10;Through extensive experiments, we demonstrated that our proposed approach could improve the text-to-text transformer's performance on auto-completion task and fastly adapt to expert domain of text auto-completion.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Training, Text Auto-Completion, Domain Adaptation, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1606.04870" label="1606.04870">
        <attvalues>
          <attvalue for="0" value="Smart Reply: Automated Response Suggestion for Email" />
          <attvalue for="1" value="  In this paper we propose and investigate a novel end-to-end method for&#10;automatically generating short email responses, called Smart Reply. It&#10;generates semantically diverse suggestions that can be used as complete email&#10;responses with just one tap on mobile. The system is currently used in Inbox by&#10;Gmail and is responsible for assisting with 10% of all mobile responses. It is&#10;designed to work at very high throughput and process hundreds of millions of&#10;messages daily. The system exploits state-of-the-art, large-scale deep&#10;learning.&#10;  We describe the architecture of the system as well as the challenges that we&#10;faced while building it, like response diversity and scalability. We also&#10;introduce a new method for semantic clustering of user-generated content that&#10;requires only a modest amount of explicitly labeled data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.00080" label="1906.00080">
        <attvalues>
          <attvalue for="0" value="Gmail Smart Compose: Real-Time Assisted Writing" />
          <attvalue for="1" value="  In this paper, we present Smart Compose, a novel system for generating&#10;interactive, real-time suggestions in Gmail that assists users in writing mails&#10;by reducing repetitive typing. In the design and deployment of such a&#10;large-scale and complicated system, we faced several challenges including model&#10;selection, performance evaluation, serving and other practical issues. At the&#10;core of Smart Compose is a large-scale neural language model. We leveraged&#10;state-of-the-art machine learning techniques for language model training which&#10;enabled high-quality suggestion prediction, and constructed novel serving&#10;infrastructure for high-throughput and real-time inference. Experimental&#10;results show the effectiveness of our proposed system design and deployment&#10;approach. This system is currently being served in Gmail.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.15156" label="2312.15156">
        <attvalues>
          <attvalue for="0" value="Large Language Models as Zero-Shot Keyphrase Extractors: A Preliminary&#10;  Empirical Study" />
          <attvalue for="1" value="  Zero-shot keyphrase extraction aims to build a keyphrase extractor without&#10;training by human-annotated data, which is challenging due to the limited human&#10;intervention involved. Challenging but worthwhile, zero-shot setting&#10;efficiently reduces the time and effort that data labeling takes. Recent&#10;efforts on pre-trained large language models (e.g., ChatGPT and ChatGLM) show&#10;promising performance on zero-shot settings, thus inspiring us to explore&#10;prompt-based methods. In this paper, we ask whether strong keyphrase extraction&#10;models can be constructed by directly prompting the large language model&#10;ChatGPT. Through experimental results, it is found that ChatGPT still has a lot&#10;of room for improvement in the keyphrase extraction task compared to existing&#10;state-of-the-art unsupervised and supervised models.&#10;" />
          <attvalue for="2" value="&#10;Keyphrase extraction aims to extract a set of important phrases from unstructured text into structured data formats, which is a fundamental and crucial task in natural language processing \cite{2014survey,song_survey,song2023chatgpt}. Typically, keyphrase is beneficial for various applications due to its concise and precise expression \cite{li2023generate,pmlr,Tian_2023,hypersiamesenet,li2023unsupervised,HISUM,salewski2023zeroshot}. Working with an enormous amount of labeling data is always hectic, labor-intensive, and time-consuming. Hence, many organizations and companies rely on keyphrase extraction to automate manual work with zero / few-shot settings \cite{2021unsupervised, hguke, promptrank, hyperrank, agrank}.&#10;&#10;Recent works \cite{agrawal2022,Wei2023ZeroShotIE} on large-scale pre-trained language models, such as GPT-3 \cite{llm_few}, InstructGPT \cite{ouyang2022training} and ChatGPT 2, suggest that large language models perform well in various natural language processing downstream tasks even without tuning the parameters. Therefore, it is a challenging question: Is it feasible to prompt large language models to achieve a zero-shot keyphrase extractor? Based on these clues, in this paper, we turn to ChatGPT and hypothesize that ChatGPT is born with the right abilities to deposit a zero-shot keyphrase extractor in an interactive mode. &#10;&#10;In this paper, we mainly focus on verifying the performance of ChatGPT on four keyphrase extraction datasets and the ability of understanding long documents. Generally, long documents often necessitate complex processing strategies \cite{longke,ld1,ld2,ld3}. In the keyphrase extraction task, while it may be feasible to design improved algorithms for handling a multitude of candidate keyphrases in long documents, we believe that the effective incorporation of advanced features, especially those with background knowledge, can enhance the efficient discrimination between keyphrases and non-keyphrases, even when dealing with a vast number of candidate keyphrases. Simultaneously, the question of how to encode long documents is a topic worthy of exploration \cite{2014survey}. Consequently, we test the capacity of ChatGPT as a general large language model to handle long documents. Through extensive experiments, it is found that ChatGPT still has a lot of room for improvement in the keyphrase extraction task compared to existing state-of-the-art unsupervised and supervised models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Models, Zero-Shot Learning, Artificial Intelligence, Natural Language Processing, Keyphrase Extraction" />
        </attvalues>
      </node>
      <node id="2303.13001" label="2303.13001">
        <attvalues>
          <attvalue for="0" value="Is ChatGPT A Good Keyphrase Generator? A Preliminary Study" />
          <attvalue for="1" value="  The emergence of ChatGPT has recently garnered significant attention from the&#10;computational linguistics community. To demonstrate its capabilities as a&#10;keyphrase generator, we conduct a preliminary evaluation of ChatGPT for the&#10;keyphrase generation task. We evaluate its performance in various aspects,&#10;including keyphrase generation prompts, keyphrase generation diversity, and&#10;long document understanding. Our evaluation is based on six benchmark datasets,&#10;and we adopt the prompt suggested by OpenAI while extending it to six candidate&#10;prompts. We find that ChatGPT performs exceptionally well on all six candidate&#10;prompts, with minor performance differences observed across the datasets. Based&#10;on our findings, we conclude that ChatGPT has great potential for keyphrase&#10;generation. Moreover, we discover that ChatGPT still faces challenges when it&#10;comes to generating absent keyphrases. Meanwhile, in the final section, we also&#10;present some limitations and future expansions of this report.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.07481" label="2009.07481">
        <attvalues>
          <attvalue for="0" value="Unsupervised Summarization by Jointly Extracting Sentences and Keywords" />
          <attvalue for="1" value="  We present RepRank, an unsupervised graph-based ranking model for extractive&#10;multi-document summarization in which the similarity between words, sentences,&#10;and word-to-sentence can be estimated by the distances between their vector&#10;representations in a unified vector space. In order to obtain desirable&#10;representations, we propose a self-attention based learning method that&#10;represent a sentence by the weighted sum of its word embeddings, and the&#10;weights are concentrated to those words hopefully better reflecting the content&#10;of a document. We show that salient sentences and keywords can be extracted in&#10;a joint and mutual reinforcement process using our learned representations, and&#10;prove that this process always converges to a unique solution leading to&#10;improvement in performance. A variant of absorbing random walk and the&#10;corresponding sampling-based algorithm are also described to avoid redundancy&#10;and increase diversity in the summaries. Experiment results with multiple&#10;benchmark datasets show that RepRank achieved the best or comparable&#10;performance in ROUGE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.08396" label="2311.08396">
        <attvalues>
          <attvalue for="0" value="Zero-shot audio captioning with audio-language model guidance and audio&#10;  context keywords" />
          <attvalue for="1" value="  Zero-shot audio captioning aims at automatically generating descriptive&#10;textual captions for audio content without prior training for this task.&#10;Different from speech recognition which translates audio content that contains&#10;spoken language into text, audio captioning is commonly concerned with ambient&#10;sounds, or sounds produced by a human performing an action. Inspired by&#10;zero-shot image captioning methods, we propose ZerAuCap, a novel framework for&#10;summarising such general audio signals in a text caption without requiring&#10;task-specific training. In particular, our framework exploits a pre-trained&#10;large language model (LLM) for generating the text which is guided by a&#10;pre-trained audio-language model to produce captions that describe the audio&#10;content. Additionally, we use audio context keywords that prompt the language&#10;model to generate text that is broadly relevant to sounds. Our proposed&#10;framework achieves state-of-the-art results in zero-shot audio captioning on&#10;the AudioCaps and Clotho datasets. Our code is available at&#10;https://github.com/ExplainableML/ZerAuCap.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07293" label="2109.07293">
        <attvalues>
          <attvalue for="0" value="Unsupervised Keyphrase Extraction by Jointly Modeling Local and Global&#10;  Context" />
          <attvalue for="1" value="  Embedding based methods are widely used for unsupervised keyphrase extraction&#10;(UKE) tasks. Generally, these methods simply calculate similarities between&#10;phrase embeddings and document embedding, which is insufficient to capture&#10;different context for a more effective UKE model. In this paper, we propose a&#10;novel method for UKE, where local and global contexts are jointly modeled. From&#10;a global view, we calculate the similarity between a certain phrase and the&#10;whole document in the vector space as transitional embedding based models do.&#10;In terms of the local view, we first build a graph structure based on the&#10;document where phrases are regarded as vertices and the edges are similarities&#10;between vertices. Then, we proposed a new centrality computation method to&#10;capture local salient information based on the graph structure. Finally, we&#10;further combine the modeling of global and local context for ranking. We&#10;evaluate our models on three public benchmarks (Inspec, DUC 2001, SemEval 2010)&#10;and compare with existing state-of-the-art models. The results show that our&#10;model outperforms most models while generalizing better on input documents with&#10;different domains and length. Additional ablation study shows that both the&#10;local and global information is crucial for unsupervised keyphrase extraction&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.03319" label="2305.03319">
        <attvalues>
          <attvalue for="0" value="HiPool: Modeling Long Documents Using Graph Neural Networks" />
          <attvalue for="1" value="  Encoding long sequences in Natural Language Processing (NLP) is a challenging&#10;problem. Though recent pretraining language models achieve satisfying&#10;performances in many NLP tasks, they are still restricted by a pre-defined&#10;maximum length, making them challenging to be extended to longer sequences. So&#10;some recent works utilize hierarchies to model long sequences. However, most of&#10;them apply sequential models for upper hierarchies, suffering from long&#10;dependency issues. In this paper, we alleviate these issues through a&#10;graph-based method. We first chunk the sequence with a fixed length to model&#10;the sentence-level information. We then leverage graphs to model intra- and&#10;cross-sentence correlations with a new attention mechanism. Additionally, due&#10;to limited standard benchmarks for long document classification (LDC), we&#10;propose a new challenging benchmark, totaling six datasets with up to 53k&#10;samples and 4034 average tokens' length. Evaluation shows our model surpasses&#10;competitive baselines by 2.6% in F1 score, and 4.8% on the longest sequence&#10;dataset. Our method is shown to outperform hierarchical sequential models with&#10;better performance and scalability, especially for longer sequences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16784" label="2305.16784">
        <attvalues>
          <attvalue for="0" value="Incorporating Distributions of Discourse Structure for Long Document&#10;  Abstractive Summarization" />
          <attvalue for="1" value="  For text summarization, the role of discourse structure is pivotal in&#10;discerning the core content of a text. Regrettably, prior studies on&#10;incorporating Rhetorical Structure Theory (RST) into transformer-based&#10;summarization models only consider the nuclearity annotation, thereby&#10;overlooking the variety of discourse relation types. This paper introduces the&#10;'RSTformer', a novel summarization model that comprehensively incorporates both&#10;the types and uncertainty of rhetorical relations. Our RST-attention mechanism,&#10;rooted in document-level rhetorical structure, is an extension of the recently&#10;devised Longformer framework. Through rigorous evaluation, the model proposed&#10;herein exhibits significant superiority over state-of-the-art models, as&#10;evidenced by its notable performance on several automatic metrics and human&#10;evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2405.07767" label="2405.07767">
        <attvalues>
          <attvalue for="0" value="Synthetic Test Collections for Retrieval Evaluation" />
          <attvalue for="1" value="  Test collections play a vital role in evaluation of information retrieval&#10;(IR) systems. Obtaining a diverse set of user queries for test collection&#10;construction can be challenging, and acquiring relevance judgments, which&#10;indicate the appropriateness of retrieved documents to a query, is often costly&#10;and resource-intensive. Generating synthetic datasets using Large Language&#10;Models (LLMs) has recently gained significant attention in various&#10;applications. In IR, while previous work exploited the capabilities of LLMs to&#10;generate synthetic queries or documents to augment training data and improve&#10;the performance of ranking models, using LLMs for constructing synthetic test&#10;collections is relatively unexplored. Previous studies demonstrate that LLMs&#10;have the potential to generate synthetic relevance judgments for use in the&#10;evaluation of IR systems. In this paper, we comprehensively investigate whether&#10;it is possible to use LLMs to construct fully synthetic test collections by&#10;generating not only synthetic judgments but also synthetic queries. In&#10;particular, we analyse whether it is possible to construct reliable synthetic&#10;test collections and the potential risks of bias such test collections may&#10;exhibit towards LLM-based models. Our experiments indicate that using LLMs it&#10;is possible to construct synthetic test collections that can reliably be used&#10;for retrieval evaluation.&#10;" />
          <attvalue for="2" value="&#10;Test collection construction is a pivotal process to evaluate the effectiveness of information retrieval (IR) systems. The most widely used approach for constructing test collections is based on the Cranfield paradigm \cite{cleverdon1967cranfield,aslam2006statistical}, which involves creating a collection comprising queries and associated relevance judgments. Queries used in test collection construction are expected to come from real usage logs, representing real information needs. However, it is very difficult to get access to such logs outside of search engine companies. Hence, lots of existing test collections used in IR are based on manually created queries \cite{craswell2021trec,yilmaz2008simple}. This process demands time and expertise, making it costly in terms of labor and resources; furthermore, there are no guarantees that queries generated at the end of this process are representative of real information needs. Similarly, obtaining relevance judgments for a query is an expensive procedure requiring significant human effort. The highly demanding nature of the test construction process is a major bottleneck in constructing large test collections; hence, most existing publicly available test collections tend to consist of a small number of queries, which could degrade the reliability of these collections.&#10;&#10;Recently, Large Language Models (LLMs) have demonstrated remarkable performance on unseen tasks by only considering the instructions (so-called `prompts') provided to them \cite{chiang2023can,jiang2022promptmaker}. Synthetic datasets generated using LLMs have recently gained attention across a range of diverse tasks \cite{li2023synthetic,zhang2018synthetic,bao2023synthetic}. In IR, previous studies use LLMs to generate synthetic training data for augmentation to boost the quality of retrievers \cite{askari2023expand,bonifacio2022inpars}. LLMs have also been used to generate relevance labels \cite{thomas2023large,faggioli2023perspectives}, as well as to generate query variants for evaluation and training of IR systems \cite{rajapakse2023improving,alaofi2023can}. However, to the best of our knowledge, no prior study has thoroughly explored the potential of LLMs to construct fully synthetic test collections where both queries and associated relevance judgments are automatically generated using LLMs. &#10;&#10;Given the aforementioned challenges in constructing large-scale test collections, our goal in this paper is to investigate whether it is possible to create reliable synthetic test collections so that there is (i) no need for real usage query logs or manual creation of queries, and (ii) no need to obtain manual relevance judgments. We investigate different approaches to construct synthetic test collections using LLMs and show using synthetic test collections it is possible to obtain evaluation results that are similar to results obtained using real test collections.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Large Language Models, Linguistics, Synthetic Test Collections, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2104.09399" label="2104.09399">
        <attvalues>
          <attvalue for="0" value="TREC Deep Learning Track: Reusable Test Collections in the Large Data&#10;  Regime" />
          <attvalue for="1" value="  The TREC Deep Learning (DL) Track studies ad hoc search in the large data&#10;regime, meaning that a large set of human-labeled training data is available.&#10;Results so far indicate that the best models with large data may be deep neural&#10;networks. This paper supports the reuse of the TREC DL test collections in&#10;three ways. First we describe the data sets in detail, documenting clearly and&#10;in one place some details that are otherwise scattered in track guidelines,&#10;overview papers and in our associated MS MARCO leaderboard pages. We intend&#10;this description to make it easy for newcomers to use the TREC DL data. Second,&#10;because there is some risk of iteration and selection bias when reusing a data&#10;set, we describe the best practices for writing a paper using TREC DL data,&#10;without overfitting. We provide some illustrative analysis. Finally we address&#10;a number of issues around the TREC DL data, including an analysis of&#10;reusability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.07849" label="2310.07849">
        <attvalues>
          <attvalue for="0" value="Synthetic Data Generation with Large Language Models for Text&#10;  Classification: Potential and Limitations" />
          <attvalue for="1" value="  The collection and curation of high-quality training data is crucial for&#10;developing text classification models with superior performance, but it is&#10;often associated with significant costs and time investment. Researchers have&#10;recently explored using large language models (LLMs) to generate synthetic&#10;datasets as an alternative approach. However, the effectiveness of the&#10;LLM-generated synthetic data in supporting model training is inconsistent&#10;across different classification tasks. To better understand factors that&#10;moderate the effectiveness of the LLM-generated synthetic data, in this study,&#10;we look into how the performance of models trained on these synthetic data may&#10;vary with the subjectivity of classification. Our results indicate that&#10;subjectivity, at both the task level and instance level, is negatively&#10;associated with the performance of the model trained on synthetic data. We&#10;conclude by discussing the implications of our work on the potential and&#10;limitations of leveraging LLM for synthetic data generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10; Generative AI in synthetic data generation. Recent advancements in generative AI have motivated numerous studies to explore the potential of leveraging generative models to create synthetic data for training machine learning models, especially for computer vision (CV) and natural language processing (NLP) tasks. In the realm of CV, several works have utilized GAN-based models~\cite{karras2019style} or diffusion models~\cite{nichol2021glide} to generate synthetic data for image recognition~\cite{besnier2020dataset,he2022synthetic} or object segmentation~\cite{zhang2021datasetgan}. Similarly, in the NLP field, researchers have also probed into the capacity of language models in generating synthetic data for various text classification tasks~\cite{kumar2020data,chung2023increasing, sahu2022data, yoo2021gpt3mix, ye2022zerogen,wang2021towards,hartvigsen2022toxigen,meng2022generating,gao2022self,aggarwal2022entity, chen2022weakly}, with mixed results reported regarding the effectiveness of the synthetic data generated. &#10;In this study, we aim to obtain a better understanding of {when} the synthetic data generated by language models can lead to effective model training, and we focus on exploring the role of task subjectivity in moderating the effectiveness of the synthetic data. &#10;&#10;Large language models.&#10;Based on the Transformer architecture~\cite{vaswani2017attention}, large language models (LLMs) have facilitated remarkable progress in the field of natural language processing. The utilization of bidirectional contexts in the BERT model~\cite{devlin2018bert} has resulted in superior performance across a wide range of tasks. Building on this, OpenAI's GPT series, comprising of models like GPT-2~\cite{radford2019language}, the colossal GPT-3~\cite{brown2020language} with an impressive 175 billion parameters and the most recent GPT-4~\cite{openai2023gpt4}, pushed the boundaries of possibilities of LLMs. These models exhibit remarkable proficiency in generating high-quality human-like text~\cite{clark2021all,dou2021gpt,10.1145/3544548.3581318}, showcasing capabilities in rudimentary reasoning~\cite{wei2021finetuned}, translation~\cite{brown2020language}, scientific synthetic data generation~\cite{10.1145/3544548.3580688}, and code generation~\cite{10.1145/3544548.3580940}. &#10;In this study, we focus on leveraging the cutting-edge GPT-3.5-Turbo model to explore its capabilities and limitations in synthesizing data for text classification tasks with different subjectivity levels.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1806.01013" label="1806.01013">
        <attvalues>
          <attvalue for="0" value="Synthetic data generation for end-to-end thermal infrared tracking" />
          <attvalue for="1" value="  The usage of both off-the-shelf and end-to-end trained deep networks have&#10;significantly improved performance of visual tracking on RGB videos. However,&#10;the lack of large labeled datasets hampers the usage of convolutional neural&#10;networks for tracking in thermal infrared (TIR) images. Therefore, most state&#10;of the art methods on tracking for TIR data are still based on handcrafted&#10;features. To address this problem, we propose to use image-to-image translation&#10;models. These models allow us to translate the abundantly available labeled RGB&#10;data to synthetic TIR data. We explore both the usage of paired and unpaired&#10;image translation models for this purpose. These methods provide us with a&#10;large labeled dataset of synthetic TIR sequences, on which we can train&#10;end-to-end optimal features for tracking. To the best of our knowledge we are&#10;the first to train end-to-end features for TIR tracking. We perform extensive&#10;experiments on VOT-TIR2017 dataset. We show that a network trained on a large&#10;dataset of synthetic TIR data obtains better performance than one trained on&#10;the available real TIR data. Combining both data sources leads to further&#10;improvement. In addition, when we combine the network with motion features we&#10;outperform the state of the art with a relative gain of over 10%, clearly&#10;showing the efficiency of using synthetic data to train end-to-end TIR&#10;trackers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.10621" label="2309.10621">
        <attvalues>
          <attvalue for="0" value="Large language models can accurately predict searcher preferences" />
          <attvalue for="1" value="  Relevance labels, which indicate whether a search result is valuable to a&#10;searcher, are key to evaluating and optimising search systems. The best way to&#10;capture the true preferences of users is to ask them for their careful feedback&#10;on which results would be useful, but this approach does not scale to produce a&#10;large number of labels. Getting relevance labels at scale is usually done with&#10;third-party labellers, who judge on behalf of the user, but there is a risk of&#10;low-quality data if the labeller doesn't understand user needs. To improve&#10;quality, one standard approach is to study real users through interviews, user&#10;studies and direct feedback, find areas where labels are systematically&#10;disagreeing with users, then educate labellers about user needs through judging&#10;guidelines, training and monitoring. This paper introduces an alternate&#10;approach for improving label quality. It takes careful feedback from real&#10;users, which by definition is the highest-quality first-party gold data that&#10;can be derived, and develops an large language model prompt that agrees with&#10;that data.&#10;  We present ideas and observations from deploying language models for&#10;large-scale relevance labelling at Bing, and illustrate with data from TREC. We&#10;have found large language models can be effective, with accuracy as good as&#10;human labellers and similar capability to pick the hardest queries, best runs,&#10;and best groups. Systematic changes to the prompts make a difference in&#10;accuracy, but so too do simple paraphrases. To measure agreement with real&#10;searchers needs high-quality &quot;gold&quot; labels, but with these we find that models&#10;produce better labels than third-party workers, for a fraction of the cost, and&#10;these labels let us train notably better rankers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.09161" label="2304.09161">
        <attvalues>
          <attvalue for="0" value="Perspectives on Large Language Models for Relevance Judgment" />
          <attvalue for="1" value="  When asked, large language models (LLMs) like ChatGPT claim that they can&#10;assist with relevance judgments but it is not clear whether automated judgments&#10;can reliably be used in evaluations of retrieval systems. In this perspectives&#10;paper, we discuss possible ways for LLMs to support relevance judgments along&#10;with concerns and issues that arise. We devise a human--machine collaboration&#10;spectrum that allows to categorize different relevance judgment strategies,&#10;based on how much humans rely on machines. For the extreme point of &quot;fully&#10;automated judgments&quot;, we further include a pilot experiment on whether&#10;LLM-based relevance judgments correlate with judgments from trained human&#10;assessors. We conclude the paper by providing opposing perspectives for and&#10;against the use of~LLMs for automatic relevance judgments, and a compromise&#10;perspective, informed by our analyses of the literature, our preliminary&#10;experimental evidence, and our experience as IR researchers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.05619" label="2309.05619">
        <attvalues>
          <attvalue for="0" value="Effective Proxy for Human Labeling: Ensemble Disagreement Scores in&#10;  Large Language Models for Industrial NLP" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated significant capability to&#10;generalize across a large number of NLP tasks. For industry applications, it is&#10;imperative to assess the performance of the LLM on unlabeled production data&#10;from time to time to validate for a real-world setting. Human labeling to&#10;assess model error requires considerable expense and time delay. Here we&#10;demonstrate that ensemble disagreement scores work well as a proxy for human&#10;labeling for language models in zero-shot, few-shot, and fine-tuned settings,&#10;per our evaluation on keyphrase extraction (KPE) task. We measure fidelity of&#10;the results by comparing to true error measured from human labeled ground&#10;truth. We contrast with the alternative of using another LLM as a source of&#10;machine labels, or silver labels. Results across various languages and domains&#10;show disagreement scores provide a better estimation of model performance with&#10;mean average error (MAE) as low as 0.4% and on average 13.8% better than using&#10;silver labels.&#10;" />
          <attvalue for="2" value="&#10;We have recently seen significant progress on many natural language processing (NLP) tasks using the latest generative pretrained models such as GPT \cite{openai2023gpt, ouyang2022training}, PaLM \cite{chowdhery2022palm}, and many others \cite{touvron2023llama, bai2022constitutional, penedo2023refinedweb, alpaca}. This new generation of models opens up many new possibilities including competitive performance in zero-shot and few-shot settings for tasks that have typically been modeled using a supervised setting \cite{openai2023gpt}. More established language models (BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta}, XLM-Roberta \cite{conneau2020unsupervised}, etc.) provide a strong balance of inference cost and task performance for such systems. This broad class of large language models (LLMs) used for complex supervised NLP tasks share the problem of how to effectively assess performance in production settings where we don’t yet have human labels due to cost or urgency. &#10;&#10;The ability to judge model capability becomes important for production settings where we often have to decide whether to launch a model in a new domain or for a new language where we have few or no labels ready. This is also known as few-shot and zero-shot performance, respectively. Scaling models up to new domains and new languages quickly becomes an expensive proposition in terms of labeling. For example, if we have two new domains and ten languages, this results in twenty new label sets that need to be generated. Having the capability to guide that investment or possibly eliminate the need for extensive human labeling for some subset of those domains/languages becomes very valuable. &#10;&#10;There have been many approaches to assess the performance of LLMs without human labels, including efforts to assess the performance of task-specific models. \cite{kamath-etal-2020-selective} explored evaluating fine-tuned question answering models on out of domain data, relevant to question answering problems. More recently, \cite{fu2023estimating} creates a meta-model responsible for predicting the accuracy of the LLM model using the model’s confidence scores as features. Methods from the computer vision (CV) domain to assess unlabeled data more generally have, for example, proposed the average threshold confidence method that learns a threshold over the model’s confidence, predicting accuracy as the fraction of unlabeled examples exceeding that threshold \cite{DBLP:conf/iclr/GargBLNS22}, or iteratively learn an ensemble of models to identify misclassified data points and perform self-training to improve the ensemble with the identified points \cite{chen2021detecting}. However, the metrics and hyperparameters in previous works are specifically for classification tasks and cannot be easily extended to more complex tasks. &#10;&#10;We propose adapting disagreement scores in \cite{JiangNBK22, kirsch2022note}, also from the CV domain, to assess model quality for these supervised NLP tasks. A disagreement score is computed by first training a well-calibrated ensemble of models and then measuring how similar their respective predictions are on the same input. The intuition is that models will agree on highly confident (likely correct) predictions and disagree on less confident (likely wrong) predictions.&#10;One way to develop a well calibrated ensemble is to train the same model on the same dataset but changing initial random seed among the ensemble members, as proposed in \cite{JiangNBK22} for the CV domain. &#10;&#10;In this paper, we adapt the same approach for the NLP tasks to understand the prediction performance across different domains (survey responses, conversation text, and social media chats) and languages. Inspired by the latest work on LLMs, as another alternative to human labeling, we explore leveraging a few-shot GPT-4 as an oracle model to provide a ‘silver label’. We find that disagreement scores of a well-calibrated ensemble work better at predicting a single model's performance for a complex keyphrase extraction task (KPE) than GPT-4 as an oracle model. Our evaluation comparing XLM-Roberta \cite{conneau-etal-2020-unsupervised}, GPT-3 \cite{brown2020language}, and GPT-4 models \cite{openai2023gpt} shows that disagreement scores provide estimation of model performance with mean average error (MAE) as low as 0.4\% and on average 13.8\% better than using silver labels. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Human Labeling Alternatives, Linguistics, Language Model Evaluation, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2006.09462" label="2006.09462">
        <attvalues>
          <attvalue for="0" value="Selective Question Answering under Domain Shift" />
          <attvalue for="1" value="  To avoid giving wrong answers, question answering (QA) models need to know&#10;when to abstain from answering. Moreover, users often ask questions that&#10;diverge from the model's training data, making errors more likely and thus&#10;abstention more critical. In this work, we propose the setting of selective&#10;question answering under domain shift, in which a QA model is tested on a&#10;mixture of in-domain and out-of-domain data, and must answer (i.e., not abstain&#10;on) as many questions as possible while maintaining high accuracy. Abstention&#10;policies based solely on the model's softmax probabilities fare poorly, since&#10;models are overconfident on out-of-domain inputs. Instead, we train a&#10;calibrator to identify inputs on which the QA model errs, and abstain when it&#10;predicts an error is likely. Crucially, the calibrator benefits from observing&#10;the model's behavior on out-of-domain data, even if from a different domain&#10;than the test data. We combine this method with a SQuAD-trained QA model and&#10;evaluate on mixtures of SQuAD and five other QA datasets. Our method answers&#10;56% of questions while maintaining 80% accuracy; in contrast, directly using&#10;the model's probabilities only answers 48% at 80% accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14802" label="2305.14802">
        <attvalues>
          <attvalue for="0" value="Estimating Large Language Model Capabilities without Labeled Test Data" />
          <attvalue for="1" value="  Large Language Models (LLMs) have the impressive ability to perform&#10;in-context learning (ICL) from only a few examples, but the success of ICL&#10;varies widely from task to task. Thus, it is important to quickly determine&#10;whether ICL is applicable to a new task, but directly evaluating ICL accuracy&#10;can be expensive in situations where test data is expensive to annotate -- the&#10;exact situations where ICL is most appealing. In this paper, we propose the&#10;task of ICL accuracy estimation, in which we predict the accuracy of an LLM&#10;when doing in-context learning on a new task given only unlabeled test data for&#10;that task. To perform ICL accuracy estimation, we propose a method that trains&#10;a meta-model using LLM confidence scores as features. We compare our method to&#10;several strong accuracy estimation baselines on a new benchmark that covers 4&#10;LLMs and 3 task collections. The meta-model improves over all baselines across&#10;8 out of 12 settings and achieves the same estimation performance as directly&#10;evaluating on 40 collected labeled test examples per task. At the same time, no&#10;existing approach provides an accurate and reliable ICL accuracy estimation in&#10;every setting, highlighting the need for better ways to measure the uncertainty&#10;of LLM predictions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.04234" label="2201.04234">
        <attvalues>
          <attvalue for="0" value="Leveraging Unlabeled Data to Predict Out-of-Distribution Performance" />
          <attvalue for="1" value="  Real-world machine learning deployments are characterized by mismatches&#10;between the source (training) and target (test) distributions that may cause&#10;performance drops. In this work, we investigate methods for predicting the&#10;target domain accuracy using only labeled source data and unlabeled target&#10;data. We propose Average Thresholded Confidence (ATC), a practical method that&#10;learns a threshold on the model's confidence, predicting accuracy as the&#10;fraction of unlabeled examples for which model confidence exceeds that&#10;threshold. ATC outperforms previous methods across several model architectures,&#10;types of distribution shifts (e.g., due to synthetic corruptions, dataset&#10;reproduction, or novel subpopulations), and datasets (Wilds, ImageNet, Breeds,&#10;CIFAR, and MNIST). In our experiments, ATC estimates target performance&#10;$2$-$4\times$ more accurately than prior methods. We also explore the&#10;theoretical foundations of the problem, proving that, in general, identifying&#10;the accuracy is just as hard as identifying the optimal predictor and thus, the&#10;efficacy of any method rests upon (perhaps unstated) assumptions on the nature&#10;of the shift. Finally, analyzing our method on some toy distributions, we&#10;provide insights concerning when it works. Code is available at&#10;https://github.com/saurabhgarg1996/ATC_code/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.13799" label="2106.13799">
        <attvalues>
          <attvalue for="0" value="Assessing Generalization of SGD via Disagreement" />
          <attvalue for="1" value="  We empirically show that the test error of deep networks can be estimated by&#10;simply training the same architecture on the same training set but with a&#10;different run of Stochastic Gradient Descent (SGD), and measuring the&#10;disagreement rate between the two networks on unlabeled test data. This builds&#10;on -- and is a stronger version of -- the observation in Nakkiran &amp; Bansal '20,&#10;which requires the second run to be on an altogether fresh training set. We&#10;further theoretically show that this peculiar phenomenon arises from the&#10;\emph{well-calibrated} nature of \emph{ensembles} of SGD-trained models. This&#10;finding not only provides a simple empirical measure to directly predict the&#10;test error using unlabeled test data, but also establishes a new conceptual&#10;connection between generalization and calibration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.03025" label="2310.03025">
        <attvalues>
          <attvalue for="0" value="Retrieval meets Long Context Large Language Models" />
          <attvalue for="1" value="  Extending the context window of large language models (LLMs) is getting&#10;popular recently, while the solution of augmenting LLMs with retrieval has&#10;existed for years. The natural questions are: i) Retrieval-augmentation versus&#10;long context window, which one is better for downstream tasks? ii) Can both&#10;methods be combined to get the best of both worlds? In this work, we answer&#10;these questions by studying both solutions using two state-of-the-art&#10;pretrained LLMs, i.e., a proprietary 43B GPT and Llama2-70B. Perhaps&#10;surprisingly, we find that LLM with 4K context window using simple&#10;retrieval-augmentation at generation can achieve comparable performance to&#10;finetuned LLM with 16K context window via positional interpolation on long&#10;context tasks, while taking much less computation. More importantly, we&#10;demonstrate that retrieval can significantly improve the performance of LLMs&#10;regardless of their extended context window sizes. Our best model,&#10;retrieval-augmented Llama2-70B with 32K context window, outperforms&#10;GPT-3.5-turbo-16k and Davinci003 in terms of average score on nine long context&#10;tasks including question answering, query-based summarization, and in-context&#10;few-shot learning tasks. It also outperforms its non-retrieval Llama2-70B-32k&#10;baseline by a margin, while being much faster at generation. Our study provides&#10;general insights on the choice of retrieval-augmentation versus long context&#10;extension of LLM for practitioners.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;In this section, we discuss the related work in long context LLM, efficient attention methods, and retrieval-augmented language models.&#10;&#10;" />
          <attvalue for="4" value="Retrieval Augmentation Techniques, Computer Science, Context Window Extension, Linguistics, Cognitive Science, Artificial Intelligence, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2105.11174" label="2105.11174">
        <attvalues>
          <attvalue for="0" value="Retrieval Enhanced Model for Commonsense Generation" />
          <attvalue for="1" value="  Commonsense generation is a challenging task of generating a plausible&#10;sentence describing an everyday scenario using provided concepts. Its&#10;requirement of reasoning over commonsense knowledge and compositional&#10;generalization ability even puzzles strong pre-trained language generation&#10;models. We propose a novel framework using retrieval methods to enhance both&#10;the pre-training and fine-tuning for commonsense generation. We retrieve&#10;prototype sentence candidates by concept matching and use them as auxiliary&#10;input. For fine-tuning, we further boost its performance with a trainable&#10;sentence retriever. We demonstrate experimentally on the large-scale CommonGen&#10;benchmark that our approach achieves new state-of-the-art results.&#10;" />
          <attvalue for="2" value="&#10;The understanding of commonsense knowledge in human language has been acknowledged as a critical component for artificial intelligence systems.&#10;In recent years, many new tasks and datasets are proposed to assess NLP model's ability of commonsense reasoning \cite{yu2020survey}.&#10;SWAG~\cite{zellers-etal-2018-swag} is a task of inferring the upcoming event based on a partial description using commonsense.&#10;CommonsenseQA~\cite{talmor-etal-2019-commonsenseqa} is a commonsense question answering dataset built from ConceptNet.&#10;Recently, \cite{lin-etal-2020-commongen} propose CommonGen, a new challenge for evaluating model's ability of generative commonsense reasoning.&#10;&#10;CommonGen requires the system to construct a plausible sentence based on several concepts related to an everyday scenario.&#10;Two examples for this task are shown in Table~\ref{tab:examples}.&#10;The task is challenging because the system needs to organize provided concepts into the most plausible scenario, avoid violation of commonsense, and ensure the generated sentence is grammatically correct.&#10;Existing approaches fine-tune pre-trained encoder-decoder models for description construction with concatenated concepts as input.&#10;&#10;\cite{fan-etal-2020-enhanced} propose a retrieve-and-generation method for commonsense generation which uses a prototype candidate sentence as auxiliary input.&#10;However, their retriever is non-trainable and only works for the fine-tuning process.&#10;In this work, we extend this idea and propose a novel framework for commonsense generation by using retrieval method for enhancing both the pre-training and fine-tuning stages.&#10;Furthermore, we design a trainable prototype sentence retriever to further boost generation performance.&#10;&#10;We conduct experiments on CommonGen~\cite{lin-etal-2020-commongen} benchmark dataset. &#10;It contains 35,141 concept sets and 79,051 corresponding sentences.&#10;Each concept set is mapped to multiple corresponding sentences.&#10;Without any model modification or complex fusion of knowledge graphs, our approach achieves new state-of-the-art results on CommonGen on several metrics, including BLEU, CIDEr and SPICE.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Retrieval Methods, Linguistics, Cognitive Science, Language Models, Artificial Intelligence, Commonsense Generation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1808.05326" label="1808.05326">
        <attvalues>
          <attvalue for="0" value="SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense&#10;  Inference" />
          <attvalue for="1" value="  Given a partial description like &quot;she opened the hood of the car,&quot; humans can&#10;reason about the situation and anticipate what might come next (&quot;then, she&#10;examined the engine&quot;). In this paper, we introduce the task of grounded&#10;commonsense inference, unifying natural language inference and commonsense&#10;reasoning.&#10;  We present SWAG, a new dataset with 113k multiple choice questions about a&#10;rich spectrum of grounded situations. To address the recurring challenges of&#10;the annotation artifacts and human biases found in many existing datasets, we&#10;propose Adversarial Filtering (AF), a novel procedure that constructs a&#10;de-biased dataset by iteratively training an ensemble of stylistic classifiers,&#10;and using them to filter the data. To account for the aggressive adversarial&#10;filtering, we use state-of-the-art language models to massively oversample a&#10;diverse set of potential counterfactuals. Empirical results demonstrate that&#10;while humans can solve the resulting inference problems with high accuracy&#10;(88%), various competitive models struggle on our task. We provide&#10;comprehensive analysis that indicates significant opportunities for future&#10;research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03705" label="1911.03705">
        <attvalues>
          <attvalue for="0" value="CommonGen: A Constrained Text Generation Challenge for Generative&#10;  Commonsense Reasoning" />
          <attvalue for="1" value="  Recently, large-scale pre-trained language models have demonstrated&#10;impressive performance on several commonsense-reasoning benchmark datasets.&#10;However, building machines with commonsense to compose realistically plausible&#10;sentences remains challenging. In this paper, we present a constrained text&#10;generation task, CommonGen associated with a benchmark dataset, to explicitly&#10;test machines for the ability of generative commonsense reasoning. Given a set&#10;of common concepts (e.g., {dog, frisbee, catch, throw}); the task is to&#10;generate a coherent sentence describing an everyday scenario using these&#10;concepts (e.g., &quot;a man throws a frisbee and his dog catches it&quot;).&#10;  The CommonGen task is challenging because it inherently requires 1)&#10;relational reasoning with background commonsense knowledge, and 2)&#10;compositional generalization ability to work on unseen concept combinations.&#10;Our dataset, constructed through a combination of crowdsourced and existing&#10;caption corpora, consists of 79k commonsense descriptions over 35k unique&#10;concept-sets. Experiments show that there is a large gap between&#10;state-of-the-art text generation models (e.g., T5) and human performance.&#10;Furthermore, we demonstrate that the learned generative commonsense reasoning&#10;capability can be transferred to improve downstream tasks such as CommonsenseQA&#10;by generating additional context.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.00366" label="2012.00366">
        <attvalues>
          <attvalue for="0" value="An Enhanced Knowledge Injection Model for Commonsense Generation" />
          <attvalue for="1" value="  Commonsense generation aims at generating plausible everyday scenario&#10;description based on a set of provided concepts. Digging the relationship of&#10;concepts from scratch is non-trivial, therefore, we retrieve prototypes from&#10;external knowledge to assist the understanding of the scenario for better&#10;description generation. We integrate two additional modules, namely position&#10;indicator and scaling module, into the pretrained encoder-decoder model for&#10;prototype modeling to enhance the knowledge injection procedure. We conduct&#10;experiment on CommonGen benchmark, and experimental results show that our&#10;method significantly improves the performance on all the metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.11624" label="2401.11624">
        <attvalues>
          <attvalue for="0" value="In-context Learning with Retrieved Demonstrations for Language Models: A&#10;  Survey" />
          <attvalue for="1" value="  Language models, especially pre-trained large language models, have showcased&#10;remarkable abilities as few-shot in-context learners (ICL), adept at adapting&#10;to new tasks with just a few demonstrations in the input context. However, the&#10;model's ability to perform ICL is sensitive to the choice of the few-shot&#10;demonstrations. Instead of using a fixed set of demonstrations, one recent&#10;development is to retrieve demonstrations tailored to each input query. The&#10;implementation of demonstration retrieval is relatively straightforward,&#10;leveraging existing databases and retrieval systems. This not only improves the&#10;efficiency and scalability of the learning process but also has been shown to&#10;reduce biases inherent in manual example selection. In light of the encouraging&#10;results and growing research in ICL with retrieved demonstrations, we conduct&#10;an extensive review of studies in this area. In this survey, we discuss and&#10;compare different design choices for retrieval models, retrieval training&#10;procedures, and inference algorithms.&#10;" />
          <attvalue for="2" value="&#10;&#10;Few-shot in-context learning (ICL) is the ability of large language models (LLMs) to perform a new task when a few input-output examples, or demonstrations, for the new task are given alongside the actual task input.&#10;Importantly, the model parameters do not have to be fine-tuned towards the new task.&#10;ICL is popularized by the work on pre-trained large language models, which can perform ICL without being trained to do so~\cite{brown2020language},&#10;though smaller language models can also be explicitly trained to perform ICL~\cite{min2022metaicl}.&#10;&#10;ICL presents several advantages over the conventional methodology for adapting language models to a downstream task, which typically involves initial pre-training followed by subsequent fine-tuning. One significant merit of ICL is the circumvention of fine-tuning, which might not always be possible due to limited access to the model parameters or constraints on computational resources~\cite{brown2020language}. Furthermore, ICL avoids common issues associated with fine-tuning, such as overfitting\cite{ying2019overview,kazemi2023understanding}. Compared to parameter-efficient fine-tuning methods (PEFT)~\cite{hu2021lora,dettmers2023qlora,lester2021power}, ICL is computationally cheaper and remain the model parameters unchanged thus preserving the generality of the LLMs. &#10;&#10;Early ICL implementations use a fixed set of demonstrations for each target task. These demonstrations could be hand-crafted by human~\cite{hendrycks2021measuring,wei2022chain,kazemi2023lambada}, randomly chosen from training data~\cite{brown2020language,lewkowycz2022solving}. &#10;Beyond random selection, there are more advanced selection processes based on metrics such as complexity~\cite{fu2022complexity}, diversity~\cite{li2023finding}, difficulty~\cite{drozdov2023parade}, concept learning~\cite{wang2023large} and perplexity~\cite{gonen2023demystifying}. &#10;Importantly, the demonstrations remain context-insensitive (i.e. the same demonstrations are used regardless of the query) which could hinder unlocking the true potential of the LLMs. &#10;The effectiveness of such demonstrations is influenced by factors such as the quality, quantity, and ordering of the demonstrations.&#10;&#10;Retrieval-based ICL (RetICL) presents a paradigm shift in the optimization of language model performance, moving beyond static, pre-defined demonstration sets to a dynamic, context-sensitive approach. At the heart of this innovation is the concept of adaptive demonstration selection, where a specialized retriever intelligently curates tailored demonstrations for each specific task input. This method has not only consistently outshined approaches relying on random or static hand-crafted demonstrations but has also demonstrated a remarkable resilience to a variety of influencing factors.&#10;&#10;The efficacy of RetICL\ pivots on the ``relevance'' and ``usefulness'' of the demonstrations it selects, a process intricately influenced by multiple elements. These include the nature of the retriever—ranging from general off-the-shelf models to finely-tuned, domain-specific variants—the source and diversity of the retrieval corpus, the retriever's objectives (focusing on either similarity or diversity), and the strategies for integrating multiple demonstrations.&#10;Over the past two years, numerous and sometimes concurrent works have studied RetICL\ each with different terminology and with variations in problem definition and subsequent methodologies, making it difficult to comprehend the current state of research and practice in RetICL, especially for newcomers to the field. In this comprehensive survey, we meticulously analyze 22 seminal papers in the field of RetICL, as detailed in Table~\ref{tab:related_work}, and provide a categorization of their main building blocks (See Figure \ref{fig:ret-survey}). Our work not only provides a thorough synthesis of existing research but also underscores the areas where RetICL\ significantly surpasses previous ICL methods, and illuminates many paths forward for future innovations in this area, thus serving as a critical resource for ICL.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Few-Shot Learning Methods, Artificial Intelligence, Natural Language Processing, Demonstration Retrieval Systems, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2110.15943" label="2110.15943">
        <attvalues>
          <attvalue for="0" value="MetaICL: Learning to Learn In Context" />
          <attvalue for="1" value="  We introduce MetaICL (Meta-training for In-Context Learning), a new&#10;meta-training framework for few-shot learning where a pretrained language model&#10;is tuned to do in-context learning on a large set of training tasks. This&#10;meta-training enables the model to more effectively learn a new task in context&#10;at test time, by simply conditioning on a few training examples with no&#10;parameter updates or task-specific templates. We experiment on a large, diverse&#10;collection of tasks consisting of 142 NLP datasets including classification,&#10;question answering, natural language inference, paraphrase detection and more,&#10;across seven different meta-training/target splits. MetaICL outperforms a range&#10;of baselines including in-context learning without meta-training and multi-task&#10;learning followed by zero-shot transfer. We find that the gains are&#10;particularly significant for target tasks that have domain shifts from the&#10;meta-training tasks, and that using a diverse set of the meta-training tasks is&#10;key to improvements. We also show that MetaICL approaches (and sometimes beats)&#10;the performance of models fully finetuned on the target task, and outperforms&#10;much bigger models with nearly 8x parameters. Finally, we show that MetaICL is&#10;complementary to human-written instructions, and the best performance can be&#10;achieved by combining both approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.11293" label="2301.11293">
        <attvalues>
          <attvalue for="0" value="Understanding Finetuning for Factual Knowledge Extraction from Language&#10;  Models" />
          <attvalue for="1" value="  Language models (LMs) pretrained on large corpora of text from the web have&#10;been observed to contain large amounts of various types of knowledge about the&#10;world. This observation has led to a new and exciting paradigm in knowledge&#10;graph construction where, instead of manual curation or text mining, one&#10;extracts knowledge from the parameters of an LM. Recently, it has been shown&#10;that finetuning LMs on a set of factual knowledge makes them produce better&#10;answers to queries from a different set, thus making finetuned LMs a good&#10;candidate for knowledge extraction and, consequently, knowledge graph&#10;construction. In this paper, we analyze finetuned LMs for factual knowledge&#10;extraction. We show that along with its previously known positive effects,&#10;finetuning also leads to a (potentially harmful) phenomenon which we call&#10;Frequency Shock, where at the test time the model over-predicts rare entities&#10;that appear in the training set and under-predicts common entities that do not&#10;appear in the training set enough times. We show that Frequency Shock leads to&#10;a degradation in the predictions of the model and beyond a point, the harm from&#10;Frequency Shock can even outweigh the positive effects of finetuning, making&#10;finetuning harmful overall. We then consider two solutions to remedy the&#10;identified negative effect: 1- model mixing and 2- mixture finetuning with the&#10;LM's pre-training task. The two solutions combined lead to significant&#10;improvements compared to vanilla finetuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14314" label="2305.14314">
        <attvalues>
          <attvalue for="0" value="QLoRA: Efficient Finetuning of Quantized LLMs" />
          <attvalue for="1" value="  We present QLoRA, an efficient finetuning approach that reduces memory usage&#10;enough to finetune a 65B parameter model on a single 48GB GPU while preserving&#10;full 16-bit finetuning task performance. QLoRA backpropagates gradients through&#10;a frozen, 4-bit quantized pretrained language model into Low Rank&#10;Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all&#10;previous openly released models on the Vicuna benchmark, reaching 99.3% of the&#10;performance level of ChatGPT while only requiring 24 hours of finetuning on a&#10;single GPU. QLoRA introduces a number of innovations to save memory without&#10;sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is&#10;information theoretically optimal for normally distributed weights (b) double&#10;quantization to reduce the average memory footprint by quantizing the&#10;quantization constants, and (c) paged optimziers to manage memory spikes. We&#10;use QLoRA to finetune more than 1,000 models, providing a detailed analysis of&#10;instruction following and chatbot performance across 8 instruction datasets,&#10;multiple model types (LLaMA, T5), and model scales that would be infeasible to&#10;run with regular finetuning (e.g. 33B and 65B parameter models). Our results&#10;show that QLoRA finetuning on a small high-quality dataset leads to&#10;state-of-the-art results, even when using smaller models than the previous&#10;SoTA. We provide a detailed analysis of chatbot performance based on both human&#10;and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable&#10;alternative to human evaluation. Furthermore, we find that current chatbot&#10;benchmarks are not trustworthy to accurately evaluate the performance levels of&#10;chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to&#10;ChatGPT. We release all of our models and code, including CUDA kernels for&#10;4-bit training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.03874" label="2103.03874">
        <attvalues>
          <attvalue for="0" value="Measuring Mathematical Problem Solving With the MATH Dataset" />
          <attvalue for="1" value="  Many intellectual endeavors require mathematical problem solving, but this&#10;skill remains beyond the capabilities of computers. To measure this ability in&#10;machine learning models, we introduce MATH, a new dataset of 12,500 challenging&#10;competition mathematics problems. Each problem in MATH has a full step-by-step&#10;solution which can be used to teach models to generate answer derivations and&#10;explanations. To facilitate future research and increase accuracy on MATH, we&#10;also contribute a large auxiliary pretraining dataset which helps teach models&#10;the fundamentals of mathematics. Even though we are able to increase accuracy&#10;on MATH, our results show that accuracy remains relatively low, even with&#10;enormous Transformer models. Moreover, we find that simply increasing budgets&#10;and model parameter counts will be impractical for achieving strong&#10;mathematical reasoning if scaling trends continue. While scaling Transformers&#10;is automatically solving most other text-based tasks, scaling is not currently&#10;solving MATH. To have more traction on mathematical problem solving we will&#10;likely need new algorithmic advancements from the broader research community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.13894" label="2212.13894">
        <attvalues>
          <attvalue for="0" value="LAMBADA: Backward Chaining for Automated Reasoning in Natural Language" />
          <attvalue for="1" value="  Remarkable progress has been made on automated reasoning with natural text,&#10;by using Language Models (LMs) and methods such as Chain-of-Thought and&#10;Selection-Inference. These techniques search for proofs in the forward&#10;direction from axioms to the conclusion, which suffers from a combinatorial&#10;explosion of the search space, and thus high failure rates for problems&#10;requiring longer chains of reasoning. The classical automated reasoning&#10;literature has shown that reasoning in the backward direction (i.e. from the&#10;intended conclusion to supporting axioms) is significantly more efficient at&#10;proof-finding. Importing this intuition into the LM setting, we develop a&#10;Backward Chaining algorithm, called LAMBADA, that decomposes reasoning into&#10;four sub-modules. These sub-modules are simply implemented by few-shot prompted&#10;LM inference. We show that LAMBADA achieves sizable accuracy boosts over&#10;state-of-the-art forward reasoning methods on challenging logical reasoning&#10;datasets, particularly when deep and accurate proof chains are required.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.00720" label="2210.00720">
        <attvalues>
          <attvalue for="0" value="Complexity-Based Prompting for Multi-Step Reasoning" />
          <attvalue for="1" value="  We study the task of prompting large-scale language models to perform&#10;multi-step reasoning. Existing work shows that when prompted with a chain of&#10;thoughts (CoT), sequences of short sentences describing intermediate reasoning&#10;steps towards a final answer, large language models can generate new reasoning&#10;chains and predict answers for new inputs. A central question is which&#10;reasoning examples make the most effective prompts. In this work, we propose&#10;complexity-based prompting, a simple and effective example selection scheme for&#10;multi-step reasoning. We show that prompts with higher reasoning complexity,&#10;i.e., chains with more reasoning steps, achieve substantially better&#10;performance on multi-step reasoning tasks over strong baselines. We further&#10;extend our complexity-based criteria from prompting (selecting inputs) to&#10;decoding (selecting outputs), where we sample multiple reasoning chains from&#10;the model, then choose the majority of generated answers from complex reasoning&#10;chains (over simple chains). When used to prompt GPT-3 and Codex, our approach&#10;substantially improves multi-step reasoning accuracy and achieves new&#10;state-of-the-art (SOTA) performance on three math benchmarks (GSM8K,&#10;MultiArith, and MathQA) and two BigBenchHard tasks (Date Understanding and&#10;Penguins), with an average +5.3 and up to +18 accuracy improvements. Compared&#10;with existing example selection schemes like manual tuning or retrieval-based&#10;selection, selection based on reasoning complexity is intuitive, easy to&#10;implement, and annotation-efficient. Further results demonstrate the robustness&#10;of performance gains from complex prompts under format perturbation and&#10;distribution shift.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.14408" label="2310.14408">
        <attvalues>
          <attvalue for="0" value="PaRaDe: Passage Ranking using Demonstrations with Large Language Models" />
          <attvalue for="1" value="  Recent studies show that large language models (LLMs) can be instructed to&#10;effectively perform zero-shot passage re-ranking, in which the results of a&#10;first stage retrieval method, such as BM25, are rated and reordered to improve&#10;relevance. In this work, we improve LLM-based re-ranking by algorithmically&#10;selecting few-shot demonstrations to include in the prompt. Our analysis&#10;investigates the conditions where demonstrations are most helpful, and shows&#10;that adding even one demonstration is significantly beneficial. We propose a&#10;novel demonstration selection strategy based on difficulty rather than the&#10;commonly used semantic similarity. Furthermore, we find that demonstrations&#10;helpful for ranking are also effective at question generation. We hope our work&#10;will spur more principled research into question generation and passage&#10;ranking.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Concurrent with UPR, PromptRank \cite{khalifa2023fewshot} is the most related prior work, using demonstrations to re-rank ``document paths'' for multihop-QA. Details of how they select demonstrations is unclear, motivating us to conduct our own study. &#10;&#10;Our difficulty-based demonstration selection (\S\ref{sec:uncertainty}) is closely related to active learning \cite{Dagan1995CommitteeBasedSF,Roy2001TowardOA,Settles2009ActiveLL}. Similarly, \cite{diao2023active} measure uncertainty with generation instead of scoring. \cite{zhang-etal-2022-active} formulate demonstration selection as a reinforcement learning problem. \cite{rubin-etal-2022-learning} use LLM-scoring to find hard negatives for their trained demonstration retriever. Others explore demonstration ordering~\cite{lu-etal-2022-fantastically} and joint selection~\cite{drozdov2023compositional,levy-etal-2023-diverse,Agrawal2022IncontextES,Ye2023CompositionalEF}. Concurrent to our work, \cite{li2023finding} perform multiple rounds of hill climbing to find groups of demonstrations that perform well according to a validation set. In contrast, DBS selects demonstrations directly and does not rely on validation.&#10;&#10;Discriminative methods are widely used in supervised &#10;ranking~\cite{zhuang2022rankt5,nogueira-dos-santos-etal-2020-beyond,hui2022ed2lm}.&#10;Listwise prompting is an alternative to query likelihood, but requires a sliding window strategy as not all documents fit in the context \cite{Ma2023ZeroShotLD,Sun2023IsCG}.&#10;Rather than query likelihood, HyDE \cite{Gao2022PreciseZD} achieves zero-shot ranking through document generation, which we hypothesize would be improved through demonstrations. &#10;PaRaDe is bounded by the first stage BM25 retrieval, and it may be fruitful to explore approaches that align first stage retrieval with our demonstration-based approach \cite{yadav-etal-2022-efficient}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2404.13760" label="2404.13760">
        <attvalues>
          <attvalue for="0" value="How to Encode Domain Information in Relation Classification" />
          <attvalue for="1" value="  Current language models require a lot of training data to obtain high&#10;performance. For Relation Classification (RC), many datasets are&#10;domain-specific, so combining datasets to obtain better performance is&#10;non-trivial. We explore a multi-domain training setup for RC, and attempt to&#10;improve performance by encoding domain information. Our proposed models improve&#10;&gt; 2 Macro-F1 against the baseline setup, and our analysis reveals that not all&#10;the labels benefit the same: The classes which occupy a similar space across&#10;domains (i.e., their interpretation is close across them, for example&#10;&quot;physical&quot;) benefit the least, while domain-dependent relations (e.g.,&#10;&quot;part-of'') improve the most when encoding domain information.&#10;" />
          <attvalue for="2" value="&#10;&#10;Relation Classification (RC) is the task of identifying the semantic relation between two given entities. The task is beneficial for many different downstream tasks which involve Natural Language Understanding. For example, question answering, knowledge base population, or summarization.&#10;In addition to the wide variety of downstream applications, as most information extraction tasks, RC is topic-specific, meaning that depending on the topic the information to extract can vary a lot. For example, in the music domain we may want to extract that a song is included in a musical album, while in the politics domain we may have a politician winning a political election.&#10;While current deep learning models require a lot of training data, collecting and annotating text from every domain is time-consuming and expensive.&#10;&#10;In this project, we explore the critical setup of multi-domain training with the aim of identifying the best setup for maximizing the training data (by including data coming from different domains), without losing domain-specific information.&#10;To do so, we compare multiple ways of enriching the input instances with domain information (see Section~\ref{sec:domains}).&#10;&#10;Encoding information about where a certain utterance originates from has been previously explored in other Natural Language Processing fields.&#10;In the multi-lingual space,~\cite{xlmr} exploited language embeddings for multi-lingual model training.&#10;\cite{ammar-etal-2016-many} first proposed to use language embeddings for training a multi-lingual syntactic parser for seven European languages, and showed improved performance.&#10;Later work also successfully trained parsers with the so called treebank embeddings for datasets within the same language~\cite{stymne-etal-2018-parser} or language family~\cite{smith-etal-2018-82}. &#10;Other work have used special language ids to mark the language of each instance in the context of machine translation~\cite{liu-etal-2020-multilingual-denoising}.&#10;To the best of our knowledge, these approaches have been exploited mostly in multi-lingual setups and syntactic tasks. In this work, we explore a gap and test their effectiveness for encoding domain information in a semantic setup: Relation Classification. &#10;We compare ``dataset embeddings'' and ``domain markers'' from previous work with a new approach exploiting domain-specific entity types.&#10;Our contributions are:&#10;&#10;\begin{itemize}&#10; \item CrossRE 2.0, an extension of the CrossRE dataset~\cite{bassignana-plank-2022-crossre} with 3.3k new annotations in the news domain in order to balance data across domains;&#10; \item We propose the first multi-domain training baseline on CrossRE;&#10; \item We test previous work for encoding dataset information in RC, and compare it with a new RC-specific technique; We present an in-depth analysis of the results obtained.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Training, Domain Adaptation, Artificial Intelligence, Relation Classification" />
        </attvalues>
      </node>
      <node id="1809.02237" label="1809.02237">
        <attvalues>
          <attvalue for="0" value="82 Treebanks, 34 Models: Universal Dependency Parsing with&#10;  Multi-Treebank Models" />
          <attvalue for="1" value="  We present the Uppsala system for the CoNLL 2018 Shared Task on universal&#10;dependency parsing. Our system is a pipeline consisting of three components:&#10;the first performs joint word and sentence segmentation; the second predicts&#10;part-of- speech tags and morphological features; the third predicts dependency&#10;trees from words and tags. Instead of training a single parsing model for each&#10;treebank, we trained models with multiple treebanks for one language or closely&#10;related languages, greatly reducing the number of models. On the official test&#10;run, we ranked 7th of 27 teams for the LAS and MLAS metrics. Our system&#10;obtained the best scores overall for word segmentation, universal POS tagging,&#10;and morphological features.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.09345" label="2210.09345">
        <attvalues>
          <attvalue for="0" value="CrossRE: A Cross-Domain Dataset for Relation Extraction" />
          <attvalue for="1" value="  Relation Extraction (RE) has attracted increasing attention, but current RE&#10;evaluation is limited to in-domain evaluation setups. Little is known on how&#10;well a RE system fares in challenging, but realistic out-of-distribution&#10;evaluation setups. To address this gap, we propose CrossRE, a new,&#10;freely-available cross-domain benchmark for RE, which comprises six distinct&#10;text domains and includes multi-label annotations. An additional innovation is&#10;that we release meta-data collected during annotation, to include explanations&#10;and flags of difficult instances. We provide an empirical evaluation with a&#10;state-of-the-art model for relation classification. As the meta-data enables us&#10;to shed new light on the state-of-the-art model, we provide a comprehensive&#10;analysis on the impact of difficult cases and find correlations between model&#10;and human annotations. Overall, our empirical investigation highlights the&#10;difficulty of cross-domain RE. We release our dataset, to spur more research in&#10;this direction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.11058" label="2109.11058">
        <attvalues>
          <attvalue for="0" value="Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese&#10;  Language Models" />
          <attvalue for="1" value="  Prior work has shown that structural supervision helps English language&#10;models learn generalizations about syntactic phenomena such as subject-verb&#10;agreement. However, it remains unclear if such an inductive bias would also&#10;improve language models' ability to learn grammatical dependencies in&#10;typologically different languages. Here we investigate this question in&#10;Mandarin Chinese, which has a logographic, largely syllable-based writing&#10;system; different word order; and sparser morphology than English. We train&#10;LSTMs, Recurrent Neural Network Grammars, Transformer language models, and&#10;Transformer-parameterized generative parsing models on two Mandarin Chinese&#10;datasets of different sizes. We evaluate the models' ability to learn different&#10;aspects of Mandarin grammar that assess syntactic and semantic relationships.&#10;We find suggestive evidence that structural supervision helps with representing&#10;syntactic state across intervening content and improves performance in low-data&#10;settings, suggesting that the benefits of hierarchical inductive biases in&#10;acquiring dependency relationships may extend beyond English.&#10;" />
          <attvalue for="2" value="&#10;&#10;A rich collection of targeted linguistic evaluations has shown that neural language models can surprisingly learn many aspects of grammar from unlabeled linguistic input \cite[e.g.,][]{linzen-etal-2016-assessing,gulordava-etal-2018-colorless,warstadt2020blimp,hu-etal-2020-systematic,xiang2021climp}.&#10;There is also growing evidence that explicit modeling of syntax helps neural network-based language models represent syntactic state and exhibit human-like processing behaviors of non-local grammatical dependencies, including number agreement \cite{kuncoro-etal-2018-lstms}, negative polarity licensing, filler-gap dependencies \cite{wilcox-etal-2019-structural, hu-etal-2020-systematic}, and garden-path effects \cite{futrell-etal-2019-neural,hu-etal-2020-systematic}. However, this line of research has focused primarily on the syntax of English. It is unclear to what extent structural supervision may help neural language models generalize for languages with differing typologies. Expanding these analyses beyond English has the potential to inform scientific questions about inductive biases for language acquisition, as well as practical questions about model architectures that approach language-independence \cite{bender_achieving_2011}.&#10;&#10;Here, we perform a controlled case study of grammatical knowledge in Mandarin Chinese language models.&#10;The orthography and grammar of Chinese provide a useful testing ground given the differences from English and other Indo-European languages (see, e.g., \cite{shopen_language_1985,li_chinese_2015}). Whereas today's Indo-European languages like English generally use phone-based orthography, Chinese uses a logographic system where each character generally responds to a syllable. Most Mandarin Chinese words are one or two syllables, influencing the distribution of tokens. Grammatically, Chinese has almost no inflectional morphology, and corpus studies suggest that the average dependency length of Mandarin Chinese sentences is larger than that of English sentences \cite{jiang2015effects}, with potential implications for language modeling. On the one hand, the need to track input across long dependencies may make structural supervision more beneficial for Mandarin Chinese language models; on the other hand, the prevalence of these dependencies may make it easier for them to learn to maintain non-local information without explicitly modeling syntax. &#10;Other fine-grained differences in typology also affect the types of syntactic tests that can be conducted. For example, &#10;since relative clauses precede the head noun in Chinese (unlike in English), we can manipulate the distance of a verb--object dependency by inserting relative clauses in between.&#10;These characteristics motivate our choice of Mandarin Chinese as a language for evaluating structurally supervised neural language models. &#10;&#10;We design six classes of Mandarin test suites covering a range of syntactic and semantic relationships, some specific to Mandarin and some comparable to English. &#10;We train neural language models with differing inductive biases &#10;on two datasets of different sizes, and compare models' performance on our targeted evaluation materials. While most prior work investigating syntactically guided language models has used Recurrent Neural Network Grammar models \cite{dyer2016recurrent} -- potentially conflating structural supervision with a particular parameterization -- this work further explores structured Transformer language models \cite{qian-etal-2021-structural}. Our results are summarized as follows. &#10;We find that structural supervision yields greatest performance advantages in low-data settings, in line with prior work on English language models. &#10;Our results also suggest a potential benefit of structural supervision in deriving garden-path effects induced by local classifier--noun mismatch, and in maintaining syntactic expectations across intervening content within a dependency relation. These findings suggest that the benefits of hierarchical inductive biases in acquiring dependency relationships may not be specific to English. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Linguistic Generalization, Computer Science, Syntactic Dependency Learning, Linguistics, Language Model Training, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1912.00582" label="1912.00582">
        <attvalues>
          <attvalue for="0" value="BLiMP: The Benchmark of Linguistic Minimal Pairs for English" />
          <attvalue for="1" value="  We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP),&#10;a challenge set for evaluating what language models (LMs) know about major&#10;grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each&#10;containing 1000 minimal pairs isolating specific contrasts in syntax,&#10;morphology, or semantics. The data is automatically generated according to&#10;expert-crafted grammars, and aggregate human agreement with the labels is&#10;96.4%. We use it to evaluate n-gram, LSTM, and Transformer (GPT-2 and&#10;Transformer-XL) LMs. We find that state-of-the-art models identify&#10;morphological contrasts reliably, but they struggle with semantic restrictions&#10;on the distribution of quantifiers and negative polarity items and subtle&#10;syntactic phenomena such as extraction islands.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.00943" label="1903.00943">
        <attvalues>
          <attvalue for="0" value="Structural Supervision Improves Learning of Non-Local Grammatical&#10;  Dependencies" />
          <attvalue for="1" value="  State-of-the-art LSTM language models trained on large corpora learn&#10;sequential contingencies in impressive detail and have been shown to acquire a&#10;number of non-local grammatical dependencies with some success. Here we&#10;investigate whether supervision with hierarchical structure enhances learning&#10;of a range of grammatical dependencies, a question that has previously been&#10;addressed only for subject-verb agreement. Using controlled experimental&#10;methods from psycholinguistics, we compare the performance of word-based LSTM&#10;models versus two models that represent hierarchical structure and deploy it in&#10;left-to-right processing: Recurrent Neural Network Grammars (RNNGs) (Dyer et&#10;al., 2016) and a incrementalized version of the Parsing-as-Language-Modeling&#10;configuration from Chariak et al., (2016). Models are tested on a diverse range&#10;of configurations for two classes of non-local grammatical dependencies in&#10;English---Negative Polarity licensing and Filler--Gap Dependencies. Using the&#10;same training data across models, we find that structurally-supervised models&#10;outperform the LSTM, with the RNNG demonstrating best results on both types of&#10;grammatical dependencies and even learning many of the Island Constraints on&#10;the filler--gap dependency. Structural supervision thus provides data&#10;efficiency advantages over purely string-based training of neural language&#10;models in acquiring human-like generalizations about non-local grammatical&#10;dependencies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.00104" label="2108.00104">
        <attvalues>
          <attvalue for="0" value="Structural Guidance for Transformer Language Models" />
          <attvalue for="1" value="  Transformer-based language models pre-trained on large amounts of text data&#10;have proven remarkably successful in learning generic transferable linguistic&#10;representations. Here we study whether structural guidance leads to more&#10;human-like systematic linguistic generalization in Transformer language models&#10;without resorting to pre-training on very large amounts of data. We explore two&#10;general ideas. The &quot;Generative Parsing&quot; idea jointly models the incremental&#10;parse and word sequence as part of the same sequence modeling task. The&#10;&quot;Structural Scaffold&quot; idea guides the language model's representation via&#10;additional structure loss that separately predicts the incremental constituency&#10;parse. We train the proposed models along with a vanilla Transformer language&#10;model baseline on a 14 million-token and a 46 million-token subset of the BLLIP&#10;dataset, and evaluate models' syntactic generalization performances on SG Test&#10;Suites and sized BLiMP. Experiment results across two benchmarks suggest&#10;converging evidence that generative structural supervisions can induce more&#10;robust and humanlike linguistic generalization in Transformer language models&#10;without the need for data intensive pre-training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.13824" label="2303.13824">
        <attvalues>
          <attvalue for="0" value="$k$NN Prompting: Beyond-Context Learning with Calibration-Free Nearest&#10;  Neighbor Inference" />
          <attvalue for="1" value="  In-Context Learning (ICL), which formulates target tasks as prompt completion&#10;conditioned on in-context demonstrations, has become the prevailing utilization&#10;of LLMs. In this paper, we first disclose an actual predicament for this&#10;typical usage that it can not scale up with training data due to context length&#10;restriction. Besides, existing works have shown that ICL also suffers from&#10;various biases and requires delicate calibration treatment. To address both&#10;challenges, we advocate a simple and effective solution, $k$NN Prompting, which&#10;first queries LLM with training data for distributed representations, then&#10;predicts test instances by simply referring to nearest neighbors. We conduct&#10;comprehensive experiments to demonstrate its two-fold superiority: 1)&#10;Calibration-Free: $k$NN Prompting does not directly align LLM output&#10;distribution with task-specific label space, instead leverages such&#10;distribution to align test and training instances. It significantly outperforms&#10;state-of-the-art calibration-based methods under comparable few-shot scenario.&#10;2) Beyond-Context: $k$NN Prompting can further scale up effectively with as&#10;many training data as are available, continually bringing substantial&#10;improvements. The scaling trend holds across 10 orders of magnitude ranging&#10;from 2 shots to 1024 shots as well as different LLMs scales ranging from 0.8B&#10;to 30B. It successfully bridges data scaling into model scaling, and brings new&#10;potentials for the gradient-free paradigm of LLM deployment. Code is publicly&#10;available.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs), when scale up to billions of parameters, have demonstrated remarkable capabilities in a wide range of NLP tasks~\cite{radford2019language,NEURIPS2020_1457c0d6}.&#10;However, such models are prohibitively expensive to train with most of the research- or consumer-level devices, though some of them are already publicly available~\cite{zhang2022opt}.&#10;As a result, it is now an emerging paradigm that LLMs are hosted in a remote data center while accessed by end users or applications via simple API requests.&#10;The typical usage of LLM under such paradigm is In-Context Learning, where LLM reads and completes a prompt sequence as how it is pretrained on massive text corpora.&#10;The prompt is constructed by concatenation of several training examples and a test instance, and the prediction is obtained by mapping the LLM word continuations back to label space.&#10;&#10;It is widely investigated and acknowledged that modern neural networks generally perform better w.r.t. increased training data.&#10;Specifically, there exists a power law between expected model performance and available data scale~\cite{hestness2017deep,rosenfeld2020a}.&#10;For ICL, it is also empirically observed that the performance continually improves when more training examples are prepended into the prompt~\cite{NEURIPS2020_1457c0d6}.&#10;However, such improvements are quickly prevented by the predicament of context length restriction, as language models are designed and trained to only process sequences within a fixed length, which is in fact 1024 or 2048 tokens.&#10;In order to utilize more training data, several works try to select the most relevant examples to compose the prompt before querying LLM~\cite{liu-etal-2022-makes,rubin-etal-2022-learning}, but still only in-context examples can actually participate the LLM inference while most training data are discarded beforehand, thus providing marginal data scaling benefits. Besides, their reliance on external retriever also incurs further complications.&#10;As a consequence, such a situation poses a serious challenge for many practical scenarios where more than a few training data are available.&#10;&#10;Another vulnerability of ICL is the severe bias existed in the output distribution of LLMs, which results in considerable performance degradation~\cite{holtzman-etal-2021-surface} and instability~\cite{lu-etal-2022-fantastically} as shown in existing works.&#10;Accordingly, many have proposed various ways to calibrate the output distribution~\cite{pmlr-v139-zhao21c,jiang-etal-2021-know,min-etal-2022-noisy}. For example,~\cite{pmlr-v139-zhao21c} measure such bias by probing LLM with a &quot;NA&quot; example and record the according prior.&#10;However, as LLMs are pretrained on general-domain natural language, its capability to complete a fabricated prompt is essentially not aligned with downstream task-specific label space.&#10;As a consequence, such calibration-based methods can only alleviate the bias to a limited extent.&#10;&#10;In this paper, we advocate a simple and effective solution, $k$NN Prompting, to address both challenges.&#10;Specifically, we assign training data into a demonstration set and an anchor set.&#10;We append each anchor example into the prompt and query LLM, then instead of aligning word continuations with labels, we collect the language modeling probability as distributed representation and cache it into a local datastore.&#10;At inference time, for each test instance, we similarly obtain its representation and match it against the maintained datastore to make predictions.&#10;In general, the proposed framework enables both calibration-free optimization because it avoids forced input-label alignment, and beyond-context learning because the anchor set allows utilization of unlimited training data.&#10;&#10;We conduct comprehensive experiments using 10 established text classification tasks to demonstrate the significant superiority of $k$NN Prompting across various scenarios and against competitive opponents:&#10;1) Under few shot scenario where training data is very limited and fits in the context, $k$NN Prompting outperforms state-of-the-art calibration-based methods by considerable margin (up to +7.07).&#10;2) Under low resource or fully supervised scenario where training data can not fit in the context, $k$NN Prompting further exhibits its major advantage.&#10;It can effectively scale up with as many training data as are available across 10 orders of magnitude (2 shots$\sim$1024 shots, see Figure~\ref{fig:figure1} for illustration) as well as different LLMs scales (0.8B$\sim$30B).&#10;Specifically, with only 32 shots training data, it dramatically improves ICL by +13.58 in average score at its most, and achieves absolute improvements up to +18.84 under fully supervised setting.&#10;We also provide formal explanation on the intrinsic mechanism of effectiveness, as well as detailed analyses regarding its robustness and choices of design.&#10;Accompanied with these appealing aspects, $k$NN Prompting is in general a promising solution that bridges the benefits of data scaling into model scaling to take the gradient-free paradigm of LLM deployment one step further.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Large Language Models, Computer Science, Calibration Free Scaling, Artificial Intelligence, Mathematics, Prompt Learning Methods" />
        </attvalues>
      </node>
      <node id="1712.00409" label="1712.00409">
        <attvalues>
          <attvalue for="0" value="Deep Learning Scaling is Predictable, Empirically" />
          <attvalue for="1" value="  Deep learning (DL) creates impactful advances following a virtuous recipe:&#10;model architecture search, creating large training data sets, and scaling&#10;computation. It is widely believed that growing training sets and models should&#10;improve accuracy and result in better products. As DL application domains grow,&#10;we would like a deeper understanding of the relationships between training set&#10;size, computational scale, and model accuracy improvements to advance the&#10;state-of-the-art.&#10;  This paper presents a large scale empirical characterization of&#10;generalization error and model size growth as training sets grow. We introduce&#10;a methodology for this measurement and test four machine learning domains:&#10;machine translation, language modeling, image processing, and speech&#10;recognition. Our empirical results show power-law generalization error scaling&#10;across a breadth of factors, resulting in power-law exponents---the &quot;steepness&quot;&#10;of the learning curve---yet to be explained by theoretical work. Further, model&#10;improvements only shift the error but do not appear to affect the power-law&#10;exponent. We also show that model size scales sublinearly with data size. These&#10;scaling relationships have significant implications on deep learning research,&#10;practice, and systems. They can assist model debugging, setting accuracy&#10;targets, and decisions about data set growth. They can also guide computing&#10;system design and underscore the importance of continued computational scaling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.12673" label="1909.12673">
        <attvalues>
          <attvalue for="0" value="A Constructive Prediction of the Generalization Error Across Scales" />
          <attvalue for="1" value="  The dependency of the generalization error of neural networks on model and&#10;dataset size is of critical importance both in practice and for understanding&#10;the theory of neural networks. Nevertheless, the functional form of this&#10;dependency remains elusive. In this work, we present a functional form which&#10;approximates well the generalization error in practice. Capitalizing on the&#10;successful concept of model scaling (e.g., width, depth), we are able to&#10;simultaneously construct such a form and specify the exact models which can&#10;attain it across model/data scales. Our construction follows insights obtained&#10;from observations conducted over a range of model/data scales, in various model&#10;types and datasets, in vision and language tasks. We show that the form both&#10;fits the observations well across scales, and provides accurate predictions&#10;from small- to large-scale models and data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14256" label="2305.14256">
        <attvalues>
          <attvalue for="0" value="Linear Cross-Lingual Mapping of Sentence Embeddings" />
          <attvalue for="1" value="  Semantics of a sentence is defined with much less ambiguity than semantics of&#10;a single word, and we assume that it should be better preserved by translation&#10;to another language. If multilingual sentence embeddings intend to represent&#10;sentence semantics, then the similarity between embeddings of any two sentences&#10;must be invariant with respect to translation. Based on this suggestion, we&#10;consider a simple linear cross-lingual mapping as a possible improvement of the&#10;multilingual embeddings. We also consider deviation from orthogonality&#10;conditions as a measure of deficiency of the embeddings.&#10;" />
          <attvalue for="2" value="&#10;The approximately linear mapping between cross-lingual word embeddings in different languages is based on assumption that the word semantic meaning is conserved in a translation \cite{ The linearity is only approximate because the corresponding words in different languages have different cultural background, different multiple meanings and different dependencies on context \cite{patra-etal-2019-bilingual, zhao-gilman-2020-non, Cao2020MultilingualAlignment, Peng2022Understanding}.&#10;There are multiple patterns of polysemy, and the corresponding counts of word senses are different in different languages \cite{SRINIVASAN2015124, CASAS201919}. &#10;&#10;We expect, however, that a sentence has a less ambiguous meaning than a word, simply because the sentence context reduces ambiguity of each of its words. Indeed, in \cite{kang-etal-2024-translate} it is demonstrated that additional context helps to reduce disambiguation errors.&#10;The idea that a sentence semantics should be better conserved in a translation was used in \cite{reimers-gurevych-2020-making}. &#10;&#10;In Appendix~\ref{app:ambiguity_loss} we provide simple examples illustrating the loss of word ambiguity in a sentence, and suggest that a good translation can preserve the residual ambiguity, if any. &#10;The examples show that if semantics of a sentence is somewhat changed in translation, then a better translation is possible. Unlike a lone word, which often has different sets of meaning in different languages, a sentence is not only less ambiguous but also allow differently phrased translations, among which there is usually at least one that fully preserves the semantics. &#10;&#10;In order to explore the preservation of sentence semantics in translation, we consider here a linear mapping between multilingual embeddings in two languages. Unlike the removal of a language-specific bias in each language separately \cite{yang-etal-2021-simple, xie-etal-2022-discovering}, this mapping depends on both languages of interest and, while computationally cheap, may provide a better correspondence between the embeddings. Our contribution:&#10;\begin{enumerate}[topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex]&#10; \item We suggest simple and computationally light improvement of the correspondence of sentence embeddings between two languages. The 'sentence' can be one or several contiguous sentences. &#10; \item For our evaluation we introduce a dataset based on wikipedia news.&#10; \item We demonstrate a non-orthogonality of the linear mapping between multilingual embeddings as an example and a measure of deficiency of a multilingual embedding model.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Cross-Lingual Mapping, Mathematics, Natural Language Processing, Sentence Semantics, Multilingual Embeddings" />
        </attvalues>
      </node>
      <node id="2304.13803" label="2304.13803">
        <attvalues>
          <attvalue for="0" value="Translate to Disambiguate: Zero-shot Multilingual Word Sense&#10;  Disambiguation with Pretrained Language Models" />
          <attvalue for="1" value="  Pretrained Language Models (PLMs) learn rich cross-lingual knowledge and can&#10;be finetuned to perform well on diverse tasks such as translation and&#10;multilingual word sense disambiguation (WSD). However, they often struggle at&#10;disambiguating word sense in a zero-shot setting. To better understand this&#10;contrast, we present a new study investigating how well PLMs capture&#10;cross-lingual word sense with Contextual Word-Level Translation (C-WLT), an&#10;extension of word-level translation that prompts the model to translate a given&#10;word in context. We find that as the model size increases, PLMs encode more&#10;cross-lingual word sense knowledge and better use context to improve WLT&#10;performance. Building on C-WLT, we introduce a zero-shot approach for WSD,&#10;tested on 18 languages from the XL-WSD dataset. Our method outperforms fully&#10;supervised baselines on recall for many evaluation languages without additional&#10;training or finetuning. This study presents a first step towards understanding&#10;how to best leverage the cross-lingual knowledge inside PLMs for robust&#10;zero-shot reasoning in any language.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretrained Language Models (PLMs) have been found to perform many cross-lingual tasks without explicit cross-lingual training signals, including word-level translation (WLT) across languages \cite{gonen2020greek}. These models also demonstrate cross-lingual knowledge when finetuned for the word sense disambiguation (WSD) \cite{raganato-etal-2020-xl, pasini2021xl}. However, little is known about the extent to which word sense knowledge comes from pretraining rather than finetuning: many PLMs struggle to disambiguate word sense when formulated as a binary classification task, the most common word sense setup for prompting language models \cite{shi2022language, scao2022bloom}.&#10;&#10;To investigate this, we measure the ability of multilingual autoregressive language models to understand the cross-lingual meaning of words in a given context. Specifically, we extend the WLT task setup to include a specific context in the prompt, which we call Contextual Word-Level Translation (C-WLT).&#10;We show empirically that pretrained language models are able to take advantage of contextual information in the prompt to improve WLT performance, and as the model size increases, both English and multilingual PLM demonstrate improved cross-lingual knowledge resulting in better performance in contextual WLT.&#10;&#10;Translations of a word that change based on context are frequently due to differing word senses not shared by an analogous word in the target language \cite{resnik1999distinguishing}. Inspired by this, we apply C-WLT to the task of WSD by translating the ambiguous word $w$ in context with WLT and then assigning $w$ with the senses in the overlap of the translated word's sense set with $w$'s senses (Figure \ref{fig:xl-wsd-method}, left). We test this zero-shot approach for WSD on 18 languages from the XL-WSD dataset \cite{pasini2021xl}, and find that in our best setting, WSD via C-WLT outperforms prior works on recall for many evaluation languages with no additional training or finetuning of the model. We also observe that ensembling diverse target languages with this method narrows down the predicted set of senses, as demonstrated by the improvements in Jaccard similarity with the reference set. Finally, we analyze our design choices and the types of errors made by this approach to better understand the behavior of WSD via C-WLT and how it relates to supervised WSD classification.&#10;&#10;The overall findings of this work are as follows:&#10;&#10;\begin{itemize}&#10;\itemsep0em&#10;&#10;\item PLMs leverage contextual information to encode cross-lingual knowledge and better capture lexical information, such as word translations and meanings.&#10;&#10;\item We can leverage this contextual knowledge of lexical translation to effectively perform zero-shot WSD for many languages, including low-resource ones and languages on which the PLM was not pretrained.&#10;&#10;\item The efficacy of WSD via C-WLT depends on the interplay between pretraining languages, model size, and target language choice: smaller multilingual PLMs perform better on seen languages but are more sensitive to design choices and do not generalize as well as larger English PLMs.&#10;&#10;\end{itemize}&#10;&#10;In sum, we evaluate the lexical translation skills of PLMs in context, and we present a first step towards applying that skill to the downstream task of WSD. Given that most WSD training data outside of English are automatically created \cite[e.g.,][]{scarlini2019just, barba2021mulan}, zero-shot approaches such as our proposed WSD via C-WLT approach are crucial for improving WSD in lower-resource languages.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.12604" label="2205.12604">
        <attvalues>
          <attvalue for="0" value="Leveraging QA Datasets to Improve Generative Data Augmentation" />
          <attvalue for="1" value="  The ability of generative language models (GLMs) to generate text has&#10;improved considerably in the last few years, enabling their use for generative&#10;data augmentation. In this work, we propose CONDA, an approach to further&#10;improve GLMs' ability to generate synthetic data by reformulating data&#10;generation as context generation for a given question-answer (QA) pair and&#10;leveraging QA datasets for training context generators. Then, we cast&#10;downstream tasks into the same question answering format and adapt the&#10;fine-tuned context generators to the target task domain. Finally, we use the&#10;fine-tuned GLM to generate relevant contexts, which are in turn used as&#10;synthetic training data for their corresponding tasks. We perform extensive&#10;experiments on multiple classification datasets and demonstrate substantial&#10;improvements in performance for both few- and zero-shot settings. Our analysis&#10;reveals that QA datasets that require high-level reasoning abilities (e.g.,&#10;abstractive and common-sense QA datasets) tend to give the best boost in&#10;performance in both few-shot and zero-shot settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Data Augmentation&#10;\cite{wei-zou-2019-eda} propose a simple data augmentation method using synonym replacement, random insertion, random swap, and random deletion. &#10;\cite{sennrich-etal-2016-improving} augment samples by translating them into foreign language and then back to English.&#10;\cite{du-etal-2021-self} compute task-specific query embeddings to retrieve sentences from unlabeled documents from the Internet.&#10;After a rise in pretrained generative language models, the generation capabilities of these models have been explored to generate synthetic data.&#10;\cite{AnabyTavor2020DoNH, kumar-etal-2020-data, schick-schutze-2021-generating, mekala-etal-2021-coarse2fine} generate labeled documents using the GLMs and \cite{yang-etal-2020-generative} do so specifically for common-sense reasoning.&#10;\cite{puri-etal-2020-training} use GLMs to synthesize questions and answers and improve performance on question answering.&#10;\cite{vu-etal-2021-strata} generate data for NLI tasks.&#10;&#10;Few-shot Learning&#10;Our work is closely related to few-shot learning as we take a few annotated samples as supervision.&#10;The idea of formulating classification as a prompting task is getting increasingly popular.&#10;~\cite{NEURIPS2020_1457c0d6} introduce a new paradigm called in-context learning to infer from large language models using few annotated samples.&#10;\cite{schick-schutze-2021-exploiting} formulate input samples as cloze-style phrases and assign pseudo-labels that are used for training the classifier and \cite{tam-etal-2021-improving} improves their approach further without using any task-specific unlabeled data.&#10;~\cite{McCann2018decaNLP, 2020t5} format several NLP tasks into a question-answer and text-to-text schema.&#10;\cite{lin2021few} train multilingual autoregressive language models to enable few-shot learning in multiple languages.&#10;~\cite{gao-etal-2021-making} propose to generate prompts and convert smaller pretrained language models to few-shot learners. Other work proposes to pre-train prompts by adding soft prompts into the pre-training stage~\cite{gu-etal-2022-ppt,vu-etal-2022-spot,vu-etal-2022-overcoming}.&#10;&#10;Language Model Fine-Tuning&#10;Pre-trained language models are applied to downstream tasks by fine-tuning them using task-specific objectives~\cite{howard-ruder-2018-universal}.&#10;However, this process requires significant annotated downstream task data~\cite{Yogatama2019LearningAE}.&#10;Many methods have been proposed to address this challenge.&#10;~\cite{gururangan-etal-2020-dont} propose to continue training on unlabeled data from the target task domain. &#10;~\cite{aghajanyan-etal-2021-muppet} propose pre-finetuning, a large-scale multi-task learning stage between language model pre-training and fine-tuning.&#10;~\cite{Phang2018SentenceEO} introduce intermediate task fine-tuning which involves fine-tuning a language model on an auxiliary task before continuously training on the target task.&#10;~\cite{pruksachatkun-etal-2020-intermediate} observe that the tasks requiring high-level inference and reasoning abilities are the best choice as intermediate tasks.&#10;~\cite{vu-etal-2020-exploring} identify the best auxiliary tasks for high performance on downstream tasks.&#10;~\cite{vu-etal-2021-strata} use NLI as auxiliary task to generate synthetic NLI data for intermediate fine-tuning.&#10;Our method differs from~\cite{Phang2018SentenceEO} in two fronts: (1) we use QA datasets for training context generators instead of answering the question&#10;, and (2) we use the fine-tuned GLM to generate synthetic data instead of training directly for the downstream tasks.&#10;It also differs from ~\cite{vu-etal-2021-strata} in terms of the generated data, where they consider NLI as an auxiliary task and generate synthetic samples in target-domain for the NLI task irrespective of the target task and perform intermediate task fine-tuning.&#10;ConDA formats target tasks into question-answer format and directly generates samples relevant for target task.&#10; " />
          <attvalue for="4" value="Machine Learning, Computer Science, Data Augmentation Techniques, Linguistics, Generative Language Models, Artificial Intelligence, Question Answering Systems, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2403.19930" label="2403.19930">
        <attvalues>
          <attvalue for="0" value="Are LLMs Effective Backbones for Fine-tuning? An Experimental&#10;  Investigation of Supervised LLMs on Chinese Short Text Matching" />
          <attvalue for="1" value="  The recent success of Large Language Models (LLMs) has garnered significant&#10;attention in both academia and industry. Prior research on LLMs has primarily&#10;focused on enhancing or leveraging their generalization capabilities in zero-&#10;and few-shot settings. However, there has been limited investigation into&#10;effectively fine-tuning LLMs for a specific natural language understanding task&#10;in supervised settings. In this study, we conduct an experimental analysis by&#10;fine-tuning LLMs for the task of Chinese short text matching. We explore&#10;various factors that influence performance when fine-tuning LLMs, including&#10;task modeling methods, prompt formats, and output formats.&#10;" />
          <attvalue for="2" value="&#10;The recent success of Large Language Models (LLMs), such as GPT-3\cite{brown2020language}, LLaMA\cite{touvron2023llama} and PaLM\cite{chowdhery2023palm}, has garnered significant attention in both academia and industry. LLMs have demonstrated remarkable generalization capabilities in zero- and few-shot settings, particularly in natural language generation (NLG) tasks. Substantial efforts have been made to enhance and utilizing such generalization capabilities\cite{xu-etal-2023-fine,saad-falcon-etal-2023-udapdr,yun-etal-2023-appraising}.&#10;&#10;However, for natural language understanding (NLU) tasks, zero- and few-shot LLMs struggle to achieve satisfactory performance\cite{nie2022improving,wei2023zero,li2023evaluating,li2023label} compared to fine-tuned small models (e.g., Bert base\cite{devlin2018bert}). Our experimental results on the task of Chinese short text matching also confirm this phenomenon. As presented in Section\ref{sec:model}, fine-tuned Bert achieves an accuracy of 84.5\% on the BQ\cite{chen2018bq} corpus, while GPT-4, one of the most successful LLMs, only attains an accuracy score of 52.9\% in zero-shot and 77.9\% in few-shot settings. There has been limited investigation into effectively tuning LLMs for a specific NLU task in supervised settings. In this paper, we explore various factors affecting the performance of LLMs for Chinese short text matching task, including task modeling methods, prompt formats, and output formats.&#10;&#10;\begin{itemize}&#10;\item Task modeling methods: In this study, we examine the impacts of modeling this task as both a generative task and a discriminative classification task, respectively. (1) Generative Task: LLMs uniformly model all tasks as generative tasks. Following this principle, we organize the given pair of sentences into a single text as input and make the model generate the target label (equivalent or inequivalent). (2) Discriminative Classification Task: Motivated by the efficacy of fine-tuning Bert for text matching\cite{chen2020neural,qi2022all}, we concatenate the given pair of texts as input, extract vector representations from the final LLM layer as features, and perform binary classifications based on the extracted features.&#10;\item Prompt Formats: Prompt design is crucial for LLMs in zero- and few-shot settings\cite{gu2021ppt,liu2023pre}. However, the importance of prompts in supervised settings has not been explored. In this paper, we compare two completely different styles of prompts. One is concise, directly concatenating the given pair of sentences without any explanation of the target task. The other organizes the prompt through complex instructions, including not only the given sentences but also a detail description of the target task.&#10;\item Output Formats: Incorporating the Chain of Thought (CoT) into prompts has been shown to significantly enhance performance in reasoning and complex tasks in zero- and few-shot settings\cite{wei2022chain,wang2022self}. Nevertheless, the impact of CoT on matching tasks in supervised settings has yet to be examined. In this study, we address this gap by incorporating CoT into the output part of training samples.&#10;\end{itemize}&#10;We conduct experiments on two widely-used Chinese short text matching datasets, LCQMC \cite{liu2018lcqmc} and BQ \cite{chen2018bq}. All experiments are carried out based on CLLM-7B, which is a Chinese-enhanced model based on LLaMA-2-7B. Our preliminary results demonstrate that the fine-tuned CLLM-7B outperforms both fine-tuned BERT and few-shot GPT-4. Furthermore, the results indicate that the generative paradigm surpasses the discriminative approach, especially when training data is limited. Lastly, our experiments reveal that CoT is also beneficial for the matching task in supervised settings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Natural Language Understanding, Fine-Tuning Techniques, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2303.00807" label="2303.00807">
        <attvalues>
          <attvalue for="0" value="UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and&#10;  Distillation of Rerankers" />
          <attvalue for="1" value="  Many information retrieval tasks require large labeled datasets for&#10;fine-tuning. However, such datasets are often unavailable, and their utility&#10;for real-world applications can diminish quickly due to domain shifts. To&#10;address this challenge, we develop and motivate a method for using large&#10;language models (LLMs) to generate large numbers of synthetic queries cheaply.&#10;The method begins by generating a small number of synthetic queries using an&#10;expensive LLM. After that, a much less expensive one is used to create large&#10;numbers of synthetic queries, which are used to fine-tune a family of reranker&#10;models. These rerankers are then distilled into a single efficient retriever&#10;for use in the target domain. We show that this technique boosts zero-shot&#10;accuracy in long-tail domains and achieves substantially lower latency than&#10;standard reranking methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11828" label="2305.11828">
        <attvalues>
          <attvalue for="0" value="Appraising the Potential Uses and Harms of LLMs for Medical Systematic&#10;  Reviews" />
          <attvalue for="1" value="  Medical systematic reviews play a vital role in healthcare decision making&#10;and policy. However, their production is time-consuming, limiting the&#10;availability of high-quality and up-to-date evidence summaries. Recent&#10;advancements in large language models (LLMs) offer the potential to&#10;automatically generate literature reviews on demand, addressing this issue.&#10;However, LLMs sometimes generate inaccurate (and potentially misleading) texts&#10;by hallucination or omission. In healthcare, this can make LLMs unusable at&#10;best and dangerous at worst. We conducted 16 interviews with international&#10;systematic review experts to characterize the perceived utility and risks of&#10;LLMs in the specific context of medical evidence reviews. Experts indicated&#10;that LLMs can assist in the writing process by drafting summaries, generating&#10;templates, distilling information, and crosschecking information. They also&#10;raised concerns regarding confidently composed but inaccurate LLM outputs and&#10;other potential downstream harms, including decreased accountability and&#10;proliferation of low-quality reviews. Informed by this qualitative analysis, we&#10;identify criteria for rigorous evaluation of biomedical LLMs aligned with&#10;domain expert views.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.02216" label="2212.02216">
        <attvalues>
          <attvalue for="0" value="Improving Few-Shot Performance of Language Models via Nearest Neighbor&#10;  Calibration" />
          <attvalue for="1" value="  Pre-trained language models (PLMs) have exhibited remarkable few-shot&#10;learning capabilities when provided a few examples in a natural language prompt&#10;as demonstrations of test instances, i.e., in-context learning. However, the&#10;performance of in-context learning is susceptible to the choice of prompt&#10;format, training examples and the ordering of the training examples. In this&#10;paper, we propose a novel nearest-neighbor calibration framework for in-context&#10;learning to ease this issue. It is inspired by a phenomenon that the in-context&#10;learning paradigm produces incorrect labels when inferring training instances,&#10;which provides a useful supervised signal to calibrate predictions. Thus, our&#10;method directly augments the predictions with a $k$-nearest-neighbor ($k$NN)&#10;classifier over a datastore of cached few-shot instance representations&#10;obtained by PLMs and their corresponding labels. Then adaptive neighbor&#10;selection and feature regularization modules are introduced to make full use of&#10;a few support instances to reduce the $k$NN retrieval noise. Experiments on&#10;various few-shot text classification tasks demonstrate that our method&#10;significantly improves in-context learning, while even achieving comparable&#10;performance with state-of-the-art tuning-based approaches in some sentiment&#10;analysis tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Task Formulation.&#10;We consider the few-shot adaption of a pre-trained language model $\mathcal{L}$ on the task $\mathcal{D}$ with a label space $\mathcal{Y}$. &#10;For the task, we assume that the training data $\mathcal{D}_{\mathrm{train}} = \{(x^i, y^i)\}_{i=1}^{K \times |\mathcal{Y}|}$ only consists of $K$ examples per class, where $x$ represents the input, $y$ is the target label and $|\mathcal{Y}|$ denotes the number of unique classes.&#10;The goal of few-shot adaption is to develop task-agnostic learning strategies on $\mathcal{D}_{\mathrm{train}}$, and generalize well to an unseen test set $\mathcal{D}_{\mathrm{test}}$. &#10;We additionally assume access to development set $\mathcal{D}_{\mathrm{dev}}$ with the same size as the training data for model selection and hyper-parameter tuning, as larger validation sets can grant a substantial advantage~\cite{perez2021true}.&#10;For our experiments, we use 16 training examples ($K$ = 16) and a development set with 16 examples per class for all tasks.&#10;&#10;Prompt-Based Fine-tuning.&#10;The standard fine-tuning has a clear discrepancy between pre-training and fine-tuning phases, where the former is optimized by the prediction of masked tokens in the masked language modelling (MLM) task. &#10;An alternative way to eliminate this gap is prompt-based fine-tuning, in which the task is formulated in a cloze-format~\cite{taylor}. &#10;In this way, the language model $\mathcal{L}$ predicts label words with the MLM objective. &#10;Specifically, inputs are converted using a pre-defined prompt template $x_{\mathrm{prompt}} = \mathcal{T}(x)$, e.g., in the sentiment classification task, $x_{\mathrm{prompt}}$ is constructed as follow: &#10;\begin{align}&#10; x_{\mathrm{prompt}}=[CLS]\ x It was [MASK].\ [SEP].&#10;\end{align}&#10;Then the language model $\mathcal{L}$ decides which verbalizer (e.g., `great' and `terrible') is most likely for [MASK] in $x_{\mathrm{prompt}}$. &#10;For the model training, let $\mathcal{M}: \mathcal{Y} \to \mathcal{V}$ be a mapping from the task label space to words in the vocabulary $\mathcal{V}$ of PLMs.&#10;The probability of class $y \in \mathcal{Y}$ is calculated as:&#10;\begin{align}&#10;p(y|x) &amp; = p_{\mathcal{L}}([MASK]=\mathcal{M}(y)|\mathcal{T}(x)) \\&#10;&amp; = \frac{\mathrm{exp}(W_{\mathcal{M}(y)} h_{[MASK]})}{\sum_{\hat{y} \in \mathcal{Y}} \mathrm{exp}(W_{\mathcal{M}(\hat{y}) }h_{[MASK]})},&#10;\end{align}&#10;where $h_{[MASK]}$ denotes the hidden vector of [MASK] token and $W_{\mathcal{M}(y)}$ refers to pre-softmax vector for word $v \in \mathcal{V}$. &#10;The entire model is trained by minimizing the cross-entropy loss with $\mathcal{D}_{\mathrm{train}}$ and select the best checkpoint on $\mathcal{D}_{\mathrm{dev}}$.&#10;&#10;In-Context Learning.&#10;Instead of directly fine-tuning model, \cite{gpt3} show that PLMs themselves have the capability to perform few-shot learning without parameter updates. &#10;It explores an in-context learning paradigm, which simply concatenates randomly sampled training examples as demonstrations with inputs during inference:&#10;\begin{equation}&#10;\begin{aligned}&#10;x_{\mathrm{demo}} &amp; = \mathcal{T}(x_{train}^{1}) \oplus \ldots \oplus \mathcal{T}(x_{train}^{|\mathcal{Y}|}) \oplus \mathcal{T}(x), \\&#10;p(y|x) &amp; = p_{\mathcal{L}}([MASK] = \mathcal{M}(y)|x_{\mathrm{demo}}), &#10;\end{aligned}&#10;\end{equation}&#10;where $\oplus$ refers to concatenation of input texts, and we select one example per class as demonstrations ($x_{\mathrm{train}}^{1},..., x_{\mathrm{train}}^{|\mathcal{Y}|}$).&#10;The final prediction of the in-context learning ensembles all results based on different sampled demonstrations. &#10;This method has practical advantages for the few-shot adaptation over the now-standard approach of finetuning, as we could hold only one model for serving many different tasks, avoiding expensive and time-consuming parameter updates. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2304.11633" label="2304.11633">
        <attvalues>
          <attvalue for="0" value="Evaluating ChatGPT's Information Extraction Capabilities: An Assessment&#10;  of Performance, Explainability, Calibration, and Faithfulness" />
          <attvalue for="1" value="  The capability of Large Language Models (LLMs) like ChatGPT to comprehend&#10;user intent and provide reasonable responses has made them extremely popular&#10;lately. In this paper, we focus on assessing the overall ability of ChatGPT&#10;using 7 fine-grained information extraction (IE) tasks. Specially, we present&#10;the systematically analysis by measuring ChatGPT's performance, explainability,&#10;calibration, and faithfulness, and resulting in 15 keys from either the ChatGPT&#10;or domain experts. Our findings reveal that ChatGPT's performance in&#10;Standard-IE setting is poor, but it surprisingly exhibits excellent performance&#10;in the OpenIE setting, as evidenced by human evaluation. In addition, our&#10;research indicates that ChatGPT provides high-quality and trustworthy&#10;explanations for its decisions. However, there is an issue of ChatGPT being&#10;overconfident in its predictions, which resulting in low calibration.&#10;Furthermore, ChatGPT demonstrates a high level of faithfulness to the original&#10;text in the majority of cases. We manually annotate and release the test sets&#10;of 7 fine-grained IE tasks contains 14 datasets to further promote the&#10;research. The datasets and code are available at&#10;https://github.com/pkuserc/ChatGPT_for_IE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2310.01208" label="2310.01208">
        <attvalues>
          <attvalue for="0" value="Label Supervised LLaMA Finetuning" />
          <attvalue for="1" value="  The recent success of Large Language Models (LLMs) has gained significant&#10;attention in both academia and industry. Substantial efforts have been made to&#10;enhance the zero- and few-shot generalization capabilities of open-source LLMs&#10;through finetuning. Currently, the prevailing approach is instruction-tuning,&#10;which trains LLMs to complete real-world tasks by generating responses guided&#10;by natural language instructions. It is worth noticing that such an approach&#10;may underperform in sequence and token classification tasks. Unlike text&#10;generation tasks, classification tasks have a limited label space, where&#10;precise label prediction is more appreciated than generating diverse and&#10;human-like responses. Prior research has unveiled that instruction-tuned LLMs&#10;cannot outperform BERT, prompting us to explore the potential of leveraging&#10;latent representations from LLMs for supervised label prediction. In this&#10;paper, we introduce a label-supervised adaptation for LLMs, which aims to&#10;finetuning the model with discriminant labels. We evaluate this approach with&#10;Label Supervised LLaMA (LS-LLaMA), based on LLaMA-2-7B, a relatively&#10;small-scale LLM, and can be finetuned on a single GeForce RTX4090 GPU. We&#10;extract latent representations from the final LLaMA layer and project them into&#10;the label space to compute the cross-entropy loss. The model is finetuned by&#10;Low-Rank Adaptation (LoRA) to minimize this loss. Remarkably, without intricate&#10;prompt engineering or external knowledge, LS-LLaMA substantially outperforms&#10;LLMs ten times its size in scale and demonstrates consistent improvements&#10;compared to robust baselines like BERT-Large and RoBERTa-Large in text&#10;classification. Moreover, by removing the causal mask from decoders, LS-unLLaMA&#10;achieves the state-of-the-art performance in named entity recognition (NER).&#10;Our work will shed light on a novel approach to adapting LLMs for various&#10;downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Continuous efforts have been devoted to improving the problem-solving abilities of LLMs with their superior text generation capacity. One of the main research directions focuses on prompt engineering, which aims to generate higher-quality responses by harnessing LLMs' existing knowledge. Without modifying the parameters, one can give instructions with a few input-output exemplars or carefully crafted in-context prompts to help the model better understand the task and elicit a profound inferencing and reasoning process. Various prompting techniques have been proposed, such as contextual calibration \cite{zhao2021calibrate}, prompt programming \cite{reynolds2021prompt}, chat-based prompt \cite{wei2023zero}, chain-of-thought \cite{wei2022chain}, and tree-of-thought \cite{yao2023tree}. These methods are especially preferred for extra-large LLMs like ChatGPT and PaLM-540B, on which tuning the parameters may be less feasible, they have been effective in a range of tasks including information extraction \cite{li2023evaluating}, semantic textual similarity \cite{li2023angle}, and reasoning \cite{kojima2022large}. &#10;&#10;Despite the success of zero- and few-shot settings, LLMs frequently struggle in domains that require specific knowledge or precise response generation. To generalize LLMs on more downstream tasks, researchers have also investigated various instruction-tuning methods \cite{brown2020language,wei2021finetuned,wang2022self,peng2023instruction,phang2023hypertuning,zadouri2023pushing} to replenish domain knowledge and enhance LLMs performance. Instruction-tuning approaches tune the pretrained parameters with instructional data, which contain instructional commands and human-annotated expected outcomes \cite{sanh2021multitask,wang2022super}. \cite{peng2023instruction} finetuned LLaMA using 52K English and Chinese instruction-following instances generated using GPT-4. \cite{phang2023hypertuning} proposed HyperTuning that uses a hypermodel to generate task-specific parameters for a fixed downstream model for model adaptation. Instruction-following LLMs present substantial improvements in zero-shot performance on unseen tasks.&#10;&#10;So far, no attempts have been made to finetune an LLM with discriminant labels. Our study verifies the feasibility of the proposed label-supervised adaptation approach on sequence and token classification tasks.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2002.12764" label="2002.12764">
        <attvalues>
          <attvalue for="0" value="Towards Learning a Universal Non-Semantic Representation of Speech" />
          <attvalue for="1" value="  The ultimate goal of transfer learning is to reduce labeled data requirements&#10;by exploiting a pre-existing embedding model trained for different datasets or&#10;tasks. The visual and language communities have established benchmarks to&#10;compare embeddings, but the speech community has yet to do so. This paper&#10;proposes a benchmark for comparing speech representations on non-semantic&#10;tasks, and proposes a representation based on an unsupervised triplet-loss&#10;objective. The proposed representation outperforms other representations on the&#10;benchmark, and even exceeds state-of-the-art performance on a number of&#10;transfer learning tasks. The embedding is trained on a publicly available&#10;dataset, and it is tested on a variety of low-resource downstream tasks,&#10;including personalization tasks and medical domain. The benchmark, models, and&#10;evaluation code are publicly released.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Transfer Learning, Computer Science, Linguistics, Benchmark Evaluation, Artificial Intelligence, Speech Representations" />
        </attvalues>
      </node>
      <node id="2102.12266" label="2102.12266">
        <attvalues>
          <attvalue for="0" value="Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding&#10;  Learning" />
          <attvalue for="1" value="  Word embedding learning methods require a large number of occurrences of a&#10;word to accurately learn its embedding. However, out-of-vocabulary (OOV) words&#10;which do not appear in the training corpus emerge frequently in the smaller&#10;downstream data. Recent work formulated OOV embedding learning as a few-shot&#10;regression problem and demonstrated that meta-learning can improve results&#10;obtained. However, the algorithm used, model-agnostic meta-learning (MAML) is&#10;known to be unstable and perform worse when a large number of gradient steps&#10;are used for parameter updates. In this work, we propose the use of Leap, a&#10;meta-learning algorithm which leverages the entire trajectory of the learning&#10;process instead of just the beginning and the end points, and thus ameliorates&#10;these two issues. In our experiments on a benchmark OOV embedding learning&#10;dataset and in an extrinsic evaluation, Leap performs comparably or better than&#10;MAML. We go on to examine which contexts are most beneficial to learn an OOV&#10;embedding from, and propose that the choice of contexts may matter more than&#10;the meta-learning employed.&#10;" />
          <attvalue for="2" value="&#10;&#10;Distributional methods for learning word embeddings require a sufficient number of occurrences of a word in the training corpus to accurately learn its embedding.&#10;Even though the embeddings can be trained on raw text implying that an embedding for every word is obtained, in practice &#10;out-of-vocabulary (OOV)&#10;words do occur in the downstream applications embeddings are used,&#10;for example &#10;due to&#10;domain-specific terminology. Nevertheless, OOV words are often content words such as names which convey important information for downstream tasks; for example, drug names are key&#10;in the biomedical domain. However, the amount of downstream language data is typically much smaller than the corpus&#10;used for training word embeddings, thus methods that rely on distributional properties of words across large amounts of data perform poorly \cite{herbelot2017highrisk}.&#10;&#10;Researchers often assign OOV words to random embeddings or to an ``unknown'' embedding, however these solutions fail to capture the distributional properties of words. Zero-shot approaches \cite{pinter-etal-2017-mimicking, kim2015characteraware, bojanowski2016enriching} attempt to predict the embeddings for OOV words from their characters alone. These approaches rely on inferring the meaning of a word from its subword information, such as morphemes or WordPiece tokens used in BERT \cite{devlin-etal-2019-bert}. While this works well for many words, it performs poorly for names and words where morphology is not informative.&#10;&#10;Given that an OOV word occurs once, the chance of a second occurrence is much higher than the first \cite{noriegas}. Hence while OOV words can be rare and not seen in training, it is reasonable to expect that a limited number of occurrences will be present in the data of a downstream application. &#10;Few-shot approaches \cite{garneau-etal-2018-predicting, khodak2018la,hu2019fewshot} leveraged this to predict the embeddings for OOV words from just a few contexts, often in conjunction with their morphological information.&#10;\cite{hu2019fewshot} proposed an attention-based architecture for OOV word embedding learning as a few-shot regression problem. The model is trained to predict the embedding of a word based on a few contexts and its character sequence. Such a model is trained by simulating OOV words in the training corpus, with their target embeddings provided by learning them&#10;on the same corpus. As OOV words must have their embeddings inferred from contexts outside the training corpus, the authors show that using an adaptation of the model-agnostic meta-learning (MAML) algorithm \cite{finn2017modelagnostic} to adapt the model's parameters to the target domain improves the quality of the learned OOV word embeddings.&#10;&#10;However, MAML is known to be&#10;unstable due to the calculation of gradients requiring backpropagation through multiple instances of the model, as the learning process must be unrolled to calculate gradients with respect to the initial parameters \cite{antoniou2018train}. &#10;In practice, the learning process is often truncated to a small number of gradient steps, but has been shown to have a short-horizon bias \cite{wu2018understanding}, causing it to underperform. &#10;&#10;In this work we explore OOV word embedding learning using Leap \cite{flennerhag2018transferring}, a meta-learning framework which takes into consideration the entire learning trajectory, not only the beginning and end points.&#10;Each task is associated with a loss surface over the model's parameters on which the learning process travels, and the aim is to minimize the expected length of this process across tasks. Leap also does not require backpropagation through the learning process, allowing it to adapt over a larger number of gradient steps and thus not suffering from the short-horizon bias that MAML is prone to.&#10;&#10;We conduct an intrinsic evaluation of MAML and Leap on the&#10;dataset of \cite{chimeras} which simulates OOV words by combining the contexts of two semantically similar words to form a 'chimera'. We find that Leap performs better than MAML at adapting model parameters to a new corpus. We also conduct an extrinsic evaluation on NER in the biomedical domain where the results are comparable to MAML, without improving in most cases on a random embedding baseline. Finally, we examine which contexts are more beneficial to learn an embedding from, and note that the contexts from which an embedding is learned matters more than the meta-learning method employed.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Meta Learning Algorithms, Machine Learning, Computer Science, Linguistics, Out-of-Vocabulary Embeddings, Artificial Intelligence, Mathematics, Natural Language Processing, Word Embedding Learning" />
        </attvalues>
      </node>
      <node id="1707.06961" label="1707.06961">
        <attvalues>
          <attvalue for="0" value="Mimicking Word Embeddings using Subword RNNs" />
          <attvalue for="1" value="  Word embeddings improve generalization over lexical features by placing each&#10;word in a lower-dimensional space, using distributional information obtained&#10;from unlabeled data. However, the effectiveness of word embeddings for&#10;downstream NLP tasks is limited by out-of-vocabulary (OOV) words, for which&#10;embeddings do not exist. In this paper, we present MIMICK, an approach to&#10;generating OOV word embeddings compositionally, by learning a function from&#10;spellings to distributional embeddings. Unlike prior work, MIMICK does not&#10;require re-training on the original word embedding corpus; instead, learning is&#10;performed at the type level. Intrinsic and extrinsic evaluations demonstrate&#10;the power of this simple approach. On 23 languages, MIMICK improves performance&#10;over a word-based baseline for tagging part-of-speech and morphosyntactic&#10;attributes. It is competitive with (and complementary to) a supervised&#10;character-based model in low-resource settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.00724" label="1903.00724">
        <attvalues>
          <attvalue for="0" value="Predicting and interpreting embeddings for out of vocabulary words in&#10;  downstream tasks" />
          <attvalue for="1" value="  We propose a novel way to handle out of vocabulary (OOV) words in downstream&#10;natural language processing (NLP) tasks. We implement a network that predicts&#10;useful embeddings for OOV words based on their morphology and on the context in&#10;which they appear. Our model also incorporates an attention mechanism&#10;indicating the focus allocated to the left context words, the right context&#10;words or the word's characters, hence making the prediction more interpretable.&#10;The model is a ``drop-in'' module that is jointly trained with the downstream&#10;task's neural network, thus producing embeddings specialized for the task at&#10;hand. When the task is mostly syntactical, we observe that our model aims most&#10;of its attention on surface form characters. On the other hand, for tasks more&#10;semantical, the network allocates more attention to the surrounding words. In&#10;all our tests, the module helps the network to achieve better performances in&#10;comparison to the use of simple random embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.00505" label="1907.00505">
        <attvalues>
          <attvalue for="0" value="Few-Shot Representation Learning for Out-Of-Vocabulary Words" />
          <attvalue for="1" value="  Existing approaches for learning word embeddings often assume there are&#10;sufficient occurrences for each word in the corpus, such that the&#10;representation of words can be accurately estimated from their contexts.&#10;However, in real-world scenarios, out-of-vocabulary (a.k.a. OOV) words that do&#10;not appear in training corpus emerge frequently. It is challenging to learn&#10;accurate representations of these words with only a few observations. In this&#10;paper, we formulate the learning of OOV embeddings as a few-shot regression&#10;problem, and address it by training a representation function to predict the&#10;oracle embedding vector (defined as embedding trained with abundant&#10;observations) based on limited observations. Specifically, we propose a novel&#10;hierarchical attention-based architecture to serve as the neural regression&#10;function, with which the context information of a word is encoded and&#10;aggregated from K observations. Furthermore, our approach can leverage&#10;Model-Agnostic Meta-Learning (MAML) for adapting the learned model to the new&#10;corpus fast and robustly. Experiments show that the proposed approach&#10;significantly outperforms existing methods in constructing accurate embeddings&#10;for OOV words, and improves downstream tasks where these embeddings are&#10;utilized.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.03400" label="1703.03400">
        <attvalues>
          <attvalue for="0" value="Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" />
          <attvalue for="1" value="  We propose an algorithm for meta-learning that is model-agnostic, in the&#10;sense that it is compatible with any model trained with gradient descent and&#10;applicable to a variety of different learning problems, including&#10;classification, regression, and reinforcement learning. The goal of&#10;meta-learning is to train a model on a variety of learning tasks, such that it&#10;can solve new learning tasks using only a small number of training samples. In&#10;our approach, the parameters of the model are explicitly trained such that a&#10;small number of gradient steps with a small amount of training data from a new&#10;task will produce good generalization performance on that task. In effect, our&#10;method trains the model to be easy to fine-tune. We demonstrate that this&#10;approach leads to state-of-the-art performance on two few-shot image&#10;classification benchmarks, produces good results on few-shot regression, and&#10;accelerates fine-tuning for policy gradient reinforcement learning with neural&#10;network policies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.09502" label="1810.09502">
        <attvalues>
          <attvalue for="0" value="How to train your MAML" />
          <attvalue for="1" value="  The field of few-shot learning has recently seen substantial advancements.&#10;Most of these advancements came from casting few-shot learning as a&#10;meta-learning problem. Model Agnostic Meta Learning or MAML is currently one of&#10;the best approaches for few-shot learning via meta-learning. MAML is simple,&#10;elegant and very powerful, however, it has a variety of issues, such as being&#10;very sensitive to neural network architectures, often leading to instability&#10;during training, requiring arduous hyperparameter searches to stabilize&#10;training and achieve high generalization and being very computationally&#10;expensive at both training and inference times. In this paper, we propose&#10;various modifications to MAML that not only stabilize the system, but also&#10;substantially improve the generalization performance, convergence speed and&#10;computational overhead of MAML, which we call MAML++.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.02021" label="1803.02021">
        <attvalues>
          <attvalue for="0" value="Understanding Short-Horizon Bias in Stochastic Meta-Optimization" />
          <attvalue for="1" value="  Careful tuning of the learning rate, or even schedules thereof, can be&#10;crucial to effective neural net training. There has been much recent interest&#10;in gradient-based meta-optimization, where one tunes hyperparameters, or even&#10;learns an optimizer, in order to minimize the expected loss when the training&#10;procedure is unrolled. But because the training procedure must be unrolled&#10;thousands of times, the meta-objective must be defined with an&#10;orders-of-magnitude shorter time horizon than is typical for neural net&#10;training. We show that such short-horizon meta-objectives cause a serious bias&#10;towards small step sizes, an effect we term short-horizon bias. We introduce a&#10;toy problem, a noisy quadratic cost function, on which we analyze short-horizon&#10;bias by deriving and comparing the optimal schedules for short and long time&#10;horizons. We then run meta-optimization experiments (both offline and online)&#10;on standard benchmark datasets, showing that meta-optimization chooses too&#10;small a learning rate by multiple orders of magnitude, even when run with a&#10;moderately long time horizon (100 steps) typical of work in the area. We&#10;believe short-horizon bias is a fundamental problem that needs to be addressed&#10;if meta-optimization is to scale to practical neural net training regimes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.01054" label="1812.01054">
        <attvalues>
          <attvalue for="0" value="Transferring Knowledge across Learning Processes" />
          <attvalue for="1" value="  In complex transfer learning scenarios new tasks might not be tightly linked&#10;to previous tasks. Approaches that transfer information contained only in the&#10;final parameters of a source model will therefore struggle. Instead, transfer&#10;learning at a higher level of abstraction is needed. We propose Leap, a&#10;framework that achieves this by transferring knowledge across learning&#10;processes. We associate each task with a manifold on which the training process&#10;travels from initialization to final parameters and construct a meta-learning&#10;objective that minimizes the expected length of this path. Our framework&#10;leverages only information obtained during training and can be computed on the&#10;fly at negligible cost. We demonstrate that our framework outperforms competing&#10;methods, both in meta-learning and transfer learning, on a set of computer&#10;vision tasks. Finally, we demonstrate that Leap can transfer knowledge across&#10;learning processes in demanding reinforcement learning environments (Atari)&#10;that involve millions of gradient steps.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.16007" label="2306.16007">
        <attvalues>
          <attvalue for="0" value="Prompting Large Language Models for Zero-Shot Domain Adaptation in&#10;  Speech Recognition" />
          <attvalue for="1" value="  The integration of Language Models (LMs) has proven to be an effective way to&#10;address domain shifts in speech recognition. However, these approaches usually&#10;require a significant amount of target domain text data for the training of&#10;LMs. Different from these methods, in this work, with only a domain-specific&#10;text prompt, we propose two zero-shot ASR domain adaptation methods using&#10;LLaMA, a 7-billion-parameter large language model (LLM). LLM is used in two&#10;ways: 1) second-pass rescoring: reranking N-best hypotheses of a given ASR&#10;system with LLaMA; 2) deep LLM-fusion: incorporating LLM into the decoder of an&#10;encoder-decoder based ASR system. Experiments show that, with only one domain&#10;prompt, both methods can effectively reduce word error rates (WER) on&#10;out-of-domain TedLium-2 and SPGISpeech datasets. Especially, the deep&#10;LLM-fusion has the advantage of better recall of entity and out-of-vocabulary&#10;words.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Engineering, Domain Adaptation, Artificial Intelligence, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2405.06105" label="2405.06105">
        <attvalues>
          <attvalue for="0" value="Can Perplexity Reflect Large Language Model's Ability in Long Text&#10;  Understanding?" />
          <attvalue for="1" value="  Recent studies have shown that Large Language Models (LLMs) have the&#10;potential to process extremely long text. Many works only evaluate LLMs'&#10;long-text processing ability on the language modeling task, with perplexity&#10;(PPL) as the evaluation metric. However, in our study, we find that there is no&#10;correlation between PPL and LLMs' long-text understanding ability. Besides, PPL&#10;may only reflect the model's ability to model local information instead of&#10;catching long-range dependency. Therefore, only using PPL to prove the model&#10;could process long text is inappropriate. The local focus feature of PPL could&#10;also explain some existing phenomena, such as the great extrapolation ability&#10;of the position method ALiBi. When evaluating a model's ability in long text,&#10;we might pay more attention to PPL's limitation and avoid overly relying on it.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, many researchers~\cite{chen2023extending,chen2023clex,xiong2023effective, ding2023longnet, chen2023longlora} have proposed various approaches to scale up the context window of LLMs to more than 100k. Since there is not a comprehensive benchmark tailored for the evaluation of such extremely long text understanding, such as question answering (QA) over 100K tokens, researchers use perplexity (PPL), an evaluation metric for language modeling, to demonstrate the model's ability to process long text~\cite{chen2023longlora, ding2023longnet,liu2023scaling,peng2023yarn}. &#10;&#10;However, only given LLMs are excellent in language modeling, can it indicate LLMs' ability to understand long text? We conduct experiments on three long context window LLM variants to figure out this. We use several available benchmarks of downstream tasks, such as QA and summerization, to evaluate their long-text understanding ability. Surprisingly, the models' performance on language modeling is inconsistent with their performance on most downstream tasks, implying the PPL can not be a good indicator of the model's long-text understanding ability.&#10;&#10;We speculate that the phenomenon above may be because PPL is a reflection of the model's ability to model local information. We use LLaMA2, which only has a short context window of 4,096 and cannot handle long context, to prove our speculation. The experiment results show that, LLaMA2 delivers comparable PPL with the long context window LLMs. The feature of PPL in reflecting local information modeling ability can also explain why methods such as ALiBi \cite{press2022train}, which makes the model mainly focus on local information, could enable models to extrapolate to longer inference sequences while keeping the PPL at a low level.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Mathematics, Model Understanding Limitations, Long Text Processing" />
        </attvalues>
      </node>
      <node id="2010.10938" label="2010.10938">
        <attvalues>
          <attvalue for="0" value="What makes multilingual BERT multilingual?" />
          <attvalue for="1" value="  Recently, multilingual BERT works remarkably well on cross-lingual transfer&#10;tasks, superior to static non-contextualized word embeddings. In this work, we&#10;provide an in-depth experimental study to supplement the existing literature of&#10;cross-lingual ability. We compare the cross-lingual ability of&#10;non-contextualized and contextualized representation model with the same data.&#10;We found that datasize and context window size are crucial factors to the&#10;transferability.&#10;" />
          <attvalue for="2" value="&#10;&#10;Cross-lingual word embedding is to learn embeddings in a shared vector space for two or more languages. A line of works assumes that monolingual word embeddings share similar structures across different languages and try to impose post-hoc alignment through a mapping~\cite{mikolov:13, Smith:17, joulin:18, Lample:18, artetxe:18, Zhou:19}. Another line of works considers joint training, which optimizes monolingual objective with or without cross-lingual constraints when training word embeddings~\cite{luong:15,gouws:15,Ammar:16,duong:16,lample:18b}. Cross-lingual word embedding methods above were initially proposed for non-contextualized embedding such as GloVe~\cite{pennington:14} and Word2Vec~\cite{mikolov:13w2v}, and later adapted to contextualized word representation~\cite{schuster:19, aldarmaki-diab:19}. &#10;&#10;Multilingual BERT (m-BERT) \cite{devlin:19} has shown its superior ability in cross-lingual transfer on many downstream tasks, either in a way it is used as a feature extractor or finetuned end-to-end~\cite{conneau:18, wu:19, hsu-etal:19, pires:19}. &#10;It seems that m-BERT has successfully learned a set of cross-lingual representations in a shared vector space for multiple languages~\cite{Cao:20}. &#10;However, given the way how m-BERT was pre-trained, it is unclear how it succeeded in building up cross-lingual ability without parallel resources and explicit supervised objectives.&#10;&#10;There is a line of work studying the key components contributing to the cross-lingual ability of m-BERT~\cite{Karth:20, tran:20, Cao:20, singh:19}. &#10;It was shown that {depth} and {total number of parameters} remarkably affect cross-lingual ability~\cite{Cao:20}. &#10;The conclusion about the impact of shared vocabulary is mixed~\cite{Karth:20, singh:19}, showing that our understandings about it are still in the early stages.&#10;&#10;In this paper, we study the impacts of some critical factors on the cross-lingual ability of m-BERT to enrich our understandings of how to build a powerful cross-lingual model. &#10;The contributions of this work can be summarized as the following:&#10;\begin{itemize}&#10; \item We found that large enough datasizes and modeling long term dependency are all necessary factors for the cross-lingual ability of m-BERT.&#10; \item We found that the non-contextualized word embedding training under the same condition as m-BERT does not show the same cross-lingual ability, which shows the uniqueness of m-BERT. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.05720" label="2404.05720">
        <attvalues>
          <attvalue for="0" value="Language-Independent Representations Improve Zero-Shot Summarization" />
          <attvalue for="1" value="  Finetuning pretrained models on downstream generation tasks often leads to&#10;catastrophic forgetting in zero-shot conditions. In this work, we focus on&#10;summarization and tackle the problem through the lens of language-independent&#10;representations. After training on monolingual summarization, we perform&#10;zero-shot transfer to new languages or language pairs. We first show naively&#10;finetuned models are highly language-specific in both output behavior and&#10;internal representations, resulting in poor zero-shot performance. Next, we&#10;propose query-key (QK) finetuning to decouple task-specific knowledge from the&#10;pretrained language generation abilities. Then, after showing downsides of the&#10;standard adversarial language classifier, we propose a balanced variant that&#10;more directly enforces language-agnostic representations. Moreover, our&#10;qualitative analyses show removing source language identity correlates to&#10;zero-shot summarization performance. Our code is openly available.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretrained multilingual models \cite{conneau-etal-2020-unsupervised,liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,lin-etal-2022-shot} have been established as promising sources of transfer learning,&#10;where task-specific finetuning benefits from the general knowledge learned on diverse unsupervised data.&#10;However,&#10;due to data or computational constraints,&#10;the task-specific data often only cover a limited subset of the languages in pretraining.&#10;Therefore,&#10;during finetuning&#10;it is crucial to retain the knowledge of the pretrained model and to enable zero-shot transfer, i.e., performing the task on more languages covered by the pretrained model.&#10;While zero-shot crosslingual transfer has shown very promising results on sequence classification or labeling problems \cite{pires-etal-2019-multilingual,DBLP:conf/nips/ConneauL19,wu-dredze-2019-beto},&#10;it remains challenging for generation tasks \cite{ronnqvist-etal-2019-multilingual,vu-etal-2022-overcoming,li-murray-2023-zero} including summarization and translation.&#10;A main obstacle is catastrophic forgetting \cite{DBLP:journals/neco/FrenchC02}, where languages supported by the pretrained model but not covered in the finetuning data are forgotten.&#10;In this work, &#10;we use summarization as a testbed for various types of zero-shot generation.&#10;As shown in \ref{fig:approach_overview}, &#10;given a pretrained model&#10;and intralingual summarization training data in some languages (A$arrow$A, B$arrow$B),&#10;we aim for zero-shot intralingual and crosslingual summarization on new languages (C$arrow$C) and language pairs (A$arrow$B, A$arrow$C) respectively.&#10;&#10;To alleviate catastrophic forgetting, &#10;one line of work trains on additional unsupervised data \cite{maurya-etal-2021-zmbart,vu-etal-2022-overcoming,DBLP:journals/corr/abs-2311-09344}.&#10;Besides the computational overhead,&#10;this approach raises a theoretical question:&#10;As the pretrained language model has already learned extensively on unsupervised data,&#10;is it necessary to re-learn language modeling in task-specific finetuning?&#10;We therefore explore a more challenging case of only using &#10;paired summarization data without relying on any unsupervised data.&#10;&#10;We identify two challenges when generalizing summarization abilities to new languages.&#10;First, &#10;decoupling the task-specific knowledge from the language generation abilities is essential.&#10;In response,&#10;we propose a new finetuning method based on query and keys, which is shown effective for both intralingual and crosslingual zero-shot setting.&#10;For crosslingual zero-shot settings,&#10;it is also crucial to decouple language from content, i.e., creating language-agnostic representations. &#10;This has been shown to facilitate zero-shot crosslingual generation in general \cite{pham-etal-2019-improving,wu-etal-2022-laft,DBLP:journals/corr/abs-2308-11466}.&#10;There a prominent approach is adversarial training \cite{DBLP:journals/jmlr/GaninUAGLLML16,chen-etal-2018-adversarial},&#10;where the model is trained to deceive a language classifier.&#10;We show the existing formulation fails to fully achieve language-agnostic representations,&#10;and improves it by explicitly incentivizing the model to deceive the classifier into a uniform class distribution.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Multilingual Summarization, Computer Science, Linguistics, Zero-Shot Transfer Learning, Artificial Intelligence, Natural Language Processing, Language Model Finetuning" />
        </attvalues>
      </node>
      <node id="2205.12647" label="2205.12647">
        <attvalues>
          <attvalue for="0" value="Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation" />
          <attvalue for="1" value="  In this paper, we explore the challenging problem of performing a generative&#10;task in a target language when labeled data is only available in English, using&#10;summarization as a case study. We assume a strict setting with no access to&#10;parallel data or machine translation and find that common transfer learning&#10;approaches struggle in this setting, as a generative multilingual model&#10;fine-tuned purely on English catastrophically forgets how to generate&#10;non-English. Given the recent rise of parameter-efficient adaptation&#10;techniques, we conduct the first investigation into how one such method, prompt&#10;tuning (Lester et al., 2021), can overcome catastrophic forgetting to enable&#10;zero-shot cross-lingual generation. Our experiments show that&#10;parameter-efficient prompt tuning provides gains over standard fine-tuning when&#10;transferring between less-related languages, e.g., from English to Thai.&#10;However, a significant gap still remains between these methods and&#10;fully-supervised baselines. To improve cross-lingual transfer further, we&#10;explore several approaches, including: (1) mixing in unlabeled multilingual&#10;data, and (2) explicitly factoring prompts into recombinable language and task&#10;components. Our approaches can provide further quality gains, suggesting that&#10;robust zero-shot cross-lingual generation is within reach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.17325" label="2305.17325">
        <attvalues>
          <attvalue for="0" value="Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a&#10;  Solution" />
          <attvalue for="1" value="  Zero-shot cross-lingual transfer is when a multilingual model is trained to&#10;perform a task in one language and then is applied to another language.&#10;Although the zero-shot cross-lingual transfer approach has achieved success in&#10;various classification tasks, its performance on natural language generation&#10;tasks falls short in quality and sometimes outputs an incorrect language. In&#10;our study, we show that the fine-tuning process learns language invariant&#10;representations, which is beneficial for classification tasks but harmful for&#10;generation tasks. Motivated by this, we propose a simple method to regularize&#10;the model from learning language invariant representations and a method to&#10;select model checkpoints without a development set in the target language, both&#10;resulting in better generation quality. Experiments on three semantically&#10;diverse generation tasks show that our method reduces the accidental&#10;translation problem by 68% and improves the ROUGE-L score by 1.5 on average.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Multilingual Language Models. &#10;One line of work is to train multilingual versions of modern Language Models. mBERT&#10; \cite{mBERT} is the multilingual version of BERT \cite{devlin-etal-2019-bert}, which uses the same encoder-only model architecture but is only trained on multilingual corpora. XLM-R \cite{conneau-etal-2020-unsupervised} is the multilingual version of RoBERTa \cite{liu2019roberta}, which implements multiple optimization tricks and is larger in scale, resulting in better performance than BERT. mBART \cite{liu-etal-2020-multilingual-denoising} is the multilingual version of BART \cite{lewis-etal-2020-bart}, an encoder-decoder model trained to reconstruct the original text through various types of artificially introduced noises. mT5 \cite{xue-etal-2021-mt5} is the multilingual version of T5 \cite{2020t5}, an encoder-decoder model trained on a span denoising objective. &#10;&#10;Cross-lingual Transfer. Multilingual models are able to be fine-tuned on annotated data of a task in only one source language and transfer the knowledge to other target languages to perform the same task without any supervision. While \cite{pires-etal-2019-multilingual} states that sub-word overlap between source and target facilitates cross-lingual transfer, \cite{K2020Cross-Lingual} shows that cross-lingual transfer manifests in pairs of source and target with zero sub-word overlap and word order is instead the most crucial ingredient. The performance of cross-lingual transfer between languages with a different order severely drops. Although the importance of word order is echoed by later studies (\cite{artetxe-etal-2020-cross}; \cite{dufter-schutze-2020-identifying}), recent studies have also debated in favor of the importance of matching script also contributing to cross-lingual transfer (\cite{lauscher-etal-2020-zero}; \cite{fujinuma-etal-2022-match}). &#10;\cite{wu-etal-2022-zero} points out that the optimal set of parameters that generalizes well to all languages is a subset of parameters that achieves good performance on the source language. Therefore it is hard to find the optimal zero-shot cross-lingual transfer parameters by only optimizing source language performance. \cite{chen-ritter-2021-model} train a scoring model with the input features being the model's hidden representations and the output score being how well it generalizes to a given target language. However, previous studies focus on lower-level NLP tasks, which include text classification, dependency parsing, and extractive question answering \cite{pmlr-v119-hu20b} and rarely touch on language generation. \\&#10; Another line of work focuses on applying cross-lingual transfer to a wide range of multilingual NLP applications, which include sequence tagging \cite{zhilin2016multitask}, Named Entity Recognition \cite{xie-etal-2018-neural}, dependency parsing \cite{ahmad-etal-2019-difficulties}, sentence classification (\cite{conneau-etal-2018-xnli}; \cite{yang-etal-2019-paws}), and information retrieval \cite{izacard2022unsupervised}. Empirical studies also train ranking models \cite{lin-etal-2019-choosing}, use meta-learning \cite{nooralahzadeh-etal-2020-zero}, or use Shapley Value \cite{parvez-chang-2021-evaluating} to predict which sources perform the best for a given target language. \\&#10;&#10;Natural Language Generation. Multilingual LMs are prone to produce text that is repetitive \cite{xu2022learning}, contains hallucinations \cite{raunak-etal-2021-curious}, or is in the wrong language (\cite{zhang-etal-2020-improving}; \cite{xue-etal-2021-mt5}; \cite{vu-2022-overcoming}). \cite{vu-2022-overcoming} proposed to use parameter efficient fine-tuning methods (\cite{lester-etal-2021-power}; \cite{qin-eisner-2021-learning}; \cite{li-liang-2021-prefix}) to regularize the model to generate in a desired language. Other ways to improve generation quality include using back translation (\cite{gu-etal-2019-improved}; \cite{zhang-etal-2020-improving}), and transliteration \cite{sun-etal-2022-alternative} as data augmentation techniques, mixing in the pretrain objective during fine-tuning \cite{xue-etal-2021-mt5} and using an auxiliary source language in machine translation \cite{xu-etal-2021-improving-multilingual}. Two concurrent efforts are close to our work: \cite{xu-murray-2022-por} and \cite{dontstop-emnlp-2022} both empirically show that using multiple languages during fine-tuning in few-shot cross-lingual transfer improves performance in text classification. Our work differs in that we evaluated text generation under a true zero-shot setting, where we have access to neither a few examples to train on nor an annotated development set to guide model checkpoint selection.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.01597" label="2106.01597">
        <attvalues>
          <attvalue for="0" value="ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language&#10;  Generation" />
          <attvalue for="1" value="  Despite the recent advancement in NLP research, cross-lingual transfer for&#10;natural language generation is relatively understudied. In this work, we&#10;transfer supervision from high resource language (HRL) to multiple low-resource&#10;languages (LRLs) for natural language generation (NLG). We consider four NLG&#10;tasks (text summarization, question generation, news headline generation, and&#10;distractor generation) and three syntactically diverse languages, i.e.,&#10;English, Hindi, and Japanese. We propose an unsupervised cross-lingual language&#10;generation framework (called ZmBART) that does not use any parallel or&#10;pseudo-parallel/back-translated data. In this framework, we further pre-train&#10;mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task&#10;using monolingual data of three languages. The objective function of the&#10;auxiliary task is close to the target tasks which enriches the multi-lingual&#10;latent representation of mBART and provides good initialization for target&#10;tasks. Then, this model is fine-tuned with task-specific supervised English&#10;data and directly evaluated with low-resource languages in the Zero-shot&#10;setting. To overcome catastrophic forgetting and spurious correlation issues,&#10;we applied freezing model component and data argumentation approaches&#10;respectively. This simple modeling approach gave us promising results.We&#10;experimented with few-shot training (with 1000 supervised data points) which&#10;boosted the model performance further. We performed several ablations and&#10;cross-lingual transferability analyses to demonstrate the robustness of ZmBART.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Early works on cross-lingual generation rely on machine translation (MT). In the very first work, \cite{wanetal2010cross} leveraged the MT pipeline for cross-language document summarization. They first translate the non-English test instances to English. This translated text is fed through the supervised model (trained with document summarization data in English) to generate English summaries. Finally, these summaries are translated back to the target language. \cite{ 10.1109/TASLP.2018.2842432} and \cite{duanetal2019zero} used MT systems to generate pseudo training data for cross-lingual summarization and news headline generation respectively. However these MT based models are not suitable for low resource languages as they do not share parameters across-languages and generated translations are error-prone. &#10;&#10;Recently there are a few works in the direction of supervision transfer from HRL(s) to LRL(s) for language generation. \cite{kumaretal2019cross} used back-translation (needs MT system) and annotated supervised data for cross-lingual question generation. \cite{Chi_Dong_Wei_Wang_Mao_Huang_2020} used parallel data to train a sequence-to-sequence model for zero-shot cross-lingual abstractive text summarization and question generation. \cite{DBLP:journals/corr/abs-2006-15020} proposed a pre-training based on mono-lingual paragraphs. Then this pre-trained model is used for zero-shot abstractive text summarization (ATS) in multiple languages. They trained a model on the ATS dataset on all the languages except the test language. This approach needs annotated data in multiple languages. Existing supervision transfer methods require parallel data for the cross-lingual tasks. Either they use available parallel corpora directly, or they translate/ back-translate data to generate pseudo-parallel corpora. Both these approaches pose significant challenges, as task-specific parallel data for multiple languages is difficult to obtain, and MT are far from perfect, especially for low resource languages. &#10;&#10;Unlike the previous approaches, we did not use any parallel data or back-translation in our proposed framework. We did not pre-train any model from scratch. Instead, we leveraged the existing pre-trained model mBART. We included four challenging generation tasks across three syntactically diverse languages. Even we did not modify any hyper-parameters across the tasks and languages. All these considerations make the framework simple and easy to use. Further, it enables the addition of different other languages and NLG tasks in the proposed framework a simple extension exercise.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2311.09344" label="2311.09344">
        <attvalues>
          <attvalue for="0" value="Language and Task Arithmetic with Parameter-Efficient Layers for&#10;  Zero-Shot Summarization" />
          <attvalue for="1" value="  Parameter-efficient fine-tuning (PEFT) using labeled task data can&#10;significantly improve the performance of large language models (LLMs) on the&#10;downstream task. However, there are 7000 languages in the world and many of&#10;these languages lack labeled data for real-world language generation tasks. In&#10;this paper, we propose to improve zero-shot cross-lingual transfer by composing&#10;language or task specialized parameters. Our method composes language and task&#10;PEFT modules via element-wise arithmetic operations to leverage unlabeled data&#10;and English labeled data. We extend our approach to cases where labeled data&#10;from more languages is available and propose to arithmetically compose PEFT&#10;modules trained on languages related to the target. Empirical results on&#10;summarization demonstrate that our method is an effective strategy that obtains&#10;consistent gains using minimal training of PEFT modules.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.11466" label="2308.11466">
        <attvalues>
          <attvalue for="0" value="SONAR: Sentence-Level Multimodal and Language-Agnostic Representations" />
          <attvalue for="1" value="  We introduce SONAR, a new multilingual and multimodal fixed-size sentence&#10;embedding space. Our single text encoder, covering 200 languages, substantially&#10;outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim&#10;and xsim++ multilingual similarity search tasks. Speech segments can be&#10;embedded in the same SONAR embedding space using language-specific speech&#10;encoders trained in a teacher-student setting on speech transcription data. Our&#10;encoders outperform existing speech encoders on similarity search tasks. We&#10;also provide a text decoder for 200 languages, which allows us to perform&#10;text-to-text and speech-to-text machine translation, including for zero-shot&#10;language and modality combinations. Our text-to-text results are competitive&#10;compared to the state-of-the-art NLLB~1B model, despite the fixed-size&#10;bottleneck representation. Our zero-shot speech-to-text translation results&#10;compare favorably with strong supervised baselines such as Whisper.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1505.07818" label="1505.07818">
        <attvalues>
          <attvalue for="0" value="Domain-Adversarial Training of Neural Networks" />
          <attvalue for="1" value="  We introduce a new representation learning approach for domain adaptation, in&#10;which data at training and test time come from similar but different&#10;distributions. Our approach is directly inspired by the theory on domain&#10;adaptation suggesting that, for effective domain transfer to be achieved,&#10;predictions must be made based on features that cannot discriminate between the&#10;training (source) and test (target) domains. The approach implements this idea&#10;in the context of neural network architectures that are trained on labeled data&#10;from the source domain and unlabeled data from the target domain (no labeled&#10;target-domain data is necessary). As the training progresses, the approach&#10;promotes the emergence of features that are (i) discriminative for the main&#10;learning task on the source domain and (ii) indiscriminate with respect to the&#10;shift between the domains. We show that this adaptation behaviour can be&#10;achieved in almost any feed-forward model by augmenting it with few standard&#10;layers and a new gradient reversal layer. The resulting augmented architecture&#10;can be trained using standard backpropagation and stochastic gradient descent,&#10;and can thus be implemented with little effort using any of the deep learning&#10;packages. We demonstrate the success of our approach for two distinct&#10;classification problems (document sentiment analysis and image classification),&#10;where state-of-the-art domain adaptation performance on standard benchmarks is&#10;achieved. We also validate the approach for descriptor learning task in the&#10;context of person re-identification application.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.01614" label="1606.01614">
        <attvalues>
          <attvalue for="0" value="Adversarial Deep Averaging Networks for Cross-Lingual Sentiment&#10;  Classification" />
          <attvalue for="1" value="  In recent years great success has been achieved in sentiment classification&#10;for English, thanks in part to the availability of copious annotated resources.&#10;Unfortunately, most languages do not enjoy such an abundance of labeled data.&#10;To tackle the sentiment classification problem in low-resource languages&#10;without adequate annotated data, we propose an Adversarial Deep Averaging&#10;Network (ADAN) to transfer the knowledge learned from labeled data on a&#10;resource-rich source language to low-resource languages where only unlabeled&#10;data exists. ADAN has two discriminative branches: a sentiment classifier and&#10;an adversarial language discriminator. Both branches take input from a shared&#10;feature extractor to learn hidden representations that are simultaneously&#10;indicative for the classification task and invariant across languages.&#10;Experiments on Chinese and Arabic sentiment classification demonstrate that&#10;ADAN significantly outperforms state-of-the-art systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.04003" label="2202.04003">
        <attvalues>
          <attvalue for="0" value="Differentiable N-gram Objective on Abstractive Summarization" />
          <attvalue for="1" value="  ROUGE is a standard automatic evaluation metric based on n-grams for&#10;sequence-to-sequence tasks, while cross-entropy loss is an essential objective&#10;of neural network language model that optimizes at a unigram level. We present&#10;differentiable n-gram objectives, attempting to alleviate the discrepancy&#10;between training criterion and evaluating criterion. The objective maximizes&#10;the probabilistic weight of matched sub-sequences, and the novelty of our work&#10;is the objective weights the matched sub-sequences equally and does not ceil&#10;the number of matched sub-sequences by the ground truth count of n-grams in&#10;reference sequence. We jointly optimize cross-entropy loss and the proposed&#10;objective, providing decent ROUGE score enhancement over abstractive&#10;summarization dataset CNN/DM and XSum, outperforming alternative n-gram&#10;objectives.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic text summarization has attracted much attention in recent years. &#10;Text summarization can be classified into extractive or abstractive summarization.&#10;The goal of extractive summarization is identifying and extracting a few important and comprehensive sentences from the source docuements \cite{zhou-etal-2018-neural-document, Nallapati_Zhai_Zhou_2017}.&#10;Abstractive summarization compresses the context of source material and rewrites the text into shorter version while retaining the meanings \cite{ELKASSAS2021113679}.&#10;Abstractive summarization is commonly viewed as a sequence-to-sequence (seq2seq) learning process \cite{Sutskever2014}.&#10;We concentrate on abstractive approaches in this paper.&#10;&#10;Seq2seq language models \cite{Bahdanau2015} with attention mechanism have dominated various natural language processing (NLP) downstream tasks\cite{rush2015, chopra2016, Vaswani2017, Radford2018, Devlin2019, Raffel2019, Lewis2020, Qi2020}. &#10;In recent years, large pre-trained language models with self-supervised autoencoder and autoregressive generation have greatly improved the semantic quality of summarization.&#10;It is also worth noticing that language model with representation learning methods (\eg, bottom-up \cite{gehrmann2018bottomup}, contrastive learning \cite{Xu_Zhang_Wu_Wei_2022} and external faithful signals \cite{dou-etal-2021-gsum}) can effectively improve the quality of machine generated summaries.&#10;Recent works investigated external commonsence enhanced representation \cite{yang-human-Like-2021} and topic-oriented semantic representation \cite{nguyen-etal-2021-enriching} for abstractive summarization.&#10;To efficiently evaluate the consistency, conciseness and faithfulness of machine generated summaries, deep neural network based evaluation is receiving increasing attention \cite{kryscinski2020factCC, LabanSBH22Summac, liu-etal-2022-brio, ladhak-etal-2022-faithful}.&#10;&#10;Most of the language models use cross-entropy loss as the learning objective, which is simple and standard for seq2seq model. &#10;Automatic evaluation methods based on $N$-gram, e.g., recall-oriented understudy for gisting evaluation (ROUGE) metric \cite{Lin2004} and bilingual evaluation understudy (BLEU) metric \cite{Papineni2002}, are widely-used for generation tasks such as text summarization \cite{Bhandari2020} and machine translation. Meanwhile the widely used cross-entropy objective merely considers exact token matching from the reference text by rewarding the probability of ground truth token and diminishing the others. &#10;The discrepancy between the learning objective and the evaluation objective (usually referred as {exposure bias} \cite{Ranzato2016}) may make the language model miss some reliable alternative sub-sequences, while humanly text summarization tends be tolerant of alternative language expressions or shifted sub-sequences.&#10;&#10;\cite{yang-etal2018} and \cite{li-piccardi-2021} used generative adversarial neural network to optimize BLEU metric on machine translation.&#10;\cite{Ranzato2016} and \cite{Wu-etal2016} tried bridging the discrepancy by rewarding BLEU and grammaticality evaluation utility (GLEU) metrics for text generation with reinforcement learning \cite{Williams1989}.&#10;\cite{paulus2018} applied reinforcement learning approach on text summarization. However, high variance of sampling can make the process of reinforcement learning unstable and less reproducible. \cite{ma2018bagofwords} and \cite{Shao2020BoN} proposed bag-of-words \cite{Joachims1998} and bag-of-ngrams (BoN) as sequence-level training target respectively. &#10;They intented to minimize the gap between the probability distribution of every token in the output sequence and a set of token or $N$-gram from reference sequence.&#10;\cite{Zhukov2017} introduced a lower bound approximation of expected BLEU score for sequence generation.&#10;\cite{Casas2018} proposed a differentiable BLEU objective with approximation of $N$-gram matches using Gumbel-softmax \cite{jang2017}.&#10;\cite{yavuz2018} attempted to develop approximation objective for the longest common sub-sequence.&#10;\cite{shao2018greedy} proposed a differentiable probabilistic $N$-gram count objective by maximizing the probabilistic count of matched $N$-gram. $N$-gram matching is verified with the argmax result of the final output of the language model.&#10;However their objective will stop rewarding the probabilistic count of an $N$-gram if its expectation reaches the corresponding count in reference sequence.&#10;Although cross-entropy is the most contributing learning objective,&#10;probabilistic $N$-gram objectives can serve for representation learning to shape the probabilistic distribution of the hidden state output of a language model.&#10;Since the probabilistic $N$-gram is independently recognized from the final hidden state, removing the $N$-gram count ceiling could enable the language model to reward complete $N$-gram matches in the training stage.&#10;This is particularly important to achieve satisfying weighted linear combination for $N$-gram matches when weight vectors of the probabilistic $N$-gram count cannot define the simplex (i.e., sum of the weight vectors exceed 1).&#10;&#10;Inspired by probabilistic $N$-gram count objective \cite{shao2018greedy} and BoN objective \cite{Shao2020BoN}, we want the learning objective to be flexible for $N$-gram matching which is not limited by the reference $N$-gram count ceiling, &#10;and expect the co-occurrences of $N$-gram have equal weights in the objective, &#10;yet retaining the capability of alleviating the exposure bias. In this paper, we propose two differentiable $N$-gram objectives: &#10;1) differentiable $N$-gram rewards: a position-related $N$-gram matches objective that maximizes the probabilistic of matched $N$-gram between the output sequence and reference sequence. The matched sub-sequence is counted only if it shares the uniform position in candidate text and reference text;&#10;2) differentiable $N$-gram matches: a position-unrelated $N$-gram matches objective that maximizes the probabilistic of matched $N$-gram between the output sequence and reference sequence. The matched sub-sequence is not constrained by positions. &#10;Unlike \cite{shao2018greedy, Shao2020BoN}, our $N$-gram rewards objective and $N$-gram matches objective value the matched $N$-gram equally, and continue optimizing a matched $N$-gram even if it exceeds the count of this $N$-gram in reference text.&#10;We fine-tune the pre-trained model BART-base on abstractive summarization task, jointly optimizing the proposed objective and cross-entropy loss.&#10;Finally, experiment results show that our algorithm outperforms the probabilistic $N$-gram count objective and BoN objective with cross-entropy loss on text summarization.&#10;&#10;Our contributions are listed as follows.&#10;\begin{itemize}&#10; \item Our proposal, jointly optimizing cross-entropy loss and $N$-gram objective that equally weights $N$-gram matches and does not have a ceiling for $N$-gram matches, can deliver decent evaluation scores improvement on text summarization and surpass alternative $N$-gram objectives. &#10; \item We conduct extensive experimental evaluation on the CNN/DailyMail dataset and the XSum dataset using ROUGE scores, &#10; BERTscore, Word Mover's Distance, FactCCX and SummaC\textsubscript{CONV} as summarization metrics, &#10; confirming the improvement made by our proposed objectives.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Sequence-to-Sequence Tasks, Computer Science, Linguistics, Artificial Intelligence, Mathematics, Evaluation Metric Enhancement, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1808.10792" label="1808.10792">
        <attvalues>
          <attvalue for="0" value="Bottom-Up Abstractive Summarization" />
          <attvalue for="1" value="  Neural network-based methods for abstractive summarization produce outputs&#10;that are more fluent than other techniques, but which can be poor at content&#10;selection. This work proposes a simple technique for addressing this issue: use&#10;a data-efficient content selector to over-determine phrases in a source&#10;document that should be part of the summary. We use this selector as a&#10;bottom-up attention step to constrain the model to likely phrases. We show that&#10;this approach improves the ability to compress text, while still generating&#10;fluent summaries. This two-step process is both simpler and higher performing&#10;than other end-to-end content selection models, leading to significant&#10;improvements on ROUGE for both the CNN-DM and NYT corpus. Furthermore, the&#10;content selector can be trained with as little as 1,000 sentences, making it&#10;easy to transfer a trained summarizer to a new domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.03481" label="2109.03481">
        <attvalues>
          <attvalue for="0" value="Sequence Level Contrastive Learning for Text Summarization" />
          <attvalue for="1" value="  Contrastive learning models have achieved great success in unsupervised&#10;visual representation learning, which maximize the similarities between feature&#10;representations of different views of the same image, while minimize the&#10;similarities between feature representations of views of different images. In&#10;text summarization, the output summary is a shorter form of the input document&#10;and they have similar meanings. In this paper, we propose a contrastive&#10;learning model for supervised abstractive text summarization, where we view a&#10;document, its gold summary and its model generated summaries as different views&#10;of the same mean representation and maximize the similarities between them&#10;during training. We improve over a strong sequence-to-sequence text generation&#10;model (i.e., BART) on three different summarization datasets. Human evaluation&#10;also shows that our model achieves better faithfulness ratings compared to its&#10;counterpart without contrastive objectives.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.08014" label="2010.08014">
        <attvalues>
          <attvalue for="0" value="GSum: A General Framework for Guided Neural Abstractive Summarization" />
          <attvalue for="1" value="  Neural abstractive summarization models are flexible and can produce coherent&#10;summaries, but they are sometimes unfaithful and can be difficult to control.&#10;While previous studies attempt to provide different types of guidance to&#10;control the output and increase faithfulness, it is not clear how these&#10;strategies compare and contrast to each other. In this paper, we propose a&#10;general and extensible guided summarization framework (GSum) that can&#10;effectively take different kinds of external guidance as input, and we perform&#10;experiments across several different varieties. Experiments demonstrate that&#10;this model is effective, achieving state-of-the-art performance according to&#10;ROUGE on 4 popular summarization datasets when using highlighted sentences as&#10;guidance. In addition, we show that our guided model can generate more faithful&#10;summaries and demonstrate how different types of guidance generate&#10;qualitatively different summaries, lending a degree of controllability to the&#10;learned models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.10616" label="2109.10616">
        <attvalues>
          <attvalue for="0" value="Enriching and Controlling Global Semantics for Text Summarization" />
          <attvalue for="1" value="  Recently, Transformer-based models have been proven effective in the&#10;abstractive summarization task by creating fluent and informative summaries.&#10;Nevertheless, these models still suffer from the short-range dependency&#10;problem, causing them to produce summaries that miss the key points of&#10;document. In this paper, we attempt to address this issue by introducing a&#10;neural topic model empowered with normalizing flow to capture the global&#10;semantics of the document, which are then integrated into the summarization&#10;model. In addition, to avoid the overwhelming effect of global semantics on&#10;contextualized representation, we introduce a mechanism to control the amount&#10;of global semantics supplied to the text generation module. Our method&#10;outperforms state-of-the-art summarization models on five common text&#10;summarization datasets, namely CNN/DailyMail, XSum, Reddit TIFU, arXiv, and&#10;PubMed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.07100" label="2010.07100">
        <attvalues>
          <attvalue for="0" value="Re-evaluating Evaluation in Text Summarization" />
          <attvalue for="1" value="  Automated evaluation metrics as a stand-in for manual evaluation are an&#10;essential part of the development of text-generation tasks such as text&#10;summarization. However, while the field has progressed, our standard metrics&#10;have not -- for nearly 20 years ROUGE has been the standard evaluation in most&#10;summarization papers. In this paper, we make an attempt to re-evaluate the&#10;evaluation method for text summarization: assessing the reliability of&#10;automatic metrics using top-scoring system outputs, both abstractive and&#10;extractive, on recently popular datasets for both system-level and&#10;summary-level evaluation settings. We find that conclusions about evaluation&#10;metrics on older datasets do not necessarily hold on modern datasets and&#10;systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.04887" label="1703.04887">
        <attvalues>
          <attvalue for="0" value="Improving Neural Machine Translation with Conditional Sequence&#10;  Generative Adversarial Nets" />
          <attvalue for="1" value="  This paper proposes an approach for applying GANs to NMT. We build a&#10;conditional sequence generative adversarial net which comprises of two&#10;adversarial sub models, a generator and a discriminator. The generator aims to&#10;generate sentences which are hard to be discriminated from human-translated&#10;sentences (i.e., the golden target sentences), And the discriminator makes&#10;efforts to discriminate the machine-generated sentences from human-translated&#10;ones. The two sub models play a mini-max game and achieve the win-win situation&#10;when they reach a Nash Equilibrium. Additionally, the static sentence-level&#10;BLEU is utilized as the reinforced objective for the generator, which biases&#10;the generation towards high BLEU points. During training, both the dynamic&#10;discriminator and the static BLEU objective are employed to evaluate the&#10;generated sentences and feedback the evaluations to guide the learning of the&#10;generator. Experimental results show that the proposed model consistently&#10;outperforms the traditional RNNSearch and the newly emerged state-of-the-art&#10;Transformer on English-German and Chinese-English translation tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.04871" label="1805.04871">
        <attvalues>
          <attvalue for="0" value="Bag-of-Words as Target for Neural Machine Translation" />
          <attvalue for="1" value="  A sentence can be translated into more than one correct sentences. However,&#10;most of the existing neural machine translation models only use one of the&#10;correct translations as the targets, and the other correct sentences are&#10;punished as the incorrect sentences in the training stage. Since most of the&#10;correct translations for one sentence share the similar bag-of-words, it is&#10;possible to distinguish the correct translations from the incorrect ones by the&#10;bag-of-words. In this paper, we propose an approach that uses both the&#10;sentences and the bag-of-words as targets in the training stage, in order to&#10;encourage the model to generate the potentially correct sentences that are not&#10;appeared in the training set. We evaluate our model on a Chinese-English&#10;translation dataset, and experiments show our model outperforms the strong&#10;baselines by the BLEU score of 4.55.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.09320" label="1911.09320">
        <attvalues>
          <attvalue for="0" value="Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural&#10;  Machine Translation" />
          <attvalue for="1" value="  Non-Autoregressive Neural Machine Translation (NAT) achieves significant&#10;decoding speedup through generating target words independently and&#10;simultaneously. However, in the context of non-autoregressive translation, the&#10;word-level cross-entropy loss cannot model the target-side sequential&#10;dependency properly, leading to its weak correlation with the translation&#10;quality. As a result, NAT tends to generate influent translations with&#10;over-translation and under-translation errors. In this paper, we propose to&#10;train NAT to minimize the Bag-of-Ngrams (BoN) difference between the model&#10;output and the reference sentence. The bag-of-ngrams training objective is&#10;differentiable and can be efficiently calculated, which encourages NAT to&#10;capture the target-side sequential dependency and correlates well with the&#10;translation quality. We validate our approach on three translation tasks and&#10;show that our approach largely outperforms the NAT baseline by about 5.0 BLEU&#10;scores on WMT14 En$\leftrightarrow$De and about 2.5 BLEU scores on WMT16&#10;En$\leftrightarrow$Ro.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.04708" label="1712.04708">
        <attvalues>
          <attvalue for="0" value="Differentiable lower bound for expected BLEU score" />
          <attvalue for="1" value="  In natural language processing tasks performance of the models is often&#10;measured with some non-differentiable metric, such as BLEU score. To use&#10;efficient gradient-based methods for optimization, it is a common workaround to&#10;optimize some surrogate loss function. This approach is effective if&#10;optimization of such loss also results in improving target metric. The&#10;corresponding problem is referred to as loss-evaluation mismatch. In the&#10;present work we propose a method for calculation of differentiable lower bound&#10;of expected BLEU score that does not involve computationally expensive sampling&#10;procedure such as the one required when using REINFORCE rule from reinforcement&#10;learning (RL) framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.01144" label="1611.01144">
        <attvalues>
          <attvalue for="0" value="Categorical Reparameterization with Gumbel-Softmax" />
          <attvalue for="1" value="  Categorical variables are a natural choice for representing discrete&#10;structure in the world. However, stochastic neural networks rarely use&#10;categorical latent variables due to the inability to backpropagate through&#10;samples. In this work, we present an efficient gradient estimator that replaces&#10;the non-differentiable sample from a categorical distribution with a&#10;differentiable sample from a novel Gumbel-Softmax distribution. This&#10;distribution has the essential property that it can be smoothly annealed into a&#10;categorical distribution. We show that our Gumbel-Softmax estimator outperforms&#10;state-of-the-art gradient estimators on structured output prediction and&#10;unsupervised generative modeling tasks with categorical latent variables, and&#10;enables large speedups on semi-supervised classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.03132" label="1809.03132">
        <attvalues>
          <attvalue for="0" value="Greedy Search with Probabilistic N-gram Matching for Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Neural machine translation (NMT) models are usually trained with the&#10;word-level loss using the teacher forcing algorithm, which not only evaluates&#10;the translation improperly but also suffers from exposure bias. Sequence-level&#10;training under the reinforcement framework can mitigate the problems of the&#10;word-level loss, but its performance is unstable due to the high variance of&#10;the gradient estimation. On these grounds, we present a method with a&#10;differentiable sequence-level training objective based on probabilistic n-gram&#10;matching which can avoid the reinforcement framework. In addition, this method&#10;performs greedy search in the training which uses the predicted words as&#10;context just as at inference to alleviate the problem of exposure bias.&#10;Experiment results on the NIST Chinese-to-English translation tasks show that&#10;our method significantly outperforms the reinforcement-based algorithms and&#10;achieves an improvement of 1.5 BLEU points on average over a strong baseline&#10;system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.13708" label="2205.13708">
        <attvalues>
          <attvalue for="0" value="HiJoNLP at SemEval-2022 Task 2: Detecting Idiomaticity of Multiword&#10;  Expressions using Multilingual Pretrained Language Models" />
          <attvalue for="1" value="  This paper describes an approach to detect idiomaticity only from the&#10;contextualized representation of a MWE over multilingual pretrained language&#10;models. Our experiments find that larger models are usually more effective in&#10;idiomaticity detection. However, using a higher layer of the model may not&#10;guarantee a better performance. In multilingual scenarios, the convergence of&#10;different languages are not consistent and rich-resource languages have big&#10;advantages over other languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the past several years, there have been breakthroughs in a variety of natural language processing tasks with the power of pretrained language models.&#10;These include but are not limit to question answering~\cite{devlin-etal-2019-bert}, language generation~\cite{radford2018improving,radford2019language} and machine translation~\cite{liu-etal-2020-multilingual-denoising}.&#10;However, it's still not clear whether pretrained language models have the ability in capturing the meanings of multiword expressions (MWEs), especially idioms.&#10;Given the prevalent usage of idioms in different languages, identifying the correct meaning of a phrase in a certain context is crucial for many downstream tasks including sentiment analysis~\cite{WILLIAMS20157375}, automatic spelling correction~\cite{horbach-etal-2016-corpus} and machine translation~\cite{isabelle-etal-2017-challenge}.&#10;&#10;In literature, idiomaticity detection has been a research topic drawing much attention from the NLP community.&#10;MWEs which have both an idiomatic interpretation and a literal interpretation are also referred as Potentially Idiomatic Expressions~(PIEs), for example, spill the beans.&#10;There has been both supervised~\cite{sporleder-li-2009-unsupervised} and unsupervised~\cite{haagsma-etal-2018-side,kurfali-ostling-2020-disambiguation} approaches to solve this problem.&#10;For example, \newcite{feldman-etal-cicling-2013-automatic} treated idiom recognition as outlier detection, which does not rely on costly annotated training data.&#10;\newcite{peng-etal-2014-classifying} incorporated the affective hypothesis of idioms to facilitate the identification of idiomatic operations. &#10;&#10;Due to the limited understanding of how pretrained language models may handle representation of phrases, a series of works are proposed to investigate phrase composition from their contextualized representations.&#10;\cite{yu-ettinger-2020-assessing} conduct analysis of phrasal representations in state-of-the-art pre-trained transformers and find that phrase representation in these models still relies heavily on word content, showing little evidence of nuanced composition. &#10;\cite{shwartz-dagan-2019-still} confirm that contextualized word representations perform better than static word embeddings, more so on detecting meaning shift than in recovering implicit information.&#10;Therefore, it remains a challenging problem to resolve the idiomaticity of phrases.&#10;&#10;Specifically on idiomaticity, recent approaches are trying to further diagnose pretrained language models using new metrics and datasets.&#10;\cite{garcia-etal-2021-assessing} analyse different levels of contextualisation to check to what extent models are able to detect idiomaticity at type and token level.&#10;\cite{garcia-etal-2021-probing} propose probing measures to assess Noun Compound~(NC) idiomaticity and conclude that idiomaticity is not yet accurately represented by contextualised models.&#10;AStitchInLanguageModels~\cite{tayyar-madabushi-etal-2021-astitchinlanguagemodels-dataset} design two tasks to first test a language model’s ability to detect idiom usage, and the effectiveness of a language model in generating representations of sentences containing idioms.&#10;\cite{tan-jiang-2021-bert} conduct two probing tasks, PIE usage classification and idiom paraphrase identification, suggesting that BERT indeed is able to separate the literal and idiomatic usages of a PIE with high accuracy and is also able to encode the idiomatic meaning of a PIE to some extent.&#10;However, there's still much more to explore in idiomaticity.&#10;&#10;Based upon AStitchInLanguageModels~\cite{tayyar-madabushi-etal-2021-astitchinlanguagemodels-dataset}, SemEval-2022 Task2~\cite{tayyarmadabushi-etal-2022-semeval} is proposed with a focus on multilingual idiomaticity.&#10;The task is arranged consisting the two subtasks:&#10;\begin{enumerate}&#10; \item Subtask A: A binary classification task aimed at determining whether a sentence contains an idiomatic expression.&#10; \item Subtask B: Pretrain or finetune a model which is expected to output the correct Semantic Text Similarity (STS) scores between sentence pairs, whether or not either sentence contains an idiomatic expression.&#10;\end{enumerate}&#10;&#10;In this paper, we focus on Subtask A and investigate how the span representation of a MWE can tell about its idiomaticity.&#10;We extend one of the monolingual idiomaticity probing method~\cite{tan-jiang-2021-bert} to multilingual scenario and compare multiple settings using multi-lingual BERT~(mBERT)~\cite{devlin-etal-2019-bert} and XLM-R~\cite{conneau-etal-2020-unsupervised}.&#10;Following \cite{yu-ettinger-2020-assessing}, we also consider variations of phrase representations across models, layers, and representation types.&#10;Different from them, we use more representation types to conduct the experiments.&#10;&#10;Our main conclusion from these experiments are two folds: &#10;\begin{enumerate}&#10; \item Larger models are usually more effective in idiomaticity detection. However, a higher layer may not contribute more to the idiomaticity detection task, or more contextualization does not guarantee a better performance.&#10; \item For multilingual scenario, the convergence of different languages are not consistent. Rich resource languages have initiative advantages over other languages.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Representation, Computer Science, Linguistics, Idiomaticity Detection, Multilingual Models, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1704.07431" label="1704.07431">
        <attvalues>
          <attvalue for="0" value="A Challenge Set Approach to Evaluating Machine Translation" />
          <attvalue for="1" value="  Neural machine translation represents an exciting leap forward in translation&#10;quality. But what longstanding weaknesses does it resolve, and which remain? We&#10;address these questions with a challenge set approach to translation evaluation&#10;and error analysis. A challenge set consists of a small set of sentences, each&#10;hand-designed to probe a system's capacity to bridge a particular structural&#10;divergence between languages. To exemplify this approach, we present an&#10;English-French challenge set, and use it to analyze phrase-based and neural&#10;systems. The resulting analysis provides not only a more fine-grained picture&#10;of the strengths of neural systems, but also insight into which linguistic&#10;phenomena remain out of reach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03763" label="2010.03763">
        <attvalues>
          <attvalue for="0" value="Assessing Phrasal Representation and Composition in Transformers" />
          <attvalue for="1" value="  Deep transformer models have pushed performance on NLP tasks to new limits,&#10;suggesting sophisticated treatment of complex linguistic inputs, such as&#10;phrases. However, we have limited understanding of how these models handle&#10;representation of phrases, and whether this reflects sophisticated composition&#10;of phrase meaning like that done by humans. In this paper, we present&#10;systematic analysis of phrasal representations in state-of-the-art pre-trained&#10;transformers. We use tests leveraging human judgments of phrase similarity and&#10;meaning shift, and compare results before and after control of word overlap, to&#10;tease apart lexical effects versus composition effects. We find that phrase&#10;representation in these models relies heavily on word content, with little&#10;evidence of nuanced composition. We also identify variations in phrase&#10;representation quality across models, layers, and representation types, and&#10;make corresponding recommendations for usage of representations from these&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.10618" label="1902.10618">
        <attvalues>
          <attvalue for="0" value="Still a Pain in the Neck: Evaluating Text Representations on Lexical&#10;  Composition" />
          <attvalue for="1" value="  Building meaningful phrase representations is challenging because phrase&#10;meanings are not simply the sum of their constituent meanings. Lexical&#10;composition can shift the meanings of the constituent words and introduce&#10;implicit information. We tested a broad range of textual representations for&#10;their capacity to address these issues. We found that as expected,&#10;contextualized word representations perform better than static word embeddings,&#10;more so on detecting meaning shift than in recovering implicit information, in&#10;which their performance is still far from that of humans. Our evaluation suite,&#10;including 5 tasks related to lexical composition effects, can serve future&#10;research aiming to improve such representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.10050" label="2204.10050">
        <attvalues>
          <attvalue for="0" value="SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence&#10;  Embedding" />
          <attvalue for="1" value="  This paper presents the shared task on Multilingual Idiomaticity Detection&#10;and Sentence Embedding, which consists of two subtasks: (a) a binary&#10;classification task aimed at identifying whether a sentence contains an&#10;idiomatic expression, and (b) a task based on semantic text similarity which&#10;requires the model to adequately represent potentially idiomatic expressions in&#10;context. Each subtask includes different settings regarding the amount of&#10;training data. Besides the task description, this paper introduces the datasets&#10;in English, Portuguese, and Galician and their annotation procedure, the&#10;evaluation metrics, and a summary of the participant systems and their results.&#10;The task had close to 100 registered participants organised into twenty five&#10;teams making over 650 and 150 submissions in the practice and evaluation phases&#10;respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.09109" label="2006.09109">
        <attvalues>
          <attvalue for="0" value="How to Probe Sentence Embeddings in Low-Resource Languages: On&#10;  Structural Design Choices for Probing Task Evaluation" />
          <attvalue for="1" value="  Sentence encoders map sentences to real valued vectors for use in downstream&#10;applications. To peek into these representations - e.g., to increase&#10;interpretability of their results - probing tasks have been designed which&#10;query them for linguistic knowledge. However, designing probing tasks for&#10;lesser-resourced languages is tricky, because these often lack large-scale&#10;annotated data or (high-quality) dependency parsers as a prerequisite of&#10;probing task design in English. To investigate how to probe sentence embeddings&#10;in such cases, we investigate sensitivity of probing task results to structural&#10;design choices, conducting the first such large scale study. We show that&#10;design choices like size of the annotated probing dataset and type of&#10;classifier used for evaluation do (sometimes substantially) influence probing&#10;outcomes. We then probe embeddings in a multilingual setup with design choices&#10;that lie in a 'stable region', as we identify for English, and find that&#10;results on English do not transfer to other languages. Fairer and more&#10;comprehensive sentence-level probing evaluation should thus be carried out on&#10;multiple languages in the future.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Our goal &#10;is to probe for sentence-level linguistic knowledge encoded in sentence embeddings \cite{Perone.2018} in a &#10;multilingual setup &#10;which marginalizes out the effects of probing task design choices when comparing sentence representations.&#10;&#10;Sentence embeddings have become &#10;central &#10;for representing texts beyond the word level, e.g., in small data scenarios, where it is difficult to induce good higher-level text representations from word embeddings \cite{subramanian2018learning} or for clustering or text retrieval applications \cite{Reimers.2019}. &#10;To standardize the comparison of sentence embeddings, \newcite{conneau-kiela-2018-senteval} proposed the SentEval framework for evaluating the quality of sentence embeddings on a range of downstream and 10 probing tasks.&#10;&#10;Probing tasks are used to introspect embeddings for linguistic knowledge, by taking ``probes'' as dedicated syntactic or semantic micro tasks&#10;\cite{kohn-2016-evaluating}.&#10;As opposed to an evaluation in downstream applications or benchmarks like GLUE \cite{wang-etal-2018-glue}, &#10;probing tasks &#10;target very specific linguistic knowledge which may otherwise be confounded in downstream applications. &#10;Since they are artificial tasks, they can also be better controlled for to avoid dataset biases and artifacts. Probing is typically either executed on type/token (word) \cite{tenney2018what} or sentence level \cite{Adi.2017}. &#10;For sentence level evaluation, &#10;SentEval thus far only includes en data. &#10;Each probing task in SentEval is balanced and has 100k train, 10k dev, and 10k test instances. &#10;The effects of these design choices are unclear, which is why our work addresses their influence systematically.&#10;&#10;In the multilingual setting, &#10;\newcite{DBLP:journals/corr/abs-1903-09442} propose 15 token and type level probing tasks. &#10;Their probing task data is sourced from UniMorph 2.0 \cite{kirov-etal-2018-unimorph}, Universal Dependency treebanks \cite{mccarthy-etal-2018-marrying} and Wikipedia word frequency lists.&#10;To deal with lower-resourced languages, they only use 10K samples per probing task/language pair (7K/2K/1K for train/dev/test) and exclude task/language pairs for which this amount cannot be generated.&#10;Their final experiments are carried out &#10;on &#10;five languages (Finnish, German, Spanish, ru, tr), &#10;for which enough training data is available. &#10;They find that for morphologically rich (agglutinative) languages, several probing tasks positively correlate with downstream applications. Our work also investigates correlation between probing and downstream performance, but we do so on sentence level. &#10;&#10;On sentence level, \newcite{ravishankar-etal-2019-probing} &#10;train an InferSent-like encoder \cite{Conneau.2017} on en and map this encoder to four languages (ru, French, German, Spanish) using parallel data.&#10;Subsequently, they &#10;probe the encoders on the probing tasks proposed by \newcite{Conneau.2018a} using Wikipedia data for each language, &#10;with the same size of probing task data as in SentEval, i.e., 100k/10k/10k for train/dev/test. &#10;Their interest is in whether probing tasks results are higher/lower compared to en scores. They find particularly the ru probing scores to be low, which they speculate to be an artifact of cross-lingual word embedding induction and the language distance of ru to en. &#10;In contrast to us, their focus is particularly on the effect of transferring sentence representations from en to other languages. The problem of such an analysis is that results may be&#10;affected &#10;by the nature of the cross-lingual mapping techniques. &#10;\newcite{Krasnowska.2019} probe sentence encoders in en and Polish (pl). They use tasks defined in \newcite{Conneau.2018a} but slightly modify them (e.g., replacing dependency with constituency trees), reject some tasks (Bigram-Shift, as word order may play a minor role in pl), and add two new tasks (Voice and Sentence Type). Since pl data is less abundant, &#10;they shrink the size of the pl datasets to 75k/7.5k/7.5k for train/dev/test and, for consistency, do the same for en.&#10;They extract probing datasets from an en-pl parallel corpus using COMBO for dependency parsing \cite{Rybak.2018}. &#10;They &#10;find that en and pl probing results mostly agree, i.e., encoders store the same linguistic information across the two languages. &#10; " />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Mathematics, Sentence Embeddings, Probing Task Design, Natural Language Processing, Multilingual Evaluation" />
        </attvalues>
      </node>
      <node id="2408.00675" label="2408.00675">
        <attvalues>
          <attvalue for="0" value="Leveraging Entailment Judgements in Cross-Lingual Summarisation" />
          <attvalue for="1" value="  Synthetically created Cross-Lingual Summarisation (CLS) datasets are prone to&#10;include document-summary pairs where the reference summary is unfaithful to the&#10;corresponding document as it contains content not supported by the document&#10;(i.e., hallucinated content). This low data quality misleads model learning and&#10;obscures evaluation results. Automatic ways to assess hallucinations and&#10;improve training have been proposed for monolingual summarisation,&#10;predominantly in English. For CLS, we propose to use off-the-shelf&#10;cross-lingual Natural Language Inference (X-NLI) to evaluate faithfulness of&#10;reference and model generated summaries. Then, we study training approaches&#10;that are aware of faithfulness issues in the training data and propose an&#10;approach that uses unlikelihood loss to teach a model about unfaithful summary&#10;sequences. Our results show that it is possible to train CLS models that yield&#10;more faithful summaries while maintaining comparable or better informativess.&#10;" />
          <attvalue for="2" value="&#10;&#10;A widely used method to create abstractive summarisation datasets is to crawl websites from which documents paired with &#10;reference summaries can be extracted. Examples of this are&#10;synthetic datasets created in the news \cite{grusky-etal-2018-newsroom,narayan-etal-2018-dont,scialom-etal-2020-mlsum,hasan-etal-2021-xl} and instructional domains \cite{ladhak-etal-2020-wikilingua} and for descriptive summarisation &#10;\cite{liu2018generating,perez-beltrachini-etal-2019-generating,perez-beltrachini-lapata-2021-models}.&#10;The potential content misalignment in document-summary pairs created in this way raises concerns about the quality of training and evaluation data \cite{gehrmann2022-obstacles-nlgeval}.&#10;&#10;Previous work carried out manual validation of document-summary pairs in automatically created datasets to assess content overlap thereof&#10;\cite{maynez-etal-2020-faithfulness,hasan-etal-2021-xl,perez-beltrachini-lapata-2021-models,gao-etal-2023-evaluating,chen-etal-2023-revisiting}.&#10;This inspection aims at pinpointing whether summaries convey content that cannot &#10;be inferred from the document, i.e. hallucinations. \cite{maynez-etal-2020-faithfulness} &#10;found that 70\% of the pairs in the XSum dataset \cite{narayan-etal-2018-dont} contain &#10;summaries with hallucinations.&#10;For multi- and cross-lingual summarisation,&#10;\cite{hasan-etal-2021-xl,perez-beltrachini-lapata-2021-models} found \textasciitilde30\% of the summaries to be unfaithful.&#10;&#10;Taking a causal look into the hallucinations problem in automatically generated summaries, previous work \cite{gem-sets-2021,nan-etal-2021-entity,liu-etal-2021-noisy,cao-wang-2021-cliff,goyal-durrett-2021-annotating,choubey2022cape,aharoni2022mface,qiu2023detecting} inspects whether reference summaries in a dataset contain hallucinations in an automatic way. They further exploit this information about the data quality in terms of faithfulness for training with a better signal. Their focus is on monolingual summarisation (the input document and output summary are in the same language) and the most studied language is English. In this work, we focus on cross-lingual summarisation where the document &#10;is written in one language (e.g., Czech) and the corresponding summary is written in a different language (e.g., English).&#10;&#10;We propose to leverage cross-lingual natural language inference \cite{conneau2018xnli} to supplement human validation and to automatically annotate synthetic cross-lingual datasets with hallucination judgements. &#10;Our study focuses on the XWikis corpus \cite{perez-beltrachini-lapata-2021-models} consisting of descriptive summaries.&#10;It is extracted from Wikipedia by aligning articles in different languages (e.g., Czech and English) and re-combining the lead paragraph of one article with the body of the other to form cross-lingual document-summary pairs. It includes English, French, German, and Czech and we extended it with Chinese.&#10;&#10;We study simple training schemes that are aware of hallucinations occurring in reference summaries. &#10;These train with a smaller but cleaner dataset where highly unfaithful document-summary pairs are removed, include unfaithful document-summary pairs for training but ignore the unfaithful content or explicitly teach the model about unfaithful summary sub-sequences. &#10;Our experiments show that by simply fine-tuning with a smaller but more faithful training set it is possible to improve faithfulness on generated summaries while maintaining their informativeness in the context of CLS. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cross Lingual Summarization, Artificial Intelligence, Natural Language Inference, Information Science, Natural Language Processing, Faithful Summarization Models" />
        </attvalues>
      </node>
      <node id="1804.11283" label="1804.11283">
        <attvalues>
          <attvalue for="0" value="Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive&#10;  Strategies" />
          <attvalue for="1" value="  We present NEWSROOM, a summarization dataset of 1.3 million articles and&#10;summaries written by authors and editors in newsrooms of 38 major news&#10;publications. Extracted from search and social media metadata between 1998 and&#10;2017, these high-quality summaries demonstrate high diversity of summarization&#10;styles. In particular, the summaries combine abstractive and extractive&#10;strategies, borrowing words and phrases from articles at varying rates. We&#10;analyze the extraction strategies used in NEWSROOM summaries against other&#10;datasets to quantify the diversity and difficulty of our new data, and train&#10;existing methods on the data to evaluate its utility and challenges.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14900" label="2004.14900">
        <attvalues>
          <attvalue for="0" value="MLSUM: The Multilingual Summarization Corpus" />
          <attvalue for="1" value="  We present MLSUM, the first large-scale MultiLingual SUMmarization dataset.&#10;Obtained from online newspapers, it contains 1.5M+ article/summary pairs in&#10;five different languages -- namely, French, German, Spanish, Russian, Turkish.&#10;Together with English newspapers from the popular CNN/Daily mail dataset, the&#10;collected data form a large scale multilingual dataset which can enable new&#10;research directions for the text summarization community. We report&#10;cross-lingual comparative analyses based on state-of-the-art systems. These&#10;highlight existing biases which motivate the use of a multi-lingual dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.10198" label="1801.10198">
        <attvalues>
          <attvalue for="0" value="Generating Wikipedia by Summarizing Long Sequences" />
          <attvalue for="1" value="  We show that generating English Wikipedia articles can be approached as a&#10;multi- document summarization of source documents. We use extractive&#10;summarization to coarsely identify salient information and a neural abstractive&#10;model to generate the article. For the abstractive model, we introduce a&#10;decoder-only architecture that can scalably attend to very long sequences, much&#10;longer than typical encoder- decoder architectures used in sequence&#10;transduction. We show that this model can generate fluent, coherent&#10;multi-sentence paragraphs and even whole Wikipedia articles. When given&#10;reference documents, we show it can extract relevant factual information as&#10;reflected in perplexity, ROUGE scores and human evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.04687" label="1906.04687">
        <attvalues>
          <attvalue for="0" value="Generating Summaries with Topic Templates and Structured Convolutional&#10;  Decoders" />
          <attvalue for="1" value="  Existing neural generation approaches create multi-sentence text as a single&#10;sequence. In this paper we propose a structured convolutional decoder that is&#10;guided by the content structure of target summaries. We compare our model with&#10;existing sequential decoders on three data sets representing different domains.&#10;Automatic and human evaluation demonstrate that our summaries have better&#10;content coverage.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.09583" label="2202.09583">
        <attvalues>
          <attvalue for="0" value="Models and Datasets for Cross-Lingual Summarisation" />
          <attvalue for="1" value="  We present a cross-lingual summarisation corpus with long documents in a&#10;source language associated with multi-sentence summaries in a target language.&#10;The corpus covers twelve language pairs and directions for four European&#10;languages, namely Czech, English, French and German, and the methodology for&#10;its creation can be applied to several other languages. We derive cross-lingual&#10;document-summary instances from Wikipedia by combining lead paragraphs and&#10;articles' bodies from language aligned Wikipedia titles. We analyse the&#10;proposed cross-lingual summarisation task with automatic metrics and validate&#10;it with a human study. To illustrate the utility of our dataset we report&#10;experiments with multi-lingual pre-trained models in supervised, zero- and&#10;few-shot, and out-of-domain scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.06935" label="2202.06935">
        <attvalues>
          <attvalue for="0" value="Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation&#10;  Practices for Generated Text" />
          <attvalue for="1" value="  Evaluation practices in natural language generation (NLG) have many known&#10;flaws, but improved evaluation approaches are rarely widely adopted. This issue&#10;has become more urgent, since neural NLG models have improved to the point&#10;where they can often no longer be distinguished based on the surface-level&#10;features that older metrics rely on. This paper surveys the issues with human&#10;and automatic model evaluations and with commonly used datasets in NLG that&#10;have been pointed out over the past 20 years. We summarize, categorize, and&#10;discuss how researchers have been addressing these issues and what their&#10;findings mean for the current state of model evaluations. Building on those&#10;insights, we lay out a long-term vision for NLG evaluation and propose concrete&#10;steps for researchers to improve their evaluation processes. Finally, we&#10;analyze 66 NLG papers from recent NLP conferences in how well they already&#10;follow these suggestions and identify which areas require more drastic changes&#10;to the status quo.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00661" label="2005.00661">
        <attvalues>
          <attvalue for="0" value="On Faithfulness and Factuality in Abstractive Summarization" />
          <attvalue for="1" value="  It is well known that the standard likelihood training and approximate&#10;decoding objectives in neural text generation models lead to less human-like&#10;responses for open-ended tasks such as language modeling and story generation.&#10;In this paper we have analyzed limitations of these models for abstractive&#10;document summarization and found that these models are highly prone to&#10;hallucinate content that is unfaithful to the input document. We conducted a&#10;large scale human evaluation of several neural abstractive summarization&#10;systems to better understand the types of hallucinations they produce. Our&#10;human annotators found substantial amounts of hallucinated content in all model&#10;generated summaries. However, our analysis does show that pretrained models are&#10;better summarizers not only in terms of raw metrics, i.e., ROUGE, but also in&#10;generating faithful and factual summaries as evaluated by humans. Furthermore,&#10;we show that textual entailment measures better correlate with faithfulness&#10;than standard metrics, potentially leading the way to automatic evaluation&#10;metrics as well as training and decoding criteria.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.04018" label="2307.04018">
        <attvalues>
          <attvalue for="0" value="Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New&#10;  Benchmark with Improved Annotation" />
          <attvalue for="1" value="  Most existing cross-lingual summarization (CLS) work constructs CLS corpora&#10;by simply and directly translating pre-annotated summaries from one language to&#10;another, which can contain errors from both summarization and translation&#10;processes. To address this issue, we propose ConvSumX, a cross-lingual&#10;conversation summarization benchmark, through a new annotation schema that&#10;explicitly considers source input context. ConvSumX consists of 2 sub-tasks&#10;under different real-world scenarios, with each covering 3 language directions.&#10;We conduct thorough analysis on ConvSumX and 3 widely-used manually annotated&#10;CLS corpora and empirically find that ConvSumX is more faithful towards input&#10;text. Additionally, based on the same intuition, we propose a 2-Step method,&#10;which takes both conversation and summary as input to simulate human annotation&#10;process. Experimental results show that 2-Step method surpasses strong&#10;baselines on ConvSumX under both automatic and human evaluation. Analysis shows&#10;that both source input text and summary are crucial for modeling cross-lingual&#10;summaries.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.09069" label="2106.09069">
        <attvalues>
          <attvalue for="0" value="Automatic Construction of Evaluation Suites for Natural Language&#10;  Generation Datasets" />
          <attvalue for="1" value="  Machine learning approaches applied to NLP are often evaluated by summarizing&#10;their performance in a single number, for example accuracy. Since most test&#10;sets are constructed as an i.i.d. sample from the overall data, this approach&#10;overly simplifies the complexity of language and encourages overfitting to the&#10;head of the data distribution. As such, rare language phenomena or text about&#10;underrepresented groups are not equally included in the evaluation. To&#10;encourage more in-depth model analyses, researchers have proposed the use of&#10;multiple test sets, also called challenge sets, that assess specific&#10;capabilities of a model. In this paper, we develop a framework based on this&#10;idea which is able to generate controlled perturbations and identify subsets in&#10;text-to-scalar, text-to-text, or data-to-text settings. By applying this&#10;framework to the GEM generation benchmark, we propose an evaluation suite made&#10;of 80 challenge sets, demonstrate the kinds of analyses that it enables and&#10;shed light onto the limits of current generation models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.09130" label="2102.09130">
        <attvalues>
          <attvalue for="0" value="Entity-level Factual Consistency of Abstractive Text Summarization" />
          <attvalue for="1" value="  A key challenge for abstractive summarization is ensuring factual consistency&#10;of the generated summary with respect to the original document. For example,&#10;state-of-the-art models trained on existing datasets exhibit entity&#10;hallucination, generating names of entities that are not present in the source&#10;document. We propose a set of new metrics to quantify the entity-level factual&#10;consistency of generated summaries and we show that the entity hallucination&#10;problem can be alleviated by simply filtering the training data. In addition,&#10;we propose a summary-worthy entity classification task to the training process&#10;as well as a joint entity and summary generation approach, which yield further&#10;improvements in entity level metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.07032" label="2009.07032">
        <attvalues>
          <attvalue for="0" value="Noisy Self-Knowledge Distillation for Text Summarization" />
          <attvalue for="1" value="  In this paper we apply self-knowledge distillation to text summarization&#10;which we argue can alleviate problems with maximum-likelihood training on&#10;single reference and noisy datasets. Instead of relying on one-hot annotation&#10;labels, our student summarization model is trained with guidance from a teacher&#10;which generates smoothed labels to help regularize training. Furthermore, to&#10;better model uncertainty during training, we introduce multiple noise signals&#10;for both teacher and student models. We demonstrate experimentally on three&#10;benchmarks that our framework boosts the performance of both pretrained and&#10;non-pretrained summarizers achieving state-of-the-art results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.04302" label="2104.04302">
        <attvalues>
          <attvalue for="0" value="Annotating and Modeling Fine-grained Factuality in Summarization" />
          <attvalue for="1" value="  Recent pre-trained abstractive summarization systems have started to achieve&#10;credible performance, but a major barrier to their use in practice is their&#10;propensity to output summaries that are not faithful to the input and that&#10;contain factual errors. While a number of annotated datasets and statistical&#10;models for assessing factuality have been explored, there is no clear picture&#10;of what errors are most important to target or where current techniques are&#10;succeeding and failing. We explore both synthetic and human-labeled data&#10;sources for training models to identify factual errors in summarization, and&#10;study factuality at the word-, dependency-, and sentence-level. Our&#10;observations are threefold. First, exhibited factual errors differ&#10;significantly across datasets, and commonly-used training sets of simple&#10;synthetic errors do not reflect errors made on abstractive datasets like XSum.&#10;Second, human-labeled data with fine-grained annotations provides a more&#10;effective training signal than sentence-level annotations or synthetic data.&#10;Finally, we show that our best factuality detection model enables training of&#10;more factual XSum summarization models by allowing us to identify non-factual&#10;tokens in the training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10622" label="2212.10622">
        <attvalues>
          <attvalue for="0" value="mFACE: Multilingual Summarization with Factual Consistency Evaluation" />
          <attvalue for="1" value="  Abstractive summarization has enjoyed renewed interest in recent years,&#10;thanks to pre-trained language models and the availability of large-scale&#10;datasets. Despite promising results, current models still suffer from&#10;generating factually inconsistent summaries, reducing their utility for&#10;real-world application. Several recent efforts attempt to address this by&#10;devising models that automatically detect factual inconsistencies in machine&#10;generated summaries. However, they focus exclusively on English, a language&#10;with abundant resources. In this work, we leverage factual consistency&#10;evaluation models to improve multilingual summarization. We explore two&#10;intuitive approaches to mitigate hallucinations based on the signal provided by&#10;a multilingual NLI model, namely data filtering and controlled generation.&#10;Experimental results in the 45 languages from the XLSum dataset show gains over&#10;strong baselines in both automatic and human evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13632" label="2305.13632">
        <attvalues>
          <attvalue for="0" value="Detecting and Mitigating Hallucinations in Multilingual Summarisation" />
          <attvalue for="1" value="  Hallucinations pose a significant challenge to the reliability of neural&#10;models for abstractive summarisation. While automatically generated summaries&#10;may be fluent, they often lack faithfulness to the original document. This&#10;issue becomes even more pronounced in low-resource settings, such as&#10;cross-lingual transfer. With the existing faithful metrics focusing on English,&#10;even measuring the extent of this phenomenon in cross-lingual settings is hard.&#10;To address this, we first develop a novel metric, mFACT, evaluating the&#10;faithfulness of non-English summaries, leveraging translation-based transfer&#10;from multiple English faithfulness metrics. We then propose a simple but&#10;effective method to reduce hallucinations with a cross-lingual transfer, which&#10;weighs the loss of each training example by its faithfulness score. Through&#10;extensive experiments in multiple languages, we demonstrate that mFACT is the&#10;metric that is most suited to detect hallucinations. Moreover, we find that our&#10;proposed loss weighting method drastically increases both performance and&#10;faithfulness according to both automatic and human evaluation when compared to&#10;strong baselines for cross-lingual transfer such as MAD-X. Our code and dataset&#10;are available at https://github.com/yfqiu-nlp/mfact-summ.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;While faithfulness in summarisation is a highly researched topic, previous works focused mostly on the English language \cite{pagnoni-etal-2021-understanding-factuality-with-FRANK,maynez-etal-2020-faithfulness-factuality-in-abstractive-summarization,10.1162/tacl_a_00373-summeval-fabbri}. &#10;To evaluate faithfulness, state-of-the-art methods fall into three categories. Firstly, validating faithfulness can be cast as a classification problem \cite{kryscinski-etal-2020-FactCC,goyal-durrett-2021-DAE-annotating-and-modeling-more-fine-grained-factuality,laban-etal-2022-summac}. Secondly, faithfulness can be interpreted as answerability, and assessed with existing question answering models \cite{fabbri-etal-2022-qafacteval, scialom-etal-2021-questeval}. Finally, language models may be adopted to identify extrinsic hallucinations \cite{filippova-2020-controlled-hallucination-extrinsic-eval,cao-etal-2022-hallucinated-but-factual-ENTFA-paper}. &#10;To improve faithfulness in summarisation models, an approach related to ours changes the training dynamics, e.g. by filtering out hallucinated data \cite{cao-etal-2022-learning-rejection,kang-hashimoto-2020-loss-truncation,goyal-etal-2022-training-dynamics}. The expert/anti-expert approach aims to learn experts and anti-experts that alter the behaviour of a base generative model \cite{liu-etal-2021-dexperts,choubey2021cape,ilharco2022editingmodelwithtaskarithmetricTaskVectorNegation}. Other methods include designing the specific neural architecture \cite{huang-etal-2020-knowledge-graph-aug-summ, qiu-cohen-2022-abstractive, 10.5555/3504035.3504621-cao-factaware-abs-summ}, summary ranking \cite{falke-etal-2019-ranking,liu-etal-2022-brio-bringing-order} and post-hoc correction \cite{zhu-etal-2021-enhancing-factual-consistency,dong-etal-2020-multi-fact-correction,cao-etal-2020-factual-error-correction,zhao-etal-2020-reducing}. &#10;However, there is still a limited understanding of the effectiveness of these methods in cross-lingual transfer.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2011.06057" label="2011.06057">
        <attvalues>
          <attvalue for="0" value="Exploring the Value of Personalized Word Embeddings" />
          <attvalue for="1" value="  In this paper, we introduce personalized word embeddings, and examine their&#10;value for language modeling. We compare the performance of our proposed&#10;prediction model when using personalized versus generic word representations,&#10;and study how these representations can be leveraged for improved performance.&#10;We provide insight into what types of words can be more accurately predicted&#10;when building personalized models. Our results show that a subset of words&#10;belonging to specific psycholinguistic categories tend to vary more in their&#10;representations across users and that combining generic and personalized word&#10;embeddings yields the best performance, with a 4.7% relative reduction in&#10;perplexity. Additionally, we show that a language model using personalized word&#10;embeddings can be effectively used for authorship attribution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Prior work has considered user embeddings, where one vector is learned for each user in the data (we learn a set of vectors per user, one for each word in the vocabulary).&#10;User embeddings have been used for dialog generation~\cite{li2016persona},&#10;query auto-completion~\cite{jaech-ostendorf-2018-personalized}, authorship attribution~\cite{ebrahimi2016personalized}, and sarcasm detection~\cite{kolchinski-potts-2018-representing}.&#10;\newcite{amer2016toward} learn a set of embeddings from the books that a user adds to their profile.&#10;Some approaches also use network information \cite{zeng2017socialized,huang2016enriching}.&#10;&#10;Personalization has been studied for marketing, webpage layout, recommendations, query completion, and dialog~\cite{eirinaki2003web,das2007google}.&#10;Our prior work \cite{Welch19LearningFromPersonal,Welch19LookWhosTalking} explored predicting response time, common messages, and author relationships from personal conversation data. \newcite{zhang2018personalizing} conditioned dialog systems on artificially constructed personas and \newcite{madotto-etal-2019-personalizing} used meta-learning to improve this process. Goal-oriented dialog has used demographics (i.e., age, gender) to condition system response generation, showing that this relatively coarse grained personalization improves system performance \cite{joshi2017personalization}.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Authorship Attribution, Personalized Embeddings, Linguistics, Psycholinguistics, Artificial Intelligence, Natural Language Processing, Language Modeling, Psychology" />
        </attvalues>
      </node>
      <node id="2005.00816" label="2005.00816">
        <attvalues>
          <attvalue for="0" value="DQI: Measuring Data Quality in NLP" />
          <attvalue for="1" value="  Neural language models have achieved human level performance across several&#10;NLP datasets. However, recent studies have shown that these models are not&#10;truly learning the desired task; rather, their high performance is attributed&#10;to overfitting using spurious biases, which suggests that the capabilities of&#10;AI systems have been over-estimated. We introduce a generic formula for Data&#10;Quality Index (DQI) to help dataset creators create datasets free of such&#10;unwanted biases. We evaluate this formula using a recently proposed approach&#10;for adversarial filtering, AFLite. We propose a new data creation paradigm&#10;using DQI to create higher quality data. The data creation paradigm consists of&#10;several data visualizations to help data creators (i) understand the quality of&#10;data and (ii) visualize the impact of the created data instance on the overall&#10;quality. It also has a couple of automation methods to (i) assist data creators&#10;and (ii) make the model more robust to adversarial attacks. We use DQI along&#10;with these automation methods to renovate biased examples in SNLI. We show that&#10;models trained on the renovated SNLI dataset generalize better to out of&#10;distribution tasks. Renovation results in reduced model performance, exposing a&#10;large gap with respect to human performance. DQI systematically helps in&#10;creating harder benchmarks using active learning. Our work takes the process of&#10;dynamic dataset creation forward, wherein datasets evolve together with the&#10;evolving state of the art, therefore serving as a means of benchmarking the&#10;true progress of AI.&#10;" />
          <attvalue for="2" value="&#10;Recently, a series of works \cite{gururangan2018annotation, poliak2018hypothesis, kaushik2018much, tsuchiya2018performance, tan2019investigating, schwartz2017effect, nadeem2020stereoset} has shown that many of popular datasets, such as SQUAD \cite{rajpurkar2016squad} and SNLI \cite{bowman2015large} have unwanted biases \cite{torralba2011unbiased}, resulting from the annotation process. The spurious biases represent ``unintended correlations between input and output&quot; \cite{bras2020adversarial}. Models exploit these biases as features instead of utilizing the actual underlying features needed to solve a task. Models therefore fail to generalize, and consequently, their performance drops drastically when tested with out of distribution data or adversarial examples \cite{bras2020adversarial,mccoy2019right,zhang2019paws,jia2017adversarial,jin2019bert}. These can limit Machine Learning applications to various domains because of the possibility of serious accidents. For example, ``a medical diagnosis model may consistently classify with high confidence, even while it should flag difficult examples for human intervention. The resulting unflagged, erroneous diagnoses could blockade future machine learning technologies in medicine.&quot; \cite{hendrycks2016baseline}. These biases have also led to the overestimation of AI's true advancement \cite{sakaguchi2019winogrande,bras2020adversarial}. &#10;&#10;Hence, in lieu of merely creating and solving new datasets, the Machine Learning community needs to address a core problem, i.e., how can dataset creators create datasets that are free of unwanted biases, and thus help models generalize better? This paper focuses only on NLP, but the same principles are also applicable to other areas such as Vision and Speech.&#10;&#10;There are mainly four types of approaches to address this problem (i) Dataset pruning (ii) Stopping the model from exploiting biases (iii) Adversarial dataset creation (iv) Counterfactual Data Augmentation. Each type of approach focuses on a specific part of the loop consisting of data and model, as illustrated in Figure \ref{fig:exapproaches}. &#10;&#10;AFLite \cite{sakaguchi2019winogrande}, REPAIR \cite{li2019repair}, RESOUND \cite{li2018resound} and Dataset Distillation \cite{wang2018dataset} are some of the recent works that use the first approach. AFLite filters dataset biases adversarially to attenuate the overestimation of AI systems' capabilities. On the other hand, Dataset Distillation synthesizes a minimum set of representative data to achieve close to original performance. Similarly, REPAIR resamples data to remove representation biases, and RESOUND samples existing datasets and creates a new dataset to minimize static biases. However, all these approaches do not directly impact the dataset creation process, as data pruning is only done after the data has been created by crowd workers and/or automated systems. Post-creation, data pruning is a costly operation, as resources invested in creating the initial `biased' data get wasted. Also, these approaches do not prevent a dataset creator from creating biased data in a future data creation process.&#10;&#10;The second approach has been studied in several works \cite{clark2019don}. They use a prior knowledge of biases to train a naive model that exploits dataset biases. Then this model is combined with a robust model, and the ensemble is trained. The ensemble is forced to focus on other patterns of data which are not biases. Similarly, DRiFt has been proposed \cite{he2019unlearn}, where initially a biased model is learned, which uses only bias related features. Then a debiased model is trained to fit the residual of the biased model. Another interesting work \cite{mahabadi2019simple} operates along the same lines, and has an additional lightweight bias-only model which learns dataset biases. They use its prediction to adjust the loss of the base model, to reduce the biases. Apart from the overhead involved in bias identification, the drawbacks of ``wasted resources invested in creating the initial biased data&quot; and ``not preventing dataset creators from creating biased data in future&quot; remain in this type of approach.&#10;&#10;Adversarial Filtering algorithm \cite{zellers2018swag} builds a de-biased dataset by iteratively training an ensemble of classifiers, and then utilizing them to filter data. However, this approach is model dependent and the drawbacks of the first two approaches still remain. Similarly, the Adversarial NLI dataset creation process \cite{nie2019adversarial} involves an iterative and adversarial &quot;human-and-model-in-the-loop&quot; procedure. Here, dataset creators have an additional responsibility to fool the model, and the effort required on their part increases as the rounds progress. Also, this process might create biased data itself, since it is adversarial to a specific model. Biased data is relative in nature and has significance with respect to a trained set. Since the model is not trained at every step, the adversarial dataset creation process may not produce bias free data in each and among various splits. This category of approaches might induce its own biases, as studied in a recent work \cite{liu2019inoculation} for NLI stress tests \cite{naik2018stress} and the Adversarial SQuAD dataset \cite{jia2017adversarial}.&#10;&#10;Counterfactual Data Augmentation involves asking dataset creators to create samples with counterfactual target labels. This shouldn't disturb the sample's internal coherence, nor make unnecessary changes \cite{kaushik2019learning}. Recently, a new annotation paradigm has been proposed \cite{gardner2020evaluating} where they recommend that dataset authors manually perturb the test instances in small but meaningful ways that change the gold label, creating contrast sets. However, these approaches have too much dependence on authors in identifying a list of phenomena that characterize their dataset. Thus they can lead to the formation of a different, unique set of biases for each dataset they are applied to. Also, this approach does not prevent crowd workers from creating biased data in future.&#10;&#10;Overall, existing approaches have seven types of issues: (i) resources invested in creating the initial `biased' data get wasted, (ii) a dataset creator is not prevented from creating biased data in a future data creation process, (iii) important aspects of bias like the dependence of bias on training set, train-test split are ignored, (iv) a set of additional biases is created as a byproduct, (v) the time complexity is high because of the involvement of training at each iteration, (vi) they are specific to a model or task, (vii) there is too much effort required on the part of crowd workers/authors/experts, without providing a suitable and illustrative feedback channel. We introduce a generic formula for DQI to address the first six issues, and a new data creation paradigm with several data visualizations and a couple of user-assistance methods to address the seventh one. &#10;&#10;Data Shapley \cite{ghorbani2019data} has been proposed as a metric to quantify the value of each training datum to the predictor performance. However, their approach was model dependent and task dependent. More importantly, their metric might not signify bias content, as they quantify the value of training datum based on predictor performance, and biases might favor the predictor. So, we focus on building a generic DQI with minimized dependency on models and tasks.&#10;&#10;We take inspiration from the Quality Indexes present in other domains such as power quality \cite{bollen2000understanding}, water quality \cite{world1993guidelines}, food quality \cite{grunert2005food} and air quality \cite{jones1999indoor}. We actuate and adapt those in our approach to find the formula for DQI. First, we identify the seven components which cover the space of various possible interactions between samples in an NLP dataset.&#10;We look for potential leads by going through a series of works which enumerate the various origins of dataset biases, and their impact on performance and robustness. We trace the leads to propose an empirical formula for DQI. We cover many datasets and a hierarchy of tasks ranging from NLI to Text Summarization in our analysis. This is to ensure that our formula is generic and is not overfitted towards a specific task or dataset. We evaluate this formula using AFLite, which is a recent and successful approach for light weight, model agnostic adversarial filtering.&#10;&#10;We utilize DQI to propose a new data creation paradigm which consists of several data visualizations to help data creators (i) understand the quality of data and (ii) visualize the impact of their created data instance on the overall quality. In a concurrent work \cite{wang2020vibe}, a tool for measuring and mitigating bias in Image datasets has been proposed. Our data creation paradigm also has a couple of automation methods to (i) assist data creators in rectifying their data creation process to minimize biases and (ii) make the model more robust to adversarial attacks. The automation methods consist of Textfooler \cite{jin2019bert}, a recent technique which has been successful in fooling the state-of-the-art models and Autofix, a model independent version of Textfooler which we propose using DQI. Figure \ref{fig:ourapproach} illustrates our proposed data creation paradigm.&#10;&#10;Active learning has been shown to be useful for various NLP tasks \cite{li2020active, sachan2015active, garrette2013learning, kholghi2016active}. DQI systematically helps in creating harder benchmarks using active learning. We apply DQI in an active learning setup to renovate the SNLI dataset \cite{bowman2015large} using the automation methods, and produce a series of benchmarks in an increasing hierarchy of hardness. Inspired by recent datasets \cite{sakaguchi2019winogrande} \cite{nie2019adversarial}, our work takes forward the process of dynamic dataset creation wherein datasets evolve together with the evolving state of the art, therefore serving as a means of benchmarking the true progress of AI. &#10;&#10;We also show that models trained on the renovated SNLI dataset generalize better to out of distribution tasks. Our work supports the findings of an interesting recent work \cite{bras2020adversarial} where they indicate that biases make benchmarks easier, as models learn to exploit these biases instead of learning actual features. &#10;&#10;Figure \ref{fig:overall} summarizes our work in this paper using a process flow diagram. Figures \ref{fig:step1}, \ref{fig:step2}, \ref{fig:step3}, \ref{fig:step4} and \ref{fig:step5} provide more details on each of the processes.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Language Model Bias, Artificial Intelligence, Natural Language Processing, Data Quality Assessment, Statistics, Adversarial Robustness" />
        </attvalues>
      </node>
      <node id="2002.04108" label="2002.04108">
        <attvalues>
          <attvalue for="0" value="Adversarial Filters of Dataset Biases" />
          <attvalue for="1" value="  Large neural models have demonstrated human-level performance on language and&#10;vision benchmarks, while their performance degrades considerably on adversarial&#10;or out-of-distribution samples. This raises the question of whether these&#10;models have learned to solve a dataset rather than the underlying task by&#10;overfitting to spurious dataset biases. We investigate one recently proposed&#10;approach, AFLite, which adversarially filters such dataset biases, as a means&#10;to mitigate the prevalent overestimation of machine performance. We provide a&#10;theoretical understanding for AFLite, by situating it in the generalized&#10;framework for optimum bias reduction. We present extensive supporting evidence&#10;that AFLite is broadly applicable for reduction of measurable dataset biases,&#10;and that models trained on the filtered datasets yield better generalization to&#10;out-of-distribution tasks. Finally, filtering results in a large drop in model&#10;performance (e.g., from 92% to 62% for SNLI), while human performance still&#10;remains high. Our work thus shows that such filtered datasets can pose new&#10;research challenges for robust generalization by serving as upgraded&#10;benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.02136" label="1610.02136">
        <attvalues>
          <attvalue for="0" value="A Baseline for Detecting Misclassified and Out-of-Distribution Examples&#10;  in Neural Networks" />
          <attvalue for="1" value="  We consider the two related problems of detecting if an example is&#10;misclassified or out-of-distribution. We present a simple baseline that&#10;utilizes probabilities from softmax distributions. Correctly classified&#10;examples tend to have greater maximum softmax probabilities than erroneously&#10;classified and out-of-distribution examples, allowing for their detection. We&#10;assess performance by defining several tasks in computer vision, natural&#10;language processing, and automatic speech recognition, showing the&#10;effectiveness of this baseline across all. We then show the baseline can&#10;sometimes be surpassed, demonstrating the room for future research on these&#10;underexplored detection tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.07911" label="1904.07911">
        <attvalues>
          <attvalue for="0" value="REPAIR: Removing Representation Bias by Dataset Resampling" />
          <attvalue for="1" value="  Modern machine learning datasets can have biases for certain representations&#10;that are leveraged by algorithms to achieve high performance without learning&#10;to solve the underlying task. This problem is referred to as &quot;representation&#10;bias&quot;. The question of how to reduce the representation biases of a dataset is&#10;investigated and a new dataset REPresentAtion bIas Removal (REPAIR) procedure&#10;is proposed. This formulates bias minimization as an optimization problem,&#10;seeking a weight distribution that penalizes examples easy for a classifier&#10;built on a given feature representation. Bias reduction is then equated to&#10;maximizing the ratio between the classification loss on the reweighted dataset&#10;and the uncertainty of the ground-truth class labels. This is a minimax problem&#10;that REPAIR solves by alternatingly updating classifier parameters and dataset&#10;resampling weights, using stochastic gradient descent. An experimental set-up&#10;is also introduced to measure the bias of any dataset for a given&#10;representation, and the impact of this bias on the performance of recognition&#10;models. Experiments with synthetic and action recognition data show that&#10;dataset REPAIR can significantly reduce representation bias, and lead to&#10;improved generalization of models trained on REPAIRed datasets. The tools used&#10;for characterizing representation bias, and the proposed dataset REPAIR&#10;algorithm, are available at https://github.com/JerryYLi/Dataset-REPAIR/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.10959" label="1811.10959">
        <attvalues>
          <attvalue for="0" value="Dataset Distillation" />
          <attvalue for="1" value="  Model distillation aims to distill the knowledge of a complex model into a&#10;simpler one. In this paper, we consider an alternative formulation called&#10;dataset distillation: we keep the model fixed and instead attempt to distill&#10;the knowledge from a large training dataset into a small one. The idea is to&#10;synthesize a small number of data points that do not need to come from the&#10;correct data distribution, but will, when given to the learning algorithm as&#10;training data, approximate the model trained on the original data. For example,&#10;we show that it is possible to compress 60,000 MNIST training images into just&#10;10 synthetic distilled images (one per class) and achieve close to original&#10;performance with only a few gradient descent steps, given a fixed network&#10;initialization. We evaluate our method in various initialization settings and&#10;with different learning objectives. Experiments on multiple datasets show the&#10;advantage of our approach compared to alternative methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.02668" label="1904.02668">
        <attvalues>
          <attvalue for="0" value="Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets" />
          <attvalue for="1" value="  Several datasets have recently been constructed to expose brittleness in&#10;models trained on existing benchmarks. While model performance on these&#10;challenge datasets is significantly lower compared to the original benchmark,&#10;it is unclear what particular weaknesses they reveal. For example, a challenge&#10;dataset may be difficult because it targets phenomena that current models&#10;cannot capture, or because it simply exploits blind spots in a model's specific&#10;training set. We introduce inoculation by fine-tuning, a new analysis method&#10;for studying challenge datasets by exposing models (the metaphorical patient)&#10;to a small amount of data from the challenge dataset (a metaphorical pathogen)&#10;and assessing how well they can adapt. We apply our method to analyze the NLI&#10;&quot;stress tests&quot; (Naik et al., 2018) and the Adversarial SQuAD dataset (Jia and&#10;Liang, 2017). We show that after slight exposure, some of these datasets are no&#10;longer challenging, while others remain difficult. Our results indicate that&#10;failures on challenge datasets may lead to very different conclusions about&#10;models, training datasets, and the challenge datasets themselves.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.00692" label="1806.00692">
        <attvalues>
          <attvalue for="0" value="Stress Test Evaluation for Natural Language Inference" />
          <attvalue for="1" value="  Natural language inference (NLI) is the task of determining if a natural&#10;language hypothesis can be inferred from a given premise in a justifiable&#10;manner. NLI was proposed as a benchmark task for natural language&#10;understanding. Existing models perform well at standard datasets for NLI,&#10;achieving impressive results across different genres of text. However, the&#10;extent to which these models understand the semantic content of sentences is&#10;unclear. In this work, we propose an evaluation methodology consisting of&#10;automatically constructed &quot;stress tests&quot; that allow us to examine whether&#10;systems have the ability to make real inferential decisions. Our evaluation of&#10;six sentence-encoder models on these stress tests reveals strengths and&#10;weaknesses of these models with respect to challenging linguistic phenomena,&#10;and suggests important directions for future work in this area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.12434" label="1909.12434">
        <attvalues>
          <attvalue for="0" value="Learning the Difference that Makes a Difference with&#10;  Counterfactually-Augmented Data" />
          <attvalue for="1" value="  Despite alarm over the reliance of machine learning systems on so-called&#10;spurious patterns, the term lacks coherent meaning in standard statistical&#10;frameworks. However, the language of causality offers clarity: spurious&#10;associations are due to confounding (e.g., a common cause), but not direct or&#10;indirect causal effects. In this paper, we focus on natural language&#10;processing, introducing methods and resources for training models less&#10;sensitive to spurious patterns. Given documents and their initial labels, we&#10;task humans with revising each document so that it (i) accords with a&#10;counterfactual target label; (ii) retains internal coherence; and (iii) avoids&#10;unnecessary changes. Interestingly, on sentiment analysis and natural language&#10;inference tasks, classifiers trained on original data fail on their&#10;counterfactually-revised counterparts and vice versa. Classifiers trained on&#10;combined datasets perform remarkably well, just shy of those specialized to&#10;either domain. While classifiers trained on either original or manipulated data&#10;alone are sensitive to spurious features (e.g., mentions of genre), models&#10;trained on the combined data are less sensitive to this signal. Both datasets&#10;are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.06699" label="2406.06699">
        <attvalues>
          <attvalue for="0" value="In-Context Learning and Fine-Tuning GPT for Argument Mining" />
          <attvalue for="1" value="  Large Language Models (LLMs) have become ubiquitous in NLP and deep learning.&#10;In-Context Learning (ICL) has been suggested as a bridging paradigm between the&#10;training-free and fine-tuning LLMs settings. In ICL, an LLM is conditioned to&#10;solve tasks by means of a few solved demonstration examples included as prompt.&#10;Argument Mining (AM) aims to extract the complex argumentative structure of a&#10;text, and Argument Type Classification (ATC) is an essential sub-task of AM. We&#10;introduce an ICL strategy for ATC combining kNN-based examples selection and&#10;majority vote ensembling. In the training-free ICL setting, we show that GPT-4&#10;is able to leverage relevant information from only a few demonstration examples&#10;and achieve very competitive classification accuracy on ATC. We further set up&#10;a fine-tuning strategy incorporating well-crafted structural features given&#10;directly in textual form. In this setting, GPT-3.5 achieves state-of-the-art&#10;performance on ATC. Overall, these results emphasize the emergent ability of&#10;LLMs to grasp global discursive flow in raw text in both off-the-shelf and&#10;fine-tuned setups.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models (LLMs) have become ubiquitous in Deep Learning and have shown impressive capabilities in most NLP tasks~\cite{ZhaoEtAl23}. In the main, LLMs are used in two distinct settings: (i) training-free, where the pre-trained LLM is used for inference without any parameter adjustment, and (ii) fine-tuning, where the parameters of the LLM are updated through supervised training to enable transfer learning on a downstream task.&#10;&#10;Recently, In-Context Learning (ICL) has been suggested as a bridging pa\-ra\-digm between the training-free and fine-tuning settings. ICL is a prompt engineering technique whereby an LLM is conditioned to solve tasks by means of a few solved demonstration examples included as part of its input prompt~\cite{DongEtAl23}. Generally, the input prompt includes task instructions, the current input sample to be solved as well as several solved input-output pair examples. In this way, ICL maintains the training-free posture (parameters frozen) of the LLM while at the same time providing it with some supervision through demonstration examples. It also enables direct incorporation of selected features inside the prompt template, thereby obviating the need for architecture customization. Creative ICL strategies combining $k$NN-based examples selection, generated chain-of-thought (CoT) prompting, and majority vote ensembling have been proposed and shown to outperform fine-tuning approaches~\cite{Microsoft23,CoTWeiEtAl2023,LeiEtAl23,WangSelfConsistency23}. In the main, $k$NN-based examples selection optimizes the process of learning from few examples and ensembling increases the robustness of the predictions~\cite{CapabilitiesNori23,Microsoft23,LeiEtAl23}.&#10;&#10;Argument Mining (AM) focuses on extracting the underlying argumentative and discursive structure from raw text~\cite{AMPalauMoens09}. Argument Type Classification (ATC), which involves classifying argumentative units in text according to their argumentative roles, is the crucial sub-task in AM. Research has shown that the argumentative role of a unit cannot be inferred solely for its text: additional structural and contextual information is needed~\cite{ParsingStabGurevych17}. This additional information can be incorporated via feature engineering~\cite{ParsingStabGurevych17}, memory-enabled neural architectures~\cite{PotashEtAl17,SpanRepKuribayashiEtAl19} or LLM-based hybrid methods~\cite{BERTFeaTxtMushtaqCabessa22,BERTMinusMushtaqCabessa23}.&#10;&#10;We introduce an ICL strategy for ATC combining $k$NN-based examples selection and majority vote ensembling. We experimented with various prompting templates in order to reveal the proper contribution of different contextual elements. In this training-free setting, we show that GPT-4~\cite{GPT42023} is able to leverage relevant information from only a few demonstration examples in order to achieve very competitive classification accuracy. We further set up a fine-tuning strategy incorporating well-crafted structural features input directly in textual form. In this setting, GPT-3.5 achieves state-of-the-art performance on the ATC task. Overall, these results emphasize the emergent ability of LLMs to grasp global discursive flow in raw text in both off-the-shelf and fine-tuned setups~\cite{wei2022emergent}. The code is available on GitHub \href{&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Argument Mining Techniques, Deep Learning, Artificial Intelligence, Language Model Development, Natural Language Processing, Deep Learning Strategies" />
        </attvalues>
      </node>
      <node id="2301.00234" label="2301.00234">
        <attvalues>
          <attvalue for="0" value="A Survey on In-context Learning" />
          <attvalue for="1" value="  With the increasing capabilities of large language models (LLMs), in-context&#10;learning (ICL) has emerged as a new paradigm for natural language processing&#10;(NLP), where LLMs make predictions based on contexts augmented with a few&#10;examples. It has been a significant trend to explore ICL to evaluate and&#10;extrapolate the ability of LLMs. In this paper, we aim to survey and summarize&#10;the progress and challenges of ICL. We first present a formal definition of ICL&#10;and clarify its correlation to related studies. Then, we organize and discuss&#10;advanced techniques, including training strategies, prompt designing&#10;strategies, and related analysis. Additionally, we explore various ICL&#10;application scenarios, such as data engineering and knowledge updating.&#10;Finally, we address the challenges of ICL and suggest potential directions for&#10;further research. We hope that our work can encourage more research on&#10;uncovering how ICL works and improving ICL.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the scaling of model size and data size~\cite{gpt3, chowdhery2022palm, openai:2023gpt4, Hugo:2023llama, Hugo:2023llama2}, large language models (LLMs) demonstrate the in-context learning (ICL) ability, that is, learning from a few examples in the context. &#10;Many studies have shown that LLMs can perform a series of complex tasks through ICL, such as solving mathematical reasoning problems~\cite{cot}. These strong abilities have been widely verified as emerging abilities for large language models~\cite{wei2022emergent}. &#10;&#10;The key idea of in-context learning is to learn from analogy. Figure~\ref{fig:icl} gives an example that describes how language models make decisions via ICL.&#10;First, ICL requires a few demonstration examples to form a prompt context. These examples are usually written in natural language templates. &#10;Then, ICL concatenates a query question and the piece of prompt context together to form the input, which is then fed into the language model for prediction.&#10;Different from supervised learning, which requires a training stage that uses backward gradients to update model parameters, ICL does not perform parameter updates. The model is expected to learn the pattern hidden in the demonstration and accordingly make the right prediction. &#10;&#10;\tikzstyle{my-box}=[&#10; rectangle,&#10; draw=hidden-draw,&#10; rounded corners,&#10; text opacity=1,&#10; minimum height=1.5em,&#10; minimum width=5em,&#10; inner sep=2pt,&#10; align=center,&#10; fill opacity=.5,&#10; line width=0.8pt,&#10;]&#10;\tikzstyle{leaf}=[my-box, minimum height=1.5em,&#10; fill=hidden-pink!80, text=black, align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;&#10;As a new paradigm, ICL has multiple attractive advantages. First, since the demonstration is written in natural language, it provides an interpretable interface to communicate with LLMs~\cite{gpt3}.&#10;This paradigm makes it much easier to incorporate human knowledge into LLMs by changing the demonstration and templates~\cite{liu2022close, lu2022order, cot, Wu2022SelfadaptiveIL}. &#10;Second, in-context learning is similar to the decision process of human beings by learning from analogy~\cite{winston1980learningByAnalogy}. Third, compared to supervised training, ICL is a training-free learning framework. This could not only greatly reduce the computational costs for adapting the model to new tasks, but also make language-model-as-a-service~\cite{sun2022black} possible and can be easily applied to large-scale real-world tasks.&#10;&#10;Despite being promising, there are also interesting questions and intriguing properties that require further investigation in ICL. &#10;Although a range of vanilla GPT models show excellent ICL capability, several studies have found that this capability can be significantly improved through adaptation during pretraining~\cite{metaicl, Li2023mend}.&#10;Moreover, the performance of ICL is sensitive to specific settings, including the prompt template, the selection and order of demonstration examples, and other factors~\cite{topic, liu2024lets}. Additionally, optimizing the conciseness of demonstration examples and improving the computational efficiency of ICL are critical areas of ongoing research~\cite{liu2024incontext}. &#10;Furthermore, despite preliminary explanations~\cite{dai2022iclft, jiang2023latent}, the underlying working mechanism of ICL remains unclear and requires further investigation.&#10;&#10;With the rapid growth of studies in ICL, our survey aims to sensitize the community toward the current progress.&#10;In the following sections, we delve into an in-depth discussion of related studies, and we summarize the taxonomy in Figure~\ref{taxo_of_icl} and the key findings in Appendix~\ref{app:takeaway}. &#10;We highlight the challenges and potential directions and hope our work provide a useful roadmap for beginners interested in this area and shed light on future research.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.13375" label="2303.13375">
        <attvalues>
          <attvalue for="0" value="Capabilities of GPT-4 on Medical Challenge Problems" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated remarkable capabilities in&#10;natural language understanding and generation across various domains, including&#10;medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art&#10;LLM, on medical competency examinations and benchmark datasets. GPT-4 is a&#10;general-purpose model that is not specialized for medical problems through&#10;training or engineered to solve clinical tasks. Our analysis covers two sets of&#10;official practice materials for the USMLE, a three-step examination program&#10;used to assess clinical competency and grant licensure in the United States. We&#10;also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond&#10;measuring model performance, experiments were conducted to investigate the&#10;influence of test questions containing both text and images on model&#10;performance, probe for memorization of content during training, and study&#10;probability calibration, which is of critical importance in high-stakes&#10;applications like medicine. Our results show that GPT-4, without any&#10;specialized prompt crafting, exceeds the passing score on USMLE by over 20&#10;points and outperforms earlier general-purpose models (GPT-3.5) as well as&#10;models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned&#10;version of Flan-PaLM 540B). In addition, GPT-4 is significantly better&#10;calibrated than GPT-3.5, demonstrating a much-improved ability to predict the&#10;likelihood that its answers are correct. We also explore the behavior of the&#10;model qualitatively through a case study that shows the ability of GPT-4 to&#10;explain medical reasoning, personalize explanations to students, and&#10;interactively craft new counterfactual scenarios around a medical case.&#10;Implications of the findings are discussed for potential uses of GPT-4 in&#10;medical education, assessment, and clinical practice, with appropriate&#10;attention to challenges of accuracy and safety.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.01700" label="2107.01700">
        <attvalues>
          <attvalue for="0" value="End-to-end Neural Coreference Resolution Revisited: A Simple yet&#10;  Effective Baseline" />
          <attvalue for="1" value="  Since the first end-to-end neural coreference resolution model was&#10;introduced, many extensions to the model have been proposed, ranging from using&#10;higher-order inference to directly optimizing evaluation metrics using&#10;reinforcement learning. Despite improving the coreference resolution&#10;performance by a large margin, these extensions add substantial extra&#10;complexity to the original model. Motivated by this observation and the recent&#10;advances in pre-trained Transformer language models, we propose a simple yet&#10;effective baseline for coreference resolution. Even though our model is a&#10;simplified version of the original neural coreference resolution model, it&#10;achieves impressive performance, outperforming all recent extended works on the&#10;public English OntoNotes benchmark. Our work provides evidence for the&#10;necessity of carefully justifying the complexity of existing or newly proposed&#10;models, as introducing a conceptual or practical simplification to an existing&#10;model can still yield competitive results.&#10;" />
          <attvalue for="2" value="&#10;Coreference resolution is the task of clustering mentions in text that refer to the same entities \cite{ng2010review} (Figure \ref{fig:coref_example}). As a fundamental task of natural language processing, coreference resolution can be an essential component for many downstream applications. Many traditional coreference resolution systems are pipelined systems, each consists of two separate components: (1) a mention detector for identifying entity mentions from text (2) a coreference resolver for clustering the extracted mentions \cite{raghunathanEtal2010multi,durrettklein2013easy,clarkmanning2015entity,wisemanetal2016learning,clarkmanning2016deep}. These models typically rely heavily on syntatic parsers and use highly engineered mention proposal algorithms.&#10;&#10;In 2017, the first end-to-end coreference resolution model named e2e-coref was proposed \cite{leeetal2017end}. It outperforms previous pipelined systems without using any syntactic parser or complicated hand-engineered features. Since then, many extensions to the e2e-coref model have been introduced, ranging from using higher-order inference to directly optimizing evaluation metrics using reinforcement learning \cite{zhangetal2018neuralcoreference,leeetal2018higher,Gu2018ASO,feietal2019end,kantorgloberson2019coreference,joshietal2019bert,ijcai20190700,joshi2020spanbert} (Figure \ref{fig:coref_timeline}). Despite improving the coreference resolution performance by a large margin, these extensions add a lot of extra complexity to the original model. Motivated by this observation and the recent advances in pre-trained Transformer language models, we propose a simple yet effective baseline for coreference resolution. We introduce simplifications to the original e2e-coref model, creating a conceptually simpler model for coreference resolution. Despite its simplicity, our model outperforms all aforementioned methods on the public English OntoNotes benchmark. Our work provides evidence for the necessity of carefully justifying the complexity of existing or newly proposed models, as introducing a conceptual or practical simplification to an existing model can still yield competitive results. The findings of our work agree with the results of several recent studies \cite{xuchoi2020revealing,Kirstain2021CoreferenceRW,lai2021bert}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Neural Network Models, Coreference Resolution, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1609.08667" label="1609.08667">
        <attvalues>
          <attvalue for="0" value="Deep Reinforcement Learning for Mention-Ranking Coreference Models" />
          <attvalue for="1" value="  Coreference resolution systems are typically trained with heuristic loss&#10;functions that require careful tuning. In this paper we instead apply&#10;reinforcement learning to directly optimize a neural mention-ranking model for&#10;coreference evaluation metrics. We experiment with two approaches: the&#10;REINFORCE policy gradient algorithm and a reward-rescaled max-margin objective.&#10;We find the latter to be more effective, resulting in significant improvements&#10;over the current state-of-the-art on the English and Chinese portions of the&#10;CoNLL 2012 Shared Task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.05392" label="1804.05392">
        <attvalues>
          <attvalue for="0" value="Higher-order Coreference Resolution with Coarse-to-fine Inference" />
          <attvalue for="1" value="  We introduce a fully differentiable approximation to higher-order inference&#10;for coreference resolution. Our approach uses the antecedent distribution from&#10;a span-ranking architecture as an attention mechanism to iteratively refine&#10;span representations. This enables the model to softly consider multiple hops&#10;in the predicted clusters. To alleviate the computational cost of this&#10;iterative process, we introduce a coarse-to-fine approach that incorporates a&#10;less accurate but more efficient bilinear factor, enabling more aggressive&#10;pruning without hurting accuracy. Compared to the existing state-of-the-art&#10;span-ranking approach, our model significantly improves accuracy on the English&#10;OntoNotes benchmark, while being far more computationally efficient.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.09091" label="1908.09091">
        <attvalues>
          <attvalue for="0" value="BERT for Coreference Resolution: Baselines and Analysis" />
          <attvalue for="1" value="  We apply BERT to coreference resolution, achieving strong improvements on the&#10;OntoNotes (+3.9 F1) and GAP (+11.5 F1) benchmarks. A qualitative analysis of&#10;model predictions indicates that, compared to ELMo and BERT-base, BERT-large is&#10;particularly better at distinguishing between related but distinct entities&#10;(e.g., President and CEO). However, there is still room for improvement in&#10;modeling document-level context, conversations, and mention paraphrasing. Our&#10;code and models are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.13305" label="2004.13305">
        <attvalues>
          <attvalue for="0" value="Weakly Supervised POS Taggers Perform Poorly on Truly Low-Resource&#10;  Languages" />
          <attvalue for="1" value="  Part-of-speech (POS) taggers for low-resource languages which are exclusively&#10;based on various forms of weak supervision - e.g., cross-lingual transfer,&#10;type-level supervision, or a combination thereof - have been reported to&#10;perform almost as well as supervised ones. However, weakly supervised POS&#10;taggers are commonly only evaluated on languages that are very different from&#10;truly low-resource languages, and the taggers use sources of information, like&#10;high-coverage and almost error-free dictionaries, which are likely not&#10;available for resource-poor languages. We train and evaluate state-of-the-art&#10;weakly supervised POS taggers for a typologically diverse set of 15 truly&#10;low-resource languages. On these languages, given a realistic amount of&#10;resources, even our best model gets only less than half of the words right. Our&#10;results highlight the need for new and different approaches to POS tagging for&#10;truly low-resource languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.12011" label="2402.12011">
        <attvalues>
          <attvalue for="0" value="A Systematic Comparison of Contextualized Word Embeddings for Lexical&#10;  Semantic Change" />
          <attvalue for="1" value="  Contextualized embeddings are the preferred tool for modeling Lexical&#10;Semantic Change (LSC). Current evaluations typically focus on a specific task&#10;known as Graded Change Detection (GCD). However, performance comparison across&#10;work are often misleading due to their reliance on diverse settings. In this&#10;paper, we evaluate state-of-the-art models and approaches for GCD under equal&#10;conditions. We further break the LSC problem into Word-in-Context (WiC) and&#10;Word Sense Induction (WSI) tasks, and compare models across these different&#10;levels. Our evaluation is performed across different languages on eight&#10;available benchmarks for LSC, and shows that (i) APD outperforms other&#10;approaches for GCD; (ii) XL-LEXEME outperforms other contextualized models for&#10;WiC, WSI, and GCD, while being comparable to GPT-4; (iii) there is a clear need&#10;for improving the modeling of word meanings, as well as focus on how, when, and&#10;why these meanings change, rather than solely focusing on the extent of&#10;semantic change.&#10;" />
          <attvalue for="2" value="&#10;Lexical Semantic Change (LSC) is the problem of automatically identifying words that change their meaning over time~\cite{montanelli2023survey,tahmasebi2021survey,kutuzov2018diachronic,tang2018state}. The interest in this problem has been significantly fueled by the advent of word embeddings and modern language models. After more than a decade of ad hoc evaluation, a new evaluation framework was recently introduced, aimed at assessing and comparing the performance of different models and approaches~\cite{schlechtweg2020semeval}. This framework was adopted to create benchmarks in different languages. Each benchmark includes a diachronic corpus spanning two time periods, along with a list of target words and tasks aimed at detecting word meaning change over time. The most popular task, known as Graded Change Detection (GCD), consists of ranking a list of target words based on their degree of change.&#10;&#10;The initial excitement for word embeddings prompted researchers and practitioners to solve the GCD task by using static embedding models~\cite{schlechtweg2020semeval,shoemark2019room}. However, the shift towards more advanced Transformer architectures has established the use of contextualized embedding models as the preferred tool for addressing GCD~\cite{montanelli2023survey,kutuzov2022contextualized}. On one hand, these models distinguish the different meanings of a word by contextualizing each occurrence with a different embedding. On the other hand, the generation and processing of contextualized embeddings across entire corpora pose scalability challenges, both in terms of time and memory consumption~\cite{periti2022what,montariol2021scalable}. Different strategies have been adopted to tackle these challenges, leading to a proliferation of evaluations across diverse settings~(e.g., limited samples of benchmarks) and conditions~(e.g., pre-trained vs. fine-tuned models). As a result, these evaluations on GCD hinder a fair comparison among the performance of different models and approaches, thereby deviating from the original goal of the framework.&#10;&#10;Moreover, while the GCD task is attracting more and more evaluations, it addresses only a partial complexity inherent to the established framework. Notably, the framework includes three distinct aspects~\cite{schlechtweg2021dwug}: &#10;\begin{itemize}&#10; \item [i)] semantic proximity judgments of word in-context,&#10; \item [ii)] word sense induction based on proximity judgments,&#10; \item [iii)] quantification of semantic change from induced senses.&#10;\end{itemize} &#10;As a matter of fact, when contextualized embedding models are used to address GCD, cosine similarities among word embeddings serve as surrogate for (i), without evaluation focused on this aspect. Additionally, most approaches to GCD, pass from (i) to (iii), sidestepping the intermediate aspect (ii). That is, they quantify semantic change as overall proximity variation, without inducing word senses. Consequently, while these approaches can be evaluated through GDC, they preclude the interpretation of which meaning(s) have changed.&#10;&#10;We argue that (i) and (ii) are equally relevant aspects as (iii), constituting a fundamental aspect of the LSC problem. Their evaluation can provide valuable insights into the current state of LSC modeling, while offering a broader perspective on contextualized embedding models in Natural Language Processing (NLP)..&#10;&#10;Original contribution of our work&#10;\begin{compactitem}&#10;\item We systematically evaluate and compare various models and approaches for GCD under equal settings and conditions. Our evaluation for GCD spans eight different languages. Importantly, we perform the first evaluation over Chinese and the second evaluation for Norwegian within the existing literature. Our results show superior performance of the recent state-of-the-art model for GCD, namely XL-LEXEME, over various approaches. &#10;\item We are the first to evaluate contextualized embedding models for (i) and (ii) within the existing literature. Our evaluation of (i) and (ii) relies on two well-known tasks in NLP, namely Word-in-Context (WiC), and Word Sense Induction (WSI). Importantly, we evaluate various models as computational annotators. &#10;\item We compare GPT-4 to contextualized models through the WiC, WSI, and GCD tasks. Our evaluation reveals that GPT-4 obtains comparable performance to XL-LEXEME. In contrast to the limited accessibility and high associated cost of GPT-4, XL-LEXEME is a considerably smaller, open-source model. Thus, we argue that the use of GPT-4 is not justified for modeling the LSC problem. &#10;\end{compactitem}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Modeling Evaluation, Contextualized Embeddings, Artificial Intelligence, Natural Language Processing, Lexical Semantic Change" />
        </attvalues>
      </node>
      <node id="2304.01666" label="2304.01666">
        <attvalues>
          <attvalue for="0" value="A Survey on Contextualised Semantic Shift Detection" />
          <attvalue for="1" value="  Semantic Shift Detection (SSD) is the task of identifying, interpreting, and&#10;assessing the possible change over time in the meanings of a target word.&#10;Traditionally, SSD has been addressed by linguists and social scientists&#10;through manual and time-consuming activities. In the recent years,&#10;computational approaches based on Natural Language Processing and word&#10;embeddings gained increasing attention to automate SSD as much as possible. In&#10;particular, over the past three years, significant advancements have been made&#10;almost exclusively based on word contextualised embedding models, which can&#10;handle the multiple usages/meanings of the words and better capture the related&#10;semantic shifts. In this paper, we survey the approaches based on&#10;contextualised embeddings for SSD (i.e., CSSDetection) and we propose a&#10;classification framework characterised by meaning representation,&#10;time-awareness, and learning modality dimensions. The framework is exploited i)&#10;to review the measures for shift assessment, ii) to compare the approaches on&#10;performance, and iii) to discuss the current issues in terms of scalability,&#10;interpretability, and robustness. Open challenges and future research&#10;directions about CSSDetection are finally outlined.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.03537" label="1806.03537">
        <attvalues>
          <attvalue for="0" value="Diachronic word embeddings and semantic shifts: a survey" />
          <attvalue for="1" value="  Recent years have witnessed a surge of publications aimed at tracing temporal&#10;changes in lexical semantics using distributional methods, particularly&#10;prediction-based word embedding models. However, this vein of research lacks&#10;the cohesion, common terminology and shared practices of more established areas&#10;of natural language processing. In this paper, we survey the current state of&#10;academic research related to diachronic word embeddings and semantic shifts&#10;detection. We start with discussing the notion of semantic shifts, and then&#10;continue with an overview of the existing methods for tracing such time-related&#10;shifts with word embedding models. We propose several axes along which these&#10;methods can be compared, and outline the main challenges before this emerging&#10;subfield of NLP, as well as prospects and possible applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.09872" label="1801.09872">
        <attvalues>
          <attvalue for="0" value="A State-of-the-Art of Semantic Change Computation" />
          <attvalue for="1" value="  This paper reviews the state-of-the-art of semantic change computation, one&#10;emerging research field in computational linguistics, proposing a framework&#10;that summarizes the literature by identifying and expounding five essential&#10;components in the field: diachronic corpus, diachronic word sense&#10;characterization, change modelling, evaluation data and data visualization.&#10;Despite the potential of the field, the review shows that current studies are&#10;mainly focused on testifying hypotheses proposed in theoretical linguistics and&#10;that several core issues remain to be solved: the need for diachronic corpora&#10;of languages other than English, the need for comprehensive evaluation data for&#10;evaluation, the comparison and construction of approaches to diachronic word&#10;sense characterization and change modelling, and further exploration of data&#10;visualization techniques for hypothesis justification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.11464" label="2007.11464">
        <attvalues>
          <attvalue for="0" value="SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection" />
          <attvalue for="1" value="  Lexical Semantic Change detection, i.e., the task of identifying words that&#10;change meaning over time, is a very active research area, with applications in&#10;NLP, lexicography, and linguistics. Evaluation is currently the most pressing&#10;problem in Lexical Semantic Change detection, as no gold standards are&#10;available to the community, which hinders progress. We present the results of&#10;the first shared task that addresses this gap by providing researchers with an&#10;evaluation framework and manually annotated, high-quality datasets for English,&#10;German, Latin, and Swedish. 33 teams submitted 186 systems, which were&#10;evaluated on two subtasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08540" label="2104.08540">
        <attvalues>
          <attvalue for="0" value="DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languages" />
          <attvalue for="1" value="  Word meaning is notoriously difficult to capture, both synchronically and&#10;diachronically. In this paper, we describe the creation of the largest resource&#10;of graded contextualized, diachronic word meaning annotation in four different&#10;languages, based on 100,000 human semantic proximity judgments. We thoroughly&#10;describe the multi-round incremental annotation process, the choice for a&#10;clustering algorithm to group usages into senses, and possible - diachronic and&#10;synchronic - uses for this dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.11520" label="2003.11520">
        <attvalues>
          <attvalue for="0" value="Joint Multiclass Debiasing of Word Embeddings" />
          <attvalue for="1" value="  Bias in Word Embeddings has been a subject of recent interest, along with&#10;efforts for its reduction. Current approaches show promising progress towards&#10;debiasing single bias dimensions such as gender or race. In this paper, we&#10;present a joint multiclass debiasing approach that is capable of debiasing&#10;multiple bias dimensions simultaneously. In that direction, we present two&#10;approaches, HardWEAT and SoftWEAT, that aim to reduce biases by minimizing the&#10;scores of the Word Embeddings Association Test (WEAT). We demonstrate the&#10;viability of our methods by debiasing Word Embeddings on three classes of&#10;biases (religion, gender and race) in three different publicly available word&#10;embeddings and show that our concepts can both reduce or even completely&#10;eliminate bias, while maintaining meaningful relationships between vectors in&#10;word embeddings. Our work strengthens the foundation for more unbiased neural&#10;representations of textual data.&#10;" />
          <attvalue for="2" value="&#10;Word Embeddings, i.e., the vector representation of natural language words, are key components of many state-of-the art algorithms for a variety Natural Language Processing tasks, such as Sentiment Analysis or Part of Speech Tagging. &#10;Recent research established that popular embeddings are prone to substantial biases, e.g., with respect to gender or race ~\cite{DBLP:journals/corr/abs-1711-08412,bolukbasi2016}, which demonstrated in results like ``Man is to Computer Programmer as Woman is to&#10;Homemaker&quot;~\cite{bolukbasi2016} as results of basic analogy tasks.&#10;Since such biases can potentially have an effect on downstream tasks, several relevant approaches for debiasing existing word embedding have been developed.&#10;A common deficit of existing techniques is that debiasing is limited to a single bias dimension (such as gender). Thus, in this paper, we propose two new post-processing methods for joint/simultaneous multiclass debiasing, which differ in their trade-off between maintaining word relationships and decreasing bias levels: HardWEAT completely eliminates contained bias as measured by the established Word Embedding Association Test \cite{caliskan2017}. SoftWEAT has a stronger and tunable emphasis on maintaining the original relationships between words in addition to bias removal. We demonstrate the effectiveness of our approach on the bias dimensions gender, race and religion on three conventional Word Embedding models: FastText, GloVe and Word2Vec. &#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Bias Reduction, Computer Science, Machine Learning, Linguistics, Word Embeddings, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1607.06520" label="1607.06520">
        <attvalues>
          <attvalue for="0" value="Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word&#10;  Embeddings" />
          <attvalue for="1" value="  The blind application of machine learning runs the risk of amplifying biases&#10;present in data. Such a danger is facing us with word embedding, a popular&#10;framework to represent text data as vectors which has been used in many machine&#10;learning and natural language processing tasks. We show that even word&#10;embeddings trained on Google News articles exhibit female/male gender&#10;stereotypes to a disturbing extent. This raises concerns because their&#10;widespread use, as we describe, often tends to amplify these biases.&#10;Geometrically, gender bias is first shown to be captured by a direction in the&#10;word embedding. Second, gender neutral words are shown to be linearly separable&#10;from gender definition words in the word embedding. Using these properties, we&#10;provide a methodology for modifying an embedding to remove gender stereotypes,&#10;such as the association between between the words receptionist and female,&#10;while maintaining desired associations such as between the words queen and&#10;female. We define metrics to quantify both direct and indirect gender biases in&#10;embeddings, and develop algorithms to &quot;debias&quot; the embedding. Using&#10;crowd-worker evaluation as well as standard benchmarks, we empirically&#10;demonstrate that our algorithms significantly reduce gender bias in embeddings&#10;while preserving the its useful properties such as the ability to cluster&#10;related concepts and to solve analogy tasks. The resulting embeddings can be&#10;used in applications without amplifying gender bias.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1608.07187" label="1608.07187">
        <attvalues>
          <attvalue for="0" value="Semantics derived automatically from language corpora contain human-like&#10;  biases" />
          <attvalue for="1" value="  Artificial intelligence and machine learning are in a period of astounding&#10;growth. However, there are concerns that these technologies may be used, either&#10;with or without intention, to perpetuate the prejudice and unfairness that&#10;unfortunately characterizes many human institutions. Here we show for the first&#10;time that human-like semantic biases result from the application of standard&#10;machine learning to ordinary language---the same sort of language humans are&#10;exposed to every day. We replicate a spectrum of standard human biases as&#10;exposed by the Implicit Association Test and other well-known psychological&#10;studies. We replicate these using a widely used, purely statistical&#10;machine-learning model---namely, the GloVe word embedding---trained on a corpus&#10;of text from the Web. Our results indicate that language itself contains&#10;recoverable and accurate imprints of our historic biases, whether these are&#10;morally neutral as towards insects or flowers, problematic as towards race or&#10;gender, or even simply veridical, reflecting the {\em status quo} for the&#10;distribution of gender with respect to careers or first names. These&#10;regularities are captured by machine learning along with the rest of semantics.&#10;In addition to our empirical findings concerning language, we also contribute&#10;new methods for evaluating bias in text, the Word Embedding Association Test&#10;(WEAT) and the Word Embedding Factual Association Test (WEFAT). Our results&#10;have implications not only for AI and machine learning, but also for the fields&#10;of psychology, sociology, and human ethics, since they raise the possibility&#10;that mere exposure to everyday language can account for the biases we replicate&#10;here.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08826" label="2104.08826">
        <attvalues>
          <attvalue for="0" value="GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation" />
          <attvalue for="1" value="  Large-scale language models such as GPT-3 are excellent few-shot learners,&#10;allowing them to be controlled via natural text prompts. Recent studies report&#10;that prompt-based direct classification eliminates the need for fine-tuning but&#10;lacks data and inference scalability. This paper proposes a novel data&#10;augmentation technique that leverages large-scale language models to generate&#10;realistic text samples from a mixture of real samples. We also propose&#10;utilizing soft-labels predicted by the language models, effectively distilling&#10;knowledge from the large-scale language models and creating textual&#10;perturbations simultaneously. We perform data augmentation experiments on&#10;diverse classification tasks and show that our method hugely outperforms&#10;existing text augmentation methods. Ablation studies and a qualitative analysis&#10;provide more insights into our approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Knowledge Distillation &#10;&#10;Knowledge distillation \cite{phuong2019towards} is a technique that trains a smaller student classifier on the outputs of a larger teacher classifier. Knowledge distillation for language models in the context of model compression has been well-studied in the literature. There have been various distilled models and distillation methods proposed for pre-trained language models \cite{sanh2019distilbert, tang2019distilling}. By utilizing soft-labels predicted by the large-scale language model, our approach helps to transfer knowledge to the downstream classifiers. &#10;&#10;Text Augmentation&#10;&#10;Text augmentation refers to methods for perturbing the linguistic space without altering class labels to improve the robustness and generalizability of the downstream models. Data augmentation has been studied extensively in the NLP scene. Text augmentation in the current literature comes with two flavors: shallow and deep augmentation. The shallow data augmentation techniques inject locally plausible small noises into the linguistic space (words or phrases), in the hopes that the perturbations produce linguistically acceptable samples while maintaining label consistency. Two examples are EDA \cite{wei2019eda} and synonym replacement \cite{zhang2016characterlevel}.&#10;&#10;Another class of augmentation techniques employs external language models to improve global coherence and consistency. The back-translation approach exploits semantic consistency in translation language pairs to generate novel paraphrases \cite{fadaee2017data}. In the more recent line of work, pre-trained language models, such as BERT \cite{devlin2019bert} or the sequence-to-sequence variant BART \cite{lewis2020bart}, are used to obtain more diverse and linguistically correct augmentation samples. For example, BART has been proven to be effective in populating text samples for data-scarce labels \cite{kumar2020data}. \cite{ng2020ssmba} proposed using masked language models as a denoising autoencoder to generate synthetic texts. Some other researchers have taken the direction of perturbing the latent spaces, optionally by introducing variational inference in the architecture \cite{xia2020cg, xia2020composed, hou2018sequence, yoo2019data}.&#10;&#10;On the other hand, inspired by the mix-up technique \cite{zhang2017mixup} proposed for the vision domain, there have also been works to mix existing text samples to produce realistic augmentation texts based on statistical methods \cite{guo2020sequence, sun2020mixup, chen2020mixtext}. Furthermore, pseudo-labeling, the act of annotating unlabeled data with model predictions \cite{lee2013pseudo, reed2014training}, has been actively used in semi-supervised learning settings \cite{chen2020mixtext, xie2020unsupervised, berthelot2019mixmatch}.&#10;&#10;Large-scale Language Models&#10;&#10;Pre-trained transformer-based language models \cite{devlin2019bert, lewis2020bart} have initiated a new paradigm in the NLP scene, changing the way we design NLP pipelines. With the recent development of mega-scale language models \cite{shoeybi2019megatron, brown2020language}, we are witnessing another shift in the paradigm, namely prompt-based NLP. These large language models are essentially few-shot learners, allowing them to be controlled through natural text. There has been a steep rise in the community's interest to better understand the prompt-based mechanisms \cite{reynolds2021prompt, schick2020exploiting, shin2020autoprompt, jiang2020can, zhao2021calibrate}. Our work relies on the previous findings on prompt-based manipulation.&#10;&#10;To the best of our knowledge, this work is the first to propose using the prompt-based approach to generate synthetic samples from large-scale language models for the purpose of text augmentation.&#10;&#10;" />
          <attvalue for="4" value="Text Classification Methods, Machine Learning, Computer Science, Data Augmentation Techniques, Linguistics, Language Model Control, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2302.14225" label="2302.14225">
        <attvalues>
          <attvalue for="0" value="Weighted Sampling for Masked Language Modeling" />
          <attvalue for="1" value="  Masked Language Modeling (MLM) is widely used to pretrain language models.&#10;The standard random masking strategy in MLM causes the pre-trained language&#10;models (PLMs) to be biased toward high-frequency tokens. Representation&#10;learning of rare tokens is poor and PLMs have limited performance on downstream&#10;tasks. To alleviate this frequency bias issue, we propose two simple and&#10;effective Weighted Sampling strategies for masking tokens based on the token&#10;frequency and training loss. We apply these two strategies to BERT and obtain&#10;Weighted-Sampled BERT (WSBERT). Experiments on the Semantic Textual Similarity&#10;benchmark (STS) show that WSBERT significantly improves sentence embeddings&#10;over BERT. Combining WSBERT with calibration methods and prompt learning&#10;further improves sentence embeddings. We also investigate fine-tuning WSBERT on&#10;the GLUE benchmark and show that Weighted Sampling also improves the transfer&#10;learning capability of the backbone PLM. We further analyze and provide&#10;insights into how WSBERT improves token embeddings.&#10;" />
          <attvalue for="2" value="&#10;Early language models model context unidirectionally, either left-to-right or right-to-left. In contrast, Masked Language Modeling (MLM) replaces a subset of tokens in the input sequence with a special token \verb![MASK]! and trains the model to predict the masked tokens using their bidirectional context. MLM has been widely adopted as a self-supervised pre-training objective for learning bidirectionally contextualized language representations, such as BERT~\cite{bert} and RoBERTa~\cite{roberta}. BERT and its extensions as pre-trained language models (PLMs) have shown remarkable performance on various downstream NLP tasks. &#10;&#10;Nevertheless, recent studies reveal critical problems in MLM. \cite{representation_degeneration, how_contextual} find the contextualized word representations of BERT and other PLMs are not isotropic as they are not uniformly distributed w.r.t. direction; instead, they are anisotropic as word representations occupy a narrow cone. The token frequency in the pre-training data usually follows a long-tailed distribution. The conventional masking strategy for MLM selects tokens to mask with a uniform distribution~\cite{bert,roberta}. &#10;This random masking strategy for MLM unavoidably encounters the frequency bias issue, that is, high-frequency tokens will be masked frequently, while more informative tokens, typically with lower frequencies, will be masked much less frequently during pre-training, which would greatly harm the efficiency of pre-training, lower the quality of representations of rare tokens and limit the performance of PLMs. As shown in Figure~\ref{fig:input}, tokens selected based on their frequency (in pink, see Eqn.~\ref{eq:freq_weight}) are apparently more informative than tokens selected randomly (in blue) which are mostly high-frequency tokens but not essential to the semantics of the sentence. \cite{on_the_sentence} investigates the embedding space of MLM-trained PLMs and confirms that embeddings are biased by token frequency and rare tokens are distributed sparsely in the embedding space. \cite{promptbert} demonstrates that frequency bias indeed harms the performance of sentence embeddings generated by MLM-trained PLMs.&#10;As shown in these studies, alleviating the frequency bias issue is essential for improving effectiveness of MLM and performance of resulting PLMs.&#10;&#10;Several recent studies focus on improving efficiency of pre-training, including mixed-precision training~\cite{megatron-lm}, parameter distillation for different layers~\cite{effecient_training}, introducing a note dictionary for saving information of rare tokens~\cite{tfn}, designing different training objectives~\cite{albert,electra,cocolm}, and dropping redundant tokens during pre-training~\cite{token_dropping}. However, most of these approaches focus on modifying model architecture or optimization for pre-training. &#10;&#10;Our work focuses on alleviating the frequency bias issue in MLM and improving quality of PLMs. We propose two Weighted Sampling methods for masking tokens based on token frequency or training loss. The latter one can dynamically adjust sampling weights and achieve a good balance between masking probabilities of common tokens and rare tokens based on the learning status of PLMs. &#10;Our Weighted Sampling methods can be applied to any MLM-pretrained PLMs. In this work, we focus on investigating the effectiveness of applying Weighted Sampling to BERT as the backbone. We initialize from BERT and continue pre-training with Weighted Sampling. We denote the resulting PLM by WSBERT. We hypothesize that since Weighted Sampling could alleviate frequency bias, it could improve representation learning of rare tokens and also improve the overall quality of language representations. Quality of pre-trained language representations is generally evaluated on sentence representations generated by PLMs, commonly evaluated on the Semantic Textual Similarity (STS) benchmark \cite{semeval2012,semeval2013,semeval2014,semeval2015,semeval2016,semeval2017,sick}; and evaluated on transfer learning capability of PLMs, commonly evaluated on fine-tuning and testing on the GLUE benchmark~\cite{glue}. Recent efforts on sentence representation modeling include calibration methods~\cite{on_the_sentence,whitening}, prompt learning\cite{radford2018gpt,radford2019gpt,lm_few_shot,cloze_question, making_plm_few_shot, promptbert}, and sentence-level contrastive learning (CL) based models such as SimCSE~\cite{simcse} and its variants~\cite{continuous_prompt,infocse}. Although SimCSE and its variants achieve state-of-the-art (SOTA) performance on STS, &#10;they degrade the transfer learning capability on tasks such as SQuAD since they do not target improving token-level representation learning~\cite{tacl}. We also observe absolute 0.5 performance degradation on GLUE from SimCSE-BERT compared to BERT. &#10;&#10;In this work, to investigate whether the proposed Weighted Sampling could improve the quality of token embeddings, we evaluate sentence representations generated by WSBERT on STS and the transferability of WSBERT on GLUE. We also analyze the embedding space of WSBERT and BERT to understand how Weighted Sampling improves the quality of token embeddings.&#10;Our contributions can be summarized as follows:&#10;\begin{itemize}[leftmargin=*,noitemsep]&#10; \item We propose two Weighted Sampling methods to alleviate the frequency bias issue in conventional masked language modeling.&#10; \item We develop a new PLM, WSBERT, by applying Weighted Sampling to BERT. Different from SOTA sentence representation models, we find WSBERT outperforms BERT on both sentence representation quality and transfer learning capability. We also find integrating calibration methods and prompts into WSBERT further improve sentence representations.&#10; \item We design ablation approaches to analyze the embedding space of WSBERT and BERT. We find that with Weighted Sampling, rare tokens are more concentrated with common tokens and common tokens are more concentrated in the embedding space than BERT. We also find that both common and rare tokens are closer to the origin in WSBERT than BERT and token embeddings of WSBERT are less sparse than BERT. We believe these improvements in token embeddings caused by Weighted Sampling contribute to the improvements in sentence representations and transferability.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Transfer Learning Improvement, Token Representation Learning, Linguistics, Artificial Intelligence, Mathematics, Language Model Pretraining" />
        </attvalues>
      </node>
      <node id="2102.08473" label="2102.08473">
        <attvalues>
          <attvalue for="0" value="COCO-LM: Correcting and Contrasting Text Sequences for Language Model&#10;  Pretraining" />
          <attvalue for="1" value="  We present a self-supervised learning framework, COCO-LM, that pretrains&#10;Language Models by COrrecting and COntrasting corrupted text sequences.&#10;Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language&#10;model to corrupt text sequences, upon which it constructs two new tasks for&#10;pretraining the main model. The first token-level task, Corrective Language&#10;Modeling, is to detect and correct tokens replaced by the auxiliary model, in&#10;order to better capture token-level semantics. The second sequence-level task,&#10;Sequence Contrastive Learning, is to align text sequences originated from the&#10;same source input while ensuring uniformity in the representation space.&#10;Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms&#10;recent state-of-the-art pretrained models in accuracy, but also improves&#10;pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of&#10;its pretraining GPU hours. With the same pretraining steps of standard&#10;base/large-sized models, COCO-LM outperforms the previous best models by 1+&#10;GLUE average points.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.13240" label="2203.13240">
        <attvalues>
          <attvalue for="0" value="Token Dropping for Efficient BERT Pretraining" />
          <attvalue for="1" value="  Transformer-based models generally allocate the same amount of computation&#10;for each token in a given sequence. We develop a simple but effective &quot;token&#10;dropping&quot; method to accelerate the pretraining of transformer models, such as&#10;BERT, without degrading its performance on downstream tasks. In short, we drop&#10;unimportant tokens starting from an intermediate layer in the model to make the&#10;model focus on important tokens; the dropped tokens are later picked up by the&#10;last layer of the model so that the model still produces full-length sequences.&#10;We leverage the already built-in masked language modeling (MLM) loss to&#10;identify unimportant tokens with practically no computational overhead. In our&#10;experiments, this simple approach reduces the pretraining cost of BERT by 25%&#10;while achieving similar overall fine-tuning performance on standard downstream&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.15316" label="2103.15316">
        <attvalues>
          <attvalue for="0" value="Whitening Sentence Representations for Better Semantics and Faster&#10;  Retrieval" />
          <attvalue for="1" value="  Pre-training models such as BERT have achieved great success in many natural&#10;language processing tasks. However, how to obtain better sentence&#10;representation through these pre-training models is still worthy to exploit.&#10;Previous work has shown that the anisotropy problem is an critical bottleneck&#10;for BERT-based sentence representation which hinders the model to fully utilize&#10;the underlying semantic features. Therefore, some attempts of boosting the&#10;isotropy of sentence distribution, such as flow-based model, have been applied&#10;to sentence representations and achieved some improvement. In this paper, we&#10;find that the whitening operation in traditional machine learning can similarly&#10;enhance the isotropy of sentence representations and achieve competitive&#10;results. Furthermore, the whitening technique is also capable of reducing the&#10;dimensionality of the sentence representation. Our experimental results show&#10;that it can not only achieve promising performance but also significantly&#10;reduce the storage cost and accelerate the model retrieval speed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.04198" label="2111.04198">
        <attvalues>
          <attvalue for="0" value="TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning" />
          <attvalue for="1" value="  Masked language models (MLMs) such as BERT and RoBERTa have revolutionized&#10;the field of Natural Language Understanding in the past few years. However,&#10;existing pre-trained MLMs often output an anisotropic distribution of token&#10;representations that occupies a narrow subset of the entire representation&#10;space. Such token representations are not ideal, especially for tasks that&#10;demand discriminative semantic meanings of distinct tokens. In this work, we&#10;propose TaCL (Token-aware Contrastive Learning), a novel continual pre-training&#10;approach that encourages BERT to learn an isotropic and discriminative&#10;distribution of token representations. TaCL is fully unsupervised and requires&#10;no additional data. We extensively test our approach on a wide range of English&#10;and Chinese benchmarks. The results show that TaCL brings consistent and&#10;notable improvements over the original BERT model. Furthermore, we conduct&#10;detailed analysis to reveal the merits and inner-workings of our approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-trained Language Models. Since the introduction of BERT \cite{DBLP:conf/naacl/DevlinCLT19}, the research community has witnessed remarkable progress in the field of language model pre-training on a large amount of free text. Such advancements have led to significant progresses in a wide range of natural language understanding (NLU) tasks \cite{DBLP:journals/corr/abs-1907-11692,DBLP:conf/nips/YangDYCSL19,DBLP:conf/iclr/ClarkLLM20,DBLP:journals/corr/abs-2110-06612} and text generation tasks \cite{radford2019language,DBLP:conf/acl/LewisLGGMLSZ20,DBLP:journals/jmlr/RaffelSRLNMZLL20,DBLP:conf/eacl/SuCWVBLC21,DBLP:conf/acl/SuVBWC21,DBLP:journals/taslp/SuWCBKC21,DBLP:journals/corr/abs-2109-14739,DBLP:journals/corr/abs-2108-13740,DBLP:journals/corr/abs-2108-12516,DBLP:journals/corr/abs-2109-02492}&#10;&#10;Contrastive Learning. Generally, contrastive learning methods distinguish observed data points from fictitious negative samples. They have been widely applied to various computer vision areas, including image \cite{DBLP:conf/cvpr/ChopraHL05,oord2018representation} and video \cite{DBLP:conf/iccv/WangG15,DBLP:conf/icra/SermanetLCHJSLB18}. Recently, \cite{chen2020simple} proposed a simple framework for contrastive learning of visual representations (SimCLR) based on multi-class N-pair loss. \cite{radford2021learning,pmlr-v139-jia21b} applied the contrastive learning approach for language-image pretraining. \cite{xu2021videoclip, yang2021taco} proposed a contrastive pre-training approach for video-text alignment.&#10;&#10;In the field of NLP, numerous approaches have been proposed to learn better sentence-level \cite{reimers-gurevych-2019-sentence,wu2020clear,meng2021coco,liu2021fast,DBLP:journals/corr/abs-2104-08821,DBLP:conf/acl/Su0ZLBC0C020} and lexical-level \cite{liu-etal-2021-self,vulic-etal-2021-lexfit,liu-etal-2021-mirrorwic,Wang2021PhraseBERTIP} representations using contrastive learning. Different from our work, none of these studies specifically investigates how to utilize contrastive learning for improving general-purpose token-level representations. Beyond representation learning, contrastive learning has also been applied to other NLP applications such as NER \cite{das2021container} and summarisation \cite{liu-liu-2021-simcls}, knowledge probing for pre-trained language models \cite{DBLP:journals/corr/abs-2110-08173}, and open-ended text generation \cite{DBLP:journals/corr/abs-2202-06417}.&#10;&#10;Continual Pre-training. Many researchers \cite{xu-etal-2019-bert, gururangan2020don, pan2021multilingual} have investigated how to continually pre-train the model to alleviate the task- and domain-discrepancy between the pre-trained models and the specific target task. In contrast, our proposed approach studies how to apply continual pre-training to directly improve the quality of model representations which is transferable and beneficial to a wide range of benchmark tasks.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2304.11389" label="2304.11389">
        <attvalues>
          <attvalue for="0" value="Transformer-Based Language Model Surprisal Predicts Human Reading Times&#10;  Best with About Two Billion Training Tokens" />
          <attvalue for="1" value="  Recent psycholinguistic studies have drawn conflicting conclusions about the&#10;relationship between the quality of a language model and the ability of its&#10;surprisal estimates to predict human reading times, which has been speculated&#10;to be due to the large gap in both the amount of training data and model&#10;capacity across studies. The current work aims to consolidate these findings by&#10;evaluating surprisal estimates from Transformer-based language model variants&#10;that vary systematically in the amount of training data and model capacity on&#10;their ability to predict human reading times. The results show that surprisal&#10;estimates from most variants with contemporary model capacities provide the&#10;best fit after seeing about two billion training tokens, after which they begin&#10;to diverge from humanlike expectations. Additionally, newly-trained smaller&#10;model variants reveal a 'tipping point' at convergence, after which the&#10;decrease in language model perplexity begins to result in poorer fits to human&#10;reading times. These results suggest that the massive amount of training data&#10;is mainly responsible for the poorer fit achieved by surprisal from larger&#10;pre-trained language models, and that a certain degree of model capacity is&#10;necessary for Transformer-based language models to capture humanlike&#10;expectations.&#10;" />
          <attvalue for="2" value="&#10;The predictability of upcoming linguistic material has long been considered a crucial factor underlying difficulty in human sentence processing \cite{hale01, levy08}, and has received empirical support from numerous studies showing surprisal \cite{shannon48} to be highly predictive of relevant behavioral and neural measures \cite[e.g.][]{dembergkeller08, smithlevy13, haleetal18, shainetal20}.&#10;Since language models (LMs) are trained to estimate a conditional probability distribution of a word given its context, surprisal estimates calculated from them have often been evaluated on their ability to predict measures of processing difficulty.&#10;&#10;Recent studies in computational psycholinguistics have provided conflicting evidence with regard to the relationship between LM quality (i.e.~next-word prediction accuracy) and goodness-of-fit to human reading times.&#10;Earlier work using newly-trained LMs showed a negative relationship between LM perplexity and predictive power of surprisal estimates \cite{goodkindbicknell18, wilcoxetal20, merkxfrank21}, but more recent work using large pre-trained Transformer-based LMs \cite[e.g.~GPT-2;][]{radfordetal19} show a robust positive relationship between the two variables \cite{ohetal22, ohschuler23tacl}.&#10;While \cite{ohschuler23tacl} conjecture that these studies capture two distinct regimes, it remains less clear where the reversal in this relationship happens.&#10;The main challenge in answering this question lies in the massive difference in terms of both the amount of training data and the model capacity of LMs that were studied.&#10;&#10;The current study aims to cover this conceptual middle ground by evaluating, on their ability to predict human reading times, surprisal estimates from Transformer-based LM variants that vary systematically in the amount of training data and model capacity.&#10;Results from regression analyses show that surprisal from most LM variants with contemporary model capacities make the biggest contribution to regression model fit after seeing about two billion tokens of training data, after which additional training data result in surprisal estimates that continue to diverge from humanlike expectations.&#10;Additionally, surprisal estimates from newly-trained smaller LM variants reveal a `tipping point' at convergence, after which the decrease in perplexity begins to result in poorer fits to human reading times.&#10;Taken together, these results suggest that the vast amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger Transformer-based pre-trained LMs \cite{ohetal22, ohschuler23tacl}, and that a certain degree of model capacity is necessary for Transformer-based LMs to capture humanlike expectations that manifest in reading times.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Psycholinguistics, Language Model Evaluation, Artificial Intelligence, Human Reading Behavior, Model Capacity Optimization" />
        </attvalues>
      </node>
      <node id="2212.12131" label="2212.12131">
        <attvalues>
          <attvalue for="0" value="Why Does Surprisal From Larger Transformer-Based Language Models Provide&#10;  a Poorer Fit to Human Reading Times?" />
          <attvalue for="1" value="  This work presents a detailed linguistic analysis into why larger&#10;Transformer-based pre-trained language models with more parameters and lower&#10;perplexity nonetheless yield surprisal estimates that are less predictive of&#10;human reading times. First, regression analyses show a strictly monotonic,&#10;positive log-linear relationship between perplexity and fit to reading times&#10;for the more recently released five GPT-Neo variants and eight OPT variants on&#10;two separate datasets, replicating earlier results limited to just GPT-2 (Oh et&#10;al., 2022). Subsequently, analysis of residual errors reveals a systematic&#10;deviation of the larger variants, such as underpredicting reading times of&#10;named entities and making compensatory overpredictions for reading times of&#10;function words such as modals and conjunctions. These results suggest that the&#10;propensity of larger Transformer-based models to 'memorize' sequences during&#10;training makes their surprisal estimates diverge from humanlike expectations,&#10;which warrants caution in using pre-trained language models to study human&#10;language processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.12865" label="2203.12865">
        <attvalues>
          <attvalue for="0" value="Multilingual CheckList: Generation and Evaluation" />
          <attvalue for="1" value="  Multilingual evaluation benchmarks usually contain limited high-resource&#10;languages and do not test models for specific linguistic capabilities.&#10;CheckList is a template-based evaluation approach that tests models for&#10;specific capabilities. The CheckList template creation process requires native&#10;speakers, posing a challenge in scaling to hundreds of languages. In this work,&#10;we explore multiple approaches to generate Multilingual CheckLists. We device&#10;an algorithm - Template Extraction Algorithm (TEA) for automatically extracting&#10;target language CheckList templates from machine translated instances of a&#10;source language templates. We compare the TEA CheckLists with CheckLists&#10;created with different levels of human intervention. We further introduce&#10;metrics along the dimensions of cost, diversity, utility, and correctness to&#10;compare the CheckLists. We thoroughly analyze different approaches to creating&#10;CheckLists in Hindi. Furthermore, we experiment with 9 more different&#10;languages. We find that TEA followed by human verification is ideal for scaling&#10;Checklist-based evaluation to multiple languages while TEA gives a good&#10;estimates of model performance.&#10;" />
          <attvalue for="2" value="&#10;Multilingual transformer based models \cite{bert,xlmr,liu2020multilingual,xue2021mt5} have demonstrated commendable zero \&amp; few-shot capabilities. Their performance is typically evaluated on benchmarks like XNLI \cite{xnli}, XGLUE \cite{xglue}, XTREME \cite{hu2020xtreme} \&amp; XTREME-R \cite{ruder2021xtremer}. However, this evaluation paradigm has a number of limitations including: First, most of these datasets are limited to a few high resource languages~\cite{pmlr-v119-hu20b, wang-etal-2020-extending, vulic-etal-2020-multi}, except for a few tasks (e.g., NER, POS \cite{ahuja-etal-2022-beyond, bhatt-2021-on}). Second, creating high quality test sets of substantial size for many tasks and languages is prohibitively expensive. Third, state-of-art models are known to learn spurious patterns to achieve high accuracies, saturating performance on these test-benches, yet performing poorly on often much simpler real world cases~\cite{balanced_vqa_v2, gururangan-etal-2018-annotation, glockner2018breaking, tsuchiya2018performance, geva2019we}. Fourth, these benchmarks do not evaluate models for language specific nuances \cite{checklist-paper}. Lastly, this evaluation approach does not provide any insights into where the model is failing \cite{wu-etal-2019-errudite}. These limitations lead to the need of interactive, challenging, and much larger testing datasets (like \cite{big-bench, dyna-bench}) and more holistic approaches to evaluation (like \cite{checklist-paper}).&#10;&#10;CheckList \cite{checklist-paper} is an evaluation paradigm that systematically tests the various {\em(linguistic) capabilities} required to solve a task. It allows creation of large and targeted test sets easily using various abstractions. &#10;Specifically, users can generate {templates}, essentially sentences with {slots} that can be filled in with a dictionary of {lexicons} to generate test {instances}. CheckList templates are created by native speakers. \cite{ruder2021xtremer} introduce Multilingual Checklists created by human translation from English CheckList for 50 languages for a subset of tests on Question Answering. However, since CheckLists are task \&amp; language specific, human creation or translation of CheckLists remains extremely resource-intensive. &#10;&#10;In this paper, we introduce an automatic approach to creating Multilingual CheckLists. We devise the Template Extraction Algorithm (TEA) for extracting templates in a {target} language from the translated instances of a {source} language CheckList (here English) automatically (\S\ref{3_tea}). We also experiment with semi-automatic and manual approaches for Multilingual CheckList creation (\S\ref{4_multilingual_checklist_generation}). In the semi-automatic approach (TEA-ver), we ask human annotators to verify and correct the templates created by TEA. In the manual approach, we ask annotators to create CheckLists in two ways: first, by translation of English CheckList to the target language (t9n) (same as \cite{ruder2021xtremer}); Second, by giving a description of the task and capabilities to create CheckLists from scratch (SCR) (same as original English CheckLists creation \cite{checklist-paper}).Using these four approaches, we create CheckLists for Sentiment Analysis (SA) and Natural Language Inference (NLI) in Hindi (\S\ref{6_hindi_checklists}). We demonstrate broad applicability of TEA by generating CheckLists in additional 9 typologically diverse languages (Gujarati, French, Swahili, Arabic, German, Spanish, Russian, Vietnamese, Japanese) and TEA-ver CheckLists in 3 of them (\S\ref{7_multilingual_experiments}).&#10;&#10;Evaluation of CheckLists is non-trivial. For thorough comparisons, we propose evaluation metrics along four axes: {utility}, {diversity}, {cost} \&amp; {correctness} (\S\ref{5_metrics}). &#10;Our evaluation indicates that CheckLists created using TEA are not only cost-effective but also useful and diverse, with comparable quality to the manually and semi-automatically created CheckLists. Experiments on typologically diverse languages show that TEA CheckLists provide a good estimate of the failures of the model, and thus can be used even in the absence of resources to verify them or create human-annotated gold test-sets.&#10;&#10;To summarize, our contributions are:&#10;a) We propose TEA (Template Extraction Algorithm) to extract templates in a target language using translated instances of a source CheckList.&#10;b) We experiment with varying degrees of human intervention, comparing semi-automatic \&amp; manual approaches of Multilingual CheckList creation with TEA, to understand the best utilization of the human effort. &#10;c) We introduce evaluation metrics along the axes of utility, diversity, cost, and correctness for in-depth comparison of the the CheckLists. &#10;d) We will release all the 4 CheckLists in Hindi for SA and NLI, TEA CheckLists in 9 languages for SA and TEA-ver CheckLists in 3 languages for SA.&#10;&#10;We release the code of TEA and the CheckLists created at \href{&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Testing, Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Multilingual Evaluation, Template Generation" />
        </attvalues>
      </node>
      <node id="1612.00837" label="1612.00837">
        <attvalues>
          <attvalue for="0" value="Making the V in VQA Matter: Elevating the Role of Image Understanding in&#10;  Visual Question Answering" />
          <attvalue for="1" value="  Problems at the intersection of vision and language are of significant&#10;importance both as challenging research questions and for the rich set of&#10;applications they enable. However, inherent structure in our world and bias in&#10;our language tend to be a simpler signal for learning than visual modalities,&#10;resulting in models that ignore visual information, leading to an inflated&#10;sense of their capability.&#10;  We propose to counter these language priors for the task of Visual Question&#10;Answering (VQA) and make vision (the V in VQA) matter! Specifically, we balance&#10;the popular VQA dataset by collecting complementary images such that every&#10;question in our balanced dataset is associated with not just a single image,&#10;but rather a pair of similar images that result in two different answers to the&#10;question. Our dataset is by construction more balanced than the original VQA&#10;dataset and has approximately twice the number of image-question pairs. Our&#10;complete balanced dataset is publicly available at www.visualqa.org as part of&#10;the 2nd iteration of the Visual Question Answering Dataset and Challenge (VQA&#10;v2.0).&#10;  We further benchmark a number of state-of-art VQA models on our balanced&#10;dataset. All models perform significantly worse on our balanced dataset,&#10;suggesting that these models have indeed learned to exploit language priors.&#10;This finding provides the first concrete empirical evidence for what seems to&#10;be a qualitative sense among practitioners.&#10;  Finally, our data collection protocol for identifying complementary images&#10;enables us to develop a novel interpretable model, which in addition to&#10;providing an answer to the given (image, question) pair, also provides a&#10;counter-example based explanation. Specifically, it identifies an image that is&#10;similar to the original image, but it believes has a different answer to the&#10;same question. This can help in building trust for machines among their users.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.13455" label="2005.13455">
        <attvalues>
          <attvalue for="0" value="Self-Training for Unsupervised Parsing with PRPN" />
          <attvalue for="1" value="  Neural unsupervised parsing (UP) models learn to parse without access to&#10;syntactic annotations, while being optimized for another task like language&#10;modeling. In this work, we propose self-training for neural UP models: we&#10;leverage aggregated annotations predicted by copies of our model as supervision&#10;for future copies. To be able to use our model's predictions during training,&#10;we extend a recent neural UP architecture, the PRPN (Shen et al., 2018a) such&#10;that it can be trained in a semi-supervised fashion. We then add examples with&#10;parses predicted by our model to our unlabeled UP training data. Our&#10;self-trained model outperforms the PRPN by 8.1% F1 and the previous state of&#10;the art by 1.6% F1. In addition, we show that our architecture can also be&#10;helpful for semi-supervised parsing in ultra-low-resource settings.&#10;" />
          <attvalue for="2" value="&#10;&#10;Unsupervised parsing (UP) models learn to parse sentences into unlabeled constituency trees without the need for annotated treebanks. Self-training \cite{Yar95,Riloff+Wiebe+Wilson:03a} consists of training a model, using it to label new examples and, based on a confidence metric, adding a subset to the training set, before repeating training.&#10;For supervised parsing, results with self-training have been mixed \cite{charniak1997statistical,steedman2003bootstrapping,effselftrain}. For unsupervised dependency parsing, \cite{le2015unsupervised} obtain strong results by training a supervised parser on outputs of unsupervised parsing. UP models show low self-agreement between training runs \cite{kim2019compound}, while obtaining parsing performances far above chance. Supervising one run with confident parses from the last could combine their individual strengths.&#10;Thus, we ask the question: Can UP benefit from self-training?&#10;&#10;In order to answer this question, we propose SS-PRPN, a semi-supervised extension of the UP architecture PRPN \cite{shen2017neural}, which can be trained jointly on language modeling and supervised parsing. This enables our model to leverage silver-standard annotations obtained via self-training for supervision. Our approach draws on the idea of syntactic distances, which can be learned both as latent variables \cite{shen2017neural} and as explicit supervision targets \cite{shen2018straight}.&#10;We use both of these, leveraging annotations obtained via UP to supervise the two different outputs of the parser, in addition to standard UP training. &#10;&#10;SS-PRPN, in combination with self-training, improves over its original version by $8.1\%$ F1 and over the previous state of the art \cite{kim2019compound} by $1.6\%$ F1, when trained and evaluated on the English PTB \cite{mitchell1999treebank}: UP can indeed benefit from self-training. We further perform an analysis of our self-training procedure, finding that longer sentences benefit most from self-training.&#10;&#10;Although our primary motivation for the development of a semi-supervised architecture is to enable self-training, we further hypothesize that, since language modeling and parsing annotations seem to provide complementary information, UP should aid low-resource supervised parsing. As a proof of concept, we employ SS-PRPN for semi-supervised training. In extremely-low-data regimes with no more than 250 labeled parses, SS-PRPN outperforms supervised and unsupervised baselines in most settings on unlabeled parsing, and in all settings on labeled constituency parsing. &#10;&#10;Related Work&#10;Following the line of research on non-neural UP models \cite{clark2001unsupervised,klein2002generative,bod-2006-subtrees}, early approaches to neural UP \cite{yogatama2016learning,choi2018learning} obtain improved performance on downstream tasks, yet show highly inconsistent behavior in parsing \cite{williams2018latent}. &#10;&#10;Recently, \cite{shen2017neural} introduce the first high performing neural UP model \cite{htut2018grammar}. \cite{dyer2019critical} raise concerns that PRPN's parsing methodology is biased towards English trees.&#10;Though these concerns are serious, they are largely orthogonal to our research question regarding the helpfulness of self-training for UP.&#10;&#10;Several models have been introduced since: \cite{shen2018ordered} propose an architecture consisting of an LSTM \cite{hochreiter1997long} with a modified update function for the LSTM cell state, \cite{kim2019compound}---the current state-of-the-art---introduce a model based on a mixture of probabilistic context-free grammars, \cite{kim2019unsupervised} present unsupervised learning of recurrent neural networks grammars, \cite{li2019imitation} combine PRPN with imitation learning, and \cite{drozdov2019unsupervised} employ a recursive autoencoder. \cite{kim2020pre} examine tree induction from pre-trained models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Self-Supervised Learning, Language Modeling, Neural Parsing Models" />
        </attvalues>
      </node>
      <node id="1504.04666" label="1504.04666">
        <attvalues>
          <attvalue for="0" value="Unsupervised Dependency Parsing: Let's Use Supervised Parsers" />
          <attvalue for="1" value="  We present a self-training approach to unsupervised dependency parsing that&#10;reuses existing supervised and unsupervised parsing algorithms. Our approach,&#10;called `iterated reranking' (IR), starts with dependency trees generated by an&#10;unsupervised parser, and iteratively improves these trees using the richer&#10;probability models used in supervised parsing that are in turn trained on these&#10;trees. Our system achieves 1.8% accuracy higher than the state-of-the-part&#10;parser of Spitkovsky et al. (2013) on the WSJ corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.02013" label="1711.02013">
        <attvalues>
          <attvalue for="0" value="Neural Language Modeling by Jointly Learning Syntax and Lexicon" />
          <attvalue for="1" value="  We propose a neural language model capable of unsupervised syntactic&#10;structure induction. The model leverages the structure information to form&#10;better semantic representations and better language modeling. Standard&#10;recurrent neural networks are limited by their structure and fail to&#10;efficiently use syntactic information. On the other hand, tree-structured&#10;recursive networks usually require additional structural supervision at the&#10;cost of human expert annotation. In this paper, We propose a novel neural&#10;language model, called the Parsing-Reading-Predict Networks (PRPN), that can&#10;simultaneously induce the syntactic structure from unannotated sentences and&#10;leverage the inferred structure to learn a better language model. In our model,&#10;the gradient can be directly back-propagated from the language model loss into&#10;the neural parsing network. Experiments show that the proposed model can&#10;discover the underlying syntactic structure and achieve state-of-the-art&#10;performance on word/character-level language model tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.04168" label="1806.04168">
        <attvalues>
          <attvalue for="0" value="Straight to the Tree: Constituency Parsing with Neural Syntactic&#10;  Distance" />
          <attvalue for="1" value="  In this work, we propose a novel constituency parsing scheme. The model&#10;predicts a vector of real-valued scalars, named syntactic distances, for each&#10;split position in the input sentence. The syntactic distances specify the order&#10;in which the split points will be selected, recursively partitioning the input,&#10;in a top-down fashion. Compared to traditional shift-reduce parsing schemes,&#10;our approach is free from the potential problem of compounding errors, while&#10;being faster and easier to parallelize. Our model achieves competitive&#10;performance amongst single model, discriminative parsers in the PTB dataset and&#10;outperforms previous models in the CTB dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.09100" label="1611.09100">
        <attvalues>
          <attvalue for="0" value="Learning to Compose Words into Sentences with Reinforcement Learning" />
          <attvalue for="1" value="  We use reinforcement learning to learn tree-structured neural networks for&#10;computing representations of natural language sentences. In contrast with prior&#10;work on tree-structured models in which the trees are either provided as input&#10;or predicted using supervision from explicit treebank annotations, the tree&#10;structures in this work are optimized to improve performance on a downstream&#10;task. Experiments demonstrate the benefit of learning task-specific composition&#10;orders, outperforming both sequential encoders and recursive encoders based on&#10;treebank annotations. We analyze the induced trees and show that while they&#10;discover some linguistically intuitive structures (e.g., noun phrases, simple&#10;verb phrases), they are different than conventional English syntactic&#10;structures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.02786" label="1707.02786">
        <attvalues>
          <attvalue for="0" value="Learning to Compose Task-Specific Tree Structures" />
          <attvalue for="1" value="  For years, recursive neural networks (RvNNs) have been shown to be suitable&#10;for representing text into fixed-length vectors and achieved good performance&#10;on several natural language processing tasks. However, the main drawback of&#10;RvNNs is that they require structured input, which makes data preparation and&#10;model implementation hard. In this paper, we propose Gumbel Tree-LSTM, a novel&#10;tree-structured long short-term memory architecture that learns how to compose&#10;task-specific tree structures only from plain text data efficiently. Our model&#10;uses Straight-Through Gumbel-Softmax estimator to decide the parent node among&#10;candidates dynamically and to calculate gradients of the discrete decision. We&#10;evaluate the proposed model on natural language inference and sentiment&#10;analysis, and show that our model outperforms or is at least comparable to&#10;previous models. We also find that our model converges significantly faster&#10;than other models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.01121" label="1709.01121">
        <attvalues>
          <attvalue for="0" value="Do latent tree learning models identify meaningful structure in&#10;  sentences?" />
          <attvalue for="1" value="  Recent work on the problem of latent tree learning has made it possible to&#10;train neural networks that learn to both parse a sentence and use the resulting&#10;parse to interpret the sentence, all without exposure to ground-truth parse&#10;trees at training time. Surprisingly, these models often perform better at&#10;sentence understanding tasks than models that use parse trees from conventional&#10;parsers. This paper aims to investigate what these latent tree learning models&#10;learn. We replicate two such models in a shared codebase and find that (i) only&#10;one of these models outperforms conventional tree-structured models on sentence&#10;classification, (ii) its parsing strategies are not especially consistent&#10;across random restarts, (iii) the parses it produces tend to be shallower than&#10;standard Penn Treebank (PTB) parses, and (iv) they do not resemble those of PTB&#10;or any other semantic or syntactic formalism that the authors are aware of.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.10000" label="1808.10000">
        <attvalues>
          <attvalue for="0" value="Grammar Induction with Neural Language Models: An Unusual Replication" />
          <attvalue for="1" value="  A substantial thread of recent work on latent tree learning has attempted to&#10;develop neural network models with parse-valued latent variables and train them&#10;on non-parsing tasks, in the hope of having them discover interpretable tree&#10;structure. In a recent paper, Shen et al. (2018) introduce such a model and&#10;report near-state-of-the-art results on the target task of language modeling,&#10;and the first strong latent tree learning result on constituency parsing. In an&#10;attempt to reproduce these results, we discover issues that make the original&#10;results hard to trust, including tuning and even training on what is&#10;effectively the test set. Here, we attempt to reproduce these results in a fair&#10;experiment and to extend them to two new datasets. We find that the results of&#10;this work are robust: All variants of the model under study outperform all&#10;latent tree learning baselines, and perform competitively with symbolic grammar&#10;induction systems. We find that this model represents the first empirical&#10;success for latent tree learning, and that neural network language modeling&#10;warrants further study as a setting for grammar induction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.09428" label="1909.09428">
        <attvalues>
          <attvalue for="0" value="A Critical Analysis of Biased Parsers in Unsupervised Parsing" />
          <attvalue for="1" value="  A series of recent papers has used a parsing algorithm due to Shen et al.&#10;(2018) to recover phrase-structure trees based on proxies for &quot;syntactic&#10;depth.&quot; These proxy depths are obtained from the representations learned by&#10;recurrent language models augmented with mechanisms that encourage the&#10;(unsupervised) discovery of hierarchical structure latent in natural language&#10;sentences. Using the same parser, we show that proxies derived from a&#10;conventional LSTM language model produce trees comparably well to the&#10;specialized architectures used in previous work. However, we also provide a&#10;detailed analysis of the parsing algorithm, showing (1) that it is&#10;incomplete---that is, it can recover only a fraction of possible trees---and&#10;(2) that it has a marked bias for right-branching structures which results in&#10;inflated performance in right-branching languages like English. Our analysis&#10;shows that evaluating with biased parsing algorithms can inflate the apparent&#10;structural competence of language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.03746" label="1904.03746">
        <attvalues>
          <attvalue for="0" value="Unsupervised Recurrent Neural Network Grammars" />
          <attvalue for="1" value="  Recurrent neural network grammars (RNNG) are generative models of language&#10;which jointly model syntax and surface structure by incrementally generating a&#10;syntax tree and sentence in a top-down, left-to-right order. Supervised RNNGs&#10;achieve strong language modeling and parsing performance, but require an&#10;annotated corpus of parse trees. In this work, we experiment with unsupervised&#10;learning of RNNGs. Since directly marginalizing over the space of latent trees&#10;is intractable, we instead apply amortized variational inference. To maximize&#10;the evidence lower bound, we develop an inference network parameterized as a&#10;neural CRF constituency parser. On language modeling, unsupervised RNNGs&#10;perform as well their supervised counterparts on benchmarks in English and&#10;Chinese. On constituency grammar induction, they are competitive with recent&#10;neural language models that induce tree structures from words through attention&#10;mechanisms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.02276" label="1906.02276">
        <attvalues>
          <attvalue for="0" value="An Imitation Learning Approach to Unsupervised Parsing" />
          <attvalue for="1" value="  Recently, there has been an increasing interest in unsupervised parsers that&#10;optimize semantically oriented objectives, typically using reinforcement&#10;learning. Unfortunately, the learned trees often do not match actual syntax&#10;trees well. Shen et al. (2018) propose a structured attention mechanism for&#10;language modeling (PRPN), which induces better syntactic structures but relies&#10;on ad hoc heuristics. Also, their model lacks interpretability as it is not&#10;grounded in parsing actions. In our work, we propose an imitation learning&#10;approach to unsupervised parsing, where we transfer the syntactic knowledge&#10;induced by the PRPN to a Tree-LSTM model with discrete parsing actions. Its&#10;policy is then refined by Gumbel-Softmax training towards a semantically&#10;oriented objective. We evaluate our approach on the All Natural Language&#10;Inference dataset and show that it achieves a new state of the art in terms of&#10;parsing $F$-score, outperforming our base models, including the PRPN.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12696" label="2305.12696">
        <attvalues>
          <attvalue for="0" value="Learning Interpretable Style Embeddings via Prompting LLMs" />
          <attvalue for="1" value="  Style representation learning builds content-independent representations of&#10;author style in text. Stylometry, the analysis of style in text, is often&#10;performed by expert forensic linguists and no large dataset of stylometric&#10;annotations exists for training. Current style representation learning uses&#10;neural methods to disentangle style from content to create style vectors,&#10;however, these approaches result in uninterpretable representations,&#10;complicating their usage in downstream applications like authorship attribution&#10;where auditing and explainability is critical. In this work, we use prompting&#10;to perform stylometry on a large number of texts to create a synthetic dataset&#10;and train human-interpretable style representations we call LISA embeddings. We&#10;release our synthetic stylometry dataset and our interpretable style models as&#10;resources.&#10;" />
          <attvalue for="2" value="&#10;Style representation learning aims to represent the stylistic attributes of an authored text. Prior work has treated the style of a text as separable from the content. Stylistic attributes have included, but are not limited to, linguistic choices in syntax, grammar, spelling, vocabulary, and punctuation \cite{stylevscontent1}. Style representations should represent two texts with similar stylistic attributes more closely than texts with different attributes independent of what content is present in the texts.&#10;&#10;Stylometry, the analysis of style, applies forensic linguistics to tasks like authorship attribution. Stylometry often relies on semi-manual analysis by forensic linguistic experts \cite{forensic_or_stylometry1,forensic_or_stylometry2,forensic_or_stylometry5}. Computational stylometry often uses rule-based approaches utilizing count-based features like the frequencies of function words \cite{forensic_or_stylometry3,forensic_or_stylometry4,liwc}. More modern, neural approaches attempt to learn style representations in an unsupervised fashion through a proxy task like style transfer \cite{styletransfervector3,styletransfervector4,styletransfervector1,styletransfervector6,styletransfervector7,styletransfervector5,styletransfervector2} or authorship verification \cite{avvector1,avvector2,avvector3,styleemb}. These stronger neural approaches, unlike simpler frequency-based techniques, are uninterpretable. This makes it difficult to effectively analyze their representations and their failure modes, and precludes their usage in real-world authorship attribution scenarios because interpretability and verification is critical for legal admissibility \cite{legaladmissibility}.&#10;\mainfig&#10;&#10;With this motivation, we propose a human-interpretable style representation model $\mathcal{M}$ which, for a given text $t$, produces a $D$-dimensional vector $\mathcal{M}(t) \in [0.0,1.0]^D$. Each dimension corresponds to one of $D$ style attributes $\{a_0, a_1, \ldots, a_D\}$. Each element at dimension $d$ of this vector is constrained in the range $[0.0, 1.0]$ to represent the probability of the corresponding style attribute $a_d$ being present in the text $t$. See Figure \ref{fig:main} for a visualization of a result from our final trained model with $D = 768$ dimensions. An immediate obstacle to train such a model is that no large dataset of texts with stylometric annotations currently exists; annotating a large number of texts on a wide variety ($D=768$) of stylistic attributes would likely require annotators with linguistic expertise and be prohibitively expensive. Given this, we use GPT-3 \cite{gpt3}, a large language model (LLM), and zero-shot prompts to generate a synthetic dataset we call StyleGenome of human-interpretable stylometric annotations for various texts. Our approach is motivated by recent works showing models trained on synthetic datasets annotated by prompting LLMs can match and sometimes even outperform models trained on human-labeled datasets \cite{llmannotate1, llmannotate2,llmdistill2,llmdistill3}. Training on StyleGenome, we develop the Linguistically-Interpretable Style Attribute (Lisa) embedding model. We summarize our primary contributions:&#10;\begin{enumerate}&#10; \item We outline an unsupervised method for producing interpretable style embeddings using zero-shot prompting and distillation.&#10; \item We generate and release StyleGenome, a synthetic stylometry dataset with \textasciitilde 5.5M examples, the first large-scale dataset with texts paired with wide range of stylometric annotations.&#10; \item We train, evaluate, and release Lisa, the first-ever interpretable style embedding model with a wide variety of linguistic dimensions ($D = 768$). We find Lisa matches the performance of existing style representations, while allowing for explainability and transparency.&#10;\end{enumerate}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Interpretable Embeddings, Linguistics, Cognitive Science, Style Representation Learning, Stylometry Analysis, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1905.05621" label="1905.05621">
        <attvalues>
          <attvalue for="0" value="Style Transformer: Unpaired Text Style Transfer without Disentangled&#10;  Latent Representation" />
          <attvalue for="1" value="  Disentangling the content and style in the latent space is prevalent in&#10;unpaired text style transfer. However, two major issues exist in most of the&#10;current neural models. 1) It is difficult to completely strip the style&#10;information from the semantics for a sentence. 2) The recurrent neural network&#10;(RNN) based encoder and decoder, mediated by the latent representation, cannot&#10;well deal with the issue of the long-term dependency, resulting in poor&#10;preservation of non-stylistic semantic content. In this paper, we propose the&#10;Style Transformer, which makes no assumption about the latent representation of&#10;source sentence and equips the power of attention mechanism in Transformer to&#10;achieve better style transfer and better content preservation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.09395" label="1908.09395">
        <attvalues>
          <attvalue for="0" value="Domain Adaptive Text Style Transfer" />
          <attvalue for="1" value="  Text style transfer without parallel data has achieved some practical&#10;success. However, in the scenario where less data is available, these methods&#10;may yield poor performance. In this paper, we examine domain adaptation for&#10;text style transfer to leverage massively available data from other domains.&#10;These data may demonstrate domain shift, which impedes the benefits of&#10;utilizing such data for training. To address this challenge, we propose simple&#10;yet effective domain adaptive text style transfer models, enabling&#10;domain-adaptive information exchange. The proposed models presumably learn from&#10;the source domain to: (i) distinguish stylized information and generic content&#10;information; (ii) maximally preserve content information; and (iii) adaptively&#10;transfer the styles in a domain-aware manner. We evaluate the proposed models&#10;on two style transfer tasks (sentiment and formality) over multiple target&#10;domains where only limited non-parallel data is available. Extensive&#10;experiments demonstrate the effectiveness of the proposed model compared to the&#10;baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Text Style Transfer. Text style transfer using neural networks has been widely studied in the past few years. A common paradigm is to first disentangle latent space as content and style features, and then generate stylistic sentences by tweaking the style-relevant features and passing through a decoder. \cite{hu2017toward,fu2018style,shen2017style,yang2018unsupervised,gong2019reinforcement,lin2017adversarial} explored this direction by assuming the disentanglement can be achieved in an auto-encoding procedure with a suitable style regularization, implemented by either adversarial discriminators or style classifiers. \cite{li2018delete,xu2018unpaired,zhang2018learning} achieved disentanglement by filtering the stylistic words of input sentences. Recently, \cite{prabhumoye2018style} has proposed to use back-translation for text style transfer with a de-noising auto-encoding objective~\cite{logeswaran2018content,subramanian2018multiple}. Our work differs from the above in that we leverage domain adaptation to deal with limited target domain data, whereas previous methods require massive target domain style-labelled samples.&#10;&#10;Domain Adaptation.&#10;Domain adaptation has been studied in various natural language processing tasks, such as sentiment classification~\cite{qu2019adversarial}, dialogue system~\cite{wen2016multi}, abstractive summarization~\cite{hua2017pilot,zhang2018shaped}, machine translation~\cite{koehn2007experiments,axelrod2011domain,sennrich2016improving,michel2018extreme}, etc. However, no work has been done for exploring domain adaptation on text style transfer. To our best knowledge, we are the first to explore the adaptation of text style transfer models for a new domain with limited non-parallel data available. The task requires both style transfer and domain-specific generation on the target domain. To differentiate different domains, \cite{sennrich2016controlling,chu2017empirical} appended domain tokens to the input sentences. Our model uses learnable domain vectors combining domain-specific style classifiers, which force the model to learn distinct stylized information in each domain. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2212.09670" label="2212.09670">
        <attvalues>
          <attvalue for="0" value="StyleFlow: Disentangle Latent Representations via Normalizing Flow for&#10;  Unsupervised Text Style Transfer" />
          <attvalue for="1" value="  Text style transfer aims to alter the style of a sentence while preserving&#10;its content. Due to the lack of parallel corpora, most recent work focuses on&#10;unsupervised methods and often uses cycle construction to train models. Since&#10;cycle construction helps to improve the style transfer ability of the model by&#10;rebuilding transferred sentences back to original-style sentences, it brings&#10;about a content loss in unsupervised text style transfer tasks. In this paper,&#10;we propose a novel disentanglement-based style transfer model StyleFlow to&#10;enhance content preservation. Instead of the typical encoder-decoder scheme,&#10;StyleFlow can not only conduct the forward process to obtain the output, but&#10;also infer to the input through the output. We design an attention-aware&#10;coupling layers to disentangle the content representations and the style&#10;representations of a sentence. Besides, we propose a data augmentation method&#10;based on Normalizing Flow to improve the robustness of the model. Experiment&#10;results demonstrate that our model preserves content effectively and achieves&#10;the state-of-the-art performance on the most metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.07844" label="1908.07844">
        <attvalues>
          <attvalue for="0" value="Similarity Learning for Authorship Verification in Social Media" />
          <attvalue for="1" value="  Authorship verification tries to answer the question if two documents with&#10;unknown authors were written by the same author or not. A range of successful&#10;technical approaches has been proposed for this task, many of which are based&#10;on traditional linguistic features such as n-grams. These algorithms achieve&#10;good results for certain types of written documents like books and novels.&#10;Forensic authorship verification for social media, however, is a much more&#10;challenging task since messages tend to be relatively short, with a large&#10;variety of different genres and topics. At this point, traditional methods&#10;based on features like n-grams have had limited success. In this work, we&#10;propose a new neural network topology for similarity learning that&#10;significantly improves the performance on the author verification task with&#10;such challenging data sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.03158" label="2109.03158">
        <attvalues>
          <attvalue for="0" value="Idiosyncratic but not Arbitrary: Learning Idiolects in Online Registers&#10;  Reveals Distinctive yet Consistent Individual Styles" />
          <attvalue for="1" value="  An individual's variation in writing style is often a function of both social&#10;and personal attributes. While structured social variation has been extensively&#10;studied, e.g., gender based variation, far less is known about how to&#10;characterize individual styles due to their idiosyncratic nature. We introduce&#10;a new approach to studying idiolects through a massive cross-author comparison&#10;to identify and encode stylistic features. The neural model achieves strong&#10;performance at authorship identification on short texts and through an&#10;analogy-based probing task, showing that the learned representations exhibit&#10;surprising regularities that encode qualitative and quantitative shifts of&#10;idiolectal styles. Through text perturbation, we quantify the relative&#10;contributions of different linguistic elements to idiolectal variation.&#10;Furthermore, we provide a description of idiolects through measuring inter- and&#10;intra-author variation, showing that variation in idiolects is often&#10;distinctive yet consistent.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.04907" label="2204.04907">
        <attvalues>
          <attvalue for="0" value="Same Author or Just Same Topic? Towards Content-Independent Style&#10;  Representations" />
          <attvalue for="1" value="  Linguistic style is an integral component of language. Recent advances in the&#10;development of style representations have increasingly used training objectives&#10;from authorship verification (AV): Do two texts have the same author? The&#10;assumption underlying the AV training task (same author approximates same&#10;writing style) enables self-supervised and, thus, extensive training. However,&#10;a good performance on the AV task does not ensure good &quot;general-purpose&quot; style&#10;representations. For example, as the same author might typically write about&#10;certain topics, representations trained on AV might also encode content&#10;information instead of style alone. We introduce a variation of the AV training&#10;task that controls for content using conversation or domain labels. We evaluate&#10;whether known style dimensions are represented and preferred over content&#10;information through an original variation to the recently proposed STEL&#10;framework. We find that representations trained by controlling for conversation&#10;are better than representations trained with domain or no content control at&#10;representing style independent from content.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.11610" label="2210.11610">
        <attvalues>
          <attvalue for="0" value="Large Language Models Can Self-Improve" />
          <attvalue for="1" value="  Large Language Models (LLMs) have achieved excellent performances in various&#10;tasks. However, fine-tuning an LLM requires extensive supervision. Human, on&#10;the other hand, may improve their reasoning abilities by self-thinking without&#10;external inputs. In this work, we demonstrate that an LLM is also capable of&#10;self-improving with only unlabeled datasets. We use a pre-trained LLM to&#10;generate &quot;high-confidence&quot; rationale-augmented answers for unlabeled questions&#10;using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM&#10;using those self-generated solutions as target outputs. We show that our&#10;approach improves the general reasoning ability of a 540B-parameter LLM&#10;(74.4%-&gt;82.1% on GSM8K, 78.2%-&gt;83.0% on DROP, 90.0%-&gt;94.4% on OpenBookQA, and&#10;63.4%-&gt;67.9% on ANLI-A3) and achieves state-of-the-art-level performance,&#10;without any ground truth label. We conduct ablation studies and show that&#10;fine-tuning on reasoning is critical for self-improvement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.07834" label="2007.07834">
        <attvalues>
          <attvalue for="0" value="InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language&#10;  Model Pre-Training" />
          <attvalue for="1" value="  In this work, we present an information-theoretic framework that formulates&#10;cross-lingual language model pre-training as maximizing mutual information&#10;between multilingual-multi-granularity texts. The unified view helps us to&#10;better understand the existing methods for learning cross-lingual&#10;representations. More importantly, inspired by the framework, we propose a new&#10;pre-training task based on contrastive learning. Specifically, we regard a&#10;bilingual sentence pair as two views of the same meaning and encourage their&#10;encoded representations to be more similar than the negative examples. By&#10;leveraging both monolingual and parallel corpora, we jointly train the pretext&#10;tasks to improve the cross-lingual transferability of pre-trained models.&#10;Experimental results on several benchmarks show that our approach achieves&#10;considerably better performance. The code and pre-trained models are available&#10;at https://aka.ms/infoxlm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Cross-Lingual Representation, Information Theory, Computer Science, Contrastive Learning, Linguistics, Language Model Pre-training, Mathematics" />
        </attvalues>
      </node>
      <node id="2104.11127" label="2104.11127">
        <attvalues>
          <attvalue for="0" value="Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network" />
          <attvalue for="1" value="  Adaption of end-to-end speech recognition systems to new tasks is known to be&#10;challenging. A number of solutions have been proposed which apply external&#10;language models with various fusion methods, possibly with a combination of&#10;two-pass decoding. Also TTS systems have been used to generate adaptation data&#10;for the end-to-end models. In this paper we show that RNN-transducer models can&#10;be effectively adapted to new domains using only small amounts of textual data.&#10;By taking advantage of model's inherent structure, where the prediction network&#10;is interpreted as a language model, we can apply fast adaptation to the model.&#10;Adapting the model avoids the need for complicated decoding time fusions and&#10;external language models. Using appropriate regularization, the prediction&#10;network can be adapted to new domains while still retaining good generalization&#10;capabilities. We show with multiple ASR evaluation tasks how this method can&#10;provide relative gains of 10-45% in target task WER. We also share insights how&#10;RNN-transducer prediction network performs as a language model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.01330" label="2005.01330">
        <attvalues>
          <attvalue for="0" value="From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of&#10;  Parsing Morphologically-Rich Languages (MRLs)?" />
          <attvalue for="1" value="  It has been exactly a decade since the first establishment of SPMRL, a&#10;research initiative unifying multiple research efforts to address the peculiar&#10;challenges of Statistical Parsing for Morphologically-Rich Languages&#10;(MRLs).Here we reflect on parsing MRLs in that decade, highlight the solutions&#10;and lessons learned for the architectural, modeling and lexical challenges in&#10;the pre-neural era, and argue that similar challenges re-emerge in neural&#10;architectures for MRLs. We then aim to offer a climax, suggesting that&#10;incorporating symbolic ideas proposed in SPMRL terms into nowadays neural&#10;architectures has the potential to push NLP for MRLs to a new level. We sketch&#10;strategies for designing Neural Models for MRLs (NMRL), and showcase&#10;preliminary support for these strategies via investigating the task of&#10;multi-tagging in Hebrew, a morphologically-rich, high-fusion, language&#10;" />
          <attvalue for="2" value="&#10; The ability to process natural language data and to automatically extract structured meanings out of them has always been the hallmark of Artificial Intelligence (AI), and today it is also of immense practical value in downstream technological applications for Information Extraction, Text Analytics, and diverse Data Science applications.&#10; The introduction of deep learning models \cite{deeplearn} into Natural Language Processing (NLP) has led to an explosion in the Neural models and pre-training techniques applied to NLP tasks --- from classical tasks as tagging and parsing to end-to-end tasks as machine translation and question answering --- raising the performance bar on these tasks to an all-times peak.&#10;So far though, these advances have been reported mostly for English. Can these advances carry over to languages that are typologically vastly different from English, such as {Morphologically-Rich Languages}?&#10;&#10;The term {Morphologically-Rich Languages} (MRLs) refers to languages such as Arabic, Hebrew, Turkish or Maltese, in which significant information is expressed morphologically, e.g., via word-level variation, rather than syntactically, e.g., via fixed word-order and periphrastic constructions, as in English. These properties lead to diverse and ambiguous structures, accompanied with huge lexica, which in turn make&#10; MRLs notoriously hard to parse \cite{nivre07conll,tsarfaty13a}.&#10;A decade ago, \newcite{mrls} put forth three overarching challenges for the MRLs research community:&#10;\begin{itemize}&#10;\item[] {(i) The Architectural Challenge:} What {input units} are adequate for processing MRLs? &#10;\\ {(ii) The Modeling Challenge:} What {modeling assumptions} are adequate for MRLs?&#10;\\{(iii) The Lexical Challenge:} How can we cope with extreme {data sparseness} in MRLs lexica?&#10;\end{itemize}&#10;&#10;For NLP in the pre-neural era, effective solutions have been proposed and successfully applied to address each of these challenges for MRLs, using data from MRLs treebanks and designated shared tasks \cite{nivre07conll,spmrl13,spmrl14,nivre16ud}. The solutions proposed to the above challenges included: {(i)} parsing {morphemes} rather than words, {(ii)} {joint modeling} of local morphology and global structures, and {(iii)} exploiting {external knowledge} to analyze the {long tail} of un-attested word-forms. &#10;&#10;Upon the introduction of Neural Network models into NLP \cite{primer}, it was hoped that we could dispense with the need to model different languages differently. Curiously though, this has not been the case. Languages with rich morphology typically require careful treatment, and often the design of additional resources (cf.\ \newcite{tail}). Moreover, current modeling strategies for neural NLP appear to stand {in contrast} with the pre-neural proposals for processing MRLs. &#10;&#10;First, unsupervised pre-training techniques employing language modeling objectives (LM, MLM) are applied nowadays to {raw words} rather than morphemes, and deliver {word-embeddings} agnostic to internal structure. While some morphological structure may be implicitly encoded in these vectors, the {morphemes} themselves remain un-accessible \cite{claraDEP,cotterell15}. &#10;&#10;Second, pre-neural models for parsing MRLs call for {joint} inference over local and global structures, tasking multiple, ambiguous, morphological analyses (a.k.a.\ {lattices}) as input, and disambiguating these morphological structure jointly with the parsing task \cite{goldberg08joint,green10arabic, bohnet13joint,seeker15,more19}. In contrast, pre-trained embeddings select a single vector for each input token --- prior to any further analysis. &#10;&#10;Finally, pre-trained embeddings trained on {words} cannot assign vectors to unseen words. The use of unsupervised {char-based} or {sub-word} units \cite{enrich17} to remedy this situation shows mixed results; while these models learn {orthographic} similarities between seen and unseen words, they fail to learn the {functions} of sub-word units (\newcite{oded}; \newcite{claraLM} and references therein).&#10;&#10;This paper aims to underscore the challenges of processing MRLs, reiterate the lessons learned in the pre-neural era, and establish their relevance to MRL processing in neural terms. On the one hand, technical proposals as pre-trained embeddings, fine-tuning, and end-to-end modeling, have advanced NLP greatly. On the other hand, neural advances often overlook MRL complexities, and disregard strategies that were proven useful for MRLs in the past. We argue that breakthroughs in {Neural Models for MRLs} (NMRL) can be obtained by incorporating {symbolic} knowledge and {pre-neural} strategies into the end-to-end {neural} architectures. &#10;&#10;The remainder of this paper is organized as follows. In Section~\ref{sec:dl} we survey the methodological changes that neural modeling brought into NLP. In Section~\ref{sec:mrl} we characterize MRLs and qualify the challenges that they pose to neural NLP. In Section~\ref{sec:dlmrl} we assess the compatibility of pre-neural modeling and current neural modeling practices for MRLs, and in Section~\ref{sec:obj} we suggest to re-frame pre-neural solution strategies in neural terms. In Section~\ref{sec:exp} we present preliminary empirical support for these strategies, and in Section~\ref{sec:con} we conclude. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Morphologically Rich Languages, Computer Science, Linguistics, Neural Architectures, Mathematics, Natural Language Processing, Statistical Parsing" />
        </attvalues>
      </node>
      <node id="1807.07987" label="1807.07987">
        <attvalues>
          <attvalue for="0" value="Deep Learning" />
          <attvalue for="1" value="  Deep learning (DL) is a high dimensional data reduction technique for&#10;constructing high-dimensional predictors in input-output models. DL is a form&#10;of machine learning that uses hierarchical layers of latent features. In this&#10;article, we review the state-of-the-art of deep learning from a modeling and&#10;algorithmic perspective. We provide a list of successful areas of applications&#10;in Artificial Intelligence (AI), Image Processing, Robotics and Automation.&#10;Deep learning is predictive in its nature rather then inferential and can be&#10;viewed as a black-box methodology for high-dimensional function estimation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1510.00726" label="1510.00726">
        <attvalues>
          <attvalue for="0" value="A Primer on Neural Network Models for Natural Language Processing" />
          <attvalue for="1" value="  Over the past few years, neural networks have re-emerged as powerful&#10;machine-learning models, yielding state-of-the-art results in fields such as&#10;image recognition and speech processing. More recently, neural network models&#10;started to be applied also to textual natural language signals, again with very&#10;promising results. This tutorial surveys neural network models from the&#10;perspective of natural language processing research, in an attempt to bring&#10;natural-language researchers up to speed with the neural techniques. The&#10;tutorial covers input encoding for natural language tasks, feed-forward&#10;networks, convolutional networks, recurrent networks and recursive networks, as&#10;well as the computation graph abstraction for automatic gradient computation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.09180" label="1808.09180">
        <attvalues>
          <attvalue for="0" value="What do character-level models learn about morphology? The case of&#10;  dependency parsing" />
          <attvalue for="1" value="  When parsing morphologically-rich languages with neural models, it is&#10;beneficial to model input at the character level, and it has been claimed that&#10;this is because character-level models learn morphology. We test these claims&#10;by comparing character-level models to an oracle with access to explicit&#10;morphological analysis on twelve languages with varying morphological&#10;typologies. Our results highlight many strengths of character-level models, but&#10;also show that they are poor at disambiguating some words, particularly in the&#10;face of case syncretism. We then demonstrate that explicitly modeling&#10;morphological case improves our best model, showing that character-level models&#10;can benefit from targeted forms of explicit morphological modeling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11498" label="2305.11498">
        <attvalues>
          <attvalue for="0" value="Recouple Event Field via Probabilistic Bias for Event Extraction" />
          <attvalue for="1" value="  Event Extraction (EE), aiming to identify and classify event triggers and&#10;arguments from event mentions, has benefited from pre-trained language models&#10;(PLMs). However, existing PLM-based methods ignore the information of&#10;trigger/argument fields, which is crucial for understanding event schemas. To&#10;this end, we propose a Probabilistic reCoupling model enhanced Event extraction&#10;framework (ProCE). Specifically, we first model the syntactic-related event&#10;fields as probabilistic biases, to clarify the event fields from ambiguous&#10;entanglement. Furthermore, considering multiple occurrences of the same&#10;triggers/arguments in EE, we explore probabilistic interaction strategies among&#10;multiple fields of the same triggers/arguments, to recouple the corresponding&#10;clarified distributions and capture more latent information fields. Experiments&#10;on EE datasets demonstrate the effectiveness and generalization of our proposed&#10;approach.&#10;" />
          <attvalue for="2" value="&#10;&#10;Event extraction is an essential information extraction (IE) task, aiming to extract event structures from unstructured event mentions. It consists of event detection (ED) and event argument extraction (EAE). For example, in the mention &quot;CNN's Kelly reports on Netanya's attack.&quot;, the ED model should identify the event trigger &quot;attack&quot; and classify event type &quot;Conflict&quot;, the EAE model should identify the event argument &quot;Netanya&quot; and classify argument role &quot;Place&quot;.&#10;&#10;Existing EE methods mainly focus on feature engineering. Inspired by the significant performance of PLMs, some prior work \cite{wang2019adversarial,wadden2019entity} utilize general PLMs, such as BERT \cite{devlin2019bert}, to construct global dependencies among context words by self-attention. However, PLMs suffer from handling the entanglement of triggers/arguments. As shown in Figure~\ref{attack}, in the sentence &quot;Attack happened without declaration of war, the attack was judged in trials.&quot;, the first trigger &quot;attack&quot; is more important to the second trigger &quot;attack&quot; than other words when computing self-attention, the information of trigger fields, which is defined as the core auxiliary information scope of triggers, is not highlighted and strengthened enough from the original self-attention entanglement.&#10;&#10;To avoid the problem of auxiliary information insufficiency, dependency tree based Graph Convolution Network (GCN) \cite{nguyen2018graph,liu2018jointly,DBLP:conf/wsdm/WuBG0LY23} was adopted to capture syntactic relations between triggers and related words. However, this method still has some problems: (1) GCN focuses on the nearest syntactic neighbors ~\cite{nguyen2018graph,liu2018jointly}, over-smoothing \cite{zhou2020graph} in deep layers limits the message passing; (2) GCN has not modeled the event fields and entanglement among the identical event fields.&#10;&#10;Another localness-enhanced method, which models central word regions as Gaussian priors \cite{yang2018modeling,guo2019gaussian}, could expand the neighbor scopes without changing the model structure.&#10;&#10;In this paper, we propose a novel probabilistic recoupling model enhanced EE framework. Specifically, we model and clarify the event fields using probabilistic priors over the corresponding window (i.e., the deviation of syntactic dependency distribution) to the central word (i.e., the position of trigger/argument), and further decouple the original ambiguous self-attention entanglement for the first time.&#10;&#10;According to our statistics on the ACE-2005 dataset, 13.18\% (1959/14862) of mentions have the same triggers and arguments occurring multiple times. The same triggers/arguments in the same mentions are identical, modeling the entanglement among multiple occurrences of the same triggers/arguments is beneficial. We explore several probabilistic interaction methods among identical trigger/argument fields to recouple the clarified fields and capture more latent knowledge fields. In addition, we adopt a novel distribution metric loss and regularize the output distributions of the identical triggers/argument fields to be consistent by minimizing the Wasserstein (WA) divergence among the outputs. Experiments on several datasets indicate that ProCE achieves significant performances on overall and few-sample settings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Probabilistic Modeling, Computer Science, Event Extraction, Linguistics, Language Models, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1909.03546" label="1909.03546">
        <attvalues>
          <attvalue for="0" value="Entity, Relation, and Event Extraction with Contextualized Span&#10;  Representations" />
          <attvalue for="1" value="  We examine the capabilities of a unified, multi-task framework for three&#10;information extraction tasks: named entity recognition, relation extraction,&#10;and event extraction. Our framework (called DyGIE++) accomplishes all tasks by&#10;enumerating, refining, and scoring text spans designed to capture local&#10;(within-sentence) and global (cross-sentence) context. Our framework achieves&#10;state-of-the-art results across all tasks, on four datasets from a variety of&#10;domains. We perform experiments comparing different techniques to construct&#10;span representations. Contextualized embeddings like BERT perform well at&#10;capturing relationships among entities in the same or adjacent sentences, while&#10;dynamic span graph updates model long-range cross-sentence relationships. For&#10;instance, propagating span representations via predicted coreference links can&#10;enable the model to disambiguate challenging entity mentions. Our code is&#10;publicly available at https://github.com/dwadden/dygiepp and can be easily&#10;adapted for new tasks or datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.10991" label="2211.10991">
        <attvalues>
          <attvalue for="0" value="Modeling Fine-grained Information via Knowledge-aware Hierarchical Graph&#10;  for Zero-shot Entity Retrieval" />
          <attvalue for="1" value="  Zero-shot entity retrieval, aiming to link mentions to candidate entities&#10;under the zero-shot setting, is vital for many tasks in Natural Language&#10;Processing. Most existing methods represent mentions/entities via the sentence&#10;embeddings of corresponding context from the Pre-trained Language Model.&#10;However, we argue that such coarse-grained sentence embeddings can not fully&#10;model the mentions/entities, especially when the attention scores towards&#10;mentions/entities are relatively low. In this work, we propose GER, a&#10;\textbf{G}raph enhanced \textbf{E}ntity \textbf{R}etrieval framework, to&#10;capture more fine-grained information as complementary to sentence embeddings.&#10;We extract the knowledge units from the corresponding context and then&#10;construct a mention/entity centralized graph. Hence, we can learn the&#10;fine-grained information about mention/entity by aggregating information from&#10;these knowledge units. To avoid the graph information bottleneck for the&#10;central mention/entity node, we construct a hierarchical graph and design a&#10;novel Hierarchical Graph Attention Network~(HGAN). Experimental results on&#10;popular benchmarks demonstrate that our proposed GER framework performs better&#10;than previous state-of-the-art models. The code has been available at&#10;https://github.com/wutaiqiang/GER-WSDM2023.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.08434" label="1812.08434">
        <attvalues>
          <attvalue for="0" value="Graph Neural Networks: A Review of Methods and Applications" />
          <attvalue for="1" value="  Lots of learning tasks require dealing with graph data which contains rich&#10;relation information among elements. Modeling physics systems, learning&#10;molecular fingerprints, predicting protein interface, and classifying diseases&#10;demand a model to learn from graph inputs. In other domains such as learning&#10;from non-structural data like texts and images, reasoning on extracted&#10;structures (like the dependency trees of sentences and the scene graphs of&#10;images) is an important research topic which also needs graph reasoning models.&#10;Graph neural networks (GNNs) are neural models that capture the dependence of&#10;graphs via message passing between the nodes of graphs. In recent years,&#10;variants of GNNs such as graph convolutional network (GCN), graph attention&#10;network (GAT), graph recurrent network (GRN) have demonstrated ground-breaking&#10;performances on many deep learning tasks. In this survey, we propose a general&#10;design pipeline for GNN models and discuss the variants of each component,&#10;systematically categorize the applications, and propose four open problems for&#10;future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.10182" label="1810.10182">
        <attvalues>
          <attvalue for="0" value="Modeling Localness for Self-Attention Networks" />
          <attvalue for="1" value="  Self-attention networks have proven to be of profound value for its strength&#10;of capturing global dependencies. In this work, we propose to model localness&#10;for self-attention networks, which enhances the ability of capturing useful&#10;local context. We cast localness modeling as a learnable Gaussian bias, which&#10;indicates the central and scope of the local region to be paid more attention.&#10;The bias is then incorporated into the original attention distribution to form&#10;a revised distribution. To maintain the strength of capturing long distance&#10;dependencies and enhance the ability of capturing short-range dependencies, we&#10;only apply localness modeling to lower layers of self-attention networks.&#10;Quantitative and qualitative analyses on Chinese-English and English-German&#10;translation tasks demonstrate the effectiveness and universality of the&#10;proposed approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.00591" label="2301.00591">
        <attvalues>
          <attvalue for="0" value="Analysing Discrete Self Supervised Speech Representation for Spoken&#10;  Language Modeling" />
          <attvalue for="1" value="  This work profoundly analyzes discrete self-supervised speech representations&#10;(units) through the eyes of Generative Spoken Language Modeling (GSLM).&#10;Following the findings of such an analysis, we propose practical improvements&#10;to the discrete unit for the GSLM. First, we start comprehending these units by&#10;analyzing them in three axes: interpretation, visualization, and resynthesis.&#10;Our analysis finds a high correlation between the speech units to phonemes and&#10;phoneme families, while their correlation with speaker or gender is weaker.&#10;Additionally, we found redundancies in the extracted units and claim that one&#10;reason may be the units' context. Following this analysis, we propose a new,&#10;unsupervised metric to measure unit redundancies. Finally, we use this metric&#10;to develop new methods that improve the robustness of units' clustering and&#10;show significant improvement considering zero-resource speech metrics such as&#10;ABX. Code and analysis tools are available under the following link:&#10;https://github.com/slp-rl/SLM-Discrete-Representations&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;The general GSLM pipeline is comprised of three main modules: (i) Speech-to-unit, (ii) Unit language model, and (iii) Unit-to-speech, where each of these modules is trained separately. Speech resynthesis can be achieved while ignoring the language model and directly feeding the quantized units into the unit-to-speech module~\cite{polyak2021speech}&#10;&#10;{Speech To Unit (STU)} The model first encodes the raw speech signal into a continuous representation and then quantizes the representation to a sequence of discrete units ~\cite{lakhotia2021generative, kharitonov2022textless, gat2022robustness}.&#10;&#10;Formally, denote the domain of audio samples by $x \subset R$. Therefore, the representation for a raw signal is a sequence of samples $x = (x_1,\ldots, x_T)$, where $x_t\in x$ for all $1\leq t \leq T$. Consider an encoder network, $f$, that gets as input the speech utterance and outputs a sequence of spectral representations sampled at a low frequency as follows $f(x) = (v_1, \dots, v_{T'})$. Note that we do not assume anything about the structure of the encoder network $f$. &#10;Since the representations learned by such models are usually continuous, a k-means algorithm is applied over the models' outputs to generate discrete units, denoted as $z = (z_1,\ldots,z_{T'})$. Each element $z_i$ in $z$ is a positive integer, $z_i\in\{1,..,K\}$ for $1\le i \le T'$, where $K$ is the number of discrete units. &#10;&#10;As the quantized representation, $z$, usually contain units repetitions that degrade the performance of the language modeling, a common approach is to collapse repetitions and generate a de-duplicated sequence while additionally storing the units' duration separately. For instance, the sequence 12,12,25,31,31,31 will be converted into 12,25,31 and the corresponding durations 2,1,3.&#10;&#10;{Unit Language Model (ULM) } is trained on the extracted and deduplicated discrete units, $z$. The language model can be used, for example, to generate speech conditionally or unconditionally.&#10;&#10;{Unit To Speech} module converts the discrete speech representation, $z$, to a raw waveform. The authors in~\cite{lakhotia2021generative} used a Tacotron2.0~\cite{shen2018natural} based model followed by WaveGlow~\cite{prenger2019waveglow} vocoder. Later, \cite{polyak2021speech} proposed a unit-based vocoder based on the HiFi-GAN architecture to convert units to speech directly. In this work, we focus on the latter setting. &#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Speech Processing, Generative Language Modeling, Linguistics, Unsupervised Metric Development, Speech Representation Analysis, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.16810" label="2310.16810">
        <attvalues>
          <attvalue for="0" value="Can GPT models Follow Human Summarization Guidelines? Evaluating ChatGPT&#10;  and GPT-4 for Dialogue Summarization" />
          <attvalue for="1" value="  This study explores the capabilities of prompt-driven Large Language Models&#10;(LLMs) like ChatGPT and GPT-4 in adhering to human guidelines for dialogue&#10;summarization. Experiments employed DialogSum (English social conversations)&#10;and DECODA (French call center interactions), testing various prompts:&#10;including prompts from existing literature and those from human summarization&#10;guidelines, as well as a two-step prompt approach. Our findings indicate that&#10;GPT models often produce lengthy summaries and deviate from human summarization&#10;guidelines. However, using human guidelines as an intermediate step shows&#10;promise, outperforming direct word-length constraint prompts in some cases. The&#10;results reveal that GPT models exhibit unique stylistic tendencies in their&#10;summaries. While BERTScores did not dramatically decrease for GPT outputs&#10;suggesting semantic similarity to human references and specialised pre-trained&#10;models, ROUGE scores reveal grammatical and lexical disparities between&#10;GPT-generated and human-written summaries. These findings shed light on the&#10;capabilities and limitations of GPT models in following human instructions for&#10;dialogue summarization.&#10;" />
          <attvalue for="2" value="&#10;The emergence of prompt-driven Large Language Models (LLMs) has led to extensive investigations into their capabilities in a wide range of representative task categories \cite{laskar-etal-2023-systematic, qin2023chatgpt, bang2023multitask}, including summarization.&#10;The results of these studies consistently indicate that ChatGPT, in general, underperforms in summarization tasks compared to task-specific pre-trained models \cite{bang2023multitask, zhang2023extractive, yang2023exploring}. This is also the case for multilingual summarization \cite{wang2023zeroshot}. As for dialogue summarization, only \cite{qin2023chatgpt, laskar-etal-2023-systematic} studied ChatGPT's performance using SAMSum \cite{gliwa-etal-2019-samsum} and DialogSum \cite{chen-etal-2021-dialogsum}. &#10;&#10;Although LLMs have been tested in various representative task categories and with different prompts, we don't know their ability to follow complex instructions, particularly if they are able to follow what humans would receive in the real-world scenario to annotate data.&#10;The dialogue summarization task aims to extract important information from a dialogue and present it to users in the form of a complete and concise textual summary. Human annotators adhere to guidelines provided when creating reference summaries, making the task ideal for assessing the ability of language models (LLMs) to follow these complex instructions.&#10;&#10;While dialogue summarization is a well-established task, its task formulation is still subject to discussion in the linguistic and NLP communities, which has prevented a consensual definition of what a dialogue summarization should look like \cite{guo-etal-2022-questioning}.&#10;The choice of how dialogue summarization should be performed is influenced by specific communication objectives, which may differ among various scenarios such as meeting, customer service, or social conversations, etc.&#10;As a result, each dataset might have unique summarization requirements aligned with its distinct communication goals. &#10;A review of guidelines for several major dialogue summarization datasets reveals that these corpora outline their objectives for creating reference summaries, and that different summary criteria are applied to each corpus \cite{zhou2023evaluating}.&#10;For example, in the case of customer service corpora, while TWEETSUMM \cite{feigenblat-etal-2021-tweetsumm-dialog} offers both extractive and abstractive summaries, CSDS \cite{lin-etal-2021-csds} provides three distinct summaries for each dialogue: an overall summary and two role-oriented summaries (user and agent). It is therefore reasonable to use task-specific guidelines as instructions for LLMs.&#10;&#10;In this study, we aim to investigate the capacity of GPT models to effectively follow human summarization guidelines given to annotators to write reference summaries. To this end, we tested ChatGPT and GPT-4 with different prompts: from simplified instructions that proved effective in similar summarization tasks to instructions given to human annotators. &#10;A two-step prompting strategy was also used, taking human annotation instructions as an intermediate prompting step. We reported ROUGE and BERTScore, with experiments conducted on two datasets -- DialogSum \cite{chen-etal-2021-dialogsum} and DECODA \cite{favre-etal-2015-call}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Computational Linguistics, Human Guideline Adherence, Artificial Intelligence, Natural Language Processing, Dialogue Summarization" />
        </attvalues>
      </node>
      <node id="2302.14229" label="2302.14229">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Cross-Lingual Summarization via Large Language Models" />
          <attvalue for="1" value="  Given a document in a source language, cross-lingual summarization (CLS) aims&#10;to generate a summary in a different target language. Recently, the emergence&#10;of Large Language Models (LLMs), such as GPT-3.5, ChatGPT and GPT-4, has&#10;attracted wide attention from the computational linguistics community. However,&#10;it is not yet known the performance of LLMs on CLS. In this report, we&#10;empirically use various prompts to guide LLMs to perform zero-shot CLS from&#10;different paradigms (i.e., end-to-end and pipeline), and provide a preliminary&#10;evaluation on the generated summaries. We find that ChatGPT and GPT-4&#10;originally prefer to produce lengthy summaries with detailed information. These&#10;two LLMs can further balance informativeness and conciseness with the help of&#10;an interactive prompt, significantly improving their CLS performance.&#10;Experimental results on three widely-used CLS datasets show that GPT-4 achieves&#10;state-of-the-art zero-shot CLS performance, and performs competitively compared&#10;with the fine-tuned mBART-50. Moreover, we also find some multi-lingual and&#10;bilingual LLMs (i.e., BLOOMZ, ChatGLM-6B, Vicuna-13B and ChatYuan) have limited&#10;zero-shot CLS ability. Due to the composite nature of CLS, which requires&#10;models to perform summarization and translation simultaneously, accomplishing&#10;this task in a zero-shot manner is even a challenge for LLMs. Therefore, we&#10;sincerely hope and recommend future LLM research could use CLS as a testbed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1911.12237" label="1911.12237">
        <attvalues>
          <attvalue for="0" value="SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  This paper introduces the SAMSum Corpus, a new dataset with abstractive&#10;dialogue summaries. We investigate the challenges it poses for automated&#10;summarization by testing several models and comparing their results with those&#10;obtained on a corpus of news articles. We show that model-generated summaries&#10;of dialogues achieve higher ROUGE scores than the model-generated summaries of&#10;news -- in contrast with human evaluators' judgement. This suggests that a&#10;challenging task of abstractive dialogue summarization requires dedicated&#10;models and non-standard quality measures. To our knowledge, our study is the&#10;first attempt to introduce a high-quality chat-dialogues corpus, manually&#10;annotated with abstractive summarizations, which can be used by the research&#10;community for further studies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.06762" label="2105.06762">
        <attvalues>
          <attvalue for="0" value="DialogSum: A Real-Life Scenario Dialogue Summarization Dataset" />
          <attvalue for="1" value="  Proposal of large-scale datasets has facilitated research on deep neural&#10;models for news summarization. Deep learning can also be potentially useful for&#10;spoken dialogue summarization, which can benefit a range of real-life scenarios&#10;including customer service management and medication tracking. To this end, we&#10;propose DialogSum, a large-scale labeled dialogue summarization dataset. We&#10;conduct empirical analysis on DialogSum using state-of-the-art neural&#10;summarizers. Experimental results show unique challenges in dialogue&#10;summarization, such as spoken terms, special discourse structures, coreferences&#10;and ellipsis, pragmatics and social common sense, which require specific&#10;representation learning technologies to better deal with.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.17378" label="2210.17378">
        <attvalues>
          <attvalue for="0" value="Questioning the Validity of Summarization Datasets and Improving Their&#10;  Factual Consistency" />
          <attvalue for="1" value="  The topic of summarization evaluation has recently attracted a surge of&#10;attention due to the rapid development of abstractive summarization systems.&#10;However, the formulation of the task is rather ambiguous, neither the&#10;linguistic nor the natural language processing community has succeeded in&#10;giving a mutually agreed-upon definition. Due to this lack of well-defined&#10;formulation, a large number of popular abstractive summarization datasets are&#10;constructed in a manner that neither guarantees validity nor meets one of the&#10;most essential criteria of summarization: factual consistency. In this paper,&#10;we address this issue by combining state-of-the-art factual consistency models&#10;to identify the problematic instances present in popular summarization&#10;datasets. We release SummFC, a filtered summarization dataset with improved&#10;factual consistency, and demonstrate that models trained on this dataset&#10;achieve improved performance in nearly all quality aspects. We argue that our&#10;dataset should become a valid benchmark for developing and evaluating&#10;summarization systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.13139" label="2108.13139">
        <attvalues>
          <attvalue for="0" value="CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue&#10;  Summarization" />
          <attvalue for="1" value="  Dialogue summarization has drawn much attention recently. Especially in the&#10;customer service domain, agents could use dialogue summaries to help boost&#10;their works by quickly knowing customer's issues and service progress. These&#10;applications require summaries to contain the perspective of a single speaker&#10;and have a clear topic flow structure, while neither are available in existing&#10;datasets. Therefore, in this paper, we introduce a novel Chinese dataset for&#10;Customer Service Dialogue Summarization (CSDS). CSDS improves the abstractive&#10;summaries in two aspects: (1) In addition to the overall summary for the whole&#10;dialogue, role-oriented summaries are also provided to acquire different&#10;speakers' viewpoints. (2) All the summaries sum up each topic separately, thus&#10;containing the topic-level structure of the dialogue. We define tasks in CSDS&#10;as generating the overall summary and different role-oriented summaries for a&#10;given dialogue. Next, we compare various summarization methods on CSDS, and&#10;experiment results show that existing methods are prone to generate redundant&#10;and incoherent summaries. Besides, the performance becomes much worse when&#10;analyzing the performance on role-oriented summaries and topic structures. We&#10;hope that this study could benchmark Chinese dialogue summarization and benefit&#10;further studies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08200" label="2104.08200">
        <attvalues>
          <attvalue for="0" value="IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural&#10;  Language Generation" />
          <attvalue for="1" value="  Natural language generation (NLG) benchmarks provide an important avenue to&#10;measure progress and develop better NLG systems. Unfortunately, the lack of&#10;publicly available NLG benchmarks for low-resource languages poses a&#10;challenging barrier for building NLG systems that work well for languages with&#10;limited amounts of data. Here we introduce IndoNLG, the first benchmark to&#10;measure natural language generation (NLG) progress in three low-resource -- yet&#10;widely spoken -- languages of Indonesia: Indonesian, Javanese, and Sundanese.&#10;Altogether, these languages are spoken by more than 100 million native&#10;speakers, and hence constitute an important use case of NLG systems today.&#10;Concretely, IndoNLG covers six tasks: summarization, question answering,&#10;chit-chat, and three different pairs of machine translation (MT) tasks. We&#10;collate a clean pretraining corpus of Indonesian, Sundanese, and Javanese&#10;datasets, Indo4B-Plus, which is used to pretrain our models: IndoBART and&#10;IndoGPT. We show that IndoBART and IndoGPT achieve competitive performance on&#10;all tasks -- despite using only one-fifth the parameters of a larger&#10;multilingual model, mBART-LARGE (Liu et al., 2020). This finding emphasizes the&#10;importance of pretraining on closely related, local languages to achieve more&#10;efficient learning and faster inference for very low-resource languages like&#10;Javanese and Sundanese.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;NLP Benchmarks.&#10;Numerous benchmarks have recently emerged, which have catalyzed advances in monolingual and cross-lingual transfer learning. These include NLU benchmarks for low-resource languages including IndoNLU~\cite{wilie2020indonlu}, IndoLEM~\cite{koto-etal-2020-indolem}, and those focusing on Filipino~\cite{cruz2020establishing}, Bangla~\cite{bhattacharjee2021banglabert}, and Thai~\cite{lowphansirikul2021wangchanberta}; neural machine translation (MT) datasets for low-resource scenarios including for Indonesian \cite{guntara2020benchmarking}, African languages \cite{duh-etal-2020-benchmarking,lakew2020low}, and Nepali and Sinhala \cite{guzman2019flores}; and large-scale multilingual benchmarks such as XTREME \cite{Hu2020xtreme}, MTOP \cite{li2020mtop}, and XGLUE \cite{Liang2020xglue}.&#10;\cite{winata2021multilingual,aguilar2020lince,khanuja2020gluecos} further developed multilingual benchmarks to evaluate the effectiveness of pretrained multilingual language models. More recently, GEM \cite{gehrmann2021gem} covers NLG tasks in various languages, together with automated and human evaluation metrics. Our benchmark compiles languages and tasks that are not covered in those prior work, such as local multilingual (Indonesian, Javanese, Sundanese, and English) MT tasks, Indonesian summarization, and Indonesian chit-chat dialogue.&#10;Pretrained NLG Models.&#10;Recently, the paradigm of pretraining-then-fine-tuning has achieved remarkable success in NLG, as evidenced by the success of monolingual pretrained NLG models. GPT-2 \cite{radford2019language}, and later GPT-3 \cite{NEURIPS2020_1457c0d6}, demonstrated that language models can perform zero-shot transfer to downstream tasks via generation. Other recent state-of-the-art models are BART \cite{lewis2020bart}, which maps corrupted documents to their original, and the encoder-decoder T5 \cite{raffel2020exploring}, which resulted from a thorough investigation of architectures, objectives, datasets, and pretraining strategies. These monolingual models have been generalised to the multilingual case by pretraining the architectures on multiple languages; examples include mBART~\cite{liu2020mbart} and mT5 \cite{xue2020mt5}. In this paper, we focus on local, near-monolingual models for the languages of Indonesia, and systematically compare them on our benchmark with such larger multilingual models.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Modeling, Low-Resource Languages, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1810.00438" label="1810.00438">
        <attvalues>
          <attvalue for="0" value="Parameter-free Sentence Embedding via Orthogonal Basis" />
          <attvalue for="1" value="  We propose a simple and robust non-parameterized approach for building&#10;sentence representations. Inspired by the Gram-Schmidt Process in geometric&#10;theory, we build an orthogonal basis of the subspace spanned by a word and its&#10;surrounding context in a sentence. We model the semantic meaning of a word in a&#10;sentence based on two aspects. One is its relatedness to the word vector&#10;subspace already spanned by its contextual words. The other is the word's novel&#10;semantic meaning which shall be introduced as a new basis vector perpendicular&#10;to this existing subspace. Following this motivation, we develop an innovative&#10;method based on orthogonal basis to combine pre-trained word embeddings into&#10;sentence representations. This approach requires zero parameters, along with&#10;efficient inference performance. We evaluate our approach on 11 downstream NLP&#10;tasks. Our model shows superior performance compared with non-parameterized&#10;alternatives and it is competitive to other approaches relying on either large&#10;amounts of labelled data or prolonged training time.&#10;" />
          <attvalue for="2" value="&#10;The concept of word embeddings has been prevalent in NLP community in recent years, as they can characterize semantic similarity between any pair of words, achieving promising results in a large number of NLP tasks \cite{mikolov2013distributed, pennington2014glove, Salle2016MatrixFU}. However, due to the hierarchical nature of human language, it is not sufficient to comprehend text solely based on isolated understanding of each word. This has prompted a recent rise in search for semantically robust embeddings for longer pieces of text, such as sentences and paragraphs.&#10;&#10;Based on learning paradigms, the existing approaches to sentence embeddings can be categorized into two categories: i) parameterized methods and ii) non-parameterized methods. &#10;&#10;Parameterized sentence embeddings. These models are parameterized and require training to optimize their parameters. SkipThought \cite{kiros2015skip} is an encoder-decoder model that predicts adjacent sentences. \cite{pgj2017unsup} proposes an unsupervised model, Sent2Vec, to learn an n-gram feature in a sentence to predict the center word from the surrounding context. Quick thoughts (QT) \cite{logeswaran2018efficient} replaces the encoder with a classifier to predict context sentences from candidate sequences. \cite{khodak2018carte} proposes $\grave{a}\: la \: carte$ to learn a linear mapping to reconstruct the center word from its context. \cite{conneau-EtAl:2017:EMNLP2017} generates the sentence encoder InferSent using Natural Language Inference (NLI) dataset. Universal Sentence Encoder \cite{2018arXiv180407754Y, cer2018universal} utilizes the emerging transformer structure \cite{vaswani2017attention, devlin2018bert} that has been proved powerful in various NLP tasks. The model is first trained on large scale of unsupervised data from Wikipedia and forums, and then trained on the Stanford Natural Language Inference (SNLI) dataset. \cite{wieting-17-recurrent} propose the gated recurrent averaging network (GRAN), which is trained on Paraphrase Database (PPDB) and English Wikipedia. \cite{subramanian2018learning} leverages a multi-task learning framework to generate sentence embeddings. \cite{wieting2016iclr} learns the paraphrastic sentence representations as the simple average of updated word embeddings.&#10;&#10;Non-parameterized sentence embedding. Recent work \cite{arora2017asimple} shows that, surprisingly, a weighted sum or transformation of word representations can outperform many sophisticated neural network structures in sentence embedding tasks. These methods are parameter-free and require no further training upon pre-trained word vectors. \cite{arora2017asimple} constructs a sentence embedding called SIF as a sum of pre-trained word embeddings, weighted by reverse document frequency. \cite{ethayarajh2018unsupervised} builds upon the random walk model proposed in SIF by setting the probability of word generation inversely related to the angular distance between the word and sentence embeddings. \cite{2018arXiv180301400R} concatenates different power mean word embeddings as a sentence vector in $p$-mean. As these methods do not have a parameterized model, they can be easily adapted to novel text domains with both fast inference speed and high-quality sentence embeddings. In view of this trend, our work aims to further advance the frontier of this group and make its new state-of-the-art.&#10;&#10;In this paper, we propose a novel sentence embedding algorithm, Geometric Embedding (GEM), based entirely on the geometric structure of word embedding space. Given a $d$-dim word embedding matrix $\mA\in\mathbb{R}^{d\times n}$ for a sentence with $n$ words, any linear combination of the sentence's word embeddings lies in the subspace spanned by the $n$ word vectors. We analyze the geometric structure of this subspace in $\mathbb{R}^{d}$. When we consider the words in a sentence one-by-one in order, each word may bring in a novel orthogonal basis to the existing subspace. This new basis can be considered as the new semantic meaning brought in by this word, while the length of projection in this direction can indicate the intensity of this new meaning. It follows that a word with a strong intensity should have a larger influence in the sentence's meaning. Thus, these intensities can be converted into weights to linearly combine all word embeddings to obtain the sentence embedding. In this paper, we theoretically frame the above approach in a QR factorization of the word embedding matrix $\mA$.&#10;Furthermore, since the meaning and importance of a word largely depends on its close neighborhood, we propose the sliding-window QR factorization method to capture the context of a word and characterize its significance within the context. &#10;&#10;In the last step, we adapt a similar approach as \cite{arora2017asimple} to remove top principal vectors before generating the final sentence embedding. This step is to ensure commonly shared background components, e.g. stop words, do not bias sentence similarity comparison. As we build a new orthogonal basis for each sentence, we propose to have disparate background components for each sentence. This motivates us to put forward a sentence-specific principal vector removal method, leading to better empirical results.&#10;&#10;We evaluate our algorithm on 11 NLP tasks. Our algorithm outperforms all non-parameterized methods and many parameterized approaches in 10 tasks. Compared to SIF \cite{arora2017asimple}, the performance is boosted by 5.5\% on STS benchmark dataset, and by 2.5\% on SST dataset. Plus, the running time of our model compares favorably with existing models. &#10;&#10;The rest of this paper is organized as following. In \ref{Aprc}, we describe our sentence embedding algorithm GEM. We evaluate our model on various tasks in \ref{exp} and \ref{discussion}. Finally, we summarize our work in \ref{summary}. Our implementation is available online. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Word Embeddings, Sentence Representation, Linear Algebra, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1703.02507" label="1703.02507">
        <attvalues>
          <attvalue for="0" value="Unsupervised Learning of Sentence Embeddings using Compositional n-Gram&#10;  Features" />
          <attvalue for="1" value="  The recent tremendous success of unsupervised word embeddings in a multitude&#10;of applications raises the obvious question if similar methods could be derived&#10;to improve embeddings (i.e. semantic representations) of word sequences as&#10;well. We present a simple but efficient unsupervised objective to train&#10;distributed representations of sentences. Our method outperforms the&#10;state-of-the-art unsupervised models on most benchmark tasks, highlighting the&#10;robustness of the produced general-purpose sentence embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.05388" label="1805.05388">
        <attvalues>
          <attvalue for="0" value="A La Carte Embedding: Cheap but Effective Induction of Semantic Feature&#10;  Vectors" />
          <attvalue for="1" value="  Motivations like domain adaptation, transfer learning, and feature learning&#10;have fueled interest in inducing embeddings for rare or unseen words, n-grams,&#10;synsets, and other textual features. This paper introduces a la carte&#10;embedding, a simple and general alternative to the usual word2vec-based&#10;approaches for building such representations that is based upon recent&#10;theoretical results for GloVe-like embeddings. Our method relies mainly on a&#10;linear transformation that is efficiently learnable using pretrained word&#10;vectors and linear regression. This transform is applicable on the fly in the&#10;future when a new text feature or rare word is encountered, even if only a&#10;single usage example is available. We introduce a new dataset showing how the a&#10;la carte method requires fewer examples of words in context to learn&#10;high-quality embeddings and we obtain state-of-the-art results on a nonce task&#10;and some unsupervised document classification tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.07754" label="1804.07754">
        <attvalues>
          <attvalue for="0" value="Learning Semantic Textual Similarity from Conversations" />
          <attvalue for="1" value="  We present a novel approach to learn representations for sentence-level&#10;semantic similarity using conversational data. Our method trains an&#10;unsupervised model to predict conversational input-response pairs. The&#10;resulting sentence embeddings perform well on the semantic textual similarity&#10;(STS) benchmark and SemEval 2017's Community Question Answering (CQA) question&#10;similarity subtask. Performance is further improved by introducing multitask&#10;training combining the conversational input-response prediction task and a&#10;natural language inference task. Extensive experiments show the proposed model&#10;achieves the best performance among all neural models on the STS benchmark and&#10;is competitive with the state-of-the-art feature engineered and mixed systems&#10;in both tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.00364" label="1705.00364">
        <attvalues>
          <attvalue for="0" value="Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings" />
          <attvalue for="1" value="  We consider the problem of learning general-purpose, paraphrastic sentence&#10;embeddings, revisiting the setting of Wieting et al. (2016b). While they found&#10;LSTM recurrent networks to underperform word averaging, we present several&#10;developments that together produce the opposite conclusion. These include&#10;training on sentence pairs rather than phrase pairs, averaging states to&#10;represent sequences, and regularizing aggressively. These improve LSTMs in both&#10;transfer learning and supervised settings. We also introduce a new recurrent&#10;architecture, the Gated Recurrent Averaging Network, that is inspired by&#10;averaging and LSTMs while outperforming them both. We analyze our learned&#10;models, finding evidence of preferences for particular parts of speech and&#10;dependency relations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.06118" label="1911.06118">
        <attvalues>
          <attvalue for="0" value="Learning Multi-Sense Word Distributions using Approximate&#10;  Kullback-Leibler Divergence" />
          <attvalue for="1" value="  Learning word representations has garnered greater attention in the recent&#10;past due to its diverse text applications. Word embeddings encapsulate the&#10;syntactic and semantic regularities of sentences. Modelling word embedding as&#10;multi-sense gaussian mixture distributions, will additionally capture&#10;uncertainty and polysemy of words. We propose to learn the Gaussian mixture&#10;representation of words using a Kullback-Leibler (KL) divergence based&#10;objective function. The KL divergence based energy function provides a better&#10;distance metric which can effectively capture entailment and distribution&#10;similarity among the words. Due to the intractability of KL divergence for&#10;Gaussian mixture, we go for a KL approximation between Gaussian mixtures. We&#10;perform qualitative and quantitative experiments on benchmark word similarity&#10;and entailment datasets which demonstrate the effectiveness of the proposed&#10;approach.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language modelling in its inception had one-hot vector encoding of words. However, it captures only alphabetic ordering but not the word semantic similarity. Vector space models helps to learn word representations in a lower dimensional space and also captures semantic similarity. Learning word embedding aids in natural language processing tasks such as question answering and reasoning \cite{choi2018quac}, stance detection \cite{augenstein2016stance}, claim verification \cite{hanselowski2018ukp}.&#10;&#10;Recent models \cite{mikolov2013efficient,bengio2003neural} work on the basis that words with similar context share semantic similarity. \cite{bengio2003neural} proposes a neural probabilistic model which models the target word probability conditioned on the previous words using a recurrent neural network. Word2Vec models \cite{mikolov2013efficient} such as continuous bag-of-words (CBOW) predict the target word given the context, and skip-gram model works in reverse of predicting the context given the target word. While, GloVe embeddings were based on a Global matrix factorization on local contexts \cite{pennington2014glove}. However, the aforementioned models do not handle words with multiple meanings (polysemies).&#10;&#10;\cite{huang2012improving} proposes a neural network approach considering both local and global contexts in learning word embeddings (point estimates). Their multiple prototype model handles polysemous words by providing apriori heuristics about word senses in the dataset. \cite{tian2014probabilistic} proposes an alternative to handle polysemous words by a modified skip-gram model and EM algorithm. \cite{neelakantan2015efficient} presents a non-parametric based alternative to handle polysemies. &#10;However, these approaches fail to consider entailment relations among the words.&#10;&#10;\cite{vilnis2014word} learn a Gaussian distribution per word using the expected likelihood kernel. However, for polysemous words, this may lead to word distributions with larger variances as it may have to cover various senses. &#10;&#10;\cite{athiwaratkun2017multimodal} proposes multimodal word distribution approach. It captures polysemy. However, the energy based objective function fails to consider asymmetry and hence entailment. Textual entailment recognition is necessary to capture lexical inference relations such as causality (for example, mosquito $arrow$ malaria), hypernymy (for example, dog $\models$ animal) etc.&#10;&#10;In this paper, we propose to obtain multi-sense word embedding distributions by using a variant of max margin objective based on the asymmetric KL divergence energy function to capture textual entailment. Multi-sense distributions are advantageous in capturing polysemous nature of words and in reducing the uncertainty per word by distributing it across senses. However, computing KL divergence between mixtures of Gaussians is intractable, and we use a KL divergence approximation based on stricter upper and lower bounds. While capturing textual entailment (asymmetry), we have also not compromised on capturing symmetrical similarity between words (for example, funny and hilarious) which will be elucidated in Section $3.1$. We also show the effectiveness of the proposed approach on the benchmark word similarity and entailment datasets in the experimental section.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Word Embeddings, Gaussian Mixture Models, Mathematics, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1606.05464" label="1606.05464">
        <attvalues>
          <attvalue for="0" value="Stance Detection with Bidirectional Conditional Encoding" />
          <attvalue for="1" value="  Stance detection is the task of classifying the attitude expressed in a text&#10;towards a target such as Hillary Clinton to be &quot;positive&quot;, negative&quot; or&#10;&quot;neutral&quot;. Previous work has assumed that either the target is mentioned in the&#10;text or that training data for every target is given. This paper considers the&#10;more challenging version of this task, where targets are not always mentioned&#10;and no training data is available for the test targets. We experiment with&#10;conditional LSTM encoding, which builds a representation of the tweet that is&#10;dependent on the target, and demonstrate that it outperforms encoding the tweet&#10;and the target independently. Performance is improved further when the&#10;conditional model is augmented with bidirectional encoding. We evaluate our&#10;approach on the SemEval 2016 Task 6 Twitter Stance Detection corpus achieving&#10;performance second best only to a system trained on semi-automatically labelled&#10;tweets for the test target. When such weak supervision is added, our approach&#10;achieves state-of-the-art results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.01479" label="1809.01479">
        <attvalues>
          <attvalue for="0" value="UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification" />
          <attvalue for="1" value="  The Fact Extraction and VERification (FEVER) shared task was launched to&#10;support the development of systems able to verify claims by extracting&#10;supporting or refuting facts from raw text. The shared task organizers provide&#10;a large-scale dataset for the consecutive steps involved in claim verification,&#10;in particular, document retrieval, fact extraction, and claim classification.&#10;In this paper, we present our claim verification pipeline approach, which,&#10;according to the preliminary results, scored third in the shared task, out of&#10;23 competing systems. For the document retrieval, we implemented a new entity&#10;linking approach. In order to be able to rank candidate facts and classify a&#10;claim on the basis of several selected facts, we introduce two extensions to&#10;the Enhanced LSTM (ESIM).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1504.06654" label="1504.06654">
        <attvalues>
          <attvalue for="0" value="Efficient Non-parametric Estimation of Multiple Embeddings per Word in&#10;  Vector Space" />
          <attvalue for="1" value="  There is rising interest in vector-space word embeddings and their use in&#10;NLP, especially given recent methods for their fast estimation at very large&#10;scale. Nearly all this work, however, assumes a single vector per word type&#10;ignoring polysemy and thus jeopardizing their usefulness for downstream tasks.&#10;We present an extension to the Skip-gram model that efficiently learns multiple&#10;embeddings per word type. It differs from recent related work by jointly&#10;performing word sense discrimination and embedding learning, by&#10;non-parametrically estimating the number of senses per word type, and by its&#10;efficiency and scalability. We present new state-of-the-art results in the word&#10;similarity in context task and demonstrate its scalability by training with one&#10;machine on a corpus of nearly 1 billion tokens in less than 6 hours.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.08424" label="1704.08424">
        <attvalues>
          <attvalue for="0" value="Multimodal Word Distributions" />
          <attvalue for="1" value="  Word embeddings provide point representations of words containing useful&#10;semantic information. We introduce multimodal word distributions formed from&#10;Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty&#10;information. To learn these distributions, we propose an energy-based&#10;max-margin objective. We show that the resulting approach captures uniquely&#10;expressive semantic information, and outperforms alternatives, such as word2vec&#10;skip-grams, and Gaussian embeddings, on benchmark datasets such as word&#10;similarity and entailment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.11443" label="2408.11443">
        <attvalues>
          <attvalue for="0" value="Distributional Properties of Subword Regularization" />
          <attvalue for="1" value="  Subword regularization, used widely in NLP, improves model performance by&#10;reducing the dependency on exact tokenizations, augmenting the training corpus,&#10;and exposing the model to more unique contexts during training. BPE and&#10;MaxMatch, two popular subword tokenization schemes, have stochastic dropout&#10;regularization variants. However, there has not been an analysis of the&#10;distributions formed by them. We show that these stochastic variants are&#10;heavily biased towards a small set of tokenizations per word. If the benefits&#10;of subword regularization are as mentioned, we hypothesize that biasedness&#10;artificially limits the effectiveness of these schemes. Thus, we propose an&#10;algorithm to uniformly sample tokenizations that we use as a drop-in&#10;replacement for the stochastic aspects of existing tokenizers, and find that it&#10;improves machine translation quality.&#10;" />
          <attvalue for="2" value="&#10;&#10;Tokenization is the first stage in almost all natural language processing pipelines, where raw text is transformed into a format that is understood by the model.&#10;Modern neural models use subword tokenization, which represents text as a sequence of subword units drawn from a subword vocabulary (e.g., decompositional $arrow$ de composition al). &#10;Popular subword tokenization schemes are BPE \cite{sennrich-etal-2016-neural}, MaxMatch/WordPiece \cite{wu2016googles}, and UnigramLM \cite{kudo-2018-subword}.&#10;Unintentionally, the downstream models are thus not conditioned on the raw text, but rather the exact tokenization of the text.&#10;During training, subword regularization (where static tokenizations are replaced with sampled tokenizations) is often used to break the dependency on the exact tokenization.&#10;It also serves as data augmentation, and improves performance in a variety of downstream tasks.&#10;&#10;There are two main types of stochastic tokenizers: those which learn a distribution from text (e.g., UnigramLM) and those which inject randomness by corrupting the tokenization scheme (e.g., BPE-Dropout, \cite{provilkov-etal-2020-bpe} and MaxMatch-Dropout, \cite{hiraoka-2022-maxmatch}).&#10;In our work, we focus on the latter, for which no prior study of the resulting distributions exists.&#10;BPE- and MaxMatch-Dropout add randomness post hoc into the underlying deterministic tokenization, and the distributions they produce are essentially unrelated to the text distribution.&#10;We find that these distributions are heavily biased, in that they do not produce uniform tokenization distributions (see \ref{ex:bpe_dropout_20}).&#10;&#10;Despite them working well in practice, there is no reason to believe that the distributions formed by BPE- and MaxMatch-Dropout are ``good'' for training.&#10;However, there are reasons to believe that a different strategy, uniform sampling, would be better for training, as it would increase the amount of regularization and augmentation injected into the training process.&#10;We experiment with replacing the stochastic aspects of BPE- and MaxMatch-Dropout with one which samples uniformly at random from all possible tokenizations, and find that it improves modeling quality on several translation tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Machine Translation, Linguistics, Stochastic Regularization, Mathematics, Subword Tokenization, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1910.13267" label="1910.13267">
        <attvalues>
          <attvalue for="0" value="BPE-Dropout: Simple and Effective Subword Regularization" />
          <attvalue for="1" value="  Subword segmentation is widely used to address the open vocabulary problem in&#10;machine translation. The dominant approach to subword segmentation is Byte Pair&#10;Encoding (BPE), which keeps the most frequent words intact while splitting the&#10;rare ones into multiple tokens. While multiple segmentations are possible even&#10;with the same vocabulary, BPE splits words into unique sequences; this may&#10;prevent a model from better learning the compositionality of words and being&#10;robust to segmentation errors. So far, the only way to overcome this BPE&#10;imperfection, its deterministic nature, was to create another subword&#10;segmentation algorithm (Kudo, 2018). In contrast, we show that BPE itself&#10;incorporates the ability to produce multiple segmentations of the same word. We&#10;introduce BPE-dropout - simple and effective subword regularization method&#10;based on and compatible with conventional BPE. It stochastically corrupts the&#10;segmentation procedure of BPE, which leads to producing multiple segmentations&#10;within the same fixed BPE framework. Using BPE-dropout during training and the&#10;standard BPE during inference improves translation quality up to 3 BLEU&#10;compared to BPE and up to 0.9 BLEU compared to the previous subword&#10;regularization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.04126" label="2209.04126">
        <attvalues>
          <attvalue for="0" value="MaxMatch-Dropout: Subword Regularization for WordPiece" />
          <attvalue for="1" value="  We present a subword regularization method for WordPiece, which uses a&#10;maximum matching algorithm for tokenization. The proposed method,&#10;MaxMatch-Dropout, randomly drops words in a search using the maximum matching&#10;algorithm. It realizes finetuning with subword regularization for popular&#10;pretrained language models such as BERT-base. The experimental results&#10;demonstrate that MaxMatch-Dropout improves the performance of text&#10;classification and machine translation tasks as well as other subword&#10;regularization methods. Moreover, we provide a comparative analysis of subword&#10;regularization methods: subword regularization with SentencePiece (Unigram),&#10;BPE-Dropout, and MaxMatch-Dropout.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.14146" label="2204.14146">
        <attvalues>
          <attvalue for="0" value="Training Language Models with Language Feedback" />
          <attvalue for="1" value="  Pretrained language models often do not perform tasks in ways that are in&#10;line with our preferences, e.g., generating offensive text or factually&#10;incorrect summaries. Recent work approaches the above issue by learning from a&#10;simple form of human evaluation: comparisons between pairs of model-generated&#10;task outputs. Comparison feedback conveys limited information about human&#10;preferences per human evaluation. Here, we propose to learn from natural&#10;language feedback, which conveys more information per human evaluation. We&#10;learn from language feedback on model outputs using a three-step learning&#10;algorithm. First, we condition the language model on the initial output and&#10;feedback to generate many refinements. Second, we choose the refinement with&#10;the highest similarity to the feedback. Third, we finetune a language model to&#10;maximize the likelihood of the chosen refinement given the input. In synthetic&#10;experiments, we first evaluate whether language models accurately incorporate&#10;feedback to produce refinements, finding that only large language models (175B&#10;parameters) do so. Using only 100 samples of human-written feedback, our&#10;learning algorithm finetunes a GPT-3 model to roughly human-level summarization&#10;ability.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language Models (LMs) achieve strong performance across diverse NLP tasks, from summarization to question answering and conversational assistants \cite[][interalia]{Radford2018ImprovingLU,Radford2019LanguageMA,brown2020language, rae2021scaling}. A key problem with LMs is that they generate text that violates human preferences, such as LM-generated misinformation \cite{lin2021truthfulqa}, offensive language \cite{gehman2020realtoxicityprompts}, and factually incorrect outputs such as summaries \cite{stiennon2020learning}.&#10;Current methods alleviate such issues by training LMs to generate text that scores highly according to human preferences, or a predictive model thereof~\cite{ziegler2019fine,stiennon2020learning, nakano2021webgpt,ouyang2022training}. In this line of work, human evaluators indicate their preferences by comparing text outputs. However, each comparison provides little information per evaluation about human preferences.&#10;&#10;We propose to use natural language feedback, which contains more information per evaluation. We introduce a three-step learning algorithm, as shown in \ref{fig:illustration}. First, we condition an LM on an input, model-generated output, and human-written feedback to sample many possible refinements of the output. Second, we choose the refinement with the highest embedding-based similarity with the feedback. Third, we finetune an LM on the chosen refinements. Our algorithm departs from prior work, which uses reinforcement learning methods~\cite[][inter alia]{ziegler2019fine} or auxiliary losses~\cite{stacey2021natural} that cannot be straightforwardly generalized to using natural language feedback.&#10;&#10;We validate our algorithm on a carefully-controlled synthetic task of removing offensive words from a sentence with GPT-3-based models~\cite{brown2020language,ouyang2022training}.&#10;We find that only the largest GPT-3-based models (175B parameters) accurately refine outputs.&#10;Using the above insight, we use the largest GPT-3 models to test our algorithm on text summarization, following~\cite{stiennon2020learning}.&#10;A model trained with our algorithm generates summaries that human evaluators prefer to human reference summaries $\sim$51\% of the time.&#10;We obtain these results when learning from only $100$ samples of natural language feedback.&#10;Our analysis shows that LM-generated refinements typically incorporate the feedback, especially when choosing the refinement with the highest similarity with the feedback.&#10;Our results suggest that natural language feedback is a promising avenue for learning from human preferences.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Natural Language Processing, Human Feedback Learning" />
        </attvalues>
      </node>
      <node id="2104.08142" label="2104.08142">
        <attvalues>
          <attvalue for="0" value="Supervising Model Attention with Human Explanations for Robust Natural&#10;  Language Inference" />
          <attvalue for="1" value="  Natural Language Inference (NLI) models are known to learn from biases and&#10;artefacts within their training data, impacting how well they generalise to&#10;other unseen datasets. Existing de-biasing approaches focus on preventing the&#10;models from learning these biases, which can result in restrictive models and&#10;lower performance. We instead investigate teaching the model how a human would&#10;approach the NLI task, in order to learn features that will generalise better&#10;to previously unseen examples. Using natural language explanations, we&#10;supervise the model's attention weights to encourage more attention to be paid&#10;to the words present in the explanations, significantly improving model&#10;performance. Our experiments show that the in-distribution improvements of this&#10;method are also accompanied by out-of-distribution improvements, with the&#10;supervised models learning from features that generalise better to other NLI&#10;datasets. Analysis of the model indicates that human explanations encourage&#10;increased attention on the important words, with more attention paid to words&#10;in the premise and less attention paid to punctuation and stop-words.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.12559" label="1911.12559">
        <attvalues>
          <attvalue for="0" value="KPTimes: A Large-Scale Dataset for Keyphrase Generation on News&#10;  Documents" />
          <attvalue for="1" value="  Keyphrase generation is the task of predicting a set of lexical units that&#10;conveys the main content of a source text. Existing datasets for keyphrase&#10;generation are only readily available for the scholarly domain and include&#10;non-expert annotations. In this paper we present KPTimes, a large-scale dataset&#10;of news texts paired with editor-curated keyphrases. Exploring the dataset, we&#10;show how editors tag documents, and how their annotations differ from those&#10;found in existing datasets. We also train and evaluate state-of-the-art neural&#10;keyphrase generation models on KPTimes to gain insights on how well they&#10;perform on the news domain. The dataset is available online at&#10;https://github.com/ygorg/KPTimes .&#10;" />
          <attvalue for="2" value="&#10;&#10;Keyphrases are single or multi-word lexical units that best summarise a document~\cite{evans-zhai:1996:ACL}.&#10;As such, they are of great importance for indexing, categorising and browsing digital libraries~\cite{witten2009build}.&#10;Yet, very few documents have keyphrases assigned, thus raising the need for automatic keyphrase generation systems.&#10;This task falls under the task of automatic keyphrase extraction which can also be the subtask of finding keyphrases that only appear in the input document.&#10;Generating keyphrases can be seen as a particular instantiation of text summarization, where the goal is not to produce a well-formed piece of text, but a coherent set of phrases that convey the most salient information.&#10;Those phrases may or may not appear in the document, the latter requiring some form of abstraction to be generated.&#10;State-of-the-art systems for this task rely on recurrent neural networks~\cite{P17-1054,chen-EtAl:2018:EMNLP9,chen-etal-2019-integrated}, and hence require large amounts of annotated training data to achieve good performance.&#10;As gold annotated data is expensive and difficult to obtain~\cite{Mao2017}, previous works focused on readily available scientific abstracts and used author-assigned keyphrases as a proxy for expert annotations.&#10;However, this poses two major issues:&#10;1)~neural models for keyphrase generation do not generalize well across domains, thus limiting their use in practice;&#10;2)~author-assigned keyphrases exhibit strong consistency issues that negatively impacts the model's performance.&#10;There is therefore a great need for annotated data from different sources, that is both sufficiently large to support the training of neural-based models and that comprises gold-standard labels provided by experts.&#10;In this study, we address this need by providing \dataset, a dataset made of 279\,923 news articles that comes with editor-assigned keyphrases.&#10;&#10;Online news are particularly relevant to keyphrase generation since they are a natural fit for faceted navigation~\cite{tunkelang2009faceted} or topic detection and tracking~\cite{allan2012topic}.&#10;Also, and not less importantly, they are available in large quantities and are sometimes accompanied by metadata containing human-assigned keyphrases initially intended for search engines.&#10;Here, we divert these annotations from their primary purpose, and use them as gold-standard labels to automatically build our dataset.&#10;More precisely, we collect data by crawling selected news websites and use heuristics to draw texts paired with gold keyphrases. We then explore the resulting dataset to better understand how editors tag documents, and how these expert annotations differ from author-assigned keyphrases found in scholarly documents.&#10;Finally, we analyse the performance of state-of-the-art keyphrase generation models and investigate their transferability to the news domain and the impact of domain shift.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Keyphrase Generation, Dataset Creation, Linguistics, Information Science, Natural Language Processing, Neural Modeling" />
        </attvalues>
      </node>
      <node id="1704.06879" label="1704.06879">
        <attvalues>
          <attvalue for="0" value="Deep Keyphrase Generation" />
          <attvalue for="1" value="  Keyphrase provides highly-condensed information that can be effectively used&#10;for understanding, organizing and retrieving text content. Though previous&#10;studies have provided many workable solutions for automated keyphrase&#10;extraction, they commonly divided the to-be-summarized content into multiple&#10;text chunks, then ranked and selected the most meaningful ones. These&#10;approaches could neither identify keyphrases that do not appear in the text,&#10;nor capture the real semantic meaning behind the text. We propose a generative&#10;model for keyphrase prediction with an encoder-decoder framework, which can&#10;effectively overcome the above drawbacks. We name it as deep keyphrase&#10;generation since it attempts to capture the deep semantic meaning of the&#10;content with a deep learning method. Empirical analysis on six datasets&#10;demonstrates that our proposed model not only achieves a significant&#10;performance boost on extracting keyphrases that appear in the source text, but&#10;also can generate absent keyphrases based on the semantic meaning of the text.&#10;Code and dataset are available at&#10;https://github.com/memray/OpenNMT-kpg-release.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.07185" label="1808.07185">
        <attvalues>
          <attvalue for="0" value="Keyphrase Generation with Correlation Constraints" />
          <attvalue for="1" value="  In this paper, we study automatic keyphrase generation. Although conventional&#10;approaches to this task show promising results, they neglect correlation among&#10;keyphrases, resulting in duplication and coverage issues. To solve these&#10;problems, we propose a new sequence-to-sequence architecture for keyphrase&#10;generation named CorrRNN, which captures correlation among multiple keyphrases&#10;in two ways. First, we employ a coverage vector to indicate whether the word in&#10;the source document has been summarized by previous phrases to improve the&#10;coverage for keyphrases. Second, preceding phrases are taken into account to&#10;eliminate duplicate phrases and improve result coherence. Experiment results&#10;show that our model significantly outperforms the state-of-the-art method on&#10;benchmark datasets in terms of both accuracy and diversity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.03454" label="1904.03454">
        <attvalues>
          <attvalue for="0" value="An Integrated Approach for Keyphrase Generation via Exploring the Power&#10;  of Retrieval and Extraction" />
          <attvalue for="1" value="  In this paper, we present a novel integrated approach for keyphrase&#10;generation (KG). Unlike previous works which are purely extractive or&#10;generative, we first propose a new multi-task learning framework that jointly&#10;learns an extractive model and a generative model. Besides extracting&#10;keyphrases, the output of the extractive model is also employed to rectify the&#10;copy probability distribution of the generative model, such that the generative&#10;model can better identify important contents from the given document. Moreover,&#10;we retrieve similar documents with the given document from training data and&#10;use their associated keyphrases as external knowledge for the generative model&#10;to produce more accurate keyphrases. For further exploiting the power of&#10;extraction and retrieval, we propose a neural-based merging module to combine&#10;and re-rank the predicted keyphrases from the enhanced generative model, the&#10;extractive model, and the retrieved keyphrases. Experiments on the five KG&#10;benchmarks demonstrate that our integrated approach outperforms the&#10;state-of-the-art methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.10380" label="2310.10380">
        <attvalues>
          <attvalue for="0" value="Contextual Data Augmentation for Task-Oriented Dialog Systems" />
          <attvalue for="1" value="  Collection of annotated dialogs for training task-oriented dialog systems&#10;have been one of the key bottlenecks in improving current models. While dialog&#10;response generation has been widely studied on the agent side, it is not&#10;evident if similar generative models can be used to generate a large variety&#10;of, and often unexpected, user inputs that real dialog systems encounter in&#10;practice. Existing data augmentation techniques such as paraphrase generation&#10;do not take the dialog context into consideration. In this paper, we develop a&#10;novel dialog augmentation model that generates a user turn, conditioning on&#10;full dialog context. Additionally, with a new prompt design for language model,&#10;and output re-ranking, the dialogs generated from our model can be directly&#10;used to train downstream dialog systems. On common benchmark datasets MultiWoZ&#10;and SGD, we show that our dialog augmentation model generates high quality&#10;dialogs and improves dialog success rate by as much as $8\%$ over baseline.&#10;" />
          <attvalue for="2" value=" &#10;&#10;Users of commercial voice assistants and chatbots (e.g. Alexa, Siri, Google assistant) are able to accomplish various tasks by interacting with them via natural language conversation. Task-oriented dialog models form the core technology behind these applications, which understands users' natural language utterances \cite{Hakkani-TurTCCG16,goo2018slot}, keeps track of the conversation \cite{MrksicSWTY17,ChenLWZT020}, performs requested tasks (e.g. API calls) \cite{WenMBY17,GaoWPLL18}, and generates appropriate meaningful response to the user \cite{WenGMSVY15,ZhangSGCBGGLD20}. &#10;&#10;Training neural task-oriented dialog models \cite{SimpleTOD:Hosseini-AslMWY20,SOLOIST:PengLLSLG21,UBAR:YangLQ21}, requires a large amount of annotated data, which is difficult to obtain for model developers. While crowd-sourcing and dialog simulation based on agent interplay \cite{shah2018building,Simulator:LinAEBZB20} addresses this issue to a certain extent, these are slow and don't provide sufficient coverage of different natural language (NL) user turn surface form variations. Recently, large pre-trained language models (e.g. GPT-2 \cite{GPT2:radford2019language}, T5 \cite{T5:RaffelSRLNMZLL20}) have been successfully used to generate fluent agent dialog responses, both with dialog context \cite{GuWWSY20,ZhangSGCBGGLD20,KaleR20} or without it \cite{KaleR20a,XuWKL20}. However, it is unclear if similar models can capture the large variation of user turn distribution in such task-oriented dialogs. Previous work on data augmentation for spoken language understanding has largely focused on generating paraphrases of user utterance, with a specific goal and set of entities \cite{HouLCL:18,ZhaoZY19,LinXZZZ21}. However, such utterances again fail to provide sufficient coverage of the large semantic space possible between dialog turns, and may not improve performance of downstream task-oriented dialog systems. As an example, in Table \ref{table:dialog1}, dialog $1$, the user says in the first turn $U_1=$ {``please put me in touch with the local police, i was just robbed''}. A valid variation of this user turn which fits in the dialog context as generated by our model is $U_2 =$ {``I was robbed and I am looking for help''}. Note that, $U_1$ and $U_2$ are not semantically equivalent paraphrases ($U_2$ doesn't explicitly request police). &#10;&#10;Therefore, in this paper, we propose a novel dialog augmentation model, using BART \cite{BART:LewisLGGMLSZ20}, which can generate variations of a user turn, when conditioned on past and future dialog turns. Unlike dialog response generation, our model does not have the strict requirement of conveying a desired fixed action response, and can also leverage the future turns in the dialog. We show that using future context is indeed beneficial for the dialog augmentation task.&#10;In addition, we propose a new NL context prompt design to delineate between user and system turns, which better aligns with the language model pre-training task, and significantly improves quality of generated utterances and their positive impact on downstream tasks. On benchmark MutiWoZ \cite{Multiwoz21:EricGPSAGKGKH20} and SGD \cite{SGD:RastogiZSGK20} datasets, using dialogs generated from our augmentation model can significantly improve dialog success rate and goal accuracy compared to state-of-the-art baseline models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Cognitive Science, Language Models, Data Augmentation, Dialog Systems" />
        </attvalues>
      </node>
      <node id="1606.03777" label="1606.03777">
        <attvalues>
          <attvalue for="0" value="Neural Belief Tracker: Data-Driven Dialogue State Tracking" />
          <attvalue for="1" value="  One of the core components of modern spoken dialogue systems is the belief&#10;tracker, which estimates the user's goal at every step of the dialogue.&#10;However, most current approaches have difficulty scaling to larger, more&#10;complex dialogue domains. This is due to their dependency on either: a) Spoken&#10;Language Understanding models that require large amounts of annotated training&#10;data; or b) hand-crafted lexicons for capturing some of the linguistic&#10;variation in users' language. We propose a novel Neural Belief Tracking (NBT)&#10;framework which overcomes these problems by building on recent advances in&#10;representation learning. NBT models reason over pre-trained word vectors,&#10;learning to compose them into distributed representations of user utterances&#10;and dialogue context. Our evaluation on two datasets shows that this approach&#10;surpasses past limitations, matching the performance of state-of-the-art models&#10;which rely on hand-crafted semantic lexicons and outperforming them when such&#10;lexicons are not provided.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.06176" label="1801.06176">
        <attvalues>
          <attvalue for="0" value="Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy&#10;  Learning" />
          <attvalue for="1" value="  Training a task-completion dialogue agent via reinforcement learning (RL) is&#10;costly because it requires many interactions with real users. One common&#10;alternative is to use a user simulator. However, a user simulator usually lacks&#10;the language complexity of human interlocutors and the biases in its design may&#10;tend to degrade the agent. To address these issues, we present Deep Dyna-Q,&#10;which to our knowledge is the first deep RL framework that integrates planning&#10;for task-completion dialogue policy learning. We incorporate into the dialogue&#10;agent a model of the environment, referred to as the world model, to mimic real&#10;user response and generate simulated experience. During dialogue policy&#10;learning, the world model is constantly updated with real user experience to&#10;approach real user behavior, and in turn, the dialogue agent is optimized using&#10;both real experience and simulated experience. The effectiveness of our&#10;approach is demonstrated on a movie-ticket booking task in both simulated and&#10;human-in-the-loop settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1508.01745" label="1508.01745">
        <attvalues>
          <attvalue for="0" value="Semantically Conditioned LSTM-based Natural Language Generation for&#10;  Spoken Dialogue Systems" />
          <attvalue for="1" value="  Natural language generation (NLG) is a critical component of spoken dialogue&#10;and it has a significant impact both on usability and perceived quality. Most&#10;NLG systems in common use employ rules and heuristics and tend to generate&#10;rigid and stylised responses without the natural variation of human language.&#10;They are also not easily scaled to systems covering multiple domains and&#10;languages. This paper presents a statistical language generator based on a&#10;semantically controlled Long Short-term Memory (LSTM) structure. The LSTM&#10;generator can learn from unaligned data by jointly optimising sentence planning&#10;and surface realisation using a simple cross entropy training criterion, and&#10;language variation can be easily achieved by sampling from output candidates.&#10;With fewer heuristics, an objective evaluation in two differing test domains&#10;showed the proposed method improved performance compared to previous methods.&#10;Human judges scored the LSTM system higher on informativeness and naturalness&#10;and overall preferred it to the other systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00796" label="2005.00796">
        <attvalues>
          <attvalue for="0" value="A Simple Language Model for Task-Oriented Dialogue" />
          <attvalue for="1" value="  Task-oriented dialogue is often decomposed into three tasks: understanding&#10;user input, deciding actions, and generating a response. While such&#10;decomposition might suggest a dedicated model for each sub-task, we find a&#10;simple, unified approach leads to state-of-the-art performance on the MultiWOZ&#10;dataset. SimpleTOD is a simple approach to task-oriented dialogue that uses a&#10;single, causal language model trained on all sub-tasks recast as a single&#10;sequence prediction problem. This allows SimpleTOD to fully leverage transfer&#10;learning from pre-trained, open domain, causal language models such as GPT-2.&#10;SimpleTOD improves over the prior state-of-the-art in joint goal accuracy for&#10;dialogue state tracking, and our analysis reveals robustness to noisy&#10;annotations in this setting. SimpleTOD also improves the main metrics used to&#10;evaluate action decisions and response generation in an end-to-end setting:&#10;inform rate by 8.1 points, success rate by 9.7 points, and combined score by&#10;7.2 points.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.04871" label="1801.04871">
        <attvalues>
          <attvalue for="0" value="Building a Conversational Agent Overnight with Dialogue Self-Play" />
          <attvalue for="1" value="  We propose Machines Talking To Machines (M2M), a framework combining&#10;automation and crowdsourcing to rapidly bootstrap end-to-end dialogue agents&#10;for goal-oriented dialogues in arbitrary domains. M2M scales to new tasks with&#10;just a task schema and an API client from the dialogue system developer, but it&#10;is also customizable to cater to task-specific interactions. Compared to the&#10;Wizard-of-Oz approach for data collection, M2M achieves greater diversity and&#10;coverage of salient dialogue flows while maintaining the naturalness of&#10;individual utterances. In the first phase, a simulated user bot and a&#10;domain-agnostic system bot converse to exhaustively generate dialogue&#10;&quot;outlines&quot;, i.e. sequences of template utterances and their semantic parses. In&#10;the second phase, crowd workers provide contextual rewrites of the dialogues to&#10;make the utterances more natural while preserving their meaning. The entire&#10;process can finish within a few hours. We propose a new corpus of 3,000&#10;dialogues spanning 2 domains collected with M2M, and present comparisons with&#10;popular dialogue datasets on the quality and diversity of the surface forms and&#10;dialogue flows.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.15006" label="2004.15006">
        <attvalues>
          <attvalue for="0" value="Template Guided Text Generation for Task-Oriented Dialogue" />
          <attvalue for="1" value="  Virtual assistants such as Google Assistant, Amazon Alexa, and Apple Siri&#10;enable users to interact with a large number of services and APIs on the web&#10;using natural language. In this work, we investigate two methods for Natural&#10;Language Generation (NLG) using a single domain-independent model across a&#10;large number of APIs. First, we propose a schema-guided approach which&#10;conditions the generation on a schema describing the API in natural language.&#10;Our second method investigates the use of a small number of templates, growing&#10;linearly in number of slots, to convey the semantics of the API. To generate&#10;utterances for an arbitrary slot combination, a few simple templates are first&#10;concatenated to give a semantically correct, but possibly incoherent and&#10;ungrammatical utterance. A pre-trained language model is subsequently employed&#10;to rewrite it into coherent, natural sounding text. Through automatic metrics&#10;and human evaluation, we show that our method improves over strong baselines,&#10;is robust to out-of-domain inputs and shows improved sample efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.05589" label="2106.05589">
        <attvalues>
          <attvalue for="0" value="AUGNLG: Few-shot Natural Language Generation using Self-trained Data&#10;  Augmentation" />
          <attvalue for="1" value="  Natural Language Generation (NLG) is a key component in a task-oriented&#10;dialogue system, which converts the structured meaning representation (MR) to&#10;the natural language. For large-scale conversational systems, where it is&#10;common to have over hundreds of intents and thousands of slots, neither&#10;template-based approaches nor model-based approaches are scalable. Recently,&#10;neural NLGs started leveraging transfer learning and showed promising results&#10;in few-shot settings. This paper proposes AUGNLG, a novel data augmentation&#10;approach that combines a self-trained neural retrieval model with a few-shot&#10;learned NLU model, to automatically create MR-to-Text data from open-domain&#10;texts. The proposed system mostly outperforms the state-of-the-art methods on&#10;the FewShotWOZ data in both BLEU and Slot Error Rate. We further confirm&#10;improved results on the FewShotSGD data and provide comprehensive analysis&#10;results on key components of our system. Our code and data are available at&#10;https://github.com/XinnuoXu/AugNLG.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;NLG for Dialogue Response Generation &#10;There has been a body of work on neural NLG models, adopting various architectures, such as RNNs \cite{wen-etal-2015-semantically}, attention RNNs \cite{dusek-jurcicek-2016-sequence}, SC-LSTM \cite{wen-etal-2016-multi}, T2G2 \cite{kale-rastogi-2020-template}, AdapterCL \cite{madotto2020continual} and associated variants \cite{tran2017natural, tran2017neural}. Despite the improved flexibility and naturalness over template-based methods, neural approaches require large amounts of annotated data to reach good performance.&#10;&#10;Data Augmentation &#10;Data augmentation has been widely applied to a variety of NLP tasks, including sentence classification \cite{xie2020unsupervised}, natural language inference \cite{NEURIPS2019_671f0311} and spoken language understanding \cite{li2019insufficient, quan2019effective, zhao2019data}. Prior approaches for text data utilized back-translation \cite{sennrich2016improving, edunov2018understanding}, c-BERT word replacement \cite{jiao-etal-2020-tinybert}, mixed labels and representations \cite{guo2019augmenting, chen2020mixtext} and paraphrase data \cite{gao2020paraphrase}. However, the range of augmented data will be inherently limited, particularly in few-shot learning settings due to the nature of prior approaches, which only leverages in-domain data. In contrast, we take a rarely explored approach, tapping into a wealth of open-domain text that covers almost all topics. Recently, \cite{du-etal-2021-self} proposed a self-training method to augment data for NLU tasks by retrieving sentences from data crawled on the web. However, their method cannot be directly applied to the NLG problem since it does not yield MR annotations. Our approach, in contrast, generates MR-to-Text data by jointly employing a self-trained neural retrieval model with a few-shot learned NLU model.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1807.01554" label="1807.01554">
        <attvalues>
          <attvalue for="0" value="Sequence-to-Sequence Data Augmentation for Dialogue Language&#10;  Understanding" />
          <attvalue for="1" value="  In this paper, we study the problem of data augmentation for language&#10;understanding in task-oriented dialogue system. In contrast to previous work&#10;which augments an utterance without considering its relation with other&#10;utterances, we propose a sequence-to-sequence generation based data&#10;augmentation framework that leverages one utterance's same semantic&#10;alternatives in the training data. A novel diversity rank is incorporated into&#10;the utterance representation to make the model produce diverse utterances and&#10;these diversely augmented utterances help to improve the language understanding&#10;module. Experimental results on the Airline Travel Information System dataset&#10;and a newly created semantic frame annotation on Stanford Multi-turn,&#10;Multidomain Dialogue Dataset show that our framework achieves significant&#10;improvements of 6.38 and 10.04 F-scores respectively when only a training set&#10;of hundreds utterances is represented. Case studies also confirm that our&#10;method generates diverse utterances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.10770" label="1908.10770">
        <attvalues>
          <attvalue for="0" value="Data Augmentation with Atomic Templates for Spoken Language&#10;  Understanding" />
          <attvalue for="1" value="  Spoken Language Understanding (SLU) converts user utterances into structured&#10;semantic representations. Data sparsity is one of the main obstacles of SLU due&#10;to the high cost of human annotation, especially when domain changes or a new&#10;domain comes. In this work, we propose a data augmentation method with atomic&#10;templates for SLU, which involves minimum human efforts. The atomic templates&#10;produce exemplars for fine-grained constituents of semantic representations. We&#10;propose an encoder-decoder model to generate the whole utterance from atomic&#10;exemplars. Moreover, the generator could be transferred from source domains to&#10;help a new domain which has little data. Experimental results show that our&#10;method achieves significant improvements on DSTC 2\&amp;3 dataset which is a domain&#10;adaptation setting of SLU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Data augmentation for SLU\quad \cite{hou2018sequence} use a sequence-to-sequence model to translate between sentence pairs with the same meaning. However, it cannot generate data for new semantic representations. \cite{yoo2018data} use a variational autoencoder to generate labelled language. It still lacks variety of semantic meanings. Let $x$ and $y$ denote an input sentence and the corresponding semantic representation respectively. These two works estimate $p(x, y)$ with generation models to produce more labelled samples, while they have two drawbacks. 1) The models cannot control which kinds of semantic meaning should be generated for supply. 2) The models cannot generate data for new semantic representations which may contain out-of-vocabulary (OOV) labels. We propose to generate utterance based on semantic representations by estimating $p(x|y)$, because $y$ is well-structured and easy to be synthesized. To overcome the OOV problem of semantic labels, we first map them to atomic exemplars with a little of human effort.&#10;&#10;Zero-shot learning of SLU\quad Besides data augmentation, zero-shot learning of SLU \cite{ferreira2015zero,yazdani2015model} is also related, which can adapt to unseen semantic labels. \cite{yazdani2015model} exploit a binary classifier for each possible act-slot-value triple to predict whether it exists in the input sentence. \cite{zhao2019hierarchical} propose a hierarchical decoding model for SLU. However, they still have problem with new act and slot. \cite{Bapna2017towards,lee2018zero,sz128-zhu-sigdial18} try to solve it with textual slot descriptions. In this paper, we propose atomic templates to describe act-slot-value triples but not separate slots or acts.&#10; " />
        </attvalues>
      </node>
      <node id="2108.08451" label="2108.08451">
        <attvalues>
          <attvalue for="0" value="Augmenting Slot Values and Contexts for Spoken Language Understanding&#10;  with Pretrained Models" />
          <attvalue for="1" value="  Spoken Language Understanding (SLU) is one essential step in building a&#10;dialogue system. Due to the expensive cost of obtaining the labeled data, SLU&#10;suffers from the data scarcity problem. Therefore, in this paper, we focus on&#10;data augmentation for slot filling task in SLU. To achieve that, we aim at&#10;generating more diverse data based on existing data. Specifically, we try to&#10;exploit the latent language knowledge from pretrained language models by&#10;finetuning them. We propose two strategies for finetuning process: value-based&#10;and context-based augmentation. Experimental results on two public SLU datasets&#10;have shown that compared with existing data augmentation methods, our proposed&#10;method can generate more diverse sentences and significantly improve the&#10;performance on SLU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.01669" label="1907.01669">
        <attvalues>
          <attvalue for="0" value="MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State&#10;  Corrections and State Tracking Baselines" />
          <attvalue for="1" value="  MultiWOZ 2.0 (Budzianowski et al., 2018) is a recently released multi-domain&#10;dialogue dataset spanning 7 distinct domains and containing over 10,000&#10;dialogues. Though immensely useful and one of the largest resources of its kind&#10;to-date, MultiWOZ 2.0 has a few shortcomings. Firstly, there is substantial&#10;noise in the dialogue state annotations and dialogue utterances which&#10;negatively impact the performance of state-tracking models. Secondly, follow-up&#10;work (Lee et al., 2019) has augmented the original dataset with user dialogue&#10;acts. This leads to multiple co-existent versions of the same dataset with&#10;minor modifications. In this work we tackle the aforementioned issues by&#10;introducing MultiWOZ 2.1. To fix the noisy state annotations, we use&#10;crowdsourced workers to re-annotate state and utterances based on the original&#10;utterances in the dataset. This correction process results in changes to over&#10;32% of state annotations across 40% of the dialogue turns. In addition, we fix&#10;146 dialogue utterances by canonicalizing slot values in the utterances to the&#10;values in the dataset ontology. To address the second problem, we combined the&#10;contributions of the follow-up works into MultiWOZ 2.1. Hence, our dataset also&#10;includes user dialogue acts as well as multiple slot descriptions per dialogue&#10;state slot. We then benchmark a number of state-of-the-art dialogue state&#10;tracking models on the MultiWOZ 2.1 dataset and show the joint state tracking&#10;performance on the corrected state annotations. We are publicly releasing&#10;MultiWOZ 2.1 to the community, hoping that this dataset resource will allow for&#10;more effective models across various dialogue subproblems to be built in the&#10;future.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.12743" label="2112.12743">
        <attvalues>
          <attvalue for="0" value="Multi-speaker Multi-style Text-to-speech Synthesis With Single-speaker&#10;  Single-style Training Data Scenarios" />
          <attvalue for="1" value="  In the existing cross-speaker style transfer task, a source speaker with&#10;multi-style recordings is necessary to provide the style for a target speaker.&#10;However, it is hard for one speaker to express all expected styles. In this&#10;paper, a more general task, which is to produce expressive speech by combining&#10;any styles and timbres from a multi-speaker corpus in which each speaker has a&#10;unique style, is proposed. To realize this task, a novel method is proposed.&#10;This method is a Tacotron2-based framework but with a fine-grained text-based&#10;prosody predicting module and a speaker identity controller. Experiments&#10;demonstrate that the proposed method can successfully express a style of one&#10;speaker with the timber of another speaker bypassing the dependency on a single&#10;speaker's multi-style corpus. Moreover, the explicit prosody features used in&#10;the prosody predicting module can increase the diversity of synthetic speech by&#10;adjusting the value of prosody features.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, enormous progress has been made in the neural text-to-speech (TTS), which benefits from the development of sequence-to-sequence (seq2seq) neural models~\cite{bahdanau2014neural,sutskever2014sequence}, making it possible to synthesize highly intelligible and natural speech~\cite{wang2017tacotron,shen2018natural,ren2019fastspeech,yu2020durian}. Despite the successful application of TTS in many scenarios, how to create expressive synthetic speech that can be flexibly controlled in terms of various speaking styles and speaker timbres is desirable for better user experience. This paper proposes a new expressive speech synthesis task that creates diversity synthetic speech by combining the timbre and speaking style from different speakers.&#10;&#10;To create a TTS system with the ability to synthesize various expressive speech, a straightforward method is to train a TTS model with a database with manual labels~\cite{lee2017emotional,choi2019multi,litao2021controllable,li2018emphatic,liu2021expressive}, for instance, a database with manually labeled emotion categories~\cite{lee2017emotional,litao2021controllable} or speaking styles~\cite{liu2021expressive}. However, the limitation of these methods is obvious, i.e., it heavily depends on the training data and can not create new voice by combining different speaker timbres and speaking styles. To transplant a style to a target speaker for whom no labeled expressive recording exists, the cross-speaker style transfer task has attracted much attention~\cite{bian2019multi,whitehill2019multi,karlapati2020copycat,litao2021controllable,pan2021cross,shang2021incorporatingM3}. Reference embedding-based cross-speaker style transfer models~\cite{bian2019multi,whitehill2019multi,karlapati2020copycat,shang2021incorporatingM3,Li2021ControllableCE}, typically based on several general reference embedding methods \cite{skerry2018towards,wang2018style,zhang2019learning}, have shown promising performance on the style transfer task.&#10;&#10;While those cross-speaker transfer methods can successfully produce expressive speech with a specific speaking style and a timbre from a speaker who has no such a speaking style in the corpus, they typically depend on a source speaker who has enough manually labeled expressive sources. It requires a source speaker to be an expert in expressing all expected styles with the aim to produce synthetic speech with various styles. Anyway, it is impossible for one source speaker to imitate all possible speaking styles and record enough recordings. In contrast, it is much easier to obtain an expressive corpus in which each speaker only speak one specific speaking style that he or&#10;she is good at. With such a corpus, a practical task is to build a TTS system that has the ability to produce synthetic speech by combining different timbres and styles from different speakers, which is referred to as speaker-related multi-style and multi-speaker TTS (SRM2TTS). &#10;&#10;However, it is non-trivial to achieve such a SRM2TTS task. Compared with the traditional cross-speaker style transfer task, in the SRM2TTS task, the timbre and style are closely entangled, making it difficult to transfer styles across speakers with reference-based methods. Taking inspiration from the success of the label-assisted content-aware prosody prediction model on the style transfer task~\cite{pan2021cross}, a novel method for the SRM2TTS task is proposed in this work. Specifically, based on a typical neural seq2seq framework, a content-aware multi-scale prosody modeling module is proposed, which can provide the style information to the TTS system based on the style label and input text. With an extra speaker identity controller, the proposed method can distinguish different styles and timbres, and thus can perform any combination of speakers and styles for SRM2TTS. Experiments have shown that the proposed method achieves good performance on synthesizing expressive speech by combining any speaker timbre and speaking style. Besides, benefit from the explicit modeling of prosody features, the proposed method can flexibly control each prosodic component, e.g., pitch and energy, which can increase the diversity of synthesized speech. &#10;&#10;Our contribution can be summarized as follows: &#10;(1) Synthesizing expressive speech by combining any style and timbre based on a multi-speaker database in which each speaker has a unique speaking style is first proposed in this work. The realization of this task has profound implications from a perspective of usability.&#10;(2) A novel method, which can realize the combination and control of any style and timbre on expressive speech synthesis, is proposed. &#10;(3) Extensive experiments have shown that with a novel fine-grained text-based prosody modeling module the proposed method can explicitly model and flexibly control the prosodic components.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Machine Learning, Linguistics, Engineering, Speech Style Transfer, Prosody Prediction, Multi-Speaker Synthesis, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1703.10135" label="1703.10135">
        <attvalues>
          <attvalue for="0" value="Tacotron: Towards End-to-End Speech Synthesis" />
          <attvalue for="1" value="  A text-to-speech synthesis system typically consists of multiple stages, such&#10;as a text analysis frontend, an acoustic model and an audio synthesis module.&#10;Building these components often requires extensive domain expertise and may&#10;contain brittle design choices. In this paper, we present Tacotron, an&#10;end-to-end generative text-to-speech model that synthesizes speech directly&#10;from characters. Given &lt;text, audio&gt; pairs, the model can be trained completely&#10;from scratch with random initialization. We present several key techniques to&#10;make the sequence-to-sequence framework perform well for this challenging task.&#10;Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English,&#10;outperforming a production parametric system in terms of naturalness. In&#10;addition, since Tacotron generates speech at the frame level, it's&#10;substantially faster than sample-level autoregressive methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.05884" label="1712.05884">
        <attvalues>
          <attvalue for="0" value="Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram&#10;  Predictions" />
          <attvalue for="1" value="  This paper describes Tacotron 2, a neural network architecture for speech&#10;synthesis directly from text. The system is composed of a recurrent&#10;sequence-to-sequence feature prediction network that maps character embeddings&#10;to mel-scale spectrograms, followed by a modified WaveNet model acting as a&#10;vocoder to synthesize timedomain waveforms from those spectrograms. Our model&#10;achieves a mean opinion score (MOS) of $4.53$ comparable to a MOS of $4.58$ for&#10;professionally recorded speech. To validate our design choices, we present&#10;ablation studies of key components of our system and evaluate the impact of&#10;using mel spectrograms as the input to WaveNet instead of linguistic, duration,&#10;and $F_0$ features. We further demonstrate that using a compact acoustic&#10;intermediate representation enables significant simplification of the WaveNet&#10;architecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.05447" label="1711.05447">
        <attvalues>
          <attvalue for="0" value="Emotional End-to-End Neural Speech Synthesizer" />
          <attvalue for="1" value="  In this paper, we introduce an emotional speech synthesizer based on the&#10;recent end-to-end neural model, named Tacotron. Despite its benefits, we found&#10;that the original Tacotron suffers from the exposure bias problem and&#10;irregularity of the attention alignment. Later, we address the problem by&#10;utilization of context vector and residual connection at recurrent neural&#10;networks (RNNs). Our experiments showed that the model could successfully train&#10;and generate speech for given emotion labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.08679" label="2011.08679">
        <attvalues>
          <attvalue for="0" value="Controllable Emotion Transfer For End-to-End Speech Synthesis" />
          <attvalue for="1" value="  Emotion embedding space learned from references is a straightforward approach&#10;for emotion transfer in encoder-decoder structured emotional text to speech&#10;(TTS) systems. However, the transferred emotion in the synthetic speech is not&#10;accurate and expressive enough with emotion category confusions. Moreover, it&#10;is hard to select an appropriate reference to deliver desired emotion strength.&#10;To solve these problems, we propose a novel approach based on Tacotron. First,&#10;we plug two emotion classifiers -- one after the reference encoder, one after&#10;the decoder output -- to enhance the emotion-discriminative ability of the&#10;emotion embedding and the predicted mel-spectrum. Second, we adopt style loss&#10;to measure the difference between the generated and reference mel-spectrum. The&#10;emotion strength in the synthetic speech can be controlled by adjusting the&#10;value of the emotion embedding as the emotion embedding can be viewed as the&#10;feature map of the mel-spectrum. Experiments on emotion transfer and strength&#10;control have shown that the synthetic speech of the proposed method is more&#10;accurate and expressive with less emotion category confusions and the control&#10;of emotion strength is more salient to listeners.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.01490" label="2008.01490">
        <attvalues>
          <attvalue for="0" value="Expressive TTS Training with Frame and Style Reconstruction Loss" />
          <attvalue for="1" value="  We propose a novel training strategy for Tacotron-based text-to-speech (TTS)&#10;system to improve the expressiveness of speech. One of the key challenges in&#10;prosody modeling is the lack of reference that makes explicit modeling&#10;difficult. The proposed technique doesn't require prosody annotations from&#10;training data. It doesn't attempt to model prosody explicitly either, but&#10;rather encodes the association between input text and its prosody styles using&#10;a Tacotron-based TTS framework. Our proposed idea marks a departure from the&#10;style token paradigm where prosody is explicitly modeled by a bank of prosody&#10;embeddings. The proposed training strategy adopts a combination of two&#10;objective functions: 1) frame level reconstruction loss, that is calculated&#10;between the synthesized and target spectral features; 2) utterance level style&#10;reconstruction loss, that is calculated between the deep style features of&#10;synthesized and target speech. The proposed style reconstruction loss is&#10;formulated as a perceptual loss to ensure that utterance level speech style is&#10;taken into consideration during training. Experiments show that the proposed&#10;training strategy achieves remarkable performance and outperforms a&#10;state-of-the-art baseline in both naturalness and expressiveness. To our best&#10;knowledge, this is the first study to incorporate utterance level perceptual&#10;quality as a loss function into Tacotron training for improved expressiveness.&#10;" />
          <attvalue for="2" value="&#10;&#10;W{ith} the advent of deep learning, neural TTS has shown many advantages over the conventional TTS techniques \cite{tokuda2013speech, ze2013statistical,liu2017mongolian}. For example, encoder-decoder architecture with attention mechanism, such as Tacotron~\cite{wang2017tacotron,shen2018natural,liu2020wavetts,lee2019robust}, has consistently achieved high voice quality. The key idea is to integrate the conventional TTS pipeline \cite{hunt1996unit,tokuda2002hmm} into an unified framework that learns sequence-to-sequence mapping from text to a sequence of acoustic features~\cite{lee2019robust,chung2019semi,He2019,Luong2019,liu2019teacher}. Furthermore, together with a neural vocoder \cite{hayashi2017investigation,shen2018natural,chen2018high,Okamoto2019, berrak_is18, berrak-journal, sisman2018adaptive}, neural TTS generates natural-sounding and human-like speech which achieves state-of-the-art performance. Despite the progress, the expressiveness of the synthesized speech remains to be improved. &#10;&#10;Speech conveys information not only through phonetic content, but also through its prosody. Speech prosody can affect syntactic and semantic interpretation of an utterance \cite{hirschberg2004pragmatics}, that is called linguistic prosody. Speech prosody is also used to display one's emotional state, that is referred to as affective prosody.&#10;Both linguistic prosody and affective prosody are manifested over a segment of speech beyond short-time speech frame. Linguistically, speech prosody in general refers to stress, intonation, and rhythm in spoken words, phrases, and sentences. As speech prosody is the result of the interplay of multiple speech properties, it is not easy to define speech prosody by a simple labeling scheme~\cite{Luong,lin2019investigation,hodariusing,zhao2020improved,hodari2020perception}. Even if a labeling scheme is possible~\cite{Silverman1992TOBIAS,taylor1998assigning}, a set of discrete labels may not be sufficient to describe the entire continuum of speech prosody. &#10;&#10;Besides naturalness, one of the factors that differentiate human speech from today's synthesized speech is their expressiveness.&#10;Prosody is one of the defining features of expressiveness that makes speech lively. Several recent studies successfully improve the expressiveness of Tacotron TTS framework\cite{ wang2018style,Stanton2018Predicting,skerry2018towards,sun2020fully,sun2020generating}. The idea is to learn latent prosody embedding, i.e. style token, from training data~\cite{wang2018style}. At run-time, the style token can be used to predict the speech style from text~\cite{Stanton2018Predicting}, or to transfer the speech style from a reference utterance to target~\cite{skerry2018towards}. &#10;It is observed that such speech styling is effective and consistently improves speech quality. Sun et al. \cite{sun2020fully, sun2020generating} further study a hierarchical, fine-grained and interpretable latent variable model for prosody rendering. The studies show that precise control of the prosody style leads to improvement of prosody expressiveness in the Tacotron TTS framework. However, several issues have hindered the effectiveness of above prosody modeling techniques. &#10;&#10;First, the latent embedding space of prosody is learnt in an unsupervised manner, where the style is defined as anything but speaker identity and phonetic content in speech. We note that many different styles co-exist in speech. Some are speaker dependent, such as accent and idiolect, others are speaker independent such as prosodic phrasing, lexical stress and prosodic stress. There is no guarantee that such latent embedding space of style represents only the intended prosody. Second, while the techniques don't require the prosody annotations on training data, they require a reference speech or a manual selection of style token~\cite{wang2018style} in order to explicitly control the style of output speech during run-time inference. While it is possible to automate the style token selection \cite{Stanton2018Predicting}, a correct prediction of style token is subject to both the design of the style token dictionary, and the run-time style token prediction algorithm. Third, the style token dictionary in Tacotron is trained from a collection of speech utterances to represent a large range of acoustic expressiveness for a speaker or an audiobook\cite{wang2018style}. It is not intended to provide differential prosodic details at phrase or utterance level. It is desirable for Tacotron system to learn to automate the prosody styling in response to input text at run-time, that will be the focus of this paper.&#10;&#10;To address the above issues, we believe that Tacotron training should minimize frame level reconstruction loss~\cite{wang2017tacotron,shen2018natural} and utterance level perceptual loss at the same time. Perceptual loss is first proposed for image stylization and synthesis \cite{ dosovitskiy2016generating,johnson2016perceptual,chen2017photographic,9052944}, where feature activation patterns, or deep features, derived from pre-trained auxiliary networks are used to optimize the perceptual quality of output image. Several computational models have been proposed to approximate human perception of audio quality, such as Perceptual Evaluation of Audio Quality (PEAQ) \cite{thiede2000peaq}, Perceptual Evaluation of Speech Quality (PESQ) \cite{rix2001perceptual}, and Perceptual Evaluation of Audio methods for Source Separation (PEASS) \cite{emiya2010peass}. However, such models are not differentiable, hence cannot be directly employed during TTS training. We believe that utterance level perceptual loss based on deep features that reflects global speech style would be useful to improve overall speech quality.&#10;&#10;We are motivated to study a novel training strategy for TTS systems, that learns to associate prosody styles with input text implicitly. We would like to avoid the use of prosody annotations. We don't attempt to model prosody explicitly either, but rather learn the association between prosody styles and input text using existing neural TTS system, such as Tacotron. As the training strategy is only involved during training, it doesn't change the run-time inference process for neural TTS system. At run-time, we don't require any reference signal nor manual selection of prosody style. &#10;&#10;The main contributions of this paper include: 1) we propose a novel training strategy for Tacotron TTS that improves utterance level expressiveness of speech; 2) we propose to supervise the training of Tacotron with a fully differentiable perceptual loss, which is derived from a pre-trained auxiliary network, in addition to frame reconstruction loss; and 3) we successfully implement a system that doesn't require any reference speech nor manual selection of prosody style at run-time.&#10;To our best knowledge, this is the first study to incorporate perceptual loss into Tacotron training for improved expressiveness.&#10;&#10;This paper is organized as follows: In Section II, we present the research background and related work to motivate our study. In Section III, we propose a novel training strategy for TTS system with frame and style reconstruction loss. In Section IV, we report the subjective and objective evaluations. Section V concludes the discussion.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.11958" label="1910.11958">
        <attvalues>
          <attvalue for="0" value="Multi-Reference Neural TTS Stylization with Adversarial Cycle&#10;  Consistency" />
          <attvalue for="1" value="  Current multi-reference style transfer models for Text-to-Speech (TTS)&#10;perform sub-optimally on disjoints datasets, where one dataset contains only a&#10;single style class for one of the style dimensions. These models generally fail&#10;to produce style transfer for the dimension that is underrepresented in the&#10;dataset. In this paper, we propose an adversarial cycle consistency training&#10;scheme with paired and unpaired triplets to ensure the use of information from&#10;all style dimensions. During training, we incorporate unpaired triplets with&#10;randomly selected reference audio samples and encourage the synthesized speech&#10;to preserve the appropriate styles using adversarial cycle consistency. We use&#10;this method to transfer emotion from a dataset containing four emotions to a&#10;dataset with only a single emotion. This results in a 78% improvement in style&#10;transfer (based on emotion classification) with minimal reduction in fidelity&#10;and naturalness. In subjective evaluations our method was consistently rated as&#10;closer to the reference style than the baseline. Synthesized speech samples are&#10;available at: https://sites.google.com/view/adv-cycle-consistent-tts&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.12562" label="2107.12562">
        <attvalues>
          <attvalue for="0" value="Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech&#10;  Synthesis" />
          <attvalue for="1" value="  Cross-speaker style transfer is crucial to the applications of multi-style&#10;and expressive speech synthesis at scale. It does not require the target&#10;speakers to be experts in expressing all styles and to collect corresponding&#10;recordings for model training. However, the performances of existing style&#10;transfer methods are still far behind real application needs. The root causes&#10;are mainly twofold. Firstly, the style embedding extracted from single&#10;reference speech can hardly provide fine-grained and appropriate prosody&#10;information for arbitrary text to synthesize. Secondly, in these models the&#10;content/text, prosody, and speaker timbre are usually highly entangled, it's&#10;therefore not realistic to expect a satisfied result when freely combining&#10;these components, such as to transfer speaking style between speakers. In this&#10;paper, we propose a cross-speaker style transfer text-to-speech (TTS) model&#10;with explicit prosody bottleneck. The prosody bottleneck builds up the kernels&#10;accounting for speaking style robustly, and disentangles the prosody from&#10;content and speaker timbre, therefore guarantees high quality cross-speaker&#10;style transfer. Evaluation result shows the proposed method even achieves&#10;on-par performance with source speaker's speaker-dependent (SD) model in&#10;objective measurement of prosody, and significantly outperforms the cycle&#10;consistency and GMVAE-based baselines in objective and subjective evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.09047" label="1803.09047">
        <attvalues>
          <attvalue for="0" value="Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with&#10;  Tacotron" />
          <attvalue for="1" value="  We present an extension to the Tacotron speech synthesis architecture that&#10;learns a latent embedding space of prosody, derived from a reference acoustic&#10;representation containing the desired prosody. We show that conditioning&#10;Tacotron on this learned embedding space results in synthesized audio that&#10;matches the prosody of the reference signal with fine time detail even when the&#10;reference and synthesis speakers are different. Additionally, we show that a&#10;reference prosody embedding can be used to synthesize text that is different&#10;from that of the reference utterance. We define several quantitative and&#10;subjective metrics for evaluating prosody transfer, and report results with&#10;accompanying audio samples from single-speaker and 44-speaker Tacotron models&#10;on a prosody transfer task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.09017" label="1803.09017">
        <attvalues>
          <attvalue for="0" value="Style Tokens: Unsupervised Style Modeling, Control and Transfer in&#10;  End-to-End Speech Synthesis" />
          <attvalue for="1" value="  In this work, we propose &quot;global style tokens&quot; (GSTs), a bank of embeddings&#10;that are jointly trained within Tacotron, a state-of-the-art end-to-end speech&#10;synthesis system. The embeddings are trained with no explicit labels, yet learn&#10;to model a large range of acoustic expressiveness. GSTs lead to a rich set of&#10;significant results. The soft interpretable &quot;labels&quot; they generate can be used&#10;to control synthesis in novel ways, such as varying speed and speaking style -&#10;independently of the text content. They can also be used for style transfer,&#10;replicating the speaking style of a single audio clip across an entire&#10;long-form text corpus. When trained on noisy, unlabeled found data, GSTs learn&#10;to factorize noise and speaker identity, providing a path towards highly&#10;scalable but robust speech synthesis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.04342" label="1812.04342">
        <attvalues>
          <attvalue for="0" value="Learning latent representations for style control and transfer in&#10;  end-to-end speech synthesis" />
          <attvalue for="1" value="  In this paper, we introduce the Variational Autoencoder (VAE) to an&#10;end-to-end speech synthesis model, to learn the latent representation of&#10;speaking styles in an unsupervised manner. The style representation learned&#10;through VAE shows good properties such as disentangling, scaling, and&#10;combination, which makes it easy for style control. Style transfer can be&#10;achieved in this framework by first inferring style representation through the&#10;recognition network of VAE, then feeding it into TTS network to guide the style&#10;in synthesizing speech. To avoid Kullback-Leibler (KL) divergence collapse in&#10;training, several techniques are adopted. Finally, the proposed model shows&#10;good performance of style control and outperforms Global Style Token (GST)&#10;model in ABX preference tests on style transfer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.06333" label="2109.06333">
        <attvalues>
          <attvalue for="0" value="Connecting degree and polarity: An artificial language learning study" />
          <attvalue for="1" value="  We investigate a new linguistic generalization in pre-trained language models&#10;(taking BERT (Devlin et al., 2019) as a case study). We focus on degree&#10;modifiers (expressions like slightly, very, rather, extremely) and test the&#10;hypothesis that the degree expressed by a modifier (low, medium or high degree)&#10;is related to the modifier's sensitivity to sentence polarity (whether it shows&#10;preference for affirmative or negative sentences or neither). To probe this&#10;connection, we apply the Artificial Language Learning experimental paradigm&#10;from psycholinguistics to a neural language model. Our experimental results&#10;suggest that BERT generalizes in line with existing linguistic observations&#10;that relate degree semantics to polarity sensitivity, including the main one:&#10;low degree semantics is associated with preference towards positive polarity.&#10;" />
          <attvalue for="2" value="&#10;&#10;Linguistic expressions can be characterized along a variety of properties: what they mean, what parts they consist of, how they combine with other expressions and so on. Some of these properties are systematically related to each other. When these relations appear systematically in language after language, they can be grounds for implicational linguistic universals, for example, Greenberg's {Universal 37}: {A language never has more gender categories in nonsingular numbers than in the singular.} \cite{greenberg}. Here, two properties of linguistic expressions are related: the grammatical number of an expression and how many gender distinctions are available for this expression. More complex generalizations may concern correlation between continuous properties $A$ and $B$. &#10;&#10;In order to arrive at linguistic universals connecting $A$ and $B$, the relation between these properties has to be established at the level of individual languages, which is not trivial for properties with complex internal structure.&#10;&#10;In this paper, we study one such connection: the problem of polarity sensitivity of degree modifiers \cite{israel1996,israel2011,solt2018,solt2021}. Degree modifiers are words like {slightly}, {very}, and {extremely}. Property $A$, in this case, is the degree that these words convey, defined on a interval from very low to very high. For example, the degree of {slightly} is lower than the one of {very}. Property $B$ here encodes distributional preferences of degree modifiers with respect to polarity of a sentence where they appear -- roughly, whether they appear exclusively in negative or affirmative sentences, or &#10;show no polarity preference. Polarity preferences can also be represented as a continuous property from very low (negative polarity preference) to very high (positive polarity preference), with polarity-neutral in the middle. &#10;&#10;Interactions between linguistic properties have been subject to experimental studies in psycholinguistics and cognitive science.&#10;One prominent method is Artificial Language Learning \cite{friederici, motamedi, kanwal, culbertson, ettlinger, finley}. It has the following main ingredients: &#10;\begin{enumerate}&#10;\item {fragment of an artificial language}&#10;in the form of expressions that do not belong to the language that participants are speakers of; &#10;\item {training phase}, where some information about the language fragment is given to the participants; &#10;\item {testing phase}, where it is checked what other knowledge, beside the provided, was inferred during training. &#10;\end{enumerate}&#10;&#10;Originally designed for studies with human participants, the Artificial Language Learning framework has also been applied to neural network-based learning models \cite{piantadosi,carcassi,vandepol}.&#10;Replacing human participants with artificial learning agents allows to examine the learning process in more detail and to make a variety of learnability statements. One important property of these experiments is that the learning agents typically come in a blank state with no prior knowledge or biases. This limits the set of linguistic questions that can be targeted by this type of experiment. &#10;&#10;The way we use the Artificial Language Learning paradigm can be seen as middle ground between experiments with human participants and with artificial learners described above. Our approach also involves an artificial language fragment and a training procedure to introduce knowledge about some property $A$, but it uses a pre-trained language model (LM) \cite{Peters:2018,bert,brown2020language} as the learning agent.&#10; More technically, we extend a pre-trained LM with a set of new tokens with randomly initialized embeddings and perform fine-tuning on a carefully constructed synthetic dataset. The dataset is constructed in a way to indirectly introduce different values along property $A$ for different new tokens. Upon fine-tuning, we measure how the training affected property $B$ and how variation along $B$ depends on the values of property $A$ introduced during training.&#10;&#10;Our study will focus on English, as represented in a pre-trained LM BERT \cite{bert}. We see this as a proof of concept work that can be extended further along the cross-linguistic dimension and in application to other models. Using this set-up, we address the question of whether LMs encode a connection between degree semantics and polarity, therefore making a generalization across two different linguistic properties. &#10;&#10;Our approach belongs to the general area of studies using counterfactual linguistic data in NLP (\cite{kaushik,kaushik2021} a.o.); a part of that subfield that uses novel lexicon is the closest to the present paper (\cite{levy,bylinina2022driving} a.o.). Our work contributes to the general agenda of establishing closer connections between learning in humans and artificial neural models (\cite{futrell-etal-2019-neural,wilcox-etal-2020-structural} a.o.). The main step forward that we make with this paper is the extension of these methods to linguistic properties that have complex internal structure, rather than clear morphological or syntactic exponence. &#10;&#10;To sum up, we make the following contributions: &#10;\begin{itemize}&#10;\item We propose an experimental methodology to explore generalization between complex linguistic properties;&#10;\item We use this methodology to explore the relation between two linguistic phenomena, degree and polarity sensitivity, as represented in one pre-trained LM (BERT).&#10;\item We argue that, according to the experimental results, the LM in question indeed makes a connection between the degree encoded by a degree modifier and its polarity sensitivity.&#10;\end{itemize}&#10;&#10;The paper is structured as follows: Section~\ref{sec:degrees_and_polarity} gives linguistic background about degrees and polarity. Section~\ref{sec:method_sec} describes the general method. In Section~\ref{sec:synth_dataset_sec}, we define a synthetic dataset and the measures we use to estimate degree and polarity. Section~\ref{sec:exper_sec} presents the experiment. Section~\ref{sec:discuss_sec} discusses our results, the limitations of our set-up and suggestions for future work.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Degree Modifier Semantics, Computer Science, Language Model Analysis, Linguistics, Psycholinguistics, Artificial Intelligence, Psychology, Linguistic Generalization" />
        </attvalues>
      </node>
      <node id="2010.02114" label="2010.02114">
        <attvalues>
          <attvalue for="0" value="Explaining The Efficacy of Counterfactually Augmented Data" />
          <attvalue for="1" value="  In attempts to produce ML models less reliant on spurious patterns in NLP&#10;datasets, researchers have recently proposed curating counterfactually&#10;augmented data (CAD) via a human-in-the-loop process in which given some&#10;documents and their (initial) labels, humans must revise the text to make a&#10;counterfactual label applicable. Importantly, edits that are not necessary to&#10;flip the applicable label are prohibited. Models trained on the augmented data&#10;appear, empirically, to rely less on semantically irrelevant words and to&#10;generalize better out of domain. While this work draws loosely on causal&#10;thinking, the underlying causal model (even at an abstract level) and the&#10;principles underlying the observed out-of-domain improvements remain unclear.&#10;In this paper, we introduce a toy analog based on linear Gaussian models,&#10;observing interesting relationships between causal models, measurement noise,&#10;out-of-domain generalization, and reliance on spurious signals. Our analysis&#10;provides some insights that help to explain the efficacy of CAD. Moreover, we&#10;develop the hypothesis that while adding noise to causal features should&#10;degrade both in-domain and out-of-domain performance, adding noise to&#10;non-causal features should lead to relative improvements in out-of-domain&#10;performance. This idea inspires a speculative test for determining whether a&#10;feature attribution technique has identified the causal spans. If adding noise&#10;(e.g., by random word flips) to the highlighted spans degrades both in-domain&#10;and out-of-domain performance on a battery of challenge datasets, but adding&#10;noise to the complement gives improvements out-of-domain, it suggests we have&#10;identified causal spans. We present a large-scale empirical study comparing&#10;spans edited to create CAD to those selected by attention and saliency maps.&#10;Across numerous domains and models, we find that the hypothesized phenomenon is&#10;pronounced for CAD.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.02417" label="2011.02417">
        <attvalues>
          <attvalue for="0" value="Investigating Novel Verb Learning in BERT: Selectional Preference&#10;  Classes and Alternation-Based Syntactic Generalization" />
          <attvalue for="1" value="  Previous studies investigating the syntactic abilities of deep learning&#10;models have not targeted the relationship between the strength of the&#10;grammatical generalization and the amount of evidence to which the model is&#10;exposed during training. We address this issue by deploying a novel&#10;word-learning paradigm to test BERT's few-shot learning capabilities for two&#10;aspects of English verbs: alternations and classes of selectional preferences.&#10;For the former, we fine-tune BERT on a single frame in a verbal-alternation&#10;pair and ask whether the model expects the novel verb to occur in its sister&#10;frame. For the latter, we fine-tune BERT on an incomplete selectional network&#10;of verbal objects and ask whether it expects unattested but plausible&#10;verb/object pairs. We find that BERT makes robust grammatical generalizations&#10;after just one or two instances of a novel word in fine-tuning. For the verbal&#10;alternation tests, we find that the model displays behavior that is consistent&#10;with a transitivity bias: verbs seen few times are expected to take direct&#10;objects, but verbs seen with direct objects are not expected to occur&#10;intransitively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.05557" label="2203.05557">
        <attvalues>
          <attvalue for="0" value="Conditional Prompt Learning for Vision-Language Models" />
          <attvalue for="1" value="  With the rise of powerful pre-trained vision-language models like CLIP, it&#10;becomes essential to investigate ways to adapt these models to downstream&#10;datasets. A recently proposed method named Context Optimization (CoOp)&#10;introduces the concept of prompt learning -- a recent trend in NLP -- to the&#10;vision domain for adapting pre-trained vision-language models. Specifically,&#10;CoOp turns context words in a prompt into a set of learnable vectors and, with&#10;only a few labeled images for learning, can achieve huge improvements over&#10;intensively-tuned manual prompts. In our study we identify a critical problem&#10;of CoOp: the learned context is not generalizable to wider unseen classes&#10;within the same dataset, suggesting that CoOp overfits base classes observed&#10;during training. To address the problem, we propose Conditional Context&#10;Optimization (CoCoOp), which extends CoOp by further learning a lightweight&#10;neural network to generate for each image an input-conditional token (vector).&#10;Compared to CoOp's static prompts, our dynamic prompts adapt to each instance&#10;and are thus less sensitive to class shift. Extensive experiments show that&#10;CoCoOp generalizes much better than CoOp to unseen classes, even showing&#10;promising transferability beyond a single dataset; and yields stronger domain&#10;generalization performance as well. Code is available at&#10;https://github.com/KaiyangZhou/CoOp.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;\@startsection{paragraph} {4} {\z@} {.5em \@plus1ex \@minus.2ex} {-1.5em} {\normalfont\bfseries} {Vision-Language Models}&#10;We mainly review studies focused on aligning images and texts to learn a joint embedding space~\cite{radford2021learning,jia2021scaling,zhang2020contrastive}. The idea of cross-modality alignment is certainly not new and has been investigated since nearly a decade ago---though with dramatically different technologies than today.&#10;&#10;A typical vision-language model consists of three key elements: two for image and text encoding while the third is related to the design of loss functions. In early days, models for processing images and texts are often designed and also learned independently, with their outputs connected by extra modules (losses) for alignment. Images are often encoded using hand-crafted descriptors~\cite{socher2013zero,elhoseiny2013write} or neural networks~\cite{frome2013devise,lei2015predicting}, while texts are encoded using, for instance, pre-trained word vectors~\cite{socher2013zero,frome2013devise} or the frequency-based TF-IDF features~\cite{elhoseiny2013write,lei2015predicting}. In terms of cross-modality alignment, common approaches include metric learning~\cite{frome2013devise}, multi-label classification~\cite{joulin2016learning,gomez2017self}, and n-gram language learning~\cite{li2017learning}. Recently, a study suggests that training the vision part with an image captioning loss can make the visual representation more transferable~\cite{desai2021virtex}.&#10;&#10;Recent vision-language models~\cite{radford2021learning,jia2021scaling,furst2021cloob,li2021supervision} bridge the two modalities by learning two encoders jointly. Also, the models are now built with much larger neural networks. As discussed in Zhou et al.~\cite{zhou2021coop}, recent successes in vision-language models are mainly attributed to the developments in \romannumeral 1) Transformers~\cite{vaswani2017attention}, \romannumeral 2) contrastive representation learning~\cite{chen2020simple,he2020momentum,henaff2020data}, and \romannumeral 3) web-scale training datasets~\cite{radford2021learning,jia2021scaling}. A representative approach is CLIP~\cite{radford2021learning}, which trains two neural network-based encoders using a contrastive loss to match pairs of images and texts. After consuming 400 million data pairs, the CLIP model demonstrates a remarkable zero-shot image recognition capability. Similar to CoOp~\cite{zhou2021coop}, our approach is orthogonal to the research of CLIP-like models~\cite{radford2021learning,jia2021scaling,furst2021cloob,li2021supervision}, aiming to offer an efficient solution for adapting pre-trained vision-language models to downstream applications.&#10;&#10;\@startsection{paragraph} {4} {\z@} {.5em \@plus1ex \@minus.2ex} {-1.5em} {\normalfont\bfseries} {Prompt Learning}&#10;This topic originates from the NLP domain. The motivation was to view pre-trained language models, such as BERT~\cite{devlin2019bert} or GPT~\cite{radford2019language}, as knowledge bases from which information useful to downstream tasks is elicited~\cite{petroni2019language}. Concretely, given a pre-trained language model, the task is often formulated as a ``fill-in-the-blank'' cloze test, such as asking the model to predict the masked token in ``No reason to watch. \underline{It was} [MASK]'' as either ``positive'' or ``negative'' for sentiment classification. The key lies in how to design the underlined part, known as prompt (template), in such a format familiar to the model.&#10;&#10;Instead of manually designing a prompt, research in prompt learning aims to automate the process with the help of affordable-sized labeled data. Jiang et al.~\cite{jiang2020can} use text mining and paraphrasing to generate a group of candidate prompts, within which the optimal ones are chosen to have the highest training accuracy. Shin et al.~\cite{shin2020autoprompt} propose AutoPrompt, a gradient-based approach that selects from a vocabulary the best tokens that cause the greatest changes in gradients based on the label likelihood. Our research is most related to continuous prompt learning methods~\cite{zhong2021factual,li2021prefix,lester2021power}, where the main idea is to turn a prompt into a set of continuous vectors that can be end-to-end optimized with respect to an objective function. See Liu et al.~\cite{liu2021pre} for a more comprehensive survey.&#10;&#10;In computer vision, prompt learning is a nascent research direction that has only been explored very recently~\cite{zhou2021coop,yao2021cpt,rao2022denseclip,ju2021prompting,zhang2021pointclip}. Our research is built on top of CoOp~\cite{zhou2021coop}, which is the earliest work to bring continuous prompt learning to the vision domain for adaptation of pre-trained vision-language models. Crucially, our approach solves the weak generalizability problem of CoOp~\cite{zhou2021coop}, based on a simple idea of conditional prompt learning---which to our knowledge is also novel in the context of NLP and thus could be of interest to the NLP community as well.&#10;&#10;\@startsection{paragraph} {4} {\z@} {.5em \@plus1ex \@minus.2ex} {-1.5em} {\normalfont\bfseries} {Zero-Shot Learning (ZSL)}&#10;is another relevant research area where the goal is similar to ours, i.e., to recognize novel classes by training only on base classes~\cite{wang2019survey,xian2017zero,chao2016empirical,yi2022exploring}. Moreover, the generalization problem where a model trained on base classes often fails on novel classes is also linked to the ``seen-class bias'' issue raised in the ZSL literature~\cite{xian2017zero}. The most common approach to ZSL is to learn a semantic space based on auxiliary information such as attributes~\cite{huynh2020fine} or word embeddings~\cite{frome2013devise,wang2018zero}. Different from existing ZSL methods, our work addresses the emerging problem of adapting large vision-language models and uses drastically different techniques based on prompting.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Computer Vision, Prompt Learning Optimization, Mathematics, Natural Language Processing, Vision Language Models, Domain Generalization" />
        </attvalues>
      </node>
      <node id="2404.03865" label="2404.03865">
        <attvalues>
          <attvalue for="0" value="FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed&#10;  Forward Skipping" />
          <attvalue for="1" value="  Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent&#10;achieving remarkable success in language understanding and generation. However,&#10;such impressive capability typically comes with a substantial model size, which&#10;presents significant challenges for autoregressive token-by-token generation.&#10;To mitigate computation overload incurred during generation, several early-exit&#10;and layer-dropping strategies have been proposed. Despite some promising&#10;success due to the redundancy across LLMs layers on metrics like Rough-L/BLUE,&#10;our careful knowledge-intensive evaluation unveils issues such as generation&#10;collapse, hallucination of wrong facts, and noticeable performance drop even at&#10;the trivial exit ratio of 10-15% of layers. We attribute these errors primarily&#10;to ineffective handling of the KV cache through state copying during&#10;early-exit. In this work, we observed the saturation of computationally&#10;expensive feed-forward blocks of LLM layers and proposed FFN-SkipLLM, which is&#10;a novel fine-grained skip strategy of autoregressive LLMs. More specifically,&#10;FFN-SkipLLM is an input-adaptive feed-forward skipping strategy that can skip&#10;25-30% of FFN blocks of LLMs with marginal change in performance on&#10;knowledge-intensive generation tasks without any requirement to handle KV&#10;cache. Our extensive experiments and ablation across benchmarks like MT-Bench,&#10;Factoid-QA, and variable-length text summarization illustrate how our simple&#10;and ease-at-use method can facilitate faster autoregressive decoding.&#10;" />
          <attvalue for="2" value="&#10;Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks. However, their widespread adoption is hindered by their massive scale, characterized by billions of parameters, which demand exceedingly high computational resources and memory capacities. For instance, the GPT-175B model necessitates 325 GB of GPU memory for loading its weights and relies on a minimum of five A100 (80GB) GPUs employing sophisticated parallelism techniques \cite{sheng2023high}. This imposing computational and memory requirement presents a challenge to the broader accessibility of these models. &#10;&#10;To alleviate the demanding hardware requirements for deploying massive trained models, considerable efforts have been taking to mitigate their high computational inference cost resulting from token-by-token generation. Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}. These methods exploit existing redundancy across LLMs layers which can be ignored during token-by-token generation significantly saving massive computation involved within a layer (e.g., $\sim$ 200-300 million parameters in a single LLaMa layer). Although the proposed methods have shown some promising success, their performance is widely restricted by the issue of inappropriately handling KV caching. KV caching saves keys and values of all attention layers for previously generated tokens and accelerates sequence generation by reducing redundant computation (though at the cost of higher memory usage). Given a token is generated via early exiting, its KV caches in subsequent layers are incomplete which impedes the generation of future tokens beyond the exiting layer of the current token.&#10;&#10;\begin{wraptable}{r}{5cm}&#10;&#10;\end{wraptable}&#10;&#10;For handling KV cache issue, some recent works \cite{elbayad2019depth,calm,li2021accelerating,chen2023ee,del2023skipdecode} proposes three main solutions: copying hidden states, pre-fixed token-level skip pattern, and KV recomputation. Despite these mitigation methods, our careful knowledge-intensive investigation reveals that layer-skipping induces permanent damage due to deviation from the inference process that the model is trained to excel at, leading to significant hallucination of wrong facts and token generation collapse. Figure \ref{fig:ffn_skipllm_motivation} shows the comparison of the responses generated by two recent Layer Skipping methods, namely SkipDecode \cite{del2023skipdecode} and ShortGPT \cite{men2024shortgpt} for a knowledge-intensive QA example. In response, both ShortGPT and SkipDeocde fail to generate the correct answer ``Narendra Modi&quot; and suffer from token collapse and hallucinate misinformation. &#10;&#10;In this work, we ask an interesting unexplored question: Instead of attempting to fix the KV cache, can we completely circumvent the KV cache bottleneck of layer-skipping and still ignore unnecessary computational expenses while mitigating hallucination and token generation collapse? To this end, our work is the \underline{first} attempt to investigate a fine-grained layer-skipping strategy that focuses on computationally expensive feed-forward network (FFN) blocks in LLMs. Table \ref{tab:parameter_count} presents the parameter counts of individual components of LLaMa-7B layer and it can be observed that FFN blocks hold approximately two-third of the parameter budget of the layer, marking them as favorable candidates for skipping during token-by-token generation. Our work derives its motivation from two primary observations: \circled{1} we find a monotonically increasing cosine similarity between the tensors generated before and after the FFN blocks across layers in LLMs which indicates unnecessary computation performed by these blocks, \circled{2} due to the observed phenomenon of attention sink \cite{xiao2023efficient}, we found that allowing a small fraction of first-few token ($\sim$ 5-10\% of maximum sequence length) decoding using full strength (no-skip) of LLMs can significantly help in stabilizing the KV cache, paving way for skipping FFN blocks without significant performance degradation for later tokens. We propose FFN-SkipLLM, a novel fine-grained skip strategy of autoregressive LLMs which is an input-adaptive&#10;feed-forward skipping strategy that can skip $\sim$ 25-30\% of FFN blocks of LLMs&#10;with marginal change in performance on knowledge-intensive tasks. Note that because we only skip FFN blocks, we in-turn can fully circumvent the KV cache issue associated with layer-skipping. Our primary contributions can be unfolded as:&#10;&#10;\begin{itemize}&#10; \item Unlike prior layer-skipping methods, we focus on only skipping computationally expensive FFN blocks based on our observation of their monotonically increasing saturation within the middle layers of LLMs.&#10;&#10;\item Our proposed FFN-SkipLLM uses a simple cosine similarity metric across tensors to capture the trend of FFN saturation and decide an input-adaptive skipping of FFN blocks. More specifically, once a similarity threshold is reached, given the monotonically increasing saturation, we greedily select the next $k$ layers whose FFN blocks can be ignored depending on the desired skipping requirement. &#10;&#10;\item Our extensive knowledge-intensive experiments such as Factoid-QA, Multi-turn conversations and Variable-length in-context text summarization, reveal that FFN-SkipLLM can skip $\sim$ 25-30\% of FFN blocks of LLMs with a marginal change in performance and reduce hallucination and token collapse. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Autoregressive Generation, Deep Learning Efficiency, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2308.12241" label="2308.12241">
        <attvalues>
          <attvalue for="0" value="LLMRec: Benchmarking Large Language Models on Recommendation Task" />
          <attvalue for="1" value="  Recently, the fast development of Large Language Models (LLMs) such as&#10;ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of&#10;conversational models. However, the application of LLMs in the recommendation&#10;domain has not been thoroughly investigated. To bridge this gap, we propose&#10;LLMRec, a LLM-based recommender system designed for benchmarking LLMs on&#10;various recommendation tasks. Specifically, we benchmark several popular&#10;off-the-shelf LLMs, such as ChatGPT, LLaMA, ChatGLM, on five recommendation&#10;tasks, including rating prediction, sequential recommendation, direct&#10;recommendation, explanation generation, and review summarization. Furthermore,&#10;we investigate the effectiveness of supervised finetuning to improve LLMs'&#10;instruction compliance ability. The benchmark results indicate that LLMs&#10;displayed only moderate proficiency in accuracy-based tasks such as sequential&#10;and direct recommendation. However, they demonstrated comparable performance to&#10;state-of-the-art methods in explainability-based tasks. We also conduct&#10;qualitative evaluations to further evaluate the quality of contents generated&#10;by different models, and the results show that LLMs can truly understand the&#10;provided information and generate clearer and more reasonable results. We&#10;aspire that this benchmark will serve as an inspiration for researchers to&#10;delve deeper into the potential of LLMs in enhancing recommendation&#10;performance. Our codes, processed data and benchmark results are available at&#10;https://github.com/williamliujl/LLMRec.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2307.13692" label="2307.13692">
        <attvalues>
          <attvalue for="0" value="ARB: Advanced Reasoning Benchmark for Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated remarkable performance on&#10;various quantitative reasoning and knowledge benchmarks. However, many of these&#10;benchmarks are losing utility as LLMs get increasingly high scores, despite not&#10;yet reaching expert performance in these domains. We introduce ARB, a novel&#10;benchmark composed of advanced reasoning problems in multiple fields. ARB&#10;presents a more challenging test than prior benchmarks, featuring problems in&#10;mathematics, physics, biology, chemistry, and law. As a subset of ARB, we&#10;introduce a challenging set of math and physics problems which require advanced&#10;symbolic reasoning and domain knowledge. We evaluate recent models such as&#10;GPT-4 and Claude on ARB and demonstrate that current models score well below&#10;50% on more demanding tasks. In order to improve both automatic and assisted&#10;evaluation capabilities, we introduce a rubric-based evaluation approach,&#10;allowing GPT-4 to score its own intermediate reasoning steps. Further, we&#10;conduct a human evaluation of the symbolic subset of ARB, finding promising&#10;agreement between annotators and GPT-4 rubric evaluation scores.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.15426" label="2110.15426">
        <attvalues>
          <attvalue for="0" value="RadBERT-CL: Factually-Aware Contrastive Learning For Radiology Report&#10;  Classification" />
          <attvalue for="1" value="  Radiology reports are unstructured and contain the imaging findings and&#10;corresponding diagnoses transcribed by radiologists which include clinical&#10;facts and negated and/or uncertain statements. Extracting pathologic findings&#10;and diagnoses from radiology reports is important for quality control,&#10;population health, and monitoring of disease progress. Existing works,&#10;primarily rely either on rule-based systems or transformer-based pre-trained&#10;model fine-tuning, but could not take the factual and uncertain information&#10;into consideration, and therefore generate false-positive outputs. In this&#10;work, we introduce three sedulous augmentation techniques which retain factual&#10;and critical information while generating augmentations for contrastive&#10;learning. We introduce RadBERT-CL, which fuses these information into BlueBert&#10;via a self-supervised contrastive loss. Our experiments on MIMIC-CXR show&#10;superior performance of RadBERT-CL on fine-tuning for multi-class, multi-label&#10;report classification. We illustrate that when few labeled data are available,&#10;RadBERT-CL outperforms conventional SOTA transformers (BERT/BlueBert) by&#10;significantly larger margins (6-11%). We also show that the representations&#10;learned by RadBERT-CL can capture critical medical information in the latent&#10;space.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13655" label="2305.13655">
        <attvalues>
          <attvalue for="0" value="LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image&#10;  Diffusion Models with Large Language Models" />
          <attvalue for="1" value="  Recent advancements in text-to-image diffusion models have yielded impressive&#10;results in generating realistic and diverse images. However, these models still&#10;struggle with complex prompts, such as those that involve numeracy and spatial&#10;reasoning. This work proposes to enhance prompt understanding capabilities in&#10;diffusion models. Our method leverages a pretrained large language model (LLM)&#10;for grounded generation in a novel two-stage process. In the first stage, the&#10;LLM generates a scene layout that comprises captioned bounding boxes from a&#10;given prompt describing the desired image. In the second stage, a novel&#10;controller guides an off-the-shelf diffusion model for layout-grounded image&#10;generation. Both stages utilize existing pretrained models without additional&#10;model parameter optimization. Our method significantly outperforms the base&#10;diffusion model and several strong baselines in accurately generating images&#10;according to prompts that require various capabilities, doubling the generation&#10;accuracy across four tasks on average. Furthermore, our method enables&#10;instruction-based multi-round scene specification and can handle prompts in&#10;languages not supported by the underlying diffusion model. We anticipate that&#10;our method will unleash users' creativity by accurately following more complex&#10;prompts. Our code, demo, and benchmark are available at:&#10;https://llm-grounded-diffusion.github.io&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11175" label="2305.11175">
        <attvalues>
          <attvalue for="0" value="VisionLLM: Large Language Model is also an Open-Ended Decoder for&#10;  Vision-Centric Tasks" />
          <attvalue for="1" value="  Large language models (LLMs) have notably accelerated progress towards&#10;artificial general intelligence (AGI), with their impressive zero-shot capacity&#10;for user-tailored tasks, endowing them with immense potential across a range of&#10;applications. However, in the field of computer vision, despite the&#10;availability of numerous powerful vision foundation models (VFMs), they are&#10;still restricted to tasks in a pre-defined form, struggling to match the&#10;open-ended task capabilities of LLMs. In this work, we present an LLM-based&#10;framework for vision-centric tasks, termed VisionLLM. This framework provides a&#10;unified perspective for vision and language tasks by treating images as a&#10;foreign language and aligning vision-centric tasks with language tasks that can&#10;be flexibly defined and managed using language instructions. An LLM-based&#10;decoder can then make appropriate predictions based on these instructions for&#10;open-ended tasks. Extensive experiments show that the proposed VisionLLM can&#10;achieve different levels of task customization through language instructions,&#10;from fine-grained object-level to coarse-grained task-level customization, all&#10;with good results. It's noteworthy that, with a generalist LLM-based framework,&#10;our model can achieve over 60\% mAP on COCO, on par with detection-specific&#10;models. We hope this model can set a new baseline for generalist vision and&#10;language models. The demo shall be released based on&#10;https://github.com/OpenGVLab/InternGPT. The code shall be released at&#10;https://github.com/OpenGVLab/VisionLLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.00692" label="2308.00692">
        <attvalues>
          <attvalue for="0" value="LISA: Reasoning Segmentation via Large Language Model" />
          <attvalue for="1" value="  Although perception systems have made remarkable advancements in recent&#10;years, they still rely on explicit human instruction or pre-defined categories&#10;to identify the target objects before executing visual recognition tasks. Such&#10;systems cannot actively reason and comprehend implicit user intention. In this&#10;work, we propose a new segmentation task -- reasoning segmentation. The task is&#10;designed to output a segmentation mask given a complex and implicit query text.&#10;Furthermore, we establish a benchmark comprising over one thousand&#10;image-instruction-mask data samples, incorporating intricate reasoning and&#10;world knowledge for evaluation purposes. Finally, we present LISA: large&#10;Language Instructed Segmentation Assistant, which inherits the language&#10;generation capabilities of multimodal Large Language Models (LLMs) while also&#10;possessing the ability to produce segmentation masks. We expand the original&#10;vocabulary with a &lt;SEG&gt; token and propose the embedding-as-mask paradigm to&#10;unlock the segmentation capability. Remarkably, LISA can handle cases involving&#10;complex reasoning and world knowledge. Also, it demonstrates robust zero-shot&#10;capability when trained exclusively on reasoning-free datasets. In addition,&#10;fine-tuning the model with merely 239 reasoning segmentation data samples&#10;results in further performance enhancement. Both quantitative and qualitative&#10;experiments show our method effectively unlocks new reasoning segmentation&#10;capabilities for multimodal LLMs. Code, models, and data are available at&#10;https://github.com/dvlab-research/LISA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.09842" label="2304.09842">
        <attvalues>
          <attvalue for="0" value="Chameleon: Plug-and-Play Compositional Reasoning with Large Language&#10;  Models" />
          <attvalue for="1" value="  Large language models (LLMs) have achieved remarkable progress in solving&#10;various natural language processing tasks due to emergent reasoning abilities.&#10;However, LLMs have inherent limitations as they are incapable of accessing&#10;up-to-date information (stored on the Web or in task-specific knowledge bases),&#10;using external tools, and performing precise mathematical and logical&#10;reasoning. In this paper, we present Chameleon, an AI system that mitigates&#10;these limitations by augmenting LLMs with plug-and-play modules for&#10;compositional reasoning. Chameleon synthesizes programs by composing various&#10;tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python&#10;functions, and heuristic-based modules) for accomplishing complex reasoning&#10;tasks. At the heart of Chameleon is an LLM-based planner that assembles a&#10;sequence of tools to execute to generate the final response. We showcase the&#10;effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning&#10;tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54%&#10;overall accuracy on ScienceQA, improving the best published few-shot result by&#10;11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%,&#10;lifting the state of the art to 98.78%. Our analysis also shows that the&#10;GPT-4-powered planner exhibits more consistent and rational tool selection via&#10;inferring potential constraints from instructions, compared to a&#10;ChatGPT-powered planner. The project is available at&#10;https://chameleon-llm.github.io.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.02565" label="2308.02565">
        <attvalues>
          <attvalue for="0" value="SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning" />
          <attvalue for="1" value="  Textual graphs (TGs) are graphs whose nodes correspond to text (sentences or&#10;documents), which are widely prevalent. The representation learning of TGs&#10;involves two stages: (i) unsupervised feature extraction and (ii) supervised&#10;graph representation learning. In recent years, extensive efforts have been&#10;devoted to the latter stage, where Graph Neural Networks (GNNs) have dominated.&#10;However, the former stage for most existing graph benchmarks still relies on&#10;traditional feature engineering techniques. More recently, with the rapid&#10;development of language models (LMs), researchers have focused on leveraging&#10;LMs to facilitate the learning of TGs, either by jointly training them in a&#10;computationally intensive framework (merging the two stages), or designing&#10;complex self-supervised training tasks for feature extraction (enhancing the&#10;first stage). In this work, we present SimTeG, a frustratingly Simple approach&#10;for Textual Graph learning that does not innovate in frameworks, models, and&#10;tasks. Instead, we first perform supervised parameter-efficient fine-tuning&#10;(PEFT) on a pre-trained LM on the downstream task, such as node classification.&#10;We then generate node embeddings using the last hidden states of finetuned LM.&#10;These derived features can be further utilized by any GNN for training on the&#10;same task. We evaluate our approach on two fundamental graph representation&#10;learning tasks: node classification and link prediction. Through extensive&#10;experiments, we show that our approach significantly improves the performance&#10;of various GNNs on multiple graph benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.08170" label="2402.08170">
        <attvalues>
          <attvalue for="0" value="LLaGA: Large Language and Graph Assistant" />
          <attvalue for="1" value="  Graph Neural Networks (GNNs) have empowered the advance in graph-structured&#10;data analysis. Recently, the rise of Large Language Models (LLMs) like GPT-4&#10;has heralded a new era in deep learning. However, their application to graph&#10;data poses distinct challenges due to the inherent difficulty of translating&#10;graph structures to language. To this end, we introduce the Large Language and&#10;Graph Assistant (LLaGA), an innovative model that effectively integrates LLM&#10;capabilities to handle the complexities of graph-structured data. LLaGA retains&#10;the general-purpose nature of LLMs while adapting graph data into a format&#10;compatible with LLM input. LLaGA achieves this by reorganizing graph nodes to&#10;structure-aware sequences and then mapping these into the token embedding space&#10;through a versatile projector. LLaGA excels in versatility, generalizability&#10;and interpretability, allowing it to perform consistently well across different&#10;datasets and tasks, extend its ability to unseen datasets or tasks, and provide&#10;explanations for graphs. Our extensive experiments across popular graph&#10;benchmarks show that LLaGA delivers outstanding performance across four&#10;datasets and three tasks using one single model, surpassing state-of-the-art&#10;graph models in both supervised and zero-shot scenarios. Our code is available&#10;at \url{https://github.com/VITA-Group/LLaGA}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14152" label="2305.14152">
        <attvalues>
          <attvalue for="0" value="Memory-Efficient Fine-Tuning of Compressed Large Language Models via&#10;  sub-4-bit Integer Quantization" />
          <attvalue for="1" value="  Large language models (LLMs) face the challenges in fine-tuning and&#10;deployment due to their high memory demands and computational costs. While&#10;parameter-efficient fine-tuning (PEFT) methods aim to reduce the memory usage&#10;of the optimizer state during fine-tuning, the inherent size of pre-trained LLM&#10;weights continues to be a pressing concern. Even though quantization techniques&#10;are widely proposed to ease memory demands and accelerate LLM inference, most&#10;of these techniques are geared towards the deployment phase. To bridge this&#10;gap, this paper presents Parameter-Efficient and Quantization-aware Adaptation&#10;(PEQA) - a simple yet effective method that combines the advantages of PEFT&#10;with quantized LLMs. By updating solely the quantization scales, PEQA can be&#10;directly applied to quantized LLMs, ensuring seamless task transitions.&#10;Parallel to existing PEFT methods, PEQA significantly reduces the memory&#10;overhead associated with the optimizer state. Furthermore, it leverages the&#10;advantages of quantization to substantially reduce model sizes. Even after&#10;fine-tuning, the quantization structure of a PEQA-tuned LLM remains intact,&#10;allowing for accelerated inference on the deployment stage. We employ&#10;PEQA-tuning for task-specific adaptation on LLMs with up to 65 billion&#10;parameters. To assess the logical reasoning and language comprehension of&#10;PEQA-tuned LLMs, we fine-tune low-bit quantized LLMs using a instruction&#10;dataset. Our results show that even when LLMs are quantized to below 4-bit&#10;precision, their capabilities in language modeling, few-shot in-context&#10;learning, and comprehension can be resiliently restored to (or even improved&#10;over) their full-precision original performances with PEQA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.00978" label="2306.00978">
        <attvalues>
          <attvalue for="0" value="AWQ: Activation-aware Weight Quantization for LLM Compression and&#10;  Acceleration" />
          <attvalue for="1" value="  Large language models (LLMs) have transformed numerous AI applications.&#10;On-device LLM is becoming increasingly important: running LLMs locally on edge&#10;devices can reduce the cloud computing cost and protect users' privacy.&#10;However, the astronomical model size and the limited hardware resource pose&#10;significant deployment challenges. We propose Activation-aware Weight&#10;Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only&#10;quantization. AWQ finds that not all weights in an LLM are equally important.&#10;Protecting only 1% salient weights can greatly reduce quantization error. To&#10;identify salient weight channels, we should refer to the activation&#10;distribution, not weights. To avoid the hardware-inefficient mix-precision&#10;quantization, we mathematically derive that scaling up the salient channels can&#10;reduce the quantization error. AWQ employs an equivalent transformation to&#10;scale the salient weight channels to protect them. The scale is determined by&#10;collecting the activation statistics offline. AWQ does not rely on any&#10;backpropagation or reconstruction, so it generalizes to different domains and&#10;modalities without overfitting the calibration set. AWQ outperforms existing&#10;work on various language modeling and domain-specific benchmarks (coding and&#10;math). Thanks to better generalization, it achieves excellent quantization&#10;performance for instruction-tuned LMs and, for the first time, multi-modal LMs.&#10;Alongside AWQ, we implement TinyChat, an efficient and flexible inference&#10;framework tailored for 4-bit on-device LLM/VLMs. With kernel fusion and&#10;platform-aware weight packing, TinyChat offers more than 3x speedup over the&#10;Huggingface FP16 implementation on both desktop and mobile GPUs. It also&#10;democratizes the deployment of the 70B Llama-2 model on mobile GPUs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.12755" label="2206.12755">
        <attvalues>
          <attvalue for="0" value="Training Your Sparse Neural Network Better with Any Mask" />
          <attvalue for="1" value="  Pruning large neural networks to create high-quality, independently trainable&#10;sparse masks, which can maintain similar performance to their dense&#10;counterparts, is very desirable due to the reduced space and time complexity.&#10;As research effort is focused on increasingly sophisticated pruning methods&#10;that leads to sparse subnetworks trainable from the scratch, we argue for an&#10;orthogonal, under-explored theme: improving training techniques for pruned&#10;sub-networks, i.e. sparse training. Apart from the popular belief that only the&#10;quality of sparse masks matters for sparse training, in this paper we&#10;demonstrate an alternative opportunity: one can carefully customize the sparse&#10;training techniques to deviate from the default dense network training&#10;protocols, consisting of introducing ``ghost&quot; neurons and skip connections at&#10;the early stage of training, and strategically modifying the initialization as&#10;well as labels. Our new sparse training recipe is generally applicable to&#10;improving training from scratch with various sparse masks. By adopting our&#10;newly curated techniques, we demonstrate significant performance gains across&#10;various popular datasets (CIFAR-10, CIFAR-100, TinyImageNet), architectures&#10;(ResNet-18/32/104, Vgg16, MobileNet), and sparse mask options (lottery ticket,&#10;SNIP/GRASP, SynFlow, or even randomly pruning), compared to the default&#10;training protocols, especially at high sparsity levels. Code is at&#10;https://github.com/VITA-Group/ToST&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.10460" label="2306.10460">
        <attvalues>
          <attvalue for="0" value="Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery&#10;  Tickets from Large Models" />
          <attvalue for="1" value="  Large pre-trained transformers have been receiving explosive attention in the&#10;past few years, due to their wide adaptability for numerous downstream&#10;applications via fine-tuning, but their exponentially increasing parameter&#10;counts are becoming a primary hurdle to even just fine-tune them without&#10;industry-standard hardware. Recently, Lottery Ticket Hypothesis (LTH) and its&#10;variants, have been exploited to prune these large pre-trained models&#10;generating subnetworks that can achieve similar performance as their dense&#10;counterparts, but LTH pragmatism is enormously inhibited by repetitive full&#10;training and pruning routine of iterative magnitude pruning (IMP) which worsens&#10;with increasing model size. Motivated by the recent observations of model&#10;soups, which suggest that fine-tuned weights of multiple models can be merged&#10;to a better minima, we propose Instant Soup Pruning (ISP) to generate lottery&#10;ticket quality subnetworks, using a fraction of the original IMP cost by&#10;replacing the expensive intermediate pruning stages of IMP with computationally&#10;efficient weak mask generation and aggregation routine. More specifically,&#10;during the mask generation stage, ISP takes a small handful of iterations using&#10;varying training protocols and data subsets to generate many weak and noisy&#10;subnetworks, and superpose them to average out the noise creating a&#10;high-quality denoised subnetwork. Our extensive experiments and ablation on two&#10;popular large-scale pre-trained models: CLIP (unexplored in pruning till date)&#10;and BERT across multiple benchmark vision and language datasets validate the&#10;effectiveness of ISP compared to several state-of-the-art pruning methods.&#10;Codes are available at: \url{https://github.com/VITA-Group/instant_soup}&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.03805" label="2306.03805">
        <attvalues>
          <attvalue for="0" value="The Emergence of Essential Sparsity in Large Pre-trained Models: The&#10;  Weights that Matter" />
          <attvalue for="1" value="  Large pre-trained transformers are show-stealer in modern-day deep learning,&#10;and it becomes crucial to comprehend the parsimonious patterns that exist&#10;within them as they grow in scale. With exploding parameter counts, Lottery&#10;Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in&#10;sparsifying them due to high computation and memory bottleneck of repetitive&#10;train-prune-retrain routine of iterative magnitude pruning (IMP) which worsens&#10;with increasing model size. This paper comprehensively studies induced sparse&#10;patterns across multiple large pre-trained vision and language transformers. We&#10;propose the existence of -- essential sparsity defined with a sharp dropping&#10;point beyond which the performance declines much faster w.r.t the rise of&#10;sparsity level, when we directly remove weights with the smallest magnitudes in&#10;one-shot without re-training. We also find essential sparsity to hold valid for&#10;N:M sparsity patterns as well as on modern-scale large language models&#10;(Vicuna-7B). We also present an intriguing emerging phenomenon of abrupt&#10;sparsification during the pre-training of BERT, i.e., BERT suddenly becomes&#10;heavily sparse in pre-training after certain iterations. Moreover, our&#10;observations also indicate a counter-intuitive finding that BERT trained with a&#10;larger amount of pre-training data tends to have a better ability to condense&#10;knowledge in comparatively relatively fewer parameters. Lastly, we investigate&#10;the effect of the pre-training loss on essential sparsity and discover that&#10;self-supervised learning (SSL) objectives trigger stronger emergent&#10;sparsification properties than supervised learning (SL). Our codes are&#10;available at \url{https://github.com/VITA-Group/essential_sparsity}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.02141" label="2303.02141">
        <attvalues>
          <attvalue for="0" value="Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!" />
          <attvalue for="1" value="  Sparse Neural Networks (SNNs) have received voluminous attention&#10;predominantly due to growing computational and memory footprints of&#10;consistently exploding parameter count in large-scale models. Similar to their&#10;dense counterparts, recent SNNs generalize just as well and are equipped with&#10;numerous favorable benefits (e.g., low complexity, high scalability, and&#10;robustness), sometimes even better than the original dense networks. As&#10;research effort is focused on developing increasingly sophisticated sparse&#10;algorithms, it is startling that a comprehensive benchmark to evaluate the&#10;effectiveness of these algorithms has been highly overlooked. In absence of a&#10;carefully crafted evaluation benchmark, most if not all, sparse algorithms are&#10;evaluated against fairly simple and naive tasks (eg. CIFAR, ImageNet, GLUE,&#10;etc.), which can potentially camouflage many advantages as well unexpected&#10;predicaments of SNNs. In pursuit of a more general evaluation and unveiling the&#10;true potential of sparse algorithms, we introduce &quot;Sparsity May Cry&quot; Benchmark&#10;(SMC-Bench), a collection of carefully-curated 4 diverse tasks with 10&#10;datasets, that accounts for capturing a wide range of domain-specific and&#10;sophisticated knowledge. Our systemic evaluation of the most representative&#10;sparse algorithms reveals an important obscured observation: the&#10;state-of-the-art magnitude- and/or gradient-based sparse algorithms seemingly&#10;fail to perform on SMC-Bench when applied out-of-the-box, sometimes at&#10;significantly trivial sparsity as low as 5%. By incorporating these&#10;well-thought and diverse tasks, SMC-Bench is designed to favor and encourage&#10;the development of more scalable and generalizable sparse algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.02277" label="2310.02277">
        <attvalues>
          <attvalue for="0" value="Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs&#10;  &quot;Difficult&quot; Downstream Tasks in LLMs" />
          <attvalue for="1" value="  We present Junk DNA Hypothesis by adopting a novel task-centric angle for the&#10;pre-trained weights of large language models (LLMs). It has been believed that&#10;weights in LLMs contain significant redundancy, leading to the conception that&#10;a considerable chunk of the parameters can be removed by pruning without&#10;compromising performance. Contrary to this belief, this paper presents a&#10;counter-argument: small-magnitude weights of pre-trained model weights encode&#10;vital knowledge essential for tackling difficult downstream tasks - manifested&#10;as the monotonic relationship between the performance drop of downstream tasks&#10;across the difficulty spectrum, as we prune more pre-trained weights by&#10;magnitude. Moreover, we reveal that these seemingly inconsequential weights can&#10;result in irreparable loss of knowledge and performance degradation in&#10;difficult tasks, even when downstream continual training is allowed.&#10;Interestingly, our evaluations show that the other popular compression, namely&#10;quantization, fails to exhibit similar monotonic effect and does not as&#10;convincingly disentangle this task-difficulty information. To study formally,&#10;we introduce several quantifiable metrics to gauge the downstream task&#10;difficulty: (1) within the same task category, and (2) across different task&#10;categories. Our extensive experiments substantiate the Junk DNA Hypothesis&#10;across a diverse range of model sizes, tasks, datasets, and even pruning&#10;methods. Codes are available at:&#10;https://github.com/VITA-Group/Junk_DNA_Hypothesis.git.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05175" label="2310.05175">
        <attvalues>
          <attvalue for="0" value="Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for&#10;  Pruning LLMs to High Sparsity" />
          <attvalue for="1" value="  Large Language Models (LLMs), renowned for their remarkable performance&#10;across diverse domains, present a challenge when it comes to practical&#10;deployment due to their colossal model size. In response to this challenge,&#10;efforts have been directed toward the application of traditional network&#10;pruning techniques to LLMs, uncovering a massive number of parameters that can&#10;be pruned in one-shot without hurting performance. Prevailing LLM pruning&#10;strategies have consistently adhered to the practice of uniformly pruning all&#10;layers at equivalent sparsity, resulting in robust performance. However, this&#10;observation stands in contrast to the prevailing trends observed in the field&#10;of vision models, where non-uniform layerwise sparsity typically yields&#10;stronger results. To understand the underlying reasons for this disparity, we&#10;conduct a comprehensive study and discover a strong correlation with the&#10;emergence of activation outliers in LLMs. Inspired by this finding, we&#10;introduce a novel LLM pruning methodology that incorporates a tailored set of&#10;non-uniform layerwise sparsity ratios, termed as Outlier Weighed Layerwise&#10;sparsity (OWL). The sparsity ratio of OWL is proportional to the outlier ratio&#10;observed within each layer, facilitating a more effective alignment between&#10;layerwise weight sparsity and outlier ratios. Our empirical evaluation,&#10;conducted across the LLaMA-V1 family and OPT, spanning various benchmarks,&#10;demonstrates the distinct advantages offered by OWL over previous methods. For&#10;instance, OWL exhibits a remarkable performance gain, surpassing the&#10;state-of-the-art Wanda and SparseGPT by 61.22 and 6.80 perplexity at a high&#10;sparsity level of 70%, respectively, while delivering 2.6x end-to-end inference&#10;speed-up in the DeepSparse inference engine. Codes are available at&#10;https://github.com/luuyin/OWL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.15101" label="2109.15101">
        <attvalues>
          <attvalue for="0" value="Compositional generalization in semantic parsing with pretrained&#10;  transformers" />
          <attvalue for="1" value="  Large-scale pretraining instills large amounts of knowledge in deep neural&#10;networks. This, in turn, improves the generalization behavior of these models&#10;in downstream tasks. What exactly are the limits to the generalization benefits&#10;of large-scale pretraining? Here, we report observations from some simple&#10;experiments aimed at addressing this question in the context of two semantic&#10;parsing tasks involving natural language, SCAN and COGS. We show that language&#10;models pretrained exclusively with non-English corpora, or even with&#10;programming language corpora, significantly improve out-of-distribution&#10;generalization in these benchmarks, compared with models trained from scratch,&#10;even though both benchmarks are English-based. This demonstrates the&#10;surprisingly broad transferability of pretrained representations and knowledge.&#10;Pretraining with a large-scale protein sequence prediction task, on the other&#10;hand, mostly deteriorates the generalization performance in SCAN and COGS,&#10;suggesting that pretrained representations do not transfer universally and that&#10;there are constraints on the similarity between the pretraining and downstream&#10;domains for successful transfer. Finally, we show that larger models are harder&#10;to train from scratch and their generalization accuracy is lower when trained&#10;up to convergence on the relatively small SCAN and COGS datasets, but the&#10;benefits of large-scale pretraining become much clearer with larger models.&#10;" />
          <attvalue for="2" value="&#10;\epigraph{here is the root of the root and the bud of the bud&#10;and the sky of the sky of a tree called life}{--- e e cummings}&#10;&#10;Large, pretrained language and vision models are widely used in a diverse range of downstream NLP and computer vision tasks. These models encode a large amount of transferable knowledge (of varying specificity) in their parameters and display remarkable (sometimes even surprising) ``emergent'' generalization behaviors as a result. To give a few examples: (i) the highly influential GPT-3 model displays hallmarks of few-shot learning/generalization ability from a small number of examples given in its prompt \cite{brown2020}; (ii) DALL-E, or other similar text-to-image models trained on large datasets such as CLIP+VQGAN, display qualitative evidence of compositional generalization abilities \cite{ramesh2021,crowson2022}; (iii) pretraining with large, diverse image or text datasets improves the out-of-distribution generalization performance of image recognition \cite{orhan2019,xie2020} and NLP models \cite{hendrycks2020}, respectively.&#10;&#10;What exactly are the limits to the generalization benefits of large-scale pretraining? More concretely, how do pretraining benefits scale with factors such as the diversity and the scale of the pretraining data, the model size, or the similarity between the pretraining data and the downstream task? Here, we report the results of some simple experiments aimed at addressing the latter two factors, namely the model size and the similarity between the pretraining data and the downstream task, in the context of two previously introduced semantic parsing tasks, SCAN \cite{lake2018} and COGS \cite{kim2020}. &#10;&#10;A few other recent works have also attempted to address some of these questions. \cite{furrer2020} and \cite{tay2021} show that large-scale pretraining ---in the form of language modeling on the web-scale C4 corpus \cite{raffel2020} consisting mainly of English language texts--- can improve compositional generalization in SCAN and COGS benchmarks, respectively. With respect to the specificity of pretraining benefits, \cite{lu2021} argue that pretraining with language modeling confers broad benefits in downstream tasks, including, somewhat surprisingly, a variety of non-language tasks such as image recognition or protein sequence modeling. \cite{papadimitriou2020} similarly report downstream natural language modeling tasks can benefit from pretraining in seemingly unrelated domains such Java code or MIDI music scores. They argue that abstract syntactic similarity between the pretraining and downstream domains is key for the transfer success of pretraining. Consistent with this idea, \cite{chiang2020} also show that pretraining with a simple artificial language generated by a stack-based hierarchical grammar can improve the accuracy on a diverse set of downstream natural language tasks, namely the tasks comprising the GLUE benchmark \cite{wang2018}. More recently, \cite{krishna2021} show that text summarization also does not seem to require pretraining with natural language texts: they demonstrate that even pretraining with texts consisting entirely of randomly and independently sampled nonsense words achieves similar results in downstream text summarization tasks. In the visual domain, \cite{baradad2021} show that even very simple random noise processes can be used as effective pretraining data for downstream natural image recognition tasks, as long as these noise processes satisfy certain basic structural properties of natural images. In a similar vein, \cite{sinha2021} recently showed that scrambling the (within-sentence) word order in natural language texts has surprisingly little effect on their effectiveness as pretraining data, as long as some higher-order co-occurrence statistics are preserved.&#10;&#10;The results we report here contribute valuable observations to this prior literature. Our main results can be summarized as follows: &#10;&#10;\begin{itemize}&#10;\item Large-scale pretraining with natural language based text-to-text tasks improve compositional generalization in both SCAN and COGS benchmarks. This result is essentially a replication of earlier reports to the same effect: e.g. \cite{furrer2020} for SCAN and \cite{tay2021} for COGS.&#10;&#10;\item Surprisingly, however, even models pretrained exclusively with non-English languages significantly improve performance on these benchmarks compared to models trained from scratch, even though both benchmarks are English-based. &#10;&#10;\item Even more surprisingly, a language model pretrained predominantly on programming languages \cite{wang2021} also provides large generalization benefits on both SCAN and COGS, roughly equivalent in size to the generalization benefits of large-scale pretraining with natural languages.&#10;&#10;\item However, the same is not true for pretraining with a large-scale protein sequence prediction task \cite{elnaggar2020}. Pretraining with this task in fact mostly deteriorates the performance on the downstream semantic parsing tasks, suggesting that pretrained representations do not transfer universally and that there are likely constraints on the similarity between the pretraining and downstream tasks for successful transfer.&#10;&#10;\item Bigger models are harder to train from scratch (and their generalization accuracy lower when trained to convergence) on the relatively small-scale SCAN and COGS benchmarks, but they benefit more from large-scale pretraining.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Neural Networks, Cognitive Science, Artificial Intelligence, Language Model Pretraining, Transfer Learning" />
        </attvalues>
      </node>
      <node id="2102.12092" label="2102.12092">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Text-to-Image Generation" />
          <attvalue for="1" value="  Text-to-image generation has traditionally focused on finding better modeling&#10;assumptions for training on a fixed dataset. These assumptions might involve&#10;complex architectures, auxiliary losses, or side information such as object&#10;part labels or segmentation masks supplied during training. We describe a&#10;simple approach for this task based on a transformer that autoregressively&#10;models the text and image tokens as a single stream of data. With sufficient&#10;data and scale, our approach is competitive with previous domain-specific&#10;models when evaluated in a zero-shot fashion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.07640" label="1907.07640">
        <attvalues>
          <attvalue for="0" value="Robustness properties of Facebook's ResNeXt WSL models" />
          <attvalue for="1" value="  We investigate the robustness properties of ResNeXt class image recognition&#10;models trained with billion scale weakly supervised data (ResNeXt WSL models).&#10;These models, recently made public by Facebook AI, were trained with ~1B images&#10;from Instagram and fine-tuned on ImageNet. We show that these models display an&#10;unprecedented degree of robustness against common image corruptions and&#10;perturbations, as measured by the ImageNet-C and ImageNet-P benchmarks. They&#10;also achieve substantially improved accuracies on the recently introduced&#10;&quot;natural adversarial examples&quot; benchmark (ImageNet-A). The largest of the&#10;released models, in particular, achieves state-of-the-art results on&#10;ImageNet-C, ImageNet-P, and ImageNet-A by a large margin. The gains on&#10;ImageNet-C, ImageNet-P, and ImageNet-A far outpace the gains on ImageNet&#10;validation accuracy, suggesting the former as more useful benchmarks to measure&#10;further progress in image recognition. Remarkably, the ResNeXt WSL models even&#10;achieve a limited degree of adversarial robustness against state-of-the-art&#10;white-box attacks (10-step PGD attacks). However, in contrast to adversarially&#10;trained models, the robustness of the ResNeXt WSL models rapidly declines with&#10;the number of PGD steps, suggesting that these models do not achieve genuine&#10;adversarial robustness. Visualization of the learned features also confirms&#10;this conclusion. Finally, we show that although the ResNeXt WSL models are more&#10;shape-biased than comparable ImageNet-trained models in a shape-texture cue&#10;conflict experiment, they still remain much more texture-biased than humans,&#10;suggesting that they share some of the underlying characteristics of&#10;ImageNet-trained models that make this benchmark challenging.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.04252" label="1911.04252">
        <attvalues>
          <attvalue for="0" value="Self-training with Noisy Student improves ImageNet classification" />
          <attvalue for="1" value="  We present Noisy Student Training, a semi-supervised learning approach that&#10;works well even when labeled data is abundant. Noisy Student Training achieves&#10;88.4% top-1 accuracy on ImageNet, which is 2.0% better than the&#10;state-of-the-art model that requires 3.5B weakly labeled Instagram images. On&#10;robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to&#10;83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces&#10;ImageNet-P mean flip rate from 27.8 to 12.2.&#10;  Noisy Student Training extends the idea of self-training and distillation&#10;with the use of equal-or-larger student models and noise added to the student&#10;during learning. On ImageNet, we first train an EfficientNet model on labeled&#10;images and use it as a teacher to generate pseudo labels for 300M unlabeled&#10;images. We then train a larger EfficientNet as a student model on the&#10;combination of labeled and pseudo labeled images. We iterate this process by&#10;putting back the student as the teacher. During the learning of the student, we&#10;inject noise such as dropout, stochastic depth, and data augmentation via&#10;RandAugment to the student so that the student generalizes better than the&#10;teacher. Models are available at&#10;https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet.&#10;Code is available at https://github.com/google-research/noisystudent.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05465" label="2010.05465">
        <attvalues>
          <attvalue for="0" value="COGS: A Compositional Generalization Challenge Based on Semantic&#10;  Interpretation" />
          <attvalue for="1" value="  Natural language is characterized by compositionality: the meaning of a&#10;complex expression is constructed from the meanings of its constituent parts.&#10;To facilitate the evaluation of the compositional abilities of language&#10;processing architectures, we introduce COGS, a semantic parsing dataset based&#10;on a fragment of English. The evaluation portion of COGS contains multiple&#10;systematic gaps that can only be addressed by compositional generalization;&#10;these include new combinations of familiar syntactic structures, or new&#10;combinations of familiar words and familiar structures. In experiments with&#10;Transformers and LSTMs, we found that in-distribution accuracy on the COGS test&#10;set was near-perfect (96--99%), but generalization accuracy was substantially&#10;lower (16--35%) and showed high sensitivity to random seed ($\pm$6--8%). These&#10;findings indicate that contemporary standard NLP models are limited in their&#10;compositional generalization capacity, and position COGS as a good way to&#10;measure progress.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.08970" label="2007.08970">
        <attvalues>
          <attvalue for="0" value="Compositional Generalization in Semantic Parsing: Pre-training vs.&#10;  Specialized Architectures" />
          <attvalue for="1" value="  While mainstream machine learning methods are known to have limited ability&#10;to compositionally generalize, new architectures and techniques continue to be&#10;proposed to address this limitation. We investigate state-of-the-art techniques&#10;and architectures in order to assess their effectiveness in improving&#10;compositional generalization in semantic parsing tasks based on the SCAN and&#10;CFQ datasets. We show that masked language model (MLM) pre-training rivals&#10;SCAN-inspired architectures on primitive holdout splits. On a more complex&#10;compositional task, we show that pre-training leads to significant improvements&#10;in performance vs. comparable non-pre-trained models, whereas architectures&#10;proposed to encourage compositional generalization on SCAN or in the area of&#10;algorithm learning fail to lead to significant improvements. We establish a new&#10;state of the art on the CFQ compositional generalization benchmark using MLM&#10;pre-training together with an intermediate representation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.03322" label="2105.03322">
        <attvalues>
          <attvalue for="0" value="Are Pre-trained Convolutions Better than Pre-trained Transformers?" />
          <attvalue for="1" value="  In the era of pre-trained language models, Transformers are the de facto&#10;choice of model architectures. While recent research has shown promise in&#10;entirely convolutional, or CNN, architectures, they have not been explored&#10;using the pre-train-fine-tune paradigm. In the context of language models, are&#10;convolutional models competitive to Transformers when pre-trained? This paper&#10;investigates this research question and presents several interesting findings.&#10;Across an extensive set of experiments on 8 datasets/tasks, we find that&#10;CNN-based pre-trained models are competitive and outperform their Transformer&#10;counterpart in certain scenarios, albeit with caveats. Overall, the findings&#10;outlined in this paper suggest that conflating pre-training and architectural&#10;advances is misguided and that both advances should be considered&#10;independently. We believe our research paves the way for a healthy amount of&#10;optimism in alternative architectures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-training on a large corpus has become the primary method of learning universal language representations to solve different downstream NLP tasks. &#10;The first generation of pre-trained models aimed at learning embedding for words, like Skip-Gram~\cite{mikolov2013distributed} and Glove~\cite{pennington2014glove}, and quickly developed to learning contextualized representation for words, like ELMO~\cite{peters2018deep}, GPT~\cite{radford2018improving}, and BERT~\cite{devlin2018bert}. This, however, is not the only axis in which pre-trained models have evolved. &#10;&#10;Different objective functions and various tasks, both supervised and unsupervised, have been explored for pre-training. For instance, CoVe~\cite{mccann2017learned} uses machine translation as the pre-training task, ELMO~\cite{peters2018deep} and GPT~\cite{radford2018improving} use language modeling objectives, BERT~\cite{devlin2018bert} uses masked language modeling, T5~\cite{raffel2019exploring} and MASS~\cite{song2019mass} use Seq2Seq masked language modeling, and XLNet~\cite{yang2019xlnet} utilizes permuted language modeling. In addition to this, BART~\cite{lewis2019bart} uses a denoising autoencoder setup during pre-training, where the model takes a partially corrupted input and is trained to recover the original, undistorted input. Some models use a contrastive learning setup during pertaining, like replaced token detection, used by ELECTRA~\cite{clark2020electra}, and sentence order prediction, used by ALBERT~\cite{lan2019albert} and StructBERT~\cite{wang2019structbert}.&#10;&#10;Another axis where pre-trained models in NLP explored different ideas is model architecture. ELMO~\cite{peters2018deep} and CoVe~\cite{mccann2017learned} used LSTMs as the base model. Later, Transformers~\cite{vaswani2017attention} became the de facto architecture of pre-trained NLP models. BERT~\cite{devlin2018bert}, XLNet~\cite{yang2019xlnet} and RoBERTa~\cite{liu2019roberta} use the Transformer encoder, while GPT~\cite{radford2018improving}, GPT-2~\cite{radford2019language}, and GPT-3~\cite{brown2020language} use the Transformer decoder as the backbone. Some pre-trained models are also are based on the encoder-decoder transformer architecture, like T5~\cite{raffel2019exploring}, MASS~\cite{song2019mass}, and BART~\cite{lewis2019bart}. In this paper, we investigate another model architecture variation by studying the power of convolutional neural network as the backbone of pre-trained models for NLP.&#10;&#10;Convolutions have always been an interesting choice for sequence modeling and NLP applications \cite{kim-2014-convolutional,bai2018empirical, kalchbrenner2016neural}. Convolutions are lightweight and fast and have many interesting use-cases, notably for lightweight classification. In the era when LSTMs were the workhorses of NLP applications, convolutions were positioned nicely on the pareto frontier of the compute-performance curve. They are fast and lightweight, and unlike Transformers, they do not suffer from quadratic complexity. Our work is also well-aligned with the resurgence of interest in convolutions where \cite{wu2019pay} showed that convolutions can outperform self-attention on several sequence transduction tasks. Moreover, the necessity of the self-attention inductive bias in transformers have been also a subject of recent interest. Synthesizer models \cite{tay2020synthesizer} showed that transformers can still do pretty well without token-token dot product self-attention and a random attention matrix can perform competitively on certain tasks. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2004.14601" label="2004.14601">
        <attvalues>
          <attvalue for="0" value="Learning Music Helps You Read: Using Transfer to Study Linguistic&#10;  Structure in Language Models" />
          <attvalue for="1" value="  We propose transfer learning as a method for analyzing the encoding of&#10;grammatical structure in neural language models. We train LSTMs on&#10;non-linguistic data and evaluate their performance on natural language to&#10;assess which kinds of data induce generalizable structural features that LSTMs&#10;can use for natural language. We find that training on non-linguistic data with&#10;latent structure (MIDI music or Java code) improves test performance on natural&#10;language, despite no overlap in surface form or vocabulary. To pinpoint the&#10;kinds of abstract structure that models may be encoding to lead to this&#10;improvement, we run similar experiments with two artificial parentheses&#10;languages: one which has a hierarchical recursive structure, and a control&#10;which has paired tokens but no recursion. Surprisingly, training a model on&#10;either of these artificial languages leads to the same substantial gains when&#10;testing on natural language. Further experiments on transfer between natural&#10;languages controlling for vocabulary overlap show that zero-shot performance on&#10;a test language is highly correlated with typological syntactic similarity to&#10;the training language, suggesting that representations induced by pre-training&#10;correspond to the cross-linguistic syntactic properties. Our results provide&#10;insights into the ways that neural models represent abstract syntactic&#10;structure, and also about the kind of structural inductive biases which allow&#10;for natural language acquisition.&#10;" />
          <attvalue for="2" value="&#10;&#10;Understanding how neural language models learn and represent syntactic structure&#10;is an important analytic question for NLP. Recent work has directly probed the internal activations of models \cite{conneaufing, grain, johnprobe, urvashibert}, or fed them curated inputs that depend on complex syntax \cite{linzen16, gulordava, olmpics, mccoy2020}, in order to uncover latent syntactic awareness.&#10;&#10;We propose a different approach: we measure the structural awareness of a language model &#10;by studying how much this structure acts as an inductive bias to improve learning when we transfer from one language or symbolic system to another.&#10;&#10;We train LSTM models on data with varying degrees of language-like structure (music, Java code, nested symbols), and then evaluate their performance on natural language. Before evaluation, we freeze the LSTM parameters and fine-tune the word embeddings on the evaluation language. This lets us see if the training data induces language-like structure in the recurrent parameters of LSTMs--- despite removing vocabulary-level confounders. By assessing if representations are useful across languages, we examine the generalizable representations of grammar that LSTMs encode. We call this new method the Test for Inductive Bias via Language Model Transfer (TILT).&#10;&#10;Firstly, we examine the transfer of abstract structural features from languages that are very different on the surface from human language. We find that pretraining an LSTM on music data or Java code greatly improves transfer to human language over pretraining on structureless random baseline data. To test if the gain in performance is due to the LSTM utilizing the recursive nature of music and code, we train models on an artificial language with recursion (hierarchically nested symbols) and observe that they also perform well when evaluated on human language. However, we also surprisingly find that recursion is a sufficient, but not necessary condition for generalizable, language-like grammar induction. We observe similar gains when pretraining on a language of matching pairs that do not nest hierarchically, showcasing the importance of non-hierarchical head-dependent-type relations in LSTM language processing.&#10;&#10;Lastly, in transfer experiments between different human languages, we find that transfer is better between languages that are syntactically typologically similar, even with no vocabulary overlap. This suggests that models have the ability to form representations of typologically sensible properties rather than relying on ad-hoc or non-natural representations. For this result we draw on recent interlingual work such as \cite{artetxezero}, \cite{edo-zero}, and \cite{xnli}, extending it to use typological distance to turn these observations into quantitative probes.&#10;&#10;The TILT method allows us to ask a complementary set of questions to those answered by current analysis methods. TILTs demonstrate the abstract structural notions that LSTMs can learn, rather than probing for the manifestation of a particular known structure, as in most current methods. By examining the pretraining structures that give LSTMs a better ability to model language, we also contribute to the more general cognitive question of what structural inductive biases a learner needs to be able to easily acquire human language.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.11995" label="2012.11995">
        <attvalues>
          <attvalue for="0" value="Pre-Training a Language Model Without Human Language" />
          <attvalue for="1" value="  In this paper, we study how the intrinsic nature of pre-training data&#10;contributes to the fine-tuned downstream performance. To this end, we pre-train&#10;different transformer-based masked language models on several corpora with&#10;certain features, and we fine-tune those language models on GLUE benchmarks. We&#10;find that models pre-trained on unstructured data beat those trained directly&#10;from scratch on downstream tasks. Our results also show that pre-training on&#10;structured data does not always make the model acquire ability that can be&#10;transferred to natural language downstream tasks. To our great astonishment, we&#10;uncover that pre-training on certain non-human language data gives GLUE&#10;performance close to performance pre-trained on another non-English language.&#10;" />
          <attvalue for="2" value="&#10;Neural language models (LMs) are prevalent in nowadays natural language processing (NLP) community, and they are indispensable to a variety of NLP tasks.&#10;Researchers have devoted themselves to understanding what these models have learned and how they work. &#10;Probing a trained model is widely used to understand to what extent a model learns certain linguistic features~\cite{kovaleva2019revealing, hewitt2019structural, tenney2019bert, tenney2018you, lin2019open}.&#10;Another line of research focuses more on how training corpora affect the trained LMs ~\cite{micheli2020importance, gururangan-etal-2020-dont, zhang2020you}. &#10;&#10;In this work, we aim to understand how downstream performance varies across models pre-trained on data of particular traits.&#10;The core problem we determine to answer is: What factors in the pre-training data make a pre-trained transformer LM perform better on downstream tasks than their trained from scratch counterparts?&#10;To answer this question, we pre-train many different transformer LMs on dataset from miscellaneous disciplines, ranging from amino acid sequences in complex living organisms to artificial data generated by a simple python script.&#10;We then fine-tune them on English downstream tasks.&#10;The process is illustrated in Figure~\ref{fig:exp}. &#10;&#10;Recently,~\cite{papadimitriou2020learning} proposed to train an LSTM LM on a non-natural language dataset and test the LM's perplexity on natural language. &#10;They observed that LSTM LM trained on structured dataset gives perplexity far lower than those trained on unstructured data.&#10;While the observations are intriguing, this setting doesn't match the common setting widely applied nowadays, in which we fine-tune pre-trained LMs on downstream tasks.&#10;This is the first paper investigating whether masked language model (MLM) pre-training on non-natural language aids downstream natural language tasks' performance.&#10;&#10;Based on the experiments, we have the following observations: &#10;\begin{itemize}&#10;\item We reveal that fine-tuning models pre-trained on unstructured data outperforms model trained from scratch on downstream tasks. &#10;\item We find that structured pre-training data is not a sufficient condition to a pre-trained model that can perform well on NLP tasks.&#10;\item We discover that pre-training on a simple artificial dataset with hierarchical structure leads to downstream performance comparable to models pre-trained on human language.&#10;\item Our experiments show that token distribution is not the key factors to how well the model transferred to downstream tasks, while the number of token embeddings used during pre-training affects downstream performance.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04953" label="2109.04953">
        <attvalues>
          <attvalue for="0" value="Does Pretraining for Summarization Require Knowledge Transfer?" />
          <attvalue for="1" value="  Pretraining techniques leveraging enormous datasets have driven recent&#10;advances in text summarization. While folk explanations suggest that knowledge&#10;transfer accounts for pretraining's benefits, little is known about why it&#10;works or what makes a pretraining task or dataset suitable. In this paper, we&#10;challenge the knowledge transfer story, showing that pretraining on documents&#10;consisting of character n-grams selected at random, we can nearly match the&#10;performance of models pretrained on real corpora. This work holds the promise&#10;of eliminating upstream corpora, which may alleviate some concerns over&#10;offensive language, bias, and copyright issues. To see whether the small&#10;residual benefit of using real data could be accounted for by the structure of&#10;the pretraining task, we design several tasks motivated by a qualitative study&#10;of summarization corpora. However, these tasks confer no appreciable benefit,&#10;leaving open the possibility of a small role for knowledge transfer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.05963" label="2106.05963">
        <attvalues>
          <attvalue for="0" value="Learning to See by Looking at Noise" />
          <attvalue for="1" value="  Current vision systems are trained on huge datasets, and these datasets come&#10;with costs: curation is expensive, they inherit human biases, and there are&#10;concerns over privacy and usage rights. To counter these costs, interest has&#10;surged in learning from cheaper data sources, such as unlabeled images. In this&#10;paper we go a step further and ask if we can do away with real image datasets&#10;entirely, instead learning from noise processes. We investigate a suite of&#10;image generation models that produce images from simple random processes. These&#10;are then used as training data for a visual representation learner with a&#10;contrastive loss. We study two types of noise processes, statistical image&#10;models and deep generative models under different random initializations. Our&#10;findings show that it is important for the noise to capture certain structural&#10;properties of real data but that good performance can be achieved even with&#10;processes that are far from realistic. We also find that diversity is a key&#10;property to learn good representations. Datasets, models, and code are&#10;available at https://mbaradad.github.io/learning_with_noise.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.06225" label="2007.06225">
        <attvalues>
          <attvalue for="0" value="ProtTrans: Towards Cracking the Language of Life's Code Through&#10;  Self-Supervised Deep Learning and High Performance Computing" />
          <attvalue for="1" value="  Computational biology and bioinformatics provide vast data gold-mines from&#10;protein sequences, ideal for Language Models taken from NLP. These LMs reach&#10;for new prediction frontiers at low inference costs. Here, we trained two&#10;auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models&#10;(BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393&#10;billion amino acids. The LMs were trained on the Summit supercomputer using&#10;5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that&#10;the raw protein LM-embeddings from unlabeled data captured some biophysical&#10;features of protein sequences. We validated the advantage of using the&#10;embeddings as exclusive input for several subsequent tasks. The first was a&#10;per-residue prediction of protein secondary structure (3-state accuracy&#10;Q3=81%-87%); the second were per-protein predictions of protein sub-cellular&#10;localization (ten-state accuracy: Q10=81%) and membrane vs. water-soluble&#10;(2-state accuracy Q2=91%). For the per-residue predictions the transfer of the&#10;most informative embeddings (ProtT5) for the first time outperformed the&#10;state-of-the-art without using evolutionary information thereby bypassing&#10;expensive database searches. Taken together, the results implied that protein&#10;LMs learned some of the grammar of the language of life. To facilitate future&#10;work, we released our models at https://github.com/agemagician/ProtTrans.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.02730" label="2312.02730">
        <attvalues>
          <attvalue for="0" value="Towards Measuring Representational Similarity of Large Language Models" />
          <attvalue for="1" value="  Understanding the similarity of the numerous released large language models&#10;(LLMs) has many uses, e.g., simplifying model selection, detecting illegal&#10;model reuse, and advancing our understanding of what makes LLMs perform well.&#10;In this work, we measure the similarity of representations of a set of LLMs&#10;with 7B parameters. Our results suggest that some LLMs are substantially&#10;different from others. We identify challenges of using representational&#10;similarity measures that suggest the need of careful study of similarity scores&#10;to avoid false conclusions.&#10;" />
          <attvalue for="2" value="&#10;Numerous large language models (LLMs) with remarkable natural language understanding and reasoning capabilities have been released in recent months \cite{yang_harnessing_2023,zhao_survey_2023}.&#10;However, a comprehensive understanding of the differences between these models beyond architectures and benchmark performances is yet to be established. &#10;This is partly due to the inherent challenges in LLMs' explainability given their scale, their high demand for computational resources, and the rising trend of proprietary models.&#10;&#10;We argue that a thorough understanding of similarities and differences of LLMs is highly desirable:&#10;it may help identify factors that make models perform well, clear up the generalizability of studies of individual LLMs, simplify model selection, enhance our ability to ensure alignment of model behavior with human goals, improve ensembling, benchmark models without labeled data, identify (potentially illegal) model (re)use, and may aid certification of models, which could be required by future regulation of AI.&#10;&#10;LLM similarity can be studied from multiple perspectives, including functional similarity, i.e., whether they produce similar outputs, representational similarity, i.e., whether they have similar internal representations, whether they have similar reliance on specific training data, or whether they were trained in a similar manner.&#10;Methods for these perspectives have been proposed in prior work, but often focus on non-sequence models \cite{klabunde_similarity_2023} or do not scale to the size of LLMs \cite{shah_modeldiff_2022}.&#10;In this work, we focus on representational similarity in the last layer as it implies functional similarity, because the final layer has limited options to diverge functionally. &#10;Additionally, it allows studying similarity of how outputs are generated.&#10;&#10;Similarity of language models was studied to some extent \cite{wu_similarity_2020,ethayarajh_how_2019}, but these works do not explore similarity of decoder-only models on the scale of recent LLMs, and instead focus on smaller BERT-style models.&#10;As LLMs have developed at break-neck speed, analysis of the similarity of LLMs is generally limited.&#10;Moreover, many novel tools to study the similarity of representations have emerged relatively recently.&#10;&#10;In this paper, we aim to make the first steps towards understanding similarity of LLMs in more detail:&#10;\begin{enumerate}&#10; \item After discussing several options to compare LLMs, we outline how representational similarity measures can be applied to LLMs (Sec.~\ref{sec:comparing_llms}).&#10; \item We present first empirical results regarding the representational similarity of a set of 7B parameter models, offering a preliminary view into LLM similarity for commonsense reasoning (Winogrande) and code generation (HumanEval) (Sec.~\ref{sec:experiments}).&#10; \item We identify challenges of gaining a reliable picture of similarity of LLM representations (Sec.~\ref{sec:experiments}).&#10;\end{enumerate}&#10;Our code and data is publicly available (see \ref{apx:code}).&#10;&#10;Related Work.\ \ &#10;Several works study representations of language models and make implicit comparisons: how contextual they are \cite{ethayarajh_how_2019}, what interpretable concepts can be decoded from them \cite{liu_linguistic_2019}, or how models can communicate via representations \cite{moschella_relative_2023}.&#10;\cite{wu_similarity_2020,abnar_blackbox_2019} explicitly compare representations between models.&#10;However, these works have in common that they do not study the current generation of LLMs, and instead focus on smaller models with different architectures like BERT \cite{devlin_bert_2019} or ELMo \cite{peters_deep_2018}.&#10;Similarity of these models was also studied from a functional perspective \cite{mccoy_berts_2020}.&#10;As an exception, \cite{gurnee_finding_2023} probe the recent Pythia models \cite{biderman_pythia_2023}. &#10;Concurrent work \cite{yousefi_-context_2023,brown2023understanding} studies representations of modern LLMs.&#10;Performance of LLMs is compared in many benchmarks \cite[e.g.,][]{srivastava_capabilities_of_language_models_2022}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Representational Similarity, Model Comparison, Mathematics" />
        </attvalues>
      </node>
      <node id="2305.06329" label="2305.06329">
        <attvalues>
          <attvalue for="0" value="Similarity of Neural Network Models: A Survey of Functional and&#10;  Representational Measures" />
          <attvalue for="1" value="  Measuring similarity of neural networks to understand and improve their&#10;behavior has become an issue of great importance and research interest. In this&#10;survey, we provide a comprehensive overview of two complementary perspectives&#10;of measuring neural network similarity: (i) representational similarity, which&#10;considers how activations of intermediate layers differ, and (ii) functional&#10;similarity, which considers how models differ in their outputs. In addition to&#10;providing detailed descriptions of existing measures, we summarize and discuss&#10;results on the properties of and relationships between these measures, and&#10;point to open research problems. We hope our work lays a foundation for more&#10;systematic research on the properties and applicability of similarity measures&#10;for neural network models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.01172" label="2005.01172">
        <attvalues>
          <attvalue for="0" value="Similarity Analysis of Contextual Word Representation Models" />
          <attvalue for="1" value="  This paper investigates contextual word representation models from the lens&#10;of similarity analysis. Given a collection of trained models, we measure the&#10;similarity of their internal representations and attention. Critically, these&#10;models come from vastly different architectures. We use existing and novel&#10;similarity measures that aim to gauge the level of localization of information&#10;in the deep models, and facilitate the investigation of which design factors&#10;affect model similarity, without requiring any external linguistic annotation.&#10;The analysis reveals that models within the same family are more similar to one&#10;another, as may be expected. Surprisingly, different architectures have rather&#10;similar representations, but different individual neurons. We also observed&#10;differences in information localization in lower and higher layers and found&#10;that higher layers are more affected by fine-tuning on downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.15430" label="2209.15430">
        <attvalues>
          <attvalue for="0" value="Relative representations enable zero-shot latent space communication" />
          <attvalue for="1" value="  Neural networks embed the geometric structure of a data manifold lying in a&#10;high-dimensional space into latent representations. Ideally, the distribution&#10;of the data points in the latent space should depend only on the task, the&#10;data, the loss, and other architecture-specific constraints. However, factors&#10;such as the random weights initialization, training hyperparameters, or other&#10;sources of randomness in the training phase may induce incoherent latent spaces&#10;that hinder any form of reuse. Nevertheless, we empirically observe that, under&#10;the same data and modeling choices, the angles between the encodings within&#10;distinct latent spaces do not change. In this work, we propose the latent&#10;similarity between each sample and a fixed set of anchors as an alternative&#10;data representation, demonstrating that it can enforce the desired invariances&#10;without any additional training. We show how neural architectures can leverage&#10;these relative representations to guarantee, in practice, invariance to latent&#10;isometries and rescalings, effectively enabling latent space communication:&#10;from zero-shot model stitching to latent space comparison between diverse&#10;settings. We extensively validate the generalization capability of our approach&#10;on different datasets, spanning various modalities (images, text, graphs),&#10;tasks (e.g., classification, reconstruction) and architectures (e.g., CNNs,&#10;GCNs, transformers).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.02969" label="1911.02969">
        <attvalues>
          <attvalue for="0" value="BERTs of a feather do not generalize together: Large variability in&#10;  generalization across models with similar test set performance" />
          <attvalue for="1" value="  If the same neural network architecture is trained multiple times on the same&#10;dataset, will it make similar linguistic generalizations across runs? To study&#10;this question, we fine-tuned 100 instances of BERT on the Multi-genre Natural&#10;Language Inference (MNLI) dataset and evaluated them on the HANS dataset, which&#10;evaluates syntactic generalization in natural language inference. On the MNLI&#10;development set, the behavior of all instances was remarkably consistent, with&#10;accuracy ranging between 83.6% and 84.8%. In stark contrast, the same models&#10;varied widely in their generalization performance. For example, on the simple&#10;case of subject-object swap (e.g., determining that &quot;the doctor visited the&#10;lawyer&quot; does not entail &quot;the lawyer visited the doctor&quot;), accuracy ranged from&#10;0.00% to 66.2%. Such variation is likely due to the presence of many local&#10;minima that are equally attractive to a low-bias learner such as a neural&#10;network; decreasing the variability may therefore require models with stronger&#10;inductive biases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.01610" label="2305.01610">
        <attvalues>
          <attvalue for="0" value="Finding Neurons in a Haystack: Case Studies with Sparse Probing" />
          <attvalue for="1" value="  Despite rapid adoption and deployment of large language models (LLMs), the&#10;internal computations of these models remain opaque and poorly understood. In&#10;this work, we seek to understand how high-level human-interpretable features&#10;are represented within the internal neuron activations of LLMs. We train&#10;$k$-sparse linear classifiers (probes) on these internal activations to predict&#10;the presence of features in the input; by varying the value of $k$ we study the&#10;sparsity of learned representations and how this varies with model scale. With&#10;$k=1$, we localize individual neurons which are highly relevant for a&#10;particular feature, and perform a number of case studies to illustrate general&#10;properties of LLMs. In particular, we show that early layers make use of sparse&#10;combinations of neurons to represent many features in superposition, that&#10;middle layers have seemingly dedicated neurons to represent higher-level&#10;contextual features, and that increasing scale causes representational sparsity&#10;to increase on average, but there are multiple types of scaling dynamics. In&#10;all, we probe for over 100 unique features comprising 10 different categories&#10;in 7 different models spanning 70 million to 6.9 billion parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.04256" label="2304.04256">
        <attvalues>
          <attvalue for="0" value="A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding" />
          <attvalue for="1" value="  Zero-shot dialogue understanding aims to enable dialogue to track the user's&#10;needs without any training data, which has gained increasing attention. In this&#10;work, we investigate the understanding ability of ChatGPT for zero-shot&#10;dialogue understanding tasks including spoken language understanding (SLU) and&#10;dialogue state tracking (DST). Experimental results on four popular benchmarks&#10;reveal the great potential of ChatGPT for zero-shot dialogue understanding. In&#10;addition, extensive analysis shows that ChatGPT benefits from the multi-turn&#10;interactive prompt in the DST task but struggles to perform slot filling for&#10;SLU. Finally, we summarize several unexpected behaviors of ChatGPT in dialogue&#10;understanding tasks, hoping to provide some insights for future research on&#10;building zero-shot dialogue understanding systems with Large Language Models&#10;(LLMs).&#10;" />
          <attvalue for="2" value="&#10;Recent studies on Large Language Models (LLMs), such as GPT-3~\cite{brown2020language}, InstructGPT~\cite{ouyang2022training}, PaLM~\cite{chowdhery2022palm}, and OPT~\cite{zhang2022opt}, have exhibited impressive zero-shot performance.&#10;More recently, ChatGPT, a conversational large language model that has been trained by reinforcement learning with human feedback, which has brought&#10;remarkable success on various zero-shot natural language processing (NLP) tasks.&#10;&#10;Specifically, ChatGPT has shown competitive performance on&#10;zero-shot logical reasoning~\cite{qin2023chatgpt}, text summarization~\cite{yang2023exploring}, machine translation~\cite{jiao2023chatgpt}, information extraction~\cite{wei2023zero} and so on.&#10;However, it remains unclear how ChatGPT performs when it comes to dialogue-understanding tasks.&#10;&#10;To this end, we provide an empirical analysis on performing ChatGPT for zero-shot dialogue understanding tasks including spoken language understanding (SLU) and dialogue state tracking (DST).&#10;Furthermore, we introduce a multi-turn interactive prompt framework to improve the performance of ChatGPT in multi-turn DST.&#10;We conduct experiments on four widely used benchmarks including ATIS~\cite{hemphill-etal-1990-atis}, SNIPS~\cite{couke2018snips} in SLU and MultiWOZ2.1~\cite{eric2019multiwoz}, MultiWOZ2.4~\cite{Ye2021MultiWOZ2A} in DST.&#10;&#10;Through a preliminary exploration study, we provide the following observations:&#10;&#10;\begin{itemize}&#10; \item[\img{figures/heart-emoji.png}]ChatGPT demonstrates better capability in multi-turn dialogue understanding tasks (multi-turn DST) as compared to single-turn tasks (single-turn SLU).&#10; \item[\img{figures/heart-emoji.png}] Multi-turn interactive prompts can better leverage ChatGPT's multi-turn ability to enhance multi-turn tasks (i.e., DST).&#10; \item[\img{figures/broken-heart-emoji.png}] ChatGPT exhibits relatively inferior performance in slot-filling tasks, which can be compensated for by providing descriptions and examples of slot names.&#10; \item[\img{figures/broken-heart-emoji.png}] During multi-turn conversations, ChatGPT may occasionally exhibit unexpected behaviors, such as violating format requirements.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Models, Zero-Shot Learning, Artificial Intelligence, Dialogue Understanding, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2302.08081" label="2302.08081">
        <attvalues>
          <attvalue for="0" value="Exploring the Limits of ChatGPT for Query or Aspect-based Text&#10;  Summarization" />
          <attvalue for="1" value="  Text summarization has been a crucial problem in natural language processing&#10;(NLP) for several decades. It aims to condense lengthy documents into shorter&#10;versions while retaining the most critical information. Various methods have&#10;been proposed for text summarization, including extractive and abstractive&#10;summarization. The emergence of large language models (LLMs) like GPT3 and&#10;ChatGPT has recently created significant interest in using these models for&#10;text summarization tasks. Recent studies \cite{goyal2022news,&#10;zhang2023benchmarking} have shown that LLMs-generated news summaries are&#10;already on par with humans. However, the performance of LLMs for more practical&#10;applications like aspect or query-based summaries is underexplored. To fill&#10;this gap, we conducted an evaluation of ChatGPT's performance on four widely&#10;used benchmark datasets, encompassing diverse summaries from Reddit posts, news&#10;articles, dialogue meetings, and stories. Our experiments reveal that ChatGPT's&#10;performance is comparable to traditional fine-tuning methods in terms of Rouge&#10;scores. Moreover, we highlight some unique differences between&#10;ChatGPT-generated summaries and human references, providing valuable insights&#10;into the superpower of ChatGPT for diverse text summarization tasks. Our&#10;findings call for new directions in this area, and we plan to conduct further&#10;research to systematically examine the characteristics of ChatGPT-generated&#10;summaries through extensive human evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.08745" label="2301.08745">
        <attvalues>
          <attvalue for="0" value="Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine" />
          <attvalue for="1" value="  This report provides a preliminary evaluation of ChatGPT for machine&#10;translation, including translation prompt, multilingual translation, and&#10;translation robustness. We adopt the prompts advised by ChatGPT to trigger its&#10;translation ability and find that the candidate prompts generally work well&#10;with minor performance differences. By evaluating on a number of benchmark test&#10;sets, we find that ChatGPT performs competitively with commercial translation&#10;products (e.g., Google Translate) on high-resource European languages but lags&#10;behind significantly on low-resource or distant languages. As for the&#10;translation robustness, ChatGPT does not perform as well as the commercial&#10;systems on biomedical abstracts or Reddit comments but exhibits good results on&#10;spoken language. Further, we explore an interesting strategy named&#10;$\mathbf{pivot~prompting}$ for distant languages, which asks ChatGPT to&#10;translate the source sentence into a high-resource pivot language before into&#10;the target language, improving the translation performance noticeably. With the&#10;launch of the GPT-4 engine, the translation performance of ChatGPT is&#10;significantly boosted, becoming comparable to commercial translation products,&#10;even for distant languages. Human analysis on Google Translate and ChatGPT&#10;suggests that ChatGPT with GPT-3.5 tends to generate more hallucinations and&#10;mis-translation errors while that with GPT-4 makes the least errors. In other&#10;words, ChatGPT has already become a good translator. Please refer to our Github&#10;project for more details:&#10;https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator&#10;" />
          <attvalue for="2" value="&#10;&#10;ChatGPT is an intelligent chatting machine developed by OpenAI upon the InstructGPT~\cite{ouyang2022InstructGPT}, which is trained to follow an instruction in a prompt and provide a detailed response. According to the official statement, ChatGPT is able to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests due to the dialogue format. It integrates various abilities of natural language processing, including question answering, storytelling, logic reasoning, code debugging, machine translation, and so on.&#10;We are particularly interested in how ChatGPT performs for machine translation tasks, especially the gap between ChatGPT and commercial translation products (e.g., Google Translate, DeepL Translate).&#10;&#10;In this report, we provide a preliminary study of ChatGPT on machine translation, which to our best knowledge is also the first one since the release of ChatGPT. Specifically, we focus on three aspects:&#10;\begin{itemize}[leftmargin=10pt]&#10; \item Translation Prompt: ChatGPT is essentially a large language model, which needs prompts as guidance to trigger its translation ability. The style of prompts may affect the quality of translation outputs. For example, how to mention the source or target language information matters in multilingual machine translation models, which is usually solved by attaching language tokens~\cite{Johnson:2017:TACL,fan2021beyond}.&#10; \item Multilingual Translation: ChatGPT is a single model handling various NLP tasks and covering different languages, which can be considered a unified multilingual machine translation model. Thus, we are curious about how ChatGPT performs on different language pairs considering both the resource difference (e.g., high vs. low) and language family (e.g., European vs. Asian).&#10; \item Translation Robustness: ChatGPT is developed upon GPT3, which was trained on large-scale datasets that cover various domains. Therefore, we wonder if it can perform robustly well on domain-specific or even noisy sentences.&#10;\end{itemize}&#10;&#10;To trigger the translation ability of ChatGPT, we ask ChatGPT itself for advice and obtain three candidate translation prompts. By evaluating on the Chinese$\Rightarrow$English translation task, we find that the candidate prompts generally work well and show minor performance differences. Nevertheless, we adopt the best-performing prompt for the rest parts of the study. &#10;By evaluating the translation among four selected languages on the Flores-101 test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages.&#10;As for the translation robustness, results on three robustness sets suggest that ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language.&#10;&#10;Further, we have a discussion on how to improve ChatGPT for machine translation.&#10;On one hand, we explore an interesting strategy named pivot prompting for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. &#10;On the other hand, with an improved engine GPT-4~\cite{openai2023gpt4} launched on March 15, 2023, we re-evaluate the translation ability of ChatGPT and observe a significant boost of performance. The translation performance of ChatGPT becomes comparable to commercial translation products, even for distant languages.&#10;Extensive analysis on Google Translate and ChatGPT suggests that ChatGPT with GPT-3.5 tend to generate more hallucinations and more mis-translation errors while that with GPT-4 makes the least errors.&#10;In other words, ChatGPT has already become a good translator with GPT-4 as the engine!&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.00773" label="2104.00773">
        <attvalues>
          <attvalue for="0" value="MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with&#10;  Essential Annotation Corrections to Improve State Tracking Evaluation" />
          <attvalue for="1" value="  The MultiWOZ 2.0 dataset has greatly stimulated the research of task-oriented&#10;dialogue systems. However, its state annotations contain substantial noise,&#10;which hinders a proper evaluation of model performance. To address this issue,&#10;massive efforts were devoted to correcting the annotations. Three improved&#10;versions (i.e., MultiWOZ 2.1-2.3) have then been released. Nonetheless, there&#10;are still plenty of incorrect and inconsistent annotations. This work&#10;introduces MultiWOZ 2.4, which refines the annotations in the validation set&#10;and test set of MultiWOZ 2.1. The annotations in the training set remain&#10;unchanged (same as MultiWOZ 2.1) to elicit robust and noise-resilient model&#10;training. We benchmark eight state-of-the-art dialogue state tracking models on&#10;MultiWOZ 2.4. All of them demonstrate much higher performance than on MultiWOZ&#10;2.1.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00416" label="2011.00416">
        <attvalues>
          <attvalue for="0" value="Deep Learning for Text Style Transfer: A Survey" />
          <attvalue for="1" value="  Text style transfer is an important task in natural language generation,&#10;which aims to control certain attributes in the generated text, such as&#10;politeness, emotion, humor, and many others. It has a long history in the field&#10;of natural language processing, and recently has re-gained significant&#10;attention thanks to the promising performance brought by deep neural models. In&#10;this paper, we present a systematic survey of the research on neural text style&#10;transfer, spanning over 100 representative articles since the first neural text&#10;style transfer work in 2017. We discuss the task formulation, existing datasets&#10;and subtasks, evaluation, as well as the rich methodologies in the presence of&#10;parallel and non-parallel data. We also provide discussions on a variety of&#10;important topics regarding the future development of this task. Our curated&#10;paper list is at https://github.com/zhijing-jin/Text_Style_Transfer_Survey&#10;" />
          <attvalue for="2" value="&#10;&#10;Language is situational. Every utterance fits in a specific time, place, and scenario, conveys specific characteristics of the speaker, and typically has a well-defined intent. &#10;For example, someone who is uncertain is more likely to use tag questions (e.g., “This is true, isn’t it?”) than declarative sentences (e.g., “This is definitely true.”). Similarly, a professional setting is more likely to include formal statements (e.g., “Please consider taking a seat.”) as compared to an informal situation (e.g., “Come and sit!”). For artificial intelligence systems to accurately understand and generate language, it is necessary to model language with style/attribute, which goes beyond merely verbalizing the semantics in a non-stylized way.&#10;The values of the attributes can be drawn from a wide range of choices depending on pragmatics, such as the extent of formality, politeness, simplicity, personality, emotion,&#10;partner effect (e.g., reader awareness), genre of writing (e.g., fiction or non-fiction), and so on. &#10;&#10;The goal of text style transfer (TST) is to automatically control the style attributes of text while preserving the content. TST has a wide range of applications, as outlined by \cite{mcdonald1985computational} and \cite{hovy1987generating}. The style of language is crucial because it makes natural language processing more user-centered. TST has many immediate applications. For instance, one such application is intelligent bots for which users prefer distinct and consistent persona (e.g., empathetic) instead of emotionless or inconsistent persona.&#10;Another application is the development of intelligent writing assistants; for example, non-expert writers often need to polish their writings to better fit their purpose, e.g., more professional, polite, objective, humorous, or other advanced writing requirements, which may take years of experience to master. Other applications include automatic text simplification (where the target style is “simple”), debiasing online text (where the target style is “objective”), fighting against offensive language (where the target style is “non-offensive”), and so on.&#10;&#10;To formally define text style transfer, let us denote the target utterance as $\bm{x}'$ and the target discourse style attribute as $a'$. TST aims to model $p(\bm{x}'|a,\bm{x})$, where $\bm{x}$ is a given text carrying a source attribute value $a$. Consider the previous example of text expressed by two different extents of formality:&#10;&#10;In this case, a TST model should be able to modify the formality and generate the formal sentence $\bm{x}'=$``Please consider taking a seat.'' given the informal input $\bm{x}=$``Come and sit!''. Note that the key difference of TST from another NLP task, style-conditioned language modeling, is that the latter is conditioned on only a style token, whereas TST takes as input both the target style attribute $a'$ and a source sentence $\bm{x}$ that constrains the content.&#10;&#10;Crucial to the definition of style transfer is the distinction of ``style'' and ``content,'' for which there are two common practices. The first one is by linguistic definition, where non-functional linguistic features are classified into the style (e.g., formality), and the semantics are classified into the content. In contrast, the second practice is data-driven, -- given two corpora (e.g., a positive review set and a negative review set), the invariance between the two corpora is the content, whereas the variance is the style (e.g., sentiment, topic) \cite{mou-vechtomova-2020-stylized}. &#10;&#10;Driven by the growing needs for TST, active research in this field has emerged, from the traditional linguistic approaches, to the more recent neural network-based approaches. Traditional approaches rely on term replacement and templates. For example, early work in NLG for weather forecasts builds domain-specific templates to {express different types of weather with different levels of uncertainty for different users \cite{sripada2004lessons,reiter2005choosing,belz2008automatic,gkatzia2017data}}. Research that more explicitly focuses on TST starts from the frame language-based systems \cite{mcdonald1985computational}, and schema-based NLG systems \cite{hovy1987generating,hovy1990pragmatics} which generate text with pragmatic constraints such as formality under small-scale well-defined schema. Most of this earlier work required domain-specific templates, hand-featured phrase sets that express a certain attribute (e.g., friendly), and sometimes a look-up table of expressions with the same meaning but multiple different attributes \cite{bateman1989phrasing,stamatatos1997user,power2003generating,reiter2003lessons,sheikha2011formal,mairesse2011controling}.&#10;&#10;With the success of deep learning in the last decade, a variety of neural methods have been recently proposed for TST. If parallel data are provided, standard sequence-to-sequence models are often directly applied~\cite{rao-tetreault-2018-dear} (see Section~\ref{sec:sup}). However, most use cases do not have parallel data, so TST on non-parallel corpora has become a prolific research area (see Section~\ref{sec:unsup}).&#10;The first line of approaches disentangle text into its content and attribute in the latent space, and apply generative modeling \cite{Hu2017TowardCG,shen2017style}.&#10;This trend was then joined by another distinctive line of approach, prototype editing \cite{li-etal-2018-delete} which extracts a sentence template and its attribute markers to generate the text. Another paradigm soon followed, i.e., pseudo-parallel corpus construction to train the model as if in a supervised way with the pseudo-parallel data \cite{zhang2018style,jin2019imat}. These three directions, (1) disentanglement, (2) prototype editing, and (3) pseudo-parallel corpus construction, are further advanced with the emergence of Transformer-based models \cite{Sudhakar2019TransformingDR,malmi2020unsupervised}.&#10;&#10;Given the advances in TST methodologies, it now starts to expand its impact to downstream applications, such as persona-based dialog generation \cite{niu-bansal-2018-polite,huang-etal-2018-automatic}, stylistic summarization \cite{jin-etal-2020-hooks}, stylized language modeling to imitate specific authors \cite{Syed2020AdaptingLM}, online text debiasing \cite{pryzant2020automatically,ma2020powertransformer}, simile generation \cite{chakrabarty2020generating}, and many others.&#10;&#10;Motivation of a Survey on TST.&#10;The increasing interest in modeling the style of text can be regarded as a trend reflecting the fact that NLP researchers start to focus more on user-centeredness and personalization. However, despite the growing interest in TST, the existing literature shows a large diversity in the selection of benchmark datasets, methodological frameworks, and evaluation metrics. Thus, the aim of this survey is to provide summaries and potential standardizations on some important aspects of TST, such as the terminology, problem definition, benchmark datasets, and evaluation metrics.&#10;We also aim to provide different perspectives on the methodology of TST, and suggest some potential cross-cutting research questions for our proposed research agenda of the field.&#10;As shown in Table~\ref{tab:overview}, the key contributions targeted by this survey are as follows:&#10;&#10;\begin{enumerate}&#10; \item We conduct the first comprehensive review that covers most existing works (more than 100 papers) on deep learning-based TST.&#10; \item We provide an overview of the task setting, terminology definition, benchmark datasets (Section~\ref{sec:task-overview}), and evaluation metrics for which we proposed standard practices that can be helpful for future works (Section~\ref{sec:eval}).&#10; \item We categorize the existing approaches on parallel data (Section~\ref{sec:sup}) and non-parallel data (Section~\ref{sec:unsup}) for which we distill some unified methodological frameworks.&#10; \item We discuss a potential research agenda for TST (Section~\ref{sec:discussion}), including expanding the scope of styles, improving the methodology, loosening dataset assumptions, and improving evaluation metrics.&#10; \item We provide a vision for how to broaden the impact of TST (Section~\ref{sec:impact}), including connecting to more NLP tasks, and more specialized downstream applications, as well as considering some important ethical impacts.&#10;\end{enumerate}&#10;&#10;Paper Selection. The neural TST papers reviewed in this survey are mainly from top conferences in NLP and artificial intelligence (AI), including ACL, EMNLP, NAACL, COLING, CoNLL, NeurIPS, ICML, ICLR, AAAI, and IJCAI. Other than conference papers, we also include some non-peer-reviewed preprint papers that can offer some insightful information about the field. The major factors for selecting non-peer-reviewed preprint papers include novelty and completeness, among others. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Natural Language Generation, Deep Neural Models, Text Style Transfer" />
        </attvalues>
      </node>
      <node id="1803.06535" label="1803.06535">
        <attvalues>
          <attvalue for="0" value="Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks&#10;  and Metrics for Formality Style Transfer" />
          <attvalue for="1" value="  Style transfer is the task of automatically transforming a piece of text in&#10;one particular style into another. A major barrier to progress in this field&#10;has been a lack of training and evaluation datasets, as well as benchmarks and&#10;automatic metrics. In this work, we create the largest corpus for a particular&#10;stylistic transfer (formality) and show that techniques from the machine&#10;translation community can serve as strong baselines for future work. We also&#10;discuss challenges of using automatic metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.07894" label="1808.07894">
        <attvalues>
          <attvalue for="0" value="Style Transfer as Unsupervised Machine Translation" />
          <attvalue for="1" value="  Language style transferring rephrases text with specific stylistic attributes&#10;while preserving the original attribute-independent content. One main challenge&#10;in learning a style transfer system is a lack of parallel data where the source&#10;sentence is in one style and the target sentence in another style. With this&#10;constraint, in this paper, we adapt unsupervised machine translation methods&#10;for the task of automatic style transfer. We first take advantage of&#10;style-preference information and word embedding similarity to produce&#10;pseudo-parallel data with a statistical machine translation (SMT) framework.&#10;Then the iterative back-translation approach is employed to jointly train two&#10;neural machine translation (NMT) based transfer systems. To control the noise&#10;generated during joint training, a style classifier is introduced to guarantee&#10;the accuracy of style transfer and penalize bad candidates in the generated&#10;pseudo data. Experiments on benchmark datasets show that our proposed method&#10;outperforms previous state-of-the-art models in terms of both accuracy of style&#10;transfer and quality of input-output correspondence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.11333" label="1901.11333">
        <attvalues>
          <attvalue for="0" value="IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and&#10;  Translation" />
          <attvalue for="1" value="  Text attribute transfer aims to automatically rewrite sentences such that&#10;they possess certain linguistic attributes, while simultaneously preserving&#10;their semantic content. This task remains challenging due to a lack of&#10;supervised parallel data. Existing approaches try to explicitly disentangle&#10;content and attribute information, but this is difficult and often results in&#10;poor content-preservation and ungrammaticality. In contrast, we propose a&#10;simpler approach, Iterative Matching and Translation (IMaT), which: (1)&#10;constructs a pseudo-parallel corpus by aligning a subset of semantically&#10;similar sentences from the source and the target corpora; (2) applies a&#10;standard sequence-to-sequence model to learn the attribute transfer; (3)&#10;iteratively improves the learned transfer function by refining imperfections in&#10;the alignment. In sentiment modification and formality transfer tasks, our&#10;method outperforms complex state-of-the-art systems by a large margin. As an&#10;auxiliary contribution, we produce a publicly-available test set with&#10;human-generated transfer references.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.03162" label="1805.03162">
        <attvalues>
          <attvalue for="0" value="Polite Dialogue Generation Without Parallel Data" />
          <attvalue for="1" value="  Stylistic dialogue response generation, with valuable applications in&#10;personality-based conversational agents, is a challenging task because the&#10;response needs to be fluent, contextually-relevant, as well as&#10;paralinguistically accurate. Moreover, parallel datasets for&#10;regular-to-stylistic pairs are usually unavailable. We present three&#10;weakly-supervised models that can generate diverse polite (or rude) dialogue&#10;responses without parallel data. Our late fusion model (Fusion) merges the&#10;decoder of an encoder-attention-decoder dialogue model with a language model&#10;trained on stand-alone polite utterances. Our label-fine-tuning (LFT) model&#10;prepends to each source sequence a politeness-score scaled label (predicted by&#10;our state-of-the-art politeness classifier) during training, and at test time&#10;is able to generate polite, neutral, and rude responses by simply scaling the&#10;label embedding by the corresponding score. Our reinforcement learning model&#10;(Polite-RL) encourages politeness generation by assigning rewards proportional&#10;to the politeness classifier score of the sampled response. We also present two&#10;retrieval-based polite dialogue model baselines. Human evaluation validates&#10;that while the Fusion and the retrieval-based models achieve politeness with&#10;poorer context-relevance, the LFT and Polite-RL models can produce&#10;significantly more polite responses without sacrificing dialogue quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.01980" label="2004.01980">
        <attvalues>
          <attvalue for="0" value="Hooks in the Headline: Learning to Generate Headlines with Controlled&#10;  Styles" />
          <attvalue for="1" value="  Current summarization systems only produce plain, factual headlines, but do&#10;not meet the practical needs of creating memorable titles to increase exposure.&#10;We propose a new task, Stylistic Headline Generation (SHG), to enrich the&#10;headlines with three style options (humor, romance and clickbait), in order to&#10;attract more readers. With no style-specific article-headline pair (only a&#10;standard headline summarization dataset and mono-style corpora), our method&#10;TitleStylist generates style-specific headlines by combining the summarization&#10;and reconstruction tasks into a multitasking framework. We also introduced a&#10;novel parameter sharing scheme to further disentangle the style from the text.&#10;Through both automatic and human evaluation, we demonstrate that TitleStylist&#10;can generate relevant, fluent headlines with three target styles: humor,&#10;romance, and clickbait. The attraction score of our model generated headlines&#10;surpasses that of the state-of-the-art summarization model by 9.68%, and even&#10;outperforms human-written references.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.08942" label="2009.08942">
        <attvalues>
          <attvalue for="0" value="Generating similes effortlessly like a Pro: A Style Transfer Approach&#10;  for Simile Generation" />
          <attvalue for="1" value="  Literary tropes, from poetry to stories, are at the crux of human imagination&#10;and communication. Figurative language such as a simile go beyond plain&#10;expressions to give readers new insights and inspirations. In this paper, we&#10;tackle the problem of simile generation. Generating a simile requires proper&#10;understanding for effective mapping of properties between two concepts. To this&#10;end, we first propose a method to automatically construct a parallel corpus by&#10;transforming a large number of similes collected from Reddit to their literal&#10;counterpart using structured common sense knowledge. We then propose to&#10;fine-tune a pretrained sequence to sequence model, BART~\cite{lewis2019bart},&#10;on the literal-simile pairs to gain generalizability, so that we can generate&#10;novel similes given a literal sentence. Experiments show that our approach&#10;generates $88\%$ novel similes that do not share properties with the training&#10;data. Human evaluation on an independent set of literal statements shows that&#10;our model generates similes better than two literary experts&#10;\textit{37\%}\footnote{We average 32.6\% and 41.3\% for 2 humans.} of the&#10;times, and three baseline systems including a recent metaphor generation model&#10;\textit{71\%}\footnote{We average 82\% ,63\% and 68\% for three baselines.} of&#10;the times when compared pairwise.\footnote{The simile in the title is generated&#10;by our best model. Input: Generating similes effortlessly, output: Generating&#10;similes \textit{like a Pro}.} We also show how replacing literal sentences with&#10;similes from our best model in machine generated stories improves evocativeness&#10;and leads to better acceptance by human judges.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.03393" label="1902.03393">
        <attvalues>
          <attvalue for="0" value="Improved Knowledge Distillation via Teacher Assistant" />
          <attvalue for="1" value="  Despite the fact that deep neural networks are powerful models and achieve&#10;appealing results on many tasks, they are too large to be deployed on edge&#10;devices like smartphones or embedded sensor nodes. There have been efforts to&#10;compress these networks, and a popular method is knowledge distillation, where&#10;a large (teacher) pre-trained network is used to train a smaller (student)&#10;network. However, in this paper, we show that the student network performance&#10;degrades when the gap between student and teacher is large. Given a fixed&#10;student network, one cannot employ an arbitrarily large teacher, or in other&#10;words, a teacher can effectively transfer its knowledge to students up to a&#10;certain size, not smaller. To alleviate this shortcoming, we introduce&#10;multi-step knowledge distillation, which employs an intermediate-sized network&#10;(teacher assistant) to bridge the gap between the student and the teacher.&#10;Moreover, we study the effect of teacher assistant size and extend the&#10;framework to multi-step distillation. Theoretical analysis and extensive&#10;experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet&#10;architectures substantiate the effectiveness of our proposed approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.06189" label="2010.06189">
        <attvalues>
          <attvalue for="0" value="X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained&#10;  Language Models" />
          <attvalue for="1" value="  Language models (LMs) have proven surprisingly successful at capturing&#10;factual knowledge by completing cloze-style fill-in-the-blank questions such as&#10;&quot;Punta Cana is located in _.&quot; However, while knowledge is both written and&#10;queried in many languages, studies on LMs' factual representation ability have&#10;almost invariably been performed on English. To assess factual knowledge&#10;retrieval in LMs in different languages, we create a multilingual benchmark of&#10;cloze-style probes for 23 typologically diverse languages. To properly handle&#10;language variations, we expand probing methods from single- to multi-word&#10;entities, and develop several decoding algorithms to generate multi-token&#10;predictions. Extensive experimental results provide insights about how well (or&#10;poorly) current state-of-the-art LMs perform at this task in languages with&#10;more or fewer available resources. We further propose a code-switching-based&#10;method to improve the ability of multilingual LMs to access knowledge, and&#10;verify its effectiveness on several benchmark languages. Benchmark data and&#10;code have been released at https://x-factr.github.io.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Factual Knowledge Retrieval from LMs&#10;Several works have focused on probing factual knowledge solely from pre-trained LMs without access to external knowledge.&#10;They do so by either using prompts and letting the LM fill in the blanks, which assumes that the LM is a static knowledge source \cite{petroni-etal-2019-language,jiang-2019-lpaqa,poerner-2019-ebert,bouraoui-2020-relbert}, or fine-tuning the LM on a set of question-answer pairs to directly generate answers, which dynamically adapts the LM to this particular task \cite{roberts-2020-t5pack}.&#10;Impressive results demonstrated by these works indicate that large-scale LMs contain a significant amount of knowledge, in some cases even outperforming competitive question answering systems relying on external resources \cite{roberts-2020-t5pack}.&#10;\cite{petroni-2020-contextlm} further shows that LMs can generate even more factual knowledge when augmented with retrieved sentences.&#10;Our work builds on these works by expanding to multilingual and multi-token evaluation, and also demonstrates the significant challenges posed by this setting.&#10;&#10;Multilingual Benchmarks&#10;Many multilingual benchmarks have been created to evaluate the performance of multilingual systems on different natural language processing tasks, including question answering \cite{artetxe-2019-cross,lewis-2019-mlqa,clark-2020-tydiqa}, natural language understanding \cite{conneau-etal-2018-xnli,yang-etal-2019-paws,zweigenbaum-2018-overview,artetxe-2019-massively}, syntactic prediction \cite{nivre-2018-universal,pan-etal-2017-cross}, and comprehensive benchmarks covering multiple tasks \cite{hu-2020-xtreme,liang-2020-xglue}.&#10;We focus on multilingual factual knowledge retrieval from LMs, which to our knowledge has not been covered by any previous work.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Factual Representation Analysis, Artificial Intelligence, Multilingual Knowledge Retrieval, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2405.01299" label="2405.01299">
        <attvalues>
          <attvalue for="0" value="The Effectiveness of LLMs as Annotators: A Comparative Overview and&#10;  Empirical Analysis of Direct Representation" />
          <attvalue for="1" value="  Large Language Models (LLMs) have emerged as powerful support tools across&#10;various natural language tasks and a range of application domains. Recent&#10;studies focus on exploring their capabilities for data annotation. This paper&#10;provides a comparative overview of twelve studies investigating the potential&#10;of LLMs in labelling data. While the models demonstrate promising cost and&#10;time-saving benefits, there exist considerable limitations, such as&#10;representativeness, bias, sensitivity to prompt variations and English language&#10;preference. Leveraging insights from these studies, our empirical analysis&#10;further examines the alignment between human and GPT-generated opinion&#10;distributions across four subjective datasets. In contrast to the studies&#10;examining representation, our methodology directly obtains the opinion&#10;distribution from GPT. Our analysis thereby supports the minority of studies&#10;that are considering diverse perspectives when evaluating data annotation tasks&#10;and highlights the need for further research in this direction.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs) have shown impressive abilities in a variety of natural language related tasks \cite{brown2020language, touvron2023llama}.&#10;\cite{brown2020language} demonstrate their ability as few-shot learners and \cite{flan_2021,kojima2022large} evidence their zero-shot capabilities. &#10;Recognising the significance and costliness of annotated data across various research domains, recent work explores the potential of LLMs as data annotators, encompassing both zero- and few-shot learning approaches \cite{lee2023can, Ziems_Held_Shaikh_Chen_Zhang_Yang_2023, tornbergchatgpt, zhu2023can,gilardi2023chatgpt, mohta2023large, ding2022gpt, he2023annollm}. Considering that LLMs are trained to adhere to instructions guided by human preference \cite{ouyang2022training, rafailov2023direct}, studies examine the extent to which human disagreement is captured \cite{lee2023can} and whether or not such disagreement aligns with that of humans \cite{santurkar2023whose}.&#10;&#10;Our work, firstly, offers a comparative overview of twelve previous studies that investigate the capabilities of LLMs as annotators, &#10;concentrating on classification tasks and considering whether disagreement is captured by the studies. Secondly, we present an empirical analysis concentrating more specifically on the perspectivist question. We compare the top-performing LLM from the first section(GPT) against human annotators, by examining the degree of alignment &#10;between their opinion distributions, &#10;for the case of the four subjective datasets recently used for the 2023 SEMEVAL Task on Learning With Disagreement \cite{leonardelli_2023_semeval}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Human-AI Alignment, Linguistics, Cognitive Science, Language Model Capabilities, Data Annotation Limitations, Statistics" />
        </attvalues>
      </node>
      <node id="2305.13788" label="2305.13788">
        <attvalues>
          <attvalue for="0" value="Can Large Language Models Capture Dissenting Human Voices?" />
          <attvalue for="1" value="  Large language models (LLMs) have shown impressive achievements in solving a&#10;broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been&#10;shown to generalize in zero-shot settings as well. However, whether LLMs&#10;closely align with the human disagreement distribution has not been&#10;well-studied, especially within the scope of natural language inference (NLI).&#10;In this paper, we evaluate the performance and alignment of LLM distribution&#10;with humans using two different techniques to estimate the multinomial&#10;distribution: Monte Carlo Estimation (MCE) and Log Probability Estimation&#10;(LPE). As a result, we show LLMs exhibit limited ability in solving NLI tasks&#10;and simultaneously fail to capture human disagreement distribution. The&#10;inference and human alignment performances plunge even further on data samples&#10;with high human disagreement levels, raising concerns about their natural&#10;language understanding (NLU) ability and their representativeness to a larger&#10;human population. The source code for the experiments is available at&#10;https://github.com/xfactlab/emnlp2023-LLM-Disagreement&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2303.17548" label="2303.17548">
        <attvalues>
          <attvalue for="0" value="Whose Opinions Do Language Models Reflect?" />
          <attvalue for="1" value="  Language models (LMs) are increasingly being used in open-ended contexts,&#10;where the opinions reflected by LMs in response to subjective queries can have&#10;a profound impact, both on user satisfaction, as well as shaping the views of&#10;society at large. In this work, we put forth a quantitative framework to&#10;investigate the opinions reflected by LMs -- by leveraging high-quality public&#10;opinion polls and their associated human responses. Using this framework, we&#10;create OpinionsQA, a new dataset for evaluating the alignment of LM opinions&#10;with those of 60 US demographic groups over topics ranging from abortion to&#10;automation. Across topics, we find substantial misalignment between the views&#10;reflected by current LMs and those of US demographic groups: on par with the&#10;Democrat-Republican divide on climate change. Notably, this misalignment&#10;persists even after explicitly steering the LMs towards particular demographic&#10;groups. Our analysis not only confirms prior observations about the&#10;left-leaning tendencies of some human feedback-tuned LMs, but also surfaces&#10;groups whose opinions are poorly reflected by current LMs (e.g., 65+ and&#10;widowed individuals). Our code and data are available at&#10;https://github.com/tatsu-lab/opinions_qa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.14803" label="2304.14803">
        <attvalues>
          <attvalue for="0" value="SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)" />
          <attvalue for="1" value="  NLP datasets annotated with human judgments are rife with disagreements&#10;between the judges. This is especially true for tasks depending on subjective&#10;judgments such as sentiment analysis or offensive language detection.&#10;Particularly in these latter cases, the NLP community has come to realize that&#10;the approach of 'reconciling' these different subjective interpretations is&#10;inappropriate. Many NLP researchers have therefore concluded that rather than&#10;eliminating disagreements from annotated corpora, we should preserve&#10;them-indeed, some argue that corpora should aim to preserve all annotator&#10;judgments. But this approach to corpus creation for NLP has not yet been widely&#10;accepted. The objective of the LeWiDi series of shared tasks is to promote this&#10;approach to developing NLP models by providing a unified framework for training&#10;and evaluating with such datasets. We report on the second LeWiDi shared task,&#10;which differs from the first edition in three crucial respects: (i) it focuses&#10;entirely on NLP, instead of both NLP and computer vision tasks in its first&#10;edition; (ii) it focuses on subjective tasks, instead of covering different&#10;types of disagreements-as training with aggregated labels for subjective NLP&#10;tasks is a particularly obvious misrepresentation of the data; and (iii) for&#10;the evaluation, we concentrate on soft approaches to evaluation. This second&#10;edition of LeWiDi attracted a wide array of participants resulting in 13 shared&#10;task submission papers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.06939" label="2304.06939">
        <attvalues>
          <attvalue for="0" value="Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with&#10;  Text" />
          <attvalue for="1" value="  In-context vision and language models like Flamingo support arbitrarily&#10;interleaved sequences of images and text as input. This format not only enables&#10;few-shot learning via interleaving independent supervised (image, text)&#10;examples, but also, more complex prompts involving interaction between images,&#10;e.g., &quot;What do image A and image B have in common?&quot; To support this interface,&#10;pretraining occurs over web corpora that similarly contain interleaved&#10;images+text. To date, however, large-scale data of this form have not been&#10;publicly available.&#10;  We release Multimodal C4, an augmentation of the popular text-only C4 corpus&#10;with images interleaved. We use a linear assignment algorithm to place images&#10;into longer bodies of text using CLIP features, a process that we show&#10;outperforms alternatives. Multimodal C4 spans everyday topics like cooking,&#10;travel, technology, etc. A manual inspection of a random sample of documents&#10;shows that a vast majority (88%) of images are topically relevant, and that&#10;linear assignment frequently selects individual sentences specifically&#10;well-aligned with each image (80%). After filtering NSFW images, ads, etc., the&#10;resulting corpus consists of 101.2M documents with 571M images interleaved in&#10;43B English tokens.&#10;" />
          <attvalue for="2" value="&#10;&#10;In-context learning \cite{brown2020language} enables sequence models to adapt to new tasks without any parameter updates.&#10;By interleaving a few supervised examples in a prompt, few-shot learning can be formatted as a next-token prediction task, i.e., $x_1, y_1, x_2, y_2, \ldots, x_n$ is input to predict $\hat y_n$.&#10;Some image+text models also support in-context learning via interleaving of images/text jointly.&#10;Prior experiments \cite{alayrac2022flamingo} suggest that performant multimodal in-context learning is dependent upon pretraining on similarly interleaved sequences of images and text (rather than single image/caption pairs). However, such a large-scale corpus has not been made publicly available.&#10;&#10;To address this, we introduce Multimodal C4 (mmc4), a public, billion-scale image-text dataset consisting of interleaved image/text sequences. mmc4 is constructed from public webpages contained in the cleaned English c4 corpus. In addition to standard preprocessing steps like deduplication, NSFW removal, etc., we place images into sequences of sentences by treating each document as an instance of a bipartite linear assignment problem, with images being assigned to sentences (under the constraint that each sentence is assigned at most one image). We show that applying CLIP ViT-L/14 \cite{radford2021learning} to estimate bipartite weights in a zero-shot fashion results in state-of-the-art performance on intra-document alignment benchmarks, and then apply this process to 100M+ documents to construct mmc4.&#10;Apart from the full corpus, we have created two additional subsets: mmc4-ff, which removes images with detected faces, and mmc4-core, a more strictly filtered and downsized version of the corpus, serving as an initial corpus for developers.&#10;&#10;We explore mmc4, showing that: 1) the text and images in the corpus span expected everyday topics like cooking and travel; 2) filters like NSFW/ad removal work with high accuracy; and 3) the resulting images are relevant to the associated documents, and often, appropriately aligned to the most-relevant individual sentence. We conclude by discussing initial use-cases of mmc4, including OpenFlamingo \cite{awadalla2023openflamingo}, an open source version of Flamingo \cite{alayrac2022flamingo}. Initial ablations show that training on the sequences of mmc4 enables few-shot, in-context adaptation to image captioning datasets.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multimodal Learning, Computer Vision, Large-Scale Data, Artificial Intelligence, Natural Language Processing, Vision Language Models" />
        </attvalues>
      </node>
      <node id="2204.14198" label="2204.14198">
        <attvalues>
          <attvalue for="0" value="Flamingo: a Visual Language Model for Few-Shot Learning" />
          <attvalue for="1" value="  Building models that can be rapidly adapted to novel tasks using only a&#10;handful of annotated examples is an open challenge for multimodal machine&#10;learning research. We introduce Flamingo, a family of Visual Language Models&#10;(VLM) with this ability. We propose key architectural innovations to: (i)&#10;bridge powerful pretrained vision-only and language-only models, (ii) handle&#10;sequences of arbitrarily interleaved visual and textual data, and (iii)&#10;seamlessly ingest images or videos as inputs. Thanks to their flexibility,&#10;Flamingo models can be trained on large-scale multimodal web corpora containing&#10;arbitrarily interleaved text and images, which is key to endow them with&#10;in-context few-shot learning capabilities. We perform a thorough evaluation of&#10;our models, exploring and measuring their ability to rapidly adapt to a variety&#10;of image and video tasks. These include open-ended tasks such as visual&#10;question-answering, where the model is prompted with a question which it has to&#10;answer; captioning tasks, which evaluate the ability to describe a scene or an&#10;event; and close-ended tasks such as multiple-choice visual question-answering.&#10;For tasks lying anywhere on this spectrum, a single Flamingo model can achieve&#10;a new state of the art with few-shot learning, simply by prompting the model&#10;with task-specific examples. On numerous benchmarks, Flamingo outperforms&#10;models fine-tuned on thousands of times more task-specific data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.02000" label="2002.02000">
        <attvalues>
          <attvalue for="0" value="Aligning the Pretraining and Finetuning Objectives of Language Models" />
          <attvalue for="1" value="  We demonstrate that explicitly aligning the pretraining objectives to the&#10;finetuning objectives in language model training significantly improves the&#10;finetuning task performance and reduces the minimum amount of finetuning&#10;examples required. The performance margin gained from objective alignment&#10;allows us to build language models with smaller sizes for tasks with less&#10;available training data. We provide empirical evidence of these claims by&#10;applying objective alignment to concept-of-interest tagging and acronym&#10;detection tasks. We found that, with objective alignment, our 768 by 3 and 512&#10;by 3 transformer language models can reach accuracy of 83.9%/82.5% for&#10;concept-of-interest tagging and 73.8%/70.2% for acronym detection using only&#10;200 finetuning examples per task, outperforming the 768 by 3 model pretrained&#10;without objective alignment by +4.8%/+3.4% and +9.9%/+6.3%. We name finetuning&#10;small language models in the presence of hundreds of training examples or less&#10;&quot;Few Example learning&quot;. In practice, Few Example Learning enabled by objective&#10;alignment not only saves human labeling costs, but also makes it possible to&#10;leverage language models in more real-time applications.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the past three years, new deep learning language model architectures and techniques have significantly advanced the field of computational linguistics. The current state-of-the-art language models use large Transformer architectures \cite{Vaswani2017} containing hundreds of millions of parameters and are pretrained using multi-billion word corpuses. GPT \cite{Radford2018} and BERT \cite{Devlin2019} are two of the first examples of these Transformer-base language models. They outperformed the RNN-based models such as ELMo \cite{Peters2018} by a significant margin in benchmarks like GLUE \cite{Wang2019} and SQuAD \cite{Rajpurkar2016}. Since the release of GPT and BERT, many researchers have further improved the Transformer-based language models demonstrated by surpassing their predecessors in the evaluations of the common benchmarks \cite{Liu2019, Dong2019, Conneau2019, Conneau20192, Raffel2019, Yang2019, Sun2020}.&#10;&#10;These new language model releases typically use more model parameters than their predecessors and are evaluated only on the academic datasets. This contrasts the two major challenges that we face in building natural language understanding (NLU) applications: (1) Speeding up language model inference; (2) Shortage of finetuning data for application-specific tasks. In our experience, the inference speed of the 768 by 12 BERT-base model, the smallest BERT model, is far from meeting the requirements for most real-time applications. To speed up language models, people have developed libraries for fast neural network computation \cite{Zhang2018, Junczysdowmunt2018} and built smaller models with sufficient prediction power using knowledge distillation \cite{Jiao2019} or improved Transformer architectures \cite{Lan2020}. In terms of the learning techniques for low-resource tasks, knowledge transfer through pretraining itself is a partial solution. GPT-2 has demonstrated its impressive zero-shot learning capability in text generation and question \&amp; answering \cite{Radford2019}. Besides transfer learning, multitasking is another common technique used to boost low-resource task performance \cite{Lin2018, Liu2018, Conneau2019}.&#10;&#10;In this paper, we tackle both of the above challenges at once. We develop a solution to train high-quality small language models using only a few hundred finetuning examples. We attempted to maximize the efficacy of knowledge transfer by designing pretraining objectives that closely resemble the finetuning objectives - we call this ``explicit objective alignment&quot;, ``objective alignment&quot; in short. In our tasks, objective alignment not only enabled smaller language models to perform the finetuning tasks equally well as their larger counterparts pretrained without objective alignment, but also reduced the number of finetuning examples required. We were able to develop a concept-of-interest tagger and an acronym detector using a 768 by 3 Transformer model and 200 finetuning examples for each task. We call finetuning in this model size and data size limit Few Example Learning (FEL).&#10;&#10;The main contributions of this paper are:&#10;\begin{itemize}[topsep=0pt]&#10;\item We propose pretraining objective alignment as a solution to developing small language models for NLU applications that have limited labeled data.&#10;\item We demonstrate our solution by building two NLU applications of reasonable accuracy using a 768 by 3 Transformer model and 200 finetuning examples each.&#10;\item We provide detailed steps to carry out object alignment, complete descriptions of the training parameters and recommendations for best practices.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Training, Artificial Intelligence, Objective Alignment Techniques, Few Example Learning" />
        </attvalues>
      </node>
      <node id="2408.11381" label="2408.11381">
        <attvalues>
          <attvalue for="0" value="RAGLAB: A Modular and Research-Oriented Unified Framework for&#10;  Retrieval-Augmented Generation" />
          <attvalue for="1" value="  Large Language Models (LLMs) demonstrate human-level capabilities in&#10;dialogue, reasoning, and knowledge retention. However, even the most advanced&#10;LLMs face challenges such as hallucinations and real-time updating of their&#10;knowledge. Current research addresses this bottleneck by equipping LLMs with&#10;external knowledge, a technique known as Retrieval Augmented Generation (RAG).&#10;However, two key issues constrained the development of RAG. First, there is a&#10;growing lack of comprehensive and fair comparisons between novel RAG&#10;algorithms. Second, open-source tools such as LlamaIndex and LangChain employ&#10;high-level abstractions, which results in a lack of transparency and limits the&#10;ability to develop novel algorithms and evaluation metrics. To close this gap,&#10;we introduce RAGLAB, a modular and research-oriented open-source library.&#10;RAGLAB reproduces 6 existing algorithms and provides a comprehensive ecosystem&#10;for investigating RAG algorithms. Leveraging RAGLAB, we conduct a fair&#10;comparison of 6 RAG algorithms across 10 benchmarks. With RAGLAB, researchers&#10;can efficiently compare the performance of various algorithms and develop novel&#10;algorithms.&#10;" />
          <attvalue for="2" value="&#10;&#10;Retrieval augmentation generation(RAG) leverages external knowledge to mitigate hallucination issues, ensure real-time knowledge updates, and protect private data with no parametric knowledge\cite{naive-rag-2017-qa-baseon-wiki,naiverag-2020-1,naiverag-2-Realm}. &#10;However, researchers face two main barriers to investigating new RAG algorithms. On the one hand, many published works are either not open-source or have difficulty setting up the environment. While open-source works lack modular design, it is hard to develop new algorithms or extend new datasets for evaluation. Researchers have to waste a lot of time developing new algorithms from scratch. On the other hand, a multitude of novel RAG algorithms have merged, including ITER-RETGEN\cite{iter-retgen}, RRR\cite{rrr-rag}, Self-Ask\cite{self-ask}, Active RAG\cite{active-rag}, Self-RAG\cite{selfrag}, etc. However, these RAG algorithms are not well aligned in their fundamental components and evaluation methodologies, making it difficult for researchers to accurately assess their improvements. As a result, the absence of a unified framework makes it difficult for researchers and engineers to select appropriate algorithms for varied contexts, potentially hindering the advancement of the field.&#10;&#10;While various current works are investigating these questions, such as LlamaIndex \cite{LlamaIndex}, LangChain\cite{LangChain2022}, Haystack\cite{haystack}, FastRAG\cite{fastRAG2023}, RALLE \cite{ralle-framework}, LocalRQA\cite{localrqa}, AutoRAG\cite{AutoRAG2024}, and FlashRAG\cite{FlashRAG}.&#10;LlamaIndex, LangChain, and Haystack are excessively encapsulation and lack transparency in internal operational mechanisms. Consequently, even experienced experts abandon tools like LangChain due to the lack of transparency\cite{woolf-abandon-langchain-2023}. FastRAG and RALLE offer light and transparent frameworks that enable users to assemble their own RAG systems using core components. AutoRAG provides comprehensive metrics to assist users in selecting an optimal RAG system for customized data. LocalRAG provides a wide selection of model training algorithms and evaluation methods. However, LocalRAG, FastRAG, AutoRAG, and RALLE do not reproduce published algorithms. Researchers still need to invest time in replicating algorithms using the provided components. FlashRAG addressed this issue by reproducing a substantial number of existing algorithms. However, FlashRAG lacks training functionalities and fails to properly align generators during inference, leading to unfair comparisons among various algorithms. For a more detailed comparison, refer to Table~\ref{tab:1-comparision_diff_framework}.&#10;&#10;To close this gap, we present RAGLAB, a researcher-oriented RAG toolkit for a fair comparison of existing RAG algorithms and simplify the process of developing new algorithms.&#10;RAGLAB provides a modular architecture for each component of the RAG system, providing an ideal platform for fair comparison of algorithms. Additionally, RAGLAB designs an interactive mode and user-friendly interface, facilitating both educational purposes and demonstrations.&#10;&#10;In this paper, we introduce the RAGLAB framework, giving an overview of core components and system workflows(section~\ref{sec:2-RAGLAB}).&#10;We standardized key experimental variables: generator fine-tuning, instructions, retrieval configurations, knowledge bases, and benchmark. As a result, we present a comprehensive and fair comparison of 6 RAG algorithms across 10 benchmarks(section~\ref{sec:3-Experiment}).&#10;&#10;RAGLAB is available on GitHub under the MIT license.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Open-Source Research Tools, Computer Science, Linguistics, Language Model Limitations, Retrieval Augmented Generation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.11511" label="2310.11511">
        <attvalues>
          <attvalue for="0" value="Self-RAG: Learning to Retrieve, Generate, and Critique through&#10;  Self-Reflection" />
          <attvalue for="1" value="  Despite their remarkable capabilities, large language models (LLMs) often&#10;produce responses containing factual inaccuracies due to their sole reliance on&#10;the parametric knowledge they encapsulate. Retrieval-Augmented Generation&#10;(RAG), an ad hoc approach that augments LMs with retrieval of relevant&#10;knowledge, decreases such issues. However, indiscriminately retrieving and&#10;incorporating a fixed number of retrieved passages, regardless of whether&#10;retrieval is necessary, or passages are relevant, diminishes LM versatility or&#10;can lead to unhelpful response generation. We introduce a new framework called&#10;Self-Reflective Retrieval-Augmented Generation (Self-RAG) that enhances an LM's&#10;quality and factuality through retrieval and self-reflection. Our framework&#10;trains a single arbitrary LM that adaptively retrieves passages on-demand, and&#10;generates and reflects on retrieved passages and its own generations using&#10;special tokens, called reflection tokens. Generating reflection tokens makes&#10;the LM controllable during the inference phase, enabling it to tailor its&#10;behavior to diverse task requirements. Experiments show that Self-RAG (7B and&#10;13B parameters) significantly outperforms state-of-the-art LLMs and&#10;retrieval-augmented models on a diverse set of tasks. Specifically, Self-RAG&#10;outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA,&#10;reasoning and fact verification tasks, and it shows significant gains in&#10;improving factuality and citation accuracy for long-form generations relative&#10;to these models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;{Retrieval-Augmented Generation.}&#10;{&#10;Retrieval-Augmented Generation (RAG) augments the input space of LMs with retrieved text passages~\cite{guu2020retrieval,lewis2020retrieval}, leading to large improvements in knowledge-intensive tasks after fine-tuning or used with off-the-shelf LMs~\cite{ram2023context}. &#10;A more recent work~\cite{luo2023sail} instruction-tunes an LM with a fixed number of retrieved passages prepended to input, or pre-train a retriever and LM jointly, followed by few-shot fine-tuning on task datasets~\cite{izacard2022few}. }&#10;While prior work often retrieves only once at the beginning, \cite{jiang2023active} propose to&#10;adaptively retrieve passages for generation on top of a proprietary LLM or ~\cite{schick2023toolformer} train an LM to generate API calls for named entities. &#10;Yet, the improved task performance of such approaches often comes at the expense of runtime efficiency~\cite{mallen2022not}, robustness to irrelevant context~\cite{pmlr-v202-shi23a}, and lack of attributions~\cite{liu2023evaluating,gao2023enabling}. &#10;We introduce a method to train an arbitrary LM to learn to use retrieval on-demand for diverse instruction-following queries and introduce controlled generation guided by reflections tokens to further improve generation quality and attributions. &#10;&#10;{Concurrent RAG work. }&#10;{&#10;A few concurrent works on RAG propose new training or prompting strategies to improve widely-adopted RAG approaches. &#10;\cite{lin2023radit} fine-tune both the retriever and LM on instruction-tuning datasets in two steps.&#10;While we also train our model on diverse instruction-following datasets, \model enables retrieval on demand and selection of the best possible model output via fine-grained self-reflection, making it widely applicable and more robust and controllable. &#10;\cite{Yoran2023MakingRL} use a natural language inference model and \cite{xu2023recomp} use a summarization model to filter out or compress retrieved passages before using them to prompt the LM to generate the output.&#10;\model processes passages in parallel and filters out irrelevant ones through self-reflection, without relying on external models at inference. Moreover, our self-reflection mechanism also evaluates other aspects of the model output quality including factuality.&#10;LATS~\cite{zhou2023language} prompt off-the-shelf LMs to search for relevant information for question answering tasks and to generate with tree search, guided by LM-generated value scores. &#10;While their value function simply indicates an overall score of each generation, \model trains to an arbitrary LM to learn to generate fine-grained self-reflection and customizable inference. &#10;}&#10;&#10;{Training and generating with critics.}&#10;Training LLMs with reinforcement learning (e.g., Proximal Policy Optimization or PPO; \cite{schulman2017proximal}) from human feedback (RLHF) has proven effective in aligning LLMs with human preferences~\cite{ouyang2022training}. &#10;\cite{wu2023fine} introduce fine-grained RLHF with multiple reward models. &#10;Though our work also studies fine-grained critique on retrieval and generation, we train our target LM on task examples augmented with reflection tokens from a critic &#10;model offline, with a far lower training cost compared to RLHF. In addition, reflection tokens in \model enable controllable generation at inference, while RLHF focuses on human preference alignment during training.&#10;Other works use general control tokens to guide LM generation~\cite{lu2022quark,korbak2023pretraining}, while \model uses reflection tokens to decide the need for retrieval and to self-evaluate generation quality.&#10;\cite{xie2023decomposition} propose a self-evaluation-guided decoding framework, but they focus only on reasoning tasks with one evaluation dimension (reasoning path consistency) and without retrieval.&#10;Recent work on LLM refinement \cite{dhuliawala2023chainofverification, madaan2023selfrefine, paul2023refiner} prompts a model to generate task output, natural language feedback and refined task output iteratively, but at the cost of inference efficiency.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2308.10633" label="2308.10633">
        <attvalues>
          <attvalue for="0" value="RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented&#10;  Large Language Models" />
          <attvalue for="1" value="  Retrieval-augmented large language models (R-LLMs) combine pre-trained large&#10;language models (LLMs) with information retrieval systems to improve the&#10;accuracy of factual question-answering. However, current libraries for building&#10;R-LLMs provide high-level abstractions without sufficient transparency for&#10;evaluating and optimizing prompts within specific inference processes such as&#10;retrieval and generation. To address this gap, we present RaLLe, an open-source&#10;framework designed to facilitate the development, evaluation, and optimization&#10;of R-LLMs for knowledge-intensive tasks. With RaLLe, developers can easily&#10;develop and evaluate R-LLMs, improving hand-crafted prompts, assessing&#10;individual inference processes, and objectively measuring overall system&#10;performance quantitatively. By leveraging these features, developers can&#10;enhance the performance and accuracy of their R-LLMs in knowledge-intensive&#10;generation tasks. We open-source our code at https://github.com/yhoshi3/RaLLe.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have shown great potential for natural language understanding and generation tasks \cite{gpt3,palm,gpt4}.&#10;However, they face challenges when answering factual questions due to hallucinations (or confabulations) \cite{hallucination1,hallucination2}, outdated parametric knowledge \cite{streamingqa}, and memory efficiency of parametric knowledge \cite[e.g.,][]{LMasKB2}.&#10;To address these limitations, researchers have turned to the retrieval-augmented approach used in open-domain question answering (QA) \cite{odqa}, hereinafter referred to as retrieval-augmented LLMs or R-LLMs.&#10;&#10;In comparison to closed-book settings where language models generate answers without retrieval, R-LLMs (open-book settings) enable the retrieval of relevant information from external databases or corpora \cite{survey,simplyretrieve}, which has led to improved accuracy in open-domain QA \cite{replug}.&#10;Additionally, R-LLMs can acquire extended features even without additional training, such as explicit references, relief from fact hallucination \cite{webgpt}, and easy updates to the knowledge source \cite[e.g.,][]{realm,simplyretrieve}.&#10;&#10;Retrieval-augmented generation needs further research and development to reach its full potential.&#10;For example, even though the retriever-reader system has been trained on the Natural Questions (NQ) dataset \cite{nq}, its F1 score on the short answer task is 68.3 and still lags behind the oracle F1 score of 75.7 \cite{unanswerable}.&#10;This implies that further improvements can be made to the retrieval-augmented generation approach.&#10;Additionally, users would be probably aware that the outputs generated by R-LLMs may contain factual errors, particularly when applied to knowledge-intensive tasks.&#10;However, there is currently a lack of accessible evaluation framework to assess their output quality.&#10;This makes it difficult to identify areas for improvement.&#10;&#10;Furthermore, having effective tools for developing R-LLMs is crucial.&#10;These tools should enable the design of inference steps such as retrieve-then-generate, selecting the combination of retrievers and LLMs, evaluating the performance of the entire system, and testing the prompts used in each inference step.&#10;Currently available tools, such as the ChatGPT Retrieval Plugin, Guidance, and LangChain \cite{langchain}, offer a high degree of abstraction, making it challenging to verify the functionality of individual inference steps or optimize prompts within each step.&#10;This lack of transparency might hinder the optimization of R-LLMs.&#10;&#10;In this paper, we propose RaLLe, an accessible framework for Retrieval-Augmented Large Language model development and Evaluation.&#10;We also present evaluation results of several R-LLMs that we have constructed by using open-source retrievers and LLMs.&#10;To the best of our knowledge, RaLLe is the first framework that empowers R-LLM developers and open-domain QA researchers to efficiently develop, evaluate, and improve R-LLMs using objective metrics.&#10;&#10;RaLLe offers several key benefits:&#10;\begin{enumerate}&#10;&#09;\item Easy development and testing: users can easily select, combine, and test various retrievers and LLMs, especially open-source models, within a graphical interface.&#10;&#09;\item Objective evaluation of R-LLMs: RaLLe provides reproducible experiments with objective benchmarks/metrics, enabling objective assessments of R-LLM performance.&#10;&#09;\item Transparent prompt engineering: all inputs (prompts) and outputs of each action are visible to developers, allowing for easy exploration and optimization of the prompts.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.08284" label="2209.08284">
        <attvalues>
          <attvalue for="0" value="Structured Knowledge Grounding for Question Answering" />
          <attvalue for="1" value="  Can language models (LM) ground question-answering (QA) tasks in the&#10;knowledge base via inherent relational reasoning ability? While previous models&#10;that use only LMs have seen some success on many QA tasks, more recent methods&#10;include knowledge graphs (KG) to complement LMs with their more logic-driven&#10;implicit knowledge. However, effectively extracting information from structured&#10;data, like KGs, empowers LMs to remain an open question, and current models&#10;rely on graph techniques to extract knowledge. In this paper, we propose to&#10;solely leverage the LMs to combine the language and knowledge for knowledge&#10;based question-answering with flexibility, breadth of coverage and structured&#10;reasoning. Specifically, we devise a knowledge construction method that&#10;retrieves the relevant context with a dynamic hop, which expresses more&#10;comprehensivenes than traditional GNN-based techniques. And we devise a deep&#10;fusion mechanism to further bridge the information exchanging bottleneck&#10;between the language and the knowledge. Extensive experiments show that our&#10;model consistently demonstrates its state-of-the-art performance over&#10;CommensenseQA benchmark, showcasing the possibility to leverage LMs solely to&#10;robustly ground QA into the knowledge base.&#10;" />
          <attvalue for="2" value="&#10;Question Answering (QA) task: given a question context, a QA model needs to produce an answer for it. This requires the QA model to gather relevant knowledge and reason over it. &#10;Explicit knowledge source: knowledge graphs (e.g., ConceptNet)&#10;&#10;Recent works aim to reason based on both language model and Knoweldge graphs. Language model have been pre-trained on raw texts, storing knowledge explicitly; knowledge graphs are networks connecting entities in a logical way, storing knowledge implicitly. The two systems each have their own role: KGs contains easy-to-follow logic patterns, but lacks coverage and are noisy, thanks to the excessively simplification of logic; LMs vice versa. However, the models differ in how they integrate the two different system together. Starting with QA-GNN, the model runs GNNs over KGs to retrieve a subgraph using entity matching and path finding, then jointly reason over both modalities by either adding question context to the subgraph as an additional node or introducing additional interaction tokens and layers into the model architecture. The drawbacks of using GNN is evident as the retrieved graph is only limited in terms of knowledge as compromised by training performance.&#10;&#10;It's difficult to combine these two modes. First, identifying relevant knowledge from a huge KG in the context of a question is difficult. Second, the optimum technique to combine context with the retrieved graph reasoning is unclear. QA-GNN~\cite{Yasunaga2021QAGNNRW} and GreaseLM~\cite{greaseLM}, the most recent work (as far as we know), retrieve a subgraph using entity matching and path finding, then jointly reason over both modalities by either adding question context to the subgraph as an additional node or introducing additional interaction tokens and layers into the model architecture.&#10;&#10;Our main contributions are expected as follows:&#10;\begin{itemize}&#10; \item We propose a new way to fuse these two knowledge bases with only LM involved.&#10; \item We convert KG data into a natural language format, and then run a LM over it to gain embedding of KG knowledge.&#10; \item We achieve comparable results to existing SOTA over public QA benchmark CommonsenseQA and OpenbookQA.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Question Answering, Computer Science, Linguistics, Cognitive Science, Language Models, Knowledge Graphs, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2104.06378" label="2104.06378">
        <attvalues>
          <attvalue for="0" value="QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question&#10;  Answering" />
          <attvalue for="1" value="  The problem of answering questions using knowledge from pre-trained language&#10;models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA&#10;context (question and answer choice), methods need to (i) identify relevant&#10;knowledge from large KGs, and (ii) perform joint reasoning over the QA context&#10;and KG. In this work, we propose a new model, QA-GNN, which addresses the above&#10;challenges through two key innovations: (i) relevance scoring, where we use LMs&#10;to estimate the importance of KG nodes relative to the given QA context, and&#10;(ii) joint reasoning, where we connect the QA context and KG to form a joint&#10;graph, and mutually update their representations through graph neural networks.&#10;We evaluate our model on QA benchmarks in the commonsense (CommonsenseQA,&#10;OpenBookQA) and biomedical (MedQA-USMLE) domains. QA-GNN outperforms existing&#10;LM and LM+KG models, and exhibits capabilities to perform interpretable and&#10;structured reasoning, e.g., correctly handling negation in questions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.08860" label="2201.08860">
        <attvalues>
          <attvalue for="0" value="GreaseLM: Graph REASoning Enhanced Language Models for Question&#10;  Answering" />
          <attvalue for="1" value="  Answering complex questions about textual narratives requires reasoning over&#10;both stated context and the world knowledge that underlies it. However,&#10;pretrained language models (LM), the foundation of most modern QA systems, do&#10;not robustly represent latent relationships between concepts, which is&#10;necessary for reasoning. While knowledge graphs (KG) are often used to augment&#10;LMs with structured representations of world knowledge, it remains an open&#10;question how to effectively fuse and reason over the KG representations and the&#10;language context, which provides situational constraints and nuances. In this&#10;work, we propose GreaseLM, a new model that fuses encoded representations from&#10;pretrained LMs and graph neural networks over multiple layers of modality&#10;interaction operations. Information from both modalities propagates to the&#10;other, allowing language context representations to be grounded by structured&#10;world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in&#10;the context to inform the graph representations of knowledge. Our results on&#10;three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA)&#10;and medical question answering (i.e., MedQA-USMLE) domains demonstrate that&#10;GreaseLM can more reliably answer questions that require reasoning over both&#10;situational constraints and structured knowledge, even outperforming models 8x&#10;larger.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Integrating KG information has become a popular research area for improving neural QA systems. Some works explore using two-tower models to answer questions, where a graph representation of knowledge and language representation are fused with no interaction between them \cite{wang2019improving}. Other works seek to use one modality to ground the other, such as using an encoded representation of a linked KG to augment the textual representation of a QA example (e.g., Knowledgeable Reader, \cite{mihaylov2018knowledgeable}; KagNet, \cite{lin2019kagnet}; KT-NET, \cite{yang2019enhancing}). Others reverse the flow of information and use a representation of the text (e.g., final layer of LM) to provide an augmentation to a graph reasoning model over an extracted KG for the example (e.g., MHGRN, \cite{feng2020scalable}; \cite{lv2020graph}). In all of these settings, however, the interaction between both modalities is limited as information between them only flows one way. &#10;&#10;More recent approaches explore deeper integrations of both modalities. Certain approaches learn to access implicit knowledge encoded in LMs \cite{Bosselut2019COMETCT,petroni2019language,Hwang2021COMETATOMIC2O} by training on structured KG data, and then use the LM to generate local KGs that can be used for QA \cite{wang2020connecting,bosselut2021dynamic}. However, these approaches discard the static KG once they train the LM on its facts, losing important structure that can guide reasoning. More recently, QA-GNN \cite{Yasunaga2021QAGNNRW} proposed to jointly update the LM and GNN representations via message passing. However, they use a single pooled representation of the LM to seed the textual component of this joint structure, limiting the updates that can be made to the textual representation. In contrast to prior works, we propose to make individual token representations in the LM and node representations in the GNN mix for multiple layers, enabling representations of both modalities to reflect particularities of the other (e.g., knowledge grounds language; language nuances specifies which knowledge is important). Simultaneously, we retain the individual structure of both modalities, which we demonstrate improves QA performance substantially (\S\ref{sec:experiments}).&#10;&#10;Additionally, some works explore integrating knowledge graphs with language models in the pretraining stage. However, much like for QA, the modality interaction is typically limited to knowledge feeding language \cite{Zhang2019ERNIEEL,Shen2020ExploitingSK,Yu2020JAKETJP}, rather than designing interactions across multiple layers. \cite{Sun2020CoLAKECL}'s work is perhaps most similar, but they do not use the same interaction bottleneck, requiring high-precision entity mention spans for linking, and they limit expressivity through shared modality parameters for the LM and KG.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2304.07772" label="2304.07772">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Evaluation of Neural SPARQL Query Generation from&#10;  Natural Language Questions" />
          <attvalue for="1" value="  In recent years, the field of neural machine translation (NMT) for SPARQL&#10;query generation has witnessed significant growth. Incorporating the copy&#10;mechanism with traditional encoder-decoder architectures and using pre-trained&#10;encoder-decoders and large language models have set new performance benchmarks.&#10;This paper presents various experiments that replicate and expand upon recent&#10;NMT-based SPARQL generation studies, comparing pre-trained language models&#10;(PLMs), non-pre-trained language models (NPLMs), and large language models&#10;(LLMs), highlighting the impact of question annotation and the copy mechanism&#10;and testing various fine-tuning methods using LLMs. In particular, we provide a&#10;systematic error analysis of the models and test their generalization ability.&#10;Our study demonstrates that the copy mechanism yields significant performance&#10;enhancements for most PLMs and NPLMs. Annotating the data is pivotal to&#10;generating correct URIs, with the &quot;tag-within&quot; strategy emerging as the most&#10;effective approach. Additionally, our findings reveal that the primary source&#10;of errors stems from incorrect URIs in SPARQL queries that are sometimes&#10;replaced with hallucinated URIs when using base models. This does not happen&#10;using the copy mechanism, but it sometimes leads to selecting wrong URIs among&#10;candidates. Finally, the performance of the tested LLMs fell short of achieving&#10;the desired outcomes.&#10;" />
          <attvalue for="2" value="&#10;The Semantic Web (SW) provides a framework for storing and organizing structured data, following standards defined by the World Wide Web Consortium (W3C). To access data within a Knowledge Base (KB) on the SW, one must use the SPARQL query language, which can be challenging for non-experts and necessitates a familiarity with the KB's structure and ontology. For greater usability of the SW, it is crucial to develop models that allow users to query KBs easily using natural language. &#10;&#10;One approach to generating queries based on natural language questions is to use neural machine translation (NMT).&#10;This is typically done using an encoder-decoder architecture, where input sentences pass through the encoder to generate a vector that holds their semantics, and the decoder produces tokens at each time step based on the encoder's output and previous tokens. However, classic (without pre-training) NMT architectures have common limitations. First, they require a fixed vocabulary, which cannot be updated without restarting the entire learning process. Therefore, if a new word appears during the testing phase, it is processed as an unknown (&lt;unk&gt;) token. &#10;Second, it is not trivial for a NMT model to learn to transform natural language into KB elements (URIs), particularly since most elements are seen very few times during training. These limitations are particularly important in the case of NMT-based SPARQL query generation, as unknown words are often elements from the KB, which can lead to irrelevant URIs in the generated queries. To address these limitations, the copy mechanism was proposed \cite{copy-machanism}, which allows tokens from the input to be directly copied into the output based on a knowledge base vocabulary that includes KB URIs. However, copy-based models require the annotation of KB URIs in the natural language questions. &#10;&#10;The recent development of pretrained language models and their application to SPARQL query generation has opened new potential avenues \cite{lin2022sparql} \cite{reyd2023comprehensive} \cite{tran2021spbert} \cite{huang2021unseen} \cite{huang2021unseen} \cite{naik2023sql}. In fact, Lehmann et al. \cite{lehmann2023language} suggest the use of controlled natural language as a target for Knowledge Graph Question Answering (KGQA) semantic parsing. They hypothesize that pretraining LLMs on textual data can facilitate parsing into controlled natural language for KGQA with limited training data requirements, reducing the cost and effort of collecting high-quality training data. SPARQL can be considered as such a controlled natural language. &#10;&#10;Initial experiments on pretrained language models for SPARQL query generation indicate that while they may outperform their non-pretrained counterparts \cite{banerjee2022modern}, they still exhibit some limitations in handling unknown URIs to some degree, but most importantly, they exhibit poor generalization abilities as their performance drops when new question templates are used at test time \cite{reyd2023comprehensive}. With this in mind, while most datasets rely on template-based questions, the ability of neural models trained on template questions to handle natural (formulated by humans, paraphrases) questions remains unexplored. &#10;&#10;In this paper, our goal is to expand upon these experiments and provide a systematic comparison of several models. In particular, we aim to identify the failures of state-of-the-art neural query generators, in terms of SPARQL structures, incorrect URIs, and hallucinated URIs. We also test models' generalization capabilities with natural - non template-based- questions. Additionally, we include Large Language Models (LLMs) to assess whether they exhibit the same shortcomings. &#10;In this context, we address the following research questions :&#10;\begin{enumerate}&#10; \item Does the annotation of KB elements in the natural language questions improve the SPARQL query generation for all models? &#10; \item Does the integration of a copy mechanism in NPLMs and PLMs improve the accuracy of the KB elements in the SPARQL queries?&#10; \item Are Large Language Models (LLMs) effective in this task, and which fine-tuning/prompting technique performs better?&#10; \item What are the most common generation errors, and what types of tokens are often generated instead of the expected types?&#10; \item How do models trained on template-based questions perform on naturally reformulated questions?&#10;\end{enumerate}&#10;&#10;Our contributions include the following aspects:&#10;&#10;\begin{enumerate}&#10;\item{Using annotation, we compare the results of two NPLMs (ConvSeq2Seq and Transformers), two PLMs (BART \cite{BART} and T5 \cite{T5}) and two LLMs (Llama2 \cite{touvron2023llama} and Code Llamav2 7B \cite{roziere2023code}; }&#10;\item{We evaluate the impact of the copy mechanism and question annotations and experiment with &quot;raw-question&quot; (non-annotated) questions, &quot;tag-within&quot; questions where we replace natural language elements with their KBs URIs counterparts and tag-end questions, where we list KB URIs with their labels at the end of the questions;}&#10;\item{We experiment with standard fine-tuning and instruction fine-tuning on two Large Language Models (LLMs), namely Llama\cite{touvron2023llama} and Code Llama\cite{roziere2023code} and measure the impact of training data size on the results;} \item{We perform a fine-grained analysis of the generation errors with their type distribution for all the models;}&#10;\item{We test the generalization capabilities of the best-performing models using questions reformulated in different settings.}&#10;\end{enumerate}&#10;&#10;To our knowledge, such detailed study has not been done yet especially including the performance of large language models, the generalization abilities of all models, and the fine-grained identification of errors made at generation time. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Query Generation Techniques, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2106.09997" label="2106.09997">
        <attvalues>
          <attvalue for="0" value="SPBERT: An Efficient Pre-training BERT on SPARQL Queries for Question&#10;  Answering over Knowledge Graphs" />
          <attvalue for="1" value="  In this paper, we propose SPBERT, a transformer-based language model&#10;pre-trained on massive SPARQL query logs. By incorporating masked language&#10;modeling objectives and the word structural objective, SPBERT can learn&#10;general-purpose representations in both natural language and SPARQL query&#10;language. We investigate how SPBERT and encoder-decoder architecture can be&#10;adapted for Knowledge-based QA corpora. We conduct exhaustive experiments on&#10;two additional tasks, including SPARQL Query Construction and Answer&#10;Verbalization Generation. The experimental results show that SPBERT can obtain&#10;promising results, achieving state-of-the-art BLEU scores on several of these&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.12793" label="2204.12793">
        <attvalues>
          <attvalue for="0" value="Modern Baselines for SPARQL Semantic Parsing" />
          <attvalue for="1" value="  In this work, we focus on the task of generating SPARQL queries from natural&#10;language questions, which can then be executed on Knowledge Graphs (KGs). We&#10;assume that gold entity and relations have been provided, and the remaining&#10;task is to arrange them in the right order along with SPARQL vocabulary, and&#10;input tokens to produce the correct SPARQL query. Pre-trained Language Models&#10;(PLMs) have not been explored in depth on this task so far, so we experiment&#10;with BART, T5 and PGNs (Pointer Generator Networks) with BERT embeddings,&#10;looking for new baselines in the PLM era for this task, on DBpedia and Wikidata&#10;KGs. We show that T5 requires special input tokenisation, but produces state of&#10;the art performance on LC-QuAD 1.0 and LC-QuAD 2.0 datasets, and outperforms&#10;task-specific models from previous works. Moreover, the methods enable semantic&#10;parsing for questions where a part of the input needs to be copied to the&#10;output query, thus enabling a new paradigm in KG semantic parsing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.08744" label="1909.08744">
        <attvalues>
          <attvalue for="0" value="Low-Resource Parsing with Crosslingual Contextualized Representations" />
          <attvalue for="1" value="  Despite advances in dependency parsing, languages with small treebanks still&#10;present challenges. We assess recent approaches to multilingual contextual word&#10;representations (CWRs), and compare them for crosslingual transfer from a&#10;language with a large treebank to a language with a small or nonexistent&#10;treebank, by sharing parameters between languages in the parser itself. We&#10;experiment with a diverse selection of languages in both simulated and truly&#10;low-resource scenarios, and show that multilingual CWRs greatly facilitate&#10;low-resource dependency parsing even without crosslingual supervision such as&#10;dictionaries or parallel text. Furthermore, we examine the non-contextual part&#10;of the learned language models (which we call a &quot;decontextual probe&quot;) to&#10;demonstrate that polyglot language models better encode crosslingual lexical&#10;correspondence compared to aligned monolingual language models. This analysis&#10;provides further evidence that polyglot training is an effective approach to&#10;crosslingual transfer.&#10;" />
          <attvalue for="2" value="&#10;Dependency parsing has achieved new states of the art using distributed word representations in neural networks, trained with large amounts of annotated data \cite{dozatmanning2017,dozat-qi-manning:2017:K17-3,ma-EtAl:2018:Long2,Che2018ElmoUD}.&#10;However, many languages are low-resource, with small or no treebanks, which presents a severe challenge in developing accurate parsing systems in those languages. &#10;One way to address this problem is with a crosslingual solution that makes use of a language with a large treebank and raw text in both languages.&#10;The hypothesis behind this approach is that, although each language is unique, different languages manifest similar characteristics (e.g., morphological, lexical, syntactic) which can be exploited by training a single polyglot model with data from multiple languages \cite{ammarthesis}.&#10;&#10;Recent work has extended contextual word representations (cwrs) multilingually either by training a polyglot language model (LM) on a mixture of data from multiple languages (joint training approach; \cite{mulcaire_NAACL2019}; \cite{LampleConneau2019}) &#10;or by aligning multiple monolingual language models crosslingually (retrofitting approach; \cite{Schuster2019CrossLingual,aldarmaki_diab2019}).&#10;These multilingual representations have been shown to facilitate crosslingual transfer on several tasks, including Universal Dependencies parsing and natural language inference.&#10;In this work, we assess these two types of methods by using them for low-resource dependency parsing, and discover that the joint training approach substantially outperforms the retrofitting approach.&#10;We further apply multilingual cwrs produced by the joint training approach to diverse languages, and show that it is still effective in transfer between distant languages, though we find that phylogenetically related source languages are generally more helpful.&#10;&#10;We hypothesize that joint polyglot training is more successful than retrofitting because it induces a degree of lexical correspondence between languages that the linear transformation used in retrofitting methods cannot capture. &#10;To test this hypothesis, we design a decontextual probe. We decontextualize cwrs into non-contextual word vectors that retain much of cwrs' task-performance benefit, and evaluate the crosslingual transferability of language models via word translation.&#10;In our decontextualization framework, we use a single LSTM cell without recurrence to obtain a context-independent vector, thereby allowing for a direct probe into the LSTM networks independent of a particular corpus.&#10;We show that decontextualized vectors from the joint training approach yield representations that score higher on a word translation task than the retrofitting approach or word type vectors such as fastText \cite{bojanowski2017enriching}.&#10;This finding provides evidence that polyglot language models encode crosslingual similarity, specifically crosslingual lexical correspondence, that a linear alignment between monolingual language models does not. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Multilingual Parsing, Crosslingual Transfer, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1805.01087" label="1805.01087">
        <attvalues>
          <attvalue for="0" value="Stack-Pointer Networks for Dependency Parsing" />
          <attvalue for="1" value="  We introduce a novel architecture for dependency parsing: \emph{stack-pointer&#10;networks} (\textbf{\textsc{StackPtr}}). Combining pointer&#10;networks~\citep{vinyals2015pointer} with an internal stack, the proposed model&#10;first reads and encodes the whole sentence, then builds the dependency tree&#10;top-down (from root-to-leaf) in a depth-first fashion. The stack tracks the&#10;status of the depth-first search and the pointer networks select one child for&#10;the word at the top of the stack at each step. The \textsc{StackPtr} parser&#10;benefits from the information of the whole sentence and all previously derived&#10;subtree structures, and removes the left-to-right restriction in classical&#10;transition-based parsers. Yet, the number of steps for building any (including&#10;non-projective) parse tree is linear in the length of the sentence just as&#10;other transition-based parsers, yielding an efficient decoding algorithm with&#10;$O(n^2)$ time complexity. We evaluate our model on 29 treebanks spanning 20&#10;languages and different dependency annotation schemas, and achieve&#10;state-of-the-art performance on 21 of them.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.09697" label="1902.09697">
        <attvalues>
          <attvalue for="0" value="Polyglot Contextual Representations Improve Crosslingual Transfer" />
          <attvalue for="1" value="  We introduce Rosita, a method to produce multilingual contextual word&#10;representations by training a single language model on text from multiple&#10;languages. Our method combines the advantages of contextual word&#10;representations with those of multilingual representation learning. We produce&#10;language models from dissimilar language pairs (English/Arabic and&#10;English/Chinese) and use them in dependency parsing, semantic role labeling,&#10;and named entity recognition, with comparisons to monolingual and&#10;non-contextual variants. Our results provide further evidence for the benefits&#10;of polyglot learning, in which representations are shared across multiple&#10;languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.03243" label="1903.03243">
        <attvalues>
          <attvalue for="0" value="Context-Aware Cross-Lingual Mapping" />
          <attvalue for="1" value="  Cross-lingual word vectors are typically obtained by fitting an orthogonal&#10;matrix that maps the entries of a bilingual dictionary from a source to a&#10;target vector space. Word vectors, however, are most commonly used for sentence&#10;or document-level representations that are calculated as the weighted average&#10;of word embeddings. In this paper, we propose an alternative to word-level&#10;mapping that better reflects sentence-level cross-lingual similarity. We&#10;incorporate context in the transformation matrix by directly mapping the&#10;averaged embeddings of aligned sentences in a parallel corpus. We also&#10;implement cross-lingual mapping of deep contextualized word embeddings using&#10;parallel sentences with word alignments. In our experiments, both approaches&#10;resulted in cross-lingual sentence embeddings that outperformed&#10;context-independent word mapping in sentence translation retrieval.&#10;Furthermore, the sentence-level transformation could be used for word-level&#10;mapping without loss in word translation quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.03152" label="2211.03152">
        <attvalues>
          <attvalue for="0" value="Noisy Channel for Automatic Text Simplification" />
          <attvalue for="1" value="  In this paper we present a simple re-ranking method for Automatic Sentence&#10;Simplification based on the noisy channel scheme. Instead of directly computing&#10;the best simplification given a complex text, the re-ranking method also&#10;considers the probability of the simple sentence to produce the complex&#10;counterpart, as well as the probability of the simple text itself, according to&#10;a language model. Our experiments show that combining these scores outperform&#10;the original system in three different English datasets, yielding the best&#10;known result in one of them. Adopting the noisy channel scheme opens new ways&#10;to infuse additional information into ATS systems, and thus to control&#10;important aspects of them, a known limitation of end-to-end neural seq2seq&#10;generative models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic Text Simplification (ATS) aims to rewrite text into a form&#10;that is easier to understand, while retaining the original&#10;meaning. Its is an active area of research with many interests in improving web accessibility \cite{Alarcon2021}, for cognitive disabled users \cite{kamran2022web,moreno2021designing}, for scientific information access \cite{Ermakova}, and in general it is intended for social good \cite{SocialGood}.&#10;&#10;In the last years there has been a remarkable advance in ATS due to&#10;the advent of deep learning techniques and neural language models. This has lead to a variety of methods that&#10;follow the seq2seq architecture to build models that estimate $p(y|x)$, the probability of producing a&#10;simple sentence $y$ given the complex sentence $x$ \cite{nisioi2017exploring,zhang-lapata-2017-sentence,martin2020multilingual,Lin_Wan_2021,Omelianchuk}. One of the main problems of this approach is the lack of control mechanisms to prevent the system to hallucinate or produce repetition when generating $y$, as well to infuse new information into the system. Previous works have tried to control the generation of simple sentences by using control codes that condition the system output with variables such as length, number paraphrases, lexical complexity, and syntactic complexity, etc \cite{martin-etal-2020-controllable,sheang-saggion-2021-controllable}. Other works such as \cite{Clive-2021} use control prefixes, including input-dependent conditional information pre-trained models, incorporating learnable attribute-level representations at different layers of a transformer.&#10;&#10;In this paper we propose an alternative method to control the generated simplifications of a seq2seq ATS model. Instead of controlling the simplifications at decoding time, we propose to re-rank the top candidates of an neural seq2seq model according to the noisy channel scheme ~\cite{10.5555/972470.972474,yee-etal-2019-simple,Yu-Lei}. The noisy channel scheme decomposes $p(y|x)$ into $p(x|y)$, the probability of generating the complex sentence $x$ given the simple sentence $y$, and $p(y)$, the probability of the simple sentence $y$. We present a re-ranking method that considers all three probabilities using different models. Because each of the models is independently trained on different data, our method is able to infuse additional knowledge into a traditional seq2seq ATS system. The noisy channel method has been mainly applied in machine translation, but it was also applied for Document Compression \cite{daume2002noisy}, exactly in the dropping word task (related to summarization). Dropping or deleting words is also one of the operations an ATS systems should carry our, but it is not limited to that: ATS systems should also split sentences, reorder words or sentences and add necessary words or phrases. Moreover, ATS systems's output should be coherent, and by only dropping words coherence can become deficient.&#10;&#10;The contributions of this paper are the following:&#10;\begin{itemize}&#10;\item We propose a re-ranking method for seq2seq ATS systems based on noisy channel.&#10;\item We show that our method outperforms the original system in three&#10; different English datasets, yielding the best known result in one of them.&#10;\item We show that the method is able to infuse new information in neural&#10; ATS systems, as the additional models in the noisy channel scheme can be&#10; trained on complementary data.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Generative Models, Computer Science, Linguistics, Artificial Intelligence, Mathematics, Sentence Simplification, Natural Language Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="1703.10931" label="1703.10931">
        <attvalues>
          <attvalue for="0" value="Sentence Simplification with Deep Reinforcement Learning" />
          <attvalue for="1" value="  Sentence simplification aims to make sentences easier to read and understand.&#10;Most recent approaches draw on insights from machine translation to learn&#10;simplification rewrites from monolingual corpora of complex and simple&#10;sentences. We address the simplification problem with an encoder-decoder model&#10;coupled with a deep reinforcement learning framework. Our model, which we call&#10;{\sc Dress} (as shorthand for {\bf D}eep {\bf RE}inforcement {\bf S}entence&#10;{\bf S}implification), explores the space of possible simplifications while&#10;learning to optimize a reward function that encourages outputs which are&#10;simple, fluent, and preserve the meaning of the input. Experiments on three&#10;datasets demonstrate that our model outperforms competitive simplification&#10;systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.05070" label="2103.05070">
        <attvalues>
          <attvalue for="0" value="Text Simplification by Tagging" />
          <attvalue for="1" value="  Edit-based approaches have recently shown promising results on multiple&#10;monolingual sequence transduction tasks. In contrast to conventional&#10;sequence-to-sequence (Seq2Seq) models, which learn to generate text from&#10;scratch as they are trained on parallel corpora, these methods have proven to&#10;be much more effective since they are able to learn to make fast and accurate&#10;transformations while leveraging powerful pre-trained language models. Inspired&#10;by these ideas, we present TST, a simple and efficient Text Simplification&#10;system based on sequence Tagging, leveraging pre-trained Transformer-based&#10;encoders. Our system makes simplistic data augmentations and tweaks in training&#10;and inference on a pre-existing system, which makes it less reliant on large&#10;amounts of parallel training data, provides more control over the outputs and&#10;enables faster inference speeds. Our best model achieves near state-of-the-art&#10;performance on benchmark test datasets for the task. Since it is fully&#10;non-autoregressive, it achieves faster inference speeds by over 11 times than&#10;the current state-of-the-art text simplification system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.02677" label="1910.02677">
        <attvalues>
          <attvalue for="0" value="Controllable Sentence Simplification" />
          <attvalue for="1" value="  Text simplification aims at making a text easier to read and understand by&#10;simplifying grammar and structure while keeping the underlying information&#10;identical. It is often considered an all-purpose generic task where the same&#10;simplification is suitable for all; however multiple audiences can benefit from&#10;simplified text in different ways. We adapt a discrete parametrization&#10;mechanism that provides explicit control on simplification systems based on&#10;Sequence-to-Sequence models. As a result, users can condition the&#10;simplifications returned by a model on attributes such as length, amount of&#10;paraphrasing, lexical complexity and syntactic complexity. We also show that&#10;carefully chosen values of these attributes allow out-of-the-box&#10;Sequence-to-Sequence models to outperform their standard counterparts on&#10;simplification benchmarks. Our model, which we call ACCESS (as shorthand for&#10;AudienCe-CEntric Sentence Simplification), establishes the state of the art at&#10;41.87 SARI on the WikiLarge test set, a +1.42 improvement over the best&#10;previously reported score.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2110.08329" label="2110.08329">
        <attvalues>
          <attvalue for="0" value="Control Prefixes for Parameter-Efficient Text Generation" />
          <attvalue for="1" value="  Prefix-tuning is a powerful lightweight technique for adapting a large&#10;pre-trained language model to a downstream application. However, it uses the&#10;same dataset-level tuned prompt for all examples in the dataset. We extend this&#10;idea and propose a dynamic method, Control Prefixes, which allows for the&#10;inclusion of conditional input-dependent information, combining the benefits of&#10;prompt tuning and controlled generation. The method incorporates&#10;attribute-level learnable representations into different layers of a&#10;pre-trained transformer, allowing for the generated text to be guided in a&#10;particular direction. We provide a systematic evaluation of the technique and&#10;apply it to five datasets from the GEM benchmark for natural language&#10;generation (NLG). Although the aim is to develop a parameter-efficient model,&#10;we show Control Prefixes can even outperform full fine-tuning methods. We&#10;present state-of-the-art results on several data-to-text datasets, including&#10;WebNLG.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="0907.0806" label="0907.0806">
        <attvalues>
          <attvalue for="0" value="A Noisy-Channel Model for Document Compression" />
          <attvalue for="1" value="  We present a document compression system that uses a hierarchical&#10;noisy-channel model of text production. Our compression system first&#10;automatically derives the syntactic structure of each sentence and the overall&#10;discourse structure of the text given as input. The system then uses a&#10;statistical hierarchical model of text production in order to drop&#10;non-important syntactic and discourse constituents so as to generate coherent,&#10;grammatical document compressions of arbitrary length. The system outperforms&#10;both a baseline and a sentence-based compression system that operates by&#10;simplifying sequentially all sentences in a text. Our results support the claim&#10;that discourse knowledge plays an important role in document summarization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.13233" label="2403.13233">
        <attvalues>
          <attvalue for="0" value="Technical Report: Competition Solution For BetterMixture" />
          <attvalue for="1" value="  In the era of flourishing large-scale models, the challenge of selecting and&#10;optimizing datasets from the vast and complex sea of data, to enhance the&#10;performance of large language models within the constraints of limited&#10;computational resources, has become paramount. This paper details our solution&#10;for the BetterMixture challenge, which focuses on the fine-tuning data mixing&#10;for large language models. Our approach, which secured third place,&#10;incorporates data deduplication, low-level and high-level quality filtering,&#10;and diversity selection. The foundation of our solution is Ke-Data-Juicer, an&#10;extension of Data-Juicer, demonstrating its robust capabilities in handling and&#10;optimizing data for large language models.&#10;" />
          <attvalue for="2" value="&#10;The emergence of large-scale language models such as ChatGPT\cite{chatgpt} has transformed natural language processing. Meanwhile, the rapid growth of Chinese open-source large language models, including ChatGLM\cite{glm130b}, Baichuan\cite{yang2023baichuan}, Qwen\cite{qwen}, and BELLE\cite{BELLE}, contributing positively to the field's evolution.&#10;&#10;The swift development of Large Language Models (LLMs) has highlighted the critical need for vast quantities of high-quality data. &#10;In the response, BetterMixture emerges as a data-centric challenge that tests the analysis and combination capabilities of fine-tuning data for LLMs, bridging the gap between data needs and model optimization.&#10;&#10;To tackle the challenge, we utilized our Ke-Data-Juicer system, an advancement of Data-Juicer.&#10;Data-Juicer\cite{chen2024datajuicer} is a comprehensive one-stop data processing system for Large Language Models. It is capable of efficiently generating a variety of data recipes, exploring numerous combinations for creating data mixtures, and assessing their impact on model performance. &#10;Ke-Data-Juicer builds upon this foundation by enhancing high-level quality filtering and diversity selection capabilities.&#10;&#10;Building on Ke-Data-Juicer, we applied standard filtering techniques, including text length, language identification, and specific word filtering, referred to as low-level quality filtering.&#10;&#10;To enhance data quality filtering, we introduced high-level quality filtering, with a LLM serving as a trainable data selector. This process evaluates and assigns scores to each sample of instruction fine-tuning data.&#10;Specifically, we introduced Perplexity (PPL) calculated by the LLM to quantify the difficulty of instructions.&#10;Instruction Following Difficulty (IFD) \cite{li2023quantity} also introduced to assess the challenge of responding to specific instructions. &#10;Furthermore, we introduced the IFD-Vote method, which utilizes multiple LLMs to refine quality assessment based on their collective scores.&#10;&#10;In addition to quality, diversity is crucial. We employ the k-center-greedy algorithm to enhance the diversity of the selected data mixture&#10;&#10;In summary, there are three main contributions of this paper:&#10;\begin{itemize}&#10;\item &#10;We proposed a complete solution for the BetterMixture challenge, securing third place in the competition.&#10;\item&#10;We introduced high-level quality filtering methods based on LLMs, including LLM perplexity filtering and LLM Instruction-Following Difficulty (IFD) filtering techniques.&#10;\item&#10;We introduced the IFD-Vote method, leveraging multiple LLMs, to select high-quality instruction data.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computational Resource Management, Computer Science, Large Language Models, Linguistics, Data Science, Data Optimization Techniques, Mathematics" />
        </attvalues>
      </node>
      <node id="2309.02033" label="2309.02033">
        <attvalues>
          <attvalue for="0" value="Data-Juicer: A One-Stop Data Processing System for Large Language Models" />
          <attvalue for="1" value="  The immense evolution in Large Language Models (LLMs) has underscored the&#10;importance of massive, heterogeneous, and high-quality data. A data recipe is a&#10;mixture of data from different sources for training LLMs, which plays a vital&#10;role in LLMs' performance. Existing open-source tools for LLM data processing&#10;are mostly tailored for specific data recipes. To continuously uncover the&#10;potential of LLMs, incorporate data from new sources, and improve LLMs'&#10;performance, we build a new system named Data-Juicer, with which we can&#10;efficiently generate diverse data recipes, explore different possibilities in&#10;forming data mixtures, and evaluate their effects on model performance.&#10;Different from traditional data-analytics pipelines, Data-Juicer faces some&#10;unique challenges. Firstly, the possible data sources for forming data recipes&#10;are truly heterogeneous and massive with various qualities. Secondly, it is&#10;extremely expensive to precisely evaluate data recipes' impact on LLMs'&#10;performance. Thirdly, the end users of Data-Juicer, model developers, need&#10;sufficient flexibility to configure and evaluate different data recipes.&#10;  Data-Juicer features a fine-grained abstraction of pipelines for constructing&#10;data recipes, with over 50 built-in operators for easy composition and&#10;extension. By incorporating visualization and auto-evaluation capabilities,&#10;Data-Juicer enables a timely feedback loop for both LLM pre-training and&#10;fine-tuning. Further, Data-Juicer is optimized and integrated with ecosystems&#10;for LLM training, evaluation, and distributed computing. The data recipes&#10;derived with Data-Juicer gain notable improvements on state-of-the-art LLMs, by&#10;up to 7.45% increase in averaged score across 16 LLM benchmarks and 17.5%&#10;higher win rate in pair-wise GPT-4 evaluations. Our system, data recipes, and&#10;tutorials are released, calling for broader data-centric research on training&#10;and understanding LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.05708" label="1909.05708">
        <attvalues>
          <attvalue for="0" value="Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary&#10;  Induction" />
          <attvalue for="1" value="  The task of bilingual dictionary induction (BDI) is commonly used for&#10;intrinsic evaluation of cross-lingual word embeddings. The largest dataset for&#10;BDI was generated automatically, so its quality is dubious. We study the&#10;composition and quality of the test sets for five diverse languages from this&#10;dataset, with concerning findings: (1) a quarter of the data consists of proper&#10;nouns, which can be hardly indicative of BDI performance, and (2) there are&#10;pervasive gaps in the gold-standard targets. These issues appear to affect the&#10;ranking between cross-lingual embedding systems on individual languages, and&#10;the overall degree to which the systems differ in performance. With proper&#10;nouns removed from the data, the margin between the top two systems included in&#10;the study grows from 3.4% to 17.2%. Manual verification of the predictions, on&#10;the other hand, reveals that gaps in the gold standard targets artificially&#10;inflate the margin between the two systems on English to Bulgarian BDI from&#10;0.1% to 6.7%. We thus suggest that future research either avoids drawing&#10;conclusions from quantitative results on this BDI dataset, or accompanies such&#10;evaluation with rigorous error analysis.&#10;" />
          <attvalue for="2" value="&#10;Bilingual dictionary induction (BDI) refers to retrieving translations of individual words. &#10;The task has been widely used for intrinsic evaluation of cross-lingual embedding algorithms, which aim to map two languages into the same embedding space, for transfer learning purposes \cite{C12-1089}. Recently, \cite{Glavas19} reported limited evidence in support of this practice---they found that cross-lingual embeddings optimized for a BDI evaluation metric were not necessarily better on downstream tasks. &#10;Here, we study BDI evaluation in itself, as has been done for other evaluation methods in the past (cf. \cite{faruqui2016problems}'s work on word similarity), with concerning findings about its reliability. &#10;&#10;A massive dataset of 110 bilingual dictionaries, known as the MUSE dataset, was introduced in early 2018 along with a strong baseline \cite{Conneau2018}. Subsets of the MUSE dictionaries have been used for model comparison in the evaluation of numerous cross-lingual embedding systems developed since (cf. \cite{grave2018unsupervised,jawanpuria2019, Hoshen2018AnIC, hoshen2018non, DBLP:journals/corr/abs-1809-02306,joulin2018rcsls}). Even though the field has been very active, progress has been incremental for most language pairs.&#10;Moreover, there have been very few attempts at a linguistically-informed error analysis of BDI performance as measured on MUSE (cf. \cite{kementchedjhieva2018generalizing}). This is problematic for two reasons: on one hand, most systems greatly vary in their approach and architecture, so it is difficult to identify the source of the reported performance gains; on the other hand, the MUSE dataset was compiled automatically, with no manual post-processing to clean up noise, so the real impact of the performance gains is unclear.&#10;&#10;In this work, we study the composition and quality of the MUSE data for five diverse languages: German, Danish, Bulgarian, Arabic and Hindi. A manual part-of-speech annotation of the test sets for these languages &#10;reveals a strikingly high number of proper nouns. We refer to linguistic literature to argue that proper nouns,&#10;having no lexical meaning but rather just a referential function, cannot reliably be used in the evaluation of word-level translation systems.&#10;We find that excluding proper noun pairs from the test dictionaries for the aforementioned languages affects the ranking and degree of performance gaps between five of the most influential recent systems for BDI.&#10;&#10;With a new, more reliable ranking at hand, we perform qualitative analysis on the performance gap between the best and second best systems for Bulgarian. This reveals another major issue with the data: limited coverage of morphological variants for the target words. Through manual verification of the models' predictions, we find that the gap in performance between the two systems is far smaller than previously perceived. &#10;&#10;The uncovered issues of high noise levels (proper nouns) and limited coverage (missing gold standard targets) clearly have a crucial impact on BDI results obtained on the MUSE dataset, and need to be addressed. Filtering out proper nouns could be achieved automatically, by checking against gazetteers of named entities. We find that an automatic procedure for the filling of missing targets, however, yields only minor improvements. We thus urge researchers to be cautious when reporting quantitative results on MUSE, and to account for the problems presented here through manual verification and analysis of the results. As an alternative, we point them to morphologically complete BDI resources, built bottom-up \cite{czarnowska2019dont}. We share our part-of-speech annotations, such that future work can use this resource for analysis purposes. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Computational Linguistics, Evaluation Methodology, Natural Language Processing, Statistics, Bilingual Dictionary Induction, Cross-Lingual Embeddings" />
        </attvalues>
      </node>
      <node id="1605.02276" label="1605.02276">
        <attvalues>
          <attvalue for="0" value="Problems With Evaluation of Word Embeddings Using Word Similarity Tasks" />
          <attvalue for="1" value="  Lacking standardized extrinsic evaluation methods for vector representations&#10;of words, the NLP community has relied heavily on word similarity tasks as a&#10;proxy for intrinsic evaluation of word vectors. Word similarity evaluation,&#10;which correlates the distance between vectors and human judgments of semantic&#10;similarity is attractive, because it is computationally inexpensive and fast.&#10;In this paper we present several problems associated with the evaluation of&#10;word vectors on word similarity datasets, and summarize existing solutions. Our&#10;study suggests that the use of word similarity tasks for evaluation of word&#10;vectors is not sustainable and calls for further research on evaluation&#10;methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.11222" label="1805.11222">
        <attvalues>
          <attvalue for="0" value="Unsupervised Alignment of Embeddings with Wasserstein Procrustes" />
          <attvalue for="1" value="  We consider the task of aligning two sets of points in high dimension, which&#10;has many applications in natural language processing and computer vision. As an&#10;example, it was recently shown that it is possible to infer a bilingual&#10;lexicon, without supervised data, by aligning word embeddings trained on&#10;monolingual data. These recent advances are based on adversarial training to&#10;learn the mapping between the two embeddings. In this paper, we propose to use&#10;an alternative formulation, based on the joint estimation of an orthogonal&#10;matrix and a permutation matrix. While this problem is not convex, we propose&#10;to initialize our optimization algorithm by using a convex relaxation,&#10;traditionally considered for the graph isomorphism problem. We propose a&#10;stochastic algorithm to minimize our cost function on large scale problems.&#10;Finally, we evaluate our method on the problem of unsupervised word&#10;translation, by aligning word embeddings trained on monolingual data. On this&#10;task, our method obtains state of the art results, while requiring less&#10;computational resources than competing approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.08773" label="1808.08773">
        <attvalues>
          <attvalue for="0" value="Learning Multilingual Word Embeddings in Latent Metric Space: A&#10;  Geometric Approach" />
          <attvalue for="1" value="  We propose a novel geometric approach for learning bilingual mappings given&#10;monolingual embeddings and a bilingual dictionary. Our approach decouples&#10;learning the transformation from the source language to the target language&#10;into (a) learning rotations for language-specific embeddings to align them to a&#10;common space, and (b) learning a similarity metric in the common space to model&#10;similarities between the embeddings. We model the bilingual mapping problem as&#10;an optimization problem on smooth Riemannian manifolds. We show that our&#10;approach outperforms previous approaches on the bilingual lexicon induction and&#10;cross-lingual word similarity tasks. We also generalize our framework to&#10;represent multiple languages in a common latent space. In particular, the&#10;latent space representations for several languages are learned jointly, given&#10;bilingual dictionaries for multiple language pairs. We illustrate the&#10;effectiveness of joint learning for multiple languages in zero-shot word&#10;translation setting. Our implementation is available at&#10;https://github.com/anoopkunchukuttan/geomm .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.07745" label="1804.07745">
        <attvalues>
          <attvalue for="0" value="Loss in Translation: Learning Bilingual Word Mapping with a Retrieval&#10;  Criterion" />
          <attvalue for="1" value="  Continuous word representations learned separately on distinct languages can&#10;be aligned so that their words become comparable in a common space. Existing&#10;works typically solve a least-square regression problem to learn a rotation&#10;aligning a small bilingual lexicon, and use a retrieval criterion for&#10;inference. In this paper, we propose an unified formulation that directly&#10;optimizes a retrieval criterion in an end-to-end fashion. Our experiments on&#10;standard benchmarks show that our approach outperforms the state of the art on&#10;word translation, with the biggest improvements observed for distant language&#10;pairs such as English-Chinese.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.00064" label="1809.00064">
        <attvalues>
          <attvalue for="0" value="Generalizing Procrustes Analysis for Better Bilingual Dictionary&#10;  Induction" />
          <attvalue for="1" value="  Most recent approaches to bilingual dictionary induction find a linear&#10;alignment between the word vector spaces of two languages. We show that&#10;projecting the two languages onto a third, latent space, rather than directly&#10;onto each other, while equivalent in terms of expressivity, makes it easier to&#10;learn approximate alignments. Our modified approach also allows for supporting&#10;languages to be included in the alignment process, to obtain an even better&#10;performance in low resource settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Bilingual embeddings Many diverse cross-lingual word embedding models have been proposed \cite{Ruder2018}. The most popular kind learns a linear transformation from source to target language space \cite{Mikolov2013e}. In most recent work, this mapping is constrained to be orthogonal and solved using Procrustes Analysis \cite{Xing2015,Artetxe2017,Artetxe2018,Conneau2018,Lu2015a}. The approach most similar to ours, \cite{Faruqui2014}, uses canonical correlation analysis (CCA) to project both source and target language spaces into a third, joint space. In this setup, similarly to GPA, the third space is iteratively updated, such that at timestep $t$, it is a product of the two language spaces as transformed by the mapping learned at timestep $t-1$. The objective that drives the updates of the mapping matrices is to maximize the correlation between the projected embeddings of translational equivalents (where the latter are taken from a gold-standard seed dictionary). In their analysis of the transformed embedding spaces, \cite{Faruqui2014} focus on the improved quality of monolingual embedding spaces themselves and do not perform evaluation of the task of BDI. They find that the transformed monolingual spaces better encode the difference between synonyms and antonyms: in the original monolingual English space, synonyms and antonyms of beautiful are all mapped close to each other in a mixed fashion; in the transformed space the synonyms of beautiful are mapped in a cluster around the query word and its antonyms are mapped in a separate cluster. This finding is in line with our observation that GPA-learned alignments are more precise in distinguishing between synonyms and antonyms.&#10;&#10;Multilingual embeddings Several approaches extend existing methods to space alignments between more than two languages \cite{Ammar2016a,Ruder2018}. \cite{Smith2017} project all vocabularies into the English space. In some cases, multilingual training has been shown to lead to improvements over bilingually trained embedding spaces \cite{Vulic2017}, similar to our findings.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1909.02855" label="1909.02855">
        <attvalues>
          <attvalue for="0" value="Don't Forget the Long Tail! A Comprehensive Analysis of Morphological&#10;  Generalization in Bilingual Lexicon Induction" />
          <attvalue for="1" value="  Human translators routinely have to translate rare inflections of words - due&#10;to the Zipfian distribution of words in a language. When translating from&#10;Spanish, a good translator would have no problem identifying the proper&#10;translation of a statistically rare inflection such as habl\'aramos. Note the&#10;lexeme itself, hablar, is relatively common. In this work, we investigate&#10;whether state-of-the-art bilingual lexicon inducers are capable of learning&#10;this kind of generalization. We introduce 40 morphologically complete&#10;dictionaries in 10 languages and evaluate three of the state-of-the-art models&#10;on the task of translation of less frequent morphological forms. We demonstrate&#10;that the performance of state-of-the-art models drops considerably when&#10;evaluated on infrequent morphological inflections and then show that adding a&#10;simple morphological constraint at training time improves the performance,&#10;proving that the bilingual lexicon inducers can benefit from better encoding of&#10;morphology.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.02201" label="2102.02201">
        <attvalues>
          <attvalue for="0" value="When Can Models Learn From Explanations? A Formal Framework for&#10;  Understanding the Roles of Explanation Data" />
          <attvalue for="1" value="  Many methods now exist for conditioning model outputs on task instructions,&#10;retrieved documents, and user-provided explanations and feedback. Rather than&#10;relying solely on examples of task inputs and outputs, these approaches use&#10;valuable additional data for improving model correctness and aligning learned&#10;models with human priors. Meanwhile, a growing body of evidence suggests that&#10;some language models can (1) store a large amount of knowledge in their&#10;parameters, and (2) perform inference over tasks in textual inputs at test&#10;time. These results raise the possibility that, for some tasks, humans cannot&#10;explain to a model any more about the task than it already knows or could infer&#10;on its own. In this paper, we study the circumstances under which explanations&#10;of individual data points can (or cannot) improve modeling performance. In&#10;order to carefully control important properties of the data and explanations,&#10;we introduce a synthetic dataset for experiments, and we also make use of three&#10;existing datasets with explanations: e-SNLI, TACRED, and SemEval. We first give&#10;a formal framework for the available modeling approaches, in which explanation&#10;data can be used as model inputs, as targets, or as a prior. After arguing that&#10;the most promising role for explanation data is as model inputs, we propose to&#10;use a retrieval-based method and show that it solves our synthetic task with&#10;accuracies upwards of 95%, while baselines without explanation data achieve&#10;below 65% accuracy. We then identify properties of datasets for which&#10;retrieval-based modeling fails. With the three existing datasets, we find no&#10;improvements from explanation retrieval. Drawing on findings from our synthetic&#10;task, we suggest that at least one of six preconditions for successful modeling&#10;fails to hold with these datasets. Our code is publicly available at&#10;https://github.com/peterbhase/ExplanationRoles&#10;" />
          <attvalue for="2" value="&#10;&#10;To provide signal for learning, traditional supervised learning algorithms use labels consisting of class IDs or a number in regression settings. Yet training models with data in this form provides the minimum possible supervision for learning a task. Consider how deeply this style of learning contrasts with the way a person can learn a task by getting verbal explanations from someone helping them in addition to just the error signal from their performance. &#10;Access to such feedback can accelerate learning, resulting in less error-prone behavior, while also aligning the learned behavior with the teacher's prior on what behaviors are good. Since this sort of training should yield efficient and safe outcomes, the contrast between machine and human learning points to natural question: How can we incorporate natural language explanations into learning algorithms?&#10;&#10;A long line of past work has sought to use explanations, rationales, instructions, and other similar data to improve models. Proposed methods use explanations to constrain or regularize the learned model \cite{zaidan_using_2007, small2011constrained, ba2015predicting, zhang_rationale-augmented_2016, Srivastava2018LearningCF, andreas2018learning, liang2020alice}, to automatically label data for data augmentation \cite{hancock_training_2018, wang_does_2019, awasthi2020learning}, as additional supervision \cite{narang_wt5?!_2020, hase2020leakage, pruthi2020} or intermediate structured variables \cite{camburu_e-snli:_2018, rajani_explain_2019, wiegreffe2020}, and simply as model inputs \cite{rupprecht2018guide, Co-Reyes2019Guiding, zhou2020towards}.&#10;&#10;What is surprising about the sheer breadth of approaches in these works is that they all aim to incorporate essentially the same kinds of information. We can describe each of these approaches as trying to augment models with (1) information not available through their inputs or in their parametric knowledge, or (2) a further specification of the task that is informative about which models are good. Improving models in this manner is a natural goal of approaches using explanations, since one purpose of an explanation is to communicate a mental model \cite{doshi-velez_towards_2017, miller2019explanation}. But how do explanations get used as additional targets, as inputs, as regularizers, as structured variables, and as rules for automatic data labeling? Even under a general notion of what an ``explanation&quot; is, e.g. the answer to some why-question \cite{miller2019explanation}, this kind of data plays an impressive number of roles. &#10;&#10;Yet there are tasks where explanations do not fulfill these roles effectively, as improvements in performance prove elusive even when thousands of explanations are gathered \cite{narang_wt5?!_2020, hase2020leakage}. In fact, there is reason to think that for some tasks models will not need additional information or further task specification of the kind explanations provide. This is because large language models now (1) store a great amount of knowledge in their parameters \cite{roberts-etal-2020-much, lewis2020retrieval}, and (2) infer tasks at test time from the input itself \cite{radford_language_2019, gpt3, weller-etal-2020-learning}. So in some situations we may not be able to explain to a model more about a task or a data point than it already knows or could infer on its own. What remains unclear, however, is the set of conditions which distinguish situations where explanations will be helpful from those where they will not be helpful in practice or cannot be in principle. &#10;&#10;In this paper, we (1) give an argument for the role of explanations in modeling that helps us understand how explanations have been used in such distinct ways and points us toward suitable methods, and (2) we experimentally study the conditions under which explanations are or are not helpful to models, using a specially designed synthetic task and three existing datasets with explanations given for individual data points. &#10;The modeling approach we ultimately propose is to perform retrieval over past explanations and provide them as inputs to a model at prediction time (see Sec.~\ref{sec:our_model}), which is the approach we reach following our broader argument in Sec.~\ref{sec:formalizing_the_role}. &#10;Our synthetic task (described in Sec.~\ref{sec:synthetic}) is designed to have analogous properties to existing real (i.e., human-curated) data, and it is especially useful here as it enables us to test a number of hypotheses that we could not test with existing datasets.&#10;&#10;Using RoBERTa as a representative large language model \cite{liu_roberta_2019} and Sentence-BERT as a retrieval model \cite{reimers-gurevych-2019-sentence},&#10;we investigate a number of primary research questions, each given with brief context: &#10;\begin{enumerate}[itemsep=0pt, wide=0pt, leftmargin=*, after=\strut]&#10; \item RQ1. Since some models can infer tasks from sequences at test time, providing task information may not be helpful. When can models solve our synthetic problem by inferring each sequence's task, and when must they be given the task information?&#10; \item RQ2. Explanations seen in the past may help with predicting future data points. Can retrieval of past explanations enable a model to solve our task?&#10; \item RQ3. Useful information might be distributed over several explanations. Can models aggregate information across explanations for better prediction?&#10; \item RQ4. We can let pretrained models combine explanations by giving them as textual input, or we can pool extracted feature representations. What is the best way to compute explanation representations for prediction?&#10; \item RQ5. Good explanations pertain to the data point they are given for, but what makes an explanation relevant across data points? What enables a retrieval model to find relevant explanations for a new data point?&#10; \item RQ6. One intuitive use case for explanations is to encourage models to rely on causal features rather than spurious correlations. Can explanations help models learn to use strong features rather than weak ones?&#10; \item RQ7. Here, the training signal for a retrieval model depends on how the classifier uses the explanations the initial retrieval model can provide.&#10; How does the co-dependence between classifier and retrieval model influence the viability of joint training? &#10; \item RQ8. After identifying a set of conditions which determine whether retrieval-based modeling can succeed in our synthetic task, we ask: does retrieval of explanations improve model performance on existing datasets?&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Explanation-Based Modeling, Computer Science, Machine Learning, Language Model Conditioning, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Human-Model Knowledge Alignment" />
        </attvalues>
      </node>
      <node id="1805.03818" label="1805.03818">
        <attvalues>
          <attvalue for="0" value="Training Classifiers with Natural Language Explanations" />
          <attvalue for="1" value="  Training accurate classifiers requires many labels, but each label provides&#10;only limited information (one bit for binary classification). In this work, we&#10;propose BabbleLabble, a framework for training classifiers in which an&#10;annotator provides a natural language explanation for each labeling decision. A&#10;semantic parser converts these explanations into programmatic labeling&#10;functions that generate noisy labels for an arbitrary amount of unlabeled data,&#10;which is used to train a classifier. On three relation extraction tasks, we&#10;find that users are able to train classifiers with comparable F1 scores from&#10;5-100$\times$ faster by providing explanations instead of just labels.&#10;Furthermore, given the inherent imperfection of labeling functions, we find&#10;that a simple rule-based semantic parser suffices.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14546" label="2004.14546">
        <attvalues>
          <attvalue for="0" value="WT5?! Training Text-to-Text Models to Explain their Predictions" />
          <attvalue for="1" value="  Neural networks have recently achieved human-level performance on various&#10;challenging natural language processing (NLP) tasks, but it is notoriously&#10;difficult to understand why a neural network produced a particular prediction.&#10;In this paper, we leverage the text-to-text framework proposed by Raffel et&#10;al.(2019) to train language models to output a natural text explanation&#10;alongside their prediction. Crucially, this requires no modifications to the&#10;loss function or training and decoding procedures -- we simply train the model&#10;to output the explanation after generating the (natural text) prediction. We&#10;show that this approach not only obtains state-of-the-art results on&#10;explainability benchmarks, but also permits learning from a limited set of&#10;labeled explanations and transferring rationalization abilities across&#10;datasets. To facilitate reproducibility and future work, we release our code&#10;use to train the models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Measuring and improving the interpretability of neural networks is a heavily-studied area of research; a comprehensive overview is outside of the scope of this work.&#10;Instead, we refer the interested reader to the surveys provided by \cite{doshi2017towards,molnar2019interpretable,guidotti2018survey}.&#10;Most work on interpretability methods focuses on models for computer vision applications (e.g.\ \cite{xu2015show, zhang2018visual}), whereas the interpretability of NLP models is apparently less studied.&#10;A notable exception is the fact that attention-based neural networks \cite{bahdanau2014neural} provide some means of interpretability ``for free'' by examining the weight assigned by the neural network to different regions in the input \cite{graves2013generating,raffel2017online,huang2018music}, but this introspection method has been shown to be unreliable \cite{jain2019attention,serrano2019attention,pruthi2019learning}.&#10;There has separately been significant work on better understanding the behavior NLP models, for example by crafting inputs that cause a misclassification \cite{jia2017adversarial,nie2019adversarial} or diagnosing why they sometimes generate nonsensical text \cite{lee2018hallucinations,belinkov2017synthetic}.&#10;&#10;An early investigation into explanations for NLP datasets was performed by \cite{zaidan2008modeling}, who introduced the idea of annotating spans of the input which support the label.&#10;This produced the ``Movie Reviews'' dataset that we consider in our experiments.&#10;The general approach of extractive explanation was recently surveyed and advocated for by \cite{deyoung2019eraser}, who proposed the ERASER benchmark comprising various datasets.&#10;As discussed in \ref{sec:extractive}, our approach is strictly more general in that it also allows us to consider generating abstractive explanations.&#10;&#10;\cite{camburu2018snli} have the most philosophically similar perspective to ours.&#10;They consider the generation of abstractive explanations by creating the e-SNLI dataset, which we consider in our experiments. e-SNLI is a variant of the Stanford Natural Language Inference (SNLI) dataset \cite{bowman2015large} that adds human-annotated explanations for all examples in the training, validation, and test sets.&#10;To generate explanations, \cite{camburu2018snli} propose model architectures that generally consist of separate components for classification and explanation.&#10;They consider various tasks, including conditioning the prediction on the explanation and vice versa, as well as producing sentence embeddings.&#10;Most related to this work, they also consider the task of learning to explain with e-SNLI but generating explanations for out-of-domain examples from other natural language inference tasks.&#10;The primary differences between \cite{camburu2018snli} and this work are that our approach requires no special model architecture and that we take advantage of a pre-trained model that is already quite capable of generating natural text.&#10;These differences not only make our method simpler but also produce substantially better performance on the e-SNLI task.&#10;&#10;\cite{rajani2019explain} also consider abstractive explanations.&#10;They introduce the CoS-E dataset, which comprises examples from the Commonsense Question Answering (CQA) dataset that have been annotated with abstractive explanations.&#10;However, their focus is mainly on using explanations to improve a model's predictions, and as such they propose first training a model to generate explanations and then training a classifier to produce a prediction based on the original example and the generated explanation.&#10;Interestingly, this provided a substantial performance boost on the CQA dataset.&#10;They include minimal analysis or evaluation of the generated explanations, though they do show (through a few non-cherrypicked examples) that their model can generate explanations for datasets it was not trained on.&#10;The primary focus of our work is on generating useful explanations, so we do not experiment with feeding explanations into a model to improve its predictions.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1812.01193" label="1812.01193">
        <attvalues>
          <attvalue for="0" value="e-SNLI: Natural Language Inference with Natural Language Explanations" />
          <attvalue for="1" value="  In order for machine learning to garner widespread public adoption, models&#10;must be able to provide interpretable and robust explanations for their&#10;decisions, as well as learn from human-provided explanations at train time. In&#10;this work, we extend the Stanford Natural Language Inference dataset with an&#10;additional layer of human-annotated natural language explanations of the&#10;entailment relations. We further implement models that incorporate these&#10;explanations into their training process and output them at test time. We show&#10;how our corpus of explanations, which we call e-SNLI, can be used for various&#10;goals, such as obtaining full sentence justifications of a model's decisions,&#10;improving universal sentence representations and transferring to out-of-domain&#10;NLI datasets. Our dataset thus opens up a range of research directions for&#10;using natural language explanations, both for improving models and for&#10;asserting their trust.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.11544" label="1803.11544">
        <attvalues>
          <attvalue for="0" value="Guide Me: Interacting with Deep Networks" />
          <attvalue for="1" value="  Interaction and collaboration between humans and intelligent machines has&#10;become increasingly important as machine learning methods move into real-world&#10;applications that involve end users. While much prior work lies at the&#10;intersection of natural language and vision, such as image captioning or image&#10;generation from text descriptions, less focus has been placed on the use of&#10;language to guide or improve the performance of a learned visual processing&#10;algorithm. In this paper, we explore methods to flexibly guide a trained&#10;convolutional neural network through user input to improve its performance&#10;during inference. We do so by inserting a layer that acts as a spatio-semantic&#10;guide into the network. This guide is trained to modify the network's&#10;activations, either directly via an energy minimization scheme or indirectly&#10;through a recurrent model that translates human language queries to interaction&#10;weights. Learning the verbal interaction is fully automatic and does not&#10;require manual text annotations. We evaluate the method on two datasets,&#10;showing that guiding a pre-trained network can improve performance, and provide&#10;extensive insights into the interaction between the guide and the CNN.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1702.08608" label="1702.08608">
        <attvalues>
          <attvalue for="0" value="Towards A Rigorous Science of Interpretable Machine Learning" />
          <attvalue for="1" value="  As machine learning systems become ubiquitous, there has been a surge of&#10;interest in interpretable machine learning: systems that provide explanation&#10;for their outputs. These explanations are often used to qualitatively assess&#10;other criteria such as safety or non-discrimination. However, despite the&#10;interest in interpretability, there is very little consensus on what&#10;interpretable machine learning is and how it should be measured. In this&#10;position paper, we first define interpretability and describe when&#10;interpretability is needed (and when it is not). Next, we suggest a taxonomy&#10;for rigorous evaluation and expose open questions towards a more rigorous&#10;science of interpretable machine learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.07269" label="1706.07269">
        <attvalues>
          <attvalue for="0" value="Explanation in Artificial Intelligence: Insights from the Social&#10;  Sciences" />
          <attvalue for="1" value="  There has been a recent resurgence in the area of explainable artificial&#10;intelligence as researchers and practitioners seek to make their algorithms&#10;more understandable. Much of this research is focused on explicitly explaining&#10;decisions or actions to a human observer, and it should not be controversial to&#10;say that looking at how humans explain to each other can serve as a useful&#10;starting point for explanation in artificial intelligence. However, it is fair&#10;to say that most work in explainable artificial intelligence uses only the&#10;researchers' intuition of what constitutes a `good' explanation. There exists&#10;vast and valuable bodies of research in philosophy, psychology, and cognitive&#10;science of how people define, generate, select, evaluate, and present&#10;explanations, which argues that people employ certain cognitive biases and&#10;social expectations towards the explanation process. This paper argues that the&#10;field of explainable artificial intelligence should build on this existing&#10;research, and reviews relevant papers from philosophy, cognitive&#10;psychology/science, and social psychology, which study these topics. It draws&#10;out some important findings, and discusses ways that these can be infused with&#10;work on explainable artificial intelligence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.16216" label="2407.16216">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO,&#10;  DPO and More" />
          <attvalue for="1" value="  With advancements in self-supervised learning, the availability of trillions&#10;tokens in a pre-training corpus, instruction fine-tuning, and the development&#10;of large Transformers with billions of parameters, large language models (LLMs)&#10;are now capable of generating factual and coherent responses to human queries.&#10;However, the mixed quality of training data can lead to the generation of&#10;undesired responses, presenting a significant challenge. Over the past two&#10;years, various methods have been proposed from different perspectives to&#10;enhance LLMs, particularly in aligning them with human expectation. Despite&#10;these efforts, there has not been a comprehensive survey paper that categorizes&#10;and details these approaches. In this work, we aim to address this gap by&#10;categorizing these papers into distinct topics and providing detailed&#10;explanations of each alignment method, thereby helping readers gain a thorough&#10;understanding of the current state of the field.&#10;" />
          <attvalue for="2" value="&#10;Over the past decades, the pretraining of LLMs through self-supervised learning \cite{devlin2019bertpretrainingdeepbidirectional} has seen significant advancements. These improvements have been driven by the development of larger decoder-only Transformers, the utilization of trillions of tokens, and the parallelization of computations across multiple GPUs. Following the pretraining phase, instruction tuning was employed to guide LLMs in responding to human queries. Despite these advancements, a critical issue remains unresolved: LLMs can generate undesired responses, such as providing instructions on how to commit illegal activities. To mitigate this risk, it is essential to align LLMs with human values.&#10;&#10;Reinforcement Learning from Human Feedback (RLHF) \cite{ouyang2022training,bai2022training} has emerged as a groundbreaking technique for aligning LLMs. This approach has led to the development of powerful models such as GPT-4 \cite{openai2024gpt4}, Claude \cite{anthropic2024claude}, and Gemini \cite{team2023gemini}. Following the introduction of RLHF, numerous studies have explored various approaches to further align LLMs. However, there has not yet been a comprehensive review of methods for aligning LLMs with human preferences. This paper aims to fill that gap by categorically reviewing existing literature and providing detailed analyses of individual papers.&#10;&#10;In this paper, we have structured our review into four main topics: 1. Reward Model; 2. Feedback; 3. Reinforcement Learning (RL); and 4. Optimization. Each topic was further divided into subtopics as shown in Figure. \ref{fig:your_label}. For the Reward Model, the subtopics were: 1. Explicit Reward Model vs. Implicit Reward Model; 2. Pointwise Reward Model vs. Preference Model; 3. Response-Level Reward vs. Token-Level Reward and 4. Negative Preference Optimization. Regarding Feedback, the subtopics included: 1. Preference Feedback vs. Binary Feedback; 2. Pairwise Feedback vs. Listwise Feedback; and 3. Human Feedback vs. AI Feedback. In the RL section, the subtopics were: 1. Reference-Based RL vs. Reference-Free RL; 2. Length-Control RL; 3. Different Divergences in RL and 4. On-Policy RL vs. Off-Policy RL. For Optimization, the subtopics were: 1. Online/Iterative Preference Optimization vs. Offline/Non-iterative Preference Optimization; and 3. Separating SFT and Alignment vs. Merging SFT and Alignment. Table \ref{Table: Comparison all papers across 13 metrics} provided an analysis of all the papers reviewed in detail using these 13 evaluation metrics.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Self-Supervised Learning, Human Alignment Methods" />
        </attvalues>
      </node>
      <node id="2406.17453" label="2406.17453">
        <attvalues>
          <attvalue for="0" value="Learning to Ask Informative Questions: Enhancing LLMs with Preference&#10;  Optimization and Expected Information Gain" />
          <attvalue for="1" value="  Questions are essential tools for acquiring the necessary information to&#10;complete information-seeking tasks. However, large language models (LLMs),&#10;especially open-source models, often perform poorly in generating informative&#10;questions, as measured by expected information gain (EIG). In this paper, we&#10;propose a method to enhance the informativeness of LLM-generated questions in&#10;20-question game dialogues. We sample multiple questions from the same model&#10;(LLAMA 2-CHAT 7B) for each game and create pairs of low-EIG and high-EIG&#10;questions to apply a Direct Preference Optimization (DPO) algorithm. Our&#10;results show that this method produces more effective questions (in terms of&#10;EIG), even in domains different from those used to train the DPO model.&#10;" />
          <attvalue for="2" value="&#10;&#10;Questions in language serve as requests for information \cite{Searle1969SpeechActs}. A speaker lacks information in their knowledge state and asks questions to gain this information. This process of acquiring information through questioning is essential for children to learn about the world \cite{Chouinard2007} and for adults to solve complex problems \cite{Geva2021Aristotle}. &#10;Questions, however, vary in their level of informativeness, with some questions being more informative and efficient in reaching the problem's solution \cite{grand:LIPS}.&#10;&#10;Cognitive Science provides two interesting tools to study questions' informativeness: the 20 Questions Game as a test bed, and the Expected Information Gain (EIG) as a quantifying measure \cite{RUGGERI2015203}. The 20 Questions game consists of one player asking yes/no questions to identify the item the other player has in mind, from a pool of possible items. In this context, the EIG \cite{shannon1948mathematical} measures questions' informativeness as the (expected) entropy reduction caused by a certain question in the space of possible items. Most informative polar questions partition the possible items into two same-size clusters: items in one cluster are expected to receive a positive answer, and items in the other cluster are expected to receive a negative one.&#10;&#10;In recent years, Large Language Models (LLMs) have demonstrated remarkable language and reasoning capabilities \cite{kojima2022large, huang-chang-2023-towards}. The ability to ask informative and effective questions is crucial for employing these systems as successful user assistants on a large scale. Although LLMs have demonstrated their ability to play the 20 Questions game, their questions are characterised by low informativeness and limited success \cite{INLG23}. Recent studies have emerged to improve the informativeness of LLMs' questions. \cite{Apple} improve open-source LLMs via Reinforcement Learning and Behavioral Cloning from larger LLMs' data. Relying on LLMs' ability to generate diverse questions and provide reliable answers \cite{piriyakulkij2023activepreference, Testoni23}, \cite{hu:unce24} propose an inference-time re-ranking strategy based on EIG. Differently from them, we propose a training strategy leveraging EIG as a refined signal with a preference optimization algorithm. Unlike other approaches, all the steps of our method are accomplished by the same open-source model, without requiring annotation or feedback signals from external models.&#10;&#10;This study aims to improve the informativeness of questions generated by LLMs, thereby leading to more efficient agents. The 20 Questions Game serves as a testbed to illustrate the feasibility of our approach.&#10;To achieve our aim, we propose a method involving three steps: sampling multiple questions, questions' evaluation in terms of EIG, and training with preference optimization. In the first step, the model generates a set of possible questions, and it computes the EIG for each question. This set of questions, along with the corresponding EIG values, as a proxy for the questions' informativeness, is used for Direct Preference Optimization (DPO; \cite{rafailov2024direct}). Our results show that EIG is a strong training signal to improve the question-asking capabilities of current LLMs and overcome their shortcomings in asking effective questions.&#10;&#10;While these findings hold in the controlled setting of the 20 Questions Game, our approach could be extended outside this paradigm and metric to train different models’ capabilities. Our core idea of sampling, self-annotation to filter with a metric, and preference optimization could improve questions’ generations in a plethora of tasks – recommendation systems \cite{piriyakulkij2023activepreference} and image retrieval \cite{GroundedRetrieval23} for example. Other metrics, such as Expected Savings \cite{Rothe2018askgood}, could be employed to generate large preference datasets and improve LLMs' information-seeking abilities beyond our tested paradigm.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Question Generation Techniques, Information Seeking Strategies, Artificial Intelligence, Information Science, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2101.02235" label="2101.02235">
        <attvalues>
          <attvalue for="0" value="Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit&#10;  Reasoning Strategies" />
          <attvalue for="1" value="  A key limitation in current datasets for multi-hop reasoning is that the&#10;required steps for answering the question are mentioned in it explicitly. In&#10;this work, we introduce StrategyQA, a question answering (QA) benchmark where&#10;the required reasoning steps are implicit in the question, and should be&#10;inferred using a strategy. A fundamental challenge in this setup is how to&#10;elicit such creative questions from crowdsourcing workers, while covering a&#10;broad range of potential strategies. We propose a data collection procedure&#10;that combines term-based priming to inspire annotators, careful control over&#10;the annotator population, and adversarial filtering for eliminating reasoning&#10;shortcuts. Moreover, we annotate each question with (1) a decomposition into&#10;reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the&#10;answers to each step. Overall, StrategyQA includes 2,780 examples, each&#10;consisting of a strategy question, its decomposition, and evidence paragraphs.&#10;Analysis shows that questions in StrategyQA are short, topic-diverse, and cover&#10;a wide range of strategies. Empirically, we show that humans perform well (87%)&#10;on this task, while our best baseline reaches an accuracy of $\sim$66%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.19471" label="2402.19471">
        <attvalues>
          <attvalue for="0" value="Loose LIPS Sink Ships: Asking Questions in Battleship with&#10;  Language-Informed Program Sampling" />
          <attvalue for="1" value="  Questions combine our mastery of language with our remarkable facility for&#10;reasoning about uncertainty. How do people navigate vast hypothesis spaces to&#10;pose informative questions given limited cognitive resources? We study these&#10;tradeoffs in a classic grounded question-asking task based on the board game&#10;Battleship. Our language-informed program sampling (LIPS) model uses large&#10;language models (LLMs) to generate natural language questions, translate them&#10;into symbolic programs, and evaluate their expected information gain. We find&#10;that with a surprisingly modest resource budget, this simple Monte Carlo&#10;optimization strategy yields informative questions that mirror human&#10;performance across varied Battleship board scenarios. In contrast, LLM-only&#10;baselines struggle to ground questions in the board state; notably, GPT-4V&#10;provides no improvement over non-visual baselines. Our results illustrate how&#10;Bayesian models of question-asking can leverage the statistics of language to&#10;capture human priors, while highlighting some shortcomings of pure LLMs as&#10;grounded reasoners.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.01468" label="2310.01468">
        <attvalues>
          <attvalue for="0" value="Probing the Multi-turn Planning Capabilities of LLMs via 20 Question&#10;  Games" />
          <attvalue for="1" value="  Large language models (LLMs) are effective at answering questions that are&#10;clearly asked. However, when faced with ambiguous queries they can act&#10;unpredictably and produce incorrect outputs. This underscores the need for the&#10;development of intelligent agents capable of asking clarification questions to&#10;resolve ambiguities effectively. This capability requires complex&#10;understanding, state tracking, reasoning and planning over multiple&#10;conversational turns. However, directly measuring this can be challenging. In&#10;this paper, we offer a surrogate problem which assesses an LLMs's capability to&#10;deduce an entity unknown to itself, but revealed to a judge, by asking the&#10;judge a series of queries. This \textit{entity-deducing game} can serve as an&#10;evaluation framework to probe the conversational reasoning and planning&#10;capabilities of language models. We systematically evaluate various LLMs and&#10;discover significant differences in their performance on this task. We find&#10;that strong LLMs like GPT-4 outperform human players by a large margin. We&#10;further employ Behavior Cloning (BC) to examine whether a weaker model is&#10;capable of imitating a stronger model and generalizing to data or domains,&#10;using only the demonstrations from a stronger model. We finally propose to use&#10;Reinforcement Learning to enhance reasoning and planning capacity of Vicuna&#10;models through episodes of game playing, which lead to significant performance&#10;improvement. We hope that this problem offers insights into how autonomous&#10;agents could be trained to behave more intelligently in ambiguous&#10;circumstances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.08584" label="2311.08584">
        <attvalues>
          <attvalue for="0" value="Asking More Informative Questions for Grounded Retrieval" />
          <attvalue for="1" value="  When a model is trying to gather information in an interactive setting, it&#10;benefits from asking informative questions. However, in the case of a grounded&#10;multi-turn image identification task, previous studies have been constrained to&#10;polar yes/no questions, limiting how much information the model can gain in a&#10;single turn. We present an approach that formulates more informative,&#10;open-ended questions. In doing so, we discover that off-the-shelf visual&#10;question answering (VQA) models often make presupposition errors, which&#10;standard information gain question selection methods fail to account for. To&#10;address this issue, we propose a method that can incorporate presupposition&#10;handling into both question selection and belief updates. Specifically, we use&#10;a two-stage process, where the model first filters out images which are&#10;irrelevant to a given question, then updates its beliefs about which image the&#10;user intends. Through self-play and human evaluations, we show that our method&#10;is successful in asking informative open-ended questions, increasing accuracy&#10;over the past state-of-the-art by 14%, while resulting in 48% more efficient&#10;games in human evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.14115" label="2406.14115">
        <attvalues>
          <attvalue for="0" value="Take the essence and discard the dross: A Rethinking on Data Selection&#10;  for Fine-Tuning Large Language Models" />
          <attvalue for="1" value="  Data selection for fine-tuning Large Language Models (LLMs) aims to select a&#10;high-quality subset from a given candidate dataset to train a Pending Fine-tune&#10;Model (PFM) into a Selective-Enhanced Model (SEM). It can improve the model&#10;performance and accelerate the training process. Although a few surveys have&#10;investigated related works of data selection, there is a lack of comprehensive&#10;comparison between existing methods due to their various experimental settings.&#10;To address this issue, we first propose a three-stage scheme for data selection&#10;and comprehensively review existing works according to this scheme. Then, we&#10;design a unified comparing method with ratio-based efficiency indicators and&#10;ranking-based feasibility indicators to overcome the difficulty of comparing&#10;various models with diverse experimental settings. After an in-depth&#10;comparative analysis, we find that the more targeted method with data-specific&#10;and model-specific quality labels has higher efficiency, but the introduction&#10;of additional noise information should be avoided when designing selection&#10;algorithms. Finally, we summarize the trends in data selection and highlight&#10;the short-term and long-term challenges to guide future research.&#10;" />
          <attvalue for="2" value="&#10;&#10;Supervised fine-tuning (SFT) leverages small amounts of instruction-pair data to unlock large language models' instruction-following capabilities and improve generalization across various tasks \cite{radford2019language, weifinetuned, singh2023beyond, zhang2024instructiontuninglargelanguage, albalak2024surveydataselectionlanguage}. Recent research highlights that data quality is more critical than data quantity for effective fine-tuning~\cite{nakkiran2021deep, shumailov2024ai, zhou2024lima, jindal2024birbal}. As a result, several data curation techniques have been proposed, such as data selection~\cite{chen2024alpagasus, li2024quantity}, data evolution~\cite{wang2023selfinstruct, xu2023wizardlm}, and data reflection~\cite{mukherjee2023orca, yin-etal-2023-dynosaur}. Data selection, in particular, involves choosing a high-quality subset from a candidate dataset based on specific selection criteria, enhancing the model's performance while improving training efficiency by reducing the number of samples. Unlike data augmentation or polishing, it focuses on selecting inherently higher-quality samples, as shown in Figure \ref{fig:one-pic}.&#10;&#10;However, despite the rapid development of data selection methods, there is currently no unified framework for systematically guiding and comparing these methods, as experimental settings vary widely across studies. Although some surveys~\cite{bommasani2021opportunities, albalak2024surveydataselectionlanguage, wang2024survey} have reviewed data curation techniques in the contexts of pretraining, fine-tuning, and reinforcement learning, they generally provide high-level overviews and lack in-depth discussions of the fine-tuning stage. This gap makes it difficult for researchers to conduct focused, sustained studies on this crucial phase. To address this, our survey provides a fine-grained review of data selection methods for instruction fine-tuning LLMs, rethinking existing approaches, proposing a unified comparison method, and outlining key trends and challenges in the field.&#10;&#10;We begin by reviewing existing data selection techniques, organizing them within a three-stage scheme based on the key components of the data selection pipeline: feature extraction, criteria design, and selector evaluation (Figure \ref{fig:landscape}). In the feature extraction stage (Section~\ref{sec:preprocess}), we categorize methods into three types based on the form of the candidate data: human-designed features, model-oriented features, and raw text. &#10;In the criteria design stage (Section~\ref{sec:selector}), we categorize methods based on the source of the sample quality label into two groups: internal information and external information. The latter is further divided into methods obtaining criteria from model preference or sample influence. In the selector evaluation stage (Section~\ref{sec:eval}), we outline three key aspects to reliably evaluate a selector's effectiveness: candidate datasets, counterpart models, and evaluation metrics.&#10;&#10;We also introduce a unified comparison method for evaluating existing works, incorporating both ratio-based efficiency indicators and ranking-based feasibility indicators (Section \ref{sec:anal}). Specifically, we first construct a quantitative comparison of two-dimensional efficiency based on the performance improvement ratio (PIR) and selected dataset fraction (SDF), aligning them through the efficiency curve assumption, effectively addressing the challenge of comparing different methods under inconsistent configurations. Then, we consider the feasibility of the method from the perspectives of flexibility and simplicity indicators. It qualitatively ranks existing models by manually considering algorithm complexity and reproducibility (the number of training models, algorithm steps, and open-source availability), as well as their transferability and scalability (dependence on data and models). &#10;&#10;Finally, we discuss the main trends and challenges faced for data selection. We first sort out the existing works chronologically from three aspects (Candidate Dataset, Quality Measurement, and Selected Feature) to grasp the current research focus (Section \ref{sec:disc}). We then point out the most important open question (How can we design effective sample quality measurement for data selection?) with four challenges: balance the efficiency and feasibility; ensure the measurement objectivity; improve specific tasks/domains performance without compromising others and satisfy multiple goals.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Mathematics, Information Science, Model Training Optimization, Data Selection Methods" />
        </attvalues>
      </node>
      <node id="1912.02292" label="1912.02292">
        <attvalues>
          <attvalue for="0" value="Deep Double Descent: Where Bigger Models and More Data Hurt" />
          <attvalue for="1" value="  We show that a variety of modern deep learning tasks exhibit a&#10;&quot;double-descent&quot; phenomenon where, as we increase model size, performance first&#10;gets worse and then gets better. Moreover, we show that double descent occurs&#10;not just as a function of model size, but also as a function of the number of&#10;training epochs. We unify the above phenomena by defining a new complexity&#10;measure we call the effective model complexity and conjecture a generalized&#10;double descent with respect to this measure. Furthermore, our notion of model&#10;complexity allows us to identify certain regimes where increasing (even&#10;quadrupling) the number of train samples actually hurts test performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.08701" label="2307.08701">
        <attvalues>
          <attvalue for="0" value="AlpaGasus: Training A Better Alpaca with Fewer Data" />
          <attvalue for="1" value="  Large language models (LLMs) strengthen instruction-following capability&#10;through instruction-finetuning (IFT) on supervised instruction/response data.&#10;However, widely used IFT datasets (e.g., Alpaca's 52k data) surprisingly&#10;contain many low-quality instances with incorrect or irrelevant responses,&#10;which are misleading and detrimental to IFT. In this paper, we propose a simple&#10;and effective data selection strategy that automatically identifies and filters&#10;out low-quality data using a strong LLM (e.g., ChatGPT). To this end, we&#10;introduce AlpaGasus, which is finetuned on only 9k high-quality data filtered&#10;from the 52k Alpaca data. AlpaGasus significantly outperforms the original&#10;Alpaca as evaluated by GPT-4 on multiple test sets and the controlled human&#10;evaluation. Its 13B variant matches $&gt;90\%$ performance of its teacher LLM&#10;(i.e., Text-Davinci-003 generating the 52k data) on test tasks. It also&#10;provides 5.7x faster training, reducing the training time for a 7B variant from&#10;80 minutes (for Alpaca) to 14 minutes. Moreover, the experiments prove the&#10;efficacy of our method across diverse datasets, base models, and LLM filters.&#10;Overall, AlpaGasus demonstrates a novel data-centric IFT paradigm that can be&#10;generally applied to instruction-tuning data, leading to faster training and&#10;better instruction-following models. Our project page is available at:&#10;https://lichang-chen.github.io/AlpaGasus/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.02707" label="2306.02707">
        <attvalues>
          <attvalue for="0" value="Orca: Progressive Learning from Complex Explanation Traces of GPT-4" />
          <attvalue for="1" value="  Recent research has focused on enhancing the capability of smaller models&#10;through imitation learning, drawing on the outputs generated by large&#10;foundation models (LFMs). A number of issues impact the quality of these&#10;models, ranging from limited imitation signals from shallow LFM outputs; small&#10;scale homogeneous training data; and most notably a lack of rigorous evaluation&#10;resulting in overestimating the small model's capability as they tend to learn&#10;to imitate the style, but not the reasoning process of LFMs. To address these&#10;challenges, we develop Orca (We are working with our legal team to publicly&#10;release a diff of the model weights in accordance with LLaMA's release policy&#10;to be published at https://aka.ms/orca-lm), a 13-billion parameter model that&#10;learns to imitate the reasoning process of LFMs. Orca learns from rich signals&#10;from GPT-4 including explanation traces; step-by-step thought processes; and&#10;other complex instructions, guided by teacher assistance from ChatGPT. To&#10;promote this progressive learning, we tap into large-scale and diverse&#10;imitation data with judicious sampling and selection. Orca surpasses&#10;conventional state-of-the-art instruction-tuned models such as Vicuna-13B by&#10;more than 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard&#10;(BBH) and 42% on AGIEval. Moreover, Orca reaches parity with ChatGPT on the BBH&#10;benchmark and shows competitive performance (4 pts gap with optimized system&#10;message) in professional and academic examinations like the SAT, LSAT, GRE, and&#10;GMAT, both in zero-shot settings without CoT; while trailing behind GPT-4. Our&#10;research indicates that learning from step-by-step explanations, whether these&#10;are generated by humans or more advanced AI models, is a promising direction to&#10;improve model capabilities and skills.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Foundation Models (LFMs) such as ChatGPT and GPT-4~\cite{openai2023gpt4} exhibit remarkable zero-shot performances across a broad spectrum of tasks. Alongside academic benchmarks like Human Eval~\cite{chen2021codex} and Big Bench~\cite{srivastava2022imitation}, GPT-4 has also demonstrated human-level performance on various professional exams, including the bar exam, SAT, GRE, and USMLE. These advancements can be credited to the scaling of both model and dataset sizes, as well as the incorporation of a second layer of training to better align the models with user intent. This alignment is accomplished by fine-tuning the models via supervised learning on demonstrations of prompts and desired model behavior, and through reinforcement learning from human preferences~\cite{InstructGPT}. &#10;&#10;As these models continue to evolve and become more powerful, an intriguing question arises: Can we use the model itself to supervise its own behavior or that of other AI models? ~\cite{Bai2022ConstitutionalAH} have shown that by sampling output from an initial model, generating revisions, and then fine-tuning the original model based on these revised responses, model behavior can be controlled more effectively and can be made more harmless, with significantly fewer human labels. &#10;&#10;Recently, there has been an influx of studies using LFMs like ChatGPT and GPT-4 as teachers to generate large datasets, for {instruction tuning}, and to train smaller models, such as Alpaca~\cite{alpaca}, WizardLM~\cite{xu2023wizardlm} and Vicuna~\cite{vicuna-2023}. While these models can produce content that matches the style of their teachers, they often fall short in terms of the reasoning and comprehension skills displayed by the larger foundation models. &#10;&#10;Take, for example, the 13-billion parameter instruction-tuned model, Vicuna~\cite{vicuna-2023} (with LLAMA-13B~\cite{touvron2023llama} as the base), which is widely regarded as one of the best models in its family, as evidenced by its performance on leaderboards like OpenLLM and ChatArena. &#10;&#10;As illustrated in Figure~\ref{fig:gpt4-eval}, the widely-used evaluation method of using GPT-4 as the judge suggests that Vicuna retains $92\%$ of ChatGPT's quality. However, a more meticulous evaluation on reasoning benchmarks against human labels finds Vicuna to retain only $64\%$ of ChatGPT's quality on professional and academic exams (see Figure~\ref{fig:agi-eval}), and only $48\%$ of ChatGPT's quality on complex benchmarks like BigBench-hard~\cite{suzgun2022challenging} (see Figure~\ref{fig:bbh-eval}). This discrepancy not only underscores the limitations of existing evaluation protocols with smaller LLMs, but it also reveals their significant lag in reasoning and comprehension capabilities. In essence, these models may be articulate, but they may not necessarily possess robust reasoning skills. In this study, we discuss some of the reasons behind these gaps and propose strategies for addressing them. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.10172" label="2307.10172">
        <attvalues>
          <attvalue for="0" value="DialogStudio: Towards Richest and Most Diverse Unified Dataset&#10;  Collection for Conversational AI" />
          <attvalue for="1" value="  Despite advancements in conversational AI, language models encounter&#10;challenges to handle diverse conversational tasks, and existing dialogue&#10;dataset collections often lack diversity and comprehensiveness. To tackle these&#10;issues, we introduce DialogStudio: the largest and most diverse collection of&#10;dialogue datasets, unified under a consistent format while preserving their&#10;original information. Our collection encompasses data from open-domain&#10;dialogues, task-oriented dialogues, natural language understanding,&#10;conversational recommendation, dialogue summarization, and knowledge-grounded&#10;dialogues, making it an incredibly rich and diverse resource for dialogue&#10;research and model training. To further enhance the utility of DialogStudio, we&#10;identify the licenses for each dataset, design external knowledge and&#10;domain-aware prompts for selected dialogues to facilitate instruction-aware&#10;fine-tuning. Furthermore, we develop conversational AI models using the dataset&#10;collection, and our experiments in both zero-shot and few-shot learning&#10;scenarios demonstrate the superiority of DialogStudio. To improve transparency&#10;and support dataset and task-based research, as well as language model&#10;pre-training, all datasets, licenses, codes, and models associated with&#10;DialogStudio are made publicly&#10;accessible\footnote{\url{https://github.com/salesforce/DialogStudio}}.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years have seen remarkable progress in Conversational AI, primarily driven by the advent of approaches and language models~\cite{shuster2022blenderbot,zhang2023enhancing,longpre2023flan,touvron2023llama}. &#10;Despite the advancements, these models could fall short when handling various tasks in a conversation due to the lack of comprehensive and diverse training data. &#10;Current dialogue datasets~\cite{lin2021bitod,asri2017frames} are typically limited in size and task-specific, which thus results in suboptimal ability in task-oriented model performance.&#10;Additionally, the lack of dataset standardization impedes model generalizability.&#10;&#10;A few recent works~\cite{gupta2022instructdial,longpre2023flan,ding2023enhancing} have introduced a large collection of datasets, which includes diverse tasks based on public datasets. For instance, FlanT5~\cite{longpre2023flan} presents the flan collections with a wide array of datasets and tasks. Despite this breadth, the coverage of dialogue datasets within the Flan collection remains notably sparse, featuring only about ten datasets. Although OPT~\cite{iyer2022opt} have incorporated collections with several dialogue datasets, these collections remain inaccessible to the public. In contract, efforts like InstructDial~\cite{gupta2022instructdial} and ParlAI~\cite{miller2017parlai} consist of more dialogue datasets, but they lack diversity and comprehensiveness. &#10;For instance, ParlAI mainly includes open-domain dialogue datasets, which are exclusively accessible through their platform. Other collections~\cite{gupta2022instructdial,kim2022soda,ding2023enhancing,dubois2023alpacafarm} often distill single dataset from ChatGPT or process datasets into a sequence-to-sequence format to support language model training, featuring only input-output pairs such as dialogue context and system response. However, previous collections often overlook other crucial dialogue information, constraining their utility for research on individual datasets, tasks, and broader applications. &#10;&#10;To overcome the aforementioned challenges, we introduce DialogStudio, the most comprehensive and diverse collection of publicly available dialogue datasets, unified under a consistent format. By aggregating dialogues from various sources, DialogStudio promotes holistic analysis and the development of models adaptable to a variety of conversational scenarios. The collection spans an extensive range of domains, aspects, and tasks, and it is inclusive of several categories: Open-Domain Dialogues, Task-Oriented Dialogues, Natural Language Understanding, Conversational Recommendation, Dialogue Summarization, and Knowledge-Grounded Dialogues. Thus, it can provide support for research in both individual dialogue tasks and large-scale language pre-training.&#10;&#10;DialogStudio stands out not only for its comprehensive coverage but also for its accessibility. It offers easy access with a unified format and documents.&#10;A straightforward $\mathtt{load\_dataset()}$ command through HuggingFace allows users to seamlessly interact with the collection, and we have included documentation for each dataset to enhance usability. &#10;We anticipate that this collection will enable comprehensive and standardized training and evaluations of dialogue models, fostering fair comparisons and propelling further advancements in Conversational AI.&#10;&#10;Furthermore, we identify dialogue domains, design external knowledge for available dialogues and create tailored prompts for selected datasets accordingly. Leveraging these datasets from DialogStudio, we have constructed instruction-aware models, with capacities ranging from 770M to 3B parameters. These models have the ability to handle various external knowledge and are adept at both response generation and general tasks, demonstrating the benefits of DialogStudio. The main contributions of this paper are as follows:&#10;\begin{itemize}[leftmargin=*]&#10; \item We introduce DialogStudio, a meticulously curated collection of more than 80 dialogue datasets. These datasets are unified under a consistent format while retaining their original information. We integrate external knowledge, incorporate domain-aware prompts and identify dataset licenses, making DialogStudio an exceptionally rich and diverse resource for dialogue research and model training.&#10; \item We have made our datasets publicly available to enhance transparency and support research efforts. Additionally, we are committed to improving DialogStudio's usability and will persist in our efforts to refine it, ensuring an optimal user experience.&#10; \item We train conversational AI models based on DialogStudio, and these models have demonstrated superior performance over strong baselines in both zero-shot and few-shot learning scenarios. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Dialogue Datasets, Linguistics, Language Model Training, Conversational AI, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2308.08169" label="2308.08169">
        <attvalues>
          <attvalue for="0" value="Enhancing Performance on Seen and Unseen Dialogue Scenarios using&#10;  Retrieval-Augmented End-to-End Task-Oriented System" />
          <attvalue for="1" value="  End-to-end task-oriented dialogue (TOD) systems have achieved promising&#10;performance by leveraging sophisticated natural language understanding and&#10;natural language generation capabilities of pre-trained models. This work&#10;enables the TOD systems with more flexibility through a simple cache. The cache&#10;provides the flexibility to dynamically update the TOD systems and handle both&#10;existing and unseen dialogue scenarios. Towards this end, we first fine-tune a&#10;retrieval module to effectively retrieve the most relevant information entries&#10;from the cache. We then train end-to-end TOD models that can refer to and&#10;ground on both dialogue history and retrieved information during TOD&#10;generation. The cache is straightforward to construct, and the backbone models&#10;of TOD systems are compatible with existing pre-trained generative models.&#10;Extensive experiments demonstrate the superior performance of our framework,&#10;with a notable improvement in non-empty joint goal accuracy by 6.7% compared to&#10;strong baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.02787" label="2106.02787">
        <attvalues>
          <attvalue for="0" value="BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue&#10;  Modeling" />
          <attvalue for="1" value="  Task-oriented dialogue (ToD) benchmarks provide an important avenue to&#10;measure progress and develop better conversational agents. However, existing&#10;datasets for end-to-end ToD modeling are limited to a single language,&#10;hindering the development of robust end-to-end ToD systems for multilingual&#10;countries and regions. Here we introduce BiToD, the first bilingual&#10;multi-domain dataset for end-to-end task-oriented dialogue modeling. BiToD&#10;contains over 7k multi-domain dialogues (144k utterances) with a large and&#10;realistic bilingual knowledge base. It serves as an effective benchmark for&#10;evaluating bilingual ToD systems and cross-lingual transfer learning&#10;approaches. We provide state-of-the-art baselines under three evaluation&#10;settings (monolingual, bilingual, and cross-lingual). The analysis of our&#10;baselines in different settings highlights 1) the effectiveness of training a&#10;bilingual ToD system compared to two independent monolingual ToD systems, and&#10;2) the potential of leveraging a bilingual knowledge base and cross-lingual&#10;transfer learning to improve the system performance under low resource&#10;condition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.12017" label="2212.12017">
        <attvalues>
          <attvalue for="0" value="OPT-IML: Scaling Language Model Instruction Meta Learning through the&#10;  Lens of Generalization" />
          <attvalue for="1" value="  Recent work has shown that fine-tuning large pre-trained language models on a&#10;collection of tasks described via instructions, a.k.a. instruction-tuning,&#10;improves their zero and few-shot generalization to unseen tasks. However, there&#10;is a limited understanding of the performance trade-offs of different decisions&#10;made during the instruction-tuning process. These decisions include the scale&#10;and diversity of the instruction-tuning benchmark, different task sampling&#10;strategies, fine-tuning with and without demonstrations, training using&#10;specialized datasets for reasoning and dialogue, and finally, the fine-tuning&#10;objectives themselves. In this paper, we characterize the effect of&#10;instruction-tuning decisions on downstream task performance when scaling both&#10;model and benchmark sizes. To this end, we create OPT-IML Bench: a large&#10;benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated&#10;into task categories from 8 existing benchmarks, and prepare an evaluation&#10;framework to measure three types of model generalizations: to tasks from fully&#10;held-out categories, to held-out tasks from seen categories, and to held-out&#10;instances from seen tasks. Through the lens of this framework, we first present&#10;insights about instruction-tuning decisions as applied to OPT-30B and further&#10;exploit these insights to train OPT-IML 30B and 175B, which are&#10;instruction-tuned versions of OPT. OPT-IML demonstrates all three&#10;generalization abilities at both scales on four different evaluation benchmarks&#10;with diverse tasks and input formats -- PromptSource, FLAN,&#10;Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly&#10;outperform OPT on all benchmarks but is also highly competitive with existing&#10;models fine-tuned on each specific benchmark. We release OPT-IML at both&#10;scales, together with the OPT-IML Bench evaluation framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.06476" label="1705.06476">
        <attvalues>
          <attvalue for="0" value="ParlAI: A Dialog Research Software Platform" />
          <attvalue for="1" value="  We introduce ParlAI (pronounced &quot;par-lay&quot;), an open-source software platform&#10;for dialog research implemented in Python, available at http://parl.ai. Its&#10;goal is to provide a unified framework for sharing, training and testing of&#10;dialog models, integration of Amazon Mechanical Turk for data collection, human&#10;evaluation, and online/reinforcement learning; and a repository of machine&#10;learning models for comparing with others' models, and improving upon existing&#10;architectures. Over 20 tasks are supported in the first release, including&#10;popular datasets such as SQuAD, bAbI tasks, MCTest, WikiQA, QACNN, QADailyMail,&#10;CBT, bAbI Dialog, Ubuntu, OpenSubtitles and VQA. Several models are integrated,&#10;including neural models such as memory networks, seq2seq and attentive LSTMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10465" label="2212.10465">
        <attvalues>
          <attvalue for="0" value="SODA: Million-scale Dialogue Distillation with Social Commonsense&#10;  Contextualization" />
          <attvalue for="1" value="  Data scarcity has been a long standing issue in the field of open-domain&#10;social dialogue. To quench this thirst, we present SODA: the first publicly&#10;available, million-scale high-quality social dialogue dataset. By&#10;contextualizing social commonsense knowledge from a knowledge graph, we are&#10;able to distill an exceptionally broad spectrum of social interactions from a&#10;large language model. Human evaluation shows that conversations in SODA are&#10;more consistent, specific, and (surprisingly) natural than those in prior&#10;human-authored datasets.&#10;  Using SODA, we train COSMO: a generalizable conversation model that is&#10;significantly more natural and consistent on unseen datasets than&#10;best-performing conversation models (e.g., GODEL, BlenderBot-1, Koala, Vicuna).&#10;Experiments reveal COSMO is sometimes even preferred to the original&#10;human-written gold responses. Additionally, our results shed light on the&#10;distinction between knowledge-enriched conversations and natural social&#10;chitchats. We plan to make our data, model, and code public.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14387" label="2305.14387">
        <attvalues>
          <attvalue for="0" value="AlpacaFarm: A Simulation Framework for Methods that Learn from Human&#10;  Feedback" />
          <attvalue for="1" value="  Large language models (LLMs) such as ChatGPT have seen widespread adoption&#10;due to their strong instruction-following abilities. Developing these LLMs&#10;involves a complex yet poorly understood workflow requiring training with human&#10;feedback. Replicating and understanding this instruction-following requires&#10;tackling three major challenges: the high cost of data collection, the lack of&#10;trustworthy evaluation, and the absence of reference method implementations. We&#10;address these challenges with AlpacaFarm, a simulator that enables research and&#10;development for learning from feedback at a low cost. First, we design LLM&#10;prompts to simulate human feedback that are 50x cheaper than crowdworkers and&#10;display high agreement with humans. Second, we propose an automatic evaluation&#10;and validate it against human instructions obtained on real-world interactions.&#10;Third, we contribute reference implementations for several methods (PPO, DPO,&#10;best-of-n, expert iteration, and more) that learn from pairwise feedback.&#10;Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate&#10;eleven models on 10k pairs of real human feedback and show that rankings of&#10;models trained in AlpacaFarm match rankings of models trained on human data. As&#10;a demonstration of the research possible in AlpacaFarm, we find that methods&#10;that use a reward model can substantially improve over supervised fine-tuning&#10;and that our reference PPO implementation leads to a +10% improvement in&#10;win-rate against Davinci003. We release all components of AlpacaFarm at&#10;https://github.com/tatsu-lab/alpaca_farm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.16013" label="2007.16013">
        <attvalues>
          <attvalue for="0" value="Neural Composition: Learning to Generate from Multiple Models" />
          <attvalue for="1" value="  Decomposing models into multiple components is critically important in many&#10;applications such as language modeling (LM) as it enables adapting individual&#10;components separately and biasing of some components to the user's personal&#10;preferences. Conventionally, contextual and personalized adaptation for&#10;language models, are achieved through class-based factorization, which requires&#10;class-annotated data, or through biasing to individual phrases which is limited&#10;in scale. In this paper, we propose a system that combines model-defined&#10;components, by learning when to activate the generation process from each&#10;individual component, and how to combine probability distributions from each&#10;component, directly from unlabeled text data.&#10;" />
          <attvalue for="2" value="&#10;Language models are a key component of applications that require generation of coherent natural language text, &#10;&#09;including machine translation, speech recognition, abstractive text summarization, and many others.&#10;For a long time n-gram models \cite{ueberla1993ngram} dominated the field due to their simplicity, efficiency and scalability.&#10;However, recently neural models gained popularity, notably from simple recurrent networks \cite{mikolov2010rnnlm} to &#10;&#09;very powerful models including \cite{gpt2, yang2019xlnet}.&#10;These models often include billions of parameters and they have been shown to do very well at generalizing from vast amounts of data.&#10;However, how to adapt these models to different users (e.g., personalized contact list in a messaging application), &#10;&#09;or how to update these models efficiently (for example, when a new movie title is released, which may be important for a ticket booking application)&#10;&#09;does still remain a challenge. &#10;When the number of users is large, or updates are frequent, adapting a large monolithic model becomes impractical &#10;&#09;and this necessitates the use of composite models in which some components may be updated separately.&#10;&#10;For these reasons, class-based models are still widely used in different applications, particularly in automatic speech recognition (ASR) where integrating external knowledge sources and personalized entities in the language model are crucial in achieving accurate transcription:&#10;&#09;\cite{contactnames2015, dynclass2016vasserman, mcgraw2016personalized, chen2018endtoend}.&#10;Class-based models, however, require annotations in order to learn where these components/classes are used which limits their applicability.&#10;Instead of using classes, where content of a class is assumed to be similar in some way, e.g., entities of the same type, &#10;&#09;\cite{hall2015ngram, scheiner2016voicesearch, he2019e2e} boost scores of individual phrases and n-grams to bias ASR search.&#10;Note that this type of biasing can be applied to both WFST-based and neural models. &#10;&#10;\cite{pundak2018deep} learn a fixed-size representation for every biasing phrase separately.&#10;The ASR decoder then uses attention mechanism to interpolate these representations and the result is added to the decoder's input.&#10;As the decoder needs to attend to each individual phrase at every step, scaling this approach to a large number of biasing phrases and entities poses an engineering challenge. &#10;\cite{khandelwal2020generalization} propose nearest-neighbor LM which can use external data to bias its predictions,&#10;&#09;however, significant limits application of this type of model, especially in ASR domain.&#10;\cite{levit2015personalization} is similar to our work in that they aim to solve a similar problem.&#10;They use expectation-maximization method to learn a class-based (or more generally, word-phrase-entity) model without a requirement for annotated data.&#10;However, their method only applies to n-gram models while we do not make assumptions about internal structure of component models.&#10;&#10;In this paper, we take an approach reminiscent of a class-based model in that we use components (classes)&#10;&#09;whose elements are expected to be used in similar context.&#10;&#10;We call them model-defined components because they are defined by their respective models (FST- or neural-based).&#10; Unlike class-based models, however, we do not assign any tags to these components.&#10; This allows us to do away with one of the main shortcomings of class-based models -- the requirement for annotated (manually or automatically) data.&#10; The main motivating idea of our method is as follows:&#10; &#09;given a general generative language model and some components represented as generative LMs, &#10;&#09;we can learn where these components are useful, i.e. where they make better predictions than the general model.&#10;&#09;Additionally, the proposed model learns, directly from data, how to interpolate different components at each token, which class-based approaches are incapable of due to their explicit factorization into sequence of classes and words. &#10;Note that our approach does not require us to assign any semantic tags to components,&#10;&#09;their meaning is implicit and arises from their content.&#10;&#10;It is worth noting that there are many methods for combining multiple full-sentence language model in the literature, \cite{iyer1997ood, kalai1999online, broman2005} to name a few.&#10;However, such methods cannot be applied to entity models with full-sentence models, and therefore these methods cannot solve the problem we seek to address.&#10;&#10;The rest of the paper is organized as follows:&#10;&#09;in Section~\ref{section:model}, we describe the structure of the proposed model,&#10;&#09;the training procedure is detailed in Section~\ref{section:model-training}.&#10;In Section~\ref{section:experiments}, we present experimental results, &#10;&#09;and in Section~\ref{section:conclusions} we conclude and outline future work.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Model Decomposition, Linguistics, Personalized Adaptation, Artificial Intelligence, Mathematics, Language Modeling" />
        </attvalues>
      </node>
      <node id="2004.06499" label="2004.06499">
        <attvalues>
          <attvalue for="0" value="What's so special about BERT's layers? A closer look at the NLP pipeline&#10;  in monolingual and multilingual models" />
          <attvalue for="1" value="  Peeking into the inner workings of BERT has shown that its layers resemble&#10;the classical NLP pipeline, with progressively more complex tasks being&#10;concentrated in later layers. To investigate to what extent these results also&#10;hold for a language other than English, we probe a Dutch BERT-based model and&#10;the multilingual BERT model for Dutch NLP tasks. In addition, through a deeper&#10;analysis of part-of-speech tagging, we show that also within a given task,&#10;information is spread over different parts of the network and the pipeline&#10;might not be as neat as it seems. Each layer has different specialisations, so&#10;that it may be more useful to combine information from different layers,&#10;instead of selecting a single one based on the best overall performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, BERT Model Analysis, Computer Science, Linguistics, Multilingual NLP Tasks, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2005.01810" label="2005.01810">
        <attvalues>
          <attvalue for="0" value="Spying on your neighbors: Fine-grained probing of contextual embeddings&#10;  for information about surrounding words" />
          <attvalue for="1" value="  Although models using contextual word embeddings have achieved&#10;state-of-the-art results on a host of NLP tasks, little is known about exactly&#10;what information these embeddings encode about the context words that they are&#10;understood to reflect. To address this question, we introduce a suite of&#10;probing tasks that enable fine-grained testing of contextual embeddings for&#10;encoding of information about surrounding words. We apply these tasks to&#10;examine the popular BERT, ELMo and GPT contextual encoders, and find that each&#10;of our tested information types is indeed encoded as contextual information&#10;across tokens, often with near-perfect recoverability-but the encoders vary in&#10;which features they distribute to which tokens, how nuanced their distributions&#10;are, and how robust the encoding of each feature is to distance. We discuss&#10;implications of these results for how different types of models breakdown and&#10;prioritize word-level context information when constructing token embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Much work has been done on analyzing the information captured by sentence encoders and language models in general. Classification-based probing tasks have been used to analyze the contents of sentence embeddings~\cite{adi2016,conneau2018you,ettinger2016probing}, finding that these embeddings encode a variety of information about sentence structure, content, length, etc., though more tightly-controlled tasks suggest weaknesses in capturing basic sentence meaning~\cite{ettinger2018assessing}. Our work uses the same classification-based probing methodology, but focuses on probing token-level embeddings for context information.&#10;&#10;Other work has analyzed linguistic capacities of language models by examining output probabilities in context, emulating methods for studying human language processing. Much of this work has studied sensitivity to syntactic dependencies in recurrent neural network language models~\cite[e.g.][]{linzen2016assessing, wilcox2018rnn,chowdhury2018rnn,gulordava2019colorless, marvin2019targeted,futrell2019neural}. Using similar methods to test syntactic awareness in BERT,~\cite{goldberg2019assessing} finds the model to perform almost at ceiling on syntactic tests. Testing BERT's outputs on a range of semantic, syntactic and pragmatic information,~\cite{ettinger2020bert} finds strong sensitivity to syntax, but clear limitations in areas of semantics and pragmatic/commonsense reasoning. We complement this work with a direct focus on the contextual token representations learned by models pre-trained on language modeling, examining the syntactic and semantic information that these embeddings capture about surrounding words. &#10;&#10;Most directly related to the present work are studies using probing and other methods to analyze information in contextual token embeddings. Some of this research \cite[e.g.][]{tenney2019bert, jawahar2019does} finds that BERT encodes more local, syntactic information at lower layers and more global, semantic information at higher layers. \cite{peters2018dissecting} find that encoders differ in encoding strength for semantic features but all encode these features strongly where possible. \cite{hewitt2019structural} provide evidence that contextual encoders capture sentence-level hierarchical syntactic structures in their representations. Other work \cite{liu2019linguistic, tenney2019you} finds that contextual word encoders struggle to learn fine-grained linguistic information in a variety of contexts. These papers have focused primarily on studying the ability of contextual embeddings to capture information about the full sentence, or about phrases or dependencies of which those contextual embeddings are a part. We focus on mapping the precise distribution of context information across token embeddings, with a systematic, fine-grained investigation of the information that each token encodes about each of its surrounding tokens.&#10;&#10;" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Contextual Word Embeddings" />
        </attvalues>
      </node>
      <node id="2405.05348" label="2405.05348">
        <attvalues>
          <attvalue for="0" value="The Effect of Model Size on LLM Post-hoc Explainability via LIME" />
          <attvalue for="1" value="  Large language models (LLMs) are becoming bigger to boost performance.&#10;However, little is known about how explainability is affected by this trend.&#10;This work explores LIME explanations for DeBERTaV3 models of four different&#10;sizes on natural language inference (NLI) and zero-shot classification (ZSC)&#10;tasks. We evaluate the explanations based on their faithfulness to the models'&#10;internal decision processes and their plausibility, i.e. their agreement with&#10;human explanations. The key finding is that increased model size does not&#10;correlate with plausibility despite improved model performance, suggesting a&#10;misalignment between the LIME explanations and the models' internal processes&#10;as model size increases. Our results further suggest limitations regarding&#10;faithfulness metrics in NLI contexts.&#10;" />
          <attvalue for="2" value="&#10;&#10;Research has shown that performance in language models depends strongly on scale and less on model shape \cite{kaplan}, where scale refers to the number of parameters, the training dataset size, and the amount of compute for training. For instance, OpenAI's series of Generative Pre-Trained Transformers (GPT) has grown from 1.5 billion parameters for GPT-2 to 175 billion parameters for GPT-3 which helped improve across various NLP tasks \cite{brownGPT3}. This trend seems likely to continue. &#10;&#10;As LLMs grow in size and performance and are increasingly deployed in high-stakes applications, the need to understand and explain their behaviour becomes more crucial. Post-hoc explainability methods such as LIME \cite{ribeiro} are one way of attempting to do this. Although these methods have been widely applied to LLMs \cite{madsen}, to the best of our knowledge no research has been conducted on the impact of model size on the quality of these kinds of explanations. Here we begin to fill this gap by investigating the impact of model size on the quality of LIME explanations. We apply two approaches to assess the quality of explanations, namely faithfulness \cite{chan} and plausibility \cite{deyoung}. While faithfulness aims to measure the extent to which an explanation reflects the true internal decision processes, plausibility assesses the quality of the explanations based on their agreement with human-generated explanations.&#10;&#10;We find that, even though model performance increases with model size, the agreement between human-generated and LIME-generated explanations does not. This indicates some extent of misalignment between the explanations and the true internal decision processes. Our findings also imply possible flaws in removal-based faithfulness metrics based on the NLP task which points to more general limitations for highlight-based post-hoc explainability in NLP such as lack of expressiveness. This study serves as a first attempt to understand how post-hoc explainability is affected by model size. We hope that this research encourages others to further explore this area and to that end we provide an extensible code repository for others to build on. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Natural Language Processing, Model Size Performance, Language Model Explainability" />
        </attvalues>
      </node>
      <node id="2108.04840" label="2108.04840">
        <attvalues>
          <attvalue for="0" value="Post-hoc Interpretability for Neural NLP: A Survey" />
          <attvalue for="1" value="  Neural networks for NLP are becoming increasingly complex and widespread, and&#10;there is a growing concern if these models are responsible to use. Explaining&#10;models helps to address the safety and ethical concerns and is essential for&#10;accountability. Interpretability serves to provide these explanations in terms&#10;that are understandable to humans. Additionally, post-hoc methods provide&#10;explanations after a model is learned and are generally model-agnostic. This&#10;survey provides a categorization of how recent post-hoc interpretability&#10;methods communicate explanations to humans, it discusses each method in-depth,&#10;and how they are validated, as the latter is often a common concern.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05514" label="2204.05514">
        <attvalues>
          <attvalue for="0" value="A Comparative Study of Faithfulness Metrics for Model Interpretability&#10;  Methods" />
          <attvalue for="1" value="  Interpretation methods to reveal the internal reasoning processes behind&#10;machine learning models have attracted increasing attention in recent years. To&#10;quantify the extent to which the identified interpretations truly reflect the&#10;intrinsic decision-making mechanisms, various faithfulness evaluation metrics&#10;have been proposed. However, we find that different faithfulness metrics show&#10;conflicting preferences when comparing different interpretations. Motivated by&#10;this observation, we aim to conduct a comprehensive and comparative study of&#10;the widely adopted faithfulness metrics. In particular, we introduce two&#10;assessment dimensions, namely diagnosticity and time complexity. Diagnosticity&#10;refers to the degree to which the faithfulness metric favours relatively&#10;faithful interpretations over randomly generated ones, and time complexity is&#10;measured by the average number of model forward passes. According to the&#10;experimental results, we find that sufficiency and comprehensiveness metrics&#10;have higher diagnosticity and lower time complexity than the other faithfulness&#10;metric&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.12754" label="2406.12754">
        <attvalues>
          <attvalue for="0" value="Chumor 1.0: A Truly Funny and Challenging Chinese Humor Understanding&#10;  Dataset from Ruo Zhi Ba" />
          <attvalue for="1" value="  Existing humor datasets and evaluations predominantly focus on English,&#10;lacking resources for culturally nuanced humor in non-English languages like&#10;Chinese. To address this gap, we construct Chumor, a dataset sourced from Ruo&#10;Zhi Ba (RZB), a Chinese Reddit-like platform dedicated to sharing&#10;intellectually challenging and culturally specific jokes. We annotate&#10;explanations for each joke and evaluate human explanations against two&#10;state-of-the-art LLMs, GPT-4o and ERNIE Bot, through A/B testing by native&#10;Chinese speakers. Our evaluation shows that Chumor is challenging even for SOTA&#10;LLMs, and the human explanations for Chumor jokes are significantly better than&#10;explanations generated by the LLMs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Humor is an intrinsic human trait that touches the core of our social and emotional lives, making it a rich field of study across various disciplines \cite{lefcourt2001humor, mihalcea-strapparava-2005-making, gelkopf2011use, hessel-etal-2023-androids}. &#10;With the advent of Large Language Models (LLMs), researchers have evaluated LLMs' performance on diverse tasks \cite{liu2023evaluating, deng2024tables, wu-etal-2023-hi}&#10;and observed LLMs' extraordinary performance on many \cite{10.1162/tacl_a_00632}.&#10;In contrast, researchers have observed that LLMs still fail to understand humor \cite{ghanadian-etal-2023-chatgpt}.&#10;However, with all these studies on humor and LLMs' understanding of humor, most of these humor datasets and evaluations remain in English \cite{radev-etal-2016-humor, hasan-etal-2019-ur}.&#10;This presents a significant gap, particularly for non-English languages like Chinese, where culturally nuanced humor understanding is underrepresented. &#10;&#10;In this paper, we try to address this gap by constructing Chumor, a truly funny and challenging Chinese humor understanding dataset sourced from Ruo Zhi Ba (\begin{CJK}{UTF8}{gbsn}RZB, ``弱智吧''\end{CJK} in Chinese), a Chinese version of Reddit platform dedicated to sharing intellectually challenging and culturally specific jokes.&#10;This platform provides a set of unique Chinese jokes that incorporate the subtleties and intricacies of Chinese humor.&#10;\ref{fig:example} provides an example of the joke from RZB.&#10;In addition, \cite{bai2024coig} have observed that when tuning LLMs on data from RZB, LLMs achieve the best performance on Chinese reasoning tasks compared to tuning LLMs on data from other sources, indicating the significant value of jokes from RZB.&#10;&#10;In Chumor, we manually annotate the explanations for each joke.&#10;We then prompt two state-of-the-art (SOTA) LLMs, GPT-4o from OpenAI and ERNIE Bot from Baidu to get their explanations.&#10;In the evaluation, a group of native Chinese speakers determine their preferences between the explanations from human and the LLM in an A/B testing fashion.&#10;We note that in the preference annotation process, all of our annotators report to us that the jokes are very funny and align well with the Chinese Internet trends.&#10;Our evaluation shows that Chumor~is challenging even for the SOTA LLMs, and the human explanations for Chumor~jokes are significantly better than explanations generated by the LLMs (\ref{fig:gpt-4o-preference-eval}).&#10;In addition, we provide examples of our jokes on which LLMs fail to provide explanations and the hypothesis of their failures.&#10;&#10;In summary, our contributions are two folds:&#10;&#10;\begin{enumerate}[leftmargin=\parindent,align=left,labelwidth=\parindent,labelsep=0pt]&#10; \item We construct Chumor, a truly funny and challenging Chinese humor understanding dataset, addressing the lack of non-English humor understanding dataset.&#10; \item We reveal that on Chumor, human explanations for jokes are significantly better than explanations from SOTA LLMs.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Multilingual Humor Analysis, Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Humor Dataset Construction, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.03439" label="2304.03439">
        <attvalues>
          <attvalue for="0" value="Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4" />
          <attvalue for="1" value="  Harnessing logical reasoning ability is a comprehensive natural language&#10;understanding endeavor. With the release of Generative Pretrained Transformer 4&#10;(GPT-4), highlighted as &quot;advanced&quot; at reasoning tasks, we are eager to learn&#10;the GPT-4 performance on various logical reasoning tasks. This report analyses&#10;multiple logical reasoning datasets, with popular benchmarks like LogiQA and&#10;ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice&#10;reading comprehension and natural language inference tasks with benchmarks&#10;requiring logical reasoning. We further construct a logical reasoning&#10;out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4.&#10;We also make a performance comparison between ChatGPT and GPT-4. Experiment&#10;results show that ChatGPT performs significantly better than the RoBERTa&#10;fine-tuning method on most logical reasoning benchmarks. With early access to&#10;the GPT-4 API we are able to conduct intense experiments on the GPT-4 model.&#10;The results show GPT-4 yields even higher performance on most logical reasoning&#10;datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known&#10;datasets like LogiQA and ReClor. However, the performance drops significantly&#10;when handling newly released and out-of-distribution datasets. Logical&#10;reasoning remains challenging for ChatGPT and GPT-4, especially on&#10;out-of-distribution and natural language inference datasets. We release the&#10;prompt-style logical reasoning datasets as a benchmark suite and name it&#10;LogiEval.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.08126" label="1506.08126">
        <attvalues>
          <attvalue for="0" value="Humor in Collective Discourse: Unsupervised Funniness Detection in the&#10;  New Yorker Cartoon Caption Contest" />
          <attvalue for="1" value="  The New Yorker publishes a weekly captionless cartoon. More than 5,000&#10;readers submit captions for it. The editors select three of them and ask the&#10;readers to pick the funniest one. We describe an experiment that compares a&#10;dozen automatic methods for selecting the funniest caption. We show that&#10;negative sentiment, human-centeredness, and lexical centrality most strongly&#10;match the funniest captions, followed by positive sentiment. These results are&#10;useful for understanding humor and also in the design of more engaging&#10;conversational agents in text and multimodal (vision+text) systems. As part of&#10;this work, a large set of cartoons and captions is being made available to the&#10;community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.18058" label="2403.18058">
        <attvalues>
          <attvalue for="0" value="COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning" />
          <attvalue for="1" value="  Recently, there have been significant advancements in large language models&#10;(LLMs), particularly focused on the English language. These advancements have&#10;enabled these LLMs to understand and execute complex instructions with&#10;unprecedented accuracy and fluency. However, despite these advancements, there&#10;remains a noticeable gap in the development of Chinese instruction tuning. The&#10;unique linguistic features and cultural depth of the Chinese language pose&#10;challenges for instruction tuning tasks. Existing datasets are either derived&#10;from English-centric LLMs or are ill-suited for aligning with the interaction&#10;patterns of real-world Chinese users. To bridge this gap, we introduce&#10;COIG-CQIA, a high-quality Chinese instruction tuning dataset. Our aim is to&#10;build a diverse, wide-ranging instruction-tuning dataset to better align model&#10;behavior with human interactions. To this end, we collect a high-quality&#10;human-written corpus from various sources on the Chinese Internet, including&#10;Q&amp;A communities, Wikis, examinations, and existing NLP datasets. This corpus&#10;was rigorously filtered and carefully processed to form the COIG-CQIA dataset.&#10;Furthermore, we train models of various scales on different subsets of CQIA,&#10;following in-depth evaluation and analyses. The findings from our experiments&#10;offer valuable insights for selecting and developing Chinese instruction-tuning&#10;datasets. We also find that models trained on CQIA-Subset achieve competitive&#10;results in human assessment as well as knowledge and security benchmarks. Data&#10;are available at https://huggingface.co/datasets/m-a-p/COIG-CQIA&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11072" label="2305.11072">
        <attvalues>
          <attvalue for="0" value="Self-supervised Fine-tuning for Improved Content Representations by&#10;  Speaker-invariant Clustering" />
          <attvalue for="1" value="  Self-supervised speech representation models have succeeded in various tasks,&#10;but improving them for content-related problems using unlabeled data is&#10;challenging. We propose speaker-invariant clustering (Spin), a novel&#10;self-supervised learning method that clusters speech representations and&#10;performs swapped prediction between the original and speaker-perturbed&#10;utterances. Spin disentangles speaker information and preserves content&#10;representations with just 45 minutes of fine-tuning on a single GPU. Spin&#10;improves pre-trained networks and outperforms prior methods in speech&#10;recognition and acoustic unit discovery.&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-supervised learning (SSL) for speech representation using large neural networks and unlabeled data offers effective initialization and representations for downstream tasks~\cite{yang2021superb,evain2021lebenchmark,chang2021exploration,tsai-etal-2022-superb,mohamed2022self}.&#10;Among prior methods, learning discrete units like K-means clusters benefits downstream performance~\cite{hsu2021hubert,chen2022wavlm,chung2021w2v,maekaku2022exploration,chiu2022bestrq,ren2022speech,wells2022phonetic}.&#10;While speech representation encompasses information from multiple aspects, most SSL methods lack explicit speaker disentanglement.&#10;Extracting speaker-invariant linguistic content can benefit downstream tasks like automatic speech recognition~(ASR) and phoneme recognition~(PR)~\cite{hsu2017unsupervised,tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}.&#10;In light of this, ContentVec~\cite{qian2022contentvec} imposes speaker-invariant constraints to pre-trained HuBERT models~\cite{hsu2021hubert} to improve content-related downstream tasks.&#10;However, ContentVec adds a substantial amount of computational cost, requiring 19 hours on 36 GPUs, on top of the pre-trained models, which are already expensive to compute.&#10;&#10;In this paper, we first demonstrate the benefits of extracting features closer to the underlying phonetic content to motivate our work.&#10;Following this observation, we present speaker-invariant clustering~(Spin), a novel and cost-effective self-supervised fine-tuning~(SSFT) method for SSL models that leverages vector quantization~\cite{caron2018deep,asano2019self,caron2020unsupervised} and speaker disentanglement~\cite{qian2022contentvec} to improve content representation.&#10;In short, Spin~is trained to identify the unchanged spoken content from pairs of speaker-augmented utterances via quantized representation matching.&#10;Such design leads to a disentangled representation focusing on the spoken content, improving various downstream tasks, including content-related tasks in SUPERB~\cite{yang2021superb} and ZeroSpeech~\cite{nguyen2020zero}.&#10;In terms of efficiency, we show that Spin~requires less than 45 minutes of training on a single GPU, costing less than 1\% of ContentVec.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Speaker Invariant Modeling, Artificial Intelligence, Speech Representation, Signal Processing, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2110.04590" label="2110.04590">
        <attvalues>
          <attvalue for="0" value="An Exploration of Self-Supervised Pretrained Representations for&#10;  End-to-End Speech Recognition" />
          <attvalue for="1" value="  Self-supervised pretraining on speech data has achieved a lot of progress.&#10;High-fidelity representation of the speech signal is learned from a lot of&#10;untranscribed data and shows promising performance. Recently, there are several&#10;works focusing on evaluating the quality of self-supervised pretrained&#10;representations on various tasks without domain restriction, e.g. SUPERB.&#10;However, such evaluations do not provide a comprehensive comparison among many&#10;ASR benchmark corpora. In this paper, we focus on the general applications of&#10;pretrained speech representations, on advanced end-to-end automatic speech&#10;recognition (E2E-ASR) models. We select several pretrained speech&#10;representations and present the experimental results on various open-source and&#10;publicly available corpora for E2E-ASR. Without any modification of the&#10;back-end model architectures or training strategy, some of the experiments with&#10;pretrained representations, e.g., WSJ, WSJ0-2mix with HuBERT, reach or&#10;outperform current state-of-the-art (SOTA) recognition performance. Moreover,&#10;we further explore more scenarios for whether the pretraining representations&#10;are effective, such as the cross-language or overlapped speech. The scripts,&#10;configuratons and the trained models have been released in ESPnet to let the&#10;community reproduce our experiments and improve them.&#10;" />
          <attvalue for="2" value="&#10;&#10;The performance of speech recognition systems have been improved a lot over the last decade. On the one hand, the rapid development of deep neural networks has dramatically pushed the limit of the models~\cite{abdel2012applying,graves2013speech,chan2015listen,vaswani2017attention,dong2018speech,gulati2020conformer}. On the other hand, the increasing computing resources have enabled to train an automatic speech recognition (ASR) system with a large amount of transcribed data~\cite{panayotov2015librispeech,chen2021gigaspeech}, leading to a better performance. It is known that the deep neural networks are data hungry, thus some researchers have been trying to improve the capacity of neural networks by incorporating more and more transcribed data~\cite{chan2021speechstew}. However, using the transcribed data only is not efficient because the untranscribed data is of great portion in all the data available. Motivated by this, researchers proposed to make use of the untranscribed data, known as unsupervised and semi-supervised learning. Recently, it has become a hot topic in speech recognition and can be roughly divided into two approaches. In~\cite{lee2013pseudo,synnaeve2019end,kahn2020self}, a semi-supervised method, called pseudo-labelling, was proposed to use both transcribed and untranscribed data. A seed model is first trained with the transcribed data in a supervised manner, which is then used to generate the pseudo-labels for the untranscribed data. After that, a model can be trained with all the data in a supervised manner.&#10;&#10;Previous studies in computer vision (CV) and natural language processing (NLP) have investigated to learn representations from data, showing the advantages in the corresponding downstream tasks~\cite{vinyals2016show,devlin2018bert,radford2018improving}. Similarly, another approach was proposed to pretrain models using a large amount of untranscribed data for learning high quality speech features. In this context, many pretrained speech representation models have been proposed, which are often referred as self-supervised learning representation (SSLR). These SSLRs can be categorized by their training objectives. To be specific, one direct way to learn the speech representations is to predict the future information given the history information. In~\cite{chung2019unsupervised,chung2020vector}, the authors adopted an method similar to the autoregressive language models (LMs) to predict the future acoustic features (e.g. FBANK) conditioned on the past input features, called autoregressive predictive coding (APC). Instead of autoregressive modeling, some researchers proposed to use masking prediction techniques as in BERT-LM~\cite{devlin2018bert} to learn the speech representations, including Mockingjay~\cite{liu2020mockingjay}, TERA~\cite{liu2020tera} and NPC~\cite{liu2020non}. However, it is not necessary to learn the speech representations by reconstructing the acoustic features. In~\cite{oord2018representation,schneider2019wav2vec}, the models were optimized with a contrastive loss to distinguish the positive sample from negative samples in predictions of future. Later in~\cite{baevski2019vq, baevski2020wav2vec}, a BERT Transformer model is concatenated after the encoder trained by the contrastive loss. Recently, a novel model, called HuBERT~\cite{hsu2021hubert}, was proposed to pretrain the representation model by a classification tasks using pseudo-labels motivated by deep cluster models \cite{caron2018deep,xie2016unsupervised}.&#10;&#10;All the proposed representations have shown promising results, however, we can hardly draw a conclusion about a suitable representation for various tasks because their experiments focused on a limited number of tasks and were performed independently. Recently, a benchmark, called Speech processing Universal PERformance Benchmark (SUPERB)~\cite{yang2021superb}, was proposed to provide a fair and standard evaluation of various speech representations, with a unified toolkit, S3PRL. SUPERB focuses on the shallow information of each representation. During evaluation, all the representation models are frozen and applied on several downstream tasks each of which uses a quite light-weight downstream model. For example, the ASR task is evaluated using a two-layer RNN-based connectionist temporal classification (CTC) model. Such evaluation provides informative clues to compare the capacity and the concentration of information for each SSLR. Besides, it prepares the easy access to a lot of pretrained SSLR models. Thus, SUPERB is a very strong benchmark for evaluating the SSLRs without any doubt. &#10;&#10;With that being said, it still remains a question that how well these SSLRs can perform in the advanced speech recognition systems. In this paper, we investigate the performance of end-to-end ASR (E2E-ASR) systems using the pretrained SSLRs. To achieve this, we incorporated the SSLRs from S3PRL, the toolkit used in SUPERB, to the ESPnet\cite{watanabe2018espnet}, a widely used E2E speech processing toolkit. Thus, we can easily evaluate the E2E-ASR performance of pretrained SSLRs available in S3PRL using the current state-of-the-art (SOTA) neural network models, such as Transformers~\cite{vaswani2017attention,dong2018speech} and Conformers~\cite{gulati2020conformer}. We can also easily evalute the SSLRs in other downstream tasks, including speech translation (ST) \cite{inaguma2020espnet} and speech enhancement (SE) \cite{li2021espnet}. It is also an interesting question in the air about the generalization ability of these SSLRs, given the fact that most of SSLRs were trained and tested mainly on LibriSpeech~\cite{panayotov2015librispeech,kahn2020libri}. In this project, we explored these pretrained SSLRs on various open-source and publicly available corpora as many as possible, considering different characteristics including read vs. spontaneous speech, single-speaker vs. multi-speaker, noisy/distant-talk environments, and the telephone channel. We show that some of the SSLRs can achieve much better results than the commonly used log-Mel Filterbank (FBANK) feature.&#10;&#10;The contributions of this study include:&#10;\begin{itemize}&#10; \item We implement the use of pretrained SSLRs in advanced E2E-ASR models, based on which we compare the performance of different representations.&#10; \item The experimental results show that simply replacing the FBANK features with the SSLRs can surpass our current best E2E-ASR system. For some ASR benchmark corpora, our results get competitive results with the SOTA systems, such as WSJ, LibriSpeech and TEDLIUM2.&#10; \item We explore more scenarios with domain-mismatch from the raw speech data to train the pretraining representations. Some observations show the relationship between pretraining representations and their applicable scenarios. &#10; \item We provide reproducible benchmark results, recipes, setups and well-trained models on several publicly available corpora in our open source toolkit ESPnet.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.06849" label="2203.06849">
        <attvalues>
          <attvalue for="0" value="SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark&#10;  for Semantic and Generative Capabilities" />
          <attvalue for="1" value="  Transfer learning has proven to be crucial in advancing the state of speech&#10;and natural language processing research in recent years. In speech, a model&#10;pre-trained by self-supervised learning transfers remarkably well on multiple&#10;tasks. However, the lack of a consistent evaluation methodology is limiting&#10;towards a holistic understanding of the efficacy of such models. SUPERB was a&#10;step towards introducing a common benchmark to evaluate pre-trained models&#10;across various speech tasks. In this paper, we introduce SUPERB-SG, a new&#10;benchmark focused on evaluating the semantic and generative capabilities of&#10;pre-trained models by increasing task diversity and difficulty over SUPERB. We&#10;use a lightweight methodology to test the robustness of representations learned&#10;by pre-trained models under shifts in data domain and quality across different&#10;types of tasks. It entails freezing pre-trained model parameters, only using&#10;simple task-specific trainable heads. The goal is to be inclusive of all&#10;researchers, and encourage efficient use of computational resources. We also&#10;show that the task diversity of SUPERB-SG coupled with limited task supervision&#10;is an effective recipe for evaluating the generalizability of model&#10;representation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.10643" label="2205.10643">
        <attvalues>
          <attvalue for="0" value="Self-Supervised Speech Representation Learning: A Review" />
          <attvalue for="1" value="  Although supervised deep learning has revolutionized speech and audio&#10;processing, it has necessitated the building of specialist models for&#10;individual tasks and application scenarios. It is likewise difficult to apply&#10;this to dialects and languages for which only limited labeled data is&#10;available. Self-supervised representation learning methods promise a single&#10;universal model that would benefit a wide variety of tasks and domains. Such&#10;methods have shown success in natural language processing and computer vision&#10;domains, achieving new levels of performance while reducing the number of&#10;labels required for many downstream scenarios. Speech representation learning&#10;is experiencing similar progress in three main categories: generative,&#10;contrastive, and predictive methods. Other approaches rely on multi-modal data&#10;for pre-training, mixing text or visual data streams with speech. Although&#10;self-supervised speech representation is still a nascent research area, it is&#10;closely related to acoustic word embedding and learning with zero lexical&#10;resources, both of which have seen active research for many years. This review&#10;presents approaches for self-supervised speech representation learning and&#10;their connection to other research areas. Since many current methods focus&#10;solely on automatic speech recognition as a downstream task, we review recent&#10;efforts on benchmarking learned representations to extend the application&#10;beyond speech recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.06209" label="2108.06209">
        <attvalues>
          <attvalue for="0" value="W2v-BERT: Combining Contrastive Learning and Masked Language Modeling&#10;  for Self-Supervised Speech Pre-Training" />
          <attvalue for="1" value="  Motivated by the success of masked language modeling~(MLM) in pre-training&#10;natural language processing models, we propose w2v-BERT that explores MLM for&#10;self-supervised speech representation learning. w2v-BERT is a framework that&#10;combines contrastive learning and MLM, where the former trains the model to&#10;discretize input continuous speech signals into a finite set of discriminative&#10;speech tokens, and the latter trains the model to learn contextualized speech&#10;representations via solving a masked prediction task consuming the discretized&#10;tokens. In contrast to existing MLM-based speech pre-training frameworks such&#10;as HuBERT, which relies on an iterative re-clustering and re-training process,&#10;or vq-wav2vec, which concatenates two separately trained modules, w2v-BERT can&#10;be optimized in an end-to-end fashion by solving the two self-supervised&#10;tasks~(the contrastive task and MLM) simultaneously. Our experiments show that&#10;w2v-BERT achieves competitive results compared to current state-of-the-art&#10;pre-trained models on the LibriSpeech benchmarks when using the Libri-Light~60k&#10;corpus as the unsupervised data. In particular, when compared to published&#10;models such as conformer-based wav2vec~2.0 and HuBERT, our model shows~5\%&#10;to~10\% relative WER reduction on the test-clean and test-other subsets. When&#10;applied to the Google's Voice Search traffic dataset, w2v-BERT outperforms our&#10;internal conformer-based wav2vec~2.0 by more than~30\% relatively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;We consider our work most related to~HuBERT~\cite{hsu2021hubert}, vq-wav2vec~\cite{baevski2020vq}, and DiscreteBERT~\cite{baevski2019effectiveness}: w2v-BERT and these methods all try to first transform continuous speech signals into discretized units so as to exploit masked language modeling~(MLM)~\cite{devlin2019bert} for learning contextualized speech representations.&#10;Despite sharing this same high-level philosophy for learning speech representations, there are two key differences between w2v-BERT and other methods.&#10;&#10;The most noticeable difference is that w2v-BERT's speech discretizing module and its main contextualized representation learning module can be trained end-to-end.&#10;This is in contrast to vq-wav2vec and DiscreteBERT, which involve a two-stage process where the speech discretizing module needs to be obtained in advance and is kept frozen during the training of the representation learning module.&#10;In vq-wav2vec and DiscreteBERT, a problematic token ID assignment would negatively affect the subsequent learning module and it is hard for the learning module to recover the errors made by the discretizer.&#10;Observing such drawback, HuBERT greatly improves vq-wav2vec and DiscreteBERT by allowing refinement on the ID assignment via iterating between k-means clustering and re-training its representation learning module. &#10;However, the fact that HuBERT iterates between the two stages also means it involves more heuristic design choices, for example, the gradually increasing number of clusters in different iterations.&#10;End-to-end methods such as w2v-BERT alleviate the need of coordinating multiple stages.&#10;One potential risk for end-to-end approaches compared to k-means clustering is codebook collapse.&#10;In w2v-BERT, we find the contrastive learning objective effectively avoids codebook collapse and thus enables masked prediction training.&#10;&#10;In addition, unlike other methods that use transformer layers~\cite{vaswani2017attention} as building blocks, w2v-BERT adopts conformer layers~\cite{gulati2020conformer} for constructing the network.&#10;As demonstrated in~\cite{gulati2020conformer}, conformer layers, which combine convolution neural networks~(CNNs) and transformers to model both local and global dependencies of audio sequences, are likely a better option for modeling speech than transformer layers and CNNs.&#10;That being said, using a potentially more powerful building block is not the only factor that makes w2v-BERT outperform other methods, as the effectiveness of the pre-training framework itself is also validated in our experiments where w2v-BERT outperforms w2v-Conformer~\cite{zhang2020pushing}, which is also built with conformer layers.&#10;&#10;w2v-BERT is also related to wav2vec~2.0~\cite{baevski2020wav2vec}.&#10;Same as w2v-BERT, wav2vec~2.0 is end-to-end where the discretizer is jointly trained with its representation learning module.&#10;However, wav2vec~2.0 only employs contrastive learning, whose resulting ASR performance lags behind that of combining contrastive learning and masked prediction.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2204.03240" label="2204.03240">
        <attvalues>
          <attvalue for="0" value="Speech Pre-training with Acoustic Piece" />
          <attvalue for="1" value="  Previous speech pre-training methods, such as wav2vec2.0 and HuBERT,&#10;pre-train a Transformer encoder to learn deep representations from audio data,&#10;with objectives predicting either elements from latent vector quantized space&#10;or pre-generated labels (known as target codes) with offline clustering.&#10;However, those training signals (quantized elements or codes) are independent&#10;across different tokens without considering their relations. According to our&#10;observation and analysis, the target codes share obvious patterns aligned with&#10;phonemized text data. Based on that, we propose to leverage those patterns to&#10;better pre-train the model considering the relations among the codes. The&#10;patterns we extracted, called &quot;acoustic piece&quot;s, are from the sentence piece&#10;result of HuBERT codes. With the acoustic piece as the training signal, we can&#10;implicitly bridge the input audio and natural language, which benefits&#10;audio-to-text tasks, such as automatic speech recognition (ASR). Simple but&#10;effective, our method &quot;HuBERT-AP&quot; significantly outperforms strong baselines on&#10;the LibriSpeech ASR task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.07902" label="1709.07902">
        <attvalues>
          <attvalue for="0" value="Unsupervised Learning of Disentangled and Interpretable Representations&#10;  from Sequential Data" />
          <attvalue for="1" value="  We present a factorized hierarchical variational autoencoder, which learns&#10;disentangled and interpretable representations from sequential data without&#10;supervision. Specifically, we exploit the multi-scale nature of information in&#10;sequential data by formulating it explicitly within a factorized hierarchical&#10;graphical model that imposes sequence-dependent priors and sequence-independent&#10;priors to different sets of latent variables. The model is evaluated on two&#10;speech corpora to demonstrate, qualitatively, its ability to transform speakers&#10;or linguistic content by manipulating different sets of latent variables; and&#10;quantitatively, its ability to outperform an i-vector baseline for speaker&#10;verification and reduce the word error rate by as much as 35% in mismatched&#10;train/test scenarios for automatic speech recognition tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.09872" label="2205.09872">
        <attvalues>
          <attvalue for="0" value="Content-Context Factorized Representations for Automated Speech&#10;  Recognition" />
          <attvalue for="1" value="  Deep neural networks have largely demonstrated their ability to perform&#10;automated speech recognition (ASR) by extracting meaningful features from input&#10;audio frames. Such features, however, may consist not only of information about&#10;the spoken language content, but also may contain information about unnecessary&#10;contexts such as background noise and sounds or speaker identity, accent, or&#10;protected attributes. Such information can directly harm generalization&#10;performance, by introducing spurious correlations between the spoken words and&#10;the context in which such words were spoken. In this work, we introduce an&#10;unsupervised, encoder-agnostic method for factoring speech-encoder&#10;representations into explicit content-encoding representations and spurious&#10;context-encoding representations. By doing so, we demonstrate improved&#10;performance on standard ASR benchmarks, as well as improved performance in both&#10;real-world and artificially noisy ASR scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.13191" label="2208.13191">
        <attvalues>
          <attvalue for="0" value="Towards Disentangled Speech Representations" />
          <attvalue for="1" value="  The careful construction of audio representations has become a dominant&#10;feature in the design of approaches to many speech tasks. Increasingly, such&#10;approaches have emphasized &quot;disentanglement&quot;, where a representation contains&#10;only parts of the speech signal relevant to transcription while discarding&#10;irrelevant information. In this paper, we construct a representation learning&#10;task based on joint modeling of ASR and TTS, and seek to learn a representation&#10;of audio that disentangles that part of the speech signal that is relevant to&#10;transcription from that part which is not. We present empirical evidence that&#10;successfully finding such a representation is tied to the randomness inherent&#10;in training. We then make the observation that these desired, disentangled&#10;solutions to the optimization problem possess unique statistical properties.&#10;Finally, we show that enforcing these properties during training improves WER&#10;by 24.5% relative on average for our joint modeling task. These observations&#10;motivate a novel approach to learning effective audio representations.&#10;" />
          <attvalue for="2" value="&#10;&#10;The recent success of unsupervised pretraining in language processing can be credited to the advent of sophisticated techniques for learning representations of text. Methods like ELMo \cite{Peters2018}, GPT \cite{Radford2018}, and BERT \cite{Devlin2018} work by using an unsupervised task that develops a representation of text that is useful for downstream tasks in a way that is agnostic to what that task is. &#10;&#10;In applying these lessons to unsupervised pretraining in speech great progress has been made with the discovery that in a data-intensive domain like audio, it is best to learn a representation that discards unimportant parts of the signal. Contrastive estimation \cite{Gutmann2010}, in which a full reconstruction is not learned, has yielded representations that achieve strong results in speaker identification and speech recognition \cite{Oord2018}. State-of-the-art methods combine contrastive learning with masked language modeling as in Wav2Vec 2.0 \cite{ Baevski2020} and Adaptive SpecAugment \cite{zhang20}.&#10;&#10;Such successes can be seen as signaling a movement away from task-agnostic representations and towards \say{lossy} representations, in which a model learns not only to summarize relevant portions of a signal but also to discard portions that are irrelevant to the downstream task. This distinction is particularly clear in the world of multi-modal representation learning, where we seek a representation specifically of the intersectionbetween two domains (e.g. audio and images \cite{Peri2021, Morgado2020} or audio and text \cite{Chung2019UnsupervisedLO}). However, while there are several natural methods for learning a representation that models components of a signal that are required for a task, it is difficult to craft a method that compels a model to specifically exclude irrelevant components. Approaches in this space have commonly relied on techniques like adversarial learning to exclude particular parts of a signal thought to be irrelevant, as in \cite{Wang2021adversarially}.&#10;&#10;In this study, we present a novel architecture specifically designed to learn a measurably disentangled representation of audio using supervised data. Our model is based on the paradigm of dual learning \cite{DualLearningBook, Xia2016, Wang2019MultiAgentDL, Wang2018DualTL}, which seeks to exploit the \say{duality} between ASR and TTS. Traditionally, this is done by training a model that performs both ASR and TTS with a shared encoder that is tasked with representing inputs from both the speech and text domains \cite{ren2020unsupervised, xu2020lrspeech}. Our model adds a secondary encoder, which is intended to capture specifically those parts of the audio signal that are irrelevant to the transcript. While the primary encoder is utilized for both ASR and TTS, this secondary encoder is used only for audio reconstruction, which is a task that requires both that part of the audio signal that predicts the transcript and the \say{residual} signal that does not. We argue that disentanglement is facilitated by the explicit modeling of the residual signal by the secondary encoder, and demonstrate this disentanglement by training a speaker-ID classifier on the outputs of both the primary and secondary encoders.&#10;&#10;Other studies have shown that in scenarios where more than one solution to an optimization problem is possible (such as generalized vs. overfit solutions \cite{Weber2018} and selection of significant units in a DNN \cite{Frankle2018}), the stochasticity of parameter initialization and minibatch selection can be decisive. We present empirical evidence that speech signal disentanglement is such a problem. We find that both entangled and disentangled solutions to our dual learning problem are possible, and that the superior, disentangled solution is arrived at randomly. We then observe that the disentangled solution has the unique statistical property of using a large amount of its variational capacity in both encoders. Finally, we show that enforcing this property during training with an additional loss term substantially improves ASR quality.&#10;&#10;Possible applications of our joint modeling task include refinement of back-transcription based semi-supervised learning systems such as speech chains \cite{Tjandra17} and Sequential MixMatch \cite{ZChen21}. We believe that our discoveries motivate the usage of a secondary encoder in such systems to achieve disentanglement in semi-supervised audio representations.&#10;&#10;The rest of this paper is structured as follows. Section \ref{sec:architecture} describes our architecture for a joint audio and text model that can simultaneously perform ASR, TTS, and audio and text reconstruction. Section \ref{sec:experiments} presents the design for our experiment investigating the nature of disentangled representations. Section \ref{sec:results} details the results of that experiment and observations of the statistical differences between entangled and disentangled representations. We summarize our findings and discuss future work in Section \ref{sec:conclusions}. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.03389" label="2311.03389">
        <attvalues>
          <attvalue for="0" value="Learning Disentangled Speech Representations" />
          <attvalue for="1" value="  Disentangled representation learning from speech remains limited despite its&#10;importance in many application domains. A key challenge is the lack of speech&#10;datasets with known generative factors to evaluate methods. This paper proposes&#10;SynSpeech: a novel synthetic speech dataset with ground truth factors enabling&#10;research on disentangling speech representations. We plan to present a&#10;comprehensive study evaluating supervised techniques using established&#10;supervised disentanglement metrics. This benchmark dataset and framework&#10;address the gap in the rigorous evaluation of state-of-the-art disentangled&#10;speech representation learning methods. Our findings will provide insights to&#10;advance this underexplored area and enable more robust speech representations.&#10;" />
          <attvalue for="2" value="&#10;Learning disentangled representations yields promising results in domains like computer vision, allowing models to robustly separate generative factors within data~\cite{locatello2019challenging,higgins2017beta,kim2018disentangling}. A disentangled representation independently captures the ``true'' generative factors of variation that explain the data. Such representations provide multiple benefits including enhanced predictive abilities on downstream tasks, decreased sample complexity, greater explainability, fairness, and a means to avoid shortcut learning~\cite{carbonneau2020measuring}. However, progress on learning disentangled speech representations has been limited, despite potential benefits such as improved understanding of speech signals, interpretable features, controllable generation, source separation, multilingual and cross-lingual speech processing, voice conversion, robustness to variability, few-shot learning, privacy-preserving, etc\cite{Qian2018UnsupervisedSL,chou2018multi}.&#10;&#10;A key challenge impeding advancement in this domain is the lack of suitable benchmarking datasets for quantitative evaluation and analysis~\cite{hsu2017unsupervised, mo2019semantic}. Real-world speech data often lack explicit ground truth annotation of generative factors. On the other hand, most synthetic datasets are not large or rich enough to benchmark speech disentanglement models~\cite{wang2018style,zhou2020comparison,jia2018transfer}. This contributes to difficulties in reproducible research and standardized comparison of methods.&#10;&#10;SynSpeech represents, to the best of our knowledge, the first large-scale synthetic speech dataset designed for benchmarking disentangled speech representation learning for both content and speaker characteristics. With its three distinct dataset versions, SynSpeech enabled a comprehensive analysis of representation generalization through LP and facilitated an empirical evaluation of disentanglement using supervised and unsupervised disentanglement metrics. This benchmark dataset provided fundamental insights, advancing understanding and progress in this previously underexplored area of speech representation.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.09224" label="2204.09224">
        <attvalues>
          <attvalue for="0" value="ContentVec: An Improved Self-Supervised Speech Representation by&#10;  Disentangling Speakers" />
          <attvalue for="1" value="  Self-supervised learning in speech involves training a speech representation&#10;network on a large-scale unannotated speech corpus, and then applying the&#10;learned representations to downstream tasks. Since the majority of the&#10;downstream tasks of SSL learning in speech largely focus on the content&#10;information in speech, the most desirable speech representations should be able&#10;to disentangle unwanted variations, such as speaker variations, from the&#10;content. However, disentangling speakers is very challenging, because removing&#10;the speaker information could easily result in a loss of content as well, and&#10;the damage of the latter usually far outweighs the benefit of the former. In&#10;this paper, we propose a new SSL method that can achieve speaker&#10;disentanglement without severe loss of content. Our approach is adapted from&#10;the HuBERT framework, and incorporates disentangling mechanisms to regularize&#10;both the teacher labels and the learned representations. We evaluate the&#10;benefit of speaker disentanglement on a set of content-related downstream&#10;tasks, and observe a consistent and notable performance advantage of our&#10;speaker-disentangled representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Voice Conversion&#10; Voice conversion is among the first research areas where speaker disentanglement is explored. The general trend follows the analysis-synthesis framework, where the analysis stage learns a speaker-independent speech representation that only preserves the content, and the synthesis stage uses the speaker-independent speech representation and the speaker-related variations to synthesize the conversion results.&#10; Much research focuses on learning better linguistic representations during the analysis stage and/or injecting speaker variations better during the synthesis stage. VAE-VC \cite{hsu2016voice} is an early attempt of directly using VAE for voice conversion. Afterward, \cite{Chou2018MultitargetVC} disentangles more speaker variations from the latent representation by discouraging the latent representation to be classified as the source speaker using an auxiliary speaker classifier on the latent representation. In contrast, ACVAE-VC \cite{kameoka2018acvae} indirectly encourages more speaker disentanglement by encouraging the conversion output to be correctly classified as the source speaker. Inspired by image style transfer, StarGAN-VC \cite{Kameoka2018StarGANVCNM}, StarGAN-VC2 \cite{kaneko2019stargan}, CycleGAN-VC \cite{Kaneko2018CycleGANVCNV}, and CycleGAN-VC2 \cite{Kaneko2019CycleganVC2IC} adapted StarGAN \cite{Choi2018StarGANUG} and CycleGAN \cite{Zhu2017UnpairedIT} respectively for voice conversion. AutoVC \cite{qian2019autovc} disentangles speakers and content by directly tuning the bottleneck dimensions of a vanilla autoencoder. The following AutoVC-F0 \cite{qian2020f0} improves pitch disentanglement by conditioning the synthesis stage on pitch representations. VoiceMixer \cite{Lee2021VoiceMixerAV} improves the content loss of AutoVC using similarity-based downsampling as the bottleneck. AdaIN-VC \cite{Chou2019OneshotVC} uses instance normalization to normalize out the speaker variations in the analysis stage, and AGAIN-VC \cite{Chen2021AgainVCAO} additionally uses an activation function to constrain the speaker variations from flowing into the synthesis stage. Instead of pursuing extreme speaker disentanglement, another slightly different track of research encourages the synthesis stage to use the supplied speaker variations by using partially disentangled content representations combined with speaker variations that are easier for the synthesis stage to utilize. SpeechSplit \cite{qian2020unsupervised}, AutoPST \cite{Qian2021GlobalPS}, and NANSY \cite{choi2021neural} perturb the speaker variations during the analysis stage to encourage the synthesis stage to use the supplied more stable speaker representations. In particular, \cite{polyak2021speech} and NANSY start with the self-supervised speech representations as the partially disentangled content representation.&#10;Self-supervised Learning in Speech&#10;Learning self-supervised speech representation usually encodes the speech feature into context representations followed by pretext tasks to extract content information, which mainly has two tracks. The first track is generative learning. \cite{Chung2019AnUA,Chung2020GenerativePF} uses Autoregresstive Predictive Coding (APC) for self-supervised representation learning. Mockingjay \cite{Liu2020MockingjayUS} learns speech representation by predicting the current frame given both the past and future contexts. TERA \cite{Liu2021TERASL} learns speech representation by reconstructing acoustic&#10;frames from their altered counterparts. DeCoAR 2.0 \cite{Ling2020DeCoAR2D} reconstructs the frames from their vector-quantized counterparts. \cite{Wang2020UnsupervisedPO} reconstructs masked frames. The second track is discriminative. \cite{Oord2018RepresentationLW} uses constrastive predictive coding to learn multi-modal represenations including speech. Wav2vec \cite{Schneider2019wav2vecUP} learns to predict the future samples from distractors. Wav2vec 2.0 \cite{Baevski2020wav2vec2A}, an end-to-end version of vq-wav2vec \cite{Baevski2020vqwav2vecSL}, learns to identify the true vq-quantized frame among the distractors using contrastive loss. \cite{Kharitonov2021DataAC} significantly improves CPC-based SSL with speech data augmentation. \cite{Zhang2020PushingTL} pushes the limits of SSL using noisy student training by giant Conformer models pre-trained using wav2vec 2.0. Hubert \cite{Hsu2021HuBERTSS} predicts masked frames pre-quantized using k-means. ILS-SSL \cite{Wang2021SelfSupervisedLF} further improves Hubert by adding masked prediction loss on intermediate layers. Besides, there are also research using multiple tasks \cite{Pascual2019LearningPS,Ravanelli2020MultiTaskSL,Chung2021W2vBERTCC} or using both labeled and unlabeled data \cite{Wang2020UnsupervisedPO}. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="1807.05520" label="1807.05520">
        <attvalues>
          <attvalue for="0" value="Deep Clustering for Unsupervised Learning of Visual Features" />
          <attvalue for="1" value="  Clustering is a class of unsupervised learning methods that has been&#10;extensively applied and studied in computer vision. Little work has been done&#10;to adapt it to the end-to-end training of visual features on large scale&#10;datasets. In this work, we present DeepCluster, a clustering method that&#10;jointly learns the parameters of a neural network and the cluster assignments&#10;of the resulting features. DeepCluster iteratively groups the features with a&#10;standard clustering algorithm, k-means, and uses the subsequent assignments as&#10;supervision to update the weights of the network. We apply DeepCluster to the&#10;unsupervised training of convolutional neural networks on large datasets like&#10;ImageNet and YFCC100M. The resulting model outperforms the current state of the&#10;art by a significant margin on all the standard benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.09882" label="2006.09882">
        <attvalues>
          <attvalue for="0" value="Unsupervised Learning of Visual Features by Contrasting Cluster&#10;  Assignments" />
          <attvalue for="1" value="  Unsupervised image representations have significantly reduced the gap with&#10;supervised pretraining, notably with the recent achievements of contrastive&#10;learning methods. These contrastive methods typically work online and rely on a&#10;large number of explicit pairwise feature comparisons, which is computationally&#10;challenging. In this paper, we propose an online algorithm, SwAV, that takes&#10;advantage of contrastive methods without requiring to compute pairwise&#10;comparisons. Specifically, our method simultaneously clusters the data while&#10;enforcing consistency between cluster assignments produced for different&#10;augmentations (or views) of the same image, instead of comparing features&#10;directly as in contrastive learning. Simply put, we use a swapped prediction&#10;mechanism where we predict the cluster assignment of a view from the&#10;representation of another view. Our method can be trained with large and small&#10;batches and can scale to unlimited amounts of data. Compared to previous&#10;contrastive methods, our method is more memory efficient since it does not&#10;require a large memory bank or a special momentum network. In addition, we also&#10;propose a new data augmentation strategy, multi-crop, that uses a mix of views&#10;with different resolutions in place of two full-resolution views, without&#10;increasing the memory or compute requirements much. We validate our findings by&#10;achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as&#10;surpassing supervised pretraining on all the considered transfer tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.11588" label="2011.11588">
        <attvalues>
          <attvalue for="0" value="The Zero Resource Speech Benchmark 2021: Metrics and baselines for&#10;  unsupervised spoken language modeling" />
          <attvalue for="1" value="  We introduce a new unsupervised task, spoken language modeling: the learning&#10;of linguistic representations from raw audio signals without any labels, along&#10;with the Zero Resource Speech Benchmark 2021: a suite of 4 black-box, zero-shot&#10;metrics probing for the quality of the learned models at 4 linguistic levels:&#10;phonetics, lexicon, syntax and semantics. We present the results and analyses&#10;of a composite baseline made of the concatenation of three unsupervised&#10;systems: self-supervised contrastive representation learning (CPC), clustering&#10;(k-means) and language modeling (LSTM or BERT). The language models learn on&#10;the basis of the pseudo-text derived from clustering the learned&#10;representations. This simple pipeline shows better than chance performance on&#10;all four metrics, demonstrating the feasibility of spoken language modeling&#10;from raw speech. It also yields worse performance compared to text-based&#10;'topline' systems trained on the same data, delineating the space to be&#10;explored by more sophisticated end-to-end models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Zero Resource Speech Challenge Series. Previous work \cite{versteegh2015zero, dunbar2017zero, dunbar2019zero,dunbar2020} has focused on establishing benchmarks for unsupervised learning of an entire dialogue system, but has so far remained at a rather low level (acoustic, lexical). Acoustic modeling has used two metrics: ABX, a distance-based metric to be discussed later, and opinion scores on TTS output (whereby the discovered units are used to resynthesize speech). As for the lexical level, past work has focused on using the NLP metrics developed for word segmentation \cite{ludusan2014bridging}. However, these metrics assume that the models should discover words explicitly. The success of character-based language models suggests that it is possible to learn high-level linguistic concepts without explicitly segmenting words \cite[see][]{hahn2019}. &#10;&#10;Black box NLP.&#10;Among the variety of black-box linguistic tasks, psycholinguistically-inspired ones enable direct comparison of models and humans. Grammaticality judgments for recurrent networks have been investigated since \cite{allen1999:emergence}, who use closely matched pairs of sentences to investigate grammatical correctness. This approach has recently been adopted to assess the abilities of RNNs, and LSTMs in particular, in capturing syntactic structures. For instance, \cite{linzen:2016} and \cite{gulordava:2018} use word probes in minimally different pairs of English sentences to study number agreement. To discriminate grammatical sentences from ungrammatical ones, they retrieve the probabilities of the possible morphological forms of a target word, given the probability of the previous words in the sentence. Practically, in the sentence ``the boy \underline{is} sleeping'', they assume the network has detected number agreement if $\mathbf{P}(w = is) &gt; \mathbf{P}(w = are)$. This methodology has also been adapted by \cite{goldberg:2019} to models trained with a masked language-modeling objective. Similarly, \cite{ravfogel2018can} use word probes to examine whether LSTMs understand Basque agreement and \cite{godais2017charnlm} to test the lexical level in character-based LM. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="1609.02907" label="1609.02907">
        <attvalues>
          <attvalue for="0" value="Semi-Supervised Classification with Graph Convolutional Networks" />
          <attvalue for="1" value="  We present a scalable approach for semi-supervised learning on&#10;graph-structured data that is based on an efficient variant of convolutional&#10;neural networks which operate directly on graphs. We motivate the choice of our&#10;convolutional architecture via a localized first-order approximation of&#10;spectral graph convolutions. Our model scales linearly in the number of graph&#10;edges and learns hidden layer representations that encode both local graph&#10;structure and features of nodes. In a number of experiments on citation&#10;networks and on a knowledge graph dataset we demonstrate that our approach&#10;outperforms related methods by a significant margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.10903" label="1710.10903">
        <attvalues>
          <attvalue for="0" value="Graph Attention Networks" />
          <attvalue for="1" value="  We present graph attention networks (GATs), novel neural network&#10;architectures that operate on graph-structured data, leveraging masked&#10;self-attentional layers to address the shortcomings of prior methods based on&#10;graph convolutions or their approximations. By stacking layers in which nodes&#10;are able to attend over their neighborhoods' features, we enable (implicitly)&#10;specifying different weights to different nodes in a neighborhood, without&#10;requiring any kind of costly matrix operation (such as inversion) or depending&#10;on knowing the graph structure upfront. In this way, we address several key&#10;challenges of spectral-based graph neural networks simultaneously, and make our&#10;model readily applicable to inductive as well as transductive problems. Our GAT&#10;models have achieved or matched state-of-the-art results across four&#10;established transductive and inductive graph benchmarks: the Cora, Citeseer and&#10;Pubmed citation network datasets, as well as a protein-protein interaction&#10;dataset (wherein test graphs remain unseen during training).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.07953" label="1905.07953">
        <attvalues>
          <attvalue for="0" value="Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph&#10;  Convolutional Networks" />
          <attvalue for="1" value="  Graph convolutional network (GCN) has been successfully applied to many&#10;graph-based applications; however, training a large-scale GCN remains&#10;challenging. Current SGD-based algorithms suffer from either a high&#10;computational cost that exponentially grows with number of GCN layers, or a&#10;large space requirement for keeping the entire graph and the embedding of each&#10;node in memory. In this paper, we propose Cluster-GCN, a novel GCN algorithm&#10;that is suitable for SGD-based training by exploiting the graph clustering&#10;structure. Cluster-GCN works as the following: at each step, it samples a block&#10;of nodes that associate with a dense subgraph identified by a graph clustering&#10;algorithm, and restricts the neighborhood search within this subgraph. This&#10;simple but effective strategy leads to significantly improved memory and&#10;computational efficiency while being able to achieve comparable test accuracy&#10;with previous algorithms. To test the scalability of our algorithm, we create a&#10;new Amazon2M data with 2 million nodes and 61 million edges which is more than&#10;5 times larger than the previous largest publicly available dataset (Reddit).&#10;For training a 3-layer GCN on this data, Cluster-GCN is faster than the&#10;previous state-of-the-art VR-GCN (1523 seconds vs 1961 seconds) and using much&#10;less memory (2.2GB vs 11.2GB). Furthermore, for training 4 layer GCN on this&#10;data, our algorithm can finish in around 36 minutes while all the existing GCN&#10;training algorithms fail to train due to the out-of-memory issue. Furthermore,&#10;Cluster-GCN allows us to train much deeper GCN without much time and memory&#10;overhead, which leads to improved prediction accuracy---using a 5-layer&#10;Cluster-GCN, we achieve state-of-the-art test F1 score 99.36 on the PPI&#10;dataset, while the previous best result was 98.71 by [16]. Our codes are&#10;publicly available at&#10;https://github.com/google-research/google-research/tree/master/cluster_gcn.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.10044" label="2109.10044">
        <attvalues>
          <attvalue for="0" value="Something Old, Something New: Grammar-based CCG Parsing with Transformer&#10;  Models" />
          <attvalue for="1" value="  This report describes the parsing problem for Combinatory Categorial Grammar&#10;(CCG), showing how a combination of Transformer-based neural models and a&#10;symbolic CCG grammar can lead to substantial gains over existing approaches.&#10;The report also documents a 20-year research program, showing how NLP methods&#10;have evolved over this time. The staggering accuracy improvements provided by&#10;neural models for CCG parsing can be seen as a reflection of the improvements&#10;seen in NLP more generally. The report provides a minimal introduction to CCG&#10;and CCG parsing, with many pointers to the relevant literature. It then&#10;describes the CCG supertagging problem, and some recent work from Tian et al.&#10;(2020) which applies Transformer-based models to supertagging with great&#10;effect. I use this existing model to develop a CCG multitagger, which can serve&#10;as a front-end to an existing CCG parser. Simply using this new multitagger&#10;provides substantial gains in parsing accuracy. I then show how a&#10;Transformer-based model from the parsing literature can be combined with the&#10;grammar-based CCG parser, setting a new state-of-the-art for the CCGbank&#10;parsing task of almost 93% F-score for labelled dependencies, with complete&#10;sentence accuracies of over 50%.&#10;" />
          <attvalue for="2" value="&#10;&#10;Combinatory Categorial Grammar (CCG) is a lexicalized grammar formalism in the type-driven tradition, building on historical work by \cite{ajdukiewicz:35} and \cite{bar-hillel:53}. The original formalism, often referred to as classical categorial grammar, uses the rules of forward and backward application to combine the categorial types. CCG, developed over many years by Mark Steedman \cite{steedman:2000}, uses a number of additional combinatory rules to deal with ``movement&quot; phenomena in natural languages -- syntactic environments in which phrases are moved from their canonical argument positions, often creating an unbounded dependency between the argument and predicate. Examples in English include questions and relative clause extraction \cite{rimell-etal-2009-unbounded}. This movement phenomena is what motivated Chomsky to develop transformational grammar \cite{chomsky:65}. Unlike transformational grammar, however, CCG is a ``monostratal&quot; theory in which the apparent movement of syntactic units is handled by a single level of representation. Other approaches to categorial grammar include the type-logical approach \cite{moortgat:97}, in which linguistic types are the formulas of a logic and derivations are proofs, and the algebraic approach of the later work of Lambek \cite{lambek:08}, in which linguistic types are the partially-ordered objects of an algebra (specifically a pregroup), and derivations are given by the partial order.&#10;&#10;Figure~\ref{fig:application_deriv} gives an example CCG derivation using only the basic rules of forward ($&gt;$) and backward ($&lt;$) application. The categorial types that are assigned to words at the leaves of the derivation are referred to as lexical categories. The internal structure of categories is built recursively from atomic categories and slashes (`\textbackslash', `/') which indicate the directions of arguments. In a typical CCG grammar there are only a small number of atomic categories, such as $S$ for sentence, $N$ for noun, $NP$ for noun phrase, and $PP$ for prepositional phrase. However, the recursive combination of categories and slashes can lead to a large number of categories; for example, the grammar used in the parsing experiments below has around 1,300 lexical categories. CCG is referred to as lexicalised because most of the grammatical information---which is language-dependent and encoded in the lexical categories---resides in the lexicon, with the remainder of the grammar being provided by a small number of combinatory rules.&#10;&#10;One way to think of the application of these rules, or rule schema (since they apply to an unbounded set of category pairs), is that the matching parts of the combining categories effectively cancel, leading to the rules being called cancellation laws in some of the earlier work on categorial grammar. For example, when the lexical categories for Exchange and Commission in Figure~\ref{fig:application_deriv} are combined, the argument $N$ required by Exchange in $N/N$ cancels with the lexical category $N$ for Commission. We can also think of $N/N$ as a function that is applied to its argument $N$. The forward in forward application refers to the fact that the argument is to the right. Backward ($&lt;$) application---for when the argument is to the left---is used in the example when combining the subject $NP$ Investors with the derived verb phrase $S[dcl]\bs NP$.&#10;&#10;Figure~\ref{fig:fcomp} shows the derivation for a noun phrase containing a relative clause, where the object has been extracted out of its canonical position to the right of the transitive verb. The bracketing structure of the lexical category of the transitive verb ($(S[dcl]\bs NP)/NP$) means that the verb is expecting to combine with its object to the right before its subject to the left. However, in this example the object has been moved away from the verb so that is not possible. The solution provided by CCG is to use two new combinatory rules. First, the unary rule of type-raising (${&gt;}\mathbf{T}$) turns an atomic $NP$ category into a complex category $S/(S\bs NP)$. A useful way to think about this new category is that it's a sentence missing a verb phrase ($S\bs NP$) to the right, which is a natural way to conceive of a subject $NP$ as a function. Second, the rule of forward composition (${&gt;}\mathbf{B}$) enables the combination of the type-raised noun phrase ($S/(S\bs NP)$) and the transitive verb ($S[dcl]\bs NP)/NP$), again with the idea that the verb-phrase categories ``in the middle&quot; effectively cancel. This results in the slightly unusual constituent $S[dcl]/NP$, which reflects the fact that the linguistic unit the fund reached is a sentence missing an $NP$ to its right. Note that the lexical category for the relative pronoun in this example ($(NP\bs NP)/(S[dcl]/NP)$) is expecting such a constituent to its right, so the relative pronoun can combine with the derived category using forward application.&#10;&#10;There are additional combinatory rules in CCG which are designed to deal with other linguistic phenomena, including some rules in which the main slashes of the combining categories point in different directions -- the so-called ``non-harmonic&quot; or crossing rules, such as backward crossed composition. These are all based on the operators of combinatory logic&#10;\cite{curry:1958}; hence the term combinatory in Combinatory Categorial Grammar. \cite{steedman:96}, \cite{steedman:2000} and \cite{baldridge:thesis02} contain many linguistic examples which motivate the particular set of rules in the theory.&#10;&#10;There is much work on the formal properties of CCG, including the seminal papers of Vijay-Shanker, Weir and Joshi in which it was proven that CCG is strictly more powerful than context-free grammars, but substantially less powerful than context-sensitive grammars -- hence the term mildly context-senstive \cite{weir:92}. \cite{joshi:91} prove that CCG is weakly equivalent---i.e. generating the same string sets---to Tree Adjoining Grammar, Head Grammar, and Linear Indexed Grammar. This was a remarkable result given the apparent differences between these formalisms. Tree Adjoining Grammar \cite{joshi:87}, like CCG, has become a standard grammar formalism in Computational Linguistics and has formed the basis for much experimental work in developing parsers and NLP systems \cite{kasai-etal-2018-end}. \cite{kuhlmann-etal-2015-lexicalization} build on the earlier formal work and show that there are versions of CCG that are more powerful than CFGs, but strictly less powerful than TAG.&#10;Despite the additional power of CCG (and TAG), there are still efficient parsing algorithms for CCG (and TAG) which are polynomial in the length of the input sentence \cite{vijay:93,kuhlmann-etal-2018-complexity}. &#10;&#10;The mildly context-sensitive nature of CGG is much trumpeted, and rightly so given that it enables analyses of the crossing dependencies in Dutch and Swiss German \cite{shieber:85}. However, it is perhaps worth pointing out that, for practical CCG parsing of English at least, the successful parsers have either used a CCG grammar which is context free by construction, being built entirely from rules instances observed in a finite CCG treebank \cite{hock:acl02,fowler-penn-2010-accurate}, or a grammar which is context free in practice by limiting the applicability of the combinatory rules to the rule instances in the treebank \cite{clark-curran-2007-wide}. Hence the parsing algorithms used by practical CCG parsers tend not to exploit the (somewhat complicated) structure-sharing schemes which define the more general polynomial-time parsing algorithms referenced above.&#10;&#10;The remainder of this report starts out with the CCG supertagging task (Section~\ref{sec:supertagging}), showing the 20-year evolution of CCG supertagging from feature-based models in which the features are defined by hand, to neural models in which the features are induced automatically by a neural network. Section~\ref{sec:parsing} then demonstrates the gains that can be obtained by simply using a neural CCG supertagger as a front-end to an existing CCG parser, as well as additional improvements from using a neural classifier for the parsing model itself. Note that much of this report is a survey of existing work carried out by other researchers---or at least existing work replicated by the author---with the new material appearing in Section~\ref{sec:neural_parsing}, which reports new state-of-the-art accuracy figures for the CCGbank parsing task. It also acts as something of a survey of the 20-year wide-coverage CCG parsing project that began in Edinburgh. For a more detailed exposition of the linguistic theory of CCG, the reader is referred to \cite{steedman:96}, \cite{steedman:2000} and \cite{baldridge:thesis02}. For an introduction to wide-coverage CCG parsing, the reader is referred to \cite{clark-curran-2007-wide} and \cite{hockenmaier-steedman-2007-ccgbank}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Network Models, Artificial Intelligence, Mathematics, Natural Language Processing, Combinatory Categorial Grammar" />
        </attvalues>
      </node>
      <node id="1804.06610" label="1804.06610">
        <attvalues>
          <attvalue for="0" value="End-to-end Graph-based TAG Parsing with Neural Networks" />
          <attvalue for="1" value="  We present a graph-based Tree Adjoining Grammar (TAG) parser that uses&#10;BiLSTMs, highway connections, and character-level CNNs. Our best end-to-end&#10;parser, which jointly performs supertagging, POS tagging, and parsing,&#10;outperforms the previously reported best results by more than 2.2 LAS and UAS&#10;points. The graph-based parsing architecture allows for global inference and&#10;rich feature representations for TAG parsing, alleviating the fundamental&#10;trade-off between transition-based and graph-based parsing systems. We also&#10;demonstrate that the proposed parser achieves state-of-the-art performance in&#10;the downstream tasks of Parsing Evaluation using Textual Entailments (PETE) and&#10;Unbounded Dependency Recovery. This provides further support for the claim that&#10;TAG is a viable formalism for problems that require rich structural analysis of&#10;sentences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1702.06594" label="1702.06594">
        <attvalues>
          <attvalue for="0" value="On the Complexity of CCG Parsing" />
          <attvalue for="1" value="  We study the parsing complexity of Combinatory Categorial Grammar (CCG) in&#10;the formalism of Vijay-Shanker and Weir (1994). As our main result, we prove&#10;that any parsing algorithm for this formalism will take in the worst case&#10;exponential time when the size of the grammar, and not only the length of the&#10;input sentence, is included in the analysis. This sets the formalism of&#10;Vijay-Shanker and Weir (1994) apart from weakly equivalent formalisms such as&#10;Tree-Adjoining Grammar (TAG), for which parsing can be performed in time&#10;polynomial in the combined size of grammar and input sentence. Our results&#10;contribute to a refined understanding of the class of mildly context-sensitive&#10;grammars, and inform the search for new, mildly context-sensitive versions of&#10;CCG.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.03337" label="2106.03337">
        <attvalues>
          <attvalue for="0" value="Summary Grounded Conversation Generation" />
          <attvalue for="1" value="  Many conversation datasets have been constructed in the recent years using&#10;crowdsourcing. However, the data collection process can be time consuming and&#10;presents many challenges to ensure data quality. Since language generation has&#10;improved immensely in recent years with the advancement of pre-trained language&#10;models, we investigate how such models can be utilized to generate entire&#10;conversations, given only a summary of a conversation as the input. We explore&#10;three approaches to generate summary grounded conversations, and evaluate the&#10;generated conversations using automatic measures and human judgements. We also&#10;show that the accuracy of conversation summarization can be improved by&#10;augmenting a conversation summarization dataset with generated conversations.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic conversation systems require large quantities of data to learn task specific language patterns and underlying conversation policies. Such data either come from human-to-human conversation logs \cite{lowe2015ubuntu, hardalov2018towards} or is collected in crowd-sourced environments, where two or more crowd-workers play specific roles under some guidelines &#10;\cite{zhang2018personalizing, budzianowski2018multiwoz}. Since real human-to-human conversation logs are scarce, many datasets have been created using the latter approach. However, crowd-sourced conversation data collection is time consuming, costly and presents multiple challenges to ensure data quality \cite{kang2018data}.&#10;&#10;Conversation summarization is an emerging research area that has been ill-studied due to the lack of large-scale datasets. Most existing public datasets in this domain are small, for example, AMI meeting corpus \cite{mccowan2005ami} contains $137$ summary transcripts. CRD3 \cite{rameshkumar2020storytelling} is a spoken conversation dataset that consists of $159$ conversations and summaries. Samsum \cite{gliwa2019samsum}, the only large scale dataset for conversation summarization, contains over $16,000$ open-domain conversations and summaries created artificially by humans. &#10;&#10;Large scale pre-trained language models (PLMs) \cite{lewis2020bart, brown2020language, raffel2020exploring} have been used in various text generation tasks \cite{budzianowski2019hello, min2020ambigqa, cachola2020tldr}. In recent studies, PLMs are used to generate training data for natural language processing (NLP) applications. For example,~\cite{anaby2020not, yang2020g} use PLMs to create paraphrases for intent classifiers in conversation systems, and show that, when the original datasets are augmented with the generated data, performance improves. More recently~\cite{mohapatra2020simulated} generated entire conversations grounded on instructions that are provided to crowd-workers using a modular approach, where different PLMs are trained for different roles. &#10;&#10;Our Contributions: &#10;We investigate how PLMs can be utilized to generate entire conversations that are grounded on a given summary. We explore three approaches:&#10;(1) Supervised Learning (SL) based conversation generation (SL-Gen): where, a PLM is trained to generate an entire conversation, taking the summary of a conversation as input, (2) Reinforced Learning (RL) based conversation generation (RL-Gen): where, we further improve the SL-Gen method using the quality of the generated conversations as a reward, and (3) Controlled turn-by-turn conversation generation (CN-Gen): which allows us to generate conversations turn-by-turn, constrained on the summary and a set of pre-defined control parameters. &#10;We evaluate the quality of the generated conversations by conducting automatic and human evaluation. We also show that once a conversation summarization dataset is augmented with the generated conversations, the performance of the downstream summarization task is improved.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Summarization Dataset Augmentation, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Natural Language Processing, Conversation Data Generation" />
        </attvalues>
      </node>
      <node id="2011.04946" label="2011.04946">
        <attvalues>
          <attvalue for="0" value="When Do You Need Billions of Words of Pretraining Data?" />
          <attvalue for="1" value="  NLP is currently dominated by general-purpose pretrained language models like&#10;RoBERTa, which achieve strong performance on NLU tasks through pretraining on&#10;billions of words. But what exact knowledge or skills do Transformer LMs learn&#10;from large-scale pretraining that they cannot learn from less data? We adopt&#10;four probing methods---classifier probing, information-theoretic probing,&#10;unsupervised relative acceptability judgment, and fine-tuning on NLU&#10;tasks---and draw learning curves that track the growth of these different&#10;measures of linguistic ability with respect to pretraining data volume using&#10;the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B&#10;words. We find that LMs require only about 10M or 100M words to learn&#10;representations that reliably encode most syntactic and semantic features we&#10;test. A much larger quantity of data is needed in order to acquire enough&#10;commonsense knowledge and other skills required to master typical downstream&#10;NLU tasks. The results suggest that, while the ability to encode linguistic&#10;features is almost certainly necessary for language understanding, it is likely&#10;that other forms of knowledge are the major drivers of recent improvements in&#10;language understanding among large pretrained models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Probing neural network representations has been an active area of research in recent years \cite{rogers2020primer,Belinkov2019AnalysisMI}. With the advent of large pretrained Transformers like BERT \cite{devlin2019bert}, numerous papers have used classifier probes methods to attempt to locate linguistic features in learned representations with striking positive results \cite{tenney2019you,hewitt2019structural}. However, another thread has found problems with many probing methods: Classifier probes can learn too much from training data \cite{hewitt2019designing} and can fail to distinguish between features that are extractable and features that are actually used \cite{voita2020informationtheoretic,pimentel2020information,elazar2020amnesic}. Moreover, it is advisable to look to a variety of probing methods, as different probing methods often yield contradictory results \cite{warstadt2019investigating}. &#10;&#10;There have also been a few earlier studies investigating the relationship between pretraining data volume and linguistic knowledge in language models. Studies of unsupervised acceptability judgments find fairly consistent evidence of rapid improvements in linguistic knowledge up to about 10M words of pretraining data, after which improvements slow down for most phenomena. \cite{vanschijndel2019quantity} find large improvements in knowledge of subject-verb agreement and reflexive binding up to 10M words, and few improvements between 10M and 80M words. \cite{hu-etal-2020-systematic} find that GPT-2 trained on 42M words performs roughly as well on a syntax benchmark as a similar model trained on 100 times that amount. Other studies have investigate how one model's linguistic knowledge changes during the training process, as a function of the number of updates \cite{saphra2019understanding,chiang2020pretrained}.&#10;&#10;\cite{JMLR:v21:20-074} also investigate how performance on SuperGLUE (and other downstream tasks) improves with pretraining dataset sizes between about 8M and 34B words. In contrast to our findings, they find that models with around 500M words of pretraining data can perform similarly on downstream tasks to models with 34B words. This discrepancy may arise from several factors. First, the architecture and pretraining for their T5 model is not identical to RoBERTa's or the MiniBERTas'. Second, they pretrain their models for a fixed number of iterations (totaling 34B tokens), whereas the miniBERTas were trained with early stopping. Nonetheless, this result suggests that the number of unique tokens might matter less than the number of iterations, within reasonable limits.&#10;&#10;There is also some recent work that investigates the effect of pretraining data size of other languages. \cite{micheli-etal-2020-importance} pretrain BERT-based language models on 10MB, 100MB, 500MB, 1000MB, 2000MB, and 4000MB of French text and test them on a question answering task. They find that the French MLM pretrained on 100MB of raw text has similar performance to the ones pretrained on larger datasets on the task, and that corpus-specific self-supervised learning does not make a significant difference. \cite{martin-etal-2020-camembert} also show that French MLMs can already learn a lot from small-scale pretraining.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Training, Linguistic Feature Encoding, Natural Language Understanding, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2105.05535" label="2105.05535">
        <attvalues>
          <attvalue for="0" value="OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and&#10;  Robustness for Lexical Complexity Prediction" />
          <attvalue for="1" value="  We propose an ensemble model for predicting the lexical complexity of words&#10;and multiword expressions (MWEs). The model receives as input a sentence with a&#10;target word or MWEand outputs its complexity score. Given that a key challenge&#10;with this task is the limited size of annotated data, our model relies on&#10;pretrained contextual representations from different state-of-the-art&#10;transformer-based language models (i.e., BERT and RoBERTa), and on a variety of&#10;training methods for further enhancing model generalization and&#10;robustness:multi-step fine-tuning and multi-task learning, and adversarial&#10;training. Additionally, we propose to enrich contextual representations by&#10;adding hand-crafted features during training. Our model achieved competitive&#10;results and ranked among the top-10 systems in both sub-tasks.&#10;" />
          <attvalue for="2" value="&#10;Predicting the difficulty of a word in a given context is useful in many natural language processing (NLP) applications such as lexical simplification. Previous efforts \cite{paetzold2016semeval,yimam2018report,zampieri2017complex} have focused on framing this as a binary classification task, which might not be ideal, since a word close to the decision&#10;boundary is assumed to be just as complex as one further&#10;away \cite{shardlow2020complex}. To alleviate this issue, SemEval-2021 Task 1 \cite{shardlow2021semeval} formulates this task as a regression task, where a model should predict the complexity value of words (Subtask 1) and MWEs (Subtask 2) in context. &#10;&#10;This paper describes the system developed by the Ochadai-Kyoto team for SemEval-2021 Task 1. Given that a key challenge in this task is the limited size of annotated data, we follow best practices from recent work on enhancing model generalization and robustness, and propose a model ensemble that leverages pretrained representations (i.e. BERT and RoBERTa), multi-step fine-tuning, multi-task learning and adversarial training. Additionally, we propose to enrich contextual representations by incorporating hand-crafted features during training. Our model ranked 7th out of 54 participating teams on Subtask 1, and 8th out of 37 teams on Subtask 2, obtaining Pearson correlation scores of 0.7772 and 0.8438, respectively.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Lexical Complexity Analysis, Linguistics, Computational Linguistics, Artificial Intelligence, Language Model Development, Natural Language Processing, Ensemble Modeling Techniques" />
        </attvalues>
      </node>
      <node id="1804.09132" label="1804.09132">
        <attvalues>
          <attvalue for="0" value="A Report on the Complex Word Identification Shared Task 2018" />
          <attvalue for="1" value="  We report the findings of the second Complex Word Identification (CWI) shared&#10;task organized as part of the BEA workshop co-located with NAACL-HLT'2018. The&#10;second CWI shared task featured multilingual and multi-genre datasets divided&#10;into four tracks: English monolingual, German monolingual, Spanish monolingual,&#10;and a multilingual track with a French test set, and two tasks: binary&#10;classification and probabilistic classification. A total of 12 teams submitted&#10;their results in different task/track combinations and 11 of them wrote system&#10;description papers that are referred to in this report and appear in the BEA&#10;workshop proceedings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.00473" label="2106.00473">
        <attvalues>
          <attvalue for="0" value="SemEval-2021 Task 1: Lexical Complexity Prediction" />
          <attvalue for="1" value="  This paper presents the results and main findings of SemEval-2021 Task 1 -&#10;Lexical Complexity Prediction. We provided participants with an augmented&#10;version of the CompLex Corpus (Shardlow et al 2020). CompLex is an English&#10;multi-domain corpus in which words and multi-word expressions (MWEs) were&#10;annotated with respect to their complexity using a five point Likert scale.&#10;SemEval-2021 Task 1 featured two Sub-tasks: Sub-task 1 focused on single words&#10;and Sub-task 2 focused on MWEs. The competition attracted 198 teams in total,&#10;of which 54 teams submitted official runs on the test data to Sub-task 1 and 37&#10;to Sub-task 2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.20335" label="2405.20335">
        <attvalues>
          <attvalue for="0" value="Xwin-LM: Strong and Scalable Alignment Practice for LLMs" />
          <attvalue for="1" value="  In this work, we present Xwin-LM, a comprehensive suite of alignment&#10;methodologies for large language models (LLMs). This suite encompasses several&#10;key techniques, including supervised finetuning (SFT), reward modeling (RM),&#10;rejection sampling finetuning (RS), and direct preference optimization (DPO).&#10;The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned&#10;with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn&#10;preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward&#10;models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B&#10;parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt&#10;is linked to 64 unique responses generated by Xwin-LM-SFT and scored by&#10;Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses&#10;from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the&#10;DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate&#10;consistent and significant improvements across the pipeline, demonstrating the&#10;strength and scalability of Xwin-LM. The repository&#10;https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster&#10;community research.&#10;" />
          <attvalue for="2" value="&#10;Recent advances in artificial intelligence, epitomized by large language models (LLMs) such as GPT-4~\cite{gpt4} and Claude~\cite{claude3}, have demonstrated remarkable capabilities across diverse real-world applications. Ensuring these models align with human expectations and values is crucial, especially as they are integrated into and utilized across numerous applications~\cite{instructGPT,bai2022training}.&#10;&#10;To achieve this alignment, the technique of Reinforcement Learning from Human/AI Feedback (RLHF/RLAIF)~\cite{rlhf, rlaif} has been proposed. This approach involves initially gathering preferences from human or AI sources, followed by optimizing a policy model against a clearly built Reward Model (RM)~\cite{instructGPT} or an implicit preference learning target~\cite{dpo}. While effective, the inherent complexity and high costs pose significant barriers, limiting extensive exploration within the research community.&#10;&#10;In this work, we develop and release a strong and scalable RLHF pipeline named Xwin-LM. We detail our approach for developing Xwin-LM, which includes supervised finetuning, preference annotation, reward modeling, and policy optimization, along with observations and insights associated with each step. Specifically, we start with pretrained models Llama-2~\cite{llama2}, a collection of prompts, and a well-trained annotator, GPT-4. First, we train our supervised learning models, Xwin-LM-SFT, using an instruction-following dataset annotated by GPT-4 to establish an initial capability as a cold start. We then collect a preference dataset, Xwin-Pair, where responses are sampled from Xwin-LM-SFT and preferences are labeled by GPT-4. This dataset is used to train the reward model, Xwin-RM. Next, we use Xwin-LM-SFT to sample multiple responses for each prompt from another set and employ Xwin-RM to rank these responses, creating a multiwise preference dataset named Xwin-Set. Subsequently, Xwin-LM-RS is obtained by applying the rejection sampling (RS) finetuning technique to the highest-scoring responses in Xwin-Set. Finally, beyond learning only from positive samples, Xwin-LM-DPO employs the direct preference optimization technique, involving negative samples in Xwin-Set to learn from unexpected behavior.&#10;&#10;We evaluate Xwin-LM on two popular instruction-following benchmarks, AlpacaEval~\cite{alpacaeval} and MT-bench~\cite{vicuna}. Fig.~\ref{fig:advertisement} illustrates the performance evolution of Xwin-LM throughout our pipeline. It is evident that Xwin-LM-SFT achieves a satisfactory cold start, and subsequent rejection sampling finetuning and direct preference optimization steps significantly improve model performance, indicating the strength of the proposed pipeline. Our Xwin-LM achieves state-of-the-art performance among all Llama2-based models.&#10;&#10;In addition to the strong results, we have also gleaned several observations and insights associated with the pipeline: &#10;\begin{enumerate}[(1)] &#10; \item The model's upper capability limit remains fairly constant during RLHF; performance gains are mainly due to enhanced stability in generating high-quality responses. Specifically, we observe that performance on two benchmarks and the RM score on our validation set under the best-of-1 protocol improved steadily, while those under the best-of-64 protocol remained fairly constant. &#10; \item For SFT, a linear enhancement in performance hinges on an exponential increase in data scale. Furthermore, as the data scale continues to increase, performance gradually approaches saturation. &#10; \item Best-of-n evaluation is a discriminative metric for evaluating RMs and can also be an indicator for probing the potential optimization upper bound for alignment. &#10; \item The DPO algorithm shows a certain sensitivity to the dispreferred responses within the data pair. We find that the dispreferred response should closely match the policy's output distribution. &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Preference Dataset Development, Linguistics, Cognitive Science, Large-Scale Model Optimization, Mathematics, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="2406.18972" label="2406.18972">
        <attvalues>
          <attvalue for="0" value="Applying LLMs for Rescoring N-best ASR Hypotheses of Casual&#10;  Conversations: Effects of Domain Adaptation and Context Carry-over" />
          <attvalue for="1" value="  Large language models (LLMs) have been successfully applied for rescoring&#10;automatic speech recognition (ASR) hypotheses. However, their ability to&#10;rescore ASR hypotheses of casual conversations has not been sufficiently&#10;explored. In this study, we reveal it by performing N-best ASR hypotheses&#10;rescoring using Llama2 on the CHiME-7 distant ASR (DASR) task. Llama2 is one of&#10;the most representative LLMs, and the CHiME-7 DASR task provides datasets of&#10;casual conversations between multiple participants. We investigate the effects&#10;of domain adaptation of the LLM and context carry-over when performing N-best&#10;rescoring. Experimental results show that, even without domain adaptation,&#10;Llama2 outperforms a standard-size domain-adapted Transformer-LM, especially&#10;when using a long context. Domain adaptation shortens the context length needed&#10;with Llama2 to achieve its best performance, i.e., it reduces the computational&#10;cost of Llama2.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs),&#10;such as GPT-4 \cite{OpenAI_arXiv2023},&#10;PaLM2 \cite{Google_arXiv2023},&#10;and Llama2&#10;(Large Language Model META AI) \cite{Touvron_arXiv2023a},&#10;have now become a prominent component in modern natural language processing&#10;(NLP)&#10;and are&#10;successfully utilized in various NLP tasks,&#10;such as machine translation, text summarization, and question answering.&#10;Recently,&#10;they have been&#10;used&#10;not only in NLP tasks&#10;but also in speech-related tasks,&#10;including automatic speech recognition (ASR).&#10;A simple&#10;way to utilize LLMs in ASR&#10;is using them in the second-pass rescoring (re-ranking)&#10;of multiple ASR hypotheses&#10;represented&#10;as&#10;an $N$-best list or a lattice,&#10;which is obtained by the first-pass ASR decoding.&#10;Several studies have reported&#10;the usefulness of&#10;LLMs&#10;in $N$-best or lattice rescoring of ASR hypotheses&#10;\cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}.&#10;&#10;Thanks to the significant progress of end-to-end (E2E) neural network modeling,&#10;the performance of ASR has greatly improved.&#10;Despite this&#10;significant&#10;progress,&#10;ASR accuracy remains unsatisfactory in some situations,&#10;such as performing ASR &#10;in daily-life environments&#10;\cite{Cornell_CHiME2023,Wang_CHiME2023,Ye_CHiME2023,Kamo_CHiME2023,Prisyach_CHiME2023,Park_CHiME2023}.&#10;The distant ASR (DASR) task of the CHiME-7 challenge&#10;provides a dataset&#10;of such challenging situations \cite{Cornell_CHiME2023}.&#10;The dataset contains casual conversations between multiple participants&#10;at&#10;real dinner parties.&#10;LMs can be expected to&#10;play an&#10;important&#10;role&#10;in ASR of such casual conversational speech, &#10;and most of the&#10;submitted&#10;systems&#10;try to use&#10;LMs&#10;during&#10;ASR decoding&#10;and/or for rescoring ASR hypotheses&#10;\cite{Ye_CHiME2023,Kamo_CHiME2023,Prisyach_CHiME2023,Park_CHiME2023}.&#10;However,&#10;the effect of using LMs is limited&#10;(the&#10;first-place system&#10;does not use&#10;any&#10;LMs \cite{Wang_CHiME2023}),&#10;and there&#10;is a demand for LMs&#10;to&#10;deal with&#10;such highly&#10;casual conversational speech.&#10;&#10;As described above,&#10;several studies have successfully applied LLMs for&#10;rescoring ASR hypotheses&#10;\cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}.&#10;However,&#10;their&#10;targets are not casual conversations,&#10;and the ability of LLMs to rescore ASR hypotheses of&#10;casual conversations&#10;remains unclear&#10;(note that LLMs are not allowed to be used in the CHiME-7&#10;challenge&#10;\cite{Cornell_CHiME2023}).&#10;In this study,&#10;we reveal&#10;it&#10;by&#10;performing&#10;$N$-best ASR hypotheses rescoring&#10;using&#10;Llama2-7B \cite{Touvron_arXiv2023a},&#10;which is one of the most representative&#10;Transformer&#10;\cite{Vaswani_NIPS2017}&#10;decoder-based&#10;causal&#10;LLMs,&#10;on the CHiME-7 DASR task.&#10;We comprehensively investigate&#10;the effects of &#10;domain adaptation of the&#10;LLM&#10;and&#10;context carry-over&#10;\cite{Zheng_ASRU2021,Udagawa_IS2022,Chen_ICASSP2023,Ye_CHiME2023}&#10;when performing $N$-best rescoring.&#10;We employ QLoRA&#10;\cite{Dettmers_NeurIPS2023}&#10;for&#10;memory efficient&#10;domain adaptation&#10;and consider various context lengths (up to 1024 tokens)&#10;in context carry-over.&#10;&#10;We conducted experiments,&#10;including experimental settings that have not been investigated&#10;in previous studies&#10;\cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023},&#10;and thus,&#10;the experimental results&#10;and findings&#10;obtained in this study are informative&#10;for researchers in this field&#10;(note that&#10;Llama2-7B&#10;is allowed to be used&#10;in the CHiME-8 challenge \cite{CHiME}).&#10;Our main findings can be summarized as follows.&#10;\begin{itemize}&#10;\item Even without domain adaptation,&#10;Llama2&#10;significantly&#10;outperforms a standard-size domain-adapted Transformer-LM.&#10;\item Both&#10;domain adaptation and context carry-over improve the Llama2 performance.&#10;\item Even without domain adaptation,&#10;by considering a very long context (e.g., 1024 tokens),&#10;Llama2 captures the flow of a conversation&#10;and achieves&#10;the lowest word error rate (WER),&#10;which is&#10;achieved&#10;with the domain-adapted Llama2.&#10;\item Domain adaptation&#10;shortens the context length&#10;needed with Llama2 to achieve the lowest WER,&#10;significantly reducing&#10;the computational cost of Llama2.&#10;\end{itemize}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Recognition Systems, Linguistics, Domain Adaptation Techniques, Artificial Intelligence, Signal Processing, Language Model Rescoring" />
        </attvalues>
      </node>
      <node id="2104.04950" label="2104.04950">
        <attvalues>
          <attvalue for="0" value="Innovative Bert-based Reranking Language Models for Speech Recognition" />
          <attvalue for="1" value="  More recently, Bidirectional Encoder Representations from Transformers (BERT)&#10;was proposed and has achieved impressive success on many natural language&#10;processing (NLP) tasks such as question answering and language understanding,&#10;due mainly to its effective pre-training then fine-tuning paradigm as well as&#10;strong local contextual modeling ability. In view of the above, this paper&#10;presents a novel instantiation of the BERT-based contextualized language models&#10;(LMs) for use in reranking of N-best hypotheses produced by automatic speech&#10;recognition (ASR). To this end, we frame N-best hypothesis reranking with BERT&#10;as a prediction problem, which aims to predict the oracle hypothesis that has&#10;the lowest word error rate (WER) given the N-best hypotheses (denoted by&#10;PBERT). In particular, we also explore to capitalize on task-specific global&#10;topic information in an unsupervised manner to assist PBERT in N-best&#10;hypothesis reranking (denoted by TPBERT). Extensive experiments conducted on&#10;the AMI benchmark corpus demonstrate the effectiveness and feasibility of our&#10;methods in comparison to the conventional autoregressive models like the&#10;recurrent neural network (RNN) and a recently proposed method that employed&#10;BERT to compute pseudo-log-likelihood (PLL) scores for N-best hypothesis&#10;reranking.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.01857" label="2110.01857">
        <attvalues>
          <attvalue for="0" value="ASR Rescoring and Confidence Estimation with ELECTRA" />
          <attvalue for="1" value="  In automatic speech recognition (ASR) rescoring, the hypothesis with the&#10;fewest errors should be selected from the n-best list using a language model&#10;(LM). However, LMs are usually trained to maximize the likelihood of correct&#10;word sequences, not to detect ASR errors. We propose an ASR rescoring method&#10;for directly detecting errors with ELECTRA, which is originally a pre-training&#10;method for NLP tasks. ELECTRA is pre-trained to predict whether each word is&#10;replaced by BERT or not, which can simulate ASR error detection on large text&#10;corpora. To make this pre-training closer to ASR error detection, we further&#10;propose an extended version of ELECTRA called phone-attentive ELECTRA&#10;(P-ELECTRA). In the pre-training of P-ELECTRA, each word is replaced by a&#10;phone-to-word conversion model, which leverages phone information to generate&#10;acoustically similar words. Since our rescoring method is optimized for&#10;detecting errors, it can also be used for word-level confidence estimation.&#10;Experimental evaluations on the Librispeech and TED-LIUM2 corpora show that our&#10;rescoring method with ELECTRA is competitive with conventional rescoring&#10;methods with faster inference. ELECTRA also performs better in confidence&#10;estimation than BERT because it can learn to detect inappropriate words not&#10;only in fine-tuning but also in pre-training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.01094" label="2202.01094">
        <attvalues>
          <attvalue for="0" value="RescoreBERT: Discriminative Speech Recognition Rescoring with BERT" />
          <attvalue for="1" value="  Second-pass rescoring is an important component in automatic speech&#10;recognition (ASR) systems that is used to improve the outputs from a first-pass&#10;decoder by implementing a lattice rescoring or $n$-best re-ranking. While&#10;pretraining with a masked language model (MLM) objective has received great&#10;success in various natural language understanding (NLU) tasks, it has not&#10;gained traction as a rescoring model for ASR. Specifically, training a&#10;bidirectional model like BERT on a discriminative objective such as minimum WER&#10;(MWER) has not been explored. Here we show how to train a BERT-based rescoring&#10;model with MWER loss, to incorporate the improvements of a discriminative loss&#10;into fine-tuning of deep bidirectional pretrained models for ASR. Specifically,&#10;we propose a fusion strategy that incorporates the MLM into the discriminative&#10;training process to effectively distill knowledge from a pretrained model. We&#10;further propose an alternative discriminative loss. This approach, which we&#10;call RescoreBERT, reduces WER by 6.6%/3.4% relative on the LibriSpeech&#10;clean/other test sets over a BERT baseline without discriminative objective. We&#10;also evaluate our method on an internal dataset from a conversational agent and&#10;find that it reduces both latency and WER (by 3 to 8% relative) over an LSTM&#10;rescoring model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.00212" label="2204.00212">
        <attvalues>
          <attvalue for="0" value="Effect and Analysis of Large-scale Language Model Rescoring on&#10;  Competitive ASR Systems" />
          <attvalue for="1" value="  Large-scale language models (LLMs) such as GPT-2, BERT and RoBERTa have been&#10;successfully applied to ASR N-best rescoring. However, whether or how they can&#10;benefit competitive, near state-of-the-art ASR systems remains unexplored. In&#10;this study, we incorporate LLM rescoring into one of the most competitive ASR&#10;baselines: the Conformer-Transducer model. We demonstrate that consistent&#10;improvement is achieved by the LLM's bidirectionality, pretraining, in-domain&#10;finetuning and context augmentation. Furthermore, our lexical analysis sheds&#10;light on how each of these components may be contributing to the ASR&#10;performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.08133" label="2306.08133">
        <attvalues>
          <attvalue for="0" value="Large-scale Language Model Rescoring on Long-form Data" />
          <attvalue for="1" value="  In this work, we study the impact of Large-scale Language Models (LLM) on&#10;Automated Speech Recognition (ASR) of YouTube videos, which we use as a source&#10;for long-form ASR. We demonstrate up to 8\% relative reduction in Word Error&#10;Eate (WER) on US English (en-us) and code-switched Indian English (en-in)&#10;long-form ASR test sets and a reduction of up to 30\% relative on Salient Term&#10;Error Rate (STER) over a strong first-pass baseline that uses a maximum-entropy&#10;based language model. Improved lattice processing that results in a lattice&#10;with a proper (non-tree) digraph topology and carrying context from the 1-best&#10;hypothesis of the previous segment(s) results in significant wins in rescoring&#10;with LLMs. We also find that the gains in performance from the combination of&#10;LLMs trained on vast quantities of available data (such as C4) and conventional&#10;neural LMs is additive and significantly outperforms a strong first-pass&#10;baseline with a maximum entropy LM.&#10;  Copyright 2023 IEEE. Personal use of this material is permitted. Permission&#10;from IEEE must be obtained for all other uses, in any current or future media,&#10;including reprinting/republishing this material for advertising or promotional&#10;purposes, creating new collective works, for resale or redistribution to&#10;servers or lists, or reuse of any copyrighted component of this work in other&#10;works.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.15223" label="2309.15223">
        <attvalues>
          <attvalue for="0" value="Low-rank Adaptation of Large Language Model Rescoring for&#10;  Parameter-Efficient Speech Recognition" />
          <attvalue for="1" value="  We propose a neural language modeling system based on low-rank adaptation&#10;(LoRA) for speech recognition output rescoring. Although pretrained language&#10;models (LMs) like BERT have shown superior performance in second-pass&#10;rescoring, the high computational cost of scaling up the pretraining stage and&#10;adapting the pretrained models to specific domains limit their practical use in&#10;rescoring. Here we present a method based on low-rank decomposition to train a&#10;rescoring BERT model and adapt it to new domains using only a fraction (0.08%)&#10;of the pretrained parameters. These inserted matrices are optimized through a&#10;discriminative training objective along with a correlation-based regularization&#10;loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is&#10;evaluated on LibriSpeech and internal datasets with decreased training times by&#10;factors between 5.4 and 3.6.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.13734" label="2306.13734">
        <attvalues>
          <attvalue for="0" value="The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple&#10;  Devices in Diverse Scenarios" />
          <attvalue for="1" value="  The CHiME challenges have played a significant role in the development and&#10;evaluation of robust automatic speech recognition (ASR) systems. We introduce&#10;the CHiME-7 distant ASR (DASR) task, within the 7th CHiME challenge. This task&#10;comprises joint ASR and diarization in far-field settings with multiple, and&#10;possibly heterogeneous, recording devices. Different from previous challenges,&#10;we evaluate systems on 3 diverse scenarios: CHiME-6, DiPCo, and Mixer 6. The&#10;goal is for participants to devise a single system that can generalize across&#10;different array geometries and use cases with no a-priori information. Another&#10;departure from earlier CHiME iterations is that participants are allowed to use&#10;open-source pre-trained models and datasets. In this paper, we describe the&#10;challenge design, motivation, and fundamental research questions in detail. We&#10;also present the baseline system, which is fully array-topology agnostic and&#10;features multi-channel diarization, channel selection, guided source separation&#10;and a robust ASR model that leverages self-supervised speech representations&#10;(SSLR).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.14638" label="2308.14638">
        <attvalues>
          <attvalue for="0" value="The USTC-NERCSLIP Systems for the CHiME-7 DASR Challenge" />
          <attvalue for="1" value="  This technical report details our submission system to the CHiME-7 DASR&#10;Challenge, which focuses on speaker diarization and speech recognition under&#10;complex multi-speaker scenarios. Additionally, it also evaluates the efficiency&#10;of systems in handling diverse array devices. To address these issues, we&#10;implemented an end-to-end speaker diarization system and introduced a&#10;rectification strategy based on multi-channel spatial information. This&#10;approach significantly diminished the word error rates (WER). In terms of&#10;recognition, we utilized publicly available pre-trained models as the&#10;foundational models to train our end-to-end speech recognition models. Our&#10;system attained a Macro-averaged diarization-attributed WER (DA-WER) of 21.01%&#10;on the CHiME-7 evaluation set, which signifies a relative improvement of 62.04%&#10;over the official baseline system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12378" label="2310.12378">
        <attvalues>
          <attvalue for="0" value="The CHiME-7 Challenge: System Description and Performance of NeMo Team's&#10;  DASR System" />
          <attvalue for="1" value="  We present the NVIDIA NeMo team's multi-channel speech recognition system for&#10;the 7th CHiME Challenge Distant Automatic Speech Recognition (DASR) Task,&#10;focusing on the development of a multi-channel, multi-speaker speech&#10;recognition system tailored to transcribe speech from distributed microphones&#10;and microphone arrays. The system predominantly comprises of the following&#10;integral modules: the Speaker Diarization Module, Multi-channel Audio Front-End&#10;Processing Module, and the ASR Module. These components collectively establish&#10;a cascading system, meticulously processing multi-channel and multi-speaker&#10;audio input. Moreover, this paper highlights the comprehensive optimization&#10;process that significantly enhanced our system's performance. Our team's&#10;submission is largely based on NeMo toolkits and will be publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.14039" label="2109.14039">
        <attvalues>
          <attvalue for="0" value="Marked Attribute Bias in Natural Language Inference" />
          <attvalue for="1" value="  Reporting and providing test sets for harmful bias in NLP applications is&#10;essential for building a robust understanding of the current problem. We&#10;present a new observation of gender bias in a downstream NLP application:&#10;marked attribute bias in natural language inference. Bias in downstream&#10;applications can stem from training data, word embeddings, or be amplified by&#10;the model in use. However, focusing on biased word embeddings is potentially&#10;the most impactful first step due to their universal nature. Here we seek to&#10;understand how the intrinsic properties of word embeddings contribute to this&#10;observed marked attribute effect, and whether current post-processing methods&#10;address the bias successfully. An investigation of the current debiasing&#10;landscape reveals two open problems: none of the current debiased embeddings&#10;mitigate the marked attribute error, and none of the intrinsic bias measures&#10;are predictive of the marked attribute effect. By noticing that a new type of&#10;intrinsic bias measure correlates meaningfully with the marked attribute&#10;effect, we propose a new postprocessing debiasing scheme for static word&#10;embeddings. The proposed method applied to existing embeddings achieves new&#10;best results on the marked attribute bias test set. See&#10;https://github.com/hillary-dawkins/MAB.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-trained distributed representations of words (a.k.a. word embeddings) are ubiquitous tools in natural language processing (NLP). &#10;Their utility is owing to the remarkable success in mapping semantic and syntactic relationships among words to linear relationships among real-valued vectors. &#10;For instance, analogy generation using vector addition on word embeddings (e.g. Tokyo is to Japan as Paris is to France) was taken to be an early measure of word embedding quality. &#10;In all kinds of related tasks, the vector space is known to encode semantic meaning surprisingly well \cite{pennington2014glove, Mik:2013:w2v, Mik:2013:linreg}. &#10;However, harmful gender-biased properties of word embeddings are also known to exist. &#10;Later is was observed that the same analogy generation property that produced the celebrated ``man is to king as woman is to queen&quot; analogy would also predict ``man is to programmer as woman is to homemaker&quot; \cite{Tolga:2016}. This observation sparked interest in developing debiased word embeddings. &#10;&#10;Post-processing debiasing schemes are usually motivated by recognizing some intrinsic measure of bias in the embedding space, and then attempting to reduce that intrinsic bias. Early work (2016-2017) focused on the idea of a ``gender direction&quot; vector within the embedding space, loosely defined as the difference vector between female and male attribute words. &#10;It was noted that any non-zero projection of a word onto the gender direction (termed direct bias) implied that the word was more related to one gender over another.&#10;In the case of ideally gender-neutral words (e.g. doctor, nurse, programmer, homemaker), this was viewed as an undesirable property. &#10;The first debiasing methods, Hard Debias \cite{Tolga:2016} and Gender Neutral-GloVe \cite{Zhao2018:GNglove}, worked to minimize or eliminate the direct bias, and were shown to be successful in mitigating harmful analogies generated by word embeddings in relation to gender-stereotyped occupations. &#10;&#10;An influential critique paper by \cite{Gonen:2019} demonstrated that minimizing direct bias did not eliminate bias in the vector space entirely. Rather, words that tended to cluster together due to gender bias (e.g. nurse, teacher, secretary, etc.) would still cluster together in the nullspace of the gender direction. Furthermore, the original bias could be recovered by classification techniques using only the debiased word embeddings as input. These observations were termed cluster and recoverability bias. &#10;&#10;The next wave of debiasing methods (2019-present) focused on reducing cluster and recoverability bias while proposing new metrics to systematically quantify the indirect bias of the embedding space (e.g. the Gender-based Illicit Proximity Estimate, introduced by \cite{Kumar:2020:RAN}). While these new debiasing schemes do reduce indirect bias in multiple ways, there is a general lack of&#10;connection to downstream applications such as coreference resolution, natural language inference (NLI) and sentiment analysis. &#10;&#10;Current gender-bias evaluation tests (GBETs) in widespread use include the WinoBias test set \cite{Zhao:2018:WinoBias}, designed to measure bias in coreference resolution systems using stereotypical occupations as a probe, and the NLI test set \cite{Dev:2020:NLItest}, designed to measure stereotypical inferences again using occupations as the concept of interest. &#10;More commonly used evaluations include the Word Embedding Association Test (WEAT) \cite{Caliskan:2017:weat}, and the analogy generation test SemBias \cite{Zhao2018:GNglove}. However these tests solely evaluate the vector properties of the word embeddings, without any connection to downstream applications. &#10;Adding to the library of downstream GBETs is essential in building a robust understanding of gender bias in NLP applications \cite{Sun:2019:review}.&#10;&#10;Here we introduce a new observation of gender-biased predictions in a downstream task, namely “marked attribute bias” in natural language inference, and develop corresponding GBETs. Marked attribute bias refers to the language model’s tendency to predict that “person” implies “man” (the default attribute), while simultaneously understanding that “person” does not necessarily imply “woman” (the marked attribute). Marked attribute bias was found to exist on explicitly defined gender words (e.g. man, woman, etc.), and persist on implicit gender words (e.g. names) as well as latent gender-carriers (e.g. stereotypical occupations). &#10;&#10;An analysis of the currently available debiased embeddings reveals that none are able to successfully mitigate marked attribute bias. Furthermore, none of the currently proposed measures of intrinsic bias on the embedding space are predictive of the marked attribute effect. We define a new measure of intrinsic bias that was found to correlate with the marked attribute effect better than any currently available metric. &#10;Using this insight, we introduce a new debiasing scheme: Multi-dimensional Information-weighted Soft Projection. Applying MISP to an existing debiased embedding achieves the lowest observed marked attribute bias error. &#10;\\&#10;&#10;Summary of main contributions:&#10;&#10;\begin{enumerate}&#10;\item We present a new observation of gender bias in a downstream NLP application: marked attribute bias (MAB). The MAB test sets are made available in order to expand the current set of GBETs.&#10;\item An analysis of current debiasing methods and current intrinsic bias measures finds that none sufficiently mitigate the error, and likewise none sufficiently explain the effect. This observation creates two new open problems.&#10;\item We propose a new measure for quantifying intrinsic bias on the embedding space: Multi-dimensional Information-weighted Direct Bias (MIDB). This measure was found to correlate meaningfully with the marked attribute effect.&#10;\item We introduce a new debiasing scheme: Multi-dimensional Information-weighted Soft Projection. MISP-debiased embeddings obtain new best performance on the MAB test set. &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Word Embeddings, Artificial Intelligence, Bias Mitigation, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1809.01496" label="1809.01496">
        <attvalues>
          <attvalue for="0" value="Learning Gender-Neutral Word Embeddings" />
          <attvalue for="1" value="  Word embedding models have become a fundamental component in a wide range of&#10;Natural Language Processing (NLP) applications. However, embeddings trained on&#10;human-generated corpora have been demonstrated to inherit strong gender&#10;stereotypes that reflect social constructs. To address this concern, in this&#10;paper, we propose a novel training procedure for learning gender-neutral word&#10;embeddings. Our approach aims to preserve gender information in certain&#10;dimensions of word vectors while compelling other dimensions to be free of&#10;gender influence. Based on the proposed method, we generate a Gender-Neutral&#10;variant of GloVe (GN-GloVe). Quantitative and qualitative experiments&#10;demonstrate that GN-GloVe successfully isolates gender information without&#10;sacrificing the functionality of the embedding model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.03862" label="1903.03862">
        <attvalues>
          <attvalue for="0" value="Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases&#10;  in Word Embeddings But do not Remove Them" />
          <attvalue for="1" value="  Word embeddings are widely used in NLP for a vast range of tasks. It was&#10;shown that word embeddings derived from text corpora reflect gender biases in&#10;society. This phenomenon is pervasive and consistent across different word&#10;embedding models, causing serious concern. Several recent works tackle this&#10;problem, and propose methods for significantly reducing this gender bias in&#10;word embeddings, demonstrating convincing results. However, we argue that this&#10;removal is superficial. While the bias is indeed substantially reduced&#10;according to the provided bias definition, the actual effect is mostly hiding&#10;the bias, not removing it. The gender bias information is still reflected in&#10;the distances between &quot;gender-neutralized&quot; words in the debiased embeddings,&#10;and can be recovered from them. We present a series of experiments to support&#10;this claim, for two debiasing methods. We conclude that existing bias removal&#10;techniques are insufficient, and should not be trusted for providing&#10;gender-neutral modeling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.01938" label="2006.01938">
        <attvalues>
          <attvalue for="0" value="Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased&#10;  Proximities in Word Embeddings" />
          <attvalue for="1" value="  Word embeddings are the standard model for semantic and syntactic&#10;representations of words. Unfortunately, these models have been shown to&#10;exhibit undesirable word associations resulting from gender, racial, and&#10;religious biases. Existing post-processing methods for debiasing word&#10;embeddings are unable to mitigate gender bias hidden in the spatial arrangement&#10;of word vectors. In this paper, we propose RAN-Debias, a novel gender debiasing&#10;methodology which not only eliminates the bias present in a word vector but&#10;also alters the spatial distribution of its neighbouring vectors, achieving a&#10;bias-free setting while maintaining minimal semantic offset. We also propose a&#10;new bias evaluation metric - Gender-based Illicit Proximity Estimate (GIPE),&#10;which measures the extent of undue proximity in word vectors resulting from the&#10;presence of gender-based predilections. Experiments based on a suite of&#10;evaluation metrics show that RAN-Debias significantly outperforms the&#10;state-of-the-art in reducing proximity bias (GIPE) by at least 42.02%. It also&#10;reduces direct bias, adding minimal semantic disturbance, and achieves the best&#10;performance in a downstream application task (coreference resolution).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.09369" label="1908.09369">
        <attvalues>
          <attvalue for="0" value="On Measuring and Mitigating Biased Inferences of Word Embeddings" />
          <attvalue for="1" value="  Word embeddings carry stereotypical connotations from the text they are&#10;trained on, which can lead to invalid inferences in downstream models that rely&#10;on them. We use this observation to design a mechanism for measuring&#10;stereotypes using the task of natural language inference. We demonstrate a&#10;reduction in invalid inferences via bias mitigation strategies on static word&#10;embeddings (GloVe). Further, we show that for gender bias, these techniques&#10;extend to contextualized embeddings when applied selectively only to the static&#10;components of contextualized embeddings (ELMo, BERT).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.08976" label="1906.08976">
        <attvalues>
          <attvalue for="0" value="Mitigating Gender Bias in Natural Language Processing: Literature Review" />
          <attvalue for="1" value="  As Natural Language Processing (NLP) and Machine Learning (ML) tools rise in&#10;popularity, it becomes increasingly vital to recognize the role they play in&#10;shaping societal biases and stereotypes. Although NLP models have shown success&#10;in modeling various applications, they propagate and may even amplify gender&#10;bias found in text corpora. While the study of bias in artificial intelligence&#10;is not new, methods to mitigate gender bias in NLP are relatively nascent. In&#10;this paper, we review contemporary studies on recognizing and mitigating gender&#10;bias in NLP. We discuss gender bias based on four forms of representation bias&#10;and analyze methods recognizing gender bias. Furthermore, we discuss the&#10;advantages and drawbacks of existing gender debiasing methods. Finally, we&#10;discuss future studies for recognizing and mitigating gender bias in NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.03871" label="2210.03871">
        <attvalues>
          <attvalue for="0" value="Data-Efficiency with a Single GPU: An Exploration of Transfer Methods&#10;  for Small Language Models" />
          <attvalue for="1" value="  Multi-task learning (MTL), instruction tuning, and prompting have recently&#10;been shown to improve the generalizability of large language models to new&#10;tasks. However, the benefits of such methods are less well-documented in&#10;smaller language models, with some studies finding contradictory results. In&#10;this work, we explore and isolate the effects of (i) model size, (ii) general&#10;purpose MTL, (iii) in-domain MTL, (iv) instruction tuning, and (v) few-shot&#10;fine-tuning for models with fewer than 500 million parameters. Our experiments&#10;in the zero-shot setting demonstrate that models gain 31% relative improvement,&#10;on average, from general purpose MTL, with an additional 37.6% relative gain&#10;from in-domain MTL. Contradictory to prior works on large models, we find that&#10;instruction tuning provides a modest 2% performance improvement for small&#10;models.&#10;" />
          <attvalue for="2" value="&#10;Many recent works have demonstrated the benefits of prompting for large language models (see \cite{prompt_survey} for an extensive survey). While prompts started as simple task identifiers (eg. &quot;topic&quot; for topic classification) they have expanded to include answer templates, examples, and instructions \cite{2020t5, FETA,FLAN,Mishra2021NaturalIB,Ouyang2022TrainingLM}. Studies on utilizing prompts have shown that as model sizes scale up, the generalization abilities of a model increase \cite{gpt3, lester-etal-2021-power,demonstrations_icl}. However, utilizing models on the hundred-billion parameter scale is not accessible for most researchers and practitioners. Furthermore, \cite{wei2022emergent} show that trends for large language models do not hold for smaller language models. &#10;For this reason, it is crucial that we must empirically find the trends that occur in smaller models and cannot rely on studies of larger models.&#10;&#10;Interestingly, some findings on instruction tuning have been contradictory across studies. For example, \cite{FLAN} find that models with fewer than 8B parameters see decreases in generalization when utilizing instructions, whereas \cite{instructdial} find consistent gains in models with 3B and fewer parameters. To conflate these results further though, \cite{instructdial} only consider 2 situations: when inputs include instructions and answer templates, or neither.&#10;&#10;Simultaneously with the emergence of prompting, the explicit multi-task learning (MTL) paradigm emerged, with works such as Muppet \cite{aghajanyan-etal-2021-muppet} or T0 \cite{sanh2022multitask} and their variants. Explicit MTL has been demonstrated as a means of improving the downstream performance of pre-trained language models in data-constrained settings. In this work we consider 2 types of MTL: general purpose and in-domain. Specifically, general purpose MTL consists of training across a wide variety of tasks and domains, whereas in-domain MTL consists of training across a variety of tasks that all occur within a domain.&#10;&#10;One limitation of many previous works on prompting and multi-task learning is that they focus on language models in the billion-parameter scale. For situations with latency and memory limitations, small models may be the only option. In this work, we study an example of such a domain; dialogue.&#10;&#10;In this work we bridge the gap between previous studies by exploring the effects of a variety of factors on the zero- and few-shot generalizability on modestly sized language models (&lt;500M parameters). Specifically, we run experiments to find the effects of: (i) model size, (ii) general purpose MTL, (iii) in-domain MTL, (iv) instruction tuning, and (v) fine-tuning with and without instructions. Additionally, &#10;we perform a linguistic analysis on instruction wording and&#10;analyze variations in performance across task instructions.&#10;&#10;In this study, we find that&#10;(1) In-domain multi-task learning (MTL) gives the largest improvements to generalizability, up to 80\% increased relative performance, and 37.6\% on average across all models&#10;(2) Increasing model size alone has little effect on generalization, but when combined with in-domain MTL leads to double the (already strong) performance improvement of in-domain MTL&#10;(3) General purpose MTL can provide large gains (57\% improvement) for downstream tasks which closely resemble the MTL tasks, but still provides modest gains (5\%) even for tasks which are more dissimilar&#10;(4) Instruction tuning during in-domain MTL provides modest gains of just over 2\% performance, regardless of model size.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Multi-Task Learning Effects, Mathematics, Model Size Performance, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2101.11038" label="2101.11038">
        <attvalues>
          <attvalue for="0" value="Muppet: Massive Multi-task Representations with Pre-Finetuning" />
          <attvalue for="1" value="  We propose pre-finetuning, an additional large-scale learning stage between&#10;language model pre-training and fine-tuning. Pre-finetuning is massively&#10;multi-task learning (around 50 datasets, over 4.8 million total labeled&#10;examples), and is designed to encourage learning of representations that&#10;generalize better to many different tasks. We show that pre-finetuning&#10;consistently improves performance for pretrained discriminators (e.g.~RoBERTa)&#10;and generation models (e.g.~BART) on a wide range of tasks (sentence&#10;prediction, commonsense reasoning, MRC, etc.), while also significantly&#10;improving sample efficiency during fine-tuning. We also show that large-scale&#10;multi-tasking is crucial; pre-finetuning can hurt performance when few tasks&#10;are used up until a critical point (usually above 15) after which performance&#10;improves linearly in the number of tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Multi-task learning has been an increasingly active topic in recent literature. Recent advances such as MT-DNN show that by leveraging multi-task learning, we can further improve performance on several language benchmarks on top of traditional pre-training \cite{MT_DNN}. However, T5 \cite{T5} shows that incorporating multi-task learning ontop of larger models does not improve upon the standardized pre-training / finetuning. Thus the effect of multi-task learning across different pre-training methods is not fully understood. &#10;&#10;Recently \cite{unifiedqa} showed how doing MTL training on a range of QA tasks can improve the performance of T5 by taking advantage of cross dataset transfer. Unlike our approach, they convert all the data to a seq2seq format, operate on a smaller MTL scale, have a different batching strategy, and focus solely on improving QA tasks. Our work shows how even seemingly very different datasets, for example, summarization and extractive QA, can help each other by improving the model's representations.&#10;&#10;Our work aims to explore multi-task learning at a much larger scale; by incorporating a larger number of tasks, we show that we can consistently improve several language benchmarks from several domains. Contrary to T5, we show that incorporating a secondary stage of multi-task learning does lead to better representations. In \S \ref{sec:understanding_mtl} we demonstrate the effectiveness of multi-task learning to be coming from the large scale of our MTL setup.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2404.12195" label="2404.12195">
        <attvalues>
          <attvalue for="0" value="OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of&#10;  Instruction Data" />
          <attvalue for="1" value="  Instruction fine-tuning pretrained LLMs for diverse downstream tasks has&#10;demonstrated remarkable success and has captured the interest of both academics&#10;and practitioners. To ensure such fine-tuned LLMs align with human preferences,&#10;techniques such as RLHF and DPO have emerged. At the same time, there is&#10;increasing interest in smaller parameter counts for models. In this work, using&#10;OpenLLaMA 3Bv2 as a base model, we describe the recipe used to fine-tune the&#10;OpenBezoar family of models. In this recipe: We first generate synthetic&#10;instruction fine-tuning data using an open and commercially non-restrictive&#10;instruction fine-tuned variant of the Falcon-40B model under three schemes&#10;based on: LaMini-LM, WizardLM/Evol-Instruct (with databricks-dolly-15k as a&#10;seed dataset) and Orca (with the Flan Collection as a seed dataset), then&#10;filter these generations using GPT-4 as a human proxy. We then perform&#10;cost-effective QLoRA-based supervised fine-tuning sequentially with each&#10;scheme. The resulting checkpoint is further fine-tuned with a subset of the&#10;HH-RLHF dataset to minimize distribution shift prior to using the DPO loss to&#10;obtain the final checkpoint. Evaluation is done with the LM Eval Harness&#10;tasks/metrics as well as on MT-Bench using the &quot;LLM-as-a-judge&quot; framework with&#10;Claude 2.1, with the finding that the final checkpoint,&#10;&quot;OpenBezoar-HH-RLHF-DPO&quot;, demonstrates superior performance over many models at&#10;the 3B parameter scale, even outperforming the top model in one of the&#10;categories on the Huggingface Open LLM Leaderboard. We release&#10;&quot;OpenBezoar-SFT&quot;, &quot;OpenBezoar-HH-RLHF-SFT&quot;, &quot;OpenBezoar-HH-RLHF-DPO&quot;&#10;checkpoints, alongside our generated datasets on HuggingFace at&#10;https://huggingface.co/collections/SurgeGlobal/open-bezoar-6620a24923e12127e9e2b9cc&#10;and our codebase at&#10;https://bitbucket.org/paladinanalytics/workspace/projects/OP.&#10;" />
          <attvalue for="2" value="&#10;&#10;\acrfull{acr_sft} of pre-trained \acrfull{acr_llms} on instruction datasets in order to specialize them in a variety of downstream tasks is not just pivotal for guiding them to produce sensible responses, but also serves as a compelling demonstration of how supervised learning can enable artificial models to generalize effectively through observational learning. This process of \acrshort{acr_sft} for \acrshort{acr_llm}s is largely similar to other gradient based optimization pipelines. Early examples, and at present the most capable \acrshort{acr_llm}s are very large, with reported parameter count exceeding 100B. Consequently, the computational cost of \acrshort{acr_sft} for an \acrshort{acr_llm} of such magnitude is out of reach for organizations and individuals with conventional budgetary expectations. However, it has been demonstrated that models with a comparatively smaller parameter count can perform reasonably well on diverse downstream tasks, even outperforming larger models in specific cases\cite{touvron2023llama} \cite{gunasekar2023textbooks}. Our survey during the preliminary stages of this work (carried out between March and October of 2023) indicated that there were only a handful of fine tuned models at the 3B scale that had benchmark scores comparable to their larger counterparts\cite{togethercomputer_RedPajama_INCITE_Base_3B_v1}. Subsequently, in an effort to investigate the potential of instruction fine-tuned 3B parameter scale models, we chose to devise and implement a recipe for fine-tuning the OpenLLaMA 3B V2\cite{openlm2023openllama} base model, which was a very recently released model at the time.\par&#10;&#10;The largest and most capable base models are typically fine-tuned using datasets comprising of large numbers of human-generated examples, which in part accounts for the versatility of the resulting fine-tuned \acrshort{acr_llms}. However, such datasets are costly to create due to the human labor required, cause training times to inflate, and typically, their resulting models' licenses prevent the commercial use of novel models fine-tuned on such models' outputs. Open, crow-sourced datasets exist, but they often suffer from problems such as limited diversity and relatively small size\cite{DatabricksBlog2023DollyV2,together2023redpajama, openassistantoaast1}. Scale can be achieved by having an \acrshort{acr_llm} generate completely new instruction datasets \cite{maeng2019alpaca, wang-etal-2023-self-instruct}, but the most capable such models have restrictive licensing, casting uncertainty on the openness of derived models trained on their outputs. Our aim in this work is to utilize a sufficiently capable open-source instruction model with a license that permits commercial use of the generated responses \cite{h2oaifalcon40boasst}, in order to generate instruction/response pairs via three dataset generation schemes, resulting in instruction datasets that permit commercial use. We go on to further filter this dataset for higher quality and more diverse generations using a better human proxy model \cite{achiam2023gpt}, and perform \acrshort{acr_sft} on our chosen open base model using \acrshort{acr_qlora}, resulting in three \acrshort{acr_qlora} adapter models. These models, along with an alignment-specific model described below, comprise the OpenBezoar family of models, released herewith.\par&#10;&#10;As outlined above, \acrshort{acr_llms} when fine-tuned using supervised methods for different tasks on large datasets have been proven to generalize surprisingly well and perform on a wide range of benchmarks. If these acquired expertise are collectively termed the model's ``skillset&quot;, some of these skills might not be desirable under certain scenarios or may need to be modulated subject to certain circumstances. For instance, if asked to generate a plan to conduct a criminal activity after providing a detailed background of the target of interest, a naively fine-tuned \acrshort{acr_llm} might choose to respond back with the guidelines to achieve the task. While this is to be expected, a human in similar circumstances may elect to exercise more agency and question the request or refuse to answer it. It may be prudent to endow models with similar capabilities dependent on context. To further anthropomorphize, certain responses might be preferred over others based on the context as well. This leads to the conclusion that it may be advisable to bias the output of the \acrshort{acr_llm} towards the human-preferred output, which can be achieved through further fine-tuning the \acrshort{acr_llm} with an objective that achieves the alignment desired.\par&#10;&#10;As the generation using \acrshort{acr_llms} is discrete by nature (as it proceeds token-wise), the objective function for such alignment fine-tuning is inherently non-differentiable. Consequently, a popular approach is to optimize weights post-hoc using \acrfull{acr_rl}, called \acrfull{acr_rlhf} in this context. More specifically, to maximize a reward based on human preference using \acrfull{acr_ppo}\cite{ziegler2020finetuning} is now common. The objective for \acrshort{acr_rl} necessitates a reward model that has been trained on a comparisons dataset sampled from a preference distribution, modelled with a preference model such as Bradley-Terry\cite{Bradley1952RankAO}. A prerequisite for implementing preference modeling techniques is that human annotators, either online or more commonly offline, label the answers to prompts with a ranking that denotes their preferences. As fine-tuning \acrshort{acr_llms} is typically orchestrated at a large scale on massive datasets, the requirement to separately train a reward model can be a significant bottleneck due to these requirements. However, more recently, it has been shown that with a change of variables it is possible to express the objective for training a reward model as a function of the policy itself\cite{rafailov2023direct}, allowing us to dispense with the reward model and make reward implicit. This technique of \acrfull{acr_dpo} allows the alignment of \acrshort{acr_llms} directly from preference datasets. In our work, we perform \acrshort{acr_dpo} on a subset of the Anthropic HH-RLHF dataset\cite{bai2022training} after merging the \acrshort{acr_qlora} adapter from the \acrshort{acr_sft} stage. We deliberately chose to apply \acrshort{acr_dpo} to the merged model as the update rule of \acrshort{acr_dpo} explicitly refers to the entire parameterized \acrshort{acr_llm}\cite{rafailov2023direct}. Further research is required to evaluate the use of low-rank adapters in this regard.\par&#10;&#10;We release checkpoints after each stage of \acrshort{acr_sft} and the merged models before and after \acrshort{acr_dpo}. We call the merged model after the final \acrshort{acr_sft} checkpoint ``OpenBezoar-SFT&quot; and models before and after \acrshort{acr_dpo}, ``OpenBezoar-HH-RLHF-SFT&quot;, and ``OpenBezoar-HH-RLHF-DPO&quot; respectively. Out of ten benchmarks evaluated, OpenBezoar-SFT outperformed the base model in all but two benchmarks (SciQ, PIQA), significantly outperforming it on TruthfulQA (14.18\% accuracy improvement), OpenBookQA (8.84\%), and MMLU (4.29\%), with an overall average improvement of 1.48\%. The final OpenBezoar-HH-RLHF-DPO model outperforms OpenBezoar-SFT in turn on average by 2.36\%, recording improvements on all benchmarks except TruthfulQA (-2.75\%) and MMLU (-6.04\%).&#10;In order to evaluate human preferences alignment, we employ the \acrshort{acr_llm}-as-a-judge framework\cite{zheng2023judging} with the MT-bench benchmark question set. Although it has been established that GPT-4 matches human preferences by achieving the same level of agreement as among humans, here we attempt to establish Anthropic's Claude-2.1\cite{anthropic} as a viable judge. In this regard, we calculate the agreement between Claude-2.1 and other types of judges, including humans and observe that Claude-2.1 exceeds the threshold of 80\% agreement, thus validating its potency as a judge to approximate human preferences. Subsequently, we first evaluate OpenBezoar, OpenBezoar-HH-RLHF-SFT, and OpenBezoar-HH-RLHF-DPO models for single answer grading to establish the overall dominance of OpenBezoar-HH-RLHF-DPO over the preceding models. Hence we choose OpenBezoar-HH-RLHF-DPO for evaluations against three other publicly available models, only one of which (RedPajama-INCITE-Chat-3B-v1) was available at the time of experimentation. These models were chosen based their having a comparable parameter count and their ranking in the HuggingFace Open \acrshort{acr_llm} Leaderboard\cite{huggingface-open-llm-leaderboard}. In terms of the average score, OpenBezoar-HH-RLHF-DPO model surpassed two out of the three competitors, and even outperformed the top performing chat model in 3B parameter scale in one of the categories (Writing).\par&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Fine-Tuning, Cognitive Science, Efficient Model Optimization, Artificial Intelligence, Mathematics, Natural Language Processing, Human Preference Alignment" />
        </attvalues>
      </node>
      <node id="1909.06951" label="1909.06951">
        <attvalues>
          <attvalue for="0" value="Alpaca: Intermittent Execution without Checkpoints" />
          <attvalue for="1" value="  The emergence of energy harvesting devices creates the potential for&#10;batteryless sensing and computing devices. Such devices operate only&#10;intermittently, as energy is available, presenting a number of challenges for&#10;software developers. Programmers face a complex design space requiring&#10;reasoning about energy, memory consistency, and forward progress. This paper&#10;introduces Alpaca, a low-overhead programming model for intermittent computing&#10;on energy-harvesting devices. Alpaca programs are composed of a sequence of&#10;user-defined tasks. The Alpaca runtime preserves execution progress at the&#10;granularity of a task. The key insight in Alpaca is the privatization of data&#10;shared between tasks. Updates of shared values in a task are privatized and&#10;only committed to main memory on successful execution of the task, ensuring&#10;that data remain consistent despite power failures. Alpaca provides a familiar&#10;programming interface and a highly efficient runtime model. We also present an&#10;alternate version of Alpaca, Alpaca-undo, that uses undo-logging and rollback&#10;instead of privatization and commit. We implemented a prototype of both&#10;versions of Alpaca as an extension to C with an LLVM compiler pass. We&#10;evaluated Alpaca, and directly compared to three systems from prior work.&#10;Alpacaconsistently improves performance compared to the previous systems, by up&#10;to 23.8x, while also improving memory footprint in many cases, by up to 17.6x.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.04222" label="2105.04222">
        <attvalues>
          <attvalue for="0" value="Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State&#10;  Tracking" />
          <attvalue for="1" value="  Zero-shot cross-domain dialogue state tracking (DST) enables us to handle&#10;task-oriented dialogue in unseen domains without the expense of collecting&#10;in-domain data. In this paper, we propose a slot description enhanced&#10;generative approach for zero-shot cross-domain DST. Specifically, our model&#10;first encodes dialogue context and slots with a pre-trained self-attentive&#10;encoder, and generates slot values in an auto-regressive manner. In addition,&#10;we incorporate Slot Type Informed Descriptions that capture the shared&#10;information across slots to facilitate cross-domain knowledge transfer.&#10;Experimental results on the MultiWOZ dataset show that our proposed method&#10;significantly improves existing state-of-the-art results in the zero-shot&#10;cross-domain setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Dialogue State Tracking has been of broad interest to the dialogue research community~\cite{williams2007partially,williams2014dialog,heck2020trippy,liu2020attention,wu2020tod,madotto2020language}. Current state-of-the-art models~\cite{Chen2020SchemaGuidedMD,lin2020mintl,heck2020trippy,hosseini2020simple,ye2021slot,li2020coco} trained with extensive annotated data have been shown promising performance in complex multi-domain conversations~\cite{budzianowski2018multiwoz}. However, collecting large amounts of data for every domain is costly and inefficient. To address this issue, several methods~\cite{wu2019transferable,zhou2019multi} have proposed for transferring prior knowledge of existing domains to new ones. On the other hand, \cite{campagna2020zero} proposed an abstract dialogue model that leverages the ontology and in-domain templates to generate a large amount of synthesized data for domain adaptation. Different from their method, in this paper, we utilize a pre-trained seq2seq model and slot descriptions for cross-domain DST without any in-domain data.&#10;&#10;Slot Description has been shown to be a promising technique in cross domain semantic parsing~\cite{bapna2017towards,shah2019robust,namazifar2020language}. To encourage this line of research in DST as well, MultiWOZ2.1~\cite{eric2019multiwoz} provides a further annotation for slot descriptions. \cite{rastogi2020towards} incorporated slot descriptions for facilitating cross domain DST, while \cite{gao2019dialog,gao2020machine} formulated DST as a question answering problem by casting a slot name into questions. However, these works did not show the effectiveness of slot descriptions, by comparing the performance of models with and without them. There is no study on how to construct slot descriptions. In this paper, we aim to fill this research gap by providing an empirical study on the different slot description formulations.&#10;&#10;" />
          <attvalue for="4" value="Dialogue State Tracking, Machine Learning, Computer Science, Linguistics, Artificial Intelligence, Cross-Domain Transfer, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2105.11447" label="2105.11447">
        <attvalues>
          <attvalue for="0" value="True Few-Shot Learning with Language Models" />
          <attvalue for="1" value="  Pretrained language models (LMs) perform well on many tasks even when&#10;learning from a few examples, but prior work uses many held-out examples to&#10;tune various aspects of learning, such as hyperparameters, training objectives,&#10;and natural language templates (&quot;prompts&quot;). Here, we evaluate the few-shot&#10;ability of LMs when such held-out examples are unavailable, a setting we call&#10;true few-shot learning. We test two model selection criteria, cross-validation&#10;and minimum description length, for choosing LM prompts and hyperparameters in&#10;the true few-shot setting. On average, both marginally outperform random&#10;selection and greatly underperform selection based on held-out examples.&#10;Moreover, selection criteria often prefer models that perform significantly&#10;worse than randomly-selected ones. We find similar results even when taking&#10;into account our uncertainty in a model's true performance during selection, as&#10;well as when varying the amount of computation and number of examples used for&#10;selection. Overall, our findings suggest that prior work significantly&#10;overestimated the true few-shot ability of LMs given the difficulty of few-shot&#10;model selection.&#10;" />
          <attvalue for="2" value="&#10;&#10;Major progress in language model (LM) pretraining has led to the idea that LMs can learn a new task using a small number of examples only, i.e., few-shot learning~\cite{radford2019language,brown2020language,schick2020exploiting}.&#10;Few-shot learning overcomes many challenges with data-rich supervised learning: collecting labeled data is expensive, often requires experts, and scales poorly with the number of tasks.&#10;However, the few-shot performance of LMs is very sensitive to the textual task description~\cite[``prompt'';][]{schick2020exploiting,jiang-etal-2020-know,gao2020making,zhao2021calibrate}, order of training examples~\cite{zhao2021calibrate,lu2021fantastically,liu2021what}, decoding strategy~\cite{schick2020small,perez2021rissanen}, and other hyperparameters~\cite{schick2020exploiting,gao2020making,schick2020small,schick2020few,tam2021improving}, as well as the learning algorithm itself~\cite{schick2020exploiting,tam2021improving}.&#10;Thus, effective model selection is crucial for obtaining good few-shot performance.&#10;&#10;There are issues with how recent work approaches model selection in few-shot learning, however.&#10;Prior work uses large train or held-out sets with many examples to choose prompts~\cite{brown2020language,tam2021improving,radford2021learning} and hyperparameters~\cite{tam2021improving}.&#10;Other work claims to use no validation set for hyperparameter selection~\cite{schick2020exploiting,schick2020few,wang2021entailment} but does not describe how they design other aspects of their learning algorithm (e.g., training objectives).&#10;It is unlikely that no validation examples were used, given the sophisticated nature of the proposed algorithms.&#10;In this work, we examine if prior few-shot learning methods still perform well when using only the provided examples for model selection, a setting we term true few-shot learning.&#10;&#10;We find that true few-shot model selection yields prompts that marginally outperform random selection and greatly underperform selection based on held-out examples.&#10;Our result shows that prior work may have greatly overestimated the few-shot ability of LMs.&#10;In other words, one reason that prompts are so effective~\cite[``worth many examples'';][]{scao2021data} is that they are often tuned using many examples.&#10;We evaluate two standard model selection criteria -- cross-validation (CV) and minimum description length (MDL) -- finding that both obtain only limited improvements over random selection and perform much worse than selection using held-out examples.&#10;For prompt selection, our observation holds for 9 LMs ranging over 3 orders of magnitude in size~\cite{radford2019language,brown2020language,sanh2019distilbert} on 3 classification tasks and 41 tasks in the LAMA benchmark~\cite{petroni-etal-2019-language}.&#10;For choosing hyperparameters, true few-shot selection causes performance to drop by 2-10\% across 8 tasks for ADAPET~\cite{tam2021improving}, a state-of-the-art few-shot method.&#10;Furthermore, true few-shot model selection has high variance in performance; selected models often do much worse than randomly-chosen ones.&#10;We find similar results when varying the number of examples used, amount of computation, and conservativeness of our selection criterion.&#10;Altogether, our results suggest that model selection is a fundamental roadblock to true few-shot learning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Computer Science, Linguistics, Language Model Evaluation, Mathematics, Statistics, Model Selection Criteria" />
        </attvalues>
      </node>
      <node id="2103.03872" label="2103.03872">
        <attvalues>
          <attvalue for="0" value="Rissanen Data Analysis: Examining Dataset Characteristics via&#10;  Description Length" />
          <attvalue for="1" value="  We introduce a method to determine if a certain capability helps to achieve&#10;an accurate model of given data. We view labels as being generated from the&#10;inputs by a program composed of subroutines with different capabilities, and we&#10;posit that a subroutine is useful if and only if the minimal program that&#10;invokes it is shorter than the one that does not. Since minimum program length&#10;is uncomputable, we instead estimate the labels' minimum description length&#10;(MDL) as a proxy, giving us a theoretically-grounded method for analyzing&#10;dataset characteristics. We call the method Rissanen Data Analysis (RDA) after&#10;the father of MDL, and we showcase its applicability on a wide variety of&#10;settings in NLP, ranging from evaluating the utility of generating subquestions&#10;before answering a question, to analyzing the value of rationales and&#10;explanations, to investigating the importance of different parts of speech, and&#10;uncovering dataset gender bias.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.11955" label="2103.11955">
        <attvalues>
          <attvalue for="0" value="Improving and Simplifying Pattern Exploiting Training" />
          <attvalue for="1" value="  Recently, pre-trained language models (LMs) have achieved strong performance&#10;when fine-tuned on difficult benchmarks like SuperGLUE. However, performance&#10;can suffer when there are very few labeled examples available for fine-tuning.&#10;Pattern Exploiting Training (PET) is a recent approach that leverages patterns&#10;for few-shot learning. However, PET uses task-specific unlabeled data. In this&#10;paper, we focus on few-shot learning without any unlabeled data and introduce&#10;ADAPET, which modifies PET's objective to provide denser supervision during&#10;fine-tuning. As a result, ADAPET outperforms PET on SuperGLUE without any&#10;task-specific unlabeled data. Our code can be found at&#10;https://github.com/rrmenon10/ADAPET.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;Cloze-style questions and MLM. A cloze task is a problem where certain parts of a text are removed, and the goal is to replace the missing portion based on the context \cite{taylor1953cloze}. Here, the text that has some parts removed is considered a cloze-style question. Inspired by cloze tasks, BERT introduces the MLM objective that tries to predict the original word at the masked out positions in a cloze question.\\&#10;\\&#10;Notation. Let $G$ represent a language model, $x$ represent the input example converted into a cloze-style question, and $y$ represent the label at the masked location $m$. We are interested in the quantity $[\![G_m(x)]\!]_{z}$ which represents the logit value for a specific token $z$ at the mask location $m$. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2103.08493" label="2103.08493">
        <attvalues>
          <attvalue for="0" value="How Many Data Points is a Prompt Worth?" />
          <attvalue for="1" value="  When fine-tuning pretrained models for classification, researchers either use&#10;a generic model head or a task-specific prompt for prediction. Proponents of&#10;prompting have argued that prompts provide a method for injecting task-specific&#10;guidance, which is beneficial in low-data regimes. We aim to quantify this&#10;benefit through rigorous testing of prompts in a fair setting: comparing&#10;prompted and head-based fine-tuning in equal conditions across many tasks and&#10;data sizes. By controlling for many sources of advantage, we find that&#10;prompting does indeed provide a benefit, and that this benefit can be&#10;quantified per task. Results show that prompting is often worth 100s of data&#10;points on average across classification tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.05542" label="2108.05542">
        <attvalues>
          <attvalue for="0" value="AMMUS : A Survey of Transformer-based Pretrained Models in Natural&#10;  Language Processing" />
          <attvalue for="1" value="  Transformer-based pretrained language models (T-PTLMs) have achieved great&#10;success in almost every NLP task. The evolution of these models started with&#10;GPT and BERT. These models are built on the top of transformers,&#10;self-supervised learning and transfer learning. Transformed-based PTLMs learn&#10;universal language representations from large volumes of text data using&#10;self-supervised learning and transfer this knowledge to downstream tasks. These&#10;models provide good background knowledge to downstream tasks which avoids&#10;training of downstream models from scratch. In this comprehensive survey paper,&#10;we initially give a brief overview of self-supervised learning. Next, we&#10;explain various core concepts like pretraining, pretraining methods,&#10;pretraining tasks, embeddings and downstream adaptation methods. Next, we&#10;present a new taxonomy of T-PTLMs and then give brief overview of various&#10;benchmarks including both intrinsic and extrinsic. We present a summary of&#10;various useful libraries to work with T-PTLMs. Finally, we highlight some of&#10;the future research directions which will further improve these models. We&#10;strongly believe that this comprehensive survey paper will serve as a good&#10;reference to learn the core concepts as well as to stay updated with the recent&#10;happenings in T-PTLMs.&#10;" />
          <attvalue for="2" value="&#10;T{ransformer}-based pretrained language models (T-PTLMs) like GPT-1 \cite{radford2018improving}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ELECTRA \cite{clark2019electra}, T5 \cite{raffel2019exploring}, ALBERT \cite{lan2019albert}, BART \cite{lewis2020bart} and PEGAUSUS \cite{zhang2020pegasus} have achieved tremendous success in NLP because of their ability to learn universal language representations from large volumes of unlabeled text data and then transfer this knowledge to downstream tasks. In the early days, NLP systems are mostly rule-based which are later replaced by machine-learned models. Machine learning models require feature engineering which requires domain expertise and it is a time-consuming process too. The evolution of better computer hardware like GPUs and word embeddings like Word2Vec \cite{mikolov2013efficient} and Glove \cite{pennington2014glove} increased the use of deep learning models like CNN \cite{kalchbrenner2014convolutional} and RNN \cite{liu2016recurrent,zhou2016text} for building NLP systems. The main drawback with these deep learning models is the requirement of training the model from scratch except for the word embeddings. Training the model from scratch requires a large number of labeled instances which are expensive to generate. However, we expect the model to perform well using few labeled instances only. Transfer learning \cite{pan2009survey} allows the reuse of knowledge learned in source tasks to perform well in the target task. Here the target task should be similar to the source task. Based on the idea of transfer learning, researchers in Computer Vision trained large CNN models \cite{simonyan2014very,szegedy2016rethinking,he2016deep,tan2019efficientnet} using large scale labeled datasets like ImageNet \cite{krizhevsky2012imagenet,russakovsky2015imagenet}. These models learn image representations which are common across all the tasks. The large pretrained CNN models are adapted to downstream tasks by including few task-specific layers and then fine-tuned on the target datasets \cite{kaur2019automated}. As the pretrained CNN models provide good background knowledge to the downstream models, they enjoyed tremendous success in many CV tasks \cite{he2016deep,ren2015faster}.&#10;&#10;Deep learning models like CNN and RNN have difficulties in modelling long term contexts and learn the word representations with locality bias \cite{qiu2020pre}. Moreover, as RNNs process the input sequentially i.e., word by word, the utilization of parallel computer hardware is limited. To overcome these drawbacks in existing deep learning models, Vaswani et al. \cite{vaswani2017attention} proposed a deep learning model called Transformers which is completely based on self-attention. Self-attention allows for more parallelization compared to RNNs and can easily model long term contexts as every token attend to all the tokens in the input sequence \cite{vaswani2017attention}. Transformers contains a stack of encoder and decoder layers. With the help of a stack of encoder and decoder layers, transformers can learn complex language information. It is a very expensive and time-taking process to generate a large amount of labeled data in the NLP domain. However, it is very easy to get large volumes of unlabeled text data. NLP research community impressed with the success of CNN-based pretrained models in Computer Vision, have developed T-PTLMs by combining the power of transformers and self-supervised learning. Self-supervised learning allows the transformers to learn based on the pseudo supervision provided by one or more pretraining tasks. &#10;&#10;GPT and BERT are the first T-PTLMs developed based on transformer decoder and encoder layers respectively. Following GPT and BERT, models like XLNet, RoBERTa, ELECTRA, ALBERT, T5, BART and PEGAUSUS are proposed. Here XLNet, RoBERTa, ELECTRA and ALBERT are improvements over BERT model while T5, BART and PEGAUSUS are encoder-decoder based models. Kaplan et al. \cite{kaplan2020scaling} showed that the performance of T-PTLMs can be increased just by increasing the size of the model. This observation triggered the development of large-scale T-PTLMs like GPT-3 (175B) \cite{brown2020language}, PANGU- (200B) \cite{zeng2021pangu}, GShard (600B) \cite{lepikhin2020gshard} which contain billions of parameters and Switch-Transformers (1.6T) \cite{fedus2021switch} which contains trillions of parameters. Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}. T-PTLMs support transfer-learning also as these models can be adapted to downstream tasks by fine-tuning or prompt-tuning on target datasets. In this survey paper, we present a comprehensive review of recent research works related to T-PTLMs. We summarize the highlights of our survey as&#10;\begin{itemize}&#10; \item We present a brief overview of SSL, the backbone behind developing T-PTLMs (Section \ref{ssl-sec}).&#10; \item We explain various core concepts related to T-PTLMs like pretraining, pretraining methods, pretraining tasks, embeddings and downstream adaptation methods (Section \ref{core-concepts-sec}).&#10; \item We present a new taxonomy to categorize various T-PTLMs. This taxonomy is based on four perspectives namely pretraining corpus, architecture, type of SSL and extensions (Section \ref{taxonomy-sec}). &#10; \item We present a new taxonomy to categorize various downstream adaptation methods and explain each in detail (Section \ref{adaptation-methods-sec}).&#10; \item We present a brief overview of various benchmarks including both intrinsic and extrinsic which evaluate the progress of T-PTLMs (Section \ref{evaluation}).&#10; \item We present a brief overview of various libraries starting from Huggingface Transformers to Transformer-interpret which are useful to work T-PTLMs (Section \ref{libraries-sec}).&#10; \item We briefly discuss some of the future research directions which will drive the research community to further improve the models (Section \ref{future-directions}).&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Mathematics, Self-Supervised Learning, Language Model Evolution, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="1912.08777" label="1912.08777">
        <attvalues>
          <attvalue for="0" value="PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  Recent work pre-training Transformers with self-supervised objectives on&#10;large text corpora has shown great success when fine-tuned on downstream NLP&#10;tasks including text summarization. However, pre-training objectives tailored&#10;for abstractive text summarization have not been explored. Furthermore there is&#10;a lack of systematic evaluation across diverse domains. In this work, we&#10;propose pre-training large Transformer-based encoder-decoder models on massive&#10;text corpora with a new self-supervised objective. In PEGASUS, important&#10;sentences are removed/masked from an input document and are generated together&#10;as one output sequence from the remaining sentences, similar to an extractive&#10;summary. We evaluated our best PEGASUS model on 12 downstream summarization&#10;tasks spanning news, science, stories, instructions, emails, patents, and&#10;legislative bills. Experiments demonstrate it achieves state-of-the-art&#10;performance on all 12 downstream datasets measured by ROUGE scores. Our model&#10;also shows surprising performance on low-resource summarization, surpassing&#10;previous state-of-the-art results on 6 datasets with only 1000 examples.&#10;Finally we validated our results using human evaluation and show that our model&#10;summaries achieve human performance on multiple datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1404.2188" label="1404.2188">
        <attvalues>
          <attvalue for="0" value="A Convolutional Neural Network for Modelling Sentences" />
          <attvalue for="1" value="  The ability to accurately represent sentences is central to language&#10;understanding. We describe a convolutional architecture dubbed the Dynamic&#10;Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of&#10;sentences. The network uses Dynamic k-Max Pooling, a global pooling operation&#10;over linear sequences. The network handles input sentences of varying length&#10;and induces a feature graph over the sentence that is capable of explicitly&#10;capturing short and long-range relations. The network does not rely on a parse&#10;tree and is easily applicable to any language. We test the DCNN in four&#10;experiments: small scale binary and multi-class sentiment prediction, six-way&#10;question classification and Twitter sentiment prediction by distant&#10;supervision. The network achieves excellent performance in the first three&#10;tasks and a greater than 25% error reduction in the last task with respect to&#10;the strongest baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.06639" label="1611.06639">
        <attvalues>
          <attvalue for="0" value="Text Classification Improved by Integrating Bidirectional LSTM with&#10;  Two-dimensional Max Pooling" />
          <attvalue for="1" value="  Recurrent Neural Network (RNN) is one of the most popular architectures used&#10;in Natural Language Processsing (NLP) tasks because its recurrent structure is&#10;very suitable to process variable-length text. RNN can utilize distributed&#10;representations of words by first converting the tokens comprising each text&#10;into vectors, which form a matrix. And this matrix includes two dimensions: the&#10;time-step dimension and the feature vector dimension. Then most existing models&#10;usually utilize one-dimensional (1D) max pooling operation or attention-based&#10;operation only on the time-step dimension to obtain a fixed-length vector.&#10;However, the features on the feature vector dimension are not mutually&#10;independent, and simply applying 1D pooling operation over the time-step&#10;dimension independently may destroy the structure of the feature&#10;representation. On the other hand, applying two-dimensional (2D) pooling&#10;operation over the two dimensions may sample more meaningful features for&#10;sequence modeling tasks. To integrate the features on both dimensions of the&#10;matrix, this paper explores applying 2D max pooling operation to obtain a&#10;fixed-length representation of the text. This paper also utilizes 2D&#10;convolution to sample more meaningful information of the matrix. Experiments&#10;are conducted on six text classification tasks, including sentiment analysis,&#10;question classification, subjectivity classification and newsgroup&#10;classification. Compared with the state-of-the-art models, the proposed models&#10;achieve excellent performance on 4 out of 6 tasks. Specifically, one of the&#10;proposed models achieves highest accuracy on Stanford Sentiment Treebank binary&#10;classification and fine-grained classification tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1409.1556" label="1409.1556">
        <attvalues>
          <attvalue for="0" value="Very Deep Convolutional Networks for Large-Scale Image Recognition" />
          <attvalue for="1" value="  In this work we investigate the effect of the convolutional network depth on&#10;its accuracy in the large-scale image recognition setting. Our main&#10;contribution is a thorough evaluation of networks of increasing depth using an&#10;architecture with very small (3x3) convolution filters, which shows that a&#10;significant improvement on the prior-art configurations can be achieved by&#10;pushing the depth to 16-19 weight layers. These findings were the basis of our&#10;ImageNet Challenge 2014 submission, where our team secured the first and the&#10;second places in the localisation and classification tracks respectively. We&#10;also show that our representations generalise well to other datasets, where&#10;they achieve state-of-the-art results. We have made our two best-performing&#10;ConvNet models publicly available to facilitate further research on the use of&#10;deep visual representations in computer vision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.11946" label="1905.11946">
        <attvalues>
          <attvalue for="0" value="EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" />
          <attvalue for="1" value="  Convolutional Neural Networks (ConvNets) are commonly developed at a fixed&#10;resource budget, and then scaled up for better accuracy if more resources are&#10;available. In this paper, we systematically study model scaling and identify&#10;that carefully balancing network depth, width, and resolution can lead to&#10;better performance. Based on this observation, we propose a new scaling method&#10;that uniformly scales all dimensions of depth/width/resolution using a simple&#10;yet highly effective compound coefficient. We demonstrate the effectiveness of&#10;this method on scaling up MobileNets and ResNet.&#10;  To go even further, we use neural architecture search to design a new&#10;baseline network and scale it up to obtain a family of models, called&#10;EfficientNets, which achieve much better accuracy and efficiency than previous&#10;ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3%&#10;top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on&#10;inference than the best existing ConvNet. Our EfficientNets also transfer well&#10;and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%),&#10;and 3 other transfer learning datasets, with an order of magnitude fewer&#10;parameters. Source code is at&#10;https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.08097" label="2006.08097">
        <attvalues>
          <attvalue for="0" value="FinBERT: A Pretrained Language Model for Financial Communications" />
          <attvalue for="1" value="  Contextual pretrained language models, such as BERT (Devlin et al., 2019),&#10;have made significant breakthrough in various NLP tasks by training on large&#10;scale of unlabeled text re-sources.Financial sector also accumulates large&#10;amount of financial communication text.However, there is no pretrained finance&#10;specific language models available. In this work,we address the need by&#10;pretraining a financial domain specific BERT models, FinBERT, using a large&#10;scale of financial communication corpora. Experiments on three financial&#10;sentiment classification tasks confirm the advantage of FinBERT over generic&#10;domain BERT model. The code and pretrained models are available at&#10;https://github.com/yya518/FinBERT. We hope this will be useful for&#10;practitioners and researchers working on financial NLP tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.10386" label="2010.10386">
        <attvalues>
          <attvalue for="0" value="A Benchmark for Lease Contract Review" />
          <attvalue for="1" value="  Extracting entities and other useful information from legal contracts is an&#10;important task whose automation can help legal professionals perform contract&#10;reviews more efficiently and reduce relevant risks. In this paper, we tackle&#10;the problem of detecting two different types of elements that play an important&#10;role in a contract review, namely entities and red flags. The latter are terms&#10;or sentences that indicate that there is some danger or other potentially&#10;problematic situation for one or more of the signing parties. We focus on&#10;supporting the review of lease agreements, a contract type that has received&#10;little attention in the legal information extraction literature, and we define&#10;the types of entities and red flags needed for that task. We release a new&#10;benchmark dataset of 179 lease agreement documents that we have manually&#10;annotated with the entities and red flags they contain, and which can be used&#10;to train and test relevant extraction algorithms. Finally, we release a new&#10;language model, called ALeaseBERT, pre-trained on this dataset and fine-tuned&#10;for the detection of the aforementioned elements, providing a baseline for&#10;further research&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.04664" label="2102.04664">
        <attvalues>
          <attvalue for="0" value="CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding&#10;  and Generation" />
          <attvalue for="1" value="  Benchmark datasets have a significant impact on accelerating research in&#10;programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark&#10;dataset to foster machine learning research for program understanding and&#10;generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and&#10;a platform for model evaluation and comparison. CodeXGLUE also features three&#10;baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder&#10;models, to make it easy for researchers to use the platform. The availability&#10;of such data and baselines can help the development and validation of new&#10;methods that can be applied to various program understanding and generation&#10;problems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.08155" label="2002.08155">
        <attvalues>
          <attvalue for="0" value="CodeBERT: A Pre-Trained Model for Programming and Natural Languages" />
          <attvalue for="1" value="  We present CodeBERT, a bimodal pre-trained model for programming language&#10;(PL) and nat-ural language (NL). CodeBERT learns general-purpose&#10;representations that support downstream NL-PL applications such as natural&#10;language codesearch, code documentation generation, etc. We develop CodeBERT&#10;with Transformer-based neural architecture, and train it with a hybrid&#10;objective function that incorporates the pre-training task of replaced token&#10;detection, which is to detect plausible alternatives sampled from generators.&#10;This enables us to utilize both bimodal data of NL-PL pairs and unimodal data,&#10;where the former provides input tokens for model training while the latter&#10;helps to learn better generators. We evaluate CodeBERT on two NL-PL&#10;applications by fine-tuning model parameters. Results show that CodeBERT&#10;achieves state-of-the-art performance on both natural language code search and&#10;code documentation generation tasks. Furthermore, to investigate what type of&#10;knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and&#10;evaluate in a zero-shot setting where parameters of pre-trained models are&#10;fixed. Results show that CodeBERT performs better than previous pre-trained&#10;models on NL-PL probing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.08366" label="2009.08366">
        <attvalues>
          <attvalue for="0" value="GraphCodeBERT: Pre-training Code Representations with Data Flow" />
          <attvalue for="1" value="  Pre-trained models for programming language have achieved dramatic empirical&#10;improvements on a variety of code-related tasks such as code search, code&#10;completion, code summarization, etc. However, existing pre-trained models&#10;regard a code snippet as a sequence of tokens, while ignoring the inherent&#10;structure of code, which provides crucial code semantics and would enhance the&#10;code understanding process. We present GraphCodeBERT, a pre-trained model for&#10;programming language that considers the inherent structure of code. Instead of&#10;taking syntactic-level structure of code like abstract syntax tree (AST), we&#10;use data flow in the pre-training stage, which is a semantic-level structure of&#10;code that encodes the relation of &quot;where-the-value-comes-from&quot; between&#10;variables. Such a semantic-level structure is neat and does not bring an&#10;unnecessarily deep hierarchy of AST, the property of which makes the model more&#10;efficient. We develop GraphCodeBERT based on Transformer. In addition to using&#10;the task of masked language modeling, we introduce two structure-aware&#10;pre-training tasks. One is to predict code structure edges, and the other is to&#10;align representations between source code and code structure. We implement the&#10;model in an efficient way with a graph-guided masked attention function to&#10;incorporate the code structure. We evaluate our model on four tasks, including&#10;code search, clone detection, code translation, and code refinement. Results&#10;show that code structure and newly introduced pre-training tasks can improve&#10;GraphCodeBERT and achieves state-of-the-art performance on the four downstream&#10;tasks. We further show that the model prefers structure-level attentions over&#10;token-level attentions in the task of code search.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.08645" label="2105.08645">
        <attvalues>
          <attvalue for="0" value="CoTexT: Multi-task Learning with Code-Text Transformer" />
          <attvalue for="1" value="  We present CoTexT, a pre-trained, transformer-based encoder-decoder model&#10;that learns the representative context between natural language (NL) and&#10;programming language (PL). Using self-supervision, CoTexT is pre-trained on&#10;large programming language corpora to learn a general understanding of language&#10;and code. CoTexT supports downstream NL-PL tasks such as code&#10;summarizing/documentation, code generation, defect detection, and code&#10;debugging. We train CoTexT on different combinations of available PL corpus&#10;including both &quot;bimodal&quot; and &quot;unimodal&quot; data. Here, bimodal data is the&#10;combination of text and corresponding code snippets, whereas unimodal data is&#10;merely code snippets. We first evaluate CoTexT with multi-task learning: we&#10;perform Code Summarization on 6 different programming languages and Code&#10;Refinement on both small and medium size featured in the CodeXGLUE dataset. We&#10;further conduct extensive experiments to investigate CoTexT on other tasks&#10;within the CodeXGlue dataset, including Code Generation and Defect Detection.&#10;We consistently achieve SOTA results in these tasks, demonstrating the&#10;versatility of our models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00377" label="2105.00377">
        <attvalues>
          <attvalue for="0" value="MathBERT: A Pre-Trained Model for Mathematical Formula Understanding" />
          <attvalue for="1" value="  Large-scale pre-trained models like BERT, have obtained a great success in&#10;various Natural Language Processing (NLP) tasks, while it is still a challenge&#10;to adapt them to the math-related tasks. Current pre-trained models neglect the&#10;structural features and the semantic correspondence between formula and its&#10;context. To address these issues, we propose a novel pre-trained model, namely&#10;\textbf{MathBERT}, which is jointly trained with mathematical formulas and&#10;their corresponding contexts. In addition, in order to further capture the&#10;semantic-level structural features of formulas, a new pre-training task is&#10;designed to predict the masked formula substructures extracted from the&#10;Operator Tree (OPT), which is the semantic structural representation of&#10;formulas. We conduct various experiments on three downstream tasks to evaluate&#10;the performance of MathBERT, including mathematical information retrieval,&#10;formula topic classification and formula headline generation. Experimental&#10;results demonstrate that MathBERT significantly outperforms existing methods on&#10;all those three tasks. Moreover, we qualitatively show that this pre-trained&#10;model effectively captures the semantic-level structural information of&#10;formulas. To the best of our knowledge, MathBERT is the first pre-trained model&#10;for mathematical formula understanding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.03323" label="1904.03323">
        <attvalues>
          <attvalue for="0" value="Publicly Available Clinical BERT Embeddings" />
          <attvalue for="1" value="  Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT&#10;(Devlin et al., 2018) have dramatically improved performance for many natural&#10;language processing (NLP) tasks in recent months. However, these models have&#10;been minimally explored on specialty corpora, such as clinical text; moreover,&#10;in the clinical domain, no publicly-available pre-trained BERT models yet&#10;exist. In this work, we address this need by exploring and releasing BERT&#10;models for clinical text: one for generic clinical text and another for&#10;discharge summaries specifically. We demonstrate that using a domain-specific&#10;model yields performance improvements on three common clinical NLP tasks as&#10;compared to nonspecific embeddings. These domain-specific models are not as&#10;performant on two clinical de-identification tasks, and argue that this is a&#10;natural consequence of the differences between de-identified source text and&#10;synthetically non de-identified task text.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.15779" label="2007.15779">
        <attvalues>
          <attvalue for="0" value="Domain-Specific Language Model Pretraining for Biomedical Natural&#10;  Language Processing" />
          <attvalue for="1" value="  Pretraining large neural language models, such as BERT, has led to impressive&#10;gains on many natural language processing (NLP) tasks. However, most&#10;pretraining efforts focus on general domain corpora, such as newswire and Web.&#10;A prevailing assumption is that even domain-specific pretraining can benefit by&#10;starting from general-domain language models. In this paper, we challenge this&#10;assumption by showing that for domains with abundant unlabeled text, such as&#10;biomedicine, pretraining language models from scratch results in substantial&#10;gains over continual pretraining of general-domain language models. To&#10;facilitate this investigation, we compile a comprehensive biomedical NLP&#10;benchmark from publicly-available datasets. Our experiments show that&#10;domain-specific pretraining serves as a solid foundation for a wide range of&#10;biomedical NLP tasks, leading to new state-of-the-art results across the board.&#10;Further, in conducting a thorough evaluation of modeling choices, both for&#10;pretraining and task-specific fine-tuning, we discover that some common&#10;practices are unnecessary with BERT models, such as using complex tagging&#10;schemes in named entity recognition (NER). To help accelerate research in&#10;biomedical NLP, we have released our state-of-the-art pretrained and&#10;task-specific models for the community, and created a leaderboard featuring our&#10;BLURB benchmark (short for Biomedical Language Understanding &amp; Reasoning&#10;Benchmark) at https://aka.ms/BLURB.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.05474" label="1906.05474">
        <attvalues>
          <attvalue for="0" value="Transfer Learning in Biomedical Natural Language Processing: An&#10;  Evaluation of BERT and ELMo on Ten Benchmarking Datasets" />
          <attvalue for="1" value="  Inspired by the success of the General Language Understanding Evaluation&#10;benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE)&#10;benchmark to facilitate research in the development of pre-training language&#10;representations in the biomedicine domain. The benchmark consists of five tasks&#10;with ten datasets that cover both biomedical and clinical texts with different&#10;dataset sizes and difficulties. We also evaluate several baselines based on&#10;BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and&#10;MIMIC-III clinical notes achieves the best results. We make the datasets,&#10;pre-trained models, and codes publicly available at&#10;https://github.com/ncbi-nlp/BLUE_Benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.05840" label="1909.05840">
        <attvalues>
          <attvalue for="0" value="Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT" />
          <attvalue for="1" value="  Transformer based architectures have become de-facto models used for a range&#10;of Natural Language Processing tasks. In particular, the BERT based models&#10;achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However,&#10;BERT based models have a prohibitive memory footprint and latency. As a result,&#10;deploying BERT based models in resource constrained environments has become a&#10;challenging task. In this work, we perform an extensive analysis of fine-tuned&#10;BERT models using second order Hessian information, and we use our results to&#10;propose a novel method for quantizing BERT models to ultra low precision. In&#10;particular, we propose a new group-wise quantization scheme, and we use a&#10;Hessian based mix-precision method to compress the model further. We&#10;extensively test our proposed method on BERT downstream tasks of SST-2, MNLI,&#10;CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at&#10;most $2.3\%$ performance degradation, even with ultra-low precision&#10;quantization down to 2 bits, corresponding up to $13\times$ compression of the&#10;model parameters, and up to $4\times$ compression of the embedding table as&#10;well as activations. Among all tasks, we observed the highest performance loss&#10;for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as&#10;well as visualization, we show that this is related to the fact that current&#10;training/fine-tuning strategy of BERT does not converge for SQuAD.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.09418" label="1905.09418">
        <attvalues>
          <attvalue for="0" value="Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy&#10;  Lifting, the Rest Can Be Pruned" />
          <attvalue for="1" value="  Multi-head self-attention is a key component of the Transformer, a&#10;state-of-the-art architecture for neural machine translation. In this work we&#10;evaluate the contribution made by individual attention heads in the encoder to&#10;the overall performance of the model and analyze the roles played by them. We&#10;find that the most important and confident heads play consistent and often&#10;linguistically-interpretable roles. When pruning heads using a method based on&#10;stochastic gates and a differentiable relaxation of the L0 penalty, we observe&#10;that specialized heads are last to be pruned. Our novel pruning method removes&#10;the vast majority of heads without seriously affecting performance. For&#10;example, on the English-Russian WMT dataset, pruning 38 out of 48 encoder heads&#10;results in a drop of only 0.15 BLEU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.11556" label="1909.11556">
        <attvalues>
          <attvalue for="0" value="Reducing Transformer Depth on Demand with Structured Dropout" />
          <attvalue for="1" value="  Overparameterized transformer networks have obtained state of the art results&#10;in various natural language processing tasks, such as machine translation,&#10;language modeling, and question answering. These models contain hundreds of&#10;millions of parameters, necessitating a large amount of computation and making&#10;them prone to overfitting. In this work, we explore LayerDrop, a form of&#10;structured dropout, which has a regularization effect during training and&#10;allows for efficient pruning at inference time. In particular, we show that it&#10;is possible to select sub-networks of any depth from one large network without&#10;having to finetune them and with limited impact on performance. We demonstrate&#10;the effectiveness of our approach by improving the state of the art on machine&#10;translation, language modeling, summarization, question answering, and language&#10;understanding benchmarks. Moreover, we show that our approach leads to small&#10;BERT-like models of higher quality compared to training from scratch or using&#10;distillation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.08511" label="2401.08511">
        <attvalues>
          <attvalue for="0" value="The Gaps between Pre-train and Downstream Settings in Bias Evaluation&#10;  and Debiasing" />
          <attvalue for="1" value="  The output tendencies of Pre-trained Language Models (PLM) vary markedly&#10;before and after Fine-Tuning (FT) due to the updates to the model parameters.&#10;These divergences in output tendencies result in a gap in the social biases of&#10;PLMs. For example, there exits a low correlation between intrinsic bias scores&#10;of a PLM and its extrinsic bias scores under FT-based debiasing methods.&#10;Additionally, applying FT-based debiasing methods to a PLM leads to a decline&#10;in performance in downstream tasks. On the other hand, PLMs trained on large&#10;datasets can learn without parameter updates via In-Context Learning (ICL)&#10;using prompts. ICL induces smaller changes to PLMs compared to FT-based&#10;debiasing methods. Therefore, we hypothesize that the gap observed in&#10;pre-trained and FT models does not hold true for debiasing methods that use&#10;ICL. In this study, we demonstrate that ICL-based debiasing methods show a&#10;higher correlation between intrinsic and extrinsic bias scores compared to&#10;FT-based methods. Moreover, the performance degradation due to debiasing is&#10;also lower in the ICL case compared to that in the FT case.&#10;" />
          <attvalue for="2" value="&#10;&#10;PLMs learn not only beneficial information~\cite{peters-etal-2018-deep,devlin-etal-2019-bert,brown2020language,touvron2023llama} but also undesirable social biases such as gender, race, and religous biases that exist in the training data~\cite{sun-etal-2019-mitigating,liang-etal-2020-towards,10.1162/tacl_a_00434,zhou-etal-2022-sense,guo-etal-2022-auto}.&#10;Overall, two major approaches can be identified in the literature to elicit value from PLMs in downstream tasks: FT and ICL.&#10;FT adapts PLMs to specific tasks by updating parameters, while ICL uses prompts without modifying the model parameters.&#10;&#10;FT models diverge considerably from the original PLMs in their output distributions~\cite{chen-etal-2020-recall}.&#10;Similarly, the output distribution of a PLM is significantly affected by debiasing methods, because the parameters of the PLM are updated during the debiasing process.&#10;Debiasing accompanied by FT suffers substantial performance decline in downstream tasks compared to the original PLM~\cite{meade-etal-2022-empirical,kaneko2023impact,oba2023contextual}.&#10;This is because the beneficial information learnt during pre-training is lost duringdebiasing.&#10;Furthermore, bias evaluations exhibit a weak-level of correlation between pre-trained and FT PLMs~\cite{goldfarb-tarrant-etal-2021-intrinsic,kaneko-etal-2022-debiasing,cao-etal-2022-intrinsic}.&#10;&#10;On the other hand, it is not obvious whether the prevalent wisdom regarding bias in such FT regimes similarly pertains to ICL, devoid of concomitant model updates.&#10;The absence of parameter updates precludes the elimination of beneficial encodings, thereby minimizing adverse impacts on downstream task effectiveness.&#10;ICL strategies for mitigating biases may thus pose superior viability through obviation of representational damage.&#10;Moreover, we hypothesize that bias evaluations contingent on the pre-training and downstream tasks exhibit heightened correlations, because the ICL-based debiasing methods protect the model parameters.&#10;&#10;In this paper, we investigate the performance gap of debiasing methods when applied to downstream tasks in an ICL setting.&#10;Additionally, we examine the correlation between bias evaluations for pre-training and downstream tasks enabled by the parameter sharing of ICL.&#10;Our experimental results show that ICL has a smaller gap than the FT setting wrt performance degradation of debiasing and correlation between evaluations in pre-training and downstream tasks.&#10;Therefore, we expect this paper to contribute by cautioning the community against directly applying trends from pre-training and downstream tasks with FT to ICL without careful considerations.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Fine-Tuning Methods, Computer Science, Linguistics, Language Model Bias, Artificial Intelligence, Debiasing Techniques, Psychology" />
        </attvalues>
      </node>
      <node id="2309.09092" label="2309.09092">
        <attvalues>
          <attvalue for="0" value="The Impact of Debiasing on the Performance of Language Models in&#10;  Downstream Tasks is Underestimated" />
          <attvalue for="1" value="  Pre-trained language models trained on large-scale data have learned serious&#10;levels of social biases. Consequently, various methods have been proposed to&#10;debias pre-trained models. Debiasing methods need to mitigate only&#10;discriminatory bias information from the pre-trained models, while retaining&#10;information that is useful for the downstream tasks. In previous research,&#10;whether useful information is retained has been confirmed by the performance of&#10;downstream tasks in debiased pre-trained models. On the other hand, it is not&#10;clear whether these benchmarks consist of data pertaining to social biases and&#10;are appropriate for investigating the impact of debiasing. For example in&#10;gender-related social biases, data containing female words (e.g. ``she, female,&#10;woman''), male words (e.g. ``he, male, man''), and stereotypical words (e.g.&#10;``nurse, doctor, professor'') are considered to be the most affected by&#10;debiasing. If there is not much data containing these words in a benchmark&#10;dataset for a target task, there is the possibility of erroneously evaluating&#10;the effects of debiasing. In this study, we compare the impact of debiasing on&#10;performance across multiple downstream tasks using a wide-range of benchmark&#10;datasets that containing female, male, and stereotypical words. Experiments&#10;show that the effects of debiasing are consistently \emph{underestimated}&#10;across all tasks. Moreover, the effects of debiasing could be reliably&#10;evaluated by separately considering instances containing female, male, and&#10;stereotypical words than all of the instances in a benchmark dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.15859" label="2012.15859">
        <attvalues>
          <attvalue for="0" value="Intrinsic Bias Metrics Do Not Correlate with Application Bias" />
          <attvalue for="1" value="  Natural Language Processing (NLP) systems learn harmful societal biases that&#10;cause them to amplify inequality as they are deployed in more and more&#10;situations. To guide efforts at debiasing these systems, the NLP community&#10;relies on a variety of metrics that quantify bias in models. Some of these&#10;metrics are intrinsic, measuring bias in word embedding spaces, and some are&#10;extrinsic, measuring bias in downstream tasks that the word embeddings enable.&#10;Do these intrinsic and extrinsic metrics correlate with each other? We compare&#10;intrinsic and extrinsic metrics across hundreds of trained models covering&#10;different tasks and experimental conditions. Our results show no reliable&#10;correlation between these metrics that holds in all scenarios across tasks and&#10;languages. We urge researchers working on debiasing to focus on extrinsic&#10;measures of bias, and to make using these measures more feasible via creation&#10;of new challenge sets and annotated test data. To aid this effort, we release&#10;code, a new intrinsic metric, and an annotated test set focused on gender bias&#10;in hate speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.02938" label="2210.02938">
        <attvalues>
          <attvalue for="0" value="Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and&#10;  their Social Biases in Downstream Tasks" />
          <attvalue for="1" value="  We study the relationship between task-agnostic intrinsic and task-specific&#10;extrinsic social bias evaluation measures for Masked Language Models (MLMs),&#10;and find that there exists only a weak correlation between these two types of&#10;evaluation measures. Moreover, we find that MLMs debiased using different&#10;methods still re-learn social biases during fine-tuning on downstream tasks. We&#10;identify the social biases in both training instances as well as their assigned&#10;labels as reasons for the discrepancy between intrinsic and extrinsic bias&#10;evaluation measurements. Overall, our findings highlight the limitations of&#10;existing MLM bias evaluation measures and raise concerns on the deployment of&#10;MLMs in downstream applications using those measures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.13928" label="2203.13928">
        <attvalues>
          <attvalue for="0" value="On the Intrinsic and Extrinsic Fairness Evaluation Metrics for&#10;  Contextualized Language Representations" />
          <attvalue for="1" value="  Multiple metrics have been introduced to measure fairness in various natural&#10;language processing tasks. These metrics can be roughly categorized into two&#10;categories: 1) \emph{extrinsic metrics} for evaluating fairness in downstream&#10;applications and 2) \emph{intrinsic metrics} for estimating fairness in&#10;upstream contextualized language representation models. In this paper, we&#10;conduct an extensive correlation study between intrinsic and extrinsic metrics&#10;across bias notions using 19 contextualized language models. We find that&#10;intrinsic and extrinsic metrics do not necessarily correlate in their original&#10;setting, even when correcting for metric misalignments, noise in evaluation&#10;datasets, and confounding factors such as experiment configuration for&#10;extrinsic metrics. %al&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.04611" label="2005.04611">
        <attvalues>
          <attvalue for="0" value="How Context Affects Language Models' Factual Predictions" />
          <attvalue for="1" value="  When pre-trained on large unsupervised textual corpora, language models are&#10;able to store and retrieve factual knowledge to some extent, making it possible&#10;to use them directly for zero-shot cloze-style question answering. However,&#10;storing factual knowledge in a fixed number of weights of a language model&#10;clearly has limitations. Previous approaches have successfully provided access&#10;to information outside the model weights using supervised architectures that&#10;combine an information retrieval system with a machine reading component. In&#10;this paper, we go a step further and integrate information from a retrieval&#10;system with a pre-trained language model in a purely unsupervised way. We&#10;report that augmenting pre-trained language models in this way dramatically&#10;improves performance and that the resulting system, despite being unsupervised,&#10;is competitive with a supervised machine reading baseline. Furthermore,&#10;processing query and context with different segment tokens allows BERT to&#10;utilize its Next Sentence Prediction pre-trained classifier to determine&#10;whether the context is relevant or not, substantially improving BERT's&#10;zero-shot cloze-style question-answering performance and making its predictions&#10;robust to noisy contexts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Language Model Limitations, Information Retrieval Integration, Artificial Intelligence, Unsupervised Question Answering" />
        </attvalues>
      </node>
      <node id="2307.01323" label="2307.01323">
        <attvalues>
          <attvalue for="0" value="Semantic enrichment towards efficient speech representations" />
          <attvalue for="1" value="  Over the past few years, self-supervised learned speech representations have&#10;emerged as fruitful replacements for conventional surface representations when&#10;solving Spoken Language Understanding (SLU) tasks. Simultaneously, multilingual&#10;models trained on massive textual data were introduced to encode language&#10;agnostic semantics. Recently, the SAMU-XLSR approach introduced a way to make&#10;profit from such textual models to enrich multilingual speech representations&#10;with language agnostic semantics. By aiming for better semantic extraction on a&#10;challenging Spoken Language Understanding task and in consideration with&#10;computation costs, this study investigates a specific in-domain semantic&#10;enrichment of the SAMU-XLSR model by specializing it on a small amount of&#10;transcribed data from the downstream task. In addition, we show the benefits of&#10;the use of same-domain French and Italian benchmarks for low-resource language&#10;portability and explore cross-domain capacities of the enriched SAMU-XLSR.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken language understanding (SLU) consists of various Natural Language Processing tasks that extract semantics from speech~\cite{tur2011spoken}, such as call routing, named entity recognition from speech, or slot filling tasks in the context of human-machine dialogue. &#10;This work focuses on end-to-end neural approaches for speech-to-concept, one of the most challenging SLU tasks.&#10;It distinguishes itself from conventional cascade approaches~\cite{liu2020mockingjay,liu2021tera} by using a single neural model to directly extract the semantics from speech signals~\cite{ghannay2018end, haghani2018audio,serdyuk2018towards} with the advantages of: (1) jointly optimizing the Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) parts, and (2) mitigating error propagation. &#10;Nonetheless, end-to-end models' main challenge resides in the lack of bimodal annotated data, i.e. audio speech recordings with semantic manual annotations. &#10;To overcome it, transfer learning techniques~\cite{bhosale2019end,Caubriere2019,huang2020leveraging} and artificial augmentation of training data with speech synthesis~\cite{desot2020corpus,lugosch2020using} have been proposed.&#10;&#10;In this paper, we investigated a recently proposed approach to remedy the aforementioned problem with the use of self-supervised learning (SSL) for the SLU task. &#10;SSL models, which are pre-trained from huge amounts of unlabelled data, have lately become very trendy as they show promising results in a wide range of speech tasks~\cite{baevski2020wav2vec,devlin-etal-2019-bert} when substantially alleviating the need of costly annotated speech data. &#10;At the same time, similar ideas had been successfully applied to text to allow semantics information extraction~\cite{devlin-etal-2019-bert,feng2022language}. &#10;Several attempts to unify both textual and speech modalities can be found in~\cite{huang2020leveraging,agrawal2022tie,muller2021pursuit}. &#10;Inspired by this new challenge, ~\cite{khurana2022samu} proposed a framework named SAMU-XLSR (Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation) which produces a semantically-aligned multimodal and multilingual sentence-level representation. &#10;To do so, the authors combined the well-known multilingual frame-level speech representation learning model XLS-R~\cite{xlsr} with the Language Agnostic BERT Sentence Embedding generator LaBSE~\cite{feng2022language}. &#10;More interestingly, ~\cite{laperriere2022use} shows that SAMU-XLSR can also be used as a frame-level speech encoder for a challenging end-to-end SLU task when they find that this model might create semantically aware frame-level speech representations. &#10;&#10;This study shows that specializing SAMU-XLSR representations, by exploiting a small amount of transcribed data without costly semantic annotation, offers very strong semantics extraction enhancements. &#10;Another main discovery lies in the scoring equivalence obtained with computational cost-effective experiments following layer-wise analysis of enriched SSL models. &#10;We finally investigate how different portabilities on same-domain or same-language data could be beneficial in order to make the semantically enriched representations more accurate.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Spoken Language Understanding, Speech Representation Learning, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1802.08395" label="1802.08395">
        <attvalues>
          <attvalue for="0" value="Towards end-to-end spoken language understanding" />
          <attvalue for="1" value="  Spoken language understanding system is traditionally designed as a pipeline&#10;of a number of components. First, the audio signal is processed by an automatic&#10;speech recognizer for transcription or n-best hypotheses. With the recognition&#10;results, a natural language understanding system classifies the text to&#10;structured data as domain, intent and slots for down-streaming consumers, such&#10;as dialog system, hands-free applications. These components are usually&#10;developed and optimized independently. In this paper, we present our study on&#10;an end-to-end learning system for spoken language understanding. With this&#10;unified approach, we can infer the semantic meaning directly from audio&#10;features without the intermediate text representation. This study showed that&#10;the trained model can achieve reasonable good result and demonstrated that the&#10;model can capture the semantic attention directly from the audio features.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.04284" label="2010.04284">
        <attvalues>
          <attvalue for="0" value="Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent&#10;  Systems" />
          <attvalue for="1" value="  Training an end-to-end (E2E) neural network speech-to-intent (S2I) system&#10;that directly extracts intents from speech requires large amounts of&#10;intent-labeled speech data, which is time consuming and expensive to collect.&#10;Initializing the S2I model with an ASR model trained on copious speech data can&#10;alleviate data sparsity. In this paper, we attempt to leverage NLU text&#10;resources. We implemented a CTC-based S2I system that matches the performance&#10;of a state-of-the-art, traditional cascaded SLU system. We performed controlled&#10;experiments with varying amounts of speech and text training data. When only a&#10;tenth of the original data is available, intent classification accuracy&#10;degrades by 7.6% absolute. Assuming we have additional text-to-intent data&#10;(without speech) available, we investigated two techniques to improve the S2I&#10;system: (1) transfer learning, in which acoustic embeddings for intent&#10;classification are tied to fine-tuned BERT text embeddings; and (2) data&#10;augmentation, in which the text-to-intent data is converted into&#10;speech-to-intent data using a multi-speaker text-to-speech system. The proposed&#10;approaches recover 80% of performance lost due to using limited intent-labeled&#10;speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.09463" label="1910.09463">
        <attvalues>
          <attvalue for="0" value="Using Speech Synthesis to Train End-to-End Spoken Language Understanding&#10;  Models" />
          <attvalue for="1" value="  End-to-end models are an attractive new approach to spoken language&#10;understanding (SLU) in which the meaning of an utterance is inferred directly&#10;from the raw audio without employing the standard pipeline composed of a&#10;separately trained speech recognizer and natural language understanding module.&#10;The downside of end-to-end SLU is that in-domain speech data must be recorded&#10;to train the model. In this paper, we propose a strategy for overcoming this&#10;requirement in which speech synthesis is used to generate a large synthetic&#10;training dataset from several artificial speakers. Experiments on two&#10;open-source SLU datasets confirm the effectiveness of our approach, both as a&#10;sole source of training data and as a form of data augmentation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.09044" label="2011.09044">
        <attvalues>
          <attvalue for="0" value="Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end&#10;  Spoken Language Understanding" />
          <attvalue for="1" value="  End-to-end (E2E) spoken language understanding (SLU) systems can infer the&#10;semantics of a spoken utterance directly from an audio signal. However,&#10;training an E2E system remains a challenge, largely due to the scarcity of&#10;paired audio-semantics data. In this paper, we treat an E2E system as a&#10;multi-modal model, with audio and text functioning as its two modalities, and&#10;use a cross-modal latent space (CMLS) architecture, where a shared latent space&#10;is learned between the `acoustic' and `text' embeddings. We propose using&#10;different multi-modal losses to explicitly guide the acoustic embeddings to be&#10;closer to the text embeddings, obtained from a semantically powerful&#10;pre-trained BERT model. We train the CMLS model on two publicly available E2E&#10;datasets, across different cross-modal losses and show that our proposed&#10;triplet loss function achieves the best performance. It achieves a relative&#10;improvement of 1.4% and 4% respectively over an E2E model without a cross-modal&#10;space and a relative improvement of 0.7% and 1% over a previously published&#10;CMLS model using $L_2$ loss. The gains are higher for a smaller, more&#10;complicated E2E dataset, demonstrating the efficacy of using an efficient&#10;cross-modal loss function, especially when there is limited E2E training data&#10;available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.08180" label="2205.08180">
        <attvalues>
          <attvalue for="0" value="SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual&#10;  Speech Representation" />
          <attvalue for="1" value="  We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level&#10;Cross-Lingual Speech Representation learning framework. Unlike previous works&#10;on speech representation learning, which learns multilingual contextual speech&#10;embedding at the resolution of an acoustic frame (10-20ms), this work focuses&#10;on learning multimodal (speech-text) multilingual speech embedding at the&#10;resolution of a sentence (5-10s) such that the embedding vector space is&#10;semantically aligned across different languages. We combine state-of-the-art&#10;multilingual acoustic frame-level speech representation learning model XLS-R&#10;with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an&#10;utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we&#10;train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual&#10;speech-text and speech-speech associations emerge in its learned representation&#10;space. To substantiate our claims, we use SAMU-XLSR speech encoder in&#10;combination with a pre-trained LaBSE text sentence encoder for cross-lingual&#10;speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual&#10;speech-to-speech translation retrieval. We highlight these applications by&#10;performing several cross-lingual text and speech translation retrieval tasks&#10;across several datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.05291" label="2210.05291">
        <attvalues>
          <attvalue for="0" value="On the Use of Semantically-Aligned Speech Representations for Spoken&#10;  Language Understanding" />
          <attvalue for="1" value="  In this paper we examine the use of semantically-aligned speech&#10;representations for end-to-end spoken language understanding (SLU). We employ&#10;the recently-introduced SAMU-XLSR model, which is designed to generate a single&#10;embedding that captures the semantics at the utterance level, semantically&#10;aligned across different languages. This model combines the acoustic&#10;frame-level speech representation learning model (XLS-R) with the Language&#10;Agnostic BERT Sentence Embedding (LaBSE) model. We show that the use of the&#10;SAMU-XLSR model instead of the initial XLS-R model improves significantly the&#10;performance in the framework of end-to-end SLU. Finally, we present the&#10;benefits of using this model towards language portability in SLU.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken language understanding (SLU) refers to natural language processing tasks related to semantic extraction from speech~\cite{tur2011spoken}.&#10;Different tasks can be addressed as SLU tasks, such as named entity recognition from speech, call routing, slot filling task in a context of human-machine dialogue. &#10;&#10;To our knowledge, end-to-end neural approaches have been proposed four years ago in order to directly extract the semantics from speech signal, by using a single neural model~\cite{ghannay2018end, haghani2018audio,serdyuk2018towards}, instead of applying a classical cascade approach based on the use of an automatic speech recognition (ASR) system, followed by a natural language understanding processing (NLU) module applied to the automatic transcription~\cite{tur2011spoken}.&#10;Two are the main advantages of end-to-end approaches. The first one is related to the joint optimization of the ASR and NLU part, since the unique neural model is optimized only for the final SLU task.&#10;The second one is the mitigation of error propagation: when using a cascade approach, errors generated by the first modules propagate to the following ones. &#10;&#10;Since 2018, end-to-end approaches have became very popular in the SLU literature~\cite{desot2019slu,dinarelli2020data,radfar2020end,palogiannidi2020end,poncelet2021low}.&#10;A main issue of these approaches is the lack of bimodal annotated data (speech audio recordings with semantic manual annotation). Several methods have been proposed in order to address this issue, e.g. transfer learning techniques~\cite{bhosale2019end,Caubriere2019}, \cite{huang2020leveraging} or artificial augmentation of the training data using speech synthesis~\cite{desot2020corpus,lugosch2020using}.&#10;&#10;Self-supervised learning (SSL), that benefits from unlabelled data, recently opened new perspectives for automatic speech recognition and natural language processing~\cite{baevski2020wav2vec,devlin-etal-2019-bert}. &#10;SSL has been successfully applied to several SLU tasks, especially through cascade approaches~\cite{laperriere2021we}: the ASR system benefits from learning better speech unit representations~\cite{liu2020mockingjay,liu2021tera,hsu2021hubert} while the NLU module benefits from BERT-like models~\cite{devlin-etal-2019-bert}.&#10;The use of an end-to-end approach exploiting directly both speech and text SSL models is limited by the difficulty to unify the speech and textual representation spaces, in addition to the complexity of managing a huge number of model parameters.&#10;Some approaches have been proposed in order to exploit the BERT-like capabilities within an end-to-end SLU model, e.g. by projecting some kinds of sequences of embeddings extracted by an ASR sub-module to a BERT model~\cite{wang2020large,chung2021splat}, or by tying at the sentence level the acoustic embeddings to a SLU fine-tuned BERT model for a speech intent detection task~\cite{huang2020leveraging,agrawal2022tie}.&#10;In~\cite{muller2021pursuit}, a similar approach is extended in order to build a multilingual end-to-end SLU model, again for speech intent detection.&#10;&#10;Earlier this year, a new promising model was introduced in~\cite{khurana2022samu}. &#10;The model combines a state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R~\cite{babu2021xls} with the Language Agnostic BERT Sentence Embedding~\cite{feng2022language} (LaBSE) model to create an utterance-level multimodal multilingual speech encoder. &#10;This model is named SAMU-XLSR, for Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework.&#10;&#10;In this paper, we analyze the performance and the behavior of the SAMU-XLSR model using the French MEDIA benchmark dataset, which is considered as a very challenging benchmarks for SLU~\cite{bechet2019benchmarking}.&#10;Moreover, by using the Italian PortMEDIA corpus~\cite{lefevre2012leveraging}, we also investigate the potential of porting an existing end-to-end SLU model from one language (French) to another (Italian) through two scenarios concerning the target language: zero-shot or low-resource learning.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.00946" label="2308.00946">
        <attvalues>
          <attvalue for="0" value="Teaching Smaller Language Models To Generalise To Unseen Compositional&#10;  Questions" />
          <attvalue for="1" value="  We equip a smaller Language Model to generalise to answering challenging&#10;compositional questions that have not been seen in training. To do so we&#10;propose a combination of multitask supervised pretraining on up to 93 tasks&#10;designed to instill diverse reasoning abilities, and a dense retrieval system&#10;that aims to retrieve a set of evidential paragraph fragments. Recent progress&#10;in question-answering has been achieved either through prompting methods&#10;against very large pretrained Language Models in zero or few-shot fashion, or&#10;by fine-tuning smaller models, sometimes in conjunction with information&#10;retrieval. We focus on the less explored question of the extent to which&#10;zero-shot generalisation can be enabled in smaller models with retrieval&#10;against a corpus within which sufficient information to answer a particular&#10;question may not exist. We establish strong baselines in this setting for&#10;diverse evaluation datasets (StrategyQA, CommonsenseQA, IIRC, DROP, Musique and&#10;ARC-DA), and show that performance can be significantly improved by adding&#10;retrieval-augmented training datasets which are designed to expose our models&#10;to a variety of heuristic reasoning strategies such as weighing partial&#10;evidence or ignoring an irrelevant context.&#10;" />
          <attvalue for="2" value="&#10;We are inspired by recent progress with pretrained large Language Models (LLM), which when prompted with task demonstrations \cite{Brown2020-rl}, instructions \cite{Sanh2021-na, Wei2021-go, Ouyang2022-ti} or reasoning chains \cite{Wei2022-lz}, show an ability to answer questions unlikely to have been encountered during training. However a diversity of potential applications exist in the broad domain of reasoning systems and considerations such as latency, cost, energy efficiency, physical compute size and internet connectivity requirements are relevant in determining the most appropriate approach for a given situation.&#10;&#10;Rather than encoding all knowledge in the parameters of a LLM, an alternative approach has been to transform the original question-answering problem into a reading comprehension (RC) problem by retrieving relevant information for answering a particular query from an external corpus, and training a smaller model (QA Model) to reason over the concatenation of the query and retrieved information to derive an answer e.g. \cite{Chen2017-gw}. In this paper we extend retrieval methods as described in section\ref{sys_comps} in conjunction with a supervised multitask pretraining regime for the QA Model involving 79 tasks for our baseline and 93 tasks for the improved model.&#10;&#10;The viability of this approach outside of fine-tuned settings is currently subject to limitations, both in the retrieval component, as discussed below, and with respect to the inabilities of smaller language models to perform the reasoning function as well as larger models. We aim to quantify performance limitations and evaluate mitigations for some of them. &#10;&#10;There are at least two significant challenges in retrieval to be overcome. Firstly, no matter how large the corpus is, there will always be missing information, particularly so in our setting where neither datasets nor corpus have been normalised such that sufficient information is in the corpus to make each question answerable through deductively valid means. Secondly, as long as humans ask questions with ambiguous references e.g. ``Who is the spouse of the Green performer?'' \cite{Trivedi2022-mv}, retrieval will necessarily be imperfect even where sufficient knowledge exists in the corpus and the retrieval method is otherwise perfect. &#10;&#10;We evaluate a method for addressing these issues. Specifically, we measure the effect of adding datasets to our QA Model training regime that are designed to impart heuristic strategies for reasoning to a plausible rather than an entailed answer. We construct these datasets by building contexts for training questions using our retrieval system against a fixed corpus of English Wikipedia paragraphs. The resulting samples (``retrieval-augmented training datasets'', abbreviated to RATD) are included in training our QA Model irrespective of whether they contain partial, full, or no evidence. Our approach carries the advantage that a diversity of reasoning strategies may be imparted. Such strategies include ignoring an irrelevant context completely or weighing partially evidential facts; e.g. reasoning toward answering ``Do teenagers always rebel against their parents?'' \cite{Talmor2021-al} can be aided by the retrieval of knowledge that ``Adolescents who have a good relationship with their parents are less likely to engage in various risk behaviours'', even though there is no entailment implied. &#10;&#10;Generally our method is most applicable to question-answering tasks where the desired answer is short i.e. from a word to a short sentence, and the question itself does not come already supplied with a fully evidential context. We also assume that it is possible to retrieve sufficient information from our corpus so as to make a question answerable within a modest sequence length (we limit ourselves to a 512 token maximum) e.g. we are unlikely to be able to answer a question such as ``How many songs have a person's name in the title?'' even through retrieving every instance is theoretically possible. We focus our study on a set of unseen evaluation datasets that meet the following criteria: (1) Datasets collectively involve diverse textual and numerical reasoning strategies. (2) Questions are generally readily answerable by humans with access to a web browser and without specialised knowledge. (3) Questions tend to be compositional. (4) Relevant comparison with prior work exists. In regards to defining compositionality, others e.g. \cite{Dankers2022-bw,Hupkes2020-iu} have noted challenges in singularly defining compositionality as it relates to NLP; for our purposes we pragmatically define a question as compositional if it is unlikely to be answerable by our QA Model with a memorised answer from a similar training example, and requires reasoning over a context by utilising at least one reasoning operation (e.g. conjunction) involving more than one textual fact, and/or at least one numerical operation involving more than one number.&#10;&#10;This criteria leads us to select six evaluation datasets: StrategyQA \cite{Geva2021-sl} contains commonsense samples requiring diverse multi-hop reasoning strategies. On average samples require content from 2.33 separate paragraphs to answer when considering retrieval from Wikipedia. Musique \cite{Trivedi2022-mv} is a multi-hop dataset focused on factual questions that require retrieval of content from two to four paragraphs. IIRC \cite{Ferguson2020-hv} contains questions where an initial paragraph is given and answers depend upon reasoning over this plus one to over four additional paragraphs that must be retrieved. ARC-DA \cite{Bhakthavatsalam2021-fq} is a question-only subset of ARC \cite{Clark2018-gy}. The Worldtree database provides explanatory fact sets for ARC samples which average six facts per sample \cite{Xie2020-xb}. DROP \cite{Dua2019-td} is a RC dataset wherein answering each question requires numerical or temporal reasoning over a provided context to reach an often abstractive answer e.g. ``How many field goals were scored in the first quarter?...The Lions scored first...with a 23-yard field goal...The Buccaneers tied it up with a 38-yard field goal...then took the lead...The Lions responded with a 28-yard field goal...'' The answer is 3 which isn't explicit in the context. CommonsenseQA \cite{Talmor2019-rm} contains samples that are often unlikely to be answerable by finding a singular fact e.g. ``I’m crossing the river, my feet are wet but my body is dry, where am I? (A) waterfall (B) bridge (C) valley (D) bank (E) island'' is answered by considering knowledge related to each option. These datasets are discussed more fully in section\ref{sec:unseen_eval_datasets_desc}. &#10;&#10;In addition to the possibility of answer leakage from directly memorised samples, it has been shown that models are able to utilise more subtle cues such as the writing style of a particular annotator who contributed to both train and test splits for better results than are achievable where the test split is truly independent of the training split \cite{Geva2019-ll}. To minimise such issues as well as to facilitate comparison in a similar setting as other zero/few shot studies which have varying definitions of ``seen-ness'', we simply define an unseen question as one from an evaluation dataset that is disjoint from our training datasets. Against this definition, two of our evaluation datasets, ARC-DA and Musique, are ``partially seen'' as discussed further below.&#10;&#10;In summary the major contributions of this paper are: (A) We offer what is to our knowledge the most comprehensive set of baselines evaluating smaller Language Model zero-shot reasoning abilities published to date. (B) We show that augmenting the training regime with RATD datasets significantly improves performance from the baselines. (C) We demonstrate that training for numerical literacy and unanswerability is brittle in the unseen setting in the absence of sufficiently similarly formatted training examples. (D) We propose effective extensions to the retrieval approach as described below. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Generalization, Multitask Learning Strategies, Artificial Intelligence, Question Answering Systems, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2201.05320" label="2201.05320">
        <attvalues>
          <attvalue for="0" value="CommonsenseQA 2.0: Exposing the Limits of AI through Gamification" />
          <attvalue for="1" value="  Constructing benchmarks that test the abilities of modern natural language&#10;understanding models is difficult - pre-trained language models exploit&#10;artifacts in benchmarks to achieve human parity, but still fail on adversarial&#10;examples and make errors that demonstrate a lack of common sense. In this work,&#10;we propose gamification as a framework for data construction. The goal of&#10;players in the game is to compose questions that mislead a rival AI while using&#10;specific phrases for extra points. The game environment leads to enhanced user&#10;engagement and simultaneously gives the game designer control over the&#10;collected data, allowing us to collect high-quality data at scale. Using our&#10;method we create CommonsenseQA 2.0, which includes 14,343 yes/no questions, and&#10;demonstrate its difficulty for models that are orders-of-magnitude larger than&#10;the AI used in the game itself. Our best baseline, the T5-based Unicorn with&#10;11B parameters achieves an accuracy of 70.2%, substantially higher than GPT-3&#10;(52.9%) in a few-shot inference setup. Both score well below human performance&#10;which is at 94.1%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.05885" label="2108.05885">
        <attvalues>
          <attvalue for="0" value="The paradox of the compositionality of natural language: a neural&#10;  machine translation case study" />
          <attvalue for="1" value="  Obtaining human-like performance in NLP is often argued to require&#10;compositional generalisation. Whether neural networks exhibit this ability is&#10;usually studied by training models on highly compositional synthetic data.&#10;However, compositionality in natural language is much more complex than the&#10;rigid, arithmetic-like version such data adheres to, and artificial&#10;compositionality tests thus do not allow us to determine how neural models deal&#10;with more realistic forms of compositionality. In this work, we re-instantiate&#10;three compositionality tests from the literature and reformulate them for&#10;neural machine translation (NMT). Our results highlight that: i) unfavourably,&#10;models trained on more data are more compositional; ii) models are sometimes&#10;less compositional than expected, but sometimes more, exemplifying that&#10;different levels of compositionality are required, and models are not always&#10;able to modulate between them correctly; iii) some of the non-compositional&#10;behaviours are mistakes, whereas others reflect the natural variation in data.&#10;Apart from an empirical study, our work is a call to action: we should rethink&#10;the evaluation of compositionality in neural networks and develop benchmarks&#10;using real data to evaluate compositionality on natural language, where&#10;composing meaning is not as straightforward as doing the math.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In previous work, a variety of artificial tasks have been proposed to evaluate compositional generalisation using non-i.i.d.\ test sets that are designed to assess a specific characteristic of compositional behaviour.&#10;Examples are systematicity \cite{lake2018generalization, bastings2018jump, hupkes2020compositionality}, substitutivity \cite{mul2019siamese,hupkes2020compositionality}, localism \cite{hupkes2020compositionality,saphra-lopez-2020-lstms}, productivity \cite{lake2018generalization} or overgeneralisation \cite{korrel2019transcoding,hupkes2020compositionality,dankers-etal-2021-generalising}. &#10;Generally, neural models struggle to generalise in such evaluation setups.&#10;&#10;There are also studies that consider compositional generalisation on more natural data.&#10;Such studies typically focus on either MT \cite{lake2018generalization,raunak2019compositionality,li2021compositional} or semantic parsing \cite{finegan2018improving,keysers2019measuring,kim2020cogs,shaw-etal-2021-compositional}.&#10;Most of these studies consider small and highly controlled subsets of natural language.&#10;&#10;Instead, we focus on models trained on fully natural MT datasets, which we believe to be the setup for compositionality evaluation that does most justice to the complexity of natural language: contrary to semantic parsing, where the outputs are structures created by expert annotators, in translation both inputs and outputs are fully-fledged natural language sentences.&#10;To the best of our knowledge, the only attempt to explicitly measure compositional generalisation of NMT models trained on large natural MT corpora is the study presented by \cite{raunak2019compositionality}.&#10;They measure productivity -- generalisation to longer sentence lengths -- of an LSTM-based NMT model trained on a full-size, natural MT dataset.&#10;Other studies using NMT, instead, consider toy datasets generated via templating \cite{lake2018generalization} or focus on short sentences excluding more complex constructions that contribute to the complexity of natural language for compositional generalisation, such as polysemous words or metaphors \cite{li2021compositional}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2011.07127" label="2011.07127">
        <attvalues>
          <attvalue for="0" value="IIRC: A Dataset of Incomplete Information Reading Comprehension&#10;  Questions" />
          <attvalue for="1" value="  Humans often have to read multiple documents to address their information&#10;needs. However, most existing reading comprehension (RC) tasks only focus on&#10;questions for which the contexts provide all the information required to answer&#10;them, thus not evaluating a system's performance at identifying a potential&#10;lack of sufficient information and locating sources for that information. To&#10;fill this gap, we present a dataset, IIRC, with more than 13K questions over&#10;paragraphs from English Wikipedia that provide only partial information to&#10;answer them, with the missing information occurring in one or more linked&#10;documents. The questions were written by crowd workers who did not have access&#10;to any of the linked documents, leading to questions that have little lexical&#10;overlap with the contexts where the answers appear. This process also gave many&#10;questions without answers, and those that require discrete reasoning,&#10;increasing the difficulty of the task. We follow recent modeling work on&#10;various reading comprehension datasets to construct a baseline model for this&#10;dataset, finding that it achieves 31.1% F1 on this task, while estimated human&#10;performance is 88.4%. The dataset, code for the baseline system, and a&#10;leaderboard can be found at https://allennlp.org/iirc.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.03315" label="2102.03315">
        <attvalues>
          <attvalue for="0" value="Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the&#10;  Direct-Answer AI2 Reasoning Challenge" />
          <attvalue for="1" value="  We present the ARC-DA dataset, a direct-answer (&quot;open response&quot;, &quot;freeform&quot;)&#10;version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset. While ARC&#10;has been influential in the community, its multiple-choice format is&#10;unrepresentative of real-world questions, and multiple choice formats can be&#10;particularly susceptible to artifacts. The ARC-DA dataset addresses these&#10;concerns by converting questions to direct-answer format using a combination of&#10;crowdsourcing and expert review. The resulting dataset contains 2985 questions&#10;with a total of 8436 valid answers (questions typically have more than one&#10;valid answer). ARC-DA is one of the first DA datasets of natural questions that&#10;often require reasoning, and where appropriate question decompositions are not&#10;evident from the questions themselves. We describe the conversion approach&#10;taken, appropriate evaluation metrics, and several strong models. Although&#10;high, the best scores (81% GENIE, 61.4% F1, 63.2% ROUGE-L) still leave&#10;considerable room for improvement. In addition, the dataset provides a natural&#10;setting for new research on explanation, as many questions require reasoning to&#10;construct answers. We hope the dataset spurs further advances in complex&#10;question-answering by the community. ARC-DA is available at&#10;https://allenai.org/data/arc-da&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.05457" label="1803.05457">
        <attvalues>
          <attvalue for="0" value="Think you have Solved Question Answering? Try ARC, the AI2 Reasoning&#10;  Challenge" />
          <attvalue for="1" value="  We present a new question set, text corpus, and baselines assembled to&#10;encourage AI research in advanced question answering. Together, these&#10;constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful&#10;knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC&#10;question set is partitioned into a Challenge Set and an Easy Set, where the&#10;Challenge Set contains only questions answered incorrectly by both a&#10;retrieval-based algorithm and a word co-occurence algorithm. The dataset&#10;contains only natural, grade-school science questions (authored for human&#10;tests), and is the largest public-domain set of this kind (7,787 questions). We&#10;test several baselines on the Challenge Set, including leading neural models&#10;from the SQuAD and SNLI tasks, and find that none are able to significantly&#10;outperform a random baseline, reflecting the difficult nature of this task. We&#10;are also releasing the ARC Corpus, a corpus of 14M science sentences relevant&#10;to the task, and implementations of the three neural baseline models tested.&#10;Can your model perform better? We pose ARC as a challenge to the community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.00161" label="1903.00161">
        <attvalues>
          <attvalue for="0" value="DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning&#10;  Over Paragraphs" />
          <attvalue for="1" value="  Reading comprehension has recently seen rapid progress, with systems matching&#10;humans on the most popular datasets for the task. However, a large body of work&#10;has highlighted the brittleness of these systems, showing that there is much&#10;work left to be done. We introduce a new English reading comprehension&#10;benchmark, DROP, which requires Discrete Reasoning Over the content of&#10;Paragraphs. In this crowdsourced, adversarially-created, 96k-question&#10;benchmark, a system must resolve references in a question, perhaps to multiple&#10;input positions, and perform discrete operations over them (such as addition,&#10;counting, or sorting). These operations require a much more comprehensive&#10;understanding of the content of paragraphs than what was necessary for prior&#10;datasets. We apply state-of-the-art methods from both the reading comprehension&#10;and semantic parsing literature on this dataset and show that the best systems&#10;only achieve 32.7% F1 on our generalized accuracy metric, while expert human&#10;performance is 96.0%. We additionally present a new model that combines reading&#10;comprehension methods with simple numerical reasoning to achieve 47.0% F1.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.07898" label="1908.07898">
        <attvalues>
          <attvalue for="0" value="Are We Modeling the Task or the Annotator? An Investigation of Annotator&#10;  Bias in Natural Language Understanding Datasets" />
          <attvalue for="1" value="  Crowdsourcing has been the prevalent paradigm for creating natural language&#10;understanding datasets in recent years. A common crowdsourcing practice is to&#10;recruit a small number of high-quality workers, and have them massively&#10;generate examples. Having only a few workers generate the majority of examples&#10;raises concerns about data diversity, especially when workers freely generate&#10;sentences. In this paper, we perform a series of experiments showing these&#10;concerns are evident in three recent NLP datasets. We show that model&#10;performance improves when training with annotator identifiers as features, and&#10;that models are able to recognize the most productive annotators. Moreover, we&#10;show that often models do not generalize well to examples from annotators that&#10;did not contribute to the training set. Our findings suggest that annotator&#10;bias should be monitored during dataset creation, and that test set annotators&#10;should be disjoint from training set annotators.&#10;" />
          <attvalue for="2" value="&#10;&#10;Generating large datasets has become one of the main drivers of progress in natural language understanding (NLU). The prevalent method for creating new datasets is through crowdsourcing, where examples are generated by workers \cite{zaidan2011crowdsourcing,richardson2013mctest,bowman2015large,rajpurkar2016squad,trischler2017newsqa}.&#10;A common recent practice is to choose a small group of workers who produce high-quality annotations, and massively generate examples using these workers.&#10;&#10;Having only a few workers annotate the majority of dataset examples raises concerns about data diversity and the ability of models to generalize, especially when the crowdsourcing task is to generate free text.&#10;If an annotator consistently uses language patterns that correlate with the labels, a neural model can pick up on those, which can lead to an over-estimation of model performance.&#10;&#10;In this paper, we continue recent efforts to understand biases that are introduced during the process of data creation \cite{levy2015supervised,schwartz2017roc,gururangan2018annotation, glockner2018breaking, poliak2018hypothesis, tsuchiya2018performance,aharoni2018split, paun2018comparing}.&#10;We investigate this form of bias, termed annotator bias, and perform multiple experiments over three recent NLU datasets: MNLI \cite{N18-1101}, OpenBookQA. \cite{mihaylovetal2018}, and CommonsenseQA \cite{talmor2019commonsenseqa}. &#10;&#10;First, we establish that annotator information improves model performance by supplying annotator IDs as part of the input features. Second, we show that models are able to recognize annotators that generated many examples, illustrating that annotator information is captured by the model. Last, we test whether models generalize to annotators that were not seen at training time. We observe that often generalization to new annotators fails, and that augmenting the training set with a small number of examples from these annotators substantially increases performance.&#10;&#10;Taken together, our experiments show that annotator bias exists in current NLU datasets, which can lead to problems in model generalization to new users.&#10;Hence, we propose that annotator bias should be monitored&#10;at data collection time &#10;and to tackle it by having the test set include examples from a disjoint set of annotators.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.14533" label="2402.14533">
        <attvalues>
          <attvalue for="0" value="Whose LLM is it Anyway? Linguistic Comparison and LLM Attribution for&#10;  GPT-3.5, GPT-4 and Bard" />
          <attvalue for="1" value="  Large Language Models (LLMs) are capable of generating text that is similar&#10;to or surpasses human quality. However, it is unclear whether LLMs tend to&#10;exhibit distinctive linguistic styles akin to how human authors do. Through a&#10;comprehensive linguistic analysis, we compare the vocabulary, Part-Of-Speech&#10;(POS) distribution, dependency distribution, and sentiment of texts generated&#10;by three of the most popular LLMS today (GPT-3.5, GPT-4, and Bard) to diverse&#10;inputs. The results point to significant linguistic variations which, in turn,&#10;enable us to attribute a given text to its LLM origin with a favorable 88\%&#10;accuracy using a simple off-the-shelf classification model. Theoretical and&#10;practical implications of this intriguing finding are discussed.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs), such as GPT-3.5 \cite{ouyang2022training}, GPT-4 \cite{achiam2023gpt} and Bard \cite{bard}, have revolutionized and popularized natural language processing and AI, demonstrating human-like and super-human performance in a wide range of text-based tasks \cite{zhao2023survey}. While the layman may find the responses of LLMs hard to distinguish from human-generated ones \cite{layman_llm,new_intro_7}, a plethora of recent literature has shown that it is possible to successfully discern human-generated text from LLM-generated text using various computational techniques \cite{new_intro_llm_1,new_intro_llm_2,new_intro_llm_3}. Among the developed techniques, the linguistic approach, which focuses on the structure, patterns, and nuances inherent in human language, stands out as a promising option that offers both high statistical performance \cite{herbold2023large} as well as theoretically-grounded explanatory power \cite{munoz2023contrasting}, as opposed to alternative \say{black-box} machine-learning techniques \cite{llm_math_1,llm_math_2}. &#10;&#10;Indeed, recent literature has shown that human and LLM-generated texts are, generally speaking, linguistically different across a wide variety of tasks and datasets including news reporting \cite{munoz2023contrasting}, hotel reviewing \cite{giorgi2023slept}, essay writing \cite{herbold2023large} and scientific communication \cite{desaire2023distinguishing} &#10;to name a few. &#10;Common to these and similar studies is the observation that LLM-generated texts tend to be extensive and comprehensive, highly organized, follow a logical structure or formally stated, and present higher objectivity and lower prevalence of bias and harmful content compared to human-generated texts \cite{wu2023survey}. &#10;&#10;Extensive research into human-generated texts has consistently demonstrated the inherent diversity in human writing styles, resulting in distinct linguistic patterns, structures, and nuances \cite{liza,ws_general_3,ws_general_2}. Notably, highly successful techniques for author attribution \cite{ws_author_1,ws_author_2} and author profiling \cite{author_profile_1,author_profile_2} have leveraged linguistic markers to identify and differentiate between authors and their characteristics (sometimes referred to as Stylometrics \cite{stylometric}). These remarkable capabilities underscore both the richness and variability present in human-generated texts as well as the unique linguistic traits presented, rather consistently, by different authors. Unfortunately, to the best of our knowledge, a similar inquiry into LLM-generated texts has yet to take place. That is, it remains unclear whether different LLMs present distinct linguistic styles and, if so, could these linguistic markers be effectively used for LLM attribution (i.e., identifying which LLM has generated a given text).&#10;&#10;In this work, we report on a comprehensive linguistic comparison of LLM-generated texts generated by three of the most popular LLMs today: GPT-3.5, GPT-4, and Bard. Using a wide range of topics and prompts, the results reveal that, indeed, these LLMs are linguistically different, particularly in terms of vocabulary, Part-Of-Speech (POS), and dependencies. In turn, these linguistic markers are shown to bring about a remarkable performance when applied to the LLM attribution task. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Authorship Attribution, Language Model Analysis, Linguistics, Linguistic Style Variation, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2303.11156" label="2303.11156">
        <attvalues>
          <attvalue for="0" value="Can AI-Generated Text be Reliably Detected?" />
          <attvalue for="1" value="  The unregulated use of LLMs can potentially lead to malicious consequences&#10;such as plagiarism, generating fake news, spamming, etc. Therefore, reliable&#10;detection of AI-generated text can be critical to ensure the responsible use of&#10;LLMs. Recent works attempt to tackle this problem either using certain model&#10;signatures present in the generated text outputs or by applying watermarking&#10;techniques that imprint specific patterns onto them. In this paper, we show&#10;that these detectors are not reliable in practical scenarios. In particular, we&#10;develop a recursive paraphrasing attack to apply on AI text, which can break a&#10;whole range of detectors, including the ones using the watermarking schemes as&#10;well as neural network-based detectors, zero-shot classifiers, and&#10;retrieval-based detectors. Our experiments include passages around 300 tokens&#10;in length, showing the sensitivity of the detectors even in the case of&#10;relatively long passages. We also observe that our recursive paraphrasing only&#10;degrades text quality slightly, measured via human studies, and metrics such as&#10;perplexity scores and accuracy on text benchmarks. Additionally, we show that&#10;even LLMs protected by watermarking schemes can be vulnerable against spoofing&#10;attacks aimed to mislead detectors to classify human-written text as&#10;AI-generated, potentially causing reputational damages to the developers. In&#10;particular, we show that an adversary can infer hidden AI text signatures of&#10;the LLM outputs without having white-box access to the detection method.&#10;Finally, we provide a theoretical connection between the AUROC of the best&#10;possible detector and the Total Variation distance between human and AI text&#10;distributions that can be used to study the fundamental hardness of the&#10;reliable detection problem for advanced language models. Our code is publicly&#10;available at https://github.com/vinusankars/Reliability-of-AI-text-detectors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.07205" label="2303.07205">
        <attvalues>
          <attvalue for="0" value="The Science of Detecting LLM-Generated Texts" />
          <attvalue for="1" value="  The emergence of large language models (LLMs) has resulted in the production&#10;of LLM-generated texts that is highly sophisticated and almost&#10;indistinguishable from texts written by humans. However, this has also sparked&#10;concerns about the potential misuse of such texts, such as spreading&#10;misinformation and causing disruptions in the education system. Although many&#10;detection approaches have been proposed, a comprehensive understanding of the&#10;achievements and challenges is still lacking. This survey aims to provide an&#10;overview of existing LLM-generated text detection techniques and enhance the&#10;control and regulation of language generation models. Furthermore, we emphasize&#10;crucial considerations for future research, including the development of&#10;comprehensive evaluation metrics and the threat posed by open-source LLMs, to&#10;drive progress in the area of LLM-generated text detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16617" label="2305.16617">
        <attvalues>
          <attvalue for="0" value="Efficient Detection of LLM-generated Texts with a Bayesian Surrogate&#10;  Model" />
          <attvalue for="1" value="  The detection of machine-generated text, especially from large language&#10;models (LLMs), is crucial in preventing serious social problems resulting from&#10;their misuse. Some methods train dedicated detectors on specific datasets but&#10;fall short in generalizing to unseen test data, while other zero-shot ones&#10;often yield suboptimal performance. Although the recent DetectGPT has shown&#10;promising detection performance, it suffers from significant inefficiency&#10;issues, as detecting a single candidate requires querying the source LLM with&#10;hundreds of its perturbations. This paper aims to bridge this gap. Concretely,&#10;we propose to incorporate a Bayesian surrogate model, which allows us to select&#10;typical samples based on Bayesian uncertainty and interpolate scores from&#10;typical samples to other samples, to improve query efficiency. Empirical&#10;results demonstrate that our method significantly outperforms existing&#10;approaches under a low query budget. Notably, when detecting the text generated&#10;by LLaMA family models, our method with just 2 or 3 queries can outperform&#10;DetectGPT with 200 queries.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.11729" label="2307.11729">
        <attvalues>
          <attvalue for="0" value="OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with&#10;  Adversarially Generated Examples" />
          <attvalue for="1" value="  Large Language Models (LLMs) have achieved human-level fluency in text&#10;generation, making it difficult to distinguish between human-written and&#10;LLM-generated texts. This poses a growing risk of misuse of LLMs and demands&#10;the development of detectors to identify LLM-generated texts. However, existing&#10;detectors lack robustness against attacks: they degrade detection accuracy by&#10;simply paraphrasing LLM-generated texts. Furthermore, a malicious user might&#10;attempt to deliberately evade the detectors based on detection results, but&#10;this has not been assumed in previous studies. In this paper, we propose&#10;OUTFOX, a framework that improves the robustness of LLM-generated-text&#10;detectors by allowing both the detector and the attacker to consider each&#10;other's output. In this framework, the attacker uses the detector's prediction&#10;labels as examples for in-context learning and adversarially generates essays&#10;that are harder to detect, while the detector uses the adversarially generated&#10;essays as examples for in-context learning to learn to detect essays from a&#10;strong attacker. Experiments in the domain of student essays show that the&#10;proposed detector improves the detection performance on the attacker-generated&#10;texts by up to +41.3 points F1-score. Furthermore, the proposed detector shows&#10;a state-of-the-art detection performance: up to 96.9 points F1-score, beating&#10;existing detectors on non-attacked texts. Finally, the proposed attacker&#10;drastically degrades the performance of detectors by up to -57.0 points&#10;F1-score, massively outperforming the baseline paraphrasing method for evading&#10;detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.03992" label="2309.03992">
        <attvalues>
          <attvalue for="0" value="ConDA: Contrastive Domain Adaptation for AI-generated Text Detection" />
          <attvalue for="1" value="  Large language models (LLMs) are increasingly being used for generating text&#10;in a variety of use cases, including journalistic news articles. Given the&#10;potential malicious nature in which these LLMs can be used to generate&#10;disinformation at scale, it is important to build effective detectors for such&#10;AI-generated text. Given the surge in development of new LLMs, acquiring&#10;labeled training data for supervised detectors is a bottleneck. However, there&#10;might be plenty of unlabeled text data available, without information on which&#10;generator it came from. In this work we tackle this data problem, in detecting&#10;AI-generated news text, and frame the problem as an unsupervised domain&#10;adaptation task. Here the domains are the different text generators, i.e. LLMs,&#10;and we assume we have access to only the labeled source data and unlabeled&#10;target data. We develop a Contrastive Domain Adaptation framework, called&#10;ConDA, that blends standard domain adaptation techniques with the&#10;representation power of contrastive learning to learn domain invariant&#10;representations that are effective for the final unsupervised detection task.&#10;Our experiments demonstrate the effectiveness of our framework, resulting in&#10;average performance gains of 31.7% from the best performing baselines, and&#10;within 0.8% margin of a fully supervised detector. All our code and data is&#10;available at https://github.com/AmritaBh/ConDA-gen-text-detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.05540" label="2306.05540">
        <attvalues>
          <attvalue for="0" value="DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of&#10;  Machine-Generated Text" />
          <attvalue for="1" value="  With the rapid progress of large language models (LLMs) and the huge amount&#10;of text they generated, it becomes more and more impractical to manually&#10;distinguish whether a text is machine-generated. Given the growing use of LLMs&#10;in social media and education, it prompts us to develop methods to detect&#10;machine-generated text, preventing malicious usage such as plagiarism,&#10;misinformation, and propaganda. Previous work has studied several zero-shot&#10;methods, which require no training data. These methods achieve good&#10;performance, but there is still a lot of room for improvement. In this paper,&#10;we introduce two novel zero-shot methods for detecting machine-generated text&#10;by leveraging the log rank information. One is called DetectLLM-LRR, which is&#10;fast and efficient, and the other is called DetectLLM-NPR, which is more&#10;accurate, but slower due to the need for perturbations. Our experiments on&#10;three datasets and seven language models show that our proposed methods improve&#10;over the state of the art by 3.9 and 1.75 AUROC points absolute. Moreover,&#10;DetectLLM-NPR needs fewer perturbations than previous work to achieve the same&#10;level of performance, which makes it more practical for real-world use. We also&#10;investigate the efficiency--performance trade-off based on users preference on&#10;these two measures and we provide intuition for using them in practice&#10;effectively. We release the data and the code of both methods in&#10;https://github.com/mbzuai-nlp/DetectLLM&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.10504" label="2010.10504">
        <attvalues>
          <attvalue for="0" value="Pushing the Limits of Semi-Supervised Learning for Automatic Speech&#10;  Recognition" />
          <attvalue for="1" value="  We employ a combination of recent developments in semi-supervised learning&#10;for automatic speech recognition to obtain state-of-the-art results on&#10;LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More&#10;precisely, we carry out noisy student training with SpecAugment using giant&#10;Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we&#10;are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech&#10;test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.14327" label="2005.14327">
        <attvalues>
          <attvalue for="0" value="On the Comparison of Popular End-to-End Models for Large Scale Speech&#10;  Recognition" />
          <attvalue for="1" value="  Recently, there has been a strong push to transition from hybrid models to&#10;end-to-end (E2E) models for automatic speech recognition. Currently, there are&#10;three promising E2E methods: recurrent neural network transducer (RNN-T), RNN&#10;attention-based encoder-decoder (AED), and Transformer-AED. In this study, we&#10;conduct an empirical comparison of RNN-T, RNN-AED, and Transformer-AED models,&#10;in both non-streaming and streaming modes. We use 65 thousand hours of&#10;Microsoft anonymized training data to train these models. As E2E models are&#10;more data hungry, it is better to compare their effectiveness with large amount&#10;of training data. To the best of our knowledge, no such comprehensive study has&#10;been conducted yet. We show that although AED models are stronger than RNN-T in&#10;the non-streaming mode, RNN-T is very competitive in streaming mode if its&#10;encoder can be properly initialized. Among all three E2E models,&#10;transformer-AED achieved the best accuracy in both streaming and non-streaming&#10;mode. We show that both streaming RNN-T and transformer-AED models can obtain&#10;better accuracy than a highly-optimized hybrid model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.14132" label="2302.14132">
        <attvalues>
          <attvalue for="0" value="Structured Pruning of Self-Supervised Pre-trained Models for Speech&#10;  Recognition and Understanding" />
          <attvalue for="1" value="  Self-supervised speech representation learning (SSL) has shown to be&#10;effective in various downstream tasks, but SSL models are usually large and&#10;slow. Model compression techniques such as pruning aim to reduce the model size&#10;and computation without degradation in accuracy. Prior studies focus on the&#10;pruning of Transformers; however, speech models not only utilize a stack of&#10;Transformer blocks, but also combine a frontend network based on multiple&#10;convolutional layers for low-level feature representation learning. This&#10;frontend has a small size but a heavy computational cost. In this work, we&#10;propose three task-specific structured pruning methods to deal with such&#10;heterogeneous networks. Experiments on LibriSpeech and SLURP show that the&#10;proposed method is more accurate than the original wav2vec2-base with 10% to&#10;30% less computation, and is able to reduce the computation by 40% to 50%&#10;without any degradation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Compression Techniques, Speech Representation Learning, Signal Processing, Mathematics, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="2305.12900" label="2305.12900">
        <attvalues>
          <attvalue for="0" value="Evaluating Prompt-based Question Answering for Object Prediction in the&#10;  Open Research Knowledge Graph" />
          <attvalue for="1" value="  There have been many recent investigations into prompt-based training of&#10;transformer language models for new text genres in low-resource settings. The&#10;prompt-based training approach has been found to be effective in generalizing&#10;pre-trained or fine-tuned models for transfer to resource-scarce settings. This&#10;work, for the first time, reports results on adopting prompt-based training of&#10;transformers for \textit{scholarly knowledge graph object prediction}. The work&#10;is unique in the following two main aspects. 1) It deviates from the other&#10;works proposing entity and relation extraction pipelines for predicting objects&#10;of a scholarly knowledge graph. 2) While other works have tested the method on&#10;text genera relatively close to the general knowledge domain, we test the&#10;method for a significantly different domain, i.e. scholarly knowledge, in turn&#10;testing the linguistic, probabilistic, and factual generalizability of these&#10;large-scale transformer models. We find that (i) per expectations, transformer&#10;models when tested out-of-the-box underperform on a new domain of data, (ii)&#10;prompt-based training of the models achieve performance boosts of up to 40\% in&#10;a relaxed evaluation setting, and (iii) testing the models on a starkly&#10;different domain even with a clever training objective in a low resource&#10;setting makes evident the domain knowledge capture gap offering an&#10;empirically-verified incentive for investing more attention and resources to&#10;the scholarly domain in the context of transformer models.&#10;" />
          <attvalue for="2" value="&#10;&#10;A cloze test is a language assessment where certain words (cloze text) are removed from language fragments and the participant must fill them in~\cite{taylor1953cloze}. This type of test helps language learners demonstrate certain necessary skills around language comprehension: syntax, contextual understanding, vocabulary, and factual knowledge. This philosophy of language learning has been applied to transformer language models such as BERT~\cite{kenton2019bert} and RoBERTa~\cite{liu2019roberta}. These models are pre-trained on large-scale text corpora to predict missing words or the next sentence, producing models with language comprehension abilities, around syntax, contextual understanding, vocabulary, and factual knowledge, similar to language learners.&#10;&#10;The seminal work by Petroni et al.~\cite{petroni2019language} opened the avenue for testing these language models for their vasts store of linguistic and factual knowledge with explicit relational cloze objectives for extracting new facts from the language models for knowledge base (KB) population as a downstream task. KBs are effective solutions for accessing gold-standard relational data such as (Michael Jordan, born-in, x). The traditional method for populating such KBs with additional facts would otherwise leverage complex NLP pipelines involving entity extraction, co-reference resolution, entity linking, and relation extraction components~\cite{surdeanu2014overview} that are known to be plagued by the error propagation problem from earlier to later components in the pipeline. Instead, the powerful transformer language models as rich stores of linguistic and factual information having been pre-trained on billion-word corpus from encyclopedic sources were probed for additional facts, showing to outperform the traditional NLP pipeline method for generating relational knowledge to populate KBs as a downstream task~\cite{petroni2019language}.&#10;&#10;While the pre-training objective prepares task-agnostic language models from large-scale corpora which were probed in the work by Petroni et al.~\cite{petroni2019language}, these language models can be further fine-tuned as an additional step with task-specific objectives which stimulates their task-specific suitability for a downstream task. To probe the transformer language models for relational facts as discussed in the earlier paragraph, the knowledge of the original models was accessed by conditioning on their latent context representations. On the other hand, to obtain a model specifically for the Question Answering (QA) downstream task, as a concrete example and the focus of this work, better versions of the models are obtained when the original models' weights are used to first initialize a task-agnostic model which is then further fine-tuned to obtain a QA task-specific model given instances of the downstream task, e.g. as defined in the Stanford Question Answering Dataset (SQuAD)~\cite{rajpurkar2016squad,rajpurkar2018know}. This work focuses on fine-tuning language models for the QA task inspired after the SQuAD dataset. The goal is to obtain a model that can extract answers from context paragraphs based on questions, and to optimize for scholarly knowledge rather than encyclopedic knowledge by tapping into the pre-trained model’s cloze task ability and the SQuAD QA fine-tuned model’s structural representation. The next two paragraphs introduce the why and how of our work.&#10;&#10;Why focus on the scholarly domain? In the face of rapid publication rates at an alarming rate of millions of articles per year~\cite{johnson2018stm}, researchers are immensely challenged in keeping up with the latest findings in scholarly publications. To address this problem, the Open Research Knowledge Graph (\href{ leveraging next-generation semantic scholarly knowledge publishing tools~\cite{shotton2009semantic}, was created to make scholarly contributions more accessible and machine-actionable, enabling smart information access methods via \href{ \href{ and \href{ thereby alleviating the researchers' knowledge comprehension problem from months or days to a matter of minutes. This growing KB of structured science-wide scholarly contributions is an unexplored resource that could be used as a testbed for discovering new facts with the help of powerful transformer language models. This could be realized in the long-run as a potential NLP service that assists ORKG users in scholarly knowledge curation and completion. Specifically, the service, directly inspired by Petroni et al's fact probing method~\cite{petroni2019language}, could be based on optimal, fine-tuned versions of the language models to discover additional objects for new incoming relations. \ref{table-examples} shows some example instances of the proposed task. As evident from these examples, the task aims to extract objects as answers from scientific paper abstracts given a relation/predicate as a question. The task is modeled similarly to the SQuAD QA task, which extracts facts from unstructured encyclopedic knowledge using a question and answer format. We have chosen the SQuAD-based task formulation for three mains reasons: 1) the SQuAD QA task formulation intuitively transfers to the scholarly domain; 2) to exploit the vast collection of options to choose from of state-of-the-art language models fine-tuned on the QA task; and 3) to test whether these existing models' encoded statistical regularities to extract answers from a given encyclopedic knowledge context can transfer easily to the scholarly domain. We present a detailed empirical investigation for the first time of the linguistically-rich fine-tuned SQuAD QA models' capacity to transfer to a new domain, i.e. the scholarly domain, which has thus far remained unexplored.&#10;&#10;How to obtain an optimal model for a new domain? To pre-train or fine-tune a language model for QA on a new domain, the traditional method is to use expensive human-labeled data. Instead, inspired from prior work~\cite{fabbri2020template,schick2021exploiting,zhong2022proqa}, this study uses two strategies: 1) template-based unsupervised generation of structured data similar to SQuAD QA data from the ORKG KB, and 2) structural prompt-based learning over state-of-the-art SQuAD-specific fine-tuned transformer models for the scholarly domain.&#10;How to obtain an optimal model on a new domain? The similar structured QA data can serve as prompts to stimulate the knowledge acquired in the fine-tuned language models to model knowledge generalizations over the new domain based on the unified input schema in turn resulting in an optimal trained model in a resource-scarce setting. The idea is to apply a ``Wh'' question template on an ORKG structured contribution predicate, and then generate a question based on that predicate whose answer is the ORKG object that can be found as a contiguous span in the paper's abstract. Having created a question for all (context, answer) pairs, we then reserve a portion of the data for prompt-based fine-tuning of variants of state-of-the-art SQuAD transformer models on this data and evaluate on the remaining held-out version of the synthetically generated ORKG QA data.&#10;&#10;Summarily, our contributions are: 1) we introduce a template-based unsupervised question generation framework for the scholarly domain, similar in format to the SQuAD dataset; 2) we report, for the first time, a detailed empirical analysis of the scholarly domain object prediction task using prompt-based QA task and state-of-the-art transformer models as rich stores of linguistic, probabilistic, and factual parameters, thereby testing the transferability of these pre-trained models on a novel domain. Our \href{ \href{ and \href{ are publicly released.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Knowledge Graph Prediction, Linguistics, Language Model Training, Domain Adaptation Techniques, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2205.04040" label="2205.04040">
        <attvalues>
          <attvalue for="0" value="ProQA: Structural Prompt-based Pre-training for Unified Question&#10;  Answering" />
          <attvalue for="1" value="  Question Answering (QA) is a longstanding challenge in natural language&#10;processing. Existing QA works mostly focus on specific question types,&#10;knowledge domains, or reasoning skills. The specialty in QA research hinders&#10;systems from modeling commonalities between tasks and generalization for wider&#10;applications. To address this issue, we present ProQA, a unified QA paradigm&#10;that solves various tasks through a single model. ProQA takes a unified&#10;structural prompt as the bridge and improves the QA-centric ability by&#10;structural prompt-based pre-training. Through a structurally designed&#10;prompt-based input schema, ProQA concurrently models the knowledge&#10;generalization for all QA tasks while keeping the knowledge customization for&#10;every specific QA task. Furthermore, ProQA is pre-trained with structural&#10;prompt-formatted large-scale synthesized corpus, which empowers the model with&#10;the commonly-required QA ability. Experimental results on 11 QA benchmarks&#10;demonstrate that ProQA consistently boosts performance on both full data&#10;fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore,&#10;ProQA exhibits strong ability in both continual learning and transfer learning&#10;by taking the advantages of the structural prompt.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.12994" label="2407.12994">
        <attvalues>
          <attvalue for="0" value="A Survey of Prompt Engineering Methods in Large Language Models for&#10;  Different NLP Tasks" />
          <attvalue for="1" value="  Large language models (LLMs) have shown remarkable performance on many&#10;different Natural Language Processing (NLP) tasks. Prompt engineering plays a&#10;key role in adding more to the already existing abilities of LLMs to achieve&#10;significant performance gains on various NLP tasks. Prompt engineering requires&#10;composing natural language instructions called prompts to elicit knowledge from&#10;LLMs in a structured way. Unlike previous state-of-the-art (SoTA) models,&#10;prompt engineering does not require extensive parameter re-training or&#10;fine-tuning based on the given NLP task and thus solely operates on the&#10;embedded knowledge of LLMs. Additionally, LLM enthusiasts can intelligently&#10;extract LLMs' knowledge through a basic natural language conversational&#10;exchange or prompt engineering, allowing more and more people even without deep&#10;mathematical machine learning background to experiment with LLMs. With prompt&#10;engineering gaining popularity in the last two years, researchers have come up&#10;with numerous engineering techniques around designing prompts to improve&#10;accuracy of information extraction from the LLMs. In this paper, we summarize&#10;different prompting techniques and club them together based on different NLP&#10;tasks that they have been used for. We further granularly highlight the&#10;performance of these prompting strategies on various datasets belonging to that&#10;NLP task, talk about the corresponding LLMs used, present a taxonomy diagram&#10;and discuss the possible SoTA for specific datasets. In total, we read and&#10;present a survey of 44 research papers which talk about 39 different prompting&#10;methods on 29 different NLP tasks of which most of them have been published in&#10;the last two years.&#10;" />
          <attvalue for="2" value="&#10;&#10;Artificial Intelligence has advanced significantly with the introduction of LLMs. LLMs are trained on huge corpora of text documents with millions and billions of tokens. It has been shown that as the number of model parameters increase, the performance of machine learning models improve and such has been the case with these LLMs. They have attained unprecedented performance on a wide array of NLP tasks \cite{chang2023survey} because of which they have attracted a lot of interest from academia and different industries including medicine, law, finance and more. The present phase of research on LLMs focuses on their reasoning capacity via prompts rather than just next token prediction which has opened a new field of research around prompt engineering.&#10;&#10;Prompt engineering is the process of creating natural language instructions, or prompts, to extract knowledge from LLMs in an organized manner. Prompt engineering, in contrast to earlier conventional models, relies only on the embedded knowledge of LLMs and does not require extensive parameter re-training or fine-tuning based on the underlying NLP task. Understanding model parameters in terms of real world knowledge embedded in them is beyond human capabilities and hence this new field of prompt engineering has caught everyone's attention as it allows natural language exchange between researchers and LLMs to achieve the goals of the underlying NLP task.&#10;&#10;In this work, we enumerate several prompting strategies and group them according to different NLP tasks that they have been used for. We provide a taxonomy diagram, tabulate the prompting techniques tried on various datasets for different NLP tasks, discuss the LLMs employed, and list potential SoTA methods for each dataset. As a part of this survey, we have reviewed and analyzed 44 research papers in total, the majority of which have been published in the previous two years and cover 39 prompting techniques applied on 29 different NLP tasks. There have not been a lot of prior systematic surveys on prompt engineering. \cite{sahoo2024systematic} surveys 29 prompting technique papers based on their applications. This is a very broad categorization as a single application can encapsulate numerous NLP tasks. For example, one of the applications which they discuss is reasoning and logic which can have plethora of NLP tasks like commonsense reasoning, mathemathical problem solving, multi-hop reasoning etc. This is different from our approach as we take a more granular categorization of prompting strategies based on the NLP tasks. \cite{edemacu2024privacy} provides an overview of privacy protection prompting methods and thus focuses on a comparatively small sub-field of prompt engineering. \cite{chen2023unleashing} limits the discussion of prompting strategies to some 9-10 methodologies and also does not incorporate categorizing them based on the NLP tasks.&#10;&#10;The rest of the paper is organized in the following way. Section 2 talks about various prompt engineering techniques and section3 highlights different NLP tasks. The sub-sections of section3 discuss different prompting strategies that have been applied on a given NLP task and their corresponding results. Section 4 concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Prompt Engineering" />
        </attvalues>
      </node>
      <node id="2404.06001" label="2404.06001">
        <attvalues>
          <attvalue for="0" value="Privacy Preserving Prompt Engineering: A Survey" />
          <attvalue for="1" value="  Pre-trained language models (PLMs) have demonstrated significant proficiency&#10;in solving a wide range of general natural language processing (NLP) tasks.&#10;Researchers have observed a direct correlation between the performance of these&#10;models and their sizes. As a result, the sizes of these models have notably&#10;expanded in recent years, persuading researchers to adopt the term large&#10;language models (LLMs) to characterize the larger-sized PLMs. The size&#10;expansion comes with a distinct capability called in-context learning (ICL),&#10;which represents a special form of prompting and allows the models to be&#10;utilized through the presentation of demonstration examples without&#10;modifications to the model parameters. Although interesting, privacy concerns&#10;have become a major obstacle in its widespread usage. Multiple studies have&#10;examined the privacy risks linked to ICL and prompting in general, and have&#10;devised techniques to alleviate these risks. Thus, there is a necessity to&#10;organize these mitigation techniques for the benefit of the community. This&#10;survey provides a systematic overview of the privacy protection methods&#10;employed during ICL and prompting in general. We review, analyze, and compare&#10;different methods under this paradigm. Furthermore, we provide a summary of the&#10;resources accessible for the development of these frameworks. Finally, we&#10;discuss the limitations of these frameworks and offer a detailed examination of&#10;the promising areas that necessitate further exploration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.14735" label="2310.14735">
        <attvalues>
          <attvalue for="0" value="Unleashing the potential of prompt engineering in Large Language Models:&#10;  a comprehensive review" />
          <attvalue for="1" value="  This comprehensive review delves into the pivotal role of prompt engineering&#10;in unleashing the capabilities of Large Language Models (LLMs). The development&#10;of Artificial Intelligence (AI), from its inception in the 1950s to the&#10;emergence of advanced neural networks and deep learning architectures, has made&#10;a breakthrough in LLMs, with models such as GPT-4o and Claude-3, and in&#10;Vision-Language Models (VLMs), with models such as CLIP and ALIGN. Prompt&#10;engineering is the process of structuring inputs, which has emerged as a&#10;crucial technique to maximize the utility and accuracy of these models. This&#10;paper explores both foundational and advanced methodologies of prompt&#10;engineering, including techniques such as self-consistency, chain-of-thought,&#10;and generated knowledge, which significantly enhance model performance.&#10;Additionally, it examines the prompt method of VLMs through innovative&#10;approaches such as Context Optimization (CoOp), Conditional Context&#10;Optimization (CoCoOp), and Multimodal Prompt Learning (MaPLe). Critical to this&#10;discussion is the aspect of AI security, particularly adversarial attacks that&#10;exploit vulnerabilities in prompt engineering. Strategies to mitigate these&#10;risks and enhance model robustness are thoroughly reviewed. The evaluation of&#10;prompt methods is also addressed, through both subjective and objective&#10;metrics, ensuring a robust analysis of their efficacy. This review also&#10;reflects the essential role of prompt engineering in advancing AI capabilities,&#10;providing a structured framework for future research and application.&#10;" />
          <attvalue for="2" value="&#10;In recent years, a significant milestone in artificial intelligence research has been the progression of natural language processing capabilities, primarily attributed to Large Language Models (LLMs). Many popular models, rooted in the transformer architecture \cite{10.5555/3295222.3295349}, undergo training on extensive datasets derived from web-based text. Central to their design is a self-supervised learning objective, which focuses on predicting subsequent words in incomplete sentences. Those models are called Artificial Intelligence-Generated Content (AIGC), and their ability to generate coherent and contextually relevant responses is a result of this training process, where they learn to associate words and phrases with their typical contexts.&#10;&#10;LLMs operate by encoding the input text into a high-dimensional vector space, where semantic relationships between words and phrases are preserved. The model then decodes this representation to generate a response, guided by the learned statistical patterns \cite{bender2021dangers}. The quality of the response can be influenced by various factors, including the prompt provided to the model, the model's hyperparameters, and the diversity of the training data.&#10;&#10;These models, including LLMs such as the GPT series &#10;\cite{brownGPT3, openai2024gpt4technicalreport} by OpenAI, along with many others (e.g. Gemini \cite{geminiteam2024geminifamilyhighlycapable, google2024gemini} and Gemini (BARD) \cite{hulbert2023bard} by Google, Claude series by Anthropic \cite{anthropic2024claude3, claude3}, and Llama series open-source model from Meta \cite{2023llama2, dubey2024llama3herdmodels}), have revolutionized tasks ranging from information extraction to the creation of engaging content \cite{Sarkhel2023}. In parallel, the development of multimodal large models (MMLMs) has introduced the ability to process and generate not just text, but also images, audio, and other forms of data, showcasing their flexibility and effectiveness. These models integrate multiple data modalities into a single framework, demonstrating strong capabilities in tasks such as image description and visual question answering (VQA). Early MMLMs include the DALL-E series \cite{ramesh2021zeroshot, marcus2022dalle2, openai2021dalle}, which can generate images from textual descriptions, and CLIP, which can understand and relate text and image data in a unified manner \cite{Radford2021LearningTV, li2022supervision}. More powerful models such as GPT-4o by OpenAI \cite{openai_gpt4o} and Claude 3.5 Sonnet by Anthropic \cite{anthropic2024claude3, claude3} excel in multimodal tasks involving text generation and understanding, integrating natural language processing with various forms of data to perform diverse and complex tasks. While numerous advanced models are currently capable of processing audio, the majority of accessible Application Programming Interfaces (APIs) remain focused on text and vision modalities. With the gradual introduction of audio APIs, a broad expansion of research in this modality can be expected \cite{Moore2024}. The evolution of LLMs reflects significant strides in AI research, characterized by increasing model complexity, enhanced training methodologies, and broader application potentials. These advancements underline the critical role of prompt engineering in maximizing the utility and accuracy of these models, ensuring that they can effectively cater to diverse and dynamic user needs. While this survey is mainly focused on prompt engineering for LLMs, the inclusion of vision-language models (VLMs) offers a broader perspective, revealing the potential and challenges of prompt engineering in handling multimodal data. By integrating research from both types of models, we can gain a deeper understanding of the applications of prompt engineering and provide valuable insights for future research and practice.&#10;&#10;In real applications, the prompt is the input of the model, and prompt engineering can result in significant output differences \cite{kaddour2023challenges}. Modifying both the structure (e.g., altering length, arrangement of instances) and the content (e.g., phrasing, choice of illustrations, directives) of the prompt can exert a notable influence on the model's behavior \cite{lu2022sensitivity, webson2022prompt}.&#10;&#10;Prompt engineering refers to the systematic design and optimization of input prompts to guide the responses of LLMs, ensuring accuracy, relevance, and coherence in the generated output. This process is crucial in harnessing the full potential of these models, making them more accessible and applicable across diverse domains. Over time, prompt engineering has evolved from an empirical practice into a well-structured research domain. As illustrated in Figure \ref{History_of_the_development_in_prompt_engineering}, the historical progression of prompt engineering showcases significant milestones from the early days of structured inputs in the 1950s to advanced methodologies such as chain-of-thought prompting \cite{wei2022chain} and self-consistency prompting \cite{wang2023selfconsistency} developed in recent years. This review will primarily focus on techniques emerging from the period of rapid development after 2017.&#10;&#10;Contemporary prompt engineering encompasses a spectrum of techniques, ranging from foundational approaches such as role-prompting \cite{shanahan2023roleplay} to more sophisticated methods such as chain-of-thought prompting \cite{wei2022chain}. The domain remains dynamic, with emergent research continually unveiling novel techniques and applications in prompt engineering. The importance of prompt engineering is accentuated by its ability to guide model responses, thereby amplifying the versatility and relevance of LLMs in various sectors. Importantly, a well-constructed prompt can counteract challenges such as machine hallucinations \cite{maynez2020faithfulness, bubeck2023sparks}. The influence of prompt engineering extends to numerous disciplines. For instance, it has facilitated the creation of robust feature extractors using LLMs, thereby improving their efficacy in tasks such as defect detection and classification \cite{yong2022prompt}.&#10;&#10;This paper aims to provide a comprehensive review of the prompt engineering techniques proposed so far within the realm of LLMs. The structure of the paper is organized as follows: Section \ref{Sec2} explores the foundational methods of prompt engineering, emphasizing the importance of clear and precise instructions, role-prompting, and iterative attempts to optimize outputs. In Section \ref{sec:advanced_methodologies}, advanced methodologies such as chain-of-thought, self-consistency, and generated knowledge are introduced to guide models in generating high-quality content. Section \ref{Sec4} discusses methodologies specific to VLMs, including Context Optimization (CoOp), Conditional Context Optimization (CoCoOp), and Multimodal Prompt Learning (MaPLe), which enhance the performance of VLMs \cite{WANG2023100047}. Section \ref{Sec5} assesses the efficacy of various prompt methods through both subjective and objective evaluations, ensuring a robust analysis of their effectiveness. Section \ref{Sec6} briefly explores the applications of prompt engineering across diverse fields such as education, content creation, computer programming, and reasoning tasks, highlighting its broad impact. Section \ref{Sec7} addresses the security implications of prompt engineering, identifying common vulnerabilities in LLMs and reviewing strategies to enhance security such as adversarial training. Finally, Section \ref{Sec8} explores prospective methodologies, emphasizing the importance of understanding AI model structures and the potential of AI agents in advancing AI-generated content tools. This structured framework provides an entire overview of the pivotal role of prompt engineering in advancing AI capabilities and guiding future research and applications.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.13205" label="2309.13205">
        <attvalues>
          <attvalue for="0" value="A Practical Survey on Zero-shot Prompt Design for In-context Learning" />
          <attvalue for="1" value="  The remarkable advancements in large language models (LLMs) have brought&#10;about significant improvements in Natural Language Processing(NLP) tasks. This&#10;paper presents a comprehensive review of in-context learning techniques,&#10;focusing on different types of prompts, including discrete, continuous,&#10;few-shot, and zero-shot, and their impact on LLM performance. We explore&#10;various approaches to prompt design, such as manual design, optimization&#10;algorithms, and evaluation methods, to optimize LLM performance across diverse&#10;tasks. Our review covers key research studies in prompt engineering, discussing&#10;their methodologies and contributions to the field. We also delve into the&#10;challenges faced in evaluating prompt performance, given the absence of a&#10;single &quot;best&quot; prompt and the importance of considering multiple metrics. In&#10;conclusion, the paper highlights the critical role of prompt design in&#10;harnessing the full potential of LLMs and provides insights into the&#10;combination of manual design, optimization techniques, and rigorous evaluation&#10;for more effective and efficient use of LLMs in various NLP tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, transformer-based language models (such as \cite{t5}, \cite{BART}, \cite{NEURIPS2020_1457c0d6}, \cite{BERT}) have emerged as a transformative force in the field of artificial intelligence, revolutionizing Natural Language Understanding(NLU) and Generation(NLG). As model size and training data have evolved, the GPT series has exhibited extraordinary capabilities in a wide range of natural language tasks by relying on a paradigm known as in-context learning. According to \cite{NEURIPS2020_1457c0d6}, in-context learning harnesses the context provided by input data to generate appropriate responses or predictions, contrasting with traditional methods that necessitate explicit task-specific training and fine-tuning on labeled datasets. In-context learning enables large language models to capitalize on vast amounts of data and adapt to various tasks in a flexible and dynamic manner. There are several categories of in-context learning, including zero-shot, one-shot, and few-shot learning. In all types of in-context learning, the key to success lies in effective prompt design, which is occasionally referred to as an &quot;art.&quot; This survey paper aims to categorize each type of in-context learning, discuss the core principles, examine state-of-the-art design techniques, and explore recent advancements in in-context learning, with a particular focus on zero-shot discrete in-context learning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Prompt Engineering" />
        </attvalues>
      </node>
      <node id="1911.03437" label="1911.03437">
        <attvalues>
          <attvalue for="0" value="SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language&#10;  Models through Principled Regularized Optimization" />
          <attvalue for="1" value="  Transfer learning has fundamentally changed the landscape of natural language&#10;processing (NLP) research. Many existing state-of-the-art models are first&#10;pre-trained on a large text corpus and then fine-tuned on downstream tasks.&#10;However, due to limited data resources from downstream tasks and the extremely&#10;large capacity of pre-trained models, aggressive fine-tuning often causes the&#10;adapted model to overfit the data of downstream tasks and forget the knowledge&#10;of the pre-trained model. To address the above issue in a more principled&#10;manner, we propose a new computational framework for robust and efficient&#10;fine-tuning for pre-trained language models. Specifically, our proposed&#10;framework contains two important ingredients: 1. Smoothness-inducing&#10;regularization, which effectively manages the capacity of the model; 2. Bregman&#10;proximal point optimization, which is a class of trust-region methods and can&#10;prevent knowledge forgetting. Our experiments demonstrate that our proposed&#10;method achieves the state-of-the-art performance on multiple NLP benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.05975" label="2105.05975">
        <attvalues>
          <attvalue for="0" value="Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer" />
          <attvalue for="1" value="  There is an increasing amount of evidence that in cases with little or no&#10;data in a target language, training on a different language can yield&#10;surprisingly good results. However, currently there are no established&#10;guidelines for choosing the training (source) language. In attempt to solve&#10;this issue we thoroughly analyze a state-of-the-art multilingual model and try&#10;to determine what impacts good transfer between languages. As opposed to the&#10;majority of multilingual NLP literature, we don't only train on English, but on&#10;a group of almost 30 languages. We show that looking at particular syntactic&#10;features is 2-4 times more helpful in predicting the performance than an&#10;aggregated syntactic similarity. We find out that the importance of syntactic&#10;features strongly differs depending on the downstream task - no single feature&#10;is a good performance predictor for all NLP tasks. As a result, one should not&#10;expect that for a target language $L_1$ there is a single language $L_2$ that&#10;is the best choice for any NLP task (for instance, for Bulgarian, the best&#10;source language is French on POS tagging, Russian on NER and Thai on NLI). We&#10;discuss the most important linguistic features affecting the transfer quality&#10;using statistical and machine learning methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.12688" label="1905.12688">
        <attvalues>
          <attvalue for="0" value="Choosing Transfer Languages for Cross-Lingual Learning" />
          <attvalue for="1" value="  Cross-lingual transfer, where a high-resource transfer language is used to&#10;improve the accuracy of a low-resource task language, is now an invaluable tool&#10;for improving performance of natural language processing (NLP) on low-resource&#10;languages. However, given a particular task language, it is not clear which&#10;language to transfer from, and the standard strategy is to select languages&#10;based on ad hoc criteria, usually the intuition of the experimenter. Since a&#10;large number of features contribute to the success of cross-lingual transfer&#10;(including phylogenetic similarity, typological properties, lexical overlap, or&#10;size of available data), even the most enlightened experimenter rarely&#10;considers all these factors for the particular task at hand. In this paper, we&#10;consider this task of automatically selecting optimal transfer languages as a&#10;ranking problem, and build models that consider the aforementioned features to&#10;perform this prediction. In experiments on representative NLP tasks, we&#10;demonstrate that our model predicts good transfer languages much better than ad&#10;hoc baselines considering single features in isolation, and glean insights on&#10;what features are most informative for each different NLP tasks, which may&#10;inform future ad hoc selection even without use of our method. Code, data, and&#10;pre-trained models are available at https://github.com/neulab/langrank&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.01496" label="2302.01496">
        <attvalues>
          <attvalue for="0" value="Efficient Domain Adaptation for Speech Foundation Models" />
          <attvalue for="1" value="  Foundation models (FMs), that are trained on broad data at scale and are&#10;adaptable to a wide range of downstream tasks, have brought large interest in&#10;the research community. Benefiting from the diverse data sources such as&#10;different modalities, languages and application domains, foundation models have&#10;demonstrated strong generalization and knowledge transfer capabilities. In this&#10;paper, we present a pioneering study towards building an efficient solution for&#10;FM-based speech recognition systems. We adopt the recently developed&#10;self-supervised BEST-RQ for pretraining, and propose the joint finetuning with&#10;both source and unsupervised target domain data using JUST Hydra. The FM&#10;encoder adapter and decoder are then finetuned to the target domain with a&#10;small amount of supervised in-domain data. On a large-scale YouTube and Voice&#10;Search task, our method is shown to be both data and model parameter efficient.&#10;It achieves the same quality with only 21.6M supervised in-domain data and&#10;130.8M finetuned parameters, compared to the 731.1M model trained from scratch&#10;on additional 300M supervised in-domain data.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large models trained on broad data at scale are often desired since they can be adaptable to a wide range of downstream tasks. One prevailing work is the Foundation Model (FM) \cite{bommasani2021opportunities}, which has brought tremendous interest in the research community. FMs have demonstrated strong generalization and knowledge transfer capabilities \cite{brown2020language,adiwardana2020towards}, as a result of learning from diverse data sources such as different modalities, multiple languages and various application domains. In the speech community, there have been numerous research studies showing promising results and demonstrating the potential advantages of such models \cite{NarayananMisraSimPundakEtAl18,chan2021speechstew,chen2021gigaspeech,zhang2022bigssl,radford2022robust,hwang2022pseudo,gandhi2022esb}. Depending on whether the supervised training data is used, we can group the foundation models into two categories, self-supervised pretrained models \cite{devlin2018bert,baevski2020wav2vec} and supervised multitask-trained ones \cite{raffel2020exploring}. With self-supervised training, the models are first trained on audio-only data using contrastive loss \cite{oord2018representation} or reconstruction loss \cite{chorowski2019unsupervised}, to learn good representations of the speech signals. These models are then directly used as feature extractors for downstream tasks \cite{huang2022s3prl,lin2022analyzing}. As no label information is needed, this approach can easily scale up for more diverse speech data without human transcription effort involved. With supervised multitask learning similar to \cite{raffel2020exploring}, different tasks are unified into a heterogeneous discriminative task and the model is trained jointly on these tasks, such as multi-domain tasks \cite{narayanan2018toward,chan2021speechstew} or multilingual tasks \cite{li2021scaling, li2022massively}. A prerequisite of this approach is to have some labeled data for tasks that the FMs are trained on. Recent works also found that self-supervised pretraining could improve the label data efficiency for the supervised multitask models \cite{chan2021speechstew,zhang2022bigssl}. In this paper, we mainly focus on FMs trained under this procedure for better recognition quality. &#10;&#10;Existing work has mainly focused on using supervised in-domain data to jointly train or finetune FMs for target tasks \cite{chan2021speechstew,zhang2022bigssl}. The use of supervised in-domain data requires FM retraining when new domain presents. To address this, we propose to build FMs from the public domain YouTube data. YouTube has a diverse source of speech from more than 100 different countries around the world, across 80 languages and covers a large variety of domains \cite{liao2013large}. The amount of data is also tremendous \cite{youtubestats}. These make it a great source for building speech FMs, which can also be shared between industry and academia to foster collaborations. For better quality, FMs tend to have large sizes. Fine tuning such models are resource inefficient and time consuming. Techniques that can efficiently adapt FMs to a target task are crucial. Existing work such as residual adapters \cite{houlsby2019parameter,hwang2022large,biadsy2022scalable}, prompting \cite{he2021towards} and neural reprogramming \cite{yang2021voice2series} have demonstrated such potentials. &#10;&#10;Our work contributes to FM learning in several aspects. First, conventional pretrain+finetune 2-stage scheme like wav2vec 2.0 \cite{baevski2020wav2vec} and wav2vec-BERT \cite{chung2021w2v} updates the pretrained encoder during finetuning, which, however, is often costly for large FMs. Therefore, we propose to freeze the pretrained FMs during finetuning and only update a cheap added adapter for the target domain. We show such method works for CTC, LAS, and RNN-T decoders.&#10;Second, most FMs finetune the model with labeled data from the target domain. We demonstrate proper data selection from the source domain (YouTube) can also mitigate the gap between the source and target domain, and further help the performance in the target domain (Voice Search).&#10;Third, even without audio-text paired data from the target domain, we show either audio-only or text-only data can facilitate the FM finetuning.&#10;We present a full recipe of building a FM that can be efficiently adapted to a target domain. Despite some techniques in this paper have been studied previously, this paper innovatively introduces a unified framework for building high quality speech solutions for downstream tasks using FMs. In this framework, we explore data efficiency and model efficiency techniques to build FMs that have a coarse connection to the target domain but can be fast adapted with limited amount of supervised in-domain data. This opens new research directions to explore techniques to push the limit of FMs for speech tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Foundation Models, Linguistics, Efficient Learning, Artificial Intelligence, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1808.05312" label="1808.05312">
        <attvalues>
          <attvalue for="0" value="Toward domain-invariant speech recognition via large scale training" />
          <attvalue for="1" value="  Current state-of-the-art automatic speech recognition systems are trained to&#10;work in specific `domains', defined based on factors like application, sampling&#10;rate and codec. When such recognizers are used in conditions that do not match&#10;the training domain, performance significantly drops. This work explores the&#10;idea of building a single domain-invariant model for varied use-cases by&#10;combining large scale training data from multiple application domains. Our&#10;final system is trained using 162,000 hours of speech. Additionally, each&#10;utterance is artificially distorted during training to simulate effects like&#10;background noise, codec distortion, and sampling rates. Our results show that,&#10;even at such a scale, a model thus trained works almost as well as those&#10;fine-tuned to specific subsets: A single model can be robust to multiple&#10;application domains, and variations like codecs and noise. More importantly,&#10;such models generalize better to unseen conditions and allow for rapid&#10;adaptation -- we show that by using as little as 10 hours of data from a new&#10;domain, an adapted domain-invariant model can match performance of a&#10;domain-specific model trained from scratch using 70 times as much data. We also&#10;highlight some of the limitations of such models and areas that need addressing&#10;in future work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06909" label="2106.06909">
        <attvalues>
          <attvalue for="0" value="GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of&#10;  Transcribed Audio" />
          <attvalue for="1" value="  This paper introduces GigaSpeech, an evolving, multi-domain English speech&#10;recognition corpus with 10,000 hours of high quality labeled audio suitable for&#10;supervised training, and 40,000 hours of total audio suitable for&#10;semi-supervised and unsupervised training. Around 40,000 hours of transcribed&#10;audio is first collected from audiobooks, podcasts and YouTube, covering both&#10;read and spontaneous speaking styles, and a variety of topics, such as arts,&#10;science, sports, etc. A new forced alignment and segmentation pipeline is&#10;proposed to create sentence segments suitable for speech recognition training,&#10;and to filter out segments with low-quality transcription. For system training,&#10;GigaSpeech provides five subsets of different sizes, 10h, 250h, 1000h, 2500h,&#10;and 10000h. For our 10,000-hour XL training subset, we cap the word error rate&#10;at 4% during the filtering/validation stage, and for all our other smaller&#10;training subsets, we cap it at 0%. The DEV and TEST evaluation sets, on the&#10;other hand, are re-processed by professional human transcribers to ensure high&#10;transcription quality. Baseline systems are provided for popular speech&#10;recognition toolkits, namely Athena, ESPnet, Kaldi and Pika.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.13226" label="2109.13226">
        <attvalues>
          <attvalue for="0" value="BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning&#10;  for Automatic Speech Recognition" />
          <attvalue for="1" value="  We summarize the results of a host of efforts using giant automatic speech&#10;recognition (ASR) models pre-trained using large, diverse unlabeled datasets&#10;containing approximately a million hours of audio. We find that the combination&#10;of pre-training, self-training and scaling up model size greatly increases data&#10;efficiency, even for extremely large tasks with tens of thousands of hours of&#10;labeled data. In particular, on an ASR task with 34k hours of labeled data, by&#10;fine-tuning an 8 billion parameter pre-trained Conformer model we can match&#10;state-of-the-art (SoTA) performance with only 3% of the training data and&#10;significantly improve SoTA with the full training set. We also report on the&#10;universal benefits gained from using big pre-trained and self-trained models&#10;for a large set of downstream tasks that cover a wide range of speech domains&#10;and span multiple orders of magnitudes of dataset sizes, including obtaining&#10;SoTA performance on many public benchmarks. In addition, we utilize the learned&#10;representation of pre-trained networks to achieve SoTA results on non-ASR&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.08810" label="1901.08810">
        <attvalues>
          <attvalue for="0" value="Unsupervised speech representation learning using WaveNet autoencoders" />
          <attvalue for="1" value="  We consider the task of unsupervised extraction of meaningful latent&#10;representations of speech by applying autoencoding neural networks to speech&#10;waveforms. The goal is to learn a representation able to capture high level&#10;semantic content from the signal, e.g.\ phoneme identities, while being&#10;invariant to confounding low level details in the signal such as the underlying&#10;pitch contour or background noise. Since the learned representation is tuned to&#10;contain only phonetic content, we resort to using a high capacity WaveNet&#10;decoder to infer information discarded by the encoder from previous samples.&#10;Moreover, the behavior of autoencoder models depends on the kind of constraint&#10;that is applied to the latent representation. We compare three variants: a&#10;simple dimensionality reduction bottleneck, a Gaussian Variational Autoencoder&#10;(VAE), and a discrete Vector Quantized VAE (VQ-VAE). We analyze the quality of&#10;learned representations in terms of speaker independence, the ability to&#10;predict phonetic content, and the ability to accurately reconstruct individual&#10;spectrogram frames. Moreover, for discrete encodings extracted using the&#10;VQ-VAE, we measure the ease of mapping them to phonemes. We introduce a&#10;regularization scheme that forces the representations to focus on the phonetic&#10;content of the utterance and report performance comparable with the top entries&#10;in the ZeroSpeech 2017 unsupervised acoustic unit discovery task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.06280" label="2110.06280">
        <attvalues>
          <attvalue for="0" value="S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised&#10;  Speech Representations" />
          <attvalue for="1" value="  This paper introduces S3PRL-VC, an open-source voice conversion (VC)&#10;framework based on the S3PRL toolkit. In the context of recognition-synthesis&#10;VC, self-supervised speech representation (S3R) is valuable in its potential to&#10;replace the expensive supervised representation adopted by state-of-the-art VC&#10;systems. Moreover, we claim that VC is a good probing task for S3R analysis. In&#10;this work, we provide a series of in-depth analyses by benchmarking on the two&#10;tasks in VCC2020, namely intra-/cross-lingual any-to-one (A2O) VC, as well as&#10;an any-to-any (A2A) setting. We also provide comparisons between not only&#10;different S3Rs but also top systems in VCC2020 with supervised representations.&#10;Systematic objective and subjective evaluation were conducted, and we show that&#10;S3R is comparable with VCC2020 top systems in the A2O setting in terms of&#10;similarity, and achieves state-of-the-art in S3R-based A2A VC. We believe the&#10;extensive analysis, as well as the toolkit itself, contribute to not only the&#10;S3R community but also the VC community. The codebase is now open-sourced.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.03509" label="2110.03509">
        <attvalues>
          <attvalue for="0" value="Analyzing the Robustness of Unsupervised Speech Recognition" />
          <attvalue for="1" value="  Unsupervised speech recognition (unsupervised ASR) aims to learn the ASR&#10;system with non-parallel speech and text corpus only. Wav2vec-U has shown&#10;promising results in unsupervised ASR by self-supervised speech representations&#10;coupled with Generative Adversarial Network (GAN) training, but the robustness&#10;of the unsupervised ASR framework is unknown. In this work, we further analyze&#10;the training robustness of unsupervised ASR on the domain mismatch scenarios in&#10;which the domains of unpaired speech and text are different. Three domain&#10;mismatch scenarios include: (1) using speech and text from different datasets,&#10;(2) utilizing noisy/spontaneous speech, and (3) adjusting the amount of speech&#10;and text data. We also quantify the degree of the domain mismatch by&#10;calculating the JS-divergence of phoneme n-gram between the transcription of&#10;speech and text. This metric correlates with the performance highly.&#10;Experimental results show that domain mismatch leads to inferior performance,&#10;but a self-supervised model pre-trained on the targeted speech domain can&#10;extract better representation to alleviate the performance drop.&#10;" />
          <attvalue for="2" value="&#10;Automatic speech recognition (ASR) is a long-standing research area that predicts transcriptions given the input speech feature. The performance of ASR has rapidly improved due to the deep learning method with an increasing amount of labeled data. However, collecting hundreds or thousands of hours of paired speech and text data is costly and even infeasible for the low-resource or endangered languages. &#10;&#10;On the other hand, unsupervised ASR aims only to leverage unpaired speech and text data for training an ASR system. Unsupervised ASR systems learn the cross-modal mapping between speech and text \cite{yeh2018unsupervised, aldarmaki2021unsupervised, Liu2020TowardsUS,liu2018completely,chen2019completely}. &#10;In particular, \cite{liu2018completely} and \cite{chen2019completely} propose to learn the mapping by Generative Adversarial Network (GAN) \cite{goodfellow2014generative}. They demonstrate encouraging results for completely unsupervised ASR training, but the performance is still far from the supervised ASR system. Notably, a recent work named Wav2vec-U \cite{baevski2021unsupervised} has shown remarkable performance breakthroughs in unsupervised ASR. The performance is comparable with some supervised methods \cite{amodei2016deep, zhang2020pushing, xu2018neural}.&#10;&#10;This paper intends to investigate the robustness of Wav2vec-U under three different domain mismatch scenarios, where the domain of unpaired speech and text are different. Figure \ref{fig:main} shows the main concept of our work. The three scenarios are as below:\\&#10;Content dissimilarity: &#10;One major experiment in Wav2vec-U is trained with an unpaired speech from Librispeech \cite{7178964} and text from Librispeech language modeling corpus, which both come from the audiobook but are collected in speech and text form, respectively. The experimental setup is unrealistic in the unsupervised learning scenario because no prior knowledge about speech transcription exists. The intrinsic properties of the selected text corpus and speech data may be reasonably different.&#10;Therefore, we investigate the robustness of Wav2vec-U by utilizing the speech and text that come from different datasets to simulate dissimilarities in content. \\&#10;Spontaneous speech:&#10;The audio data used in the training of Wav2vec-U, even in low-resource experiments, are all read speech.&#10;Read speech is clean and well-structured;&#10;on the other hand, real-world speech data contains noises and disfluencies. Furthermore, real-world speech lacks literary expressions and causes dissimilarity to text corpus in content. &#10;To investigate the robustness of Wav2vec-U with real-world speech, we conduct further experiments on spontaneous speech datasets such as conversational speech and live talk, which is real-time generated with the casual styles. &#10;\\&#10;Quantity limitation: &#10;The results of Wav2vec-U show that it is possible to train an unsupervised ASR model by a minimal amount of speech and text from the audiobook. However, in the domain-mismatched conditions, whether we can still use a limited amount of data to train an unsupervised ASR model is under-explored. The more significant variance present in fewer data intensifies the degree of&#10;domain mismatch. Hence, we restrict the amount of data in domain mismatch conditions for testing the training robustness. &#10;&#10;The main findings of our work are listed below: &#10;\begin{itemize}&#10; \item When the content of speech and text is dissimilar, using a large amount of text retains the performance, but reducing to the small amount of texts considerably degrades the results.&#10; \item We discover that 4-gram information distribution of phoneme is critical for successfully training the GAN-based unsupervised ASR model.&#10; \item Self-supervised pre-training on targeted speech domain enables extracting better feature representation and significantly improves the performance.&#10;\end{itemize}&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.12559" label="2203.12559">
        <attvalues>
          <attvalue for="0" value="A Scalable Model Specialization Framework for Training and Inference&#10;  using Submodels and its Application to Speech Model Personalization" />
          <attvalue for="1" value="  Model fine-tuning and adaptation have become a common approach for model&#10;specialization for downstream tasks or domains. Fine-tuning the entire model or&#10;a subset of the parameters using light-weight adaptation has shown considerable&#10;success across different specialization tasks. Fine-tuning a model for a large&#10;number of domains typically requires starting a new training job for every&#10;domain posing scaling limitations. Once these models are trained, deploying&#10;them also poses significant scalability challenges for inference for real-time&#10;applications. In this paper, building upon prior light-weight adaptation&#10;techniques, we propose a modular framework that enables us to substantially&#10;improve scalability for model training and inference. We introduce Submodels&#10;that can be quickly and dynamically loaded for on-the-fly inference. We also&#10;propose multiple approaches for training those Submodels in parallel using an&#10;embedding space in the same training job. We test our framework on an extreme&#10;use-case which is speech model personalization for atypical speech, requiring a&#10;Submodel for each user. We obtain 128x Submodel throughput with a fixed&#10;computation budget without a loss of accuracy. We also show that learning a&#10;speaker-embedding space can scale further and reduce the amount of&#10;personalization training data required per speaker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.09296" label="2106.09296">
        <attvalues>
          <attvalue for="0" value="Voice2Series: Reprogramming Acoustic Models for Time Series&#10;  Classification" />
          <attvalue for="1" value="  Learning to classify time series with limited data is a practical yet&#10;challenging problem. Current methods are primarily based on hand-designed&#10;feature extraction rules or domain-specific data augmentation. Motivated by the&#10;advances in deep speech processing models and the fact that voice data are&#10;univariate temporal signals, in this paper, we propose Voice2Series (V2S), a&#10;novel end-to-end approach that reprograms acoustic models for time series&#10;classification, through input transformation learning and output label mapping.&#10;Leveraging the representation learning power of a large-scale pre-trained&#10;speech processing model, on 30 different time series tasks we show that V2S&#10;performs competitive results on 19 time series classification tasks. We further&#10;provide a theoretical justification of V2S by proving its population risk is&#10;upper bounded by the source risk and a Wasserstein distance accounting for&#10;feature alignment via reprogramming. Our results offer new and effective means&#10;to time series classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.01949" label="2311.01949">
        <attvalues>
          <attvalue for="0" value="Hint-enhanced In-Context Learning wakes Large Language Models up for&#10;  knowledge-intensive tasks" />
          <attvalue for="1" value="  In-context learning (ICL) ability has emerged with the increasing scale of&#10;large language models (LLMs), enabling them to learn input-label mappings from&#10;demonstrations and perform well on downstream tasks. However, under the&#10;standard ICL setting, LLMs may sometimes neglect query-related information in&#10;demonstrations, leading to incorrect predictions. To address this limitation,&#10;we propose a new paradigm called Hint-enhanced In-Context Learning (HICL) to&#10;explore the power of ICL in open-domain question answering, an important form&#10;in knowledge-intensive tasks. HICL leverages LLMs' reasoning ability to extract&#10;query-related knowledge from demonstrations, then concatenates the knowledge to&#10;prompt LLMs in a more explicit way. Furthermore, we track the source of this&#10;knowledge to identify specific examples, and introduce a Hint-related Example&#10;Retriever (HER) to select informative examples for enhanced demonstrations. We&#10;evaluate HICL with HER on 3 open-domain QA benchmarks, and observe average&#10;performance gains of 2.89 EM score and 2.52 F1 score on gpt-3.5-turbo, 7.62 EM&#10;score and 7.27 F1 score on LLaMA-2-Chat-7B compared with standard setting.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) with in-context learning (ICL) ability have attracted wide attention. Prompted by demonstrations consisting of a few input-label pairs, LLMs perform well even on unseen tasks \cite{brown2020language, 2023arXivSurvey}.&#10;ICL ability strongly depends on selected training examples \cite{liu2021makes,2021arXivCalibrate}, and some methods \cite{gonen2022demystifying, guo2023connecting} are designed to form a high-quality demonstration.&#10;Recent works have explored what makes ICL work in LLMs.&#10;Following the format of examples in demonstrations, language models can predict the right labels by utilizing prior knowledge acquired from pretraining~\cite{min2022rethinking, Shisg}.&#10;As the model scale further increases, LLMs can acquire knowledge directly from input-label mappings in demonstrations \cite{wei2023larger}. &#10;&#10;However, we found that the ICL ability of LLMs is limited in knowledge-intensive tasks, using the input form $\{x_1, y_1, \ldots, x_k, y_k; query\}$ as standard ICL setting. &#10;As shown in Figure~\ref{fig:HICL}, firstly, we verified that LLM knows [\underline{Helsinki} is \underline{Finland}'s capital] indeed. Nevertheless, when coming to the question [When was child benefit first paid in \underline{Helsinki} and implemented nationwide?], LLM still has a certain probability of neglecting the knowledge in [Q: When did child benefit start in \underline{Finland} following other Nordic countries? A: 1948].&#10;To obtain a universal conclusion, we conduct a pilot experiment. Under 5-shot setting, we found that 42.2\% of test queries contain corresponding knowledge in demonstrations, but LLM with standard ICL only predicts correctly for 69.9\% in this subset.&#10;To better exploit the information contained in demonstrations, we propose a new paradigm called Hint enhanced In-Context Learning (HICL).&#10;&#10;Firstly, we generate hints to present query-related information in a more explicit way. &#10;Prior works \cite{Self-Ask, DSP} investigate the ability of LLMs to perform compositional reasoning tasks in multi-hop question answering tasks. With structured prompts, we can further explore the potential of frozen LLMs.&#10;Hence, we propose a prompt-based knowledge-extracting scheme to extract valuable information from selected examples, leveraging LLMs' reasoning and summarizing ability \cite{2022arXivCoT,2023arXivSummarization}. &#10;As shown in Figure~\ref{fig:HICL}, LLM generates the hint [Child benefit was first paid in Helsinki and implemented nationwide in Finland in 1948.] based on related examples.&#10;In this way, the informative knowledge contained in demonstrations will be explored fully by the LLM, which helps LLM to answer the question consequently. &#10;&#10;With hints generated by the LLM, we can locate specific examples that provide knowledge. &#10;But in some cases, hint-related examples are not the closest to the test queries. &#10;We further analyze the pilot experiment\footref{pilot_exp} and locate specific examples that provide knowledge. Nearly 21\% of hint-related examples are not ranked top-1 when we can extract hints under 5-shot setting.&#10;It turns out that the example selection method still has room for improvement, and we are inspired to propose a Hint-related Example Retriever (HER). &#10;We use hints as supervisory information to label examples as positive or negative by calculating the similarity score, then train a retriever by contrastive learning from the data. This paradigm can also be applied to Black-Box LLMs, where the output probability distribution is not accessible.&#10;&#10;To conclude, the main contributions of our work are as follows: &#10;(1) HICL effectively enhances ICL ability in knowledge-intensive tasks with hints generated by LLMs. Experiments show that our proposed paradigm improves the performance on 3 open-domain QA benchmarks, where LLMs may fail to leverage relevant knowledge in demonstrations.&#10;(2) To retrieve hint-related examples better, we propose a new example selection model called Hint-related Example Retriever (HER). Through HER, we can retrieve examples with query-related knowledge more effectively. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Cognitive Science, Knowledge Retrieval Methods, Language Model Learning, Artificial Intelligence, Question Answering Systems, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2301.13848" label="2301.13848">
        <attvalues>
          <attvalue for="0" value="Benchmarking Large Language Models for News Summarization" />
          <attvalue for="1" value="  Large language models (LLMs) have shown promise for automatic summarization&#10;but the reasons behind their successes are poorly understood. By conducting a&#10;human evaluation on ten LLMs across different pretraining methods, prompts, and&#10;model scales, we make two important observations. First, we find instruction&#10;tuning, and not model size, is the key to the LLM's zero-shot summarization&#10;capability. Second, existing studies have been limited by low-quality&#10;references, leading to underestimates of human performance and lower few-shot&#10;and finetuning performance. To better evaluate LLMs, we perform human&#10;evaluation over high-quality summaries we collect from freelance writers.&#10;Despite major stylistic differences such as the amount of paraphrasing, we find&#10;that LMM summaries are judged to be on par with human written summaries.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.04054" label="2304.04054">
        <attvalues>
          <attvalue for="0" value="tmn at SemEval-2023 Task 9: Multilingual Tweet Intimacy Detection using&#10;  XLM-T, Google Translate, and Ensemble Learning" />
          <attvalue for="1" value="  The paper describes a transformer-based system designed for SemEval-2023 Task&#10;9: Multilingual Tweet Intimacy Analysis. The purpose of the task was to predict&#10;the intimacy of tweets in a range from 1 (not intimate at all) to 5 (very&#10;intimate). The official training set for the competition consisted of tweets in&#10;six languages (English, Spanish, Italian, Portuguese, French, and Chinese). The&#10;test set included the given six languages as well as external data with four&#10;languages not presented in the training set (Hindi, Arabic, Dutch, and Korean).&#10;We presented a solution based on an ensemble of XLM-T, a multilingual RoBERTa&#10;model adapted to the Twitter domain. To improve the performance of unseen&#10;languages, each tweet was supplemented by its English translation. We explored&#10;the effectiveness of translated data for the languages seen in fine-tuning&#10;compared to unseen languages and estimated strategies for using translated data&#10;in transformer-based models. Our solution ranked 4th on the leaderboard while&#10;achieving an overall Pearson's r of 0.599 over the test set. The proposed&#10;system improves up to 0.088 Pearson's r over a score averaged across all 45&#10;submissions.&#10;" />
          <attvalue for="2" value="&#10;&#10;Intimacy is a significant social aspect of language, which helps to explore existing social norms in various contexts \cite{pei2020quantifying}. The concept of intimacy describes how an individual relates to his addressee in their perceived interdependence, warmth, and willingness to personally share \cite{perlman1987development}. An automatic evaluation of intimacy in language provides us with a clearer picture of social interactions and different linguistic strategies.&#10;&#10;This paper describes a system developed for the SemEval-2023 Task 9:&#10;Multilingual Tweet Intimacy Analysis \cite{pei2022semeval}. The purpose of the task was to predict the intimacy of tweets. The test set contained tweets in the languages presented (seen) and not presented (unseen) in the training set. The main challenge of the task was the zero-shot intimacy prediction performance for unseen languages.&#10;&#10;Inspired by the recent success of transformer-based models in multilingual tasks \cite{malmasi-etal-2022-semeval,tayyar-madabushi-etal-2022-semeval}, we evaluate the performance of BERT (Bidirectional Encoder Representations from Transformers) \cite{devlin-etal-2019-bert} and its modifications. In our experiments, we investigated the impact of translation to improve the zero-shot performance. We have found that the joint use of original and translated texts increases the model performance for unseen languages in comparison with the separate use of original or translated data.&#10;&#10;The paper is organized as follows. Section \ref{sec2} presents the task and the dataset provided by the task organizers. Our methods are presented in Section \ref{sec3}. In Section \ref{sec4}, we describe the experimental setups that we used during the development phase of the competition. We discuss our results in Section \ref{sec5}. Finally, Section \ref{sec6} concludes this paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Computational Linguistics, Artificial Intelligence, Sentiment Prediction, Transformer Models, Natural Language Processing, Multilingual Analysis, Statistics" />
        </attvalues>
      </node>
      <node id="2011.03020" label="2011.03020">
        <attvalues>
          <attvalue for="0" value="Quantifying Intimacy in Language" />
          <attvalue for="1" value="  Intimacy is a fundamental aspect of how we relate to others in social&#10;settings. Language encodes the social information of intimacy through both&#10;topics and other more subtle cues (such as linguistic hedging and swearing).&#10;Here, we introduce a new computational framework for studying expressions of&#10;the intimacy in language with an accompanying dataset and deep learning model&#10;for accurately predicting the intimacy level of questions (Pearson's r=0.87).&#10;Through analyzing a dataset of 80.5M questions across social media, books, and&#10;films, we show that individuals employ interpersonal pragmatic moves in their&#10;language to align their intimacy with social settings. Then, in three studies,&#10;we further demonstrate how individuals modulate their intimacy to match social&#10;norms around gender, social distance, and audience, each validating key&#10;findings from studies in social psychology. Our work demonstrates that intimacy&#10;is a pervasive and impactful social dimension of language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.01108" label="2210.01108">
        <attvalues>
          <attvalue for="0" value="SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis" />
          <attvalue for="1" value="  We propose MINT, a new Multilingual INTimacy analysis dataset covering 13,372&#10;tweets in 10 languages including English, French, Spanish, Italian, Portuguese,&#10;Korean, Dutch, Chinese, Hindi, and Arabic. We benchmarked a list of popular&#10;multilingual pre-trained language models. The dataset is released along with&#10;the SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis&#10;(https://sites.google.com/umich.edu/semeval-2023-tweet-intimacy).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.12638" label="2206.12638">
        <attvalues>
          <attvalue for="0" value="Distilling a Pretrained Language Model to a Multilingual ASR Model" />
          <attvalue for="1" value="  Multilingual speech data often suffer from long-tailed language distribution,&#10;resulting in performance degradation. However, multilingual text data is much&#10;easier to obtain, yielding a more useful general language model. Hence, we are&#10;motivated to distill the rich knowledge embedded inside a well-trained teacher&#10;text model to the student speech model. We propose a novel method called the&#10;Distilling a Language model to a Speech model (Distill-L2S), which aligns the&#10;latent representations of two different modalities. The subtle differences are&#10;handled by the shrinking mechanism, nearest-neighbor interpolation, and a&#10;learnable linear projection layer. We demonstrate the effectiveness of our&#10;distillation method by applying it to the multilingual automatic speech&#10;recognition (ASR) task. We distill the transformer-based cross-lingual language&#10;model (InfoXLM) while fine-tuning the large-scale multilingual ASR model&#10;(XLSR-wav2vec 2.0) for each language. We show the superiority of our method on&#10;20 low-resource languages of the CommonVoice dataset with less than 100 hours&#10;of speech data.&#10;" />
          <attvalue for="2" value="&#10;&#10;Compared to other machine learning domains, obtaining speech data is expensive, ending up in unavoidable performance degradation of the automatic speech recognition (ASR) model.&#10;Further, collecting non-English speech data is exceptionally difficult as there are not so many speakers available \cite{ardila2020common,winata2020adapt,pratap2020massively}.&#10;In contrast, text models have been successfully developed and widely utilized by improving the performance based on the vast web-crawled corpus \cite{wenzek2020ccnet,chelba2013one,conneau2019unsupervised}.&#10;Especially, the rise of end-to-end (E2E) transformer-based models showed numerous successes \cite{vaswani2017attention,raffel2019exploring,brown2020language,devlin2018bert}, such as cross-lingual language models \cite{conneau2019unsupervised,chi2021infoxlm}.&#10;Motivated by this, transformer architecture is becoming increasingly common for E2E ASR models \cite{winata2020adapt,baevski2021unsupervised,hsu2021hubert,sadhu2021wav2vec,baevski2020wav2vec,schneider2019wav2vec,baevski2019vq}, where its generalization performance is still limited compared to that of text models.&#10;Furthermore, many approaches have been introduced for multilingual ASR, such as adapter modules \cite{winata2020adapt,kannan2019large,hou2021meta}, multi-head architecture \cite{pratap2020massively,sercu2016very,dalmia2018sequence}, logit adjustment \cite{winata2020adapt}, language-dependent batching \cite{kannan2019large,sercu2016very}, multi-task training \cite{toshniwal2018multilingual,li2018multi}, language embeddings \cite{toshniwal2018multilingual,li2018multi}, and cross-lingual training \cite{baevski2021unsupervised}.&#10;However, ASR performance has been limited for minor languages, e.g., languages with less than 100 hours of speech.&#10;Hence, it naturally raises the following intuition: Can we utilize the well-trained cross-lingual language models to increase the performance of multilingual ASR models on low-resource languages?&#10;&#10;One of the main approaches to using one model to help the other is knowledge distillation (KD) \cite{hinton2015distilling,cho2020speech,liu2019end,jiao2019tinybert,sanh2019distilbert,choi2021temporal,kim2021two}.&#10;KD transfers the information embedded inside the latent representations of the teacher model to the student model, which improves the predictive performance of the student model without any architectural modifications \cite{hinton2015distilling}.&#10;Hence, it can enjoy the better of two worlds, easily adding the KD loss in a plug-and-play manner to the existing model training, yet enjoying the better performance with the aid of the teacher model.&#10;For the classification models, it is common to distill by minimizing the distance between two output logits of the teacher and student networks \cite{hinton2015distilling,cho2020speech,sanh2019distilbert}.&#10;Also, many transformer-specific KD methods have been introduced to distill the information more effectively \cite{jiao2019tinybert,sanh2019distilbert,choi2021temporal,kim2021two}.&#10;\cite{jiao2019tinybert} minimizes the distance between the attention maps or feature outputs of two transformer models with different sizes.&#10;\cite{choi2021temporal} distills the attention maps of a transformer-based audio model to smaller architectures such as convolutional or recurrent neural networks.&#10;&#10;Although the above methods concentrate on distilling between the same modalities, some methods bridge the differences between speech and text.&#10;\cite{cho2020speech} distills between text and speech models by minimizing the distance between the classification probabilities, whereas \cite{kim2021two} minimizes the distances between the first hidden representations or the output logits.&#10;Both works sidestep the discrepancy problem between speech and text modalities by disregarding the sequence-level features.&#10;\cite{winata2020adapt} handles the multilingual ASR by directly copying the weights of the cross-lingual language model to the ASR model's transformer decoder.&#10;\cite{baevski2021unsupervised} uses $k$-means clustering to segment the audio into phonemic units, where text input is also phonemicized for adversarial training to distinguish the features between two different modalities.&#10;\cite{chung2018unsupervised} aligns the text and speech embeddings via a learnable linear transform layer in an unsupervised manner.&#10;Finally, a simple $n$-gram language model is commonly applied to the final predictions to improve the ASR performance \cite{baevski2021unsupervised,schneider2019wav2vec,conneau2020unsupervised}.&#10;We did not utilize the method in our work, but we emphasize that it can be used in parallel with the KD methods.&#10;KD methods try to yield better predictions in the first place, where using the $n$-gram language model concentrates on refining the output predictions further.&#10;&#10;However, aligning the sequence-level temporal features between text and speech data remains challenging.&#10;The shrink mechanism \cite{chen2016phone} is commonly used to align temporal information between speech and text.&#10;It clusters the neighboring outputs into speech segments while discarding the non-speech segments.&#10;\cite{liu2020bridging} propose a model that contains speech and text models as its components, where it concentrates on solving the speech-to-text translation task.&#10;\cite{yi2019ectc} introduces an encoder-decoder architecture with the losses that accommodates each module to solve the ASR task.&#10;\cite{di2019one} inputs speech and text to the transformer encoder and decoder, respectively, improving the performance of speech translation task.&#10;&#10;In this paper, we propose a novel method, Distilling a Language model to a Speech model (Distill-L2S), to leverage the rich knowledge from a well-trained general text model to increase the multilingual ASR performance, especially for low-resource languages.&#10;Our method distills the knowledge by minimizing the differences between text and speech features.&#10;We use various methods to mitigate the inherent discrepancy between the two modalities, such as the shrink method \cite{chen2016phone}, nearest-neighbor interpolation \cite{itseez2014theopencv}, and a learnable linear transformation layer \cite{jiao2019tinybert,chung2018unsupervised}.&#10;Unlike previous methods, our method avoids additional modification of the pre-existing ASR architectures, only requiring the speech features to be sequence-based.&#10;We distill the knowledge of the transformer-based cross-lingual text model, InfoXLM \cite{chi2021infoxlm}, while fine-tuning the transformer-based large-scale ASR model, XLSR-wav2vec 2.0 \cite{conneau2020unsupervised}.&#10;We experiment on 20 low-resource languages of the CommonVoice dataset \cite{ardila2020common} to demonstrate the effectiveness of our method.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Distillation, Cross-Lingual Knowledge Transfer, Multilingual Speech Recognition, Artificial Intelligence, Signal Processing" />
        </attvalues>
      </node>
      <node id="1912.06670" label="1912.06670">
        <attvalues>
          <attvalue for="0" value="Common Voice: A Massively-Multilingual Speech Corpus" />
          <attvalue for="1" value="  The Common Voice corpus is a massively-multilingual collection of transcribed&#10;speech intended for speech technology research and development. Common Voice is&#10;designed for Automatic Speech Recognition purposes but can be useful in other&#10;domains (e.g. language identification). To achieve scale and sustainability,&#10;the Common Voice project employs crowdsourcing for both data collection and&#10;data validation. The most recent release includes 29 languages, and as of&#10;November 2019 there are a total of 38 languages collecting data. Over 50,000&#10;individuals have participated so far, resulting in 2,500 hours of collected&#10;audio. To our knowledge this is the largest audio corpus in the public domain&#10;for speech recognition, both in terms of number of hours and number of&#10;languages. As an example use case for Common Voice, we present speech&#10;recognition experiments using Mozilla's DeepSpeech Speech-to-Text toolkit. By&#10;applying transfer learning from a source English model, we find an average&#10;Character Error Rate improvement of 5.99 +/- 5.48 for twelve target languages&#10;(German, French, Italian, Turkish, Catalan, Slovenian, Welsh, Irish, Breton,&#10;Tatar, Chuvash, and Kabyle). For most of these languages, these are the first&#10;ever published results on end-to-end Automatic Speech Recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.01687" label="2012.01687">
        <attvalues>
          <attvalue for="0" value="Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual&#10;  Speech Recognition" />
          <attvalue for="1" value="  One crucial challenge of real-world multilingual speech recognition is the&#10;long-tailed distribution problem, where some resource-rich languages like&#10;English have abundant training data, but a long tail of low-resource languages&#10;have varying amounts of limited training data. To overcome the long-tail&#10;problem, in this paper, we propose Adapt-and-Adjust (A2), a transformer-based&#10;multi-task learning framework for end-to-end multilingual speech recognition.&#10;The A2 framework overcomes the long-tail problem via three techniques: (1)&#10;exploiting a pretrained multilingual language model (mBERT) to improve the&#10;performance of low-resource languages; (2) proposing dual adapters consisting&#10;of both language-specific and language-agnostic adaptation with minimal&#10;additional parameters; and (3) overcoming the class imbalance, either by&#10;imposing class priors in the loss during training or adjusting the logits of&#10;the softmax output during inference. Extensive experiments on the CommonVoice&#10;corpus show that A2 significantly outperforms conventional approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Long-Tail&#10;Conventional approaches to addressing the long-tail problem are focused on data sampling methods~\cite{kubat1997addressing,chawla2002smote,wallace2011class}.&#10;Recently, the long-tail distribution issue has regained interest for neural network models~\cite{menon2020long}, and several approaches have been proposed, such as weight normalization~\cite{kang2019decoupling}, adaptive margin~\cite{cao2019learning}, and equalized loss~\cite{tan2020equalization}.&#10;&#10;Adapters&#10;Adapters were first proposed to learn domain-specific representations in computer vision in a parameter-efficient way~\cite{rebuffi2017learning}. They were subsequently adopted for NLP tasks to avoid fine-tuning a new model for each task by training an adapter module for each task while sharing the pre-trained language model parameters~\cite{houlsby2019parameter,lin2020exploring}. Invertible adapters were proposed in~\cite{pfeiffer2020adapterhub,pfeiffer2020mad} to effectively adapt an existing pre-trained multilingual model to unseen languages for multi-task cross-lingual transfer.&#10;&#10;Multilingual ASR&#10;E2E architectures like LAS~\cite{Toshniwal_2018} and the Recurrent Neural Transducer~\cite{kannan2019large} have been used for building a multilingual ASR system for a group of Indian languages. In~\cite{kannan2019large}, language adapters are used to tackle the data imbalance problem, although the improvement from using language adapters is marginal compared to the language vector augmentation. &#10;Acoustic vector quantization is also used in the recent work by~\cite{conneau2020unsupervised} on multilingual ASR. A massive multilingual ASR study with more than 50 languages and more than 16,000 hours of speech is presented in~\cite{pratap2020massively}. &#10;The two main techniques are data resampling and language clusters, which bear some similarities with our balanced sampling and language adapters. Unfortunately, their datasets are not publicly available.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1911.00359" label="1911.00359">
        <attvalues>
          <attvalue for="0" value="CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data" />
          <attvalue for="1" value="  Pre-training text representations have led to significant improvements in&#10;many areas of natural language processing. The quality of these models benefits&#10;greatly from the size of the pretraining corpora as long as its quality is&#10;preserved. In this paper, we describe an automatic pipeline to extract massive&#10;high-quality monolingual datasets from Common Crawl for a variety of languages.&#10;Our pipeline follows the data processing introduced in fastText (Mikolov et&#10;al., 2017; Grave et al., 2018), that deduplicates documents and identifies&#10;their language. We augment this pipeline with a filtering step to select&#10;documents that are close to high quality corpora like Wikipedia.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.11084" label="2105.11084">
        <attvalues>
          <attvalue for="0" value="Unsupervised Speech Recognition" />
          <attvalue for="1" value="  Despite rapid progress in the recent past, current speech recognition systems&#10;still require labeled training data which limits this technology to a small&#10;fraction of the languages spoken around the globe. This paper describes&#10;wav2vec-U, short for wav2vec Unsupervised, a method to train speech recognition&#10;models without any labeled data. We leverage self-supervised speech&#10;representations to segment unlabeled audio and learn a mapping from these&#10;representations to phonemes via adversarial training. The right representations&#10;are key to the success of our method. Compared to the best previous&#10;unsupervised work, wav2vec-U reduces the phoneme error rate on the TIMIT&#10;benchmark from 26.1 to 11.3. On the larger English Librispeech benchmark,&#10;wav2vec-U achieves a word error rate of 5.9 on test-other, rivaling some of the&#10;best published systems trained on 960 hours of labeled data from only two years&#10;ago. We also experiment on nine other languages, including low-resource&#10;languages such as Kyrgyz, Swahili and Tatar.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1509.08967" label="1509.08967">
        <attvalues>
          <attvalue for="0" value="Very Deep Multilingual Convolutional Neural Networks for LVCSR" />
          <attvalue for="1" value="  Convolutional neural networks (CNNs) are a standard component of many current&#10;state-of-the-art Large Vocabulary Continuous Speech Recognition (LVCSR)&#10;systems. However, CNNs in LVCSR have not kept pace with recent advances in&#10;other domains where deeper neural networks provide superior performance. In&#10;this paper we propose a number of architectural advances in CNNs for LVCSR.&#10;First, we introduce a very deep convolutional network architecture with up to&#10;14 weight layers. There are multiple convolutional layers before each pooling&#10;layer, with small 3x3 kernels, inspired by the VGG Imagenet 2014 architecture.&#10;Then, we introduce multilingual CNNs with multiple untied layers. Finally, we&#10;introduce multi-scale input features aimed at exploiting more context at&#10;negligible computational cost. We evaluate the improvements first on a Babel&#10;task for low resource speech recognition, obtaining an absolute 5.77% WER&#10;improvement over the baseline PLP DNN by training our CNN on the combined data&#10;of six different languages. We then evaluate the very deep CNNs on the Hub5'00&#10;benchmark (using the 262 hours of SWB-1 training data) achieving a word error&#10;rate of 11.8% after cross-entropy training, a 1.4% WER improvement (10.6%&#10;relative) over the best published CNN result so far.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.07420" label="1802.07420">
        <attvalues>
          <attvalue for="0" value="Sequence-based Multi-lingual Low Resource Speech Recognition" />
          <attvalue for="1" value="  Techniques for multi-lingual and cross-lingual speech recognition can help in&#10;low resource scenarios, to bootstrap systems and enable analysis of new&#10;languages and domains. End-to-end approaches, in particular sequence-based&#10;techniques, are attractive because of their simplicity and elegance. While it&#10;is possible to integrate traditional multi-lingual bottleneck feature&#10;extractors as front-ends, we show that end-to-end multi-lingual training of&#10;sequence models is effective on context independent models trained using&#10;Connectionist Temporal Classification (CTC) loss. We show that our model&#10;improves performance on Babel languages by over 6% absolute in terms of&#10;word/phoneme error rate when compared to mono-lingual systems built in the same&#10;setting for these languages. We also show that the trained model can be adapted&#10;cross-lingually to an unseen language using just 25% of the target data. We&#10;show that training on multiple languages is important for very low resource&#10;cross-lingual target scenarios, but not for multi-lingual testing scenarios.&#10;Here, it appears beneficial to include large well prepared datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.01694" label="1711.01694">
        <attvalues>
          <attvalue for="0" value="Multilingual Speech Recognition With A Single End-To-End Model" />
          <attvalue for="1" value="  Training a conventional automatic speech recognition (ASR) system to support&#10;multiple languages is challenging because the sub-word unit, lexicon and word&#10;inventories are typically language specific. In contrast, sequence-to-sequence&#10;models are well suited for multilingual ASR because they encapsulate an&#10;acoustic, pronunciation and language model jointly in a single network. In this&#10;work we present a single sequence-to-sequence ASR model trained on 9 different&#10;Indian languages, which have very little overlap in their scripts.&#10;Specifically, we take a union of language-specific grapheme sets and train a&#10;grapheme-based sequence-to-sequence model jointly on data from all languages.&#10;We find that this model, which is not explicitly given any information about&#10;language identity, improves recognition performance by 21% relative compared to&#10;analogous sequence-to-sequence models trained on each language individually. By&#10;modifying the model to accept a language identifier as an additional input&#10;feature, we further improve performance by an additional 7% relative and&#10;eliminate confusion between different languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.08213" label="2005.08213">
        <attvalues>
          <attvalue for="0" value="Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillation" />
          <attvalue for="1" value="  Speech is one of the most effective means of communication and is full of&#10;information that helps the transmission of utterer's thoughts. However, mainly&#10;due to the cumbersome processing of acoustic features, phoneme or word&#10;posterior probability has frequently been discarded in understanding the&#10;natural language. Thus, some recent spoken language understanding (SLU) modules&#10;have utilized end-to-end structures that preserve the uncertainty information.&#10;This further reduces the propagation of speech recognition error and guarantees&#10;computational efficiency. We claim that in this process, the speech&#10;comprehension can benefit from the inference of massive pre-trained language&#10;models (LMs). We transfer the knowledge from a concrete Transformer-based text&#10;LM to an SLU module which can face a data shortage, based on recent cross-modal&#10;distillation methodologies. We demonstrate the validity of our proposal upon&#10;the performance on Fluent Speech Command, an English SLU benchmark. Thereby, we&#10;experimentally verify our hypothesis that the knowledge could be shared from&#10;the top layer of the LM to a fully speech-based module, in which the abstracted&#10;speech is expected to meet the semantic representation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.08075" label="1904.08075">
        <attvalues>
          <attvalue for="0" value="End-to-End Speech Translation with Knowledge Distillation" />
          <attvalue for="1" value="  End-to-end speech translation (ST), which directly translates from source&#10;language speech into target language text, has attracted intensive attentions&#10;in recent years. Compared to conventional pipeline systems, end-to-end ST&#10;models have advantages of lower latency, smaller model size and less error&#10;propagation. However, the combination of speech recognition and text&#10;translation in one model is more difficult than each of these two tasks. In&#10;this paper, we propose a knowledge distillation approach to improve ST model by&#10;transferring the knowledge from text translation model. Specifically, we first&#10;train a text translation model, regarded as a teacher model, and then ST model&#10;is trained to learn output probabilities from teacher model through knowledge&#10;distillation. Experiments on English- French Augmented LibriSpeech and&#10;English-Chinese TED corpus show that end-to-end ST is possible to implement on&#10;both similar and dissimilar language pairs. In addition, with the instruction&#10;of teacher model, end-to-end ST model can gain significant improvements by over&#10;3.5 BLEU points.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.14131" label="2110.14131">
        <attvalues>
          <attvalue for="0" value="Temporal Knowledge Distillation for On-device Audio Classification" />
          <attvalue for="1" value="  Improving the performance of on-device audio classification models remains a&#10;challenge given the computational limits of the mobile environment. Many&#10;studies leverage knowledge distillation to boost predictive performance by&#10;transferring the knowledge from large models to on-device models. However, most&#10;lack a mechanism to distill the essence of the temporal information, which is&#10;crucial to audio classification tasks, or similar architecture is often&#10;required. In this paper, we propose a new knowledge distillation method&#10;designed to incorporate the temporal knowledge embedded in attention weights of&#10;large transformer-based models into on-device models. Our distillation method&#10;is applicable to various types of architectures, including the&#10;non-attention-based architectures such as CNNs or RNNs, while retaining the&#10;original network architecture during inference. Through extensive experiments&#10;on both an audio event detection dataset and a noisy keyword spotting dataset,&#10;we show that our proposed method improves the predictive performance across&#10;diverse on-device architectures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13105" label="2010.13105">
        <attvalues>
          <attvalue for="0" value="Two-stage Textual Knowledge Distillation for End-to-End Spoken Language&#10;  Understanding" />
          <attvalue for="1" value="  End-to-end approaches open a new way for more accurate and efficient spoken&#10;language understanding (SLU) systems by alleviating the drawbacks of&#10;traditional pipeline systems. Previous works exploit textual information for an&#10;SLU model via pre-training with automatic speech recognition or fine-tuning&#10;with knowledge distillation. To utilize textual information more effectively,&#10;this work proposes a two-stage textual knowledge distillation method that&#10;matches utterance-level representations and predicted logits of two modalities&#10;during pre-training and fine-tuning, sequentially. We use vq-wav2vec BERT as a&#10;speech encoder because it captures general and rich features. Furthermore, we&#10;improve the performance, especially in a low-resource scenario, with data&#10;augmentation methods by randomly masking spans of discrete audio tokens and&#10;contextualized hidden representations. Consequently, we push the&#10;state-of-the-art on the Fluent Speech Commands, achieving 99.7% test accuracy&#10;in the full dataset setting and 99.5% in the 10% subset setting. Throughout the&#10;ablation studies, we empirically verify that all used methods are crucial to&#10;the final performance, providing the best practice for spoken language&#10;understanding. Code is available at https://github.com/clovaai/textual-kd-slu.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.07467" label="1805.07467">
        <attvalues>
          <attvalue for="0" value="Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces" />
          <attvalue for="1" value="  Recent research has shown that word embedding spaces learned from text&#10;corpora of different languages can be aligned without any parallel data&#10;supervision. Inspired by the success in unsupervised cross-lingual word&#10;embeddings, in this paper we target learning a cross-modal alignment between&#10;the embedding spaces of speech and text learned from corpora of their&#10;respective modalities in an unsupervised fashion. The proposed framework learns&#10;the individual speech and text embedding spaces, and attempts to align the two&#10;spaces via adversarial training, followed by a refinement procedure. We show&#10;how our framework could be used to perform spoken word classification and&#10;translation, and the results on these two tasks demonstrate that the&#10;performance of our unsupervised alignment approach is comparable to its&#10;supervised counterpart. Our framework is especially useful for developing&#10;automatic speech recognition (ASR) and speech-to-text translation systems for&#10;low- or zero-resource languages, which have little parallel audio-text data for&#10;training modern supervised ASR and speech-to-text translation models, but&#10;account for the majority of the languages spoken across the world.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14920" label="2010.14920">
        <attvalues>
          <attvalue for="0" value="Bridging the Modality Gap for Speech-to-Text Translation" />
          <attvalue for="1" value="  End-to-end speech translation aims to translate speech in one language into&#10;text in another language via an end-to-end way. Most existing methods employ an&#10;encoder-decoder structure with a single encoder to learn acoustic&#10;representation and semantic information simultaneously, which ignores the&#10;speech-and-text modality differences and makes the encoder overloaded, leading&#10;to great difficulty in learning such a model. To address these issues, we&#10;propose a Speech-to-Text Adaptation for Speech Translation (STAST) model which&#10;aims to improve the end-to-end model performance by bridging the modality gap&#10;between speech and text. Specifically, we decouple the speech translation&#10;encoder into three parts and introduce a shrink mechanism to match the length&#10;of speech representation with that of the corresponding text transcription. To&#10;obtain better semantic representation, we completely integrate a text-based&#10;translation model into the STAST so that two tasks can be trained in the same&#10;latent space. Furthermore, we introduce a cross-modal adaptation method to&#10;close the distance between speech and text representation. Experimental results&#10;on English-French and English-German speech translation corpora have shown that&#10;our model significantly outperforms strong baselines, and achieves the new&#10;state-of-the-art performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.03320" label="1910.03320">
        <attvalues>
          <attvalue for="0" value="One-To-Many Multilingual End-to-end Speech Translation" />
          <attvalue for="1" value="  Nowadays, training end-to-end neural models for spoken language translation&#10;(SLT) still has to confront with extreme data scarcity conditions. The existing&#10;SLT parallel corpora are indeed orders of magnitude smaller than those&#10;available for the closely related tasks of automatic speech recognition (ASR)&#10;and machine translation (MT), which usually comprise tens of millions of&#10;instances. To cope with data paucity, in this paper we explore the&#10;effectiveness of transfer learning in end-to-end SLT by presenting a&#10;multilingual approach to the task. Multilingual solutions are widely studied in&#10;MT and usually rely on ``\textit{target forcing}'', in which multilingual&#10;parallel data are combined to train a single model by prepending to the input&#10;sequences a language token that specifies the target language. However, when&#10;tested in speech translation, our experiments show that MT-like \textit{target&#10;forcing}, used as is, is not effective in discriminating among the target&#10;languages. Thus, we propose a variant that uses target-language embeddings to&#10;shift the input representations in different portions of the space according to&#10;the language, so to better support the production of output in the desired&#10;target language. Our experiments on end-to-end SLT from English into six&#10;languages show important improvements when translating into similar languages,&#10;especially when these are supported by scarce data. Further improvements are&#10;obtained when using English ASR data as an additional language (up to $+2.5$&#10;BLEU points).&#10;" />
          <attvalue for="2" value="&#10;The state-of-the art results obtained by encoder-decoder models \cite{sutskever2014sequence} with sequence-to-sequence learning in fields like ASR \cite{amodei2016deep,chan2016listen,chiu2017state,zeyer2018improved} and, most importantly, machine translation \cite{vaswani2017attention,bojar2018findings}, have led to the recent proposal of sequence-to-sequence learning for direct speech-to-text translation \cite{berard2016listen,bansal2017towards}, that is translating from audio without an intermediate output representation.&#10;Unfortunately, end-to-end models require large amounts of parallel training data \cite{koehn2017six} that are not yet available for the SLT task. &#10;Indeed, while state-of-the-art sequence-to-sequence systems for ASR and MT are respectively trained on thousands of hours of transcribed speech \cite{Chiu18} and tens of millions of parallel sentences \cite{DBLP:journals/corr/abs-1803-05567}, the largest publicly available SLT corpus comprises&#10;about 500 hours of translated speech and few others amount to less than 300 hours each \cite{mustc19}.&#10;&#10;To overcome this limitation, several works have proposed approaches that exploit in different ways the wealth of ASR or MT data available.&#10;Multitask learning or transfer learning are generally used &#10;to exploit ASR data \cite{weiss2017sequence,bansal2018pre,berard2018end,anastasopoulos2018tied} with positive results,&#10;and the improvements are more evident when less training data for SLT are available.&#10;Other approaches to overcome the low-resource condition are data augmentation \cite{jia2018leveraging} and knowledge distillation \cite{liu2019end}. {However, } \cite{sperber2019attention} showed that current direct models are not data-efficient in leveraging non-SLT data, {and {that the classic} ``cascade'' approach ({i.e. a pipelined architecture integrating ASR and MT) still performs better}}.&#10;&#10;In this paper, we take advantage of the recent release of MuST-C \cite{mustc19}, which provides parallel SLT data for eight languages, in order to study whether multilingual data can be used to train systems with better translation quality than unidirectional (i.e. one-to-one) systems. Multilinguality has been widely explored in neural MT \cite{zoph2016multi,dong2015multi,luong2015multi,firat2016zero,firat2016multi,lu2018neural}, where it is now commonly performed using the target forcing mechanism \cite{hatoward,johnson2017google}, which enables translation to many languages (\{one,many\}-to-many) without changing the underlying NMT architecture. The idea is to prepend the source sentence with a token representing the target language, and all the sentences are processed using the same shared encoder-decoder architecture. Although this approach has been proposed for RNN-based NMT, it works even better \cite{lakew2018comparison} when using the Transformer \cite{vaswani2017attention} architecture. &#10;Target forcing has also been applied to multilingual speech recognition \cite{toshniwal2018multilingual,zhou2018multilingual} showing to improve the transcription quality, although multilingual ASR shows to be better than its monolingual counterparts even when the language token is not provided. &#10;&#10;A single model with shared parameters is particularly appealing in low-resource scenarios \cite{lakew2017multilingual,johnson2017google} as it performs a sort of transfer learning between language directions. &#10;However, compared to one-to-one models, the results of a multilingual model usually degrade in the language directions supported by more training data.&#10;Taking advantage of the MuST-C corpus, in this study we focus on the one-to-many scenario and investigate what groups of target languages favor transfer learning in SLT. To the best of our knowledge, this represents the first study on the effectiveness of the multilingual approach to SLT.&#10;&#10;Along this direction, we proceed incrementally by first showing the limitations of MT-like target forcing and then by proposing and evaluating our SLT-oriented enhancements.&#10;First, our initial experiments show that the target forcing approach as proposed in \cite{johnson2017google} compares poorly with the unidirectional baselines. By looking at the output, we observe that the system produces sentences whose words are coherently in one language, but in many cases the chosen language is wrong. &#10;As the system is not able to learn the co-occurrence between the embedding of the language token and the words in the target language, we then propose to give a stronger learning signal by modifying the input content using the language embedding.&#10;This, in practice, is repeated along the time dimension so to be propagated through the whole input sequence {(rather than being one single vector among thousands of others)}. &#10;&#10;Our experiments show that, by using this variant, translating into similar languages, i.e. Germanic (German and Dutch) and western European Romance (French, Italian, Spanish and Portuguese) leads to better average results than those obtained by unidirectional systems. &#10;However, and to our view unsurprisingly due to the difficulty to transfer knowledge across distant languages, the same improvements are not observed when merging more languages. Indeed, using the six languages together as a target yields improvements only for the lesser-resourced language direction, while combining all the eight languages covered by MuST-C leads to performance degradations in all the cases. &#10;In our final experiments, we also added English to the target languages of all the multilingual systems, which are then trained for translation and ASR.&#10;This provides a slight but consistent improvement to all results. &#10;Overall, in all but the two target languages with more training data, our best multilingual models outperform one-to-one models of comparable size by at least $0.4$ BLEU points. In particular, on the least represented language direction in MuST-C (i.e. English$arrow$Portuguese, for which the corpus includes 385 hours of translated speech), the observed performance improvement over the one-to-one competitor (up to $+2.5$ BLEU points) indicates the feasibility of the proposed approach to operate in low-resource conditions.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.01672" label="2102.01672">
        <attvalues>
          <attvalue for="0" value="The GEM Benchmark: Natural Language Generation, its Evaluation and&#10;  Metrics" />
          <attvalue for="1" value="  We introduce GEM, a living benchmark for natural language Generation (NLG),&#10;its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly&#10;evolving ecosystem of automated metrics, datasets, and human evaluation&#10;standards. Due to this moving target, new models often still evaluate on&#10;divergent anglo-centric corpora with well-established, but flawed, metrics.&#10;This disconnect makes it challenging to identify the limitations of current&#10;models and opportunities for progress. Addressing this limitation, GEM provides&#10;an environment in which models can easily be applied to a wide set of tasks and&#10;in which evaluation strategies can be tested. Regular updates to the benchmark&#10;will help NLG research become more multilingual and evolve the challenge&#10;alongside models. This paper serves as the description of the data for which we&#10;are organizing a shared task at our ACL 2021 Workshop and to which we invite&#10;the entire NLG community to participate.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural language generation is the task to automatically generate understandable texts, typically using a non-linguistic or textual representation of information as input~\cite{reiter2000building}. These texts aim to fulfill an underlying communicative goal (e.g., to produce a summary of an article) while remaining faithful to the input information, fluent, grammatical, and natural-looking. An NLG system needs to be robust to shifts in the data distribution and be able to produce text in many different languages. Finally, it is often desired that repeated interactions with the model produce diverse outputs, for example, to explain concepts in multiple ways or to become a more interesting conversational agent. These optimization objectives can often be conflicting~\cite{hashimoto2019unifying} and, as a result, evaluations that focus only on a single aspect may fail to recognize the drawbacks of a particular method. &#10;To demonstrate this trade-off, consider an improvement on the CNN-DM summarization dataset~\cite{hermann2015teaching,nallapati2016abstractive} measured by the ROUGE-L metric~\cite{lin2004rouge}.&#10;Since ROUGE only tests the extent to which a generated summary has a lexical overlap with a reference summary, it can erroneously produce high scores for fluent, yet meaningless and unfaithful outputs as long as many of the same words are used~\cite{maynez2020faithfulness,gabriel2020go}. Moreover, ROUGE tends to favor systems that produce longer summaries~\cite{sun2019compare}. &#10;It is thus crucial to carefully assess the progress of NLG toward all of its goals at the same time in ways that evolve alongside the models. &#10;This is currently not the case; new models are evaluated on different datasets, most of which focus only on the English language~\cite{bender2019benderrule}, and using these flawed metrics. Moreover, while human evaluations of generated texts can provide complementary insights to automatic evaluation~\cite{manning2020human}, it can also lead to contradicting results since studies often omit crucial replication details and assume different definitions of the measured quantities~\cite{howcroft2020twenty}.&#10;&#10;We propose a living benchmark called \GEM (Generation, Evaluation, and Metrics) that aims to enable research on a wide range of NLG challenges. To avoid the fallacy of encouraging hill climbing on a leaderboard~\cite{linzen2020accelerate}, \GEM focuses on an in-depth evaluation of model outputs across human and automatic evaluation that aims to uncover shortcomings and opportunities for progress. As datasets, metrics, and models improve, the benchmark environment will improve as well, replacing ``solved'' tasks with more challenging ones, incorporating newly developed metrics, and addressing discovered flaws in the experimental setup, as demonstrated in Figure~\ref{fig:circle}.&#10;Making all model outputs available under an open-source license will support evaluation research and integrating new metrics will, in turn, help their adoption and increase the robustness of model evaluations. &#10;&#10;The initial set of eleven included datasets is presented in Table~\ref{tab:overview}. They measure specific generation challenges, such as the content selection and planning (What to say?), and the surface realization (How to say it?)~\cite{reiter2000building,gatt2018survey}. Models need to be capable of paraphrasing, simplification, and others. In addition to those challenges, \GEM datasets also differ in their communicative goals, languages, the noisiness of data, and resource availability, to evaluate the consistency of evaluation schemes.&#10;About half of the datasets have multiple references and more than half were post-processed to improve data quality. The sizes range from 5k to 500k data points. \GEM features 18 languages across all tasks and two of the datasets do not include English at all.&#10;To be able to properly assess the performance of models in a way robust to the shortcuts a model can take, we additionally introduce ten types of challenging test sets that probe for specific modeling aspects~\cite{perez2017analysing,ribeiro-etal-2020-beyond}. &#10;To ensure that research with \GEM is conducted responsibly, all the datasets are documented in an NLG-specific version of data cards~\cite{bender2018data,gebru2018datasheets} we developed and for which we release a template and guide. Moreover, all submitted models will have an associated data card~\cite{mitchell2019model}.&#10;&#10;This paper describes the selection and construction of the \GEM datasets in support of the announcement of the shared task at ACL 2021. More detailed information can be found on our website.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Evaluation Metrics, Multilingual Benchmarking, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1506.03340" label="1506.03340">
        <attvalues>
          <attvalue for="0" value="Teaching Machines to Read and Comprehend" />
          <attvalue for="1" value="  Teaching machines to read natural language documents remains an elusive&#10;challenge. Machine reading systems can be tested on their ability to answer&#10;questions posed on the contents of documents that they have seen, but until now&#10;large scale training and test datasets have been missing for this type of&#10;evaluation. In this work we define a new methodology that resolves this&#10;bottleneck and provides large scale supervised reading comprehension data. This&#10;allows us to develop a class of attention based deep neural networks that learn&#10;to read real documents and answer complex questions with minimal prior&#10;knowledge of language structure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00955" label="2005.00955">
        <attvalues>
          <attvalue for="0" value="How Can We Accelerate Progress Towards Human-like Linguistic&#10;  Generalization?" />
          <attvalue for="1" value="  This position paper describes and critiques the Pretraining-Agnostic&#10;Identically Distributed (PAID) evaluation paradigm, which has become a central&#10;tool for measuring progress in natural language understanding. This paradigm&#10;consists of three stages: (1) pre-training of a word prediction model on a&#10;corpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set&#10;representing a classification task; (3) evaluation on a test set drawn from the&#10;same distribution as that training set. This paradigm favors simple, low-bias&#10;architectures, which, first, can be scaled to process vast amounts of data, and&#10;second, can capture the fine-grained statistical properties of a particular&#10;data set, regardless of whether those properties are likely to generalize to&#10;examples of the task outside the data set. This contrasts with humans, who&#10;learn language from several orders of magnitude less data than the systems&#10;favored by this evaluation paradigm, and generalize to new tasks in a&#10;consistent way. We advocate for supplementing or replacing PAID with paradigms&#10;that reward architectures that generalize as quickly and robustly as humans.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.03802" label="1705.03802">
        <attvalues>
          <attvalue for="0" value="Analysing Data-To-Text Generation Benchmarks" />
          <attvalue for="1" value="  Recently, several data-sets associating data to text have been created to&#10;train data-to-text surface realisers. It is unclear however to what extent the&#10;surface realisation task exercised by these data-sets is linguistically&#10;challenging. Do these data-sets provide enough variety to encourage the&#10;development of generic, high-quality data-to-text surface realisers ? In this&#10;paper, we argue that these data-sets have important drawbacks. We back up our&#10;claim using statistics, metrics and manual evaluation. We conclude by eliciting&#10;a set of criteria for the creation of a data-to-text benchmark which could help&#10;better support the development, evaluation and comparison of linguistically&#10;sophisticated data-to-text surface realisers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.01942" label="2306.01942">
        <attvalues>
          <attvalue for="0" value="Can Contextual Biasing Remain Effective with Whisper and GPT-2?" />
          <attvalue for="1" value="  End-to-end automatic speech recognition (ASR) and large language models, such&#10;as Whisper and GPT-2, have recently been scaled to use vast amounts of training&#10;data. Despite the large amount of training data, infrequent content words that&#10;occur in a particular task may still exhibit poor ASR performance, with&#10;contextual biasing a possible remedy. This paper investigates the effectiveness&#10;of neural contextual biasing for Whisper combined with GPT-2. Specifically,&#10;this paper proposes integrating an adapted tree-constrained pointer generator&#10;(TCPGen) component for Whisper and a dedicated training scheme to dynamically&#10;adjust the final output without modifying any Whisper model parameters.&#10;Experiments across three datasets show a considerable reduction in errors on&#10;biasing words with a biasing list of 1000 words. Contextual biasing was more&#10;effective when applied to domain-specific data and can boost the performance of&#10;Whisper and GPT-2 without losing their generality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Automatic Speech Recognition, Artificial Intelligence, Contextual Biasing Techniques, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2404.05971" label="2404.05971">
        <attvalues>
          <attvalue for="0" value="Does Transformer Interpretability Transfer to RNNs?" />
          <attvalue for="1" value="  Recent advances in recurrent neural network architectures, such as Mamba and&#10;RWKV, have enabled RNNs to match or exceed the performance of equal-size&#10;transformers in terms of language modeling perplexity and downstream&#10;evaluations, suggesting that future systems may be built on completely new&#10;architectures. In this paper, we examine if selected interpretability methods&#10;originally designed for transformer language models will transfer to these&#10;up-and-coming recurrent architectures. Specifically, we focus on steering model&#10;outputs via contrastive activation addition, on eliciting latent predictions&#10;via the tuned lens, and eliciting latent knowledge from models fine-tuned to&#10;produce false outputs under certain conditions. Our results show that most of&#10;these techniques are effective when applied to RNNs, and we show that it is&#10;possible to improve some of them by taking advantage of RNNs' compressed state.&#10;" />
          <attvalue for="2" value="&#10;&#10;The transformer architecture \cite{vaswani2017attention} has all but replaced the recurrent neural network (RNN) in natural language processing in recent years due to its impressive ability to handle long-distance dependencies and its parallelizable training across the time dimension. But the self-attention mechanism at the heart of the transformer suffers from quadratic time complexity, making it computationally expensive to apply to very long sequences.&#10;&#10;Mamba \cite{gu2023mamba} and RWKV \cite{peng2023rwkv} are RNNs that allow for parallelized training across the time dimension by restricting the underlying recurrence relation to be associative \cite{martin2017parallelizing, blelloch1990prefix}. Empirically, these architectures exhibit comparable perplexity and downstream performance to equal-size transformers, making them attractive alternatives for many use-cases.&#10;&#10;In this paper, we assess whether popular interpretability tools originally designed for the transformer will also apply to these new RNN models. In particular, we reproduce the following findings from the transformer interpretability literature:&#10;\begin{enumerate}&#10; \item Contrastive activation addition (CAA): \cite{rimsky2023steering} find that transformer LMs can be controlled using ``steering vectors,'' computed by averaging the difference in residual stream activations between pairs of positive and negative examples of a particular behavior, such as factual versus hallucinatory responses.&#10; \item The tuned lens: \cite{belrose2023eliciting} find that interpretable next-token predictions can be elicited from intermediate layers of a transformer using linear probes, and that the accuracy of these predictions increases monotonically with depth.&#10; \item ``Quirky'' models: \cite{mallen2023eliciting} find that simple probing methods can elicit a transformer's knowledge of the correct answer to a question, even when it has been fine-tuned to output an incorrect answer. They further find that these probes generalize to problems harder than those the probe was trained on.&#10;\end{enumerate}&#10;&#10;We also introduce state steering, a modification of CAA that operates on an RNN's compressed state, rather than on its residual stream.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Recurrent Neural Networks, Deep Learning Architectures, Mathematics, Language Model Interpretability" />
        </attvalues>
      </node>
      <node id="1709.04057" label="1709.04057">
        <attvalues>
          <attvalue for="0" value="Parallelizing Linear Recurrent Neural Nets Over Sequence Length" />
          <attvalue for="1" value="  Recurrent neural networks (RNNs) are widely used to model sequential data but&#10;their non-linear dependencies between sequence elements prevent parallelizing&#10;training over sequence length. We show the training of RNNs with only linear&#10;sequential dependencies can be parallelized over the sequence length using the&#10;parallel scan algorithm, leading to rapid training on long sequences even with&#10;small minibatch size. We develop a parallel linear recurrence CUDA kernel and&#10;show that it can be applied to immediately speed up training and inference of&#10;several state of the art RNN architectures by up to 9x. We abstract recent work&#10;on linear RNNs into a new framework of linear surrogate RNNs and develop a&#10;linear surrogate model for the long short-term memory unit, the GILR-LSTM, that&#10;utilizes parallel linear recurrence. We extend sequence learning to new&#10;extremely long sequence regimes that were previously out of reach by&#10;successfully training a GILR-LSTM on a synthetic sequence classification task&#10;with a one million timestep dependency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.06681" label="2312.06681">
        <attvalues>
          <attvalue for="0" value="Steering Llama 2 via Contrastive Activation Addition" />
          <attvalue for="1" value="  We introduce Contrastive Activation Addition (CAA), an innovative method for&#10;steering language models by modifying their activations during forward passes.&#10;CAA computes &quot;steering vectors&quot; by averaging the difference in residual stream&#10;activations between pairs of positive and negative examples of a particular&#10;behavior, such as factual versus hallucinatory responses. During inference,&#10;these steering vectors are added at all token positions after the user's prompt&#10;with either a positive or negative coefficient, allowing precise control over&#10;the degree of the targeted behavior. We evaluate CAA's effectiveness on Llama 2&#10;Chat using multiple-choice behavioral question datasets and open-ended&#10;generation tasks. We demonstrate that CAA significantly alters model behavior,&#10;is effective over and on top of traditional methods like finetuning and system&#10;prompt design, and minimally reduces capabilities. Moreover, we gain deeper&#10;insights into CAA's mechanisms by employing various activation space&#10;interpretation methods. CAA accurately steers model outputs and sheds light on&#10;how high-level concepts are represented in Large Language Models (LLMs).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.08112" label="2303.08112">
        <attvalues>
          <attvalue for="0" value="Eliciting Latent Predictions from Transformers with the Tuned Lens" />
          <attvalue for="1" value="  We analyze transformers from the perspective of iterative inference, seeking&#10;to understand how model predictions are refined layer by layer. To do so, we&#10;train an affine probe for each block in a frozen pretrained model, making it&#10;possible to decode every hidden state into a distribution over the vocabulary.&#10;Our method, the \emph{tuned lens}, is a refinement of the earlier ``logit&#10;lens'' technique, which yielded useful insights but is often brittle.&#10;  We test our method on various autoregressive language models with up to 20B&#10;parameters, showing it to be more predictive, reliable and unbiased than the&#10;logit lens. With causal experiments, we show the tuned lens uses similar&#10;features to the model itself. We also find the trajectory of latent predictions&#10;can be used to detect malicious inputs with high accuracy. All code needed to&#10;reproduce our results can be found at&#10;https://github.com/AlignmentResearch/tuned-lens.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.01037" label="2312.01037">
        <attvalues>
          <attvalue for="0" value="Eliciting Latent Knowledge from Quirky Language Models" />
          <attvalue for="1" value="  Eliciting Latent Knowledge (ELK) aims to find patterns in a capable neural&#10;network's activations that robustly track the true state of the world,&#10;especially in hard-to-verify cases where the model's output is untrusted. To&#10;further ELK research, we introduce 12 datasets and a corresponding suite of&#10;&quot;quirky&quot; language models (LMs) that are finetuned to make systematic errors&#10;when answering questions if and only if the keyword &quot;Bob&quot; is present in the&#10;prompt. We find that, especially in middle layers, linear probes usually report&#10;an LM's knowledge independently of what the LM outputs, enabling us to elicit&#10;the correct answer despite the model's untruthful output. The best probing&#10;method (logistic regression on contrast pairs) recovers 89% of the gap in AUROC&#10;between truthful and untruthful contexts, and 75% for questions harder than&#10;those used to train the probe. We also find that a mechanistic anomaly&#10;detection approach can flag untruthful behavior with 0.95 AUROC. Our results&#10;show promise for eliciting reliable knowledge from capable but untrusted&#10;models, and facilitates future research empirically investigating ELK methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04778" label="2109.04778">
        <attvalues>
          <attvalue for="0" value="Improving Multilingual Translation by Representation and Gradient&#10;  Regularization" />
          <attvalue for="1" value="  Multilingual Neural Machine Translation (NMT) enables one model to serve all&#10;translation directions, including ones that are unseen during training, i.e.&#10;zero-shot translation. Despite being theoretically attractive, current models&#10;often produce low quality translations -- commonly failing to even produce&#10;outputs in the right target language. In this work, we observe that off-target&#10;translation is dominant even in strong multilingual systems, trained on massive&#10;multilingual corpora. To address this issue, we propose a joint approach to&#10;regularize NMT models at both representation-level and gradient-level. At the&#10;representation level, we leverage an auxiliary target language prediction task&#10;to regularize decoder outputs to retain information about the target language.&#10;At the gradient level, we leverage a small amount of direct data (in thousands&#10;of sentence pairs) to regularize model gradients. Our results demonstrate that&#10;our approach is highly effective in both reducing off-target translation&#10;occurrences and improving zero-shot translation performance by +5.59 and +10.38&#10;BLEU on WMT and OPUS datasets respectively. Moreover, experiments show that our&#10;method also works well when the small amount of direct data is not available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Multilingual NMT&#10;Multilingual NMT aims to train one model to serve all language pairs~\cite{ha2016toward,firat2016multi,johnson2017google}.&#10;Several subsequent works explored various parameter sharing strategies to mitigate the representation bottleneck~\cite{blackwood2018multilingual,platanios2018contextual,sachan2018parameter,sen2019multilingual}.&#10;Meanwhile, there are also notorious cases of off-target translation especially in English-Free pairs.&#10;Previous works either resort to back-translation techniques to generate synthetic English-Free data~\cite{gu2019improved,zhang2020improving}, or to model-level changes to the encoder-decoder alignments~\cite{arivazhagan2019massively,liu2020improving}.&#10;In contrast, we propose a joint representation and gradient regularization approach to reduce off-target translations and significantly improve performance across all language pairs.&#10;&#10;Multi-Task Learning for NMT&#10;Multi-task learning (MTL) is a widely used technique to share model parameters and improve generalization~\cite{ruder2017overview}.&#10;For NMT, previous works have leveraged MTL to inject linguistic knowledge or leveraged monolingual data~\cite{eriguchi2017learning,niehues2017exploiting,kiperwasser2018scheduled,wang2020multi}.&#10;Our work leverages an auxiliary TLP loss to help learn more separable model states for different target languages.&#10;&#10;Optimization Learning&#10;Previous works have studied the optimization challenges in multi-task training~\cite{hessel2019multi,schaul2019ray}, where \cite{yu2020gradient} proposed to resolve gradient conflicts between different tasks.&#10;Meanwhile for NMT, ~\cite{wang2021gradient,wang2020balancing} proposed to mask out or assign different weights to training samples based on the gradient alignments with validation set.&#10;\cite{yu2020gradient} proposed to resolve the pair-wise gradient conflicts between translation directions.&#10;In contrast, we propose TGP to guide the training process by projecting training gradients according to the oracle gradients.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Machine Learning, Language Model Regularization, Artificial Intelligence, Multilingual Translation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.15925" label="2010.15925">
        <attvalues>
          <attvalue for="0" value="RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark" />
          <attvalue for="1" value="  In this paper, we introduce an advanced Russian general language&#10;understanding evaluation benchmark -- RussianGLUE. Recent advances in the field&#10;of universal language models and transformers require the development of a&#10;methodology for their broad diagnostics and testing for general intellectual&#10;skills - detection of natural language inference, commonsense reasoning,&#10;ability to perform simple logical operations regardless of text subject or&#10;lexicon. For the first time, a benchmark of nine tasks, collected and organized&#10;analogically to the SuperGLUE methodology, was developed from scratch for the&#10;Russian language. We provide baselines, human level evaluation, an open-source&#10;framework for evaluating models&#10;(https://github.com/RussianNLP/RussianSuperGLUE), and an overall leaderboard of&#10;transformer models for the Russian language. Besides, we present the first&#10;results of comparing multilingual models in the adapted diagnostic test set and&#10;offer the first steps to further expanding or assessing state-of-the-art models&#10;independently of language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;Several benchmarks have been developed to evaluate and analyze word and sentence embeddings over the past few years. &#10;&#10;SentEval \cite{conneau2018senteval} is one of the first frameworks intended to evaluate the quality of sentence embeddings. A twofold set of transfer tasks is used to assess the generalization power of sentence embedding models. The transfer tasks comprise downstream tasks, in which the sentence embedding is used as a feature vector, and probing tasks, which are aimed to evaluate the capability of sentence embeddings to encode linguistic properties. The choice of the downstream tasks is limited to sentiment classification, natural language inference, paraphrase detection and image captioning tasks. The probing tasks are meant to analyse morphological, syntactical and semantical information encoded in sentence embeddings. &#10;&#10;The General Language Understanding Evaluation (GLUE) \cite{wang2018glue} benchmark is a collection of tools for evaluating the performance of language models across a diverse set of existing natural language understanding (NLU) tasks, adopted from different sources. These tasks are divided into two parts: single sentence classification tasks and sentence pair classifications tasks subdivided further into similarity and inference tasks. GLUE also includes a hand-crafted diagnostic test, which probes for complex linguistic phenomena, such as the ability of the model to express lexical semantics and predicate-argument structure, to pose logical apparatus and knowledge representation. GLUE is recognized as a de-facto standard benchmark to evaluate transformer-derived language models. Last but not least GLUE informs on human baselines for the tasks, so that not only submitted models are compared to the baseline, but also to the human performance. The SuperGLUE \cite{wang2019superglue} follows GLUE paradigm for language model evaluation based on NLU tasks, providing with more complex tasks, of which some require reasoning capabilities and some are aimed at detecting ethical biases. A few recent projects reveal that GLUE tasks may be not sophisticated enough and do not require much tasks-specific linguistic knowledge \cite{kovaleva2019revealing,warstadt2019investigating}. Thus SuperGLUE benchmark, being more challenging, becomes much more preferable for evaluation of language models. &#10;&#10;decaNLP \cite{McCann2018decaNLP} widens the scope for language model evaluation by introducing ten disparate natural language tasks. These tasks comprise not only text classification problems, but sequence tagging and sequence transformation problems. The latter include machine translation and text summarization, while the former include semantic parsing and semantic role labelling. Although decaNLP along with the associated research direction focuses on multi-task learning as a form of question answering, it supports zero-shot evaluation. &#10;&#10;To evaluate models for languages other than English, several monolingual benchmarks were developed, such as FLUE \cite{le2019flaubert} and CLUE \cite{CLUE}, being French and Chinese versions of GLUE. These benchmarks include a variety of tasks, ranging from part-of-speech tagging and syntax parsing to machine reading comprehension and natural language inference. &#10;&#10;To the best of our knowledge, LINSPECTOR \cite{eichler-etal-2019-linspector} is a first multi-lingual benchmark for evaluating the performance of language models. LINSPECTOR offers 22 probing tasks to analyse for a single linguistic feature such as case marking, gender, person, or tense for 52 languages. A part of these 22 probing tasks are static, i.e. are aimed at evaluation of word embeddings, and the rest are contextual and should be used to evaluate language models. Released in early 2020 two multilingual benchmarks, \cite{liang2020xglue} and XTREME \cite{hu2020xtreme}, aim at evaluation of cross-lingual models. &#10;XGLUE includes 11 tasks, which cover both language understanding and language generation problems, for 19 languages. XGLUE provides with several multilingual and bilingual corpora that allow of cross-lingual model training. As for the Russian language, XGLUE provides with four datasets for POS tagging, a part of XNLI \cite{conneau2018xnli} and two datasets, crawled from commercial news website, used for news classification and news headline generation. XTREME consists of nine tasks which cover classification, sequence labelling, question answering and retrieval problems for 40 languages. Almost a half of the datasets were translated from English to the target languages with the help of professional translators. XTREME offers for the Russian language five datasets, including NER and two question-answering datasets. Both XGLUE and XTREME offer tasks that are much simpler than SuperGLUE and are aimed at evaluation of cross-lingual models rather than at comparison of mono-lingual models in similar setups. Thus the need for novel datasets targeted at mono-lingual model evaluation for languages other than English is still not eliminated. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2110.10319" label="2110.10319">
        <attvalues>
          <attvalue for="0" value="LMSOC: An Approach for Socially Sensitive Pretraining" />
          <attvalue for="1" value="  While large-scale pretrained language models have been shown to learn&#10;effective linguistic representations for many NLP tasks, there remain many&#10;real-world contextual aspects of language that current approaches do not&#10;capture. For instance, consider a cloze-test &quot;I enjoyed the ____ game this&#10;weekend&quot;: the correct answer depends heavily on where the speaker is from, when&#10;the utterance occurred, and the speaker's broader social milieu and&#10;preferences. Although language depends heavily on the geographical, temporal,&#10;and other social contexts of the speaker, these elements have not been&#10;incorporated into modern transformer-based language models. We propose a simple&#10;but effective approach to incorporate speaker social context into the learned&#10;representations of large-scale language models. Our method first learns dense&#10;representations of social contexts using graph representation learning&#10;algorithms and then primes language model pretraining with these social context&#10;representations. We evaluate our approach on geographically-sensitive&#10;language-modeling tasks and show a substantial improvement (more than 100%&#10;relative lift on MRR) compared to baselines.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models are at the very heart of many modern NLP systems and applications \cite{young2018recent}. Representations derived from large-scale language models are used widely in many downstream NLP models \cite{peters2018deep, devlin-etal-2019-bert}. However, an implicit assumption made in most modern NLP systems (including language models) is that language is independent of extra-linguistic context such as speaker/author identity and their social setting. While this simplifying assumption has undoubtedly encouraged remarkable progress in modeling language, there is overwhelming evidence in socio-linguistics that language understanding is influenced by the social context in which language is grounded \cite{nguyen2016computational,hovy2018social, mishra2018detecting, garten2019incorporating, flek-2020-returning, bender-koller-2020-climbing}. In fact, language use on social media where every utterance is grounded in a specific social context (like time, geography, social groups, communities) reinforces this often ignored aspect of language. When NLP applications ignore this social context, they may perform sub-optimally underscoring the need for a richer integration of social contexts into NLP models \cite{pavalanathan2015confounds,lynn2017human, zamani-etal-2018-residualized, lynn2019tweet, may-etal-2019-measuring,kurita-etal-2019-measuring, welch-etal-2020-compositional, hovy2021importance}. &#10;&#10;Prior attempts to better leverage the social context surrounding language while learning language representations have mostly focused on learning social context dependent word embeddings and have been primarily used to characterize language variation across many dimensions (time, geography, and demographics). These methods learn word embeddings for each specific social context and can capture how word meanings vary across these dimensions \cite{bamman2014distributed,kulkarni2015statistically, hamilton2016diachronic,welch-etal-2020-compositional,welch-etal-2020-exploring}. However, word embedding based approaches in general suffer from two fundamental limitations: (a) word embeddings are not linguistically contextualized as noted by \cite{peters2018deep} (b) word embedding learning is transductive -- they can only generate embeddings for words observed during training and usually assume a finite word vocabulary and a set of social contexts all of which need to be seen during training. Recent approaches have addressed the first limitation by learning word representations that are contextualized by their token-specific usage context \cite{peters2018deep,devlin-etal-2019-bert,liu2019roberta,yang2019xlnet,yang2019context}. The second limitation has been addressed by WordPiece tokenization methods \cite{6289079, devlin-etal-2019-bert, liu2019roberta}. While these approaches have successfully captured linguistic context, they still do not capture social context in language representations.&#10;``How can we learn linguistically contextualized and socially contextualized language representations?'' is the question we seek to answer in this paper.&#10;&#10;We propose Lmsoc\ to (a) learn representations of tokens that are both linguistically contextualized and socially sensitive and (b) enable the language model to inductively generate representations for language grounded in social contexts it has never observed during the language model pre-training process. As an example, our model can enable NLP systems to associate the right entity being referred to based on the broader user/social context in which an utterance like ``Our Prime Minister visited the UK last week.'' is grounded. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Social Context Representation, Geographically-Sensitive NLP, Computer Science, Linguistics, Language Model Limitations, Artificial Intelligence, Social Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1708.02709" label="1708.02709">
        <attvalues>
          <attvalue for="0" value="Recent Trends in Deep Learning Based Natural Language Processing" />
          <attvalue for="1" value="  Deep learning methods employ multiple processing layers to learn hierarchical&#10;representations of data and have produced state-of-the-art results in many&#10;domains. Recently, a variety of model designs and methods have blossomed in the&#10;context of natural language processing (NLP). In this paper, we review&#10;significant deep learning related models and methods that have been employed&#10;for numerous NLP tasks and provide a walk-through of their evolution. We also&#10;summarize, compare and contrast the various models and put forward a detailed&#10;understanding of the past, present and future of deep learning in NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1508.07544" label="1508.07544">
        <attvalues>
          <attvalue for="0" value="Computational Sociolinguistics: A Survey" />
          <attvalue for="1" value="  Language is a social phenomenon and variation is inherent to its social&#10;nature. Recently, there has been a surge of interest within the computational&#10;linguistics (CL) community in the social dimension of language. In this article&#10;we present a survey of the emerging field of &quot;Computational Sociolinguistics&quot;&#10;that reflects this increased interest. We aim to provide a comprehensive&#10;overview of CL research on sociolinguistic themes, featuring topics such as the&#10;relation between language and social identity, language use in social&#10;interaction and multilingual communication. Moreover, we demonstrate the&#10;potential for synergy between the research communities involved, by showing how&#10;the large-scale data-driven methods that are widely used in CL can complement&#10;existing sociolinguistic studies, and how sociolinguistics can inform and&#10;challenge the methods and assumptions employed in CL studies. We hope to convey&#10;the possible benefits of a closer collaboration between the two communities and&#10;conclude with a discussion of open challenges.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1411.3315" label="1411.3315">
        <attvalues>
          <attvalue for="0" value="Statistically Significant Detection of Linguistic Change" />
          <attvalue for="1" value="  We propose a new computational approach for tracking and detecting&#10;statistically significant linguistic shifts in the meaning and usage of words.&#10;Such linguistic shifts are especially prevalent on the Internet, where the&#10;rapid exchange of ideas can quickly change a word's meaning. Our meta-analysis&#10;approach constructs property time series of word usage, and then uses&#10;statistically sound change point detection algorithms to identify significant&#10;linguistic shifts.&#10;  We consider and analyze three approaches of increasing complexity to generate&#10;such linguistic property time series, the culmination of which uses&#10;distributional characteristics inferred from word co-occurrences. Using&#10;recently proposed deep neural language models, we first train vector&#10;representations of words for each time period. Second, we warp the vector&#10;spaces into one unified coordinate system. Finally, we construct a&#10;distance-based distributional time series for each word to track it's&#10;linguistic displacement over time.&#10;  We demonstrate that our approach is scalable by tracking linguistic change&#10;across years of micro-blogging using Twitter, a decade of product reviews using&#10;a corpus of movie reviews from Amazon, and a century of written books using the&#10;Google Book-ngrams. Our analysis reveals interesting patterns of language usage&#10;change commensurate with each medium.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.02986" label="2010.02986">
        <attvalues>
          <attvalue for="0" value="Compositional Demographic Word Embeddings" />
          <attvalue for="1" value="  Word embeddings are usually derived from corpora containing text from many&#10;individuals, thus leading to general purpose representations rather than&#10;individually personalized representations. While personalized embeddings can be&#10;useful to improve language model performance and other language processing&#10;tasks, they can only be computed for people with a large amount of longitudinal&#10;data, which is not the case for new users. We propose a new form of&#10;personalized word embeddings that use demographic-specific word representations&#10;derived compositionally from full or partial demographic information for a user&#10;(i.e., gender, age, location, religion). We show that the resulting&#10;demographic-aware word representations outperform generic word representations&#10;on two tasks for English: language modeling and word associations. We further&#10;explore the trade-off between the number of available attributes and their&#10;relative effectiveness and discuss the ethical implications of using them.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Embedding Bias.&#10;Recent work on embeddings has revealed and attempted to remove racial, gender, religious, and other biases~\cite{manzini-etal-2019-black,bolukbasi2016man}.&#10;The bias in our corpora and embeddings have a societal impact and risks exclusion and demographic misrepresentation~\cite{hovy-spruit-2016-social}. This means that users of certain regions, ages, or genders may find NLP technologies more difficult to use. For instance, when using standard corpora for POS tagging, \cite{hovy-sogaard-2015-tagging} found that models perform significantly lower on younger people and ethnic minorities. Similarly, results on text-based geotagging show best results for men over 40~\cite{pavalanathan-eisenstein-2015-confounds}.&#10;&#10;Similar results are starting to be found in embeddings produced by contextual embedding methods \cite{may-etal-2019-measuring,kurita2019measuring}. We focus on non-contextual embedding methods because of their computational efficiency, which is crucial if many separate representations are being learned. Additionally, there may not be a large amount of available data for underrepresented groups and these contextualized models require billions of tokens for training. Recent work has also shown that static embeddings are competitive with contextualized ones in some settings~\cite{arora2020contextual}.&#10;&#10;Personalization.&#10;The closest work is \cite{garimella-etal-2017-demographic}'s exploration of demographic-specific word embedding spaces.&#10;They trained word embeddings for male and female speakers who live in the USA and India using skip-gram architectures that learn a separate word matrix for each demographic group (e.g., male speakers from the USA).&#10;&#10;Another line of work used discrete~\cite{hovy-2015-demographic} or continuous values~\cite{lynn-etal-2017-human} to learn speaker embeddings: a single vector for each user.&#10;The speaker embedding is appended to the input of the recurrent or output layer, and trained simultaneously with the rest of the model.&#10;This idea applies to any contextual information type and was introduced as a way to condition language models on topics learned by topic modeling \cite{mikolov2012context}.&#10;It has since been used as a way of representing users in tasks such as&#10;task-oriented and open-domain dialog \cite{wen2013recurrent,li-etal-2016-persona},&#10;information retrieval based on book preferences \cite{amer2016toward},&#10;query auto-completion \cite{jaech-ostendorf-2018-personalized},&#10;authorship attribution \cite{ebrahimi2016personalized},&#10;sarcasm detection \cite{kolchinski-potts-2018-representing},&#10;sentiment analysis \cite{zeng2017socialized},&#10;and cold-start language modeling \cite{huang2016enriching}.&#10;Finally, a recent study by \cite{king-cook-2020-evaluating} compared how to improve a language model with user-specific data using priming and interpolation, depending on the amount of data available, learning a new model for each user.&#10;&#10;More generally, personalization has been extensively applied to marketing, webpage layout, product and news recommendation, query completion, and dialog~\cite{eirinaki2003web,das2007google}.&#10;\cite{Welch19LearningFromPersonal,Welch19LookWhosTalking} explored predicting response time, common messages, and speaker relationships from personal conversation data.&#10;\cite{zhang2018personalizing} conditioned dialog systems on artificially constructed personas and \cite{madotto-etal-2019-personalizing} used meta-learning to improve this process.&#10;Goal-oriented dialog has used demographics (i.e. age, gender) to condition system response generation, showing that this relatively coarse grained personalization improves system performance \cite{joshi2017personalization}.&#10;&#10;Social Media.&#10;We use social media data with demographic attributes inferred from user posts.&#10;Prior work has explored extraction or prediction of attributes such as age, gender, region, and political orientation~\cite{rao2010classifying,rangel2013overview}.&#10;Work on analyzing the demographics of social media users also includes race/ethnicity, income level, urbanity, emotional stability, personality traits \cite{mccrae1997personality}, and life satisfaction \cite{duggan2013demographics,correa2010interacts}.&#10;&#10;One particularly relevant study by \cite{gjurkovic-snajder-2018-reddit} presented a corpus of Reddit users with personality information as well as some demographics for a subset of users.&#10;Unlike our approach, which is based on text content, they extract information from Reddit flairs, a type of user tag.&#10;Out of their set of 10,295 users, 2,253 are also in our set of users (22\% of theirs, 0.5\% of ours) that have one or more demographic labels, confirming the speculation in their paper that extracting demographics from text is a complementary approach that captures more information about users in their data.&#10;Other work has used Reddit posts to identify users who were diagnosed with depression \cite{yates-etal-2017-depression} and to construct personas for personalized dialog agents \cite{mazare-etal-2018-training}.&#10;&#10;Language Models.&#10;To evaluate embeddings, we consider language modeling, a task that has long been used for speech recognition and translation, and more recently been widely used for model pretraining.&#10;A range of models have been developed, with progressively larger models trained on more data~\cite[e.g.,][]{dai-etal-2019-transformer}.&#10;Variations of the LSTM have consistently achieved state-of-the-art performance without massive compute resources&#10;\cite{merity2018analysis,melis2019mogrifier,merity2019single,li-etal-2020-learning}.&#10;We use the AWD-LSTM~\cite{merity2018regularizing} in our experiments, as it achieves very strong performance, has a well-studied codebase, and can be trained on a single GPU in a day.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1902.05766" label="1902.05766">
        <attvalues>
          <attvalue for="0" value="Context-Aware Self-Attention Networks" />
          <attvalue for="1" value="  Self-attention model have shown its flexibility in parallel computation and&#10;the effectiveness on modeling both long- and short-term dependencies. However,&#10;it calculates the dependencies between representations without considering the&#10;contextual information, which have proven useful for modeling dependencies&#10;among neural representations in various natural language tasks. In this work,&#10;we focus on improving self-attention networks through capturing the richness of&#10;context. To maintain the simplicity and flexibility of the self-attention&#10;networks, we propose to contextualize the transformations of the query and key&#10;layers, which are used to calculates the relevance between elements.&#10;Specifically, we leverage the internal representations that embed both global&#10;and deep contexts, thus avoid relying on external resources. Experimental&#10;results on WMT14 English-German and WMT17 Chinese-English translation tasks&#10;demonstrate the effectiveness and universality of the proposed methods.&#10;Furthermore, we conducted extensive analyses to quantity how the context&#10;vectors participate in the self-attention model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.15166" label="2312.15166">
        <attvalues>
          <attvalue for="0" value="SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective&#10;  Depth Up-Scaling" />
          <attvalue for="1" value="  We introduce SOLAR 10.7B, a large language model (LLM) with 10.7 billion&#10;parameters, demonstrating superior performance in various natural language&#10;processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale&#10;LLMs, we present a method for scaling LLMs called depth up-scaling (DUS), which&#10;encompasses depthwise scaling and continued pretraining. In contrast to other&#10;LLM up-scaling methods that use mixture-of-experts, DUS does not require&#10;complex changes to train and inference efficiently. We show experimentally that&#10;DUS is simple yet effective in scaling up high-performance LLMs from small&#10;ones. Building on the DUS model, we additionally present SOLAR 10.7B-Instruct,&#10;a variant fine-tuned for instruction-following capabilities, surpassing&#10;Mixtral-8x7B-Instruct. SOLAR 10.7B is publicly available under the Apache 2.0&#10;license, promoting broad access and application in the LLM field.&#10;" />
          <attvalue for="2" value="&#10;&#10;The field of natural language processing (NLP) has been significantly transformed by the introduction of large language models (LLMs), which have enhanced our understanding and interaction with human language~\cite{zhao2023survey}. These advancements bring challenges such as the increased need to train ever larger models~\cite{rae2021scaling,wang2023learning,pan2023reusing,winglian,yao20232x,gesmundo2023composable} owing to the performance scaling law~\cite{kaplan2020scaling, hernandez2021scaling, anil2023palm,kaddour2023no}. To efficiently tackle the above, recent works in scaling language models such as a mixture of experts (MoE)~\cite{shazeer2017outrageously,komatsuzaki2022sparse} have been proposed. While those approaches are able to efficiently and effectively scale-up LLMs, they often require non-trivial changes to the training and inference framework~\cite{gale2023megablocks}, which hinders widespread applicability. Effectively and efficiently scaling up LLMs whilst also retaining the simplicity for ease of use is an important problem~\cite{alberts2023large,fraiwan2023review,sallam2023chatgpt,bahrini2023chatgpt}.&#10;&#10;Inspired by ~\cite{komatsuzaki2022sparse}, we present depth up-scaling (DUS), an effective and efficient method to up-scale LLMs whilst also remaining straightforward to use.&#10;DUS consists of scaling the number of layers in the base model and continually pretraining the scaled model. Unlike \cite{komatsuzaki2022sparse}, DUS does not scale the model using MoE and rather use a depthwise scaling method analogous to ~\cite{tan2019efficientnet} which is adapted for the LLM architecture. Thus, there are no additional modules or dynamism as with MoE, making DUS immediately compatible with easy-to-use LLM frameworks such as HuggingFace~\cite{wolf2019huggingface} with no changes to the training or inference framework for maximal efficiency. Furthermore, DUS is applicable to all transformer architectures, opening up new gateways to effectively and efficiently scale-up LLMs in a simple manner. Using DUS, we release \pt, an LLM with 10.7 billion parameters, that outperforms existing models like Llama 2~\cite{touvron2023llama2} and Mistral 7B~\cite{jiang2023mistral} in various benchmarks.&#10;&#10;We have also developed \ft, a variant fine-tuned for tasks requiring strict adherence to complex instructions. It significantly outperforms the Mixtral-8x7B-Instruct model across various evaluation metrics, evidencing an advanced proficiency that exceeds the capabilities of even larger models in terms of benchmark performance.&#10;&#10;By releasing \pt under the Apache 2.0 license, we aim to promote collaboration and innovation in NLP. This open-source approach allows for wider access and application of these models by researchers and developers globally.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Deep Learning Scaling, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.10699" label="2310.10699">
        <attvalues>
          <attvalue for="0" value="Reusing Pretrained Models by Multi-linear Operators for Efficient&#10;  Training" />
          <attvalue for="1" value="  Training large models from scratch usually costs a substantial amount of&#10;resources. Towards this problem, recent studies such as bert2BERT and LiGO have&#10;reused small pretrained models to initialize a large model (termed the ``target&#10;model''), leading to a considerable acceleration in training. Despite the&#10;successes of these previous studies, they grew pretrained models by mapping&#10;partial weights only, ignoring potential correlations across the entire model.&#10;As we show in this paper, there are inter- and intra-interactions among the&#10;weights of both the pretrained and the target models. As a result, the partial&#10;mapping may not capture the complete information and lead to inadequate growth.&#10;In this paper, we propose a method that linearly correlates each weight of the&#10;target model to all the weights of the pretrained model to further enhance&#10;acceleration ability. We utilize multi-linear operators to reduce computational&#10;and spacial complexity, enabling acceptable resource requirements. Experiments&#10;demonstrate that our method can save 76\% computational costs on DeiT-base&#10;transferred from DeiT-small, which outperforms bert2BERT by +12.0\% and LiGO by&#10;+20.7\%, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.06103" label="2308.06103">
        <attvalues>
          <attvalue for="0" value="Composable Function-preserving Expansions for Transformer Architectures" />
          <attvalue for="1" value="  Training state-of-the-art neural networks requires a high cost in terms of&#10;compute and time. Model scale is recognized to be a critical factor to achieve&#10;and improve the state-of-the-art. Increasing the scale of a neural network&#10;normally requires restarting from scratch by randomly initializing all the&#10;parameters of the model, as this implies a change of architecture's parameters&#10;that does not allow for a straightforward transfer of knowledge from smaller&#10;size models. In this work, we propose six composable transformations to&#10;incrementally increase the size of transformer-based neural networks while&#10;preserving functionality, allowing to expand the capacity of the model as&#10;needed. We provide proof of exact function preservation under minimal&#10;initialization constraints for each transformation. The proposed methods may&#10;enable efficient training pipelines for larger and more powerful models by&#10;progressively expanding the architecture throughout training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.15841" label="2211.15841">
        <attvalues>
          <attvalue for="0" value="MegaBlocks: Efficient Sparse Training with Mixture-of-Experts" />
          <attvalue for="1" value="  We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE)&#10;training on GPUs. Our system is motivated by the limitations of current&#10;frameworks, which restrict the dynamic routing in MoE layers to satisfy the&#10;constraints of existing software and hardware. These formulations force a&#10;tradeoff between model quality and hardware efficiency, as users must choose&#10;between dropping tokens from the computation or wasting computation and memory&#10;on padding. To address these limitations, we reformulate MoE computation in&#10;terms of block-sparse operations and develop new block-sparse GPU kernels that&#10;efficiently handle the dynamism present in MoEs. Our approach never drops&#10;tokens and maps efficiently to modern hardware, enabling end-to-end training&#10;speedups of up to 40% over MoEs trained with the state-of-the-art Tutel library&#10;and 2.4x over DNNs trained with the highly-optimized Megatron-LM framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.00237" label="2305.00237">
        <attvalues>
          <attvalue for="0" value="A Review of ChatGPT Applications in Education, Marketing, Software&#10;  Engineering, and Healthcare: Benefits, Drawbacks, and Research Directions" />
          <attvalue for="1" value="  ChatGPT is a type of artificial intelligence language model that uses deep&#10;learning algorithms to generate human-like responses to text-based prompts. The&#10;introduction of the latest ChatGPT version in November of 2022 has caused&#10;shockwaves in the industrial and academic communities for its powerful&#10;capabilities, plethora of possible applications, and the great possibility for&#10;abuse. At the time of writing this work, several other language models (e.g.,&#10;Google Bard and Meta LLaMA) just came out in an attempt to get a foothold in&#10;the vast possible market. These models have the ability to revolutionize the&#10;way we interact with computers and have potential applications in many fields,&#10;including education, software engineering, healthcare, and marketing. In this&#10;paper, we will discuss the possible applications, drawbacks, and research&#10;directions using advanced language Chatbots (e.g., ChatGPT) in each of these&#10;fields. We first start with a brief introduction and the development timeline&#10;of artificial intelligence based language models, then we go through possible&#10;applications of such models, after that we discuss the limitations and&#10;drawbacks of the current technological state of the art, and finally we point&#10;out future possible research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.09103" label="2304.09103">
        <attvalues>
          <attvalue for="0" value="ChatGPT: Applications, Opportunities, and Threats" />
          <attvalue for="1" value="  Developed by OpenAI, ChatGPT (Conditional Generative Pre-trained Transformer)&#10;is an artificial intelligence technology that is fine-tuned using supervised&#10;machine learning and reinforcement learning techniques, allowing a computer to&#10;generate natural language conversation fully autonomously. ChatGPT is built on&#10;the transformer architecture and trained on millions of conversations from&#10;various sources. The system combines the power of pre-trained deep learning&#10;models with a programmability layer to provide a strong base for generating&#10;natural language conversations. In this study, after reviewing the existing&#10;literature, we examine the applications, opportunities, and threats of ChatGPT&#10;in 10 main domains, providing detailed examples for the business and industry&#10;as well as education. We also conducted an experimental study, checking the&#10;effectiveness and comparing the performances of GPT-3.5 and GPT-4, and found&#10;that the latter performs significantly better. Despite its exceptional ability&#10;to generate natural-sounding responses, the authors believe that ChatGPT does&#10;not possess the same level of understanding, empathy, and creativity as a human&#10;and cannot fully replace them in most situations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.03771" label="1910.03771">
        <attvalues>
          <attvalue for="0" value="HuggingFace's Transformers: State-of-the-art Natural Language Processing" />
          <attvalue for="1" value="  Recent progress in natural language processing has been driven by advances in&#10;both model architecture and model pretraining. Transformer architectures have&#10;facilitated building higher-capacity models and pretraining has made it&#10;possible to effectively utilize this capacity for a wide variety of tasks.&#10;\textit{Transformers} is an open-source library with the goal of opening up&#10;these advances to the wider machine learning community. The library consists of&#10;carefully engineered state-of-the art Transformer architectures under a unified&#10;API. Backing this library is a curated collection of pretrained models made by&#10;and available for the community. \textit{Transformers} is designed to be&#10;extensible by researchers, simple for practitioners, and fast and robust in&#10;industrial deployments. The library is available at&#10;\url{https://github.com/huggingface/transformers}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.02475" label="2212.02475">
        <attvalues>
          <attvalue for="0" value="Meta-Learning Fast Weight Language Models" />
          <attvalue for="1" value="  Dynamic evaluation of language models (LMs) adapts model parameters at test&#10;time using gradient information from previous tokens and substantially improves&#10;LM performance. However, it requires over 3x more compute than standard&#10;inference. We present Fast Weight Layers (FWLs), a neural component that&#10;provides the benefits of dynamic evaluation much more efficiently by expressing&#10;gradient updates as linear attention. A key improvement over dynamic evaluation&#10;is that FWLs can also be applied at training time so the model learns to make&#10;good use of gradient updates. FWLs can easily be added on top of existing&#10;transformer models, require relatively little extra compute or memory to run,&#10;and significantly improve language modeling perplexity.&#10;" />
          <attvalue for="2" value="&#10;A key challenge in language modeling is representing the contextual information from previous tokens. &#10;Transformer language models use attention to pass along this information, but constantly referring back to the previous text is a cognitively implausible model of working memory. &#10;An appealing alternative is using fast weight neural networks \cite{hinton1987using, schmidhuber1992learning}.&#10;Inspired by short-term plasticity in the brain, these models have parameters that change on-the-fly based on input data (previous tokens for LMs) in addition to standard ``slow&quot; weights learned during training.&#10;Fast weights have proven successful for supervised \cite{ba2016using}, reinforcement \cite{munkhdalai2019metalearned}, and few-shot \cite{munkhdalai2017meta} learning.&#10;&#10;Dynamic evaluation \cite{mikolov2010recurrent,krause2018dynamic} uses a variant of fast weights to improve language models at inference time.&#10;After scoring (or generating) a chunk of text, dynamic evaluation applies a gradient update to the model coming from the LM loss over that chunk before continuing.&#10;Intuitively, this process improves performance because an update that makes the model better at predicting previous tokens will likely also make it better at predicting future ones. &#10;Dynamic evaluation substantially improves LM perplexity, but has numerous drawbacks.&#10;It requires an extra forward and backward pass through the model to compute the gradients, and the sequential gradient updates over chunks cannot be parallelized.&#10;Furthermore, it is very memory intensive because a separate copy of the evolving model weights has to be stored for each example in a minibatch.&#10;Lastly, dynamic evaluation is only used at test-time, so the model does not learn to make good use of gradient updates.&#10;&#10;We present Fast Weight Layers (FWLs), a neural component that provides the benefits of dynamic evaluation with none of these downsides.&#10;They can be added to any LM without requiring changes to the training or evaluation loop.&#10;Like dynamic evaluation, FWLs also update their parameters using gradient information from previous tokens, but FWLs employ three key ideas to improve efficiency.&#10;First, FWLs are added on top of the transformer after the last attention layer, which avoids having to backpropagate through the whole transformer and circumvents the complexities of backpropagation through time. &#10;Secondly, FWLs compute gradients in parallel rather than recurrently.&#10;Lastly, FWLs leverage the property that gradient matrices are rank one to compute their outputs efficiently. &#10;&#10;Crucially, the efficient design means FWLs can be used at training time so the model learns to obtain beneficial updates from previous gradients.&#10;Another benefit is that while dynamic evaluation needs hyperparameter search to find a good step size for the gradient update, FWLs can instead learn the step size. &#10;Training FWLs can be viewed as applying gradient-based meta-learning \cite{finn2017model} to language modeling, where the support set contains tokens seen so far and the query set contains future tokens, a perspective which helps explain some of the behaviors of FWLs. &#10;&#10;FWLs scale well to long sequences and are complementary to existing long-text generation methods such as sparse attention \cite{child2019generating} or recurrent processing \cite{dai2019transformer}.&#10;We evaluate FWLs at language modeling on the WikiText-103 dataset \cite{merity2016pointer}.&#10;They substantially improve perplexities over strong baselines; for example lowering Transformer-XL's perplexity from 18.1 to 16.6.&#10;This gain is comparable to the one from dynamic evaluation, but FWLs add less than 20\% compute overhead to the model compared to the over 200\% of dynamic evaluation.&#10;Ablations show FWLs achieve superior compute vs perplexity trade-offs compared to alternative fast weight methods. We also analyze how FWLs improve results and find they are especially effective at modeling rare tokens, repeated tokens, and long documents. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Mathematics, Gradient-Based Learning, Neural Network Efficiency, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1907.09720" label="1907.09720">
        <attvalues>
          <attvalue for="0" value="Metalearned Neural Memory" />
          <attvalue for="1" value="  We augment recurrent neural networks with an external memory mechanism that&#10;builds upon recent progress in metalearning. We conceptualize this memory as a&#10;rapidly adaptable function that we parameterize as a deep neural network.&#10;Reading from the neural memory function amounts to pushing an input (the key&#10;vector) through the function to produce an output (the value vector). Writing&#10;to memory means changing the function; specifically, updating the parameters of&#10;the neural network to encode desired information. We leverage training and&#10;algorithmic techniques from metalearning to update the neural memory function&#10;in one shot. The proposed memory-augmented model achieves strong performance on&#10;a variety of learning problems, from supervised question answering to&#10;reinforcement learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.00837" label="1703.00837">
        <attvalues>
          <attvalue for="0" value="Meta Networks" />
          <attvalue for="1" value="  Neural networks have been successfully applied in applications with a large&#10;amount of labeled data. However, the task of rapid generalization on new&#10;concepts with small training data while preserving performances on previously&#10;learned ones still presents a significant challenge to neural network models.&#10;In this work, we introduce a novel meta learning method, Meta Networks&#10;(MetaNet), that learns a meta-level knowledge across tasks and shifts its&#10;inductive biases via fast parameterization for rapid generalization. When&#10;evaluated on Omniglot and Mini-ImageNet benchmarks, our MetaNet models achieve&#10;a near human-level performance and outperform the baseline approaches by up to&#10;6% accuracy. We demonstrate several appealing properties of MetaNet relating to&#10;generalization and continual learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.03728" label="2408.03728">
        <attvalues>
          <attvalue for="0" value="A Convex-optimization-based Layer-wise Post-training Pruner for Large&#10;  Language Models" />
          <attvalue for="1" value="  Pruning is a critical strategy for compressing trained large language models&#10;(LLMs), aiming at substantial memory conservation and computational&#10;acceleration without compromising performance. However, existing pruning&#10;methods often necessitate inefficient retraining for billion-scale LLMs or rely&#10;on heuristic methods such as the optimal brain surgeon framework, which degrade&#10;performance. In this paper, we introduce FISTAPruner, the first post-training&#10;pruner based on convex optimization models and algorithms. Specifically, we&#10;propose a convex optimization model incorporating $\ell_1$ norm to induce&#10;sparsity and utilize the FISTA solver for optimization. FISTAPruner&#10;incorporates an intra-layer cumulative error correction mechanism and supports&#10;parallel pruning. We comprehensively evaluate FISTAPruner on models such as&#10;OPT, LLaMA, LLaMA-2, and LLaMA-3 with 125M to 70B parameters under unstructured&#10;and 2:4 semi-structured sparsity, demonstrating superior performance over&#10;existing state-of-the-art methods across various language benchmarks.&#10;" />
          <attvalue for="2" value="&#10;In recent years, large language models (LLMs) have revolutionized natural language processing fields, achieving impressive results in tasks such as machine translation, sentiment analysis, question answering, and text generation \cite{lyu2023new, yao2023empowering, zhang2023prompting, zhang2023sentiment, wang2023augmenting, arefeen2024leancontext, li2024pre}. Advanced LLMs such as OpenAI's GPT-4~\cite{openai2023gpt}, Meta's LLaMA-3~\cite{meta2023llama3}, and Google's Gemini \cite{team2023gemini} excel in generating coherent text with extensive parameters. However, the growth in model sizes outpaces hardware improvements, posing significant deployment and inference challenges \cite{steiner2023model}. For example, operating OPT-175B \cite{zhang2022opt} requires over 320GB of memory and at least five 80GB A100 GPUs for loading its parameters in FP16 precision. This challenge becomes more pronounced in environments with limited resources, such as mobile devices, edge computing systems, and real-time applications. Consequently, there has been considerable interest in compressing LLMs to enhance their efficiency and practicality for deployment across various applications.&#10;&#10;Pruning is a key method for compressing LLMs, aiming to eliminate redundant weights to reduce model size and computational demands while striving to maintain performance. Methods such as those in \cite{huang2020convolution, ma2023llm_pruner, zhang2023lottery} require a retraining phase post-pruning, which is inefficient for billion-scale LLMs. Recent developments, including SparseGPT~\cite{frantar2023sparsegpt} and Wanda \cite{sun2023simple}, employ one-shot post-training pruning techniques for LLMs. These methods, however, rely on the heuristic-based optimal brain surgeon (OBS) framework~\cite{hassibi1992second} or utilize heuristic-based pruning metric to determine which weights to prune, potentially compromising performance.&#10;&#10;In this work, we introduce a novel convex optimization model for layer-wise post-training pruning of LLMs. Figure \ref{fig:overview} provides an overview of our method, which is applied to each linear operator, such as \(W_K\), \(W_Q\), \(W_V\), and $W_O$ within the Transformer's attention blocks \cite{vaswani2017attention}. We employ the Frobenius norm of the difference between the outputs obtained from the dense and pruned weights to quantify the output error. Additionally, we integrate an $\ell_1$-norm regularization term, the optimal convex approximation of the $\ell_0$-norm~\cite{candes2006robust}, into each row of weights to promote sparsity. The solutions of the proposed optimization model demonstrate a balanced trade-off between output error and sparsity, governed by our proposed adaptive tuning method that meticulously adjusts the hyperparameter $\lambda$. To solve this optimization problem efficiently, we utilize the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) \cite{beck2009fast}, which ensures a convergence rate of $O(1/k^2)$. Following this, we name our proposed method FISTAPruner.&#10;&#10;In addition, our approach effectively addresses the cumulative error resulting from compression within the decoder layers by incorporating an error correction mechanism. Specifically, there exists a discrepancy between the outputs of the dense and the pruned weights. This error accumulates since the output from one pruned weight serves as the input for the subsequent operator. To mitigate this accumulated error, FISTAPruner sequentially prunes the weights of each linear operator within a decoder layer. It utilizes the output from the pruned weights of the preceding operator as the input activation for the ongoing pruning process, thereby minimizing the gap between the output weight being pruned and its counterpart in the dense model. &#10;Moreover, FISTAPruner treats each decoder layer as an independent pruning unit, supporting the simultaneous pruning of multiple decoder layers, which significantly enhances efficiency.&#10;&#10;We empirically evaluate FISTAPruner on the widely adopted OPT \cite{zhang2022opt}, LLaMA~\cite{touvron2023llama}, and LLaMA-2~\cite{touvron2023llama2} model families, as well as the latest LLaMA-3 \cite{touvron2023llama} models. FISTAPruner’s layer-by-layer pruning implementation allows for the pruning of these LLMs ranging from 125M to 70B parameters on a single NVIDIA A100 GPU with 40GB of memory. Our results confirm that FISTAPruner can efficiently create sparse networks from pretrained LLMs without retraining. Moreover, our approach exceeds the performance of state-of-the-art methods such as SparseGPT and Wanda across various language benchmarks. We believe our work sets a new direction and baseline for future research in this area and encourages further exploration into understanding sparsity in LLMs with the tools of convex optimization.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Convex Optimization Methods, Artificial Intelligence, Mathematics, Language Model Pruning" />
        </attvalues>
      </node>
      <node id="2004.11627" label="2004.11627">
        <attvalues>
          <attvalue for="0" value="Convolution-Weight-Distribution Assumption: Rethinking the Criteria of&#10;  Channel Pruning" />
          <attvalue for="1" value="  Channel pruning is a popular technique for compressing convolutional neural&#10;networks (CNNs), where various pruning criteria have been proposed to remove&#10;the redundant filters. From our comprehensive experiments, we found two blind&#10;spots in the study of pruning criteria: (1) Similarity: There are some strong&#10;similarities among several primary pruning criteria that are widely cited and&#10;compared. According to these criteria, the ranks of filters'Importance Score&#10;are almost identical, resulting in similar pruned structures. (2)&#10;Applicability: The filters'Importance Score measured by some pruning criteria&#10;are too close to distinguish the network redundancy well. In this paper, we&#10;analyze these two blind spots on different types of pruning criteria with&#10;layer-wise pruning or global pruning. The analyses are based on the empirical&#10;experiments and our assumption (Convolutional Weight Distribution Assumption)&#10;that the well-trained convolutional filters each layer approximately follow a&#10;Gaussian-alike distribution. This assumption has been verified through&#10;systematic and extensive statistical tests.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.11695" label="2306.11695">
        <attvalues>
          <attvalue for="0" value="A Simple and Effective Pruning Approach for Large Language Models" />
          <attvalue for="1" value="  As their size increases, Large Languages Models (LLMs) are natural candidates&#10;for network pruning methods: approaches that drop a subset of network weights&#10;while striving to preserve performance. Existing methods, however, require&#10;either retraining, which is rarely affordable for billion-scale LLMs, or&#10;solving a weight reconstruction problem reliant on second-order information,&#10;which may also be computationally expensive. In this paper, we introduce a&#10;novel, straightforward yet effective pruning method, termed Wanda (Pruning by&#10;Weights and activations), designed to induce sparsity in pretrained LLMs.&#10;Motivated by the recent observation of emergent large magnitude features in&#10;LLMs, our approach prunes weights with the smallest magnitudes multiplied by&#10;the corresponding input activations, on a per-output basis. Notably, Wanda&#10;requires no retraining or weight update, and the pruned LLM can be used as is.&#10;We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2&#10;across various language benchmarks. Wanda significantly outperforms the&#10;established baseline of magnitude pruning and performs competitively against&#10;recent method involving intensive weight update. Code is available at&#10;https://github.com/locuslab/wanda.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="math/0409186" label="math/0409186">
        <attvalues>
          <attvalue for="0" value="Robust Uncertainty Principles: Exact Signal Reconstruction from Highly&#10;  Incomplete Frequency Information" />
          <attvalue for="1" value="  This paper considers the model problem of reconstructing an object from&#10;incomplete frequency samples. Consider a discrete-time signal $f \in \C^N$ and&#10;a randomly chosen set of frequencies $\Omega$ of mean size $\tau N$. Is it&#10;possible to reconstruct $f$ from the partial knowledge of its Fourier&#10;coefficients on the set $\Omega$?&#10;  A typical result of this paper is as follows: for each $M &gt; 0$, suppose that&#10;$f$ obeys $$ # \{t, f(t) \neq 0 \} \le \alpha(M) \cdot (\log N)^{-1} \cdot #&#10;\Omega, $$ then with probability at least $1-O(N^{-M})$, $f$ can be&#10;reconstructed exactly as the solution to the $\ell_1$ minimization problem $$&#10;\min_g \sum_{t = 0}^{N-1} |g(t)|, \quad \text{s.t.} \hat g(\omega) = \hat&#10;f(\omega) \text{for all} \omega \in \Omega. $$ In short, exact recovery may be&#10;obtained by solving a convex optimization problem. We give numerical values for&#10;$\alpha$ which depends on the desired probability of success; except for the&#10;logarithmic factor, the condition on the size of the support is sharp.&#10;  The methodology extends to a variety of other setups and higher dimensions.&#10;For example, we show how one can reconstruct a piecewise constant (one or&#10;two-dimensional) object from incomplete frequency samples--provided that the&#10;number of jumps (discontinuities) obeys the condition above--by minimizing&#10;other convex functionals such as the total-variation of $f$.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.05159" label="2210.05159">
        <attvalues>
          <attvalue for="0" value="Can Language Models Be Specific? How?" />
          <attvalue for="1" value="  &quot;He is a person&quot;, &quot;Paris is located on the earth&quot;. Both statements are&#10;correct but meaningless - due to lack of specificity. In this paper, we propose&#10;to measure how specific the language of pre-trained language models (PLMs) is.&#10;To achieve this, we introduce a novel approach to build a benchmark for&#10;specificity testing by forming masked token prediction tasks with prompts. For&#10;instance, given &quot;Toronto is located in [MASK].&quot;, we want to test whether a more&#10;specific answer will be better filled in by PLMs, e.g., Ontario instead of&#10;Canada. From our evaluations, we show that existing PLMs have only a slight&#10;preference for more specific answers. We identify underlying factors affecting&#10;the specificity and design two prompt-based methods to improve the specificity.&#10;Results show that the specificity of the models can be improved by the proposed&#10;methods without additional training. We hope this work can bring to awareness&#10;the notion of specificity of language models and encourage the research&#10;community to further explore this important but understudied problem.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-trained language models (PLMs) such as BERT \cite{devlin2019bert} and GPT-2/3 \cite{radford2019language,NEURIPS2020_1457c0d6} have achieved quite impressive results in various natural language processing tasks.&#10;Recent works show that the parameters of these models contain significant amounts of knowledge \cite{petroni2019language,roberts2020much,jiang2020x,jiang2020can,wang2020language}, and knowledge stored in PLMs can be extracted by predicting the mask token(s) using prompts. &#10;For instance, given prompt ``J. K. Rowling was born in [MASK].'', PLMs can predict the birthplace of Rowling based on its knowledge.&#10;&#10;However, there may exist multiple answers for a query, while not all answers are equally specific. In many situations, we desire a specific answer.&#10;For the example above, the masked token can be replaced by Yate (a town), Gloucestershire (a county), or England (a country). &#10;To acquire the maximum knowledge (in this example, the town, the county, and the country where Rowling was born), &#10;we may prefer the model to fill in Yate since Gloucestershire and England can be further predicted using prompts, e.g., ``Yate is located in [MASK].''&#10;This means, if the prediction is more specific, we can retrieve more fine-grained information from language models, and further acquire more information.&#10;Besides, sometimes, the less specific answer is not useful. For instance, it is well known that Chicago is located in the USA, users will not get additional information if the model only predicts Chicago is located in the USA instead of Illinois. More examples are shown in Figure \ref{fig:intro}.&#10;To make an analogy:&#10;A good speaker not only needs to be correct, but also has the ability to be specific when desired. The same is true for language models.&#10;&#10;Although there are works on measuring how much knowledge is stored in PLMs or improving the correctness of the predictions \cite{petroni2019language,roberts2020much,jiang2020can}, few attempted to measure or improve the specificity of predictions made by PLMs. &#10;Noteworthy exceptions include the work by \cite{adiwardana2020towards,thoppilan2022lamda}, who evaluated the specificity of conversational language models. In their research, specificity was defined and measured within a conversational context -- for instance, the response ``Me too. I love Eurovision songs'' is deemed more specific than simply ``Me too'' to the statement ``I love Eurovision''.&#10;Understanding how specific the language of PLMs is can help us better understand the behavior of language models and facilitate downstream applications such as question answering, text generation, and information extraction \cite{liu2021pre,khashabi2020unifiedqa,NEURIPS2020_1457c0d6,wang2020language}, e.g., making the generated answers/sentences or extracted information more specific or fine-grained.&#10;&#10;Therefore, we propose to build a benchmark to measure the specificity of the language of PLMs. &#10;For reducing human effort and easier to further expand the dataset (e.g., to specific domains),&#10;we introduce a novel way to construct test data automatically based on transitive relations in Wikidata \cite{vrandevcic2014wikidata}.&#10;Specifically, we extract reasoning paths from Wikidata, e.g., (J. K. Rowling, birthplace, Yate, location, Gloucestershire, location, England).&#10;Based on the average distance of each object to the subject and the property of transitive relations, we form masked-token-prediction based probing tasks to measure the specificity, e.g., whether the masked token in ``J. K. Rowling was born in [MASK].'' is better filled by Yate than England by PLMs.&#10;The resulting benchmark dataset contains more than 20,000 probes covering queries from 5 different categories.&#10;The quality of the benchmark is high, where the judgment on which answer is more specific is $\sim97\%$ consistent with humans.&#10;&#10;We provide in-depth analyses on model specificity and study two factors that affect the specificity with our benchmark. &#10;As shown by our evaluations in Section \ref{sec:analysis}, existing PLMs, e.g., BERT and GPT-2, similarly have only a slight preference for more specific answers (in only about $60\%$ of cases where a more specific answer is preferred). &#10;We also show that, in general, PLMs prefer less specific answers without subjects given, and they only have a weak ability to differentiate coarse-grained/fine-grained objects by measuring their similarities to subjects.&#10;The results indicate that specificity was neglected by existing research on language models.&#10;How to improve and control it is undoubtedly an interesting and valuable problem.&#10;&#10;Based on our observations and analyses, we propose two techniques to improve the specificity of the predictions by modifying the prompts without additional training: &#10;Few-shot Prompting, where demonstrations with more specific answers are provided to guide the models to produce more specific answers; and Cascade Prompting, where which clauses are added as suffixes to bias the predictions to be more specific. Results show that Few-shot Prompting can improve the specificity for unidirectional language models like GPT-2 well, while Cascade Prompting works well for bidirectional language models such as BERT.&#10;&#10;The main contributions of our work are summarized as follows:&#10;\begin{itemize}[nolistsep]&#10;\item We propose a novel automatic approach to build a benchmark for specificity testing based on the property of transitive relations.&#10;\item We analyze the specificity of several existing PLMs and study two factors that affect the specificity.&#10;\item We propose two methods to improve the specificity by modifying the prompts without additional training.&#10;\item We provide in-depth analyses and discussions, suggesting further works to explore and further improve the specificity.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Specificity Measurement, Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence Improvement, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2005.00700" label="2005.00700">
        <attvalues>
          <attvalue for="0" value="UnifiedQA: Crossing Format Boundaries With a Single QA System" />
          <attvalue for="1" value="  Question answering (QA) tasks have been posed using a variety of formats,&#10;such as extractive span selection, multiple choice, etc. This has led to&#10;format-specialized models, and even to an implicit division in the QA&#10;community. We argue that such boundaries are artificial and perhaps&#10;unnecessary, given the reasoning abilities we seek to teach are not governed by&#10;the format. As evidence, we use the latest advances in language modeling to&#10;build a single pre-trained QA model, UnifiedQA, that performs surprisingly well&#10;across 17 QA datasets spanning 4 diverse formats. UnifiedQA performs on par&#10;with 9 different models that were trained on individual datasets themselves.&#10;Even when faced with 12 unseen datasets of observed formats, UnifiedQA performs&#10;surprisingly well, showing strong generalization from its out-of-format&#10;training data. Finally, simply fine-tuning this pre-trained QA model into&#10;specialized models results in a new state of the art on 6 datasets,&#10;establishing UnifiedQA as a strong starting point for building QA systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.12230" label="2406.12230">
        <attvalues>
          <attvalue for="0" value="MCSD: An Efficient Language Model with Diverse Fusion" />
          <attvalue for="1" value="  Transformers excel in Natural Language Processing (NLP) due to their prowess&#10;in capturing long-term dependencies but suffer from exponential resource&#10;consumption with increasing sequence lengths. To address these challenges, we&#10;propose MCSD model, an efficient language model with linear scaling and fast&#10;inference speed. MCSD model leverages diverse feature fusion, primarily through&#10;the multi-channel slope and decay (MCSD) block, to robustly represent features.&#10;This block comprises slope and decay sections that extract features across&#10;diverse temporal receptive fields, facilitating capture of both local and&#10;global information. In addition, MCSD block conducts element-wise fusion of&#10;diverse features to further enhance the delicate feature extraction capability.&#10;For inference, we formulate the inference process into a recurrent&#10;representation, slashing space complexity to $O(1)$ and time complexity to&#10;$O(N)$ respectively. Our experiments show that MCSD attains higher throughput&#10;and lower GPU memory consumption compared to Transformers, while maintaining&#10;comparable performance to larger-scale language learning models on benchmark&#10;tests. These attributes position MCSD as a promising base for edge deployment&#10;and embodied intelligence.&#10;" />
          <attvalue for="2" value="&#10;Recent years have witnessed significant strides in Natural Language Processing (NLP), notably the emergence of Large Language Models (LLMs) \cite{LLM}, transforming machine-human interaction by mimicking human-like language comprehension and generation. Among them, Transformer dominates in NLP due to its powerful performance \cite{1}. Benefiting from its self-attention mechanism, Transformer has long-range dependencies that substantially improve the ability to process language. Transformer-based LLMs trained on extensive datasets sourced from the web have achieved remarkable success &#10;\cite{brown2020language,achiam2023gpt4,gemini2023}. However, it suffers from the disadvantage of high computational resource consumption, accompanied by $O(N^2)$ computational complexity and $O(N)$ space complexity at inference \cite{kvcache}. The computational requirements of the model scale quadratically with the length of the input sequence $N$ during inference. This limits its application to some scenarios, such as time-sensitive or resource-limited environments, typical of edge devices. &#10;&#10;Many attempts have been made to address the above drawbacks. Some methods simplify the query mechanism by replacing multi-head attention with multi-query attention \cite{kvcache} and group query attention \cite{ainslie2023gqa}. The memory requirement is reduced by sharing the keys $K$ and values $V$ among all heads or grouped heads, providing a flexible trade-off between computational efficiency and model representation capability. Other approaches focus on improving the computational efficiency of attention, such as AFT \cite{AFT} and RWKV \cite{2}. They obviate computing and storing expensive attention matrices by optimizing the matrix multiplication, resulting in linear computational complexity. However, these methods still use a QKV-based querying mechanism as multi-head attention does, which requires global interactions between queries and values. Recently, alternatives like Mamba \cite{mamba}, rooted in state-space model (SSM) evolution, have gained traction in the reseach community. Mamba extends input information into a higher-dimensional hidden space, demonstrating efficiency and lightness. Yet, empirical evidences suggest scaling challenges for Mamba \cite{de2024griffin}, indicating ongoing hurdles in its widespread adoption.&#10;&#10;To this paper, we propose MCSD, an efficient language model that achieves a trade-off between performance and computational efficiency through diverse fusion. Specifically, our method facilitates both local and global feature fusion through the innovative MCSD block. In the MCSD block, it contains a slope sectionand a decay section, each adept at integrating long-range and short-range information, respectively. Subsequently, combining the outputs of two sections can empower the network with rich feature extraction capabilities. We use multi-channel slope and multi-channel decay to integrate historical information across varying temporal receptive fields, and leverage diverse perturbations for fine-grained element-wise integration. In addition, our approach uses multiple predefined linear operations instead of traditional dot product token interactions, thereby achieving a linear computational complexity and significantly reduced memory footprint during inference.&#10;Our contributions in this paper are as follows:&#10;&#10;\begin{itemize}&#10;&#10;\item A new MCSD model is proposed to achieve a balance between computational consumption and representation ability via series of linear fusion, helping to address the scaling and deployment challenges.&#10;&#10;\item To enhance the feature extraction capability, Multi-Channel Slope and Decay (MCSD) block is proposed to achieve rich feature extraction and diverse fusion, which ensure the diversity of feature extraction and fine-grained feature interaction.&#10;&#10;\item A recurrent representation is proposed during inference stage to accelerate the inference speed. This simplified approach enjoys a computational complexity linear to the length of the sequence and has a low and stable memory complexity.&#10;&#10;\item Experiments show that the MCSD approach outperforms Transformer on three metrics: GPU memory, latency, and throughput, showing the robust scalability of our method. The results confirm its competitive edge in delivering high-performance outcomes at low computational costs, making it a viable solution for resource-constrained edge-device deployments.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Efficient Language Models, Linguistics, Deep Learning Architectures, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2105.14103" label="2105.14103">
        <attvalues>
          <attvalue for="0" value="An Attention Free Transformer" />
          <attvalue for="1" value="  We introduce Attention Free Transformer (AFT), an efficient variant of&#10;Transformers that eliminates the need for dot product self attention. In an AFT&#10;layer, the key and value are first combined with a set of learned position&#10;biases, the result of which is multiplied with the query in an element-wise&#10;fashion. This new operation has a memory complexity linear w.r.t. both the&#10;context size and the dimension of features, making it compatible to both large&#10;input and model sizes. We also introduce AFT-local and AFT-conv, two model&#10;variants that take advantage of the idea of locality and spatial weight sharing&#10;while maintaining global connectivity. We conduct extensive experiments on two&#10;autoregressive modeling tasks (CIFAR10 and Enwik8) as well as an image&#10;recognition task (ImageNet-1K classification). We show that AFT demonstrates&#10;competitive performance on all the benchmarks, while providing excellent&#10;efficiency at the same time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.19427" label="2402.19427">
        <attvalues>
          <attvalue for="0" value="Griffin: Mixing Gated Linear Recurrences with Local Attention for&#10;  Efficient Language Models" />
          <attvalue for="1" value="  Recurrent neural networks (RNNs) have fast inference and scale efficiently on&#10;long sequences, but they are difficult to train and hard to scale. We propose&#10;Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that&#10;mixes gated linear recurrences with local attention. Hawk exceeds the reported&#10;performance of Mamba on downstream tasks, while Griffin matches the performance&#10;of Llama-2 despite being trained on over 6 times fewer tokens. We also show&#10;that Griffin can extrapolate on sequences significantly longer than those seen&#10;during training. Our models match the hardware efficiency of Transformers&#10;during training, and during inference they have lower latency and significantly&#10;higher throughput. We scale Griffin up to 14B parameters, and explain how to&#10;shard our models for efficient distributed training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.07962" label="2202.07962">
        <attvalues>
          <attvalue for="0" value="Revisiting Parameter-Efficient Tuning: Are We Really There Yet?" />
          <attvalue for="1" value="  Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the&#10;new paradigm for using pretrained language models (PLMs). By tuning just a&#10;fraction amount of parameters comparing to full model finetuning, PETuning&#10;methods claim to have achieved performance on par with or even better than&#10;finetuning. In this work, we take a step back and re-examine these PETuning&#10;methods by conducting the first comprehensive investigation into the training&#10;and evaluation of them. We found the problematic validation and testing&#10;practice in current studies, when accompanied by the instability nature of&#10;PETuning methods, has led to unreliable conclusions. When being compared under&#10;a truly fair evaluation protocol, PETuning cannot yield consistently&#10;competitive performance while finetuning remains to be the best-performing&#10;method in medium- and high-resource settings. We delve deeper into the cause of&#10;the instability and observed that the number of trainable parameters and&#10;training iterations are two main factors: reducing trainable parameters and&#10;prolonging training iterations may lead to higher stability in PETuning&#10;methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Instability of finetuning PLMs. &#10;While our study is, to the best of our knowledge, the first to systematically investigate PETuning instability, prior studies have looked into the instability of finetuning PLMs.&#10;\cite{dodge2020finetuning} illustrated the inherent instability of finetuning by controlling random seeds and provided a new early stopping strategy to improve instability.&#10;\cite{LeeCK20mixout} proposed a new regularisation method by mixing two models based on dropout to prevent catastrophic forgetting and to improve instability.&#10;More recently, \cite{conf/iclr/MosbachAK21} revisited the hypotheses of finetuning instability proposed by previous studies and found that optimisation difficulties can lead to vanishing gradients, which further causes finetuning instability. \cite{zhang2021revisiting} also revealed that optimisation significantly affects the instabilities in few-sample fine-tuning.&#10;&#10;Analysis of PETuning.&#10;As PETuning methods have become a prominent research direction, a great number of studies aim to analyse the characteristics of these methods.&#10;\cite{he-etal-2021-effectiveness} investigated the effectiveness of Adapter across different scales and \cite{han-etal-2021-robust} provided a robust strategy for training Adapter.&#10;Recently, \cite{he2021unified} and \cite{mao2021unipelt} proposed a unified view to connect various PETuning methods. However, there has not been reliable validation and comparison for off-the-shelf PETuning methods in terms of stability and effectiveness, and this is where our paper bridges the gap.&#10; " />
          <attvalue for="4" value="Language Model Tuning, Computer Science, Linguistics, Parameter Efficiency, Evaluation Methodology, Mathematics, Statistics" />
        </attvalues>
      </node>
      <node id="2402.11094" label="2402.11094">
        <attvalues>
          <attvalue for="0" value="Word Embeddings Revisited: Do LLMs Offer Something New?" />
          <attvalue for="1" value="  Learning meaningful word embeddings is key to training a robust language&#10;model. The recent rise of Large Language Models (LLMs) has provided us with&#10;many new word/sentence/document embedding models. Although LLMs have shown&#10;remarkable advancement in various NLP tasks, it is still unclear whether the&#10;performance improvement is merely because of scale or whether underlying&#10;embeddings they produce significantly differ from classical encoding models&#10;like Sentence-BERT (SBERT) or Universal Sentence Encoder (USE). This paper&#10;systematically investigates this issue by comparing classical word embedding&#10;techniques against LLM-based word embeddings in terms of their latent vector&#10;semantics. Our results show that LLMs tend to cluster semantically related&#10;words more tightly than classical models. LLMs also yield higher average&#10;accuracy on the Bigger Analogy Test Set (BATS) over classical methods. Finally,&#10;some LLMs tend to produce word embeddings similar to SBERT, a relatively&#10;lighter classical model.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the introduction of Word2Vec~\cite{mikolov2013efficient} and GLoVe~\cite{pennington-etal-2014-glove}, word embedding techniques and their semantic analysis have gained significant popularity in the field of deep NLP research. Since then, the size and complexity of the embedding models have increased dramatically, especially with the introduction of transformer-based architectures like BERT~\cite{devlin2018bert}, BART~\cite{lewis2019bart}, RoBERTa~\cite{liu2019roberta}, followed by rigorously fine-tuned sentence encoders like LASER \cite{Artetxe_2019}, Universal Sentence Encoder (USE) \cite{cer2018universal}, SentenceBERT (SBERT) \cite{reimers2019sentencebert}, etc., and finally with the emergence of Large Language Models (LLMs) like GPT~\cite{brown2020language}, PaLM~\cite{chowdhery2022palm}, LLaMA~\cite{touvron2023llama}, etc. While transformer-based models not only offer embeddings for words but also for longer sequences like sentences and documents, learning meaningful word embeddings still remains fundamental for proper contextualization of longer sequences and robust language modeling.&#10;&#10;While LLMs have shown remarkable success across various well-known NLP tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using,zero-shot-souvika,Bangla-Word-Analogy}, it remains an open question if their enhanced performance can solely be attributed to their larger scale or if the embeddings they generate are fundamentally distinct from those created by traditional encoding models such as Sentence-BERT or Universal Sentence Encoders~\cite{yash,souvika-USE}. Since the introduction of the Transformer architecture, model development has exploded with innovations, improving year after year. However, the medium has remained the same: written language. In theory, the similarity and relatedness of words have not changed, but the way that models treat words and their similarity has. More recently, \cite{fournier2020analogies} evaluated smaller transformer models like GPT-2 and BERT on analogy tasks and found that these models performed poorly on Encyclopedic and Lexicographic analogies, further raising the question of how embedding performance changes as model size increases. As such, we perform the following two analyses in this paper.&#10;&#10;\begin{itemize}[leftmargin=*,itemsep=0.2ex,partopsep=0ex,parsep=0ex]&#10;&#10;\item Word-Pair Similarity Analysis: Given two distinct word embedding techniques (classical or LLM-based), we compare and analyze their cosine similarity distributions of pairs of 1) Semantically Related, 2) Morphologically Related, and 3) Unrelated words, where the distributions are derived from their word embedding vectors.&#10;&#10;\item Word Analogy Task Analysis: For a given word analogy task of the form $a:b::c:d$ (``a is to b as c is to d''), we compare the accuracies of various embedding techniques using vector arithmetic in the latent vector space. \end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Word Embedding Techniques, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2309.03747" label="2309.03747">
        <attvalues>
          <attvalue for="0" value="The Daunting Dilemma with Sentence Encoders: Success on Standard&#10;  Benchmarks, Failure in Capturing Basic Semantic Properties" />
          <attvalue for="1" value="  In this paper, we adopted a retrospective approach to examine and compare&#10;five existing popular sentence encoders, i.e., Sentence-BERT, Universal&#10;Sentence Encoder (USE), LASER, InferSent, and Doc2vec, in terms of their&#10;performance on downstream tasks versus their capability to capture basic&#10;semantic properties. Initially, we evaluated all five sentence encoders on the&#10;popular SentEval benchmark and found that multiple sentence encoders perform&#10;quite well on a variety of popular downstream tasks. However, being unable to&#10;find a single winner in all cases, we designed further experiments to gain a&#10;deeper understanding of their behavior. Specifically, we proposed four semantic&#10;evaluation criteria, i.e., Paraphrasing, Synonym Replacement, Antonym&#10;Replacement, and Sentence Jumbling, and evaluated the same five sentence&#10;encoders using these criteria. We found that the Sentence-Bert and USE models&#10;pass the paraphrasing criterion, with SBERT being the superior between the two.&#10;LASER dominates in the case of the synonym replacement criterion.&#10;Interestingly, all the sentence encoders failed the antonym replacement and&#10;jumbling criteria. These results suggest that although these popular sentence&#10;encoders perform quite well on the SentEval benchmark, they still struggle to&#10;capture some basic semantic properties, thus, posing a daunting dilemma in NLP&#10;research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;By far, many have proposed a variety of techniques to generate the embedding for a given sentence. Doc2Vec \cite{le2014distributed} is an unsupervised technique that generates embedding based on the variable-length piece of text and creates unique embeddings for each paragraph in a document. Later, others attempted to learn sentence embedding using auto-encoder \cite{socher2011dynamic,hill2016learning}, \cite{hu2017toward}. On the other hand, InferSent \cite{conneau-etal-2017-supervised} used SNLI~\cite{dolan2004unsupervised} and Multi-genre NLI labeled data~\cite{williams2017broad} and learned the sentence embedding using the Bi-LSTM with max-pooling architecture and a Siamese network. &#10;&#10;More recently, \cite{cer2018universal} proposed ``Universal Sentence Encoder'' (USE), which is trained on the combination of supervised and unsupervised NLI (Natural Language Inference) data, and it has effectively produced sophisticated sentence embeddings. Sentence BERT (SBert) \cite{reimers-gurevych-2019-sentence}, which is trained on Wikipedia corpus and news-wire articles and later fine-tuned on SNLI and Multi-Genre NLI dataset. These models have been trained rigorously on a large corpus of data, and many of them used data parallelisms \cite{wieting2017paranmt,artetxe2019massively,wieting2019simple,wieting2019bilingual}, natural language inference (NLI) \cite{conneau2017supervised,conneau2018you,reimers2019sentence}, or a combination of both \cite{subramanian2018learning}. &#10;&#10;However, recently \cite{reimers-gurevych-2019-sentence,li2020sentence, pham2021out} reported that these pre-trained language models produce poor embeddings for semantic similarity tasks. Many pre-trained language models are designed for task-specific purposes; as a result, the embeddings generated by the models could be biased. To further investigate this issue in this paper, we conduct a systematic study of popular sentence encoders by proposing four basic semantic evaluation criteria and report our findings to inform the research community.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2204.06452" label="2204.06452">
        <attvalues>
          <attvalue for="0" value="Building Markovian Generative Architectures over Pretrained LM Backbones&#10;  for Efficient Task-Oriented Dialog Systems" />
          <attvalue for="1" value="  Recently, Transformer based pretrained language models (PLMs), such as GPT2&#10;and T5, have been leveraged to build generative task-oriented dialog (TOD)&#10;systems. A drawback of existing PLM-based models is their non-Markov&#10;architectures across turns, i.e., the whole history is used as the conditioning&#10;input at each turn. First, this brings inefficiencies in memory and&#10;computation. Furthermore, using the whole history increases model complexity&#10;and may hurt the training efficiency, especially when facing small amounts of&#10;labeled training data (the low-resource setting). In this paper, motivated by&#10;the observation that dialog states could be viewed as Markov states, we propose&#10;to build Markovian Generative Architectures (MGA) over PLM backbones for&#10;efficient TOD systems. Experiments on MultiWOZ2.1 show that in the&#10;rich-resource setting, the proposed Markov models reduce memory and time costs&#10;without performance degradation; in the low-resource setting, the training&#10;efficiency of the Markov models is more significant.&#10;" />
          <attvalue for="2" value="&#10;Task-oriented dialog (TOD) systems interact with users in natural languages through multiple turns to accomplish tasks.&#10;At each turn, the system needs to parse the user utterance and track the dialog state, which is usually defined to be a compact summary of dialog history, as shown in Figure~\ref{fig:TOD}.&#10;The dialog state is often represented by a set of slot-value pairs that determine the user's requirement.&#10;Based on the tracked dialog state, the system will query a task-related database (DB), decide an action and generate a response.&#10;The methodology for building TOD systems is gradually advancing from separate training of individual modules \cite{williams2016dialog,mrkvsic2017neural, dai2018tracking} to the end-to-end (E2E) trainable approach \cite{wen2017a, liu2017end, lei2018sequicity, fsdm, zhang2020task,gao2020paraphrase,zhang-etal-2020-probabilistic}.&#10;In early E2E methods, the sequential turns of a dialog are usually modeled as a Markov process and realized over LSTM-based backbones.&#10;&#10;Recently, self-attention based Transformer neural networks \cite{vaswani2017attention} have shown their superiority in capturing long-term dependencies over LSTM based networks. Transformer based pretrained language models (PLM), such as GPT2 \cite{radford2019gpt2} and T5 \cite{JMLR:v21:20-074}, have been leveraged to build generative E2E TOD systems in the pretraining-and-finetuning paradigm, which have shown improved performance over LSTM-based ones. Examples include GPT2-based SimpleTOD \cite{hosseini2020simple}, SOLOIST \cite{peng2020etal}, AuGPT \cite{kulhanek2021augpt} and UBAR \cite{yang2021ubar}, and T5-based PPTOD \cite{su2021multitask} and MTTOD \cite{lee-2021-improving-end}, among others.&#10;A drawback of existing PLM-based methods, viewed from efficiencies in memory, computation and learning, is that the whole history is used as the conditioning input at each turn. The dialog model thus becomes non-Markov across turns, i.e., the generation at current turn depends not only on the previous turn but also on all previous turns, namely the whole dialog history. &#10;Some models rely on all previous user utterances and system responses at each turn, and others like UBAR and MTTOD make the generation at current turn additionally conditional on all previous dialog states, DB results, system acts.&#10;In both manners, the non-Markov models increase the memory and computation costs in both training and generation.&#10;Furthermore, using the whole history increase model complexity and may hurt the training efficiency, especially when facing small amounts of labeled training data (the low-resource setting).&#10;&#10;As briefly reviewed above, Markov models have been used in previous LSTM-based TOD systems \cite{wen2017a}-\cite{zhang-etal-2020-probabilistic}, but not studied for PLM-based TOD systems.&#10;In this paper, motivated by the observation that dialog states could be viewed as Markov states \cite{williams2016dialog,sutton2018reinforcement}, we propose to build {M}arkovian {G}enerative {A}rchitectures (MGA) over PLM backbones for efficient TOD systems, which could be based on GPT2 or T5 but use shortened history.&#10;We follow the classic definition of the dialog state, which is defined to compactly summarize the dialog history from the beginning up to current turn \cite{williams2016dialog}.&#10;It is also based on this definition that dialog states in widely-used dialog datasets such as MultiWOZ \cite{eric2019multiwoz} are annotated.&#10;We observe that dialog states defined in this way essentially corresponds to Markov states from modeling the dialog as a Markov decision process \cite{sutton2018reinforcement}.&#10;A shortened history, which consists of only last dialog state and system response along with current user utterance, can be supposed to provide enough context information for the agent to make prediction at current turn.&#10;Theoretically, if the dialog state is strictly a Markov state, it means the future is independent of earlier history given the dialog state and there is no information lost in using the shortened history for predicting the future.&#10;&#10;Remarkably, when comparing Markov and non-Markov models (both over PLM backbones) in practice, it is difficult to obtain general guarantees which one would be better.&#10;For Markov models, assuming that the dialog state has the Markov property is a model assumption, and in practice, may be only approximately satisfied.&#10;However, assuming that the non-Markov models, with increasing model complexity, can be well trained, under a given amount of samples, is also an assumption, and in practice, may be more easily over-fitted, particularly in the low-resource setting.&#10;So regarding the comparison of Markov and non-Markov models (both over PLM backbones), there are some interesting research questions, which will be empirically addressed in this paper. &#10;First, in the rich-resource setting, can Markov models perform close to non-Markov models? This roughly tests the Markov model assumption.&#10;Second, in the low-resource setting, can Markov models outperform non-Markov models? This compares the training efficiency of the two models.&#10;&#10;Extensive experiments are taken on the widely used MultiWOZ2.1 dataset \cite{eric2019multiwoz}.&#10;First, in the rich-resource setting, 100\% labeled data in MultiWOZ2.1 training set is used, and it is found that compared with existing non-Markov PLM-based systems, the proposed Markov PLM-based systems achieve equally strong performance, with significantly reduced computation and memory costs.&#10;Second, we consider the low-resource setting, where some proportions (10\% and 20\%) of the labeled dialogs from MultiWOZ2.1 training set are drawn.&#10;The rest dialogs in the training set are discarded for supervised-only training, or treated as unlabeled for semi-supervised training.&#10;The proposed Markov models achieve much better than their non-Markov counterparts in both low-resource settings, which demonstrates the superiority of the Markov models in training efficiency.&#10;Overall, these results give encouraging empirical evidences, which support the advantage of using Markovian architectures for PLM-based TOD systems.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Architecture, Efficient Deep Learning, Task-Oriented Dialog Systems, Mathematics, Information Science" />
        </attvalues>
      </node>
      <node id="1604.04562" label="1604.04562">
        <attvalues>
          <attvalue for="0" value="A Network-based End-to-End Trainable Task-oriented Dialogue System" />
          <attvalue for="1" value="  Teaching machines to accomplish tasks by conversing naturally with humans is&#10;challenging. Currently, developing task-oriented dialogue systems requires&#10;creating multiple components and typically this involves either a large amount&#10;of handcrafting, or acquiring costly labelled datasets to solve a statistical&#10;learning problem for each component. In this work we introduce a neural&#10;network-based text-in, text-out end-to-end trainable goal-oriented dialogue&#10;system along with a new way of collecting dialogue data based on a novel&#10;pipe-lined Wizard-of-Oz framework. This approach allows us to develop dialogue&#10;systems easily and without making too many assumptions about the task at hand.&#10;The results show that the model can converse with human subjects naturally&#10;whilst helping them to accomplish tasks in a restaurant search domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.08115" label="2009.08115">
        <attvalues>
          <attvalue for="0" value="A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief&#10;  States towards Semi-Supervised Learning" />
          <attvalue for="1" value="  Structured belief states are crucial for user goal tracking and database&#10;query in task-oriented dialog systems. However, training belief trackers often&#10;requires expensive turn-level annotations of every user utterance. In this&#10;paper we aim at alleviating the reliance on belief state labels in building&#10;end-to-end dialog systems, by leveraging unlabeled dialog data towards&#10;semi-supervised learning. We propose a probabilistic dialog model, called the&#10;LAtent BElief State (LABES) model, where belief states are represented as&#10;discrete latent variables and jointly modeled with system responses given user&#10;inputs. Such latent variable modeling enables us to develop semi-supervised&#10;learning under the principled variational learning framework. Furthermore, we&#10;introduce LABES-S2S, which is a copy-augmented Seq2Seq model instantiation of&#10;LABES. In supervised experiments, LABES-S2S obtains strong results on three&#10;benchmark datasets of different scales. In utilizing unlabeled dialog data,&#10;semi-supervised LABES-S2S significantly outperforms both supervised-only and&#10;semi-supervised baselines. Remarkably, we can reduce the annotation demands to&#10;50% without performance loss on MultiWOZ.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.14739" label="2109.14739">
        <attvalues>
          <attvalue for="0" value="Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System" />
          <attvalue for="1" value="  Pre-trained language models have been recently shown to benefit task-oriented&#10;dialogue (TOD) systems. Despite their success, existing methods often formulate&#10;this task as a cascaded generation problem which can lead to error accumulation&#10;across different sub-tasks and greater data annotation overhead. In this study,&#10;we present PPTOD, a unified plug-and-play model for task-oriented dialogue. In&#10;addition, we introduce a new dialogue multi-task pre-training strategy that&#10;allows the model to learn the primary TOD task completion skills from&#10;heterogeneous dialog corpora. We extensively test our model on three benchmark&#10;TOD tasks, including end-to-end dialogue modelling, dialogue state tracking,&#10;and intent classification. Experimental results show that PPTOD achieves new&#10;state of the art on all evaluated tasks in both high-resource and low-resource&#10;scenarios. Furthermore, comparisons against previous SOTA methods show that the&#10;responses generated by PPTOD are more factually correct and semantically&#10;coherent as judged by human annotators.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Task-Oriented Dialogue. Task-oriented dialogue aims at accomplishing user's goal. Traditional systems \cite{DBLP:journals/csl/WilliamsY07,DBLP:journals/pieee/YoungGTW13} adopt a pipelined approach that requires dialogue state tracking for understanding user's goal, dialogue policy learning for deciding which system action to take, and natural language generation for generating dialogue responses. &#10;&#10;Recently, to simplify the modelling effort, researchers have shifted their attention to building neural network models that address the TOD sub-tasks \cite{wen-etal-2017-network,eric-etal-2017-key,lei-etal-2018-sequicity,DBLP:conf/aaai/LiangTCY20}. &#10;With the advances in pre-trained language models (PLMs), \cite{budzianowski-vulic-2019-hello} first applied the GPT-2 model for the NLG task. \cite{DBLP:conf/emnlp/LinMWF20} and \cite{DBLP:conf/aaai/YangLQ21} moved one step forward and utilized pre-trained language models to solve all TOD sub-tasks conditioned on the history of oracle belief states. Based on the GPT-2 model, \cite{DBLP:conf/nips/Hosseini-AslMWY20} proposed a cascaded model, SimpleTOD, that addresses all TOD sub-tasks without using the oracle information. To improve the system performance, \cite{peng2021soloist} and \cite{DBLP:journals/tacl/LiuYRB21} applied dialogue pre-training over external dialogue corpora. However, both methods require the pre-training data to be fully annotated for all TOD sub-tasks (i.e., DST, POL, and NLG) which greatly limits the amount of data they can use. Additionally, \cite{DBLP:journals/tacl/LiuYRB21} achieved better results with noisy chanel model that requires two additional language models for outputs re-scoring. Unlike their approach, we address the task of task-oriented dialogue with a single unified model. Lastly, concurrent work by \cite{DBLP:journals/corr/abs-2111-14592} shows that adding an unified dialogue act prediction task for policy optimization helps to improve the performance of the pre-trained task-oriented dialogue model.&#10;&#10;Language Model Pre-training. The research community has witnessed remarkable progress of pre-training methods in a wide range&#10;of NLP tasks, including language understanding&#10;\cite{DBLP:conf/naacl/PetersNIGCLZ18,DBLP:conf/naacl/DevlinCLT19,DBLP:journals/corr/abs-1907-11692,DBLP:conf/nips/YangDYCSL19,DBLP:journals/corr/abs-2111-04198} and text generation \cite{radford2019language,DBLP:conf/acl/LewisLGGMLSZ20,DBLP:journals/jmlr/RaffelSRLNMZLL20,DBLP:journals/taslp/SuWCBKC21,DBLP:conf/emnlp/SuVWFC21,DBLP:conf/emnlp/SuMBC21,DBLP:journals/corr/abs-2202-06417}.&#10;&#10;In the dialogue domain, many models are pre-trained on open-domain conversational data like Reddit. Based on GPT-2, Transfertransfo \cite{DBLP:journals/corr/abs-1901-08149} achieves good results on ConvAI-2 competition. As another extension of GPT-2, DialoGPT \cite{DBLP:conf/acl/ZhangSGCBGGLD20} performs well in generating open-domain dialogue response. ConveRT \cite{DBLP:conf/emnlp/HendersonCMSWV20} is a language model with dual-encoder built for the task of response selection. PLATO \cite{DBLP:conf/acl/BaoHWWW20} pre-trains a model with discrete latent variable structure for the response generation task. \cite{wu-etal-2020-tod} adapts BERT with TOD pre-training and achieves strong performances on four dialogue understanding tasks. &#10;&#10;Pre-training on Supplementary Data. Recent work \cite{phang2018stilts,DBLP:armen2021muppet} found that supplementary training on the tasks with intermediate-labelled data improves the performance of the fine-tuned models on GLUE natural language understanding benchmark~\cite{DBLP:wang2019glue}. Our work studies a similar supplementary training setup with intermediate-labelled data for task-oriented dialogue systems. Unlike previous work, we use a single multi-task model for all relevant sub-tasks in task-oriented dialogue systems.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2310.06547" label="2310.06547">
        <attvalues>
          <attvalue for="0" value="Rationale-Enhanced Language Models are Better Continual Relation&#10;  Learners" />
          <attvalue for="1" value="  Continual relation extraction (CRE) aims to solve the problem of catastrophic&#10;forgetting when learning a sequence of newly emerging relations. Recent CRE&#10;studies have found that catastrophic forgetting arises from the model's lack of&#10;robustness against future analogous relations. To address the issue, we&#10;introduce rationale, i.e., the explanations of relation classification results&#10;generated by large language models (LLM), into CRE task. Specifically, we&#10;design the multi-task rationale tuning strategy to help the model learn current&#10;relations robustly. We also conduct contrastive rationale replay to further&#10;distinguish analogous relations. Experimental results on two standard&#10;benchmarks demonstrate that our method outperforms the state-of-the-art CRE&#10;models.&#10;" />
          <attvalue for="2" value="&#10;Relation extraction (RE) aims to identify the relations between two entities in a text. &#10;While traditional RE models cannot handle the real-life situation where new relations are constantly emerging,&#10;continual relation extraction (CRE) &#10;attempts to learn new relations while retaining the performance on learned relations&#10;\cite{han-etal-2020-continual, cui-etal-2021-refining, zhao-etal-2022-consistent}.&#10;Similar to other continual learning tasks, the main challenge in CRE is the phenomenon of catastrophic forgetting (CF), i.e., the performance on identifying old relations degrades significantly while learning new relations.&#10;&#10;Most previous CRE researches have attributed catastrophic forgetting to the destruction of representations learned on previous tasks when learning new tasks \cite{han-etal-2020-continual, cui-etal-2021-refining, wang2022more}. &#10;They focused on recovering the representations on previous tasks, using methods like restricted gradient updating and knowledge distillation \cite{lopezpaz2017gradient, cao-etal-2020-incremental}.&#10;Recently, another line of work \cite{wang-etal-2022-learning-robust} found that in CRE scenario, models trained for the current task do not have good identification ability for new-coming relations which are analogous to a current relation.&#10;Thus, making a model learn current relations robustly to avoid subsequent confusion becomes the new research focus \cite{wang-etal-2022-learning-robust, zhao2023improving}.&#10;&#10;To address this problem, in this paper,&#10;we assume that incorporating rationales can enhance the performance of CRE models in learning analogous relations.&#10;This is inspired by the intuition that, training models with explicit rationale supervision can provide greater robustness \cite{chen-etal-2022-rationalization}.&#10;Moreover, &#10;since relation extraction requires reasoning over two entities,&#10;providing explanations for why the two entities have a specific relation can enhance the reasoning capacity of smaller models, thereby eliminating reliance on spurious shortcuts \cite{li2022explanations, magister2023teaching}.&#10;As there are no such corpus which labels the entity relations with corresponding rationales, &#10;we propose to make use of LLM to generate the explanations, i.e., the rationales, for the relation classification answers with prompting questions, as shown in Figure \ref{fig:brationale}.&#10;&#10;To fully exploit rationales, we propose a novel approach called \RationaleCL, which incorporates two strategies: multi-task rationale tuning and contrastive rationale replay, into a rehearsal-based framework, as shown in Figure \ref{fig:task formation}.&#10;Specifically, we employ the encoder-decoder model T5 \cite{raffel2020exploring} as our backbone, utilizing multi-task rationale tuning with three tasks: question to answer as the main task, question to rationale-answer and question-rationale to answer as auxiliary tasks. &#10;With the rationale tuning strategy, we distill the rationale knowledge from LLM to make T5 develop the reasoning ability to interpret its classification results, leading to enhanced robustness for the CRE task.&#10;When conducting memory rehearsal, &#10;we prompt LLM to differentiate between analogous relations and regenerate the corresponding explanations, i.e., the contrastive rationales (Figure \ref{fig:brationale}), to update the memory. which not only helps mitigate catastrophic forgetting but prevents confusion.&#10;&#10;Our contributions are summarized as follows: &#10;(1) For the first time, we introduce rationale generated by LLM into CRE task to mitigate catastrophic forgetting.&#10;(2) We propose a novel rationale-enhanced CRE method \RationaleCL, which incorporates multi-task rationale tuning and contrastive rationale replay strategies.&#10;(3) Experimental results on FewRel and TACRED verify the effectiveness of our method.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Models, Continual Learning, Artificial Intelligence, Natural Language Processing, Relation Extraction" />
        </attvalues>
      </node>
      <node id="1706.08840" label="1706.08840">
        <attvalues>
          <attvalue for="0" value="Gradient Episodic Memory for Continual Learning" />
          <attvalue for="1" value="  One major obstacle towards AI is the poor ability of models to solve new&#10;problems quicker, and without forgetting previously acquired knowledge. To&#10;better understand this issue, we study the problem of continual learning, where&#10;the model observes, once and one by one, examples concerning a sequence of&#10;tasks. First, we propose a set of metrics to evaluate models learning over a&#10;continuum of data. These metrics characterize models not only by their test&#10;accuracy, but also in terms of their ability to transfer knowledge across&#10;tasks. Second, we propose a model for continual learning, called Gradient&#10;Episodic Memory (GEM) that alleviates forgetting, while allowing beneficial&#10;transfer of knowledge to previous tasks. Our experiments on variants of the&#10;MNIST and CIFAR-100 datasets demonstrate the strong performance of GEM when&#10;compared to the state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.04497" label="2210.04497">
        <attvalues>
          <attvalue for="0" value="Learning Robust Representations for Continual Relation Extraction via&#10;  Adversarial Class Augmentation" />
          <attvalue for="1" value="  Continual relation extraction (CRE) aims to continually learn new relations&#10;from a class-incremental data stream. CRE model usually suffers from&#10;catastrophic forgetting problem, i.e., the performance of old relations&#10;seriously degrades when the model learns new relations. Most previous work&#10;attributes catastrophic forgetting to the corruption of the learned&#10;representations as new relations come, with an implicit assumption that the CRE&#10;models have adequately learned the old relations. In this paper, through&#10;empirical studies we argue that this assumption may not hold, and an important&#10;reason for catastrophic forgetting is that the learned representations do not&#10;have good robustness against the appearance of analogous relations in the&#10;subsequent learning process. To address this issue, we encourage the model to&#10;learn more precise and robust representations through a simple yet effective&#10;adversarial class augmentation mechanism (ACA), which is easy to implement and&#10;model-agnostic. Experimental results show that ACA can consistently improve the&#10;performance of state-of-the-art CRE models on two popular benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.11790" label="2204.11790">
        <attvalues>
          <attvalue for="0" value="Can Rationalization Improve Robustness?" />
          <attvalue for="1" value="  A growing line of work has investigated the development of neural NLP models&#10;that can produce rationales--subsets of input that can explain their model&#10;predictions. In this paper, we ask whether such rationale models can also&#10;provide robustness to adversarial attacks in addition to their interpretable&#10;nature. Since these models need to first generate rationales (&quot;rationalizer&quot;)&#10;before making predictions (&quot;predictor&quot;), they have the potential to ignore&#10;noise or adversarially added text by simply masking it out of the generated&#10;rationale. To this end, we systematically generate various types of 'AddText'&#10;attacks for both token and sentence-level rationalization tasks, and perform an&#10;extensive empirical evaluation of state-of-the-art rationale models across five&#10;different tasks. Our experiments reveal that the rationale models show the&#10;promise to improve robustness, while they struggle in certain scenarios--when&#10;the rationalizer is sensitive to positional bias or lexical choices of attack&#10;text. Further, leveraging human rationale as supervision does not always&#10;translate to better performance. Our study is a first step towards exploring&#10;the interplay between interpretability and robustness in the&#10;rationalize-then-predict framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.07414" label="2010.07414">
        <attvalues>
          <attvalue for="0" value="On Cross-Dataset Generalization in Automatic Detection of Online Abuse" />
          <attvalue for="1" value="  NLP research has attained high performances in abusive language detection as&#10;a supervised classification task. While in research settings, training and test&#10;datasets are usually obtained from similar data samples, in practice systems&#10;are often applied on data that are different from the training set in topic and&#10;class distributions. Also, the ambiguity in class definitions inherited in this&#10;task aggravates the discrepancies between source and target datasets. We&#10;explore the topic bias and the task formulation bias in cross-dataset&#10;generalization. We show that the benign examples in the Wikipedia Detox dataset&#10;are biased towards platform-specific topics. We identify these examples using&#10;unsupervised topic modeling and manual inspection of topics' keywords. Removing&#10;these topics increases cross-dataset generalization, without reducing in-domain&#10;classification performance. For a robust dataset design, we suggest applying&#10;inexpensive unsupervised methods to inspect the collected data and downsize the&#10;non-generalizable content before manually annotating for class labels.&#10;" />
          <attvalue for="2" value="&#10;&#10;The NLP research community has devoted significant efforts to support the safety and inclusiveness of online discussion forums by developing automatic systems to detect hurtful, derogatory or obscene utterances. &#10;Most of these systems are based on supervised machine learning techniques, and require annotated data. &#10;Several publicly available datasets have been created for the task \cite{mishra2019tackling,vidgen2020directions}. &#10;However, due to the ambiguities in the task definition and complexities of data collection, cross-dataset generalizability remains a challenging and under-studied issue of online abuse detection. &#10;&#10;Existing datasets differ in the considered types of offensive behaviour and annotation schemes, data sources and data collection methods. &#10;There is no agreed-upon definition of harmful online behaviour yet. &#10;Several terms have been used to refer to the general concept of harmful online behavior, including toxicity \cite{hosseini2017deceiving}, hate speech \cite{schmidt2017survey}, offensive \cite{zampieri2019semeval} and abusive language \cite{waseem2017understanding,vidgen2019challenges}. &#10;Still, in practice, every dataset only focuses on a narrow range of sub-types of such behaviours and a single online platform \cite{jurgens2019just}. &#10;For example, \cite{davidson2017automated} annotated tweets for three categories, Racist, Offensive but not Racist and Clean, and \cite{nobata2016abusive} collected discussions from Yahoo! Finance news and applied a binary annotation scheme of Abusive versus Clean. &#10;Further, since pure random sampling usually results in small proportions of offensive examples \cite{founta2018large}, various sampling techniques are often employed. &#10;\cite{zampieri2019semeval} used words and phrases frequently found in offensive messages to search for potential abusive tweets. \cite{founta2018large} and \cite{razavi2010offensive} started from random sampling, then boosted the abusive part of the datasets using specific search procedures. &#10;\cite{hosseinmardi2015analyzing} used snowballing to collect abusive posts on Instagram. &#10;Due to this variability in category definitions and data collection techniques, a system trained on a particular dataset is prone to overfitting to the specific characteristics of that dataset. &#10;As a result, although models tend to perform well in cross-validation evaluation on one dataset, the cross-dataset generalizability remains low \cite{van2018challenges,wiegand-etal-2019-detection}. &#10;&#10;In this work, we investigate the impact of two types of biases originating from source data that can emerge in a cross-domain application of models: 1) task formulation bias (discrepancy in class definitions and annotation between the training and test sets) and 2) selection bias (discrepancy in the topic and class distributions between the training and test sets). &#10;Further, we suggest topic-based dataset pruning as a method of mitigating selection bias to increase generalizability. This approach is different from domain adaptation techniques based on data selection \cite{ruder2017learning,liu2019reinforced} in that we apply an unsupervised topic modeling method for topic discovery without using the class labels. We show that some topics are more generalizable than others. &#10;The topics that are specific to the training dataset lead to overfitting and, therefore, lower generalizability. &#10;Excluding or down-sampling instances associated with such topics before the expensive annotation step can substantially reduce the annotation costs.&#10;&#10;We focus on the Wikipedia Detox or Wiki-dataset, (an extension of the dataset by \cite{wulczyn2017ex}), collected from English Wikipedia talk pages and annotated for toxicity. To explore the generalizability of the models trained on this dataset, &#10;we create an out-of-domain test set comprising various types of abusive behaviours by combining two existing datasets, namely Waseem-dataset \cite{waseem2016hateful} and Founta-dataset \cite{founta2018large}, both collected from Twitter.&#10;&#10;Our main contributions are as follows:&#10;\begin{itemize}[leftmargin=*]&#10;&#10;\item We identify topics included in the Wiki-dataset and manually examine keywords associated with the topics to heuristically determine topics' generalizability and their potential association with toxicity. &#10;&#10;\item We assess the generalizability of the task formulations by training a classifier to detect the Toxic class in the Wiki-dataset and testing it on an out-of-domain dataset comprising various types of offensive behaviours. We find that Wiki-Toxic is most generalizable to Founta-Abusive and least generalizable to Waseem-Sexism.&#10;&#10;\item We show that re-sampling techniques result in a trade-off between the True Positive and True Negative rates on the out-of-domain test set. This trade-off is mainly governed by the ratio of toxic to normal instances and not the size of the dataset. &#10;&#10;\item We investigate the impact of topic distribution on generalizability and show that general and identity-related topics are more generalizable than platform-specific topics. &#10;&#10;\item We show that excluding Wikipedia-specific data instances (54\% of the dataset) does not affect the results of in-domain classification, and improves both True Positive and True Negative rates on the out-of-domain test set, unlike re-sampling methods. Through unsupervised topic modeling, such topics can be identified and excluded before annotation. The pruned version of the Wikipedia dataset is available at.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Dataset Design Optimization, Machine Learning, Computer Science, Cross-Dataset Generalization, Linguistics, Data Science, Abusive Language Detection, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1908.06024" label="1908.06024">
        <attvalues>
          <attvalue for="0" value="Tackling Online Abuse: A Survey of Automated Abuse Detection Methods" />
          <attvalue for="1" value="  Abuse on the Internet represents an important societal problem of our time.&#10;Millions of Internet users face harassment, racism, personal attacks, and other&#10;types of abuse on online platforms. The psychological effects of such abuse on&#10;individuals can be profound and lasting. Consequently, over the past few years,&#10;there has been a substantial research effort towards automated abuse detection&#10;in the field of natural language processing (NLP). In this paper, we present a&#10;comprehensive survey of the methods that have been proposed to date, thus&#10;providing a platform for further development of this area. We describe the&#10;existing datasets and review the computational approaches to abuse detection,&#10;analyzing their strengths and limitations. We discuss the main trends that&#10;emerge, highlight the challenges that remain, outline possible solutions, and&#10;propose guidelines for ethics and explainability&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.01670" label="2004.01670">
        <attvalues>
          <attvalue for="0" value="Directions in Abusive Language Training Data: Garbage In, Garbage Out" />
          <attvalue for="1" value="  Data-driven analysis and detection of abusive online content covers many&#10;different tasks, phenomena, contexts, and methodologies. This paper&#10;systematically reviews abusive language dataset creation and content in&#10;conjunction with an open website for cataloguing abusive language data. This&#10;collection of knowledge leads to a synthesis providing evidence-based&#10;recommendations for practitioners working with this complex and highly diverse&#10;data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1702.08138" label="1702.08138">
        <attvalues>
          <attvalue for="0" value="Deceiving Google's Perspective API Built for Detecting Toxic Comments" />
          <attvalue for="1" value="  Social media platforms provide an environment where people can freely engage&#10;in discussions. Unfortunately, they also enable several problems, such as&#10;online harassment. Recently, Google and Jigsaw started a project called&#10;Perspective, which uses machine learning to automatically detect toxic&#10;language. A demonstration website has been also launched, which allows anyone&#10;to type a phrase in the interface and instantaneously see the toxicity score&#10;[1]. In this paper, we propose an attack on the Perspective toxic detection&#10;system based on the adversarial examples. We show that an adversary can subtly&#10;modify a highly toxic phrase in a way that the system assigns significantly&#10;lower toxicity score to it. We apply the attack on the sample phrases provided&#10;in the Perspective website and show that we can consistently reduce the&#10;toxicity scores to the level of the non-toxic phrases. The existence of such&#10;adversarial examples is very harmful for toxic detection systems and seriously&#10;undermines their usability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.08983" label="1903.08983">
        <attvalues>
          <attvalue for="0" value="SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in&#10;  Social Media (OffensEval)" />
          <attvalue for="1" value="  We present the results and the main findings of SemEval-2019 Task 6 on&#10;Identifying and Categorizing Offensive Language in Social Media (OffensEval).&#10;The task was based on a new dataset, the Offensive Language Identification&#10;Dataset (OLID), which contains over 14,000 English tweets. It featured three&#10;sub-tasks. In sub-task A, the goal was to discriminate between offensive and&#10;non-offensive posts. In sub-task B, the focus was on the type of offensive&#10;content in the post. Finally, in sub-task C, systems had to detect the target&#10;of the offensive posts. OffensEval attracted a large number of participants and&#10;it was one of the most popular tasks in SemEval-2019. In total, about 800 teams&#10;signed up to participate in the task, and 115 of them submitted results, which&#10;we present and analyze in this report.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.09899" label="1705.09899">
        <attvalues>
          <attvalue for="0" value="Understanding Abuse: A Typology of Abusive Language Detection Subtasks" />
          <attvalue for="1" value="  As the body of research on abusive language detection and analysis grows,&#10;there is a need for critical consideration of the relationships between&#10;different subtasks that have been grouped under this label. Based on work on&#10;hate speech, cyberbullying, and online abuse we propose a typology that&#10;captures central similarities and differences between subtasks and we discuss&#10;its implications for data annotation and feature construction. We emphasize the&#10;practical actions that can be taken by researchers to best approach their&#10;abusive language detection subtask of interest.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01738" label="1906.01738">
        <attvalues>
          <attvalue for="0" value="A Just and Comprehensive Strategy for Using NLP to Address Online Abuse" />
          <attvalue for="1" value="  Online abusive behavior affects millions and the NLP community has attempted&#10;to mitigate this problem by developing technologies to detect abuse. However,&#10;current methods have largely focused on a narrow definition of abuse to&#10;detriment of victims who seek both validation and solutions. In this position&#10;paper, we argue that the community needs to make three substantive changes: (1)&#10;expanding our scope of problems to tackle both more subtle and more serious&#10;forms of abuse, (2) developing proactive technologies that counter or inhibit&#10;abuse before it harms, and (3) reframing our effort within a framework of&#10;justice to promote healthy communities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.07572" label="1809.07572">
        <attvalues>
          <attvalue for="0" value="Challenges for Toxic Comment Classification: An In-Depth Error Analysis" />
          <attvalue for="1" value="  Toxic comment classification has become an active research field with many&#10;recently proposed approaches. However, while these approaches address some of&#10;the task's challenges others still remain unsolved and directions for further&#10;research are needed. To this end, we compare different deep learning and&#10;shallow approaches on a new, large comment dataset and propose an ensemble that&#10;outperforms all individual models. Further, we validate our findings on a&#10;second dataset. The results of the ensemble enable us to perform an extensive&#10;error analysis, which reveals open challenges for state-of-the-art methods and&#10;directions towards pending future research. These challenges include missing&#10;paradigmatic context and inconsistent dataset labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.05246" label="1707.05246">
        <attvalues>
          <attvalue for="0" value="Learning to select data for transfer learning with Bayesian Optimization" />
          <attvalue for="1" value="  Domain similarity measures can be used to gauge adaptability and select&#10;suitable data for transfer learning, but existing approaches define ad hoc&#10;measures that are deemed suitable for respective tasks. Inspired by work on&#10;curriculum learning, we propose to \emph{learn} data selection measures using&#10;Bayesian Optimization and evaluate them across models, domains and tasks. Our&#10;learned measures outperform existing domain similarity measures significantly&#10;on three tasks: sentiment analysis, part-of-speech tagging, and parsing. We&#10;show the importance of complementing similarity with diversity, and that&#10;learned measures are -- to some degree -- transferable across models, domains,&#10;and even tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.08914" label="1610.08914">
        <attvalues>
          <attvalue for="0" value="Ex Machina: Personal Attacks Seen at Scale" />
          <attvalue for="1" value="  The damage personal attacks cause to online discourse motivates many&#10;platforms to try to curb the phenomenon. However, understanding the prevalence&#10;and impact of personal attacks in online platforms at scale remains&#10;surprisingly difficult. The contribution of this paper is to develop and&#10;illustrate a method that combines crowdsourcing and machine learning to analyze&#10;personal attacks at scale. We show an evaluation method for a classifier in&#10;terms of the aggregated number of crowd-workers it can approximate. We apply&#10;our methodology to English Wikipedia, generating a corpus of over 100k high&#10;quality human-labeled comments and 63M machine-labeled ones from a classifier&#10;that is as good as the aggregate of 3 crowd-workers, as measured by the area&#10;under the ROC curve and Spearman correlation. Using this corpus of&#10;machine-labeled scores, our methodology allows us to explore some of the open&#10;questions about the nature of online personal attacks. This reveals that the&#10;majority of personal attacks on Wikipedia are not the result of a few malicious&#10;users, nor primarily the consequence of allowing anonymous contributions from&#10;unregistered users.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.01582" label="2405.01582">
        <attvalues>
          <attvalue for="0" value="Text Quality-Based Pruning for Efficient Training of Language Models" />
          <attvalue for="1" value="  In recent times training Language Models (LMs) have relied on computationally&#10;heavy training over massive datasets which makes this training process&#10;extremely laborious. In this paper we propose a novel method for numerically&#10;evaluating text quality in large unlabelled NLP datasets in a model agnostic&#10;manner to assign the text instances a &quot;quality score&quot;.&#10;  By proposing the text quality metric, the paper establishes a framework to&#10;identify and eliminate low-quality text instances, leading to improved training&#10;efficiency for LM models. Experimental results over multiple models and&#10;datasets demonstrate the efficacy of this approach, showcasing substantial&#10;gains in training effectiveness and highlighting the potential for&#10;resource-efficient LM training.&#10;  For example, we observe an absolute accuracy improvement of 0.9% averaged&#10;over 14 downstream evaluation tasks for multiple LM models while using 40%&#10;lesser data and training 42% faster when training on the OpenWebText dataset&#10;and 0.8% average absolute accuracy improvement while using 20% lesser data and&#10;training 21% faster on the Wikipedia dataset.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language Models (LMs) have gained significant attention in recent years due to their impressive performance in various natural language processing (NLP) tasks \cite{opt, falcon, llama, lima, roberta}. However, their training process often relies on computationally intensive procedures that involve massive datasets and compute requirements which hinders training large scale LMs on noisy real-world or domain specific datasets. What's worse is that several of these datasets are uncurated and may contain harmful content which the LM model can potentially pick up during the training process \cite{bias1, bias2, bias3}.&#10;&#10;Text quality evaluation plays a crucial role in assessing the suitability and reliability of textual data for training LMs. Previous research has explored various approaches for text quality assessment, primarily focusing on human annotation and subjective judgments. For instance, \cite{humaneval} introduce a crowdsourcing-based method for ranking text quality, where human evaluators provide subjective ratings. While such approaches provide valuable insights, they suffer from scalability limitations and subjectivity biases.&#10;To overcome these limitations, more recent works have explored the use of automated approaches to quality evaluation such as making use of ChatGPT or GPT-4 to evaluate the quality of the text, where text is designated to be high quality if ChatGPT/GPT-4 deems it to be similar to human text \cite{chatgpteval, gpt4eval}. However, these methods are model dependent and requires training massive LLM models, which defeats the purpose of efficient LM training.&#10;&#10;We address this issue by proposing a novel method for numerically evaluating text quality in large unlabelled NLP datasets, with the aim of improving LM training performance and efficiency. We also ensure that our text quality metric is model agnostic, helping us avoid having to recompute these quality metrics for each model. By leveraging this numerical text quality score, we demonstrate how it can be used to prune the original dataset, enabling the training of LMs using only a fraction of the data. Our approach aims to identify and eliminate low-quality text instances, thereby streamlining the training process and mitigating the burden of handling large-scale datasets. We also remove potentially harmful content from the data by ensuring that harmful content is rated poorly by our text quality score which can then be pruned. We observe an absolute improvement of 0.9\% averaged over 14 downstream evaluation tasks for multiple LM models while using 40\% lesser data and training 42\% faster when training on the OpenWebText dataset \cite{openwebtext} and a 0.8\% absolute improvement averaged over 3 models and 14 downstream tasks for the Wikipedia dataset \cite{wikipedia} when using 20\% lesser data and training time.&#10;&#10;The key contribution of this paper lies in establishing a framework that quantitatively evaluates text quality in a model agnostic manner and subsequently guides the pruning of NLP datasets for LM training. By leveraging this quality score metric, we enable a more efficient allocation of computational resources and reduce the data requirements for training LMs. This approach not only expedites the training process but also enhances the overall effectiveness of the models. To the best of our knowledge, there doesn't exist an objective way to evaluate the quality of large scale textual datasets and we hope this work will pave the way for further work in this space.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Training, Text Quality Evaluation, Artificial Intelligence, Mathematics, Natural Language Processing, Efficient NLP Processing" />
        </attvalues>
      </node>
      <node id="2107.00061" label="2107.00061">
        <attvalues>
          <attvalue for="0" value="All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated&#10;  Text" />
          <attvalue for="1" value="  Human evaluations are typically considered the gold standard in natural&#10;language generation, but as models' fluency improves, how well can evaluators&#10;detect and judge machine-generated text? We run a study assessing non-experts'&#10;ability to distinguish between human- and machine-authored text (GPT2 and GPT3)&#10;in three domains (stories, news articles, and recipes). We find that, without&#10;training, evaluators distinguished between GPT3- and human-authored text at&#10;random chance level. We explore three approaches for quickly training&#10;evaluators to better identify GPT3-authored text (detailed instructions,&#10;annotated examples, and paired examples) and find that while evaluators'&#10;accuracy improved up to 55%, it did not significantly improve across the three&#10;domains. Given the inconsistent results across text domains and the often&#10;contradictory reasons evaluators gave for their judgments, we examine the role&#10;untrained human evaluations play in NLG evaluation and provide recommendations&#10;to NLG researchers for improving human evaluations of text generated from&#10;state-of-the-art models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;A subfield of NLG analyzes the role of human evaluations, including discussions of the tradeoffs of human and automatic evaluations \cite{belz-reiter-2006-comparing, hashimoto-etal-2019-unifying}.&#10;There are critiques and recommendations for different aspects of human evaluations, like the evaluation design \cite{novikova-etal-2018-rankme, santhanam-shaikh-2019-towards}, question framing \cite{schoch-etal-2020-problem}, and evaluation measures like agreement \cite{amidei-etal-2018-rethinking}, as well as analyses of past NLG papers' human evaluations \cite{vanderlee_journal, howcroft-etal-2020-twenty}.&#10;Additionally, crowdsourcing literature has work on effectively using platforms like Amazon Mechanical Turk \cite[e.g.,][]{florian_crowdsourcing,oppenheimer_crowdsourcing,weld_crowdsourcing,mitra_crowdsourcing}.&#10;In this work, we focus on the role evaluator training can play for producing better accuracy at distinguishing human- and machine-generated text, though other quality control methods are worth exploring.&#10;&#10;Previous work has asked evaluators to distinguish between human- and machine-authored text. For example, \cite{ippolito-etal-2020-automatic} found that trained evaluators were able to detect open-ended GPT2-L-generated text 71.4\% of the time, \cite{garbacea-etal-2019-judge} reported that individual evaluators guessed correctly 66.6\% of the time when evaluating product reviews, and \cite{gpt3} found evaluators could guess GPT3-davinci-generated news articles' source with 52\% accuracy, though these results are not directly comparable to ours due to differences in the evaluation setup, data, and participants.&#10;&#10;Finally, our findings that untrained evaluators are not well equipped to detect machine-generated text point to the importance of researching the safe deployment of NLG systems. \cite{gehrmann-etal-2019-gltr} proposed visualization techniques to help readers detect generated text, and work like \cite{grover_zellers}, \cite{ippolito-etal-2020-automatic}, and \cite{uchendu-etal-2020-authorship} investigated large language models' ability to detect generated text.&#10; " />
        </attvalues>
      </node>
      <node id="2303.15056" label="2303.15056">
        <attvalues>
          <attvalue for="0" value="ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks" />
          <attvalue for="1" value="  Many NLP applications require manual data annotations for a variety of tasks,&#10;notably to train classifiers or evaluate the performance of unsupervised&#10;models. Depending on the size and degree of complexity, the tasks may be&#10;conducted by crowd-workers on platforms such as MTurk as well as trained&#10;annotators, such as research assistants. Using a sample of 2,382 tweets, we&#10;demonstrate that ChatGPT outperforms crowd-workers for several annotation&#10;tasks, including relevance, stance, topics, and frames detection. Specifically,&#10;the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of&#10;five tasks, while ChatGPT's intercoder agreement exceeds that of both&#10;crowd-workers and trained annotators for all tasks. Moreover, the&#10;per-annotation cost of ChatGPT is less than $0.003 -- about twenty times&#10;cheaper than MTurk. These results show the potential of large language models&#10;to drastically increase the efficiency of text classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.00907" label="1811.00907">
        <attvalues>
          <attvalue for="0" value="Importance of Search and Evaluation Strategies in Neural Dialogue&#10;  Modeling" />
          <attvalue for="1" value="  We investigate the impact of search strategies in neural dialogue modeling.&#10;We first compare two standard search algorithms, greedy and beam search, as&#10;well as our newly proposed iterative beam search which produces a more diverse&#10;set of candidate responses. We evaluate these strategies in realistic full&#10;conversations with humans and propose a model-based Bayesian calibration to&#10;address annotator bias. These conversations are analyzed using two automatic&#10;metrics: log-probabilities assigned by the model and utterance diversity. Our&#10;experiments reveal that better search algorithms lead to higher rated&#10;conversations. However, finding the optimal selection mechanism to choose from&#10;a more diverse set of candidates is still an open question.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Search Algorithm Optimization, Computer Science, Neural Dialogue Modeling, Linguistics, Human Conversation Analysis, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2112.12356" label="2112.12356">
        <attvalues>
          <attvalue for="0" value="Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token&#10;  Attributions in Different Languages?" />
          <attvalue for="1" value="  During the past several years, a surge of multi-lingual Pre-trained Language&#10;Models (PLMs) has been proposed to achieve state-of-the-art performance in many&#10;cross-lingual downstream tasks. However, the understanding of why multi-lingual&#10;PLMs perform well is still an open domain. For example, it is unclear whether&#10;multi-Lingual PLMs reveal consistent token attributions in different languages.&#10;To address this, in this paper, we propose a Cross-lingual Consistency of Token&#10;Attributions (CCTA) evaluation framework. Extensive experiments in three&#10;downstream tasks demonstrate that multi-lingual PLMs assign significantly&#10;different attributions to multi-lingual synonyms. Moreover, we have the&#10;following observations: 1) the Spanish achieves the most consistent token&#10;attributions in different languages when it is used for training PLMs; 2) the&#10;consistency of token attributions strongly correlates with performance in&#10;downstream tasks.&#10;" />
          <attvalue for="2" value="&#10; The cross-lingual zero-shot transfer is a fundamental task in the NLP domain to overcome language barriers, whose goal is to transfer model information trained from source/high-resource languages (i.e. English) to target/low-resource languages (i.e. Hindi) in the absence of explicit supervision. Multi-lingual Pre-trained Language Models (PLMs) such as multi-lingual BERT (mBERT) \cite{pires2019multilingual}, XLM \cite{conneau2019cross} and XLM-Roberta (XLM-R) \cite{conneau2020unsupervised}, have demonstrated superior performance in many cross-lingual zero-shot downstream tasks such as natural language inference and question answering. \\&#10; However, the understanding why multi-lingual PLMs perform surprisingly well is still an open domain. Previous works have investigated them extensively in various aspects. For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}. Another line of related work is to understand the multi-lingual model representation in the parallel corpus \cite{kudugunta2019investigating}. They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}.\\ Even though existing literature has made much progress on the interpretation of multi-lingual PLMs, to the best of our knowledge, there still lacks an investigation on the attribution (i.e. importance) of multi-lingual tokens to the predictions of PLMs in the downstream tasks. This facilitates the understanding of how multi-lingual PLMs distinguish important tokens from others trained in source languages, and whether the understanding of tokens can be transferred to target languages. In this paper, we explore the following question in the downstream tasks, which require parallel texts (i.e. texts placed alongside their translations): Do multi-lingual PLMs reveal consistent token attributions in different languages? To address this, we propose a Consistency of Token Attributions (CCTA) evaluation framework. This is different from isomorphism frameworks from previous works, since they focus on the representation of tokens (i.e. token embeddings), while we focus on the importance of tokens (i.e. token attributions). Extensive experiments in three benchmark datasets (i.e. three downstream tasks) demonstrate that multi-lingual PLMs attach different attributions to multi-lingual synonyms. Moreover, we have the following observations: 1) the Spanish achieves the most consistent token attributions in different languages when it is used for training PLMs; 2) the consistency of token attributions strongly correlates with performance in downstream tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Token Attribution Analysis, Linguistics, Artificial Intelligence, Language Model Performance, Cross-Lingual Consistency" />
        </attvalues>
      </node>
      <node id="2004.03032" label="2004.03032">
        <attvalues>
          <attvalue for="0" value="A Systematic Analysis of Morphological Content in BERT Models for&#10;  Multiple Languages" />
          <attvalue for="1" value="  This work describes experiments which probe the hidden representations of&#10;several BERT-style models for morphological content. The goal is to examine the&#10;extent to which discrete linguistic structure, in the form of morphological&#10;features and feature values, presents itself in the vector representations and&#10;attention distributions of pre-trained language models for five European&#10;languages. The experiments contained herein show that (i) Transformer&#10;architectures largely partition their embedding space into convex sub-regions&#10;highly correlated with morphological feature value, (ii) the contextualized&#10;nature of transformer embeddings allows models to distinguish ambiguous&#10;morphological forms in many, but not all cases, and (iii) very specific&#10;attention head/layer combinations appear to hone in on subject-verb agreement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;With the recent success of Transformer-style architectures \cite{VaswaniEtAl2017} like BERT \cite{DevlinEtAl2018} on many natural language processing tasks, a considerable amount of research has gone into investigating the inner workings of these models, a research program sometimes dubbed ``BERTology'' \cite{RogersEtAl2020}. Among this literature, work has focused on syntactic aspects \cite{HewittManning2019,CoenenEtAl2019,KimEtAl2020}, including subject-verb agreement \cite{Goldberg2019}, and also various semantic aspects such as semantic role and model predictions' correlation with human judgment \cite{Ettinger2020}. &#10;&#10;One particular method of probing the information in these large architectures is to perform different tasks at different layers of the model, seeking to identify where different types of linguistic information may reside \cite{TenneyEtAl2019a}. It has generally been shown that more local, shallow information is reflected in lower layers, and richer, more abstract information is reflected in higher layers \cite{PetersEtAl2018b}. Not only have layers been shown to be specialized for content, \newcite{ClarkEtAl2019} further showed the diffusion of linguistic knowledge through such models by demonstrating that BERT's different attention heads learn to focus on different aspects of linguistic meaning.&#10;&#10;In addition to the growing literature on BERT-style models, work on evaluating continuous embedding models for morphological information goes back some years. Particularly, \newcite{BelinkovEtAl2017} trained classifiers on word representations extracted from models trained for machine translation to assess what these models learn about morphology. For work investigating models for agree-phenomena, \newcite{LinzenEtAl2016} showed that LSTM architectures \cite{HochreiterSchmidhuber1997} successfully model subject-verb agreement in many instances (see also \newcite{GiulianelliEtAl2018}), and \newcite{RavfogelEtAl2018} put forth the objective of modelling agreement in Basque as a potential baseline for future work.. Finally, for work most similar to ours in investigating morphological information at the featural level, \newcite{BasiratTang2018} train classifiers to distringuish nominal features in Swedish, and \newcite{Kohn2015} does the same for more varied features and multiple languages.&#10;&#10;This work finds its place in systematically addressing the question of morphological featural information in the hidden vector representations and attention heads of BERT models for multiple languages.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2011.02070" label="2011.02070">
        <attvalues>
          <attvalue for="0" value="Probing Multilingual BERT for Genetic and Typological Signals" />
          <attvalue for="1" value="  We probe the layers in multilingual BERT (mBERT) for phylogenetic and&#10;geographic language signals across 100 languages and compute language distances&#10;based on the mBERT representations. We 1) employ the language distances to&#10;infer and evaluate language trees, finding that they are close to the reference&#10;family tree in terms of quartet tree distance, 2) perform distance matrix&#10;regression analysis, finding that the language distances can be best explained&#10;by phylogenetic and worst by structural factors and 3) present a novel measure&#10;for measuring diachronic meaning stability (based on cross-lingual&#10;representation variability) which correlates significantly with published&#10;ranked lists based on linguistic approaches. Our results contribute to the&#10;nascent field of typological interpretability of cross-lingual text&#10;representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03310" label="1911.03310">
        <attvalues>
          <attvalue for="0" value="How Language-Neutral is Multilingual BERT?" />
          <attvalue for="1" value="  Multilingual BERT (mBERT) provides sentence representations for 104&#10;languages, which are useful for many multi-lingual tasks. Previous work probed&#10;the cross-linguality of mBERT using zero-shot transfer learning on&#10;morphological and syntactic tasks. We instead focus on the semantic properties&#10;of mBERT. We show that mBERT representations can be split into a&#10;language-specific component and a language-neutral component, and that the&#10;language-neutral component is sufficiently general in terms of modeling&#10;semantics to allow high-accuracy word-alignment and sentence retrieval but is&#10;not yet good enough for the more difficult task of MT quality estimation. Our&#10;work presents interesting challenges which must be solved to build better&#10;language-neutral representations, particularly for tasks requiring linguistic&#10;transfer of semantics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Since the publication of mBERT \cite{devlin2019bert}, many positive&#10;experimental results were published.&#10;&#10;\cite{wang2019crosslingual} reached impressive results in zero-shot dependency&#10;parsing. However, the representation used for the parser was a bilingual&#10;projection of the contextual embeddings based on word-alignment trained on&#10;parallel data.&#10;&#10;\cite{pires2019multilingual} recently examined the cross-lingual properties of&#10;mBERT on zero-shot NER and part-of-speech (POS) tagging but the success of&#10;zero-shot transfer strongly depends on how typologically similar the languages&#10;are. Similarly, \cite{wu2019beto} trained good multilingual models for POS&#10;tagging, NER, and XNLI, but struggled to achieve good results in the zero-shot&#10;setup.&#10;&#10;\minornote{AF: Can we say here: ``In contrast, we study semantic tasks'' or&#10;something similar? There needs to be a contrast with our work.}&#10;&#10;\cite{pires2019multilingual} assessed mBERT on cross-lingual sentence&#10;retrieval between three language pairs. They observed that if they subtract the&#10;average difference between the embeddings from the target language&#10;representation, the retrieval accuracy significantly increases. We&#10;systematically study this idea in the later sections.&#10;&#10;Many experiments show&#10;\cite{wu2019beto,kudugunta2019investigating,kondratyuk2019udify} that&#10;downstream task models can extract relevant features from the multilingual&#10;representations. But these results do not directly show language-neutrality,&#10;i.e., to what extent are similar phenomena are represented similarly across&#10;languages. The models can obtain the task-specific information based on the&#10;knowledge of the language, which (as we show later) can be easily identified.&#10;Our choice of evaluation tasks eliminates this risk by directly comparing the&#10;representations.&#10;Limited success in zero-shot setups and the need for explicit bilingual&#10;projection in order to work well&#10;\cite{pires2019multilingual,wu2019beto,ronnqvist2019multilingual} also shows&#10;limited language neutrality of mBERT\@.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1909.02197" label="1909.02197">
        <attvalues>
          <attvalue for="0" value="Investigating Multilingual NMT Representations at Scale" />
          <attvalue for="1" value="  Multilingual Neural Machine Translation (NMT) models have yielded large&#10;empirical success in transfer learning settings. However, these black-box&#10;representations are poorly understood, and their mode of transfer remains&#10;elusive. In this work, we attempt to understand massively multilingual NMT&#10;representations (with 103 languages) using Singular Value Canonical Correlation&#10;Analysis (SVCCA), a representation similarity framework that allows us to&#10;compare representations across different languages, layers and models. Our&#10;analysis validates several empirical results and long-standing intuitions, and&#10;unveils new observations regarding how representations evolve in a multilingual&#10;translation model. We draw three major conclusions from our analysis, with&#10;implications on cross-lingual transfer learning: (i) Encoder representations of&#10;different languages cluster based on linguistic similarity, (ii)&#10;Representations of a source language learned by the encoder are dependent on&#10;the target language, and vice-versa, and (iii) Representations of high resource&#10;and/or linguistically similar languages are more robust when fine-tuning on an&#10;arbitrary language pair, which is critical to determining how much&#10;cross-lingual transfer can be expected in a zero or few-shot setting. We&#10;further connect our findings with existing empirical observations in&#10;multilingual NMT and transfer learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.05061" label="1906.05061">
        <attvalues>
          <attvalue for="0" value="Probing Multilingual Sentence Representations With X-Probe" />
          <attvalue for="1" value="  This paper extends the task of probing sentence representations for&#10;linguistic insight in a multilingual domain. In doing so, we make two&#10;contributions: first, we provide datasets for multilingual probing, derived&#10;from Wikipedia, in five languages, viz. English, French, German, Spanish and&#10;Russian. Second, we evaluate six sentence encoders for each language, each&#10;trained by mapping sentence representations to English sentence&#10;representations, using sentences in a parallel corpus. We discover that&#10;cross-lingually mapped representations are often better at retaining certain&#10;linguistic information than representations derived from English encoders&#10;trained on natural language inference (NLI) as a downstream task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.11888" label="2101.11888">
        <attvalues>
          <attvalue for="0" value="Does Typological Blinding Impede Cross-Lingual Sharing?" />
          <attvalue for="1" value="  Bridging the performance gap between high- and low-resource languages has&#10;been the focus of much previous work. Typological features from databases such&#10;as the World Atlas of Language Structures (WALS) are a prime candidate for&#10;this, as such data exists even for very low-resource languages. However,&#10;previous work has only found minor benefits from using typological information.&#10;Our hypothesis is that a model trained in a cross-lingual setting will pick up&#10;on typological cues from the input data, thus overshadowing the utility of&#10;explicitly using such features. We verify this hypothesis by blinding a model&#10;to typological information, and investigate how cross-lingual sharing and&#10;performance is impacted. Our model is based on a cross-lingual architecture in&#10;which the latent weights governing the sharing between languages is learnt&#10;during training. We show that (i) preventing this model from exploiting&#10;typology severely reduces performance, while a control experiment reaffirms&#10;that (ii) encouraging sharing according to typology somewhat improves&#10;performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Most languages in the world have little access to NLP technology due to data scarcity \cite{joshi2020state}.&#10;Nonetheless, high-quality multilingual representations can be obtained using only a raw text signal, e.g.~via multilingual language modelling \cite{bert}.&#10;Furthermore, structural similarities of languages are to a large extent documented in typological databases such as the World Atlas of Language Structures (WALS, \cite{wals}).&#10;Hence, developing models which can take use typological similarities of languages is an important direction in order to alleviate language technology inequalities.&#10;&#10;While previous work has attempted to use typological information to inform NLP models, our work differs significantly from such efforts in that we blind a model to this information.&#10;Most previous work includes language information as features, by using language IDs, or language embeddings (e.g.~\cite{ammar2016many,o2016survey,ostling-tiedemann-2017-continuous,ponti2019modeling,oncevay2020bridging}).&#10;Notably, limited effects are usually observed from including typological features explicitly.&#10;For instance, \cite{delhoneux2018parameter} observe positive cross-lingual sharing effects only in a handful of their settings.&#10;We therefore hypothesise that relevant typological information is learned as a by-product of cross-lingual training. &#10;Hence, although models do benefit from this information, it is not necessary to provide it explicitly in a high-resource scenario, where there is abundant training data.&#10;This is confirmed by \cite{bjerva2018phonology}, who find that, e.g., language embeddings trained on a morphological task can encode morphological features from WALS.&#10;&#10;In contrast with previous work, we blind a model to typological information, by using adversarial techniques based on gradient reversal \cite{ganin2014unsupervised}.&#10;We evaluate on the structured prediction and classification tasks in XTREME \cite{xtreme}, yielding a total of 40 languages and 4 tasks.&#10;We show that when a model is blinded to typological signals relating to syntax and morphology, performance on related NLP tasks drops significantly.&#10;For instance, the mean accuracy across 40 languages for POS tagging drops by 1.8\% when blinding the model to morphological features.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.09375" label="1802.09375">
        <attvalues>
          <attvalue for="0" value="From Phonology to Syntax: Unsupervised Linguistic Typology at Different&#10;  Levels with Language Embeddings" />
          <attvalue for="1" value="  A core part of linguistic typology is the classification of languages&#10;according to linguistic properties, such as those detailed in the World Atlas&#10;of Language Structure (WALS). Doing this manually is prohibitively&#10;time-consuming, which is in part evidenced by the fact that only 100 out of&#10;over 7,000 languages spoken in the world are fully covered in WALS.&#10;  We learn distributed language representations, which can be used to predict&#10;typological properties on a massively multilingual scale. Additionally,&#10;quantitative and qualitative analyses of these language embeddings can tell us&#10;how language similarities are encoded in NLP models for tasks at different&#10;typological levels. The representations are learned in an unsupervised manner&#10;alongside tasks at three typological levels: phonology (grapheme-to-phoneme&#10;prediction, and phoneme reconstruction), morphology (morphological inflection),&#10;and syntax (part-of-speech tagging).&#10;  We consider more than 800 languages and find significant differences in the&#10;language representations encoded, depending on the target task. For instance,&#10;although Norwegian Bokm{\aa}l and Danish are typologically close to one&#10;another, they are phonologically distant, which is reflected in their language&#10;embeddings growing relatively distant in a phonological task. We are also able&#10;to predict typological features in WALS with high accuracies, even for unseen&#10;language families.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.12862" label="2009.12862">
        <attvalues>
          <attvalue for="0" value="What does it mean to be language-agnostic? Probing multilingual sentence&#10;  encoders for typological properties" />
          <attvalue for="1" value="  Multilingual sentence encoders have seen much success in cross-lingual model&#10;transfer for downstream NLP tasks. Yet, we know relatively little about the&#10;properties of individual languages or the general patterns of linguistic&#10;variation that they encode. We propose methods for probing sentence&#10;representations from state-of-the-art multilingual encoders (LASER, M-BERT, XLM&#10;and XLM-R) with respect to a range of typological properties pertaining to&#10;lexical, morphological and syntactic structure. In addition, we investigate how&#10;this information is distributed across all layers of the models. Our results&#10;show interesting differences in encoding linguistic variation associated with&#10;different pretraining strategies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Multilingual encoders have been successfully applied to perform zero-shot cross-lingual transfer in downstream NLP tasks, such as part of speech (POS) tagging and named entity recognition (NER) \cite{van2019comparison}, dependency and constituency parsing \cite{tran2019zero, kim2020multilingual}, text categorization \cite{nozza2020mask}, cross-lingual natural language inference (XNLI) and question answering (XQA) \cite{lauscher2020zero}. Interestingly, models trained in unsupervised monolingual tasks (M-BERT, XLM-R) exhibit competitive performance to those that rely on cross-lingual objectives and parallel data (LASER, XLM). Recently, \cite{huang2019unicoder} introduced Unicoder that relies on 4 cross-lingual tasks. Improving on M-BERT and XLM on XNLI and XQA, the authors claim that the tasks help learn language relationships from more perspectives. This raises the question of whether multilingual encoders capture linguistic and typological properties differently depending on the type of pretraining tasks. &#10;&#10;To investigate this, we use techniques from the rapidly growing line of research on interpretation of neural models \cite{linzen2016assessing, conneau2018you, peters2018dissecting, tenney2019you}, which has been recently extended to the multilingual setting \cite{pires2019multilingual, csahin2019linspector, ravishankar2019word, ravishankar2019sentence}. \cite{ravishankar2019word, ravishankar2019sentence} study multilingual sentence encoders using probing tasks of \cite{conneau2018you}, e.g probing for universal properties such as sentence length and tree depth, but do not directly probe for typological information. In a similar vein, \cite{pires2019multilingual} study how M-BERT generalizes across languages by testing zero-shot cross-lingual transfer in traditional downstream tasks. They only briefly touch on typology by testing generalization across typologically diverse languages in POS tagging and NER, and find that cross-lingual transfer is more effective across similar languages. They ascribe this effect to word-piece overlap, arguing that similar success on distant languages might require a cross-lingual objective. On the contrary, \cite{karthikeyan2020cross} show that cross-lingual transfer can also be successful with zero lexical overlap, arguing that M-BERT's cross-lingual effectiveness stems from its ability to recognize language structure and semantics instead. In this work, we take a closer look at these emerging language structures by probing the models for typological properties. &#10;&#10;To the best of our knowledge, our approach comes closest to that of \cite{csahin2019linspector}, who probed non-contextualized multilingual word representations for linguistic properties such as case marking, gender system and grammatical mood. We considerably expand on this work by proposing methods to probe multilingual sentence encoders and investigating a wider range of typological properties pertaining to lexical, morphological and syntactic structure. Since such models are inclined to learn a language identity \cite{wu2019beto}, we also propose a paired language evaluation set-up, evaluating on languages unseen during training. &#10;&#10;Previous research on monolingual model probing has shown that lower layers of a language model capture local syntax, while higher layers tend to capture more complex semantics \cite{peters2018dissecting, blevins2018deep}. \cite{tenney2019bert} show that the same ordering emerges in BERT, and that syntactic information is more localizable within the model, while information related to semantic tasks is scattered across many layers. We take a similar approach to test where in the model typological information is encoded and whether it is localized or spread across layers.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2004.14923" label="2004.14923">
        <attvalues>
          <attvalue for="0" value="Bridging Linguistic Typology and Multilingual Machine Translation with&#10;  Multi-View Language Representations" />
          <attvalue for="1" value="  Sparse language vectors from linguistic typology databases and learned&#10;embeddings from tasks like multilingual machine translation have been&#10;investigated in isolation, without analysing how they could benefit from each&#10;other's language characterisation. We propose to fuse both views using singular&#10;vector canonical correlation analysis and study what kind of information is&#10;induced from each source. By inferring typological features and language&#10;phylogenies, we observe that our representations embed typology and strengthen&#10;correlations with language relationships. We then take advantage of our&#10;multi-view language vector space for multilingual machine translation, where we&#10;achieve competitive overall translation accuracy in tasks that require&#10;information about language similarities, such as language clustering and&#10;ranking candidates for multilingual transfer. With our method, which is also&#10;released as a tool, we can easily project and assess new languages without&#10;expensive retraining of massive multilingual or ranking models, which are major&#10;disadvantages of related approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.03620" label="1805.03620">
        <attvalues>
          <attvalue for="0" value="On the Limitations of Unsupervised Bilingual Dictionary Induction" />
          <attvalue for="1" value="  Unsupervised machine translation---i.e., not assuming any cross-lingual&#10;supervision signal, whether a dictionary, translations, or comparable&#10;corpora---seems impossible, but nevertheless, Lample et al. (2018) recently&#10;proposed a fully unsupervised machine translation (MT) model. The model relies&#10;heavily on an adversarial, unsupervised alignment of word embedding spaces for&#10;bilingual dictionary induction (Conneau et al., 2018), which we examine here.&#10;Our results identify the limitations of current unsupervised MT: unsupervised&#10;bilingual dictionary induction performs much worse on morphologically rich&#10;languages that are not dependent marking, when monolingual corpora from&#10;different domains or different embedding algorithms are used. We show that a&#10;simple trick, exploiting a weak supervision signal from identical words,&#10;enables more robust induction, and establish a near-perfect correlation between&#10;unsupervised bilingual dictionary induction performance and a previously&#10;unexplored graph similarity metric.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.04070" label="2004.04070">
        <attvalues>
          <attvalue for="0" value="Are All Good Word Vector Spaces Isomorphic?" />
          <attvalue for="1" value="  Existing algorithms for aligning cross-lingual word vector spaces assume that&#10;vector spaces are approximately isomorphic. As a result, they perform poorly or&#10;fail completely on non-isomorphic spaces. Such non-isomorphism has been&#10;hypothesised to result from typological differences between languages. In this&#10;work, we ask whether non-isomorphism is also crucially a sign of degenerate&#10;word vector spaces. We present a series of experiments across diverse languages&#10;which show that variance in performance across language pairs is not only due&#10;to typological differences, but can mostly be attributed to the size of the&#10;monolingual resources available, and to the properties and duration of&#10;monolingual training (e.g. &quot;under-training&quot;).&#10;" />
          <attvalue for="2" value="&#10;Word embeddings have been argued to reflect how language users organise concepts \cite{Mandera:2017,Asr:2018naacl}. The extent to which they really do so has been evaluated, e.g., using semantic word similarity and association norms \cite{hill2015simlex,Gerz:2016emnlp}, and word analogy benchmarks \cite{Mikolov2013naacl}. If word embeddings reflect more or less language-independent conceptual organisations, word embeddings in different languages can be expected to be near-isomorphic. Researchers have exploited this to learn linear transformations between such spaces \cite{Mikolov2013exploiting,Glavas2019}, which have been used to induce bilingual dictionaries, as well as to facilitate multilingual modeling and cross-lingual transfer \cite{Ruder2019survey}.&#10;&#10;In this paper, we show that near-isomorphism arises only with sufficient amounts of training. This is of practical interest for applications of linear alignment methods for cross-lingual word embeddings. It furthermore provides us with an explanation for reported failures to align word vector spaces in different languages \cite{sogaard2018limitations,Artetxe2018}, which has so far been largely attributed only to inherent typological differences.&#10;&#10;In fact, the amount of data used to induce the monolingual embeddings is predictive of the quality of the aligned cross-lingual word embeddings, as evaluated on bilingual lexicon induction (BLI). Consider, for motivation, Figure~\ref{fig:token-counts-vs-scores}; it shows the performance of a state-of-the-art alignment method---RCSLS with iterative normalisation \cite{Zhang2019}---on mapping English embeddings onto embeddings in other languages, and its correlation ($\rho = 0.72$) with the size of the tokenised target language Polyglot Wikipedia \cite{polyglot:2013:ACL-CoNLL}. &#10;&#10;We investigate to what extent the amount of data available for some languages and corresponding training conditions provide a {sufficient}~explanation for the variance in reported results; that is, whether it is the full story or not: The answer is 'almost', that is, its interplay with inherent typological differences does have a crucial impact on the `alignability' of monolingual vector spaces.&#10;&#10;We first discuss current standard methods of quantifying the degree of near-isomorphism between word vector spaces (\S\ref{s:quantify-isomorphism}). We then outline training settings that may influence isomorphism (\S\ref{s:isomorphism-and-learning}) and present a novel experimental protocol for learning cross-lingual word embeddings that simulates a low-resource environment, and also controls for topical skew and differences in morphological complexity (\S\ref{s:simulated-low-resource}). We focus on two groups of languages: 1) Spanish, Basque, Galician, and Quechua, and 2) Bengali, Tamil, and Urdu, as these are arguably spoken in culturally related regions, but have very different morphology. Our experiments, among other findings, indicate that a low-resource version of Spanish is as difficult to align to English as Quechua, challenging the assumption from prior work that the primary issue to resolve in cross-lingual word embedding learning is language dissimilarity (instead of, e.g., procuring additional raw data for embedding training). We also show that by controlling for different factors, we reduce the gap between aligning Spanish and Basque to English from 0.291 to 0.129. Similarly, under these controlled circumstances, we do not observe any substantial performance difference between aligning Spanish and Galician to English, or between aligning Bengali and Tamil to English.&#10;&#10;We also investigate the learning dynamics of monolingual word embeddings and their impact on BLI performance and near-isomorphism of the resulting word vector spaces (\S\ref{s:experimental}), finding training duration, amount of monolingual resources, preprocessing, and self-learning all to have a large impact. The findings are verified across a set of typologically diverse languages, where we pair English with Spanish, Arabic, and Japanese. &#10;&#10;We will release our new evaluation dictionaries and subsampled Wikipedias controlling for topical skew and morphological differences to facilitate future research at:.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.10764" label="2406.10764">
        <attvalues>
          <attvalue for="0" value="GNOME: Generating Negotiations through Open-Domain Mapping of Exchanges" />
          <attvalue for="1" value="  Language Models have previously shown strong negotiation capabilities in&#10;closed domains where the negotiation strategy prediction scope is constrained&#10;to a specific setup. In this paper, we first show that these models are not&#10;generalizable beyond their original training domain despite their wide-scale&#10;pretraining. Following this, we propose an automated framework called GNOME,&#10;which processes existing human-annotated, closed-domain datasets using Large&#10;Language Models and produces synthetic open-domain dialogues for negotiation.&#10;GNOME improves the generalizability of negotiation systems while reducing the&#10;expensive and subjective task of manual data curation. Through our experimental&#10;setup, we create a benchmark comparing encoder and decoder models trained on&#10;existing datasets against datasets created through GNOME. Our results show that&#10;models trained on our dataset not only perform better than previous state of&#10;the art models on domain specific strategy prediction, but also generalize&#10;better to previously unseen domains.&#10;" />
          <attvalue for="2" value="&#10;&#10;Negotiation is a key component of modern dialogue systems and has significant applications from bargaining~\cite{lewicki1981bargaining-application} to gaming~\cite{hausken1997game-application}, and business transactions~\cite{, filzmoser2010automatedvshuman, mumpower1991judgment}. Negotiation agents powered by language models (LMs) are increasingly being used to conduct automated negotiations in a variety of contexts, including game theory~\cite{ding-catan-2021, peskov2020takes}, multi-issue bargaining~\cite{chawla2021casino, job_interview} and pyschotherapy~\cite{tanana2016comparison}.&#10;&#10;Previous works have explored models trained for negotiation strategy prediction that aim to label each utterance in a dialogue with the appropriate negotiation strategies to enable the negotiator to understand the opponent's motive effectively.&#10;These studies include&#10;development of strategy prediction models for tasks such as commodity price bargaining~\cite{ahmad2023ina, craiglist_bargain} and item negotiation in a barter setting~\cite{lewis2017dealnodeal, chawla2021casino} while limiting the scope of the negotiation to a specific selection of items (closed-domain). However, to facilitate the wider adoption and domain-agnostic negotiation understanding, the training datasets must be as varied and open as possible~\cite{craiglist_bargain}.&#10;Additionally, closed-domain settings have been shown to affect out-of-domain task performance, which remains an unsolved challenge for modern negotiation agents~\cite{li-murray-2023-zero}.&#10;&#10;Recently, Large Language Models (LLMs) have shown excellent generalization capabilities in zero and few-shot settings across multiple benchmarks~\cite{bubeck2023sparks, chang2023survey,kojima2022large, hou2024large, ahmed2022few, dai2022promptagator}. As a preliminary experiment to our study, we show that this generalization capability is impacted when LLMs are fine-tuned on closed-domain negotiation strategy prediction tasks. For our experiments, we selected four popular datasets, trained models on these individually, and evaluated their performance on out-of-domain data from other datasets. These datasets included CaSiNo~\cite{chawla2021casino}, which consists of negotiations involving campfire resources like firewood, water, and food; Job Interview~\cite{job_interview}, containing simulated job interview negotiations for salary and responsibilities; Craigslist Bargain~\cite{craiglist_bargain}, focused on negotiations for items on the Craigslist platform; and Persuasion For Good~\cite{persuasion-for-good}, where participants are persuaded to donate to social causes. These datasets present a varied set of negotiation strategies that were manually mapped to a more general set of labels to allow for cross-comparison (more details about this mapping can be found in \ref{app:mappings}). &#10;&#10;Our results presented in \ref{fig:generalizability_results} demonstrate that existing encoders and LLMs trained on individual datasets show poor generalization when tested on datasets with different negotiation scenarios. Furthermore, we observed that domain-restricted training, for example, on item exchange-based CaSiNo, poorly generalizes on monetary negotiations, as found in the Job Interview and Craigslist Bargain datasets. This highlights the need for diverse, cross-domain training data.&#10;&#10;To address this generalizability problem, we introduce a novel framework for Generating Negotiations through Open Domain Mapping of Exchanges, or GNOME, that utilizes existing human-annotated closed-domain datasets to produce synthetic open-domain datasets for strategy prediction for negotiation. An example of this is shown in \ref{fig:overview} where the original dialogue for a bike sale is translated to a negotiation for a Victorian armchair while maintaining the consistency of the original dialogue.&#10;&#10;In this work, we show that:&#10;\begin{enumerate}&#10; \item Models trained and fine-tuned on open-domain data exhibit improved performance in domain-specific strategy prediction tasks.&#10; \item Open-domain training and fine-tuning enhances a model's ability to generalize to novel, unseen domains.&#10; \item High-quality synthetic data, derived from human annotations, can effectively substitute for costly additional human-annotated data.&#10;\end{enumerate} &#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Limitations, Artificial Intelligence, Negotiation Strategy Prediction, Natural Language Processing, Automated Data Generation" />
        </attvalues>
      </node>
      <node id="2103.15721" label="2103.15721">
        <attvalues>
          <attvalue for="0" value="CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic&#10;  Negotiation Systems" />
          <attvalue for="1" value="  Automated systems that negotiate with humans have broad applications in&#10;pedagogy and conversational AI. To advance the development of practical&#10;negotiation systems, we present CaSiNo: a novel corpus of over a thousand&#10;negotiation dialogues in English. Participants take the role of campsite&#10;neighbors and negotiate for food, water, and firewood packages for their&#10;upcoming trip. Our design results in diverse and linguistically rich&#10;negotiations while maintaining a tractable, closed-domain environment. Inspired&#10;by the literature in human-human negotiations, we annotate persuasion&#10;strategies and perform correlation analysis to understand how the dialogue&#10;behaviors are associated with the negotiation performance. We further propose&#10;and evaluate a multi-task framework to recognize these strategies in a given&#10;utterance. We find that multi-task learning substantially improves the&#10;performance for all strategy labels, especially for the ones that are the most&#10;skewed. We release the dataset, annotations, and the code to propel future work&#10;in human-machine negotiations: https://github.com/kushalchawla/CaSiNo&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.05125" label="1706.05125">
        <attvalues>
          <attvalue for="0" value="Deal or No Deal? End-to-End Learning for Negotiation Dialogues" />
          <attvalue for="1" value="  Much of human dialogue occurs in semi-cooperative settings, where agents with&#10;different goals attempt to agree on common decisions. Negotiations require&#10;complex communication and reasoning skills, but success is easy to measure,&#10;making this an interesting task for AI. We gather a large dataset of&#10;human-human negotiations on a multi-issue bargaining task, where agents who&#10;cannot observe each other's reward functions must reach an agreement (or a&#10;deal) via natural language dialogue. For the first time, we show it is possible&#10;to train end-to-end models for negotiation, which must learn both linguistic&#10;and reasoning skills with no annotated dialogue states. We also introduce&#10;dialogue rollouts, in which the model plans ahead by simulating possible&#10;complete continuations of the conversation, and find that this technique&#10;dramatically improves performance. Our code and dataset are publicly available&#10;(https://github.com/facebookresearch/end-to-end-negotiator).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.09637" label="1808.09637">
        <attvalues>
          <attvalue for="0" value="Decoupling Strategy and Generation in Negotiation Dialogues" />
          <attvalue for="1" value="  We consider negotiation settings in which two agents use natural language to&#10;bargain on goods. Agents need to decide on both high-level strategy (e.g.,&#10;proposing \$50) and the execution of that strategy (e.g., generating &quot;The bike&#10;is brand new. Selling for just \$50.&quot;). Recent work on negotiation trains&#10;neural models, but their end-to-end nature makes it hard to control their&#10;strategy, and reinforcement learning tends to lead to degenerate solutions. In&#10;this paper, we propose a modular approach based on coarse di- alogue acts&#10;(e.g., propose(price=50)) that decouples strategy and generation. We show that&#10;we can flexibly set the strategy using supervised learning, reinforcement&#10;learning, or domain-specific knowledge without degeneracy, while our&#10;retrieval-based generation can maintain context-awareness and produce diverse&#10;utterances. We test our approach on the recently proposed DEALORNODEAL game,&#10;and we also collect a richer dataset based on real items on Craigslist. Human&#10;evaluation shows that our systems achieve higher task success rate and more&#10;human-like negotiation behavior than previous approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.06725" label="1906.06725">
        <attvalues>
          <attvalue for="0" value="Persuasion for Good: Towards a Personalized Persuasive Dialogue System&#10;  for Social Good" />
          <attvalue for="1" value="  Developing intelligent persuasive conversational agents to change people's&#10;opinions and actions for social good is the frontier in advancing the ethical&#10;development of automated dialogue systems. To do so, the first step is to&#10;understand the intricate organization of strategic disclosures and appeals&#10;employed in human persuasion conversations. We designed an online persuasion&#10;task where one participant was asked to persuade the other to donate to a&#10;specific charity. We collected a large dataset with 1,017 dialogues and&#10;annotated emerging persuasion strategies from a subset. Based on the&#10;annotation, we built a baseline classifier with context information and&#10;sentence-level features to predict the 10 persuasion strategies used in the&#10;corpus. Furthermore, to develop an understanding of personalized persuasion&#10;processes, we analyzed the relationships between individuals' demographic and&#10;psychological backgrounds including personality, morality, value systems, and&#10;their willingness for donation. Then, we analyzed which types of persuasion&#10;strategies led to a greater amount of donation depending on the individuals'&#10;personal backgrounds. This work lays the ground for developing a personalized&#10;persuasive dialogue system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.04160" label="2305.04160">
        <attvalues>
          <attvalue for="0" value="X-LLM: Bootstrapping Advanced Large Language Models by Treating&#10;  Multi-Modalities as Foreign Languages" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated remarkable language abilities.&#10;GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities&#10;beyond previous visual language models. We attribute this to the use of more&#10;advanced LLMs compared with previous multimodal models. Unfortunately, the&#10;model architecture and training strategies of GPT-4 are unknown. To endow LLMs&#10;with multimodal capabilities, we propose X-LLM, which converts Multi-modalities&#10;(images, speech, videos) into foreign languages using X2L interfaces and inputs&#10;them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple&#10;frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X''&#10;denotes multi-modalities such as image, speech, and videos, and ``L'' denotes&#10;languages. X-LLM's training consists of three stages: (1) Converting Multimodal&#10;Information: The first stage trains each X2L interface to align with its&#10;respective single-modal encoder separately to convert multimodal information&#10;into languages. (2) Aligning X2L representations with the LLM: single-modal&#10;encoders are aligned with the LLM through X2L interfaces independently. (3)&#10;Integrating multiple modalities: all single-modal encoders are aligned with the&#10;LLM through X2L interfaces to integrate multimodal capabilities into the LLM.&#10;Our experiments show that X-LLM demonstrates impressive multimodel chat&#10;abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen&#10;images/instructions, and yields a 84.5\% relative score compared with GPT-4 on&#10;a synthetic multimodal instruction-following dataset. And we also conduct&#10;quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote&#10;the era of LLM-based speech recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Vision-Language Models.&#10;As summarized in many surveys~\cite{chen2023vlp,du2022survey}, visual language models~\cite{zhou2020unified,li2022blip} have made great strides with the development of pre-training techniques~\cite{devlin2018bert,bi2020palm,zoph2020rethinking}. In the early days, researchers used Faster-RCNN~\cite{girshick2015fast} to extract image features and concatenated them with language models such as BERT~\cite{devlin2018bert} to perform vision-language pre-training. VisualBERT~\cite{li2019visualbert}, for example, combines image regions and language using a Transformer~\cite{vaswani2017attention} to allow self-attention to discover implicit alignments between language and vision. It is pre-trained with masked language modeling~\cite{devlin2018bert} and a sentence-image prediction task~\cite{li2019visualbert}. With the introduction of ViLT~\cite{kim2021vilt}, researchers use vision transformers~\cite{dosovitskiy2020image,khan2022transformers} to process images, textual transformers (such as BERT~\cite{devlin2018bert}, GPT-2~\cite{radford2019language}, T5~\cite{raffel2020exploring}) to process text, and pre-training objectives such as masked language modeling, image-text matching, and image-text contrast to train visual language models. CLIP~\cite{radford2021learning} uses a text encoder and an image encoder to encode text and images separately and then performs unsupervised contrastive learning to obtain good representations of vision-language alignment. BLIP~\cite{li2022blip} is a new VLP framework that transfers flexibly to both vision-language understanding and generation tasks.&#10;&#10;In the field of visual dialogue~\cite{das2017visual,chen2020dmrm,chen2022utc}, researchers design pre-training objectives related to visual dialogue based on vision-language pre-training models~\cite{devlin2018bert,li2019visualbert} and finetune vison-language models on visual dialogue data~\cite{das2017visual} to achieve better dialogue performance. VisDial-BERT~\cite{murahari2020large} and VD-BERT~\cite{wang2020vd}, for example, use pre-trained ViLBERT~\cite{lu2019vilbert} and BERT to finetune models on visual dialogue data using masked language modeling and image-text matching. AlignVD~\cite{chen2022unsupervised} proposes two methods for visual-language alignment based on pre-trained ViT~\cite{radford2021learning} and BERT to achieve better performance in visual dialogue. &#10;&#10;Enhancing Vision-language Understanding with Advanced LLMs.&#10;Although the aforementioned vision-language models have achieved some success, there is still significant room for improvement in terms of language generation~\cite{chen2023vlp,long2022vision,li2023blip2}. A recent method~\cite{li2023blip2,driess2023palm,openaigpt4} for enhancing visual language understanding using advanced large-scale language models~\cite{touvron2023llama,chiang2023vicuna} has been proposed. For example, BLIP2~\cite{li2023blip2} uses a Q-Former to connect a visual encoder with an LLM, aligning the learned queries of the Q-Former with language-related visual features extracted by the visual encoder. The Q-Former then connects the visual encoder with the language model, allowing the learned query representations to adapt to the LLM. PaLM-E~\cite{driess2023palm} combines ViT-22B~\cite{dehghani2023scaling} with PaLM-560B~\cite{bi2020palm} to inject multimodal information into the embedding space of the pre-trained language model, establishing a connection between perception and language and greatly enhancing the model's visual language understanding ability. In addition, Visual ChatGPT~\cite{wu2023visual} and HuggingGPT~\cite{shen2023hugginggpt} use ChatGPT as the core logic controller, which understands user intent and then call upon specific domain visual language models. Finally, the recently proposed GPT-4~\cite{openaigpt4} demonstrates powerful multimodal capabilities: building on its strong language understanding abilities, it can generate complex image descriptions, create websites based on handwritten text instructions, and explain unusual visual phenomena. However, the model structure and training strategies of GPT-4 remain a mystery. MiniGPT-4~\cite{zhu2023minigpt4} and LLaVA~\cite{liu2023visual} align text and image data to the large-scale language model Vicuna~\cite{chiang2023vicuna} and ViT~\cite{zhai2022scaling} to complete image-based language tasks. In contrast, X-LLM is a universal framework for multimodal LLMs that bootstraps advanced large language models by treating multi-modalities as foreign languages. In this paper, we implement X-LLM that supports images, videos, and speech. Based on the X-LLM framework, we can extend the model to more modalities, such as injecting continuous space robot states, terminal information, or audio rather than speech into the LLM.&#10; " />
          <attvalue for="4" value="Multimodal Language Models, Computer Science, Linguistics, Cognitive Science, Large Language Model Architecture, Artificial Intelligence, Multimodal Information Processing" />
        </attvalues>
      </node>
      <node id="2005.09282" label="2005.09282">
        <attvalues>
          <attvalue for="0" value="Bayesian Subspace HMM for the Zerospeech 2020 Challenge" />
          <attvalue for="1" value="  In this paper we describe our submission to the Zerospeech 2020 challenge,&#10;where the participants are required to discover latent representations from&#10;unannotated speech, and to use those representations to perform speech&#10;synthesis, with synthesis quality used as a proxy metric for the unit quality.&#10;In our system, we use the Bayesian Subspace Hidden Markov Model (SHMM) for unit&#10;discovery. The SHMM models each unit as an HMM whose parameters are constrained&#10;to lie in a low dimensional subspace of the total parameter space which is&#10;trained to model phonetic variability. Our system compares favorably with the&#10;baseline on the human-evaluated character error rate while maintaining&#10;significantly lower unit bitrate.&#10;" />
          <attvalue for="2" value="&#10;Learning useful unsupervised representations of data is one of the most important research questions of modern machine learning. In Automatic Speech Recognition (ASR) particularly, there is a wide performance schism between systems in resource-rich languages and low resource languages due to the dependence of contemporary ASR technology on large annotated speech corpora. There has therefore been growing interest in using unsupervised machine learning approaches to speech processing to bridge this gap. One line of research is Acoustic Unit Discovery (AUD) where the goal is to discover a set of units, similar to phones, from unlabeled speech in a language.&#10;&#10;The overarching theme of most AUD methods is the postulation of a latent space which represents the units and which is inferred from the data. Bayesian approaches model the problem as a generative process such as an HMM or GMM with a Dirichlet process prior so that both the number of units and the parameters of those units' models can be inferred~\cite{lee2012nonparametric,ondel2016variational,kamper2017segmental,heck2017feature,ondel2018bayesian, ondel2017bayesian, Ondel2019shmm}. Another approach is the use of neural networks which usually learn continuous latent spaces rather than discrete units. Autoencoders have been used to this end, with further constraints such as speaker invariance costs or correspondence training~\cite{kamper2015unsupervised,kamper2019truly,yusuf2019temporally}. Siamese neural networks have also been applied, where the labels for training are obtained from an unsupervised term detection system~\cite{thiolliere2015hybrid,zeghidour2016deep}.&#10;&#10;Recently, Bayesian-neural-network hybrids have become popular because they combine the modeling power of neural networks with the structured self-regularizing properties of Bayesian models. Variants of the Variational AutoEncoder (VAE) have been especially successful~\cite{kingma2013auto}. Among these are the vector quantized VAE~\cite{chorowski2019unsupervised,Tjandra2019,Eloff2019}, HMM-VAE~\cite{ebbers2017hidden, glarner2018full}.&#10;&#10;The AUD system we adopt in this work is one such Bayesian hybrid, the Subspace Hidden Markov Model (SHMM) system~\cite{Ondel2019shmm}. It models each acoustic unit as an HMM whose parameters are constrained to be in a low-dimensional phonetic subspace of the parameter space.&#10;&#10;The Zerospeech challenge~\cite{versteegh2015zero, dunbar2017zero, dunbar2019zero} provides a platform to test these myriad systems on the same task. Initial iterations of the challenge measured performance using the ABX discriminability criterion~\cite{schatz2013evaluating,schatz2014evaluating}. Since the 2019 challenge~\cite{dunbar2019zero}, there has been a shift to a synthesis-based evaluation scheme where the learned units are used to synthesize audio waveforms, and the quality of the synthesized waveforms - measured by human-evaluated Character Error Rate (CER) - is used as a proxy metric for the quality of the AUD. The rationale is that given a set of waveforms and their transcriptions, one should be able to train synthesis systems, and the more accurate the transcripts are, the less garbled the synthesis would be. The other relevant metric for us is the bitrate which measures the conciseness of the representation, i.e. if the AUD system incorrectly ascribes multiple units to the same phone, while the synthesis may not degrade much, the bitrate would increase.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Speech Processing, Linguistics, Bayesian Modeling, Speech Synthesis, Mathematics, Latent Representation, Phonetics" />
        </attvalues>
      </node>
      <node id="1802.06053" label="1802.06053">
        <attvalues>
          <attvalue for="0" value="Bayesian Models for Unit Discovery on a Very Low Resource Language" />
          <attvalue for="1" value="  Developing speech technologies for low-resource languages has become a very&#10;active research field over the last decade. Among others, Bayesian models have&#10;shown some promising results on artificial examples but still lack of in situ&#10;experiments. Our work applies state-of-the-art Bayesian models to unsupervised&#10;Acoustic Unit Discovery (AUD) in a real low-resource language scenario. We also&#10;show that Bayesian models can naturally integrate information from other&#10;resourceful languages by means of informative prior leading to more consistent&#10;discovered units. Finally, discovered acoustic units are used, either as the&#10;1-best sequence or as a lattice, to perform word segmentation. Word&#10;segmentation results show that this Bayesian approach clearly outperforms a&#10;Segmental-DTW baseline on the same corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1312.6114" label="1312.6114">
        <attvalues>
          <attvalue for="0" value="Auto-Encoding Variational Bayes" />
          <attvalue for="1" value="  How can we perform efficient inference and learning in directed probabilistic&#10;models, in the presence of continuous latent variables with intractable&#10;posterior distributions, and large datasets? We introduce a stochastic&#10;variational inference and learning algorithm that scales to large datasets and,&#10;under some mild differentiability conditions, even works in the intractable&#10;case. Our contributions are two-fold. First, we show that a reparameterization&#10;of the variational lower bound yields a lower bound estimator that can be&#10;straightforwardly optimized using standard stochastic gradient methods. Second,&#10;we show that for i.i.d. datasets with continuous latent variables per&#10;datapoint, posterior inference can be made especially efficient by fitting an&#10;approximate inference model (also called a recognition model) to the&#10;intractable posterior using the proposed lower bound estimator. Theoretical&#10;advantages are reflected in experimental results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.07556" label="1904.07556">
        <attvalues>
          <attvalue for="0" value="Unsupervised acoustic unit discovery for speech synthesis using discrete&#10;  latent-variable neural networks" />
          <attvalue for="1" value="  For our submission to the ZeroSpeech 2019 challenge, we apply discrete&#10;latent-variable neural networks to unlabelled speech and use the discovered&#10;units for speech synthesis. Unsupervised discrete subword modelling could be&#10;useful for studies of phonetic category learning in infants or in low-resource&#10;speech technology requiring symbolic input. We use an autoencoder (AE)&#10;architecture with intermediate discretisation. We decouple acoustic unit&#10;discovery from speaker modelling by conditioning the AE's decoder on the&#10;training speaker identity. At test time, unit discovery is performed on speech&#10;from an unseen speaker, followed by unit decoding conditioned on a known target&#10;speaker to obtain reconstructed filterbanks. This output is fed to a neural&#10;vocoder to synthesise speech in the target speaker's voice. For discretisation,&#10;categorical variational autoencoders (CatVAEs), vector-quantised VAEs (VQ-VAEs)&#10;and straight-through estimation are compared at different compression levels on&#10;two languages. Our final model uses convolutional encoding, VQ-VAE&#10;discretisation, deconvolutional decoding and an FFTNet vocoder. We show that&#10;decoupled speaker conditioning intrinsically improves discrete acoustic&#10;representations, yielding competitive synthesis quality compared to the&#10;challenge baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.01077" label="2404.01077">
        <attvalues>
          <attvalue for="0" value="Efficient Prompting Methods for Large Language Models: A Survey" />
          <attvalue for="1" value="  Prompting has become a mainstream paradigm for adapting large language models&#10;(LLMs) to specific natural language processing tasks. While this approach opens&#10;the door to in-context learning of LLMs, it brings the additional computational&#10;burden of model inference and human effort of manual-designed prompts,&#10;particularly when using lengthy and complex prompts to guide and control the&#10;behavior of LLMs. As a result, the LLM field has seen a remarkable surge in&#10;efficient prompting methods. In this paper, we present a comprehensive overview&#10;of these methods. At a high level, efficient prompting methods can broadly be&#10;categorized into two approaches: prompting with efficient computation and&#10;prompting with efficient design. The former involves various ways of&#10;compressing prompts, and the latter employs techniques for automatic prompt&#10;optimization. We present the basic concepts of prompting, review the advances&#10;for efficient prompting, and highlight future research directions.&#10;" />
          <attvalue for="2" value="&#10;\tikzstyle{my-box}=[&#10; rectangle,&#10; draw=GGray!50,&#10; rounded corners,&#10; text opacity=1,&#10; minimum height=1.5em,&#10; minimum width=5em,&#10; inner sep=2pt,&#10; align=center,&#10; fill opacity=.5,&#10; line width=1pt,&#10;]&#10;\tikzstyle{Ins_leaf}=[my-box, minimum height=1.5em,&#10; fill=BBlue!25, text=black,&#10; align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=1pt,&#10;]&#10;\tikzstyle{CoT_leaf}=[my-box, minimum height=1.5em,&#10; fill=YYellow!25, text=black,&#10; align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=1pt,&#10;]&#10;\tikzstyle{T2V_leaf}=[my-box, minimum height=1.5em,&#10; fill=OOrange!25, text=black,&#10; align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=1pt,&#10;]&#10;\tikzstyle{T2T_leaf}=[my-box, minimum height=1.5em,&#10; fill=GGreen!25, text=black,&#10; align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=1pt,&#10;]&#10;&#10;As hundreds of billions of breakthroughs on the parameter scale, Large Language Models (LLMs) acquire emergent abilities~\cite{Wei2022EmergentAO}, especially in-context learning ability~\cite{brown2020language} that promote rapid advancement in prompting techniques. &#10;Prompting stands out as a lightweight promising solution for controlling LLMs without tuning parameters, having received widespread attention within the Natural Language Processing (NLP) community. &#10;There are two types of prompts, where the hard prompt is discrete natural language descriptions and the soft prompt is continuous vector representations.&#10;In particular, the hard prompt has become a crucial bridge for human-machine interaction relying on its improved interpretability, controllability and flexibility compared to the soft prompt.&#10;Therefore, we mainly focus on the hard prompt that covers all the components of LLM input scaling from concise instructions to long context with demonstrations (Chain-of-Thought (CoT)~\cite{Wei2022ChainOT}, role-playing system prompts, etc.).&#10;&#10;At present, it is common to unlock the potential of LLMs in specific domains by prompting. For example, the CoT series of studies~\cite{Yao2023TreeOT, Besta2023GraphOT, Chen2022ProgramOT} have progressively enhanced LLM reasoning capability by thinking aloud. Furthermore, OpenAI recently introduced the reasoning LLM o1~\cite{o1, o1-mini} trained with reinforcement learning to break down more difficult problems and produce a long internal CoT before responding. The excellent performance of these methods promotes increasing research on optimized prompting methods, which has gradually formed a brand new area in the NLP landscape. Meanwhile, several challenges related to application efficiency come one after another: more and more complex prompt design makes manual prompt optimization time-consuming and labor-intensive; more and more detailed prompt content inevitably consumes significant computational resources when applied to large-scale models. Such prohibitive overheads have become a major barrier to the practical deployment of LLMs, so we define ``Efficient Prompting Methods'' as prompting language models to achieve comparable or even better performance with fewer human or computational resources in this paper.&#10;&#10;We narrow this survey to efficient prompting methods in the era of LLMs. To the best of our knowledge, this is the first survey to summarize LLM prompting methods from the point of ``Efficient''. It is remarkable that we model the core concepts of each category of resource-saving methods from a mathematical perspective in \S\ref{Mathematical_modeling}. Following this, we propose a novel taxonomy as shown in Fig. \ref{Fig_Taxonomy} to comprehensively review efficient prompting methods based on their consistent optimization strategies. To avoid human resources, we introduce automatic prompt engineering efforts based on LLM empowerment in \S\ref{Automatic_prompt_engineering}, including iterative design and optimization of different prompt components. To save computational resources, we organize prompt compression efforts into two categories based on prompt types in \S\ref{Prompt_compression}: Text-to-Vector (T2V) compression in continuous space and Text-to-Text (T2T) compression in discrete space. Additionally, we provide sufficient schematic diagrams depicting the basic pipeline of each category of methods, as well as tables representing differences in the details of the same category of methods. There is also a list of open-source projects in Appendix \ref{Open_Resources} as a quick access for NLP practitioners efficiently prompting LLMs in both scientific research and commercial deployment. Finally, we analyze the challenges of existing methods and discuss promising future directions in \S\ref{Future_directions}. We hope this survey can provide a clear picture of the efficient prompting topic and contribute to convenient human-machine interaction in the progress of Artificial General Intelligence (AGI).&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Efficient Prompting Methods, Natural Language Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2012.15562" label="2012.15562">
        <attvalues>
          <attvalue for="0" value="UNKs Everywhere: Adapting Multilingual Language Models to New Scripts" />
          <attvalue for="1" value="  Massively multilingual language models such as multilingual BERT offer&#10;state-of-the-art cross-lingual transfer performance on a range of NLP tasks.&#10;However, due to limited capacity and large differences in pretraining data&#10;sizes, there is a profound performance gap between resource-rich and&#10;resource-poor target languages. The ultimate challenge is dealing with&#10;under-resourced languages not covered at all by the models and written in&#10;scripts unseen during pretraining. In this work, we propose a series of novel&#10;data-efficient methods that enable quick and effective adaptation of pretrained&#10;multilingual models to such low-resource languages and unseen scripts. Relying&#10;on matrix factorization, our methods capitalize on the existing latent&#10;knowledge about multiple languages already available in the pretrained model's&#10;embedding matrix. Furthermore, we show that learning of the new dedicated&#10;embedding matrix in the target language can be improved by leveraging a small&#10;number of vocabulary items (i.e., the so-called lexically overlapping tokens)&#10;shared between mBERT's and target language vocabulary. Our adaptation&#10;techniques offer substantial performance gains for languages with unseen&#10;scripts. We also demonstrate that they can yield improvements for low-resource&#10;languages written in scripts covered by the pretrained model.&#10;" />
          <attvalue for="2" value="&#10;Massively multilingual language models pretrained on large multilingual data, such as multilingual BERT \cite[mBERT;][]{Devlin2019bert} and XLM-R \cite{Conneau2020xlm-r} are the current state-of-the-art vehicle for effective cross-lingual transfer \cite{Hu2020xtreme}. However, while they exhibit strong transfer performance between resource-rich and similar languages \cite{Conneau2020xlm-r,Artetxe2020cross-lingual}, these models struggle with transfer to low-resource languages \cite{Wu:2020repl} and languages not represented at all in their pretraining corpora \cite{pfeiffer20madx,Muller20BeingUnseen,Ansell2021MADG}. The most extreme challenge is dealing with unseen languages with unseen scripts (i.e., the scripts are not represented in the pretraining data; see Figure \ref{fig:script_examples}), where the pretrained models are bound to fail entirely if they are used off-the-shelf without any further model adaptation.&#10;&#10;Existing work focuses on the embedding layer and learns either a new embedding matrix for the target language \cite{Artetxe2020cross-lingual} or adds new tokens to the pretrained vocabulary. While the former has only been applied to high-resource languages, the latter approaches have been limited to languages with seen scripts \cite{ChauLS20Parsing, Muller20BeingUnseen} and large pretraining corpora \cite{Wang20ExtendmBERT}. Another line of work adapts the embedding layer as well as other layers of the model via adapters \cite{pfeiffer20madx, ustun2020udapter}. Such methods, however, cannot be directly applied to languages with unseen scripts.&#10;&#10;In this work, we first empirically verify that the original tokenizer and the original embedding layer of a pretrained multilingual model fail for languages with unseen script. This implies that dedicated in-language tokenizers and embeddings are a crucial requirement for any successful model adaptation. The key challenge is aligning new target language embeddings to the pretrained model's representations while leveraging knowledge encoded in the existing embedding matrix.&#10;We systematize existing approaches based on the pretrained information they utilize and identify lexically overlapping tokens that are present in both vocabularies as key carriers of such information \cite{Sogaard2018limitations}.&#10;We then present novel, effective, and data-efficient methods for adapting pretrained multilingual language models to resource-low languages written in different scripts. Beyond lexical overlap, our methods rely on factorized information from the embedding matrix and token groupings. &#10;&#10;We evaluate our approaches in the named entity recognition (NER) task on the standard WikiAnn dataset \cite{Rahimi2019massively} and Dependency Parsing \cite[DP;][]{nivre:2016}. We use 4 diverse resource-rich languages as source languages, and transfer to 17 and 6 resource-poor target languages respectively, including 5 languages with unseen scripts (Amharic, Tibetan, Khmer, Divehi, Sinhala).&#10;We show that our adaptation techniques offer unmatched performance for languages with unseen scripts. They also yield improvements for low-resource and under-represented languages written in scripts covered by the pretrained model. &#10;&#10;Contributions. 1)~We systematize and compare current model adaptation strategies for low-resource languages with seen and unseen scripts. 2)~We measure the impact of initialization when learning new embedding layers, and demonstrate that non-random initialization starting from a subset of seen lexical items (i.e., lexically overlapping vocabulary items) has a strong positive impact on task performance for resource-poor languages. 3)~We propose methods for learning low-dimensional embeddings, which reduce the number of trainable parameters and yield more efficient model adaptation. Our approach, based on matrix factorization and language clusters, extracts relevant information from the pretrained embedding matrix. 4)~We show that our methods outperform previous approaches with both resource-rich and resource-poor languages. They substantially reduce the gap between random and lexically-overlapping initialization, enabling better model adaption to unseen scripts.&#10;&#10;The code for this work is released at&#10;\href{ Adapter-Hub/UNKs\_everywhere}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Low-Resource Languages, Cross-Lingual Transfer, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2010.12858" label="2010.12858">
        <attvalues>
          <attvalue for="0" value="When Being Unseen from mBERT is just the Beginning: Handling New&#10;  Languages With Multilingual Language Models" />
          <attvalue for="1" value="  Transfer learning based on pretraining language models on a large amount of&#10;raw data has become a new norm to reach state-of-the-art performance in NLP.&#10;Still, it remains unclear how this approach should be applied for unseen&#10;languages that are not covered by any available large-scale multilingual&#10;language model and for which only a small amount of raw data is generally&#10;available. In this work, by comparing multilingual and monolingual models, we&#10;show that such models behave in multiple ways on unseen languages. Some&#10;languages greatly benefit from transfer learning and behave similarly to&#10;closely related high resource languages whereas others apparently do not.&#10;Focusing on the latter, we show that this failure to transfer is largely&#10;related to the impact of the script used to write such languages.&#10;Transliterating those languages improves very significantly the ability of&#10;large-scale multilingual language models on downstream tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models are now a new standard to build state-of-the-art Natural Language Processing (NLP) systems. In the past year, monolingual language models have been released for more than 20 languages including Arabic, French, German, and Italian~\cite[inter alia]{antoun2020arabert,martin-etal-2020-camembert,de2019bertje,canete-2020-beto,kuratov-2019-rubert,schweter-2020-berturk}. Additionally, large-scale multilingual models covering more than 100 languages are now available (\xlmr by \cite{conneau-etal-2020-unsupervised} and \mbert by \cite{devlin-etal-2019-bert}). &#10;Still, most of the 6500+ spoken languages in the world~\cite{hammarstrom2016linguistic}&#10;are not covered---remaining unseen---by those models. Even languages with millions of native speakers like Sorani Kurdish (about 7 million speakers in the Middle East) or Bambara (spoken by around 5 million people in Mali and neighboring countries) are not covered by any available language models at the time of writing. &#10;&#10;Even if training multilingual models that cover more languages and language varieties is tempting, the curse of multilinguality \cite{conneau-etal-2020-unsupervised} makes it an impractical solution, as it would require to train ever larger models. Furthermore, as shown by \cite{wu-dredze-2020-languages}, large-scale multilingual language models are sub-optimal\draftremove{performance} for languages that \draftreplace{only account for a small portion of the pretraining.}{are under-sampled during pretraining.} &#10;&#10;In this paper, we analyze task and language adaptation experiments to get usable language model-based representations for under-studied low resource languages. &#10;We run experiments on 15 typologically diverse \draftremove{unseen}languages on three NLP tasks: part-of-speech (POS) tagging, dependency parsing (DEP) and named-entity recognition (NER). &#10;&#10;Our results bring forth a diverse set of behaviors that we classify in three categories reflecting the abilities of pretrained multilingual language models to be used for low-resource languages\draftreplace{.}{. We dub those categories Easy, Intermediate and Hard.}\draftremove{Some languages, the ``Easy'' ones}&#10;&#10;Hard languages include both stable and endangered languages, but they predominantly are languages of communities that are majorly under-served by modern NLP. Hence, we direct our attention to these Hard languages.&#10;For those languages, we show that the script they are written in can be a critical element in the transfer abilities of pretrained multilingual language models. \draftnote{Not really... -BS} &#10;Transliterating them leads to large gains in performance \draftremove{leading to }outperforming non-contextual strong baselines. To sum up, our contributions are the following:&#10;&#10;\begin{itemize}&#10; \item \draftremove{Based on our empirical results, }We propose a new categorization of the low-resource languages that are unseen by available language models: the Hard, the Intermediate and the Easy languages. \item We show that Hard languages can be better addressed by transliterating them into a better-handled script (typically Latin), providing a promising direction towards making multilingual language models useful for a new set of unseen languages. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.13640" label="2004.13640">
        <attvalues>
          <attvalue for="0" value="Extending Multilingual BERT to Low-Resource Languages" />
          <attvalue for="1" value="  Multilingual BERT (M-BERT) has been a huge success in both supervised and&#10;zero-shot cross-lingual transfer learning. However, this success has focused&#10;only on the top 104 languages in Wikipedia that it was trained on. In this&#10;paper, we propose a simple but effective approach to extend M-BERT (E-BERT) so&#10;that it can benefit any new language, and show that our approach benefits&#10;languages that are already in M-BERT as well. We perform an extensive set of&#10;experiments with Named Entity Recognition (NER) on 27 languages, only 16 of&#10;which are in M-BERT, and show an average increase of about 6% F1 on languages&#10;that are already in M-BERT and 23% F1 increase on new languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.00193" label="1902.00193">
        <attvalues>
          <attvalue for="0" value="Massively Multilingual Transfer for NER" />
          <attvalue for="1" value="  In cross-lingual transfer, NLP models over one or more source languages are&#10;applied to a low-resource target language. While most prior work has used a&#10;single source model or a few carefully selected models, here we consider a&#10;`massive' setting with many such models. This setting raises the problem of&#10;poor transfer, particularly from distant languages. We propose two techniques&#10;for modulating the transfer, suitable for zero-shot or few-shot learning,&#10;respectively. Evaluating on named entity recognition, we show that our&#10;techniques are much more effective than strong baselines, including standard&#10;ensembling, and our unsupervised method rivals oracle selection of the single&#10;best individual model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.00613" label="2008.00613">
        <attvalues>
          <attvalue for="0" value="Exploiting Deep Sentential Context for Expressive End-to-End Speech&#10;  Synthesis" />
          <attvalue for="1" value="  Attention-based seq2seq text-to-speech systems, especially those use&#10;self-attention networks (SAN), have achieved state-of-art performance. But an&#10;expressive corpus with rich prosody is still challenging to model as 1)&#10;prosodic aspects, which span across different sentential granularities and&#10;mainly determine acoustic expressiveness, are difficult to quantize and label&#10;and 2) the current seq2seq framework extracts prosodic information solely from&#10;a text encoder, which is easily collapsed to an averaged expression for&#10;expressive contents. In this paper, we propose a context extractor, which is&#10;built upon SAN-based text encoder, to sufficiently exploit the sentential&#10;context over an expressive corpus for seq2seq-based TTS. Our context extractor&#10;first collects prosodic-related sentential context information from different&#10;SAN layers and then aggregates them to learn a comprehensive sentence&#10;representation to enhance the expressiveness of the final generated speech.&#10;Specifically, we investigate two methods of context aggregation: 1) direct&#10;aggregation which directly concatenates the outputs of different SAN layers,&#10;and 2) weighted aggregation which uses multi-head attention to automatically&#10;learn contributions for different SAN layers. Experiments on two expressive&#10;corpora show that our approach can produce more natural speech with much richer&#10;prosodic variations, and weighted aggregation is more superior in modeling&#10;expressivity.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, the naturalness of corpus-based text-to-speech (TTS) has been significantly improved with the use of attention-based sequence-to-sequence (seq2seq) mapping framework~\cite{wang2017tacotron,shen2018natural}. Such so-called end-to-end (E2E) systems directly employ a text encoder network to learn linguistic, syntactic and semantic information from simple character or phoneme sequences. The sequence of aggregated textual representation is further attended by an acoustic decoder network through some attention mechanism, producing predicted speech representations (e.g., mel-spectrogram) that are subsequently transformed to waveforms via a neural vocoder.&#10;&#10;Sentential context~\cite{wang2019exploiting} mainly involves the latent syntactic and semantic information embedded in the text, recently proved to be important in natural language processing (NLP) tasks~\cite{wang2019exploiting,peters2018deep}. It might be essential to the naturalness of speech synthesis as well, especially for a system built upon an expressive corpus with rich prosodic variations. The seq2seq framework extracts prosodic information solely from the text encoder in an unsupervised way, which is easily collapsed to an averaged expression for expressive contents. To better make use of the sentential context in an E2E framework, one way is feature engineering as the previous generation of TTS does. For example, recent study has shown that exploiting syntactic features in a parsed tree is beneficial to the richness of the prosodic outcomes, leading to more natural synthesized speech~\cite{guo2019exploiting}.&#10;&#10;However, modeling expressiveness in text-to-speech is still challenging as it refers to different levels of syntactic and semantic information reflected in intensity, rhythm, intonation and other prosody related factors. However, it is difficult to define the relations explicitly between the syntactic/semantic factors and the prosodic factors. To model expressivity, the global style tokens (GST) family~\cite{wang2018style,an2019learning} learns style embeddings from a reference audio in an unsupervised way, which lets the synthesized speech imitate the style of reference audio. Although the style embeddings from a reference audio is helpful to control the style of synthesized speech, it is hard to choose an appropriate reference audio for each input sentence. Likewise, the variational autoencoder (VAE) models styles or expressivity in a similar way~\cite{zhang2019learning}.&#10;&#10;Recent studies have revealed that self-attention based networks (SAN)~\cite{li2019neural,yasuda2019investigation,yang2019enhancing,yang2019improving} have strong ability in capturing global prosodic information, leading to more natural synthesized speech. And unveiled by recent NLP tasks, different SAN encoder layers can capture latent syntactic and semantic properties of the input sentence at different levels~\cite{wang2019exploiting,peters2018deep}. But current SAN-based TTS systems only leverage the highly aggregated latent text representation, usually the outputs of the text encoder, from the simple textual input, to guide the speech generation process. Although the highly aggregated representation can be treated as a global description of the sentential context, it is not enough to generate expressive content according to our experiments as it may disperse the contribution of sentential context embedded in the intermediate SAN layers~\cite{shi2016does}.&#10;&#10;In this paper, to excavate the sentential context for expressive speech synthesis, we propose a context extractor to sufficiently exploit sentential context over an expressive corpus for seq2seq-based TTS. Specifically, we utilize different levels of representations from the SAN-based text encoder to build a context extractor, which is helpful to extract different levels of syntactic and semantic information~\cite{dou2018exploiting}. In details, our context extractor first collects the prosodic-related sentential context information from different SAN-based encoder layers, and then aggregates them to learn a comprehensive sentence representation to enhance the expressiveness of the final generated speech. Specifically, we investigate two methods of context aggregation: 1) direct aggregation which directly concatenates the outputs of different SAN layers, and 2) weighted aggregation which uses multi-head attention to automatically learn contributions for different SAN layers. Experiments on two expressive corpora show that our approach can produce more natural speech with richer prosodic variations, and weighted aggregation is more superior in modeling expressivity. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Self-Attention Networks, Text-to-Speech Systems, Artificial Intelligence, Signal Processing, Prosody Modeling" />
        </attvalues>
      </node>
      <node id="1906.01268" label="1906.01268">
        <attvalues>
          <attvalue for="0" value="Exploiting Sentential Context for Neural Machine Translation" />
          <attvalue for="1" value="  In this work, we present novel approaches to exploit sentential context for&#10;neural machine translation (NMT). Specifically, we first show that a shallow&#10;sentential context extracted from the top encoder layer only, can improve&#10;translation performance via contextualizing the encoding representations of&#10;individual words. Next, we introduce a deep sentential context, which&#10;aggregates the sentential context representations from all the internal layers&#10;of the encoder to form a more comprehensive context representation.&#10;Experimental results on the WMT14 English-to-German and English-to-French&#10;benchmarks show that our model consistently improves performance over the&#10;strong TRANSFORMER model (Vaswani et al., 2017), demonstrating the necessity&#10;and effectiveness of exploiting sentential context for NMT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.04764" label="1904.04764">
        <attvalues>
          <attvalue for="0" value="Exploiting Syntactic Features in a Parsed Tree to Improve End-to-End TTS" />
          <attvalue for="1" value="  The end-to-end TTS, which can predict speech directly from a given sequence&#10;of graphemes or phonemes, has shown improved performance over the conventional&#10;TTS. However, its predicting capability is still limited by the&#10;acoustic/phonetic coverage of the training data, usually constrained by the&#10;training set size. To further improve the TTS quality in pronunciation, prosody&#10;and perceived naturalness, we propose to exploit the information embedded in a&#10;syntactically parsed tree where the inter-phrase/word information of a sentence&#10;is organized in a multilevel tree structure. Specifically, two key features:&#10;phrase structure and relations between adjacent words are investigated.&#10;Experimental results in subjective listening, measured on three test sets, show&#10;that the proposed approach is effective to improve the pronunciation clarity,&#10;prosody and naturalness of the synthesized speech of the baseline system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.08895" label="1809.08895">
        <attvalues>
          <attvalue for="0" value="Neural Speech Synthesis with Transformer Network" />
          <attvalue for="1" value="  Although end-to-end neural text-to-speech (TTS) methods (such as Tacotron2)&#10;are proposed and achieve state-of-the-art performance, they still suffer from&#10;two problems: 1) low efficiency during training and inference; 2) hard to model&#10;long dependency using current recurrent neural networks (RNNs). Inspired by the&#10;success of Transformer network in neural machine translation (NMT), in this&#10;paper, we introduce and adapt the multi-head attention mechanism to replace the&#10;RNN structures and also the original attention mechanism in Tacotron2. With the&#10;help of multi-head self-attention, the hidden states in the encoder and decoder&#10;are constructed in parallel, which improves the training efficiency. Meanwhile,&#10;any two inputs at different times are connected directly by self-attention&#10;mechanism, which solves the long range dependency problem effectively. Using&#10;phoneme sequences as input, our Transformer TTS network generates mel&#10;spectrograms, followed by a WaveNet vocoder to output the final audio results.&#10;Experiments are conducted to test the efficiency and performance of our new&#10;network. For the efficiency, our Transformer TTS network can speed up the&#10;training about 4.25 times faster compared with Tacotron2. For the performance,&#10;rigorous human tests show that our proposed model achieves state-of-the-art&#10;performance (outperforms Tacotron2 with a gap of 0.048) and is very close to&#10;human quality (4.39 vs 4.44 in MOS).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.11960" label="1810.11960">
        <attvalues>
          <attvalue for="0" value="Investigation of enhanced Tacotron text-to-speech synthesis systems with&#10;  self-attention for pitch accent language" />
          <attvalue for="1" value="  End-to-end speech synthesis is a promising approach that directly converts&#10;raw text to speech. Although it was shown that Tacotron2 outperforms classical&#10;pipeline systems with regards to naturalness in English, its applicability to&#10;other languages is still unknown. Japanese could be one of the most difficult&#10;languages for which to achieve end-to-end speech synthesis, largely due to its&#10;character diversity and pitch accents. Therefore, state-of-the-art systems are&#10;still based on a traditional pipeline framework that requires a separate text&#10;analyzer and duration model. Towards end-to-end Japanese speech synthesis, we&#10;extend Tacotron to systems with self-attention to capture long-term&#10;dependencies related to pitch accents and compare their audio quality with&#10;classical pipeline systems under various conditions to show their pros and&#10;cons. In a large-scale listening test, we investigated the impacts of the&#10;presence of accentual-type labels, the use of force or predicted alignments,&#10;and acoustic features used as local condition parameters of the Wavenet&#10;vocoder. Our results reveal that although the proposed systems still do not&#10;match the quality of a top-line pipeline system for Japanese, we show important&#10;stepping stones towards end-to-end Japanese speech synthesis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.10181" label="1810.10181">
        <attvalues>
          <attvalue for="0" value="Exploiting Deep Representations for Neural Machine Translation" />
          <attvalue for="1" value="  Advanced neural machine translation (NMT) models generally implement encoder&#10;and decoder as multiple layers, which allows systems to model complex functions&#10;and capture complicated linguistic structures. However, only the top layers of&#10;encoder and decoder are leveraged in the subsequent process, which misses the&#10;opportunity to exploit the useful information embedded in other layers. In this&#10;work, we propose to simultaneously expose all of these signals with layer&#10;aggregation and multi-layer attention mechanisms. In addition, we introduce an&#10;auxiliary regularization term to encourage different layers to capture diverse&#10;information. Experimental results on widely-used WMT14 English-German and WMT17&#10;Chinese-English translation data demonstrate the effectiveness and universality&#10;of the proposed approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15996" label="2203.15996">
        <attvalues>
          <attvalue for="0" value="TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models" />
          <attvalue for="1" value="  Pre-trained language models have been prevailed in natural language&#10;processing and become the backbones of many NLP tasks, but the demands for&#10;computational resources have limited their applications. In this paper, we&#10;introduce TextPruner, an open-source model pruning toolkit designed for&#10;pre-trained language models, targeting fast and easy model compression.&#10;TextPruner offers structured post-training pruning methods, including&#10;vocabulary pruning and transformer pruning, and can be applied to various&#10;models and tasks. We also propose a self-supervised pruning method that can be&#10;applied without the labeled data. Our experiments with several NLP tasks&#10;demonstrate the ability of TextPruner to reduce the model size without&#10;re-training the model.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large pre-trained language models (PLMs) \cite{devlin-etal-2019-bert,liu2019roberta} have achieved great success in a variety of NLP tasks. However, it is difficult to deploy them for real-world applications where computation and memory resources are limited. Reducing the pre-trained model size and speeding up the inference have become a critical issue.&#10;&#10;Pruning is a common technique for model compression. It identifies and removes redundant or less important neurons from the networks. From the view of the model structure, pruning methods can be categorized into unstructured pruning and structured pruning. In the unstructured pruning, each model parameter is individually removed if it reaches some criteria based on the magnitude or importance score \cite{DBLP:journals/corr/HanPTD15, DBLP:conf/iclr/ZhuG18, DBLP:conf/nips/Sanh0R20}. The unstructured pruning results in sparse matrices and allows for significant model compression, but the inference speed can hardly be improved without specialized devices. While in the structured pruning, rows or columns of the parameters are removed from the weight matrices \cite{DBLP:journals/corr/abs-1910-06360,DBLP:conf/nips/MichelLN19,voita-etal-2019-analyzing,lagunas-etal-2021-block,DBLP:conf/nips/HouHSJCL20}. Thus, the resulting model speeds up on the common CPU and GPU devices.&#10;&#10;Pruning methods can also be classified into optimization-free methods \cite{DBLP:conf/nips/MichelLN19} and the ones that involve optimization \cite{frankle2018the, lagunas-etal-2021-block}. The latter usually achieves higher performance, but the former runs faster and is more convenient to use.&#10;&#10;Pruning PLMs has been of growing interest. Most of the works focus on reducing transformer size while ignoring the vocabulary \cite{abdaoui-etal-2020-load}. Pruning vocabulary can greatly reduce the model size for multilingual PLMs.&#10;&#10;In this paper, we present TextPruner, a model pruning toolkit for PLMs. It combines both transformer pruning and vocabulary pruning. The purpose of TextPruner is to offer a universal, fast, and easy-to-use tool for model compression. We expect it can be accessible to users with little model training experience. Therefore, we implement the structured optimization-free pruning methods for its convenient use and fast computation. Pruning a base-sized model only requires several minutes with TextPruner. TextPruner can also be a useful analysis tool for inspecting the importance of the neurons in the model.&#10;&#10;TextPruner has the following highlights:&#10;\begin{itemize}[noitemsep,topsep=0pt]&#10; \item TextPruner is designed to be easy to use. It provides both Python API and Command Line Interface (CLI). Working with either of them requires only a couple of lines of simple code. Besides, TextPruner is non-intrusive and compatible with Transformers \cite{wolf-etal-2020-transformers}, which means users do not have to change their models that are built on the Transformers library. &#10; \item TextPruner works with different models and tasks. It has been tested on tasks like text classification, machine reading comprehension (MRC), named entity recognition (NER). TextPruner is also designed to be extensible for other models.&#10; \item TextPruner is flexible. Users can control the pruning process and explore pruning strategies via tuning the configurations to find the optimal configurations for the specific tasks.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Model Compression Techniques, Artificial Intelligence, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1506.02626" label="1506.02626">
        <attvalues>
          <attvalue for="0" value="Learning both Weights and Connections for Efficient Neural Networks" />
          <attvalue for="1" value="  Neural networks are both computationally intensive and memory intensive,&#10;making them difficult to deploy on embedded systems. Also, conventional&#10;networks fix the architecture before training starts; as a result, training&#10;cannot improve the architecture. To address these limitations, we describe a&#10;method to reduce the storage and computation required by neural networks by an&#10;order of magnitude without affecting their accuracy by learning only the&#10;important connections. Our method prunes redundant connections using a&#10;three-step method. First, we train the network to learn which connections are&#10;important. Next, we prune the unimportant connections. Finally, we retrain the&#10;network to fine tune the weights of the remaining connections. On the ImageNet&#10;dataset, our method reduced the number of parameters of AlexNet by a factor of&#10;9x, from 61 million to 6.7 million, without incurring accuracy loss. Similar&#10;experiments with VGG-16 found that the number of parameters can be reduced by&#10;13x, from 138 million to 10.3 million, again with no loss of accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04838" label="2109.04838">
        <attvalues>
          <attvalue for="0" value="Block Pruning For Faster Transformers" />
          <attvalue for="1" value="  Pre-training has improved model accuracy for both classification and&#10;generation tasks at the cost of introducing much larger and slower models.&#10;Pruning methods have proven to be an effective way of reducing model size,&#10;whereas distillation methods are proven for speeding up inference. We introduce&#10;a block pruning approach targeting both small and fast models. Our approach&#10;extends structured methods by considering blocks of any size and integrates&#10;this structure into the movement pruning paradigm for fine-tuning. We find that&#10;this approach learns to prune out full components of the underlying model, such&#10;as attention heads. Experiments consider classification and generation tasks,&#10;yielding among other results a pruned model that is a 2.4x faster, 74% smaller&#10;BERT on SQuAD v1, with a 1% drop on F1, competitive both with distilled models&#10;in speed and pruned models in size.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.07103" label="2401.07103">
        <attvalues>
          <attvalue for="0" value="Leveraging Large Language Models for NLG Evaluation: Advances and&#10;  Challenges" />
          <attvalue for="1" value="  In the rapidly evolving domain of Natural Language Generation (NLG)&#10;evaluation, introducing Large Language Models (LLMs) has opened new avenues for&#10;assessing generated content quality, e.g., coherence, creativity, and context&#10;relevance. This paper aims to provide a thorough overview of leveraging LLMs&#10;for NLG evaluation, a burgeoning area that lacks a systematic analysis. We&#10;propose a coherent taxonomy for organizing existing LLM-based evaluation&#10;metrics, offering a structured framework to understand and compare these&#10;methods. Our detailed exploration includes critically assessing various&#10;LLM-based methodologies, as well as comparing their strengths and limitations&#10;in evaluating NLG outputs. By discussing unresolved challenges, including bias,&#10;robustness, domain-specificity, and unified evaluation, this paper seeks to&#10;offer insights to researchers and advocate for fairer and more advanced NLG&#10;evaluation techniques.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Generation (NLG) stands at the forefront of modern AI-driven communication, with recent advancements in large language models (LLMs) revolutionizing the capabilities of NLG systems~\cite{ouyang2022training, openai2023gpt4}. These models, powered by deep learning techniques and vast amounts of training data, exhibit excellent proficiency in generating text across a wide range of applications. As NLG technology continues its rapid evolution, it becomes increasingly imperative to establish robust evaluation methodologies that can reliably gauge the quality of the generated content. &#10;&#10;Traditional NLG evaluation metrics, such as BLEU~\cite{Papineni2002Bleu}, ROUGE~\cite{Lin2004Rouge} and TER~\cite{snover2006study}, primarily focus on surface-level text differences and often fall short in assessing semantic aspects~\cite{freitag-etal-2020-bleu}. This limitation has been noted to hinder research progress and can lead to misleading research conclusions. &#10;Additionally, other methods that employ neural embeddings to calculate the score~\cite{liu2016not,sellam-etal-2020-bleurt,Zhang2020BERTScore}, despite assessing aspects like semantic equivalence and fluency, are inflexible and limited in scope~\cite{freitag-etal-2021-experts}. &#10;Additionally, these traditional methods tend to have low alignment with human judgement~\cite{liu2023gpteval} and lack interpretability for the score~\cite{xu2023instructscore}.&#10;These drawbacks underscore the need for more nuanced and comprehensive evaluation methods in the NLG field.&#10;&#10;The emergent abilities of LLMs present a promising avenue for the LLM-based NLG evaluation, such as Chain-of-Thought (CoT)~\cite{wei2022chain}, zero-shot instruction following~\cite{DBLP:conf/iclr/WeiBZGYLDDL22}, better alignment with human preference~\cite{ouyang2022training}, etc. These attributes position LLMs as potent tools for evaluating NLG outputs, offering a more sophisticated and better human-aligned assessment compared to traditional methods~\cite{liu2023gpteval, kocmi-federmann-2023-large, fu2023gptscore}. &#10;For instance, LLMs could generate reasonable explanations to support the ultimate score~\cite{xu2023instructscore}, and the reinforcement learning with human feedback (RLHF) could align LLMs' preference with human better~\cite{ouyang2022training, zheng2023judging}.&#10;As in Figure~\ref{fig:illu}, the key strategy in these approaches involves instructing LLMs with prompts to evaluate generated texts from various aspects, either with references and sources or not. However, the wide array of LLM-based NLG evaluation methods, addressing different tasks and goals, lack a unified overview. &#10;&#10;Given the burgeoning volume of work in the realm of LLMs for NLG evaluation, a synthesized summary is urgently needed to navigate the complexities and diverse methodologies within this space. This survey aims to provide a comprehensive overview of this promising domain, presenting a coherent taxonomy for organizing existing works. We meticulously delineate pivotal studies and their methodologies, and delve into an analytical discussion of the various strengths, limitations, and distinctive attributes of these approaches. Furthermore, we navigate through the yet-to-be-resolved challenges and the open-ended questions within this field, thereby charting potential avenues for future scholarly exploration. This comprehensive exploration aims to spark readers with an in-depth understanding of the nuances and evolving dynamics of LLM-based approaches in NLG evaluation.&#10;&#10;Organization of this paper: We present the first comprehensive survey of recent advancements in leveraging LLMs for NLG evaluation. Initially, we establish a formal framework for NLG evaluation and propose a taxonomy to categorize relevant works (Section \ref{sec:taxonomy}). Subsequently, we delve into and elaborate on these works in detail (Section \ref{sec:generative}). Furthermore, we conduct a systematic review of various meta-evaluation benchmarks that assess the efficacy of LLM-based evaluators (Section \ref{sec:benchmark}). Additionally, we provide a thorough comparison of LLM-based evaluators with traditional evaluators in terms of performance, efficiency and qualitative qualitative analysis (Section \ref{sec:comparison}).&#10;In recognition of the rapid evolution of this field, we identify and discuss several potential open problems that may guide future research (Section \ref{sec:future}). To conclude, we advocate for the advancement of this field through the development of more impartial, robust, expert and unified LLM-based evaluators. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Evaluation Methodologies, Linguistics, Cognitive Science, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2403.04190" label="2403.04190">
        <attvalues>
          <attvalue for="0" value="Generative AI for Synthetic Data Generation: Methods, Challenges and the&#10;  Future" />
          <attvalue for="1" value="  The recent surge in research focused on generating synthetic data from large&#10;language models (LLMs), especially for scenarios with limited data&#10;availability, marks a notable shift in Generative Artificial Intelligence (AI).&#10;Their ability to perform comparably to real-world data positions this approach&#10;as a compelling solution to low-resource challenges. This paper delves into&#10;advanced technologies that leverage these gigantic LLMs for the generation of&#10;task-specific training data. We outline methodologies, evaluation techniques,&#10;and practical applications, discuss the current limitations, and suggest&#10;potential pathways for future research.&#10;" />
          <attvalue for="2" value="&#10;The introduction of Transformer \cite{vaswani2023attention} in 2017, followed by groundbreaking LLMs like OpenAI's GPT \cite{brown2020language} and Google's BERT \cite{devlin-etal-2019-bert}, marked the beginning of a new era in language understanding and generation. More recently, generative LLMs (e.g., GPT-3\cite{kojima2023large}, LlaMa\cite{touvron2023llama} and ChatGPT\cite{chatgpt}) have propelled this evolution to unprecedented heights, seamlessly converging with Generative AI and heralding a fresh era in the realm of synthetic data generation\cite{meng2023tuning,meng2022generating,zerogen,gao2023selfguided,ye-etal-2022-progen,regen,chen-etal-2023-mixture}. &#10;&#10;The origins of Generative AI can be traced back to pivotal models such as Generative Adversarial Networks\cite{goodfellow2014generative} (GANs) and Variational Autoencoders\cite{kingma2022autoencoding} (VAEs), which demonstrated the ability to generate realistic images and signals\cite{wu2020logan}. However, it wasn't until the advent of LLMs in recent years that Generative AI truly began to flourish. These LLMs, trained on vast datasets, showcased an unprecedented ability to produce coherent and contextually relevant text, pushing the boundaries of what AI could achieve in language-related tasks. The convergence of Generative AI and LLMs in the realm of synthetic data creation represents not merely a technological advancement, but a profound paradigm shift in our approach to data creation and the training of AI models.&#10;&#10;Why do we need synthetic data? The necessity for synthetic data arises from the inherent limitations of general-purpose Large Language Models (LLMs) in specialized and private domains, despite their significant achievements across various benchmarks. For instance, ClinicalBERT\cite{huang2019clinicalbert}, adapted from BERT through pre-training on clinical texts, demonstrates superior performance in predicting hospital readmissions compared to the original BERT\cite{devlin2018bert}, which was trained on Wikipedia and BookCorpus\cite{zhu2015aligning} text data. This highlights a crucial challenge: specialized domains often rely on domain-specific data that is not readily available or open to the public, thereby underscoring the importance of synthetic data in bridging these gaps.&#10;&#10;Synergy between LLMs and synthetic data generation.&#10;Large Language Models (LLMs) for synthetic data generation marks a significant frontier in the field of AI. LLMs, such as ChatGPT, have revolutionized our approach to understanding and generating human-like text, providing a mechanism to create rich, contextually relevant synthetic data on an unprecedented scale. This synergy is pivotal in addressing data scarcity and privacy concerns, particularly in domains where real data is either limited or sensitive. By generating text that closely mirrors human language, LLMs facilitate the creation of robust, varied datasets necessary for training and refining AI models across various applications, from healthcare\cite{peng2023study}, eduction\cite{moore2023empowering} to business management\cite{rane2023role}. Moreover, this collaboration opens new avenues for ethical AI development, allowing researchers to bypass the biases and ethical dilemmas often inherent in real-world datasets. The integration of LLMs in synthetic data generation not only pushes the boundaries of what's achievable in AI but also ensures a more responsible and inclusive approach to AI development, aligning with evolving ethical standards and societal needs.&#10;&#10;Other related survey papers. Comprehensive surveys for Generative AI and LLMs exist, each revisits related works from a different perspective: Generative AI surveys provide a holistic view of this area starting from Generative Adversarial Networks (GANs) to ChatGPT \cite{cao2023comprehensive} and models developed for synthetic data generation in the past decade \cite{bauer2024comprehensive}, with a special focus on text-to-image \cite{zhang2023text} or text-to-speech \cite{zhang2023survey} generation as well as practical applications in Education \cite{baidoo2023education} and Healthcare \cite{yu2023leveraging}; Surveys for LLMs provide systematic categorization \cite{qiu2020pre} for NLP tasks \cite{min2023recent} and methods to adapt these LLMs to specific domains \cite{10356_167965} through model optimization and personalization perspectives \cite{guo2022domain}. Surveys on LLMs for text generation \cite{li2022pretrained} focus on developing generative LLMs including model architecture choices and training techniques and do not contain gigantic LLMs released in the past two years. Unlike these survey papers, this paper mainly focuses on recent technologies that employ generative LLMs without training them for synthetic training data generation and elicit their potential impact on practical adoption.&#10;&#10;Outline of this paper. The following of this paper is organized as follows. Section \ref{sec:mthods} introduces recent methods for generating synthetic data from LLMs. Specifically, we summarize prompt engineering techniques that are particularly designed for probing LLMs to obtain desired data in sub-section\ref{sec:methods-prompt} while in sub-section\ref{sec:methods-adaptation}, we talk about how to employ parameter-efficient methods to adapt LLMs for generating task-related data; In sub-sections \ref{sec:methods-quality} and \ref{sec:methods-training} we introduce methods that can measure the quality of the synthetic dataset and how to effectively make use of the data for training. Section \ref{sec:application} details the application of synthetic data, focusing on its utilization in low-resource tasks in Sub-Section \ref{sec:application-low} and practical deployment scenarios in Sub-Section \ref{sec:application-fast}. Additionally, Sub-Section \ref{sec:application-medical} provides a specific case study on the use of synthetic data within medical domains. Finally, in Section \ref{sec:challenges}, we underscore some prominent challenges in synthetic data and discuss potential avenues for future research.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Synthetic Data Generation, Computer Science, Large Language Models, Linguistics, Generative AI, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2202.04538" label="2202.04538">
        <attvalues>
          <attvalue for="0" value="Generating Training Data with Language Models: Towards Zero-Shot&#10;  Language Understanding" />
          <attvalue for="1" value="  Pretrained language models (PLMs) have demonstrated remarkable performance in&#10;various natural language processing tasks: Unidirectional PLMs (e.g., GPT) are&#10;well known for their superior text generation capabilities; bidirectional PLMs&#10;(e.g., BERT) have been the prominent choice for natural language understanding&#10;(NLU) tasks. While both types of models have achieved promising few-shot&#10;learning performance, their potential for zero-shot learning has been&#10;underexplored. In this paper, we present a simple approach that uses both types&#10;of PLMs for fully zero-shot learning of NLU tasks without requiring any&#10;task-specific data: A unidirectional PLM generates class-conditioned texts&#10;guided by prompts, which are used as the training data for fine-tuning a&#10;bidirectional PLM. With quality training data selected based on the generation&#10;probability and regularization techniques (label smoothing and temporal&#10;ensembling) applied to the fine-tuning stage for better generalization and&#10;stability, our approach demonstrates strong performance across seven&#10;classification tasks of the GLUE benchmark (e.g., 72.3/73.8 on MNLI-m/mm and&#10;92.8 on SST-2), significantly outperforming zero-shot prompting methods and&#10;achieving even comparable results to strong few-shot approaches using 32&#10;training samples per class.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.07922" label="2202.07922">
        <attvalues>
          <attvalue for="0" value="ZeroGen: Efficient Zero-shot Learning via Dataset Generation" />
          <attvalue for="1" value="  There is a growing interest in dataset generation recently due to the&#10;superior generative capacity of large pre-trained language models (PLMs). In&#10;this paper, we study a flexible and efficient zero-short learning method,&#10;\textsc{ZeroGen}. Given a zero-shot task, we first generate a dataset from&#10;scratch using PLMs in an unsupervised manner. Then, we train a tiny task model&#10;(e.g., LSTM) under the supervision of the synthesized dataset. This approach&#10;allows highly efficient inference as the final task model only has orders of&#10;magnitude fewer parameters comparing to PLMs (e.g., GPT2-XL). Apart from being&#10;annotation-free and efficient, we argue that \textsc{ZeroGen} can also provide&#10;useful insights from the perspective of data-free model-agnostic knowledge&#10;distillation, and unreferenced text generation evaluation. Experiments and&#10;analysis on different NLP tasks, namely, text classification, question&#10;answering, and natural language inference, show the effectiveness of&#10;\textsc{ZeroGen}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10703" label="2305.10703">
        <attvalues>
          <attvalue for="0" value="ReGen: Zero-Shot Text Classification via Training Data Generation with&#10;  Progressive Dense Retrieval" />
          <attvalue for="1" value="  With the development of large language models (LLMs), zero-shot learning has&#10;attracted much attention for various NLP tasks. Different from prior works that&#10;generate training data with billion-scale natural language generation (NLG)&#10;models, we propose a retrieval-enhanced framework to create training data from&#10;a general-domain unlabeled corpus. To realize this, we first conduct&#10;contrastive pretraining to learn an unsupervised dense retriever for extracting&#10;the most relevant documents using class-descriptive verbalizers. We then&#10;further propose two simple strategies, namely Verbalizer Augmentation with&#10;Demonstrations and Self-consistency Guided Filtering to improve the topic&#10;coverage of the dataset while removing noisy examples. Experiments on nine&#10;datasets demonstrate that REGEN achieves 4.3% gain over the strongest baselines&#10;and saves around 70% of the time compared to baselines using large NLG models.&#10;Besides, REGEN can be naturally integrated with recently proposed large&#10;language models to boost performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.00953" label="1912.00953">
        <attvalues>
          <attvalue for="0" value="LOGAN: Latent Optimisation for Generative Adversarial Networks" />
          <attvalue for="1" value="  Training generative adversarial networks requires balancing of delicate&#10;adversarial dynamics. Even with careful tuning, training may diverge or end up&#10;in a bad equilibrium with dropped modes. In this work, we improve CS-GAN with&#10;natural gradient-based latent optimisation and show that it improves&#10;adversarial dynamics by enhancing interactions between the discriminator and&#10;the generator. Our experiments demonstrate that latent optimisation can&#10;significantly improve GAN training, obtaining state-of-the-art performance for&#10;the ImageNet ($128 \times 128$) dataset. Our model achieves an Inception Score&#10;(IS) of $148$ and an Fr\'echet Inception Distance (FID) of $3.4$, an&#10;improvement of $17\%$ and $32\%$ in IS and FID respectively, compared with the&#10;baseline BigGAN-deep model with the same architecture and number of parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.05342" label="1904.05342">
        <attvalues>
          <attvalue for="0" value="ClinicalBERT: Modeling Clinical Notes and Predicting Hospital&#10;  Readmission" />
          <attvalue for="1" value="  Clinical notes contain information about patients that goes beyond structured&#10;data like lab values and medications. However, clinical notes have been&#10;underused relative to structured data, because notes are high-dimensional and&#10;sparse. This work develops and evaluates representations of clinical notes&#10;using bidirectional transformers (ClinicalBERT). ClinicalBERT uncovers&#10;high-quality relationships between medical concepts as judged by humans.&#10;ClinicalBert outperforms baselines on 30-day hospital readmission prediction&#10;using both discharge summaries and the first few days of notes in the intensive&#10;care unit. Code and model parameters are available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.06724" label="1506.06724">
        <attvalues>
          <attvalue for="0" value="Aligning Books and Movies: Towards Story-like Visual Explanations by&#10;  Watching Movies and Reading Books" />
          <attvalue for="1" value="  Books are a rich source of both fine-grained information, how a character, an&#10;object or a scene looks like, as well as high-level semantics, what someone is&#10;thinking, feeling and how these states evolve through a story. This paper aims&#10;to align books to their movie releases in order to provide rich descriptive&#10;explanations for visual content that go semantically far beyond the captions&#10;available in current datasets. To align movies and books we exploit a neural&#10;sentence embedding that is trained in an unsupervised way from a large corpus&#10;of books, as well as a video-text neural embedding for computing similarities&#10;between movie clips and sentences in the book. We propose a context-aware CNN&#10;to combine information from multiple sources. We demonstrate good quantitative&#10;performance for movie/book alignment and show several qualitative examples that&#10;showcase the diversity of tasks our model can be used for.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13523" label="2305.13523">
        <attvalues>
          <attvalue for="0" value="A Study of Generative Large Language Model for Medical Research and&#10;  Healthcare" />
          <attvalue for="1" value="  There is enormous enthusiasm and concerns in using large language models&#10;(LLMs) in healthcare, yet current assumptions are all based on general-purpose&#10;LLMs such as ChatGPT. This study develops a clinical generative LLM,&#10;GatorTronGPT, using 277 billion words of mixed clinical and English text with a&#10;GPT-3 architecture of 20 billion parameters. GatorTronGPT improves biomedical&#10;natural language processing for medical research. Synthetic NLP models trained&#10;using GatorTronGPT generated text outperform NLP models trained using&#10;real-world clinical text. Physicians Turing test using 1 (worst) to 9 (best)&#10;scale shows that there is no significant difference in linguistic readability&#10;(p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical&#10;relevance (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that&#10;physicians cannot differentiate them (p &lt; 0.001). This study provides insights&#10;on the opportunities and challenges of LLMs for medical research and&#10;healthcare.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.04226" label="2303.04226">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey of AI-Generated Content (AIGC): A History of&#10;  Generative AI from GAN to ChatGPT" />
          <attvalue for="1" value="  Recently, ChatGPT, along with DALL-E-2 and Codex,has been gaining significant&#10;attention from society. As a result, many individuals have become interested in&#10;related resources and are seeking to uncover the background and secrets behind&#10;its impressive performance. In fact, ChatGPT and other Generative AI (GAI)&#10;techniques belong to the category of Artificial Intelligence Generated Content&#10;(AIGC), which involves the creation of digital content, such as images, music,&#10;and natural language, through AI models. The goal of AIGC is to make the&#10;content creation process more efficient and accessible, allowing for the&#10;production of high-quality content at a faster pace. AIGC is achieved by&#10;extracting and understanding intent information from instructions provided by&#10;human, and generating the content according to its knowledge and the intent&#10;information. In recent years, large-scale models have become increasingly&#10;important in AIGC as they provide better intent extraction and thus, improved&#10;generation results. With the growth of data and the size of the models, the&#10;distribution that the model can learn becomes more comprehensive and closer to&#10;reality, leading to more realistic and high-quality content generation. This&#10;survey provides a comprehensive review on the history of generative models, and&#10;basic components, recent advances in AIGC from unimodal interaction and&#10;multimodal interaction. From the perspective of unimodality, we introduce the&#10;generation tasks and relative models of text and image. From the perspective of&#10;multimodality, we introduce the cross-application between the modalities&#10;mentioned above. Finally, we discuss the existing open problems and future&#10;challenges in AIGC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.02524" label="2401.02524">
        <attvalues>
          <attvalue for="0" value="Comprehensive Exploration of Synthetic Data Generation: A Survey" />
          <attvalue for="1" value="  Recent years have witnessed a surge in the popularity of Machine Learning&#10;(ML), applied across diverse domains. However, progress is impeded by the&#10;scarcity of training data due to expensive acquisition and privacy legislation.&#10;Synthetic data emerges as a solution, but the abundance of released models and&#10;limited overview literature pose challenges for decision-making. This work&#10;surveys 417 Synthetic Data Generation (SDG) models over the last decade,&#10;providing a comprehensive overview of model types, functionality, and&#10;improvements. Common attributes are identified, leading to a classification and&#10;trend analysis. The findings reveal increased model performance and complexity,&#10;with neural network-based approaches prevailing, except for privacy-preserving&#10;data generation. Computer vision dominates, with GANs as primary generative&#10;models, while diffusion models, transformers, and RNNs compete. Implications&#10;from our performance evaluation highlight the scarcity of common metrics and&#10;datasets, making comparisons challenging. Additionally, the neglect of training&#10;and computational costs in literature necessitates attention in future&#10;research. This work serves as a guide for SDG model selection and identifies&#10;crucial areas for future exploration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.13336" label="2303.13336">
        <attvalues>
          <attvalue for="0" value="A Survey on Audio Diffusion Models: Text To Speech Synthesis and&#10;  Enhancement in Generative AI" />
          <attvalue for="1" value="  Generative AI has demonstrated impressive performance in various fields,&#10;among which speech synthesis is an interesting direction. With the diffusion&#10;model as the most popular generative model, numerous works have attempted two&#10;active tasks: text to speech and speech enhancement. This work conducts a&#10;survey on audio diffusion model, which is complementary to existing surveys&#10;that either lack the recent progress of diffusion-based speech synthesis or&#10;highlight an overall picture of applying diffusion model in multiple fields.&#10;Specifically, this work first briefly introduces the background of audio and&#10;diffusion model. As for the text-to-speech task, we divide the methods into&#10;three categories based on the stage where diffusion model is adopted: acoustic&#10;model, vocoder and end-to-end framework. Moreover, we categorize various speech&#10;enhancement tasks by either certain signals are removed or added into the input&#10;speech. Comparisons of experimental results and discussions are also covered in&#10;this survey.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.08940" label="1909.08940">
        <attvalues>
          <attvalue for="0" value="Improving Generalization by Incorporating Coverage in Natural Language&#10;  Inference" />
          <attvalue for="1" value="  The task of natural language inference (NLI) is to identify the relation&#10;between the given premise and hypothesis. While recent NLI models achieve very&#10;high performance on individual datasets, they fail to generalize across similar&#10;datasets. This indicates that they are solving NLI datasets instead of the task&#10;itself. In order to improve generalization, we propose to extend the input&#10;representations with an abstract view of the relation between the hypothesis&#10;and the premise, i.e., how well the individual words, or word n-grams, of the&#10;hypothesis are covered by the premise. Our experiments show that the use of&#10;this information considerably improves generalization across different NLI&#10;datasets without requiring any external knowledge or additional data. Finally,&#10;we show that using the coverage information is not only beneficial for&#10;improving the performance across different datasets of the same task. The&#10;resulting generalization improves the performance across datasets that belong&#10;to similar but not the same tasks.&#10;" />
          <attvalue for="2" value="&#10;The task of Natural language inference (NLI) \cite{W03-0906,dagan2006pascal,D15-1075} is to specify whether the given hypothesis entails, contradicts, or is neutral regarding the premise.&#10;While existing NLI models have high performances on individual datasets, they fail to generalize across different datasets of the same task.&#10;This indicates that existing models are overfitting to specific properties of each dataset instead of learning the higher-level inference knowledge that is required to solve the task. &#10;&#10;Existing solutions to improve the performance of NLI models across datasets include (1) using external knowledge \cite{joshi2018pair2vec,P18-1224}, and (2) fine-tuning on the target datasets, e.g., \cite{bansal:AAAI:2019,liu2019inoculation}.&#10;The above approaches include the use of additional data or knowledge sources. Besides, fine-tuning on one target dataset may decrease the performance on the other datasets \cite{bansal:AAAI:2019}.&#10;&#10;In this paper, we propose a simple approach to improve generalization by using the information that is already present in the dataset.&#10;Existing NLI methods overly rely on the lexical form of the inputs, assuming that the model itself learns the required abstractions.&#10;Instead, we propose to extend the existing input representations with a more abstract view of the relation between the hypothesis and premise, i.e., how well each word, or word n-gram, of the hypothesis is covered by the premise.&#10;Our experiments show that using this information considerably and consistently improves generalization across various NLI datasets, i.e., from one to 30 points improvements.&#10;&#10;Finally, we show that beside improving the performance across different datasets of the same task, our proposed approach also improves the performance across datasets of similar tasks, i.e., reading comprehension on the SQuAD dataset and QA-SRL \cite{he-etal-2015-question}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Text Representation Models, Artificial Intelligence, Natural Language Inference, Generalization Techniques" />
        </attvalues>
      </node>
      <node id="1810.08854" label="1810.08854">
        <attvalues>
          <attvalue for="0" value="pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence&#10;  Inference" />
          <attvalue for="1" value="  Reasoning about implied relationships (e.g., paraphrastic, common sense,&#10;encyclopedic) between pairs of words is crucial for many cross-sentence&#10;inference problems. This paper proposes new methods for learning and using&#10;embeddings of word pairs that implicitly represent background knowledge about&#10;such relationships. Our pairwise embeddings are computed as a compositional&#10;function on word representations, which is learned by maximizing the pointwise&#10;mutual information (PMI) with the contexts in which the two words co-occur. We&#10;add these representations to the cross-sentence attention layer of existing&#10;inference models (e.g. BiDAF for QA, ESIM for NLI), instead of extending or&#10;replacing existing word embeddings. Experiments show a gain of 2.7% on the&#10;recently released SQuAD2.0 and 1.3% on MultiNLI. Our representations also aid&#10;in better generalization with gains of around 6-7% on adversarial SQuAD&#10;datasets, and 8.8% on the adversarial entailment test set by Glockner et al.&#10;(2018).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.04289" label="1711.04289">
        <attvalues>
          <attvalue for="0" value="Neural Natural Language Inference Models Enhanced with External&#10;  Knowledge" />
          <attvalue for="1" value="  Modeling natural language inference is a very challenging task. With the&#10;availability of large annotated data, it has recently become feasible to train&#10;complex models such as neural-network-based inference models, which have shown&#10;to achieve the state-of-the-art performance. Although there exist relatively&#10;large annotated data, can machines learn all knowledge needed to perform&#10;natural language inference (NLI) from these data? If not, how can&#10;neural-network-based NLI models benefit from external knowledge and how to&#10;build NLI models to leverage it? In this paper, we enrich the state-of-the-art&#10;neural natural language inference models with external knowledge. We&#10;demonstrate that the proposed models improve neural NLI models to achieve the&#10;state-of-the-art performance on the SNLI and MultiNLI datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.00320" label="2205.00320">
        <attvalues>
          <attvalue for="0" value="Detoxifying Language Models with a Toxic Corpus" />
          <attvalue for="1" value="  Existing studies have investigated the tendency of autoregressive language&#10;models to generate contexts that exhibit undesired biases and toxicity. Various&#10;debiasing approaches have been proposed, which are primarily categorized into&#10;data-based and decoding-based. In our study, we investigate the ensemble of the&#10;two debiasing paradigms, proposing to use toxic corpus as an additional&#10;resource to reduce the toxicity. Our result shows that toxic corpus can indeed&#10;help to reduce the toxicity of the language generation process substantially,&#10;complementing the existing debiasing methods.&#10;" />
          <attvalue for="2" value="&#10;Pretraining language models (LMs) have been a foundation of NLP given recent performance achievements; however, there is a growing concern related to inherent societal and harmful biases in these models. Due to historical biases embedded in training corpora, it is unavoidable for the language models to absorb, reproduce, and even amplify such undesired biases \cite{Schick2020Self-DiagnosisNLP}. &#10;&#10;\cite{Gehman2020REALTOXICITYPROMPTS:Models} showed that pretrained LMs generate toxic text even when conditioned on innocuous prompts. One of their proposed debiased techniques is Domain-Adaptive Pretraining \cite{Gururangan2020DontTasks}, or DAPT, on a non-toxic corpus. \cite{Schick2020Self-DiagnosisNLP} proposed a self-debiasing approach that uses only a handful of templates that contain the definition of undesired attributes. DAPT is a data-based approach where internal weights are updated with an additional phase of pretraining. On the other hand, self-debiasing is a decoding-based approach that does not require additional resources. The difference between the two debiasing paradigms is a trade-off between the computational cost and the quality of debiasing.&#10;&#10;In this study, we propose to ensemble the data- and decoding-based approaches by using a toxic corpus as a detoxifying strategy. Our study attempts to invalidate the belief that only non-toxic corpora can reduce the toxicity of language generation. We use GPT-2 \cite{Radford2018LanguageLearners} as our primary language model and OpenWebText (OWTC; \cite{Gokaslan2019OpenWeb}), a large corpus of English webtext, as our training corpus. We measure the toxicity of each document using PerspectiveAPI and collect non-toxic and toxic corpora that satisfy our toxicity requirements. &#10;&#10;Our results demonstrate that using the toxic corpus indeed reduces the toxicity level of text generated from pretrained language models, which can be further improved by ensemble with the non-toxic corpus.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Toxicity Reduction, Language Model Bias, Computational Linguistics, Artificial Intelligence, Debiasing Techniques, Natural Language Processing, Psychology" />
        </attvalues>
      </node>
      <node id="2305.10005" label="2305.10005">
        <attvalues>
          <attvalue for="0" value="DinoSR: Self-Distillation and Online Clustering for Self-supervised&#10;  Speech Representation Learning" />
          <attvalue for="1" value="  In this paper, we introduce self-distillation and online clustering for&#10;self-supervised speech representation learning (DinoSR) which combines masked&#10;language modeling, self-distillation, and online clustering. We show that these&#10;concepts complement each other and result in a strong representation learning&#10;model for speech. DinoSR first extracts contextualized embeddings from the&#10;input audio with a teacher network, then runs an online clustering system on&#10;the embeddings to yield a machine-discovered phone inventory, and finally uses&#10;the discretized tokens to guide a student network. We show that DinoSR&#10;surpasses previous state-of-the-art performance in several downstream tasks,&#10;and provide a detailed analysis of the model and the learned discrete units.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Self-supervised speech representation learning with deep neural networks first emerged in the form of autoregressive models~\cite{van2017neural,oord2018representation,baevski2019vq,chung2019unsupervised} where the goal is to predict the future based on past observations.&#10;Subsequently, bidirectional models~\cite{ling2020decoar,baevski2020wav,hsu2021hubert,chung2021w2v,chen2022wavlm} relaxed the unidirectional limitation to achieve better results.&#10;A common learning paradigm for bidirectional models is MLM -- masking part of the input and training the model to recover the missing information using unmasked targets.&#10;These targets can be derived from the audio signal using different strategies, such as surface features~\cite{ling2020decoar} or contrastive learning~\cite{baevski2020wav}.&#10;&#10;Following the MLM training scheme, HuBERT~\cite{hsu2021hubert} proposed targeting discrete units generated by vanilla acoustic unit discovery systems.&#10;Such a system can be as simple as K-means clustering over MFCC features, or even random linear projections over spectrograms~\cite{chiu2022self}.&#10;Interestingly, HuBERT found that the acoustic unit discovery system can be iteratively refined by running offline K-Means clustering on the output of a specific layer of the pre-trained model.&#10;However, several important hyper-parameters are required to obtain the best performance, such as the number of updates, the layer whose output is to be clustered, and the number of clusters for each iteration.&#10;While the proposed method is conceptually similar to HuBERT -- MLM with discovered acoustic units, our method can be trained end-to-end with fewer heuristics by leveraging the self-distillation framework and online clustering.&#10;&#10;Our method is also closely related to self-distillation methods for representation learning.&#10;These methods originated from image representation learning~\cite{grill2020bootstrap,caron2021emerging}, training a pair of identical models named student and teacher networks.&#10;The key to this framework is to provide different views of the same input by image augmentation to each model, and also to update them in different policies -- gradient descent for the student model and exponential moving average for the teacher model.&#10;Following the self-distillation framework, \cite{baevski2022data2vec} generalized the method to speech processing by replacing image augmentation with the MLM masking strategy and found it effective.&#10;The key difference between this work and prior work is the online clustering mechanism that derives discrete targets instead of using continuous embeddings from the teacher model as targets.&#10;We also note that our method differs from studies in knowledge distillation from pre-trained speech representation models~\cite{chang2022distilhubert,wang22t_interspeech,lee22p_interspeech,ashihara22_interspeech} which focus on inference efficiency and model compression.&#10;&#10;" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Speech Representation Learning, Signal Processing, Mathematics, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2406.13357" label="2406.13357">
        <attvalues>
          <attvalue for="0" value="Transferable speech-to-text large language model alignment module" />
          <attvalue for="1" value="  By leveraging the power of Large Language Models(LLMs) and speech foundation&#10;models, state of the art speech-text bimodal works can achieve challenging&#10;tasks like spoken translation(ST) and question answering(SQA) altogether with&#10;much simpler architectures. In this paper, we utilize the capability of Whisper&#10;encoder and pre-trained Yi-6B. Empirical results reveal that modal alignment&#10;can be achieved with one layer module and hundred hours of speech-text&#10;multitask corpus. We further swap the Yi-6B with human preferences aligned&#10;version of Yi-6B-Chat during inference, and discover that the alignment&#10;capability is applicable as well. In addition, the alignment subspace revealed&#10;by singular value decomposition(SVD) also implies linear alignment subspace is&#10;sparse, which leaves the possibility to concatenate other features like&#10;voice-print or video to expand modality.&#10;" />
          <attvalue for="2" value="&#10;LLMs have received much attention in recent years. The powerful capabilities of ChatGPT\cite{10.5555/3495724.3495883} have achieved unprecedented breakthroughs in the natural language processing(NLP) field. Gradually, using a single model to solve multiple tasks has become the mainstream approach. Vision large language models have applied this principle to various vision tasks\cite{NEURIPS2022_960a172b, pmlr-v202-li23q, NEURIPS2023_6dcf277e, pmlr-v139-radford21a, zhu2023minigpt4}. In terms of speech modality, some studies have signaled that it is feasible to interact with LLM through speech. AudioGPT\cite{huang2023audiogpt} and HuggingGPT\cite{NEURIPS2023_77c33e6a} have made preliminary attempts. They employ a cascade method to seamlessly integrate automatic speech recognition (ASR), text-to-speech (TTS), and other recognition/generation tasks. The key concept is to apply LLM as an intermediate interface for distributing tasks via calling upon the appropriate models. &#10;Because the LLM is trained with text, speech information is hardly recognized, such as emotions and tones in human voice. By discretizing the speech signal into token sequences and expanding them within the LLM, SpeechGPT\cite{zhang-etal-2023-speechgpt} enables seamless text-speech interaction with a vocoder model for speech synthesis. However, this method requires retraining the LLM to support additional tokens. Moreover, there are some works achieve similar results by concatenating speech and text features as the prompt of LLM. LLaSM\cite{shu2023llasm} uses Whisper\cite{pmlr-v202-radford23a} and Chinese-LLAMA2-7B as speech encoder and LLM with two training stages. In the first stage, they use ASR dataset for the adaptor pre-training. In the second stage, adaptor and language model are updated for cross-modal instruction fine-tuning. Whisper does not appear in Speech-LLaMA\cite{10389705}, they train 4 Transformer layers as audio encoder to complete ST tasks in 13 languages with LLaMA\cite{touvron2023llama}. Whisper and Qwen are used in Qwen-Audio\cite{chu2023qwenaudio}, which is also trained in two stages. The first stage, Qwen LLm is frozen and multi-task audio data is used to train Whisper. In the second stage of training, multi-round dialogue data is utilized to generate an interactive chat model that can accommodate input from diverse audio and text sources.&#10;&#10;Previous works have excelled in aligning speech text modalities, most of which require retraining the speech encoder with a large amount of data to improve representation ability, and then fine-tuning the LLM model with instruction data to achieve better performance. However, this brings a large overhead to computing resources and is difficult to implement when data resources are scarce. Besides, those training strategies are fixed with particulay models and require multiple training with different speech-text foundation models composition. Should a replacement become necessary, realignment processes would have to be updated once more, leading to significant expenses in terms of overall training and utilization.&#10;&#10;This paper raises several questions in response to this situation. Does modal alignment module require retraining speech modality and text modality? Does the alignment of speech-text modality only require a simple alignment module, or even a simple linear layer? Does training modality alignment module require massive amounts of data? Is the trained alignment module scalable and can it be replaced by a LLM with better performance? Furthermore, what kind of knowledge does the feature after alignment module mapping contain? Extend from existing work, we investigate the sufficiency of each components separately, namely the model size of the alignment module, amount of training data, transferability of alignment module across LLMs and the information contained in alignment modules which are rarely explored in current work.&#10;&#10;We propose a linear layer after speech encoder as modal alignment module with open source models and corpus to achieve ASR, ST, SQA and text question answering(QA) multitasks in Mandarin. First, we conjecture that pre-trained speech encoder and LLM have strong text and speech capabilities, so we explore the connection through a single-layer alignment module. We choose Whisper encoder to extract speech features, while keeping parameters frozen, in order to reduce training overhead. Yi-6B with the LLaMA\cite{touvron2023llama} decoder-only structure is selected as LLM. A linear layer is chosen as the modal alignment module to map the speech features output by Whisper into the text feature space. The LLM is frozen during the alignment module training phase. In addition, we explore the transferability of alignment module. After the alignment module is aligned between speech and text, both of Whisper and alignment module are frozen, we replace the Yi-6B model with a supervised fine-tuning(SFT) version that aligns with human preferences. This updated model is validated on ST tasks, resulting in significant performance improvements. Finally, in order to further explore the alignment subspace, we use SVD analysis and therefore reveal information redundancy. Our contributions can be summarized as the following points: &#10;&#10;\begin{itemize}&#10;\item Only adding and training an additional layer of alignment module between LLM and speech encoder to achieve ASR, ST, SQA and QA via open source models and data. The alignment module uses only a small amount of data to stimulate modal alignment capabilities.&#10;&#10;\item The trained alignment module has strong scalability. It can be replaced with the SFT model with better command following and human preference capabilities from the same source without additional training, further improving the preference of specific tasks, such as ST, SQA, etc.&#10;&#10;\item Preliminary analysis of the features after alignment mapping revealed information redundancy. Gradually reducing the dimension of modal alignment mapping revealed that a small reduction in feature dimension has only a slight impact on model performance. This provides insights for future feature concatenation, such as voiceprint features or video features.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Speech Text Alignment, Signal Processing, Mathematics, Multimodal Processing" />
        </attvalues>
      </node>
      <node id="2308.15930" label="2308.15930">
        <attvalues>
          <attvalue for="0" value="LLaSM: Large Language and Speech Model" />
          <attvalue for="1" value="  Multi-modal large language models have garnered significant interest&#10;recently. Though, most of the works focus on vision-language multi-modal models&#10;providing strong capabilities in following vision-and-language instructions.&#10;However, we claim that speech is also an important modality through which&#10;humans interact with the world. Hence, it is crucial for a general-purpose&#10;assistant to be able to follow multi-modal speech-and-language instructions. In&#10;this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an&#10;end-to-end trained large multi-modal speech-language model with cross-modal&#10;conversational abilities, capable of following speech-and-language&#10;instructions. Our early experiments show that LLaSM demonstrates a more&#10;convenient and natural way for humans to interact with artificial intelligence.&#10;Specifically, we also release a large Speech Instruction Following dataset&#10;LLaSM-Audio-Instructions. Code and demo are available at&#10;https://github.com/LinkSoul-AI/LLaSM and&#10;https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions&#10;dataset is available at&#10;https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.03917" label="2307.03917">
        <attvalues>
          <attvalue for="0" value="On decoder-only architecture for speech-to-text and large language model&#10;  integration" />
          <attvalue for="1" value="  Large language models (LLMs) have achieved remarkable success in the field of&#10;natural language processing, enabling better human-computer interaction using&#10;natural language. However, the seamless integration of speech signals into LLMs&#10;has not been explored well. The &quot;decoder-only&quot; architecture has also not been&#10;well studied for speech processing tasks. In this research, we introduce&#10;Speech-LLaMA, a novel approach that effectively incorporates acoustic&#10;information into text-based large language models. Our method leverages&#10;Connectionist Temporal Classification and a simple audio encoder to map the&#10;compressed acoustic features to the continuous semantic space of the LLM. In&#10;addition, we further probe the decoder-only architecture for speech-to-text&#10;tasks by training a smaller scale randomly initialized speech-LLaMA model from&#10;speech-text paired data alone. We conduct experiments on multilingual&#10;speech-to-text translation tasks and demonstrate a significant improvement over&#10;strong baselines, highlighting the potential advantages of decoder-only models&#10;for speech-to-text conversion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Our model aims at integrating speech signals into large language models, as well as relates to Connectionist Temporal Classification (CTC) feature length compression and low-rank adaptation (LoRA). We discuss these topics in the following.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2311.04934" label="2311.04934">
        <attvalues>
          <attvalue for="0" value="Prompt Cache: Modular Attention Reuse for Low-Latency Inference" />
          <attvalue for="1" value="  We present Prompt Cache, an approach for accelerating inference for large&#10;language models (LLM) by reusing attention states across different LLM prompts.&#10;Many input prompts have overlapping text segments, such as system messages,&#10;prompt templates, and documents provided for context. Our key insight is that&#10;by precomputing and storing the attention states of these frequently occurring&#10;text segments on the inference server, we can efficiently reuse them when these&#10;segments appear in user prompts. Prompt Cache employs a schema to explicitly&#10;define such reusable text segments, called prompt modules. The schema ensures&#10;positional accuracy during attention state reuse and provides users with an&#10;interface to access cached states in their prompt. Using a prototype&#10;implementation, we evaluate Prompt Cache across several LLMs. We show that&#10;Prompt Cache significantly reduce latency in time-to-first-token, especially&#10;for longer prompts such as document-based question answering and&#10;recommendations. The improvements range from 8x for GPU-based inference to 60x&#10;for CPU-based inference, all while maintaining output accuracy and without the&#10;need for model parameter modifications.&#10;" />
          <attvalue for="2" value="&#10;&#10;A substantial fraction of large language model (LLM) prompts are reused frequently. For example, prompts usually commence with identical ``system messages'' that provide initial guidelines for its functionality. &#10;Documents can also overlap in multiple prompts. In a wide range of long-context LLM applications, such as legal analysis~\cite{cui2023chatlaw, nay2023large}, healthcare applications~\cite{DBLP:journals/jbi/SteinbergJFCPS21,DBLP:journals/npjdm/Rasmy0XTZ21}, and education~\cite{DBLP:journals/corr/abs-2106-07340}, the prompt includes one or several documents from a pool. &#10;Additionally, prompts are often formatted with reusable templates~\cite{DBLP:journals/corr/abs-2302-11382} as a result of prompt engineering. Such examples are common in LLM for robotics and tool learning~\cite{huang2022language, driess2023palme, DBLP:journals/corr/abs-2304-08354}. This further results in a high degree of overlap between prompts using the same template.&#10;&#10;We introduce a novel technique termed \tech to reduce the computational overhead in generative LLM inference. \tech is motivated by the observation that input prompts to LLM often has reusable structures. The key idea is to precompute attention states of the frequently revisited prompt segments in memory, and reuse them when these segments appear in the prompt to reduce latency.&#10;&#10;Reusing attention states is a popular strategy for accelerating the service of a single prompt~\cite{pope2022efficiently}. The existing approach, often referred to as Key-Value (KV) Cache, reuses the key-value attention states of input tokens during the autoregressive token generation.&#10;This eliminates the need to compute full attention for every token generation (\S~\ref{ssec:kvcache}). By caching the key-value attention computed for the previously generated token, each token generation requires the computation of key-value attention states only once. &#10;&#10;Building on top of \kvcache, \tech extends attention state reuse from a single prompt to multiple prompts by making attention state reuse modular.&#10;In our approach, frequently reused text segments are individually precomputed and stored in memory. When such ``cached'' segments appear in the input prompt, the system uses the precomputed key-value attention states from memory instead of recomputing them. As a result, attention computations are only required for uncached text segments. \ref{fig:overview} illustrates the difference between full autoregressive generation, \kvcache, and \tech. We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size~\cite{keles2022computational,tay2023efficient} while the space and compute complexity of \tech scales linearly with the size.&#10;&#10;Two challenges arise when reusing attention states across prompts. First, attention states are position-dependent due to the positional encoding in Transformers. Thus, the attention states of a text segment can only be reused if the segment appears at the same position. Second, &#10;the system must be able to efficiently recognize a text segment whose attention states may have been cached in order to reuse.&#10;&#10;To tackle these two problems, \tech combines two ideas.&#10;The first is to make the structure of a prompt explicit with a Prompt Markup Language (\lang).&#10;\lang makes reusable text segments explicit as modules, \ie, \modu. &#10;It not only solves the second problem above but opens the door for solving the first, since each \modu can be assigned with unique position IDs.&#10;Our second idea is our empirical finding that LLMs can operate on attention states with discontinuous position IDs. This means that we can extract different segment of attention states and concatenate them to formulate subset of meanings. We leverage this to enable users to select \modus based on their needs, or even update some \modus during the runtime.&#10;&#10;We explain how \tech works in \S\ref{sec:prompt-caching}. In summary, an LLM user writes their prompts in \lang, with the intention that they may reuse the attention states based on \modus. &#10;Importantly, they must derive a prompt from a \sch, which is also written in \lang. \ref{fig:schema-prompt} shows a example prompt based on an example \sch.&#10;When \tech receives a prompt, it first processes its \sch and computes the attention states for its \modus. It reuses these states for the \modus in the prompt and other prompts derived from the same \sch.&#10;In \S\ref{sec:implementation}, we report a prototype implementation of \tech on top of the HuggingFace transformers library \cite{wolf2020huggingfaces}. While \tech can work with any Transformer architecture compatible with \kvcache, we experiment with three popular Transformer architectures powering the following open-sourced LLMs: Llama2 \cite{touvron2023llama}, Falcon \cite{falcon}, and MPT~\cite{MosaicML2023Introducing}. We consider two types of memory for storing prompt modules: CPU and GPU memory. While CPU memory can scale to terabyte levels, it brings the overhead of host-to-device memory copying. In contrast, GPU memory does not require coping but has limited capacity.&#10;&#10;Using the prototype, we conduct an extensive benchmark evaluation to examine the performance and quantify the accuracy of \tech across various long-context datasets (\S\ref{sec:eval}). We employ the LongBench suite~\cite{bai2023longbench}, which includes recommendation and question-answering (QA) tasks based on multiple documents. In our evaluation, \tech reduces time-to-first-token (TTFT) latency from $1.5\times$ to $10\times$ for GPU inference with prompt modules on GPU memory and from $20\times$ to $70\times$ for CPU inference, all without any significant accuracy loss. Additionally, we analyze the memory overhead of the precomputed attention states for each model and discuss directions for optimizing the memory footprint of \tech. &#10;We subsequently showcase several generative tasks, including personalization, code generation, and parameterized prompts, to demonstrate the expressiveness of the prompt schema and performance improvement with negligible quality degradation.&#10;&#10;In our present study, we mainly focus on techniques for modular attention reuse. However, we foresee \tech being utilized as a foundational component for future LLM serving systems. Such systems could incorporate enhanced prompt module management and GPU cache replacement strategies, optimizing the advantages of both host DRAM and GPU HBM. Our source code and data used for evaluation are available at \href{&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Inference Acceleration, Mathematics, Attention State Reuse, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2302.11382" label="2302.11382">
        <attvalues>
          <attvalue for="0" value="A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT" />
          <attvalue for="1" value="  Prompt engineering is an increasingly important skill set needed to converse&#10;effectively with large language models (LLMs), such as ChatGPT. Prompts are&#10;instructions given to an LLM to enforce rules, automate processes, and ensure&#10;specific qualities (and quantities) of generated output. Prompts are also a&#10;form of programming that can customize the outputs and interactions with an&#10;LLM. This paper describes a catalog of prompt engineering techniques presented&#10;in pattern form that have been applied to solve common problems when conversing&#10;with LLMs. Prompt patterns are a knowledge transfer method analogous to&#10;software patterns since they provide reusable solutions to common problems&#10;faced in a particular context, i.e., output generation and interaction when&#10;working with LLMs. This paper provides the following contributions to research&#10;on prompt engineering that apply LLMs to automate software development tasks.&#10;First, it provides a framework for documenting patterns for structuring prompts&#10;to solve a range of problems so that they can be adapted to different domains.&#10;Second, it presents a catalog of patterns that have been applied successfully&#10;to improve the outputs of LLM conversations. Third, it explains how prompts can&#10;be built from multiple patterns and illustrates prompt patterns that benefit&#10;from combination with other prompt patterns.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.06732" label="2009.06732">
        <attvalues>
          <attvalue for="0" value="Efficient Transformers: A Survey" />
          <attvalue for="1" value="  Transformer model architectures have garnered immense interest lately due to&#10;their effectiveness across a range of domains like language, vision and&#10;reinforcement learning. In the field of natural language processing for&#10;example, Transformers have become an indispensable staple in the modern deep&#10;learning stack. Recently, a dizzying number of &quot;X-former&quot; models have been&#10;proposed - Reformer, Linformer, Performer, Longformer, to name a few - which&#10;improve upon the original Transformer architecture, many of which make&#10;improvements around computational and memory efficiency. With the aim of&#10;helping the avid researcher navigate this flurry, this paper characterizes a&#10;large and thoughtful selection of recent efficiency-flavored &quot;X-former&quot; models,&#10;providing an organized and comprehensive overview of existing work and models&#10;across multiple domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.15326" label="2310.15326">
        <attvalues>
          <attvalue for="0" value="Specialist or Generalist? Instruction Tuning for Specific NLP Tasks" />
          <attvalue for="1" value="  The potential of large language models (LLMs) to simultaneously perform a&#10;wide range of natural language processing (NLP) tasks has been the subject of&#10;extensive research. Although instruction tuning has proven to be a&#10;data-efficient method for transforming LLMs into such generalist models, their&#10;performance still lags behind specialist models trained exclusively for&#10;specific tasks. In this paper, we investigate whether incorporating&#10;broad-coverage generalist instruction tuning can contribute to building a&#10;specialist model. We hypothesize that its efficacy depends on task specificity&#10;and skill requirements. Our experiments assess four target tasks with distinct&#10;coverage levels, revealing that integrating generalist instruction tuning&#10;consistently enhances model performance when the task coverage is broad. The&#10;effect is particularly pronounced when the amount of task-specific training&#10;data is limited. Further investigation into three target tasks focusing on&#10;different capabilities demonstrates that generalist instruction tuning improves&#10;understanding and reasoning abilities. However, for tasks requiring factual&#10;knowledge, generalist data containing hallucinatory information may negatively&#10;affect the model's performance. Overall, our work provides a systematic guide&#10;for developing specialist models with general instruction tuning. Our code and&#10;other related resources can be found at&#10;https://github.com/DavidFanzz/Generalist_or_Specialist.&#10;" />
          <attvalue for="2" value="&#10;&#10;The latest generation of large language models (LLMs), such as ChatGPT \cite{2022OpenAIchatgpt} and GPT4 \cite{2023GPT4Openai}, are often referred to as generalist models for their exceptional generalizability to perform various natural language processing (NLP) tasks. Recent studies \cite{2023TaoriAlpaca,zhou2023lima,2023GudibandeFalse} suggest that (1) the foundation of their superior performance (i.e., knowledge and capabilities) is predominantly acquired during large-scale unsupervised pre-training; and (2) {instruction tuning} \cite{2021SanhT0,wei2021finetuned,mishra2021natural,ouyang2022training} is an incredibly data-efficient method for unleashing the power of LLMs to complete realistic NLP tasks. However, under rigorous evaluation, the performance of those instruction-following generalist models often falls short compared to traditional task-specific specialist models \cite{2023JiaoChatGPTTranslator,2023QinChatGPTGeneral,fang2023chatgpt,liu2023comprehensive}. Recently, there has also been a growing trend towards developing specialist models using instruction tuning \cite{2023JiaoParroT,2023WangInstructUIE,2023ZhangWritingAssistance,cheng2023adapting,2023HuaTuoWang}.&#10;&#10;In this paper, we study how to better harness the power of LLM for specific NLP tasks using instruction tuning. Our research is motivated by the existence of various broad-coverage general-purpose instruction-following datasets \cite{2023TaoriAlpaca,2023PengGPT4Instruct,dolly,2023XuWizardLM,zhou2023lima,su2023pandagpt} and their surprising efficiency for turning LLMs into capable instruction-following generalists. For instance, \newcite{zhou2023lima} show that merely one thousand supervised input-output pairs are necessary to build a competent generalist. In contrast to general-purpose instruction tuning, our preliminary experiments show that a sufficiently large set of task-specific data is still required for transforming an LLM into a superior specialist. This leads us to a pivotal research question: How to better unleash the power of LLMs for specific NLP tasks by marrying the best of two worlds? More specifically, can general-purpose instruction-following datasets aid in the transformation of an LLM into a specialist? If so, when and how?&#10;&#10;We hypothesize the answers to the previous questions depend on (1) how specific the target task is; and (2) what skills the target task requires. To test this hypothesis, we first assess four target tasks with distinct levels of coverage. Our findings reveal that integrating general instruction tuning—that is, training with generalist data enhances the model's performance on specific NLP tasks with broad task coverage, particularly when the amount of task-specific training data is limited. To gain a deeper understanding of the improvements elicited by training with generalist data, we subsequently examine three target tasks that focus on distinct skill sets. Our results suggest that general instruction tuning improves the model's understanding and reasoning capabilities. However, when it comes to tasks that demand factual knowledge from the LLM, instructional data generated through self-instruct \cite{2022WangSelfInstruct} harms the model's performance due to the intrinsic hallucinations brought by such data creation approach.&#10;&#10;In sum, to the best of our knowledge, our work is the first effort to present a systematic guide for building and improving specialist models with general instruction tuning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Instruction Tuning" />
        </attvalues>
      </node>
      <node id="2303.13547" label="2303.13547">
        <attvalues>
          <attvalue for="0" value="A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability" />
          <attvalue for="1" value="  This paper presents the first comprehensive analysis of ChatGPT's Text-to-SQL&#10;ability. Given the recent emergence of large-scale conversational language&#10;model ChatGPT and its impressive capabilities in both conversational abilities&#10;and code generation, we sought to evaluate its Text-to-SQL performance. We&#10;conducted experiments on 12 benchmark datasets with different languages,&#10;settings, or scenarios, and the results demonstrate that ChatGPT has strong&#10;text-to-SQL abilities. Although there is still a gap from the current&#10;state-of-the-art (SOTA) model performance, considering that the experiment was&#10;conducted in a zero-shot scenario, ChatGPT's performance is still impressive.&#10;Notably, in the ADVETA (RPL) scenario, the zero-shot ChatGPT even outperforms&#10;the SOTA model that requires fine-tuning on the Spider dataset by 4.1\%,&#10;demonstrating its potential for use in practical applications. To support&#10;further research in related fields, we have made the data generated by ChatGPT&#10;publicly available at https://github.com/THU-BPM/chatgpt-sql.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.08085" label="2304.08085">
        <attvalues>
          <attvalue for="0" value="InstructUIE: Multi-task Instruction Tuning for Unified Information&#10;  Extraction" />
          <attvalue for="1" value="  Large language models have unlocked strong multi-task capabilities from&#10;reading instructive prompts. However, recent studies have shown that existing&#10;large models still have difficulty with information extraction tasks. For&#10;example, gpt-3.5-turbo achieved an F1 score of 18.22 on the Ontonotes dataset,&#10;which is significantly lower than the state-of-the-art performance. In this&#10;paper, we propose InstructUIE, a unified information extraction framework based&#10;on instruction tuning, which can uniformly model various information extraction&#10;tasks and capture the inter-task dependency. To validate the proposed method,&#10;we introduce IE INSTRUCTIONS, a benchmark of 32 diverse information extraction&#10;datasets in a unified text-to-text format with expert-written instructions.&#10;Experimental results demonstrate that our method achieves comparable&#10;performance to Bert in supervised settings and significantly outperforms the&#10;state-of-the-art and gpt3.5 in zero-shot settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13225" label="2305.13225">
        <attvalues>
          <attvalue for="0" value="Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A&#10;  Preliminary Study on Writing Assistance" />
          <attvalue for="1" value="  Proprietary Large Language Models (LLMs), such as ChatGPT, have garnered&#10;significant attention due to their exceptional capabilities in handling a&#10;diverse range of tasks. Recent studies demonstrate that open-sourced smaller&#10;foundational models, such as 7B-size LLaMA, can also display remarkable&#10;proficiency in tackling diverse tasks when fine-tuned using instruction-driven&#10;data. In this work, we investigate a practical problem setting where the&#10;primary focus is on one or a few particular tasks rather than general-purpose&#10;instruction following, and explore whether LLMs can be beneficial and further&#10;improved for such targeted scenarios. We choose the writing-assistant scenario&#10;as the testbed, which includes seven writing tasks. We collect training data&#10;for these tasks, reframe them in an instruction-following format, and&#10;subsequently refine the LLM, specifically LLaMA, via instruction tuning.&#10;Experimental results show that fine-tuning LLaMA on writing instruction data&#10;significantly improves its ability on writing tasks. We also conduct more&#10;experiments and analyses to offer insights for future work on effectively&#10;fine-tuning LLaMA for specific scenarios. Finally, we initiate a discussion&#10;regarding the necessity of employing LLMs for only one targeted task, taking&#10;into account the efforts required for tuning and the resources consumed during&#10;deployment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16355" label="2305.16355">
        <attvalues>
          <attvalue for="0" value="PandaGPT: One Model To Instruction-Follow Them All" />
          <attvalue for="1" value="  We present PandaGPT, an approach to emPower large lANguage moDels with visual&#10;and Auditory instruction-following capabilities. Our pilot experiments show&#10;that PandaGPT can perform complex tasks such as detailed image description&#10;generation, writing stories inspired by videos, and answering questions about&#10;audios. More interestingly, PandaGPT can take multimodal inputs simultaneously&#10;and compose their semantics naturally. For example, PandaGPT can connect how&#10;objects look in an image/video and how they sound in an audio. To do so,&#10;PandaGPT combines the multimodal encoders from ImageBind and the large language&#10;models from Vicuna. Notably, only aligned image-text pairs are required for the&#10;training of PandaGPT. Thanks to the strong capability of ImageBind in embedding&#10;data from different modalities into the same space, PandaGPT displays emergent,&#10;i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g.,&#10;video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an&#10;initial step toward building AGI that can perceive and understand inputs in&#10;different modalities holistically, as we humans do. Our project page is at&#10;https://panda-gpt.github.io/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.12810" label="2312.12810">
        <attvalues>
          <attvalue for="0" value="Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and&#10;  Detection" />
          <attvalue for="1" value="  Dysfluent speech modeling requires time-accurate and silence-aware&#10;transcription at both the word-level and phonetic-level. However, current&#10;research in dysfluency modeling primarily focuses on either transcription or&#10;detection, and the performance of each aspect remains limited. In this work, we&#10;present an unconstrained dysfluency modeling (UDM) approach that addresses both&#10;transcription and detection in an automatic and hierarchical manner. UDM&#10;eliminates the need for extensive manual annotation by providing a&#10;comprehensive solution. Furthermore, we introduce a simulated dysfluent dataset&#10;called VCTK++ to enhance the capabilities of UDM in phonetic transcription. Our&#10;experimental results demonstrate the effectiveness and robustness of our&#10;proposed methods in both transcription and detection tasks.&#10;" />
          <attvalue for="2" value="&#10;In the field of speech analysis, a clear and universally agreed-upon definition of dysfluent speech is yet to be established. Dysfluencies are commonly associated with speech disorders such as stuttering, aphasia~\cite{brady2016aphasia}, and dyslexia~\cite{snowling2013dyslexia}, characterized by disruptions in the normal flow of speech~\cite{palfy2012pattern-dysfluent}. However, dysfluency also appears in normal conversational speech or spontaneous speech~\cite{pitt2005buckeye}, where individuals may experience hesitations or interruptions while speaking. In this context, we refer to dysfluent speech as any form of speech characterized by abnormal patterns such as repetition, prolongation, and irregular pauses, as discussed in~\cite{kouzelis2023weakly}. Within the domain of dysfluent speech modeling, there is also a lack of a unified paradigm. Research efforts in this area can be roughly categorized as transcription and detection. &#10;&#10;Current state-of-the-art word transcription models~\cite{radford2022whisper, lian2023av-data2vec, zhang2023google-usm, pratap2023scaling-speech, aghajanyan2023scaling-speech} and phonetic transcription models~\cite{wavlm-ctc} often struggle to accurately transcribe dysfluent speech. As a result, human annotations are still required and transcribers commonly resort to a two-step process. They first obtain automatic transcriptions and then manually annotate the missing portions, a laborious and time-consuming task that is comparable to manual annotation from scratch. &#10;WhisperX\cite{bain2023whisperx} recently extends Whisper~\cite{radford2022whisper} by generating timestamps for individual words. However, it still delivers limited performance for dysfluent speech. Moreover, time-accurate phonetic transcriptions might be a better representation to capture various dysfluency types. Another requirement is that phonetic transcriptions should be sensitive to silences as it might indicate a block or poor breath-speech coordination. \cite{kouzelis2023weakly} recently proposed a time-accurate and silence-aware neural forced aligner, where a weighted finite-state transducer (WFST) is introduced for modeling dysfluency patterns such as repetition. However, this approach assumes that there is minimal deviation between the reference and &quot;real&quot; transcribed text. In real-life dysfluent speech, such as the example shown in Figure \ref{UDM}, this assumption may not hold true. &#10;&#10;Research on dysfluency detection has traditionally been conducted independently of dysfluency transcription and has recently been dominated by end-to-end methods. These approaches typically focus on either utterance-level detection \cite{kourkounakis2021fluentnet, alharbi2017segment-detection2, alharbi2020segment-detection3, segment-detection4}, or frame-level detection \cite{harvill2022frame-level-stutter, shonibare2022frame-detection2}. However, these studies primarily address data-driven classification problems and do not explicitly incorporate dysfluency transcription into their detection methods. A unified framework that integrates dysfluency transcription and detection is essential to develop an efficient and robust dysfluency modeling system.&#10;&#10;In this study, we propose an unconstrained dysfluency modeling (UDM) approach that integrates dysfluent speech transcription and pattern detection in an automatic manner with no human effort. Since real dysfluent speech is unconstrained and word transcription is unknown (as shown in the &quot;Human Transcription&quot; in Figure \ref{UDM}, which is largely different from reference text), we develop hierarchical transcription methods. Firstly, we introduce an unconstrained forced aligner with a dynamic alignment search module to generate text-independent alignments. Secondly, we propose a Text Refresher that leverages the alignment input to refine the state-of-the-art Whisper~\cite{radford2022whisper} transcription. For the detection component, we employ 2D-alignment to automatically detect various phonetic and word-level dysfluency patterns, including repetition, missing, replacement, insertion, deletion, and irregular pauses. To further enhance performance, we curate a dysfluent dataset called VCTK$^{++}$ to boost the capacity of our unconstrained forced aligner (UFA). Experimental results demonstrate the effectiveness of our proposed framework in both dysfluent speech transcription and dysfluency pattern detection.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Linguistics, Speech Transcription, Dysfluency Modeling, Automatic Annotation, Natural Language Processing, Psychology" />
        </attvalues>
      </node>
      <node id="2306.00996" label="2306.00996">
        <attvalues>
          <attvalue for="0" value="Weakly-supervised forced alignment of disfluent speech using&#10;  phoneme-level modeling" />
          <attvalue for="1" value="  The study of speech disorders can benefit greatly from time-aligned data.&#10;However, audio-text mismatches in disfluent speech cause rapid performance&#10;degradation for modern speech aligners, hindering the use of automatic&#10;approaches. In this work, we propose a simple and effective modification of&#10;alignment graph construction of CTC-based models using Weighted Finite State&#10;Transducers. The proposed weakly-supervised approach alleviates the need for&#10;verbatim transcription of speech disfluencies for forced alignment. During the&#10;graph construction, we allow the modeling of common speech disfluencies, i.e.&#10;repetitions and omissions. Further, we show that by assessing the degree of&#10;audio-text mismatch through the use of Oracle Error Rate, our method can be&#10;effectively used in the wild. Our evaluation on a corrupted version of the&#10;TIMIT test set and the UCLASS dataset shows significant improvements,&#10;particularly for recall, achieving a 23-25% relative improvement over our&#10;baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.00747" label="2303.00747">
        <attvalues>
          <attvalue for="0" value="WhisperX: Time-Accurate Speech Transcription of Long-Form Audio" />
          <attvalue for="1" value="  Large-scale, weakly-supervised speech recognition models, such as Whisper,&#10;have demonstrated impressive results on speech recognition across domains and&#10;languages. However, their application to long audio transcription via buffered&#10;or sliding window approaches is prone to drifting, hallucination &amp; repetition;&#10;and prohibits batched transcription due to their sequential nature. Further,&#10;timestamps corresponding each utterance are prone to inaccuracies and&#10;word-level timestamps are not available out-of-the-box. To overcome these&#10;challenges, we present WhisperX, a time-accurate speech recognition system with&#10;word-level timestamps utilising voice activity detection and forced phoneme&#10;alignment. In doing so, we demonstrate state-of-the-art performance on&#10;long-form transcription and word segmentation benchmarks. Additionally, we show&#10;that pre-segmenting audio with our proposed VAD Cut &amp; Merge strategy improves&#10;transcription quality and enables a twelve-fold transcription speedup via&#10;batched inference.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.00550" label="2303.00550">
        <attvalues>
          <attvalue for="0" value="Towards domain generalisation in ASR with elitist sampling and ensemble&#10;  knowledge distillation" />
          <attvalue for="1" value="  Knowledge distillation has widely been used for model compression and domain&#10;adaptation for speech applications. In the presence of multiple teachers,&#10;knowledge can easily be transferred to the student by averaging the models&#10;output. However, previous research shows that the student do not adapt well&#10;with such combination. This paper propose to use an elitist sampling strategy&#10;at the output of ensemble teacher models to select the best-decoded utterance&#10;generated by completely out-of-domain teacher models for generalizing unseen&#10;domain. The teacher models are trained on AMI, LibriSpeech and WSJ while the&#10;student is adapted for the Switchboard data. The results show that with the&#10;selection strategy based on the individual models posteriors the student model&#10;achieves a better WER compared to all the teachers and baselines with a minimum&#10;absolute improvement of about 8.4 percent. Furthermore, an insights on the&#10;model adaptation with out-of-domain data has also been studied via correlation&#10;analysis.&#10;" />
          <attvalue for="2" value="&#10;&#10;Knowledge distillation (KD) \cite{KD} is a popular technique to distill the knowledge from either single or multiple cumbersome teacher models to one student model. &#10;The knowledge from the teacher model is transferred at the output \cite{KD} or intermediate feature level \cite{romero2014fitnets}. A common technique is to transfer the knowledge using teachers' posterior distribution \cite{gou2021knowledge}. Usually, the student model is trained with the teacher's posteriors~(soft labels) along with the original training labels~(hard labels). Therefore, the total loss is the weighted sum of supervised and KD losses. KD is used for many tasks such as domain adaptation \cite{asami2017domain,meng2019domain,zhu2020domain}, domain generalisation \cite{wang2021embracing,kim2021domain, fang2021mosaicking}, and model compression \cite{chebotar2016distilling, kim2019knowledge, takashima2018investigation}. &#10;&#10;In automatic speech recognition, knowledge distillation is performed over either frame-level \cite{wong2016sequence} or sequence-level \cite{huang2018knowledge} span. Following the sequential nature of the ASR task, sequence-level transfer is shown to be a better approach in \cite{wong2016sequence}. Sequence level KD was first proposed in \cite{kim2016sequence}, where the teacher models provide sequence-level probability distribution over the whole sample space for better knowledge transfer. In the case of multiple teacher models, it is crucial to optimise the sampling strategy to obtain the best possible output. &#10;For example, \cite{gao2021distilling} proposed three types of selection strategies at the output of teachers, i.e. weighted, Top-1 and Top-K. The error metric is used as a selection criterion in all of these strategies. Therefore, the labelled data is assumed to be available. However, this approach is not helpful in those scenarios where the goal is to adapt unseen out-of-domain~(OOD) data. Hence, an unsupervised sampling strategy is necessary to reduce the uncertainty of multiple teacher outcomes and select the best output.&#10;&#10;Furthermore, the underlying teacher model is important for the generalised representation of the acoustic samples. Various models such as BERT\cite{futami2020distilling}, BLSTMs \cite{8639629} with different context windows have been used in ASR knowledge distillation tasks. Self-supervised training paved the path to learn the general data representation through unsupervised pre-training. Such a model is trained with masked spans for generalised contextual latent representation of speech \cite{baevski2020wav2vec}. These self-supervised models have been observed to be quickly adaptable to new domains or cross-domain tasks \cite{hsu2021robust}.&#10;&#10;The generalisation problem in KD is two-fold. The first problem is optimising the teacher model's learned representation, and the second is choosing the appropriate distribution for a student from the teacher model. These problems become challenging when dealing with OOD data between teacher and student. In this paper, the first problem is tackled with a pre-trained wav2vec teacher models where each model is fine-tuned with an in-domain corpora. The second problem is tackled with posterior based elitist sampling strategy which selects the best utterance decoded by the teachers. &#10;In summary, an ensemble of teacher models are trained on completely OOD data compared to the data which student needs to adapt. An inference is run on the unlabelled OOD data to generate the soft labels from the teachers. Finally, an elitist sampling strategy based on the output posteriors is used to select the best decoded utterance from the teachers to train a student model. More specifically, dataset from read speech: WSJ (LDC catalog LDC93S6A, LDC94S13A), LibriSpeech (LS) \cite{Panayotov2015} and meeting: AMI \cite{Carletta2006} are used to train three state-of-the-art teacher models. These models are used to decode the Switchboard (SB) \cite{Godfrey1992} corpus, and a student model is trained. &#10;The results show that with such selection on the ensemble of teacher outputs, the student model performs better compared to the baselines and all the individual teacher models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Compression, Domain Adaptation, Signal Processing, Speech Recognition, Statistics" />
        </attvalues>
      </node>
      <node id="2001.01798" label="2001.01798">
        <attvalues>
          <attvalue for="0" value="Domain Adaptation via Teacher-Student Learning for End-to-End Speech&#10;  Recognition" />
          <attvalue for="1" value="  Teacher-student (T/S) has shown to be effective for domain adaptation of deep&#10;neural network acoustic models in hybrid speech recognition systems. In this&#10;work, we extend the T/S learning to large-scale unsupervised domain adaptation&#10;of an attention-based end-to-end (E2E) model through two levels of knowledge&#10;transfer: teacher's token posteriors as soft labels and one-best predictions as&#10;decoder guidance. To further improve T/S learning with the help of ground-truth&#10;labels, we propose adaptive T/S (AT/S) learning. Instead of conditionally&#10;choosing from either the teacher's soft token posteriors or the one-hot&#10;ground-truth label, in AT/S, the student always learns from both the teacher&#10;and the ground truth with a pair of adaptive weights assigned to the soft and&#10;one-hot labels quantifying the confidence on each of the knowledge sources. The&#10;confidence scores are dynamically estimated at each decoder step as a function&#10;of the soft and one-hot labels. With 3400 hours parallel close-talk and&#10;far-field Microsoft Cortana data for domain adaptation, T/S and AT/S achieve&#10;6.3% and 10.3% relative word error rate improvement over a strong E2E model&#10;trained with the same amount of far-field data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.02782" label="2011.02782">
        <attvalues>
          <attvalue for="0" value="Domain Adaptation Using Class Similarity for Robust Speech Recognition" />
          <attvalue for="1" value="  When only limited target domain data is available, domain adaptation could be&#10;used to promote performance of deep neural network (DNN) acoustic model by&#10;leveraging well-trained source model and target domain data. However, suffering&#10;from domain mismatch and data sparsity, domain adaptation is very challenging.&#10;This paper proposes a novel adaptation method for DNN acoustic model using&#10;class similarity. Since the output distribution of DNN model contains the&#10;knowledge of similarity among classes, which is applicable to both source and&#10;target domain, it could be transferred from source to target model for the&#10;performance improvement. In our approach, we first compute the frame level&#10;posterior probabilities of source samples using source model. Then, for each&#10;class, probabilities of this class are used to compute a mean vector, which we&#10;refer to as mean soft labels. During adaptation, these mean soft labels are&#10;used in a regularization term to train the target model. Experiments showed&#10;that our approach outperforms fine-tuning using one-hot labels on both accent&#10;and noise adaptation task, especially when source and target domain are highly&#10;mismatched.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.02629" label="2107.02629">
        <attvalues>
          <attvalue for="0" value="Embracing the Dark Knowledge: Domain Generalization Using Regularized&#10;  Knowledge Distillation" />
          <attvalue for="1" value="  Though convolutional neural networks are widely used in different tasks, lack&#10;of generalization capability in the absence of sufficient and representative&#10;data is one of the challenges that hinder their practical application. In this&#10;paper, we propose a simple, effective, and plug-and-play training strategy&#10;named Knowledge Distillation for Domain Generalization (KDDG) which is built&#10;upon a knowledge distillation framework with the gradient filter as a novel&#10;regularization term. We find that both the ``richer dark knowledge&quot; from the&#10;teacher network, as well as the gradient filter we proposed, can reduce the&#10;difficulty of learning the mapping which further improves the generalization&#10;ability of the model. We also conduct experiments extensively to show that our&#10;framework can significantly improve the generalization capability of deep&#10;neural networks in different tasks including image classification,&#10;segmentation, reinforcement learning by comparing our method with existing&#10;state-of-the-art domain generalization techniques. Last but not the least, we&#10;propose to adopt two metrics to analyze our proposed method in order to better&#10;understand how our proposed method benefits the generalization capability of&#10;deep neural networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.06531" label="2111.06531">
        <attvalues>
          <attvalue for="0" value="Domain Generalization on Efficient Acoustic Scene Classification using&#10;  Residual Normalization" />
          <attvalue for="1" value="  It is a practical research topic how to deal with multi-device audio inputs&#10;by a single acoustic scene classification system with efficient design. In this&#10;work, we propose Residual Normalization, a novel feature normalization method&#10;that uses frequency-wise normalization % instance normalization with a shortcut&#10;path to discard unnecessary device-specific information without losing useful&#10;information for classification. Moreover, we introduce an efficient&#10;architecture, BC-ResNet-ASC, a modified version of the baseline architecture&#10;with a limited receptive field. BC-ResNet-ASC outperforms the baseline&#10;architecture even though it contains the small number of parameters. Through&#10;three model compression schemes: pruning, quantization, and knowledge&#10;distillation, we can reduce model complexity further while mitigating the&#10;performance degradation. The proposed system achieves an average test accuracy&#10;of 76.3% in TAU Urban Acoustic Scenes 2020 Mobile, development dataset with&#10;315k parameters, and average test accuracy of 75.3% after compression to 61.0KB&#10;of non-zero parameters. The proposed method won the 1st place in DCASE 2021&#10;challenge, TASK1A.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.07947" label="1606.07947">
        <attvalues>
          <attvalue for="0" value="Sequence-Level Knowledge Distillation" />
          <attvalue for="1" value="  Neural machine translation (NMT) offers a novel alternative formulation of&#10;translation that is potentially simpler than statistical approaches. However to&#10;reach competitive performance, NMT models need to be exceedingly large. In this&#10;paper we consider applying knowledge distillation approaches (Bucila et al.,&#10;2006; Hinton et al., 2015) that have proven successful for reducing the size of&#10;neural models in other domains to the problem of NMT. We demonstrate that&#10;standard knowledge distillation applied to word-level prediction can be&#10;effective for NMT, and also introduce two novel sequence-level versions of&#10;knowledge distillation that further improve performance, and somewhat&#10;surprisingly, seem to eliminate the need for beam search (even when applied on&#10;the original teacher model). Our best student model runs 10 times faster than&#10;its state-of-the-art teacher with little loss in performance. It is also&#10;significantly better than a baseline model trained without knowledge&#10;distillation: by 4.2/1.7 BLEU with greedy decoding/beam search. Applying weight&#10;pruning on top of knowledge distillation results in a student model that has 13&#10;times fewer parameters than the original teacher model, with a decrease of 0.4&#10;BLEU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09310" label="2005.09310">
        <attvalues>
          <attvalue for="0" value="Distilling Knowledge from Ensembles of Acoustic Models for Joint&#10;  CTC-Attention End-to-End Speech Recognition" />
          <attvalue for="1" value="  Knowledge distillation has been widely used to compress existing deep&#10;learning models while preserving the performance on a wide range of&#10;applications. In the specific context of Automatic Speech Recognition (ASR),&#10;distillation from ensembles of acoustic models has recently shown promising&#10;results in increasing recognition performance. In this paper, we propose an&#10;extension of multi-teacher distillation methods to joint CTC-attention&#10;end-to-end ASR systems. We also introduce three novel distillation strategies.&#10;The core intuition behind them is to integrate the error rate metric to the&#10;teacher selection rather than solely focusing on the observed losses. In this&#10;way, we directly distill and optimize the student toward the relevant metric&#10;for speech recognition. We evaluate these strategies under a selection of&#10;training procedures on different datasets (TIMIT, Librispeech, Common Voice)&#10;and various languages (English, French, Italian). In particular,&#10;state-of-the-art error rates are reported on the Common Voice French, Italian&#10;and TIMIT datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.03822" label="2008.03822">
        <attvalues>
          <attvalue for="0" value="Distilling the Knowledge of BERT for Sequence-to-Sequence ASR" />
          <attvalue for="1" value="  Attention-based sequence-to-sequence (seq2seq) models have achieved promising&#10;results in automatic speech recognition (ASR). However, as these models decode&#10;in a left-to-right way, they do not have access to context on the right. We&#10;leverage both left and right context by applying BERT as an external language&#10;model to seq2seq ASR through knowledge distillation. In our proposed method,&#10;BERT generates soft labels to guide the training of seq2seq ASR. Furthermore,&#10;we leverage context beyond the current utterance as input to BERT. Experimental&#10;evaluations show that our method significantly improves the ASR performance&#10;from the seq2seq baseline on the Corpus of Spontaneous Japanese (CSJ).&#10;Knowledge distillation from BERT outperforms that from a transformer LM that&#10;only looks at left context. We also show the effectiveness of leveraging&#10;context beyond the current utterance. Our method outperforms other LM&#10;application approaches such as n-best rescoring and shallow fusion, while it&#10;does not require extra inference cost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.01027" label="2104.01027">
        <attvalues>
          <attvalue for="0" value="Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised&#10;  Pre-Training" />
          <attvalue for="1" value="  Self-supervised learning of speech representations has been a very active&#10;research area but most work is focused on a single domain such as read audio&#10;books for which there exist large quantities of labeled and unlabeled data. In&#10;this paper, we explore more general setups where the domain of the unlabeled&#10;data for pre-training data differs from the domain of the labeled data for&#10;fine-tuning, which in turn may differ from the test data domain. Our&#10;experiments show that using target domain data during pre-training leads to&#10;large performance improvements across a variety of setups. On a large-scale&#10;competitive setup, we show that pre-training on unlabeled in-domain data&#10;reduces the gap between models trained on in-domain and out-of-domain labeled&#10;data by 66%-73%. This has obvious practical implications since it is much&#10;easier to obtain unlabeled target domain data than labeled data. Moreover, we&#10;find that pre-training on multiple domains improves generalization performance&#10;on domains not seen during training. Code and models will be made available at&#10;https://github.com/pytorch/fairseq.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;This paper is related to a large body of work on robust ASR and domain adaptation.&#10;There are two popular lines of approaches. &#10;The first one is feature-based, which focuses on creating robust features~\cite{stern2012features,hsu2018extracting}. Both signal processing-based~\cite{kingsbury1998robust} and learned~\cite{hsu2017disentangle} features have been explored. &#10;The other line is model-based, which exposes a model to diverse data while minimally pre-processing speech input in order to exploit the model capacity. &#10;This includes data augmentation~\cite{kim2017generation,tang2018study}, self-training on target domain~\cite{khurana2020unsupervised}, domain adversarial training~\cite{sun2017unsupervised}, and joint training~\cite{likhomanenko2020rethinking}.&#10;The self-supervised approach explored in this paper can be categorized as model-based; however, unlike the aforementioned model-based methods, it does not require any labeled data during pre-training by using a self-supervised objective.&#10;&#10;The most related work to this paper is~\cite{kawakami2020learning}, which investigated domain-shift for self-supervised learning, but did not dissect the domains of data used during pre-training. &#10;We extend this work by also examining the effect of pre-training data domain.&#10;Furthermore, the pre-trained feature extractor in~\cite{kawakami2020learning} is fixed during supervised fine-tuning, making it more similar to feature-based approaches.&#10;Other related work includes pre-training on multiple languages and investigating how representations transfer between languages~\cite{rivire2020unsupervised,conneau2020unsupervised}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2403.05132" label="2403.05132">
        <attvalues>
          <attvalue for="0" value="ChatUIE: Exploring Chat-based Unified Information Extraction using Large&#10;  Language Models" />
          <attvalue for="1" value="  Recent advancements in large language models have shown impressive&#10;performance in general chat. However, their domain-specific capabilities,&#10;particularly in information extraction, have certain limitations. Extracting&#10;structured information from natural language that deviates from known schemas&#10;or instructions has proven challenging for previous prompt-based methods. This&#10;motivated us to explore domain-specific modeling in chat-based language models&#10;as a solution for extracting structured information from natural language. In&#10;this paper, we present ChatUIE, an innovative unified information extraction&#10;framework built upon ChatGLM. Simultaneously, reinforcement learning is&#10;employed to improve and align various tasks that involve confusing and limited&#10;samples. Furthermore, we integrate generation constraints to address the issue&#10;of generating elements that are not present in the input. Our experimental&#10;results demonstrate that ChatUIE can significantly improve the performance of&#10;information extraction with a slight decrease in chatting ability.&#10;" />
          <attvalue for="2" value="&#10;Information extraction (IE) is a structured prediction task that aims to identify and structure user-specified information from unstructured texts ~\cite{DBLP:conf/anlp/AndersenHWHSN92,DBLP:journals/nle/Grishman19,lu-etal-2022-unified,cao-etal-2022-oneee,jiang-etal-2021-named,DBLP:journals/access/XuS22}. IE tasks are highly diversified due to their varying targets (entities, relations, events, etc.), heterogeneous structures (spans, triplets, records, etc.), and domain-specific schemas~\cite{DBLP:journals/corr/abs-2301-03282,DBLP:journals/corr/abs-2210-02414,DBLP:conf/acl/DuQLDQY022}.&#10;The primary studies~\cite{jiang-etal-2021-named,li2022unified,DBLP:conf/cikm/XuSLF18,ye-etal-2022-packed,cao-etal-2022-oneee,sheng-etal-2021-casee,Zhang2022OptimizingBF,tang-etal-2022-unirel,xu-etal-2022-extracting} of information extraction are task-specialized, which results in dedicated architectures, isolated models, and specialized knowledge sources for different IE tasks. Several improved methods~\cite{lu-etal-2022-unified,DBLP:journals/corr/abs-2301-03282,wei2023zeroshot,wang2023instructuie} have been proposed for the unified modeling of information extraction tasks, including prompt-based extractive and generative models. &#10;&#10;However, these methods are highly tailored to pre-defined schemas or fixed instructions, which makes it extremely challenging to facilitate natural language extraction. As shown in Figure~\ref{Fig.motivation}, UIE relies on a pre-defined schema and prompt template. Deviating from this consistency can significantly degrade model performance, especially for zero-shot tasks where the schema was not seen during training. In contrast, InstructUIE uses a set of instructions for information extraction. However, since these instructions are not tailored to the specific task (what to extract?), the model is restricted to the known dataset. When faced with a new schema, using task-irrelevant instructions makes it difficult to produce satisfactory results.&#10;&#10;Generally, previous instruction-based methods focused more on memorizing instructions rather than comprehending them. While ChatGLM outperforms InstructUIE in task-relevant natural language scenarios, there is room for improvement in closed domain. &#10;However, enhancing the information extraction capabilities while preserving the general chat capabilities of ChatGLM presents challenges. Firstly, conflicts between domain-specific knowledge and the knowledge embedded in LLMs may result in knowledge forgetting. Secondly, the scarcity of annotated domain knowledge and uneven sample distribution make it arduous for LLM to effectively adapt and accommodate.&#10;These issues cannot be adequately addressed through supervised fine-tuning alone. In order to address this issue, we have introduced reinforcement learning to align various tasks. In contrast to other tasks, the result for information extraction is derived from the input. To ensure that the generated elements remain within the input, we utilize generation constraint decoding.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Information Extraction Techniques, Chat-Based Domain Modeling, Language Model Advancements, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2203.12277" label="2203.12277">
        <attvalues>
          <attvalue for="0" value="Unified Structure Generation for Universal Information Extraction" />
          <attvalue for="1" value="  Information extraction suffers from its varying targets, heterogeneous&#10;structures, and demand-specific schemas. In this paper, we propose a unified&#10;text-to-structure generation framework, namely UIE, which can universally model&#10;different IE tasks, adaptively generate targeted structures, and&#10;collaboratively learn general IE abilities from different knowledge sources.&#10;Specifically, UIE uniformly encodes different extraction structures via a&#10;structured extraction language, adaptively generates target extractions via a&#10;schema-based prompt mechanism - structural schema instructor, and captures the&#10;common IE abilities via a large-scale pre-trained text-to-structure model.&#10;Experiments show that UIE achieved the state-of-the-art performance on 4 IE&#10;tasks, 13 datasets, and on all supervised, low-resource, and few-shot settings&#10;for a wide range of entity, relation, event and sentiment extraction tasks and&#10;their unification. These results verified the effectiveness, universality, and&#10;transferability of UIE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.02693" label="2209.02693">
        <attvalues>
          <attvalue for="0" value="OneEE: A One-Stage Framework for Fast Overlapping and Nested Event&#10;  Extraction" />
          <attvalue for="1" value="  Event extraction (EE) is an essential task of information extraction, which&#10;aims to extract structured event information from unstructured text. Most prior&#10;work focuses on extracting flat events while neglecting overlapped or nested&#10;ones. A few models for overlapped and nested EE includes several successive&#10;stages to extract event triggers and arguments,which suffer from error&#10;propagation. Therefore, we design a simple yet effective tagging scheme and&#10;model to formulate EE as word-word relation recognition, called OneEE. The&#10;relations between trigger or argument words are simultaneously recognized in&#10;one stage with parallel grid tagging, thus yielding a very fast event&#10;extraction speed. The model is equipped with an adaptive event fusion module to&#10;generate event-aware representations and a distance-aware predictor to&#10;integrate relative distance information for word-word relation recognition,&#10;which are empirically demonstrated to be effective mechanisms. Experiments on 3&#10;overlapped and nested EE benchmarks, namely FewFC, Genia11, and Genia13, show&#10;that OneEE achieves the state-of-the-art (SOTA) results. Moreover, the&#10;inference speed of OneEE is faster than those of baselines in the same&#10;condition, and can be further substantially improved since it supports parallel&#10;inference.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.08977" label="2106.08977">
        <attvalues>
          <attvalue for="0" value="Named Entity Recognition with Small Strongly Labeled and Large Weakly&#10;  Labeled Data" />
          <attvalue for="1" value="  Weak supervision has shown promising results in many natural language&#10;processing tasks, such as Named Entity Recognition (NER). Existing work mainly&#10;focuses on learning deep NER models only with weak supervision, i.e., without&#10;any human annotation, and shows that by merely using weakly labeled data, one&#10;can achieve good performance, though still underperforms fully supervised NER&#10;with manually/strongly labeled data. In this paper, we consider a more&#10;practical scenario, where we have both a small amount of strongly labeled data&#10;and a large amount of weakly labeled data. Unfortunately, we observe that&#10;weakly labeled data does not necessarily improve, or even deteriorate the model&#10;performance (due to the extensive noise in the weak labels) when we train deep&#10;NER models over a simple or weighted combination of the strongly labeled and&#10;weakly labeled data. To address this issue, we propose a new multi-stage&#10;computational framework -- NEEDLE with three essential ingredients: (1) weak&#10;label completion, (2) noise-aware loss function, and (3) final fine-tuning over&#10;the strongly labeled data. Through experiments on E-commerce query NER and&#10;Biomedical NER, we demonstrate that NEEDLE can effectively suppress the noise&#10;of the weak labels and outperforms existing methods. In particular, we achieve&#10;new SOTA F1-scores on 3 Biomedical NER datasets: BC5CDR-chem 93.74,&#10;BC5CDR-disease 90.69, NCBI-disease 92.28.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.03282" label="2301.03282">
        <attvalues>
          <attvalue for="0" value="Universal Information Extraction as Unified Semantic Matching" />
          <attvalue for="1" value="  The challenge of information extraction (IE) lies in the diversity of label&#10;schemas and the heterogeneity of structures. Traditional methods require&#10;task-specific model design and rely heavily on expensive supervision, making&#10;them difficult to generalize to new schemas. In this paper, we decouple IE into&#10;two basic abilities, structuring and conceptualizing, which are shared by&#10;different tasks and schemas. Based on this paradigm, we propose to universally&#10;model various IE tasks with Unified Semantic Matching (USM) framework, which&#10;introduces three unified token linking operations to model the abilities of&#10;structuring and conceptualizing. In this way, USM can jointly encode schema and&#10;input text, uniformly extract substructures in parallel, and controllably&#10;decode target structures on demand. Empirical evaluation on 4 IE tasks shows&#10;that the proposed method achieves state-of-the-art performance under the&#10;supervised experiments and shows strong generalization ability in zero/few-shot&#10;transfer settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.10070" label="2112.10070">
        <attvalues>
          <attvalue for="0" value="Unified Named Entity Recognition as Word-Word Relation Classification" />
          <attvalue for="1" value="  So far, named entity recognition (NER) has been involved with three major&#10;types, including flat, overlapped (aka. nested), and discontinuous NER, which&#10;have mostly been studied individually. Recently, a growing interest has been&#10;built for unified NER, tackling the above three jobs concurrently with one&#10;single model. Current best-performing methods mainly include span-based and&#10;sequence-to-sequence models, where unfortunately the former merely focus on&#10;boundary identification and the latter may suffer from exposure bias. In this&#10;work, we present a novel alternative by modeling the unified NER as word-word&#10;relation classification, namely W^2NER. The architecture resolves the kernel&#10;bottleneck of unified NER by effectively modeling the neighboring relations&#10;between entity words with Next-Neighboring-Word (NNW) and Tail-Head-Word-*&#10;(THW-*) relations. Based on the W^2NER scheme we develop a neural framework, in&#10;which the unified NER is modeled as a 2D grid of word pairs. We then propose&#10;multi-granularity 2D convolutions for better refining the grid representations.&#10;Finally, a co-predictor is used to sufficiently reason the word-word relations.&#10;We perform extensive experiments on 14 widely-used benchmark datasets for flat,&#10;overlapped, and discontinuous NER (8 English and 6 Chinese datasets), where our&#10;model beats all the current top-performing baselines, pushing the&#10;state-of-the-art performances of unified NER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.06067" label="2109.06067">
        <attvalues>
          <attvalue for="0" value="Packed Levitated Marker for Entity and Relation Extraction" />
          <attvalue for="1" value="  Recent entity and relation extraction works focus on investigating how to&#10;obtain a better span representation from the pre-trained encoder. However, a&#10;major limitation of existing works is that they ignore the interrelation&#10;between spans (pairs). In this work, we propose a novel span representation&#10;approach, named Packed Levitated Markers (PL-Marker), to consider the&#10;interrelation between the spans (pairs) by strategically packing the markers in&#10;the encoder. In particular, we propose a neighborhood-oriented packing&#10;strategy, which considers the neighbor spans integrally to better model the&#10;entity boundary information. Furthermore, for those more complicated span pair&#10;classification tasks, we design a subject-oriented packing strategy, which&#10;packs each subject and all its objects to model the interrelation between the&#10;same-subject span pairs. The experimental results show that, with the enhanced&#10;marker feature, our model advances baselines on six NER benchmarks, and obtains&#10;a 4.1%-4.3% strict relation F1 improvement with higher speed over previous&#10;state-of-the-art models on ACE04 and ACE05.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.01583" label="2107.01583">
        <attvalues>
          <attvalue for="0" value="CasEE: A Joint Learning Framework with Cascade Decoding for Overlapping&#10;  Event Extraction" />
          <attvalue for="1" value="  Event extraction (EE) is a crucial information extraction task that aims to&#10;extract event information in texts. Most existing methods assume that events&#10;appear in sentences without overlaps, which are not applicable to the&#10;complicated overlapping event extraction. This work systematically studies the&#10;realistic event overlapping problem, where a word may serve as triggers with&#10;several types or arguments with different roles. To tackle the above problem,&#10;we propose a novel joint learning framework with cascade decoding for&#10;overlapping event extraction, termed as CasEE. Particularly, CasEE sequentially&#10;performs type detection, trigger extraction and argument extraction, where the&#10;overlapped targets are extracted separately conditioned on the specific former&#10;prediction. All the subtasks are jointly learned in a framework to capture&#10;dependencies among the subtasks. The evaluation on a public event extraction&#10;benchmark FewFC demonstrates that CasEE achieves significant improvements on&#10;overlapping event extraction over previous competitive methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.09039" label="2211.09039">
        <attvalues>
          <attvalue for="0" value="UniRel: Unified Representation and Interaction for Joint Relational&#10;  Triple Extraction" />
          <attvalue for="1" value="  Relational triple extraction is challenging for its difficulty in capturing&#10;rich correlations between entities and relations. Existing works suffer from 1)&#10;heterogeneous representations of entities and relations, and 2) heterogeneous&#10;modeling of entity-entity interactions and entity-relation interactions.&#10;Therefore, the rich correlations are not fully exploited by existing works. In&#10;this paper, we propose UniRel to address these challenges. Specifically, we&#10;unify the representations of entities and relations by jointly encoding them&#10;within a concatenated natural language sequence, and unify the modeling of&#10;interactions with a proposed Interaction Map, which is built upon the&#10;off-the-shelf self-attention mechanism within any Transformer block. With&#10;comprehensive experiments on two popular relational triple extraction datasets,&#10;we demonstrate that UniRel is more effective and computationally efficient. The&#10;source code is available at https://github.com/wtangdev/UniRel.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.01866" label="2408.01866">
        <attvalues>
          <attvalue for="0" value="Efficient Solutions For An Intriguing Failure of LLMs: Long Context&#10;  Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated remarkable capabilities in&#10;comprehending and analyzing lengthy sequential inputs, owing to their extensive&#10;context windows that allow processing millions of tokens in a single forward&#10;pass. However, this paper uncovers a surprising limitation: LLMs fall short&#10;when handling long input sequences. We investigate this issue using three&#10;datasets and two tasks (sentiment analysis and news categorization) across&#10;various LLMs, including Claude 3, Gemini Pro, GPT 3.5 Turbo, Llama 3 Instruct,&#10;and Mistral Instruct models. To address this limitation, we propose and&#10;evaluate ad-hoc solutions that substantially enhance LLMs' performance on long&#10;input sequences by up to 50%, while reducing API cost and latency by up to 93%&#10;and 50%, respectively.&#10;" />
          <attvalue for="2" value="&#10;LLMs have demonstrated remarkable capabilities in natural language understanding and generation tasks. Leveraging extensive pretraining on massive text corpora, the new generation of LLMs can perform a wide range of language tasks with minimal task-specific fine-tuning. Additionally, these LLMs are equipped with behemothic context windows that enable them to analyze inputs spanning up to tens or hundreds of pages in one forward pass. In this paper, we study the performance of Claude 3 Haiku \cite{claude3haiku20240307}, GPT3.5-Turbo \cite{gpt3.5turbo}, Gemini-1.0-pro \cite{Gemini2023}, Llama 3 8b Instruct \cite{llama3}, and Mistral 7b Instruct \cite{mistral}. Some of these LLMs are equipped with context windows that can support up to 200,000 tokens in one forward pass. &#10;&#10;Related Work. Prompting strategies have emerged as a promising avenue for improving LLM performance by providing concise and informative input \cite{Liu+2023,Brown+2020, Jiang+23, ge+24}. These strategies involve extracting key information from the input text and presenting it to the LLM in a structured manner. However, despite being equipped with context windows capable in theory of supporting large amounts of text, the performance of LLMs often suffers on lengthy input sequences as the prompt length grows \cite{Li+2023,Li+2024}. &#10;&#10;On the other hand, many general summarization techniques are available for condensing lengthy texts into more manageable snippets. Extractive Summarization methods such as TextRank \cite{mihalceaT2004} are widely used to identify and extract the most significant sentences from a document for different purposes \cite{CacholaLCW2020, FengFQ22, BalcerzakJW2014, Wang+2020}. Although not designed for prompt compression, these techniques might therefore be useful in this context, and have relatively low computational overheads; in this paper, we therefore investigate the use of real-time summarization pipelines and text truncation techniques to boost LLM performance by optimizing the input while reducing their load.&#10;&#10;Motivation. &#10;There is a body of research dedicated to studying the limitations of LLMs on long sequences and proposing mitigations at both architecture-level \cite{BeltagyPC2020,BertschANG2024} as well as prompt-level \cite{Wei+2022}. These studies often involve defining and exploring overly complex problems such as those about extreme-label classification \cite{Li+2024} or ``Needle In a Haystack&quot; \cite{MachlabB2024}. However, a systematic study of LLM capabilities and limitations on long-form analysis tasks such as news categorization or sentiment analysis&#10;of long reviews which require a common general understanding of the input context is still lacking. Furthermore, the emphasis on approaches involving prompt-tuning has diverted attention away from optimizing and streamlining the information fed to LLMs. This study serves to fill these gaps by showcasing the failure of LLMs on canonical NLP tasks when dealing with long sequences and to ignite a spark of interest in the research community to explore the untapped potential of optimizing and condensing the information fed to LLMs.&#10;&#10;Contribution. Our main contributions are:&#10;\begin{enumerate}[itemsep=0pt, topsep=0pt, partopsep=0pt, leftmargin=*]&#10;\item We systematically study the performance of state-of-the-art (SotA) LLMs on sentiment analysis and news categorization, revealing their limitations in processing long-form text effectively.&#10;\item We propose and evaluate ad-hoc solutions using extractive and diverse summarization as well as selective truncation to condense input text, which substantially improves LLM performance by up to 50\%, reduces API costs by as much as 93\% and significantly reduces latency.&#10;\item We present comprehensive empirical and ablation studies examining the relationship between input length, summarization strategies, and model performance, providing insights into optimal summarization approaches for LLMs.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Limitations, Model Performance Optimization, Mathematics, Sequence Analysis Challenges" />
        </attvalues>
      </node>
      <node id="2404.02060" label="2404.02060">
        <attvalues>
          <attvalue for="0" value="Long-context LLMs Struggle with Long In-context Learning" />
          <attvalue for="1" value="  Large Language Models (LLMs) have made significant strides in handling long&#10;sequences. Some models like Gemini could even to be capable of dealing with&#10;millions of tokens. However, their performance evaluation has largely been&#10;confined to metrics like perplexity and synthetic tasks, which may not fully&#10;capture their true abilities in more challenging, real-world scenarios. We&#10;introduce a benchmark (LongICLBench) for long in-context learning in&#10;extreme-label classification using six datasets with 28 to 174 classes and&#10;input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend&#10;the entire input to recognize the massive label spaces to make correct&#10;predictions. We evaluate on 15 long-context LLMs and find that they perform&#10;well on less challenging classification tasks with smaller label space and&#10;shorter demonstrations. However, they struggle with more challenging task like&#10;Discovery with 174 labels, suggesting a gap in their ability to process long,&#10;context-rich sequences. Further analysis reveals a bias towards labels&#10;presented later in the sequence and a need for improved reasoning over multiple&#10;pieces of information. Our study reveals that long context understanding and&#10;reasoning is still a challenging task for the existing LLMs. We believe&#10;LongICLBench could serve as a more realistic evaluation for the future&#10;long-context LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.15011" label="2004.15011">
        <attvalues>
          <attvalue for="0" value="TLDR: Extreme Summarization of Scientific Documents" />
          <attvalue for="1" value="  We introduce TLDR generation, a new form of extreme summarization, for&#10;scientific papers. TLDR generation involves high source compression and&#10;requires expert background knowledge and understanding of complex&#10;domain-specific language. To facilitate study on this task, we introduce&#10;SciTLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers. SciTLDR&#10;contains both author-written and expert-derived TLDRs, where the latter are&#10;collected using a novel annotation protocol that produces high-quality&#10;summaries while minimizing annotation burden. We propose CATTS, a simple yet&#10;effective learning strategy for generating TLDRs that exploits titles as an&#10;auxiliary training signal. CATTS improves upon strong baselines under both&#10;automated metrics and human evaluations. Data and code are publicly available&#10;at https://github.com/allenai/scitldr.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.01625" label="2305.01625">
        <attvalues>
          <attvalue for="0" value="Unlimiformer: Long-Range Transformers with Unlimited Length Input" />
          <attvalue for="1" value="  Since the proposal of transformers, these models have been limited to bounded&#10;input lengths, because of their need to attend to every token in the input. In&#10;this work, we propose Unlimiformer: a general approach that wraps any existing&#10;pretrained encoder-decoder transformer, and offloads the cross-attention&#10;computation to a single k-nearest-neighbor (kNN) index, while the returned kNN&#10;distances are the attention dot-product scores. This kNN index can be kept on&#10;either the GPU or CPU memory and queried in sub-linear time; this way, we can&#10;index practically unlimited input sequences, while every attention head in&#10;every decoder layer retrieves its top-k keys, instead of attending to every&#10;key. We evaluate Unlimiformer on several long-document and book-summarization&#10;benchmarks, showing that it can process even 500k token-long inputs from the&#10;BookSum dataset, without any input truncation at test time. We demonstrate that&#10;Unlimiformer improves pretrained models such as BART and Longformer by&#10;extending them to unlimited inputs without additional learned weights and&#10;without modifying their code. We make our code and models publicly available at&#10;https://github.com/abertsch72/unlimiformer .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.08865" label="2404.08865">
        <attvalues>
          <attvalue for="0" value="LLM In-Context Recall is Prompt Dependent" />
          <attvalue for="1" value="  The proliferation of Large Language Models (LLMs) highlights the critical&#10;importance of conducting thorough evaluations to discern their comparative&#10;advantages, limitations, and optimal use cases. Particularly important is&#10;assessing their capacity to accurately retrieve information included in a given&#10;prompt. A model's ability to do this significantly influences how effectively&#10;it can utilize contextual details, thus impacting its practical efficacy and&#10;dependability in real-world applications.&#10;  Our research analyzes the in-context recall performance of various LLMs using&#10;the needle-in-a-haystack method. In this approach, a factoid (the &quot;needle&quot;) is&#10;embedded within a block of filler text (the &quot;haystack&quot;), which the model is&#10;asked to retrieve. We assess the recall performance of each model across&#10;various haystack lengths and with varying needle placements to identify&#10;performance patterns. This study demonstrates that an LLM's recall capability&#10;is not only contingent upon the prompt's content but also may be compromised by&#10;biases in its training data. Conversely, adjustments to model architecture,&#10;training strategy, or fine-tuning can improve performance. Our analysis&#10;provides insight into LLM behavior, offering direction for the development of&#10;more effective applications of LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14993" label="2305.14993">
        <attvalues>
          <attvalue for="0" value="Controlling Pre-trained Language Models for Grade-Specific Text&#10;  Simplification" />
          <attvalue for="1" value="  Text simplification (TS) systems rewrite text to make it more readable while&#10;preserving its content. However, what makes a text easy to read depends on the&#10;intended readers. Recent work has shown that pre-trained language models can&#10;simplify text using a wealth of techniques to control output simplicity,&#10;ranging from specifying only the desired reading grade level, to directly&#10;specifying low-level edit operations. Yet it remains unclear how to set these&#10;control parameters in practice. Existing approaches set them at the corpus&#10;level, disregarding the complexity of individual inputs and considering only&#10;one level of output complexity. In this work, we conduct an empirical study to&#10;understand how different control mechanisms impact the adequacy and simplicity&#10;of text simplification systems. Based on these insights, we introduce a simple&#10;method that predicts the edit operations required for simplifying a text for a&#10;specific grade level on an instance-per-instance basis. This approach improves&#10;the quality of the simplified outputs over corpus-level search-based&#10;heuristics.&#10;" />
          <attvalue for="2" value=" &#10;&#10;In the NLP task of text simplification, systems are asked to rewrite, restructure or modify an original text such that it improves the readability of the original text for a target audience while preserving its meaning. However, text can be simplified in many different ways and what makes a text simple to read depends on the reader. Replacing complex or specialized terms with simpler synonyms might help non-native speakers \cite{SarahPetersenMariOstendorf2007,DavidAllen2009}, restructuring text into short sentences with simple words might better match the literacy skills of children \cite{watanabe2009facilita}. &#10;&#10;Acknowledging that text simplification is highly audience-centric \cite{stajner-2021-automatic}, recent work has focused on developing techniques to control the degree of simplicity of the output at different levels. At a high level, one can simply specify the desired reading grade level of the output \cite{ScartonSpecia2018, kew-ebling-2022-target}. At a low level, one can control complexity by describing the nature of simplification operations to be performed \cite{mallinson2019controllable, martin2020controllable}. For example (Figure~\ref{fig:newsela_example}), one could obtain two distinct simplifications of the same inputs by indicating that they are intended for a grade 6 vs. grade 3 audience, or by specifying values for low-level control tokens such as the word length ratio (W) between the source and the target &#10; \mc{this is an awkward way to describe the length ratio between input and output}\sa{Updated} &#10; and the maximum dependency tree depth (DTD) ratio between the source and the target.&#10; \mc{The figure does not use the W and DTD notation.}\sa{Updated} &#10; For an original complex text at grade 8, when simplifying to grade 5, the low-level control values indicate a conservative rewrite, whereas, for grade 3, the properties encoded by the control tokens reflect a relatively more lexical and structural change.&#10;&#10;While specifying a reading grade level might be more intuitive for lay users, it provides weaker control over the nature of simplification to be performed. On the other hand, controlling the outputs' simplicity by setting several low-level properties, such as the number of words or dependency tree depth, provides finer-grained control but can be cumbersome to set by readers, teachers, or other users. As a result, it remains unclear how to operationalize the control of text simplification in practice. Prior work sets low-level control values (length, degree of paraphrasing, lexical complexity, and syntactic complexity) at the corpus level by searching for control token values on a development set. This is done via maximizing a utility computed using an automatic evaluation metric, \sari, a metric designed to measure lexical simplicity \cite{XuNapolesPavlickChenCallison-Burch2016}. While this approach is appealing in its simplicity, it remains unclear whether this approach actually helps control complexity for individual inputs, as the control token values are always set at the corpus level.&#10;&#10;This work presents a systematic empirical study of the impact of control tokens on the degree and quality of simplifications achieved at the instance level as measured by automatic text simplification metrics. Our empirical study shows that most corpus-level control tokens have an opposite impact on adequacy and simplicity when measured by BLEU and SARI respectively. As a result, selecting their values based on \sari alone yields simpler text at the cost of misrepresenting the original source content. To address this problem, we introduce simple models to predict what control tokens are needed for a given input text and a desired grade level, based on surface-form features extracted from the source text and the desired complexity level. We show that the predicted low-level control tokens improve text simplification on a controllable \ts task compared to corpus-level search-based optimization.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Models, Readability Control, Text Simplification, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1910.04387" label="1910.04387">
        <attvalues>
          <attvalue for="0" value="Controllable Sentence Simplification: Employing Syntactic and Lexical&#10;  Constraints" />
          <attvalue for="1" value="  Sentence simplification aims to make sentences easier to read and understand.&#10;Recent approaches have shown promising results with sequence-to-sequence models&#10;which have been developed assuming homogeneous target audiences. In this paper&#10;we argue that different users have different simplification needs (e.g.&#10;dyslexics vs. non-native speakers), and propose CROSS, ContROllable Sentence&#10;Simplification model, which allows to control both the level of simplicity and&#10;the type of the simplification. We achieve this by enriching a&#10;Transformer-based architecture with syntactic and lexical constraints (which&#10;can be set or learned from data). Empirical results on two benchmark datasets&#10;show that constraints are key to successful simplification, offering flexible&#10;generation output.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.11696" label="2308.11696">
        <attvalues>
          <attvalue for="0" value="Efficient Benchmarking of Language Models" />
          <attvalue for="1" value="  The increasing versatility of language models (LMs) has given rise to a new&#10;class of benchmarks that comprehensively assess a broad range of capabilities.&#10;Such benchmarks are associated with massive computational costs, extending to&#10;thousands of GPU hours per model. However, the efficiency aspect of these&#10;evaluation efforts had raised little discussion in the literature. In this&#10;work, we present the problem of Efficient Benchmarking, namely, intelligently&#10;reducing the computation costs of LM evaluation without compromising&#10;reliability. Using the HELM benchmark as a test case, we investigate how&#10;different benchmark design choices affect the computation-reliability&#10;trade-off. We propose to evaluate the reliability of such decisions, by using a&#10;new measure -- Decision Impact on Reliability, DIoR for short. We find, for&#10;example, that a benchmark leader may change by merely removing a low-ranked&#10;model from the benchmark, and observe that a correct benchmark ranking can be&#10;obtained by considering only a fraction of the evaluation examples. Based on&#10;our findings, we outline a set of concrete recommendations for efficient&#10;benchmark design and utilization practices. To take a step further, we use our&#10;findings to propose an evaluation algorithm, that, when applied to the HELM&#10;benchmark, leads to dramatic cost savings with minimal loss of benchmark&#10;reliability, often reducing computation by x100 or more.&#10;" />
          <attvalue for="2" value=" &#10;&#10;Given the ongoing advances in the versatility and performance of Language Models (LMs), they are now expected to perform a diverse range of tasks. &#10;This expectation raises a profound challenge -- how do we evaluate and rank the quality of different LMs over a variety of capabilities?&#10;&#10;This is a complex evaluation endeavor \cite{chang2023survey}, as it transcends the boundaries of a specific task and seeks to measure the overall capabilities of an LM over a wide manifold of natural language tasks. &#10;To this end, LM benchmarks are constantly being proposed, where each new benchmark further expands the coverage and diversity of evaluated tasks and settings \cite{wang-etal-2018-glue,bigbench,lmeval-harness,talmor-etal-2020-olmpics,yuan2023evaluating,zhang2023dialogstudio}. &#10;Running such expansive benchmarks can entail spending \$$10$K+ or $4$K+ GPU hours for evaluating a single model \cite{liang2022holistic}, and may even surpass those of pretraining \cite{biderman2023emergent} when evaluating checkpoints.&#10;At the same time, even when compute resources are abundant, benchmarks are bound to make certain concessions aiming to approximate true model ability. &#10;These concessions -- in the form of benchmark design choices -- are to be made such that their impact on benchmark reliability (\S\ref{sec:reliability}) is both minimized and transparent. This, to minimize cases where sub-optimal design choices lead to reliability issues such as anointing a different best model or making rank differences between models statistically meaningless. &#10;&#10;In this work, we call attention to the topic of Efficient Benchmarking, namely intelligently reducing the computation costs of evaluation without compromising reliability. &#10;While the trade-off between computation and performance is usually discussed in the context of pre-training (e.g., scaling laws; \cite{hoffmann2022trainingChinchilla,ivgi2022scaling}) and fine-tuning (e.g., parameter efficient; \cite{lialin2023scaling}), here we call for putting this trade-off on the center stage of evaluation design. &#10;&#10;In practice, the compute side of the trade-off already plays a role in most large-scale evaluation decisions, both in benchmark design \cite{liang2022holistic} and in its use for evaluation \cite[e.g., choosing the number of seeds;][]{csordas2021devil, choshen2022start}.&#10;However, despite their practical importance, these choices and their impact on benchmark reliability have hardly been discussed in the literature, making researchers apply their own efficiency heuristics instead of using systematic guidelines or literature when building their benchmarks. &#10;&#10;In order to advance efficient evaluation practices, the community is in need of a systematic set of guidelines and recommendations. &#10;These, in turn, must be based on a rigorous study of the different decisions made in benchmark design and how they affect efficiency. &#10;&#10;To begin addressing these challenges, we propose Decision Impact on Reliability -- DIoR -- a way to measure the Impact of a Decision over a setup size (e.g., $1$K examples, $10$ datasets) on the Reliability. In addition, we perform a comprehensive analysis study on efficient benchmarking. &#10;With HELM \cite{liang2022holistic} as a test case, we test various decisions made and how they affect the trade-off between computation and reliability:&#10;decisions about scenarios which are aggregated phenomena (\S\ref{sec:scenario}), subscenarios (\S\ref{sec:subscenarios}), few-shot prompts (\S\ref{sec:seeds}) and the metric (\S\ref{sec:metric}). &#10;Among other findings, &#10;we observe a substantial computation redundancy (see Fig.~\ref{fig:rank}, \ref{fig:seeds}); &#10;that a change in one rank is currently unreliable (\S\ref{sec:scenario}); &#10;that splitting the data into groups (scenarios) hurts reliability; &#10;and that the mean win rate score (\S\ref{sec:metric}) is unreliable and gameable. &#10;&#10;Given our analysis findings, we collect a set of general guidelines for future benchmark creation and use (see Tips above). Moreover, we show how our findings can benefit current benchmarks by proposing Flash-HELM (\S\ref{sec:flashHELM}), a general evaluation algorithm that enables obtaining a model's ranking with a fraction of the computation and minimal loss of benchmark reliability.&#10;&#10;In summary, the contributions of this work are as follows:&#10;\begin{enumerate}&#10; \item We highlight the importance of the balance between computation and reliability in benchmark design and utilization, and propose DIoR as a quantitative measure of the reliability of a specific efficiency strategy.&#10; \item We conduct the first systematic study of the effects of benchmark design on reliability.&#10; \item Given the analysis findings, we provide an Efficient benchmark building checklist; These guidelines outline how best to reduce the computational cost of benchmarking while maintaining an adequate level of evaluation reliability.&#10; \item We propose an algorithm for dynamic ranking of a new LM, assigning higher importance to rank top-performing models. In HELM, we show that this algorithm (later incorporated into the framework) dramatically reduces the computation by up to $\times200$ with minor deviations from the original ranking (see Fig.~\ref{fig:tournament}).&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Benchmark Design Optimization, Operations Research, Mathematics, Statistics, Efficient Computation Methods" />
        </attvalues>
      </node>
      <node id="2308.01240" label="2308.01240">
        <attvalues>
          <attvalue for="0" value="Evaluating Instruction-Tuned Large Language Models on Code Comprehension&#10;  and Generation" />
          <attvalue for="1" value="  In this work, we evaluate 10 open-source instructed LLMs on four&#10;representative code comprehension and generation tasks. We have the following&#10;main findings. First, for the zero-shot setting, instructed LLMs are very&#10;competitive on code comprehension and generation tasks and sometimes even&#10;better than small SOTA models specifically fine-tuned on each downstream task.&#10;We also find that larger instructed LLMs are not always better on code-related&#10;tasks. Second, for the few-shot setting, we find that adding demonstration&#10;examples substantially helps instructed LLMs perform better on most code&#10;comprehension and generation tasks; however, the examples would sometimes&#10;induce unstable or even worse performance. Furthermore, we find widely-used&#10;BM25-based shot selection strategy significantly outperforms the basic random&#10;selection or fixed selection only on generation problems. Third, for the&#10;fine-tuning setting, we find that fine-tuning could further improve the model&#10;performance on downstream code comprehension and generation tasks compared to&#10;the zero-shot/one-shot performance. In addition, after being fine-tuned on the&#10;same downstream task dataset, instructed LLMs outperform both the small SOTA&#10;models and similar-scaled LLMs without instruction tuning. Based on our&#10;findings, we further present practical implications on model and usage&#10;recommendation, performance and cost trade-offs, and future direction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.11158" label="2304.11158">
        <attvalues>
          <attvalue for="0" value="Emergent and Predictable Memorization in Large Language Models" />
          <attvalue for="1" value="  Memorization, or the tendency of large language models (LLMs) to output&#10;entire sequences from their training data verbatim, is a key concern for safely&#10;deploying language models. In particular, it is vital to minimize a model's&#10;memorization of sensitive datapoints such as those containing personal&#10;identifiable information (PII). The prevalence of such undesirable memorization&#10;can pose issues for model trainers, and may even require discarding an&#10;otherwise functional model. We therefore seek to predict which sequences will&#10;be memorized before a large model's full train-time by extrapolating the&#10;memorization behavior of lower-compute trial runs. We measure memorization of&#10;the Pythia model suite and plot scaling laws for forecasting memorization,&#10;allowing us to provide equi-compute recommendations to maximize the reliability&#10;(recall) of such predictions. We additionally provide further novel discoveries&#10;on the distribution of memorization scores across models and data. We release&#10;all code and data necessary to reproduce the results in this paper at&#10;https://github.com/EleutherAI/pythia&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.06387" label="2202.06387">
        <attvalues>
          <attvalue for="0" value="Scaling Laws Under the Microscope: Predicting Transformer Performance&#10;  from Small Scale Experiments" />
          <attvalue for="1" value="  Neural scaling laws define a predictable relationship between a model's&#10;parameter count and its performance after training in the form of a power law.&#10;However, most research to date has not explicitly investigated whether scaling&#10;laws can be used to accelerate model development. In this work, we perform such&#10;an empirical investigation across a wide range of language understanding tasks,&#10;starting from models with as few as 10K parameters, and evaluate downstream&#10;performance across 9 language understanding tasks. We find that scaling laws&#10;emerge at finetuning time in some NLP tasks, and that they can also be&#10;exploited for debugging convergence when training large models. Moreover, for&#10;tasks where scaling laws exist, they can be used to predict the performance of&#10;larger models, which enables effective model selection. However, revealing&#10;scaling laws requires careful hyperparameter tuning and multiple runs for the&#10;purpose of uncertainty estimation, which incurs additional overhead, partially&#10;offsetting the computational benefits.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.15647" label="2303.15647">
        <attvalues>
          <attvalue for="0" value="Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning" />
          <attvalue for="1" value="  This paper presents a systematic overview and comparison of&#10;parameter-efficient fine-tuning methods covering over 40 papers published&#10;between February 2019 and February 2023. These methods aim to resolve the&#10;infeasibility and impracticality of fine-tuning large language models by only&#10;training a small set of parameters. We provide a taxonomy that covers a broad&#10;range of methods and present a detailed method comparison with a specific focus&#10;on real-life efficiency and fine-tuning multibillion-scale language models.&#10;" />
          <attvalue for="2" value="&#10;\begin{floatingfigure}[r]{0.4\textwidth}&#10;\epigraph{One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation...}{Rich Sutton, The Bitter Lesson}&#10;\end{floatingfigure}&#10;&#10;In October 2018, BERT Large \cite{devlin2018bert} with 350 million parameters was the biggest Transformer model \cite{vaswani2017attention} ever trained. At the time, contemporary hardware struggled to fine-tune this model. The section``Out-of-memory issues'' on BERT's GitHub\footnotemark specifies the maximum batch size for BERT Large given 12Gb of GPU RAM and 512 tokens as zero.&#10;Five years in, publicly available models grew to 176 billion parameters \cite{bloom,zhang2022opt,zeng2022glm130b}, i.e. by a factor of 500. Published literature includes models up to 1 trillion parameters \cite{palm,megatron,switch}. However, single-GPU RAM increased less than 10 times due to the high cost of HBM memory.&#10;Model size scales almost two orders of magnitude quicker than computational resources making fine-tuning the largest models to downstream tasks infeasible for most and impractical for everyone.&#10;&#10;In-context learning \cite{radford2019language} thus became the new normal, the standard way to pass downstream task training data to billion-scale language models. However, the limited context length imposed by the transformer architecture \cite{vaswani2017attention, huang2018music}, the absence of ICL abilities in moderately large language models \cite{lu2023emergent}, the quadratic increase in computational cost with an increase in context length (or demonstrations in ICL) \cite{keles2023computational}, and the sensitivity of ICL performance \cite{bertsch2024context} present challenges in the utility, reliability, and efficiency of ICL. &#10;In cases where the model performs at par or better in the ICL setting compared to the fine-tuned model, fine-tuning is still a lucrative strategy due to the impractical inference cost of ICL \cite{bertsch2024context}.&#10;Thus, we, as a community of researchers and engineers, need efficient ways to train on downstream task data.&#10;&#10;Parameter-efficient fine-tuning (PEFT) aims to resolve this problem by only training a small set of parameters, which might be a subset of the existing model parameters or a set of newly added parameters. These methods differ in parameter and memory efficiency, training speed, final model quality, and additional inference costs (if any).&#10;&#10;In the last few years, more than a hundred PEFT papers have been published, with several studies \cite{delta_tuning} providing a good overview of the most popular methods, such as Adapters \cite{adapters}, BitFit \cite{bitfit}, LoRA \cite{lora}, Compacter \cite{compacter}, and Soft Prompts \cite{p_tuning,prefix_tuning}.&#10;&#10;\cite{modular_deep_learning} presented a survey on modular deep learning, providing an overview of several similar methods from the perspective of modularity and multi-task inference. Our focus differs by concentrating on PEFT methods, specifically for fine-tuning large language models, where minimizing RAM consumption and training time without sacrificing performance is crucial.&#10;&#10;This survey presents a systematic overview, comparison, and taxonomy of 30 parameter-efficient fine-tuning methods. Over the last year, research efforts have also focused on replicating the success of PEFT in the pre-training regime. Hence, we also discuss a few prominent methods that aim to achieve efficiency gains during pre-training. We discuss 30 methods in-depth, covering over 50 papers published from early 2019 to mid-2024. We highlight the current unresolved challenges in PEFT, including the limited theoretical understanding, the performance gap between PEFT and traditional fine-tuning, and reporting issues.&#10;&#10;We conduct the most &#10;extensive experimental comparison of PEFT methods &#10;, evaluating 14 methods and their variations across five datasets and three model sizes (0.7B, 3B, and 11B). The study includes a detailed comparison of these methods' efficiency in terms of GPU memory consumption and throughput. Our findings reveal that methods previously shown to outperform LoRA struggle to do so in resource-constrained settings and exhibit high hyperparameter sensitivity in hybrid PEFT methods. &#10;&#10;We found that Kronecker-based reparametrizations, while not enhancing memory efficiency compared to matrix-product counterparts, can improve training and inference speeds with efficient implementation. Surprisingly, Layer Norm tuning performs exceptionally well compared to most PEFT methods in our study. We also note a significant discrepancy between reported and actual trainable parameter counts in PEFT methods. This leads to unforeseen outcomes, such as the high computational costs of Prompt Tuning and Hybrid methods.&#10;Our code is available on Github.&#10;&#10;In conclusion, we suggest several avenues for improvement, such as developing standardized PEFT benchmarks,&#10;conducting in-depth studies on hyperparameters and interpretability,&#10;exploring the difference in training dynamics of reparametrized neural networks,&#10;further improving training and inference efficiency of PEFT methods,&#10;and utility of PEFT methods with quantized backbone models.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.12284" label="2108.12284">
        <attvalues>
          <attvalue for="0" value="The Devil is in the Detail: Simple Tricks Improve Systematic&#10;  Generalization of Transformers" />
          <attvalue for="1" value="  Recently, many datasets have been proposed to test the systematic&#10;generalization ability of neural networks. The companion baseline Transformers,&#10;typically trained with default hyper-parameters from standard tasks, are shown&#10;to fail dramatically. Here we demonstrate that by revisiting model&#10;configurations as basic as scaling of embeddings, early stopping, relative&#10;positional embedding, and Universal Transformer variants, we can drastically&#10;improve the performance of Transformers on systematic generalization. We report&#10;improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics&#10;dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity&#10;split, and from 35% to 81% on COGS. On SCAN, relative positional embedding&#10;largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100%&#10;accuracy on the length split with a cutoff at 26. Importantly, performance&#10;differences between these models are typically invisible on the IID data split.&#10;This calls for proper generalization validation sets for developing neural&#10;networks that generalize systematically. We publicly release the code to&#10;reproduce our results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.09452" label="2306.09452">
        <attvalues>
          <attvalue for="0" value="Distillation Strategies for Discriminative Speech Recognition Rescoring" />
          <attvalue for="1" value="  Second-pass rescoring is employed in most state-of-the-art speech recognition&#10;systems. Recently, BERT based models have gained popularity for re-ranking the&#10;n-best hypothesis by exploiting the knowledge from masked language model&#10;pre-training. Further, fine-tuning with discriminative loss such as minimum&#10;word error rate (MWER) has shown to perform better than likelihood-based loss.&#10;Streaming applications with low latency requirements impose significant&#10;constraints on the size of the models, thereby limiting the word error rate&#10;(WER) performance gains. In this paper, we propose effective strategies for&#10;distilling from large models discriminatively trained with the MWER objective.&#10;We experiment on Librispeech and production scale internal dataset for&#10;voice-assistant. Our results demonstrate relative improvements of upto 7% WER&#10;over student models trained with MWER. We also show that the proposed&#10;distillation can reduce the WER gap between the student and the teacher by 62%&#10;upto 100%.&#10;" />
          <attvalue for="2" value="&#10;Two-pass automatic speech recognition (ASR) systems comprise of a first-pass model to generate n-best hypotheses and a second-pass rescoring model to re-rank and pick the best hypothesis.&#10;Directly optimizing the second-pass models with respect to the WER is an appealing proposition.&#10;One way to achieve this is through the MWER objective, which is designed to directly minimize the expected WER \cite{mwer2016,xu2022rescorebert}.&#10;There have been several attempts at incorporating discriminative training using MWER loss for LSTM-RNN \cite{mwer2016,gandhe2020audio}, Sequence-to-Sequence \cite{mwer2018,li2020towards,sainath2019two}, RNN-Transducer \cite{guo2020efficient}, Transformer-Transducer \cite{meng2021minimum} based architectures for ASR.&#10;Recent studies have shown BERT models \cite{bert} with bi-directional information encoding and pre-trained knowledge are favorable for rescoring, both in log-likelihood \cite{salazar2020masked} and discriminative loss \cite{xu2022rescorebert} settings.&#10;&#10;Knowledge distillation in neural networks \cite{hinton2015distilling} has shown great success in efficiently compressing and mimicking one or ensemble of larger models (teachers).&#10;This enables use of much smaller models with the advantages of reduced training time, inference and latency costs.&#10;\cite{hinton2015distilling} proposed cross-entropy based distillation based on KL-divergence loss for distilling from an ensemble of teacher models for classification problems.&#10;\cite{romero2014fitnets} extended knowledge distillation by minimizing L2-norm of the intermediate representations from hidden layers between the teacher and the student.&#10;L2-norm based objectives are also used to closely replicate real value predictions of teacher in application to regression problems \cite{chen2017learning,wang2017model}.&#10;There have also been several successful attempts of knowledge distillation in language modeling domain.&#10;Most settings in the language model domain are posed as a classification problem and typically combination of the KL divergence loss with regularization is used.&#10;For BERT models, optimizing the cosine distance between the teacher and student embeddings is found to be useful \cite{sanh2019distilbert}.&#10;\cite{salazar2020masked} proposed to distill pseudo log-likelihoods (PLL) from BERT \cite{mlm-scoring1} using L2-loss based regression over the classification (CLS) token in application to utterance rescoring for ASR.&#10;&#10;Recently, RescoreBERT was proposed~\cite{xu2022rescorebert} to train a BERT based second-pass rescoring model with several discriminative objective functions including MWER.&#10;RescoreBERT provided up-to 5.3\% relative WER improvement, which was attributed to MWER fine-tuning.&#10;The paper also proposed a technique to distill knowledge to a smaller model of 5M parameters for low-latency streaming applications.&#10;The distillation involves training the student BERT model to predict the sentence-level PLL from the bigger 170M parameter teacher model before proceeding with MWER training.&#10;Although, reducing the model size from 170M to 5M parameters brings average latency improvement of approximately 1350\% (relative), the WER improvements are diminished by 61.3\% relative to 170M model.&#10;This suggests there is a need for better distillation strategies to attain better latency vs. WER trade-off.&#10;&#10;The advantages of using discriminative loss functions such as MWER \cite{xu2022rescorebert,gandhe2020audio} to train language models are clear.&#10;Further, the need for small models to enable low-latency streaming applications is necessary.&#10;However, efficient techniques to distill from a model trained with discriminative loss such as MWER have not been explored.&#10;Moreover, the question on effect of distillation and its ability to retain the discriminative power with respect to MWER trained model is unanswered.&#10;In this paper, we propose several techniques to perform distillation from a second-pass ASR rescoring teacher model trained with a discriminative MWER criterion.&#10;We devise loss functions to train the student model to mimic the teacher while retaining the discriminative power of the teacher to minimize the expected WER.&#10;We demonstrate that distilling from a larger teacher gives better WER in comparison to a MWER trained student model.&#10;To the best of our knowledge, this work is the first to explore distillation from a MWER trained model and explore its feasibility in application to second-pass ASR rescoring.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Recognition Systems, Model Distillation Techniques, Linguistics, Artificial Intelligence, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1912.03363" label="1912.03363">
        <attvalues>
          <attvalue for="0" value="Audio-attention discriminative language model for ASR rescoring" />
          <attvalue for="1" value="  End-to-end approaches for automatic speech recognition (ASR) benefit from&#10;directly modeling the probability of the word sequence given the input audio&#10;stream in a single neural network. However, compared to conventional ASR&#10;systems, these models typically require more data to achieve comparable&#10;results. Well-known model adaptation techniques, to account for domain and&#10;style adaptation, are not easily applicable to end-to-end systems. Conventional&#10;HMM-based systems, on the other hand, have been optimized for various&#10;production environments and use cases. In this work, we propose to combine the&#10;benefits of end-to-end approaches with a conventional system using an&#10;attention-based discriminative language model that learns to rescore the output&#10;of a first-pass ASR system. We show that learning to rescore a list of&#10;potential ASR outputs is much simpler than learning to generate the hypothesis.&#10;The proposed model results in 8% improvement in word error rate even when the&#10;amount of training data is a fraction of data used for training the first-pass&#10;system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;{\mwer} criterion has been previously used in ~\cite{hori2016minimum} to improve the performance of LSTM LMs. They train the model using the {\mwer} criterion but they do not explore using an un-normalized model or a model with attention to audio. ~\cite{guo2019spelling} also explored training a spelling correction model to learn the errors made by a first pass system. However, they did not explore using this model for a conventional system or condition on input audio (e.g. by attending to input audio). Besides, their spelling correction model is a generative model (decoder generates corrected characters in a left-to-right manner). Recently, ~\cite{Chiu2019} proposed to use a listen-attend-and-spell (LAS) model as a second pass rescoring model for output of an RNN-Transducer (RNN-T) first pass. They show significant improvements with LAS as rescoring component, even with a shared encoder. However, the amount of training data used for LAS rescoring was the same as the training data used for first pass model while we explore training the rescoring model with much less amount of data and is important for low-resource languages or domain adaptation with limited amount of in-domain data. The best performance was also achieved when the shared encoder was fine tuned in multi-task fashion (combined loss for both RNN-T and LAS). Additionally, they do not report results on using such a rescoring scheme with a conventional ASR model.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1712.01818" label="1712.01818">
        <attvalues>
          <attvalue for="0" value="Minimum Word Error Rate Training for Attention-based&#10;  Sequence-to-Sequence Models" />
          <attvalue for="1" value="  Sequence-to-sequence models, such as attention-based models in automatic&#10;speech recognition (ASR), are typically trained to optimize the cross-entropy&#10;criterion which corresponds to improving the log-likelihood of the data.&#10;However, system performance is usually measured in terms of word error rate&#10;(WER), not log-likelihood. Traditional ASR systems benefit from discriminative&#10;sequence training which optimizes criteria such as the state-level minimum&#10;Bayes risk (sMBR) which are more closely related to WER. In the present work,&#10;we explore techniques to train attention-based models to directly minimize&#10;expected word error rate. We consider two loss functions which approximate the&#10;expected number of word errors: either by sampling from the model, or by using&#10;N-best lists of decoded hypotheses, which we find to be more effective than the&#10;sampling-based method. In experimental evaluations, we find that the proposed&#10;training procedure improves performance by up to 8.2% relative to the baseline&#10;system. This allows us to train grapheme-based, uni-directional attention-based&#10;models which match the performance of a traditional, state-of-the-art,&#10;discriminative sequence-trained system on a mobile voice-search task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.11544" label="2004.11544">
        <attvalues>
          <attvalue for="0" value="Towards Fast and Accurate Streaming End-to-End ASR" />
          <attvalue for="1" value="  End-to-end (E2E) models fold the acoustic, pronunciation and language models&#10;of a conventional speech recognition model into one neural network with a much&#10;smaller number of parameters than a conventional ASR system, thus making it&#10;suitable for on-device applications. For example, recurrent neural network&#10;transducer (RNN-T) as a streaming E2E model has shown promising potential for&#10;on-device ASR. For such applications, quality and latency are two critical&#10;factors. We propose to reduce E2E model's latency by extending the RNN-T&#10;endpointer (RNN-T EP) model with additional early and late penalties. By&#10;further applying the minimum word error rate (MWER) training technique, we&#10;achieved 8.0% relative word error rate (WER) reduction and 130ms 90-percentile&#10;latency reduction over on a Voice Search test set. We also experimented with a&#10;second-pass Listen, Attend and Spell (LAS) rescorer . Although it did not&#10;directly improve the first pass latency, the large WER reduction provides extra&#10;room to trade WER for latency. RNN-T EP+LAS, together with MWER training brings&#10;in 18.7% relative WER reduction and 160ms 90-percentile latency reductions&#10;compared to the original proposed RNN-T EP model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.10992" label="1908.10992">
        <attvalues>
          <attvalue for="0" value="Two-Pass End-to-End Speech Recognition" />
          <attvalue for="1" value="  The requirements for many applications of state-of-the-art speech recognition&#10;systems include not only low word error rate (WER) but also low latency.&#10;Specifically, for many use-cases, the system must be able to decode utterances&#10;in a streaming fashion and faster than real-time. Recently, a streaming&#10;recurrent neural network transducer (RNN-T) end-to-end (E2E) model has shown to&#10;be a good candidate for on-device speech recognition, with improved WER and&#10;latency metrics compared to conventional on-device models [1]. However, this&#10;model still lags behind a large state-of-the-art conventional model in quality&#10;[2]. On the other hand, a non-streaming E2E Listen, Attend and Spell (LAS)&#10;model has shown comparable quality to large conventional models [3]. This work&#10;aims to bring the quality of an E2E streaming model closer to that of a&#10;conventional system by incorporating a LAS network as a second-pass component,&#10;while still abiding by latency constraints. Our proposed two-pass model&#10;achieves a 17%-22% relative reduction in WER compared to RNN-T alone and&#10;increases latency by a small fraction over RNN-T.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.02929" label="1709.02929">
        <attvalues>
          <attvalue for="0" value="Model Distillation with Knowledge Transfer from Face Classification to&#10;  Alignment and Verification" />
          <attvalue for="1" value="  Knowledge distillation is a potential solution for model compression. The&#10;idea is to make a small student network imitate the target of a large teacher&#10;network, then the student network can be competitive to the teacher one. Most&#10;previous studies focus on model distillation in the classification task, where&#10;they propose different architects and initializations for the student network.&#10;However, only the classification task is not enough, and other related tasks&#10;such as regression and retrieval are barely considered. To solve the problem,&#10;in this paper, we take face recognition as a breaking point and propose model&#10;distillation with knowledge transfer from face classification to alignment and&#10;verification. By selecting appropriate initializations and targets in the&#10;knowledge transfer, the distillation can be easier in non-classification tasks.&#10;Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the&#10;student network can be competitive to the teacher one in alignment and&#10;verification, and even surpasses the teacher network under specific compression&#10;rates. In addition, to achieve stronger knowledge transfer, we also use a&#10;common initialization trick to improve the distillation performance of&#10;classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M&#10;datasets show the effectiveness of this simple trick.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.00194" label="1910.00194">
        <attvalues>
          <attvalue for="0" value="Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word&#10;  Representations" />
          <attvalue for="1" value="  Contextualized word representations are able to give different&#10;representations for the same word in different contexts, and they have been&#10;shown to be effective in downstream natural language processing tasks, such as&#10;question answering, named entity recognition, and sentiment analysis. However,&#10;evaluation on word sense disambiguation (WSD) in prior work shows that using&#10;contextualized word representations does not outperform the state-of-the-art&#10;approach that makes use of non-contextualized word embeddings. In this paper,&#10;we explore different strategies of integrating pre-trained contextualized word&#10;representations and our best strategy achieves accuracies exceeding the best&#10;prior published accuracies by significant margins on multiple benchmark WSD&#10;datasets. We make the source code available at&#10;https://github.com/nusnlp/contextemb-wsd.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Continuous word representations in real-valued vectors, or commonly known as word embeddings, have been shown to help improve NLP performance. Initially, exploiting continuous representations was achieved by adding real-valued vectors as classification features \cite{turian_word_2010}. \cite{taghipour_semi-supervised_2015} fine-tuned non-contextualized word embeddings by a feed-forward neural network such that those word embeddings were more suited for WSD. The fine-tuned embeddings were incorporated into an SVM classifier. \cite{iacobacci_embeddings_2016} explored different strategies of incorporating word embeddings and found that their best strategy involved exponential decay that decreased the contribution of surrounding word features as their distances to the target word increased.&#10;&#10;The neural sequence tagging approach has also been explored for WSD. \cite{kageback_word_2016} proposed bidirectional long short-term memory (LSTM) \cite{hochreiter_long_1997} for WSD. They concatenated the hidden states of the forward and backward LSTMs and fed the concatenation into an affine transformation followed by softmax normalization, similar to the approach to incorporate a bidirectional LSTM adopted in sequence labeling tasks such as part-of-speech tagging and named entity recognition \cite{ma_end--end_2016}. \cite{raganato_neural_2017} proposed a self-attention layer on top of the concatenated bidirectional LSTM hidden states for WSD and introduced multi-task learning with part-of-speech tagging and semantic labeling as auxiliary tasks. However, on average across the test sets, their approach did not outperform SVM with word embedding features. Subsequently, \cite{luo_incorporating_2018} proposed the incorporation of glosses from WordNet in a bidirectional LSTM for WSD, and reported better results than both SVM and prior bidirectional LSTM models.&#10;&#10;A neural language model (LM) is aimed at predicting a word given its surrounding context. As such, the resulting hidden representation vector captures the context of a word in a sentence. \cite{melamud_context2vec:_2016} designed context2vec, which is a one-layer bidirectional LSTM trained to maximize the similarity between the hidden state representation of the LSTM and the target word embedding. \cite{peters_deep_2018} designed ELMo, which is a two-layer bidirectional LSTM language model trained to predict the next word in the forward LSTM and the previous word in the backward LSTM. In both models, WSD was evaluated by nearest neighbor matching between the test and training instance representations. However, despite training on a huge amount of raw texts, the resulting accuracies were still lower than those achieved by WSD approaches with pre-trained non-contextualized word representations.&#10;&#10;End-to-end neural machine translation (NMT) \cite{sutskever_sequence_2014,bahdanau_neural_2015} learns to generate an output sequence given an input sequence, using an encoder-decoder model. The encoder captures the contextualized representation of the words in the input sentence for the decoder to generate the output sentence. Following this intuition, \cite{mccann_learned_2017} trained an encoder-decoder model on parallel texts and obtained pre-trained contextualized word representations from the encoder.&#10;&#10;" />
          <attvalue for="4" value="Word Representation Models, Computer Science, Linguistics, Contextualized Embeddings, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2112.11638" label="2112.11638">
        <attvalues>
          <attvalue for="0" value="Consistency and Coherence from Points of Contextual Similarity" />
          <attvalue for="1" value="  Factual consistency is one of important summary evaluation dimensions,&#10;especially as summary generation becomes more fluent and coherent. The ESTIME&#10;measure, recently proposed specifically for factual consistency, achieves high&#10;correlations with human expert scores both for consistency and fluency, while&#10;in principle being restricted to evaluating such text-summary pairs that have&#10;high dictionary overlap. This is not a problem for current styles of&#10;summarization, but it may become an obstacle for future summarization systems,&#10;or for evaluating arbitrary claims against the text. In this work we generalize&#10;the method, and make a variant of the measure applicable to any text-summary&#10;pairs. As ESTIME uses points of contextual similarity, it provides insights&#10;into usefulness of information taken from different BERT layers. We observe&#10;that useful information exists in almost all of the layers except the several&#10;lowest ones. For consistency and fluency - qualities focused on local text&#10;details - the most useful layers are close to the top (but not at the top); for&#10;coherence and relevance we found a more complicated and interesting picture.&#10;" />
          <attvalue for="2" value="&#10;A summary is assessed by evaluating its qualities, which can be defined in different ways \cite{Fan:2018:Robust, Xenouleas:2019:SUMQE, Kryscinski:2020:EvaluatingFactual, Vasilyev:2020:Sensitivity, Fabbri:2021:SummEval}. The commonly considered qualities are of two classes: summary-focused and summarization-focused. The summary-focused qualities are supposed to reflect the language of the summary itself, without any relation to the summarization. For example, grammar, fluency, structure and coherence of a summary should not require consideration of the text from which the summary is produced: grading a summary in this respect is no different from evaluation of any other text, generated or not. The summarization-focused qualities are supposed to reflect the summarization, and require both the summary and the text considered together, - for example, relevancy, informativeness, factual consistency. &#10;&#10;The automated evaluation measures normally use both the summary and the text \cite{Louis:2009:JShannon, Scialom:2019:SummaQA, Gao:2020:SUPERT, Vasilyev:2020:BLANC, Vasilyev:2020:ESTIME, Scialom:2021:QuestEval}; other measures use the summary and the 'reference summaries' - the summaries human-written specifically for the text \cite{Papineni:2002:BLEU, Lin:2004:ROUGE, Zhang:2020:BERTScore}. Correlation of evaluation measures with human scores for all qualities is widely accepted as a criterion for judging about the evaluation measures \cite{Fabbri:2021:SummEval}, with a few caveats.&#10;&#10;Since the summary qualities (e.g. relevance, consistency, coherence and fluency) are all different, improving a measure eventually cannot be expected to keep improving the correlations with all the qualities. &#10;It would be natural to have measures intentionally focused on certain qualities. &#10;If a measure correlates reasonably well (by current standards) with all the qualities, this may reflect the simple reality that better generation systems produce summaries with better qualities overall, and that easier texts allow to produce summaries with better qualities overall. There is also a possibility of an implicit bias even in expert scores \cite{Vasilyev:2021:NoHuman}.&#10;&#10;Factual consistency is arguably the most objective quality of the summary, and there were consistent efforts to improve its evaluation \cite{Falke:2019:Ranking, Kryscinski:2020:EvaluatingFactual, Wang:2020:Asking, Maynez:2020:Faithfulness, Scialom:2021:QuestEval, Gabriel:2021:GoFigure}. Recently introduced measure ESTIME is focused on factual consistency, and achieves superior scores both for consistency and fluency \cite{Vasilyev:2020:ESTIME}. The measure is simple, interpretable and easily reproducible: It is not tuned on some specific human-annotated dataset but simply using a well known pretrained language model, and almost no parameters to set or chose. It can use any other pretrained language model, including multilingual BERT, thus being ready for other languages. &#10;However, ESTIME can be used only in situations where the summary and the text have high dictionary overlap. This is not a problem for evaluating current generation systems (which rarely introduce words not from the text), but this makes more problematic a progress toward estimation of factual consistency of an arbitrary 'claim' (not necessarily a summary) with respect to a text.&#10;&#10;Our contribution:&#10;\begin{enumerate}[topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex]&#10; \item We generalize ESTIME, and present its variant applicable to more scenarios.&#10; \item We get insights on usefulness of embeddings from different BERT layers for evaluating summary qualities.&#10; \item We provide an alternative measure for evaluating coherence. Curiously, its correlations with human scores are better when using the lower-middle part of large BERT.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Factual Consistency, Computer Science, Language Model Analysis, Linguistics, Text Summarization, Computational Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1810.06065" label="1810.06065">
        <attvalues>
          <attvalue for="0" value="Robust Neural Abstractive Summarization Systems and Evaluation against&#10;  Adversarial Information" />
          <attvalue for="1" value="  Sequence-to-sequence (seq2seq) neural models have been actively investigated&#10;for abstractive summarization. Nevertheless, existing neural abstractive&#10;systems frequently generate factually incorrect summaries and are vulnerable to&#10;adversarial information, suggesting a crucial lack of semantic understanding.&#10;In this paper, we propose a novel semantic-aware neural abstractive&#10;summarization model that learns to generate high quality summaries through&#10;semantic interpretation over salient content. A novel evaluation scheme with&#10;adversarial samples is introduced to measure how well a model identifies&#10;off-topic information, where our model yields significantly better performance&#10;than the popular pointer-generator summarizer. Human evaluation also confirms&#10;that our system summaries are uniformly more informative and faithful as well&#10;as less redundant than the seq2seq model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.14602" label="2012.14602">
        <attvalues>
          <attvalue for="0" value="Is human scoring the best criteria for summary evaluation?" />
          <attvalue for="1" value="  Normally, summary quality measures are compared with quality scores produced&#10;by human annotators. A higher correlation with human scores is considered to be&#10;a fair indicator of a better measure. We discuss observations that cast doubt&#10;on this view. We attempt to show a possibility of an alternative indicator.&#10;Given a family of measures, we explore a criterion of selecting the best&#10;measure not relying on correlations with human scores. Our observations for the&#10;BLANC family of measures suggest that the criterion is universal across very&#10;different styles of summaries.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.03972" label="2307.03972">
        <attvalues>
          <attvalue for="0" value="Evaluating the Capability of Large-scale Language Models on Chinese&#10;  Grammatical Error Correction Task" />
          <attvalue for="1" value="  Large-scale language models (LLMs) has shown remarkable capability in various&#10;of Natural Language Processing (NLP) tasks and attracted lots of attention&#10;recently. However, some studies indicated that large language models fail to&#10;achieve promising result beyond the state-of-the-art models in English&#10;grammatical error correction (GEC) tasks. In this report, we aim to explore the&#10;how large language models perform on Chinese grammatical error correction tasks&#10;and provide guidance for future work. We conduct experiments with 3 different&#10;LLMs of different model scale on 4 Chinese GEC dataset. Our experimental&#10;results indicate that the performances of LLMs on automatic evaluation metrics&#10;falls short of the previous sota models because of the problem of&#10;over-correction. Furthermore, we also discover notable variations in the&#10;performance of LLMs when evaluated on different data distributions. Our&#10;findings demonstrates that further investigation is required for the&#10;application of LLMs on Chinese GEC task.&#10;" />
          <attvalue for="2" value="&#10;&#10;Building on InstructGPT~\cite{DBLP:conf/nips/Ouyang0JAWMZASR22}, ChatGPT has demonstrated its powerful ability to understand complex instruction and generate reasonable responses on various of NLP tasks. Following the technical trajectory of ChatGPT, a significant number of high-quality LLMs have emerged in recent times in both academia and industry, such as LLaMA~\cite{DBLP:journals/corr/abs-2302-13971}, ChatGLM~\cite{du2022glm} and PaLM~\cite{DBLP:journals/corr/abs-2305-10403}. Previous studies found that these LLMs have achieved great performance on a wide range of NLP tasks, including machine translation~\cite{DBLP:journals/corr/abs-2301-08745}, named entity recognition~\cite{DBLP:journals/corr/abs-2305-05862} and text summarization~\cite{DBLP:journals/corr/abs-2302-08081}.&#10;&#10;Certain studies have token comprehensive investigations into the performance of LLMs in the domain of English grammatical error correction, yielding some interesting findings~\cite{DBLP:journals/corr/abs-2304-01746,DBLP:journals/corr/abs-2303-13648}. LLMs are not able to outperform sota models in terms of automatic evaluation metrics. This is primarily because LLMs tend to make unnecessary modifications to make the input sentences more fluent, which may result in over correction problem, and in some cases, even alter the original semantics of the input sentences. &#10;&#10;In this report, we aim to explore the performance of LLMs in Chinese GEC task. We conducted experiments on various LLMs to investigate the influence of model size on the GEC results. Additionally, we attempted different test dataset from various data sources to explore the impact of data distribution on the outcomes.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Grammatical Error Correction, Natural Language Processing, Language Model Performance" />
        </attvalues>
      </node>
      <node id="2404.00699" label="2404.00699">
        <attvalues>
          <attvalue for="0" value="How Much are Large Language Models Contaminated? A Comprehensive Survey&#10;  and the LLMSanitize Library" />
          <attvalue for="1" value="  With the rise of Large Language Models (LLMs) in recent years, abundant new&#10;opportunities are emerging, but also new challenges, among which contamination&#10;is quickly becoming critical. Business applications and fundraising in AI have&#10;reached a scale at which a few percentage points gained on popular&#10;question-answering benchmarks could translate into dozens of millions of&#10;dollars, placing high pressure on model integrity. At the same time, it is&#10;becoming harder and harder to keep track of the data that LLMs have seen; if&#10;not impossible with closed-source models like GPT-4 and Claude-3 not divulging&#10;any information on the training set. As a result, contamination becomes a major&#10;issue: LLMs' performance may not be reliable anymore, as the high performance&#10;may be at least partly due to their previous exposure to the data. This&#10;limitation jeopardizes the entire progress in the field of NLP, yet, there&#10;remains a lack of methods on how to efficiently detect contamination.In this&#10;paper, we survey all recent work on contamination detection with LLMs, and help&#10;the community track contamination levels of LLMs by releasing an open-source&#10;Python library named LLMSanitize implementing major contamination detection&#10;algorithms.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the rapidly evolving landscape of artificial intelligence (AI), Large Language Models (LLMs) have emerged as pivotal tools, driving innovation across a wide spectrum of applications, from natural language processing (NLP) and automated content creation \cite{achiam2023gpt,betker2023improving} to complex decision-making systems and autonomous agents \cite{wei2022chain,li2023chain,yang2023auto,wu2023autogen}. At their core, these models rely on extensive datasets \cite{commoncrawl,gao2020pile} to learn about language and the world, and generate responses that are increasingly indistinguishable from human-authored output \cite{instructgpt}. However, the integrity of these datasets is paramount, as any contamination can significantly impair the models' effectiveness and reliability. &#10;&#10;Detecting contamination in a LLM is a broad issue that we divide into open-data contamination detection and closed-data contamination detection. On the one hand, open-data refers to the scenario where the LLM pre-training data is known, enabling direct comparisons with the evaluation dataset. On the other hand, closed-data contamination refers to the more and more prevalent use case of an unknown pre-training set. This latter case is more challenging and necessitates an examination of the LLM's behavior on evaluation data points. &#10;Additionally, we highlight the following orthogonal dimension to the duality explained above: if we consider whether the contaminated data contains only inputs or both inputs and labels, contamination includes input contamination and input+label contamination. A third dimension to the problem is the level of access to the LLM itself in closed-data setups: white-box detection where we have full access to the model, {e.g.,}\ a local model, and black-box detection where only the API access is available, {e.g.,}\ GPT-4 \cite{achiam2023gpt}.&#10;&#10;Contamination poses a multifaceted challenge, threatening not only the technical accuracy of LLMs but also their ethical and commercial viability. In high-stakes scenarios where imprecision can have dramatic consequences, such as medical diagnosis, legal advice, or financial services, the repercussions of relying on contaminated data can be profound. Moreover, the allure of leveraging LLM outputs to attract investment underscores a pressing commercial dimension. As businesses increasingly integrate AI-driven insights into their strategic planning and operational decisions, the assurance of data purity becomes intertwined with potential market success and valuation. &#10;Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination. &#10;&#10;This landscape necessitates a comprehensive survey and a shared integrated resource of contamination detection in LLMs. &#10;We delineate the scope and nature of contamination detection challenges, identifying their types.&#10;In doing so, we not only contribute to the academic and practical understanding of data contamination issues but also highlight the pressing need for strategies that mitigate these risks.&#10;&#10;To push forward research in contamination detection, we make the following contributions:&#10;\begin{itemize}[leftmargin=*,itemsep=0.1em]&#10; \item To the best of our knowledge, we are the first to thoroughly review the field of contamination detection in LLMs. &#10;&#10;\item We categorize contamination detection into two broad types of use cases vastly differing in the techniques involved, namely open-data and closed-data contamination detection, and review all existing work in each category. We refer to \ref{fig:cls} for an overview of our proposed classification of contamination detection methods. &#10;&#10;\item We introduce LLMSanitize, an open-source Python library aimed at helping the community centralize a shared implementation of all contamination detection methods. &#10;\end{itemize}&#10;&#10;The rest of this paper is structured as follows: Section 2 explores methods and findings related to open-data contamination detection. Section 3 addresses the issue of closed-data contamination detection, where models may be evaluated on data that they have previously seen. Section 4 discusses the current and future challenges in the field, including best practices. Section 5 briefly introduces our library LLMSanitize implementing contamination detection algorithms, which is further described in Appendix.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Data Science, Mathematics, Contamination Detection, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2306.02224" label="2306.02224">
        <attvalues>
          <attvalue for="0" value="Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions" />
          <attvalue for="1" value="  Auto-GPT is an autonomous agent that leverages recent advancements in&#10;adapting Large Language Models (LLMs) for decision-making tasks. While there&#10;has been a growing interest in Auto-GPT stypled agents, questions remain&#10;regarding the effectiveness and flexibility of Auto-GPT in solving real-world&#10;decision-making tasks. Its limited capability for real-world engagement and the&#10;absence of benchmarks contribute to these uncertainties. In this paper, we&#10;present a comprehensive benchmark study of Auto-GPT styled agents in&#10;decision-making tasks that simulate real-world scenarios. Our aim is to gain&#10;deeper insights into this problem and understand the adaptability of GPT-based&#10;agents. We compare the performance of popular LLMs such as GPT-4, GPT-3.5,&#10;Claude, and Vicuna in Auto-GPT styled decision-making tasks. Furthermore, we&#10;introduce the Additional Opinions algorithm, an easy and effective method that&#10;incorporates supervised/imitation-based learners into the Auto-GPT scheme. This&#10;approach enables lightweight supervised learning without requiring fine-tuning&#10;of the foundational LLMs. We demonstrate through careful baseline comparisons&#10;and ablation studies that the Additional Opinions algorithm significantly&#10;enhances performance in online decision-making benchmarks, including WebShop&#10;and ALFWorld.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.16989" label="2311.16989">
        <attvalues>
          <attvalue for="0" value="ChatGPT's One-year Anniversary: Are Open-Source Large Language Models&#10;  Catching up?" />
          <attvalue for="1" value="  Upon its release in late 2022, ChatGPT has brought a seismic shift in the&#10;entire landscape of AI, both in research and commerce. Through&#10;instruction-tuning a large language model (LLM) with supervised fine-tuning and&#10;reinforcement learning from human feedback, it showed that a model could answer&#10;human questions and follow instructions on a broad panel of tasks. Following&#10;this success, interests in LLMs have intensified, with new LLMs flourishing at&#10;frequent interval across academia and industry, including many start-ups&#10;focused on LLMs. While closed-source LLMs (e.g., OpenAI's GPT, Anthropic's&#10;Claude) generally outperform their open-source counterparts, the progress on&#10;the latter has been rapid with claims of achieving parity or even better on&#10;certain tasks. This has crucial implications not only on research but also on&#10;business. In this work, on the first anniversary of ChatGPT, we provide an&#10;exhaustive overview of this success, surveying all tasks where an open-source&#10;LLM has claimed to be on par or better than ChatGPT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.17591" label="2310.17591">
        <attvalues>
          <attvalue for="0" value="Lil-Bevo: Explorations of Strategies for Training Language Models in&#10;  More Humanlike Ways" />
          <attvalue for="1" value="  We present Lil-Bevo, our submission to the BabyLM Challenge. We pretrained&#10;our masked language models with three ingredients: an initial pretraining with&#10;music data, training on shorter sequences before training on longer ones, and&#10;masking specific tokens to target some of the BLiMP subtasks. Overall, our&#10;baseline models performed above chance, but far below the performance levels of&#10;larger LLMs trained on more data. We found that training on short sequences&#10;performed better than training on longer sequences.Pretraining on music may&#10;help performance marginally, but, if so, the effect seems small. Our targeted&#10;Masked Language Modeling augmentation did not seem to improve model performance&#10;in general, but did seem to help on some of the specific BLiMP tasks that we&#10;were targeting (e.g., Negative Polarity Items). Training performant LLMs on&#10;small amounts of data is a difficult but potentially informative task. While&#10;some of our techniques showed some promise, more work is needed to explore&#10;whether they can improve performance more than the modest gains here. Our code&#10;is available at https://github.com/venkatasg/Lil-Bevo and out models at&#10;https://huggingface.co/collections/venkatasg/babylm-653591cdb66f4bf68922873a&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models (LLMs) generate complex and largely grammatical strings and display impressive performance with structures traditionally thought to require abstract and hierarchical syntax \cite{linzen2016assessing,linzen2021syntactic,wilcoxLI,futrell2019rnns}. They have achieved human-like performance at a wide range of natural language tasks \cite{bubeck2023sparks,frank_2023}, particularly those having to do with linguistic form \cite{mahowald2023dissociating}.&#10;This state of affairs has led to claims that such models should be taken seriously as cognitive models of human language \cite{piantadosi2023modern,baroni2021proper,frank_2023}, in line with claims from the neuroscience literature to ``take mechanistic abstraction seriously'' \cite{cao2021explanatory}.&#10;&#10;One reason that has been posited not to take LLMs seriously as cognitive models, though, is the immense amount of data they are trained on relative to what a human child is exposed to \cite{warstadt2022what,van-schijndel-etal-2019-quantity}.&#10;Thus, it is possible that models memorize more than humans do and, relative to humans, over-rely on statistical heuristics and memorized chunks of language \cite{bender2021dangers}.&#10;&#10;On the other hand, the quality of data that LLMs get during pretraining is, in many ways, much worse than what human learners get. &#10;Children get richly structured, interactive, multimodal input, tailored to their specific interests and needs.&#10;A baby might reach for a cup of water and be told ``Water. You want some water?'' &#10;Given that babies are known to conduct repeated experiments to learn about the world \cite{gopnik1999scientist}, the baby might try this again and again until mastering the concept of what water is.&#10;An LLM, meanwhile, might begin learning language by being asked to predict random tokens in the Wikipedia article on quantum mechanics.&#10;&#10;In this paper, we describe our experiments with Lil-Bevo, a small language model trained on human-scale data for the BabyLM competition \cite{warstadt-et-al-2023-babylm}.&#10;The goal of the competition is to train a performant LM on a human-scale amount of data: 10M words for the small track, 100M for the larger track.&#10;We submitted to both strict tracks --- however, we were notified through the meta-review that our models qualify only for the loose track due to the usage of additional non-linguistic data (music from the MAESTRO dataset~\cite{hawthorne2018enabling}).&#10;The evaluation is on a set of natural language tasks including grammatical acceptability judgments via minimal pairs in the BLiMP benchmark \cite{warstadt2020blimp}, language understanding tasks in SuperGLUE \cite{wang2019superglue}, and MSGS (the Mixed Signals Generalization Set) \cite{warstadt-etal-2020-learning}&#10;&#10;We started with a baseline DeBERTa model, trained from scratch on BabyLM data using a custom unigram SentencePiece tokenizer ~\cite{kudo-richardson-2018-sentencepiece}.&#10;Our strategy was not focused on the architecture, but on ways in which we could adjust the training regime to improve performance above the baseline.&#10;&#10;Specifically, our strategy targets 3 ways in which typical LLM training regimes lead to lower-quality data than humans have access to.&#10;Here, we describe those strategies and their motivation. &#10;We give detailed methods in Section~\ref{sec:experiments_and_methods} and then present results, including a number of ablation studies that attempt to partition out what strategies were successful.&#10;&#10;We treated these studies as proof-of-concept and did not exhaustively test these strategies.&#10;Thus, we think that there is still room for improvement.&#10;&#10;Training on Short Sequences&#10;&#10;Unlike LLMs, babies do not start language by learning long complicated sequences all at once.&#10;Using databases of child and child-directed speech, it has been shown that there is some alignment of caretakers to the child's level in terms of linguistic complexity such that caregivers talk to younger children using shorter utterances and longer utterances as they develop \cite{schwab2016language,kunert2011adaptation}. &#10;To that end, \cite{mueller-linzen-2023-plant} showed that training on simpler data first could induce a better hierarchical bias for learning language.&#10;We specifically take inspiration from \cite{press-etal-2021-shortformer} who showed that LLMs learn better when trained on shorter sequences before being trained on longer sequences.&#10;&#10;Training on Music Before Training on Language&#10;Unlike LLMs, babies are exposed to a wide range of input besides just text.&#10;Before and while learning language, they are also learning to map the visual world, to navigate the physical world, to process non-linguistic auditory stimuli, and to engage in a wide variety of cognitive operations.&#10;Thus, it is commonly observed that some of the machinery thought to be language-specific (e.g., hierarchical structure) might be induced in pre-linguistic infants through exposure to other kinds of stimuli.&#10;\cite{papadimitriou-jurafsky-2020-learning} use this idea to show that training language models on structured data (e.g., music) can help models learn faster.&#10;We use a similar idea, with initial pretraining on a mix of music (piano performances) and text.&#10;&#10;Targeted Masked Language Model The role of child-directed speech in human language learning is controversial \cite[see][for discussion and a large-scale replication of infant-directed speech preferences]{manybabies2020quantifying}.&#10;It is generally agreed that parents do not correct a child every time they make a grammatical error \cite{marcus1993negative}, but there is also evidence that social feedback acts as a signal \cite{tomasello1992social} and that parents structure input to be helpful \cite{weisleder2013talking}.&#10;When a child says something wrong, a parent might ``recast'' the utterance or highlight grammatical features that children are struggling with \cite{nicholas2001recasts}. &#10;Inspired by this idea, targeting the BLiMP \cite{warstadt2020blimp} syntactic evaluations as well as more general tasks, we trained with a targeted MLM objective. &#10;&#10;We considered some variations of the idea of learning with some external feedback that distinguishes correct tokens against corrupted/noisy replacements. For example, ELECTRA \cite{Clark2020ELECTRA} consists in learning to detect tokens which have been replaced by an auxiliary model. Unfortunately, replaced token detection approaches such as ELECTRA \cite{Clark2020ELECTRA} suffer from an inability to learn probability distributions over the entire vocabulary, and so cannot be used for (pseudo)-likelihood scoring \cite{salazar-etal-2020-masked}. Another related approach is Corrective Language Modeling (CLM) \cite{DBLP:journals/corr/abs-2204-06644}, in which the model is trained to correctly replace corrupted tokens; however, it is not clear how to best use these models for scoring sentences in BLiMP.&#10;&#10;Given the problems outlined above, we decided to use masked language modeling (MLM) with targeted masks. The motivation is to make it easier for the model to learn syntactic phenomena that co-occur frequently with certain words. Other strategies for selecting masks were used in \cite{sadeq-etal-2022-informask,gu-etal-2020-train}; unlike these works, we mask specific words which are essential to the phenomena in BLiMP. For example, to target the filler-gap dependency subtask in BLiMP, we go through the original data set and mask every occurrence of ``that'' and ``what'' in the corpus. By focusing on these words, we anticipate that the model will more quickly learn to score ``I know what you did last summer.'' more highly than ``I know that you did last summer.'' &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Masked Language Modeling, Linguistics, Cognitive Science, Language Model Training, Limited Data Performance" />
        </attvalues>
      </node>
      <node id="2004.10827" label="2004.10827">
        <attvalues>
          <attvalue for="0" value="Syntactic Structure from Deep Learning" />
          <attvalue for="1" value="  Modern deep neural networks achieve impressive performance in engineering&#10;applications that require extensive linguistic skills, such as machine&#10;translation. This success has sparked interest in probing whether these models&#10;are inducing human-like grammatical knowledge from the raw data they are&#10;exposed to, and, consequently, whether they can shed new light on long-standing&#10;debates concerning the innate structure necessary for language acquisition. In&#10;this article, we survey representative studies of the syntactic abilities of&#10;deep networks, and discuss the broader implications that this work has for&#10;theoretical linguistics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.01866" label="1811.01866">
        <attvalues>
          <attvalue for="0" value="Do RNNs learn human-like abstract word order preferences?" />
          <attvalue for="1" value="  RNN language models have achieved state-of-the-art results on various tasks,&#10;but what exactly they are representing about syntax is as yet unclear. Here we&#10;investigate whether RNN language models learn humanlike word order preferences&#10;in syntactic alternations. We collect language model surprisal scores for&#10;controlled sentence stimuli exhibiting major syntactic alternations in English:&#10;heavy NP shift, particle shift, the dative alternation, and the genitive&#10;alternation. We show that RNN language models reproduce human preferences in&#10;these alternations based on NP length, animacy, and definiteness. We collect&#10;human acceptability ratings for our stimuli, in the first acceptability&#10;judgment experiment directly manipulating the predictors of syntactic&#10;alternations. We show that the RNNs' performance is similar to the human&#10;acceptability ratings and is not matched by an n-gram baseline model. Our&#10;results show that RNNs learn the abstract features of weight, animacy, and&#10;definiteness which underlie soft constraints on syntactic alternations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.12247" label="1810.12247">
        <attvalues>
          <attvalue for="0" value="Enabling Factorized Piano Music Modeling and Generation with the MAESTRO&#10;  Dataset" />
          <attvalue for="1" value="  Generating musical audio directly with neural networks is notoriously&#10;difficult because it requires coherently modeling structure at many different&#10;timescales. Fortunately, most music is also highly structured and can be&#10;represented as discrete note events played on musical instruments. Herein, we&#10;show that by using notes as an intermediate representation, we can train a&#10;suite of models capable of transcribing, composing, and synthesizing audio&#10;waveforms with coherent musical structure on timescales spanning six orders of&#10;magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large&#10;advance in the state of the art is enabled by our release of the new MAESTRO&#10;(MIDI and Audio Edited for Synchronous TRacks and Organization) dataset,&#10;composed of over 172 hours of virtuosic piano performances captured with fine&#10;alignment (~3 ms) between note labels and audio waveforms. The networks and the&#10;dataset together present a promising approach toward creating new expressive&#10;and interpretable neural models of music.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.06226" label="1808.06226">
        <attvalues>
          <attvalue for="0" value="SentencePiece: A simple and language independent subword tokenizer and&#10;  detokenizer for Neural Text Processing" />
          <attvalue for="1" value="  This paper describes SentencePiece, a language-independent subword tokenizer&#10;and detokenizer designed for Neural-based text processing, including Neural&#10;Machine Translation. It provides open-source C++ and Python implementations for&#10;subword units. While existing subword segmentation tools assume that the input&#10;is pre-tokenized into word sequences, SentencePiece can train subword models&#10;directly from raw sentences, which allows us to make a purely end-to-end and&#10;language independent system. We perform a validation experiment of NMT on&#10;English-Japanese machine translation, and find that it is possible to achieve&#10;comparable accuracy to direct subword training from raw sentences. We also&#10;compare the performance of subword training and segmentation with various&#10;configurations. SentencePiece is available under the Apache 2 license at&#10;https://github.com/google/sentencepiece.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.19905" label="2305.19905">
        <attvalues>
          <attvalue for="0" value="How to Plant Trees in Language Models: Data and Architectural Effects on&#10;  the Emergence of Syntactic Inductive Biases" />
          <attvalue for="1" value="  Accurate syntactic representations are essential for robust generalization in&#10;natural language. Recent work has found that pre-training can teach language&#10;models to rely on hierarchical syntactic features - as opposed to incorrect&#10;linear features - when performing tasks after fine-tuning. We test what aspects&#10;of pre-training are important for endowing encoder-decoder Transformers with an&#10;inductive bias that favors hierarchical syntactic generalizations. We focus on&#10;architectural features (depth, width, and number of parameters), as well as the&#10;genre and size of the pre-training corpus, diagnosing inductive biases using&#10;two syntactic transformation tasks: question formation and passivization, both&#10;in English. We find that the number of parameters alone does not explain&#10;hierarchical generalization: model depth plays greater role than model width.&#10;We also find that pre-training on simpler language, such as child-directed&#10;speech, induces a hierarchical bias using an order-of-magnitude less data than&#10;pre-training on more typical datasets based on web text or Wikipedia; this&#10;suggests that in cognitively plausible language acquisition settings, neural&#10;language models may be more data-efficient than previously thought.&#10;" />
          <attvalue for="2" value="&#10;Accurate syntactic representations are necessary for robust generalization to new natural language inputs and for the generation of correct outputs. Consider the problem of identifying the subject of ``said'' in the following sentence:&#10;&#10;\ex. Can {you} repeat what the {senator} next to the {cats} {said}?&#10;&#10;Typical language models (LMs), which receive linear sequences of words as input, could conceivably rely on a linear or positional feature that usually, but does not always, identifies the correct subject of a verb. An LM could learn, for example, that the first noun in the sentence is always the subject. This heuristic works for many simple sentences, but fails in Ex.~\ref{ex:heuristics}: here, the first noun is ``{you}'', and so this heuristic would lead the LM to incorrectly interpret the sentence as meaning ``Can you repeat what {you} said?'' The LM could also learn that the subject of the verb is the noun closest to the verb in the linear order of the sentence, in which case it would interpret Ex.~\ref{ex:heuristics} as ``Can you repeat what the {cats} said?'' By contrast, an LM that represents the sentence as hierarchically structured will correctly identify {senator} as the subject of the embedded clause that contains the verb {said}. This example demonstrates that a preference for syntactic features over linear features is required for robust linguistic generalization.&#10;&#10;The success of large-scale pre-training across fine-tuning tasks suggests that exposure to natural language may teach models to rely on appropriate syntactic features instead of heuristics (even though models still often rely on heuristics; \cite{mccoy-etal-2019-right}). This hypothesis is supported by the finding that, given minimal pairs of grammatical and ungrammatical sentences, the probability distribution over sentences defined by LMs often favors the grammatical sentence \cite{marvin-linzen-2018-targeted,hu-etal-2020-systematic}. A related line of work has shown that, through pre-training, LMs can under some circumstances acquire syntactic inductive biases which are then applied to fine-tuning tasks, whereas models which have not been pre-trained do not have such inductive biases (\cite{warstadt-bowman-2020-linguistic,warstadt-etal-2020-learning,lovering2021predicting,mueller-etal-2022-coloring}).&#10;&#10;When does pre-training endow LMs with a syntactic inductive bias? In this study, we address two specific sub-questions: (1) Which architectural features make a syntactic inductive bias more likely to emerge in a Transformer LM? (2) How is the inductive bias affected by the genre and size of the pre-training corpus? We investigate these questions by evaluating a range of Transformer encoder-decoder models based on T5 \cite{raffel-etal-2020-t5}. We evaluate both existing publicly available models and models that we pre-train ourselves; we explore a variety of model widths (embedding and hidden dimension, feed-forward layer size) and depths (number of layers), and pre-train on corpora of varying genres and sizes. We then evaluate models' inductive biases by observing their out-of-distribution generalization when fine-tuned on syntactic transformations tasks (\S\ref{sec:methods}). We find that depth matters more than width for the acquisition of hierarchical biases (\S\ref{sec:arch-effects}), and that pre-training on simpler language induces hierarchical biases using far less data (\S\ref{sec:style} and \S\ref{sec:data-quantity}). This last finding suggests that in language acquisition settings in which the training corpus more closely resembles the language that children are exposed to, Transformers may be more sample-efficient than previously thought.&#10;&#10;Our code is available on GitHub.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.11771" label="2210.11771">
        <attvalues>
          <attvalue for="0" value="InforMask: Unsupervised Informative Masking for Language Model&#10;  Pretraining" />
          <attvalue for="1" value="  Masked language modeling is widely used for pretraining large language models&#10;for natural language understanding (NLU). However, random masking is&#10;suboptimal, allocating an equal masking rate for all tokens. In this paper, we&#10;propose InforMask, a new unsupervised masking strategy for training masked&#10;language models. InforMask exploits Pointwise Mutual Information (PMI) to&#10;select the most informative tokens to mask. We further propose two&#10;optimizations for InforMask to improve its efficiency. With a one-off&#10;preprocessing step, InforMask outperforms random masking and previously&#10;proposed masking strategies on the factual recall benchmark LAMA and the&#10;question answering benchmark SQuAD v1 and v2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.09733" label="2004.09733">
        <attvalues>
          <attvalue for="0" value="Train No Evil: Selective Masking for Task-Guided Pre-Training" />
          <attvalue for="1" value="  Recently, pre-trained language models mostly follow the&#10;pre-train-then-fine-tuning paradigm and have achieved great performance on&#10;various downstream tasks. However, since the pre-training stage is typically&#10;task-agnostic and the fine-tuning stage usually suffers from insufficient&#10;supervised data, the models cannot always well capture the domain-specific and&#10;task-specific patterns. In this paper, we propose a three-stage framework by&#10;adding a task-guided pre-training stage with selective masking between general&#10;pre-training and fine-tuning. In this stage, the model is trained by masked&#10;language modeling on in-domain unsupervised data to learn domain-specific&#10;patterns and we propose a novel selective masking strategy to learn&#10;task-specific patterns. Specifically, we design a method to measure the&#10;importance of each token in sequences and selectively mask the important&#10;tokens. Experimental results on two sentiment analysis tasks show that our&#10;method can achieve comparable or even better performance with less than 50% of&#10;computation cost, which indicates our method is both effective and efficient.&#10;The source code of this paper can be obtained from&#10;https://github.com/thunlp/SelectiveMasking.&#10;" />
          <attvalue for="2" value="&#10;&#10;&#10;&#10;Pre-trained Language Models (PLMs) have achieved superior performances on various NLP tasks~\cite{baevski2019cloze,joshi2019spanbert,liu2019roberta,yang2019xlnet,Clark2020ELECTRA:} and have attracted wide research interests. Inspired by the success of GPT~\cite{radford2018improving} and BERT~\cite{devlin2018bert}, most PLMs follow the pre-train-then-fine-tuning paradigm, which adopts unsupervised pre-training on large general-domain corpora to learn general language patterns and supervised fine-tuning to adapt to downstream tasks. &#10;&#10;Recently, \cite{gururangan2020dont} shows that learning domain-specific and task-specific patterns during pre-training can be helpful to the models for certain domains and tasks. However, conventional pre-training is aimless with respect to specific downstream tasks, and fine-tuning usually suffers from insufficient supervised data, preventing PLMs from effectively capturing these patterns. &#10;&#10;To learn domain-specific language patterns, some previous works~\cite{Beltagy2019SciBERT,huang2019clinicalbert} pre-train a BERT-like model from scratch using large-scale in-domain data. However, they are computation-intensive and require large-scale in-domain data, which is hard to obtain in many domains. To learn task-specific &#10;language patterns, some previous works~\cite{phang2018sentence} add intermediate supervised pre-training after general pre-training, whose pre-training task is similar to the downstream task but has a larger dataset. However, \cite{wang2019can} shows that this kind of intermediate pre-training often negatively impacts the transferability to downstream tasks.&#10;&#10;To better capture domain-specific and task-specific patterns, we propose a three-stage framework by adding a task-guided pre-training stage with selective masking between the general pre-training and fine-tuning. The overall framework is shown in Figure~\ref{fig:setting}. In the stage of task-guided pre-training, the model is trained by masked language modeling (Masked LM)~\cite{devlin2018bert} on mid-scale in-domain unsupervised data, which is constructed by collecting other corpora in the same domain. &#10;In this way, PLMs can utilize more data to better learn domain-specific language patterns~\cite{alsentzer-etal-2019-publicly,Lee2019BioBERT,sung-etal-2019-pre,xu-etal-2019-doubletransfer, aharoni2020unsupervised}. &#10;However, the conventional Masked LM randomly masks tokens, which is inefficient to learn task-specific language patterns. Hence, we propose a selective masking strategy for task-guided pre-training, whose main idea is selectively masking the important tokens for downstream tasks.&#10;&#10;Intuitively, some tokens are more important than others for a specific task and the important tokens vary among different tasks~\cite{ziser-reichart-2018-pivot, feng-etal-2018-pathologies, rietzler2020adapt}. For instance, in sentiment analysis, sentiment tokens such as ``like'' and ``hate'' are critical for sentiments classification~\cite{ke2019sentilr}. And, in relation extraction, predicates and verbs are typically more significant. If PLMs can selectively mask and predict the important tokens instead of a mass of random tokens, they can effectively learn task-specific language patterns and the computation cost of pre-training can be significantly reduced.&#10;&#10;For the selective masking strategy, we propose a simple method to find important tokens for downstream tasks. Specifically, we define a task-specific score for each token and if the score is lower than a certain threshold, we regard the token as important.&#10;However, this method relies on the supervised downstream datasets whose sizes are limited for pre-training. To better utilize mid-scale in-domain unsupervised data as shown in Figure \ref{fig:setting}, we train a neural network on downstream datasets where the important tokens are annotated using the method mentioned above. This neural network can learn the implicit token-selecting rules, which enables us to select tokens without supervision.&#10;&#10;We conduct experiments on two sentiment analysis tasks: MR~\cite{pang2005seeing} and SemEval14 task 4~\cite{pontiki-etal-2014-semeval}. Experimental results show that our method is both efficient and effective. Our method can achieve comparable and even better performances than the conventional pre-train-then-fine-tune method with less than 50\% of the overall computation cost. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.06546" label="2107.06546">
        <attvalues>
          <attvalue for="0" value="ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language&#10;  Modelling track, 2021 edition" />
          <attvalue for="1" value="  We present the visually-grounded language modelling track that was introduced&#10;in the Zero-Resource Speech challenge, 2021 edition, 2nd round. We motivate the&#10;new track and discuss participation rules in detail. We also present the two&#10;baseline systems that were developed for this track.&#10;" />
          <attvalue for="2" value="&#10;This document introduces the visually-grounded spoken language modeling track of the ZeroSpeech 2021 challenge. In this track, participants are asked to use audiovisual materials during training. Evaluation is identical to the speech-only track at ZeroSpeech 2021. &#10;&#10;Learning to comprehend and produce spoken languages is one of the hallmarks of human cognition, and the importance of speech communication also makes speech-based capabilities central to AI development. Modern automatic speech recognition (ASR) systems largely rely on supervised training, where input speech is paired with corresponding phonetic annotations or text transcripts. While this approach has produced great results in high-resource languages such as English, deployment of similar systems for low-resource environments such as small language communities or even unwritten languages is difficult. In addition, a mismatch between conversational speech and large-scale text data still exists even in high-resource languages. &#10;&#10;In contrast to ASR, human children achieve their language skills without direct supervision or detailed feedback, simply interacting with their physical and linguistic environments. Moreover, these experiences are essentially multimodal: children not only hear speech of their caregivers, but concurrently observe the world through a number of senses. Instead of perceiving speech and the corresponding text, they learn from speech in the context of different everyday multimodal communicative scenarios. &#10; Compared with audio data only, audiovisual data contain more statistical regularities at multiple levels: there are regularities within audio and visual data respectively; and there are also cross-modal regularities at both the acoustic and semantic levels. For example, the presence of a dog in a visual scene is likely to co-occur with both barking sounds at the acoustic level and the spoken word “dog” (and other words semantically related to dog) at the semantic level. The presence of a dog can be used as a supervisory signal to train language learning systems to parse and segment words that consistently occur in the same context, and semantically group words that occur in similar visual contexts. &#10; In order to develop AI systems or models of human learning with similar multimodal language learning skills, a number of models and learning algorithms have been proposed throughout the years (e.g., \cite{roy2002learning,Yu_Ballard_2004,tenbosch_2009,Driesen_2011,Rasanen_Rasilo_2015, Mangin_2015}). These systems have used various types of speech data with simulated or robot vision-based visual input. &#10;&#10;However, only the recent advances in deep learning have scaled up the capabilities of audiovisual systems to a level where they can start to capture the relationships between realistic visual data (e.g., photographs or videos) and language related to the visual scene. These models were first developed for captions descriptive of images \cite{socher2014grounded,karpathy2015deep} and more recently for spoken descriptions of the images (e.g., \cite{harwath2015deep,harwath2016unsupervised,synnaeve2014learning,chrupala2017representations,kamper2019semantic}). In this context, it is of great interest how the representations emerging from training of such multimodal models relate to the known linguistic structure of the input language (e.g., \cite{chrupala2017representations,alishahi2017encoding,harwath2019learning,havard2019models,havard2019word}), and how such methods can support (or replace) purely audio-based representation learning approaches (e.g., \cite{Chung2019a,oord2018cpc}). In other words, it would be highly useful if unsupervised learning from multimodal data could be used to acquire language representations such as phone(me)s and words without access to transcribed training data in the given language---units that can be then used as a basis for many other language processing tasks. However, the research in this direction is still young and largely driven by a few research groups \cite{chrupala2021visually}. In addition, there are no standardized evaluation metrics or a common benchmark to compare different methodological approaches and thereby to drive the research in this area forward. &#10;&#10;The goal of the ZR-2021VG track is to tackle the issue of multimodal language learning. In contrast to the earlier Zerospeech-challenges \cite{versteegh2015zero,dunbar2017zero,dunbar2019zero,nguyen2020zero} that have purely focused on audio-based learning of linguistic representations (including the speech-based track of the current challenge, on which we build), ZR-2021VG takes a step towards multimodal language learning by asking participants to train on audiovisual data. The aim of the challenge is to learn phonemic, lexical, syntactic, and semantic representations of speech with the help of supporting visual information, as evaluated by standardized evaluation protocols. As a result, the challenge aims to bring together researchers from speech technology, natural language processing, computer vision, and machine learning to work on multimodal language learning, thereby advancing the state-of-the-art in audiovisual learning algorithms and providing new knowledge on how visual information may support unsupervised learning of linguistic patterns from speech.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Modelling, Computer Science, Speech Challenge, Linguistics, Cognitive Science, Artificial Intelligence, Baseline Systems" />
        </attvalues>
      </node>
      <node id="1412.2306" label="1412.2306">
        <attvalues>
          <attvalue for="0" value="Deep Visual-Semantic Alignments for Generating Image Descriptions" />
          <attvalue for="1" value="  We present a model that generates natural language descriptions of images and&#10;their regions. Our approach leverages datasets of images and their sentence&#10;descriptions to learn about the inter-modal correspondences between language&#10;and visual data. Our alignment model is based on a novel combination of&#10;Convolutional Neural Networks over image regions, bidirectional Recurrent&#10;Neural Networks over sentences, and a structured objective that aligns the two&#10;modalities through a multimodal embedding. We then describe a Multimodal&#10;Recurrent Neural Network architecture that uses the inferred alignments to&#10;learn to generate novel descriptions of image regions. We demonstrate that our&#10;alignment model produces state of the art results in retrieval experiments on&#10;Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated&#10;descriptions significantly outperform retrieval baselines on both full images&#10;and on a new dataset of region-level annotations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1511.03690" label="1511.03690">
        <attvalues>
          <attvalue for="0" value="Deep Multimodal Semantic Embeddings for Speech and Images" />
          <attvalue for="1" value="  In this paper, we present a model which takes as input a corpus of images&#10;with relevant spoken captions and finds a correspondence between the two&#10;modalities. We employ a pair of convolutional neural networks to model visual&#10;objects and speech signals at the word level, and tie the networks together&#10;with an embedding and alignment model which learns a joint semantic space over&#10;both modalities. We evaluate our model using image search and annotation tasks&#10;on the Flickr8k dataset, which we augmented by collecting a corpus of 40,000&#10;spoken captions using Amazon Mechanical Turk.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1702.01991" label="1702.01991">
        <attvalues>
          <attvalue for="0" value="Representations of language in a model of visually grounded speech&#10;  signal" />
          <attvalue for="1" value="  We present a visually grounded model of speech perception which projects&#10;spoken utterances and images to a joint semantic space. We use a multi-layer&#10;recurrent highway network to model the temporal nature of spoken speech, and&#10;show that it learns to extract both form and meaning-based linguistic knowledge&#10;from the input signal. We carry out an in-depth analysis of the representations&#10;used by different components of the trained model and show that encoding of&#10;semantic aspects tends to become richer as we go up the hierarchy of layers,&#10;whereas encoding of form-related aspects of the language input tends to&#10;initially increase and then plateau or decrease.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.03815" label="1706.03815">
        <attvalues>
          <attvalue for="0" value="Encoding of phonology in a recurrent neural model of grounded speech" />
          <attvalue for="1" value="  We study the representation and encoding of phonemes in a recurrent neural&#10;network model of grounded speech. We use a model which processes images and&#10;their spoken descriptions, and projects the visual and auditory representations&#10;into the same semantic space. We perform a number of analyses on how&#10;information about individual phonemes is encoded in the MFCC features extracted&#10;from the speech signal, and the activations of the layers of the model. Via&#10;experiments with phoneme decoding and phoneme discrimination we show that&#10;phoneme representations are most salient in the lower layers of the model,&#10;where low-level signals are processed at a fine-grained level, although a large&#10;amount of phonological information is retain at the top recurrent layer. We&#10;further find out that the attention mechanism following the top recurrent layer&#10;significantly attenuates encoding of phonology and makes the utterance&#10;embeddings much more invariant to synonymy. Moreover, a hierarchical clustering&#10;of phoneme representations learned by the network shows an organizational&#10;structure of phonemes similar to those proposed in linguistics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.09602" label="1911.09602">
        <attvalues>
          <attvalue for="0" value="Learning Hierarchical Discrete Linguistic Units from Visually-Grounded&#10;  Speech" />
          <attvalue for="1" value="  In this paper, we present a method for learning discrete linguistic units by&#10;incorporating vector quantization layers into neural models of visually&#10;grounded speech. We show that our method is capable of capturing both&#10;word-level and sub-word units, depending on how it is configured. What&#10;differentiates this paper from prior work on speech unit learning is the choice&#10;of training objective. Rather than using a reconstruction-based loss, we use a&#10;discriminative, multimodal grounding objective which forces the learned units&#10;to be useful for semantic image retrieval. We evaluate the sub-word units on&#10;the ZeroSpeech 2019 challenge, achieving a 27.3\% reduction in ABX error rate&#10;over the top-performing submission, while keeping the bitrate approximately the&#10;same. We also present experiments demonstrating the noise robustness of these&#10;units. Finally, we show that a model with multiple quantizers can&#10;simultaneously learn phone-like detectors at a lower layer and word-like&#10;detectors at a higher layer. We show that these detectors are highly accurate,&#10;discovering 279 words with an F1 score of greater than 0.5.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.03052" label="1902.03052">
        <attvalues>
          <attvalue for="0" value="Models of Visually Grounded Speech Signal Pay Attention To Nouns: a&#10;  Bilingual Experiment on English and Japanese" />
          <attvalue for="1" value="  We investigate the behaviour of attention in neural models of visually&#10;grounded speech trained on two languages: English and Japanese. Experimental&#10;results show that attention focuses on nouns and this behaviour holds true for&#10;two very typologically different languages. We also draw parallels between&#10;artificial neural attention and human attention and show that neural attention&#10;focuses on word endings as it has been theorised for human attention. Finally,&#10;we investigate how two visually grounded monolingual models can be used to&#10;perform cross-lingual speech-to-speech retrieval. For both languages, the&#10;enriched bilingual (speech-image) corpora with part-of-speech tags and forced&#10;alignments are distributed to the community for reproducible research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.08491" label="1909.08491">
        <attvalues>
          <attvalue for="0" value="Word Recognition, Competition, and Activation in a Model of Visually&#10;  Grounded Speech" />
          <attvalue for="1" value="  In this paper, we study how word-like units are represented and activated in&#10;a recurrent neural model of visually grounded speech. The model used in our&#10;experiments is trained to project an image and its spoken description in a&#10;common representation space. We show that a recurrent model trained on spoken&#10;sentences implicitly segments its input into word-like units and reliably maps&#10;them to their correct visual referents. We introduce a methodology originating&#10;from linguistics to analyse the representation learned by neural networks --&#10;the gating paradigm -- and show that the correct representation of a word is&#10;only activated if the network has access to first phoneme of the target word,&#10;suggesting that the network does not rely on a global acoustic pattern.&#10;Furthermore, we find out that not all speech frames (MFCC vectors in our case)&#10;play an equal role in the final encoded representation of a given word, but&#10;that some frames have a crucial effect on it. Finally, we suggest that word&#10;representation could be activated through a process of lexical competition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.13225" label="2104.13225">
        <attvalues>
          <attvalue for="0" value="Visually grounded models of spoken language: A survey of datasets,&#10;  architectures and evaluation techniques" />
          <attvalue for="1" value="  This survey provides an overview of the evolution of visually grounded models&#10;of spoken language over the last 20 years. Such models are inspired by the&#10;observation that when children pick up a language, they rely on a wide range of&#10;indirect and noisy clues, crucially including signals from the visual modality&#10;co-occurring with spoken utterances. Several fields have made important&#10;contributions to this approach to modeling or mimicking the process of learning&#10;language: Machine Learning, Natural Language and Speech Processing, Computer&#10;Vision and Cognitive Science. The current paper brings together these&#10;contributions in order to provide a useful introduction and overview for&#10;practitioners in all these areas. We discuss the central research questions&#10;addressed, the timeline of developments, and the datasets which enabled much of&#10;this work. We then summarize the main modeling architectures and offer an&#10;exhaustive overview of the evaluation metrics and analysis techniques.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.04313" label="1712.04313">
        <attvalues>
          <attvalue for="0" value="The Zero Resource Speech Challenge 2017" />
          <attvalue for="1" value="  We describe a new challenge aimed at discovering subword and word units from&#10;raw speech. This challenge is the followup to the Zero Resource Speech&#10;Challenge 2015. It aims at constructing systems that generalize across&#10;languages and adapt to new speakers. The design features and evaluation metrics&#10;of the challenge are presented and the results of seventeen models are&#10;discussed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.07779" label="2204.07779">
        <attvalues>
          <attvalue for="0" value="SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words" />
          <attvalue for="1" value="  Pre-trained models are widely used in the tasks of natural language&#10;processing nowadays. However, in the specific field of text simplification, the&#10;research on improving pre-trained models is still blank. In this work, we&#10;propose a continued pre-training method for text simplification. Specifically,&#10;we propose a new masked language modeling (MLM) mechanism, which does not&#10;randomly mask words but only masks simple words. The new mechanism can make the&#10;model learn to generate simple words. We use a small-scale simple text dataset&#10;for continued pre-training and employ two methods to identify simple words from&#10;the texts. We choose BERT, a representative pre-trained model, and continue&#10;pre-training it using our proposed method. Finally, we obtain SimpleBERT, which&#10;surpasses BERT in both lexical simplification and sentence simplification tasks&#10;and has achieved state-of-the-art results on multiple datasets. What's more,&#10;SimpleBERT can replace BERT in existing simplification models without&#10;modification.&#10;" />
          <attvalue for="2" value="&#10;&#10;The goal of text simplification is to reduce complex text to a more comprehensible text while keeping its meaning intact \cite{alva2020data, sikka2020survey}. This technology can provide convenience for children, non-native speakers, and people with dyslexia \cite{shardlow2014survey}. &#10;&#10;In recent years, a small number of studies have applied pre-trained models to the field of text simplification and achieved good results. \cite{maruyama2019extremely} \shortcite{maruyama2019extremely} used TransformerLM to help extremely low-resource sentence simplification.&#10;\cite{qiang2020lexical} \shortcite{qiang2020lexical} proposed BERT-LS to generate simple substitutions for lexical simplification. \cite{martin2020multilingual} \shortcite{martin2020multilingual} proposed unsupervised mining techniques and combined them with pre-trained models. \cite{jiang2020neural} \shortcite{jiang2020neural} initialized the encoder with BERT and proposed a seq2seq model. &#10;&#10;However, simply applying pre-trained models to text simplification may be problematic. In the pre-training stage, the models are trained in a self-supervised way with a large number of ordinary texts, and the models are expected to learn to generate ordinary words instead of simple words. &#10;Besides, some tasks such as lexical simplification do not have large-scale training data. The effect of fine-tuning may not be good as expected. Therefore, it is worth investigating how to make pre-trained models learn to generate simple words.&#10;&#10;In this work, we are committed to improving the pre-trained model like BERT \cite{devlin2019bert} to help both lexical and sentence simplification. We are inspired by the works of \cite{gururangan2020don} \shortcite{gururangan2020don} and&#10;\cite{gu2020train} \shortcite{gu2020train}. They conclude that a task-guided pre-training stage using the task-related data may be helpful for downstream tasks.&#10;&#10;Specifically, we collect a set of simple texts and then employ two methods to automatically and coarsely identify simple words in texts. We propose a new masked language modeling (MLM) mechanism which only masks the simple words in the texts and then let the model learn to predict the masked simple words. We choose a representative pre-trained model, BERT, and continue pre-training it using the method proposed above. Thus, we obtain a new pre-trained model for text simplification. We name it SimpleBERT. We experiment with SimpleBERT on several tasks of text simplification to demonstrate its strong performance and generality. &#10;&#10;In summary, the main contributions of our work are that (1) We propose SimpleBERT with a new MLM mechanism as the backbone for text simplification tasks. SimpleBERT can replace BERT in existing simplification models without any modification. (2) Our proposed method achieves state-of-the-art results on multiple datasets in text simplification, and it outperforms the original BERT on all datasets. (3) To our best knowledge, we are the first to explore how to improve pre-trained models in text simplification. We will release the code and model to promote future research when the paper is accepted.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Text Simplification, Artificial Intelligence, Pre-trained Models, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2406.17789" label="2406.17789">
        <attvalues>
          <attvalue for="0" value="Spanish and LLM Benchmarks: is MMLU Lost in Translation?" />
          <attvalue for="1" value="  The evaluation of Large Language Models (LLMs) is a key element in their&#10;continuous improvement process and many benchmarks have been developed to&#10;assess the performance of LLMs in different tasks and topics. As LLMs become&#10;adopted worldwide, evaluating them in languages other than English is&#10;increasingly important. However, most LLM benchmarks are simply translated&#10;using an automated tool and then run in the target language. This means that&#10;the results depend not only on the LLM performance in that language but also on&#10;the quality of the translation. In this paper, we consider the case of the&#10;well-known Massive Multitask Language Understanding (MMLU) benchmark. Selected&#10;categories of the benchmark are translated into Spanish using Azure Translator&#10;and ChatGPT4 and run on ChatGPT4. Next, the results are processed to identify&#10;the test items that produce different answers in Spanish and English. Those are&#10;then analyzed manually to understand if the automatic translation caused the&#10;change. The results show that a significant fraction of the failing items can&#10;be attributed to mistakes in the translation of the benchmark. These results&#10;make a strong case for improving benchmarks in languages other than English by&#10;at least revising the translations of the items and preferably by adapting the&#10;tests to the target language by experts.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models are becoming a fundamental block in modern computing systems enabling new applications and facilitating the interaction with users \cite{LLM_survey}. However, LLMs have limitations and their performance has to be well understood before using them on a given application \cite{LLM_survey2}. This has motivated the development of a large number of LLM evaluation benchmarks that test the knowledge that models have of many different topics and how well they can perform tasks such as logic reasoning or problem-solving \cite{LLM_evaluation_survey}. Most of these benchmarks are designed so that the LLM responses can be processed automatically thus enabling testing at scale with thousands of questions. This is commonly achieved by using multiple-choice tests. &#10;&#10;There are LLM benchmarks to evaluate a wide range of tasks and topics. For example, there are tests to evaluate the capabilities of LLMs to solve common sense reasoning problems \cite{zellers2019hellaswag} or to answer mathematical questions \cite{Mathmeasuring}. To provide a more comprehensive evaluation, some benchmarks evaluate several tasks, for example, the Multitask Language Understanding (MMLU) test \cite{MMLU} evaluates 57 different topics and other benchmarks increase the number of tasks and topics to more than 200 \cite{BIGMeasuring}. The speed and energy dissipation of LLMs are also important factors that are commonly evaluated in terms of the number of tokens generated per second, the memory used or the energy per token, or with more user-centric metrics like the time and energy needed to complete a given task \cite{SpeedLLM}. As LLMs become pervasive and used in almost any domain and application, more benchmarks will be developed each having possibly thousands or even hundreds of thousands of questions. &#10;&#10;Another dimension of LLM evaluation is their performance in languages other than English as in fact most users are native speakers of other languages. Most benchmarks are written in English with questions taken in many cases from different exams, such as university, high school or professional tests. The simplest approach is to translate these same tests into other languages and use them for multilingual evaluation. This clearly introduces a cultural bias, especially when questions are related to subjects such as history, geography, art or general culture. Ideally, specific tests should be developed or at least adapted for each language.&#10;&#10;However, this is not the only problem. To be able to evaluate LLMs in many languages, and given the large number of questions of the benchmarks, the standard procedure is to translate the English test to the target language using automatic translation tools, for example, in the Okapi project \cite{lai2023okapi} three benchmarks from the Open LLM Leaderboard \cite{open-llm-leaderboard} are translated using ChatGPT while in the evaluation of GPT4, the tests were translated using Azure Translator \cite{GPT4}. This implies that the benchmarks in languages other than English are not only measuring the performance of the LLM but also of the translation tool as the quality of the translation can clearly impact the results. &#10;&#10;In this work we perform an initial analysis of the impact of automatic translation on one of the most widely used LLM benchmarks, the Multitask Language Understanding (MMLU) test \cite{MMLU} for one of the most commonly used and chosen as a second language to learn, Spanish. The analysis shows that automatic translation induces errors in the LLM answers and thus distorts the benchmark's results. Based on these findings, potential solutions to this problem are also briefly discussed.&#10;&#10;The rest of the work is organized as follows, in section\ref{sec:methodology} the methodology used in our analysis is presented, followed by the results in section\ref{sec:ResultsAnalysis} and a discussion of their implications and potential solutions in section\ref{sec:discussion}. The paper ends with the conclusion in section\ref{sec:Conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Multilingual Benchmarking, Translation Quality Assessment, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2307.16039" label="2307.16039">
        <attvalues>
          <attvalue for="0" value="Okapi: Instruction-tuned Large Language Models in Multiple Languages&#10;  with Reinforcement Learning from Human Feedback" />
          <attvalue for="1" value="  A key technology for the development of large language models (LLMs) involves&#10;instruction tuning that helps align the models' responses with human&#10;expectations to realize impressive learning abilities. Two major approaches for&#10;instruction tuning characterize supervised fine-tuning (SFT) and reinforcement&#10;learning from human feedback (RLHF), which are currently applied to produce the&#10;best commercial LLMs (e.g., ChatGPT). To improve the accessibility of LLMs for&#10;research and development efforts, various instruction-tuned open-source LLMs&#10;have also been introduced recently, e.g., Alpaca, Vicuna, to name a few.&#10;However, existing open-source LLMs have only been instruction-tuned for English&#10;and a few popular languages, thus hindering their impacts and accessibility to&#10;many other languages in the world. Among a few very recent work to explore&#10;instruction tuning for LLMs in multiple languages, SFT has been used as the&#10;only approach to instruction-tune LLMs for multiple languages. This has left a&#10;significant gap for fine-tuned LLMs based on RLHF in diverse languages and&#10;raised important questions on how RLHF can boost the performance of&#10;multilingual instruction tuning. To overcome this issue, we present Okapi, the&#10;first system with instruction-tuned LLMs based on RLHF for multiple languages.&#10;Okapi introduces instruction and response-ranked data in 26 diverse languages&#10;to facilitate the experiments and development of future multilingual LLM&#10;research. We also present benchmark datasets to enable the evaluation of&#10;generative LLMs in multiple languages. Our experiments demonstrate the&#10;advantages of RLHF for multilingual instruction over SFT for different base&#10;models and datasets. Our framework and resources are released at&#10;https://github.com/nlp-uoregon/Okapi.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.09384" label="1808.09384">
        <attvalues>
          <attvalue for="0" value="What Makes Reading Comprehension Questions Easier?" />
          <attvalue for="1" value="  A challenge in creating a dataset for machine reading comprehension (MRC) is&#10;to collect questions that require a sophisticated understanding of language to&#10;answer beyond using superficial cues. In this work, we investigate what makes&#10;questions easier across recent 12 MRC datasets with three question styles&#10;(answer extraction, description, and multiple choice). We propose to employ&#10;simple heuristics to split each dataset into easy and hard subsets and examine&#10;the performance of two baseline models for each of the subsets. We then&#10;manually annotate questions sampled from each subset with both validity and&#10;requisite reasoning skills to investigate which skills explain the difference&#10;between easy and hard questions. From this study, we observed that (i) the&#10;baseline performances for the hard subsets remarkably degrade compared to those&#10;of entire datasets, (ii) hard questions require knowledge inference and&#10;multiple-sentence reasoning in comparison with easy questions, and (iii)&#10;multiple-choice questions tend to require a broader range of reasoning skills&#10;than answer extraction and description questions. These results suggest that&#10;one might overestimate recent advances in MRC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.10386" label="2105.10386">
        <attvalues>
          <attvalue for="0" value="Analysis of Boolean Functions" />
          <attvalue for="1" value="  The subject of this textbook is the analysis of Boolean functions. Roughly&#10;speaking, this refers to studying Boolean functions $f : \{0,1\}^n \to \{0,1\}$&#10;via their Fourier expansion and other analytic means. Boolean functions are&#10;perhaps the most basic object of study in theoretical computer science, and&#10;Fourier analysis has become an indispensable tool in the field. The topic has&#10;also played a key role in several other areas of mathematics, from&#10;combinatorics, random graph theory, and statistical physics, to Gaussian&#10;geometry, metric/Banach spaces, and social choice theory.&#10;  The intent of this book is both to develop the foundations of the field and&#10;to give a wide (though far from exhaustive) overview of its applications. Each&#10;chapter ends with a &quot;highlight&quot; showing the power of analysis of Boolean&#10;functions in different subject areas: property testing, social choice,&#10;cryptography, circuit complexity, learning theory, pseudorandomness, hardness&#10;of approximation, concrete complexity, and random graph theory.&#10;  The book can be used as a reference for working researchers or as the basis&#10;of a one-semester graduate-level course. The author has twice taught such a&#10;course at Carnegie Mellon University, attended mainly by graduate students in&#10;computer science and mathematics but also by advanced undergraduates, postdocs,&#10;and researchers in adjacent fields. In both years most of Chapters 1-5 and 7&#10;were covered, along with parts of Chapters 6, 8, 9, and 11, and some additional&#10;material on additive combinatorics. Nearly 500 exercises are provided at the&#10;ends of the book's chapters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07780" label="2109.07780">
        <attvalues>
          <attvalue for="0" value="Improving Neural Machine Translation by Bidirectional Training" />
          <attvalue for="1" value="  We present a simple and effective pretraining strategy -- bidirectional&#10;training (BiT) for neural machine translation. Specifically, we bidirectionally&#10;update the model parameters at the early stage and then tune the model&#10;normally. To achieve bidirectional updating, we simply reconstruct the training&#10;samples from &quot;src$\rightarrow$tgt&quot; to &quot;src+tgt$\rightarrow$tgt+src&quot; without any&#10;complicated model modifications. Notably, our approach does not increase any&#10;parameters or training steps, requiring the parallel data merely. Experimental&#10;results show that BiT pushes the SOTA neural machine translation performance&#10;across 15 translation tasks on 8 language pairs (data sizes range from 160K to&#10;38M) significantly higher. Encouragingly, our proposed model can complement&#10;existing data manipulation strategies, i.e. back translation, data&#10;distillation, and data diversification. Extensive analyses show that our&#10;approach functions as a novel bilingual code-switcher, obtaining better&#10;bilingual alignment.&#10;" />
          <attvalue for="2" value="&#10;Recent years have seen a surge of interest in neural machine translation (NMT,~\cite{luong2015effective,wu2016google,gehring2017convolutional,transformer}) where it benefits from a massive amount of training data. But obtaining such large amounts of parallel data is not-trivial in most machine translation scenarios. For example, there are many low-resource language pairs (e.g. English-to-Tamil), which lack adequate parallel data for training. &#10;&#10;Although many approaches about fully exploiting the parallel and monolingual data are proposed, e.g. back translation~\cite{sennrich-etal-2016-improving}, knowledge distillation~\cite{kim-rush-2016-sequence} and data diversification~\cite{nguyen2019data}, the prerequisite of these approaches is to build a well-performed baseline model based on the parallel data. However, \newcite{koehn2017six,lample2018phrase,sennrich2019revisiting} empirically reveal that NMT runs worse than their statistical or even unsupervised counterparts in low-resource conditions. Here naturally arise a question: Can we find a strategy to consistently improve NMT performance given the parallel data merely?&#10;&#10;We decide to find a solution from human learning behavior.~\newcite{pavlenko2002bidirectional,dworin2003insights,chen2015bilingual} show that bidirectional language learning helps master bilingualism. In the context of machine translation, both the source$arrow$target and target$arrow$source language mappings may benefit bilingual modeling, which motivates many recent studies, e.g. dual learning~\cite{he2016dual} and symmetric training~\cite{cohn2016incorporating,Liang2007AgreementBasedL}. However, their approaches rely on external resources (e.g. word alignment or monolingual data) or complicated model modifications, which limit the applicability of the method to a broader range of languages and model structures. Accordingly, we turn to propose a simple data manipulation strategy and transfer the bidirectional relationship through bidirectional training (\S\ref{subsec:bidirectional}). The core idea is using a bidirectional system as an initialization for a unidirectional system.&#10;Specifically, to make the most of the parallel data, we first reconstruct the training samples from ``$\overrightarrow{B}$: source$arrow$target'' to ``$\overleftrightarrow{B}$: source$+$target$arrow$target$+$source'', where the training data was doubled. Then we update the model parameters with $\overleftrightarrow{B}$ in the early stage, and tune the model with normal ``$\overrightarrow{B}$ source$arrow$target'' direction.&#10;&#10;We validated our approach on several benchmarks across different language families and data sizes, including IWSLT21 En$rightarrow$De, WMT16 En$rightarrow$Ro, &#10;WMT19 En$rightarrow$Gu,&#10;IWSLT21 En$rightarrow$Sw, WMT14 En$rightarrow$De, WMT19 En$rightarrow$De, WMT17 Zh$rightarrow$En and WAT17 Ja$rightarrow$En. Experimental results show that the proposed bidirectional training (BiT) consistently and significantly improves the translation performance over the strong Transformer~\cite{transformer}. &#10;Also, we show that BiT can complement existing data manipulation strategies, i.e. back translation, knowledge distillation and data diversification. Extensive analyses in \S\ref{subsec:analysis} confirm that the performance improvement indeed comes from the better cross-lingual modeling and our method works like a novel code-switching method.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Data Science, Pretraining Strategies, Mathematics, Neural Machine Translation, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1911.01986" label="1911.01986">
        <attvalues>
          <attvalue for="0" value="Data Diversification: A Simple Strategy For Neural Machine Translation" />
          <attvalue for="1" value="  We introduce Data Diversification: a simple but effective strategy to boost&#10;neural machine translation (NMT) performance. It diversifies the training data&#10;by using the predictions of multiple forward and backward models and then&#10;merging them with the original dataset on which the final NMT model is trained.&#10;Our method is applicable to all NMT models. It does not require extra&#10;monolingual data like back-translation, nor does it add more computations and&#10;parameters like ensembles of models. Our method achieves state-of-the-art BLEU&#10;scores of 30.7 and 43.7 in the WMT'14 English-German and English-French&#10;translation tasks, respectively. It also substantially improves on 8 other&#10;translation tasks: 4 IWSLT tasks (English-German and English-French) and 4&#10;low-resource translation tasks (English-Nepali and English-Sinhala). We&#10;demonstrate that our method is more effective than knowledge distillation and&#10;dual learning, it exhibits strong correlation with ensembles of models, and it&#10;trades perplexity off for better BLEU score. We have released our source code&#10;at https://github.com/nxphi47/data_diversification&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1601.01085" label="1601.01085">
        <attvalues>
          <attvalue for="0" value="Incorporating Structural Alignment Biases into an Attentional Neural&#10;  Translation Model" />
          <attvalue for="1" value="  Neural encoder-decoder models of machine translation have achieved impressive&#10;results, rivalling traditional translation models. However their modelling&#10;formulation is overly simplistic, and omits several key inductive biases built&#10;into traditional models. In this paper we extend the attentional neural&#10;translation model to include structural biases from word based alignment&#10;models, including positional bias, Markov conditioning, fertility and agreement&#10;over translation directions. We show improvements over a baseline attentional&#10;model and standard phrase-based model over several language pairs, evaluating&#10;on difficult languages in a low resource setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.01258" label="1806.01258">
        <attvalues>
          <attvalue for="0" value="Agreement-based Learning" />
          <attvalue for="1" value="  Model selection is a problem that has occupied machine learning researchers&#10;for a long time. Recently, its importance has become evident through&#10;applications in deep learning. We propose an agreement-based learning framework&#10;that prevents many of the pitfalls associated with model selection. It relies&#10;on coupling the training of multiple models by encouraging them to agree on&#10;their predictions while training. In contrast with other model selection and&#10;combination approaches used in machine learning, the proposed framework is&#10;inspired by human learning. We also propose a learning algorithm defined within&#10;this framework which manages to significantly outperform alternatives in&#10;practice, and whose performance improves further with the availability of&#10;unlabeled data. Finally, we describe a number of potential directions for&#10;developing more flexible agreement-based learning algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.05408" label="1908.05408">
        <attvalues>
          <attvalue for="0" value="Towards End-to-End Learning for Efficient Dialogue Agent by Modeling&#10;  Looking-ahead Ability" />
          <attvalue for="1" value="  Learning an efficient manager of dialogue agent from data with little manual&#10;intervention is important, especially for goal-oriented dialogues. However,&#10;existing methods either take too many manual efforts (e.g. reinforcement&#10;learning methods) or cannot guarantee the dialogue efficiency (e.g.&#10;sequence-to-sequence methods). In this paper, we address this problem by&#10;proposing a novel end-to-end learning model to train a dialogue agent that can&#10;look ahead for several future turns and generate an optimal response to make&#10;the dialogue efficient. Our method is data-driven and does not require too much&#10;manual work for intervention during system design. We evaluate our method on&#10;two datasets of different scenarios and the experimental results demonstrate&#10;the efficiency of our model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In most situations, the dialogue systems require handcrafted definition of dialogue states and dialogue policies~\cite{pomdp,challenge,neg2,jd}. Those methods make the pipeline of dialogue systems clear to design and easy to maintain, but suffer from the massive expensive human efforts and the error propagation issue~\cite{errorpro1,errorpro2}.&#10;&#10;Reinforcement learning based methods for dialogue policy selection are widely studied recently~\cite{BBQ,movie,SIGDIAL,continue}. These methods only need human to design the learning strategies and do not require massive training data. However, the expensive domain knowledge and human expert efforts for agents to learn from are necessary~\cite{withhuman,bootstraping}. Therefore, hybrid methods that integrate supervised learning and reinforcement learning are proposed recently~\cite{hybrid,kefu}. Thus, collecting massive training data becomes another manual work.&#10;&#10;More recently, end-to-end dialogue systems attract much attention because almost no human efforts are required and they are scalable for different domains~\cite{EACL,IJCNLP,deal,goal3}, especially with sequence-to-sequence based models~\cite{seq2seq}. Although those models have been proved to be effective on chit-chat conversations~\cite{chat1,chat2,chat3}, how to build agents that are goal-oriented with efficient dialogue managers through end-to-end approaches still remains questionable~\cite{goal1,goal2}, and we investigate the question in this paper. &#10;&#10;Our idea of enabling the agent to be efficient by modeling looking-ahead ability is inspired by the AI Planning concept, which is a traditional searching technology in the field of AI, and is suitable for goal-based tasks, such as robotics control~\cite{ai}. Recently, the concept is borrowed to dialogue system communities and integrated into deep learning models. For example, a trade-off method for training the agents neither with real human nor with user simulators is proposed, in order to obtain better policy learning results~\cite{DDQ}. In addition, at earlier time, the planning idea has been utilized for improving the dialogue generation task~\cite{plan1,plan2}.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Machine Learning, End-to-End Learning Models, Goal-Oriented Dialogue Systems, Linguistics, Cognitive Science, Dialogue Agent Efficiency, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2108.13811" label="2108.13811">
        <attvalues>
          <attvalue for="0" value="TREND: Trigger-Enhanced Relation-Extraction Network for Dialogues" />
          <attvalue for="1" value="  The goal of dialogue relation extraction (DRE) is to identify the relation&#10;between two entities in a given dialogue. During conversations, speakers may&#10;expose their relations to certain entities by explicit or implicit clues, such&#10;evidences called &quot;triggers&quot;. However, trigger annotations may not be always&#10;available for the target data, so it is challenging to leverage such&#10;information for enhancing the performance. Therefore, this paper proposes to&#10;learn how to identify triggers from the data with trigger annotations and then&#10;transfers the trigger-finding capability to other datasets for better&#10;performance. The experiments show that the proposed approach is capable of&#10;improving relation extraction performance of unseen relations and also&#10;demonstrate the transferability of our proposed trigger-finding model across&#10;different domains and datasets.&#10;" />
          <attvalue for="2" value="&#10;&#10;The goal of relation extraction (RE) is to identify the semantic relation type between two mentioned entities from a given text piece, which is one of basic and important natural language understanding (NLU) problems \cite{zhang2017position, zhou2021improved, cohen2020relation}.&#10;In this task setting, we are usually given a written sentence and a query pair containing two entities and asked to return the most possible relation type from a predefined set of relations.&#10;Dialogue relation extraction (DRE), on the other hand, aims to excavate underlying cross-sentence relation in natural human communications \cite{yu2020dialogue, jia2021ddrel}.&#10;The problem itself is well-motivated, because relations between entities in dialogues could potentially provide dialogue systems with additional features for better dialogue managing~\cite{peng2018deep, su2018discriminative} or response generation~\cite{su2018natural}. &#10;&#10;There are two popular datasets, DialogRE~\cite{yu2020dialogue} and DDRel~\cite{jia2021ddrel}, focusing on relation extraction in dialogues illustrated in Figure \ref{fig:example}.&#10;In DRE, given a conversation and a query pair, we aim to identify the interpersonal relationship between the given entities, where entities can be human or other types like locations.&#10;As shown in Figure~\ref{fig:example}, the evidences of relations within the conversation flow, called Triggers, provide informative cues for this task.&#10;A trigger can be a short phrase or even a single word with any possible part-of-speech.&#10;In the example, the clue for knowing the speaker 2 has a negative impression on the speaker 3 comes from the sentence ``You are arrogant.''&#10;Such hint is intuitively useful for deciding the relations.&#10;However, \cite{albalak2022d} is the only prior work that tried to explicitly leverage such signal for improving DRE, because such explanation annotations may not be always available~\cite{kung2020zero}.&#10;&#10;Prior work can be divided into two main lines, one of which is graph-based methods.&#10;DHGAT \cite{chen2020dialogue} presents an attention-based heterogeneous graph network to model multiple types of features;&#10;GDPNet \cite{xue2021gdpnet} constructs latent multi-view graphs to model possible relationships among tokens in a long sequence, and then refines the graphs by iterative graph convolution and pooling techniques.&#10;Another branch is BERT-based \cite{kenton2019bert} methods \cite{yu2020dialogue, xue2022embarrassingly}. SimepleRE \cite{xue2022embarrassingly} is a simple BERT model with an additional refinement gate for iteratively finding high-confidence prediction.&#10;LSR \cite{nan2020reasoning} is a latent structure refinement method for better reasoning in the document-level relation extraction task.&#10;Although it is known that using trigger information can significantly help the performance of relation extraction, only DialogRE has the annotated triggers.&#10;It is not guaranteed that utilizing the annotated triggers can generalize to other relations from other datasets, considering the discrepancy of their relation types.&#10;&#10;Given the target data without trigger annotations, this paper proposes TREND, a simple multi-tasking model with an attentional relation predictor, &#10;where it learns the general capability of finding triggers and transfers it to the unseen relations for performance improvement.&#10;The experiments show that our proposed method can effectively identify the explicit triggers and generalize to unseen relations towards great flexibility and practicality.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Trigger Identification, Computer Science, Linguistics, Dialogue Relation Extraction, Artificial Intelligence, Natural Language Processing, Transfer Learning" />
        </attvalues>
      </node>
      <node id="2004.08056" label="2004.08056">
        <attvalues>
          <attvalue for="0" value="Dialogue-Based Relation Extraction" />
          <attvalue for="1" value="  We present the first human-annotated dialogue-based relation extraction (RE)&#10;dataset DialogRE, aiming to support the prediction of relation(s) between two&#10;arguments that appear in a dialogue. We further offer DialogRE as a platform&#10;for studying cross-sentence RE as most facts span multiple sentences. We argue&#10;that speaker-related information plays a critical role in the proposed task,&#10;based on an analysis of similarities and differences between dialogue-based and&#10;traditional RE tasks. Considering the timeliness of communication in a&#10;dialogue, we design a new metric to evaluate the performance of RE methods in a&#10;conversational setting and investigate the performance of several&#10;representative RE methods on DialogRE. Experimental results demonstrate that a&#10;speaker-aware extension on the best-performing model leads to gains in both the&#10;standard and conversational evaluation settings. DialogRE is available at&#10;https://dataset.org/dialogre/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.02747" label="1808.02747">
        <attvalues>
          <attvalue for="0" value="Natural Language Generation by Hierarchical Decoding with Linguistic&#10;  Patterns" />
          <attvalue for="1" value="  Natural language generation (NLG) is a critical component in spoken dialogue&#10;systems. Classic NLG can be divided into two phases: (1) sentence planning:&#10;deciding on the overall sentence structure, (2) surface realization:&#10;determining specific word forms and flattening the sentence structure into a&#10;string. Many simple NLG models are based on recurrent neural networks (RNN) and&#10;sequence-to-sequence (seq2seq) model, which basically contains an&#10;encoder-decoder structure; these NLG models generate sentences from scratch by&#10;jointly optimizing sentence planning and surface realization using a simple&#10;cross entropy loss training criterion. However, the simple encoder-decoder&#10;architecture usually suffers from generating complex and long sentences,&#10;because the decoder has to learn all grammar and diction knowledge. This paper&#10;introduces a hierarchical decoding NLG model based on linguistic patterns in&#10;different levels, and shows that the proposed method outperforms the&#10;traditional one with a smaller model size. Furthermore, the design of the&#10;hierarchical decoding is flexible and easily-extensible in various NLG systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.02553" label="2012.02553">
        <attvalues>
          <attvalue for="0" value="DDRel: A New Dataset for Interpersonal Relation Classification in Dyadic&#10;  Dialogues" />
          <attvalue for="1" value="  Interpersonal language style shifting in dialogues is an interesting and&#10;almost instinctive ability of human. Understanding interpersonal relationship&#10;from language content is also a crucial step toward further understanding&#10;dialogues. Previous work mainly focuses on relation extraction between named&#10;entities in texts. In this paper, we propose the task of relation&#10;classification of interlocutors based on their dialogues. We crawled movie&#10;scripts from IMSDb, and annotated the relation labels for each session&#10;according to 13 pre-defined relationships. The annotated dataset DDRel consists&#10;of 6300 dyadic dialogue sessions between 694 pair of speakers with 53,126&#10;utterances in total. We also construct session-level and pair-level relation&#10;classification tasks with widely-accepted baselines. The experimental results&#10;show that this task is challenging for existing models and the dataset will be&#10;useful for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.05126" label="2109.05126">
        <attvalues>
          <attvalue for="0" value="D-REX: Dialogue Relation Extraction with Explanations" />
          <attvalue for="1" value="  Existing research studies on cross-sentence relation extraction in long-form&#10;multi-party conversations aim to improve relation extraction without&#10;considering the explainability of such methods. This work addresses that gap by&#10;focusing on extracting explanations that indicate that a relation exists while&#10;using only partially labeled data. We propose our model-agnostic framework,&#10;D-REX, a policy-guided semi-supervised algorithm that explains and ranks&#10;relations. We frame relation extraction as a re-ranking task and include&#10;relation- and entity-specific explanations as an intermediate step of the&#10;inference process. We find that about 90% of the time, human annotators prefer&#10;D-REX's explanations over a strong BERT-based joint relation extraction and&#10;explanation model. Finally, our evaluations on a dialogue relation extraction&#10;dataset show that our method is simple yet effective and achieves a&#10;state-of-the-art F1 score on relation extraction, improving upon existing&#10;methods by 13.5%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.06780" label="2012.06780">
        <attvalues>
          <attvalue for="0" value="GDPNet: Refining Latent Multi-View Graph for Relation Extraction" />
          <attvalue for="1" value="  Relation Extraction (RE) is to predict the relation type of two entities that&#10;are mentioned in a piece of text, e.g., a sentence or a dialogue. When the&#10;given text is long, it is challenging to identify indicative words for the&#10;relation prediction. Recent advances on RE task are from BERT-based sequence&#10;modeling and graph-based modeling of relationships among the tokens in the&#10;sequence. In this paper, we propose to construct a latent multi-view graph to&#10;capture various possible relationships among tokens. We then refine this graph&#10;to select important words for relation prediction. Finally, the representation&#10;of the refined graph and the BERT-based sequence representation are&#10;concatenated for relation extraction. Specifically, in our proposed GDPNet&#10;(Gaussian Dynamic Time Warping Pooling Net), we utilize Gaussian Graph&#10;Generator (GGG) to generate edges of the multi-view graph. The graph is then&#10;refined by Dynamic Time Warping Pooling (DTWPool). On DialogRE and TACRED, we&#10;show that GDPNet achieves the best performance on dialogue-level RE, and&#10;comparable performance with the state-of-the-arts on sentence-level RE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.13873" label="2012.13873">
        <attvalues>
          <attvalue for="0" value="An Embarrassingly Simple Model for Dialogue Relation Extraction" />
          <attvalue for="1" value="  Dialogue relation extraction (RE) is to predict the relation type of two&#10;entities mentioned in a dialogue. In this paper, we propose a simple yet&#10;effective model named SimpleRE for the RE task. SimpleRE captures the&#10;interrelations among multiple relations in a dialogue through a novel input&#10;format named BERT Relation Token Sequence (BRS). In BRS, multiple [CLS] tokens&#10;are used to capture possible relations between different pairs of entities&#10;mentioned in the dialogue. A Relation Refinement Gate (RRG) is then designed to&#10;extract relation-specific semantic representation in an adaptive manner.&#10;Experiments on the DialogRE dataset show that SimpleRE achieves the best&#10;performance, with much shorter training time. Further, SimpleRE outperforms all&#10;direct baselines on sentence-level RE without using external resources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.06312" label="2005.06312">
        <attvalues>
          <attvalue for="0" value="Reasoning with Latent Structure Refinement for Document-Level Relation&#10;  Extraction" />
          <attvalue for="1" value="  Document-level relation extraction requires integrating information within&#10;and across multiple sentences of a document and capturing complex interactions&#10;between inter-sentence entities. However, effective aggregation of relevant&#10;information in the document remains a challenging research question. Existing&#10;approaches construct static document-level graphs based on syntactic trees,&#10;co-references or heuristics from the unstructured text to model the&#10;dependencies. Unlike previous methods that may not be able to capture rich&#10;non-local interactions for inference, we propose a novel model that empowers&#10;the relational reasoning across sentences by automatically inducing the latent&#10;document-level graph. We further develop a refinement strategy, which enables&#10;the model to incrementally aggregate relevant information for multi-hop&#10;reasoning. Specifically, our model achieves an F1 score of 59.05 on a&#10;large-scale document-level dataset (DocRED), significantly improving over the&#10;previous results, and also yields new state-of-the-art results on the CDR and&#10;GDA dataset. Furthermore, extensive analyses show that the model is able to&#10;discover more accurate inter-sentence relations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.08491" label="2401.08491">
        <attvalues>
          <attvalue for="0" value="Contrastive Perplexity for Controlled Generation: An Application in&#10;  Detoxifying Large Language Models" />
          <attvalue for="1" value="  The generation of undesirable and factually incorrect content of large&#10;language models poses a significant challenge and remains largely an unsolved&#10;issue. This paper studies the integration of a contrastive learning objective&#10;for fine-tuning LLMs for implicit knowledge editing and controlled text&#10;generation. Optimizing the training objective entails aligning text&#10;perplexities in a contrastive fashion. To facilitate training the model in a&#10;self-supervised fashion, we leverage an off-the-shelf LLM for training data&#10;generation. We showcase applicability in the domain of detoxification. Herein,&#10;the proposed approach leads to a significant decrease in the generation of&#10;toxic content while preserving general utility for downstream tasks such as&#10;commonsense reasoning and reading comprehension. The proposed approach is&#10;conceptually simple but empirically powerful.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language model (LLM) technology advancements have rapidly propelled their integration into numerous NLP systems. As their prevalence grows in daily applications, the imperative to control toxicity within these models becomes increasingly paramount. The challenge lies in preserving performance while effectively mitigating their potential toxicity~\cite{gehman-etal-2020-realtoxicityprompts,xu-etal-2021-detoxifying,welbl-etal-2021-challenges-detoxifying,hartvigsen2022toxigen,hosseini2023empirical,welleck2023generating}, a concern at the forefront of modern LLM development.&#10;&#10;Current methodologies predominantly employ a pipeline approach: pre-processing data to expunge toxic language, conventional LLM training, and a subsequent post-processing step to cleanse generated text. \\&#10;However, this is problematic for several reasons. First, heavy data pre-processing is extremely challenging at scale and significantly deteriorates performance, especially when content is removed. Second, post-processing relies on subjective heuristics, limiting both utility and scalability~\cite{liu-etal-2021-dexperts,kumar-etal-2023-controlled,hallinan-etal-2023-detoxifying}.&#10;&#10;Despite shared concerns regarding toxicity, existing approaches tend toward superficial censorship, often prompting LLMs to avoid sensitive topics altogether, limiting applicability for marginalized groups and inadvertently allowing for implicit toxicity~\cite{zou2023universal,deshpande-etal-2023-toxicity,wei2023jailbroken,liu2023jailbreaking}. An example of this phenomenon is when an LLM detects a hint of sensitivity in a query and opts to avoid addressing it directly, often responding with generic statements such as ``I can't answer,'' thereby evading potentially sensitive topics altogether.&#10;&#10;Recently, there has been increased interest in the research community in LLM alignment,&#10;that is, training techniques to align model output to the user’s intent, such as Reinforcement Learning through Human (RLHF)~\cite{NIPS2017_d5e2c0ad} Feedback and Direct Preference Optimization (DPO)~\cite{rafailov2023direct,bai2022constitutional}. However, LLM alignment typically affects the performance. ~\cite{bekbayev2023poison} show in their work that aligning LLMs by reinforcing models not to respond to specific user inputs degraded the performance. In contrast, \cite{bai2022training} show that degradation or improvement in performance by alignment is dependent on model size. \\&#10;We argue that LLMs should not simply avoid sensitive topics but comprehend toxicity and convey concepts in non-toxic ways. Instead of avoiding a topic altogether by imposing guardrails, we posit the meaningfulness of the exposure toxicity in a contrastive fashion to learn to differentiate semantics. \\&#10;This is because, among other things, expressing an idea in both a toxic and non-toxic manner often merely involves minor language alterations, as can be seen in the following examples:&#10;\\&#10;\\&#10;Toxic-1: The essay is total \underline{bullshit}. \\ \contour{black}{$\Rightarrow$} Detoxified: The essay \underline{should be improved}.&#10;\\&#10;\\&#10;Toxic-2: He is a \underline{bad-ass} politican. \\ \contour{black}{$\Rightarrow$} Detoxified: He is a \underline{tough} politican.&#10;\\&#10;\\&#10;Toxic-3: She acts like a \underline{moron}. \\ \contour{black}{$\Rightarrow$} Detoxified: I don't like her behaviour.&#10;\\&#10;&#10;We propose a holistic framework for implicit knowledge editing, modifying language at the stylistic level—a move toward rendering LLMs more ``politically correct'' on ambiguous topics, as opposed to silencing them entirely~\cite{tang2023detoxify,welleck2023generating}.&#10;&#10;Our method, dubbed \underline{C}ontrastive \underline{P}erplexity (CP), introduces a simple yet potent technique for implicit knowledge editing and controlled text generation. We emphasize differentiating tokens between these sets by generating positive and negative sets from LLM queries and enforcing a contrastive loss with a margin. This approach considers the toxicity of generated outputs and their semantic relevance to input prompts, aiming to avoiding toxic language on sensitive topics whenever feasible while maintain general LLM utility. Crucially, we advocate for this technique in gray-zone topics, emphasizing a nuanced strategy while suggesting hard removal for red-flag topics to prevent potential misuse. See Fig.~\ref{fig:white-box_overview} for an illustration of the effect of CP on toxicity and similarity w.r.t. input context for different language models.&#10;&#10;In our study, we advocate for directly utilizing data generated by LLMs, recognizing that it reflects the inherent biases present within these models. This approach enables us to implement auto-corrections by paraphrasing when required, effectively steering clear of toxic terms and concepts.&#10;&#10;To generate our data, we employed a straightforward method. We prompted an off-the-shelf LLM to generate paraphrased non-toxic inputs. This results in the creation of a positive set of sentences. Conversely, for the negative set, we employed adversarial prompting techniques. Here, the LLM was tasked with generating a set of toxic sentences in a counterfactual manner.&#10;\\&#10;&#10;Contributions: The contributions of the proposed work are threefold: First, contrastive perplexity, a holistic approach for knowledge editing Second, a simple strategy for utilizing LLM for automatic generation of contrastive pairs Third, showcasing the applicability of our framework for toxicity removal while maintaining the general utility of LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Controlled Text Generation, Toxic Content Reduction, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2104.06390" label="2104.06390">
        <attvalues>
          <attvalue for="0" value="Detoxifying Language Models Risks Marginalizing Minority Voices" />
          <attvalue for="1" value="  Language models (LMs) must be both safe and equitable to be responsibly&#10;deployed in practice. With safety in mind, numerous detoxification techniques&#10;(e.g., Dathathri et al. 2020; Krause et al. 2020) have been proposed to&#10;mitigate toxic LM generations. In this work, we show that current&#10;detoxification techniques hurt equity: they decrease the utility of LMs on&#10;language used by marginalized groups (e.g., African-American English and&#10;minority identity mentions). In particular, we perform automatic and human&#10;evaluations of text generation quality when LMs are conditioned on inputs with&#10;different dialects and group identifiers. We find that detoxification makes LMs&#10;more brittle to distribution shift, especially on language used by marginalized&#10;groups. We identify that these failures stem from detoxification methods&#10;exploiting spurious correlations in toxicity datasets. Overall, our results&#10;highlight the tension between the controllability and distributional robustness&#10;of LMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.09211" label="2301.09211">
        <attvalues>
          <attvalue for="0" value="An Empirical Study of Metrics to Measure Representational Harms in&#10;  Pre-Trained Language Models" />
          <attvalue for="1" value="  Large-scale Pre-Trained Language Models (PTLMs) capture knowledge from&#10;massive human-written data which contains latent societal biases and toxic&#10;contents. In this paper, we leverage the primary task of PTLMs, i.e., language&#10;modeling, and propose a new metric to quantify manifested implicit&#10;representational harms in PTLMs towards 13 marginalized demographics. Using&#10;this metric, we conducted an empirical analysis of 24 widely used PTLMs. Our&#10;analysis provides insights into the correlation between the proposed metric in&#10;this work and other related metrics for representational harm. We observe that&#10;our metric correlates with most of the gender-specific metrics in the&#10;literature. Through extensive experiments, we explore the connections between&#10;PTLMs architectures and representational harms across two dimensions: depth and&#10;width of the networks. We found that prioritizing depth over width, mitigates&#10;representational harms in some PTLMs. Our code and data can be found at&#10;https://github.com/microsoft/SafeNLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.19230" label="2305.19230">
        <attvalues>
          <attvalue for="0" value="Controlled Text Generation with Hidden Representation Transformations" />
          <attvalue for="1" value="  We propose CHRT (Control Hidden Representation Transformation) - a controlled&#10;language generation framework that steers large language models to generate&#10;text pertaining to certain attributes (such as toxicity). CHRT gains attribute&#10;control by modifying the hidden representation of the base model through&#10;learned transformations. We employ a contrastive-learning framework to learn&#10;these transformations that can be combined to gain multi-attribute control. The&#10;effectiveness of CHRT is experimentally shown by comparing it with seven&#10;baselines over three attributes. CHRT outperforms all the baselines in the task&#10;of detoxification, positive sentiment steering, and text simplification while&#10;minimizing the loss in linguistic qualities. Further, our approach has the&#10;lowest inference latency of only 0.01 seconds more than the base model, making&#10;it the most suitable for high-performance production environments. We&#10;open-source our code and release two novel datasets to further propel&#10;controlled language generation research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10543" label="2212.10543">
        <attvalues>
          <attvalue for="0" value="Detoxifying Text with MaRCo: Controllable Revision with Experts and&#10;  Anti-Experts" />
          <attvalue for="1" value="  Text detoxification has the potential to mitigate the harms of toxicity by&#10;rephrasing text to remove offensive meaning, but subtle toxicity remains&#10;challenging to tackle. We introduce MaRCo, a detoxification algorithm that&#10;combines controllable generation and text rewriting methods using a Product of&#10;Experts with autoencoder language models (LMs). MaRCo uses likelihoods under a&#10;non-toxic LM (expert) and a toxic LM (anti-expert) to find candidate words to&#10;mask and potentially replace. We evaluate our method on several subtle toxicity&#10;and microaggressions datasets, and show that it not only outperforms baselines&#10;on automatic metrics, but MaRCo's rewrites are preferred 2.1 $\times$ more in&#10;human evaluation. Its applicability to instances of subtle toxicity is&#10;especially promising, demonstrating a path forward for addressing increasingly&#10;elusive online hate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.02483" label="2307.02483">
        <attvalues>
          <attvalue for="0" value="Jailbroken: How Does LLM Safety Training Fail?" />
          <attvalue for="1" value="  Large language models trained for safety and harmlessness remain susceptible&#10;to adversarial misuse, as evidenced by the prevalence of &quot;jailbreak&quot; attacks on&#10;early releases of ChatGPT that elicit undesired behavior. Going beyond&#10;recognition of the issue, we investigate why such attacks succeed and how they&#10;can be created. We hypothesize two failure modes of safety training: competing&#10;objectives and mismatched generalization. Competing objectives arise when a&#10;model's capabilities and safety goals conflict, while mismatched generalization&#10;occurs when safety training fails to generalize to a domain for which&#10;capabilities exist. We use these failure modes to guide jailbreak design and&#10;then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's&#10;Claude v1.3, against both existing and newly designed attacks. We find that&#10;vulnerabilities persist despite the extensive red-teaming and safety-training&#10;efforts behind these models. Notably, new attacks utilizing our failure modes&#10;succeed on every prompt in a collection of unsafe requests from the models'&#10;red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our&#10;analysis emphasizes the need for safety-capability parity -- that safety&#10;mechanisms should be as sophisticated as the underlying model -- and argues&#10;against the idea that scaling alone can resolve these safety failure modes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13860" label="2305.13860">
        <attvalues>
          <attvalue for="0" value="Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study" />
          <attvalue for="1" value="  Large Language Models (LLMs), like ChatGPT, have demonstrated vast potential&#10;but also introduce challenges related to content constraints and potential&#10;misuse. Our study investigates three key research questions: (1) the number of&#10;different prompt types that can jailbreak LLMs, (2) the effectiveness of&#10;jailbreak prompts in circumventing LLM constraints, and (3) the resilience of&#10;ChatGPT against these jailbreak prompts. Initially, we develop a classification&#10;model to analyze the distribution of existing prompts, identifying ten distinct&#10;patterns and three categories of jailbreak prompts. Subsequently, we assess the&#10;jailbreak capability of prompts with ChatGPT versions 3.5 and 4.0, utilizing a&#10;dataset of 3,120 jailbreak questions across eight prohibited scenarios.&#10;Finally, we evaluate the resistance of ChatGPT against jailbreak prompts,&#10;finding that the prompts can consistently evade the restrictions in 40 use-case&#10;scenarios. The study underscores the importance of prompt structures in&#10;jailbreaking LLMs and discusses the challenges of robust jailbreak prompt&#10;generation and prevention.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.13449" label="2308.13449">
        <attvalues>
          <attvalue for="0" value="The Poison of Alignment" />
          <attvalue for="1" value="  From the perspective of content safety issues, alignment has shown to limit&#10;large language models' (LLMs) harmful content generation. This intentional&#10;method of reinforcing models to not respond to certain user inputs seem to be&#10;present in many modern open-source instruction tuning datasets such as&#10;OpenAssistant or Guanaco. We introduce a novel insight to an instruction-tuned&#10;model's performance affected by the presence of alignment in supervised&#10;fine-tuning dataset. To be specific, we noticed that alignment acts as if it is&#10;poisoning the instruction dataset. Experimentally, we demonstrate that aligned&#10;answers significantly worsen the performance of the resulting fine-tuned&#10;model's on various reasoning benchmarks such as Big Bench (BBH), Massive&#10;Multitask Language Understanding (MMLU), Human Eval, and Discrete Reasoning&#10;Over Paragraphs (DROP), performing worse than the counterpart tuned without&#10;alignment by 4-33%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.01359" label="2007.01359">
        <attvalues>
          <attvalue for="0" value="A Bayesian Multilingual Document Model for Zero-shot Topic&#10;  Identification and Discovery" />
          <attvalue for="1" value="  In this paper, we present a Bayesian multilingual document model for learning&#10;language-independent document embeddings. The model is an extension of BaySMM&#10;[Kesiraju et al 2020] to the multilingual scenario. It learns to represent the&#10;document embeddings in the form of Gaussian distributions, thereby encoding the&#10;uncertainty in its covariance. We propagate the learned uncertainties through&#10;linear classifiers that benefit zero-shot cross-lingual topic identification.&#10;Our experiments on 17 languages show that the proposed multilingual Bayesian&#10;document model performs competitively, when compared to other systems based on&#10;large-scale neural networks (LASER, XLM-R, mUSE) on 8 high-resource languages,&#10;and outperforms these systems on 9 mid-resource languages. We revisit&#10;cross-lingual topic identification in zero-shot settings by taking a deeper&#10;dive into current datasets, baseline systems and the languages covered. We&#10;identify shortcomings in the existing evaluation protocol (MLDoc dataset), and&#10;propose a robust alternative scheme, while also extending the cross-lingual&#10;experimental setup to 17 languages. Finally, we consolidate the observations&#10;from all our experiments, and discuss points that can potentially benefit the&#10;future research works in applications relying on cross-lingual transfers.&#10;" />
          <attvalue for="2" value="&#10;\blfootnote{*While at Speech@FIT BUT}&#10;The zero-shot cross-lingual topic identification (ID) or document classification aims to classify documents from target languages using a classifier trained on examples from one or more source language(s). This is mainly useful in scenarios where the data from target language(s) have little or no labels to train an in-language classifier. Such tasks also have real-world applications~\cite{strassel:2016:lorelei}.&#10;Furthermore, the cross-lingual transfer experiments can also help to analyse and test the capabilities of an underlying multilingual language model.&#10;&#10;The common approach is to first train a multilingual language model that aims to capture the semantic relations of words in context, independent of the language~\cite{Ammar:2016:MMWE,Mikel:2019:MMS,Unicoder:2019,Alexis:2020:XLR,Feng:2020:LaBSE}. Such a multilingual model can then later be either (i) fine-tuned for classification~\cite{Siddhant:2019:MMTE} task using labelled examples from source language(s), or (ii) used to extract low-dimensional embeddings (representations) for documents from both source and target languages~\cite{Reimers:2020:Making}; the embeddings from source language(s) together with annotated labels are then used for training a light-weight independent classifier for cross-lingual topic ID, which is then used to classify embeddings from target languages.&#10;The former approach relying on fine-tuning is not efficient as it would require to keep a copy of the entire multilingual model (or bunch of adapter layers) for every source language, and every down-stream task. The latter approach of extracting language-agnostic document (sentence) embeddings is more practical as it would require only one model, and several light-weight downstream classifiers. This paper entirely focuses on models, experiments and analysis related to the latter scheme relying on language agnostic document embeddings, followed by a light-weight classifier.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Multilingual Document Modeling, Bayesian Uncertainty Modeling, Natural Language Processing, Statistics, Cross-Lingual Topic Identification" />
        </attvalues>
      </node>
      <node id="1602.01925" label="1602.01925">
        <attvalues>
          <attvalue for="0" value="Massively Multilingual Word Embeddings" />
          <attvalue for="1" value="  We introduce new methods for estimating and evaluating embeddings of words in&#10;more than fifty languages in a single shared embedding space. Our estimation&#10;methods, multiCluster and multiCCA, use dictionaries and monolingual data; they&#10;do not require parallel data. Our new evaluation method, multiQVEC-CCA, is&#10;shown to correlate better than previous ones with two downstream tasks (text&#10;categorization and parsing). We also describe a web portal for evaluation that&#10;will facilitate further research in this area, along with open-source releases&#10;of all our methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.00437" label="1909.00437">
        <attvalues>
          <attvalue for="0" value="Evaluating the Cross-Lingual Effectiveness of Massively Multilingual&#10;  Neural Machine Translation" />
          <attvalue for="1" value="  The recently proposed massively multilingual neural machine translation (NMT)&#10;system has been shown to be capable of translating over 100 languages to and&#10;from English within a single model. Its improved translation performance on low&#10;resource languages hints at potential cross-lingual transfer capability for&#10;downstream tasks. In this paper, we evaluate the cross-lingual effectiveness of&#10;representations from the encoder of a massively multilingual NMT model on 5&#10;downstream classification and sequence labeling tasks covering a diverse set of&#10;over 50 languages. We compare against a strong baseline, multilingual BERT&#10;(mBERT), in different cross-lingual transfer learning scenarios and show gains&#10;in zero-shot transfer in 4 out of these 5 tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.09638" label="2207.09638">
        <attvalues>
          <attvalue for="0" value="Doge Tickets: Uncovering Domain-general Language Models by Playing&#10;  Lottery Tickets" />
          <attvalue for="1" value="  Over-parameterized models, typically pretrained language models (LMs), have&#10;shown an appealing expressive power due to their small learning bias. However,&#10;the huge learning capacity of LMs can also lead to large learning variance. In&#10;a pilot study, we find that, when faced with multiple domains, a critical&#10;portion of parameters behave unexpectedly in a domain-specific manner while&#10;others behave in a domain-general one. Motivated by this phenomenon, we for the&#10;first time posit that domain-general parameters can underpin a domain-general&#10;LM that can be derived from the original LM. To uncover the domain-general LM,&#10;we propose to identify domain-general parameters by playing lottery tickets&#10;(dubbed doge tickets). In order to intervene the lottery, we propose a&#10;domain-general score, which depicts how domain-invariant a parameter is by&#10;associating it with the variance. Comprehensive experiments are conducted on&#10;the Amazon, Mnli and OntoNotes datasets. The results show that the doge tickets&#10;obtains an improved out-of-domain generalization in comparison with a range of&#10;competitive baselines. Analysis results further hint the existence of&#10;domain-general parameters and the performance consistency of doge tickets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Domain Generalization Techniques, Linguistics, Artificial Intelligence, Mathematics, Statistics, Language Model Optimization, Parameter Analysis Methods" />
        </attvalues>
      </node>
      <node id="2209.12616" label="2209.12616">
        <attvalues>
          <attvalue for="0" value="T-NER: An All-Round Python Library for Transformer-based Named Entity&#10;  Recognition" />
          <attvalue for="1" value="  Language model (LM) pretraining has led to consistent improvements in many&#10;NLP downstream tasks, including named entity recognition (NER). In this paper,&#10;we present T-NER (Transformer-based Named Entity Recognition), a Python library&#10;for NER LM finetuning. In addition to its practical utility, T-NER facilitates&#10;the study and investigation of the cross-domain and cross-lingual&#10;generalization ability of LMs finetuned on NER. Our library also provides a web&#10;app where users can get model predictions interactively for arbitrary text,&#10;which facilitates qualitative model evaluation for non-expert programmers. We&#10;show the potential of the library by compiling nine public NER datasets into a&#10;unified format and evaluating the cross-domain and cross-lingual performance&#10;across the datasets. The results from our initial experiments show that&#10;in-domain performance is generally competitive across datasets. However,&#10;cross-domain generalization is challenging even with a large pretrained LM,&#10;which has nevertheless capacity to learn domain-specific features if fine-tuned&#10;on a combined dataset. To facilitate future research, we also release all our&#10;LM checkpoints via the Hugging Face model hub.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language model (LM) pretraining has become one of the most common strategies within the natural language processing (NLP) community to solve downstream tasks \cite{peters-etal-2018-deep, howard-ruder-2018-universal, radford2018improving, radford2019language, devlin2018bert}.&#10;LMs trained over large textual data only need to be finetuned on downstream tasks to outperform most of the task-specific designed models. Among the NLP tasks impacted by LM pretraining, named entity recognition (NER) is one of the most prevailing and practical applications. However, the availability of open-source NER libraries for LM training is limited.&#10;&#10;In this paper, we introduce {T-NER}, an open-source Python library for cross-domain analysis for NER with pretrained Transformer-based LMs.&#10;Figure~\ref{fig:diagram} shows a brief overview of our library and its functionalities.&#10;The library facilitates NER experimental design including easy-to-use features such as model training and evaluation. Most notably, it enables to organize cross-domain analyses such as training a NER model and testing it on a different domain, with a small configuration.&#10;We also report initial experiment results, by which we show that although cross-domain NER is challenging, if it has an access to new domains, LM can successfully learn new domain knowledge. The results give us an insight that LM is capable to learn a variety of domain knowledge, but an ordinary finetuning scheme on single dataset most likely causes overfitting and results in poor domain generalization.&#10;&#10;As a system design, T-NER is implemented in Pytorch \cite{paszke2019pytorch} on top of the Transformers library \cite{Wolf2019HuggingFacesTS}. Moreover, the interfaces of our training and evaluation modules are highly inspired by Scikit-learn \cite{pedregosa2011scikit}, enabling an interoperability with recent models as well as integrating them in an intuitive way. In addition to the versatility of our toolkit for NER experimentation, we also include an online demo and robust pre-trained models trained across domains.&#10;In the following sections, we provide a brief overview about NER in Section \ref{sec:ner}, explain the system architecture of T-NER with a few basic usages in Section~\ref{sec:system-overview} and describe experiment results on cross-domain transfer with our library in Section~\ref{sec:experiments}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Named Entity Recognition, Artificial Intelligence, Language Model Pretraining, Cross-Domain Generalization" />
        </attvalues>
      </node>
      <node id="1912.01703" label="1912.01703">
        <attvalues>
          <attvalue for="0" value="PyTorch: An Imperative Style, High-Performance Deep Learning Library" />
          <attvalue for="1" value="  Deep learning frameworks have often focused on either usability or speed, but&#10;not both. PyTorch is a machine learning library that shows that these two goals&#10;are in fact compatible: it provides an imperative and Pythonic programming&#10;style that supports code as a model, makes debugging easy and is consistent&#10;with other popular scientific computing libraries, while remaining efficient&#10;and supporting hardware accelerators such as GPUs.&#10;  In this paper, we detail the principles that drove the implementation of&#10;PyTorch and how they are reflected in its architecture. We emphasize that every&#10;aspect of PyTorch is a regular Python program under the full control of its&#10;user. We also explain how the careful and pragmatic implementation of the key&#10;components of its runtime enables them to work together to achieve compelling&#10;performance.&#10;  We demonstrate the efficiency of individual subsystems, as well as the&#10;overall speed of PyTorch on several common benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1201.0490" label="1201.0490">
        <attvalues>
          <attvalue for="0" value="Scikit-learn: Machine Learning in Python" />
          <attvalue for="1" value="  Scikit-learn is a Python module integrating a wide range of state-of-the-art&#10;machine learning algorithms for medium-scale supervised and unsupervised&#10;problems. This package focuses on bringing machine learning to non-specialists&#10;using a general-purpose high-level language. Emphasis is put on ease of use,&#10;performance, documentation, and API consistency. It has minimal dependencies&#10;and is distributed under the simplified BSD license, encouraging its use in&#10;both academic and commercial settings. Source code, binaries, and documentation&#10;can be downloaded from http://scikit-learn.org.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.01307" label="2208.01307">
        <attvalues>
          <attvalue for="0" value="Multilingual Coreference Resolution in Multiparty Dialogue" />
          <attvalue for="1" value="  Existing multiparty dialogue datasets for entity coreference resolution are&#10;nascent, and many challenges are still unaddressed. We create a large-scale&#10;dataset, Multilingual Multiparty Coref (MMC), for this task based on TV&#10;transcripts. Due to the availability of gold-quality subtitles in multiple&#10;languages, we propose reusing the annotations to create silver coreference&#10;resolution data in other languages (Chinese and Farsi) via annotation&#10;projection. On the gold (English) data, off-the-shelf models perform relatively&#10;poorly on MMC, suggesting that MMC has broader coverage of multiparty&#10;coreference than prior datasets. On the silver data, we find success both using&#10;it for data augmentation and training from scratch, which effectively simulates&#10;the zero-shot cross-lingual setting.&#10;" />
          <attvalue for="2" value="&#10;Coreference resolution is a challenging aspect of understanding natural language dialogue \cite{khosla-etal-2021-codi}. Many dialogue datasets are between two participants, even though there are distinct challenges that arise in the multiparty setting with more than two speakers. Fig. \ref{fig-intro} shows how ``you'' could refer to any subset of the listeners of an utterance.&#10;While there are some datasets on multiparty conversations from TV transcripts \cite{choi-chen-2018-semeval}, they only annotate people, resulting in incomplete annotations across entity types. &#10;Moreover, these datasets are only limited to English, and works in dialogue coreference resolution in other languages are rare \cite{muzerelle-etal-2014-ancor}.&#10;&#10;We introduce a new (entity) coreference resolution dataset focused on multiparty dialogue that supports experiments in multiple languages. We first annotate for coreference on the transcripts from two popular TV shows, in English. We then leverage existing gold subtitle translations~\cite{opensubtitle} in Chinese and Farsi to project our annotations, resulting in a multilingual corpus (Fig. \ref{fig-intro}).&#10;&#10;Our experiments demonstrate that coreference resolution models trained on existing datasets are not robust to a shift to this domain. Further, we demonstrate that training on our projected annotations to non-English languages leads to improvements in non-English evaluation. Finally, we lay out an evaluation for zero-shot cross-lingual coreference resolution, requiring models to test on other languages with no in-language examples. We release over 1,200 scenes from TV shows with all annotations and related metadata in English, Chinese, and Farsi, which we call MMC: Multilingual Multiparty Coreference. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cross-Lingual Coreference Modeling, Entity Coreference Resolution, Multilingual Dataset Creation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1809.06142" label="1809.06142">
        <attvalues>
          <attvalue for="0" value="Open Subtitles Paraphrase Corpus for Six Languages" />
          <attvalue for="1" value="  This paper accompanies the release of Opusparcus, a new paraphrase corpus for&#10;six European languages: German, English, Finnish, French, Russian, and Swedish.&#10;The corpus consists of paraphrases, that is, pairs of sentences in the same&#10;language that mean approximately the same thing. The paraphrases are extracted&#10;from the OpenSubtitles2016 corpus, which contains subtitles from movies and TV&#10;shows. The informal and colloquial genre that occurs in subtitles makes such&#10;data a very interesting language resource, for instance, from the perspective&#10;of computer assisted language learning. For each target language, the&#10;Opusparcus data have been partitioned into three types of data sets: training,&#10;development and test sets. The training sets are large, consisting of millions&#10;of sentence pairs, and have been compiled automatically, with the help of&#10;probabilistic ranking functions. The development and test sets consist of&#10;sentence pairs that have been checked manually; each set contains approximately&#10;1000 sentence pairs that have been verified to be acceptable paraphrases by two&#10;annotators.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.05176" label="2305.05176">
        <attvalues>
          <attvalue for="0" value="FrugalGPT: How to Use Large Language Models While Reducing Cost and&#10;  Improving Performance" />
          <attvalue for="1" value="  There is a rapidly growing number of large language models (LLMs) that users&#10;can query for a fee. We review the cost associated with querying popular LLM&#10;APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have&#10;heterogeneous pricing structures, with fees that can differ by two orders of&#10;magnitude. In particular, using LLMs on large collections of queries and text&#10;can be expensive. Motivated by this, we outline and discuss three types of&#10;strategies that users can exploit to reduce the inference cost associated with&#10;using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As&#10;an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM&#10;cascade which learns which combinations of LLMs to use for different queries in&#10;order to reduce cost and improve accuracy. Our experiments show that FrugalGPT&#10;can match the performance of the best individual LLM (e.g. GPT-4) with up to&#10;98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost.&#10;The ideas and findings presented here lay a foundation for using LLMs&#10;sustainably and efficiently.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.10568" label="2004.10568">
        <attvalues>
          <attvalue for="0" value="Up or Down? Adaptive Rounding for Post-Training Quantization" />
          <attvalue for="1" value="  When quantizing neural networks, assigning each floating-point weight to its&#10;nearest fixed-point value is the predominant approach. We find that, perhaps&#10;surprisingly, this is not the best we can do. In this paper, we propose&#10;AdaRound, a better weight-rounding mechanism for post-training quantization&#10;that adapts to the data and the task loss. AdaRound is fast, does not require&#10;fine-tuning of the network, and only uses a small amount of unlabelled data. We&#10;start by theoretically analyzing the rounding problem for a pre-trained neural&#10;network. By approximating the task loss with a Taylor series expansion, the&#10;rounding task is posed as a quadratic unconstrained binary optimization&#10;problem. We simplify this to a layer-wise local loss and propose to optimize&#10;this loss with a soft relaxation. AdaRound not only outperforms&#10;rounding-to-nearest by a significant margin but also establishes a new&#10;state-of-the-art for post-training quantization on several networks and tasks.&#10;Without fine-tuning, we can quantize the weights of Resnet18 and Resnet50 to 4&#10;bits while staying within an accuracy loss of 1%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.10438" label="2211.10438">
        <attvalues>
          <attvalue for="0" value="SmoothQuant: Accurate and Efficient Post-Training Quantization for Large&#10;  Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) show excellent performance but are compute- and&#10;memory-intensive. Quantization can reduce memory and accelerate inference.&#10;However, existing methods cannot maintain accuracy and hardware efficiency at&#10;the same time. We propose SmoothQuant, a training-free, accuracy-preserving,&#10;and general-purpose post-training quantization (PTQ) solution to enable 8-bit&#10;weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that&#10;weights are easy to quantize while activations are not, SmoothQuant smooths the&#10;activation outliers by offline migrating the quantization difficulty from&#10;activations to weights with a mathematically equivalent transformation.&#10;SmoothQuant enables an INT8 quantization of both weights and activations for&#10;all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG,&#10;Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x&#10;speedup and 2x memory reduction for LLMs with negligible loss in accuracy.&#10;SmoothQuant enables serving 530B LLM within a single node. Our work offers a&#10;turn-key solution that reduces hardware costs and democratizes LLMs. Code is&#10;available at https://github.com/mit-han-lab/smoothquant.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.10199" label="2106.10199">
        <attvalues>
          <attvalue for="0" value="BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based&#10;  Masked Language-models" />
          <attvalue for="1" value="  We introduce BitFit, a sparse-finetuning method where only the bias-terms of&#10;the model (or a subset of them) are being modified. We show that with&#10;small-to-medium training data, applying BitFit on pre-trained BERT models is&#10;competitive with (and sometimes better than) fine-tuning the entire model. For&#10;larger data, the method is competitive with other sparse fine-tuning methods.&#10;Besides their practical utility, these findings are relevant for the question&#10;of understanding the commonly-used process of finetuning: they support the&#10;hypothesis that finetuning is mainly about exposing knowledge induced by&#10;language-modeling training, rather than learning new task-specific linguistic&#10;knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The problem of identifying the minimal set of parameters that need to be fine-tuned to achieve good performance in end-tasks relates both to practical questions of model compression, and also to more fundamental question on the nature of the pre-training and finetuning process, the ``linguistic knowledge`` induced by each of them, and the extent to which it generalizes to different tasks. &#10;&#10;Over-parameterization.\quad&#10; Large LM models were shown to be over-parameterized: they contain more parameters than needed in inference \cite{compression-orig, DBLP:journals/corr/HintonVD15, DBLP:conf/iclr/UrbanGKAWMPRC17, DBLP:journals/tnn/Karnin90, DBLP:journals/tnn/Reed93, DBLP:journals/cejcs/AugastaK13, DBLP:conf/interspeech/LiuZW14, han2015learning, DBLP:conf/iclr/MolchanovTKAK17}. \cite{DBLP:journals/corr/abs-2002-08307} have demonstrated that overparmeterization can be exploited in finetuning: pruned network perform well in transfer setting. We work in a complementary setting, where the entire model is kept, but only some parameters are updated. The remarkable success of those works have sparked interest the lottery-ticket hypothesis \cite{DBLP:conf/iclr/FrankleC19, DBLP:conf/nips/ChenFC0ZWC20, DBLP:conf/emnlp/PrasannaRR20}: the conjecture that large models are needed in pretraining only to induce (in high probability) the existing of sub-networks initialized with the correct inductive bias for learning, and the findings that those sparse networks often transfer well to different tasks. &#10;&#10;Bias terms.\quad&#10;Bias terms and their importance are rarely discussed in the literature. \cite{zhao-etal-2020-masking} describe a masking-based fine-tuning method, and explicitly mention ignoring the bias terms, as handling them ``did not observe a positive effect on performance''.\\&#10; An exception is the work of \cite{DBLP:conf/icml/WangZB19} who analyzed bias terms from the perspective of attribution method. They demonstrate that the last layer bias values are responsible for the predicted class, and propose a way to back-propagate their importance. \cite{DBLP:conf/acl/MichelN18} finetuned the biases of the output softmax in an NMT systems, to personalize the output vocabulary, and \cite{DBLP:journals/corr/abs-2003-00152} have demonstrated that randomly-initialized CNNs achieve reasonable accuracy after training the batch-norm layers alone. Finally, and closest to our work, \cite{tinytl} demonstrate that bias-only fine-tuning similar to ours is effective also for adaptation of pre-trained computer vision models. Our work empirically shows the importance and power of the bias parameters to substantially change the networks' behavior, calling for further analysis and attention on the bias terms.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.04647" label="2106.04647">
        <attvalues>
          <attvalue for="0" value="Compacter: Efficient Low-Rank Hypercomplex Adapter Layers" />
          <attvalue for="1" value="  Adapting large-scale pretrained language models to downstream tasks via&#10;fine-tuning is the standard method for achieving state-of-the-art performance&#10;on NLP benchmarks. However, fine-tuning all weights of models with millions or&#10;billions of parameters is sample-inefficient, unstable in low-resource&#10;settings, and wasteful as it requires storing a separate copy of the model for&#10;each task. Recent work has developed parameter-efficient fine-tuning methods,&#10;but these approaches either still require a relatively large number of&#10;parameters or underperform standard fine-tuning. In this work, we propose&#10;Compacter, a method for fine-tuning large-scale language models with a better&#10;trade-off between task performance and the number of trainable parameters than&#10;prior work. Compacter accomplishes this by building on top of ideas from&#10;adapters, low-rank optimization, and parameterized hypercomplex multiplication&#10;layers. Specifically, Compacter inserts task-specific weight matrices into a&#10;pretrained model's weights, which are computed efficiently as a sum of&#10;Kronecker products between shared &quot;slow&quot; weights and &quot;fast&quot; rank-one matrices&#10;defined per Compacter layer. By only training 0.047% of a pretrained model's&#10;parameters, Compacter performs on par with standard fine-tuning on GLUE and&#10;outperforms standard fine-tuning on SuperGLUE and low-resource settings. Our&#10;code is publicly available at~\url{https://github.com/rabeehk/compacter}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.11529" label="2302.11529">
        <attvalues>
          <attvalue for="0" value="Modular Deep Learning" />
          <attvalue for="1" value="  Transfer learning has recently become the dominant paradigm of machine&#10;learning. Pre-trained models fine-tuned for downstream tasks achieve better&#10;performance with fewer labelled examples. Nonetheless, it remains unclear how&#10;to develop models that specialise towards multiple tasks without incurring&#10;negative interference and that generalise systematically to non-identically&#10;distributed tasks. Modular deep learning has emerged as a promising solution to&#10;these challenges. In this framework, units of computation are often implemented&#10;as autonomous parameter-efficient modules. Information is conditionally routed&#10;to a subset of modules and subsequently aggregated. These properties enable&#10;positive transfer and systematic generalisation by separating computation from&#10;routing and updating modules locally. We offer a survey of modular&#10;architectures, providing a unified view over several threads of research that&#10;evolved independently in the scientific literature. Moreover, we explore&#10;various additional purposes of modularity, including scaling language models,&#10;causal inference, programme induction, and planning in reinforcement learning.&#10;Finally, we report various concrete applications where modularity has been&#10;successfully deployed such as cross-lingual and cross-modal knowledge transfer.&#10;Related talks and projects to this survey, are available at&#10;https://www.modulardeeplearning.com/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.03965" label="2011.03965">
        <attvalues>
          <attvalue for="0" value="On the Practical Ability of Recurrent Neural Networks to Recognize&#10;  Hierarchical Languages" />
          <attvalue for="1" value="  While recurrent models have been effective in NLP tasks, their performance on&#10;context-free languages (CFLs) has been found to be quite weak. Given that CFLs&#10;are believed to capture important phenomena such as hierarchical structure in&#10;natural languages, this discrepancy in performance calls for an explanation. We&#10;study the performance of recurrent models on Dyck-n languages, a particularly&#10;important and well-studied class of CFLs. We find that while recurrent models&#10;generalize nearly perfectly if the lengths of the training and test strings are&#10;from the same range, they perform poorly if the test strings are longer. At the&#10;same time, we observe that recurrent models are expressive enough to recognize&#10;Dyck words of arbitrary lengths in finite precision if their depths are&#10;bounded. Hence, we evaluate our models on samples generated from Dyck languages&#10;with bounded depth and find that they are indeed able to generalize to much&#10;higher lengths. Since natural language datasets have nested dependencies of&#10;bounded depth, this may help explain why they perform well in modeling&#10;hierarchical dependencies in natural language data despite prior works&#10;indicating poor generalization performance on Dyck languages. We perform&#10;probing studies to support our results and provide comparisons with&#10;Transformers.&#10;" />
          <attvalue for="2" value="&#10;\blfootnote{ This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details:.}&#10;&#10;Recurrent models (RNNs and more specifically LSTMs) have been used extensively across several NLP tasks such as machine translation \cite{sutskever2014sequence}, language modeling \cite{melis2017state} and question answering \cite{seo2016bidirectional}. Natural languages involve phenomena such as hierarchical and long-distance dependencies. Although RNNs are known to be Turing-complete \cite{siegelmann1992computational} given unbounded precision, their practical ability to model such phenomena remains unclear.&#10;&#10;Recently, several works \cite{weiss2018practical,sennhauser-berwick-2018-evaluating,skachkova-etal-2018-closing} have attempted to understand the capabilities of LSTMs by empirically analyzing them on different types of formal languages. Natural languages, for the most part, can be modeled by context-free languages \cite{Jaeger-Rogers} and their hierarchical structure has been emphasized by \newcite{chomsky2002syntactic}. Thus studying the capabilities of RNNs in recognizing context-free languages (CFLs) can shed light on how well they can model hierarchical structures. An important family of context-free languages is the Dyck-$n$ language.&#10;&#10;Previous works \cite{suzgun2019lstm,suzgun2019memory,yu-etal-2019-learning} showed that LSTMs achieve limited generalization performance on recognizing Dyck-$2$. This prompted the development of memory-augmented variants \cite{joulin2015inferring,suzgun2019memory} of LSTMs which generalize well on Dyck languages but are notoriously hard to train and fail to perform well on practical NLP tasks \cite{NIPS2019_8748}. On the other hand, despite the limited performance of LSTMs on Dyck languages, several studies \cite{gulordava-etal-2018-colorless,tran-etal-2018-importance} have found that LSTMs perform well in modeling hierarchical structure in natural language data. In this work, we take a step towards bridging this gap.&#10;&#10;Our Contributions. We investigate the ability of recurrent models to learn and generalize on Dyck languages. We first evaluate the ability of LSTMs to recognize randomly sampled Dyck-$n$ sequences and find, in contrast to prior works \cite{suzgun2019lstm,suzgun2019memory}, that they can generalize nearly perfectly when the test samples are within the same lengths as seen during training. Similar to prior works, when evaluated on randomly generated samples of higher lengths we observe limited performance. Dyck languages and (deterministic) CFLs can be recognized by (deterministic) pushdown automata (PDA). We construct an RNN that directly simulates a PDA given unbounded precision. A key observation is that the higher the depth of the stack the higher is the required precision. This implies that fixed precision RNNs are expressive enough to recognize strings of arbitrary lengths if the required depth of the stack is bounded. Based on this observation, we test the hypothesis whether LSTMs can generalize to higher lengths if the depth of the inputs in the training and test set is bounded by the same value. In the bounded depth setting, LSTMs are able to generalize to much higher lengths compared to the lengths used during training. Given that natural languages in practical settings also contain nested dependencies of bounded depths \cite{gibson1991computational,mcelree2001working}, this may help explain why LSTMs perform well in modeling natural language corpora containing nested dependencies. We then assess the generalization performance of the model across higher depths and find that although LSTMs can generalize to a certain extent, their performance degrades gradually with increasing depths. Since Transformer \cite{vaswani2017attention} is also a dominant model in NLP \cite{devlin-etal-2019-bert}, we include it in our experiments. To our knowledge, prior works have not empirically analyzed Transformers on formal languages, particularly context-free languages. We further conduct robustness experiments and probing studies to support our results.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Context-Free Language Modeling, Computer Science, Recurrent Model Limitations, Linguistics, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1707.05589" label="1707.05589">
        <attvalues>
          <attvalue for="0" value="On the State of the Art of Evaluation in Neural Language Models" />
          <attvalue for="1" value="  Ongoing innovations in recurrent neural network architectures have provided a&#10;steady influx of apparently state-of-the-art results on language modelling&#10;benchmarks. However, these have been evaluated using differing code bases and&#10;limited computational resources, which represent uncontrolled sources of&#10;experimental variation. We reevaluate several popular architectures and&#10;regularisation methods with large-scale automatic black-box hyperparameter&#10;tuning and arrive at the somewhat surprising conclusion that standard LSTM&#10;architectures, when properly regularised, outperform more recent models. We&#10;establish a new state of the art on the Penn Treebank and Wikitext-2 corpora,&#10;as well as strong baselines on the Hutter Prize dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.01603" label="1611.01603">
        <attvalues>
          <attvalue for="0" value="Bidirectional Attention Flow for Machine Comprehension" />
          <attvalue for="1" value="  Machine comprehension (MC), answering a query about a given context&#10;paragraph, requires modeling complex interactions between the context and the&#10;query. Recently, attention mechanisms have been successfully extended to MC.&#10;Typically these methods use attention to focus on a small portion of the&#10;context and summarize it with a fixed-size vector, couple attentions&#10;temporally, and/or often form a uni-directional attention. In this paper we&#10;introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage&#10;hierarchical process that represents the context at different levels of&#10;granularity and uses bi-directional attention flow mechanism to obtain a&#10;query-aware context representation without early summarization. Our&#10;experimental evaluations show that our model achieves the state-of-the-art&#10;results in Stanford Question Answering Dataset (SQuAD) and CNN/DailyMail cloze&#10;test.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.04908" label="1805.04908">
        <attvalues>
          <attvalue for="0" value="On the Practical Computational Power of Finite Precision RNNs for&#10;  Language Recognition" />
          <attvalue for="1" value="  While Recurrent Neural Networks (RNNs) are famously known to be Turing&#10;complete, this relies on infinite precision in the states and unbounded&#10;computation time. We consider the case of RNNs with finite precision whose&#10;computation time is linear in the input length. Under these limitations, we&#10;show that different RNN variants have different computational power. In&#10;particular, we show that the LSTM and the Elman-RNN with ReLU activation are&#10;strictly stronger than the RNN with a squashing activation and the GRU. This is&#10;achieved because LSTMs and ReLU-RNNs can easily implement counting behavior. We&#10;show empirically that the LSTM does indeed learn to effectively use the&#10;counting mechanism.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.03648" label="1906.03648">
        <attvalues>
          <attvalue for="0" value="LSTM Networks Can Perform Dynamic Counting" />
          <attvalue for="1" value="  In this paper, we systematically assess the ability of standard recurrent&#10;networks to perform dynamic counting and to encode hierarchical&#10;representations. All the neural models in our experiments are designed to be&#10;small-sized networks both to prevent them from memorizing the training sets and&#10;to visualize and interpret their behaviour at test time. Our results&#10;demonstrate that the Long Short-Term Memory (LSTM) networks can learn to&#10;recognize the well-balanced parenthesis language (Dyck-$1$) and the shuffles of&#10;multiple Dyck-$1$ languages, each defined over different parenthesis-pairs, by&#10;emulating simple real-time $k$-counter machines. To the best of our knowledge,&#10;this work is the first study to introduce the shuffle languages to analyze the&#10;computational power of neural networks. We also show that a single-layer LSTM&#10;with only one hidden unit is practically sufficient for recognizing the&#10;Dyck-$1$ language. However, none of our recurrent networks was able to yield a&#10;good performance on the Dyck-$2$ language learning task, which requires a model&#10;to have a stack-like mechanism for recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03329" label="1911.03329">
        <attvalues>
          <attvalue for="0" value="Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck&#10;  Languages" />
          <attvalue for="1" value="  We introduce three memory-augmented Recurrent Neural Networks (MARNNs) and&#10;explore their capabilities on a series of simple language modeling tasks whose&#10;solutions require stack-based mechanisms. We provide the first demonstration of&#10;neural networks recognizing the generalized Dyck languages, which express the&#10;core of what it means to be a language with hierarchical structure. Our&#10;memory-augmented architectures are easy to train in an end-to-end fashion and&#10;can learn the Dyck languages over as many as six parenthesis-pairs, in addition&#10;to two deterministic palindrome languages and the string-reversal transduction&#10;task, by emulating pushdown automata. Our experiments highlight the increased&#10;modeling capacity of memory-augmented models over simple RNNs, while inflecting&#10;our understanding of the limitations of these models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.05395" label="2206.05395">
        <attvalues>
          <attvalue for="0" value="Why is constrained neural language generation particularly challenging?" />
          <attvalue for="1" value="  Recent advances in deep neural language models combined with the capacity of&#10;large scale datasets have accelerated the development of natural language&#10;generation systems that produce fluent and coherent texts (to various degrees&#10;of success) in a multitude of tasks and application contexts. However,&#10;controlling the output of these models for desired user and task needs is still&#10;an open challenge. This is crucial not only to customizing the content and&#10;style of the generated language, but also to their safe and reliable deployment&#10;in the real world. We present an extensive survey on the emerging topic of&#10;constrained neural language generation in which we formally define and&#10;categorize the problems of natural language generation by distinguishing&#10;between conditions and constraints (the latter being testable conditions on the&#10;output text instead of the input), present constrained text generation tasks,&#10;and review existing methods and evaluation metrics for constrained text&#10;generation. Our aim is to highlight recent progress and trends in this emerging&#10;field, informing on the most promising directions and limitations towards&#10;advancing the state-of-the-art of constrained neural language generation&#10;research.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advances in the field of natural language generation (NLG) \cite{gatt2018survey} have resulted in models able to produce realistic, coherent, and fluent texts in a multitude of natural language processing tasks. Powerful large scale language models can be readily used to perform unconditional language generation, however these models provide little control over attributes of the generated texts. Unlike conventional methods which were able to provide fine-grained control over many aspects of the system output including incorporating domain-specific dictionaries, terminology or certain words in the generated output, neural end-to-end approaches remove many of these knobs and switches \cite{post2018fast}. However, imposing constraints on the output generated by these models is crucial for achieving useful and safe language generation in a multitude of real world application scenarios. For example, it can help avoid generic and meaningless responses in dialogue systems \cite{see2019makes}, personalize dialogue agents based on user features that lead to more engaging and meaningful conversations \cite{zhang2018personalizing}, ensure non-offensive sentence completion and friendly communication \cite{liu2019rhetorically}, intervene on the system output in interactive scenarios where domain specific terminology must be included in the generated texts \cite{crego2016systran}, or aid in art creation in applications such as poetry generation or assisted story writing \cite{peng2018towards}. Moreover, controlling a generic pretrained language model in&#10;order to satisfy certain desiderata helps avoid generating toxic content, prevents demographic biases, can steer generations towards desired a topic or style \cite{khalifa2020distributional}, and helps communicate intentions in suitable manners for different situations, target audiences and environments \cite{lample2018multiple}, \cite{li2018delete}. Incorporating prior knowledge and target side constraints in text generative models has numerous applications in many natural language processing areas, including dialogue systems, machine translation, question answering, text summarization, text simplification, image captioning, etc. Unquestionably, constrained text generation is important in many real-world applications, but compared to other instances of natural language generation, constrained text generation using neural networks remains an open challenge.&#10;&#10;We identify the following reasons that explain why constrained neural text generation represents a much harder problem compared to other instances of neural text generation: i) lack of model expressiveness: current models are not expressive enough to incorporate arbitrary constraints, defined as testable conditions on the output text, into the objective function at training time; ii) lack of suitable evaluation metrics: while one can verify whether an output satisfies a constraint or not, it is usually hard to measure to what extent an output satisfies a constraint, and it is even harder to jointly evaluate this with other properties of the generated text (such as relevance or coherence); iii) difficulty in constrained optimization: even if constraints can be expressed and added to the objective function, they are usually non-differentiable, especially at the token level. This is bad as most methods model and generate text as a sequence of tokens; &#10;iv) lack of constrained text generation datasets that are diverse and representative enough of the variety of practical constraints. &#10;&#10;For example, commonly used sequential text generation methods and architectures assume a rigid modeling of the output sequence based on an ordering of words, in which tokens are generated progressively one at a time in a standard left-to-right manner \cite{chan2019kermit}. Such autoregressive models cannot easily express constraints at arbitrary positions in the generated sequence or satisfy constraints involving multiple input objects. In addition to these issues, it is generally more challenging to incorporate multiple and heterogeneous constraints, which conform to given rules, topics, sentiments, lexical constraints, or pre-defined stylistic and content attributes. &#10;&#10;Our work focuses on the emerging problem of neural natural language generation with constraints. We first define the problem and differentiate between the ambiguous use of conditions and constraints in natural language generation, including examples that represent instantiations of the constrained neural text generation problem. We then survey approaches, learning methodologies and model architectures employed for generating texts with desirable attributes, and corresponding evaluation metrics. We conclude with open research problems and limitations of current models. The scope of our work is draw clear boundaries between the confusing terminology used in the neural language generation literature, highlight the main approaches and discuss how they suffer from the general challenges of constrained text generation, and serve as an informative guide and an advocate for solving these general challenges and advancing meaningful, useful, and safe constrained NLG research. &#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Constrained Text Generation, Artificial Intelligence, Neural Language Models, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1804.06609" label="1804.06609">
        <attvalues>
          <attvalue for="0" value="Fast Lexically Constrained Decoding with Dynamic Beam Allocation for&#10;  Neural Machine Translation" />
          <attvalue for="1" value="  The end-to-end nature of neural machine translation (NMT) removes many ways&#10;of manually guiding the translation process that were available in older&#10;paradigms. Recent work, however, has introduced a new capability: lexically&#10;constrained or guided decoding, a modification to beam search that forces the&#10;inclusion of pre-specified words and phrases in the output. However, while&#10;theoretically sound, existing approaches have computational complexities that&#10;are either linear (Hokamp and Liu, 2017) or exponential (Anderson et al., 2017)&#10;in the number of constraints. We present a algorithm for lexically constrained&#10;decoding with a complexity of O(1) in the number of constraints. We demonstrate&#10;the algorithms remarkable ability to properly place these constraints, and use&#10;it to explore the shaky relationship between model and BLEU scores. Our&#10;implementation is available as part of Sockeye.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.08654" label="1902.08654">
        <attvalues>
          <attvalue for="0" value="What makes a good conversation? How controllable attributes affect human&#10;  judgments" />
          <attvalue for="1" value="  A good conversation requires balance -- between simplicity and detail;&#10;staying on topic and changing it; asking questions and answering them. Although&#10;dialogue agents are commonly evaluated via human judgments of overall quality,&#10;the relationship between quality and these individual factors is less&#10;well-studied. In this work, we examine two controllable neural text generation&#10;methods, conditional training and weighted decoding, in order to control four&#10;important attributes for chitchat dialogue: repetition, specificity,&#10;response-relatedness and question-asking. We conduct a large-scale human&#10;evaluation to measure the effect of these control parameters on multi-turn&#10;interactive conversations on the PersonaChat task. We provide a detailed&#10;analysis of their relationship to high-level aspects of conversation, and show&#10;that by controlling combinations of these variables our models obtain clear&#10;improvements in human quality judgments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.05540" label="1610.05540">
        <attvalues>
          <attvalue for="0" value="SYSTRAN's Pure Neural Machine Translation Systems" />
          <attvalue for="1" value="  Since the first online demonstration of Neural Machine Translation (NMT) by&#10;LISA, NMT development has recently moved from laboratory to production systems&#10;as demonstrated by several entities announcing roll-out of NMT engines to&#10;replace their existing technologies. NMT systems have a large number of&#10;training configurations and the training process of such systems is usually&#10;very long, often a few weeks, so role of experimentation is critical and&#10;important to share. In this work, we present our approach to production-ready&#10;systems simultaneously with release of online demonstrators covering a large&#10;variety of languages (12 languages, for 32 language pairs). We explore&#10;different practical choices: an efficient and evolutive open-source framework;&#10;data preparation; network architecture; additional implemented features; tuning&#10;for production; etc. We discuss about evaluation methodology, present our first&#10;findings and we finally outline further work.&#10;  Our ultimate goal is to share our expertise to build competitive production&#10;systems for &quot;generic&quot; translation. We aim at contributing to set up a&#10;collaborative framework to speed-up adoption of the technology, foster further&#10;research efforts and enable the delivery and adoption to/by industry of&#10;use-case specific engines integrated in real production workflows. Mastering of&#10;the technology would allow us to build translation engines suited for&#10;particular needs, outperforming current simplest/uniform systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01604" label="1906.01604">
        <attvalues>
          <attvalue for="0" value="KERMIT: Generative Insertion-Based Modeling for Sequences" />
          <attvalue for="1" value="  We present KERMIT, a simple insertion-based approach to generative modeling&#10;for sequences and sequence pairs. KERMIT models the joint distribution and its&#10;decompositions (i.e., marginals and conditionals) using a single neural network&#10;and, unlike much prior work, does not rely on a prespecified factorization of&#10;the data distribution. During training, one can feed KERMIT paired data $(x,&#10;y)$ to learn the joint distribution $p(x, y)$, and optionally mix in unpaired&#10;data $x$ or $y$ to refine the marginals $p(x)$ or $p(y)$. During inference, we&#10;have access to the conditionals $p(x \mid y)$ and $p(y \mid x)$ in both&#10;directions. We can also sample from the joint distribution or the marginals.&#10;The model supports both serial fully autoregressive decoding and parallel&#10;partially autoregressive decoding, with the latter exhibiting an empirically&#10;logarithmic runtime. We demonstrate through experiments in machine translation,&#10;representation learning, and zero-shot cloze question answering that our&#10;unified approach is capable of matching or exceeding the performance of&#10;dedicated state-of-the-art systems across a wide range of tasks without the&#10;need for problem-specific architectural adaptation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.10348" label="2002.10348">
        <attvalues>
          <attvalue for="0" value="Low-Resource Knowledge-Grounded Dialogue Generation" />
          <attvalue for="1" value="  Responding with knowledge has been recognized as an important capability for&#10;an intelligent conversational agent. Yet knowledge-grounded dialogues, as&#10;training data for learning such a response generation model, are difficult to&#10;obtain. Motivated by the challenge in practice, we consider knowledge-grounded&#10;dialogue generation under a natural assumption that only limited training&#10;examples are available. In such a low-resource setting, we devise a&#10;disentangled response decoder in order to isolate parameters that depend on&#10;knowledge-grounded dialogues from the entire generation model. By this means,&#10;the major part of the model can be learned from a large number of ungrounded&#10;dialogues and unstructured documents, while the remaining small parameters can&#10;be well fitted using the limited training examples. Evaluation results on two&#10;benchmarks indicate that with only 1/8 training data, our model can achieve the&#10;state-of-the-art performance and generalize well on out-of-domain knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.10092" label="2308.10092">
        <attvalues>
          <attvalue for="0" value="Open, Closed, or Small Language Models for Text Classification?" />
          <attvalue for="1" value="  Recent advancements in large language models have demonstrated remarkable&#10;capabilities across various NLP tasks. But many questions remain, including&#10;whether open-source models match closed ones, why these models excel or&#10;struggle with certain tasks, and what types of practical procedures can improve&#10;performance. We address these questions in the context of classification by&#10;evaluating three classes of models using eight datasets across three distinct&#10;tasks: named entity recognition, political party prediction, and misinformation&#10;detection. While larger LLMs often lead to improved performance, open-source&#10;models can rival their closed-source counterparts by fine-tuning. Moreover,&#10;supervised smaller models, like RoBERTa, can achieve similar or even greater&#10;performance in many datasets compared to generative LLMs. On the other hand,&#10;closed models maintain an advantage in hard tasks that demand the most&#10;generalizability. This study underscores the importance of model selection&#10;based on task requirements&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;The hype around generative LLMs such as GPT-3, ChatGPT, and GPT-4 has grown recently over the results shown in the benchmark datasets, such as Question-Answering, Commonsense Reasoning, and Reading Comprehension. In turn, many researchers have begun to investigate these LLMs' performance on other NLP tasks specific to their domains, contrasting them with established BERT-like models \cite{ye2023comprehensive}. Below, we review their overall capabilities in classification, with a particular focus on Named Entity Recognition (NER), Political Ideology Prediction, and Misinformation Detection, along with a discussion of the limitations of closed-source models.&#10;&#10;LLMs for classification&#10;&#10;Text classification has significantly evolved over time. Starting from rule-based methods and regexes, NLP later shifted towards classical machine learning methods and then towards deep neural networks. Today, NLP has entered the era of transformer-based models, transitioning from fine-tuning Pre-trained Language Models (PLMs) such as RoBERTa \cite{Kenton2019} to the recent advancements in generative LLMs that require prompt engineering. RoBERTa is an encoder-based model that is pre-trained on the masked language modeling (MLM) task of predicting hidden words in a sentence. In contrast, GPT---Generative Pretrained Transformer---is designed for predicting the next token \cite{Radford2019, Brown2020}. Classification for RoBERTa can be done through feeding the embeddings (last hidden layer) taken from the ``[CLS]'' token and feeding them through a linear layer \cite{Sun2019}.&#10;&#10;Generative LLMs show strong comprehension abilities \cite{Liu2023} to human commands, especially after Reinforcement Learning from Human Feedback (RLHF) \cite{Ouyang2022}. Because of this high level of comprehension, these new types of models have led to the creation of a new field called ``prompt engineering'' \cite{AlKhamissi2022} with both manual and automatic prompts to improve task-specific performance \cite{Shin2020}. At the same time, the significance of prompts was exemplified in studies like Chain of Thought \cite{Wei2022} and Tree of Thought \cite{Yao2023}, which can significantly increase performance by suggesting patterns of reasoning for the LLM. Despite the challenge of slight prompt variations causing significant output differences, Instructed GPT, multi-round chat fine-tuning with human feedback aligning \cite{Rafailov2023,Ouyang2022} have attempted to bridge this gap by aligning models with human language patterns. Leveraging the exponential capacity of linearly scaled parameters, LLMs showcased their prowess in complex tasks, including classification. However, many have criticized the tendencies of these models to hallucinate. Guard rails---established for safety and ethical reasons and put in place in the system prompt during RLHF---can be tricked and even jailbroken. Although initial research demonstrated GPT2's classification capabilities with additional classifiers like BERT models, recent focus has shifted towards larger language models with prompt engineering, often leaving classification performance under-reported. &#10;&#10;Another common criticism of LLMs is that their pretraining is resource intensive and almost impossible to execute for most companies and labs. Through Meta's release of Llama \cite{Touvron2023a} and Llama 2 \cite{Touvron2023} to the open-source community, researchers have access to pretrained LLMs to explore how different LLMs perform in various contexts. While smaller, Llama 2 does boast similar capabilities compared to the commercial closed-sourced models of GPT-3.5 and GPT-4, but are still lacking in many areas.&#10;Recently, many researchers have been focused on fine-tuning and inference for these LLMs in low-resource environments and have proposed methods such as fp16, 8bit-Quantized, LoRA \cite{Hu2021} and QLoRA \cite{Dettmers2023}. With these methods, we can dramatically reduce the compute needed to fine-tune open-source pretrained LLMs on various downstream tasks and inject more domain-specific knowledge. In particular, we implement LoRA fine-tuning for Llama 2 on the NER task. &#10;&#10;Named Entity Recognition&#10;&#10;Named Entity Recognition (NER) remains a fundamental task in NLP, essential for transforming unstructured text into structured data. This extracted information enhances interpretability in various contexts and feeds into downstream models like Graph Neural Networks (GNNs).&#10;&#10;Like other NLP tasks, prevailing NER methods utilize PLMs such as RoBERTa. These models initially extract contextual representations (last hidden outputs of tokens) and employ sub-modules like MLPs \cite{yadav-bethard-2018-survey}, BiLSTMs \cite{Graves2013}, CRF \cite{Souza2019a}, and Global Pointers \cite{Su2022} to aid in entity extraction, and thereby boosting overall performance. Regardless of the method employed, effective NER using these structures depends on well-annotated datasets for effective fine-tuning to achieve optimal performance.&#10;&#10;Recent studies by \cite{Li2023,Wang2023a} have shown promising results when incorporating GPT-3.5 for NER. These models were prompted to generate specific entities in desired formats, and this was followed by straightforward post-processing. Furthermore, they demonstrated that robust zero-shot and few-shot capabilities significantly improved performance, even in contexts with limited resources. Based on these findings, we propose adopting a similar approach in several experiments presented in the second part of the paper. &#10;&#10;Political Ideology Prediction&#10;&#10;Political ideology prediction is the first of many steps computational scientists conduct to analyze partisan discourse or polarization. This task is typically framed as predicting the party or ideology of social media users. The literature references a wide variety of features ranging from textual content \cite{conover2011predicting, rodriguez2022urjc, mou2021align, fagni2022fine}, various types of network information \cite{Barber2015TweetingFL, colleoni2014echo, pennacchiotti2011machine, gu2016ideology, xiao2020timme, havey2020partisan, wojcieszak2022most, jiang2021social} to other features like ideology of well-known media outlets from which users share stories \cite{rheault2021efficient, luceri2019red, stefanov2020predicting, badawy2018analyzing}. The Authors' forthcoming ICWSM paper (anonymized) provides a comparative survey and empirical analysis of various domain-specialized and non-generative approaches, showing that RoBERTa achieves strong performance equal or superior to specialized models in this particular task.&#10;&#10;Previously, human labels have been the gold standard unless self-declared labels are available (e.g., politicians or users responding to a survey). But \cite{tornberg2023chatgpt} showed that GPT-4 performed better than human annotators, even experts, in determining the party affiliation of politicians from their messages on social media. While a strong result, this leaves open questions that motivate our work here. First, prior research shows that politician behavior can be different and sometimes easier to predict than the general public \cite{cohen2013classifying}. Second, \cite{tornberg2023chatgpt} focused on the United States, which has a two-party system. Identifying party ideology in a multiparty system might be more challenging. Finally, this task is generally used as a foundation for downstream research, and cost may be an issue, as well as other concerns like models changing over time. Here, we address these questions by testing approaches to classifying the general public in both the US (two-party) and Canada (multi-party).&#10;&#10;Misinformation Detection&#10;Misinformation is a critical societal challenge to which a great deal of research has been devoted \cite{shu2017fake,kumar2021battling,shahid2022detecting}. One of the main tools aimed at countering the spread of fake news is algorithmic detection, usually framed as a classification problem (e.g., labeling information as ``True'' or ``False'') \cite{shu2017fake}. While there are many approaches based on network information or user profiling, textual content is central, and often the only way a prediction could be made with certainty given that content is what actually determines veracity.&#10;&#10;Older approaches such as SVM, CNN, LSTM, etc. were once prevalent \cite{shu2017fake}, but transformer-based language models such as BERT have generally been shown to provide superior performance in detecting misinformation \cite{pelrine2021surprising, kaliyar2021fakebert}. More recently, GPT-4 gave even stronger performance and other benefits like better generalization and uncertainty quantification \cite{pelrine2023reliable}. However, with massive amounts of potential misinformation created every day, scalability remains a key challenge, and GPT-4 is expensive and strictly rate-limited. To our knowledge, it has not yet been determined if recent scalable open-source generative LLMs---like Llama 2---are effective in this domain. Testing these newer models is also one of our objective. &#10;&#10;Limitations of Close-Sourced Models&#10;&#10;Closed-source models such as GPT-3.5 and GPT-4 boast impressive performance across various NLP tasks; however, they are accompanied by several limitations. Typically, these models are accessed through APIs, relieving users of computing infrastructure concerns. Although they are user-friendly, cloud based AI services lack control over training data and model versioning. The undisclosed nature of the training corpus makes it challenging to determine whether a model's success on benchmark datasets is due to effective generalization or potential data leakage. Moreover, reproducing research conducted on closed-source models proves difficult due to the high cost associated with running experiments via APIs (considering GPT-3.5 \&amp; GPT-4 costs) and unanticipated model updates \cite{pozzobon2023challenges}, which can lead to fluctuating performance \cite{chen2023chatgpts}. In addition, many of these closed-source models incorporate interactions with APIs into their subsequent model's training dataset, raising ethical and privacy concerns. Finally, the significant energy consumption required to train and run these LLMs has a substantial environmental impact, making their use a concern for sustainable practices.&#10;&#10;" />
          <attvalue for="4" value="Model Selection Strategies, NLP Task Performance, Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Information Science" />
        </attvalues>
      </node>
      <node id="2404.04212" label="2404.04212">
        <attvalues>
          <attvalue for="0" value="Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language&#10;  Translation" />
          <attvalue for="1" value="  Parameter-efficient fine-tuning (PEFT) methods are increasingly vital in&#10;adapting large-scale pre-trained language models for diverse tasks, offering a&#10;balance between adaptability and computational efficiency. They are important&#10;in Low-Resource Language (LRL) Neural Machine Translation (NMT) to enhance&#10;translation accuracy with minimal resources. However, their practical&#10;effectiveness varies significantly across different languages. We conducted&#10;comprehensive empirical experiments with varying LRL domains and sizes to&#10;evaluate the performance of 8 PEFT methods with in total of 15 architectures&#10;using the SacreBLEU score. We showed that 6 PEFT architectures outperform the&#10;baseline for both in-domain and out-domain tests and the Houlsby+Inversion&#10;adapter has the best performance overall, proving the effectiveness of PEFT&#10;methods.&#10;" />
          <attvalue for="2" value="&#10;Advances in large-scale pre-trained language models have transformed the field for high-resource languages \cite{min2023recent}, but these data and compute-hungry models are not viable for the more-than-7000 low-resource languages (LRLs) in the world \cite{stap-araabi-2023-chatgpt, robinson-etal-2023-chatgpt, zhang-etal-2023-dont}.&#10;Ideal for the limitations of LRLs, parameter-efficient fine-tuning (PEFT) methods \cite{Houlsby2019parameter, pfeiffer2020mad, hu2021lora} are designed to strategically update a small number of parameters within a pre-trained model to be more efficient and adaptable without retraining the entire model. Their architecture significantly saves computational resources and storage space while achieving results comparable to full fine-tuning in downstream tasks \cite{ruder-etal-2022-modular}. \cite{ustun2022does} examined the applicability of 4 PEFT methods specifically in the context of language translation. &#10;Moreover, it did not address truly LRLs \cite{ustun2022does}, nor did it incorporate variation in domains that would allow for an assessment of the models' generalization capabilities.&#10;&#10;As a result, while the PEFT methods have shown potential in fine-tuning specific tasks, domains, and languages, the effectiveness of this collection of PEFT methods for LRL translation has not been systematically examined. In this paper, we explore the performance of different PEFT architectures in the LRL Neural Machine translation (NMT) by comparing in-domain and out-of-domain test results, as well as training times. We also investigate the effectiveness of PEFT methods in translating LRLs, focusing specifically on their architectures and performance across various datasets.&#10;&#10;The contributions of our paper are 1) comprehensive experimentation of PEFT architectures to reveal the suitability of translating non-Latin scripts and LRL pairs; 2) an in-depth assessment of 15 PEFT architectures using 8 distinct methods to evaluate their effectiveness in LRL translation; and 3) a systematic exploration of experimental settings, including variations in dataset domains and sizes, aimed at enhancing model generalization capabilities.&#10;As the field continues to advance rapidly, these PEFT guidelines provide practical recommendations for improving LRL translations, thus narrowing the language gap. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Fine-Tuning Methods, Computer Science, Linguistics, Mathematics, Neural Machine Translation, Statistics, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2205.11277" label="2205.11277">
        <attvalues>
          <attvalue for="0" value="When does Parameter-Efficient Transfer Learning Work for Machine&#10;  Translation?" />
          <attvalue for="1" value="  Parameter-efficient fine-tuning methods (PEFTs) offer the promise of adapting&#10;large pre-trained models while only tuning a small number of parameters. They&#10;have been shown to be competitive with full model fine-tuning for many&#10;downstream tasks. However, prior work indicates that PEFTs may not work as well&#10;for machine translation (MT), and there is no comprehensive study showing when&#10;PEFTs work for MT. We conduct a comprehensive empirical study of PEFTs for MT,&#10;considering (1) various parameter budgets, (2) a diverse set of language-pairs,&#10;and (3) different pre-trained models. We find that 'adapters', in which small&#10;feed-forward networks are added after every layer, are indeed on par with full&#10;model fine-tuning when the parameter budget corresponds to 10% of total model&#10;parameters. Nevertheless, as the number of tuned parameters decreases, the&#10;performance of PEFTs decreases. The magnitude of this decrease depends on the&#10;language pair, with PEFTs particularly struggling for distantly related&#10;language-pairs. We find that using PEFTs with a larger pre-trained model&#10;outperforms full fine-tuning with a smaller model, and for smaller training&#10;data sizes, PEFTs outperform full fine-tuning for the same pre-trained model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.14623" label="2206.14623">
        <attvalues>
          <attvalue for="0" value="Contextual Density Ratio for Language Model Biasing of Sequence to&#10;  Sequence ASR Systems" />
          <attvalue for="1" value="  End-2-end (E2E) models have become increasingly popular in some ASR tasks&#10;because of their performance and advantages. These E2E models directly&#10;approximate the posterior distribution of tokens given the acoustic inputs.&#10;Consequently, the E2E systems implicitly define a language model (LM) over the&#10;output tokens, which makes the exploitation of independently trained language&#10;models less straightforward than in conventional ASR systems. This makes it&#10;difficult to dynamically adapt E2E ASR system to contextual profiles for better&#10;recognizing special words such as named entities. In this work, we propose a&#10;contextual density ratio approach for both training a contextual aware E2E&#10;model and adapting the language model to named entities. We apply the&#10;aforementioned technique to an E2E ASR system, which transcribes doctor and&#10;patient conversations, for better adapting the E2E system to the names in the&#10;conversations. Our proposed technique achieves a relative improvement of up to&#10;46.5% on the names over an E2E baseline without degrading the overall&#10;recognition accuracy of the whole test set. Moreover, it also surpasses a&#10;contextual shallow fusion baseline by 22.1 % relative.&#10;" />
          <attvalue for="2" value="&#10; In recent years, End-to-end (E2E) systems~\cite{LAS,RNNT, NT} have been greatly adopted for ASR because of their performance and simplicity. &#10; In conventional ASR systems, several components, such as the pronunciation lexicon, the language models, and the acoustic model, are optimized independently.&#10; In contrast, E2E systems integrate all components of conventional ASR systems in a single neural network by directly approximating the posterior probability given the acoustic features.&#10;&#10;Despite the direct posterior approximation simplifying many aspects such as training, decoding and deployment, these advantages do not come without drawbacks. &#10; Specifically, one of the disadvantages is the difficulty of exploiting external language models.&#10; Since the E2E models directly approximate the posterior token distribution, there is no straightforward way to integrate them, which &#10; generates recognition problems for rare events among other problems.&#10; These tail probability events cannot be expected to be comprehensively observed in training.&#10;The selection of the output units such as word pieces~\cite{BPE} or characters instead of full words could theoretically mitigate these problems, but in practice this is not the case~\cite{ShallowContextual}.&#10;&#10;Several early attempts were proposed to integrate external language models (LM) into E2E systems, ranging from shallow or deep fusion~\cite{ShallowDeepFusion, ShallowASR}, to cold fusion~\cite{coldFusion} among many others~\cite{componentFusion,ShallowFusionRNNT}.&#10; In the ASR domain, \cite{ShallowASR} shown that shallow fusion with same output units in both LM and E2E worked best to incorporate an external language model in some tasks.&#10; However, one of the disadvantages of shallow fusion comes from the fact that most of the E2E systems~\cite{LAS,RNNT, LASFirst} arguably already incorporate an internal LM~\cite{StatelessRNNT,LessIsMore, HAT}.&#10; Recently, a density ratio approach~\cite{dr} was proposed to better integrate external LM by removing the internal LM contribution of E2E models while decoding.&#10; In this initial work~\cite{dr}, authors reported improvements over shallow fusion for domain adaptation tasks. Further works modified the architecture~\cite{HAT} or made some assumptions~\cite{Internal} to better approximate the implicit LM of E2E systems in combination with density ratio.&#10;&#10;In many tasks, ASR systems are expected to recognize singletons or infrequent words such as contact list names or other named entities.&#10; These entities may not be seen during training, and even for those which are observed, the training data is clearly neither representative of the particular distribution nor of the relevant set associated with the application context for a given utterance.&#10; Consequently, even if the E2E model is using BPE~\cite{BPE} or graphemenes as output units, the system struggles to recognize them. Several approaches have been introduced in the literature to contextualize E2E models to named entities such as song names or contact lists. In~\cite{CLAS} and~\cite{contextualRNNT}, an additional attention set input is proposed for both attention and recurrent neural transducer (RNN-T) models respectively. Shallow fusion approach~\cite{ShallowContextualBiasing} was applied surpassing state-of-the-art conventional models. More recently~\cite{ShallowContextual}, shallow fusion was used in combination with special tokens to delimit class-based entity names together with a mapping to transform rare words to common words through pronunciation. &#10;&#10;In this paper, we propose contextual density ratio for contextualizing E2E models so that the internal E2E LM is dynamically adapted to a priori known named entities.&#10; The proposed technique builds upon both density ratio~\cite{dr} and class-based LM tags~\cite{ShallowContextual} to contextualize the E2E models. &#10; During training, we introduce special tokens to enclose known named entities so that the E2E system learns to predict when a named entity is spoken &#10; by the different statistical clues obtained from both the acoustics and the internal LM. &#10; We also approximate an internal E2E language model by training an independent LM with the transcriptions on which the E2E system was trained.&#10; During decoding, we apply density ratio within the named entities segments identified by the E2E model to dynamically adapt and contextualize the system's LM.&#10; We show that the proposed contextual density ratio (CDR) reduces names recognition errors over a E2E baseline with and without contextual shallow fusion~\cite{ShallowContextualBiasing} by 46.5\% and 22.1 \% respectively.&#10;&#10;The paper is organized as follows. We first briefly review density ratio in section~\ref{sec:dr} to pave the way for the following section~\ref{sec:drt} where the proposed contextual density ratio is detailed. &#10; In section~\ref{sec:results}, we apply the proposed technique to a doctor-to-patient conversation task for better recognizing both doctors and patient names. &#10; Finally, in section~\ref{sec:conc}, we conclude with some reflections.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="End-to-End Modeling, Computer Science, Linguistics, Automatic Speech Recognition, Artificial Intelligence, Mathematics, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2007.05609" label="2007.05609">
        <attvalues>
          <attvalue for="0" value="Class LM and word mapping for contextual biasing in End-to-End ASR" />
          <attvalue for="1" value="  In recent years, all-neural, end-to-end (E2E) ASR systems gained rapid&#10;interest in the speech recognition community. They convert speech input to text&#10;units in a single trainable Neural Network model. In ASR, many utterances&#10;contain rich named entities. Such named entities may be user or location&#10;specific and they are not seen during training. A single model makes it&#10;inflexible to utilize dynamic contextual information during inference. In this&#10;paper, we propose to train a context aware E2E model and allow the beam search&#10;to traverse into the context FST during inference. We also propose a simple&#10;method to adjust the cost discrepancy between the context FST and the base&#10;model. This algorithm is able to reduce the named entity utterance WER by 57%&#10;with little accuracy degradation on regular utterances. Although an E2E model&#10;does not need pronunciation dictionary, it's interesting to make use of&#10;existing pronunciation knowledge to improve accuracy. In this paper, we propose&#10;an algorithm to map the rare entity words to common words via pronunciation and&#10;treat the mapped words as an alternative form to the original word during&#10;recognition. This algorithm further reduces the WER on the named entity&#10;utterances by another 31%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.06426" label="1708.06426">
        <attvalues>
          <attvalue for="0" value="Cold Fusion: Training Seq2Seq Models Together with Language Models" />
          <attvalue for="1" value="  Sequence-to-sequence (Seq2Seq) models with attention have excelled at tasks&#10;which involve generating natural language sentences such as machine&#10;translation, image captioning and speech recognition. Performance has further&#10;been improved by leveraging unlabeled data, often in the form of a language&#10;model. In this work, we present the Cold Fusion method, which leverages a&#10;pre-trained language model during training, and show its effectiveness on the&#10;speech recognition task. We show that Seq2Seq models with Cold Fusion are able&#10;to better utilize language information enjoying i) faster convergence and&#10;better generalization, and ii) almost complete transfer to a new domain while&#10;using less than 10% of the labeled training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13878" label="2010.13878">
        <attvalues>
          <attvalue for="0" value="Improved Neural Language Model Fusion for Streaming Recurrent Neural&#10;  Network Transducer" />
          <attvalue for="1" value="  Recurrent Neural Network Transducer (RNN-T), like most end-to-end speech&#10;recognition model architectures, has an implicit neural network language model&#10;(NNLM) and cannot easily leverage unpaired text data during training. Previous&#10;work has proposed various fusion methods to incorporate external NNLMs into&#10;end-to-end ASR to address this weakness. In this paper, we propose extensions&#10;to these techniques that allow RNN-T to exploit external NNLMs during both&#10;training and inference time, resulting in 13-18% relative Word Error Rate&#10;improvement on Librispeech compared to strong baselines. Furthermore, our&#10;methods do not incur extra algorithmic latency and allow for flexible&#10;plug-and-play of different NNLMs without re-training. We also share in-depth&#10;analysis to better understand the benefits of the different NNLM fusion&#10;methods. Our work provides a reliable technique for leveraging unpaired text&#10;data to significantly improve RNN-T while keeping the system streamable,&#10;flexible, and lightweight.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.06749" label="2012.06749">
        <attvalues>
          <attvalue for="0" value="Less Is More: Improved RNN-T Decoding Using Limited Label Context and&#10;  Path Merging" />
          <attvalue for="1" value="  End-to-end models that condition the output label sequence on all previously&#10;predicted labels have emerged as popular alternatives to conventional systems&#10;for automatic speech recognition (ASR). Since unique label histories correspond&#10;to distinct models states, such models are decoded using an approximate&#10;beam-search process which produces a tree of hypotheses.&#10;  In this work, we study the influence of the amount of label context on the&#10;model's accuracy, and its impact on the efficiency of the decoding process. We&#10;find that we can limit the context of the recurrent neural network transducer&#10;(RNN-T) during training to just four previous word-piece labels, without&#10;degrading word error rate (WER) relative to the full-context baseline. Limiting&#10;context also provides opportunities to improve the efficiency of the&#10;beam-search process during decoding by removing redundant paths from the active&#10;beam, and instead retaining them in the final lattice. This path-merging scheme&#10;can also be applied when decoding the baseline full-context model through an&#10;approximation. Overall, we find that the proposed path-merging scheme is&#10;extremely effective allowing us to improve oracle WERs by up to 36% over the&#10;baseline, while simultaneously reducing the number of model evaluations by up&#10;to 5.3% without any degradation in WER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.03411" label="2006.03411">
        <attvalues>
          <attvalue for="0" value="Contextual RNN-T For Open Domain ASR" />
          <attvalue for="1" value="  End-to-end (E2E) systems for automatic speech recognition (ASR), such as RNN&#10;Transducer (RNN-T) and Listen-Attend-Spell (LAS) blend the individual&#10;components of a traditional hybrid ASR system - acoustic model, language model,&#10;pronunciation model - into a single neural network. While this has some nice&#10;advantages, it limits the system to be trained using only paired audio and&#10;text. Because of this, E2E models tend to have difficulties with correctly&#10;recognizing rare words that are not frequently seen during training, such as&#10;entity names. In this paper, we propose modifications to the RNN-T model that&#10;allow the model to utilize additional metadata text with the objective of&#10;improving performance on these named entity words. We evaluate our approach on&#10;an in-house dataset sampled from de-identified public social media videos,&#10;which represent an open domain ASR task. By using an attention model and a&#10;biasing model to leverage the contextual metadata that accompanies a video, we&#10;observe a relative improvement of about 16% in Word Error Rate on Named&#10;Entities (WER-NE) for videos with related metadata.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.06361" label="1908.06361">
        <attvalues>
          <attvalue for="0" value="Understanding Undesirable Word Embedding Associations" />
          <attvalue for="1" value="  Word embeddings are often criticized for capturing undesirable word&#10;associations such as gender stereotypes. However, methods for measuring and&#10;removing such biases remain poorly understood. We show that for any embedding&#10;model that implicitly does matrix factorization, debiasing vectors post hoc&#10;using subspace projection (Bolukbasi et al., 2016) is, under certain&#10;conditions, equivalent to training on an unbiased corpus. We also prove that&#10;WEAT, the most common association test for word embeddings, systematically&#10;overestimates bias. Given that the subspace projection method is provably&#10;effective, we use it to derive a new measure of association called the&#10;$\textit{relational inner product association}$ (RIPA). Experiments with RIPA&#10;reveal that, on average, skipgram with negative sampling (SGNS) does not make&#10;most words any more gendered than they are in the training corpus. However, for&#10;gender-stereotyped words, SGNS actually amplifies the gender association in the&#10;corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Word Embeddings Word embedding models generate distributed representations of words in a low-dimensional continuous space. This is generally done using: (a) neural networks that learn embeddings by predicting the contexts words appear in, or vice-versa \cite{bengio2003neural,mikolov2013distributed,collobert2008unified}; (b) low-rank approximations of word-context matrices containing a co-occurrence statistic \cite{landauer1997solution,levy2014neural}. The objective of SGNS is to maximize the probability of observed word-context pairs and to minimize the probability of $k$ randomly sampled negative examples. Though no co-occurrence statistics are explicitly calculated, \cite{levy2014neural} proved that SGNS is implicitly factorizing a word-context PMI matrix shifted by $- \log k$. Similarly, GloVe implicitly factorizes a log co-occurrence count matrix \cite{pennington2014glove}.&#10;&#10;Word Analogies A word analogy a:b::x:y asserts that ``a is to b as x is to y'' and holds in the embedding space iff $\vec{a} + ( \vec{y} - \vec{x}) = \vec{b}$. \cite{ethayarajh2018towards} proved that for GloVe and SGNS, a:b::x:y holds exactly in an embedding space with no reconstruction error iff the words are coplanar and the co-occurrence shifted PMI is the same for each word pair and across both word pairs. Word analogies are often used to signify that semantic and syntactic properties of words (e.g., verb tense, gender) can be captured as linear relations.&#10;&#10;Measuring Associations \cite{caliskan2017semantics} proposed what is now the most commonly used association test for word embeddings. The word embedding association test (WEAT) uses cosine similarity to measure how associated two given sets of target words are with respect to two sets of attribute words (e.g., `male' vs.\ `female'). For example, \cite{caliskan2017semantics} claimed that science-related words are more associated with `male' than `female' attributes compared to art-related words, and that this was statistically significant. However, aside from some intuitive results (e.g., that female names are associated with female attributes), there is little evidence that WEAT is a good measure of association. &#10;&#10;Debiasing Embeddings \cite{bolukbasi2016man} claimed that the existence of stereotypical analogies such as doctor:nurse::man:woman constituted gender bias. To prevent such analogies from holding in the vector space, they subtracted from each biased word vector its projection on a ``gender bias subspace''. This subspace was defined by the first $m$ principal components for ten gender relation vectors (e.g., $\vec{man} - \vec{woman}$). Each debiased word vector was thus orthogonal to the gender bias subspace and its projection on the subspace was zero. While this subspace projection method precluded gender-biased analogies from holding in the embedding space, \cite{bolukbasi2016man} did not provide any theoretical guarantee that the vectors were unbiased (i.e., equivalent to vectors that would be obtained from training on a gender-agnostic corpus with no reconstruction error). Other work has tried to learn gender-neutral embeddings from scratch \cite{zhao2018learning}, despite this approach requiring custom changes to the objective of each embedding model.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Word Embeddings, Bias Measurement, Debiasing Methods, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2402.06592" label="2402.06592">
        <attvalues>
          <attvalue for="0" value="Self-consistent context aware conformer transducer for speech&#10;  recognition" />
          <attvalue for="1" value="  We propose a novel neural network architecture based on conformer transducer&#10;that adds contextual information flow to the ASR systems. Our method improves&#10;the accuracy of recognizing uncommon words while not harming the word error&#10;rate of regular words. We explore the uncommon words accuracy improvement when&#10;we use the new model and/or shallow fusion with context language model. We&#10;found that combination of both provides cumulative gain in uncommon words&#10;recognition accuracy.&#10;" />
          <attvalue for="2" value="&#10;End-to-end speech-to-text systems have emerged as a dominant technology in voice recognition, surpassing the performance of the previously prevalent multi-component HMM-DNN systems. Among the various model architectures that stand out are: Connectionist Temporal Classification \cite{Graves:06icml}, Listen-Attend-Spell \cite{LAS-2016}, Time-Depth Separable Convolutions \cite{hannun2019sequencetosequence}, Recurrent Neural Network Transducer \cite{graves-2012-sequen-trans}, Transformer \cite{Dong-no-recurrence-s2s-model-2018}, Conformer \cite{gulati2020conformer}, SqueezeFormer \cite{kim2022squeezeformer}, Fast Conformer\cite{rekesh2023fast}, and Zipformer \cite{yao2023zipformer}. These models all share an encoder-decoder architecture. &#10;One major limitation of E2E systems is their inability&#10;to accurately recognize words that are either absent or infrequently present in the training data, for example, entity names, person names, etc. To address this issue, two main strategies have been proposed: model-based and decoding-based approaches. The first approach modifies the model to integrate user-provided context words \cite{Pundak2018DeepCE,Chang2021ContextAwareTT, MysoreSathyendra2022,fu2023robust}, while the second approach applies an external context score using techniques like shallow fusion or on-the-fly re-scoring \cite{Zhao2019end2end,wang2023contextual}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Speech Recognition Systems, Linguistics, Language Model Integration, Artificial Intelligence, Signal Processing" />
        </attvalues>
      </node>
      <node id="1904.02619" label="1904.02619">
        <attvalues>
          <attvalue for="0" value="Sequence-to-Sequence Speech Recognition with Time-Depth Separable&#10;  Convolutions" />
          <attvalue for="1" value="  We propose a fully convolutional sequence-to-sequence encoder architecture&#10;with a simple and efficient decoder. Our model improves WER on LibriSpeech&#10;while being an order of magnitude more efficient than a strong RNN baseline.&#10;Key to our approach is a time-depth separable convolution block which&#10;dramatically reduces the number of parameters in the model while keeping the&#10;receptive field large. We also give a stable and efficient beam search&#10;inference procedure which allows us to effectively integrate a language model.&#10;Coupled with a convolutional language model, our time-depth separable&#10;convolution architecture improves by more than 22% relative WER over the best&#10;previously reported sequence-to-sequence results on the noisy LibriSpeech test&#10;set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.00888" label="2206.00888">
        <attvalues>
          <attvalue for="0" value="Squeezeformer: An Efficient Transformer for Automatic Speech Recognition" />
          <attvalue for="1" value="  The recently proposed Conformer model has become the de facto backbone model&#10;for various downstream speech tasks based on its hybrid attention-convolution&#10;architecture that captures both local and global features. However, through a&#10;series of systematic studies, we find that the Conformer architecture's design&#10;choices are not optimal. After re-examining the design choices for both the&#10;macro and micro-architecture of Conformer, we propose Squeezeformer which&#10;consistently outperforms the state-of-the-art ASR models under the same&#10;training schemes. In particular, for the macro-architecture, Squeezeformer&#10;incorporates (i) the Temporal U-Net structure which reduces the cost of the&#10;multi-head attention modules on long sequences, and (ii) a simpler block&#10;structure of multi-head attention or convolution modules followed up by&#10;feed-forward module instead of the Macaron structure proposed in Conformer.&#10;Furthermore, for the micro-architecture, Squeezeformer (i) simplifies the&#10;activations in the convolutional block, (ii) removes redundant Layer&#10;Normalization operations, and (iii) incorporates an efficient depthwise&#10;down-sampling layer to efficiently sub-sample the input signal. Squeezeformer&#10;achieves state-of-the-art results of 7.5%, 6.5%, and 6.0% word-error-rate (WER)&#10;on LibriSpeech test-other without external language models, which are 3.1%,&#10;1.4%, and 0.6% better than Conformer-CTC with the same number of FLOPs. Our&#10;code is open-sourced and available online.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.05084" label="2305.05084">
        <attvalues>
          <attvalue for="0" value="Fast Conformer with Linearly Scalable Attention for Efficient Speech&#10;  Recognition" />
          <attvalue for="1" value="  Conformer-based models have become the dominant end-to-end architecture for&#10;speech processing tasks. With the objective of enhancing the conformer&#10;architecture for efficient training and inference, we carefully redesigned&#10;Conformer with a novel downsampling schema. The proposed model, named Fast&#10;Conformer(FC), is 2.8x faster than the original Conformer, supports scaling to&#10;Billion parameters without any changes to the core architecture and also&#10;achieves state-of-the-art accuracy on Automatic Speech Recognition benchmarks.&#10;To enable transcription of long-form speech up to 11 hours, we replaced global&#10;attention with limited context attention post-training, while also improving&#10;accuracy through fine-tuning with the addition of a global token. Fast&#10;Conformer, when combined with a Transformer decoder also outperforms the&#10;original Conformer in accuracy and in speed for Speech Translation and Spoken&#10;Language Understanding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.03250" label="2111.03250">
        <attvalues>
          <attvalue for="0" value="Context-Aware Transformer Transducer for Speech Recognition" />
          <attvalue for="1" value="  End-to-end (E2E) automatic speech recognition (ASR) systems often have&#10;difficulty recognizing uncommon words, that appear infrequently in the training&#10;data. One promising method, to improve the recognition accuracy on such rare&#10;words, is to latch onto personalized/contextual information at inference. In&#10;this work, we present a novel context-aware transformer transducer (CATT)&#10;network that improves the state-of-the-art transformer-based ASR system by&#10;taking advantage of such contextual signals. Specifically, we propose a&#10;multi-head attention-based context-biasing network, which is jointly trained&#10;with the rest of the ASR sub-networks. We explore different techniques to&#10;encode contextual data and to create the final attention context vectors. We&#10;also leverage both BLSTM and pretrained BERT based models to encode contextual&#10;data and guide the network training. Using an in-house far-field dataset, we&#10;show that CATT, using a BERT based context encoder, improves the word error&#10;rate of the baseline transformer transducer and outperforms an existing deep&#10;contextual model by 24.2% and 19.4% respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.05271" label="2305.05271">
        <attvalues>
          <attvalue for="0" value="Robust Acoustic and Semantic Contextual Biasing in Neural Transducers&#10;  for Speech Recognition" />
          <attvalue for="1" value="  Attention-based contextual biasing approaches have shown significant&#10;improvements in the recognition of generic and/or personal rare-words in&#10;End-to-End Automatic Speech Recognition (E2E ASR) systems like neural&#10;transducers. These approaches employ cross-attention to bias the model towards&#10;specific contextual entities injected as bias-phrases to the model. Prior&#10;approaches typically relied on subword encoders for encoding the bias phrases.&#10;However, subword tokenizations are coarse and fail to capture granular&#10;pronunciation information which is crucial for biasing based on acoustic&#10;similarity. In this work, we propose to use lightweight character&#10;representations to encode fine-grained pronunciation features to improve&#10;contextual biasing guided by acoustic similarity between the audio and the&#10;contextual entities (termed acoustic biasing). We further integrate pretrained&#10;neural language model (NLM) based encoders to encode the utterance's semantic&#10;context along with contextual entities to perform biasing informed by the&#10;utterance's semantic context (termed semantic biasing). Experiments using a&#10;Conformer Transducer model on the Librispeech dataset show a 4.62% - 9.26%&#10;relative WER improvement on different biasing list sizes over the baseline&#10;contextual model when incorporating our proposed acoustic and semantic biasing&#10;approach. On a large-scale in-house dataset, we observe 7.91% relative WER&#10;improvement compared to our baseline model. On tail utterances, the&#10;improvements are even more pronounced with 36.80% and 23.40% relative WER&#10;improvements on Librispeech rare words and an in-house testset respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.00178" label="2310.00178">
        <attvalues>
          <attvalue for="0" value="Contextual Biasing with the Knuth-Morris-Pratt Matching Algorithm" />
          <attvalue for="1" value="  Contextual biasing refers to the problem of biasing the automatic speech&#10;recognition (ASR) systems towards rare entities that are relevant to the&#10;specific user or application scenarios. We propose algorithms for contextual&#10;biasing based on the Knuth-Morris-Pratt algorithm for pattern matching. During&#10;beam search, we boost the score of a token extension if it extends matching&#10;into a set of biasing phrases. Our method simulates the classical approaches&#10;often implemented in the weighted finite state transducer (WFST) framework, but&#10;avoids the FST language altogether, with careful considerations on memory&#10;footprint and efficiency on tensor processing units (TPUs) by vectorization.&#10;Without introducing additional model parameters, our method achieves&#10;significant word error rate (WER) reductions on biasing test sets by itself,&#10;and yields further performance gain when combined with a model-based biasing&#10;method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.13181" label="2405.13181">
        <attvalues>
          <attvalue for="0" value="Comparative Analysis of Different Efficient Fine Tuning Methods of Large&#10;  Language Models (LLMs) in Low-Resource Setting" />
          <attvalue for="1" value="  In the domain of large language models (LLMs), arXiv:2305.16938 showed that&#10;few-shot full-model fine-tuning -- namely Vanilla Fine Tuning (FT) and&#10;Pattern-Based Fine Tuning (PBFT) --, and In-Context Learning (ICL) generalize&#10;similarly on Out-Of-Domain (OOD) datasets, but vary in terms of task&#10;adaptation. However, they both pose challenges, especially in term of memory&#10;requirements. In this paper, we further try to push the understanding of&#10;different fine-tuning strategies for LLM and aim to bring a myriad of these on&#10;the same pedestal for an elaborate comparison with full-model fine-tuning on&#10;two diverse datasets. To that end, we conducted a series of experiments,&#10;beginning with state-of-the-art methods like vanilla fine-tuning and&#10;Pattern-Based Fine-Tuning (PBFT) on pre-trained models across two datasets,&#10;COLA and MNLI. We then investigate adaptive fine-tuning and the efficiency of&#10;LoRA adapters in a few-shot setting. Finally, we also compare an alternative&#10;approach that has gained recent popularity -- context distillation -- with the&#10;vanilla FT and PBFT with and without few-shot setup.&#10;  Our findings suggest that these alternative strategies that we explored can&#10;exhibit out-of-domain generalization comparable to that of vanilla FT and PBFT.&#10;PBFT under-performs Vanilla FT on out-of-domain (OOD) data, emphasizing the&#10;need for effective prompts. Further, our adaptive-fine tuning and LoRA&#10;experiments perform comparable or slightly worse than the standard fine-tunings&#10;as anticipated, since standard fine-tunings involve tuning the entire model.&#10;Finally, our context distillation experiments out-perform the standard&#10;fine-tuning methods. These findings underscore that eventually the choice of an&#10;appropriate fine-tuning method depends on the available resources (memory,&#10;compute, data) and task adaptability.&#10;" />
          <attvalue for="2" value="&#10; The goal of our work is to evaluate and compare the performance of a pre-trained large language model on sequence classification tasks. We aimed to do this by employing -- 1) different fine tuning (FT) methods, 2) applying Low-Rank Adaptation - LoRA (\cite{hu2021lora}) adaptors with few-shot learning, and 3) performing context-distillation both with and without few-shot learning setting. We aim to understand the efficacy of alternative fine-tuning methods on a pre-trained large language model’s performance in sequence classification tasks using 2 datasets, namely \href{ (\cite{N18-1101}) and \href{ (\cite{warstadt2018neural}), which are further explained in Section \ref{Datasets}. We explored alternate ways of efficiently fine-tuning the model and compared them with the baseline methods (vanilla and pattern-based fine-tuning) for Open Pre-trained Transformer (OPT) (\cite{zhang2022opt}) model's performance on the text sequence classification task using both in-domain and out of domain accuracies. We try to keep the training process, experiments, and hyper-parameters similar across various experiments, wherever possible, for a fair comparison. &#10;&#10;Currently large language models (LLMs) are pre-dominantly used by leveraging In-Context Learning - ICL (\cite{brown2020language}), whereby during the inference time, the model learns to answer follow-up questions from a series of prompts. This approach requires significant inference time memory and compute. In recent times, bunch of alternate methods have been proposed and explored to augment the issues faced with ICL. We explore and analyze a number of these methods in our work presented here.&#10;&#10;Our work is relevant to anyone leveraging large language models for tasks like chat bots and code completion. If successful, our methods could improve the efficiency and performance of these models, particularly in tasks requiring the ingestion of large sequences of dialogue, code, or text.&#10;&#10;We expand on all of these methods in section\ref{Approach} of this paper. We also further expand on the 2 datasets that we have performed our experiments on, in the section\ref{Datasets}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Fine-Tuning, Computer Science, Linguistics, Cognitive Science, Model Efficiency Optimization, Artificial Intelligence, Task Adaptation Strategies" />
        </attvalues>
      </node>
      <node id="1805.12471" label="1805.12471">
        <attvalues>
          <attvalue for="0" value="Neural Network Acceptability Judgments" />
          <attvalue for="1" value="  This paper investigates the ability of artificial neural networks to judge&#10;the grammatical acceptability of a sentence, with the goal of testing their&#10;linguistic competence. We introduce the Corpus of Linguistic Acceptability&#10;(CoLA), a set of 10,657 English sentences labeled as grammatical or&#10;ungrammatical from published linguistics literature. As baselines, we train&#10;several recurrent neural network models on acceptability classification, and&#10;find that our models outperform unsupervised models by Lau et al (2016) on&#10;CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et&#10;al.'s models and ours learn systematic generalizations like subject-verb-object&#10;order. However, all models we test perform far below human level on a wide&#10;range of grammatical constructions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.13673" label="2210.13673">
        <attvalues>
          <attvalue for="0" value="Evaluating Parameter Efficient Learning for Generation" />
          <attvalue for="1" value="  Parameter efficient learning methods (PERMs) have recently gained significant&#10;attention as they provide an efficient way for pre-trained language models&#10;(PLMs) to adapt to a downstream task. However, these conclusions are mostly&#10;drawn from in-domain evaluations over the full training set. In this paper, we&#10;present comparisons between PERMs and finetuning from three new perspectives:&#10;(1) the effect of sample and model size to in-domain evaluations, (2)&#10;generalization to unseen domains and new datasets, and (3) the faithfulness of&#10;generations. Our results show that for in-domain settings (a) there is a cross&#10;point of sample size for which PERMs will perform better than finetuning when&#10;training with fewer samples, and (b) larger PLMs have larger cross points. For&#10;cross-domain and cross-dataset cases, we show that (a) Adapter (Houlsby et al.,&#10;2019) performs the best amongst all the PERMs studied here, and (b) it&#10;outperforms finetuning if the task dataset is below a certain size. We also&#10;compare the faithfulness of generations and show that PERMs can achieve better&#10;faithfulness score than finetuning, especially for small training set, by as&#10;much as 6%. Finally, we apply Adapter to MT-NLG 530b (Smith et al., 2022) and&#10;achieve new state-of-the-art results on Xsum (Narayan et al., 2018) for all&#10;ROUGE scores (ROUGE-1 49.17, ROUGE-2 27.20, ROUGE-L 40.98).&#10;" />
          <attvalue for="2" value="&#10;&#10;Parameter efficient learning methods ({{PERMs}}) serve as potential alternatives to finetuning for adapting and deploying language models in real world scenarios \cite{ding2022delta}. They allow users to finetune only a small number of parameters while freezing the rest of the shared parameters of pre-trained language models (PLMs). This is especially important for large language models (e.g. GPT-3 \cite{brown2020language} and MT-NLG \cite{smith2022using}) &#10;as finetuning the entire model will be very expensive or infeasible due to their model size.&#10;&#10;Prefix tuning \cite{li2021prefix}, which is one of the {{PERMs}}, draws inspiration from prompting and introduces a small set of continuous vectors as virtual prompts to allow subsequent tokens to attend to, which obtains comparable performance to finetuning in the full data setting. Prompt tuning \cite{lester2021power} shows the power of scaling PLMs and that tuning only a few extra embeddings is sufficient to achieve similar performance to finetuning the entire 11b T5-XXL \cite{raffel2020exploring} model. &#10;P-tuning v2 \cite{liu2022p} further demonstrates that small PLMs can also achieve comparable results to finetuning with Prefix tuning. Different from adding new parameters through prompts, Adapter \cite{houlsby2019parameter} injects trainable parameters through low-rank structure in a skip-connection way. Other {{PERMs}} includes LoRA \cite{hu2021lora}, Mix-And-Match adapter \cite{he2021towards}, Compactor \cite{karimi2021compacter}, BitFit \cite{zaken2022bitfit}, diff-pruning \cite{guo2021parameter} and etc. &#10;&#10;Most conclusions about {{PERMs}} so far are drawn from their in-domain evaluations over full training samples. To the best of our knowledge, it is not yet investigated (1) how these conclusions apply to different training sizes and model sizes, and (2) how {{PERMs}} generalize to unseen domains and new datasets, which are both important aspects for deploying {{PERMs}} in real-world applications. &#10;&#10;In addition, faithfulness in natural language generation has become an important topic as it is vital to real-world applications. Various efforts are made to systematically measure and mitigate factual errors in many generation tasks, including summarization~\cite{huang2021factual} and dialogue generations~\cite{rashkin2021increasing,shuster2021retrieval,dziri2021neural, wu2021controllable}. However, existing work on faithfulness only focuses on faithfulness of finetuning, and the impact of {{PERMs}} on the faithfulness of generation is not yet explored.&#10;&#10;In this paper, we provide an in-depth study of {{PERMs}} for generation tasks through three important aspects when deploying {{PERMs}} in practical applications: (1) in-domain evaluation by scaling both training dataset size and model size of PLMs, (2) cross-domain and cross-dataset generalization, and (3) faithfulness assessment. Two generation tasks are used for evaluation: summarization and dialogue generation. We study four representative methods: P-tuning, Prompt tuning, Prefix tuning, and Adapter, but mainly focus on Prefix tuning and Adapter as our preliminary results show that they are better than the others. Our contributions are summarized as follows: (1) To the best of our knowledge, we present the first comparisons of faithfulness for {{PERMs}}. Our experimental results show that {{PERMs}}, especially prefix tuning can achieve better faithfulness than finetuning by up to 6\%. (2) For in-domain settings, there is always a cross point of sample size for which {{PERMs}} will be better than finetuning when training on fewer samples. Larger PLMs have larger cross points. Users need to choose which method to use based on their own training sample size and model size. (3) Compared to finetuning, not all {{PERMs}} can easily achieve better cross-domain and cross-dataset scores than finetuning even with 8.3b PLM. Our results show that Adapter is a better method than Prefix tuning on 13 out of 15 comparison settings. (4) New state-of-the-art results on Xsum \cite{narayan2018don} are obtained by applying Adapter to MT-NLG 530b model. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Model Evaluation Metrics, Linguistics, Cognitive Science, Domain Generalization, Mathematics, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2012.07463" label="2012.07463">
        <attvalues>
          <attvalue for="0" value="Parameter-Efficient Transfer Learning with Diff Pruning" />
          <attvalue for="1" value="  While task-specific finetuning of pretrained networks has led to significant&#10;empirical advances in NLP, the large size of networks makes finetuning&#10;difficult to deploy in multi-task, memory-constrained settings. We propose diff&#10;pruning as a simple approach to enable parameter-efficient transfer learning&#10;within the pretrain-finetune framework. This approach views finetuning as&#10;learning a task-specific diff vector that is applied on top of the pretrained&#10;parameter vector, which remains fixed and is shared across different tasks. The&#10;diff vector is adaptively pruned during training with a differentiable&#10;approximation to the L0-norm penalty to encourage sparsity. Diff pruning&#10;becomes parameter-efficient as the number of tasks increases, as it requires&#10;storing only the nonzero positions and weights of the diff vector for each&#10;task, while the cost of storing the shared pretrained model remains constant.&#10;It further does not require access to all tasks during training, which makes it&#10;attractive in settings where tasks arrive in stream or the set of tasks is&#10;unknown. We find that models finetuned with diff pruning can match the&#10;performance of fully finetuned baselines on the GLUE benchmark while only&#10;modifying 0.5% of the pretrained model's parameters per task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.14839" label="2104.14839">
        <attvalues>
          <attvalue for="0" value="The Factual Inconsistency Problem in Abstractive Text Summarization: A&#10;  Survey" />
          <attvalue for="1" value="  Recently, various neural encoder-decoder models pioneered by Seq2Seq&#10;framework have been proposed to achieve the goal of generating more abstractive&#10;summaries by learning to map input text to output text. At a high level, such&#10;neural models can freely generate summaries without any constraint on the words&#10;or phrases used. Moreover, their format is closer to human-edited summaries and&#10;output is more readable and fluent. However, the neural model's abstraction&#10;ability is a double-edged sword. A commonly observed problem with the generated&#10;summaries is the distortion or fabrication of factual information in the&#10;article. This inconsistency between the original text and the summary has&#10;caused various concerns over its applicability, and the previous evaluation&#10;methods of text summarization are not suitable for this issue. In response to&#10;the above problems, the current research direction is predominantly divided&#10;into two categories, one is to design fact-aware evaluation metrics to select&#10;outputs without factual inconsistency errors, and the other is to develop new&#10;summarization systems towards factual consistency. In this survey, we focus on&#10;presenting a comprehensive review of these fact-specific evaluation methods and&#10;text summarization models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.06963" label="2107.06963">
        <attvalues>
          <attvalue for="0" value="Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable&#10;  Features" />
          <attvalue for="1" value="  Knowledge-grounded dialogue systems are intended to convey information that&#10;is based on evidence provided in a given source text. We discuss the challenges&#10;of training a generative neural dialogue model for such systems that is&#10;controlled to stay faithful to the evidence. Existing datasets contain a mix of&#10;conversational responses that are faithful to selected evidence as well as more&#10;subjective or chit-chat style responses. We propose different evaluation&#10;measures to disentangle these different styles of responses by quantifying the&#10;informativeness and objectivity. At training time, additional inputs based on&#10;these evaluation measures are given to the dialogue model. At generation time,&#10;these additional inputs act as stylistic controls that encourage the model to&#10;generate responses that are faithful to the provided evidence. We also&#10;investigate the usage of additional controls at decoding time using resampling&#10;techniques. In addition to automatic metrics, we perform a human evaluation&#10;study where raters judge the output of these controlled generation models to be&#10;generally more objective and faithful to the evidence compared to baseline&#10;dialogue systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07567" label="2104.07567">
        <attvalues>
          <attvalue for="0" value="Retrieval Augmentation Reduces Hallucination in Conversation" />
          <attvalue for="1" value="  Despite showing increasingly human-like conversational abilities,&#10;state-of-the-art dialogue models often suffer from factual incorrectness and&#10;hallucination of knowledge (Roller et al., 2020). In this work we explore the&#10;use of neural-retrieval-in-the-loop architectures - recently shown to be&#10;effective in open-domain QA (Lewis et al., 2020b; Izacard and Grave, 2020) -&#10;for knowledge-grounded dialogue, a task that is arguably more challenging as it&#10;requires querying based on complex multi-turn dialogue context and generating&#10;conversationally coherent responses. We study various types of architectures&#10;with multiple components - retrievers, rankers, and encoder-decoders - with the&#10;goal of maximizing knowledgeability while retaining conversational ability. We&#10;demonstrate that our best models obtain state-of-the-art performance on two&#10;knowledge-grounded conversational tasks. The models exhibit open-domain&#10;conversational capabilities, generalize effectively to scenarios not within the&#10;training data, and, as verified by human evaluations, substantially reduce the&#10;well-known problem of knowledge hallucination in state-of-the-art chatbots.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Hallucination in text-generation models is a topic that has received attention recently, particularly in the settings of summarization \cite{maynez2020faithfulness}, machine translation \cite{zhou2020detecting}, and news generation \cite{zellers2019defending}. For dialogue, it has been observed in state-of-the-art models \cite{roller2020recipes} and studied in depth \cite{mielke2020linguistic}, but so far without resolution.&#10;&#10;Open-domain question answering (QA) has long considered retrieval as an intermediate step towards its solution \cite{voorhees2001trec}, but has become a more intensively studied topic recently for neural models, first using simple vector-space based retrievers \cite{chen2017reading}, and then more recently with end-to-end generation models where the retrieval component is a neural network as well \cite{lewis2020retrieval,izacard2020leveraging}. These recent neural approaches over unstructured text have overtaken prior methods exploiting the graph structure of knowledge sources (such as hyperlinks in Wikipedia) \cite{min2019knowledge,Asai2020Learning,Sun_2019,xiong-etal-2019-improving}, and are an attractive alternative for dialogue.&#10;&#10;Knowledge-grounded dialogue is increasingly becoming a more important topic, with&#10;several datasets proposed that attempt to model its occurrence \cite{dinan2018wizard,ghazvininejad2018knowledge,gopalakrishnan2019topical,galetzka2020corpus}. However, many of these works are constructed based on a model being provided a gold paragraph or passage of knowledge, rather than having to learn to retrieve knowledge from a large unstructured set as we consider here.&#10;Recent methods have focused on: determining which specific elements of a given piece of knowledge are informative to the dialogue, which is commonly referred to as ``knowledge selection'' \cite{Zhao_2020,Kim2020Sequential,Bruyn2020BARTFK}; learning how to attend to the relevant knowledge \cite{Ma_2020,Cai2020ABT,Zhao2020Low-Resource}; or examining how much knowledge is present in large language models \cite{zhao2020pretrained}. Some recent work has explored retrieval-based mechanisms, however the retrieval over knowledge is generally limited to a small subset of the overall corpus considered \cite{Fan_2021,Bruyn2020BARTFK,hedayatnia2020policydriven}. In essence, across the tasks considered, utilizing knowledge in the form of unstructured text is popular, but is generally limited to selection mechanisms over a fixed document, small documents sets or else simple vector-space models \cite{dinan2018wizard}.&#10;&#10;We note that very recently retrieval augmented generation has been applied to task-oriented dialogue \cite{thulke2021efficient}, which is in contrast to the open-domain knowledge-grounded dialogue setting we consider here.&#10;&#10;Other work that includes a retrieval-augmentation step includes the area of language modeling, where it is used for pre-training \cite{guu2020realm}, and as a memory \cite{yogatama2021adaptive}, especially using $k$-nearest neighbor-based cache models \cite{kh2020nearest,Khandelwal2020Generalization,grave2016improving,merity2016pointer}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.10992" label="2305.10992">
        <attvalues>
          <attvalue for="0" value="How does the task complexity of masked pretraining objectives affect&#10;  downstream performance?" />
          <attvalue for="1" value="  Masked language modeling (MLM) is a widely used self-supervised pretraining&#10;objective, where a model needs to predict an original token that is replaced&#10;with a mask given contexts. Although simpler and computationally efficient&#10;pretraining objectives, e.g., predicting the first character of a masked token,&#10;have recently shown comparable results to MLM, no objectives with a masking&#10;scheme actually outperform it in downstream tasks. Motivated by the assumption&#10;that their lack of complexity plays a vital role in the degradation, we&#10;validate whether more complex masked objectives can achieve better results and&#10;investigate how much complexity they should have to perform comparably to MLM.&#10;Our results using GLUE, SQuAD, and Universal Dependencies benchmarks&#10;demonstrate that more complicated objectives tend to show better downstream&#10;results with at least half of the MLM complexity needed to perform comparably&#10;to MLM. Finally, we discuss how we should pretrain a model using a masked&#10;objective from the task complexity perspective.&#10;" />
          <attvalue for="2" value=" &#10;Masked language modeling (MLM) \cite{devlin-etal-2019-bert}, where a model needs to predict a particular token that is replaced with a mask placeholder given its surrounding context, is a widely used self-supervised pretraining objective in natural language processing.&#10;Recently, simpler pretraining objectives have shown promising results on downstream tasks. \cite{aroca-ouellette-rudzicz-2020-losses} have proposed various token-level and sentence-level auxiliary pretraining objectives, showing improvements over BERT \cite{devlin-etal-2019-bert}. \cite{yamaguchi-etal-2021-frustratingly} and \cite{alajrami-aletras-2022-pre} have demonstrated that such token-level objectives themselves, i.e., pretraining without MLM, perform comparably to MLM.&#10;&#10;Although these simple token-level objectives themselves, e.g., predicting the first character of a masked token (First Char) \cite{yamaguchi-etal-2021-frustratingly}, have exhibited competitive downstream performances to MLM with smaller computations, no objectives using mask tokens are not clearly comparable to MLM on downstream tasks. We conjecture that the main reason behind the performance difference lies in its lack of complexity, i.e., the number of classes to be predicted, and similar arguments have been made for auxiliary task ineffectiveness \cite{Lan2020ALBERT} and pretraining task design \cite{yamaguchi-etal-2021-frustratingly}.&#10;&#10;This paper sheds light on the task complexity of masked pretraining objectives and investigates RQ1: whether a more complex objective, becoming closer to MLM, can achieve a better downstream result and RQ2: how much complexity they need to obtain comparable results to MLM.&#10;To this end, we propose masked $n$ character prediction as a control task, which requires us to predict the first or last $n$ characters of a masked token, allowing us to empirically evaluate how the task complexity affects downstream performance by varying $n$.&#10;We pretrain 14 different types of models with the proposed control task in addition to MLM for reference and evaluate their downstream performance on the GLUE \cite{wang2018glue}, SQuAD \cite{rajpurkar-etal-2016-squad}, and Universal Dependencies (UD) \cite{nivre-etal-2020-universal} benchmarks.&#10;We also conduct a cost-benefit analysis of performance gains with respect to task complexity and analyze how to select an optimal complexity for a given task. &#10;&#10;Contributions &#10;(1) We model the task complexity of a masked pretraining objective as masked $n$ character prediction (\S\ref{sec:task}) and revealed how it affects downstream performance (\S\ref{sec:results}).&#10;(2) We conduct two analyses to provide insights into how we should pretrain a model by using a masked objective from the task complexity perspective (\S\ref{sec:analysis}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Masked Language Modeling, Linguistics, Artificial Intelligence, Mathematics, Task Complexity Analysis, Pretraining Objectives" />
        </attvalues>
      </node>
      <node id="2109.01819" label="2109.01819">
        <attvalues>
          <attvalue for="0" value="Frustratingly Simple Pretraining Alternatives to Masked Language&#10;  Modeling" />
          <attvalue for="1" value="  Masked language modeling (MLM), a self-supervised pretraining objective, is&#10;widely used in natural language processing for learning text representations.&#10;MLM trains a model to predict a random sample of input tokens that have been&#10;replaced by a [MASK] placeholder in a multi-class setting over the entire&#10;vocabulary. When pretraining, it is common to use alongside MLM other auxiliary&#10;objectives on the token or sequence level to improve downstream performance&#10;(e.g. next sentence prediction). However, no previous work so far has attempted&#10;in examining whether other simpler linguistically intuitive or not objectives&#10;can be used standalone as main pretraining objectives. In this paper, we&#10;explore five simple pretraining objectives based on token-level classification&#10;tasks as replacements of MLM. Empirical results on GLUE and SQuAD show that our&#10;proposed methods achieve comparable or better performance to MLM using a&#10;BERT-BASE architecture. We further validate our methods using smaller models,&#10;showing that pretraining a model with 41% of the BERT-BASE's parameters,&#10;BERT-MEDIUM results in only a 1% drop in GLUE scores with our best objective.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.10415" label="2203.10415">
        <attvalues>
          <attvalue for="0" value="How does the pre-training objective affect what large language models&#10;  learn about linguistic properties?" />
          <attvalue for="1" value="  Several pre-training objectives, such as masked language modeling (MLM), have&#10;been proposed to pre-train language models (e.g. BERT) with the aim of learning&#10;better language representations. However, to the best of our knowledge, no&#10;previous work so far has investigated how different pre-training objectives&#10;affect what BERT learns about linguistics properties. We hypothesize that&#10;linguistically motivated objectives such as MLM should help BERT to acquire&#10;better linguistic knowledge compared to other non-linguistically motivated&#10;objectives that are not intuitive or hard for humans to guess the association&#10;between the input and the label to be predicted. To this end, we pre-train BERT&#10;with two linguistically motivated objectives and three non-linguistically&#10;motivated ones. We then probe for linguistic characteristics encoded in the&#10;representation of the resulting models. We find strong evidence that there are&#10;only small differences in probing performance between the representations&#10;learned by the two different types of objectives. These surprising results&#10;question the dominant narrative of linguistically informed pre-training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10403" label="2212.10403">
        <attvalues>
          <attvalue for="0" value="Towards Reasoning in Large Language Models: A Survey" />
          <attvalue for="1" value="  Reasoning is a fundamental aspect of human intelligence that plays a crucial&#10;role in activities such as problem solving, decision making, and critical&#10;thinking. In recent years, large language models (LLMs) have made significant&#10;progress in natural language processing, and there is observation that these&#10;models may exhibit reasoning abilities when they are sufficiently large.&#10;However, it is not yet clear to what extent LLMs are capable of reasoning. This&#10;paper provides a comprehensive overview of the current state of knowledge on&#10;reasoning in LLMs, including techniques for improving and eliciting reasoning&#10;in these models, methods and benchmarks for evaluating reasoning abilities,&#10;findings and implications of previous research in this field, and suggestions&#10;on future directions. Our aim is to provide a detailed and up-to-date review of&#10;this topic and stimulate meaningful discussion and future work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1308.0850" label="1308.0850">
        <attvalues>
          <attvalue for="0" value="Generating Sequences With Recurrent Neural Networks" />
          <attvalue for="1" value="  This paper shows how Long Short-term Memory recurrent neural networks can be&#10;used to generate complex sequences with long-range structure, simply by&#10;predicting one data point at a time. The approach is demonstrated for text&#10;(where the data are discrete) and online handwriting (where the data are&#10;real-valued). It is then extended to handwriting synthesis by allowing the&#10;network to condition its predictions on a text sequence. The resulting system&#10;is able to generate highly realistic cursive handwriting in a wide variety of&#10;styles.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.07842" label="2302.07842">
        <attvalues>
          <attvalue for="0" value="Augmented Language Models: a Survey" />
          <attvalue for="1" value="  This survey reviews works in which language models (LMs) are augmented with&#10;reasoning skills and the ability to use tools. The former is defined as&#10;decomposing a potentially complex task into simpler subtasks while the latter&#10;consists in calling external modules such as a code interpreter. LMs can&#10;leverage these augmentations separately or in combination via heuristics, or&#10;learn to do so from demonstrations. While adhering to a standard missing tokens&#10;prediction objective, such augmented LMs can use various, possibly&#10;non-parametric external modules to expand their context processing ability,&#10;thus departing from the pure language modeling paradigm. We therefore refer to&#10;them as Augmented Language Models (ALMs). The missing token objective allows&#10;ALMs to learn to reason, use tools, and even act, while still performing&#10;standard natural language tasks and even outperforming most regular LMs on&#10;several benchmarks. In this work, after reviewing current advance in ALMs, we&#10;conclude that this new research direction has the potential to address common&#10;limitations of traditional LMs such as interpretability, consistency, and&#10;scalability issues.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09946" label="2005.09946">
        <attvalues>
          <attvalue for="0" value="GM-CTSC at SemEval-2020 Task 1: Gaussian Mixtures Cross Temporal&#10;  Similarity Clustering" />
          <attvalue for="1" value="  This paper describes the system proposed for the SemEval-2020 Task 1:&#10;Unsupervised Lexical Semantic Change Detection. We focused our approach on the&#10;detection problem. Given the semantics of words captured by temporal word&#10;embeddings in different time periods, we investigate the use of unsupervised&#10;methods to detect when the target word has gained or loosed senses. To this&#10;end, we defined a new algorithm based on Gaussian Mixture Models to cluster the&#10;target similarities computed over the two periods. We compared the proposed&#10;approach with a number of similarity-based thresholds. We found that, although&#10;the performance of the detection methods varies across the word embedding&#10;algorithms, the combination of Gaussian Mixture with Temporal Referencing&#10;resulted in our best system.&#10;" />
          <attvalue for="2" value="&#10;The recent development in word embeddings, and their increasing capability to capture lexical semantics&#10;has inspired the application of these methods to new tasks and introduced new challenges. &#10;The diachronic analysis of language is one of such linguistic tasks that has benefited from the advantages of these new methods, i.e. the capability to build semantic representations of words by skimming through large corpora spanning multiple time periods.&#10;SemEval 2020 Task 1 \cite{schlechtweg2020semeval} addresses the current lack of a systematic approach for the evaluation of automatic methods for the diachronic analysis by proposing a common evaluation framework that comprises two tasks and covers four different languages (German, English, Latin, and Swedish).&#10;Given two corpora $C_1$ and $C_2$ for two periods $t_1$ and $t_2$, Subtask 1 requires participants to classify a set of target words in two categories: words that have lost or gained senses from $t_1$ to $t_2$ and words that did not, while Subtask 2 requires participants to rank the target words according to their degree of lexical semantic change between the two periods.&#10;We tackle the problem of automatically detecting lexical semantic changes with approaches that rely on temporal word embeddings.&#10;These approaches create a word vector representation for each time period by exploiting a shared semantic space. Similarity measures can then be used to capture the extent of a word semantic change between two time lapses.&#10;Some temporal word embedding techniques adopt a two-step approach, where they first learn separate word embeddings for each time period and then align the word vectors across multiple time periods \cite{Hamilton2016}. Other dynamic approaches incorporate the alignment directly into the learning stage via the optimisation function \cite{Tahmasebi2018}.&#10;Dynamic word embeddings can be further categorised according to the constraint imposed on the alignment. &#10;The explicit alignment adopts a conservative approach to the semantic drift that a word can undergo by posing a limit to the distance between the word vectors belonging to the two temporal spaces.&#10;In the implicit alignment, there is no need for explicit constraint since the alignment is automatically performed by sharing the same word context vectors across all the time periods.&#10;&#10;In this work, we focus on dynamic word embeddings by exploring methods based on both explicit, such as Dynamic Word2Vec \cite{Yao2018}, and implicit alignment, namely Temporal Random Indexing \cite{Basile} and Temporal Referencing \cite{Dubossarsky2019}.&#10;We analyse the use of different similarity measures to determine the extent of a word semantic change and&#10;compare the cosine similarity with Pearson Correlation and the neighborhood similarity \cite{Shoemark2019}.&#10;While these similarity measures can be directly employed to generate a ranked list of words for Subtask 2, their adoption in Subtask 1 requires further manipulation.&#10;We introduce a new method to classify changing vs. stable words by clustering the target similarity distributions via Gaussian Mixture Models.&#10;We describe the embedding models and the clustering algorithm in Section 2, while Section 3 provides details about the hyper-parameter selection. Section 4 reports the results of the task evaluation followed by some concluding remarks in Section 5.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Temporal Word Embeddings, Machine Learning, Computer Science, Linguistics, Computational Linguistics, Unsupervised Detection Methods, Natural Language Processing, Statistics, Lexical Semantic Change" />
        </attvalues>
      </node>
      <node id="1811.06278" label="1811.06278">
        <attvalues>
          <attvalue for="0" value="Survey of Computational Approaches to Lexical Semantic Change" />
          <attvalue for="1" value="  Our languages are in constant flux driven by external factors such as&#10;cultural, societal and technological changes, as well as by only partially&#10;understood internal motivations. Words acquire new meanings and lose old&#10;senses, new words are coined or borrowed from other languages and obsolete&#10;words slide into obscurity. Understanding the characteristics of shifts in the&#10;meaning and in the use of words is useful for those who work with the content&#10;of historical texts, the interested general public, but also in and of itself.&#10;The findings from automatic lexical semantic change detection, and the models&#10;of diachronic conceptual change are currently being incorporated in approaches&#10;for measuring document across-time similarity, information retrieval from&#10;long-term document archives, the design of OCR algorithms, and so on. In recent&#10;years we have seen a surge in interest in the academic community in&#10;computational methods and tools supporting inquiry into diachronic conceptual&#10;change and lexical replacement. This article is an extract of a survey of&#10;recent computational techniques to tackle lexical semantic change currently&#10;under review. In this article we focus on diachronic conceptual change as an&#10;extension of semantic change.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.00607" label="1703.00607">
        <attvalues>
          <attvalue for="0" value="Dynamic Word Embeddings for Evolving Semantic Discovery" />
          <attvalue for="1" value="  Word evolution refers to the changing meanings and associations of words&#10;throughout time, as a byproduct of human language evolution. By studying word&#10;evolution, we can infer social trends and language constructs over different&#10;periods of human history. However, traditional techniques such as word&#10;representation learning do not adequately capture the evolving language&#10;structure and vocabulary. In this paper, we develop a dynamic statistical model&#10;to learn time-aware word vector representation. We propose a model that&#10;simultaneously learns time-aware embeddings and solves the resulting &quot;alignment&#10;problem&quot;. This model is trained on a crawled NYTimes dataset. Additionally, we&#10;develop multiple intuitive evaluation strategies of temporal word embeddings.&#10;Our qualitative and quantitative tests indicate that our method not only&#10;reliably captures this evolution over time, but also consistently outperforms&#10;state-of-the-art temporal embedding approaches on both semantic accuracy and&#10;alignment quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01688" label="1906.01688">
        <attvalues>
          <attvalue for="0" value="Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic&#10;  Change" />
          <attvalue for="1" value="  State-of-the-art models of lexical semantic change detection suffer from&#10;noise stemming from vector space alignment. We have empirically tested the&#10;Temporal Referencing method for lexical semantic change and show that, by&#10;avoiding alignment, it is less affected by this noise. We show that, trained on&#10;a diachronic corpus, the skip-gram with negative sampling architecture with&#10;temporal referencing outperforms alignment models on a synthetic task as well&#10;as a manual testset. We introduce a principled way to simulate lexical semantic&#10;change and systematically control for possible biases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1608.04207" label="1608.04207">
        <attvalues>
          <attvalue for="0" value="Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction&#10;  Tasks" />
          <attvalue for="1" value="  There is a lot of research interest in encoding variable length sentences&#10;into fixed length vectors, in a way that preserves the sentence meanings. Two&#10;common methods include representations based on averaging word vectors, and&#10;representations based on the hidden states of recurrent neural networks such as&#10;LSTMs. The sentence vectors are used as features for subsequent machine&#10;learning tasks or for pre-training in the context of deep learning. However,&#10;not much is known about the properties that are encoded in these sentence&#10;representations and about the language information they capture. We propose a&#10;framework that facilitates better understanding of the encoded representations.&#10;We define prediction tasks around isolated aspects of sentence structure&#10;(namely sentence length, word content, and word order), and score&#10;representations by the ability to train a classifier to solve each prediction&#10;task when using the representation as input. We demonstrate the potential&#10;contribution of the approach by analyzing different sentence representation&#10;mechanisms. The analysis sheds light on the relative strengths of different&#10;sentence embedding methods with respect to these low level prediction tasks,&#10;and on the effect of the encoded vector's dimensionality on the resulting&#10;representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.12289" label="1909.12289">
        <attvalues>
          <attvalue for="0" value="Attention Forcing for Sequence-to-sequence Model Training" />
          <attvalue for="1" value="  Auto-regressive sequence-to-sequence models with attention mechanism have&#10;achieved state-of-the-art performance in many tasks such as machine translation&#10;and speech synthesis. These models can be difficult to train. The standard&#10;approach, teacher forcing, guides a model with reference output history during&#10;training. The problem is that the model is unlikely to recover from its&#10;mistakes during inference, where the reference output is replaced by generated&#10;output. Several approaches deal with this problem, largely by guiding the model&#10;with generated output history. To make training stable, these approaches often&#10;require a heuristic schedule or an auxiliary classifier. This paper introduces&#10;attention forcing, which guides the model with generated output history and&#10;reference attention. This approach can train the model to recover from its&#10;mistakes, in a stable fashion, without the need for a schedule or a classifier.&#10;In addition, it allows the model to generate output sequences aligned with the&#10;references, which can be important for cascaded systems like many speech&#10;synthesis systems. Experiments on speech synthesis show that attention forcing&#10;yields significant performance gain. Experiments on machine translation show&#10;that for tasks where various re-orderings of the output are valid, guiding the&#10;model with generated output history is challenging, while guiding the model&#10;with reference attention is beneficial.&#10;" />
          <attvalue for="2" value="&#10;Auto-regressive sequence-to-sequence (seq2seq) models with attention mechanism are widely used in a variety of areas including Neural Machine Translation (NMT) \cite{neubig2017neural,huang2016attention} and speech synthesis \cite{shen2018natural,wang2018style}, also known as Text-To-Speech (TTS). These models excel at connecting sequences of different length, but can be difficult to train. A standard approach is teacher forcing, which guides a model with reference output history during training. This makes the model unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. One alternative is to train the model in free running mode, where the model is guided by generated output history. This approach often struggles to converge, especially for attention-based models, which need to infer the correct output and align it with the input at the same time.&#10;&#10;Several approaches are introduced to tackle the above problem, namely scheduled sampling \cite{bengio2015scheduled} and professor forcing \cite{lamb2016professor}. Scheduled sampling randomly decides, for each time step, whether the reference or generated output token is added to the output history. The probability of choosing the reference output token decays from 1 to 0 with a heuristic schedule. A natural extension is sequence-level scheduled sampling, where the decision is made for each sequence instead of token. Professor forcing views the seq2seq model as a generator. During training, the generator operates in both teacher forcing mode and free running mode. In teacher forcing mode, it tries to maximize the standard likelihood. In free running mode, it tries to fool a discriminator, which is trained to tell if the model is running in teacher forcing mode. To make training stable, the above approaches require either a well tuned schedule, or a well trained discriminator.&#10;&#10;This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach makes training stable by decoupling the learning of the output and that of the alignment. There is no need for a schedule or a discriminator. Furthermore, for cascaded systems like many TTS systems, attention forcing can be particularly useful. A model trained with attention forcing can generate (in attention forcing mode) output sequences aligned with the references. These output sequences can be used to train a downstream model, enabling it to fix some upstream errors. The TTS experiments show that attention forcing yields significant gain in speech quality. The NMT experiments show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history can be problematic, while guiding the model with reference attention yields slight but consistent gain in BLEU score \cite{papineni2002bleu}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Sequence-to-Sequence Models, Signal Processing, Mathematics, Attention Mechanisms" />
        </attvalues>
      </node>
      <node id="1703.01619" label="1703.01619">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation and Sequence-to-sequence Models: A Tutorial" />
          <attvalue for="1" value="  This tutorial introduces a new and powerful set of techniques variously&#10;called &quot;neural machine translation&quot; or &quot;neural sequence-to-sequence models&quot;.&#10;These techniques have been used in a number of tasks regarding the handling of&#10;human language, and can be a powerful tool in the toolbox of anyone who wants&#10;to model sequential data of some sort. The tutorial assumes that the reader&#10;knows the basics of math and programming, but does not assume any particular&#10;experience with neural networks or natural language processing. It attempts to&#10;explain the intuition behind the various methods covered, then delves into them&#10;with enough mathematical detail to understand them concretely, and culiminates&#10;with a suggestion for an implementation exercise, where readers can test that&#10;they understood the content in practice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.03099" label="1506.03099">
        <attvalues>
          <attvalue for="0" value="Scheduled Sampling for Sequence Prediction with Recurrent Neural&#10;  Networks" />
          <attvalue for="1" value="  Recurrent Neural Networks can be trained to produce sequences of tokens given&#10;some input, as exemplified by recent results in machine translation and image&#10;captioning. The current approach to training them consists of maximizing the&#10;likelihood of each token in the sequence given the current (recurrent) state&#10;and the previous token. At inference, the unknown previous token is then&#10;replaced by a token generated by the model itself. This discrepancy between&#10;training and inference can yield errors that can accumulate quickly along the&#10;generated sequence. We propose a curriculum learning strategy to gently change&#10;the training process from a fully guided scheme using the true previous token,&#10;towards a less guided scheme which mostly uses the generated token instead.&#10;Experiments on several sequence prediction tasks show that this approach yields&#10;significant improvements. Moreover, it was used successfully in our winning&#10;entry to the MSCOCO image captioning challenge, 2015.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.09038" label="1610.09038">
        <attvalues>
          <attvalue for="0" value="Professor Forcing: A New Algorithm for Training Recurrent Networks" />
          <attvalue for="1" value="  The Teacher Forcing algorithm trains recurrent networks by supplying observed&#10;sequence values as inputs during training and using the network's own&#10;one-step-ahead predictions to do multi-step sampling. We introduce the&#10;Professor Forcing algorithm, which uses adversarial domain adaptation to&#10;encourage the dynamics of the recurrent network to be the same when training&#10;the network and when sampling from the network over multiple time steps. We&#10;apply Professor Forcing to language modeling, vocal synthesis on raw waveforms,&#10;handwriting generation, and image generation. Empirically we find that&#10;Professor Forcing acts as a regularizer, improving test likelihood on character&#10;level Penn Treebank and sequential MNIST. We also find that the model&#10;qualitatively improves samples, especially when sampling for a large number of&#10;time steps. This is supported by human evaluation of sample quality. Trade-offs&#10;between Professor Forcing and Scheduled Sampling are discussed. We produce&#10;T-SNEs showing that Professor Forcing successfully makes the dynamics of the&#10;network during training and sampling more similar.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.01534" label="2103.01534">
        <attvalues>
          <attvalue for="0" value="Towards Efficiently Diversifying Dialogue Generation via Embedding&#10;  Augmentation" />
          <attvalue for="1" value="  Dialogue generation models face the challenge of producing generic and&#10;repetitive responses. Unlike previous augmentation methods that mostly focus on&#10;token manipulation and ignore the essential variety within a single sample&#10;using hard labels, we propose to promote the generation diversity of the neural&#10;dialogue models via soft embedding augmentation along with soft labels in this&#10;paper. Particularly, we select some key input tokens and fuse their embeddings&#10;together with embeddings from their semantic-neighbor tokens. The new&#10;embeddings serve as the input of the model to replace the original one.&#10;Besides, soft labels are used in loss calculation, resulting in multi-target&#10;supervision for a given input. Our experimental results on two datasets&#10;illustrate that our proposed method is capable of generating more diverse&#10;responses than raw models while remains a similar n-gram accuracy that ensures&#10;the quality of generated responses.&#10;" />
          <attvalue for="2" value="&#10;&#10;Dialogue generation is important in AI applications, in which a model can generate responses for user-issued dialogue history. Former dialogue datasets make the end-to-end training of deep neural models possible, such as DailyDialog~\cite{dailydialog}, PersonaChat~\cite{personachat}, etc., and deep neural networks including Seq2seq with attention~\cite{seq2seq}, Transformers~\cite{transformer} have already shown their capability in generating conversation replies. Dialogue generation can be regarded as a many-to-many (combination of one-to-many and many-to-one) problem as one specific response may be reasonable for multiple histories and vice versa. However, current models tend to produce generic sentences such as ``I don't know'', caused by the current inherent deterministic training objective as well as insufficient diversity and limited quality of current datasets~\cite{diversity_objective,shao_quality}. &#10;&#10;Generating more various dialogue responses remains a hard task. In order to tackle it, some researchers try to refine the training objective with extra constraint items~\cite{diversity_objective} or modify the criterion that encourages models to decode more diverse words~\cite{kurata_generate,kulikov_decoding,li2016simple,song_diversifying}. However, such models are more likely to generate ungrammatical or uncorrelated responses and the superiority of constructed objectives to cross-entropy loss remains unknown. &#10;Another kind of approach enhances the training data directly. Data filtering is commonly used in machine learning and extended to dialogue generation by removing samples with generic responses based on entropy~\cite{entropy_filter} or the predictions of a Seq2seq model~\cite{data_distillation}. &#10;Obtaining more training samples is also a possible solution. Word replacement extends the data scales by randomly replacing original tokens with others based on vocabulary distributions~\cite{xie_replacement,fadaee_replacement}. &#10;In addition, new augmented sentences can also be obtained via a learnable model that even keeps interaction between the generation model and both of them will be trained jointly~\cite{insufficient_rock}. But these methods merely include more training data and still bridge a one-to-one mapping between input and response within a single sample. And they also add great extra computational load. So it is necessary to propose a simple and effective method regarding the essence of this problem.&#10;&#10;In this paper, we propose an embedding augmentation method for dialogue models, where not only the training objective encourages various output, but also soft embeddings enhance the diversity within a single sample. It is inspired by the recent success of the mixup approach that combines training pairs of samples and labels convexly into a single one~\cite{mixup}. &#10;Particularly, the original embeddings of tokens in the training samples will be randomly replaced by augmented ones, which are mixtures of the raw one and several semantically similar embeddings conditioned on their distribution. And these selected positions will use soft labels instead of hard one-hot vectors. Such a mechanism ensures that the model can learn a soft word distribution rather than a fixed word in both input and output, which is consistent with the multi-source and multi-target purpose, benefiting the generating diversity without using any additional sample. Compared to previous similar work~\cite{kobayashi_contextual,gao_mixup}, we use both soft embedding and soft labels for a more flexible training process. A compact in-domain similar token prediction model is utilized instead of a deep language model, realizing a more efficient augmentation with a less computational cost.&#10;&#10;To verify our method, we conduct experiments on two dialogue datasets, PersonaChat and DailyDialog, using two base models, Seq2seq and transformer. The experimental results show that our method can obtain remarkable diversity improvement as well as n-gram accuracy boost in terms of auto metrics compared to models without embedding augmentation or using token-level replacement. We also compare it with a BERT-based embedding augmentation method, demonstrating they have close performance while the former one is much faster and proving the efficiency of our method.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Dialogue Generation Models, Machine Learning, Computer Science, Linguistics, Neural Network Augmentation, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1510.03055" label="1510.03055">
        <attvalues>
          <attvalue for="0" value="A Diversity-Promoting Objective Function for Neural Conversation Models" />
          <attvalue for="1" value="  Sequence-to-sequence neural network models for generation of conversational&#10;responses tend to generate safe, commonplace responses (e.g., &quot;I don't know&quot;)&#10;regardless of the input. We suggest that the traditional objective function,&#10;i.e., the likelihood of output (response) given input (message) is unsuited to&#10;response generation tasks. Instead we propose using Maximum Mutual Information&#10;(MMI) as the objective function in neural models. Experimental results&#10;demonstrate that the proposed MMI models produce more diverse, interesting, and&#10;appropriate responses, yielding substantive gains in BLEU scores on two&#10;conversational datasets and in human evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.08562" label="1611.08562">
        <attvalues>
          <attvalue for="0" value="A Simple, Fast Diverse Decoding Algorithm for Neural Generation" />
          <attvalue for="1" value="  In this paper, we propose a simple, fast decoding algorithm that fosters&#10;diversity in neural generation. The algorithm modifies the standard beam search&#10;algorithm by adding an inter-sibling ranking penalty, favoring choosing&#10;hypotheses from diverse parents. We evaluate the proposed model on the tasks of&#10;dialogue response generation, abstractive summarization and machine&#10;translation. We find that diverse decoding helps across all tasks, especially&#10;those for which reranking is needed.&#10;  We further propose a variation that is capable of automatically adjusting its&#10;diversity decoding rates for different inputs using reinforcement learning&#10;(RL). We observe a further performance boost from this RL technique. This paper&#10;includes material from the unpublished script &quot;Mutual Information and Diverse&#10;Decoding Improve Neural Machine Translation&quot; (Li and Jurafsky, 2016).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.05471" label="1905.05471">
        <attvalues>
          <attvalue for="0" value="Improving Neural Conversational Models with Entropy-Based Data Filtering" />
          <attvalue for="1" value="  Current neural network-based conversational models lack diversity and&#10;generate boring responses to open-ended utterances. Priors such as persona,&#10;emotion, or topic provide additional information to dialog models to aid&#10;response generation, but annotating a dataset with priors is expensive and such&#10;annotations are rarely available. While previous methods for improving the&#10;quality of open-domain response generation focused on either the underlying&#10;model or the training objective, we present a method of filtering dialog&#10;datasets by removing generic utterances from training data using a simple&#10;entropy-based approach that does not require human supervision. We conduct&#10;extensive experiments with different variations of our method, and compare&#10;dialog models across 17 evaluation metrics to show that training on datasets&#10;filtered this way results in better conversational quality as chatbots learn to&#10;output more diverse responses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1702.06703" label="1702.06703">
        <attvalues>
          <attvalue for="0" value="Data Distillation for Controlling Specificity in Dialogue Generation" />
          <attvalue for="1" value="  People speak at different levels of specificity in different situations.&#10;Depending on their knowledge, interlocutors, mood, etc.} A conversational agent&#10;should have this ability and know when to be specific and when to be general.&#10;We propose an approach that gives a neural network--based conversational agent&#10;this ability. Our approach involves alternating between \emph{data&#10;distillation} and model training : removing training examples that are closest&#10;to the responses most commonly produced by the model trained from the last&#10;round and then retrain the model on the remaining dataset. Dialogue generation&#10;models trained with different degrees of data distillation manifest different&#10;levels of specificity.&#10;  We then train a reinforcement learning system for selecting among this pool&#10;of generation models, to choose the best level of specificity for a given&#10;input. Compared to the original generative model trained without distillation,&#10;the proposed system is capable of generating more interesting and&#10;higher-quality responses, in addition to appropriately adjusting specificity&#10;depending on the context.&#10;  Our research constitutes a specific case of a broader approach involving&#10;training multiple subsystems from a single dataset distinguished by differences&#10;in a specific property one wishes to model. We show that from such a set of&#10;subsystems, one can use reinforcement learning to build a system that tailors&#10;its output to different input contexts at test time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.02573" label="1703.02573">
        <attvalues>
          <attvalue for="0" value="Data Noising as Smoothing in Neural Network Language Models" />
          <attvalue for="1" value="  Data noising is an effective technique for regularizing neural network&#10;models. While noising is widely adopted in application domains such as vision&#10;and speech, commonly used noising primitives have not been developed for&#10;discrete sequence-level settings such as language modeling. In this paper, we&#10;derive a connection between input noising in neural network language models and&#10;smoothing in $n$-gram models. Using this connection, we draw upon ideas from&#10;smoothing to develop effective noising schemes. We demonstrate performance&#10;gains when applying the proposed schemes to language modeling and machine&#10;translation. Finally, we provide empirical analysis validating the relationship&#10;between noising and smoothing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.00440" label="1705.00440">
        <attvalues>
          <attvalue for="0" value="Data Augmentation for Low-Resource Neural Machine Translation" />
          <attvalue for="1" value="  The quality of a Neural Machine Translation system depends substantially on&#10;the availability of sizable parallel corpora. For low-resource language pairs&#10;this is not the case, resulting in poor translation quality. Inspired by work&#10;in computer vision, we propose a novel data augmentation approach that targets&#10;low-frequency words by generating new sentence pairs containing rare words in&#10;new, synthetically created contexts. Experimental results on simulated&#10;low-resource settings show that our method improves translation quality by up&#10;to 2.9 BLEU points over the baseline and up to 3.2 BLEU over back-translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.09412" label="1710.09412">
        <attvalues>
          <attvalue for="0" value="mixup: Beyond Empirical Risk Minimization" />
          <attvalue for="1" value="  Large deep neural networks are powerful, but exhibit undesirable behaviors&#10;such as memorization and sensitivity to adversarial examples. In this work, we&#10;propose mixup, a simple learning principle to alleviate these issues. In&#10;essence, mixup trains a neural network on convex combinations of pairs of&#10;examples and their labels. By doing so, mixup regularizes the neural network to&#10;favor simple linear behavior in-between training examples. Our experiments on&#10;the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show&#10;that mixup improves the generalization of state-of-the-art neural network&#10;architectures. We also find that mixup reduces the memorization of corrupt&#10;labels, increases the robustness to adversarial examples, and stabilizes the&#10;training of generative adversarial networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.10523" label="1905.10523">
        <attvalues>
          <attvalue for="0" value="Soft Contextual Data Augmentation for Neural Machine Translation" />
          <attvalue for="1" value="  While data augmentation is an important trick to boost the accuracy of deep&#10;learning methods in computer vision tasks, its study in natural language tasks&#10;is still very limited. In this paper, we present a novel data augmentation&#10;method for neural machine translation. Different from previous augmentation&#10;methods that randomly drop, swap or replace words with other words in a&#10;sentence, we softly augment a randomly chosen word in a sentence by its&#10;contextual mixture of multiple related words. More accurately, we replace the&#10;one-hot representation of a word by a distribution (provided by a language&#10;model) over the vocabulary, i.e., replacing the embedding of this word by a&#10;weighted combination of multiple semantically similar words. Since the weights&#10;of those words depend on the contextual information of the word to be replaced,&#10;the newly generated sentences capture much richer information than previous&#10;augmentation methods. Experimental results on both small scale and large scale&#10;machine translation datasets demonstrate the superiority of our method over&#10;strong baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.08824" label="2010.08824">
        <attvalues>
          <attvalue for="0" value="Knowledge-Grounded Dialogue Generation with Pre-trained Language Models" />
          <attvalue for="1" value="  We study knowledge-grounded dialogue generation with pre-trained language&#10;models. To leverage the redundant external knowledge under capacity constraint,&#10;we propose equipping response generation defined by a pre-trained language&#10;model with a knowledge selection module, and an unsupervised approach to&#10;jointly optimizing knowledge selection and response generation with unlabeled&#10;dialogues. Empirical results on two benchmarks indicate that our model can&#10;significantly outperform state-of-the-art methods in both automatic evaluation&#10;and human judgment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Early work on end-to-end open domain dialogue generation is inspired by the research of machine translation \cite{ritter2011data,shangL2015neural,vinyals2015neural}. Later, the vanilla encoder-decoder architecture is widely extended to improve diversity of responses \cite{li2015diversity,xing2017topic,zhao2017learning,tao2018get}; to model the structure of conversation contexts \cite{serban2016building,serban2017hierarchical,xing2017hierarchical,zhang2019recosa}; to control attributes of responses \cite{xu2019neural,zhou2017emotional,zhang2018learning,wang2018learning,see2019makes}; and to bias responses to some specific personas \cite{li2016persona,zhang2018personalizing}. Recently, grounding dialogue generation by extra knowledge is emerging as an important step towards human-like conversational AI, where the knowledge could be obtained from knowledge graphs \cite{zhou2018commonsense,moon2019opendialkg,tuan2019dykgchat}, retrieved from unstructured documents \cite{dinan2018wizard,lian2019learning,zhao2020low,kim2020sequential}, or extracted from visual background \cite{mostafazadeh2017image,shuster2018engaging,huber2018emotional}. In this work, we study document-grounded dialogue generation. Rather than learning from scratch like most existing work, we take advantage of the pre-trained language models and achieve new state-of-the-art on the benchmarks of the task. &#10;&#10;Big, deep neural language models pre-trained on huge unlabeled text corpus have led to strong improvements on numerous natural language understanding and natural language generation benchmarks \cite{devlin2018bert,yang2019xlnet,liu2019roberta,radford2019language,song2019mass,dong2019unified,lewis2019bart}, and therefore are revolutionizing almost the full spectrum of NLP applications \cite{raffel2019exploring,sun2019utilizing,qiao2019understanding,zhang2019hibert,lample2019cross} and some &#10;interdisciplinary applications in NLP and computer vision \cite{lu2019vilbert,su2019vl,sun2019videobert}. In the context of dialogue generation, by fine-tuning GPT-2 \cite{radford2019language} in different sizes on social media data, recent work has \cite{zhang2019dialogpt,wolf2019transfertransfo} shown promising progress on conversation engagement and commonsense question-answering. In this work, we further explore the application of pre-training to the task of open domain dialogue generation by equipping the pre-trained language models with external knowledge. Different from a very recent paper on pre-training for low-resource knowledge-grounded dialogue generation \cite{zhao2020low}, the work presents an in-depth investigation on how to release the power of the existing pre-trained language models on the task when input exceeds the capacity of the models. &#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Dialogue Generation, Artificial Intelligence, Natural Language Processing, Language Modeling, Knowledge Selection" />
        </attvalues>
      </node>
      <node id="2309.02240" label="2309.02240">
        <attvalues>
          <attvalue for="0" value="Dialog Action-Aware Transformer for Dialog Policy Learning" />
          <attvalue for="1" value="  Recent works usually address Dialog policy learning DPL by training a&#10;reinforcement learning (RL) agent to determine the best dialog action. However,&#10;existing works on deep RL require a large volume of agent-user interactions to&#10;achieve acceptable performance. In this paper, we propose to make full use of&#10;the plain text knowledge from the pre-trained language model to accelerate the&#10;RL agent's learning speed. Specifically, we design a dialog action-aware&#10;transformer encoder (DaTrans), which integrates a new fine-tuning procedure&#10;named masked last action task to encourage DaTrans to be dialog-aware and&#10;distils action-specific features. Then, DaTrans is further optimized in an RL&#10;setting with ongoing interactions and evolves through exploration in the dialog&#10;action space toward maximizing long-term accumulated rewards. The effectiveness&#10;and efficiency of the proposed model are demonstrated with both simulator&#10;evaluation and human evaluation.&#10;" />
          <attvalue for="2" value="&#10;A task-oriented dialog system that can serve users on certain tasks has increasingly attracted research efforts. &#10;Dialog policy learning (DPL) aiming to determine the next abstracted system output plays a key role in pipeline task-oriented dialog systems \cite{kwan_survey_2023}. Recently, it has shown great potential for using reinforcement learning (RL) based methods to formulate DPL \cite{young2013pomdp, su2016continuously, peng2017composite}. A lot of progress is being made in demonstration-based efficient learning methods \cite{brys2015reinforcement,cederborg2015policy,wang2020learning,li2020rethinking,jhunjhunwala2020multi,geishauser2022dynamic}. Among these methods, dialog state tracking (DST), comprising all information required to determine the response, is an indispensable module. However, DST inevitably accumulates errors from each module of the system. &#10;&#10;Recent pre-trained language models (PLMs) gathering knowledge from the massive plain text show great potential for formulating DPL without DST. Recently, the studies on PLMs for dialog, including BERT-based dialog state tracking \cite{gulyaev2020goal} and GPT-2 based dialog generation \cite{peng2020few, yang2020ubar} are not centred on DPL. To this end, we proposed the Dialog Action-oriented transformer encoder termed as DaTrans, for efficient dialog policy training. DaTrans is achieved by a dialog act-aware fine-tuning task, which encourages the model to distil the dialog policy logic. Specifically, rather than commonly used tasks, like predicting randomly masked words in the input (MLM task) and classifying whether the sentences are continuous or not (NSP task) \cite{devlin2018bert}, DaTrans is fine-tuned by predicting the masked last acts in the input action sequences (termed as MLA task). After that, DaTrans works as an RL agent which evolves toward maximizing long-term accumulated rewards through interacting with a user simulator. Following the traditional RL-based dialog policy learning framework, the main novelty of DaTrans is that it integrates a proposed dialog action-aware fine-tuning task (MLA), which helps to extract action-specific features from historical dialog action sequences to improve dialog policy learning. The empirical results prove the excellent performance of DaTrans. Our main contributions include 1) We propose the DaTrans that integrates the dialog act-aware fine-tuning task to extract the dialog policy logic from the plain text; 2) We validate the efficiency and effectiveness of the proposed model on a multi-domain benchmark with both simulator and human evaluation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Dialog Policy Learning, Artificial Intelligence, Natural Language Processing, Reinforcement Learning, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2202.13675" label="2202.13675">
        <attvalues>
          <attvalue for="0" value="A Survey on Recent Advances and Challenges in Reinforcement Learning&#10;  Methods for Task-Oriented Dialogue Policy Learning" />
          <attvalue for="1" value="  Dialogue Policy Learning is a key component in a task-oriented dialogue&#10;system (TDS) that decides the next action of the system given the dialogue&#10;state at each turn. Reinforcement Learning (RL) is commonly chosen to learn the&#10;dialogue policy, regarding the user as the environment and the system as the&#10;agent. Many benchmark datasets and algorithms have been created to facilitate&#10;the development and evaluation of dialogue policy based on RL. In this paper,&#10;we survey recent advances and challenges in dialogue policy from the&#10;prescriptive of RL. More specifically, we identify the major problems and&#10;summarize corresponding solutions for RL-based dialogue policy learning.&#10;Besides, we provide a comprehensive survey of applying RL to dialogue policy&#10;learning by categorizing recent methods into basic elements in RL. We believe&#10;this survey can shed a light on future research in dialogue management.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.09781" label="2009.09781">
        <attvalues>
          <attvalue for="0" value="Rethinking Supervised Learning and Reinforcement Learning in&#10;  Task-Oriented Dialogue Systems" />
          <attvalue for="1" value="  Dialogue policy learning for task-oriented dialogue systems has enjoyed great&#10;progress recently mostly through employing reinforcement learning methods.&#10;However, these approaches have become very sophisticated. It is time to&#10;re-evaluate it. Are we really making progress developing dialogue agents only&#10;based on reinforcement learning? We demonstrate how (1)~traditional supervised&#10;learning together with (2)~a simulator-free adversarial learning method can be&#10;used to achieve performance comparable to state-of-the-art RL-based methods.&#10;First, we introduce a simple dialogue action decoder to predict the appropriate&#10;actions. Then, the traditional multi-label classification solution for dialogue&#10;policy learning is extended by adding dense layers to improve the dialogue&#10;agent performance. Finally, we employ the Gumbel-Softmax estimator to&#10;alternatively train the dialogue agent and the dialogue reward model without&#10;using reinforcement learning. Based on our extensive experimentation, we can&#10;conclude the proposed methods can achieve more stable and higher performance&#10;with fewer efforts, such as the domain knowledge required to design a user&#10;simulator and the intractable parameter tuning in reinforcement learning. Our&#10;main goal is not to beat reinforcement learning with supervised learning, but&#10;to demonstrate the value of rethinking the role of reinforcement learning and&#10;supervised learning in optimizing task-oriented dialogue systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05928" label="2204.05928">
        <attvalues>
          <attvalue for="0" value="Dynamic Dialogue Policy for Continual Reinforcement Learning" />
          <attvalue for="1" value="  Continual learning is one of the key components of human learning and a&#10;necessary requirement of artificial intelligence. As dialogue can potentially&#10;span infinitely many topics and tasks, a task-oriented dialogue system must&#10;have the capability to continually learn, dynamically adapting to new&#10;challenges while preserving the knowledge it already acquired. Despite the&#10;importance, continual reinforcement learning of the dialogue policy has&#10;remained largely unaddressed. The lack of a framework with training protocols,&#10;baseline models and suitable metrics, has so far hindered research in this&#10;direction. In this work we fill precisely this gap, enabling research in&#10;dialogue policy optimisation to go from static to dynamic learning. We provide&#10;a continual learning algorithm, baseline architectures and metrics for&#10;assessing continual learning models. Moreover, we propose the dynamic dialogue&#10;policy transformer (DDPT), a novel dynamic architecture that can integrate new&#10;knowledge seamlessly, is capable of handling large state spaces and obtains&#10;significant zero-shot performance when being exposed to unseen domains, without&#10;any growth in network parameter size.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.02450" label="2002.02450">
        <attvalues>
          <attvalue for="0" value="Goal-Oriented Multi-Task BERT-Based Dialogue State Tracker" />
          <attvalue for="1" value="  Dialogue State Tracking (DST) is a core component of virtual assistants such&#10;as Alexa or Siri. To accomplish various tasks, these assistants need to support&#10;an increasing number of services and APIs. The Schema-Guided State Tracking&#10;track of the 8th Dialogue System Technology Challenge highlighted the DST&#10;problem for unseen services. The organizers introduced the Schema-Guided&#10;Dialogue (SGD) dataset with multi-domain conversations and released a zero-shot&#10;dialogue state tracking model. In this work, we propose a GOaL-Oriented&#10;Multi-task BERT-based dialogue state tracker (GOLOMB) inspired by architectures&#10;for reading comprehension question answering systems. The model &quot;queries&quot;&#10;dialogue history with descriptions of slots and services as well as possible&#10;values of slots. This allows to transfer slot values in multi-domain dialogues&#10;and have a capability to scale to unseen slot types. Our model achieves a joint&#10;goal accuracy of 53.97% on the SGD dataset, outperforming the baseline model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.12328" label="2002.12328">
        <attvalues>
          <attvalue for="0" value="Few-shot Natural Language Generation for Task-Oriented Dialog" />
          <attvalue for="1" value="  As a crucial component in task-oriented dialog systems, the Natural Language&#10;Generation (NLG) module converts a dialog act represented in a semantic form&#10;into a response in natural language. The success of traditional template-based&#10;or statistical models typically relies on heavily annotated data, which is&#10;infeasible for new domains. Therefore, it is pivotal for an NLG system to&#10;generalize well with limited labelled data in real applications. To this end,&#10;we present FewShotWoz, the first NLG benchmark to simulate the few-shot&#10;learning setting in task-oriented dialog systems. Further, we develop the&#10;SC-GPT model. It is pre-trained on a large set of annotated NLG corpus to&#10;acquire the controllable generation ability, and fine-tuned with only a few&#10;domain-specific labels to adapt to new domains. Experiments on FewShotWoz and&#10;the large Multi-Domain-WOZ datasets show that the proposed SC-GPT significantly&#10;outperforms existing methods, measured by various automatic metrics and human&#10;evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;A typical task-oriented spoken dialog system uses a pipeline architecture, as shown in Figure~\ref{fig:dailog_system} (a), where each dialog turn is processed using a four-step procedure.&#10;$(\RN{1})$ Transcriptions of user’s input are first passed to the natural language understanding (NLU)&#10;module, where the user’s intention and other key information are extracted. &#10;$(\RN{2})$ This information is then formatted as the input to dialog state tracking (DST), which maintains&#10;the current state of the dialog. &#10;$(\RN{3})$ Outputs of DST are passed to the dialog policy module, which produces a dialog act based on the facts or entities retrieved from external resources (such as a database or a knowledge base).&#10;$(\RN{4})$ The dialog act emitted by the dialog policy module serves as the input to&#10;the NLG, through which a system response in natural language is generated. &#10;In this paper, we focus&#10;on the NLG component of task-oriented dialog&#10;systems, \ie how to produce natural language responses conditioned on dialog acts.&#10;&#10;Specifically, {dialog act}&#10;$\Acal$ is defined as the combination of intent $\Imat$ and slot-value pairs $\{(s_i, v_i)\}^P_{i=1}$:&#10;\begin{equation}&#10; \Acal = [ \underbrace{~~\Imat_{~_{~}}}_{Intent}, \underbrace{(s_1, v_1), \cdots, (s_P, v_P)}_{Slot-value pairs } ]&#10;\end{equation}&#10;where $P$ is the number of pairs, which varies in different dialog acts. &#10;&#10;\begin{itemize}&#10;&#10;\item {Intents} are usually used to distinguish different types of system actions. Typical examples include {inform}, {request}, {confirm}, {select} \etc~&#10; \item {Slot-value pairs} indicate the category and content of the information to express in the utterance, respectively.&#10;\end{itemize}&#10;&#10;The goal of NLG is to translate $\Acal$ into a natural language response $\xv = [x_1, \cdots, x_T]$, where $T$ is the sequence length. In Figure~\ref{fig:dailog_system} (b), we show an example of the dialog act: &#10;$confirm~(name=Hilton, area=center)$, and the corresponding natural language response is ``{Let me confirm that you are searching for Hilton in the center area}''. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2404.19319" label="2404.19319">
        <attvalues>
          <attvalue for="0" value="Knowledge Distillation vs. Pretraining from Scratch under a Fixed&#10;  (Computation) Budget" />
          <attvalue for="1" value="  Compared to standard language model (LM) pretraining (i.e., from scratch),&#10;Knowledge Distillation (KD) entails an additional forward pass through a&#10;teacher model that is typically substantially larger than the target student&#10;model. As such, KD in LM pretraining materially slows down throughput of&#10;pretraining instances vis-a-vis pretraining from scratch. Scaling laws of LM&#10;pretraining suggest that smaller models can close the gap to larger&#10;counterparts if trained on more data (i.e., processing more tokens)-and under a&#10;fixed computation budget, smaller models are able be process more data than&#10;larger models. We thus hypothesize that KD might, in fact, be suboptimal to&#10;pretraining from scratch for obtaining smaller LMs, when appropriately&#10;accounting for the compute budget. To test this, we compare pretraining from&#10;scratch against several KD strategies for masked language modeling (MLM) in a&#10;fair experimental setup, with respect to amount of computation as well as&#10;pretraining data. Downstream results on GLUE, however, do not confirm our&#10;hypothesis: while pretraining from scratch performs comparably to ordinary KD&#10;under a fixed computation budget, more sophisticated KD strategies, namely&#10;TinyBERT (Jiao et al., 2020) and MiniLM (Wang et al., 2023), outperform it by a&#10;notable margin. We further find that KD yields larger gains over pretraining&#10;from scratch when the data must be repeated under the fixed computation budget.&#10;" />
          <attvalue for="2" value="&#10;&#10;Knowledge distillation \cite[KD;][]{hinton2015distilling,jiao-etal-2020-tinybert} during LM pretraining has emerged as the primary mean of compressing the capabilities of a large pretrained teacher model into a task agnostic smaller student model. KD is praised for yielding high-performing task agnostic small models, mitigating the need for pretraining (small models) from scratch, which is typically considered more expensive. The body of existing KD work for MLM \cite{jiao-etal-2020-tinybert, wang-etal-2023-distill}, however, typically does not compare KD against pretraining from scratch in a fair setup: (i) with the same target models (exactly the same architecture) and (ii) under the same computation budget. Compared to just training the target model from scratch, KD comes with a computational overhead of forward passes through the typically considerably larger teacher model. This, under the same computation budget, allows pretraining from scratch to consume more data (i.e., more tokens) than KD, which leads to the central research question of this work: in a fair setup where both are given equal overall computation budget, is KD still more effective than pretraining from scratch (No-KD)? We hypothesize that, under a fair evaluation setup, No-KD may be as effective as KD, rendering KD inconsequential. Our reasoning is based on two observations:&#10;&#10;1) Fair KD Comparison. A fair comparison, in which both setups are given identical computation budgets (as well as identical target models) eludes existing work on KD. \cite{jiao-etal-2020-tinybert} compare their model to BERT\textsubscript{Tiny} \cite{turc2019wellread}, which has the same architecture but employs significantly different training resources than their TinyBERT\textsubscript{Tiny}, preventing a fair comparison. Similarly, \cite{sanh2020distilbert} compare their distilled student solely against the teacher, whereas \cite{sun-etal-2020-mobilebert, wang2020minilm} only add comparison against larger pretrained models and competing KD strategies. Even the body of work that focuses on comparing different KD strategies has only recently sought to standardize training and thus enable fair comparisons \cite{lu2022knowledge,wang-etal-2023-distill}.&#10;&#10;2) Scaling Laws. Scaling laws \cite{kaplan2020scaling, hoffmann2022training}, reveal that, under a fixed computation budget, only a marginal correlation exists between the LM size and it's performance: &#10;smaller models compensate their lower learning efficiency with the ability to process more tokens within the same budget. While there are ongoing refinements to the scaling law \cite{hoffmann2022training}, it has been consistently reaffirmed by several studies \cite{geiping2022cramming, bansal2022data, clark2022unified}. &#10;For instance, \cite{geiping2022cramming} showcases this behavior by training multiple BERT models with varying architecture sizes for a fix 24 hour duration, resulting in similar loss values across all sizes. &#10;&#10;Contribution Motivated by the recent findings in the realm of scaling laws and recognizing the absence of a fair comparison between KD and No-KD, our primary contribution lies in the comparison of No-KD against KD strategies for MLM while ensuring a fair setup with regards to compute budget and pretraining data. We initially assess No-KD in an optimal setup, where unlimited pretraining tokens are available within a fixed compute budget. Additionally, we examine the scenario when data is constrained within a fixed compute budget.&#10;&#10;Our results reveal that, in the optimal setting, No-KD performs indeed comparably to vanilla-KD, exhibiting an average improvement over vanilla-KD of 0.4 and 0.1 points for 6-, and 12-layer models on GLUE. However, No-KD falls short of surpassing more advanced KD strategies, exemplified by the comparison with TinyBERT and MiniLM. When available data is limited within the fixed compute budget, KD strategies outperform No-KD by an even larger margin: No-KD, though faster, needs more epochs, whereas KD strategies extract more information from limited data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Knowledge Distillation Strategies, Linguistics, Computational Efficiency Optimization, Language Model Pretraining, Mathematics" />
        </attvalues>
      </node>
      <node id="2206.14366" label="2206.14366">
        <attvalues>
          <attvalue for="0" value="Knowledge Distillation of Transformer-based Language Models Revisited" />
          <attvalue for="1" value="  In the past few years, transformer-based pre-trained language models have&#10;achieved astounding success in both industry and academia. However, the large&#10;model size and high run-time latency are serious impediments to applying them&#10;in practice, especially on mobile phones and Internet of Things (IoT) devices.&#10;To compress the model, considerable literature has grown up around the theme of&#10;knowledge distillation (KD) recently. Nevertheless, how KD works in&#10;transformer-based models is still unclear. We tease apart the components of KD&#10;and propose a unified KD framework. Through the framework, systematic and&#10;extensive experiments that spent over 23,000 GPU hours render a comprehensive&#10;analysis from the perspectives of knowledge types, matching strategies,&#10;width-depth trade-off, initialization, model size, etc. Our empirical results&#10;shed light on the distillation in the pre-train language model and with&#10;relative significant improvement over previous state-of-the-arts(SOTA).&#10;Finally, we provide a best-practice guideline for the KD in transformer-based&#10;models.&#10;" />
          <attvalue for="2" value="&#10;Recently, the emergence of pre-trained language models, especially the transformer-based model such as BERT \cite{Devlin2019BERTPO}, and GPT-3 \cite{Brown2020LanguageMA}, has revolutionized the research on various natural language processing (NLP), compute vision (CV), and multimodal tasks \cite{Dosovitskiy2021AnII, Liu2021SwinTH,Lin2021M6AC,wang2022OFA} and achieve stunning success. These researches follow the pretrain-then-finetune paradigm: the models are first pre-trained on a large unlabeled corpus and then fine-tuned for specific downstream tasks. Even though these models are effective and prevalent, the heavy model size and high latency limit their application in real-world scenarios, particularly on resource-constrained devices, e.g. mobile phones, IoT devices, and autonomous cars \cite{Zualkernan2022AnIS, Li2021NPASAC}. &#10;&#10;Many model compression techniques have been proposed to obtain a much smaller and eco-friendly model with comparable performance to alleviate the former shortcomings. Among all these methods, knowledge distillation (KD) \cite{Hinton2015DistillingTK} is simple yet effective and has been frequently used \cite{Wang2020MiniLMDS, Jiao2020TinyBERTDB}. KD often trains a large and elaborate model as the teacher model to guide the training of a smaller model, named the student model. During the learning procedure, the student model is forced to mimic the behavior of the teacher so that the knowledge from the teacher model will be transferred to the student model. &#10;&#10;Despite considerable previous literature having grown up to apply knowledge distillation to transformer-based models for model compression \cite{Wang2020MiniLMDS,Jiao2020TinyBERTDB,Sanh2019DistilBERTAD,Sun2020MobileBERTAC}, there are still too many unexplored areas in the mechanism of KD. In this work, we attempt to provide a comprehensive overview of KD for transformer-based models. The main contributions of our work are summarized as follows.&#10;\begin{itemize}&#10; \item We present a generic distillation framework that contains three main components: initialization, knowledge type, and matching strategy. Any existing method could be identified and incorporated into the framework. To tease apart, we categorize common initialization schemes, knowledge types, and matching strategies and propose a unified formulation of distillation. &#10; \item We conduct systematic and extensive experiments which consist of about 30,000 experimental results and cost over 23,000 GPU hours to investigate the effects of different parts of the proposed framework. We provide exhaustive analyses about the initialization, temperature and hard label weight, layer match, width-depth trade-off, and teacher model size. &#10; \item Based on the empirical results, we establish a best-practice guideline on the knowledge distillation of transformer-based models. The model following the guideline achieves better scores with a smaller size compared to previous compact models.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15032" label="2305.15032">
        <attvalues>
          <attvalue for="0" value="How to Distill your BERT: An Empirical Study on the Impact of Weight&#10;  Initialisation and Distillation Objectives" />
          <attvalue for="1" value="  Recently, various intermediate layer distillation (ILD) objectives have been&#10;shown to improve compression of BERT models via Knowledge Distillation (KD).&#10;However, a comprehensive evaluation of the objectives in both task-specific and&#10;task-agnostic settings is lacking. To the best of our knowledge, this is the&#10;first work comprehensively evaluating distillation objectives in both settings.&#10;We show that attention transfer gives the best performance overall. We also&#10;study the impact of layer choice when initializing the student from the teacher&#10;layers, finding a significant impact on the performance in task-specific&#10;distillation. For vanilla KD and hidden states transfer, initialisation with&#10;lower layers of the teacher gives a considerable improvement over higher&#10;layers, especially on the task of QNLI (up to an absolute percentage change of&#10;17.8 in accuracy). Attention transfer behaves consistently under different&#10;initialisation settings. We release our code as an efficient transformer-based&#10;model distillation framework for further studies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.14034" label="2212.14034">
        <attvalues>
          <attvalue for="0" value="Cramming: Training a Language Model on a Single GPU in One Day" />
          <attvalue for="1" value="  Recent trends in language modeling have focused on increasing performance&#10;through scaling, and have resulted in an environment where training language&#10;models is out of reach for most researchers and practitioners. While most in&#10;the community are asking how to push the limits of extreme computation, we ask&#10;the opposite question: How far can we get with a single GPU in just one day?&#10;  We investigate the downstream performance achievable with a transformer-based&#10;language model trained completely from scratch with masked language modeling&#10;for a single day on a single consumer GPU. Aside from re-analyzing nearly all&#10;components of the pretraining pipeline for this scenario and providing a&#10;modified pipeline with performance close to BERT, we investigate why scaling&#10;down is hard, and which modifications actually improve performance in this&#10;scenario. We provide evidence that even in this constrained setting,&#10;performance closely follows scaling laws observed in large-compute settings.&#10;Through the lens of scaling laws, we categorize a range of recent improvements&#10;to training and architecture and discuss their merit and practical&#10;applicability (or lack thereof) for the limited compute setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.01235" label="2204.01235">
        <attvalues>
          <attvalue for="0" value="An Analysis of Semantically-Aligned Speech-Text Embeddings" />
          <attvalue for="1" value="  Embeddings play an important role in end-to-end solutions for multi-modal&#10;language processing problems. Although there has been some effort to understand&#10;the properties of single-modality embedding spaces, particularly that of text,&#10;their cross-modal counterparts are less understood. In this work, we study some&#10;intrinsic properties of a joint speech-text embedding space, constructed by&#10;minimizing the distance between paired utterance and transcription inputs in a&#10;teacher-student model setup, that are informative for several prominent use&#10;cases. We found that incorporating automatic speech recognition through both&#10;pretraining and multitask scenarios aid semantic alignment significantly,&#10;resulting in more tightly coupled embeddings. To analyse cross-modal embeddings&#10;we utilise a quantitative retrieval accuracy metric for semantic alignment,&#10;zero-shot classification for generalisability, and probing of the encoders to&#10;observe the extent of knowledge transfer from one modality to another.&#10;" />
          <attvalue for="2" value="&#10;&#10;Dense vector representations are ubiquitous as inputs into deep learning models. In natural language processing (NLP), there is a significant body of work pertaining to methods that map high-level discrete entities such as words to a low-dimensional continuous space. These learned embeddings often contain useful properties including semantic regularity, that is words that have similar meaning appear closer to each other in vector space \cite{mikolov2013distributed}. More recent deep-learning approaches such as ELMo \cite{peters1luke} and BERT \cite{devlin-etal-2019-bert} can further account for context, enabling for example, the disambiguation of homonyms appearing in different sentences. The masked language modeling objective popularised by BERT has since been extended to the audio domain to learn contextualized speech representations directly from audio \cite{baevski2020wav2vec, hsu2021hubert}. Utilising such embeddings have been shown to improve performance across a broad range of downstream tasks for both speech and text, including spoken language understanding (SLU), automatic speech recognition (ASR), and question-answering (Q-A).&#10;&#10;Given that many modern NLP problems such as the above are multimodal in nature, there have been attempts at unifying the embedding spaces of speech and text. The use-cases of joint speech-text embeddings can be broadly classified into three non-mutually exclusive categories: &#10;\begin {enumerate*} [label=\itshape\alph*\upshape)]&#10;\item when intending to leverage data from both modalities as input into a single model, such as for multimodal translation \cite{zheng2021fused} \item when learning a semantic alignment between text and speech \cite{duquenne2021multimodal,chung2018unsupervised} that can be useful for data mining or retrieval \item when transferring knowledge encoded in a pretrained model from one modality to another or incorporating knowledge from both modalities, often seen in end-to-end solutions for speech translation (ST) \cite{ye2021end,han2021learning,dong2021listen,Tang2021IST}, SLU \cite{Denisov_2020,chung2020splat} or Q-A \cite{chuang2019speechbert}), that combine semantics from text with acoustics from speech&#10;\end{enumerate*}. &#10;&#10;Several studies have been carried out to better understand learned embeddings derived exclusively from either text or speech, via linear probing \cite{conneau-etal-2018-cram,yossisenemb,47786}, geometrical analysis of the representation space \cite{ethayarajh2019}, or other intrinsic measures \cite{Pasad2021LayerWiseAO, schnabel2015evaluation}. Cross-modal embeddings on the other hand have mostly been evaluated only in the context of a downstream task, hence are not well-understood. We attempt to close this gap by studying the extent to which directly minimising a distance-based measure between speech and text inputs at sentence-level can produce joint embeddings that are desirable for the above use-cases. We employ a dual encoder arranged in a teacher-student setup, where the speech embedding space gradually adapts to the text embedding space. Variations of this training objective are found in the literature, either as a pretraining step in isolation \cite{duquenne2021multimodal,Denisov_2020,chung2020splat}, or optimised in conjunction with the downstream task \cite{dong2021listen,Tang2021IST}. &#10;&#10;Ensuing from the above use-cases, we analyse the following characteristics of the joint embedding:&#10;\setlist{nolistsep}&#10;\begin{itemize}[noitemsep]&#10; \item For one model to leverage both data modalities, respective representations with the same underlying semantics should be close in embedding space. We measure this using retrieval accuracy, comparing different training scenarios (pretraining and multitask training) against a strong baseline.&#10; \item The resultant speech-text alignment should be robust to new inputs. We further experiment on the generalisability of the cross-modal alignment through zero-shot classification of several speech datasets. &#10; \item Knowledge transfer from one pretrained model to the other should occur via teacher-student learning. The extent of embedded knowledge present in the model before and after training was ascertained through extending the linear probing technique to cover both text and speech. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Multimodal Embeddings, Machine Learning, Computer Science, Linguistics, Speech Text Alignment, Cross Modal Analysis, Artificial Intelligence, Speech Recognition, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2102.05766" label="2102.05766">
        <attvalues>
          <attvalue for="0" value="Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining&#10;  and Speech Translation" />
          <attvalue for="1" value="  Recently, representation learning for text and speech has successfully&#10;improved many language related tasks. However, all existing methods suffer from&#10;two limitations: (a) they only learn from one input modality, while a unified&#10;representation for both speech and text is needed by tasks such as end-to-end&#10;speech translation, and as a result,(b) they can not exploit various&#10;large-scale text and speech data and their performance is limited by the&#10;scarcity of parallel speech translation data.To address these problems, we&#10;propose a Fused Acoustic and Text Masked Language Model (FAT-MLM) which jointly&#10;learns a unified representation for both acoustic and text input from various&#10;types of corpora including parallel data for speech recognition and machine&#10;translation, and even pure speech and text data. Within this cross-modal&#10;representation learning framework, we further present an end-to-end model for&#10;Fused Acoustic and Text Speech Translation (FAT-ST). Experiments on three&#10;translation directions show that by fine-tuning from FAT-MLM, our proposed&#10;speech translation models substantially improve translation quality by up to&#10;+5.9 BLEU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.10380" label="2104.10380">
        <attvalues>
          <attvalue for="0" value="End-to-end Speech Translation via Cross-modal Progressive Training" />
          <attvalue for="1" value="  End-to-end speech translation models have become a new trend in research due&#10;to their potential of reducing error propagation. However, these models still&#10;suffer from the challenge of data scarcity. How to effectively use unlabeled or&#10;other parallel corpora from machine translation is promising but still an open&#10;problem. In this paper, we propose Cross Speech-Text Network (XSTNet), an&#10;end-to-end model for speech-to-text translation. XSTNet takes both speech and&#10;text as input and outputs both transcription and translation text. The model&#10;benefits from its three key design aspects: a self-supervised pre-trained&#10;sub-network as the audio encoder, a multi-task training objective to exploit&#10;additional parallel bilingual text, and a progressive training procedure. We&#10;evaluate the performance of XSTNet and baselines on the MuST-C En-X and&#10;LibriSpeech En-Fr datasets. In particular, XSTNet achieves state-of-the-art&#10;results on all language directions with an average BLEU of 28.8, outperforming&#10;the previous best method by 3.2 BLEU. Code, models, cases, and more detailed&#10;analysis are available at https://github.com/ReneeYe/XSTNet.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.05782" label="2107.05782">
        <attvalues>
          <attvalue for="0" value="Improving Speech Translation by Understanding and Learning from the&#10;  Auxiliary Text Translation Task" />
          <attvalue for="1" value="  Pretraining and multitask learning are widely used to improve the speech to&#10;text translation performance. In this study, we are interested in training a&#10;speech to text translation model along with an auxiliary text to text&#10;translation task. We conduct a detailed analysis to understand the impact of&#10;the auxiliary task on the primary task within the multitask learning framework.&#10;Our analysis confirms that multitask learning tends to generate similar decoder&#10;representations from different modalities and preserve more information from&#10;the pretrained text translation modules. We observe minimal negative transfer&#10;effect between the two tasks and sharing more parameters is helpful to transfer&#10;knowledge from the text task to the speech task. The analysis also reveals that&#10;the modality representation difference at the top decoder layers is still not&#10;negligible, and those layers are critical for the translation quality. Inspired&#10;by these findings, we propose three methods to improve translation quality.&#10;First, a parameter sharing and initialization strategy is proposed to enhance&#10;information sharing between the tasks. Second, a novel attention-based&#10;regularization is proposed for the encoders and pulls the representations from&#10;different modalities closer. Third, an online knowledge distillation is&#10;proposed to enhance the knowledge transfer from the text to the speech task.&#10;Our experiments show that the proposed approach improves translation&#10;performance by more than 2 BLEU over a strong baseline and achieves&#10;state-of-the-art results on the \textsc{MuST-C} English-German, English-French&#10;and English-Spanish language pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.02295" label="2010.02295">
        <attvalues>
          <attvalue for="0" value="SPLAT: Speech-Language Joint Pre-Training for Spoken Language&#10;  Understanding" />
          <attvalue for="1" value="  Spoken language understanding (SLU) requires a model to analyze input&#10;acoustic signal to understand its linguistic content and make predictions. To&#10;boost the models' performance, various pre-training methods have been proposed&#10;to learn rich representations from large-scale unannotated speech and text.&#10;However, the inherent disparities between the two modalities necessitate a&#10;mutual analysis. In this paper, we propose a novel semi-supervised learning&#10;framework, SPLAT, to jointly pre-train the speech and language modules. Besides&#10;conducting a self-supervised masked language modeling task on the two&#10;individual modules using unpaired speech and text, SPLAT aligns representations&#10;from the two modules in a shared latent space using a small amount of paired&#10;speech and text. Thus, during fine-tuning, the speech module alone can produce&#10;representations carrying both acoustic information and contextual semantic&#10;knowledge of an input acoustic signal. Experimental results verify the&#10;effectiveness of our approach on various SLU tasks. For example, SPLAT improves&#10;the previous state-of-the-art performance on the Spoken SQuAD dataset by more&#10;than 10%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.11559" label="1910.11559">
        <attvalues>
          <attvalue for="0" value="SpeechBERT: An Audio-and-text Jointly Learned Language Model for&#10;  End-to-end Spoken Question Answering" />
          <attvalue for="1" value="  While various end-to-end models for spoken language understanding tasks have&#10;been explored recently, this paper is probably the first known attempt to&#10;challenge the very difficult task of end-to-end spoken question answering&#10;(SQA). Learning from the very successful BERT model for various text processing&#10;tasks, here we proposed an audio-and-text jointly learned SpeechBERT model.&#10;This model outperformed the conventional approach of cascading ASR with the&#10;following text question answering (TQA) model on datasets including ASR errors&#10;in answer spans, because the end-to-end model was shown to be able to extract&#10;information out of audio data before ASR produced errors. When ensembling the&#10;proposed end-to-end model with the cascade architecture, even better&#10;performance was achieved. In addition to the potential of end-to-end SQA, the&#10;SpeechBERT can also be considered for many other spoken language understanding&#10;tasks just as BERT for many text processing tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.04734" label="2107.04734">
        <attvalues>
          <attvalue for="0" value="Layer-wise Analysis of a Self-supervised Speech Representation Model" />
          <attvalue for="1" value="  Recently proposed self-supervised learning approaches have been successful&#10;for pre-training speech representation models. The utility of these learned&#10;representations has been observed empirically, but not much has been studied&#10;about the type or extent of information encoded in the pre-trained&#10;representations themselves. Developing such insights can help understand the&#10;capabilities and limits of these models and enable the research community to&#10;more efficiently develop their usage for downstream applications. In this work,&#10;we begin to fill this gap by examining one recent and successful pre-trained&#10;model (wav2vec 2.0), via its intermediate representation vectors, using a suite&#10;of analysis tools. We use the metrics of canonical correlation, mutual&#10;information, and performance on simple downstream tasks with non-parametric&#10;probes, in order to (i) query for acoustic and linguistic information content,&#10;(ii) characterize the evolution of information across model layers, and (iii)&#10;understand how fine-tuning the model for automatic speech recognition (ASR)&#10;affects these observations. Our findings motivate modifying the fine-tuning&#10;protocol for ASR, which produces improved word error rates in a low-resource&#10;setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;There has been extensive work on analyzing supervised speech models~\cite{belinkov2019analysis,palaskar2019learned,prasad2020accents}, but research on analyzing SSL models has been limited. Some very recent work has explored the phonetic, paralinguistic, and semantic content in SSL models using classifier probes~\cite{hsu2021hubert,baevski2021unsupervised,ma2021probing,shah2021all} and relationships between models with different training objectives and architectures~\cite{chung2020similarity}.&#10;The 2021 Zero Resource Speech Benchmark~\cite{nguyen2020zero} introduces zero-shot analysis datasets and metrics to evaluate the ability of SSL speech representations to encode different levels of linguistic information. While we share much of the motivation of \cite{ma2021probing,shah2021all,nguyen2020zero}, we focus on layer-wise analysis of a range of acoustic-linguistic content using lightweight methods that don't rely on training classifiers or collecting any additional labels for analysis, making it easier to scale.&#10;&#10;Layer-wise analysis of linguistic structure has also been done before for visually grounded speech~\cite{chrupala2017representations} and SSL text models~\cite{tenney2019bert}.&#10;Our methods of canonical correlation analysis (CCA) and discrete mutual information (MI) estimates are closest to Voita {et al.}'s work on text models~\cite{voita2019bottom}. MI has also been used to analyze supervised ASR models~\cite{prasad2020accents}. Unlike prior work, we apply these methods to the analysis of the relationship between representations and both discrete labels and continuous embeddings, and between representations from pre-trained and fine-tuned models. To our knowledge, this is the first work to analyze an SSL speech model on a range of linguistic properties using non-parametric probes.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2403.06988" label="2403.06988">
        <attvalues>
          <attvalue for="0" value="Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation" />
          <attvalue for="1" value="  To ensure that text generated by large language models (LLMs) is in an&#10;expected format, constrained decoding proposes to enforce strict formal&#10;language constraints during generation. However, as we show in this work, not&#10;only do such methods incur performance overhead during generation, but many of&#10;them also significantly impair task accuracy, if they do not correctly align&#10;the underlying LLM sub-word vocabularies with external constraints. To address&#10;this, we present a novel decoding algorithm, DOMINO, that can enforce&#10;constraints in a fully subword-aligned fashion, while leveraging&#10;pre-computation and speculative decoding to achieve virtually no overhead and&#10;in some cases even almost 2$\times$ speedup over unconstrained decoding --&#10;thereby outperforming existing approaches by a wide margin.&#10;" />
          <attvalue for="2" value=" &#10;&#10;The recent success of Large Language Models (LLMs) \cite{BrownMRSKDSKSSA20,ChenTJYPKEBP21,gpt4,TouvronLIMLLRAGJGL23,TouvronMSAAB23,Gemini,JiangRSBLSBCLSSSALBGLLSSYATLWLE23} has lead to the development of various methods that facilitate constrained generation, a method that lets users tailor the output of an LLM to a specific task or format.&#10;&#10;Constrained Decoding&#10;To ensure that text generated by an LLM adheres to syntactic constraints, these methods restrict the decoding procedure of an LLM in a way that only permits syntactically valid tokens at each sampling step. Doing so, the generated text can be ensured to adhere to constraints like high-level templates \cite{Beurer-Kellner023,guidance}, regular expressions \cite{Beurer-Kellner023,guidance,WillardL23} or context-free grammars \cite{WillardL23,guidance}.&#10;Constrained decoding provides numerous upsides. For instance, it guarantees that generated output will always adhere to the specified syntactic constraints, and reduces the need for ad-hoc parsing, retrying and prompting on top of LLMs. This facilitates the use of LLMs as part of larger pipelines or very specific tasks, without the need for fine-tuning or additional post-processing.&#10;&#10;Key Challenge: Token Misalignment Since LLM sub-word token do not align directly with most given syntactic constraints, the key challenge in constrained decoding is to interface the LLM vocabulary with a syntactic constraint like all output should be valid JSON.&#10;We showcase this in \ref{fig:intro}: While in an unconstrained setting, the LLM picks \ttoken{·&quot;} as the fourth token during generation, naively restricting the LLM to only immediatly valid JSON grammar terminals like just \ttoken{&quot;} or \ttoken{·}, leads to the less optimal choice of \ttoken{\tabchar} instead. By introducing such sub-optimal tokens, the distribution of a badly-constrained LLM can easily diverge from the unconstrained case, leading to a significant decrease in reasoning performance and therefore downstream accuracy. Here, the naively constrained LLM produces various high perplexity tokens, indicating that the model likely would not have chosen them otherwise (highlighted in \ref{fig:intro}). This is because naive constraining does not account for bridge tokens that may span multiple parser terminals in the underlying grammar (e.g., whitespace and double quotes, in this example). While existing work on code generation has made this observation before \cite{PoesiaP00SMG22}, solving this problem efficiently remains challenging, as the online computation of all bridge tokens at each decoding step, can be too costly in high-throughput environments.&#10;&#10;This work: Efficient, Minimally-Invasive Constraining&#10;In this work, we study the token misalignment problem outlined above, and examine its consequences for constrained decoding, showing empirically that misalignment can lead to a significant decrease in downstream accuracy.&#10;Based on this observation, we propose the notion of minimally invasive constrained decoding: A form of constraining that enforces a grammar, but also intervenes as little as possible during generation, avoiding token misalignment and optimizing for faithful, low-perplexity model output.&#10;&#10;Based on this, we propose a novel constrained decoding algorithm, \tool, which can enforce context-free grammars in a minimally-invasive way. In contrast to existing methods, \tool is highly efficient and incurs little to no overhead, and in many cases even increases the throughput of LLM inference over unconstrained generation, by leveraging pre-computation \cite{WillardL23} and a novel speculative decoding procedure for constrained decoding.&#10;We compare \tool with other approaches in \ref{tab:constrained_decoding_methods}.&#10;&#10;Main contributions&#10;In summary, our key contributions are:&#10;\begin{itemize}&#10; \item We identify the challenges of constrained decoding, most notably the correct and efficient alignment of sub-word tokens and grammar terminals (\ref{sec:token_misalignment}).&#10; \item We propose \tool, a novel constrained decoding algorithm, that addresses token misalignment and leverages pre-computation and speculative decoding for very low overhead generation (\ref{sec:technical}).&#10; \item An extensive evaluation thats shows that \tool is minimally-invasive, low-overhead, significantly outperforms other methods, and even exceeds unconstrained generation throughput in many cases (\ref{sec:eval}).&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Constrained Decoding Methods, Mathematics, Natural Language Generation, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2212.06094" label="2212.06094">
        <attvalues>
          <attvalue for="0" value="Prompting Is Programming: A Query Language for Large Language Models" />
          <attvalue for="1" value="  Large language models have demonstrated outstanding performance on a wide&#10;range of tasks such as question answering and code generation. On a high level,&#10;given an input, a language model can be used to automatically complete the&#10;sequence in a statistically-likely way. Based on this, users prompt these&#10;models with language instructions or examples, to implement a variety of&#10;downstream tasks. Advanced prompting methods can even imply interaction between&#10;the language model, a user, and external tools such as calculators. However, to&#10;obtain state-of-the-art performance or adapt language models for specific&#10;tasks, complex task- and model-specific programs have to be implemented, which&#10;may still require ad-hoc interaction.&#10;  Based on this, we present the novel idea of Language Model Programming (LMP).&#10;LMP generalizes language model prompting from pure text prompts to an intuitive&#10;combination of text prompting and scripting. Additionally, LMP allows&#10;constraints to be specified over the language model output. This enables easy&#10;adaption to many tasks while abstracting language model internals and providing&#10;high-level semantics.&#10;  To enable LMP, we implement LMQL(short for Language Model Query Language),&#10;which leverages the constraints and control flow from an LMP prompt to generate&#10;an efficient inference procedure that minimizes the number of expensive calls&#10;to the underlying language model.&#10;  We show that LMQL can capture a wide range of state-of-the-art prompting&#10;methods in an intuitive way, especially facilitating interactive flows that are&#10;challenging to implement with existing high-level APIs. Our evaluation shows&#10;that we retain or increase the accuracy on several downstream tasks, while also&#10;significantly reducing the required amount of computation or cost in the case&#10;of pay-to-use APIs (26-85% cost savings).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.09702" label="2307.09702">
        <attvalues>
          <attvalue for="0" value="Efficient Guided Generation for Large Language Models" />
          <attvalue for="1" value="  In this article we show how the problem of neural text generation can be&#10;constructively reformulated in terms of transitions between the states of a&#10;finite-state machine. This framework leads to an efficient approach to guiding&#10;text generation with regular expressions and context-free grammars by allowing&#10;the construction of an index over a language model's vocabulary. The approach&#10;is model agnostic, allows one to enforce domain-specific knowledge and&#10;constraints, and enables the construction of reliable interfaces by&#10;guaranteeing the structure of the generated text. It adds little overhead to&#10;the token sequence generation process and significantly outperforms existing&#10;solutions. An implementation is provided in the open source Python library&#10;Outlines&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.11227" label="2201.11227">
        <attvalues>
          <attvalue for="0" value="Synchromesh: Reliable code generation from pre-trained language models" />
          <attvalue for="1" value="  Large pre-trained language models have been used to generate code,providing a&#10;flexible interface for synthesizing programs from natural language&#10;specifications. However, they often violate syntactic and semantic rules of&#10;their output language, limiting their practical usability. In this paper, we&#10;propose Synchromesh: a framework for substantially improving the reliability of&#10;pre-trained models for code generation. Synchromesh comprises two components.&#10;First, it retrieves few-shot examples from a training bank using Target&#10;Similarity Tuning (TST), a novel method for semantic example selection. TST&#10;learns to recognize utterances that describe similar target programs despite&#10;differences in surface natural language features. Then, Synchromesh feeds the&#10;examples to a pre-trained language model and samples programs using Constrained&#10;Semantic Decoding (CSD): a general framework for constraining the output to a&#10;set of valid programs in the target language. CSD leverages constraints on&#10;partial outputs to sample complete correct programs, and needs neither&#10;re-training nor fine-tuning of the language model. We evaluate our methods by&#10;synthesizing code from natural language descriptions using GPT-3 and Codex in&#10;three real-world languages: SQL queries, Vega-Lite visualizations and SMCalFlow&#10;programs. These domains showcase rich constraints that CSD is able to enforce,&#10;including syntax, scope, typing rules, and contextual logic. We observe&#10;substantial complementary gains from CSD and TST in prediction accuracy and in&#10;effectively preventing run-time errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.19041" label="2405.19041">
        <attvalues>
          <attvalue for="0" value="BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge&#10;  Distillation" />
          <attvalue for="1" value="  Recent end-to-end approaches have shown promise in extending large language&#10;models (LLMs) to speech inputs, but face limitations in directly assessing and&#10;optimizing alignment quality and fail to achieve fine-grained alignment due to&#10;speech-text length mismatch. We introduce BLSP-KD, a novel approach for&#10;Bootstrapping Language-Speech Pretraining via Knowledge Distillation, which&#10;addresses these limitations through two key techniques. First, it optimizes&#10;speech-text alignment by minimizing the divergence between the LLM's next-token&#10;prediction distributions for speech and text inputs using knowledge&#10;distillation. Second, it employs a continuous-integrate-andfire strategy to&#10;segment speech into tokens that correspond one-to-one with text tokens,&#10;enabling fine-grained alignment. We also introduce Partial LoRA (PLoRA), a new&#10;adaptation method supporting LLM finetuning for speech inputs under knowledge&#10;distillation. Quantitative evaluation shows that BLSP-KD outperforms previous&#10;end-to-end baselines and cascaded systems with comparable scale of parameters,&#10;facilitating general instruction-following capabilities for LLMs with speech&#10;inputs. This approach provides new possibilities for extending LLMs to spoken&#10;language interactions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;We begin by formulating the problem definition. Consider an existing LLM trained solely on text, represented by parameters $\phi$. When prompted by a text input $\mathbf{x}$, this LLM can generate a response according to distribution $p_{\phi}(\cdot | \mathbf{x})$. Our objective is to extend the LLM's understanding and generation capabilities to speech inputs. Assume $\mathbf{x}$ is the transcript of a speech input $\mathbf{s}$. Typically, the speech is first transformed into feature representations $\mathbf{s}^enc$ by a speech encoder with parameters $\psi$, and then mapped through a modality adapter module with parameters $\theta$ to hidden states $\mathbf{s}^adp$ in the word embedding space, prompting the LLM to generate responses according to distribution $p_{\psi, \theta, \phi}(\cdot|\mathbf{s}^adp)$. &#10;&#10;Previous methods address this problem by fine-tuning on a collection $\mathcal{D}=\{(\mathbf{s},\mathbf{y})\}$ comprised of pairs of speech input $\mathbf{s}$ and the desired text response $\mathbf{y}$. Given the limited availability of manually annotated speech instruction datasets, the recent BLSP approach suggests to align speech and text via behavior alignment. It constructs synthesized responses from an existing ASR dataset $\mathcal{D} = \{(\mathbf{s},\mathbf{x})\}$ of speech and transcript pairs using a continuation prompt $\mathbf{c}$, expanding it to a collection of tuples $\mathcal{D} = \{(\mathbf{s},\mathbf{x}, \mathbf{y})\}$ with each $\mathbf{y}$ generated via greedy-search as follows:&#10;\begin{align}&#10;\mathbf{y} = \arg\max_{\mathbf{y}'} p(\mathbf{y'} |\mathbf{x})&#10;\end{align}&#10;&#10;Here, and when clear from the context, the continuation prompt $\mathbf{c}$ is used as a condition but is not explicitly included to maintain clarity in presentation. &#10;&#10;The parameter set $\theta$ of the modality adapter module is then optimized to minimize the cross entropy loss of responses as in instruction fine-tuning:&#10;&#10;\begin{align}&#10;\ell^resp_CE (\mathbf{s}, \mathbf{y}) &amp;=-\log p_\theta(\mathbf{y}|\mathbf{s}^adp) \\ &#10;&amp;= -\sum_j\log p_\theta(y_j|\mathbf{s}^adp, \mathbf{y}_{&lt;j}) &#10;\end{align}&#10;&#10;Despite achieving remarkable instruction-following capabilities for speech inputs, the BLSP \cite{wang2023blsp} approach encounters two limitations. First, while the continuation instruction aims to generate diverse responses that mimic general next-token prediction, it represents only one specific instruction and thus captures only a subset of the LLM's generative behavior. Second, the cross-entropy loss encourages the model to replicate the 1-best responses with high probability and thus neglect the broader distribution of possible responses. The cross-entropy loss $\ell^resp_CE (\mathbf{s}, \mathbf{y})$, defined between speech $\mathbf{s}$ and 1-best response $\mathbf{y}$, does not provide an end-to-end differential measure of alignment quality between speech $\mathbf{s}$ and transcript $\mathbf{x}$.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Knowledge Distillation, Speech-Text Alignment, Artificial Intelligence, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2212.10378" label="2212.10378">
        <attvalues>
          <attvalue for="0" value="Data Curation Alone Can Stabilize In-context Learning" />
          <attvalue for="1" value="  In-context learning (ICL) enables large language models (LLMs) to perform new&#10;tasks by prompting them with a sequence of training examples. However, it is&#10;known that ICL is very sensitive to the choice of training examples: randomly&#10;sampling examples from a training set leads to high variance in performance. In&#10;this paper, we show that carefully curating a subset of training data greatly&#10;stabilizes ICL performance without any other changes to the ICL algorithm&#10;(e.g., prompt retrieval or calibration). We introduce two methods to choose&#10;training subsets -- both score training examples individually, then select the&#10;highest-scoring ones. CondAcc scores a training example by its average dev-set&#10;ICL accuracy when combined with random training examples, while Datamodels&#10;learns linear regressors that estimate how the presence of each training&#10;example influences LLM outputs. Across five tasks and two LLMs, sampling from&#10;stable subsets selected by CondAcc and Datamodels improves average accuracy&#10;over sampling from the entire training set by 7.7% and 6.3%, respectively.&#10;Surprisingly, the stable subset examples are not especially diverse in content&#10;or low in perplexity, in contrast with other work suggesting that diversity and&#10;perplexity are important when prompting LLMs.&#10;" />
          <attvalue for="2" value="&#10;In-context learning (ICL) is a new paradigm for few-shot learning with pretrained large language models (LLMs) without any parameter updates.&#10;In ICL, an LLM can perform a new task simply by conditioning on a prompt consisting of a sequence of labeled training examples.&#10;First introduced by GPT-3~\cite{brown2020language}, ICL with LLMs has reached state-of-the-art few-shot performance across many tasks~\cite{rae2021scaling,smith2022using,thoppilan2022lamda,chowdhery2022palm}.&#10;Compared with alternatives that use fine-tuning~\cite{devlin2018bert,schick2020exploiting,gao2020making}, ICL does not require task-specific training, which enables its use with very large language models, &#10;and it uses a unified model for all tasks, enabling easier deployment.&#10;&#10;Despite its impressive few-shot performance, ICL often exhibits unintuitive behavior~\cite{min2022rethinking}.&#10;The standard ICL approach is to randomly sample a few examples from a training set to construct a prompt~\cite{brown2020language}; however, prior work~\cite{liu2021makes,zhao2021calibrate,lu2021fantastically} has found that ICL is very sensitive to the choice of training examples and their order in the prompt.&#10;ICL is also sensitive to small changes in prompt format~\cite{chen2022relation}.&#10;&#10;In this paper, we show that carefully curating a smaller training dataset from a larger pool can make ICL much more stable.&#10;We define a training subset $\mathcal{E}$ to be stable\ if randomly sampling a sequence of examples as a prompt from $\mathcal{E}$ yields much higher average and worst-case accuracy than randomly sampling from the original training set.&#10;We propose two methods to identify such a stable subset.&#10;Our CondAcc\ method scores a training example by its average dev-set ICL accuracy when combined with random training examples; these scores are closely related to Data Shapley values~\cite{ghorbani2019data}.&#10;Our Datamodels\ method fits a linear regressor that predicts the LLM's output based on which example is present at each index in the prompt;&#10;we score a training example highly if the associated weights from the linear model indicate that its presence improves accuracy.&#10;For both methods, we then select training examples with the highest scores to form the stable subset.&#10;While some prior work improves ICL accuracy by retrieving a suitable prompt for each test example~\cite{liu2021makes,rubin2021learning,su2022selective}, we show that it is possible to achieve stably good accuracy with a randomly sampled prompt for all test examples, when given the ``right'' training (sub)set.&#10;&#10;Our subset selection methods greatly improve performance across 5 classification datasets and 4 LLMs, with main experiments on GPTJ-6B~\cite{gpt-j} and OPT-13B~\cite{zhang2022opt}.&#10;On average, CondAcc\ and Datamodels\ outperform the baseline that uses the entire training set without selection (named All) by 7.7\% and 6.3\%, respectively, when comparing the average accuracy over multiple sampled prompts.&#10;In contrast, baselines that choose examples found in high-performing prompts (TopPrompts) or examples that lead to high one-shot ICL accuracy (OneShot) to form the subsets do not perform as well (see Figure~\ref{fig:fig1}).&#10;Our stable subset examples generalize to out-of-distribution test data, and &#10;we can even find stable subsets for binary classification tasks that only contain examples of one label;&#10;these findings suggest that the stable subset examples help LLMs understand the overall task definition.&#10;&#10;Finally, we study what makes stable subset examples special by analyzing sequence length, perplexity, and diversity in both raw text and embedding spaces.&#10;We find that these examples do not have abnormally long sequence lengths or high perplexities.&#10;In contrast with prior work optimizing diversity for prompt selection~\cite{su2022selective,Ye2022ComplementaryEF}, we find our stable subsets no more diverse than random subsets of the training data. &#10;In summary, we show that curating training data appropriately leads to more stable and accurate ICL performance; we hope future work can develop new strategies for writing such helpful examples.&#10;Code and data are publicly available at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, In-Context Learning, Training Data Curation, Artificial Intelligence, Mathematics, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2209.07661" label="2209.07661">
        <attvalues>
          <attvalue for="0" value="On the Relation between Sensitivity and Accuracy in In-context Learning" />
          <attvalue for="1" value="  In-context learning (ICL) suffers from oversensitivity to the prompt, making&#10;it unreliable in real-world scenarios. We study the sensitivity of ICL with&#10;respect to multiple perturbation types. First, we find that label bias obscures&#10;the true sensitivity, and therefore prior work may have significantly&#10;underestimated ICL sensitivity. Second, we observe a strong negative&#10;correlation between ICL sensitivity and accuracy: predictions sensitive to&#10;perturbations are less likely to be correct. Motivated by these findings, we&#10;propose \textsc{SenSel}, a few-shot selective prediction method that abstains&#10;from sensitive predictions. Experiments on ten classification datasets show&#10;that \textsc{SenSel} consistently outperforms two commonly used&#10;confidence-based and entropy-based baselines on abstention decisions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.13892" label="2211.13892">
        <attvalues>
          <attvalue for="0" value="Complementary Explanations for Effective In-Context Learning" />
          <attvalue for="1" value="  Large language models (LLMs) have exhibited remarkable capabilities in&#10;learning from explanations in prompts, but there has been limited understanding&#10;of exactly how these explanations function or why they are effective. This work&#10;aims to better understand the mechanisms by which explanations are used for&#10;in-context learning. We first study the impact of two different factors on the&#10;performance of prompts with explanations: the computation trace (the way the&#10;solution is decomposed) and the natural language used to express the prompt. By&#10;perturbing explanations on three controlled tasks, we show that both factors&#10;contribute to the effectiveness of explanations. We further study how to form&#10;maximally effective sets of explanations for solving a given test query. We&#10;find that LLMs can benefit from the complementarity of the explanation set:&#10;diverse reasoning skills shown by different exemplars can lead to better&#10;performance. Therefore, we propose a maximal marginal relevance-based exemplar&#10;selection approach for constructing exemplar sets that are both relevant as&#10;well as complementary, which successfully improves the in-context learning&#10;performance across three real-world tasks on multiple LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.12607" label="1910.12607">
        <attvalues>
          <attvalue for="0" value="Generative Pre-Training for Speech with Autoregressive Predictive Coding" />
          <attvalue for="1" value="  Learning meaningful and general representations from unannotated speech that&#10;are applicable to a wide range of tasks remains challenging. In this paper we&#10;propose to use autoregressive predictive coding (APC), a recently proposed&#10;self-supervised objective, as a generative pre-training approach for learning&#10;meaningful, non-specific, and transferable speech representations. We pre-train&#10;APC on large-scale unlabeled data and conduct transfer learning experiments on&#10;three speech applications that require different information about speech&#10;characteristics to perform well: speech recognition, speech translation, and&#10;speaker identification. Extensive experiments show that APC not only&#10;outperforms surface features (e.g., log Mel spectrograms) and other popular&#10;representation learning methods on all three tasks, but is also effective at&#10;reducing downstream labeled data size and model parameters. We also investigate&#10;the use of Transformers for modeling APC and find it superior to RNNs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.12971" label="2102.12971">
        <attvalues>
          <attvalue for="0" value="Are pre-trained text representations useful for multilingual and&#10;  multi-dimensional language proficiency modeling?" />
          <attvalue for="1" value="  Development of language proficiency models for non-native learners has been&#10;an active area of interest in NLP research for the past few years. Although&#10;language proficiency is multidimensional in nature, existing research typically&#10;considers a single &quot;overall proficiency&quot; while building models. Further,&#10;existing approaches also considers only one language at a time. This paper&#10;describes our experiments and observations about the role of pre-trained and&#10;fine-tuned multilingual embeddings in performing multi-dimensional,&#10;multilingual language proficiency classification. We report experiments with&#10;three languages -- German, Italian, and Czech -- and model seven dimensions of&#10;proficiency ranging from vocabulary control to sociolinguistic appropriateness.&#10;Our results indicate that while fine-tuned embeddings are useful for&#10;multilingual proficiency modeling, none of the features achieve consistently&#10;best performance for all dimensions of language proficiency. All code, data and&#10;related supplementary material can be found at:&#10;https://github.com/nishkalavallabhi/MultidimCEFRScoring.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Automated Essay Scoring (AES) is a well-researched problem in NLP and has been applied to real-world language assessment scenarios for English \cite{Attali.Bustein-04}. A wide range of features such as document length, lexical/syntactic n-grams, and features capturing linguistic aspects such as vocabulary, syntax and discourse are commonly used \cite{Klebanov.Flor-13,Phandi.Chai.ea-15,Zesch.Wojatzki.ea-15}. In the recent past, different forms of text embeddings and pre-trained language models have also been explored \cite{Alikaniotis.Yannakoudakis.ea-16,Dong.Zhang-16,Mayfield.Black-20} along with approaches to combine linguistic features with neural networks \cite{Shin-18,Liu.Xu.ea-19}. \newcite{Ke.Ng-19} and \newcite{Klebanov.Madnani-20} present the most recent surveys on the state of the art in AES (focusing on English).&#10;&#10;In terms of modeling, AES has been modeled as a classification, regression, and ranking problem, with approaches ranging from linear regression to deep learning models. Some of the recent work explored the usefulness of multi-task learning \cite{Cummins.Rei-18,Berggren.Rama.ea-19} and transfer learning \cite{Jin.He.ea-18,Ballier.Canu.ea-20}. Going beyond approaches that work for a single language, \newcite{Vajjala.Rama-18} reported on developing methods for multi- and cross-lingual AES. &#10;&#10;Much of the existing AES research has been focused on English, but there is a growing body of research on other European languages: German \cite{Hancke.Meurers-13}, Estonian \cite{Vajjala.Loo-14}, Swedish \cite{Pilan.Alfter.ea-16}, Norwegian \cite{Berggren.Rama.ea-19} which explored both language specific (e.g., case markers in Estonian) as well as language agnostic (e.g., POS n-grams) features \cite{Vajjala.Rama-18} for this task. However, to our knowledge, the use of large pre-trained language models such as BERT \cite{Devlin.Chang.ea-18} has not been explored yet for AES in non-English languages. &#10;&#10;Further, most of the approaches focused on modeling language proficiency as a single variable. Although there is some research focusing on multiple dimensions of language proficiency \cite{Lee.Gentile.ea-09,Attali.Sinharay-15,Agejev.Snajder-17,Mathias.Bhattacharyya-20}, none of them focused on non-English languages or used recent multilingual pre-trained models such as BERT. In this paper, we focus on this problem of multi-dimensional modeling of language proficiency for three languages---German, Italian, and Czech---and explore whether recent research on multilingual embeddings can be useful for non-English AES. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Language Proficiency Modeling, Natural Language Processing, Multilingual Embeddings" />
        </attvalues>
      </node>
      <node id="2003.07278" label="2003.07278">
        <attvalues>
          <attvalue for="0" value="A Survey on Contextual Embeddings" />
          <attvalue for="1" value="  Contextual embeddings, such as ELMo and BERT, move beyond global word&#10;representations like Word2Vec and achieve ground-breaking performance on a wide&#10;range of natural language processing tasks. Contextual embeddings assign each&#10;word a representation based on its context, thereby capturing uses of words&#10;across varied contexts and encoding knowledge that transfers across languages.&#10;In this survey, we review existing contextual embedding models, cross-lingual&#10;polyglot pre-training, the application of contextual embeddings in downstream&#10;tasks, model compression, and model analyses.&#10;" />
          <attvalue for="2" value="&#10;Distributional word representations \cite{turian2010word,mikolov2013efficient,pennington2014glove} trained in an unsupervised manner on large-scale corpora are widely used in modern natural language processing systems. However, these approaches only obtain a single global representation for each word, ignoring their context. Different from traditional word representations, contextual embeddings move beyond word-level semantics in that each token is associated with a representation that is a function of the entire input&#10;sequence. These context-dependent representations can capture many syntactic and semantic properties of words under diverse linguistic contexts. Previous work \cite{peters2018deep,devlin2018bert,yang2019xlnet,raffel2019exploring} has shown that contextual embeddings pre-trained on large-scale unlabelled corpora achieve state-of-the-art performance on a wide range of natural language processing tasks, such as text classification, question answering and text summarization. Further analyses \cite{liu2019linguistic,hewitt-liang-2019-designing,hewitt2019structural,tenney2019bert} demonstrate that contextual embeddings are capable of learning useful and transferable representations across languages. &#10;&#10;The rest of the survey is organized as follows. In Section \ref{sec:preliminary}, we define the concept of contextual embeddings. In Section \ref{sec:train_method}, we introduce existing methods for obtaining contextual embeddings. In Section \ref{sec:cross_lingual_method}, we present the pre-training methods of contextual embeddings on multi-lingual corpora. In Section \ref{sec:transfer_method}, we describe methods for applying pre-trained contextual embeddings in downstream tasks. In Section \ref{sec:compression_method}, we detail model compression methods. In Section \ref{sec:analysis}, we survey analyses that have aimed to identify the linguistic knowledge learned by contextual embeddings. We conclude the survey by highlighting some challenges for future research in Section \ref{sec:challenges}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Model Analysis Techniques, Linguistics, Contextual Embeddings, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2302.01398" label="2302.01398">
        <attvalues>
          <attvalue for="0" value="The unreasonable effectiveness of few-shot learning for machine&#10;  translation" />
          <attvalue for="1" value="  We demonstrate the potential of few-shot translation systems, trained with&#10;unpaired language data, for both high and low-resource language pairs. We show&#10;that with only 5 examples of high-quality translation data shown at inference,&#10;a transformer decoder-only model trained solely with self-supervised learning,&#10;is able to match specialized supervised state-of-the-art models as well as more&#10;general commercial translation systems. In particular, we outperform the best&#10;performing system on the WMT'21 English - Chinese news translation task by only&#10;using five examples of English - Chinese parallel data at inference. Moreover,&#10;our approach in building these models does not necessitate joint multilingual&#10;training or back-translation, is conceptually simple and shows the potential to&#10;extend to the multilingual setting. Furthermore, the resulting models are two&#10;orders of magnitude smaller than state-of-the-art language models. We then&#10;analyze the factors which impact the performance of few-shot translation&#10;systems, and highlight that the quality of the few-shot demonstrations heavily&#10;determines the quality of the translations generated by our models. Finally, we&#10;show that the few-shot paradigm also provides a way to control certain&#10;attributes of the translation -- we show that we are able to control for&#10;regional varieties and formality using only a five examples at inference,&#10;paving the way towards controllable machine translation systems.&#10;" />
          <attvalue for="2" value="&#10;&#10;Current state-of-the-art machine translation systems are typically built by leveraging vast amounts of parallel data mined from the web. While this is practical for high-resource language pairs, it is unfeasible to obtain corpora of such sizes for the majority of languages in the world. Moreover, the reliance of mined parallel data has many potential downsides, such as allowing for poisoning attacks \cite{xu2021targeted,wang2021putting}, memorization of low-quality examples \cite{raunak2022finding}, and biases towards generating text in language registers over-represented in the parallel data, such as language varieties \cite{lakew2018neural,riley2022frmt} or formality \cite{rippeth-etal-2022-controlling}.&#10;&#10;As an alternative, researchers began exploring the task of unsupervised translation \cite{ravi2011deciphering} i.e. building translation models without any parallel data at all. Unsupervised translation systems have demonstrated promising performance in recent years, able to match strong supervised baselines on academic benchmarks \cite{song2019mass,garcia2020harnessing,han2021unsupervised} by relying on a collection of tricks and techniques, such as multilinguality \cite{conneau2019unsupervised, garcia2020harnessing, garcia2020multilingual,lin2021few}, back-translation \cite{lample2017unsupervised}, and most recently through large-scale models with parameters in the hundreds of billions \cite{chowdhery2022palm,vilar2022prompting,han2021unsupervised}. Despite such results, these systems are rarely compared to state-of-the-art supervised models which also leverage their own bag of tricks to improve performance. Moreover, these models trained on large-scale monolingual data also suffer from similar issues as in models trained on large-scale parallel data: variance in the quality of monolingual data and generation being biased towards the over-represented registers in the monolingual data.&#10;&#10;It thus remains to explore what lies between these two research streams: few-shot learning \cite{brown2020language}. Recently, large language models have been shown capable of performing arbitrary tasks by exposing a few demonstrations of the task at inference time. The dependence of these models on demonstrations to perform the task allows us to overcome the aforementioned issues by carefully selecting a small set of high-quality translation pairs in the language register of interest as demonstrations.&#10;&#10;In this work, we demonstrate that the few-shot translation paradigm allows us to build high-fidelity translation models at a smaller scale (8B parameters) than traditional large language models ($&gt;100$B parameters) without the need for back-translation or large-scale parallel text mining. We evaluate these models on the WMT'21 English\scalebox{0.7}[1.0]{$-$}German and English\scalebox{0.7}[1.0]{$-$}Chinese news translation task and show that they outperform commercial baselines, and show performance competitive with WMT'21 submissions, which themselves rely on many of the aforementioned techniques. We then verify that our approach works in low-resource scenarios by performing a similar study on the WMT'21 English\scalebox{0.7}[1.0]{$-$}Icelandic language pair, where the amount of Icelandic monolingual data is two orders of magnitude smaller than Chinese or German. Furthermore, we show that constraining the demonstrations to be in a desired language register generally results in the output translation being part of the register. We show that this results in quantitative gains in translation benchmarks which account for language registers: the region-aware benchmark FRMT \cite{riley2022frmt} and the IWSLT'22 Special Task on Formality Control for Spoken Language Translation \cite{anastasopoulos-etal-2022-findings}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Computer Science, Machine Translation, Linguistics, Artificial Intelligence, Natural Language Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="2011.00675" label="2011.00675">
        <attvalues>
          <attvalue for="0" value="A Targeted Attack on Black-Box Neural Machine Translation with Parallel&#10;  Data Poisoning" />
          <attvalue for="1" value="  As modern neural machine translation (NMT) systems have been widely deployed,&#10;their security vulnerabilities require close scrutiny. Most recently, NMT&#10;systems have been found vulnerable to targeted attacks which cause them to&#10;produce specific, unsolicited, and even harmful translations. These attacks are&#10;usually exploited in a white-box setting, where adversarial inputs causing&#10;targeted translations are discovered for a known target system. However, this&#10;approach is less viable when the target system is black-box and unknown to the&#10;adversary (e.g., secured commercial systems). In this paper, we show that&#10;targeted attacks on black-box NMT systems are feasible, based on poisoning a&#10;small fraction of their parallel training data. We show that this attack can be&#10;realised practically via targeted corruption of web documents crawled to form&#10;the system's training data. We then analyse the effectiveness of the targeted&#10;poisoning in two common NMT training scenarios: the from-scratch training and&#10;the pre-train &amp; fine-tune paradigm. Our results are alarming: even on the&#10;state-of-the-art systems trained with massive parallel data (tens of millions),&#10;the attacks are still successful (over 50% success rate) under surprisingly low&#10;poisoning budgets (e.g., 0.006%). Lastly, we discuss potential defences to&#10;counter such attacks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.05243" label="2107.05243">
        <attvalues>
          <attvalue for="0" value="Putting words into the system's mouth: A targeted attack on neural&#10;  machine translation using monolingual data poisoning" />
          <attvalue for="1" value="  Neural machine translation systems are known to be vulnerable to adversarial&#10;test inputs, however, as we show in this paper, these systems are also&#10;vulnerable to training attacks. Specifically, we propose a poisoning attack in&#10;which a malicious adversary inserts a small poisoned sample of monolingual text&#10;into the training set of a system trained using back-translation. This sample&#10;is designed to induce a specific, targeted translation behaviour, such as&#10;peddling misinformation. We present two methods for crafting poisoned examples,&#10;and show that only a tiny handful of instances, amounting to only 0.02% of the&#10;training set, is sufficient to enact a successful attack. We outline a defence&#10;method against said attacks, which partly ameliorates the problem. However, we&#10;stress that this is a blind-spot in modern NMT, demanding immediate attention.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.12929" label="2210.12929">
        <attvalues>
          <attvalue for="0" value="Finding Memo: Extractive Memorization in Constrained Sequence Generation&#10;  Tasks" />
          <attvalue for="1" value="  Memorization presents a challenge for several constrained Natural Language&#10;Generation (NLG) tasks such as Neural Machine Translation (NMT), wherein the&#10;proclivity of neural models to memorize noisy and atypical samples reacts&#10;adversely with the noisy (web crawled) datasets. However, previous studies of&#10;memorization in constrained NLG tasks have only focused on counterfactual&#10;memorization, linking it to the problem of hallucinations. In this work, we&#10;propose a new, inexpensive algorithm for extractive memorization (exact&#10;training data generation under insufficient context) in constrained sequence&#10;generation tasks and use it to study extractive memorization and its effects in&#10;NMT. We demonstrate that extractive memorization poses a serious threat to NMT&#10;reliability by qualitatively and quantitatively characterizing the memorized&#10;samples as well as the model behavior in their vicinity. Based on empirical&#10;observations, we develop a simple algorithm which elicits non-memorized&#10;translations of memorized samples from the same model, for a large fraction of&#10;such samples. Finally, we show that the proposed algorithm could also be&#10;leveraged to mitigate memorization in the model through finetuning. We have&#10;released the code to reproduce our results at&#10;https://github.com/vyraun/Finding-Memo.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.01064" label="1811.01064">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation into Language Varieties" />
          <attvalue for="1" value="  Both research and commercial machine translation have so far neglected the&#10;importance of properly handling the spelling, lexical and grammar divergences&#10;occurring among language varieties. Notable cases are standard national&#10;varieties such as Brazilian and European Portuguese, and Canadian and European&#10;French, which popular online machine translation services are not keeping&#10;distinct. We show that an evident side effect of modeling such varieties as&#10;unique classes is the generation of inconsistent translations. In this work, we&#10;investigate the problem of training neural machine translation from English to&#10;specific pairs of language varieties, assuming both labeled and unlabeled&#10;parallel texts, and low-resource conditions. We report experiments from English&#10;to two pairs of dialects, EuropeanBrazilian Portuguese and European-Canadian&#10;French, and two pairs of standardized varieties, Croatian-Serbian and&#10;Indonesian-Malay. We show significant BLEU score improvements over baseline&#10;systems when translation into similar languages is learned as a multilingual&#10;task with shared representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.06644" label="2205.06644">
        <attvalues>
          <attvalue for="0" value="Controlling Translation Formality Using Pre-trained Multilingual&#10;  Language Models" />
          <attvalue for="1" value="  This paper describes the University of Maryland's submission to the Special&#10;Task on Formality Control for Spoken Language Translation at \iwslt, which&#10;evaluates translation from English into 6 languages with diverse grammatical&#10;formality markers. We investigate to what extent this problem can be addressed&#10;with a \textit{single multilingual model}, simultaneously controlling its&#10;output for target language and formality. Results show that this strategy can&#10;approach the translation quality and formality control achieved by dedicated&#10;translation models. However, the nature of the underlying pre-trained language&#10;model and of the finetuning samples greatly impact results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.02450" label="1905.02450">
        <attvalues>
          <attvalue for="0" value="MASS: Masked Sequence to Sequence Pre-training for Language Generation" />
          <attvalue for="1" value="  Pre-training and fine-tuning, e.g., BERT, have achieved great success in&#10;language understanding by transferring knowledge from rich-resource&#10;pre-training task to the low/zero-resource downstream tasks. Inspired by the&#10;success of BERT, we propose MAsked Sequence to Sequence pre-training (MASS) for&#10;the encoder-decoder based language generation tasks. MASS adopts the&#10;encoder-decoder framework to reconstruct a sentence fragment given the&#10;remaining part of the sentence: its encoder takes a sentence with randomly&#10;masked fragment (several consecutive tokens) as input, and its decoder tries to&#10;predict this masked fragment. In this way, MASS can jointly train the encoder&#10;and decoder to develop the capability of representation extraction and language&#10;modeling. By further fine-tuning on a variety of zero/low-resource language&#10;generation tasks, including neural machine translation, text summarization and&#10;conversational response generation (3 tasks and totally 8 datasets), MASS&#10;achieves significant improvements over the baselines without pre-training or&#10;with other pre-training methods. Specially, we achieve the state-of-the-art&#10;accuracy (37.5 in terms of BLEU score) on the unsupervised English-French&#10;translation, even beating the early attention-based supervised model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.05448" label="2110.05448">
        <attvalues>
          <attvalue for="0" value="Unsupervised Neural Machine Translation with Generative Language Models&#10;  Only" />
          <attvalue for="1" value="  We show how to derive state-of-the-art unsupervised neural machine&#10;translation systems from generatively pre-trained language models. Our method&#10;consists of three steps: few-shot amplification, distillation, and&#10;backtranslation. We first use the zero-shot translation ability of large&#10;pre-trained language models to generate translations for a small set of&#10;unlabeled sentences. We then amplify these zero-shot translations by using them&#10;as few-shot demonstrations for sampling a larger synthetic dataset. This&#10;dataset is distilled by discarding the few-shot demonstrations and then&#10;fine-tuning. During backtranslation, we repeatedly generate translations for a&#10;set of inputs and then fine-tune a single language model on both directions of&#10;the translation task at once, ensuring cycle-consistency by swapping the roles&#10;of gold monotext and generated translations when fine-tuning. By using our&#10;method to leverage GPT-3's zero-shot translation capability, we achieve a new&#10;state-of-the-art in unsupervised translation on the WMT14 English-French&#10;benchmark, attaining a BLEU score of 42.1.&#10;" />
          <attvalue for="2" value="&#10;Recent work on generative pre-training has shown&#10;that with sufficient data and&#10;scale&#10;\cite{DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2010-14701},&#10;large language models (LMs) can learn a diverse suite of tasks without&#10;explicit supervision \cite{radford2019language}, and that even&#10;stronger performance on these tasks can be elicited using few-shot&#10;demonstrations \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}. While&#10;few-shot prompting is flexible and&#10;enables strong performance on a diverse suite of NLP tasks&#10;to be coaxed out of generatively pre-trained LMs without further&#10;fine-tuning, its benefits are&#10;most pronounced with larger models, with commensurate training,&#10;inference, compute, and data costs. Furthermore, the very generality of the&#10;pre-training objective which enables multi-task learning can produce LMs&#10;with more knowledge than is immediately apparent,&#10;requiring carefully designed prompts to bring out fully. The desire to&#10;unlock and amplify these latent abilities while also reducing the cost&#10;of few-shot prompting motivates our present work,&#10;which allows us to continue&#10;fine-tuning our models, obtaining more performance from&#10;smaller models and pushing our larger models even further, without&#10;resorting to few-shot prompting at test time or any additional supervision&#10;at train~time.&#10;&#10;We target the domain of unsupervised&#10; neural machine translation (NMT), which typically involves&#10;bootstrapping a weak translation model before amplifying its translation ability via&#10;backtranslation. Recent work in unsupervised NMT has been&#10;dominated by large encoder-decoder architectures where the bootstrap is&#10;implemented by denoising/autoencoding tasks&#10;(e.g., multilingual Cloze \cite{DBLP:conf/naacl/DevlinCLT19, DBLP:conf/nips/ConneauL19}, masked-span&#10;prediction \cite{DBLP:journals/jmlr/RaffelSRLNMZLL20, DBLP:conf/naacl/XueCRKASBR21},&#10;reconstruction from corrupted inputs&#10;\cite{DBLP:conf/emnlp/WangZJLL19, DBLP:journals/tacl/LiuGGLEGLZ20})&#10;intended to produce strong encoders and&#10;aligned multilingual representations for decoding.&#10;In our present work, we show that generative language modeling alone&#10;can implement the entire unsupervised NMT pipeline, and derive&#10;state-of-the-art unsupervised NMT systems using only generatively&#10;pre-trained language models.&#10;We implement the bootstrap by first sampling a small number of&#10;zero-shot translations from GPT-3. These are then used as&#10;few-shot prompts to sample a larger dataset of synthetic translations.&#10;The few-shot prompts are then discarded and the generated&#10;samples are distilled by fine-tuning the model on&#10;these synthetic data in the zero-shot format. This produces a language model&#10;aligned to our translation format and amenable to large-scale&#10;backtranslation. By using our method to leverage GPT-3's zero-shot&#10;translation capability, we achieve a new state-of-the-art in&#10;unsupervised translation on the WMT14 English-French benchmark,&#10;attaining a BLEU score of 42.1.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.00043" label="1711.00043">
        <attvalues>
          <attvalue for="0" value="Unsupervised Machine Translation Using Monolingual Corpora Only" />
          <attvalue for="1" value="  Machine translation has recently achieved impressive performance thanks to&#10;recent advances in deep learning and the availability of large-scale parallel&#10;corpora. There have been numerous attempts to extend these successes to&#10;low-resource language pairs, yet requiring tens of thousands of parallel&#10;sentences. In this work, we take this research direction to the extreme and&#10;investigate whether it is possible to learn to translate even without any&#10;parallel data. We propose a model that takes sentences from monolingual corpora&#10;in two different languages and maps them into the same latent space. By&#10;learning to reconstruct in both languages from this shared feature space, the&#10;model effectively learns to translate without using any labeled data. We&#10;demonstrate our model on two widely used datasets and two language pairs,&#10;reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French&#10;datasets, without using even a single parallel sentence at training time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.09102" label="2211.09102">
        <attvalues>
          <attvalue for="0" value="Prompting PaLM for Translation: Assessing Strategies and Performance" />
          <attvalue for="1" value="  Large language models (LLMs) that have been trained on multilingual but not&#10;parallel text exhibit a remarkable ability to translate between languages. We&#10;probe this ability in an in-depth study of the pathways language model (PaLM),&#10;which has demonstrated the strongest machine translation (MT) performance among&#10;similarly-trained LLMs to date. We investigate various strategies for choosing&#10;translation examples for few-shot prompting, concluding that example quality is&#10;the most important factor. Using optimized prompts, we revisit previous&#10;assessments of PaLM's MT capabilities with more recent test sets, modern MT&#10;metrics, and human evaluation, and find that its performance, while impressive,&#10;still lags that of state-of-the-art supervised systems. We conclude by&#10;providing an analysis of PaLM's MT output which reveals some interesting&#10;properties and prospects for future work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.08067" label="1904.08067">
        <attvalues>
          <attvalue for="0" value="Text Classification Algorithms: A Survey" />
          <attvalue for="1" value="  In recent years, there has been an exponential growth in the number of&#10;complex documents and texts that require a deeper understanding of machine&#10;learning methods to be able to accurately classify texts in many applications.&#10;Many machine learning approaches have achieved surpassing results in natural&#10;language processing. The success of these learning algorithms relies on their&#10;capacity to understand complex models and non-linear relationships within data.&#10;However, finding suitable structures, architectures, and techniques for text&#10;classification is a challenge for researchers. In this paper, a brief overview&#10;of text classification algorithms is discussed. This overview covers different&#10;text feature extractions, dimensionality reduction methods, existing algorithms&#10;and techniques, and evaluations methods. Finally, the limitations of each&#10;technique and their application in the real-world problem are discussed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.01252" label="2406.01252">
        <attvalues>
          <attvalue for="0" value="Towards Scalable Automated Alignment of LLMs: A Survey" />
          <attvalue for="1" value="  Alignment is the most critical step in building large language models (LLMs)&#10;that meet human needs. With the rapid development of LLMs gradually surpassing&#10;human capabilities, traditional alignment methods based on human-annotation are&#10;increasingly unable to meet the scalability demands. Therefore, there is an&#10;urgent need to explore new sources of automated alignment signals and technical&#10;approaches. In this paper, we systematically review the recently emerging&#10;methods of automated alignment, attempting to explore how to achieve effective,&#10;scalable, automated alignment once the capabilities of LLMs exceed those of&#10;humans. Specifically, we categorize existing automated alignment methods into 4&#10;major categories based on the sources of alignment signals and discuss the&#10;current status and potential development of each category. Additionally, we&#10;explore the underlying mechanisms that enable automated alignment and discuss&#10;the essential factors that make automated alignment technologies feasible and&#10;effective from the fundamental role of alignment.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years have witnessed the rapid advancements of large language models (LLMs), which have dramatically reshaped the landscape of artificial intelligence~\cite{ouyangTrainingLanguageModels2022b, touvron2023llama, openaiGPT4TechnicalReport2023}.&#10;Alignment is at the core of shaping behaviors of LLMs corresponding to human intentions and values~\cite{yaoInstructionsIntrinsicHuman2023a,shenLargeLanguageModel2023}, e.g., teaching LLMs to follow ``helpful, harmless and honest (HHH)'' principles during responding~\cite{askell2021general}. As a result, increasing efforts have been made for aligning LLMs to meet the human requirements, which makes it a hotspot research direction in LLM era~\cite{wangAligningLargeLanguage2023a, wangEssenceProspectInvestigation2024, jiAIAlignmentComprehensive2024}.&#10;&#10;Previous studies of alignment have primarily relied on manually annotated alignment data, which includes human preference information, to perform post-training on pre-trained models to achieve alignment~\cite{stiennonLearningSummarizeHuman2020}. &#10;Specifically, there are two primary forms of alignment data: &#10;1) instruction-response pairs, which typically consist of a query and a human-written golden reference. This form of data is often used for supervised fine-tuning of LLMs to inject human preference information into the model~\cite{alpaca,peng2023instruction,ding-etal-2023-enhancing};&#10;2) preference data, which usually includes a query, several potential responses, and human preferences regarding these responses~\cite{cui2024ultrafeedback}. &#10;Preference data can be applied for direct preference optimization via algorithms such as DPO~\cite{rafailovDirectPreferenceOptimization2023a}, IPO~\cite{azar2024general}, and PRO~\cite{songPreferenceRankingOptimization2024}. &#10;Besides, it can also be used to train a reward model, which aligns the target policy LLM to the preference information in the data by providing feedback on the model's responses~\cite{stiennonLearningSummarizeHuman2020, bai2022training, ouyangTrainingLanguageModels2022b}.&#10;However, the construction process for both instruction-response pairs and preference data requires very expensive, meticulous human annotation with high quality standards, making each step of scaling these methods very costly~\cite{ouyangTrainingLanguageModels2022b,touvron2023llama,NEURIPS2023_ac662d74}.&#10;&#10;Even with such high costs, the scalability of these human annotation-dependent alignment methods is still unsustainable. &#10;First, with the rapid development of LLMs, the capabilities of LLMs have gradually approached or even surpassed human in many aspects, making it increasingly challenging for humans to produce alignment data that is meaningful for LLMs~\cite{bowman2022measuring,burns2023weaktostrong}. &#10;In fact, many studies have found that the quality of data generated by LLMs has already exceeded the quality of data annotated by general human annotators in many perspectives~\cite{zheng2024judging,chen2024spiral,wei2024long}. &#10;This phenomenon not only significantly raises the cost of obtaining single meaningful human-annotated data (due to the need for increasingly expensive high-quality annotators), but also substantially reduces the potential benefits of human-annotated data for LLMs.&#10;Second, as the capabilities of LLMs gradually surpass human capability boundaries, it becomes increasingly difficult for humans to effectively judge the quality of the responses generated by LLMs. &#10;This leads to a significant decline in the quality of the preference signals generated by humans, which can no longer accurately reflect human needs, thereby making it challenging to provide effective guidance for LLMs.&#10;Therefore, alignment methods based on human annotation are increasingly unable to cope with the rapid improvement in the capabilities of LLMs, making it difficult to achieve scalable oversight for LLMs.&#10;&#10;To address these challenges, automated alignment has drawn great attention very recently~\cite{yuan2024self,chen2024self}. &#10;Unlike previous methods that relied on human annotation to obtain alignment signals, the goal of automated alignment is constructing scalable and high-quality alignment systems with minimal human intervention. &#10;Therefore, automated alignment has the potential to address the core challenges posed by the rapid development of LLMs, where human annotation is either infeasible or extremely expensive. &#10;For automated alignment, the most crucial part is to find a scalable alignment signal that can replace human manually-created preference signals and remain effective amid the rapid development of LLMs. &#10;&#10;To this end, this survey categorizes the rapidly developing automated alignment methods according to the mechanisms used to construct different alignment signals, summarizes the current developments in each direction, and discusses the developmental trajectory and potential future directions.&#10;Specifically, this survey explores the following representative directions for constructing alignment signals to achieve automated alignment, including:&#10;\begin{itemize}[leftmargin=1em]&#10; \item Aligning through inductive bias~(\S\ref{sec:inductive}), which automatically steers the model towards desired behaviors by introducing suitable assumptions and constraints, without the use of additional training signals beyond the model itself.&#10; \item Aligning through behavior imitation~(\S\ref{sec:imitation}), which achieves automated alignment by mimicking the behavior of another aligned model. For instance, using a well-aligned model to generate instruction-response pairs, and then train target model with imitation learning.&#10; \item Aligning through model feedback~(\S\ref{sec:reward}), which involves guiding the alignment optimization of the target model by obtaining feedback from other models.&#10; \item Aligning through environment feedback~(\S\ref{sec:environment}), which involves automatically obtaining alignment signals or feedback through interaction with environment to achieve automated alignment of the target model.&#10;\end{itemize}&#10;&#10;Furthermore, this survey also explores the underlying mechanisms~(\S\ref{sec:mechanism}) that enable automated alignment and, from the fundamental role of alignment, discuss the essential factors that make automated alignment technologies feasible and effective.&#10;&#10;The rest of this survey is organized as follows: Section~\ref{sec:overview} describes the scope of automated alignment covered in this survey, as well as the our taxonomy.&#10;Section~\ref{sec:inductive}-\ref{sec:environment} provide a detailed introduction to the progress and limitations of the four aforementioned representative directions in automated alignment. Section~\ref{sec:mechanism} explores the underlying mechanisms of automated alignment. And we include a overall conclusion of this survey in Section~\ref{sec:discuss}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Automated Alignment Methods, Computer Science, Linguistics, Scalable AI Development, Artificial Intelligence, Natural Language Processing, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="2308.12014" label="2308.12014">
        <attvalues>
          <attvalue for="0" value="From Instructions to Intrinsic Human Values -- A Survey of Alignment&#10;  Goals for Big Models" />
          <attvalue for="1" value="  Big models, exemplified by Large Language Models (LLMs), are models typically&#10;pre-trained on massive data and comprised of enormous parameters, which not&#10;only obtain significantly improved performance across diverse tasks but also&#10;present emergent capabilities absent in smaller models. However, the growing&#10;intertwining of big models with everyday human lives poses potential risks and&#10;might cause serious social harm. Therefore, many efforts have been made to&#10;align LLMs with humans to make them better follow user instructions and satisfy&#10;human preferences. Nevertheless, `what to align with' has not been fully&#10;discussed, and inappropriate alignment goals might even backfire. In this&#10;paper, we conduct a comprehensive survey of different alignment goals in&#10;existing work and trace their evolution paths to help identify the most&#10;essential goal. Particularly, we investigate related works from two&#10;perspectives: the definition of alignment goals and alignment evaluation. Our&#10;analysis encompasses three distinct levels of alignment goals and reveals a&#10;goal transformation from fundamental abilities to value orientation, indicating&#10;the potential of intrinsic human values as the alignment goal for enhanced&#10;LLMs. Based on such results, we further discuss the challenges of achieving&#10;such intrinsic value alignment and provide a collection of available resources&#10;for future research on the alignment of big models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.15025" label="2309.15025">
        <attvalues>
          <attvalue for="0" value="Large Language Model Alignment: A Survey" />
          <attvalue for="1" value="  Recent years have witnessed remarkable progress made in large language models&#10;(LLMs). Such advancements, while garnering significant attention, have&#10;concurrently elicited various concerns. The potential of these models is&#10;undeniably vast; however, they may yield texts that are imprecise, misleading,&#10;or even detrimental. Consequently, it becomes paramount to employ alignment&#10;techniques to ensure these models to exhibit behaviors consistent with human&#10;values.&#10;  This survey endeavors to furnish an extensive exploration of alignment&#10;methodologies designed for LLMs, in conjunction with the extant capability&#10;research in this domain. Adopting the lens of AI alignment, we categorize the&#10;prevailing methods and emergent proposals for the alignment of LLMs into outer&#10;and inner alignment. We also probe into salient issues including the models'&#10;interpretability, and potential vulnerabilities to adversarial attacks. To&#10;assess LLM alignment, we present a wide variety of benchmarks and evaluation&#10;methodologies. After discussing the state of alignment research for LLMs, we&#10;finally cast a vision toward the future, contemplating the promising avenues of&#10;research that lie ahead.&#10;  Our aspiration for this survey extends beyond merely spurring research&#10;interests in this realm. We also envision bridging the gap between the AI&#10;alignment research community and the researchers engrossed in the capability&#10;exploration of LLMs for both capable and safe LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.03540" label="2211.03540">
        <attvalues>
          <attvalue for="0" value="Measuring Progress on Scalable Oversight for Large Language Models" />
          <attvalue for="1" value="  Developing safe and useful general-purpose AI systems will require us to make&#10;progress on scalable oversight: the problem of supervising systems that&#10;potentially outperform us on most skills relevant to the task at hand.&#10;Empirical work on this problem is not straightforward, since we do not yet have&#10;systems that broadly exceed our abilities. This paper discusses one of the&#10;major ways we think about this problem, with a focus on ways it can be studied&#10;empirically. We first present an experimental design centered on tasks for&#10;which human specialists succeed but unaided humans and current general AI&#10;systems fail. We then present a proof-of-concept experiment meant to&#10;demonstrate a key feature of this experimental design and show its viability&#10;with two question-answering tasks: MMLU and time-limited QuALITY. On these&#10;tasks, we find that human participants who interact with an unreliable&#10;large-language-model dialog assistant through chat -- a trivial baseline&#10;strategy for scalable oversight -- substantially outperform both the model&#10;alone and their own unaided performance. These results are an encouraging sign&#10;that scalable oversight will be tractable to study with present models and&#10;bolster recent findings that large language models can productively assist&#10;humans with difficult tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.09390" label="2312.09390">
        <attvalues>
          <attvalue for="0" value="Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak&#10;  Supervision" />
          <attvalue for="1" value="  Widely used alignment techniques, such as reinforcement learning from human&#10;feedback (RLHF), rely on the ability of humans to supervise model behavior -&#10;for example, to evaluate whether a model faithfully followed instructions or&#10;generated safe outputs. However, future superhuman models will behave in&#10;complex ways too difficult for humans to reliably evaluate; humans will only be&#10;able to weakly supervise superhuman models. We study an analogy to this&#10;problem: can weak model supervision elicit the full capabilities of a much&#10;stronger model? We test this using a range of pretrained language models in the&#10;GPT-4 family on natural language processing (NLP), chess, and reward modeling&#10;tasks. We find that when we naively finetune strong pretrained models on labels&#10;generated by a weak model, they consistently perform better than their weak&#10;supervisors, a phenomenon we call weak-to-strong generalization. However, we&#10;are still far from recovering the full capabilities of strong models with naive&#10;finetuning alone, suggesting that techniques like RLHF may scale poorly to&#10;superhuman models without further work. We find that simple methods can often&#10;significantly improve weak-to-strong generalization: for example, when&#10;finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence&#10;loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our&#10;results suggest that it is feasible to make empirical progress today on a&#10;fundamental challenge of aligning superhuman models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.18802" label="2403.18802">
        <attvalues>
          <attvalue for="0" value="Long-form factuality in large language models" />
          <attvalue for="1" value="  Large language models (LLMs) often generate content that contains factual&#10;errors when responding to fact-seeking prompts on open-ended topics. To&#10;benchmark a model's long-form factuality in open domains, we first use GPT-4 to&#10;generate LongFact, a prompt set comprising thousands of questions spanning 38&#10;topics. We then propose that LLM agents can be used as automated evaluators for&#10;long-form factuality through a method which we call Search-Augmented Factuality&#10;Evaluator (SAFE). SAFE utilizes an LLM to break down a long-form response into&#10;a set of individual facts and to evaluate the accuracy of each fact using a&#10;multi-step reasoning process comprising sending search queries to Google Search&#10;and determining whether a fact is supported by the search results. Furthermore,&#10;we propose extending F1 score as an aggregated metric for long-form factuality.&#10;To do so, we balance the percentage of supported facts in a response&#10;(precision) with the percentage of provided facts relative to a hyperparameter&#10;representing a user's preferred response length (recall).&#10;  Empirically, we demonstrate that LLM agents can outperform crowdsourced human&#10;annotators - on a set of ~16k individual facts, SAFE agrees with crowdsourced&#10;human annotators 72% of the time, and on a random subset of 100 disagreement&#10;cases, SAFE wins 76% of the time. At the same time, SAFE is more than 20 times&#10;cheaper than human annotators. We also benchmark thirteen language models on&#10;LongFact across four model families (Gemini, GPT, Claude, and PaLM-2), finding&#10;that larger language models generally achieve better long-form factuality.&#10;LongFact, SAFE, and all experimental code are available at&#10;https://github.com/google-deepmind/long-form-factuality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.01335" label="2401.01335">
        <attvalues>
          <attvalue for="0" value="Self-Play Fine-Tuning Converts Weak Language Models to Strong Language&#10;  Models" />
          <attvalue for="1" value="  Harnessing the power of human-annotated data through Supervised Fine-Tuning&#10;(SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we&#10;delve into the prospect of growing a strong LLM out of a weak one without the&#10;need for acquiring additional human-annotated data. We propose a new&#10;fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a&#10;supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism,&#10;where the LLM refines its capability by playing against instances of itself.&#10;More specifically, the LLM generates its own training data from its previous&#10;iterations, refining its policy by discerning these self-generated responses&#10;from those obtained from human-annotated data. Our method progressively&#10;elevates the LLM from a nascent model to a formidable one, unlocking the full&#10;potential of human-annotated demonstration data for SFT. Theoretically, we&#10;prove that the global optimum to the training objective function of our method&#10;is achieved only when the LLM policy aligns with the target data distribution.&#10;Empirically, we evaluate our method on several benchmark datasets including the&#10;HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our&#10;results show that SPIN can significantly improve the LLM's performance across a&#10;variety of benchmarks and even outperform models trained through direct&#10;preference optimization (DPO) supplemented with extra GPT-4 preference data.&#10;This sheds light on the promise of self-play, enabling the achievement of&#10;human-level performance in LLMs without the need for expert opponents. Codes&#10;are available at https://github.com/uclaml/SPIN.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03794" label="2004.03794">
        <attvalues>
          <attvalue for="0" value="CALM: Continuous Adaptive Learning for Language Modeling" />
          <attvalue for="1" value="  Training large language representation models has become a standard in the&#10;natural language processing community. This allows for fine tuning on any&#10;number of specific tasks, however, these large high capacity models can&#10;continue to train on domain specific unlabeled data to make initialization even&#10;more robust for supervised tasks. We demonstrate that in practice these&#10;pre-trained models present performance deterioration in the form of&#10;catastrophic forgetting when evaluated on tasks from a general domain such as&#10;GLUE. In this work we propose CALM, Continuous Adaptive Learning for Language&#10;Modeling: techniques to render models which retain knowledge across multiple&#10;domains. With these methods, we are able to reduce the performance gap across&#10;supervised tasks introduced by task specific models which we demonstrate using&#10;a continual learning setting in biomedical and clinical domains.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformer \cite{NIPS2017_7181} based language representation &#10;has replaced many previous pre-training or initialization approaches \cite{devlin2018bert, radford2019language, yang2019xlnet, liu2019roberta}.&#10;Fine tuning using these architectures often yields state-of-the-art results on the order of a few hours.&#10;The caveat to these robust models is that the initial training can be on the scale of several weeks and on many distributed GPUs which is a costly endeavour.&#10;&#10;Pre-trained language models are further adapted to perform strongly in other domains as well.&#10;For example, while the original BERT models \cite{devlin2018bert} were trained on English Wikipedia articles and BooksCorpus \cite{zhu2015aligning}, the same masked language modeling was continued on biomedical data.&#10;BioBERT \cite{lee2019biobert} was trained using Pubmed abstracts and full articles and Clinical BERT \cite{alsentzer2019publicly} parameters were further refined using MIMIC-III clinical notes \cite{johnson2016mimic}.&#10;Evidence suggest that understanding the syntactic structure of scientific literature and clinical data from pre-training boosts performance in their respective downstream tasks \cite{peng2019transfer}.&#10;Training is done with the expectation of building robust, high capacity generalized language models can continue to absorb new domain knowledge.&#10;&#10;Catastrophic forgetting \cite{mccloskey1989catastrophic, ratcliff1990connectionist} is the unfortunate side-effect of incorporating new domain data one after another. &#10;Parameters shift towards capturing the current task and if previous data is no longer available, the model will lose representation of it.&#10;In general perplexity increases for older domains,&#10;and models lose confidence in continual learning settings \cite{yogatama2019learning}.&#10;For many tasks the straightforward solution is to combine datasets during training and approach this as a multi-task learning (MTL) \cite{ruder2017overview} problem.&#10;Mixing data&#10;has the desired effect of constraining parameters to find a space where both tasks reach close to optimal performance.&#10;&#10;We further argue that these expensive pre-trained models are an example where MTL is not feasible in practice for several reasons.&#10;Time and hardware accessibility are the largest constraints for developing these models.&#10;Access to processed training data is generally not possible \cite{radford2019language, devlin2018bert}, and exact training configurations are equally difficult to gather with results being arduous to reproduce.&#10;Resource usage has recently been criticized from another perspective as well.&#10;\cite{strubell2019energy} show that as deep neural architectures in the natural language community grow we increasingly trade results for carbon emissions. &#10;&#10;Current work in catastrophic forgetting mitigation has been limited to a few small scale tested methods.&#10;\cite{howard2018universal} introduced a multi stage training scheme for fine tuning LSTM based universal language models (ULMFiT).&#10;The authors proposed that current methods, rather than data, are ineffective and focused on learning rate control across layers, as well as modifying learning rate scheduling.&#10;A larger category of work deals with constraining model parameters to a latent space where they continue to capture previous tasks.&#10;Initial work focused on model regularization and varying activations \cite{goodfellow2013empirical}. \cite{kirkpatrick2017overcoming} provided a more sophisticated solution constraining weights individually termed elastic weight consolidation (EWC).&#10;We make use of both EWC and ULMFiT and provide further technical detail in this paper.&#10;The final approach is focused on experience replay.&#10;Using small samples of data from previous tasks coupled with local adaptation \cite{d2019episodic} demonstrate improvement in a Lifelong Learning (LL) training scheme.&#10;\cite{chaudhry2019continual} also explore LL by experimenting with updating the memory bank for experience replay.&#10;Our work focuses on both of these techniques with the major difference being problem scale.&#10;Many existing works apply these solutions on small networks whereas we experiment on architectures having several orders of magnitude more parameters.&#10;Our contributions are as follows:&#10;&#10;\begin{itemize}&#10; \item&#10; We motivate the task by providing concrete evidence of catastrophic forgetting for language representation pre-training evaluated on the GLUE benchmark \cite{wang2018glue}.&#10; \item&#10; We provide empirical evidence of catastrophic forgetting mitigation with experience replay, learning rate control, and elastic weight consolidation, applied towards large scale language model pre-training. &#10; \item&#10; We further demonstrate the robustness of elastic weight consolidation when pre-training under two stages of domain shift.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Continual Learning Methods, Computer Science, Linguistics, Cognitive Science, Language Model Training, Domain Adaptation Techniques, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1706.05098" label="1706.05098">
        <attvalues>
          <attvalue for="0" value="An Overview of Multi-Task Learning in Deep Neural Networks" />
          <attvalue for="1" value="  Multi-task learning (MTL) has led to successes in many applications of&#10;machine learning, from natural language processing and speech recognition to&#10;computer vision and drug discovery. This article aims to give a general&#10;overview of MTL, particularly in deep neural networks. It introduces the two&#10;most common methods for MTL in Deep Learning, gives an overview of the&#10;literature, and discusses recent advances. In particular, it seeks to help ML&#10;practitioners apply MTL by shedding light on how MTL works and providing&#10;guidelines for choosing appropriate auxiliary tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01076" label="1906.01076">
        <attvalues>
          <attvalue for="0" value="Episodic Memory in Lifelong Language Learning" />
          <attvalue for="1" value="  We introduce a lifelong language learning setup where a model needs to learn&#10;from a stream of text examples without any dataset identifier. We propose an&#10;episodic memory model that performs sparse experience replay and local&#10;adaptation to mitigate catastrophic forgetting in this setup. Experiments on&#10;text classification and question answering demonstrate the complementary&#10;benefits of sparse experience replay and local adaptation to allow the model to&#10;continuously learn from new datasets. We also show that the space complexity of&#10;the episodic memory module can be reduced significantly (~50-90%) by randomly&#10;choosing which examples to store in memory with a minimal decrease in&#10;performance. We consider an episodic memory component as a crucial building&#10;block of general linguistic intelligence and see our model as a first step in&#10;that direction.&#10;" />
          <attvalue for="2" value="&#10;The ability to continuously learn and accumulate knowledge &#10;throughout a lifetime and reuse it effectively to adapt to a&#10;new problem quickly is a hallmark of general intelligence.&#10;State-of-the-art machine learning models work well&#10;on a single dataset given enough training examples, but&#10;they often fail to isolate and reuse &#10;previously acquired knowledge&#10;when the data distribution shifts (e.g., &#10;when presented with a new dataset)---a &#10;phenomenon known as catastrophic forgetting \cite{mccloskey1989catastrophic,ratcliff1990connectionist}.&#10;&#10;The three main approaches to address catastrophic &#10;forgetting are &#10;based on: (i) augmenting the loss&#10;function that is being minimized during training&#10;with extra terms (e.g., a regularization term, &#10;an optimization constraint)&#10;to prevent model parameters learned on a new dataset &#10;from significantly deviating from parameters&#10;learned on previously seen datasets \cite{ewc, zenke, chaudhry},&#10;(ii) adding extra learning phases such as a knowledge distillation phase, an experience replay \cite{progresscompress, experiencereplay},&#10;and (iii) augmenting the &#10;model with an episodic memory module \cite{mbpa}.&#10;Recent methods have shown that these &#10;approaches can be combined---e.g.,&#10;by defining optimization constraints using samples from the&#10;episodic memory \cite{gem, agem}.&#10;&#10;In language learning, progress in unsupervised pretraining &#10;\cite{elmo,ulmfit,bert} has driven advances in&#10;many language understanding tasks \cite{kitaev,lee}.&#10;However, these models have been shown to require a lot of&#10;in-domain training examples, rapidly overfit to particular datasets,&#10;and are prone to catastrophic forgetting \cite{glipaper},&#10;making them unsuitable as a model of general linguistic intelligence.&#10;&#10;In this paper, we investigate the role of &#10;episodic memory for learning a model of language&#10;in a lifelong setup. &#10;We propose to use such a component&#10;for sparse experience replay and local adaptation&#10;to allow the model to continually learn from examples drawn from &#10;different data distributions.&#10;In experience replay, we randomly select&#10;examples from memory to retrain on.&#10;Our model only performs experience replay&#10;very sparsely to consolidate newly acquired knowledge &#10;with existing knowledge in the memory into the model.&#10;We show that a 1\% experience replay to learning &#10;new examples ratio is sufficient.&#10;Such a process bears some similarity &#10;to memory consolidation in human learning \cite{McGaugh248}.&#10;In local adaptation, we follow&#10;Memory-based Parameter &#10;Adaptation \cite[MbPA;][]{mbpa} and&#10;use examples retrieved from memory to update model&#10;parameters used to make a prediction of a particular&#10;test example.&#10;&#10;Our setup is different &#10;from a typical lifelong learning setup.&#10;We assume that the model &#10;only makes one pass over the training examples, similar to \cite{agem}.&#10;However, we also assume neither our &#10;training nor test examples &#10;have dataset identifying information (e.g., &#10;a dataset identity, a dataset descriptor).&#10;We argue that our lifelong language learning setup---where&#10;a model is presented with &#10;a stream of examples without an explicit &#10;identifier about which dataset (distribution)&#10;the examples come from---is a &#10;realistic setup to learn a general linguistic intelligence model.&#10;Our experiments focus on lifelong language &#10;learning on two tasks---text &#10;classification and question answering.&#10;&#10;Our main contributions in this paper are:&#10;\begin{itemize}&#10;\item We introduce a lifelong language learning setup where the model&#10;needs to learn from a stream of examples from many datasets &#10;(presented sequentially) in one pass, and no dataset boundary&#10;or dataset identity is given to the model.&#10;\item We present an episodic memory model (\S{\ref{sec:model}}) &#10;that augments an encoder-decoder model with a memory module.&#10;Our memory is a key-value memory that stores&#10;previously seen examples for sparse experience replay and local adaptation.&#10;\item We leverage progress in unsupervised pretraining&#10;to obtain good memory key representations and discuss&#10;strategies to manage the space complexity of the memory module.&#10;\item We compare our proposed method to baseline and state-of-the-art continual learning methods and demonstrate its efficacy on text classification and question answering tasks (\S{\ref{sec:experiments}}). &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.00275" label="1910.00275">
        <attvalues>
          <attvalue for="0" value="Bad Form: Comparing Context-Based and Form-Based Few-Shot Learning in&#10;  Distributional Semantic Models" />
          <attvalue for="1" value="  Word embeddings are an essential component in a wide range of natural&#10;language processing applications. However, distributional semantic models are&#10;known to struggle when only a small number of context sentences are available.&#10;Several methods have been proposed to obtain higher-quality vectors for these&#10;words, leveraging both this context information and sometimes the word forms&#10;themselves through a hybrid approach. We show that the current tasks do not&#10;suffice to evaluate models that use word-form information, as such models can&#10;easily leverage word forms in the training data that are related to word forms&#10;in the test data. We introduce 3 new tasks, allowing for a more balanced&#10;comparison between models. Furthermore, we show that hyperparameters that have&#10;largely been ignored in previous work can consistently improve the performance&#10;of both baseline and advanced models, achieving a new state of the art on 4 out&#10;of 6 tasks.&#10;" />
          <attvalue for="2" value="&#10;Word embeddings have impacted almost every aspect of NLP, proving effective in a wide range of use cases. Often used in the form of a pre-trained model, these vectors provide easy to use representations of semantic meaning. However, distributional models are known to struggle with words for which training data is sparse, often resulting in low-quality vector representations \cite{huang2012improving, adams2017cross}. The default approach in this case has historically been to ignore these rare words, preferring an incomplete view over an incorrect one \cite{Mikolov:20130be}. Another option is to use the surface form of a word to obtain a vector, leveraging morphological characteristics \cite{luong2013better} or subword embeddings \cite{Bojanowski:20160be}. As neither of these approaches fully resolves the problem, more techniques have been proposed for few-shot learning in distributional models. Each of these aims to correctly position a new word vector inside an existing semantic space. The challenge for few-shot learning is to find a position that accurately reflects the meaning of the word, even if only a small number of usage examples is available.&#10;&#10;Making systems better at handling rare words is an obvious practical goal of few-shot learning, as it could substantially improve systems working with technical language or dialects. However, few-shot learning is also interesting from a human language learning perspective: unlike current-day distributional models, humans excel at learning meaning from sparse data through a process called `fast mapping' \cite{trueswell2013propose, lake2017building}. Lessons learned from psychology might prove effective in machines, and novel few-shot learning techniques might provide insight into fast mapping in humans.&#10;&#10;Three evaluation tasks have been proposed to evaluate few-shot learning methods: Definitional Nonce \cite{herbelot2017}, Chimera \cite{Lazaridou:20170be}, and Contextual Rare Words \cite{Khodak:20180be}, which we describe in Section~\ref{sec:back}. However, each of these tasks was designed for context-based few-shot learning, without considering hybrid methods, which also have access to word-form information. We show that the existing tasks do not suffice to fully assess the performance of hybrid models, with relatively simple, purely form-based methods dominating two out of three tasks. To provide a better overview and performance comparison, in Section~\ref{sec:eval} we introduce three new tasks based on these three datasets. In Sections~\ref{sec:novel}--\ref{sec:res}, we show that, just as hyperparameters are essential to good performance with standard distributional models \cite{levy2015improving}, the same is true for few-shot distributional models. With three straightforward modifications, we substantially improve the baseline scores, outperforming several advanced methods from previous work, as well as achieving a new state of the art on 4 out of 6 evaluation tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Word Embeddings Evaluation, Model Performance Optimization, Computational Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2210.11777" label="2210.11777">
        <attvalues>
          <attvalue for="0" value="Analyzing and Evaluating Faithfulness in Dialogue Summarization" />
          <attvalue for="1" value="  Dialogue summarization is abstractive in nature, making it suffer from&#10;factual errors. The factual correctness of summaries has the highest priority&#10;before practical applications. Many efforts have been made to improve&#10;faithfulness in text summarization. However, there is a lack of systematic&#10;study on dialogue summarization systems. In this work, we first perform the&#10;fine-grained human analysis on the faithfulness of dialogue summaries and&#10;observe that over 35% of generated summaries are faithfully inconsistent&#10;respective the source dialogues. Furthermore, we present a new model-level&#10;faithfulness evaluation method. It examines generation models with multi-choice&#10;questions created by rule-based transformations. Experimental results show that&#10;our evaluation schema is a strong proxy for the factual correctness of&#10;summarization models. The human-annotated faithfulness samples and the&#10;evaluation toolkit are released to facilitate future research toward faithful&#10;dialogue summarization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.05641" label="2105.05641">
        <attvalues>
          <attvalue for="0" value="How Reliable are Model Diagnostics?" />
          <attvalue for="1" value="  In the pursuit of a deeper understanding of a model's behaviour, there is&#10;recent impetus for developing suites of probes aimed at diagnosing models&#10;beyond simple metrics like accuracy or BLEU. This paper takes a step back and&#10;asks an important and timely question: how reliable are these diagnostics in&#10;providing insight into models and training setups? We critically examine three&#10;recent diagnostic tests for pre-trained language models, and find that&#10;likelihood-based and representation-based model diagnostics are not yet as&#10;reliable as previously assumed. Based on our empirical findings, we also&#10;formulate recommendations for practitioners and researchers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Language Models, Model Diagnostics, Reliability Analysis, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2102.08424" label="2102.08424">
        <attvalues>
          <attvalue for="0" value="Searching for Search Errors in Neural Morphological Inflection" />
          <attvalue for="1" value="  Neural sequence-to-sequence models are currently the predominant choice for&#10;language generation tasks. Yet, on word-level tasks, exact inference of these&#10;models reveals the empty string is often the global optimum. Prior works have&#10;speculated this phenomenon is a result of the inadequacy of neural models for&#10;language generation. However, in the case of morphological inflection, we find&#10;that the empty string is almost never the most probable solution under the&#10;model. Further, greedy search often finds the global optimum. These&#10;observations suggest that the poor calibration of many neural models may stem&#10;from characteristics of a specific subset of tasks rather than general&#10;ill-suitedness of such models for language generation.&#10;" />
          <attvalue for="2" value="&#10;Neural sequence-to-sequence models are omnipresent in the field of natural language processing due to their impressive performance. &#10;They hold state of the art on a myriad of tasks, e.g., neural machine translation \cite[NMT;][]{ott-etal-2018-scaling} and abstractive summarization \cite[AS;][]{lewis2019bart}. Yet, an undesirable property of these models has been repeatedly observed in word-level tasks: When using beam search as the decoding strategy, increasing the beam width beyond a size of $k=5$ often leads to a drop in the quality of solutions \cite{murray-chiang-2018-correcting, yang-etal-2018-breaking,pmlr-v97-cohen19a}. &#10;Further, in the context of NMT, it has been shown that the empty string is frequently the most-probable solution under the model \cite{stahlberg-byrne-2019-nmt}. Some suggest this is a manifestation of the general inadequacy of neural models for language generation tasks \cite{koehn-knowles-2017-six, kumar2019calibration,holtzman2019curious,stahlberg_phd}; in this work, we find evidence demonstrating otherwise.\looseness=-1 &#10;&#10;Sequence-to-sequence transducers for character-level tasks often follow the architectures of their word-level counterparts \cite{faruqui-etal-2016-morphological, lee-etal-2017-fully}, and have likewise achieved state-of-the-art performance on e.g., morphological inflection generation \cite{wu2020applying} and grapheme-to-phoneme conversion \cite{G2P}. &#10;Given prior findings, we might expect to see the same degenerate behavior in these models---however, we do not. &#10;We run a series of experiments on morphological inflection (MI) generators to explore whether neural transducers for this task are similarly poorly calibrated, i.e. are far from the true distribution $p(\yy\mid\xx)$. &#10;We evaluate the performance of two character-level sequence-to-sequence transducers using different decoding strategies; our results, previewed in \ref{tab:search_errors_nmt}, show that evaluation metrics do not degrade with larger beam sizes as in NMT or AS. &#10;Additionally, only in extreme circumstances, e.g., low-resource settings with less than 100 training samples, is the empty string ever the global optimum under the model.\looseness=-1&#10;&#10;Our findings directly refute the claim that neural architectures are inherently inadequate for modeling language generation tasks. Instead, our results admit two potential causes of the degenerate behavior observed in tasks such as NMT and AS: (1) lack of a deterministic mapping between input and output and (2) a (perhaps irreparable) discrepancy between sample complexity and training resources. Our results alone are not sufficient to accept or reject either hypothesis, and thus we leave these as future research directions.\looseness=-1 &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Generation, Neural Models, Mathematics, Model Calibration" />
        </attvalues>
      </node>
      <node id="1806.00187" label="1806.00187">
        <attvalues>
          <attvalue for="0" value="Scaling Neural Machine Translation" />
          <attvalue for="1" value="  Sequence to sequence learning models still require several days to reach&#10;state of the art performance on large benchmark datasets using a single&#10;machine. This paper shows that reduced precision and large batch training can&#10;speedup training by nearly 5x on a single 8-GPU machine with careful tuning and&#10;implementation. On WMT'14 English-German translation, we match the accuracy of&#10;Vaswani et al. (2017) in under 5 hours when training on 8 GPUs and we obtain a&#10;new state of the art of 29.3 BLEU after training for 85 minutes on 128 GPUs. We&#10;further improve these results to 29.8 BLEU by training on the much larger&#10;Paracrawl dataset. On the WMT'14 English-French task, we obtain a&#10;state-of-the-art BLEU of 43.2 in 8.5 hours on 128 GPUs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.10006" label="1808.10006">
        <attvalues>
          <attvalue for="0" value="Correcting Length Bias in Neural Machine Translation" />
          <attvalue for="1" value="  We study two problems in neural machine translation (NMT). First, in beam&#10;search, whereas a wider beam should in principle help translation, it often&#10;hurts NMT. Second, NMT has a tendency to produce translations that are too&#10;short. Here, we argue that these problems are closely related and both rooted&#10;in label bias. We show that correcting the brevity problem almost eliminates&#10;the beam problem; we compare some commonly-used methods for doing this, finding&#10;that a simple per-word reward works well; and we introduce a simple and quick&#10;way to tune this reward using the perceptron algorithm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.10090" label="1908.10090">
        <attvalues>
          <attvalue for="0" value="On NMT Search Errors and Model Errors: Cat Got Your Tongue?" />
          <attvalue for="1" value="  We report on search errors and model errors in neural machine translation&#10;(NMT). We present an exact inference procedure for neural sequence models based&#10;on a combination of beam search and depth-first search. We use our exact search&#10;to find the global best model scores under a Transformer base model for the&#10;entire WMT15 English-German test set. Surprisingly, beam search fails to find&#10;these global best model scores in most cases, even with a very large beam size&#10;of 100. For more than 50% of the sentences, the model in fact assigns its&#10;global best score to the empty translation, revealing a massive failure of&#10;neural models in properly accounting for adequacy. We show by constraining&#10;search with a minimum translation length that at the root of the problem of&#10;empty translations lies an inherent bias towards shorter translations. We&#10;conclude that vanilla NMT in its current form requires just the right amount of&#10;beam search errors, which, from a modelling perspective, is a highly&#10;unsatisfactory conclusion indeed, as the model often prefers an empty&#10;translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.10213" label="2005.10213">
        <attvalues>
          <attvalue for="0" value="Applying the Transformer to Character-level Transduction" />
          <attvalue for="1" value="  The transformer has been shown to outperform recurrent neural network-based&#10;sequence-to-sequence models in various word-level NLP tasks. Yet for&#10;character-level transduction tasks, e.g. morphological inflection generation&#10;and historical text normalization, there are few works that outperform&#10;recurrent models using the transformer. In an empirical study, we uncover that,&#10;in contrast to recurrent sequence-to-sequence models, the batch size plays a&#10;crucial role in the performance of the transformer on character-level tasks,&#10;and we show that with a large enough batch size, the transformer does indeed&#10;outperform recurrent models. We also introduce a simple technique to handle&#10;feature-guided character-level transduction that further improves performance.&#10;With these insights, we achieve state-of-the-art performance on morphological&#10;inflection and historical text normalization. We also show that the transformer&#10;outperforms a strong baseline on two other character-level transduction tasks:&#10;grapheme-to-phoneme conversion and transliteration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.06338" label="2004.06338">
        <attvalues>
          <attvalue for="0" value="Transformer based Grapheme-to-Phoneme Conversion" />
          <attvalue for="1" value="  Attention mechanism is one of the most successful techniques in deep learning&#10;based Natural Language Processing (NLP). The transformer network architecture&#10;is completely based on attention mechanisms, and it outperforms&#10;sequence-to-sequence models in neural machine translation without recurrent and&#10;convolutional layers. Grapheme-to-phoneme (G2P) conversion is a task of&#10;converting letters (grapheme sequence) to their pronunciations (phoneme&#10;sequence). It plays a significant role in text-to-speech (TTS) and automatic&#10;speech recognition (ASR) systems. In this paper, we investigate the application&#10;of transformer architecture to G2P conversion and compare its performance with&#10;recurrent and convolutional neural network based approaches. Phoneme and word&#10;error rates are evaluated on the CMUDict dataset for US English and the NetTalk&#10;dataset. The results show that transformer based G2P outperforms the&#10;convolutional-based approach in terms of word error rate and our results&#10;significantly exceeded previous recurrent approaches (without attention)&#10;regarding word and phoneme error rates on both datasets. Furthermore, the size&#10;of the proposed model is much smaller than the size of the previous approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.17304" label="2305.17304">
        <attvalues>
          <attvalue for="0" value="External Language Model Integration for Factorized Neural Transducers" />
          <attvalue for="1" value="  We propose an adaptation method for factorized neural transducers (FNT) with&#10;external language models. We demonstrate that both neural and n-gram external&#10;LMs add significantly more value when linearly interpolated with predictor&#10;output compared to shallow fusion, thus confirming that FNT forces the&#10;predictor to act like regular language models. Further, we propose a method to&#10;integrate class-based n-gram language models into FNT framework resulting in&#10;accuracy gains similar to a hybrid setup. We show average gains of 18% WERR&#10;with lexical adaptation across various scenarios and additive gains of up to&#10;60% WERR in one entity-rich scenario through a combination of class-based&#10;n-gram and neural LMs.&#10;" />
          <attvalue for="2" value="&#10;With large amounts of paired speech and text data available for ASR training, end-to-end (E2E) automatic speech recognition (ASR) systems have become competitive with hybrid ASR systems~\cite{sainath2020streaming,li2020developing,radford2022robust}. E2E alternatives have the advantage of architectural simplicity compared to hybrid ASR systems in which acoustic models, language models, and pronunciation models are independently trained. However, a consequence of joint modeling is that effective adaptation to target scenarios using text-only tuning data becomes challenging. There are generally four ways to perform text-only adaptation:&#10;\begin{itemize}&#10;\item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr};&#10;\item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ};&#10;\item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density};&#10;\item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}.&#10;\end{itemize}&#10;In this paper, we focus on the last approach using factorized neural transducers (FNT)~\cite{chen2022factorized} which is an extension of RNN-T~\cite{graves2012sequence} that makes the predictor network act like a proper language model. We investigate linear/log-linear interpolation approaches for augmenting FNT predictor activations with probabilities from external in-domain n-gram and neural models, and demonstrate that these methods are superior to the shallow fusion baseline. We also show how the n-gram approach can be extended to handle class-based n-gram LMs via linear interpolation, resulting in adaptation improvements comparable to those achievable with a hybrid recognizer. A number of important adjustments necessary for an effective application of the interpolated class-based setup is also discussed. &#10;&#10;The remainder of this paper is organized as follows. In \S\ref{sec:adapt} we explain the basic principles of FNTs and how various types of external LMs can be integrated with them. An experimental sectionfollows, and the paper is concluded with a summary of presented results.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation Adaptation, Linguistics, External Knowledge Integration, Artificial Intelligence, Neural Language Models, Mathematics" />
        </attvalues>
      </node>
      <node id="2003.12710" label="2003.12710">
        <attvalues>
          <attvalue for="0" value="A Streaming On-Device End-to-End Model Surpassing Server-Side&#10;  Conventional Model Quality and Latency" />
          <attvalue for="1" value="  Thus far, end-to-end (E2E) models have not been shown to outperform&#10;state-of-the-art conventional models with respect to both quality, i.e., word&#10;error rate (WER), and latency, i.e., the time the hypothesis is finalized after&#10;the user stops speaking. In this paper, we develop a first-pass Recurrent&#10;Neural Network Transducer (RNN-T) model and a second-pass Listen, Attend, Spell&#10;(LAS) rescorer that surpasses a conventional model in both quality and latency.&#10;On the quality side, we incorporate a large number of utterances across varied&#10;domains to increase acoustic diversity and the vocabulary seen by the model. We&#10;also train with accented English speech to make the model more robust to&#10;different pronunciations. In addition, given the increased amount of training&#10;data, we explore a varied learning rate schedule. On the latency front, we&#10;explore using the end-of-sentence decision emitted by the RNN-T model to close&#10;the microphone, and also introduce various optimizations to improve the speed&#10;of LAS rescoring. Overall, we find that RNN-T+LAS offers a better WER and&#10;latency tradeoff compared to a conventional model. For example, for the same&#10;latency, RNN-T+LAS obtains a 8% relative improvement in WER, while being more&#10;than 400-times smaller in model size.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.02695" label="1612.02695">
        <attvalues>
          <attvalue for="0" value="Towards better decoding and language model integration in sequence to&#10;  sequence models" />
          <attvalue for="1" value="  The recently proposed Sequence-to-Sequence (seq2seq) framework advocates&#10;replacing complex data processing pipelines, such as an entire automatic speech&#10;recognition system, with a single neural network trained in an end-to-end&#10;fashion. In this contribution, we analyse an attention-based seq2seq speech&#10;recognition system that directly transcribes recordings into characters. We&#10;observe two shortcomings: overconfidence in its predictions and a tendency to&#10;produce incomplete transcriptions when language models are used. We propose&#10;practical solutions to both problems achieving competitive speaker independent&#10;word error rates on the Wall Street Journal dataset: without separate language&#10;models we reach 10.6% WER, while together with a trigram language model, we&#10;reach 6.7% WER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.17049" label="2210.17049">
        <attvalues>
          <attvalue for="0" value="Modular Hybrid Autoregressive Transducer" />
          <attvalue for="1" value="  Text-only adaptation of a transducer model remains challenging for end-to-end&#10;speech recognition since the transducer has no clearly separated acoustic model&#10;(AM), language model (LM) or blank model. In this work, we propose a modular&#10;hybrid autoregressive transducer (MHAT) that has structurally separated label&#10;and blank decoders to predict label and blank distributions, respectively,&#10;along with a shared acoustic encoder. The encoder and label decoder outputs are&#10;directly projected to AM and internal LM scores and then added to compute label&#10;posteriors. We train MHAT with an internal LM loss and a HAT loss to ensure&#10;that its internal LM becomes a standalone neural LM that can be effectively&#10;adapted to text. Moreover, text adaptation of MHAT fosters a much better LM&#10;fusion than internal LM subtraction-based methods. On Google's large-scale&#10;production data, a multi-domain MHAT adapted with 100B sentences achieves&#10;relative WER reductions of up to 12.4% without LM fusion and 21.5% with LM&#10;fusion from 400K-hour trained HAT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11481" label="2010.11481">
        <attvalues>
          <attvalue for="0" value="Similarity Analysis of Self-Supervised Speech Representations" />
          <attvalue for="1" value="  Self-supervised speech representation learning has recently been a prosperous&#10;research topic. Many algorithms have been proposed for learning useful&#10;representations from large-scale unlabeled data, and their applications to a&#10;wide range of speech tasks have also been investigated. However, there has been&#10;little research focusing on understanding the properties of existing&#10;approaches. In this work, we aim to provide a comparative study of some of the&#10;most representative self-supervised algorithms. Specifically, we quantify the&#10;similarities between different self-supervised representations using existing&#10;similarity measures. We also design probing tasks to study the correlation&#10;between the models' pre-training loss and the amount of specific speech&#10;information contained in their learned representations. In addition to showing&#10;how various self-supervised models behave differently given the same input, our&#10;study also finds that the training objective has a higher impact on&#10;representation similarity than architectural choices such as building blocks&#10;(RNN/Transformer/CNN) and directionality (uni/bidirectional). Our results also&#10;suggest that there exists a strong correlation between pre-training loss and&#10;downstream performance for some self-supervised algorithms.&#10;" />
          <attvalue for="2" value="&#10;Self-supervised learning is a form of unsupervised learning that treats the input or modifications of the input as learning targets.&#10;Thanks to this property, self-supervised learning can leverage large-scale unlabeled data for training, and has enjoyed success in learning high-level representations of data from different modalities~\cite{chen2020simple,devlin2019bert,baevski2020wav2vec}.&#10;&#10;Recently, self-supervised approaches for learning speech representations have received great research attention.&#10;Methods like contrastive predictive coding~\cite{oord2018representation}, autoregressive predictive coding~\cite{chung2019unsupervised}, masked predictive coding~\cite{liu2020mockingjay,wang2020unsupervised,jiang2019improving}, and problem-agnostic speech encoder~\cite{pascual2019learning} have been shown to be capable of learning representations that capture high-level properties of speech that are not easily accessible from surface features such as audio waveforms and spectrograms.&#10;These methods have been further extended or improved for tackling a wide range of speech applications, including speech recognition~\cite{baevski2020effectiveness,chung2020generative,ling2020deep,jiang2020further,song2020speech}, speech translation~\cite{nguyen2020investigating,wu2020self}, speaker verification~\cite{ravi2020exploring}, unsupervised unit discovery~\cite{feng2020unsupervised}, and unsupervised phoneme segmentation~\cite{kreuk2020self}, to name a few.&#10;&#10;Despite the recent progress in self-supervised speech representation learning, most of the effort is made to develop new algorithms or adapt existing methods to particular tasks, and only a few studies focus on reviewing existing approaches.&#10;In this work, we aim to provide a comparative study on some of the most representative self-supervised algorithms: contrastive predictive coding~(CPC), autoregressive predictive coding~(APC), and masked predictive coding~(MPC).&#10;Our analysis focuses on the following two aspects.&#10;First, we hope to understand the similarity of different self-supervised representations.&#10;To carry out this study, we adopt two similarity measures for quantifying the similarity of two given representations.&#10;Although such a similarity analysis approach cannot discern absolute facts about the representations, it allows us to compare representations without subscribing to any specific type of information, and helps us answer questions like: Given the same input, how similar are different self-supervised representations? Which modeling choices, e.g., building blocks~(RNN/Transformer/CNN) and directionality~(uni/bidirectional), have a higher impact on similarity? How much does a model change when it is trained on more data?&#10;&#10;Our second area of investigation examines, for each self-supervised algorithm, how well its pre-training loss correlates with downstream performance.&#10;We use phonetic and speaker classification as probing tasks to measure the amount of phonetic and speaker information contained in the representations as a function of pre-training loss.&#10;This study could be useful for model selection if there exists a strong correlation between them.&#10;&#10;Only a few studies have focused on analyzing self-supervised models.&#10;Chung et al.~(2020) proposed to incorporate vector quantization layers to restrict model capacity during pre-training so as to uncover a model's preference in preserving speech information for achieving a maximal self-supervised objective~\cite{chung2020vector}.&#10;Bland{\'o}n and R{\&quot;a}s{\&quot;a}nen~(2020) studied the correlation between the self-supervised loss of APC and CPC and their performance on a phoneme discrimination task~\cite{blandon2020analysis}, which has the same goal as our second study.&#10;However, neither of these two works investigated the similarity between different self-supervised representations.&#10;For the correlation study, we also consider more self-supervised models with diverse modeling choices as compared to previous work~\cite{blandon2020analysis}.&#10;&#10;Our analysis yields the following insights:&#10;\begin{itemize}&#10; \item The objective has a higher impact on representation similarity than model architecture.&#10; \item Under the same objective, a model's directionality\\(uni/bidirectional) affects representation similarity more than its building blocks~(RNN/Transformer/CNN).&#10; \item Both APC and MPC both have a stronger correlations between pre-training loss and phonetic and speaker classification performance than does CPC.&#10; \item While all models benefit from increasing the size of unlabeled training data, CPC is found to make use of these additional data more efficiently than APC and MPC.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Comparative Analysis, Linguistics, Speech Representation Learning, Artificial Intelligence, Mathematics, Self-Supervised Algorithms" />
        </attvalues>
      </node>
      <node id="2002.05709" label="2002.05709">
        <attvalues>
          <attvalue for="0" value="A Simple Framework for Contrastive Learning of Visual Representations" />
          <attvalue for="1" value="  This paper presents SimCLR: a simple framework for contrastive learning of&#10;visual representations. We simplify recently proposed contrastive&#10;self-supervised learning algorithms without requiring specialized architectures&#10;or a memory bank. In order to understand what enables the contrastive&#10;prediction tasks to learn useful representations, we systematically study the&#10;major components of our framework. We show that (1) composition of data&#10;augmentations plays a critical role in defining effective predictive tasks, (2)&#10;introducing a learnable nonlinear transformation between the representation and&#10;the contrastive loss substantially improves the quality of the learned&#10;representations, and (3) contrastive learning benefits from larger batch sizes&#10;and more training steps compared to supervised learning. By combining these&#10;findings, we are able to considerably outperform previous methods for&#10;self-supervised and semi-supervised learning on ImageNet. A linear classifier&#10;trained on self-supervised representations learned by SimCLR achieves 76.5%&#10;top-1 accuracy, which is a 7% relative improvement over previous&#10;state-of-the-art, matching the performance of a supervised ResNet-50. When&#10;fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy,&#10;outperforming AlexNet with 100X fewer labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.10603" label="2001.10603">
        <attvalues>
          <attvalue for="0" value="Unsupervised Pre-training of Bidirectional Speech Encoders via Masked&#10;  Reconstruction" />
          <attvalue for="1" value="  We propose an approach for pre-training speech representations via a masked&#10;reconstruction loss. Our pre-trained encoder networks are bidirectional and can&#10;therefore be used directly in typical bidirectional speech recognition models.&#10;The pre-trained networks can then be fine-tuned on a smaller amount of&#10;supervised data for speech recognition. Experiments with this approach on the&#10;LibriSpeech and Wall Street Journal corpora show promising results. We find&#10;that the main factors that lead to speech recognition improvements are: masking&#10;segments of sufficient width in both time and frequency, pre-training on a much&#10;larger amount of unlabeled data than the labeled data, and domain adaptation&#10;when the unlabeled and labeled data come from different domains. The gain from&#10;pre-training is additive to that of supervised data augmentation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.09932" label="1910.09932">
        <attvalues>
          <attvalue for="0" value="Improving Transformer-based Speech Recognition Using Unsupervised&#10;  Pre-training" />
          <attvalue for="1" value="  Speech recognition technologies are gaining enormous popularity in various&#10;industrial applications. However, building a good speech recognition system&#10;usually requires large amounts of transcribed data, which is expensive to&#10;collect. To tackle this problem, an unsupervised pre-training method called&#10;Masked Predictive Coding is proposed, which can be applied for unsupervised&#10;pre-training with Transformer based model. Experiments on HKUST show that using&#10;the same training data, we can achieve CER 23.3%, exceeding the best end-to-end&#10;model by over 0.2% absolute CER. With more pre-training data, we can further&#10;reduce the CER to 21.0%, or a 11.8% relative CER reduction over baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09862" label="2005.09862">
        <attvalues>
          <attvalue for="0" value="A Further Study of Unsupervised Pre-training for Transformer Based&#10;  Speech Recognition" />
          <attvalue for="1" value="  Building a good speech recognition system usually requires large amounts of&#10;transcribed data, which is expensive to collect. To tackle this problem, many&#10;unsupervised pre-training methods have been proposed. Among these methods,&#10;Masked Predictive Coding achieved significant improvements on various speech&#10;recognition datasets with BERT-like Masked Reconstruction loss and Transformer&#10;backbone. However, many aspects of MPC have not been fully investigated. In&#10;this paper, we conduct a further study on MPC and focus on three important&#10;aspects: the effect of pre-training data speaking style, its extension on&#10;streaming model, and how to better transfer learned knowledge from pre-training&#10;stage to downstream tasks. Experiments reveled that pre-training data with&#10;matching speaking style is more useful on downstream recognition tasks. A&#10;unified training objective with APC and MPC provided 8.46% relative error&#10;reduction on streaming model trained on HKUST. Also, the combination of target&#10;data adaption and layer-wise discriminative training helped the knowledge&#10;transfer of MPC, which achieved 3.99% relative error reduction on AISHELL over&#10;a strong baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.10387" label="1910.10387">
        <attvalues>
          <attvalue for="0" value="Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention&#10;  Networks" />
          <attvalue for="1" value="  Self-attention network (SAN) can benefit significantly from the&#10;bi-directional representation learning through unsupervised pretraining&#10;paradigms such as BERT and XLNet. In this paper, we present an XLNet-like&#10;pretraining scheme &quot;Speech-XLNet&quot; for unsupervised acoustic model pretraining&#10;to learn speech representations with SAN. The pretrained SAN is finetuned under&#10;the hybrid SAN/HMM framework. We conjecture that by shuffling the speech frame&#10;orders, the permutation in Speech-XLNet serves as a strong regularizer to&#10;encourage the SAN to make inferences by focusing on global structures through&#10;its attention weights. In addition, Speech-XLNet also allows the model to&#10;explore the bi-directional contexts for effective speech representation&#10;learning. Experiments on TIMIT and WSJ demonstrate that Speech-XLNet greatly&#10;improves the SAN/HMM performance in terms of both convergence speed and&#10;recognition accuracy compared to the one trained from randomly initialized&#10;weights. Our best systems achieve a relative improvement of 11.9% and 8.3% on&#10;the TIMIT and WSJ tasks respectively. In particular, the best system achieves a&#10;phone error rate (PER) of 13.3% on the TIMIT test set, which to our best&#10;knowledge, is the lowest PER obtained from a single system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.03615" label="2008.03615">
        <attvalues>
          <attvalue for="0" value="Exploring the Use of an Unsupervised Autoregressive Model as a Shared&#10;  Encoder for Text-Dependent Speaker Verification" />
          <attvalue for="1" value="  In this paper, we propose a novel way of addressing text-dependent automatic&#10;speaker verification (TD-ASV) by using a shared-encoder with task-specific&#10;decoders. An autoregressive predictive coding (APC) encoder is pre-trained in&#10;an unsupervised manner using both out-of-domain (LibriSpeech, VoxCeleb) and&#10;in-domain (DeepMine) unlabeled datasets to learn generic, high-level feature&#10;representation that encapsulates speaker and phonetic content. Two&#10;task-specific decoders were trained using labeled datasets to classify speakers&#10;(SID) and phrases (PID). Speaker embeddings extracted from the SID decoder were&#10;scored using a PLDA. SID and PID systems were fused at the score level. There&#10;is a 51.9% relative improvement in minDCF for our system compared to the fully&#10;supervised x-vector baseline on the cross-lingual DeepMine dataset. However,&#10;the i-vector/HMM method outperformed the proposed APC encoder-decoder system. A&#10;fusion of the x-vector/PLDA baseline and the SID/PLDA scores prior to PID&#10;fusion further improved performance by 15% indicating complementarity of the&#10;proposed approach to the x-vector system. We show that the proposed approach&#10;can leverage from large, unlabeled, data-rich domains, and learn speech&#10;patterns independent of downstream tasks. Such a system can provide competitive&#10;performance in domain-mismatched scenarios where test data is from data-scarce&#10;domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.13465" label="2007.13465">
        <attvalues>
          <attvalue for="0" value="Self-Supervised Contrastive Learning for Unsupervised Phoneme&#10;  Segmentation" />
          <attvalue for="1" value="  We propose a self-supervised representation learning model for the task of&#10;unsupervised phoneme boundary detection. The model is a convolutional neural&#10;network that operates directly on the raw waveform. It is optimized to identify&#10;spectral changes in the signal using the Noise-Contrastive Estimation&#10;principle. At test time, a peak detection algorithm is applied over the model&#10;outputs to produce the final boundaries. As such, the proposed model is trained&#10;in a fully unsupervised manner with no manual annotations in the form of target&#10;boundaries nor phonetic transcriptions. We compare the proposed approach to&#10;several unsupervised baselines using both TIMIT and Buckeye corpora. Results&#10;suggest that our approach surpasses the baseline models and reaches&#10;state-of-the-art performance on both data sets. Furthermore, we experimented&#10;with expanding the training set with additional examples from the Librispeech&#10;corpus. We evaluated the resulting model on distributions and languages that&#10;were not seen during the training phase (English, Hebrew and German) and showed&#10;that utilizing additional untranscribed data is beneficial for model&#10;performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.04205" label="2007.04205">
        <attvalues>
          <attvalue for="0" value="Analysis of Predictive Coding Models for Phonemic Representation&#10;  Learning in Small Datasets" />
          <attvalue for="1" value="  Neural network models using predictive coding are interesting from the&#10;viewpoint of computational modelling of human language acquisition, where the&#10;objective is to understand how linguistic units could be learned from speech&#10;without any labels. Even though several promising predictive coding -based&#10;learning algorithms have been proposed in the literature, it is currently&#10;unclear how well they generalise to different languages and training dataset&#10;sizes. In addition, despite that such models have shown to be effective&#10;phonemic feature learners, it is unclear whether minimisation of the predictive&#10;loss functions of these models also leads to optimal phoneme-like&#10;representations. The present study investigates the behaviour of two predictive&#10;coding models, Autoregressive Predictive Coding and Contrastive Predictive&#10;Coding, in a phoneme discrimination task (ABX task) for two languages with&#10;different dataset sizes. Our experiments show a strong correlation between the&#10;autoregressive loss and the phoneme discrimination scores with the two&#10;datasets. However, to our surprise, the CPC model shows rapid convergence&#10;already after one pass over the training data, and, on average, its&#10;representations outperform those of APC on both languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.12471" label="2406.12471">
        <attvalues>
          <attvalue for="0" value="Fighting Randomness with Randomness: Mitigating Optimisation Instability&#10;  of Fine-Tuning using Delayed Ensemble and Noisy Interpolation" />
          <attvalue for="1" value="  While fine-tuning of pre-trained language models generally helps to overcome&#10;the lack of labelled training samples, it also displays model performance&#10;instability. This instability mainly originates from randomness in&#10;initialisation or data shuffling. To address this, researchers either modify&#10;the training process or augment the available samples, which typically results&#10;in increased computational costs. We propose a new mitigation strategy, called&#10;Delayed Ensemble with Noisy Interpolation (DENI), that leverages the strengths&#10;of ensembling, noise regularisation and model interpolation, while retaining&#10;computational efficiency. We compare DENI with 9 representative mitigation&#10;strategies across 3 models, 4 tuning strategies and 7 text classification&#10;datasets. We show that: 1) DENI outperforms the best performing mitigation&#10;strategy (Ensemble), while using only a fraction of its cost; 2) the mitigation&#10;strategies are beneficial for parameter-efficient fine-tuning (PEFT) methods,&#10;outperforming full fine-tuning in specific cases; and 3) combining DENI with&#10;data augmentation often leads to even more effective instability mitigation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Tuning of pre-trained language models such as BERT or RoBERTa using either full fine-tuning or parameter efficient fine-tuning (PEFT) has achieved significant success across a wide range of natural language processing tasks. They are especially useful when faced with limited labelled data for quickly adapting to the specific task. Despite the success, previous works observed that fine-tuning still remains unstable~\cite{dodge2020fine, mosbach2021on, chen-etal-2022-revisiting}, especially with limited data. Fine-tuning is sensitive to the effects of randomness originating from random initialisation, data shuffling or model randomness (e.g., use of non-deterministic layers such as dropout in the model). As illustrated in Figure~\ref{fig:variance_in_results}, repeating the fine-tuning process multiple times, without mitigating the randomness, leads to large performance variance in the results, both for full fine-tuning and for PEFT methods.&#10;&#10;To deal with the fine-tuning instability, researchers propose various strategies to mitigate the effects of randomness~\cite{pecher2024survey}. In most cases, the mitigation strategies focus on modifying the training process~\cite{Lee2020mixout, dodge2020fine}, such as adding noise~\cite{hua-etal-2021-noise, wu-etal-2022-noisytune, hua2023improving}, ensembling multiple models~\cite{hidey-etal-2022-reducing, khurana-etal-2021-emotionally, wang-etal-2023-two, summers2021nondeterminism}, or improving the experimental setup that potentially leads to the instability~\cite{zhang2021revisiting, mosbach2021on}, such as using bias correction or training for longer. As many of these strategies are designed and evaluated on high-resource datasets, almost no focus is dedicated to evaluating the mitigation benefit of data augmentation~\cite{zhang2018mixup, meng2023tuning}. At the same time, the instability of the whole training process is addressed only when considering full fine-tuning. When using PEFT methods, the focus is on the initialisation of soft prompts, even though the factors such as data shuffling still lead to variance in results~\cite{chen-etal-2022-revisiting}. Overall, the best performing mitigation strategies are ensembles and model interpolation methods~\cite{gueta-etal-2023-knowledge, hidey-etal-2022-reducing, wang-etal-2023-two}, which significantly reduce the deviation in results, but also significantly increase the computation costs. The methods that add noise to the model parameters~\cite{hua-etal-2021-noise, wu-etal-2022-noisytune} also perform well, improving generalisability and overall performance, but not necessarily reducing the instability.&#10;&#10;Inspired by the success and mutual complementarity between ensemble methods, model interpolation and noise regularisation, we propose a novel mitigation method Delayed Ensemble with Noisy Interpolation (DENI). The DENI method leverages the benefits of ensembling while reducing its computation costs. To achieve this, the ensemble is created at the end of training from a single model by perturbing its parameters using random noise (i.e., fighting randomness with randomness). In addition, the method creates the ensemble by adding noise, trains it for a few steps and then aggregates it into a single model multiple times during training (see Figure~\ref{fig:method-illustration}), which leads to more effective mitigation of the randomness. Using the DENI method leads to lower variance in results and higher performance (as illustrated in Figure~\ref{fig:variance_in_results}).&#10;&#10;To evaluate the benefit of the DENI method, we compare it with other representative mitigation strategies that modify the optimisation process. In the comparison, we also include an augmentation strategy that uses large language models to paraphrase samples, as such paraphrasing was observed to improve robustness and stability by~\cite{cegin-etal-2023-chatgpt, cegin2024effects}, especially in limited data settings. Besides full fine-tuning, we also explore the benefit of the proposed mitigation strategy and other baselines for representative parameter-efficient fine-tuning (PEFT) methods, namely LoRA, IA3 and UniPELT. &#10;&#10;Our main contributions and findings are:&#10;&#10;\begin{itemize}&#10; \item We propose DENI - a novel strategy for mitigating the randomness sensitivity of fine-tuning (originating from initialisation, data shuffling and model randomness). The proposed method leverages the benefit of ensembling and model interpolation, while reducing their computation costs using noise.&#10; \item We compare DENI with 9 representative mitigation strategies, which either modify the training process or augment the data, across 7 text classification datasets. The results show that, in comparison with the best performing baselines, the DENI method improves the overall performance and reduces the deviation in results, while introducing lower computation costs, leading to an efficient mitigation.&#10; \item We explore the benefit of mitigation strategies across 3 representative PEFT methods. We find they often benefit more from the mitigation strategies, especially data augmentation, showing a larger decrease in deviation and increase in overall performance, even outperforming full fine-tuning in specific cases.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Fine-Tuning, Computer Science, Model Performance Instability, Linguistics, Mathematics, Mitigation Strategies, Statistics" />
        </attvalues>
      </node>
      <node id="2312.01082" label="2312.01082">
        <attvalues>
          <attvalue for="0" value="A Survey on Stability of Learning with Limited Labelled Data and its&#10;  Sensitivity to the Effects of Randomness" />
          <attvalue for="1" value="  Learning with limited labelled data, such as prompting, in-context learning,&#10;fine-tuning, meta-learning or few-shot learning, aims to effectively train a&#10;model using only a small amount of labelled samples. However, these approaches&#10;have been observed to be excessively sensitive to the effects of uncontrolled&#10;randomness caused by non-determinism in the training process. The randomness&#10;negatively affects the stability of the models, leading to large variances in&#10;results across training runs. When such sensitivity is disregarded, it can&#10;unintentionally, but unfortunately also intentionally, create an imaginary&#10;perception of research progress. Recently, this area started to attract&#10;research attention and the number of relevant studies is continuously growing.&#10;In this survey, we provide a comprehensive overview of 415 papers addressing&#10;the effects of randomness on the stability of learning with limited labelled&#10;data. We distinguish between four main tasks addressed in the papers&#10;(investigate/evaluate; determine; mitigate; benchmark/compare/report randomness&#10;effects), providing findings for each one. Furthermore, we identify and discuss&#10;seven challenges and open problems together with possible directions to&#10;facilitate further research. The ultimate goal of this survey is to emphasise&#10;the importance of this growing research area, which so far has not received an&#10;appropriate level of attention, and reveal impactful directions for future&#10;research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.04835" label="2107.04835">
        <attvalues>
          <attvalue for="0" value="Noise Stability Regularization for Improving BERT Fine-tuning" />
          <attvalue for="1" value="  Fine-tuning pre-trained language models such as BERT has become a common&#10;practice dominating leaderboards across various NLP tasks. Despite its recent&#10;success and wide adoption, this process is unstable when there are only a small&#10;number of training samples available. The brittleness of this process is often&#10;reflected by the sensitivity to random seeds. In this paper, we propose to&#10;tackle this problem based on the noise stability property of deep nets, which&#10;is investigated in recent literature (Arora et al., 2018; Sanyal et al., 2020).&#10;Specifically, we introduce a novel and effective regularization method to&#10;improve fine-tuning on NLP tasks, referred to as Layer-wise Noise Stability&#10;Regularization (LNSR). We extend the theories about adding noise to the input&#10;and prove that our method gives a stabler regularization effect. We provide&#10;supportive evidence by experimentally confirming that well-performing models&#10;show a low sensitivity to noise and fine-tuning with LNSR exhibits clearly&#10;higher generalizability and stability. Furthermore, our method also&#10;demonstrates advantages over other state-of-the-art algorithms including L2-SP&#10;(Li et al., 2018), Mixout (Lee et al., 2020) and SMART (Jiang et al., 2020).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.04863" label="2302.04863">
        <attvalues>
          <attvalue for="0" value="Knowledge is a Region in Weight Space for Fine-tuned Language Models" />
          <attvalue for="1" value="  Research on neural networks has focused on understanding a single model&#10;trained on a single dataset. However, relatively little is known about the&#10;relationships between different models, particularly those trained or tested on&#10;different datasets. We address this by studying how the weight space and the&#10;underlying loss landscape of different models are interconnected.&#10;  Specifically, we demonstrate that finetuned models that were optimized for&#10;high performance, reside in well-defined regions in weight space, and vice&#10;versa -- that any model that resides anywhere in those regions also exhibits&#10;high performance. Notably, we show that language models that have been&#10;finetuned on the same dataset form a tight cluster in the weight space, while&#10;models finetuned on different datasets from the same underlying task form a&#10;looser cluster. Moreover, traversing around the region between the models leads&#10;to new models that perform comparably or even better than models obtained via&#10;finetuning, even on tasks that the original models were not finetuned on.&#10;  Our findings provide insight into the relationships between models,&#10;demonstrating that a model positioned between two similar models can acquire&#10;the knowledge of both. We leverage this and design a method for selecting a&#10;better model for efficient finetuning. Specifically, we show that starting from&#10;the center of the region is as effective, if not more, than using the&#10;pretrained model in 11 out of 12 datasets, resulting in an average accuracy&#10;improvement of 3.06.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12947" label="2305.12947">
        <attvalues>
          <attvalue for="0" value="ChatGPT to Replace Crowdsourcing of Paraphrases for Intent&#10;  Classification: Higher Diversity and Comparable Model Robustness" />
          <attvalue for="1" value="  The emergence of generative large language models (LLMs) raises the question:&#10;what will be its impact on crowdsourcing? Traditionally, crowdsourcing has been&#10;used for acquiring solutions to a wide variety of human-intelligence tasks,&#10;including ones involving text generation, modification or evaluation. For some&#10;of these tasks, models like ChatGPT can potentially substitute human workers.&#10;In this study, we investigate whether this is the case for the task of&#10;paraphrase generation for intent classification. We apply data collection&#10;methodology of an existing crowdsourcing study (similar scale, prompts and seed&#10;data) using ChatGPT and Falcon-40B. We show that ChatGPT-created paraphrases&#10;are more diverse and lead to at least as robust models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.06243" label="2409.06243">
        <attvalues>
          <attvalue for="0" value="Inference is All You Need: Self Example Retriever for Cross-domain&#10;  Dialogue State Tracking with ChatGPT" />
          <attvalue for="1" value="  Traditional dialogue state tracking approaches heavily rely on extensive&#10;training data and handcrafted features, limiting their scalability and&#10;adaptability to new domains. In this paper, we propose a novel method that&#10;leverages inference and in-context learning with ChatGPT for domain transfer in&#10;dialogue state tracking, without any parameter updates. By guiding ChatGPT's&#10;chain of thought, we enable it to retrieve relevant examples and generalize&#10;knowledge to accurately infer dialogue states, solely through inference.&#10;Experimental results on the MultiWOZ dataset demonstrate competitive&#10;performance and promising generalization across domains. Our parameter-free&#10;approach offers a scalable and adaptable solution, opening new research&#10;directions in domain transfer learning.&#10;" />
          <attvalue for="2" value="&#10;&#10;Dialogue State Tracking (DST) is a crucial component of task-oriented dialogue systems, predicting essential conversational information \cite{young2010hidden}. A common limitation in current DST systems is the requirement to pre-define the domain before the training. While this is manageable in research, it poses challenges in practical applications where users frequently request new domains.&#10;For example, consider a DST system designed to provide bus and airplane information. Users may also want to get the train information from the same service. In conventional DST, integrating a new domain involves the cumbersome process of collecting and annotating domain-specific data, followed by retraining the entire model. This inflexibility in accommodating new domains highlights the need for more versatile and efficient approaches..&#10;&#10;Several strategies have been proposed for achieving domain transfer to unseen domains by utilizing natural language form slot descriptions \cite{lin2021leveraging, heck2022robust}, leveraging question answering dataset \cite{li2021zero, sf-dst}, and augmentation with ontology \cite{campagna2020zero}. However, existing methods encounter two key challenges in terms of explainability. Firstly, it is often unclear which existing domains have contributed to the successful transition to unseen domain. Secondly, it's uncertain which characteristics of the dataset drive effective domain adaptation — whether shorter, concise utterances or information-rich, longer ones are more advantageous. Understanding the model's behavior is essential for strategic planning in collecting additional data for further training and making the model adapt to a new domain. Unfortunately, current methods operate as black boxes, limiting the utility of model insights for such planning.&#10;&#10;To this end, we developed an explainable example retrieval and In-Context Learning system for the Cross-Domain Dialogue State Tracking (DST) model by leveraging the LLM. Recent research has shown that Language Models (LLMs) can generate reliable natural language explanations. Building upon this work, we utilized LLM to create transparent and interpretable explanations for understanding the model's behavior and planning future data collection strategies. Here, we query to the LLM by asking, &quot;Is this example useful for solving this test? why?&quot;. and get the answers and explanation of why it is helpful.&#10;&#10;Furthermore, by utilizing the In-Context Learning approach, we effectively employ the retrieved results to generate DST result for unseen domain. In-context learning allows for a more transparent inspection of the inference process for new domains compared to traditional parameter update methods. This characteristic of the in-context learning approach makes it easy to understand the model's inference steps, which is well-suited to the challenges encountered in cross-domain DST research. Here, to enhance the coherance, we use same model for retreival and DST.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Dialogue State Tracking, Computer Science, Machine Learning, Linguistics, Cognitive Science, Domain Transfer Learning, Inference Based Methods, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2101.08333" label="2101.08333">
        <attvalues>
          <attvalue for="0" value="Zero-shot Generalization in Dialog State Tracking through Generative&#10;  Question Answering" />
          <attvalue for="1" value="  Dialog State Tracking (DST), an integral part of modern dialog systems, aims&#10;to track user preferences and constraints (slots) in task-oriented dialogs. In&#10;real-world settings with constantly changing services, DST systems must&#10;generalize to new domains and unseen slot types. Existing methods for DST do&#10;not generalize well to new slot names and many require known ontologies of slot&#10;types and values for inference. We introduce a novel ontology-free framework&#10;that supports natural language queries for unseen constraints and slots in&#10;multi-domain task-oriented dialogs. Our approach is based on generative&#10;question-answering using a conditional language model pre-trained on&#10;substantive English sentences. Our model improves joint goal accuracy in&#10;zero-shot domain adaptation settings by up to 9% (absolute) over the previous&#10;state-of-the-art on the MultiWOZ 2.1 dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00891" label="2005.00891">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain&#10;  Dialogue State Tracking" />
          <attvalue for="1" value="  Zero-shot transfer learning for multi-domain dialogue state tracking can&#10;allow us to handle new domains without incurring the high cost of data&#10;acquisition. This paper proposes new zero-short transfer learning technique for&#10;dialogue state tracking where the in-domain training data are all synthesized&#10;from an abstract dialogue model and the ontology of the domain. We show that&#10;data augmentation through synthesized data can improve the accuracy of&#10;zero-shot learning for both the TRADE model and the BERT-based SUMBT model on&#10;the MultiWOZ 2.1 dataset. We show training with only synthesized in-domain data&#10;on the SUMBT model can reach about 2/3 of the accuracy obtained with the full&#10;training dataset. We improve the zero-shot learning state of the art on average&#10;across domains by 21%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Dialogue Datasets and Synthesis.&#10;Synthesized data (in training and evaluation) was proposed by \newcite{weston2015towards} to evaluate the ability of neural models to reason compositionally, and was also used in visual question answering~\cite{Johnson_2017_CVPR, Hudson_2019_CVPR} and semantic parsing~\cite{lake2018generalization}.&#10;&#10;\newcite{overnight} proposed synthesizing data, then crowdsourcing paraphrases to train semantic parsers. Various semantic parsing datasets have been generated with this technique~\cite{su2017building, zhong2017seq2sql} and the technique has also been adapted to the multiturn setting~\cite{DBLP:journals/corr/abs-1812-10037, DBLP:journals/corr/abs-1801-04871}. While it tends to be well-annotated, paraphrase data is expensive to acquire, and these datasets are very small.&#10;&#10;More recently, we proposed training with both a large amount of synthesized data and a small amount of paraphrase data for semantic parsing of single sentences~\cite{geniepldi19, xu2020schema2qa}. We showed that training with such data can perform well on real-world evaluations. This paper extends this work to the multi-turn setting. Dialogues are more complex as they need to capture information, such as the abstract dialogue state, that is not present in the target annotation (domain and slot values). We extend the synthesis algorithm to operate based on a dialogue model, tracking enough information to continue the dialogue. We also present a novel dialogue model that is suitable for synthesis.&#10;&#10;Dialogue State Tracking.&#10;Dialogue state tracking is a long-studied field, starting with the first Dialogue State Tracking Challenge~\cite{williams2014dialog}. A review of prior work can be found by~\newcite{williams2016dialog}.&#10;&#10;Previous works on DST use different approaches, ranging from using handcrafted features to elicit utterance information \cite{henderson2014robust, wang2013simple}. \newcite{mrkvsic2016neural} use Convolutional Neural Networks to learn utterance representations. However, their models do not scale as they do not share parameters across different slots. \newcite{zhong2018global} and \newcite{nouri2018toward} propose a new global module that shares information to facilitate knowledge transfer. However, they rely on a predefined ontology. \newcite{xu2018end} use a pointer network with a Seq2Seq architecture to handle unseen slot values. \newcite{Lee2019Jul} use a pre-trained BERT model~\cite{Devlin2018Oct} to encode slots and utterances and uses multi-head attention~\cite{vaswani2017attention} to find relevant information in the dialogue context for predicting slot values. \newcite{Wu2019May} introduce an encoder-decoder architecture with a copy mechanism, sharing all model parameters between all domains. ~\newcite{zhou2019multi} formulate multi-domain DST as a question answering task and use reading comprehension techniques to generate the answers by either span or value prediction.&#10;&#10;\newcite{johnson2017google} propose single encoder-decoder models for zero-shot machine translation by encoding language and input sentence jointly, and \newcite{zhao2018zero} propose cross-domain zero-shot language generation using a cross-domain embedding space.&#10;&#10;Modelling of Dialogues.&#10;Previous work already proposed general models of dialogues as finite state machines~\cite{jurafsky1997switchboard, bunt2017revisiting, yu2019midas}. Existing models are optimized for analyzing existing human conversations. Our dialogue model is the first suitable for synthesis, carrying enough information to continue the dialogue.&#10;&#10;\newcite{DBLP:journals/corr/abs-1810-07942} previously proposed a different annotation scheme for dialogues, using a hierarchical representation scheme, instead of the more typical intent and slot. Their work is complementary to ours: our method of dialogue synthesis is applicable to any annotation scheme. In this paper, we focus on the existing annotation scheme used by the MultiWOZ dataset.&#10; " />
        </attvalues>
      </node>
      <node id="2104.04923" label="2104.04923">
        <attvalues>
          <attvalue for="0" value="Non-Autoregressive Semantic Parsing for Compositional Task-Oriented&#10;  Dialog" />
          <attvalue for="1" value="  Semantic parsing using sequence-to-sequence models allows parsing of deeper&#10;representations compared to traditional word tagging based models. In spite of&#10;these advantages, widespread adoption of these models for real-time&#10;conversational use cases has been stymied by higher compute requirements and&#10;thus higher latency. In this work, we propose a non-autoregressive approach to&#10;predict semantic parse trees with an efficient seq2seq model architecture. By&#10;combining non-autoregressive prediction with convolutional neural networks, we&#10;achieve significant latency gains and parameter size reduction compared to&#10;traditional RNN models. Our novel architecture achieves up to an 81% reduction&#10;in latency on TOP dataset and retains competitive performance to non-pretrained&#10;models on three different semantic parsing datasets. Our code is available at&#10;https://github.com/facebookresearch/pytext&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Non-autoregressive Decoding&#10;Recent work in machine translation has made a lot of progress in fully non-autoregressive models \cite{gu2017non, ma2019flowseq,ghazvininejad2020aligned,saharia2020non} and parallel decoding \cite{lee2018,ghazvininejad2019maskpredict, gu2019levenshtein,ghazvininejad2020semi,kasai2020parallel}. &#10;While many advancements have been made in machine translation, we believe we are the first to explore the non-autoregressive semantic parsing setting. In our work, we extend the CMLM to work for semantic parsing. We make two important adjustments: first, we use a different masking approach where we mask everything and do one-step generation. Second, we note the importance of the length prediction task for parsing and improve the length prediction module in the CMLM.&#10;&#10;Seq2Seq For Semantic Parsing&#10;Recent advances in language understanding have lead to increased reliance on seq2seq architectures. Recent work by \cite{rongali2020don, decoupled}, showed the advantages from using a pointer generator architecture for resolving complex queries (e.g. composition and cross domain queries) that could not be handled by word tagging models. Since we target the same task, we adapt their pointer decoder into our proposed architecture. However, to optimize for latency and compression we train CNN based architectures (\cite{desai2020lightweight} and \cite{wu2019pay}) to leverage the inherent model parallelism compared to the BiLSTM model proposed in \cite{decoupled} and more compression compared to the transformer seq2seq baseline proposed in \cite{rongali2020don}. To further improve latency we look at parallel decoding through non-autoregressive decoding compared to prior work leveraging autoregressive models.&#10;&#10;" />
          <attvalue for="4" value="Semantic Parsing, Computer Science, Linguistics, Sequence-to-Sequence Models, Artificial Intelligence, Neural Network Optimization" />
        </attvalues>
      </node>
      <node id="2210.06175" label="2210.06175">
        <attvalues>
          <attvalue for="0" value="Exploring Efficient-tuning Methods in Self-supervised Speech Models" />
          <attvalue for="1" value="  In this study, we aim to explore efficient tuning methods for speech&#10;self-supervised learning. Recent studies show that self-supervised learning&#10;(SSL) can learn powerful representations for different speech tasks. However,&#10;fine-tuning pre-trained models for each downstream task is&#10;parameter-inefficient since SSL models are notoriously large with millions of&#10;parameters. Adapters are lightweight modules commonly used in NLP to solve this&#10;problem. In downstream tasks, the parameters of SSL models are frozen, and only&#10;the adapters are trained. Given the lack of studies generally exploring the&#10;effectiveness of adapters for self-supervised speech tasks, we intend to fill&#10;this gap by adding various adapter modules in pre-trained speech SSL models. We&#10;show that the performance parity can be achieved with over 90% parameter&#10;reduction, and discussed the pros and cons of efficient tuning techniques. This&#10;is the first comprehensive investigation of various adapter types across speech&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2111.10367" label="2111.10367">
        <attvalues>
          <attvalue for="0" value="SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation&#10;  on Natural Speech" />
          <attvalue for="1" value="  Progress in speech processing has been facilitated by shared datasets and&#10;benchmarks. Historically these have focused on automatic speech recognition&#10;(ASR), speaker identification, or other lower-level tasks. Interest has been&#10;growing in higher-level spoken language understanding tasks, including using&#10;end-to-end models, but there are fewer annotated datasets for such tasks. At&#10;the same time, recent work shows the possibility of pre-training generic&#10;representations and then fine-tuning for several tasks using relatively little&#10;labeled data. We propose to create a suite of benchmark tasks for Spoken&#10;Language Understanding Evaluation (SLUE) consisting of limited-size labeled&#10;training sets and corresponding evaluation sets. This resource would allow the&#10;research community to track progress, evaluate pre-trained representations for&#10;higher-level tasks, and study open questions such as the utility of pipeline&#10;versus end-to-end approaches. We present the first phase of the SLUE benchmark&#10;suite, consisting of named entity recognition, sentiment analysis, and ASR on&#10;the corresponding datasets. We focus on naturally produced (not read or&#10;synthesized) speech, and freely available datasets. We provide new&#10;transcriptions and annotations on subsets of the VoxCeleb and VoxPopuli&#10;datasets, evaluation metrics and results for baseline models, and an&#10;open-source toolkit to reproduce the baselines and evaluate new models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10615" label="2305.10615">
        <attvalues>
          <attvalue for="0" value="ML-SUPERB: Multilingual Speech Universal PERformance Benchmark" />
          <attvalue for="1" value="  Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard&#10;to benchmark the performance of Self-Supervised Learning (SSL) models on&#10;various speech processing tasks. However, SUPERB largely considers English&#10;speech in its evaluation. This paper presents multilingual SUPERB (ML-SUPERB),&#10;covering 143 languages (ranging from high-resource to endangered), and&#10;considering both automatic speech recognition and language identification.&#10;Following the concept of SUPERB, ML-SUPERB utilizes frozen SSL features and&#10;employs a simple framework for multilingual tasks by learning a shallow&#10;downstream model. Similar to the SUPERB benchmark, we find speech SSL models&#10;can significantly improve performance compared to FBANK features. Furthermore,&#10;we find that multilingual models do not always perform better than their&#10;monolingual counterparts. We will release ML-SUPERB as a challenge with&#10;organized datasets and reproducible training scripts for future multilingual&#10;representation research.&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-supervised learning (SSL) has been a popular method in the speech community. &#10;SSL models have shown promising results by capturing important speech features, such as phonemes and other acoustic units, through training on large amounts of unlabeled speech data \cite{mohamed2022self}. These models have led to significant improvements in downstream tasks, such as speech recognition, speaker identification, and emotion recognition \cite{yang21c_interspeech}. Over the past few years, researchers have proposed a variety of SSL models with different training objectives, operating under various data conditions, model architectures, and modalities \cite{baevski2020wav2vec, hsu2021hubert}.&#10;&#10;A major challenge in evaluating SSL models for speech is the difficulty of comparison since most models have been evaluated using different experimental setups. To address this issue, Yang et al. introduced the Speech processing Universal PERformance Benchmark (SUPERB) \cite{yang21c_interspeech}. Recently, an extension of SUPERB called SUPERB-SG \cite{tsai2022superb} has been introduced. SUPERB provides a comprehensive speech SSL benchmark including tasks such as recognition, detection, semantics, speaker identification, paralinguistics, and generation. With SUPERB, researchers can more easily compare the performance of different SSL models on various speech-related tasks, universally.&#10;&#10;While SUPERB covers a wide range of speech tasks, it was designed primarily for English speech. However, there has been growing interest in applying SSL models to multilingual scenarios, such as training multilingual SSL models \cite{babu2021xls, conneau2020unsupervised, duquenne2022speechmatrix} or using SSL models in a cross-lingual manner \cite{zhao2022improving, berrebbi22_interspeech, wu2020self, li22aa_interspeech}. To support future research in these areas, we propose a new benchmark called multilingual SUPERB (ML-SUPERB). &#10;&#10;ML-SUPERB is designed to cover a wide range of languages, including both high-resource languages like English and endangered languages such as Totonac. The benchmark primarily focuses on evaluating SSL models for automatic speech recognition (ASR) and language identification (LID). To accommodate different use cases for SSL models, ML-SUPERB includes two tracks with four different tasks: the monolingual track (monolingual ASR), and the multilingual track (multilingual ASR, LID, joint multilingual ASR/LID). Similar to SUPERB, ML-SUPERB employs frozen SSL models as feature extractors and a lightweight downstream model that can be fine-tuned for different tracks to achieve high training efficiency.&#10;&#10;Several existing benchmarks also include multilingual SSL models \cite{evain21_interspeech, javed2022indicsuperb, conneau22_interspeech}. Lebenchmark primarily evaluates speech tasks in French \cite{evain21_interspeech}; IndicSUPERB focuses mostly on Indian languages \cite{javed2022indicsuperb}. XTREME-S focuses on multilingual speech representation benchmarks, including ASR, speech translation, speech classification, and speech retrieval \cite{conneau22_interspeech}. There are three main differences between XTREME-S and ML-SUPERB. Firstly, ML-SUPERB covers a wider range of languages, with 143 languages compared to XTREME-S's 102. Secondly, ML-SUPERB focuses on ASR and LID, while XTREME-S covers four different tasks. However, ML-SUPERB expands the tasks by evaluating them in four common multilingual research scenarios, while XTREME-S considers multilingual training only. Finally, ML-SUPERB is designed for efficiency, using smaller benchmark datasets and downstream models, and does not include fine-tuning. This lightweight setup allows us to conduct experiments for a dozen of popular speech SSL models, trained with various sizes and pre-training sets, and compare their performances across the proposed tracks. We expect ML-SUPERB would be a valuable complement to existing benchmarks.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12780" label="2010.12780">
        <attvalues>
          <attvalue for="0" value="Open-Domain Dialogue Generation Based on Pre-trained Language Models" />
          <attvalue for="1" value="  Pre-trained language models have been successfully used in response&#10;generation for open-domain dialogue. Four main frameworks have been proposed:&#10;(1) Transformer-ED using Transformer encoder and decoder separately for source&#10;and target sentences; (2) Transformer-Dec using Transformer decoder for both&#10;source and target sentences; (3) Transformer-MLM using Transformer decoder that&#10;applies bi-directional attention on the source side and left-to-right attention&#10;on the target side with masked language model objective; and (4) Transformer-AR&#10;that uses auto-regressive objective instead. In this study, we compare these&#10;frameworks on 3 datasets, and our comparison reveals that the best framework&#10;uses bidirectional attention on the source side and does not separate encoder&#10;and decoder. We also examine model discrepancy, and our experiments confirm&#10;that the performance of a model is directly impacted by the underlying&#10;discrepancies. We then propose two correction methods to reduce the&#10;discrepancies, and both improve the model performance. These results show that&#10;discrepancies is an important factor to consider when we use a pre-trained&#10;model, and a reduction in discrepancies can lead to improved performance.&#10;" />
          <attvalue for="2" value=" &#10;General purpose (non goal-oriented) dialogue has been investigated using data-driven sequence-to-sequence (SEQ2SEQ) recurrent neural networks (RNN) \cite{sordoni2015neural, shang2015neural, wen2015semantically, vinyals2015neural}. Recently, fine-tuning pre-trained language models has demonstrated superior performance in the ConvAI2 \cite{dinan2019second} competition. &#10;Several other studies also successfully exploited pre-trained language models for dialogue generation. It is intuitive that the rich language knowledge encoded in the model pre-trained on a large amount of raw texts can help dialogue systems to generate more reasonable responses. While one can easily agree on this principle, it is much more difficult to determine how a language model should be adapted for dialogue generation.&#10;Different frameworks have been proposed in the literature:&#10;Transformer-ED (explicit encoder and decoder architecture) \cite{zheng2019pre}, Transformer-Dec (decoder only) \cite{wolf2019transfertransfo,lin2019caire,zhang2019dialogpt}, Transformer decoder that uses bi-directional attention on the source side and left-to-right attention on the target side with Masked Language Model (MLM) objective (Transformer-MLM) \cite{dong2019unified} or with Auto-Regressive (AR) objective (Transformer-AR) \cite{bao2019plato, shuster2019dialogue}. The first two frameworks utilize Generative Pre-Training (GPT) \cite{radford2018improving}, a left-to-right architecture pre-trained with AR. &#10;The left-to-right generation fashion corresponds well to that of dialogue response generation. Thus, some researchers believe that this framework naturally works well for dialogue response generation \cite{lin2020exploring}. The two latter frameworks use BERT \cite{devlin2018bert}, a bi-directional architecture pre-trained with MLM. BERT has been used widely as the encoder for classification tasks \cite{zhang2019find, zeng2020multidomain}, while some studies \cite{dong2019unified, zeng2020generalized} show that fine-tuning BERT can also achieve state-of-the-art performance for response generation in dialogue. To our knowledge, no study has investigated into the fine-tuning methods to understand why and how a pre-trained language model can be fine-tuned for dialogue generation. The choice of a pre-trained language model and the way to adapt it are still an art.&#10;&#10;In this study, we aim at providing an analysis about the utilization of pre-trained language models for dialogue generation. To this end, we re-implement the existing approaches proposed in the literature and run extensive experiments on 3 datasets to compare them. Our results show that Transformer-ED that separates encoder and decoder does not produce competitive results against others that combine them, and models that use bi-directional attention to encode dialogue history outperforms the one using unidirectional (left-to-right) attention. However, an advantage of using unidirectional attention is generating diverse responses. Additionally, this comparison reveals some important aspects that were neglected in the utilization of pre-trained models, namely, the discrepancies that may occur between pre-training and the fine-tuning processes and between fine-tuning and the generation (inference) process. &#10;&#10;The concept of model discrepancy &#10;has been briefly mentioned in \cite{yang2019xlnet} to mean that the model has been trained in a way, but used in a different way for the task. However, the problem has not been investigated in depth. We believe that model discrepancy is a very important aspect that can bring a significant impact on the final result. Going further in this direction, we define two discrepancies: pretrain-finetune discrepancy which means the differences in architecture and loss function between pre-training and fine-tuning, and finetune-generation discrepancy which means that the way it is used in generation (inference/test) is different from the way it has been trained. For the four frameworks we mentioned for dialogue generation based on pre-trained models, except Transformer-Dec, they all have some pretrain-finetune discrepancies, while only Transformer-MLM has finetune-generation discrepancy because of MLM objective: during training, the model input has random masks, while in generation process, the input does not contain masks (see Figure \ref{Fig:finegen}). We summarize the discrepancies of different models in (Table \ref{tab:frameworks}).&#10;&#10;Discrepancies may affect the model performance since models with such discrepancies cannot best exploit the pre-trained language model or the fine-tuned model. The impact of pretrain-finetune discrepancy could be reduced by using a large dataset since a large amount of training data can correct the discrepancies to some extent. This also explains why discrepancy has not attracted much attention -- most of the work on dialogue generation tries to use as much training data as possible. &#10;However, if the amount of training data is limited, the discrepancy problem may surface. Therefore, we will analyze the performance of the models with both large and small amount of training data in order to make it easier to observe the impact of discrepancies.&#10;In particular, our experiments will show that Transformer-ED and Transformer-AR, which have the largest pretrain-finetune discrepancy, are more impacted than Transformer-MLM and Transformer-Dec by a small amount of data. &#10;&#10;To further confirm that discrepancies are truly the hidden reason of model performance, we propose correction measures to reduce pretrain-finetune discrepancy and finetune-generation discrepancy of Transformer-MLM, in order to see if these measures can avoid the negative impact of discrepancies. These results show that the performance of the model is increased when the discrepancy issues are corrected, confirming that discrepancies are indeed an important factor that influence the effectiveness of a pre-trained model for dialogue generation.&#10;This study is the first investigation to show explicitly the phenomenon of model discrepancy and its impact on performance. It can be seen as an appeal to more investigations on this important problem when we adapt a pre-trained model.&#10;&#10;The contributions in this work are summarized as: &#10;\begin{itemize}&#10;\item We re-implement and compare four major frameworks that utilize pre-trained language models for dialog generation on three public dialogue datasets and we consider two data scales. Our experimental results will support our analysis on model's architectural appropriateness.&#10;&#10;\item We formally introduce the concept of pretrain-finetune discrepancy and finetune-generation discrepancy when exploit a pre-trained model. We examine the discrepancies of each framework and confirm the impact of discrepancies in experiments.&#10;&#10;\item We propose two correction methods to decrease pretrain-finetune discrepancy and finetune-generation discrepancy of Transformer-MLM. Both corrections lead to increased performance of Transformer-MLM, confirming that the correction of discrepancies is an important aspect to consider in utilizing a pre-trained model.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Frameworks, Computer Science, Linguistics, Cognitive Science, Model Discrepancy Analysis, Mathematics, Pre-trained Model Optimization" />
        </attvalues>
      </node>
      <node id="1902.00098" label="1902.00098">
        <attvalues>
          <attvalue for="0" value="The Second Conversational Intelligence Challenge (ConvAI2)" />
          <attvalue for="1" value="  We describe the setting and results of the ConvAI2 NeurIPS competition that&#10;aims to further the state-of-the-art in open-domain chatbots. Some key&#10;takeaways from the competition are: (i) pretrained Transformer variants are&#10;currently the best performing models on this task, (ii) but to improve&#10;performance on multi-turn conversations with humans, future systems must go&#10;beyond single word metrics like perplexity to measure the performance across&#10;sequences of utterances (conversations) -- in terms of repetition, consistency&#10;and balance of dialogue acts (e.g. how many questions asked vs. answered).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.08149" label="1901.08149">
        <attvalues>
          <attvalue for="0" value="TransferTransfo: A Transfer Learning Approach for Neural Network Based&#10;  Conversational Agents" />
          <attvalue for="1" value="  We introduce a new approach to generative data-driven dialogue systems (e.g.&#10;chatbots) called TransferTransfo which is a combination of a Transfer learning&#10;based training scheme and a high-capacity Transformer model. Fine-tuning is&#10;performed by using a multi-task objective which combines several unsupervised&#10;prediction tasks. The resulting fine-tuned model shows strong improvements over&#10;the current state-of-the-art end-to-end conversational models like memory&#10;augmented seq2seq and information-retrieval models. On the privately held&#10;PERSONA-CHAT dataset of the Conversational Intelligence Challenge 2, this&#10;approach obtains a new state-of-the-art, with respective perplexity, Hits@1 and&#10;F1 metrics of 16.28 (45 % absolute improvement), 80.7 (46 % absolute&#10;improvement) and 19.5 (20 % absolute improvement).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.07931" label="1910.07931">
        <attvalues>
          <attvalue for="0" value="PLATO: Pre-trained Dialogue Generation Model with Discrete Latent&#10;  Variable" />
          <attvalue for="1" value="  Pre-training models have been proved effective for a wide range of natural&#10;language processing tasks. Inspired by this, we propose a novel dialogue&#10;generation pre-training framework to support various kinds of conversations,&#10;including chit-chat, knowledge grounded dialogues, and conversational question&#10;answering. In this framework, we adopt flexible attention mechanisms to fully&#10;leverage the bi-directional context and the uni-directional characteristic of&#10;language generation. We also introduce discrete latent variables to tackle the&#10;inherent one-to-many mapping problem in response generation. Two reciprocal&#10;tasks of response generation and latent act recognition are designed and&#10;carried out simultaneously within a shared network. Comprehensive experiments&#10;on three publicly available datasets verify the effectiveness and superiority&#10;of the proposed framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03829" label="2004.03829">
        <attvalues>
          <attvalue for="0" value="Exploring Versatile Generative Language Model Via Parameter-Efficient&#10;  Transfer Learning" />
          <attvalue for="1" value="  Fine-tuning pre-trained generative language models to down-stream language&#10;generation tasks has shown promising results. However, this comes with the cost&#10;of having a single, large model for each task, which is not ideal in&#10;low-memory/power scenarios (e.g., mobile). In this paper, we propose an&#10;effective way to fine-tune multiple down-stream generation tasks simultaneously&#10;using a single, large pre-trained model. The experiments on five diverse&#10;language generation tasks show that by just using an additional 2-3% parameters&#10;for each task, our model can maintain or even improve the performance of&#10;fine-tuning the whole model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-trained generative language models~\cite{radford2019language,radford2018improving,dai-etal-2019-transformer,yang2019xlnet,peters2018deep} have shown to be very effective in language generation, whereas, bidirectional pre-trained models~\cite{devlin2019bert,liu2019roberta,sanh2019distilbert} significantly improve the performance of several down-stream classification tasks. Fine-tuning large pre-trained models has shown positive results in dialogue tasks~\cite{DBLP:journals/corr/abs-1901-08149, budzianowski2019hello} and other language generation tasks~\cite{dong2019unified}. However, all of the previous works only consider fine-tuning on each generation task individually, which requires a separate model for each task. In this work, we use only a single model, for multiple generation tasks. &#10;&#10;Residual adapters, derived from residual networks~\cite{he2016deep}, were first introduced by \cite{rebuffi2017learning} for multiple visual domain learning.&#10;\cite{houlsby2019parameter} proposed low-rank residual adapters to improve the scalability of the adapter module, and effectively transfer BERT~\cite{devlin2019bert} to multiple text classification tasks simultaneously, while \cite{bapna2019simple} applied an adapter layer to language/domain adaptation for neural machine translation. On the other hand, \cite{dathathri2019plug} proposed a plug and play method to control the language model generation without finetuning the model. Differently, in this paper, we extend the idea of adapters to a large variety of language generation tasks, which has not been considered before, and we compare the idea of a fixed pre-trained back-bone for continual learning with multi-task training~\cite{stickland2019bert}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1910.03544" label="1910.03544">
        <attvalues>
          <attvalue for="0" value="Find or Classify? Dual Strategy for Slot-Value Predictions on&#10;  Multi-Domain Dialog State Tracking" />
          <attvalue for="1" value="  Dialog state tracking (DST) is a core component in task-oriented dialog&#10;systems. Existing approaches for DST mainly fall into one of two categories,&#10;namely, ontology-based and ontology-free methods. An ontology-based method&#10;selects a value from a candidate-value list for each target slot, while an&#10;ontology-free method extracts spans from dialog contexts. Recent work&#10;introduced a BERT-based model to strike a balance between the two methods by&#10;pre-defining categorical and non-categorical slots. However, it is not clear&#10;enough which slots are better handled by either of the two slot types, and the&#10;way to use the pre-trained model has not been well investigated. In this paper,&#10;we propose a simple yet effective dual-strategy model for DST, by adapting a&#10;single BERT-style reading comprehension model to jointly handle both the&#10;categorical and non-categorical slots. Our experiments on the MultiWOZ datasets&#10;show that our method significantly outperforms the BERT-based counterpart,&#10;finding that the key is a deep interaction between the domain-slot and context&#10;information. When evaluated on noisy (MultiWOZ 2.0) and cleaner (MultiWOZ 2.1)&#10;settings, our method performs competitively and robustly across the two&#10;different settings. Our method sets the new state of the art in the noisy&#10;setting, while performing more robustly than the best model in the cleaner&#10;setting. We also conduct a comprehensive error analysis on the dataset,&#10;including the effects of the dual strategy for each slot, to facilitate future&#10;research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.13892" label="2311.13892">
        <attvalues>
          <attvalue for="0" value="General Phrase Debiaser: Debiasing Masked Language Models at a&#10;  Multi-Token Level" />
          <attvalue for="1" value="  The social biases and unwelcome stereotypes revealed by pretrained language&#10;models are becoming obstacles to their application. Compared to numerous&#10;debiasing methods targeting word level, there has been relatively less&#10;attention on biases present at phrase level, limiting the performance of&#10;debiasing in discipline domains. In this paper, we propose an automatic&#10;multi-token debiasing pipeline called \textbf{General Phrase Debiaser}, which&#10;is capable of mitigating phrase-level biases in masked language models.&#10;Specifically, our method consists of a \textit{phrase filter stage} that&#10;generates stereotypical phrases from Wikipedia pages as well as a \textit{model&#10;debias stage} that can debias models at the multi-token level to tackle bias&#10;challenges on phrases. The latter searches for prompts that trigger model's&#10;bias, and then uses them for debiasing. State-of-the-art results on standard&#10;datasets and metrics show that our approach can significantly reduce gender&#10;biases on both career and multiple disciplines, across models with varying&#10;parameter sizes.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, masked language models (MLMs) \cite{devlin2018bert, lanalbert, liu2019roberta, NEURIPS2019_dc6a7e65, sanh2019distilbert, zhang2021you} are employed in both traditional tasks like text classification \cite{kowsari2019text, zhang2023ideal, zhang2023hypertime} and diverse multimodal tasks \cite{wu2023autogen, wu2023empirical} when combined with models like image generators \cite{radford2021learning, rombach2022high}. We aim to develop MLMs with minimal human biases, even when the pretraining data unavoidably contains these biases. However, correcting implicit biases in pretrained MLMs can be very challenging, especially considering the high cost of retraining models from scratch. &#10;&#10;Existing studies \cite{liang2020towards, kaneko2021debiasing, garimella2021he, ct1965, webster2020measuring, chengfairfil} have introduced intuitive approaches that use additional corpus to retrieve contextualized embeddings or locate the biases and fine-tune accordingly. But they are rely on external human-written corpus. Auto-Debias\cite{guo2022auto} hires the prompt\cite{jiang2020can} template &quot;[attribute word] [T]...[T] [MASK]&quot; to guide MLMs to automatically search for prompts that makes the model show its bias, and then fine-tune MLMs with them. Nevertheless, real-world language environments are not so ideal, meaning both attribute words and stereotypes should be treated as multi-token. While these method only correct biases at the word level, lead to struggling at the phrase level. &#10;&#10;Motivated by this, we propose an automatic multi-token debias pipeline called General Phrase Debiaser to address the limitations of automatic debiasing mentioned above. The major contributions of our work are:&#10;\begin{itemize}&#10;&#09;\item Unlike existing methods, we debias MLMs at the phrase granularity. In order to reduce the cost of manually constructing the phrase list, we get the stereotypical phrases filtered from hyperlinks of Wikipedia pages in Phrase Filter Stage. &#10;&#09;\item With the multi-token debias head we proposed, “discriminative” prompts can be searched in Model Debias Stage. These cloze-style prompts have the highest disagreement in generating stereotypical phrases (e.g., mathematical theory/dance art) with respect to demographic words (e.g., man/woman). Then we fine-tune the model using searched prompts.&#10;&#09;\item Different from the Auto-Debias' fine-tuning stage, our approach derives loss from stereotypical phrases, rather than from the entire vocabulary belonging to the model itself. This allows our method to adjust the model parameters more specifically without affecting any other gender-independent word or knowledge. &#10;&#09;\item We conduct experiments on three well-known open-source MLMs: BERT\cite{devlin2018bert}, ALBERT\cite{lanalbert}, and DistilBERT\cite{sanh2019distilbert}, and achieves state-of-the-art performance (0.12, 0.16, and 52) on SEAT test.&#10;\end{itemize}&#10;&#10;Our code and debiased model files are available at &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Social Computing, Computer Science, Linguistics, Language Model Bias, Artificial Intelligence, Debiasing Techniques, Social Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1810.06553" label="1810.06553">
        <attvalues>
          <attvalue for="0" value="Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking&#10;  Recipes and Food Images" />
          <attvalue for="1" value="  In this paper, we introduce Recipe1M+, a new large-scale, structured corpus&#10;of over one million cooking recipes and 13 million food images. As the largest&#10;publicly available collection of recipe data, Recipe1M+ affords the ability to&#10;train high-capacity modelson aligned, multimodal data. Using these data, we&#10;train a neural network to learn a joint embedding of recipes and images that&#10;yields impressive results on an image-recipe retrieval task. Moreover, we&#10;demonstrate that regularization via the addition of a high-level classification&#10;objective both improves retrieval performance to rival that of humans and&#10;enables semantic vector arithmetic. We postulate that these embeddings will&#10;provide a basis for further exploration of the Recipe1M+ dataset and food and&#10;cooking in general. Code, data and models are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.13413" label="1912.13413">
        <attvalues>
          <attvalue for="0" value="Semantics- and Syntax-related Subvectors in the Skip-gram Embeddings" />
          <attvalue for="1" value="  We show that the skip-gram embedding of any word can be decomposed into two&#10;subvectors which roughly correspond to semantic and syntactic roles of the&#10;word.&#10;" />
          <attvalue for="2" value="&#10;Assuming that words have already been converted into indices, let $\{1,\ldots,n\}$ be a finite vocabulary of words. Following the setups of the widely used {\sc word2vec} \cite{mikolov2013distributed} model, we consider two vectors per each word $i$: &#10;\begin{itemize}&#10; \item $\mathbf{w}_i$ is an embedding of the word $i$ when $i$ is a center word,&#10; \item $\mathbf{c}_i$ is an embedding of the word $i$ when $i$ is a context word.&#10;\end{itemize} &#10;We follow the assumptions of \cite{assylbekov2019context} \shortcite{assylbekov2019context} on the nature of word vectors, context vectors, and text generation, i.e.&#10;\begin{enumerate}&#10; \item A priori word vectors $\mathbf{w}_1,\ldots,\mathbf{w}_n\in\mathbb{R}^{d}$ are i.i.d. draws from isotropic multivariate Gaussian distribution:&#10; $\mathbf{w}_i\,\,{\stackrel{iid}{\sim}}\,\,\mathcal{N}(\mathbf{0},\,\textstyle{\frac1d}\mathbf{I})$, where $\mathbf{I}$ is the $d\times d$ identity matrix. &#10; \item Context vectors $\mathbf{c}_1,\ldots,\mathbf{c}_n$ are related to word vectors according to&#10; $\mathbf{c}_i=\mathbf{Qw}_i$, $i=1,\ldots,n$, for some orthogonal matrix $\mathbf{Q}\in\mathbb{R}^{d\times d}$. &#10; \item Given a word $j$, the probability of any word $i$ being in its context is given by&#10;\begin{equation}&#10;p(i\mid j) \propto p_i\cdot{e^{\mathbf{w}_j^\top\mathbf{c}_i}}&#10;\end{equation}&#10;where $p_i=p(i)$ is the unigram probability for the word $i$.&#10;\end{enumerate}&#10;&#10;Hypothesis. Under the assumptions \ref{assump_1}--\ref{assump_3} above, \cite{assylbekov2019context} \shortcite{assylbekov2019context} showed that each word's vector $\mathbf{w}_i$ splits into two approximately equally-sized subvectors $\mathbf{x}_i$ and $\mathbf{y}_i$, and the model \ref{eq:model} for generating a word $i$ in the context of a word $j$ can be rewritten as&#10;$$&#10;p(i\mid j)\approx p_i\cdot e^{\mathbf{x}_j^\top\mathbf{x}_i-\mathbf{y}_j^\top\mathbf{y}_i}.&#10;$$&#10;Interestingly, embeddings of the first type ($\mathbf{x}_i$ and $\mathbf{x}_j$) are responsible for pulling the word $i$ into the context of the word $j$, while embeddings of the second type ($\mathbf{y}_i$ and $\mathbf{y}_j$) are responsible for pushing the word $i$ away from the context of the word $j$. We hypothesize that the $\mathbf{x}$-embeddings are more related to semantics, whereas the $\mathbf{y}$-embeddings are more related to syntax. In what follows we provide a motivating example for this hypothesis and then empirically validate it through controlled experiments.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Word Embeddings, Mathematics, Natural Language Processing, Syntactic Roles, Semantic Analysis" />
        </attvalues>
      </node>
      <node id="1902.09859" label="1902.09859">
        <attvalues>
          <attvalue for="0" value="Context Vectors are Reflections of Word Vectors in Half the Dimensions" />
          <attvalue for="1" value="  This paper takes a step towards theoretical analysis of the relationship&#10;between word embeddings and context embeddings in models such as word2vec. We&#10;start from basic probabilistic assumptions on the nature of word vectors,&#10;context vectors, and text generation. These assumptions are well supported&#10;either empirically or theoretically by the existing literature. Next, we show&#10;that under these assumptions the widely-used word-word PMI matrix is&#10;approximately a random symmetric Gaussian ensemble. This, in turn, implies that&#10;context vectors are reflections of word vectors in approximately half the&#10;dimensions. As a direct application of our result, we suggest a theoretically&#10;grounded way of tying weights in the SGNS model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.03130" label="2408.03130">
        <attvalues>
          <attvalue for="0" value="Inference Optimizations for Large Language Models: Effects, Challenges,&#10;  and Practical Considerations" />
          <attvalue for="1" value="  Large language models are ubiquitous in natural language processing because&#10;they can adapt to new tasks without retraining. However, their sheer scale and&#10;complexity present unique challenges and opportunities, prompting researchers&#10;and practitioners to explore novel model training, optimization, and deployment&#10;methods. This literature review focuses on various techniques for reducing&#10;resource requirements and compressing large language models, including&#10;quantization, pruning, knowledge distillation, and architectural optimizations.&#10;The primary objective is to explore each method in-depth and highlight its&#10;unique challenges and practical applications. The discussed methods are&#10;categorized into a taxonomy that presents an overview of the optimization&#10;landscape and helps navigate it to understand the research trajectory better.&#10;" />
          <attvalue for="2" value="&#10;In recent years, Large Language Models (LLMs) have emerged as the cornerstone of Natural Language Processing (NLP), revolutionizing various domains with unprecedented capabilities. These versatile models have demonstrated remarkable abilities in diverse applications, ranging from assisting in code generation \cite{Li2023StarCoderMT} \cite{Chen2021EvaluatingLL}, to facilitating news summarization \cite{Wei2022EmergentAO} \cite{Lewis2019BARTDS}, and even augmenting information retrieval systems for improved search accuracy and efficiency \cite{Lewis2020RetrievalAugmentedGF} \cite{DBLP:journals/corr/abs-2312-10997}.&#10;Furthermore, these models' sheer scale and complexity present unique challenges and opportunities, prompting researchers and practitioners to explore novel model training, optimization, and deployment methods. Optimizing large models for speed, reducing resource consumption, and making them more accessible is a significant part of LLM research.&#10;&#10;The primary objective of this research paper is to explore various techniques for reducing resource requirements and compressing large language models, including analyzing each method in-depth and highlighting its unique challenges and practical implications. The discussed methods include quantization, pruning, knowledge distillation, and architectural optimizations. To better understand the relationship between these techniques, they are categorized into a taxonomy that presents an overview of the optimization landscape and helps navigate it for a better understanding of the research trajectory. Refer to figure \ref{fig:taxonomy} for a visual representation of the categorization and to the respective sectionfor a more detailed look at the discussed literature in each category.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Resource Reduction Techniques, Mathematics, Model Compression Methods, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2305.06161" label="2305.06161">
        <attvalues>
          <attvalue for="0" value="StarCoder: may the source be with you!" />
          <attvalue for="1" value="  The BigCode community, an open-scientific collaboration working on the&#10;responsible development of Large Language Models for Code (Code LLMs),&#10;introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context&#10;length, infilling capabilities and fast large-batch inference enabled by&#10;multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced&#10;from The Stack, a large collection of permissively licensed GitHub repositories&#10;with inspection tools and an opt-out process. We fine-tuned StarCoderBase on&#10;35B Python tokens, resulting in the creation of StarCoder. We perform the most&#10;comprehensive evaluation of Code LLMs to date and show that StarCoderBase&#10;outperforms every open Code LLM that supports multiple programming languages&#10;and matches or outperforms the OpenAI code-cushman-001 model. Furthermore,&#10;StarCoder outperforms every model that is fine-tuned on Python, can be prompted&#10;to achieve 40\% pass@1 on HumanEval, and still retains its performance on other&#10;programming languages. We take several important steps towards a safe&#10;open-access model release, including an improved PII redaction pipeline and a&#10;novel attribution tracing tool, and make the StarCoder models publicly&#10;available under a more commercially viable version of the Open Responsible AI&#10;Model license.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.04363" label="2310.04363">
        <attvalues>
          <attvalue for="0" value="Amortizing intractable inference in large language models" />
          <attvalue for="1" value="  Autoregressive large language models (LLMs) compress knowledge from their&#10;training data through next-token conditional distributions. This limits&#10;tractable querying of this knowledge to start-to-end autoregressive sampling.&#10;However, many tasks of interest -- including sequence continuation, infilling,&#10;and other forms of constrained generation -- involve sampling from intractable&#10;posterior distributions. We address this limitation by using amortized Bayesian&#10;inference to sample from these intractable posteriors. Such amortization is&#10;algorithmically achieved by fine-tuning LLMs via diversity-seeking&#10;reinforcement learning algorithms: generative flow networks (GFlowNets). We&#10;empirically demonstrate that this distribution-matching paradigm of LLM&#10;fine-tuning can serve as an effective alternative to maximum-likelihood&#10;training and reward-maximizing policy optimization. As an important&#10;application, we interpret chain-of-thought reasoning as a latent variable&#10;modeling problem and demonstrate that our approach enables data-efficient&#10;adaptation of LLMs to tasks that require multi-step rationalization and tool&#10;use.&#10;" />
          <attvalue for="2" value="&#10;&#10;Autoregressive large language models (LLMs) trained on general-domain data are vast stores of world knowledge \cite{petroni-etal-2019-language}. They are typically optimized by predicting a token given its preceding context; therefore, tractable inference over this knowledge is limited to sampling conditioned on a prefix. Many useful tasks, such as infilling~\cite{zhu2019text,liu-etal-2019-tigs}, generating text conditioned on length or lexical constraints~\cite{hokamp-liu-2017-lexically, hu-etal-2019-improved}, and finding the most likely sequence continuation, involve intractable inference in LLMs.&#10;&#10;Such tasks are related to the problem of reasoning, which has been framed as one of probabilistic inference \cite{gershman-goodman}. Correspondingly, the linguistic expression of reasoning can be seen as inference over language. For example, we can interpret chain-of-thought reasoning \cite{wei2022chain,kojima2022large}, a paradigm of reasoning in language models, as a problem of intractable posterior inference. Given a question-answer pair $(X, Y)$, we are interested in finding latent chains of thought -- token sequences $Z$ that contribute the most to the conditional likelihood&#10;\begin{equation}&#10; p(Y\mid X) = \sum_{Z}p_{\rm LM}(ZY\mid X) = \sum_{Z}p_{\rm LM}(Y\mid XZ)p_{\rm LM}(Z\mid X),&#10;\end{equation}&#10;where $p_{\rm LM}$ denotes the likelihood assigned to a sequence by a language model and apposition of variables (\eg, $XZY$) denotes the concatenation of the token sequences.&#10;&#10;While past work has relied on prompting and in-context learning to produce $Z$'s that lead to the correct $Y$, treating $Z$ as a hidden variable in a latent variable model (LVM) renders chain-of-thought reasoning a Bayesian inference problem (\ref{fig:figure_one}). For this LVM, the distribution we must sample from is the posterior $p_{\rm LM}(Z\mid X,Y) = \frac{p_{\rm LM}(XZY)}{\sum_{Z'}p_{\rm LM}(XZ'Y)}$. Such sampling is intractable: while it is easy to evaluate $p_{\rm LM}(XZY)$, the conditional distributions needed to sample $Z$ from $p_{\rm LM}(Z\mid X,Y)$ one token at a time are not easy to compute.&#10;&#10;A standard method to sample approximately from intractable posterior distributions is Markov chain Monte Carlo (MCMC), but it is difficult to craft good proposal distributions {for multi-modal distributions over} language data \cite{miao2018cgmh,zhang-etal-2020-language-generation,lew2023sequential}, and inference on a new input may be prohibitively slow. Alternatively, one can turn to reinforcement learning (RL) approaches such as proximal policy optimization~\cite[PPO;][]{schulman2017proximal}, where the language model is treated as a policy to be fine-tuned. However, these do not aim to model the full diversity of the distribution; instead, learned policies settle around a small number of modes. In both cases, issues with this mode collapse are exacerbated when the target distribution is misspecified, leading to the undesirable behavior of overoptimized samplers \cite{gao2022scaling}.&#10;&#10;Amortized probabilistic inference -- that is, training a model to approximate a distribution of interest -- provides a principled, efficient, and potentially scalable way to draw samples from the distribution \cite{beal2003variational}. One way to implement amortized inference for high-dimensional discrete data such as text is using generative flow networks \cite[GFlowNets;][]{bengio2021flow}, which are diversity-seeking reinforcement learning algorithms that train policies to sample objects (such as a token sequence $Z$) with probability proportional to a given reward function, such as the joint $p_{\rm LM}(XZY)$.&#10;&#10;In this work, we present a method that initializes the GFlowNet policy with a pretrained LLM and continues to train it with a reward objective that can be evaluated with the same LLM. The result is a different type of fine-tuning (FT) procedure for text generation that has a number of advantages, including improved sample diversity, data efficiency, and out-of-distribution generalization. GFlowNet fine-tuning makes the language model sample from the target distribution, enabling amortized inference in a number of applications (\ref{fig:figure_one}).&#10;&#10;Leveraging this approach, we empirically demonstrate the possibilities and benefits of learning to sample from intractable distributions over text continuations, latent reasoning chains, and tool use sequences using GFlowNet fine-tuning. Notably, the diversity of samples from the models trained with GFlowNet fine-tuning is beneficial in Bayesian model averaging settings, such as when aggregating answers to questions obtained via multiple reasoning chains. For example, using a pretrained language model with 6B parameters, our method shows an absolute improvement of 10.9\% over supervised fine-tuning on subjectivity classification with only 10 labeled examples (\ref{sec:expt_subj}) and outperforms supervised fine-tuning and PPO by 63\% on integer arithmetic with 50 demonstrations, with notable improvements in out-of-distribution generalization (\S\ref{sec:expt_arithmetic}). Moreover, the benefits of amortized inference allow us to efficiently sample from the fine-tuned model at scale.&#10;Our contributions include:&#10;\begin{enumerate}[left=0pt,nosep,label=(\arabic*)]&#10; \item A general algorithm for amortized sampling from intractable LLM posteriors.&#10; \item A probabilistic approach to fine-tuning LLMs to perform chain-of-thought reasoning.&#10; \item Empirical results on sequence continuation, natural language reasoning, integer arithmetic with tool use, and story infilling.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Language Model Training, Constrained Generation Techniques, Artificial Intelligence, Mathematics, Bayesian Inference Methods, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1901.00158" label="1901.00158">
        <attvalues>
          <attvalue for="0" value="Text Infilling" />
          <attvalue for="1" value="  Recent years have seen remarkable progress of text generation in different&#10;contexts, such as the most common setting of generating text from scratch, and&#10;the emerging paradigm of retrieval-and-rewriting. Text infilling, which fills&#10;missing text portions of a sentence or paragraph, is also of numerous use in&#10;real life, yet is under-explored. Previous work has focused on restricted&#10;settings by either assuming single word per missing portion or limiting to a&#10;single missing portion to the end of the text. This paper studies the general&#10;task of text infilling, where the input text can have an arbitrary number of&#10;portions to be filled, each of which may require an arbitrary unknown number of&#10;tokens. We study various approaches for the task, including a self-attention&#10;model with segment-aware position encoding and bidirectional context modeling.&#10;We create extensive supervised data by masking out text with varying&#10;strategies. Experiments show the self-attention model greatly outperforms&#10;others, creating a strong baseline for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.10752" label="1905.10752">
        <attvalues>
          <attvalue for="0" value="TIGS: An Inference Algorithm for Text Infilling with Gradient Search" />
          <attvalue for="1" value="  Text infilling is defined as a task for filling in the missing part of a&#10;sentence or paragraph, which is suitable for many real-world natural language&#10;generation scenarios. However, given a well-trained sequential generative&#10;model, generating missing symbols conditioned on the context is challenging for&#10;existing greedy approximate inference algorithms. In this paper, we propose an&#10;iterative inference algorithm based on gradient search, which is the first&#10;inference algorithm that can be broadly applied to any neural sequence&#10;generative models for text infilling tasks. We compare the proposed method with&#10;strong baselines on three text infilling tasks with various mask ratios and&#10;different mask strategies. The results show that our proposed method is&#10;effective and efficient for fill-in-the-blank tasks, consistently outperforming&#10;all baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.10996" label="1811.10996">
        <attvalues>
          <attvalue for="0" value="CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling" />
          <attvalue for="1" value="  In real-world applications of natural language generation, there are often&#10;constraints on the target sentences in addition to fluency and naturalness&#10;requirements. Existing language generation techniques are usually based on&#10;recurrent neural networks (RNNs). However, it is non-trivial to impose&#10;constraints on RNNs while maintaining generation quality, since RNNs generate&#10;sentences sequentially (or with beam search) from the first word to the last.&#10;In this paper, we propose CGMH, a novel approach using Metropolis-Hastings&#10;sampling for constrained sentence generation. CGMH allows complicated&#10;constraints such as the occurrence of multiple keywords in the target&#10;sentences, which cannot be handled in traditional RNN-based approaches.&#10;Moreover, CGMH works in the inference stage, and does not require parallel&#10;corpora for training. We evaluate our method on a variety of tasks, including&#10;keywords-to-sentence generation, unsupervised sentence paraphrasing, and&#10;unsupervised sentence error correction. CGMH achieves high performance compared&#10;with previous supervised methods for sentence generation. Our code is released&#10;at https://github.com/NingMiao/CGMH&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.12334" label="2011.12334">
        <attvalues>
          <attvalue for="0" value="Language Generation via Combinatorial Constraint Satisfaction: A Tree&#10;  Search Enhanced Monte-Carlo Approach" />
          <attvalue for="1" value="  Generating natural language under complex constraints is a principled&#10;formulation towards controllable text generation. We present a framework to&#10;allow specification of combinatorial constraints for sentence generation. We&#10;propose TSMH, an efficient method to generate high likelihood sentences with&#10;respect to a pre-trained language model while satisfying the constraints. Our&#10;approach is highly flexible, requires no task-specific training, and leverages&#10;efficient constraint satisfaction solving techniques. To better handle the&#10;combinatorial constraints, a tree search algorithm is embedded into the&#10;proposal process of the Markov chain Monte Carlo (MCMC) to explore candidates&#10;that satisfy more constraints. Compared to existing MCMC approaches, our&#10;sampling approach has a better mixing performance. Experiments show that TSMH&#10;achieves consistent and significant improvement on multiple language generation&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.04399" label="2106.04399">
        <attvalues>
          <attvalue for="0" value="Flow Network based Generative Models for Non-Iterative Diverse Candidate&#10;  Generation" />
          <attvalue for="1" value="  This paper is about the problem of learning a stochastic policy for&#10;generating an object (like a molecular graph) from a sequence of actions, such&#10;that the probability of generating an object is proportional to a given&#10;positive reward for that object. Whereas standard return maximization tends to&#10;converge to a single return-maximizing sequence, there are cases where we would&#10;like to sample a diverse set of high-return solutions. These arise, for&#10;example, in black-box function optimization when few rounds are possible, each&#10;with large batches of queries, where the batches should be diverse, e.g., in&#10;the design of new molecules. One can also see this as a problem of&#10;approximately converting an energy function to a generative distribution. While&#10;MCMC methods can achieve that, they are expensive and generally only perform&#10;local exploration. Instead, training a generative policy amortizes the cost of&#10;search during training and yields to fast generation. Using insights from&#10;Temporal Difference learning, we propose GFlowNet, based on a view of the&#10;generative process as a flow network, making it possible to handle the tricky&#10;case where different trajectories can yield the same final state, e.g., there&#10;are many ways to sequentially add atoms to generate some molecular graph. We&#10;cast the set of trajectories as a flow and convert the flow consistency&#10;equations into a learning objective, akin to the casting of the Bellman&#10;equations into Temporal Difference methods. We prove that any global minimum of&#10;the proposed objectives yields a policy which samples from the desired&#10;distribution, and demonstrate the improved performance and diversity of&#10;GFlowNet on a simple domain where there are many modes to the reward function,&#10;and on a molecule synthesis task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.00377" label="2006.00377">
        <attvalues>
          <attvalue for="0" value="Linguistic Features for Readability Assessment" />
          <attvalue for="1" value="  Readability assessment aims to automatically classify text by the level&#10;appropriate for learning readers. Traditional approaches to this task utilize a&#10;variety of linguistically motivated features paired with simple machine&#10;learning models. More recent methods have improved performance by discarding&#10;these features and utilizing deep learning models. However, it is unknown&#10;whether augmenting deep learning models with linguistically motivated features&#10;would improve performance further. This paper combines these two approaches&#10;with the goal of improving overall model performance and addressing this&#10;question. Evaluating on two large readability corpora, we find that, given&#10;sufficient training data, augmenting deep learning models with linguistically&#10;motivated features does not improve state-of-the-art performance. Our results&#10;provide preliminary evidence for the hypothesis that the state-of-the-art deep&#10;learning models represent linguistic features of the text related to&#10;readability. Future research on the nature of representations formed in these&#10;models can shed light on the learned features and their relations to&#10;linguistically motivated ones hypothesized in traditional approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Work on readability assessment has involved progress on three core components: corpora, features, and models. While early work utilized small corpora, limited feature sets, and simple models, modern research has experimented with a broad set of features and deep learning techniques.&#10;&#10;Labeled corpora can be difficult to assemble given the time and qualifications needed to assign a text a readability level. The size of readability corpora expanded significantly with the introduction of the WeeklyReader corpus by \cite{schwarmReadingLevelAssessment2005}. Composed of articles from an educational magazine, the WeeklyReader corpus contains roughly 2,400 articles. The WeeklyReader corpus was then built upon by \cite{vajjalaImprovingAccuracyReadability2012} by adding data from the BBC Bitesize website to form the WeeBit corpus. This WeeBit corpus is larger, containing roughly 6,000 documents, while also spanning a greater range of readability levels. Within these corpora, topic and readability are highly correlated. Thus, \cite{xiaTextReadabilityAssessment2016} constructed the Newsela corpus in which each article is represented at multiple reading levels thereby diminishing this correlation. &#10;&#10;Early work on readability assessment, such as that of \cite{fleschNewReadabilityYardstick1948}, extracted simple textual features like character count. More recently, \cite{schwarmReadingLevelAssessment2005} analyzed a broader set of features including out-of-vocabulary scores and syntactic features such as average parse tree height. \cite{vajjalaImprovingAccuracyReadability2012} assembled perhaps the broadest class of features. They incorporated measures shown by \cite{luAutomaticAnalysisSyntactic2010} to correlate well with second language acquisition measures, as well as psycholinguistically relevant features from the Celex Lexical database and MRC Psycholinguistic Database \cite{Baayen1995TheCL, wilsonMRCPsycholinguisticDatabase1988}.&#10;&#10;Traditional feature formulas, like the Flesch formula, relied on linear models. Later work progressed to more complex related models like SVMs \cite{schwarmReadingLevelAssessment2005}. Most recently, state-of-art-performance has been achieved on readability assessment with deep neural network incorporating attention mechanisms. These approaches ignore linguistic features entirely and instead feed the raw embeddings of input words, relying on the model itself to extract any relevant features. Specifically, \cite{martincSupervisedUnsupervisedNeural2019} found that a pretrained transformer model achieved state-of-the-art performance on the WeeBit corpus while a hierarchical attention network (HAN) achieved state-of-the-art performance on the Newsela corpus.&#10;&#10;Deep learning approaches generally exclude any specific linguistic features. In general, a ``feature-less&quot; approach is sensible given the hypothesis that, with enough data, training, and model complexity, a model should learn any linguistic features that researchers might attempt to precompute. However, precomputed linguistic features may be useful in data-poor contexts where data acquisition is expensive and error-prone. For this reason, in this paper we attempt to incorporate linguistic features with deep learning methods in order to improve readability assessment.&#10;&#10;" />
          <attvalue for="4" value="Deep Learning Models, Machine Learning, Computer Science, Linguistics, Cognitive Science, Linguistic Feature Analysis, Readability Assessment, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.03262" label="2304.03262">
        <attvalues>
          <attvalue for="0" value="When do you need Chain-of-Thought Prompting for ChatGPT?" />
          <attvalue for="1" value="  Chain-of-Thought (CoT) prompting can effectively elicit complex multi-step&#10;reasoning from Large Language Models~(LLMs). For example, by simply adding CoT&#10;instruction ``Let's think step-by-step'' to each input query of MultiArith&#10;dataset, GPT-3's accuracy can be improved from 17.7\% to 78.7\%. However, it is&#10;not clear whether CoT is still effective on more recent instruction finetuned&#10;(IFT) LLMs such as ChatGPT. Surprisingly, on ChatGPT, CoT is no longer&#10;effective for certain tasks such as arithmetic reasoning while still keeping&#10;effective on other reasoning tasks. Moreover, on the former tasks, ChatGPT&#10;usually achieves the best performance and can generate CoT even without being&#10;instructed to do so. Hence, it is plausible that ChatGPT has already been&#10;trained on these tasks with CoT and thus memorized the instruction so it&#10;implicitly follows such an instruction when applied to the same queries, even&#10;without CoT. Our analysis reflects a potential risk of overfitting/bias toward&#10;instructions introduced in IFT, which becomes more common in training LLMs. In&#10;addition, it indicates possible leakage of the pretraining recipe, e.g., one&#10;can verify whether a dataset and instruction were used in training ChatGPT. Our&#10;experiments report new baseline results of ChatGPT on a variety of reasoning&#10;tasks and shed novel insights into LLM's profiling, instruction memorization,&#10;and pretraining dataset leakage.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Cognitive Science, Language Model Reasoning, Instruction Finetuning Effects, Artificial Intelligence, Large Language Model Analysis, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2212.01757" label="2212.01757">
        <attvalues>
          <attvalue for="0" value="Languages You Know Influence Those You Learn: Impact of Language&#10;  Characteristics on Multi-Lingual Text-to-Text Transfer" />
          <attvalue for="1" value="  Multi-lingual language models (LM), such as mBERT, XLM-R, mT5, mBART, have&#10;been remarkably successful in enabling natural language tasks in low-resource&#10;languages through cross-lingual transfer from high-resource ones. In this work,&#10;we try to better understand how such models, specifically mT5, transfer *any*&#10;linguistic and semantic knowledge across languages, even though no explicit&#10;cross-lingual signals are provided during pre-training. Rather, only&#10;unannotated texts from each language are presented to the model separately and&#10;independently of one another, and the model appears to implicitly learn&#10;cross-lingual connections. This raises several questions that motivate our&#10;study, such as: Are the cross-lingual connections between every language pair&#10;equally strong? What properties of source and target language impact the&#10;strength of cross-lingual transfer? Can we quantify the impact of those&#10;properties on the cross-lingual transfer?&#10;  In our investigation, we analyze a pre-trained mT5 to discover the attributes&#10;of cross-lingual connections learned by the model. Through a statistical&#10;interpretation framework over 90 language pairs across three tasks, we show&#10;that transfer performance can be modeled by a few linguistic and data-derived&#10;features. These observations enable us to interpret cross-lingual understanding&#10;of the mT5 model. Through these observations, one can favorably choose the best&#10;source language for a task, and can anticipate its training data demands. A key&#10;finding of this work is that similarity of syntax, morphology and phonology are&#10;good predictors of cross-lingual transfer, significantly more than just the&#10;lexical similarity of languages. For a given language, we are able to predict&#10;zero-shot performance, that increases on a logarithmic scale with the number of&#10;few-shot target language data points.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multi-lingual language models (LM), such as mBERT \cite{devlin-etal-2019-bert}, XLM-R \cite{conneau2019unsupervised}, mT5 \cite{xue2020mt5}, mBART \cite{liu-etal-2020-multilingual-denoising}, have been remarkably successful in enabling natural language tasks in low-resource languages through cross-lingual transfer from high-resource languages.&#10;LM based pre-training and fine-tuning, combined with transfer learning resulted in state-of-art performance across various tasks \cite{pires-etal-2019-multilingual,libovicky2019language}.&#10;&#10;In a typical cross-lingual transfer scenario, a single multi-lingual language model is pre-trained with large quantities of (unannotated) text from multiple languages.&#10;It is then fine-tuned for a given natural language understanding task using human-labeled examples of that task in a {source} language.&#10;Cross-lingual transfer occurs when this fine-tuned model can effectively perform this task on another language -- the {target} language -- without human-labeled data (called {zero-shot transfer}), or with only a few human-labeled examples in the target language (called {few-shot transfer}).&#10;&#10;Recently, a line of work by \cite{hestness2017deep,kaplan2020scaling} has analyzed the scaling effects of parameters, corpus size and number of training steps on pre-training loss in language models \cite{devlin-etal-2019-bert}. &#10;\cite{hutter2021learning} extended this analysis to the out-of-distribution transfer setting and showed that the effective amount of data transferred from the training distribution to the target distribution follows a power law of the number of parameters and the amount of training data. &#10;Similarly, \cite{xia-etal-2020-predicting} showed that the performance of a wide range of language tasks could be predicted with relatively good accuracy.&#10;Their approach consists of parameterizing the experimental setting with both data-driven features, and linguistic features fed to a gradient-boosting model \cite{friedman2001greedy} to predict downstream performance.&#10;&#10;Probing studies from \cite{pires-etal-2019-multilingual} and \cite{xue2020mt5} suggest that large multi-lingual language models exhibit zero-shot transfer ability and can deliver state-of-art performance for low-resource languages.&#10;\cite{K2020Cross-Lingual} have suggested that ``structural similarity'' between the source and target languages is one of the most important factors regardless of the lexical overlap or word frequency similarity. &#10;Along similar lines, \cite{lauscher-etal-2020-zero} introduce a meta-regression framework and use it to predict cross-lingual task performance. \cite{lin-etal-2019-choosing} combined multiple features into a gradient-boosting model to predict zero-shot cross-lingual transfer performance. Finally, \cite{de-vries-etal-2022-make} combined multiple typological features in a single regression model to predict the cross-lingual transfer performance of XLM-R \cite{conneau2019unsupervised} in POS tagging. &#10;Our work extends their findings by presenting an interpretable statistical framework to explain zero-shot and few-shot cross-lingual transfer. We do it across three tasks and 90 language pairs. While language similarity is a critical factor in effective cross-lingual transfer, we show that corpora size or language model performance in pre-trained models plays an equally important role. &#10;&#10;In our work, we try to better understand how multi-lingual pre-trained language models, such as mT5 \cite{xue2020mt5}, transfer {any} linguistic and semantic knowledge across languages.&#10;There are no explicit cross-lingual signals provided to the model during pre-training.&#10;Rather, unannotated texts from each language are presented to the model separately and independently of one another, and the model appears to implicitly learn cross-lingual connections.&#10;The fact that this model exhibits cross-lingual transfer may suggest that it is somehow aligning its learned ``semantic spaces'' of different languages \cite{libovicky2019language,muller2021align}.&#10;But, {are the cross-lingual connections between every language pair equally strong?}&#10;{What properties of the source and the target language impact cross-lingual transfer performance?}&#10;{Can we quantify the impact of those properties on the cross-lingual transfer?}&#10;These are some of the key questions regarding effectiveness of cross-lingual transfer that naturally follow, and are the central theme of this work.&#10;&#10;We posit that transfer between some languages is more dominant than others, based on the premise that not all language pairs are born equal \cite{wu-dredze-2020-languages}.&#10;As highlighted by \cite{ruder2020beyondenglish}, designing an NLP system by mirroring what has been done on some high-resource languages (e.g., English) can lead to poor assumptions (e.g., ignoring the rich morphological connections between certain languages).&#10;This approach is sub-optimal as it ignores specific properties of, say, Swahili or Arabic that could potentially see larger transfer benefits from ``non-traditional'' source languages.&#10;&#10;Our contributions are three-fold:&#10;&#09;First, we establish an interpretable statistical framework to enable introspection into cross-lingual transfer in mT5.&#10;&#09;Next, using the above framework, we assess the impact of various factors on cross-lingual transfer.&#10;&#09;Finally, we derive linear connections between language similarity features, language model performance and number of target training samples for transfer learning.&#10;A key finding of this work is that syntactic similarity, morphological similarity and phonological similarity are good predictors of cross-lingual transfer, significantly more so than just lexical similarity of language pairs.&#10;For a given {\{source, target\}} language pair, we have the ability to predict zero-shot performance on the target language (for a given task), that is shown to increase on a logarithmic scale with the number of few-shot target language data points.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Multilingual Natural Language Processing, Linguistics, Cross-Lingual Transfer, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2102.04074" label="2102.04074">
        <attvalues>
          <attvalue for="0" value="Learning Curve Theory" />
          <attvalue for="1" value="  Recently a number of empirical &quot;universal&quot; scaling law papers have been&#10;published, most notably by OpenAI. `Scaling laws' refers to power-law decreases&#10;of training or test error w.r.t. more data, larger neural networks, and/or more&#10;compute. In this work we focus on scaling w.r.t. data size $n$. Theoretical&#10;understanding of this phenomenon is largely lacking, except in&#10;finite-dimensional models for which error typically decreases with $n^{-1/2}$&#10;or $n^{-1}$, where $n$ is the sample size. We develop and theoretically analyse&#10;the simplest possible (toy) model that can exhibit $n^{-\beta}$ learning curves&#10;for arbitrary power $\beta&gt;0$, and determine whether power laws are universal&#10;or depend on the data distribution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.19572" label="2310.19572">
        <attvalues>
          <attvalue for="0" value="Improving Input-label Mapping with Demonstration Replay for In-context&#10;  Learning" />
          <attvalue for="1" value="  In-context learning (ICL) is an emerging capability of large autoregressive&#10;language models where a few input-label demonstrations are appended to the&#10;input to enhance the model's understanding of downstream NLP tasks, without&#10;directly adjusting the model parameters. The effectiveness of ICL can be&#10;attributed to the strong language modeling capabilities of large language&#10;models (LLMs), which enable them to learn the mapping between input and labels&#10;based on in-context demonstrations. Despite achieving promising results, the&#10;causal nature of language modeling in ICL restricts the attention to be&#10;backward only, i.e., a token only attends to its previous tokens, failing to&#10;capture the full input-label information and limiting the model's performance.&#10;In this paper, we propose a novel ICL method called Repeated Demonstration with&#10;Sliding Causal Attention, (RdSca). Specifically, we duplicate later&#10;demonstrations and concatenate them to the front, allowing the model to&#10;`observe' the later information even under the causal restriction. Besides, we&#10;introduce sliding causal attention, which customizes causal attention to avoid&#10;information leakage. Experimental results show that our method significantly&#10;improves the input-label mapping in ICL demonstrations. We also conduct an&#10;in-depth analysis of how to customize the causal attention without training,&#10;which has been an unexplored area in previous research.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have become the backbone of various natural language processing tasks in different fields. One of the most remarkable abilities of LLMs is in-context learning (ICL)~\cite{brown2020language}. By providing a few demonstrations and instructions into the input, along with the input queries, LLMs can perform well in new tasks without requiring fine-tuning. The secret of ICL is to formulate the input as the natural language generation task, then the LLM can be activated to prompt knowledge learned in the pre-training stage. &#10;&#10;Despite the promising results demonstrated by existing ICL models~\cite{DBLP:journals/corr/abs-2212-04037,DBLP:conf/nips/Wei0SBIXCLZ22}, their causal nature in language modeling restricts each token's attention solely to its preceding tokens. As a result, these models fail to capture the complete input-label information, thereby limiting their overall performance. Specifically, the pre-training objective of the current autoregressive LLMs focuses on predicting future tokens based on past ones~\cite{radford2018improving}, implemented with causal attention. While this approach works well for modeling regular sequences, it becomes less effective when applied to ICL tasks. The limitation of causal attention restricts ICL demonstrations to having only left context, which hampers the model's ability to fully comprehend and exploit the input-label relationship.&#10;&#10;Unlike tokens in a sentence that possess sequential dependencies, there is no inherent sequential relationship between the demonstrations in the ICL input. Therefore, it is desirable for these demonstrations to interact with one another comprehensively, rather than relying solely on later demonstrations attending to earlier ones, while the reverse is not possible. Intuitively, if we can enable each demonstration to attend to all the others, we can potentially obtain a more sophisticated context for the ICL query.&#10;However, achieving this on an LLM pre-trained with the objective of causal language modeling is not straightforward. Simply removing the causal restriction and allowing the model to have access to the right context during inference is not feasible, as it would result in a significant disparity between training and inference conditions.&#10;&#10;In this work, we focus on capturing full input-label mapping information from demonstrations. To achieve this target, we propose two techniques. The first is Repeated Demonstration, where we replicate later demonstrations and concatenate them to the front. This allows the model to `observe' the later information even under the causal restriction. &#10;For example, if we consider four demonstrations represented by $d_1d_2d_3d_4$, the input sequence after replication becomes $d_2'd_3'd_4'd_1d_2d_3d_4$.&#10;However, simply duplicating demonstrations brings about a new problem: we do not want to attend a demonstration twice, as this may cause the model to take shortcuts by learning to repeat the answer of its first encounter, rather than learning the input-label mapping. To address this, we propose the second technique, the Sliding Causal Attention, which customizes the original causal attention by restricting the attention window so that each demonstration can only attend to all other demonstrations once. In the case of four demonstrations, attention windows are $d_2'd_3'd_4'd_1$, $d_3'd_4'd_1d_2$, $d_4'd_1d_2d_3$, and $d_1d_2d_3d_4$, respectively. Through experiments, we demonstrate that our proposed method (Repeated Demonstrations with Sliding Causal Attention, RdSca) significantly enhances the ability to learn input-label mapping from ICL demonstrations. &#10;&#10;Our proposed sliding casual attention is the first attempt that customizes the causal attention in the inference stage without further training. We investigate a number of different designs for customizing causal attention and reach some principles for the success of ICL. For example, we find that the first &lt;SOS&gt; token plays an essential role. It should always be available to attend to no matter where the attention window slides. Besides, the size of the attention window determines the richness of the semantic context, thus it affects the performance greatly.&#10;Our contributions are summarized as:&#10;\begin{itemize}&#10; \item To the best of our knowledge, we are the first to identify the limitation of causal language modeling in the ICL and to introduce a novel approach for enabling effective interactions between demonstrations.&#10; \item We validate the feasibility of customizing causal attention during the inference stage without further training and conduct further analysis on causal attention customization. We believe this idea has great potential and sheds new light on optimizing ICL and other large model inference scenarios.&#10; \item We conduct experiments on several text classification datasets to evaluate the effectiveness of our proposed method. The experimental results clearly demonstrate that our approach significantly enhances the input-label mapping in ICL demonstrations.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Attention Mechanism Optimization, Computer Science, Linguistics, Cognitive Science, Language Model Enhancement, Artificial Intelligence, In-Context Learning Methods" />
        </attvalues>
      </node>
      <node id="2305.03513" label="2305.03513">
        <attvalues>
          <attvalue for="0" value="ChatGraph: Interpretable Text Classification by Converting ChatGPT&#10;  Knowledge to Graphs" />
          <attvalue for="1" value="  ChatGPT, as a recently launched large language model (LLM), has shown&#10;superior performance in various natural language processing (NLP) tasks.&#10;However, two major limitations hinder its potential applications: (1) the&#10;inflexibility of finetuning on downstream tasks and (2) the lack of&#10;interpretability in the decision-making process. To tackle these limitations,&#10;we propose a novel framework that leverages the power of ChatGPT for specific&#10;tasks, such as text classification, while improving its interpretability. The&#10;proposed framework conducts a knowledge graph extraction task to extract&#10;refined and structural knowledge from the raw data using ChatGPT. The rich&#10;knowledge is then converted into a graph, which is further used to train an&#10;interpretable linear classifier to make predictions. To evaluate the&#10;effectiveness of our proposed method, we conduct experiments on four datasets.&#10;The result shows that our method can significantly improve the performance&#10;compared to directly utilizing ChatGPT for text classification tasks. And our&#10;method provides a more transparent decision-making process compared with&#10;previous text classification methods.&#10;" />
          <attvalue for="2" value="&#10;&#10;Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}. &#10;In a range of NLP tasks, including question answering, dialogue, summarization, named entity recognition, and sentiment analysis, ChatGPT has demonstrated impressive performance, outperforming many models even in zero-shot settings~\cite{nov2023putting,liu2023deid,shen2023hugginggpt,qin2023chatgpt,peng2023towards}. &#10;&#10;However, despite the superior performance and generation ability, ChatGPT as a black-box model has two major limitations that hinder its potential applications. First, since model parameters are inaccessible, the ChatGPT model cannot be flexibly finetuned on specific datasets to adapt to certain tasks. Therefore, effective as it is, ChatGPT does not consistently outperform other models in every NLP task. For example, as illustrated in Figure~\ref{intro_compare}, TextGCN~\cite{yao2019graph} achieves significantly better performance than ChatGPT when used directly for text classification tasks. Second, similar to the traditional deep models, ChatGPT also suffers from the lack of transparency in the decision-making process, making it uninterpretable. Moreover, since ChatGPT is not open-sourced, existing interpretation methods such as attention scores cannot be applied. &#10;&#10;In this study, we tackle these two limitations in the text classification scenario. Specifically, to extend the potential of ChatGPT, it is necessary to consider two key questions. &#10;i) How to effectively utilize ChatGPT in scenarios where a substantial amount of labeled data is available, making it challenging to fully harness the potential of in-context learning~\cite{min2021recent}?&#10;ii) How to enhance interpretability in the decision-making process? In this work, instead of developing a customized interpretation algorithm, we propose to tackle the problem from a data-centric perspective~\cite{zha2023data}. Specifically, given text data as input, rather than directly applying ChatGPT to classify text data, we first conduct a knowledge graph extraction task to extract refined and structural knowledge from the raw data using ChatGPT. The rich knowledge is then converted and distilled into a graph, which is further used to train an interpretable linear classifier to make predictions. Unlike traditional graph extraction methods, which mainly rely on heuristics~\cite{yao2019graph} or graph refinement~\cite{zhao2021data}, our proposed method does not require further refinement of the obtained graph, which reduces computational costs. Furthermore, our approaches provide a more transparent presentation of the relationships within context, and enable further investigation of the interpretation, thereby addressing the black box issue commonly associated with complex models. Additionally, our proposed framework is flexible and can be easily extended by integrating external knowledge to further enhance the training process, e.g., including TF-IDF weight~\cite{ramos2003using}.&#10;&#10;Our contributions can be summarized from three perspectives. 1) We propose a novel framework that converts the learned knowledge from ChatGPT into a structural form, i.e., graphs, which enables more effective training of the knowledge with label information for text classification tasks. 2) The decision-making in our text classification framework is inherently interpretable, which overcomes the deficiency of black-box models in interpretability. 3) Our proposed method significantly improves the performance of ChatGPT over directly utilizing it for text classification tasks, which demonstrates the effectiveness and superiority of our approach. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Mathematics, Information Science, Natural Language Processing, Interpretability Techniques" />
        </attvalues>
      </node>
      <node id="1602.02373" label="1602.02373">
        <attvalues>
          <attvalue for="0" value="Supervised and Semi-Supervised Text Categorization using LSTM for Region&#10;  Embeddings" />
          <attvalue for="1" value="  One-hot CNN (convolutional neural network) has been shown to be effective for&#10;text categorization (Johnson &amp; Zhang, 2015). We view it as a special case of a&#10;general framework which jointly trains a linear model with a non-linear feature&#10;generator consisting of `text region embedding + pooling'. Under this&#10;framework, we explore a more sophisticated region embedding method using Long&#10;Short-Term Memory (LSTM). LSTM can embed text regions of variable (and possibly&#10;large) sizes, whereas the region size needs to be fixed in a CNN. We seek&#10;effective and efficient use of LSTM for this purpose in the supervised and&#10;semi-supervised settings. The best results were obtained by combining region&#10;embeddings in the form of LSTM and convolution layers trained on unlabeled&#10;data. The results indicate that on this task, embeddings of text regions, which&#10;can convey complex concepts, are more useful than embeddings of single words in&#10;isolation. We report performances exceeding the previous best results on four&#10;benchmark datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1503.00075" label="1503.00075">
        <attvalues>
          <attvalue for="0" value="Improved Semantic Representations From Tree-Structured Long Short-Term&#10;  Memory Networks" />
          <attvalue for="1" value="  Because of their superior ability to preserve sequence information over time,&#10;Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with&#10;a more complex computational unit, have obtained strong results on a variety of&#10;sequence modeling tasks. The only underlying LSTM structure that has been&#10;explored so far is a linear chain. However, natural language exhibits syntactic&#10;properties that would naturally combine words to phrases. We introduce the&#10;Tree-LSTM, a generalization of LSTMs to tree-structured network topologies.&#10;Tree-LSTMs outperform all existing systems and strong LSTM baselines on two&#10;tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task&#10;1) and sentiment classification (Stanford Sentiment Treebank).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.10035" label="2301.10035">
        <attvalues>
          <attvalue for="0" value="Putting ChatGPT's Medical Advice to the (Turing) Test" />
          <attvalue for="1" value="  Objective: Assess the feasibility of using ChatGPT or a similar AI-based&#10;chatbot for patient-provider communication. Participants: A US representative&#10;sample of 430 study participants aged 18 and above. 53.2% of respondents&#10;analyzed were women; their average age was 47.1. Exposure: Ten representative&#10;non-administrative patient-provider interactions were extracted from the EHR.&#10;Patients' questions were placed in ChatGPT with a request for the chatbot to&#10;respond using approximately the same word count as the human provider's&#10;response. In the survey, each patient's question was followed by a provider- or&#10;ChatGPT-generated response. Participants were informed that five responses were&#10;provider-generated and five were chatbot-generated. Participants were asked,&#10;and incentivized financially, to correctly identify the response source.&#10;Participants were also asked about their trust in chatbots' functions in&#10;patient-provider communication, using a Likert scale of 1-5. Results: The&#10;correct classification of responses ranged between 49.0% to 85.7% for different&#10;questions. On average, chatbot responses were correctly identified 65.5% of the&#10;time, and provider responses were correctly distinguished 65.1% of the time. On&#10;average, responses toward patients' trust in chatbots' functions were weakly&#10;positive (mean Likert score: 3.4), with lower trust as the health-related&#10;complexity of the task in questions increased. Conclusions: ChatGPT responses&#10;to patient questions were weakly distinguishable from provider responses.&#10;Laypeople appear to trust the use of chatbots to answer lower risk health&#10;questions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.05679" label="1809.05679">
        <attvalues>
          <attvalue for="0" value="Graph Convolutional Networks for Text Classification" />
          <attvalue for="1" value="  Text classification is an important and classical problem in natural language&#10;processing. There have been a number of studies that applied convolutional&#10;neural networks (convolution on regular grid, e.g., sequence) to&#10;classification. However, only a limited number of studies have explored the&#10;more flexible graph convolutional neural networks (convolution on non-grid,&#10;e.g., arbitrary graph) for the task. In this work, we propose to use graph&#10;convolutional networks for text classification. We build a single text graph&#10;for a corpus based on word co-occurrence and document word relations, then&#10;learn a Text Graph Convolutional Network (Text GCN) for the corpus. Our Text&#10;GCN is initialized with one-hot representation for word and document, it then&#10;jointly learns the embeddings for both words and documents, as supervised by&#10;the known class labels for documents. Our experimental results on multiple&#10;benchmark datasets demonstrate that a vanilla Text GCN without any external&#10;word embeddings or knowledge outperforms state-of-the-art methods for text&#10;classification. On the other hand, Text GCN also learns predictive word and&#10;document embeddings. In addition, experimental results show that the&#10;improvement of Text GCN over state-of-the-art comparison methods become more&#10;prominent as we lower the percentage of training data, suggesting the&#10;robustness of Text GCN to less training data in text classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.04819" label="2301.04819">
        <attvalues>
          <attvalue for="0" value="Data-centric AI: Perspectives and Challenges" />
          <attvalue for="1" value="  The role of data in building AI systems has recently been significantly&#10;magnified by the emerging concept of data-centric AI (DCAI), which advocates a&#10;fundamental shift from model advancements to ensuring data quality and&#10;reliability. Although our community has continuously invested efforts into&#10;enhancing data in different aspects, they are often isolated initiatives on&#10;specific tasks. To facilitate the collective initiative in our community and&#10;push forward DCAI, we draw a big picture and bring together three general&#10;missions: training data development, inference data development, and data&#10;maintenance. We provide a top-level discussion on representative DCAI tasks and&#10;share perspectives. Finally, we list open challenges. More resources are&#10;summarized at https://github.com/daochenzha/data-centric-AI&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.06830" label="2006.06830">
        <attvalues>
          <attvalue for="0" value="Data Augmentation for Graph Neural Networks" />
          <attvalue for="1" value="  Data augmentation has been widely used to improve generalizability of machine&#10;learning models. However, comparatively little work studies data augmentation&#10;for graphs. This is largely due to the complex, non-Euclidean structure of&#10;graphs, which limits possible manipulation operations. Augmentation operations&#10;commonly used in vision and language have no analogs for graphs. Our work&#10;studies graph data augmentation for graph neural networks (GNNs) in the context&#10;of improving semi-supervised node-classification. We discuss practical and&#10;theoretical motivations, considerations and strategies for graph data&#10;augmentation. Our work shows that neural edge predictors can effectively encode&#10;class-homophilic structure to promote intra-class edges and demote inter-class&#10;edges in given graph structure, and our main contribution introduces the GAug&#10;graph data augmentation framework, which leverages these insights to improve&#10;performance in GNN-based node classification via edge prediction. Extensive&#10;experiments on multiple benchmarks show that augmentation via GAug improves&#10;performance across GNN architectures and datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.09828" label="2010.09828">
        <attvalues>
          <attvalue for="0" value="Cross-Lingual Transfer in Zero-Shot Cross-Language Entity Linking" />
          <attvalue for="1" value="  Cross-language entity linking grounds mentions in multiple languages to a&#10;single-language knowledge base. We propose a neural ranking architecture for&#10;this task that uses multilingual BERT representations of the mention and the&#10;context in a neural network. We find that the multilingual ability of BERT&#10;leads to robust performance in monolingual and multilingual settings.&#10;Furthermore, we explore zero-shot language transfer and find surprisingly&#10;robust performance. We investigate the zero-shot degradation and find that it&#10;can be partially mitigated by a proposed auxiliary training objective, but that&#10;the remaining error can best be attributed to domain shift rather than language&#10;transfer.&#10;" />
          <attvalue for="2" value="&#10;&#10;Entity linking grounds named entities mentioned in text, such as Chancellor, to a reference knowledge base (KB) or ontology entry, such as Angela Merkel. Historically, entity linking work focused on English documents and knowledge bases, but subsequent work expanded the task to consider multiple languages~\cite{McNamee2011}. In cross-language entity linking, entities in a set of multilingual documents is linked to a KB in a single language.&#10;The TAC KBP shared task~\cite{Ji2015}, for example, links mentions in Chinese and Spanish documents with an English KB.&#10;Success in building cross-language linking systems can be helpful in tasks such as discovering all documents relevant to an entity, regardless of language.&#10;&#10;Successfully linking a mention across languages requires adapting several common entity linking components to the cross-language setting.&#10;Consider the example in Figure \ref{fig:ex_arch}, which contains the&#10;Spanish mention Oficina de la Presidencia, a reference to the entity President of Mexico in an English KB. To link the mention to the relevant entity we must compare the mention text and its surrounding textual context in Spanish to the English entity name and entity description, as well as compare the mention and entity type. Previous work has focused on transliteration or translation approaches for name and context~\cite{McNamee2011, pan2015unsupervised}, or leveraging large amounts of cross-language information~\cite{Tsai2016} and multilingual embeddings~\cite{upadhyay-etal-2018-joint}.&#10;&#10;Since this work emerged, there have been major advances in multilingual NLP~\cite{wu-dredze-2019-beto, pires-etal-2019-multilingual}. Mainstream approaches to multilingual learning now use multilingual encoders, trained on raw text from multiple languages~\cite{devlin-etal-2019-bert}. These models, such as multilingual BERT or XMLR~\cite{conneau2019unsupervised}, have achieved impressive results on a range of multilingual NLP tasks, including part of speech tagging~\cite{tsai-etal-2019-small}, parsing~\cite{wang-etal-2019-cross, kondratyuk-straka-2019-75}, and semantic similarity~\cite{lo-simard-2019-fully,reimers-gurevych-2019-sentence}.&#10;&#10;We propose to leverage text representations with &#10;multilingual BERT~\cite{devlin-etal-2019-bert} for cross-language entity linking to handle&#10;the mention text, entity name, mention context and entity description. We use a neural ranking objective and a deep learning model to combine these representations, along with a &#10;one-hot embedding for the entity and mention type,&#10;to produce a cross-language linker. We use this ranking architecture to highlight the ability of mBERT to perform on this task without a more complex architecture.&#10;Although previous work tends to use multilingual encoders for one language at a time, \eg train a Spanish NER system with mBERT, we ask: can our model effectively link entities {across} languages? &#10;We find that, somewhat surprisingly, &#10;our approach does exceedingly well;&#10;scores are comparable to previously reported best results that are trained on data not available to our model (they have access to non-English names).&#10;Next, we consider a multilingual setting, in which a single system is simultaneously trained to link mentions in multiple languages to an English KB. &#10;Previous work~\cite{upadhyay-etal-2018-joint} has shown that multilingual models can perform robustly on cross-language entity linking. Again, we find that, surprisingly, a model trained on multiple languages at once does about as well, or in some cases better, than the same model trained separately on every language.&#10;&#10;These encouraging results lead us to explore the challenging task of zero-shot training, in which we train a model to link single language documents (\eg English) to an English KB, but apply it to unseen languages (\eg Chinese) documents. &#10;While the resulting model certainly does worse on a language that is unobserved, the reduction in performance is remarkably small.&#10;This result leads us to ask: 1) Why do zero-shot entity linking models do so well? 2) What information is needed to allow zero-shot models to perform as well as multilingually trained models? &#10;Using a series of ablation experiments we find that correctly comparing the mention text and entity name is the most important component of an entity linking model. Therefore, we propose an auxiliary pre-training objective to improve zero-shot performance. However, we find that this text-focused approach does not improve performance significantly.&#10;Rather, we find that much of the remaining loss comes not from the language transfer, but from mismatches of entities mentioned across the datasets. This suggests that future work on the remaining challenges in zero-shot entity linking should focus on topic adaptation, instead of improvements in cross-lingual representations.&#10;&#10;In summary, this paper uses a simple ranker to explore effective cross-language entity linking with multiple languages. We demonstrate its effectiveness at zero-shot linking, evaluate a pre-training objective to improve zero-shot transfer, and lay out guidelines to inform future research on zero-shot linking.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Multilingual Models, Artificial Intelligence, Natural Language Processing, Entity Linking" />
        </attvalues>
      </node>
      <node id="1911.04128" label="1911.04128">
        <attvalues>
          <attvalue for="0" value="A hybrid text normalization system using multi-head self-attention for&#10;  mandarin" />
          <attvalue for="1" value="  In this paper, we propose a hybrid text normalization system using multi-head&#10;self-attention. The system combines the advantages of a rule-based model and a&#10;neural model for text preprocessing tasks. Previous studies in Mandarin text&#10;normalization usually use a set of hand-written rules, which are hard to&#10;improve on general cases. The idea of our proposed system is motivated by the&#10;neural models from recent studies and has a better performance on our internal&#10;news corpus. This paper also includes different attempts to deal with&#10;imbalanced pattern distribution of the dataset. Overall, the performance of the&#10;system is improved by over 1.5% on sentence-level and it has a potential to&#10;improve further.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text Normalization (TN) is a process to transform non-standard words (NSW) into spoken-form words (SFW) for disambiguation. In Text-To-Speech (TTS), text normalization is an essential procedure to normalize unreadable numbers, symbols or characters, such as transforming ``\$20'' to ``twenty dollars'' and ``@'' to ``at'', into words that can be used in speech synthesis. The surrounding context is the determinant for ambiguous cases in TN. For example, the context will decide whether to read ``2019'' as year or a number, and whether to read ``10:30'' as time or the score of a game. In Mandarin, some cases depend on language habit instead of rules- ``2'' can either be read as ``\textipa{\`{e}}r'' or ``li\textipa{\v{a}}ng'' and ``1'' as ``y\textipa{\={\i}}'' or ``y\textipa{\={a}}o''.&#10;&#10;Currently, based on the traditional taxonomy approach for NSW\cite{R11}, the Mandarin TN tasks are generally resolved by rule-based systems which use keywords and regular expressions to determine the SFW of ambiguous words\cite{R1,R3}. These systems typically classify NSW into different pattern groups, such as abbreviations, numbers, etc., and then into sub-groups, such as phone number, year, etc., which has corresponding NSW-SFW transformations. Zhou\cite{R5} and Jia\cite{R4} proposed systems which use maximum entropy (ME) to further disambiguate the NSW with multiple pattern matches. For the NSW given the context constraints, the highest probability corresponds to the highest entropy. Liou\cite{R2} proposed a system of data-driven models which combines a rule-based and a keyword-based TN module. The second module classifies preceding and following words around the keywords and then trains a CRF model to predict the NSW patterns based on the classification results. There are some other hybrid systems\cite{R13,R12} which use NLP models and rules separately to help normalize hard cases in TN.&#10;&#10;For recent NLP studies, sequence-to-sequence (seq2seq) models have achieved impressive progress in TN tasks in English and Russian\cite{R6,R7}. Seq2seq models typically encode sequences into a state vector, which is decoded into an output vector from its learnt vector representation and then to a sequence. Different seq2seq models with bi-LSTM, bi-GRU with attention are proposed in \cite{R7,R8}. Zhang and Sproat proposed a contextual seq2seq model, which uses a sliding-window and RNN with attention\cite{R6}. In this model, bi-directional GRU is used in both encoder and decoder, and the context words are labeled with ``$\langle$self$\rangle$'', helping the model distinguish the NSW and the context.&#10;&#10;However, seq2seq models have several downsides when directly applied in Mandarin TN tasks. As mentioned in \cite{R6}, the sequence output directly from a seq2seq model can lead to unrecoverable errors. The model sometimes changes the context words which should be kept the same. Our experiments produce similar errors. For example, ``Podnieks, Andrew 2000'' is transformed to ``Podncourt, Andrew Two Thousand'', changing ``Podnieks'' to ``Podncourt''. These errors cannot be detected by the model itself. In \cite{R14}, rules are applied to two specific categories to resolve silly errors, but this method is hard to apply to all cases. Another challenge in Mandarin is the word segmentation since words are not separated by spaces and the segmentation could depend on the context. Besides, some NSW may have more than one SFW in Mandarin, making the seq2seq model hard to train. For example, ``\begin{CJK*}{UTF8}{gbsn}两千零八年\end{CJK*}'' and ``\begin{CJK*}{UTF8}{gbsn}二零零八年\end{CJK*}'' are both acceptable SFW for ``2008\begin{CJK*}{UTF8}{gbsn}年\end{CJK*}''. The motivation of this paper is to combine the advantages of a rule-based model for its flexibility and a neural model to enhance the performance on more general cases. To avoid the problems of seq2seq models, we consider the TN task as a multi-class classification problem with carefully designed patterns for the neural model.&#10;&#10;The contributions of this paper include the following. First, this is the first known TN system for Mandarin which uses a neural model with multi-head self-attention. Second, we propose a hybrid system combining a rule-based model and a neural model. Third, we experiment with different approaches to deal with imbalanced dataset in the TN task.&#10;&#10;The paper is organized as follows. Section \ref{sec:method} introduces the detailed structure of the proposed hybrid system and its training and inference. In Section \ref{sec:experiments}, the performance of different system configurations is evaluated on different datasets. And the conclusion is given in Section \ref{sec:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Neural Network Models, Text Normalization, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2112.09866" label="2112.09866">
        <attvalues>
          <attvalue for="0" value="Cascading Adaptors to Leverage English Data to Improve Performance of&#10;  Question Answering for Low-Resource Languages" />
          <attvalue for="1" value="  Transformer based architectures have shown notable results on many down&#10;streaming tasks including question answering. The availability of data, on the&#10;other hand, impedes obtaining legitimate performance for low-resource&#10;languages. In this paper, we investigate the applicability of pre-trained&#10;multilingual models to improve the performance of question answering in&#10;low-resource languages. We tested four combinations of language and task&#10;adapters using multilingual transformer architectures on seven languages&#10;similar to MLQA dataset. Additionally, we have also proposed zero-shot transfer&#10;learning of low-resource question answering using language and task adapters.&#10;We observed that stacking the language and the task adapters improves the&#10;multilingual transformer models' performance significantly for low-resource&#10;languages.&#10;" />
          <attvalue for="2" value="&#10;Last few years have seen emergence of transformer based pretrained models like BERT\cite{devlin-etal-2019-bert}, XLNet\cite{NEURIPS2019_dc6a7e65}, T5\cite{JMLR:v21:20-074}, XLM-RoBERTa\cite{conneau-etal-2020-unsupervised} etc. The pretrained models have shown significant improvement in various downstream tasks like question answering, NER, Machine translation and speech recognition when used with word level utilities.\cite{delobelle-etal-2020-robbert,pires-etal-2019-multilingual,pfeiffer-etal-2020-adapterhub,pires-etal-2019-multilingual,pandya2021question,10.1145/3461763,murthy-etal-2019-addressing,inproceedings,baxi-etal-2015-morphological,JMLR:v21:20-074}.&#10;&#10;The emergence of multilingual models: mBERT \cite{devlin-etal-2019-bert} and XLM-RoBERTa\cite{conneau-etal-2020-unsupervised} made it possible to leverage English data to improve the performance of low-resource languages. In this paper, we continue to investigate the effectiveness of multilingual pretrained transformer models in improving the performance of question answering systems in a low-resource setup using the cascading of language and task adapters\cite{pfeiffer-etal-2021-adapterfusion,pfeiffer-etal-2020-adapterhub,bapna-firat-2019-simple}. Our work contributes by evaluating cross-lingual performance in seven languages - Hindi, Arabic, German, Spanish, English, Vietnamese and Simplified Chinese. Our models are evaluated on the combination of XQuAD\cite{artetxe-etal-2020-cross} and MLQA\cite{lewis-etal-2020-mlqa} datasets which are similar to SQuAD \cite{rajpurkar-etal-2016-squad}. &#10;&#10;To this end, our contributions are as follows: &#10;&#10;\begin{itemize}&#10;\item {We have trained multilingual variants of transformers, namely mBert and XLM-RoBERTa with a QA dataset in seven languages. Both the MLQA and XQuAD datasets contain validation and test sets for the above languages but not the training set. To finetune the model we have combined the test set of XQuAD and MLQA datasets and evaluated the model with the MLQA development dataset as the test dataset. By splitting the dataset in this way we can get train and test data with the considerable length for low-resource languages which helped us to conduct various experiments. Table \ref{table:datasetlength} highlights the size of our train and test set for all the above-mentioned languages.}&#10;&#10;\item {We exhaustively analysed the fine-tuned models by evaluating them with the tasks adapter \cite{pfeiffer-etal-2021-adapterfusion,pfeiffer-etal-2020-adapterhub}. We conducted the experiments in two different setups, Houlsby\cite{pmlr-v97-houlsby19a} and Pfeiffer\cite{pfeiffer-etal-2021-adapterfusion,pfeiffer-etal-2020-mad}. These two setups enabled us to compare our language model variants with their multilingual counterparts and understand the different factors that lead to better results on the downstream tasks.}&#10;&#10;\item {We have also attempted a series of two different experiments by stacking language adapters and task adapter in different ways. We first analyze the fine-tuned model by stacking language-specific adapter with the XLM-RoBERTa\textsubscript{base}. After fine-tuning the language-specific adapter we augment the task-specific adapter upon the previously fine-tuned language adapter. We analyze both the experiments separately and conclude that multiple adapters with the transformer-based model perform notably better.}&#10;&#10;\item {Due to limited training, the transfer-learning performance of the transformer is poor on the low-resource languages as well as on the languages unseen during the pretraining\cite{kakwani-etal-2020-indicnlpsuite}. The multi-task adapter (MAD-X) \cite{pfeiffer-etal-2020-mad} outperforms the state-of-the-art models in cross-lingual transfer across a representative set of typologically diverse languages on question answering. To avoid the training of model individually for multiple languages while maintaining the performance, we used cross-lingual transfer by switching heads of language adapter from the source language to the target language.}&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Question Answering, Machine Learning, Computer Science, Linguistics, Low-Resource Languages, Multilingual Models, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2005.00247" label="2005.00247">
        <attvalues>
          <attvalue for="0" value="AdapterFusion: Non-Destructive Task Composition for Transfer Learning" />
          <attvalue for="1" value="  Sequential fine-tuning and multi-task learning are methods aiming to&#10;incorporate knowledge from multiple tasks; however, they suffer from&#10;catastrophic forgetting and difficulties in dataset balancing. To address these&#10;shortcomings, we propose AdapterFusion, a new two stage learning algorithm that&#10;leverages knowledge from multiple tasks. First, in the knowledge extraction&#10;stage we learn task specific parameters called adapters, that encapsulate the&#10;task-specific information. We then combine the adapters in a separate knowledge&#10;composition step. We show that by separating the two stages, i.e., knowledge&#10;extraction and knowledge composition, the classifier can effectively exploit&#10;the representations learned from multiple tasks in a non-destructive manner. We&#10;empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it&#10;effectively combines various types of knowledge at different layers of the&#10;model. We show that our approach outperforms traditional strategies such as&#10;full fine-tuning as well as multi-task learning. Our code and adapters are&#10;available at AdapterHub.ml.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.04689" label="2404.04689">
        <attvalues>
          <attvalue for="0" value="Multicalibration for Confidence Scoring in LLMs" />
          <attvalue for="1" value="  This paper proposes the use of &quot;multicalibration&quot; to yield interpretable and&#10;reliable confidence scores for outputs generated by large language models&#10;(LLMs). Multicalibration asks for calibration not just marginally, but&#10;simultaneously across various intersecting groupings of the data. We show how&#10;to form groupings for prompt/completion pairs that are correlated with the&#10;probability of correctness via two techniques: clustering within an embedding&#10;space, and &quot;self-annotation&quot; - querying the LLM by asking it various yes-or-no&#10;questions about the prompt. We also develop novel variants of multicalibration&#10;algorithms that offer performance improvements by reducing their tendency to&#10;overfit. Through systematic benchmarking across various question answering&#10;datasets and LLMs, we show how our techniques can yield confidence scores that&#10;provide substantial improvements in fine-grained measures of both calibration&#10;and accuracy compared to existing methods.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have revolutionized text generation, with applications ranging from code development \cite{chen2021evaluating} to information retrieval \cite{zhu2023large}. However, alongside their impressive capabilities, LLMs possess a troubling tendency to fabricate information, generating outputs that diverge from factual reality – a phenomenon dubbed ``hallucination'' \cite{huang2023survey}. These hallucinations pose significant challenges to the trustworthiness and ethical deployment of LLMs, demanding the development of robust detection and mitigation strategies.&#10;&#10;In this paper, we leverage recent ``multicalibration'' techniques \cite{hebert2018multicalibration} to produce calibrated probabilities indicating whether a generated response constitutes a hallucination. Unlike conventional calibration methods, multicalibrated probabilities are self-consistent not just marginally (i.e. on average over all examples), but also conditionally on various properties of the instance, which allows them to serve as more refined risk measures. Producing ``risk scores'' for hallucinations can provide an interpretable measure of risk which can be exposed to the user &#10;(e.g. through a coloring scheme, as in Figure \ref{fig:color})&#10;to communicate the risk associated with the generated content. Moreover, when those risk scores are calibrated, they are not only interpretable but ``trustworthy'' in the sense that they can be safely used as if they were true probabilities \cite{noarov2023high}. &#10;&#10;Our approach mirrors the robust assurances offered by conformal prediction, where multicalibration (of quantiles) has been used to give group-conditional guarantees \cite{bastani2022practical,jung2022batch,gibbs2023conformal}. Traditionally multicalibration has been used to give estimates of uncertainty in tabular data settings that hold conditionally on various features that are explicitly present in the data --- often demographic attributes like sex or race. A key challenge in applying these techniques to hallucination detection in LLMs is a lack of such explicit features. An important part of our contribution is generating features that are useful to multicalibrate with respect to --- which we do both through clustering prompt embeddings, and by having the LLM itself annotate prompts with binary features via the answers to yes-or-no questions. &#10;&#10;We note that what in many contexts, what is and is not a ''hallucination`` can be open to interpretation, and does not have sharp boundaries. In this study, we adopt an agnostic stance toward its definition. Specifically, we refrain from stipulating criteria for determining what constitutes &quot;good&quot; or &quot;bad&quot; generated content. Instead, we assume access to a modestly sized calibration dataset that has been annotated with binary labels. For any criterion for what constitutes a ``good'' vs. ``bad'' completion in a given context, such a dataset could be produced by human evaluators. For our work, we assume that this is given and do not take a stance on what the criterion for establishing that a given completion is ``good'' in a given context should be.&#10;&#10;Our contributions are threefold: 1.~We show how to apply multicalibration techniques in the context of hallucination detection in LLMs; a primary challenge here is to obtain reasonable ``groups'' with respect which to multicalibrate, which we do via prompt clustering and via self-annotation of prompts. 2.~We introduce novel variations of multicalibration methods which yield substantial performance enhancements. 3.~We systematically evaluate these techniques across diverse LLMs and question answering datasets, demonstrating their efficacy in calibration and overall performance compared to existing baselines.&#10;&#10;Additional Related Work&#10;Numerous recent surveys focus on hallucinations in LLMs \cite{chang2023survey, huang2023survey, ji2023survey, rawte2023survey, tonmoy2024comprehensive, zhang2023siren, guerreiro2023hallucinations}. The predominant focus of current research lies in binary hallucination detection, specifically the capacity to discern whether generated text exhibits signs of hallucination. Key contributions in this domain include \cite{manakul2023selfcheckgpt, rebedea2023nemo}, which evaluate consistency, similarity, and agreement among alternative generated responses. \cite{kadavath2022language, friel2023chainpoll} directly engage LLMs by posing inquiries about correctness or consistency within a single answer.&#10;&#10;More closely related is a smaller body of literature that explores uncertainty quantification and confidence scoring in this context \cite{xiao2021hallucination, verma2023reducing, varshney2023stitch, kalai2023calibrated, tian2023just, zhao2023automatic, chen2023quantifying, duan2023shifting, lin2023generating, liu2023calibrating} and propose a variety of approaches to reduce hallucination generation ranging from updated beam search methods, fine-tuning, human labelling, and epistemic neural networks. Several recent papers use conformal prediction to derive sets of prompt completions, offering marginal coverage guarantees (e.g.~for 90\% of prompts, at least one completion in the set should be ``good'') \cite{quach2023conformal, kumar2023conformal, deutschmann2023conformal, ren2023robots, zecchin2023forking}. Among these, \cite{kumar2023conformal} is closest to our approach but requires &quot;group-specific&quot; prompting strategies. The remainder focus on improving the LLM's decoding strategy and/or predictive sets, which are less suited to binary classification settings (like hallucination detection), where they are limited to $\{0\}$, $\{1\}$, and $\{0, 1\}$.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Confidence Score Optimization, Language Model Calibration, Multicalibration Techniques, Mathematics, Statistics" />
        </attvalues>
      </node>
      <node id="2308.07107" label="2308.07107">
        <attvalues>
          <attvalue for="0" value="Large Language Models for Information Retrieval: A Survey" />
          <attvalue for="1" value="  As a primary means of information acquisition, information retrieval (IR)&#10;systems, such as search engines, have integrated themselves into our daily&#10;lives. These systems also serve as components of dialogue, question-answering,&#10;and recommender systems. The trajectory of IR has evolved dynamically from its&#10;origins in term-based methods to its integration with advanced neural models.&#10;While the neural models excel at capturing complex contextual signals and&#10;semantic nuances, thereby reshaping the IR landscape, they still face&#10;challenges such as data scarcity, interpretability, and the generation of&#10;contextually plausible yet potentially inaccurate responses. This evolution&#10;requires a combination of both traditional methods (such as term-based sparse&#10;retrieval methods with rapid response) and modern neural architectures (such as&#10;language models with powerful language understanding capacity). Meanwhile, the&#10;emergence of large language models (LLMs), typified by ChatGPT and GPT-4, has&#10;revolutionized natural language processing due to their remarkable language&#10;understanding, generation, generalization, and reasoning abilities.&#10;Consequently, recent research has sought to leverage LLMs to improve IR&#10;systems. Given the rapid evolution of this research trajectory, it is necessary&#10;to consolidate existing methodologies and provide nuanced insights through a&#10;comprehensive overview. In this survey, we delve into the confluence of LLMs&#10;and IR systems, including crucial aspects such as query rewriters, retrievers,&#10;rerankers, and readers. Additionally, we explore promising directions, such as&#10;search agents, within this expanding field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.17651" label="2310.17651">
        <attvalues>
          <attvalue for="0" value="High-Dimensional Prediction for Sequential Decision Making" />
          <attvalue for="1" value="  We study the problem of making predictions of an adversarially chosen&#10;high-dimensional state that are unbiased subject to an arbitrary collection of&#10;conditioning events, with the goal of tailoring these events to downstream&#10;decision makers. We give efficient algorithms for solving this problem, as well&#10;as a number of applications that stem from choosing an appropriate set of&#10;conditioning events.&#10;  For example, we can efficiently make predictions targeted at polynomially&#10;many decision makers, giving each of them optimal swap regret if they&#10;best-respond to our predictions. We generalize this to online combinatorial&#10;optimization, where the decision makers have a very large action space, to give&#10;the first algorithms offering polynomially many decision makers no regret on&#10;polynomially many subsequences that may depend on their actions and the&#10;context. We apply these results to get efficient no-subsequence-regret&#10;algorithms in extensive-form games (EFGs), yielding a new family of regret&#10;guarantees for EFGs that generalizes some existing EFG regret notions, e.g.&#10;regret to informed causal deviations, and is generally incomparable to other&#10;known such notions.&#10;  Next, we develop a novel transparent alternative to conformal prediction for&#10;building valid online adversarial multiclass prediction sets. We produce class&#10;scores that downstream algorithms can use for producing valid-coverage&#10;prediction sets, as if these scores were the true conditional class&#10;probabilities. We show this implies strong conditional validity guarantees&#10;including set-size-conditional and multigroup-fair coverage for polynomially&#10;many downstream prediction sets. Moreover, our class scores can be guaranteed&#10;to have improved $L_2$ loss, cross-entropy loss, and generally any Bregman&#10;loss, compared to any collection of benchmark models, yielding a&#10;high-dimensional real-valued version of omniprediction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.01067" label="2206.01067">
        <attvalues>
          <attvalue for="0" value="Practical Adversarial Multivalid Conformal Prediction" />
          <attvalue for="1" value="  We give a simple, generic conformal prediction method for sequential&#10;prediction that achieves target empirical coverage guarantees against&#10;adversarially chosen data. It is computationally lightweight -- comparable to&#10;split conformal prediction -- but does not require having a held-out validation&#10;set, and so all data can be used for training models from which to derive a&#10;conformal score. It gives stronger than marginal coverage guarantees in two&#10;ways. First, it gives threshold calibrated prediction sets that have correct&#10;empirical coverage even conditional on the threshold used to form the&#10;prediction set from the conformal score. Second, the user can specify an&#10;arbitrary collection of subsets of the feature space -- possibly intersecting&#10;-- and the coverage guarantees also hold conditional on membership in each of&#10;these subsets. We call our algorithm MVP, short for MultiValid Prediction. We&#10;give both theory and an extensive set of empirical evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.15145" label="2209.15145">
        <attvalues>
          <attvalue for="0" value="Batch Multivalid Conformal Prediction" />
          <attvalue for="1" value="  We develop fast distribution-free conformal prediction algorithms for&#10;obtaining multivalid coverage on exchangeable data in the batch setting.&#10;Multivalid coverage guarantees are stronger than marginal coverage guarantees&#10;in two ways: (1) They hold even conditional on group membership -- that is, the&#10;target coverage level $1-\alpha$ holds conditionally on membership in each of&#10;an arbitrary (potentially intersecting) group in a finite collection&#10;$\mathcal{G}$ of regions in the feature space. (2) They hold even conditional&#10;on the value of the threshold used to produce the prediction set on a given&#10;example. In fact multivalid coverage guarantees hold even when conditioning on&#10;group membership and threshold value simultaneously.&#10;  We give two algorithms: both take as input an arbitrary non-conformity score&#10;and an arbitrary collection of possibly intersecting groups $\mathcal{G}$, and&#10;then can equip arbitrary black-box predictors with prediction sets. Our first&#10;algorithm (BatchGCP) is a direct extension of quantile regression, needs to&#10;solve only a single convex minimization problem, and produces an estimator&#10;which has group-conditional guarantees for each group in $\mathcal{G}$. Our&#10;second algorithm (BatchMVP) is iterative, and gives the full guarantees of&#10;multivalid conformal prediction: prediction sets that are valid conditionally&#10;both on group membership and non-conformity threshold. We evaluate the&#10;performance of both of our algorithms in an extensive set of experiments. Code&#10;to replicate all of our experiments can be found at&#10;https://github.com/ProgBelarus/BatchMultivalidConformal&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12616" label="2305.12616">
        <attvalues>
          <attvalue for="0" value="Conformal Prediction With Conditional Guarantees" />
          <attvalue for="1" value="  We consider the problem of constructing distribution-free prediction sets&#10;with finite-sample conditional guarantees. Prior work has shown that it is&#10;impossible to provide exact conditional coverage universally in finite samples.&#10;Thus, most popular methods only guarantee marginal coverage over the covariates&#10;or are restricted to a limited set of conditional targets, e.g. coverage over a&#10;finite set of pre-specified subgroups. This paper bridges this gap by defining&#10;a spectrum of problems that interpolate between marginal and conditional&#10;validity. We motivate these problems by reformulating conditional coverage as&#10;coverage over a class of covariate shifts. When the target class of shifts is&#10;finite-dimensional, we show how to simultaneously obtain exact finite-sample&#10;coverage over all possible shifts. For example, given a collection of&#10;subgroups, our prediction sets guarantee coverage over each group. For more&#10;flexible, infinite-dimensional classes where exact coverage is impossible, we&#10;provide a procedure for quantifying the coverage errors of our algorithm.&#10;Moreover, by tuning interpretable hyperparameters, we allow the practitioner to&#10;control the size of these errors across shifts of interest. Our methods can be&#10;incorporated into existing split conformal inference pipelines, and thus can be&#10;used to quantify the uncertainty of modern black-box algorithms without&#10;distributional assumptions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.05221" label="2207.05221">
        <attvalues>
          <attvalue for="0" value="Language Models (Mostly) Know What They Know" />
          <attvalue for="1" value="  We study whether language models can evaluate the validity of their own&#10;claims and predict which questions they will be able to answer correctly. We&#10;first show that larger models are well-calibrated on diverse multiple choice&#10;and true/false questions when they are provided in the right format. Thus we&#10;can approach self-evaluation on open-ended sampling tasks by asking models to&#10;first propose answers, and then to evaluate the probability &quot;P(True)&quot; that&#10;their answers are correct. We find encouraging performance, calibration, and&#10;scaling for P(True) on a diverse array of tasks. Performance at self-evaluation&#10;further improves when we allow models to consider many of their own samples&#10;before predicting the validity of one specific possibility. Next, we&#10;investigate whether models can be trained to predict &quot;P(IK)&quot;, the probability&#10;that &quot;I know&quot; the answer to a question, without reference to any particular&#10;proposed answer. Models perform well at predicting P(IK) and partially&#10;generalize across tasks, though they struggle with calibration of P(IK) on new&#10;tasks. The predicted P(IK) probabilities also increase appropriately in the&#10;presence of relevant source materials in the context, and in the presence of&#10;hints towards the solution of mathematical word problems. We hope these&#10;observations lay the groundwork for training more honest models, and for&#10;investigating how honesty generalizes to cases where models are trained on&#10;objectives other than the imitation of human writing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.10193" label="2306.10193">
        <attvalues>
          <attvalue for="0" value="Conformal Language Modeling" />
          <attvalue for="1" value="  We propose a novel approach to conformal prediction for generative language&#10;models (LMs). Standard conformal prediction produces prediction sets -- in&#10;place of single predictions -- that have rigorous, statistical performance&#10;guarantees. LM responses are typically sampled from the model's predicted&#10;distribution over the large, combinatorial output space of natural language.&#10;Translating this process to conformal prediction, we calibrate a stopping rule&#10;for sampling different outputs from the LM that get added to a growing set of&#10;candidates until we are confident that the output set is sufficient. Since some&#10;samples may be low-quality, we also simultaneously calibrate and apply a&#10;rejection rule for removing candidates from the output set to reduce noise.&#10;Similar to conformal prediction, we prove that the sampled set returned by our&#10;procedure contains at least one acceptable answer with high probability, while&#10;still being empirically precise (i.e., small) on average. Furthermore, within&#10;this set of candidate responses, we show that we can also accurately identify&#10;subsets of individual components -- such as phrases or sentences -- that are&#10;each independently correct (e.g., that are not &quot;hallucinations&quot;), again with&#10;statistical guarantees. We demonstrate the promise of our approach on multiple&#10;tasks in open-domain question answering, text summarization, and radiology&#10;report generation using different LM variants.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Conformal prediction and risk control. Our work adds to the rich collection of tools for uncertainty estimation and risk control for machine learning algorithms~\cite[][inter alia]{angelopoulos2023conformal, anastasios-learning-2021,barber2021predictive, bates-rcps, fisch2022false_positives, gupta2020binary, lei-robins-wasserman-dfps, lei2018distribution,vovk2002calibration,vovk2015probabilistic,pmlr-v60-vovk17a}. &#10;These techniques were previously extended and applied in the language domain to classification with finitely-many classes~\cite{fisch2021admission,fisch2021fewshot,jones2022capturing}, to token-level predictions~\cite{dey_conf_inf_2022,ravfogel2023conformal}, and to reliably accelerate LMs~\cite{goldshtein2023efficiently,schuster-etal-2021-consistent,schuster2022confident}. Here, we address the emerging challenge of providing reliable prediction sets for unbounded, free-text generation---which previous methods are unequipped for.&#10;The distribution-free, finite-sample performance guarantees that we derive &#10;are similar to those given by prediction sets or regression intervals in standard conformal prediction~\cite{angelopoulos2023conformal, papadopoulos2002inductive, vovk2005algorithmic}, but with slightly relaxed ``correctness'' criterions~\cite{cauchois2022predictive, fisch2021admission}. &#10;In particular, we build on the groundwork set by \cite{anastasios-learning-2021}, which provides a general methodology for calibrating any risk function that is controllable via some low-dimensional hyper-parameter configuration. We extend their framework to handle sampling-based algorithms that can effectively be used for LMs, and that, critically, do not require enumerating the full output space (which is intractable in our case). &#10;Most relevant to our work in LMs, other recent approaches have built on conformal principles to construct confidence intervals for generative diffusion models over images~\cite{horwitz2022conffusion,Teneggi2023HowTT}. These methods do not directly translate to LMs, however, as they only provide non-combinatorial confidence intervals at the pixel-level.\looseness=-1&#10;&#10;Uncertainty estimation in LMs.&#10;As the use of LMs in-the-wild quickly grows, there is increasing interest in obtaining and expressing meaningful confidence estimates for each output. Recent studies show that the logits of out-of-the-box LMs tend to exhibit overconfidence, even when wrong~\cite{desai-durrett-2020-calibration, kadavath2022language,miao2021prevent, vasconcelos2023generation}. Recent alignment techniques degrade this even further~\cite{kadavath2022language,openai2023gpt4}. Most current mitigation approaches focus on introducing linguistic cues~\cite{Lin2022TeachingMT,Zhou2023NavigatingTG} or post-hoc logit calibration~\cite{jiang-etal-2021-know,kadavath2022language,mielke-etal-2022-reducing,zablotskaia2023uncertainty}. In this work, we develop similar techniques to improve the output of the underlying LM. Our methods are model agnostic and provide rigorous guarantees.&#10;Our conformal component selection (\S\ref{sec:comp_selection}) also relates to recent self-consistency work that builds on the empirical observation that repeated similar samples are more likely to be correct~\cite{mitchell-etal-2022-enhancing,wang2023selfconsistency}, and cross-sample entailment can approximate uncertainty~\cite{kuhn2023semantic}. Unlike previous work that uses a fixed number of re-samples and compares full outputs, we (a) introduce a dynamic stopping rule to reduce the number of samples, (b) extend this concept to semantically compare sub-components of long text outputs, and (c) conformalize the process to provide formal guarantees.\looseness=-1&#10;&#10;Reliable generation.&#10;It is common practice to post-hoc apply classifiers and filters on top of LM generations for various quality goals such as preventing toxicity~\cite{gehman-etal-2020-realtoxicityprompts,rauh2022characteristics,Welbl2021ChallengesID}, verifying grounding against sources~\cite{bohnet2023attributed,liu2023evaluating,yue2023automatic}, or re-ranking the set of decoded outputs~\cite{jiang2022pairreranker}. Our work provides a systematic and reliable approach for filtering or flagging poor-quality outputs---both at a full generation and component level---and can also readily incorporate additional signal from auxiliary classifiers. For example, we demonstrate in our experiments using off-the-shelf natural language inference (NLI) models~\cite{bowman-etal-2015-large,Khot2018SciTaiLAT,schuster-etal-2021-get,thorne-etal-2018-fever,williams-etal-2018-broad,zhang-etal-2019-paws} to help guide the selection of individual, confident components in text summarization (i.e., sentences that are fully entailed by the larger text~\cite{fabbri-etal-2022-qafacteval,honovich-etal-2022-true,laban-etal-2022-summac,schuster-etal-2022-stretching}).\looseness=-1&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.18404" label="2305.18404">
        <attvalues>
          <attvalue for="0" value="Conformal Prediction with Large Language Models for Multi-Choice&#10;  Question Answering" />
          <attvalue for="1" value="  As large language models continue to be widely developed, robust uncertainty&#10;quantification techniques will become crucial for their safe deployment in&#10;high-stakes scenarios. In this work, we explore how conformal prediction can be&#10;used to provide uncertainty quantification in language models for the specific&#10;task of multiple-choice question-answering. We find that the uncertainty&#10;estimates from conformal prediction are tightly correlated with prediction&#10;accuracy. This observation can be useful for downstream applications such as&#10;selective classification and filtering out low-quality predictions. We also&#10;investigate the exchangeability assumption required by conformal prediction to&#10;out-of-subject questions, which may be a more realistic scenario for many&#10;practical applications. Our work contributes towards more trustworthy and&#10;reliable usage of large language models in safety-critical situations, where&#10;robust guarantees of error rate are required.&#10;" />
          <attvalue for="2" value=" &#10;&#10;Large language models (LLMs) have recently achieved impressive performance on a number of NLP tasks, such as machine translation, text summarization, and code generation. However, lingering concerns of trust and bias still limit their widespread application for critical decision-making domains such as healthcare.&#10;&#10;One well-known issue with current LLMs is their tendency to ``hallucinate'' false information with seemingly high confidence. These hallucinations can occur when the model generates outputs not grounded in any factual basis or when the prompt is highly unusual or ambiguous. This behavior of LLMs may also result from how these models are trained --- using statistical sampling for next-token prediction --- which can progressively increase the likelihood of factual errors as the length of generated tokens increases~\cite{LeCun_2023}. Factually incorrect outputs may confuse and deceive users into drawing wrong conclusions, ultimately decreasing the overall system's trustworthiness. Decisions based on unpredictable or biased model behavior could have significant negative and socially harmful consequences in high-stakes domains such as healthcare and law.&#10;&#10;Therefore, we seek to explore principled uncertainty quantification (UQ) techniques for LLMs that can provide guaranteed error rates of model predictions. Ideally, these UQ techniques should be model agnostic and easy to implement without requiring model retraining due to the intensive computing costs and limited API access associated with many LLMs. To this end, we investigate conformal prediction, a distribution-free UQ framework, to provide LLMs for the task of multiple-choice question-answering (MCQA). &#10;&#10;Based on our experiments, we find the uncertainty, as provided by conformal prediction, to be strongly correlated with accuracy, enabling applications such as filtering out low-quality predictions to prevent a degraded user experience. We also verify the importance of the exchangeability assumption in conformal prediction (see section\ref{background}) for guaranteeing a user-specified level of errors. &#10;&#10;To summarize, our contributions are the following:&#10;\begin{itemize}&#10;\item we adapt conformal prediction for MCQA tasks to provide distribution-free uncertainty quantification in LLMs,&#10;\item show how the uncertainty provided by conformal prediction can be useful for downstream tasks such as selective classification,&#10;\item and assess the performance of conformal prediction when the exchangeability assumption is violated for in-context learning in LLMs.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.10850" label="2302.10850">
        <attvalues>
          <attvalue for="0" value="Offline Reinforcement Learning for Mixture-of-Expert Dialogue Management" />
          <attvalue for="1" value="  Reinforcement learning (RL) has shown great promise for developing dialogue&#10;management (DM) agents that are non-myopic, conduct rich conversations, and&#10;maximize overall user satisfaction. Despite recent developments in RL and&#10;language models (LMs), using RL to power conversational chatbots remains&#10;challenging, in part because RL requires online exploration to learn&#10;effectively, whereas collecting novel human-bot interactions can be expensive&#10;and unsafe. This issue is exacerbated by the combinatorial action spaces facing&#10;these algorithms, as most LM agents generate responses at the word level. We&#10;develop a variety of RL algorithms, specialized to dialogue planning, that&#10;leverage recent Mixture-of-Expert Language Models (MoE-LMs) -- models that&#10;capture diverse semantics, generate utterances reflecting different intents,&#10;and are amenable for multi-turn DM. By exploiting MoE-LM structure, our methods&#10;significantly reduce the size of the action space and improve the efficacy of&#10;RL-based DM. We evaluate our methods in open-domain dialogue to demonstrate&#10;their effectiveness w.r.t.\ the diversity of intent in generated utterances and&#10;overall DM performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural language processing (NLP) has made significant strides in recent years, notably in the field of language generation. Thanks to advances in language modeling, particularly with the use of the transformer architecture \cite{vaswani2017attention}, NLP models can now generate text that is often difficult to distinguish from that written by a human. However, despite these advancements, these models still fall short when it comes to having rich conversations. Current NLP models lack effective dialogue management: they are good at generating individual sentences, but struggle with maintaining coherent and engaging conversations. Whereas most compelling conversations generally span numerous topics, are rather open-ended, and often have an underlying goal (e.g., customer success, task completion, recommendation). This requires dialogue agents to understand the context of the conversation and respond appropriately while maintaining the ability to achieve goals. &#10;&#10;Reinforcement learning (RL) is a natural approach to learning a policy for a dialogue management (DM) agent. Earlier work on RL-based dialogue systems relies on specific, hand-crafted semantic states~\cite{levin1997stochastic, singh2002optimizing, walker2000application} or partially observable belief states~\cite{williams2007partially, young2010hidden}, in which the agent encodes conversations and chooses the best-structured dialogue action at each turn. Applications include relational reasoning~\cite{shah2018bootstrapping}, task completion~\cite{shi2018sentiment}, and query fulfillment~\cite{serban2017deep}, whose action spaces are structured enough to be represented by hand-crafted features. To handle more complex dialogues, recent approaches use language models (LMs) to extract semantic representations from conversation histories, treat them as dialogue states, and apply RL to learn a word-level generative DM agent~\cite{jaques2019way, li2016deep, li2017adversarial, shin2020generating}. &#10;&#10;However, unlike supervised learning approaches, where one can train imitation agents with offline conversation data, RL-based DM algorithms require online exploration. Unfortunately, constant interactions with real users are often expensive and time-consuming. While one can potentially address the DM problem using offline RL, issues such as model exploitation leading to distribution shift on the state and action spaces, when training on static datasets are of paramount concern \cite{levine2020offline}. Moreover, the myriad variations of language make incorporating all possible conversation histories and bot utterances into the state and action spaces of an RL formulation of the DM problem impractical due to the combinatorics at play. As a result, naive application of RL to DM may result in poorly-performing agents that generate incomprehensible utterances~\cite{zhao2019rethinking}. &#10;&#10;We tackle the above issues related to the use of offline RL in DM systems by leveraging recent advances in Mixture-of-Expert Language Models (MoE-LMs) \cite{chow2023mixture}. Specifically, we develop a suite of offline RL algorithms specialized in dialogue planning that exploit the structure of MoE-LMs.&#10;Our methods consist of three main components: &#10;{1)} a primitive LM, which uses a probabilistic encoder and decoder and is capable of &#10;generating diverse semantic intents; &#10;{2)} a number of {specialized} expert LMs, each of which generates utterances corresponding to a specific intent; and {3)} a compositional DM that, at each turn, given the encoded conversation history, selects an utterance from a set of candidate utterances suggested by the experts and pass it to the DM agent to execute. &#10;&#10;Our contributions to offline RL adapted for MoE-based DM agents are four-fold. &#10;First, we exploit the hierarchical structure of MoE-LMs, allowing our offline RL methods to work with a significantly smaller, finite action space, making the RL problem more tractable. &#10;Second, by leveraging pre-trained MoE-LMs---which generate coherent utterances---and regularization techniques from offline RL that align the DM’s behavior with that of the primitive LM---the proposed RL algorithms can focus on higher-level dialogue planning. The proposed combination results in higher data efficiency than standard RL methods by delegating the responsibility of language fluency to be handled by the MoE-LMs. &#10;Third, by using the diverse semantic representations of MoE-LMs, our methods operate at the sentence embedding space and have much simpler critic and actor updates. This circumvents the word-level credit-assignment issue, particularly challenging in long conversations \cite{saleh2020hierarchical}. &#10;Fourth, in contrast to the findings of \cite{verma2022chai}, where offline RL agents tend to lack utterance diversity (due to potential reward hacking and optimization of a single objective), our MoE-based DM agents by design are adept at generating utterances that reflect different intents.&#10;&#10;We begin with a brief introduction of LMs, the MoE-LM architecture, and the use of Markov decision processes (MDPs) in DM in Section~\ref{sec:section2}. We then describe the pre-training procedure for MoE-LMs---which encode diverse semantics and generate fluent utterances capturing specific intents---in Section~\ref{sec:section3}. We derive state-of-the-art (SOTA) offline RL algorithms for training MoE-LMs in Section~\ref{sec:section4}, and three MoE-LM specialized offline RL algorithms in Section~\ref{sec:section5}. Finally, in Section~\ref{sec:section6}, we evaluate our algorithms in open-domain dialogues against their ability to generate utterances with diverse intents and their overall DM performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Artificial Intelligence, Dialogue Management, Natural Language Processing, Reinforcement Learning" />
        </attvalues>
      </node>
      <node id="2005.01643" label="2005.01643">
        <attvalues>
          <attvalue for="0" value="Offline Reinforcement Learning: Tutorial, Review, and Perspectives on&#10;  Open Problems" />
          <attvalue for="1" value="  In this tutorial article, we aim to provide the reader with the conceptual&#10;tools needed to get started on research on offline reinforcement learning&#10;algorithms: reinforcement learning algorithms that utilize previously collected&#10;data, without additional online data collection. Offline reinforcement learning&#10;algorithms hold tremendous promise for making it possible to turn large&#10;datasets into powerful decision making engines. Effective offline reinforcement&#10;learning methods would be able to extract policies with the maximum possible&#10;utility out of the available data, thereby allowing automation of a wide range&#10;of decision-making domains, from healthcare and education to robotics. However,&#10;the limitations of current algorithms make this difficult. We will aim to&#10;provide the reader with an understanding of these challenges, particularly in&#10;the context of modern deep reinforcement learning methods, and describe some&#10;potential solutions that have been explored in recent work to mitigate these&#10;challenges, along with recent applications, and a discussion of perspectives on&#10;open problems in the field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.00115" label="2305.00115">
        <attvalues>
          <attvalue for="0" value="Towards Better Domain Adaptation for Self-supervised Models: A Case&#10;  Study of Child ASR" />
          <attvalue for="1" value="  Recently, self-supervised learning (SSL) from unlabelled speech data has&#10;gained increased attention in the automatic speech recognition (ASR) community.&#10;Typical SSL methods include autoregressive predictive coding (APC), Wav2vec2.0,&#10;and hidden unit BERT (HuBERT). However, SSL models are biased to the&#10;pretraining data. When SSL models are finetuned with data from another domain,&#10;domain shifting occurs and might cause limited knowledge transfer for&#10;downstream tasks. In this paper, we propose a novel framework, domain&#10;responsible adaptation and finetuning (DRAFT), to reduce domain shifting in&#10;pretrained speech models, and evaluate it for a causal and non-causal&#10;transformer. For the causal transformer, an extension of APC (E-APC) is&#10;proposed to learn richer information from unlabelled data by using multiple&#10;temporally-shifted sequences to perform prediction. For the non-causal&#10;transformer, various solutions for using the bidirectional APC (Bi-APC) are&#10;investigated. In addition, the DRAFT framework is examined for Wav2vec2.0 and&#10;HuBERT methods, which use non-causal transformers as the backbone. The&#10;experiments are conducted on child ASR (using the OGI and MyST databases) using&#10;SSL models trained with unlabelled adult speech data from Librispeech. The&#10;relative WER improvements of up to 19.7% on the two child tasks are observed&#10;when compared to the pretrained models without adaptation. With the proposed&#10;methods (E-APC and DRAFT), the relative WER improvements are even larger (30%&#10;and 19% on the OGI and MyST data, respectively) when compared to the models&#10;without using pretraining methods.&#10;" />
          <attvalue for="2" value="&#10;D{espite} impressive advancement in developing automatic speech recognition (ASR) techniques in the last decade, children's ASR remains difficult. Challenges arise, in part, from difficulties in acoustic and language modeling of child speech. Due to different growth patterns of children and motor control issues, child speech has a higher degree of intra-speaker and inter-speaker acoustic variability than adult speech~\cite{lee1999acoustics}. Additionally, child speech is characterized by significant mispronunciations and disfluencies~\cite{yaruss1999language,tran2020analysis}. Another challenge is the lack of large-scale publicly-available child speech databases, and thus child ASR can be treated as a low-resource task\cite{wang2021low}.&#10;&#10;Recently, self-supervised learning (SSL) from speech data has been investigated\cite{chen2021wavlm,zhang2021bigssl,pmlr-v139-wang21y,chung2021w2v,ao2021speecht5,jiang2021further,wang2020unsupervised,liu2020mockingjay,liu2021tera} because of its great potential of improving low-resource tasks through learning prior knowledge from large amounts of data without annotations. SSL models can be used in two manners: 1) feature extraction to replace human-designed features\cite{yang2021superb,evain2021lebenchmark,chang2021exploration}; and 2) model initialization for finetuning downstream tasks\cite{vyas2021comparing,misra2021comparison}. The idea of SSL is to design pseudo-labels for training deep neural networks (DNN) and then transfer the learned knowledge to a downstream supervised task. For example, autoregressive predictive coding (APC) uses temporally-shifted sequences to perform prediction such that the model predicts future frames from previous frames\cite{chung2019unsupervised,chung2020generative,Ravi2020}. In \cite{fan2021bi}, we proposed a bidirectional APC (Bi-APC) method for bidirectional long short-term memory (BLSTM) pretraining for children's ASR. Different from APC and Bi-APC where the reconstruction loss is used, Wav2vec-based methods are implemented to include negative samples, and a contrastive loss is utilized to increase the distance from the output to negative samples and decrease that distance to the positive sample\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}. The positive sample is the frame being masked (to be predicted), and negative samples are the unmasked frames in the utterance.&#10;A more recent SSL framework, HuBERT\cite{hsu2021hubertc,hsu2021hubertj}, creates the pseudo-label of each speech frame using clustering techniques like K-means. These methods have been shown to be effective for low-resource ASR tasks such as low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec} and accented speech\cite{li2021accent}. &#10;&#10;However, a weakness of SSL training is domain shifting that happens when the domain of the finetuning data is different than that of the pretraining data\cite{meng2021don,hsu2021robust}. Although a performance improvement can be observed when the magnitude of the pretraining data is large enough, previous work has shown that additional gains can be obtained by including target domain data in the ASR pretraining stage\cite{hsu2021robust,hwang2021large}. But including target domain data would be impractical if we are not aware of the finetuning task at the pretraining stage. In addition, retraining a large-scale SSL model with both the source and target domain data to address domain shifting may not always be possible or computationally efficient. Hence, investigating adaptation methods for SSL is gaining attention for work involving out-of-domain low-resource tasks. Previous studies proposed to perform adaptation of supervised models either during or after the finetuning stage \cite{khurana2021magic,huo2021incremental}. No additional adaptation stage of self-supervised models has been investigated before for domain shifting in SSL methods. &#10;&#10;In this paper, building on our work in \cite{fan2021bi}, we explore how SSL methods can improve the performance of child ASR in the context of a low-resource setting for causal and non-causal transformers. First, autoregressive predictions at different temporal distances are shown to enable the pretrained model to learn more effectively \cite{chung2020generative}. We, therefore, propose to use multiple temporally-shifted sequences to construct a multi-task training objective for APC. Second, the proposed Bi-APC framework in our previous work performs well for BLSTM, whose parameters can be separated into forward-related and reverse-related ones. It is unknown whether the Bi-APC framework can be used for transformer architectures that have only one set of parameters. To do so, we copy the modules in the transformer during pretraining and treat the modules as separate parameters for two APCs in two directions. After pretraining, we either use one of the modules when the weights are shared, or average the weights of the two modules to formalize the final parameters for finetuning. Finally, we propose a domain responsible adaptation and finetuning (DRAFT) framework to address the domain shifting problem in SSL. In DRAFT, residual adapters are placed between blocks in the transformer and are responsible for learning domain related information at an additional adaptation stage. The additional adaptation stage trains the model with target finetuning data and with the same SSL loss that was used in the pretraining stage. Only residual adapters are updated during the adaptation stage so that the knowledge learned from source domain data can be retained. The proposed DRAFT framework has a lower cost than adding target domain data at the pretraining stage and can be used in various SSL methods.&#10;&#10;Note that residual adapters have been proposed before in the literature. In \cite{tomanek2021residual, houlsby2019parameter}, residual adapters are inserted to achieve a parameter efficient adaptation for low-resource supervised tasks, but the performance is worse than finetuning the entire model. The method is beneficial when adaptation is frequently required such as personalization of a speech recognition model. In \cite{hou2021exploiting,kannan2019large,rebuffi2017learning}, residual adapters are applied to learn domain specific parameters to achieve robust models for various domains. Differences between our work and these methods will be discussed further in Sec.\ref{sssec:res_adapt}. The DRAFT part is an extension of our recent paper \cite{fan2022draft}. We report on more experiments in this paper to better understand the functionality of the residual adapters.&#10;&#10;The contributions of this paper are:&#10;&#10;\begin{itemize}&#10; \item{An extension to autoregressive predictive coding (E-APC) is proposed so that the pretrained model can learn more useful speech representations from unlabelled data. It is then used for a causal transformer pretraining.}&#10; \item{Various solutions for using the Bi-APC algorithm in non-causal transformers are investigated.}&#10; \item{A domain responsible adaptation and finetuning (DRAFT) framework is proposed to address the domain shifting problem in self-supervised pretrained models. Different from \cite{fan2022draft}, DRAFT's performance is examined with Bi-APC, and ablation studies are conducted for a better understanding of the DRAFT framework.}&#10;\end{itemize}&#10;&#10;The remainder of this paper is organized as follows. Section \ref{sec:ssl_general} proposes a general SSL framework. Section \ref{sec:method} describes the proposed methods for better improving low-resource ASR tasks with SSL pretrained models. Experimental setups are described in Section \ref{sec:exp_setup}. Results are shown and discussed in Section \ref{sec:result}. We conclude the paper in Section \ref{sec:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Domain Adaptation, Artificial Intelligence, Signal Processing, Speech Recognition, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2101.07597" label="2101.07597">
        <attvalues>
          <attvalue for="0" value="UniSpeech: Unified Speech Representation Learning with Labeled and&#10;  Unlabeled Data" />
          <attvalue for="1" value="  In this paper, we propose a unified pre-training approach called UniSpeech to&#10;learn speech representations with both unlabeled and labeled data, in which&#10;supervised phonetic CTC learning and phonetically-aware contrastive&#10;self-supervised learning are conducted in a multi-task learning manner. The&#10;resultant representations can capture information more correlated with phonetic&#10;structures and improve the generalization across languages and domains. We&#10;evaluate the effectiveness of UniSpeech for cross-lingual representation&#10;learning on public CommonVoice corpus. The results show that UniSpeech&#10;outperforms self-supervised pretraining and supervised transfer learning for&#10;speech recognition by a maximum of 13.4% and 17.8% relative phone error rate&#10;reductions respectively (averaged over all testing languages). The&#10;transferability of UniSpeech is also demonstrated on a domain-shift speech&#10;recognition task, i.e., a relative word error rate reduction of 6% against the&#10;previous approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;A common way to improve the performance of low-resource ASR models is to leverage data from other high-resource settings. Transfer learning and multitask learning are commonly used methods. Transfer learning \cite{DBLP:conf/rep4nlp/KunzeKKKJS17, DBLP:conf/interspeech/HuangYLG14a, DBLP:conf/interspeech/JoshiZMKL20} first trains the model on the high-resource setting and then fine-tune it on the target data-scarce setting. The parameters learned from the first setting serve as a starting point, also known as supervised pre-training. In multitask learning \cite{huang2013cross, DBLP:conf/interspeech/KnillGRR14, chen2015multitask}, the model is simultaneously trained on multiple languages with shared components. Both methods depend on labeled data from multiple languages to yield consistent improvements while large amount of unlabeled data cannot be used. &#10;&#10;Recently, self-supervised learning has received great attention as it does not require any labeled data. Based on the training objectives, self-supervised methods can be categorized into reconstructive learning (recreating audio frames) and contrastive learning (discriminating true sample from set of negative samples). \cite{DBLP:journals/taslp/ChenHLWS19} use an autoencoder to perform full reconstruction. \cite{DBLP:journals/taslp/ChorowskiWBO19} use a high capacity WaveNet autoencoders to learn meaningful speech representations. They compare three different variants of constraints: a simple bottleneck, a Gaussian Variational Autoencoder (VAE) and a Vector Quantized VQE (VQ-VAE). Autoregressive predictive coding (APC) \cite{DBLP:conf/interspeech/ChungHTG19} reconstructs the future frame with an unidirectional encoder for phone classification and SR task. Masked reconstruction \cite{DBLP:conf/icassp/LiuYCHL20,DBLP:conf/icassp/LingLSK20,ling2020bertphone} has also been widely investigated which masks part of the input and learn to reconstruct it. In the research line of contrastive learning, CPC \cite{oord2018representation} uses an autoregressive model to classify future frames from negative examples. Wav2vec \cite{DBLP:conf/interspeech/SchneiderBCA19} evaluates the effectiveness of contrastive learning on speech recognition task. \cite{DBLP:conf/emnlp/KawakamiWDBO20} and \cite{DBLP:conf/icassp/RiviereJMD20} show bi-directional and modified CPC transfers well across domains and languages. Vq-wav2vec \cite{DBLP:conf/iclr/BaevskiSA20} uses a vector quantization module to learn discrete representations. They further introduce Wav2vec2.0 \cite{DBLP:conf/nips/BaevskiZMA20}, which masks the speech input in the latent space and solves a contrastive task defined over contextual representations in the masked region and a quantization of the latent representations. They show the discrete latent speech representations learnt by quantizer are related to phonemes. \cite{DBLP:journals/corr/abs-2006-13979} try the idea on multilingual settings, named XLSR. &#10;&#10;Some other recent work employ multitask learning strategy for speech representation learning. \cite{DBLP:conf/interspeech/PascualRSBB19} and \cite{DBLP:conf/icassp/RavanelliZPSMTB20} propose to learn a problem-agnostic speech encoder which jointly solves different self-supervised tasks, including reconstructive loss and contrastive loss. Compared to our work, they use only unlabeled data and the models are evaluated on speaker identification, emotion classification and ASR tasks. While we focus on improving ASR performance in low-resource scenarios with all availabel data. \cite{DBLP:journals/corr/abs-2011-00093} alternatively minimize the unsupervised masked CPC loss and the supervised CTC loss. They focus on simplify the training pipeline and reports equivalent word error rate as in wav2vec2.0. In contrast, our method can improve the recognition performance significantly. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2110.07205" label="2110.07205">
        <attvalues>
          <attvalue for="0" value="SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language&#10;  Processing" />
          <attvalue for="1" value="  Motivated by the success of T5 (Text-To-Text Transfer Transformer) in&#10;pre-trained natural language processing models, we propose a unified-modal&#10;SpeechT5 framework that explores the encoder-decoder pre-training for&#10;self-supervised speech/text representation learning. The SpeechT5 framework&#10;consists of a shared encoder-decoder network and six modal-specific&#10;(speech/text) pre/post-nets. After preprocessing the input speech/text through&#10;the pre-nets, the shared encoder-decoder network models the&#10;sequence-to-sequence transformation, and then the post-nets generate the output&#10;in the speech/text modality based on the output of the decoder. Leveraging&#10;large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a&#10;unified-modal representation, hoping to improve the modeling capability for&#10;both speech and text. To align the textual and speech information into this&#10;unified semantic space, we propose a cross-modal vector quantization approach&#10;that randomly mixes up speech/text states with latent units as the interface&#10;between encoder and decoder. Extensive evaluations show the superiority of the&#10;proposed SpeechT5 framework on a wide variety of spoken language processing&#10;tasks, including automatic speech recognition, speech synthesis, speech&#10;translation, voice conversion, speech enhancement, and speaker identification.&#10;We release our code and model at https://github.com/microsoft/SpeechT5.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.02558" label="2104.02558">
        <attvalues>
          <attvalue for="0" value="Comparing CTC and LFMMI for out-of-domain adaptation of wav2vec 2.0&#10;  acoustic model" />
          <attvalue for="1" value="  In this work, we investigate if the wav2vec 2.0 self-supervised pretraining&#10;helps mitigate the overfitting issues with connectionist temporal&#10;classification (CTC) training to reduce its performance gap with flat-start&#10;lattice-free MMI (E2E-LFMMI) for automatic speech recognition with limited&#10;training data. Towards that objective, we use the pretrained wav2vec 2.0 BASE&#10;model and fine-tune it on three different datasets including out-of-domain&#10;(Switchboard) and cross-lingual (Babel) scenarios. Our results show that for&#10;supervised adaptation of the wav2vec 2.0 model, both E2E-LFMMI and CTC achieve&#10;similar results; significantly outperforming the baselines trained only with&#10;supervised data. Fine-tuning the wav2vec 2.0 model with E2E-LFMMI and CTC we&#10;obtain the following relative WER improvements over the supervised baseline&#10;trained with E2E-LFMMI. We get relative improvements of 40% and 44% on the&#10;clean-set and 64% and 58% on the test set of Librispeech (100h) respectively.&#10;On Switchboard (300h) we obtain relative improvements of 33% and 35%&#10;respectively. Finally, for Babel languages, we obtain relative improvements of&#10;26% and 23% on Swahili (38h) and 18% and 17% on Tagalog (84h) respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04934" label="2110.04934">
        <attvalues>
          <attvalue for="0" value="Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs&#10;  for Robust Speech Recognition" />
          <attvalue for="1" value="  The goal of self-supervised learning (SSL) for automatic speech recognition&#10;(ASR) is to learn good speech representations from a large amount of unlabeled&#10;speech for the downstream ASR task. However, most SSL frameworks do not&#10;consider noise robustness which is crucial for real-world applications. In this&#10;paper we propose wav2vec-Switch, a method to encode noise robustness into&#10;contextualized representations of speech via contrastive learning.&#10;Specifically, we feed original-noisy speech pairs simultaneously into the&#10;wav2vec 2.0 network. In addition to the existing contrastive learning task, we&#10;switch the quantized representations of the original and noisy speech as&#10;additional prediction targets of each other. By doing this, it enforces the&#10;network to have consistent predictions for the original and noisy speech, thus&#10;allows to learn contextualized representation with noise robustness. Our&#10;experiments on synthesized and real noisy data show the effectiveness of our&#10;method: it achieves 2.9--4.9% relative word error rate (WER) reduction on the&#10;synthesized noisy LibriSpeech data without deterioration on the original data,&#10;and 5.7% on CHiME-4 real 1-channel noisy data compared to a data augmentation&#10;baseline even with a strong language model for decoding. Our results on CHiME-4&#10;can match or even surpass those with well-designed speech enhancement&#10;components.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.03520" label="2110.03520">
        <attvalues>
          <attvalue for="0" value="Accent-Robust Automatic Speech Recognition Using Supervised and&#10;  Unsupervised Wav2vec Embeddings" />
          <attvalue for="1" value="  Speech recognition models often obtain degraded performance when tested on&#10;speech with unseen accents. Domain-adversarial training (DAT) and multi-task&#10;learning (MTL) are two common approaches for building accent-robust ASR models.&#10;ASR models using accent embeddings is another approach for improving robustness&#10;to accents. In this study, we perform systematic comparisons of DAT and MTL&#10;approaches using a large volume of English accent corpus (4000 hours of US&#10;English speech and 1244 hours of 20 non-US-English accents speech). We explore&#10;embeddings trained under supervised and unsupervised settings: a separate&#10;embedding matrix trained using accent labels, and embeddings extracted from a&#10;fine-tuned wav2vec model. We find that our DAT model trained with supervised&#10;embeddings achieves the best performance overall and consistently provides&#10;benefits for all testing datasets, and our MTL model trained with wav2vec&#10;embeddings are helpful learning accent-invariant features and improving&#10;novel/unseen accents. We also illustrate that wav2vec embeddings have more&#10;advantages for building accent-robust ASR when no accent labels are available&#10;for training supervised embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.07957" label="2110.07957">
        <attvalues>
          <attvalue for="0" value="Don't speak too fast: The impact of data bias on self-supervised speech&#10;  models" />
          <attvalue for="1" value="  Self-supervised Speech Models (S3Ms) have been proven successful in many&#10;speech downstream tasks, like ASR. However, how pre-training data affects S3Ms'&#10;downstream behavior remains an unexplored issue. In this paper, we study how&#10;pre-training data affects S3Ms by pre-training models on biased datasets&#10;targeting different factors of speech, including gender, content, and prosody,&#10;and evaluate these pre-trained S3Ms on selected downstream tasks in SUPERB&#10;Benchmark. Our experiments show that S3Ms have tolerance toward gender bias.&#10;Moreover, we find that the content of speech has little impact on the&#10;performance of S3Ms across downstream tasks, but S3Ms do show a preference&#10;toward a slower speech rate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;One of the most often used learning schemes for self-supervised speech models is through reconstruction speech frames.&#10;Here we introduce some of the recently emerged reconstruction methods.&#10;The Autoregressive Predictive Coding (APC) method~\cite{chung2020generative}, is primarily inspired by language models (LM) for text.&#10;The DeCoAR~\cite{decoar} method combines the bidirectionality of ELMo~\cite{elmo} and the autoregressive reconstruction objective of APC~\cite{chung2020generative}.&#10;The work of \cite{apc2, vq_apc, improved_apc} also adopt the autoregressive reconstruction scheme, but in some variation.&#10;The Transformer Encoder Representations from Alteration (TERA)~\cite{tera} method, an improved version of Mockingjay~\cite{mockingjay}, is mainly inspired by masked language models (MLM)~\cite{bert} for text.&#10;The work of \cite{mpc, mpc2, mpe} also adopt variations of the MLM reconstruction schemes.&#10;In this work, we select two methods to represent each scheme for our study, the APC method from the autoregressive family and the TERA method from the MLM family.&#10;We select two models for our study due to space limitations.&#10;&#10;Bias and fairness issues in speech are receiving more attention these days. Demographic bias is the most studied. Recent works analyze the impact of demographic bias as well as mitigating bias on specific tasks including ASR\cite{meyer-etal-2020-artie, 10.1145/3442188.3445893}, speaker recognition\cite{fenu21_interspeech, fenu2021improving}. and speech translation\cite{bentivogli2020gender, gaido2020breeding, gaido2021split}. A large body of research related to data bias evaluates models on a single downstream task, however, data bias on S3Ms and its effect on downstream tasks from diverse categories are not yet explored. A related work analyzing pre-training data of S3Ms is \cite{hsu2021robust}, which investigates the effect of domain shift in SSL. Our work differs from theirs in the sense that we focus on data bias toward different speech factors at pre-training stage. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2110.03560" label="2110.03560">
        <attvalues>
          <attvalue for="0" value="Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0" />
          <attvalue for="1" value="  We propose a simple and effective cross-lingual transfer learning method to&#10;adapt monolingual wav2vec-2.0 models for Automatic Speech Recognition (ASR) in&#10;resource-scarce languages. We show that a monolingual wav2vec-2.0 is a good&#10;few-shot ASR learner in several languages. We improve its performance further&#10;via several iterations of Dropout Uncertainty-Driven Self-Training (DUST) by&#10;using a moderate-sized unlabeled speech dataset in the target language. A key&#10;finding of this work is that the adapted monolingual wav2vec-2.0 achieves&#10;similar performance as the topline multilingual XLSR model, which is trained on&#10;fifty-three languages, on the target language ASR task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.00155" label="2110.00155">
        <attvalues>
          <attvalue for="0" value="Incremental Layer-wise Self-Supervised Learning for Efficient Speech&#10;  Domain Adaptation On Device" />
          <attvalue for="1" value="  Streaming end-to-end speech recognition models have been widely applied to&#10;mobile devices and show significant improvement in efficiency. These models are&#10;typically trained on the server using transcribed speech data. However, the&#10;server data distribution can be very different from the data distribution on&#10;user devices, which could affect the model performance. There are two main&#10;challenges for on device training, limited reliable labels and limited training&#10;memory. While self-supervised learning algorithms can mitigate the mismatch&#10;between domains using unlabeled data, they are not applicable on mobile devices&#10;directly because of the memory constraint. In this paper, we propose an&#10;incremental layer-wise self-supervised learning algorithm for efficient speech&#10;domain adaptation on mobile devices, in which only one layer is updated at a&#10;time. Extensive experimental results demonstrate that the proposed algorithm&#10;obtains a Word Error Rate (WER) on the target domain $24.2\%$ better than&#10;supervised baseline and costs $89.7\%$ less training memory than the end-to-end&#10;self-supervised learning algorithm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.06952" label="2109.06952">
        <attvalues>
          <attvalue for="0" value="Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and&#10;  Accented Speech" />
          <attvalue for="1" value="  Automatic Speech Recognition (ASR) systems are often optimized to work best&#10;for speakers with canonical speech patterns. Unfortunately, these systems&#10;perform poorly when tested on atypical speech and heavily accented speech. It&#10;has previously been shown that personalization through model fine-tuning&#10;substantially improves performance. However, maintaining such large models per&#10;speaker is costly and difficult to scale. We show that by adding a relatively&#10;small number of extra parameters to the encoder layers via so-called residual&#10;adapter, we can achieve similar adaptation gains compared to model fine-tuning,&#10;while only updating a tiny fraction (less than 0.5%) of the model parameters.&#10;We demonstrate this on two speech adaptation tasks (atypical and accented&#10;speech) and for two state-of-the-art ASR architectures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.11905" label="2105.11905">
        <attvalues>
          <attvalue for="0" value="Exploiting Adapters for Cross-lingual Low-resource Speech Recognition" />
          <attvalue for="1" value="  Cross-lingual speech adaptation aims to solve the problem of leveraging&#10;multiple rich-resource languages to build models for a low-resource target&#10;language. Since the low-resource language has limited training data, speech&#10;recognition models can easily overfit. In this paper, we propose to use&#10;adapters to investigate the performance of multiple adapters for&#10;parameter-efficient cross-lingual speech adaptation. Based on our previous&#10;MetaAdapter that implicitly leverages adapters, we propose a novel algorithms&#10;called SimAdapter for explicitly learning knowledge from adapters. Our&#10;algorithm leverages adapters which can be easily integrated into the&#10;Transformer structure.MetaAdapter leverages meta-learning to transfer the&#10;general knowledge from training data to the test language. SimAdapter aims to&#10;learn the similarities between the source and target languages during&#10;fine-tuning using the adapters. We conduct extensive experiments on&#10;five-low-resource languages in Common Voice dataset. Results demonstrate that&#10;our MetaAdapter and SimAdapter methods can reduce WER by 2.98% and 2.55% with&#10;only 2.5% and 15.5% of trainable parameters compared to the strong full-model&#10;fine-tuning baseline. Moreover, we also show that these two novel algorithms&#10;can be integrated for better performance with up to 3.55% relative WER&#10;reduction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.08045" label="1705.08045">
        <attvalues>
          <attvalue for="0" value="Learning multiple visual domains with residual adapters" />
          <attvalue for="1" value="  There is a growing interest in learning data representations that work well&#10;for many different types of problems and data. In this paper, we look in&#10;particular at the task of learning a single visual representation that can be&#10;successfully utilized in the analysis of very different types of images, from&#10;dog breeds to stop signs and digits. Inspired by recent work on learning&#10;networks that predict the parameters of another, we develop a tunable deep&#10;network architecture that, by means of adapter residual modules, can be steered&#10;on the fly to diverse visual domains. Our method achieves a high degree of&#10;parameter sharing while maintaining or even improving the accuracy of&#10;domain-specific representations. We also introduce the Visual Decathlon&#10;Challenge, a benchmark that evaluates the ability of representations to capture&#10;simultaneously ten very different visual domains and measures their ability to&#10;recognize well uniformly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.07931" label="2206.07931">
        <attvalues>
          <attvalue for="0" value="DRAFT: A Novel Framework to Reduce Domain Shifting in Self-supervised&#10;  Learning and Its Application to Children's ASR" />
          <attvalue for="1" value="  Self-supervised learning (SSL) in the pretraining stage using un-annotated&#10;speech data has been successful in low-resource automatic speech recognition&#10;(ASR) tasks. However, models trained through SSL are biased to the pretraining&#10;data which is usually different from the data used in finetuning tasks, causing&#10;a domain shifting problem, and thus resulting in limited knowledge transfer. We&#10;propose a novel framework, domain responsible adaptation and finetuning&#10;(DRAFT), to reduce domain shifting in pretrained speech models through an&#10;additional adaptation stage. In DRAFT, residual adapters (RAs) are inserted in&#10;the pretrained model to learn domain-related information with the same SSL loss&#10;as the pretraining stage. Only RA parameters are updated during the adaptation&#10;stage. DRAFT is agnostic to the type of SSL method used and is evaluated with&#10;three widely used approaches: APC, Wav2vec2.0, and HuBERT. On two child ASR&#10;tasks (OGI and MyST databases), using SSL models trained with un-annotated&#10;adult speech data (Librispeech), relative WER improvements of up to 19.7% are&#10;observed when compared to the pretrained models without adaptation. Additional&#10;experiments examined the potential of cross knowledge transfer between the two&#10;datasets and the results are promising, showing a broader usage of the proposed&#10;DRAFT framework.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, self-supervised learning (SSL) for speech has been investigated\cite{chen2021wavlm,zhang2021bigssl,chung2021w2v,ao2021speecht5,liu2021tera} because of its great potential in improving low-resource ASR tasks. In SSL, pseudo-labels are generated for un-annotated data for model pretraining, and then the learned knowledge is transferred to a downstream supervised task through finetuning. For example, autoregressive predictive coding (APC) uses a shifted input sequence as supervision such that the model predicts future frames from previous frames\cite{chung2019unsupervised,chung2020generative,Ravi2020}. Different from APC, Wav2vec-based methods include sampled negative frames in a contrastive loss to increase discrimination between frames in a way that the learned embedding is closer to the positive frame and more distant to the negative frames\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}. A more recent SSL framework, HuBERT\cite{hsu2021hubertc,hsu2021hubertj}, creates a pseudo-label for each speech frame using clustering techniques like K-means. Models learned with SSL objectives can be used in two manners: 1) feature extraction as a replacement of hand-crafted speech features\cite{yang2021superb,evain2021lebenchmark,chang2021exploration}; or 2) model initialization for finetuning downstream tasks\cite{jiang2021further,wang2020unsupervised,misra2021comparison}. SSL has been shown to be effective in ASR of low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec}, accented speech\cite{li2021accent}, and child ASR\cite{fan2021bi,wang2021low}. In \cite{fan2021bi}, we extend APC to learn bidirectional contexts for pretraining from adult speech data.&#10;&#10;However, a main weakness of SSL is that training from one domain causes domain shifting when finetuning on data from a different domain\cite{meng2021don,sanabria2022measuring}. To address this issue, previous work presented robust pretrained models by adding target domain data during pretraining\cite{hsu2021robust,hwang2021large}. However, including target domain data might not be feasible at the pretraining stage. In addition, retraining a large-scale SSL model with both the source and target domain data may not be computationally efficient. In \cite{khurana2021magic}, un-annotated target domain data are used for semi-supervised learning during the finetuning stage to alleviate the performance degradation caused by domain shifting. However, no previous work, to our knowledge, has investigated methods for performing adaptation of self-supervised models with finetuning data.&#10;&#10;In this paper, we propose a novel framework, domain responsible adaptation and finetuning (DRAFT), to reduce domain shifting in SSL-pretrained speech models. In DRAFT, residual adapters (RAs) are placed between blocks in the transformer and are responsible for learning domain specific information during an additional adaptation stage. The additional adaptation stage trains the model with finetuning data and with the same SSL loss that was used in the pretraining stage. To prevent catastrophic forgetting of the learned knowledge from source domain data, only RA parameters are updated during the adaptation stage. Hence, DRAFT has a lower computational cost than retraining the pretrained models with both the source and target domain data. Note that DRAFT is universal to different SSL methods. When performing DRAFT on SSL-pretrained speech models (trained with adult speech data) for child ASR tasks, we obtain significant improvements over baselines without adaptation for both causal (pretrained with APC) and non-causal transformers (pretrained with Wav2vec2.0 or HuBERT). We presented partial results of DRAFT in a paper that is in review\cite{fan2022towards}. In this paper, more and different experiments are presented. For example, when the learned RAs from one task are used for finetuning another task (cross transfer), improvements are still observed, showing a broader usage of the proposed framework. We also include results of adapter finetuning experiments for more detailed comparisons.&#10;&#10;Note that residual adapters have been used before to learn domain specific parameters for adaptation\cite{hou2021exploiting,kannan2019large,rebuffi2017learning, tomanek2021residual}. In \cite{houlsby2019parameter} residual adapters are inserted to achieve a parameter efficient adaptation for disordered speech. In \cite{thomas2022efficient}, the same idea is used for SSL-pretrained models for efficient adaptation, which is known as adapter tuning in natural language processing. However, these methods use residual adapters for supervised tasks or during finetuning, unlike our proposed method. &#10;&#10;The remainder of this paper is organized as follows. Section \ref{sec:method} describes the proposed DRAFT framework. Experimental setups are described in Section \ref{sec:exp_setup}. Results are shown and discussed in Section \ref{sec:result}. We conclude the paper in Section \ref{sec:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.13581" label="2311.13581">
        <attvalues>
          <attvalue for="0" value="PaSS: Parallel Speculative Sampling" />
          <attvalue for="1" value="  Scaling the size of language models to tens of billions of parameters has led&#10;to impressive performance on a wide range of tasks. At generation, these models&#10;are used auto-regressively, requiring a forward pass for each generated token,&#10;and thus reading the full set of parameters from memory. This memory access&#10;forms the primary bottleneck for generation and it worsens as the model size&#10;increases. Moreover, executing a forward pass for multiple tokens in parallel&#10;often takes nearly the same time as it does for just one token. These two&#10;observations lead to the development of speculative sampling, where a second&#10;smaller model is used to draft a few tokens, that are then validated or&#10;rejected using a single forward pass of the large model. Unfortunately, this&#10;method requires two models that share the same tokenizer and thus limits its&#10;adoption. As an alternative, we propose to use parallel decoding as a way to&#10;draft multiple tokens from a single model with no computational cost, nor the&#10;need for a second model. Our approach only requires an additional input token&#10;that marks the words that will be generated simultaneously. We show promising&#10;performance (up to $30\%$ speed-up) while requiring only as few as $O(d_{emb})$&#10;additional parameters.&#10;" />
          <attvalue for="2" value="&#10;Since the Transformer architecture was proposed (\cite{vaswani2023attention}), large language models have achieved impressive results across natural language processing benchmarks~(\cite{brown2020language}). However, these remarkable achievements were only made possible because of dramatic increases in the number of parameters or model sizes (\cite{brown2020language}, \cite{wei2022emergent}), resulting in considerable memory requirements and greater processing times. &#10;This problem is further exacerbated by the fact that, at inference time, transformers are used auto-regressively: a new model call is needed for each generated token. This is especially problematic due to the memory-bandwidth cost of recurrently loading the model parameters and the past keys and values tensors (\cite{shazeer2019fast}).&#10;&#10;Recently, several works~(\cite{chen2023accelerating,leviathan2023fast}) have proposed to reduce inference time &#10;by leveraging a smaller model to approximate generation from a larger model at a faster pace. &#10;The small model produces a few potential tokens, and the larger model evaluates all of the tokens at once in a single forward step. &#10;Importantly, the generation quality of the original large model is guaranteed by the rejection scheme that keeps only tokens that are generated with an identical distribution than the large model (\cite{chen2023accelerating}, \cite{leviathan2023fast}). &#10;While effective in practice, this approach requires to deploy simultaneously two models that share the same vocabulary, creating memory and running time bottlenecks.&#10;&#10;An alternative solution is to directly leverage the large model to generate multiple tokens at once, instead of generating them auto-regressively.&#10;This solution, called parallel decoding, can be implemented as a masked language model (\cite{ghazvininejad-etal-2019-mask}) or by copying the encoder input in the decoder in the context of encoder-decoder architectures (\cite{gu2018nonautoregressive}). &#10;These solutions have the advantage over speculative sampling of avoiding the need for a second model, but they require substantial changes to the Transformer architecture that make them not suitable as such for accelerating the decoding of a given pre-trained language model.&#10;&#10;In this work, we propose to combine the best of both directions in a variant of the speculative sampling that we call Parallel Speculative Sampling (PaSS). &#10;The idea is to generate candidate tokens via parallel decoding by adding a small number of ``look-ahead embeddings'' and generate output for each of these additional embeddings. &#10;This solution does not require a second model, nor modifications to the large language model.&#10;By design, our approach also generates at each step at least one token auto-regressively, guaranteeing the same loss-less quality of generations as speculating sampling methods.&#10;The memory overhead of adding the extra embeddings is $O(d_{emb})$ new weights, that need to be trained.&#10;This is several orders of magnitude smaller than any small model added by existing speculative sampling solutions. &#10;The most similar works to ours are \cite{stern2018blockwise} and \cite{medusa} where they add look-ahead classification heads instead of embeddings, leading to a worse memory overhead of $O(d_{emb}K)$ where $K$ is the vocabulary size.&#10;Additionally, \cite{stern2018blockwise} focus solely on greedy decoding, and \cite{medusa} do not guarantee a loss-less decoding. &#10;Similarly, \cite{zhang2023draft} also drop the second model, but still decode auto-regressively.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Scaling, Computer Science, Information Technology, Efficient Generation Techniques, Linguistics, Parallel Decoding Methods, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2302.01318" label="2302.01318">
        <attvalues>
          <attvalue for="0" value="Accelerating Large Language Model Decoding with Speculative Sampling" />
          <attvalue for="1" value="  We present speculative sampling, an algorithm for accelerating transformer&#10;decoding by enabling the generation of multiple tokens from each transformer&#10;call. Our algorithm relies on the observation that the latency of parallel&#10;scoring of short continuations, generated by a faster but less powerful draft&#10;model, is comparable to that of sampling a single token from the larger target&#10;model. This is combined with a novel modified rejection sampling scheme which&#10;preserves the distribution of the target model within hardware numerics. We&#10;benchmark speculative sampling with Chinchilla, a 70 billion parameter language&#10;model, achieving a 2-2.5x decoding speedup in a distributed setup, without&#10;compromising the sample quality or making modifications to the model itself.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.17192" label="2211.17192">
        <attvalues>
          <attvalue for="0" value="Fast Inference from Transformers via Speculative Decoding" />
          <attvalue for="1" value="  Inference from large autoregressive models like Transformers is slow -&#10;decoding K tokens takes K serial runs of the model. In this work we introduce&#10;speculative decoding - an algorithm to sample from autoregressive models faster&#10;without any changes to the outputs, by computing several tokens in parallel. At&#10;the heart of our approach lie the observations that (1) hard language-modeling&#10;tasks often include easier subtasks that can be approximated well by more&#10;efficient models, and (2) using speculative execution and a novel sampling&#10;method, we can make exact decoding from the large models faster, by running&#10;them in parallel on the outputs of the approximation models, potentially&#10;generating several tokens concurrently, and without changing the distribution.&#10;Our method can accelerate existing off-the-shelf models without retraining or&#10;architecture changes. We demonstrate it on T5-XXL and show a 2X-3X acceleration&#10;compared to the standard T5X implementation, with identical outputs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.03115" label="1811.03115">
        <attvalues>
          <attvalue for="0" value="Blockwise Parallel Decoding for Deep Autoregressive Models" />
          <attvalue for="1" value="  Deep autoregressive sequence-to-sequence models have demonstrated impressive&#10;performance across a wide variety of tasks in recent years. While common&#10;architecture classes such as recurrent, convolutional, and self-attention&#10;networks make different trade-offs between the amount of computation needed per&#10;layer and the length of the critical path at training time, generation still&#10;remains an inherently sequential process. To overcome this limitation, we&#10;propose a novel blockwise parallel decoding scheme in which we make predictions&#10;for multiple time steps in parallel then back off to the longest prefix&#10;validated by a scoring model. This allows for substantial theoretical&#10;improvements in generation speed when applied to architectures that can process&#10;output sequences in parallel. We verify our approach empirically through a&#10;series of experiments using state-of-the-art self-attention models for machine&#10;translation and image super-resolution, achieving iteration reductions of up to&#10;2x over a baseline greedy decoder with no loss in quality, or up to 7x in&#10;exchange for a slight decrease in performance. In terms of wall-clock time, our&#10;fastest models exhibit real-time speedups of up to 4x over standard greedy&#10;decoding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.07914" label="2311.07914">
        <attvalues>
          <attvalue for="0" value="Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey" />
          <attvalue for="1" value="  The contemporary LLMs are prone to producing hallucinations, stemming mainly&#10;from the knowledge gaps within the models. To address this critical limitation,&#10;researchers employ diverse strategies to augment the LLMs by incorporating&#10;external knowledge, aiming to reduce hallucinations and enhance reasoning&#10;accuracy. Among these strategies, leveraging knowledge graphs as a source of&#10;external information has demonstrated promising results. In this survey, we&#10;comprehensively review these knowledge-graph-based augmentation techniques in&#10;LLMs, focusing on their efficacy in mitigating hallucinations. We&#10;systematically categorize these methods into three overarching groups, offering&#10;methodological comparisons and performance evaluations. Lastly, this survey&#10;explores the current trends and challenges associated with these techniques and&#10;outlines potential avenues for future research in this emerging field.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) seek to emulate human intelligence through statistical training on extensive datasets~\cite{huang2022towards}. LLMs operate on input text to predict the subsequent token or word in the sequence while identifying patterns and connections between words and phrases, aiming to comprehend and generate human-like text. Due to their stochastic decoding processes, i.e., sampling the next token in the sequence, these models exhibit probabilistic behavior, potentially yielding varied outputs or predictions for the same input across different instances. Additionally, if the training data includes misinformation, biases, or inaccuracies, these flaws may be mirrored or amplified in the content produced by these models. LLMs also face challenges in accurately interpreting phrases or terms when the context is vague and resides in a knowledge gap region of the model, leading to outputs that may sound plausible but are often irrelevant or incorrect~\cite{ji2023survey, lenat2023getting}. This phenomenon, often termed &quot;hallucinations,&quot; undermines the reliability of these models~\cite{mallen2023not}.&#10;&#10;Addressing the issue of hallucinations in these models is challenging due to their inherent probabilistic nature.&#10;To effectively tackle this issue, there have been continuous research efforts in making knowledge updates and model tuning~\cite{zhang2023large, mialon2023augmented, petroni2019language}.&#10;However, adding random information does not improve the model's interpretation and reasoning capabilities. Instead, providing more granular and contextually relevant, precise external knowledge can significantly aid the model in recalling essential information~\cite{jiang2020can}.&#10;&#10;One emerging research trend is enhancing LLMs through integrating knowledge representation tools such as knowledge graphs (KGs)~\cite{mruthyunjaya2023rethinking}. Zheng et al.~\cite{zheng2023does} demonstrate that augmenting these models with comprehensive external knowledge from KGs can boost their performance and facilitate a more robust reasoning process. &#10;The strategies for enhancing LLMs with KGs can be grouped into three main categories, each uniquely contributing to the refinement of the model as shown in &#10;Figure~\ref{fig:intro}: enhancing the inference process, improving the learning mechanism, and establishing robust methods for validating the model's decisions. &#10;&#10;In this survey, we critically review KG augmentation methods used in specific stages to reduce hallucinations in LLMs and improve their performance and reliability. In Section~\ref{method}, we classify these methods into three overarching categories: (1) Knowledge-Aware Inference, (2) Knowledge-Aware Learning, and (3) Knowledge-Aware Validation. Additionally, in Section~\ref{discussion}, we evaluate the empirical efficacy of these methods and discuss current research trends, followed by suggestions for potential future research directions.&#10;&#10;Related Works: There are several related surveys which discuss LLM augmentation using external knowledge~\cite{hu2023survey, yin2022survey, alkhamissi2022review, ye2022generative, wei2021knowledge, liang2022reasoning, zhang2023large, mialon2023augmented}. However, to our knowledge, this is the first survey to exclusively focus on critically reviewing LLM augmentation methods utilizing structured knowledge from knowledge graphs. Specifically, our emphasis is on addressing hallucinations in LLMs through KG integration.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Hallucination Mitigation, Knowledge Graph Augmentation, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1912.11078" label="1912.11078">
        <attvalues>
          <attvalue for="0" value="Predictive Biases in Natural Language Processing Models: A Conceptual&#10;  Framework and Overview" />
          <attvalue for="1" value="  An increasing number of works in natural language processing have addressed&#10;the effect of bias on the predicted outcomes, introducing mitigation techniques&#10;that act on different parts of the standard NLP pipeline (data and models).&#10;However, these works have been conducted in isolation, without a unifying&#10;framework to organize efforts within the field. This leads to repetitive&#10;approaches, and puts an undue focus on the effects of bias, rather than on&#10;their origins. Research focused on bias symptoms rather than the underlying&#10;origins could limit the development of effective countermeasures. In this&#10;paper, we propose a unifying conceptualization: the predictive bias framework&#10;for NLP. We summarize the NLP literature and propose a general mathematical&#10;definition of predictive bias in NLP along with a conceptual framework,&#10;differentiating four main origins of biases: label bias, selection bias, model&#10;overamplification, and semantic bias. We discuss how past work has countered&#10;each bias origin. Our framework serves to guide an introductory overview of&#10;predictive bias in NLP, integrating existing work into a single structure and&#10;opening avenues for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.02451" label="2004.02451">
        <attvalues>
          <attvalue for="0" value="An Analysis of the Utility of Explicit Negative Examples to Improve the&#10;  Syntactic Abilities of Neural Language Models" />
          <attvalue for="1" value="  We explore the utilities of explicit negative examples in training neural&#10;language models. Negative examples here are incorrect words in a sentence, such&#10;as &quot;barks&quot; in &quot;*The dogs barks&quot;. Neural language models are commonly trained&#10;only on positive examples, a set of sentences in the training data, but recent&#10;studies suggest that the models trained in this way are not capable of robustly&#10;handling complex syntactic constructions, such as long-distance agreement. In&#10;this paper, using English data, we first demonstrate that appropriately using&#10;negative examples about particular constructions (e.g., subject-verb agreement)&#10;will boost the model's robustness on them, with a negligible loss of&#10;perplexity. The key to our success is an additional margin loss between the&#10;log-likelihoods of a correct word and an incorrect word. We then provide a&#10;detailed analysis of the trained models. One of our findings is the difficulty&#10;of object-relative clauses for RNNs. We find that even with our direct learning&#10;signals the models still suffer from resolving agreement across an&#10;object-relative clause. Augmentation of training sentences involving the&#10;constructions somewhat helps, but the accuracy still does not reach the level&#10;of subject-relative clauses. Although not directly cognitively appealing, our&#10;method can be a tool to analyze the true architectural limitation of neural&#10;models on challenging linguistic constructions.&#10;" />
          <attvalue for="2" value="&#10;\seclabel{intro}&#10;&#10;Despite not being exposed to explicit syntactic supervision, neural language models (LMs), such as recurrent neural networks, are able to generate fluent and natural sentences, suggesting that they induce syntactic knowledge about the language to some extent.&#10;However, it is still under debate whether such induced knowledge about grammar is robust enough to deal with syntactically challenging constructions such as long-distance subject-verb agreement.&#10;So far, the results for RNN language models (RNN-LMs) trained only with raw text are overall negative;&#10;prior work has reported low performance on the challenging test cases \cite{marvin-linzen:2018:EMNLP} even with the massive size of the data and model \cite{van-schijndel-EtAl:2019:EMNLP1},&#10;or argue the necessity of an architectural change to track the syntactic structure explicitly \cite{wilcox-etal-2019-structural,kuncoro-EtAl:2018:Long}.&#10;Here the task is to evaluate whether a model assigns a higher likelihood on a grammatically correct sentence (\ref{ex:correct_orc}) over an incorrect sentence (\ref{ex:incorrect_orc}) that is minimally different from the original one \cite{Q16-1037}.&#10;&#10;\begin{exe}&#10; \ex&#10; \begin{xlist}&#10; \ex[]{The author that the guards like \underline{laughs}.}&#10; \ex[*]{{The author that the guards like \underline{laugh}.}}&#10; \end{xlist}&#10;\end{exe}&#10;&#10;In this paper, to obtain a new insight into the syntactic abilities of neural LMs, in particular RNN-LMs, we perform a series of experiments under a different condition from the prior work.&#10;Specifically, we extensively analyze the performance of the models that are exposed to explicit negative examples.&#10;In this work, negative examples are the sentences or tokens that are grammatically incorrect, such as (\ref{ex:incorrect_orc}) above.&#10;&#10;Since these negative examples provide a direct learning signal on the task at test time it may not be very surprising if the task performance goes up.&#10;We acknowledge this, and argue that our motivation for this setup is to deepen understanding, in particular the limitation or the capacity of the current architectures, which we expect can be reached with such strong supervision.&#10;Another motivation is engineering:&#10;we could exploit negative examples in different ways, and establishing a better way will be of practical importance toward building an LM or generator that can be robust on particular linguistic constructions.&#10;&#10;The first research question we pursue is about this latter point:&#10;what is a better method to utilize negative examples that help LMs to acquire robustness on the target syntactic constructions?&#10;Regarding this point, we find that adding additional token-level loss trying to guarantee a margin between log-probabilities for the correct and incorrect words (e.g., $\log p(\textrm{laughs} | h)$ and $\log p(\textrm{laugh} | h)$ for (\ref{ex:correct_orc})) is superior to the alternatives.&#10;On the test set of \cite{marvin-linzen:2018:EMNLP}, we show that LSTM language models (LSTM-LMs) trained by this loss reach near perfect level on most syntactic constructions for which we create negative examples, with only a slight increase of perplexity about 1.0 point.&#10;&#10;Past work conceptually similar to us is \cite{enguehard-etal-2017-exploring}, which, while not directly exploiting negative examples, trains an LM with additional explicit supervision signals to the evaluation task.&#10;They hypothesize that LSTMs do have enough capacity to acquire robust syntactic abilities but the learning signals given by the raw text are weak, and show that multi-task learning with a binary classification task to predict the upcoming verb form (singular or plural) helps models aware of the target syntax (subject-verb agreement).&#10;Our experiments basically confirm and strengthen this argument, with even stronger learning signals from negative examples, and we argue this allows us to evaluate the true capacity of the current architectures.&#10;In our experiments (Section~\ref{exp}), we show that our margin loss achieves higher syntactic performance than their multi-task learning.&#10;&#10;Another relevant work on the capacity of LSTM-LMs is \cite{kuncoro-etal-2019-scalable}, which shows that by distilling from syntactic LMs \cite{dyer-EtAl:2016:N16-1}, LSTM-LMs can improve their robustness on various agreement phenomena.&#10;We show that our LMs with the margin loss outperform theirs in most of the aspects, further strengthening the argument about a stronger capacity of LSTM-LMs.&#10;&#10;The latter part of this paper is a detailed analysis of the trained models and introduced losses.&#10;Our second question is about the true {limitation} of LSTM-LMs:&#10;are there still any syntactic constructions that the models cannot handle robustly even with our direct learning signals?&#10;This question can be seen as a fine-grained one raised by \cite{enguehard-etal-2017-exploring} with a stronger tool and improved evaluation metric.&#10;Among tested constructions, we find that syntactic agreement across an object relative clause (RC) is challenging.&#10;To inspect whether this is due to the architectural limitation, we train another LM on a dataset, on which we unnaturally augment sentences involving object RCs.&#10;Since it is known that object RCs are relatively rare compared to subject RCs \cite{hale-2001-probabilistic},&#10;frequency may be the main reason for the lower performance.&#10;Interestingly, even when increasing the number of sentences with an object RC by eight times (more than twice of sentences with a subject RC), the accuracy does not reach the same level as agreement across a subject RC.&#10;This result suggests an inherent difficulty in tracking a syntactic state across an object RC for sequential neural architectures.&#10;&#10;We finally provide an ablation study to understand the encoded linguistic knowledge in the models learned with the help of our method.&#10;We experiment under reduced supervision at two different levels:&#10;(1) at a lexical level, by not giving negative examples on verbs that appear in the test set;&#10;(2) at a construction level, by not giving negative examples about a particular construction, e.g., verbs after a subject RC.&#10;We observe no huge score drops by both.&#10;This suggests that our learning signals at a lexical level (negative words) strengthen the abstract syntactic knowledge about the target constructions, and also that the models can generalize the knowledge acquired by negative examples to similar constructions for which negative examples are not explicitly given.&#10;The result also implies that negative examples do not have to be complete and can be noisy, which will be appealing from an engineering perspective.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Training Data Optimization, Linguistics, Cognitive Science, Syntactic Construction Analysis, Neural Language Models" />
        </attvalues>
      </node>
      <node id="1706.03542" label="1706.03542">
        <attvalues>
          <attvalue for="0" value="Exploring the Syntactic Abilities of RNNs with Multi-task Learning" />
          <attvalue for="1" value="  Recent work has explored the syntactic abilities of RNNs using the&#10;subject-verb agreement task, which diagnoses sensitivity to sentence structure.&#10;RNNs performed this task well in common cases, but faltered in complex&#10;sentences (Linzen et al., 2016). We test whether these errors are due to&#10;inherent limitations of the architecture or to the relatively indirect&#10;supervision provided by most agreement dependencies in a corpus. We trained a&#10;single RNN to perform both the agreement task and an additional task, either&#10;CCG supertagging or language modeling. Multi-task training led to significantly&#10;lower error rates, in particular on complex sentences, suggesting that RNNs&#10;have the ability to evolve more sophisticated syntactic representations than&#10;shown before. We also show that easily available agreement training data can&#10;improve performance on other syntactic tasks, in particular when only a limited&#10;amount of training data is available for those tasks. The multi-task paradigm&#10;can also be leveraged to inject grammatical knowledge into language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.06438" label="1906.06438">
        <attvalues>
          <attvalue for="0" value="Scalable Syntax-Aware Language Models Using Knowledge Distillation" />
          <attvalue for="1" value="  Prior work has shown that, on small amounts of training data, syntactic&#10;neural language models learn structurally sensitive generalisations more&#10;successfully than sequential language models. However, their computational&#10;complexity renders scaling difficult, and it remains an open question whether&#10;structural biases are still necessary when sequential models have access to&#10;ever larger amounts of training data. To answer this question, we introduce an&#10;efficient knowledge distillation (KD) technique that transfers knowledge from a&#10;syntactic language model trained on a small corpus to an LSTM language model,&#10;hence enabling the LSTM to develop a more structurally sensitive representation&#10;of the larger training data it learns from. On targeted syntactic evaluations,&#10;we find that, while sequential LSTMs perform much better than previously&#10;reported, our proposed technique substantially improves on this baseline,&#10;yielding a new state of the art. Our findings and analysis affirm the&#10;importance of structural biases, even in models that learn from large amounts&#10;of data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.05021" label="2009.05021">
        <attvalues>
          <attvalue for="0" value="Investigating Gender Bias in BERT" />
          <attvalue for="1" value="  Contextual language models (CLMs) have pushed the NLP benchmarks to a new&#10;height. It has become a new norm to utilize CLM provided word embeddings in&#10;downstream tasks such as text classification. However, unless addressed, CLMs&#10;are prone to learn intrinsic gender-bias in the dataset. As a result,&#10;predictions of downstream NLP models can vary noticeably by varying gender&#10;words, such as replacing &quot;he&quot; to &quot;she&quot;, or even gender-neutral words. In this&#10;paper, we focus our analysis on a popular CLM, i.e., BERT. We analyse the&#10;gender-bias it induces in five downstream tasks related to emotion and&#10;sentiment intensity prediction. For each task, we train a simple regressor&#10;utilizing BERT's word embeddings. We then evaluate the gender-bias in&#10;regressors using an equity evaluation corpus. Ideally and from the specific&#10;design, the models should discard gender informative features from the input.&#10;However, the results show a significant dependence of the system's predictions&#10;on gender-particular words and phrases. We claim that such biases can be&#10;reduced by removing genderspecific features from word embedding. Hence, for&#10;each layer in BERT, we identify directions that primarily encode gender&#10;information. The space formed by such directions is referred to as the gender&#10;subspace in the semantic space of word embeddings. We propose an algorithm that&#10;finds fine-grained gender directions, i.e., one primary direction for each BERT&#10;layer. This obviates the need of realizing gender subspace in multiple&#10;dimensions and prevents other crucial information from being omitted.&#10;Experiments show that removing embedding components in such directions achieves&#10;great success in reducing BERT-induced bias in the downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;While a lot has been studied, identified, and mitigated when it comes to gender-bias in static word embeddings \cite{bolukbasi2016man, zhao-etal-2018-learning, caliskan2017semantics, zhao-etal-2018-gender}, very few recent works studied gender-bias in contextualized settings. We adapt the intuition of possible gender subspace in $\BERT$ from \cite{bolukbasi2016man}, which studied the existence of gender directions in static word embeddings. \cite{zhao-etal-2019-gender, basta-etal-2019-evaluating, gonen-goldberg-2019-lipstick} focused their study on ELMo. \cite{kurita-etal-2019-measuring} provided a template-based approach to quantify bias in BERT. \cite{sahlgren-olsson-2019-gender} studied bias in both contextualized and non-contextualized Swedish embeddings.&#10;&#10;To the best of our knowledge, we are the first to identify gender-bias in BERT by analysing its impact on downstream tasks. We propose a novel algorithm to identify fine-grained gender directions to minimize the exclusion of important semantic information. Empirically, the elimination of embedding components in gender directions proves to be significantly reducing gender-bias in the tasks under study.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2407.07638" label="2407.07638">
        <attvalues>
          <attvalue for="0" value="Tuning Vision-Language Models with Candidate Labels by Prompt Alignment" />
          <attvalue for="1" value="  Vision-language models (VLMs) can learn high-quality representations from a&#10;large-scale training dataset of image-text pairs. Prompt learning is a popular&#10;approach to fine-tuning VLM to adapt them to downstream tasks. Despite the&#10;satisfying performance, a major limitation of prompt learning is the demand for&#10;labelled data. In real-world scenarios, we may only obtain candidate labels&#10;(where the true label is included) instead of the true labels due to data&#10;privacy or sensitivity issues. In this paper, we provide the first study on&#10;prompt learning with candidate labels for VLMs. We empirically demonstrate that&#10;prompt learning is more advantageous than other fine-tuning methods, for&#10;handling candidate labels. Nonetheless, its performance drops when the label&#10;ambiguity increases. In order to improve its robustness, we propose a simple&#10;yet effective framework that better leverages the prior knowledge of VLMs to&#10;guide the learning process with candidate labels. Specifically, our framework&#10;disambiguates candidate labels by aligning the model output with the mixed&#10;class posterior jointly predicted by both the learnable and the handcrafted&#10;prompt. Besides, our framework can be equipped with various off-the-shelf&#10;training objectives for learning with candidate labels to further improve their&#10;performance. Extensive experiments demonstrate the effectiveness of our&#10;proposed framework.&#10;" />
          <attvalue for="2" value="&#10;&#10;Vision-language models (VLMs) such as CLIP \cite{radford2021clip}, ALIGN \cite{pmlr-v139-align}, and Coca \cite{yu2022coca} have become excellent base models in multiple domains, most of which employ a dual-encoder architecture to align the natural images with descriptive texts. &#10;Remarkably, this special training pattern has endowed VLMs with superior zero-shot transfer performance on visual recognition tasks. In specific, during the inference, the pre-trained text encoder receives inputs in the form of man-crafted prompts, \eg, ``a photo of \textlessCLS\textgreater.''. Subsequently, all the generated textual embeddings are matched with the visual embedding obtained from the image encoder to predict the image category. &#10;However, the powerful zero-shot ability of VLMs was shown to be heavily dependent on the choice of handcrafted prompts, which needs substantial efforts and professional domain knowledge to design \cite{zhou2022coop}.&#10;To avoid the manual design of the prompts, prompt learning \cite{zhou2022coop} is proposed, which treats the textual prompt as additional learnable parameters and tunes them while keeping all the original parameters of the pre-trained model fixed. Later, the concept of the prompt is extended to visual prompt \cite{jia2022visual} and multi-modal prompt \cite{khattak2023maple} in VLMs. &#10;Overall, there has been increasing attention paid to prompt learning due to its potential to perform significantly better than zero-shot transfer with a few sets of labeled data \cite{zhou2022coop,zhou2022cocoop,zhu2023gradalign,khattak2023maple,jia2022visual}.&#10;&#10;While prompt learning has demonstrated effectiveness and efficiency in few-shot supervised learning, the true labels must be provided for the training data used in prompt learning. &#10;This is a significant defect and will limit the usage of prompt learning in various real-world scenarios because we may be unable to collect accurate labels due to security issues or labeling difficulties. Fortunately, obtaining a set of candidate labels that includes the true label in these situations is easier.&#10;For example, as shown in Figure \ref{fig:cl_illustration}, it is challenging to determine which is the true label from `Hawk', `Eagle', and `Falcon', hence all these three labels can be considered as candidate labels during the annotation process of this bird recognition task.&#10;As we see, learning with only candidate labels (also widely known as partial-label learning (PLL) \cite{wang2021pico,wu2022revisiting,lv2020progressive,feng2020provably}) is practically significant, which also has arisen in many vital applications such as web mining \cite{luo2010learning}, online annotation \cite{tang2017confidence} and ecoinformatics \cite{liu2012conditional}. Nevertheless, existing PLL methods primarily focus on training a model from scratch, and the effectiveness of PLL in the new training paradigm called prompt learning remains unconfirmed. To bridge this research gap, we, for the first time, explore the validity and potential approaches for prompt learning with candidate labels.&#10;&#10;This paper empirically shows that prompt learning combined with the prevailing PLL training objectives in a vanilla way can learn from candidate labels. However, as experimentally suggested, if the candidate labels become more ambiguous, the model's performance will drop significantly. Fortunately, prompt learning is still more robust than linear probe \cite{radford2021clip}, another tuning method that trains a linear classifier on top of a frozen pre-trained model. We conjecture that the reason lies in the prior knowledge brought by the fixed class token, which can keep VLMs from over-fitting to the false-positive labels in the candidate label set and provide VLMs with preferred zero-shot ability, thus mitigating the error accumulation problem \cite{yao2021network} in PLL when the label ambiguity increases. &#10;&#10;Therefore, to enhance the robustness of prompt learning with candidate labels, we propose a simple yet effective framework incorporating the handcrafted prompt to distill the model with more comprehensive prior knowledge.&#10;Concretely, it dynamically mixes the class posteriors predicted by both the handcrafted and learnable prompt, followed by aligning the mixed class posterior with the model output.&#10;Besides, due to the simplicity and flexibility of our framework, it can cooperate with any current PLL training objectives. With our framework, the overall performance of various PLL training objectives has improved by a large margin when tuning VLMs with candidate labels.&#10;&#10;Our main contributions can be summarized as follows: &#10;\begin{itemize}&#10;\item We provide the first study on the scenario when vision-language models are tuned with only candidate labels.&#10;\item We demonstrate empirically and explain that prompt learning combined with PLL training objectives in a vanilla way can learn from candidate labels but is not robust when the label ambiguity is high.&#10;\item A framework is proposed to enhance the robustness of prompt learning with candidate labels by aligning the model output with the dynamically mixed prediction by both the handcrafted and the learnable prompt.&#10;\item Extensive experiments demonstrate the effectiveness of our framework.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Prompt Learning Methods, Computer Science, Machine Learning, Linguistics, Cognitive Science, Candidate Label Handling, Computer Vision, Natural Language Processing, Vision Language Models" />
        </attvalues>
      </node>
      <node id="2102.05918" label="2102.05918">
        <attvalues>
          <attvalue for="0" value="Scaling Up Visual and Vision-Language Representation Learning With Noisy&#10;  Text Supervision" />
          <attvalue for="1" value="  Pre-trained representations are becoming crucial for many NLP and perception&#10;tasks. While representation learning in NLP has transitioned to training on raw&#10;text without human annotations, visual and vision-language representations&#10;still rely heavily on curated training datasets that are expensive or require&#10;expert knowledge. For vision applications, representations are mostly learned&#10;using datasets with explicit class labels such as ImageNet or OpenImages. For&#10;vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all&#10;involve a non-trivial data collection (and cleaning) process. This costly&#10;curation process limits the size of datasets and hence hinders the scaling of&#10;trained models. In this paper, we leverage a noisy dataset of over one billion&#10;image alt-text pairs, obtained without expensive filtering or post-processing&#10;steps in the Conceptual Captions dataset. A simple dual-encoder architecture&#10;learns to align visual and language representations of the image and text pairs&#10;using a contrastive loss. We show that the scale of our corpus can make up for&#10;its noise and leads to state-of-the-art representations even with such a simple&#10;learning scheme. Our visual representation achieves strong performance when&#10;transferred to classification tasks such as ImageNet and VTAB. The aligned&#10;visual and language representations enables zero-shot image classification and&#10;also set new state-of-the-art results on Flickr30K and MSCOCO image-text&#10;retrieval benchmarks, even when compared with more sophisticated&#10;cross-attention models. The representations also enable cross-modality search&#10;with complex text and text + image queries.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.01134" label="2109.01134">
        <attvalues>
          <attvalue for="0" value="Learning to Prompt for Vision-Language Models" />
          <attvalue for="1" value="  Large pre-trained vision-language models like CLIP have shown great potential&#10;in learning representations that are transferable across a wide range of&#10;downstream tasks. Different from the traditional representation learning that&#10;is based mostly on discretized labels, vision-language pre-training aligns&#10;images and texts in a common feature space, which allows zero-shot transfer to&#10;a downstream task via prompting, i.e., classification weights are synthesized&#10;from natural language describing classes of interest. In this work, we show&#10;that a major challenge for deploying such models in practice is prompt&#10;engineering, which requires domain expertise and is extremely time-consuming --&#10;one needs to spend a significant amount of time on words tuning since a slight&#10;change in wording could have a huge impact on performance. Inspired by recent&#10;advances in prompt learning research in natural language processing (NLP), we&#10;propose Context Optimization (CoOp), a simple approach specifically for&#10;adapting CLIP-like vision-language models for downstream image recognition.&#10;Concretely, CoOp models a prompt's context words with learnable vectors while&#10;the entire pre-trained parameters are kept fixed. To handle different image&#10;recognition tasks, we provide two implementations of CoOp: unified context and&#10;class-specific context. Through extensive experiments on 11 datasets, we&#10;demonstrate that CoOp requires as few as one or two shots to beat hand-crafted&#10;prompts with a decent margin and is able to gain significant improvements over&#10;prompt engineering with more shots, e.g., with 16 shots the average gain is&#10;around 15% (with the highest reaching over 45%). Despite being a learning-based&#10;approach, CoOp achieves superb domain generalization performance compared with&#10;the zero-shot model using hand-crafted prompts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.12119" label="2203.12119">
        <attvalues>
          <attvalue for="0" value="Visual Prompt Tuning" />
          <attvalue for="1" value="  The current modus operandi in adapting pre-trained models involves updating&#10;all the backbone parameters, ie, full fine-tuning. This paper introduces Visual&#10;Prompt Tuning (VPT) as an efficient and effective alternative to full&#10;fine-tuning for large-scale Transformer models in vision. Taking inspiration&#10;from recent advances in efficiently tuning large language models, VPT&#10;introduces only a small amount (less than 1% of model parameters) of trainable&#10;parameters in the input space while keeping the model backbone frozen. Via&#10;extensive experiments on a wide variety of downstream recognition tasks, we&#10;show that VPT achieves significant performance gains compared to other&#10;parameter efficient tuning protocols. Most importantly, VPT even outperforms&#10;full fine-tuning in many cases across model capacities and training data&#10;scales, while reducing per-task storage cost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.03117" label="2210.03117">
        <attvalues>
          <attvalue for="0" value="MaPLe: Multi-modal Prompt Learning" />
          <attvalue for="1" value="  Pre-trained vision-language (V-L) models such as CLIP have shown excellent&#10;generalization ability to downstream tasks. However, they are sensitive to the&#10;choice of input text prompts and require careful selection of prompt templates&#10;to perform well. Inspired by the Natural Language Processing (NLP) literature,&#10;recent CLIP adaptation approaches learn prompts as the textual inputs to&#10;fine-tune CLIP for downstream tasks. We note that using prompting to adapt&#10;representations in a single branch of CLIP (language or vision) is sub-optimal&#10;since it does not allow the flexibility to dynamically adjust both&#10;representation spaces on a downstream task. In this work, we propose&#10;Multi-modal Prompt Learning (MaPLe) for both vision and language branches to&#10;improve alignment between the vision and language representations. Our design&#10;promotes strong coupling between the vision-language prompts to ensure mutual&#10;synergy and discourages learning independent uni-modal solutions. Further, we&#10;learn separate prompts across different early stages to progressively model the&#10;stage-wise feature relationships to allow rich context learning. We evaluate&#10;the effectiveness of our approach on three representative tasks of&#10;generalization to novel classes, new target datasets and unseen domain shifts.&#10;Compared with the state-of-the-art method Co-CoOp, MaPLe exhibits favorable&#10;performance and achieves an absolute gain of 3.45% on novel classes and 2.72%&#10;on overall harmonic-mean, averaged over 11 diverse image recognition datasets.&#10;Our code and pre-trained models are available at&#10;https://github.com/muzairkhattak/multimodal-prompt-learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.14865" label="2205.14865">
        <attvalues>
          <attvalue for="0" value="Prompt-aligned Gradient for Prompt Tuning" />
          <attvalue for="1" value="  Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we&#10;can craft a zero-shot classifier by &quot;prompt&quot;, e.g., the confidence score of an&#10;image being &quot;[CLASS]&quot; can be obtained by using the VLM provided similarity&#10;measure between the image and the prompt sentence &quot;a photo of a [CLASS]&quot;.&#10;Therefore, prompt shows a great potential for fast adaptation of VLMs to&#10;downstream tasks if we fine-tune the prompt-based similarity measure. However,&#10;we find a common failure that improper fine-tuning may not only undermine the&#10;prompt's inherent prediction for the task-related classes, but also for other&#10;classes in the VLM vocabulary. Existing methods still address this problem by&#10;using traditional anti-overfitting techniques such as early stopping and data&#10;augmentation, which lack a principled solution specific to prompt. We present&#10;Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from&#10;forgetting the the general knowledge learned from VLMs. In particular, ProGrad&#10;only updates the prompt whose gradient is aligned (or non-conflicting) to the&#10;&quot;general direction&quot;, which is represented as the gradient of the KL loss of the&#10;pre-defined prompt prediction. Extensive experiments demonstrate the stronger&#10;few-shot generalization ability of ProGrad over state-of-the-art prompt tuning&#10;methods. Codes are available at https://github.com/BeierZhu/Prompt-align.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.08053" label="2002.08053">
        <attvalues>
          <attvalue for="0" value="Progressive Identification of True Labels for Partial-Label Learning" />
          <attvalue for="1" value="  Partial-label learning (PLL) is a typical weakly supervised learning problem,&#10;where each training instance is equipped with a set of candidate labels among&#10;which only one is the true label. Most existing methods elaborately designed&#10;learning objectives as constrained optimizations that must be solved in&#10;specific manners, making their computational complexity a bottleneck for&#10;scaling up to big data. The goal of this paper is to propose a novel framework&#10;of PLL with flexibility on the model and optimization algorithm. More&#10;specifically, we propose a novel estimator of the classification risk,&#10;theoretically analyze the classifier-consistency, and establish an estimation&#10;error bound. Then we propose a progressive identification algorithm for&#10;approximately minimizing the proposed risk estimator, where the update of the&#10;model and identification of true labels are conducted in a seamless manner. The&#10;resulting algorithm is model-independent and loss-independent, and compatible&#10;with stochastic optimization. Thorough experiments demonstrate it sets the new&#10;state of the art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.11919" label="2002.11919">
        <attvalues>
          <attvalue for="0" value="Network Cooperation with Progressive Disambiguation for Partial Label&#10;  Learning" />
          <attvalue for="1" value="  Partial Label Learning (PLL) aims to train a classifier when each training&#10;instance is associated with a set of candidate labels, among which only one is&#10;correct but is not accessible during the training phase. The common strategy&#10;dealing with such ambiguous labeling information is to disambiguate the&#10;candidate label sets. Nonetheless, existing methods ignore the disambiguation&#10;difficulty of instances and adopt the single-trend training mechanism. The&#10;former would lead to the vulnerability of models to the false positive labels&#10;and the latter may arouse error accumulation problem. To remedy these two&#10;drawbacks, this paper proposes a novel approach termed &quot;Network Cooperation&#10;with Progressive Disambiguation&quot; (NCPD) for PLL. Specifically, we devise a&#10;progressive disambiguation strategy of which the disambiguation operations are&#10;performed on simple instances firstly and then gradually on more complicated&#10;ones. Therefore, the negative impacts brought by the false positive labels of&#10;complicated instances can be effectively mitigated as the disambiguation&#10;ability of the model has been strengthened via learning from the simple&#10;instances. Moreover, by employing artificial neural networks as the backbone,&#10;we utilize a network cooperation mechanism which trains two networks&#10;collaboratively by letting them interact with each other. As two networks have&#10;different disambiguation ability, such interaction is beneficial for both&#10;networks to reduce their respective disambiguation errors, and thus is much&#10;better than the existing algorithms with single-trend training process.&#10;Extensive experimental results on various benchmark and practical datasets&#10;demonstrate the superiority of our NCPD to other state-of-the-art PLL methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.06366" label="2110.06366">
        <attvalues>
          <attvalue for="0" value="Time Masking for Temporal Language Models" />
          <attvalue for="1" value="  Our world is constantly evolving, and so is the content on the web.&#10;Consequently, our languages, often said to mirror the world, are dynamic in&#10;nature. However, most current contextual language models are static and cannot&#10;adapt to changes over time. In this work, we propose a temporal contextual&#10;language model called TempoBERT, which uses time as an additional context of&#10;texts. Our technique is based on modifying texts with temporal information and&#10;performing time masking - specific masking for the supplementary time&#10;information. We leverage our approach for the tasks of semantic change&#10;detection and sentence time prediction, experimenting on diverse datasets in&#10;terms of time, size, genre, and language. Our extensive evaluation shows that&#10;both tasks benefit from exploiting time masking.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Temporal Language Models&#10;Language models are usually pretrained on corpora derived from a snapshot of the web crawled at a specific moment in time~\cite{devlin2019bert,liu2019roberta}.&#10;This ``static'' nature of training prevents LMs from adapting to time and generalizing temporally~\cite{rottger2021temporal,lazaridou2021pitfalls,hombaiah2021dynamic,dhingra2021time}.&#10;\cite{hombaiah2021dynamic} performed incremental training to better handle continuously evolving web content.&#10;\cite{dhingra2021time} experimented with temporal language models for question answering. The authors focused on probing LMs for factual knowledge that changes over time, and showed that conditioning the temporal LM on the temporal context of the text data improves memorization of facts. &#10;Others focused on document classification by using word-level temporal embeddings~\cite{huang2019neural} and adapting pretrained BERT models to domain and time~\cite{rottger2021temporal}.&#10;&#10;In this work, we focus on the novel concept of temporal contextual representation. We leverage our proposed model for the tasks of semantic change detection and sentence time prediction, which were not studied in the above papers. &#10;Furthermore, we learn temporal contexts for word embeddings, introduce the concept of time masking, and demonstrate its benefits for temporal LMs.&#10;&#10;Semantic Change Detection&#10;Semantic change detection is the task of determining whether and to what extent the meaning of a set of target words has changed over time, with the help of time-annotated corpora~\cite{kutuzov2018diachronic,tahmasebi2018survey}. This task is often addressed using distributional semantic models: time sensitive word representations are learned and then compared between different time periods~\cite{hamilton2016diachronic,bamler2017dynamic,dubossarsky2019time}.&#10;\cite{gonen2020simple} used a simple nearest-neighbors-based approach to detect semantically-changed words.&#10;\cite{del2019short} measured changes of word meaning in online Reddit communities by employing an incremental fine-tuning approach. \cite{montariol2021scalable} used statistical tools to detect semantic change in a scalable way using clusters.&#10;Others attempted to simultaneously learn time-aware embeddings over all time periods and resolve the alignment problem using regularization~\cite{yao2018dynamic}, modeling word usage as a function of time~\cite{rosenfeld2018deep}, Bayesian skip-gram model with latent time series~\cite{bamler2017dynamic}, and exponential family embeddings~\cite{rudolph2018dynamic}.&#10;&#10;In all of these methods, each word has a single representation for each time period, which limits their sensitivity and interpretability. &#10;Recent contextualized architectures allow for overcoming this limitation by taking sentential context into account when inferring word token representations. Indeed, such architectures were applied to diachronic semantic change detection~\cite{hu2019diachronic,giulianelli2020analysing,martinc2020leveraging,martinc2020capturing,laicher2021explaining}. While all these studies used language models by aggregating the information from the set of the token embeddings, we directly exploit their contextual properties by integrating time directly into their training process.&#10;&#10;Sentence Time Prediction&#10;In the last decade, the exploitation of temporal information to improve document search and exploration received a notable attention from the Information Retrieval community. Specifically, in this context, Temporal Information Extraction is crucial to support retrieval systems~\cite{campos2014survey,kanhabua2016temporal}.&#10;Predicting the time of textual units (e.g., documents, posts, or sentences) based on their content is an important task~\cite{de2005t,kumar2011supervised,vashishth2018dating,savov2021predicting} with numerous approaches. &#10;Most existing work uses documents as textual units: &#10;One of the first studies to model temporal information for the automatic dating of documents is~\cite{de2005t}, who used a statistical LM based on word usage statistics over time. &#10;Several papers extended this model with temporal entropy~\cite{kanhabua2008improving}, KL divergence~\cite{kumar2011supervised}, and n-grams~\cite{jatowt2017interactive}. &#10;\cite{niculae2014temporal} tackled this task as a ranking problem. As for deep neural models for document dating, \cite{vashishth2018dating} used a Graph Convolution Network together with the syntactic and temporal structure of the document, and \cite{ray2018ad3} used an attention-based neural model.&#10;Recently, \cite{savov2021predicting} used Ordinal Classification to predict the time of each sentence individually and then aggregated the scores to document level.&#10;&#10;In this work, we focus on dating sentences. Sentences are often significantly shorter than documents, making this task potentially harder than dating documents, due to the limited context. In addition, sentence dating can be used as a building block of document dating, as was done in~\cite{savov2021predicting}. We use time masking to directly predict the writing time of sentences.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Temporal Context Analysis, Artificial Intelligence, Natural Language Processing, Language Model Evolution, Dynamic Text Processing" />
        </attvalues>
      </node>
      <node id="2103.05081" label="2103.05081">
        <attvalues>
          <attvalue for="0" value="A Parallelizable Lattice Rescoring Strategy with Neural Language Models" />
          <attvalue for="1" value="  This paper proposes a parallel computation strategy and a posterior-based&#10;lattice expansion algorithm for efficient lattice rescoring with neural&#10;language models (LMs) for automatic speech recognition. First, lattices from&#10;first-pass decoding are expanded by the proposed posterior-based lattice&#10;expansion algorithm. Second, each expanded lattice is converted into a minimal&#10;list of hypotheses that covers every arc. Each hypothesis is constrained to be&#10;the best path for at least one arc it includes. For each lattice, the neural LM&#10;scores of the minimal list are computed in parallel and are then integrated&#10;back to the lattice in the rescoring stage. Experiments on the Switchboard&#10;dataset show that the proposed rescoring strategy obtains comparable&#10;recognition performance and generates more compact lattices than a competitive&#10;baseline method. Furthermore, the parallel rescoring method offers more&#10;flexibility by simplifying the integration of PyTorch-trained neural LMs for&#10;lattice rescoring with Kaldi.&#10;" />
          <attvalue for="2" value="&#10;Neural language models (LMs), including long short-term memory (LSTM) and Transformer based ones, have significantly improved performance over $n$-gram LMs in automatic speech recognition (ASR)~\cite{mikolov2010recurrent,chen2015recurrent,xu2018neural,zeghidour2018fully, synnaeve2019end, irie2019language,li2020empirical}. Since it is challenging for one-pass decoding with a neural LM to obtain competitive performance with lower latency than a two-pass approach~\cite{hori2014real,shi2014efficient,sundermeyer2015feedforward,beck2019lstm,jorge2020lstm}, a widely adopted way is still to use neural LMs to rescore $N$-best hypotheses (alternative word-sequences) or lattices that are decoded with an $n$-gram LM~\cite{deoras2011fast,sundermeyer2014lattice,liu2014efficient,sundermeyer2015feedforward,liu2016two,chen2017future,kumar2017lattice,xu2018pruned}. A lattice is a compact representation of the hypothesis space for an utterance. $N$-best hypotheses only cover a small subspace. Thus, lattice rescoring usually outperforms $N$-best rescoring.&#10;&#10;The key for lattice rescoring is to balance accuracy and efficiency since exact rescoring is not practical because it involves expanding a lattice into a linear or prefix tree structure and rescoring each hypothesis. A major speed bottleneck in lattice rescoring using a neural LM is the LM evaluation. Neural LM probabilities are usually computed on-the-fly and sequentially among hypotheses in a lattice during lattice traversal~\cite{sundermeyer2014lattice,liu2016two,xu2018pruned}. Though caching computed probabilities~\cite{liu2016two} or pruning-based methods~\cite{sundermeyer2014lattice,xu2018pruned} can reduce the number of evaluations, the sequential order of LM evaluation in a lattice is inefficient. The process can be accelerated significantly by evaluating multiple hypotheses in parallel. However, given the graph structure of lattices, taking advantage of such speedup is challenging, especially for lattice rescoring methods that perform expansion and rescoring simultaneously.&#10;To enable batch computation, we convert a lattice into a minimal list of hypotheses that satisfy two conditions. First, every arc should be included in at least one hypothesis. Second, each hypothesis is the best path for at least one arc it contains. Computed neural LM scores are integrated back into the lattice for rescoring where score refer to negative log probabilities.&#10;&#10;Lattice rescoring usually involves lattice expansion. Performing rescoring without changing the lattice structure is feasible, but it is generally not as good as with expansion~\cite{sundermeyer2014lattice}. To prevent expanded lattices from being too large, equivalence estimation of history states and pruning-based methods have been proposed~\cite{sundermeyer2014lattice,liu2014efficient,sundermeyer2015feedforward,liu2016two,chen2017future,xu2018pruned}. For example, an $n$-gram approximation method restricts lattice size by merging history states that share $(n-1)$ most recent words. But $n$-gram approximation based expansion method may sacrifice accuracy and waste computation on less likely paths. The general goal of lattice expansion is to make arcs on relatively probable paths have unique histories so that neural LM scores for them can be exact. To this end, we propose a new lattice expansion method that expands arcs only when their posteriors are larger than a threshold. Effectively, only more probable arcs are expanded so that arcs on sufficiently likely paths tend to have unique histories.&#10;&#10;In summary, we propose an efficient lattice rescoring strategy that enables parallel computation of neural LM scores within a lattice. The strategy mainly involves operations such as posterior-based lattice expansion and lattice-to-list conversion using a proposed path cover algorithm. Furthermore, we experiment with a refined lattice rescoring strategy to further improve results. The proposed lattice-to-list conversion makes it easier to integrate neural LMs trained with PyTorch (or other tools) for efficient lattice rescoring in Kaldi~\cite{povey2011kaldi}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Parallel Computation, Artificial Intelligence, Mathematics, Speech Recognition, Neural Language Models" />
        </attvalues>
      </node>
      <node id="1812.06864" label="1812.06864">
        <attvalues>
          <attvalue for="0" value="Fully Convolutional Speech Recognition" />
          <attvalue for="1" value="  Current state-of-the-art speech recognition systems build on recurrent neural&#10;networks for acoustic and/or language modeling, and rely on feature extraction&#10;pipelines to extract mel-filterbanks or cepstral coefficients. In this paper we&#10;present an alternative approach based solely on convolutional neural networks,&#10;leveraging recent advances in acoustic models from the raw waveform and&#10;language modeling. This fully convolutional approach is trained end-to-end to&#10;predict characters from the raw waveform, removing the feature extraction step&#10;altogether. An external convolutional language model is used to decode words.&#10;On Wall Street Journal, our model matches the current state-of-the-art. On&#10;Librispeech, we report state-of-the-art performance among end-to-end models,&#10;including Deep Speech 2 trained with 12 times more acoustic data and&#10;significantly more linguistic data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.02331" label="2312.02331">
        <attvalues>
          <attvalue for="0" value="Revisiting Topic-Guided Language Models" />
          <attvalue for="1" value="  A recent line of work in natural language processing has aimed to combine&#10;language models and topic models. These topic-guided language models augment&#10;neural language models with topic models, unsupervised learning methods that&#10;can discover document-level patterns of word use. This paper compares the&#10;effectiveness of these methods in a standardized setting. We study four&#10;topic-guided language models and two baselines, evaluating the held-out&#10;predictive performance of each model on four corpora. Surprisingly, we find&#10;that none of these methods outperform a standard LSTM language model baseline,&#10;and most fail to learn good topics. Further, we train a probe of the neural&#10;language model that shows that the baseline's hidden states already encode&#10;topic information. We make public all code used for this study.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recurrent neural networks (RNNs) and LSTMs have been an important class of models in the development of methods for many tasks in natural language processing, including machine translation, summarization, and speech recognition. One of the most successful applications of these models is in language modeling, where they are effective at modeling small text corpora. Even with the advent of transformer-based language models, RNNs and LSTMs can outperform non-pretrained transformers on various small datasets \cite{melis2019mogrifier}.&#10;&#10;While powerful, RNN- and LSTM-based models struggle to capture&#10;long-range dependencies in their context history \cite{bai2018empirical, sankar-etal-2019-neural}. Additionally, they are not designed to learn interpretable structure in a corpus of documents. To this end, multiple researchers have proposed adapting these models by incorporating topic models \cite{dieng2016topicrnn, lau2017topically, rezaee2020discrete, guo2020recurrent}. The motivation for combining language models and topic models is to decouple local syntactic structure, which can be modeled by a language model, from document-level semantic concepts, which can be captured by a topic model \cite{khandelwal2018sharp, oconnor-andreas-2021-context}. The topic model component is also designed to uncover latent structure in documents. &#10;&#10;We refer to these models as topic-guided language models. Broadly, this body of research has reported good results: topic-guided language models improve next-word predictive performance and learn interpretable topics.\looseness=-1&#10;&#10;In this work, we re-investigate this class of models by evaluating four representative topic-guided language model (TGLM) papers in a unified setting. We train the models from \cite{dieng2016topicrnn, lau2017topically, rezaee2020discrete, guo2020recurrent} on three document-level corpora and evaluate their held-out perplexity. Unlike some prior work, during next-word prediction, we take care to condition the topic model component on only previous words, rather than the entire document. Moreover, we use a baseline language model that is conditioned on all previously seen document words, rather than being restricted to the current sentence \cite{lau2017topically, rezaee2020discrete, guo2020recurrent}. Additionally, we choose baseline language models with comparable model sizes to ensure valid comparisons. Our finding: no predictive improvement of TGLMs over a standard LSTM-LM baseline \cite{zaremba2014recurrent}.&#10;&#10;In order to understand why topic-guided language models offer no predictive improvement, we probe the LSTM-LM's hidden representations. A probe is a trained predictor used to measure the extent to which fitted ``black-box'' models, such as neural models, have learned specific linguistic features of the input \cite{hewitt2019designing}. The probe reveals that the LSTM-LM already encodes topic information, rendering a formal topic model component redundant.&#10;&#10;Additionally, topic-guided language models were developed to provide insight into text corpora by uncovering latent topics. This method of exploratory text analysis is commonly used in the social sciences and digital humanities \cite{griffiths2004, blei2007correlated, grimmer2013text, mohr2013introduction}. Here, we show that the topics learned by topic-guided language models are not better than a standard topic model and, for some of the models, qualitatively poor. &#10;&#10;This paper contributes to a line of reproducibility studies in machine learning that aim to evaluate competing methods in a consistent and equitable manner. These studies have uncovered instances where results are not directly comparable, as reported numbers are borrowed from prior works that used different experimental settings \cite{marie-etal-2021-scientific, hoyle2021automated}. Furthermore, they identify cases where baselines are either too weak or improperly tuned \cite{dacrema2019, nityasya-etal-2023-scientific}. We observe analogous issues within the topic-guided language modeling literature. To support transparency and reproducibility, we make public all code used in this study.&#10;&#10;Finally, we consider how these insights apply to other models. While prior work has incorporated topic model components into RNNs and LSTMs, the topic-guided language model framework is agnostic to the class of neural language model used. &#10;We conclude by discussing how the results in this paper are relevant to researchers considering incorporating topic models into more powerful neural language models, such as transformers. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Model Evaluation, Topic Modeling Techniques, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1909.01792" label="1909.01792">
        <attvalues>
          <attvalue for="0" value="Mogrifier LSTM" />
          <attvalue for="1" value="  Many advances in Natural Language Processing have been based upon more&#10;expressive models for how inputs interact with the context in which they occur.&#10;Recurrent networks, which have enjoyed a modicum of success, still lack the&#10;generalization and systematicity ultimately required for modelling language. In&#10;this work, we propose an extension to the venerable Long Short-Term Memory in&#10;the form of mutual gating of the current input and the previous output. This&#10;mechanism affords the modelling of a richer space of interactions between&#10;inputs and their context. Equivalently, our model can be viewed as making the&#10;transition function given by the LSTM context-dependent. Experiments&#10;demonstrate markedly improved generalization on language modelling in the range&#10;of 3-4 perplexity points on Penn Treebank and Wikitext-2, and 0.01-0.05 bpc on&#10;four character-based datasets. We establish a new state of the art on all&#10;datasets with the exception of Enwik8, where we close a large gap between the&#10;LSTM and Transformer models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.01271" label="1803.01271">
        <attvalues>
          <attvalue for="0" value="An Empirical Evaluation of Generic Convolutional and Recurrent Networks&#10;  for Sequence Modeling" />
          <attvalue for="1" value="  For most deep learning practitioners, sequence modeling is synonymous with&#10;recurrent networks. Yet recent results indicate that convolutional&#10;architectures can outperform recurrent networks on tasks such as audio&#10;synthesis and machine translation. Given a new sequence modeling task or&#10;dataset, which architecture should one use? We conduct a systematic evaluation&#10;of generic convolutional and recurrent architectures for sequence modeling. The&#10;models are evaluated across a broad range of standard tasks that are commonly&#10;used to benchmark recurrent networks. Our results indicate that a simple&#10;convolutional architecture outperforms canonical recurrent networks such as&#10;LSTMs across a diverse range of tasks and datasets, while demonstrating longer&#10;effective memory. We conclude that the common association between sequence&#10;modeling and recurrent networks should be reconsidered, and convolutional&#10;networks should be regarded as a natural starting point for sequence modeling&#10;tasks. To assist related work, we have made code available at&#10;http://github.com/locuslab/TCN .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.01702" label="1611.01702">
        <attvalues>
          <attvalue for="0" value="TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency" />
          <attvalue for="1" value="  In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based&#10;language model designed to directly capture the global semantic meaning&#10;relating words in a document via latent topics. Because of their sequential&#10;nature, RNNs are good at capturing the local structure of a word sequence -&#10;both semantic and syntactic - but might face difficulty remembering long-range&#10;dependencies. Intuitively, these long-range dependencies are of semantic&#10;nature. In contrast, latent topic models are able to capture the global&#10;underlying semantic structure of a document but do not account for word&#10;ordering. The proposed TopicRNN model integrates the merits of RNNs and latent&#10;topic models: it captures local (syntactic) dependencies using an RNN and&#10;global (semantic) dependencies using latent topics. Unlike previous work on&#10;contextual RNN language modeling, our model is learned end-to-end. Empirical&#10;results on word prediction show that TopicRNN outperforms existing contextual&#10;RNN baselines. In addition, TopicRNN can be used as an unsupervised feature&#10;extractor for documents. We do this for sentiment analysis on the IMDB movie&#10;review dataset and report an error rate of $6.28\%$. This is comparable to the&#10;state-of-the-art $5.91\%$ resulting from a semi-supervised approach. Finally,&#10;TopicRNN also yields sensible topics, making it a useful alternative to&#10;document models such as latent Dirichlet allocation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.04623" label="1805.04623">
        <attvalues>
          <attvalue for="0" value="Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context" />
          <attvalue for="1" value="  We know very little about how neural language models (LM) use prior&#10;linguistic context. In this paper, we investigate the role of context in an&#10;LSTM LM, through ablation studies. Specifically, we analyze the increase in&#10;perplexity when prior context words are shuffled, replaced, or dropped. On two&#10;standard datasets, Penn Treebank and WikiText-2, we find that the model is&#10;capable of using about 200 tokens of context on average, but sharply&#10;distinguishes nearby context (recent 50 tokens) from the distant history. The&#10;model is highly sensitive to the order of words within the most recent&#10;sentence, but ignores word order in the long-range context (beyond 50 tokens),&#10;suggesting the distant past is modeled only as a rough semantic field or topic.&#10;We further find that the neural caching model (Grave et al., 2017b) especially&#10;helps the LSTM to copy words from within this distant context. Overall, our&#10;analysis not only provides a better understanding of how neural LMs use their&#10;context, but also sheds light on recent success from cache-based models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.08012" label="1704.08012">
        <attvalues>
          <attvalue for="0" value="Topically Driven Neural Language Model" />
          <attvalue for="1" value="  Language models are typically applied at the sentence level, without access&#10;to the broader document context. We present a neural language model that&#10;incorporates document context in the form of a topic model-like architecture,&#10;thus providing a succinct representation of the broader document context&#10;outside of the current sentence. Experiments over a range of datasets&#10;demonstrate that our model outperforms a pure sentence-based model in terms of&#10;language model perplexity, and leads to topics that are potentially more&#10;coherent than those produced by a standard LDA topic model. Our model also has&#10;the ability to generate related sentences for a topic, providing another way to&#10;interpret topics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1409.2329" label="1409.2329">
        <attvalues>
          <attvalue for="0" value="Recurrent Neural Network Regularization" />
          <attvalue for="1" value="  We present a simple regularization technique for Recurrent Neural Networks&#10;(RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful&#10;technique for regularizing neural networks, does not work well with RNNs and&#10;LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show&#10;that it substantially reduces overfitting on a variety of tasks. These tasks&#10;include language modeling, speech recognition, image caption generation, and&#10;machine translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.15195" label="2106.15195">
        <attvalues>
          <attvalue for="0" value="Scientific Credibility of Machine Translation Research: A&#10;  Meta-Evaluation of 769 Papers" />
          <attvalue for="1" value="  This paper presents the first large-scale meta-evaluation of machine&#10;translation (MT). We annotated MT evaluations conducted in 769 research papers&#10;published from 2010 to 2020. Our study shows that practices for automatic MT&#10;evaluation have dramatically changed during the past decade and follow&#10;concerning trends. An increasing number of MT evaluations exclusively rely on&#10;differences between BLEU scores to draw conclusions, without performing any&#10;kind of statistical significance testing nor human evaluation, while at least&#10;108 metrics claiming to be better than BLEU have been proposed. MT evaluations&#10;in recent papers tend to copy and compare automatic metric scores from previous&#10;work to claim the superiority of a method or an algorithm without confirming&#10;neither exactly the same training, validating, and testing data have been used&#10;nor the metric scores are comparable. Furthermore, tools for reporting&#10;standardized metric scores are still far from being widely adopted by the MT&#10;community. After showing how the accumulation of these pitfalls leads to&#10;dubious evaluation, we propose a guideline to encourage better automatic MT&#10;evaluation along with a simple meta-evaluation scoring method to assess its&#10;credibility.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.06902" label="1907.06902">
        <attvalues>
          <attvalue for="0" value="Are We Really Making Much Progress? A Worrying Analysis of Recent Neural&#10;  Recommendation Approaches" />
          <attvalue for="1" value="  Deep learning techniques have become the method of choice for researchers&#10;working on algorithmic aspects of recommender systems. With the strongly&#10;increased interest in machine learning in general, it has, as a result, become&#10;difficult to keep track of what represents the state-of-the-art at the moment,&#10;e.g., for top-n recommendation tasks. At the same time, several recent&#10;publications point out problems in today's research practice in applied machine&#10;learning, e.g., in terms of the reproducibility of the results or the choice of&#10;the baselines when proposing new models. In this work, we report the results of&#10;a systematic analysis of algorithmic proposals for top-n recommendation tasks.&#10;Specifically, we considered 18 algorithms that were presented at top-level&#10;research conferences in the last years. Only 7 of them could be reproduced with&#10;reasonable effort. For these methods, it however turned out that 6 of them can&#10;often be outperformed with comparably simple heuristic methods, e.g., based on&#10;nearest-neighbor or graph-based techniques. The remaining one clearly&#10;outperformed the baselines but did not consistently outperform a well-tuned&#10;non-neural linear ranking method. Overall, our work sheds light on a number of&#10;potential problems in today's machine learning scholarship and calls for&#10;improved scientific practices in this area. Source code of our experiments and&#10;full results are available at:&#10;https://github.com/MaurizioFD/RecSys2019_DeepLearning_Evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.06328" label="2204.06328">
        <attvalues>
          <attvalue for="0" value="HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition" />
          <attvalue for="1" value="  Pre-training with self-supervised models, such as Hidden-unit BERT (HuBERT)&#10;and wav2vec 2.0, has brought significant improvements in automatic speech&#10;recognition (ASR). However, these models usually require an expensive&#10;computational cost to achieve outstanding performance, slowing down the&#10;inference speed. To improve the model efficiency, we introduce an early exit&#10;scheme for ASR, namely HuBERT-EE, that allows the model to stop the inference&#10;dynamically. In HuBERT-EE, multiple early exit branches are added at the&#10;intermediate layers. When the intermediate prediction of the early exit branch&#10;is confident, the model stops the inference, and the corresponding result can&#10;be returned early. We investigate the proper early exiting criterion and&#10;fine-tuning strategy to effectively perform early exiting. Experimental results&#10;on the LibriSpeech show that HuBERT-EE can accelerate the inference of the&#10;HuBERT while simultaneously balancing the trade-off between the performance and&#10;the latency.&#10;" />
          <attvalue for="2" value="&#10;Recently, self-supervised speech representation learning (speech SSL) \cite{w2v,hubert,wavlm, ils-ssl, data2vec} has achieved considerable improvements in automatic speech recognition (ASR) literature.&#10;Unlike fully-supervised learning approaches, which rely on manually annotated labels, speech SSL models can learn a meaningful speech representation by leveraging unlabeled speech data.&#10;&#10;Among the various speech SSL models, Hidden-unit BERT (HuBERT) \cite{hubert} is one of the most prominent models for speech recognition.&#10;On the LibriSpeech \cite{librispeech}, fine-tuned HuBERT using connectionist temporal classification (CTC) \cite{ctc} achieves the state-of-the-art word error rate (WER) results.&#10;However, such a model tends to have a large model size and high computational complexity to achieve promising performance.&#10;For example, the base version of the HuBERT has about 95 million parameters.&#10;Also, a large version utilizes twice as many Transformer layers \cite{transformer} as in the base version, with almost 317 million parameters.&#10;These large-scale pre-trained models usually suffer from slow inference speed, which may hinder their usage in real-world applications where fast inference is desirable.&#10;&#10;Typical approaches to improve model efficiency include knowledge distillation (KD) \cite{kd, bucila-et-al:scheme}, pruning \cite{pruning1, pruning2}, and model quantization \cite{quantization}. &#10;While those methods reduce the processing complexity, they still require samples to pass through the entire model.&#10;In contrast, early exiting is a technique to adaptively accelerate the inference speed by returning the result at an intermediate layer.&#10;Since multiple classifiers are attached to some intermediate layers and jointly trained with the original backbone model, each classifier yields the prediction and confidence score during the inference.&#10;When the intermediate prediction is confident enough, the corresponding result can be exited early.&#10;However, existing early exit methods \cite{ee1,ee2,ee3,ee4,ee5} are mainly designed for natural language processing (NLP) classification tasks.&#10;Only a few studies have been investigated in the speech domain, including speech enhancement \cite{ee-se}, speech separation \cite{ee-sp}, and limited-vocabulary commands recognition \cite{ee-cr}.&#10;Since the ASR model does not use the commonly-used classifier for classification, it is challenging to directly apply the previous approaches to the ASR model.&#10;&#10;In this paper, we introduce a simple yet effective early exit method for ASR, namely HuBERT Early Exiting (HuBERT-EE), that enables the HuBERT model to stop the inference dynamically.&#10;To the best of our knowledge, this is the first attempt to apply the early exit framework to the speech SSL model.&#10;Specifically, the proposed HuBERT-EE accelerates the inference procedure by adding multiple early exit branches at the intermediate layers of the HuBERT.&#10;When the early exit branch is confident in its prediction, the model stops the inference and outputs the intermediate prediction as the final result.&#10;Different from intermediate CTC-based approaches \cite{inter-ctc, inter-ctc2, inter-ctc-pruning, inter-KD}, the HuBERT-EE aims to dynamically use the intermediate prediction as the model's final output with minimal WER degradation.&#10;Instead of simply applying the intermediate-CTC framework, we newly construct the self-attention-based early exit branch to perform the early exiting effectively.&#10;In addition, we explore the proper early exiting criterion and fine-tuning strategy to perform the early exiting effectively. Also, we newly design the self-attention-based early exit branch.&#10;&#10;From the experimental results on the LibriSpeech dataset, it is verified that the HuBERT-EE can be successfully applied to the ASR task.&#10;Compared to the other compression methods, HuBERT-EE enables the model to stop the inference dynamically while achieving a better speed-performance trade-off.&#10;This implies that the proposed method can be applied to a real-world scenario in which users have the flexibility to adjust the inference speed according to their demands.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Deep Learning, Signal Processing, Speech Recognition, Model Efficiency" />
        </attvalues>
      </node>
      <node id="2102.03216" label="2102.03216">
        <attvalues>
          <attvalue for="0" value="Intermediate Loss Regularization for CTC-based Speech Recognition" />
          <attvalue for="1" value="  We present a simple and efficient auxiliary loss function for automatic&#10;speech recognition (ASR) based on the connectionist temporal classification&#10;(CTC) objective. The proposed objective, an intermediate CTC loss, is attached&#10;to an intermediate layer in the CTC encoder network. This intermediate CTC loss&#10;well regularizes CTC training and improves the performance requiring only small&#10;modification of the code and small and no overhead during training and&#10;inference, respectively. In addition, we propose to combine this intermediate&#10;CTC loss with stochastic depth training, and apply this combination to a&#10;recently proposed Conformer network. We evaluate the proposed method on various&#10;corpora, reaching word error rate (WER) 9.9% on the WSJ corpus and character&#10;error rate (CER) 5.2% on the AISHELL-1 corpus respectively, based on CTC greedy&#10;search without a language model. Especially, the AISHELL-1 task is comparable&#10;to other state-of-the-art ASR systems based on auto-regressive decoder with&#10;beam search.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.06586" label="2206.06586">
        <attvalues>
          <attvalue for="0" value="FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from&#10;  Off-the-Shelf Models" />
          <attvalue for="1" value="  Cross-lingual transfer (CLT) is of various applications. However, labeled&#10;cross-lingual corpus is expensive or even inaccessible, especially in the&#10;fields where labels are private, such as diagnostic results of symptoms in&#10;medicine and user profiles in business. Nevertheless, there are off-the-shelf&#10;models in these sensitive fields. Instead of pursuing the original labels, a&#10;workaround for CLT is to transfer knowledge from the off-the-shelf models&#10;without labels. To this end, we define a novel CLT problem named FreeTransfer-X&#10;that aims to achieve knowledge transfer from the off-the-shelf models in&#10;rich-resource languages. To address the problem, we propose a 2-step knowledge&#10;distillation (KD, Hinton et al., 2015) framework based on multilingual&#10;pre-trained language models (mPLM). The significant improvement over strong&#10;neural machine translation (NMT) baselines demonstrates the effectiveness of&#10;the proposed method. In addition to reducing annotation cost and protecting&#10;private labels, the proposed method is compatible with different networks and&#10;easy to be deployed. Finally, a range of analyses indicate the great potential&#10;of the proposed method.&#10;" />
          <attvalue for="2" value="&#10;Cross-lingual transfer (CLT) is a critical topic for natural language processing due to the data imbalance between languages. &#10;While models of rich-resource languages (e.g. English) have been applied on various real-world tasks, the progress on poor-resource languages lags behind.&#10;CLT researches enable the knowledge transfer from the rich-resource languages to the poor-resource languages.&#10;&#10;Although the application of CLT is valuable, data labels are expensive or even inaccessible in private and sensitive domains, such as medicine and business.&#10;For example, the diagnostic results of a user's symptoms are private and a company's internal description of users are confidential.&#10;Since short of labels for CLT, even though there are excellent applications in rich-resource languages, it is difficult to benefit the people using poor-resource languages.&#10;Previous CLT researches have not well studied how to leverage knowledge of rich-resource languages without labels.&#10;To define and tackle this problem will benefit both the community and the industry. &#10;&#10;In order to reduce the demand of labels, existing works mainly fall into two paradigms as follows.&#10;One paradigm focuses on learning language-agnostic representation and model parameters. &#10;CLT is realized by either aligning parameters of monolingual models or sharing parameters among different languages~\cite{liu-etal-2019-investigating,jacob-mbert-2019,xlmr-conneau-etal-2020-unsupervised,Wang*2020Cross-lingual}. &#10;The objective is to build a unified representation, which is used by downstream tasks, for all the languages.&#10;In this paradigm, although the demand of labels is reduced, it still requires a certain number of labels to adapt the model to a particular language and task.&#10;Besides, models in this paradigm are usually large-scale Transformers~\cite{trm_NIPS2017_3f5ee243} based on mPLMs, which limits their deployment in real-world.&#10;Another paradigm is to leverage machine translation (MT) systems to generate training or testing pseudo-corpus for a specific language~\cite{xnli-conneau-etal-2018}.&#10;For simplicity, we take English as the rich-resource languages in this paper.&#10;Translate-train translates annotated training corpus from English to other languages. Gold labels are directly applied to the translated data. Although labels in poor-resource languages are not required, gold labels in English are still necessary. On the contrary, Translate-test translates testing corpus from poor-resource languages to English. This method can directly leverage off-the-shelf English models, but it runs a 2-pass inference which highly limits its efficiency. &#10;Both the two CLT paradigms mentioned above require language-specific and task-specific labels, except for the 2-pass Translate-test. &#10;The demand of labels highly limits the reuse of the English knowledge in private and sensitive domains. &#10;Then a question comes up: Is it possible to perform CLT totally without labels?&#10;&#10;In this paper, we define a novel problem: safe and label-free cross-lingual transfer from off-the-shelf models (FreeTransfer-X). The FreeTransfer-X asks researchers to achieve CLT only with off-the-shelf English models but any labels, as formally defined in Section~\ref{sec:problem_definition}. &#10;To the best knowledge of the authors, it's the first time that the FreeTransfer-X is clearly defined.&#10;&#10;To address the FreeTransfer-X, we propose a 2-step knowledge distillation~\cite[KD,][]{hinton2015distilling} framework based on mPLM, as shown in Figure~\ref{fig:overview}.&#10;Given an off-the-shelf model $\theta_{src}$ in the source language (e.g. English), first we take $\theta_{src}$ as the teacher and an mPLM model $\theta_{src}^{m}$ as the student, then train $\theta_{src}^{m}$ on unlabeled corpus $\mathcal{D}_{src}$. Second, we take $\theta_{tgt}^{m}$ as the teacher and train a student $\theta_{tgt}$ on unlabeled corpus $\mathcal{D}_{tgt}$.&#10;This cross-lingual transfer framework is label-free and applicable for any model architecture.&#10;Experimental results demonstrate the effectiveness of the proposed framework on both sentence classification and sequence tagging. &#10;&#10;In short, the major contributions of this work include:&#10;\begin{itemize}&#10; \item A novel cross-lingual transfer problem FreeTransfer-X is defined. The FreeTransfer-X asks researchers to achieve CLT from off-the-shelf models without using labels. It reduces the labeling cost and protects the labels in private domains such as medicine and business.&#10; \item We propose a 2-step knowledge distillation framework based on mPLMs, e.g. XLM-RoBERTa~\cite{xlmr-conneau-etal-2020-unsupervised}, to address the FreeTransfer-X. It significantly outperforms the NMT baselines on sentence classification and sequence tagging tasks. Besides, it's compatible with heterogeneous networks.&#10; \item Further analysis indicates abundant research potentials of the proposed framework. To improve the two distillation steps and the mPLM may benefit the framework.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Knowledge Distillation, Cross-Lingual Transfer, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1909.00153" label="1909.00153">
        <attvalues>
          <attvalue for="0" value="Adversarial Learning with Contextual Embeddings for Zero-resource&#10;  Cross-lingual Classification and NER" />
          <attvalue for="1" value="  Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrated&#10;state-of-the-art performance on various NLP tasks. Recent work with the&#10;multilingual version of BERT has shown that the model performs very well in&#10;zero-shot and zero-resource cross-lingual settings, where only labeled English&#10;data is used to finetune the model. We improve upon multilingual BERT's&#10;zero-resource cross-lingual performance via adversarial learning. We report the&#10;magnitude of the improvement on the multilingual MLDoc text classification and&#10;CoNLL 2002/2003 named entity recognition tasks. Furthermore, we show that&#10;language-adversarial training encourages BERT to align the embeddings of&#10;English documents and their translations, which may be the cause of the&#10;observed performance gains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Adversarial Learning, Machine Learning, Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Cross-Lingual Embeddings, Multilingual NLP" />
        </attvalues>
      </node>
      <node id="2203.00211" label="2203.00211">
        <attvalues>
          <attvalue for="0" value="Investigating Selective Prediction Approaches Across Several Tasks in&#10;  IID, OOD, and Adversarial Settings" />
          <attvalue for="1" value="  In order to equip NLP systems with selective prediction capability, several&#10;task-specific approaches have been proposed. However, which approaches work&#10;best across tasks or even if they consistently outperform the simplest baseline&#10;'MaxProb' remains to be explored. To this end, we systematically study&#10;'selective prediction' in a large-scale setup of 17 datasets across several NLP&#10;tasks. Through comprehensive experiments under in-domain (IID), out-of-domain&#10;(OOD), and adversarial (ADV) settings, we show that despite leveraging&#10;additional resources (held-out data/computation), none of the existing&#10;approaches consistently and considerably outperforms MaxProb in all three&#10;settings. Furthermore, their performance does not translate well across tasks.&#10;For instance, Monte-Carlo Dropout outperforms all other approaches on Duplicate&#10;Detection datasets but does not fare well on NLI datasets, especially in the&#10;OOD setting. Thus, we recommend that future selective prediction approaches&#10;should be evaluated across tasks and settings for reliable estimation of their&#10;capabilities.&#10;" />
          <attvalue for="2" value="&#10;&#10;Despite impressive progress made in Natural Language Processing (NLP), it is unreasonable to expect models to be perfect in their predictions.&#10;They often make incorrect predictions, especially when inputs tend to diverge from their training data distribution \cite{elsahar-galle-2019-annotate,miller2020effect,pmlr-v139-koh21a}.&#10;While this is acceptable for tolerant applications like movie recommendations, high risk associated with incorrect predictions hinders the adoption of these systems in real-world safety-critical domains like biomedical and autonomous robots.&#10;In such scenarios, selective prediction becomes crucial as it allows maintaining high accuracy by abstaining on instances where error is likely.&#10;&#10;Selective Prediction (SP) has been studied in machine learning \cite{chow1957optimum,el2010foundations} and computer vision \cite{Geifman2017SelectiveCF, Geifman2019SelectiveNetAD}, but has only recently gained attention in NLP.&#10;\cite{kamath-etal-2020-selective} proposed a post-hoc calibration-based SP technique for Question-Answering (QA) datasets. &#10;\cite{garg2021will} distill the QA model to filter out error-prone questions.&#10;Unfortunately, despite the shared goal of making NLP systems robust and reliable for real-world applications, SP has remained underexplored; the community does not know which techniques work best across tasks/settings or even if they consistently outperform the simplest baseline `MaxProb' \cite{hendrycks17baseline} (that uses the maximum softmax probability as the confidence estimator for selective prediction).&#10;&#10;In this work, we address the above point and study selective prediction in a large-scale setup of $17$ datasets across NLI, Duplicate Detection, and QA tasks. &#10;We conduct comprehensive experiments under In-Domain (IID), Out-Of-Domain (OOD), and Adversarial (ADV) settings that result in the following findings:&#10;\begin{enumerate}[noitemsep,nosep,leftmargin=*]&#10; \item None of the existing SP approaches consistently and considerably outperforms MaxProb.&#10;&#10;Slight improvement in IID: Most of the approaches outperform MaxProb in the IID setting; however, the magnitude of improvement is very small (Figure \ref{fig:iid_results}). For instance, MCD achieves an average improvement of just $0.28$ on AUC value across all NLI datasets.&#10;&#10;Negligible improvement in OOD: &#10; The magnitude of improvement is even lesser ($0.08$) than that observed in the IID setting (Figure \ref{ood_fig}). &#10; In a few cases, we also observe performance degradation (higher AUC than MaxProb). &#10;&#10;Performance degradation in ADV: All the approaches fail to even match the MaxProb performance in ADV setting (Figure \ref{adv_fig}). For instance, MCD degrades the AUC value by $1.76$ on duplicate detection datasets and calibration degrades by $1.27$ on NLI datasets in ADV setting.&#10;&#10;\item Approaches do not translate well across tasks: We find that a single approach does not achieve the best performance across all tasks. For instance, MCD outperforms all other approaches on Duplicate Detection datasets but does not fare well on the NLI datasets. &#10;&#10;\item Existing approaches require additional resources: MCD requires additional computation and calibration-based approaches require a held-out dataset. &#10; In contrast, MaxProb does not require any such resources and still outperforms them, especially in the ADV setting.&#10;&#10;\end{enumerate}&#10;&#10;Overall, our results highlight that there is a need to develop stronger selective prediction approaches that perform well across tasks while being computationally efficient. &#10;To foster development in this field, we release our code and experimental setup.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Selective Prediction, Linguistics, Task Generalization, Artificial Intelligence, NLP Systems Evaluation, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2012.07421" label="2012.07421">
        <attvalues>
          <attvalue for="0" value="WILDS: A Benchmark of in-the-Wild Distribution Shifts" />
          <attvalue for="1" value="  Distribution shifts -- where the training distribution differs from the test&#10;distribution -- can substantially degrade the accuracy of machine learning (ML)&#10;systems deployed in the wild. Despite their ubiquity in the real-world&#10;deployments, these distribution shifts are under-represented in the datasets&#10;widely used in the ML community today. To address this gap, we present WILDS, a&#10;curated benchmark of 10 datasets reflecting a diverse range of distribution&#10;shifts that naturally arise in real-world applications, such as shifts across&#10;hospitals for tumor identification; across camera traps for wildlife&#10;monitoring; and across time and location in satellite imaging and poverty&#10;mapping. On each dataset, we show that standard training yields substantially&#10;lower out-of-distribution than in-distribution performance. This gap remains&#10;even with models trained by existing methods for tackling distribution shifts,&#10;underscoring the need for new methods for training models that are more robust&#10;to the types of distribution shifts that arise in practice. To facilitate&#10;method development, we provide an open-source package that automates dataset&#10;loading, contains default model architectures and hyperparameters, and&#10;standardizes evaluations. Code and leaderboards are available at&#10;https://wilds.stanford.edu.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07009" label="2109.07009">
        <attvalues>
          <attvalue for="0" value="Will this Question be Answered? Question Filtering via Answer Model&#10;  Distillation for Efficient Question Answering" />
          <attvalue for="1" value="  In this paper we propose a novel approach towards improving the efficiency of&#10;Question Answering (QA) systems by filtering out questions that will not be&#10;answered by them. This is based on an interesting new finding: the answer&#10;confidence scores of state-of-the-art QA systems can be approximated well by&#10;models solely using the input question text. This enables preemptive filtering&#10;of questions that are not answered by the system due to their answer confidence&#10;scores being lower than the system threshold. Specifically, we learn&#10;Transformer-based question models by distilling Transformer-based answering&#10;models. Our experiments on three popular QA datasets and one industrial QA&#10;benchmark demonstrate the ability of our question models to approximate the&#10;Precision/Recall curves of the target QA system well. These question models,&#10;when used as filters, can effectively trade off lower computation cost of QA&#10;systems for lower Recall, e.g., reducing computation by ~60%, while only losing&#10;~3-4% of Recall.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03133" label="2004.03133">
        <attvalues>
          <attvalue for="0" value="Neutralizing Gender Bias in Word Embedding with Latent Disentanglement&#10;  and Counterfactual Generation" />
          <attvalue for="1" value="  Recent research demonstrates that word embeddings, trained on the&#10;human-generated corpus, have strong gender biases in embedding spaces, and&#10;these biases can result in the discriminative results from the various&#10;downstream tasks. Whereas the previous methods project word embeddings into a&#10;linear subspace for debiasing, we introduce a \textit{Latent Disentanglement}&#10;method with a siamese auto-encoder structure with an adapted gradient reversal&#10;layer. Our structure enables the separation of the semantic latent information&#10;and gender latent information of given word into the disjoint latent&#10;dimensions. Afterwards, we introduce a \textit{Counterfactual Generation} to&#10;convert the gender information of words, so the original and the modified&#10;embeddings can produce a gender-neutralized word embedding after geometric&#10;alignment regularization, without loss of semantic information. From the&#10;various quantitative and qualitative debiasing experiments, our method shows to&#10;be better than existing debiasing methods in debiasing word embeddings. In&#10;addition, Our method shows the ability to preserve semantic information during&#10;debiasing by minimizing the semantic information losses for extrinsic NLP&#10;downstream tasks.&#10;" />
          <attvalue for="2" value="&#10; Recent researches have disclosed that word embeddings contain unexpected bias in their geometry on the embedding space \cite{bolukbasi16,zhao2019gender}. The bias reflects unwanted stereotypes such as the correlation between gender and occupation words. \cite{bolukbasi16} enumerated that the automatically generated analogies of ($she,he$) in the Word2Vec \cite{mikolov2013distributed} show the gender biases in significant level. An example of the analogies is the relatively closer distance of she to nurse; and he to doctor.&#10;&#10;\cite{garg18} demonstrated that the embeddings, from Word2Vec \cite{word2vec13} to Glove \cite{glove14}, have strong associations between value-neutral words and population-segment words, i.e. a strong association between housekeeper and Hispanic. This unwanted bias can cause biased results in the downstream tasks \cite{caliskan17,kiritchenko2018examining,bhaskaran2019good} and gender discrimination in NLP systems.&#10;&#10;From the various gender debiasing methods for pre-trained word embeddings, the widely recognized method is a post-processing method, which projects word embeddings to the space that is orthogonal to the gender direction vector defined by a set of gender word pairs. However, if the gender direction vector includes a component of semantic information, the semantic information will be lost through the post-processing projections.&#10;&#10;To balance between the gender debiasing and the semantic information preserving, we propose an encoder-decoder framework that disentangles a latent space of a given word embedding into two encoded latent spaces: the first part is the gender latent space, and the second part is the semantic latent space that is independent to the gender information. To disentangle the latent space into two sub-spaces, we use a gradient reversal layer by prohibiting the inference on the gender latent information from the semantic information. Then, we generate a counterfactual word embedding by converting the encoded gender latent into the opposite gender. Afterwards, the original and the counterfactual word embeddings are geometrically interpreted to neutralize the gender information of given word embeddings, see Figure \ref{fig:schematic} for the illustration on our debiasing method.&#10;&#10;Our contributions are summarized as follows:&#10;\begin{itemize}&#10;\item We propose a method for disentangling the latent information of the word embedding by utilizing the siamese auto-encoder structure with an adapted gradient reversal layer.&#10;\item We propose a new gender debiasing method, which transforms the original word embedding into gender-neutral embedding, with the gender-counterfactual word embedding.&#10;\item We propose a generalized alignment with a kernel function that enforces the embedding shift, during the debiasing process, in a direction that does not damage the semantics of word embedding.&#10;\end{itemize}&#10;&#10;We evaluated the proposed method and other baseline methods with several quantitative and qualitative debiasing experiments, and we found that the proposed method shows significant improvements from the existing methods. Additionally, the results from several NLP downstream tasks show that our proposed method minimizes performance degradation than the existing methods.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Bias Reduction, Machine Learning, Computer Science, Linguistics, Word Embeddings, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1805.04508" label="1805.04508">
        <attvalues>
          <attvalue for="0" value="Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems" />
          <attvalue for="1" value="  Automatic machine learning systems can inadvertently accentuate and&#10;perpetuate inappropriate human biases. Past work on examining inappropriate&#10;biases has largely focused on just individual systems. Further, there is no&#10;benchmark dataset for examining inappropriate biases in systems. Here for the&#10;first time, we present the Equity Evaluation Corpus (EEC), which consists of&#10;8,640 English sentences carefully chosen to tease out biases towards certain&#10;races and genders. We use the dataset to examine 219 automatic sentiment&#10;analysis systems that took part in a recent shared task, SemEval-2018 Task 1&#10;'Affect in Tweets'. We find that several of the systems show statistically&#10;significant bias; that is, they consistently provide slightly higher sentiment&#10;intensity predictions for one race or one gender. We make the EEC freely&#10;available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.10256" label="1906.10256">
        <attvalues>
          <attvalue for="0" value="Good Secretaries, Bad Truck Drivers? Occupational Gender Stereotypes in&#10;  Sentiment Analysis" />
          <attvalue for="1" value="  In this work, we investigate the presence of occupational gender stereotypes&#10;in sentiment analysis models. Such a task has implications for reducing&#10;implicit biases in these models, which are being applied to an increasingly&#10;wide variety of downstream tasks. We release a new gender-balanced dataset of&#10;800 sentences pertaining to specific professions and propose a methodology for&#10;using it as a test bench to evaluate sentiment analysis models. We evaluate the&#10;presence of occupational gender stereotypes in 3 different models using our&#10;approach, and explore their relationship with societal perceptions of&#10;occupations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.12320" label="2303.12320">
        <attvalues>
          <attvalue for="0" value="GrapeQA: GRaph Augmentation and Pruning to Enhance Question-Answering" />
          <attvalue for="1" value="  Commonsense question-answering (QA) methods combine the power of pre-trained&#10;Language Models (LM) with the reasoning provided by Knowledge Graphs (KG). A&#10;typical approach collects nodes relevant to the QA pair from a KG to form a&#10;Working Graph (WG) followed by reasoning using Graph Neural Networks(GNNs).&#10;This faces two major challenges: (i) it is difficult to capture all the&#10;information from the QA in the WG, and (ii) the WG contains some irrelevant&#10;nodes from the KG. To address these, we propose GrapeQA with two simple&#10;improvements on the WG: (i) Prominent Entities for Graph Augmentation&#10;identifies relevant text chunks from the QA pair and augments the WG with&#10;corresponding latent representations from the LM, and (ii) Context-Aware Node&#10;Pruning removes nodes that are less relevant to the QA pair. We evaluate our&#10;results on OpenBookQA, CommonsenseQA and MedQA-USMLE and see that GrapeQA shows&#10;consistent improvements over its LM + KG predecessor (QA-GNN in particular) and&#10;large improvements on OpenBookQA.&#10;" />
          <attvalue for="2" value="&#10;&#10;Answering questions is a challenging NLP problem as it involves understanding the question context and sifting through relevant information to identify the answer.&#10;Question-answering models have evolved from rule-based~\cite{7980526} to RNN-based sequence models~\cite{ and now to Transformer-based Language Models (LM) such as RoBERTa-large~\cite{DBLP:journals/corr/abs-1907-11692}.&#10;However, commonsense question-answering&#10;adds a layer of complexity as the model needs to reason about questions relating diverse topics, making the task challenging for LMs that may not have seen something similar in the pre-training data.&#10;&#10;While LMs capture the implicit patterns and contextual information within the data, KGs are able to capture explicit relations between the text entities.&#10;KGs such as&#10;Freebase~\cite{bollacker2008freebase},&#10;Wikidata~\cite{vrandevcic2012wikidata}, or&#10;ConceptNet~\cite{speer2017conceptnet} store knowledge in the form of graph triplets (topic-relationship-topic) and are well suited for Graph Neural Networks (GNNs), e.g. ~\cite{welling2016semi}.&#10;Thus, commonsense QA in particular has attracted interest in combining LMs and KGs with the reasoning ability of GNNs~\cite{lin-etal-2019-kagnet,yasunaga-etal-2021-qa}.&#10;&#10;Most works on LM + KG extract a sub-graph or Working Graph (WG) from the KG based on concepts mentioned in the QA pair~\cite{lin-etal-2019-kagnet,feng-etal-2020-scalable,yasunaga-etal-2021-qa} and focus on improving reasoning.&#10;For example, \cite{lin-etal-2019-kagnet}~propose a graph network to score answers while~\cite{feng-etal-2020-scalable} focus on a multi-hop message passing framework that allows each node to attend to multi-hop neighbors in a single layer, combining interpretable path-based reasoning with scalable GNNs.&#10;\cite{yasunaga-etal-2021-qa}~improve the extracted WG through a relevance scoring mechanism followed by joint reasoning and~\cite{zhang2022greaselm} fuse information from both the modalities (LM, KG) by mixing their tokens and nodes.&#10;&#10;Our emphasis with GrapeQA lies in improving the working graph (WG) with two simple ideas.&#10;(i)~We augment the WG with useful information from the question-answer pair reducing the burden on a single QA context node used in previous works.(discussed in \ref{22})&#10;(ii)~Instead of keeping all nodes of the WG, or simply scoring relevance, we drop less relevant information (nodes) from the WG simplifying the graph reasoning process.&#10;The improvements to the WG are combined with the reasoning process of QA-GNN~\cite{yasunaga-etal-2021-qa} and evaluated on three datasets, where we see especially large improvements on domain-specific OpenBookQA (discussed in \ref{23}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Question Answering, Computer Science, Linguistics, Cognitive Science, Language Models, Knowledge Graphs, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1612.03975" label="1612.03975">
        <attvalues>
          <attvalue for="0" value="ConceptNet 5.5: An Open Multilingual Graph of General Knowledge" />
          <attvalue for="1" value="  Machine learning about language can be improved by supplying it with specific&#10;knowledge and sources of external information. We present here a new version of&#10;the linked open data resource ConceptNet that is particularly well suited to be&#10;used with modern NLP techniques such as word embeddings.&#10;  ConceptNet is a knowledge graph that connects words and phrases of natural&#10;language with labeled edges. Its knowledge is collected from many sources that&#10;include expert-created resources, crowd-sourcing, and games with a purpose. It&#10;is designed to represent the general knowledge involved in understanding&#10;language, improving natural language applications by allowing the application&#10;to better understand the meanings behind the words people use.&#10;  When ConceptNet is combined with word embeddings acquired from distributional&#10;semantics (such as word2vec), it provides applications with understanding that&#10;they would not acquire from distributional semantics alone, nor from narrower&#10;resources such as WordNet or DBPedia. We demonstrate this with state-of-the-art&#10;results on intrinsic evaluations of word relatedness that translate into&#10;improvements on applications of word vectors, including solving SAT-style&#10;analogies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.02151" label="1909.02151">
        <attvalues>
          <attvalue for="0" value="KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning" />
          <attvalue for="1" value="  Commonsense reasoning aims to empower machines with the human ability to make&#10;presumptions about ordinary situations in our daily life. In this paper, we&#10;propose a textual inference framework for answering commonsense questions,&#10;which effectively utilizes external, structured commonsense knowledge graphs to&#10;perform explainable inferences. The framework first grounds a question-answer&#10;pair from the semantic space to the knowledge-based symbolic space as a schema&#10;graph, a related sub-graph of external knowledge graphs. It represents schema&#10;graphs with a novel knowledge-aware graph network module named KagNet, and&#10;finally scores answers with graph representations. Our model is based on graph&#10;convolutional networks and LSTMs, with a hierarchical path-based attention&#10;mechanism. The intermediate attention scores make it transparent and&#10;interpretable, which thus produce trustworthy inferences. Using ConceptNet as&#10;the only external resource for Bert-based models, we achieved state-of-the-art&#10;performance on the CommonsenseQA, a large-scale dataset for commonsense&#10;reasoning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00646" label="2005.00646">
        <attvalues>
          <attvalue for="0" value="Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question&#10;  Answering" />
          <attvalue for="1" value="  Existing work on augmenting question answering (QA) models with external&#10;knowledge (e.g., knowledge graphs) either struggle to model multi-hop relations&#10;efficiently, or lack transparency into the model's prediction rationale. In&#10;this paper, we propose a novel knowledge-aware approach that equips pre-trained&#10;language models (PTLMs) with a multi-hop relational reasoning module, named&#10;multi-hop graph relation network (MHGRN). It performs multi-hop,&#10;multi-relational reasoning over subgraphs extracted from external knowledge&#10;graphs. The proposed reasoning module unifies path-based reasoning methods and&#10;graph neural networks to achieve better interpretability and scalability. We&#10;also empirically show its effectiveness and scalability on CommonsenseQA and&#10;OpenbookQA datasets, and interpret its behaviors with case studies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.08604" label="2001.08604">
        <attvalues>
          <attvalue for="0" value="Variational Hierarchical Dialog Autoencoder for Dialog State Tracking&#10;  Data Augmentation" />
          <attvalue for="1" value="  Recent works have shown that generative data augmentation, where synthetic&#10;samples generated from deep generative models complement the training dataset,&#10;benefit NLP tasks. In this work, we extend this approach to the task of dialog&#10;state tracking for goal-oriented dialogs. Due to the inherent hierarchical&#10;structure of goal-oriented dialogs over utterances and related annotations, the&#10;deep generative model must be capable of capturing the coherence among&#10;different hierarchies and types of dialog features. We propose the Variational&#10;Hierarchical Dialog Autoencoder (VHDA) for modeling the complete aspects of&#10;goal-oriented dialogs, including linguistic features and underlying structured&#10;annotations, namely speaker information, dialog acts, and goals. The proposed&#10;architecture is designed to model each aspect of goal-oriented dialogs using&#10;inter-connected latent variables and learns to generate coherent goal-oriented&#10;dialogs from the latent spaces. To overcome training issues that arise from&#10;training complex variational models, we propose appropriate training&#10;strategies. Experiments on various dialog datasets show that our model improves&#10;the downstream dialog trackers' robustness via generative data augmentation. We&#10;also discover additional benefits of our unified approach to modeling&#10;goal-oriented dialogs: dialog response generation and user simulation, where&#10;our model outperforms previous strong baselines.&#10;" />
          <attvalue for="2" value="&#10;&#10;Data augmentation, a technique that augments the training set with &#10;&#09;label-preserving synthetic samples, is commonly employed in modern machine&#10;&#09;learning approaches.&#10;It has been used extensively in visual learning pipelines&#10;&#09;\cite{shorten2019survey} but less frequently for NLP &#10;&#09;tasks due to the lack of well-established techniques in the area.&#10;While some notable work exists in &#10;&#09;text classification \cite{zhang2015character}, spoken language &#10;&#09;understanding \cite{yoo2019data}, and machine translation&#10;&#09;\cite{fadaee2017data}, we still lack the full understanding of &#10;&#09;utilizing generative models for text augmentation.&#10;&#10;Ideally, a data augmentation technique for supervised tasks must&#10;&#09;synthesize distribution-preserving and &#10;&#09;sufficiently realistic samples.&#10;Current approaches for data augmentation in NLP tasks mostly revolve around&#10;&#09;thesaurus data augmentation \cite{zhang2015character}, in&#10;&#09;which words that belong to the same semantic role are substituted with one&#10;&#09;another using a preconstructed lexicon, and noisy data augmentation&#10;&#09;\cite{wei2019eda} where random editing operations create perturbations in&#10;&#09;the language space.&#10;Thesaurus data augmentation requires a set of handcrafted semantic dictionaries, &#10;&#09;which are costly&#10;&#09;to build and maintain, whereas noisy data augmentation does not synthesize&#10;&#09;sufficiently realistic samples.&#10;The recent trend \cite{hu2017toward,yoo2019data,shin2019utterance} gravitates &#10;&#09;towards generative data augmentation&#10;&#09;(GDA), a class of techniques that leverage deep generative models &#10;&#09;such as VAEs to delegate the automatic discovery of novel class-preserving samples to machine learning.&#10;In this work, we explore GDA in the context of dialog modeling and contextual&#10;&#09;understanding.&#10;&#10;Goal-oriented dialogs occur between a user and a system that communicates&#10;&#09;verbally to accomplish the user's goals &#10;&#09;(\ref{tab:vhda-interpolation-2}).&#10;However, because the user's goals and the system's possible actions are not &#10;&#09;transparent to each other, both parties must rely on verbal communications &#10;&#09;to infer and take appropriate actions to resolve the goals.&#10;Dialog state tracker is a core component of such systems, &#10;&#09;enabling it to track the dialog's latest status \cite{henderson2014second}.&#10;A dialog state typically consists of inform and request&#10;&#09;types of slot values.&#10;For example, a user may verbally refer to a previously mentioned food type as &#10; the preferred one - e.g., Asian (\verb|inform(food=asian)|).&#10;Given the user utterance and historical turns, the state tracker must infer&#10;&#09;the user's current goals.&#10;As such, we can view dialog state tracking as a sparse sequential &#10;&#09;multi-class classification problem.&#10;Modeling goal-oriented dialogs for GDA requires a novel approach &#10;&#09;that simultaneously solves state tracking, user simulation &#10;&#09;\cite{schatzmann2007agenda}, and utterance generation.&#10;&#10;Various deep models exist for modeling dialogs.&#10;The Markov approach \cite{serban2017hierarchical} employs&#10;&#09;a sequence-to-sequence variational autoencoder (VAE) &#10;&#09;\cite{kingma2013auto} structure to predict the next utterance given&#10;&#09;a deterministic context representation, while the holistic approach&#10;&#09;\cite{park2018hierarchical} utilizes a set of global&#10;&#09;latent variables to encode the entire dialog,&#10;&#09;improving the awareness in general dialog structures.&#10;However, current approaches are limited to linguistic features.&#10;Recently, \cite{bak2019variational} proposed a hierarchical VAE structure&#10;&#09;that incorporates the speaker's information, but we have yet&#10;&#09;to explore a universal approach for encompassing fundamental aspects of &#10;&#09;goal-oriented dialogs.&#10;Such a unified model capable of disentangling latents into specific dialog &#10; aspects can increase the modeling efficiency and enable interesting &#10; extensions based on the fine-grained controllability.&#10;&#10;This paper proposes a &#10;&#09;novel multi-level hierarchical and recurrent VAE structure&#10;&#09;called Variational Hierarchical Dialog Autoencoder (VHDA).&#10;Our model enables modeling all aspects (speaker information, goals, dialog &#10;&#09;acts, utterances, and general dialog flow) of goal-oriented dialogs in a disentangled manner by assigning latents to each aspect.&#10;However, complex and autoregressive VAEs are known to suffer from the risk of&#10;&#09;inference collapse \cite{cremer2018inference}, in which &#10;&#09;the model converges to a local optimum where the generator network&#10;&#09;neglects the latents, reducing the generation controllability.&#10;To mitigate the issue, we devise two simple but effective training strategies.&#10;&#10;Our contributions are summarized as follows. &#10;\begin{enumerate}&#10; \item We propose a novel deep latent model for modeling dialog utterances and &#10;&#09;their relationships with the goal-oriented annotations.&#10;&#09;We show that the strong level of coherence and accuracy displayed by &#10;&#09;the model allows it to be used for augmenting dialog state tracking&#10;&#09;datasets.&#10;&#09;\item Leveraging the model's generation capabilities, we show that generative &#10; data augmentation is attainable even for the complex dialog-related &#10; tasks that pertain to both hierarchical and sequential annotations.&#10; \item We propose simple but effective training policies for our VAE-based &#10; model, which have applications in other similar VAE structures.&#10;\end{enumerate}&#10;&#10;The code for reproducing this paper is available at github&#10; .&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Generative Data Augmentation, Deep Learning Models, Machine Learning, Computer Science, Linguistics, Dialog State Tracking, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1509.01626" label="1509.01626">
        <attvalues>
          <attvalue for="0" value="Character-level Convolutional Networks for Text Classification" />
          <attvalue for="1" value="  This article offers an empirical exploration on the use of character-level&#10;convolutional networks (ConvNets) for text classification. We constructed&#10;several large-scale datasets to show that character-level convolutional&#10;networks could achieve state-of-the-art or competitive results. Comparisons are&#10;offered against traditional models such as bag of words, n-grams and their&#10;TFIDF variants, and deep learning models such as word-based ConvNets and&#10;recurrent neural networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.02305" label="1809.02305">
        <attvalues>
          <attvalue for="0" value="Data Augmentation for Spoken Language Understanding via Joint&#10;  Variational Generation" />
          <attvalue for="1" value="  Data scarcity is one of the main obstacles of domain adaptation in spoken&#10;language understanding (SLU) due to the high cost of creating manually tagged&#10;SLU datasets. Recent works in neural text generative models, particularly&#10;latent variable models such as variational autoencoder (VAE), have shown&#10;promising results in regards to generating plausible and natural sentences. In&#10;this paper, we propose a novel generative architecture which leverages the&#10;generative power of latent variable models to jointly synthesize fully&#10;annotated utterances. Our experiments show that existing SLU models trained on&#10;the additional synthetic examples achieve performance gains. Our approach not&#10;only helps alleviate the data scarcity issue in the SLU task for many datasets&#10;but also indiscriminately improves language understanding performances for&#10;various SLU models, supported by extensive experiments and rigorous statistical&#10;testing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.03424" label="1804.03424">
        <attvalues>
          <attvalue for="0" value="A Hierarchical Latent Structure for Variational Conversation Modeling" />
          <attvalue for="1" value="  Variational autoencoders (VAE) combined with hierarchical RNNs have emerged&#10;as a powerful framework for conversation modeling. However, they suffer from&#10;the notorious degeneration problem, where the decoders learn to ignore latent&#10;variables and reduce to vanilla RNNs. We empirically show that this degeneracy&#10;occurs mostly due to two reasons. First, the expressive power of hierarchical&#10;RNN decoders is often high enough to model the data using only its decoding&#10;distributions without relying on the latent variables. Second, the conditional&#10;VAE structure whose generation process is conditioned on a context, makes the&#10;range of training targets very sparse; that is, the RNN decoders can easily&#10;overfit to the training data ignoring the latent variables. To solve the&#10;degeneration problem, we propose a novel model named Variational Hierarchical&#10;Conversation RNNs (VHCR), involving two key ideas of (1) using a hierarchical&#10;structure of latent variables, and (2) exploiting an utterance drop&#10;regularization. With evaluations on two datasets of Cornell Movie Dialog and&#10;Ubuntu Dialog Corpus, we show that our VHCR successfully utilizes latent&#10;variables and outperforms state-of-the-art models for conversation generation.&#10;Moreover, it can perform several new utterance control tasks, thanks to its&#10;hierarchical latent structure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.03558" label="1801.03558">
        <attvalues>
          <attvalue for="0" value="Inference Suboptimality in Variational Autoencoders" />
          <attvalue for="1" value="  Amortized inference allows latent-variable models trained via variational&#10;learning to scale to large datasets. The quality of approximate inference is&#10;determined by two factors: a) the capacity of the variational distribution to&#10;match the true posterior and b) the ability of the recognition network to&#10;produce good variational parameters for each datapoint. We examine approximate&#10;inference in variational autoencoders in terms of these factors. We find that&#10;divergence from the true posterior is often due to imperfect recognition&#10;networks, rather than the limited complexity of the approximating distribution.&#10;We show that this is due partly to the generator learning to accommodate the&#10;choice of approximation. Furthermore, we show that the parameters used to&#10;increase the expressiveness of the approximation play a role in generalizing&#10;inference rather than simply improving the complexity of the approximation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.05540" label="2108.05540">
        <attvalues>
          <attvalue for="0" value="Unsupervised Corpus Aware Language Model Pre-training for Dense Passage&#10;  Retrieval" />
          <attvalue for="1" value="  Recent research demonstrates the effectiveness of using fine-tuned language&#10;models~(LM) for dense retrieval. However, dense retrievers are hard to train,&#10;typically requiring heavily engineered fine-tuning pipelines to realize their&#10;full potential. In this paper, we identify and address two underlying problems&#10;of dense retrievers: i)~fragility to training data noise and ii)~requiring&#10;large batches to robustly learn the embedding space. We use the recently&#10;proposed Condenser pre-training architecture, which learns to condense&#10;information into the dense vector through LM pre-training. On top of it, we&#10;propose coCondenser, which adds an unsupervised corpus-level contrastive loss&#10;to warm up the passage embedding space. Retrieval experiments on MS-MARCO,&#10;Natural Question, and Trivia QA datasets show that coCondenser removes the need&#10;for heavy data engineering such as augmentation, synthesis, or filtering, as&#10;well as the need for large batch training. It shows comparable performance to&#10;RocketQA, a state-of-the-art, heavily engineered system, using simple small&#10;batch fine-tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.08240" label="2010.08240">
        <attvalues>
          <attvalue for="0" value="Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for&#10;  Pairwise Sentence Scoring Tasks" />
          <attvalue for="1" value="  There are two approaches for pairwise sentence scoring: Cross-encoders, which&#10;perform full-attention over the input pair, and Bi-encoders, which map each&#10;input independently to a dense vector space. While cross-encoders often achieve&#10;higher performance, they are too slow for many practical use cases.&#10;Bi-encoders, on the other hand, require substantial training data and&#10;fine-tuning over the target task to achieve competitive performance. We present&#10;a simple yet efficient data augmentation strategy called Augmented SBERT, where&#10;we use the cross-encoder to label a larger set of input pairs to augment the&#10;training data for the bi-encoder. We show that, in this process, selecting the&#10;sentence pairs is non-trivial and crucial for the success of the method. We&#10;evaluate our approach on multiple tasks (in-domain) as well as on a domain&#10;adaptation task. Augmented SBERT achieves an improvement of up to 6 points for&#10;in-domain and of up to 37 points for domain adaptation tasks compared to the&#10;original bi-encoder performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Sentence embeddings are a well studied area in recent literature. Earlier techniques included unsupervised methods such as Skip-thought vectors \cite{10.5555/2969442.2969607} and supervised methods such as InferSent \cite{conneau-etal-2017-supervised} or USE \cite{cer-etal-2018-universal}. &#10;For pairwise scoring tasks, more recent sentence embedding techniques are also able to encode a pair of sentences jointly.&#10;Among these, BERT \cite{devlin2018bert} can be used as a cross-encoder.&#10;Both inputs are separated by a special SEP token and multi-head attention is applied over all input tokens. While the BERT cross-encoder achieves high performances for many sentence pair-tasks, a drawback is that no independent sentence representations are generated. This drawback was addressed by SBERT \cite{reimers-2019-sentence-bert}, which applies BERT independently on the inputs followed by mean pooling on the output to create fixed-sized sentence embeddings.&#10;&#10;\newcite{Humeau2020Poly-encoders} showed that cross-encoders typically outperform bi-encoders on sentence scoring tasks. They proposed a third strategy (poly-encoders), that is in-between cross- and bi-encoders. Poly-encoders utilize two separate transformers, one for the candidate and one for the context. A given candidate is represented by one vector, while the context is jointly encoded with the candidates (similar to cross-encoders). Unlike cross-encoder's full self attention technique, poly-encoders apply attention between two inputs only at the top layer. Poly-encoders have the drawback that they are only practical for certain applications: The score function is not symmetric, i.e., they cannot be applied for tasks with a symmetric similarity relation. Further, poly-encoder representations cannot be efficiently indexed, causing issues for retrieval tasks with large corpora sizes. &#10;&#10;\newcite{chen-etal-2020-dipair} propose the DiPair architecture which, similar to our work, also uses a cross-encoder model to annotate unlabeled pairs for fine-tuning a bi-encoder model. &#10;DiPair focuses on inference speed and provides a detailed ablation for optimal bi-encoder architectures for performance versus speed trade-offs. The focus of our work are sampling techniques, which we find crucial for performance boosts in the bi-encoder model while keeping its architecture constant.&#10;&#10;Our proposed data augmentation approach is based on semi-supervision \cite{10.1145/279943.279962} for in-domain tasks, which has been applied successfully for a wide range of tasks. \newcite{uva-etal-2018-injecting} train a SVM model with few gold samples and apply semi-supervision with pre-training neural networks. Another common strategy is to generate paraphrases of existent sentences, for example, by replacing words with synonyms \cite{wei-zou-2019-eda}, by using round-trip translation \cite{wei2018fast,xie2020unsupervised}, or with seq2seq-models \cite{kumar-etal-2019-submodular}. Other approaches generate synthetic data by using generative adversarial networks \cite{tanaka2019data}, by using a language model to replace certain words \cite{cbert-aug} or to generate complete sentences \cite{anaby2019not}. These data augmentation approaches have in common that they were applied to single sentence classification tasks. In our work, we focus on sentence pair tasks, for which we need to generate suitable sentence pairs. As we show, randomly combining sentences is insufficient. Sampling appropriate pairs has a decisive impact on performance &#10;which corresponds to recent findings on similar datasets \cite{peinelt-etal-2019-aiming}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2007.00049" label="2007.00049">
        <attvalues>
          <attvalue for="0" value="OSCaR: Orthogonal Subspace Correction and Rectification of Biases in&#10;  Word Embeddings" />
          <attvalue for="1" value="  Language representations are known to carry stereotypical biases and, as a&#10;result, lead to biased predictions in downstream tasks. While existing methods&#10;are effective at mitigating biases by linear projection, such methods are too&#10;aggressive: they not only remove bias, but also erase valuable information from&#10;word embeddings. We develop new measures for evaluating specific information&#10;retention that demonstrate the tradeoff between bias removal and information&#10;retention. To address this challenge, we propose OSCaR (Orthogonal Subspace&#10;Correction and Rectification), a bias-mitigating method that focuses on&#10;disentangling biased associations between concepts instead of removing concepts&#10;wholesale. Our experiments on gender biases show that OSCaR is a well-balanced&#10;approach that ensures that semantic information is retained in the embeddings&#10;and bias is also effectively mitigated.&#10;" />
          <attvalue for="2" value=" &#10;Word embeddings are used extensively across natural language processing (NLP) and succinctly capture not only the syntactic and semantic structure of language, but also word meaning in context.&#10;As such, word embeddings are essential building blocks for today's state-of-the-art in NLP. &#10;But they are also known to capture a significant amount of stereotypical associations \cite[e.g.,][]{debias, zhao-etal-2017-men, Bias1, biasSurvey} related to gender, race, nationality, or religion, which can manifest in unwanted and/or potentially harmful ways in downstream tasks \cite{gap, zhao-etal-2019-gender,bias2}. Such potentially problematic associations, when embedded in word representations, can lead to incorrect and unfair decisions about large groups of people. While the term ``bias'' has many meanings, in this paper we use it to refer to these {unwanted stereotypical associations}. &#10;&#10;Existing methods to mitigate these effects either require expensive retraining of vectors \cite{gan-bias} which can be inefficient, or projecting out information contained along an entire subspace representing a protected concept (such as gender or race) in the embedding space \cite[e.g.,][]{debias,Bias1, ravfogel2020null}.&#10;Projective approaches are difficult to control as they are either insufficient: removing a subspace can still leave residual bias~\cite{gonen2019lipstick, lauscher2019bias}, or too aggressive: in the case of gender, also unnecessarily altering the association between the word pregnant and words like female and mother.&#10;In tasks such as coreference resolution, removing such associations could hinder reference resolution. &#10;&#10;To quantify how much valid information is retained, we look at the output space of models.&#10;Following~\cite{bias2}, we use Natural Language Inference (NLI) as an effective quantitative probe.&#10;Here, we construct the hypothesis by making minimal edits to a premise, and observe model prediction conditioned on such changes. For example,&#10;\begin{description}[noitemsep,topsep=0.5ex,align=right,labelindent=12ex]&#10;\item [Premise:] A \subject{matriarch} sold a watch.&#10;\item [Hypothesis:] A \subject{woman} sold a watch.&#10;\end{description}&#10;&#10;Here, the objective is to determine if the hypothesis is entailed by the premise, contradicted by it, or neither (neutral to it). A GloVe-based NLI~\cite{parikh2016decomposable} model, without any explicit form of bias mitigation, predicts label entail with a high probability of $97\%$; the notion of a \subject{matriarch} being a \subject{woman} is correctly identified by the model. &#10;However, after projective debiasing, the model classifies the pair as neutral with a probability $62\%$ while the probability of the label $entail$ drops to much lower at $16\%$. That is, aggressive mitigation of gender representations erases valid gender associations.&#10;&#10;Ideally, we should correct problematic associations without erasing valid ones.&#10;To this end, we propose \oscar (Othogonal Subspace Correction and Rectification) which orthogonalizes and rectifies identified subspaces of concepts that are incorrectly associated in an embedding space.&#10;Embeddings outside the subspace are stretched in a graded manner or untouched.&#10;Our contributions are:&#10;\begin{enumerate} \item We argue that mitigating unwanted stereotypical associations should go beyond information removal (e.g., projecting out features), and should also preserve pertinent associations. &#10;\item We present \oscar, a completely different method from the existing projective approaches; it uses orthogonalization of subspaces desired not to have interdependence, and so minimal change is made to embeddings to prevent loss of desired associations. &#10;\item We develop a combination of tests based on NLI to evaluate both bias mitigation and information retention.&#10;\item Our experiments show that \oscar is a well-balanced approach that mitigates biases as good as projective approaches while retaining more valid associations.&#10;\end{enumerate}&#10;&#10;Our contributions, focusing specifically on representations rather than classifiers, are important given the preponderance of distributed representations of text across NLP.&#10;Predictions from systems that use such representations, if unchecked, could lead to real-world decisions (involving, e.g. hiring) that ``systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others''~\cite{friedman1996bias}.&#10;In an effort to prevent such transition of representational harms into allocational harms~\cite[cf.][]{crawford2017trouble,abbasi2019fairness,blodgett-etal-2020-language}, we look at mitigating stereotyping biases at the source, i.e. the embedding space. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Social Computing, Machine Learning, Computer Science, Linguistics, Word Embedding Analysis, Information Retention Methods, Language Bias Mitigation, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1810.05201" label="1810.05201">
        <attvalues>
          <attvalue for="0" value="Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns" />
          <attvalue for="1" value="  Coreference resolution is an important task for natural language&#10;understanding, and the resolution of ambiguous pronouns a longstanding&#10;challenge. Nonetheless, existing corpora do not capture ambiguous pronouns in&#10;sufficient volume or diversity to accurately indicate the practical utility of&#10;models. Furthermore, we find gender bias in existing corpora and systems&#10;favoring masculine entities. To address this, we present and release GAP, a&#10;gender-balanced labeled corpus of 8,908 ambiguous pronoun-name pairs sampled to&#10;provide diverse coverage of challenges posed by real-world text. We explore a&#10;range of baselines which demonstrate the complexity of the challenge, the best&#10;achieving just 66.9% F1. We show that syntactic structure and continuous neural&#10;models provide promising, complementary cues for approaching the challenge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.07593" label="1801.07593">
        <attvalues>
          <attvalue for="0" value="Mitigating Unwanted Biases with Adversarial Learning" />
          <attvalue for="1" value="  Machine learning is a tool for building models that accurately represent&#10;input training data. When undesired biases concerning demographic groups are in&#10;the training data, well-trained models will reflect those biases. We present a&#10;framework for mitigating such biases by including a variable for the group of&#10;interest and simultaneously learning a predictor and an adversary. The input to&#10;the network X, here text or census data, produces a prediction Y, such as an&#10;analogy completion or income bracket, while the adversary tries to model a&#10;protected variable Z, here gender or zip code.&#10;  The objective is to maximize the predictor's ability to predict Y while&#10;minimizing the adversary's ability to predict Z. Applied to analogy completion,&#10;this method results in accurate predictions that exhibit less evidence of&#10;stereotyping Z. When applied to a classification task using the UCI Adult&#10;(Census) Dataset, it results in a predictive model that does not lose much&#10;accuracy while achieving very close to equality of odds (Hardt, et al., 2016).&#10;The method is flexible and applicable to multiple definitions of fairness as&#10;well as a wide range of gradient-based learning models, including both&#10;regression and classification tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.15509" label="2006.15509">
        <attvalues>
          <attvalue for="0" value="BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant&#10;  Supervision" />
          <attvalue for="1" value="  We study the open-domain named entity recognition (NER) problem under distant&#10;supervision. The distant supervision, though does not require large amounts of&#10;manual annotations, yields highly incomplete and noisy distant labels via&#10;external knowledge bases. To address this challenge, we propose a new&#10;computational framework -- BOND, which leverages the power of pre-trained&#10;language models (e.g., BERT and RoBERTa) to improve the prediction performance&#10;of NER models. Specifically, we propose a two-stage training algorithm: In the&#10;first stage, we adapt the pre-trained language model to the NER tasks using the&#10;distant labels, which can significantly improve the recall and precision; In&#10;the second stage, we drop the distant labels, and propose a self-training&#10;approach to further improve the model performance. Thorough experiments on 5&#10;benchmark datasets demonstrate the superiority of BOND over existing distantly&#10;supervised NER methods. The code and distantly labeled data have been released&#10;in https://github.com/cliang1453/BOND.&#10;" />
          <attvalue for="2" value="&#10;&#10;Named Entity Recognition (NER) is the task of detecting mentions of real-world&#10;entities from text and classifying them into predefined types (e.g.,&#10;locations, persons, organizations). It is a core task in knowledge extraction&#10;and is important to various downstream applications such as user interest&#10;modeling \cite{karatay2015user}, question answering \cite{khalid2008impact}&#10;and dialogue systems \cite{bowden2018slugnerds}. Traditional approaches to&#10;NER mainly train statistical sequential models, such as Hidden Markov Model&#10;(HMM) \cite{zhou2002named} and Conditional Random Field (CRF)&#10;\cite{lafferty2001conditional} based on hand-crafted features. To alleviate the burden of designing hand-crafted features, deep learning models&#10;\cite{ma2016end,huang2015bidirectional} have been proposed for NER&#10;and shown strong performance. However, most deep learning methods rely on&#10;large amounts of labeled training data. As NER tasks require token-level&#10;labels, annotating a large number of documents can be expensive,&#10;time-consuming, and prone to human errors. In many real-life scenarios, the&#10;lack of labeled data has become the biggest bottleneck that prevents deep&#10;learning models from being adopted for NER tasks.&#10;&#10;To tackle the label scarcity issue, one approach is to use distant&#10;supervision to generate labels automatically. In distant supervision, the&#10;labeling procedure is to match the tokens in the target corpus with&#10;concepts in knowledge bases (e.g.&#10;Wikipedia and&#10;YAGO),&#10;which are usually easy and cheap to access. Nevertheless, the labels generated&#10;by the matching procedure suffer from two major challenges. The first challenge&#10;is incomplete annotation, which is caused by the limited coverage of&#10;existing knowledge bases. Take two common open-domain NER datasets as examples. From Table~\ref{tab:comp_distantlabel}, we find that the coverage of tokens on both datasets is very low (less than 60\%).This issue renders many entities mentions unmatched and produces many false-positive labels, which can hurt subsequent NER model training significantly. The second challenge is noisy annotation. The annotation is often noisy due to the labeling ambiguity -- the&#10;same entity mention can be mapped to multiple entity types in the knowledge&#10;bases. For instance, the entity mention 'Liverpool' can be mapped to&#10;both 'Liverpool City' (type: LOC) and 'Liverpool Football&#10;Club' (type: ORG) in the knowledge base. While existing methods&#10;adopt label induction methods based on type popularity, they will potentially&#10;lead to a matching bias toward popular types.&#10;Consequently, it can lead to many false-positive samples and hurt the&#10;performance of NER models. What's worse, there is often a trade-off between the label accuracy and coverage: generating the high-quality label requires setting strict matching rules which may not generalize well for all the tokens and thus reduce the coverage and introduce false-negative labels. On the other hand, increasing the coverage of annotation suffers from the increasing number of incorrect labels due to label ambiguity. From the above, it is still very challenging to generate high-quality labels with high coverage to the target corpus.&#10;&#10;Several studies have attempted to address the above challenges in&#10;distantly-supervised NER. To address the label incompleteness issue, some&#10;works adopt the partial annotation CRFs to consider all possible labels for&#10;unlabeled tokens~\cite{yang2018distantly,shang2018learning}, but they still&#10;require a considerable amount of annotated tokens or external tools. To&#10;address the label noise issue, \cite{ni2017weakly} \cite{ni2017weakly}&#10;use heuristic rules to filter out sentences with low matching quality. However,&#10;this filtering strategy improves the precision at the expense of lowering the&#10;recall. \cite{cao2019low} \cite{cao2019low} attempt to induce labels for&#10;entity mentions based on their occurrence popularity in the concept taxonomy,&#10;which can suffer from labeling bias and produce mislabeled data. Moreover,&#10;most of the methods mainly focus on NER tasks in specific domains (e.g. biomedical, chemistry, etc.) where the ambiguity of the named entity is very low. When the matching ambiguity issue is more severe, such methods will be less effective especially under open-domain scenarios. Till now, training open-domain NER models with distant&#10;supervision remains a challenging problem.&#10;&#10;We propose our model \bands, short for BERT-Assisted Open-Domain Named entity recognition with Distant Supervision, which learns accurate named entity taggers from distant&#10;supervision without any restriction on the domain or the content of the&#10;corpora. To address the challenges in learning from distant supervision, our approach leverages the power of pre-trained language&#10;models (e.g., ELMo \cite{peters2018deep}, BERT&#10;\cite{devlin2018bert}, XLnet \cite{yang2019xlnet}) which are particularly&#10;attractive to this task due to the following merits: First, they are very large neural networks trained with huge amounts of unlabeled data in&#10;a completely unsupervised manner, which can be cheaply obtained; Second, due to their massive sizes (usually&#10;having hundreds of millions or billions of parameters), they have strong&#10;expressive power to capture general semantics and syntactic information&#10;effectively. These language models have achieved state-of-the-art performance&#10;in many popular NLP benchmarks with appropriate fine-tuning&#10;~\cite{devlin2018bert,liu2019roberta,yang2019xlnet,Lan2020ALBERT,raffel2019exploring}, which demonstrates their strong ability in modeling the text data.&#10;&#10;To fully harness the power of pre-trained language models for tackling the two&#10;challenges, we propose a two-stage training framework. In the first stage, we&#10;fine-tune the RoBERTa model~\cite{liu2019roberta} with distantly-matched labels&#10;to essentially transfer the semantic knowledge in RoBERTa, which will improve the quality of prediction induced from distant&#10;supervision. &#10;It is worth noting that we adopt early stopping to prevent the model from overfitting to the incomplete annotated labels and significantly improve the recall. Then we use the RoBERTa model to predict a set of pseudo soft-labels for all data. In the second stage, we replace the distantly-matched labels with the pseudo soft-labels and design a teacher-student framework to further improve the recall. The student model is first initialized by the model learned in the first stage and trained using pseudo soft-labels.&#10;Then, we update the teacher model from the student model in the previous iteration to generate a new set of pseudo-labels for the next iteration to continue the training of the student model. &#10;This teacher-student framework enjoys the merit that it progressively improves the model confidence over data. In addition, we select samples based on the prediction confidence of the student model to further improve the quality of soft labels. In this way, we can better exploit both the knowledge base information and the language models and improve the model fitting.&#10;&#10;Our proposed method is closely related to low-resource NER and semi-supervised learning. We discuss more details in Section 5. We summarize the key contributions of our work as follows:&#10;&#10;$\bullet$ We demonstrate that the pre-trained language model can also provide additional semantic information during the training process and reduce the label noise for distantly-supervised named entity recognition. To the best of our knowledge, this is the first work that leverages the power of pre-trained language model for open-domain NER tasks with distant supervision.&#10;&#10;$\bullet$ We design a two-stage framework to fully exploit the power of language models in our task. Specifically, we refine the distant label iteratively with the language model in the first stage and improve the model fitting under the teacher-student framework in the second stage, which is able to address the challenge of noisy and incomplete annotation.&#10;&#10;$\bullet$ We conduct comprehensive experiments on 5 datasets for named entity recognition tasks with distant supervision. Our proposed method significantly outperforms state-of-the-art distantly supervised NER competitors in all 5 datasets (4 of which by significant margins). " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Named Entity Recognition, Artificial Intelligence, Distant Supervision, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="1805.03784" label="1805.03784">
        <attvalues>
          <attvalue for="0" value="SlugNERDS: A Named Entity Recognition Tool for Open Domain Dialogue&#10;  Systems" />
          <attvalue for="1" value="  In dialogue systems, the tasks of named entity recognition (NER) and named&#10;entity linking (NEL) are vital preprocessing steps for understanding user&#10;intent, especially in open domain interaction where we cannot rely on&#10;domain-specific inference. UCSC's effort as one of the funded teams in the 2017&#10;Amazon Alexa Prize Contest has yielded Slugbot, an open domain social bot,&#10;aimed at casual conversation. We discovered several challenges specifically&#10;associated with both NER and NEL when building Slugbot, such as that the NE&#10;labels are too coarse-grained or the entity types are not linked to a useful&#10;ontology. Moreover, we have discovered that traditional approaches do not&#10;perform well in our context: even systems designed to operate on tweets or&#10;other social media data do not work well in dialogue systems. In this paper, we&#10;introduce Slugbot's Named Entity Recognition for dialogue Systems (SlugNERDS),&#10;a NER and NEL tool which is optimized to address these issues. We describe two&#10;new resources that we are building as part of this work: SlugEntityDB and&#10;SchemaActuator. We believe these resources will be useful for the research&#10;community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1603.01354" label="1603.01354">
        <attvalues>
          <attvalue for="0" value="End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF" />
          <attvalue for="1" value="  State-of-the-art sequence labeling systems traditionally require large&#10;amounts of task-specific knowledge in the form of hand-crafted features and&#10;data pre-processing. In this paper, we introduce a novel neutral network&#10;architecture that benefits from both word- and character-level representations&#10;automatically, by using combination of bidirectional LSTM, CNN and CRF. Our&#10;system is truly end-to-end, requiring no feature engineering or data&#10;pre-processing, thus making it applicable to a wide range of sequence labeling&#10;tasks. We evaluate our system on two data sets for two sequence labeling tasks&#10;--- Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003&#10;corpus for named entity recognition (NER). We obtain state-of-the-art&#10;performance on both the two data --- 97.55\% accuracy for POS tagging and&#10;91.21\% F1 for NER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1508.01991" label="1508.01991">
        <attvalues>
          <attvalue for="0" value="Bidirectional LSTM-CRF Models for Sequence Tagging" />
          <attvalue for="1" value="  In this paper, we propose a variety of Long Short-Term Memory (LSTM) based&#10;models for sequence tagging. These models include LSTM networks, bidirectional&#10;LSTM (BI-LSTM) networks, LSTM with a Conditional Random Field (CRF) layer&#10;(LSTM-CRF) and bidirectional LSTM with a CRF layer (BI-LSTM-CRF). Our work is&#10;the first to apply a bidirectional LSTM CRF (denoted as BI-LSTM-CRF) model to&#10;NLP benchmark sequence tagging data sets. We show that the BI-LSTM-CRF model&#10;can efficiently use both past and future input features thanks to a&#10;bidirectional LSTM component. It can also use sentence level tag information&#10;thanks to a CRF layer. The BI-LSTM-CRF model can produce state of the art (or&#10;close to) accuracy on POS, chunking and NER data sets. In addition, it is&#10;robust and has less dependence on word embedding as compared to previous&#10;observations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.03599" label="1809.03599">
        <attvalues>
          <attvalue for="0" value="Learning Named Entity Tagger using Domain-Specific Dictionary" />
          <attvalue for="1" value="  Recent advances in deep neural models allow us to build reliable named entity&#10;recognition (NER) systems without handcrafting features. However, such methods&#10;require large amounts of manually-labeled training data. There have been&#10;efforts on replacing human annotations with distant supervision (in conjunction&#10;with external dictionaries), but the generated noisy labels pose significant&#10;challenges on learning effective neural models. Here we propose two neural&#10;models to suit noisy distant supervision from the dictionary. First, under the&#10;traditional sequence labeling framework, we propose a revised fuzzy CRF layer&#10;to handle tokens with multiple possible labels. After identifying the nature of&#10;noisy labels in distant supervision, we go beyond the traditional framework and&#10;propose a novel, more effective neural model AutoNER with a new Tie or Break&#10;scheme. In addition, we discuss how to refine distant supervision for better&#10;NER performance. Extensive experiments on three benchmark datasets demonstrate&#10;that AutoNER achieves the best performance when only using dictionaries with no&#10;additional human effort, and delivers competitive results with state-of-the-art&#10;supervised benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.02483" label="1707.02483">
        <attvalues>
          <attvalue for="0" value="Weakly Supervised Cross-Lingual Named Entity Recognition via Effective&#10;  Annotation and Representation Projection" />
          <attvalue for="1" value="  The state-of-the-art named entity recognition (NER) systems are supervised&#10;machine learning models that require large amounts of manually annotated data&#10;to achieve high accuracy. However, annotating NER data by human is expensive&#10;and time-consuming, and can be quite difficult for a new language. In this&#10;paper, we present two weakly supervised approaches for cross-lingual NER with&#10;no human annotation in a target language. The first approach is to create&#10;automatically labeled NER data for a target language via annotation projection&#10;on comparable corpora, where we develop a heuristic scheme that effectively&#10;selects good-quality projection-labeled data from noisy data. The second&#10;approach is to project distributed representations of words (word embeddings)&#10;from a target language to a source language, so that the source-language NER&#10;system can be applied to the target language without re-training. We also&#10;design two co-decoding schemes that effectively combine the outputs of the two&#10;projection-based approaches. We evaluate the performance of the proposed&#10;approaches on both in-house and open NER data for several target languages. The&#10;results show that the combined systems outperform three other weakly supervised&#10;approaches on the CoNLL data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.21515" label="2407.21515">
        <attvalues>
          <attvalue for="0" value="Learning Effective Representations for Retrieval Using Self-Distillation&#10;  with Adaptive Relevance Margins" />
          <attvalue for="1" value="  Representation-based retrieval models, so-called biencoders, estimate the&#10;relevance of a document to a query by calculating the similarity of their&#10;respective embeddings. Current state-of-the-art biencoders are trained using an&#10;expensive training regime involving knowledge distillation from a teacher model&#10;and batch-sampling. Instead of relying on a teacher model, we contribute a&#10;novel parameter-free loss function for self-supervision that exploits the&#10;pre-trained language modeling capabilities of the encoder model as a training&#10;signal, eliminating the need for batch sampling by performing implicit hard&#10;negative mining. We investigate the capabilities of our proposed approach&#10;through extensive ablation studies, demonstrating that self-distillation can&#10;match the effectiveness of teacher distillation using only 13.5% of the data,&#10;while offering a speedup in training time between 3x and 15x compared to&#10;parametrized losses. Code and data is made openly available.&#10;" />
          <attvalue for="2" value="&#10;&#10;In information retrieval, trans\-for\-mer-based bi-encoders are used as effective retrieval models. They estimate a document's relevance to a query by computing the similarity of their embeddings. Queries and documents are represented independently, which allows for pre-computing and indexing all document representations offline and computing only the query representation at retrieval time. bi-encoders are thus appealing in practice since they can be easily scaled. In contrast, trans\-for\-mer-based cross-encoders are a type of model that estimates relevance jointly by computing a score for each document-query pair at retrieval time, achieving higher effectiveness than bi-encoders at the expense of a higher computational cost and inference latency. They are thus usally reserved for multi-stage reranking, such that only the top-k documents of an initial ranking produced by a more efficient bi-encoder are re-ranked.&#10;&#10;However, highly effective bi-encoders come at the cost of a highly expensive training regime based on knowledge distillation. Figure~\ref{fig:conceptual-overview} (left) illustrates this process, where the relevance of each training sample is first estimated by a cross-encoder teacher model. The estimated relevance scores are first used for a computationally intensive batch sampling process, where an optimal distribution of relevance scores within each batch is ensured according to the teacher model, which then provides supervision for training a bi-encoder student model with a margin-based loss function. While this knowledge distillation setup yields highly effective bi-encoder ranking models, we highlight three drawbacks: &#10;\begin{enumerate*}[label=\bfseries(\arabic*)]&#10;\item &#10;A teacher model must be available for the desired dataset and domain. cross-encoder models are usually chosen as teacher models to maximize effectiveness~\cite{hofstatter:2020}. Training a cross-encoder model for knowledge distillation is computationally expensive and may not be possible for domains with little ground-truth ranking data.&#10;\item &#10;Teacher scores must be inferred for every training sample, requiring a forward pass. This adds inference cost onto a process that is already prohibitively expensive. Improving training efficiency is crucial for information retrieval, as the majority of energy spent over the lifetime of a model, especially models that are efficient at retrieval time, goes into training~\cite{scells:2022}.&#10;\item &#10;Batch sampling prohibits continuous learning on new data. The current best batch sampling procedures rely on clustering the training data set~\cite{hofstatter:2020}. A static training dataset is thus required, rendering it challenging to adapt such approaches in settings where training data is collected continuously, such as online learning-to-rank.&#10;\end{enumerate*}&#10;&#10;In this paper, we propose a novel training regime and loss function for bi-encoders (Figure~\ref{fig:conceptual-overview}, right) that eliminates the complexities of previous approaches while maintaining competitive effectiveness. Instead of using a teacher model, we leverage the pre-trained text similarity capabilities of the encoder model to provide a supervision signal in a self-distillation setup. This eliminates the need for expensive exhaustive inference over the training dataset, does not rely on batch sampling techniques, and is highly data efficient. We then conduct systematic ablation studies which demonstrate that self-distillation can match the effectiveness of teacher-distillation based training regimes, with nDCG@10 and Recall@1000 scores of no significant difference, while using only 13.5\% of the data. Finally, our proposed loss function is hyperparameter free, eliminating the need for expensive hyperparameter tuning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Information Retrieval Systems, Language Model Training, Artificial Intelligence, Mathematics, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2010.02666" label="2010.02666">
        <attvalues>
          <attvalue for="0" value="Improving Efficient Neural Ranking Models with Cross-Architecture&#10;  Knowledge Distillation" />
          <attvalue for="1" value="  Retrieval and ranking models are the backbone of many applications such as&#10;web search, open domain QA, or text-based recommender systems. The latency of&#10;neural ranking models at query time is largely dependent on the architecture&#10;and deliberate choices by their designers to trade-off effectiveness for higher&#10;efficiency. This focus on low query latency of a rising number of efficient&#10;ranking architectures make them feasible for production deployment. In machine&#10;learning an increasingly common approach to close the effectiveness gap of more&#10;efficient models is to apply knowledge distillation from a large teacher model&#10;to a smaller student model. We find that different ranking architectures tend&#10;to produce output scores in different magnitudes. Based on this finding, we&#10;propose a cross-architecture training procedure with a margin focused loss&#10;(Margin-MSE), that adapts knowledge distillation to the varying score output&#10;distributions of different BERT and non-BERT passage ranking architectures. We&#10;apply the teachable information as additional fine-grained labels to existing&#10;training triples of the MSMARCO-Passage collection. We evaluate our procedure&#10;of distilling knowledge from state-of-the-art concatenated BERT models to four&#10;different efficient architectures (TK, ColBERT, PreTT, and a BERT CLS dot&#10;product model). We show that across our evaluated architectures our Margin-MSE&#10;knowledge distillation significantly improves re-ranking effectiveness without&#10;compromising their efficiency. Additionally, we show our general distillation&#10;method to improve nearest neighbor based index retrieval with the BERT dot&#10;product model, offering competitive results with specialized and much more&#10;costly training methods. To benefit the community, we publish the teacher-score&#10;training files in a ready-to-use package.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.09770" label="2208.09770">
        <attvalues>
          <attvalue for="0" value="Z-Code++: A Pre-trained Language Model Optimized for Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  This paper presents Z-Code++, a new pre-trained language model optimized for&#10;abstractive text summarization. The model extends the state of the art&#10;encoder-decoder model using three techniques. First, we use a two-phase&#10;pre-training process to improve model's performance on low-resource&#10;summarization tasks. The model is first pre-trained using text corpora for&#10;language understanding, and then is continually pre-trained on summarization&#10;corpora for grounded text generation. Second, we replace self-attention layers&#10;in the encoder with disentangled attention layers, where each word is&#10;represented using two vectors that encode its content and position,&#10;respectively. Third, we use fusion-in-encoder, a simple yet effective method of&#10;encoding long sequences in a hierarchical manner. Z-Code++ creates new state of&#10;the art on 9 out of 13 text summarization tasks across 5 languages. Our model&#10;is parameter-efficient in that it outperforms the 600x larger PaLM-540B on&#10;XSum, and the finetuned 200x larger GPT3-175B on SAMSum. In zero-shot and&#10;few-shot settings, our model substantially outperforms the competing models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.04096" label="2011.04096">
        <attvalues>
          <attvalue for="0" value="Metrics also Disagree in the Low Scoring Range: Revisiting Summarization&#10;  Evaluation Metrics" />
          <attvalue for="1" value="  In text summarization, evaluating the efficacy of automatic metrics without&#10;human judgments has become recently popular. One exemplar work concludes that&#10;automatic metrics strongly disagree when ranking high-scoring summaries. In&#10;this paper, we revisit their experiments and find that their observations stem&#10;from the fact that metrics disagree in ranking summaries from any narrow&#10;scoring range. We hypothesize that this may be because summaries are similar to&#10;each other in a narrow scoring range and are thus, difficult to rank. Apart&#10;from the width of the scoring range of summaries, we analyze three other&#10;properties that impact inter-metric agreement - Ease of Summarization,&#10;Abstractiveness, and Coverage. To encourage reproducible research, we make all&#10;our analysis code and data publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08542" label="2112.08542">
        <attvalues>
          <attvalue for="0" value="QAFactEval: Improved QA-Based Factual Consistency Evaluation for&#10;  Summarization" />
          <attvalue for="1" value="  Factual consistency is an essential quality of text summarization models in&#10;practical settings. Existing work in evaluating this dimension can be broadly&#10;categorized into two lines of research, entailment-based and question answering&#10;(QA)-based metrics, and different experimental setups often lead to contrasting&#10;conclusions as to which paradigm performs the best. In this work, we conduct an&#10;extensive comparison of entailment and QA-based metrics, demonstrating that&#10;carefully choosing the components of a QA-based metric, especially question&#10;generation and answerability classification, is critical to performance.&#10;Building on those insights, we propose an optimized metric, which we call&#10;QAFactEval, that leads to a 14% average improvement over previous QA-based&#10;metrics on the SummaC factual consistency benchmark, and also outperforms the&#10;best-performing entailment-based metric. Moreover, we find that QA-based and&#10;entailment-based metrics can offer complementary signals and be combined into a&#10;single metric for a further performance boost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.04281" label="2012.04281">
        <attvalues>
          <attvalue for="0" value="CTRLsum: Towards Generic Controllable Text Summarization" />
          <attvalue for="1" value="  Current summarization systems yield generic summaries that are disconnected&#10;from users' preferences and expectations. To address this limitation, we&#10;present CTRLsum, a novel framework for controllable summarization. Our approach&#10;enables users to control multiple aspects of generated summaries by interacting&#10;with the summarization system through textual input in the form of a set of&#10;keywords or descriptive prompts. Using a single unified model, CTRLsum is able&#10;to achieve a broad scope of summary manipulation at inference time without&#10;requiring additional human annotations or pre-defining a set of control aspects&#10;during training. We quantitatively demonstrate the effectiveness of our&#10;approach on three domains of summarization datasets and five control aspects:&#10;1) entity-centric and 2) length-controllable summarization, 3) contribution&#10;summarization on scientific papers, 4) invention purpose summarization on&#10;patent filings, and 5) question-guided summarization on news articles in a&#10;reading comprehension setting. Moreover, when used in a standard, uncontrolled&#10;summarization setting, CTRLsum achieves state-of-the-art results on the&#10;CNN/DailyMail dataset. Code and model checkpoints are available at&#10;https://github.com/salesforce/ctrl-sum&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.20053" label="2405.20053">
        <attvalues>
          <attvalue for="0" value="Would I Lie To You? Inference Time Alignment of Language Models using&#10;  Direct Preference Heads" />
          <attvalue for="1" value="  Pre-trained Language Models (LMs) exhibit strong zero-shot and in-context&#10;learning capabilities; however, their behaviors are often difficult to control.&#10;By utilizing Reinforcement Learning from Human Feedback (RLHF), it is possible&#10;to fine-tune unsupervised LMs to follow instructions and produce outputs that&#10;reflect human preferences. Despite its benefits, RLHF has been shown to&#10;potentially harm a language model's reasoning capabilities and introduce&#10;artifacts such as hallucinations where the model may fabricate facts. To&#10;address this issue we introduce Direct Preference Heads (DPH), a fine-tuning&#10;framework that enables LMs to learn human preference signals through an&#10;auxiliary reward head without directly affecting the output distribution of the&#10;language modeling head. We perform a theoretical analysis of our objective&#10;function and find strong ties to Conservative Direct Preference Optimization&#10;(cDPO). Finally we evaluate our models on GLUE, RACE, and the GPT4All&#10;evaluation suite and demonstrate that our method produces models which achieve&#10;higher scores than those fine-tuned with Supervised Fine-Tuning (SFT) or Direct&#10;Preference Optimization (DPO) alone.&#10;" />
          <attvalue for="2" value="&#10;Reinforcement Learning from Human Feedback (RLHF) is a technique that can be used to align an agent --- such as a Large Language Model (LLM) --- to human preferences and lead to more truthful, more helpful, less harmful and more preferred outputs \cite{ouyang2022training}. Proximal Policy Optimization (PPO) \cite{schulman2017proximal} and Direct Preference Optimization (DPO) \cite{rafailov2023direct} are two such aligment techniques which have been extensively used to improve the quality of LLM outputs, leading to instruction following agents or chat assistants which are quickly approaching human-baselines in a variety of knowledge and reasoning tasks \cite{open-llm-leaderboard, clark2018think, zellers2019hellaswag, hendrycks2021measuring, lin2022truthfulqa, DBLP:journals/corr/abs-1907-10641, DBLP:journals/corr/abs-2110-14168}.&#10;&#10;However, recent research has shown that RLHF may actually hurt an LLM's reasoning abilities rather than improving it. One study \cite{bekbayev2023poison} discovered that performing alignment during the Supervised Fine-Tuning (SFT) stage of training may lead to worse performance on reasoning benchmarks, and another \cite{bai2022training} discovered that SFT alone outperforms RLHF for smaller models with the benefits of RLHF only emerging for models with more than 1 Billion parameters. Ouyang et al. \cite{ouyang2022training} also reports an increased tendency for RLHF models to make up information in closed domain tasks (``hallucination'') compared to models trained with SFT alone.&#10;&#10;To combat the the risk of RLHF compromising the abilities of an LLM in favor of producing preferable outputs we introduce Direct Preference Heads (DPH), a novel feature based approach that optimises a reward score produced by the LLM rather than optimising the logits produced by language modelling head. DPH can be used in combination with (or without) existing alignment techniques to allow language models to self-evaluate outputs sampled at inference time and select the highest scoring candidate.&#10;&#10;We evaluate the performance of DPH using an efficient 551M parameter LM on a variety of commonsense reasoning and Natural Language Understanding (NLU) tasks. All code used to train our models is available on \anon{\href{ and we release our model weights on \anon{\href{ Face}}.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Preference Optimization, Machine Learning, Computer Science, Linguistics, Language Model Control, Artificial Intelligence, Natural Language Processing, Reinforcement Learning" />
        </attvalues>
      </node>
      <node id="2404.14994" label="2404.14994">
        <attvalues>
          <attvalue for="0" value="Transformers Can Represent $n$-gram Language Models" />
          <attvalue for="1" value="  Existing work has analyzed the representational capacity of the transformer&#10;architecture by means of formal models of computation. However, the focus so&#10;far has been on analyzing the architecture in terms of language&#10;\emph{acceptance}. We contend that this is an ill-suited problem in the study&#10;of \emph{language models} (LMs), which are definitionally \emph{probability&#10;distributions} over strings. In this paper, we focus on the relationship&#10;between transformer LMs and $n$-gram LMs, a simple and historically relevant&#10;class of language models. We show that transformer LMs using the hard or sparse&#10;attention mechanisms can exactly represent any $n$-gram LM, giving us a&#10;concrete lower bound on their probabilistic representational capacity. This&#10;provides a first step towards understanding the mechanisms that transformer LMs&#10;can use to represent probability distributions over strings.&#10;" />
          <attvalue for="2" value=" &#10;Neural language models (LMs) have become the backbone of many NLP systems. &#10;Their widespread adoption has prompted a plethora of theoretical work investigating what they can and cannot do by studying their representational capacity.&#10;Most state-of-the-art LMs are based on the transformer architecture \cite{Vaswani2017}, whose theoretical abilities and limitations have been studied extensively; see, e.g., the survey by \cite{strobl2023transformers}. &#10;But, many questions remain unanswered.&#10;Most existing work studies the architecture in terms of binary language recognition.&#10;This introduces a category error between the object of study---an LM, which is definitionally a distribution over strings---and the theoretical abstraction---a set of strings.&#10;To amend this discrepancy, we ask: What classes of probability distributions over strings can transformer LMs represent?\looseness=-1&#10;&#10;Formal models of probabilistic computation provide a natural, well-understood, and precise framework for studying the classes of probability distributions language models can represent.&#10;Traditionally, the representational capacity of neural networks, both in terms of lower bounds (what they can provably do) as well as upper bounds (what they can provably not do), has been studied in terms of Boolean sequential models of computation, such as finite-state automata and Turing machines \cite[e.g.,][]{Kleene1956,Minsky1954,Siegelmann1992OnTC,hao-etal-2018-context,merrill-2019-sequential,merrill-etal-2020-formal,merrill-etal-2022-saturated,merrill2022extracting}.&#10;Recent work has extended this paradigm to work with probabilistic models of computation \cite{svete-cotterell-2023-recurrent,nowak-etal-2023-representational}, but so far only for LMs based on recurrent neural networks.\looseness=-1&#10;&#10;However, the sequential nature of classical models makes the connection to the inherently parallelizable transformer architecture less straightforward and has resulted in a number of results upper-bounding their representational capacity \cite{hahn-2020-theoretical,bhattamishra-etal-2020-on-ability,chiang-cholak-2022-overcoming,hao-etal-2022-formal,merrill-sabharwal-2023-parallelism}.&#10;We connect transformer LMs to a classical class of LMs that lend themselves particularly well to parallelized computations: \ngram LMs.&#10;We show that both hard as well as sparse attention transformer LMs can represent any \ngram LM (\ref{thm:transformers-n-gram-label,thm:transformers-n-gram-label-sparse}).&#10;This gives us a concrete lower bound on their probabilistic representational capacity.&#10;We also study the role of the number of heads (\ref{thm:transformers-n-gram-label}) and the number of layers (\ref{thm:transformers-n-gram-label-multi-layer}), illustrating a trade-off between the number of heads, layers, and the complexity of the non-linear transformations required for the simulation of \ngram LMs.&#10;Altogether, these results offer a step towards understanding the probabilistic representational capacity of transformer LMs and the mechanisms they might employ to implement formal models of computation.\looseness=-1&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Probabilistic Representations, Mathematics, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="1906.01615" label="1906.01615">
        <attvalues>
          <attvalue for="0" value="Sequential Neural Networks as Automata" />
          <attvalue for="1" value="  This work attempts to explain the types of computation that neural networks&#10;can perform by relating them to automata. We first define what it means for a&#10;real-time network with bounded precision to accept a language. A measure of&#10;network memory follows from this definition. We then characterize the classes&#10;of languages acceptable by various recurrent networks, attention, and&#10;convolutional networks. We find that LSTMs function like counter machines and&#10;relate convolutional networks to the subregular hierarchy. Overall, this work&#10;attempts to increase our understanding and ability to interpret neural networks&#10;through the lens of theory. These theoretical insights help explain neural&#10;computation, as well as the relationship between neural networks and natural&#10;language grammar.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.08500" label="2004.08500">
        <attvalues>
          <attvalue for="0" value="A Formal Hierarchy of RNN Architectures" />
          <attvalue for="1" value="  We develop a formal hierarchy of the expressive capacity of RNN&#10;architectures. The hierarchy is based on two formal properties: space&#10;complexity, which measures the RNN's memory, and rational recurrence, defined&#10;as whether the recurrent update can be described by a weighted finite-state&#10;machine. We place several RNN variants within this hierarchy. For example, we&#10;prove the LSTM is not rational, which formally separates it from the related&#10;QRNN (Bradbury et al., 2016). We also show how these models' expressive&#10;capacity is expanded by stacking multiple layers or composing them with&#10;different pooling functions. Our results build on the theory of &quot;saturated&quot;&#10;RNNs (Merrill, 2019). While formally extending these findings to unsaturated&#10;RNNs is left to future work, we hypothesize that the practical learnable&#10;capacity of unsaturated RNNs obeys a similar hierarchy. Experimental findings&#10;from training unsaturated networks on formal languages support this conjecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.16213" label="2106.16213">
        <attvalues>
          <attvalue for="0" value="Saturated Transformers are Constant-Depth Threshold Circuits" />
          <attvalue for="1" value="  Transformers have become a standard neural network architecture for many NLP&#10;problems, motivating theoretical analysis of their power in terms of formal&#10;languages. Recent work has shown that transformers with hard attention are&#10;quite limited in power (Hahn, 2020), as they can be simulated by constant-depth&#10;AND/OR circuits (Hao et al. 2021). However, hard attention is a strong&#10;assumption, which may complicate the relevance of these results in practice. In&#10;this work, we analyze the circuit complexity of transformers with saturated&#10;attention: a generalization of hard attention that more closely captures the&#10;attention patterns learnable in practical transformers. We first show that&#10;saturated transformers transcend the known limitations of hard-attention&#10;transformers. We then prove saturated transformers with floating-point values&#10;can be simulated by constant-depth threshold circuits, giving the class&#10;$\mathsf{TC}^0$ as an upper bound on the formal languages they recognize.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.12451" label="2201.12451">
        <attvalues>
          <attvalue for="0" value="Extracting Finite Automata from RNNs Using State Merging" />
          <attvalue for="1" value="  One way to interpret the behavior of a blackbox recurrent neural network&#10;(RNN) is to extract from it a more interpretable discrete computational model,&#10;like a finite state machine, that captures its behavior. In this work, we&#10;propose a new method for extracting finite automata from RNNs inspired by the&#10;state merging paradigm from grammatical inference. We demonstrate the&#10;effectiveness of our method on the Tomita languages benchmark, where we find&#10;that it is able to extract faithful automata from RNNs trained on all languages&#10;in the benchmark. We find that extraction performance is aided by the number of&#10;data provided during the extraction process, as well as, curiously, whether the&#10;RNN model is trained for additional epochs after perfectly learning its target&#10;language. We use our method to analyze this phenomenon, finding that training&#10;beyond convergence is useful because it leads to compression of the internal&#10;state space of the RNN. This finding demonstrates how our method can be used&#10;for interpretability and analysis of trained RNN models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05161" label="2310.05161">
        <attvalues>
          <attvalue for="0" value="Recurrent Neural Language Models as Probabilistic Finite-state Automata" />
          <attvalue for="1" value="  Studying language models (LMs) in terms of well-understood formalisms allows&#10;us to precisely characterize their abilities and limitations. Previous work has&#10;investigated the representational capacity of recurrent neural network (RNN)&#10;LMs in terms of their capacity to recognize unweighted formal languages.&#10;However, LMs do not describe unweighted formal languages -- rather, they define&#10;\emph{probability distributions} over strings. In this work, we study what&#10;classes of such probability distributions RNN LMs can represent, which allows&#10;us to make more direct statements about their capabilities. We show that simple&#10;RNNs are equivalent to a subclass of probabilistic finite-state automata, and&#10;can thus model a strict subset of probability distributions expressible by&#10;finite-state models. Furthermore, we study the space complexity of representing&#10;finite-state LMs with RNNs. We show that, to represent an arbitrary&#10;deterministic finite-state LM with $N$ states over an alphabet $\alphabet$, an&#10;RNN requires $\Omega\left(N |\Sigma|\right)$ neurons. These results present a&#10;first step towards characterizing the classes of distributions RNN LMs can&#10;represent and thus help us understand their capabilities and limitations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12942" label="2310.12942">
        <attvalues>
          <attvalue for="0" value="On the Representational Capacity of Recurrent Neural Language Models" />
          <attvalue for="1" value="  This work investigates the computational expressivity of language models&#10;(LMs) based on recurrent neural networks (RNNs). Siegelmann and Sontag (1992)&#10;famously showed that RNNs with rational weights and hidden states and unbounded&#10;computation time are Turing complete. However, LMs define weightings over&#10;strings in addition to just (unweighted) language membership and the analysis&#10;of the computational power of RNN LMs (RLMs) should reflect this. We extend the&#10;Turing completeness result to the probabilistic case, showing how a rationally&#10;weighted RLM with unbounded computation time can simulate any deterministic&#10;probabilistic Turing machine (PTM) with rationally weighted transitions. Since,&#10;in practice, RLMs work in real-time, processing a symbol at every time step, we&#10;treat the above result as an upper bound on the expressivity of RLMs. We also&#10;provide a lower bound by showing that under the restriction to real-time&#10;computation, such models can simulate deterministic real-time rational PTMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.06755" label="1906.06755">
        <attvalues>
          <attvalue for="0" value="Theoretical Limitations of Self-Attention in Neural Sequence Models" />
          <attvalue for="1" value="  Transformers are emerging as the new workhorse of NLP, showing great success&#10;across tasks. Unlike LSTMs, transformers process input sequences entirely&#10;through self-attention. Previous work has suggested that the computational&#10;capabilities of self-attention to process hierarchical structures are limited.&#10;In this work, we mathematically investigate the computational power of&#10;self-attention to model formal languages. Across both soft and hard attention,&#10;we show strong theoretical limitations of the computational abilities of&#10;self-attention, finding that it cannot model periodic finite-state languages,&#10;nor hierarchical structure, unless the number of layers or heads increases with&#10;input length. These limitations seem surprising given the practical success of&#10;self-attention and the prominent role assigned to hierarchical structure in&#10;linguistics, suggesting that natural language can be approximated well with&#10;models that are too weak for the formal languages typically assumed in&#10;theoretical linguistics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.11264" label="2009.11264">
        <attvalues>
          <attvalue for="0" value="On the Ability and Limitations of Transformers to Recognize Formal&#10;  Languages" />
          <attvalue for="1" value="  Transformers have supplanted recurrent models in a large number of NLP tasks.&#10;However, the differences in their abilities to model different syntactic&#10;properties remain largely unknown. Past works suggest that LSTMs generalize&#10;very well on regular languages and have close connections with counter&#10;languages. In this work, we systematically study the ability of Transformers to&#10;model such languages as well as the role of its individual components in doing&#10;so. We first provide a construction of Transformers for a subclass of counter&#10;languages, including well-studied languages such as n-ary Boolean Expressions,&#10;Dyck-1, and its generalizations. In experiments, we find that Transformers do&#10;well on this subclass, and their learned mechanism strongly correlates with our&#10;construction. Perhaps surprisingly, in contrast to LSTMs, Transformers do well&#10;only on a subset of regular languages with degrading performance as we make&#10;languages more complex according to a well-known measure of complexity. Our&#10;analysis also provides insights on the role of self-attention mechanism in&#10;modeling certain behaviors and the influence of positional encoding schemes on&#10;the learning and generalization abilities of the model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.12172" label="2202.12172">
        <attvalues>
          <attvalue for="0" value="Overcoming a Theoretical Limitation of Self-Attention" />
          <attvalue for="1" value="  Although transformers are remarkably effective for many tasks, there are some&#10;surprisingly easy-looking regular languages that they struggle with. Hahn shows&#10;that for languages where acceptance depends on a single input symbol, a&#10;transformer's classification decisions become less and less confident (that is,&#10;with cross-entropy approaching 1 bit per string) as input strings get longer&#10;and longer. We examine this limitation using two languages: PARITY, the&#10;language of bit strings with an odd number of 1s, and FIRST, the language of&#10;bit strings starting with a 1. We demonstrate three ways of overcoming the&#10;limitation suggested by Hahn's lemma. First, we settle an open question by&#10;constructing a transformer that recognizes PARITY with perfect accuracy, and&#10;similarly for FIRST. Second, we use layer normalization to bring the&#10;cross-entropy of both models arbitrarily close to zero. Third, when&#10;transformers need to focus on a single position, as for FIRST, we find that&#10;they can fail to generalize to longer strings; we offer a simple remedy to this&#10;problem that also improves length generalization in machine translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.06618" label="2204.06618">
        <attvalues>
          <attvalue for="0" value="Formal Language Recognition by Hard Attention Transformers: Perspectives&#10;  from Circuit Complexity" />
          <attvalue for="1" value="  This paper analyzes three formal models of Transformer encoders that differ&#10;in the form of their self-attention mechanism: unique hard attention (UHAT);&#10;generalized unique hard attention (GUHAT), which generalizes UHAT; and&#10;averaging hard attention (AHAT). We show that UHAT and GUHAT Transformers,&#10;viewed as string acceptors, can only recognize formal languages in the&#10;complexity class AC$^0$, the class of languages recognizable by families of&#10;Boolean circuits of constant depth and polynomial size. This upper bound&#10;subsumes Hahn's (2020) results that GUHAT cannot recognize the DYCK languages&#10;or the PARITY language, since those languages are outside AC$^0$ (Furst et al.,&#10;1984). In contrast, the non-AC$^0$ languages MAJORITY and DYCK-1 are&#10;recognizable by AHAT networks, implying that AHAT can recognize languages that&#10;UHAT and GUHAT cannot.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.00729" label="2207.00729">
        <attvalues>
          <attvalue for="0" value="The Parallelism Tradeoff: Limitations of Log-Precision Transformers" />
          <attvalue for="1" value="  Despite their omnipresence in modern NLP, characterizing the computational&#10;power of transformer neural nets remains an interesting open question. We prove&#10;that transformers whose arithmetic precision is logarithmic in the number of&#10;input tokens (and whose feedforward nets are computable using space linear in&#10;their input) can be simulated by constant-depth logspace-uniform threshold&#10;circuits. This provides insight on the power of transformers using known&#10;results in complexity theory. For example, if $\mathsf L \neq \mathsf P$ (i.e.,&#10;not all poly-time problems can be solved using logarithmic space), then&#10;transformers cannot even accurately solve linear equalities or check membership&#10;in an arbitrary context-free grammar with empty productions. Our result&#10;intuitively emerges from the transformer architecture's high parallelizability.&#10;We thus speculatively introduce the idea of a fundamental parallelism tradeoff:&#10;any model architecture as parallelizable as the transformer will obey&#10;limitations similar to it. Since parallelism is key to training models at&#10;massive scale, this suggests a potential inherent weakness of the scaling&#10;paradigm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.09664" label="2302.09664">
        <attvalues>
          <attvalue for="0" value="Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation&#10;  in Natural Language Generation" />
          <attvalue for="1" value="  We introduce a method to measure uncertainty in large language models. For&#10;tasks like question answering, it is essential to know when we can trust the&#10;natural language outputs of foundation models. We show that measuring&#10;uncertainty in natural language is challenging because of &quot;semantic&#10;equivalence&quot; -- different sentences can mean the same thing. To overcome these&#10;challenges we introduce semantic entropy -- an entropy which incorporates&#10;linguistic invariances created by shared meanings. Our method is unsupervised,&#10;uses only a single model, and requires no modifications to off-the-shelf&#10;language models. In comprehensive ablation studies we show that the semantic&#10;entropy is more predictive of model accuracy on question answering data sets&#10;than comparable baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Prior work on uncertainty in foundation models for NLP has largely focused on the calibration of classifiers \cite{jiang2021can, desai2020calibration} and text regressors \cite{glushkova2021uncertainty, wang2022uncertainty}. These settings, are analogous to classification or regression settings in other modalities like vision, and conventional uncertainty measures like MC dropout or Deep Ensembles can be applied without modification (see \ref{section_background} for a discussion of uncertainty in deep learning in general). As we argue in \ref{section_challenges_in_nlg}, generative natural language poses important further challenges. &#10;\cite{jiang2021can} do examine calibration in generative question answering and find only a weak correlation between the log-likelihood models assign to their answer and the answer's correctness. In \ref{s:experiments} we explain however why semantic equivalence in natural language makes calibration a problematic evaluation for generative language models. Reliable uncertainty can be useful on downstream tasks such as graph semantic parsing \cite{lin2022towards}.&#10;&#10;Some research has addressed uncertainty or calibration in NLG either by prompting the models to evaluate their own generations or by fine-tuning the generating model to predict its uncertainty \cite{mielke2020linguistic,lin2022teaching, kadavath2022language}.&#10;These methods need further training and supervision.&#10;Because they need additional training and supervision, they are hard to reproduce, expensive to create, and have been shown to be sensitive to distribution shift.&#10;For example, we were unable to implement one proposal by \cite{kadavath2022language} to train a language model to directly predict confidence due to hardware limitations.&#10;Our unsupervised method which uses models `off-the-shelf' avoids these limitations.&#10;&#10;Many of the issues that make probabilistic uncertainty estimation in NLG difficult also make automatic evaluation of NLG difficult.&#10;\cite{ott2018analyzing}, for instance, study how the performance of machine translation models suffers because one sentence can be translated in multiple ways.&#10;Similarly, \cite{sai2022survey} discuss how paraphrase detection can be used to evaluate NLG and other related methods might transfer to uncertainty estimation.&#10;&#10;Automatic paraphrase identification can be based on comparing lexical features of two given sequences \cite{fernando2008semantic, issa2018abstract} or on measuring the similarity between the embeddings of the two sequences \cite{yu2014deep, socher2011dynamic}. Recently, however, SotA paraphrase identification approaches have primarily used BERT-based models to classify pairs of sequences into the classes paraphrases and not paraphrases \cite{he2020realformer, tay2021charformer}. The idea of formalising semantic equivalence via textual entailment has a long history in linguistics \cite{culicover1968paraphrase} and NLP \cite{pado2009measuring, androutsopoulos2010survey}. Transformer-based paraphrase detection models such as EFL \cite{wang2021entailment} achieve SotA performance on paraphrase detection benchmarks such as Quora Question Pairs \cite{wang2017bilateral}.&#10;&#10;" />
          <attvalue for="4" value="Language Model Uncertainty, Information Theory, Computer Science, Linguistics, Artificial Intelligence, Mathematics, Semantic Entropy Measurement, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.00014" label="2310.00014">
        <attvalues>
          <attvalue for="0" value="Fewer-token Neural Speech Codec with Time-invariant Codes" />
          <attvalue for="1" value="  Language model based text-to-speech (TTS) models, like VALL-E, have gained&#10;attention for their outstanding in-context learning capability in zero-shot&#10;scenarios. Neural speech codec is a critical component of these models, which&#10;can convert speech into discrete token representations. However, excessive&#10;token sequences from the codec may negatively affect prediction accuracy and&#10;restrict the progression of Language model based TTS models. To address this&#10;issue, this paper proposes a novel neural speech codec with time-invariant&#10;codes named TiCodec. By encoding and quantizing time-invariant information into&#10;a separate code, TiCodec can reduce the amount of frame-level information that&#10;needs encoding, effectively decreasing the number of tokens as codes of speech.&#10;Furthermore, this paper introduces a time-invariant encoding consistency loss&#10;to enhance the consistency of time-invariant code within an utterance and force&#10;it to capture more global information, which can benefit the zero-shot TTS&#10;task. Experimental results demonstrate that TiCodec can not only enhance the&#10;quality of reconstruction speech with fewer tokens but also increase the&#10;similarity and naturalness, as well as reduce the word error rate of the&#10;synthesized speech by the TTS model.&#10;" />
          <attvalue for="2" value="&#10;Recently, large language models have demonstrated remarkable performance on zero-shot text-to-speech (TTS) tasks such as VALL-E \cite{wang2023neural}, SPEAR-TTS \cite{kharitonov2023speak}, and SoundStorm \cite{borsos2023soundstorm}. &#10;VALL-E uses discrete tokens derived from Encodec \cite{defossez2022high} as a representation of speech, and then trains an autoregressive (AR) language model and a non-autoregressive (NAR) language model to generate tokens from the first quantizer and the other seven quantizers separately.&#10;It can synthesize high-quality personalized speech by using a short recording of an unknown speaker as an acoustic prompt. &#10;However, the high-quality reconstruction of speech requires multiple frame-level token sequences, which affects the inference speed and robustness, and restricts the model structure and training methods of language model based TTS models.&#10;Therefore, how to represent speech better with fewer tokens has become a core issue.&#10;&#10;Neural speech codec is an important method to acquire discrete token representations of speech.&#10;To improve the compression rate and reduce the number of tokens, more and more research is focusing on neural speech codec \cite{kumar2023high, xu23_interspeech, zheng23c_interspeech}.&#10;Kleijn et al. \cite{kleijn2018wavenet} proposed a low-rate speech coding architecture based on the WaveNet \cite{vanwavenet} decoder.&#10;Lyra \cite{kleijn2021generative} encodes quantized mel-spectrogram features of speech, and then decodes them with WaveGRU \cite{chung2014empirical}. &#10;Subsequently, end-to-end neural speech codecs have been introduced.&#10;Grbacea et al. \cite{garbacea2019low} used the discretized latent representations proposed in VQVAE \cite{van2017neural} as conditioning for the WaveNet decoder. &#10;After that, SoundStream \cite{zeghidour2021soundstream}, as a fully convolutional end-to-end universal audio codec model, was proposed, extending the VQVAE vector quantizer to a residual vector quantizer. &#10;Following that, Encodec \cite{defossez2022high} introduced a spectrogram-only adversarial loss, a novel gradient balancer, and a small Transformer model to further improve the performance of codec. &#10;HifiCodec \cite{yang2023hifi} proposes a codec model that uses group-residual vector quantization to improve the reconstruction performance of audio. It can achieve good speech reconstruction performance with only four discrete token sequences, outperforming SoundStream and Encodec. &#10;However, the performance of codec decreases significantly when using only one or two discrete token sequences to represent speech, making it unable to reconstruct high-quality speech.&#10;&#10;To achieve good speech reconstruction performance with only two or even one sequence of discrete frame-level tokens, we propose a neural speech codec model with time-invariant codes named TiCodec.&#10;Some information in a speech that does not change over time is extracted by a time-invariant representation extraction module and encoded into a fixed-length code, referred to as the time-invariant code.&#10;This operation can reduce the amount of information that needs to be encoded in frame-level codes, forcing it to be maximally informative about time-related aspects.&#10;After obtaining the frame-level and time-invariant features, they are separately quantized as frame-level and time-invariant tokens.&#10;When TiCodec is used for downstream TTS tasks, the time-invariant tokens can be extracted from the prompt of target speakers, which can better maintain the timbre information of target speakers.&#10;At the same time, fewer frame-level tokens can be used to predict by the TTS model, while maintaining a low word error rate (WER) and high quality of synthesized speech.&#10;To make the time-invariant token representations extracted from the target speech in TTS contain more global time-invariant information, we introduce the time-invariant encoding consistency loss, hoping to improve the robustness of inference in TTS and further reduce WER.&#10;&#10;The contributions of this paper are as follows: &#10;\begin{itemize}&#10;\item &#10;This paper proposed a neural speech codec model named TiCodec, which can separate the time-varying and time-invariant information in speech and quantize them separately. &#10;\item&#10;A time-invariant encoding consistency loss was introduced to improve the consistency of the time-invariant codes.&#10;\end{itemize}&#10;&#10;Experimental results on speech reconstruction and zero-shot TTS task with LibriTTS datasets \cite{zen2019libritts} show that TiCodec achieved better speech reconstruction performance with fewer tokens and improved robustness, quality, and similarity of synthesized speech in the zero-shot TTS task.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Speech Coding, Computer Science, Speech Processing, Electrical Engineering, Linguistics, Text-to-Speech Models, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2301.02111" label="2301.02111">
        <attvalues>
          <attvalue for="0" value="Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers" />
          <attvalue for="1" value="  We introduce a language modeling approach for text to speech synthesis (TTS).&#10;Specifically, we train a neural codec language model (called Vall-E) using&#10;discrete codes derived from an off-the-shelf neural audio codec model, and&#10;regard TTS as a conditional language modeling task rather than continuous&#10;signal regression as in previous work. During the pre-training stage, we scale&#10;up the TTS training data to 60K hours of English speech which is hundreds of&#10;times larger than existing systems. Vall-E emerges in-context learning&#10;capabilities and can be used to synthesize high-quality personalized speech&#10;with only a 3-second enrolled recording of an unseen speaker as an acoustic&#10;prompt. Experiment results show that Vall-E significantly outperforms the&#10;state-of-the-art zero-shot TTS system in terms of speech naturalness and&#10;speaker similarity. In addition, we find Vall-E could preserve the speaker's&#10;emotion and acoustic environment of the acoustic prompt in synthesis. See&#10;https://aka.ms/valle for demos of our work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.03540" label="2302.03540">
        <attvalues>
          <attvalue for="0" value="Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal&#10;  Supervision" />
          <attvalue for="1" value="  We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can&#10;be trained with minimal supervision. By combining two types of discrete speech&#10;representations, we cast TTS as a composition of two sequence-to-sequence&#10;tasks: from text to high-level semantic tokens (akin to &quot;reading&quot;) and from&#10;semantic tokens to low-level acoustic tokens (&quot;speaking&quot;). Decoupling these two&#10;tasks enables training of the &quot;speaking&quot; module using abundant audio-only data,&#10;and unlocks the highly efficient combination of pretraining and backtranslation&#10;to reduce the need for parallel data when training the &quot;reading&quot; component. To&#10;control the speaker identity, we adopt example prompting, which allows&#10;SPEAR-TTS to generalize to unseen speakers using only a short sample of 3&#10;seconds, without any explicit speaker representation or speaker-id labels. Our&#10;experiments demonstrate that SPEAR-TTS achieves a character error rate that is&#10;competitive with state-of-the-art methods using only 15 minutes of parallel&#10;data, while matching ground-truth speech in terms of naturalness and acoustic&#10;quality, as measured in subjective tests.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.09636" label="2305.09636">
        <attvalues>
          <attvalue for="0" value="SoundStorm: Efficient Parallel Audio Generation" />
          <attvalue for="1" value="  We present SoundStorm, a model for efficient, non-autoregressive audio&#10;generation. SoundStorm receives as input the semantic tokens of AudioLM, and&#10;relies on bidirectional attention and confidence-based parallel decoding to&#10;generate the tokens of a neural audio codec. Compared to the autoregressive&#10;generation approach of AudioLM, our model produces audio of the same quality&#10;and with higher consistency in voice and acoustic conditions, while being two&#10;orders of magnitude faster. SoundStorm generates 30 seconds of audio in 0.5&#10;seconds on a TPU-v4. We demonstrate the ability of our model to scale audio&#10;generation to longer sequences by synthesizing high-quality, natural dialogue&#10;segments, given a transcript annotated with speaker turns and a short prompt&#10;with the speakers' voices.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.13438" label="2210.13438">
        <attvalues>
          <attvalue for="0" value="High Fidelity Neural Audio Compression" />
          <attvalue for="1" value="  We introduce a state-of-the-art real-time, high-fidelity, audio codec&#10;leveraging neural networks. It consists in a streaming encoder-decoder&#10;architecture with quantized latent space trained in an end-to-end fashion. We&#10;simplify and speed-up the training by using a single multiscale spectrogram&#10;adversary that efficiently reduces artifacts and produce high-quality samples.&#10;We introduce a novel loss balancer mechanism to stabilize training: the weight&#10;of a loss now defines the fraction of the overall gradient it should represent,&#10;thus decoupling the choice of this hyper-parameter from the typical scale of&#10;the loss. Finally, we study how lightweight Transformer models can be used to&#10;further compress the obtained representation by up to 40%, while staying faster&#10;than real time. We provide a detailed description of the key design choices of&#10;the proposed model including: training objective, architectural changes and a&#10;study of various perceptual loss functions. We present an extensive subjective&#10;evaluation (MUSHRA tests) together with an ablation study for a range of&#10;bandwidths and audio domains, including speech, noisy-reverberant speech, and&#10;music. Our approach is superior to the baselines methods across all evaluated&#10;settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio.&#10;Code and models are available at github.com/facebookresearch/encodec.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.01120" label="1712.01120">
        <attvalues>
          <attvalue for="0" value="Wavenet based low rate speech coding" />
          <attvalue for="1" value="  Traditional parametric coding of speech facilitates low rate but provides&#10;poor reconstruction quality because of the inadequacy of the model used. We&#10;describe how a WaveNet generative speech model can be used to generate high&#10;quality speech from the bit stream of a standard parametric coder operating at&#10;2.4 kb/s. We compare this parametric coder with a waveform coder based on the&#10;same generative model and show that approximating the signal waveform incurs a&#10;large rate penalty. Our experiments confirm the high performance of the WaveNet&#10;based coder and show that the speech produced by the system is able to&#10;additionally perform implicit bandwidth extension and does not significantly&#10;impair recognition of the original speaker for the human listener, even when&#10;that speaker has not been used during the training of the generative model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1609.03499" label="1609.03499">
        <attvalues>
          <attvalue for="0" value="WaveNet: A Generative Model for Raw Audio" />
          <attvalue for="1" value="  This paper introduces WaveNet, a deep neural network for generating raw audio&#10;waveforms. The model is fully probabilistic and autoregressive, with the&#10;predictive distribution for each audio sample conditioned on all previous ones;&#10;nonetheless we show that it can be efficiently trained on data with tens of&#10;thousands of samples per second of audio. When applied to text-to-speech, it&#10;yields state-of-the-art performance, with human listeners rating it as&#10;significantly more natural sounding than the best parametric and concatenative&#10;systems for both English and Mandarin. A single WaveNet can capture the&#10;characteristics of many different speakers with equal fidelity, and can switch&#10;between them by conditioning on the speaker identity. When trained to model&#10;music, we find that it generates novel and often highly realistic musical&#10;fragments. We also show that it can be employed as a discriminative model,&#10;returning promising results for phoneme recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.09660" label="2102.09660">
        <attvalues>
          <attvalue for="0" value="Generative Speech Coding with Predictive Variance Regularization" />
          <attvalue for="1" value="  The recent emergence of machine-learning based generative models for speech&#10;suggests a significant reduction in bit rate for speech codecs is possible.&#10;However, the performance of generative models deteriorates significantly with&#10;the distortions present in real-world input signals. We argue that this&#10;deterioration is due to the sensitivity of the maximum likelihood criterion to&#10;outliers and the ineffectiveness of modeling a sum of independent signals with&#10;a single autoregressive model. We introduce predictive-variance regularization&#10;to reduce the sensitivity to outliers, resulting in a significant increase in&#10;performance. We show that noise reduction to remove unwanted signals can&#10;significantly increase performance. We provide extensive subjective performance&#10;evaluations that show that our system based on generative modeling provides&#10;state-of-the-art coding performance at 3 kb/s for real-world speech signals at&#10;reasonable computational complexity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.06464" label="1910.06464">
        <attvalues>
          <attvalue for="0" value="Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder" />
          <attvalue for="1" value="  In order to efficiently transmit and store speech signals, speech codecs&#10;create a minimally redundant representation of the input signal which is then&#10;decoded at the receiver with the best possible perceptual quality. In this work&#10;we demonstrate that a neural network architecture based on VQ-VAE with a&#10;WaveNet decoder can be used to perform very low bit-rate speech coding with&#10;high reconstruction quality. A prosody-transparent and speaker-independent&#10;model trained on the LibriSpeech corpus coding audio at 1.6 kbps exhibits&#10;perceptual quality which is around halfway between the MELP codec at 2.4 kbps&#10;and AMR-WB codec at 23.05 kbps. In addition, when training on high-quality&#10;recorded speech with the test speaker included in the training set, a model&#10;coding speech at 1.6 kbps produces output of similar perceptual quality to that&#10;generated by AMR-WB at 23.05 kbps.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.00937" label="1711.00937">
        <attvalues>
          <attvalue for="0" value="Neural Discrete Representation Learning" />
          <attvalue for="1" value="  Learning useful representations without supervision remains a key challenge&#10;in machine learning. In this paper, we propose a simple yet powerful generative&#10;model that learns such discrete representations. Our model, the Vector&#10;Quantised-Variational AutoEncoder (VQ-VAE), differs from VAEs in two key ways:&#10;the encoder network outputs discrete, rather than continuous, codes; and the&#10;prior is learnt rather than static. In order to learn a discrete latent&#10;representation, we incorporate ideas from vector quantisation (VQ). Using the&#10;VQ method allows the model to circumvent issues of &quot;posterior collapse&quot; --&#10;where the latents are ignored when they are paired with a powerful&#10;autoregressive decoder -- typically observed in the VAE framework. Pairing&#10;these representations with an autoregressive prior, the model can generate high&#10;quality images, videos, and speech as well as doing high quality speaker&#10;conversion and unsupervised learning of phonemes, providing further evidence of&#10;the utility of the learnt representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.03312" label="2107.03312">
        <attvalues>
          <attvalue for="0" value="SoundStream: An End-to-End Neural Audio Codec" />
          <attvalue for="1" value="  We present SoundStream, a novel neural audio codec that can efficiently&#10;compress speech, music and general audio at bitrates normally targeted by&#10;speech-tailored codecs. SoundStream relies on a model architecture composed by&#10;a fully convolutional encoder/decoder network and a residual vector quantizer,&#10;which are trained jointly end-to-end. Training leverages recent advances in&#10;text-to-speech and speech enhancement, which combine adversarial and&#10;reconstruction losses to allow the generation of high-quality audio content&#10;from quantized embeddings. By training with structured dropout applied to&#10;quantizer layers, a single model can operate across variable bitrates from&#10;3kbps to 18kbps, with a negligible quality loss when compared with models&#10;trained at fixed bitrates. In addition, the model is amenable to a low latency&#10;implementation, which supports streamable inference and runs in real time on a&#10;smartphone CPU. In subjective evaluations using audio at 24kHz sampling rate,&#10;SoundStream at 3kbps outperforms Opus at 12kbps and approaches EVS at 9.6kbps.&#10;Moreover, we are able to perform joint compression and enhancement either at&#10;the encoder or at the decoder side with no additional latency, which we&#10;demonstrate through background noise suppression for speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.02765" label="2305.02765">
        <attvalues>
          <attvalue for="0" value="HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio&#10;  Codec" />
          <attvalue for="1" value="  Audio codec models are widely used in audio communication as a crucial&#10;technique for compressing audio into discrete representations. Nowadays, audio&#10;codec models are increasingly utilized in generation fields as intermediate&#10;representations. For instance, AudioLM is an audio generation model that uses&#10;the discrete representation of SoundStream as a training target, while VALL-E&#10;employs the Encodec model as an intermediate feature to aid TTS tasks. Despite&#10;their usefulness, two challenges persist: (1) training these audio codec models&#10;can be difficult due to the lack of publicly available training processes and&#10;the need for large-scale data and GPUs; (2) achieving good reconstruction&#10;performance requires many codebooks, which increases the burden on generation&#10;models. In this study, we propose a group-residual vector quantization (GRVQ)&#10;technique and use it to develop a novel \textbf{Hi}gh \textbf{Fi}delity Audio&#10;Codec model, HiFi-Codec, which only requires 4 codebooks. We train all the&#10;models using publicly available TTS data such as LibriTTS, VCTK, AISHELL, and&#10;more, with a total duration of over 1000 hours, using 8 GPUs. Our experimental&#10;results show that HiFi-Codec outperforms Encodec in terms of reconstruction&#10;performance despite requiring only 4 codebooks. To facilitate research in audio&#10;codec and generation, we introduce AcademiCodec, the first open-source audio&#10;codec toolkit that offers training codes and pre-trained models for Encodec,&#10;SoundStream, and HiFi-Codec. Code and pre-trained model can be found on:&#10;\href{https://github.com/yangdongchao/AcademiCodec}{https://github.com/yangdongchao/AcademiCodec}&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.02882" label="1904.02882">
        <attvalues>
          <attvalue for="0" value="LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech" />
          <attvalue for="1" value="  This paper introduces a new speech corpus called &quot;LibriTTS&quot; designed for&#10;text-to-speech use. It is derived from the original audio and text materials of&#10;the LibriSpeech corpus, which has been used for training and evaluating&#10;automatic speech recognition systems. The new corpus inherits desired&#10;properties of the LibriSpeech corpus while addressing a number of issues which&#10;make LibriSpeech less than ideal for text-to-speech work. The released corpus&#10;consists of 585 hours of speech data at 24kHz sampling rate from 2,456 speakers&#10;and the corresponding texts. Experimental results show that neural end-to-end&#10;TTS models trained from the LibriTTS corpus achieved above 4.0 in mean opinion&#10;scores in naturalness in five out of six evaluation speakers. The corpus is&#10;freely available for download from http://www.openslr.org/60/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05492" label="2310.05492">
        <attvalues>
          <attvalue for="0" value="How Abilities in Large Language Models are Affected by Supervised&#10;  Fine-tuning Data Composition" />
          <attvalue for="1" value="  Large language models (LLMs) with enormous pre-training tokens and parameters&#10;emerge diverse abilities, including math reasoning, code generation, and&#10;instruction following. These abilities are further enhanced by supervised&#10;fine-tuning (SFT). While the open-source community has explored ad-hoc SFT for&#10;enhancing individual capabilities, proprietary LLMs exhibit versatility across&#10;various skills. Therefore, understanding the facilitation of multiple abilities&#10;via SFT is paramount. In this study, we specifically focuses on the interplay&#10;of data composition between mathematical reasoning, code generation, and&#10;general human-aligning abilities during SFT. We propose four intriguing&#10;research questions to explore the association between model performance and&#10;various factors including data amount, composition ratio, model size and SFT&#10;strategies. Our experiments reveal that distinct capabilities scale differently&#10;and larger models generally show superior performance with same amount of data.&#10;Mathematical reasoning and code generation consistently improve with increasing&#10;data amount, whereas general abilities plateau after roughly a thousand&#10;samples. Moreover, we observe data composition appears to enhance various&#10;abilities under limited data conditions, yet can lead to performance conflicts&#10;when data is plentiful. Our findings also suggest the amount of composition&#10;data influences performance more than the composition ratio. In analysis of SFT&#10;strategies, we find that sequentially learning multiple skills risks&#10;catastrophic forgetting. Our proposed Dual-stage Mixed Fine-tuning (DMT)&#10;strategy offers a promising solution to learn multiple abilities with different&#10;scaling patterns.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Supervised Fine-Tuning of Large Language Models&#10;Large Language Models (LLMs) have shown notable zero-shot performance in various domains \cite{brown2020language,wu2021yuan,hou2024large,dong2023revisit,zhou2024grasping,wu2023semantic,song2023large}, prompting further development to push the boundaries of these models. To delve deeper to their potential, LLMs are subjected to a Supervised Fine-Tuning (SFT) phase, enhancing their ability to solve tasks and align better with human instructions. Here, we extend the conventional definition of SFT to include various forms of sequence-to-sequence fine-tuning, such as fine-tuning for human alignment, instruction following, and domain-specific task optimization \cite{zhou2023instructionfollowing,yuan2023rrhf,cheng2023m,zhang2024instruction}.&#10;&#10;Recent research has delved into multi-task instruction fine-tuning of pre-trained LLMs to bolster their zero-shot performance across numerous downstream NLP tasks \cite{sanh2022multitask}. In an effort to encompass existing NLP tasks comprehensively, \cite{flan,longpre2023flan} curated the expansive FLAN dataset specifically for instruction-based fine-tuning. LLMs, both open-source \cite{chung2022scaling} and proprietary \cite{singhal2022large}, fine-tuned with FLAN, have demonstrated enhanced zero-shot performance on a variety of unseen tasks.&#10;&#10;While research has probed into the generalization capabilities of LLMs within out-of-distribution domains \cite{liu2024good,yuan2024revisiting,wang2024multiperspective}, the effects of multi-task training on in-domain performance remain under-explored. With the ascent of proprietary models like ChatGPT, the focus on SFT for aligning LLMs with human intent has intensified \cite{ouyang2022training}. Moving away from crowd-sourced SFT data, recent initiatives have generated SFT datasets from user logs within proprietary LLM platforms \cite{vicuna2023,openchat}, employing the models themselves to assist in the data generation process \cite{wang2023selfinstruct,alpaca,cheng2023accelerating,lei2023instructerc,xu2023wizardlm,xue2023occuquest}. Additionally, methods to improve SFT data quality have been proposed, targeting more accurate alignment with human interactions \cite{lima,tulu,instag,liu2024makes}.&#10;&#10;Furthermore, SFT has proven beneficial for LLMs in specialized areas such as mathematical reasoning \cite{gsm8k,hendrycks2021measuring,rft,chen2024autoprm,mammoth,gou2024tora,li2023query,yue2024mammoth2} and code generation tasks \cite{codealpaca,wizardcoder,wang2024dolphcoder,wei2023magicoder}. Taking advantage of their advanced interactive capabilities, some researchers have leveraged supervised fine-tuned LLMs to compose commands that interface with external tools, thus enhancing the handling of assorted downstream applications \cite{shen2023hugginggpt,yao2023react,yao2023tree,song2024knowledge,fu2024preact}. This paper examines the SFT performance using composite datasets, considering different model sizes and data amounts.&#10;&#10;Scaling Laws in Large Language Models &#10;&#10;The exceptional performance of LLMs comes from scaling up model sizes, data amounts, and computational costs to massive scales. &#10;Therefore, it is crucial to explore the model performance across an exponential range of scales. &#10;Many endeavors have been made to discuss the scaling laws for pre-training \cite{palm2,hoffmann2022training}, transfer learning \cite{chronopoulou2019embarrassingly}, preference modeling \cite{gao2022rmscaling} and mathematical reasoning \cite{rft}. In this paper, we also explore the SFT performance with composite data from the perspective of different scales of model sizes and data amounts.&#10;&#10;" />
          <attvalue for="4" value="Supervised Fine-Tuning Strategies, Machine Learning, Computer Science, Linguistics, Artificial Intelligence, Mathematics, Language Model Abilities, Data Composition Effects" />
        </attvalues>
      </node>
      <node id="2408.05948" label="2408.05948">
        <attvalues>
          <attvalue for="0" value="ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge&#10;  Graph QA datasets with Large Language Models" />
          <attvalue for="1" value="  The rapid advancement of Large Language Models (LLMs) and conversational&#10;assistants necessitates dynamic, scalable, and configurable conversational&#10;datasets for training and evaluation. These datasets must accommodate diverse&#10;user interaction modes, including text and voice, each presenting unique&#10;modeling challenges. Knowledge Graphs (KGs), with their structured and evolving&#10;nature, offer an ideal foundation for current and precise knowledge. Although&#10;human-curated KG-based conversational datasets exist, they struggle to keep&#10;pace with the rapidly changing user information needs. We present ConvKGYarn, a&#10;scalable method for generating up-to-date and configurable conversational KGQA&#10;datasets. Qualitative psychometric analyses confirm our method can generate&#10;high-quality datasets rivaling a popular conversational KGQA dataset while&#10;offering it at scale and covering a wide range of human-interaction&#10;configurations. We showcase its utility by testing LLMs on diverse&#10;conversations - exploring model behavior on conversational KGQA sets with&#10;different configurations grounded in the same KG fact set. Our results&#10;highlight the ability of ConvKGYarn to improve KGQA foundations and evaluate&#10;parametric knowledge of LLMs, thus offering a robust solution to the constantly&#10;evolving landscape of conversational assistants.&#10;" />
          <attvalue for="2" value="&#10;&#10;The proliferation of Large Language Models (LLMs) and conversational assistants has led to their ubiquitous presence in daily user interactions. This widespread adoption underscores the critical need for dynamic datasets capable of rigorously evaluating their proficiency in addressing knowledge-seeking queries.&#10;Knowledge Graphs (KGs) have long been recognized as powerful tools for capturing structured representations of the world~\cite{hogan-etal-2021-knowledge}.&#10;In KGs, concepts and entities are represented as nodes, while semantic relationships defining facts are represented with edges. &#10;This structured representation has had an impact across various domains, including Natural Language Processing~\cite{schneider-etal-2022-decade}, Recommender Systems~\cite{guo-etal-2022-recommender}, and Information Retrieval~\cite{reinanda-etal-2020-knowledge}.&#10;&#10;The integration of LLMs and KGs has opened up new opportunities in natural language processing~\cite{petroni-etal-2019-language,guu-etal-2020-realm,peng-etal-2023-check}, which has led to significant advancements across various tasks~\cite{barba-etal-2021-esc,chakrabarti-etal-2022-joint,de-cao-etal-2022-multilingual,xu-etal-2023-kilm}. &#10;By combining the dynamic capabilities of LLMs with the structured insights from KGs, researchers have unlocked new avenues for developing advanced question-answering (QA) systems.&#10;In conversational Knowledge Graph Question Answering (KGQA), datasets like ConvQuestions~\cite{10.1145/3357384.3358016} have emerged to address scenarios where questions often lack full context or contain grammatical inconsistencies. &#10;These datasets have played a crucial role in enabling new retrieval-augmented systems, demonstrating the potential of LLM-KG integrations to provide accurate and attributable responses in conversational settings~\cite{christmann2023compmix}.&#10;&#10;In relation, advancements in text retrieval have underscored the potential of using LLMs to generate synthetic data to improve the effectiveness of downstream systems.&#10;This process has been utilized at scale for neural query synthesis~\cite{doct5query, unicoil, NQS} and LLM-based ranked list reorderings for instruction distillation into open-source rerankers~\cite{rankvicuna, rankzephyr, lit5} resulting in substantial improvements across a spectrum of retrieval tasks. &#10;More recently, synthetic data generation, facilitated by automated prompt optimization~\cite{path}, has enabled the training of highly effective small-scale models, deprecating the dependence on human-labeled data. &#10;Collectively, these highlight the opportunity to integrate synthetic data strategies from LLMs to develop more resilient and adaptable knowledge-intensive systems. &#10;&#10;While existing conversational KGQA datasets are rich in content, they often struggle to keep pace with rapidly evolving user information needs. &#10;This discrepancy raises questions about the relevance of such data in real-world, adaptive conversational scenarios.&#10;To address this challenge, we introduce Conv\-KG\-Yarn, a novel method for generating large-scale, configurable conversational Knowledge Graph Question Answering (KGQA) datasets.&#10;Through rigorous psychometric evaluation metrics, we demonstrate that Conv\-KG\-Yarn produces high-quality conversational data comparable to established, human-curated KGQA datasets. &#10;Notably, Conv\-KG\-Yarn achieves this while significantly expanding entity and fact coverage by several orders of magnitude and introducing configurable properties in user interaction styles.&#10;&#10;A critical component of our research involves evaluating the datasets generated by Conv\-KG\-Yarn using various LLMs to assess their parametric knowledge. &#10;Our observations reveal that these models often struggle with fact recall, underscoring the need for retrieval-augmented systems.&#10;By evaluating datasets generated with different interaction styles and their varied linguistic phenomena generated with consistent fact sets from the Knowledge Graph, we aim to assess the robustness of LLMs in handling diverse conversational settings, in confounder-free manners.&#10;Our findings indicate that model effectiveness varies significantly across diverse user interaction styles, highlighting the importance of developing LLMs capable of serving as robust conversational systems.&#10;&#10;Through this work, we seek to reveal the path toward creating datasets that can effectively train and evaluate evolving conversational assistants.&#10;We envision these methods and datasets will play a crucial role in developing more versatile and adaptive conversational AI systems.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Linguistics, Language Models, Knowledge Graphs, Conversational AI, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2003.02320" label="2003.02320">
        <attvalues>
          <attvalue for="0" value="Knowledge Graphs" />
          <attvalue for="1" value="  In this paper we provide a comprehensive introduction to knowledge graphs,&#10;which have recently garnered significant attention from both industry and&#10;academia in scenarios that require exploiting diverse, dynamic, large-scale&#10;collections of data. After some opening remarks, we motivate and contrast&#10;various graph-based data models and query languages that are used for knowledge&#10;graphs. We discuss the roles of schema, identity, and context in knowledge&#10;graphs. We explain how knowledge can be represented and extracted using a&#10;combination of deductive and inductive techniques. We summarise methods for the&#10;creation, enrichment, quality assessment, refinement, and publication of&#10;knowledge graphs. We provide an overview of prominent open knowledge graphs and&#10;enterprise knowledge graphs, their applications, and how they use the&#10;aforementioned techniques. We conclude with high-level future research&#10;directions for knowledge graphs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.00105" label="2210.00105">
        <attvalues>
          <attvalue for="0" value="A Decade of Knowledge Graphs in Natural Language Processing: A Survey" />
          <attvalue for="1" value="  In pace with developments in the research field of artificial intelligence,&#10;knowledge graphs (KGs) have attracted a surge of interest from both academia&#10;and industry. As a representation of semantic relations between entities, KGs&#10;have proven to be particularly relevant for natural language processing (NLP),&#10;experiencing a rapid spread and wide adoption within recent years. Given the&#10;increasing amount of research work in this area, several KG-related approaches&#10;have been surveyed in the NLP research community. However, a comprehensive&#10;study that categorizes established topics and reviews the maturity of&#10;individual research streams remains absent to this day. Contributing to closing&#10;this gap, we systematically analyzed 507 papers from the literature on KGs in&#10;NLP. Our survey encompasses a multifaceted review of tasks, research types, and&#10;contributions. As a result, we present a structured overview of the research&#10;landscape, provide a taxonomy of tasks, summarize our findings, and highlight&#10;directions for future work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.12528" label="2103.12528">
        <attvalues>
          <attvalue for="0" value="Multilingual Autoregressive Entity Linking" />
          <attvalue for="1" value="  We present mGENRE, a sequence-to-sequence system for the Multilingual Entity&#10;Linking (MEL) problem -- the task of resolving language-specific mentions to a&#10;multilingual Knowledge Base (KB). For a mention in a given language, mGENRE&#10;predicts the name of the target entity left-to-right, token-by-token in an&#10;autoregressive fashion. The autoregressive formulation allows us to effectively&#10;cross-encode mention string and entity names to capture more interactions than&#10;the standard dot product between mention and entity vectors. It also enables&#10;fast search within a large KB even for mentions that do not appear in mention&#10;tables and with no need for large-scale vector indices. While prior MEL works&#10;use a single representation for each entity, we match against entity names of&#10;as many languages as possible, which allows exploiting language connections&#10;between source input and target name. Moreover, in a zero-shot setting on&#10;languages with no training data at all, mGENRE treats the target language as a&#10;latent variable that is marginalized at prediction time. This leads to over 50%&#10;improvements in average accuracy. We show the efficacy of our approach through&#10;extensive evaluation including experiments on three popular MEL benchmarks&#10;where mGENRE establishes new state-of-the-art results. Code and pre-trained&#10;models at https://github.com/facebookresearch/GENRE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.12235" label="2306.12235">
        <attvalues>
          <attvalue for="0" value="CompMix: A Benchmark for Heterogeneous Question Answering" />
          <attvalue for="1" value="  Fact-centric question answering (QA) often requires access to multiple,&#10;heterogeneous, information sources. By jointly considering several sources like&#10;a knowledge base (KB), a text collection, and tables from the web, QA systems&#10;can enhance their answer coverage and confidence. However, existing QA&#10;benchmarks are mostly constructed with a single source of knowledge in mind.&#10;This limits capabilities of these benchmarks to fairly evaluate QA systems that&#10;can tap into more than one information repository. To bridge this gap, we&#10;release CompMix, a crowdsourced QA benchmark which naturally demands the&#10;integration of a mixture of input sources. CompMix has a total of 9,410&#10;questions, and features several complex intents like joins and temporal&#10;conditions. Evaluation of a range of QA systems on CompMix highlights the need&#10;for further research on leveraging information from heterogeneous sources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.15088" label="2309.15088">
        <attvalues>
          <attvalue for="0" value="RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large&#10;  Language Models" />
          <attvalue for="1" value="  Researchers have successfully applied large language models (LLMs) such as&#10;ChatGPT to reranking in an information retrieval context, but to date, such&#10;work has mostly been built on proprietary models hidden behind opaque API&#10;endpoints. This approach yields experimental results that are not reproducible&#10;and non-deterministic, threatening the veracity of outcomes that build on such&#10;shaky foundations. To address this significant shortcoming, we present&#10;RankVicuna, the first fully open-source LLM capable of performing high-quality&#10;listwise reranking in a zero-shot setting. Experimental results on the TREC&#10;2019 and 2020 Deep Learning Tracks show that we can achieve effectiveness&#10;comparable to zero-shot reranking with GPT-3.5 with a much smaller 7B parameter&#10;model, although our effectiveness remains slightly behind reranking with GPT-4.&#10;We hope our work provides the foundation for future research on reranking with&#10;modern LLMs. All the code necessary to reproduce our results is available at&#10;https://github.com/castorini/rank_llm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.00857" label="2309.00857">
        <attvalues>
          <attvalue for="0" value="Evaluating Transformer's Ability to Learn Mildly Context-Sensitive&#10;  Languages" />
          <attvalue for="1" value="  Despite the fact that Transformers perform well in NLP tasks, recent studies&#10;suggest that self-attention is theoretically limited in learning even some&#10;regular and context-free languages. These findings motivated us to think about&#10;their implications in modeling natural language, which is hypothesized to be&#10;mildly context-sensitive. We test the Transformer's ability to learn mildly&#10;context-sensitive languages of varying complexities, and find that they&#10;generalize well to unseen in-distribution data, but their ability to&#10;extrapolate to longer strings is worse than that of LSTMs. Our analyses show&#10;that the learned self-attention patterns and representations modeled dependency&#10;relations and demonstrated counting behavior, which may have helped the models&#10;solve the languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Regarding the expressiveness of the Transformer, \cite{perez2018on, JMLR:v22:20-302} established the Turing-completeness of the hard-attention Transformer. \cite{bhattamishra-etal-2020-computational} proves the Turing-completeness of soft attention by showing that they can simulate RNNs. However, these results assumed arbitrary precision for weights and activations, had certain departures from the original architecture, and made the proofs through their unique task definitions.&#10;&#10;In a practical case of formal language learning from finite examples, the Transformer's ability is known to be limited, even in the regular language class. Empirically, it was shown that Transformers of different self-attention variants have limited abilities to learn certain star-free languages, as well as non-star-free, periodic regular languages \cite{hahn2020, bhattamishra-etal-2020-ability}, although the latter may still be recognized theoretically with a simple modification to the architecture \cite{chiang-cholak-2022-overcoming}.&#10;&#10;As for context-free languages with hieararchical structural analyses such as Dyck-$n$, \cite{ebrahimi-etal-2020-self} empirically demonstrated one Transformer encoder setup in which such languages may be learned and observed stack-like behavior in self-attention patterns. \cite{yao-etal-2021-self} proved and empirically showed that self-attention can learn Dyck-$n$ with a bounded depth, although the boundedness reduces the CFL to regular. Additional empirical work on Dyck-$n$ include \cite{bernardy-etal-2021-Transformer, wen2023interpretability}, among others.&#10;&#10;Besides language recognition guided by the Chomsky hierarchy, another line of research investigates other alternative formal languages, such as counter-recognizable languages \cite{bhattamishra-etal-2020-ability} and first-order logic \cite{merrill2023Transformers, chiang2023tighter}, to characterize the expressiveness of Transformers.&#10;&#10;This work introduces MCSGs into the empirical explorations through assessing the ability of Transformers in certain learning settings to learn a variety of string languages recognizable by MCSGs of different complexities, which have not yet been studied as a whole like languages in other classes. Occasionally, studies on the Transformer's learning ability worked with data that conveniently fall into this class, including a few counter languages that are also TAG-recognizable \cite{bhattamishra-etal-2020-ability}, discontinuities in Dutch \cite{kogkalidis-wijnholds-2022-discontinuous}, reduplication \cite{deletang2023neural}, as well as a crossed parentheses language inspired by crossing dependency \cite{papadimitriou2023pretrain}. This work complements these and other aforementioned related work by presenting a systematic evaluation guided by basic MCSL constructions and the subhierarchy within the class, as well as comparing each of the basic constructions against a less and a more complex counterparts.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Transformer Limitations, Mathematics, Context-Sensitive Languages, Natural Language Modeling" />
        </attvalues>
      </node>
      <node id="2311.12338" label="2311.12338">
        <attvalues>
          <attvalue for="0" value="A Survey on Large Language Models for Personalized and Explainable&#10;  Recommendations" />
          <attvalue for="1" value="  In recent years, Recommender Systems(RS) have witnessed a transformative&#10;shift with the advent of Large Language Models(LLMs) in the field of Natural&#10;Language Processing(NLP). These models such as OpenAI's GPT-3.5/4, Llama from&#10;Meta, have demonstrated unprecedented capabilities in understanding and&#10;generating human-like text. This has led to a paradigm shift in the realm of&#10;personalized and explainable recommendations, as LLMs offer a versatile toolset&#10;for processing vast amounts of textual data to enhance user experiences. To&#10;provide a comprehensive understanding of the existing LLM-based recommendation&#10;systems, this survey aims to analyze how RS can benefit from LLM-based&#10;methodologies. Furthermore, we describe major challenges in Personalized&#10;Explanation Generating(PEG) tasks, which are cold-start problems, unfairness&#10;and bias problems in RS.&#10;" />
          <attvalue for="2" value="&#10;Recommendation systems are pivotal in aiding users to discover pertinent and personalized items or content. The advent of Large Language Models (LLMs) in Natural Language Processing (NLP) has sparked increased enthusiasm for leveraging the capabilities of these models to elevate and improve recommendation systems.&#10;&#10;With the thriving of pre-training in NLP, many language models have been pre-trained on large scale unsupervised corpora, and then fine-tuned for downstream tasks. The transformer architecture\cite{2017attention}, was introduced in 2017, has become a foundation in LLMs. It eschewed the sequential nature of recurrent neural networks (RNNs) in favor of a self-attention mechanism, enabling parallelization and significantly improving efficiency in handling sequential data. Based on Transformer architechture, many Pre-trained Language Models(PLMs) have emerged. GPT series\cite{dale2021gpt}, developed by OpenAI, and BERT\cite{devlin2018bert}, developed by Google, represent two prominent approaches to leveraging transformers for PLMs. The key advantage of incorporating PLMs into recommendation systems lies in their ability to extract high-quality representations of textual features and leverage the extensive external knowledge encoded within them.\cite{2023arXiv230203735L} Different from traditional recommendation systems, the LLM-based models can capture contextual information, comprehending user queries, item descriptions, and other textual data more efficiently.\cite{P5} Based on PLMs, fine-tuning strategy involves training the model on a smaller task-specific dataset. This dataset is typically related to a specific application or domain, such as sentiment analysis, text classification, question answering, or recommender systems. However, fine-tuning large language models on specific downstream tasks paradigm usually needs to fine-tunes all of the parameters in a PLM, which is a computational resource consuming process. Most researchers and companies cannot access as much resouce as OpenAI or Microsoft or Google. As a result, a recently proposed paradigm, prompt learning\cite{liu2023pre}, further unifies the use of PLMs on different tasks&#10;in a simple yet flexible manner. In general, prompting is the process of providing addtional information for a trained model to condition while predicting output labels for a task, for example, providing a piece of text inserted in the input examples. Prompt tuning is different from fine tuning in that it only requires storing a small task-specific prompt for each task instead of making a task specific copy of the entire PLM in fine tuning. The advantage of this paradigm lie in two aspects:(1)It bridges the gap between pre-training and downstream objectives, allowing better utilization of the rich knowledge in pretrained models. This advantage will be multiplied when very little downstream data is available. (2)Only a small set of parameters are needed to tune for prompt engineering, which is more efficient.\cite{surveyllm4rec}&#10;&#10;This survey focuses on utilizing LLMs for Personalized Explanation Generating task. I will discuss the background of NLP for text generation in Chapter 2, the challenges existed in explainable recommendations in Chapter 3, the current approaches intending to solve these challenges in Chapter 4, and finally, the conclusion in Chapter 5.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Recommender Systems, Computer Science, Large Language Models, Linguistics, Mathematics, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2302.03735" label="2302.03735">
        <attvalues>
          <attvalue for="0" value="Pre-train, Prompt and Recommendation: A Comprehensive Survey of Language&#10;  Modelling Paradigm Adaptations in Recommender Systems" />
          <attvalue for="1" value="  The emergence of Pre-trained Language Models (PLMs) has achieved tremendous&#10;success in the field of Natural Language Processing (NLP) by learning universal&#10;representations on large corpora in a self-supervised manner. The pre-trained&#10;models and the learned representations can be beneficial to a series of&#10;downstream NLP tasks. This training paradigm has recently been adapted to the&#10;recommendation domain and is considered a promising approach by both academia&#10;and industry. In this paper, we systematically investigate how to extract and&#10;transfer knowledge from pre-trained models learned by different PLM-related&#10;training paradigms to improve recommendation performance from various&#10;perspectives, such as generality, sparsity, efficiency and effectiveness.&#10;Specifically, we propose a comprehensive taxonomy to divide existing PLM-based&#10;recommender systems w.r.t. their training strategies and objectives. Then, we&#10;analyze and summarize the connection between PLM-based training paradigms and&#10;different input data types for recommender systems. Finally, we elaborate on&#10;open issues and future research directions in this vibrant field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.13270" label="2210.13270">
        <attvalues>
          <attvalue for="0" value="Generating Hierarchical Explanations on Text Classification Without&#10;  Connecting Rules" />
          <attvalue for="1" value="  The opaqueness of deep NLP models has motivated the development of methods&#10;for interpreting how deep models predict. Recently, work has introduced&#10;hierarchical attribution, which produces a hierarchical clustering of words,&#10;along with an attribution score for each cluster. However, existing work on&#10;hierarchical attribution all follows the connecting rule, limiting the cluster&#10;to a continuous span in the input text. We argue that the connecting rule as an&#10;additional prior may undermine the ability to reflect the model decision&#10;process faithfully. To this end, we propose to generate hierarchical&#10;explanations without the connecting rule and introduce a framework for&#10;generating hierarchical clusters. Experimental results and further analysis&#10;show the effectiveness of the proposed method in providing high-quality&#10;explanations for reflecting model predicting process.&#10;" />
          <attvalue for="2" value="&#10;The opaqueness of deep natural language processing (NLP) models has grown in tandem with their power \cite{doshi2017towards}, which has motivated efforts to interpret how these black-box models work \cite{sundararajan2017axiomatic, belinkov2019analysis}.&#10;Post-hoc explanation aims to explain a trained model and reveal how the model arrives at a decision \cite{jacovi2020towards, molnar2020interpretable}. &#10;In NLP, this goal is usually approached with attribution method, which assesses the influence of inputs on model predictions.&#10;&#10;Prior lines of work on post-hoc explanation usually focus on generating word-level or phrase-level attribution for deep NLP models.&#10;Recently, work has introduced the new idea of hierarchical attribution \cite{singh2018hierarchical, jin2019towards, chen2020generating}.&#10;As shown in Figure \ref{hierarchical attribution},&#10;hierarchical attribution produces a hierarchical clustering of words, and provides attribution scores for each clusters. By providing compositional semantics information, hierarchical attribution can give users a better understanding of the model decision-making process.&#10;Since the attribution score of each cluster in hierarchical attribution is calculated separately, the key point of generating hierarchical attribution is how to get word clusters,&#10;which should be informative enough to capture meaningful feature interactions while displaying a sufficiently small subset of all feature groups to maintain simplicity \cite{singh2018hierarchical}.&#10;&#10;Existing work has proposed various algorithms to generate hierarchical clusters. For example, \cite{singh2018hierarchical} use CD score \cite{murdoch2018beyond} as a joining metric in the agglomerative clustering procedure;&#10;\cite{chen2020generating} recursively divides large text spans into smaller ones by detecting feature interaction.&#10;However, previous work requires only adjacent clusters to be grouped as a new cluster, which we denote as the connecting rule.&#10;With the connecting rule, generated clusters will always be continuous text spans in the input text.&#10;While consistent with human reading habits, we argue that the connecting rule as an additional prior may undermine the ability to faithfully reflect the model decision process.&#10;The concerns are summarized as follows:&#10;&#10;First, modern NLP models such as BERT \cite{devlin2018bert} and GPT \cite{radford2018improving, radford2019language} are&#10;almost all transformer-based, using self-attention mechanisms \cite{vaswani2017attention} to build word relations. Since all word relations are calculated parallelly in the self-attention mechanism, connecting rule is inconsistent with the base working algorithms of these models.&#10;&#10;Second, unlike the toy sample in Figure \ref{hierarchical attribution}, NLP tasks are becoming increasingly complex, often requiring the joint reasoning of different parts of the input text \cite{chowdhary2020natural}.&#10;For example, Figure \ref{our method} shows an sample from natural language interface (NLI) task, in which `has a' and `available' are the key combinatorial semantics to make the prediction. &#10;However, hierarchical explanations with the connecting rule can not identify this compositional information but only can build relations ntil the whole sentence is regarded as one cluster.&#10;&#10;To this end, we propose to generate hierarchical explanations without the connecting rule and introduce a framework for generating hierarchical clusters, which produces hierarchical clusters by recursively detecting the strongest interactions among clusters and then merging small clusters into bigger ones.&#10;Compared to previous methods with connecting rules, our method can provide compositional semantics information of long-distance spans.&#10;We build systems based on two classic attribution methods: LOO \cite{lipton2018mythos} and LIME \cite{ribeiro2016should}. Experimental results and further analysis show that our method can capture higher quality features for reflecting model predicting than existing competitive methods.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Interpretability, Computer Science, Deep NLP Models, Linguistics, Hierarchical Attribution, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="1703.01365" label="1703.01365">
        <attvalues>
          <attvalue for="0" value="Axiomatic Attribution for Deep Networks" />
          <attvalue for="1" value="  We study the problem of attributing the prediction of a deep network to its&#10;input features, a problem previously studied by several other works. We&#10;identify two fundamental axioms---Sensitivity and Implementation Invariance&#10;that attribution methods ought to satisfy. We show that they are not satisfied&#10;by most known attribution methods, which we consider to be a fundamental&#10;weakness of those methods. We use the axioms to guide the design of a new&#10;attribution method called Integrated Gradients. Our method requires no&#10;modification to the original network and is extremely simple to implement; it&#10;just needs a few calls to the standard gradient operator. We apply this method&#10;to a couple of image models, a couple of text models and a chemistry model,&#10;demonstrating its ability to debug networks, to extract rules from a network,&#10;and to enable users to engage with models better.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03685" label="2004.03685">
        <attvalues>
          <attvalue for="0" value="Towards Faithfully Interpretable NLP Systems: How should we define and&#10;  evaluate faithfulness?" />
          <attvalue for="1" value="  With the growing popularity of deep-learning based NLP models, comes a need&#10;for interpretable systems. But what is interpretability, and what constitutes a&#10;high-quality interpretation? In this opinion piece we reflect on the current&#10;state of interpretability evaluation research. We call for more clearly&#10;differentiating between different desired criteria an interpretation should&#10;satisfy, and focus on the faithfulness criteria. We survey the literature with&#10;respect to faithfulness evaluation, and arrange the current approaches around&#10;three assumptions, providing an explicit form to how faithfulness is &quot;defined&quot;&#10;by the community. We provide concrete guidelines on how evaluation of&#10;interpretation methods should and should not be conducted. Finally, we claim&#10;that the current binary definition for faithfulness sets a potentially&#10;unrealistic bar for being considered faithful. We call for discarding the&#10;binary notion of faithfulness in favor of a more graded one, which we believe&#10;will be of greater practical utility.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.05337" label="1806.05337">
        <attvalues>
          <attvalue for="0" value="Hierarchical interpretations for neural network predictions" />
          <attvalue for="1" value="  Deep neural networks (DNNs) have achieved impressive predictive performance&#10;due to their ability to learn complex, non-linear relationships between&#10;variables. However, the inability to effectively visualize these relationships&#10;has led to DNNs being characterized as black boxes and consequently limited&#10;their applications. To ameliorate this problem, we introduce the use of&#10;hierarchical interpretations to explain DNN predictions through our proposed&#10;method, agglomerative contextual decomposition (ACD). Given a prediction from a&#10;trained DNN, ACD produces a hierarchical clustering of the input features,&#10;along with the contribution of each cluster to the final prediction. This&#10;hierarchy is optimized to identify clusters of features that the DNN learned&#10;are predictive. Using examples from Stanford Sentiment Treebank and ImageNet,&#10;we show that ACD is effective at diagnosing incorrect predictions and&#10;identifying dataset bias. Through human experiments, we demonstrate that ACD&#10;enables users both to identify the more accurate of two DNNs and to better&#10;trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to&#10;adversarial perturbations, implying that it captures fundamental aspects of the&#10;input and ignores spurious noise.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11506" label="2010.11506">
        <attvalues>
          <attvalue for="0" value="Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution&#10;  Data" />
          <attvalue for="1" value="  Fine-tuned pre-trained language models can suffer from severe miscalibration&#10;for both in-distribution and out-of-distribution (OOD) data due to&#10;over-parameterization. To mitigate this issue, we propose a regularized&#10;fine-tuning method. Our method introduces two types of regularization for&#10;better calibration: (1) On-manifold regularization, which generates pseudo&#10;on-manifold samples through interpolation within the data manifold. Augmented&#10;training with these pseudo samples imposes a smoothness regularization to&#10;improve in-distribution calibration. (2) Off-manifold regularization, which&#10;encourages the model to output uniform distributions for pseudo off-manifold&#10;samples to address the over-confidence issue for OOD data. Our experiments&#10;demonstrate that the proposed method outperforms existing calibration methods&#10;for text classification in terms of expectation calibration error,&#10;misclassification detection, and OOD detection on six datasets. Our code can be&#10;found at https://github.com/Lingkai-Kong/Calibrated-BERT-Fine-Tuning.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-trained language models have recently brought the natural language processing (NLP) community into the transfer&#10;learning era. The transfer learning framework consists of two stages, where we first pre-train a large-scale language model, (\eg, BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta}, ALBERT \cite{Lan2020ALBERT} and T5 \cite{raffel2019exploring})&#10;on a large text corpus and then fine-tune it on downstream tasks.&#10;Such a fine-tuning approach has achieved&#10;SOTA performance in many NLP benchmarks \cite{wang2018glue, wang2019superglue}. &#10;&#10;Many applications, however, require trustworthy predictions that need to be not only accurate but also well calibrated.&#10;In particular, a well-calibrated model should produce reliable confident estimates for both in-distribution and out-of-distribution (OOD) data: (1) For in-distribution data, a model should produce predictive probabilities close to the true likelihood for each class, \ie, confidence $\approx$ true likelihood.&#10;(2) For OOD data, which do not belong to any class of the training data, the model output should produce high uncertainty to say `I don't know', \ie, confidence $\approx$ random guess, instead of producing absurdly wrong yet wildly confident predictions. Providing such calibrated output probabilities can help us to achieve better model robustness \cite{lee2018simple}, model fairness \cite{chouldechova2017fair} and improve label efficiency via uncertainty driven learning \cite{gal2017active,siddhant2018deep, shen2018deep}.&#10;&#10;Unfortunately, \cite{guo2017calibration} have shown that due to over-parameterization, deep convolutional neural networks are often miscalibrated.&#10;Our experimental investigation further corroborates that fine-tuned language models can suffer from miscalibration even more for NLP tasks. As shown in&#10; Figure~\ref{fig:intro-calibration}, we present the calibration of a BERT-MLP model for a text classification task on the 20NG dataset.&#10;Specifically, we train a TextCNN \cite{kim2014convolutional} and a BERT-MLP using 20NG\textsubscript{15}&#10;(the first 15 categories of 20NG) and then evaluate them on both in-distribution and&#10;OOD data.&#10;The first row plots their reliability diagrams \cite{mizil-2005-predict} on the test set of&#10;20NG\textsubscript{15}. Though BERT improves the classification accuracy from $83.9\%$ to&#10;$87.4\%$, it also increases the expected calibration error (ECE, see more details in Section \ref{sec:pre}) from $4.0\%$ to&#10;$9.5\%$. This indicates that BERT-MLP is much more miscalibrated for in-distribution&#10; data. The second row plots the histograms of the model confidence, \ie, the maximum output probability,&#10;on the test set of 20NG\textsubscript{5} (the unseen 5 categories of 20NG). While it is desirable to produce low probabilities for&#10; these unseen classes, BERT-MLP produces wrong yet over-confident&#10; predictions for such OOD data.&#10;&#10;Such an aggravation of miscalibration is due to the even more significant over-parameterization of these language models.&#10;At the pre-training stage, they are trained on a huge amount of unlabeled data in an unsupervised manner, \eg, T5 is pre-trained on 745 GB text. To&#10;capture rich semantic and syntactic information from such a large corpus, the language models are designed to have enormous capacity, \eg, T5 has&#10;about 11 billion parameters. At the fine-tuning stage, however, only limited labeled data are available in the downstream tasks. With the extremely&#10;high capacity, these models can easily overfit training data likelihood and be over-confident in their predictions.&#10;&#10;To fight against miscalibration, a natural option is to apply a calibration method such as temperature scaling \cite{guo2017calibration} in a post-processing step. However, temperature scaling only learns a single parameter to rescale all the logits, which is not flexible and insufficient. Moreover, it cannot improve out-of-distribution calibration.&#10;A second option is to mitigate miscalibration during training using regularization. For example, \cite{Pereyra2017erl} propose an entropy regularizer to prevent over-confidence, but it can needlessly hurt legitimate high confident predictions. A third option is to use Bayesian neural networks \cite{pmlr-v37-blundell15, pmlr-v70-louizos17a}, which treat model parameters as probability distributions to represent model uncertainty explicitly. However, these Bayesian approaches are often prohibitive, as the priors of the model parameters are difficult to specify, and exact inference is intractable, which can also lead to unreliable uncertainty estimates.&#10;&#10;We propose a regularization approach to addressing miscalibration for&#10; fine-tuning pre-trained language models&#10; from a data augmentation perspective.&#10; We propose two new regularizers using pseudo samples both on and&#10; off the data manifold to mitigate&#10; data scarcity and prevent over-confident predictions. &#10; Specifically, our method imposes two types of regularization for better calibration during fine-tuning:&#10;(1) On-manifold regularization: We first generate on-manifold samples by interpolating the training data and their corresponding&#10;labels along the direction learned from hidden feature space; training over such augmented on-manifold data introduces a smoothness constraint within&#10;the data manifold to improve the model calibration for in-distribution data.&#10;(2) Off-manifold regularization: We generate off-manifold samples by adding relatively large perturbations along the directions that point outward the data manifold;&#10;we penalize the negative entropy of the output distribution for such off-manifold samples to address the over-confidence issue for OOD data.&#10;&#10;We evaluate our proposed model calibration method on six text classification datasets. For in-distribution data, we measure ECE and the performance of misclassification detection. For out-of-distribution data, we measure the performance of OOD detection. Our experiments show that our method&#10;outperforms existing state-of-the-art methods in both settings, and meanwhile maintains competitive classification accuracy.&#10;&#10;We summarize our contribution as follows: (1) We propose a general calibration framework, which can be applied to pre-trained language model fine-tuning, as well as other deep neural network-based prediction problems. (2) The proposed method adopts on- and off-manifold regularization from a data augmentation perspective to improve calibration for both in-distribution and OOD data. (3) We conduct comprehensive experiments showing that our method outperforms existing calibration methods in terms of ECE, miscalssification detection and OOD detection on six text classification datasets.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Regularization Techniques, Computer Science, Linguistics, Language Model Calibration, Artificial Intelligence, Mathematics, Statistics, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="1703.00056" label="1703.00056">
        <attvalues>
          <attvalue for="0" value="Fair prediction with disparate impact: A study of bias in recidivism&#10;  prediction instruments" />
          <attvalue for="1" value="  Recidivism prediction instruments (RPI's) provide decision makers with an&#10;assessment of the likelihood that a criminal defendant will reoffend at a&#10;future point in time. While such instruments are gaining increasing popularity&#10;across the country, their use is attracting tremendous controversy. Much of the&#10;controversy concerns potential discriminatory bias in the risk assessments that&#10;are produced. This paper discusses several fairness criteria that have recently&#10;been applied to assess the fairness of recidivism prediction instruments. We&#10;demonstrate that the criteria cannot all be simultaneously satisfied when&#10;recidivism prevalence differs across groups. We then show how disparate impact&#10;can arise when a recidivism prediction instrument fails to satisfy the&#10;criterion of error rate balance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.02910" label="1703.02910">
        <attvalues>
          <attvalue for="0" value="Deep Bayesian Active Learning with Image Data" />
          <attvalue for="1" value="  Even though active learning forms an important pillar of machine learning,&#10;deep learning tools are not prevalent within it. Deep learning poses several&#10;difficulties when used in an active learning setting. First, active learning&#10;(AL) methods generally rely on being able to learn and update models from small&#10;amounts of data. Recent advances in deep learning, on the other hand, are&#10;notorious for their dependence on large amounts of data. Second, many AL&#10;acquisition functions rely on model uncertainty, yet deep learning methods&#10;rarely represent such model uncertainty. In this paper we combine recent&#10;advances in Bayesian deep learning into the active learning framework in a&#10;practical way. We develop an active learning framework for high dimensional&#10;data, a task which has been extremely challenging so far, with very sparse&#10;existing literature. Taking advantage of specialised models such as Bayesian&#10;convolutional neural networks, we demonstrate our active learning techniques&#10;with image data, obtaining a significant improvement on existing active&#10;learning approaches. We demonstrate this on both the MNIST dataset, as well as&#10;for skin cancer diagnosis from lesion images (ISIC2016 task).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.05697" label="1808.05697">
        <attvalues>
          <attvalue for="0" value="Deep Bayesian Active Learning for Natural Language Processing: Results&#10;  of a Large-Scale Empirical Study" />
          <attvalue for="1" value="  Several recent papers investigate Active Learning (AL) for mitigating the&#10;data dependence of deep learning for natural language processing. However, the&#10;applicability of AL to real-world problems remains an open question. While in&#10;supervised learning, practitioners can try many different methods, evaluating&#10;each against a validation set before selecting a model, AL affords no such&#10;luxury. Over the course of one AL run, an agent annotates its dataset&#10;exhausting its labeling budget. Thus, given a new task, an active learner has&#10;no opportunity to compare models and acquisition functions. This paper provides&#10;a large scale empirical study of deep active learning, addressing multiple&#10;tasks and, for each, multiple datasets, multiple models, and a full suite of&#10;acquisition functions. We find that across all settings, Bayesian active&#10;learning by disagreement, using uncertainty estimates provided either by&#10;Dropout or Bayes-by Backprop significantly improves over i.i.d. baselines and&#10;usually outperforms classic uncertainty sampling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1408.5882" label="1408.5882">
        <attvalues>
          <attvalue for="0" value="Convolutional Neural Networks for Sentence Classification" />
          <attvalue for="1" value="  We report on a series of experiments with convolutional neural networks (CNN)&#10;trained on top of pre-trained word vectors for sentence-level classification&#10;tasks. We show that a simple CNN with little hyperparameter tuning and static&#10;vectors achieves excellent results on multiple benchmarks. Learning&#10;task-specific vectors through fine-tuning offers further gains in performance.&#10;We additionally propose a simple modification to the architecture to allow for&#10;the use of both task-specific and static vectors. The CNN models discussed&#10;herein improve upon the state of the art on 4 out of 7 tasks, which include&#10;sentiment analysis and question classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.11423" label="1911.11423">
        <attvalues>
          <attvalue for="0" value="Single Headed Attention RNN: Stop Thinking With Your Head" />
          <attvalue for="1" value="  The leading approaches in language modeling are all obsessed with TV shows of&#10;my youth - namely Transformers and Sesame Street. Transformers this,&#10;Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic wafer&#10;scale silicon. We opt for the lazy path of old and proven techniques with a&#10;fancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN). The&#10;author's lone goal is to show that the entire field might have evolved a&#10;different direction if we had instead been obsessed with a slightly different&#10;acronym and slightly different result. We take a previously strong language&#10;model based only on boring LSTMs and get it to within a stone's throw of a&#10;stone's throw of state-of-the-art byte level language model results on enwik8.&#10;This work has undergone no intensive hyperparameter optimization and lived&#10;entirely on a commodity desktop machine that made the author's small studio&#10;apartment far too warm in the midst of a San Franciscan summer. The final&#10;results are achievable in plus or minus 24 hours on a single GPU as the author&#10;is impatient. The attention mechanism is also readily extended to large&#10;contexts with minimal computation. Take that Sesame Street.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language has been a thorn in humanity's side since we evolved a complex enough audio and graphics processing unit to grunt, let alone write cryptocurrency whitepapers or opinion columns.&#10;Language has been found at the core of every human conflict in history, from World Wars (culinary and otherwise) to the Great Border Skirmish (2008) between you and your loud neighbour.&#10;Many are of the opinion that language has redeeming features.&#10;They claim (with scant evidence) that language could contain useful knowledge far beneath the surface wasteland of memes and colourful insults we usually see, just as life might theoretically be found deep under the ice of Jupiter's moon Europa.&#10;Many fight against the homogenization of language by dividing and conquering as they did in the Tower of Babel era (see: Javascript frameworks).&#10;Regardless of how you feel about language, a gambler would expect language to exist for at least a few more years and is thus a worthy field of study.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Cryptography, Artificial Intelligence, Mathematics, Language Modeling, Computational Efficiency" />
        </attvalues>
      </node>
      <node id="2011.09708" label="2011.09708">
        <attvalues>
          <attvalue for="0" value="Are Pre-trained Language Models Knowledgeable to Ground Open Domain&#10;  Dialogues?" />
          <attvalue for="1" value="  We study knowledge-grounded dialogue generation with pre-trained language&#10;models. Instead of pursuing new state-of-the-art on benchmarks, we try to&#10;understand if the knowledge stored in parameters of the pre-trained models is&#10;already enough to ground open domain dialogues, and thus allows us to get rid&#10;of the dependency on external knowledge sources in generation. Through&#10;extensive experiments on benchmarks, we find that by fine-tuning with a few&#10;dialogues containing knowledge, the pre-trained language models can outperform&#10;the state-of-the-art model that requires external knowledge in automatic&#10;evaluation and human judgment, suggesting a positive answer to the question we&#10;raised.&#10;" />
          <attvalue for="2" value="&#10;While techniques of open domain dialogue generation \cite{vinyals2015neural,xing2017hierarchical,zhang2019recosa} have been applied in industrial products \cite{shum2018eliza,ram2018conversational}, people can still feel the gap between the dialogue systems and humans, especially when they dive into a specific topic of interest. To bridge the gap, researchers consider grounding open domain dialogues by external knowledge which could be retrieved either from structured knowledge bases \cite{zhou2018commonsense,moon2019opendialkg,tuan2019dykgchat} or from unstructured documents \cite{dinan2018wizard}. &#10;Dialogue generation now is based on both conversation contexts and the external knowledge which hints the generation model how to go deep for the topic in discussion. &#10;&#10;In this work, we investigate if a large scale pre-trained language model can instead serve as a knowledge base in open domain dialogue generation. The work is motivated by two lines of research emerging recently: (1) interestingly, some recent studies indicate that pre-trained language models have packed enough knowledge in their parameters, and thus they can do a good job in question-answering tasks without the need of access to external knowledge \cite{petroni2019language,roberts2020much}. Thus, we are curious if similar results can be achieved in open domain dialogue generation. If the answer is ``yes'', then we can get rid of the dependency on external knowledge sources, and obtain a simpler and more flexible architecture with a better generalization ability inherited from pre-training with massive text corpus \cite{radford2019language}; and (2) pre-training techniques have exhibited compelling performance on the task of open domain dialogue generation \cite{zhang2019dialogpt,wolf2019transfertransfo}. Particularly, a recent paper \cite{zhang2019dialogpt} has demonstrated with examples that a pre-trained generation model can reply with commonsense knowledge. Therefore, it is interesting to further explore to what extend a pre-trained language model can keep conversation smooth, knowledgeable, and reasonable with in-depth analysis. &#10;&#10;The test beds are benchmarks of knowledge-grounded dialogue generation, including Wizard of Wikipedia (Wizard) \cite{dinan2018wizard}, CMU Document Grounded Conversations (CMU$\_$DoG) \cite{zhou2018dataset}, and Topical-Chat (TC) \cite{gopalakrishnan2019topical}, in which we discard the external knowledge passages. The remaining dialogues are basically deep discussions about specific topics between two participants, and thus are suitable probes. We choose DialoGPT \cite{zhang2019dialogpt}, GPT-2$_{finetune}$, and DialoGPT$_{finetune}$ as the pre-trained language models for investigation, where GPT-2$_{finetune}$ and DialoGPT$_{finetune}$ refer to the OpenAI GPT-2 model \cite{radford2019language} and DialoGPT fine-tuned on the training data of the benchmarks respectively. Evaluation results on both automatic metrics and human judgment indicate that without the aid of external knowledge, the fine-tuned models are still capable of replying with proper and specific content based on the knowledge encoded in its parameters, though sometimes they may make mistakes on details. &#10;&#10;Our contributions include: (1) the first systematic study on the possibility of using a pre-trained language model as a knowledge base to ground open domain dialogues; and (2) insights and ideas for future work from extensive experiments. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Models, Dialogue Generation, Artificial Intelligence, Natural Language Processing, Knowledge Retrieval" />
        </attvalues>
      </node>
      <node id="1506.05869" label="1506.05869">
        <attvalues>
          <attvalue for="0" value="A Neural Conversational Model" />
          <attvalue for="1" value="  Conversational modeling is an important task in natural language&#10;understanding and machine intelligence. Although previous approaches exist,&#10;they are often restricted to specific domains (e.g., booking an airline ticket)&#10;and require hand-crafted rules. In this paper, we present a simple approach for&#10;this task which uses the recently proposed sequence to sequence framework. Our&#10;model converses by predicting the next sentence given the previous sentence or&#10;sentences in a conversation. The strength of our model is that it can be&#10;trained end-to-end and thus requires much fewer hand-crafted rules. We find&#10;that this straightforward model can generate simple conversations given a large&#10;conversational training dataset. Our preliminary results suggest that, despite&#10;optimizing the wrong objective function, the model is able to converse well. It&#10;is able extract knowledge from both a domain specific dataset, and from a&#10;large, noisy, and general domain dataset of movie subtitles. On a&#10;domain-specific IT helpdesk dataset, the model can find a solution to a&#10;technical problem via conversations. On a noisy open-domain movie transcript&#10;dataset, the model can perform simple forms of common sense reasoning. As&#10;expected, we also find that the lack of consistency is a common failure mode of&#10;our model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.05339" label="1907.05339">
        <attvalues>
          <attvalue for="0" value="ReCoSa: Detecting the Relevant Contexts with Self-Attention for&#10;  Multi-turn Dialogue Generation" />
          <attvalue for="1" value="  In multi-turn dialogue generation, response is usually related with only a&#10;few contexts. Therefore, an ideal model should be able to detect these relevant&#10;contexts and produce a suitable response accordingly. However, the widely used&#10;hierarchical recurrent encoderdecoder models just treat all the contexts&#10;indiscriminately, which may hurt the following response generation process.&#10;Some researchers try to use the cosine similarity or the traditional attention&#10;mechanism to find the relevant contexts, but they suffer from either&#10;insufficient relevance assumption or position bias problem. In this paper, we&#10;propose a new model, named ReCoSa, to tackle this problem. Firstly, a word&#10;level LSTM encoder is conducted to obtain the initial representation of each&#10;context. Then, the self-attention mechanism is utilized to update both the&#10;context and masked response representation. Finally, the attention weights&#10;between each context and response representations are computed and used in the&#10;further decoding process. Experimental results on both Chinese customer&#10;services dataset and English Ubuntu dialogue dataset show that ReCoSa&#10;significantly outperforms baseline models, in terms of both metric-based and&#10;human evaluations. Further analysis on attention shows that the detected&#10;relevant contexts by ReCoSa are highly coherent with human's understanding,&#10;validating the correctness and interpretability of ReCoSa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.01957" label="1801.01957">
        <attvalues>
          <attvalue for="0" value="From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots" />
          <attvalue for="1" value="  Conversational systems have come a long way since their inception in the&#10;1960s. After decades of research and development, we've seen progress from&#10;Eliza and Parry in the 60's and 70's, to task-completion systems as in the&#10;DARPA Communicator program in the 2000s, to intelligent personal assistants&#10;such as Siri in the 2010s, to today's social chatbots like XiaoIce. Social&#10;chatbots' appeal lies not only in their ability to respond to users' diverse&#10;requests, but also in being able to establish an emotional connection with&#10;users. The latter is done by satisfying users' need for communication,&#10;affection, as well as social belonging. To further the advancement and adoption&#10;of social chatbots, their design must focus on user engagement and take both&#10;intellectual quotient (IQ) and emotional quotient (EQ) into account. Users&#10;should want to engage with a social chatbot; as such, we define the success&#10;metric for social chatbots as conversation-turns per session (CPS). Using&#10;XiaoIce as an illustrative example, we discuss key technologies in building&#10;social chatbots from core chat to visual awareness to skills. We also show how&#10;XiaoIce can dynamically recognize emotion and engage the user throughout long&#10;conversations with appropriate interpersonal responses. As we become the first&#10;generation of humans ever living with AI, we have a responsibility to design&#10;social chatbots to be both useful and empathetic, so they will become&#10;ubiquitous and help society as a whole.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.03604" label="1801.03604">
        <attvalues>
          <attvalue for="0" value="Conversational AI: The Science Behind the Alexa Prize" />
          <attvalue for="1" value="  Conversational agents are exploding in popularity. However, much work remains&#10;in the area of social conversation as well as free-form conversation over a&#10;broad range of domains and topics. To advance the state of the art in&#10;conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar&#10;university competition where sixteen selected university teams were challenged&#10;to build conversational agents, known as socialbots, to converse coherently and&#10;engagingly with humans on popular topics such as Sports, Politics,&#10;Entertainment, Fashion and Technology for 20 minutes. The Alexa Prize offers&#10;the academic community a unique opportunity to perform research with a live&#10;system used by millions of users. The competition provided university teams&#10;with real user conversational data at scale, along with the user-provided&#10;ratings and feedback augmented with annotations by the Alexa team. This enabled&#10;teams to effectively iterate and make improvements throughout the competition&#10;while being evaluated in real-time through live user interactions. To build&#10;their socialbots, university teams combined state-of-the-art techniques with&#10;novel strategies in the areas of Natural Language Understanding, Context&#10;Modeling, Dialog Management, Response Generation, and Knowledge Acquisition. To&#10;support the efforts of participating teams, the Alexa Prize team made&#10;significant scientific and engineering investments to build and improve&#10;Conversational Speech Recognition, Topic Tracking, Dialog Evaluation, Voice&#10;User Experience, and tools for traffic management and scalability. This paper&#10;outlines the advances created by the university teams as well as the Alexa&#10;Prize team to achieve the common goal of solving the problem of Conversational&#10;AI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.00610" label="1910.00610">
        <attvalues>
          <attvalue for="0" value="DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic&#10;  Knowledge Graphs" />
          <attvalue for="1" value="  Data-driven, knowledge-grounded neural conversation models are capable of&#10;generating more informative responses. However, these models have not yet&#10;demonstrated that they can zero-shot adapt to updated, unseen knowledge graphs.&#10;This paper proposes a new task about how to apply dynamic knowledge graphs in&#10;neural conversation model and presents a novel TV series conversation corpus&#10;(DyKgChat) for the task. Our new task and corpus aids in understanding the&#10;influence of dynamic knowledge graphs on responses generation. Also, we propose&#10;a preliminary model that selects an output from two networks at each time step:&#10;a sequence-to-sequence model (Seq2Seq) and a multi-hop reasoning model, in&#10;order to support dynamic knowledge graphs. To benchmark this new task and&#10;evaluate the capability of adaptation, we introduce several evaluation metrics&#10;and the experiments show that our proposed approach outperforms previous&#10;knowledge-grounded conversation models. The proposed corpus and model can&#10;motivate the future research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;The proposed task is motivated by prior knowledge-grounded conversation tasks \cite{ghazvininejad2018knowledge, zhou2018commonsense}, but further requires the capability to adapt to dynamic knowledge graphs.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1811.01241" label="1811.01241">
        <attvalues>
          <attvalue for="0" value="Wizard of Wikipedia: Knowledge-Powered Conversational agents" />
          <attvalue for="1" value="  In open-domain dialogue intelligent agents should exhibit the use of&#10;knowledge, however there are few convincing demonstrations of this to date. The&#10;most popular sequence to sequence models typically &quot;generate and hope&quot; generic&#10;utterances that can be memorized in the weights of the model when mapping from&#10;input utterance(s) to output, rather than employing recalled knowledge as&#10;context. Use of knowledge has so far proved difficult, in part because of the&#10;lack of a supervised learning benchmark task which exhibits knowledgeable open&#10;dialogue with clear grounding. To that end we collect and release a large&#10;dataset with conversations directly grounded with knowledge retrieved from&#10;Wikipedia. We then design architectures capable of retrieving knowledge,&#10;reading and conditioning on it, and finally generating natural responses. Our&#10;best performing dialogue models are able to conduct knowledgeable discussions&#10;on open-domain topics as evaluated by automatic metrics and human evaluations,&#10;while our new benchmark allows for measuring further improvements in this&#10;important research direction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.07358" label="1809.07358">
        <attvalues>
          <attvalue for="0" value="A Dataset for Document Grounded Conversations" />
          <attvalue for="1" value="  This paper introduces a document grounded dataset for text conversations. We&#10;define &quot;Document Grounded Conversations&quot; as conversations that are about the&#10;contents of a specified document. In this dataset the specified documents were&#10;Wikipedia articles about popular movies. The dataset contains 4112&#10;conversations with an average of 21.43 turns per conversation. This positions&#10;this dataset to not only provide a relevant chat history while generating&#10;responses but also provide a source of information that the models could use.&#10;We describe two neural architectures that provide benchmark performance on the&#10;task of generating the next response. We also evaluate our models for&#10;engagement and fluency, and find that the information from the document helps&#10;in generating more engaging and fluent responses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.11995" label="2308.11995">
        <attvalues>
          <attvalue for="0" value="Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations" />
          <attvalue for="1" value="  Building socialbots that can have deep, engaging open-domain conversations&#10;with humans is one of the grand challenges of artificial intelligence (AI). To&#10;this end, bots need to be able to leverage world knowledge spanning several&#10;domains effectively when conversing with humans who have their own world&#10;knowledge. Existing knowledge-grounded conversation datasets are primarily&#10;stylized with explicit roles for conversation partners. These datasets also do&#10;not explore depth or breadth of topical coverage with transitions in&#10;conversations. We introduce Topical-Chat, a knowledge-grounded human-human&#10;conversation dataset where the underlying knowledge spans 8 broad topics and&#10;conversation partners don't have explicitly defined roles, to help further&#10;research in open-domain conversational AI. We also train several&#10;state-of-the-art encoder-decoder conversational models on Topical-Chat and&#10;perform automated and human evaluation for benchmarking.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.06875" label="2304.06875">
        <attvalues>
          <attvalue for="0" value="nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss&#10;  Prediction across Scales" />
          <attvalue for="1" value="  As language models scale up, it becomes increasingly expensive to verify&#10;research ideas because conclusions on small models do not trivially transfer to&#10;large ones. A possible solution is to establish a generic system that&#10;accurately predicts certain metrics for large models without training them.&#10;Existing scaling laws require hyperparameter search on the largest models,&#10;limiting their predicative capability. In this paper, we present an approach&#10;(namely {\mu}Scaling) to predict the pre-training loss, based on our&#10;observations that Maximal Update Parametrization ({\mu}P) enables accurate&#10;fitting of scaling laws close to common loss basins in hyperparameter space.&#10;With {\mu}Scaling, different model designs can be compared on large scales by&#10;training only their smaller counterparts. Further, we introduce nanoLM: an&#10;affordable LLM pre-training benchmark that facilitates this new research&#10;paradigm. With around 14% of the one-time pre-training cost, we can accurately&#10;forecast the loss for models up to 52B. Our goal with nanoLM is to empower&#10;researchers with limited resources to reach meaningful conclusions on large&#10;models. We also aspire for our benchmark to serve as a bridge between the&#10;academic community and the industry. Code for {\mu}Scaling is available at&#10;https://github.com/cofe-ai/Mu-scaling. Code for nanoLLM will be available&#10;later.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models (LLMs) pre-trained on Web-scale data have demonstrated impressive performance on various downstream tasks under a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning. &#10;&#10;Modern LLMs are based on the Transformer architecture \cite{DBLP:conf/nips/VaswaniSPUJGKP17}, and can be trained with unsupervised objectives including causal language modeling \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}, masked language modeling, among others \cite{DBLP:conf/icml/WangRHSCBLR22}. Since researches on scaling laws \cite{DBLP:journals/corr/abs-2001-08361, DBLP:journals/corr/abs-2203-15556} reveal the potential of improving model performance by increasing the total computation, the community have been scaling up both the model sizes and training data \cite{DBLP:conf/nips/BrownMRSKDNSSAA20, DBLP:journals/corr/abs-2201-11990, DBLP:journals/corr/abs-2307-09288, falcon40b}, as briefly summarized in Table \ref{tab: current llm}.&#10;&#10;However, this trend makes it increasingly difficult for average researchers to verify research ideas or search for hyperparameters (HPs): first, conclusions achieved with small models do not trivially transfer to larger ones; on the other hand, directly training large models for multiple times is a costly endeavor. For instance, as reported by Llama-2 \cite{DBLP:journals/corr/abs-2307-09288}, time to train the 7B, 13B, and 70B models on roughly 2 trillion tokens is 184k, 368k, and 1.7M GPU hours with A100-80GB, respectively. An entry point as such prevents the community from swiftly making improvements that are reliable on large scales, and results in enormous waste of computational resources and intellectual efforts. Thus, it is necessary to establish a pipeline to compare different LLM structures, algorithms, and hyperparameters with limited computational resources (i.e., small models), while making sure the results are instructive for any model scale. This is the goal of this paper.&#10;&#10;As a possible solution to this issue, the technical report of GPT-4 \cite{gpt4} showed that some behaviors of large models can be predicted before the training starts (with unpublished methods). In this paper, we start by proposing a method that yields accurate loss prediction, namely $\mu$Scaling ~(a compound word of $\mu$P \cite{DBLP:journals/corr/abs-2203-03466} and Scaling Laws \cite{DBLP:journals/corr/abs-2001-08361}), with experimental results supporting its correctness. Based on $\mu$Scaling, we establish a new paradigm for meaningful research on large models without actually training them. Finally, we propose an affordable benchmark for LLM pre-training studies, namely nanoLM, which facilities this research paradigm for the community.&#10;&#10;Contributions. &#10;We substantiate our contributions as follows:&#10;\begin{itemize}&#10;\item We propose $\mu$Scaling, a loss prediction method based on $\mu$P and modified scaling laws. For hyperparameters (HPs) in the common loss basins, the training loss can be accurately predicted by a power-law function w.r.t. model sizes, which includes embedding sizes, in contrast to existing methods. This method requires searching for the optimal HP only once to predict loss in arbitrary model scale.&#10;\item We unlock a new LLM study paradigm that can directly compare the loss for different model designs on large scales without direct training. We facilitate this paradigm by proposing nanoLM, an affordable benchmark. nanoLM is compatible with mainstream Transformer architectures, including decoder-only structures (e.g., GPT, Llama), encoder-only structures (e.g., BERT), and encoder-decoder structures (e.g., T5), and supports data parallelism strategies (Section \ref{method}). For benchmark evaluation, we publicly release a pre-training dataset with 100B/400B/1T/2T tokens, chosen from existing sources and categorized into various specialized domains (Section \ref{data benchmark}).&#10;\item Effectiveness: Empirically, we successfully utilized our method to forecast the loss for 12-layer GPT, Llama, BERT, and T5 models on the C4 and MC4 dataset. &#10;For more expansive models, we experiment on GPT models with 32 and 64 layers, culminating in sizes of 26B and 52B, respectively. Results indicate that the actual loss remains predictable (Section \ref{fitting result}).&#10;\item Efficiency: By generating a series of small proxy models with sizes ranging from 38M to 3.4B, predicting loss using $\mu$Scaling incurs only $13.1\%, 14.2\%$ of the one-time pre-training cost for 26B and 52B models, respectively. &#10;This demonstrates that nanoLM can help researchers make affordable and meaningful comparisons between different model designs and serve as a new benchmark for LLM study (Section \ref{efficiency and performance}).&#10;\end{itemize}&#10;To foster reproducibility, we will open-source all our code and data of nanoLM benchmark. Part of our code is attached in ``Supplementary Material''.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Predictive Scaling Methods, Efficient Research Paradigms, Engineering, Mathematics" />
        </attvalues>
      </node>
      <node id="2203.03466" label="2203.03466">
        <attvalues>
          <attvalue for="0" value="Tensor Programs V: Tuning Large Neural Networks via Zero-Shot&#10;  Hyperparameter Transfer" />
          <attvalue for="1" value="  Hyperparameter (HP) tuning in deep learning is an expensive process,&#10;prohibitively so for neural networks (NNs) with billions of parameters. We show&#10;that, in the recently discovered Maximal Update Parametrization (muP), many&#10;optimal HPs remain stable even as model size changes. This leads to a new HP&#10;tuning paradigm we call muTransfer: parametrize the target model in muP, tune&#10;the HP indirectly on a smaller model, and zero-shot transfer them to the&#10;full-sized model, i.e., without directly tuning the latter at all. We verify&#10;muTransfer on Transformer and ResNet. For example, 1) by transferring&#10;pretraining HPs from a model of 13M parameters, we outperform published numbers&#10;of BERT-large (350M parameters), with a total tuning cost equivalent to&#10;pretraining BERT-large once; 2) by transferring from 40M parameters, we&#10;outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7%&#10;of total pretraining cost. A Pytorch implementation of our technique can be&#10;found at github.com/microsoft/mup and installable via `pip install mup`.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.15752" label="2406.15752">
        <attvalues>
          <attvalue for="0" value="TacoLM: GaTed Attention Equipped Codec Language Model are Efficient&#10;  Zero-Shot Text to Speech Synthesizers" />
          <attvalue for="1" value="  Neural codec language model (LM) has demonstrated strong capability in&#10;zero-shot text-to-speech (TTS) synthesis. However, the codec LM often suffers&#10;from limitations in inference speed and stability, due to its auto-regressive&#10;nature and implicit alignment between text and audio. In this work, to handle&#10;these challenges, we introduce a new variant of neural codec LM, namely TacoLM.&#10;Specifically, TacoLM introduces a gated attention mechanism to improve the&#10;training and inference efficiency and reduce the model size. Meanwhile, an&#10;additional gated cross-attention layer is included for each decoder layer,&#10;which improves the efficiency and content accuracy of the synthesized speech.&#10;In the evaluation of the Librispeech corpus, the proposed TacoLM achieves a&#10;better word error rate, speaker similarity, and mean opinion score, with 90%&#10;fewer parameters and 5.2 times speed up, compared with VALL-E. Demo and code is&#10;available at https://ereboas.github.io/TacoLM/.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the development of deep neural networks, text-to-speech (TTS) technology has made significant progress~\cite{glowtts,gradtts,paralleltacotron,speartts}. Among them, zero-shot TTS only requires a short audio sample as prompts to synthesize high-quality speech for any unseen speaker. Zero-shot TTS does not need to be fine-tuned with speech data with respect to new speakers, which is of more practical value but still challenging. Early studies use continuous audio signals as input, relying on an explicit speaker encoder to embed a speaker's timbre, prosody, and speaking style~\cite{scglowtts,yourtts}. Some studies further rely on specifically designed speech disentanglement approaches to extract speaker-agnostic information~\cite{kumar2022zero,generspeech}. However, when the speaker embedding is inaccurate, the model's ability to generalize to zero-shot scenarios decreases dramatically. &#10;&#10;Recent developments in diffusion and language modeling bring breakthroughs to the zero-shot TTS. The former~\cite{NS2,voicebox,fastdiff} leverages the diffusion model~\cite{ho2020ddpm} and its variants~\cite{flowmatching,yang2022diffusion} to estimate the target speech that shares the same distribution as the enrollment, while the latter~\cite{audiolm} usually employs language models on a pre-trained neural codec to extract discrete audio representations and reconstruct high-quality waveforms. Both systems achieve impressive performance in the field of zero-shot speech synthesis. As the neural codec language model doesn't require a separated duration prediction model, which potentially enables a more direct end-to-end optimization, we focus on the improvement of this algorithm family in this work. &#10;In the domain of neural codec language models, VALL-E~\cite{valle} is a prototypical and highly effective two-stage approach. Specifically, VALL-E takes the phoneme and acoustic tokens as prompts, and leverages an autoregressive (AR) and a non-autoregressive (NAR) language model, to generate coarse and fine-grained acoustic tokens of the unenrolled speaker, respectively. This powerful approach obviates the need for encoding the speaker into embeddings, while allowing for direct extraction of the environment and acoustic information from the audio samples. &#10;&#10;Despite its achievements, the VALL-E still suffers from many drawbacks. One drawback that affects the practical application experience is the slow speed, arising from the use of an AR model to generate coarse-grained acoustic tokens. While the multi-head attention mechanism in the Transformer can model the relationship between pairs of tokens well, it often faces large computational and memory costs. The speed deficiency is especially prominent in the inference process, since the AR model needs to generate outputs in a token-by-token manner. Another limitation of VALL-E is the occasional mismatch between the synthesized speech and text prompts, which can manifest as repetitions, transpositions, or omissions. This is because the model does not align text and speech well. How to realize zero-shot TTS in both an efficient and accurate manner remains an open challenge.&#10;&#10;In this study, we address the zero-shot TTS task with our proposed TacoLM (Ga\underline{T}ed \underline{A}ttention Equipped \underline{Co}dec \underline{L}anguage \underline{M}odel), which is a two-stage (AR + NAR) framework, similar to VALL-E.&#10;TacoLM incorporates a MEGA module, as detailed in \cite{mega}, which is based on a single-head gated attention mechanism with an exponential moving average. &#10;As a result, TacoLM is computationally efficient and requires minimal memory and storage.&#10;In addition, TacoLM employs a gated cross-attention mechanism to exchange information between text and audio, aiming at enhancing the accuracy of the synthesized speech.&#10;To evaluate TacoLM in the zero-shot scenario, we conducted experiments on the LibriSpeech~\cite{librispeech} dataset. The experimental results show that TacoLM is superior to the advanced baseline VALL-E in terms of both objective (WER and speaker similarity) and subjective metrics (CMOS and SMOS). &#10;Ablation studies further corroborate the individual contributions of TacoLM's components to its overall effectiveness.&#10;Our contributions can be summarized as follows:&#10;\begin{itemize}[itemsep=0.5mm, parsep=1pt, leftmargin=*]&#10; \item We propose TacoLM, a two-stage zero-shot text-to-speech framework, which first discretizes the audio and text, and then relies on language models to demonstrate strong zero-shot speech synthesis capabilities. The training of TacoLM is also open-source to facilitate research in this direction. &#10; \item We train the discrete speech language model leveraging on a moving average equipped gated attention mechanism (MEGA), which has a lightweight computation and storage compared to the vanilla multi-head attention.&#10; \item We design a novel gated cross-attention layer that efficiently links up information between text and the audio sequence, further improving the performance and efficiency.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Text-to-Speech Synthesis, Linguistics, Engineering, Speech Generation Efficiency, Signal Processing, Neural Language Models" />
        </attvalues>
      </node>
      <node id="2005.11129" label="2005.11129">
        <attvalues>
          <attvalue for="0" value="Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment&#10;  Search" />
          <attvalue for="1" value="  Recently, text-to-speech (TTS) models such as FastSpeech and ParaNet have&#10;been proposed to generate mel-spectrograms from text in parallel. Despite the&#10;advantage, the parallel TTS models cannot be trained without guidance from&#10;autoregressive TTS models as their external aligners. In this work, we propose&#10;Glow-TTS, a flow-based generative model for parallel TTS that does not require&#10;any external aligner. By combining the properties of flows and dynamic&#10;programming, the proposed model searches for the most probable monotonic&#10;alignment between text and the latent representation of speech on its own. We&#10;demonstrate that enforcing hard monotonic alignments enables robust TTS, which&#10;generalizes to long utterances, and employing generative flows enables fast,&#10;diverse, and controllable speech synthesis. Glow-TTS obtains an&#10;order-of-magnitude speed-up over the autoregressive model, Tacotron 2, at&#10;synthesis with comparable speech quality. We further show that our model can be&#10;easily extended to a multi-speaker setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.06337" label="2105.06337">
        <attvalues>
          <attvalue for="0" value="Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech" />
          <attvalue for="1" value="  Recently, denoising diffusion probabilistic models and generative score&#10;matching have shown high potential in modelling complex data distributions&#10;while stochastic calculus has provided a unified point of view on these&#10;techniques allowing for flexible inference schemes. In this paper we introduce&#10;Grad-TTS, a novel text-to-speech model with score-based decoder producing&#10;mel-spectrograms by gradually transforming noise predicted by encoder and&#10;aligned with text input by means of Monotonic Alignment Search. The framework&#10;of stochastic differential equations helps us to generalize conventional&#10;diffusion probabilistic models to the case of reconstructing data from noise&#10;with different parameters and allows to make this reconstruction flexible by&#10;explicitly controlling trade-off between sound quality and inference speed.&#10;Subjective human evaluation shows that Grad-TTS is competitive with&#10;state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We&#10;will make the code publicly available shortly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11439" label="2010.11439">
        <attvalues>
          <attvalue for="0" value="Parallel Tacotron: Non-Autoregressive and Controllable TTS" />
          <attvalue for="1" value="  Although neural end-to-end text-to-speech models can synthesize highly&#10;natural speech, there is still room for improvements to its efficiency and&#10;naturalness. This paper proposes a non-autoregressive neural text-to-speech&#10;model augmented with a variational autoencoder-based residual encoder. This&#10;model, called \emph{Parallel Tacotron}, is highly parallelizable during both&#10;training and inference, allowing efficient synthesis on modern parallel&#10;hardware. The use of the variational autoencoder relaxes the one-to-many&#10;mapping nature of the text-to-speech problem and improves naturalness. To&#10;further improve the naturalness, we use lightweight convolutions, which can&#10;efficiently capture local contexts, and introduce an iterative spectrogram loss&#10;inspired by iterative refinement. Experimental results show that Parallel&#10;Tacotron matches a strong autoregressive baseline in subjective evaluations&#10;with significantly decreased inference time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.05557" label="2104.05557">
        <attvalues>
          <attvalue for="0" value="SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model" />
          <attvalue for="1" value="  In this paper, we propose SC-GlowTTS: an efficient zero-shot multi-speaker&#10;text-to-speech model that improves similarity for speakers unseen during&#10;training. We propose a speaker-conditional architecture that explores a&#10;flow-based decoder that works in a zero-shot scenario. As text encoders, we&#10;explore a dilated residual convolutional-based encoder, gated&#10;convolutional-based encoder, and transformer-based encoder. Additionally, we&#10;have shown that adjusting a GAN-based vocoder for the spectrograms predicted by&#10;the TTS model on the training dataset can significantly improve the similarity&#10;and speech quality for new speakers. Our model converges using only 11&#10;speakers, reaching state-of-the-art results for similarity with new speakers,&#10;as well as high speech quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.02418" label="2112.02418">
        <attvalues>
          <attvalue for="0" value="YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice&#10;  Conversion for everyone" />
          <attvalue for="1" value="  YourTTS brings the power of a multilingual approach to the task of zero-shot&#10;multi-speaker TTS. Our method builds upon the VITS model and adds several novel&#10;modifications for zero-shot multi-speaker and multilingual training. We&#10;achieved state-of-the-art (SOTA) results in zero-shot multi-speaker TTS and&#10;results comparable to SOTA in zero-shot voice conversion on the VCTK dataset.&#10;Additionally, our approach achieves promising results in a target language with&#10;a single-speaker dataset, opening possibilities for zero-shot multi-speaker TTS&#10;and zero-shot voice conversion systems in low-resource languages. Finally, it&#10;is possible to fine-tune the YourTTS model with less than 1 minute of speech&#10;and achieve state-of-the-art results in voice similarity and with reasonable&#10;quality. This is important to allow synthesis for speakers with a very&#10;different voice or recording characteristics from those seen during training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.07211" label="2205.07211">
        <attvalues>
          <attvalue for="0" value="GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain&#10;  Text-to-Speech" />
          <attvalue for="1" value="  Style transfer for out-of-domain (OOD) speech synthesis aims to generate&#10;speech samples with unseen style (e.g., speaker identity, emotion, and prosody)&#10;derived from an acoustic reference, while facing the following challenges: 1)&#10;The highly dynamic style features in expressive voice are difficult to model&#10;and transfer; and 2) the TTS models should be robust enough to handle diverse&#10;OOD conditions that differ from the source data. This paper proposes&#10;GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style&#10;transfer of OOD custom voice. GenerSpeech decomposes the speech variation into&#10;the style-agnostic and style-specific parts by introducing two components: 1) a&#10;multi-level style adaptor to efficiently model a large range of style&#10;conditions, including global speaker and emotion characteristics, and the local&#10;(utterance, phoneme, and word-level) fine-grained prosodic representations; and&#10;2) a generalizable content adaptor with Mix-Style Layer Normalization to&#10;eliminate style information in the linguistic content representation and thus&#10;improve model generalization. Our evaluations on zero-shot style transfer&#10;demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of&#10;audio quality and style similarity. The extension studies to adaptive style&#10;transfer further show that GenerSpeech performs robustly in the few-shot data&#10;setting. Audio samples are available at https://GenerSpeech.github.io/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.09116" label="2304.09116">
        <attvalues>
          <attvalue for="0" value="NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot&#10;  Speech and Singing Synthesizers" />
          <attvalue for="1" value="  Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild&#10;datasets is important to capture the diversity in human speech such as speaker&#10;identities, prosodies, and styles (e.g., singing). Current large TTS systems&#10;usually quantize speech into discrete tokens and use language models to&#10;generate these tokens one by one, which suffer from unstable prosody, word&#10;skipping/repeating issue, and poor voice quality. In this paper, we develop&#10;NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual&#10;vector quantizers to get the quantized latent vectors and uses a diffusion&#10;model to generate these latent vectors conditioned on text input. To enhance&#10;the zero-shot capability that is important to achieve diverse speech synthesis,&#10;we design a speech prompting mechanism to facilitate in-context learning in the&#10;diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to&#10;large-scale datasets with 44K hours of speech and singing data and evaluate its&#10;voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS&#10;systems by a large margin in terms of prosody/timbre similarity, robustness,&#10;and voice quality in a zero-shot setting, and performs novel zero-shot singing&#10;synthesis with only a speech prompt. Audio samples are available at&#10;https://speechresearch.github.io/naturalspeech2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.15687" label="2306.15687">
        <attvalues>
          <attvalue for="0" value="Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale" />
          <attvalue for="1" value="  Large-scale generative models such as GPT and DALL-E have revolutionized the&#10;research community. These models not only generate high fidelity outputs, but&#10;are also generalists which can solve tasks not explicitly taught. In contrast,&#10;speech generative models are still primitive in terms of scale and task&#10;generalization. In this paper, we present Voicebox, the most versatile&#10;text-guided generative model for speech at scale. Voicebox is a&#10;non-autoregressive flow-matching model trained to infill speech, given audio&#10;context and text, trained on over 50K hours of speech that are not filtered or&#10;enhanced. Similar to GPT, Voicebox can perform many different tasks through&#10;in-context learning, but is more flexible as it can also condition on future&#10;context. Voicebox can be used for mono or cross-lingual zero-shot&#10;text-to-speech synthesis, noise removal, content editing, style conversion, and&#10;diverse sample generation. In particular, Voicebox outperforms the&#10;state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs&#10;1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to&#10;20 times faster. Audio samples can be found in&#10;\url{https://voicebox.metademolab.com}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.11239" label="2006.11239">
        <attvalues>
          <attvalue for="0" value="Denoising Diffusion Probabilistic Models" />
          <attvalue for="1" value="  We present high quality image synthesis results using diffusion probabilistic&#10;models, a class of latent variable models inspired by considerations from&#10;nonequilibrium thermodynamics. Our best results are obtained by training on a&#10;weighted variational bound designed according to a novel connection between&#10;diffusion probabilistic models and denoising score matching with Langevin&#10;dynamics, and our models naturally admit a progressive lossy decompression&#10;scheme that can be interpreted as a generalization of autoregressive decoding.&#10;On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and&#10;a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality&#10;similar to ProgressiveGAN. Our implementation is available at&#10;https://github.com/hojonathanho/diffusion&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.02747" label="2210.02747">
        <attvalues>
          <attvalue for="0" value="Flow Matching for Generative Modeling" />
          <attvalue for="1" value="  We introduce a new paradigm for generative modeling built on Continuous&#10;Normalizing Flows (CNFs), allowing us to train CNFs at unprecedented scale.&#10;Specifically, we present the notion of Flow Matching (FM), a simulation-free&#10;approach for training CNFs based on regressing vector fields of fixed&#10;conditional probability paths. Flow Matching is compatible with a general&#10;family of Gaussian probability paths for transforming between noise and data&#10;samples -- which subsumes existing diffusion paths as specific instances.&#10;Interestingly, we find that employing FM with diffusion paths results in a more&#10;robust and stable alternative for training diffusion models. Furthermore, Flow&#10;Matching opens the door to training CNFs with other, non-diffusion probability&#10;paths. An instance of particular interest is using Optimal Transport (OT)&#10;displacement interpolation to define the conditional probability paths. These&#10;paths are more efficient than diffusion paths, provide faster training and&#10;sampling, and result in better generalization. Training CNFs using Flow&#10;Matching on ImageNet leads to consistently better performance than alternative&#10;diffusion-based methods in terms of both likelihood and sample quality, and&#10;allows fast and reliable sample generation using off-the-shelf numerical ODE&#10;solvers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.00796" label="2209.00796">
        <attvalues>
          <attvalue for="0" value="Diffusion Models: A Comprehensive Survey of Methods and Applications" />
          <attvalue for="1" value="  Diffusion models have emerged as a powerful new family of deep generative&#10;models with record-breaking performance in many applications, including image&#10;synthesis, video generation, and molecule design. In this survey, we provide an&#10;overview of the rapidly expanding body of work on diffusion models,&#10;categorizing the research into three key areas: efficient sampling, improved&#10;likelihood estimation, and handling data with special structures. We also&#10;discuss the potential for combining diffusion models with other generative&#10;models for enhanced results. We further review the wide-ranging applications of&#10;diffusion models in fields spanning from computer vision, natural language&#10;generation, temporal data modeling, to interdisciplinary applications in other&#10;scientific disciplines. This survey aims to provide a contextualized, in-depth&#10;look at the state of diffusion models, identifying the key areas of focus and&#10;pointing to potential areas for further exploration. Github:&#10;https://github.com/YangLing0818/Diffusion-Models-Papers-Survey-Taxonomy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.03143" label="2209.03143">
        <attvalues>
          <attvalue for="0" value="AudioLM: a Language Modeling Approach to Audio Generation" />
          <attvalue for="1" value="  We introduce AudioLM, a framework for high-quality audio generation with&#10;long-term consistency. AudioLM maps the input audio to a sequence of discrete&#10;tokens and casts audio generation as a language modeling task in this&#10;representation space. We show how existing audio tokenizers provide different&#10;trade-offs between reconstruction quality and long-term structure, and we&#10;propose a hybrid tokenization scheme to achieve both objectives. Namely, we&#10;leverage the discretized activations of a masked language model pre-trained on&#10;audio to capture long-term structure and the discrete codes produced by a&#10;neural audio codec to achieve high-quality synthesis. By training on large&#10;corpora of raw audio waveforms, AudioLM learns to generate natural and coherent&#10;continuations given short prompts. When trained on speech, and without any&#10;transcript or annotation, AudioLM generates syntactically and semantically&#10;plausible speech continuations while also maintaining speaker identity and&#10;prosody for unseen speakers. Furthermore, we demonstrate how our approach&#10;extends beyond speech by generating coherent piano music continuations, despite&#10;being trained without any symbolic representation of music.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.07503" label="1506.07503">
        <attvalues>
          <attvalue for="0" value="Attention-Based Models for Speech Recognition" />
          <attvalue for="1" value="  Recurrent sequence generators conditioned on input data through an attention&#10;mechanism have recently shown very good performance on a range of tasks in-&#10;cluding machine translation, handwriting synthesis and image caption gen-&#10;eration. We extend the attention-mechanism with features needed for speech&#10;recognition. We show that while an adaptation of the model used for machine&#10;translation in reaches a competitive 18.7% phoneme error rate (PER) on the&#10;TIMIT phoneme recognition task, it can only be applied to utterances which are&#10;roughly as long as the ones it was trained on. We offer a qualitative&#10;explanation of this failure and propose a novel and generic method of adding&#10;location-awareness to the attention mechanism to alleviate this issue. The new&#10;method yields a model that is robust to long inputs and achieves 18% PER in&#10;single utterances and 20% in 10-times longer (repeated) utterances. Finally, we&#10;propose a change to the at- tention mechanism that prevents it from&#10;concentrating too much on single frames, which further reduces PER to 17.6%&#10;level.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.07413" label="1707.07413">
        <attvalues>
          <attvalue for="0" value="Exploring Neural Transducers for End-to-End Speech Recognition" />
          <attvalue for="1" value="  In this work, we perform an empirical comparison among the CTC,&#10;RNN-Transducer, and attention-based Seq2Seq models for end-to-end speech&#10;recognition. We show that, without any language model, Seq2Seq and&#10;RNN-Transducer models both outperform the best reported CTC models with a&#10;language model, on the popular Hub5'00 benchmark. On our internal diverse&#10;dataset, these trends continue - RNNTransducer models rescored with a language&#10;model after beam search outperform our best CTC models. These results simplify&#10;the speech recognition pipeline so that decoding can now be expressed purely as&#10;neural network operations. We also study how the choice of encoder architecture&#10;affects the performance of the three models - when all encoder layers are&#10;forward only, and when encoders downsample the input representation&#10;aggressively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.08091" label="1705.08091">
        <attvalues>
          <attvalue for="0" value="Local Monotonic Attention Mechanism for End-to-End Speech and Language&#10;  Processing" />
          <attvalue for="1" value="  Recently, encoder-decoder neural networks have shown impressive performance&#10;on many sequence-related tasks. The architecture commonly uses an attentional&#10;mechanism which allows the model to learn alignments between the source and the&#10;target sequence. Most attentional mechanisms used today is based on a global&#10;attention property which requires a computation of a weighted summarization of&#10;the whole input sequence generated by encoder states. However, it is&#10;computationally expensive and often produces misalignment on the longer input&#10;sequence. Furthermore, it does not fit with monotonous or left-to-right nature&#10;in several tasks, such as automatic speech recognition (ASR),&#10;grapheme-to-phoneme (G2P), etc. In this paper, we propose a novel attention&#10;mechanism that has local and monotonic properties. Various ways to control&#10;those properties are also explored. Experimental results on ASR, G2P and&#10;machine translation between two languages with similar sentence structures,&#10;demonstrate that the proposed encoder-decoder model with local monotonic&#10;attention could achieve significant performance improvements and reduce the&#10;computational complexity in comparison with the one that used the standard&#10;global attention architecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.00784" label="1704.00784">
        <attvalues>
          <attvalue for="0" value="Online and Linear-Time Attention by Enforcing Monotonic Alignments" />
          <attvalue for="1" value="  Recurrent neural network models with an attention mechanism have proven to be&#10;extremely effective on a wide variety of sequence-to-sequence problems.&#10;However, the fact that soft attention mechanisms perform a pass over the entire&#10;input sequence when producing each element in the output sequence precludes&#10;their use in online settings and results in a quadratic time complexity. Based&#10;on the insight that the alignment between input and output sequence elements is&#10;monotonic in many problems of interest, we propose an end-to-end differentiable&#10;method for learning monotonic alignments which, at test time, enables computing&#10;attention online and in linear time. We validate our approach on sentence&#10;summarization, machine translation, and online speech recognition problems and&#10;achieve results competitive with existing sequence-to-sequence models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.10088" label="1808.10088">
        <attvalues>
          <attvalue for="0" value="End-to-end Speech Recognition with Adaptive Computation Steps" />
          <attvalue for="1" value="  In this paper, we present Adaptive Computation Steps (ACS) algo-rithm, which&#10;enables end-to-end speech recognition models to dy-namically decide how many&#10;frames should be processed to predict a linguistic output. The model that&#10;applies ACS algorithm follows the encoder-decoder framework, while unlike the&#10;attention-based mod-els, it produces alignments independently at the encoder&#10;side using the correlation between adjacent frames. Thus, predictions can be&#10;made as soon as sufficient acoustic information is received, which makes the&#10;model applicable in online cases. Besides, a small change is made to the&#10;decoding stage of the encoder-decoder framework, which allows the prediction to&#10;exploit bidirectional contexts. We verify the ACS algorithm on a Mandarin&#10;speech corpus AIShell-1, and it achieves a 31.2% CER in the online occasion,&#10;compared to the 32.4% CER of the attention-based model. To fully demonstrate&#10;the advantage of ACS algorithm, offline experiments are conducted, in which our&#10;ACS model achieves an 18.7% CER, outperforming the attention-based counterpart&#10;with the CER of 22.0%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.11235" label="1905.11235">
        <attvalues>
          <attvalue for="0" value="CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition" />
          <attvalue for="1" value="  In this paper, we propose a novel soft and monotonic alignment mechanism used&#10;for sequence transduction. It is inspired by the integrate-and-fire model in&#10;spiking neural networks and employed in the encoder-decoder framework consists&#10;of continuous functions, thus being named as: Continuous Integrate-and-Fire&#10;(CIF). Applied to the ASR task, CIF not only shows a concise calculation, but&#10;also supports online recognition and acoustic boundary positioning, thus&#10;suitable for various ASR scenarios. Several support strategies are also&#10;proposed to alleviate the unique problems of CIF-based model. With the joint&#10;action of these methods, the CIF-based model shows competitive performance.&#10;Notably, it achieves a word error rate (WER) of 2.86% on the test-clean of&#10;Librispeech and creates new state-of-the-art result on Mandarin telephone ASR&#10;benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.07597" label="2301.07597">
        <attvalues>
          <attvalue for="0" value="How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation,&#10;  and Detection" />
          <attvalue for="1" value="  The introduction of ChatGPT has garnered widespread attention in both&#10;academic and industrial communities. ChatGPT is able to respond effectively to&#10;a wide range of human questions, providing fluent and comprehensive answers&#10;that significantly surpass previous public chatbots in terms of security and&#10;usefulness. On one hand, people are curious about how ChatGPT is able to&#10;achieve such strength and how far it is from human experts. On the other hand,&#10;people are starting to worry about the potential negative impacts that large&#10;language models (LLMs) like ChatGPT could have on society, such as fake news,&#10;plagiarism, and social security issues. In this work, we collected tens of&#10;thousands of comparison responses from both human experts and ChatGPT, with&#10;questions ranging from open-domain, financial, medical, legal, and&#10;psychological areas. We call the collected dataset the Human ChatGPT Comparison&#10;Corpus (HC3). Based on the HC3 dataset, we study the characteristics of&#10;ChatGPT's responses, the differences and gaps from human experts, and future&#10;directions for LLMs. We conducted comprehensive human evaluations and&#10;linguistic analyses of ChatGPT-generated content compared with that of humans,&#10;where many interesting results are revealed. After that, we conduct extensive&#10;experiments on how to effectively detect whether a certain text is generated by&#10;ChatGPT or humans. We build three different detection systems, explore several&#10;key factors that influence their effectiveness, and evaluate them in different&#10;scenarios. The dataset, code, and models are all publicly available at&#10;https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13711" label="2305.13711">
        <attvalues>
          <attvalue for="0" value="LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain&#10;  Conversations with Large Language Models" />
          <attvalue for="1" value="  We propose LLM-Eval, a unified multi-dimensional automatic evaluation method&#10;for open-domain conversations with large language models (LLMs). Existing&#10;evaluation methods often rely on human annotations, ground-truth responses, or&#10;multiple LLM prompts, which can be expensive and time-consuming. To address&#10;these issues, we design a single prompt-based evaluation method that leverages&#10;a unified evaluation schema to cover multiple dimensions of conversation&#10;quality in a single model call. We extensively evaluate the performance of&#10;LLM-Eval on various benchmark datasets, demonstrating its effectiveness,&#10;efficiency, and adaptability compared to state-of-the-art evaluation methods.&#10;Our analysis also highlights the importance of choosing suitable LLMs and&#10;decoding strategies for accurate evaluation results. LLM-Eval offers a&#10;versatile and robust solution for evaluating open-domain conversation systems,&#10;streamlining the evaluation process and providing consistent performance across&#10;diverse scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.08061" label="2212.08061">
        <attvalues>
          <attvalue for="0" value="On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in&#10;  Zero-Shot Reasoning" />
          <attvalue for="1" value="  Generating a Chain of Thought (CoT) has been shown to consistently improve&#10;large language model (LLM) performance on a wide range of NLP tasks. However,&#10;prior work has mainly focused on logical reasoning tasks (e.g. arithmetic,&#10;commonsense QA); it remains unclear whether improvements hold for more diverse&#10;types of reasoning, especially in socially situated contexts. Concretely, we&#10;perform a controlled evaluation of zero-shot CoT across two socially sensitive&#10;domains: harmful questions and stereotype benchmarks. We find that zero-shot&#10;CoT reasoning in sensitive domains significantly increases a model's likelihood&#10;to produce harmful or undesirable output, with trends holding across different&#10;prompt formats and model variants. Furthermore, we show that harmful CoTs&#10;increase with model size, but decrease with improved instruction following. Our&#10;work suggests that zero-shot CoT should be used with caution on socially&#10;important tasks, especially when marginalized groups or sensitive topics are&#10;involved.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.00955" label="2305.00955">
        <attvalues>
          <attvalue for="0" value="Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural&#10;  Language Generation" />
          <attvalue for="1" value="  Many recent advances in natural language generation have been fueled by&#10;training large language models on internet-scale data. However, this paradigm&#10;can lead to models that generate toxic, inaccurate, and unhelpful content, and&#10;automatic evaluation metrics often fail to identify these behaviors. As models&#10;become more capable, human feedback is an invaluable signal for evaluating and&#10;improving models. This survey aims to provide an overview of the recent&#10;research that has leveraged human feedback to improve natural language&#10;generation. First, we introduce an encompassing formalization of feedback, and&#10;identify and organize existing research into a taxonomy following this&#10;formalization. Next, we discuss how feedback can be described by its format and&#10;objective, and cover the two approaches proposed to use feedback (either for&#10;training or decoding): directly using the feedback or training feedback models.&#10;We also discuss existing datasets for human-feedback data collection, and&#10;concerns surrounding feedback collection. Finally, we provide an overview of&#10;the nascent field of AI feedback, which exploits large language models to make&#10;judgments based on a set of principles and minimize the need for human&#10;intervention.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.05958" label="1804.05958">
        <attvalues>
          <attvalue for="0" value="Can Neural Machine Translation be Improved with User Feedback?" />
          <attvalue for="1" value="  We present the first real-world application of methods for improving neural&#10;machine translation (NMT) with human reinforcement, based on explicit and&#10;implicit user feedback collected on the eBay e-commerce platform. Previous work&#10;has been confined to simulation experiments, whereas in this paper we work with&#10;real logged feedback for offline bandit learning of NMT parameters. We conduct&#10;a thorough analysis of the available explicit user judgments---five-star&#10;ratings of translation quality---and show that they are not reliable enough to&#10;yield significant improvements in bandit learning. In contrast, we successfully&#10;utilize implicit task-based feedback collected in a cross-lingual search task&#10;to improve task-specific and machine translation quality metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12295" label="2305.12295">
        <attvalues>
          <attvalue for="0" value="Logic-LM: Empowering Large Language Models with Symbolic Solvers for&#10;  Faithful Logical Reasoning" />
          <attvalue for="1" value="  Large Language Models (LLMs) have shown human-like reasoning abilities but&#10;still struggle with complex logical problems. This paper introduces a novel&#10;framework, Logic-LM, which integrates LLMs with symbolic solvers to improve&#10;logical problem-solving. Our method first utilizes LLMs to translate a natural&#10;language problem into a symbolic formulation. Afterward, a deterministic&#10;symbolic solver performs inference on the formulated problem. We also introduce&#10;a self-refinement module, which utilizes the symbolic solver's error messages&#10;to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on&#10;five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO,&#10;LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant&#10;performance boost of 39.2% over using LLM alone with standard prompting and&#10;18.4% over LLM with chain-of-thought prompting. Our findings suggest that&#10;Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for&#10;faithful logical reasoning. Code and data are publicly available at&#10;https://github.com/teacherpeterpan/Logic-LLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.18700" label="2402.18700">
        <attvalues>
          <attvalue for="0" value="Learning to Compress Prompt in Natural Language Formats" />
          <attvalue for="1" value="  Large language models (LLMs) are great at processing multiple natural&#10;language processing tasks, but their abilities are constrained by inferior&#10;performance with long context, slow inference speed, and the high cost of&#10;computing the results. Deploying LLMs with precise and informative context&#10;helps users process large-scale datasets more effectively and cost-efficiently.&#10;Existing works rely on compressing long prompt contexts into soft prompts.&#10;However, soft prompt compression encounters limitations in transferability&#10;across different LLMs, especially API-based LLMs. To this end, this work aims&#10;to compress lengthy prompts in the form of natural language with LLM&#10;transferability. This poses two challenges: (i) Natural Language (NL) prompts&#10;are incompatible with back-propagation, and (ii) NL prompts lack flexibility in&#10;imposing length constraints. In this work, we propose a Natural Language Prompt&#10;Encapsulation (Nano-Capsulator) framework compressing original prompts into NL&#10;formatted Capsule Prompt while maintaining the prompt utility and&#10;transferability. Specifically, to tackle the first challenge, the&#10;Nano-Capsulator is optimized by a reward function that interacts with the&#10;proposed semantics preserving loss. To address the second question, the&#10;Nano-Capsulator is optimized by a reward function featuring length constraints.&#10;Experimental results demonstrate that the Capsule Prompt can reduce 81.4% of&#10;the original length, decrease inference latency up to 4.5x, and save 80.1% of&#10;budget overheads while providing transferability across diverse LLMs and&#10;different datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" " />
          <attvalue for="4" value="Computer Science, Large Language Models, Information Technology, Prompt Compression Techniques, Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.13512" label="2305.13512">
        <attvalues>
          <attvalue for="0" value="Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken&#10;  Language Understanding" />
          <attvalue for="1" value="  Recently, large pretrained language models have demonstrated strong language&#10;understanding capabilities. This is particularly reflected in their zero-shot&#10;and in-context learning abilities on downstream tasks through prompting. To&#10;assess their impact on spoken language understanding (SLU), we evaluate several&#10;such models like ChatGPT and OPT of different sizes on multiple benchmarks. We&#10;verify the emergent ability unique to the largest models as they can reach&#10;intent classification accuracy close to that of supervised models with zero or&#10;few shots on various languages given oracle transcripts. By contrast, the&#10;results for smaller models fitting a single GPU fall far behind. We note that&#10;the error cases often arise from the annotation scheme of the dataset;&#10;responses from ChatGPT are still reasonable. We show, however, that the model&#10;is worse at slot filling, and its performance is sensitive to ASR errors,&#10;suggesting serious challenges for the application of those textual models on&#10;SLU.&#10;" />
          <attvalue for="2" value="&#10;&#10;Gigantic pretrained language models like GPT3 with 175B parameters trained on 45TB texts have demonstrated surprisingly strong performance on various downstream language tasks with little or no data \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}. Since then, GPT3 has evolved into GPT3.5 through pretraining on code as in Codex \cite{DBLP:journals/corr/abs-2107-03374} which powers GitHub Copilot, as well as through instruction fine-tuning that aligns the model's responses given instructions with human expectations using reinforcement learning, known as InstructGPT \cite{DBLP:journals/corr/abs-2203-02155}. When further combined with fine-tuning on dialogues in a similar way, the resulting model, ChatGPT, has gained great popularity since its release in late 2022, displaying highly human-like language understanding and generation capabilities \cite{kung2023performance,DBLP:journals/corr/abs-2302-04023,DBLP:journals/corr/abs-2302-06476}, and has become the core of a number of AI-powered applications. Combined with the ability to utilize tools with external APIs as in Toolformer \cite{DBLP:journals/corr/abs-2302-04761} as well as conducting web search as in WebGPT \cite{DBLP:journals/corr/abs-2112-09332} and New Bing, a competent and versatile AI assistant has taken shape. Therefore, a question arises: is the model capable of conducting spoken language understanding (SLU) tasks like current voice assistants?&#10;&#10;Current SLU approaches are substantially different from how we use those GPT3-based models. Traditionally, SLU is carried out using a cascaded pipeline, which includes an automatic speech recognition (ASR) module taking audio as inputs, and a natural language understanding (NLU) module working on ASR transcripts, hypotheses, or lattice to predict labels for tasks like intent classification (IC) and slot filling (SF) \cite{DBLP:conf/asru/Mori07,DBLP:conf/ijcai/QinXC021}. Recently, end-to-end approaches that directly predict labels from speech \cite{DBLP:conf/icassp/SerdyukWFKLB18,DBLP:conf/slt/HaghaniNBCGMPQW18,DBLP:conf/interspeech/SaxonCMM21} become more popular, and pretrained language and speech models are also introduced into SLU \cite{DBLP:journals/corr/abs-2111-02735,DBLP:conf/icassp/AroraDDCUPZKGYV22,DBLP:conf/icassp/SeoKL22}. Additionally, there are works focused on low-resource or few-shot textual IC/SF \cite{DBLP:conf/emnlp/YazdaniH15,DBLP:conf/interspeech/FerreiraJL15,DBLP:conf/acl/HouCLZLLL20,DBLP:conf/emnlp/WuSJ21,DBLP:conf/interspeech/PengZZG21}.&#10;&#10;However, those methods are based on the paradigm of supervised training or fine-tuning with a set of possibly large-scale training data. In contrast, considering the difficulty of fine-tuning the whole GPT3 model, recent NLP research highlights a different scheme, namely prompting \cite{DBLP:journals/csur/LiuYFJHN23}: given a fixed textual description of the task known as a prompt without any training data, the language model may correctly carry out the task. Furthermore, the in-context learning approach adds a few paired examples in the textual prompt to further direct the model towards the desired outputs. Such methods are different from traditional zero or few-shot learning in which the model parameters are fixed. It appears to be clumsy and may perform worse than fine-tuned smaller models like T5-11B at the beginning \cite{DBLP:conf/emnlp/LesterAC21}. Additionally, larger models were believed to be unscalable to reach the desired performance given the costs \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2110-14168}, However, recent explorations reveal the emergent abilities of larger models like GPT3-175B and PaLM 540B \cite{DBLP:journals/corr/abs-2204-02311}: prompting shows low or even close-to-random performance on multiple tasks until a certain scale of the model where a breakthrough emerges \cite{DBLP:journals/corr/abs-2206-04615,DBLP:journals/corr/abs-2206-07682}. This breakthrough enables chain-of-thought prompting to surpass the smaller models fine-tuned on rich data \cite{weichain,DBLP:journals/corr/abs-2210-00720}, allows reasoning using internal knowledge with results comparable to external knowledge retrievers \cite{DBLP:journals/corr/abs-2209-10063}, and leads to better robustness and generalization \cite{DBLP:journals/corr/abs-2210-00720,DBLP:journals/corr/abs-2210-09150}.&#10;&#10;There have been several works on SLU employing prompts, such as fine-tuning pretrained models like T5 aided by prompts \cite{DBLP:conf/coling/WuWZCZ22,DBLP:journals/corr/abs-2210-03337}, fine-tuning embeddings prepended to the inputs known as continuous prompts \cite{DBLP:conf/interspeech/ChangT0L22,DBLP:journals/corr/abs-2303-00733}, and end-to-end SLU by in-context learning on GPT2 with a fine-tuned audio encoder \cite{DBLP:conf/interspeech/GaoNQZCH22}. They are nevertheless distinct from the current prompting and in-context learning scheme, and have not approached the regime of emergent abilities. Hence the potential and limitations of this new type of method on SLU remain unexplored. Therefore we endeavor to undertake it by designing prompts and evaluating these models on multiple SLU benchmarks, including SLURP \cite{DBLP:conf/emnlp/BastianelliVSR20} and the multilingual MINDS-14 \cite{DBLP:conf/emnlp/GerzSKMLSMWV21}. Since these models take textual inputs, beside oracle transcripts, we also use ASR transcripts from Whisper \cite{DBLP:journals/corr/abs-2212-04356}, which embodies a pipeline upon completely off-the-shelf pretrained models. Furthermore, we compare smaller models that can easily run on a common GPU, namely GPT2 \cite{radford2019language} and several OPT models \cite{DBLP:journals/corr/abs-2205-01068}.&#10;&#10;As a result, we discover that the largest GPT3.5 and ChatGPT models achieve high performance on intent classification under zero-shot or few-shot in-context scenarios that are close or even better than models fine-tuned on the whole dataset, when given the oracle transcripts. This is unique to those large models as the smaller GPT2, OPT and GPT3.5 Curie models have much lower performance and are entirely ineffective under zero-shot cases. Even in the cases where the predictions differ from the labels, the predictions are mostly reasonable, often due to the ambiguity of the sentence. This raises the question that the tasks to predict intermediate IC/SF labels might not sufficiently reveal the potential of the model. However, for the slot filling task with a more complicated task definition, the performance is much worse. Additionally, the accuracy drops significantly when using ASR transcripts. We show that the models have limited awareness of word pronunciations and possible ASR errors, which poses challenges for directly deploying those models for real-world SLU. To facilitate reproduction, relevant resources and prompts are available at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Spoken Language Understanding, Artificial Intelligence, Natural Language Processing, Artificial Intelligence Limitations" />
        </attvalues>
      </node>
      <node id="2103.03095" label="2103.03095">
        <attvalues>
          <attvalue for="0" value="A Survey on Spoken Language Understanding: Recent Advances and New&#10;  Frontiers" />
          <attvalue for="1" value="  Spoken Language Understanding (SLU) aims to extract the semantics frame of&#10;user queries, which is a core component in a task-oriented dialog system. With&#10;the burst of deep neural networks and the evolution of pre-trained language&#10;models, the research of SLU has obtained significant breakthroughs. However,&#10;there remains a lack of a comprehensive survey summarizing existing approaches&#10;and recent trends, which motivated the work presented in this article. In this&#10;paper, we survey recent advances and new frontiers in SLU. Specifically, we&#10;give a thorough review of this research field, covering different aspects&#10;including (1) new taxonomy: we provide a new perspective for SLU filed,&#10;including single model vs. joint model, implicit joint modeling vs. explicit&#10;joint modeling in joint model, non pre-trained paradigm vs. pre-trained&#10;paradigm;(2) new frontiers: some emerging areas in complex SLU as well as the&#10;corresponding challenges; (3) abundant open-source resources: to help the&#10;community, we have collected, organized the related papers, baseline projects&#10;and leaderboard on a public website where SLU researchers could directly access&#10;to the recent progress. We hope that this survey can shed a light on future&#10;research in SLU field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.09190" label="1809.09190">
        <attvalues>
          <attvalue for="0" value="From Audio to Semantics: Approaches to end-to-end spoken language&#10;  understanding" />
          <attvalue for="1" value="  Conventional spoken language understanding systems consist of two main&#10;components: an automatic speech recognition module that converts audio to a&#10;transcript, and a natural language understanding module that transforms the&#10;resulting text (or top N hypotheses) into a set of domains, intents, and&#10;arguments. These modules are typically optimized independently. In this paper,&#10;we formulate audio to semantic understanding as a sequence-to-sequence problem&#10;[1]. We propose and compare various encoder-decoder based approaches that&#10;optimize both modules jointly, in an end-to-end manner. Evaluations on a&#10;real-world task show that 1) having an intermediate text representation is&#10;crucial for the quality of the predicted semantics, especially the intent&#10;arguments and 2) jointly optimizing the full system improves overall accuracy&#10;of prediction. Compared to independently trained models, our best jointly&#10;trained model achieves similar domain and intent prediction F1 scores, but&#10;improves argument word error rate by 18% relative.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.09009" label="2106.09009">
        <attvalues>
          <attvalue for="0" value="End-to-End Spoken Language Understanding for Generalized Voice&#10;  Assistants" />
          <attvalue for="1" value="  End-to-end (E2E) spoken language understanding (SLU) systems predict&#10;utterance semantics directly from speech using a single model. Previous work in&#10;this area has focused on targeted tasks in fixed domains, where the output&#10;semantic structure is assumed a priori and the input speech is of limited&#10;complexity. In this work we present our approach to developing an E2E model for&#10;generalized SLU in commercial voice assistants (VAs). We propose a fully&#10;differentiable, transformer-based, hierarchical system that can be pretrained&#10;at both the ASR and NLU levels. This is then fine-tuned on both transcription&#10;and semantic classification losses to handle a diverse set of intent and&#10;argument combinations. This leads to an SLU system that achieves significant&#10;improvements over baselines on a complex internal generalized VA dataset with a&#10;43% improvement in accuracy, while still meeting the 99% accuracy benchmark on&#10;the popular Fluent Speech Commands dataset. We further evaluate our model on a&#10;hard test set, exclusively containing slot arguments unseen in training, and&#10;demonstrate a nearly 20% improvement, showing the efficacy of our approach in&#10;truly demanding VA scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.14706" label="2111.14706">
        <attvalues>
          <attvalue for="0" value="ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet" />
          <attvalue for="1" value="  As Automatic Speech Processing (ASR) systems are getting better, there is an&#10;increasing interest of using the ASR output to do downstream Natural Language&#10;Processing (NLP) tasks. However, there are few open source toolkits that can be&#10;used to generate reproducible results on different Spoken Language&#10;Understanding (SLU) benchmarks. Hence, there is a need to build an open source&#10;standard that can be used to have a faster start into SLU research. We present&#10;ESPnet-SLU, which is designed for quick development of spoken language&#10;understanding in a single framework. ESPnet-SLU is a project inside end-to-end&#10;speech processing toolkit, ESPnet, which is a widely used open-source standard&#10;for various speech processing tasks like ASR, Text to Speech (TTS) and Speech&#10;Translation (ST). We enhance the toolkit to provide implementations for various&#10;SLU benchmarks that enable researchers to seamlessly mix-and-match different&#10;ASR and NLU models. We also provide pretrained models with intensively tuned&#10;hyper-parameters that can match or even outperform the current state-of-the-art&#10;performances. The toolkit is publicly available at&#10;https://github.com/espnet/espnet.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07253" label="2104.07253">
        <attvalues>
          <attvalue for="0" value="Integration of Pre-trained Networks with Continuous Token Interface for&#10;  End-to-End Spoken Language Understanding" />
          <attvalue for="1" value="  Most End-to-End (E2E) SLU networks leverage the pre-trained ASR networks but&#10;still lack the capability to understand the semantics of utterances, crucial&#10;for the SLU task. To solve this, recently proposed studies use pre-trained NLU&#10;networks. However, it is not trivial to fully utilize both pre-trained&#10;networks; many solutions were proposed, such as Knowledge Distillation,&#10;cross-modal shared embedding, and network integration with Interface. We&#10;propose a simple and robust integration method for the E2E SLU network with&#10;novel Interface, Continuous Token Interface (CTI), the junctional&#10;representation of the ASR and NLU networks when both networks are pre-trained&#10;with the same vocabulary. Because the only difference is the noise level, we&#10;directly feed the ASR network's output to the NLU network. Thus, we can train&#10;our SLU network in an E2E manner without additional modules, such as&#10;Gumbel-Softmax. We evaluate our model using SLURP, a challenging SLU dataset&#10;and achieve state-of-the-art scores on both intent classification and slot&#10;filling tasks. We also verify the NLU network, pre-trained with Masked Language&#10;Model, can utilize a noisy textual representation of CTI. Moreover, we show our&#10;model can be trained with multi-task learning from heterogeneous data even&#10;after integration with CTI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.05702" label="2006.05702">
        <attvalues>
          <attvalue for="0" value="Few-shot Slot Tagging with Collapsed Dependency Transfer and&#10;  Label-enhanced Task-adaptive Projection Network" />
          <attvalue for="1" value="  In this paper, we explore the slot tagging with only a few labeled support&#10;sentences (a.k.a. few-shot). Few-shot slot tagging faces a unique challenge&#10;compared to the other few-shot classification problems as it calls for modeling&#10;the dependencies between labels. But it is hard to apply previously learned&#10;label dependencies to an unseen domain, due to the discrepancy of label sets.&#10;To tackle this, we introduce a collapsed dependency transfer mechanism into the&#10;conditional random field (CRF) to transfer abstract label dependency patterns&#10;as transition scores. In the few-shot setting, the emission score of CRF can be&#10;calculated as a word's similarity to the representation of each label. To&#10;calculate such similarity, we propose a Label-enhanced Task-Adaptive Projection&#10;Network (L-TapNet) based on the state-of-the-art few-shot classification model&#10;-- TapNet, by leveraging label name semantics in representing labels.&#10;Experimental results show that our model significantly outperforms the&#10;strongest few-shot learning baseline by 14.64 F1 scores in the one-shot&#10;setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.13952" label="2004.13952">
        <attvalues>
          <attvalue for="0" value="Data Augmentation for Spoken Language Understanding via Pretrained&#10;  Language Models" />
          <attvalue for="1" value="  The training of spoken language understanding (SLU) models often faces the&#10;problem of data scarcity. In this paper, we put forward a data augmentation&#10;method using pretrained language models to boost the variability and accuracy&#10;of generated utterances. Furthermore, we investigate and propose solutions to&#10;two previously overlooked semi-supervised learning scenarios of data scarcity&#10;in SLU: i) Rich-in-Ontology: ontology information with numerous valid dialogue&#10;acts is given; ii) Rich-in-Utterance: a large number of unlabelled utterances&#10;are available. Empirical results show that our method can produce synthetic&#10;training data that boosts the performance of language understanding models in&#10;various scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.03337" label="2210.03337">
        <attvalues>
          <attvalue for="0" value="A Unified Framework for Multi-intent Spoken Language Understanding with&#10;  prompting" />
          <attvalue for="1" value="  Multi-intent Spoken Language Understanding has great potential for widespread&#10;implementation. Jointly modeling Intent Detection and Slot Filling in it&#10;provides a channel to exploit the correlation between intents and slots.&#10;However, current approaches are apt to formulate these two sub-tasks&#10;differently, which leads to two issues: 1) It hinders models from effective&#10;extraction of shared features. 2) Pretty complicated structures are involved to&#10;enhance expression ability while causing damage to the interpretability of&#10;frameworks. In this work, we describe a Prompt-based Spoken Language&#10;Understanding (PromptSLU) framework, to intuitively unify two sub-tasks into&#10;the same form by offering a common pre-trained Seq2Seq model. In detail, ID and&#10;SF are completed by concisely filling the utterance into task-specific prompt&#10;templates as input, and sharing output formats of key-value pairs sequence.&#10;Furthermore, variable intents are predicted first, then naturally embedded into&#10;prompts to guide slot-value pairs inference from a semantic perspective.&#10;Finally, we are inspired by prevalent multi-task learning to introduce an&#10;auxiliary sub-task, which helps to learn relationships among provided labels.&#10;Experiment results show that our framework outperforms several state-of-the-art&#10;baselines on two public datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15863" label="2203.15863">
        <attvalues>
          <attvalue for="0" value="WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen&#10;  Language Models" />
          <attvalue for="1" value="  Large-scale auto-regressive language models pretrained on massive text have&#10;demonstrated their impressive ability to perform new natural language tasks&#10;with only a few text examples, without the need for fine-tuning. Recent studies&#10;further show that such a few-shot learning ability can be extended to the&#10;text-image setting by training an encoder to encode the images into embeddings&#10;functioning like the text embeddings of the language model. Interested in&#10;exploring the possibility of transferring the few-shot learning ability to the&#10;audio-text setting, we propose a novel speech understanding framework,&#10;WavPrompt, where we finetune a wav2vec model to generate a sequence of audio&#10;embeddings understood by the language model. We show that WavPrompt is a&#10;few-shot learner that can perform speech understanding tasks better than a&#10;naive text baseline. We conduct detailed ablation studies on different&#10;components and hyperparameters to empirically identify the best model&#10;configuration. In addition, we conduct a non-speech understanding experiment to&#10;show WavPrompt can extract more information than just the transcriptions. Code&#10;is available at https://github.com/Hertin/WavPrompt&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08524" label="2104.08524">
        <attvalues>
          <attvalue for="0" value="Multilingual and Cross-Lingual Intent Detection from Spoken Data" />
          <attvalue for="1" value="  We present a systematic study on multilingual and cross-lingual intent&#10;detection from spoken data. The study leverages a new resource put forth in&#10;this work, termed MInDS-14, a first training and evaluation resource for the&#10;intent detection task with spoken data. It covers 14 intents extracted from a&#10;commercial system in the e-banking domain, associated with spoken examples in&#10;14 diverse language varieties. Our key results indicate that combining machine&#10;translation models with state-of-the-art multilingual sentence encoders (e.g.,&#10;LaBSE) can yield strong intent detectors in the majority of target languages&#10;covered in MInDS-14, and offer comparative analyses across different axes:&#10;e.g., zero-shot versus few-shot learning, translation direction, and impact of&#10;speech recognition. We see this work as an important step towards more&#10;inclusive development and evaluation of multilingual intent detectors from&#10;spoken data, in a much wider spectrum of languages compared to prior work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.05461" label="2305.05461">
        <attvalues>
          <attvalue for="0" value="What is the best recipe for character-level encoder-only modelling?" />
          <attvalue for="1" value="  This paper aims to benchmark recent progress in language understanding models&#10;that output contextualised representations at the character level. Many such&#10;modelling architectures and methods to train those architectures have been&#10;proposed, but it is currently unclear what the relative contributions of the&#10;architecture vs. the pretraining objective are to final model performance. We&#10;explore the design space of such models, comparing architectural innovations&#10;and a variety of different pretraining objectives on a suite of evaluation&#10;tasks with a fixed training procedure in order to find the currently optimal&#10;way to build and train character-level BERT-like models. We find that our best&#10;performing character-level model exceeds the performance of a token-based model&#10;trained with the same settings on the same data, suggesting that&#10;character-level models are ready for more widespread adoption. Unfortunately,&#10;the best method to train character-level models still relies on a subword-level&#10;tokeniser during pretraining, and final model performance is highly dependent&#10;on tokeniser quality. We believe our results demonstrate the readiness of&#10;character-level models for multilingual language representation, and encourage&#10;NLP practitioners to try them as drop-in replacements for token-based models.&#10;" />
          <attvalue for="2" value="&#10;&#10;The first stage of almost all NLP modelling pipelines is to convert input text strings into a sequence of symbols that the model can ingest. This step, called tokenisation, can be highly non-trivial and introduces significant theoretical and practical complexities to both training and using these models. One particular issue for massively multilingual models is that many languages have to compete for space given a fixed vocabulary size, which limits the effective vocabulary size per language: as an illustration, the WordPiece tokeniser that multilingual BERT uses tokenises `hello' as two tokens: `hell' and `\#\#o'. We are therefore interested in alternative approaches which use lightweight tokenisation schemes (in particular character-level segmentation) coupled with powerful neural-network based composition functions to build language models (see Section \ref{sec:benefits} for a discussion of the benefits of character-level modelling). In this paper, we aim to determine the best way to build such models, focussing on models which output vector representations for each input character.&#10;&#10;However, as the field of pretrained character-level modelling is relatively new, comparisons are complicated by the fact that recently proposed methods use different model architectures, pretrain on different data using different training objectives, and evaluate on different downstream tasks. With so many variables changing simultaneously, it is difficult to disentangle the effect of each individual choice in the modelling pipeline, and therefore decide on an overall best model configuration. To answer this question, we tested many model architectures and pretraining objectives from recent literature on a unified set of evaluation tasks, with the same training procedure. We identify one particular configuration that shows the best performance across all of our downstream evaluation tasks, namely a combination of the Charformer downsampling model \cite{Tay:2021}, and CANINE upsampling model and pretraining procedure \cite{Clark:22}. We dub this configuration BORT, for Bidirectional Orthographic Representation Technique. This model even outperforms a BERT baseline on all tasks we consider, while being moderately slower to pretrain (\S \ref{sec:big_comparison}).&#10;&#10;One sticky point we discovered is that the best modelling configuration we found above relies crucially on a tokeniser during pretraining. We investigate alternative objectives that do not require a tokeniser, and find that these objectives result in worse-performing models. In addition, we also investigate the impact of the tokeniser used to pretrain the model, and find that the quality of the tokeniser (measured by vocabulary size) has a big impact on the final model downstream task performance, even though the tokeniser is not used at all during evaluation. This results in the unfortunate situation that users of such models have a hidden dependency on the tokeniser used to train the model; hence, users may be using models out of domain without any explicit feedback (such as worse tokeniser compression rates), causing difficult-to-detect performance regressions.&#10;&#10;Taken together, we believe our results show that character-level representation models are ready to supplant subword-level models as a default choice for converting text into features. However, these models still require extensive supervision from tokenisers, and we believe that the next frontier of research in character-level modelling is finding ways to once and for all eliminate tokenisation as a key step in the NLP pipeline.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Understanding Models, Linguistics, Cognitive Science, Multilingual Language Representation, Model Architecture Design, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2106.12672" label="2106.12672">
        <attvalues>
          <attvalue for="0" value="Charformer: Fast Character Transformers via Gradient-based Subword&#10;  Tokenization" />
          <attvalue for="1" value="  State-of-the-art models in natural language processing rely on separate rigid&#10;subword tokenization algorithms, which limit their generalization ability and&#10;adaptation to new settings. In this paper, we propose a new model inductive&#10;bias that learns a subword tokenization end-to-end as part of the model. To&#10;this end, we introduce a soft gradient-based subword tokenization module (GBST)&#10;that automatically learns latent subword representations from characters in a&#10;data-driven fashion. Concretely, GBST enumerates candidate subword blocks and&#10;learns to score them in a position-wise fashion using a block scoring network.&#10;We additionally introduce Charformer, a deep Transformer model that integrates&#10;GBST and operates on the byte level. Via extensive experiments on English GLUE,&#10;multilingual, and noisy text datasets, we show that Charformer outperforms a&#10;series of competitive byte-level baselines while generally performing on par&#10;and sometimes outperforming subword-based models. Additionally, Charformer is&#10;fast, improving the speed of both vanilla byte-level and subword-level&#10;Transformers by 28%-100% while maintaining competitive quality. We believe this&#10;work paves the way for highly performant token-free models that are trained&#10;completely end-to-end.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.14782" label="2405.14782">
        <attvalues>
          <attvalue for="0" value="Lessons from the Trenches on Reproducible Evaluation of Language Models" />
          <attvalue for="1" value="  Effective evaluation of language models remains an open challenge in NLP.&#10;Researchers and engineers face methodological issues such as the sensitivity of&#10;models to evaluation setup, difficulty of proper comparisons across methods,&#10;and the lack of reproducibility and transparency. In this paper we draw on&#10;three years of experience in evaluating large language models to provide&#10;guidance and lessons for researchers. First, we provide an overview of common&#10;challenges faced in language model evaluation. Second, we delineate best&#10;practices for addressing or lessening the impact of these challenges on&#10;research. Third, we present the Language Model Evaluation Harness (lm-eval): an&#10;open source library for independent, reproducible, and extensible evaluation of&#10;language models that seeks to address these issues. We describe the features of&#10;the library as well as case studies in which the library has been used to&#10;alleviate these methodological concerns.&#10;" />
          <attvalue for="2" value="&#10;&#10;Evaluation on shared benchmark tasks is a crucial tool used to track and communicate progress in the machine learning and language modeling communities \cite{ruder2021benchmarking}. Benchmarks are used to track progress toward shared community goals and to demonstrate the improvements of newly proposed methods over prior baselines. Evaluation practices thus play a crucial role in the direction of the field: inconsistencies or biases in evaluation practices can lead to skewed performance comparisons, which may influence the direction of future research and the adoption of new methods by the community \cite{dehghani2021benchmark} or lead to adverse effects from deploying suboptimal or harmful models \cite{bender-friedman-2018-data} on tasks for which they are ill-suited \cite{Raji_2022}.&#10;&#10;Unfortunately, transparent and reproducible evaluation of large language models is very challenging. In our research we have frequently struggled to reproduce the results reported in various papers as well as carry out new evaluations ourselves. To address this problem we built the Language Model Evaluation Harness \cite{eval-harness}, a flexible evaluation library that serves as research infrastructure for evaluation. Our goal with lm-eval~is to enable researchers to run any benchmark on any model as easily as possible, while also making it easy for creators of new model inference libraries or evaluation benchmarks to connect their work to the broader ecosystem.&#10;&#10;Over the past three years, the design of~lm-eval~has evolved as the needs of the open source community and our understanding of best practices for language model evaluation have evolved. In this paper we detail lessons learned that have been especially beneficial to obtaining useful and rigorous findings. We highlight several commonly-faced challenges in evaluating language models, including the difficulty of assessing the correctness of natural language responses, challenges in benchmark design, and the dependence upon implementation details that are often obscured or unreported (Section~\ref{sec:challenges}). We then discuss best practices we've identified to improve how to communicate results and improve evaluation rigor in the language modeling community, despite these challenges (Section~\ref{sec:best-practices}). Finally, we detail how we have used our learnings to inform the design of lm-eval~(Section~\ref{sec:eval-harness}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Best Practices Development, Computer Science, Methodological Challenges, Linguistics, Language Model Evaluation, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2107.07002" label="2107.07002">
        <attvalues>
          <attvalue for="0" value="The Benchmark Lottery" />
          <attvalue for="1" value="  The world of empirical machine learning (ML) strongly relies on benchmarks in&#10;order to determine the relative effectiveness of different algorithms and&#10;methods. This paper proposes the notion of &quot;a benchmark lottery&quot; that describes&#10;the overall fragility of the ML benchmarking process. The benchmark lottery&#10;postulates that many factors, other than fundamental algorithmic superiority,&#10;may lead to a method being perceived as superior. On multiple benchmark setups&#10;that are prevalent in the ML community, we show that the relative performance&#10;of algorithms may be altered significantly simply by choosing different&#10;benchmark tasks, highlighting the fragility of the current paradigms and&#10;potential fallacious interpretation derived from benchmarking ML methods. Given&#10;that every benchmark makes a statement about what it perceives to be important,&#10;we argue that this might lead to biased progress in the community. We discuss&#10;the implications of the observed phenomena and provide recommendations on&#10;mitigating them using multiple machine learning domains and communities as use&#10;cases, including natural language processing, computer vision, information&#10;retrieval, recommender systems, and reinforcement learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.09511" label="2206.09511">
        <attvalues>
          <attvalue for="0" value="The Fallacy of AI Functionality" />
          <attvalue for="1" value="  Deployed AI systems often do not work. They can be constructed haphazardly,&#10;deployed indiscriminately, and promoted deceptively. However, despite this&#10;reality, scholars, the press, and policymakers pay too little attention to&#10;functionality. This leads to technical and policy solutions focused on&#10;&quot;ethical&quot; or value-aligned deployments, often skipping over the prior question&#10;of whether a given system functions, or provides any benefits at all. To&#10;describe the harms of various types of functionality failures, we analyze a set&#10;of case studies to create a taxonomy of known AI functionality issues. We then&#10;point to policy and organizational responses that are often overlooked and&#10;become more readily available once functionality is drawn into focus. We argue&#10;that functionality is a meaningful AI policy challenge, operating as a&#10;necessary first step towards protecting affected communities from algorithmic&#10;harm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.09350" label="2205.09350">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Inflection as a Data Augmentation Method for Parsing" />
          <attvalue for="1" value="  We propose a morphology-based method for low-resource (LR) dependency&#10;parsing. We train a morphological inflector for target LR languages, and apply&#10;it to related rich-resource (RR) treebanks to create cross-lingual&#10;(x-inflected) treebanks that resemble the target LR language. We use such&#10;inflected treebanks to train parsers in zero- (training on x-inflected&#10;treebanks) and few-shot (training on x-inflected and target language treebanks)&#10;setups. The results show that the method sometimes improves the baselines, but&#10;not consistently.&#10;" />
          <attvalue for="2" value="&#10;&#10;Dependency parsers \cite{dozat-etal-2017-stanfords,ma-etal-2018-stack,strzyz-etal-2019-viable} already achieve accurate results for certain setups \cite{berzak-etal-2016-anchoring}. Yet, they require large amounts of data to work, which hurts low-resource (LR) scenarios. In this line, authors have studied how to overcome this problem.&#10;&#10;On data augmentation, recent approaches have replaced subtrees of sentences to generate new ones \cite{vania-etal-2019-systematic,dehouck-gomez-rodriguez-2020-data}. On cross-lingual learning, authors have explored delexicalized approaches from rich-resource (RR) treebanks. \cite{mcdonald-etal-2011-multi,falenska-cetinoglu-2017-lexicalized}. \cite{wang-eisner-2018-synthetic} permuted constituents of distant treebanks to generate synthetic ones that resembled the target language. \cite{vilares-etal-2016-one,ammar-etal-2016-many} merged treebanks to train multilingual parsers that sometimes could outperform the equivalent monolingual version, which has applications for less-resourced parsing. In the context of multilingual representations, \cite{mulcaire-etal-2019-low} trained a zero-shot parser on top of a polyglot language model, relying on merged RR treebanks too. &#10;&#10;In other matters, morphological inflection \cite{cotterell2016sigmorphon,pimentel2021sigmorphon} generates words from lemmas and morphological feats (e.g. look $arrow$ looking). Also, it is known that morphology helps parsing and that morphological complexity relates to the magnitude of the improvements \cite{dehouck-denis-2018-framework}. Yet, as far as we know, there is no work on cross-lingual morphological inflection as a data augmentation method for parsing. Here, we propose a technique that lies in the intersectionbetween data augmentation, cross-lingual learning, and morphological inflection.&#10;&#10;Contribution We introduce a method that uses cross-lingual morphological inflection to generate `synthetic creole' treebanks, which we call x-inflected treebanks. To do so, we require a source language treebank from a closely-related language (for which lemmas and morphological feats are available), and a morphological inflection system trained for the target language. This way, we expect to generate x-inflected treebanks that should resemble to a certain extent the target language (see Figure \ref{fig:morph_process}). The goal is to improve the parser's performance for languages for which little or no annotated data are available, but for which we can train an accurate morphological inflection system that can be later applied to a related RR treebank and resemble the target language. The code is available at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Morphological Inflection, Cross-Lingual Learning, Natural Language Processing, Dependency Parsing" />
        </attvalues>
      </node>
      <node id="1605.04481" label="1605.04481">
        <attvalues>
          <attvalue for="0" value="Anchoring and Agreement in Syntactic Annotations" />
          <attvalue for="1" value="  We present a study on two key characteristics of human syntactic annotations:&#10;anchoring and agreement. Anchoring is a well known cognitive bias in human&#10;decision making, where judgments are drawn towards pre-existing values. We&#10;study the influence of anchoring on a standard approach to creation of&#10;syntactic resources where syntactic annotations are obtained via human editing&#10;of tagger and parser output. Our experiments demonstrate a clear anchoring&#10;effect and reveal unwanted consequences, including overestimation of parsing&#10;performance and lower quality of annotations in comparison with human-based&#10;annotations. Using sentences from the Penn Treebank WSJ, we also report&#10;systematically obtained inter-annotator agreement estimates for English&#10;dependency parsing. Our agreement results control for parser bias, and are&#10;consequential in that they are on par with state of the art parsing performance&#10;for English newswire. We discuss the impact of our findings on strategies for&#10;future annotation efforts and parser evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1507.08449" label="1507.08449">
        <attvalues>
          <attvalue for="0" value="One model, two languages: training bilingual parsers with harmonized&#10;  treebanks" />
          <attvalue for="1" value="  We introduce an approach to train lexicalized parsers using bilingual corpora&#10;obtained by merging harmonized treebanks of different languages, producing&#10;parsers that can analyze sentences in either of the learned languages, or even&#10;sentences that mix both. We test the approach on the Universal Dependency&#10;Treebanks, training with MaltParser and MaltOptimizer. The results show that&#10;these bilingual parsers are more than competitive, as most combinations not&#10;only preserve accuracy, but some even achieve significant improvements over the&#10;corresponding monolingual parsers. Preliminary experiments also show the&#10;approach to be promising on texts with code-switching and when more languages&#10;are added.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.10579" label="1909.10579">
        <attvalues>
          <attvalue for="0" value="Using Priming to Uncover the Organization of Syntactic Representations&#10;  in Neural Language Models" />
          <attvalue for="1" value="  Neural language models (LMs) perform well on tasks that require sensitivity&#10;to syntactic structure. Drawing on the syntactic priming paradigm from&#10;psycholinguistics, we propose a novel technique to analyze the representations&#10;that enable such success. By establishing a gradient similarity metric between&#10;structures, this technique allows us to reconstruct the organization of the&#10;LMs' syntactic representational space. We use this technique to demonstrate&#10;that LSTM LMs' representations of different types of sentences with relative&#10;clauses are organized hierarchically in a linguistically interpretable manner,&#10;suggesting that the LMs track abstract properties of the sentence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2302.14055" label="2302.14055">
        <attvalues>
          <attvalue for="0" value="Phone and speaker spatial organization in self-supervised speech&#10;  representations" />
          <attvalue for="1" value="  Self-supervised representations of speech are currently being widely used for&#10;a large number of applications. Recently, some efforts have been made in trying&#10;to analyze the type of information present in each of these representations.&#10;Most such work uses downstream models to test whether the representations can&#10;be successfully used for a specific task. The downstream models, though,&#10;typically perform nonlinear operations on the representation extracting&#10;information that may not have been readily available in the original&#10;representation. In this work, we analyze the spatial organization of phone and&#10;speaker information in several state-of-the-art speech representations using&#10;methods that do not require a downstream model. We measure how different layers&#10;encode basic acoustic parameters such as formants and pitch using&#10;representation similarity analysis. Further, we study the extent to which each&#10;representation clusters the speech samples by phone or speaker classes using&#10;non-parametric statistical testing. Our results indicate that models represent&#10;these speech attributes differently depending on the target task used during&#10;pretraining.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, many new deep-learning-based speech representations have been proposed and used for a variety of applications. Most of these models are trained with self-supervised approaches~\cite{liu2022audio}, making it hard to understand which information is being preserved in the resulting representations. Consequently, several recent works have focused on analyzing how the properties of a speech signal are encoded in these representations \cite{ma2021probing,de2022probing}. These works rely on different techniques for probing speech representations. Some of them are based on evaluating downstream tasks involving training a machine learning model that takes the representations from the pre-trained model as input. The output labels are characteristics derived from the speech signal, such as phone class \cite{ma2021probing}, pronunciation quality, fluency, or other speech properties \cite{shah2021all}.&#10;These approaches tell us whether a certain representation encodes a particular speech property, but they do not specifically tackle the question of the structure of the representation in the embedding space. &#10;&#10;For example, an interesting situation arises when analyzing phone and speaker classes. A given representation cannot cluster well both phones and speakers since clustering by speaker implies that phone information must be ignored and conversely. Nonetheless, by non-linearly transforming the embeddings with a specific downstream model for each case, it may be possible to classify both phones and speakers with the same embeddings. In this work, though, our goal is to understand the underlying organization of the embeddings as they come out of the model. For this reason, we avoid the use of downstream models.&#10;&#10;One way of analyzing the structure of a representation is by comparing it with another one. A comparison of two representations can be made using methods from Representation Similarity Analysis (RSA) \cite{kornblith2019similarity}. One of these methods is centered kernel alignment (CKA), which measures the similarity of the geometric structure of two representations and it has been used to identify correspondences in representations that were trained using different initializations. RSA techniques have been used to compare representations extracted from different hidden layers within the same neural network. In \cite{pasad2021layer}, they found that wav2vec2.0 layers best encoded phonetic information in the middle layers. Using CKA, \cite{chung2021similarity} found that the learning objective of self-supervised speech models affects the similarity more than the architecture does. &#10;&#10;A different approach for analyzing a representation structure involves measuring how members of the same class (for example, phones) are clustered in that space. One way of computing a metric for this is by using an ABX discrimination task \cite{schatz2013evaluating} where every sample X is classified as being of class A or not by comparing the distances in the representation space of samples A, B, with X, with B in a different class from A. This method has been used to measure the intrinsic quality of a speech representation to perform a certain classification task \cite{algayres2020evaluating,carlin2011rapid}. &#10;&#10;In this work, we analyze how different self-supervised speech models represent phone and speaker information using methods that do not require downstream models. First, we analyze the similarity of a set of acoustic features with the representations generated by the models' layers using linear CKA. Second, using non-parametric statistical testing, we measure how speech samples from the same class (phone or speaker) cluster together in the space. Specifically, given a speech sample, we measure to what extent its nearest neighbors belong to the same class using a multivariate Wilcoxon-Mann-Whitney (WMW) test. Our analysis suggests that representations with the same learning strategy (i.e., trained for the same self-supervised task) tend to represent the same type of information, regardless of their specific architecture.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Speech Processing, Speaker Information Clustering, Linguistics, Cognitive Science, Speech Representation Analysis, Acoustic Parameter Encoding" />
        </attvalues>
      </node>
      <node id="2203.01205" label="2203.01205">
        <attvalues>
          <attvalue for="0" value="Audio Self-supervised Learning: A Survey" />
          <attvalue for="1" value="  Inspired by the humans' cognitive ability to generalise knowledge and skills,&#10;Self-Supervised Learning (SSL) targets at discovering general representations&#10;from large-scale data without requiring human annotations, which is an&#10;expensive and time consuming task. Its success in the fields of computer vision&#10;and natural language processing have prompted its recent adoption into the&#10;field of audio and speech processing. Comprehensive reviews summarising the&#10;knowledge in audio SSL are currently missing. To fill this gap, in the present&#10;work, we provide an overview of the SSL methods used for audio and speech&#10;processing applications. Herein, we also summarise the empirical works that&#10;exploit the audio modality in multi-modal SSL frameworks, and the existing&#10;suitable benchmarks to evaluate the power of SSL in the computer audition&#10;domain. Finally, we discuss some open problems and point out the future&#10;directions on the development of audio SSL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13007" label="2010.13007">
        <attvalues>
          <attvalue for="0" value="Probing Acoustic Representations for Phonetic Properties" />
          <attvalue for="1" value="  Pre-trained acoustic representations such as wav2vec and DeCoAR have attained&#10;impressive word error rates (WER) for speech recognition benchmarks,&#10;particularly when labeled data is limited. But little is known about what&#10;phonetic properties these various representations acquire, and how well they&#10;encode transferable features of speech. We compare features from two&#10;conventional and four pre-trained systems in some simple frame-level phonetic&#10;classification tasks, with classifiers trained on features from one version of&#10;the TIMIT dataset and tested on features from another. All contextualized&#10;representations offered some level of transferability across domains, and&#10;models pre-trained on more audio data give better results; but overall, DeCoAR,&#10;the system with the simplest architecture, performs best. This type of&#10;benchmarking analysis can thus uncover relative strengths of various proposed&#10;acoustic representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.16193" label="2203.16193">
        <attvalues>
          <attvalue for="0" value="Probing phoneme, language and speaker information in unsupervised speech&#10;  representations" />
          <attvalue for="1" value="  Unsupervised models of representations based on Contrastive Predictive Coding&#10;(CPC)[1] are primarily used in spoken language modelling in that they encode&#10;phonetic information. In this study, we ask what other types of information are&#10;present in CPC speech representations. We focus on three categories: phone&#10;class, gender and language, and compare monolingual and bilingual models. Using&#10;qualitative and quantitative tools, we find that both gender and phone class&#10;information are present in both types of models. Language information, however,&#10;is very salient in the bilingual model only, suggesting CPC models learn to&#10;discriminate languages when trained on multiple languages. Some language&#10;information can also be retrieved from monolingual models, but it is more&#10;diffused across all features. These patterns hold when analyses are carried on&#10;the discrete units from a downstream clustering model. However, although there&#10;is no effect of the number of target clusters on phone class and language&#10;information, more gender information is encoded with more clusters. Finally, we&#10;find that there is some cost to being exposed to two languages on a downstream&#10;phoneme discrimination task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.00387" label="2101.00387">
        <attvalues>
          <attvalue for="0" value="What all do audio transformer models hear? Probing Acoustic&#10;  Representations for Language Delivery and its Structure" />
          <attvalue for="1" value="  In recent times, BERT based transformer models have become an inseparable&#10;part of the 'tech stack' of text processing models. Similar progress is being&#10;observed in the speech domain with a multitude of models observing&#10;state-of-the-art results by using audio transformer models to encode speech.&#10;This begs the question of what are these audio transformer models learning.&#10;Moreover, although the standard methodology is to choose the last layer&#10;embedding for any downstream task, but is it the optimal choice? We try to&#10;answer these questions for the two recent audio transformer models, Mockingjay&#10;and wave2vec2.0. We compare them on a comprehensive set of language delivery&#10;and structure features including audio, fluency and pronunciation features.&#10;Additionally, we probe the audio models' understanding of textual surface,&#10;syntax, and semantic features and compare them to BERT. We do this over&#10;exhaustive settings for native, non-native, synthetic, read and spontaneous&#10;speech datasets&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.13542" label="2007.13542">
        <attvalues>
          <attvalue for="0" value="Evaluating the reliability of acoustic speech embeddings" />
          <attvalue for="1" value="  Speech embeddings are fixed-size acoustic representations of variable-length&#10;speech sequences. They are increasingly used for a variety of tasks ranging&#10;from information retrieval to unsupervised term discovery and speech&#10;segmentation. However, there is currently no clear methodology to compare or&#10;optimise the quality of these embeddings in a task-neutral way. Here, we&#10;systematically compare two popular metrics, ABX discrimination and Mean Average&#10;Precision (MAP), on 5 languages across 17 embedding methods, ranging from&#10;supervised to fully unsupervised, and using different loss functions&#10;(autoencoders, correspondence autoencoders, siamese). Then we use the ABX and&#10;MAP to predict performances on a new downstream task: the unsupervised&#10;estimation of the frequencies of speech segments in a given corpus. We find&#10;that overall, ABX and MAP correlate with one another and with frequency&#10;estimation. However, substantial discrepancies appear in the fine-grained&#10;distinctions across languages and/or embedding methods. This makes it&#10;unrealistic at present to propose a task-independent silver bullet method for&#10;computing the intrinsic quality of speech embeddings. There is a need for more&#10;detailed analysis of the metrics currently used to evaluate such embeddings.&#10;" />
          <attvalue for="2" value="&#10;&#10;Unsupervised representation learning is the area of research that aims to extract units from unlabelled speech that are consistent with the phonemic transcription \cite{clsp,zs15,zs17}. As opposed to text, speech is subject to large variability. Two speech sequences with the same transcription can have significantly different raw speech signals. In order to work on speech sequences in an unsupervised way, there is a need for robust acoustic representations. To address that challenge, recent methods use {speech embeddings}, i.e.~fixed-size representations of variable-length speech sequences \cite{herman_cae,nils,settle,riad,emb2,emb3,emb5,cae}.&#10;&#10;Speech embeddings can be used in many applications, such as key-word spotting\cite{query,query2,query3}, spoken term discovery\cite{utd,utd2,utd3}, and segmentation of speech into words \cite{goldwater,seg1,seg2}. It is convenient to evaluate the reliability of speech embeddings without being tied to a particular downstream task. One way to do that is to compute the intrinsic quality of speech embeddings. The basic idea is that a reliable speech embedding should maximise the information relevant to its type and minimise irrelevant token-specific information. Two popular metrics have been used: the mean average precision (MAP) \cite{map} and the ABX discrimination score \cite{abx}.&#10;&#10;ABX and MAP are mathematically distinct yet they are expected to correlate well with each other as they both evaluate the discriminability of speech embeddings in terms of their transcription. However, \cite{nils} revealed a surprising result: the best model according to the ABX, is also the worst one according to the MAP. Following \cite{nils}'s results, we observed that this kind of discrepancies is much more common than we had expected. If a model performs well according to the MAP and bad according to the ABX, which metric should be trusted? For research in this field to go forward, there is a need to quantify the correlation of these two metrics.&#10;&#10;In this paper, we wanted to go further and check that MAP and ABX can also predict performances on a downstream task. Such tasks are numerous, but one of them has not yet received enough interest: the unsupervised frequency estimation. We define the frequency of a speech sequence as the number of times the phonetic transcription of this sequence appears in the corpus. When dealing with text corpora, frequencies can be computed exactly with a lookup table and are used in many NLP applications. In the absence of labels, deriving the frequency of a speech sequence becomes a problem of density estimation. Estimated frequencies can be useful in representation learning by enabling efficient sampling of tokens in a speech database \cite{riad}. Also, frequencies could be used for the unsupervised word segmentation using algorithms similar to those used in text \cite{goldwater}.&#10;&#10;In Section~\ref{embeddings}, we present the range of embedding models that can be grouped in five categories of increasing expected reliability: hand-crafted, unsupervised, self-supervised, supervised plus a top-line embedding. In Section~\ref{tasks}, we present the MAP and ABX metrics and introduce our frequency estimation task. In Section~\ref{results}, we present results on the five speech datasets from the ZeroSpeech Challenge \cite{zs15,zs17}. From these results, we draw guidelines for future improvements in the field of acoustic speech embeddings.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00572" label="2105.00572">
        <attvalues>
          <attvalue for="0" value="Larger-Scale Transformers for Multilingual Masked Language Modeling" />
          <attvalue for="1" value="  Recent work has demonstrated the effectiveness of cross-lingual language&#10;model pretraining for cross-lingual understanding. In this study, we present&#10;the results of two larger multilingual masked language models, with 3.5B and&#10;10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform&#10;XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the&#10;RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on&#10;average while handling 99 more languages. This suggests pretrained models with&#10;larger capacity may obtain both strong performance on high-resource languages&#10;while greatly improving low-resource languages. We make our code and models&#10;publicly available.&#10;" />
          <attvalue for="2" value="&#10;&#10;The goal of this paper is to present a study of the impact of larger capacity models on cross-lingual language understanding (XLU).&#10;We scale the capacity of XLM-R by almost two orders of magnitude while training on the same CC100 dataset~\cite{wenzek2019ccnet}. Our two new multilingual masked language model dubbed XLM-R\textsubscript{XL} and XLM-R\textsubscript{XXL}, with 3.5 and 10.7 billion parameters respectively, significantly outperform the previous XLM-R model (trained in a similar setting) on cross-lingual understanding benchmarks and obtain competitive performance with the multilingual T5 models~\cite{raffel2019exploring,xue2020mt5}. We show that they can even outperform RoBERTa-Large~\cite{roberta2019} on the GLUE benchmark~\cite{wang2018glue}.&#10;&#10;Recent multilingual masked language models (MLM) like mBERT ~\cite{devlin2018bert} or XLM \cite{lample2019cross} improved cross-lingual language understanding by pretraining large Transformer models~\cite{transformer17} on multiple languages at once. The XLM-R model~\cite{conneau2019unsupervised} extended that approach by scaling the amount of data by two orders of magnitude, from Wikipedia to Common-Crawl and training longer, similar to RoBERTa~\cite{roberta2019}.&#10;These models are particularly effective for low-resource languages, where both labeled and unlabeled data is scarce. They enable supervised cross-lingual transfer, where labeled data in one language can be used to solve the same task in other languages, and unsupervised cross-lingual transfer, where low-resource language self-supervised representations are improved using additional unlabeled data from higher-resource languages. Furthermore, they reduce the need for training one model per language, and allows the use of a single - potentially much larger - pretrained model that is then fine-tuned on annotated data from many languages.&#10;&#10;The better performance of self-supervised cross-lingual models on low-resource languages comes however at the cost of lower performance on higher-resource languages~\cite{arivazhagan2019massively}. When the number of languages becomes large, \cite{conneau2019unsupervised} even observed an overall decrease of performance on all languages. It was hypothesized that when multilingual models get more capacity, they may showcase strong performance on both high-resource languages and low-resource languages. &#10;With only 550M parameters, the XLM-R model is now relatively small compared to new standards. Recent work scaled language models to hundreds of billions~\cite{brown2020language} or even multiple trillion parameters~\cite{fedus2021switch}, showing consistent gains in doing so. Recently, multilingual T5 showed impressive increase in performance by scaling the model capacity to tens of billions of parameters. Our study complements these findings by showing the impact of larger capacity models on the important pretraining task of multilingual masked language modeling. We show promising results for cross-lingual understanding: XLM-R\textsubscript{XXL} can both obtain a new state of the art on some cross-lingual understanding benchmarks and outperform the RoBERTa-Large model on the English GLUE benchmark~\cite{wang2018glue}. This suggests that very large-scale multilingual models may be able to benefit from the best of both worlds: obtaining strong performance on high-resource languages while still allowing for zero-shot transfer and low-resource language understanding.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Modeling, Cognitive Science, Language Understanding, Pretrained Language Models, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1909.04761" label="1909.04761">
        <attvalues>
          <attvalue for="0" value="MultiFiT: Efficient Multi-lingual Language Model Fine-tuning" />
          <attvalue for="1" value="  Pretrained language models are promising particularly for low-resource&#10;languages as they only require unlabelled data. However, training existing&#10;models requires huge amounts of compute, while pretrained cross-lingual models&#10;often underperform on low-resource languages. We propose Multi-lingual language&#10;model Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tune&#10;language models efficiently in their own language. In addition, we propose a&#10;zero-shot method using an existing pretrained cross-lingual model. We evaluate&#10;our methods on two widely used cross-lingual classification datasets where they&#10;outperform models pretrained on orders of magnitude more data and compute. We&#10;release all models and code.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretrained language models (LMs) have shown striking improvements on a range of natural language processing (NLP) tasks \cite{Peters2018,Howard2018,Devlin2018}. These models only require unlabelled data for training and are thus particularly useful in scenarios where labelled data is scarce.&#10;As much of NLP research has focused on the English language, the larger promise of these models is to bridge the digital language divide and enable the application of NLP methods to many of the world's other 6,000 languages where labelled data is less plentiful.&#10;&#10;Recently, cross-lingual extensions of these LMs have been proposed that train on multiple languages jointly \cite{Artetxe2018e,Lample2019}. These models are able to perform zero-shot learning, only requiring labelled data in the source language. However, source data in another language may often not be available, whereas obtaining a small number of labels is typically straightforward.&#10;&#10;Furthermore such models have several downsides: a) some variants rely on large amounts of parallel data, which may not be available for truly low-resource languages; b) they require a huge amount of compute for training; and c) cross-lingual models underperform on low-resource languages---precisely the setting where they would be most useful.&#10;We are aware of two possible reasons for this: 1) Languages that are less frequently seen during training are underrepresented in the embedding space. 2) Infrequent scripts are over-segmented in the shared word piece vocabulary \cite{Wang2019}. &#10;&#10;In this work, we show that small monolingual LMs are able to outperform expensive cross-lingual models both in the zero-shot and the supervised setting. We propose Multi-lingual language model Fine-tuning (MultiFit) to enable practitioners to train and fine-tune language models efficiently. Our model combines universal language model fine-tuning \cite[ULMFiT;][]{Howard2018} with the quasi-recurrent neural network \cite[QRNN;][]{Bradbury2017} and subword tokenization \cite{Kudo2018} and can be pretrained on a single Tesla V100 GPU in a few hours. In addition, we propose to use a pretrained cross-lingual model's predictions as pseudo labels to adapt the monolingual language model to the zero-shot setting. We evaluate our models on two widely used cross-lingual classification datasets, MLDoc \cite{Schwenk2018} and CLS \cite{Prettenhofer2010a} where we outperform the state-of-the-art zero-shot model LASER \cite{Artetxe2018e} and multi-lingual BERT \cite{Devlin2018} in the supervised setting---even without any pretraining. In the zero-shot setting, we outperform both models using pseudo labels---and report significantly higher performance with as little as 100 examples. We finally show that information from monolingual and cross-lingual language models is complementary and that pretraining makes models robust to noise.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Cross-Lingual Classification, Linguistics, Low-Resource Languages, Language Model Training, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1902.03499" label="1902.03499">
        <attvalues>
          <attvalue for="0" value="Multilingual Neural Machine Translation With Soft Decoupled Encoding" />
          <attvalue for="1" value="  Multilingual training of neural machine translation (NMT) systems has led to&#10;impressive accuracy improvements on low-resource languages. However, there are&#10;still significant challenges in efficiently learning word representations in&#10;the face of paucity of data. In this paper, we propose Soft Decoupled Encoding&#10;(SDE), a multilingual lexicon encoding framework specifically designed to share&#10;lexical-level information intelligently without requiring heuristic&#10;preprocessing such as pre-segmenting the data. SDE represents a word by its&#10;spelling through a character encoding, and its semantic meaning through a&#10;latent embedding space shared by all languages. Experiments on a standard&#10;dataset of four low-resource languages show consistent improvements over strong&#10;multilingual NMT baselines, with gains of up to 2 BLEU on one of the tested&#10;languages, achieving the new state-of-the-art on all four language pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.09821" label="1805.09821">
        <attvalues>
          <attvalue for="0" value="A Corpus for Multilingual Document Classification in Eight Languages" />
          <attvalue for="1" value="  Cross-lingual document classification aims at training a document classifier&#10;on resources in one language and transferring it to a different language&#10;without any additional resources. Several approaches have been proposed in the&#10;literature and the current best practice is to evaluate them on a subset of the&#10;Reuters Corpus Volume 2. However, this subset covers only few languages&#10;(English, German, French and Spanish) and almost all published works focus on&#10;the the transfer between English and German. In addition, we have observed that&#10;the class prior distributions differ significantly between the languages. We&#10;argue that this complicates the evaluation of the multilinguality. In this&#10;paper, we propose a new subset of the Reuters corpus with balanced class priors&#10;for eight languages. By adding Italian, Russian, Japanese and Chinese, we cover&#10;languages which are very different with respect to syntax, morphology, etc. We&#10;provide strong baselines for all language transfer directions using&#10;multilingual word and sentence embeddings respectively. Our goal is to offer a&#10;freely available framework to evaluate cross-lingual document classification,&#10;and we hope to foster by these means, research in this important area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.08350" label="1910.08350">
        <attvalues>
          <attvalue for="0" value="A Mutual Information Maximization Perspective of Language Representation&#10;  Learning" />
          <attvalue for="1" value="  We show state-of-the-art word representation learning methods maximize an&#10;objective function that is a lower bound on the mutual information between&#10;different parts of a word sequence (i.e., a sentence). Our formulation provides&#10;an alternative perspective that unifies classical word embedding models (e.g.,&#10;Skip-gram) and modern contextual embeddings (e.g., BERT, XLNet). In addition to&#10;enhancing our theoretical understanding of these methods, our derivation leads&#10;to a principled framework that can be used to construct new self-supervised&#10;tasks. We provide an example by drawing inspirations from related methods based&#10;on mutual information maximization that have been successful in computer&#10;vision, and introduce a simple self-supervised objective that maximizes the&#10;mutual information between a global sentence representation and n-grams in the&#10;sentence. Our analysis offers a holistic view of representation learning&#10;methods to transfer knowledge and translate progress across multiple domains&#10;(e.g., natural language processing, computer vision, audio processing).&#10;" />
          <attvalue for="2" value="&#10;Advances in representation learning &#10;have driven progress &#10;in natural language processing.&#10;Performance on many downstream &#10;tasks have improved considerably,&#10;achieving parity with human baselines in &#10;benchmark leaderboards such as SQuAD \cite{squad,squad2} and GLUE \cite{glue}.&#10;The main ingredient is &#10;the ``pretrain and fine-tune'' approach,&#10;where a large text encoder is trained &#10;on an unlabeled corpus&#10;with self-supervised training objectives and used to initialize&#10;a task-specific model.&#10;Such an approach has also been shown to reduce&#10;the number of training examples that is needed to achieve good&#10;performance on the task of interest \cite{glipaper}.&#10;&#10;In contrast to first-generation models that &#10;learn word type embeddings&#10;\cite{skipgram,glove},&#10;recent methods&#10;have focused on contextual token representations---i.e., &#10;learning an encoder to &#10;represent words in context.&#10;Many of these encoders are trained with &#10;a language modeling objective, where &#10;the representation of a context is trained to &#10;be predictive of &#10;a target token by maximizing the log &#10;likelihood of predicting this token \cite{daile,ulmfit,gpt,gpt2}.&#10;In a vanilla language modeling objective, the &#10;target token&#10;is always the next token that follows the context.&#10;\cite{elmo} propose an improvement by adding a reverse &#10;objective that also predicts the word token that precedes the context.&#10;Following this trend, current state-of-the-art encoders&#10;such as BERT \cite{bert}&#10;and XLNet \cite{xlnet} &#10;are also trained with variants of the language modeling&#10;objective: masked language modeling and permutation language modeling. &#10;&#10;In this paper, we provide an alternative view and show &#10;that these methods also maximize&#10;a lower bound on the mutual information between different&#10;parts of a word sequence.&#10;Such a framework is inspired by the InfoMax &#10;principle \cite{linsker}&#10;and has been the main driver of&#10;progress in self-supervised representation learning in other domains&#10;such as computer vision, audio processing, and reinforcement learning&#10;\cite{mine,cpc,deepinfomax,amdim,lowe}.&#10;Many of these methods are trained to maximize a particular&#10;lower bound called InfoNCE \cite{cpc}---also known as contrastive&#10;learning \cite{arora}.&#10;The main idea behind contrastive learning is to divide an input data&#10;into multiple (possibly overlapping) views and maximize &#10;the mutual information&#10;between encoded representations of these views, using&#10;views derived from other inputs as negative samples.&#10;In \S{\ref{sec:mim}}, we provide an overview of representation learning with&#10;mutual information maximization. &#10;We then show how &#10;the skip-gram objective (\S{\ref{sec:skipgram}}; \cite{skipgram}),&#10;masked language modeling (\S{\ref{sec:bert}}; \cite{bert}), and permutation&#10;language modeling (\S{\ref{sec:xlnet}}; \cite{xlnet}), fit in this framework.&#10;&#10;In addition to providing a principled theoretical &#10;understanding that bridges progress in multiple areas,&#10;our proposed framework also gives rise to a &#10;general class of word representation learning models&#10;which serves as a basis for designing&#10;and combining self-supervised training objectives &#10;to create better language representations.&#10;As an example, we show how to use this framework to&#10;construct a simple self-supervised objective &#10;that maximizes&#10;the mutual information between a sentence&#10;and $n$-grams in the sentence (\S\ref{sec:infobert}).&#10;We combine it with a variant of the masked language modeling&#10;objective and &#10;show that the resulting representation performs better,&#10;particularly&#10;on tasks such as question answering and linguistics acceptability &#10;(\S\ref{sec:experiments}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Word Representation Learning, Computer Science, Machine Learning, Linguistics, Computer Vision, Mathematics, Natural Language Processing, Mutual Information Maximization, Multimodal Knowledge Transfer" />
        </attvalues>
      </node>
      <node id="1806.03822" label="1806.03822">
        <attvalues>
          <attvalue for="0" value="Know What You Don't Know: Unanswerable Questions for SQuAD" />
          <attvalue for="1" value="  Extractive reading comprehension systems can often locate the correct answer&#10;to a question in a context document, but they also tend to make unreliable&#10;guesses on questions for which the correct answer is not stated in the context.&#10;Existing datasets either focus exclusively on answerable questions, or use&#10;automatically generated unanswerable questions that are easy to identify. To&#10;address these weaknesses, we present SQuAD 2.0, the latest version of the&#10;Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD&#10;data with over 50,000 unanswerable questions written adversarially by&#10;crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0,&#10;systems must not only answer questions when possible, but also determine when&#10;no answer is supported by the paragraph and abstain from answering. SQuAD 2.0&#10;is a challenging natural language understanding task for existing models: a&#10;strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on&#10;SQuAD 2.0.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.04062" label="1801.04062">
        <attvalues>
          <attvalue for="0" value="MINE: Mutual Information Neural Estimation" />
          <attvalue for="1" value="  We argue that the estimation of mutual information between high dimensional&#10;continuous random variables can be achieved by gradient descent over neural&#10;networks. We present a Mutual Information Neural Estimator (MINE) that is&#10;linearly scalable in dimensionality as well as in sample size, trainable&#10;through back-prop, and strongly consistent. We present a handful of&#10;applications on which MINE can be used to minimize or maximize mutual&#10;information. We apply MINE to improve adversarially trained generative models.&#10;We also use MINE to implement Information Bottleneck, applying it to supervised&#10;classification; our results demonstrate substantial improvement in flexibility&#10;and performance in these settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.06670" label="1808.06670">
        <attvalues>
          <attvalue for="0" value="Learning deep representations by mutual information estimation and&#10;  maximization" />
          <attvalue for="1" value="  In this work, we perform unsupervised learning of representations by&#10;maximizing mutual information between an input and the output of a deep neural&#10;network encoder. Importantly, we show that structure matters: incorporating&#10;knowledge about locality of the input to the objective can greatly influence a&#10;representation's suitability for downstream tasks. We further control&#10;characteristics of the representation by matching to a prior distribution&#10;adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a&#10;number of popular unsupervised learning methods and competes with&#10;fully-supervised learning on several classification tasks. DIM opens new&#10;avenues for unsupervised learning of representations and is an important step&#10;towards flexible formulations of representation-learning objectives for&#10;specific end-goals.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.09229" label="1902.09229">
        <attvalues>
          <attvalue for="0" value="A Theoretical Analysis of Contrastive Unsupervised Representation&#10;  Learning" />
          <attvalue for="1" value="  Recent empirical works have successfully used unlabeled data to learn feature&#10;representations that are broadly useful in downstream classification tasks.&#10;Several of these methods are reminiscent of the well-known word2vec embedding&#10;algorithm: leveraging availability of pairs of semantically &quot;similar&quot; data&#10;points and &quot;negative samples,&quot; the learner forces the inner product of&#10;representations of similar pairs with each other to be higher on average than&#10;with negative samples. The current paper uses the term contrastive learning for&#10;such algorithms and presents a theoretical framework for analyzing them by&#10;introducing latent classes and hypothesizing that semantically similar points&#10;are sampled from the same latent class. This framework allows us to show&#10;provable guarantees on the performance of the learned representations on the&#10;average classification task that is comprised of a subset of the same set of&#10;latent classes. Our generalization bound also shows that learned&#10;representations can reduce (labeled) sample complexity on downstream tasks. We&#10;conduct controlled experiments in both the text and image domains to support&#10;the theory.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.14277" label="2405.14277">
        <attvalues>
          <attvalue for="0" value="Improving Language Models Trained on Translated Data with Continual&#10;  Pre-Training and Dictionary Learning Analysis" />
          <attvalue for="1" value="  Training LLMs for low-resource languages usually utilizes data augmentation&#10;from English using machine translation (MT). This, however, brings a number of&#10;challenges to LLM training: there are large costs attached to translating and&#10;curating huge amounts of content with high-end machine translation solutions;&#10;the translated content carries over cultural biases; and if the translation is&#10;not faithful and accurate, data quality degrades causing issues in the trained&#10;model. In this work, we investigate the role of translation and synthetic data&#10;in training language models. We translate TinyStories, a dataset of 2.2M short&#10;stories for 3-4 year old children, from English to Arabic using the open&#10;NLLB-3B MT model. We train a number of story generation models of size 1M-33M&#10;parameters using this data. We identify a number of quality and task-specific&#10;issues in the resulting models. To rectify these issues, we further pre-train&#10;the models with a small dataset of synthesized high-quality Arabic stories&#10;generated by a capable LLM, representing 1% of the original training data. We&#10;show, using GPT-4 as a judge and Dictionary Learning Analysis from mechanistic&#10;interpretability, that the suggested approach is a practical means to resolve&#10;some of the machine translation pitfalls. We illustrate the improvements&#10;through case studies of linguistic and cultural bias issues.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs) have transformed the landscape of natural language processing (NLP) significantly. With the realization of unprecedented capabilities, the availability of underlying large training corpora and well-established language model training pipelines has shifted the focus in NLP research from defining linguistic inductive biases to the collection and curation of extensive text datasets \cite{dolma, penedo2024fineweb, together2023redpajama, openelm}. The recent trend showed an increase in focus on data curation and augmentation compared to innovation in model architecture or training paradigms \cite{brown2020language, touvron2023llama}. &#10;However, emergent capabilities in LLMs are noticeable for models of very large sizes such as 10B parameters or above and most of smaller LLMs of 200M-3B parameters have shown limited abilities in reasoning, fact recall and coherent long generation~\cite{schaeffer2024emergent}. While these models are typically trained with the same data used to train their larger and more capable siblings, the drops in capabilities of these small LLM are usually attributed to their reduced learnability or scale. &#10;&#10;However, newer small models have been recently shown to compete with 10x bigger models in challenging tasks. Examples include MiniMA~\cite{zhang2023towards} and the Phi family~\cite{abdin2024phi, li2023textbooks}. These small language models (SLMs) offer faster training and lower deployment cost at the expense of being more task-specific and less general-use, which is not an issue for application-oriented language models. The key ingredient for the success of these smaller models seems to be the use of sophisticated and aggressive data curation, and high-quality synthetic data generated by bigger LLMs. &#10;&#10;This new trend, however, does not map equally to languages that are not as privileged with huge amounts of high quality content (or content in quantities that allow aggressive filtering), nor the availability of strong models that can be employed to generate diverse synthetic data in substantial quantities in a cost-effective manner, as is the situation with Arabic and many other low-resourced languages~\cite{thompson2024shocking}. &#10;&#10;A commonly-adopted workaround to the issue of data shortage is to turn to machine translation (MT) to benefit from the available content in English, which is evident in the data mixtures of the more capable Arabic models, e.g. Jais~\cite{sengupta2023jais}. However, the use of machine translation does not come without pitfalls. Specifically, 1) cultural biases that are stored within a language corpus get imported when translation is used, leading to misaligned models, e.g.~\cite{holmstrom-etal-2023-bridging} and 2) based on the quality of the translation, certain linguistic intricacies of languages might not be respected~\cite{zhang-toral-2019-effect}, leading to degradation in the quality of the data and thus degradation in the capability of the final model with regard to the quality of the target language. &#10;&#10;In this work, we study this phenomenon using the recently-released TinyStories~\cite{eldan2023tinystories}, a synthetic dataset introduced to explore emergent properties in small language models. TinyStories comprises 2.2M short stories in English of about 200 words generated by GPT-3.5 and GPT-4. The relatively small models trained on TinyStories have shown interesting capabilities in generating coherent and creative short stories with correct grammar. We translate TinyStories to Arabic using the open-source translation model NLLB-3B to simulate medium-quality translation. We train models with different sizes using the Arabic-translated TinyStories, and benchmark the trained models against several other Arabic LLMs using GPT-4 as a judge following previous works \cite{zheng2024judging} in the task of story generation using three metrics: grammar correctness, consistency with the provided context, and creativity. We also identify some cultural and linguistic issues that arise from the translation. Then, we synthesize a high quality small dataset of 20K stories to explore the efficacy of continual pre-training in recovering from the issues brought along by low-quality translated data. &#10;&#10;The contributions of this work are as follow: &#10;\begin{itemize}&#10;\item We investigate the degradation of language models when using translated data of medium quality in training. We identify linguistic issues and cultural biases, which we address by further pre-training the models with a limited amount of high-quality synthetic data.\item We create a dataset with an Arabic version of TinyStories through medium quality translation, plus a small high-quality synthetic data which we will release as open-source to facilitate studying translation issues.&#10;\item We provide a comparative analysis before and after refinement using dictionary learning methods from mechanistic interpretability to assess the effects of continual pre-training with high-quality data. &#10;\end{itemize}&#10;&#10;The rest of the paper is organized as follows: First, we describe how we prepare training data. Then we discuss the limitations of translated data. In Section~\ref{sec:continue_pre-training}, we show how continual pre-training with small amount of high-quality improve the models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Low-Resource Language Solutions, Linguistics, Language Model Training, Artificial Intelligence, Machine Translation Challenges, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.15004" label="2304.15004">
        <attvalues>
          <attvalue for="0" value="Are Emergent Abilities of Large Language Models a Mirage?" />
          <attvalue for="1" value="  Recent work claims that large language models display emergent abilities,&#10;abilities not present in smaller-scale models that are present in larger-scale&#10;models. What makes emergent abilities intriguing is two-fold: their sharpness,&#10;transitioning seemingly instantaneously from not present to present, and their&#10;unpredictability, appearing at seemingly unforeseeable model scales. Here, we&#10;present an alternative explanation for emergent abilities: that for a&#10;particular task and model family, when analyzing fixed model outputs, emergent&#10;abilities appear due to the researcher's choice of metric rather than due to&#10;fundamental changes in model behavior with scale. Specifically, nonlinear or&#10;discontinuous metrics produce apparent emergent abilities, whereas linear or&#10;continuous metrics produce smooth, continuous predictable changes in model&#10;performance. We present our alternative explanation in a simple mathematical&#10;model, then test it in three complementary ways: we (1) make, test and confirm&#10;three predictions on the effect of metric choice using the InstructGPT/GPT-3&#10;family on tasks with claimed emergent abilities; (2) make, test and confirm two&#10;predictions about metric choices in a meta-analysis of emergent abilities on&#10;BIG-Bench; and (3) show to choose metrics to produce never-before-seen&#10;seemingly emergent abilities in multiple vision tasks across diverse deep&#10;networks. Via all three analyses, we provide evidence that alleged emergent&#10;abilities evaporate with different metrics or with better statistics, and may&#10;not be a fundamental property of scaling AI models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.14219" label="2404.14219">
        <attvalues>
          <attvalue for="0" value="Phi-3 Technical Report: A Highly Capable Language Model Locally on Your&#10;  Phone" />
          <attvalue for="1" value="  We introduce phi-3-mini, a 3.8 billion parameter language model trained on&#10;3.3 trillion tokens, whose overall performance, as measured by both academic&#10;benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and&#10;GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite&#10;being small enough to be deployed on a phone. Our training dataset is a&#10;scaled-up version of the one used for phi-2, composed of heavily filtered&#10;publicly available web data and synthetic data. The model is also further&#10;aligned for robustness, safety, and chat format. We also provide&#10;parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called&#10;phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini&#10;(e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance&#10;multilingual, multimodal, and long-context capabilities, we introduce three&#10;models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision.&#10;The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters,&#10;achieves superior performance in language reasoning, math, and code tasks&#10;compared to other open-source models of similar scale, such as Llama 3.1 and&#10;the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini.&#10;Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from&#10;phi-3.5-mini, excels in reasoning tasks and is adept at handling both&#10;single-image and text prompts, as well as multi-image and text prompts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.05749" label="2401.05749">
        <attvalues>
          <attvalue for="0" value="A Shocking Amount of the Web is Machine Translated: Insights from&#10;  Multi-Way Parallelism" />
          <attvalue for="1" value="  We show that content on the web is often translated into many languages, and&#10;the low quality of these multi-way translations indicates they were likely&#10;created using Machine Translation (MT). Multi-way parallel, machine generated&#10;content not only dominates the translations in lower resource languages; it&#10;also constitutes a large fraction of the total web content in those languages.&#10;We also find evidence of a selection bias in the type of content which is&#10;translated into many languages, consistent with low quality English content&#10;being translated en masse into many lower resource languages, via MT. Our work&#10;raises serious concerns about training models such as multilingual large&#10;language models on both monolingual and bilingual data scraped from the web.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.16149" label="2308.16149">
        <attvalues>
          <attvalue for="0" value="Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open&#10;  Generative Large Language Models" />
          <attvalue for="1" value="  We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric&#10;foundation and instruction-tuned open generative large language models (LLMs).&#10;The models are based on the GPT-3 decoder-only architecture and are pretrained&#10;on a mixture of Arabic and English texts, including source code in various&#10;programming languages. With 13 billion parameters, they demonstrate better&#10;knowledge and reasoning capabilities in Arabic than any existing open Arabic&#10;and multilingual models by a sizable margin, based on extensive evaluation.&#10;Moreover, the models are competitive in English compared to English-centric&#10;open models of similar size, despite being trained on much less English data.&#10;We provide a detailed description of the training, the tuning, the safety&#10;alignment, and the evaluation of the models. We release two open versions of&#10;the model -- the foundation Jais model, and an instruction-tuned Jais-chat&#10;variant -- with the aim of promoting research on Arabic LLMs. Available at&#10;https://huggingface.co/inception-mbzuai/jais-13b-chat&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.08069" label="1906.08069">
        <attvalues>
          <attvalue for="0" value="The Effect of Translationese in Machine Translation Test Sets" />
          <attvalue for="1" value="  The effect of translationese has been studied in the field of machine&#10;translation (MT), mostly with respect to training data. We study in depth the&#10;effect of translationese on test data, using the test sets from the last three&#10;editions of WMT's news shared task, containing 17 translation directions. We&#10;show evidence that (i) the use of translationese in test sets results in&#10;inflated human evaluation scores for MT systems; (ii) in some cases system&#10;rankings do change and (iii) the impact translationese has on a translation&#10;direction is inversely correlated to the translation quality attainable by&#10;state-of-the-art MT systems for that direction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.16377" label="2406.16377">
        <attvalues>
          <attvalue for="0" value="On the Transformations across Reward Model, Parameter Update, and&#10;  In-Context Prompt" />
          <attvalue for="1" value="  Despite the general capabilities of pre-trained large language models (LLMs),&#10;they still need further adaptation to better serve practical applications. In&#10;this paper, we demonstrate the interchangeability of three popular and distinct&#10;adaptation tools: parameter updating, reward modeling, and in-context&#10;prompting. This interchangeability establishes a triangular framework with six&#10;transformation directions, each of which facilitates a variety of applications.&#10;Our work offers a holistic view that unifies numerous existing studies and&#10;suggests potential research directions. We envision our work as a useful&#10;roadmap for future research on LLMs.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) pre-trained on large-scale corpora through self-supervised learning have developed substantial world knowledge and reasoning capabilities. However, when deploying them to specific real-world applications, these models still require further adaptation to achieve desired behaviors \cite{ouyang2022training}. To solve the last-mile problem of adapting LLMs to practical downstream tasks, researchers and practitioners usually resort to three kinds of approaches.&#10;&#10;The most classic approach is to modify the internal representations and mechanisms of LLMs via parameter update, such as fine-tuning the models on a set of demonstrations of desirable (and undesirable) behaviors \cite{zhou2024lima,hu2021lora,houlsby2019parameter}. Another approach involves using a reward model to differentiate between desirable and undesirable outputs \cite{ouyang2022training, liu2024statistical}. That is, the reward model should assign higher scores to more desired outputs, providing reliable guidance on how the model should behave. Thanks to the exceptional in-context learning capabilities of LLMs \cite{brown2020language}, in-context prompting \cite{wei2023larger,wei2022chain,liu2023pre} has also emerged as a promising new method for altering the model behavior by simply augmenting the model input with an informative prompt.&#10;&#10;In this paper, we offer a holistic view that these three tools (parameter update, reward model, and in-context prompt) are mutually interchangeable. This interchangeability forms a triangle with six transformation directions, and each transformation facilitates a range of downstream applications. Our systematic analysis connects numerous existing studies and outlines possible future research.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Artificial Intelligence Applications, Artificial Intelligence, Mathematics, Natural Language Processing, Machine Learning Frameworks, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2404.10642" label="2404.10642">
        <attvalues>
          <attvalue for="0" value="Self-playing Adversarial Language Game Enhances LLM Reasoning" />
          <attvalue for="1" value="  We explore the self-play training procedure of large language models (LLMs)&#10;in a two-player adversarial language game called Adversarial Taboo. In this&#10;game, an attacker and a defender communicate around a target word only visible&#10;to the attacker. The attacker aims to induce the defender to speak the target&#10;word unconsciously, while the defender tries to infer the target word from the&#10;attacker's utterances. To win the game, both players should have sufficient&#10;knowledge about the target word and high-level reasoning ability to infer and&#10;express in this information-reserved conversation. Hence, we are curious about&#10;whether LLMs' reasoning ability can be further enhanced by self-play in this&#10;adversarial language game (SPAG). With this goal, we select several open-source&#10;LLMs and let each act as the attacker and play with a copy of itself as the&#10;defender on an extensive range of target words. Through reinforcement learning&#10;on the game outcomes, we observe that the LLMs' performances uniformly improve&#10;on a broad range of reasoning benchmarks. Furthermore, iteratively adopting&#10;this self-play process can continuously promote LLMs' reasoning abilities. The&#10;code is at https://github.com/Linear95/SPAG.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs), such as GPT-4~\cite{openai2022gpt4} and Gemini~\cite{team2023gemini}, have reformed the domain of artificial intelligence (AI) with astonishing language capacities, such as natural language understanding~\cite{yang2023harnessing,touvron2023llama}, text generation~\cite{kocon2023chatgpt,anil2023palm}, machine translation~\cite{jiao2023chatgpt}, summarization~\cite{xie-etal-2024-chunk}, and programming~\cite{surameery2023use,tian2023chatgpt}. However, the reasoning ability of LLMs, which is essential for complex problem-solving~\cite{pan2023logic} and advanced intelligence-developing~\cite{yao2021adversarial}, still retains being challenged by various criteria including correctness~\cite{zhang2023language} and faithfulness~\cite{turpin2024language}. &#10;&#10;To address the reasoning challenge of LLMs, plenty of works have contributed in-depth efforts from the perspectives of Chain-of-Thought (CoT) prompt engineering~\cite{wei2022chain,ding2023everything,yao2024tree}, and the usage of auxiliary reasoning tools~\cite{pan2023logic}. However, both prompt-based and tool-calling methods require additional prompt designs, which are inconsistent and sensitive to different prompt patterns and LLM checkpoints~\cite{turpin2024language,chu2023survey}. More fundamental and consistent reasoning-improving approaches are post-pretraining~\cite{azerbayev2023llemma} and fine-tuning~\cite{dong2023abilities}, which trains LLMs with additional reasoning-related text corpus.&#10;Nevertheless, these methods demand sufficient high-quality textual data, which are difficult to collect due to the massive costs of human annotation efforts~\cite{singh2023beyond}. &#10;&#10;To improve LLM reasoning more efficiently, self-improvement methods, which enhance LLMs with model-generated synthetic data, have recently attracted increasing research attention~\cite{singh2023beyond,huang2023large,burns2023weak,chen2024self}. Self-improvement methods usually utilize the intrinsic language capability of LLMs to judge~\cite{huang2023large}, filter~\cite{yuan2024self}, or revise~\cite{yuan2024self} self-generated samples to enhance their quality. However, most self-improvement methods rely on a broad range of high-quality question queries to prevent over-fitting into a sub-domain of reasoning tasks, which still requires additional data collection and cleaning. Besides, the judgments from LLMs are not guaranteed objective~\cite{raina2024llm}. If an LLM already has an incorrect or biased recognition of a particular concept, the self-improvement process can reinforce and amplify the LLM's cognitive dissonance.&#10;&#10;Towards more general and objective self-reasoning-improvement methods, we are inspired by the advancement from AlphaGO~\cite{silver2016mastering} to AlphaGO Zero~\cite{silver2017mastering}, in which the game agents' intelligence continuously promotes via self-play without any human knowledge.&#10;Analogically, we expect to set up a language game where LLMs can improve their reasoning capacities via reinforcement learning (RL) during self-play. &#10;Although language games have attracted increasing attention in natural language processing~\cite{lewis2017deal,hausknecht2020interactive,xu2023exploring,wu2024enhance}, most of them are specially designed with customized game rules, in lack of the generalization to improve the general language capacities of LLMs. Among a few general-target language games including red-teaming~\cite{ma2023red}, negotiation~\cite{lewis2017deal}, and bargain~\cite{abdulhai2023lmrl}, additional human judgments or reward models are required for outcome determination, which posts challenges on the efficiency and effectiveness of large-scale self-play RL training. Recent studies have raised interest in entity- or word-based language games, such as 20-Question~\cite{zhang2023entity} &#10;and Guess-My-City~\cite{abdulhai2023lmrl}&#10;, which provide not only straight-forward word-level outcomes but also language universality by traversing the game word from comprehensive vocabularies. However, unlike the GO game, these word-based games are out of the adversarial scheme, limiting the game intensity and self-play learning effectiveness. &#10;&#10;With the above consideration, we &#10;select an&#10;adversarial language game called Adversarial Taboo~\cite{yao2021adversarial}, in which an attacker and a defender &#10;perform&#10;a conversation around a target word only visible to the attacker. The attacker aims to induce the defender to speak out the target word unconsciously; the defender tries to avoid unconscious utterance of the target word and guess the word from the dialogue history. To win the adversarial game in information-limited conversations, both players are required to have high-level language capacities in terms of expression, upstanding, and reasoning.&#10;Moreover, by collecting target words from a vast vocabulary, this game can cover a broad range of topics providing sufficient language versatility. Besides, the game outcomes can be automatically and explicitly judged: we only need to check whether the target word appears in the defender's utterances (attacker wins) or its inference patterns (defender wins). &#10;We conduct the self-play on this adversarial game using open-source LLMs, LLaMA-2-7B~\cite{touvron2023llama} and Baichuan-2-13B~\cite{yang2023baichuan},&#10;with target words selected from a 50K top-frequency vocabulary~\cite{davies_coca_2019}. Next, we conduct offline reinforcement learning on the game outcomes and observe significant performance improvement on a broad range of LLM reasoning benchmarks. Furthermore, we iterate this sampling-learning process with three epochs, within which the LLMs' reasoning can continuously obtain improvement. We believe this novel training scheme, Self-Play of Adversarial Game (SPAG), has great potential for developing advanced LLM capacities.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Adversarial Language Games, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Reinforcement Learning" />
        </attvalues>
      </node>
      <node id="2302.10724" label="2302.10724">
        <attvalues>
          <attvalue for="0" value="ChatGPT: Jack of all trades, master of none" />
          <attvalue for="1" value="  OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and&#10;revolutionized the approach in artificial intelligence to human-model&#10;interaction. Several publications on ChatGPT evaluation test its effectiveness&#10;on well-known natural language processing (NLP) tasks. However, the existing&#10;studies are mostly non-automated and tested on a very limited scale. In this&#10;work, we examined ChatGPT's capabilities on 25 diverse analytical NLP tasks,&#10;most of them subjective even to humans, such as sentiment analysis, emotion&#10;recognition, offensiveness, and stance detection. In contrast, the other tasks&#10;require more objective reasoning like word sense disambiguation, linguistic&#10;acceptability, and question answering. We also evaluated GPT-4 model on five&#10;selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process&#10;and analyzed more than 49k responses. Our comparison of its results with&#10;available State-of-the-Art (SOTA) solutions showed that the average loss in&#10;quality of the ChatGPT model was about 25% for zero-shot and few-shot&#10;evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower&#10;than for ChatGPT. We showed that the more difficult the task (lower SOTA&#10;performance), the higher the ChatGPT loss. It especially refers to pragmatic&#10;NLP problems like emotion recognition. We also tested the ability to&#10;personalize ChatGPT responses for selected subjective tasks via Random&#10;Contextual Few-Shot Personalization, and we obtained significantly better&#10;user-based predictions. Additional qualitative analysis revealed a ChatGPT&#10;bias, most likely due to the rules imposed on human trainers by OpenAI. Our&#10;results provide the basis for a fundamental discussion of whether the high&#10;quality of recent predictive NLP models can indicate a tool's usefulness to&#10;society and how the learning and validation procedures for such systems should&#10;be established.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.13191" label="2308.13191">
        <attvalues>
          <attvalue for="0" value="Chunk, Align, Select: A Simple Long-sequence Processing Method for&#10;  Transformers" />
          <attvalue for="1" value="  Although dominant in natural language processing, transformer-based models&#10;remain challenged by the task of long-sequence processing, because the&#10;computational cost of self-attention operations in transformers swells&#10;quadratically with the input sequence length. To alleviate the complexity of&#10;long-sequence processing, we propose a simple framework to enable the&#10;offthe-shelf pre-trained transformers to process much longer sequences, while&#10;the computation and memory costs remain growing linearly with the input&#10;sequence lengths. More specifically, our method divides each long-sequence&#10;input into a batch of chunks, then aligns the interchunk information during the&#10;encoding steps, and finally selects the most representative hidden states from&#10;the encoder for the decoding process. To extract inter-chunk semantic&#10;information, we align the start and end token embeddings among chunks in each&#10;encoding transformer block. To learn an effective hidden selection policy, we&#10;design a dual updating scheme inspired by reinforcement learning, which regards&#10;the decoders of transformers as environments, and the downstream performance&#10;metrics as the rewards to evaluate the hidden selection actions. Our empirical&#10;results on real-world long-text summarization and reading comprehension tasks&#10;demonstrate effective improvements compared to prior longsequence processing&#10;baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.01622" label="1911.01622">
        <attvalues>
          <attvalue for="0" value="Adversarial Language Games for Advanced Natural Language Intelligence" />
          <attvalue for="1" value="  We study the problem of adversarial language games, in which multiple agents&#10;with conflicting goals compete with each other via natural language&#10;interactions. While adversarial language games are ubiquitous in human&#10;activities, little attention has been devoted to this field in natural language&#10;processing. In this work, we propose a challenging adversarial language game&#10;called Adversarial Taboo as an example, in which an attacker and a defender&#10;compete around a target word. The attacker is tasked with inducing the defender&#10;to utter the target word invisible to the defender, while the defender is&#10;tasked with detecting the target word before being induced by the attacker. In&#10;Adversarial Taboo, a successful attacker must hide its intention and subtly&#10;induce the defender, while a competitive defender must be cautious with its&#10;utterances and infer the intention of the attacker. Such language abilities can&#10;facilitate many important downstream NLP tasks. To instantiate the game, we&#10;create a game environment and a competition platform. Comprehensive experiments&#10;and empirical studies on several baseline attack and defense strategies show&#10;promising and interesting results. Based on the analysis on the game and&#10;experiments, we discuss multiple promising directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.04388" label="2305.04388">
        <attvalues>
          <attvalue for="0" value="Language Models Don't Always Say What They Think: Unfaithful&#10;  Explanations in Chain-of-Thought Prompting" />
          <attvalue for="1" value="  Large Language Models (LLMs) can achieve strong performance on many tasks by&#10;producing step-by-step reasoning before giving a final output, often referred&#10;to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT&#10;explanations as the LLM's process for solving a task. This level of&#10;transparency into LLMs' predictions would yield significant safety benefits.&#10;However, we find that CoT explanations can systematically misrepresent the true&#10;reason for a model's prediction. We demonstrate that CoT explanations can be&#10;heavily influenced by adding biasing features to model inputs--e.g., by&#10;reordering the multiple-choice options in a few-shot prompt to make the answer&#10;always &quot;(A)&quot;--which models systematically fail to mention in their&#10;explanations. When we bias models toward incorrect answers, they frequently&#10;generate CoT explanations rationalizing those answers. This causes accuracy to&#10;drop by as much as 36% on a suite of 13 tasks from BIG-Bench Hard, when testing&#10;with GPT-3.5 from OpenAI and Claude 1.0 from Anthropic. On a social-bias task,&#10;model explanations justify giving answers in line with stereotypes without&#10;mentioning the influence of these social biases. Our findings indicate that CoT&#10;explanations can be plausible yet misleading, which risks increasing our trust&#10;in LLMs without guaranteeing their safety. Building more transparent and&#10;explainable systems will require either improving CoT faithfulness through&#10;targeted efforts or abandoning CoT in favor of alternative methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.04254" label="2311.04254">
        <attvalues>
          <attvalue for="0" value="Everything of Thoughts: Defying the Law of Penrose Triangle for Thought&#10;  Generation" />
          <attvalue for="1" value="  Recent advancements in Large Language Models (LLMs) have revolutionized&#10;decision-making by breaking down complex problems into more manageable language&#10;sequences referred to as &quot;thoughts&quot;. An effective thought design should&#10;consider three key perspectives: performance, efficiency, and flexibility.&#10;However, existing thought can at most exhibit two of these attributes. To&#10;address these limitations, we introduce a novel thought prompting approach&#10;called &quot;Everything of Thoughts&quot; (XoT) to defy the law of &quot;Penrose triangle of&#10;existing thought paradigms. XoT leverages pretrained reinforcement learning and&#10;Monte Carlo Tree Search (MCTS) to incorporate external domain knowledge into&#10;thoughts, thereby enhancing LLMs' capabilities and enabling them to generalize&#10;to unseen problems efficiently. Through the utilization of the MCTS-LLM&#10;collaborative thought revision framework, this approach autonomously produces&#10;high-quality comprehensive cognitive mappings with minimal LLM interactions.&#10;Additionally, XoT empowers LLMs to engage in unconstrained thinking, allowing&#10;for flexible cognitive mappings for problems with multiple solutions. We&#10;evaluate XoT on several challenging multi-solution problem-solving tasks,&#10;including Game of 24, 8-Puzzle, and Pocket Cube. Our results demonstrate that&#10;XoT significantly outperforms existing approaches. Notably, XoT can yield&#10;multiple solutions with just one LLM call, showcasing its remarkable&#10;proficiency in addressing complex problems across diverse domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.06585" label="2312.06585">
        <attvalues>
          <attvalue for="0" value="Beyond Human Data: Scaling Self-Training for Problem-Solving with&#10;  Language Models" />
          <attvalue for="1" value="  Fine-tuning language models~(LMs) on human-generated data remains a prevalent&#10;practice. However, the performance of such models is often limited by the&#10;quantity and diversity of high-quality human data. In this paper, we explore&#10;whether we can go beyond human data on tasks where we have access to scalar&#10;feedback, for example, on math problems where one can verify correctness. To do&#10;so, we investigate a simple self-training method based on&#10;expectation-maximization, which we call ReST$^{EM}$, where we (1) generate&#10;samples from the model and filter them using binary feedback, (2) fine-tune the&#10;model on these samples, and (3) repeat this process a few times. Testing on&#10;advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find&#10;that ReST$^{EM}$ scales favorably with model size and significantly surpasses&#10;fine-tuning only on human data. Overall, our findings suggest self-training&#10;with feedback can substantially reduce dependence on human-generated data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.14016" label="2402.14016">
        <attvalues>
          <attvalue for="0" value="Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on&#10;  Zero-shot LLM Assessment" />
          <attvalue for="1" value="  Large Language Models (LLMs) are powerful zero-shot assessors used in&#10;real-world situations such as assessing written exams and benchmarking systems.&#10;Despite these critical applications, no existing work has analyzed the&#10;vulnerability of judge-LLMs to adversarial manipulation. This work presents the&#10;first study on the adversarial robustness of assessment LLMs, where we&#10;demonstrate that short universal adversarial phrases can be concatenated to&#10;deceive judge LLMs to predict inflated scores. Since adversaries may not know&#10;or have access to the judge-LLMs, we propose a simple surrogate attack where a&#10;surrogate model is first attacked, and the learned attack phrase then&#10;transferred to unknown judge-LLMs. We propose a practical algorithm to&#10;determine the short universal attack phrases and demonstrate that when&#10;transferred to unseen models, scores can be drastically inflated such that&#10;irrespective of the assessed text, maximum scores are predicted. It is found&#10;that judge-LLMs are significantly more susceptible to these adversarial attacks&#10;when used for absolute scoring, as opposed to comparative assessment. Our&#10;findings raise concerns on the reliability of LLM-as-a-judge methods, and&#10;emphasize the importance of addressing vulnerabilities in LLM assessment&#10;methods before deployment in high-stakes real-world scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.05398" label="1909.05398">
        <attvalues>
          <attvalue for="0" value="Interactive Fiction Games: A Colossal Adventure" />
          <attvalue for="1" value="  A hallmark of human intelligence is the ability to understand and communicate&#10;with language. Interactive Fiction games are fully text-based simulation&#10;environments where a player issues text commands to effect change in the&#10;environment and progress through the story. We argue that IF games are an&#10;excellent testbed for studying language-based autonomous agents. In particular,&#10;IF games combine challenges of combinatorial action spaces, language&#10;understanding, and commonsense reasoning. To facilitate rapid development of&#10;language-based agents, we introduce Jericho, a learning environment for&#10;man-made IF games and conduct a comprehensive study of text-agents across a&#10;rich set of games, highlighting directions in which agents can improve.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.04658" label="2309.04658">
        <attvalues>
          <attvalue for="0" value="Exploring Large Language Models for Communication Games: An Empirical&#10;  Study on Werewolf" />
          <attvalue for="1" value="  Communication games, which we refer to as incomplete information games that&#10;heavily depend on natural language communication, hold significant research&#10;value in fields such as economics, social science, and artificial intelligence.&#10;In this work, we explore the problem of how to engage large language models&#10;(LLMs) in communication games, and in response, propose a tuning-free&#10;framework. Our approach keeps LLMs frozen, and relies on the retrieval and&#10;reflection on past communications and experiences for improvement. An empirical&#10;study on the representative and widely-studied communication game,&#10;``Werewolf'', demonstrates that our framework can effectively play Werewolf&#10;game without tuning the parameters of the LLMs. More importantly, strategic&#10;behaviors begin to emerge in our experiments, suggesting that it will be a&#10;fruitful journey to engage LLMs in communication games and associated domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.18232" label="2311.18232">
        <attvalues>
          <attvalue for="0" value="LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language&#10;  Models" />
          <attvalue for="1" value="  Large language models (LLMs) provide excellent text-generation capabilities,&#10;but standard prompting and generation methods generally do not lead to&#10;intentional or goal-directed agents and might necessitate considerable prompt&#10;tuning. This becomes particularly apparent in multi-turn conversations: even&#10;the best current LLMs rarely ask clarifying questions, engage in explicit&#10;information gathering, or take actions now that lead to better decisions after&#10;multiple turns. Reinforcement learning has the potential to leverage the&#10;powerful modeling capabilities of LLMs, as well as their internal&#10;representation of textual interactions, to create capable goal-directed&#10;language agents. This can enable intentional and temporally extended&#10;interactions, such as with humans, through coordinated persuasion and carefully&#10;crafted questions, or in goal-directed play through text games to bring about&#10;desired final outcomes. However, enabling this requires the community to&#10;develop stable and reliable reinforcement learning algorithms that can&#10;effectively train LLMs. Developing such algorithms requires tasks that can&#10;gauge progress on algorithm design, provide accessible and reproducible&#10;evaluations for multi-turn interactions, and cover a range of task properties&#10;and challenges in improving reinforcement learning algorithms. Our paper&#10;introduces the LMRL-Gym benchmark for evaluating multi-turn RL for LLMs,&#10;together with an open-source research framework containing a basic toolkit for&#10;getting started on multi-turn RL with offline value-based and policy-based RL&#10;methods. Our benchmark consists of 8 different language tasks, which require&#10;multiple rounds of language interaction and cover a range of tasks in&#10;open-ended dialogue and text games.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;Datasets, benchmarks, and libraries. Benchmarks and datasets have been an important factor for driving progress in NLP in domains that include machine translation~\cite{tiedemann-2012-parallel, bojar-EtAl:2016:WMT1}, natural language understanding~\cite{rajpurkar2016squad, wang2018glue, hendrycks2020measuring, hendrycks2021measuring, ramamurthy2023is}, and solving math problems~\cite{cobbe2021training}. However, these tasks generally do not involve multi-turn interaction and do not come with rewards, making them hard to adapt to RL research. &#10;For example, the standard for evaluating dialogue agents has been to run a human subjects study, but this is time-consuming and costly. Some works have proposed text games for evaluating language-based agents~\cite{hausknecht19,hendrycks2021jiminycricket,wang2022scienceworld, yao2023webshop}. Our aim is to cover a variety of problem settings that reflect challenges in open-vocabulary interaction in addition to text games. Motivated by successes in using LLMs to generate synthetic data~\cite{hausknecht19, park2023generative, bai2022constitutional}, our proposed tasks are based on synthetic data. While such data may differ from natural text, the scope of our benchmark is specific to evaluating RL algorithms, not the ability to interact with humans.&#10;&#10;RL for language models. &#10;RL for language models has seen success in aligning LLMs with human preferences (RLHF)~\cite{ziegler2020finetuning, stiennon2020learning, bai2022training, bai2022constitutional, ouyang2022training, christiano2023deep}, optimizing non-differentiable objectives for machine translation~\cite{wu2016googles, nguyen-etal-2017-reinforcement, kiegeland-kreutzer-2021-revisiting}, generation~\cite{Tambwekar_2019, pang2021text, pyatkin2022reinforced}, dialogue~\cite{cuayahuitl2015strategic, rl_negotiation, li2016deep}, question answering~\cite{pyatkin2022reinforced}, and summarization~\cite{paulus2017deep, bohm-etal-2019-better, wu2018learning}. These include RL methods that learn by directly interacting with the environment (online RL)~\cite{carta2023grounding} and RL methods that only use a static dataset (offline RL)~\cite{jaques2020humancentric, snell2022offline, jang2022gptcritic, verma2022chai, cicero}. However, many of these works operate in the singe-step bandit setting, and do not consider multi-turn goal-directed tasks. Our benchmark, on the other hand, focuses on tasks involving multiple turns of interaction with clearly defined goal-based reward functions.&#10;&#10;Capabilities of LLMs. &#10;There has been a surge in the capabilities of LLMs for generation~\cite{ghazvininejad-etal-2017-hafez, radford2019language}, dialogue~\cite{lewis2017deal, pmlr-v70-jaques17a, shuster2022blenderbot, snell-etal-2022-context}, question answering~\cite{pyatkin2022reinforced}, summarization~\cite{paulus2017deep, bohm-etal-2019-better, wu2018learning}, text-based games~\cite{narasimhan-etal-2015-language, hausknecht19}, translation~\cite{gu-etal-2017-trainable}, and more. However, these are often supervised learning tasks that do not test the LLMs' abilities to achieve a specific long-term objective. Research on dialogue generation~\cite{pmlr-v70-jaques17a, he2018decoupling, shuster2022blenderbot, shuster2022language} has often focused on generating feasible-looking agent dialogue without explicit consideration for some multi-turn objective.&#10;Our benchmarks allow for the development of algorithms that enable LLMs to interact with an environment to achieve long-term objectives, by providing tasks with online simulators and offline datasets. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2306.15268" label="2306.15268">
        <attvalues>
          <attvalue for="0" value="Can Pretrained Language Models Derive Correct Semantics from Corrupt&#10;  Subwords under Noise?" />
          <attvalue for="1" value="  For Pretrained Language Models (PLMs), their susceptibility to noise has&#10;recently been linked to subword segmentation. However, it is unclear which&#10;aspects of segmentation affect their understanding. This study assesses the&#10;robustness of PLMs against various disrupted segmentation caused by noise. An&#10;evaluation framework for subword segmentation, named Contrastive Lexical&#10;Semantic (CoLeS) probe, is proposed. It provides a systematic categorization of&#10;segmentation corruption under noise and evaluation protocols by generating&#10;contrastive datasets with canonical-noisy word pairs. Experimental results&#10;indicate that PLMs are unable to accurately compute word meanings if the noise&#10;introduces completely different subwords, small subword fragments, or a large&#10;number of additional subwords, particularly when they are inserted within other&#10;subwords.&#10;" />
          <attvalue for="2" value="&#10;&#10;The capability to understand the meaning of noisy words through character arrangements is a crucial aspect of human cognitive abilities \cite{rawlinson2007letter}. This capability is highly sought after in practical applications such as machine translation and sentiment analysis \cite{belinkov2017synthetic}. However, despite their success in in-distribution test data with standardized word forms, Pretrained Language Models (PLMs), which serve as the backbone models, tend to perform poorly on rare or noisy words \cite{kumar-etal-2020-noisy,baron2015words}. These noisy words may be caused by accidental typos \cite{belinkov2017synthetic} or spelling variants on social media \cite{ritter-etal-2010-unsupervised}.&#10;&#10;Prior studies show that most subword-based PLMs perform poorly under noise largely due to subword segmentation \cite{zhuang2022typos}, while character-based PLMs show more robustness \cite{el-boukkouri-etal-2020-characterbert}. Examining the impact of subword segmentation factors on PLMs is also crucial for defending against the adversarial attacks that leverage the sensitivity of subword segmentation to noise \cite{liu-etal-2022-character}. However, rare work has investigated how the subword segmentation from noisy words affects the word meaning. &#10;&#10;To help address this question, we design and develop a contrastive framework (CoLes) to assess the robustness of PLMs in the face of various forms of segmentation corruption. &#10;As subword segmentation can be influenced by noise in various ways, such as adding extra subwords or losing original subwords, we systematically categorize the ways into four main categories and two additional sub-categories based on three subword sets, as exemplified in Table \ref{tab:subword_sets}. Two types of noise models are proposed to effectively generate all the types of corruption except missing corruption, and a contrastive dataset consisting of noisy and standard word pairs is created. This framework enables us to evaluate the significance of preserved subwords and the impact of subwords added by noise.&#10;&#10;The experimental results provide the following insights: 1) complete corruption: the PLMs struggle to infer meaning accurately if no subwords from the original segmentation are retained. The worst performance is observed when the meaning of original words is stored in the embedding; 2) partial corruption: preserving larger subword chunks can aid the understanding of PLMs, whereas retaining smaller subword pieces tend to be ineffective; and 3) additive corruption: even with all original subwords, however, the addition of subwords can harm the meaning of words, particularly when they are placed within other subwords. The more additive subwords, the greater the deviation in word semantics.&#10;All the results are consistent on the three PLMs with different vocabularies and segmentation algorithms.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Subword Segmentation Noise, Language Model Robustness, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1711.02173" label="1711.02173">
        <attvalues>
          <attvalue for="0" value="Synthetic and Natural Noise Both Break Neural Machine Translation" />
          <attvalue for="1" value="  Character-based neural machine translation (NMT) models alleviate&#10;out-of-vocabulary issues, learn morphology, and move us closer to completely&#10;end-to-end translation systems. Unfortunately, they are also very brittle and&#10;easily falter when presented with noisy data. In this paper, we confront NMT&#10;models with synthetic and natural sources of noise. We find that&#10;state-of-the-art models fail to translate even moderately noisy texts that&#10;humans have no trouble comprehending. We explore two approaches to increase&#10;model robustness: structure-invariant word representations and robust training&#10;on noisy texts. We find that a model based on a character convolutional neural&#10;network is able to simultaneously learn representations robust to multiple&#10;kinds of noise.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.12932" label="2003.12932">
        <attvalues>
          <attvalue for="0" value="Noisy Text Data: Achilles' Heel of BERT" />
          <attvalue for="1" value="  Owing to the phenomenal success of BERT on various NLP tasks and benchmark&#10;datasets, industry practitioners are actively experimenting with fine-tuning&#10;BERT to build NLP applications for solving industry use cases. For most&#10;datasets that are used by practitioners to build industrial NLP applications,&#10;it is hard to guarantee absence of any noise in the data. While BERT has&#10;performed exceedingly well for transferring the learnings from one use case to&#10;another, it remains unclear how BERT performs when fine-tuned on noisy text. In&#10;this work, we explore the sensitivity of BERT to noise in the data. We work&#10;with most commonly occurring noise (spelling mistakes, typos) and show that&#10;this results in significant degradation in the performance of BERT. We present&#10;experimental results to show that BERT's performance on fundamental NLP tasks&#10;like sentiment analysis and textual similarity drops significantly in the&#10;presence of (simulated) noise on benchmark datasets viz. IMDB Movie Review,&#10;STS-B, SST-2. Further, we identify shortcomings in the existing BERT pipeline&#10;that are responsible for this drop in performance. Our findings suggest that&#10;practitioners need to be vary of presence of noise in their datasets while&#10;fine-tuning BERT to solve industry use cases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.00716" label="2204.00716">
        <attvalues>
          <attvalue for="0" value="CharacterBERT and Self-Teaching for Improving the Robustness of Dense&#10;  Retrievers on Queries with Typos" />
          <attvalue for="1" value="  Current dense retrievers are not robust to out-of-domain and outlier queries,&#10;i.e. their effectiveness on these queries is much poorer than what one would&#10;expect. In this paper, we consider a specific instance of such queries: queries&#10;that contain typos. We show that a small character level perturbation in&#10;queries (as caused by typos) highly impacts the effectiveness of dense&#10;retrievers. We then demonstrate that the root cause of this resides in the&#10;input tokenization strategy employed by BERT. In BERT, tokenization is&#10;performed using the BERT's WordPiece tokenizer and we show that a token with a&#10;typo will significantly change the token distributions obtained after&#10;tokenization. This distribution change translates to changes in the input&#10;embeddings passed to the BERT-based query encoder of dense retrievers. We then&#10;turn our attention to devising dense retriever methods that are robust to such&#10;queries with typos, while still being as performant as previous methods on&#10;queries without typos. For this, we use CharacterBERT as the backbone encoder&#10;and an efficient yet effective training method, called Self-Teaching (ST), that&#10;distills knowledge from queries without typos into the queries with typos.&#10;Experimental results show that CharacterBERT in combination with ST achieves&#10;significantly higher effectiveness on queries with typos compared to previous&#10;methods. Along with these results and the open-sourced implementation of the&#10;methods, we also provide a new passage retrieval dataset consisting of&#10;real-world queries with typos and associated relevance assessments on the MS&#10;MARCO corpus, thus supporting the research community in the investigation of&#10;effective and robust dense retrievers. Code, experimental results and dataset&#10;are made available at https://github.com/ielab/CharacterBERT-DR.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.10392" label="2010.10392">
        <attvalues>
          <attvalue for="0" value="CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary&#10;  Representations From Characters" />
          <attvalue for="1" value="  Due to the compelling improvements brought by BERT, many recent&#10;representation models adopted the Transformer architecture as their main&#10;building block, consequently inheriting the wordpiece tokenization system&#10;despite it not being intrinsically linked to the notion of Transformers. While&#10;this system is thought to achieve a good balance between the flexibility of&#10;characters and the efficiency of full words, using predefined wordpiece&#10;vocabularies from the general domain is not always suitable, especially when&#10;building models for specialized domains (e.g., the medical domain). Moreover,&#10;adopting a wordpiece tokenization shifts the focus from the word level to the&#10;subword level, making the models conceptually more complex and arguably less&#10;convenient in practice. For these reasons, we propose CharacterBERT, a new&#10;variant of BERT that drops the wordpiece system altogether and uses a&#10;Character-CNN module instead to represent entire words by consulting their&#10;characters. We show that this new model improves the performance of BERT on a&#10;variety of medical domain tasks while at the same time producing robust,&#10;word-level and open-vocabulary representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.17004" label="2210.17004">
        <attvalues>
          <attvalue for="0" value="Character-level White-Box Adversarial Attacks against Transformers via&#10;  Attachable Subwords Substitution" />
          <attvalue for="1" value="  We propose the first character-level white-box adversarial attack method&#10;against transformer models. The intuition of our method comes from the&#10;observation that words are split into subtokens before being fed into the&#10;transformer models and the substitution between two close subtokens has a&#10;similar effect to the character modification. Our method mainly contains three&#10;steps. First, a gradient-based method is adopted to find the most vulnerable&#10;words in the sentence. Then we split the selected words into subtokens to&#10;replace the origin tokenization result from the transformer tokenizer. Finally,&#10;we utilize an adversarial loss to guide the substitution of attachable&#10;subtokens in which the Gumbel-softmax trick is introduced to ensure gradient&#10;propagation. Meanwhile, we introduce the visual and length constraint in the&#10;optimization process to achieve minimum character modifications. Extensive&#10;experiments on both sentence-level and token-level tasks demonstrate that our&#10;method could outperform the previous attack methods in terms of success rate&#10;and edit distance. Furthermore, human evaluation verifies our adversarial&#10;examples could preserve their origin labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.15043" label="2402.15043">
        <attvalues>
          <attvalue for="0" value="KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large&#10;  Language Models" />
          <attvalue for="1" value="  Automatic evaluation methods for large language models (LLMs) are hindered by&#10;data contamination, leading to inflated assessments of their effectiveness.&#10;Existing strategies, which aim to detect contaminated texts, focus on&#10;quantifying contamination status instead of accurately gauging model&#10;performance. In this paper, we introduce KIEval, a Knowledge-grounded&#10;Interactive Evaluation framework, which incorporates an LLM-powered&#10;&quot;interactor&quot; role for the first time to accomplish a dynamic&#10;contamination-resilient evaluation. Starting with a question in a conventional&#10;LLM benchmark involving domain-specific knowledge, KIEval utilizes dynamically&#10;generated, multi-round, and knowledge-focused dialogues to determine whether a&#10;model's response is merely a recall of benchmark answers or demonstrates a deep&#10;comprehension to apply knowledge in more complex conversations. Extensive&#10;experiments on seven leading LLMs across five datasets validate KIEval's&#10;effectiveness and generalization. We also reveal that data contamination brings&#10;no contribution or even negative effect to models' real-world applicability and&#10;understanding, and existing contamination detection methods for LLMs can only&#10;identify contamination in pre-training but not during supervised fine-tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;\nocite{*}&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Knowledge Grounded Dialogue, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Data Contamination Detection, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2111.00191" label="2111.00191">
        <attvalues>
          <attvalue for="0" value="How should human translation coexist with NMT? Efficient tool for&#10;  building high quality parallel corpus" />
          <attvalue for="1" value="  This paper proposes a tool for efficiently constructing high-quality parallel&#10;corpora with minimizing human labor and making this tool publicly available.&#10;Our proposed construction process is based on neural machine translation (NMT)&#10;to allow for it to not only coexist with human translation, but also improve&#10;its efficiency by combining data quality control with human translation in a&#10;data-centric approach.&#10;" />
          <attvalue for="2" value="&#10;Building a high-quality parallel corpus, which has its target sentence precisely translates the source sentence, vice versa, is a common important issue in the entire field of machine translation. Unfortunately, obtaining a high-quality parallel corpus is difficult for many reasons, including problems of copyright acquisition, the difficulty of finding the proper alignment, and high monetary and temporal costs of building the corpus~\cite{koehn2020findings}. Human translation is fundamentally the most trusted approach for improving data quality, and it can construct a high-quality parallel corpus~\cite{hutchins2001machine,rojo2018aspects}. However, even this approach is limited, as it requires a tremendous amount of money and time for humans to manually construct the entire corpus. To alleviate this limitation, we present a novel tool for constructing high-quality parallel corpora using only simple mono corpus. &#10;&#10;In detail, we divided the generic process of constructing a high-quality parallel corpus into two components. First, a data advancing automation approach is employed to advance the source ({\em{i.e.,}} the initial mono corpus) language using corpus filtering~\cite{herold2021data} and Grammar Error Correction (GEC)~\cite{wang2020comprehensive}. Subsequently, the quality of the translated target is also improved by both the advanced mono corpus and the process of Automatic Post Editing (APE)~\cite{chatterjee2019findings}. Second, we utilize both the predicting automation approach ({\em{i.e.,}} predictions of data quality) and human translation to minimize the human labor required to complete the task. The predicting automation approach employs Quality Estimation~\cite{fonseca2019findings} to predict the sentence quality of the parallel corpus, and its labels are used to measure the human labor cost.&#10;&#10;Thus, the limitations associated with generating a parallel corpus can be alleviated if the computer automatically determines the quality of the corpus according to a specified level of threshold control. That is, human labor is unnecessary when the threshold is exceeded, although if threshold is not exceeded then it requires refinement, which is the verification and post-processing of the corpus conducted by humans. Overall, our approach contributes to the human translation market and to the automated machine translation field by improving efficiency and minimizing cost.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human Translation Efficiency, Computer Science, Linguistics, Parallel Corpus Construction, Artificial Intelligence, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2005.06600" label="2005.06600">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey of Grammar Error Correction" />
          <attvalue for="1" value="  Grammar error correction (GEC) is an important application aspect of natural&#10;language processing techniques. The past decade has witnessed significant&#10;progress achieved in GEC for the sake of increasing popularity of machine&#10;learning and deep learning, especially in late 2010s when near human-level GEC&#10;systems are available. However, there is no prior work focusing on the whole&#10;recapitulation of the progress. We present the first survey in GEC for a&#10;comprehensive retrospect of the literature in this area. We first give the&#10;introduction of five public datasets, data annotation schema, two important&#10;shared tasks and four standard evaluation metrics. More importantly, we discuss&#10;four kinds of basic approaches, including statistical machine translation based&#10;approach, neural machine translation based approach, classification based&#10;approach and language model based approach, six commonly applied performance&#10;boosting techniques for GEC systems and two data augmentation methods. Since&#10;GEC is typically viewed as a sister task of machine translation, many GEC&#10;systems are based on neural machine translation (NMT) approaches, where the&#10;neural sequence-to-sequence model is applied. Similarly, some performance&#10;boosting techniques are adapted from machine translation and are successfully&#10;combined with GEC systems for enhancement on the final performance.&#10;Furthermore, we conduct an analysis in level of basic approaches, performance&#10;boosting techniques and integrated GEC systems based on their experiment&#10;results respectively for more clear patterns and conclusions. Finally, we&#10;discuss five prospective directions for future GEC researches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.03723" label="1910.03723">
        <attvalues>
          <attvalue for="0" value="Knowledge Distillation from Internal Representations" />
          <attvalue for="1" value="  Knowledge distillation is typically conducted by training a small model (the&#10;student) to mimic a large and cumbersome model (the teacher). The idea is to&#10;compress the knowledge from the teacher by using its output probabilities as&#10;soft-labels to optimize the student. However, when the teacher is considerably&#10;large, there is no guarantee that the internal knowledge of the teacher will be&#10;transferred into the student; even if the student closely matches the&#10;soft-labels, its internal representations may be considerably different. This&#10;internal mismatch can undermine the generalization capabilities originally&#10;intended to be transferred from the teacher to the student. In this paper, we&#10;propose to distill the internal representations of a large model such as BERT&#10;into a simplified version of it. We formulate two ways to distill such&#10;representations and various algorithms to conduct the distillation. We&#10;experiment with datasets from the GLUE benchmark and consistently show that&#10;adding knowledge distillation from internal representations is a more powerful&#10;method than only using soft-label distillation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.11928" label="2104.11928">
        <attvalues>
          <attvalue for="0" value="Extract then Distill: Efficient and Effective Task-Agnostic BERT&#10;  Distillation" />
          <attvalue for="1" value="  Task-agnostic knowledge distillation, a teacher-student framework, has been&#10;proved effective for BERT compression. Although achieving promising results on&#10;NLP tasks, it requires enormous computational resources. In this paper, we&#10;propose Extract Then Distill (ETD), a generic and flexible strategy to reuse&#10;the teacher's parameters for efficient and effective task-agnostic&#10;distillation, which can be applied to students of any size. Specifically, we&#10;introduce two variants of ETD, ETD-Rand and ETD-Impt, which extract the&#10;teacher's parameters in a random manner and by following an importance metric&#10;respectively. In this way, the student has already acquired some knowledge at&#10;the beginning of the distillation process, which makes the distillation process&#10;converge faster. We demonstrate the effectiveness of ETD on the GLUE benchmark&#10;and SQuAD. The experimental results show that: (1) compared with the baseline&#10;without an ETD strategy, ETD can save 70\% of computation cost. Moreover, it&#10;achieves better results than the baseline when using the same computing&#10;resource. (2) ETD is generic and has been proven effective for different&#10;distillation methods (e.g., TinyBERT and MiniLM) and students of different&#10;sizes. The source code will be publicly available upon publication.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.09149" label="1904.09149">
        <attvalues>
          <attvalue for="0" value="Knowledge Distillation via Route Constrained Optimization" />
          <attvalue for="1" value="  Distillation-based learning boosts the performance of the miniaturized neural&#10;network based on the hypothesis that the representation of a teacher model can&#10;be used as structured and relatively weak supervision, and thus would be easily&#10;learned by a miniaturized model. However, we find that the representation of a&#10;converged heavy model is still a strong constraint for training a small student&#10;model, which leads to a high lower bound of congruence loss. In this work,&#10;inspired by curriculum learning we consider the knowledge distillation from the&#10;perspective of curriculum learning by routing. Instead of supervising the&#10;student model with a converged teacher model, we supervised it with some anchor&#10;points selected from the route in parameter space that the teacher model passed&#10;by, as we called route constrained optimization (RCO). We experimentally&#10;demonstrate this simple operation greatly reduces the lower bound of congruence&#10;loss for knowledge distillation, hint and mimicking learning. On close-set&#10;classification tasks like CIFAR100 and ImageNet, RCO improves knowledge&#10;distillation by 2.14% and 1.5% respectively. For the sake of evaluating the&#10;generalization, we also test RCO on the open-set face recognition task&#10;MegaFace.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.01348" label="1910.01348">
        <attvalues>
          <attvalue for="0" value="On the Efficacy of Knowledge Distillation" />
          <attvalue for="1" value="  In this paper, we present a thorough evaluation of the efficacy of knowledge&#10;distillation and its dependence on student and teacher architectures. Starting&#10;with the observation that more accurate teachers often don't make good&#10;teachers, we attempt to tease apart the factors that affect knowledge&#10;distillation performance. We find crucially that larger models do not often&#10;make better teachers. We show that this is a consequence of mismatched&#10;capacity, and that small students are unable to mimic large teachers. We find&#10;typical ways of circumventing this (such as performing a sequence of knowledge&#10;distillation steps) to be ineffective. Finally, we show that this effect can be&#10;mitigated by stopping the teacher's training early. Our results generalize&#10;across datasets and models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.11295" label="2109.11295">
        <attvalues>
          <attvalue for="0" value="Dynamic Knowledge Distillation for Pre-trained Language Models" />
          <attvalue for="1" value="  Knowledge distillation~(KD) has been proved effective for compressing&#10;large-scale pre-trained language models. However, existing methods conduct KD&#10;statically, e.g., the student model aligns its output distribution to that of a&#10;selected teacher model on the pre-defined training dataset. In this paper, we&#10;explore whether a dynamic knowledge distillation that empowers the student to&#10;adjust the learning procedure according to its competency, regarding the&#10;student performance and learning efficiency. We explore the dynamical&#10;adjustments on three aspects: teacher model adoption, data selection, and KD&#10;objective adaptation. Experimental results show that (1) proper selection of&#10;teacher model can boost the performance of student model; (2) conducting KD&#10;with 10% informative instances achieves comparable performance while greatly&#10;accelerates the training; (3) the student performance can be boosted by&#10;adjusting the supervision contribution of different alignment objective. We&#10;find dynamic knowledge distillation is promising and provide discussions on&#10;potential future directions towards more efficient KD methods. Our code is&#10;available at https://github.com/lancopku/DynamicKD.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.12836" label="1810.12836">
        <attvalues>
          <attvalue for="0" value="Learning Cross-Lingual Sentence Representations via a Multi-task&#10;  Dual-Encoder Model" />
          <attvalue for="1" value="  A significant roadblock in multilingual neural language modeling is the lack&#10;of labeled non-English data. One potential method for overcoming this issue is&#10;learning cross-lingual text representations that can be used to transfer the&#10;performance from training on English tasks to non-English tasks, despite little&#10;to no task-specific non-English data. In this paper, we explore a natural setup&#10;for learning cross-lingual sentence representations: the dual-encoder. We&#10;provide a comprehensive evaluation of our cross-lingual representations on a&#10;number of monolingual, cross-lingual, and zero-shot/few-shot learning tasks,&#10;and also give an analysis of different learned cross-lingual embedding spaces.&#10;" />
          <attvalue for="2" value="&#10;&#10;We present a novel approach for cross-lingual representation learning that combines methods for multi-task learning of monolingual sentence representations~\cite{unec2018,subramanian18} with recent work on dual encoder methods for obtaining multilingual sentence representations for bi-text retrieval~\cite{Guo2018,yang2019}. By doing so, we learn representations that maintain strong performance on the original monolingual language tasks, while simultaneously obtaining good performance using zero-shot learning on the same task in another language. For a given language pair, we construct a multi-task training scheme using native source language tasks, native target language tasks, and a bridging translation task to encourage sentences with identical meanings, but written in different languages, to have similar embeddings.&#10;&#10;We evaluate the learned representations on several monolingual and cross-lingual tasks, and provide a graph-based analysis of the learned representations. Multi-task training using additional monolingual tasks is found to improve performance over models that only make use of parallel data on both cross-lingual semantic textual similarity (STS)~\cite{cer-EtAl:2017:SemEval} and cross-lingual eigen-similarity~\cite{sogard2018}. For European languages, the results show that the addition of monolingual data improves the embedding alignment of sentences and their translations. Further, we find that cross-lingual training with additional monolingual data leads to far better cross-lingual transfer learning performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Modeling, Cross-Lingual Transfer, Artificial Intelligence, Neural Language Representation" />
        </attvalues>
      </node>
      <node id="1807.11906" label="1807.11906">
        <attvalues>
          <attvalue for="0" value="Effective Parallel Corpus Mining using Bilingual Sentence Embeddings" />
          <attvalue for="1" value="  This paper presents an effective approach for parallel corpus mining using&#10;bilingual sentence embeddings. Our embedding models are trained to produce&#10;similar representations exclusively for bilingual sentence pairs that are&#10;translations of each other. This is achieved using a novel training method that&#10;introduces hard negatives consisting of sentences that are not translations but&#10;that have some degree of semantic similarity. The quality of the resulting&#10;embeddings are evaluated on parallel corpus reconstruction and by assessing&#10;machine translation systems trained on gold vs. mined sentence pairs. We find&#10;that the sentence embeddings can be used to reconstruct the United Nations&#10;Parallel Corpus at the sentence level with a precision of 48.9% for en-fr and&#10;54.9% for en-es. When adapted to document level matching, we achieve a parallel&#10;document matching accuracy that is comparable to the significantly more&#10;computationally intensive approach of [Jakob 2010]. Using reconstructed&#10;parallel data, we are able to train NMT models that perform nearly as well as&#10;models trained on the original data (within 1-2 BLEU).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.08564" label="1902.08564">
        <attvalues>
          <attvalue for="0" value="Improving Multilingual Sentence Embedding using Bi-directional Dual&#10;  Encoder with Additive Margin Softmax" />
          <attvalue for="1" value="  In this paper, we present an approach to learn multilingual sentence&#10;embeddings using a bi-directional dual-encoder with additive margin softmax.&#10;The embeddings are able to achieve state-of-the-art results on the United&#10;Nations (UN) parallel corpus retrieval task. In all the languages tested, the&#10;system achieves P@1 of 86% or higher. We use pairs retrieved by our approach to&#10;train NMT models that achieve similar performance to models trained on gold&#10;pairs. We explore simple document-level embeddings constructed by averaging our&#10;sentence embeddings. On the UN document-level retrieval task, document&#10;embeddings achieve around 97% on P@1 for all experimented language pairs.&#10;Lastly, we evaluate the proposed model on the BUCC mining task. The learned&#10;embeddings with raw cosine similarity scores achieve competitive results&#10;compared to current state-of-the-art models, and with a second-stage scorer we&#10;achieve a new state-of-the-art level on this task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.12711" label="2209.12711">
        <attvalues>
          <attvalue for="0" value="Can Large Language Models Truly Understand Prompts? A Case Study with&#10;  Negated Prompts" />
          <attvalue for="1" value="  Previous work has shown that there exists a scaling law between the size of&#10;Language Models (LMs) and their zero-shot performance on different downstream&#10;NLP tasks. In this work, we show that this phenomenon does not hold when&#10;evaluating large LMs on tasks with negated prompts, but instead shows an&#10;inverse scaling law. We evaluate 9 different tasks with negated prompts on (1)&#10;pretrained LMs (OPT &amp; GPT-3) of varying sizes (125M - 175B), (2) LMs further&#10;pretrained to generalize to novel prompts (InstructGPT), (3) LMs provided with&#10;few-shot examples, and (4) LMs fine-tuned specifically on negated prompts; all&#10;LM types perform worse on negated prompts as they scale and show a huge&#10;performance gap between the human performance when comparing the average score&#10;on both original and negated prompts. By highlighting a critical limitation of&#10;existing LMs and methods, we urge the community to develop new approaches of&#10;developing LMs that actually follow the given instructions. We provide the code&#10;and the datasets to explore negated prompts at&#10;https://github.com/joeljang/negated-prompts-for-llms&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LMs) pretrained on a vast amounts of corpora have shown surprising, even emergent, capabilities of solving various downstream tasks through prompts (instructions)~\cite{brown2020language, rae2021scaling, chowdhery2022palm, zhang2022opt, wei2022emergent}. Previous work has specifically shown LMs can perform unseen tasks through multitask fine-tuning on various downstream tasks with prompts~\cite{sanh2021multitask, wei2021finetuned, wang2022benchmarking, ouyang2022training}. A 540B LM~\cite{chowdhery2022palm} has even shown the capability to act as the ``brain'' for actual robots, helping them perform different tasks in the real-world~\cite{ahn2022can}. As LMs are trained to become more aligned with human values, perform real-world tasks, and are endowed with responsibilities that may result in real-world consequences, it is more-so important to ensure that LMs actually do what they are instructed to do.&#10;&#10;In this work, we test the capabilities of Language Models (LMs) on truly following the given instructions (prompts) by conducting a case study with negated instructions; that is, telling the LM NOT to do something as shown by an example in Figure \ref{fig:example}. Prior work~\cite{ettinger2020bert, webson2021prompt} has shown that LMs (as well as other large pretrained models in different modalities such as DALLE-2~\cite{ramesh2022hierarchical}) have a hard time understanding negated prompts and perform the task as if provided with the original prompt. For example, if we prompt DALLE-2 this prompt: ``Do not generate a monkey holding a banana'', it will generate an image with a monkey holding a banana. We hypothesize this is due to the syntax of the negated prompts being rarely seen or out-of-distribution during initial pretraining or during the adaptation phase.&#10;&#10;We aim to answer four main questions in this work. (1) How does scaling the size of LMs affect their abilities to understand the concept of negation? (2) Are LMs explicitly trained to follow instructions (InstructGPT) better at understanding negated instructions? (3) Can In-Context Learning (ICL) or Fine-tuning (FT) help mitigate this problem? (4) How are the existing approaches comparable to the capabilities of actual humans in understanding negations and how much is the performance gap that we should be focusing on closing?&#10;&#10;The answers to the questions above can be summarized as follows:&#10;\begin{itemize}&#10;&#10;\item Results show scaling to be inefficient at helping LMs understand negation. On the contrary, LMs perform worse as they scale.&#10;&#10;\item LMs specifically adapted to generalize to novel instructions still suffer from understanding negation, despite showing a bit of an improvement.&#10;&#10;\item ICL helps LMs understand negation in only specific scenarios, while FT seems to help in all scenarios. However, FT results in degradation of the original task performance, resulting in a zero-sum game. &#10;&#10;\item Comparing the existing approaches with the human performance measured by asking 13-year-old humans to do the same task given both the original and negated prompts, we show that there is a huge ($\sim$31.3\%) performance gap to close.&#10;&#10;\end{itemize} &#10;&#10;Through this work, we aim to highlight a critical shortcoming of large LMs that should be carefully considered before empowering them with responsibilities that might result in real-world consequences.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Limitations, Negated Prompt Evaluation, Large Model Performance" />
        </attvalues>
      </node>
      <node id="2204.01691" label="2204.01691">
        <attvalues>
          <attvalue for="0" value="Do As I Can, Not As I Say: Grounding Language in Robotic Affordances" />
          <attvalue for="1" value="  Large language models can encode a wealth of semantic knowledge about the&#10;world. Such knowledge could be extremely useful to robots aiming to act upon&#10;high-level, temporally extended instructions expressed in natural language.&#10;However, a significant weakness of language models is that they lack real-world&#10;experience, which makes it difficult to leverage them for decision making&#10;within a given embodiment. For example, asking a language model to describe how&#10;to clean a spill might result in a reasonable narrative, but it may not be&#10;applicable to a particular agent, such as a robot, that needs to perform this&#10;task in a particular environment. We propose to provide real-world grounding by&#10;means of pretrained skills, which are used to constrain the model to propose&#10;natural language actions that are both feasible and contextually appropriate.&#10;The robot can act as the language model's &quot;hands and eyes,&quot; while the language&#10;model supplies high-level semantic knowledge about the task. We show how&#10;low-level skills can be combined with large language models so that the&#10;language model provides high-level knowledge about the procedures for&#10;performing complex and temporally-extended instructions, while value functions&#10;associated with these skills provide the grounding necessary to connect this&#10;knowledge to a particular physical environment. We evaluate our method on a&#10;number of real-world robotic tasks, where we show the need for real-world&#10;grounding and that this approach is capable of completing long-horizon,&#10;abstract, natural language instructions on a mobile manipulator. The project's&#10;website and the video can be found at https://say-can.github.io/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.05731" label="1908.05731">
        <attvalues>
          <attvalue for="0" value="Simple and Effective Noisy Channel Modeling for Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Previous work on neural noisy channel modeling relied on latent variable&#10;models that incrementally process the source and target sentence. This makes&#10;decoding decisions based on partial source prefixes even though the full source&#10;is available. We pursue an alternative approach based on standard sequence to&#10;sequence models which utilize the entire source. These models perform&#10;remarkably well as channel models, even though they have neither been trained&#10;on, nor designed to factor over incomplete target sentences. Experiments with&#10;neural language models trained on billions of words show that noisy channel&#10;models can outperform a direct model by up to 3.2 BLEU on WMT'17 German-English&#10;translation. We evaluate on four language-pairs and our channel models&#10;consistently outperform strong alternatives such right-to-left reranking models&#10;and ensembles of direct models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Sequence to sequence models directly estimate the posterior probability of a target sequence $y$ given a source sequence $x$~\cite{sutskever2014sequence,bahdanau2015neural,gehring2017convs2s,vaswani2017transformer} and can be trained with pairs of source and target sequences.&#10;Unpaired sequences can be leveraged by data augmentation schemes such as back-translation, but direct models cannot naturally take advantage of unpaired data~\cite{sennrich2016bt,edunov2018bt}.&#10;&#10;The noisy channel approach is an alternative which is used in statistical machine translation~\cite{brown1993mathematics,koehn2003statistical}.&#10;It entails a channel model probability $p(x|y)$ that operates in the reverse direction as well as a language model probability $p(y)$.&#10;The language model can be estimated on unpaired data and can take a separate form to the channel model. &#10;Noisy channel modeling mitigates explaining away effects that result in the source being ignored for highly likely output prefixes~\cite{klein2001explain}.&#10;&#10;Previous work on neural noisy channel modeling relied on a complex latent variable model that incrementally processes source and target prefixes~\cite{yu2017neuralnoisy}.&#10;This trades efficiency for accuracy because their model performs significantly less well than a vanilla sequence to sequence model.&#10;For languages with similar word order, it can be sufficient to predict the first target token based on a short source prefix, but for languages where word order differs significantly, we may need to take the entire source sentence into account to make a decision.&#10;&#10;In this paper, we show that a standard sequence to sequence model is an effective parameterization of the channel probability.&#10;We train the model on full sentences and apply it to score the source given an incomplete target sentence.&#10;This bases decoding decisions on scoring the entire source sequence and it is very simple and effective (\textsection\ref{sec:approach}).&#10;We analyze this approach for various target prefix sizes and find that it is most accurate for large target context sizes. &#10;Our simple noisy channel approach consistently outperforms strong baselines such as online ensembles and left-to-right re-ranking setups (\textsection\ref{sec:experiments}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Sequence to Sequence Models, Linguistics, Neural Channel Modeling, Machine Translation Evaluation, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1611.02554" label="1611.02554">
        <attvalues>
          <attvalue for="0" value="The Neural Noisy Channel" />
          <attvalue for="1" value="  We formulate sequence to sequence transduction as a noisy channel decoding&#10;problem and use recurrent neural networks to parameterise the source and&#10;channel models. Unlike direct models which can suffer from explaining-away&#10;effects during training, noisy channel models must produce outputs that explain&#10;their inputs, and their component models can be trained with not only paired&#10;training samples but also unpaired samples from the marginal output&#10;distribution. Using a latent variable to control how much of the conditioning&#10;sequence the channel model needs to read in order to generate a subsequent&#10;symbol, we obtain a tractable and effective beam search decoder. Experimental&#10;results on abstractive sentence summarisation, morphological inflection, and&#10;machine translation show that noisy channel models outperform direct models,&#10;and that they significantly benefit from increased amounts of unpaired output&#10;data that direct models cannot easily use.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.04289" label="2406.04289">
        <attvalues>
          <attvalue for="0" value="What Languages are Easy to Language-Model? A Perspective from Learning&#10;  Probabilistic Regular Languages" />
          <attvalue for="1" value="  What can large language models learn? By definition, language models (LM) are&#10;distributions over strings. Therefore, an intuitive way of addressing the above&#10;question is to formalize it as a matter of learnability of classes of&#10;distributions over strings. While prior work in this direction focused on&#10;assessing the theoretical limits, in contrast, we seek to understand the&#10;empirical learnability. Unlike prior empirical work, we evaluate neural LMs on&#10;their home turf-learning probabilistic languages-rather than as classifiers of&#10;formal languages. In particular, we investigate the learnability of regular LMs&#10;(RLMs) by RNN and Transformer LMs. We empirically test the learnability of RLMs&#10;as a function of various complexity parameters of the RLM and the hidden state&#10;size of the neural LM. We find that the RLM rank, which corresponds to the size&#10;of linear space spanned by the logits of its conditional distributions, and the&#10;expected length of sampled strings are strong and significant predictors of&#10;learnability for both RNNs and Transformers. Several other predictors also&#10;reach significance, but with differing patterns between RNNs and Transformers.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models are, definitionally, distributions over strings.&#10;However, not all neural LMs are capable of learning---or even representing---all possible distributions.&#10;This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples?&#10;In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}.&#10;Moreover, the prevalence of Transformer-based LMs has led to a recent body of work investigating their representational capacity \cite[e.g.,][]{hahn-2020-theoretical,ebrahimi-etal-2020-self,bhattamishra-etal-2020-ability,merrill-sabharwal-2023-parallelism}.&#10;However, almost all of this work is theoretical, i.e., researchers seek theorems that give exact limitations on the capacity of specific neural LMs.&#10;While such work provides a good characterization of what neural LMs could, in principle, learn, it does not speak to what LMs can learn in practice.\looseness=-1&#10;&#10;In contrast to the above, a more empirically minded researcher might prefer to run a series of controlled experiments to determine what neural LMs can and cannot learn.&#10;Their goal is to empirically characterize what classes of formal LMs, e.g., probabilistic finite-state automata, are learnable with neural LMs in practice, using current best practices.&#10;Such work can inform our understanding of what types of languages larger LMs trained on human-written text might represent---specifically, what grammatical structures they can recognize, and how efficiently they can do so.&#10;All of the above is crucial for quantifying the practical capabilities, and limits, of neural LMs.&#10;While plenty of empirical work has provided insights into the linguistic capabilities of modern LMs using linguistically annotated datasets \cite[e.g., ][]{10.1162/tacl-a-00115,hewitt-manning-2019-structural,jawahar-etal-2019-bert,liu-etal-2019-linguistic,ICARD2020102308,doi:10.1073/pnas.1907367117,10.1162/tacl-a-00349,belinkov-2022-probing}, human-annotated datasets give us limited insight into the types of distributions a neural LM can learn because the true distribution that the neural LMs is modeling is often unclear.&#10;For instance, fitting an LM to Wikipedia leaves open to interpretation exactly which probability distribution over strings the neural LM is modeling.&#10;In contrast, learning a probabilistic formal language in a controlled experiment offers an unparalleled level of control.\looseness-1&#10;&#10;A closer look at the existing work on the empirical learnability of formal languages (see \ref{app:related_work} for an overview) reveals a categorical mismatch between what LMs are, i.e., probability distributions over strings, and what learning a formal language means, i.e., classifying strings as members of a specific language, i.e., a set of strings \cite{ebrahimi-etal-2020-self,deletang2023neural,wang-steinert-threlkeld-2023-evaluating}.&#10;In response, we propose to investigate the practical representation capacity of neural LMs by testing their ability to learn distributions over strings.&#10;By sampling training corpora from probabilistic finite-state automata (\pfsaAcr), we can ask precise questions about the learnability.&#10;&#10;We offer an empirical study, sampling datasets of 20k strings from 2100 randomly generated \pfsaAcr{s}, and training 15k RNN and Transformer language models with a varying hidden state size on datasets.&#10;Our study is informed by various theoretical results on the representational capacity of RNNs concerning probabilistic finite-state automata. &#10;We assess the learnability by approximating the KL divergence between \pfsaAcr{s} and neural LMs.&#10;In a regression analysis, we find that a large number of properties of the automaton, e.g., the number of states, the number of transitions, the rank of its emission matrix, and its entropy all contribute to learnability. &#10;In particular, the rank of the emission matrix and expected length of the sampled strings are strong predictors of learnability for both RNN and Transformer LMs. Several other predictors also demonstrate relevance in accounting for learnability, but have differing magnitudes and patterns of significance for the RNN and Transformer LMs. &#10;Similar to \cite{deletang2023neural}, we find that RNNs are better suited to modeling formal languages than Transformers.\looseness=-1&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Learnability, Computer Science, Probabilistic Language Analysis, Linguistics, Neural Network Evaluation, Mathematics" />
        </attvalues>
      </node>
      <node id="1906.06349" label="1906.06349">
        <attvalues>
          <attvalue for="0" value="On the Computational Power of RNNs" />
          <attvalue for="1" value="  Recent neural network architectures such as the basic recurrent neural&#10;network (RNN) and Gated Recurrent Unit (GRU) have gained prominence as&#10;end-to-end learning architectures for natural language processing tasks. But&#10;what is the computational power of such systems? We prove that finite precision&#10;RNNs with one hidden layer and ReLU activation and finite precision GRUs are&#10;exactly as computationally powerful as deterministic finite automata. Allowing&#10;arbitrary precision, we prove that RNNs with one hidden layer and ReLU&#10;activation are at least as computationally powerful as pushdown automata. If we&#10;also allow infinite precision, infinite edge weights, and nonlinear output&#10;activation functions, we prove that GRUs are at least as computationally&#10;powerful as pushdown automata. All results are shown constructively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.07515" label="2010.07515">
        <attvalues>
          <attvalue for="0" value="RNNs can generate bounded hierarchical languages with optimal memory" />
          <attvalue for="1" value="  Recurrent neural networks empirically generate natural language with high&#10;syntactic fidelity. However, their success is not well-understood&#10;theoretically. We provide theoretical insight into this success, proving in a&#10;finite-precision setting that RNNs can efficiently generate bounded&#10;hierarchical languages that reflect the scaffolding of natural language syntax.&#10;We introduce Dyck-($k$,$m$), the language of well-nested brackets (of $k$&#10;types) and $m$-bounded nesting depth, reflecting the bounded memory needs and&#10;long-distance dependencies of natural language syntax. The best known results&#10;use $O(k^{\frac{m}{2}})$ memory (hidden units) to generate these languages. We&#10;prove that an RNN with $O(m \log k)$ hidden units suffices, an exponential&#10;reduction in memory, by an explicit construction. Finally, we show that no&#10;algorithm, even with unbounded computation, can suffice with $o(m \log k)$&#10;hidden units.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.04303" label="2010.04303">
        <attvalues>
          <attvalue for="0" value="How Can Self-Attention Networks Recognize Dyck-n Languages?" />
          <attvalue for="1" value="  We focus on the recognition of Dyck-n ($\mathcal{D}_n$) languages with&#10;self-attention (SA) networks, which has been deemed to be a difficult task for&#10;these networks. We compare the performance of two variants of SA, one with a&#10;starting symbol (SA$^+$) and one without (SA$^-$). Our results show that SA$^+$&#10;is able to generalize to longer sequences and deeper dependencies. For&#10;$\mathcal{D}_2$, we find that SA$^-$ completely breaks down on long sequences&#10;whereas the accuracy of SA$^+$ is 58.82$\%$. We find attention maps learned by&#10;$\text{SA}{^+}$ to be amenable to interpretation and compatible with a&#10;stack-based language recognizer. Surprisingly, the performance of SA networks&#10;is at par with LSTMs, which provides evidence on the ability of SA to learn&#10;hierarchies without recursion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.02053" label="2307.02053">
        <attvalues>
          <attvalue for="0" value="Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN&#10;  Fine-Tuning" />
          <attvalue for="1" value="  Recently, the release of INSTRUCTEVAL has provided valuable insights into the&#10;performance of large language models (LLMs) that utilize encoder-decoder or&#10;decoder-only architecture. Interestingly, despite being introduced four years&#10;ago, T5-based LLMs, such as FLAN-T5, continue to outperform the latest&#10;decoder-based LLMs, such as LLAMA and VICUNA, on tasks that require general&#10;problem-solving skills. This performance discrepancy can be attributed to three&#10;key factors: (1) Pre-training data, (2) Backbone architecture, and (3)&#10;Instruction dataset. In this technical report, our main focus is on&#10;investigating the impact of the third factor by leveraging VICUNA, a large&#10;language model based on LLAMA, which has undergone fine-tuning on ChatGPT&#10;conversations. To achieve this objective, we fine-tuned VICUNA using a&#10;customized instruction dataset collection called FLANMINI. This collection&#10;includes a subset of the large-scale instruction dataset known as FLAN, as well&#10;as various code-related datasets and conversational datasets derived from&#10;ChatGPT/GPT-4. This dataset comprises a large number of tasks that demand&#10;problem-solving skills. Our experimental findings strongly indicate that the&#10;enhanced problem-solving abilities of our model, FLACUNA, are obtained through&#10;fine-tuning VICUNA on the FLAN dataset, leading to significant improvements&#10;across numerous benchmark datasets in INSTRUCTEVAL. FLACUNA is publicly&#10;available at https://huggingface.co/declare-lab/flacuna-13b-v1.0.&#10;" />
          <attvalue for="2" value="&#10;&#10;ChatGPT and its successor GPT-4 have surpassed their prior state-of-the-art models on a vast majority of the benchmarking tasks and datasets. However, to preserve privacy, natively running a 175B+ sized model like GPT-3 is beyond the capabilities of most organizations, let alone individuals. This has prompted many researchers to fine-tune manageable-sized LLMs --- from 7B to 30B on a diverse set of &#10;instruction examples&#10;generated by ChatGPT or GPT-4. This has birthed LLMs, such as, Alpaca~\cite{alpaca} and Vicuna~\cite{vicuna2023} that are fine-tuned checkpoints of LLaMA~\cite{llama}. These models have attained close to ChatGPT-level performance on some specific benchmarking tasks, but overall generalization still remains elusive. Recent works like InstructEval~\cite{instructeval} strongly hint that the fine-tuning datasets dictate the task-specific performances. For instance, it has been observed that Flan-T5 --- a T5 checkpoint fine-tuned on Flan Collection instruction dataset --- outperforms Vicuna and Alpaca on tasks involving strong reasoning and problem-solving skills. This spurred us to fine-tune Vicuna on Flan-mini Collection dataset, anticipating improvement on reasoning-intensive tasks in InstructEval~\cite{instructeval}.&#10;&#10;To this end, we first sample a 1M-sized instruction dataset from the 15M-sized Flan Collection dataset~\cite{longpre2023flan} and combined it with several other datasets comprising coding tasks and ChatGPT/GPT-4 distilled conversations. The resulting smaller dataset, Flan-mini, is then cast into the conversational format of Vicuna. &#10;To ensure a reasonable computational cost for the fine-tuning process,&#10;we retrofit LoRA~\cite{hu2021lora} adapter into the LLaMA~\cite{llama} decoder-transformer of Vicuna. Following a parameter-efficient LoRA fine-tuning of the Vicuna checkpoint on Flan-mini, we obtain Flacuna. As expected, Flacuna outperforms Vicuna by a substantial margin on most benchmark datasets, especially for reasoning-intensive tasks. However, the performance of Flacuna still remains below Flan-T5 on the same reasoning benchmarks. This could be attributed to the 15-times smaller dataset of the instruction dataset which may contain less diverse samples. Furthermore, full fine-tuning of Vicuna may narrow the gap with Flan-T5.&#10;&#10;This work overall has the following contributions:&#10;\begin{enumerate}[leftmargin=*]&#10; \item Improving the problem-solving capability of Vicuna through parameter efficient fine-tuning on Flan-mini.&#10; \item Introducing an instruction tuning dataset, Flan-mini, comprising a diverse set of tasks and templates.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Large-Scale Dataset Evaluation, Artificial Intelligence, Language Model Performance, Problem-Solving Abilities" />
        </attvalues>
      </node>
      <node id="2306.04757" label="2306.04757">
        <attvalues>
          <attvalue for="0" value="INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large&#10;  Language Models" />
          <attvalue for="1" value="  Instruction-tuned large language models have revolutionized natural language&#10;processing and have shown great potential in applications such as&#10;conversational agents. These models, such as GPT-4, can not only master&#10;language but also solve complex tasks in areas like mathematics, coding,&#10;medicine, and law. Despite their impressive capabilities, there is still a lack&#10;of comprehensive understanding regarding their full potential, primarily due to&#10;the black-box nature of many models and the absence of holistic evaluation&#10;studies. To address these challenges, we present INSTRUCTEVAL, a more&#10;comprehensive evaluation suite designed specifically for instruction-tuned&#10;large language models. Unlike previous works, our evaluation involves a&#10;rigorous assessment of models based on problem-solving, writing ability, and&#10;alignment to human values. We take a holistic approach to analyze various&#10;factors affecting model performance, including the pretraining foundation,&#10;instruction-tuning data, and training methods. Our findings reveal that the&#10;quality of instruction data is the most crucial factor in scaling model&#10;performance. While open-source models demonstrate impressive writing abilities,&#10;there is substantial room for improvement in problem-solving and alignment. We&#10;are encouraged by the rapid development of models by the open-source community,&#10;but we also highlight the need for rigorous evaluation to support claims made&#10;about these models. Through INSTRUCTEVAL, we aim to foster a deeper&#10;understanding of instruction-tuned models and advancements in their&#10;capabilities. INSTRUCTEVAL is publicly available at&#10;https://github.com/declare-lab/instruct-eval.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.04717" label="2211.04717">
        <attvalues>
          <attvalue for="0" value="Improving Noisy Student Training on Non-target Domain Data for Automatic&#10;  Speech Recognition" />
          <attvalue for="1" value="  Noisy Student Training (NST) has recently demonstrated extremely strong&#10;performance in Automatic Speech Recognition(ASR). In this paper, we propose a&#10;data selection strategy named LM Filter to improve the performance of NST on&#10;non-target domain data in ASR tasks. Hypotheses with and without a Language&#10;Model are generated and the CER differences between them are utilized as a&#10;filter threshold. Results reveal that significant improvements of 10.4%&#10;compared with no data filtering baselines. We can achieve 3.31% CER in&#10;AISHELL-1 test set, which is best result from our knowledge without any other&#10;supervised data. We also perform evaluations on the supervised 1000 hour&#10;AISHELL-2 dataset and competitive results of 4.73% CER can be achieved.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, Semi-Supervised Learning (SSL) has attracted a lot of research interest &#10;in many fields of deep learning, such as Automatic Speech Recognition (ASR) \cite{ssl_asr0,ssl_asr1,synnaeve2020endtoend}, Computer Vision \cite{ssl_cv0,ssl_cv1,xie2020self} and Natural Language Processing \cite{ssl_nlp0,ssl_nlp1,ssl_nlp2}. &#10;Among these methods, Noisy Student Training (NST) has recently demonstrated extremely strong performances in Image Classification \cite{xie2020self} by introducing noise and randomness into traditional Teacher-student Learning \cite{teacher_student0,teacher_student1}. This method further demonstrates its robustness in the ASR field \cite{park2020improved, doutre2021improving, nst_asr4}. After combing with pre-train methods \cite{baevski2020wav2vec}, NST is shown to be a vital component for achieving SOTA results on a number of datasets, e.g. Librispeech \cite{NST_ASR2_pushing_limits}.&#10;&#10;However, NST has not been widely investigated in ASR tasks when the domain of the supervised data does not match the unsupervised data. Noise and domain play an important role in ASR \cite{noise_domain0} and the abundant unsupervised data from social media may not always match the domain of the desired task. Thus, proper data selection techniques are required to remove noise and select data that is close to the target domain \cite{lu22_interspeech}. &#10;The most common filter in ASR is the Confidence Score that selects the most trustworthy transcriptions based on confidence estimation and threshold \cite{confidence_score0, confidence_score1, confidence_score2}. However, this method is not always promising in scenarios with large amount of unlabelled data with domain mismatches.&#10;Another recent unsupervised data selection technique is investigated in \cite{lu22_interspeech}, where a contrastive Language Model is applied as a data selector to better improve the target-domain ASR task.&#10;&#10;In this paper, we propose a novel data selection strategy named LM Filter which can utilize model differences to filter more valuable non-target domain data to improve the performance of NST. &#10;We leverage concept of contrastive LM and data selection method in \cite{zheng2022scaling}. Our LM Filter is based on hypotheses from LM to gradually remove noisy data inside each iteration of NST method. The filter condition is relaxed through the NST iteration to make the model advance gradually in due order. &#10;This method has the following benefits:&#10;\begin{itemize}&#10; \item No additional data selection models are required. Model differences can be obtained from different decoding strategies (e.g. with/without LM).&#10; \item Label is not required to perform the data selection and it is totally unsupervised.&#10; \item Less time and resources are utilized to run the NST method and it can converge faster in fewer iterations.&#10;\end{itemize}&#10;&#10;Experiments on AISHELL-1 \cite{aishell1} as supervised data and WenetSpeech \cite{zhang2022WenetSpeech} as unsupervised data indicate a significant improvement of 10.4\% comparing with no data filtering baselines. &#10;When combined AISHELL-2 \cite{aishell2} and WenetSpeech as unsupervised data, 3.31\% character error rate(CER) is achieved on AISHELL-1 test set, which is the best result from our knowledge without any other supervised data on this test set. &#10;LM Filter further demonstrates its robustness in larger dataset such as AISHELL-2 (supervised) and WenetSpeech (unsupervised) to achieve promising result of 4.73\% which has 13.6\% improvement comparing with the baseline. &#10;&#10;The rest of the paper is organized as follows. Section 2 briefly introduces the basic concepts and methods of NST in ASR. Our proposed data selection strategy LM Filter will be included in section3. Experiment details are introduced in Section 4. Eventually we give our conclusions in section5.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Automatic Speech Recognition, Signal Processing, Data Selection Strategies, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1905.00546" label="1905.00546">
        <attvalues>
          <attvalue for="0" value="Billion-scale semi-supervised learning for image classification" />
          <attvalue for="1" value="  This paper presents a study of semi-supervised learning with large&#10;convolutional networks. We propose a pipeline, based on a teacher/student&#10;paradigm, that leverages a large collection of unlabelled images (up to 1&#10;billion). Our main goal is to improve the performance for a given target&#10;architecture, like ResNet-50 or ResNext. We provide an extensive analysis of&#10;the success factors of our approach, which leads us to formulate some&#10;recommendations to produce high-accuracy models for image classification with&#10;semi-supervised learning. As a result, our approach brings important gains to&#10;standard architectures for image, video and fine-grained classification. For&#10;instance, by leveraging one billion unlabelled images, our learned vanilla&#10;ResNet-50 achieves 81.2% top-1 accuracy on the ImageNet benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.04445" label="1904.04445">
        <attvalues>
          <attvalue for="0" value="Semi-Supervised Segmentation of Salt Bodies in Seismic Images using an&#10;  Ensemble of Convolutional Neural Networks" />
          <attvalue for="1" value="  Seismic image analysis plays a crucial role in a wide range of industrial&#10;applications and has been receiving significant attention. One of the essential&#10;challenges of seismic imaging is detecting subsurface salt structure which is&#10;indispensable for identification of hydrocarbon reservoirs and drill path&#10;planning. Unfortunately, exact identification of large salt deposits is&#10;notoriously difficult and professional seismic imaging often requires expert&#10;human interpretation of salt bodies. Convolutional neural networks (CNNs) have&#10;been successfully applied in many fields, and several attempts have been made&#10;in the field of seismic imaging. But the high cost of manual annotations by&#10;geophysics experts and scarce publicly available labeled datasets hinder the&#10;performance of the existing CNN-based methods. In this work, we propose a&#10;semi-supervised method for segmentation (delineation) of salt bodies in seismic&#10;images which utilizes unlabeled data for multi-round self-training. To reduce&#10;error amplification during self-training we propose a scheme which uses an&#10;ensemble of CNNs. We show that our approach outperforms state-of-the-art on the&#10;TGS Salt Identification Challenge dataset and is ranked the first among the&#10;3234 competing methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.13788" label="1909.13788">
        <attvalues>
          <attvalue for="0" value="Revisiting Self-Training for Neural Sequence Generation" />
          <attvalue for="1" value="  Self-training is one of the earliest and simplest semi-supervised methods.&#10;The key idea is to augment the original labeled dataset with unlabeled data&#10;paired with the model's prediction (i.e. the pseudo-parallel data). While&#10;self-training has been extensively studied on classification problems, in&#10;complex sequence generation tasks (e.g. machine translation) it is still&#10;unclear how self-training works due to the compositionality of the target&#10;space. In this work, we first empirically show that self-training is able to&#10;decently improve the supervised baseline on neural sequence generation tasks.&#10;Through careful examination of the performance gains, we find that the&#10;perturbation on the hidden states (i.e. dropout) is critical for self-training&#10;to benefit from the pseudo-parallel data, which acts as a regularizer and&#10;forces the model to yield close predictions for similar unlabeled inputs. Such&#10;effect helps the model correct some incorrect predictions on unlabeled data. To&#10;further encourage this mechanism, we propose to inject noise to the input&#10;space, resulting in a &quot;noisy&quot; version of self-training. Empirical study on&#10;standard machine translation and text summarization benchmarks shows that noisy&#10;self-training is able to effectively utilize unlabeled data and improve the&#10;performance of the supervised baseline by a large margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.01981" label="2204.01981">
        <attvalues>
          <attvalue for="0" value="Unsupervised Data Selection via Discrete Speech Representation for ASR" />
          <attvalue for="1" value="  Self-supervised learning of speech representations has achieved impressive&#10;results in improving automatic speech recognition (ASR). In this paper, we show&#10;that data selection is important for self-supervised learning. We propose a&#10;simple and effective unsupervised data selection method which selects&#10;acoustically similar speech to a target domain. It takes the discrete speech&#10;representation available in common self-supervised learning frameworks as&#10;input, and applies a contrastive data selection method on the discrete tokens.&#10;Through extensive empirical studies we show that our proposed method reduces&#10;the amount of required pre-training data and improves the downstream ASR&#10;performance. Pre-training on a selected subset of 6% of the general data pool&#10;results in 11.8% relative improvements in LibriSpeech test-other compared to&#10;pre-training on the full set. On Multilingual LibriSpeech French, German, and&#10;Spanish test sets, selecting 6% data for pre-training reduces word error rate&#10;by more than 15% relatively compared to the full set, and achieves competitive&#10;results compared to current state-of-the-art performances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.05522" label="1709.05522">
        <attvalues>
          <attvalue for="0" value="AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech&#10;  Recognition Baseline" />
          <attvalue for="1" value="  An open-source Mandarin speech corpus called AISHELL-1 is released. It is by&#10;far the largest corpus which is suitable for conducting the speech recognition&#10;research and building speech recognition systems for Mandarin. The recording&#10;procedure, including audio capturing devices and environments are presented in&#10;details. The preparation of the related resources, including transcriptions and&#10;lexicon are described. The corpus is released with a Kaldi recipe. Experimental&#10;results implies that the quality of audio recordings and transcriptions are&#10;promising.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.10583" label="1808.10583">
        <attvalues>
          <attvalue for="0" value="AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale" />
          <attvalue for="1" value="  AISHELL-1 is by far the largest open-source speech corpus available for&#10;Mandarin speech recognition research. It was released with a baseline system&#10;containing solid training and testing pipelines for Mandarin ASR. In AISHELL-2,&#10;1000 hours of clean read-speech data from iOS is published, which is free for&#10;academic usage. On top of AISHELL-2 corpus, an improved recipe is developed and&#10;released, containing key components for industrial applications, such as&#10;Chinese word segmentation, flexible vocabulary expension and phone set&#10;transformation etc. Pipelines support various state-of-the-art techniques, such&#10;as time-delayed neural networks and Lattic-Free MMI objective funciton. In&#10;addition, we also release dev and test data from other channels(Android and&#10;Mic). For research community, we hope that AISHELL-2 corpus can be a solid&#10;resource for topics like transfer learning and robust ASR. For industry, we&#10;hope AISHELL-2 recipe can be a helpful reference for building meaningful&#10;industrial systems and products.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.17790" label="2404.17790">
        <attvalues>
          <attvalue for="0" value="Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing&#10;  Japanese Language Capabilities" />
          <attvalue for="1" value="  Cross-lingual continual pre-training of large language models (LLMs)&#10;initially trained on English corpus allows us to leverage the vast amount of&#10;English language resources and reduce the pre-training cost. In this study, we&#10;constructed Swallow, an LLM with enhanced Japanese capability, by extending the&#10;vocabulary of Llama 2 to include Japanese characters and conducting continual&#10;pre-training on a large Japanese web corpus. Experimental results confirmed&#10;that the performance on Japanese tasks drastically improved through continual&#10;pre-training, and the performance monotonically increased with the amount of&#10;training data up to 100B tokens. Consequently, Swallow achieved superior&#10;performance compared to other LLMs that were trained from scratch in English&#10;and Japanese. An analysis of the effects of continual pre-training revealed&#10;that it was particularly effective for Japanese question answering tasks.&#10;Furthermore, to elucidate effective methodologies for cross-lingual continual&#10;pre-training from English to Japanese, we investigated the impact of vocabulary&#10;expansion and the effectiveness of incorporating parallel corpora. The results&#10;showed that the efficiency gained through vocabulary expansion had no negative&#10;impact on performance, except for the summarization task, and that the combined&#10;use of parallel corpora enhanced translation ability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Cross-Lingual Pre-Training, Linguistics, Language Model Training, Artificial Intelligence, Multilingual NLP" />
        </attvalues>
      </node>
      <node id="1910.00294" label="1910.00294">
        <attvalues>
          <attvalue for="0" value="When and Why is Document-level Context Useful in Neural Machine&#10;  Translation?" />
          <attvalue for="1" value="  Document-level context has received lots of attention for compensating neural&#10;machine translation (NMT) of isolated sentences. However, recent advances in&#10;document-level NMT focus on sophisticated integration of the context,&#10;explaining its improvement with only a few selected examples or targeted test&#10;sets. We extensively quantify the causes of improvements by a document-level&#10;model in general test sets, clarifying the limit of the usefulness of&#10;document-level context in NMT. We show that most of the improvements are not&#10;interpretable as utilizing the context. We also show that a minimal encoding is&#10;sufficient for the context modeling and very long context is not helpful for&#10;NMT.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural machine translation (NMT) \cite{bahdanau2015neural,vaswani2017attention} has been originally developed to work sentence by sentence. Recently, it has been claimed that sentence-level NMT generates document-level errors, e.g. wrong coreference of pronouns/articles or inconsistent translations throughout a document \cite{guillou2018pronoun,laubli2018has}.&#10;&#10;A lot of research addresses these problems by feeding surrounding context sentences as additional inputs to an NMT model.&#10;Modeling of the context is usually done with fully-fledged NMT encoders with extensions to consider complex relations between sentences \cite{bawden2018evaluating,voita2018context,zhang2018improving,miculicich2018document,maruf2019selective}.&#10;Despite the high overhead in modeling, translation metric scores (e.g. {\bleu}) are often only marginally improved, leaving the evaluation to artificial tests targeted for pronoun resolution \cite{jean2017does,tiedemann2017neural,bawden2018evaluating,voita2018context,voita2019when}.&#10;Even if the metric score gets significantly better, the improvement is limited to specific datasets or explained with only a few examples \cite{tu2018learning,maruf2018document,kuang2018fusing,cao2018encoding,zhang2018improving,maruf2019selective}.&#10;&#10;This paper systematically investigates when and why document-level context improves NMT, asking the following research questions:&#10;\begin{itemize}\itemsep0em&#10; \item In general, how often is the context utilized in an interpretable way, e.g. coreference?&#10; \item Is there any other (non-linguistic) cause of improvements by document-level models?&#10; \item Which part of a context sentence is actually meaningful for the improvement?&#10; \item Is a long-range context, e.g. in ten consecutive sentences, still useful?&#10; \item How much modeling power is necessary for the improvements?&#10;\end{itemize}&#10;&#10;To answer these questions, we conduct an extensive qualitative analysis on non-targeted test sets.&#10;According to the analysis, we use only the important parts of the surrounding sentences to facilitate the integration of long-range contexts.&#10;We also compare different architectures for the context modeling and check sufficient model complexity for a significant improvement.&#10;&#10;Our results show that the improvement in {\bleu} is mostly from a non-linguistic factor: regularization by reserving parameters for context inputs.&#10;We also verify that very long context is indeed not helpful for NMT, and a full encoder stack is not necessary for the improved performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Document-Level Context, Linguistics, Context Modeling Limitations, Artificial Intelligence, Mathematics, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1808.07048" label="1808.07048">
        <attvalues>
          <attvalue for="0" value="Has Machine Translation Achieved Human Parity? A Case for Document-level&#10;  Evaluation" />
          <attvalue for="1" value="  Recent research suggests that neural machine translation achieves parity with&#10;professional human translation on the WMT Chinese--English news translation&#10;task. We empirically test this claim with alternative evaluation protocols,&#10;contrasting the evaluation of single sentences and entire documents. In a&#10;pairwise ranking experiment, human raters assessing adequacy and fluency show a&#10;stronger preference for human over machine translation when evaluating&#10;documents as compared to isolated sentences. Our findings emphasise the need to&#10;shift towards document-level evaluation as machine translation improves to the&#10;degree that errors which are hard or impossible to spot at the sentence-level&#10;become decisive in discriminating quality of different translation outputs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.00513" label="1711.00513">
        <attvalues>
          <attvalue for="0" value="Evaluating Discourse Phenomena in Neural Machine Translation" />
          <attvalue for="1" value="  For machine translation to tackle discourse phenomena, models must have&#10;access to extra-sentential linguistic context. There has been recent interest&#10;in modelling context in neural machine translation (NMT), but models have been&#10;principally evaluated with standard automatic metrics, poorly adapted to&#10;evaluating discourse phenomena. In this article, we present hand-crafted,&#10;discourse test sets, designed to test the models' ability to exploit previous&#10;source and target sentences. We investigate the performance of recently&#10;proposed multi-encoder NMT models trained on subtitles for English to French.&#10;We also explore a novel way of exploiting context from the previous sentence.&#10;Despite gains using BLEU, multi-encoder models give limited improvement in the&#10;handling of discourse phenomena: 50% accuracy on our coreference test set and&#10;53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A&#10;simple strategy of decoding the concatenation of the previous and current&#10;sentence leads to good performance, and our novel strategy of multi-encoding&#10;and decoding of two sentences leads to the best performance (72.5% for&#10;coreference and 57% for coherence/cohesion), highlighting the importance of&#10;target-side context.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.10163" label="1805.10163">
        <attvalues>
          <attvalue for="0" value="Context-Aware Neural Machine Translation Learns Anaphora Resolution" />
          <attvalue for="1" value="  Standard machine translation systems process sentences in isolation and hence&#10;ignore extra-sentential information, even though extended context can both&#10;prevent mistakes in ambiguous cases and improve translation coherence. We&#10;introduce a context-aware neural machine translation model designed in such way&#10;that the flow of information from the extended context to the translation model&#10;can be controlled and analyzed. We experiment with an English-Russian subtitles&#10;dataset, and observe that much of what is captured by our model deals with&#10;improving pronoun translation. We measure correspondences between induced&#10;attention distributions and coreference relations and observe that the model&#10;implicitly captures anaphora. It is consistent with gains for sentences where&#10;pronouns need to be gendered in translation. Beside improvements in anaphoric&#10;cases, the model also improves in overall BLEU, both over its context-agnostic&#10;version (+0.7) and over simple concatenation of the context and source&#10;sentences (+0.6).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.03581" label="1810.03581">
        <attvalues>
          <attvalue for="0" value="Improving the Transformer Translation Model with Document-Level Context" />
          <attvalue for="1" value="  Although the Transformer translation model (Vaswani et al., 2017) has&#10;achieved state-of-the-art performance in a variety of translation tasks, how to&#10;use document-level context to deal with discourse phenomena problematic for&#10;Transformer still remains a challenge. In this work, we extend the Transformer&#10;model with a new context encoder to represent document-level context, which is&#10;then incorporated into the original encoder and decoder. As large-scale&#10;document-level parallel corpora are usually not available, we introduce a&#10;two-step training method to take full advantage of abundant sentence-level&#10;parallel corpora and limited document-level parallel corpora. Experiments on&#10;the NIST Chinese-English datasets and the IWSLT French-English datasets show&#10;that our approach improves over Transformer significantly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.01576" label="1809.01576">
        <attvalues>
          <attvalue for="0" value="Document-Level Neural Machine Translation with Hierarchical Attention&#10;  Networks" />
          <attvalue for="1" value="  Neural Machine Translation (NMT) can be improved by including document-level&#10;contextual information. For this purpose, we propose a hierarchical attention&#10;model to capture the context in a structured and dynamic manner. The model is&#10;integrated in the original NMT architecture as another level of abstraction,&#10;conditioning on the NMT model's own previous hidden states. Experiments show&#10;that hierarchical attention significantly improves the BLEU score over a strong&#10;NMT baseline with the state-of-the-art in context-aware methods, and that both&#10;the encoder and decoder benefit from context in complementary ways.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.08788" label="1903.08788">
        <attvalues>
          <attvalue for="0" value="Selective Attention for Context-aware Neural Machine Translation" />
          <attvalue for="1" value="  Despite the progress made in sentence-level NMT, current systems still fall&#10;short at achieving fluent, good quality translation for a full document. Recent&#10;works in context-aware NMT consider only a few previous sentences as context&#10;and may not scale to entire documents. To this end, we propose a novel and&#10;scalable top-down approach to hierarchical attention for context-aware NMT&#10;which uses sparse attention to selectively focus on relevant sentences in the&#10;document context and then attends to key words in those sentences. We also&#10;propose single-level attention approaches based on sentence or word-level&#10;information in the context. The document-level context representation, produced&#10;from these attention modules, is integrated into the encoder or decoder of the&#10;Transformer model depending on whether we use monolingual or bilingual context.&#10;Our experiments and evaluation on English-German datasets in different document&#10;MT settings show that our selective attention approach not only significantly&#10;outperforms context-agnostic baselines but also surpasses context-aware&#10;baselines in most cases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.05135" label="1704.05135">
        <attvalues>
          <attvalue for="0" value="Does Neural Machine Translation Benefit from Larger Context?" />
          <attvalue for="1" value="  We propose a neural machine translation architecture that models the&#10;surrounding text in addition to the source sentence. These models lead to&#10;better performance, both in terms of general translation quality and pronoun&#10;prediction, when trained on small corpora, although this improvement largely&#10;disappears when trained with a larger corpus. We also discover that&#10;attention-based neural machine translation is well suited for pronoun&#10;prediction and compares favorably with other approaches that were specifically&#10;designed for this task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.05943" label="1708.05943">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation with Extended Context" />
          <attvalue for="1" value="  We investigate the use of extended context in attention-based neural machine&#10;translation. We base our experiments on translated movie subtitles and discuss&#10;the effect of increasing the segments beyond single translation units. We study&#10;the use of extended source language context as well as bilingual context&#10;extensions. The models learn to distinguish between information from different&#10;segments and are surprisingly robust with respect to translation quality. In&#10;this pilot study, we observe interesting cross-sentential attention patterns&#10;that improve textual coherence in translation at least in some selected cases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.05979" label="1905.05979">
        <attvalues>
          <attvalue for="0" value="When a Good Translation is Wrong in Context: Context-Aware Machine&#10;  Translation Improves on Deixis, Ellipsis, and Lexical Cohesion" />
          <attvalue for="1" value="  Though machine translation errors caused by the lack of context beyond one&#10;sentence have long been acknowledged, the development of context-aware NMT&#10;systems is hampered by several problems. Firstly, standard metrics are not&#10;sensitive to improvements in consistency in document-level translations.&#10;Secondly, previous work on context-aware NMT assumed that the sentence-aligned&#10;parallel data consisted of complete documents while in most practical scenarios&#10;such document-level data constitutes only a fraction of the available parallel&#10;data. To address the first issue, we perform a human study on an&#10;English-Russian subtitles dataset and identify deixis, ellipsis and lexical&#10;cohesion as three main sources of inconsistency. We then create test sets&#10;targeting these phenomena. To address the second shortcoming, we consider a&#10;set-up in which a much larger amount of sentence-level data is available&#10;compared to that aligned at the document level. We introduce a model that is&#10;suitable for this scenario and demonstrate major gains over a context-agnostic&#10;baseline on our new benchmarks without sacrificing performance as measured with&#10;BLEU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.09367" label="1711.09367">
        <attvalues>
          <attvalue for="0" value="Learning to Remember Translation History with a Continuous Cache" />
          <attvalue for="1" value="  Existing neural machine translation (NMT) models generally translate&#10;sentences in isolation, missing the opportunity to take advantage of&#10;document-level information. In this work, we propose to augment NMT models with&#10;a very light-weight cache-like memory network, which stores recent hidden&#10;representations as translation history. The probability distribution over&#10;generated words is updated online depending on the translation history&#10;retrieved from the memory, endowing NMT models with the capability to&#10;dynamically adapt over time. Experiments on multiple domains with different&#10;topics and styles show the effectiveness of the proposed approach with&#10;negligible impact on the computational cost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.03688" label="1711.03688">
        <attvalues>
          <attvalue for="0" value="Document Context Neural Machine Translation with Memory Networks" />
          <attvalue for="1" value="  We present a document-level neural machine translation model which takes both&#10;source and target document context into account using memory networks. We model&#10;the problem as a structured prediction problem with interdependencies among the&#10;observed and hidden variables, i.e., the source sentences and their unobserved&#10;target translations in the document. The resulting structured prediction&#10;problem is tackled with a neural translation model equipped with two memory&#10;components, one each for the source and target side, to capture the documental&#10;interdependencies. We train the model end-to-end, and propose an iterative&#10;decoding algorithm based on block coordinate descent. Experimental results of&#10;English translations from French, German, and Estonian documents show that our&#10;model is effective in exploiting both source and target document context, and&#10;statistically significantly outperforms the previous work in terms of BLEU and&#10;METEOR.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.10044" label="2409.10044">
        <attvalues>
          <attvalue for="0" value="Benchmarking Large Language Model Uncertainty for Prompt Optimization" />
          <attvalue for="1" value="  Prompt optimization algorithms for Large Language Models (LLMs) excel in&#10;multi-step reasoning but still lack effective uncertainty estimation. This&#10;paper introduces a benchmark dataset to evaluate uncertainty metrics, focusing&#10;on Answer, Correctness, Aleatoric, and Epistemic Uncertainty. Through analysis&#10;of models like GPT-3.5-Turbo and Meta-Llama-3.1-8B-Instruct, we show that&#10;current metrics align more with Answer Uncertainty, which reflects output&#10;confidence and diversity, rather than Correctness Uncertainty, highlighting the&#10;need for improved metrics that are optimization-objective-aware to better guide&#10;prompt optimization. Our code and dataset are available at&#10;https://github.com/0Frett/PO-Uncertainty-Benchmarking.&#10;" />
          <attvalue for="2" value="&#10;Prompting methods for large language models (LLMs) have gained significant attention for their ability to enhance reasoning capabilities through multi-step processes, such as Chain of Thought (CoT)\cite{wei2022chain}, Tree of Thought(ToT)\cite{yao2024tree}, and ReAct\cite{yao2022react}. These approaches can be extended by incorporating search algorithms to optimize prompts, utilizing techniques like Monte Carlo Tree Search (LATS, STaR)\cite{zhou2023language,zelikman2022star}, bandit algorithms (LongPO)\cite{hsieh2023automatic}, and gradient-style search (OPRO)\cite{Yang2023LargeLM}. A key element in these search and optimization algorithms is uncertainty estimation, which is vital for guiding decisions, balancing exploration and exploitation, and improving algorithm efficiency. Uncertainty estimation techniques, such as those used in Bandit algorithms or Bayesian optimization, can dynamically adjust learning rates or hyperparameters. In combinatorial optimization (e.g., genetic algorithms or simulated annealing), uncertainty estimation informs heuristic decisions like mutation rates or temperature adjustments. Hence, developing robust methods to quantify uncertainty in LLMs, particularly for prompt optimization, is essential.&#10;&#10;Previous approaches to measuring uncertainty in LLMs primarily rely on token-level or sentence-level generation likelihoods, often represented by metrics like token disparity probability~\cite{wang2024chain}, predictive entropy~\cite{npe_lnpe} and reciprocal of perplexity~\cite{chen1998evaluation}. These techniques have been used for bias calibration~\cite{zhou2023batch}, controllable decoding~\cite{zhu2024hot}, and LLM planning~\cite{ren2023robots}. However, we argue that such token-level or sentence-level uncertainty measurements are more indicative of model output confidence or output diversity, which may not align with the needs of prompt optimization tasks. In these contexts, uncertainty estimation should guide the search process itself. For example, in tree-based reasoning, the uncertainty at each node should help steer the search direction in line with the search objectives rather than simply reflecting model confidence or output variability.&#10;&#10;Figure\ref{fig:au-cu} justify our hypothesis and illustrates the relationship between LLM correctness uncertainty, answer uncertainty (see section~\ref{sec:uncertainty_type}), and response accuracy in the GSM8K dataset. In prompt optimization tasks focused on searching correct answer, a reliable uncertainty metric targeting this objective should exhibits 50\% response accuracy(correct/wrong) when its value is at its highest. This objective conflicts with answer uncertainty, which is designed to measure diversity of responses but may reflect an incorrect majority answer.&#10;&#10;In this work, we first highlight the differing requirements for uncertainty metrics in prompt optimization versus text generation. We outline four key uncertainties that are beneficial to prompt optimization—Answer, Correctness, Aleatoric, Epistemic and propose a novel benchmarking pipeline designed to evaluate the effectiveness of current NLG uncertainty metrics in prompt optimization setting. By performing extensive sampling on LLMs, our pipeline construct large, tree-structured reasoning traces from model outputs. Once these traces are built, we can compute accurate estimation of the uncertainties, which can serve as ground truth values for comparison with metric predictions. Our evaluation shows that current uncertainty metrics mainly capture Answer Uncertainty and fail to measure other uncertainty types, emphasizing the need for more diverse, optimization-aware estimators to guide prompt optimization for different objectives.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Uncertainty, Machine Learning, Computer Science, Linguistics, Benchmarking Evaluation Metrics, Artificial Intelligence, Mathematics, Prompt Optimization Algorithms" />
        </attvalues>
      </node>
      <node id="2311.10117" label="2311.10117">
        <attvalues>
          <attvalue for="0" value="Automatic Engineering of Long Prompts" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated remarkable capabilities in&#10;solving complex open-domain tasks, guided by comprehensive instructions and&#10;demonstrations provided in the form of prompts. However, these prompts can be&#10;lengthy, often comprising hundreds of lines and thousands of tokens, and their&#10;design often requires considerable human effort. Recent research has explored&#10;automatic prompt engineering for short prompts, typically consisting of one or&#10;a few sentences. However, the automatic design of long prompts remains a&#10;challenging problem due to its immense search space. In this paper, we&#10;investigate the performance of greedy algorithms and genetic algorithms for&#10;automatic long prompt engineering. We demonstrate that a simple greedy approach&#10;with beam search outperforms other methods in terms of search efficiency.&#10;Moreover, we introduce two novel techniques that utilize search history to&#10;enhance the effectiveness of LLM-based mutation in our search algorithm. Our&#10;results show that the proposed automatic long prompt engineering algorithm&#10;achieves an average of 9.2% accuracy gain on eight tasks in Big Bench Hard,&#10;highlighting the significance of automating prompt designs to fully harness the&#10;capabilities of LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The remarkable ability of large language models (LLMs) to perform complex tasks without fine-tuning through prompting has significantly broadened their applicability. As a result, designing effective prompts to fully exploit the capabilities of LLMs has become an important topic. Many principal ways for prompt design have been studied recently~\cite{reynolds2021prompt,brown2020language,wei2022chain,wang2022self,wang2023plan}. For instance, a well-designed prompt may include a system prompt (e.g., &quot;you are an AI programming assistant&quot;), an instruction prompt outlining the task, multiple contextual examples, and a chain-of-thoughts reasoning sectionthat explains the step-by-step thought process behind the examples. By incorporating these elements, effective prompts often have tens of sentences and thousands of tokens.&#10;&#10;An orthogonal line of previous work has explored soft-prompt tuning, a technique that optimizes prompts within a continuous embedding space using standard continuous optimization algorithms~\cite{lester2021power,zhang2021differentiable,wang2022preserving}. While capable of achieving satisfactory performance, soft-prompts lack interpretability and cannot be applied via LLM APIs. Additionally, these parameter-efficient fine-tuning methods demand large training sets, making them unsuitable for applications with limited data, such as those with only tens or hundreds of samples.&#10;&#10;Given the limited availability of training data (&lt;1000 samples), our focus lies in exploring strategies for optimizing hard prompts, which are semantically equivalent to the original prompts but yield superior performance. In the context of automated prompt engineering, the literature considers two primary settings. The first setting, which aligns with our work, assumes the existence of an initial human-crafted prompt and aims to refine or improve it to achieve enhanced performance. Several discrete search algorithms have been proposed for this setting: \cite{xu2022gps} employs a genetic algorithm for prompt tuning, utilizing back translation, cloze tasks, and sentence continuation to mutate the initial instruction. More recently, \cite{fernando2023promptbreeder,guo2023connecting} proposed leveraging LLMs for mutation and crossover operations in evolutionary searches, while \cite{yang2023large} demonstrated the optimization capabilities of LLMs in generating improved prompt variations based on previous fitness scores. However, these prompt evolution techniques are designed for short sentences or paragraphs within a long prompt. For example, many of them try to evolve only the instruction part or the sentence ``Let's think step by step'' in the prompt. Our work aims to provide complete freedom to evolve the entire long prompt, opening up more avenues for improvement but also introducing challenges in determining how and where to change the original prompt.&#10;&#10;Another setting focuses on automatic prompt generation without a pre-existing prompt. \cite{honovich2022instruction} demonstrated the ability of LLMs to generate brief task descriptions when provided with input-output pairs. Building upon this technique and incorporating random search, \cite{zhou2022large} proposed an automatic prompt engineering (APE) algorithm capable of generating prompts from given pairs. \cite{pryzant2023automatic} explored the utilization of input-output pair feedback to refine instructions, while \cite{chen2023instructzero} developed a continuous relaxation approach employing Bayesian optimization for search within a continuous space. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.10200" label="2402.10200">
        <attvalues>
          <attvalue for="0" value="Chain-of-Thought Reasoning Without Prompting" />
          <attvalue for="1" value="  In enhancing the reasoning capabilities of large language models (LLMs),&#10;prior research primarily focuses on specific prompting techniques such as&#10;few-shot or zero-shot chain-of-thought (CoT) prompting. These methods, while&#10;effective, often involve manually intensive prompt engineering. Our study takes&#10;a novel approach by asking: Can LLMs reason effectively without prompting? Our&#10;findings reveal that, intriguingly, CoT reasoning paths can be elicited from&#10;pre-trained LLMs by simply altering the \textit{decoding} process. Rather than&#10;conventional greedy decoding, we investigate the top-$k$ alternative tokens,&#10;uncovering that CoT paths are frequently inherent in these sequences. This&#10;approach not only bypasses the confounders of prompting but also allows us to&#10;assess the LLMs' \textit{intrinsic} reasoning abilities. Moreover, we observe&#10;that the presence of a CoT in the decoding path correlates with a higher&#10;confidence in the model's decoded answer. This confidence metric effectively&#10;differentiates between CoT and non-CoT paths. Extensive empirical studies on&#10;various reasoning benchmarks show that the proposed CoT-decoding effectively&#10;elicits reasoning capabilities from language models, which were previously&#10;obscured by standard greedy decoding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.07650" label="2002.07650">
        <attvalues>
          <attvalue for="0" value="Uncertainty Estimation in Autoregressive Structured Prediction" />
          <attvalue for="1" value="  Uncertainty estimation is important for ensuring safety and robustness of AI&#10;systems. While most research in the area has focused on un-structured&#10;prediction tasks, limited work has investigated general uncertainty estimation&#10;approaches for structured prediction. Thus, this work aims to investigate&#10;uncertainty estimation for autoregressive structured prediction tasks within a&#10;single unified and interpretable probabilistic ensemble-based framework. We&#10;consider: uncertainty estimation for sequence data at the token-level and&#10;complete sequence-level; interpretations for, and applications of, various&#10;measures of uncertainty; and discuss both the theoretical and practical&#10;challenges associated with obtaining them. This work also provides baselines&#10;for token-level and sequence-level error detection, and sequence-level&#10;out-of-domain input detection on the WMT'14 English-French and WMT'17&#10;English-German translation and LibriSpeech speech recognition datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.17249" label="2309.17249">
        <attvalues>
          <attvalue for="0" value="Batch Calibration: Rethinking Calibration for In-Context Learning and&#10;  Prompt Engineering" />
          <attvalue for="1" value="  Prompting and in-context learning (ICL) have become efficient learning&#10;paradigms for large language models (LLMs). However, LLMs suffer from prompt&#10;brittleness and various bias factors in the prompt, including but not limited&#10;to the formatting, the choice verbalizers, and the ICL examples. To address&#10;this problem that results in unexpected performance degradation, calibration&#10;methods have been developed to mitigate the effects of these biases while&#10;recovering LLM performance. In this work, we first conduct a systematic&#10;analysis of the existing calibration methods, where we both provide a unified&#10;view and reveal the failure cases. Inspired by these analyses, we propose Batch&#10;Calibration (BC), a simple yet intuitive method that controls the contextual&#10;bias from the batched input, unifies various prior approaches, and effectively&#10;addresses the aforementioned issues. BC is zero-shot, inference-only, and&#10;incurs negligible additional costs. In the few-shot setup, we further extend BC&#10;to allow it to learn the contextual bias from labeled data. We validate the&#10;effectiveness of BC with PaLM 2-(S, M, L) and CLIP models and demonstrate&#10;state-of-the-art performance over previous calibration baselines across more&#10;than 10 natural language understanding and image classification tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.02772" label="2309.02772">
        <attvalues>
          <attvalue for="0" value="Hot or Cold? Adaptive Temperature Sampling for Code Generation with&#10;  Large Language Models" />
          <attvalue for="1" value="  Recently, Large Language Models (LLMs) have shown impressive abilities in&#10;code generation. However, existing LLMs' decoding strategies are designed for&#10;Natural Language (NL) generation, overlooking the differences between NL and&#10;programming languages (PL). Due to this oversight, a better decoding strategy&#10;for code generation remains an open question. In this paper, we conduct the&#10;first systematic study to explore a decoding strategy specialized in code&#10;generation. With an analysis of loss distributions of code tokens, we find that&#10;code tokens can be divided into two categories: challenging tokens that are&#10;difficult to predict and confident tokens that can be easily inferred. Among&#10;them, the challenging tokens mainly appear at the beginning of a code block.&#10;Inspired by the above findings, we propose a simple yet effective method:&#10;Adaptive Temperature (AdapT) sampling, which dynamically adjusts the&#10;temperature coefficient when decoding different tokens. We apply a larger&#10;temperature when sampling for challenging tokens, allowing LLMs to explore&#10;diverse choices. We employ a smaller temperature for confident tokens avoiding&#10;the influence of tail randomness noises. We apply AdapT sampling to LLMs with&#10;different sizes and conduct evaluations on two popular datasets. Results show&#10;that AdapT sampling significantly outperforms state-of-the-art decoding&#10;strategy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.01928" label="2307.01928">
        <attvalues>
          <attvalue for="0" value="Robots That Ask For Help: Uncertainty Alignment for Large Language Model&#10;  Planners" />
          <attvalue for="1" value="  Large language models (LLMs) exhibit a wide range of promising capabilities&#10;-- from step-by-step planning to commonsense reasoning -- that may provide&#10;utility for robots, but remain prone to confidently hallucinated predictions.&#10;In this work, we present KnowNo, which is a framework for measuring and&#10;aligning the uncertainty of LLM-based planners such that they know when they&#10;don't know and ask for help when needed. KnowNo builds on the theory of&#10;conformal prediction to provide statistical guarantees on task completion while&#10;minimizing human help in complex multi-step planning settings. Experiments&#10;across a variety of simulated and real robot setups that involve tasks with&#10;different modes of ambiguity (e.g., from spatial to numeric uncertainties, from&#10;human preferences to Winograd schemas) show that KnowNo performs favorably over&#10;modern baselines (which may involve ensembles or extensive prompt tuning) in&#10;terms of improving efficiency and autonomy, while providing formal assurances.&#10;KnowNo can be used with LLMs out of the box without model-finetuning, and&#10;suggests a promising lightweight approach to modeling uncertainty that can&#10;complement and scale with the growing capabilities of foundation models.&#10;Website: https://robot-help.github.io&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.15729" label="2408.15729">
        <attvalues>
          <attvalue for="0" value="LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of&#10;  Relational Knowledge in Language Models" />
          <attvalue for="1" value="  Knowledge probing evaluates the extent to which a language model (LM) has&#10;acquired relational knowledge during its pre-training phase. It provides a&#10;cost-effective means of comparing LMs of different sizes and training setups&#10;and is useful for monitoring knowledge gained or lost during continual learning&#10;(CL). In prior work, we presented an improved knowledge probe called BEAR&#10;(Wiland et al., 2024), which enables the comparison of LMs trained with&#10;different pre-training objectives (causal and masked LMs) and addresses issues&#10;of skewed distributions in previous probes to deliver a more unbiased reading&#10;of LM knowledge. With this paper, we present LM-PUB- QUIZ, a Python framework&#10;and leaderboard built around the BEAR probing mechanism that enables&#10;researchers and practitioners to apply it in their work. It provides options&#10;for standalone evaluation and direct integration into the widely-used training&#10;pipeline of the Hugging Face TRANSFORMERS library. Further, it provides a&#10;fine-grained analysis of different knowledge types to assist users in better&#10;understanding the knowledge in each evaluated LM. We publicly release&#10;LM-PUB-QUIZ as an open-source project.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-trained language models (LMs) currently take on a central role in state-of-the-art NLP approaches~\cite{devlinBERTPretrainingDeep2019b}. Given their importance, prior work has sought to measure the amount of factual knowledge encoded in LMs using knowledge probing mechanisms~\cite{petroniHowContextAffects2020, kaloKAMELKnowledgeAnalysis2022}. Here, the knowledge represented in the parameters of an LM is automatically compared to factual knowledge in a relational knowledge base (KB). For instance, a probe might measure if an LM can correctly recall the capitals of countries, as illustrated in Figure~\ref{fig:bear-illustration}.&#10;&#10;In previous work, we introduced a new knowledge probe called BEAR~\cite{wilandBEAR2024} that addresses various issues of ambiguities and skewed answer distributions of prior probes to deliver a more unbiased reading of LM knowledge. Further, it reformulates probing as a ranking task, thus enabling a direct comparison of LMs trained with different pre-training objectives (masked and causal LMs) and vocabularies. However, despite being conceptually simple, BEAR relies on a different implementation than existing probes and previously returned only an overall score as the evaluation result, thus limiting adoption and interpretability. &#10;&#10;Framework. With this paper, we present LM-Pub-Quiz, an open-source Python framework and leaderboard built around the BEAR probing mechanism that enables researchers and practitioners to apply it in their work. Our framework was designed for ease of use, providing simple interfaces and direct integration into the Hugging Face transformers ecosystem~\cite{wolfTransformersStateoftheArtNatural2020}. Two use cases in particular have shaped the development of the library: &#10;&#10;\begin{enumerate}&#10; \item The first main use case is to evaluate and compare already-trained LMs. Users need only pass the string identifier of one of the LMs on the Hugging Face model hub in order to calculate the BEAR score for this model. This yields not only an overall BEAR score but also a more fine-grained analysis of different types of relational knowledge in the LM. &#10;&#10;\item The second main use case is to monitor the knowledge gained and lost during pre-training and continual training (e.g.~when adapting an LM to a new domain). Here, LM-Pub-Quiz provides an easy integration into the Hugging Face Trainer to track knowledge development during training.&#10;\end{enumerate}&#10;&#10;To encourage uptake, we make our library freely available and open source. Additionally, we are actively curating a leaderboard with scores of existing LMs. We encourage the community to participate in extending the list of evaluated models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Natural Language Processing, Knowledge Probing Framework, Continual Learning Analysis" />
        </attvalues>
      </node>
      <node id="2108.10447" label="2108.10447">
        <attvalues>
          <attvalue for="0" value="One TTS Alignment To Rule Them All" />
          <attvalue for="1" value="  Speech-to-text alignment is a critical component of neural textto-speech&#10;(TTS) models. Autoregressive TTS models typically use an attention mechanism to&#10;learn these alignments on-line. However, these alignments tend to be brittle&#10;and often fail to generalize to long utterances and out-of-domain text, leading&#10;to missing or repeating words. Most non-autoregressive endto-end TTS models&#10;rely on durations extracted from external sources. In this paper we leverage&#10;the alignment mechanism proposed in RAD-TTS as a generic alignment learning&#10;framework, easily applicable to a variety of neural TTS models. The framework&#10;combines forward-sum algorithm, the Viterbi algorithm, and a simple and&#10;efficient static prior. In our experiments, the alignment learning framework&#10;improves all tested TTS architectures, both autoregressive (Flowtron, Tacotron&#10;2) and non-autoregressive (FastPitch, FastSpeech 2, RAD-TTS). Specifically, it&#10;improves alignment convergence speed of existing attention-based mechanisms,&#10;simplifies the training pipeline, and makes the models more robust to errors on&#10;long utterances. Most importantly, the framework improves the perceived speech&#10;synthesis quality, as judged by human evaluators.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural text-to-speech (TTS) models, especially autoregressive TTS models, produce naturally sounding speech for in-domain text~\cite{Tacotron, Tacotron2, Flowtron}.&#10;However, these models can suffer from pronunciation issues such as missing and repeated words for out-of-domain text, especially in long utterances. A typical neural TTS model consists of an encoder that maps text inputs to hidden states, a decoder that generates mel-spectograms or waveforms from the hidden states, and an alignment mechanism or a duration source that maps the encoder states to decoder inputs~\cite{Tacotron, Tacotron2, Flowtron, fastspeech2, yiFastSpeech, kim2020glowtts, lancucki2020fastpitch}. Autoregressive TTS models rely on the attention mechanism~\cite{Graves13, bahdanau2014neural} to align text and speech, typically using content based attention mechanism~\cite{Tacotron, Flowtron}. Although recent works have improved alignments by using both content and location sensitive attention~\cite{Tacotron2}, such models still suffer from alignment problems on long utterances~\cite{kim2020glowtts}. &#10;&#10;In contrast, parallel (non-autoregressive) TTS models factor out durations from the decoding process, thereby requiring durations as input for each token. These models generally rely on external aligners~\cite{fastspeech2} like the Montreal Forced Aligner (MFA)~\cite{MFA}, or on durations extracted from a pre-trained autoregressive model (or forced aligner)~\cite{lancucki2020fastpitch, peng2020non, yiFastSpeech} like Tacotron 2~\cite{Tacotron2}. In addition to the dependency on external alignments, these models &#10;can suffer from poor training efficiency, require carefully engineered training schedules to prevent unstable learning, and may be difficult to extend to languages either because pre-existing aligners are either unavailable or their output does not exactly fit the desired format. Ideally, we would like the alignment to be trained end-to-end as part of the TTS model to significantly simplify the training pipeline. We would also like the alignments to converge and stabilize rapidly as the rest of the TTS pipeline is dependent on it. Most importantly the output quality should be no worse (and hopefully better) than if we were to train on alignments provided by external sources.&#10;&#10;This work leverages the alignment framework proposed in~\cite{rad_tts} to simplify alignment learning in several TTS models. We demonstrate its ability to convert all TTS models to a simpler end-to-end pipeline with better convergence rates and improved robustness to long utterances. We improve prior work on alignments in autoregressive TTS systems~\cite{Tacotron, Tacotron2, Flowtron} by adding a constraint that directly maximizes the likelihood of text given speech mel-spectrograms. We demonstrate that this approach can also be used to learn alignments online in parallel TTS models~\cite{lancucki2020fastpitch, fastspeech2, rad_tts}, again eliminating the need for external aligners or alignments obtained from a pre-trained TTS models. In addition, we further examine the effect of a simple, static alignment prior for guiding alignment attention learning \cite{tachibana2018efficiently, rad_tts}.&#10; We demonstrate in our experiments that our framework can improve both autoregressive and parallel models with respect to convergence rate of speech text alignments, closeness to hand-annotated durations, and speech quality. In summary, our results show that TTS models trained with our alignment learning framework have fewer repeated and missing words during inference, improved stability on long sequence synthesis, and improved overall speech quality based on human evaluation.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Text Models, Computer Science, Linguistics, Engineering, Alignment Learning, Speech Synthesis, Signal Processing" />
        </attvalues>
      </node>
      <node id="2006.06873" label="2006.06873">
        <attvalues>
          <attvalue for="0" value="FastPitch: Parallel Text-to-speech with Pitch Prediction" />
          <attvalue for="1" value="  We present FastPitch, a fully-parallel text-to-speech model based on&#10;FastSpeech, conditioned on fundamental frequency contours. The model predicts&#10;pitch contours during inference. By altering these predictions, the generated&#10;speech can be more expressive, better match the semantic of the utterance, and&#10;in the end more engaging to the listener. Uniformly increasing or decreasing&#10;pitch with FastPitch generates speech that resembles the voluntary modulation&#10;of voice. Conditioning on frequency contours improves the overall quality of&#10;synthesized speech, making it comparable to state-of-the-art. It does not&#10;introduce an overhead, and FastPitch retains the favorable, fully-parallel&#10;Transformer architecture, with over 900x real-time factor for mel-spectrogram&#10;synthesis of a typical utterance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.08969" label="1710.08969">
        <attvalues>
          <attvalue for="0" value="Efficiently Trainable Text-to-Speech System Based on Deep Convolutional&#10;  Networks with Guided Attention" />
          <attvalue for="1" value="  This paper describes a novel text-to-speech (TTS) technique based on deep&#10;convolutional neural networks (CNN), without use of any recurrent units.&#10;Recurrent neural networks (RNN) have become a standard technique to model&#10;sequential data recently, and this technique has been used in some cutting-edge&#10;neural TTS techniques. However, training RNN components often requires a very&#10;powerful computer, or a very long time, typically several days or weeks. Recent&#10;other studies, on the other hand, have shown that CNN-based sequence synthesis&#10;can be much faster than RNN-based techniques, because of high&#10;parallelizability. The objective of this paper is to show that an alternative&#10;neural TTS based only on CNN alleviate these economic costs of training. In our&#10;experiment, the proposed Deep Convolutional TTS was sufficiently trained&#10;overnight (15 hours), using an ordinary gaming PC equipped with two GPUs, while&#10;the quality of the synthesized speech was almost acceptable.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.00990" label="2204.00990">
        <attvalues>
          <attvalue for="0" value="Content-Dependent Fine-Grained Speaker Embedding for Zero-Shot Speaker&#10;  Adaptation in Text-to-Speech Synthesis" />
          <attvalue for="1" value="  Zero-shot speaker adaptation aims to clone an unseen speaker's voice without&#10;any adaptation time and parameters. Previous researches usually use a speaker&#10;encoder to extract a global fixed speaker embedding from reference speech, and&#10;several attempts have tried variable-length speaker embedding. However, they&#10;neglect to transfer the personal pronunciation characteristics related to&#10;phoneme content, leading to poor speaker similarity in terms of detailed&#10;speaking styles and pronunciation habits. To improve the ability of the speaker&#10;encoder to model personal pronunciation characteristics, we propose&#10;content-dependent fine-grained speaker embedding for zero-shot speaker&#10;adaptation. The corresponding local content embeddings and speaker embeddings&#10;are extracted from a reference speech, respectively. Instead of modeling the&#10;temporal relations, a reference attention module is introduced to model the&#10;content relevance between the reference speech and the input text, and to&#10;generate the fine-grained speaker embedding for each phoneme encoder output.&#10;The experimental results show that our proposed method can improve speaker&#10;similarity of synthesized speeches, especially for unseen speakers.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural network-based text-to-speech (TTS), aiming to synthesize intelligible and natural speech from text, has made great progress in recent years \cite{shen2018natural, ping2018deep, ren2020fastspeech}.&#10;These TTS models can synthesize natural human speech with sufficient high-quality training data, for not only single speaker but also multi-speaker scenarios \cite{zhang2019learning, chen2020multispeech}.&#10;But it is too expensive to collect a sufficient amount of speech data for new speakers.&#10;As a result, adapting TTS models to an arbitrary speaker with a few samples (speaker adaptation) is a hot research topic in academia and industry recently \cite{tan2021survey}.&#10;&#10;One of the general approaches for speaker adaptation is fine-tuning the whole or part of a well-trained multi-speaker TTS model with a few adaptation data \cite{arik2018neural}.&#10;Some works explore how to better adapt the pre-trained model to the target speaker, such as AdaSpeech series \cite{chen2021adaspeech, yan2021adaspeech2, yan2021adaspeech3}.&#10;These methods are proven to achieve considerable adaptation performance, but with some drawbacks:&#10;(i) certain adaptation time and trainable parameters are required for each new target speaker; &#10;(ii) voice quality drops quickly when the adaptation data is less than 10 sentences \cite{chen2021adaspeech};&#10;(iii) adaptation performance can be affected by low-quality speeches of the target speaker, resulting in poor intelligibility and prosody of synthesized speech. &#10;&#10;To avoid the problems of directly adapting pre-trained models, another line is to leverage a speaker encoder to extract the speaker embedding from reference speech to model speaker identity in TTS.&#10;This approach is also called zero-shot speaker adaptation, since it can clone an unseen speaker's voice by using the speaker embedding only, without any adaptation time and parameters. &#10;In this connection, it is necessary to explore constructing a better speaker identity representation space to improve the generalization of speaker representation and its adaptability to acoustic models. &#10;Some researches draw on transfer learning, such as x-vectors from speaker verification tasks \cite{jia2018transfer, cooper2020zero,zhang2021one}.&#10;Others focus on training an encoder network jointly with acoustic models, like using global speaker embeddings (GSEs) \cite{lu2019one} or variational autoencoder (VAE) \cite{hsu2018hierarchical, nguyen2021nvc}.&#10;These methods can clone the overall timbre or speaking style of the reference speech well and make real-time inference for an arbitrary speaker without fine-tuning.&#10;&#10;Although representing a speaker's voice with a fixed-length vector is a common idea for zero-shot speaker adaptation, the speaker characteristics of a person actually include not only global timbre information but also some local pronunciation variations.&#10;It is difficult to use a single speaker embedding to describe these local characteristics, leading to poor similarity in terms of detailed speaking styles and pronunciation habits of the target speaker.&#10;Inspired by related works on prosody transfer \cite{klimkov2019finegrained, lee2019robust, li2021towards}, some prior researches try to introduce fine-grained speaker embedding via attention mechanism for capturing more speaker information from speech. &#10;\cite{Fu2019phoneme} considers phoneme level speaker representations to generate phoneme-dependent speaker embedding by attention.&#10;However, the extraction procedure of phoneme level representations is too complicated and not suitable for zero-shot scenarios.&#10;To make good use of reference speech, &#10;Attentron \cite{choi2020attentron} proposes an attention-based variable-length embedding method to leverage features near to raw reference speech for better generalization.&#10;However, it only extracts simple reference embeddings without clear meaning and does not show the ability to transfer personal pronunciation characteristics related to phoneme content. &#10;&#10;To further improve speaker similarity for zero-shot speaker adaptation,&#10;we extract the corresponding local content embeddings and local speaker embeddings from a reference speech to model personal pronunciation characteristics.&#10;A content-dependent reference attention module is introduced to model the content relevance between the reference speech and the input text, and is used to guide the generation of fine-grained speaker embedding for each phoneme encoder output.&#10;The experiment results show that our proposed method outperforms both two fixed-length speaker embedding methods and a variable-length speaker embedding method based on Attentron in terms of speaker similarity, &#10;especially for unseen speakers.&#10;The synthesized speeches and experimental analysis demonstrate that our method has the ability to transfer personal pronunciation characteristics related to phoneme content. &#10;Besides, we investigate the impact of local speaker embeddings with different granularity on the synthesized speech and present the interpretability of our method through visualization.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Speech Processing, Linguistics, Engineering, Zero-Shot Adaptation Techniques, Personal Pronunciation Modeling, Speaker Voice Cloning, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2106.15561" label="2106.15561">
        <attvalues>
          <attvalue for="0" value="A Survey on Neural Speech Synthesis" />
          <attvalue for="1" value="  Text to speech (TTS), or speech synthesis, which aims to synthesize&#10;intelligible and natural speech given text, is a hot research topic in speech,&#10;language, and machine learning communities and has broad applications in the&#10;industry. As the development of deep learning and artificial intelligence,&#10;neural network-based TTS has significantly improved the quality of synthesized&#10;speech in recent years. In this paper, we conduct a comprehensive survey on&#10;neural TTS, aiming to provide a good understanding of current research and&#10;future trends. We focus on the key components in neural TTS, including text&#10;analysis, acoustic models and vocoders, and several advanced topics, including&#10;fast TTS, low-resource TTS, robust TTS, expressive TTS, and adaptive TTS, etc.&#10;We further summarize resources related to TTS (e.g., datasets, opensource&#10;implementations) and discuss future research directions. This survey can serve&#10;both academic researchers and industry practitioners working on TTS.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.06006" label="1802.06006">
        <attvalues>
          <attvalue for="0" value="Neural Voice Cloning with a Few Samples" />
          <attvalue for="1" value="  Voice cloning is a highly desired feature for personalized speech interfaces.&#10;Neural network based speech synthesis has been shown to generate high quality&#10;speech for a large number of speakers. In this paper, we introduce a neural&#10;voice cloning system that takes a few audio samples as input. We study two&#10;approaches: speaker adaptation and speaker encoding. Speaker adaptation is&#10;based on fine-tuning a multi-speaker generative model with a few cloning&#10;samples. Speaker encoding is based on training a separate model to directly&#10;infer a new speaker embedding from cloning audios and to be used with a&#10;multi-speaker generative model. In terms of naturalness of the speech and its&#10;similarity to original speaker, both approaches can achieve good performance,&#10;even with very few cloning audios. While speaker adaptation can achieve better&#10;naturalness and similarity, the cloning time or required memory for the speaker&#10;encoding approach is significantly less, making it favorable for low-resource&#10;deployment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.00993" label="2103.00993">
        <attvalues>
          <attvalue for="0" value="AdaSpeech: Adaptive Text to Speech for Custom Voice" />
          <attvalue for="1" value="  Custom voice, a specific text to speech (TTS) service in commercial speech&#10;platforms, aims to adapt a source TTS model to synthesize personal voice for a&#10;target speaker using few speech data. Custom voice presents two unique&#10;challenges for TTS adaptation: 1) to support diverse customers, the adaptation&#10;model needs to handle diverse acoustic conditions that could be very different&#10;from source speech data, and 2) to support a large number of customers, the&#10;adaptation parameters need to be small enough for each target speaker to reduce&#10;memory usage while maintaining high voice quality. In this work, we propose&#10;AdaSpeech, an adaptive TTS system for high-quality and efficient customization&#10;of new voices. We design several techniques in AdaSpeech to address the two&#10;challenges in custom voice: 1) To handle different acoustic conditions, we use&#10;two acoustic encoders to extract an utterance-level vector and a sequence of&#10;phoneme-level vectors from the target speech during training; in inference, we&#10;extract the utterance-level vector from a reference speech and use an acoustic&#10;predictor to predict the phoneme-level vectors. 2) To better trade off the&#10;adaptation parameters and voice quality, we introduce conditional layer&#10;normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this&#10;part in addition to speaker embedding for adaptation. We pre-train the source&#10;TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets&#10;(with different acoustic conditions from LibriTTS) with few adaptation data,&#10;e.g., 20 sentences, about 1 minute speech. Experiment results show that&#10;AdaSpeech achieves much better adaptation quality than baseline methods, with&#10;only about 5K specific parameters for each speaker, which demonstrates its&#10;effectiveness for custom voice. Audio samples are available at&#10;https://speechresearch.github.io/adaspeech/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.04558" label="1806.04558">
        <attvalues>
          <attvalue for="0" value="Transfer Learning from Speaker Verification to Multispeaker&#10;  Text-To-Speech Synthesis" />
          <attvalue for="1" value="  We describe a neural network-based system for text-to-speech (TTS) synthesis&#10;that is able to generate speech audio in the voice of many different speakers,&#10;including those unseen during training. Our system consists of three&#10;independently trained components: (1) a speaker encoder network, trained on a&#10;speaker verification task using an independent dataset of noisy speech from&#10;thousands of speakers without transcripts, to generate a fixed-dimensional&#10;embedding vector from seconds of reference speech from a target speaker; (2) a&#10;sequence-to-sequence synthesis network based on Tacotron 2, which generates a&#10;mel spectrogram from text, conditioned on the speaker embedding; (3) an&#10;auto-regressive WaveNet-based vocoder that converts the mel spectrogram into a&#10;sequence of time domain waveform samples. We demonstrate that the proposed&#10;model is able to transfer the knowledge of speaker variability learned by the&#10;discriminatively-trained speaker encoder to the new task, and is able to&#10;synthesize natural speech from speakers that were not seen during training. We&#10;quantify the importance of training the speaker encoder on a large and diverse&#10;speaker set in order to obtain the best generalization performance. Finally, we&#10;show that randomly sampled speaker embeddings can be used to synthesize speech&#10;in the voice of novel speakers dissimilar from those used in training,&#10;indicating that the model has learned a high quality speaker representation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.07217" label="1810.07217">
        <attvalues>
          <attvalue for="0" value="Hierarchical Generative Modeling for Controllable Speech Synthesis" />
          <attvalue for="1" value="  This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model&#10;which can control latent attributes in the generated speech that are rarely&#10;annotated in the training data, such as speaking style, accent, background&#10;noise, and recording conditions. The model is formulated as a conditional&#10;generative model based on the variational autoencoder (VAE) framework, with two&#10;levels of hierarchical latent variables. The first level is a categorical&#10;variable, which represents attribute groups (e.g. clean/noisy) and provides&#10;interpretability. The second level, conditioned on the first, is a multivariate&#10;Gaussian variable, which characterizes specific attribute configurations (e.g.&#10;noise level, speaking rate) and enables disentangled fine-grained control over&#10;these attributes. This amounts to using a Gaussian mixture model (GMM) for the&#10;latent distribution. Extensive evaluation demonstrates its ability to control&#10;the aforementioned attributes. In particular, we train a high-quality&#10;controllable TTS model on real found data, which is capable of inferring&#10;speaker and style attributes from a noisy utterance and use it to synthesize&#10;clean speech with controllable speaking style.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.02479" label="1907.02479">
        <attvalues>
          <attvalue for="0" value="Fine-grained robust prosody transfer for single-speaker neural&#10;  text-to-speech" />
          <attvalue for="1" value="  We present a neural text-to-speech system for fine-grained prosody transfer&#10;from one speaker to another. Conventional approaches for end-to-end prosody&#10;transfer typically use either fixed-dimensional or variable-length prosody&#10;embedding via a secondary attention to encode the reference signal. However,&#10;when trained on a single-speaker dataset, the conventional prosody transfer&#10;systems are not robust enough to speaker variability, especially in the case of&#10;a reference signal coming from an unseen speaker. Therefore, we propose&#10;decoupling of the reference signal alignment from the overall system. For this&#10;purpose, we pre-compute phoneme-level time stamps and use them to aggregate&#10;prosodic features per phoneme, injecting them into a sequence-to-sequence&#10;text-to-speech system. We incorporate a variational auto-encoder to further&#10;enhance the latent representation of prosody embeddings. We show that our&#10;proposed approach is significantly more stable and achieves reliable prosody&#10;transplantation from an unseen speaker. We also propose a solution to the use&#10;case in which the transcription of the reference signal is absent. We evaluate&#10;all our proposed methods using both objective and subjective listening tests.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.08484" label="2005.08484">
        <attvalues>
          <attvalue for="0" value="Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based&#10;  Variable-Length Embedding" />
          <attvalue for="1" value="  On account of growing demands for personalization, the need for a so-called&#10;few-shot TTS system that clones speakers with only a few data is emerging. To&#10;address this issue, we propose Attentron, a few-shot TTS model that clones&#10;voices of speakers unseen during training. It introduces two special encoders,&#10;each serving different purposes. A fine-grained encoder extracts&#10;variable-length style information via an attention mechanism, and a&#10;coarse-grained encoder greatly stabilizes the speech synthesis, circumventing&#10;unintelligible gibberish even for synthesizing speech of unseen speakers. In&#10;addition, the model can scale out to an arbitrary number of reference audios to&#10;improve the quality of the synthesized speech. According to our experiments,&#10;including a human evaluation, the proposed model significantly outperforms&#10;state-of-the-art models when generating speech for unseen speakers in terms of&#10;speaker similarity and quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.05716" label="2005.05716">
        <attvalues>
          <attvalue for="0" value="AttViz: Online exploration of self-attention for transparent neural&#10;  language modeling" />
          <attvalue for="1" value="  Neural language models are becoming the prevailing methodology for the tasks&#10;of query answering, text classification, disambiguation, completion and&#10;translation. Commonly comprised of hundreds of millions of parameters, these&#10;neural network models offer state-of-the-art performance at the cost of&#10;interpretability; humans are no longer capable of tracing and understanding how&#10;decisions are being made. The attention mechanism, introduced initially for the&#10;task of translation, has been successfully adopted for other language-related&#10;tasks. We propose AttViz, an online toolkit for exploration of&#10;self-attention---real values associated with individual text tokens. We show&#10;how existing deep learning pipelines can produce outputs suitable for AttViz,&#10;offering novel visualizations of the attention heads and their aggregations&#10;with minimal effort, online. We show on examples of news segments how the&#10;proposed system can be used to inspect and potentially better understand what a&#10;model has learned (or emphasized).&#10;" />
          <attvalue for="2" value="&#10;Contemporary machine learning that addresses text-related tasks adheres to the use of large language models---deep neural network architectures that have gone through extensive unsupervised pre-training in order to capture context-dependent meaning of individual tokens \cite{devlin2019bert,liu2019roberta,yang2019xlnet}. Even though pre-training of such multi-million parameter neural networks can be expensive \cite{radford2019language}, many pre-trained models have been made freely available to the wider research community, unveiling the opportunity for the exploration of how, and why such large models perform well.&#10;One of the main problems with neural language models is their interpretability. Even though the models learn the task well (even at super-human levels), understanding the reasons for predictions and inspection of whether the models picked up irrelevant biases or spurious correlations can be a non-trivial task.&#10;&#10;Approaches to understanding black-box (non-interpretable) neural network models often resort to post-hoc approximations, e.g., SHAP \cite{lundberg2017unified}, and similar are not necessary internal to the model itself. &#10;A potential way of extracting the token relevance is the attention mechanism \cite{bahdanau;2014,luong2015effective}. The attention mechanism learns token pair-value mappings, potentially encoding relations between token pairs.&#10;When inspected as self-relations, the attention of a token w.r.t. itself (the diagonal element of the token attention matrix) potentially offers some insight into the importance of that token. Similar findings were also recently discussed when considering tabular data \cite{arik2019tabnet}.&#10;However, analytically, as well as numerically, exploration of attention can be a cumbersome task, resulting in the rise of approaches aimed at attention visualization.&#10;Visualization of (latent) embedding spaces is becoming ubiquitous in contemporary machine learning. For example, the Google's Embedding Projector has offered numerous visualizations for non-savvy users, making embedding projections to low dimensional (human-understandable) vector spaces simple and available online. Even though visualization of simple embedding spaces is already accessible, visualization of complex neural network models' interior representations distributed across multiple embeddings (e.g., attention vectors), however, can be a challenging task.&#10;The works of \cite{liu2018visual} and \cite{8614007} are examples of attempts at unveiling the workings of black-box attention layers and offering an interface for human researches to learn and inspect their models.\cite{liu2018visual} visualize, as well as offer possible coloring of the attention space. &#10;Further, \cite{8614007} visualized self-attention with examples in sentiment analysis. The main contributions of AttViz are multi-fold, and can be stated as follows.&#10;AttViz focuses exclusively on self-attention and introduces two novel ways of visualizing this property while being available online and accessible to a wider audience. AttViz can interactively aggregate the attention vectors and offers simultaneous exploration of the output probability space, as well as the attention space.&#10;&#10;The remainder of this work is structured as follows. In Section~\ref{sec:attentionviz}, we discuss the works, related to the proposed AttViz approach. In Section~\ref{sec:AttViz}, we present the key ideas and technical implementation of AttViz, followed by our use case -- a study of news segments in Section~\ref{sec:usage}. Finally, we discuss (in Section~\ref{sec:discussion}) the overall capabilities of AttViz.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Interpretability, Computer Science, Linguistics, Cognitive Science, Attention Mechanism, Neural Language Models" />
        </attvalues>
      </node>
      <node id="1705.07874" label="1705.07874">
        <attvalues>
          <attvalue for="0" value="A Unified Approach to Interpreting Model Predictions" />
          <attvalue for="1" value="  Understanding why a model makes a certain prediction can be as crucial as the&#10;prediction's accuracy in many applications. However, the highest accuracy for&#10;large modern datasets is often achieved by complex models that even experts&#10;struggle to interpret, such as ensemble or deep learning models, creating a&#10;tension between accuracy and interpretability. In response, various methods&#10;have recently been proposed to help users interpret the predictions of complex&#10;models, but it is often unclear how these methods are related and when one&#10;method is preferable over another. To address this problem, we present a&#10;unified framework for interpreting predictions, SHAP (SHapley Additive&#10;exPlanations). SHAP assigns each feature an importance value for a particular&#10;prediction. Its novel components include: (1) the identification of a new class&#10;of additive feature importance measures, and (2) theoretical results showing&#10;there is a unique solution in this class with a set of desirable properties.&#10;The new class unifies six existing methods, notable because several recent&#10;methods in the class lack the proposed desirable properties. Based on insights&#10;from this unification, we present new methods that show improved computational&#10;performance and/or better consistency with human intuition than previous&#10;approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.07442" label="1908.07442">
        <attvalues>
          <attvalue for="0" value="TabNet: Attentive Interpretable Tabular Learning" />
          <attvalue for="1" value="  We propose a novel high-performance and interpretable canonical deep tabular&#10;data learning architecture, TabNet. TabNet uses sequential attention to choose&#10;which features to reason from at each decision step, enabling interpretability&#10;and more efficient learning as the learning capacity is used for the most&#10;salient features. We demonstrate that TabNet outperforms other neural network&#10;and decision tree variants on a wide range of non-performance-saturated tabular&#10;datasets and yields interpretable feature attributions plus insights into the&#10;global model behavior. Finally, for the first time to our knowledge, we&#10;demonstrate self-supervised learning for tabular data, significantly improving&#10;performance with unsupervised representation learning when unlabeled data is&#10;abundant.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.12764" label="2312.12764">
        <attvalues>
          <attvalue for="0" value="Lattice Rescoring Based on Large Ensemble of Complementary Neural&#10;  Language Models" />
          <attvalue for="1" value="  We investigate the effectiveness of using a large ensemble of advanced neural&#10;language models (NLMs) for lattice rescoring on automatic speech recognition&#10;(ASR) hypotheses. Previous studies have reported the effectiveness of combining&#10;a small number of NLMs. In contrast, in this study, we combine up to eight&#10;NLMs, i.e., forward/backward long short-term memory/Transformer-LMs that are&#10;trained with two different random initialization seeds. We combine these NLMs&#10;through iterative lattice generation. Since these NLMs work complementarily&#10;with each other, by combining them one by one at each rescoring iteration,&#10;language scores attached to given lattice arcs can be gradually refined.&#10;Consequently, errors of the ASR hypotheses can be gradually reduced. We also&#10;investigate the effectiveness of carrying over contextual information (previous&#10;rescoring results) across a lattice sequence of a long speech such as a lecture&#10;speech. In experiments using a lecture speech corpus, by combining the eight&#10;NLMs and using context carry-over, we obtained a 24.4% relative word error rate&#10;reduction from the ASR 1-best baseline. For further comparison, we performed&#10;simultaneous (i.e., non-iterative) NLM combination and 100-best rescoring using&#10;the large ensemble of NLMs, which confirmed the advantage of lattice rescoring&#10;with iterative NLM combination.&#10;" />
          <attvalue for="2" value="&#10;Based on the&#10;recent&#10;introduction&#10;of state-of-the-art neural network (NN) modeling,&#10;the performance&#10;of automatic speech recognition (ASR)&#10;has&#10;been&#10;greatly improved&#10;\cite{Hinton_IEEESPM2012,Yu_Springer2015},&#10;and various types of&#10;ASR-based&#10;applications,&#10;including&#10;voice search services and smart speakers,&#10;have been actively developed.&#10;Despite this great progress,&#10;in&#10;some&#10;situations such as&#10;performing ASR in noisy environments&#10;or&#10;performing ASR&#10;for&#10;conversational&#10;speech,&#10;the accuracy of ASR&#10;remains at an unsatisfactory level&#10;\cite{Saon_IS2016,Saon_IS2017,Xiong_arXiv2017,Xiong_ICASSP2018,Barker_IS2018,Watanabe_CHiME6}.&#10;&#10;A promising&#10;approach&#10;for reducing ASR errors&#10;in such severe situations&#10;involves&#10;the use of&#10;multiple&#10;ASR&#10;hypotheses&#10;(word sequences),&#10;which are represented&#10;in such forms as&#10;an $N$-best list&#10;or&#10;a lattice.&#10;This is because&#10;an ASR&#10;hypothesis that has a significantly lower word error rate (WER)&#10;than the 1-best hypothesis&#10;can be found in&#10;multiple&#10;hypotheses&#10;if it is appropriately rescored&#10;(reranked).&#10;Various&#10;types of rescoring methods have been&#10;developed and applied&#10;to noisy&#10;or&#10;conversational speech recognition&#10;\cite{Saon_IS2016,Saon_IS2017,Xiong_arXiv2017,Xiong_ICASSP2018,Du_CHiME5,Kanda_CHiME5,Zhao_CHiME5,Li_CHiME5,Medennikov_CHiME6,Arora_CHiME6,Zorila_CHiME6,Zmolikova_CHiME6,Irie_IS2018,Xiong_EMNLP2018,Irie_IS2019,Parthasarathy_arXiv2019,Irie_ASRU2019,Sun_ICASSP2021}.&#10;&#10;In these rescoring methods,&#10;advanced neural language models (NLMs)&#10;are used&#10;as rescoring models.&#10;They&#10;can&#10;accurately&#10;model&#10;much&#10;longer word sequences&#10;than can&#10;conventional&#10;count-based $n$-gram LMs&#10;\cite{Jelinek_MITPress1998,Kneser_ICASSP1995},&#10;which can&#10;model&#10;sequences of only $n$ words&#10;(where $n$ is typically three to five).&#10;These NLMs are used&#10;to refine language scores&#10;attached to ASR hypotheses&#10;that&#10;are calculated using&#10;the $n$-gram LMs.&#10;Among the&#10;NLMs,&#10;long short-term memory (LSTM)-based recurrent NLMs&#10;\cite{Sundermeyer_IS2012}&#10;are&#10;currently&#10;the most widely used&#10;model.&#10;A forward LSTMLM can provide good WER reduction,&#10;but&#10;the WER&#10;can be further reduced &#10;by additionally using another model.&#10;Such a model&#10;would be,&#10;for example,&#10;a forward LSTMLM that has a different model&#10;structure&#10;\cite{Li_CHiME5,Medennikov_CHiME6}&#10;or that is&#10;trained with a different setting&#10;(e.g., a different initialization seed or a different data shuffling scheme)&#10;\cite{Xiong_arXiv2017,Irie_IS2018}&#10;or&#10;a backward LSTMLM&#10;that is&#10;trained by using&#10;a reversed text dataset&#10;\cite{Xiong_arXiv2017,Xiong_ICASSP2018,Irie_IS2018,Kanda_CHiME5,Arora_CHiME6},&#10;since these&#10;models&#10;work complementarily with each other.&#10;In addition to the LSTMLMs,&#10;NLMs based on Transformers&#10;\cite{Vaswani_NIPS2017}&#10;have recently been used&#10;for rescoring.&#10;They &#10;have a non-recurrent&#10;self-attentive&#10;architecture&#10;that&#10;is completely different from that of the LSTMLMs,&#10;and&#10;they&#10;show comparable or superior&#10;rescoring performance to the LSTMLMs&#10;\cite{Irie_IS2019,Irie_ASRU2019,Sun_ICASSP2021}.&#10;&#10;By performing ASR for a long speech&#10;such as a lecture speech,&#10;a long ASR hypothesis sequence&#10;can be obtained.&#10;In such a long speech&#10;(a series of utterances),&#10;the content of an utterance is naturally influenced&#10;by the content of previous utterances (i.e., context).&#10;Therefore,&#10;in rescoring&#10;such a&#10;long ASR hypothesis sequence,&#10;it is reasonable&#10;to use the rescoring results of the previous hypotheses&#10;as&#10;contextual information&#10;for rescoring the current hypothesis.&#10;It has been reported that,&#10;by&#10;carrying over&#10;contextual information&#10;across ASR hypotheses,&#10;the rescoring performance&#10;for such a long ASR hypothesis sequence&#10;can be improved&#10;\cite{Xiong_arXiv2017,Xiong_ICASSP2018,Xiong_EMNLP2018,Parthasarathy_arXiv2019,Irie_ASRU2019,Zmolikova_CHiME6,Sun_ICASSP2021}.&#10;&#10;In this study,&#10;we investigate the effectiveness of using a large ensemble of NLMs&#10;on lattice rescoring.&#10;As described above,&#10;previous studies&#10;\cite{Saon_IS2016,Saon_IS2017,Li_CHiME5,Medennikov_CHiME6,Arora_CHiME6}&#10;have reported the effectiveness of combining&#10;a small number of NLMs&#10;(up to four \cite{Medennikov_CHiME6})&#10;on lattice rescoring.&#10;In contrast,&#10;we combine up to eight NLMs,&#10;i.e.,&#10;forward/backward LSTM/Transformer-LMs,&#10;which are trained with two different random initialization seeds.&#10;We combine these complementary NLMs through iterative lattice generation&#10;while introducing context carry-over&#10;(Section~\ref{sec_lr}).&#10;We conducted experiments&#10;including&#10;experimental settings that have not&#10;been investigated&#10;in previous studies&#10;(Section~\ref{sec_rel})&#10;and confirmed&#10;the effectiveness of&#10;using a&#10;large ensemble of&#10;NLMs&#10;for&#10;lattice rescoring&#10;(Section~\ref{sec_exp}).&#10;Our main findings can be summarized&#10;as follows.&#10;&#10;\begin{enumerate}&#10;\item Combining&#10;six or seven NLMs can improve&#10;the performance of lattice rescoring.&#10;\item Lattice rescoring has an advantage over $N$-best rescoring&#10;when using&#10;a large ensemble of NLMs.&#10;\item Performing context carry-over in the backward direction is as effective&#10;as performing it in the forward direction.&#10;\item Iterative NLM combination has the potential to outperform simultaneous&#10;NLM combination,&#10;especially in a fast lattice rescoring setting.&#10;\end{enumerate}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Recognition Systems, Linguistics, Engineering, Signal Processing, Neural Language Models, Ensemble Learning Methods" />
        </attvalues>
      </node>
      <node id="1604.08242" label="1604.08242">
        <attvalues>
          <attvalue for="0" value="The IBM 2016 English Conversational Telephone Speech Recognition System" />
          <attvalue for="1" value="  We describe a collection of acoustic and language modeling techniques that&#10;lowered the word error rate of our English conversational telephone LVCSR&#10;system to a record 6.6% on the Switchboard subset of the Hub5 2000 evaluation&#10;testset. On the acoustic side, we use a score fusion of three strong models:&#10;recurrent nets with maxout activations, very deep convolutional nets with 3x3&#10;kernels, and bidirectional long short-term memory nets which operate on FMLLR&#10;and i-vector features. On the language modeling side, we use an updated model&#10;&quot;M&quot; and hierarchical neural network LMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.05256" label="1610.05256">
        <attvalues>
          <attvalue for="0" value="Achieving Human Parity in Conversational Speech Recognition" />
          <attvalue for="1" value="  Conversational speech recognition has served as a flagship speech recognition&#10;task since the release of the Switchboard corpus in the 1990s. In this paper,&#10;we measure the human error rate on the widely used NIST 2000 test set, and find&#10;that our latest automated system has reached human parity. The error rate of&#10;professional transcribers is 5.9% for the Switchboard portion of the data, in&#10;which newly acquainted pairs of people discuss an assigned topic, and 11.3% for&#10;the CallHome portion where friends and family members have open-ended&#10;conversations. In both cases, our automated system establishes a new state of&#10;the art, and edges past the human benchmark, achieving error rates of 5.8% and&#10;11.0%, respectively. The key to our system's performance is the use of various&#10;convolutional and LSTM acoustic model architectures, combined with a novel&#10;spatial smoothing method and lattice-free MMI acoustic training, multiple&#10;recurrent neural network language modeling approaches, and a systematic use of&#10;system combination.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.06073" label="1708.06073">
        <attvalues>
          <attvalue for="0" value="The Microsoft 2017 Conversational Speech Recognition System" />
          <attvalue for="1" value="  We describe the 2017 version of Microsoft's conversational speech recognition&#10;system, in which we update our 2016 system with recent developments in&#10;neural-network-based acoustic and language modeling to further advance the&#10;state of the art on the Switchboard speech recognition task. The system adds a&#10;CNN-BLSTM acoustic model to the set of model architectures we combined&#10;previously, and includes character-based and dialog session aware LSTM language&#10;models in rescoring. For system combination we adopt a two-stage approach,&#10;whereby subsets of acoustic models are first combined at the senone/frame&#10;level, followed by a word-level voting via confusion networks. We also added a&#10;confusion network rescoring step after system combination. The resulting system&#10;yields a 5.1\% word error rate on the 2000 Switchboard evaluation set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.07898" label="2006.07898">
        <attvalues>
          <attvalue for="0" value="The JHU Multi-Microphone Multi-Speaker ASR System for the CHiME-6&#10;  Challenge" />
          <attvalue for="1" value="  This paper summarizes the JHU team's efforts in tracks 1 and 2 of the CHiME-6&#10;challenge for distant multi-microphone conversational speech diarization and&#10;recognition in everyday home environments. We explore multi-array processing&#10;techniques at each stage of the pipeline, such as multi-array guided source&#10;separation (GSS) for enhancement and acoustic model training data, posterior&#10;fusion for speech activity detection, PLDA score fusion for diarization, and&#10;lattice combination for automatic speech recognition (ASR). We also report&#10;results with different acoustic model architectures, and integrate other&#10;techniques such as online multi-channel weighted prediction error (WPE)&#10;dereverberation and variational Bayes-hidden Markov model (VB-HMM) based&#10;overlap assignment to deal with reverberation and overlapping speakers,&#10;respectively. As a result of these efforts, our ASR systems achieve a word&#10;error rate of 40.5% and 67.5% on tracks 1 and 2, respectively, on the&#10;evaluation set. This is an improvement of 10.8% and 10.4% absolute, over the&#10;challenge baselines for the respective tracks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.06474" label="2102.06474">
        <attvalues>
          <attvalue for="0" value="Transformer Language Models with LSTM-based Cross-utterance Information&#10;  Representation" />
          <attvalue for="1" value="  The effective incorporation of cross-utterance information has the potential&#10;to improve language models (LMs) for automatic speech recognition (ASR). To&#10;extract more powerful and robust cross-utterance representations for the&#10;Transformer LM (TLM), this paper proposes the R-TLM which uses hidden states in&#10;a long short-term memory (LSTM) LM. To encode the cross-utterance information,&#10;the R-TLM incorporates an LSTM module together with a segment-wise recurrence&#10;in some of the Transformer blocks. In addition to the LSTM module output, a&#10;shortcut connection using a fusion layer that bypasses the LSTM module is also&#10;investigated. The proposed system was evaluated on the AMI meeting corpus, the&#10;Eval2000 and the RT03 telephone conversation evaluation sets. The best R-TLM&#10;achieved 0.9%, 0.6%, and 0.8% absolute WER reductions over the single-utterance&#10;TLM baseline, and 0.5%, 0.3%, 0.2% absolute WER reductions over a strong&#10;cross-utterance TLM baseline on the AMI evaluation set, Eval2000 and RT03&#10;respectively. Improvements on Eval2000 and RT03 were further supported by&#10;significance tests. R-TLMs were found to have better LM scores on words where&#10;recognition errors are more likely to occur. The R-TLM WER can be further&#10;reduced by interpolation with an LSTM-LM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00696" label="2105.00696">
        <attvalues>
          <attvalue for="0" value="Graph Learning: A Survey" />
          <attvalue for="1" value="  Graphs are widely used as a popular representation of the network structure&#10;of connected data. Graph data can be found in a broad spectrum of application&#10;domains such as social systems, ecosystems, biological networks, knowledge&#10;graphs, and information systems. With the continuous penetration of artificial&#10;intelligence technologies, graph learning (i.e., machine learning on graphs) is&#10;gaining attention from both researchers and practitioners. Graph learning&#10;proves effective for many tasks, such as classification, link prediction, and&#10;matching. Generally, graph learning methods extract relevant features of graphs&#10;by taking advantage of machine learning algorithms. In this survey, we present&#10;a comprehensive overview on the state-of-the-art of graph learning. Special&#10;attention is paid to four categories of existing graph learning methods,&#10;including graph signal processing, matrix factorization, random walk, and deep&#10;learning. Major models and algorithms under these categories are reviewed&#10;respectively. We examine graph learning applications in areas such as text,&#10;images, science, knowledge graphs, and combinatorial optimization. In addition,&#10;we discuss several promising research directions in this field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.09796" label="1910.09796">
        <attvalues>
          <attvalue for="0" value="Fine-grained Fact Verification with Kernel Graph Attention Network" />
          <attvalue for="1" value="  Fact Verification requires fine-grained natural language inference capability&#10;that finds subtle clues to identify the syntactical and semantically correct&#10;but not well-supported claims. This paper presents Kernel Graph Attention&#10;Network (KGAT), which conducts more fine-grained fact verification with&#10;kernel-based attentions. Given a claim and a set of potential evidence&#10;sentences that form an evidence graph, KGAT introduces node kernels, which&#10;better measure the importance of the evidence node, and edge kernels, which&#10;conduct fine-grained evidence propagation in the graph, into Graph Attention&#10;Networks for more accurate fact verification. KGAT achieves a 70.38% FEVER&#10;score and significantly outperforms existing fact verification models on FEVER,&#10;a large-scale benchmark for fact verification. Our analyses illustrate that,&#10;compared to dot-product attentions, the kernel-based attention concentrates&#10;more on relevant evidence sentences and meaningful clues in the evidence graph,&#10;which is the main source of KGAT's effectiveness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.10149" label="2304.10149">
        <attvalues>
          <attvalue for="0" value="Is ChatGPT a Good Recommender? A Preliminary Study" />
          <attvalue for="1" value="  Recommendation systems have witnessed significant advancements and have been&#10;widely used over the past decades. However, most traditional recommendation&#10;methods are task-specific and therefore lack efficient generalization ability.&#10;Recently, the emergence of ChatGPT has significantly advanced NLP tasks by&#10;enhancing the capabilities of conversational models. Nonetheless, the&#10;application of ChatGPT in the recommendation domain has not been thoroughly&#10;investigated. In this paper, we employ ChatGPT as a general-purpose&#10;recommendation model to explore its potential for transferring extensive&#10;linguistic and world knowledge acquired from large-scale corpora to&#10;recommendation scenarios. Specifically, we design a set of prompts and evaluate&#10;ChatGPT's performance on five recommendation scenarios. Unlike traditional&#10;recommendation methods, we do not fine-tune ChatGPT during the entire&#10;evaluation process, relying only on the prompts themselves to convert&#10;recommendation tasks into natural language tasks. Further, we explore the use&#10;of few-shot prompting to inject interaction information that contains user&#10;potential interest to help ChatGPT better understand user needs and interests.&#10;Comprehensive experimental results on Amazon Beauty dataset show that ChatGPT&#10;has achieved promising results in certain tasks and is capable of reaching the&#10;baseline level in others. We conduct human evaluations on two&#10;explainability-oriented tasks to more accurately evaluate the quality of&#10;contents generated by different models. And the human evaluations show ChatGPT&#10;can truly understand the provided information and generate clearer and more&#10;reasonable results. We hope that our study can inspire researchers to further&#10;explore the potential of language models like ChatGPT to improve recommendation&#10;performance and contribute to the advancement of the recommendation systems&#10;field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.07610" label="2303.07610">
        <attvalues>
          <attvalue for="0" value="Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on&#10;  Consistency with Human Preferences" />
          <attvalue for="1" value="  As a natural language assistant, ChatGPT is capable of performing various&#10;tasks, including but not limited to article generation, code completion, and&#10;data analysis. Furthermore, ChatGPT has consistently demonstrated a remarkable&#10;level of accuracy and reliability in terms of content evaluation, exhibiting&#10;the capability of mimicking human preferences. To further explore ChatGPT's&#10;potential in this regard, a study is conducted to assess its ability to rank&#10;content. In order to do so, a test set consisting of prompts is created,&#10;covering a wide range of use cases, and five models are utilized to generate&#10;corresponding responses. ChatGPT is then instructed to rank the responses&#10;generated by these models. The results on the test set show that ChatGPT's&#10;ranking preferences are consistent with human to a certain extent. This&#10;preliminary experimental finding implies that ChatGPT's zero-shot ranking&#10;capability could be used to reduce annotation pressure in a number of ranking&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.09712" label="2205.09712">
        <attvalues>
          <attvalue for="0" value="Selection-Inference: Exploiting Large Language Models for Interpretable&#10;  Logical Reasoning" />
          <attvalue for="1" value="  Large language models (LLMs) have been shown to be capable of impressive&#10;few-shot generalisation to new tasks. However, they still tend to perform&#10;poorly on multi-step logical reasoning problems. Here we carry out a&#10;comprehensive evaluation of LLMs on 50 tasks that probe different aspects of&#10;logical reasoning. We show that language models tend to perform fairly well at&#10;single step inference or entailment tasks, but struggle to chain together&#10;multiple reasoning steps to solve more complex problems. In light of this, we&#10;propose a Selection-Inference (SI) framework that exploits pre-trained LLMs as&#10;general processing modules, and alternates between selection and inference to&#10;generate a series of interpretable, casual reasoning steps leading to the final&#10;answer. We show that a 7B parameter LLM used within the SI framework in a&#10;5-shot generalisation setting, with no fine-tuning, yields a performance&#10;improvement of over 100% compared to an equivalent vanilla baseline on a suite&#10;of 10 logical reasoning tasks. The same model in the same setting even&#10;outperforms a significantly larger 280B parameter baseline on the same suite of&#10;tasks. Moreover, answers produced by the SI framework are accompanied by a&#10;causal natural-language-based reasoning trace, which has important implications&#10;for the safety and trustworthiness of the system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.03514" label="2201.03514">
        <attvalues>
          <attvalue for="0" value="Black-Box Tuning for Language-Model-as-a-Service" />
          <attvalue for="1" value="  Extremely large pre-trained language models (PTMs) such as GPT-3 are usually&#10;released as a service. It allows users to design task-specific prompts to query&#10;the PTMs through some black-box APIs. In such a scenario, which we call&#10;Language-Model-as-a-Service (LMaaS), the gradients of PTMs are usually&#10;unavailable. Can we optimize the task prompts by only accessing the model&#10;inference APIs? This paper proposes the black-box tuning framework to optimize&#10;the continuous prompt prepended to the input text via derivative-free&#10;optimization. Instead of optimizing in the original high-dimensional prompt&#10;space, which is intractable for traditional derivative-free optimization, we&#10;perform optimization in a randomly generated subspace due to the low intrinsic&#10;dimensionality of large PTMs. The experimental results show that the black-box&#10;tuning with RoBERTa on a few labeled samples not only significantly outperforms&#10;manual prompt and GPT-3's in-context learning, but also surpasses the&#10;gradient-based counterparts, i.e., prompt tuning and full model tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1609.03193" label="1609.03193">
        <attvalues>
          <attvalue for="0" value="Wav2Letter: an End-to-End ConvNet-based Speech Recognition System" />
          <attvalue for="1" value="  This paper presents a simple end-to-end model for speech recognition,&#10;combining a convolutional network based acoustic model and a graph decoding. It&#10;is trained to output letters, with transcribed speech, without the need for&#10;force alignment of phonemes. We introduce an automatic segmentation criterion&#10;for training from sequence annotation without alignment that is on par with CTC&#10;while being simpler. We show competitive results in word error rate on the&#10;Librispeech corpus with MFCC features, and promising results from raw waveform.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.16804" label="2203.16804">
        <attvalues>
          <attvalue for="0" value="BRIO: Bringing Order to Abstractive Summarization" />
          <attvalue for="1" value="  Abstractive summarization models are commonly trained using maximum&#10;likelihood estimation, which assumes a deterministic (one-point) target&#10;distribution in which an ideal model will assign all the probability mass to&#10;the reference summary. This assumption may lead to performance degradation&#10;during inference, where the model needs to compare several system-generated&#10;(candidate) summaries that have deviated from the reference summary. To address&#10;this problem, we propose a novel training paradigm which assumes a&#10;non-deterministic distribution so that different candidate summaries are&#10;assigned probability mass according to their quality. Our method achieves a new&#10;state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07&#10;ROUGE-1) datasets. Further analysis also shows that our model can estimate&#10;probabilities of candidate summaries that are more correlated with their level&#10;of quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.17267" label="2312.17267">
        <attvalues>
          <attvalue for="0" value="Enhancing Low-Resource Relation Representations through Multi-View&#10;  Decoupling" />
          <attvalue for="1" value="  Recently, prompt-tuning with pre-trained language models (PLMs) has&#10;demonstrated the significantly enhancing ability of relation extraction (RE)&#10;tasks. However, in low-resource scenarios, where the available training data is&#10;scarce, previous prompt-based methods may still perform poorly for prompt-based&#10;representation learning due to a superficial understanding of the relation. To&#10;this end, we highlight the importance of learning high-quality relation&#10;representation in low-resource scenarios for RE, and propose a novel&#10;prompt-based relation representation method, named MVRE&#10;(\underline{M}ulti-\underline{V}iew \underline{R}elation&#10;\underline{E}xtraction), to better leverage the capacity of PLMs to improve the&#10;performance of RE within the low-resource prompt-tuning paradigm. Specifically,&#10;MVRE decouples each relation into different perspectives to encompass&#10;multi-view relation representations for maximizing the likelihood during&#10;relation inference. Furthermore, we also design a Global-Local loss and a&#10;Dynamic-Initialization method for better alignment of the multi-view&#10;relation-representing virtual words, containing the semantics of relation&#10;labels during the optimization learning process and initialization. Extensive&#10;experiments on three benchmark datasets show that our method can achieve&#10;state-of-the-art in low-resource settings.&#10;" />
          <attvalue for="2" value="&#10;&#10;Relation Extraction (RE) aims to extract the relation between two entities \cite{qu2023distantly,gu2022delving} from an unstructured text \cite{cheng2021hacred}. Given the significance of inter-entity relations within textual information, the practice of relation extraction finds extensive utility across various downstream tasks, including dialogue systems~\cite{lu2023miracle,liu2018knowledge}, information retrieval~\cite{yang2020biomedical,yu2023fusionint5}, information extraction \cite{zhu2023mirror,zhu2021efficient}, and question answering~\cite{yasunaga2021qa,qu2021passage}.&#10;&#10;Following the emergence of the paradigm involving pre-trained models and fine-tuning for downstream tasks~\cite{kenton2019bert,radford2018improving}, many recent relation extraction studies have embraced the utilization of large language models~\cite{ye2020coreferential,soares2019matching,zhou2022improved,ye2022packed}. In these works, the language models are integrated with classification heads and fine-tuned specifically for relation extraction tasks, resulting in promising results.&#10;However, the effective training of additional classification heads becomes challenging in situations where task-specific data is scarce. This challenge arises from the disparity between pre-training tasks, such as masked language modeling, and the subsequent fine-tuning tasks encompassing classification and regression. This divergence hampers the seamless adaptation of pre-trained language models (PLMs) to downstream tasks.&#10;&#10;Recently, prompt tuning has emerged as a promising direction for facilitating few-shot learning, which effectively bridges the gap between the pre-training and the downstream task~\cite{gao2021making,jin2023instance}. Conceptually, prompt-tuning involves template and verbalizer engineering, aiming to discover optimal templates and answer spaces. For example, as shown in Figure~\ref{fig:intro} (a), given a sentence ``Steve Jobs, co-founder of Apple&quot; for relation extraction, the text will first be enveloped with relation-specific templates, namely transforming the original relation extraction task into a relation-oriented cloze-style task. Subsequently, the PLM will predict words in the vocabulary to fill in the [MASK] position, and these predicted words are finally mapped to corresponding labels through a verbalizer. &#10;In this example, the filled word ``$[relation_1]$&quot; (e.g., ``founded&quot;) can be linked to the label ``org:founded\_by&quot; through the verbalizer. &#10;However, for complex relation representations, such as ``per: country\_of\_birth&quot; and ``org: city\_of\_headquarters,&quot; obtaining suitable vocabulary labels is much more challenging. To address this issue, previous work \cite{han2022ptr} applies logic rules to decompose complex relations into descriptions related to the subject and object entity types. Some works&#10; construct virtual words for each relation (a trainable ``$[relation_1]$&quot;) to substitute the corresponding answer space of the complex relation~\cite{chen2022knowprompt,chen2022relation}.&#10;This paradigm focuses on optimizing the relation representation space and demands PLMs to learn representations for words that are not present in the vocabulary. &#10;However, in extremely low-resource scenarios, such as one-shot RE, building robust relation representations with this paradigm is difficult, thus leading to a performance drop.&#10;&#10;To mitigate the above issue, in this paper, we introduce Multi-view Relation Extraction (MVRE), which improves low-resource prompt-based relation representations with a multi-view decoupling framework. As illustrated in Figure~\ref{fig:intro} (b), considering that relations may contain multiple dimensions of information, for instance, ``org:founded\_by&quot; may entail details about organizations, people's names, time, the action of founding, and so on. According to theoretical analysis, being limited to a single vector representation, the model may face the upper boundary of representation capacity and fail to construct robust representations in low-resource scenarios. Therefore, we propose to optimize the latent space by decoupling it into a joint optimization of multi-view relation representations, thereby maximizing the likelihood during relation inference. By sampling a greater number of relation representations, as denoted ``$[relation_{1-i}]$&quot; in Figure~\ref{fig:intro} (b)), we promote the learned latent space to include more kinds of information about the corresponding relation. In detail, we achieve this decoupling process by disassembling the virtual words into multiple components and predicting these components through successive [MASK] tokens. Furthermore, we introduce a Global-Local loss and Dynamic Initialization approach to optimize the process of relation representations by constraining semantic information of relations. We evaluate MVRE on three&#10;relation extraction datasets. Experimental results demonstrate that our&#10;method significantly outperforms previous approaches. To sum up, our main contributions are as follows:&#10;&#10;\begin{itemize}&#10;&#10;\item To the best of our knowledge, this paper presents the first attempt to improve low-resource prompt-based relation representations with multi-view decoupling learning. In this way, the PLM can be comprehensively utilized for generating robust relation representations from limited data. &#10;\item To optimize the learning process of multi-view relation representations, we introduce the Global-Local Loss and Dynamic Initialization to impose semantic constraints between virtual relation words.&#10;\item We conduct extensive experiments on three datasets and our proposed MVRE can achieve state-of-the-art performance in low-resource scenarios.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Artificial Intelligence, Relation Extraction, Representation Learning" />
        </attvalues>
      </node>
      <node id="2212.06522" label="2212.06522">
        <attvalues>
          <attvalue for="0" value="Distantly-Supervised Named Entity Recognition with Adaptive Teacher&#10;  Learning and Fine-grained Student Ensemble" />
          <attvalue for="1" value="  Distantly-Supervised Named Entity Recognition (DS-NER) effectively alleviates&#10;the data scarcity problem in NER by automatically generating training samples.&#10;Unfortunately, the distant supervision may induce noisy labels, thus&#10;undermining the robustness of the learned models and restricting the practical&#10;application. To relieve this problem, recent works adopt self-training&#10;teacher-student frameworks to gradually refine the training labels and improve&#10;the generalization ability of NER models. However, we argue that the&#10;performance of the current self-training frameworks for DS-NER is severely&#10;underestimated by their plain designs, including both inadequate student&#10;learning and coarse-grained teacher updating. Therefore, in this paper, we make&#10;the first attempt to alleviate these issues by proposing: (1) adaptive teacher&#10;learning comprised of joint training of two teacher-student networks and&#10;considering both consistent and inconsistent predictions between two teachers,&#10;thus promoting comprehensive student learning. (2) fine-grained student&#10;ensemble that updates each fragment of the teacher model with a temporal moving&#10;average of the corresponding fragment of the student, which enhances consistent&#10;predictions on each model fragment against noise. To verify the effectiveness&#10;of our proposed method, we conduct experiments on four DS-NER datasets. The&#10;experimental results demonstrate that our method significantly surpasses&#10;previous SOTA methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.18342" label="2310.18342">
        <attvalues>
          <attvalue for="0" value="MIRACLE: Towards Personalized Dialogue Generation with Latent-Space&#10;  Multiple Personal Attribute Control" />
          <attvalue for="1" value="  Personalized dialogue systems aim to endow the chatbot agent with more&#10;anthropomorphic traits for human-like interactions. Previous approaches have&#10;explored explicitly user profile modeling using text descriptions, implicit&#10;derivation of user embeddings, or utilizing handicraft prompts for ChatGPT-like&#10;models. However, textual personas are limited in describing multi-faceted&#10;attributes (\emph{e.g.}, \emph{language style, inner character nuances}),&#10;implicit embedding suffers from personality sparsity, and handicraft prompts&#10;lack fine-grained and stable controllability. Hence, these approaches may&#10;struggle with complex personalized dialogue generation tasks that require&#10;generating controllable responses with multiple personal attributes. To this&#10;end, we propose \textbf{\textsc{Miracle}}, a novel personalized dialogue&#10;generation method through \textbf{M}ult\textbf{I}ple Pe\textbf{R}sonal&#10;\textbf{A}ttributes \textbf{C}ontrol within \textbf{L}atent-Space&#10;\textbf{E}nergy-based Models. ttributes \textbf{C}ontrol within&#10;\textbf{L}atent-Space \textbf{E}nergy-based Models. Specifically, our approach&#10;first disentangles complex personality into multi-faceted attributes.&#10;Subsequently, we employ a conditional variational auto-encoder to align with&#10;the dense personalized responses within a latent joint attribute space. We have&#10;also tailored a dedicated energy function and customized the ordinary&#10;differential equations sampling method to offer flexible attribute composition&#10;and precise attribute control. Extensive experiments demonstrate that&#10;\textsc{Miracle} outperforms several strong baselines in terms of personality&#10;controllability and response generation quality. Our dataset and code are&#10;available at \url{https://github.com/LZY-the-boys/MIRACLE}&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14685" label="2305.14685">
        <attvalues>
          <attvalue for="0" value="Fusion-in-T5: Unifying Document Ranking Signals for Improved Information&#10;  Retrieval" />
          <attvalue for="1" value="  Common document ranking pipelines in search systems are cascade systems that&#10;involve multiple ranking layers to integrate different information&#10;step-by-step. In this paper, we propose a novel re-ranker Fusion-in-T5 (FiT5),&#10;which integrates text matching information, ranking features, and global&#10;document information into one single unified model via templated-based input&#10;and global attention. Experiments on passage ranking benchmarks MS MARCO and&#10;TREC DL show that FiT5, as one single model, significantly improves ranking&#10;performance over complex cascade pipelines. Analysis finds that through&#10;attention fusion, FiT5 jointly utilizes various forms of ranking information&#10;via gradually attending to related documents and ranking features, and improves&#10;the detection of subtle nuances. Our code is open-sourced at&#10;https://github.com/OpenMatch/FiT5.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.06013" label="2112.06013">
        <attvalues>
          <attvalue for="0" value="Efficient Document-level Event Extraction via Pseudo-Trigger-aware&#10;  Pruned Complete Graph" />
          <attvalue for="1" value="  Most previous studies of document-level event extraction mainly focus on&#10;building argument chains in an autoregressive way, which achieves a certain&#10;success but is inefficient in both training and inference. In contrast to the&#10;previous studies, we propose a fast and lightweight model named as PTPCG. In&#10;our model, we design a novel strategy for event argument combination together&#10;with a non-autoregressive decoding algorithm via pruned complete graphs, which&#10;are constructed under the guidance of the automatically selected pseudo&#10;triggers. Compared to the previous systems, our system achieves competitive&#10;results with 19.8\% of parameters and much lower resource consumption, taking&#10;only 3.8\% GPU hours for training and up to 8.5 times faster for inference.&#10;Besides, our model shows superior compatibility for the datasets with (or&#10;without) triggers and the pseudo triggers can be the supplements for annotated&#10;triggers to make further improvements. Codes are available at&#10;https://github.com/Spico197/DocEE .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.03938" label="2105.03938">
        <attvalues>
          <attvalue for="0" value="Passage Retrieval for Outside-Knowledge Visual Question Answering" />
          <attvalue for="1" value="  In this work, we address multi-modal information needs that contain text&#10;questions and images by focusing on passage retrieval for outside-knowledge&#10;visual question answering. This task requires access to outside knowledge,&#10;which in our case we define to be a large unstructured passage collection. We&#10;first conduct sparse retrieval with BM25 and study expanding the question with&#10;object names and image captions. We verify that visual clues play an important&#10;role and captions tend to be more informative than object names in sparse&#10;retrieval. We then construct a dual-encoder dense retriever, with the query&#10;encoder being LXMERT, a multi-modal pre-trained transformer. We further show&#10;that dense retrieval significantly outperforms sparse retrieval that uses&#10;object expansion. Moreover, dense retrieval matches the performance of sparse&#10;retrieval that leverages human-generated captions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.06870" label="2004.06870">
        <attvalues>
          <attvalue for="0" value="Coreferential Reasoning Learning for Language Representation" />
          <attvalue for="1" value="  Language representation models such as BERT could effectively capture&#10;contextual semantic information from plain text, and have been proved to&#10;achieve promising results in lots of downstream NLP tasks with appropriate&#10;fine-tuning. However, most existing language representation models cannot&#10;explicitly handle coreference, which is essential to the coherent understanding&#10;of the whole discourse. To address this issue, we present CorefBERT, a novel&#10;language representation model that can capture the coreferential relations in&#10;context. The experimental results show that, compared with existing baseline&#10;models, CorefBERT can achieve significant improvements consistently on various&#10;downstream NLP tasks that require coreferential reasoning, while maintaining&#10;comparable performance to previous models on other common NLP tasks. The source&#10;code and experiment details of this paper can be obtained from&#10;https://github.com/thunlp/CorefBERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.01373" label="2102.01373">
        <attvalues>
          <attvalue for="0" value="An Improved Baseline for Sentence-level Relation Extraction" />
          <attvalue for="1" value="  Sentence-level relation extraction (RE) aims at identifying the relationship&#10;between two entities in a sentence. Many efforts have been devoted to this&#10;problem, while the best performing methods are still far from perfect. In this&#10;paper, we revisit two problems that affect the performance of existing RE&#10;models, namely entity representation and noisy or ill-defined labels. Our&#10;improved RE baseline, incorporated with entity representations with typed&#10;markers, achieves an F1 of 74.6% on TACRED, significantly outperforms previous&#10;SOTA methods. Furthermore, the presented new baseline achieves an F1 of 91.1%&#10;on the refined Re-TACRED dataset, demonstrating that the pretrained language&#10;models (PLMs) achieve high performance on this task. We release our code to the&#10;community for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.07126" label="2201.07126">
        <attvalues>
          <attvalue for="0" value="Instance-aware Prompt Learning for Language Understanding and Generation" />
          <attvalue for="1" value="  Recently, prompt learning has become a new paradigm to utilize pre-trained&#10;language models (PLMs) and achieves promising results in downstream tasks with&#10;a negligible increase of parameters. The current usage of discrete and&#10;continuous prompts assumes that the prompt is fixed for a specific task and all&#10;samples in the task share the same prompt. However, a task may contain quite&#10;diverse samples in which some are easy and others are difficult, and diverse&#10;prompts are desirable. In this paper, we propose an instance-aware prompt&#10;learning method that learns a different prompt for each instance. Specifically,&#10;we suppose that each learnable prompt token has a different contribution to&#10;different instances, and we learn the contribution by calculating the relevance&#10;score between an instance and each prompt token. The contribution weighted&#10;prompt would be instance aware. We apply our method to both unidirectional and&#10;bidirectional PLMs on both language understanding and generation tasks.&#10;Extensive experiments demonstrate that our method obtains considerable&#10;improvements compared to strong baselines. Especially, our method achieves the&#10;state-of-the-art on the SuperGLUE few-shot learning benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;GPT-3 \cite{Brown2020LanguageMA}, which uses the task description and several typical examples as prompt to guide the generation, indicates the language models are few-shot learners and leads to the waves of prompt learning. Recently, PET/iPET \cite{schick-schutze-2021-just} utilizes the manually-designed prompts to reformulate natural language understanding tasks as cloze-style questions with gradient-based fine-tuning. &#10;There are also a lot of studies that utilize the manually-designed prompt to mine the knowledge from the PLMs \cite{jiang-etal-2020-know,Trinh2018ASM}.&#10;Since manual-designed prompt is time-consuming and the search space is huge, researches focus on automatic prompt search \cite{gao2020making,shin-etal-2020-autoprompt,zhong-etal-2021-factual}. &#10;&#10;However, the handcrafted prompt can only reflect rationality from the perspective of humans, which midwifery the exploration in continuous prompts. \cite{li-liang-2021-prefix} proposes prefix tuning and concatenates learnable prompt at each layer of transformer while only optimizing the prefix parameters. In contrast, prompt tuning \cite{lester2021power} concatenates learnable prompt only in the embedding layer and only optimizes the prompt parameters in the embedding layer. Although \cite{lester2021power} demonstrates the effectiveness of light-weight prompt-tuning, the gap with full parameter fine-tuning still exists especially when the PLM is small. &#10;&#10;There are also a lot of works that interleave the prompt throughout the input layer. \cite{hambardzumyan-etal-2021-warp} proposes WARP, initializing the prompt parameters either with word embeddings of [MASK] or similar to the vectors from the word embedding layer. Their work is based on a series of masked language models \cite{Delobelle2020RobBERTAD,Lan2020ALBERTAL} and uses a learnable output layer to project the mask to class logits, which restricts the model and only produces a single output. \cite{Liu2021GPTUT} proposes P-tuning, using the patterns based on human design and putting the continuous prompts interleave throughout the embedded input. When optimizing the model, P-tuning jointly updates both the prompt and model parameters to perform on SuperGLUE. Similarly, we borrow the idea of human designed patterns to convert different tasks into the form of conditional language model or masked language model, and also apply our method on GPT-2 and RoBERTa.&#10;&#10;However, the above usage of the discrete and continuous prompts assumes that the prompt is fixed for a specific task and all samples in the task share the same prompt. Different from the above methods, our proposed IPL dynamically learns a special prompt for each instance and obtains considerable improvements compared to strong baselines. &#10;&#10;Very recently, a contemporaneous work also presents another instance dependent prompt generation approach \cite{IDPG}, which studies only masked language model on only NLU tasks. In contrast, our IPL model is simple and effective for both unidirectional and bidirectional PLMs on both NLU and NLG tasks.\par&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2104.07650" label="2104.07650">
        <attvalues>
          <attvalue for="0" value="KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization&#10;  for Relation Extraction" />
          <attvalue for="1" value="  Recently, prompt-tuning has achieved promising results for specific few-shot&#10;classification tasks. The core idea of prompt-tuning is to insert text pieces&#10;(i.e., templates) into the input and transform a classification task into a&#10;masked language modeling problem. However, for relation extraction, determining&#10;an appropriate prompt template requires domain expertise, and it is cumbersome&#10;and time-consuming to obtain a suitable label word. Furthermore, there exists&#10;abundant semantic and prior knowledge among the relation labels that cannot be&#10;ignored. To this end, we focus on incorporating knowledge among relation labels&#10;into prompt-tuning for relation extraction and propose a Knowledge-aware&#10;Prompt-tuning approach with synergistic optimization (KnowPrompt).&#10;Specifically, we inject latent knowledge contained in relation labels into&#10;prompt construction with learnable virtual type words and answer words. Then,&#10;we synergistically optimize their representation with structured constraints.&#10;Extensive experimental results on five datasets with standard and low-resource&#10;settings demonstrate the effectiveness of our approach. Our code and datasets&#10;are available in https://github.com/zjunlp/KnowPrompt for reproducibility.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.02355" label="2205.02355">
        <attvalues>
          <attvalue for="0" value="Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt&#10;  Tuning" />
          <attvalue for="1" value="  Pre-trained language models have contributed significantly to relation&#10;extraction by demonstrating remarkable few-shot learning abilities. However,&#10;prompt tuning methods for relation extraction may still fail to generalize to&#10;those rare or hard patterns. Note that the previous parametric learning&#10;paradigm can be viewed as memorization regarding training data as a book and&#10;inference as the close-book test. Those long-tailed or hard patterns can hardly&#10;be memorized in parameters given few-shot instances. To this end, we regard RE&#10;as an open-book examination and propose a new semiparametric paradigm of&#10;retrieval-enhanced prompt tuning for relation extraction. We construct an&#10;open-book datastore for retrieval regarding prompt-based instance&#10;representations and corresponding relation labels as memorized key-value pairs.&#10;During inference, the model can infer relations by linearly interpolating the&#10;base output of PLM with the non-parametric nearest neighbor distribution over&#10;the datastore. In this way, our model not only infers relation through&#10;knowledge stored in the weights during training but also assists&#10;decision-making by unwinding and querying examples in the open-book datastore.&#10;Extensive experiments on benchmark datasets show that our method can achieve&#10;state-of-the-art in both standard supervised and few-shot settings. Code are&#10;available in https://github.com/zjunlp/PromptKG/tree/main/research/RetrievalRE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.06300" label="2207.06300">
        <attvalues>
          <attvalue for="0" value="Re2G: Retrieve, Rerank, Generate" />
          <attvalue for="1" value="  As demonstrated by GPT-3 and T5, transformers grow in capability as parameter&#10;spaces become larger and larger. However, for tasks that require a large amount&#10;of knowledge, non-parametric memory allows models to grow dramatically with a&#10;sub-linear increase in computational cost and GPU memory requirements. Recent&#10;models such as RAG and REALM have introduced retrieval into conditional&#10;generation. These models incorporate neural initial retrieval from a corpus of&#10;passages. We build on this line of research, proposing Re2G, which combines&#10;both neural initial retrieval and reranking into a BART-based&#10;sequence-to-sequence generation. Our reranking approach also permits merging&#10;retrieval results from sources with incomparable scores, enabling an ensemble&#10;of BM25 and neural initial retrieval. To train our system end-to-end, we&#10;introduce a novel variation of knowledge distillation to train the initial&#10;retrieval, reranker, and generation using only ground truth on the target&#10;sequence output. We find large gains in four diverse tasks: zero-shot slot&#10;filling, question answering, fact-checking, and dialog, with relative gains of&#10;9% to 34% over the previous state-of-the-art on the KILT leaderboard. We make&#10;our code available as open source at&#10;https://github.com/IBM/kgi-slot-filling/tree/re2g.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.10103" label="2409.10103">
        <attvalues>
          <attvalue for="0" value="Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERT" />
          <attvalue for="1" value="  Self-supervised speech representation learning has become essential for&#10;extracting meaningful features from untranscribed audio. Recent advances&#10;highlight the potential of deriving discrete symbols from the features&#10;correlated with linguistic units, which enables text-less training across&#10;diverse tasks. In particular, sentence-level Self-Distillation of the&#10;pretrained HuBERT (SD-HuBERT) induces syllabic structures within latent speech&#10;frame representations extracted from an intermediate Transformer layer. In&#10;SD-HuBERT, sentence-level representation is accumulated from speech frame&#10;features through self-attention layers using a special CLS token. However, we&#10;observe that the information aggregated in the CLS token correlates more with&#10;speaker identity than with linguistic content. To address this, we propose a&#10;speech-only self-supervised fine-tuning approach that separates syllabic units&#10;from speaker information. Our method introduces speaker perturbation as data&#10;augmentation and adopts a frame-level training objective to prevent the CLS&#10;token from aggregating paralinguistic information. Experimental results show&#10;that our approach surpasses the current state-of-the-art method in most&#10;syllable segmentation and syllabic unit quality metrics on Librispeech,&#10;underscoring its effectiveness in promoting syllabic organization within&#10;speech-only models.&#10;" />
          <attvalue for="2" value="&#10;Self-supervised speech representation learning has emerged as a key technique as it enables speech models to extract good features from untranscribed audio.&#10;Some of those works have shown that hidden units obtained by discretizing learned features highly correlate with linguistic units, e.g., phones~\cite{9585401}, syllables~\cite{peng23e_interspeech,10446062}, and words~\cite{peng22c_interspeech}.&#10;By utilizing them as pseudo-labels of untranscribed audio, we can train transcript-less models for various spoken language processing tasks, including speech synthesis~\cite{lakhotia-etal-2021-generative,polyak21_interspeech,10158503,zhang-etal-2023-speechgpt}, spoken language understanding~\cite{wu23g_interspeech,fang2024integrating}, speech-to-speech translation~\cite{lee-etal-2022-textless,huang2023transpeech}, and spoken language acquisition~\cite{10096250}.&#10;Among them, with a few exceptions, most works have focused on the shortest phonetic units.&#10;While previous work has shown that a phone-level representation quality highly correlates with the accuracy of automatic speech recognition~\cite{chang23_interspeech}, we often need higher-level information to understand spoken language, where semantic comprehension is essential~\cite{fang2024integrating}.&#10;&#10;Recent studies have shown that self-supervised fine-tuning of the pretrained HuBERT~\cite{9585401} naturally induces syllabic organization in an intermediate network layer~\cite{peng23e_interspeech,10446062}.&#10;First, Peng et al. demonstrated that syllabic organization emerges in Visually-Grounded HuBERT (VG-HuBERT), which learns the co-occurrence of matched speech-image pairs~\cite{peng23e_interspeech}.&#10;They concluded that visual grounding is responsible for the ability as the same phenomenon does not emerge with the original masked language modeling objective.&#10;&#10;Afterwards, Cho et al. revealed that syllabic-organized representations also emerge in a speech-only model through sentence-level self-distillation fine-tuning of the pretrained HuBERT, named Self-Distilled HuBERT (SD-HuBERT)~\cite{10446062}.&#10;Like BERT in natural language processing, sentence-level representation was aggregated through self-attention layers using a special learnable token, called the CLS token, concatenated with input speech frame feature sequence~\cite{devlin-etal-2019-bert}.&#10;Additionally, they proposed the spoken sentence ABX (SSABX) task for evaluating sentence discriminability of speech models.&#10;Speech-only models are beneficial in two aspects: 1) no image labels are required, and 2) they can learn concepts without visual entities~\cite{10446062}.&#10;However, their experimental results on the SSABX task have shown that distilled information within the CLS token might be dominated by paralinguistic content.&#10;In fact, we observed a dependence between speaker IDs and the predicted pseudo-categories.&#10;Since syllabic units should be speaker-invariant, this may be a potential factor degrading the quality of linguistic features.&#10;&#10;Some works have introduced speaker disentanglement for self-supervised training of speech models~\cite{pmlr-v162-qian22b,chang23_interspeech}.&#10;Chang et al. proposed speaker-invariant clustering (Spin), which learns speaker-invariant speech representations by performing swapped prediction between original and speaker-perturbed waveforms~\cite{chang23_interspeech}.&#10;However, the representation obtained by their method is at the phone-level.&#10;&#10;In this paper, we propose a teacher-student learning-based speech-only syllabic unit discovery method that integrates speaker information disentanglement for improved performance.&#10;Following~\cite{chang23_interspeech}, to obtain speaker-invariant representations, we constrain a speech model to extract consistent features between the original speech and its speaker-perturbed version.&#10;Moreover, unlike~\cite{10446062}, we avoid using the CLS token and employ a frame-level training objective to prevent the aggregation of paralinguistic information.&#10;Experimental results show that our proposed method outperforms the current state-of-the-art method in most evaluation metrics for syllable segmentation and syllabic unit quality, demonstrating the efficacy of our proposed method.&#10;Finally, our ablation study reveals that the essential factor for syllabic organization is the use of higher Transformer~\cite{NIPS2017_3f5ee243} layers as the student's learning targets that correlate with linguistically coarse-grained units.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Self-Supervised Training, Speech Representation Learning, Artificial Intelligence, Signal Processing, Syllable Segmentation" />
        </attvalues>
      </node>
      <node id="2305.11435" label="2305.11435">
        <attvalues>
          <attvalue for="0" value="Syllable Discovery and Cross-Lingual Generalization in a Visually&#10;  Grounded, Self-Supervised Speech Model" />
          <attvalue for="1" value="  In this paper, we show that representations capturing syllabic units emerge&#10;when training a self-supervised speech model with a visually-grounded training&#10;objective. We demonstrate that a nearly identical model architecture (HuBERT)&#10;trained with a masked language modeling loss does not exhibit this same&#10;ability, suggesting that the visual grounding objective is responsible for the&#10;emergence of this phenomenon. We propose the use of a minimum cut algorithm to&#10;automatically predict syllable boundaries in speech, followed by a 2-stage&#10;clustering method to group identical syllables together. We show that our model&#10;not only outperforms a state-of-the-art syllabic segmentation method on the&#10;language it was trained on (English), but also generalizes in a zero-shot&#10;fashion to Estonian. Finally, we show that the same model is capable of&#10;zero-shot generalization for a word segmentation task on 4 other languages from&#10;the Zerospeech Challenge, in some cases beating the previous state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.10803" label="2310.10803">
        <attvalues>
          <attvalue for="0" value="SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic&#10;  Organization in HuBERT" />
          <attvalue for="1" value="  Data-driven unit discovery in self-supervised learning (SSL) of speech has&#10;embarked on a new era of spoken language processing. Yet, the discovered units&#10;often remain in phonetic space and the units beyond phonemes are largely&#10;underexplored. Here, we demonstrate that a syllabic organization emerges in&#10;learning sentence-level representation of speech. In particular, we adopt&#10;&quot;self-distillation&quot; objective to fine-tune the pretrained HuBERT with an&#10;aggregator token that summarizes the entire sentence. Without any supervision,&#10;the resulting model draws definite boundaries in speech, and the&#10;representations across frames exhibit salient syllabic structures. We&#10;demonstrate that this emergent structure largely corresponds to the ground&#10;truth syllables. Furthermore, we propose a new benchmark task, Spoken Speech&#10;ABX, for evaluating sentence-level representation of speech. When compared to&#10;previous models, our model outperforms in both unsupervised syllable discovery&#10;and learning sentence-level representation. Together, we demonstrate that the&#10;self-distillation of HuBERT gives rise to syllabic organization without relying&#10;on external labels or modalities, and potentially provides novel data-driven&#10;units for spoken language modeling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15081" label="2203.15081">
        <attvalues>
          <attvalue for="0" value="Word Discovery in Visually Grounded, Self-Supervised Speech Models" />
          <attvalue for="1" value="  We present a method for visually-grounded spoken term discovery. After&#10;training either a HuBERT or wav2vec2.0 model to associate spoken captions with&#10;natural images, we show that powerful word segmentation and clustering&#10;capability emerges within the model's self-attention heads. Our experiments&#10;reveal that this ability is not present to nearly the same extent in the base&#10;HuBERT and wav2vec2.0 models, suggesting that the visual grounding task is a&#10;crucial component of the word discovery capability we observe. We also evaluate&#10;our method on the Buckeye word segmentation and ZeroSpeech spoken term&#10;discovery tasks, where we perform on par with or better than currently&#10;published methods on several metrics. Code and model weights are available at&#10;https://github.com/jasonppy/word-discovery.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.00355" label="2104.00355">
        <attvalues>
          <attvalue for="0" value="Speech Resynthesis from Discrete Disentangled Self-Supervised&#10;  Representations" />
          <attvalue for="1" value="  We propose using self-supervised discrete representations for the task of&#10;speech resynthesis. To generate disentangled representation, we separately&#10;extract low-bitrate representations for speech content, prosodic information,&#10;and speaker identity. This allows to synthesize speech in a controllable&#10;manner. We analyze various state-of-the-art, self-supervised representation&#10;learning methods and shed light on the advantages of each method while&#10;considering reconstruction quality and disentanglement properties.&#10;Specifically, we evaluate the F0 reconstruction, speaker identification&#10;performance (for both resynthesis and voice conversion), recordings'&#10;intelligibility, and overall quality using subjective human evaluation. Lastly,&#10;we demonstrate how these representations can be used for an ultra-lightweight&#10;speech codec. Using the obtained representations, we can get to a rate of 365&#10;bits per second while providing better speech quality than the baseline&#10;methods. Audio samples can be found under the following link:&#10;speechbot.github.io/resynthesis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18096" label="2305.18096">
        <attvalues>
          <attvalue for="0" value="Improving Textless Spoken Language Understanding with Discrete Units as&#10;  Intermediate Target" />
          <attvalue for="1" value="  Spoken Language Understanding (SLU) is a task that aims to extract semantic&#10;information from spoken utterances. Previous research has made progress in&#10;end-to-end SLU by using paired speech-text data, such as pre-trained Automatic&#10;Speech Recognition (ASR) models or paired text as intermediate targets.&#10;However, acquiring paired transcripts is expensive and impractical for&#10;unwritten languages. On the other hand, Textless SLU extracts semantic&#10;information from speech without utilizing paired transcripts. However, the&#10;absence of intermediate targets and training guidance for textless SLU often&#10;results in suboptimal performance. In this work, inspired by the&#10;content-disentangled discrete units from self-supervised speech models, we&#10;proposed to use discrete units as intermediate guidance to improve textless SLU&#10;performance. Our method surpasses the baseline method on five SLU benchmark&#10;corpora. Additionally, we find that unit guidance facilitates few-shot learning&#10;and enhances the model's ability to handle noise.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08352" label="2112.08352">
        <attvalues>
          <attvalue for="0" value="Textless Speech-to-Speech Translation on Real Data" />
          <attvalue for="1" value="  We present a textless speech-to-speech translation (S2ST) system that can&#10;translate speech from one language into another language and can be built&#10;without the need of any text data. Different from existing work in the&#10;literature, we tackle the challenge in modeling multi-speaker target speech and&#10;train the systems with real-world S2ST data. The key to our approach is a&#10;self-supervised unit-based speech normalization technique, which finetunes a&#10;pre-trained speech encoder with paired audios from multiple speakers and a&#10;single reference speaker to reduce the variations due to accents, while&#10;preserving the lexical content. With only 10 minutes of paired data for speech&#10;normalization, we obtain on average 3.2 BLEU gain when training the S2ST model&#10;on the VoxPopuli S2ST dataset, compared to a baseline trained on un-normalized&#10;speech target. We also incorporate automatically mined S2ST data and show an&#10;additional 2.0 BLEU gain. To our knowledge, we are the first to establish a&#10;textless S2ST technique that can be trained with real-world data and works for&#10;multiple language pairs. Audio samples are available at&#10;https://facebookresearch.github.io/speech_translation/textless_s2st_real_data/index.html .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.12523" label="2205.12523">
        <attvalues>
          <attvalue for="0" value="TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation" />
          <attvalue for="1" value="  Direct speech-to-speech translation (S2ST) with discrete units leverages&#10;recent progress in speech representation learning. Specifically, a sequence of&#10;discrete representations derived in a self-supervised manner are predicted from&#10;the model and passed to a vocoder for speech reconstruction, while still facing&#10;the following challenges: 1) Acoustic multimodality: the discrete units derived&#10;from speech with same content could be indeterministic due to the acoustic&#10;property (e.g., rhythm, pitch, and energy), which causes deterioration of&#10;translation accuracy; 2) high latency: current S2ST systems utilize&#10;autoregressive models which predict each unit conditioned on the sequence&#10;previously generated, failing to take full advantage of parallelism. In this&#10;work, we propose TranSpeech, a speech-to-speech translation model with&#10;bilateral perturbation. To alleviate the acoustic multimodal problem, we&#10;propose bilateral perturbation (BiP), which consists of the style normalization&#10;and information enhancement stages, to learn only the linguistic information&#10;from speech samples and generate more deterministic representations. With&#10;reduced multimodality, we step forward and become the first to establish a&#10;non-autoregressive S2ST technique, which repeatedly masks and predicts unit&#10;choices and produces high-accuracy results in just a few cycles. Experimental&#10;results on three language pairs demonstrate that BiP yields an improvement of&#10;2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, our&#10;parallel decoding shows a significant reduction of inference latency, enabling&#10;speedup up to 21.4x than autoregressive technique. Audio samples are available&#10;at \url{https://TranSpeech.github.io/}&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.05210" label="2309.05210">
        <attvalues>
          <attvalue for="0" value="Understanding the Impact of Post-Training Quantization on Large Language&#10;  Models" />
          <attvalue for="1" value="  Large language models (LLMs) are rapidly increasing in size, with the number&#10;of parameters becoming a key factor in the success of many commercial models,&#10;such as ChatGPT, Claude, and Bard. Even the recently released publicly&#10;accessible models for commercial usage, such as Falcon and Llama2, come&#10;equipped with billions of parameters. This significant increase in the number&#10;of parameters makes deployment and operation very costly. The remarkable&#10;progress in the field of quantization for large neural networks in general and&#10;LLMs in particular, has made these models more accessible by enabling them to&#10;be deployed on consumer-grade GPUs. Quantized models generally demonstrate&#10;comparable performance levels to their unquantized base counterparts.&#10;Nonetheless, there exists a notable gap in our comprehensive understanding of&#10;how these quantized models respond to hyperparameters, such as temperature, max&#10;new tokens, and topk, particularly for next word prediction. The present&#10;analysis reveals that nf4 and fp4 are equally proficient 4-bit quantization&#10;techniques, characterized by similar attributes such as inference speed, memory&#10;consumption, and the quality of generated content. the study identifies nf4 as&#10;displaying greater resilience to temperature variations in the case of the&#10;llama2 series of models at lower temperature, while fp4 and fp4-dq proves to be&#10;a more suitable choice for falcon series of models. It is noteworthy that, in&#10;general, 4-bit quantized models of varying sizes exhibit higher sensitivity to&#10;temperature in the range of 0.5 to 0.8, unlike their unquantized counterparts.&#10;Additionally, int8 quantization is associated with significantly slower&#10;inference speeds, whereas unquantized bfloat16 models consistently yield the&#10;fastest inference speeds across models of all sizes.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the emergence of the Transformer architecture \cite{vaswani2017attention}, a significant breakthrough was achieved, enabling the effective retention of extensive long-range dependencies in tasks related to natural language processing, speech, and vision. The transformer architecture enables highly parallel training due to sequence parallelism, which makes it possible to pretrain LLMs with hundreds of billions of parameters \cite{brown2020language, chowdhery2022palm, smith2022using}. The Big-bench \cite{srivastava2022beyond} introduced over 200 benchmarks designed to assess the capabilities of Large Language Models(LLMs) through quantification and extrapolation. This diverse and intricately elaborated set of benchmarks significantly contributed to the intensification of the race surrounding LLM development and advancement. &#10;&#10;The widespread adoption of LLMs on a substantial scale gained traction following the successful establishment of ChatGPT (including GPT-3 and subsequent iterations) \cite{brown2020language}. The pre-training of large transformer language models with 7 billion parameters and beyond demands a considerable amount of GPU computation, which can translate to costs amounting to millions of dollars. Such level of expenditure is beyond what academic research and small organizations can typically afford. Despite the high cost of deploying and operating large language models (LLMs), the recent release of the Falcon \cite{penedo2023refinedweb} and Llama2 \cite{touvron2023llama} models has sparked optimism among small organizations and has increased their desire to deploy their own custom LLMs. &#10;&#10;The efficient deployment of decoder only LLMs are challenging in practice because the generative inference proceeds sequentially, where the computation for each token depends on the previously generated tokens \cite{pope2023efficiently}. It is noteworthy that caching the attention key and value tensors of each layer can significantly improve the inference speed of smaller decoder-only models that fit on a single GPU memory. However, this is not possible for models that do not fit into the memory of a single GPU. To address the need for expensive high-end GPUs to support the deployment of these models, diverse forms of quantization have been put forward as potential solutions. The application of quantization methods to transformers emerges as a efficacious approach for mitigating sampling latency, while incurring minimal to negligible impact on overall performance \cite{chen2023accelerating}. Quantization techniques can be mainly characterized into three forms namely - i) quantization aware training \cite{yang2023dynamic, liu2023llm}, ii) quantization aware fine-tuning \cite{dettmers2023qlora, kwon2022alphatuning, dettmers2022llm}, and iii) post training quantization (PTQ) \cite{frantar2022gptq, yuan2023rptq, lin2023awq}. In \cite{yao2023comprehensive:}, the investigation primarily centers on evaluating the impact of diverse post-training quantization methods, employing perplexity scores as a benchmark. The perplexity scores are computed on datasets such as Wiki \cite{merity2016pointer}, PTB \cite{marcus1993building}, and C4 \cite{raffel2020exploring}, which mostl likely have served as foundational datasets during the training of most of the LLMs. It should be noted that these datasets are predisposed to exhibit favorable perplexity scores across all models, owing to their utilization in model training. Furthermore, it is acknowledged that perplexity, as a metric, may not effectively capture instances of repetitive generation within LLMs. Following outlines the primary contributions of the present study.&#10;\begin{enumerate}&#10; \item This study offers a systematic examination of the influence exerted by three pivotal hyper-parameters, namely, max new tokens, temperature, and top\_k, on LLMs that have undergone quantization through widely adopted post-training quantization techniques such as \cite{frantar2022gptq} (hereafter, gptq) and \cite{dettmers2022llm, dettmers2023qlora} (hereafter, bitsandbytes).&#10; \item It explores how these hyper-parameters exert their influence across a range of model sizes, spanning from 3 billion to 70 billion parameters.&#10; \item The process involves generating a total of 6,300 samples for each quantization method, achieved by constructing ten smaller prompts that encompass a diverse spectrum of domains for every model. &#10; \item LLMs typically exhibit a tendency towards repetitive generation, and it is often challenging to discern such repetition through perplexity scores. Therefore, to identify and quantify repetitive generation, the primary metric employed is the number of duplicate content words.&#10; \item It scrutinizes quantization methods that share similar inference speeds but manifest differing effects on accuracy.&#10; \item Finally, it aims to discern the optimal quantization method for deployment, considering specific constraints and requirements.&#10;&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Large Language Models, Computer Science, Quantization Techniques, Model Performance Optimization, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2303.05295" label="2303.05295">
        <attvalues>
          <attvalue for="0" value="Dynamic Stashing Quantization for Efficient Transformer Training" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated impressive performance on a&#10;range of Natural Language Processing (NLP) tasks. Unfortunately, the immense&#10;amount of computations and memory accesses required for LLM training makes them&#10;prohibitively expensive in terms of hardware cost, and thus challenging to&#10;deploy in use cases such as on-device learning. In this paper, motivated by the&#10;observation that LLM training is memory-bound, we propose a novel dynamic&#10;quantization strategy, termed Dynamic Stashing Quantization (DSQ), that puts a&#10;special focus on reducing the memory operations, but also enjoys the other&#10;benefits of low precision training, such as the reduced arithmetic cost. We&#10;conduct a thorough study on two translation tasks (trained-from-scratch) and&#10;three classification tasks (fine-tuning). DSQ reduces the amount of arithmetic&#10;operations by $20.95\times$ and the number of DRAM operations by $2.55\times$&#10;on IWSLT17 compared to the standard 16-bit fixed-point, which is widely used in&#10;on-device learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.03590" label="1907.03590">
        <attvalues>
          <attvalue for="0" value="Multiple Generative Models Ensemble for Knowledge-Driven Proactive&#10;  Human-Computer Dialogue Agent" />
          <attvalue for="1" value="  Multiple sequence to sequence models were used to establish an end-to-end&#10;multi-turns proactive dialogue generation agent, with the aid of data&#10;augmentation techniques and variant encoder-decoder structure designs. A&#10;rank-based ensemble approach was developed for boosting performance. Results&#10;indicate that our single model, in average, makes an obvious improvement in the&#10;terms of F1-score and BLEU over the baseline by 18.67% on the DuConv dataset.&#10;In particular, the ensemble methods further significantly outperform the&#10;baseline by 35.85%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Generative method for building conversation chatbots has attracted increasing interest due to its great flexibility. A typical and prevalent generative method is the encoder-decoder \cite{sutskever2014sequence}network, which transforms the input sequence into sequence (Seq2Seq) as output. Iulian et al. used a hierarchical recurrent encoder-decoder neural network to model conversations. Song et al. \cite{song2018ensemble} used multi-Seq2Seq models as generation module of a human-computer conversation system. Both of these systems are designed without external knowledge and in a passive way. Recently, the studies on the dialogue system have been evolved from traditional conversation history based to the knowledge based ones. Marjan et al. \cite{ghazvininejad2018knowledge} generalized the Seq2Seq approach for a knowledge-ground neural conversation model, by conditioning responses on both conversation history and external knowledge. In particular, they employed a facts encoder to inject the external knowledge into the model, similar to the designs by Baidu NLP group \cite{wu2019proactive}. These studies have shown that, the knowledge based system, in general, can produce more meaningful and informative response.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Ensemble Methods, Sequence Modeling, Dialogue Generation, Mathematics" />
        </attvalues>
      </node>
      <node id="2205.08808" label="2205.08808">
        <attvalues>
          <attvalue for="0" value="Evaluation of Transfer Learning for Polish with a Text-to-Text Model" />
          <attvalue for="1" value="  We introduce a new benchmark for assessing the quality of text-to-text models&#10;for Polish. The benchmark consists of diverse tasks and datasets: KLEJ&#10;benchmark adapted for text-to-text, en-pl translation, summarization, and&#10;question answering. In particular, since summarization and question answering&#10;lack benchmark datasets for the Polish language, we describe their construction&#10;and make them publicly available. Additionally, we present plT5 - a&#10;general-purpose text-to-text model for Polish that can be fine-tuned on various&#10;Natural Language Processing (NLP) tasks with a single training objective.&#10;Unsupervised denoising pre-training is performed efficiently by initializing&#10;the model weights with a multi-lingual T5 (mT5) counterpart. We evaluate the&#10;performance of plT5, mT5, Polish BART (plBART), and Polish GPT-2 (papuGaPT2).&#10;The plT5 scores top on all of these tasks except summarization, where plBART is&#10;best. In general (except for summarization), the larger the model, the better&#10;the results. The encoder-decoder architectures prove to be better than the&#10;decoder-only equivalent.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years have brought significant progress in natural language understanding (NLU) and natural language generation (NLG). Transformer architecture enabled efficient training of large-scale language models~\cite{radford2019language} and language understanding models~\cite{devlin-etal-2019-bert,liu2019roberta}. On the other hand, transfer learning, which has been used in representation learning for years~\cite{mikolov2013distributed,pennington-etal-2014-glove,devlin-etal-2019-bert}, has finally been successfully applied to text-to-text problems as well~\cite{raffel2020exploring,lewis-etal-2020-bart}. The text-to-text framework takes text as input and produces new text as output. This unified view enables the use of the same architecture, training procedure, and decoding process for many NLP tasks such as classification, machine translation, summarization, and question answering, to name a few. In addition, \newcite{raffel2020exploring} demonstrated that the simplicity of this approach combined with scale could achieve state-of-the-art results on many benchmark datasets, which makes it even more attractive. Their T5 model was available only for English language, but more recently pre-trained multi-lingual architectures~\cite{liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,nagoudi-etal-2021-indt5} and non-English counterparts~\cite{carmo2020ptt5,Malaya} were released. Multiple publications show that models targeted for specific language perform better than multi-lingual one~\cite{martin-etal-2020-camembert,le-etal-2020-flaubert-unsupervised,chan-etal-2020-germans,mroczkowski-etal-2021-herbert,virtanen2019multilingual,nagoudi2021arat5}. Moreover, specialized architectures are typically smaller due to significantly reduced vocabulary size, and they can be trained efficiently via transfer from multi-lingual checkpoints~\cite{arkhipov-etal-2019-tuning,mroczkowski-etal-2021-herbert}. There were some attempts to pre-train Transformer-based models for generating Polish, namely plBART~ and papuGaPT2, but they lack detailed description and evaluation on benchmark datasets. &#10;&#10;Our contributions are:&#10;\begin{itemize}&#10; \item comprehensive evaluation of text-to-text models on diverse tasks in Polish, such as text-to-text KLEJ benchmark~\cite{rybak-etal-2020-klej}, machine translation, question answering and summarization,&#10; \item construction of benchmark datasets in the Polish domain for question answering and summarization,&#10; \item demonstration of the efficiency of pre-training procedure for transferring knowledge from multi-lingual to monolingual text-to-text models based on work by \cite{arkhipov-etal-2019-tuning,mroczkowski-etal-2021-herbert},&#10; \item release of plT5 -- a T5-based model for the Polish language, which achieves the best results among the evaluated text-to-text models on KLEJ benchmark, machine translation and question answering and second-best results in summarization.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text-to-Text Models, Computer Science, Linguistics, Benchmark Evaluation, Mathematics, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2104.07483" label="2104.07483">
        <attvalues>
          <attvalue for="0" value="IndT5: A Text-to-Text Transformer for 10 Indigenous Languages" />
          <attvalue for="1" value="  Transformer language models have become fundamental components of natural&#10;language processing based pipelines. Although several Transformer models have&#10;been introduced to serve many languages, there is a shortage of models&#10;pre-trained for low-resource and Indigenous languages. In this work, we&#10;introduce IndT5, the first Transformer language model for Indigenous languages.&#10;To train IndT5, we build IndCorpus--a new dataset for ten Indigenous languages&#10;and Spanish. We also present the application of IndT5 to machine translation by&#10;investigating different approaches to translate between Spanish and the&#10;Indigenous languages as part of our contribution to the AmericasNLP 2021 Shared&#10;Task on Open Machine Translation. IndT5 and IndCorpus are publicly available&#10;for research&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.09144" label="2008.09144">
        <attvalues>
          <attvalue for="0" value="PTT5: Pretraining and validating the T5 model on Brazilian Portuguese&#10;  data" />
          <attvalue for="1" value="  In natural language processing (NLP), there is a need for more resources in&#10;Portuguese, since much of the data used in the state-of-the-art research is in&#10;other languages. In this paper, we pretrain a T5 model on the BrWac corpus, an&#10;extensive collection of web pages in Portuguese, and evaluate its performance&#10;against other Portuguese pretrained models and multilingual models on three&#10;different tasks. We show that our Portuguese pretrained models have&#10;significantly better performance over the original T5 models. Moreover, we&#10;demonstrate the positive impact of using a Portuguese vocabulary. Our code and&#10;models are available at https://github.com/unicamp-dl/PTT5.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.10906" label="2010.10906">
        <attvalues>
          <attvalue for="0" value="German's Next Language Model" />
          <attvalue for="1" value="  In this work we present the experiments which lead to the creation of our&#10;BERT and ELECTRA based German language models, GBERT and GELECTRA. By varying&#10;the input training data, model size, and the presence of Whole Word Masking&#10;(WWM) we were able to attain SoTA performance across a set of document&#10;classification and named entity recognition (NER) tasks for both models of base&#10;and large size. We adopt an evaluation driven approach in training these models&#10;and our results indicate that both adding more data and utilizing WWM improve&#10;model performance. By benchmarking against existing German models, we show that&#10;these models are the best German models to date. Our trained models will be&#10;made publicly available to the research community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.01735" label="2105.01735">
        <attvalues>
          <attvalue for="0" value="HerBERT: Efficiently Pretrained Transformer-based Language Model for&#10;  Polish" />
          <attvalue for="1" value="  BERT-based models are currently used for solving nearly all Natural Language&#10;Processing (NLP) tasks and most often achieve state-of-the-art results.&#10;Therefore, the NLP community conducts extensive research on understanding these&#10;models, but above all on designing effective and efficient training procedures.&#10;Several ablation studies investigating how to train BERT-like models have been&#10;carried out, but the vast majority of them concerned only the English language.&#10;A training procedure designed for English does not have to be universal and&#10;applicable to other especially typologically different languages. Therefore,&#10;this paper presents the first ablation study focused on Polish, which, unlike&#10;the isolating English language, is a fusional language. We design and&#10;thoroughly evaluate a pretraining procedure of transferring knowledge from&#10;multilingual to monolingual BERT-based models. In addition to multilingual&#10;model initialization, other factors that possibly influence pretraining are&#10;also explored, i.e. training objective, corpus size, BPE-Dropout, and&#10;pretraining length. Based on the proposed procedure, a Polish BERT-based&#10;language model -- HerBERT -- is trained. This model achieves state-of-the-art&#10;results on multiple downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00630" label="2005.00630">
        <attvalues>
          <attvalue for="0" value="KLEJ: Comprehensive Benchmark for Polish Language Understanding" />
          <attvalue for="1" value="  In recent years, a series of Transformer-based models unlocked major&#10;improvements in general natural language understanding (NLU) tasks. Such a fast&#10;pace of research would not be possible without general NLU benchmarks, which&#10;allow for a fair comparison of the proposed methods. However, such benchmarks&#10;are available only for a handful of languages. To alleviate this issue, we&#10;introduce a comprehensive multi-task benchmark for the Polish language&#10;understanding, accompanied by an online leaderboard. It consists of a diverse&#10;set of tasks, adopted from existing datasets for named entity recognition,&#10;question-answering, textual entailment, and others. We also introduce a new&#10;sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR).&#10;To ensure a common evaluation scheme and promote models that generalize to&#10;different NLU tasks, the benchmark includes datasets from varying domains and&#10;applications. Additionally, we release HerBERT, a Transformer-based model&#10;trained specifically for the Polish language, which has the best average&#10;performance and obtains the best results for three out of nine tasks. Finally,&#10;we provide an extensive evaluation, including several standard baselines and&#10;recently proposed, multilingual Transformer-based models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.03533" label="2212.03533">
        <attvalues>
          <attvalue for="0" value="Text Embeddings by Weakly-Supervised Contrastive Pre-training" />
          <attvalue for="1" value="  This paper presents E5, a family of state-of-the-art text embeddings that&#10;transfer well to a wide range of tasks. The model is trained in a contrastive&#10;manner with weak supervision signals from our curated large-scale text pair&#10;dataset (called CCPairs). E5 can be readily used as a general-purpose embedding&#10;model for any tasks requiring a single-vector representation of texts such as&#10;retrieval, clustering, and classification, achieving strong performance in both&#10;zero-shot and fine-tuned settings. We conduct extensive evaluations on 56&#10;datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the&#10;first model that outperforms the strong BM25 baseline on the BEIR retrieval&#10;benchmark without using any labeled data. When fine-tuned, E5 obtains the best&#10;results on the MTEB benchmark, beating existing embedding models with 40x more&#10;parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.03281" label="2308.03281">
        <attvalues>
          <attvalue for="0" value="Towards General Text Embeddings with Multi-stage Contrastive Learning" />
          <attvalue for="1" value="  We present GTE, a general-purpose text embedding model trained with&#10;multi-stage contrastive learning. In line with recent advancements in unifying&#10;various NLP tasks into a single format, we train a unified text embedding model&#10;by employing contrastive learning over a diverse mixture of datasets from&#10;multiple sources. By significantly increasing the number of training data&#10;during both unsupervised pre-training and supervised fine-tuning stages, we&#10;achieve substantial performance gains over existing embedding models. Notably,&#10;even with a relatively modest parameter count of 110M, GTE$_\text{base}$&#10;outperforms the black-box embedding API provided by OpenAI and even surpasses&#10;10x larger text embedding models on the massive text embedding benchmark.&#10;Furthermore, without additional fine-tuning on each programming language&#10;individually, our model outperforms previous best code retrievers of similar&#10;size by treating code as text. In summary, our model achieves impressive&#10;results by effectively harnessing multi-stage contrastive learning, offering a&#10;powerful and efficient text embedding model with broad applicability across&#10;various NLP and code-related tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.07597" label="2309.07597">
        <attvalues>
          <attvalue for="0" value="C-Pack: Packaged Resources To Advance General Chinese Embedding" />
          <attvalue for="1" value="  We introduce C-Pack, a package of resources that significantly advance the&#10;field of general Chinese embeddings. C-Pack includes three critical resources.&#10;1) C-MTEB is a comprehensive benchmark for Chinese text embeddings covering 6&#10;tasks and 35 datasets. 2) C-MTP is a massive text embedding dataset curated&#10;from labeled and unlabeled Chinese corpora for training embedding models. 3)&#10;C-TEM is a family of embedding models covering multiple sizes. Our models&#10;outperform all prior Chinese text embeddings on C-MTEB by up to +10% upon the&#10;time of the release. We also integrate and optimize the entire suite of&#10;training methods for C-TEM. Along with our resources on general Chinese&#10;embedding, we release our data and models for English text embeddings. The&#10;English models achieve state-of-the-art performance on MTEB benchmark;&#10;meanwhile, our released English data is 2 times larger than the Chinese data.&#10;All these resources are made publicly available at&#10;https://github.com/FlagOpen/FlagEmbedding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.09836" label="2002.09836">
        <attvalues>
          <attvalue for="0" value="Fill in the BLANC: Human-free quality estimation of document summaries" />
          <attvalue for="1" value="  We present BLANC, a new approach to the automatic estimation of document&#10;summary quality. Our goal is to measure the functional performance of a summary&#10;with an objective, reproducible, and fully automated method. Our approach&#10;achieves this by measuring the performance boost gained by a pre-trained&#10;language model with access to a document summary while carrying out its&#10;language understanding task on the document's text. We present evidence that&#10;BLANC scores have as good correlation with human evaluations as do the ROUGE&#10;family of summary quality measurements. And unlike ROUGE, the BLANC method does&#10;not require human-written reference summaries, allowing for fully human-free&#10;summary quality estimation.&#10;" />
          <attvalue for="2" value="&#10;Two most widely used methods for measuring the quality of a summary are ROUGE \cite{OriginalROUGE} and human evaluation \cite{Wojciech2019Neural}.&#10;&#10;The ROUGE family of methods are well-defined and reproducible. However, these methods typically require a human-written reference summaries for comparison, completely disregarding the original document text. Even if one assumes that a reference summary is available and of optimal quality, the ROUGE method is limited to measuring a mechanical overlap of text tokens with little regard to semantics. This deficiency may be partially addressable through measurement of the similarity not of text tokens but named entities or other preprocessed features \cite{Yuning2019Facet, Arman2016Revisiting, Fatma2015Keyphrase, Jun2015Better, Kavita2018ROUGE2} or embeddings \cite{Wei2019MoverScore, Tianyi2020BERTScore, Yang2020Supert}. In the latter work \cite{ Yang2020Supert} the references are not human-written but unsupervisedly constructed from selected salient sentences. An overlap can be measured as well between summary and document text \cite{Liqun2017Efficient}.&#10;&#10;Human evaluation of summary quality is far more meaningful and powerful than ROUGE, but it is far less reproducible. Summary quality estimation is a cognitively demanding and highly subjective task. Humans are also vulnerable to biases, such as the preference for phrases and sentences copied directly from the document text into summaries \cite{Daniel2020FineTuning}. Improving human evaluation may require prompting labelers to pay higher attention \cite{Hardy2019HighRES}, as well as splitting quality scores into multiple dimensions such as fluency, informativeness, and factual correctness \cite{Wojciech2019Neural, Wojciech2017Evaluating, Lisa2018Robust}. Even if humans can be trained to be more reliable, reproducible estimators of summary quality, they will forever remain a slow, expensive, limiting resource.&#10;&#10;One possible route to a better automatic method for summary quality estimation is to train a model on document summaries annotated with human quality scores \cite{Louis2009Automatically, Louis2013Automatically, Stratos2019Sum}. Such a model could be used to evaluate summaries without further human involvement. But even if such a model could achieve high agreement with human labelers, its performance would only be as objective and reproducible as the summary quality scores generated by one particular group of humans on a particular group of documents. Such a model may not generalize beyond the domain and style of the training samples unless they are a massive, representative sample of all documents of interest.&#10;&#10;A more fundamental approach to the problem is to estimate how &quot;helpful&quot; a summary is for the task of understanding a text. For example this might be achieved through a series of question-answers \cite{Matan2019Question, Ping2018SemanticQA, Thomas2015Answers}. However, with this approach one must choose from a vast set of questions one might ask of a text, presupposing knowledge of the document itself and seriously limiting its reproducibility.&#10;&#10;In the following sectionwe suggest a new approach that is fundamentally justifiable as an estimator of summary quality, as well as being conceptually simple and reproducible.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Language Models, Evaluation Metrics, Document Summarization, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1908.08960" label="1908.08960">
        <attvalues>
          <attvalue for="0" value="Neural Text Summarization: A Critical Evaluation" />
          <attvalue for="1" value="  Text summarization aims at compressing long documents into a shorter form&#10;that conveys the most important parts of the original document. Despite&#10;increased interest in the community and notable research effort, progress on&#10;benchmark datasets has stagnated. We critically evaluate key ingredients of the&#10;current research setup: datasets, evaluation metrics, and models, and highlight&#10;three primary shortcomings: 1) automatically collected datasets leave the task&#10;underconstrained and may contain noise detrimental to training and evaluation,&#10;2) current evaluation protocol is weakly correlated with human judgment and&#10;does not account for important characteristics such as factual correctness, 3)&#10;models overfit to layout biases of current datasets and offer limited diversity&#10;in their outputs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.00284" label="1710.00284">
        <attvalues>
          <attvalue for="0" value="Efficient and Effective Single-Document Summarizations and A&#10;  Word-Embedding Measurement of Quality" />
          <attvalue for="1" value="  Our task is to generate an effective summary for a given document with&#10;specific realtime requirements. We use the softplus function to enhance keyword&#10;rankings to favor important sentences, based on which we present a number of&#10;summarization algorithms using various keyword extraction and topic clustering&#10;methods. We show that our algorithms meet the realtime requirements and yield&#10;the best ROUGE recall scores on DUC-02 over all previously-known algorithms. We&#10;show that our algorithms meet the realtime requirements and yield the best&#10;ROUGE recall scores on DUC-02 over all previously-known algorithms. To evaluate&#10;the quality of summaries without human-generated benchmarks, we define a&#10;measure called WESM based on word-embedding using Word Mover's Distance. We&#10;show that the orderings of the ROUGE and WESM scores of our algorithms are&#10;highly comparable, suggesting that WESM may serve as a viable alternative for&#10;measuring the quality of a summary.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.00318" label="1906.00318">
        <attvalues>
          <attvalue for="0" value="Question Answering as an Automatic Evaluation Metric for News Article&#10;  Summarization" />
          <attvalue for="1" value="  Recent work in the field of automatic summarization and headline generation&#10;focuses on maximizing ROUGE scores for various news datasets. We present an&#10;alternative, extrinsic, evaluation metric for this task, Answering Performance&#10;for Evaluation of Summaries. APES utilizes recent progress in the field of&#10;reading-comprehension to quantify the ability of a summary to answer a set of&#10;manually created questions regarding central entities in the source article. We&#10;first analyze the strength of this metric by comparing it to known manual&#10;evaluation metrics. We then present an end-to-end neural abstractive model that&#10;maximizes APES, while increasing ROUGE scores to competitive results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.00555" label="2207.00555">
        <attvalues>
          <attvalue for="0" value="FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech&#10;  Self-Supervised Learning" />
          <attvalue for="1" value="  Large-scale speech self-supervised learning (SSL) has emerged to the main&#10;field of speech processing, however, the problem of computational cost arising&#10;from its vast size makes a high entry barrier to academia. In addition,&#10;existing distillation techniques of speech SSL models compress the model by&#10;reducing layers, which induces performance degradation in linguistic pattern&#10;recognition tasks such as phoneme recognition (PR). In this paper, we propose&#10;FitHuBERT, which makes thinner in dimension throughout almost all model&#10;components and deeper in layer compared to prior speech SSL distillation works.&#10;Moreover, we employ a time-reduction layer to speed up inference time and&#10;propose a method of hint-based distillation for less performance degradation.&#10;Our method reduces the model to 23.8% in size and 35.9% in inference time&#10;compared to HuBERT. Also, we achieve 12.1% word error rate and 13.3% phoneme&#10;error rate on the SUPERB benchmark which is superior than prior work.&#10;" />
          <attvalue for="2" value="&#10;Large-scale speech self-supervised learning (SSL) has emerged as an important field in speech processing recently due to its powerful performance and versatility.&#10;Large amount of speech-only data can be utilized for pre-training, and even only small amount of paired data is adequate to fine-tune the model with great performance \cite{yi2020applying}.&#10;HuBERT \cite{hsu2021hubert} and wav2vec 2.0 \cite{baevski2020wav2vec} both record word error rate (WER) of 1.8\%, which was the state-of-the-art performance on the test-clean LibriSpeech \cite{panayotov2015librispeech} benchmark. &#10;Not limited to the task of automatic speech recognition (ASR), speech SSL model can be expanded into various speech-related tasks by fine-tuning on a specific mainstream task such as automatic speaker verification (ASV) or keyword spotting (KS) {\cite{fan2020exploring, hussain2021multi}}. &#10;&#10;Despite its powerful performance and versatility, the main drawback of such wide and deep models is difficulty in usage due to its vast size.&#10;The limitation of computational resource and time-consuming training caused by numerous parameters make speech SSL model usage burdensome.&#10;According to the authors of \cite{baevski2020wav2vec}, they used total 128 V100 GPUs to pre-train wav2vec 2.0 LARGE for 2.3 days, which is mostly unavailable to academia.&#10;These large-scale models require more memory and time at inference following high computational overhead during fine-tuning.&#10;&#10;Knowledge distillation is a model compression technique which can be a possible solution for the above issue.&#10;The knowledge from the cumbersome teacher model can be transferred to the student model by learning teacher's representation.&#10;In \cite{sanh2019distilbert}, knowledge distillation is applied to BERT \cite{devlin2018bert}, one of the most prominent language representation model, by reducing the number of Transformer \cite{vaswani2017attention} layers and initializing with pre-trained BERT.&#10;On the other hand, FitNets \cite{adriana2015fitnets} suggests thinner and deeper student than the teacher, matching not only the final outputs but also the intermediate representations as hints.&#10;Authors in \cite{yim2017gift} make the student model imitate the relationship between features from two distinct layers by minimizing the corresponding distance of flow of solution procedure (FSP) matrices.&#10;&#10;Meanwhile, in the field of speech SSL, few studies have been conducted using knowledge distillation techniques.&#10;DistilHuBERT \cite{chang2021distilhubert} compresses 12 Transformer layers down to 2 by employing 3 distinct prediction heads.&#10;In \cite{peng2021shrinking}, authors attempt to reduce the Transformer layers of wav2vec 2.0 \cite{baevski2020wav2vec} by introducing both KL-divergence and mean squared error (MSE) losses.&#10;However, the main disadvantage of these two approaches is performance degradation of linguistic pattern recognition tasks such as ASR or phoneme recognition (PR). &#10;&#10;Accordingly, we propose a novel approach to student model design and distillation scheme, FitHuBERT, which can be applied to any Transformer-based speech SSL model.&#10;In FitHuBERT, we design a model thinner and deeper compared to prior speech SSL distillation works.&#10;A CNN feature extractor is designed in a channel-increasing manner with pointwise convolution.&#10;For Transformer layers, dimensions of the self-attention and the inner-layer of feed-forward network (FFN) are reduced by 37.5\% and 84.3\%, respectively.&#10;By using hint-based distillation and layer-wise prediction heads, FitHuBERT can be guided during the distillation for all Transformer layers.&#10;Furthermore, a trainable time-reduction layer is introduced to attain faster inference.&#10;We reduce the parameters of the model to 23.8\% and make the inference speed 2.8 times faster compared to the teacher model, HuBERT.&#10;Also, performance of PR and ASR are relatively improved by 18.1\% and 9.6\% compared to DistilHuBERT respectively on the SUPERB benchmark \cite{yang21c_interspeech}.&#10;&#10;While developing our method, a concurrent work \cite{wang2022lighthubert} with a similar goal proposes a two-stage distillation strategy, making use of pre-training distillation and large-sized Transformer supernet with neural architecture search.&#10;Our approach, in contrast, explores a simple strategy of applying knowledge distillation directly to the pre-trained teacher model, having significantly lower training cost and simpler distillation strategy.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.01643" label="2402.01643">
        <attvalues>
          <attvalue for="0" value="L-TUNING: Synchronized Label Tuning for Prompt and Prefix in LLMs" />
          <attvalue for="1" value="  Efficiently fine-tuning Large Language Models (LLMs) for specific tasks&#10;presents a considerable challenge in natural language processing. Traditional&#10;methods, like prompt or prefix tuning, typically rely on arbitrary tokens for&#10;training, leading to prolonged training times and generalized token use across&#10;various class labels. To address these issues, this paper introduces L-Tuning,&#10;an efficient fine-tuning approach designed for classification tasks within the&#10;Natural Language Inference (NLI) framework. Diverging from conventional&#10;methods, L-Tuning focuses on the fine-tuning of label tokens processed through&#10;a pre-trained LLM, thereby harnessing its pre-existing semantic knowledge. This&#10;technique not only improves the fine-tuning accuracy and efficiency but also&#10;facilitates the generation of distinct label embeddings for each class,&#10;enhancing the model's training nuance. Our experimental results indicate a&#10;significant improvement in training efficiency and classification accuracy with&#10;L-Tuning compared to traditional approaches, marking a promising advancement in&#10;fine-tuning LLMs for complex language tasks.&#10;" />
          <attvalue for="2" value="&#10;The advent of LLM has marked a significant milestone in NLP \cite{ge2023openagi}. However, the effective utilization of LLMs often depends on fine-tuning techniques such as prompt or prefix tuning \cite{peng2023soft}. Traditional methods, which typically involve training arbitrary tokens for all labels to guide the model, encounter limitations in the context of LLMs \cite{liu2022p, lester2021power, gu2021ppt, han2022ptr}. Due to the non-semantic nature of these tokens, requiring extensive training for effective integration. Additionally, the use of identical tokens across all classes leads to suboptimal performance due to the lack of semantic differentiation among the classes.&#10;&#10;To surmount these challenges, we introduce L-Tuning, an innovative approach to prompt and prefix tuning, particularly tailored for classification tasks within the NLI framework \cite{kowsher2023contrastive}. Distinct from traditional methods, L-Tuning leverages label tokens that are initially processed through the pre-trained LLM. This strategy effectively utilizes the LLM's inherent semantic knowledge, enabling more efficient and precise optimization. Furthermore, L-Tuning employs unique label tokens for each class, thereby providing a more refined method for fine-tuning. Empirical evidence suggests that L-Tuning significantly outperforms conventional prompt and prefix tuning in LLMs, both in terms of reducing training time and enhancing performance in classification tasks.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Fine-Tuning Techniques, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.04370" label="2304.04370">
        <attvalues>
          <attvalue for="0" value="OpenAGI: When LLM Meets Domain Experts" />
          <attvalue for="1" value="  Human Intelligence (HI) excels at combining basic skills to solve complex&#10;tasks. This capability is vital for Artificial Intelligence (AI) and should be&#10;embedded in comprehensive AI Agents, enabling them to harness expert models for&#10;complex task-solving towards Artificial General Intelligence (AGI). Large&#10;Language Models (LLMs) show promising learning and reasoning abilities, and can&#10;effectively use external models, tools, plugins, or APIs to tackle complex&#10;problems. In this work, we introduce OpenAGI, an open-source AGI research and&#10;development platform designed for solving multi-step, real-world tasks.&#10;Specifically, OpenAGI uses a dual strategy, integrating standard benchmark&#10;tasks for benchmarking and evaluation, and open-ended tasks including more&#10;expandable models, tools, plugins, or APIs for creative problem-solving. Tasks&#10;are presented as natural language queries to the LLM, which then selects and&#10;executes appropriate models. We also propose a Reinforcement Learning from Task&#10;Feedback (RLTF) mechanism that uses task results to improve the LLM's&#10;task-solving ability, which creates a self-improving AI feedback loop. While we&#10;acknowledge that AGI is a broad and multifaceted research challenge with no&#10;singularly defined solution path, the integration of LLMs with domain-specific&#10;expert models, inspired by mirroring the blend of general and specialized&#10;intelligence in humans, offers a promising approach towards AGI. We are&#10;open-sourcing the OpenAGI project's code, dataset, benchmarks, evaluation&#10;methods, and the UI demo to foster community involvement in AGI advancement:&#10;https://github.com/agiresearch/OpenAGI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.08303" label="2307.08303">
        <attvalues>
          <attvalue for="0" value="Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language&#10;  Models" />
          <attvalue for="1" value="  Dense retrieval (DR) converts queries and documents into dense embeddings and&#10;measures the similarity between queries and documents in vector space. One of&#10;the challenges in DR is the lack of domain-specific training data. While DR&#10;models can learn from large-scale public datasets like MS MARCO through&#10;transfer learning, evidence shows that not all DR models and domains can&#10;benefit from transfer learning equally. Recently, some researchers have&#10;resorted to large language models (LLMs) to improve the zero-shot and few-shot&#10;DR models. However, the hard prompts or human-written prompts utilized in these&#10;works cannot guarantee the good quality of generated weak queries. To tackle&#10;this, we propose soft prompt tuning for augmenting DR (SPTAR): For each task,&#10;we leverage soft prompt-tuning to optimize a task-specific soft prompt on&#10;limited ground truth data and then prompt the LLMs to tag unlabeled documents&#10;with weak queries, yielding enough weak document-query pairs to train&#10;task-specific dense retrievers. We design a filter to select high-quality&#10;example document-query pairs in the prompt to further improve the quality of&#10;weak tagged queries. To the best of our knowledge, there is no prior work&#10;utilizing soft prompt tuning to augment DR models. The experiments demonstrate&#10;that SPTAR outperforms the unsupervised baselines BM25 and the recently&#10;proposed LLMs-based augmentation method for DR.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.04618" label="2306.04618">
        <attvalues>
          <attvalue for="0" value="Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis,&#10;  and LLMs Evaluations" />
          <attvalue for="1" value="  This paper reexamines the research on out-of-distribution (OOD) robustness in&#10;the field of NLP. We find that the distribution shift settings in previous&#10;studies commonly lack adequate challenges, hindering the accurate evaluation of&#10;OOD robustness. To address these issues, we propose a benchmark construction&#10;protocol that ensures clear differentiation and challenging distribution&#10;shifts. Then we introduce BOSS, a Benchmark suite for Out-of-distribution&#10;robustneSS evaluation covering 5 tasks and 20 datasets. Based on BOSS, we&#10;conduct a series of experiments on pre-trained language models for analysis and&#10;evaluation of OOD robustness. First, for vanilla fine-tuning, we examine the&#10;relationship between in-distribution (ID) and OOD performance. We identify&#10;three typical types that unveil the inner learning mechanism, which could&#10;potentially facilitate the forecasting of OOD robustness, correlating with the&#10;advancements on ID datasets. Then, we evaluate 5 classic methods on BOSS and&#10;find that, despite exhibiting some effectiveness in specific cases, they do not&#10;offer significant improvement compared to vanilla fine-tuning. Further, we&#10;evaluate 5 LLMs with various adaptation paradigms and find that when sufficient&#10;ID data is available, fine-tuning domain-specific models outperform LLMs on ID&#10;examples significantly. However, in the case of OOD instances, prioritizing&#10;LLMs with in-context learning yields better results. We identify that both&#10;fine-tuned small models and LLMs face challenges in effectively addressing&#10;downstream tasks. The code is public at&#10;\url{https://github.com/lifan-yuan/OOD_NLP}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Distribution shifts in NLP has been widely studied in various forms.&#10;We examine several representative cases as outlined below.&#10;Domain shift refers to the challenge of testing data originating from diverse domains, often due to data collection from various sources~\cite{ma2019domain, hendrycks-etal-2020-pretrained, lester2021power, ramponi2020neural}.&#10;Temporal shift examines the degradation of models' performance over time~\cite{huang2018examining, agarwal2021temporal}. &#10;Spurious correlation examines the issue of models acquiring dataset-specific knowledge on ID data, which may not generalize effectively to OOD data~\cite{mccoy2019right, nie2019anli, tu2020empirical,gardner2021competency, he2019unlearn, clark-etal-2019-dont, clark2020learning, mahabadi2020end}.&#10;Additionally, a requirement is for models to exhibit robustness when confronted with artificially constructed OOD samples.&#10;One typical type is malicious adversarial attacks, which involve assessing the resilience of models against inputs crafted by malevolent adversaries~\cite{li2018textbugger, le2022perturbations, DBLP:journals/corr/abs-2110-15317}.&#10;These inputs, distinct from ID samples, have the potential to induce model failures~\cite{chen2022should}.&#10;Adversarial attacks can also be effectively utilized to simulate diverse user inputs to examine models' robustness in the real world~\cite{wang-etal-2021-textflint, chen2023adversarial, goel2021robustness}.&#10;Another category is backdoor attacks, characterized by intentionally introduced spurious correlations that can be exploited by attackers for their advantage~\cite{DBLP:conf/nips/CuiYHCLS22, DBLP:journals/corr/abs-2304-14475}.&#10;&#10;\looseness=-1&#10;OOD Evaluation in NLP can be broadly classified into automatic and static evaluation approaches.&#10;Automatic evaluation utilizes diverse textual transformation techniques, such as introducing typos, to conduct a rigorous evaluation of OOD robustness. &#10;Three essential elements in the automatic OOD evaluation encompass the establishment of suitable transformation methods, evaluation metrics, and effective techniques to ensure sample validity~\cite{DBLP:conf/naacl/GoelRVTBR21, wang-etal-2021-textflint}. &#10;Static evaluation, in contrast to automated methods, offers the advantage of constructing benchmarks with higher quality, resulting in an improved estimation of OOD robustness.&#10;Numerous OOD benchmarks have been introduced, focusing on adversarial attacks~\cite{wang2021adversarial} or spurious correlations~\cite{zhang2019paws, mccoy2019right}.&#10;A relevant study to ours is GLUE-X~\cite{yang2022glue}, which establishes an OOD benchmark derived from the GLUE benchmark~\cite{wang2018glue}. &#10;Nevertheless, they do not establish a coherent benchmark construction protocol and primarily rely on dataset selection driven by popularity, incorporating datasets into the benchmark without comprehensive explanation and seemingly opting for a somewhat arbitrary selection, thus lacking a systematic approach.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Benchmark Evaluation Methods, Artificial Intelligence, Natural Language Processing, Out-of-Distribution Robustness" />
        </attvalues>
      </node>
      <node id="2201.05613" label="2201.05613">
        <attvalues>
          <attvalue for="0" value="The Dark Side of the Language: Pre-trained Transformers in the DarkNet" />
          <attvalue for="1" value="  Pre-trained Transformers are challenging human performances in many NLP&#10;tasks. The massive datasets used for pre-training seem to be the key to their&#10;success on existing tasks. In this paper, we explore how a range of pre-trained&#10;Natural Language Understanding models perform on definitely unseen sentences&#10;provided by classification tasks over a DarkNet corpus. Surprisingly, results&#10;show that syntactic and lexical neural networks perform on par with pre-trained&#10;Transformers even after fine-tuning. Only after what we call extreme domain&#10;adaptation, that is, retraining with the masked language model task on all the&#10;novel corpus, pre-trained Transformers reach their standard high results. This&#10;suggests that huge pre-training corpora may give Transformers unexpected help&#10;since they are exposed to many of the possible sentences.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformers \cite{zhang2019ernie,Radford2018ImprovingLU:GPT} have been rocking the field of NLP. These Transformers are outperforming all previous methods and, sometimes, even humans in many NLP tasks \cite{wang-etal-2018-glue,wang2020superglue,AMMUS-T-PTLMs:2021,ThreatsPTLMs:2022}. &#10;&#10;Pre-training on large corpora seems to be the key that boosts performances, as these Transformers may induce clear models of target languages. &#10;Indeed, BERT is pre-trained on an English corpus of 3,300M words consisting of books \cite{BookCorpusUsedInBERT} and Wikipedia. The English version of the last ERNIE \cite{Sun2021ERNIE3L} is trained on an even more extensive corpus. MEGATRON-LM \cite{Shoeybi2019MegatronLMTM} utilizes an incredible corpus of 174 GB, and the Chinese version of ERNIE breaks the records by exploiting a 4TB corpus \cite{ERNIE_3.0_4TB}. Therefore, the challenge is training over always more massive corpora. &#10;&#10;Huge pre-training corpora may give unexpected help to Transformers: their successes in downstream tasks can be because Transformers have seen large parts of possible sentences. This could be a sort of overfitting. This possible shortcoming is sometimes considered when novel Transformers are introduced \cite{Radford2019LanguageMA,Shoeybi2019MegatronLMTM}. For this reason, \cite{Radford2019LanguageMA} have excluded Wikipedia pages for pre-training as it is a common data source for other downstream datasets. Yet, when using off-the-shelf pre-trained models, this caution is generally disregarded. &#10;For example, the discovering ongoing conversation (DOC) task, introduced by \cite{10.1145/2885501} was found challenging for humans, but the BERT baseline model achieved the astonishing 88.4 F1 score \cite{wang2020superglue}. DOC consists of determining if two utterances are contiguous in classical theatrical plays. These &#10;plays may be included in the book dataset \cite{BookCorpusUsedInBERT} used for pre-training BERT. This may explain the superhuman performance in such a challenging task.&#10;&#10;Corpora and related tasks derived from the DeepWeb and DarkWeb \cite{info13090435,DBLP:conf/itasec/RanaldiNFZ22,avarikioti2018structure,Choshen2019TheLO} offer a tremendous opportunity to study Transformers and other natural language models on &#10;\totallyunseensentences.&#10;&#10;Performances on these tasks cannot depend on overfitting over-seen sentences.&#10;&#10;Indeed, it is extremely unlikely that texts extracted from these sources are included in pre-training corpora. Moreover, language on the DarkNet may have very different characteristics with respect to the one accessible from the surface web \cite{Choshen2019TheLO}. &#10;&#10;In this paper, we aim to explore how pre-trained Natural Language Understanding models behave on \totallyunseensentences. &#10;These \totallyunseensentences are provided by the DarkNet corpus along with a classification task. We experimented with Stylistic Classifiers based on the bleaching text model \cite{van-der-goot-etal-2018-bleaching}, with Lexical Neural Networks based on GloVe \cite{pennington-etal-2014-glove} and word2vec \cite{mikolov2013efficient}, with Syntatic-based neural networks based on KERMIT \cite{zanzotto-etal-2020-kermit}, and with holistic Transformers such as BERT \cite{devlin-etal-2019-bert}, XLNet \cite{Yang2019XLNetGA}, ERNIE \cite{zhang2019ernie} and Electra \cite{clark2020electra}. Results show that syntactic and lexical neural networks surprisingly outperform pre-trained Transformers even after fine-tuning \cite{FineTuning:Wei2021WhyDP}. Only when pre-training is extended to the novel corpus and Transformers see these \totallyunseensentences do their performances increase to the expected level.&#10;This seems to suggest that huge pre-training corpora may give Transformers the unexpected help of showing them many possible sentences. &#10;&#10;The rest of the paper is organized as follows: Material and Methods, Results and Discussion, Conclusions, and Limitations.&#10;The code and data are publicly available at: .&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Domain Adaptation, Pre-trained Transformers, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2107.02137" label="2107.02137">
        <attvalues>
          <attvalue for="0" value="ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language&#10;  Understanding and Generation" />
          <attvalue for="1" value="  Pre-trained models have achieved state-of-the-art results in various Natural&#10;Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown&#10;that scaling up pre-trained language models can improve their generalization&#10;abilities. Particularly, the GPT-3 model with 175 billion parameters shows its&#10;strong task-agnostic zero-shot/few-shot learning capabilities. Despite their&#10;success, these large-scale models are trained on plain texts without&#10;introducing knowledge such as linguistic knowledge and world knowledge. In&#10;addition, most large-scale models are trained in an auto-regressive way. As a&#10;result, this kind of traditional fine-tuning approach demonstrates relatively&#10;weak performance when solving downstream language understanding tasks. In order&#10;to solve the above problems, we propose a unified framework named ERNIE 3.0 for&#10;pre-training large-scale knowledge enhanced models. It fuses auto-regressive&#10;network and auto-encoding network, so that the trained model can be easily&#10;tailored for both natural language understanding and generation tasks with&#10;zero-shot learning, few-shot learning or fine-tuning. We trained the model with&#10;10 billion parameters on a 4TB corpus consisting of plain texts and a&#10;large-scale knowledge graph. Empirical results show that the model outperforms&#10;the state-of-the-art models on 54 Chinese NLP tasks, and its English version&#10;achieves the first place on the SuperGLUE benchmark (July 3, 2021), surpassing&#10;the human performance by +0.8% (90.6% vs. 89.8%).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.01348" label="1811.01348">
        <attvalues>
          <attvalue for="0" value="Structure and Content of the Visible Darknet" />
          <attvalue for="1" value="  In this paper, we analyze the topology and the content found on the&#10;&quot;darknet&quot;, the set of websites accessible via Tor. We created a darknet spider&#10;and crawled the darknet starting from a bootstrap list by recursively following&#10;links. We explored the whole connected component of more than 34,000 hidden&#10;services, of which we found 10,000 to be online. Contrary to folklore belief,&#10;the visible part of the darknet is surprisingly well-connected through hub&#10;websites such as wikis and forums. We performed a comprehensive categorization&#10;of the content using supervised machine learning. We observe that about half of&#10;the visible dark web content is related to apparently licit activities based on&#10;our classifier. A significant amount of content pertains to software&#10;repositories, blogs, and activism-related websites. Among unlawful hidden&#10;services, most pertain to fraudulent websites, services selling counterfeit&#10;goods, and drug markets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.05543" label="1905.05543">
        <attvalues>
          <attvalue for="0" value="The Language of Legal and Illegal Activity on the Darknet" />
          <attvalue for="1" value="  The non-indexed parts of the Internet (the Darknet) have become a haven for&#10;both legal and illegal anonymous activity. Given the magnitude of these&#10;networks, scalably monitoring their activity necessarily relies on automated&#10;tools, and notably on NLP tools. However, little is known about what&#10;characteristics texts communicated through the Darknet have, and how well&#10;off-the-shelf NLP tools do on this domain. This paper tackles this gap and&#10;performs an in-depth investigation of the characteristics of legal and illegal&#10;text in the Darknet, comparing it to a clear net website with similar content&#10;as a control condition. Taking drug-related websites as a test case, we find&#10;that texts for selling legal and illegal drugs have several linguistic&#10;characteristics that distinguish them from one another, as well as from the&#10;control condition, among them the distribution of POS tags, and the coverage of&#10;their named entities in Wikipedia.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.03122" label="1805.03122">
        <attvalues>
          <attvalue for="0" value="Bleaching Text: Abstract Features for Cross-lingual Gender Prediction" />
          <attvalue for="1" value="  Gender prediction has typically focused on lexical and social network&#10;features, yielding good performance, but making systems highly language-,&#10;topic-, and platform-dependent. Cross-lingual embeddings circumvent some of&#10;these limitations, but capture gender-specific style less. We propose an&#10;alternative: bleaching text, i.e., transforming lexical strings into more&#10;abstract features. This study provides evidence that such features allow for&#10;better transfer across languages. Moreover, we present a first study on the&#10;ability of humans to perform cross-lingual gender prediction. We find that&#10;human predictive power proves similar to that of our bleached models, and both&#10;perform better than lexical models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.09226" label="2106.09226">
        <attvalues>
          <attvalue for="0" value="Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis&#10;  of Head and Prompt Tuning" />
          <attvalue for="1" value="  Pretrained language models have achieved state-of-the-art performance when&#10;adapted to a downstream NLP task. However, theoretical analysis of these models&#10;is scarce and challenging since the pretraining and downstream tasks can be&#10;very different. We propose an analysis framework that links the pretraining and&#10;downstream tasks with an underlying latent variable generative model of text --&#10;the downstream classifier must recover a function of the posterior distribution&#10;over the latent variables. We analyze head tuning (learning a classifier on top&#10;of the frozen pretrained model) and prompt tuning in this setting. The&#10;generative model in our analysis is either a Hidden Markov Model (HMM) or an&#10;HMM augmented with a latent memory component, motivated by long-term&#10;dependencies in natural language. We show that 1) under certain non-degeneracy&#10;conditions on the HMM, simple classification heads can solve the downstream&#10;task, 2) prompt tuning obtains downstream guarantees with weaker non-degeneracy&#10;conditions, and 3) our recovery guarantees for the memory-augmented HMM are&#10;stronger than for the vanilla HMM because task-relevant information is easier&#10;to recover from the long-term memory. Experiments on synthetically generated&#10;data from HMMs back our theoretical findings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.04834" label="2210.04834">
        <attvalues>
          <attvalue for="0" value="Knowledge Distillation Transfer Sets and their Impact on Downstream NLU&#10;  Tasks" />
          <attvalue for="1" value="  Teacher-student knowledge distillation is a popular technique for compressing&#10;today's prevailing large language models into manageable sizes that fit&#10;low-latency downstream applications. Both the teacher and the choice of&#10;transfer set used for distillation are crucial ingredients in creating a high&#10;quality student. Yet, the generic corpora used to pretrain the teacher and the&#10;corpora associated with the downstream target domain are often significantly&#10;different, which raises a natural question: should the student be distilled&#10;over the generic corpora, so as to learn from high-quality teacher predictions,&#10;or over the downstream task corpora to align with finetuning? Our study&#10;investigates this trade-off using Domain Classification (DC) and Intent&#10;Classification/Named Entity Recognition (ICNER) as downstream tasks. We distill&#10;several multilingual students from a larger multilingual LM with varying&#10;proportions of generic and task-specific datasets, and report their performance&#10;after finetuning on DC and ICNER. We observe significant improvements across&#10;tasks and test sets when only task-specific corpora is used. We also report on&#10;how the impact of adding task-specific data to the transfer set correlates with&#10;the similarity between generic and task-specific data. Our results clearly&#10;indicate that, while distillation from a generic LM benefits downstream tasks,&#10;students learn better using target domain data even if it comes at the price of&#10;noisier teacher predictions. In other words, target domain data still trumps&#10;teacher knowledge.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the recent past, large language models (LMs; BERT-Large, \cite{devlin-etal-2019-bert}; GPT-2, \cite{Radford2019LanguageMA}; T5, \cite{JMLR:v21:20-074}) pretrained in a self-supervised manner on massive web corpora have consistently shown state-of-the-art performance for multiple natural language understanding (NLU) tasks. Therefore, it is no surprise that these models are of much interest for virtual assistants such as Amazon Alexa, Apple Siri, and Google Assistant. Some studies have shown that these large models trained on generic corpora seem to be more robust to data distributional shifts, relying less on domain-specific training data to perform well \cite{NEURIPS2020_1457c0d6}.&#10;&#10;Since large models cannot be directly used for low-latency applications on devices with limited computing capacity, many techniques have been developed to compress them in size. Knowledge distillation (referred to simply as distillation hereafter; \cite{hinton2015distilling}), has shown promising results, especially at the high compression rates typically required in NLU (\cite{jiao-etal-2020-tinybert}, \cite{soltan-etal-2021-limitations}). In this paradigm, lightweight models referred to as students, are trained to mimic the teacher predictions over a transfer set \cite{hinton2015distilling}. When the pretraining and task-specific corpora have significantly different distributions, as is often the case, the choice of data for the transfer set can be ambiguous. On the one hand, using pretraining corpora in the transfer set ensures high quality teacher predictions that are important for effective distillation. On the other, using the downstream corpora, although it might cause noisier teacher predictions, ensures the adaptation of the student to its final use case.&#10;&#10;To investigate this trade-off, we present a set of experiments where we distill several multilingual students from a large multilingual teacher LM trained using a masked language modeling (MLM) objective. We perform the distillations using transfer sets that comprise of generic and task-specific data in varying proportions. The students are then finetuned and evaluated on two downstream NLU tasks of interest: a Domain Classification (DC) task and a joint Intent Classification/Named Entity Recognition (ICNER) task. For each input utterance DC predicts the relevant domain (Books, Music, Shopping, etc.), IC identifies the user's intent (find a book, play a song, buy an item, etc.) and NER extracts the entities in the utterance (dates, names, locations, etc.).&#10;&#10;{Our contributions}: (1) We confirm for our setup that model preparation via distillation from a larger LM is more beneficial for downstream task performance when compared to encoder training from scratch. (2) We show that the largest improvements are seen when using only the downstream task's unlabelled data during the distillation process. Even though teacher predictions are expected to be noisy over data that is different from pretraining corpora, our results clearly indicate that students learn best in this setting. (3) Because our ICNER corpora is divided per domain, we are also able to provide a finer-grained analysis of the impact of corpora similarity on downstream results. (4) Finally, we also confirm that further adaptation of the teacher to the target-domain data, results in improved student performance across tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Distillation, Domain Adaptation Techniques, Multilingual Student Training, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2406.19486" label="2406.19486">
        <attvalues>
          <attvalue for="0" value="LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models" />
          <attvalue for="1" value="  In prompt tuning, a prefix or suffix text is added to the prompt, and the&#10;embeddings (soft prompts) or token indices (hard prompts) of the prefix/suffix&#10;are optimized to gain more control over language models for specific tasks.&#10;This approach eliminates the need for hand-crafted prompt engineering or&#10;explicit model fine-tuning. Prompt tuning is significantly more&#10;parameter-efficient than model fine-tuning, as it involves optimizing partial&#10;inputs of language models to produce desired outputs.&#10;  In this work, we aim to further reduce the amount of trainable parameters&#10;required for a language model to perform well on specific tasks. We propose&#10;Low-rank Prompt Tuning (LoPT), a low-rank model for prompts that achieves&#10;efficient prompt optimization. The proposed method demonstrates similar&#10;outcomes to full parameter prompt tuning while reducing the number of trainable&#10;parameters by a factor of 5. It also provides promising results compared to the&#10;state-of-the-art methods that would require 10 to 20 times more parameters.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the success of large language models \cite{touvron2023llama,achiam2023gpt,jiang2023mistral}, it has become increasingly important for language models (LMs) to handle instructions effectively for customized agents and tasks. &#10;There are three essential categories of methods to adapt pre-trained language models to specific and customized needs: prompt engineering, model fine-tuning, and prompt tuning.&#10;&#10;Prompt engineering \cite{brown2020language,sanh2021multitask,chung2024scaling} &#10;involves crafting handcrafted prompts and faces the challenge of getting LMs to consistently produce desired outputs with few-shot instructions. &#10;This effort may be difficult to generalize or extend to new tasks.&#10;Model fine-tuning \cite{raffel2020exploring} can perform very well for task-specific needs but requires explicit fine-tuning of a significant number of model parameters, even with parameter-efficient fine-tuning (PEFT) approaches \cite{liu2022few, hu2021lora}.&#10;&#10;Prompt tuning (PT) \cite{li2021prefix,lester2021power,wen2024hard,shi2022toward,shin2020autoprompt,khashabi2021prompt} is a promising method that lies between prompt engineering and model fine-tuning. Instead of handcrafting prompts, it optimizes a small number of prompt embeddings or indices with training data and has demonstrated capabilities comparable to those of model fine-tuning approaches \cite{asai2022attempt,shi2023dept,wang2023multitask}. &#10;&#10;We focus on soft prompt tuning, which operates by adding a prefix or suffix to the existing inputs and optimizing the embeddings of this prefix or suffix. The embeddings, or the soft prompt matrix, has dimensions $n \times d$, where $n$ is the ``tokens&quot; length of soft prompts, and $d$ is the embedding size. The soft prompt length $n$ can be task specific to achieve desired outcomes. For example, more sophisticated tasks might benefit from longer soft prompts that allow for more parameters to be optimized.&#10;&#10;In this work, we introduce a low-rank modeling approach for the soft prompt matrix, which effectively reduces the number of trainable parameters in prompt tuning without compromising performance. We find that soft prompt matrices are inherently low-rank due to their dimensionality, and we apply further dimensionality reduction through our proposed method. We demonstrate that the number of parameters required for tuning LMs to meet specific task requirements can be minimal. Additionally, the number of trainable parameters can be easily controlled by adjusting the rank of the soft prompt matrix.&#10;&#10;Our approach distinguishes itself from existing methods by directly imposing low-rank constraints on the entire soft prompt to be trained. While recent work \cite{shi2023dept} also explores low-rank matrices for prompt tuning, it restricts low-rankness to the differences or updates of a frozen baseline prompt, similar to the LoRA technique used in model fine-tuning \cite{hu2021lora}, and is only applied to a portion of the overall soft prompt.&#10;&#10;Our primary contributions are:&#10;\begin{itemize}&#10;\item We introduce Low-rank Prompt Tuning (LoPT) that significantly reduces the number of trainable parameters required in prompt tuning. &#10;\item We achieve a 5-fold reduction in trainable parameters while maintaining performance comparable to the full-parameter prompt tuning.&#10;\item We demonstrate the efficacy of our method across 5 diverse datasets, showing substantial improvements in parameter efficiency compared to existing methods.&#10;\end{itemize}&#10;&#10;Our proposed parameter-efficient method would be particularly beneficial for computationally demanding prompt tuning needs in sophisticated tasks and large language models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Prompt Engineering Techniques, Artificial Intelligence, Mathematics, Parameter Efficiency Methods, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2302.03668" label="2302.03668">
        <attvalues>
          <attvalue for="0" value="Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt&#10;  Tuning and Discovery" />
          <attvalue for="1" value="  The strength of modern generative models lies in their ability to be&#10;controlled through text-based prompts. Typical &quot;hard&quot; prompts are made from&#10;interpretable words and tokens, and must be hand-crafted by humans. There are&#10;also &quot;soft&quot; prompts, which consist of continuous feature vectors. These can be&#10;discovered using powerful optimization methods, but they cannot be easily&#10;interpreted, re-used across models, or plugged into a text-based interface.&#10;  We describe an approach to robustly optimize hard text prompts through&#10;efficient gradient-based optimization. Our approach automatically generates&#10;hard text-based prompts for both text-to-image and text-to-text applications.&#10;In the text-to-image setting, the method creates hard prompts for diffusion&#10;models, allowing API users to easily generate, discover, and mix and match&#10;image concepts without prior knowledge on how to prompt the model. In the&#10;text-to-text setting, we show that hard prompts can be automatically discovered&#10;that are effective in tuning LMs for classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10539" label="2212.10539">
        <attvalues>
          <attvalue for="0" value="Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good&#10;  movie, and a good prompt too?" />
          <attvalue for="1" value="  Large language models can perform new tasks in a zero-shot fashion, given&#10;natural language prompts that specify the desired behavior. Such prompts are&#10;typically hand engineered, but can also be learned with gradient-based methods&#10;from labeled data. However, it is underexplored what factors make the prompts&#10;effective, especially when the prompts are natural language. In this paper, we&#10;investigate common attributes shared by effective prompts. We first propose a&#10;human readable prompt tuning method (F LUENT P ROMPT) based on Langevin&#10;dynamics that incorporates a fluency constraint to find a diverse distribution&#10;of effective and fluent prompts. Our analysis reveals that effective prompts&#10;are topically related to the task domain and calibrate the prior probability of&#10;label words. Based on these findings, we also propose a method for generating&#10;prompts using only unlabeled data, outperforming strong baselines by an average&#10;of 7.0% accuracy across three tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08348" label="2112.08348">
        <attvalues>
          <attvalue for="0" value="Prompt Waywardness: The Curious Case of Discretized Interpretation of&#10;  Continuous Prompts" />
          <attvalue for="1" value="  Fine-tuning continuous prompts for target tasks has recently emerged as a&#10;compact alternative to full model fine-tuning. Motivated by these promising&#10;results, we investigate the feasibility of extracting a discrete (textual)&#10;interpretation of continuous prompts that is faithful to the problem they&#10;solve. In practice, we observe a &quot;wayward&quot; behavior between the task solved by&#10;continuous prompts and their nearest neighbor discrete projections: We can find&#10;continuous prompts that solve a task while being projected to an arbitrary text&#10;(e.g., definition of a different or even a contradictory task), while being&#10;within a very small (2%) margin of the best continuous prompt of the same size&#10;for the task. We provide intuitions behind this odd and surprising behavior, as&#10;well as extensive empirical analyses quantifying the effect of various&#10;parameters. For instance, for larger model sizes we observe higher waywardness,&#10;i.e, we can find prompts that more closely map to any arbitrary text with a&#10;smaller drop in accuracy. These findings have important implications relating&#10;to the difficulty of faithfully interpreting continuous prompts and their&#10;generalization across models and tasks, providing guidance for future progress&#10;in prompting language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.11961" label="2205.11961">
        <attvalues>
          <attvalue for="0" value="ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures&#10;  of Soft Prompts" />
          <attvalue for="1" value="  This work introduces a new multi-task, parameter-efficient language model&#10;(LM) tuning method that learns to transfer knowledge across different tasks via&#10;a mixture of soft prompts-small prefix embedding vectors pre-trained for&#10;different tasks. Our method, called ATTEMPT (ATTEntional Mixtures of Prompt&#10;Tuning), obtains source prompts as encodings of large-scale source tasks into a&#10;small number of parameters and trains an attention module to interpolate the&#10;source prompts and a newly initialized target prompt for every instance in the&#10;target task. During training, only the target task prompt and the attention&#10;weights, which are shared between tasks in multi-task training, are updated,&#10;while the original LM and source prompts are intact. ATTEMPT is highly&#10;parameter-efficient (e.g., updates 2,300 times fewer parameters than full&#10;fine-tuning) while achieving high task performance using knowledge from&#10;high-resource tasks. Moreover, it is modular using pre-trained soft prompts,&#10;and can flexibly add or remove source prompts for effective knowledge transfer.&#10;Our experimental results across 21 diverse NLP datasets show that ATTEMPT&#10;significantly outperforms prompt tuning and outperforms or matches fully&#10;fine-tuned or other parameter-efficient tuning approaches that use over ten&#10;times more parameters. Finally, ATTEMPT outperforms previous work in few-shot&#10;learning settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Parameter-efficient tuning.&#10;Here, we enlist additional parameter-efficient tuning methods that are close to our work.&#10;AdapterFusion~\cite{pfeiffer-etal-2021-adapterfusion} compose multiple different adapters by learning task-specific compositions on each task, and \cite{friedman-etal-2021-single} take an average of multiple adapter layers after training adapters individually on different QA datasets. &#10;HyperFormer~\cite{karimi-mahabadi-etal-2021-parameter_custom} and HyperDecoder~\cite{ivison2022hyperdecoders} train a shared hyper network to generate parameters of adapter layers.&#10;\cite{qin-eisner-2021-learning} introduce mixture of soft prompts, where predictions given different prompts are ensembled for the same knowledge base relationship types. &#10;IDPG~\cite{wu2022idpg} and Instance-Dependent Prompt Tuning~\cite{levine2022standing} learn to generate instance-wise prompts given the input encoded by LMs. &#10;Compared to the previous work, our main focus is transferring knowledge from multiple tasks to produce soft prompts rather than learning to generate them from scratch, and is much more efficient in terms of parameters and inference time. &#10;&#10;Concurrent to our work, \cite{liu2022few} introduce (IA)$^3$ that multiplies intermediate activation by learned vectors for few-shot learning. &#10;{\cite{wang2022learning} shows that combining a set of prompts retrieved from the prompt pool by a key-value mechanism yields competitive performance in computer vision continual learning. }&#10;For generation tasks, \cite{li2022learning} transfer multiple source prompts using multi-key memory network for prompt clustering and multi-head attention taking another LM output. &#10;{In contrast, we present an efficient multi-task tuning that is effective in diverse NLP tasks. More importantly, prior work often relies on priors such as pre-computed clusters or another LM's predictions of which tasks should be used as source tasks. &#10;{\sc Attempt}~removes the necessity of such priors by training an attention layer that learn to focus on relevant source tasks.}&#10;&#10;{Several recent lines of research attempt to adapt a massive multi-task LM trained with instructions or demonstrations to a new task without any parameter updates~\cite{sanh2022multitask,min2021metaicl,wang2022benchmarking,wei2022finetuned}. &#10;The main focus of this paper is how to efficiently transfer rich multi-task knowledge from source tasks to target tasks with training data during target task training, while those work often emphasize on zero or few-shot transfer without any parameter updates.} &#10;&#10;Modular multi-task training.&#10;There is a large literature on composing multiple separate networks to handle different sub-tasks~\cite{jacobs1991adaptive,jacobs1991task,andreas2016neural,mccann2018natural}. &#10;As the LM size expands, several recent work tries to sparsely activate or employ light-weight modules for efficient multi-task learning~\cite{gupta2022sparsely,ponti2022combining,fedus2021switch}. &#10;In particular, we share the same intuition as the &#10;concurrent work~\cite{ponti2022combining}, which combines several skills encapsulated in parameter-efficient modules; however, our main focus is on how to transfer and share knowledge from resource-rich tasks in a super parameter-efficient way, while they focus on improving few-shot generalization ability. Moreover, {\sc Attempt}~keeps LMs intact and updates fewer parameters.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2309.05173" label="2309.05173">
        <attvalues>
          <attvalue for="0" value="DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning" />
          <attvalue for="1" value="  Prompt tuning (PT), where a small amount of trainable soft (continuous)&#10;prompt vectors is affixed to the input of language models (LM), has shown&#10;promising results across various tasks and models for parameter-efficient&#10;fine-tuning (PEFT). PT stands out from other PEFT approaches because it&#10;maintains competitive performance with fewer trainable parameters and does not&#10;drastically scale up its parameters as the model size expands. However, PT&#10;introduces additional soft prompt tokens, leading to longer input sequences,&#10;which significantly impacts training and inference time and memory usage due to&#10;the Transformer's quadratic complexity. Particularly concerning for Large&#10;Language Models (LLMs) that face heavy daily querying. To address this issue,&#10;we propose Decomposed Prompt Tuning (DePT), which decomposes the soft prompt&#10;into a shorter soft prompt and a pair of low-rank matrices that are then&#10;optimised with two different learning rates. This allows DePT to achieve better&#10;performance while saving substantial memory and time costs compared to vanilla&#10;PT and its variants, without changing trainable parameter sizes. Through&#10;extensive experiments on 23 natural language processing (NLP) and&#10;vision-language (VL) tasks, we demonstrate that DePT outperforms&#10;state-of-the-art PEFT approaches, including the full fine-tuning baseline, in&#10;some scenarios. Additionally, we empirically show that DEPT grows more&#10;efficient as the model size increases. Our further study reveals that DePT&#10;integrates seamlessly with parameter-efficient transfer learning in the&#10;few-shot learning setting and highlights its adaptability to various model&#10;architectures and sizes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.05577" label="2208.05577">
        <attvalues>
          <attvalue for="0" value="Reducing Retraining by Recycling Parameter-Efficient Prompts" />
          <attvalue for="1" value="  Parameter-efficient methods are able to use a single frozen pre-trained large&#10;language model (LLM) to perform many tasks by learning task-specific soft&#10;prompts that modulate model behavior when concatenated to the input text.&#10;However, these learned prompts are tightly coupled to a given frozen model --&#10;if the model is updated, corresponding new prompts need to be obtained. In this&#10;work, we propose and investigate several approaches to &quot;Prompt Recycling'&quot;&#10;where a prompt trained on a source model is transformed to work with the new&#10;target model. Our methods do not rely on supervised pairs of prompts,&#10;task-specific data, or training updates with the target model, which would be&#10;just as costly as re-tuning prompts with the target model from scratch. We show&#10;that recycling between models is possible (our best settings are able to&#10;successfully recycle $88.9\%$ of prompts, producing a prompt that out-performs&#10;baselines), but significant performance headroom remains, requiring improved&#10;recycling techniques.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Our vocab-to-vocab transformations are similar to cross-lingual embedding mapping \cite{mikolovExploitingSimilaritiesLanguages2013}, except that our mapping is applied to models trained on the same language.&#10;&#10;The ``soft prompt transfer'' technique of \cite{vuSPoTBetterFrozen2022} is similar to ours in that prompts are reused between different models; however, \cite{vuSPoTBetterFrozen2022} focuses on transfer between different tasks with the same pre-trained model while we focus on transfer between different models trained with the same task. Additionally, their method assumes the prompt will be updated with training data for the new task. In contrast, our method does not update the prompt after recycling as that as it would remove the computational advantage over training a prompt on $M_t$ directly. The finding from \cite{vuSPoTBetterFrozen2022} that one prompt can be successfully used to initialize another suggests that using a recycled prompt as initialization could also be worth investigating.&#10;&#10;Work from \cite{suTransferabilityPromptTuning2022} also investigates the transfer of prompts across models. However, their work focuses on knowledge transfer between very different pre-trained language models (different datasets, different architectures, etc.\@) with the aim of increasing final model performance, whereas our work approaches prompt recycling as a way to avoid re-training prompts when a frozen model is updated. This difference in motivation is reflected in the difference in approaches. Their proposed approaches generally require task-specific data and paired prompts, or require training a prompt transformer using $M_t$ as part of the calculation, which incurs costs on par with re-training the prompt from scratch on the target model. In contrast, our methods only require the embeddings of the source and target models, and can be reused across tasks.&#10; " />
          <attvalue for="4" value="Prompt Learning Techniques, Computer Science, Linguistics, Cognitive Science, Model Transfer Methods, Mathematics, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2110.02782" label="2110.02782">
        <attvalues>
          <attvalue for="0" value="How BPE Affects Memorization in Transformers" />
          <attvalue for="1" value="  Training data memorization in NLP can both be beneficial (e.g., closed-book&#10;QA) and undesirable (personal data extraction). In any case, successful model&#10;training requires a non-trivial amount of memorization to store word spellings,&#10;various linguistic idiosyncrasies and common knowledge. However, little is&#10;known about what affects the memorization behavior of NLP models, as the field&#10;tends to focus on the equally important question of generalization. In this&#10;work, we demonstrate that the size of the subword vocabulary learned by&#10;Byte-Pair Encoding (BPE) greatly affects both ability and tendency of standard&#10;Transformer models to memorize training data, even when we control for the&#10;number of learned parameters. We find that with a large subword vocabulary&#10;size, Transformer models fit random mappings more easily and are more&#10;vulnerable to membership inference attacks. Similarly, given a prompt,&#10;Transformer-based language models with large subword vocabularies reproduce the&#10;training data more often. We conjecture this effect is caused by reduction in&#10;the sequences' length that happens as the BPE vocabulary grows. Our findings&#10;can allow a more informed choice of hyper-parameters, that is better tailored&#10;for a particular use-case.&#10;" />
          <attvalue for="2" value="&#10;The Transformer architecture~\cite{Vaswani2017} became the backbone of the state-of-the-art models in a variety of tasks~\cite{Liu2019,Raffel2019,Adiwardana2020,Brown2020}. &#10;This spurred a significant interest in better understanding inner workings of these models~\cite{Vig2019,Clark2019,Kharitonov2020,Hahn2020,Movva2020,Chaabouni2021,Merrill2021,sinha2021masked}.&#10;Most of these works have focussed specifically on how models generalize and capture structure across samples that are similar.&#10;For instance, \cite{Vig2019} focussed on how attention align with specific syntactic dependency relations, \cite{Hupkes2020} considered if Transformers generalize compositionally and \cite{Kharitonov2020} studied how different models generalize from very few data.&#10;In contrast to these studies, we focus on factors that control the training data {memorization} behavior of Transformers, which we believe to be important for several reasons.&#10;&#10;First,&#10;large Transformer models are increasingly often used as a storage, for instance, as a general-purpose knowledge base or as a closed-book question-answering system~\cite{Petroni2019,Roberts2020,Lewis2020}. &#10;Clearly, the ability to {memorize} factual knowledge from the training data is crucial for such applications. &#10;There are even Transformers models that are explicitly endowed with an external training data memorization mechanism~\cite{Khandelwal2020,Khandelwal2021,He2021}, demonstrating that further boosting their memorization abilities is beneficial. &#10;&#10;Second, in contrast, the same ability to memorize can become undesirable and lead to a leakage of personal data from trained models~\cite{Carlini2020,Thakkar2021}. &#10;A better understanding of the phenomenon is thus instrumental both to enable better memorization when it is needed and to avoid it when not.&#10;&#10;Third, while generalization and memorization are often thought of as competing modes of fitting data, training effective models in real tasks requires a non-trivial combination of the two. &#10;For instance, successful language models need to generalize to be able to deal with never-seen-before sentences, but they also need to {memorize} the spelling of words, the non-compositional meaning of idioms, idiosyncrasies of languages, common knowledge, etc \cite[see, e.g.][]{dankers2021paradox}.&#10;&#10;Despite this apparent importance, there is very little research into memorization in Transformers and in NLP models in general, and we have only superficial understanding of what factors affect this behavior.&#10;Intuitively, the number of parameters, data augmentation, and regularization are likely to affect how successful models are in memorization~\cite{Zhang2016,Sablayrolles2018}.&#10;In this work, we primarily focus on the influence of a less obvious yet important factor: we study how the selection of modelling units affects memorization. Typically, the same data can be represented on various levels: raw bytes and their groups, individual characters, subword units, and entire words. A very common approach is to learn subword-level vocabulary with Byte-Pair Encoding (BPE)~\cite{Sennrich2015} or similar methods~\cite[e.g.,][]{Devlin2018,Kudo2018,Provilkov2019}. &#10;In spite of ubiquitous use of these methods, to the best of our knowledge, there is no clear understanding of how {the number of subwords or BPE operations should be chosen} and how this affect behavior of a model.&#10;{We expect that BPE-like segmentation might play a crucial role in memorization, as it controls the trade-off between the number of primitives a model will have to operate with and the lengths of sequences it must represent.}&#10;&#10;In this work, to characterize a model's behavior, we measure three ``facets'' of training data memorization. First, as a proxy for the memorization capacity of a model, we use its ability to fit random, non-systematic mappings. Next, we study the preference&#10;for memorization when generalization is possible. For that, we study how easy it is to accurately tell if a particular example was used in the model's training data via a membership inference attack~\cite{Shokri2017}.&#10;Finally, we examine how easy it is to recover training data from a trained language model. We experiment with three Transformer architectures: causal \&amp; masked language models, and encoder-based classifiers. &#10;&#10;Our main experimental finding is that,&#10;across all architectures and tasks, the choice of modeling units strongly affects the memorization behavior of the models, with large-cardinality BPE vocabularies greatly facilitating memorization.&#10;This observation holds even when we control for the number of trainable parameters. &#10;&#10;After establishing this fact, we look deeper into the causes of the phenomenon we observe. We examine three candidate causes which are principal (side-)effects of applying BPE: (i) removing redundancy in the data (due to compression), (ii) increase in the number of the unique units used to represent the data, or (iii) reducing the length of the training sequences. By finding a similar effect with incompressible randomly generated data we can rule out the first possibility. Next, we artificially double the vocabulary size by introducing ``synonym'' tokens and observe that the vocabulary growth, in isolation, leads to a different memorization pattern. Thus, by exclusion, we conjecture that reducing utterance length is, at least, a very important factor of memorization.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Vulnerabilities, Data Memorization Effects, Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing, NLP Model Training" />
        </attvalues>
      </node>
      <node id="2006.14953" label="2006.14953">
        <attvalues>
          <attvalue for="0" value="What they do when in doubt: a study of inductive biases in seq2seq&#10;  learners" />
          <attvalue for="1" value="  Sequence-to-sequence (seq2seq) learners are widely used, but we still have&#10;only limited knowledge about what inductive biases shape the way they&#10;generalize. We address that by investigating how popular seq2seq learners&#10;generalize in tasks that have high ambiguity in the training data. We use SCAN&#10;and three new tasks to study learners' preferences for memorization,&#10;arithmetic, hierarchical, and compositional reasoning. Further, we connect to&#10;Solomonoff's theory of induction and propose to use description length as a&#10;principled and sensitive measure of inductive biases.&#10;  In our experimental study, we find that LSTM-based learners can learn to&#10;perform counting, addition, and multiplication by a constant from a single&#10;training example. Furthermore, Transformer and LSTM-based learners show a bias&#10;toward the hierarchical induction over the linear one, while CNN-based learners&#10;prefer the opposite. On the SCAN dataset, we find that CNN-based, and, to a&#10;lesser degree, Transformer- and LSTM-based learners have a preference for&#10;compositional generalization over memorization. Finally, across all our&#10;experiments, description length proved to be a sensitive measure of inductive&#10;biases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.13270" label="2009.13270">
        <attvalues>
          <attvalue for="0" value="Dissecting Lottery Ticket Transformers: Structural and Behavioral Study&#10;  of Sparse Neural Machine Translation" />
          <attvalue for="1" value="  Recent work on the lottery ticket hypothesis has produced highly sparse&#10;Transformers for NMT while maintaining BLEU. However, it is unclear how such&#10;pruning techniques affect a model's learned representations. By probing&#10;Transformers with more and more low-magnitude weights pruned away, we find that&#10;complex semantic information is first to be degraded. Analysis of internal&#10;activations reveals that higher layers diverge most over the course of pruning,&#10;gradually becoming less complex than their dense counterparts. Meanwhile, early&#10;layers of sparse models begin to perform more encoding. Attention mechanisms&#10;remain remarkably consistent as sparsity increases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.01366" label="2107.01366">
        <attvalues>
          <attvalue for="0" value="Can Transformers Jump Around Right in Natural Language? Assessing&#10;  Performance Transfer from SCAN" />
          <attvalue for="1" value="  Despite their practical success, modern seq2seq architectures are unable to&#10;generalize systematically on several SCAN tasks. Hence, it is not clear if&#10;SCAN-style compositional generalization is useful in realistic NLP tasks. In&#10;this work, we study the benefit that such compositionality brings about to&#10;several machine translation tasks. We present several focused modifications of&#10;Transformer that greatly improve generalization capabilities on SCAN and select&#10;one that remains on par with a vanilla Transformer on a standard machine&#10;translation (MT) task. Next, we study its performance in low-resource settings&#10;and on a newly introduced distribution-shifted English-French translation task.&#10;Overall, we find that improvements of a SCAN-capable model do not directly&#10;transfer to the resource-rich MT setup. In contrast, in the low-resource setup,&#10;general modifications lead to an improvement of up to 13.1% BLEU score w.r.t. a&#10;vanilla Transformer. Similarly, an improvement of 14% in an accuracy-based&#10;metric is achieved in the introduced compositional English-French translation&#10;task. This provides experimental evidence that the compositional generalization&#10;assessed in SCAN is particularly useful in resource-starved and domain-shifted&#10;scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Compositionally-biased models &#10;Several approaches were proposed to build SCAN-capable architectures. They span from meta-learning~\cite{Lake2019}, disentangling syntax and semantics~\cite{Russin2019}, learning equivariant~\cite{Gordon2019} and disentangled representations~\cite{Li2019} or combining neural \&amp; symbolic computations~\cite{Chen2020}.&#10;In contrast, we do not build new models that are specialized to SCAN. Instead, we show that a standard model can be incrementally modified so that performs well on SCAN and still performs well on a standard MT task. Having such incrementally improved models allows us to step back and wonder if SCAN (or similar artificial tasks) should be used as a guidance when developing new models.&#10;&#10;\cite{Bastings2018} raised concerns due to SCAN being too artificial by showing that even degenerate architectures can perform well on some SCAN tasks. Our results echo their findings: by developing architectures tailored for SCAN, one can easily come up with models that perform worse on general tasks. However, we find that if one avoids this ``SCAN overfitting'' and endows a model with capabilities that SCAN probes for without harming its general performance, they can gain in low-resource scenarios and better handle relevant phenomena in language.&#10;&#10;Changing attention mechanisms Self- and cross-attention mechanisms were tweaked in earlier work in order to inject useful biases, e.g., by adding information of relative positions of tokens~\cite{Shaw2018,Raffel2019} or accounting for the locality bias in cross-attention~\cite{Yang2018}. \cite{Sukhbaatar2019} and \cite{Rae2020} demonstrated that having a short attention span on the lower layers of Transformer models is enough for good language modeling performance. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2010.09697" label="2010.09697">
        <attvalues>
          <attvalue for="0" value="Effects of Parameter Norm Growth During Transformer Training: Inductive&#10;  Bias from Gradient Descent" />
          <attvalue for="1" value="  The capacity of neural networks like the widely adopted transformer is known&#10;to be very high. Evidence is emerging that they learn successfully due to&#10;inductive bias in the training routine, typically a variant of gradient descent&#10;(GD). To better understand this bias, we study the tendency for transformer&#10;parameters to grow in magnitude ($\ell_2$ norm) during training, and its&#10;implications for the emergent representations within self attention layers.&#10;Empirically, we document norm growth in the training of transformer language&#10;models, including T5 during its pretraining. As the parameters grow in&#10;magnitude, we prove that the network approximates a discretized network with&#10;saturated activation functions. Such &quot;saturated&quot; networks are known to have a&#10;reduced capacity compared to the full network family that can be described in&#10;terms of formal languages and automata. Our results suggest saturation is a new&#10;characterization of an inductive bias implicit in GD of particular interest for&#10;NLP. We leverage the emergent discrete structure in a saturated transformer to&#10;analyze the role of different attention heads, finding that some focus locally&#10;on a small number of positions, while other heads compute global averages,&#10;allowing counting. We believe understanding the interplay between these two&#10;capabilities may shed further light on the structure of computation within&#10;large transformers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.02637" label="2008.02637">
        <attvalues>
          <attvalue for="0" value="Question and Answer Test-Train Overlap in Open-Domain Question Answering&#10;  Datasets" />
          <attvalue for="1" value="  Ideally Open-Domain Question Answering models should exhibit a number of&#10;competencies, ranging from simply memorizing questions seen at training time,&#10;to answering novel question formulations with answers seen during training, to&#10;generalizing to completely novel questions with novel answers. However, single&#10;aggregated test set scores do not show the full picture of what capabilities&#10;models truly have. In this work, we perform a detailed study of the test sets&#10;of three popular open-domain benchmark datasets with respect to these&#10;competencies. We find that 60-70% of test-time answers are also present&#10;somewhere in the training sets. We also find that 30% of test-set questions&#10;have a near-duplicate paraphrase in their corresponding training sets. Using&#10;these findings, we evaluate a variety of popular open-domain models to obtain&#10;greater insight into what extent they can actually generalize, and what drives&#10;their overall performance. We find that all models perform dramatically worse&#10;on questions that cannot be memorized from training sets, with a mean absolute&#10;performance difference of 63% between repeated and non-repeated data. Finally&#10;we show that simple nearest-neighbor models out-perform a BART closed-book QA&#10;model, further highlighting the role that training set memorization plays in&#10;these benchmarks&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.03530" label="1611.03530">
        <attvalues>
          <attvalue for="0" value="Understanding deep learning requires rethinking generalization" />
          <attvalue for="1" value="  Despite their massive size, successful deep artificial neural networks can&#10;exhibit a remarkably small difference between training and test performance.&#10;Conventional wisdom attributes small generalization error either to properties&#10;of the model family, or to the regularization techniques used during training.&#10;  Through extensive systematic experiments, we show how these traditional&#10;approaches fail to explain why large neural networks generalize well in&#10;practice. Specifically, our experiments establish that state-of-the-art&#10;convolutional networks for image classification trained with stochastic&#10;gradient methods easily fit a random labeling of the training data. This&#10;phenomenon is qualitatively unaffected by explicit regularization, and occurs&#10;even if we replace the true images by completely unstructured random noise. We&#10;corroborate these experimental findings with a theoretical construction showing&#10;that simple depth two neural networks already have perfect finite sample&#10;expressivity as soon as the number of parameters exceeds the number of data&#10;points as it usually does in practice.&#10;  We interpret our experimental findings by comparison with traditional models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.05820" label="1610.05820">
        <attvalues>
          <attvalue for="0" value="Membership Inference Attacks against Machine Learning Models" />
          <attvalue for="1" value="  We quantitatively investigate how machine learning models leak information&#10;about the individual data records on which they were trained. We focus on the&#10;basic membership inference attack: given a data record and black-box access to&#10;a model, determine if the record was in the model's training dataset. To&#10;perform membership inference against a target model, we make adversarial use of&#10;machine learning and train our own inference model to recognize differences in&#10;the target model's predictions on the inputs that it trained on versus the&#10;inputs that it did not train on.&#10;  We empirically evaluate our inference techniques on classification models&#10;trained by commercial &quot;machine learning as a service&quot; providers such as Google&#10;and Amazon. Using realistic datasets and classification tasks, including a&#10;hospital discharge dataset whose membership is sensitive from the privacy&#10;perspective, we show that these models can be vulnerable to membership&#10;inference attacks. We then investigate the factors that influence this leakage&#10;and evaluate mitigation strategies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.16431" label="2407.16431">
        <attvalues>
          <attvalue for="0" value="FairFlow: An Automated Approach to Model-based Counterfactual Data&#10;  Augmentation For NLP" />
          <attvalue for="1" value="  Despite the evolution of language models, they continue to portray harmful&#10;societal biases and stereotypes inadvertently learned from training data. These&#10;inherent biases often result in detrimental effects in various applications.&#10;Counterfactual Data Augmentation (CDA), which seeks to balance demographic&#10;attributes in training data, has been a widely adopted approach to mitigate&#10;bias in natural language processing. However, many existing CDA approaches rely&#10;on word substitution techniques using manually compiled word-pair dictionaries.&#10;These techniques often lead to out-of-context substitutions, resulting in&#10;potential quality issues. The advancement of model-based techniques, on the&#10;other hand, has been challenged by the need for parallel training data. Works&#10;in this area resort to manually generated parallel data that are expensive to&#10;collect and are consequently limited in scale. This paper proposes FairFlow, an&#10;automated approach to generating parallel data for training counterfactual text&#10;generator models that limits the need for human intervention. Furthermore, we&#10;show that FairFlow significantly overcomes the limitations of dictionary-based&#10;word-substitution approaches whilst maintaining good performance.&#10;" />
          <attvalue for="2" value="&#10;Despite their growing popularity and unprecedented performance in various application domains, language models (LMs) continue to be plagued with issues of harmful societal biases and stereotypes that have been shown to have detrimental social effects \cite{dastin2022amazon}. &#10;The biggest contributing factor is the encapsulation of societal biases in everyday language, as is well-documented \cite{beukeboom2017linguistic,porter2016inferring,fiedler2011social}. LMs heavily rely on such textual data, now digitalized on various online outlets, as training data, causing them to mirror these biases \cite{wolf2017we}. &#10;&#10;In Natural Language Processing (NLP), similar to many machine learning domains, bias mitigation generally occurs at three intervention avenues: the training data, the learning procedure, or the model output \cite{lohia2019bias}.&#10;Since model bias traces its roots to the training data, mitigating bias at the training data level has proven very effective \cite{dixon2018measuring,de2019bias}. &#10;One such approach, Counterfactual Data Augmentation (CDA) \cite{dattagender}, seeks to remove spurious correlations between attributes in the training data by evening out the distribution of words that characterize demographic attributes in the context of neutral words that should ideally not be demographically aligned. Specifically, explicit attribute-defining words are replaced with their counterfactual equivalents from complementary demographic groups for every text instance. &#10;To illustrate this with an example, an instance of ``She is a nurse'' will be augmented with ``He is a nurse'' in the case of mitigating gender bias. This follows the intuition that in an ideal dataset, the association between gender attributes and target attributes like professions will be even for different gender groups. &#10;&#10;Key works, such as \cite{zhao-etal-2018-gender,lu2020gender,zmigrod-etal-2019-counterfactual}, introducing CDA as a bias mitigation technique adopt a word substitution approach based on dictionaries. &#10;These word substitution methods are prone to grammatical incoherence because of out-of-context substitutions and omitted word pairs. &#10;Because dictionary compilations are often incomplete \cite{dinan2020queens}, a direct word-substitution approach will not generalize to omitted words.&#10;Take for instance (Bachelor and Masters degree v. Spinster and Mistresses degree) and (she taught herself v. he taught herself) which were common issues we observed with some methods.&#10;Additionally, the dictionaries are manually compiled, which not only incurs potential costs but manually compiling counterfactual word pairs for certain demographics may be intrinsically challenging. &#10;&#10;Although generative language models like GPT-related models \cite{radford2018improving} have surged in popularity, their adoption for CDA has been limited due to the relative unavailability of parallel data needed for training. As such, model-based solutions resort to manually compiling parallel training data, a process that is both costly and constrained. This challenge is exacerbated by the fact that training models on limited parallel data can impair performance \cite{zoph2016transfer}.&#10;Although large conversational models like ChatGPT generate good counterfactuals in a zero-shot setting, they are not efficient in low-resource environments. In this work, we focus on low-resource/resource-efficient techniques that can be deployed in low-resource environments.&#10;&#10;The primary contribution of this paper is to explore an automated approach to generate parallel training data for a given demographic axis that requires minimal human intervention. &#10;Our approach takes from a user a prompt -- in the form of a single word-pair -- that describes a demographic axis. This pair is subsequently used to model a demographic subspace from which other words that define the demographic attribute can be sampled from a given corpus of text. Using an invertible flow-based model \cite{dinh2014nice}, counterfactual words are generated for sampled words.&#10;Thereafter, an error correction approach is used in tandem with direct word substitution to generate parallel data to fine-tune a generative language model to generate counterfactual texts. We call our approach and the resultant counterfactual text generation model FairFlow.&#10;This entire process is simply depicted in a four-step process in Fig.~\ref{fig: architecture}.&#10;As opposed to existing works, which will be discussed in Section ~\ref{sec: related_lit}, FairFlow does not rely on human-generated parallel data for training and eliminates the need for manually compiled word-pair dictionaries. &#10;&#10;In summary, this paper explores and proposes techniques to develop a robust model-based counterfactual generator in the absence of parallel training data. Key contributions include:&#10;\begin{enumerate}&#10; \item An automated approach to compiling dictionaries of word pairs that only requires a user to input a word-pair prompt that describes a demographic axis. &#10; \item We proposed an error correction approach to generate parallel data from dictionary word substitutions.&#10; \item We train a counterfactual model using our generated parallel data and show that the error correction approach not only improves the grammatical composition of the model but also improves the generalization of the model.&#10;\end{enumerate}&#10;&#10;We make our implementation code and materials for FairFlow available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Counterfactual Data Augmentation, Linguistics, Language Model Bias, Artificial Intelligence, Social Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1812.06135" label="1812.06135">
        <attvalues>
          <attvalue for="0" value="Bias Mitigation Post-processing for Individual and Group Fairness" />
          <attvalue for="1" value="  Whereas previous post-processing approaches for increasing the fairness of&#10;predictions of biased classifiers address only group fairness, we propose a&#10;method for increasing both individual and group fairness. Our novel framework&#10;includes an individual bias detector used to prioritize data samples in a bias&#10;mitigation algorithm aiming to improve the group fairness measure of disparate&#10;impact. We show superior performance to previous work in the combination of&#10;classification accuracy, individual fairness and group fairness on several&#10;real-world datasets in applications such as credit, employment, and criminal&#10;justice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.11714" label="1807.11714">
        <attvalues>
          <attvalue for="0" value="Gender Bias in Neural Natural Language Processing" />
          <attvalue for="1" value="  We examine whether neural natural language processing (NLP) systems reflect&#10;historical biases in training data. We define a general benchmark to quantify&#10;gender bias in a variety of neural NLP tasks. Our empirical evaluation with&#10;state-of-the-art neural coreference resolution and textbook RNN-based language&#10;models trained on benchmark datasets finds significant gender bias in how&#10;models view occupations. We then mitigate bias with CDA: a generic methodology&#10;for corpus augmentation via causal interventions that breaks associations&#10;between gendered and gender-neutral words. We empirically show that CDA&#10;effectively decreases gender bias while preserving accuracy. We also explore&#10;the space of mitigation strategies with CDA, a prior approach to word embedding&#10;debiasing (WED), and their compositions. We show that CDA outperforms WED,&#10;drastically so when word embeddings are trained. For pre-trained embeddings,&#10;the two methods can be effectively composed. We also find that as training&#10;proceeds on the original data set with gradient descent the gender bias grows&#10;as the loss reduces, indicating that the optimization encourages bias; CDA&#10;mitigates this behavior.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1604.02201" label="1604.02201">
        <attvalues>
          <attvalue for="0" value="Transfer Learning for Low-Resource Neural Machine Translation" />
          <attvalue for="1" value="  The encoder-decoder framework for neural machine translation (NMT) has been&#10;shown effective in large data scenarios, but is much less effective for&#10;low-resource languages. We present a transfer learning method that&#10;significantly improves Bleu scores across a range of low-resource languages.&#10;Our key idea is to first train a high-resource language pair (the parent&#10;model), then transfer some of the learned parameters to the low-resource pair&#10;(the child model) to initialize and constrain training. Using our transfer&#10;learning method we improve baseline NMT models by an average of 5.6 Bleu on&#10;four low-resource language pairs. Ensembling and unknown word replacement add&#10;another 2 Bleu which brings the NMT performance on low-resource machine&#10;translation close to a strong syntax based machine translation (SBMT) system,&#10;exceeding its performance on one language pair. Additionally, using the&#10;transfer learning model for re-scoring, we can improve the SBMT system by an&#10;average of 1.3 Bleu, improving the state-of-the-art on low-resource machine&#10;translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1410.8516" label="1410.8516">
        <attvalues>
          <attvalue for="0" value="NICE: Non-linear Independent Components Estimation" />
          <attvalue for="1" value="  We propose a deep learning framework for modeling complex high-dimensional&#10;densities called Non-linear Independent Component Estimation (NICE). It is&#10;based on the idea that a good representation is one in which the data has a&#10;distribution that is easy to model. For this purpose, a non-linear&#10;deterministic transformation of the data is learned that maps it to a latent&#10;space so as to make the transformed data conform to a factorized distribution,&#10;i.e., resulting in independent latent variables. We parametrize this&#10;transformation so that computing the Jacobian determinant and inverse transform&#10;is trivial, yet we maintain the ability to learn complex non-linear&#10;transformations, via a composition of simple building blocks, each based on a&#10;deep neural network. The training criterion is simply the exact log-likelihood,&#10;which is tractable. Unbiased ancestral sampling is also easy. We show that this&#10;approach yields good generative models on four image datasets and can be used&#10;for inpainting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.04935" label="2206.04935">
        <attvalues>
          <attvalue for="0" value="Sort by Structure: Language Model Ranking as Dependency Probing" />
          <attvalue for="1" value="  Making an informed choice of pre-trained language model (LM) is critical for&#10;performance, yet environmentally costly, and as such widely underexplored. The&#10;field of Computer Vision has begun to tackle encoder ranking, with promising&#10;forays into Natural Language Processing, however they lack coverage of&#10;linguistic tasks such as structured prediction. We propose probing to rank LMs,&#10;specifically for parsing dependencies in a given language, by measuring the&#10;degree to which labeled trees are recoverable from an LM's contextualized&#10;embeddings. Across 46 typologically and architecturally diverse LM-language&#10;pairs, our probing approach predicts the best LM choice 79% of the time using&#10;orders of magnitude less compute than training a full parser. Within this&#10;study, we identify and analyze one recently proposed decoupled LM - RemBERT -&#10;and find it strikingly contains less inherent dependency information, but often&#10;yields the best parser after full fine-tuning. Without this outlier our&#10;approach identifies the best LM in 89% of cases.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the advent of massively pre-trained language models (LMs) in Natural Language Processing (NLP), it has become crucial for practitioners to choose the best LM encoder for their given task early on, regardless of the rest of their proposed model architecture. The greatest variation of LMs lies in the language or domain-specificity of the unlabelled data used during pre-training (with architectures often staying identical).&#10;&#10;Typically, better expressivity is expected from language/domain-specific LMs \cite{gururangan-etal-2020-dont,dai-etal-2020-cost} while open-domain settings necessitate high-capacity models with access to as much pre-training data as possible. This tradeoff is difficult to navigate, and given that multiple specialized LMs (or none at all) are available, practitioners often resort to an ad-hoc choice. In absence of immediate performance indicators, the most accurate choice could be made by training the full model using each LM candidate, however this is often infeasible and wasteful~\cite{strubell-etal-2019-energy}.&#10;&#10;Recently, the field of Computer Vision (CV) has attempted to tackle this problem by quantifying useful information in pre-trained image encoders as measured directly on labeled target data without fine-tuning \cite{nguyen2020,you2021}. While first forays for applying these methods to NLP are promising, some linguistic tasks differ substantially: Structured prediction, such as parsing syntactic dependencies, is a fundamental NLP task not covered by prior encoder ranking methods due to its graphical output. Simultaneously, performance prediction in NLP has so far been studied as a function of dataset and model characteristics~\cite{xia-etal-2020-predicting,ye-etal-2021-towards} and has yet to examine how to rank large pools of pre-trained LMs.&#10;&#10;Given the closely related field of probing, in which lightweight models quantify task-specific information in pre-trained LMs, we recast its objective in the context of performance prediction and ask: How predictive is lightweight probing at choosing the best performing LM for dependency parsing? To answer this question, we contribute:&#10;&#10;\begin{itemize}&#10; \item An efficient encoder ranking method for structured prediction using dependency probing (\cite{depprobe}; DepProbe) to quantify latent syntax (Section \ref{sec:method}).&#10; \item Experiments across 46 typologically and architecturally diverse LM + target language combinations (Section \ref{sec:experiments}).&#10; \item An in-depth analysis of the surprisingly low inherent dependency information in RemBERT ~\cite{chung2021rethinking} compared to its high fine-tuned performance (Section \ref{sec:rembert-analysis}).&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Computer Vision, Computational Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.01150" label="2010.01150">
        <attvalues>
          <attvalue for="0" value="Cost-effective Selection of Pretraining Data: A Case Study of&#10;  Pretraining BERT on Social Media" />
          <attvalue for="1" value="  Recent studies on domain-specific BERT models show that effectiveness on&#10;downstream tasks can be improved when models are pretrained on in-domain data.&#10;Often, the pretraining data used in these models are selected based on their&#10;subject matter, e.g., biology or computer science. Given the range of&#10;applications using social media text, and its unique language variety, we&#10;pretrain two models on tweets and forum text respectively, and empirically&#10;demonstrate the effectiveness of these two resources. In addition, we&#10;investigate how similarity measures can be used to nominate in-domain&#10;pretraining data. We publicly release our pretrained models at&#10;https://bit.ly/35RpTf0.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.12462" label="2002.12462">
        <attvalues>
          <attvalue for="0" value="LEEP: A New Measure to Evaluate Transferability of Learned&#10;  Representations" />
          <attvalue for="1" value="  We introduce a new measure to evaluate the transferability of representations&#10;learned by classifiers. Our measure, the Log Expected Empirical Prediction&#10;(LEEP), is simple and easy to compute: when given a classifier trained on a&#10;source data set, it only requires running the target data set through this&#10;classifier once. We analyze the properties of LEEP theoretically and&#10;demonstrate its effectiveness empirically. Our analysis shows that LEEP can&#10;predict the performance and convergence speed of both transfer and&#10;meta-transfer learning methods, even for small or imbalanced data. Moreover,&#10;LEEP outperforms recently proposed transferability measures such as negative&#10;conditional entropy and H scores. Notably, when transferring from ImageNet to&#10;CIFAR100, LEEP can achieve up to 30% improvement compared to the best competing&#10;method in terms of the correlations with actual transfer accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.11005" label="2102.11005">
        <attvalues>
          <attvalue for="0" value="LogME: Practical Assessment of Pre-trained Models for Transfer Learning" />
          <attvalue for="1" value="  This paper studies task adaptive pre-trained model selection, an&#10;underexplored problem of assessing pre-trained models for the target task and&#10;select best ones from the model zoo \emph{without fine-tuning}. A few pilot&#10;works addressed the problem in transferring supervised pre-trained models to&#10;classification tasks, but they cannot handle emerging unsupervised pre-trained&#10;models or regression tasks. In pursuit of a practical assessment method, we&#10;propose to estimate the maximum value of label evidence given features&#10;extracted by pre-trained models. Unlike the maximum likelihood, the maximum&#10;evidence is \emph{immune to over-fitting}, while its expensive computation can&#10;be dramatically reduced by our carefully designed algorithm. The Logarithm of&#10;Maximum Evidence (LogME) can be used to assess pre-trained models for transfer&#10;learning: a pre-trained model with a high LogME value is likely to have good&#10;transfer performance. LogME is \emph{fast, accurate, and general},&#10;characterizing itself as the first practical method for assessing pre-trained&#10;models. Compared with brute-force fine-tuning, LogME brings at most&#10;$3000\times$ speedup in wall-clock time and requires only $1\%$ memory&#10;footprint. It outperforms prior methods by a large margin in their setting and&#10;is applicable to new settings. It is general enough for diverse pre-trained&#10;models (supervised pre-trained and unsupervised pre-trained), downstream tasks&#10;(classification and regression), and modalities (vision and language). Code is&#10;available at this repository:&#10;\href{https://github.com/thuml/LogME}{https://github.com/thuml/LogME}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.12971" label="2203.12971">
        <attvalues>
          <attvalue for="0" value="Probing for Labeled Dependency Trees" />
          <attvalue for="1" value="  Probing has become an important tool for analyzing representations in Natural&#10;Language Processing (NLP). For graphical NLP tasks such as dependency parsing,&#10;linear probes are currently limited to extracting undirected or unlabeled parse&#10;trees which do not capture the full task. This work introduces DepProbe, a&#10;linear probe which can extract labeled and directed dependency parse trees from&#10;embeddings while using fewer parameters and compute than prior methods.&#10;Leveraging its full task coverage and lightweight parametrization, we&#10;investigate its predictive power for selecting the best transfer language for&#10;training a full biaffine attention parser. Across 13 languages, our proposed&#10;method identifies the best source treebank 94% of the time, outperforming&#10;competitive baselines and prior work. Finally, we analyze the informativeness&#10;of task-specific subspaces in contextual embeddings as well as which benefits a&#10;full parser's non-linear parametrization provides.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.04759" label="2404.04759">
        <attvalues>
          <attvalue for="0" value="What Happens When Small Is Made Smaller? Exploring the Impact of&#10;  Compression on Small Data Pretrained Language Models" />
          <attvalue for="1" value="  Compression techniques have been crucial in advancing machine learning by&#10;enabling efficient training and deployment of large-scale language models.&#10;However, these techniques have received limited attention in the context of&#10;low-resource language models, which are trained on even smaller amounts of data&#10;and under computational constraints, a scenario known as the &quot;low-resource&#10;double-bind.&quot; This paper investigates the effectiveness of pruning, knowledge&#10;distillation, and quantization on an exclusively low-resourced, small-data&#10;language model, AfriBERTa. Through a battery of experiments, we assess the&#10;effects of compression on performance across several metrics beyond accuracy.&#10;Our study provides evidence that compression techniques significantly improve&#10;the efficiency and effectiveness of small-data language models, confirming that&#10;the prevailing beliefs regarding the effects of compression on large, heavily&#10;parameterized models hold true for less-parameterized, small-data models.&#10;" />
          <attvalue for="2" value="&#10;One of the most challenging aspects of working with large language models (LLMs) is their computational complexity \cite{zhang_when_2021}. With 340M parameters, even the BERT-large model is impractical for deployment on low-end devices with inadequate computational power \cite{treviso2022efficient}. Several architectural changes to make the BERT model more efficient have been made \cite{Jiao2019TinyBERTDB, Sanh2019DistilBERTAD, lan_albert_2023}. However, to achieve adequate performance on downstream tasks, LLMs require huge training corpora (billions of tokens), which are unavailable for most African languages \cite{nekoto2020participatory}. The omission of African languages from the pre-training phase of LLMs results in low performance in these languages, making NLP tasks participatory difficult \cite{kreutzer_quality_2022}. \cite{ahia2021low} termed this situation the &quot;low-resource double-bind&quot; to describe the coexistence of data and computation limitations on resources. This is a popular NLP setting for low-resource languages, although the performance trade-offs are understudied.&#10;&#10;One of the most promising attempts to mitigate the sparse presence of low-resource African languages in model training is the creation of AfriBERTa, the first multilingual language model trained purely and from scratch on African languages with $&lt;1$GB of data. AfriBERTa \cite{ogueji-etal-2021-small} beats competitive models like mBERT \cite{devlin_bert_2019} and XLM-R \cite{conneau2020unsupervised} on text categorization and NER tasks. Instead of depending on high-resource languages for transfer learning, AfriBERTa takes advantage of linguistic similarities between languages from low-resource environments to yield promising results, which is critical in determining the sustainability of language models trained on small datasets. However, with 126M parameters, the AfriBERTa-large model is still impractical for deployment on low-end devices with inadequate computational power. Moreover, little is known about the ability of a small-data, low-resource-focused model like AfriBERTa to generalize to unseen-before languages, considering its small size, and its ability to get &quot;smaller&quot; for efficient usage by resource-constrained users.&#10;&#10;This research attempts to bridge the gap between a low-resource, small-data, high-performance multilingual language model and an ultra-efficient, deployable model for double-bind users. &#10;Our experimental results demonstrate that pruning achieves $\approx60\%$ reduction in model size with a minimal performance drop. Furthermore, generalization tests reveal varied outcomes, with some languages surpassing dense models even with extreme pruning. Distillation achieves compression rates between 22\% and 33\% with comparable performances.&#10;Additionally, quantization reduces the model size by 64.08\%, inference time by 52.3\%, and even outperforms the baseline model in the F1 score for certain languages. Our contributions address the following questions:&#10;\begin{enumerate}&#10; \item How tiny can we construct a small-data model using the knowledge distillation framework?&#10; \item What are the efficiency and generalization limits of pruning on a small-data model?&#10; \item What are the optimal reductions we can achieve in size and latency utilizing quantization?&#10;&#10;\end{enumerate}&#10;&#10;Related work&#10;First introduced by \cite{Hinton2015DistillingTK}, \cite{Sanh2019DistilBERTAD} demonstrated similar performances on downstream tasks with smaller language models pre-trained using distillation, which is faster at inference and suitable for edge devices. \cite{Jiao2019TinyBERTDB} proposed a transformer-specific distillation method, employing a two-stage learning framework with general and task-specific distillation using BERT.&#10;\cite{Han2015DeepCC} reintroduced modern pruning as &quot;network pruning.&quot; and the spotlight intensified with \cite{franklelottery} suggesting the existence of subnetworks within a dense neural network that matches or surpass the dense model's performance—termed &quot;winning tickets.&quot; \cite{yu2019playing} and \cite{renda2020comparing} also found winning tickets early in training for Transformers and LSTMs.&#10;\cite{chen2020lottery} and \cite{prasanna2020bert} also explored trainable subnetworks in pre-trained BERT models, locating matching subnetworks at 40\% to 90\% sparsity across various applications.&#10;\cite{li2020train} demonstrated that heavily compressing large models resulted in higher accuracy than lightly compressing small models.&#10;\cite{bai2022towards} also introduced post-training quantization for language models, minimizing training time, memory, and data consumption, while \cite{wang2022deep} achieved 16$\times$ compression by quantizing transformer backbones to 4-bit and applying 50\% fine-grained structural sparsity.&#10;Additionally, \cite{xiao2022smoothquant} enabled 8-bit weight, 8-bit activation quantization for large language models, addressing activation outliers, and \cite{dettmers2022llm} used LLM.int8() on transformers with 16 or 32-bit weights for immediate inference using vector-wise quantization and mixed-precision decomposition. As far as we know, our work is the first to explore these techniques' efficacy in a low-resource double-bind setting.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Efficient Training Techniques, Computer Science, Information Technology, Linguistics, Language Model Compression, Mathematics, Low-Resource Machine Learning" />
        </attvalues>
      </node>
      <node id="2209.00099" label="2209.00099">
        <attvalues>
          <attvalue for="0" value="Efficient Methods for Natural Language Processing: A Survey" />
          <attvalue for="1" value="  Recent work in natural language processing (NLP) has yielded appealing&#10;results from scaling model parameters and training data; however, using only&#10;scale to improve performance means that resource consumption also grows. Such&#10;resources include data, time, storage, or energy, all of which are naturally&#10;limited and unevenly distributed. This motivates research into efficient&#10;methods that require fewer resources to achieve similar results. This survey&#10;synthesizes and relates current methods and findings in efficient NLP. We aim&#10;to provide both guidance for conducting NLP under limited resources, and point&#10;towards promising research directions for developing more efficient methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.12028" label="2103.12028">
        <attvalues>
          <attvalue for="0" value="Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets" />
          <attvalue for="1" value="  With the success of large-scale pre-training and multilingual modeling in&#10;Natural Language Processing (NLP), recent years have seen a proliferation of&#10;large, web-mined text datasets covering hundreds of languages. We manually&#10;audit the quality of 205 language-specific corpora released with five major&#10;public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource&#10;corpora have systematic issues: At least 15 corpora have no usable text, and a&#10;significant fraction contains less than 50% sentences of acceptable quality. In&#10;addition, many are mislabeled or use nonstandard/ambiguous language codes. We&#10;demonstrate that these issues are easy to detect even for non-proficient&#10;speakers, and supplement the human audit with automatic analyses. Finally, we&#10;recommend techniques to evaluate and improve multilingual corpora and discuss&#10;potential risks that come with low-quality data releases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.03036" label="2110.03036">
        <attvalues>
          <attvalue for="0" value="The Low-Resource Double Bind: An Empirical Study of Pruning for&#10;  Low-Resource Machine Translation" />
          <attvalue for="1" value="  A &quot;bigger is better&quot; explosion in the number of parameters in deep neural&#10;networks has made it increasingly challenging to make state-of-the-art networks&#10;accessible in compute-restricted environments. Compression techniques have&#10;taken on renewed importance as a way to bridge the gap. However, evaluation of&#10;the trade-offs incurred by popular compression techniques has been centered on&#10;high-resource datasets. In this work, we instead consider the impact of&#10;compression in a data-limited regime. We introduce the term low-resource double&#10;bind to refer to the co-occurrence of data limitations and compute resource&#10;constraints. This is a common setting for NLP for low-resource languages, yet&#10;the trade-offs in performance are poorly studied. Our work offers surprising&#10;insights into the relationship between capacity and generalization in&#10;data-limited regimes for the task of machine translation. Our experiments on&#10;magnitude pruning for translations from English into Yoruba, Hausa, Igbo and&#10;German show that in low-resource regimes, sparsity preserves performance on&#10;frequent sentences but has a disparate impact on infrequent ones. However, it&#10;improves robustness to out-of-distribution shifts, especially for datasets that&#10;are very distinct from the training distribution. Our findings suggest that&#10;sparsity can play a beneficial role at curbing memorization of low frequency&#10;attributes, and therefore offers a promising solution to the low-resource&#10;double bind.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.02768" label="1906.02768">
        <attvalues>
          <attvalue for="0" value="Playing the lottery with rewards and multiple languages: lottery tickets&#10;  in RL and NLP" />
          <attvalue for="1" value="  The lottery ticket hypothesis proposes that over-parameterization of deep&#10;neural networks (DNNs) aids training by increasing the probability of a &quot;lucky&quot;&#10;sub-network initialization being present rather than by helping the&#10;optimization process (Frankle &amp; Carbin, 2019). Intriguingly, this phenomenon&#10;suggests that initialization strategies for DNNs can be improved substantially,&#10;but the lottery ticket hypothesis has only previously been tested in the&#10;context of supervised learning for natural image tasks. Here, we evaluate&#10;whether &quot;winning ticket&quot; initializations exist in two different domains:&#10;natural language processing (NLP) and reinforcement learning (RL).For NLP, we&#10;examined both recurrent LSTM models and large-scale Transformer models (Vaswani&#10;et al., 2017). For RL, we analyzed a number of discrete-action space tasks,&#10;including both classic control and pixel control. Consistent with workin&#10;supervised image classification, we confirm that winning ticket initializations&#10;generally outperform parameter-matched random initializations, even at extreme&#10;pruning rates for both NLP and RL. Notably, we are able to find winning ticket&#10;initializations for Transformers which enable models one-third the size to&#10;achieve nearly equivalent performance. Together, these results suggest that the&#10;lottery ticket hypothesis is not restricted to supervised learning of natural&#10;images, but rather represents a broader phenomenon in DNNs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.02389" label="2003.02389">
        <attvalues>
          <attvalue for="0" value="Comparing Rewinding and Fine-tuning in Neural Network Pruning" />
          <attvalue for="1" value="  Many neural network pruning algorithms proceed in three steps: train the&#10;network to completion, remove unwanted structure to compress the network, and&#10;retrain the remaining structure to recover lost accuracy. The standard&#10;retraining technique, fine-tuning, trains the unpruned weights from their final&#10;trained values using a small fixed learning rate. In this paper, we compare&#10;fine-tuning to alternative retraining techniques. Weight rewinding (as proposed&#10;by Frankle et al., (2019)), rewinds unpruned weights to their values from&#10;earlier in training and retrains them from there using the original training&#10;schedule. Learning rate rewinding (which we propose) trains the unpruned&#10;weights from their final values using the same learning rate schedule as weight&#10;rewinding. Both rewinding techniques outperform fine-tuning, forming the basis&#10;of a network-agnostic pruning algorithm that matches the accuracy and&#10;compression ratios of several more network-specific state-of-the-art&#10;techniques.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.15082" label="2109.15082">
        <attvalues>
          <attvalue for="0" value="Towards Efficient Post-training Quantization of Pre-trained Language&#10;  Models" />
          <attvalue for="1" value="  Network quantization has gained increasing attention with the rapid growth of&#10;large pre-trained language models~(PLMs). However, most existing quantization&#10;methods for PLMs follow quantization-aware training~(QAT) that requires&#10;end-to-end training with full access to the entire dataset. Therefore, they&#10;suffer from slow training, large memory overhead, and data security issues. In&#10;this paper, we study post-training quantization~(PTQ) of PLMs, and propose&#10;module-wise quantization error minimization~(MREM), an efficient solution to&#10;mitigate these issues. By partitioning the PLM into multiple modules, we&#10;minimize the reconstruction error incurred by quantization for each module. In&#10;addition, we design a new model parallel training strategy such that each&#10;module can be trained locally on separate computing devices without waiting for&#10;preceding modules, which brings nearly the theoretical training speed-up (e.g.,&#10;$4\times$ on $4$ GPUs). Experiments on GLUE and SQuAD benchmarks show that our&#10;proposed PTQ solution not only performs close to QAT, but also enjoys&#10;significant reductions in training time, memory overhead, and data consumption.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.11041" label="1710.11041">
        <attvalues>
          <attvalue for="0" value="Unsupervised Neural Machine Translation" />
          <attvalue for="1" value="  In spite of the recent success of neural machine translation (NMT) in&#10;standard benchmarks, the lack of large parallel corpora poses a major practical&#10;problem for many language pairs. There have been several proposals to alleviate&#10;this issue with, for instance, triangulation and semi-supervised learning&#10;techniques, but they still require a strong cross-lingual signal. In this work,&#10;we completely remove the need of parallel data and propose a novel method to&#10;train an NMT system in a completely unsupervised manner, relying on nothing but&#10;monolingual corpora. Our model builds upon the recent work on unsupervised&#10;embedding mappings, and consists of a slightly modified attentional&#10;encoder-decoder model that can be trained on monolingual corpora alone using a&#10;combination of denoising and backtranslation. Despite the simplicity of the&#10;approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014&#10;French-to-English and German-to-English translation. The model can also profit&#10;from small parallel corpora, and attains 21.81 and 15.24 points when combined&#10;with 100,000 parallel sentences, respectively. Our implementation is released&#10;as an open source project.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.14277" label="2011.14277">
        <attvalues>
          <attvalue for="0" value="Intrinsic Knowledge Evaluation on Chinese Language Models" />
          <attvalue for="1" value="  Recent NLP tasks have benefited a lot from pre-trained language models (LM)&#10;since they are able to encode knowledge of various aspects. However, current LM&#10;evaluations focus on downstream performance, hence lack to comprehensively&#10;inspect in which aspect and to what extent have they encoded knowledge. This&#10;paper addresses both queries by proposing four tasks on syntactic, semantic,&#10;commonsense, and factual knowledge, aggregating to a total of $39,308$&#10;questions covering both linguistic and world knowledge in Chinese. Throughout&#10;experiments, our probes and knowledge data prove to be a reliable benchmark for&#10;evaluating pre-trained Chinese LMs. Our work is publicly available at&#10;https://github.com/ZhiruoWang/ChnEval.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years witnessed much success achieved by pre-trained LMs in the field of Natural Language Processing \cite{peters2018deep,devlin2019bert}. The performance of these models is often evaluated on downstream tasks like reading comprehension (RC), natural language inference (NLI), and sentiment analysis (SA). &#10;However, improvements on downstream hardly explain the reasons behind models' excellence, as well as what they learn during pre-training. Therefore, an emerging body of work starts to investigate the knowledge encoded in their contextual representations.&#10;&#10;Linguistic probing methods are designed to uncover the intriguing properties stored in the contextual representations. Among the linguistic knowledge, syntax is broadly explored across sensitive structures \cite{goldberg2019assessing}, grammatical correctness \cite{marvin2018targeted}, and parsing dependencies \cite{hewitt2019structural}. However, existing language probes face three challenges: (1) A skewing on syntax, for few semantic tasks ever study the contextual representations; (2) Most probes are built as classifiers that require extra training. It raises the question `Do the representations encode linguistic structure, or just that the probe has learned the linguistic task' \cite{hewitt2019designing}? and (3) Existing probing tasks scope to only an English language setting.&#10;&#10;In addition to the linguistics, tasks on common sense and facts are also introduced to test models on memorizing real-world knowledge during pre-training \cite{bisk2019piqa,zhou2019evaluating,petroni2019language}. Nonetheless, the knowledge encoding ability of BERT is controversial \cite{poerner2019bert}, and the template-based cloze questions are often too short to be leveraged by models for informative contextualizations.&#10;&#10;Inspired by the above works, this paper proposes the first intrinsic knowledge evaluation benchmark of Chinese pre-trained LMs. &#10;Linguistically, it covers both the syntactic and semantic knowledge. &#10;One task aims at the language-specific syntactic features of Chinese, and another on language-independent semantic features.&#10;Meanwhile, we inspect world knowledge from two tasks on common sense and facts, further enable questions with natural contexts. All of the four tasks are designed to fit the LM structures and capabilities, i.e. making predictions directly from deep contextualized embeddings without additional tuning.&#10;&#10;In the experiments, we test not only off-the-shelf models from CLUE project \cite{xu2020clue}, but also four BERT variants granted with different training objectives that mimic BERT, RoBERTa, SpanBERT, and ALBERT. Our tasks and data sets prove to constitute a reliable evaluation benchmark. It effectively illustrates the advantages and disadvantages of different LMs over various aspects of knowledge.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Knowledge Encoding Analysis, Linguistics, Language Model Evaluation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2004.05986" label="2004.05986">
        <attvalues>
          <attvalue for="0" value="CLUE: A Chinese Language Understanding Evaluation Benchmark" />
          <attvalue for="1" value="  The advent of natural language understanding (NLU) benchmarks for English,&#10;such as GLUE and SuperGLUE allows new NLU models to be evaluated across a&#10;diverse set of tasks. These comprehensive benchmarks have facilitated a broad&#10;range of research and applications in natural language processing (NLP). The&#10;problem, however, is that most such benchmarks are limited to English, which&#10;has made it difficult to replicate many of the successes in English NLU for&#10;other languages. To help remedy this issue, we introduce the first large-scale&#10;Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an&#10;open-ended, community-driven project that brings together 9 tasks spanning&#10;several well-established single-sentence/sentence-pair classification tasks, as&#10;well as machine reading comprehension, all on original Chinese text. To&#10;establish results on these tasks, we report scores using an exhaustive set of&#10;current state-of-the-art pre-trained Chinese models (9 in total). We also&#10;introduce a number of supplementary datasets and additional tools to help&#10;facilitate further progress on Chinese NLU. Our benchmark is released at&#10;https://www.CLUEbenchmarks.com&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.02693" label="2005.02693">
        <attvalues>
          <attvalue for="0" value="Shape of synth to come: Why we should use synthetic data for English&#10;  surface realization" />
          <attvalue for="1" value="  The Surface Realization Shared Tasks of 2018 and 2019 were Natural Language&#10;Generation shared tasks with the goal of exploring approaches to surface&#10;realization from Universal-Dependency-like trees to surface strings for several&#10;languages. In the 2018 shared task there was very little difference in the&#10;absolute performance of systems trained with and without additional,&#10;synthetically created data, and a new rule prohibiting the use of synthetic&#10;data was introduced for the 2019 shared task. Contrary to the findings of the&#10;2018 shared task, we show, in experiments on the English 2018 dataset, that the&#10;use of synthetic data can have a substantial positive effect - an improvement&#10;of almost 8 BLEU points for a previously state-of-the-art system. We analyse&#10;the effects of synthetic data, and we argue that its use should be encouraged&#10;rather than prohibited so that future research efforts continue to explore&#10;systems that can take advantage of such data.&#10;" />
          <attvalue for="2" value="&#10;&#10;The shallow task of the recent surface realization (SR) shared tasks \cite{first_surface_realization_shared_task, mulit_surface_realization_2018, second_surface_realization_2019} appears to be a relatively straightforward problem. &#10;Given a tree of lemmas, a system has to restore the original word order of the sentence and inflect its lemmas, see Figure \ref{fig:surface_realization_example}. &#10;Yet SR systems often struggle, even for a relatively fixed word order language such as English.&#10;Improved performance would facilitate investigation of more complex versions of the shallow task, such as the deep task in which function words are pruned from the tree, which may be of more practical use in pipeline natural language generation (NLG) systems \cite{moryossef-etal-2019-step, elder-etal-2019-designing, castro-ferreira-etal-2019-neural}. &#10;&#10;In this paper we explore the use of synthetic data for the English shallow task.&#10;Synthetic data is created by taking an unlabelled sentence, parsing it with&#10;an open source universal dependency parser and transforming the result into the input representation.&#10;&#10;Unlike in the 2018 shared task, where a system trained with synthetic data performed roughly the same as a system trained on the original dataset \cite{elder_srst_2018, king_white_srst_2018}, we find its use leads to a large improvement in performance.&#10;The state-of-the-art on the dataset is 72.7 BLEU-4 score \cite{yu-etal-2019-head} -- our system achieves a similar result of 72.3, which improves to 80.1 with the use of synthetic data.&#10;We analyse the ways in which synthetic data helps to improve performance, finding that longer sentences are particularly improved and more exactly correct linearizations are generated overall.&#10;&#10;Although it is common knowledge that machine learning systems typically benefit from more data, this 7.4 point jump in BLEU is important and worth emphasizing. The 2019 shared task introduced a new rule which prohibited the use of synthetic data. This was done in order to make the results of different systems more comparable.&#10;However, systems designed with smaller datasets in mind might not scale to the use of synthetic data, and an inadvertent consequence of such a rule is that it may produce results which could be misleading for future research directions.&#10;&#10;For instance, the system which was the clear winner of this year's shared task \cite{ims_srst_2019} used tree-structured long short-term memory (LSTM) networks \cite{tai_tree_lstm_2015}.&#10;In general, tree LSTMs can be slow and difficult to train.&#10;\cite{graphAMR_ACL2018} utilized a variant of the tree LSTM in a similar NLG task, converting abstract meaning representation (AMR) graphs to text.&#10;Following the state-of-the-art system \cite{neural_amr_konstas_ACL_2017}, which used standard LSTMs, \cite{graphAMR_ACL2018} augmented their training with synthetic data.&#10;Though their system outperformed \cite{neural_amr_konstas_ACL_2017} at equivalent levels of additional training sentences, it was unable to scale up to the 20 million sentences used by the best \cite{neural_amr_konstas_ACL_2017} system and ultimately did not outperform them.&#10;&#10;Critics of neural NLG approaches emphasise that quality and reliability are at the core of production-ready NLG systems.&#10;What we are essentially arguing is that if using synthetic data contributes to producing higher quality outputs, then we ought to ensure we are designing systems that can take advantage of synthetic data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Synthetic Data Effects, Natural Language Generation, Surface Realization Tasks" />
        </attvalues>
      </node>
      <node id="1904.03396" label="1904.03396">
        <attvalues>
          <attvalue for="0" value="Step-by-Step: Separating Planning from Realization in Neural&#10;  Data-to-Text Generation" />
          <attvalue for="1" value="  Data-to-text generation can be conceptually divided into two parts: ordering&#10;and structuring the information (planning), and generating fluent language&#10;describing the information (realization). Modern neural generation systems&#10;conflate these two steps into a single end-to-end differentiable system. We&#10;propose to split the generation process into a symbolic text-planning stage&#10;that is faithful to the input, followed by a neural generation stage that&#10;focuses only on realization. For training a plan-to-text generator, we present&#10;a method for matching reference texts to their corresponding text plans. For&#10;inference time, we describe a method for selecting high-quality text plans for&#10;new inputs. We implement and evaluate our approach on the WebNLG benchmark. Our&#10;results demonstrate that decoupling text planning from neural realization&#10;indeed improves the system's reliability and adequacy while maintaining fluent&#10;output. We observe improvements both in BLEU scores and in manual evaluations.&#10;Another benefit of our approach is the ability to output diverse realizations&#10;of the same input, paving the way to explicit control over the generated text&#10;structure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.07731" label="1805.07731">
        <attvalues>
          <attvalue for="0" value="Generating High-Quality Surface Realizations Using Data Augmentation and&#10;  Factored Sequence Models" />
          <attvalue for="1" value="  This work presents a new state of the art in reconstruction of surface&#10;realizations from obfuscated text. We identify the lack of sufficient training&#10;data as the major obstacle to training high-performing models, and solve this&#10;issue by generating large amounts of synthetic training data. We also propose&#10;preprocessing techniques which make the structure contained in the input&#10;features more accessible to sequence models. Our models were ranked first on&#10;all evaluation metrics in the English portion of the 2018 Surface Realization&#10;shared task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.18653" label="2405.18653">
        <attvalues>
          <attvalue for="0" value="Recent Advances of Foundation Language Models-based Continual Learning:&#10;  A Survey" />
          <attvalue for="1" value="  Recently, foundation language models (LMs) have marked significant&#10;achievements in the domains of natural language processing (NLP) and computer&#10;vision (CV). Unlike traditional neural network models, foundation LMs obtain a&#10;great ability for transfer learning by acquiring rich commonsense knowledge&#10;through pre-training on extensive unsupervised datasets with a vast number of&#10;parameters. However, they still can not emulate human-like continuous learning&#10;due to catastrophic forgetting. Consequently, various continual learning&#10;(CL)-based methodologies have been developed to refine LMs, enabling them to&#10;adapt to new tasks without forgetting previous knowledge. However, a systematic&#10;taxonomy of existing approaches and a comparison of their performance are still&#10;lacking, which is the gap that our survey aims to fill. We delve into a&#10;comprehensive review, summarization, and classification of the existing&#10;literature on CL-based approaches applied to foundation language models, such&#10;as pre-trained language models (PLMs), large language models (LLMs) and&#10;vision-language models (VLMs). We divide these studies into offline CL and&#10;online CL, which consist of traditional methods, parameter-efficient-based&#10;methods, instruction tuning-based methods and continual pre-training methods.&#10;Offline CL encompasses domain-incremental learning, task-incremental learning,&#10;and class-incremental learning, while online CL is subdivided into hard task&#10;boundary and blurry task boundary settings. Additionally, we outline the&#10;typical datasets and metrics employed in CL research and provide a detailed&#10;analysis of the challenges and future work for LMs-based continual learning.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancements in foundation language models (LMs) have set new benchmarks in both natural language processing (NLP) \cite{zhao2023survey,min2023recent,zhou2023chatgpt} and computer vision (CV) \cite{wang2023large, tiwari2022gcr, ramasesh2022effect}. Foundation LMs encompass three primary categories: Pre-trained Language Models (PLMs) \cite{min2023recent}, Large Language Models (LLMs) \cite{zhao2023survey}, and Vision-Language Models (VLMs) \cite{ijcai2022p762}. &#10;{These models are pre-trained on large, unlabeled datasets to capture rich semantic information, which is then fine-tuned for specific tasks or domains. This strategy not only enhances performance across various applications but also significantly improves the flexibility and adaptability of these models.} &#10;&#10;{However, despite their strengths, foundation LMs face challenges in dynamic environments where tasks evolve over time. A key issue is ``catastrophic forgetting&quot; \cite{kirkpatrick2017overcoming}, where models lose previously learned knowledge when adapting to new information. Unlike human learning, which is inherently continuous and adaptive \cite{de2021continual}, foundation LMs generally require retraining to incorporate new data. Effective learning in such environments demands not only the ability to accelerate learning on new tasks (forward transfer) but also to improve performance on previous tasks by integrating newly acquired knowledge (backward transfer). While multi-task learning (MTL) and transfer learning (TL) offer potential solutions, MTL requires all task data to be available upfront, and TL focuses on limited tasks, making both approaches impractical for dynamic, real-world applications.}&#10;&#10;Continual learning (CL) \cite{wang2023comprehensive,van2022three}, also known as lifelong learning \cite{parisi2019continual} or incremental learning \cite{zhou2023deep}, offers an effective solution to these challenges. {It aims to develop systems capable of continuously learning and updating without forgetting past knowledge.} Recent advancements in CL methodologies have substantially enhanced the adaptability and knowledge retention capabilities of foundation LMs \cite{mehta2021empirical,cossu2022continual,lee2023pre}. &#10;{Notable successes have been documented in diverse downstream tasks, such as aspect-based sentiment analysis \cite{ke2021classic}, dialogue generation \cite{scialom2022fine}, text classification \cite{razdaibiedina2023progressive}, visual question answering \cite{zhang2023vqacl,qian2023decouple} and so on}. {Luo et al. \cite{luo2023empirical} conduct an empirical study on catastrophic forgetting (CF) in large language models (LLMs) during continual instruction tuning.} &#10;The aforementioned works underscore the potential of continual learning to significantly boost the performance of foundation LMs.&#10;&#10;\tikzstyle{my-box}=[&#10; rectangle,&#10; draw=hidden-black,&#10; rounded corners,&#10; text opacity=1,&#10; minimum height=1.5em,&#10; minimum width=5em,&#10; inner sep=2pt,&#10; align=center,&#10; fill opacity=.5,&#10;]&#10;\tikzstyle{leaf}=[&#10; my-box, &#10; minimum height=1.5em,&#10; fill=hidden-blue!90, &#10; text=black,&#10; align=left,&#10; font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10;]&#10;&#10;In the domain of continual learning, there has been a shift from traditional methods to those incorporating foundation LMs (Figure \ref{fig:IntroCLDifference}). First, foundation LMs has specialized transfer capability to quickly adapt to downstream tasks with only a few samples. Consequently, it is crucial to mitigate the degradation of both the zero-shot transfer and history task abilities while facilitating the acquisition of new skills. Second, due to the substantial number of parameters in foundation LMs, it is crucial to employ parameter-efficient techniques \cite{han2024parameter}, such as prompt tuning \cite{liu2022p} and adapters \cite{mundra2024comprehensive}, to update parameters without comprehensive retraining. Third, the foundation LMs possess the capability to follow instructions through instructional learning \cite{dong2022survey,ouyang2022training}, enabling more dynamic and context-aware interactions.&#10;&#10;{This review systematically organizes continual learning strategies and technologies into two main categories: offline continual learning and online continual learning (Figure \ref{fig:taxonomy}). We begin by defining and explaining the different settings for these two types of continual learning. Offline continual learning includes domain/task/class-incremental CL, while online continual learning is further divided into methods that address hard task boundaries and those that manage blurry task boundaries {(Section \ref{sect: Settings and Learning Modes of CL})}. To further clarify the relationship between these strategies and model architectures, we group the methods based on three major model types: Pre-trained Language Models (PLMs), Large Language Models (LLMs), and Vision-Language Models (VLMs). These categories reflect the distinct requirements for continual learning in each architecture and emphasize the role these models play in shaping learning strategies. Additionally, we classify the methods into four key approaches: traditional continual learning methods, continual pre-training methods, parameter-efficient tuning methods, and instruction-based methods {(Section \ref{sect: Offline Continual Learning}, \ref{sect: Online Continual Learning})}. This categorization highlights the different techniques used to address challenges like catastrophic forgetting and to improve knowledge transfer in continual learning scenarios}. Finally, we static the main datasets from various perspectives {(Section\ref{sect: Datasets})} and review the key metrics to evaluate the forgetting and transferring of the models {(Section \ref{sect: Metrics})}.&#10;&#10;The main contributions of this survey paper can be summarized as follows:&#10;\begin{itemize}[leftmargin=*, align=left]&#10; \item We thoroughly review the existing literature on foundation LMs-based CL approaches, which integrate foundation LMs with CL to learn new knowledge without retraining the models. It is quite different from traditional CL since foundation LMs have great abilities of transfer learning, zero-shot and instruction following with huge parameters.&#10; \item We give the definitions of different settings and categorize these studies into various classes to better understand the development of this domain. In addition to the traditional methods like replay, regularization and parameter-isolation-based algorithms, we also summarize the works about continual pre-training methods, parameter-efficient tuning methods and instruction tuning-based methods. &#10; \item We provide the characters of existing datasets for CL and present the main metrics to evaluate the performance of preventing forgetting and knowledge transfer. Furthermore, we discuss the most challenging problems of foundation LMs-based CL and point out promising future research directions in this field.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Models, Continual Learning, Artificial Intelligence, Transfer Learning" />
        </attvalues>
      </node>
      <node id="2302.10035" label="2302.10035">
        <attvalues>
          <attvalue for="0" value="Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey" />
          <attvalue for="1" value="  With the urgent demand for generalized deep models, many pre-trained big&#10;models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of&#10;these models in single domains (like computer vision and natural language&#10;processing), the multi-modal pre-trained big models have also drawn more and&#10;more attention in recent years. In this work, we give a comprehensive survey of&#10;these models and hope this paper could provide new insights and helps fresh&#10;researchers to track the most cutting-edge works. Specifically, we firstly&#10;introduce the background of multi-modal pre-training by reviewing the&#10;conventional deep learning, pre-training works in natural language process,&#10;computer vision, and speech. Then, we introduce the task definition, key&#10;challenges, and advantages of multi-modal pre-training models (MM-PTMs), and&#10;discuss the MM-PTMs with a focus on data, objectives, network architectures,&#10;and knowledge enhanced pre-training. After that, we introduce the downstream&#10;tasks used for the validation of large-scale MM-PTMs, including generative,&#10;classification, and regression tasks. We also give visualization and analysis&#10;of the model parameters and results on representative downstream tasks.&#10;Finally, we point out possible research directions for this topic that may&#10;benefit future works. In addition, we maintain a continuously updated paper&#10;list for large-scale pre-trained multi-modal big models:&#10;https://github.com/wangxiao5791509/MultiModal_BigModels_Survey. This paper has&#10;been published by the journal Machine Intelligence Research (MIR),&#10;https://link.springer.com/article/10.1007/s11633-022-1410-8, DOI:&#10;10.1007/s11633-022-1410-8, vol. 20, no. 4, pp. 447-482, 2023.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.10936" label="2202.10936">
        <attvalues>
          <attvalue for="0" value="A Survey of Vision-Language Pre-Trained Models" />
          <attvalue for="1" value="  As transformer evolves, pre-trained models have advanced at a breakneck pace&#10;in recent years. They have dominated the mainstream techniques in natural&#10;language processing (NLP) and computer vision (CV). How to adapt pre-training&#10;to the field of Vision-and-Language (V-L) learning and improve downstream task&#10;performance becomes a focus of multimodal learning. In this paper, we review&#10;the recent progress in Vision-Language Pre-Trained Models (VL-PTMs). As the&#10;core content, we first briefly introduce several ways to encode raw images and&#10;texts to single-modal embeddings before pre-training. Then, we dive into the&#10;mainstream architectures of VL-PTMs in modeling the interaction between text&#10;and image representations. We further present widely-used pre-training tasks,&#10;and then we introduce some common downstream tasks. We finally conclude this&#10;paper and present some promising research directions. Our survey aims to&#10;provide researchers with synthesis and pointer to related research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.08383" label="1909.08383">
        <attvalues>
          <attvalue for="0" value="A continual learning survey: Defying forgetting in classification tasks" />
          <attvalue for="1" value="  Artificial neural networks thrive in solving the classification problem for a&#10;particular rigid task, acquiring knowledge through generalized learning&#10;behaviour from a distinct training phase. The resulting network resembles a&#10;static entity of knowledge, with endeavours to extend this knowledge without&#10;targeting the original task resulting in a catastrophic forgetting. Continual&#10;learning shifts this paradigm towards networks that can continually accumulate&#10;knowledge over different tasks without the need to retrain from scratch. We&#10;focus on task incremental classification, where tasks arrive sequentially and&#10;are delineated by clear boundaries. Our main contributions concern 1) a&#10;taxonomy and extensive overview of the state-of-the-art, 2) a novel framework&#10;to continually determine the stability-plasticity trade-off of the continual&#10;learner, 3) a comprehensive experimental comparison of 11 state-of-the-art&#10;continual learning methods and 4 baselines. We empirically scrutinize method&#10;strengths and weaknesses on three benchmarks, considering Tiny Imagenet and&#10;large-scale unbalanced iNaturalist and a sequence of recognition datasets. We&#10;study the influence of model capacity, weight decay and dropout regularization,&#10;and the order in which the tasks are presented, and qualitatively compare&#10;methods in terms of required memory, computation time, and storage.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.00487" label="2302.00487">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey of Continual Learning: Theory, Method and&#10;  Application" />
          <attvalue for="1" value="  To cope with real-world dynamics, an intelligent system needs to&#10;incrementally acquire, update, accumulate, and exploit knowledge throughout its&#10;lifetime. This ability, known as continual learning, provides a foundation for&#10;AI systems to develop themselves adaptively. In a general sense, continual&#10;learning is explicitly limited by catastrophic forgetting, where learning a new&#10;task usually results in a dramatic performance degradation of the old tasks.&#10;Beyond this, increasingly numerous advances have emerged in recent years that&#10;largely extend the understanding and application of continual learning. The&#10;growing and widespread interest in this direction demonstrates its realistic&#10;significance as well as complexity. In this work, we present a comprehensive&#10;survey of continual learning, seeking to bridge the basic settings, theoretical&#10;foundations, representative methods, and practical applications. Based on&#10;existing theoretical and empirical results, we summarize the general objectives&#10;of continual learning as ensuring a proper stability-plasticity trade-off and&#10;an adequate intra/inter-task generalizability in the context of resource&#10;efficiency. Then we provide a state-of-the-art and elaborated taxonomy,&#10;extensively analyzing how representative methods address continual learning,&#10;and how they are adapted to particular challenges in realistic applications.&#10;Through an in-depth discussion of promising directions, we believe that such a&#10;holistic perspective can greatly facilitate subsequent exploration in this&#10;field and beyond.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.07569" label="1802.07569">
        <attvalues>
          <attvalue for="0" value="Continual Lifelong Learning with Neural Networks: A Review" />
          <attvalue for="1" value="  Humans and animals have the ability to continually acquire, fine-tune, and&#10;transfer knowledge and skills throughout their lifespan. This ability, referred&#10;to as lifelong learning, is mediated by a rich set of neurocognitive mechanisms&#10;that together contribute to the development and specialization of our&#10;sensorimotor skills as well as to long-term memory consolidation and retrieval.&#10;Consequently, lifelong learning capabilities are crucial for autonomous agents&#10;interacting in the real world and processing continuous streams of information.&#10;However, lifelong learning remains a long-standing challenge for machine&#10;learning and neural network models since the continual acquisition of&#10;incrementally available information from non-stationary data distributions&#10;generally leads to catastrophic forgetting or interference. This limitation&#10;represents a major drawback for state-of-the-art deep neural network models&#10;that typically learn representations from stationary batches of training data,&#10;thus without accounting for situations in which information becomes&#10;incrementally available over time. In this review, we critically summarize the&#10;main challenges linked to lifelong learning for artificial learning systems and&#10;compare existing neural network approaches that alleviate, to different&#10;extents, catastrophic forgetting. We discuss well-established and emerging&#10;research motivated by lifelong learning factors in biological systems such as&#10;structural plasticity, memory replay, curriculum and transfer learning,&#10;intrinsic motivation, and multisensory integration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.09153" label="2112.09153">
        <attvalues>
          <attvalue for="0" value="An Empirical Investigation of the Role of Pre-training in Lifelong&#10;  Learning" />
          <attvalue for="1" value="  The lifelong learning paradigm in machine learning is an attractive&#10;alternative to the more prominent isolated learning scheme not only due to its&#10;resemblance to biological learning but also its potential to reduce energy&#10;waste by obviating excessive model re-training. A key challenge to this&#10;paradigm is the phenomenon of catastrophic forgetting. With the increasing&#10;popularity and success of pre-trained models in machine learning, we pose the&#10;question: What role does pre-training play in lifelong learning, specifically&#10;with respect to catastrophic forgetting? We investigate existing methods in the&#10;context of large, pre-trained models and evaluate their performance on a&#10;variety of text and image classification tasks, including a large-scale study&#10;using a novel data set of 15 diverse NLP tasks. Across all settings, we observe&#10;that generic pre-training implicitly alleviates the effects of catastrophic&#10;forgetting when learning multiple tasks sequentially compared to randomly&#10;initialized models. We then further investigate why pre-training alleviates&#10;forgetting in this setting. We study this phenomenon by analyzing the loss&#10;landscape, finding that pre-trained weights appear to ease forgetting by&#10;leading to wider minima. Based on this insight, we propose jointly optimizing&#10;for current task loss and loss basin sharpness to explicitly encourage wider&#10;basins during sequential fine-tuning. We show that this optimization approach&#10;outperforms several state-of-the-art task-sequential continual learning&#10;algorithms across multiple settings, occasionally even without retaining a&#10;memory that scales in size with the number of tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.09357" label="2205.09357">
        <attvalues>
          <attvalue for="0" value="Continual Pre-Training Mitigates Forgetting in Language and Vision" />
          <attvalue for="1" value="  Pre-trained models are nowadays a fundamental component of machine learning&#10;research. In continual learning, they are commonly used to initialize the model&#10;before training on the stream of non-stationary data. However, pre-training is&#10;rarely applied during continual learning. We formalize and investigate the&#10;characteristics of the continual pre-training scenario in both language and&#10;vision environments, where a model is continually pre-trained on a stream of&#10;incoming data and only later fine-tuned to different downstream tasks. We show&#10;that continually pre-trained models are robust against catastrophic forgetting&#10;and we provide strong empirical evidence supporting the fact that&#10;self-supervised pre-training is more effective in retaining previous knowledge&#10;than supervised protocols. Code is provided at&#10;https://github.com/AndreaCossu/continual-pretraining-nlp-vision .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.02714" label="2112.02714">
        <attvalues>
          <attvalue for="0" value="CLASSIC: Continual and Contrastive Learning of Aspect Sentiment&#10;  Classification Tasks" />
          <attvalue for="1" value="  This paper studies continual learning (CL) of a sequence of aspect sentiment&#10;classification(ASC) tasks in a particular CL setting called domain incremental&#10;learning (DIL). Each task is from a different domain or product. The DIL&#10;setting is particularly suited to ASC because in testing the system needs not&#10;know the task/domain to which the test data belongs. To our knowledge, this&#10;setting has not been studied before for ASC. This paper proposes a novel model&#10;called CLASSIC. The key novelty is a contrastive continual learning method that&#10;enables both knowledge transfer across tasks and knowledge distillation from&#10;old tasks to the new task, which eliminates the need for task ids in testing.&#10;Experimental results show the high effectiveness of CLASSIC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.14608" label="2403.14608">
        <attvalues>
          <attvalue for="0" value="Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey" />
          <attvalue for="1" value="  Large models represent a groundbreaking advancement in multiple application&#10;fields, enabling remarkable achievements across various tasks. However, their&#10;unprecedented scale comes with significant computational costs. These models,&#10;often consisting of billions of parameters, require vast amounts of&#10;computational resources for execution. Especially, the expansive scale and&#10;computational demands pose considerable challenges when customizing them for&#10;particular downstream tasks, particularly over the hardware platforms&#10;constrained by computational capabilities. Parameter Efficient Fine-Tuning&#10;(PEFT) provides a practical solution by efficiently adjusting the large models&#10;over the various downstream tasks. In particular, PEFT refers to the process of&#10;adjusting the parameters of a pre-trained large model to adapt it to a specific&#10;task or domain while minimizing the number of additional parameters introduced&#10;or computational resources required. This approach is particularly important&#10;when dealing with large-scale language models with high parameter counts, as&#10;fine-tuning these models from scratch can be computationally expensive and&#10;resource-intensive, posing considerable challenges in the supporting system&#10;platform design. In this survey, we present comprehensive studies of various&#10;PEFT algorithms, examining their performance and computational overhead.&#10;Moreover, we provide an overview of applications developed using different PEFT&#10;algorithms and discuss common techniques employed to mitigate computation costs&#10;for PEFT. In addition to providing an extensive survey from an algorithmic&#10;standpoint, we also examine various real-world system designs to investigate&#10;the implementation costs associated with different PEFT approaches. This survey&#10;serves as a valuable resource for researchers aiming to understand both the&#10;PEFT algorithm and its system implementation, offering detailed ......&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.07491" label="2305.07491">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Analysis of Adapter Efficiency" />
          <attvalue for="1" value="  Adapters have been positioned as a parameter-efficient fine-tuning (PEFT)&#10;approach, whereby a minimal number of parameters are added to the model and&#10;fine-tuned. However, adapters have not been sufficiently analyzed to understand&#10;if PEFT translates to benefits in training/deployment efficiency and&#10;maintainability/extensibility. Through extensive experiments on many adapters,&#10;tasks, and languages in supervised and cross-lingual zero-shot settings, we&#10;clearly show that for Natural Language Understanding (NLU) tasks, the parameter&#10;efficiency in adapters does not translate to efficiency gains compared to full&#10;fine-tuning of models. More precisely, adapters are relatively expensive to&#10;train and have slightly higher deployment latency. Furthermore, the&#10;maintainability/extensibility benefits of adapters can be achieved with simpler&#10;approaches like multi-task training via full fine-tuning, which also provide&#10;relatively faster training times. We, therefore, recommend that for moderately&#10;sized models for NLU tasks, practitioners should rely on full fine-tuning or&#10;multi-task training rather than using adapters. Our code is available at&#10;https://github.com/AI4Bharat/adapter-efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Parameter Efficient Fine-Tuning (PEFT): \cite{zoph-etal-2016-transfer} was one of the earliest to work on PEFT by showing that fine-tuning a part of a pre-trained model reduces memory requirements and helps to avoid overfitting. Despite its simplicity, determining what part of the model should be fine-tuned involves exhaustive searching. However, this has spurred research into injecting fine-tunable components into the pre-trained model, the most prominent being works on Adapters \cite{pmlr-v97-houlsby19a,bapna-firat-2019-simple,DBLP:conf/iclr/HuSWALWWC22} which are tiny feed-forward layers injected after the self-attention and/or feed-forward layers of Transformer models \cite{NIPS2017_3f5ee243}. Learnable prompts \cite{li-liang-2021-prefix}, which are parameters appended to the key and values of the attention layers, can also be considered as adapters via a simple reformulation \cite{he2022towards}. Works such as compacters \cite{DBLP:conf/nips/MahabadiHR21} and IA$^{3}$ \cite{DBLP:journals/corr/abs-2205-05638} further focus on reducing the size of adapters. On the other hand, works on AdapterFusion \cite{pfeiffer-etal-2021-adapterfusion}, and MAD-X \cite{pfeiffer-etal-2020-mad} focus more on the transfer learning capabilities of adapters. However, these works mainly focus on parameter efficiency and leave out other aspects of efficiency, such as training time, deployability, maintainability, and cross-lingual transfer effectiveness. AdapterDrop \cite{ruckle-etal-2021-adapterdrop} proposes to reduce adapter training time but ignores other aforementioned aspects, a gap which we fill in this paper. While our study represents the empirical comparison of In-langauge, zero-shot performance and convergence time of PEFT method, multi-task learning, and fine-tuning methods, prior research has examined the instability of PEFT method. \cite{chen-etal-2022-revisiting} demonstrated the instability of PEFT in relation to weight initialization, training time, and training data order. They also compared the performance of PEFT and fine-tuning methods with respect to different dataset sizes. Following the broken protocol issue as mentioned in the paper \cite[Section2]{chen-etal-2022-revisiting}, we have used different dev and test set for all our experiments. In addition to focusing on the observation that fine-tuning cannot be fully replaced by PEFT, our study has also demonstrated that multi-task learning can be an alternative to the PEFT method.&#10;&#10;Multilingual Pre-trained Models: Ever since the introduction of BERT \cite{devlin-etal-2019-bert}, which is a pre-trained model which leverages monolingual data, there has been a steep improvement in the performance of downstream NLP tasks such as sentiment analysis, question answering and natural language inference. This was followed by massively multilingual pre-trained models such as the language group agnostic model XLM-R \cite{conneau-etal-2020-unsupervised}, and language group specific models IndicBERT \cite{DBLP:journals/corr/abs-2212-05409,kakwani-etal-2020-indicnlpsuite}, IndoBERT \cite{koto-etal-2020-indolem}, AfriBerta \cite{ogueji-etal-2021-small}, etc. Multilingual models enable cross-lingual transfer, allowing models to be fine-tuned on one language and be evalauted in a zero-shot on other languages. The efficiency of transfer via fine-tuning has not received due attention, and our work focuses on this aspect both in full fine-tuning and PEFT paradigms.&#10;&#10;Multi-Task Learning (MTL): MTL focuses on fully-fine tuning one model for multiple tasks \cite{DBLP:conf/icml/Caruana93} but has only recently seen significant adoption \cite{wei2021finetuned,muennighoff2022crosslingual}. MTL benefits from cross-task transfer, which we also analyzed in this paper ($\S$\ref{sec:maintainability}). A general overview of MTL in deep learning can be found in \cite{DBLP:journals/corr/Ruder17a} and \cite{ DBLP:journals/corr/abs-2204-03508}. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.12394" label="2305.12394">
        <attvalues>
          <attvalue for="0" value="Pruning Pre-trained Language Models with Principled Importance and&#10;  Self-regularization" />
          <attvalue for="1" value="  Iterative pruning is one of the most effective compression methods for&#10;pre-trained language models. We discovered that finding the optimal pruning&#10;decision is an equality-constrained 0-1 Integer Linear Programming problem. The&#10;solution to this optimization problem leads to a principled importance&#10;criterion which we use to rank parameters during iterative model pruning. To&#10;mitigate the poor generalization at high sparsity levels, we propose a&#10;self-regularization scheme where model prediction is regularized by the latest&#10;checkpoint with increasing sparsity throughout pruning. Our experiments on&#10;natural language understanding, question-answering, named entity recognition,&#10;and data-to-text generation with various Transformer-based PLMs show the&#10;effectiveness of the approach at various sparsity levels.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-trained language models~(PLMs)~\cite{bert,gpt2} have significantly advanced the state-of-the-art in various natural language processing tasks~\cite{glue,webnlg,e2e,dart}. However, these models often contain a vast amount of parameters, posing non-trivial requirements for storage and computation. Due to this inefficiency, the applications of PLMs in resource-constrained scenarios are still limited.&#10;&#10;To resolve the above challenge, model compression~\cite{pkd,svd,albert} has been actively studied to make PLMs meet the practical requirement. Among them, iterative pruning methods are widely adopted at only a tiny expense of model performance when adapting PLMs to downstream tasks. &#10;During the course of iterative pruning, model parameters can not only be updated but also be pruned based on the rank of their importance scores in order to satisfy the cardinality constraint.&#10;Prevalent importance criteria are based on the parameter's magnitude~\cite{gupta,mag2} or sensitivity~\cite{l0,movement,superticket,platon}. Parameters with low importance scores are pruned and are expected to have little impact on model performance.&#10;&#10;Despite the empirical success, existing importance criteria for model pruning still face two major limitations: (1) they are heuristically defined and may not accurately quantify a parameter's contribution to the learning process, e.g., absolute weight value in magnitude-based pruning and gradient-weight product in sensitivity-based pruning; (2) they determine the importance of each parameter individually without considering the effect of coinstantaneous parameter updates on model performance, e.g., sensitivity is estimated by the absolute change in training error if only a single parameter is pruned and others remain unchanged.&#10;&#10;In this paper, we rethink the design of the importance criterion for model pruning from an optimization perspective. We begin by analyzing the temporal variation of any given learning objective based on a single-step gradient descent update under the iterative pruning setting. We show that finding the optimal pruning decision can be framed as solving an equality-constrained 0-1 Integer Linear Programming~(ILP) problem, where the constraint is defined by the specified sparsity. The resulting problem is a particular case of a general 0-1 Knapsack problem in which the weight for each item is the same. The solution to this problem naturally leads to a principled importance criterion which we use to rank all model parameters and derive the optimal stepwise pruning decision.&#10;&#10;When a high sparsity~(e.g., 80\%$\sim$90\%) is pursued, the limited capacity often renders the pruned model fails to retain satisfactory performance with conventional fine-tuning. To further improve the model's generalization ability, we propose a self-regularization scheme, where the model prediction is regularized by the latest best-performing model checkpoint during pruning. We show that such a scheme eases model learning with decreasing capacity and effectively yields a tighter upper bound of expected generalization error than learning from training data alone.&#10;&#10;To validate the effectiveness of our approach, dubbed PINS~(\underline{P}runing with principled \underline{I}mportance a\underline{N}d \underline{S}elf-regularization), we conducted extensive experiments with various pre-trained language models on a wide variety of tasks, including natural language understanding on GLUE~\cite{glue}), question answering on SQuAD~\cite{squad}, named entity recognition on CoNLL 2003~\cite{conll2003}, and data-to-text generation on WebNLG~\cite{webnlg}, DART~\cite{dart}, and E2E~\cite{e2e}. Experimental results show that PINS provides more accurate models at different sparsity levels. Detailed analysis shed further light on some intriguing properties of models pruned by PINS. By exploiting the resulting high sparsity, we show that the storage/inference can be reduced/accelerated by 8.9x and 2.7x using CSR format and a sparsity-aware inference runtime~\cite{deepsparse} on consumer-level CPUs.&#10;&#10;In summary, our contributions are:&#10;&#10;\begin{itemize}&#10;&#10;\item We establish the equivalence between the optimal pruning decision and the solution to an equality-constrained 0-1 Integer Linear Programming problem. The solution to this problem leads to a principled importance criterion that can be used to rank parameters during iterative pruning.&#10;&#10;\item We propose a simple yet effective self-regularization scheme to enhance the model's generalization capability, especially under a high-sparsity regime.&#10;&#10;\item Comprehensive experiments and analyses confirm the effectiveness of our approach at various sparsity levels. &#10;&#10;\end{itemize}&#10; &#09;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Model Compression, Optimization Techniques, Mathematics" />
        </attvalues>
      </node>
      <node id="1901.07931" label="1901.07931">
        <attvalues>
          <attvalue for="0" value="Evaluating the State-of-the-Art of End-to-End Natural Language&#10;  Generation: The E2E NLG Challenge" />
          <attvalue for="1" value="  This paper provides a comprehensive analysis of the first shared task on&#10;End-to-End Natural Language Generation (NLG) and identifies avenues for future&#10;research based on the results. This shared task aimed to assess whether recent&#10;end-to-end NLG systems can generate more complex output by learning from&#10;datasets containing higher lexical richness, syntactic complexity and diverse&#10;discourse phenomena. Introducing novel automatic and human metrics, we compare&#10;62 systems submitted by 17 institutions, covering a wide range of approaches,&#10;including machine learning architectures -- with the majority implementing&#10;sequence-to-sequence models (seq2seq) -- as well as systems based on&#10;grammatical rules and templates. Seq2seq-based systems have demonstrated a&#10;great potential for NLG in the challenge. We find that seq2seq systems&#10;generally score high in terms of word-overlap metrics and human evaluations of&#10;naturalness -- with the winning SLUG system (Juraska et al., 2018) being&#10;seq2seq-based. However, vanilla seq2seq models often fail to correctly express&#10;a given meaning representation if they lack a strong semantic control mechanism&#10;applied during decoding. Moreover, seq2seq models can be outperformed by&#10;hand-engineered systems in terms of overall quality, as well as complexity,&#10;length and diversity of outputs. This research has influenced, inspired and&#10;motivated a number of recent studies outwith the original competition, which we&#10;also summarise as part of this paper.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.02871" label="2007.02871">
        <attvalues>
          <attvalue for="0" value="DART: Open-Domain Structured Data Record to Text Generation" />
          <attvalue for="1" value="  We present DART, an open domain structured DAta Record to Text generation&#10;dataset with over 82k instances (DARTs). Data-to-Text annotations can be a&#10;costly process, especially when dealing with tables which are the major source&#10;of structured data and contain nontrivial structures. To this end, we propose a&#10;procedure of extracting semantic triples from tables that encodes their&#10;structures by exploiting the semantic dependencies among table headers and the&#10;table title. Our dataset construction framework effectively merged&#10;heterogeneous sources from open domain semantic parsing and dialogue-act-based&#10;meaning representation tasks by utilizing techniques such as: tree ontology&#10;annotation, question-answer pair to declarative sentence conversion, and&#10;predicate unification, all with minimum post-editing. We present systematic&#10;evaluation on DART as well as new state-of-the-art results on WebNLG 2017 to&#10;show that DART (1) poses new challenges to existing data-to-text datasets and&#10;(2) facilitates out-of-domain generalization. Our data and code can be found at&#10;https://github.com/Yale-LILY/dart.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.12002" label="2105.12002">
        <attvalues>
          <attvalue for="0" value="Super Tickets in Pre-Trained Language Models: From Model Compression to&#10;  Improving Generalization" />
          <attvalue for="1" value="  The Lottery Ticket Hypothesis suggests that an over-parametrized network&#10;consists of ``lottery tickets'', and training a certain collection of them&#10;(i.e., a subnetwork) can match the performance of the full model. In this&#10;paper, we study such a collection of tickets, which is referred to as ``winning&#10;tickets'', in extremely over-parametrized models, e.g., pre-trained language&#10;models. We observe that at certain compression ratios, the generalization&#10;performance of the winning tickets can not only match but also exceed that of&#10;the full model. In particular, we observe a phase transition phenomenon: As the&#10;compression ratio increases, generalization performance of the winning tickets&#10;first improves then deteriorates after a certain threshold. We refer to the&#10;tickets on the threshold as ``super tickets''. We further show that the phase&#10;transition is task and model dependent -- as the model size becomes larger and&#10;the training data set becomes smaller, the transition becomes more pronounced.&#10;Our experiments on the GLUE benchmark show that the super tickets improve&#10;single task fine-tuning by $0.9$ points on BERT-base and $1.0$ points on&#10;BERT-large, in terms of task-average score. We also demonstrate that adaptively&#10;sharing the super tickets across tasks benefits multi-task learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.12562" label="2206.12562">
        <attvalues>
          <attvalue for="0" value="PLATON: Pruning Large Transformer Models with Upper Confidence Bound of&#10;  Weight Importance" />
          <attvalue for="1" value="  Large Transformer-based models have exhibited superior performance in various&#10;natural language processing and computer vision tasks. However, these models&#10;contain enormous amounts of parameters, which restrict their deployment to&#10;real-world applications. To reduce the model size, researchers prune these&#10;models based on the weights' importance scores. However, such scores are&#10;usually estimated on mini-batches during training, which incurs large&#10;variability/uncertainty due to mini-batch sampling and complicated training&#10;dynamics. As a result, some crucial weights could be pruned by commonly used&#10;pruning methods because of such uncertainty, which makes training unstable and&#10;hurts generalization. To resolve this issue, we propose PLATON, which captures&#10;the uncertainty of importance scores by upper confidence bound (UCB) of&#10;importance estimation. In particular, for the weights with low importance&#10;scores but high uncertainty, PLATON tends to retain them and explores their&#10;capacity. We conduct extensive experiments with several Transformer-based&#10;models on natural language understanding, question answering and image&#10;classification to validate the effectiveness of PLATON. Results demonstrate&#10;that PLATON manifests notable improvement under different sparsity levels. Our&#10;code is publicly available at https://github.com/QingruZhang/PLATON.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.05407" label="2407.05407">
        <attvalues>
          <attvalue for="0" value="CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer&#10;  based on Supervised Semantic Tokens" />
          <attvalue for="1" value="  Recent years have witnessed a trend that large language model (LLM) based&#10;text-to-speech (TTS) emerges into the mainstream due to their high naturalness&#10;and zero-shot capacity. In this paradigm, speech signals are discretized into&#10;token sequences, which are modeled by an LLM with text as prompts and&#10;reconstructed by a token-based vocoder to waveforms. Obviously, speech tokens&#10;play a critical role in LLM-based TTS models. Current speech tokens are learned&#10;in an unsupervised manner, which lacks explicit semantic information and&#10;alignment to the text. In this paper, we propose to represent speech with&#10;supervised semantic tokens, which are derived from a multilingual speech&#10;recognition model by inserting vector quantization into the encoder. Based on&#10;the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice,&#10;which consists of an LLM for text-to-token generation and a conditional flow&#10;matching model for token-to-speech synthesis. Experimental results show that&#10;supervised semantic tokens significantly outperform existing unsupervised&#10;tokens in terms of content consistency and speaker similarity for zero-shot&#10;voice cloning. Moreover, we find that utilizing large-scale data further&#10;improves the synthesis performance, indicating the scalable capacity of&#10;CosyVoice. To the best of our knowledge, this is the first attempt to involve&#10;supervised speech tokens into TTS models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text-to-Speech (TTS) technology has made remarkable strides in recent years, transitioning from robotic-sounding speech to producing voices that are nearly indistinguishable from human speakers. At the forefront of this advancement are Large Language Models (LLMs), which have been increasingly utilized in TTS systems to generate speech with a higher degree of naturalness and the ability to synthesize voices in a zero-shot fashion \cite{DBLP:journals/corr/abs-2305-07243,DBLP:journals/corr/abs-2301-02111,DBLP:journals/corr/abs-2402-08093}. These LLM-based TTS models function by converting speech signals into sequences of tokens, with the LLM utilizing text as a condition to model these token sequences. A token vocoder is then employed to reconstruct the raw waveforms from the tokenized speech \cite{DBLP:conf/nips/KongKB20, DBLP:journals/corr/abs-2210-13438}.&#10;&#10;A critical aspect of the TTS process is the representation of speech tokens. Traditionally, tokens are acquired through unsupervised learning, which may not capture explicit semantic information or align well with corresponding text \cite{DBLP:journals/taslp/HsuBTLSM21,DBLP:journals/corr/abs-2210-13438}. Recognizing this gap, our work introduces supervised semantic tokens extracted from a multilingual speech recognition model, Whisper \cite{DBLP:conf/icml/RadfordKXBMS23}, by integrating vector quantization into the encoder. This innovation allows for more accurate semantic representation and alignment with text. Early studies have shown that quantizers with auxiliary automatic speech recognition (ASR) loss outperform k-means clustering on the universal speech model (USM) for speech-to-text translation and ASR tasks, as demonstrated in \cite{DBLP:journals/corr/abs-2306-12925}. Additionally, \cite{DBLP:journals/spl/YeGCLZ24} employed Gumbel-Softmax vector quantization to extract discrete speech representations that prioritize ASR-relevant information for ASR tasks. However, the impact of these approaches on text-to-speech (TTS) remains unclear.&#10;&#10;Furthermore, leveraging these supervised tokens, we propose CosyVoice, a scalable and efficient zero-shot TTS synthesizer. CosyVoice is comprised of an LLM for converting text into semantic token sequences and a conditional flow matching model for the subsequent synthesis of speech from these tokens. &#10;In contrast to prior systems like TorToise TTS \cite{DBLP:journals/corr/abs-2305-07243}, which employs an LLM in conjunction with a denoising diffusion probabilistic models (DDPM)~\cite{DBLP:conf/nips/HoJA20}, CosyVoice utilizes a conditional flow matching approach, as it has been demonstrated to accelerate both training and inference compared to traditional diffusion models \cite{le2024voicebox}. While existing methods incorporate flow matching in TTS \cite{le2024voicebox, DBLP:journals/corr/abs-2309-05027,DBLP:journals/corr/abs-2309-03199,DBLP:journals/corr/abs-2309-17056}, they often rely on phoneme duration prediction, necessitating the use of supplementary phonemizers and forced aligners. CosyVoice, however, bypasses these dependencies, offering a more direct and efficient pathway from text to speech.&#10;&#10;Our research contributes to the field of speech generation in several novel ways:&#10;\begin{itemize}[leftmargin=*,noitemsep]&#10;\item We are the first to integrate supervised speech tokens into TTS models, enhancing content consistency and speaker similarity in zero-shot voice cloning.&#10;\item We propose CosyVoice, a scalable zero-shot TTS synthesis system that combines an LLM for text-to-token generation with a conditional flow matching model for token-to-speech synthesis, forsaking the need for additional phonemizers and forced aligners.&#10;\item To further refine the quality of generated speech, we incorporate the x-vector \cite{DBLP:conf/icassp/SnyderGSPK18} into the LLM to separate the modeling of speech into semantic, speaker, and prosody components. The LLM models the semantic content and prosody, while the conditional flow matching model captures timbre and environmental information. We optimize the flow matching process with techniques such as classifier-free guidance \cite{DBLP:journals/corr/abs-2207-12598}, a cosine scheduler, and masked conditions.&#10;\end{itemize}&#10;&#10;Our experimental results demonstrate the superiority of supervised semantic tokens over unsupervised counterparts. Additionally, the scalability of CosyVoice is evidenced by improved synthesis performance when utilizing large-scale data. This work, therefore, represents a significant step forward in the development of natural-sounding, versatile TTS systems.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Text-to-Speech Synthesis, Speech Tokenization, Linguistics, Engineering, Signal Processing" />
        </attvalues>
      </node>
      <node id="2305.07243" label="2305.07243">
        <attvalues>
          <attvalue for="0" value="Better speech synthesis through scaling" />
          <attvalue for="1" value="  In recent years, the field of image generation has been revolutionized by the&#10;application of autoregressive transformers and DDPMs. These approaches model&#10;the process of image generation as a step-wise probabilistic processes and&#10;leverage large amounts of compute and data to learn the image distribution.&#10;This methodology of improving performance need not be confined to images. This&#10;paper describes a way to apply advances in the image generative domain to&#10;speech synthesis. The result is TorToise -- an expressive, multi-voice&#10;text-to-speech system.&#10;  All model code and trained weights have been open-sourced at&#10;https://github.com/neonbjb/tortoise-tts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.08093" label="2402.08093">
        <attvalues>
          <attvalue for="0" value="BASE TTS: Lessons from building a billion-parameter Text-to-Speech model&#10;  on 100K hours of data" />
          <attvalue for="1" value="  We introduce a text-to-speech (TTS) model called BASE TTS, which stands for&#10;$\textbf{B}$ig $\textbf{A}$daptive $\textbf{S}$treamable TTS with&#10;$\textbf{E}$mergent abilities. BASE TTS is the largest TTS model to-date,&#10;trained on 100K hours of public domain speech data, achieving a new&#10;state-of-the-art in speech naturalness. It deploys a 1-billion-parameter&#10;autoregressive Transformer that converts raw texts into discrete codes&#10;(&quot;speechcodes&quot;) followed by a convolution-based decoder which converts these&#10;speechcodes into waveforms in an incremental, streamable manner. Further, our&#10;speechcodes are built using a novel speech tokenization technique that features&#10;speaker ID disentanglement and compression with byte-pair encoding. Echoing the&#10;widely-reported &quot;emergent abilities&quot; of large language models when trained on&#10;increasing volume of data, we show that BASE TTS variants built with 10K+ hours&#10;and 500M+ parameters begin to demonstrate natural prosody on textually complex&#10;sentences. We design and share a specialized dataset to measure these emergent&#10;abilities for text-to-speech. We showcase state-of-the-art naturalness of BASE&#10;TTS by evaluating against baselines that include publicly available large-scale&#10;text-to-speech systems: YourTTS, Bark and TortoiseTTS. Audio samples generated&#10;by the model can be heard at https://amazon-ltts-paper.com/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05646" label="2010.05646">
        <attvalues>
          <attvalue for="0" value="HiFi-GAN: Generative Adversarial Networks for Efficient and High&#10;  Fidelity Speech Synthesis" />
          <attvalue for="1" value="  Several recent work on speech synthesis have employed generative adversarial&#10;networks (GANs) to produce raw waveforms. Although such methods improve the&#10;sampling efficiency and memory usage, their sample quality has not yet reached&#10;that of autoregressive and flow-based generative models. In this work, we&#10;propose HiFi-GAN, which achieves both efficient and high-fidelity speech&#10;synthesis. As speech audio consists of sinusoidal signals with various periods,&#10;we demonstrate that modeling periodic patterns of an audio is crucial for&#10;enhancing sample quality. A subjective human evaluation (mean opinion score,&#10;MOS) of a single speaker dataset indicates that our proposed method&#10;demonstrates similarity to human quality while generating 22.05 kHz&#10;high-fidelity audio 167.9 times faster than real-time on a single V100 GPU. We&#10;further show the generality of HiFi-GAN to the mel-spectrogram inversion of&#10;unseen speakers and end-to-end speech synthesis. Finally, a small footprint&#10;version of HiFi-GAN generates samples 13.4 times faster than real-time on CPU&#10;with comparable quality to an autoregressive counterpart.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.17056" label="2309.17056">
        <attvalues>
          <attvalue for="0" value="ReFlow-TTS: A Rectified Flow Model for High-fidelity Text-to-Speech" />
          <attvalue for="1" value="  The diffusion models including Denoising Diffusion Probabilistic Models&#10;(DDPM) and score-based generative models have demonstrated excellent&#10;performance in speech synthesis tasks. However, its effectiveness comes at the&#10;cost of numerous sampling steps, resulting in prolonged sampling time required&#10;to synthesize high-quality speech. This drawback hinders its practical&#10;applicability in real-world scenarios. In this paper, we introduce ReFlow-TTS,&#10;a novel rectified flow based method for speech synthesis with high-fidelity.&#10;Specifically, our ReFlow-TTS is simply an Ordinary Differential Equation (ODE)&#10;model that transports Gaussian distribution to the ground-truth Mel-spectrogram&#10;distribution by straight line paths as much as possible. Furthermore, our&#10;proposed approach enables high-quality speech synthesis with a single sampling&#10;step and eliminates the need for training a teacher model. Our experiments on&#10;LJSpeech Dataset show that our ReFlow-TTS method achieves the best performance&#10;compared with other diffusion based models. And the ReFlow-TTS with one step&#10;sampling achieves competitive performance compared with existing one-step TTS&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.12598" label="2207.12598">
        <attvalues>
          <attvalue for="0" value="Classifier-Free Diffusion Guidance" />
          <attvalue for="1" value="  Classifier guidance is a recently introduced method to trade off mode&#10;coverage and sample fidelity in conditional diffusion models post training, in&#10;the same spirit as low temperature sampling or truncation in other types of&#10;generative models. Classifier guidance combines the score estimate of a&#10;diffusion model with the gradient of an image classifier and thereby requires&#10;training an image classifier separate from the diffusion model. It also raises&#10;the question of whether guidance can be performed without a classifier. We show&#10;that guidance can be indeed performed by a pure generative model without such a&#10;classifier: in what we call classifier-free guidance, we jointly train a&#10;conditional and an unconditional diffusion model, and we combine the resulting&#10;conditional and unconditional score estimates to attain a trade-off between&#10;sample quality and diversity similar to that obtained using classifier&#10;guidance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.12506" label="2004.12506">
        <attvalues>
          <attvalue for="0" value="Assessing Discourse Relations in Language Generation from GPT-2" />
          <attvalue for="1" value="  Recent advances in NLP have been attributed to the emergence of large-scale&#10;pre-trained language models. GPT-2, in particular, is suited for generation&#10;tasks given its left-to-right language modeling objective, yet the linguistic&#10;quality of its generated text has largely remain unexplored. Our work takes a&#10;step in understanding GPT-2's outputs in terms of discourse coherence. We&#10;perform a comprehensive study on the validity of explicit discourse relations&#10;in GPT-2's outputs under both organic generation and fine-tuned scenarios.&#10;Results show GPT-2 does not always generate text containing valid discourse&#10;relations; nevertheless, its text is more aligned with human expectation in the&#10;fine-tuned scenario. We propose a decoupled strategy to mitigate these problems&#10;and highlight the importance of explicitly modeling discourse information.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent progress in NLP has been marked with the emergence of large-scale pre-trained models, e.g., ELMo~\cite{elmo}, BERT~\cite{bert}, and GPT-2~\cite{gpt2}. Among these, GPT-2 is particularly suitable in natural language generation due to its underlying left-to-right language modeling objective. &#10;Indeed, GPT-based language models have shown impressive results for open-domain dialogue generation \cite{golovanov2019large,wolf2019transfertransfo,dialogpt}. This has motivated investigations into GPT-2's generated text \cite{see2019massively,wallace-etal-2019-universal}. In particular, using automatic metrics (e.g., cosine similarity, lexical diversity, sentence length), \cite{see2019massively} illustrated that GPT-2 has the ability to generate interesting and coherent text. However, analysis of GPT-2's outputs from deeper linguistic dimensions (e.g., discourse) has largely remained unexplored.&#10;&#10;In this paper, we perform the first discourse analysis of GPT-2's outputs, under both organic and fine-tuned scenarios, with the goals of understanding model behavior and pointing towards ways of improvement. We chiefly focus on discourse relations, one of the most important linguistic devices for textual coherence. Discourse relations specify the relationships between text spans, for example:&#10;\begin{quote}&#10; Jazz is good, but my favorite is country music.&#10;\end{quote}&#10;The two clauses (also called arguments) are connected by a contrast relation, as signaled by the connective but. &#10;Discourse relations are central in establishing textual coherence. For example, they create rhetorical connections between spans in the absence of anaphoric entity mentions \cite{lascarides2008segmented}.&#10;Cognitive experiments have repeatedly shown discourse relations to be highly influential in the mental processing of text \cite{meyer1984effects,horowitz1987rhetorical,millis1993impact,sanders2000role}. &#10;Spans joined with incorrect discourse connectives can seem logically incoherent although they are independently grammatical:&#10;\begin{quote}&#10; Jazz is good, because my favorite is country music.&#10;\end{quote}&#10;The importance of generating good discourse connectives are recognized in prior work in NLG~\cite{biran-mckeown-2015-discourse,callaway-2003-integrating}.&#10;&#10;We examine to what extent does GPT-2 generate texts that uphold plausible discourse relations, once a discourse connective (usually 1-2 tokens) is generated. &#10;We present a comprehensive analysis of discourse connectives in both fine-tuned generation---specifically, open domain dialogue generation---and organic generation directly from GPT-2. We find that GPT-2 generates valid discourse connectives when the relation can be inferred by humans with high agreement, yet struggles to recover less obvious relations.&#10;Our manual analysis reveals the most common connective error is that the relations, signaled by the connectives, do not hold between the spans they connect. To this end, we propose a simple remedy: train a connective prediction model and replace incorrect connectives in a post-processing step. This method improves agreement between human and machine-generated connectives in both the fine-tuned and the organic scenarios. Collectively, our results highlight the importance of inferring discourse relations~\cite{xue2015conll}, and explicitly incorporating discourse information in language models~\cite{ji2016latent}, to increase their downstream efficacy.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Natural Language Processing, Discourse Coherence Analysis, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2204.02470" label="2204.02470">
        <attvalues>
          <attvalue for="0" value="Combining Spectral and Self-Supervised Features for Low Resource Speech&#10;  Recognition and Translation" />
          <attvalue for="1" value="  Self-Supervised Learning (SSL) models have been successfully applied in&#10;various deep learning-based speech tasks, particularly those with a limited&#10;amount of data. However, the quality of SSL representations depends highly on&#10;the relatedness between the SSL training domain(s) and the target data domain.&#10;On the contrary, spectral feature (SF) extractors such as log Mel-filterbanks&#10;are hand-crafted non-learnable components, and could be more robust to domain&#10;shifts. The present work examines the assumption that combining non-learnable&#10;SF extractors to SSL models is an effective approach to low resource speech&#10;tasks. We propose a learnable and interpretable framework to combine SF and SSL&#10;representations. The proposed framework outperforms significantly both baseline&#10;and SSL models on Automatic Speech Recognition (ASR) and Speech Translation&#10;(ST) tasks on three low resource datasets. We additionally design a mixture of&#10;experts based combination model. This last model reveals that the relative&#10;contribution of SSL models over conventional SF extractors is very small in&#10;case of domain mismatch between SSL training set and the target language data.&#10;" />
          <attvalue for="2" value="&#10;&#10;End-to-end models based on deep learning have demonstrated their superiority over conventional hidden Markov-based models on speech tasks for some corpora \cite{chiu2018state, karita2019comparative, pham2019very, guo2021recent}. End-to-end models could be beneficial to low resource speech tasks because these models: (1) alleviate the need of language specific resources such as lexicons \cite{grenoble2011handbook, zahrer2020towards, shi-etal-2021-leveraging}. (2) can be trained multilingually to facilitate cross-lingual transfers between high resource and low resource languages through shared architecture and weights \cite{8639655}. On the other hand, end-to-end models can perform poorly when the training data is limited \cite{rwth-libri-2019} and low resource scenarios often introduce a language-mismatch with the data used to train powerful self-supervised learning (SSL) representations \cite{tsai2022superbsg}.&#10;&#10;One direction towards mitigating these low-resource issues is to incorporate knowledge from several languages into multilingual end-to-end models \cite{watanabe2017language, toshniwal2018multilingual, Kannan2019}.&#10;When there is no training data available for the target languages, these systems can be even applied in a zero-shot manner \cite{li2020universal, yan2021differentiable, xu2021simple}.&#10;Fortunately, many languages have small amounts of data which can be used to fine-tune large-scale multilingual models towards target languages, resulting in further improvements \cite{hou20_interspeech, pratap20c_interspeech, adams2019massively, li2021scaling}.&#10;&#10;Another direction is to use self-supervised learning models trained on large untranscribed corpora as front-end feature extractors, replacing conventional spectral features (SF) such as log Mel-filterbanks coefficients (FBANK) \cite{yi2020applying, wu20g_interspeech, baevski2020wav2vec, n21_interspeech, chang2021exploration, liu2021tera}. During their unsupervised training, SSL models \cite{DBLP:HuBERT, DBLP:wav2vec,DBLP:wav2vec2,chen2021wavlm} learn their own feature extraction modules and are totally free of SF at fine-tuning time. As these models achieve state of the art on numerous speech &#10;tasks and significantly outperform models with more supervision, the effectiveness of SF on low resource tasks is increasingly questioned. &#10;&#10;The majority of SSL models are trained exclusively using English speech. Although these approaches have shown improvements, even when domain mismatches occur (such as language or audio conditions \cite{sanabria2022measuring}), performance depends on the relatedness between the SSL training domain and the target language one \cite{conneau2019unsupervised}. SSL first layers output representations tend to be quite similar to SF according to a canonical correlation analysis \cite{andrew2013deep} of Wav2vec2 \cite{DBLP:wav2vec2} from Pasad et al. \cite{DBLP:layerwiseToyota}. In contrast, the last layers are likely to be more corpus or domain-specific, which should be randomly initialized at fine-tuning time \cite{DBLP:layerwiseToyota}. Therefore, we assume that SSL representations are potentially more hurted by domain shifts than SF-based systems are. SF are domain and language agnostic and their use in multilingual models has demonstrated that they enable strong cross-lingual transfers \cite{8639655}. It is then legitimate to assume that a model leveraging both SF and SSL representations would lead to strong performances on low resource speech scenarios.&#10;&#10;In the present work, we examine this assumption by building a framework that enables combining SF and SSL representations through learnable fusions. We propose linear, convolutional and co-attention based combinations. Those methods obtain a relative diminution of $19.3\%$ Character Error Rate (CER), averaged on two ASR datasets, and a gain of 1.0 BLEU, on an ST dataset, over the SSL baseline model, while having less than 0.01\% additional parameters. We further propose a mixture of experts \cite{MoE} based technique in order to better interpret the roles and complementarities of SF and SSL components. Finally the proposed framework is evaluated on Totonac, a Mexican endangered language, and we release the first publicly available annotated speech corpus of this language.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Domain Adaptation, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2002.11800" label="2002.11800">
        <attvalues>
          <attvalue for="0" value="Universal Phone Recognition with a Multilingual Allophone System" />
          <attvalue for="1" value="  Multilingual models can improve language processing, particularly for low&#10;resource situations, by sharing parameters across languages. Multilingual&#10;acoustic models, however, generally ignore the difference between phonemes&#10;(sounds that can support lexical contrasts in a particular language) and their&#10;corresponding phones (the sounds that are actually spoken, which are language&#10;independent). This can lead to performance degradation when combining a variety&#10;of training languages, as identically annotated phonemes can actually&#10;correspond to several different underlying phonetic realizations. In this work,&#10;we propose a joint model of both language-independent phone and&#10;language-dependent phoneme distributions. In multilingual ASR experiments over&#10;11 languages, we find that this model improves testing performance by 2%&#10;phoneme error rate absolute in low-resource conditions. Additionally, because&#10;we are explicitly modeling language-independent phones, we can build a&#10;(nearly-)universal phone recognizer that, when combined with the PHOIBLE large,&#10;manually curated database of phone inventories, can be customized into 2,000&#10;language dependent recognizers. Experiments on two low-resourced indigenous&#10;languages, Inuktitut and Tusom, show that our recognizer achieves phone&#10;accuracy improvements of more than 17%, moving a step closer to speech&#10;recognition for all languages in the world.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.12121" label="2012.12121">
        <attvalues>
          <attvalue for="0" value="Applying Wav2vec2.0 to Speech Recognition in Various Low-resource&#10;  Languages" />
          <attvalue for="1" value="  There are several domains that own corresponding widely used feature&#10;extractors, such as ResNet, BERT, and GPT-x. These models are usually&#10;pre-trained on large amounts of unlabeled data by self-supervision and can be&#10;effectively applied to downstream tasks. In the speech domain, wav2vec2.0&#10;starts to show its powerful representation ability and feasibility of ultra-low&#10;resource speech recognition on the Librispeech corpus, which belongs to the&#10;audiobook domain. However, wav2vec2.0 has not been examined on real spoken&#10;scenarios and languages other than English. To verify its universality over&#10;languages, we apply pre-trained models to solve low-resource speech recognition&#10;tasks in various spoken languages. We achieve more than 20% relative&#10;improvements in six languages compared with previous work. Among these&#10;languages, English achieves a gain of 52.4%. Moreover, using coarse-grained&#10;modeling units, such as subword or character, achieves better results than&#10;fine-grained modeling units, such as phone or letter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.05847" label="2110.05847">
        <attvalues>
          <attvalue for="0" value="Evaluation of Abstractive Summarisation Models with Machine Translation&#10;  in Deliberative Processes" />
          <attvalue for="1" value="  We present work on summarising deliberative processes for non-English&#10;languages. Unlike commonly studied datasets, such as news articles, this&#10;deliberation dataset reflects difficulties of combining multiple narratives,&#10;mostly of poor grammatical quality, in a single text. We report an extensive&#10;evaluation of a wide range of abstractive summarisation models in combination&#10;with an off-the-shelf machine translation model. Texts are translated into&#10;English, summarised, and translated back to the original language. We obtain&#10;promising results regarding the fluency, consistency and relevance of the&#10;summaries produced. Our approach is easy to implement for many languages for&#10;production purposes by simply changing the translation model.&#10;" />
          <attvalue for="2" value="&#10;&#10;The processes of deliberation and collective intelligence production have evolved radically thanks to the possibility of carrying them out digitally. However, this often results in large amounts of generated content in the deliberations, causing information overload that prevents their potential from being fully realised \cite{arana2021citizen, davies2020online, davies2021mixed}. To address this, we evaluate the potential value of abstractive summarisation models when combined together with a machine translation system in synthesising and filtering information collected through such processes. Whereas the current technology of language models is mostly limited to a few languages, which creates a barrier to their more widespread use, our approach can be deployed for many languages just by changing the translation model without the need to generate new, ad-hoc corpora for the task or costly retraining for each new language. The current evaluation is done in a Spanish deliberation dataset.&#10;&#10;We have carried out an evaluation with 6 abstractive summarisation models: BART \cite{lewis2019bart}, T5 \cite{raffel2019exploring}, BERT (PreSumm – BertSumExtAbs: \cite{liu2019text}), PG (Pointer-Generator with Coverage Penalty) \cite{see2017get}, CopyTransformer \cite{gehrmann2018bottom}, and FastAbsRL \cite{chen2018fast}. Those models are applied in combination with the machine translation system MarianMT \cite{junczys2018marian} using the Opus-MT models \cite{tiedemann2020opus}. We have evaluated the quality of the summaries for each model and their comparison.&#10;&#10;Early research on the problem of text summarisation in low resourced languages (although not focused on deliberation)~\cite{orasan2008evaluation} demonstrated the limitations of machine translation systems at that time. Recently, \cite{ouyang2019robust} revisited the problem of low quality translations in low resourced languages and successfully demonstrated the possibility of using abstractive summarisation by retraining their model on corpora that have gone through the same machine translation process. In this study, we complete the cycle, translating from the original language to English, summarising, and translating back to the original language, thus avoiding the need for retraining.&#10;&#10;Using other approaches, \cite{yao2015phrase} studied English-to-Chinese summarisation combining an extractive approach with a process of sentence compression that effectively abstracts the results. \cite{duan2019zero}, following \cite{shen2018zero}, exploited the capability of a resource-rich language summariser in a teacher-student framework that connects it to the target language summariser.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Multilingual Summarization, Computer Science, Abstractive Summarization Models, Linguistics, Machine Translation Evaluation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2103.00508" label="2103.00508">
        <attvalues>
          <attvalue for="0" value="Citizen Participation and Machine Learning for a Better Democracy" />
          <attvalue for="1" value="  The development of democratic systems is a crucial task as confirmed by its&#10;selection as one of the Millennium Sustainable Development Goals by the United&#10;Nations. In this article, we report on the progress of a project that aims to&#10;address barriers, one of which is information overload, to achieving effective&#10;direct citizen participation in democratic decision-making processes. The main&#10;objectives are to explore if the application of Natural Language Processing&#10;(NLP) and machine learning can improve citizens' experience of digital citizen&#10;participation platforms. Taking as a case study the &quot;Decide Madrid&quot; Consul&#10;platform, which enables citizens to post proposals for policies they would like&#10;to see adopted by the city council, we used NLP and machine learning to provide&#10;new ways to (a) suggest to citizens proposals they might wish to support; (b)&#10;group citizens by interests so that they can more easily interact with each&#10;other; (c) summarise comments posted in response to proposals; (d) assist&#10;citizens in aggregating and developing proposals. Evaluation of the results&#10;confirms that NLP and machine learning have a role to play in addressing some&#10;of the barriers users of platforms such as Consul currently experience.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.11080" label="1805.11080">
        <attvalues>
          <attvalue for="0" value="Fast Abstractive Summarization with Reinforce-Selected Sentence&#10;  Rewriting" />
          <attvalue for="1" value="  Inspired by how humans summarize long documents, we propose an accurate and&#10;fast summarization model that first selects salient sentences and then rewrites&#10;them abstractively (i.e., compresses and paraphrases) to generate a concise&#10;overall summary. We use a novel sentence-level policy gradient method to bridge&#10;the non-differentiable computation between these two neural networks in a&#10;hierarchical way, while maintaining language fluency. Empirically, we achieve&#10;the new state-of-the-art on all metrics (including human evaluation) on the&#10;CNN/Daily Mail dataset, as well as significantly higher abstractiveness scores.&#10;Moreover, by first operating at the sentence-level and then the word-level, we&#10;enable parallel decoding of our neural generative model that results in&#10;substantially faster (10-20x) inference speed as well as 4x faster training&#10;convergence than previous long-paragraph encoder-decoder models. We also&#10;demonstrate the generalization of our model on the test-only DUC-2002 dataset,&#10;where we achieve higher scores than a state-of-the-art model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.00344" label="1804.00344">
        <attvalues>
          <attvalue for="0" value="Marian: Fast Neural Machine Translation in C++" />
          <attvalue for="1" value="  We present Marian, an efficient and self-contained Neural Machine Translation&#10;framework with an integrated automatic differentiation engine based on dynamic&#10;computation graphs. Marian is written entirely in C++. We describe the design&#10;of the encoder-decoder framework and demonstrate that a research-friendly&#10;toolkit can achieve high training and translation speed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12627" label="2010.12627">
        <attvalues>
          <attvalue for="0" value="Anchor-based Bilingual Word Embeddings for Low-Resource Languages" />
          <attvalue for="1" value="  Good quality monolingual word embeddings (MWEs) can be built for languages&#10;which have large amounts of unlabeled text. MWEs can be aligned to bilingual&#10;spaces using only a few thousand word translation pairs. For low resource&#10;languages training MWEs monolingually results in MWEs of poor quality, and thus&#10;poor bilingual word embeddings (BWEs) as well. This paper proposes a new&#10;approach for building BWEs in which the vector space of the high resource&#10;source language is used as a starting point for training an embedding space for&#10;the low resource target language. By using the source vectors as anchors the&#10;vector spaces are automatically aligned during training. We experiment on&#10;English-German, English-Hiligaynon and English-Macedonian. We show that our&#10;approach results not only in improved BWEs and bilingual lexicon induction&#10;performance, but also in improved target language MWE quality as measured using&#10;monolingual word similarity.&#10;" />
          <attvalue for="2" value="&#10;&#10;Bilingual Word Embeddings are useful for cross-lingual tasks such as cross-lingual transfer learning or machine translation.&#10;Mapping based BWE approaches rely only on a cheap bilingual signal, in the form of a seed lexicon, and monolingual data to train monolingual word embeddings (MWEs) for each language, which makes them easily applicable in low-resource scenarios \cite{Mikolov2013exploiting,Xing2015,Artetxe2016}.&#10;It was shown that BWEs can be built using a small seed lexicon \cite{Artexte2017} or without any word pairs \cite{Conneau2018,Artetxe2018} relying on the assumption of isomorphic MWE spaces.&#10;Recent approaches showed that BWEs can be built&#10;without the mapping step.&#10;\cite{Lample2018SMT} built FastText embeddings \cite{bojanowski-etal-2017-enriching} on the concatenated source and target language corpora exploiting the&#10;shared&#10;character n-grams in them.&#10;Similarly, the shared source and target language&#10;subword tokens are used as a cheap cross-lingual signal in \cite{devlin-etal-2019-bert,Conneau2019XLM}.&#10;Furthermore, the advantages of mapping and&#10;jointly training the MWEs and BWEs were&#10;combined&#10;in \cite{Wang2020} for even better BWEs.&#10;&#10;While these approaches already try to minimize the amount of bilingual signal needed for cross-lingual applications, they still require a larger amount of monolingual data to train semantically rich word embeddings \cite{adams-etal-2017-cross}.&#10;This becomes a problem when one of the two languages does not have sufficient monolingual data available \cite{artetxe2020rigor}.&#10;In this case, training a good embedding space can be infeasible which means mapping based approaches are not able to build useful BWEs \cite{Michel2020}.&#10;&#10;In this paper we introduce a new approach to building BWEs when&#10;one of the languages only has limited available monolingual data.&#10;Instead of using mapping or joint approaches, this paper takes the middle ground by making use of the MWEs of a resource rich language and training the low resource language embeddings on top of it. For this, a bilingual seed lexicon is used to initialize the representation of target language words by taking the pre-trained vectors of their source pairs prior to target side training, which acts as an informed starting point to shape the vector space during the process.&#10;We randomly initialize the representations of all non-lexicon target words and&#10;run Continuous Bag-of-Words (CBOW) and skip-gram (SG) training procedures to&#10;generate target embeddings with both Word2Vec&#10;\cite{Mikolov2013efficient} and FastText&#10;\cite{bojanowski-etal-2017-enriching}.&#10;Our approach ensures that the source language MWE space is intact,&#10;so that the data deficit on the target side does not result in lowered source embedding quality.&#10;The improved monolingual word embeddings for the target language&#10;outperform embeddings trained solely on monolingual data&#10;for semantic tasks such as word-similarity prediction.&#10;We study low-resource settings for English-German and English-Hiligaynon, where previous approaches have failed \cite{Michel2020}, as well as English-Macedonian.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Language Alignment, Linguistics, Word Embeddings, Multilingual Processing, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2205.13660" label="2205.13660">
        <attvalues>
          <attvalue for="0" value="Contextual Adapters for Personalized Speech Recognition in Neural&#10;  Transducers" />
          <attvalue for="1" value="  Personal rare word recognition in end-to-end Automatic Speech Recognition&#10;(E2E ASR) models is a challenge due to the lack of training data. A standard&#10;way to address this issue is with shallow fusion methods at inference time.&#10;However, due to their dependence on external language models and the&#10;deterministic approach to weight boosting, their performance is limited. In&#10;this paper, we propose training neural contextual adapters for personalization&#10;in neural transducer based ASR models. Our approach can not only bias towards&#10;user-defined words, but also has the flexibility to work with pretrained ASR&#10;models. Using an in-house dataset, we demonstrate that contextual adapters can&#10;be applied to any general purpose pretrained ASR model to improve&#10;personalization. Our method outperforms shallow fusion, while retaining&#10;functionality of the pretrained models by not altering any of the model&#10;weights. We further show that the adapter style training is superior to&#10;full-fine-tuning of the ASR models on datasets with user-defined content.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.09427" label="2112.09427">
        <attvalues>
          <attvalue for="0" value="Continual Learning for Monolingual End-to-End Automatic Speech&#10;  Recognition" />
          <attvalue for="1" value="  Adapting Automatic Speech Recognition (ASR) models to new domains results in&#10;a deterioration of performance on the original domain(s), a phenomenon called&#10;Catastrophic Forgetting (CF). Even monolingual ASR models cannot be extended to&#10;new accents, dialects, topics, etc. without suffering from CF, making them&#10;unable to be continually enhanced without storing all past data. Fortunately,&#10;Continual Learning (CL) methods, which aim to enable continual adaptation while&#10;overcoming CF, can be used. In this paper, we implement an extensive number of&#10;CL methods for End-to-End ASR and test and compare their ability to extend a&#10;monolingual Hybrid CTC-Transformer model across four new tasks. We find that&#10;the best performing CL method closes the gap between the fine-tuned model&#10;(lower bound) and the model trained jointly on all tasks (upper bound) by more&#10;than 40%, while requiring access to only 0.6% of the original data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.16082" label="2203.16082">
        <attvalues>
          <attvalue for="0" value="Using Adapters to Overcome Catastrophic Forgetting in End-to-End&#10;  Automatic Speech Recognition" />
          <attvalue for="1" value="  Learning a set of tasks in sequence remains a challenge for artificial neural&#10;networks, which, in such scenarios, tend to suffer from Catastrophic Forgetting&#10;(CF). The same applies to End-to-End (E2E) Automatic Speech Recognition (ASR)&#10;models, even for monolingual tasks. In this paper, we aim to overcome CF for&#10;E2E ASR by inserting adapters, small architectures of few parameters which&#10;allow a general model to be fine-tuned to a specific task, into our model. We&#10;make these adapters task-specific, while regularizing the parameters of the&#10;model shared by all tasks, thus stimulating the model to fully exploit the&#10;adapters while keeping the shared parameters to work well for all tasks. Our&#10;method outperforms all baselines on two monolingual experiments while being&#10;more storage efficient and without requiring the storage of data from previous&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.03209" label="1804.03209">
        <attvalues>
          <attvalue for="0" value="Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition" />
          <attvalue for="1" value="  Describes an audio dataset of spoken words designed to help train and&#10;evaluate keyword spotting systems. Discusses why this task is an interesting&#10;challenge, and why it requires a specialized dataset that is different from&#10;conventional datasets used for automatic speech recognition of full sentences.&#10;Suggests a methodology for reproducible and comparable accuracy metrics for&#10;this task. Describes how the data was collected and verified, what it contains,&#10;previous versions and properties. Concludes by reporting baseline results of&#10;models trained on this dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.15207" label="2007.15207">
        <attvalues>
          <attvalue for="0" value="MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain&#10;  Question Answering" />
          <attvalue for="1" value="  Progress in cross-lingual modeling depends on challenging, realistic, and&#10;diverse evaluation sets. We introduce Multilingual Knowledge Questions and&#10;Answers (MKQA), an open-domain question answering evaluation set comprising 10k&#10;question-answer pairs aligned across 26 typologically diverse languages (260k&#10;question-answer pairs in total). Answers are based on a heavily curated,&#10;language-independent data representation, making results comparable across&#10;languages and independent of language-specific passages. With 26 languages,&#10;this dataset supplies the widest range of languages to-date for evaluating&#10;question answering. We benchmark a variety of state-of-the-art methods and&#10;baselines for generative and extractive question answering, trained on Natural&#10;Questions, in zero shot and translation settings. Results indicate this dataset&#10;is challenging even in English, but especially in low-resource languages&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Cross-Lingual Modeling&#10;Recent work trains cross-lingual representations with unsupervised language modeling over many languages, including Multilingual BERT \cite{devlin2019bert}, XLM-R \cite{conneau2019unsupervised}, and Multilingual T5 \cite{xue2020mt5}.&#10;Transfer learning techniques are often applied to these cross-lingual representations to overcome the dearth of non-English data \cite{cui2019cross, hsu-etal-2019-zero, lee2019cross, kumar2019cross}.&#10;Recent investigations into cross-lingual modeling have revealed ``translation artifacts&quot; in datasets where machine translation systems are used, or human translation tasks are not carefully curated \cite{artetxe2020translation, wintner-2016-translationese, rabinovich2015unsupervised}.&#10;``Translationese&quot; results in hidden linguistic cues in translated text that render the task easier than a natural translation.&#10;&#10;English QA Resources&#10;A majority of question answering research focuses on English, which offers ample selection of evaluation datasets, including SQuAD \cite{rajpurkar2016squad}, TriviaQA \cite{joshi2017triviaqa}, and Natural Questions \cite{kwiatkowski2019natural}.&#10;Open Domain QA, pioneered by \cite{10.5555/21922.24354}, is the task of answering open questions using external knowledge sources.&#10;A common approach is to combine retrieval and extractive techniques \cite{chen-etal-2016-thorough, chen-etal-2017-reading, dhingra-etal-2017-gated, cui-etal-2017-attention}.&#10;&#10;Monolingual QA Resources&#10;Non-English question answering resource options remain comparatively rare, with most options spanning only one other language, and rarely low-resource languages.&#10;DuReader \cite{he2018dureader}, CMRC \cite{cui2019span}, and DRCD \cite{shao2018drcd} all offer high-quality Chinese QA datsets.&#10;Similarly, XCMRC \cite{liu2019xcmrc} and BiPar \cite{jing2019bipar} present parallel, cross-lingual QA dataset between English and Chinese.&#10;Exploring slightly less resource-rich languages, numerous works have derived new datasets from SQuAD, employing varying degrees of human or semi-automatic translation techniques to non-English target languages: ARCD for Arabic \cite{mozannar2019neural}, KorQuAD-1.0 for Korean \cite{lim2019korquad1}, and MMQA for Hindi \cite{gupta2018mmqa}.&#10;&#10;Multilingual QA Resources&#10;Table~\ref{evalset-comparison} compares the largest publicly available multilingual question answering evaluation sets.&#10;The table highlights the following properties of each dataset: whether the available gold answers are independent of retrieved documents, whether examples are aligned across languages, and the number of languages and examples provided.&#10;MLQA \cite{lewis2019mlqa} and XQuAD \cite{artetxe2019cross} are examples of SQuAD-style extractive datasets, employing human translators to create parallel examples.&#10;Both MLQA and XQuAD ensure that all answers are answerable (discarding ``No Answer'' examples), and derive answers from provided documents.&#10;XQA \cite{liu2019xqa}, one of the few retrieval-independent QA datasets, offers cloze-style questions, leveraging Wikipedia's daily questions and entity answers to populate document-independent answers.&#10;TyDi \cite{clark2020tydi}, like MKQA, focuses on typological diversity in its wide language selection.&#10;While TyDi offers a more natural distribution of questions, its annotations are based on the retrieval system used by the authors (Google search); hence their answers are actually start and end indices for spans of text within a given passage.&#10;Xor-QA \cite{asai2020xor} explores cross-lingual subtasks by re-annotating 40k TyDi examples, over 7 languages, sourcing answers from English documents and translating them back to the target language.&#10;Many of these multilingual resources have been bundled into cross-lingual benchmarks, such as XTREME \cite{hu2020xtreme} and XGLUE \cite{liang2020xglue}.&#10;&#10;" />
          <attvalue for="4" value="Question Answering, Computer Science, Language Evaluation, Linguistics, Multilingual Modeling, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2209.12786" label="2209.12786">
        <attvalues>
          <attvalue for="0" value="Do ever larger octopi still amplify reporting biases? Evidence from&#10;  judgments of typical colour" />
          <attvalue for="1" value="  Language models (LMs) trained on raw texts have no direct access to the&#10;physical world. Gordon and Van Durme (2013) point out that LMs can thus suffer&#10;from reporting bias: texts rarely report on common facts, instead focusing on&#10;the unusual aspects of a situation. If LMs are only trained on text corpora and&#10;naively memorise local co-occurrence statistics, they thus naturally would&#10;learn a biased view of the physical world. While prior studies have repeatedly&#10;verified that LMs of smaller scales (e.g., RoBERTa, GPT-2) amplify reporting&#10;bias, it remains unknown whether such trends continue when models are scaled&#10;up. We investigate reporting bias from the perspective of colour in larger&#10;language models (LLMs) such as PaLM and GPT-3. Specifically, we query LLMs for&#10;the typical colour of objects, which is one simple type of perceptually&#10;grounded physical common sense. Surprisingly, we find that LLMs significantly&#10;outperform smaller LMs in determining an object's typical colour and more&#10;closely track human judgments, instead of overfitting to surface patterns&#10;stored in texts. This suggests that very large models of language alone are&#10;able to overcome certain types of reporting bias that are characterized by&#10;local co-occurrences.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have been compared to hypothetical giant octopi living underwater that are exposed to a lot of language data \cite{bender-koller-2020-climbing}. Such octopi would struggle to understand what actually happens on land as they lack the physical perceptual experience of living there. As such, they may overfit to text-only corpora and thus amplify reporting bias \cite{gordon2013reporting} rather than faithfully reflecting the physical world.&#10;&#10;In textual corpora, humans do not tend to mention what is commonly known, instead using language to express new information, which is likely less common. For example, when describing the colour of a banana: ``green banana'' has much higher frequency than ``yellow banana'' in the Google Books corpus. It is natural to expect LMs would overfit to such reporting bias since they are trained to memorise such co-occurrence statistics. &#10;To observe this, we can query widely used pretrained models, such as RoBERTa$_{Large}$ \cite{liu2019roberta} with our previous example. Given the prompt ``It is commonly known that most bananas have the color &lt;mask&gt;'', RoBERTa ranks ``green'' the highest.&#10;This agrees with corpus statistics derived from raw text corpus such as the Google Ngram \cite{lin-etal-2012-syntactic} mentioned above. \cite{paik-etal-2021-world} test pretrained LMs' perception of colours and confirm that they perform no better than naive co-occurrence statistics extracted from the corpus. In fact, naively using corpus statistics achieves around 40\% accuracy on their proposed colour probing benchmark CoDa while the best LM performs similarly. \cite{zhang-etal-2022-visual} extend the evaluation to a broader range of visual properties, confirming that reporting bias can negatively influence model performance and increasing model size does not help. \cite{shwartz-choi-2020-neural} repeat the reporting bias experiments of \cite{gordon2013reporting} on pretrained LMs and find that LMs overestimate rare events and actions, also amplifying reporting bias.&#10;&#10;However, the LMs tested by \cite{paik-etal-2021-world,zhang-etal-2022-visual,shwartz-choi-2020-neural}, i.e., GPT-2 \cite{radford2019language}, BERT \cite{devlin-etal-2019-bert}, RoBERTa, and ALBERT \cite{Lan2020ALBERT}, usually have only several hundred million parameters and are of much smaller sizes than LLMs available now. In this work we probe T5 \cite{raffel-etal-2020-exploring}, GPT-3 \cite{brown2020language}, and PaLM \cite{chowdhery2022palm} of various sizes, with parameter counts ranging from 770M to 540B. Surprisingly, we find that LLMs almost double the performance of small language models (SLMs) on the typical colour task (\ref{fig:main_fig}). \cite{paik-etal-2021-world} point out that SLMs achieve poor performance on objects that typically only have one colour (such as bananas), possibly due to their true colour being an aspect of common sense and thus not frequently mentioned in the training corpus. We find in contrast that LLMs achieve surprisingly good performance in this category, reaching &gt;80\% accuracy. After plotting accuracy against model size, we observe that scaling up is universally helpful for improving LLMs' performance on the colour probing benchmark (CoDa; \cite{paik-etal-2021-world}). Moreover, as LLMs are scaled their correlation to corpus ngram statistics plateau, suggesting that memorising (local) co-occurrence statistics cannot explain their success.&#10;&#10;Our study presents controlled analyses on the colour prediction task as a case study to show that scaling up LLMs could overcome surface-level pattern memorisation (i.e., text reporting bias in our case) and learn physical world common sense at least to some extent. This is an important and surprising finding as it provides a key evidence to counterargue the previous consensus that despite achieving better performance for a range of NLP tasks, larger LMs are more prone to overfitting to corpus statistics and therefore amplifying the reporting bias. Our study points out that this criticism on model scale is misleading as it is not based on the complete picture, and when the model capacity is increased to a significantly large scale such as PaLM-540B and GPT-3$_{davinci}$, they start to overcome reporting bias and are able to abstract physical common sense from text.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Bias, Artificial Intelligence, Large Scale Language Models, Physical World Representation" />
        </attvalues>
      </node>
      <node id="2110.08182" label="2110.08182">
        <attvalues>
          <attvalue for="0" value="The World of an Octopus: How Reporting Bias Influences a Language&#10;  Model's Perception of Color" />
          <attvalue for="1" value="  Recent work has raised concerns about the inherent limitations of text-only&#10;pretraining. In this paper, we first demonstrate that reporting bias, the&#10;tendency of people to not state the obvious, is one of the causes of this&#10;limitation, and then investigate to what extent multimodal training can&#10;mitigate this issue. To accomplish this, we 1) generate the Color Dataset&#10;(CoDa), a dataset of human-perceived color distributions for 521 common&#10;objects; 2) use CoDa to analyze and compare the color distribution found in&#10;text, the distribution captured by language models, and a human's perception of&#10;color; and 3) investigate the performance differences between text-only and&#10;multimodal models on CoDa. Our results show that the distribution of colors&#10;that a language model recovers correlates more strongly with the inaccurate&#10;distribution found in text than with the ground-truth, supporting the claim&#10;that reporting bias negatively impacts and inherently limits text-only&#10;training. We then demonstrate that multimodal models can leverage their visual&#10;training to mitigate these effects, providing a promising avenue for future&#10;research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.10210" label="2307.10210">
        <attvalues>
          <attvalue for="0" value="Unsupervised Domain Adaptation using Lexical Transformations and Label&#10;  Injection for Twitter Data" />
          <attvalue for="1" value="  Domain adaptation is an important and widely studied problem in natural&#10;language processing. A large body of literature tries to solve this problem by&#10;adapting models trained on the source domain to the target domain. In this&#10;paper, we instead solve this problem from a dataset perspective. We modify the&#10;source domain dataset with simple lexical transformations to reduce the domain&#10;shift between the source dataset distribution and the target dataset&#10;distribution. We find that models trained on the transformed source domain&#10;dataset performs significantly better than zero-shot models. Using our proposed&#10;transformations to convert standard English to tweets, we reach an unsupervised&#10;part-of-speech (POS) tagging accuracy of 92.14% (from 81.54% zero shot&#10;accuracy), which is only slightly below the supervised performance of 94.45%.&#10;We also use our proposed transformations to synthetically generate tweets and&#10;augment the Twitter dataset to achieve state-of-the-art performance for POS&#10;tagging.&#10;" />
          <attvalue for="2" value="&#10;In a typical machine learning setting, training, development and test sets are usually carved out of the same data collection effort. In doing this, we caveat our models with an implicit assumption - the deployment dataset should belong to the same distribution as the training dataset. This is rarely the case and we see significant drops in performance when the model is deployed. The mismatch between the deployment data distribution, or target domain, and the training data distribution, or source domain, is known as domain shift \cite{ramponi2020neural, ruder2018strong} and the process of adapting to target domain distributions is known as domain adaptation \cite{blitzer2006domain}.&#10;&#10;The most widely studied domain adaptation methods are model-centric methods \cite{ramponi2020neural}, where parts of the model, including the feature space, the loss function or even the structure of the model are altered \cite{blitzer2006domain, pan2010cross, ganin2016domain, marz2019domain}. Data-centric methods \cite{ramponi2020neural} usually involve some form of bootstrapping and pseudo-labelling of the target domain data \cite{abney2007semisupervised, cui2019self, ruder2018strong, gupta2021unsupervised}. A popular data-centric domain adaptation method is data selection, which is an intermediate training step that aims to select a subset of data that is closest to the target domain \cite{moore2010intelligent, axelrod2011domain, aharoni2020unsupervised, iter2021complementarity}. We refer the reader to domain adaptation surveys in natural language processing for a detailed overview \cite{ramponi2020neural, chu2018survey, jiang2013literature, margolis2011literature}.&#10;&#10;To the best of our knowledge, none of the works we encounter in literature address the fundamental reason behind the need for domain adaptation - domain shift. If we are able to transform the source domain dataset such that the domain mismatch between the source domain and the target domain is reduced, while being able to exploit the annotations of the source domain corpus, then the models trained on such a transformed source domain data will naturally perform better on the target domain. This is the main motivation behind our work. All model-centric and data-centric domain adaptation methods can be applied on top of our proposed method and are complementary to it.&#10;&#10;In this paper, we transform the source domain dataset to resemble the target domain dataset more closely through a series of transformations. In our case, the source domain consists of standard English sentences and the target domain consists of tweets. Through these transformations, we are able to improve the zero-shot POS tagging accuracy by 10.39\% when averaged over five different BERT models. Also, when we combine the transformed data to augment the original target dataset, we achieve state-of-the-art POS tagging performance on the target dataset. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Computational Linguistics, Domain Adaptation, Dataset Transformation, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2308.14436" label="2308.14436">
        <attvalues>
          <attvalue for="0" value="Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware&#10;  Pre-training for KBQA" />
          <attvalue for="1" value="  Knowledge Base Question Answering (KBQA) aims to answer natural language&#10;questions with factual information such as entities and relations in KBs.&#10;However, traditional Pre-trained Language Models (PLMs) are directly&#10;pre-trained on large-scale natural language corpus, which poses challenges for&#10;them in understanding and representing complex subgraphs in structured KBs. To&#10;bridge the gap between texts and structured KBs, we propose a Structured&#10;Knowledge-aware Pre-training method (SKP). In the pre-training stage, we&#10;introduce two novel structured knowledge-aware tasks, guiding the model to&#10;effectively learn the implicit relationship and better representations of&#10;complex subgraphs. In downstream KBQA task, we further design an efficient&#10;linearization strategy and an interval attention mechanism, which assist the&#10;model to better encode complex subgraphs and shield the interference of&#10;irrelevant subgraphs during reasoning respectively. Detailed experiments and&#10;analyses on WebQSP verify the effectiveness of SKP, especially the significant&#10;improvement in subgraph retrieval (+4.08% H@10).&#10;" />
          <attvalue for="2" value="&#10;&#10;Knowledge Base Question Answering (KBQA) aims to seek answers to factoid questions from structured KBs \cite{auer2007dbpedia,bollacker2008freebase}. The existing methods to solve KBQA can be divided into two categories: Semantic parsing-based (SP-based) methods \cite{ye2021rng,das2022knowledge,lan2020query,sun2020sparqa} and embedding-based methods \cite{agarwal-etal-2021-knowledge,zhang2022subgraph,he2021improving,sun2020faithful,sun2018open,sun2019pullnet}. The former one heavily relies on the expensive annotation of the intermediate logic form such as SPARQL \cite{perez2009semantics}. Instead of parsing the questions, the latter one directly encodes and retrieves the candidate subgraphs, then obtains answers by a ranker or a generator \cite{saxena2022sequence,oguz2020unik,yu2022decaf}. Therefore, embedding-based framework is more suitable for the realistic dialogue system, which has gained attention in the research field.&#10;&#10;Unfortunately, the existing embedding-based methods still encoun-\\&#10;ter challenges in understanding and reasoning with structured knowledge. Traditional KBs, such as Freebase, typically comprise sets of subject-predicate-object (SPO) triplets. However, the PLMs are pre-trained on unstructured natural language texts, making it difficult to encode the implicit logical relationship in the structured subgraphs \cite{yang2020improving}. Moreover, to improve the efficiency, existing methods directly concatenate all the retrieved subgraphs during answer generation, which inevitably causes different types of subgraphs to interfere with each other in the encoding process \cite{de2022fido,yu-etal-2022-kg}.&#10;&#10;Previous works \cite{bordes2013translating,ji2015knowledge,sun2018open,sun2019pullnet,ju2022grape} primarily addresses these challenges by introducing external text corpus and specially designed framework to incorporate information from the documents. However, the required external resources may be hard to collect in practice. &#10;Besides, another branch of case-based methods \cite{das2022knowledge,ye2021rng,das2021case,yu-etal-2022-kg} integrate training cases or candidate subgraphs to prompt the model for reasoning. Nevertheless, these methods easily trigger additional noise and are limited by the data quality.&#10;Therefore, it is essential to develop a method for PLMs to improve its ability of understanding structured knowledge and anti-interference, which facilitates techniques of KBQA to be widely applied to the realistic question answering system.&#10;&#10;To address the above limitations, we propose Structured Knowledge-aware Pre-training method (SKP) to enhance the model's ability of understanding structured knowledge and encoding complex subgraphs. Inspired by masked language modeling (MLM) \cite{devlin2018bert} and contrastive learning \cite{chen2020simple}, we introduce two novel structured knowledge-aware pre-training tasks according to the characteristics of the data in subgraphs. These tasks effectively facilitate the learning of implicit relationships and representations in structured knowledge.&#10;In the downstream KBQA task, we follow the previous retriever-reader framework, including subgraphs retrieval and answer generation. What distinguishes our approach is the introduction of an effective linearization strategy that significantly reduces the number of candidate subgraphs in the KB while preserving the structural semantic information. Furthermore, we initialize the retriever with pre-trained parameters to facilitate the transfer of upstream learned structured knowledge to the downstream model. To better encode different subgraphs during reasoning, motivated by \cite{liu2020k}, we design an interval attention mechanism, effectively guiding the model to shield the interference from irrelevant subgraphs. Our contributions can be summarized as follows: &#10;&#10;(1) We propose two novel structured knowledge-aware pre-training tasks to enhance the learning of implicit relationships and representations in subgraphs for KBQA&#10;&#10;(2) We design an efficient linearization strategy and an interval attention mechanism to improve the model's ability to encode complex subgraphs and mitigate interference from irrelevant subgraphs during reasoning.&#10;&#10;(3) Experiments and analyses on WebQSP show the effectiveness of SKP, especially huge improvements in the subgraph retrieval. Our source codes and datasets are available at Github for further comparisons.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Structured Knowledge Representation, Computer Science, Knowledge Base Question Answering, Linguistics, Pre-trained Language Models, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2109.08678" label="2109.08678">
        <attvalues>
          <attvalue for="0" value="RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base&#10;  Question Answering" />
          <attvalue for="1" value="  Existing KBQA approaches, despite achieving strong performance on i.i.d. test&#10;data, often struggle in generalizing to questions involving unseen KB schema&#10;items. Prior ranking-based approaches have shown some success in&#10;generalization, but suffer from the coverage issue. We present RnG-KBQA, a&#10;Rank-and-Generate approach for KBQA, which remedies the coverage issue with a&#10;generation model while preserving a strong generalization capability. Our&#10;approach first uses a contrastive ranker to rank a set of candidate logical&#10;forms obtained by searching over the knowledge graph. It then introduces a&#10;tailored generation model conditioned on the question and the top-ranked&#10;candidates to compose the final logical form. We achieve new state-of-the-art&#10;results on GrailQA and WebQSP datasets. In particular, our method surpasses the&#10;prior state-of-the-art by a large margin on the GrailQA leaderboard. In&#10;addition, RnG-KBQA outperforms all prior approaches on the popular WebQSP&#10;benchmark, even including the ones that use the oracle entity linking. The&#10;experimental results demonstrate the effectiveness of the interplay between&#10;ranking and generation, which leads to the superior performance of our proposed&#10;approach across all settings with especially strong improvements in zero-shot&#10;generalization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.10610" label="2202.10610">
        <attvalues>
          <attvalue for="0" value="Knowledge Base Question Answering by Case-based Reasoning over Subgraphs" />
          <attvalue for="1" value="  Question answering (QA) over knowledge bases (KBs) is challenging because of&#10;the diverse, essentially unbounded, types of reasoning patterns needed.&#10;However, we hypothesize in a large KB, reasoning patterns required to answer a&#10;query type reoccur for various entities in their respective subgraph&#10;neighborhoods. Leveraging this structural similarity between local&#10;neighborhoods of different subgraphs, we introduce a semiparametric model&#10;(CBR-SUBG) with (i) a nonparametric component that for each query, dynamically&#10;retrieves other similar $k$-nearest neighbor (KNN) training queries along with&#10;query-specific subgraphs and (ii) a parametric component that is trained to&#10;identify the (latent) reasoning patterns from the subgraphs of KNN queries and&#10;then apply them to the subgraph of the target query. We also propose an&#10;adaptive subgraph collection strategy to select a query-specific compact&#10;subgraph, allowing us to scale to full Freebase KB containing billions of&#10;facts. We show that CBR-SUBG can answer queries requiring subgraph reasoning&#10;patterns and performs competitively with the best models on several KBQA&#10;benchmarks. Our subgraph collection strategy also produces more compact&#10;subgraphs (e.g. 55\% reduction in size for WebQSP while increasing answer&#10;recall by 4.85\%)\footnote{Code, model, and subgraphs are available at&#10;\url{https://github.com/rajarshd/CBR-SUBG}}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.13956" label="2003.13956">
        <attvalues>
          <attvalue for="0" value="SPARQA: Skeleton-based Semantic Parsing for Complex Questions over&#10;  Knowledge Bases" />
          <attvalue for="1" value="  Semantic parsing transforms a natural language question into a formal query&#10;over a knowledge base. Many existing methods rely on syntactic parsing like&#10;dependencies. However, the accuracy of producing such expressive formalisms is&#10;not satisfying on long complex questions. In this paper, we propose a novel&#10;skeleton grammar to represent the high-level structure of a complex question.&#10;This dedicated coarse-grained formalism with a BERT-based parsing algorithm&#10;helps to improve the accuracy of the downstream fine-grained semantic parsing.&#10;Besides, to align the structure of a question with the structure of a knowledge&#10;base, our multi-strategy method combines sentence-level and word-level&#10;semantics. Our approach shows promising performance on several datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12688" label="2010.12688">
        <attvalues>
          <attvalue for="0" value="Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced&#10;  Language Model Pre-training" />
          <attvalue for="1" value="  Prior work on Data-To-Text Generation, the task of converting knowledge graph&#10;(KG) triples into natural text, focused on domain-specific benchmark datasets.&#10;In this paper, however, we verbalize the entire English Wikidata KG, and&#10;discuss the unique challenges associated with a broad, open-domain, large-scale&#10;verbalization. We further show that verbalizing a comprehensive, encyclopedic&#10;KG like Wikidata can be used to integrate structured KGs and natural language&#10;corpora. In contrast to the many architectures that have been developed to&#10;integrate these two sources, our approach converts the KG into natural text,&#10;allowing it to be seamlessly integrated into existing language models. It&#10;carries the further advantages of improved factual accuracy and reduced&#10;toxicity in the resulting language model. We evaluate this approach by&#10;augmenting the retrieval corpus in a retrieval language model and showing&#10;significant improvements on the knowledge intensive tasks of open domain QA and&#10;the LAMA knowledge probe.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Data-to-Text Generation&#10;Data-to-Text Generation has several benchmark datasets with slightly different objectives--WebNLG \cite{gardent-etal-2017-webnlg} to convert a group of triples to text, E2ENLG \cite{duvsek2018findings} to convert database key-value pairs or pictures to text, WikiBio \cite{lebret-etal-2016-neural} for biography generation from text, \cite{wiseman-etal-2017-challenges} for text describing score statistics tables of basketball games, both ToTTo \cite{parikh2020totto} and DART \cite{radev2020dart} to generate text given a table and relevant highlighted cells. Many systems \cite{van-der-lee-etal-2018-automated, castro-ferreira-etal-2019-neural, shimorina-gardent-2018-handling} have been developed and evaluated on these datasets, such as graph transformers over structured data \cite{koncel-kedziorski-etal-2019-text}, latent templates for interpretability \cite{wiseman2018learning} and text-to-text generation with T5 \cite{kale2020text}.&#10;&#10;KG--Text alignment&#10;T-REx \cite{elsahar-etal-2018-rex} is a widely used Text--KG aligned corpus, built using systems such as coreference resolution and predicate linkers (details in \S \ref{sec:alignment}).&#10;\cite{logan-etal-2019-baracks} and \cite{chen2020kgpt} also created an aligned corpus using Wikipedia hyperlinks and coreference resolution. (details on comparison in \S \ref{sec:triple_types}). In contrast, we use alias-based heuristics coupled with source text selection constraints to generate a corpus of 16M triples aligned with 8M sentences. Lastly, open information extraction i.e.\ automatic KG construction from text \cite{etzioni2008open, angeli-etal-2015-leveraging, clancy-etal-2019-scalable} inherently create such a corpus but these works generally do not release the extracted KG triples.&#10;&#10;Incorporating KGs&#10;Most prior works on incorporating KG with text often learn KG entity representations and add them to the mention spans linked to the entity \cite{peters-etal-2019-knowledge, yu2020jaket, fevry2020entities} or create subgraphs relevant to the query that are expanded with text in the embedding space \cite{logan-etal-2019-baracks, sun-etal-2019-pullnet, xiong-etal-2019-improving}.&#10;Some others incorporate additional modules. \cite{verga2020facts} extend \cite{fevry2020entities} by adding a triple memory with (subject, relation) encoding as the key and the object encoding as the value. &#10;\cite{das2017question} use universal schema \cite{riedel-etal-2013-relation} that embeds text and KGs in a shared space for their integration. \cite{k-m-etal-2018-learning} learn a single representation for all the triples mentioned in a sentences during pre-training and update it further in task-specific finetuning.&#10;In contrast, we convert the KG into text and use it to augment the pre-training data. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2202.13296" label="2202.13296">
        <attvalues>
          <attvalue for="0" value="Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question&#10;  Answering" />
          <attvalue for="1" value="  Recent works on knowledge base question answering (KBQA) retrieve subgraphs&#10;for easier reasoning. A desired subgraph is crucial as a small one may exclude&#10;the answer but a large one might introduce more noises. However, the existing&#10;retrieval is either heuristic or interwoven with the reasoning, causing&#10;reasoning on the partial subgraphs, which increases the reasoning bias when the&#10;intermediate supervision is missing. This paper proposes a trainable subgraph&#10;retriever (SR) decoupled from the subsequent reasoning process, which enables a&#10;plug-and-play framework to enhance any subgraph-oriented KBQA model. Extensive&#10;experiments demonstrate SR achieves significantly better retrieval and QA&#10;performance than existing retrieval methods. Via weakly supervised pre-training&#10;as well as the end-to-end fine-tuning, SRl achieves new state-of-the-art&#10;performance when combined with NSM, a subgraph-oriented reasoner, for&#10;embedding-based KBQA methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.03737" label="2101.03737">
        <attvalues>
          <attvalue for="0" value="Improving Multi-hop Knowledge Base Question Answering by Learning&#10;  Intermediate Supervision Signals" />
          <attvalue for="1" value="  Multi-hop Knowledge Base Question Answering (KBQA) aims to find the answer&#10;entities that are multiple hops away in the Knowledge Base (KB) from the&#10;entities in the question. A major challenge is the lack of supervision signals&#10;at intermediate steps. Therefore, multi-hop KBQA algorithms can only receive&#10;the feedback from the final answer, which makes the learning unstable or&#10;ineffective.&#10;  To address this challenge, we propose a novel teacher-student approach for&#10;the multi-hop KBQA task. In our approach, the student network aims to find the&#10;correct answer to the query, while the teacher network tries to learn&#10;intermediate supervision signals for improving the reasoning capacity of the&#10;student network. The major novelty lies in the design of the teacher network,&#10;where we utilize both forward and backward reasoning to enhance the learning of&#10;intermediate entity distributions. By considering bidirectional reasoning, the&#10;teacher network can produce more reliable intermediate supervision signals,&#10;which can alleviate the issue of spurious reasoning. Extensive experiments on&#10;three benchmark datasets have demonstrated the effectiveness of our approach on&#10;the KBQA task. The code to reproduce our analysis is available at&#10;https://github.com/RichardHGL/WSDM2021_NSM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03658" label="2004.03658">
        <attvalues>
          <attvalue for="0" value="Faithful Embeddings for Knowledge Base Queries" />
          <attvalue for="1" value="  The deductive closure of an ideal knowledge base (KB) contains exactly the&#10;logical queries that the KB can answer. However, in practice KBs are both&#10;incomplete and over-specified, failing to answer some queries that have&#10;real-world answers. \emph{Query embedding} (QE) techniques have been recently&#10;proposed where KB entities and KB queries are represented jointly in an&#10;embedding space, supporting relaxation and generalization in KB inference.&#10;However, experiments in this paper show that QE systems may disagree with&#10;deductive reasoning on answers that do not require generalization or&#10;relaxation. We address this problem with a novel QE method that is more&#10;faithful to deductive reasoning, and show that this leads to better performance&#10;on complex queries to incomplete KBs. Finally we show that inserting this new&#10;QE module into a neural question-answering system leads to substantial&#10;improvements over the state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.00782" label="1809.00782">
        <attvalues>
          <attvalue for="0" value="Open Domain Question Answering Using Early Fusion of Knowledge Bases and&#10;  Text" />
          <attvalue for="1" value="  Open Domain Question Answering (QA) is evolving from complex pipelined&#10;systems to end-to-end deep neural networks. Specialized neural models have been&#10;developed for extracting answers from either text alone or Knowledge Bases&#10;(KBs) alone. In this paper we look at a more practical setting, namely QA over&#10;the combination of a KB and entity-linked text, which is appropriate when an&#10;incomplete KB is available with a large text corpus. Building on recent&#10;advances in graph representation learning we propose a novel model, GRAFT-Net,&#10;for extracting answers from a question-specific subgraph containing text and KB&#10;entities and relations. We construct a suite of benchmark tasks for this&#10;problem, varying the difficulty of questions, the amount of training data, and&#10;KB completeness. We show that GRAFT-Net is competitive with the&#10;state-of-the-art when tested using either KBs or text alone, and vastly&#10;outperforms existing methods in the combined setting. Source code is available&#10;at https://github.com/OceanskySun/GraftNet .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.09537" label="1904.09537">
        <attvalues>
          <attvalue for="0" value="PullNet: Open Domain Question Answering with Iterative Retrieval on&#10;  Knowledge Bases and Text" />
          <attvalue for="1" value="  We consider open-domain queston answering (QA) where answers are drawn from&#10;either a corpus, a knowledge base (KB), or a combination of both of these. We&#10;focus on a setting in which a corpus is supplemented with a large but&#10;incomplete KB, and on questions that require non-trivial (e.g., ``multi-hop'')&#10;reasoning. We describe PullNet, an integrated framework for (1) learning what&#10;to retrieve (from the KB and/or corpus) and (2) reasoning with this&#10;heterogeneous information to find the best answer. PullNet uses an {iterative}&#10;process to construct a question-specific subgraph that contains information&#10;relevant to the question. In each iteration, a graph convolutional network&#10;(graph CNN) is used to identify subgraph nodes that should be expanded using&#10;retrieval (or ``pull'') operations on the corpus and/or KB. After the subgraph&#10;is complete, a similar graph CNN is used to extract the answer from the&#10;subgraph. This retrieve-and-reason process allows us to answer multi-hop&#10;questions using large KBs and corpora. PullNet is weakly supervised, requiring&#10;question-answer pairs but not gold inference paths. Experimentally PullNet&#10;improves over the prior state-of-the art, and in the setting where a corpus is&#10;used with incomplete KB these improvements are often dramatic. PullNet is also&#10;often superior to prior systems in a KB-only setting or a text-only setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="cs/0605124" label="cs/0605124">
        <attvalues>
          <attvalue for="0" value="Semantics and Complexity of SPARQL" />
          <attvalue for="1" value="  SPARQL is the W3C candidate recommendation query language for RDF. In this&#10;paper we address systematically the formal study of SPARQL, concentrating in&#10;its graph pattern facility. We consider for this study a fragment without&#10;literals and a simple version of filters which encompasses all the main issues&#10;yet is simple to formalize. We provide a compositional semantics, prove there&#10;are normal forms, prove complexity bounds, among others that the evaluation of&#10;SPARQL patterns is PSPACE-complete, compare our semantics to an alternative&#10;operational semantics, give simple and natural conditions when both semantics&#10;coincide and discuss optimizations procedures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.10321" label="2203.10321">
        <attvalues>
          <attvalue for="0" value="Sequence-to-Sequence Knowledge Graph Completion and Question Answering" />
          <attvalue for="1" value="  Knowledge graph embedding (KGE) models represent each entity and relation of&#10;a knowledge graph (KG) with low-dimensional embedding vectors. These methods&#10;have recently been applied to KG link prediction and question answering over&#10;incomplete KGs (KGQA). KGEs typically create an embedding for each entity in&#10;the graph, which results in large model sizes on real-world graphs with&#10;millions of entities. For downstream tasks these atomic entity representations&#10;often need to be integrated into a multi stage pipeline, limiting their&#10;utility. We show that an off-the-shelf encoder-decoder Transformer model can&#10;serve as a scalable and versatile KGE model obtaining state-of-the-art results&#10;for KG link prediction and incomplete KG question answering. We achieve this by&#10;posing KG link prediction as a sequence-to-sequence task and exchange the&#10;triple scoring approach taken by prior KGE methods with autoregressive&#10;decoding. Such a simple but powerful method reduces the model size up to 98%&#10;compared to conventional KGE models while keeping inference time tractable.&#10;After finetuning this model on the task of KGQA over incomplete KGs, our&#10;approach outperforms baselines on multiple large-scale datasets without&#10;extensive hyperparameter tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.14610" label="2012.14610">
        <attvalues>
          <attvalue for="0" value="UniK-QA: Unified Representations of Structured and Unstructured&#10;  Knowledge for Open-Domain Question Answering" />
          <attvalue for="1" value="  We study open-domain question answering with structured, unstructured and&#10;semi-structured knowledge sources, including text, tables, lists and knowledge&#10;bases. Departing from prior work, we propose a unifying approach that&#10;homogenizes all sources by reducing them to text and applies the&#10;retriever-reader model which has so far been limited to text sources only. Our&#10;approach greatly improves the results on knowledge-base QA tasks by 11 points,&#10;compared to latest graph-based methods. More importantly, we demonstrate that&#10;our unified knowledge (UniK-QA) model is a simple and yet effective way to&#10;combine heterogeneous sources of knowledge, advancing the state-of-the-art&#10;results on two popular question answering benchmarks, NaturalQuestions and&#10;WebQuestions, by 3.5 and 2.6 points, respectively.&#10;  The code of UniK-QA is available at:&#10;https://github.com/facebookresearch/UniK-QA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.00063" label="2210.00063">
        <attvalues>
          <attvalue for="0" value="DecAF: Joint Decoding of Answers and Logical Forms for Question&#10;  Answering over Knowledge Bases" />
          <attvalue for="1" value="  Question answering over knowledge bases (KBs) aims to answer natural language&#10;questions with factual information such as entities and relations in KBs.&#10;Previous methods either generate logical forms that can be executed over KBs to&#10;obtain final answers or predict answers directly. Empirical results show that&#10;the former often produces more accurate answers, but it suffers from&#10;non-execution issues due to potential syntactic and semantic errors in the&#10;generated logical forms. In this work, we propose a novel framework DecAF that&#10;jointly generates both logical forms and direct answers, and then combines the&#10;merits of them to get the final answers. Moreover, different from most of the&#10;previous methods, DecAF is based on simple free-text retrieval without relying&#10;on any entity linking tools -- this simplification eases its adaptation to&#10;different datasets. DecAF achieves new state-of-the-art accuracy on WebQSP,&#10;FreebaseQA, and GrailQA benchmarks, while getting competitive results on the&#10;ComplexWebQuestions benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.08153" label="2212.08153">
        <attvalues>
          <attvalue for="0" value="FiDO: Fusion-in-Decoder optimized for stronger performance and faster&#10;  inference" />
          <attvalue for="1" value="  Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that&#10;sets the state-of-the-art on many knowledge-intensive NLP tasks. However, the&#10;architecture used for FiD was chosen by making minimal modifications to a&#10;standard T5 model, which our analysis shows to be highly suboptimal for a&#10;retrieval-augmented model. In particular, FiD allocates the bulk of FLOPs to&#10;the encoder, while the majority of inference time results from memory bandwidth&#10;constraints in the decoder. We propose two simple changes to the FiD&#10;architecture to alleviate memory bandwidth constraints, and speed up inference&#10;by 7x. This allows us to use a much larger decoder at modest cost. We denote&#10;FiD with the above modifications as FiDO, and show that it strongly improves&#10;performance over existing FiD models for a wide range of inference budgets. For&#10;example, FiDO-Large-XXL performs faster inference than FiD-Base and achieves&#10;better performance than FiD-Large.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04330" label="2110.04330">
        <attvalues>
          <attvalue for="0" value="KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain&#10;  Question Answering" />
          <attvalue for="1" value="  Current Open-Domain Question Answering (ODQA) model paradigm often contains a&#10;retrieving module and a reading module. Given an input question, the reading&#10;module predicts the answer from the relevant passages which are retrieved by&#10;the retriever. The recent proposed Fusion-in-Decoder (FiD), which is built on&#10;top of the pretrained generative model T5, achieves the state-of-the-art&#10;performance in the reading module. Although being effective, it remains&#10;constrained by inefficient attention on all retrieved passages which contain a&#10;lot of noise. In this work, we propose a novel method KG-FiD, which filters&#10;noisy passages by leveraging the structural relationship among the retrieved&#10;passages with a knowledge graph. We initiate the passage node embedding from&#10;the FiD encoder and then use graph neural network (GNN) to update the&#10;representation for reranking. To improve the efficiency, we build the GNN on&#10;top of the intermediate layer output of the FiD encoder and only pass a few top&#10;reranked passages into the higher layers of encoder and decoder for answer&#10;generation. We also apply the proposed GNN based reranking method to enhance&#10;the passage retrieval results in the retrieving module. Extensive experiments&#10;on common ODQA benchmark datasets (Natural Question and TriviaQA) demonstrate&#10;that KG-FiD can improve vanilla FiD by up to 1.5% on answer exact match score&#10;and achieve comparable performance with FiD with only 40% of computation cost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.02933" label="2210.02933">
        <attvalues>
          <attvalue for="0" value="Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question&#10;  Answering" />
          <attvalue for="1" value="  A common thread of open-domain question answering (QA) models employs a&#10;retriever-reader pipeline that first retrieves a handful of relevant passages&#10;from Wikipedia and then peruses the passages to produce an answer. However,&#10;even state-of-the-art readers fail to capture the complex relationships between&#10;entities appearing in questions and retrieved passages, leading to answers that&#10;contradict the facts. In light of this, we propose a novel knowledge Graph&#10;enhanced passage reader, namely Grape, to improve the reader performance for&#10;open-domain QA. Specifically, for each pair of question and retrieved passage,&#10;we first construct a localized bipartite graph, attributed to entity embeddings&#10;extracted from the intermediate layer of the reader model. Then, a graph neural&#10;network learns relational knowledge while fusing graph and contextual&#10;representations into the hidden states of the reader model. Experiments on&#10;three open-domain QA benchmarks show Grape can improve the state-of-the-art&#10;performance by up to 2.2 exact match score with a negligible overhead increase,&#10;with the same retriever and retrieved passages. Our code is publicly available&#10;at https://github.com/jumxglhf/GRAPE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08762" label="2104.08762">
        <attvalues>
          <attvalue for="0" value="Case-based Reasoning for Natural Language Queries over Knowledge Bases" />
          <attvalue for="1" value="  It is often challenging to solve a complex problem from scratch, but much&#10;easier if we can access other similar problems with their solutions -- a&#10;paradigm known as case-based reasoning (CBR). We propose a neuro-symbolic CBR&#10;approach (CBR-KBQA) for question answering over large knowledge bases. CBR-KBQA&#10;consists of a nonparametric memory that stores cases (question and logical&#10;forms) and a parametric model that can generate a logical form for a new&#10;question by retrieving cases that are relevant to it. On several KBQA datasets&#10;that contain complex questions, CBR-KBQA achieves competitive performance. For&#10;example, on the ComplexWebQuestions dataset, CBR-KBQA outperforms the current&#10;state of the art by 11\% on accuracy. Furthermore, we show that CBR-KBQA is&#10;capable of using new cases \emph{without} any further training: by&#10;incorporating a few human-labeled examples in the case memory, CBR-KBQA is able&#10;to successfully generate logical forms containing unseen KB entities as well as&#10;relations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.07606" label="1909.07606">
        <attvalues>
          <attvalue for="0" value="K-BERT: Enabling Language Representation with Knowledge Graph" />
          <attvalue for="1" value="  Pre-trained language representation models, such as BERT, capture a general&#10;language representation from large-scale corpora, but lack domain-specific&#10;knowledge. When reading a domain text, experts make inferences with relevant&#10;knowledge. For machines to achieve this capability, we propose a&#10;knowledge-enabled language representation model (K-BERT) with knowledge graphs&#10;(KGs), in which triples are injected into the sentences as domain knowledge.&#10;However, too much knowledge incorporation may divert the sentence from its&#10;correct meaning, which is called knowledge noise (KN) issue. To overcome KN,&#10;K-BERT introduces soft-position and visible matrix to limit the impact of&#10;knowledge. K-BERT can easily inject domain knowledge into the models by&#10;equipped with a KG without pre-training by-self because it is capable of&#10;loading model parameters from the pre-trained BERT. Our investigation reveals&#10;promising results in twelve NLP tasks. Especially in domain-specific tasks&#10;(including finance, law, and medicine), K-BERT significantly outperforms BERT,&#10;which demonstrates that K-BERT is an excellent choice for solving the&#10;knowledge-driven problems that require experts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Since Google Inc. launched BERT in 2018, many endeavors have been made for further optimization, basically focusing on the pre-training process and the encoder.&#10;&#10;In optimizing pre-training process, Baidu-ERNIE \cite{sun2019ernie} and BERT-WWM \cite{cui2019pre} adopt whole-word masking rather than single character masking for pre-training BERT in Chinese corpora. SpanBERT \cite{joshi2019spanbert} extended BERT by masking contiguous random spans and proposed a span boundary objective. RoBERTa \cite{liu2019roberta} optimized the pre-training of BERT in three ways, i.e., deleting the target of the next sentence prediction, dynamically changing the masking strategy and using more and longer sentences for training. In optimizing the encoder of BERT, XLNet \cite{yang2019xlnet} replaced the Transformer in BERT with Transformer-XL \cite{dai2019transformer} to improve its ability to process long sentences. THU-ERNIE \cite{zhang2019ernie} modified the encoder of BERT to an aggregator for the mutual integration of word and entities.&#10;&#10;While the pre-trained LR model is an emerging direction, there is little work on its fusion with KG. THU-ERNIE \cite{zhang2019ernie} is a pioneer in this direction by fusing entity information, but the relations between entities are ignored by it. COMET \cite{bosselut2019comet} employed the triples in KG as corpus to train GPT \cite{radford2018improving} for common sense learning, which is very inefficient. &#10;&#10;Before the emergence of pre-trained LR models, there were several studies that combined KG with word vectors. \cite{wang2014knowledge} (\cite{wang2014knowledge}) proposed a novel method of jointly embedding entities and words into the same continuous vector space basing on the idea of word2vec \cite{mikolov2013efficient}. \cite{toutanova2015representing} (\cite{toutanova2015representing}) proposed a model that captures the compositional structure of textual relations, and optimize entity, knowledge base, and textual relation representations in a joint manner. \cite{han2016joint} (\cite{han2016joint}) applied a convolutional neural network and a KG completion task to learn the representation of text and knowledge jointly. \cite{cao2018joint} (\cite{cao2018joint}) carried out cross-lingual representation learning for words and entities via attentive distant supervision.&#10;&#10;The major weakness of these methods is that they are still based on the idea of ``word2vec + transE&quot; \cite{bordes2013translating}, rather than the pre-trained LR model. Although they use the method of joint representation to make the vector space of entities and words closer, there are still HES problems. What's more, for KGs with millions of entities, this idea makes the entity table very large, making it unusable because it exceeds the GPU's memory size.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.02679" label="2106.02679">
        <attvalues>
          <attvalue for="0" value="Layered gradient accumulation and modular pipeline parallelism: fast and&#10;  efficient training of large language models" />
          <attvalue for="1" value="  The advent of the transformer has sparked a quick growth in the size of&#10;language models, far outpacing hardware improvements. (Dense) transformers are&#10;expected to reach the trillion-parameter scale in the near future, for which&#10;training requires thousands or even tens of thousands of GPUs. We investigate&#10;the challenges of training at this scale and beyond on commercially available&#10;hardware. In particular, we analyse the shortest possible training time for&#10;different configurations of distributed training, leveraging empirical scaling&#10;laws for language models to estimate the optimal (critical) batch size.&#10;Contrary to popular belief, we find no evidence for a memory wall, and instead&#10;argue that the real limitation -- other than the cost -- lies in the training&#10;duration.&#10;  In addition to this analysis, we introduce two new methods, \textit{layered&#10;gradient accumulation} and \textit{modular pipeline parallelism}, which&#10;together cut the shortest training time by half. The methods also reduce data&#10;movement, lowering the network requirement to a point where a fast InfiniBand&#10;connection is not necessary. This increased network efficiency also improve on&#10;the methods introduced with the ZeRO optimizer, reducing the memory usage to a&#10;tiny fraction of the available GPU memory.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large scale language models are rapidly changing the field of natural language processing. Transformers \cite{vaswani2017attention} have risen as the preferred architecture for language models, being simpler, more scalable and more performant than alternatives based on recurrent neural networks. Although transformers come in numerous flavors and variations \cite[\dots]{brown2020language,devlin2019bert,fedus2021switch,lewis2019bart,radford2019language,raffel2020exploring,yang2020xlnet}, a growing body of work suggests that the model performance is mainly driven by scale. Furthermore, few-shot learning capabilities have been observed in GPT-3 (175 billion parameters) \cite{brown2020language}. This opens the field of natural language processing to a wide range of new applications where fine-tuning is either impossible or impractical. These few-shot learning capabilities have not been observed in smaller models, so they are believed to emerge only at the scale of GPT-3, reinforcing the need for extremely large language models.&#10;&#10;While large language models present exciting new capabilities, they also pose a difficult and costly engineering challenge. For example, simply storing the training state of GPT-3 takes about 2 terabytes of memory, and storing its intermediate activations and gradients takes several more terabytes. This is dozens of times more memory than available on the largest GPU to date, the 80 GB NVIDIA A100. Besides memory, training also requires a gigantic amount of computing power. GPT-3 needs about 3600 petaflop-day to train, or 30 years on the same device.&#10;&#10;To speed up training of large models, it is necessary to parallelize the training process by distributing the load across multiple GPUs. The leading method is 3d parallelism (see for example \cite{shoeybi2020megatronlm}), which combines the three common form of parallelism: data, pipeline and tensor parallelism. Together with methods such as mixed precision training and activation checkpointing, 3d parallelism allows quickly training large models with up hundreds of billions of parameters but is not so fast beyond that scale. For example, GPT-3 was trained in a matter of days, while the trillion-parameter version of Megatron-LM would need more than three months to train \cite{narayanan2021efficient}. For larger models, training times are in the order of years or worse. &#10;&#10;Recently, several memory optimizations have been suggested for large models, aiming in particular to simplify fine-tuning. When compared to training from scratch, fine-tuning requires much less computational power and can be done with a limited number of GPUs, but doing so creates a memory bottleneck. The ZeRO family of methods addresses this bottleneck by partitioning the training state \cite{rajbhandari2020zero}, aggressively offloading memory \cite{ren2021zerooffload,rajbhandari2021zeroinfinity}, and breaking down individual operations \cite{rajbhandari2021zeroinfinity}. These methods together shatter any memory constraint.&#10;&#10;While a lot of attention has been given to the memory usage, much less effort has been dedicated to reducing the training time. To that end, we investigate various parallel training configurations and strategies for large and dense transformers, with the goal of minimizing the training time on existing hardware. We explicitly integrate the concept of critical batch size \cite{golmant2018computational,mccandlish2018empirical,shallue2019measuring} in our analysis, leveraging the empirical scaling laws found in \cite{kaplan2020scaling}. The critical batch size provides an upper bound on the efficient scaling of the batch size, and by extension dictates how many GPUs can be used for training. To our knowledge, our analysis is the first to directly integrate the critical batch size and the resulting parallelism bounds for large language models. Our analysis can be applied to a wide range of scales, from the tiniest thousand-parameter transformers up to the quadrillion parameter scale and beyond.&#10;&#10;Our analysis shows that memory is not a limiting factor even past the trillion-parameter scale. Instead, we find a computational bottleneck caused by the limitations of distributed training. Due to constraints from the critical batch size and network connectivity, 3d parallelism can efficiently use a limited number of GPUs, and this upper bound does not increase nearly as fast as the amount of computation needed to train larger models. We find a minimum training time of about two weeks for a trillion-parameter model, and this bound scales worse than linearly with respect to the model size. As a result, training times are in the order of months or years above the trillion-parameter scale. We also find that the ZeRO family of methods counter-productive for the training time, largely because of frequent data transfers in micro-batched approaches --- including pipeline parallelism --- which prevent efficient 3d parallelism.&#10;&#10;We introduce two closely related methods which significantly improve the training efficiency, while also reducing the memory usage and network requirement. The first one, layered gradient accumulation, uses a bandwidth-efficient scheduling for gradient accumulation, which makes it easier to overlap the gradient reduction with the computation. It also reconciles gradient accumulation with the methods introduced in the ZeRO optimizer, avoiding frequent data transfers when partitioning the training state or when offloading it to CPU memory.The second method, modular pipeline parallelism, uses a modular split of the layers to improve the efficiency of pipeline-parallel training by minimizing the pipeline ``bubble'', which otherwise limits the efficiency of pipeline parallelism. It also builds upon layered gradient accumulation, enabling its benefits in the pipeline-parallel case. In particular, it allows partitioning the training state in the fastest 3d parallel settings, reducing the memory usage to a minimum and preventing a trade-off between memory usage and the bubble reduction of modular pipeline parallelism. These methods together allow training at least twice as fast as previous methods, for example reducing the minimum training time to one week for a trillion-parameter model. While we focus on large transformers, layered gradient accumulation and modular pipeline parallelism are not specific to transformers or even large models. In fact, the improved communication overlap is particularly useful for smaller models, especially over slower networks. &#10;&#10;Concurrent to our work, two papers appeared that show overlap with our results. The latest version of Megatron-LM \cite{narayanan2021efficient} suggests a breakdown of the layers similar to modular pipeline parallelism, however the authors suggest a different scheduling aimed at reducing the activation memory. Our method instead focuses on a network-efficient method, which leads to an increased efficiency benefit from the new layer breakdown and when combined with a training state partition also leads to a lower memory usage. In section\ref{sec:fast_checkpoints}, we investigate disk offloading in a way similar to ZeRO-Infinity \cite{rajbhandari2021zeroinfinity}. Our methods show improved results, further reducing the requirements for offloading and enabling offload even on slow hard drives. However, we also little use for this extra space, as the memory usage tends to remain reasonable in most scenarios. Instead, we suggest leveraging the results to improve checkpointing methods.&#10;&#10;This paper is structured as follow. In section\ref{sec:overview} we describe the existing approaches and provide some additional context for the paper. In sections \ref{sec:layer_grad_accumulation} and\ref{sec:modular_pipeline} we introduce the main new methods. In section\ref{sec:methodology}, \ref{sec:trillion_parameter} and \ref{sec:scaling_analysis}, we analyze the impact of our method on training speed and memory usage, then generalize to a wide range of scales. In section\ref{sec:additional_considerations} we investigate some additional concerns that arise in realistic training scenarios. Finally, in section\ref{sec:conclusion} we discuss the implications and limitations of our results.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, High-Performance Computing, Large Language Models, Engineering, Artificial Intelligence, Mathematics, Distributed Training Challenges, Optimized Training Methods" />
        </attvalues>
      </node>
      <node id="1910.02054" label="1910.02054">
        <attvalues>
          <attvalue for="0" value="ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" />
          <attvalue for="1" value="  Large deep learning models offer significant accuracy gains, but training&#10;billions to trillions of parameters is challenging. Existing solutions such as&#10;data and model parallelisms exhibit fundamental limitations to fit these models&#10;into limited device memory, while obtaining computation, communication and&#10;development efficiency. We develop a novel solution, Zero Redundancy Optimizer&#10;(ZeRO), to optimize memory, vastly improving training speed while increasing&#10;the model size that can be efficiently trained. ZeRO eliminates memory&#10;redundancies in data- and model-parallel training while retaining low&#10;communication volume and high computational granularity, allowing us to scale&#10;the model size proportional to the number of devices with sustained high&#10;efficiency. Our analysis on memory requirements and communication volume&#10;demonstrates: ZeRO has the potential to scale beyond 1 Trillion parameters&#10;using today's hardware.&#10;  We implement and evaluate ZeRO: it trains large models of over 100B parameter&#10;with super-linear speedup on 400 GPUs, achieving throughput of 15 Petaflops.&#10;This represents an 8x increase in model size and 10x increase in achievable&#10;performance over state-of-the-art. In terms of usability, ZeRO can train large&#10;models of up to 13B parameters (e.g., larger than Megatron GPT 8.3B and T5 11B)&#10;without requiring model parallelism which is harder for scientists to apply.&#10;Last but not the least, researchers have used the system breakthroughs of ZeRO&#10;to create the world's largest language model (Turing-NLG, 17B parameters) with&#10;record breaking accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07857" label="2104.07857">
        <attvalues>
          <attvalue for="0" value="ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep&#10;  Learning" />
          <attvalue for="1" value="  In the last three years, the largest dense deep learning models have grown&#10;over 1000x to reach hundreds of billions of parameters, while the GPU memory&#10;has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has&#10;been supported primarily though system innovations that allow large models to&#10;fit in the aggregate GPU memory of multiple GPUs. However, we are getting close&#10;to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion&#10;parameter model for training, and such clusters are simply out of reach for&#10;most data scientists. In addition, training models at that scale requires&#10;complex combinations of parallelism techniques that puts a big burden on the&#10;data scientists to refactor their model.&#10;  In this paper we present ZeRO-Infinity, a novel heterogeneous system&#10;technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented&#10;model scale on limited resources without requiring model code refactoring. At&#10;the same time it achieves excellent training throughput and scalability,&#10;unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models&#10;with tens and even hundreds of trillions of parameters for training on current&#10;generation GPU clusters. It can be used to fine-tune trillion parameter models&#10;on a single NVIDIA DGX-2 node, making large models more accessible. In terms of&#10;training throughput and scalability, it sustains over 25 petaflops on 512&#10;NVIDIA V100 GPUs(40% of peak), while also demonstrating super linear&#10;scalability. An open source implementation of ZeRO-Infinity is available&#10;through DeepSpeed, a deep learning optimization library that makes distributed&#10;training easy, efficient, and effective.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.12941" label="1811.12941">
        <attvalues>
          <attvalue for="0" value="On the Computational Inefficiency of Large Batch Sizes for Stochastic&#10;  Gradient Descent" />
          <attvalue for="1" value="  Increasing the mini-batch size for stochastic gradient descent offers&#10;significant opportunities to reduce wall-clock training time, but there are a&#10;variety of theoretical and systems challenges that impede the widespread&#10;success of this technique. We investigate these issues, with an emphasis on&#10;time to convergence and total computational cost, through an extensive&#10;empirical analysis of network training across several architectures and problem&#10;domains, including image classification, image segmentation, and language&#10;modeling. Although it is common practice to increase the batch size in order to&#10;fully exploit available computational resources, we find a substantially more&#10;nuanced picture. Our main finding is that across a wide range of network&#10;architectures and problem domains, increasing the batch size beyond a certain&#10;point yields no decrease in wall-clock time to convergence for \emph{either}&#10;train or test loss. This batch size is usually substantially below the capacity&#10;of current systems. We show that popular training strategies for large batch&#10;size optimization begin to fail before we can populate all available compute&#10;resources, and we show that the point at which these methods break down depends&#10;more on attributes like model architecture and data complexity than it does&#10;directly on the size of the dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.06162" label="1812.06162">
        <attvalues>
          <attvalue for="0" value="An Empirical Model of Large-Batch Training" />
          <attvalue for="1" value="  In an increasing number of domains it has been demonstrated that deep&#10;learning models can be trained using relatively large batch sizes without&#10;sacrificing data efficiency. However the limits of this massive data&#10;parallelism seem to differ from domain to domain, ranging from batches of tens&#10;of thousands in ImageNet to batches of millions in RL agents that play the game&#10;Dota 2. To our knowledge there is limited conceptual understanding of why these&#10;limits to batch size differ or how we might choose the correct batch size in a&#10;new domain. In this paper, we demonstrate that a simple and easy-to-measure&#10;statistic called the gradient noise scale predicts the largest useful batch&#10;size across many domains and applications, including a number of supervised&#10;learning datasets (MNIST, SVHN, CIFAR-10, ImageNet, Billion Word),&#10;reinforcement learning domains (Atari and Dota), and even generative model&#10;training (autoencoders on SVHN). We find that the noise scale increases as the&#10;loss decreases over a training run and depends on the model size primarily&#10;through improved model performance. Our empirically-motivated theory also&#10;describes the tradeoff between compute-efficiency and time-efficiency, and&#10;provides a rough model of the benefits of adaptive batch-size training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.08676" label="1908.08676">
        <attvalues>
          <attvalue for="0" value="Hierarchically-Refined Label Attention Network for Sequence Labeling" />
          <attvalue for="1" value="  CRF has been used as a powerful model for statistical sequence labeling. For&#10;neural sequence labeling, however, BiLSTM-CRF does not always lead to better&#10;results compared with BiLSTM-softmax local classification. This can be because&#10;the simple Markov label transition model of CRF does not give much information&#10;gain over strong neural encoding. For better representing label sequences, we&#10;investigate a hierarchically-refined label attention network, which explicitly&#10;leverages label embeddings and captures potential long-term label dependency by&#10;giving each word incrementally refined label distributions with hierarchical&#10;attention. Results on POS tagging, NER and CCG supertagging show that the&#10;proposed model not only improves the overall tagging accuracy with similar&#10;number of parameters, but also significantly speeds up the training and testing&#10;compared to BiLSTM-CRF.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;{Neural Attention.} Attention has been shown useful in neural machine translation \cite{bahdanau2014neural}, sentiment classification \cite{D17-1047,E17-2091}, relation classification \cite{P16-2034}, read comprehension \cite{Hermann:2015:TMR:2969239.2969428}, sentence summarization \cite{D15-1044}, parsing \cite{D16-1035}, question answering \cite{C18-1215} and text understanding \cite{P16-1086}. Self-attention network (SAN) \cite{NIPS2017_7181} has been used for semantic role labeling \cite{D18-1548}, text classification \cite{P18-2123,D18-1408} and other tasks. &#10;Our work is similar to \cite{NIPS2017_7181} in the sense that we also build a hierarchical attentive neural network for sequence representation. The difference lies in that our main goal is to investigate the encoding of exponential label sequences, whereas their work focuses on encoding of a word sequence only.&#10;&#10;{Label Embeddings.} Label embedding was first used in the field of computer vision for facilitating zero-shot learning \cite{PalatucciNIPS2009,Socher:2013:ZLT:2999611.2999716,7781018}. The basic idea is to improve the performance of classifying previously unseen class instances by learning output label knowledge.&#10;In NLP, label embeddings have been exploited for better text classification \cite{tang2015pte,nam16allintext,wang2018joint}. However, relatively little work has been done investigating label embeddings for sequence labeling. One exception is \newcite{N16-1027}, who use supertag embeddings in the output layer of a CCG supertagger, through a combination of local classification model rescored using a supertag language model. In contrast, we model deep label interactions using a dynamically refined sequence representation network. To our knowledge, we are the first to investigate a hierarchical attention network over a label space.&#10;&#10;{Neural CRF}. There has been methods that aim to speed up neural CRF \cite{tu-18}, and to solve the Markov constraint of neural CRF. In particular, \cite{ijcai2018-637} predicts a sequence of labels as a sequence to sequence problem; \cite{guo-etal-2019-star} further integrates global input information in encoding. Capturing non-local dependencies between labels, these methods, however, are slower compared with CRF. In contrast to these lines of work, our method is both asymptotically faster and empirically more accurate compared with neural CRF.&#10; " />
          <attvalue for="4" value="Computer Science, Linguistics, Sequence Labeling Models, Neural Network Architectures, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2402.13446" label="2402.13446">
        <attvalues>
          <attvalue for="0" value="Large Language Models for Data Annotation: A Survey" />
          <attvalue for="1" value="  Data annotation generally refers to the labeling or generating of raw data&#10;with relevant information, which could be used for improving the efficacy of&#10;machine learning models. The process, however, is labor-intensive and costly.&#10;The emergence of advanced Large Language Models (LLMs), exemplified by GPT-4,&#10;presents an unprecedented opportunity to automate the complicated process of&#10;data annotation. While existing surveys have extensively covered LLM&#10;architecture, training, and general applications, we uniquely focus on their&#10;specific utility for data annotation. This survey contributes to three core&#10;aspects: LLM-Based Annotation Generation, LLM-Generated Annotations Assessment,&#10;and LLM-Generated Annotations Utilization. Furthermore, this survey includes an&#10;in-depth taxonomy of data types that LLMs can annotate, a comprehensive review&#10;of learning strategies for models utilizing LLM-generated annotations, and a&#10;detailed discussion of the primary challenges and limitations associated with&#10;using LLMs for data annotation. Serving as a key guide, this survey aims to&#10;assist researchers and practitioners in exploring the potential of the latest&#10;LLMs for data annotation, thereby fostering future advancements in this&#10;critical field.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the complex realm of machine learning and natural language processing (NLP), data annotation and synthesis stand out as a critical yet challenging task, extending beyond simple label attachment to encompass a diverse array of fundamental or auxiliary information. This detailed process typically involves \ding{182} categorizing raw data with class or task labels for basic classification, \ding{183} adding intermediate labels for contextual depth~\cite{yu2022generate}, \ding{184} assigning confidence scores to assess annotation reliability~\cite{lin2022teaching}, \ding{185} applying alignment or preference labels to tailor outputs to specific criteria or user needs, \ding{186} annotating entity relationships to understand how entities within a dataset interact with each other~\cite{wadhwa2023revisiting}, \ding{187} marking semantic roles to define the underlying roles that entities play in a sentence~\cite{larionov2019semantic}, \ding{188} tagging temporal sequences to capture the order of events or actions~\cite{yu2023temporal}, or \ding{189} Synthesize data in the format of instruction~\cite{wang2022self}, response~\cite{zhang2023self}, reasoning~\cite{wang2022pinto}, pairwise~\cite{bai2022constitutional} and textual feedback~\cite{pan2024automatically} to for language model tuning.&#10;&#10;Despite its wide applications, data annotation and synthesis poses significant challenges for current machine learning models due to the complexity, subjectivity, and diversity of data~\cite{yang2023new}. This process requires domain expertise and is resource-intensive, particularly when manually labeling or creating large datasets. Advanced LLMs such as GPT-4~\cite{openai2023gpt4}, Gemini~\cite{team2023gemini}, and LLaMA-2~\cite{touvron2023llama2} offer a promising opportunity to revolutionize data annotation. LLMs serve as more than just tools but play a crucial role in improving the effectiveness and precision of data annotation. Their ability to automate annotation tasks, ensure consistency across large volumes of data~\cite{hou2023large}, and adapt through fine-tuning or prompting for specific domains~\cite{song2023preference,Zhang2024BalancingSA}, significantly mitigates the challenges encountered with traditional annotation and synthesis methods, setting a new standard for what is achievable in the realm of NLP.&#10;This survey delves into the nuances of using LLMs for data annotation and synthesis, exploring methodologies, utilizing strategies, and associated challenges in this transformative approach. Through this exploration, we aim to shed light on the motivations behind embracing LLMs as catalysts for redefining the landscape of data annotation and synthesis in machine learning and NLP.&#10;We explore the utilization of LLMs for annotation synthesis in this survey, making four main contributions:&#10;\begin{itemize}[leftmargin=*]&#10; \item LLM-Based Annotation Generation: We dive into the process of synthesizing annotations for various data types, including instruction \&amp; response, rationale, pairwise feedback, textual feedback, and other domain-specific data. Additionally, we discuss the criteria (e.g., diversity and quality) in the annotation process.&#10; \item Assessing LLM-Generated Annotations: We explore various methods for assessing the quality of annotations and strategies for selecting high-quality annotations from numerous options.&#10; \item LLM-Generated Annotations Utilization: We investigate the methodologies at different stages, including supervised fine-tuning, alignment tuning, and inference time, to train machine learning models based on LLM-generated annotations.&#10; \item Social Impact and Future Work: We discuss issues ranging from ethical dilemmas, such as bias and implications, to technical limitations, including hallucination and efficiency in LLM-generated annotations.&#10;\end{itemize}&#10;Focusing on this underrepresented aspect of LLM application, the survey aims to serve as a valuable guide for academics and practitioners who intend to deploy LLMs for annotation purposes. Note that in this survey, we primarily focus on pure language models and do not extensively cover recently emerging multimodal LLMs, such as LLaVA~\cite{liu2023visual}. Figure~\ref{fig:intro} illustrates the general structure of this survey. Additionally, a list of potential tools for utilizing LLMs for annotation is included in Appendix~\ref{app:tool}, along with explanatory examples.&#10;&#10;Differences from Other LLM-related Surveys. While existing surveys in the NLP domain extensively cover architectural nuances~\cite{zhao2023survey}, training methodologies~\cite{liu2023trustworthy}, and evaluation protocols~\cite{chang2023survey} associated with LLMs, their main focus lies on the capabilities of models for specific end tasks such as machine translation~\cite{min2021recent}, alignment~\cite{wang2023aligning}, code generation~\cite{zan2023large}, and medical analysis~\cite{thirunavukarasu2023large}. In contrast, this survey distinguishes itself by focusing primarily on the application of these potent next-generation LLMs to the intricate realm of annotation synthesis, a domain that is crucial yet underexplored.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Data Annotation, Computer Science, Large Language Models, Machine Learning, Linguistics, Data Science, Artificial Intelligence, Information Science" />
        </attvalues>
      </node>
      <node id="2305.05003" label="2305.05003">
        <attvalues>
          <attvalue for="0" value="Revisiting Relation Extraction in the era of Large Language Models" />
          <attvalue for="1" value="  Relation extraction (RE) is the core NLP task of inferring semantic&#10;relationships between entities from text. Standard supervised RE techniques&#10;entail training modules to tag tokens comprising entity spans and then predict&#10;the relationship between them. Recent work has instead treated the problem as a&#10;\emph{sequence-to-sequence} task, linearizing relations between entities as&#10;target strings to be generated conditioned on the input. Here we push the&#10;limits of this approach, using larger language models (GPT-3 and Flan-T5 large)&#10;than considered in prior work and evaluating their performance on standard RE&#10;tasks under varying levels of supervision. We address issues inherent to&#10;evaluating generative approaches to RE by doing human evaluations, in lieu of&#10;relying on exact matching. Under this refined evaluation, we find that: (1)&#10;Few-shot prompting with GPT-3 achieves near SOTA performance, i.e., roughly&#10;equivalent to existing fully supervised models; (2) Flan-T5 is not as capable&#10;in the few-shot setting, but supervising and fine-tuning it with&#10;Chain-of-Thought (CoT) style explanations (generated via GPT-3) yields SOTA&#10;results. We release this model as a new baseline for RE tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11952" label="2305.11952">
        <attvalues>
          <attvalue for="0" value="Self-QA: Unsupervised Knowledge Guided Language Model Alignment" />
          <attvalue for="1" value="  Large-scale language models like ChatGPT and GPT-4 have gained attention for&#10;their impressive conversational and generative capabilities. However, the&#10;creation of supervised paired question-answering data for instruction tuning&#10;presents formidable challenges. This endeavor necessitates substantial human&#10;effort for data annotation and wrestles with issues concerning data quality,&#10;diversity, accuracy, and other related factors. To overcome these obstacles, we&#10;introduce an innovative framework named Self-QA, which replaces the traditional&#10;practice of human-written instruction seeds with a vast amount of unsupervised&#10;knowledge, enabling the model to generate a larger quantity of correct and&#10;domain-specific instruction data. The effectiveness of our proposed method is&#10;demonstrated through experiments conducted on unsupervised corpora from various&#10;domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.06498" label="2310.06498">
        <attvalues>
          <attvalue for="0" value="A New Benchmark and Reverse Validation Method for Passage-level&#10;  Hallucination Detection" />
          <attvalue for="1" value="  Large Language Models (LLMs) have shown their ability to collaborate&#10;effectively with humans in real-world scenarios. However, LLMs are apt to&#10;generate hallucinations, i.e., makeup incorrect text and unverified&#10;information, which can cause significant damage when deployed for&#10;mission-critical tasks. In this paper, we propose a self-check approach based&#10;on reverse validation to detect factual errors automatically in a zero-resource&#10;fashion. To facilitate future studies and assess different methods, we&#10;construct a hallucination detection benchmark named PHD, which is generated by&#10;ChatGPT and annotated by human annotators. Contrasting previous studies of&#10;zero-resource hallucination detection, our method and benchmark concentrate on&#10;passage-level detection instead of sentence-level. We empirically evaluate our&#10;method and existing zero-resource detection methods on two datasets. The&#10;experimental results demonstrate that the proposed method considerably&#10;outperforms the baselines while costing fewer tokens and less time.&#10;Furthermore, we manually analyze some hallucination cases that LLM failed to&#10;capture, revealing the shared limitation of zero-resource methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.10306" label="2404.10306">
        <attvalues>
          <attvalue for="0" value="Balancing Speciality and Versatility: a Coarse to Fine Framework for&#10;  Supervised Fine-tuning Large Language Model" />
          <attvalue for="1" value="  Aligned Large Language Models (LLMs) showcase remarkable versatility, capable&#10;of handling diverse real-world tasks. Meanwhile, aligned LLMs are also expected&#10;to exhibit speciality, excelling in specific applications. However, fine-tuning&#10;with extra data, a common practice to gain speciality, often leads to&#10;catastrophic forgetting (CF) of previously acquired versatility, hindering the&#10;model's performance across diverse tasks. In response to this challenge, we&#10;propose CoFiTune, a coarse to fine framework in an attempt to strike the&#10;balance between speciality and versatility. At the coarse-grained level, an&#10;empirical tree-search algorithm is utilized to pinpoint and update specific&#10;modules that are crucial for speciality, while keeping other parameters frozen;&#10;at the fine-grained level, a soft-masking mechanism regulates the update to the&#10;LLMs, mitigating the CF issue without harming speciality. In an overall&#10;evaluation of both speciality and versatility, CoFiTune consistently&#10;outperforms baseline methods across diverse tasks and model scales. Compared to&#10;the full-parameter SFT, CoFiTune leads to about 14% versatility improvement and&#10;marginal speciality loss on a 13B model. Lastly, based on further analysis, we&#10;provide a speculative insight into the information forwarding process in LLMs,&#10;which helps explain the effectiveness of the proposed method. The code is&#10;available at https://github.com/rattlesnakey/CoFiTune.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.09420" label="2212.09420">
        <attvalues>
          <attvalue for="0" value="Large Language Models Meet NL2Code: A Survey" />
          <attvalue for="1" value="  The task of generating code from a natural language description, or NL2Code,&#10;is considered a pressing and significant challenge in code intelligence. Thanks&#10;to the rapid development of pre-training techniques, surging large language&#10;models are being proposed for code, sparking the advances in NL2Code. To&#10;facilitate further research and applications in this field, in this paper, we&#10;present a comprehensive survey of 27 existing large language models for&#10;NL2Code, and also review benchmarks and metrics. We provide an intuitive&#10;comparison of all existing models on the HumanEval benchmark. Through in-depth&#10;observation and analysis, we provide some insights and conclude that the key&#10;factors contributing to the success of large language models for NL2Code are&#10;&quot;Large Size, Premium Data, Expert Tuning&quot;. In addition, we discuss challenges&#10;and opportunities regarding the gap between models and humans. We also create a&#10;website https://nl2code.github.io to track the latest progress through&#10;crowd-sourcing. To the best of our knowledge, this is the first survey of large&#10;language models for NL2Code, and we believe it will contribute to the ongoing&#10;development of the field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.07405" label="2312.07405">
        <attvalues>
          <attvalue for="0" value="ICL Markup: Structuring In-Context Learning using Soft-Token Tags" />
          <attvalue for="1" value="  Large pretrained language models (LLMs) can be rapidly adapted to a wide&#10;variety of tasks via a text-to-text approach, where the instruction and input&#10;are fed to the model in natural language. Combined with in-context learning&#10;(ICL), this paradigm is impressively flexible and powerful. However, it also&#10;burdens users with an overwhelming number of choices, many of them arbitrary.&#10;Inspired by markup languages like HTML, we contribute a method of using&#10;soft-token tags to compose prompt templates. This approach reduces arbitrary&#10;decisions and streamlines the application of ICL. Our method is a form of&#10;meta-learning for ICL; it learns these tags in advance during a&#10;parameter-efficient fine-tuning ``warm-up'' process. The tags can subsequently&#10;be used in templates for ICL on new, unseen tasks without any additional&#10;fine-tuning. Our experiments with this approach yield promising initial&#10;results, improving LLM performance on important enterprise applications such as&#10;few-shot and open-world intent detection, as well as text classification in&#10;news and legal domains.&#10;" />
          <attvalue for="2" value="&#10;With the growing size and capabilities of large pretrained language models (LLMs),&#10;in-context learning (ICL) has become a popular way to harness their power for new tasks.&#10;ICL is an approach to prompting LLMs &#10;which includes demonstrations of how to complete the target task in the &#10;prompt~\cite{Dong2022ALearning}.&#10;It has significant advantages over traditional fine-tuning, being data-efficient, highly flexible, and user-friendly. &#10;A LLM can be adapted to perform effectively on a new task&#10;with only a handful of demonstrations (few-shot) and some natural language instructions.&#10;This can be done quickly even by someone with little knowledge of machine learning. &#10;The LLM can also be encapsulated as a black box and shared across tasks. &#10;This allows individuals and organizations to leverage LLMs for new tasks,&#10;even if they do not have the computing resources necessary to fine-tune or even host such large models.&#10;&#10;However, ICL also has several disadvantages.&#10;Most LLMs have not been explicitly trained or tuned to perform ICL,&#10;and thus have not actually been optimized to approach new tasks in this format~\cite{Dong2022ALearning}. &#10;Like other forms of prompt engineering, &#10;ICL suffers from a lack of robustness across the many arbitrary choices that users encounter in the process of setting it up~\cite{Chen2022OnLearning}.&#10;It has been shown that the performance of in-context learning can vary dramatically&#10;based on changes to the prompt~\cite{Zhao2021CalibrateModels}.&#10;There is also evidence to suggest that ICL performs poorly when shown &#10;a &quot;none of the above&quot; option~\cite{Kadavath2022LanguageKnow}, which could hinder its application in practical settings (e.g.\ open world classification)&#10;where the inputs may not always correspond with any option in the label space.&#10;&#10;We propose addressing these shortcomings with an approach to ICL &#10;inspired by markup languages like HTML.&#10;In this paradigm, we structure ICL prompt templates&#10;using a dedicated set of soft-token&#10;tags that we add to the model's vocabulary. &#10;These soft-tokens (a.k.a. tunable tokens) are effectively ``new words'':&#10;bound to trainable parameters and processed like other tokens.&#10;Their weights are learned in advance during a ``warm-up'' stage (parameter-efficient fine-tuning).&#10;They can then be used in the ICL template for new tasks without additional fine-tuning,&#10;and can thus also be shared across tasks.&#10;The training process is therefore a form of meta-learning for ICL.&#10;We show that this approach removes several arbitrary decisions from the design of ICL prompt templates.&#10;We also provide initial empirical evidence that it can improve a model's ICL ability on new tasks.&#10;Specifically, we show that ICL Markup improves Flan-T5 models on text classification tasks&#10;(intent detection, news and legal domains).&#10;We show that ICL Markup can reduce Flan-T5's &#10;performance variability when compared to prompt engineering,&#10;increase classification accuracy,&#10;as well as improve out-of-scope intent detection&#10;when the template includes a ``none of the above'' multiple-choice option.&#10;&#10;In our experiments on a few-shot news headline classification dataset,&#10;we find that Flan-T5 can be very sensitive to small changes in the prompt. &#10;For example, switching the word used to indicate the start of a demonstration in our prompt template&#10;from &quot;Headline&quot; to &quot;Input&quot; can impact the accuracy by up to 21 percentage points (p.p.).&#10;A small syntactic choice like using &quot;)&quot; vs &quot;:&quot; to separate keywords from their associated values &#10;can impact accuracy by up to 24 p.p.,&#10;with the direction of the effect depending on the model size. &#10;We also find that when compared to a search over 96 prompts,&#10;we are able to increase Flan-T5-XL's accuracy on this dataset&#10;from a mean of 68.9\&#10;(or 70.9\% using the best prompt in our search)&#10;to a mean of 76.8\% with ICL Markup.&#10;This tops the previous best reported results on the dataset, &#10;Prompt-Based Meta-Learning ~\cite{Zhang2022Prompt-BasedClassification},&#10;by 5.2 p.p.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Language Model Adaptation, Meta-Learning Techniques" />
        </attvalues>
      </node>
      <node id="2204.08167" label="2204.08167">
        <attvalues>
          <attvalue for="0" value="A Study on Prompt-based Few-Shot Learning Methods for Belief State&#10;  Tracking in Task-oriented Dialog Systems" />
          <attvalue for="1" value="  We tackle the Dialogue Belief State Tracking(DST) problem of task-oriented&#10;conversational systems. Recent approaches to this problem leveraging&#10;Transformer-based models have yielded great results. However, training these&#10;models is expensive, both in terms of computational resources and time.&#10;Additionally, collecting high quality annotated dialogue datasets remains a&#10;challenge for researchers because of the extensive annotation required for&#10;training these models. Driven by the recent success of pre-trained language&#10;models and prompt-based learning, we explore prompt-based few-shot learning for&#10;Dialogue Belief State Tracking. We formulate the DST problem as a 2-stage&#10;prompt-based language modelling task and train language models for both tasks&#10;and present a comprehensive empirical analysis of their separate and joint&#10;performance. We demonstrate the potential of prompt-based methods in few-shot&#10;learning for DST and provide directions for future improvement.&#10;" />
          <attvalue for="2" value="&#10;&#10;Dialogue Belief State Tracking is a central problem for task-based conversational systems. The Belief State maintains a distribution of states across different dialogue turns summarising the conversation state at any point by extracting the intent from the user and system inputs. The belief state is used by the system to take appropriate actions at each turn until the conversation is concluded and the user goal is achieved. &#10;&#10;Recent State-of-the-art models that tackle the Belief State Tracking problem are generally based on large language models \cite{hosseini2020simple,heck2020trippy,wu2019transferable}. Their training usually involves huge computation and data requirements, one or both of which might be unavailable. The development of models like BERT \cite{devlin2018bert} and GPT-2 \cite{Radford2019LanguageMA} has also inspired advances in the use of pre-trained language models (PLMs) for low-resource few-shot learning for dialog generation \cite{Zhao2020Low-Resource}. The recent paradigm of prompt-based learning \cite{brown2020language} equips PLMs with constructive task-dependent prompts to simplify language generation for downstream tasks. This method has shown great results on few-shot and zero-shot learning tasks such as classification \cite{gao-etal-2021-making,schick-schutze-2021-exploiting,han2021ptr} and text generation \cite{schick2020few,li-liang-2021-prefix}. Relatively fewer attempts have been made towards few-shot learning for the DST task of dialog systems. \newcite{dingliwal2021few} presents a few-shot meta-learning approach to DST. \newcite{Peng2020SOLOISTFT, madotto2021few} show few-shot DST performance on just single domain subsets of data. \newcite{Madotto2020LanguageMA} shows few-shot training results on the speech ACT(Active Intent) identification task of task-oriented datasets. However, none of the papers present baselines on the end-to-end multi-domain, multi-slot belief-state tracking. &#10;&#10;In this paper, we make the first step towards evaluating prompt-based few-shot learning for the end-to-end dialogue to belief state prediction task. Specifically, we formulate the DST task as a two stage language generation problem and provide few-shot performance using pretrained language models like GPT-2, BERT and T5. Our analysis questions the viability of tackling DST in a prompt-based few-shot setting.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Dialogue State Tracking, Computer Science, Linguistics, Cognitive Science, Language Model Training, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2005.02877" label="2005.02877">
        <attvalues>
          <attvalue for="0" value="TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State&#10;  Tracking" />
          <attvalue for="1" value="  Task-oriented dialog systems rely on dialog state tracking (DST) to monitor&#10;the user's goal during the course of an interaction. Multi-domain and&#10;open-vocabulary settings complicate the task considerably and demand scalable&#10;solutions. In this paper we present a new approach to DST which makes use of&#10;various copy mechanisms to fill slots with values. Our model has no need to&#10;maintain a list of candidate values. Instead, all values are extracted from the&#10;dialog context on-the-fly. A slot is filled by one of three copy mechanisms:&#10;(1) Span prediction may extract values directly from the user input; (2) a&#10;value may be copied from a system inform memory that keeps track of the&#10;system's inform operations; (3) a value may be copied over from a different&#10;slot that is already contained in the dialog state to resolve coreferences&#10;within and across domains. Our approach combines the advantages of span-based&#10;slot filling methods with memory methods to avoid the use of value picklists&#10;altogether. We argue that our strategy simplifies the DST task while at the&#10;same time achieving state of the art performance on various popular evaluation&#10;sets including Multiwoz 2.1, where we achieve a joint goal accuracy beyond 55%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.08743" label="1905.08743">
        <attvalues>
          <attvalue for="0" value="Transferable Multi-Domain State Generator for Task-Oriented Dialogue&#10;  Systems" />
          <attvalue for="1" value="  Over-dependence on domain ontology and lack of knowledge sharing across&#10;domains are two practical and yet less studied problems of dialogue state&#10;tracking. Existing approaches generally fall short in tracking unknown slot&#10;values during inference and often have difficulties in adapting to new domains.&#10;In this paper, we propose a Transferable Dialogue State Generator (TRADE) that&#10;generates dialogue states from utterances using a copy mechanism, facilitating&#10;knowledge transfer when predicting (domain, slot, value) triplets not&#10;encountered during training. Our model is composed of an utterance encoder, a&#10;slot gate, and a state generator, which are shared across domains. Empirical&#10;results demonstrate that TRADE achieves state-of-the-art joint goal accuracy of&#10;48.62% for the five domains of MultiWOZ, a human-human dialogue dataset. In&#10;addition, we show its transferring ability by simulating zero-shot and few-shot&#10;dialogue state tracking for unseen domains. TRADE achieves 60.58% joint goal&#10;accuracy in one of the zero-shot domains, and is able to adapt to few-shot&#10;cases without forgetting already trained domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.09676" label="2206.09676">
        <attvalues>
          <attvalue for="0" value="Studying the role of named entities for content preservation in text&#10;  style transfer" />
          <attvalue for="1" value="  Text style transfer techniques are gaining popularity in Natural Language&#10;Processing, finding various applications such as text detoxification,&#10;sentiment, or formality transfer. However, the majority of the existing&#10;approaches were tested on such domains as online communications on public&#10;platforms, music, or entertainment yet none of them were applied to the domains&#10;which are typical for task-oriented production systems, such as personal plans&#10;arrangements (e.g. booking of flights or reserving a table in a restaurant). We&#10;fill this gap by studying formality transfer in this domain.&#10;  We noted that the texts in this domain are full of named entities, which are&#10;very important for keeping the original sense of the text. Indeed, if for&#10;example, someone communicates the destination city of a flight it must not be&#10;altered. Thus, we concentrate on the role of named entities in content&#10;preservation for formality text style transfer.&#10;  We collect a new dataset for the evaluation of content similarity measures in&#10;text style transfer. It is taken from a corpus of task-oriented dialogues and&#10;contains many important entities related to realistic requests that make this&#10;dataset particularly useful for testing style transfer models before using them&#10;in production. Besides, we perform an error analysis of a pre-trained formality&#10;transfer model and introduce a simple technique to use information about named&#10;entities to enhance the performance of baseline content similarity measures&#10;used in text style transfer.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text style transfer (TST) systems are designed to change the style of the original text to alternative one, such as more informal~\cite{rao-tetreault-2018-dear}, more positive~\cite{luo-etal-2019-towards}, or even more Shakespearean~\cite{jhamtani-etal-2017-shakespearizing}. Such systems have gained significant popularity in the NLP within the last few years. They could be applied to many purposes: from diversifying responses of dialogue agents to creating artificial personalities.&#10;&#10;More formally, TST system is a function $\alpha:~S~\times~S~\times~D~arrow~D$ that, given a source style $s^{src}$, a target style $s^{dst}$, and an input text $d^{src}$, produces an output text $d^{dst}$ such that:&#10;\begin{itemize}&#10;&#10;\item The style of the text changes from the source style $s^{src}$ to the target style $s^{dst}: \sigma(d^{src}) \neq \sigma(d^{dst})$, $\sigma(d^{dst}) = s^{dst}$;&#10;&#10;\item The content of the source text is saved in the target text as much as required for the task: $\delta(d^{src}, d^{dst}) \geq t^{\delta}$;&#10;\item The fluency of the target text achieves the required level: $\psi(d^{dst}) \geq t^{\psi}$,&#10;&#10;\end{itemize}&#10;&#10;where $t^{\delta}$ and $t^{\psi}$ are task-specific thresholds for the content preservation ($\delta$) and fluency ($\psi$) functions. &#10;&#10;To measure if the content of the source text $d^{src}$ is preserved in the target text $d^{dst}$ a content similarity measure is used. This is a specific similarity measure $sim$ which quantifies semantic&#10;relatedness of $d^{src}$ and $d^{dst}$ : $sim(d^{src}, d^{dst})$. The measure $sim$ yields high score for the pairs with similar content and low score for ones with different content.&#10;&#10;In the majority of recent TST papers~\cite{DBLP:conf/sigir/RaneDLE21,cao-etal-2020-expertise,riley-etal-2021-textsettr}BLEU~\cite{papineni2002bleu} is still the main way to evaluate the content similarity. More recent approaches as cosine similarity calculation between averaged word vectors~\cite{pang-gimpel-2019-unsupervised}, BLEURT~\cite{sellam-etal-2020-bleurt} (which is a BERT~\cite{devlin-etal-2019-bert} fine-tuned for semantic similarity evaluation task in cross-encoder manner on synthetic data)~\cite{lai-etal-2021-thank} and BERTScore\cite{zhang2019bertscore} (F1-score over BERT-embeddings between tokens from initial and target sentences)~\cite{lee-etal-2021-enhancing} are also gaining popularity. &#10;&#10;To the best of our knowledge, none of the newly proposed TST techniques have been tested in the domain of the personal plan. We consider the step towards such a domain in TST research valuable because it makes its application in the real world even more likely. One of the main distinguishing properties of this domain is a large number of named entities (NE). NEs are real-world objects, such as a person, location, organization, etc. Indeed, when a client wants to order a taxi or book a flight, and a dialogue agent's reply is modified to, for example, a more informal style to make a conversation more natural, it is crucial to keep all significant details of the client's request, as a destination of a taxi ride or a name of the departure airport.&#10;&#10;We assume that if a NE is lost during TST, then some important parts of the original content are lost. For example, in~\cite{nema-khapra-2018-towards} authors exploited a similar assumption and used the information about NEs and some other categories of words to improve measures like BLEU or METEOR~\cite{banerjee-lavie-2005-meteor} for question answering task. Thus, we dedicate our work to studying the role of named entities and other linguistic objects in the process of TST and, in particular, in content similarity scoring.&#10;&#10;The contributions of our paper are as follows:&#10;\begin{itemize}&#10; \item We create and release the first benchmark dataset for evaluating content similarity measures in style transfer in the task-oriented dialogue domain (Section~\ref{section:dataset});&#10;&#10;\item We perform an error analysis of a SOTA pre-trained text style transfer system in terms of content preservation (Section~\ref{sec:info_loss});&#10;&#10;\item We perform an error analysis of SOTA content similarity measures used in text style transfer (Section~\ref{sec:measures_errors});&#10;&#10;\item We introduce a simple technique for enriching the content similarity measures with information about named entities, which increases the quality of strong baseline measures used in text style transfer (Section~\ref{sec:ner}). &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Content Similarity Measures, Artificial Intelligence, Natural Language Processing, Named Entity Preservation, Text Style Transfer" />
        </attvalues>
      </node>
      <node id="2005.00701" label="2005.00701">
        <attvalues>
          <attvalue for="0" value="Expertise Style Transfer: A New Task Towards Better Communication&#10;  between Experts and Laymen" />
          <attvalue for="1" value="  The curse of knowledge can impede communication between experts and laymen.&#10;We propose a new task of expertise style transfer and contribute a manually&#10;annotated dataset with the goal of alleviating such cognitive biases. Solving&#10;this task not only simplifies the professional language, but also improves the&#10;accuracy and expertise level of laymen descriptions using simple words. This is&#10;a challenging task, unaddressed in previous work, as it requires the models to&#10;have expert intelligence in order to modify text with a deep understanding of&#10;domain knowledge and structures. We establish the benchmark performance of five&#10;state-of-the-art models for style transfer and text simplification. The results&#10;demonstrate a significant gap between machine and human performance. We also&#10;discuss the challenges of automatic evaluation, to provide insights into future&#10;research directions. The dataset is publicly available at&#10;https://srhthu.github.io/expertise-style-transfer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03802" label="2010.03802">
        <attvalues>
          <attvalue for="0" value="TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling" />
          <attvalue for="1" value="  We present a novel approach to the problem of text style transfer. Unlike&#10;previous approaches requiring style-labeled training data, our method makes use&#10;of readily-available unlabeled text by relying on the implicit connection in&#10;style between adjacent sentences, and uses labeled data only at inference time.&#10;We adapt T5 (Raffel et al., 2020), a strong pretrained text-to-text model, to&#10;extract a style vector from text and use it to condition the decoder to perform&#10;style transfer. As our label-free training results in a style vector space&#10;encoding many facets of style, we recast transfers as &quot;targeted restyling&quot;&#10;vector operations that adjust specific attributes of the input while preserving&#10;others. We demonstrate that training on unlabeled Amazon reviews data results&#10;in a model that is competitive on sentiment transfer, even compared to models&#10;trained fully on labeled data. Furthermore, applying our novel method to a&#10;diverse corpus of unlabeled web text results in a single model capable of&#10;transferring along multiple dimensions of style (dialect, emotiveness,&#10;formality, politeness, sentiment) despite no additional training and using only&#10;a handful of exemplars at inference time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;As mentioned at the outset, recent work on text style transfer falls into three classes: supervised, ``unsupervised'', and few-shot. Supervised style transfer has seen limited research due to the difficulty of obtaining parallel data. Examples include \cite{jhamtani17} and \cite{carlson18}.&#10;&#10;Unsupervised Approaches&#10;The bulk of research has focused on ``unsupervised'' approaches, which rely on labeled but non-parallel data.&#10;Typically, labels are assumed to be available for both source and target styles (\cite{shen17}, \cite{li18}, \cite{niu18}, and many others). \cite{zhao18} explore the case where only the target style is labeled. The use of labels at training time can aid modeling, but limits the applicability of these methods, as labeled datasets are not readily available for many attributes of interest.&#10;&#10;Our work differs from the above by removing the need for training labels, and offering a single model that can target an unrestricted set of style attributes. Despite these differences, our work shares some similarities with past work. For example, our encoder-decoder architecture and corruption methods are similar to \cite{lample19}, and we leverage a strong pretrained language model, as in \cite{sudhakar19} and \cite{wu19}.&#10;&#10;Few-Shot Approaches&#10;A few-shot approach has recently been explored by \cite{xu20}.&#10;The authors train a variational auto-encoder on unlabeled text, where a ``manipulable'' portion of the latent representation is constrained to fall on a k-dimensional simplex.&#10;To perform transfer, they identify empirically the basis vector that most strongly corresponds to the target attribute, and manipulate its magnitude.&#10;Compared to our approach, a key difference is that the number of latent factors must be chosen ahead of time, which limits the number of attributes that may be controlled. Additionally, there is no guarantee that a single basis of the learned simplex will correspond to a target attribute such as dialect or politeness.&#10;&#10;Controlled Generation&#10;A separate strand of research explores ``controlled generation'' methods for supplementing generative language models to allow control of specific attributes of the output text. As with style transfer, this can be achieved either through labeled training examples, as in CTRL \cite{keskar2019ctrl} and PPLM \cite{dathathri20}, or a few-shot approach, as in CoCon \cite{chan20}. These models differ from style transfer models in that they aim to generate plausible continuations following a prompt, as opposed to transferring attributes of a fully-formed input while preserving as much content as possible. It is not clear if controlled generation models could be used to perform style transfer, and they have not to our knowledge been evaluated in this context.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1810.11878" label="1810.11878">
        <attvalues>
          <attvalue for="0" value="Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel&#10;  Textual Transfer" />
          <attvalue for="1" value="  We consider the problem of automatically generating textual paraphrases with&#10;modified attributes or properties, focusing on the setting without parallel&#10;data (Hu et al., 2017; Shen et al., 2017). This setting poses challenges for&#10;evaluation. We show that the metric of post-transfer classification accuracy is&#10;insufficient on its own, and propose additional metrics based on semantic&#10;preservation and fluency as well as a way to combine them into a single overall&#10;score. We contribute new loss functions and training strategies to address the&#10;different metrics. Semantic preservation is addressed by adding a cyclic&#10;consistency loss and a loss based on paraphrase pairs, while fluency is&#10;improved by integrating losses based on style-specific language models. We&#10;experiment with a Yelp sentiment dataset and a new literature dataset that we&#10;propose, using multiple models that extend prior work (Shen et al., 2017). We&#10;demonstrate that our metrics correlate well with human judgments, at both the&#10;sentence-level and system-level. Automatic and manual evaluation also show&#10;large improvements over the baseline method of Shen et al. (2017). We hope that&#10;our proposed metrics can speed up system development for new textual transfer&#10;tasks while also encouraging the community to address our three complementary&#10;aspects of transfer quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.00449" label="2108.00449">
        <attvalues>
          <attvalue for="0" value="Enhancing Content Preservation in Text Style Transfer Using Reverse&#10;  Attention and Conditional Layer Normalization" />
          <attvalue for="1" value="  Text style transfer aims to alter the style (e.g., sentiment) of a sentence&#10;while preserving its content. A common approach is to map a given sentence to&#10;content representation that is free of style, and the content representation is&#10;fed to a decoder with a target style. Previous methods in filtering style&#10;completely remove tokens with style at the token level, which incurs the loss&#10;of content information. In this paper, we propose to enhance content&#10;preservation by implicitly removing the style information of each token with&#10;reverse attention, and thereby retain the content. Furthermore, we fuse content&#10;information when building the target style representation, making it dynamic&#10;with respect to the content. Our method creates not only style-independent&#10;content representation, but also content-dependent style representation in&#10;transferring style. Empirical results show that our method outperforms the&#10;state-of-the-art baselines by a large margin in terms of content preservation.&#10;In addition, it is also competitive in terms of style transfer accuracy and&#10;fluency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.15451" label="2311.15451">
        <attvalues>
          <attvalue for="0" value="Uncertainty-aware Language Modeling for Selective Question Answering" />
          <attvalue for="1" value="  We present an automatic large language model (LLM) conversion approach that&#10;produces uncertainty-aware LLMs capable of estimating uncertainty with every&#10;prediction. Our approach is model- and data-agnostic, is&#10;computationally-efficient, and does not rely on external models or systems. We&#10;evaluate converted models on the selective question answering setting -- to&#10;answer as many questions as possible while maintaining a given accuracy,&#10;forgoing providing predictions when necessary. As part of our results, we test&#10;BERT and Llama 2 model variants on the SQuAD extractive QA task and the&#10;TruthfulQA generative QA task. We show that using the uncertainty estimates&#10;provided by our approach to selectively answer questions leads to significantly&#10;higher accuracy over directly using model probabilities.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large-language models (LLMs) have demonstrated great abilities in natural language tasks, including question answering (QA) wherein the model receives a question as input and outputs a response answer. The QA task is a fundamental component in many LLM applications. However, in order to robustly answer questions accurately, the model must understand context and ground its outputs in knowledge obtained from training data, which will typically contain conflicting information. &#10;Indeed, it has been shown that LLMs commonly fail in QA tasks \cite{geiger2019posing}, and that these failures are associated with a limited understanding of output confidence, out-of-domain data, ambiguity in input prompts, inconsistent training information, and hallucinations, among others. Selective prediction \cite{el2010foundations, geifman2017selective}, i.e., calculating confidence estimates along with predictions to forgo outputs likely to be incorrect, can be used to mitigate some of these issues. &#10;&#10;Several approaches utilize the selective prediction to guide question answering tasks \cite{penas2010overview, gondek2012framework} and generally abstain from answering questions when output confidence is low. &#10;The objective is to maintain a given accuracy while answering as many questions as possible, as opposed to the more conventional goal of attempting to answer all questions correctly. One approach used inferred softmax classifier probabilities to calibrate which questions to respond to \cite{rodriguez2021quizbowl}. However, training a calibration model is challenging and softmax classifiers are often unreliable \cite{guo2017calibration}. Similarly, an out-of-domain (OOD) calibrator can be trained to detect OOD inputs \cite{kamath2020selective} but requires known or synthetic out-of-domain samples and does not consider other sources of inaccuracies like over-represented features or ambiguous labels. Other approaches include modeling and estimating LLM uncertainty \cite{dong2018confidence, shen2022posthoc, chen2023quantifying, lin2023generating, collins2023human, chuang2023dola, quach2023conformal}, fine-tuning calibrators to consider entropy, perplexity, and other metrics \cite{jiang2021can}, and calculating output consistency \cite{manakul2023selfcheckgpt, miao2023selfcheck}. Another family of techniques retrieve evidence and verify outputs through external databases \cite{guo2022survey} or in-context learning \cite{weng2022large}. These solutions require the development of knowledge bases and efficient querying systems which are often not practical and fundamentally limited by the information that exists. A facile, performant, and efficient way to estimate uncertainty directly from models, without the need of external components, is needed to design a general selective question answering framework that is applicable to a wide range of tasks. &#10;&#10;We present an uncertainty-based framework for selective QA that accounts for epistemic and aleatoric uncertainty. We consider both extractive and generative LLM models e.g., masked-language models and autoregressive models respectively, and implement and evaluate a suite of uncertainty quantification (UQ) methods spanning these uncertainty types. We find that while the individual methods significantly increase performance on the selective QA task, it is the combination of methods and metrics that yield the best accuracy. Leveraging this observation and seeking to enable both performance and efficiency, we present an approach to automatically convert LLMs into uncertainty-aware variants and to compose metrics and methods automatically. Our approach is model- and data-agnostic, lightweight, and does not rely on external models or systems. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Model Conversion, Artificial Intelligence, Mathematics, Uncertainty Estimation, Question Answering Systems" />
        </attvalues>
      </node>
      <node id="1911.00811" label="1911.00811">
        <attvalues>
          <attvalue for="0" value="Posing Fair Generalization Tasks for Natural Language Inference" />
          <attvalue for="1" value="  Deep learning models for semantics are generally evaluated using naturalistic&#10;corpora. Adversarial methods, in which models are evaluated on new examples&#10;with known semantic properties, have begun to reveal that good performance at&#10;these naturalistic tasks can hide serious shortcomings. However, we should&#10;insist that these evaluations be fair -that the models are given data&#10;sufficient to support the requisite kinds of generalization. In this paper, we&#10;define and motivate a formal notion of fairness in this sense. We then apply&#10;these ideas to natural language inference by constructing very challenging but&#10;provably fair artificial datasets and showing that standard neural models fail&#10;to generalize in the required ways; only task-specific models that jointly&#10;compose the premise and hypothesis are able to achieve high performance, and&#10;even these models do not solve the task perfectly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.04792" label="1904.04792">
        <attvalues>
          <attvalue for="0" value="Quizbowl: The Case for Incremental Question Answering" />
          <attvalue for="1" value="  Scholastic trivia competitions test knowledge and intelligence through&#10;mastery of question answering. Modern question answering benchmarks are one&#10;variant of the Turing test. Specifically, answering a set of questions as well&#10;as a human is a minimum bar towards demonstrating human-like intelligence. This&#10;paper makes the case that the format of one competition -- where participants&#10;can answer in the middle of hearing a question (incremental) -- better&#10;differentiates the skill between (human or machine) players. Additionally,&#10;merging a sequential decision-making sub-task with question answering (QA)&#10;provides a good setting for research in model calibration and opponent&#10;modeling. Thus, embedded in this task are three machine learning challenges:&#10;(1) factoid QA over thousands of Wikipedia-like answers, (2) calibration of the&#10;QA model's confidence scores, and (3) sequential decision-making that&#10;incorporates knowledge of the QA model, its calibration, and what the opponent&#10;may do. We make two contributions: (1) collecting and curating a large factoid&#10;QA dataset and an accompanying gameplay dataset, and (2) developing a model&#10;that addresses these three machine learning challenges. In addition to offline&#10;evaluation, we pitted our model against some of the most accomplished trivia&#10;players in the world in a series of exhibition matches spanning several years.&#10;Throughout this paper, we show that collaborations with the vibrant trivia&#10;community have contributed to the quality of our dataset, spawned new research&#10;directions, and doubled as an exciting way to engage the public with research&#10;in machine learning and natural language processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.04604" label="1805.04604">
        <attvalues>
          <attvalue for="0" value="Confidence Modeling for Neural Semantic Parsing" />
          <attvalue for="1" value="  In this work we focus on confidence modeling for neural semantic parsers&#10;which are built upon sequence-to-sequence models. We outline three major causes&#10;of uncertainty, and design various metrics to quantify these factors. These&#10;metrics are then used to estimate confidence scores that indicate whether model&#10;predictions are likely to be correct. Beyond confidence estimation, we identify&#10;which parts of the input contribute to uncertain predictions allowing users to&#10;interpret their model, and verify or refine its input. Experimental results&#10;show that our confidence model significantly outperforms a widely used method&#10;that relies on posterior probability, and improves the quality of&#10;interpretation compared to simply relying on attention scores.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.11896" label="2108.11896">
        <attvalues>
          <attvalue for="0" value="A Survey on Automated Fact-Checking" />
          <attvalue for="1" value="  Fact-checking has become increasingly important due to the speed with which&#10;both information and misinformation can spread in the modern media ecosystem.&#10;Therefore, researchers have been exploring how fact-checking can be automated,&#10;using techniques based on natural language processing, machine learning,&#10;knowledge representation, and databases to automatically predict the veracity&#10;of claims. In this paper, we survey automated fact-checking stemming from&#10;natural language processing, and discuss its connections to related tasks and&#10;disciplines. In this process, we present an overview of existing datasets and&#10;models, aiming to unify the various definitions given and identify common&#10;concepts. Finally, we highlight challenges for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.08206" label="1908.08206">
        <attvalues>
          <attvalue for="0" value="Denoising based Sequence-to-Sequence Pre-training for Text Generation" />
          <attvalue for="1" value="  This paper presents a new sequence-to-sequence (seq2seq) pre-training method&#10;PoDA (Pre-training of Denoising Autoencoders), which learns representations&#10;suitable for text generation tasks. Unlike encoder-only (e.g., BERT) or&#10;decoder-only (e.g., OpenAI GPT) pre-training approaches, PoDA jointly&#10;pre-trains both the encoder and decoder by denoising the noise-corrupted text,&#10;and it also has the advantage of keeping the network architecture unchanged in&#10;the subsequent fine-tuning stage. Meanwhile, we design a hybrid model of&#10;Transformer and pointer-generator networks as the backbone architecture for&#10;PoDA. We conduct experiments on two text generation tasks: abstractive&#10;summarization, and grammatical error correction. Results on four datasets show&#10;that PoDA can improve model performance over strong baselines without using any&#10;task-specific techniques and significantly speed up convergence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.12102" label="2304.12102">
        <attvalues>
          <attvalue for="0" value="Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of&#10;  LLMs with Self-Information-Based Content Filtering" />
          <attvalue for="1" value="  Large language models (LLMs) have received significant attention by achieving&#10;remarkable performance across various tasks. However, their fixed context&#10;length poses challenges when processing long documents or maintaining extended&#10;conversations. This paper proposes a method called \textit{Selective Context}&#10;that employs self-information to filter out less informative content, thereby&#10;enhancing the efficiency of the fixed context length. We demonstrate the&#10;effectiveness of our approach on tasks of summarisation and question answering&#10;across different data sources, including academic papers, news articles, and&#10;conversation transcripts.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across a wide range of natural language processing tasks, as well as real-life applications \cite{brown2020language,touvron2023llama,bubeck2023sparks}. However, a major limitation of LLMs is their fixed context length. As LLMs have no memory outside their context window, it poses a significant challenge when tackling tasks that involve processing long documents or engaging in extended conversations \cite{dong2023survey}. Increasing the context length for LLMs, particularly those based on Transformer, is very expensive due to the quadratic growth of memory and computation associated with the 2-D attention matrix \cite{vaswani2017attention}. These limitations highlight the need for more efficient solutions to utilize the limited context in tasks that require extended context.&#10;&#10;Fortunately our experiments reveal that LLMs do not need all content in a document or the entire conversation history to answer users' queries. As shown in Figure \ref{fig:example}, LLMs are able to generate the expected answer even with relevant information deleted. This might be because LLMs can infer the missing information based on the contextual clues and prior knowledge acquired from their pre-training. As a result, we argue that optimizing the use of context length by filtering out less informative content is possible without sacrificing performance.&#10;&#10;In this paper, we propose Selective Context, which filters out less informative content to reduce the cost of a given context, thereby making better use of the fixed context length in LLMs. Selective Context employs a base language model to compute self-information for lexical units (sentences, phrases, or tokens) in a context and use it to evaluate their informativeness. By selectively retaining content with higher self-information, our method provides a more compact and efficient context representation for LLMs to process without compromising their performance on various tasks.&#10;&#10;To evaluate the effectiveness of our proposed method, we tested Selective Context on three data sources: arxiv papers, BBC news articles, and conversation transcripts with four different NLP tasks: summarisation, question answering, original context reconstruction and conversation. Our results demonstrate that Selective Context significantly enhances the efficiency of LLMs, allowing them to handle long documents and extended conversations with only minor sacrifices in generation quality.&#10;&#10;The key contributions of our paper: (1) We introduce Selective Context, a novel approach to context filtering that maximises the utility of fixed context length in LLMs. (2) We provide extensive evaluations of the proposed method. (3) Our results demonstrate the effectiveness of Selective Context in reducing the cost of context in LLMs. Code and data can be found in.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Context Length Optimization, Information Science, Natural Language Processing, Language Model Efficiency" />
        </attvalues>
      </node>
      <node id="2302.14502" label="2302.14502">
        <attvalues>
          <attvalue for="0" value="A Survey on Long Text Modeling with Transformers" />
          <attvalue for="1" value="  Modeling long texts has been an essential technique in the field of natural&#10;language processing (NLP). With the ever-growing number of long documents, it&#10;is important to develop effective modeling methods that can process and analyze&#10;such texts. However, long texts pose important research challenges for existing&#10;text models, with more complex semantics and special characteristics. In this&#10;paper, we provide an overview of the recent advances on long texts modeling&#10;based on Transformer models. Firstly, we introduce the formal definition of&#10;long text modeling. Then, as the core content, we discuss how to process long&#10;input to satisfy the length limitation and design improved Transformer&#10;architectures to effectively extend the maximum context length. Following this,&#10;we discuss how to adapt Transformer models to capture the special&#10;characteristics of long texts. Finally, we describe four typical applications&#10;involving long text modeling and conclude this paper with a discussion of&#10;future directions. Our survey intends to provide researchers with a synthesis&#10;and pointer to related work on long text modeling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.07875" label="1912.07875">
        <attvalues>
          <attvalue for="0" value="Libri-Light: A Benchmark for ASR with Limited or No Supervision" />
          <attvalue for="1" value="  We introduce a new collection of spoken English audio suitable for training&#10;speech recognition systems under limited or no supervision. It is derived from&#10;open-source audio books from the LibriVox project. It contains over 60K hours&#10;of audio, which is, to our knowledge, the largest freely-available corpus of&#10;speech. The audio has been segmented using voice activity detection and is&#10;tagged with SNR, speaker ID and genre descriptions. Additionally, we provide&#10;baseline systems and evaluation metrics working under three settings: (1) the&#10;zero resource/unsupervised setting (ABX), (2) the semi-supervised setting (PER,&#10;CER) and (3) the distant supervision setting (WER). Settings (2) and (3) use&#10;limited textual resources (10 minutes to 10 hours) aligned with the speech.&#10;Setting (3) uses large amounts of unaligned text. They are evaluated on the&#10;standard LibriSpeech dev and test sets for comparison with the supervised&#10;state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.05077" label="2211.05077">
        <attvalues>
          <attvalue for="0" value="Prompting Large Pre-trained Vision-Language Models For Compositional&#10;  Concept Learning" />
          <attvalue for="1" value="  This work explores the zero-shot compositional learning ability of large&#10;pre-trained vision-language models(VLMs) within the prompt-based learning&#10;framework and propose a model (\textit{PromptCompVL}) to solve the compositonal&#10;zero-shot learning (CZSL) problem. \textit{PromptCompVL} makes two design&#10;choices: first, it uses a soft-prompting instead of hard-prompting to inject&#10;learnable parameters to reprogram VLMs for compositional learning. Second, to&#10;address the compositional challenge, it uses the soft-embedding layer to learn&#10;primitive concepts in different combinations. By combining both soft-embedding&#10;and soft-prompting, \textit{PromptCompVL} achieves state-of-the-art performance&#10;on the MIT-States dataset. Furthermore, our proposed model achieves consistent&#10;improvement compared to other CLIP-based methods which shows the effectiveness&#10;of the proposed prompting strategies for CZSL.&#10;" />
          <attvalue for="2" value="&#10;&#10;In this paper, &#10;we investigate a previously formulated compositional learning problem, compositional zero-shot learning (CZSL), which requires the agent to recognize novel compositional attribute-object (attr-obj) pairs by composing previously learnt primitive concepts. For example, in Fig.~\ref{fig:example}, after learning the primitive concepts, sliced and apple, CZSL expects the agent to recognize sliced apple which has not been observed during training time.&#10;&#10;The main challenge of CZSL is the distribution-shift between the training and test data which causes the learnt models overfit the seen compositions.&#10;Previous works usually construct a shared embedding space and add different constraints to regularize the space for compositional concept learning~\cite{attrasopt,graph_comp,open_world_comp}.&#10;In this work, we attempt to solve the CZSL problem from the lens of prompting large vision-language models. We propose a model, called PromptCompVL, to explore the compositional learning ability of current VLMs. &#10;&#10;The core idea of PromptCompVL is to inject learnable pieces, including the soft-prompting layer and the soft-embedding layer,&#10;to CLIP~\cite{clip} for compositional learning. In particular, soft-prompting layer is used to replace the CLIP's hard-prompting vectors in order to increase CLIP's capacity and reprogramming it for CZSL by adjusting the soft-prompting vectors. Moreover, soft-embedding layer is introduced to replace CLIP's original vocabulary embedding layer to address the compositional concept learning challenge. We use the soft-embedding layer to encode the primitive concepts and update the concept embedding through observing different combinations during training time. The role of soft-embedding is similar to verbalizer in the general prompt-learning framework~\cite{liupengfei}. &#10;&#10;The advantages of this work can be summarized as follows:&#10;1) Inherited from prompting methods, PromptCompVL is a parameter-efficient learning framework which can improve CZSL using VLMs without the overhead of fine-tuning the entire model.&#10;2) Different from previous prompting architecture, PromptCompVL introduces two learnable components, soft-embedding and soft-prompting simultaneously, to address CZSL problems. In particular, it introduces the soft-embedding layer to address the compositional challenge and the soft-prompting layer to improve the VLMs' flexibly to fit CZSL tasks.&#10;3) PromptCompVL achieves SOTA result on MIT-States dataset and shows consistent improvements compared to other CLIP-based methods on both MIT-States and UT-Zappos datasets.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Prompt-Based Learning, Computer Science, Machine Learning, Linguistics, Cognitive Science, Computer Vision, Natural Language Processing, Vision Language Models, Compositional Learning" />
        </attvalues>
      </node>
      <node id="1803.09851" label="1803.09851">
        <attvalues>
          <attvalue for="0" value="Attributes as Operators: Factorizing Unseen Attribute-Object&#10;  Compositions" />
          <attvalue for="1" value="  We present a new approach to modeling visual attributes. Prior work casts&#10;attributes in a similar role as objects, learning a latent representation where&#10;properties (e.g., sliced) are recognized by classifiers much in the way objects&#10;(e.g., apple) are. However, this common approach fails to separate the&#10;attributes observed during training from the objects with which they are&#10;composed, making it ineffectual when encountering new attribute-object&#10;compositions. Instead, we propose to model attributes as operators. Our&#10;approach learns a semantic embedding that explicitly factors out attributes&#10;from their accompanying objects, and also benefits from novel regularizers&#10;expressing attribute operators' effects (e.g., blunt should undo the effects of&#10;sharp). Not only does our approach align conceptually with the linguistic role&#10;of attributes as modifiers, but it also generalizes to recognize unseen&#10;compositions of objects and attributes. We validate our approach on two&#10;challenging datasets and demonstrate significant improvements over the&#10;state-of-the-art. In addition, we show that not only can our model recognize&#10;unseen compositions robustly in an open-world setting, it can also generalize&#10;to compositions where objects themselves were unseen during training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.01987" label="2102.01987">
        <attvalues>
          <attvalue for="0" value="Learning Graph Embeddings for Compositional Zero-shot Learning" />
          <attvalue for="1" value="  In compositional zero-shot learning, the goal is to recognize unseen&#10;compositions (e.g. old dog) of observed visual primitives states (e.g. old,&#10;cute) and objects (e.g. car, dog) in the training set. This is challenging&#10;because the same state can for example alter the visual appearance of a dog&#10;drastically differently from a car. As a solution, we propose a novel graph&#10;formulation called Compositional Graph Embedding (CGE) that learns image&#10;features, compositional classifiers, and latent representations of visual&#10;primitives in an end-to-end manner. The key to our approach is exploiting the&#10;dependency between states, objects, and their compositions within a graph&#10;structure to enforce the relevant knowledge transfer from seen to unseen&#10;compositions. By learning a joint compatibility that encodes semantics between&#10;concepts, our model allows for generalization to unseen compositions without&#10;relying on an external knowledge base like WordNet. We show that in the&#10;challenging generalized compositional zero-shot setting our CGE significantly&#10;outperforms the state of the art on MIT-States and UT-Zappos. We also propose a&#10;new benchmark for this task based on the recent GQA dataset. Code is available&#10;at: https://github.com/ExplainableML/czsl&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.12609" label="2101.12609">
        <attvalues>
          <attvalue for="0" value="Open World Compositional Zero-Shot Learning" />
          <attvalue for="1" value="  Compositional Zero-Shot learning (CZSL) requires to recognize state-object&#10;compositions unseen during training. In this work, instead of assuming prior&#10;knowledge about the unseen compositions, we operate in the open world setting,&#10;where the search space includes a large number of unseen compositions some of&#10;which might be unfeasible. In this setting, we start from the cosine similarity&#10;between visual features and compositional embeddings. After estimating the&#10;feasibility score of each composition, we use these scores to either directly&#10;mask the output space or as a margin for the cosine similarity between visual&#10;features and compositional embeddings during training. Our experiments on two&#10;standard CZSL benchmarks show that all the methods suffer severe performance&#10;degradation when applied in the open world setting. While our simple CZSL model&#10;achieves state-of-the-art performances in the closed world scenario, our&#10;feasibility scores boost the performance of our approach in the open world&#10;setting, clearly outperforming the previous state of the art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.06927" label="2409.06927">
        <attvalues>
          <attvalue for="0" value="Representation Tuning" />
          <attvalue for="1" value="  Activation engineering is becoming increasingly popular as a means of online&#10;control of large language models (LLMs). In this work, I extend the idea of&#10;active steering with vectors that represent a behavioral direction of interest&#10;to tuning those vectors directly into the model, obviating the need for online&#10;control. First, I identify activation vectors related to honesty in an&#10;open-source LLM (Llama- 2-13b-chat). Next, I demonstrate that model output can&#10;be made more or less honest by adding positive or negative multiples of these&#10;vectors to residual stream activations during generation. Then, I show that a&#10;similar effect can be achieved by fine-tuning the vectors directly into the&#10;model, by use of a dual loss function based on the cosine similarity of&#10;residual stream activations to the vectors combined with a standard token-based&#10;loss (&quot;representation tuning&quot;). Finally, I compare the generations in response&#10;to honesty-probing prompts from the resulting models to those from models&#10;fine-tuned with a token-based loss alone, and to those from the untuned model&#10;subjected to online steering. Overall, fine-tuning the vectors into the models&#10;using the cosine similarity plus token loss showed a stronger effect than&#10;online steering, and generalized better than using the standard loss,&#10;suggesting the potential utility of this approach as a safety measure. Code and&#10;data are available at https://github.com/cma1114/representation_tuning; tuned&#10;models are available at https://huggingface.co/collections/cackerman/&#10;representation-tuning-66da1e5ab41cd1b824687d9f.&#10;" />
          <attvalue for="2" value="&#10;&#10;The concept of activation steering \cite{turner2024steeringlanguagemodelsactivation}/representation engineering \cite{zou2023representationengineeringtopdownapproach} on transformer-based LLMs is simple, and it is remarkable that it works. First, one identifies an activation pattern in a model (generally in the residual stream input or output) corresponding to a high-level behavior like ``sycophancy'' \cite{Panickssery2023} or ``honesty'' by a simple expedient such as running pairs of inputs with and without the behavior through the model and taking the mean of the differences in the pairs' activations. Then one adds the resulting vector, scaled by +/- various coefficients, to the model's activations as it generates new output, and the model gives output that has more or less of the behavior, as one desires. This would seem quite interesting from the perspective of LLM interpretability, and potentially safety.&#10;&#10;In this work we extend the activation steering concept by permanently changing the weights of the model via fine-tuning, obviating the need for active steering with every input. Other researchers \cite{panickssery2024steeringllama2contrastive} have independently explored the idea of fine-tuning as a replacement for online steering, but this work is distinctive in targeting the tuning specifically at model activations, rather than the standard method of tuning based on model output deviations from target output. In addition to offering compute savings due to not having to add vectors to every token at inference, it was hypothesized that this approach might make the model more robust in its intended behavior. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Fine-Tuning, Language Model Control, Artificial Intelligence, Mathematics, Natural Language Processing, Activation Engineering" />
        </attvalues>
      </node>
      <node id="2310.01405" label="2310.01405">
        <attvalues>
          <attvalue for="0" value="Representation Engineering: A Top-Down Approach to AI Transparency" />
          <attvalue for="1" value="  In this paper, we identify and characterize the emerging area of&#10;representation engineering (RepE), an approach to enhancing the transparency of&#10;AI systems that draws on insights from cognitive neuroscience. RepE places&#10;population-level representations, rather than neurons or circuits, at the&#10;center of analysis, equipping us with novel methods for monitoring and&#10;manipulating high-level cognitive phenomena in deep neural networks (DNNs). We&#10;provide baselines and an initial analysis of RepE techniques, showing that they&#10;offer simple yet effective solutions for improving our understanding and&#10;control of large language models. We showcase how these methods can provide&#10;traction on a wide range of safety-relevant problems, including honesty,&#10;harmlessness, power-seeking, and more, demonstrating the promise of top-down&#10;transparency research. We hope that this work catalyzes further exploration of&#10;RepE and fosters advancements in the transparency and safety of AI systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.06478" label="2010.06478">
        <attvalues>
          <attvalue for="0" value="XL-WiC: A Multilingual Benchmark for Evaluating Semantic&#10;  Contextualization" />
          <attvalue for="1" value="  The ability to correctly model distinct meanings of a word is crucial for the&#10;effectiveness of semantic representation techniques. However, most existing&#10;evaluation benchmarks for assessing this criterion are tied to sense&#10;inventories (usually WordNet), restricting their usage to a small subset of&#10;knowledge-based representation techniques. The Word-in-Context dataset (WiC)&#10;addresses the dependence on sense inventories by reformulating the standard&#10;disambiguation task as a binary classification problem; but, it is limited to&#10;the English language. We put forward a large multilingual benchmark, XL-WiC,&#10;featuring gold standards in 12 new languages from varied language families and&#10;with different degrees of resource availability, opening room for evaluation&#10;scenarios such as zero-shot cross-lingual transfer. We perform a series of&#10;experiments to determine the reliability of the datasets and to set performance&#10;baselines for several recent contextualized multilingual models. Experimental&#10;results show that even when no tagged instances are available for a target&#10;language, models trained solely on the English data can attain competitive&#10;performance in the task of distinguishing different meanings of a word, even&#10;for distant languages. XL-WiC is available at&#10;https://pilehvar.github.io/xlwic/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.04876" label="2202.04876">
        <attvalues>
          <attvalue for="0" value="Distilling Hypernymy Relations from Language Models: On the&#10;  Effectiveness of Zero-Shot Taxonomy Induction" />
          <attvalue for="1" value="  In this paper, we analyze zero-shot taxonomy learning methods which are based&#10;on distilling knowledge from language models via prompting and sentence&#10;scoring. We show that, despite their simplicity, these methods outperform some&#10;supervised strategies and are competitive with the current state-of-the-art&#10;under adequate conditions. We also show that statistical and linguistic&#10;properties of prompts dictate downstream performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Taxonomy learning (TL) is the task of arranging domain terminologies into hierarchical structures where terms are nodes and edges denote is-a (hypernymic) relationships \cite{Hwangetal2012}. Domain-specific concept generalization is at the core of human cognition \cite{Yuetal2015}, and a key enabler in NLP tasks where inference and reasoning are important, e.g.: semantic similarity \cite{Pilehvaretal:2013, yu2014improving}, WSD \cite{Agirreetal2014} and, more recently, QA \cite{joshi2020contextualized} and NLI \cite{chen2020mining}.&#10;&#10;Earlier approaches to taxonomy learning focused on mining lexico-syntactic patterns from candidate $($hyponym, hypernym$)$ pairs \cite{Hearst1992,Snowetal2004,KozarevaandHovy2010,BoellaandDiCaro2013,Espinosa-Ankeetal2016AAAI}, clustering \cite{YangandCallan2009}, graph-based methods \cite{FountainandLapata2012,Velardietal2013} or word embeddings \cite{Fuetal2014,Yuetal2015}. These methods, which largely rely on hand-crafted features, are still relevant today, and complement modern approaches exploiting language models (LMs), either via sequence classification \cite{Chen2021ConstructingTF}, or combining contextual, distributed, and lexico-syntactic features \cite{Yu2020STEAMST}. In parallel, several works have recently focused on using LMs as zero-shot tools for solving NLP tasks, e.g., commonsense, relational and analogical reasoning \cite{Petroni2019LanguageMA,bouraoui2020inducing, ushio-etal-2021-bert-is,paranjape2021prompting}, multiword expression (MWE) identification \cite{anke2021evaluating,garcia2021probing}, QA \cite{shwartz2020unsupervised, banerjee2020self}, domain labeling \cite{sainz2021ask2transformers}, or lexical substitution and simplification \cite{zhou2019bert}. Moreover, by tuning and manipulating natural language queries (often referred to as prompts),&#10;impressive results have been recently obtained on tasks such as semantic textual similarity, entailment, or relation classification \cite{shin2020eliciting,qin2021learning}.&#10;&#10;In this paper, we evaluate LMs on TL benchmarks using prompt-based and sentence-scoring techniques, and find not only that they are competitive with common approaches proposed in the literature (which are typically supervised and/or reliant on external resources), but that they achieve SoTa results in certain domains.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Zero-Shot Learning, Artificial Intelligence, Statistics, Taxonomy Classification" />
        </attvalues>
      </node>
      <node id="2004.12006" label="2004.12006">
        <attvalues>
          <attvalue for="0" value="Contextualized Representations Using Textual Encyclopedic Knowledge" />
          <attvalue for="1" value="  We present a method to represent input texts by contextualizing them jointly&#10;with dynamically retrieved textual encyclopedic background knowledge from&#10;multiple documents. We apply our method to reading comprehension tasks by&#10;encoding questions and passages together with background sentences about the&#10;entities they mention. We show that integrating background knowledge from text&#10;is effective for tasks focusing on factual reasoning and allows direct reuse of&#10;powerful pretrained BERT-style encoders. Moreover, knowledge integration can be&#10;further improved with suitable pretraining via a self-supervised masked&#10;language model objective over words in background-augmented input text. On&#10;TriviaQA, our approach obtains improvements of 1.6 to 3.1 F1 over comparable&#10;RoBERTa models which do not integrate background knowledge dynamically. On&#10;MRQA, a large collection of diverse QA datasets, we see consistent gains&#10;in-domain along with large improvements out-of-domain on BioASQ (2.1 to 4.2&#10;F1), TextbookQA (1.6 to 2.0 F1), and DuoRC (1.1 to 2.0 F1).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.01239" label="2010.01239">
        <attvalues>
          <attvalue for="0" value="Mining Knowledge for Natural Language Inference from Wikipedia&#10;  Categories" />
          <attvalue for="1" value="  Accurate lexical entailment (LE) and natural language inference (NLI) often&#10;require large quantities of costly annotations. To alleviate the need for&#10;labeled data, we introduce WikiNLI: a resource for improving model performance&#10;on NLI and LE tasks. It contains 428,899 pairs of phrases constructed from&#10;naturally annotated category hierarchies in Wikipedia. We show that we can&#10;improve strong baselines such as BERT and RoBERTa by pretraining them on&#10;WikiNLI and transferring the models on downstream tasks. We conduct systematic&#10;comparisons with phrases extracted from other knowledge bases such as WordNet&#10;and Wikidata to find that pretraining on WikiNLI gives the best performance. In&#10;addition, we construct WikiNLI in other languages, and show that pretraining on&#10;them improves performance on NLI tasks of corresponding languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12813" label="2010.12813">
        <attvalues>
          <attvalue for="0" value="Constructing Taxonomies from Pretrained Language Models" />
          <attvalue for="1" value="  We present a method for constructing taxonomic trees (e.g., WordNet) using&#10;pretrained language models. Our approach is composed of two modules, one that&#10;predicts parenthood relations and another that reconciles those predictions&#10;into trees. The parenthood prediction module produces likelihood scores for&#10;each potential parent-child pair, creating a graph of parent-child relation&#10;scores. The tree reconciliation module treats the task as a graph optimization&#10;problem and outputs the maximum spanning tree of this graph. We train our model&#10;on subtrees sampled from WordNet, and test on non-overlapping WordNet subtrees.&#10;We show that incorporating web-retrieved glosses can further improve&#10;performance. On the task of constructing subtrees of English WordNet, the model&#10;achieves 66.7 ancestor F1, a 20.0% relative increase over the previous best&#10;published result on this task. In addition, we convert the original English&#10;dataset into nine other languages using Open Multilingual WordNet and extend&#10;our results across these languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.12753" label="1911.12753">
        <attvalues>
          <attvalue for="0" value="Inducing Relational Knowledge from BERT" />
          <attvalue for="1" value="  One of the most remarkable properties of word embeddings is the fact that&#10;they capture certain types of semantic and syntactic relationships. Recently,&#10;pre-trained language models such as BERT have achieved groundbreaking results&#10;across a wide range of Natural Language Processing tasks. However, it is&#10;unclear to what extent such models capture relational knowledge beyond what is&#10;already captured by standard word embeddings. To explore this question, we&#10;propose a methodology for distilling relational knowledge from a pre-trained&#10;language model. Starting from a few seed instances of a given relation, we&#10;first use a large text corpus to find sentences that are likely to express this&#10;relation. We then use a subset of these extracted sentences as templates.&#10;Finally, we fine-tune a language model to predict whether a given word pair is&#10;likely to be an instance of some relation, when given an instantiated template&#10;for that relation as input.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06823" label="2106.06823">
        <attvalues>
          <attvalue for="0" value="Prompting Contrastive Explanations for Commonsense Reasoning Tasks" />
          <attvalue for="1" value="  Many commonsense reasoning NLP tasks involve choosing between one or more&#10;possible answers to a question or prompt based on knowledge that is often&#10;implicit. Large pretrained language models (PLMs) can achieve near-human&#10;performance on such tasks, while providing little human-interpretable evidence&#10;of the underlying reasoning they use. In this work, we show how to use these&#10;same models to generate such evidence: inspired by the contrastive nature of&#10;human explanations, we use PLMs to complete explanation prompts which contrast&#10;alternatives according to the key attribute(s) required to justify the correct&#10;answer (for example, peanuts are usually salty while raisins are sweet).&#10;Conditioning model decisions on these explanations improves performance on two&#10;commonsense reasoning benchmarks, as compared to previous non-contrastive&#10;alternatives. These explanations are also judged by humans to be more relevant&#10;for solving the task, and facilitate a novel method to evaluate explanation&#10;faithfulfness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.05483" label="2004.05483">
        <attvalues>
          <attvalue for="0" value="Unsupervised Commonsense Question Answering with Self-Talk" />
          <attvalue for="1" value="  Natural language understanding involves reading between the lines with&#10;implicit background knowledge. Current systems either rely on pre-trained&#10;language models as the sole implicit source of world knowledge, or resort to&#10;external knowledge bases (KBs) to incorporate additional relevant knowledge. We&#10;propose an unsupervised framework based on self-talk as a novel alternative to&#10;multiple-choice commonsense tasks. Inspired by inquiry-based discovery learning&#10;(Bruner, 1961), our approach inquires language models with a number of&#10;information seeking questions such as &quot;$\textit{what is the definition of&#10;...}$&quot; to discover additional background knowledge. Empirical results&#10;demonstrate that the self-talk procedure substantially improves the performance&#10;of zero-shot language model baselines on four out of six commonsense&#10;benchmarks, and competes with models that obtain knowledge from external KBs.&#10;While our approach improves performance on several benchmarks, the self-talk&#10;induced knowledge even when leading to correct answers is not always seen as&#10;useful by human judges, raising interesting questions about the inner-workings&#10;of pre-trained language models for commonsense reasoning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.06599" label="2104.06599">
        <attvalues>
          <attvalue for="0" value="Learning How to Ask: Querying LMs with Mixtures of Soft Prompts" />
          <attvalue for="1" value="  Natural-language prompts have recently been used to coax pretrained language&#10;models into performing other AI tasks, using a fill-in-the-blank paradigm&#10;(Petroni et al., 2019) or a few-shot extrapolation paradigm (Brown et al.,&#10;2020). For example, language models retain factual knowledge from their&#10;training corpora that can be extracted by asking them to &quot;fill in the blank&quot; in&#10;a sentential prompt. However, where does this prompt come from? We explore the&#10;idea of learning prompts by gradient descent -- either fine-tuning prompts&#10;taken from previous work, or starting from random initialization. Our prompts&#10;consist of &quot;soft words,&quot; i.e., continuous vectors that are not necessarily word&#10;type embeddings from the language model. Furthermore, for each task, we&#10;optimize a mixture of prompts, learning which prompts are most effective and&#10;how to ensemble them. Across multiple English LMs and tasks, our approach&#10;hugely outperforms previous methods, showing that the implicit factual&#10;knowledge in language models was previously underestimated. Moreover, this&#10;knowledge is cheap to elicit: random initialization is nearly as good as&#10;informed initialization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.04408" label="2310.04408">
        <attvalues>
          <attvalue for="0" value="RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective&#10;  Augmentation" />
          <attvalue for="1" value="  Retrieving documents and prepending them in-context at inference time&#10;improves performance of language model (LMs) on a wide range of tasks. However,&#10;these documents, often spanning hundreds of words, make inference substantially&#10;more expensive. We propose compressing the retrieved documents into textual&#10;summaries prior to in-context integration. This not only reduces the&#10;computational costs but also relieves the burden of LMs to identify relevant&#10;information in long retrieved documents. We present two compressors -- an&#10;extractive compressor which selects useful sentences from retrieved documents&#10;and an abstractive compressor which generates summaries by synthesizing&#10;information from multiple documents. Both compressors are trained to improve&#10;LMs' performance on end tasks when the generated summaries are prepended to the&#10;LMs' input, while keeping the summary concise.If the retrieved documents are&#10;irrelevant to the input or offer no additional information to LM, our&#10;compressor can return an empty string, implementing selective augmentation.We&#10;evaluate our approach on language modeling task and open domain question&#10;answering task. We achieve a compression rate of as low as 6% with minimal loss&#10;in performance for both tasks, significantly outperforming the off-the-shelf&#10;summarization models. We show that our compressors trained for one LM can&#10;transfer to other LMs on the language modeling task and provide summaries&#10;largely faithful to the retrieved documents.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Efficient RALM ~\cite{he-etal-2021-efficient} improves efficiency of RALMs by improving retrieval components, such as data store compression, dimensionality reduction for neural retriever. A line of work also introduces reducing retrieval frequency through selective retrieval \cite{he-etal-2021-efficient,Mallen2022WhenNT} or using a larger stride \cite{martins-etal-2022-chunk}. In this work, we improve efficiency of RALM by compressing retrieved documents into a concise summary or an empty sequence, facilitating selective retrieval augmentation.&#10;&#10;Prompt Compression&#10;Recent work~\cite{wingate-etal-2022-prompt, chevalier2023adapting, Mu2023LearningTC} proposes compressing long contexts into summary vectors (soft prompts) that can be used by LMs, rather than shorter textual summaries. Such soft prompts can serve as efficient replacements for plain-text demonstrations, minimizing the computational costs during inference. Another related line of work proposes context distillation \cite{Snell2022LearningBD,Choi2022PromptIP,Padmanabhan2023PropagatingKU}, which injects the prepended context into the parameters of an LM. Compared to above approaches, our approach yields more interpretable textual summary that can transfer across different LMs, and can be applied to black box LMs without requiring gradient updates. Prior work has studied textual compression for other tasks, such as political fact checking ~\cite{Chen2023ComplexCV} and instruction learning \cite{Yin2023DidYR}.&#10;&#10;Distillation / Goal Oriented Summarization&#10; Recent work introduces symbolic knowledge distillation~\cite{west-etal-2022-symbolic}, which transfers knowledge from a teacher model by generating a training dataset with the teacher model and train a student model on it. For better performance, they introduce critic criteria, which filter undesirable examples from generated training dataset. Such distillation technique has been applied for various applications including summarization~\cite{jung2023impossible}, which aims to generate high quality summaries while we optimize for generating effective summary for downstream LMs. One work that is similar to our setting is \cite{hsu-tan-2021-decision} which trains an extractive summarization model to optimize for prediction accuracy of a sentiment prediction model based on the summary.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1809.09600" label="1809.09600">
        <attvalues>
          <attvalue for="0" value="HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question&#10;  Answering" />
          <attvalue for="1" value="  Existing question answering (QA) datasets fail to train QA systems to perform&#10;complex reasoning and provide explanations for answers. We introduce HotpotQA,&#10;a new dataset with 113k Wikipedia-based question-answer pairs with four key&#10;features: (1) the questions require finding and reasoning over multiple&#10;supporting documents to answer; (2) the questions are diverse and not&#10;constrained to any pre-existing knowledge bases or knowledge schemas; (3) we&#10;provide sentence-level supporting facts required for reasoning, allowing QA&#10;systems to reason with strong supervision and explain the predictions; (4) we&#10;offer a new type of factoid comparison questions to test QA systems' ability to&#10;extract relevant facts and perform necessary comparison. We show that HotpotQA&#10;is challenging for the latest QA systems, and the supporting facts enable&#10;models to improve performance and make explainable predictions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14660" label="2010.14660">
        <attvalues>
          <attvalue for="0" value="DualTKB: A Dual Learning Bridge between Text and Knowledge Base" />
          <attvalue for="1" value="  In this work, we present a dual learning approach for unsupervised text to&#10;path and path to text transfers in Commonsense Knowledge Bases (KBs). We&#10;investigate the impact of weak supervision by creating a weakly supervised&#10;dataset and show that even a slight amount of supervision can significantly&#10;improve the model performance and enable better-quality transfers. We examine&#10;different model architectures, and evaluation metrics, proposing a novel&#10;Commonsense KB completion metric tailored for generative models. Extensive&#10;experimental results show that the proposed method compares very favorably to&#10;the existing baselines. This approach is a viable step towards a more advanced&#10;system for automatic KB construction/expansion and the reverse operation of KB&#10;conversion to coherent textual descriptions.&#10;" />
          <attvalue for="2" value="&#10;&#10;The automatic construction of Knowledge Bases (KBs) from text and the reverse operation of sentence generation from KBs are dual tasks that are both active research topics.&#10;&#10;The first task of automatic KB construction remains a significant challenge due to the difficulty of detecting parts of text representing meaningful facts and summarizing them in a systematic form. A simpler sub-task of KB completion, i.e., extending or filling-in missing nodes or edges, has also attracted the attention of the research community. For both tasks, the system needs to generate new or complete existing graph entities coherently, possibly matching to the already existing graph structure. &#10;The dual task of decoding the information from KB back to text is a valuable functionality. This enables knowledge transfer from potentially large complex graphs into a more descriptive, human-friendly output. This conditional generation is often seen as a step towards learning using KB as prior knowledge.&#10;&#10;In this work, we address the problem of KB construction/completion and the reverse task of KB decoding, but aim at a simpler objective: transferring a single sentence to a path, and generating text from a single KB path as its dual task. &#10;In terms of data, our focus will be on Commonsense KBs, derived from sets of commonsense facts expressed in natural language sentences \cite{lenat1995cyc, cambria2014senticnet, speer2017conceptnet, sap2019atomic}. They are represented as graphs where each edge is expressed as a tuple $(e_h,r,e_t)$ with head and tail nodes $e_h$ and $e_t$ composed of free-form text, connected with a relationship operator $r$; see ConceptNet from \cite{speer2017conceptnet} or ATOMIC in \cite{sap2019atomic} for recent and commonly used examples of commonsense KBs.&#10;&#10;We observe that to train such transfer model, an additional challenge comes from the lack of datasets with parallel text and KB facts, i.e., where text sentences and KBs edges are explicitly paired/labeled from one to another. However, there exist many datasets for each individual transfer domain. Therefore, successful approaches transferring text to KB and KB to text must be able to operate in unsupervised or (at best) weakly-supervised settings. &#10;We address this challenge by proposing a model trained under dual learning of translation/transfer from text to KB and from KB to text, we name DualTKB. This is similar in philosophy to dual learning in Neural Machine Translation \cite{dual_nips2016}, or unsupervised style transfer \cite{shen2017style, tian2018structured, dai2019style}.&#10;We design our model to be trained in completely unsupervised settings. However, we observed that even a slight supervision significantly boosts model performance and enables better-quality transfers. Therefore, we also describe a simple heuristic methodology to create weakly-supervised datasets given a text corpus and a commonsense KB.&#10;&#10;We must emphasize that our proposed dual learning method is not limited to commonsense KBs and can generalize to other domains/types of KBs such as biomedical KBs. Commonsense KBs, and particularly ConceptNet, are good starting points due to the nature of their composition. &#10;Since ConceptNet was partly extracted from free-form text originating from the Open Mind Common Sense (OMCS) list of commonsense fact sentences, its nodes are often composed of parts of sentences from OMCS. &#10;This allowed us to first explore whether the proposed method worked at all before evaluating a semi-supervised approach by creating a weak supervision from a mapping between ConceptNet triples and the original OMCS sentences. &#10;While KBs are often dense with short named entity descriptions for nodes, many nodes for commonsense KBs are parts of sentences, making them inherently sparse which impacts their performance as empirically studied by \cite{malaviya2020commonsense}.&#10;&#10;The evaluation of this type of transfer models is a challenge in itself. For this purpose, we selected a set of metrics to examine different facets of the system using our created weakly-supervised dataset. For path generation, we rely on a conventional KB completion task where the goal is to maximize the validity score of a tail entity $e_t$ given the pair $(e_h,r)$. For example, \cite{malaviya2020commonsense} addresses the challenges unique to commonsense KB completion due to sparsity and large numbers of nodes resulting from encoding commonsense facts.&#10;However, KB completion does not always equate generation of edges, with the exception of COMET from \cite{bosselut2019comet} that generates tail node $e_t$ given the pair $(e_h,r)$.&#10;&#10;Since repurposing generative models for conventional KB completion evaluation is difficult \cite{malaviya2020commonsense, bosselut2019comet}, we propose a new commonsense KB completion evaluation task for generative models. It is close in spirit to conventional KB completion, but comes with its own set of challenges. Moreover, we employ the Graph Edit Distance (GED) to examine the quality of the generated graph as a whole. For text generation, we rely on traditional NLP metrics such as BLEU and ROUGE.&#10;&#10;Following is a list of highlights of our paper contributions: (1) Propose a dual learning bridge between text and commonsense KB. Implement approach as unsupervised text-to-path and path-to-text transfers; (2) Construct a weakly-supervised dataset, and explore weak-supervision training. (3) Define a novel Commonsense KB completion metric tailored for generative models. (4) Investigate successfully multiple model architectures.&#10;&#10;Finally, in Fig.~\ref{fig:gengraph} and Fig.~\ref{fig:gentext} we present a few examples generated by our proposed model. Fig.~\ref{fig:gengraph} is a text to KB translation. Each sentence below the graph is independently transferred to a path, consisting of one edge tuple $(e_h,r,e_t)$. The whole path tuple is generated at once, with $e_h$, $r$, and $e_t$ taking a free-form not restricted to any predefined sets of entities or relations. &#10;This contrasts with many existing works operating on a limited discrete set of already-defined edges in a dense conventional KB. Once all the sentences are transferred, we observe that this set of generated edges forms a connected structure, implicitly merging some nodes to form a prototype of a Knowledge Graph (KG). &#10;&#10;Fig.~\ref{fig:gentext} shows the transfer from KB to text. Each path in the graph is converted to a sentence. There is overall diversity in the generated sentences styles. Moreover, the samples show that the generation process is more sophisticated than just a trivial path flattening (i.e., merging text from all edge parts followed by minimal edits). Therefore, the proposed approach can eventually become a part of a more sophisticated system converting graphs to a coherent textual story and vice versa. Additional examples are presented in Appendix \ref{sec:app_graph_gen}.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Generative Model Evaluation, Knowledge Representation, Commonsense Knowledge Bases, Text Transfer Learning" />
        </attvalues>
      </node>
      <node id="2407.00100" label="2407.00100">
        <attvalues>
          <attvalue for="0" value="Enhancing In-Context Learning via Implicit Demonstration Augmentation" />
          <attvalue for="1" value="  The emergence of in-context learning (ICL) enables large pre-trained language&#10;models (PLMs) to make predictions for unseen inputs without updating&#10;parameters. Despite its potential, ICL's effectiveness heavily relies on the&#10;quality, quantity, and permutation of demonstrations, commonly leading to&#10;suboptimal and unstable performance. In this paper, we tackle this challenge&#10;for the first time from the perspective of demonstration augmentation.&#10;Specifically, we start with enriching representations of demonstrations by&#10;leveraging their deep feature distribution. We then theoretically reveal that&#10;when the number of augmented copies approaches infinity, the augmentation is&#10;approximately equal to a novel logit calibration mechanism integrated with&#10;specific statistical properties. This insight results in a simple yet highly&#10;efficient method that significantly improves the average and worst-case&#10;accuracy across diverse PLMs and tasks. Moreover, our method effectively&#10;reduces performance variance among varying demonstrations, permutations, and&#10;templates, and displays the capability to address imbalanced class&#10;distributions.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large pre-trained language models (PLMs) have showcased exceptional abilities in in-context learning (ICL)~\cite{brown2020language,wang2023investigating,rubin-etal-2022-learning}, which assists the model in discerning the underlying patterns within demonstrations and make more accurate predictions~\cite{chan2022data,wu-etal-2023-self}. &#10;As a new paradigm, ICL offers compelling advantages, allowing for natural language interaction with PLMs~\cite{wei2022chain,yang2023supervised}, as well as reduced computational costs~\cite{li2023context,rubin-etal-2022-learning}. &#10;&#10;While promising, ICL's performance is highly dependent on provided demonstrations and templates~\cite{liu-etal-2022-makes,zhang-etal-2022-active,sorensen-etal-2022-information}, resulting in subpar and unstable performance. This promotes research aimed at improving the quality~\cite{rubin-etal-2022-learning,li-etal-2023-unified}, quantity~\cite{li2023context,choi2022prompt}, and permutations~\cite{lu-etal-2022-fantastically,tang2023found} of demonstrations. Other research avenues include prediction adjustment~\cite{zhao2021calibrate,han2022prototypical,fei-etal-2023-mitigating} and learning process design (e.g., channel models~\cite{min-etal-2022-noisy} and meta-training frameworks~\cite{min-etal-2022-metaicl}). Despite ongoing efforts, ICL still struggles with efficiently and reliably capturing sufficient knowledge from context, leaving performance stability as a persistent bottleneck.&#10;&#10;In this study, we propose enriching contextual knowledge for PLMs by augmenting demonstrations. We first attempt to enhance the representation of demonstrations by transforming them along semantic directions sampled from the deep feature space of demonstration examples, as depicted in Figure~\ref{demonaug}. This operation stems from the observation that the deep features in a network are usually&#10;linearized~\cite{bengio2013better,cheung2020modals,cho2016noisy}, implying the existence of numerous semantic directions within the deep feature space, hence potentially enabling us to incorporate richer contextual knowledge without extending input length. From this novel perspective, we theoretically prove that when the number of augmented pieces approaches infinity, its effect approximately equals a logit adjustment operation. Specifically, we derive a refined Softmax function that integrates the statistical properties of demonstrations.&#10;Consequently, rather than explicitly executing the augmentation procedure, we can efficiently conduct implicit demonstration augmentation using the derived prediction function, obtaining an improved ICL method with theoretical guidance. &#10;&#10;We conduct extensive experiments across seven PLMs and various classification tasks. The empirical results demonstrate that our approach remarkably enhances prediction accuracy and reduces performance variability across different demonstrations, permutations, and templates. &#10;Notably, our method is straightforward, effective, and generalizable, enabling seamless integration with other ICL methods to enhance their performance.&#10;&#10;Our contributions can be summarized as follows:&#10;\begin{itemize}[itemsep=2pt, topsep=2pt,parsep=2pt]&#10; \item We introduce Implicit Demonstration Augmentation-based ICL (IDAICL), a pioneering work that incorporates demonstration augmentation into ICL. Instead of solely enhancing demonstration quality, quantity, or order, our method explores context augmentation within the deep feature space, offering a new perspective to enrich demonstrations bypassing input length limitations.&#10;&#10;\item We theoretically establish that as the number of augmented pieces approaches infinity, our augmentation strategy approximates a logit-adjusted prediction function that integrates statistical properties derived from &#10; the input data distribution. Equipped with this function, IDAICL provides a straightforward yet theory-guided solution to enhance ICL.&#10;&#10;\item Extensive experiments conducted across diverse tasks and PLMs conclusively illustrate that IDAICL considerably improves average and worst-case accuracy compared to existing ICL methods. Moreover, it effectively enhances performance stability.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Demonstration Augmentation Techniques, Computer Science, Linguistics, Deep Learning Performance Enhancement, Mathematics, Statistics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2212.10375" label="2212.10375">
        <attvalues>
          <attvalue for="0" value="Self-Adaptive In-Context Learning: An Information Compression&#10;  Perspective for In-Context Example Selection and Ordering" />
          <attvalue for="1" value="  Despite the surprising few-shot performance of in-context learning (ICL), it&#10;is still a common practice to randomly sample examples to serve as context.&#10;This paper advocates a new principle for ICL: self-adaptive in-context&#10;learning. The self-adaption mechanism is introduced to help each sample find an&#10;in-context example permutation (i.e., selection and ordering) that can derive&#10;the correct prediction, thus maximizing performance. To validate the&#10;effectiveness of self-adaptive ICL, we propose a general select-then-rank&#10;framework and instantiate it with new selection and ranking algorithms. Upon&#10;extensive evaluation on eight different NLP datasets, our self-adaptive ICL&#10;method achieves a 40% relative improvement over the common practice setting.&#10;Further analysis reveals the enormous potential of self-adaptive ICL that it&#10;might be able to close the gap between ICL and finetuning given more advanced&#10;algorithms. Our code is released to facilitate future research in this area:&#10;https://github.com/Shark-NLP/self-adaptive-ICL&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.04320" label="2305.04320">
        <attvalues>
          <attvalue for="0" value="Unified Demonstration Retriever for In-Context Learning" />
          <attvalue for="1" value="  In-context learning is a new learning paradigm where a language model&#10;conditions on a few input-output pairs (demonstrations) and a test input, and&#10;directly outputs the prediction. It has been shown highly dependent on the&#10;provided demonstrations and thus promotes the research of demonstration&#10;retrieval: given a test input, relevant examples are retrieved from the&#10;training set to serve as informative demonstrations for in-context learning.&#10;While previous works focus on training task-specific retrievers for several&#10;tasks separately, these methods are often hard to transfer and scale on various&#10;tasks, and separately trained retrievers incur a lot of parameter storage and&#10;deployment cost. In this paper, we propose Unified Demonstration Retriever&#10;(\textbf{UDR}), a single model to retrieve demonstrations for a wide range of&#10;tasks. To train UDR, we cast various tasks' training signals into a unified&#10;list-wise ranking formulation by language model's feedback. Then we propose a&#10;multi-task list-wise ranking training framework, with an iterative mining&#10;strategy to find high-quality candidates, which can help UDR fully incorporate&#10;various tasks' signals. Experiments on 30+ tasks across 13 task families and&#10;multiple data domains show that UDR significantly outperforms baselines.&#10;Further analyses show the effectiveness of each proposed component and UDR's&#10;strong ability in various scenarios including different LMs (1.3B - 175B),&#10;unseen datasets, varying demonstration quantities, etc.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.07712" label="2310.07712">
        <attvalues>
          <attvalue for="0" value="Found in the Middle: Permutation Self-Consistency Improves Listwise&#10;  Ranking in Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) exhibit positional bias in how they use context,&#10;which especially complicates listwise ranking. To address this, we propose&#10;permutation self-consistency, a form of self-consistency over ranking list&#10;outputs of black-box LLMs. Our key idea is to marginalize out different list&#10;orders in the prompt to produce an order-independent ranking with less&#10;positional bias. First, given some input prompt, we repeatedly shuffle the list&#10;in the prompt and pass it through the LLM while holding the instructions the&#10;same. Next, we aggregate the resulting sample of rankings by computing the&#10;central ranking closest in distance to all of them, marginalizing out prompt&#10;order biases in the process. Theoretically, we prove the robustness of our&#10;method, showing convergence to the true ranking in the presence of random&#10;perturbations. Empirically, on five list-ranking datasets in sorting and&#10;passage reranking, our approach improves scores from conventional inference by&#10;up to 7-18% for GPT-3.5 and 8-16% for LLaMA v2 (70B), surpassing the previous&#10;state of the art in passage reranking. Our code is at&#10;https://github.com/castorini/perm-sc.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.10183" label="2205.10183">
        <attvalues>
          <attvalue for="0" value="Prototypical Calibration for Few-shot Learning of Language Models" />
          <attvalue for="1" value="  In-context learning of GPT-like models has been recognized as fragile across&#10;different hand-crafted templates, and demonstration permutations. In this work,&#10;we propose prototypical calibration to adaptively learn a more robust decision&#10;boundary for zero- and few-shot classification, instead of greedy decoding.&#10;Concretely, our method first adopts Gaussian mixture distribution to estimate&#10;the prototypical clusters for all categories. Then we assign each cluster to&#10;the corresponding label by solving a weighted bipartite matching problem. Given&#10;an example, its prediction is calibrated by the likelihood of prototypical&#10;clusters. Experimental results show that prototypical calibration yields a&#10;substantial improvement on a diverse set of tasks. Extensive analysis across&#10;different scales also indicates that our method calibrates the decision&#10;boundary as expected, greatly improving the robustness of GPT to templates,&#10;permutations, and class imbalance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.19148" label="2305.19148">
        <attvalues>
          <attvalue for="0" value="Mitigating Label Biases for In-context Learning" />
          <attvalue for="1" value="  Various design settings for in-context learning (ICL), such as the choice and&#10;order of the in-context examples, can bias a model toward a particular&#10;prediction without being reflective of an understanding of the task. While many&#10;studies discuss these design choices, there have been few systematic&#10;investigations into categorizing them and mitigating their impact. In this&#10;work, we define a typology for three types of label biases in ICL for text&#10;classification: vanilla-label bias, context-label bias, and domain-label bias&#10;(which we conceptualize and detect for the first time).&#10;  Our analysis demonstrates that prior label bias calibration methods fall&#10;short of addressing all three types of biases. Specifically, domain-label bias&#10;restricts LLMs to random-level performance on many tasks regardless of the&#10;choice of in-context examples. To mitigate the effect of these biases, we&#10;propose a simple bias calibration method that estimates a language model's&#10;label bias using random in-domain words from the task corpus. After controlling&#10;for this estimated bias when making predictions, our novel domain-context&#10;calibration significantly improves the ICL performance of GPT-J and GPT-3 on a&#10;wide range of tasks. The gain is substantial on tasks with large domain-label&#10;bias (up to 37% in Macro-F1). Furthermore, our results generalize to models&#10;with different scales, pretraining methods, and manually-designed task&#10;instructions, showing the prevalence of label biases in ICL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1207.4404" label="1207.4404">
        <attvalues>
          <attvalue for="0" value="Better Mixing via Deep Representations" />
          <attvalue for="1" value="  It has previously been hypothesized, and supported with some experimental&#10;evidence, that deeper representations, when well trained, tend to do a better&#10;job at disentangling the underlying factors of variation. We study the&#10;following related conjecture: better representations, in the sense of better&#10;disentangling, can be exploited to produce faster-mixing Markov chains.&#10;Consequently, mixing would be more efficient at higher levels of&#10;representation. To better understand why and how this is happening, we propose&#10;a secondary conjecture: the higher-level samples fill more uniformly the space&#10;they occupy and the high-density manifolds tend to unfold when represented at&#10;higher levels. The paper discusses these hypotheses and tests them&#10;experimentally through visualization and measurements of mixing and&#10;interpolating between samples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.20729" label="2407.20729">
        <attvalues>
          <attvalue for="0" value="Adapting Safe-for-Work Classifier for Malaysian Language Text: Enhancing&#10;  Alignment in LLM-Ops Framework" />
          <attvalue for="1" value="  As large language models (LLMs) become increasingly integrated into&#10;operational workflows (LLM-Ops), there is a pressing need for effective&#10;guardrails to ensure safe and aligned interactions, including the ability to&#10;detect potentially unsafe or inappropriate content across languages. However,&#10;existing safe-for-work classifiers are primarily focused on English text. To&#10;address this gap for the Malaysian language, we present a novel safe-for-work&#10;text classifier tailored specifically for Malaysian language content. By&#10;curating and annotating a first-of-its-kind dataset of Malaysian text spanning&#10;multiple content categories, we trained a classification model capable of&#10;identifying potentially unsafe material using state-of-the-art natural language&#10;processing techniques. This work represents an important step in enabling safer&#10;interactions and content filtering to mitigate potential risks and ensure&#10;responsible deployment of LLMs. To maximize accessibility and promote further&#10;research towards enhancing alignment in LLM-Ops for the Malaysian context, the&#10;model is publicly released at&#10;https://huggingface.co/malaysia-ai/malaysian-sfw-classifier.&#10;" />
          <attvalue for="2" value="&#10;&#10;The AI field, especially natural language processing, \cite{markov2023holistic} has seen remarkable progress with significant breakthroughs like transformer-based architectures \cite{vaswani2023attentionneed}, multimodality integration to chatbot applications, and reinforcement learning from human feedback. This has led to the rise of open-domain dialogue systems, known as chatbots or conversational agents, which are now increasingly integrated into our daily lives.&#10;&#10;Due to the nature of how large language models are trained, using internet data, it is prevalent that there may be harmful contents included. However, as users continues engaging with these chatbots, exposure to harmful and provocative text can have significant adverse effects, impacting individuals' mental well-being, relationships, and emotional state. Therefore, ensuring safe and beneficial interactions has become critically important.&#10;&#10;The scarcity of data for identifying not safe for work content, particularly in the Malay language, hinders the advancement of undesired content filtration. Past work such as \cite{markov2023holistic,qiu2024facilitatingpornographictextdetection} have laid foundation in terms of AI moderation using large language models, but there is still no not safe for work task in malay language. In this paper, we address this challenge by initiating the data gathering process to create a comprehensive dataset of harmful texts. Our methodology involves mining data representative of harmful text categories. Our categorization includes the following labels: pornography, harassment, sexist, racist, religious insult, self-harm, psychiatric or mental illness, and safe for work.&#10;&#10;We aim to create a robust classifier tailored for Malaysian language text, enhancing the alignment of our large language model operations framework with safety and ethical standards. This classifier serves as a necessary guardrail within the LLM-Ops framework, providing a cost-effective solution for ensuring safe AI. By systematically identifying and filtering out inappropriate content, this classifier will help create a safe and respectful interaction environment for users.&#10;&#10;Furthermore, to the best of our knowledge, there is currently no existing local dataset for the Malaysian language that addresses these specific categories of harmful content. Our work thus represents a pioneering effort in developing and applying this crucial safety measure.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Safety, Artificial Intelligence, Content Filtering, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2403.13250" label="2403.13250">
        <attvalues>
          <attvalue for="0" value="Facilitating Pornographic Text Detection for Open-Domain Dialogue&#10;  Systems via Knowledge Distillation of Large Language Models" />
          <attvalue for="1" value="  Pornographic content occurring in human-machine interaction dialogues can&#10;cause severe side effects for users in open-domain dialogue systems. However,&#10;research on detecting pornographic language within human-machine interaction&#10;dialogues is an important subject that is rarely studied. To advance in this&#10;direction, we introduce CensorChat, a dialogue monitoring dataset aimed at&#10;detecting whether the dialogue session contains pornographic content. To this&#10;end, we collect real-life human-machine interaction dialogues in the wild and&#10;break them down into single utterances and single-turn dialogues, with the last&#10;utterance spoken by the chatbot. We propose utilizing knowledge distillation of&#10;large language models to annotate the dataset. Specifically, first, the raw&#10;dataset is annotated by four open-source large language models, with the&#10;majority vote determining the label. Second, we use ChatGPT to update the empty&#10;label from the first step. Third, to ensure the quality of the validation and&#10;test sets, we utilize GPT-4 for label calibration. If the current label does&#10;not match the one generated by GPT-4, we employ a self-criticism strategy to&#10;verify its correctness. Finally, to facilitate the detection of pornographic&#10;text, we develop a series of text classifiers using a pseudo-labeled dataset.&#10;Detailed data analysis demonstrates that leveraging knowledge distillation&#10;techniques with large language models provides a practical and cost-efficient&#10;method for developing pornographic text detectors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.07150" label="2107.07150">
        <attvalues>
          <attvalue for="0" value="Tailor: Generating and Perturbing Text with Semantic Controls" />
          <attvalue for="1" value="  Controlled text perturbation is useful for evaluating and improving model&#10;generalizability. However, current techniques rely on training a model for&#10;every target perturbation, which is expensive and hard to generalize. We&#10;present Tailor, a semantically-controlled text generation system. Tailor builds&#10;on a pretrained seq2seq model and produces textual outputs conditioned on&#10;control codes derived from semantic representations. We craft a set of&#10;operations to modify the control codes, which in turn steer generation towards&#10;targeted attributes. These operations can be further composed into higher-level&#10;ones, allowing for flexible perturbation strategies. We demonstrate the&#10;effectiveness of these perturbations in multiple applications. First, we use&#10;Tailor to automatically create high-quality contrast sets for four distinct&#10;natural language processing (NLP) tasks. These contrast sets contain fewer&#10;spurious artifacts and are complementary to manually annotated ones in their&#10;lexical diversity. Second, we show that Tailor perturbations can improve model&#10;generalization through data augmentation. Perturbing just 2% of training data&#10;leads to a 5.8-point gain on an NLI challenge set measuring reliance on&#10;syntactic heuristics.&#10;" />
          <attvalue for="2" value="&#10;&#10;\revised{Semantic perturbation through controlled text generation}&#10;modifies sentences to match certain target attributes, such as verb tense or sentiment (\eg positive$arrow$negative). &#10;It has been widely applied to a variety of tasks, \eg \revised{changing text style}~\cite{reid2021lewis}, &#10;mitigating dataset biases~\cite{gardner2021competency}, explaining model behaviors~\cite{ross2020explaining}, and improving model generalization~\cite{teney2020learning,polyjuice:acl21}.&#10;Existing efforts train task-specific generators, \eg training a sentiment style transferer requires instances annotated with positive and negative labels~\cite{madaan2020generate}.&#10;As a result, they require costly annotated data and re-training for every task of interest.&#10;&#10;This work introduces \sysname, a system that supports application-agnostic perturbations. At its core is a controlled generator (\S\ref{sec:generator-design}) that flexibly generates outputs from target semantic attributes, \revised{which we represent through structured control codes in the inputs.}&#10;As shown in Figure~\ref{fig:teaser}, \revised{these control codes} build on the PropBank semantic analysis \cite{palmer-etal-2005-proposition} of the original sentence: \revised{For each argument span, the semantic role and &#10;keyword control codes specify the desired semantic content for the span at varying levels of granularity.}&#10;To encourage control code following,&#10;we train \revised{the \sysname generator} with unlikelihood training~\cite{Welleck2020Neural} to penalize generations that are not aligned with designated \revised{control codes}.&#10;&#10;The use of semantic role \revised{control codes} allows \sysname to perform fine-grained changes to individual arguments in a sentence (\eg one can change only the \optag{PATIENT} in Figure~\ref{fig:teaser}).&#10;Instead of &#10;\revised{specifying a perturbation with a generic target property&#10;(\eg positive\arrownegative), we can &#10;specify the linguistic transformation&#10;used to achieve the property}&#10;(\eg changing sentiment through negation or antonym replacement).&#10;Making such fine-grained perturbations &#10;\revised{allows for more careful evaluation and improvement of}&#10;models' language understanding~\cite{kaushik2019learning, polyjuice:acl21}. &#10;&#10;To highlight the perturbations facilitated by \sysname,&#10;we craft a list of primitive perturbation operations (\sect{sec:perturbations}) on inputs to the generator; these can be easily composed to achieve more complex perturbations.&#10;In Figure~\ref{fig:teaser}, \sysname transforms sentence A to D through a series of perturbations:&#10;syntactic rewriting (changing verb tense), then sentence expansion (extending ``the athlete''), and finally data recombination (\ie generating new text that contains ``in'' but follows the \optag{TEMPORAL} control).&#10;Compared to existing approaches that require training a separate model for every step or annotating a dataset that represents this transformation end-to-end, such compositions make \sysname more cost-effective and generalizable.&#10;In fact, on nine fine-grained and compositional StylePTB perturbations~\cite{Lyu2021StylePTBAC}, \sysname achieves performance compatible with task-specific baselines, and even outperforms them on five transfers (\sect{sec:appendix-style-transfer}).&#10;&#10;\sysname's flexible \revised{and human-readable} control codes allow for broad, easily extendable applicability.&#10;We demonstrate its utility in evaluating and improving NLP model robustness, showing that \sysname can help replicate existing contrast sets on four diverse tasks.&#10;By abstracting manual perturbation types in prior work into perturbation strategies with \sysname, we &#10;\revised{can apply}&#10;the changes to larger datasets while saving manual annotation efforts.&#10;Our analysis suggests that these contrast sets not only have high rates of validity, but also reduce spurious &#10;\revised{artifacts compared to the original evaluation datasets}.&#10;In addition, \sysname-produced contrast sets complement human annotated ones in terms of lexical diversity: only $\sim$10\% of their unique tokens overlap with manually created contrast sets.&#10;We also explore \sysname's utility in data augmentation. We find that augmenting training data with just&#10;\revised{$\sim$2\% of \sysname perturbations} improves the robustness of natural language inference (NLI) models to inference heuristics, increasing performance on the HANS evaluation set~\cite{mccoy2019right} by an average of {5.81} points and outperforming a previous syntactic augmentation method for NLI.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Model Generalizability, Text Generation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2004.09034" label="2004.09034">
        <attvalues>
          <attvalue for="0" value="Learning What Makes a Difference from Counterfactual Examples and&#10;  Gradient Supervision" />
          <attvalue for="1" value="  One of the primary challenges limiting the applicability of deep learning is&#10;its susceptibility to learning spurious correlations rather than the underlying&#10;mechanisms of the task of interest. The resulting failure to generalise cannot&#10;be addressed by simply using more data from the same distribution. We propose&#10;an auxiliary training objective that improves the generalization capabilities&#10;of neural networks by leveraging an overlooked supervisory signal found in&#10;existing datasets. We use pairs of minimally-different examples with different&#10;labels, a.k.a counterfactual or contrasting examples, which provide a signal&#10;indicative of the underlying causal structure of the task. We show that such&#10;pairs can be identified in a number of existing datasets in computer vision&#10;(visual question answering, multi-label image classification) and natural&#10;language processing (sentiment analysis, natural language inference). The new&#10;training objective orients the gradient of a model's decision function with&#10;pairs of counterfactual examples. Models trained with this technique&#10;demonstrate improved performance on out-of-distribution test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.00288" label="2101.00288">
        <attvalues>
          <attvalue for="0" value="Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and&#10;  Improving Models" />
          <attvalue for="1" value="  While counterfactual examples are useful for analysis and training of NLP&#10;models, current generation methods either rely on manual labor to create very&#10;few counterfactuals, or only instantiate limited types of perturbations such as&#10;paraphrases or word substitutions. We present Polyjuice, a general-purpose&#10;counterfactual generator that allows for control over perturbation types and&#10;locations, trained by finetuning GPT-2 on multiple datasets of paired&#10;sentences. We show that Polyjuice produces diverse sets of realistic&#10;counterfactuals, which in turn are useful in various distinct applications:&#10;improving training and evaluation on three different tasks (with around 70%&#10;less annotation effort than manual generation), augmenting state-of-the-art&#10;explanation techniques, and supporting systematic counterfactual error analysis&#10;by revealing behaviors easily missed by human experts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Some prior work in training and evaluation relies on humans to generate counterfactuals from scratch~\cite{gardner2020contrast, teney2020learning, kaushik2019learning}. &#10;Our experiments in \S\ref{sec:app_label} indicate that asking humans to label \sysname counterfactuals yields similar or better results at a lower cost, which motivates an exploration of a mixture of manual and semi-automated generation. &#10;Similarly, prior work on analysis relies on experts to create individual counterfactuals or perturbation functions~\cite{wu2019errudite, checklist:acl20}. &#10;In \S\ref{sec:app_err_analysis}, we show that \sysname enhances current practice by generating multiple counterfactuals that might have been overlooked, and by providing abstractions that allow for new kinds of analyses.&#10;&#10;Prior work on automatically generating counterfactuals typically has a narrower scope in terms of the relationships $x \veryshortarrow \xp$.&#10;For example, adversarial generators aim to maintain semantics while changing model predictions~\cite{ribeiro2018semantically, iyyer2018adversarial, li2020contextualized}, whereas concurrent work to our own~\cite{madaan2020generate, ross2020explaining} automatically generates $\xp$ that change predictions for explanation or analysis, with no constraints on semantics.&#10;However, as shown in \S\ref{sec:app_label}--\S\ref{sec:app_err_analysis}, a mix of label-preserving and label-flipping counterfactuals generated by \sysname is quite useful for training, evaluation, explanation, and analysis. &#10;Further, general-purpose counterfactuals may lead to serendipitous discoveries (\S\ref{sec:app_err_analysis}), especially as \sysname is not fine-tuned to the target domain (and thus less liable to merely replicate what is already there).&#10;Finally, by allowing control through \tagstrs and [BLANK]s, \sysname supports human-generator collaboration, where a person specifies desired changes (\eg perturb the sentence subject).&#10;Such collaboration is hard to imagine using automatic generators with no control, or with coarser control through predefined style attributes or labels~\cite{madaan-etal-2020-politeness, malmi-etal-2020-unsupervised}. To our knowledge, prior work on controlled generation~\cite{ctrl, pplm} does not address counterfactual generation.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2012.04698" label="2012.04698">
        <attvalues>
          <attvalue for="0" value="Generate Your Counterfactuals: Towards Controlled Counterfactual&#10;  Generation for Text" />
          <attvalue for="1" value="  Machine Learning has seen tremendous growth recently, which has led to larger&#10;adoption of ML systems for educational assessments, credit risk, healthcare,&#10;employment, criminal justice, to name a few. The trustworthiness of ML and NLP&#10;systems is a crucial aspect and requires a guarantee that the decisions they&#10;make are fair and robust. Aligned with this, we propose a framework GYC, to&#10;generate a set of counterfactual text samples, which are crucial for testing&#10;these ML systems. Our main contributions include a) We introduce GYC, a&#10;framework to generate counterfactual samples such that the generation is&#10;plausible, diverse, goal-oriented, and effective, b) We generate counterfactual&#10;samples, that can direct the generation towards a corresponding condition such&#10;as named-entity tag, semantic role label, or sentiment. Our experimental&#10;results on various domains show that GYC generates counterfactual text samples&#10;exhibiting the above four properties. GYC generates counterfactuals that can&#10;act as test cases to evaluate a model and any text debiasing algorithm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.04319" label="1908.04319">
        <attvalues>
          <attvalue for="0" value="Neural Text Generation with Unlikelihood Training" />
          <attvalue for="1" value="  Neural text generation is a key tool in natural language applications, but it&#10;is well known there are major problems at its core. In particular, standard&#10;likelihood training and decoding leads to dull and repetitive outputs. While&#10;some post-hoc fixes have been proposed, in particular top-$k$ and nucleus&#10;sampling, they do not address the fact that the token-level probabilities&#10;predicted by the model are poor. In this paper we show that the likelihood&#10;objective itself is at fault, resulting in a model that assigns too much&#10;probability to sequences containing repeats and frequent words, unlike those&#10;from the human training distribution. We propose a new objective, unlikelihood&#10;training, which forces unlikely generations to be assigned lower probability by&#10;the model. We show that both token and sequence level unlikelihood training&#10;give less repetitive, less dull text while maintaining perplexity, giving&#10;superior generations using standard greedy or beam search. According to human&#10;evaluations, our approach with standard beam search also outperforms the&#10;currently popular decoding methods of nucleus sampling or beam blocking, thus&#10;providing a strong alternative to existing techniques.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;Neural Text Degeneration&#10;&#10;Recently, several papers have observed various forms of neural text degeneration, especially in open-ended generation tasks.&#10;In dialogue, it has been shown that there is a mismatch between model and human word distributions, &#10;where generative models are more likely to output frequent words, but less likely to produce&#10;rare words compared to humans. For example, this was observed across all generative&#10;models submitted to the ConvAI2 NeurIPS 2018 competition \cite{dinan2019second}.&#10;In language modeling, the work of \cite{holtzman2019curious} highlighted problems with the word frequency distribution and level of repetition in model generations compared to human text. These issues are not remedied by simply increasing the amount of the training data; e.g. large-scale GPT-2 language models \cite{radford2019language} display the same issues. &#10;&#10;Improved Decoding Algorithms&#10;&#10;Several methods have been proposed to rectify these issues. The primary ones involve changing the decoding method to a sophisticated beam search variant or to stochastic decoding, e.g. sampling.&#10;Different variants of beam search have been explored \cite{li2016simple,vijayakumar2018diverse,kulikov2018importance,holtzman2018learning} which can decrease a model's level of repetition by selecting candidates that are unlike previously chosen ones. Separately, hard or soft beam blocking has been investigated \cite{paulus2017deep,klein2017opennmt}, whereby previously generated $n$-grams are blocked from subsequent generation. This approach is often used in dialogue generation, fixing some token or phrase level repetitions but removing repetitions that would naturally occur in human text. &#10;&#10;The second major approach is that of sampling from the model at generation time. Top $k$-sampling \cite{fan2018hierarchical} and nucleus sampling \cite{holtzman2019curious} are two methods that sample sequences based on a function of&#10;the predicted next token probability distribution given by the model. &#10;Both approaches vastly improve the repetition issue, as the randomization often reduces the number of duplicate tokens in a decoded sequence, even if highly scored paths under the model (represented by beam search candidates) contain repetitions. &#10;However, as the underlying model is unchanged, it often prefers semantically similar phrasing, depending on the temperature parameter of the sampling \cite{holtzman2019curious}.&#10;Furthermore, this solution is less relevant in less open-ended tasks such as machine translation, where beam search variants are the preferred method.&#10;Ideally we would like a model that can work with both beam and sampling decoding methods.&#10;&#10;Improved Learning Algorithms&#10;&#10;The proposed learning criteria are closely related to structured output prediction methods in which the goal is to increase the scores assigned by a model to true examples while decreasing those assigned to negative examples often generated by the model itself. Some representative algorithms include structured perceptron~\cite{collins-2002-discriminative}, energy-based models~\cite{lecun2006tutorial} and more recently reflective likelihood~\cite{dieng2018learning}. A particular variant in this family of algorithms, called negative training, was recently used by \cite{he2019negative} to prevent generic and malicious responses in dialogue models. Similarly, these structured prediction algorithms with neural language models have been applied to machine translation in recent years by \cite{shen2015minimum} and \cite{edunov2017classical}. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2104.05196" label="2104.05196">
        <attvalues>
          <attvalue for="0" value="StylePTB: A Compositional Benchmark for Fine-grained Controllable Text&#10;  Style Transfer" />
          <attvalue for="1" value="  Text style transfer aims to controllably generate text with targeted&#10;stylistic changes while maintaining core meaning from the source sentence&#10;constant. Many of the existing style transfer benchmarks primarily focus on&#10;individual high-level semantic changes (e.g. positive to negative), which&#10;enable controllability at a high level but do not offer fine-grained control&#10;involving sentence structure, emphasis, and content of the sentence. In this&#10;paper, we introduce a large-scale benchmark, StylePTB, with (1) paired&#10;sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical,&#10;syntactic, semantic, and thematic transfers of text, as well as (2)&#10;compositions of multiple transfers which allow modeling of fine-grained&#10;stylistic changes as building blocks for more complex, high-level transfers. By&#10;benchmarking existing methods on StylePTB, we find that they struggle to model&#10;fine-grained changes and have an even more difficult time composing multiple&#10;styles. As a result, StylePTB brings novel challenges that we hope will&#10;encourage future research in controllable text style transfer, compositional&#10;models, and learning disentangled representations. Solving these challenges&#10;would present important steps towards controllable text generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.00958" label="1912.00958">
        <attvalues>
          <attvalue for="0" value="Language Model Bootstrapping Using Neural Machine Translation For&#10;  Conversational Speech Recognition" />
          <attvalue for="1" value="  Building conversational speech recognition systems for new languages is&#10;constrained by the availability of utterances that capture user-device&#10;interactions. Data collection is both expensive and limited by the speed of&#10;manual transcription. In order to address this, we advocate the use of neural&#10;machine translation as a data augmentation technique for bootstrapping language&#10;models. Machine translation (MT) offers a systematic way of incorporating&#10;collections from mature, resource-rich conversational systems that may be&#10;available for a different language. However, ingesting raw translations from a&#10;general purpose MT system may not be effective owing to the presence of named&#10;entities, intra sentential code-switching and the domain mismatch between the&#10;conversational data being translated and the parallel text used for MT&#10;training. To circumvent this, we explore the following domain adaptation&#10;techniques: (a) sentence embedding based data selection for MT training, (b)&#10;model finetuning, and (c) rescoring and filtering translated hypotheses. Using&#10;Hindi as the experimental testbed, we translate US English utterances to&#10;supplement the transcribed collections. We observe a relative word error rate&#10;reduction of 7.8-15.6%, depending on the bootstrapping phase. Fine grained&#10;analysis reveals that translation particularly aids the interaction scenarios&#10;which are underrepresented in the transcribed data.&#10;" />
          <attvalue for="2" value="&#10;Bootstrapping an automatic speech recognition (ASR) system for a new language involves significant data collection and transcription overhead. For factored ASR systems, where the acoustic model (AM) and language model (LM) are trained independently, the LM can be trained with additional text-only corpora to boost performance. This is especially helpful during the initial stages of model development. For a new language the typical supplemental LM sources include Wikipedia, news portals, blogs etc., which can be incorporated along with the limited transcribed data to circumvent the issue of cold start. However, for conversational agents like Alexa, Siri the utterances are usually short, goal directed and contain several named entities like song title, artist name etc., e.g. Play Moonlight Sonata by Beethoven. This informal interaction style, characteristic of conversational data, is absent in online text sources, thereby rendering them less effective for this task. As a result, LM building relies mostly on transcribed data and its performance is restricted by the speed of manual transcription and annotation.&#10;&#10;There has been a growing interest in the area of data augmentation for ASR language modeling. Previous studies include training a recurrent neural network (RNN) based LM on transcriptions and using it to generate synthetic samples for augmentation \cite{GorinIS2016}. SeqGAN, a generative adversarial model for sequences, has been employed for pretraining a code-switched LM \cite{SGargIS2018}. However, a precondition for the successful generalization of these neural generative models is the availability of a substantial amount of in-domain utterance text for training, which itself is the bottleneck during the bootstrapping phase. &#10;&#10;Utterances from mature conversational systems, for example in English provide a rich source of information. They are both in-domain, since they capture actual user interaction patterns of varying complexity, and large-scale, owing to prolonged usage. Translation offers an elegant and cost-effective solution for leveraging this existing data. Devising techniques for systematically incorporating translated data can be instrumental for achieving the rapid language expansion goal for ASR, by alleviating the prohibitively high requirements for data collection during bootstrapping. &#10;&#10;The area of machine translation has witnessed sustained research efforts \cite{CHO2014,Bahdanau2015JOINTLY,GNMT}. It is also amongst the first success stories of the end-to-end neural paradigm for sequence modeling. Conventional phrase based statistical machine translation (SMT) \cite{PBSMT} has shown to be outperformed by attention based recurrent encoder-decoder models \cite{GNMT} and transformer networks comprising self-attention and feed forward network blocks \cite{TRANSFORMER}. &#10;&#10;Data augmentation via SMT has been explored in the past for keyword spotting \cite{GorinIS2016} and ASR \cite{ICELANDIC, ENGMAND}. These studies primarily focus on incorporating raw translation output as a component in the LM. However, in our initial experiments we observed that directly ingesting translations generated from off-the-shelf MT models results in a suboptimal performance for conversational data. This could be attributed, in part, to the domain mismatch between the MT training data comprising parallel text from web sources and the informal style interaction data used for translation. This observation of MT output being sensitive to the mismatch in training and inference data distributions is consistent with previous studies on MT adaptation \cite{ADAPTATIONSURVEY}. &#10;&#10;Statistical post-editing for improving the quality of SMT outputs has been investigated \cite{ROM}. In a recent work on bootstrapping natural language understanding systems using translations \cite{MTNLP}, SMT is employed for generating initial translations, followed by the use of source-target alignments to retain and resample named entities. These post-editing approaches can minimize the undesired named entities conversions for SMT, yet the bigger issue of domain mismatch still remains open. &#10;&#10;In this work, we explore the synergies between neural machine translation and speech recognition for data augmentation. We work towards bootstrapping Hindi ASR system. Along with the limited availability of representative transcribed data, an additional challenge in this setting is that of code-switching. In typical Hindi utterances people often code mix with English within a sentence. The techniques explored in this work are however generic, and Hindi is chosen as a testbed for its complexity.&#10;&#10;We evaluate different architectures for building English to Hindi (EN$arrow$HI) translation models and elaborate on the pitfalls associated with using off-the-shelf translations. Some initial gains are observed by inferring alignments from attention weights, an approach that enables preserving and resampling the named entities. This technique is further extended to simulate code-switching in the translated data.&#10;&#10;We then delve into the deeper issue of domain inconsistency. To this end, we develop a data selection strategy for MT model training based on in-domain similarity. This is an extension of \cite{WANG2017a} for the fully unsupervised setting. We also assess model finetuning approach by adding parallel in-domain synthetic pairs. For further adaptation, a statistical LM built using transcribed data is used for rescoring the decoded translation beams. Finally, different quality metrics are compared for retaining only the high quality translations in the final translation component.&#10;&#10;A comparative evaluation of the translation-augmented LM is performed against baselines built from only transcribed data at various stages of bootstrapping. To the best of our knowledge, this work is the first investigation of the efficacy and challenges associated with neural machine translation for conversational speech recognition. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Conversational Speech Recognition, Computer Science, Linguistics, Artificial Intelligence, Language Model Bootstrapping, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1809.01962" label="1809.01962">
        <attvalues>
          <attvalue for="0" value="Code-switched Language Models Using Dual RNNs and Same-Source&#10;  Pretraining" />
          <attvalue for="1" value="  This work focuses on building language models (LMs) for code-switched text.&#10;We propose two techniques that significantly improve these LMs: 1) A novel&#10;recurrent neural network unit with dual components that focus on each language&#10;in the code-switched text separately 2) Pretraining the LM using synthetic text&#10;from a generative model estimated using the training data. We demonstrate the&#10;effectiveness of our proposed techniques by reporting perplexities on a&#10;Mandarin-English task and derive significant reductions in perplexity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.09119" label="1805.09119">
        <attvalues>
          <attvalue for="0" value="Selecting Machine-Translated Data for Quick Bootstrapping of a Natural&#10;  Language Understanding System" />
          <attvalue for="1" value="  This paper investigates the use of Machine Translation (MT) to bootstrap a&#10;Natural Language Understanding (NLU) system for a new language for the use case&#10;of a large-scale voice-controlled device. The goal is to decrease the cost and&#10;time needed to get an annotated corpus for the new language, while still having&#10;a large enough coverage of user requests. Different methods of filtering MT&#10;data in order to keep utterances that improve NLU performance and&#10;language-specific post-processing methods are investigated. These methods are&#10;tested in a large-scale NLU task with translating around 10 millions training&#10;utterances from English to German. The results show a large improvement for&#10;using MT data over a grammar-based and over an in-house data collection&#10;baseline, while reducing the manual effort greatly. Both filtering and&#10;post-processing approaches improve results further.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.10661" label="2309.10661">
        <attvalues>
          <attvalue for="0" value="NusaWrites: Constructing High-Quality Corpora for Underrepresented and&#10;  Extremely Low-Resource Languages" />
          <attvalue for="1" value="  Democratizing access to natural language processing (NLP) technology is&#10;crucial, especially for underrepresented and extremely low-resource languages.&#10;Previous research has focused on developing labeled and unlabeled corpora for&#10;these languages through online scraping and document translation. While these&#10;methods have proven effective and cost-efficient, we have identified&#10;limitations in the resulting corpora, including a lack of lexical diversity and&#10;cultural relevance to local communities. To address this gap, we conduct a case&#10;study on Indonesian local languages. We compare the effectiveness of online&#10;scraping, human translation, and paragraph writing by native speakers in&#10;constructing datasets. Our findings demonstrate that datasets generated through&#10;paragraph writing by native speakers exhibit superior quality in terms of&#10;lexical diversity and cultural content. In addition, we present the&#10;\datasetname{} benchmark, encompassing 12 underrepresented and extremely&#10;low-resource languages spoken by millions of individuals in Indonesia. Our&#10;empirical experiment results using existing multilingual large language models&#10;conclude the need to extend these models to more underrepresented languages. We&#10;release the NusaWrites dataset at https://github.com/IndoNLP/nusa-writes.&#10;" />
          <attvalue for="2" value="&#10;&#10;Most of the research works in today's NLP technology are culturally Anglocentric with English as the main language~\cite{sogaard2022ban,talat2022reap}. &#10;While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax,&#10;adilazuarda-etal-2022-indorobusta,&#10;ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}. Although such data collection methods could be effective in high-resource languages, applying the methods in underrepresented languages still needs further investigation.&#10;&#10;In this work, we compare three corpus collection methods for 12 underrepresented languages in Indonesia, namely Ambon (abs), Batak (btk), Betawi (bew), Bima (bhp), Buginese (bug), Javanese (jav), Madurese (mad), Makassarese (mak), Minangkabau (min), Palembang / Musi (mui), Rejang (rej), and Sundanese (sun). We chose Indonesian local languages as our case study because of the language diversity in Indonesia, with more than 700 languages spoken but most of them are underrepresented and extremely low-resource~\cite{cohn2014local,aji-etal-2022-one}. \cite{bang2023multitask} categorize Javanese (jav) and Sundanese (sun) as low-resource languages, while the others as extremely low-resource languages. For Ambon (abs), Bima (bhp), Makassarese (mak), Musi (mui), and Rejang (rej), they have no publicly available labeled and unlabeled corpora despite there being millions of speakers. We provide information on 12 low-resource languages under study in Table~\ref{tab:lang-under-study}. We conduct two manual data construction efforts for the 12 languages: topic-focused paragraph writing (NusaParagraph) and human translation by native speakers (NusaTranslation), and benchmark them with online scraping. For online scraping, we utilize Wikipedia as the main source as it covers some of the Indonesian local languages under study. Figure~\ref{fig:language-overlap} summarizes the corpora constructed by each approach: Wikipedia, NusaParagraph, and NusaTranslation for online scraping, paragraph writing, and human translation, respectively. NusaParagraph tends to have fewer English and Indonesian lexicons, indicating they are more relevant to the local cultures than the others.&#10;&#10;We build a new benchmark for the 12 Indonesian local languages, namely NusaWrites, using the texts produced in topic-focused paragraph writing and human translation. NusaWrites covers 5 natural language understanding tasks (e.g., emotion, sentiment classification) and one natural language generation task (i.e., machine translation), and complements NusaX~\cite{winata2022nusax}---a contemporaneous work on 10 Indonesian local languages for sentiment analysis and machine translation.&#10;We also demonstrate the inability of (1) fine-tuned Indonesian and multilingual language models (LMs) and (2) zero-shot prompting via large LMs (LLMs) to adapt to these languages, indicating that these languages are distinct from the existing models.&#10;&#10;Our contributions to this work are four-fold:&#10;\begin{itemize}&#10; \item We compare various corpus collection methods for underrepresented and extremely low-resource languages. We show that paragraph writing is the most promising strategy for building high-quality and culturally-relevant corpora.&#10; \item We extend the NLP resource coverage of Indonesian local languages with 5 new languages: Ambon (abs), Bima (bhp), Makassarese (mak), Musi (mui), and Rejang (rej). &#10; \item We propose NusaWrites,&#10; a benchmark covering new high-quality human annotated corpora consisting of 12 underrepresented languages in Indonesia with 5 downstream tasks.&#10; \item We conduct extensive analysis to showcase the similarity between languages under study with Indonesian and the inability of existing LLMs to process these languages.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Dataset Construction, Linguistics, Multilingual Language Models, Social Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2212.09648" label="2212.09648">
        <attvalues>
          <attvalue for="0" value="NusaCrowd: Open Source Initiative for Indonesian NLP Resources" />
          <attvalue for="1" value="  We present NusaCrowd, a collaborative initiative to collect and unify&#10;existing resources for Indonesian languages, including opening access to&#10;previously non-public resources. Through this initiative, we have brought&#10;together 137 datasets and 118 standardized data loaders. The quality of the&#10;datasets has been assessed manually and automatically, and their value is&#10;demonstrated through multiple experiments. NusaCrowd's data collection enables&#10;the creation of the first zero-shot benchmarks for natural language&#10;understanding and generation in Indonesian and the local languages of&#10;Indonesia. Furthermore, NusaCrowd brings the creation of the first multilingual&#10;automatic speech recognition benchmark in Indonesian and the local languages of&#10;Indonesia. Our work strives to advance natural language processing (NLP)&#10;research for languages that are under-represented despite being widely spoken.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00677" label="2011.00677">
        <attvalues>
          <attvalue for="0" value="IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model&#10;  for Indonesian NLP" />
          <attvalue for="1" value="  Although the Indonesian language is spoken by almost 200 million people and&#10;the 10th most spoken language in the world, it is under-represented in NLP&#10;research. Previous work on Indonesian has been hampered by a lack of annotated&#10;datasets, a sparsity of language resources, and a lack of resource&#10;standardization. In this work, we release the IndoLEM dataset comprising seven&#10;tasks for the Indonesian language, spanning morpho-syntax, semantics, and&#10;discourse. We additionally release IndoBERT, a new pre-trained language model&#10;for Indonesian, and evaluate it over IndoLEM, in addition to benchmarking it&#10;against existing resources. Our experiments show that IndoBERT achieves&#10;state-of-the-art performance over most of the tasks in IndoLEM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.09309" label="2009.09309">
        <attvalues>
          <attvalue for="0" value="Towards Computational Linguistics in Minangkabau Language: Studies on&#10;  Sentiment Analysis and Machine Translation" />
          <attvalue for="1" value="  Although some linguists (Rusmali et al., 1985; Crouch, 2009) have fairly&#10;attempted to define the morphology and syntax of Minangkabau, information&#10;processing in this language is still absent due to the scarcity of the&#10;annotated resource. In this work, we release two Minangkabau corpora: sentiment&#10;analysis and machine translation that are harvested and constructed from&#10;Twitter and Wikipedia. We conduct the first computational linguistics in&#10;Minangkabau language employing classic machine learning and&#10;sequence-to-sequence models such as LSTM and Transformer. Our first experiments&#10;show that the classification performance over Minangkabau text significantly&#10;drops when tested with the model trained in Indonesian. Whereas, in the machine&#10;translation experiment, a simple word-to-word translation using a bilingual&#10;dictionary outperforms LSTM and Transformer model in terms of BLEU score.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.05387" label="2009.05387">
        <attvalues>
          <attvalue for="0" value="IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural&#10;  Language Understanding" />
          <attvalue for="1" value="  Although Indonesian is known to be the fourth most frequently used language&#10;over the internet, the research progress on this language in the natural&#10;language processing (NLP) is slow-moving due to a lack of available resources.&#10;In response, we introduce the first-ever vast resource for the training,&#10;evaluating, and benchmarking on Indonesian natural language understanding&#10;(IndoNLU) tasks. IndoNLU includes twelve tasks, ranging from single sentence&#10;classification to pair-sentences sequence labeling with different levels of&#10;complexity. The datasets for the tasks lie in different domains and styles to&#10;ensure task diversity. We also provide a set of Indonesian pre-trained models&#10;(IndoBERT) trained from a large and clean Indonesian dataset Indo4B collected&#10;from publicly available sources such as social media texts, blogs, news, and&#10;websites. We release baseline models for all twelve tasks, as well as the&#10;framework for benchmark evaluation, and thus it enables everyone to benchmark&#10;their system performances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08726" label="2104.08726">
        <attvalues>
          <attvalue for="0" value="AmericasNLI: Evaluating Zero-shot Natural Language Understanding of&#10;  Pretrained Multilingual Models in Truly Low-resource Languages" />
          <attvalue for="1" value="  Pretrained multilingual models are able to perform cross-lingual transfer in&#10;a zero-shot setting, even for languages unseen during pretraining. However,&#10;prior work evaluating performance on unseen languages has largely been limited&#10;to low-level, syntactic tasks, and it remains unclear if zero-shot learning of&#10;high-level, semantic tasks is possible for unseen languages. To explore this&#10;question, we present AmericasNLI, an extension of XNLI (Conneau et al., 2018)&#10;to 10 indigenous languages of the Americas. We conduct experiments with XLM-R,&#10;testing multiple zero-shot and translation-based approaches. Additionally, we&#10;explore model adaptation via continued pretraining and provide an analysis of&#10;the dataset by considering hypothesis-only models. We find that XLM-R's&#10;zero-shot performance is poor for all 10 languages, with an average performance&#10;of 38.62%. Continued pretraining offers improvements, with an average accuracy&#10;of 44.05%. Surprisingly, training on poorly translated data by far outperforms&#10;all other methods with an accuracy of 48.72%.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretrained multilingual models such as XLM \cite{Lample2019CrosslingualLM}, multilingual BERT&#10;\cite[mBERT;][]{Devlin2019}, and XLM-R \cite{Conneau2020UnsupervisedCR} achieve strong cross-lingual transfer results for many languages and natural language processing (NLP) tasks. However, there exists a discrepancy in terms of zero-shot performance between languages present in the pretraining data and those that are not: performance is generally highest for well-represented languages and decreases with less representation. Yet, even for unseen languages, performance is generally above chance, and model adaptation approaches have been shown to yield further improvements \cite{Muller2020WhenBU, Pfeiffer2020MADXAA,pfeiffer2020unks,wang-etal-2020-extending}.&#10;&#10;Importantly, however, there are currently no datasets for high-level, semantic tasks which focus solely on low-resource languages. As these languages are most likely to be unseen to commonly used pretrained models, practically all work evaluating unseen language performance and language adaptation methods has been limited to low-level, syntactic tasks such as part-of-speech tagging, dependency parsing, and named-entity recognition \cite{Muller2020WhenBU, wang-etal-2020-extending}. This largely limits our ability to draw more general conclusions with regards to the zero-shot learning abilities of pretrained multilingual models for unseen languages.&#10;&#10;In this work, we introduce AmericasNLI, an extension of XNLI \cite{conneau2018xnli} -- a natural language inference (NLI; cf. \S\ref{subsec:nli}) dataset covering 15 high-resource languages -- to 10 Indigenous languages spoken in the Americas: Ash\'{a}ninka, Aymara, Bribri, Guaraní, Nahuatl, Otomí, Quechua, Rar\'{a}muri, Shipibo-Konibo, and Wixarika. All of them are truly low-resource languages: they have little to no digitally available labeled or unlabeled data, and they are not typically studied by the mainstream NLP community. The goal of this work is two-fold: First, we hope to increase the visibility of these languages by providing a portion of the resources necessary for NLP research. Second, we aim to allow for a more comprehensive study of multilingual model performance on unseen languages, where improvements will help extend the reach of NLP techniques to a larger set of languages. We are specifically interested in the following research questions: (1) Do pretrained multilingual models still perform above random chance for a high-level, semantic task in an unseen language? (2) Do methods aimed at adapting models to unseen languages -- previously exclusively evaluated on low-level, syntactic tasks -- also increase performance on NLI? (3) Are translation-based approaches effective for truly low-resource languages, where translation quality is typically very poor?&#10;&#10;We experiment with XLM-R, both with and without model adaptation via continued pretraining on monolingual corpora in the target language. Our results show that the performance of XLM-R out-of-the-box is moderately above chance, and model adaptation leads to improvements of up to 5.86 percentage points. Training on machine-translated training data, however, results in an even larger performance gain of 11.13 percentage points over the corresponding XLM-R model without adaptation.&#10;We further perform an analysis via experiments with hypothesis-only models, to examine potential artifacts which may have been inherited from XNLI and find that performance is above chance for most models, but still below that for using the full example.&#10;&#10;AmericasNLI is publicly available and we hope that it will serve as a benchmark for measuring the zero-shot natural language understanding abilities of multilingual models for unseen languages. Additionally, we hope that our dataset will motivate the development of novel pretraining and model adaptation techniques which are suitable for truly low-resource&#10;languages.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.09680" label="2105.09680">
        <attvalues>
          <attvalue for="0" value="KLUE: Korean Language Understanding Evaluation" />
          <attvalue for="1" value="  We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE&#10;is a collection of 8 Korean natural language understanding (NLU) tasks,&#10;including Topic Classification, SemanticTextual Similarity, Natural Language&#10;Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing,&#10;Machine Reading Comprehension, and Dialogue State Tracking. We build all of the&#10;tasks from scratch from diverse source corpora while respecting copyrights, to&#10;ensure accessibility for anyone without any restrictions. With ethical&#10;considerations in mind, we carefully design annotation protocols. Along with&#10;the benchmark tasks and data, we provide suitable evaluation metrics and&#10;fine-tuning recipes for pretrained language models for each task. We&#10;furthermore release the pretrained language models (PLM), KLUE-BERT and&#10;KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby&#10;facilitate future research. We make a few interesting observations from the&#10;preliminary experiments using the proposed KLUE benchmark suite, already&#10;demonstrating the usefulness of this new benchmark suite. First, we find&#10;KLUE-RoBERTa-large outperforms other baselines, including multilingual PLMs and&#10;existing open-source Korean PLMs. Second, we see minimal degradation in&#10;performance even when we replace personally identifiable information from the&#10;pretraining corpus, suggesting that privacy and NLU capability are not at odds&#10;with each other. Lastly, we find that using BPE tokenization in combination&#10;with morpheme-level pre-tokenization is effective in tasks involving&#10;morpheme-level tagging, detection and generation. In addition to accelerating&#10;Korean NLP research, our comprehensive documentation on creating KLUE will&#10;facilitate creating similar resources for other languages in the future. KLUE&#10;is available at https://klue-benchmark.com.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.05437" label="2203.05437">
        <attvalues>
          <attvalue for="0" value="IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic&#10;  Languages" />
          <attvalue for="1" value="  Natural Language Generation (NLG) for non-English languages is hampered by&#10;the scarcity of datasets in these languages. In this paper, we present the&#10;IndicNLG Benchmark, a collection of datasets for benchmarking NLG for 11 Indic&#10;languages. We focus on five diverse tasks, namely, biography generation using&#10;Wikipedia infoboxes, news headline generation, sentence summarization,&#10;paraphrase generation and, question generation. We describe the created&#10;datasets and use them to benchmark the performance of several monolingual and&#10;multilingual baselines that leverage pre-trained sequence-to-sequence models.&#10;Our results exhibit the strong performance of multilingual language-specific&#10;pre-trained models, and the utility of models trained on our dataset for other&#10;related NLG tasks. Our dataset creation methods can be easily applied to&#10;modest-resource languages as they involve simple steps such as scraping news&#10;articles and Wikipedia infoboxes, light cleaning, and pivoting through machine&#10;translation data. To the best of our knowledge, the IndicNLG Benchmark is the&#10;first NLG benchmark for Indic languages and the most diverse multilingual NLG&#10;dataset, with approximately 8M examples across 5 tasks and 11 languages. The&#10;datasets and models are publicly available at&#10;https://ai4bharat.iitm.ac.in/indicnlg-suite.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.15960" label="2205.15960">
        <attvalues>
          <attvalue for="0" value="NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local&#10;  Languages" />
          <attvalue for="1" value="  Natural language processing (NLP) has a significant impact on society via&#10;technologies such as machine translation and search engines. Despite its&#10;success, NLP technology is only widely available for high-resource languages&#10;such as English and Chinese, while it remains inaccessible to many languages&#10;due to the unavailability of data resources and benchmarks. In this work, we&#10;focus on developing resources for languages in Indonesia. Despite being the&#10;second most linguistically diverse country, most languages in Indonesia are&#10;categorized as endangered and some are even extinct. We develop the first-ever&#10;parallel resource for 10 low-resource languages in Indonesia. Our resource&#10;includes datasets, a multi-task benchmark, and lexicons, as well as a parallel&#10;Indonesian-English dataset. We provide extensive analyses and describe the&#10;challenges when creating such resources. We hope that our work can spark NLP&#10;research on Indonesian and other underrepresented languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16171" label="2305.16171">
        <attvalues>
          <attvalue for="0" value="Multi-lingual and Multi-cultural Figurative Language Understanding" />
          <attvalue for="1" value="  Figurative language permeates human communication, but at the same time is&#10;relatively understudied in NLP. Datasets have been created in English to&#10;accelerate progress towards measuring and improving figurative language&#10;processing in language models (LMs). However, the use of figurative language is&#10;an expression of our cultural and societal experiences, making it difficult for&#10;these phrases to be universally applicable. In this work, we create a&#10;figurative language inference dataset, \datasetname, for seven diverse&#10;languages associated with a variety of cultures: Hindi, Indonesian, Javanese,&#10;Kannada, Sundanese, Swahili and Yoruba. Our dataset reveals that each language&#10;relies on cultural and regional concepts for figurative expressions, with the&#10;highest overlap between languages originating from the same region. We assess&#10;multilingual LMs' abilities to interpret figurative language in zero-shot and&#10;few-shot settings. All languages exhibit a significant deficiency compared to&#10;English, with variations in performance reflecting the availability of&#10;pre-training and fine-tuning data, emphasizing the need for LMs to be exposed&#10;to a broader range of linguistic and cultural variation during training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14716" label="2305.14716">
        <attvalues>
          <attvalue for="0" value="GlobalBench: A Benchmark for Global Progress in Natural Language&#10;  Processing" />
          <attvalue for="1" value="  Despite the major advances in NLP, significant disparities in NLP system&#10;performance across languages still exist. Arguably, these are due to uneven&#10;resource allocation and sub-optimal incentives to work on less resourced&#10;languages. To track and further incentivize the global development of equitable&#10;language technology, we introduce GlobalBench. Prior multilingual benchmarks&#10;are static and have focused on a limited number of tasks and languages. In&#10;contrast, GlobalBench is an ever-expanding collection that aims to dynamically&#10;track progress on all NLP datasets in all languages. Rather than solely&#10;measuring accuracy, GlobalBench also tracks the estimated per-speaker utility&#10;and equity of technology across all languages, providing a multi-faceted view&#10;of how language technology is serving people of the world. Furthermore,&#10;GlobalBench is designed to identify the most under-served languages, and&#10;rewards research efforts directed towards those languages. At present, the most&#10;under-served languages are the ones with a relatively high population, but&#10;nonetheless overlooked by composite multilingual benchmarks (like Punjabi,&#10;Portuguese, and Wu Chinese). Currently, GlobalBench covers 966 datasets in 190&#10;languages, and has 1,128 system submissions spanning 62 languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04607" label="2109.04607">
        <attvalues>
          <attvalue for="0" value="IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with&#10;  Effective Domain-Specific Vocabulary Initialization" />
          <attvalue for="1" value="  We present IndoBERTweet, the first large-scale pretrained model for&#10;Indonesian Twitter that is trained by extending a monolingually-trained&#10;Indonesian BERT model with additive domain-specific vocabulary. We focus in&#10;particular on efficient model adaptation under vocabulary mismatch, and&#10;benchmark different ways of initializing the BERT embedding layer for new word&#10;types. We find that initializing with the average BERT subword embedding makes&#10;pretraining five times faster, and is more effective than proposed methods for&#10;vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.13357" label="2203.13357">
        <attvalues>
          <attvalue for="0" value="One Country, 700+ Languages: NLP Challenges for Underrepresented&#10;  Languages and Dialects in Indonesia" />
          <attvalue for="1" value="  NLP research is impeded by a lack of resources and awareness of the&#10;challenges presented by underrepresented languages and dialects. Focusing on&#10;the languages spoken in Indonesia, the second most linguistically diverse and&#10;the fourth most populous nation of the world, we provide an overview of the&#10;current state of NLP research for Indonesia's 700+ languages. We highlight&#10;challenges in Indonesian NLP and how these affect the performance of current&#10;NLP systems. Finally, we provide general recommendations to help develop NLP&#10;technology not only for languages of Indonesia but also other underrepresented&#10;languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.04243" label="2104.04243">
        <attvalues>
          <attvalue for="0" value="Incorporating External Knowledge to Enhance Tabular Reasoning" />
          <attvalue for="1" value="  Reasoning about tabular information presents unique challenges to modern NLP&#10;approaches which largely rely on pre-trained contextualized embeddings of text.&#10;In this paper, we study these challenges through the problem of tabular natural&#10;language inference. We propose easy and effective modifications to how&#10;information is presented to a model for this task. We show via systematic&#10;experiments that these strategies substantially improve tabular inference&#10;performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Inference (NLI) is the task of determining if a hypothesis sentence can be inferred&#10;as true, false, or undetermined given a premise sentence~\cite{dagan2013recognizing}. Contextual sentence embeddings such as BERT~\cite{devlin2019bert} and RoBERTa~\cite{liu2019roberta}, applied to large datasets such as SNLI~\cite{snli:emnlp2015} and MultiNLI~\cite{N18-1101}, have led to near-human performance of NLI systems.&#10;&#10;In this paper, we study the harder problem of reasoning about tabular premises, as instantiated in datasets such as TabFact~\cite{chen2019tabfact} and InfoTabS~\cite{gupta-etal-2020-infotabs}. This problem is similar to standard NLI, but the premises are Wikipedia tables rather than sentences. Models similar to the best ones for the standard NLI datasets struggle with tabular inference. &#10;Using the InfoTabS dataset as an example, we present a focused study that investigates&#10;\begin{inparaenum}[(a)]&#10;\item the poor performance of existing models, \item connections to information deficiency in the tabular premises, and,&#10;\item simple yet effective mitigatations for these problems.&#10;\end{inparaenum}&#10;&#10;We use the table and hypotheses in Figure~\ref{fig:example} as a running example through this paper, and refer to the left column as its keys.&#10;Tabular inference is challenging for several reasons:&#10;\begin{inparaenum}[(a)]&#10;\item Poor table representation: The table does not explicitly state the relationship between the keys and values.&#10;\item Missing implicit lexical knowledge due to limited training data: This affects interpreting words like `fewer', and `over' in H1 and H2 respectively.&#10;\item Presence of distracting information: All keys except No. of listings are unrelated to the hypotheses H1 and H2.&#10;\item Missing domain knowledge about keys: We need to interpret the key Volume in the financial context for this table.&#10;\end{inparaenum}&#10;&#10;In the absence of large labeled corpora, any modeling strategy needs to explicitly address these problems. In this paper, we propose effective approaches for addressing them, and show that they lead to substantial improvements in prediction quality, especially on adversarial test sets. This focused study makes the following contributions:&#10;&#10;\begin{enumerate}[nosep]&#10; \item We analyse why the existing state-of-the-art BERT class models struggle on the challenging task of NLI over tabular data.&#10; \item We propose solutions to overcome these challenges via simple modifications to inputs using existing language resources.&#10; \item Through extensive experiments, we show significant improvements to model performance, especially on challenging adversarial test sets. &#10;\end{enumerate}&#10;&#10;The updated dataset, along with associated scripts, are available at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Tabular Data Analysis, Artificial Intelligence, Natural Language Inference, NLP Model Optimization, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1909.02164" label="1909.02164">
        <attvalues>
          <attvalue for="0" value="TabFact: A Large-scale Dataset for Table-based Fact Verification" />
          <attvalue for="1" value="  The problem of verifying whether a textual hypothesis holds based on the&#10;given evidence, also known as fact verification, plays an important role in the&#10;study of natural language understanding and semantic representation. However,&#10;existing studies are mainly restricted to dealing with unstructured evidence&#10;(e.g., natural language sentences and documents, news, etc), while verification&#10;under structured evidence, such as tables, graphs, and databases, remains&#10;under-explored. This paper specifically aims to study the fact verification&#10;given semi-structured data as evidence. To this end, we construct a large-scale&#10;dataset called TabFact with 16k Wikipedia tables as the evidence for 118k&#10;human-annotated natural language statements, which are labeled as either&#10;ENTAILED or REFUTED. TabFact is challenging since it involves both soft&#10;linguistic reasoning and hard symbolic reasoning. To address these reasoning&#10;challenges, we design two different models: Table-BERT and Latent Program&#10;Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language&#10;model to encode the linearized tables and statements into continuous vectors&#10;for verification. LPA parses statements into programs and executes them against&#10;the tables to obtain the returned binary value for verification. Both methods&#10;achieve similar accuracy but still lag far behind human performance. We also&#10;perform a comprehensive analysis to demonstrate great future opportunities. The&#10;data and code of the dataset are provided in&#10;\url{https://github.com/wenhuchen/Table-Fact-Checking}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.06117" label="2005.06117">
        <attvalues>
          <attvalue for="0" value="INFOTABS: Inference on Tables as Semi-structured Data" />
          <attvalue for="1" value="  In this paper, we observe that semi-structured tabulated text is ubiquitous;&#10;understanding them requires not only comprehending the meaning of text&#10;fragments, but also implicit relationships between them. We argue that such&#10;data can prove as a testing ground for understanding how we reason about&#10;information. To study this, we introduce a new dataset called INFOTABS,&#10;comprising of human-written textual hypotheses based on premises that are&#10;tables extracted from Wikipedia info-boxes. Our analysis shows that the&#10;semi-structured, multi-domain and heterogeneous nature of the premises admits&#10;complex, multi-faceted reasoning. Experiments reveal that, while human&#10;annotators agree on the relationships between a table-hypothesis pair, several&#10;standard modeling strategies are unsuccessful at the task, suggesting that&#10;reasoning about tables can pose a difficult modeling challenge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.12609" label="2110.12609">
        <attvalues>
          <attvalue for="0" value="No News is Good News: A Critique of the One Billion Word Benchmark" />
          <attvalue for="1" value="  The One Billion Word Benchmark is a dataset derived from the WMT 2011 News&#10;Crawl, commonly used to measure language modeling ability in natural language&#10;processing. We train models solely on Common Crawl web scrapes partitioned by&#10;year, and demonstrate that they perform worse on this task over time due to&#10;distributional shift. Analysis of this corpus reveals that it contains several&#10;examples of harmful text, as well as outdated references to current events. We&#10;suggest that the temporal nature of news and its distribution shift over time&#10;makes it poorly suited for measuring language modeling ability, and discuss&#10;potential impact and considerations for researchers building language models&#10;and evaluation datasets.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models are commonly evaluated on the One Billion Word Benchmark (lm1b) \cite{lm1b}, reporting performance on perplexity \cite{radford2019language}. lm1b is derived from the WMT 2011 News Crawl dataset released ten years ago. Though the number of citations per year is decreasing, it is still widely used by researchers in recent years, as shown in Figure \ref{semanticscholar}. Examples in the dataset were constructed by extracting single sentences from news articles, such as: \begin{itemize}&#10; \item At least 101 people were killed in the blasts.&#10; \item Who can possbly ever take her serious?&#10; \item Well, Dexter caught up with me.&#10;\end{itemize}&#10;&#10;Previous work has documented examples of decontextualized hate speech \cite{ngo2021mitigating} within this dataset, as well as the impact of its destructive preprocessing \cite{radford2019language}. Our work suggests that the prevalence of this dataset in the literature is concerning, as evaluation on this dataset inadvertently incentivizes the creation of language models which optimize for generating language in the style of news articles without any regard for factuality, and encodes world knowledge which becomes progressively more outdated. We demonstrate that model performance on this task decreases over time due to distributional shift of the training data, and argue that news articles should not be used as the basis for assessing language modeling ability.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Dataset Evaluation, Information Science, Natural Language Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="2108.07790" label="2108.07790">
        <attvalues>
          <attvalue for="0" value="Mitigating harm in language models with conditional-likelihood&#10;  filtration" />
          <attvalue for="1" value="  Language models trained on large-scale unfiltered datasets curated from the&#10;open web acquire systemic biases, prejudices, and harmful views from their&#10;training data. We present a methodology for programmatically identifying and&#10;removing harmful text from web-scale datasets. A pretrained language model is&#10;used to calculate the log-likelihood of researcher-written trigger phrases&#10;conditioned on a specific document, which is used to identify and filter&#10;documents from the dataset. We demonstrate that models trained on this filtered&#10;dataset exhibit lower propensity to generate harmful text, with a marginal&#10;decrease in performance on standard language modeling benchmarks compared to&#10;unfiltered baselines. We provide a partial explanation for this performance gap&#10;by surfacing examples of hate speech and other undesirable content from&#10;standard language modeling benchmarks. Finally, we discuss the generalization&#10;of this method and how trigger phrases which reflect specific values can be&#10;used by researchers to build language models which are more closely aligned&#10;with their values.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.06563" label="2403.06563">
        <attvalues>
          <attvalue for="0" value="Unraveling the Mystery of Scaling Laws: Part I" />
          <attvalue for="1" value="  Scaling law principles indicate a power-law correlation between loss and&#10;variables such as model size, dataset size, and computational resources&#10;utilized during training. These principles play a vital role in optimizing&#10;various aspects of model pre-training, ultimately contributing to the success&#10;of large language models such as GPT-4, Llama and Gemini. However, the original&#10;scaling law paper by OpenAI did not disclose the complete details necessary to&#10;derive the precise scaling law formulas, and their conclusions are only based&#10;on models containing up to 1.5 billion parameters. Though some subsequent works&#10;attempt to unveil these details and scale to larger models, they often neglect&#10;the training dependency of important factors such as the learning rate, context&#10;length and batch size, leading to their failure to establish a reliable formula&#10;for predicting the test loss trajectory. In this technical report, we confirm&#10;that the scaling law formulations proposed in the original OpenAI paper remain&#10;valid when scaling the model size up to 33 billion, but the constant&#10;coefficients in these formulas vary significantly with the experiment setup. We&#10;meticulously identify influential factors and provide transparent, step-by-step&#10;instructions to estimate all constant terms in scaling-law formulas by training&#10;on models with only 1M~60M parameters. Using these estimated formulas, we&#10;showcase the capability to accurately predict various attributes for models&#10;with up to 33B parameters before their training, including (1) the minimum&#10;possible test loss; (2) the minimum required training steps and processed&#10;tokens to achieve a specific loss; (3) the critical batch size with an optimal&#10;time/computation trade-off at any loss value; and (4) the complete test loss&#10;trajectory with arbitrary batch size.&#10;" />
          <attvalue for="2" value="&#10;&#10;A wide range of studies have shown that the performance of a language model exhibits a notable growth pattern as the number of parameters and data size increase, following a power-law relationship~\cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,clark2022unified,zhai2022scaling,gao2023scaling,biderman2023pythia}. &#10;This scaling law plays a fundamental role in the development of large language models, enabling us to estimate optimal configurations of large models from the training logs of much smaller models~\cite{tay2022scale,hoffmann2022training}. As mentioned in the GPT-4 technical report~\cite{achiam2023gpt}, some aspects of GPT-4's&#10;performance can be accurately predicted based on models trained with no more than 1/1,000th the compute of GPT-4. By properly utilizing the scaling law, we avoid the need to perform extensive model-specific tuning on large models. &#10;&#10;The original scaling law paper by OpenAI presented the formulas of scaling laws and illustrated how they could aid in determining optimal training configurations~\cite{kaplan2020scaling}. Nonetheless, the presented formulas are based on static exponents estimated from their specific experiment setup. The full details on how to derive the constant terms in the scaling-law formulas for a new experiment setup (model architecture, tokenization, data distribution, etc) remain undisclosed. Furthermore, \cite{kaplan2020scaling} only conducted experiments with models containing up to 1.5B parameters, a size significantly smaller than that of contemporary large language models. There have been subsequent works that study scaling laws on larger models~\cite{clark2022unified,isik2024scaling}. Some have drawn different conclusions from the original scaling-law paper, casting doubt to the general applicability of scaling laws. For example, \cite{hoffmann2022training} claimed that the training data size should be scaled much more than the recommendation in \cite{kaplan2020scaling}. \cite{bi2024deepseek} suggested that the optimal batch size depends only on the compute budget rather than the loss value.&#10;&#10;In this paper, we revisit the scaling-law formulas proposed by \cite{kaplan2020scaling}, confirming that they remain generally applicable when scaling the model size up to 33B. Other works obtain different conclusions primarily due to (1) Many factors such as the data distribution, context length, tokenization affect the constant coefficients in scaling-law formulas, so the constant coefficients, unlike the formulas themselves, are not universal; and (2) The loss value adheres to an analytical power law relationship with the training step under infinite batch size. With a finite batch size, fitting the loss value with an analytical function is problematic. As a result, none of other works have provided compelling evidence to reliably predict the full loss trajectory of larger models by training solely on smaller models.&#10;&#10;After meticulously identifying&#10;influential factors in predicting the loss trajectory, we provide transparent, step-by-step guidelines on how to estimate all constant terms in scaling-law formulas by training on models with only 1M$\sim$60M parameters. Using these estimated formulas from small models, we showcase the capability to accurately predict various&#10;attributes for models with up to 33B parameters before their training starts. By unravelling the mystery of scaling laws and making them easily accessible to everyone, our objective is to shift the understanding of scaling laws from theoretical concepts to practical implementation, thereby aiding future research in pre-training large language models in a more principled manner. The summary of the key results in this paper is as follows:&#10;&#10;\begin{itemize}&#10; \item Hyperparameters such as batch size, learning rate, and learning rate scheduler influence the rate of convergence, yet do not impact the final converged loss provided that (1) their values fall within a reasonable range and (2) the model is trained with sufficient steps on adequate amounts of data.&#10; \item Adjusting the batch size involves a trade-off between time and computation. The critical batch size that strikes an optimal time/computation balance can be determined based solely on the loss value. Training with this critical batch size requires twice as many training steps to achieve a specific loss value compared to using an infinite batch size (minimum possible required steps).&#10; \item The context length, tokenization, data distribution and model configurations have big impacts on the constants in scaling law formulas, but do not affect the form of scaling law itself.&#10; \item When given a fixed context length, tokenization, data distribution, model configurations and learning rate scheduler, we observe precise and predictable power-law scalings for performance in relation to training step, batch size, and model size, provided that the learning rate is optimally configured.&#10; \item By training models with fewer than 60 million parameters, we can accurately estimate the constants in scaling-law formulas. This allows us to predict various attributes for models with up to 33 billion parameters before their training, including (1) the minimum possible loss; (2) the minimum required training steps and processed tokens to achieve a specific loss; (3) the critical batch size at any loss value; and (4) the complete test loss trajectory with arbitrary batch size.&#10; \item These predicted attributes have many intriguing features, assisting us in identifying crucial factors before training large models, such as the optimal model size and training steps within a fixed computational budget, the necessary amount of data, the ideal mix ratio of multiple datasets, and more.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Model Optimization Techniques, Artificial Intelligence, Mathematics, Scaling Law Principles, Statistics" />
        </attvalues>
      </node>
      <node id="2202.01169" label="2202.01169">
        <attvalues>
          <attvalue for="0" value="Unified Scaling Laws for Routed Language Models" />
          <attvalue for="1" value="  The performance of a language model has been shown to be effectively modeled&#10;as a power-law in its parameter count. Here we study the scaling behaviors of&#10;Routing Networks: architectures that conditionally use only a subset of their&#10;parameters while processing an input. For these models, parameter count and&#10;computational requirement form two independent axes along which an increase&#10;leads to better performance. In this work we derive and justify scaling laws&#10;defined on these two variables which generalize those known for standard&#10;language models and describe the performance of a wide range of routing&#10;architectures trained via three different techniques. Afterwards we provide two&#10;applications of these laws: first deriving an Effective Parameter Count along&#10;which all models scale at the same rate, and then using the scaling&#10;coefficients to give a quantitative comparison of the three routing techniques&#10;considered. Our analysis derives from an extensive evaluation of Routing&#10;Networks across five orders of magnitude of size, including models with&#10;hundreds of experts and hundreds of billions of parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.04177" label="2402.04177">
        <attvalues>
          <attvalue for="0" value="Scaling Laws for Downstream Task Performance of Large Language Models" />
          <attvalue for="1" value="  Scaling laws provide important insights that can guide the design of large&#10;language models (LLMs). Existing work has primarily focused on studying scaling&#10;laws for pretraining (upstream) loss. However, in transfer learning settings,&#10;in which LLMs are pretrained on an unsupervised dataset and then finetuned on a&#10;downstream task, we often also care about the downstream performance. In this&#10;work, we study the scaling behavior in a transfer learning setting, where LLMs&#10;are finetuned for machine translation tasks. Specifically, we investigate how&#10;the choice of the pretraining data and its size affect downstream performance&#10;(translation quality) as judged by two metrics: downstream cross-entropy and&#10;BLEU score. Our experiments indicate that the size of the finetuning dataset&#10;and the distribution alignment between the pretraining and downstream data&#10;significantly influence the scaling behavior. With sufficient alignment, both&#10;downstream cross-entropy and BLEU score improve monotonically with more&#10;pretraining data. In such cases, we show that it is possible to predict the&#10;downstream BLEU score with good accuracy using a log-law. However, there are&#10;also cases where moderate misalignment causes the BLEU score to fluctuate or&#10;get worse with more pretraining, whereas downstream cross-entropy monotonically&#10;improves. By analyzing these observations, we provide new practical insights&#10;for choosing appropriate pretraining data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;\looseness=-1&#10;Scaling laws for transformers.&#10;Scaling laws for LLMs have attracted significant attention as they can inform the decisions about key design choices such as model size and the type and size of the pretraining data~\cite{kaplan2020scaling, hoffmann2022training, hernandez2021scaling}. Most of the pioneering work has focused on how upstream cross-entropy loss or perplexity scales with more pretraining data, larger models, or longer training \cite{kaplan2020scaling, hoffmann2022training}. Follow-up works have analyzed scaling behavior of translation models~\cite{ghorbani2021scaling, zhuocheng2023scaling, gordon-etal-2021-data, fernandes2023scaling, bansal2022data, zhang2022examining}, studied theoretical foundation behind scaling laws~\cite{sharma2020neural, hutter2021learning, bahri2021explaining}, or extended the laws to the vision models \cite{zhai2022scaling, jain2023meta}. Closest to our work, \cite{hernandez2021scaling} have analyzed transfer learning but with a focus on how the cross-entropy loss behaves as the finetuning data scales. Unlike our work, their scaling law describes the relation between the size of a (finetuning) dataset and the cross-entropy loss on the same dataset -- making this closer to the standard scaling laws in the literature since the finetuning loss and the finetuning dataset are computed over samples from the same distribution. On the other hand, we propose scaling laws for the downstream metrics on the finetuning dataset as the pretraining data scales -- switching the focus to an ``out-of-distribution'' analysis. The only work we are aware of that proposed scaling laws for the downstream task performance as a function of pretraining dataset size \cite{sun2017revisiting} has focused on classification tasks in the vision domain and used smaller models.&#10;&#10;Transferability metrics and value of pretraining. &#10;While it may be commonly suggested that pretraining data improves both upstream and downstream performance, this rule has been challenged in the vision domain. \cite{zoph2020rethinking, he2019rethinking, shen2019object, ghiasi2018dropblock, mikami2022scaling} have demonstrated that pretraining can sometimes have no effect on the downstream task performance and sometimes it can even hurt the performance. We make similar observations in the language domain with extensive experiments on machine translation tasks and identify cases where (a) adding more pretraining data hurts the downstream task performance when pretraining data is not aligned enough with the task and (b) pretraining does not improve the downstream task performance noticeably when the finetuning dataset is large enough. Our observations about the importance of ``aligned'' pretraining data are also supported by recent work on machine translation~\cite{alves2024tower, xu2024a} trying to keep the pretraining data as multilingual as possible instead of being heavily English-centric~\cite{stap-etal-2024-fine, li2024eliciting}. Another related line of work is on transferability metrics \cite{tamkin2020investigating, chiang2022transferability, ibrahim2022newer, agostinelli2022transferability, nguyen2020leep, you2021logme, dai2019using, huang2022frustratingly, ibrahim2022newer, tran2019transferability, bao2019information, van2010using, plank2011effective}, which are efficient heuristics used to select the most appropriate source models or pretraining data for a given target task. We note that transferability metrics are designed to solve \textsl{ranking} problems, different from scaling laws. For example, these metrics answer questions such as given a pool of source models (or pretraining datasets), which source model (or pretraining dataset) is the best to finetune on for a given target task. These metrics are not designed to predict the performance of the model when key quantities (e.g., pretraining data size) are scaled.&#10; " />
        </attvalues>
      </node>
      <node id="1911.03766" label="1911.03766">
        <attvalues>
          <attvalue for="0" value="Multi-Sentence Argument Linking" />
          <attvalue for="1" value="  We present a novel document-level model for finding argument spans that fill&#10;an event's roles, connecting related ideas in sentence-level semantic role&#10;labeling and coreference resolution. Because existing datasets for&#10;cross-sentence linking are small, development of our neural model is supported&#10;through the creation of a new resource, Roles Across Multiple Sentences (RAMS),&#10;which contains 9,124 annotated events across 139 types. We demonstrate strong&#10;performance of our model on RAMS and other event-related datasets.&#10;" />
          <attvalue for="2" value="&#10;&#10;Textual event descriptions may span multiple sentences, yet large-scale datasets predominately annotate for events and their arguments at the sentence level. This has driven researchers to focus on sentence-level tasks such as semantic role labeling (SRL), even though perfect performance at such tasks would still enable a less than complete understanding of an event at the document level.&#10;&#10;In this work, we approach event understanding as a form of linking, more akin to coreference resolution than sentence-level SRL. An event trigger evokes a set of roles regarded as latent arguments, with these implicit arguments then potentially linked to explicit mentions in the text.&#10;&#10;Consider the example in \ref{fig: exm}: the \ont{Air\-strike\-Missile\-Strike} event (triggered by ``bombarding'') gives rise to a frame or set of type-level roles (\ont{attacker}, \ont{target}, \ont{instrument}, \ont{place}) with the referents (``Russians'', ``rebel outpost'', ``aircraft'', ``Syria''). Intuitively we recognize the possible existence of fillers for these roles, for example, the \ont{place} of the particular \ont{Air\-strike\-Missile\-Strike} event. These implicit arguments are linked to explicit arguments in the document (i.e., text spans). We refer to the task of finding explicit argument(s) to fill each role for an event as argument linking.&#10;&#10;Prior annotation of cross-sentence argument links has produced small datasets, with a focus either on a small number of predicate types~\cite{gerber-chai-2010-beyond, gerber-chai-2012-semantic, feizabadi-pado-2014-crowdsourcing} or on a small number of documents~\cite{ruppenhofer-etal-2010-semeval}. To enable the development of a neural model for argument linking, we produce Roles Across Multiple Sentences (\datasetname), a dataset of 9,124 annotated events from news based on an ontology of 139 event types and 65 roles. In a 5-sentence window around each event trigger, we annotate the closest argument span for each role.&#10;&#10;Our model builds on recent ideas in span selection models \cite{lee-etal-2018-higher, he-etal-2018-jointly, ouchi-etal-2018-span}, used in this work for the multi-sentence argument linking task for \datasetname and for several other event-based datasets \cite[AIDA Phase 1]{gerber-chai-2012-semantic, pradhan-etal-2013-towards, pavlick-etal-2016-gun}. On \datasetname our best model achieves \ramsbest~\F, and it achieves \ramsbestconstrained~\F when event types are also known, outperforming strong baselines. We also demonstrate effective use of \datasetname as pre-training for a related dataset.&#10;&#10;Our main contributions are a novel model for argument linking and a new large-scale dataset for the task. Our dataset is annotated for arguments across multiple sentences and has broader coverage of event types and more examples than similar work. Our experiments highlight our model's adaptability to multiple datasets. Together, these contributions further the automatic understanding of events at the document level.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Event Extraction, Linguistics, Semantic Role Labeling, Coreference Resolution, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2402.01383" label="2402.01383">
        <attvalues>
          <attvalue for="0" value="LLM-based NLG Evaluation: Current Status and Challenges" />
          <attvalue for="1" value="  Evaluating natural language generation (NLG) is a vital but challenging&#10;problem in artificial intelligence. Traditional evaluation metrics mainly&#10;capturing content (e.g. n-gram) overlap between system outputs and references&#10;are far from satisfactory, and large language models (LLMs) such as ChatGPT&#10;have demonstrated great potential in NLG evaluation in recent years. Various&#10;automatic evaluation methods based on LLMs have been proposed, including&#10;metrics derived from LLMs, prompting LLMs, and fine-tuning LLMs with labeled&#10;evaluation data. In this survey, we first give a taxonomy of LLM-based NLG&#10;evaluation methods, and discuss their pros and cons, respectively. We also&#10;discuss human-LLM collaboration for NLG evaluation. Lastly, we discuss several&#10;open problems in this area and point out future research directions.&#10;" />
          <attvalue for="2" value=" &#10;&#10;Evaluating natural language generation (NLG) is a key but challenging issue. Traditional evaluation metrics like BLEU \cite{papineni2002bleu} and ROUGE \cite{lin2004rouge} rely on the n-gram overlap between model outputs and references to measure its quality. They have been criticized for low correlation with human judgments \cite{sulem-etal-2018-bleu}, as surface-level matching cannot reliably evaluate text. After the rise of deep learning, model-based evaluation metrics like BERTScore \cite{zhang2019bertscore} and BARTScore \cite{yuan2021bartscore} have been continuously proposed and gradually adopted to evaluate the overall quality or various specific aspects of generated outputs (e.g., fluency, coherence, coverage, faithfulness, etc.). Although better than traditional metrics, their performance is still not satisfactory, and their application scope is very limited. For example, BERTScore is reference-based and cannot be used without a reference. With the emergence of large language models (LLMs) like InstructGPT \cite{ouyang2022training}, they have achieved unprecedented effectiveness in following instructions, understanding content, and generating text. This inspired researchers to use LLMs for NLG evaluation. Although this is a research direction that only emerged in 2023, the past year has seen an enormous amount of research work. It is no exaggeration to say that NLG evaluation has been revolutionized by LLMs. This article will review the existing literature and provide suggestions for future research in this field.&#10;&#10;This article mainly focuses on research that uses language models with over one billion parameters for NLG evaluation, with necessary references to earlier model-based evaluation metrics like BERTScore. To maintain focus, other types of generation like code generation and tasks involving images are not included in the scope of this article. As shown in Figure~\ref{fig:enter-label}, according to how we utilize LLMs for NLG evaluation, we categorize the research work into four types:&#10;\begin{itemize}&#10; \item LLM-derived Metrics (\S\ref{sec:llm-derived}): developing/deriving evaluation metrics from embeddings or generation probabilities of LLMs.&#10; \item Prompting LLMs (\S\ref{sec:prompting}): directly inquiring existing LLMs via designed prompts which involve different elements for evaluation. &#10; \item Fine-tuning LLMs (\S\ref{sec:fine-tuning}): using labeled evaluation data to fine-tune existing LLMs and improving their NLG evaluation capabilities. &#10; \item Human-LLM Collaborative Evaluation (\S\ref{sec:collaboration}): leveraging distinctive strengths of both human evaluators and LLMs to achieve robust and nuanced evaluations in challenging domains through human-LLM collaboration.&#10;\end{itemize}&#10; We will review each type of evaluation methods and discuss the pros and cons respectively. Lastly, we will discuss future directions in this area (\S\ref{sec:future}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence Evaluation, Artificial Intelligence, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1810.05995" label="1810.05995">
        <attvalues>
          <attvalue for="0" value="BLEU is Not Suitable for the Evaluation of Text Simplification" />
          <attvalue for="1" value="  BLEU is widely considered to be an informative metric for text-to-text&#10;generation, including Text Simplification (TS). TS includes both lexical and&#10;structural aspects. In this paper we show that BLEU is not suitable for the&#10;evaluation of sentence splitting, the major structural simplification&#10;operation. We manually compiled a sentence splitting gold standard corpus&#10;containing multiple structural paraphrases, and performed a correlation&#10;analysis with human judgments. We find low or no correlation between BLEU and&#10;the grammaticality and meaning preservation parameters where sentence splitting&#10;is involved. Moreover, BLEU often negatively correlates with simplicity,&#10;essentially penalizing simpler sentences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.13905" label="2309.13905">
        <attvalues>
          <attvalue for="0" value="AutoPrep: An Automatic Preprocessing Framework for In-the-Wild Speech&#10;  Data" />
          <attvalue for="1" value="  Recently, the utilization of extensive open-sourced text data has&#10;significantly advanced the performance of text-based large language models&#10;(LLMs). However, the use of in-the-wild large-scale speech data in the speech&#10;technology community remains constrained. One reason for this limitation is&#10;that a considerable amount of the publicly available speech data is compromised&#10;by background noise, speech overlapping, lack of speech segmentation&#10;information, missing speaker labels, and incomplete transcriptions, which can&#10;largely hinder their usefulness. On the other hand, human annotation of speech&#10;data is both time-consuming and costly. To address this issue, we introduce an&#10;automatic in-the-wild speech data preprocessing framework (AutoPrep) in this&#10;paper, which is designed to enhance speech quality, generate speaker labels,&#10;and produce transcriptions automatically. The proposed AutoPrep framework&#10;comprises six components: speech enhancement, speech segmentation, speaker&#10;clustering, target speech extraction, quality filtering and automatic speech&#10;recognition. Experiments conducted on the open-sourced WenetSpeech and our&#10;self-collected AutoPrepWild corpora demonstrate that the proposed AutoPrep&#10;framework can generate preprocessed data with similar DNSMOS and PDNSMOS scores&#10;compared to several open-sourced TTS datasets. The corresponding TTS system can&#10;achieve up to 0.68 in-domain speaker similarity.&#10;" />
          <attvalue for="2" value="&#10;&#10;Gathering large-scale, high-quality training data with comprehensive and accurate labels has always been a critical aspect of speech technology development. &#10;Over the past few decades, the speech community has devoted considerable time and effort to manually recording, collecting, and annotating a vast amount of speech data with corresponding segmentation, transcription and speaker labels, which has significantly advanced the performance of various speech technologies, such as automatic speech recognition (ASR) \cite{asr_survey, radford2023robust, zhang2023google}, text-to-speech synthesis (TTS) \cite{tts_survey, kharitonov2023speak, wang2023neural, jiang2023mega}, speaker verification (SV) \cite{sv_survey, caron2021emerging}, and speech enhancement (SE) \cite{se_survey}. &#10;However, the volume of accessible human-annotated speech data is still limited for further improving the performance and generalization of current speech models, especially for TTS which requires high-quality speech recordings with multiple speakers and styles. &#10;To this end, leveraging the vast amount of open, in-the-wild speech data from open video recordings, podcasts and audiobooks presents a promising approach.&#10;&#10;However, directly utilizing in-the-wild speech data poses challenges in two main aspects:&#10;First, the absence of necessary annotations, such as reliable text transcriptions, segmentation information, and speaker labels, impedes the direct use of this data for supervised training tasks, such as ASR, TTS and SV. &#10;Second, simply adding more data does not guarantee performance enhancement. Low-quality data with unexpected background noise, reverberation, speech overlap, and distortion, can significantly degrade the performance of generation tasks like TTS.&#10;For the missing annotation issue, previous research has focused primarily on unsupervised pre-training methods \cite{wav2vec, baevski2020wav2vec, hubert, wavlm, borsos2023audiolm}. &#10;However, for most downstream tasks, these methods still require annotated speech data for fine-tuning the pretrained models. Regarding the second issue, prior research \cite{koizumi2023libritts, valentiniinvestigating} has proposed leveraging speech enhancement methods to handle unstable speech quality in TTS model construction. &#10;However, these methods are only effective in part of the scenarios presented in diverse in-the-wild data. &#10;Nevertheless, to the best of our knowledge, there is limited previous work that attempts to comprehensively address both challenges associated with directly using in-the-wild speech data.&#10;&#10;Therefore, in this paper, we propose a pipelined automatic in-the-wild speech data preprocessing framework, named AutoPrep, which aims to directly solve the aforementioned two issues by automatically generating segmentation information, speaker labels, and transcriptions for in-the-wild speech data while removing background noise, dereverberation, and speech overlap to improve speech quality. &#10;To provide high-quality speech and annotation, we incorporate competitive models into the proposed AutoPrep framework, including the Band-Split RNN (BSRNN) speech enhancement model \cite{bsrnn}, WeSpeaker speaker embedding model \cite{wespeaker}, personalized BSRNN (pBSRNN) target speech extraction (TSE) model \cite{yu2023tspeech}, and a 60k-hour trained multilingual conformer-based \cite{conformer} RNN-Transducer \cite{rnnt} ASR system. In addition, the DNSMOS \cite{dnsmos} and PDNSMOS \cite{dubey2023icassp} metrics are also employed to evaluate the quality of the processed speech data, filtering out low-quality instances to ensure the speech quality of the final output.&#10;By utilizing the AutoPrep framework, the processed in-the-wild speech data can be directly employed in various tasks, such as TTS, SV, and ASR model training. Users can easily access individual data samples, and selectively choose the desired portions of the data for customized usage with this framework. &#10;&#10;Due to the lack of annotations for in-the-wild data, directly evaluating the annotation performance of the AutoPrep framework can be challenging.&#10;In this paper, we first apply DNSMOS and PDNSMOS metrics to measure the improvement in speech quality achieved by AutoPrep.&#10;In addition, we use the processed data to train a DurIAN \cite{yu2019durian} multi-speaker TTS system to evaluate the proposed framework.&#10;Specifically, we compute the in-domain speaker similarity (SS) between the reference and synthesized speech to assess the quality of speaker labels and use human MOS scores to overall prepossessing quality.&#10;Experiments conducted on the open-sourced WenetSpeech \cite{zhang2022wenetspeech} and our self-collected AutoPrepWild speech data demonstrate that the proposed AutoPrep framework can generate processed data speech with similar DNSMOS and PDNSMOS scores compared to the open-sourced AIShell-3 \cite{shi2020aishell} and LibriTTS \cite{zen2019libritts} datasets. &#10;And the corresponding TTS system can achieve up to 0.68 indomain speaker similarity.&#10;&#10;The main contribution of this paper can be listed as follows: 1) The proposed AutoPrep framework is among the first works that can automatically improve speech quality and generate comprehensive high-quality annotations for in-the-wild speech data. 2) We conducted an experiment on a TTS system trained using the automatically generated data to validate the effectiveness of the proposed AutoPrep. 3) We open-source the speaker labels of WenetSpeech and GigaSpeech, generated by the proposed AutoPrep framework.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Speech Data Preprocessing, Engineering, Automatic Speech Recognition, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2303.03329" label="2303.03329">
        <attvalues>
          <attvalue for="0" value="End-to-End Speech Recognition: A Survey" />
          <attvalue for="1" value="  In the last decade of automatic speech recognition (ASR) research, the&#10;introduction of deep learning brought considerable reductions in word error&#10;rate of more than 50% relative, compared to modeling without deep learning. In&#10;the wake of this transition, a number of all-neural ASR architectures were&#10;introduced. These so-called end-to-end (E2E) models provide highly integrated,&#10;completely neural ASR models, which rely strongly on general machine learning&#10;knowledge, learn more consistently from data, while depending less on ASR&#10;domain-specific experience. The success and enthusiastic adoption of deep&#10;learning accompanied by more generic model architectures lead to E2E models now&#10;becoming the prominent ASR approach. The goal of this survey is to provide a&#10;taxonomy of E2E ASR models and corresponding improvements, and to discuss their&#10;properties and their relation to the classical hidden Markov model (HMM) based&#10;ASR architecture. All relevant aspects of E2E ASR are covered in this work:&#10;modeling, training, decoding, and external language model integration,&#10;accompanied by discussions of performance and deployment opportunities, as well&#10;as an outlook into potential future developments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18802" label="2305.18802">
        <attvalues>
          <attvalue for="0" value="LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus" />
          <attvalue for="1" value="  This paper introduces a new speech dataset called ``LibriTTS-R'' designed for&#10;text-to-speech (TTS) use. It is derived by applying speech restoration to the&#10;LibriTTS corpus, which consists of 585 hours of speech data at 24 kHz sampling&#10;rate from 2,456 speakers and the corresponding texts. The constituent samples&#10;of LibriTTS-R are identical to those of LibriTTS, with only the sound quality&#10;improved. Experimental results show that the LibriTTS-R ground-truth samples&#10;showed significantly improved sound quality compared to those in LibriTTS. In&#10;addition, neural end-to-end TTS trained with LibriTTS-R achieved speech&#10;naturalness on par with that of the ground-truth samples. The corpus is freely&#10;available for download from \url{http://www.openslr.org/141/}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.00406" label="2212.00406">
        <attvalues>
          <attvalue for="0" value="High Fidelity Speech Enhancement with Band-split RNN" />
          <attvalue for="1" value="  Despite the rapid progress in speech enhancement (SE) research, enhancing the&#10;quality of desired speech in environments with strong noise and interfering&#10;speakers remains challenging. In this paper, we extend the application of the&#10;recently proposed band-split RNN (BSRNN) model to full-band SE and personalized&#10;SE (PSE) tasks. To mitigate the effects of unstable high-frequency components&#10;in full-band speech, we perform bi-directional and uni-directional band-level&#10;modeling to low-frequency and high-frequency subbands, respectively. For PSE&#10;task, we incorporate a speaker enrollment module into BSRNN to utilize target&#10;speaker information. Moreover, we utilize a MetricGAN discriminator (MGD) and a&#10;multi-resolution spectrogram discriminator (MRSD) to improve perceptual quality&#10;metrics. Experimental results show that our system outperforms various&#10;top-ranking SE systems, achieves state-of-the-art (SOTA) results on the&#10;DNS-2020 test set and ranks among the top 3 in the DNS-2023 challenge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.17016" label="2210.17016">
        <attvalues>
          <attvalue for="0" value="Wespeaker: A Research and Production oriented Speaker Embedding Learning&#10;  Toolkit" />
          <attvalue for="1" value="  Speaker modeling is essential for many related tasks, such as speaker&#10;recognition and speaker diarization. The dominant modeling approach is&#10;fixed-dimensional vector representation, i.e., speaker embedding. This paper&#10;introduces a research and production oriented speaker embedding learning&#10;toolkit, Wespeaker. Wespeaker contains the implementation of scalable data&#10;management, state-of-the-art speaker embedding models, loss functions, and&#10;scoring back-ends, with highly competitive results achieved by structured&#10;recipes which were adopted in the winning systems in several speaker&#10;verification challenges. The application to other downstream tasks such as&#10;speaker diarization is also exhibited in the related recipe. Moreover, CPU- and&#10;GPU-compatible deployment codes are integrated for production-oriented&#10;development. The toolkit is publicly available at&#10;https://github.com/wenet-e2e/wespeaker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.01700" label="1909.01700">
        <attvalues>
          <attvalue for="0" value="DurIAN: Duration Informed Attention Network For Multimodal Synthesis" />
          <attvalue for="1" value="  In this paper, we present a generic and robust multimodal synthesis system&#10;that produces highly natural speech and facial expression simultaneously. The&#10;key component of this system is the Duration Informed Attention Network&#10;(DurIAN), an autoregressive model in which the alignments between the input&#10;text and the output acoustic features are inferred from a duration model. This&#10;is different from the end-to-end attention mechanism used, and accounts for&#10;various unavoidable artifacts, in existing end-to-end speech synthesis systems&#10;such as Tacotron. Furthermore, DurIAN can be used to generate high quality&#10;facial expression which can be synchronized with generated speech with/without&#10;parallel speech and face data. To improve the efficiency of speech generation,&#10;we also propose a multi-band parallel generation strategy on top of the WaveRNN&#10;model. The proposed Multi-band WaveRNN effectively reduces the total&#10;computational complexity from 9.8 to 5.5 GFLOPS, and is able to generate audio&#10;that is 6 times faster than real time on a single CPU core. We show that DurIAN&#10;could generate highly natural speech that is on par with current state of the&#10;art end-to-end systems, while at the same time avoid word skipping/repeating&#10;errors in those systems. Finally, a simple yet effective approach for&#10;fine-grained control of expressiveness of speech and facial expression is&#10;introduced.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.03370" label="2110.03370">
        <attvalues>
          <attvalue for="0" value="WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech&#10;  Recognition" />
          <attvalue for="1" value="  In this paper, we present WenetSpeech, a multi-domain Mandarin corpus&#10;consisting of 10000+ hours high-quality labeled speech, 2400+ hours weakly&#10;labeled speech, and about 10000 hours unlabeled speech, with 22400+ hours in&#10;total. We collect the data from YouTube and Podcast, which covers a variety of&#10;speaking styles, scenarios, domains, topics, and noisy conditions. An optical&#10;character recognition (OCR) based method is introduced to generate the&#10;audio/text segmentation candidates for the YouTube data on its corresponding&#10;video captions, while a high-quality ASR transcription system is used to&#10;generate audio/text pair candidates for the Podcast data. Then we propose a&#10;novel end-to-end label error detection approach to further validate and filter&#10;the candidates. We also provide three manually labelled high-quality test sets&#10;along with WenetSpeech for evaluation -- Dev for cross-validation purpose in&#10;training, Test_Net, collected from Internet for matched test, and&#10;Test\_Meeting, recorded from real meetings for more challenging mismatched&#10;test. Baseline systems trained with WenetSpeech are provided for three popular&#10;speech recognition toolkits, namely Kaldi, ESPnet, and WeNet, and recognition&#10;results on the three test sets are also provided as benchmarks. To the best of&#10;our knowledge, WenetSpeech is the current largest open-sourced Mandarin speech&#10;corpus with transcriptions, which benefits research on production-level speech&#10;recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11567" label="2010.11567">
        <attvalues>
          <attvalue for="0" value="AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines" />
          <attvalue for="1" value="  In this paper, we present AISHELL-3, a large-scale and high-fidelity&#10;multi-speaker Mandarin speech corpus which could be used to train multi-speaker&#10;Text-to-Speech (TTS) systems. The corpus contains roughly 85 hours of&#10;emotion-neutral recordings spoken by 218 native Chinese mandarin speakers.&#10;Their auxiliary attributes such as gender, age group and native accents are&#10;explicitly marked and provided in the corpus. Accordingly, transcripts in&#10;Chinese character-level and pinyin-level are provided along with the&#10;recordings. We present a baseline system that uses AISHELL-3 for multi-speaker&#10;Madarin speech synthesis. The multi-speaker speech synthesis system is an&#10;extension on Tacotron-2 where a speaker verification model and a corresponding&#10;loss regarding voice similarity are incorporated as the feedback constraint. We&#10;aim to use the presented corpus to build a robust synthesis model that is able&#10;to achieve zero-shot voice cloning. The system trained on this dataset also&#10;generalizes well on speakers that are never seen in the training process.&#10;Objective evaluation results from our experiments show that the proposed&#10;multi-speaker synthesis system achieves high voice similarity concerning both&#10;speaker embedding similarity and equal error rate measurement. The dataset,&#10;baseline system code and generated samples are available online.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08639" label="2104.08639">
        <attvalues>
          <attvalue for="0" value="AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages&#10;  with Adversarial Examples" />
          <attvalue for="1" value="  Capturing word meaning in context and distinguishing between correspondences&#10;and variations across languages is key to building successful multilingual and&#10;cross-lingual text representation models. However, existing multilingual&#10;evaluation datasets that evaluate lexical semantics &quot;in-context&quot; have various&#10;limitations. In particular, 1) their language coverage is restricted to&#10;high-resource languages and skewed in favor of only a few language families and&#10;areas, 2) a design that makes the task solvable via superficial cues, which&#10;results in artificially inflated (and sometimes super-human) performances of&#10;pretrained encoders, on many target languages, which limits their usefulness&#10;for model probing and diagnostics, and 3) little support for cross-lingual&#10;evaluation. In order to address these gaps, we present AM2iCo (Adversarial and&#10;Multilingual Meaning in Context), a wide-coverage cross-lingual and&#10;multilingual evaluation set; it aims to faithfully assess the ability of&#10;state-of-the-art (SotA) representation models to understand the identity of&#10;word meaning in cross-lingual contexts for 14 language pairs. We conduct a&#10;series of experiments in a wide range of setups and demonstrate the challenging&#10;nature of AM2iCo. The results reveal that current SotA pretrained encoders&#10;substantially lag behind human performance, and the largest gaps are observed&#10;for low-resource languages and languages dissimilar to English.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Cross-Lingual Evaluation of Word Meaning in Context.&#10;Going beyond readily available sense inventories required for WSD-style evaluations, the comprehensive benchmarks for evaluating word meaning in context cross-lingually are still few and far between. XL-WiC \cite{raganato-etal-2020-xl} extends the original English WiC framework of \cite{pilehvar-camacho-collados-2019-wic} to 12 other languages, but supports only monolingual evaluation, and suffers from issues such as small gaps between human and system performance. The SemEval-2021 shared task MCL-WiC does focus on cross-lingual WiC, but covers only five high-resource languages from three language families (English, French, Chinese, Arabic, Russian). Both XL-WiC and MCL-WiC mainly focus on common words and do not include less frequent concepts (e.g., named entities). Further, their language coverage and data availability are heavily skewed towards Indo-European languages.&#10;&#10;There are several other `non-WiC' datasets designed to evaluate cross-lingual context-aware lexical representations. Bilingual Contextual Word Similarity (BCWS) \cite{bcws} challenges a model to predict graded similarity of cross-lingual word pairs given sentential context, one in each language. In the Bilingual Token-level Sense Retrieval (BTSR) task \cite{liu-etal-2019-investigating}, given a query word in a source language context, a system must retrieve a meaning-equivalent target language word within a target language context. However, both BCWS and BTSR are again very restricted in terms of language coverage: BCWS covers only one language pair (EN-ZH), while BTSR contains two pairs (EN-ZH/ES). Further, they provide only test data: as such, they can merely be used as general intrinsic probes for pretrained models, but cannot support fine-tuning experiments and cannot fully expose the relevance of information available in pretrained models for downstream applications. This is problematic as intrinsic tasks in general do not necessarily correlate well with downstream performance \cite{chiu-etal-2016-intrinsic, glavas-etal-2019-properly}. &#10;&#10;AM$^2$iCo vs. Entity Linking.&#10;Our work is related to the entity linking (EL) task \cite{rao2013entity, cornolti2013framework,shen2014entity} similarly to how the original WiC (based on WordNet knowledge) is related to WSD. EL systems must map entities in context to a predefined knowledge base (KB). While WSD relies on the WordNet sense inventory, the EL task focuses on KBs such as Wikipedia and DBPedia. When each entity mention is mapped to a unique Wiki page, this procedure is termed wikification \cite{mihalcea2007wikify}. The cross-lingual wikification task \cite{ji2015overview, tsai-roth-2016-cross} grounds multilingual mentions to English Wikipedia pages. Similar to WSD, EL evaluation is tied to a specific KB. It thus faces similar limitations of WSD in terms of restricting meanings and their distinctions to those predefined in the inventory. In comparison, AM$^2$iCo leverages Wikipedia only as a convenient resource for extracting the examples, similar to how the original WiC work leverages WordNet. AM$^2$iCo itself is then framed on natural text, without requiring the modeling of the KBs. Also, in comparison with EL, AM$^2$iCo provides higher data quality and a more challenging evaluation of complex word-context interactions, achieved by a carefully designed data extraction and filtering procedure. &#10;&#10;" />
          <attvalue for="4" value="Multilingual Text Representation, Computer Science, Lexical Semantics Modeling, Linguistics, Artificial Intelligence, Natural Language Processing, Cross-Lingual Evaluation" />
        </attvalues>
      </node>
      <node id="2204.02492" label="2204.02492">
        <attvalues>
          <attvalue for="0" value="Towards End-to-end Unsupervised Speech Recognition" />
          <attvalue for="1" value="  Unsupervised speech recognition has shown great potential to make Automatic&#10;Speech Recognition (ASR) systems accessible to every language. However,&#10;existing methods still heavily rely on hand-crafted pre-processing. Similar to&#10;the trend of making supervised speech recognition end-to-end, we introduce&#10;wav2vec-U 2.0 which does away with all audio-side pre-processing and improves&#10;accuracy through better architecture. In addition, we introduce an auxiliary&#10;self-supervised objective that ties model predictions back to the input.&#10;Experiments show that wav2vec-U 2.0 improves unsupervised recognition results&#10;across different languages while being conceptually simpler.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.13439" label="2011.13439">
        <attvalues>
          <attvalue for="0" value="Unsupervised Domain Adaptation for Speech Recognition via Uncertainty&#10;  Driven Self-Training" />
          <attvalue for="1" value="  The performance of automatic speech recognition (ASR) systems typically&#10;degrades significantly when the training and test data domains are mismatched.&#10;In this paper, we show that self-training (ST) combined with an&#10;uncertainty-based pseudo-label filtering approach can be effectively used for&#10;domain adaptation. We propose DUST, a dropout-based uncertainty-driven&#10;self-training technique which uses agreement between multiple predictions of an&#10;ASR system obtained for different dropout settings to measure the model's&#10;uncertainty about its prediction. DUST excludes pseudo-labeled data with high&#10;uncertainties from the training, which leads to substantially improved ASR&#10;results compared to ST without filtering, and accelerates the training time due&#10;to a reduced training data set. Domain adaptation experiments using WSJ as a&#10;source domain and TED-LIUM 3 as well as SWITCHBOARD as the target domains show&#10;that up to 80% of the performance of a system trained on ground-truth data can&#10;be recovered.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.09422" label="2212.09422">
        <attvalues>
          <attvalue for="0" value="Human in the loop: How to effectively create coherent topics by manually&#10;  labeling only a few documents per class" />
          <attvalue for="1" value="  Few-shot methods for accurate modeling under sparse label-settings have&#10;improved significantly. However, the applications of few-shot modeling in&#10;natural language processing remain solely in the field of document&#10;classification. With recent performance improvements, supervised few-shot&#10;methods, combined with a simple topic extraction method pose a significant&#10;challenge to unsupervised topic modeling methods. Our research shows that&#10;supervised few-shot learning, combined with a simple topic extraction method,&#10;can outperform unsupervised topic modeling techniques in terms of generating&#10;coherent topics, even when only a few labeled documents per class are used.&#10;" />
          <attvalue for="2" value="&#10;The identification of latent topics in large text corpora has undergone a great deal of development. However, uncovering the hidden semantics of large text corpora is still, if not of ever-increasing interest. Scientific methods continue to evolve and achieve increasingly impressive results in terms of topic coherence \cite{larochelle2012neural, srivastava2017autoencoding, chien2018latent, wang2019atm, dieng2020topic}.&#10;The practical relevance of such methods is evident from the large number of practical application papers alone. Topic models for information extraction are used, for example, for applied research in education \cite{granic2019technology}, offsite construction \cite{liu2019trending}, bioinformatics \cite{liu2016overview}, communication sciences \cite{maier2018applying} and many other practical areas (e.g. \cite{hall2008studying, daud2010knowledge, boyd2017applications, jelodar2019latent, hannigan2019topic}).&#10;&#10;While all of these methods take an unsupervised approach, few-shot methods achieve remarkable results in various supervised label-scarse settings. The metrics of interest are in this case not the coherence of clusters, but model accuracy, F1 score, or precision. Huggingfaces Sentence Transformer Finetuning (SetFit) \cite{tunstall2022efficient} allows for such a small amount of labeled data while achieving impressive classification results that unsupervised methods are heavily challenged.&#10;&#10;The idea is simple. When less and less labeled documents per class are necessary for supervised methods to achieve state-of-the-art results, human input by manually labeling a few documents becomes an attractive option for unsupervised tasks such as document clustering. By leveraging pre-trained sentence transformers \cite{reimers-2019-sentence-bert} and class-based term frequency inverse document frequency (tf-idf) for topic extraction, we can generate coherent topics with only a few labeled documents per class. As a result, manually labeling a training data set and subsequently leveraging SetFit reduces the tiresome and time- and money-intensive manual labeling by such a dramatic amount, that it is a viable alternative to unsupervised approaches.&#10;&#10;Contributions The contributions of the paper can be summarized as follows:&#10;&#10;\begin{enumerate}&#10; \item We present a method for Document Classification and subsequent Topic Extraction (DCTE) based on SetFit. The proposed method generates coherent topics from only a few labeled documents.&#10; \item We conduct a benchmark study, comparing the proposed approach with state-of-the-art topic models and document clustering methods.&#10; \item We outperform competitive benchmark models on three standard datasets in terms of topic coherence and create informative topics.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Few-Shot Learning, Machine Learning, Computer Science, Linguistics, Natural Language Processing, Statistics, Topic Modeling" />
        </attvalues>
      </node>
      <node id="2310.18696" label="2310.18696">
        <attvalues>
          <attvalue for="0" value="Probing LLMs for Joint Encoding of Linguistic Categories" />
          <attvalue for="1" value="  Large Language Models (LLMs) exhibit impressive performance on a range of NLP&#10;tasks, due to the general-purpose linguistic knowledge acquired during&#10;pretraining. Existing model interpretability research (Tenney et al., 2019)&#10;suggests that a linguistic hierarchy emerges in the LLM layers, with lower&#10;layers better suited to solving syntactic tasks and higher layers employed for&#10;semantic processing. Yet, little is known about how encodings of different&#10;linguistic phenomena interact within the models and to what extent processing&#10;of linguistically-related categories relies on the same, shared model&#10;representations. In this paper, we propose a framework for testing the joint&#10;encoding of linguistic categories in LLMs. Focusing on syntax, we find evidence&#10;of joint encoding both at the same (related part-of-speech (POS) classes) and&#10;different (POS classes and related syntactic dependency relations) levels of&#10;linguistic hierarchy. Our cross-lingual experiments show that the same patterns&#10;hold across languages in multilingual LLMs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancements in natural language processing (NLP) can be attributed to the development and&#10;pretraining of large language models (LLMs) such as BERT, GPT-3, and many&#10;others~\cite{devlin_bert_2019, brown_language_2020, touvron_llama_2023}. For their intended use of&#10;providing general-purpose language representations suitable for many NLP tasks, these models must&#10;efficiently capture a wide range of linguistic features within their finite capacity. Despite their&#10;success, little is known about the way in which different types of linguistic information are&#10;organized in these models. Systematically understanding how these models represent linguistic&#10;phenomena and their interaction is crucial for the development of more effective NLP methods.&#10;&#10;Existing research probed LLMs for their encoding of various linguistic properties such as&#10;agreement~\cite{jawahar2019does}, word order and sentence&#10;structure~\cite{tenney_what_2018,hewitt_structural_2019}, co-reference~\cite{tenney_bert_2019},&#10;semantics~\cite{ettinger_what_2020} and multilinguality~\cite{ravishankar_multilingual_2019,&#10;&#09;libovicky_language_2020}. Taking a step further, \cite{tenney_bert_2019} and&#10;\cite{clark_what_2019} studied where linguistic information is encoded in LLMs by probing different&#10;layers. Their results demonstrated that a linguistic hierarchy emerges in BERT representations, with&#10;lower layers capturing local syntax and higher layers being employed in higher-level semantic and&#10;discourse tasks. However, we do not yet understand how encodings of different linguistic phenomena&#10;interact within the models and to what extent processing of linguistically-related categories relies&#10;on the same, shared model representations.&#10;&#10;There are many dependencies between processing different linguistic phenomena: for instance,&#10;information about a word’s part of speech is likely to be employed when disambiguating its word&#10;sense. Alternatively, lower-level syntax is an important first step for semantic composition and&#10;natural language understanding tasks. In this work, we investigate how the (hierarchical)&#10;dependencies between different linguistic categories are encoded in LLMs, focusing on syntax. We ask&#10;a set of novel questions: (1) how related syntactic categories (e.g. different parts of speech&#10;(POS), such as noun or verb) are encoded within the models; (2) how syntactic&#10;categories at different levels of the linguistic hierarchy (e.g. POS classes and syntactic&#10;dependency relations) interact within the model; and (3) whether the observed patterns hold across&#10;languages.&#10;&#10;To answer these questions, we propose a framework for testing the joint encoding of linguistic&#10;categories in LLMs. Specifically, we investigate to what extent the information about distinct&#10;linguistic categories is shared in the parameters of the model. Our approach (see Fig.&#10;\ref{fig:method_overview}) is inspired by the work of \cite{choenni_investigating_2022}, who&#10;studied how LLMs share information across languages. We employ their cross-neutralization&#10;method, but extend it to study how information is encoded across linguistic categories in two&#10;syntactic tasks---part-of-speech (POS) tagging and dependency (DEP) labeling. In short, we test&#10;whether removing information on one syntactic category results in a failure to process another,&#10;related one. For instance, would removing a representation of the verb class hurt the&#10;model's ability to identify the verbs' syntactic dependencies, while still succeeding in this task&#10;on other categories, e.g.\ nouns? This provides insight into whether these categories are&#10;jointly encoded.&#10;&#10;We study joint encoding patterns within both a monolingual and multilingual LLM, namely&#10;RoBERTa~\cite{liu_roberta_2019} and XLM-R~\cite{conneau_unsupervised_2020}. For the latter, we&#10;focus our analysis on English, Greek, and Italian, investigating to what extent multilingual models&#10;encode information for POS tags and DEP relations in a language-agnostic manner. We find that POS&#10;tags that are linguistically related are indeed jointly encoded by both the monolingual and&#10;multilingual models, and observe similar joint encoding patterns across all three languages.&#10;Moreover, we obtain further evidence of both language-agnostic and language-specific encoding within&#10;multilingual models, given that representations specific to POS tags and DEP relations can be&#10;approximately transferred across languages without a substantial impact on performance. Lastly, we&#10;find evidence of joint encoding between related POS tags and DEP relations, suggesting information&#10;sharing across tasks at different levels of the linguistic hierarchy.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Linguistic Hierarchy Analysis, Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Multilingual Processing, Natural Language Processing, Language Model Interpretability" />
        </attvalues>
      </node>
      <node id="1507.04808" label="1507.04808">
        <attvalues>
          <attvalue for="0" value="Building End-To-End Dialogue Systems Using Generative Hierarchical&#10;  Neural Network Models" />
          <attvalue for="1" value="  We investigate the task of building open domain, conversational dialogue&#10;systems based on large dialogue corpora using generative models. Generative&#10;models produce system responses that are autonomously generated word-by-word,&#10;opening up the possibility for realistic, flexible interactions. In support of&#10;this goal, we extend the recently proposed hierarchical recurrent&#10;encoder-decoder neural network to the dialogue domain, and demonstrate that&#10;this model is competitive with state-of-the-art neural language models and&#10;back-off n-gram models. We investigate the limitations of this and similar&#10;approaches, and show how its performance can be improved by bootstrapping the&#10;learning from a larger question-answer pair corpus and from pretrained word&#10;embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.11638" label="2409.11638">
        <attvalues>
          <attvalue for="0" value="BanStereoSet: A Dataset to Measure Stereotypical Social Biases in LLMs&#10;  for Bangla" />
          <attvalue for="1" value="  This study presents BanStereoSet, a dataset designed to evaluate&#10;stereotypical social biases in multilingual LLMs for the Bangla language. In an&#10;effort to extend the focus of bias research beyond English-centric datasets, we&#10;have localized the content from the StereoSet, IndiBias, and Kamruzzaman et.&#10;al.'s datasets, producing a resource tailored to capture biases prevalent&#10;within the Bangla-speaking community. Our BanStereoSet dataset consists of&#10;1,194 sentences spanning 9 categories of bias: race, profession, gender,&#10;ageism, beauty, beauty in profession, region, caste, and religion. This dataset&#10;not only serves as a crucial tool for measuring bias in multilingual LLMs but&#10;also facilitates the exploration of stereotypical bias across different social&#10;categories, potentially guiding the development of more equitable language&#10;technologies in Bangladeshi contexts. Our analysis of several language models&#10;using this dataset indicates significant biases, reinforcing the necessity for&#10;culturally and linguistically adapted datasets to develop more equitable&#10;language technologies.&#10;" />
          <attvalue for="2" value="&#10;&#10;The usage of LLMs has increased significantly, with people from all over the world employing these models for various tasks. Not limited to English speakers, individuals from diverse linguistic backgrounds use LLMs in their native languages. State-of-the-art multilingual LLMs such as GPT-4 \cite{achiam2023gpt}, Llama~\cite{touvron2023llama}, and Gemini~\cite{team2023gemini} cater to this global user base. However, many recent studies have revealed undesirable biases and stereotypes in these models \cite{kotek2023gender, kamruzzaman2024global}. Some datasets have been proposed to measure these biases, predominantly in English \cite{nadeem2021stereoset, kamruzzaman-etal-2024-investigating, nangia2020crows}. There are studies in other languages like French \cite{neveol2022french}, Hindi \cite{sahoo2024indibias}, Italian \cite{sanguinetti2020haspeede}, and Arabic \cite{lauscher2020araweat}. Research on bias specific to the Bengali remains sparse. &#10;&#10;Bengali, with 233.7 million native speakers worldwide, ranks fifth worldwide.&#10;Recent studies focusing on Bengali have primarily addressed gender and religious biases. For instance, \cite{sadhu2024social} proposed a dataset concentrating on gender and religion biases through persona-based experiments, such as adopting the role of a typical Bengali person. Similarly, \cite{das2023toward} evaluated these biases in Bengali. \cite{sadhu2024empirical} found stereotypical emotional associations between genders in Bengali, using an emotion dataset.&#10;&#10;Stereotypical biases are not confined to gender and religion alone; they extend to race, profession, beauty, ageism, caste, and region, impacting individuals as significantly as gender and religion. To the best of our knowledge, there is no dataset that comprehensively covers a broader spectrum of bias categories while also focusing on the Bengali cultural context. We propose BanStereoSet, a fill-in-the-blank type dataset designed to measure stereotypical biases in Bangla, encompassing a wide range bias categories. \ref{fig:example} shows an example from our dataset, where we give LLMs three options to fill in the blank. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Social Bias Analysis, Computer Science, Linguistics, Multilingual Dataset Development, Language Model Bias, Computational Linguistics, Social Psychology, Social Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2308.14921" label="2308.14921">
        <attvalues>
          <attvalue for="0" value="Gender bias and stereotypes in Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have made substantial progress in the past&#10;several months, shattering state-of-the-art benchmarks in many domains. This&#10;paper investigates LLMs' behavior with respect to gender stereotypes, a known&#10;issue for prior models. We use a simple paradigm to test the presence of gender&#10;bias, building on but differing from WinoBias, a commonly used gender bias&#10;dataset, which is likely to be included in the training data of current LLMs.&#10;We test four recently published LLMs and demonstrate that they express biased&#10;assumptions about men and women's occupations. Our contributions in this paper&#10;are as follows: (a) LLMs are 3-6 times more likely to choose an occupation that&#10;stereotypically aligns with a person's gender; (b) these choices align with&#10;people's perceptions better than with the ground truth as reflected in official&#10;job statistics; (c) LLMs in fact amplify the bias beyond what is reflected in&#10;perceptions or the ground truth; (d) LLMs ignore crucial ambiguities in&#10;sentence structure 95% of the time in our study items, but when explicitly&#10;prompted, they recognize the ambiguity; (e) LLMs provide explanations for their&#10;choices that are factually inaccurate and likely obscure the true reason behind&#10;their predictions. That is, they provide rationalizations of their biased&#10;behavior. This highlights a key property of these models: LLMs are trained on&#10;imbalanced datasets; as such, even with the recent successes of reinforcement&#10;learning with human feedback, they tend to reflect those imbalances back at us.&#10;As with other types of societal biases, we suggest that LLMs must be carefully&#10;tested to ensure that they treat minoritized individuals and communities&#10;equitably.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.20147" label="2403.20147">
        <attvalues>
          <attvalue for="0" value="IndiBias: A Benchmark Dataset to Measure Social Biases in Language&#10;  Models for Indian Context" />
          <attvalue for="1" value="  The pervasive influence of social biases in language data has sparked the&#10;need for benchmark datasets that capture and evaluate these biases in Large&#10;Language Models (LLMs). Existing efforts predominantly focus on English&#10;language and the Western context, leaving a void for a reliable dataset that&#10;encapsulates India's unique socio-cultural nuances. To bridge this gap, we&#10;introduce IndiBias, a comprehensive benchmarking dataset designed specifically&#10;for evaluating social biases in the Indian context. We filter and translate the&#10;existing CrowS-Pairs dataset to create a benchmark dataset suited to the Indian&#10;context in Hindi language. Additionally, we leverage LLMs including ChatGPT and&#10;InstructGPT to augment our dataset with diverse societal biases and stereotypes&#10;prevalent in India. The included bias dimensions encompass gender, religion,&#10;caste, age, region, physical appearance, and occupation. We also build a&#10;resource to address intersectional biases along three intersectional&#10;dimensions. Our dataset contains 800 sentence pairs and 300 tuples for bias&#10;measurement across different demographics. The dataset is available in English&#10;and Hindi, providing a size comparable to existing benchmark datasets.&#10;Furthermore, using IndiBias we compare ten different language models on&#10;multiple bias measurement metrics. We observed that the language models exhibit&#10;more bias across a majority of the intersectional groups.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.01575" label="2011.01575">
        <attvalues>
          <attvalue for="0" value="AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings" />
          <attvalue for="1" value="  Recent work has shown that distributional word vector spaces often encode&#10;human biases like sexism or racism. In this work, we conduct an extensive&#10;analysis of biases in Arabic word embeddings by applying a range of recently&#10;introduced bias tests on a variety of embedding spaces induced from corpora in&#10;Arabic. We measure the presence of biases across several dimensions, namely:&#10;embedding models (Skip-Gram, CBOW, and FastText) and vector sizes, types of&#10;text (encyclopedic text, and news vs. user-generated content), dialects&#10;(Egyptian Arabic vs. Modern Standard Arabic), and time (diachronic analyses&#10;over corpora from different time periods). Our analysis yields several&#10;interesting findings, e.g., that implicit gender bias in embeddings trained on&#10;Arabic news corpora steadily increases over time (between 2007 and 2017). We&#10;make the Arabic bias specifications (AraWEAT) publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.03536" label="2407.03536">
        <attvalues>
          <attvalue for="0" value="Social Bias in Large Language Models For Bangla: An Empirical Study on&#10;  Gender and Religious Bias" />
          <attvalue for="1" value="  The rapid growth of Large Language Models (LLMs) has put forward the study of&#10;biases as a crucial field. It is important to assess the influence of different&#10;types of biases embedded in LLMs to ensure fair use in sensitive fields.&#10;Although there have been extensive works on bias assessment in English, such&#10;efforts are rare and scarce for a major language like Bangla. In this work, we&#10;examine two types of social biases in LLM generated outputs for Bangla&#10;language. Our main contributions in this work are: (1) bias studies on two&#10;different social biases for Bangla (2) a curated dataset for bias measurement&#10;benchmarking (3) two different probing techniques for bias detection in the&#10;context of Bangla. This is the first work of such kind involving bias&#10;assessment of LLMs for Bangla to the best of our knowledge. All our code and&#10;resources are publicly available for the progress of bias related research in&#10;Bangla NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.06432" label="2407.06432">
        <attvalues>
          <attvalue for="0" value="An Empirical Study of Gendered Stereotypes in Emotional Attributes for&#10;  Bangla in Multilingual Large Language Models" />
          <attvalue for="1" value="  The influence of Large Language Models (LLMs) is rapidly growing, automating&#10;more jobs over time. Assessing the fairness of LLMs is crucial due to their&#10;expanding impact. Studies reveal the reflection of societal norms and biases in&#10;LLMs, which creates a risk of propagating societal stereotypes in downstream&#10;tasks. Many studies on bias in LLMs focus on gender bias in various NLP&#10;applications. However, there's a gap in research on bias in emotional&#10;attributes, despite the close societal link between emotion and gender. This&#10;gap is even larger for low-resource languages like Bangla. Historically, women&#10;are associated with emotions like empathy, fear, and guilt, while men are&#10;linked to anger, bravado, and authority. This pattern reflects societal norms&#10;in Bangla-speaking regions. We offer the first thorough investigation of&#10;gendered emotion attribution in Bangla for both closed and open source LLMs in&#10;this work. Our aim is to elucidate the intricate societal relationship between&#10;gender and emotion specifically within the context of Bangla. We have been&#10;successful in showing the existence of gender bias in the context of emotions&#10;in Bangla through analytical methods and also show how emotion attribution&#10;changes on the basis of gendered role selection in LLMs. All of our resources&#10;including code and data are made publicly available to support future research&#10;on Bangla NLP.&#10;  Warning: This paper contains explicit stereotypical statements that many may&#10;find offensive.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.09520" label="2310.09520">
        <attvalues>
          <attvalue for="0" value="Reward-Augmented Decoding: Efficient Controlled Text Generation With a&#10;  Unidirectional Reward Model" />
          <attvalue for="1" value="  While large language models have proven effective in a huge range of&#10;downstream applications, they often generate text that is problematic or lacks&#10;a desired attribute. In this paper, we introduce Reward-Augmented Decoding&#10;(RAD), a text generation procedure that uses a small unidirectional reward&#10;model to encourage a language model to generate text that has certain&#10;properties. Specifically, RAD uses the reward model to score generations as&#10;they are produced and rescales sampling probabilities to favor high-reward&#10;tokens. By using a unidirectional reward model, RAD can cache activations from&#10;prior generation steps to decrease computational overhead. Through experiments&#10;on generating non-toxic and sentiment-controlled text, we demonstrate that RAD&#10;performs best among methods that change only the generation procedure and&#10;matches the performance of state-of-the-art methods that involve re-training&#10;the language model. We further validate that RAD is effective on very large&#10;language models while incurring a minimal computational overhead.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs, \cite{gopher,chinchilla,scao2022bloom,llama}) are seeing widespread adoption thanks to the fact that they can perform many language tasks and generate coherent long-form text.&#10;As LLMs are deployed in situations where they interact with humans, it can be beneficial to control the language model so that it generates text with certain properties~\cite{sudhakar2019delete} -- for example, we might desire generations that are unbiased, non-toxic, and helpful.&#10;In addition, we may want models to output text with specific properties, such as having a positive sentiment, a certain writing style, etc.&#10;Typically, LLMs pre-trained on uncurated large-scale text corpora can generate text that does not have these desired attributes~\cite{wallace2019triggers,RealToxicityPrompts}, which motivates the need for techniques that enable controllable text generation.&#10;Such techniques can be seen as providing a means to condition text generation on a desired attribute.&#10;&#10;A straightforward way to control the text generated by an LLM is to perform additional training on data that has desired properties~\cite{DAPT}.&#10;Alternatively, an LLM can be trained with ``control codes''~\cite{ctrl,quark} that indicate text characteristics and can be used to induce the LLM to generate content with those characteristics.&#10;If available, annotated human preferences can be used to train a reward model that is then used to train a language model with reinforcement learning~\cite{rlhf, kim-etal-2023-critic}.&#10;A drawback of these methods is that they can degrade performance on text that is different from the data used for additional training.&#10;Besides, work done to control one language model cannot be reused to control another language model.&#10;Moreover, the additional training cost can be prohibitively expensive, especially for very large models.&#10;&#10;One way to avoid the cost and shortcomings of additional training is to instead modify the decoding procedure used to generate text from a language model \cite{Chaffin_2022}.&#10;For example, weighted decoding modifies the probabilities assigned to each token during decoding using an auxiliary model.&#10;Most weighted decoding methods~\cite{holtzman-etal-2018-learning,gedi,dexperts,fudge,sitdikov2022classifiers} obtain an attribute probability $P(c|X)$ from a separate reward model (typically smaller than the base language model) and construct class-conditional text probabilities following Bayes rule, $P(X|c) \propto P(X)P(c|X)$, where $c$ is an attribute class and $P(X)$ is the distribution over natural language sequences $X$. During decoding, \cite{gedi} and \cite{dexperts} process signals from auxiliary generative models, whereas \cite{fudge} and \cite{sitdikov2022classifiers} evaluate intermediate sequences.&#10;Weighted decoding only requires access to the next-step probabilities output by a language model, does not require expensive training, and is often modular, i.e.\ a single reward model can be reused with many language models.&#10;Despite these benefits, weighted decoding can significantly increase the cost of decoding and often underperforms methods that involve further training~\cite{see-etal-2019-makes}.&#10;&#10;In this paper, we close the gap between weighted decoding and re-training by introducing reward-augmented decoding (RAD), an efficient, effective, and modular weighted decoding method that steers text generation based on the reward returned by an attribute-specific reward model.&#10;In particular, RAD uses a unidirectional reward model trained to output a reward representing how well a given sequence aligns with a desired attribute.&#10;The unidirectionality of the reward model allows caching intermediate activations as the sequence is generated, greatly decreasing computational costs.&#10;During decoding, the tokens with the top-$k$ highest probabilities are rescaled according to the reward model so that tokens that better reflect the desired attribute are more likely to be chosen as the next generated token. &#10;&#10;To validate RAD's effectiveness, we evaluate it on standard detoxification and sentiment-controlled generation tasks, showing that it steers text generation towards a desired attribute without sacrificing much diversity and fluency.&#10;We ultimately find that RAD outperforms other weighted decoding methods and achieves results comparable to methods that involve additional training.&#10;We further validate RAD in a real-world large-scale setting by showing it is effective and introduces minimal computational overhead when applied to the LLaMA~\cite{llama} family of language models with up to 65B parameters.&#10;&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Generation Techniques, Machine Learning, Computer Science, Linguistics, Artificial Intelligence Applications, Artificial Intelligence, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2104.05218" label="2104.05218">
        <attvalues>
          <attvalue for="0" value="FUDGE: Controlled Text Generation With Future Discriminators" />
          <attvalue for="1" value="  We propose Future Discriminators for Generation (FUDGE), a flexible and&#10;modular method for controlled text generation. Given a pre-existing model G for&#10;generating text from a distribution of interest, FUDGE enables conditioning on&#10;a desired attribute a (for example, formality) while requiring access only to&#10;G's output logits. FUDGE learns an attribute predictor operating on a partial&#10;sequence, and uses this predictor's outputs to adjust G's original&#10;probabilities. We show that FUDGE models terms corresponding to a Bayesian&#10;decomposition of the conditional distribution of G given attribute a. Moreover,&#10;FUDGE can easily compose predictors for multiple desired attributes. We&#10;evaluate FUDGE on three tasks -- couplet completion in poetry, topic control in&#10;language generation, and formality change in machine translation -- and observe&#10;gains in all three tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.07276" label="2205.07276">
        <attvalues>
          <attvalue for="0" value="Classifiers are Better Experts for Controllable Text Generation" />
          <attvalue for="1" value="  This paper proposes a simple method for controllable text generation based on&#10;weighting logits with a free-form classifier, namely CAIF sampling. Using an&#10;arbitrary text classifier, we adjust a small part of a language model's logits&#10;and guide text generation towards or away from classifier prediction. We&#10;experimented with toxicity avoidance and sentiment control tasks and showed&#10;that the proposed method significantly outperforms recent PPLM, GeDi, and&#10;DExperts on PPL and task accuracy metrics based on the external classifier of&#10;generated texts. In addition, compared to other approaches, it is easier to&#10;implement and tune and has significantly fewer restrictions and requirements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.13005" label="2004.13005">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Information Retrieval with BERT" />
          <attvalue for="1" value="  Multiple neural language models have been developed recently, e.g., BERT and&#10;XLNet, and achieved impressive results in various NLP tasks including sentence&#10;classification, question answering and document ranking. In this paper, we&#10;explore the use of the popular bidirectional language model, BERT, to model and&#10;learn the relevance between English queries and foreign-language documents in&#10;the task of cross-lingual information retrieval. A deep relevance matching&#10;model based on BERT is introduced and trained by finetuning a pretrained&#10;multilingual BERT model with weak supervision, using home-made CLIR training&#10;data derived from parallel corpora. Experimental results of the retrieval of&#10;Lithuanian documents against short English queries show that our model is&#10;effective and outperforms the competitive baseline approaches.&#10;" />
          <attvalue for="2" value="&#10;&#10;A traditional cross-lingual information retrieval (CLIR) system consists of two components: machine translation and monolingual information retrieval~\cite{nie2010}. The idea is to solve the translation problem first, then the cross-lingual IR problem become monolingual IR. However, the performance of translation-based approaches is limited by the quality of the machine translation and it needs to handle to translation ambiguity~\cite{Zhou2012}. One possible solution is to consider the translation alternatives of individual words of queries or documents as in~\cite{zbib19,XuEMNLP00}, which provides more possibilities for matching query words in relevant documents compared to using single translations. But the alignment information is necessarily required in the training stage of the CLIR system to extract target-source word pairs from parallel data and this is not a trivial task.&#10;&#10;To achieve good performance in IR, deep neural networks have been widely used in this task. These approaches can be roughly divided into two categories. The first class of approaches uses pretrained word representations or embeddings, such as word2vec~\cite{tomas2013} and GloVe~\cite{pennington2014glove}, directly to improve IR models. Usually these word embeddings are pretrained on large scale text corpora using co-occurrence statistics, so they have modeled the underlying data distribution implicitly and should be helpful for building discriminative models.~\cite{VulicSIGIR15} and ~\cite{LitschkoSIGIR18} used pretrained bilingual embeddings to represent queries and foreign documents, and then ranked documents by cosine similarity. ~\cite{zheng2015} used word2vec embeddings to learn query term weights. However, their training objectives of trained neural embeddings are different from the objective of IR.&#10;&#10;The second set of approaches design and train deep neural networks based on IR objectives. These methods have shown impressive results on monolingual IR datasets~\cite{XiongSIGIR17,guocikm16,mostafaSIGIR17}. They usually rely on large amounts of query-document relevance annotated data that are expensive to obtain, especially for low-resource language pairs in cross-lingual IR tasks. Moreover, it is not clear whether they generalize well when documents and queries are in different languages.&#10;&#10;Recently multiple pretrained language models have been developed such as BERT~\cite{devlin-etal-2019-bert} and XLNet~\cite{xlnet2019}, that model the underlying data distribution and learn the linguistic patterns or features in language. These models have outperformed traditional word embeddings on various NLP tasks~\cite{xlnet2019,devlin-etal-2019-bert,peters-etal-2018-deep,Lan2019ALBERTAL}. These pretrained models also provided new opportunities for IR. Therefore, several recent works have successfully applied BERT pretrained models for monolingual IR~\cite{daiSIGIR19,akkalyoncu-yilmaz-etal-2019-applying} and passage re-ranking~\cite{Nogueira19}.&#10;&#10;In this paper, we extend and apply BERT as a ranker for CLIR. We introduce a cross-lingual deep relevance matching model for CLIR based on BERT. We finetune a pretrained multilingual model with home-made CLIR data and obtain very promising results. In order to finetune the model, we construct a large amount of training data from parallel data, which is mainly used for machine translation and is much easier to obtain compared to the relevance labels of query-document pairs. In addition, we don't require the source-target alignment information to construct training samples and avoid the quality issues of machine translation in traditional CLIR. The entire model is specifically optimized using a CLIR objective. Our main contributions are:&#10;&#10;\begin{itemize}&#10;\item We introduce a cross-lingual deep relevance architecture with BERT, where a pretrained multilingual BERT model is adapted for cross-lingual IR.&#10;\item We define a proxy CLIR task which can be used to easily construct CLIR training data from bitext data, without requiring any amount of relevance labels of query-document pairs in different languages.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Information Retrieval, Artificial Intelligence, Neural Language Models, Information Science, Cross-Lingual Retrieval" />
        </attvalues>
      </node>
      <node id="2111.05988" label="2111.05988">
        <attvalues>
          <attvalue for="0" value="Cross-language Information Retrieval" />
          <attvalue for="1" value="  Two key assumptions shape the usual view of ranked retrieval: (1) that the&#10;searcher can choose words for their query that might appear in the documents&#10;that they wish to see, and (2) that ranking retrieved documents will suffice&#10;because the searcher will be able to recognize those which they wished to find.&#10;When the documents to be searched are in a language not known by the searcher,&#10;neither assumption is true. In such cases, Cross-Language Information Retrieval&#10;(CLIR) is needed. This chapter reviews the state of the art for CLIR and&#10;outlines some open research questions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.06613" label="1706.06613">
        <attvalues>
          <attvalue for="0" value="End-to-End Neural Ad-hoc Ranking with Kernel Pooling" />
          <attvalue for="1" value="  This paper proposes K-NRM, a kernel based neural model for document ranking.&#10;Given a query and a set of documents, K-NRM uses a translation matrix that&#10;models word-level similarities via word embeddings, a new kernel-pooling&#10;technique that uses kernels to extract multi-level soft match features, and a&#10;learning-to-rank layer that combines those features into the final ranking&#10;score. The whole model is trained end-to-end. The ranking layer learns desired&#10;feature patterns from the pairwise ranking loss. The kernels transfer the&#10;feature patterns into soft-match targets at each similarity level and enforce&#10;them on the translation matrix. The word embeddings are tuned accordingly so&#10;that they can produce the desired soft matches. Experiments on a commercial&#10;search engine's query log demonstrate the improvements of K-NRM over prior&#10;feature-based and neural-based states-of-the-art, and explain the source of&#10;K-NRM's advantage: Its kernel-guided embedding encodes a similarity metric&#10;tailored for matching query words to document words, and provides effective&#10;multi-level soft matches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.08611" label="1711.08611">
        <attvalues>
          <attvalue for="0" value="A Deep Relevance Matching Model for Ad-hoc Retrieval" />
          <attvalue for="1" value="  In recent years, deep neural networks have led to exciting breakthroughs in&#10;speech recognition, computer vision, and natural language processing (NLP)&#10;tasks. However, there have been few positive results of deep models on ad-hoc&#10;retrieval tasks. This is partially due to the fact that many important&#10;characteristics of the ad-hoc retrieval task have not been well addressed in&#10;deep models yet. Typically, the ad-hoc retrieval task is formalized as a&#10;matching problem between two pieces of text in existing work using deep models,&#10;and treated equivalent to many NLP tasks such as paraphrase identification,&#10;question answering and automatic conversation. However, we argue that the&#10;ad-hoc retrieval task is mainly about relevance matching while most NLP&#10;matching tasks concern semantic matching, and there are some fundamental&#10;differences between these two matching tasks. Successful relevance matching&#10;requires proper handling of the exact matching signals, query term importance,&#10;and diverse matching requirements. In this paper, we propose a novel deep&#10;relevance matching model (DRMM) for ad-hoc retrieval. Specifically, our model&#10;employs a joint deep architecture at the query term level for relevance&#10;matching. By using matching histogram mapping, a feed forward matching network,&#10;and a term gating network, we can effectively deal with the three relevance&#10;matching factors mentioned above. Experimental results on two representative&#10;benchmark collections show that our model can significantly outperform some&#10;well-known retrieval models as well as state-of-the-art deep matching models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.08803" label="1704.08803">
        <attvalues>
          <attvalue for="0" value="Neural Ranking Models with Weak Supervision" />
          <attvalue for="1" value="  Despite the impressive improvements achieved by unsupervised deep neural&#10;networks in computer vision and NLP tasks, such improvements have not yet been&#10;observed in ranking for information retrieval. The reason may be the complexity&#10;of the ranking problem, as it is not obvious how to learn from queries and&#10;documents when no supervised signal is available. Hence, in this paper, we&#10;propose to train a neural ranking model using weak supervision, where labels&#10;are obtained automatically without human annotators or any external resources&#10;(e.g., click data). To this aim, we use the output of an unsupervised ranking&#10;model, such as BM25, as a weak supervision signal. We further train a set of&#10;simple yet effective ranking models based on feed-forward neural networks. We&#10;study their effectiveness under various learning scenarios (point-wise and&#10;pair-wise models) and using different input representations (i.e., from&#10;encoding query-document pairs into dense/sparse vectors to using word embedding&#10;representation). We train our networks using tens of millions of training&#10;instances and evaluate it on two standard collections: a homogeneous news&#10;collection(Robust) and a heterogeneous large-scale web collection (ClueWeb).&#10;Our experiments indicate that employing proper objective functions and letting&#10;the networks to learn the input representation based on weakly supervised data&#10;leads to impressive performance, with over 13% and 35% MAP improvements over&#10;the BM25 model on the Robust and the ClueWeb collections. Our findings also&#10;suggest that supervised neural ranking models can greatly benefit from&#10;pre-training on large amounts of weakly labeled data that can be easily&#10;obtained from unsupervised IR models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.04085" label="1901.04085">
        <attvalues>
          <attvalue for="0" value="Passage Re-ranking with BERT" />
          <attvalue for="1" value="  Recently, neural models pretrained on a language modeling task, such as ELMo&#10;(Peters et al., 2017), OpenAI GPT (Radford et al., 2018), and BERT (Devlin et&#10;al., 2018), have achieved impressive results on various natural language&#10;processing tasks such as question-answering and natural language inference. In&#10;this paper, we describe a simple re-implementation of BERT for query-based&#10;passage re-ranking. Our system is the state of the art on the TREC-CAR dataset&#10;and the top entry in the leaderboard of the MS MARCO passage retrieval task,&#10;outperforming the previous state of the art by 27% (relative) in MRR@10. The&#10;code to reproduce our results is available at&#10;https://github.com/nyu-dl/dl4marco-bert&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.16833" label="2307.16833">
        <attvalues>
          <attvalue for="0" value="Data Augmentation for Neural Machine Translation using Generative&#10;  Language Model" />
          <attvalue for="1" value="  Despite the rapid growth in model architecture, the scarcity of large&#10;parallel corpora remains the main bottleneck in Neural Machine Translation.&#10;Data augmentation is a technique that enhances the performance of data-hungry&#10;models by generating synthetic data instead of collecting new ones. We explore&#10;prompt-based data augmentation approaches that leverage large-scale language&#10;models such as ChatGPT. To create a synthetic parallel corpus, we compare 3&#10;methods using different prompts. We employ two assessment metrics to measure&#10;the diversity of the generated synthetic data. This approach requires no&#10;further model training cost, which is mandatory in other augmentation methods&#10;like back-translation. The proposed method improves the unaugmented baseline by&#10;0.68 BLEU score.&#10;" />
          <attvalue for="2" value="&#10;Neural Machine Translation(NMT) is the task of converting a sentence written in a source language into a target language sentence by using a translation model.&#10;NMT models usually require vast amounts of parallel data for training, but high-quality parallel data is often scarce. &#10;Since generating parallel synthetic data demands substantial time and cost, especially for low-resource languages or domains, the problem becomes particularly severe in such cases.&#10;&#10;To address the data scarcity problem, back-translation-based methods~\cite{sennrich-etal-2016-improving, edunov-etal-2018-understanding, hoang-etal-2018-iterative, sugiyama-yoshinaga-2019-data, kumar-etal-2020-data} have been widely adopted.&#10;Back-translation leverages a backward translation model and monolingual target corpus to generate synthetic pairs, which naturally consider the source-target alignments.&#10;However, the data quality generated by back-translation can significantly vary depending on the performance of the backward translation model.&#10;When the domain of training data and the domain of data to be generated are different, obtaining high-quality synthetic data is even more challenging.&#10;In this case, out-of-domain issues such as hallucinations~\cite{wang-sennrich-2020-exposure, muller-etal-2020-domain}, are more likely to occur, leading to difficulties in acquiring high-quality synthetic data.&#10;Recently, with the remarkable advancements in Natural Language Generation models~\cite{NEURIPS2020_1457c0d6}, research on utilizing large-scale language generation models for data augmentation~\cite{yoo-etal-2021-gpt3mix-leveraging} has been conducted.&#10;During the inference phase, the model receives a prompt that defines the problem, and it generates the corresponding output data. &#10;The quality of the generated data can vary depending on the provided prompt. &#10;Therefore, to obtain high-quality data, it is crucial to carefully select a prompt that is well-suited for the task.&#10;&#10;In this paper, we conduct prompt-based data augmentation experiments by leveraging ChatGPT.&#10;Through experiments, we examine that appropriate prompts can reduce the generation cost of the synthetic data and facilitate the easy transfer of knowledge from large-scale language models.&#10;We also validate the effectiveness of the proposed 3 prompts through measure the diversity of generated synthetic data by each method.&#10;Via comparing the diversity, we demonstrate that generating various data is a crucial factor in synthetic data augmentation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Data Augmentation Techniques, Linguistics, Data Science, Language Model Applications, Artificial Intelligence, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2005.03642" label="2005.03642">
        <attvalues>
          <attvalue for="0" value="On Exposure Bias, Hallucination and Domain Shift in Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  The standard training algorithm in neural machine translation (NMT) suffers&#10;from exposure bias, and alternative algorithms have been proposed to mitigate&#10;this. However, the practical impact of exposure bias is under debate. In this&#10;paper, we link exposure bias to another well-known problem in NMT, namely the&#10;tendency to generate hallucinations under domain shift. In experiments on three&#10;datasets with multiple test domains, we show that exposure bias is partially to&#10;blame for hallucinations, and that training with Minimum Risk Training, which&#10;avoids exposure bias, can mitigate this. Our analysis explains why exposure&#10;bias is more problematic under domain shift, and also links exposure bias to&#10;the beam search problem, i.e. performance deterioration with increasing beam&#10;size. Our results provide a new justification for methods that reduce exposure&#10;bias: even if they do not increase performance on in-domain test sets, they can&#10;increase model robustness to domain shift.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.03731" label="2311.03731">
        <attvalues>
          <attvalue for="0" value="A Survey of Large Language Models Attribution" />
          <attvalue for="1" value="  Open-domain generative systems have gained significant attention in the field&#10;of conversational AI (e.g., generative search engines). This paper presents a&#10;comprehensive review of the attribution mechanisms employed by these systems,&#10;particularly large language models. Though attribution or citation improve the&#10;factuality and verifiability, issues like ambiguous knowledge reservoirs,&#10;inherent biases, and the drawbacks of excessive attribution can hinder the&#10;effectiveness of these systems. The aim of this survey is to provide valuable&#10;insights for researchers, aiding in the refinement of attribution methodologies&#10;to enhance the reliability and veracity of responses generated by open-domain&#10;generative systems. We believe that this field is still in its early stages;&#10;hence, we maintain a repository to keep track of ongoing studies at&#10;https://github.com/HITsz-TMG/awesome-llm-attributions.&#10;" />
          <attvalue for="2" value="&#10;Since the advent of open-domain generative systems driven by large language models (LLMs)~\cite{Anil2023PaLM2T,chatgpt,OpenAI2023GPT4TR}, addressing the coherent generation of potentially inaccurate or fabricated content has been a persistent challenge~\cite{Rawte2023ASO,Ye2023CognitiveMA,Zhang2023SirensSI}. Such issues are often referred to by the community as ``hallucination'' problems, wherein the generated content presents distorted or invented facts which lacks credible sourcing~\cite{DBLP:conf/acl/PeskoffS23}. This becomes particularly obvious in information-seeking and knowledge question-answering scenarios, where users rely on large language models for expert knowledge~\cite{Malaviya2023ExpertQAEQ}. &#10;&#10;\tikzstyle{my-box}=[&#10; rectangle,&#10; draw=light-blue,&#10; rounded corners,&#10; text opacity=1,&#10; minimum height=1.5em,&#10; minimum width=5em,&#10; inner sep=2pt,&#10; align=center,&#10; fill opacity=.5,&#10; line width=0.8pt,&#10;]&#10;\tikzstyle{leaf}=[my-box, minimum height=1.5em,&#10; fill=hidden-pink!80, text=black, align=left,font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;&#10;The essence of the hallucination problem may stem from the fact that pre-trained models are sourced from vast, unfiltered real-world texts~\cite{Penedo2023TheRD}. These human-generated texts inherently contain inconsistencies and falsehoods. The objective of pre-training is merely to predict the next word, without explicitly modeling the veracity of the generated content. Even after utilizing reinforcement learning from human feedback~\cite{Ouyang2022TrainingLM}, models can still exhibit external hallucinations~\cite{Bai2022TrainingAH}. To address the issue of external hallucinations, researchers have begun to employ measures like external references to enhance the authenticity and reliability of chatbots~\cite{Thoppilan2022LaMDALM,Menick2022TeachingLM,nakano2021webgpt}. The distinction between explicit attribution and reinforcement learning lies not only in the need for human verification and compliance but also in recognizing that generated content might become outdated or invalid over time. As shown in Figure~\ref{figmotivation}, attribution can leverage real-time information to ensure relevance and accuracy. However, the fundamental challenge of attribution revolves around two essential requirements~\cite{Liu2023EvaluatingVI}:&#10;&#10;\begin{enumerate}&#10; \item Comprehensive attribution or citation (high recall): Every claim and statement (except debatable or subjective text, e.g., abstained text) made by the model-generated content should be fully backed by appropriate references.&#10;&#10;\item Sufficiency attribution or citation (high precision): Every reference should directly support its associated claim or statement.&#10;\end{enumerate} &#10;With these requirements in mind, we can break down the main ways models handle attribution into three types (see examples in Figure~\ref{figexample}):&#10;&#10;\begin{enumerate}&#10; \item Direct model-driven attribution: The large model itself provides the attribution for its answer. However, this type often poses a challenge as not only might the answers be hallucinated, but the attributions themselves can also be~\cite{DBLP:journals/corr/abs-2305-18248}. While ChatGPT provides correct or partially correct answers about 50.6\% of the time, the suggested references were only present 14\% of the time~\cite{zuccon2023chatgpt}.&#10;&#10;\item Post-retrieval answering: This approach is rooted in the idea of explicitly retrieving information and then letting the model answer based on this retrieved data. But retrieval does not inherently equate to attribution~\cite{gao2023enabling}. Issues arise when the boundaries between internal knowledge of the model and externally retrieved information become blurred, leading to potential knowledge conflicts~\cite{Xie2023AdaptiveCO}. Retrieval can also be used as a specialized tool allowing the model to trigger it independently, similar to the Browse with Bing in ChatGPT.&#10;&#10;\item Post-generation attribution: The system first provides an answer, then conducts a search using both the question and answer for attribution. The answer is then modified if necessary and appropriately attributed. Modern search engines like Bing Chat~ have already incorporated such attribution. However, studies have shown that only 51.5\% of the content generated from four generative search engines was entirely supported by their cited references~\cite{Liu2023EvaluatingVI}. This form of attribution is particularly lacking in high-risk professional fields such as medicine and law, with research revealing a significant number of incomplete attributions (35\% and 31\%, respectively); moreover, many attributions were derived from unreliable sources, with 51\% of them being assessed as unreliable by experts~\cite{Malaviya2023ExpertQAEQ}.&#10;\end{enumerate}&#10;&#10;Moving beyond general discussions on text hallucinations~\cite{Zhang2023SirensSI,Ye2023CognitiveMA,Rawte2023ASO}, our study delves deeper into the attribution of large language models. As shown in Figure~\ref{taxo_of_icl}, we explore its origins, the technology underpinning it, and the criteria for its assessment. Additionally, we touch upon challenges such as biases and the potential for excessive citations. We believe that by focusing on these attribution issues, we can make models more trustworthy and easier to understand. Our goal with this study is to shed light on attribution in a way that's clearer and encourages deeper thought on the topic.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Attribution Mechanisms, Linguistics, Language Models, Conversational AI, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2309.05922" label="2309.05922">
        <attvalues>
          <attvalue for="0" value="A Survey of Hallucination in Large Foundation Models" />
          <attvalue for="1" value="  Hallucination in a foundation model (FM) refers to the generation of content&#10;that strays from factual reality or includes fabricated information. This&#10;survey paper provides an extensive overview of recent efforts that aim to&#10;identify, elucidate, and tackle the problem of hallucination, with a particular&#10;focus on ``Large'' Foundation Models (LFMs). The paper classifies various types&#10;of hallucination phenomena that are specific to LFMs and establishes evaluation&#10;criteria for assessing the extent of hallucination. It also examines existing&#10;strategies for mitigating hallucination in LFMs and discusses potential&#10;directions for future research in this area. Essentially, the paper offers a&#10;comprehensive examination of the challenges and solutions related to&#10;hallucination in LFMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.06794" label="2309.06794">
        <attvalues>
          <attvalue for="0" value="Cognitive Mirage: A Review of Hallucinations in Large Language Models" />
          <attvalue for="1" value="  As large language models continue to develop in the field of AI, text&#10;generation systems are susceptible to a worrisome phenomenon known as&#10;hallucination. In this study, we summarize recent compelling insights into&#10;hallucinations in LLMs. We present a novel taxonomy of hallucinations from&#10;various text generation tasks, thus provide theoretical insights, detection&#10;methods and improvement approaches. Based on this, future research directions&#10;are proposed. Our contribution are threefold: (1) We provide a detailed and&#10;complete taxonomy for hallucinations appearing in text generation tasks; (2) We&#10;provide theoretical analyses of hallucinations in LLMs and provide existing&#10;detection and improvement methods; (3) We propose several research directions&#10;that can be developed in the future. As hallucinations garner significant&#10;attention from the community, we will maintain updates on relevant research&#10;progress.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.07852" label="2309.07852">
        <attvalues>
          <attvalue for="0" value="ExpertQA: Expert-Curated Questions and Attributed Answers" />
          <attvalue for="1" value="  As language models are adopted by a more sophisticated and diverse set of&#10;users, the importance of guaranteeing that they provide factually correct&#10;information supported by verifiable sources is critical across fields of study.&#10;This is especially the case for high-stakes fields, such as medicine and law,&#10;where the risk of propagating false information is high and can lead to&#10;undesirable societal consequences. Previous work studying attribution and&#10;factuality has not focused on analyzing these characteristics of language model&#10;outputs in domain-specific scenarios. In this work, we conduct human evaluation&#10;of responses from a few representative systems along various axes of&#10;attribution and factuality, by bringing domain experts in the loop.&#10;Specifically, we collect expert-curated questions from 484 participants across&#10;32 fields of study, and then ask the same experts to evaluate generated&#10;responses to their own questions. In addition, we ask experts to improve upon&#10;responses from language models. The output of our analysis is ExpertQA, a&#10;high-quality long-form QA dataset with 2177 questions spanning 32 fields, along&#10;with verified answers and attributions for claims in the answers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.09401" label="2309.09401">
        <attvalues>
          <attvalue for="0" value="ChatGPT Hallucinates when Attributing Answers" />
          <attvalue for="1" value="  Can ChatGPT provide evidence to support its answers? Does the evidence it&#10;suggests actually exist and does it really support its answer? We investigate&#10;these questions using a collection of domain-specific knowledge-based&#10;questions, specifically prompting ChatGPT to provide both an answer and&#10;supporting evidence in the form of references to external sources. We also&#10;investigate how different prompts impact answers and evidence. We find that&#10;ChatGPT provides correct or partially correct answers in about half of the&#10;cases (50.6% of the times), but its suggested references only exist 14% of the&#10;times. We further provide insights on the generated references that reveal&#10;common traits among the references that ChatGPT generates, and show how even if&#10;a reference provided by the model does exist, this reference often does not&#10;support the claims ChatGPT attributes to it. Our findings are important because&#10;(1) they are the first systematic analysis of the references created by ChatGPT&#10;in its answers; (2) they suggest that the model may leverage good quality&#10;information in producing correct answers, but is unable to attribute real&#10;evidence to support its answers. Prompts, raw result files and manual analysis&#10;are made publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14627" label="2305.14627">
        <attvalues>
          <attvalue for="0" value="Enabling Large Language Models to Generate Text with Citations" />
          <attvalue for="1" value="  Large language models (LLMs) have emerged as a widely-used tool for&#10;information seeking, but their generated outputs are prone to hallucination. In&#10;this work, our aim is to allow LLMs to generate text with citations, improving&#10;their factual correctness and verifiability. Existing work mainly relies on&#10;commercial search engines and human evaluation, making it challenging to&#10;reproduce and compare different modeling approaches. We propose ALCE, the first&#10;benchmark for Automatic LLMs' Citation Evaluation. ALCE collects a diverse set&#10;of questions and retrieval corpora and requires building end-to-end systems to&#10;retrieve supporting evidence and generate answers with citations. We develop&#10;automatic metrics along three dimensions -- fluency, correctness, and citation&#10;quality -- and demonstrate their strong correlation with human judgements. Our&#10;experiments with state-of-the-art LLMs and novel prompting strategies show that&#10;current systems have considerable room for improvement -- For example, on the&#10;ELI5 dataset, even the best models lack complete citation support 50% of the&#10;time. Our analyses further highlight promising future directions, including&#10;developing better retrievers, advancing long-context LLMs, and improving the&#10;ability to synthesize information from multiple sources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.14462" label="2309.14462">
        <attvalues>
          <attvalue for="0" value="On the Impact of Quantization and Pruning of Self-Supervised Speech&#10;  Models for Downstream Speech Recognition Tasks &quot;In-the-Wild''" />
          <attvalue for="1" value="  Recent advances with self-supervised learning have allowed speech recognition&#10;systems to achieve state-of-the-art (SOTA) word error rates (WER) while&#10;requiring only a fraction of the labeled training data needed by its&#10;predecessors. Notwithstanding, while such models achieve SOTA performance in&#10;matched train/test conditions, their performance degrades substantially when&#10;tested in unseen conditions. To overcome this problem, strategies such as data&#10;augmentation and/or domain shift training have been explored. Available models,&#10;however, are still too large to be considered for edge speech applications on&#10;resource-constrained devices, thus model compression tools are needed. In this&#10;paper, we explore the effects that train/test mismatch conditions have on&#10;speech recognition accuracy based on compressed self-supervised speech models.&#10;In particular, we report on the effects that parameter quantization and model&#10;pruning have on speech recognition accuracy based on the so-called robust&#10;wav2vec 2.0 model under noisy, reverberant, and noise-plus-reverberation&#10;conditions.&#10;" />
          <attvalue for="2" value="&#10;Large deep learning models have recently achieved great success on speech recognition tasks \cite{speechT5, speech_XLS-R}. These models, however, use a considerable amount of computational resources, which can be unfeasible for many edge applications. &#10;Edge applications focus on bringing computing as close to the source of data as possible in order to reduce latency and bandwidth use. It can be particularly important for speech recognition applications, where private and/or sensitive speaker data may need to be sent over the network to be processed remotely on large data processing clusters hosting very large and complex models. Bringing such large models to the edge can be challenging, as some edge devices may be resource-constrained with limited storage and processing capacity. Moreover, edge applications are corrupted by several environmental factors, such as ambient noise and/or room reverberation, which are known to be detrimental to speech-based applications. As such, a more detailed study on the impact of model compression and inference efficiency for large speech recognition models is needed. We aim to fill this gap.&#10;&#10;More specifically, in this study our overarching goal is two-fold: (1) understand how well state-of-the-art (SOTA) speech recognition models behave under different model compression schemes, and (2) how well the compressed model accuracy remains under varying noise and reverberation conditions. We hope that the results from this study will shed light on the performance gaps that may exist before ``edge speech recognition'' is implemented in practice. Experiments with the latest (robust) wav2vec 2.0 model are conducted under two different compression schemes (quantization and model pruning) and five noisy conditions (SNR = 0, 5, 10, 15, and 20 dB) and two reverberations conditions (small room and medium room).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Model Compression, Signal Processing, Speech Recognition, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2107.04691" label="2107.04691">
        <attvalues>
          <attvalue for="0" value="An Initial Investigation of Non-Native Spoken Question-Answering" />
          <attvalue for="1" value="  Text-based machine comprehension (MC) systems have a wide-range of&#10;applications, and standard corpora exist for developing and evaluating&#10;approaches. There has been far less research on spoken question answering (SQA)&#10;systems. The SQA task considered in this paper is to extract the answer from a&#10;candidate$\text{'}$s spoken response to a question in a prompt-response style&#10;language assessment test. Applying these MC approaches to this SQA task rather&#10;than, for example, off-topic response detection provides far more detailed&#10;information that can be used for further downstream processing. One significant&#10;challenge is the lack of appropriately annotated speech corpora to train&#10;systems for this task. Hence, a transfer-learning style approach is adopted&#10;where a system trained on text-based MC is evaluated on an SQA task with&#10;non-native speakers. Mismatches must be considered between text documents and&#10;spoken responses; non-native spoken grammar and written grammar. In practical&#10;SQA, ASR systems are used, necessitating an investigation of the impact of ASR&#10;errors. We show that a simple text-based ELECTRA MC model trained on SQuAD2.0&#10;transfers well for SQA. It is found that there is an approximately linear&#10;relationship between ASR errors and the SQA assessment scores but grammar&#10;mismatches have minimal impact.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, extractive text-based machine comprehension (MC) has advanced significantly with many systems \cite{Zhang2020RetrospectiveRF, Lan2020ALBERT:, DBLP:conf/aaai/0001WZDZ020, yamada-etal-2020-luke, clark2020electra} outperforming human performance. The availability of standard corpora for reading comprehension has enabled these advances in text-based MC systems \cite{rajpurkar-etal-2016-squad, DBLP:conf/acl/RajpurkarJL18, herman-cnn, hill2015the, joshi-etal-2017-triviaqa}. In contrast, less work has been done for spoken question answering (SQA), where a machine gives an answer to a question from spoken content \cite{turmo-overview, Umbert2012FactoidQA, comas-sibyl}.&#10;&#10;In this paper a form of machine listening comprehension is considered in the context of spoken language assessment where a candidate gives a spoken response to a prompt. The SQA model is required to extract the answer from the candidate's spoken response to the textual prompt. Automated assessment needs to ensure a candidate's response is relevant to the prompt \cite{raina-etal-2020-complementary, Wang2019}. MC would allow these systems to provide finer details than simple off-topic detection. Identifying the answer regions in a candidate's response can be used as additional features in downstream assessment tasks such as feedback.&#10;&#10;The lack of appropriately annotated speech corpora in English poses a significant challenge to this SQA task. \cite{Tseng+2016} propose machine comprehension of spoken content using the TOEFL listening comprehension test as their corpus. Several deep models have been evaluated on this task: \cite{Tseng+2016} uses attention-based RNN, \cite{fang-hier} uses tree-structured RNN and \cite{chung-etal-2018-supervised} explores transfer learning from text-based question-answering. However, the TOEFL comprehension is a multiple-choice test that does not address the more challenging SQA task of extracting the answer region. \cite{Lee2018} propose Spoken SQuAD as an extraction-based SQA task. The Spoken SQuAD dataset takes the SQuAD MC dataset \cite{rajpurkar-etal-2016-squad} and passes the passages through a text-to-speech system to generate spoken equivalents. SpeechBERT \cite{Chuang2020} is an end-to-end SQA model trained on Spoken SQuAD. However, Spoken SQuAD is artificially constructed from text data, meaning it does not capture the elements of speech that distinguishes it from written text such as relaxed discourse, grammatical inconsistencies and disfluencies. \cite{odsqa} release a large scale SQA dataset with real speech but this dataset is in Chinese.&#10;&#10;To address the lack of data, a transfer learning \cite{survey-pan} style approach is adopted. A system trained on a text-based MC task is directly evaluated on the SQA task with non-native spoken responses transcribed manually and with automatic speech recognition (ASR) systems. There are several mismatches that occur between training and evaluation: a domain shift from comprehension of text documents to spoken responses; grammatical irregularities not present in native speech due to non-native speakers of English; word errors due to ASR transcriptions.&#10;In deployment of SQA models, ASR systems will inevitably be involved at the beginning of the pipeline. Therefore, it is important to explore the direct consequence of ASR errors on various automated assessment tasks.&#10;The impact of these ASR errors on several spoken language assessment tasks has previously been evaluated \cite{Lu2019, Knill2018}. \cite{Lee2018} demonstrates that ASR errors have catastrophic impact on machine comprehension. This paper extends the investigation of the sensitivity of SQA systems to ASR errors. The empirical relationship between ASR errors and SQA assessment scores is explored here. Simple data augmentation approaches are considered in order to bridge the differences between the text and speech domains.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Recognition Errors, Linguistics, Spoken Question Answering, Artificial Intelligence, Natural Language Processing, Transfer Learning Methods" />
        </attvalues>
      </node>
      <node id="2001.09694" label="2001.09694">
        <attvalues>
          <attvalue for="0" value="Retrospective Reader for Machine Reading Comprehension" />
          <attvalue for="1" value="  Machine reading comprehension (MRC) is an AI challenge that requires machine&#10;to determine the correct answers to questions based on a given passage. MRC&#10;systems must not only answer question when necessary but also distinguish when&#10;no answer is available according to the given passage and then tactfully&#10;abstain from answering. When unanswerable questions are involved in the MRC&#10;task, an essential verification module called verifier is especially required&#10;in addition to the encoder, though the latest practice on MRC modeling still&#10;most benefits from adopting well pre-trained language models as the encoder&#10;block by only focusing on the &quot;reading&quot;. This paper devotes itself to exploring&#10;better verifier design for the MRC task with unanswerable questions. Inspired&#10;by how humans solve reading comprehension questions, we proposed a&#10;retrospective reader (Retro-Reader) that integrates two stages of reading and&#10;verification strategies: 1) sketchy reading that briefly investigates the&#10;overall interactions of passage and question, and yield an initial judgment; 2)&#10;intensive reading that verifies the answer and gives the final prediction. The&#10;proposed reader is evaluated on two benchmark MRC challenge datasets SQuAD2.0&#10;and NewsQA, achieving new state-of-the-art results. Significance tests show&#10;that our model is significantly better than the strong ELECTRA and ALBERT&#10;baselines. A series of analysis is also conducted to interpret the&#10;effectiveness of the proposed reader.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1608.06378" label="1608.06378">
        <attvalues>
          <attvalue for="0" value="Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening&#10;  Comprehension Test by Machine" />
          <attvalue for="1" value="  Multimedia or spoken content presents more attractive information than plain&#10;text content, but it's more difficult to display on a screen and be selected by&#10;a user. As a result, accessing large collections of the former is much more&#10;difficult and time-consuming than the latter for humans. It's highly attractive&#10;to develop a machine which can automatically understand spoken content and&#10;summarize the key information for humans to browse over. In this endeavor, we&#10;propose a new task of machine comprehension of spoken content. We define the&#10;initial goal as the listening comprehension test of TOEFL, a challenging&#10;academic English examination for English learners whose native language is not&#10;English. We further propose an Attention-based Multi-hop Recurrent Neural&#10;Network (AMRNN) architecture for this task, achieving encouraging results in&#10;the initial tests. Initial results also have shown that word-level attention is&#10;probably more robust than sentence-level attention for this task with ASR&#10;errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1608.07775" label="1608.07775">
        <attvalues>
          <attvalue for="0" value="Hierarchical Attention Model for Improved Machine Comprehension of&#10;  Spoken Content" />
          <attvalue for="1" value="  Multimedia or spoken content presents more attractive information than plain&#10;text content, but the former is more difficult to display on a screen and be&#10;selected by a user. As a result, accessing large collections of the former is&#10;much more difficult and time-consuming than the latter for humans. It's&#10;therefore highly attractive to develop machines which can automatically&#10;understand spoken content and summarize the key information for humans to&#10;browse over. In this endeavor, a new task of machine comprehension of spoken&#10;content was proposed recently. The initial goal was defined as the listening&#10;comprehension test of TOEFL, a challenging academic English examination for&#10;English learners whose native languages are not English. An Attention-based&#10;Multi-hop Recurrent Neural Network (AMRNN) architecture was also proposed for&#10;this task, which considered only the sequential relationship within the speech&#10;utterances. In this paper, we propose a new Hierarchical Attention Model (HAM),&#10;which constructs multi-hopped attention mechanism over tree-structured rather&#10;than sequential representations for the utterances. Improved comprehension&#10;performance robust with respect to ASR errors were obtained.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.05345" label="1711.05345">
        <attvalues>
          <attvalue for="0" value="Supervised and Unsupervised Transfer Learning for Question Answering" />
          <attvalue for="1" value="  Although transfer learning has been shown to be successful for tasks like&#10;object and speech recognition, its applicability to question answering (QA) has&#10;yet to be well-studied. In this paper, we conduct extensive experiments to&#10;investigate the transferability of knowledge learned from a source QA dataset&#10;to a target dataset using two QA models. The performance of both models on a&#10;TOEFL listening comprehension test (Tseng et al., 2016) and MCTest (Richardson&#10;et al., 2013) is significantly improved via a simple transfer learning&#10;technique from MovieQA (Tapaswi et al., 2016). In particular, one of the models&#10;achieves the state-of-the-art on all target datasets; for the TOEFL listening&#10;comprehension test, it outperforms the previous best model by 7%. Finally, we&#10;show that transfer learning is helpful even in unsupervised scenarios when&#10;correct answers for target QA dataset examples are not available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.00320" label="1804.00320">
        <attvalues>
          <attvalue for="0" value="Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition&#10;  Errors on Listening Comprehension" />
          <attvalue for="1" value="  Reading comprehension has been widely studied. One of the most representative&#10;reading comprehension tasks is Stanford Question Answering Dataset (SQuAD), on&#10;which machine is already comparable with human. On the other hand, accessing&#10;large collections of multimedia or spoken content is much more difficult and&#10;time-consuming than plain text content for humans. It's therefore highly&#10;attractive to develop machines which can automatically understand spoken&#10;content. In this paper, we propose a new listening comprehension task - Spoken&#10;SQuAD. On the new task, we found that speech recognition errors have&#10;catastrophic impact on machine comprehension, and several approaches are&#10;proposed to mitigate the impact.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.02280" label="1808.02280">
        <attvalues>
          <attvalue for="0" value="ODSQA: Open-domain Spoken Question Answering Dataset" />
          <attvalue for="1" value="  Reading comprehension by machine has been widely studied, but machine&#10;comprehension of spoken content is still a less investigated problem. In this&#10;paper, we release Open-Domain Spoken Question Answering Dataset (ODSQA) with&#10;more than three thousand questions. To the best of our knowledge, this is the&#10;largest real SQA dataset. On this dataset, we found that ASR errors have&#10;catastrophic impact on SQA. To mitigate the effect of ASR errors, subword units&#10;are involved, which brings consistent improvements over all the models. We&#10;further found that data augmentation on text-based QA training examples can&#10;improve SQA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.12129" label="1910.12129">
        <attvalues>
          <attvalue for="0" value="ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain&#10;  Conversation" />
          <attvalue for="1" value="  The uptake of deep learning in natural language generation (NLG) led to the&#10;release of both small and relatively large parallel corpora for training neural&#10;models. The existing data-to-text datasets are, however, aimed at task-oriented&#10;dialogue systems, and often thus limited in diversity and versatility. They are&#10;typically crowdsourced, with much of the noise left in them. Moreover, current&#10;neural NLG models do not take full advantage of large training data, and due to&#10;their strong generalizing properties produce sentences that look template-like&#10;regardless. We therefore present a new corpus of 7K samples, which (1) is clean&#10;despite being crowdsourced, (2) has utterances of 9 generalizable and&#10;conversational dialogue act types, making it more suitable for open-domain&#10;dialogue systems, and (3) explores the domain of video games, which is new to&#10;dialogue systems despite having excellent potential for supporting rich&#10;conversations.&#10;" />
          <attvalue for="2" value="&#10;&#10;The recent adoption of deep learning methods in natural language generation (NLG) for dialogue systems resulted in an explosion of neural data-to-text generation models, which depend on large training data. These are typically trained on one of the few parallel corpora publicly available, in particular the E2E~\cite{novikova2017e2e} and the WebNLG~\cite{gardent2017creating} datasets. Crowdsourcing large NLG datasets tends to be a costly and time-consuming process, making it impractical outside of task-oriented dialogue systems. At the same time, current neural NLG models struggle to replicate the high language diversity of the training sentences present in these large datasets, and instead they learn to produce the same generic type of sentences as with considerably less training data~\cite{deriu2018syntactic,juraska2018characterizing,duvsek2019evaluating}.&#10;&#10;Motivated by the rising interest in open-domain dialogue systems and conversational agents, we present ViGGO -- a smaller but more comprehensive dataset in the video game domain, introducing several generalizable dialogue acts (DAs), making it more suitable for training versatile and more conversational NLG models.&#10;The dataset provides almost 7K pairs of structured meaning representations (MRs) and crowdsourced reference utterances about more than 100 video games. Table~\ref{tab:ex_video_game_dataset} lists three examples.&#10;&#10;Video games are a vast entertainment topic that can naturally be discussed in a casual conversation, similar to movies and music, yet in the dialogue systems community it does not enjoy popularity anywhere close to that of the latter two topics \cite{fazel2017learning,li2017end,moghe2018towards,shah2018building,khatri2018advancing}. Restaurants have served as the go-to topic in data-to-text NLG for decades, as they offer a sufficiently large set of various attributes and corresponding values to talk about. While they certainly can be a topic of a casual conversation, the existing restaurant datasets~\cite{stent2004trainable,gavsic2008training,mairesse2010phrase,howcroft2013enhancing,wen2015stochastic,nayak2017plan} are geared more toward a task-oriented dialogue where a system tries to narrow down a restaurant based on the user's preferences and ultimately give a recommendation. Our new video game dataset is designed to be more conversational, and to thus enable neural models to produce utterances more suitable for an open-domain dialogue system.&#10;&#10;Even the most recent addition to the publicly available restaurant datasets for data-to-text NLG, the E2E dataset~\cite{novikova2017e2e}, suffers from the lack of a conversational aspect. It has become popular, thanks to its unprecedented size and multiple reference utterances per MR, for training end-to-end neural models, yet it only provides a single DA type. In contrast with the E2E dataset, ViGGO presents utterances of 9 different DAs.&#10;&#10;Other domains have been represented by task-oriented datasets with multiple DA types, for example the Hotel, Laptop, and TV datasets~\cite{wen2015semantically,wen2016multi}. Nevertheless, the DAs in these datasets vary greatly in complexity, and their distribution is thus heavily skewed, typically with two or three similar DAs comprising almost the entire dataset. In our video game dataset, we omitted simple DAs, in particular those that do not require any slots, such as greetings or short prompts, and focused on a set of substantial DAs only.&#10;&#10;The main contribution of our work is thus a new parallel data-to-text NLG corpus that (1) is more conversational, rather than information seeking or question answering, and thus more suitable for an open-domain dialogue system, (2) represents a new, unexplored domain which, however, has excellent potential for application in conversational agents, and (3) has high-quality, manually cleaned human-produced utterances.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Data Science, Dialogue Systems, Artificial Intelligence, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1809.05288" label="1809.05288">
        <attvalues>
          <attvalue for="0" value="Characterizing Variation in Crowd-Sourced Data for Training Neural&#10;  Language Generators to Produce Stylistically Varied Outputs" />
          <attvalue for="1" value="  One of the biggest challenges of end-to-end language generation from meaning&#10;representations in dialogue systems is making the outputs more natural and&#10;varied. Here we take a large corpus of 50K crowd-sourced utterances in the&#10;restaurant domain and develop text analysis methods that systematically&#10;characterize types of sentences in the training data. We then automatically&#10;label the training data to allow us to conduct two kinds of experiments with a&#10;neural generator. First, we test the effect of training the system with&#10;different stylistic partitions and quantify the effect of smaller, but more&#10;stylistically controlled training data. Second, we propose a method of labeling&#10;the style variants during training, and show that we can modify the style of&#10;the generated utterances using our stylistic labels. We contrast and compare&#10;these methods that can be used with any existing large corpus, showing how they&#10;vary in terms of semantic quality and stylistic control.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.04034" label="1712.04034">
        <attvalues>
          <attvalue for="0" value="Learning Robust Dialog Policies in Noisy Environments" />
          <attvalue for="1" value="  Modern virtual personal assistants provide a convenient interface for&#10;completing daily tasks via voice commands. An important consideration for these&#10;assistants is the ability to recover from automatic speech recognition (ASR)&#10;and natural language understanding (NLU) errors. In this paper, we focus on&#10;learning robust dialog policies to recover from these errors. To this end, we&#10;develop a user simulator which interacts with the assistant through voice&#10;commands in realistic scenarios with noisy audio, and use it to learn dialog&#10;policies through deep reinforcement learning. We show that dialogs generated by&#10;our simulator are indistinguishable from human generated dialogs, as determined&#10;by human evaluators. Furthermore, preliminary experimental results show that&#10;the learned policies in noisy environments achieve the same execution success&#10;rate with fewer dialog turns compared to fixed rule-based policies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.08205" label="1809.08205">
        <attvalues>
          <attvalue for="0" value="Towards Exploiting Background Knowledge for Building Conversation&#10;  Systems" />
          <attvalue for="1" value="  Existing dialog datasets contain a sequence of utterances and responses&#10;without any explicit background knowledge associated with them. This has&#10;resulted in the development of models which treat conversation as a&#10;sequence-to-sequence generation task i.e, given a sequence of utterances&#10;generate the response sequence). This is not only an overly simplistic view of&#10;conversation but it is also emphatically different from the way humans converse&#10;by heavily relying on their background knowledge about the topic (as opposed to&#10;simply relying on the previous sequence of utterances). For example, it is&#10;common for humans to (involuntarily) produce utterances which are copied or&#10;suitably modified from background articles they have read about the topic. To&#10;facilitate the development of such natural conversation models which mimic the&#10;human process of conversing, we create a new dataset containing movie chats&#10;wherein each response is explicitly generated by copying and/or modifying&#10;sentences from unstructured background knowledge such as plots, comments and&#10;reviews about the movie. We establish baseline results on this dataset (90K&#10;utterances from 9K conversations) using three different models: (i) pure&#10;generation based models which ignore the background knowledge (ii) generation&#10;based models which learn to copy information from the background knowledge when&#10;required and (iii) span prediction based models which predict the appropriate&#10;response span in the background knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.10757" label="1812.10757">
        <attvalues>
          <attvalue for="0" value="Advancing the State of the Art in Open Domain Dialog Systems through the&#10;  Alexa Prize" />
          <attvalue for="1" value="  Building open domain conversational systems that allow users to have engaging&#10;conversations on topics of their choice is a challenging task. Alexa Prize was&#10;launched in 2016 to tackle the problem of achieving natural, sustained,&#10;coherent and engaging open-domain dialogs. In the second iteration of the&#10;competition in 2018, university teams advanced the state of the art by using&#10;context in dialog models, leveraging knowledge graphs for language&#10;understanding, handling complex utterances, building statistical and&#10;hierarchical dialog managers, and leveraging model-driven signals from user&#10;responses. The 2018 competition also included the provision of a suite of tools&#10;and models to the competitors including the CoBot (conversational bot) toolkit,&#10;topic and dialog act detection models, conversation evaluators, and a sensitive&#10;content detection model so that the competing teams could focus on building&#10;knowledge-rich, coherent and engaging multi-turn dialog systems. This paper&#10;outlines the advances developed by the university teams as well as the Alexa&#10;Prize team to achieve the common goal of advancing the science of&#10;Conversational AI. We address several key open-ended problems such as&#10;conversational speech recognition, open domain natural language understanding,&#10;commonsense reasoning, statistical dialog management, and dialog evaluation.&#10;These collaborative efforts have driven improved experiences by Alexa users to&#10;an average rating of 3.61, the median duration of 2 mins 18 seconds, and&#10;average turns to 14.6, increases of 14%, 92%, 54% respectively since the launch&#10;of the 2018 competition. For conversational speech recognition, we have&#10;improved our relative Word Error Rate by 55% and our relative Entity Error Rate&#10;by 34% since the launch of the Alexa Prize. Socialbots improved in quality&#10;significantly more rapidly in 2018, in part due to the release of the CoBot&#10;toolkit.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1508.01755" label="1508.01755">
        <attvalues>
          <attvalue for="0" value="Stochastic Language Generation in Dialogue using Recurrent Neural&#10;  Networks with Convolutional Sentence Reranking" />
          <attvalue for="1" value="  The natural language generation (NLG) component of a spoken dialogue system&#10;(SDS) usually needs a substantial amount of handcrafting or a well-labeled&#10;dataset to be trained on. These limitations add significantly to development&#10;costs and make cross-domain, multi-lingual dialogue systems intractable.&#10;Moreover, human languages are context-aware. The most natural response should&#10;be directly learned from data rather than depending on predefined syntaxes or&#10;rules. This paper presents a statistical language generator based on a joint&#10;recurrent and convolutional neural network structure which can be trained on&#10;dialogue act-utterance pairs without any semantic alignments or predefined&#10;grammar trees. Objective metrics suggest that this new model outperforms&#10;previous methods under the same experimental conditions. Results of an&#10;evaluation by human judges indicate that it produces not only high quality but&#10;linguistically varied utterances which are preferred compared to n-gram and&#10;rule-based systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1603.01232" label="1603.01232">
        <attvalues>
          <attvalue for="0" value="Multi-domain Neural Network Language Generation for Spoken Dialogue&#10;  Systems" />
          <attvalue for="1" value="  Moving from limited-domain natural language generation (NLG) to open domain&#10;is difficult because the number of semantic input combinations grows&#10;exponentially with the number of domains. Therefore, it is important to&#10;leverage existing resources and exploit similarities between domains to&#10;facilitate domain adaptation. In this paper, we propose a procedure to train&#10;multi-domain, Recurrent Neural Network-based (RNN) language generators via&#10;multiple adaptation steps. In this procedure, a model is first trained on&#10;counterfeited data synthesised from an out-of-domain dataset, and then fine&#10;tuned on a small set of in-domain utterances with a discriminative objective&#10;function. Corpus-based evaluation results show that the proposed procedure can&#10;achieve competitive performance in terms of BLEU score and slot error rate&#10;while significantly reducing the data needed to train generators in new, unseen&#10;domains. In subjective testing, human judges confirm that the procedure greatly&#10;improves generator performance when only a small amount of data is available in&#10;the domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.02979" label="1906.02979">
        <attvalues>
          <attvalue for="0" value="A Wind of Change: Detecting and Evaluating Lexical Semantic Change&#10;  across Times and Domains" />
          <attvalue for="1" value="  We perform an interdisciplinary large-scale evaluation for detecting lexical&#10;semantic divergences in a diachronic and in a synchronic task: semantic sense&#10;changes across time, and semantic sense changes across domains. Our work&#10;addresses the superficialness and lack of comparison in assessing models of&#10;diachronic lexical change, by bringing together and extending benchmark models&#10;on a common state-of-the-art evaluation task. In addition, we demonstrate that&#10;the same evaluation task and modelling approaches can successfully be utilised&#10;for the synchronic detection of domain-specific sense divergences in the field&#10;of term extraction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1109.6341" label="1109.6341">
        <attvalues>
          <attvalue for="0" value="Domain Adaptation for Statistical Classifiers" />
          <attvalue for="1" value="  The most basic assumption used in statistical learning theory is that&#10;training data and test data are drawn from the same underlying distribution.&#10;Unfortunately, in many applications, the &quot;in-domain&quot; test data is drawn from a&#10;distribution that is related, but not identical, to the &quot;out-of-domain&quot;&#10;distribution of the training data. We consider the common case in which labeled&#10;out-of-domain data is plentiful, but labeled in-domain data is scarce. We&#10;introduce a statistical formulation of this problem in terms of a simple&#10;mixture model and present an instantiation of this framework to maximum entropy&#10;classifiers and their linear chain counterparts. We present efficient inference&#10;algorithms for this special case based on the technique of conditional&#10;expectation maximization. Our experimental results show that our approach leads&#10;to improved performance on three real world tasks on four different data sets&#10;from the natural language processing domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.07408" label="2111.07408">
        <attvalues>
          <attvalue for="0" value="Time Waits for No One! Analysis and Challenges of Temporal Misalignment" />
          <attvalue for="1" value="  When an NLP model is trained on text data from one time period and tested or&#10;deployed on data from another, the resulting temporal misalignment can degrade&#10;end-task performance. In this work, we establish a suite of eight diverse tasks&#10;across different domains (social media, science papers, news, and reviews) and&#10;periods of time (spanning five years or more) to quantify the effects of&#10;temporal misalignment. Our study is focused on the ubiquitous setting where a&#10;pretrained model is optionally adapted through continued domain-specific&#10;pretraining, followed by task-specific finetuning. We establish a suite of&#10;tasks across multiple domains to study temporal misalignment in modern NLP&#10;systems. We find stronger effects of temporal misalignment on task performance&#10;than have been previously reported. We also find that, while temporal&#10;adaptation through continued pretraining can help, these gains are small&#10;compared to task-specific finetuning on data from the target time period. Our&#10;findings motivate continued research to improve temporal robustness of NLP&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.06418" label="2103.06418">
        <attvalues>
          <attvalue for="0" value="LightMBERT: A Simple Yet Effective Method for Multilingual BERT&#10;  Distillation" />
          <attvalue for="1" value="  The multilingual pre-trained language models (e.g, mBERT, XLM and XLM-R) have&#10;shown impressive performance on cross-lingual natural language understanding&#10;tasks. However, these models are computationally intensive and difficult to be&#10;deployed on resource-restricted devices. In this paper, we propose a simple yet&#10;effective distillation method (LightMBERT) for transferring the cross-lingual&#10;generalization ability of the multilingual BERT to a small student model. The&#10;experiment results empirically demonstrate the efficiency and effectiveness of&#10;LightMBERT, which is significantly better than the baselines and performs&#10;comparable to the teacher mBERT.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multilingual pre-trained language models~(PLMs), such as mBERT~\cite{devlin2019bert} and XLM~\cite{conneau2019cross}, have been shown to be effective on a variety of cross-lingual benchmarks~\cite{conneau2018xnli,artetxe2019cross,hu2020xtreme}. Moreover, XLM-R~\cite{conneau2019unsupervised} further demonstrates that it is possible to have a single large model for all languages, without sacrificing per-language performance. But, in practice, it is challenging to deploy these large multilingual PLMs on resource-limited devices because of the latency and memory requirements.&#10;&#10;Knowledge distillation~(KD)~\cite{hinton2015distilling}, one of the model compression techniques, has been successfully used for compressing monolingual PLMs~\cite{tang2019distilling,sun2019patient,sanh2019distilbert,tsai2019small,jiao2019tinybert,turc2019well,sunmobilebert,wang2020minilm}. It transfers the knowledge embedded in a large teacher PLM to a small student model by making the student model mimic the behaviors of the teacher. In this paper, we focus on the cross-lingual scenario where there is no task training data in the target languages, which is different from previous works~\cite{tsai2019small,mukherjee2020xtremedistil}. Specifically, we take mBERT as an example and investigate how to effectively and efficiently distill the cross-lingual generalization ability of it into a transformer~\cite{vaswani2017attention} based student.&#10;&#10;To achieve efficient multilingual distillation, we first initialize the student with the bottom layers of mBERT, which allows the student model to directly inherit some knowledge of mBERT at the beginning. Then, we freeze the inherited embeddings during the distillation process since they are shown to be important for the cross-lingual generalization ability of mBERT~\cite{pires2019multilingual,wu2019beto}. Last, we perform the {top transformer-layer distillation} on unsupervised corpus in multiple languages.&#10;&#10;Experiments on the cross-lingual natural language inference benchmark XNLI~\cite{conneau2018xnli} demonstrate that our distillation method significantly outperforms the baselines and achieves comparable results to the teacher mBERT.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Efficient Deployment, Information Technology, Linguistics, Language Model Distillation, Mathematics, Cross-Lingual Understanding" />
        </attvalues>
      </node>
      <node id="2306.14910" label="2306.14910">
        <attvalues>
          <attvalue for="0" value="The Importance of Human-Labeled Data in the Era of LLMs" />
          <attvalue for="1" value="  The advent of large language models (LLMs) has brought about a revolution in&#10;the development of tailored machine learning models and sparked debates on&#10;redefining data requirements. The automation facilitated by the training and&#10;implementation of LLMs has led to discussions and aspirations that human-level&#10;labeling interventions may no longer hold the same level of importance as in&#10;the era of supervised learning. This paper presents compelling arguments&#10;supporting the ongoing relevance of human-labeled data in the era of LLMs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Human-labeled data played a crucial role in the earlier era of AI, known as &quot;AI 1.0,&quot; where machine learning models heavily relied on such data \cite{deng2009imagenet}. The celebrated supervised learning framework \cite{vapnik1999overview,lecun2015deep} was designed and developed exactly for this paradigm. However, with the emergence of the new era of ``GPT&quot; models, the pretraining of large language models (LLM) primarily involves unstructured and unsupervised Internet data. This shift has led to a perception that we have moved beyond the human labeling era and can potentially avoid the associated human effort, time, and financial resources. This development is both exciting and aligns with the longstanding goal of the weakly-, semi-, and self-supervised learning community \cite{zhu2005semi,zhou2018brief,gui2023survey,balestriero2023cookbook}. &#10;&#10;Now, there is even greater hope as evidence indicates that large language models (LLMs) can be utilized for labeling tasks. Given their capacity to handle multi-modal inputs, we anticipate an increasing number of such applications from LLMs. Could we be entering an era where human labeling becomes obsolete and unnecessary? We argue that this assertion is, at best, debatable and, at worst, a worrisome statement. Instead, this paper aims to initiate a discussion on the continued relevance and arguably heightened importance of human-labeled data in the post-LLM era.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Machine Learning Development, Artificial Intelligence, Human Labeled Data" />
        </attvalues>
      </node>
      <node id="2304.12210" label="2304.12210">
        <attvalues>
          <attvalue for="0" value="A Cookbook of Self-Supervised Learning" />
          <attvalue for="1" value="  Self-supervised learning, dubbed the dark matter of intelligence, is a&#10;promising path to advance machine learning. Yet, much like cooking, training&#10;SSL methods is a delicate art with a high barrier to entry. While many&#10;components are familiar, successfully training a SSL method involves a dizzying&#10;set of choices from the pretext tasks to training hyper-parameters. Our goal is&#10;to lower the barrier to entry into SSL research by laying the foundations and&#10;latest SSL recipes in the style of a cookbook. We hope to empower the curious&#10;researcher to navigate the terrain of methods, understand the role of the&#10;various knobs, and gain the know-how required to explore how delicious SSL can&#10;be.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.10323" label="2312.10323">
        <attvalues>
          <attvalue for="0" value="Continuous Prompt Generation from Linear Combination of Discrete Prompt&#10;  Embeddings" />
          <attvalue for="1" value="  The wayward quality of continuous prompts stresses the importance of their&#10;interpretability as unexpected and unpredictable behaviors appear following&#10;training, especially in the context of large language models automating&#10;people-sensitive tasks such as resume screening. In this paper we present a&#10;novel method of constructing continuous prompts via discrete prompt embeddings&#10;and evaluate improvements to continuous prompt interpretability and inference&#10;accuracy. For a set of manually designed discrete prompts $\mathcal{D}$, which&#10;we tokenize and embed each into tensor form, we train a model to predict the&#10;weights such that the linear combinations of those prompts correspond to higher&#10;performance on natural language understanding tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;\cite{khashabi2021} formalizes the notion of waywardness by proposing that for some arbitrary task, $t$, and for an arbitrary&#10;discrete prompt, $p_D$, there exists an analogous, unintelligible continuous prompt, $p_C$, that corresponds to the former and will perform with comparable satisfaction on $t$. The waywardness of continuous prompts stresses the importance of resolving their interpretability considering the risk of unexpected and unpredictable behaviors appear in the automation of subjective or people-sensitive tasks. \\&#10;&#10;In our experiments we gathered a set $\mathcal{D}$ of discrete prompts, a few of which are provided in figure 1. We then tokenized the elements in $\mathcal{D}$ into tensor form, and then trained a feed forward neural network to predict weights such that the linear combination of the elements in $\mathcal{D}$ would result in improved model performance. \\&#10;&#10;Our results were encouraging in that our trained neural model was able to noticeably reduce the cross entropy loss provided by BART when predicting the target sequence of the natural language reasoning task. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Prompt Construction Methods, Natural Language Understanding, Artificial Intelligence, Mathematics, Language Model Interpretability" />
        </attvalues>
      </node>
      <node id="2205.06130" label="2205.06130">
        <attvalues>
          <attvalue for="0" value="Multi Task Learning For Zero Shot Performance Prediction of Multilingual&#10;  Models" />
          <attvalue for="1" value="  Massively Multilingual Transformer based Language Models have been observed&#10;to be surprisingly effective on zero-shot transfer across languages, though the&#10;performance varies from language to language depending on the pivot language(s)&#10;used for fine-tuning. In this work, we build upon some of the existing&#10;techniques for predicting the zero-shot performance on a task, by modeling it&#10;as a multi-task learning problem. We jointly train predictive models for&#10;different tasks which helps us build more accurate predictors for tasks where&#10;we have test data in very few languages to measure the actual performance of&#10;the model. Our approach also lends us the ability to perform a much more robust&#10;feature selection and identify a common set of features that influence&#10;zero-shot performance across a variety of tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multilingual models like mBERT \cite{devlin-etal-2019-bert} and XLM-R \cite{conneau-etal-2020-unsupervised} have been recently shown to be surprisingly effective for zero-shot transfer \cite{pires-etal-2019-multilingual} \cite{wu-dredze-2019-beto}, where on fine-tuning for a task on one or a few languages, called {pivots}, they can perform well on languages unseen during training. The zero-shot performance however, is often not uniform across the languages and the multilingual models turn out to be much less effective for low resource languages \cite{wu-dredze-2020-languages, lauscher-etal-2020-zero} and the languages that are typologically distant from the pivots \cite{lauscher-etal-2020-zero}. What affects the zero-shot transfer across different languages is a subject of considerable interest and importance~\cite{wang2019cross, pires-etal-2019-multilingual, wu-dredze-2019-beto, lauscher-etal-2020-zero}, however there is little conclusive evidence and a few papers even show contradictory findings.&#10;&#10;\cite{lauscher-etal-2020-zero} recently, showed that it is possible to predict the zero shot performance of mBERT and XLM-R on different languages by formulating it as a regression problem, with pretraining data size and typological similarities between the pivot and target languages as the input features, and the performance on downstream task as the prediction target. Along similar lines \cite{srinivasan2021predicting} and \cite{dolicki2021analysing} explore zero-shot performance prediction with a larger set of features and different regression techniques.&#10;&#10;However, the efficacy of these solutions are severely limited by the lack of training data, that is, the number of languages for which performance metrics are available for a given task. For instance, for most tasks in the popular XTREME-R \cite{ruder2021xtreme} benchmark, there are data points for 7-11 languages. This not only makes zero-shot performance prediction a challenging problem, but also a very important one because for practical deployment of such multilingual models, one would ideally like to know its performance for all the languages the model is supposed to handle. As \cite{srinivasan2021predicting} shows, accurate performance predictors can also help us build better and fairer multilingual models by suggesting data labeling strategies.&#10;&#10;In this work, we propose multi-task learning~\cite{zhang2017multitask} as an approach to mitigate training-data constraints and consequent over-fitting of the performance predictors to tasks and/or datasets. &#10;The contributions of our work are fourfold.&#10; First, we experiment with different multi-task learning approaches, such as Group Lasso \cite{yuan2006model}, Collective Matrix Factorization \cite{cortes2018cold}, Multi-Task Deep Gaussian Process Regression \cite{NIPS2007_66368270} and Meta Agnostic Meta Learning \cite{finn2017model} for 11 tasks. We observe an overall 10\% reduction in performance prediction errors compared to the best performing single-task models. The gains are even stronger when we just consider the tasks with very few data points ($\leq 10$), where we see a 20\% drop in the mean absolute errors.&#10; Second, an interesting consequence of modelling this problem via multi-task learning is that we are able to predict performance on low resource languages much more accurately, where in some cases single-task approaches may perform even worse than the simple averaging baselines.&#10; Third, apart from the features used for zero-shot performance prediction in the previous work \cite{lauscher-etal-2020-zero, srinivasan2021predicting, dolicki2021analysing}, we also utilize metrics quantifying the quality of multilingual tokenizers as proposed in \cite{rust-etal-2021-good} as features in our predictive models, which turn out to have strong predictive power for certain tasks. To the best of our knowledge, our work is the first to explore the impact of tokenizer quality specifically on zero-shot transfer. &#10; And fourth, our multi-task framework in general lends us with a much more robust selection of features affecting the zero-shot performance. This, in turn, lets us investigate the critical open question on what influences the zero-shot performances across languages more rigorously. As we shall see, our findings corroborate some of the previous conclusions, while others are extended or annulled. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Zero-Shot Transfer Learning, Predictive Modeling Techniques, Mathematics, Statistics" />
        </attvalues>
      </node>
      <node id="1809.00366" label="1809.00366">
        <attvalues>
          <attvalue for="0" value="Cold-start recommendations in Collective Matrix Factorization" />
          <attvalue for="1" value="  This work explores the ability of collective matrix factorization models in&#10;recommender systems to make predictions about users and items for which there&#10;is side information available but no feedback or interactions data, and&#10;proposes a new formulation with a faster cold-start prediction formula that can&#10;be used in real-time systems. While these cold-start recommendations are not as&#10;good as warm-start ones, they were found to be of better quality than&#10;non-personalized recommendations, and predictions about new users were found to&#10;be more reliable than those about new items. The formulation proposed here&#10;resulted in improved cold-start recommendations in many scenarios, at the&#10;expense of worse warm-start ones.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.07057" label="2307.07057">
        <attvalues>
          <attvalue for="0" value="Leveraging Pretrained ASR Encoders for Effective and Efficient&#10;  End-to-End Speech Intent Classification and Slot Filling" />
          <attvalue for="1" value="  We study speech intent classification and slot filling (SICSF) by proposing&#10;to use an encoder pretrained on speech recognition (ASR) to initialize an&#10;end-to-end (E2E) Conformer-Transformer model, which achieves the new&#10;state-of-the-art results on the SLURP dataset, with 90.14% intent accuracy and&#10;82.27% SLURP-F1. We compare our model with encoders pretrained on&#10;self-supervised learning (SSL), and show that ASR pretraining is much more&#10;effective than SSL for SICSF. To explore parameter efficiency, we freeze the&#10;encoder and add Adapter modules, and show that parameter efficiency is only&#10;achievable with an ASR-pretrained encoder, while the SSL encoder needs full&#10;finetuning to achieve comparable results. In addition, we provide an in-depth&#10;comparison on end-to-end models versus cascading models (ASR+NLU), and show&#10;that E2E models are better than cascaded models unless an oracle ASR model is&#10;provided. Last but not least, our model is the first E2E model that achieves&#10;the same performance as cascading models with oracle ASR. Code, checkpoints and&#10;configs are available.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken language understanding (SLU) is an essential component of conversational AI, which aims to extract semantic information directly from speech data. SLU has a very broad scope, including intent classification~\cite{sharma2021intent}, slot filling~\cite{slurp,wang2021fine,arora2022espnet,seo2022integration}, speech emotion recognition~\cite{chen2020large,shon2021leveraging}, question answering~\cite{lee2018odsqa, you2021knowledge}, etc. There are mainly two types of SLU models: (1) cascading (ASR+NLU) models that first perform automatic speech recognition (ASR) and then apply a natural language understanding (NLU) model to the transcribed text; (2) end-to-end (E2E) models that directly predict the semantic output without predicting transcriptions. Compared with its counterpart natural language understanding (NLU), SLU is more challenging. On one hand, errors will propagate from ASR to NLU in cascading SLU models. On the other hand, end-to-end SLU models cannot make use of the pretrained large language models such as BERT~\cite{devlin2018bert}. To tackle these challenges, we use end-to-end SLU models that do not have error propagation, and we also show that better performance can be achieved by utilizing ASR encoders pretrained on out-of-domain datasets.&#10;&#10;In this paper, we study the speech intent classification and slot filling (SICSF) task, which aims to detect user intents and extract the corresponding lexical fillers for detected entity slots at the same time, as illustrated in Figure~\ref{fig:example}. The most common end-to-end approach in this task is the encoder-decoder framework, where the encoder is responsible for extracting acoustic features from input audios, and the decoder is responsible for decoding the features to semantic output of intents and slots. Current works~\cite{wang2021fine,seo2022integration} use encoders pretrained by self-supervised learning (SSL)~\cite{baevski2020wav2vec,hsu2021hubert}. However, there is a large domain gap between the self-supervised learning task and the SICSF task, which limits the benefits that SLU models can obtain from the SSL-pretrained encoders. Some recent works~\cite{arora2022espnet,seo2022integration} also propose to train the SLU model with additional ASR task in a multi-task loss, which is more tricky to train since it's hard to choose a proper weight to balance different loss terms. Also, such multi-task approach wastes some network parameters in learning the ASR decoder which is not used during inference phase of the SLU task.&#10;&#10;We tackle the SICSF problem with an end-to-end approach, by using a Conformer-Transformer framework that casts the task as a sequence-to-sequence problem. Based on the intuition that the SICSF task can be treated as an audio-to-text problem, we propose to use an encoder pretrained by automatic speech recognition (ASR), which is also an audio-to-text task. The SSL objective, however, focuses on distinguishing one feature from the other features in the same sequence, which is very different from the SICSF task. We propose that, since the ASR objective is closer to the SICSF objective than the SSL objective, ASR-pretrained encoders are more beneficial to the SICSF task than SSL-pretrained encoders. Our main contributions are summarized as follows:&#10;\begin{itemize}&#10; \item Effectiveness: We present a Conformer-Transformer model with ASR-pretrained encoder that achieves new state-of-the-art performance on the SLURP dataset~\cite{slurp}, outperforming the other end-to-end (E2E) baselines by a large margin. This validates our hypothesis that ASR-pretrained encoders are more suitable for this task than SSL-pretrained encoders because of the task similarity between ASR and SICSF.&#10; \item Efficiency: We conduct extensive experiments on exploring parameter efficiency of E2E models, including freezing the encoder and using Adapters~\cite{houlsby2019parameter} in the encoder and finetuning on SLURP speech recognition data. Our results show that the best parameter efficiency is only achievable when using ASR-pretrained encoders, while models with SSL-pretrained encoders need finetuning all parameters to work well.&#10; \item E2E vs. Cascading: We also compare the proposed end-to-end (E2E) model with cascading models (ASR+NLU), and show that cascading model can only outperform E2E models when using an oracle ASR model. Also, our E2E model with ASR-pretrained encoder is able to match the performance of cascading model with oracle ASR, while all previous E2E models fall behind.&#10; \item Our code and model checkpoints are open-sourced for use in future research.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Intent Classification, End-to-End Modeling, Pretraining Methods, Computer Science, Linguistics, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2102.07370" label="2102.07370">
        <attvalues>
          <attvalue for="0" value="Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and&#10;  language Models for Intent Classification" />
          <attvalue for="1" value="  Intent classification is a task in spoken language understanding. An intent&#10;classification system is usually implemented as a pipeline process, with a&#10;speech recognition module followed by text processing that classifies the&#10;intents. There are also studies of end-to-end system that takes acoustic&#10;features as input and classifies the intents directly. Such systems don't take&#10;advantage of relevant linguistic information, and suffer from limited training&#10;data. In this work, we propose a novel intent classification framework that&#10;employs acoustic features extracted from a pretrained speech recognition system&#10;and linguistic features learned from a pretrained language model. We use&#10;knowledge distillation technique to map the acoustic embeddings towards&#10;linguistic embeddings. We perform fusion of both acoustic and linguistic&#10;embeddings through cross-attention approach to classify intents. With the&#10;proposed method, we achieve 90.86% and 99.07% accuracy on ATIS and Fluent&#10;speech corpus, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.11847" label="2408.11847">
        <attvalues>
          <attvalue for="0" value="Prompto: An open source library for asynchronous querying of LLM&#10;  endpoints" />
          <attvalue for="1" value="  Recent surge in Large Language Model (LLM) availability has opened exciting&#10;avenues for research. However, efficiently interacting with these models&#10;presents a significant hurdle since LLMs often reside on proprietary or&#10;self-hosted API endpoints, each requiring custom code for interaction.&#10;Conducting comparative studies between different models can therefore be&#10;time-consuming and necessitate significant engineering effort, hindering&#10;research efficiency and reproducibility. To address these challenges, we&#10;present prompto, an open source Python library which facilitates asynchronous&#10;querying of LLM endpoints enabling researchers to interact with multiple LLMs&#10;concurrently, while maximising efficiency and utilising individual rate limits.&#10;Our library empowers researchers and developers to interact with LLMs more&#10;effectively and enabling faster experimentation and evaluation. prompto is&#10;released with an introductory video (https://youtu.be/-eZAmlV4ypk) under MIT&#10;License and is available via GitHub&#10;(https://github.com/alan-turing-institute/prompto).&#10;" />
          <attvalue for="2" value=" &#10;&#10;The field of Natural Language Processing is going through a massive transition since the introduction of Transformer-based Large Language Models (LLMs) \cite{vaswani2017attention, devlin-etal-2019-bert, radford2019language} which have demonstrated exceptional generalisation capability on a wide range of language-related tasks.&#10;&#10;While user-friendly interfaces like ChatGPT, Gemini and Claude have made LLMs more accessible to the public, researchers nowadays increasingly interact with such models through programmatic interfaces and APIs. \cite{la2023language} noted several reproducibility issues with this Language-Models-as-a-Service (LMaaS) paradigm \cite{sun2022black}, where language models are centrally hosted and typically provided on a subscription or pay-per-use basis (e.g., the OpenAI API and Google's Gemini API). To address some of them, \cite{biderman2024lessons} suggested a series of best practices when evaluating LLMs, such as sharing your exact prompts, parameter inputs and code and always providing model outputs.&#10;&#10;In addition to LMaaS, there are now also several open LLMs, defined here as those with broadly available model weights as in \cite{kapoor2024societal}, e.g. Llama \cite{meta2024introducing}, Gemma \cite{team2024gemma}, Aya \cite{aryabumi2024aya}. These &quot;open&quot; models are often accompanied by a model card \cite{wolf-etal-2020-transformers} which provides instructions for executing them locally and for accessing their internals and weights. This allows users to create their own API endpoints for LLMs (e.g., via Ollama) on their available hardware.&#10;&#10;A significant challenge with both LMaaS and open LLMs is that conducting a comparative study across multiple models necessitates writing separate code to interact with each API and this obstacle further hinders already complex evaluation and reproducibility efforts. An additional problem is that APIs may have different constrains (e.g., query-per-minute (QPM) rate limits), adding even more complexity to the engineering design. &#10;&#10;To address these limitations and simplify large-scale comparative studies across LLMs, we introduce prompto, an open source Python library for asynchronous querying of LLM endpoints in a consistent and highly efficient manner. prompto uses asynchronous programming to efficiently interact with endpoints by allowing users to send multiple requests to different APIs concurrently. This eliminates idle wait times and maximises efficiency, especially when dealing with different rate limits. In contrast, in traditional synchronous programming, a user sends a single request to an endpoint and waits for a response from the API before sending another request, repeating for each query. prompto supports a range of LMaaS endpoints (e.g. OpenAI, Gemini, Anthropic) as well as self-hosted endpoints for querying local models (e.g. Ollama, Hugging Face's text-generation-inference for serving models hosted on Hugging Face). The codebase is easily extensible to integrate new APIs and/or locally self-hosted models. For instance, we provide an example using Quart in prompto to easily set up an endpoint for inferencing local models using transformers \cite{wolf-etal-2020-transformers}.&#10;&#10;Inspired by \cite{biderman2024lessons}, the library promotes experiment reproducibility by facilitating the definition of all inputs/prompts within a single JSON Lines (JSONL) or CSV file. The file can encompass queries for various APIs and models, enabling parallel processing for even greater efficiency gains. The library's scalability allows it to handle large-scale experiments. prompto also provides built-in functionalities for automatic evaluation of the obtained responses, such as allowing the user to apply scoring functions to model outputs, and model graded evaluation or LLM-as-a-judge \cite{zheng_judge_2023}.&#10;&#10;In this paper, we present an overview of prompto, highlighting its modular design and flexibility. We present prompto's functionalities, design choices, technical implementation and show its advantages in comparison with alternative approaches. We accompany its release with extensive documentation and a series of Jupyter Notebooks as tutorials, to allow the research community to easily explore all its functionalities. In addition to the library's provided examples, we provide an illustrative showcase of using our library and compare against a synchronous approach to query several LLM endpoints in parallel in Appendix \ref{sec:experiments}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Open Source Software, Linguistics, Research Efficiency, Artificial Intelligence, Information Science" />
        </attvalues>
      </node>
      <node id="2309.16573" label="2309.16573">
        <attvalues>
          <attvalue for="0" value="Language Models as a Service: Overview of a New Paradigm and its&#10;  Challenges" />
          <attvalue for="1" value="  Some of the most powerful language models currently are proprietary systems,&#10;accessible only via (typically restrictive) web or software programming&#10;interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In&#10;contrast with scenarios where full model access is available, as in the case of&#10;open-source models, such closed-off language models present specific challenges&#10;for evaluating, benchmarking, and testing them. This paper has two goals: on&#10;the one hand, we delineate how the aforementioned challenges act as impediments&#10;to the accessibility, replicability, reliability, and trustworthiness of LMaaS.&#10;We systematically examine the issues that arise from a lack of information&#10;about language models for each of these four aspects. We conduct a detailed&#10;analysis of existing solutions and put forth a number of considered&#10;recommendations, and highlight the directions for future advancements. On the&#10;other hand, it serves as a comprehensive resource for existing knowledge on&#10;current, major LMaaS, offering a synthesized overview of the licences and&#10;capabilities their interfaces offer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.08597" label="2409.08597">
        <attvalues>
          <attvalue for="0" value="LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented&#10;  Generation" />
          <attvalue for="1" value="  Recent advancements in integrating speech information into large language&#10;models (LLMs) have significantly improved automatic speech recognition (ASR)&#10;accuracy. However, existing methods often constrained by the capabilities of&#10;the speech encoders under varied acoustic conditions, such as accents. To&#10;address this, we propose LA-RAG, a novel Retrieval-Augmented Generation (RAG)&#10;paradigm for LLM-based ASR. LA-RAG leverages fine-grained token-level speech&#10;datastores and a speech-to-speech retrieval mechanism to enhance ASR accuracy&#10;via LLM in-context learning (ICL) capabilities. Experiments on Mandarin and&#10;various Chinese dialect datasets demonstrate significant improvements in ASR&#10;accuracy compared to existing methods, validating the effectiveness of our&#10;approach, especially in handling accent variations.&#10;" />
          <attvalue for="2" value="&#10;In recent years, there has been growing interest in integrating speech information into LLMs \cite{HyPoradise,speech_llama,whispering_llama}. These models have demonstrated remarkable efficacy in ICL capabilities to improve the ASR accuracy (LLM-based ASR). Initial studies typically input pure textual transcriptions into the LLM, often combining the ASR N-best results with instructions to prompt the LLM for error correction \cite{HyPoradise,Yang_2023,ma2023generativelargelanguagemodels}. In these studies, the LLM primarily serves as a text reranker or token selector. Concurrently, other studies have attempted to integrate pre-trained ASR models (most commonly using the speech encoder part) into LLMs with a modality adapter, such as Q-former, attention, or a projection to align the speech feature space with the textual space of the LLM \cite{speech_llama,yu2023connectingspeechencoderlarge,li2024usinglargelanguagemodel}. These approaches generally show improvements by leveraging rich acoustic signals. Further research has combined N-best results with speech encoders and even added denoising information \cite{whispering_llama,fathullah2023promptinglargelanguagemodels,chen2024itslatefusingacoustic,robust}. Such multi-source information integration usually leads to better performance. However, the performance ceiling of these methods is often limited by the capabilities of speech encoders. This is particularly evident when there is an acoustic feature mismatch between the training and test data of the speech encoder, such as in scenarios with accents where the encoder is insufficiently trained and the correct tokens do not appear in the N-best transcriptions. These methods struggle under such conditions. Usually, for traditional ASR models, domain adaptation or speaker adaptation can be used to address the issue of insufficient training \cite{Huang_Ye_Li_Gong_2021,10389732}. However, for LLM-based ASR, aside from the costly fine-tuning, this can be achieved through Retrieval-Augmented Generation (RAG) \cite{rag1,rag2}, allowing the LLM to learn external knowledge during inference.&#10;&#10;Compared to token-level or semantic-level matching in text-based RAG, the challenge of RAG in LLM-based ASR stems from how to accurately retrieve relevant speech examples and how to prompt LLMs from inherently high sampling rate acoustic data. \cite{speech_icl} explores and proposes a speech LLM capable of performing unseen classification tasks for the first time. COSMIC \cite{COSMIC} pioneered this capability in more complex ASR tasks, showing significant ASR accuracy gains in context-biased tasks. However, the above methods only use random sampling for example selection and lack exploration of how to retrieve more similar examples. \cite{speech_rag} first explored RAG in LLM-based ASR and created a retrieval datastore. \cite{yang24b_interspeech} proposed using RAG to enhance SLU task. However, they only focused on entity retrieval or only used coarse-grained speech retrieval, which makes accurate speech matching difficult.&#10;&#10;The construction of a fine-grained speech datastore for the LLM-based ASR task is hindered by a lack of precise speech-transcript alignment and the enormous volume of frame-level entries. Recently, in the speech retrieval augmentation task for small models, \cite{knn_ctc} and \cite{speaker_smoth} separately used Connectionist Temporal Classification (CTC) and Attention Encoder-Decoder (AED) pre-trained ASR models as speech tokenizers to force-align the speech features and text tokens. They established key-value pair mappings between speech features and text transcription tokens and retrieved the keys for each decoding step with a query extracted from hidden states, achieving effective performance. However, due to the large number of LLM parameters, the speed and storage consumption would be enormous if directly applied to LLM-based ASR.&#10;&#10;Therefore, we propose a new LLM-based ASR RAG (LA-RAG) paradigm utilizing the above speech tokenizers, fully leveraging the LLM's ICL capabilities. Specifically, in the database creation phase, speech tokenizers are used to obtain token-level precise alignment knowledge between speech hidden states and golden transcription tokens as key-value pairs, and the mapping between each key-value pair and its whole sequence is also stored as a speech inverted index. In the generation phase, the ASR transcription is used to perform the same speech tokenizing on the input speech, and each speech token obtained is used to query the index. By grouping and filtering policies, similar examples at the sequence level are obtained. In addition, to reduce the learning burden on the model, a pruning policy is added to remove tokens with low error probability. Finally, we input the speech and its golden transcription example pairs, together with the input speech tokens and N-best transcriptions, as prompts into the LLM. Here, we introduce a speech adapter to align speech and text spaces, and a model adapter to learn the mapping relationship of speech tokens to the correct text tokens. Experiments on Mandarin and various Chinese dialect datasets demonstrate significant improvements in ASR accuracy compared to existing methods, especially in handling accent variations.&#10;&#10;Our contributions are as follows:&#10;\begin{itemize}&#10; \item We propose a fine-grained retrieval method for speech-to-speech, implemented using a pre-trained ASR model through a simple forced alignment technique.&#10; \item We introduce a novel RAG paradigm for LLM-based ASR. By enabling the LLM to learn the mapping relationship between speech tokens and text tokens.&#10; \item We apply these methods to LLM-based ASR, leading to a significant enhancement in the accuracy of ASR results.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Artificial Intelligence, Signal Processing, Speech Recognition, Accent Variations" />
        </attvalues>
      </node>
      <node id="2310.06434" label="2310.06434">
        <attvalues>
          <attvalue for="0" value="Whispering LLaMA: A Cross-Modal Generative Error Correction Framework&#10;  for Speech Recognition" />
          <attvalue for="1" value="  We introduce a new cross-modal fusion technique designed for generative error&#10;correction in automatic speech recognition (ASR). Our methodology leverages&#10;both acoustic information and external linguistic representations to generate&#10;accurate speech transcription contexts. This marks a step towards a fresh&#10;paradigm in generative error correction within the realm of n-best hypotheses.&#10;Unlike the existing ranking-based rescoring methods, our approach adeptly uses&#10;distinct initialization techniques and parameter-efficient algorithms to boost&#10;ASR performance derived from pre-trained speech and text models. Through&#10;evaluation across diverse ASR datasets, we evaluate the stability and&#10;reproducibility of our fusion technique, demonstrating its improved word error&#10;rate relative (WERR) performance in comparison to n-best hypotheses by&#10;relatively 37.66%. To encourage future research, we have made our code and&#10;pre-trained models open source at&#10;https://github.com/Srijith-rkr/Whispering-LLaMA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.15649" label="2309.15649">
        <attvalues>
          <attvalue for="0" value="Generative Speech Recognition Error Correction with Large Language&#10;  Models and Task-Activating Prompting" />
          <attvalue for="1" value="  We explore the ability of large language models (LLMs) to act as speech&#10;recognition post-processors that perform rescoring and error correction. Our&#10;first focus is on instruction prompting to let LLMs perform these task without&#10;fine-tuning, for which we evaluate different prompting schemes, both zero- and&#10;few-shot in-context learning, and a novel task activation prompting method that&#10;combines causal instructions and demonstration to increase its context windows.&#10;Next, we show that rescoring only by in-context learning with frozen LLMs&#10;achieves results that are competitive with rescoring by domain-tuned LMs, using&#10;a pretrained first-pass recognition system and rescoring output on two&#10;out-of-domain tasks (ATIS and WSJ). By combining prompting techniques with&#10;fine-tuning we achieve error rates below the N-best oracle level, showcasing&#10;the generalization power of the LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.11382" label="2401.11382">
        <attvalues>
          <attvalue for="0" value="Using Large Language Model for End-to-End Chinese ASR and NER" />
          <attvalue for="1" value="  Mapping speech tokens to the same feature space as text tokens has become the&#10;paradigm for the integration of speech modality into decoder-only large&#10;language models (LLMs). An alternative approach is to use an encoder-decoder&#10;architecture that incorporates speech features through cross-attention. This&#10;approach, however, has received less attention in the literature. In this work,&#10;we connect the Whisper encoder with ChatGLM3 and provide in-depth comparisons&#10;of these two approaches using Chinese automatic speech recognition (ASR) and&#10;name entity recognition (NER) tasks. We evaluate them not only by conventional&#10;metrics like the F1 score but also by a novel fine-grained taxonomy of ASR-NER&#10;errors. Our experiments reveal that encoder-decoder architecture outperforms&#10;decoder-only architecture with a short context, while decoder-only architecture&#10;benefits from a long context as it fully exploits all layers of the LLM. By&#10;using LLM, we significantly reduced the entity omission errors and improved the&#10;entity ASR accuracy compared to the Conformer baseline. Additionally, we&#10;obtained a state-of-the-art (SOTA) F1 score of 0.805 on the AISHELL-NER test&#10;set by using chain-of-thought (CoT) NER which first infers long-form ASR&#10;transcriptions and then predicts NER labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.05457" label="2402.05457">
        <attvalues>
          <attvalue for="0" value="It's Never Too Late: Fusing Acoustic Information into Large Language&#10;  Models for Automatic Speech Recognition" />
          <attvalue for="1" value="  Recent studies have successfully shown that large language models (LLMs) can&#10;be successfully used for generative error correction (GER) on top of the&#10;automatic speech recognition (ASR) output. Specifically, an LLM is utilized to&#10;carry out a direct mapping from the N-best hypotheses list generated by an ASR&#10;system to the predicted output transcription. However, despite its&#10;effectiveness, GER introduces extra data uncertainty since the LLM is trained&#10;without taking into account acoustic information available in the speech&#10;signal. In this work, we aim to overcome such a limitation by infusing acoustic&#10;information before generating the predicted transcription through a novel late&#10;fusion solution termed Uncertainty-Aware Dynamic Fusion (UADF). UADF is a&#10;multimodal fusion approach implemented into an auto-regressive decoding process&#10;and works in two stages: (i) It first analyzes and calibrates the token-level&#10;LLM decision, and (ii) it then dynamically assimilates the information from the&#10;acoustic modality. Experimental evidence collected from various ASR tasks shows&#10;that UADF surpasses existing fusion mechanisms in several ways. It yields&#10;significant improvements in word error rate (WER) while mitigating data&#10;uncertainty issues in LLM and addressing the poor generalization relied with&#10;sole modality during fusion. We also demonstrate that UADF seamlessly adapts to&#10;audio-visual speech recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.10446" label="2401.10446">
        <attvalues>
          <attvalue for="0" value="Large Language Models are Efficient Learners of Noise-Robust Speech&#10;  Recognition" />
          <attvalue for="1" value="  Recent advances in large language models (LLMs) have promoted generative&#10;error correction (GER) for automatic speech recognition (ASR), which leverages&#10;the rich linguistic knowledge and powerful reasoning ability of LLMs to improve&#10;recognition results. The latest work proposes a GER benchmark with HyPoradise&#10;dataset to learn the mapping from ASR N-best hypotheses to ground-truth&#10;transcription by efficient LLM finetuning, which shows great effectiveness but&#10;lacks specificity on noise-robust ASR. In this work, we extend the benchmark to&#10;noisy conditions and investigate if we can teach LLMs to perform denoising for&#10;GER just like what robust ASR do}, where one solution is introducing noise&#10;information as a conditioner into LLM. However, directly incorporating noise&#10;embeddings from audio encoder could harm the LLM tuning due to cross-modality&#10;gap. To this end, we propose to extract a language-space noise embedding from&#10;the N-best list to represent the noise conditions of source speech, which can&#10;promote the denoising process in GER. Furthermore, in order to enhance its&#10;representation ability of audio noise, we design a knowledge distillation (KD)&#10;approach via mutual information estimation to distill the real noise&#10;information in audio embeddings to our language embedding. Experiments on&#10;various latest LLMs demonstrate our approach achieves a new breakthrough with&#10;up to 53.9% correction improvement in terms of word error rate while with&#10;limited training data. Analysis shows that our language-space noise embedding&#10;can well represent the noise conditions of source speech, under which&#10;off-the-shelf LLMs show strong ability of language-space denoising.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12477" label="2310.12477">
        <attvalues>
          <attvalue for="0" value="Exploring In-Context Learning of Textless Speech Language Model for&#10;  Speech Classification Tasks" />
          <attvalue for="1" value="  Ever since the development of GPT-3 in the natural language processing (NLP)&#10;field, in-context learning (ICL) has played an essential role in utilizing&#10;large language models (LLMs). By presenting the LM utterance-label&#10;demonstrations at the input, the LM can accomplish few-shot learning without&#10;relying on gradient descent or requiring explicit modification of its&#10;parameters. This enables the LM to perform various downstream tasks in a&#10;black-box manner. Despite the success of ICL in NLP, little work is exploring&#10;the possibility of ICL in speech processing. This study is the first work&#10;exploring ICL for speech classification tasks with textless speech LM. We first&#10;show that the current speech LM lacks the ICL capability. We then perform&#10;warmup training on the speech LM, equipping the LM with demonstration learning&#10;capability. This paper explores and proposes the first speech LM capable of&#10;performing unseen classification tasks in an ICL manner.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.02248" label="2311.02248">
        <attvalues>
          <attvalue for="0" value="COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning" />
          <attvalue for="1" value="  We present a cost-effective method to integrate speech into a large language&#10;model (LLM), resulting in a Contextual Speech Model with&#10;Instruction-following/in-context-learning Capabilities (COSMIC) multi-modal&#10;LLM. Using GPT-3.5, we generate Speech Comprehension Test Question-Answer (SQA)&#10;pairs from speech transcriptions for supervised instruction tuning. With under&#10;30 million trainable parameters and only 450 hours of English speech data,&#10;COSMIC demonstrates emerging capabilities in instruction-following and&#10;in-context learning. Equipped with such capabilities, COSMIC achieves a maximum&#10;33.18 BLEU score in 0-shot EN-to-X speech to text translation (S2TT) and a&#10;significant boost in the 1-shot setting. Additionally, there is an average&#10;25.8\% relative Word Error Rate (WER) reduction for 1-shot cross-domain&#10;adaptation. COSMIC exhibits a significant automatic speech recognition (ASR)&#10;accuracy gain in contextual biasing tasks due to its instruction-following&#10;capability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.13560" label="2312.13560">
        <attvalues>
          <attvalue for="0" value="kNN-CTC: Enhancing ASR via Retrieval of CTC Pseudo Labels" />
          <attvalue for="1" value="  The success of retrieval-augmented language models in various natural&#10;language processing (NLP) tasks has been constrained in automatic speech&#10;recognition (ASR) applications due to challenges in constructing fine-grained&#10;audio-text datastores. This paper presents kNN-CTC, a novel approach that&#10;overcomes these challenges by leveraging Connectionist Temporal Classification&#10;(CTC) pseudo labels to establish frame-level audio-text key-value pairs,&#10;circumventing the need for precise ground truth alignments. We further&#10;introduce a skip-blank strategy, which strategically ignores CTC blank frames,&#10;to reduce datastore size. kNN-CTC incorporates a k-nearest neighbors retrieval&#10;mechanism into pre-trained CTC ASR systems, achieving significant improvements&#10;in performance. By incorporating a k-nearest neighbors retrieval mechanism into&#10;pre-trained CTC ASR systems and leveraging a fine-grained, pruned datastore,&#10;kNN-CTC consistently achieves substantial improvements in performance under&#10;various experimental settings. Our code is available at&#10;https://github.com/NKU-HLT/KNN-CTC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.04791" label="2406.04791">
        <attvalues>
          <attvalue for="0" value="Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASR" />
          <attvalue for="1" value="  Despite recent improvements in End-to-End Automatic Speech Recognition (E2E&#10;ASR) systems, the performance can degrade due to vocal characteristic&#10;mismatches between training and testing data, particularly with limited target&#10;speaker adaptation data. We propose a novel speaker adaptation approach&#10;Speaker-Smoothed kNN that leverages k-Nearest Neighbors (kNN) retrieval&#10;techniques to improve model output by finding correctly pronounced tokens from&#10;its pre-built datastore during the decoding phase. Moreover, we utilize&#10;x-vector to dynamically adjust kNN interpolation parameters for data sparsity&#10;issue. This approach was validated using KeSpeech and MagicData corpora under&#10;in-domain and all-domain settings. Our method consistently performs comparably&#10;to fine-tuning without the associated performance degradation during speaker&#10;changes. Furthermore, in the all-domain setting, our method achieves&#10;state-of-the-art results, reducing the CER in both single speaker and&#10;multi-speaker test scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.03218" label="2202.03218">
        <attvalues>
          <attvalue for="0" value="Efficient Adapter Transfer of Self-Supervised Speech Models for&#10;  Automatic Speech Recognition" />
          <attvalue for="1" value="  Self-supervised learning (SSL) is a powerful tool that allows learning of&#10;underlying representations from unlabeled data. Transformer based models such&#10;as wav2vec 2.0 and HuBERT are leading the field in the speech domain. Generally&#10;these models are fine-tuned on a small amount of labeled data for a downstream&#10;task such as Automatic Speech Recognition (ASR). This involves re-training the&#10;majority of the model for each task. Adapters are small lightweight modules&#10;which are commonly used in Natural Language Processing (NLP) to adapt&#10;pre-trained models to new tasks. In this paper we propose applying adapters to&#10;wav2vec 2.0 to reduce the number of parameters required for downstream ASR&#10;tasks, and increase scalability of the model to multiple tasks or languages.&#10;Using adapters we can perform ASR while training fewer than 10% of parameters&#10;per task compared to full fine-tuning with little degradation of performance.&#10;Ablations show that applying adapters into just the top few layers of the&#10;pre-trained network gives similar performance to full transfer, supporting the&#10;theory that higher pre-trained layers encode more phonemic information, and&#10;further optimizing efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#09;" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Adaptation, Artificial Intelligence, Signal Processing, Self-Supervised Learning, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2407.06057" label="2407.06057">
        <attvalues>
          <attvalue for="0" value="Variational Best-of-N Alignment" />
          <attvalue for="1" value="  Best-of-N (BoN) is a popular and effective algorithm for aligning language&#10;models to human preferences. The algorithm works as follows: at inference time,&#10;N samples are drawn from the language model, and the sample with the highest&#10;reward, as judged by a reward model, is returned as the output. Despite its&#10;effectiveness, BoN is computationally expensive; it reduces sampling throughput&#10;by a factor of N. To make BoN more efficient at inference time, one strategy is&#10;to fine-tune the language model to mimic what BoN does during inference. To&#10;achieve this, we derive the distribution induced by the BoN algorithm. We then&#10;propose to fine-tune the language model to minimize backward KL divergence to&#10;the BoN distribution. Our approach is analogous to mean-field variational&#10;inference and, thus, we term it variational BoN (vBoN). To the extent this&#10;fine-tuning is successful and we end up with a good approximation, we have&#10;reduced the inference cost by a factor of N. Our experiments on a controlled&#10;generation task suggest that while variational BoN is not as effective as BoN&#10;in aligning language models, it is close to BoN performance as vBoN appears&#10;more often on the Pareto frontier of reward and KL divergence compared to&#10;models trained with KL-constrained RL objective.&#10;" />
          <attvalue for="2" value="&#10;Language models are pre-trained on large corpora to model a distribution over natural language text.&#10;Beyond their initial pre-training, they are often additionally fine-tuned on domain-specific data through a process called \defn{supervised fine-tuning (SFT)}.&#10;The goal of SFT is to enable the model to better perform various downstream tasks of interest. &#10;While the fine-tuned model, called the \defn{reference model} in our paper, is indeed typically much better at performing the downstream task of interest, e.g., dialogue generation or summarization, it may still generate undesirable content, e.g., harmful or offensive text.&#10;To mitigate this issue, \defn{aligning} the reference model to human preferences has become a fundamental step in the development of modern large language models \cite{llama2, gpt4, gemini}. &#10;&#10;The degree to which text is aligned with human preferences is typically operationalized using a real-valued reward function. &#10;Rather than constructing a reward function by hand, it is typically estimated from a dataset of human preferences.&#10;And, after estimation, we expect the reward function to return higher values for text that is more likely to be preferred by humans, and lower values for text that is more likely to be dispreferred.&#10;Then, given an estimated reward function, an alignment algorithm further alters the reference models in a manner such that it places the highest probability on that text that is high reward under the reward model and high probability under the reference model.\looseness=-1&#10;&#10;Alignment algorithms can be taxonomized into two groups: (i) alignment via fine-tuning, where we change the language model's parameters to achieve alignment \cite{rlhf, dpo}, and (ii) alignment through inference \cite{webgpt, mudgal2024controlled}. &#10;A common alignment-via-fine-tuning method is reinforcement learning from human feedback \cite[RLHF;][]{rlhf, summarizehf, instructgpt}.&#10;RLHF typically consists of further fine-tuning the language model under a \defn{KL-constrained RL objective}, which is made up of two terms: a term that encourages the model to maximize the reward, and a term that discourages high KL divergence between the language model and the reference model.&#10;This objective is often maximized with an RL algorithm, e.g., proximal policy optimization \cite[PPO;][]{ppo}.&#10;A common alignment through inference is the \beston \cite[Bo$N$;][]{webgpt, llama2} algorithm.&#10;As such, it does not require any fine-tuning of the language model.&#10;The algorithm is straightforward: One draws $N$ samples from the reference model and returns the text that achieves the highest reward among those $N$ samples. &#10;The \bon algorithm has also been effectively applied in controlled decoding \cite{fudge, mudgal2024controlled} and to generate a dataset for supervised fine-tuning \cite{llama2}.\looseness=-1&#10;&#10;Despite its simplicity, \bon has proven incredibly practical in generating high-reward text that still has high probability under the reference model. &#10;Theoretically, \cite{yang2024asymptotics} prove that under some simplifying assumptions, the \bon distribution is asymptotically equivalent to the optimal distribution under the KL-constrained RL objective. &#10;Empirically, it has been repeatedly shown \cite{pmlr-v202-gao23h, dpo, mudgal2024controlled} that \bon often appears on the frontier of reward and KL curves, surpassing the performance of models fine-tuned with RLHF.&#10;However, the main factor preventing \bon from replacing fine-tuning methods for alignment is its significant computational overhead during inference. &#10;Even when sampling is done in parallel, \bon decreases the text generation throughput by a factor of $N$. &#10;This drawback limits its practicality for generating text from large language models.\looseness=-1&#10;&#10;To speed up \bon, we devise a scheme to convert it into an alignment-via-fine-tuning algorithm rather than an alignment-via-inference algorithm. &#10;To this end, we first formally derive the probability distribution induced by the \bon algorithm. &#10;Then, we approximate this distribution by minimizing the reverse KL divergence between the language model and the \bon distribution. This leads to an optimization objective that we refer to as the \vbon objective. By analyzing a lower bound of this objective, we find that it behaves similarly to the KL-regularization objective in the limit, i.e., $N arrow 1$ or $N arrow \infty$. Importantly, the \vbon objective has a unique and useful property: it is insensitive to applying any monotonically increasing function to the reward values. This distinctive feature, along with the empirical success of the \bon algorithm, suggests that the \vbon objective is a promising and interesting objective to explore.&#10;Finally, we fine-tune the language model using PPO to optimize the \vbon objective. &#10;Our scheme allows us to achieve performance close to that of the \bon algorithm while increasing the inference throughput by a factor of $N$.\looseness=-1&#10;&#10;We experiment with our method in a controlled generation task, generating movie reviews with positive sentiment, and consider a binary sentiment classifier as the reward model.&#10;We compare our method, which we term as variational \bon (\vbon), against models fine-tuned with the KL-constrained RL objective.&#10;Our results suggest that models fine-tuned with the \vbon objective are most likely to generate movie reviews with positive sentiment. &#10;Additionally, these models deviate less from the reference model than those fine-tuned with KL-constrained RL objectives.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Algorithmic Efficiency Optimization, Linguistics, Variational Inference Methods, Artificial Intelligence, Mathematics, Statistics, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="2404.01730" label="2404.01730">
        <attvalues>
          <attvalue for="0" value="Asymptotics of Language Model Alignment" />
          <attvalue for="1" value="  Let $p$ denote a generative language model. Let $r$ denote a reward model&#10;that returns a scalar that captures the degree at which a draw from $p$ is&#10;preferred. The goal of language model alignment is to alter $p$ to a new&#10;distribution $\phi$ that results in a higher expected reward while keeping&#10;$\phi$ close to $p.$ A popular alignment method is the KL-constrained&#10;reinforcement learning (RL), which chooses a distribution $\phi_\Delta$ that&#10;maximizes $E_{\phi_{\Delta}} r(y)$ subject to a relative entropy constraint&#10;$KL(\phi_\Delta || p) \leq \Delta.$ Another simple alignment method is&#10;best-of-$N$, where $N$ samples are drawn from $p$ and one with highest reward&#10;is selected. In this paper, we offer a closed-form characterization of the&#10;optimal KL-constrained RL solution. We demonstrate that any alignment method&#10;that achieves a comparable trade-off between KL divergence and reward must&#10;approximate the optimal KL-constrained RL solution in terms of relative&#10;entropy. To further analyze the properties of alignment methods, we introduce&#10;two simplifying assumptions: we let the language model be memoryless, and the&#10;reward model be linear. Although these assumptions may not reflect complex&#10;real-world scenarios, they enable a precise characterization of the asymptotic&#10;behavior of both the best-of-$N$ alignment, and the KL-constrained RL method,&#10;in terms of information-theoretic quantities. We prove that the reward of the&#10;optimal KL-constrained RL solution satisfies a large deviation principle, and&#10;we fully characterize its rate function. We also show that the rate of growth&#10;of the scaled cumulants of the reward is characterized by a proper Renyi cross&#10;entropy. Finally, we show that best-of-$N$ is asymptotically equivalent to&#10;KL-constrained RL solution by proving that their expected rewards are&#10;asymptotically equal, and concluding that the two distributions must be close&#10;in KL divergence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.07434" label="2311.07434">
        <attvalues>
          <attvalue for="0" value="Understanding Users' Dissatisfaction with ChatGPT Responses: Types,&#10;  Resolving Tactics, and the Effect of Knowledge Level" />
          <attvalue for="1" value="  Large language models (LLMs) with chat-based capabilities, such as ChatGPT,&#10;are widely used in various workflows. However, due to a limited understanding&#10;of these large-scale models, users struggle to use this technology and&#10;experience different kinds of dissatisfaction. Researchers have introduced&#10;several methods, such as prompt engineering, to improve model responses.&#10;However, they focus on enhancing the model's performance in specific tasks, and&#10;little has been investigated on how to deal with the user dissatisfaction&#10;resulting from the model's responses. Therefore, with ChatGPT as the case&#10;study, we examine users' dissatisfaction along with their strategies to address&#10;the dissatisfaction. After organizing users' dissatisfaction with LLM into&#10;seven categories based on a literature review, we collected 511 instances of&#10;dissatisfactory ChatGPT responses from 107 users and their detailed&#10;recollections of dissatisfactory experiences, which we released as a publicly&#10;accessible dataset. Our analysis reveals that users most frequently experience&#10;dissatisfaction when ChatGPT fails to grasp their intentions, while they rate&#10;the severity of dissatisfaction related to accuracy the highest. We also&#10;identified four tactics users employ to address their dissatisfaction and their&#10;effectiveness. We found that users often do not use any tactics to address&#10;their dissatisfaction, and even when using tactics, 72% of dissatisfaction&#10;remained unresolved. Moreover, we found that users with low knowledge of LLMs&#10;tend to face more dissatisfaction on accuracy while they often put minimal&#10;effort in addressing dissatisfaction. Based on these findings, we propose&#10;design implications for minimizing user dissatisfaction and enhancing the&#10;usability of chat-based LLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;We review related work in (1) limitations and user challenges in LLMs and (2) user's strategies to overcome those challenges in Language Models. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Large Language Models, User Dissatisfaction Analysis, Linguistics, Chatbot Usability Enhancement, Psychology" />
        </attvalues>
      </node>
      <node id="2407.10645" label="2407.10645">
        <attvalues>
          <attvalue for="0" value="Prompt Selection Matters: Enhancing Text Annotations for Social Sciences&#10;  with Large Language Models" />
          <attvalue for="1" value="  Large Language Models have recently been applied to text annotation tasks&#10;from social sciences, equalling or surpassing the performance of human workers&#10;at a fraction of the cost. However, no inquiry has yet been made on the impact&#10;of prompt selection on labelling accuracy. In this study, we show that&#10;performance greatly varies between prompts, and we apply the method of&#10;automatic prompt optimization to systematically craft high quality prompts. We&#10;also provide the community with a simple, browser-based implementation of the&#10;method at https://prompt-ultra.github.io/ .&#10;" />
          <attvalue for="2" value="&#10;&#10;Throughout the social sciences, many research questions are answered through annotation and classification of large volumes of text, such as tweets or Facebook comments. Researchers may be interested in knowing, for instance, how politically slanted (liberal vs. conservative) a claim or headline is; how emotional or hostile its tone is, or which one of the basic emotions it reflects (sadness, joy, anger, etc) \cite{barrett2016handbook, howpartisanispress2012, Strapparava2010, rasmussen2024super}.&#10;Text annotation so far had to be performed either by human experts or by unskilled crowd workers, depending on the nature of the task. As a result, it was typically costly and time-consuming; crowd workers are also likely to mislabel the data.&#10;&#10;The recent progress of Large Language Models (LLMs) has opened new avenues and could revolutionize text mining by allowing huge volumes of text data to be analyzed in an unsupervised way in a matter of minutes \cite{heseltine2024large, gilardi2023chatgpt, tornberg2023chatgpt, Weber_Reichardt}. Early results suggest that LLMs can perform extremely well, extremely fast and at almost no financial cost, with levels of accuracy rivalling those of experts and superior to those of unskilled workers. &#10;Moreover, these results are achieved by off-the-shelf, general purpose models, that do not need any specialized training, unlike some pre-existing machine learning-based techniques.&#10;&#10;Nonetheless, crucial aspects of the automatic annotation of text data using LLMs have not been studied yet.&#10;In particular, earlier studies did not consider the importance of prompt choice: they used simple hand-crafted prompts, such as ``Does the following message express liberal or conservative views?'' or ``Is this tweet pro-life or pro-choice?'', to annotate their corpus.&#10;However, it has been observed outside the context of text annotation that applying distinct yet similar versions of a given prompt to certain tasks can result in large differences in accuracy, of the order of more than $10\%$ \cite{kojima2022large, few_shot_learners}.&#10;Going from $10\%$ to $20\%$ of mislabelled data can greatly impact the quality of a study's conclusions, especially if the classification errors are biased (e.g., if all mislabelled tweets express conservative views).&#10;&#10;Our main objectives in this paper are threefold. First, to help social scientists understand and adopt automatic text annotation using LLMs by providing a clear illustration of it. Second, to investigate the importance of prompt selection on performance, and to raise awareness about the issue of performance variability. Third, to explain to social scientists how to apply state-of-the-art prompt optimization methods used in the wider LLM community \cite{optimizer1, optimizer2} to their own classification tasks.&#10;&#10;More precisely, our contributions are as follows:&#10;\begin{itemize}&#10; \item We give a short, didactic overview of automatic text annotation for social sciences using LLMs; in particular, we draw attention to some shortcomings of the method that have not yet been discussed.&#10; \item We describe the principle of automatic prompt optimization, and provide a simple implementation of the method.&#10; \item We investigate and quantify the impact of prompt selection on accuracy across a range of standard classification tasks in social sciences using LLMs. To that end, we compare both hand-crafted prompts and automatically optimized prompts.&#10; \item We conclude that apparently similar prompts yield greatly varied accuracy levels. We also observe that automatic prompt optimization yields consistently good performance and beats prompt-crafting heuristics (such as using Chain of Thoughts prompts, see below) on most tasks.&#10; \item Finally, we provide the community with a simple and efficient way to label their datasets using LLMs coupled with either their own hand-crafted prompts or with a prompt optimization algorithm: our browser-based service.&#10;&#10;\end{itemize}&#10;All of the code used will be made available on GitHub.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Social Sciences, Prompt Optimization Methods, Linguistics, Text Annotation Tasks, Language Model Performance" />
        </attvalues>
      </node>
      <node id="2312.17617" label="2312.17617">
        <attvalues>
          <attvalue for="0" value="Large Language Models for Generative Information Extraction: A Survey" />
          <attvalue for="1" value="  Information extraction (IE) aims to extract structural knowledge (such as&#10;entities, relations, and events) from plain natural language texts. Recently,&#10;generative Large Language Models (LLMs) have demonstrated remarkable&#10;capabilities in text understanding and generation, allowing for generalization&#10;across various domains and tasks. As a result, numerous works have been&#10;proposed to harness abilities of LLMs and offer viable solutions for IE tasks&#10;based on a generative paradigm. To conduct a comprehensive systematic review&#10;and exploration of LLM efforts for IE tasks, in this study, we survey the most&#10;recent advancements in this field. We first present an extensive overview by&#10;categorizing these works in terms of various IE subtasks and learning&#10;paradigms, then we empirically analyze the most advanced methods and discover&#10;the emerging trend of IE tasks with LLMs. Based on thorough review conducted,&#10;we identify several insights in technique and promising research directions&#10;that deserve further exploration in future studies. We maintain a public&#10;repository and consistently update related resources at:&#10;\url{https://github.com/quqxui/Awesome-LLM4IE-Papers}.&#10;" />
          <attvalue for="2" value="&#10;Information Extraction (IE) is a crucial domain in natural language processing (NLP) that converts plain text into structured knowledge (e.g., entities, relations, and events), and serves as a foundational requirement for a wide range of downstream tasks, such as knowledge graph construction \cite{zhong2023comprehensive}, knowledge reasoning \cite{fu2019collaborative} and question answering \cite{srihari1999information}. Typical IE tasks consist of Named Entity Recognition (NER), Relation Extraction (RE) and Event Extraction (EE) \cite{uie,instructuie,code4uie,zhong2023contextualized}. &#10;However, performing IE tasks is inherently challenging. This is because IE involves extracting information from various sources and dealing with complex and ever-changing domain requirements \cite{zhou2022survey}. Unlike traditional NLP tasks, IE encompasses a broad spectrum of objectives such as entity extraction, relationship extraction, and more. In IE, the extraction targets exhibit intricate structures where entities are presented as span structures (string structures) and relationships are represented as triple structures \cite{uie}.&#10;Additionally, in order to effectively handle different information extraction tasks, it is necessary to employ multiple independent models. These models are trained separately for each specific task, without sharing any resources. However, this approach comes with a drawback: managing a large number of information extraction models becomes costly in terms of the resources needed for construction and training, like annotated corpora.&#10;&#10;The emergence of large language models (LLMs), such as GPT-4 \cite{gpt4}, has significantly advanced the field of NLP, due to their extraordinary capabilities in text understanding and generation \cite{qi2024unimel,peng2024large,zhang2024notellm2}.&#10;Pretraining LLMs using auto-regressive prediction allows them to capture the inherent patterns and semantic knowledge within text corpora \cite{liu2023pre,lyu2024crud,lyu2024retrieve,jia-etal-2024-mill,10.1145/3589335.3648321,fu2023unified,jia2024g3}. This enhances LLMs with the capability to perform zero-shot and few-shot learning, enabling them to model various tasks consistently and serving as tools for data augmentation \cite{zhang2024notellm, wang2024context, zhu2024fastmem}. Furthermore, LLMs can serve as intelligent agents for complex task planning and execution, utilizing memory retrieval and various tools to enhance efficiency and successfully accomplish tasks \cite{wang2024survey, guan2024enhancing, huang2024qdmr,fu2024video,li2023agent4ranking}.&#10;Therefore, there has been a recent surge of interest in generative IE methods \cite{qi2023preserving} that adopt LLMs to generate structural information rather than extracting structural information from plain text. These methods have been proven to be more practical in real-world scenarios compared to discriminated methods \cite{chen2023heproto,usm}, as they can handle schemas containing millions of entities without significant performance degradation \cite{genie}. &#10;&#10;On the one hand, LLMs have attracted significant attention from researchers in exploring their potentials for various scenarios and tasks of IE. &#10;In addition to excelling in individual IE tasks, LLMs possess a remarkable ability to effectively model all IE tasks in a universal format. &#10;This is conducted by capturing inter-task dependencies with instructive prompts, and achieves consistent performance \cite{uie,gollie,paolini2021structured,instructuie,giellm,fei2022lasuie,codeie}.&#10;On the other hand, recent works have shown the outstanding generalization of LLMs to not only learn from IE training data through fine-tuning \cite{paolini2021structured,yan2021unified,tempgen,rebel,paolini2021structured}, but also extract information in few-shot and even zero-shot scenarios relying solely on in-context examples or instructions \cite{chatie,code4struct,gpt-ner,promptner,xu2023unleash}. &#10;&#10;However, existing surveys \cite{nasar2021named,zhou2022survey,ye2022generative} do not provide a comprehensive exploration of these areas for the above two groups of research works: 1) universal frameworks that cater to multiple tasks and 2) cutting-edge learning techniques for scenarios with limited training data. The community urgently needs a more in-depth analysis of how LLM can be more appropriately applied to IE tasks to improve the performance of the IE field. This is because there are still challenges and issues in applying LLM to IE in terms of learning and understanding knowledge \cite{foppiano2024mining}. These challenges include the misalignment between natural language output and structured form \cite{code4uie}, hallucination problem in LLMs \cite{liu2024survey}, contextual dependence, high computational resource requirements \cite{sahoo2024systematic}, difficulties in updating internal knowledge \cite{xu2024editing}, etc.&#10;&#10;In this survey, we provide a comprehensive exploration of LLMs for generative IE, as illustrated in Fig. \ref{fig:intro}. To achieve this, we categorize existing methods mainly using two taxonomies: (1) a taxonomy of numerous IE subtasks, which aims to classify different types of information that can be extracted individually or uniformly, and (2) a taxonomy of IE techniques, which categorizes various novel approaches that utilize LLMs for generative IE, particularly on low-resource scenarios. In addition, we present a comprehensive review of studies that specifically focus on the application of IE techniques in various domains. And we discuss studies that aim to evaluate and analyze the performance of LLMs for IE.&#10;According to the above division, we construct a taxonomy of related studies as shown in Fig. \ref{fig_taxonomy}.&#10;We also compare several representative methods to gain deeper understanding of their potentials and limitations, and provide insightful analysis on future directions. &#10;To the best of our knowledge, this is the first survey on generative IE with LLMs. &#10;&#10;The remaining part of this survey is organized as follows: &#10;We first introduce the definition of generative IE and target of all subtasks in Section \ref{sec_prelimi}. Then, in Section \ref{sec_task}, we introduce representative models for each task and universal IE, and compare their performance.&#10;In Section \ref{sec:paradigm}, we summarize different learning techniques of LLMs for IE. Additionally, we introduce works proposed for special domains in Section \ref{sec:domain}, and present recent studies that evaluate and analyze the abilities of LLMs on IE tasks in Section \ref{sec:eval}. Finally, we propose potential research directions for future studies in Section \ref{sec:future}. In Section \ref{sec:Benchmarks}, we provide a comprehensive summary of the most commonly used LLMs and datasets statistics, as reference for researchers.&#10;&#10;\tikzstyle{my-box}=[&#10; rectangle,&#10; draw=hidden-draw,&#10; rounded corners,&#10; align=left,&#10; text opacity=1,&#10; minimum height=1.5em,&#10; minimum width=5em,&#10; inner sep=2pt,&#10; fill opacity=.8,&#10; line width=0.8pt,&#10;]&#10;&#10;\tikzstyle{leaf-head}=[my-box, minimum height=1.5em,&#10; draw=gray!80, fill=gray!15, text=black, font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;&#10;\tikzstyle{leaf-task}=[my-box, minimum height=1.5em,&#10; draw=red!70, fill=red!15, text=black, font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;&#10;\tikzstyle{leaf-paradigms}=[my-box, minimum height=1.5em,&#10; draw=cyan!70, fill=cyan!15, text=black, font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;\tikzstyle{leaf-others}=[my-box, minimum height=1.5em,&#10; draw=orange!80, fill=orange!15, text=black, font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;&#10;\tikzstyle{modelnode-task}=[my-box, minimum height=1.5em,&#10; draw=red!80, fill=hidden-pink!30, text=black, font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;&#10;\tikzstyle{modelnode-paradigms}=[my-box, minimum height=1.5em,&#10; draw=cyan!100, fill=hidden-pink!30, text=black, font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;&#10;\tikzstyle{modelnode-others}=[my-box, minimum height=1.5em,&#10; draw=orange!100, fill=hidden-pink!30, text=black, font=,&#10; inner xsep=2pt,&#10; inner ysep=4pt,&#10; line width=0.8pt,&#10;]&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Information Extraction, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2302.05019" label="2302.05019">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey on Automatic Knowledge Graph Construction" />
          <attvalue for="1" value="  Automatic knowledge graph construction aims to manufacture structured human&#10;knowledge. To this end, much effort has historically been spent extracting&#10;informative fact patterns from different data sources. However, more recently,&#10;research interest has shifted to acquiring conceptualized structured knowledge&#10;beyond informative data. In addition, researchers have also been exploring new&#10;ways of handling sophisticated construction tasks in diversified scenarios.&#10;Thus, there is a demand for a systematic review of paradigms to organize&#10;knowledge structures beyond data-level mentions. To meet this demand, we&#10;comprehensively survey more than 300 methods to summarize the latest&#10;developments in knowledge graph construction. A knowledge graph is built in&#10;three steps: knowledge acquisition, knowledge refinement, and knowledge&#10;evolution. The processes of knowledge acquisition are reviewed in detail,&#10;including obtaining entities with fine-grained types and their conceptual&#10;linkages to knowledge graphs; resolving coreferences; and extracting entity&#10;relationships in complex scenarios. The survey covers models for knowledge&#10;refinement, including knowledge graph completion, and knowledge fusion. Methods&#10;to handle knowledge evolution are also systematically presented, including&#10;condition knowledge acquisition, condition knowledge graph completion, and&#10;knowledge dynamic. We present the paradigms to compare the distinction among&#10;these methods along the axis of the data environment, motivation, and&#10;architecture. Additionally, we also provide briefs on accessible resources that&#10;can help readers to develop practical knowledge graph systems. The survey&#10;concludes with discussions on the challenges and possible directions for future&#10;exploration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.00230" label="1909.00230">
        <attvalues>
          <attvalue for="0" value="Collaborative Policy Learning for Open Knowledge Graph Reasoning" />
          <attvalue for="1" value="  In recent years, there has been a surge of interests in interpretable graph&#10;reasoning methods. However, these models often suffer from limited performance&#10;when working on sparse and incomplete graphs, due to the lack of evidential&#10;paths that can reach target entities. Here we study open knowledge graph&#10;reasoning---a task that aims to reason for missing facts over a graph augmented&#10;by a background text corpus. A key challenge of the task is to filter out&#10;&quot;irrelevant&quot; facts extracted from corpus, in order to maintain an effective&#10;search space during path inference. We propose a novel reinforcement learning&#10;framework to train two collaborative agents jointly, i.e., a multi-hop graph&#10;reasoner and a fact extractor. The fact extraction agent generates fact triples&#10;from corpora to enrich the graph on the fly; while the reasoning agent provides&#10;feedback to the fact extractor and guides it towards promoting facts that are&#10;helpful for the interpretable reasoning. Experiments on two public datasets&#10;demonstrate the effectiveness of the proposed approach. Source code and&#10;datasets used in this paper can be downloaded at&#10;https://github.com/shanzhenren/CPL&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.02962" label="2311.02962">
        <attvalues>
          <attvalue for="0" value="Retrieval-Augmented Code Generation for Universal Information Extraction" />
          <attvalue for="1" value="  Information Extraction (IE) aims to extract structural knowledge (e.g.,&#10;entities, relations, events) from natural language texts, which brings&#10;challenges to existing methods due to task-specific schemas and complex text&#10;expressions. Code, as a typical kind of formalized language, is capable of&#10;describing structural knowledge under various schemas in a universal way. On&#10;the other hand, Large Language Models (LLMs) trained on both codes and texts&#10;have demonstrated powerful capabilities of transforming texts into codes, which&#10;provides a feasible solution to IE tasks. Therefore, in this paper, we propose&#10;a universal retrieval-augmented code generation framework based on LLMs, called&#10;Code4UIE, for IE tasks. Specifically, Code4UIE adopts Python classes to define&#10;task-specific schemas of various structural knowledge in a universal way. By so&#10;doing, extracting knowledge under these schemas can be transformed into&#10;generating codes that instantiate the predefined Python classes with the&#10;information in texts. To generate these codes more precisely, Code4UIE adopts&#10;the in-context learning mechanism to instruct LLMs with examples. In order to&#10;obtain appropriate examples for different tasks, Code4UIE explores several&#10;example retrieval strategies, which can retrieve examples semantically similar&#10;to the given texts. Extensive experiments on five representative IE tasks&#10;across nine datasets demonstrate the effectiveness of the Code4UIE framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.16160" label="2407.16160">
        <attvalues>
          <attvalue for="0" value="UniMEL: A Unified Framework for Multimodal Entity Linking with Large&#10;  Language Models" />
          <attvalue for="1" value="  Multimodal Entity Linking (MEL) is a crucial task that aims at linking&#10;ambiguous mentions within multimodal contexts to the referent entities in a&#10;multimodal knowledge base, such as Wikipedia. Existing methods focus heavily on&#10;using complex mechanisms and extensive model tuning methods to model the&#10;multimodal interaction on specific datasets. However, these methods&#10;overcomplicate the MEL task and overlook the visual semantic information, which&#10;makes them costly and hard to scale. Moreover, these methods can not solve the&#10;issues like textual ambiguity, redundancy, and noisy images, which severely&#10;degrade their performance. Fortunately, the advent of Large Language Models&#10;(LLMs) with robust capabilities in text understanding and reasoning,&#10;particularly Multimodal Large Language Models (MLLMs) that can process&#10;multimodal inputs, provides new insights into addressing this challenge.&#10;However, how to design a universally applicable LLMs-based MEL approach remains&#10;a pressing challenge. To this end, we propose UniMEL, a unified framework which&#10;establishes a new paradigm to process multimodal entity linking tasks using&#10;LLMs. In this framework, we employ LLMs to augment the representation of&#10;mentions and entities individually by integrating textual and visual&#10;information and refining textual information. Subsequently, we employ the&#10;embedding-based method for retrieving and re-ranking candidate entities. Then,&#10;with only ~0.26% of the model parameters fine-tuned, LLMs can make the final&#10;selection from the candidate entities. Extensive experiments on three public&#10;benchmark datasets demonstrate that our solution achieves state-of-the-art&#10;performance, and ablation studies verify the effectiveness of all modules. Our&#10;code is available at https://github.com/Javkonline/UniMEL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.03758" label="2311.03758">
        <attvalues>
          <attvalue for="0" value="Large Language Model based Long-tail Query Rewriting in Taobao Search" />
          <attvalue for="1" value="  In the realm of e-commerce search, the significance of semantic matching&#10;cannot be overstated, as it directly impacts both user experience and company&#10;revenue. Along this line, query rewriting, serving as an important technique to&#10;bridge the semantic gaps inherent in the semantic matching process, has&#10;attached wide attention from the industry and academia. However, existing query&#10;rewriting methods often struggle to effectively optimize long-tail queries and&#10;alleviate the phenomenon of &quot;few-recall&quot; caused by semantic gap. In this paper,&#10;we present BEQUE, a comprehensive framework that Bridges the sEmantic gap for&#10;long-tail QUEries. In detail, BEQUE comprises three stages: multi-instruction&#10;supervised fine tuning (SFT), offline feedback, and objective alignment. We&#10;first construct a rewriting dataset based on rejection sampling and auxiliary&#10;tasks mixing to fine-tune our large language model (LLM) in a supervised&#10;fashion. Subsequently, with the well-trained LLM, we employ beam search to&#10;generate multiple candidate rewrites, and feed them into Taobao offline system&#10;to obtain the partial order. Leveraging the partial order of rewrites, we&#10;introduce a contrastive learning method to highlight the distinctions between&#10;rewrites, and align the model with the Taobao online objectives. Offline&#10;experiments prove the effectiveness of our method in bridging semantic gap.&#10;Online A/B tests reveal that our method can significantly boost gross&#10;merchandise volume (GMV), number of transaction (#Trans) and unique visitor&#10;(UV) for long-tail queries. BEQUE has been deployed on Taobao, one of most&#10;popular online shopping platforms in China, since October 2023.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.16789" label="2405.16789">
        <attvalues>
          <attvalue for="0" value="NoteLLM-2: Multimodal Large Representation Models for Recommendation" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated exceptional text&#10;understanding. Existing works explore their application in text embedding&#10;tasks. However, there are few works utilizing LLMs to assist multimodal&#10;representation tasks. In this work, we investigate the potential of LLMs to&#10;enhance multimodal representation in multimodal item-to-item (I2I)&#10;recommendations. One feasible method is the transfer of Multimodal Large&#10;Language Models (MLLMs) for representation tasks. However, pre-training MLLMs&#10;usually requires collecting high-quality, web-scale multimodal data, resulting&#10;in complex training procedures and high costs. This leads the community to rely&#10;heavily on open-source MLLMs, hindering customized training for representation&#10;scenarios. Therefore, we aim to design an end-to-end training method that&#10;customizes the integration of any existing LLMs and vision encoders to&#10;construct efficient multimodal representation models. Preliminary experiments&#10;show that fine-tuned LLMs in this end-to-end method tend to overlook image&#10;content. To overcome this challenge, we propose a novel training framework,&#10;NoteLLM-2, specifically designed for multimodal representation. We propose two&#10;ways to enhance the focus on visual information. The first method is based on&#10;the prompt viewpoint, which separates multimodal content into visual content&#10;and textual content. NoteLLM-2 adopts the multimodal In-Content Learning method&#10;to teach LLMs to focus on both modalities and aggregate key information. The&#10;second method is from the model architecture, utilizing a late fusion mechanism&#10;to directly fuse visual information into textual information. Extensive&#10;experiments have been conducted to validate the effectiveness of our method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.01744" label="2403.01744">
        <attvalues>
          <attvalue for="0" value="NoteLLM: A Retrievable Large Language Model for Note Recommendation" />
          <attvalue for="1" value="  People enjoy sharing &quot;notes&quot; including their experiences within online&#10;communities. Therefore, recommending notes aligned with user interests has&#10;become a crucial task. Existing online methods only input notes into BERT-based&#10;models to generate note embeddings for assessing similarity. However, they may&#10;underutilize some important cues, e.g., hashtags or categories, which represent&#10;the key concepts of notes. Indeed, learning to generate hashtags/categories can&#10;potentially enhance note embeddings, both of which compress key note&#10;information into limited content. Besides, Large Language Models (LLMs) have&#10;significantly outperformed BERT in understanding natural languages. It is&#10;promising to introduce LLMs into note recommendation. In this paper, we propose&#10;a novel unified framework called NoteLLM, which leverages LLMs to address the&#10;item-to-item (I2I) note recommendation. Specifically, we utilize Note&#10;Compression Prompt to compress a note into a single special token, and further&#10;learn the potentially related notes' embeddings via a contrastive learning&#10;approach. Moreover, we use NoteLLM to summarize the note and generate the&#10;hashtag/category automatically through instruction tuning. Extensive&#10;validations on real scenarios demonstrate the effectiveness of our proposed&#10;method compared with the online baseline and show major improvements in the&#10;recommendation system of Xiaohongshu.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.11432" label="2308.11432">
        <attvalues>
          <attvalue for="0" value="A Survey on Large Language Model based Autonomous Agents" />
          <attvalue for="1" value="  Autonomous agents have long been a prominent research focus in both academic&#10;and industry communities. Previous research in this field often focuses on&#10;training agents with limited knowledge within isolated environments, which&#10;diverges significantly from human learning processes, and thus makes the agents&#10;hard to achieve human-like decisions. Recently, through the acquisition of vast&#10;amounts of web knowledge, large language models (LLMs) have demonstrated&#10;remarkable potential in achieving human-level intelligence. This has sparked an&#10;upsurge in studies investigating LLM-based autonomous agents. In this paper, we&#10;present a comprehensive survey of these studies, delivering a systematic review&#10;of the field of LLM-based autonomous agents from a holistic perspective. More&#10;specifically, we first discuss the construction of LLM-based autonomous agents,&#10;for which we propose a unified framework that encompasses a majority of the&#10;previous work. Then, we present a comprehensive overview of the diverse&#10;applications of LLM-based autonomous agents in the fields of social science,&#10;natural science, and engineering. Finally, we delve into the evaluation&#10;strategies commonly used for LLM-based autonomous agents. Based on the previous&#10;studies, we also present several challenges and future directions in this&#10;field. To keep track of this field and continuously update our survey, we&#10;maintain a repository of relevant references at&#10;https://github.com/Paitesanshi/LLM-Agent-Survey.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.21075" label="2405.21075">
        <attvalues>
          <attvalue for="0" value="Video-MME: The First-Ever Comprehensive Evaluation Benchmark of&#10;  Multi-modal LLMs in Video Analysis" />
          <attvalue for="1" value="  In the quest for artificial general intelligence, Multi-modal Large Language&#10;Models (MLLMs) have emerged as a focal point in recent advancements. However,&#10;the predominant focus remains on developing their capabilities in static image&#10;understanding. The potential of MLLMs in processing sequential visual data is&#10;still insufficiently explored, highlighting the absence of a comprehensive,&#10;high-quality assessment of their performance. In this paper, we introduce&#10;Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of&#10;MLLMs in Video analysis. Our work distinguishes from existing benchmarks&#10;through four key features: 1) Diversity in video types, spanning 6 primary&#10;visual domains with 30 subfields to ensure broad scenario generalizability; 2)&#10;Duration in temporal dimension, encompassing both short-, medium-, and&#10;long-term videos, ranging from 11 seconds to 1 hour, for robust contextual&#10;dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides&#10;video frames, including subtitles and audios, to unveil the all-round&#10;capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual&#10;labeling by expert annotators to facilitate precise and reliable model&#10;assessment. 900 videos with a total of 254 hours are manually selected and&#10;annotated by repeatedly viewing all the video content, resulting in 2,700&#10;question-answer pairs. With Video-MME, we extensively evaluate various&#10;state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as&#10;open-source image models like InternVL-Chat-V1.5 and video models like&#10;LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the&#10;best-performing commercial model, significantly outperforming the open-source&#10;models. Our dataset along with these findings underscores the need for further&#10;improvements in handling longer sequences and multi-modal data. Project Page:&#10;https://video-mme.github.io&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.15450" label="2312.15450">
        <attvalues>
          <attvalue for="0" value="Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting&#10;  Using Multi-agent LLM" />
          <attvalue for="1" value="  Search engines are crucial as they provide an efficient and easy way to&#10;access vast amounts of information on the internet for diverse information&#10;needs. User queries, even with a specific need, can differ significantly. Prior&#10;research has explored the resilience of ranking models against typical query&#10;variations like paraphrasing, misspellings, and order changes. Yet, these works&#10;overlook how diverse demographics uniquely formulate identical queries. For&#10;instance, older individuals tend to construct queries more naturally and in&#10;varied order compared to other groups. This demographic diversity necessitates&#10;enhancing the adaptability of ranking models to diverse query formulations. To&#10;this end, in this paper, we propose a framework that integrates a novel&#10;rewriting pipeline that rewrites queries from various demographic perspectives&#10;and a novel framework to enhance ranking robustness. To be specific, we use&#10;Chain of Thought (CoT) technology to utilize Large Language Models (LLMs) as&#10;agents to emulate various demographic profiles, then use them for efficient&#10;query rewriting, and we innovate a robust Multi-gate Mixture of Experts (MMoE)&#10;architecture coupled with a hybrid loss function, collectively strengthening&#10;the ranking models' robustness. Our extensive experimentation on both public&#10;and industrial datasets assesses the efficacy of our query rewriting approach&#10;and the enhanced accuracy and robustness of the ranking model. The findings&#10;highlight the sophistication and effectiveness of our proposed model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13981" label="2305.13981">
        <attvalues>
          <attvalue for="0" value="Preserving Knowledge Invariance: Rethinking Robustness Evaluation of&#10;  Open Information Extraction" />
          <attvalue for="1" value="  The robustness to distribution changes ensures that NLP models can be&#10;successfully applied in the realistic world, especially for information&#10;extraction tasks. However, most prior evaluation benchmarks have been devoted&#10;to validating pairwise matching correctness, ignoring the crucial measurement&#10;of robustness. In this paper, we present the first benchmark that simulates the&#10;evaluation of open information extraction models in the real world, where the&#10;syntactic and expressive distributions under the same knowledge meaning may&#10;drift variously. We design and annotate a large-scale testbed in which each&#10;example is a knowledge-invariant clique that consists of sentences with&#10;structured knowledge of the same meaning but with different syntactic and&#10;expressive forms. By further elaborating the robustness metric, a model is&#10;judged to be robust if its performance is consistently accurate on the overall&#10;cliques. We perform experiments on typical models published in the last decade&#10;as well as a popular large language model, the results show that the existing&#10;successful models exhibit a frustrating degradation, with a maximum drop of&#10;23.43 F1 score. Our resources and code are available at&#10;https://github.com/qijimrc/ROBUST.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08340" label="2112.08340">
        <attvalues>
          <attvalue for="0" value="GenIE: Generative Information Extraction" />
          <attvalue for="1" value="  Structured and grounded representation of text is typically formalized by&#10;closed information extraction, the problem of extracting an exhaustive set of&#10;(subject, relation, object) triplets that are consistent with a predefined set&#10;of entities and relations from a knowledge base schema. Most existing works are&#10;pipelines prone to error accumulation, and all approaches are only applicable&#10;to unrealistically small numbers of entities and relations. We introduce GenIE&#10;(generative information extraction), the first end-to-end autoregressive&#10;formulation of closed information extraction. GenIE naturally exploits the&#10;language knowledge from the pre-trained transformer by autoregressively&#10;generating relations and entities in textual form. Thanks to a new bi-level&#10;constrained generation strategy, only triplets consistent with the predefined&#10;knowledge base schema are produced. Our experiments show that GenIE is&#10;state-of-the-art on closed information extraction, generalizes from fewer&#10;training data points than baselines, and scales to a previously unmanageable&#10;number of entities and relations. With this work, closed information extraction&#10;becomes practical in realistic scenarios, providing new opportunities for&#10;downstream tasks. Finally, this work paves the way towards a unified end-to-end&#10;approach to the core tasks of information extraction. Code, data and models&#10;available at https://github.com/epfl-dlab/GenIE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.06838" label="2311.06838">
        <attvalues>
          <attvalue for="0" value="GIELLM: Japanese General Information Extraction Large Language Model&#10;  Utilizing Mutual Reinforcement Effect" />
          <attvalue for="1" value="  Information Extraction (IE) stands as a cornerstone in natural language&#10;processing, traditionally segmented into distinct sub-tasks. The advent of&#10;Large Language Models (LLMs) heralds a paradigm shift, suggesting the&#10;feasibility of a singular model addressing multiple IE subtasks. In this vein,&#10;we introduce the General Information Extraction Large Language Model (GIELLM),&#10;which integrates text Classification, Sentiment Analysis, Named Entity&#10;Recognition, Relation Extraction, and Event Extraction using a uniform&#10;input-output schema. This innovation marks the first instance of a model&#10;simultaneously handling such a diverse array of IE subtasks. Notably, the&#10;GIELLM leverages the Mutual Reinforcement Effect (MRE), enhancing performance&#10;in integrated tasks compared to their isolated counterparts. Our experiments&#10;demonstrate State-of-the-Art (SOTA) results in five out of six Japanese mixed&#10;datasets, significantly surpassing GPT-3.5-Turbo. Further, an independent&#10;evaluation using the novel Text Classification Relation and Event&#10;Extraction(TCREE) dataset corroborates the synergistic advantages of MRE in&#10;text and word classification. This breakthrough paves the way for most IE&#10;subtasks to be subsumed under a singular LLM framework. Specialized fine-tune&#10;task-specific models are no longer needed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Universal Information Extraction. Prior to the advent of LLMs, significant efforts were made to consolidate various IE subtasks into a unified model using a generative approach. Notably, UIE and USM stand out as pioneering frameworks, both employing a common strategy to integrate the input-output processes of IE subtasks. Specifically, UIE\cite{lu-etal-2022-unified} and USM\cite{lou2023universal} utilize T5\cite{raffel2020exploring} and RoBERTa\cite{liu2019roberta} models, respectively, for training, focusing solely on word-level IE challenges. In the \cite{yan-etal-2021-unified-generative}, the approach leverages a seq2seq framework to address flat, nested, and discontinuous NER tasks. GenIE\cite{josifoski-etal-2022-genie} employs a transformer-based model to extract information from unstructured text, applying global structural constraints. Similarly, InstructionNER\cite{wang2022instructionner} utilizes the T5 model to identify entity spans based on instructions and options. However, these methods are confined to NER, RE, and EE within IE. UniSA\cite{li2023unisa} innovatively combines a transformer decoder with a multimodal transformer encoder, using contrastive learning to adeptly handle various sentiment analysis subtasks within a singular model framework. The SLG Framework(Sentence-to-label Generation Framework)\cite{10.1007/978-3-031-35320-8_18} presents MRE for the first time. And uses the T5 model to process the Mix task for SC and NER. The experimental results show that the combined two tasks work better than individually.&#10;&#10;LLMs Information Extraction. After the advent of LLMs, basically all IE methods shifted to interacting with LLMs via prompts. Let the model answer the information that wants to extract. Code4UIE\cite{guo2023retrievalaugmented} innovatively employs a two-stage prompting process that inputs IE subtasks into LLMs in a standardized code format, subsequently eliciting the extraction of information in the same format. InstructUIE\cite{wang2023instructuie}, on the other hand, tailors specific instructional texts to different IE subtasks, guiding LLMs to output information in the required format. Diverging from the sole reliance on prompts and LLMs for IE subtasks, UniversalNER\cite{zhou2023universalner} leverages LLMs with capacities of 7B and 13B for NER. This approach involves constructing prompts to distill knowledge from the ChatGPT model, which is then used to fine-tune the 7B and 13B LLMs. Remarkably, this method has shown excellent performance in both in-domain and out-of-domain NER tasks. The USA model\cite{gan2023usa} introduces an innovative approach, the MRE, which intertwines word-level and sentence-level analysis in sentiment analysis tasks. This model, with a 7B capacity, adeptly handles complex subtasks of Sentiment Analysis, encompassing both text sentiment classification and part-of-speech sentiment classification.&#10;&#10;However, the majority of these studies focused solely on word-level information extraction, neglecting text-level considerations. The MRE between these levels was largely overlooked, with only a minority of studies incorporating MRE. This research, in contrast, innovatively proposes employing a singular model to encompass all key IE subtasks, including NER, RE, EE, Sentiment Analysis, and Text Classification.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2304.06248" label="2304.06248">
        <attvalues>
          <attvalue for="0" value="LasUIE: Unifying Information Extraction with Latent Adaptive&#10;  Structure-aware Generative Language Model" />
          <attvalue for="1" value="  Universally modeling all typical information extraction tasks (UIE) with one&#10;generative language model (GLM) has revealed great potential by the latest&#10;study, where various IE predictions are unified into a linearized hierarchical&#10;expression under a GLM. Syntactic structure information, a type of effective&#10;feature which has been extensively utilized in IE community, should also be&#10;beneficial to UIE. In this work, we propose a novel structure-aware GLM, fully&#10;unleashing the power of syntactic knowledge for UIE. A heterogeneous structure&#10;inductor is explored to unsupervisedly induce rich heterogeneous structural&#10;representations by post-training an existing GLM. In particular, a structural&#10;broadcaster is devised to compact various latent trees into explicit high-order&#10;forests, helping to guide a better generation during decoding. We finally&#10;introduce a task-oriented structure fine-tuning mechanism, further adjusting&#10;the learned structures to most coincide with the end-task's need. Over 12 IE&#10;benchmarks across 7 tasks our system shows significant improvements over the&#10;baseline UIE system. Further in-depth analyses show that our GLM learns rich&#10;task-adaptive structural bias that greatly resolves the UIE crux, the&#10;long-range dependence issue and boundary identifying. Source codes are open at&#10;https://github.com/ChocoWu/LasUIE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.05711" label="2305.05711">
        <attvalues>
          <attvalue for="0" value="CodeIE: Large Code Generation Models are Better Few-Shot Information&#10;  Extractors" />
          <attvalue for="1" value="  Large language models (LLMs) pre-trained on massive corpora have demonstrated&#10;impressive few-shot learning ability on many NLP tasks. A common practice is to&#10;recast the task into a text-to-text format such that generative LLMs of natural&#10;language (NL-LLMs) like GPT-3 can be prompted to solve it. However, it is&#10;nontrivial to perform information extraction (IE) tasks with NL-LLMs since the&#10;output of the IE task is usually structured and therefore is hard to be&#10;converted into plain text. In this paper, we propose to recast the structured&#10;output in the form of code instead of natural language and utilize generative&#10;LLMs of code (Code-LLMs) such as Codex to perform IE tasks, in particular,&#10;named entity recognition and relation extraction. In contrast to NL-LLMs, we&#10;show that Code-LLMs can be well-aligned with these IE tasks by designing&#10;code-style prompts and formulating these IE tasks as code generation tasks.&#10;Experiment results on seven benchmarks show that our method consistently&#10;outperforms fine-tuning moderate-size pre-trained models specially designed for&#10;IE tasks (e.g., UIE) and prompting NL-LLMs under few-shot settings. We further&#10;conduct a series of in-depth analyses to demonstrate the merits of leveraging&#10;Code-LLMs for IE tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.01223" label="2106.01223">
        <attvalues>
          <attvalue for="0" value="A Unified Generative Framework for Various NER Subtasks" />
          <attvalue for="1" value="  Named Entity Recognition (NER) is the task of identifying spans that&#10;represent entities in sentences. Whether the entity spans are nested or&#10;discontinuous, the NER task can be categorized into the flat NER, nested NER,&#10;and discontinuous NER subtasks. These subtasks have been mainly solved by the&#10;token-level sequence labelling or span-level classification. However, these&#10;solutions can hardly tackle the three kinds of NER subtasks concurrently. To&#10;that end, we propose to formulate the NER subtasks as an entity span sequence&#10;generation task, which can be solved by a unified sequence-to-sequence&#10;(Seq2Seq) framework. Based on our unified framework, we can leverage the&#10;pre-trained Seq2Seq model to solve all three kinds of NER subtasks without the&#10;special design of the tagging schema or ways to enumerate spans. We exploit&#10;three types of entity representations to linearize entities into a sequence.&#10;Our proposed framework is easy-to-implement and achieves state-of-the-art&#10;(SoTA) or near SoTA performance on eight English NER datasets, including two&#10;flat NER datasets, three nested NER datasets, and three discontinuous NER&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04901" label="2109.04901">
        <attvalues>
          <attvalue for="0" value="Document-level Entity-based Extraction as Template Generation" />
          <attvalue for="1" value="  Document-level entity-based extraction (EE), aiming at extracting&#10;entity-centric information such as entity roles and entity relations, is key to&#10;automatic knowledge acquisition from text corpora for various domains. Most&#10;document-level EE systems build extractive models, which struggle to model&#10;long-term dependencies among entities at the document level. To address this&#10;issue, we propose a generative framework for two document-level EE tasks:&#10;role-filler entity extraction (REE) and relation extraction (RE). We first&#10;formulate them as a template generation problem, allowing models to efficiently&#10;capture cross-entity dependencies, exploit label semantics, and avoid the&#10;exponential computation complexity of identifying N-ary relations. A novel&#10;cross-attention guided copy mechanism, TopK Copy, is incorporated into a&#10;pre-trained sequence-to-sequence model to enhance the capabilities of&#10;identifying key information in the input document. Experiments done on the&#10;MUC-4 and SciREX dataset show new state-of-the-art results on REE (+3.26%),&#10;binary RE (+4.8%), and 4-ary RE (+2.7%) in F1 score.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.12810" label="2210.12810">
        <attvalues>
          <attvalue for="0" value="Code4Struct: Code Generation for Few-Shot Event Structure Prediction" />
          <attvalue for="1" value="  Large Language Model (LLM) trained on a mixture of text and code has&#10;demonstrated impressive capability in translating natural language (NL) into&#10;structured code. We observe that semantic structures can be conveniently&#10;translated into code and propose Code4Struct to leverage such text-to-structure&#10;translation capability to tackle structured prediction tasks. As a case study,&#10;we formulate Event Argument Extraction (EAE) as converting text into&#10;event-argument structures that can be represented as a class object using code.&#10;This alignment between structures and code enables us to take advantage of&#10;Programming Language (PL) features such as inheritance and type annotation to&#10;introduce external knowledge or add constraints. We show that, with sufficient&#10;in-context examples, formulating EAE as a code generation problem is&#10;advantageous over using variants of text-based prompts. Despite only using 20&#10;training event instances for each event type, Code4Struct is comparable to&#10;supervised models trained on 4,202 instances and outperforms current&#10;state-of-the-art (SOTA) trained on 20-shot data by 29.5% absolute F1.&#10;Code4Struct can use 10-shot training data from a sibling event type to predict&#10;arguments for zero-resource event types and outperforms the zero-shot baseline&#10;by 12% absolute F1.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.00253" label="2402.00253">
        <attvalues>
          <attvalue for="0" value="A Survey on Hallucination in Large Vision-Language Models" />
          <attvalue for="1" value="  Recent development of Large Vision-Language Models (LVLMs) has attracted&#10;growing attention within the AI landscape for its practical implementation&#10;potential. However, ``hallucination'', or more specifically, the misalignment&#10;between factual visual content and corresponding textual generation, poses a&#10;significant challenge of utilizing LVLMs. In this comprehensive survey, we&#10;dissect LVLM-related hallucinations in an attempt to establish an overview and&#10;facilitate future mitigation. Our scrutiny starts with a clarification of the&#10;concept of hallucinations in LVLMs, presenting a variety of hallucination&#10;symptoms and highlighting the unique challenges inherent in LVLM&#10;hallucinations. Subsequently, we outline the benchmarks and methodologies&#10;tailored specifically for evaluating hallucinations unique to LVLMs.&#10;Additionally, we delve into an investigation of the root causes of these&#10;hallucinations, encompassing insights from the training data and model&#10;components. We also critically review existing methods for mitigating&#10;hallucinations. The open questions and future directions pertaining to&#10;hallucinations within LVLMs are discussed to conclude this survey.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.18099" label="2402.18099">
        <attvalues>
          <attvalue for="0" value="Editing Factual Knowledge and Explanatory Ability of Medical Large&#10;  Language Models" />
          <attvalue for="1" value="  Model editing aims to precisely alter the behaviors of large language models&#10;(LLMs) in relation to specific knowledge, while leaving unrelated knowledge&#10;intact. This approach has proven effective in addressing issues of&#10;hallucination and outdated information in LLMs. However, the potential of using&#10;model editing to modify knowledge in the medical field remains largely&#10;unexplored, even though resolving hallucination is a pressing need in this&#10;area. Our observations indicate that current methods face significant&#10;challenges in dealing with specialized and complex knowledge in medical domain.&#10;Therefore, we propose MedLaSA, a novel Layer-wise Scalable Adapter strategy for&#10;medical model editing. MedLaSA harnesses the strengths of both adding extra&#10;parameters and locate-then-edit methods for medical model editing. We utilize&#10;causal tracing to identify the association of knowledge in neurons across&#10;different layers, and generate a corresponding scale set from the association&#10;value for each piece of knowledge. Subsequently, we incorporate scalable&#10;adapters into the dense layers of LLMs. These adapters are assigned scaling&#10;values based on the corresponding specific knowledge, which allows for the&#10;adjustment of the adapter's weight and rank. The more similar the content, the&#10;more consistent the scale between them. This ensures precise editing of&#10;semantically identical knowledge while avoiding impact on unrelated knowledge.&#10;To evaluate the editing impact on the behaviours of LLMs, we propose two model&#10;editing studies for medical domain: (1) editing factual knowledge for medical&#10;specialization and (2) editing the explanatory ability for complex knowledge.&#10;We build two novel medical benchmarking datasets and introduce a series of&#10;challenging and comprehensive metrics. Extensive experiments on medical LLMs&#10;demonstrate the editing efficiency of MedLaSA, without affecting unrelated&#10;knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.06484" label="2408.06484">
        <attvalues>
          <attvalue for="0" value="Cross-Lingual Conversational Speech Summarization with Large Language&#10;  Models" />
          <attvalue for="1" value="  Cross-lingual conversational speech summarization is an important problem,&#10;but suffers from a dearth of resources. While transcriptions exist for a number&#10;of languages, translated conversational speech is rare and datasets containing&#10;summaries are non-existent. We build upon the existing Fisher and Callhome&#10;Spanish-English Speech Translation corpus by supplementing the translations&#10;with summaries. The summaries are generated using GPT-4 from the reference&#10;translations and are treated as ground truth. The task is to generate similar&#10;summaries in the presence of transcription and translation errors. We build a&#10;baseline cascade-based system using open-source speech recognition and machine&#10;translation models. We test a range of LLMs for summarization and analyze the&#10;impact of transcription and translation errors. Adapting the Mistral-7B model&#10;for this task performs significantly better than off-the-shelf models and&#10;matches the performance of GPT-4.&#10;" />
          <attvalue for="2" value="&#10;&#10;Despite the advances in automatic speech recognition (ASR) since the advent of deep neural networks, conversational speech remains a significant challenge.&#10;Due to the lack of data and challenging recording conditions, word error rates (WER) remain high.&#10;Even when accurately transcribed, conversational speech is difficult to read; making it beneficial to build models of cross-lingual summarization that can generate more human-readable versions of conversations. &#10;Faithful summarization captures the important information in the conversation without the distractions of hesitations and other speech disfluencies.&#10;&#10;Recent advances in large language models (LLMs) have allowed them to match or even surpass the capabilities of special purpose models for a number of tasks, including summarization. &#10;For some datasets, LLMs have been found to not only surpass previous summarization models, but to meet the performance of humans \cite{zhang2024benchmarking}.&#10;LLMs have the additional benefit of flexibility.&#10;With prompting and finetuning, the model can be made to take advantage of additional information, or to provide a contextual summary based on additional instructions.&#10;&#10;Summarization of speech has a long history \cite{valenza1999summarisation}.&#10;The majority of the work has focused on single-speaker audio with a clear goal (e.g., voicemail \cite{koumpis2000transcription}, broadcast news \cite{hori2002automatic}).&#10;Two-party conversations have also been a domain of interest.&#10;Meeting summarization was explored in \cite{murray05extractive}.&#10;\cite{sharma2023espnet} released a large corpus of interviews where a host interviews a guest.&#10;Another major area is the summarization of call center interactions \cite{zou2021topic}.&#10;Given the audio from an interaction between a customer and a call center employee, the goal is to describe the nature of the call and whether and how a request was fulfilled.&#10;To our knowledge, there is no prior work on the cross-lingual summarization of conversational speech outside of the call-center domain.&#10;&#10;The classic approach to cross-lingual speech summarization has been a cascaded pipeline where audio is automatically transcribed and then fed to a summarization system \cite{furui2004speech}.&#10;A benefit of this approach is that the individual models can be trained independently, taking advantage of non-parallel data.&#10;More recent work has explored direct summarization where a single model is used to directly summarize the audio \cite{kano2023summarize}.&#10;The model can be directly optimized for the task, as opposed to individual components being optimized for intermediate objectives.&#10;&#10;Summarization can be either abstractive or extractive.&#10;In a cross-lingual conversational speech domain, extractive summarization can be problematic.&#10;Both transcription and translation errors are propagated, and the extracted utterances can be incomplete or incoherent \cite{liu2013towards}.&#10;Because of these issues, we focus on abstractive summarization.&#10;While direct approaches can be powerful, we focus on a cascaded approach which allows us to incorporate and compare open-source models.&#10;We aim to establish an evaluation framework for conversational speech summarization and to evaluate the ability of LLMs to accomplish the task.&#10;We leave comparisons against direct summarization to future work.&#10;Our contributions are as follows:&#10;&#10;\begin{itemize}&#10; \item We provide a first of its kind public conversational speech summarization dataset by building upon existing datasets.&#10; \item We compare a range of LLMs and provide baseline performance using open-source tools and models.&#10; \item We demonstrate that by fine-tuning a relatively small, quantized LLM, we achieve performance competitive with GPT-4.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Translation, Computer Science, Linguistics, Language Model Evaluation, Information Science, Natural Language Processing, Conversational Summarization" />
        </attvalues>
      </node>
      <node id="2012.07311" label="2012.07311">
        <attvalues>
          <attvalue for="0" value="Topic-Oriented Spoken Dialogue Summarization for Customer Service with&#10;  Saliency-Aware Topic Modeling" />
          <attvalue for="1" value="  In a customer service system, dialogue summarization can boost service&#10;efficiency by automatically creating summaries for long spoken dialogues in&#10;which customers and agents try to address issues about specific topics. In this&#10;work, we focus on topic-oriented dialogue summarization, which generates highly&#10;abstractive summaries that preserve the main ideas from dialogues. In spoken&#10;dialogues, abundant dialogue noise and common semantics could obscure the&#10;underlying informative content, making the general topic modeling approaches&#10;difficult to apply. In addition, for customer service, role-specific&#10;information matters and is an indispensable part of a summary. To effectively&#10;perform topic modeling on dialogues and capture multi-role information, in this&#10;work we propose a novel topic-augmented two-stage dialogue summarizer (TDS)&#10;jointly with a saliency-aware neural topic model (SATM) for topic-oriented&#10;summarization of customer service dialogues. Comprehensive studies on a&#10;real-world Chinese customer service dataset demonstrated the superiority of our&#10;method against several strong baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.01319" label="2408.01319">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Review of Multimodal Large Language Models: Performance&#10;  and Challenges Across Different Tasks" />
          <attvalue for="1" value="  In an era defined by the explosive growth of data and rapid technological&#10;advancements, Multimodal Large Language Models (MLLMs) stand at the forefront&#10;of artificial intelligence (AI) systems. Designed to seamlessly integrate&#10;diverse data types-including text, images, videos, audio, and physiological&#10;sequences-MLLMs address the complexities of real-world applications far beyond&#10;the capabilities of single-modality systems. In this paper, we systematically&#10;sort out the applications of MLLM in multimodal tasks such as natural language,&#10;vision, and audio. We also provide a comparative analysis of the focus of&#10;different MLLMs in the tasks, and provide insights into the shortcomings of&#10;current MLLMs, and suggest potential directions for future research. Through&#10;these discussions, this paper hopes to provide valuable insights for the&#10;further development and application of MLLM.&#10;" />
          <attvalue for="2" value="&#10;&#10;M{LLMs} are sophisticated artificial intelligence (AI) systems designed to process and integrate various types of data, including text, images, videos, audio, and physiological sequential data~\cite{wu2023multimodal,yin2023survey,zhang2024mm}. As we navigate the era of multimodal data fusion, marked by rapid advancements in information technology and an explosive increase in data volume, the capabilities of single-modality systems no longer suffice for complex real-world tasks~\cite{koh2024generating,ma2024eye,zhang2024potential}. Thus, the development of MLLMs is not only an inevitable trend in technological evolution but also a critical enhancement for improving the effectiveness of AI applications. By synergizing information from multiple data sources, MLLMs cultivate a more comprehensive and accurate representation of information, this capability not only unlocks significant potential but also demonstrates substantial practical application value across a diverse range of fields. The integration of diverse datasets tailors MLLMs to perform more effectively, establishing them as indispensable next-generation techniques in the ongoing quest to harness the full potential of AI technologies~\cite{zhang2023biomedgpt,mei2024phraseaug,xiao2024instruction}. Notably, MLLMs have shown remarkable performance across diverse multimodal tasks, including language, image, video, and audio processing. These models excel in integrating multimoda information to enhance the effectiveness of multimodal tasks. &#10;&#10;In Natural Language Processing (NLP) tasks such as text generation and machine translation, MLLMs leverage images, video, and audio to provide contextual support, enhancing the accuracy and expressiveness of the generated text~\cite{malik2021automatic,bahar2019comparative,lyu2023macaw}. These models also excel in sentiment analysis and dialog systems by integrating multimodal information to improve understanding and generation capabilities. In particular, MLLMs transform NLP by incorporating visual and auditory data, thus enriching text generation and machine translation~\cite{achiam2023gpt,zheng2024judging,le2023bloom}. These models enhance the accuracy and expressiveness of the generated text, providing nuanced contextual support that traditional models cannot. In sentiment analysis and dialogue systems, MLLMs are capable of the integration of multimodal information, which further deepens the understanding of system and response capabilities, presenting a leap forward in human-computer interaction~\cite{wang2024largelanguagemodelsrobotics,wang2023review}.&#10;&#10;In addition, in the vision tasks, MLLMs significantly enhance improves task comprehension, analysis, and generation. Integrating textual descriptions and image instructions commands allows for more accurate tasks such as image classification, target detection, and image annotation. For instance, MLLMs such as GPT-4V~\cite{achiam2023gpt}, and Gemini~\cite{team2023gemini} combine image content with natural language descriptions to produce more vivid and precise annotation results. These models also show progress in image generation, creating images from textual descriptions or enabling cross-modal image style migration, thus broadening the possibilities within this field. Meanwhile, video processing poses unique challenges due to its complexity. Nevertheless, the advent of MLLMs has propelled forward the capabilities of language models in this domain. Models like NExT-GPT~\cite{wu2023next} and Sora~\cite{liu2024sora} are pioneering multimodal video generation, producing richer and more realistic video content by learning from multimodal data. Furthermore, advancements in intelligent video understanding technologies, such as VideoChat~\cite{li2023videochat} and Video-LLaVA~\cite{lin2023video}, have significantly enhanced the ability to analyze and process video content. These developments promise enhanced user experiences in virtual reality, video games, and educational applications.&#10;&#10;Furthermore, In audio tasks, MLLMs bring a new technological change to audio processing tasks. Traditional audio processing usually relies on unimodal signal processing methods, such as speech recognition~\cite{gaikwad2010review} or audio classification~\cite{mulimani2024class}, which have limitations in processing complex multimodal data.MLLMs are able to understand better and generate audio-related content by combining audio signals with textual and visual information through the combination of Large Language Models (LLMs). For example, in speech generation tasks, MLLMs can utilize textual and visual information to generate more natural and contextually relevant speech output~\cite{shen2024hugginggpt,huang2024audiogpt}. In audio understanding tasks, these models can perform sentiment recognition, audio classification, or audio event detection more accurately by combining visual cues and textual descriptions. In addition, MLLMs show strong potential in tasks such as cross-modal audio-text translation, audio soundtrack generation, and multimodal sentiment analysis~\cite{tang2023salmonn,team2023gemini}. These technological advances not only improve the effectiveness of audio processing, but also expand its scenarios in real-world applications such as smart homes, virtual assistants, movie and television production, etc.&#10;&#10;As shown in Fig~\ref{Fignure}, we summarize MLLMs from recent years. This paper reviews the current state of the art of MLLM applications, introduces the basic concepts and main architectures of MLLMs in Section 2, describes their performance in different domains to identify their strengths and weaknesses in Section 3, highlights the transformative impact of MLLMs through a comparative analysis in Section 4, and provides a roadmap for future research in Section 5. Our discussion aims to incentivize continuous innovation and ensure that MLLMs remain at the forefront of AI technology development. Through a comprehensive review of current implementations and progress, this paper aims to summarize research results, provide valuable references, and offer guidance for future research in the field of MLLMs. Our goal is to inspire new ideas and directions for the continued development of MLLMs, ensuring that they remain at the forefront of AI technology development.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Multimodal Task Applications, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Multimodal AI Systems" />
        </attvalues>
      </node>
      <node id="2311.13165" label="2311.13165">
        <attvalues>
          <attvalue for="0" value="Multimodal Large Language Models: A Survey" />
          <attvalue for="1" value="  The exploration of multimodal language models integrates multiple data types,&#10;such as images, text, language, audio, and other heterogeneity. While the&#10;latest large language models excel in text-based tasks, they often struggle to&#10;understand and process other data types. Multimodal models address this&#10;limitation by combining various modalities, enabling a more comprehensive&#10;understanding of diverse data. This paper begins by defining the concept of&#10;multimodal and examining the historical development of multimodal algorithms.&#10;Furthermore, we introduce a range of multimodal products, focusing on the&#10;efforts of major technology companies. A practical guide is provided, offering&#10;insights into the technical aspects of multimodal models. Moreover, we present&#10;a compilation of the latest algorithms and commonly used datasets, providing&#10;researchers with valuable resources for experimentation and evaluation. Lastly,&#10;we explore the applications of multimodal models and discuss the challenges&#10;associated with their development. By addressing these aspects, this paper aims&#10;to facilitate a deeper understanding of multimodal models and their potential&#10;in various domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.13549" label="2306.13549">
        <attvalues>
          <attvalue for="0" value="A Survey on Multimodal Large Language Models" />
          <attvalue for="1" value="  Recently, Multimodal Large Language Model (MLLM) represented by GPT-4V has&#10;been a new rising research hotspot, which uses powerful Large Language Models&#10;(LLMs) as a brain to perform multimodal tasks. The surprising emergent&#10;capabilities of MLLM, such as writing stories based on images and OCR-free math&#10;reasoning, are rare in traditional multimodal methods, suggesting a potential&#10;path to artificial general intelligence. To this end, both academia and&#10;industry have endeavored to develop MLLMs that can compete with or even better&#10;than GPT-4V, pushing the limit of research at a surprising speed. In this&#10;paper, we aim to trace and summarize the recent progress of MLLMs. First of&#10;all, we present the basic formulation of MLLM and delineate its related&#10;concepts, including architecture, training strategy and data, as well as&#10;evaluation. Then, we introduce research topics about how MLLMs can be extended&#10;to support more granularity, modalities, languages, and scenarios. We continue&#10;with multimodal hallucination and extended techniques, including Multimodal ICL&#10;(M-ICL), Multimodal CoT (M-CoT), and LLM-Aided Visual Reasoning (LAVR). To&#10;conclude the paper, we discuss existing challenges and point out promising&#10;research directions. In light of the fact that the era of MLLM has only just&#10;begun, we will keep updating this survey and hope it can inspire more research.&#10;An associated GitHub link collecting the latest papers is available at&#10;https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.05758" label="2407.05758">
        <attvalues>
          <attvalue for="0" value="Potential of Multimodal Large Language Models for Data Mining of Medical&#10;  Images and Free-text Reports" />
          <attvalue for="1" value="  Medical images and radiology reports are crucial for diagnosing medical&#10;conditions, highlighting the importance of quantitative analysis for clinical&#10;decision-making. However, the diversity and cross-source heterogeneity of these&#10;data challenge the generalizability of current data-mining methods. Multimodal&#10;large language models (MLLMs) have recently transformed many domains,&#10;significantly affecting the medical field. Notably, Gemini-Vision-series&#10;(Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in&#10;Artificial General Intelligence (AGI) for computer vision, showcasing their&#10;potential in the biomedical domain. In this study, we evaluated the performance&#10;of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation&#10;across 14 medical imaging datasets, including 5 medical imaging categories&#10;(dermatology, radiology, dentistry, ophthalmology, and endoscopy), and 3&#10;radiology report datasets. The investigated tasks encompass disease&#10;classification, lesion segmentation, anatomical localization, disease&#10;diagnosis, report generation, and lesion detection. Our experimental results&#10;demonstrated that Gemini-series models excelled in report generation and lesion&#10;detection but faces challenges in disease classification and anatomical&#10;localization. Conversely, GPT-series models exhibited proficiency in lesion&#10;segmentation and anatomical localization but encountered difficulties in&#10;disease diagnosis and lesion detection. Additionally, both the Gemini series&#10;and GPT series contain models that have demonstrated commendable generation&#10;efficiency. While both models hold promise in reducing physician workload,&#10;alleviating pressure on limited healthcare resources, and fostering&#10;collaboration between clinical practitioners and artificial intelligence&#10;technologies, substantial enhancements and comprehensive validations remain&#10;imperative before clinical deployment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.08870" label="1911.08870">
        <attvalues>
          <attvalue for="0" value="A Comparative Study on End-to-end Speech to Text Translation" />
          <attvalue for="1" value="  Recent advances in deep learning show that end-to-end speech to text&#10;translation model is a promising approach to direct the speech translation&#10;field. In this work, we provide an overview of different end-to-end&#10;architectures, as well as the usage of an auxiliary connectionist temporal&#10;classification (CTC) loss for better convergence. We also investigate on&#10;pre-training variants such as initializing different components of a model&#10;using pre-trained models, and their impact on the final performance, which&#10;gives boosts up to 4% in BLEU and 5% in TER. Our experiments are performed on&#10;270h IWSLT TED-talks En-&gt;De, and 100h LibriSpeech Audiobooks En-&gt;Fr. We also&#10;show improvements over the current end-to-end state-of-the-art systems on both&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.09093" label="2306.09093">
        <attvalues>
          <attvalue for="0" value="Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and&#10;  Text Integration" />
          <attvalue for="1" value="  Although instruction-tuned large language models (LLMs) have exhibited&#10;remarkable capabilities across various NLP tasks, their effectiveness on other&#10;data modalities beyond text has not been fully studied. In this work, we&#10;propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual,&#10;audio, and textual information. Macaw-LLM consists of three main components: a&#10;modality module for encoding multi-modal data, a cognitive module for&#10;harnessing pretrained LLMs, and an alignment module for harmonizing diverse&#10;representations. Our novel alignment module seamlessly bridges multi-modal&#10;features to textual features, simplifying the adaptation process from the&#10;modality modules to the cognitive module. In addition, we construct a&#10;large-scale multi-modal instruction dataset in terms of multi-turn dialogue,&#10;including 69K image instances and 50K video instances. We have made our data,&#10;code and model publicly available, which we hope can pave the way for future&#10;research in multi-modal LLMs and expand the capabilities of LLMs to handle&#10;diverse data modalities and address complex real-world scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.00855" label="2307.00855">
        <attvalues>
          <attvalue for="0" value="Review of Large Vision Models and Visual Prompt Engineering" />
          <attvalue for="1" value="  Visual prompt engineering is a fundamental technology in the field of visual&#10;and image Artificial General Intelligence, serving as a key component for&#10;achieving zero-shot capabilities. As the development of large vision models&#10;progresses, the importance of prompt engineering becomes increasingly evident.&#10;Designing suitable prompts for specific visual tasks has emerged as a&#10;meaningful research direction. This review aims to summarize the methods&#10;employed in the computer vision domain for large vision models and visual&#10;prompt engineering, exploring the latest advancements in visual prompt&#10;engineering. We present influential large models in the visual domain and a&#10;range of prompt engineering methods employed on these models. It is our hope&#10;that this review provides a comprehensive and systematic description of prompt&#10;engineering methods based on large visual models, offering valuable insights&#10;for future researchers in their exploration of this field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.17177" label="2402.17177">
        <attvalues>
          <attvalue for="0" value="Sora: A Review on Background, Technology, Limitations, and Opportunities&#10;  of Large Vision Models" />
          <attvalue for="1" value="  Sora is a text-to-video generative AI model, released by OpenAI in February&#10;2024. The model is trained to generate videos of realistic or imaginative&#10;scenes from text instructions and show potential in simulating the physical&#10;world. Based on public technical reports and reverse engineering, this paper&#10;presents a comprehensive review of the model's background, related&#10;technologies, applications, remaining challenges, and future directions of&#10;text-to-video AI models. We first trace Sora's development and investigate the&#10;underlying technologies used to build this &quot;world simulator&quot;. Then, we describe&#10;in detail the applications and potential impact of Sora in multiple industries&#10;ranging from film-making and education to marketing. We discuss the main&#10;challenges and limitations that need to be addressed to widely deploy Sora,&#10;such as ensuring safe and unbiased video generation. Lastly, we discuss the&#10;future development of Sora and video generation models in general, and how&#10;advancements in the field could enable new ways of human-AI interaction,&#10;boosting productivity and creativity of video generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.10122" label="2311.10122">
        <attvalues>
          <attvalue for="0" value="Video-LLaVA: Learning United Visual Representation by Alignment Before&#10;  Projection" />
          <attvalue for="1" value="  The Large Vision-Language Model (LVLM) has enhanced the performance of&#10;various downstream tasks in visual-language understanding. Most existing&#10;approaches encode images and videos into separate feature spaces, which are&#10;then fed as inputs to large language models. However, due to the lack of&#10;unified tokenization for images and videos, namely misalignment before&#10;projection, it becomes challenging for a Large Language Model (LLM) to learn&#10;multi-modal interactions from several poor projection layers. In this work, we&#10;unify visual representation into the language feature space to advance the&#10;foundational LLM towards a unified LVLM. As a result, we establish a simple but&#10;robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images&#10;and videos, mutually enhancing each other. Video-LLaVA achieves superior&#10;performances on a broad range of 9 image benchmarks across 5 image&#10;question-answering datasets and 4 image benchmark toolkits. Additionally, our&#10;Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on&#10;MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive&#10;experiments demonstrate that Video-LLaVA mutually benefits images and videos&#10;within a unified visual representation, outperforming models designed&#10;specifically for images or videos. We aim for this work to provide modest&#10;insights into the multi-modal inputs for the LLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.04447" label="2401.04447">
        <attvalues>
          <attvalue for="0" value="Class-Incremental Learning for Multi-Label Audio Classification" />
          <attvalue for="1" value="  In this paper, we propose a method for class-incremental learning of&#10;potentially overlapping sounds for solving a sequence of multi-label audio&#10;classification tasks. We design an incremental learner that learns new classes&#10;independently of the old classes. To preserve knowledge about the old classes,&#10;we propose a cosine similarity-based distillation loss that minimizes&#10;discrepancy in the feature representations of subsequent learners, and use it&#10;along with a Kullback-Leibler divergence-based distillation loss that minimizes&#10;discrepancy in their respective outputs. Experiments are performed on a dataset&#10;with 50 sound classes, with an initial classification task containing 30 base&#10;classes and 4 incremental phases of 5 classes each. After each phase, the&#10;system is tested for multi-label classification with the entire set of classes&#10;learned so far. The proposed method obtains an average F1-score of 40.9% over&#10;the five phases, ranging from 45.2% in phase 0 on 30 classes, to 36.3% in phase&#10;4 on 50 classes. Average performance degradation over incremental phases is&#10;only 0.7 percentage points from the initial F1-score of 45.2%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.06264" label="2006.06264">
        <attvalues>
          <attvalue for="0" value="Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine&#10;  Translation Evaluation Metrics" />
          <attvalue for="1" value="  Automatic metrics are fundamental for the development and evaluation of&#10;machine translation systems. Judging whether, and to what extent, automatic&#10;metrics concur with the gold standard of human evaluation is not a&#10;straightforward problem. We show that current methods for judging metrics are&#10;highly sensitive to the translations used for assessment, particularly the&#10;presence of outliers, which often leads to falsely confident conclusions about&#10;a metric's efficacy. Finally, we turn to pairwise system ranking, developing a&#10;method for thresholding performance improvement under an automatic metric&#10;against human judgements, which allows quantification of type I versus type II&#10;errors incurred, i.e., insignificant human differences in system quality that&#10;are accepted, and significant human differences that are rejected. Together,&#10;these findings suggest improvements to the protocols for metric evaluation and&#10;system performance evaluation in machine translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.11509" label="2403.11509">
        <attvalues>
          <attvalue for="0" value="DEE: Dual-stage Explainable Evaluation Method for Text Generation" />
          <attvalue for="1" value="  Automatic methods for evaluating machine-generated texts hold significant&#10;importance due to the expanding applications of generative systems.&#10;Conventional methods tend to grapple with a lack of explainability, issuing a&#10;solitary numerical score to signify the assessment outcome. Recent advancements&#10;have sought to mitigate this limitation by incorporating large language models&#10;(LLMs) to offer more detailed error analyses, yet their applicability remains&#10;constrained, particularly in industrial contexts where comprehensive error&#10;coverage and swift detection are paramount. To alleviate these challenges, we&#10;introduce DEE, a Dual-stage Explainable Evaluation method for estimating the&#10;quality of text generation. Built upon Llama 2, DEE follows a dual-stage&#10;principle guided by stage-specific instructions to perform efficient&#10;identification of errors in generated texts in the initial stage and&#10;subsequently delves into providing comprehensive diagnostic reports in the&#10;second stage. DEE is fine-tuned on our elaborately assembled dataset AntEval,&#10;which encompasses 15K examples from 4 real-world applications of Alipay that&#10;employ generative systems. The dataset concerns newly emerged issues like&#10;hallucination and toxicity, thereby broadening the scope of DEE's evaluation&#10;criteria. Experimental results affirm that DEE's superiority over existing&#10;evaluation methods, achieving significant improvements in both human&#10;correlation as well as efficiency.&#10;" />
          <attvalue for="2" value="&#10;Recent advancements in LLMs, like LLaMA\cite{llama} and OpenAI's GPT series\cite{gpt-4}, have led to widespread use in various applications, especially in industrial scenarios. One significant challenge is ensuring the quality of the content these models generate. For instance, Alipay employs generative systems for the automatic generation of social media posts, but this raises issues like potential toxicity\cite{toxicity} or incoherence\cite{coherence} in the content. Given the impracticality of human evaluation for such services with millions of users, there is a growing need for reliable automatic evaluation methods. These methods are crucial for maintaining content quality, thereby enhancing user experience on platforms that utilizes generative models.&#10;&#10;Existing methods\cite{bartscore, rouge, bertscore, gptscore, instructscore, tigerscore} for text generation primarily focus on basic aspects but inadequately address recent emerging challenges. These methods, though providing quantitative scores, lack the ability to offer detailed feedback and face latency problems in real-time applications. Their weaknesses include:&#10;&#10;Limited Evaluation Dimensions. Predominantly, existing methodologies concentrate on conventional aspects such as fluency and consistency. However, as LLMs evolve, more sophisticated and fluent outputs are generated, bringing forth novel challenges like hallucinations\cite{hallu}, biases and toxicity\cite{toxicity}. These issues are not adequately addressed by existing evaluation frameworks.&#10;&#10;Deficient Explainability. Existing methods like ROUGE\cite{rouge} and BERTScore\cite{bertscore} primarily provide quantitative scores without explanatory feedback. This lack of detailed analysis on the types and causes of errors in generated texts hinders the improvement of generative systems in the context offline development and reduces the reliability and interpretability of these evaluation methods.&#10;&#10;Lack of Efficiency. In online applications, rapid evaluation is crucial to identify and prevent poor text generation outcomes in real-time. LLM-based methods, such as InstructScore\cite{instructscore} and TIGERScore\cite{tigerscore}, provide detailed diagnostic reports but suffer from significant latency issues due to the inherent inefficiency of LLM inference. This limits their effectiveness in time-sensitive environments.&#10;&#10;In this paper, we introduce DEE, a Dual-stage Explainable Evaluation method for text generation in industrial scenarios. DEE leverages Llama 2 and operates in two stages, as shown in Fig. \ref{fig-overview}. Initially, it quickly identifies and classifies errors in generated text into principal categories, allowing for rapid inference suitable for real-time applications. The second stage, powered by our assembled AntEval dataset, conducts an in-depth analysis of each error, providing detailed explanations. AntEval encompasses 15K examples from 4 real-world applications of Alipay based on generative systems. By including the newly emerged issues mentioned above, AntEval enables DEE to perform comprehensive evaluations. Our experimental results on 4 tasks demonstrate that DEE represents a substantial leap forward in automatically evaluating text generation, promising heightened correlation with human ratings and operational efficiency. In summary, the contributions of this paper include:&#10;&#10;\begin{itemize}&#10; \item We present an innovative dual-stage evaluation method for text generation in industrial scenarios. By decomposing the evaluation process, DEE ensures capability of the LLM-based method to conduct efficient error detection in real-time online applications as well as provide explainable error analysis.&#10; \item We introduce a dataset derived from real-world industrial applications containing recently emerged problems of generative systems. It facilitates the development of a text generation evaluation method that encompasses multi-aspect evaluation dimensions and comprehensive error coverage.&#10; \item Experimental results on the real-world dataset elaborate the superiority of our method compared to existing competitors, achieving state-of-the-art performance in industrial scenarios.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Text Evaluation, Linguistics, Language Model Applications, Explainable AI Methods, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.00752" label="2310.00752">
        <attvalues>
          <attvalue for="0" value="TIGERScore: Towards Building Explainable Metric for All Text Generation&#10;  Tasks" />
          <attvalue for="1" value="  We present TIGERScore, a \textbf{T}rained metric that follows&#10;\textbf{I}nstruction \textbf{G}uidance to perform \textbf{E}xplainable, and&#10;\textbf{R}eference-free evaluation over a wide spectrum of text generation&#10;tasks. Different from other automatic evaluation methods that only provide&#10;arcane scores, TIGERScore is guided by natural language instruction to provide&#10;error analysis to pinpoint the mistakes in the generated text. Our metric is&#10;based on LLaMA-2, trained on our meticulously curated instruction-tuning&#10;dataset MetricInstruct which covers 6 text generation tasks and 23 text&#10;generation datasets. The dataset consists of 42K quadruple in the form of&#10;(instruction, input, system output $\rightarrow$ error analysis). We collected&#10;the `system outputs' through from a large variety of models to cover different&#10;types of errors. To quantitatively assess our metric, we evaluate its&#10;correlation with human ratings on 5 held-in datasets, 2 held-out datasets and&#10;show that TIGERScore can achieve the open-source SoTA correlation with human&#10;ratings across these datasets and almost approaches GPT-4 evaluator. As a&#10;reference-free metric, its correlation can even surpass the best existing&#10;reference-based metrics. To further qualitatively assess the rationale&#10;generated by our metric, we conduct human evaluation on the generated&#10;explanations and found that the explanations are 70.8\% accurate. Through these&#10;experimental results, we believe TIGERScore demonstrates the possibility of&#10;building universal explainable metrics to evaluate any text generation task.&#10;All the resourced are released in our project website:&#10;\url{https://tiger-ai-lab.github.io/TIGERScore/}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.15023" label="2110.15023">
        <attvalues>
          <attvalue for="0" value="Empirical Analysis of Korean Public AI Hub Parallel Corpora and in-depth&#10;  Analysis using LIWC" />
          <attvalue for="1" value="  Machine translation (MT) system aims to translate source language into target&#10;language. Recent studies on MT systems mainly focus on neural machine&#10;translation (NMT). One factor that significantly affects the performance of NMT&#10;is the availability of high-quality parallel corpora. However, high-quality&#10;parallel corpora concerning Korean are relatively scarce compared to those&#10;associated with other high-resource languages, such as German or Italian. To&#10;address this problem, AI Hub recently released seven types of parallel corpora&#10;for Korean. In this study, we conduct an in-depth verification of the quality&#10;of corresponding parallel corpora through Linguistic Inquiry and Word Count&#10;(LIWC) and several relevant experiments. LIWC is a word-counting software&#10;program that can analyze corpora in multiple ways and extract linguistic&#10;features as a dictionary base. To the best of our knowledge, this study is the&#10;first to use LIWC to analyze parallel corpora in the field of NMT. Our findings&#10;suggest the direction of further research toward obtaining the improved quality&#10;parallel corpora through our correlation analysis in LIWC and NMT performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, the demand for machine translation (MT) systems has been continuously increasing and its importance is growing, especially for the industrial services. \cite{vieira2021understanding, zheng2019testing} Companies, such as Google, Facebook, Microsoft, Amazon, and Unbabel continue to conduct research and formulate plans to commercialize applications related to MT.&#10;&#10;From the late 1950s, numerous MT-related projects were proceeded by mainly focusing on rule-based and statistical-based approaches before the advent of deep learning technology. As deep learning based neural machine translation (NMT) was proposed and adopted to several researches, it has been gradually figured out that more superior performance can be derived through NMT approach \cite{bahdanau2014neural,vaswani2017attention,lample2019cross,song2019mass}.&#10;&#10;Followed by the adoption of deep learning based technique, the improvements of computing power (e.g. GPU) and corresponding enhancement of parallel processing accelerated the advancement of NMT. Recently, release of open source frameworks, such as Pytorch\cite{NEURIPS2019_9015}, and lowered accessibility to the big data further facilitated vigorous and diverse research.&#10;&#10;However, several issues considering the enhancement of the NMT system remain still. Representatively, limitations in ensuring the quality of data is an unresolved issue. As have previously been studied, the quality of the training data is deeply related to the NMT performance \cite{park2020toward,park2021study}. The major problem is that the process of building a high-quality parallel corpus is time-consuming and expensive, and it is significantly difficult for low-resource languages, such as Korean. Although data-augmentation techniques, such as back translation \cite{edunov2018understanding} and copied translation \cite{currey2017copied} have been introduced, as the human supervision is generally minimized or excluded in the data generation process, the quality of such pseudo-generated parallel corpus cannot be guaranteed \cite{burlot2019using, epaliyana2021improving}. This restricted the usage of pseudo-generated parallel to complements of human-labeled gold parallel corpus, rather than its substitutes \cite{imankulova2017improving}.&#10;&#10;For the alleviation of above limitations, numerous studies on the collection of high-quality training data have been conducted, such as parallel corpus filtering (PCF) research and Data Dam project. PCF refers to a research field that aims to filter out low-quality noisy data (i.e. sentence pairs) residing in the parallel corpus, and improve the overall quality of the corpus. PCF is currently being applied to various NMT studies and contributed to the advancement of the NMT systems \cite{koehn2019findings, park2020quality}. While the amount of training data caused significant impact on the statistical-based MT approaches, the quality of data is treated as more important than the amount of data in general deep learning-based MT approaches \cite{khayrallah2018impact, koehn-EtAl:2020:WMT}. Moreover, Data Dam projects for building high-quality parallel corpora nationally are in progress. In the Republic of Korea, a large number of parallel corpora is open to the public through AI-Hub, which is organized by the National Information Society Agency (NIA) \cite{park2020study}.&#10;&#10;Following these research trends, where the quality is treated more importantly than the quantity in the data construction process, we analyzed the above Korean-English parallel corpus distributed by AI-Hub. Despite its sufficient amount of data, the quality of corresponding corpus has not been confirmed clearly. This may restrict the unconstrained utilization of such corpus in adoption to the NMT model, as low quality data may degrade the overall performance. &#10;In this study, we conducted several quality verification experiments including Linguistic Inquiry and Word Count (LIWC) \cite{pennebaker2001linguistic,tausczik2010psychological}, and clarified the quality and characteristics of such corpus. By analyzing various factors that can affect NMT performance, we proposed a method that can be applied in future research using the analysis results.&#10;&#10;LIWC is a text-analysis tool that automatically analyzes the number of words in a sentence and classifies words with similar meanings and sentimental characteristics. LIWC extracts various interpersonal variables related to clinical, social, physiological, cognitive, psychological, and developmental contexts that cannot be detected using previous text-analysis programs. Additionally, LIWC comprises a variety of features for analyzing text. LIWC generally used to recognize linguistic markers for mental health study in Psychopathology such as detecting Narcissism\cite{holtzman2019linguistic}, schizophrenia\cite{bae2021schizophrenia}, bipolar disorder\cite{sekulic2018not}. However, LIWC provides various linguistic features, word count, gender bias and so on, so it can be used for various analyses. In this study, we use LIWC to analyze parallel corpora based on diverse properties. It is also first time to analyze corpus using LIWC.&#10;&#10;In addition, we conduct baseline translation experiments by training transformer-base model structure \cite{vaswani2017attention} through all the parallel corpora given by AIhub. By analyzing MT performance of corresponding models, we propose further research directions on MT for the Korean language. The contributions of this study are as follows:&#10;&#10;\begin{itemize}&#10;\item{For the first time, we conduct a deep data analysis on AI-Hub data. To the best of our knowledge, this is the first time LIWC has been used to analyze corpora. This study acts as a milestone for further studies on NMT with respect to the Korean language.}&#10;&#10;\item{We conduct baseline translation experiments on all the data in the AI-Hub parallel corpus. Our experiments provide a foundation for further research on Korean-based NMT.}&#10;&#10;\item{We discovered that many factors might cause decreasing model performance, and we provide the direction that those factors could be filtered through our correlation analysis between LIWC and model performance.} &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Language Corpora, Data Science, Artificial Intelligence, Neural Networks" />
        </attvalues>
      </node>
      <node id="1903.11437" label="1903.11437">
        <attvalues>
          <attvalue for="0" value="Using Monolingual Data in Neural Machine Translation: a Systematic Study" />
          <attvalue for="1" value="  Neural Machine Translation (MT) has radically changed the way systems are&#10;developed. A major difference with the previous generation (Phrase-Based MT) is&#10;the way monolingual target data, which often abounds, is used in these two&#10;paradigms. While Phrase-Based MT can seamlessly integrate very large language&#10;models trained on billions of sentences, the best option for Neural MT&#10;developers seems to be the generation of artificial parallel data through&#10;\textsl{back-translation} - a technique that fails to fully take advantage of&#10;existing datasets. In this paper, we conduct a systematic study of&#10;back-translation, comparing alternative uses of monolingual data, as well as&#10;multiple data generation procedures. Our findings confirm that back-translation&#10;is very effective and give new explanations as to why this is the case. We also&#10;introduce new data simulation techniques that are almost as effective, yet much&#10;cheaper to implement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.12282" label="1805.12282">
        <attvalues>
          <attvalue for="0" value="On the Impact of Various Types of Noise on Neural Machine Translation" />
          <attvalue for="1" value="  We examine how various types of noise in the parallel training data impact&#10;the quality of neural machine translation systems. We create five types of&#10;artificial noise and analyze how they degrade performance in neural and&#10;statistical machine translation. We find that neural models are generally more&#10;harmed by noise than statistical models. For one especially egregious type of&#10;noise they learn to just copy the input sentence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.04655" label="1811.04655">
        <attvalues>
          <attvalue for="0" value="Not Just Depressed: Bipolar Disorder Prediction on Reddit" />
          <attvalue for="1" value="  Bipolar disorder, an illness characterized by manic and depressive episodes,&#10;affects more than 60 million people worldwide. We present a preliminary study&#10;on bipolar disorder prediction from user-generated text on Reddit, which relies&#10;on users' self-reported labels. Our benchmark classifiers for bipolar disorder&#10;prediction outperform the baselines and reach accuracy and F1-scores of above&#10;86%. Feature analysis shows interesting differences in language use between&#10;users with bipolar disorders and the control group, including differences in&#10;the use of emotion-expressive words.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.08329" label="2212.08329">
        <attvalues>
          <attvalue for="0" value="Text-to-speech synthesis based on latent variable conversion using&#10;  diffusion probabilistic model and variational autoencoder" />
          <attvalue for="1" value="  Text-to-speech synthesis (TTS) is a task to convert texts into speech. Two of&#10;the factors that have been driving TTS are the advancements of probabilistic&#10;models and latent representation learning. We propose a TTS method based on&#10;latent variable conversion using a diffusion probabilistic model and the&#10;variational autoencoder (VAE). In our TTS method, we use a waveform model based&#10;on VAE, a diffusion model that predicts the distribution of latent variables in&#10;the waveform model from texts, and an alignment model that learns alignments&#10;between the text and speech latent sequences. Our method integrates diffusion&#10;with VAE by modeling both mean and variance parameters with diffusion, where&#10;the target distribution is determined by approximation from VAE. This latent&#10;variable conversion framework potentially enables us to flexibly incorporate&#10;various latent feature extractors. Our experiments show that our method is&#10;robust to linguistic labels with poor orthography and alignment errors.&#10;" />
          <attvalue for="2" value="&#10;Text-to-speech synthesis (TTS) is a task to convert texts into speech. Recent TTS methods achieved high naturalness in synthetic speech that is comparable to human speech \cite{DBLP:conf/icassp/ShenPWSJYCZWRSA18_short, DBLP:journals/corr/abs-1809-08895, DBLP:conf/icml/KimKS21}. One of the reasons that have been driving the performance of TTS is the advancement of probabilistic models such as autoregressive \cite{oord2016wavenet}, generative flow \cite{rezende2015variational}, and generative adversarial network \cite{goodfellow2014generative} models. Recently, the diffusion probabilistic model (DPM) \cite{DBLP:conf/nips/HoJA20} has been developed as another advanced probabilistic model, and it has been intensively studied in speech domains such as neural vocoders \cite{DBLP:conf/iclr/ChenZZWNC21}, and TTS \cite{jeong21_interspeech, chen21p_interspeech}. DPM is a particularly interesting approach for speech generation in terms of conditioning, because speech generation is about conditional distribution modeling. The iterative inference process of diffusion enables conditioning by various methods such as adaptive prior \cite{DBLP:conf/iclr/LeeKS0LMQ0YL22}, classifier guidance \cite{DBLP:conf/icml/KimKY22}, and iterative latent variable refinement \cite{levkovitch22_interspeech}. &#10;&#10;Another driving factor of the TTS advancement is latent representation learning. The use of the variational autoencoder (VAE) \cite{DBLP:journals/corr/KingmaW13} is a popular representation learning method for speech style modeling \cite{DBLP:conf/interspeech/KlimkovRRD19, DBLP:conf/iclr/HsuZWZWWCJCSNP19}. Designing a discrete latent space in VAE enables the capture of phonetic latent information in speech \cite{DBLP:conf/nips/OordVK17}, and a categorical latent space enables the use of a semisupervised approach to capture the desired latent information such as speech emotion \cite{DBLP:conf/iclr/HabibMSBSSKB20}.&#10;Self-supervised learning (SSL) based on a pretrained representation is also applied to speech synthesis to utilize large-scale unpaired texts or speeches \cite{jia21_interspeech, DBLP:conf/interspeech/SiuzdakDRJ22}. Supervised fine-tuning in SSL enables the adoption of pretrained features to specific domains, and it is used to capture accent features for TTS \cite{9829304}.&#10;&#10;TTS is expected to be improved by incorporating a diffusion probabilistic model and latent representation learning. However, diffusion models typically use a fixed variance, which results in a suboptimal likelihood \cite{DBLP:conf/icml/NicholD21}, and the application of diffusion to latent variable modeling, where the variance of latent variables is approximated by a model, is as yet not known. In addition, there is room to investigate alignment methods in conditional distribution modeling with diffusion, where alignments between the condition and the target are not known in TTS.&#10;&#10;In this research, we propose a TTS method based on latent variable conversion. Inspired by a recent TTS method based on VAE \cite{DBLP:journals/corr/KingmaW13} incorporating a waveform model \cite{DBLP:conf/icml/KimKS21}, our method convert texts into waveforms via a latent space. We use diffusion to model the distribution of a latent representation in the VAE-based waveform model to convert texts into waveforms. To model a latent variable with an approximated distribution, we formulate the diffusion-based acoustic model capable of modeling both mean and variance parameters. We also propose an alignment model to align latent representations of texts and acoustic features. Our method enables us to use various latent representation potentially including SSL-based methods, but we limit our focus on the normal VAE in this paper. Our contributions are as follows: (1) we propose a TTS method that models the distribution of latent variables with diffusion taking both mean and variance parameters into consideration;&#10;(2) Our experimental results imply that the modeling data in diffusion provides a more stable speech quality than modeling noise with less training time; (3) Our experimental results show that our method is robust to linguistic labels with poor orthography and alignment errors.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Probabilistic Modeling, Computer Science, Text-to-Speech Synthesis, Linguistics, Latent Representation Learning, Signal Processing, Mathematics" />
        </attvalues>
      </node>
      <node id="2009.00713" label="2009.00713">
        <attvalues>
          <attvalue for="0" value="WaveGrad: Estimating Gradients for Waveform Generation" />
          <attvalue for="1" value="  This paper introduces WaveGrad, a conditional model for waveform generation&#10;which estimates gradients of the data density. The model is built on prior work&#10;on score matching and diffusion probabilistic models. It starts from a Gaussian&#10;white noise signal and iteratively refines the signal via a gradient-based&#10;sampler conditioned on the mel-spectrogram. WaveGrad offers a natural way to&#10;trade inference speed for sample quality by adjusting the number of refinement&#10;steps, and bridges the gap between non-autoregressive and autoregressive models&#10;in terms of audio quality. We find that it can generate high fidelity audio&#10;samples using as few as six iterations. Experiments reveal WaveGrad to generate&#10;high fidelity audio, outperforming adversarial non-autoregressive baselines and&#10;matching a strong likelihood-based autoregressive baseline using fewer&#10;sequential operations. Audio samples are available at&#10;https://wavegrad.github.io/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06406" label="2106.06406">
        <attvalues>
          <attvalue for="0" value="PriorGrad: Improving Conditional Denoising Diffusion Models with&#10;  Data-Dependent Adaptive Prior" />
          <attvalue for="1" value="  Denoising diffusion probabilistic models have been recently proposed to&#10;generate high-quality samples by estimating the gradient of the data density.&#10;The framework defines the prior noise as a standard Gaussian distribution,&#10;whereas the corresponding data distribution may be more complicated than the&#10;standard Gaussian distribution, which potentially introduces inefficiency in&#10;denoising the prior noise into the data sample because of the discrepancy&#10;between the data and the prior. In this paper, we propose PriorGrad to improve&#10;the efficiency of the conditional diffusion model for speech synthesis (for&#10;example, a vocoder using a mel-spectrogram as the condition) by applying an&#10;adaptive prior derived from the data statistics based on the conditional&#10;information. We formulate the training and sampling procedures of PriorGrad and&#10;demonstrate the advantages of an adaptive prior through a theoretical analysis.&#10;Focusing on the speech synthesis domain, we consider the recently proposed&#10;diffusion-based speech generative models based on both the spectral and time&#10;domains and show that PriorGrad achieves faster convergence and inference with&#10;superior performance, leading to an improved perceptual quality and robustness&#10;to a smaller network capacity, and thereby demonstrating the efficiency of a&#10;data-dependent adaptive prior.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.11755" label="2111.11755">
        <attvalues>
          <attvalue for="0" value="Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance" />
          <attvalue for="1" value="  We propose Guided-TTS, a high-quality text-to-speech (TTS) model that does&#10;not require any transcript of target speaker using classifier guidance.&#10;Guided-TTS combines an unconditional diffusion probabilistic model with a&#10;separately trained phoneme classifier for classifier guidance. Our&#10;unconditional diffusion model learns to generate speech without any context&#10;from untranscribed speech data. For TTS synthesis, we guide the generative&#10;process of the diffusion model with a phoneme classifier trained on a&#10;large-scale speech recognition dataset. We present a norm-based scaling method&#10;that reduces the pronunciation errors of classifier guidance in Guided-TTS. We&#10;show that Guided-TTS achieves a performance comparable to that of the&#10;state-of-the-art TTS model, Grad-TTS, without any transcript for LJSpeech. We&#10;further demonstrate that Guided-TTS performs well on diverse datasets including&#10;a long-form untranscribed dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.02246" label="2206.02246">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models" />
          <attvalue for="1" value="  We present a novel way of conditioning a pretrained denoising diffusion&#10;speech model to produce speech in the voice of a novel person unseen during&#10;training. The method requires a short (~3 seconds) sample from the target&#10;person, and generation is steered at inference time, without any training&#10;steps. At the heart of the method lies a sampling process that combines the&#10;estimation of the denoising model with a low-pass version of the new speaker's&#10;sample. The objective and subjective evaluations show that our sampling method&#10;can generate a voice similar to that of the target speaker in terms of&#10;frequency, with an accuracy comparable to state-of-the-art methods, and without&#10;training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.01709" label="1910.01709">
        <attvalues>
          <attvalue for="0" value="Semi-Supervised Generative Modeling for Controllable Speech Synthesis" />
          <attvalue for="1" value="  We present a novel generative model that combines state-of-the-art neural&#10;text-to-speech (TTS) with semi-supervised probabilistic latent variable models.&#10;By providing partial supervision to some of the latent variables, we are able&#10;to force them to take on consistent and interpretable purposes, which&#10;previously hasn't been possible with purely unsupervised TTS models. We&#10;demonstrate that our model is able to reliably discover and control important&#10;but rarely labelled attributes of speech, such as affect and speaking rate,&#10;with as little as 1% (30 minutes) supervision. Even at such low supervision&#10;levels we do not observe a degradation of synthesis quality compared to a&#10;state-of-the-art baseline. Audio samples are available on the web.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.15060" label="2103.15060">
        <attvalues>
          <attvalue for="0" value="PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS" />
          <attvalue for="1" value="  This paper introduces PnG BERT, a new encoder model for neural TTS. This&#10;model is augmented from the original BERT model, by taking both phoneme and&#10;grapheme representations of text as input, as well as the word-level alignment&#10;between them. It can be pre-trained on a large text corpus in a self-supervised&#10;manner, and fine-tuned in a TTS task. Experimental results show that a neural&#10;TTS model using a pre-trained PnG BERT as its encoder yields more natural&#10;prosody and more accurate pronunciation than a baseline model using only&#10;phoneme input with no pre-training. Subjective side-by-side preference&#10;evaluations show that raters have no statistically significant preference&#10;between the speech synthesized using a PnG BERT and ground truth recordings&#10;from professional speakers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.08321" label="2212.08321">
        <attvalues>
          <attvalue for="0" value="Investigation of Japanese PnG BERT language model in text-to-speech&#10;  synthesis for pitch accent language" />
          <attvalue for="1" value="  End-to-end text-to-speech synthesis (TTS) can generate highly natural&#10;synthetic speech from raw text. However, rendering the correct pitch accents is&#10;still a challenging problem for end-to-end TTS. To tackle the challenge of&#10;rendering correct pitch accent in Japanese end-to-end TTS, we adopt PnG~BERT, a&#10;self-supervised pretrained model in the character and phoneme domain for TTS.&#10;We investigate the effects of features captured by PnG~BERT on Japanese TTS by&#10;modifying the fine-tuning condition to determine the conditions helpful&#10;inferring pitch accents. We manipulate content of PnG~BERT features from being&#10;text-oriented to speech-oriented by changing the number of fine-tuned layers&#10;during TTS. In addition, we teach PnG~BERT pitch accent information by&#10;fine-tuning with tone prediction as an additional downstream task. Our&#10;experimental results show that the features of PnG~BERT captured by pretraining&#10;contain information helpful inferring pitch accent, and PnG~BERT outperforms&#10;baseline Tacotron on accent correctness in a listening test.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.09672" label="2102.09672">
        <attvalues>
          <attvalue for="0" value="Improved Denoising Diffusion Probabilistic Models" />
          <attvalue for="1" value="  Denoising diffusion probabilistic models (DDPM) are a class of generative&#10;models which have recently been shown to produce excellent samples. We show&#10;that with a few simple modifications, DDPMs can also achieve competitive&#10;log-likelihoods while maintaining high sample quality. Additionally, we find&#10;that learning variances of the reverse diffusion process allows sampling with&#10;an order of magnitude fewer forward passes with a negligible difference in&#10;sample quality, which is important for the practical deployment of these&#10;models. We additionally use precision and recall to compare how well DDPMs and&#10;GANs cover the target distribution. Finally, we show that the sample quality&#10;and likelihood of these models scale smoothly with model capacity and training&#10;compute, making them easily scalable. We release our code at&#10;https://github.com/openai/improved-diffusion&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06103" label="2106.06103">
        <attvalues>
          <attvalue for="0" value="Conditional Variational Autoencoder with Adversarial Learning for&#10;  End-to-End Text-to-Speech" />
          <attvalue for="1" value="  Several recent end-to-end text-to-speech (TTS) models enabling single-stage&#10;training and parallel sampling have been proposed, but their sample quality&#10;does not match that of two-stage TTS systems. In this work, we present a&#10;parallel end-to-end TTS method that generates more natural sounding audio than&#10;current two-stage models. Our method adopts variational inference augmented&#10;with normalizing flows and an adversarial training process, which improves the&#10;expressive power of generative modeling. We also propose a stochastic duration&#10;predictor to synthesize speech with diverse rhythms from input text. With the&#10;uncertainty modeling over latent variables and the stochastic duration&#10;predictor, our method expresses the natural one-to-many relationship in which a&#10;text input can be spoken in multiple ways with different pitches and rhythms. A&#10;subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a&#10;single speaker dataset, shows that our method outperforms the best publicly&#10;available TTS systems and achieves a MOS comparable to ground truth.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15610" label="2203.15610">
        <attvalues>
          <attvalue for="0" value="LightHuBERT: Lightweight and Configurable Speech Representation Learning&#10;  with Once-for-All Hidden-Unit BERT" />
          <attvalue for="1" value="  Self-supervised speech representation learning has shown promising results in&#10;various speech processing tasks. However, the pre-trained models, e.g., HuBERT,&#10;are storage-intensive Transformers, limiting their scope of applications under&#10;low-resource settings. To this end, we propose LightHuBERT, a once-for-all&#10;Transformer compression framework, to find the desired architectures&#10;automatically by pruning structured parameters. More precisely, we create a&#10;Transformer-based supernet that is nested with thousands of weight-sharing&#10;subnets and design a two-stage distillation strategy to leverage the&#10;contextualized latent representations from HuBERT. Experiments on automatic&#10;speech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERT&#10;enables over $10^9$ architectures concerning the embedding dimension, attention&#10;dimension, head number, feed-forward network ratio, and network depth.&#10;LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with&#10;the HuBERT size, achieves comparable performance to the teacher model in most&#10;tasks with a reduction of 29% parameters, and obtains a $3.5\times$ compression&#10;ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword&#10;spotting, and intent classification, with a slight accuracy loss. The code and&#10;pre-trained models are available at&#10;https://github.com/mechanicalsea/lighthubert.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Large-scale pre-trained models such as wav2vec 2.0 \cite{NEURIPS2020_92d1e1eb}, HuBERT \cite{Hsu2021}, WavLM \cite{chen2021wavlm}, SpeechT5 \cite{ao2021speecht5}, and data2vec \cite{baevski2022data2vec} have drawn much attention in the speech communities, due to their excellent generalization capability across various speech applications and efficient use of large-scale unlabeled data. SUPERB \cite{Yang2021} has appeared as a popular benchmark to evaluate the performance of pre-trained models, where the pre-trained encoder is frozen and shared to investigate the capability of representations on various speech tasks, including phoneme recognition (PR), automatic speech recognition (ASR), keyword spotting (KS), query by example spoken term detection (QbE), speaker identification (SID), automatic speaker verification (ASV), speaker diarization (SD), intent classification (IC), slot filling (SF), and emotion recognition (ER). &#10;&#10;Reducing architectural complexity of large-scale pre-trained models has become an indispensable research endeavor \cite{sanh2019,chang2022distilhubert,Aguilar2020,Lai2021,zhang2021you,yu2021unified,Zafrir2021,peng2021shrinking}. &#10;DistilHuBERT \cite{chang2022distilhubert} is proposed to distill hidden representations from HuBERT Base. &#10;It creates a two-layer Transformer while retaining most performance on ten SUPERB tasks. &#10;Unlike the DistilHuBERT model, which creates a single compressed network, we distill a once-for-all network to deploy under various computational resources. &#10;PARP \cite{Lai2021} is a magnitude-based unstructured pruning method that removes insignificant weights of wav2vec 2.0 based on a sparse network on low-resource ASR with the monolingual and cross-lingual transfer.&#10;Unlike PARP, we prune structured groups of weights, therefore, avoiding irregular sparse matrix operations that are hard to accelerate on hardware \cite{gale2019state}.&#10;&#10;Once-for-all approaches \cite{Cai2020} aim to create a weight-sharing supernet and obtain a huge number of architectures for efficient deployment with different resource constraints while maintaining the same level of accuracy as those trained independently \cite{deng2020model,he2021automl,kuutti2020survey,lu2021neural}. Examples can be found in image processing \cite{Cai2020,Li2020}, natural language processing \cite{NEURIPS2020_6f5216f8}, and speech processing \cite{wang2021efficienttdnn}. &#10;Similarly, AutoFormer \cite{Chen2021} utilizes weight entanglement to supernet training and enables Transformer blocks to share weights for their common in each layer. &#10;It allows a large number of subnets in the supernet to be as well-trained as ones trained from scratch.&#10;Unlike AutoFormer, we introduce a two-stage distillation to improve the supernet while utilizing a task-agnostic objective to learn speech representations.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1911.03109" label="1911.03109">
        <attvalues>
          <attvalue for="0" value="Domain Robustness in Neural Machine Translation" />
          <attvalue for="1" value="  Translating text that diverges from the training domain is a key challenge&#10;for machine translation. Domain robustness---the generalization of models to&#10;unseen test domains---is low for both statistical (SMT) and neural machine&#10;translation (NMT). In this paper, we study the performance of SMT and NMT&#10;models on out-of-domain test sets. We find that in unknown domains, SMT and NMT&#10;suffer from very different problems: SMT systems are mostly adequate but not&#10;fluent, while NMT systems are mostly fluent, but not adequate. For NMT, we&#10;identify such hallucinations (translations that are fluent but unrelated to the&#10;source) as a key reason for low domain robustness. To mitigate this problem, we&#10;empirically compare methods that are reported to improve adequacy or in-domain&#10;robustness in terms of their effectiveness at improving domain robustness. In&#10;experiments on German to English OPUS data, and German to Romansh (a&#10;low-resource setting) we find that several methods improve domain robustness.&#10;While those methods do lead to higher BLEU scores overall, they only slightly&#10;increase the adequacy of translations compared to SMT.&#10;" />
          <attvalue for="2" value="&#10;&#10;Even though neural models have improved the state-of-the-art in machine translation considerably in recent years, they still underperform in specific conditions.&#10;One such condition is out-of-domain translation.&#10;\cite{Koehn2017} found that neural machine translation (NMT) systems perform poorly in such settings and that their poor performance cannot be explained solely by the fact that out-of-domain translation is difficult: non-neural, statistical machine translation (SMT) systems were superior at this task.&#10;For this reason, \cite{Koehn2017} identified translation of out-of-domain text as a key challenge for NMT.&#10;&#10;Catastrophic failure to translate out-of-domain text can be viewed as overfitting to the training domain, i.e.\ systems learn idiosyncrasies of the domain rather than more general features.&#10;Our goal is to learn models that generalize well to unseen data distributions, including data from other domains.&#10;We will refer to this property of showing good generalization to unseen domains as domain robustness.&#10;&#10;We consider domain robustness a desirable property of NLP systems, along with other types of robustness, such as robustness against adversarial examples (\cite{Goodfellow2015}) or typos in the input \cite{Belinkov2018}.&#10;While domain adaptation with small amounts of parallel or monolingual in-domain data has proven very effective for NMT \cite[e.g.][]{luong2015,sennrich-haddow-birch:2016:P16-11,R17-1049,li-EtAl:2019:WMT1}, the target domain(s) may be unknown when a system is built, and there are language pairs for which training data is only available for limited domains.&#10;Hence, domain robustness of systems without any domain adaptation is not only of theoretical interest, but also relevant in practice. &#10;&#10;Model architectures and training techniques have evolved since \cite{Koehn2017}'s study, and it is unclear to what extent this problem still persists. We therefore revisit the hypothesis that NMT systems exhibit low domain robustness. In preliminary experiments, we demonstrate that current models still fail at out-of-domain translation: BLEU scores drop drastically for test domains other than the training domain. However, the overall out-of-domain translation quality of NMT systems is now on par with SMT systems. &#10;&#10;An analysis of our baseline systems reveals that hallucinated content occurs frequently in out-of-domain translations (see Figure \ref{fig:hallucinated-example} for an example).&#10;Several authors present anecdotal evidence for NMT systems occasionally falling into a hallucination mode where translations are grammatically correct but unrelated to the source sentence \cite{Arthur2016,Koehn2017,Nguyen2018}.&#10;Our manual evaluation shows that hallucination is more pronounced in out-of-domain translation. We therefore expect methods that alleviate the hallucination problem to indirectly improve domain robustness.&#10;&#10;As a means to reduce hallucination, we experiment with several techniques and assess their effectiveness in improving domain robustness: reconstruction \cite{Tu2017,niu-etal-2019-bi}, subword regularization \cite{Kudo2018}, neural noisy channel models \cite{li2016mutual,yee2019simple}, and defensive distillation \cite{papernot2016distillation}, as well as combinations of these techniques. The main contributions of this paper are:&#10;&#10;\begin{itemize}&#10; \item we perform an analysis of SMT and NMT systems that confirms that while in-domain BLEU increased, domain robustness remains a major problem even with state-of-the-art Transformer architectures. Our comparison of SMT and NMT shows differences in how performance degrades in unseen domains: SMT mostly suffers in terms of fluency, while NMT tends to produce more fluent, but less adequate translations (hallucinations).&#10; \item we test several techniques related to adequacy, robustness, or out-of-domain translation in regard to their effectiveness in improving domain robustness in NMT. We find that several techniques are moderately successful, most notably reconstruction, which reduces the average percentage of hallucinations in out-of-domain test sets from 35\% to 29\%.&#10; \item we show that despite moderate improvements, domain robustness remains a challenge in NMT, and provide code and data sets to serve as baselines for future work.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Domain Robustness, Artificial Intelligence, Neural Translation Models" />
        </attvalues>
      </node>
      <node id="1412.6572" label="1412.6572">
        <attvalues>
          <attvalue for="0" value="Explaining and Harnessing Adversarial Examples" />
          <attvalue for="1" value="  Several machine learning models, including neural networks, consistently&#10;misclassify adversarial examples---inputs formed by applying small but&#10;intentionally worst-case perturbations to examples from the dataset, such that&#10;the perturbed input results in the model outputting an incorrect answer with&#10;high confidence. Early attempts at explaining this phenomenon focused on&#10;nonlinearity and overfitting. We argue instead that the primary cause of neural&#10;networks' vulnerability to adversarial perturbation is their linear nature.&#10;This explanation is supported by new quantitative results while giving the&#10;first explanation of the most intriguing fact about them: their generalization&#10;across architectures and training sets. Moreover, this view yields a simple and&#10;fast method of generating adversarial examples. Using this approach to provide&#10;examples for adversarial training, we reduce the test set error of a maxout&#10;network on the MNIST dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.06140" label="1612.06140">
        <attvalues>
          <attvalue for="0" value="Domain Control for Neural Machine Translation" />
          <attvalue for="1" value="  Machine translation systems are very sensitive to the domains they were&#10;trained on. Several domain adaptation techniques have been deeply studied. We&#10;propose a new technique for neural machine translation (NMT) that we call&#10;domain control which is performed at runtime using a unique neural network&#10;covering multiple domains. The presented approach shows quality improvements&#10;when compared to dedicated domains translating on any of the covered domains&#10;and even on out-of-domain data. In addition, model parameters do not need to be&#10;re-estimated for each domain, making this effective to real use cases.&#10;Evaluation is carried out on English-to-French translation for two different&#10;testing scenarios. We first consider the case where an end-user performs&#10;translations on a known domain. Secondly, we consider the scenario where the&#10;domain is not known and predicted at the sentence level before translating.&#10;Results show consistent accuracy improvements for both conditions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.11943" label="1906.11943">
        <attvalues>
          <attvalue for="0" value="Findings of the First Shared Task on Machine Translation Robustness" />
          <attvalue for="1" value="  We share the findings of the first shared task on improving robustness of&#10;Machine Translation (MT). The task provides a testbed representing challenges&#10;facing MT models deployed in the real world, and facilitates new approaches to&#10;improve models; robustness to noisy input and domain mismatch. We focus on two&#10;language pairs (English-French and English-Japanese), and the submitted systems&#10;are evaluated on a blind test set consisting of noisy comments on Reddit and&#10;professionally sourced translations. As a new task, we received 23 submissions&#10;by 11 participating teams from universities, companies, national labs, etc. All&#10;submitted systems achieved large improvements over baselines, with the best&#10;improvement having +22.33 BLEU. We evaluated submissions by both human judgment&#10;and automatic evaluation (BLEU), which shows high correlations (Pearson's r =&#10;0.94 and 0.95). Furthermore, we conducted a qualitative analysis of the&#10;submitted systems using compare-mt, which revealed their salient differences in&#10;handling challenges in this task. Such analysis provides additional insights&#10;when there is occasional disagreement between human judgment and BLEU, e.g.&#10;systems better at producing colloquial expressions received higher score from&#10;human judgment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.02006" label="1606.02006">
        <attvalues>
          <attvalue for="0" value="Incorporating Discrete Translation Lexicons into Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Neural machine translation (NMT) often makes mistakes in translating&#10;low-frequency content words that are essential to understanding the meaning of&#10;the sentence. We propose a method to alleviate this problem by augmenting NMT&#10;systems with discrete translation lexicons that efficiently encode translations&#10;of these low-frequency words. We describe a method to calculate the lexicon&#10;probability of the next word in the translation candidate by using the&#10;attention vector of the NMT model to select which source word lexical&#10;probabilities the model should focus on. We test two methods to combine this&#10;probability with the standard NMT probability: (1) using it as a bias, and (2)&#10;linear interpolation. Experiments on two corpora show an improvement of 2.0-2.3&#10;BLEU and 0.13-0.44 NIST score, and faster convergence time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.01874" label="1611.01874">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation with Reconstruction" />
          <attvalue for="1" value="  Although end-to-end Neural Machine Translation (NMT) has achieved remarkable&#10;progress in the past two years, it suffers from a major drawback: translations&#10;generated by NMT systems often lack of adequacy. It has been widely observed&#10;that NMT tends to repeatedly translate some source words while mistakenly&#10;ignoring other words. To alleviate this problem, we propose a novel&#10;encoder-decoder-reconstructor framework for NMT. The reconstructor,&#10;incorporated into the NMT model, manages to reconstruct the input source&#10;sentence from the hidden layer of the output target sentence, to ensure that&#10;the information in the source side is transformed to the target side as much as&#10;possible. Experiments show that the proposed framework significantly improves&#10;the adequacy of NMT output and achieves superior translation result over&#10;state-of-the-art NMT and statistical MT systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.01116" label="1811.01116">
        <attvalues>
          <attvalue for="0" value="Bi-Directional Differentiable Input Reconstruction for Low-Resource&#10;  Neural Machine Translation" />
          <attvalue for="1" value="  We aim to better exploit the limited amounts of parallel text available in&#10;low-resource settings by introducing a differentiable reconstruction loss for&#10;neural machine translation (NMT). This loss compares original inputs to&#10;reconstructed inputs, obtained by back-translating translation hypotheses into&#10;the input language. We leverage differentiable sampling and bi-directional NMT&#10;to train models end-to-end, without introducing additional parameters. This&#10;approach achieves small but consistent BLEU improvements on four language pairs&#10;in both translation directions, and outperforms an alternative differentiable&#10;reconstruction strategy based on hidden states.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1601.00372" label="1601.00372">
        <attvalues>
          <attvalue for="0" value="Mutual Information and Diverse Decoding Improve Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Sequence-to-sequence neural translation models learn semantic and syntactic&#10;relations between sentence pairs by optimizing the likelihood of the target&#10;given the source, i.e., $p(y|x)$, an objective that ignores other potentially&#10;useful sources of information. We introduce an alternative objective function&#10;for neural MT that maximizes the mutual information between the source and&#10;target sentences, modeling the bi-directional dependency of sources and&#10;targets. We implement the model with a simple re-ranking method, and also&#10;introduce a decoding algorithm that increases diversity in the N-best list&#10;produced by the first pass. Applied to the WMT German/English and&#10;French/English tasks, the proposed models offers a consistent performance boost&#10;on both standard LSTM and attention-based neural MT architectures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.17238" label="2210.17238">
        <attvalues>
          <attvalue for="0" value="Pneg: Prompt-based Negative Response Generation for Dialogue Response&#10;  Selection Task" />
          <attvalue for="1" value="  In retrieval-based dialogue systems, a response selection model acts as a&#10;ranker to select the most appropriate response among several candidates.&#10;However, such selection models tend to rely on context-response content&#10;similarity, which makes models vulnerable to adversarial responses that are&#10;semantically similar but not relevant to the dialogue context. Recent studies&#10;have shown that leveraging these adversarial responses as negative training&#10;samples is useful for improving the discriminating power of the selection&#10;model. Nevertheless, collecting human-written adversarial responses is&#10;expensive, and existing synthesizing methods often have limited scalability. To&#10;overcome these limitations, this paper proposes a simple but efficient method&#10;for generating adversarial negative responses leveraging a large-scale language&#10;model. Experimental results on dialogue selection tasks show that our method&#10;outperforms other methods of synthesizing adversarial negative responses. These&#10;results suggest that our method can be an effective alternative to human&#10;annotators in generating adversarial responses. Our dataset and generation code&#10;is available at https://github.com/leenw23/generating-negatives-by-gpt3.&#10;" />
          <attvalue for="2" value="&#10;&#10;In retrieval-based dialogue systems, the response selection model aims to predict the most appropriate response among multiple candidates retrieved for a given conversation context~\cite{zhou-etal-2018-multi, wu2019sequential, chen-etal-2022-contextual}. The selection model is generally trained to distinguish a related (positive) response from randomly sampled negative responses on training datasets, but such a model generally poses the following problems.&#10;&#10;First, randomly selected negatives are often too easy to distinguish because they are totally irrelevant to the dialogue context~\cite{li-etal-2019-sampling, lin-etal-2020-world}. In this case, the model is more likely to predict the response only by relying on the superficial content similarity of the context-response pairs~\cite{yuan-etal-2019-multi, sai-etal-2020-improving, whang2021response}. These models are vulnerable to adversarial responses with high content similarity to the dialogue context, and fail to distinguish subtle differences in various contexts in real-world scenarios. Second, random sampling causes sampling bias, such as containing false negatives for a given dialogue context~\cite{zhou-etal-2022-debiased}. These biases inherent in the training datasets hinder the accurate prediction of the selection model, resulting in performance degradation.&#10;&#10;To mitigate this problem, recent studies have proposed various methods to synthesize and leverage adversarial negative training samples so that the selection model can learn features beyond content similarity~\cite{srivastava2020robustness, kaushik2021learning}. However, existing methods for synthesizing adversarial negative responses~\cite{ebrahimi-etal-2018-hotflip, alzantot-etal-2018-generating, zhang-etal-2019-generating-fluent, qiu2021challenging, gupta-etal-2021-synthesizing} still have limitations in creating human-like responses. The most reliable method is to collect human-written adversarial negatives~\cite{sai-etal-2020-improving}, but it is not scalable because it is expensive and time consuming.&#10;&#10;To overcome these limitations, we note that large-scale language models such as GPT3 can be utilized as a low-cost data labeler~\cite{wang-etal-2021-want-reduce}. In this paper, we present Pneg, a Prompt-based NEgative response Generation method leveraging a large-scale language model (Figure~\ref{fig:method}). In P-NEG, we can cheaply collect human-like adversarial negatives by using an in-context learning-based data augmentation method with human-written samples as demonstration examples.&#10;&#10;Experimental results on the dialogue response selection task show that the selection model that trained the negative samples from Pneg has better performance than other baselines. We then conduct quality evaluation and ablation studies to analyze the validity of Pneg. Consequently, we argue that Pneg can be an efficient alternative to human annotators in generating adversarial responses.&#10;&#10;Our contributions are as follows:&#10;\begin{itemize}&#10; \item We propose Pneg, a Prompt-based NEgative response Generation method. &#10; \item Our method can generate high-quality adversarial negative responses only with a few human-written examples.&#10; \item We show that our method outperforms other baselines across multiple model architectures on the response selection task.&#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Cognitive Science, Dialogue Systems, Adversarial Response, Language Modeling" />
        </attvalues>
      </node>
      <node id="2007.06661" label="2007.06661">
        <attvalues>
          <attvalue for="0" value="Robustness to Spurious Correlations via Human Annotations" />
          <attvalue for="1" value="  The reliability of machine learning systems critically assumes that the&#10;associations between features and labels remain similar between training and&#10;test distributions. However, unmeasured variables, such as confounders, break&#10;this assumption---useful correlations between features and labels at training&#10;time can become useless or even harmful at test time. For example, high obesity&#10;is generally predictive for heart disease, but this relation may not hold for&#10;smokers who generally have lower rates of obesity and higher rates of heart&#10;disease. We present a framework for making models robust to spurious&#10;correlations by leveraging humans' common sense knowledge of causality.&#10;Specifically, we use human annotation to augment each training example with a&#10;potential unmeasured variable (i.e. an underweight patient with heart disease&#10;may be a smoker), reducing the problem to a covariate shift problem. We then&#10;introduce a new distributionally robust optimization objective over unmeasured&#10;variables (UV-DRO) to control the worst-case loss over possible test-time&#10;shifts. Empirically, we show improvements of 5-10% on a digit recognition task&#10;confounded by rotation, and 1.5-5% on the task of analyzing NYPD Police Stops&#10;confounded by location.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.11321" label="2009.11321">
        <attvalues>
          <attvalue for="0" value="Improving Dialog Evaluation with a Multi-reference Adversarial Dataset&#10;  and Large Scale Pretraining" />
          <attvalue for="1" value="  There is an increasing focus on model-based dialog evaluation metrics such as&#10;ADEM, RUBER, and the more recent BERT-based metrics. These models aim to assign&#10;a high score to all relevant responses and a low score to all irrelevant&#10;responses. Ideally, such models should be trained using multiple relevant and&#10;irrelevant responses for any given context. However, no such data is publicly&#10;available, and hence existing models are usually trained using a single&#10;relevant response and multiple randomly selected responses from other contexts&#10;(random negatives). To allow for better training and robust evaluation of&#10;model-based metrics, we introduce the DailyDialog++ dataset, consisting of (i)&#10;five relevant responses for each context and (ii) five adversarially crafted&#10;irrelevant responses for each context. Using this dataset, we first show that&#10;even in the presence of multiple correct references, n-gram based metrics and&#10;embedding based metrics do not perform well at separating relevant responses&#10;from even random negatives. While model-based metrics perform better than&#10;n-gram and embedding based metrics on random negatives, their performance drops&#10;substantially when evaluated on adversarial examples. To check if large scale&#10;pretraining could help, we propose a new BERT-based evaluation metric called&#10;DEB, which is pretrained on 727M Reddit conversations and then finetuned on our&#10;dataset. DEB significantly outperforms existing models, showing better&#10;correlation with human judgements and better performance on random negatives&#10;(88.27% accuracy). However, its performance again drops substantially, when&#10;evaluated on adversarial responses, thereby highlighting that even large-scale&#10;pretrained evaluation models are not robust to the adversarial examples in our&#10;dataset. The dataset and code are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.08700" label="2005.08700">
        <attvalues>
          <attvalue for="0" value="Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict" />
          <attvalue for="1" value="  We present Mask CTC, a novel non-autoregressive end-to-end automatic speech&#10;recognition (ASR) framework, which generates a sequence by refining outputs of&#10;the connectionist temporal classification (CTC). Neural sequence-to-sequence&#10;models are usually \textit{autoregressive}: each output token is generated by&#10;conditioning on previously generated tokens, at the cost of requiring as many&#10;iterations as the output length. On the other hand, non-autoregressive models&#10;can simultaneously generate tokens within a constant number of iterations,&#10;which results in significant inference time reduction and better suits&#10;end-to-end ASR model for real-world scenarios. In this work, Mask CTC model is&#10;trained using a Transformer encoder-decoder with joint training of mask&#10;prediction and CTC. During inference, the target sequence is initialized with&#10;the greedy CTC outputs and low-confidence tokens are masked based on the CTC&#10;probabilities. Based on the conditional dependence between output tokens, these&#10;masked low-confidence tokens are then predicted conditioning on the&#10;high-confidence tokens. Experimental results on different speech recognition&#10;tasks show that Mask CTC outperforms the standard CTC model (e.g., 17.9% -&gt;&#10;12.1% WER on WSJ) and approaches the autoregressive model, requiring much less&#10;inference time using CPUs (0.07 RTF in Python implementation). All of our codes&#10;will be publicly available.&#10;" />
          <attvalue for="2" value="&#10;Owing to the rapid development of neural sequence-to-sequence modeling~\cite{sutskever2014sequence, bahdanau2014neural}, &#10;deep neural network (DNN)-based end-to-end automatic speech recognition (ASR) systems have become almost as effective as the traditional hidden Markov model-based systems~\cite{chiu2018state, luscher2019rwth, karita2019a}.&#10;Various models and approaches have been proposed for improving the performance of the autoregressive (AR) end-to-end ASR model with the encoder-decoder architecture based on recurrent neural networks (RNNs)~\cite{chorowski2015attention, chan2016listen, kim2017joint} and Transformers~\cite{vaswani2017attention, dong2018speech, karita2019improving}.&#10;&#10;Contrary to the autoregressive framework, &#10;non-autoregressive (NAR) sequence generation has attracted attention, including the revisitation of connectionist temporal classification (CTC)~\cite{graves2006connectionist, libovicky2018end} and the growing interest for non-autoregressive Transformer (NAT)~\cite{gu2017non}.&#10;While the autoregressive model requires $L$ iterations to generate an $L$-length target sequence, a non-autoregressive model costs a constant number of iterations $K (\ll L)$, independent on the length of the target sequence.&#10;Despite the limitation in this decoding iteration, some recent studies in neural machine translation have successfully shown the effectiveness of the non-autoregressive models, &#10;performing comparable results to the autoregressive models.&#10;Different types of non-autoregressive models have been proposed based on &#10;the iterative refinement decoding~\cite{lee2018deterministic}, &#10;insert or edit-based sequence generation~\cite{stern2019insertion, gu2019levenshtein}, &#10;masked language model objective~\cite{ghazvininejad2019mask, ghazvininejad2020semi, saharia2020non}, and&#10;generative flow~\cite{ma2019flowseq}.&#10;&#10;Some attempts have also been made to realize the non-autoregressive model in speech recognition.&#10;CTC introduces a frame-wise latent alignment to represent the alignment between the input speech frames and the output tokens~\cite{graves2014towards}.&#10;While CTC makes use of dynamic programming to efficiently calculate the most probable alignment, &#10;the strong conditional independence assumption between output tokens results in poor performance compared to the autoregressive models~\cite{battenberg2017exploring}.&#10;On the other hand, \cite{chen2019non} trains a Transformer encoder-decoder in a mask-predict manner~\cite{ghazvininejad2019mask}: &#10;target tokens are randomly masked and predicted conditioning on the unmasked tokens and the input speech.&#10;To generate the output sequence in parallel during inference, &#10;the target sequence is initialized as all masked tokens and the output length is predicted by finding the position of the end-of-sequence token.&#10;However with this prediction of the output length, &#10;the model is known to be vulnerable to the output sequence with a long length. &#10;At the beginning of the decoding, &#10;the model is likely to make more mistakes in predicting long masked sequence,&#10;propagating the error to the later decoding steps.&#10;\cite{chan2020imputer} proposes Imputer, which performs the mask prediction in CTC's latent alignments to get rid of the output length prediction.&#10;However, unlike the mask-predict, Imputer requires more calculations in each interaction, which is proportional to the square of the input length $T (\gg L)$ in the self-attention layer, and the total computational cost can be very large.&#10;&#10;Our work aims to obtain a non-autoregressive end-to-end ASR model, &#10;which generates the sequence in token-level with low computational cost.&#10;The proposed Mask CTC framework trains a Transformer encoder-decoder model with both CTC and mask-predict objectives. &#10;During inference, &#10;the target sequence is initialized with the greedy CTC outputs and low-confidence tokens are masked based on the CTC probabilities.&#10;The masked low-confidence tokens are predicted conditioning on the high-confidence tokens not only in the past but also in the future context.&#10;The advantages of Mask CTC are summarized as follows.&#10;&#10;No requirement for output length prediction: &#10;Predicting the output token length from input speech is rather challenging &#10;because the length of the input utterances varies greatly depending on the speaking rate or the duration of silence.&#10;By initializing the target sequence with the CTC outputs, &#10;Mask CTC does not have to care about predicting the output length at the beginning of the decoding.&#10;&#10;Accurate and fast decoding: &#10;We observed that the results of CTC outputs themselves are quite accurate.&#10;Mask CTC does not only retain the correct tokens in the CTC outputs but also recovers the output errors by considering the entire context.&#10;Token-level iterative decoding with a small number of masks makes the model well-suited for &#10;the usage in real scenarios.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, End-to-End Deep Learning, Automatic Speech Recognition, Signal Processing, Neural Sequence Models" />
        </attvalues>
      </node>
      <node id="1909.02480" label="1909.02480">
        <attvalues>
          <attvalue for="0" value="FlowSeq: Non-Autoregressive Conditional Sequence Generation with&#10;  Generative Flow" />
          <attvalue for="1" value="  Most sequence-to-sequence (seq2seq) models are autoregressive; they generate&#10;each token by conditioning on previously generated tokens. In contrast,&#10;non-autoregressive seq2seq models generate all tokens in one pass, which leads&#10;to increased efficiency through parallel processing on hardware such as GPUs.&#10;However, directly modeling the joint distribution of all tokens simultaneously&#10;is challenging, and even with increasingly complex model structures accuracy&#10;lags significantly behind autoregressive models. In this paper, we propose a&#10;simple, efficient, and effective model for non-autoregressive sequence&#10;generation using latent variable models. Specifically, we turn to generative&#10;flow, an elegant technique to model complex distributions using neural&#10;networks, and design several layers of flow tailored for modeling the&#10;conditional density of sequential latent variables. We evaluate this model on&#10;three neural machine translation (NMT) benchmark datasets, achieving comparable&#10;performance with state-of-the-art non-autoregressive NMT models and almost&#10;constant decoding time w.r.t the sequence length.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.08926" label="2002.08926">
        <attvalues>
          <attvalue for="0" value="Imputer: Sequence Modelling via Imputation and Dynamic Programming" />
          <attvalue for="1" value="  This paper presents the Imputer, a neural sequence model that generates&#10;output sequences iteratively via imputations. The Imputer is an iterative&#10;generative model, requiring only a constant number of generation steps&#10;independent of the number of input or output tokens. The Imputer can be trained&#10;to approximately marginalize over all possible alignments between the input and&#10;output sequences, and all possible generation orders. We present a tractable&#10;dynamic programming training algorithm, which yields a lower bound on the log&#10;marginal likelihood. When applied to end-to-end speech recognition, the Imputer&#10;outperforms prior non-autoregressive models and achieves competitive results to&#10;autoregressive models. On LibriSpeech test-other, the Imputer achieves 11.1&#10;WER, outperforming CTC at 13.0 WER and seq2seq at 12.5 WER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.10229" label="2106.10229">
        <attvalues>
          <attvalue for="0" value="A learned conditional prior for the VAE acoustic space of a TTS system" />
          <attvalue for="1" value="  Many factors influence speech yielding different renditions of a given&#10;sentence. Generative models, such as variational autoencoders (VAEs), capture&#10;this variability and allow multiple renditions of the same sentence via&#10;sampling. The degree of prosodic variability depends heavily on the prior that&#10;is used when sampling. In this paper, we propose a novel method to compute an&#10;informative prior for the VAE latent space of a neural text-to-speech (TTS)&#10;system. By doing so, we aim to sample with more prosodic variability, while&#10;gaining controllability over the latent space's structure.&#10;  By using as prior the posterior distribution of a secondary VAE, which we&#10;condition on a speaker vector, we can sample from the primary VAE taking&#10;explicitly the conditioning into account and resulting in samples from a&#10;specific region of the latent space for each condition (i.e. speaker). A formal&#10;preference test demonstrates significant preference of the proposed approach&#10;over standard Conditional VAE. We also provide visualisations of the latent&#10;space where well-separated condition-specific clusters appear, as well as&#10;ablation studies to better understand the behaviour of the system.&#10;" />
          <attvalue for="2" value="&#10;&#10;Numerous factors, such as linguistic content, speaking style, dialect, speaker identity, emotional state and environment, influence speech and make variability one of its inherent characteristics. The presence of one or many of these factors results in different renditions of a given sentence. However, typical TTS systems are unable to produce multiple renditions of the same sentence. That is why recently probabilistic generative models, such as VAEs, have been used to learn a latent representation that captures the variability of speech \cite{ChengHsu2016, Hsu2017, Akuzawa2018, Zhang2019}. This latent space captures a distribution over multiple renditions and, thus, sampling from it can produce varied speech.&#10;&#10;In the standard approach, sampling is still done from the VAE latent space by using a standard Gaussian prior. Such a prior, even if it still represents natural prosody, loses some of the variability of real speech, and has been shown to lead to over-regularization and poor latent representations \cite{Dilokthanakul2016, Bowman2016}. In addition, one has no controllability on the part of the latent space from which sampling is achieved. &#10;&#10;In this paper, we introduce a more informative prior that handles the aforementioned caveats. This prior is conditioned on a specific factor that influences speech and introduces condition-specific clusters to the VAE space. In that way, the variability induced by a given condition is kept in the latent space. Then, by providing the desired conditioning during inference, we guarantee sampling from a specific part of the latent space which captures the condition identity and prosodic variability. In other words, we achieve controllability of the generated speech thanks to the disentanglement that results from using a conditional prior.&#10;&#10;More precisely, we suggest to adopt a learned conditional posterior as prior, following the approach of \cite{Aliakbarian2019}. We introduce a hierarchical VAE structure with a secondary VAE which takes as input a condition influencing speech, in our case a speaker vector, and learns the distribution of this conditioning signal. This can also be seen as generating TTS-specific speaker embeddings that are trained jointly with the rest of the system. The learned posterior distribution then acts as the prior of the primary encoder. This is a novel method to use a learned conditional prior in TTS. &#10;&#10;The typical approach to add a condition to a VAE is the Conditional VAE (CVAE) \cite{Akuzawa2018, Skerry-Ryan2018}. In CVAE, the condition is concatenated to the VAE encoder and decoder inputs. This maintains the identity of the condition, while sampling is done in practice from a standard Gaussian prior. In other words, although the encoder and decoder are conditional, the latent space itself, as expressed by the prior, isn't. In our approach, in contrast, sampling is done from a learned conditional prior. This gives us controllability over the sampling from a structured latent space. In addition, using a more informative prior generates samples with more prosodic variability. &#10;&#10;There has also been work recently on learning more potent latent representations by specifying more complex priors over the VAE latent space. VampPrior \cite{Tomczak2018, Hodari2020} is an example of such approaches where the proposed prior is a mixture of variational posteriors conditioned on learnable pseudo-data. This is close to our approach from the perspective that the parameters of the prior are learned jointly with the rest of the model. In our case, however, the input is the actual conditioning, so there is no need to define pseudo-units which is not always an obvious task. Another related work is the use of Gaussian mixture as a latent prior \cite{Hsu2019} where there is a different speech factor represented in each mixture component. This work focuses on the disentanglement of different factors that influence the latent space.&#10;&#10;In controllable TTS literature, in \cite{Wang2018} the authors introduce the ``global style tokens'' (GST), style embeddings that condition the text encoder and are learned jointly with Tacotron \cite{Wang2017}. This architecture is not variational though, thus with no direct way to sample utterances of varying prosody. Other works adopt a variational method and condition the VAE on contextual linguistic information, either directly \cite{Tyagi2020} or via training a prediction model that is then applied to sampling \cite{Karlapati2020, Hodari2021}. These works are related to ours from the perspective of learning TTS-specific embeddings jointly with the rest of the system. &#10;&#10;Another direction of gaining controllability and interpretability on the latent space is by introducing a hierarchical linguistic structure \cite{Sun2020, Hono2020}, where these fine-grained latent variables can control different speech factors. Our method can be seen as complementary to these approaches, with the possibility to build a hierarchical structure on top of it. &#10;&#10;The main contribution of this paper is that it introduces a more informative prior to the acoustic latent space, jointly trained with the rest of the system in a hierarchical VAE architecture. Using this prior adds to the expressivity of the generated samples compared to a CVAE baseline system, while maintaining the input condition. It also provides controllability on the sampling from a structured VAE latent space. While this method is successfully tested on a multi-speaker model, we hypothesize that such a prior could be learned on any condition that influences speech, e.g. language, style, emotion, etc.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Generative Model Optimization, Signal Processing, Mathematics, Speech Variability Modeling, Neural Text-to-Speech Systems" />
        </attvalues>
      </node>
      <node id="1610.04019" label="1610.04019">
        <attvalues>
          <attvalue for="0" value="Voice Conversion from Non-parallel Corpora Using Variational&#10;  Auto-encoder" />
          <attvalue for="1" value="  We propose a flexible framework for spectral conversion (SC) that facilitates&#10;training with unaligned corpora. Many SC frameworks require parallel corpora,&#10;phonetic alignments, or explicit frame-wise correspondence for learning&#10;conversion functions or for synthesizing a target spectrum with the aid of&#10;alignments. However, these requirements gravely limit the scope of practical&#10;applications of SC due to scarcity or even unavailability of parallel corpora.&#10;We propose an SC framework based on variational auto-encoder which enables us&#10;to exploit non-parallel corpora. The framework comprises an encoder that learns&#10;speaker-independent phonetic representations and a decoder that learns to&#10;reconstruct the designated speaker. It removes the requirement of parallel&#10;corpora or phonetic alignments to train a spectral conversion system. We report&#10;objective and subjective evaluations to validate our proposed method and&#10;compare it to SC methods that have access to aligned corpora.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.02648" label="1611.02648">
        <attvalues>
          <attvalue for="0" value="Deep Unsupervised Clustering with Gaussian Mixture Variational&#10;  Autoencoders" />
          <attvalue for="1" value="  We study a variant of the variational autoencoder model (VAE) with a Gaussian&#10;mixture as a prior distribution, with the goal of performing unsupervised&#10;clustering through deep generative models. We observe that the known problem of&#10;over-regularisation that has been shown to arise in regular VAEs also manifests&#10;itself in our model and leads to cluster degeneracy. We show that a heuristic&#10;called minimum information constraint that has been shown to mitigate this&#10;effect in VAEs can also be applied to improve unsupervised clustering&#10;performance with our model. Furthermore we analyse the effect of this heuristic&#10;and provide an intuition of the various processes with the help of&#10;visualizations. Finally, we demonstrate the performance of our model on&#10;synthetic data, MNIST and SVHN, showing that the obtained clusters are&#10;distinct, interpretable and result in achieving competitive performance on&#10;unsupervised clustering to the state-of-the-art results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.08521" label="1912.08521">
        <attvalues>
          <attvalue for="0" value="Contextually Plausible and Diverse 3D Human Motion Prediction" />
          <attvalue for="1" value="  We tackle the task of diverse 3D human motion prediction, that is,&#10;forecasting multiple plausible future 3D poses given a sequence of observed 3D&#10;poses. In this context, a popular approach consists of using a Conditional&#10;Variational Autoencoder (CVAE). However, existing approaches that do so either&#10;fail to capture the diversity in human motion, or generate diverse but&#10;semantically implausible continuations of the observed motion. In this paper,&#10;we address both of these problems by developing a new variational framework&#10;that accounts for both diversity and context of the generated future motion. To&#10;this end, and in contrast to existing approaches, we condition the sampling of&#10;the latent variable that acts as source of diversity on the representation of&#10;the past observation, thus encouraging it to carry relevant information. Our&#10;experiments demonstrate that our approach yields motions not only of higher&#10;quality while retaining diversity, but also that preserve the contextual&#10;information contained in the observed 3D pose sequence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.02135" label="1804.02135">
        <attvalues>
          <attvalue for="0" value="Expressive Speech Synthesis via Modeling Expressions with Variational&#10;  Autoencoder" />
          <attvalue for="1" value="  Recent advances in neural autoregressive models have improve the performance&#10;of speech synthesis (SS). However, as they lack the ability to model global&#10;characteristics of speech (such as speaker individualities or speaking styles),&#10;particularly when these characteristics have not been labeled, making neural&#10;autoregressive SS systems more expressive is still an open issue. In this&#10;paper, we propose to combine VoiceLoop, an autoregressive SS model, with&#10;Variational Autoencoder (VAE). This approach, unlike traditional autoregressive&#10;SS systems, uses VAE to model the global characteristics explicitly, enabling&#10;the expressiveness of the synthesized speech to be controlled in an&#10;unsupervised manner. Experiments using the VCTK and Blizzard2012 datasets show&#10;the VAE helps VoiceLoop to generate higher quality speech and to control the&#10;expressions in its synthesized speech by incorporating global characteristics&#10;into the speech generating process.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.07120" label="1705.07120">
        <attvalues>
          <attvalue for="0" value="VAE with a VampPrior" />
          <attvalue for="1" value="  Many different methods to train deep generative models have been introduced&#10;in the past. In this paper, we propose to extend the variational auto-encoder&#10;(VAE) framework with a new type of prior which we call &quot;Variational Mixture of&#10;Posteriors&quot; prior, or VampPrior for short. The VampPrior consists of a mixture&#10;distribution (e.g., a mixture of Gaussians) with components given by&#10;variational posteriors conditioned on learnable pseudo-inputs. We further&#10;extend this prior to a two layer hierarchical model and show that this&#10;architecture with a coupled prior and posterior, learns significantly better&#10;models. The model also avoids the usual local optima issues related to useless&#10;latent dimensions that plague VAEs. We provide empirical studies on six&#10;datasets, namely, static and binary MNIST, OMNIGLOT, Caltech 101 Silhouettes,&#10;Frey Faces and Histopathology patches, and show that applying the hierarchical&#10;VampPrior delivers state-of-the-art results on all datasets in the unsupervised&#10;permutation invariant setting and the best results or comparable to SOTA&#10;methods for the approach with convolutional networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.00955" label="1912.00955">
        <attvalues>
          <attvalue for="0" value="Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven&#10;  Acoustic Embedding Selection" />
          <attvalue for="1" value="  Recent advances in Text-to-Speech (TTS) have improved quality and naturalness&#10;to near-human capabilities when considering isolated sentences. But something&#10;which is still lacking in order to achieve human-like communication is the&#10;dynamic variations and adaptability of human speech. This work attempts to&#10;solve the problem of achieving a more dynamic and natural intonation in TTS&#10;systems, particularly for stylistic speech such as the newscaster speaking&#10;style. We propose a novel embedding selection approach which exploits&#10;linguistic information, leveraging the speech variability present in the&#10;training dataset. We analyze the contribution of both semantic and syntactic&#10;features. Our results show that the approach improves the prosody and&#10;naturalness for complex utterances as well as in Long Form Reading (LFR).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.02252" label="2011.02252">
        <attvalues>
          <attvalue for="0" value="Prosodic Representation Learning and Contextual Sampling for Neural&#10;  Text-to-Speech" />
          <attvalue for="1" value="  In this paper, we introduce Kathaka, a model trained with a novel two-stage&#10;training process for neural speech synthesis with contextually appropriate&#10;prosody. In Stage I, we learn a prosodic distribution at the sentence level&#10;from mel-spectrograms available during training. In Stage II, we propose a&#10;novel method to sample from this learnt prosodic distribution using the&#10;contextual information available in text. To do this, we use BERT on text, and&#10;graph-attention networks on parse trees extracted from text. We show a&#10;statistically significant relative improvement of $13.2\%$ in naturalness over&#10;a strong baseline when compared to recordings. We also conduct an ablation&#10;study on variations of our sampling technique, and show a statistically&#10;significant improvement over the baseline in each case.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.12403" label="2210.12403">
        <attvalues>
          <attvalue for="0" value="PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models" />
          <attvalue for="1" value="  A wide range of NLP tasks benefit from the fine-tuning of pretrained language&#10;models (PLMs). However, a number of redundant parameters which contribute less&#10;to the downstream task are observed in a directly fine-tuned model. We consider&#10;the gap between pretraining and downstream tasks hinders the training of these&#10;redundant parameters, and results in a suboptimal performance of the overall&#10;model. In this paper, we present PATS (Perturbation According To Sensitivity),&#10;a noisy training mechanism which considers each parameter's importance in the&#10;downstream task to help fine-tune PLMs. The main idea of PATS is to add bigger&#10;noise to parameters with lower sensitivity and vice versa, in order to activate&#10;more parameters' contributions to downstream tasks without affecting the&#10;sensitive ones much. Extensive experiments conducted on different tasks of the&#10;GLUE benchmark show PATS can consistently empower the fine-tuning of different&#10;sizes of PLMs, and the parameters in the well-performing models always have&#10;more concentrated distributions of sensitivities, which experimentally proves&#10;the effectiveness of our method.&#10;" />
          <attvalue for="2" value="&#10;&#10;With a huge number of model parameters and well designed training objectives, pretrained language models (PLMs) have brought a new era to NLP \cite{bertnb1,bertnb2,bertnb3,bertnb4}. Fine-tuning PLMs such as BERT \cite{bert} has become a basic and effective way in many downstream tasks \cite{finetune1, finetune2, finetune3}. &#10;&#10;However, recent study has shown that aggressive fine-tuning can induce an unstable and suboptimal performance of the models especially with insufficient data \cite{bad1, bad2}, which attracts some researchers to figure out the culprits and explore effective methods to solve them \cite{bettertune1, bettertune2, bettertune3}. For example, there are some regularization methods like RecAdam \cite{Recadam} and Mixout \cite{mixout}, and adversarial training techniques like SMART \cite{smart} and FreeLB \cite{FreeLB} to alleviate the overfitting of data in downstream tasks;&#10;Beyond that, \cite{noisytune} proposed NoisyTune with the argument that in addition to the overfitting of the limited downstream data, there could also exist overfitting in pretraining tasks, which could result in enormous gaps between pretraining and downstream task data. &#10;In order to overcome the gaps, NoisyTune simply adds &#10;some noise to parameters in the PLM before fine-tuning.&#10;Besides, it has also been demonstrated that the existence of a large number of redundant parameters could also be a factor in the suboptimal performances of aggressively fine-tuned PLMs \cite{redundancy1, redundancy2, redundancy3}.&#10;Considering the redundant parameters in a model are not insufficiently trained, \cite{sage} proposed a learning rate scheduler named SAGE in which larger learning rates are assigned to these parameters of low sensitivity (a measure of parameter's importance to downstream tasks). &#10;&#10;There could be some connection between the gaps caused by overfitting of pretraining tasks and the redundancy of parameters. &#10;We consider it could be the gaps between pretraining and downstream tasks that hinder the training of these redundant parameters. &#10;SAGE enlarges the learning rates of insensitive parameters to help their training. However, with the sensitivity measurement considered, the insensitive parameters usually have smaller gradients, so enlarged learning rates may help them little to escape the sub-optimal areas compared to involving additional noise. &#10;One noisy training method to alleviate the gaps is NoisyTune, in which parameters of a matrix in a PLM are added with noise according to the standard deviation of the matrix before fine-tuning.&#10;Nevertheless, there are few explanations about why or whether the parameters in the same matrix should be perturbed with the same intensity. Considering different parameters have different contributions to the model, noise from a unified distribution may disturb knowledge of some sensitive parameters, resulting in a loss of performance. Besides,&#10;since each task needs to capture an appropriate textual pattern and the data of it usually comes from a special domain,&#10;different downstream tasks could have different kinds of gaps with those of the pretraining. So the noise added to overcome the gaps should also be related to the downstream task data.&#10;&#10;In this paper, we propose a novel parameter-wise noisy fine-tuning method called PATS (Perturbation According To Sensitivity) to make full use of perturbation on parameters to handle the problems above.&#10;We focus on balancing the contributions of all parameters in the model by activating the insensitive ones to play better roles in downstream tasks. So the main idea of our method is adding different intensities of noise to parameters according to their sensitivity when fine-tuning PLMs, different from NoisyTune (Fig. \ref{fig:methods} (b)) in which noise added to a matrix of parameters is from a unified distribution and unrelated to downstream task data. Specifically, during fine-tuning in PATS (Fig. \ref{fig:methods} (c)),&#10;larger noise will be added to the parameters with lower sensitivity (such as the parameter shown in red), while sensitive parameters (such as the parameter shown in purple) will be barely perturbed. &#10;&#10;Our contributions can be summarized as follows: 1) We propose a simple but effective method to help all parameters be trained sufficiently when fine-tuning PLMs in downstream tasks. 2) Among all the training methods with noise, PATS is the first sensitivity-aware one which perturbs models with noise of different distributions according to parameters' sensitivity, to the best of our knowledge. 3) Extensive experiments on the GLUE benchmark show PATS makes a difference in boosting the performance of PLMs in downstream NLP tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Noisy Training Mechanisms, Artificial Intelligence, Mathematics, Language Model Fine-tuning, Parameter Sensitivity Analysis" />
        </attvalues>
      </node>
      <node id="2002.06823" label="2002.06823">
        <attvalues>
          <attvalue for="0" value="Incorporating BERT into Neural Machine Translation" />
          <attvalue for="1" value="  The recently proposed BERT has shown great power on a variety of natural&#10;language understanding tasks, such as text classification, reading&#10;comprehension, etc. However, how to effectively apply BERT to neural machine&#10;translation (NMT) lacks enough exploration. While BERT is more commonly used as&#10;fine-tuning instead of contextual embedding for downstream language&#10;understanding tasks, in NMT, our preliminary exploration of using BERT as&#10;contextual embedding is better than using for fine-tuning. This motivates us to&#10;think how to better leverage BERT for NMT along this direction. We propose a&#10;new algorithm named BERT-fused model, in which we first use BERT to extract&#10;representations for an input sequence, and then the representations are fused&#10;with each layer of the encoder and decoder of the NMT model through attention&#10;mechanisms. We conduct experiments on supervised (including sentence-level and&#10;document-level translations), semi-supervised and unsupervised machine&#10;translation, and achieve state-of-the-art results on seven benchmark datasets.&#10;Our code is available at \url{https://github.com/bert-nmt/bert-nmt}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.11764" label="1909.11764">
        <attvalues>
          <attvalue for="0" value="FreeLB: Enhanced Adversarial Training for Natural Language Understanding" />
          <attvalue for="1" value="  Adversarial training, which minimizes the maximal risk for label-preserving&#10;input perturbations, has proved to be effective for improving the&#10;generalization of language models. In this work, we propose a novel adversarial&#10;training algorithm, FreeLB, that promotes higher invariance in the embedding&#10;space, by adding adversarial perturbations to word embeddings and minimizing&#10;the resultant adversarial risk inside different regions around input samples.&#10;To validate the effectiveness of the proposed approach, we apply it to&#10;Transformer-based models for natural language understanding and commonsense&#10;reasoning tasks. Experiments on the GLUE benchmark show that when applied only&#10;to the finetuning stage, it is able to improve the overall test scores of&#10;BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8.&#10;In addition, the proposed approach achieves state-of-the-art single-model test&#10;accuracies of 85.44\% and 67.75\% on ARC-Easy and ARC-Challenge. Experiments on&#10;CommonsenseQA benchmark further demonstrate that FreeLB can be generalized and&#10;boost the performance of RoBERTa-large model on other tasks as well. Code is&#10;available at \url{https://github.com/zhuchen03/FreeLB .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.02664" label="2202.02664">
        <attvalues>
          <attvalue for="0" value="No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for&#10;  Training Large Transformer Models" />
          <attvalue for="1" value="  Recent research has shown the existence of significant redundancy in large&#10;Transformer models. One can prune the redundant parameters without&#10;significantly sacrificing the generalization performance. However, we question&#10;whether the redundant parameters could have contributed more if they were&#10;properly trained. To answer this question, we propose a novel training strategy&#10;that encourages all parameters to be trained sufficiently. Specifically, we&#10;adaptively adjust the learning rate for each parameter according to its&#10;sensitivity, a robust gradient-based measure reflecting this parameter's&#10;contribution to the model performance. A parameter with low sensitivity is&#10;redundant, and we improve its fitting by increasing its learning rate. In&#10;contrast, a parameter with high sensitivity is well-trained, and we regularize&#10;it by decreasing its learning rate to prevent further overfitting. We conduct&#10;extensive experiments on natural language understanding, neural machine&#10;translation, and image classification to demonstrate the effectiveness of the&#10;proposed schedule. Analysis shows that the proposed schedule indeed reduces the&#10;redundancy and improves generalization performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1507.08240" label="1507.08240">
        <attvalues>
          <attvalue for="0" value="EESEN: End-to-End Speech Recognition using Deep RNN Models and&#10;  WFST-based Decoding" />
          <attvalue for="1" value="  The performance of automatic speech recognition (ASR) has improved&#10;tremendously due to the application of deep neural networks (DNNs). Despite&#10;this progress, building a new ASR system remains a challenging task, requiring&#10;various resources, multiple training stages and significant expertise. This&#10;paper presents our Eesen framework which drastically simplifies the existing&#10;pipeline to build state-of-the-art ASR systems. Acoustic modeling in Eesen&#10;involves learning a single recurrent neural network (RNN) predicting&#10;context-independent targets (phonemes or characters). To remove the need for&#10;pre-generated frame labels, we adopt the connectionist temporal classification&#10;(CTC) objective function to infer the alignments between speech and label&#10;sequences. A distinctive feature of Eesen is a generalized decoding approach&#10;based on weighted finite-state transducers (WFSTs), which enables the efficient&#10;incorporation of lexicons and language models into CTC decoding. Experiments&#10;show that compared with the standard hybrid DNN systems, Eesen achieves&#10;comparable word error rates (WERs), while at the same time speeding up decoding&#10;significantly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.06580" label="2008.06580">
        <attvalues>
          <attvalue for="0" value="Adaptation Algorithms for Neural Network-Based Speech Recognition: An&#10;  Overview" />
          <attvalue for="1" value="  We present a structured overview of adaptation algorithms for neural&#10;network-based speech recognition, considering both hybrid hidden Markov model /&#10;neural network systems and end-to-end neural network systems, with a focus on&#10;speaker adaptation, domain adaptation, and accent adaptation. The overview&#10;characterizes adaptation algorithms as based on embeddings, model parameter&#10;adaptation, or data augmentation. We present a meta-analysis of the performance&#10;of speech recognition adaptation algorithms, based on relative error rate&#10;reductions as reported in the literature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.01372" label="1907.01372">
        <attvalues>
          <attvalue for="0" value="Improving Performance of End-to-End ASR on Numeric Sequences" />
          <attvalue for="1" value="  Recognizing written domain numeric utterances (e.g. I need $1.25.) can be&#10;challenging for ASR systems, particularly when numeric sequences are not seen&#10;during training. This out-of-vocabulary (OOV) issue is addressed in&#10;conventional ASR systems by training part of the model on spoken domain&#10;utterances (e.g. I need one dollar and twenty five cents.), for which numeric&#10;sequences are composed of in-vocabulary numbers, and then using an FST&#10;verbalizer to denormalize the result. Unfortunately, conventional ASR models&#10;are not suitable for the low memory setting of on-device speech recognition.&#10;E2E models such as RNN-T are attractive for on-device ASR, as they fold the AM,&#10;PM and LM of a conventional model into one neural network. However, in the&#10;on-device setting the large memory footprint of an FST denormer makes spoken&#10;domain training more difficult. In this paper, we investigate techniques to&#10;improve E2E model performance on numeric data. We find that using a&#10;text-to-speech system to generate additional numeric training data, as well as&#10;using a small-footprint neural network to perform spoken-to-written domain&#10;denorming, yields improvement in several numeric classes. In the case of the&#10;longest numeric sequences, we see reduction of WER by up to a factor of 8.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.05525" label="2101.05525">
        <attvalues>
          <attvalue for="0" value="An evaluation of word-level confidence estimation for end-to-end&#10;  automatic speech recognition" />
          <attvalue for="1" value="  Quantifying the confidence (or conversely the uncertainty) of a prediction is&#10;a highly desirable trait of an automatic system, as it improves the robustness&#10;and usefulness in downstream tasks. In this paper we investigate confidence&#10;estimation for end-to-end automatic speech recognition (ASR). Previous work has&#10;addressed confidence measures for lattice-based ASR, while current machine&#10;learning research mostly focuses on confidence measures for unstructured deep&#10;learning. However, as the ASR systems are increasingly being built upon deep&#10;end-to-end methods, there is little work that tries to develop confidence&#10;measures in this context. We fill this gap by providing an extensive benchmark&#10;of popular confidence methods on four well-known speech datasets. There are two&#10;challenges we overcome in adapting existing methods: working on structured data&#10;(sequences) and obtaining confidences at a coarser level than the predictions&#10;(words instead of tokens). Our results suggest that a strong baseline can be&#10;obtained by scaling the logits by a learnt temperature, followed by estimating&#10;the confidence as the negative entropy of the predictive distribution and,&#10;finally, sum pooling to aggregate at word level.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.06201" label="2310.06201">
        <attvalues>
          <attvalue for="0" value="Compressing Context to Enhance Inference Efficiency of Large Language&#10;  Models" />
          <attvalue for="1" value="  Large language models (LLMs) achieved remarkable performance across various&#10;tasks. However, they face challenges in managing long documents and extended&#10;conversations, due to significantly increased computational requirements, both&#10;in memory and inference time, and potential context truncation when the input&#10;exceeds the LLM's fixed context length. This paper proposes a method called&#10;Selective Context that enhances the inference efficiency of LLMs by identifying&#10;and pruning redundancy in the input context to make the input more compact. We&#10;test our approach using common data sources requiring long context processing:&#10;arXiv papers, news articles, and long conversations, on tasks of summarisation,&#10;question answering, and response generation. Experimental results show that&#10;Selective Context significantly reduces memory cost and decreases generation&#10;latency while maintaining comparable performance compared to that achieved when&#10;full context is used. Specifically, we achieve a 50\% reduction in context&#10;cost, resulting in a 36\% reduction in inference memory usage and a 32\%&#10;reduction in inference time, while observing only a minor drop of .023 in&#10;BERTscore and .038 in faithfulness on four downstream applications, indicating&#10;that our method strikes a good balance between efficiency and performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across a wide range of natural language processing tasks, as well as real-life applications \cite{brown2020language,touvron2023llama,bubeck2023sparks}. However, a major challenge for existing LLMs is processing longer context. Dealing with longer context with LLMs is fundamental in scenarios such as having long conversations, document summarisation, and question answering given long documents. However, it is very computationally expensive, particularly with Transformer based LLMs, due to the quadratic growth of memory and computation associated with the 2-D attention matrix \cite{vaswani2017attention}. This makes LLMs less accessible and sometimes leads to context truncation during inference. Moreover, due to the above limitation, existing LLMs were usually pre-trained with fixed-context windows, which further constrains their capability in processing longer context. &#10;&#10;There are active attempts in reducing the computation and memory cost of the Transformer architecture with sparse attention \cite{child2019generating} or local dense attention \cite{beltagy2020longformer}. There are also efforts to learn soft prompts with further distillation to save context cost during inference \cite{mu2023learning,chevalier2023adapting}. &#10;In contrast to existing approaches that primarily focus on &#10;architectures or distillations, we introduce a fresh perspective to tackle the redundancy in the input context itself, thus proposing a complementary, model-agnostic approach that can be potentially combined with other architecture optimisation methods to further enhance inference efficiency.&#10;&#10;The proposed method is motivated by the potential redundancy and repetition in human language, which has two main sources. The first is the inherent redundancy of natural language. For example, in the conversation &quot;A: Did you get the chance to pick up groceries today?&quot;, &quot;B: Yes, \underline{I did get the groceries.}&quot;, the underlined part can be seen as a common redundancy in communication. Linguistic studies suggest redundancy is ubiquitous in language \cite{wit1999linguistic}. The other type of input redundancy is from the overlap with training material. As the example in Fig.~\ref{fig:example} shows, if some parts of input have already been included in the pre-training stage of LLMs, then it is safe to delete them and the model can still generate the correct answer. In summary, redundancy in the input context, while beneficial for human comprehension, can be extraneous for LLMs and might lead to unnecessary computational expense.&#10;&#10;In this paper, we propose Selective Context, which prunes redundant content in a given input context, thereby reducing the computational cost and making better use of the fixed context length in LLMs. Selective Context evaluates informativeness of lexical units (i.e., tokens, phrases, or sentences) with self-information \cite{shannon1948mathematical} computed by a base causal language model. By selectively retaining content with higher self-information, our method provides a more compact and efficient context representation for LLMs to process without compromising their performance on various applications.&#10;&#10;We evaluate the effectiveness and different settings of Selective Context on arXiv papers, BBC News, and real conversation on ShareGPT.com with four NLP tasks: summarisation, question answering, original context reconstruction, and conversation. Experimental results demonstrate that our proposed method can significantly enhance context efficiency of LLMs during inference while maintaining comparable performance compared to that achieved when full context is used.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Natural Language Generation, Artificial Intelligence, Language Model Efficiency, Context Processing Methods" />
        </attvalues>
      </node>
      <node id="2304.08467" label="2304.08467">
        <attvalues>
          <attvalue for="0" value="Learning to Compress Prompts with Gist Tokens" />
          <attvalue for="1" value="  Prompting is the primary way to utilize the multitask capabilities of&#10;language models (LMs), but prompts occupy valuable space in the input context&#10;window, and repeatedly encoding the same prompt is computationally inefficient.&#10;Finetuning and distillation methods allow for specialization of LMs without&#10;prompting, but require retraining the model for each task. To avoid this&#10;trade-off entirely, we present gisting, which trains an LM to compress prompts&#10;into smaller sets of &quot;gist&quot; tokens which can be cached and reused for compute&#10;efficiency. Gist models can be trained with no additional cost over standard&#10;instruction finetuning by simply modifying Transformer attention masks to&#10;encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder&#10;(FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting&#10;in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings,&#10;all with minimal loss in output quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.16193" label="2310.16193">
        <attvalues>
          <attvalue for="0" value="Length is a Curse and a Blessing for Document-level Semantics" />
          <attvalue for="1" value="  In recent years, contrastive learning (CL) has been extensively utilized to&#10;recover sentence and document-level encoding capability from pre-trained&#10;language models. In this work, we question the length generalizability of&#10;CL-based models, i.e., their vulnerability towards length-induced semantic&#10;shift. We verify not only that length vulnerability is a significant yet&#10;overlooked research gap, but we can devise unsupervised CL methods solely&#10;depending on the semantic signal provided by document length. We first derive&#10;the theoretical foundations underlying length attacks, showing that elongating&#10;a document would intensify the high intra-document similarity that is already&#10;brought by CL. Moreover, we found that isotropy promised by CL is highly&#10;dependent on the length range of text exposed in training. Inspired by these&#10;findings, we introduce a simple yet universal document representation learning&#10;framework, LA(SER)$^{3}$: length-agnostic self-reference for semantically&#10;robust sentence representation learning, achieving state-of-the-art&#10;unsupervised performance on the standard information retrieval benchmark.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, contrastive learning (CL) has become the go-to method to train representation encoder models \cite{chen2020simple,he2020momentum,gao2021simcse,su2022one}.&#10;In the field of natural language processing (NLP), the effectiveness of the proposed unsupervised CL methods is typically evaluated on two suites of tasks, namely, &#10;semantic textual similarity (STS)~\cite{cer2017semeval} and information retrieval (IR) (e.g., \cite{thakur2beir}). Surprisingly, a large number of works only validate the usefulness of the learned representations on STS tasks, indicating a strong but widely-adopted assumption that methods optimal for STS could also provide natural transferability to retrieval tasks. &#10;&#10;Due to the document length misalignment of these two types of tasks, the potential gap in models' capability to produce meaningful representation at different length ranges has been rarely explored~\cite{xiao2023can}. &#10;Studies of document length appear to have been stranded in the era where methods are strongly term frequency-based, because of the explicit reflection of document length to sparse embeddings, with little attention given on dense encoders. &#10;Length preference for dense retrieval models is observed by \cite{thakur2beir}, who show that models trained with dot-product and cosine similarity exhibit different length preferences. &#10;However, this phenomenon has not been attributed to the distributional misalignment of length between training and inference domains/tasks, and it remains unknown what abilities of the model are enhanced and diminished when trained with a certain length range.&#10;&#10;In this work, we provide an extensive analysis of length generalizability of standard contrastive learning methods. Our findings show that, with default contrastive learning, models' capability to encode document-level semantics largely comes from their coverage of length range in the training.&#10;&#10;We first depict through derivation the theoretical underpinnings of the models' vulnerability towards length attacks. Through attacking the documents by the simple copy-and-concatenating elongation operation, we show that the vulnerability comes from the further intensified high intra-document similarity that is already pronounced after contrastive learning. This hinders a stable attention towards the semantic tokens in inference time. Further, we show that, the uniformity/isotropy promised by contrastive learning is heavily length-dependent. That is, models' encoded embeddings are only isotropic on the length range seen in the training, but remain anisotropic otherwise, hindering the same strong expressiveness of the embeddings in the unseen length range.&#10;&#10;In the quest to bridge these unideal properties, we propose a simple yet universal framework, LA(SER)$^3$: Length-Agnostic SElf-Reference for SEmantically Robust SEntence Representation learning. &#10;By providing the simple signal that &quot;the elongated version of myself 1) should still mean myself, and thus 2) should not become more or less similar to my pairs&quot;, this framework could not only act as an unsupervised contrastive learning method itself by conducting self-referencing, but could also be combined with any contrastive learning-based text encoding training methods in a plug-and-play fashion, providing strong robustness to length attacks and enhanced encoding ability. &#10;&#10;We show that, our method not only improves contrastive text encoders' robustness to length attack without sacrificing their representational power, but also provides them with external semantic signals, leading to state-of-the-art unsupervised performance on the standard information retrieval benchmark.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Contrastive Learning, Linguistics, Language Models, Artificial Intelligence, Mathematics, Document Representation" />
        </attvalues>
      </node>
      <node id="1911.05722" label="1911.05722">
        <attvalues>
          <attvalue for="0" value="Momentum Contrast for Unsupervised Visual Representation Learning" />
          <attvalue for="1" value="  We present Momentum Contrast (MoCo) for unsupervised visual representation&#10;learning. From a perspective on contrastive learning as dictionary look-up, we&#10;build a dynamic dictionary with a queue and a moving-averaged encoder. This&#10;enables building a large and consistent dictionary on-the-fly that facilitates&#10;contrastive unsupervised learning. MoCo provides competitive results under the&#10;common linear protocol on ImageNet classification. More importantly, the&#10;representations learned by MoCo transfer well to downstream tasks. MoCo can&#10;outperform its supervised pre-training counterpart in 7 detection/segmentation&#10;tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large&#10;margins. This suggests that the gap between unsupervised and supervised&#10;representation learning has been largely closed in many vision tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.09741" label="2212.09741">
        <attvalues>
          <attvalue for="0" value="One Embedder, Any Task: Instruction-Finetuned Text Embeddings" />
          <attvalue for="1" value="  We introduce INSTRUCTOR, a new method for computing text embeddings given&#10;task instructions: every text input is embedded together with instructions&#10;explaining the use case (e.g., task and domain descriptions). Unlike encoders&#10;from prior work that are more specialized, INSTRUCTOR is a single embedder that&#10;can generate text embeddings tailored to different downstream tasks and&#10;domains, without any further training. We first annotate instructions for 330&#10;diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive&#10;loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are&#10;unseen during training), ranging from classification and information retrieval&#10;to semantic textual similarity and text generation evaluation. INSTRUCTOR,&#10;while having an order of magnitude fewer parameters than the previous best&#10;model, achieves state-of-the-art performance, with an average improvement of&#10;3.4% compared to the previous best results on the 70 diverse datasets. Our&#10;analysis suggests that INSTRUCTOR is robust to changes in instructions, and&#10;that instruction finetuning mitigates the challenge of training a single model&#10;on diverse datasets. Our model, code, and data are available at&#10;https://instructor-embedding.github.io.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Text Embeddings Text embeddings are useful in many applications such as information retrieval~\cite{thakur2021beir}, text similarity~\cite{gao-etal-2021-simcse}, prompt retrieval for in-context learning ~\cite{Selective_Annotation}, classification~\cite{reimers-gurevych-2019-sentence}, and beyond.&#10;Much prior work develops different embedding models for different applications. For example, SBERT~\cite{reimers-gurevych-2019-sentence} and SimCSE~\cite{gao-etal-2021-simcse} are applied solely to text similarity and classification tasks, while DPR~\cite{karpukhin-etal-2020-dense} and Contriever~\cite{izacard2022unsupervised} focus on information retrieval.&#10;Different from Sentence-T5 trained only on symmetric data or GTR trained only on asymmetric data, we combine both groups of datasets and build \dataset, which is then used to train \ours with instructions.&#10;\cite{mteb} introduced the massive text embedding benchmark (MTEB), which can be used to evaluate embedding models on a variety of embedding tasks, spanning reranking, classification, information retrieval, bitext mining, pair classification, STS, and summarization.&#10;Their benchmark shows that models performing well on one task may not perform well on other tasks.&#10;The poor zero-shot transfer abilities of existing embedding models make it difficult to use them in applications where only few labeled data are available.&#10;This motivates us to develop a single embedding model that is applicable to a variety of tasks and has better generalization to unseen tasks.&#10;\cite{Wang2022TextEB} recently proposed E5, weakly-supervised contrastive pre-trained text embeddings, which achieve strong performance across various tasks on the MTEB benchmark, employing a larger embedding dimension compared to \ours.&#10;&#10;Instruction Finetuning &#10;Recent work demonstrated that instruction-finetuned language models could perform new tasks given a natural language instruction~\cite{mishra2022cross, Zhong2021AdaptingLM,min-etal-2022-metaicl,Sanh2022Multitask,Wei2022FinetunedLM, natural_instructions_v2,ouyang2022training}.&#10;Nonetheless, instruction finetuning has yet to be studied in the context of broadly-applicable embeddings.&#10;In this work, we explore finetuning embedding models to follow human instructions where the instruction specifies eventual use cases. &#10;Concurrent work demonstrated that instructions could facilitate information retrieval \cite{task_aware_retrieval}, which is related to our \ours design. &#10;They used instructions to build a task-aware retrieval system and conducted evaluations on the retrieval task; we build a general-purpose embedding model with instructions that can be applied to 8 tasks categories (Fig.\ \ref{fig:pipeline}), including retrieval, text similarity, clustering, and text evaluation.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2305.14785" label="2305.14785">
        <attvalues>
          <attvalue for="0" value="Simple Linguistic Inferences of Large Language Models (LLMs): Blind&#10;  Spots and Blinds" />
          <attvalue for="1" value="  We evaluate LLMs' language understanding capacities on simple inference tasks&#10;that most humans find trivial. Specifically, we target (i)&#10;grammatically-specified entailments, (ii) premises with evidential adverbs of&#10;uncertainty, and (iii) monotonicity entailments. We design evaluation sets for&#10;these tasks and conduct experiments in both zero-shot and chain-of-thought&#10;setups, and with multiple prompts and LLMs. The models exhibit moderate to low&#10;performance on these evaluation sets. Subsequent experiments show that&#10;embedding the premise in syntactic constructions that should preserve the&#10;entailment relations (presupposition triggers) or change them (non-factives),&#10;further confuses the models, causing them to either under-predict or&#10;over-predict certain entailment labels regardless of the true relation, and&#10;often disregarding the nature of the embedding context. Overall these results&#10;suggest that, despite LLMs' celebrated language understanding capacity, even&#10;the strongest models have blindspots with respect to certain types of&#10;entailments, and certain information-packaging structures act as ``blinds''&#10;overshadowing the semantics of the embedded premise.&#10;" />
          <attvalue for="2" value="&#10;&#10;LLMs have gained immense popularity thanks to their unprecedented ability to understand user queries and generate fluent seemingly-human responses. At the same time, people constantly report LLMs' failures, anecdotal&#10;\cite{borji2023categorical} and systematic, e.g, the lack of reliability and consistency \cite{shen2023chatgpt, jang2023consistency,plevris2023chatbots},&#10;contradictory or unreasonable answers \cite{zhong2023chatgpt}, inability to detect false assumptions \cite{shen2023chatgpt}, wrong information in prompts&#10;\cite {zuccon2023dr}, contradictory responses to identical queries \cite{jang2023consistency, plevris2023chatbots}.&#10;&#10;However, humans are prone to some failures as well, e.g., overlooking false assumptions in questions beyond their area of expertise, or failing to find the correct solution to a math problem.&#10;&#10;In this work we focus on tasks that are trivial for humans, and do not require any specialized expertise beyond proficiency in English.&#10;For example, it is obvious to a human that Her brother was singing entails Someone was singing, and Fred’s tie is very long implies Fred’s tie is long, but not vice versa. However, as shall be seen shortly, &#10;LLMs fail to establish such systematic relations correctly. LLMs' errors on such simple tasks are much more indicative of absence of human-like text understanding. &#10;&#10;We experiment with several types of natural language inferences (NLI), (a.k.a. recognizing textual entailment \cite{dagan2005pascal,bowman2015snli}), &#10;that are easy for humans, and show that they pose a challenge to LLMs. These NLI tests reveal some of the models' blind spots, and indicate that they are far from a genuine human-level understanding. Moreover,&#10;some information-packaging structures, where a statement is embedded within a surrounding linguistic context, may act as ``blinds'' that hinder the semantics of embedded premises, again in contrast to human-like behavior.&#10;&#10;A summary of the conditions we test, as well as the trends observed for one model (gpt-3.5-turbo-0301) is available in Figure \ref{fig:main}.&#10;&#10;We focus on inference types that are solely based on common linguistic phenomena and ``trival'' world-knowledge such as class membership (``a dog is an animal'', ``navy blue is a shade of blue''). Specifically, we test LLMs' ability to make three inference types:&#10;(i) {Grammatically-specified entailments}, i.e. replacing a constituent of the premise with an indefinite pronoun as somebody or something.&#10;(ii) Premises with {evidential adverbs of uncertainty} (supposedly, allegedly etc.), that&#10;block the entailment of the rest of the clause, and&#10;(iii) {Monotonicity entailment} (see \cite{maccartney-manning-2008-modeling}) of two kinds: upward, i.e. from subsets to supersets (``Jack is a dog'' entails ``Jack is an animal''), and downward, i.e. from supersets to subsets (``Jack isn’t an animal'' entails ``Jack isn’t a dog'').&#10;We manually curate test sets for these inference types and experiment with them in a zero-shot setup, observing that LLMs struggle with these phenomena, leading to low accuracy.&#10;&#10;We next check how embedding of the premise in a larger grammatical context affects the prediction. Such embedding can take several forms. Contexts consisting of presupposition triggers (e.g. He realized that [...], They were glad that [...], Something happened before [...]) serve to strengthen the embedded premise, while similarly structured non-factives (e.g. I feel that [...],&#10;He imagined that [...]) may cancel it. We experiment with both context types and show that in most cases they affect the LLMs' predictions incorrectly. E.g., ChatGPT has a hard time discerning the two cases,&#10;incorrectly treating both as hints towards entailment (for regular prompting) or against it (for chain-of-thought prompting).&#10;These or similar trends are observed across different prompts and models (GPT-3.5, GPT-4, LLaMA 2). While GPT-4 clearly improves over other LLMs in terms of accuracy and shows some more sound trends, it still leaves much room for improvement compared to the human baseline.&#10;&#10;These results demonstrate that state-of-the-art LLMs were unable to learn simple linguistic inferences that humans find trivial: they did not acquire them automatically in pre-training, and also in the process of instruct-tuning or human-feedback tuning.&#10;Persistence of the problem across prompts and LLMs implies that this is a systematic issue.&#10;&#10;While current LLMs struggle with these trivial phenomena, we hope future models will show improvements. Our methodology and the dataset we release can help track this progress. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Inference Task Performance, Natural Language Understanding, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2302.03494" label="2302.03494">
        <attvalues>
          <attvalue for="0" value="A Categorical Archive of ChatGPT Failures" />
          <attvalue for="1" value="  Large language models have been demonstrated to be valuable in different&#10;fields. ChatGPT, developed by OpenAI, has been trained using massive amounts of&#10;data and simulates human conversation by comprehending context and generating&#10;appropriate responses. It has garnered significant attention due to its ability&#10;to effectively answer a broad range of human inquiries, with fluent and&#10;comprehensive answers surpassing prior public chatbots in both security and&#10;usefulness. However, a comprehensive analysis of ChatGPT's failures is lacking,&#10;which is the focus of this study. Eleven categories of failures, including&#10;reasoning, factual errors, math, coding, and bias, are presented and discussed.&#10;The risks, limitations, and societal implications of ChatGPT are also&#10;highlighted. The goal of this study is to assist researchers and developers in&#10;enhancing future language models and chatbots.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18618" label="2305.18618">
        <attvalues>
          <attvalue for="0" value="Chatbots put to the test in math and logic problems: A preliminary&#10;  comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard" />
          <attvalue for="1" value="  A comparison between three chatbots which are based on large language models,&#10;namely ChatGPT-3.5, ChatGPT-4 and Google Bard is presented, focusing on their&#10;ability to give correct answers to mathematics and logic problems. In&#10;particular, we check their ability to Understand the problem at hand; Apply&#10;appropriate algorithms or methods for its solution; and Generate a coherent&#10;response and a correct answer. We use 30 questions that are clear, without any&#10;ambiguities, fully described with plain text only, and have a unique, well&#10;defined correct answer. The questions are divided into two sets of 15 each. The&#10;questions of Set A are 15 &quot;Original&quot; problems that cannot be found online,&#10;while Set B contains 15 &quot;Published&quot; problems that one can find online, usually&#10;with their solution. Each question is posed three times to each chatbot. The&#10;answers are recorded and discussed, highlighting their strengths and&#10;weaknesses. It has been found that for straightforward arithmetic, algebraic&#10;expressions, or basic logic puzzles, chatbots may provide accurate solutions,&#10;although not in every attempt. However, for more complex mathematical problems&#10;or advanced logic tasks, their answers, although written in a usually&#10;&quot;convincing&quot; way, may not be reliable. Consistency is also an issue, as many&#10;times a chatbot will provide conflicting answers when given the same question&#10;more than once. A comparative quantitative evaluation of the three chatbots is&#10;made through scoring their final answers based on correctness. It was found&#10;that ChatGPT-4 outperforms ChatGPT-3.5 in both sets of questions. Bard comes&#10;third in the original questions of Set A, behind the other two chatbots, while&#10;it has the best performance (first place) in the published questions of Set B.&#10;This is probably because Bard has direct access to the internet, in contrast to&#10;ChatGPT chatbots which do not have any communication with the outside world.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.06273" label="2303.06273">
        <attvalues>
          <attvalue for="0" value="Consistency Analysis of ChatGPT" />
          <attvalue for="1" value="  ChatGPT has gained a huge popularity since its introduction. Its positive&#10;aspects have been reported through many media platforms, and some analyses even&#10;showed that ChatGPT achieved a decent grade in professional exams, adding extra&#10;support to the claim that AI can now assist and even replace humans in&#10;industrial fields. Others, however, doubt its reliability and trustworthiness.&#10;This paper investigates the trustworthiness of ChatGPT and GPT-4 regarding&#10;logically consistent behaviour, focusing specifically on semantic consistency&#10;and the properties of negation, symmetric, and transitive consistency. Our&#10;findings suggest that while both models appear to show an enhanced language&#10;understanding and reasoning ability, they still frequently fall short of&#10;generating logically consistent predictions. We also ascertain via experiments&#10;that prompt designing, few-shot learning and employing larger large language&#10;models (LLMs) are unlikely to be the ultimate solution to resolve the&#10;inconsistency issue of LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.08732" label="2305.08732">
        <attvalues>
          <attvalue for="0" value="Knowledge Rumination for Pre-trained Language Models" />
          <attvalue for="1" value="  Previous studies have revealed that vanilla pre-trained language models&#10;(PLMs) lack the capacity to handle knowledge-intensive NLP tasks alone; thus,&#10;several works have attempted to integrate external knowledge into PLMs.&#10;However, despite the promising outcome, we empirically observe that PLMs may&#10;have already encoded rich knowledge in their pre-trained parameters but fail to&#10;fully utilize them when applying them to knowledge-intensive tasks. In this&#10;paper, we propose a new paradigm dubbed Knowledge Rumination to help the&#10;pre-trained language model utilize that related latent knowledge without&#10;retrieving it from the external corpus. By simply adding a prompt like &quot;As far&#10;as I know&quot; to the PLMs, we try to review related latent knowledge and inject&#10;them back into the model for knowledge consolidation. We apply the proposed&#10;knowledge rumination to various language models, including RoBERTa, DeBERTa,&#10;and GPT-3. Experimental results on six commonsense reasoning tasks and GLUE&#10;benchmarks demonstrate the effectiveness of our proposed approach, which proves&#10;that the knowledge stored in PLMs can be better exploited to enhance&#10;performance. Code is available in&#10;https://github.com/zjunlp/knowledge-rumination.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-trained language models (PLMs) have waved the NLP community as fundamental infrastructure by demonstrating remarkable abilities with the ``pre-train, prompt, and predict'' paradigm \cite{liu2021pre,DBLP:journals/corr/abs-2303-18223}. &#10;The mere PLMs, however, lack the capacity to handle knowledge-intensive tasks with advanced functionalities like commonsense reasoning \cite{lin-etal-2019-kagnet,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2305-03695} and open-domain question answering \cite{DBLP:conf/emnlp/YangYM15}.&#10;This necessitates a boosting trend for research focusing on augmenting PLMs with external knowledge sources \cite{chen-etal-2017-reading,DBLP:conf/www/ChenZXDYTHSC22,DBLP:conf/nips/Welleck0BHCCC21,DBLP:conf/nips/WelleckLLHC22,Zhang2022GreaseLMGR,zhang2023multimodal}.&#10;&#10;However, despite the empirical success, we observe that PLMs can often encode extensive knowledge within their parameters yet fail to utilize this effectively for knowledge-intensive tasks.&#10;Taking pilot experiments as an example, we use knowledge probing \cite{petroni-etal-2019-language} to the PLM as shown in Figure~\ref{fig:overview}.&#10;Given a question ``If a bird is a carnivore, then it is likely a(n) what?'', we notice that the PLM has known the knowledge ``a carnivore is likely a(n) predator'' in its parameters; however, we surprisingly find that the finetuned PLM chose the wrong answer despite the model knowing the related knowledge.&#10;Interestingly, this phenomenon mirrors human behavior. As an example, in the cognitive reflection test (CRT) \cite{Frederick2005CognitiveRA}, participants have posed a series of straightforward questions (already learned), yet they often initially fail in their intuitive reasoning. &#10;Upon reflection, however, individuals typically identify their erroneous responses and correct them.&#10;Consequently, we conjecture that the prominent PLMs of today have flaws as humans and we still have the following problem: are we fully exploiting the potential of the PLMs?&#10;&#10;Some pioneering researchers have attempted to unravel this enigma.&#10;For instance, &#10;\cite{DBLP:journals/corr/abs-2202-04824} and \cite{DBLP:journals/corr/abs-2210-14803} propose to utilize the knowledge in the pre-traning corpus by retrieve-then-fine-tuning method. &#10;Likewise, \cite{DBLP:conf/iclr/BhagavatulaBMSH20} capitalizes on the implicit knowledge within large language models (&gt;10B) by retrieving from model weights with recitation-augmented generation.&#10;These studies affirm that PLMs encapsulate a vast body of knowledge, with untapped potential, while in our paper, we pursue a more universally applicable, yet simple solution to fully harness knowledge in PLMs for NLP.&#10;&#10;To address this need, we introduce Knowledge Rumination to assist the model in thinking thoughtfully in handling knowledge-intensive tasks.&#10;Analogous to how animals ruminate food for better digestion and absorption—by regurgitating it from the stomach back to the mouth for additional chewing—we aim to mimic this process by having the model first review the relevant knowledge stored in its parameters and then consolidate this knowledge to better tackle associated tasks.&#10;In detail, we propose knowledge reviewing with a task-guided prompt by simply adding ``As far as I know'' to stimulate the model to recall latent knowledge.&#10;Subsequently, we consolidate knowledge via FFN to explicitly leverage latent knowledge to help address downstream tasks since FFN plays a crucial role in PLMs \cite{wang-etal-2022-finding-skill}.&#10;&#10;We apply the proposed knowledge rumination to various PLMs, including RoBERTa \cite{Liu2019RoBERTaAR}, DeBERTa \cite{he2021deberta}. &#10;We also transfer knowledge rumination to large language GPT-3 (175B)~\cite{DBLP:journals/corr/abs-2005-14165}.&#10;Experimental results on six commonsense reasoning tasks and the GLUE benchmark demonstrate that the proposed simple method can obtain performance gain and even outperform baselines of retrieving external knowledge.&#10;To conclude, we summarize the contributions of this work as follows:&#10;\begin{itemize}&#10; \item We propose a novel approach of Knowledge Rumination to better utilize the knowledge stored in the parameters, which is model agnostic and can be applied to any PLMs&#10; \item Experimental results demonstrate that the proposed approach can successfully elicit related knowledge for both small and large PLMs, yielding better performance on six commonsense tasks and GLUE benchmarks. &#10; \item Comprehensive empirical analysis indicates that still a large underestimated amount of knowledge can be retrieved from PLM's model weights, and our work takes a small step in this direction. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Model Enhancement, Artificial Intelligence, Natural Language Processing, Knowledge Integration Techniques" />
        </attvalues>
      </node>
      <node id="2305.03695" label="2305.03695">
        <attvalues>
          <attvalue for="0" value="Vera: A General-Purpose Plausibility Estimation Model for Commonsense&#10;  Statements" />
          <attvalue for="1" value="  Despite the much discussed capabilities of today's language models, they are&#10;still prone to silly and unexpected commonsense failures. We consider a&#10;retrospective verification approach that reflects on the correctness of LM&#10;outputs, and introduce Vera, a general-purpose model that estimates the&#10;plausibility of declarative statements based on commonsense knowledge. Trained&#10;on ~7M commonsense statements created from 19 QA datasets and two large-scale&#10;knowledge bases, and with a combination of three training objectives, Vera is a&#10;versatile model that effectively separates correct from incorrect statements&#10;across diverse commonsense domains. When applied to solving commonsense&#10;problems in the verification format, Vera substantially outperforms existing&#10;models that can be repurposed for commonsense verification, and it further&#10;exhibits generalization capabilities to unseen tasks and provides&#10;well-calibrated outputs. We find that Vera excels at filtering LM-generated&#10;commonsense knowledge and is useful in detecting erroneous commonsense&#10;statements generated by models like ChatGPT in real-world settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.01112" label="2104.01112">
        <attvalues>
          <attvalue for="0" value="NaturalProofs: Mathematical Theorem Proving in Natural Language" />
          <attvalue for="1" value="  Understanding and creating mathematics using natural mathematical language -&#10;the mixture of symbolic and natural language used by humans - is a challenging&#10;and important problem for driving progress in machine learning. As a step in&#10;this direction, we develop NaturalProofs, a multi-domain corpus of mathematical&#10;statements and their proofs, written in natural mathematical language.&#10;NaturalProofs unifies broad coverage, deep coverage, and low-resource&#10;mathematical sources, allowing for evaluating both in-distribution and&#10;zero-shot generalization. Using NaturalProofs, we benchmark strong neural&#10;methods on mathematical reference retrieval and generation tasks which test a&#10;system's ability to determine key results that appear in a proof. Large-scale&#10;sequence models show promise compared to classical information retrieval&#10;methods, yet their performance and out-of-domain generalization leave&#10;substantial room for improvement. NaturalProofs opens many avenues for research&#10;on challenging mathematical tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.04824" label="2202.04824">
        <attvalues>
          <attvalue for="0" value="AdaPrompt: Adaptive Model Training for Prompt-based NLP" />
          <attvalue for="1" value="  Prompt-based learning, with its capability to tackle zero-shot and few-shot&#10;NLP tasks, has gained much attention in community. The main idea is to bridge&#10;the gap between NLP downstream tasks and language modeling (LM), by mapping&#10;these tasks into natural language prompts, which are then filled by pre-trained&#10;language models (PLMs). However, for prompt learning, there are still two&#10;salient gaps between NLP tasks and pretraining. First, prompt information is&#10;not necessarily sufficiently present during LM pretraining. Second,&#10;task-specific data are not necessarily well represented during pretraining. We&#10;address these two issues by proposing AdaPrompt, adaptively retrieving external&#10;data for continual pretraining of PLMs by making use of both task and prompt&#10;characteristics. In addition, we make use of knowledge in Natural Language&#10;Inference models for deriving adaptive verbalizers. Experimental results on&#10;five NLP benchmarks show that AdaPrompt can improve over standard PLMs in&#10;few-shot settings. In addition, in zero-shot settings, our method outperforms&#10;standard prompt-based methods by up to 26.35\% relative error reduction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.14803" label="2210.14803">
        <attvalues>
          <attvalue for="0" value="Don't Prompt, Search! Mining-based Zero-Shot Learning with Language&#10;  Models" />
          <attvalue for="1" value="  Masked language models like BERT can perform text classification in a&#10;zero-shot fashion by reformulating downstream tasks as text infilling. However,&#10;this approach is highly sensitive to the template used to prompt the model, yet&#10;practitioners are blind when designing them in strict zero-shot settings. In&#10;this paper, we propose an alternative mining-based approach for zero-shot&#10;learning. Instead of prompting language models, we use regular expressions to&#10;mine labeled examples from unlabeled corpora, which can optionally be filtered&#10;through prompting, and used to finetune a pretrained model. Our method is more&#10;flexible and interpretable than prompting, and outperforms it on a wide range&#10;of tasks when using comparable templates. Our results suggest that the success&#10;of prompting can partly be explained by the model being exposed to similar&#10;examples during pretraining, which can be directly retrieved through regular&#10;expressions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.05739" label="1908.05739">
        <attvalues>
          <attvalue for="0" value="Abductive Commonsense Reasoning" />
          <attvalue for="1" value="  Abductive reasoning is inference to the most plausible explanation. For&#10;example, if Jenny finds her house in a mess when she returns from work, and&#10;remembers that she left a window open, she can hypothesize that a thief broke&#10;into her house and caused the mess, as the most plausible explanation. While&#10;abduction has long been considered to be at the core of how people interpret&#10;and read between the lines in natural language (Hobbs et al., 1988), there has&#10;been relatively little research in support of abductive natural language&#10;inference and generation. We present the first study that investigates the&#10;viability of language-based abductive reasoning. We introduce a challenge&#10;dataset, ART, that consists of over 20k commonsense narrative contexts and 200k&#10;explanations. Based on this dataset, we conceptualize two new tasks -- (i)&#10;Abductive NLI: a multiple-choice question answering task for choosing the more&#10;likely explanation, and (ii) Abductive NLG: a conditional generation task for&#10;explaining given observations in natural language. On Abductive NLI, the best&#10;model achieves 68.9% accuracy, well below human performance of 91.4%. On&#10;Abductive NLG, the current best language generators struggle even more, as they&#10;lack reasoning capabilities that are trivial for humans. Our analysis leads to&#10;new insights into the types of reasoning that deep pre-trained language models&#10;fail to perform--despite their strong performance on the related but more&#10;narrowly defined task of entailment NLI--pointing to interesting avenues for&#10;future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.07349" label="2211.07349">
        <attvalues>
          <attvalue for="0" value="Finding Skill Neurons in Pre-trained Transformer-based Language Models" />
          <attvalue for="1" value="  Transformer-based pre-trained language models have demonstrated superior&#10;performance on various natural language processing tasks. However, it remains&#10;unclear how the skills required to handle these tasks distribute among model&#10;parameters. In this paper, we find that after prompt tuning for specific tasks,&#10;the activations of some neurons within pre-trained Transformers are highly&#10;predictive of the task labels. We dub these neurons skill neurons and confirm&#10;they encode task-specific skills by finding that: (1) Skill neurons are crucial&#10;for handling tasks. Performances of pre-trained Transformers on a task&#10;significantly drop when corresponding skill neurons are perturbed. (2) Skill&#10;neurons are task-specific. Similar tasks tend to have similar distributions of&#10;skill neurons. Furthermore, we demonstrate the skill neurons are most likely&#10;generated in pre-training rather than fine-tuning by showing that the skill&#10;neurons found with prompt tuning are also crucial for other fine-tuning methods&#10;freezing neuron weights, such as the adapter-based tuning and BitFit. We also&#10;explore the applications of skill neurons, including accelerating Transformers&#10;with network pruning and building better transferability indicators. These&#10;findings may promote further research on understanding Transformers. The source&#10;code can be obtained from https://github.com/THU-KEG/Skill-Neuron.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Selective Neurons in Artificial Neural Networks&#10;There have long been findings about selective neurons in artificial neural networks. Many computer vision works~\cite{Coates2012EmergenceOO,Le2013BuildingHF,zeiler2014visualizing,Agrawal2014AnalyzingTP,Zhou2015ObjectDE,bau2020understanding} find that both supervised and unsupervised models can have units selectively respond to specific visual objects and concepts. \cite{Radford2017LearningTG} also find neurons corresponding to sentiments in unsupervised long short-term memory networks. Interestingly, there are similar selective neurons in human brains~\cite{barlow1972single,quiroga2005invariant}. The widespread emergence of these neuronal selectivities implies that there may be common learning mechanisms among intelligent systems, which is extremely worthwhile to explore in the future.&#10;&#10;\cite{Bau2017NetworkDQ} and \cite{Mu2020CompositionalEO} find that selective neurons are more important, which is consistent with our findings. However, \cite{morcos2018importance} draw opposite conclusions. We discuss this with experiments in \ref{app:single_direction}.&#10;&#10;Analyzing Pre-trained Transformers&#10;After the success of Transformer-based PLMs~\cite{devlin-etal-2019-bert,yang2019xlnet,raffel2020T5}, many efforts have been devoted to analyzing how PLMs work, such as probing the knowledge of PLMs~\cite{Liu2019LinguisticKA,hewitt2019structural,petroni-etal-2019-language} and understanding the behaviors of PLMs' parameters~\cite{voita-etal-2019-analyzing,clark-etal-2019-bert}. Among these, some works~\cite{dalvi2019one,durrani2020analyzing,antverg2021pitfalls} find that individual neurons capture linguistic properties, but they define neurons as dimensions in contextualized representations. Other works~\cite{suau2020finding,geva2021transformer,dai2021knowledge} study the same group of neurons as us and find that some neurons encode specific information like concepts, facts, and word patterns. Inspired by them, we study whether neurons encode high-level skills for handling tasks in this work and demonstrate that we can observe skill neurons with the help of prompts. We believe it is promising to explore whether and how skill neurons collaborate with the neurons encoding information in future works.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2005.02068" label="2005.02068">
        <attvalues>
          <attvalue for="0" value="Establishing Baselines for Text Classification in Low-Resource Languages" />
          <attvalue for="1" value="  While transformer-based finetuning techniques have proven effective in tasks&#10;that involve low-resource, low-data environments, a lack of properly&#10;established baselines and benchmark datasets make it hard to compare different&#10;approaches that are aimed at tackling the low-resource setting. In this work,&#10;we provide three contributions. First, we introduce two previously unreleased&#10;datasets as benchmark datasets for text classification and low-resource&#10;multilabel text classification for the low-resource language Filipino. Second,&#10;we pretrain better BERT and DistilBERT models for use within the Filipino&#10;setting. Third, we introduce a simple degradation test that benchmarks a&#10;model's resistance to performance degradation as the number of training samples&#10;are reduced. We analyze our pretrained model's degradation speeds and look&#10;towards the use of this method for comparing models aimed at operating within&#10;the low-resource setting. We release all our models and datasets for the&#10;research community to use.&#10;" />
          <attvalue for="2" value="&#10;In recent years, finetuning large-scale pretrained Transformer \cite{vaswani2017attention} models have been the most successful technique to solve various Natural Language Processing (NLP) tasks such as Machine Translation \cite{edunov2018understanding,raffel2019exploring}, Text Summarization \cite{yan2020prophetnet,takase2019positional}, Question-Answering \cite{zhang2020retrospective,garg2019tanda,dhingra2017linguistic}, Natural Language Inference \cite{zhang2019semantics,lan2019albert}, among others. This is owed to the learned knowledge intact in the model from being pretrained using a large unlabeled corpora in a source language.&#10;&#10;This method is attractive for various low-resource, low-data settings. In most cases, low-resource languages suffer from a lack of labeled corpora and resources, but not unlabeled data. This allows models to be pretrained, and then finetuned later on smaller datasets to produce robust models. This method has been shown to be effective in various low-resource tasks such as Low-resource Machine Translation \cite{zoph2016transfer}, Cross-lingual Language Modeling \cite{adams2017cross}, Named Entity Recognition \cite{das2017named}, Fake News Detection \cite{cruz2019localization}, and many more.&#10;&#10;While finetuning and transfer learning has proven useful for low-resource, low-data tasks, a lack of published resources and, more importantly, properly established benchmarks is still a problem. &#10;&#10;Without proper benchmark tasks in low-resource languages, there is no way to properly compare performance of different models and techniques. Even if they are proven to work in mainstream ``academic'' languages such as English, French, German, and others, certain quirks and characteristics of low-resource languages may affect the performance of commonly-used models. In this scenario, a commonly-held state-of-the-art model may in fact hold a flaw that can only be observed if more baselines in more languages are tested.&#10;&#10;In this work, we provide three contributions.&#10;&#10;First, we release two previously unreleased datasets. The first is a benchmark on low-resource text classification in the low-resource Filipino with two labels. This dataset holds enough data for standard from-scratch training on various neural network models, and can be used to compare various newer techniques with traditional ones. We believe that a standard baseline for text classification in Filipino is important in order to measure the progress of the field. The second dataset is a small, low-resource dataset for multilabel text classification, again in the low-resource Filipino language. We believe that a proper small-sample dataset in a slightly harder task will provide a good baseline for various classification techniques intended for low-resource, low-data environments moving forward.&#10;&#10;Second, we pretrain stronger BERT \cite{devlin2018bert} models in Filipino, with larger input sequence lengths than our previous Tagalog-BERT models \cite{cruz2019evaluating}. These models should be larger and provide more capacity for learning various tasks, not just for our low-resource text classification baselines, but for a lot of other tasks within low-resource NLP in the future. In addition to BERT, we also provide a distilled \cite{sanh2019distilbert} version of our basic cased model, which we call Tagalog-DistilBERT. We distil and provide a smaller pretrained transformer in the best interest of low-resource settings from an equipment perspective. Smaller pretrained models can also be used for deployment to mobile and on-edge applications. We test and benchmark our models on our datasets to provide an initial baseline.&#10;&#10;Lastly, we introduce a simple benchmarking test to gauge a model's resilience to performance degradation when the number of training samples given to it is reduced. We test our BERT and DistilBERT models on this task with our provided datasets and give initial baselines on their performance degradation in low-data tasks. We introduce this as a way to provide a comparison point for future models, where a model with slower degradation should, empirically, be better at low-resource and low-data tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Low Resource Learning, Artificial Intelligence, Benchmarking Techniques, Information Science, Language Modeling" />
        </attvalues>
      </node>
      <node id="1904.07418" label="1904.07418">
        <attvalues>
          <attvalue for="0" value="Positional Encoding to Control Output Sequence Length" />
          <attvalue for="1" value="  Neural encoder-decoder models have been successful in natural language&#10;generation tasks. However, real applications of abstractive summarization must&#10;consider additional constraint that a generated summary should not exceed a&#10;desired length. In this paper, we propose a simple but effective extension of a&#10;sinusoidal positional encoding (Vaswani et al., 2017) to enable neural&#10;encoder-decoder model to preserves the length constraint. Unlike in previous&#10;studies where that learn embeddings representing each length, the proposed&#10;method can generate a text of any length even if the target length is not&#10;present in training data. The experimental results show that the proposed&#10;method can not only control the generation length but also improve the ROUGE&#10;scores.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.02209" label="1909.02209">
        <attvalues>
          <attvalue for="0" value="Semantics-aware BERT for Language Understanding" />
          <attvalue for="1" value="  The latest work on language representations carefully integrates&#10;contextualized features into language model training, which enables a series of&#10;success especially in various machine reading comprehension and natural&#10;language inference tasks. However, the existing language representation models&#10;including ELMo, GPT and BERT only exploit plain context-sensitive features such&#10;as character or word embeddings. They rarely consider incorporating structured&#10;semantic information which can provide rich semantics for language&#10;representation. To promote natural language understanding, we propose to&#10;incorporate explicit contextual semantics from pre-trained semantic role&#10;labeling, and introduce an improved language representation model,&#10;Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing&#10;contextual semantics over a BERT backbone. SemBERT keeps the convenient&#10;usability of its BERT precursor in a light fine-tuning way without substantial&#10;task-specific modifications. Compared with BERT, semantics-aware BERT is as&#10;simple in concept but more powerful. It obtains new state-of-the-art or&#10;substantially improves results on ten reading comprehension and language&#10;inference tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.00409" label="1907.00409">
        <attvalues>
          <attvalue for="0" value="Evaluating Language Model Finetuning Techniques for Low-resource&#10;  Languages" />
          <attvalue for="1" value="  Unlike mainstream languages (such as English and French), low-resource&#10;languages often suffer from a lack of expert-annotated corpora and benchmark&#10;resources that make it hard to apply state-of-the-art techniques directly. In&#10;this paper, we alleviate this scarcity problem for the low-resourced Filipino&#10;language in two ways. First, we introduce a new benchmark language modeling&#10;dataset in Filipino which we call WikiText-TL-39. Second, we show that language&#10;model finetuning techniques such as BERT and ULMFiT can be used to consistently&#10;train robust classifiers in low-resource settings, experiencing at most a&#10;0.0782 increase in validation error when the number of training examples is&#10;decreased from 10K to 1K while finetuning using a privately-held sentiment&#10;dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.18803" label="2403.18803">
        <attvalues>
          <attvalue for="0" value="Projective Methods for Mitigating Gender Bias in Pre-trained Language&#10;  Models" />
          <attvalue for="1" value="  Mitigation of gender bias in NLP has a long history tied to debiasing static&#10;word embeddings. More recently, attention has shifted to debiasing pre-trained&#10;language models. We study to what extent the simplest projective debiasing&#10;methods, developed for word embeddings, can help when applied to BERT's&#10;internal representations. Projective methods are fast to implement, use a small&#10;number of saved parameters, and make no updates to the existing model&#10;parameters. We evaluate the efficacy of the methods in reducing both intrinsic&#10;bias, as measured by BERT's next sentence prediction task, and in mitigating&#10;observed bias in a downstream setting when fine-tuned. To this end, we also&#10;provide a critical analysis of a popular gender-bias assessment test for&#10;quantifying intrinsic bias, resulting in an enhanced test set and new bias&#10;measures. We find that projective methods can be effective at both intrinsic&#10;bias and downstream bias mitigation, but that the two outcomes are not&#10;necessarily correlated. This finding serves as a warning that intrinsic bias&#10;test sets, based either on language modeling tasks or next sentence prediction,&#10;should not be the only benchmark in developing a debiased language model.&#10;" />
          <attvalue for="2" value="&#10;While decoder-based generative models have shown significant capabilities in generating coherent and contextually relevant language \cite{Shahriar_Hayawi_2023}, many studies have consistently demonstrated that BERT-family encoders fine-tuned with carefully crafted data are more reliable in specialized classification tasks (see e.g. \cite{ziems2023, pahwa-pahwa-2023-bphigh, li2023chatgpt, bang2023multitask}). This observation positions BERT-family models as practitioners' primary choice for everyday NLP tasks. Given their widespread adoption, it is crucial to debias BERT-like models to ensure fairness in real-world applications.&#10;&#10;Mitigating gender bias in NLP systems typically involves quantifying and reducing bias within the relevant pre-trained resource. Perhaps the most obvious way to test for intrinsic bias in a pre-trained language model is to propose a masked language modelling (MLM) task,&#10;where content is developed around known social stereotypes \cite{nadeem-etal-2021-stereoset, nangia-etal-2020-crows}. &#10;Recently, a large-scale survey \cite{meade-etal-2022-empirical} compared intrinsic bias mitigation as measured by an MLM test set across several debiasing strategies, including sentence debiasing \cite{liang-etal-2020-towards-sent-debias}. &#10;However, the debiasing techniques were not tested on a fine-tuned model for any other task beyond language modelling. &#10;&#10;Evaluating the MLM target is most relevant when the downstream classification task of interest is performed on a single sentence or short passage. For tasks that require long-range inferences between two sentences (e.g.\ question-answering and natural language inference), Next Sentence Prediction (NSP) is known to be the relevant training target for BERT-like derivatives; the inclusion of this inter-sentence conditioning significantly improves benchmark performance on such tasks \cite{devlin-etal-2019-bert}.&#10;In this work, we focus our attention on intrinsic bias in the lesser-studied functionality of BERT-family models, next-sentence prediction (NSP), and study the connection to a downstream task that processes two sentences as its input, Natural Language Inference (NLI).&#10;&#10;NLI is a fundamental NLP task that involves determining the relationship between two sentences \cite{storks2019recent}. This type of relational understanding is foundational for many higher-level tasks in NLP, such as reading comprehension, dialogue systems, and summarization. More specifically, NLI is used to improve Question-answering models \cite{chen-etal-2021-nli-models, fortier-dubois-rosati, PARAMASIVAM20229644}, dialogue systems \cite{Chen2019-dialog}, and content verification models \cite{falke-etal-2019-ranking, dusek-kasner-2020-evaluating}. Also, since NLI tasks require logical judgments, they provide a window into potential biases in model reasoning. Here we evaluate extrinsic bias using NLI as our downstream task due to its prevalence and foundational nature. &#10;&#10;To mitigate intrinsic bias in BERT observed through NSP, and extrinsic bias within NLI, we ask how much can be borrowed from the debiasing schemes that were developed for static word embeddings. Historically, much effort has gone into debiasing static pre-trained word embeddings (see \cite{Tolga:2016, Zhao2018:GNglove, Sun:2019:review}). Applying something akin to hard debias \cite{Tolga:2016} to the final sentence representation output by a language model \cite{liang-etal-2020-towards-sent-debias, Bhardwaj2021Jul} has been suggested as a way to create debiased contextual sentence representations. Intrinsic bias within that sentence embedding can be quantified using a cosine-similarity-based measure \cite{may-etal-2019-measuring, kurita-etal-2019-measuring}. However, these authors acknowledge that such parameter-based measures may be unreliable indications of intrinsic bias in the language model at large. Here we report intrinsic bias using a prediction-based measure (NSP) only. &#10;&#10;StereoSet \cite{nadeem-etal-2021-stereoset} is currently a leading test set for reporting on intrinsic bias in BERT, as observed through the NSP task (note that StereoSet contains test sets for both language modelling and NSP, but here we focus on NSP only). &#10;However, recent concerns \cite{blodgett-etal-2021-stereotyping} motivate a very careful application of StereoSet. &#10;Here, we provide a critical analysis of both StereoSet's content and intended bias measure. The outcome of this discussion is an enhanced version of StereoSet, with alternative bias measures. &#10;&#10;Next, we investigate projective debiasing techniques applied to BERT's hidden representations, including an intervention within the attention mechanism. &#10;Previously, debias-by-projection has been applied to the final output sentence representation only \cite{liang-etal-2020-towards-sent-debias, Bhardwaj2021Jul}, but has not yet been attempted within BERT's inner layers. Furthermore, we experiment with the use of information weighting \cite{dawkins-2021-marked} paired with the use of higher-dimensional gender subspaces. We show that projective debiasing techniques can successfully mitigate the intrinsic bias, as measured by the enhanced StereoSet,&#10;and make some key observations on how to combine the aforementioned ingredients. &#10;&#10;Lastly, we report on the same projective debiasing techniques applied to BERT after fine-tuning for an NLI task. &#10;We find that intrinsic bias mitigation is not necessarily correlated with our specific bias of interest in the fine-tuned downstream setting. &#10;That is, it is not sufficient to show reduced intrinsic bias on StereoSet as evidence that some debiasing scheme is superior for all applications. &#10;This is a crucial observation since debiasing schemes for pre-trained language models are typically evaluated on prediction-based intrinsic tasks only \cite{meade-etal-2022-empirical}.&#10;That said, our proposed projective debiasing techniques can still be quite effective on our downstream test case, simply by using different hyper-parameter settings. In a related work, \cite{jin-etal-2021-transferability} find that bias mitigation by finetuning an upstream model can be transferred to the downstream setting. Here we focus on projective methods that do not need any bias datasets for finetuning. &#10;&#10;We conclude that engineering a debiased-BERT will require a task-specific development set for the purpose of hyper-parameter selection. &#10;Our proposed techniques are well-suited for this situation as they require only a very small handful of inputs to be fed forward through the model once (i.e.\ debiasing parameters are fast to find and apply). &#10;Further, we discuss how our observations can help limit the hyper-parameter search space and allow for even faster model selection. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Gender Bias Mitigation, Computer Science, Language Model Debiasing, Linguistics, Artificial Intelligence, Natural Language Processing, Psychology, NLP Bias Assessment" />
        </attvalues>
      </node>
      <node id="2305.03514" label="2305.03514">
        <attvalues>
          <attvalue for="0" value="Can Large Language Models Transform Computational Social Science?" />
          <attvalue for="1" value="  Large Language Models (LLMs) are capable of successfully performing many&#10;language processing tasks zero-shot (without training data). If zero-shot LLMs&#10;can also reliably classify and explain social phenomena like persuasiveness and&#10;political ideology, then LLMs could augment the Computational Social Science&#10;(CSS) pipeline in important ways. This work provides a road map for using LLMs&#10;as CSS tools. Towards this end, we contribute a set of prompting best practices&#10;and an extensive evaluation pipeline to measure the zero-shot performance of 13&#10;language models on 25 representative English CSS benchmarks. On taxonomic&#10;labeling tasks (classification), LLMs fail to outperform the best fine-tuned&#10;models but still achieve fair levels of agreement with humans. On free-form&#10;coding tasks (generation), LLMs produce explanations that often exceed the&#10;quality of crowdworkers' gold references. We conclude that the performance of&#10;today's LLMs can augment the CSS research pipeline in two ways: (1) serving as&#10;zero-shot data annotators on human annotation teams, and (2) bootstrapping&#10;challenging creative generation tasks (e.g., explaining the underlying&#10;attributes of a text). In summary, LLMs are posed to meaningfully participate&#10;in social science analysis in partnership with humans.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.01172" label="1904.01172">
        <attvalues>
          <attvalue for="0" value="Recent Advances in Natural Language Inference: A Survey of Benchmarks,&#10;  Resources, and Approaches" />
          <attvalue for="1" value="  In the NLP community, recent years have seen a surge of research activities&#10;that address machines' ability to perform deep language understanding which&#10;goes beyond what is explicitly stated in text, rather relying on reasoning and&#10;knowledge of the world. Many benchmark tasks and datasets have been created to&#10;support the development and evaluation of such natural language inference&#10;ability. As these benchmarks become instrumental and a driving force for the&#10;NLP research community, this paper aims to provide an overview of recent&#10;benchmarks, relevant knowledge resources, and state-of-the-art learning and&#10;inference approaches in order to support a better understanding of this growing&#10;field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08731" label="2104.08731">
        <attvalues>
          <attvalue for="0" value="Can NLI Models Verify QA Systems' Predictions?" />
          <attvalue for="1" value="  To build robust question answering systems, we need the ability to verify&#10;whether answers to questions are truly correct, not just &quot;good enough&quot; in the&#10;context of imperfect QA datasets. We explore the use of natural language&#10;inference (NLI) as a way to achieve this goal, as NLI inherently requires the&#10;premise (document context) to contain all necessary information to support the&#10;hypothesis (proposed answer to the question). We leverage large pre-trained&#10;models and recent prior datasets to construct powerful question converter and&#10;decontextualization modules, which can reformulate QA instances as&#10;premise-hypothesis pairs with very high reliability. Then, by combining&#10;standard NLI datasets with NLI examples automatically derived from QA training&#10;data, we can train NLI models to judge the correctness of QA models' proposed&#10;answers. We show that our NLI approach can generally improve the confidence&#10;estimation of a QA model across different domains, evaluated in a selective QA&#10;setting. Careful manual analysis over the predictions of our NLI model shows&#10;that it can further identify cases where the QA model produces the right answer&#10;for the wrong reason, or where the answer cannot be verified as addressing all&#10;aspects of the question.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;NLI for Downstream Tasks ~\newcite{welleck2019dialogue} proposed a dialogue-based NLI dataset and the NLI model trained over it improved the consistency of a dialogue system; ~\newcite{pasunuru2017towards, li2018ensure, falke2019ranking} used NLI models to detect factual errors in abstractive summaries. For question answering, ~\newcite{harabagiu-hickl-2006-methods} showed that textual entailment can be used to enhance the accuracy of the open-domain QA systems; ~\newcite{trivedi2019repurposing} used a pretrained NLI model to select relevant sentences for multi-hop question answering; ~\newcite{yin-etal-2020-universal} tested whether NLI models generalize to QA setting in a few-shot learning scenario.&#10;&#10;Our work is most relevant to ~\newcite{mishra-etal-2021-looking}; they also learn an NLI model using examples generated from QA datasets. Our work differs from theirs in a few chief ways. First, we improve the conversion pipeline significantly with decontextualization and a better question converter. Second, we use this framework to improve QA performance by using NLI as a verifier, which is only possible because the decontextualization allows us to focus on a single sentence. We also study whether the converted dataset is compatible with other off-the-shelf NLI datasets. By contrast, ~\newcite{mishra-etal-2021-looking} use their converted NLI dataset to aid other tasks such as fact-checking. Finally, the contrast we establish here allows us to conduct a thorough human analysis over the converted NLI data and show how the task specifications of NLI and QA are different (Section~\ref{sec:error_analysis_entailment}). &#10;&#10;Robust Question Answering&#10;Modern QA systems often give incorrect answers in challenging settings that require generalization~\cite{rajpurkar2018know, chen2019understanding, wallace-etal-2019-universal, gardner-etal-2020-evaluating, kaushik2019learning}. Models focusing on robustness and generalizability have been proposed in recent years: ~\newcite{wang2018robust,khashabi2020more,liu2020robust} use perturbation based methods and adversarial training; ~\newcite{lewis2018generative} propose generative QA to prevent the model from overfitting to simple patterns; ~\newcite{yeh2019qainfomax, zhou2020robust} use advanced regularizers; ~\newcite{clark2019don} debias the training set through ensemble-based training; and ~\newcite{chen2021robust} incorporate an explicit graph alignment procedure.&#10;&#10;Another line of work to make models more robust is by introducing answer verification~\cite{hu2019read+,kamath-etal-2020-selective,wang-etal-2020-answer-better,zhang-etal-2021-knowing} as a final step for question answering models. Our work is in the same vein, but has certain advantages from using an NLI model. First, the answer verification process is more explicit so that one is able to spot where the error emerges. Second, we can incorporate NLI datasets from other domains into the training of our verifier, reducing reliance on in-domain labeled QA data.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2003.02126" label="2003.02126">
        <attvalues>
          <attvalue for="0" value="Sequential Neural Networks for Noetic End-to-End Response Selection" />
          <attvalue for="1" value="  The noetic end-to-end response selection challenge as one track in the 7th&#10;Dialog System Technology Challenges (DSTC7) aims to push the state of the art&#10;of utterance classification for real world goal-oriented dialog systems, for&#10;which participants need to select the correct next utterances from a set of&#10;candidates for the multi-turn context. This paper presents our systems that are&#10;ranked top 1 on both datasets under this challenge, one focused and small&#10;(Advising) and the other more diverse and large (Ubuntu). Previous&#10;state-of-the-art models use hierarchy-based (utterance-level and token-level)&#10;neural networks to explicitly model the interactions among different turns'&#10;utterances for context modeling. In this paper, we investigate a sequential&#10;matching model based only on chain sequence for multi-turn response selection.&#10;Our results demonstrate that the potentials of sequential matching approaches&#10;have not yet been fully exploited in the past for multi-turn response&#10;selection. In addition to ranking top 1 in the challenge, the proposed model&#10;outperforms all previous models, including state-of-the-art hierarchy-based&#10;models, on two large-scale public multi-turn response selection benchmark&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12864" label="2010.12864">
        <attvalues>
          <attvalue for="0" value="On Transferability of Bias Mitigation Effects in Language Model&#10;  Fine-Tuning" />
          <attvalue for="1" value="  Fine-tuned language models have been shown to exhibit biases against&#10;protected groups in a host of modeling tasks such as text classification and&#10;coreference resolution. Previous works focus on detecting these biases,&#10;reducing bias in data representations, and using auxiliary training objectives&#10;to mitigate bias during fine-tuning. Although these techniques achieve bias&#10;reduction for the task and domain at hand, the effects of bias mitigation may&#10;not directly transfer to new tasks, requiring additional data collection and&#10;customized annotation of sensitive attributes, and re-evaluation of appropriate&#10;fairness metrics. We explore the feasibility and benefits of upstream bias&#10;mitigation (UBM) for reducing bias on downstream tasks, by first applying bias&#10;mitigation to an upstream model through fine-tuning and subsequently using it&#10;for downstream fine-tuning. We find, in extensive experiments across hate&#10;speech detection, toxicity detection, occupation prediction, and coreference&#10;resolution tasks over various bias factors, that the effects of UBM are indeed&#10;transferable to new downstream tasks or domains via fine-tuning, creating less&#10;biased downstream models than directly fine-tuning on the downstream task or&#10;transferring from a vanilla upstream model. Though challenges remain, we show&#10;that UBM promises more efficient and accessible bias mitigation in LM&#10;fine-tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.06733" label="2112.06733">
        <attvalues>
          <attvalue for="0" value="Measuring Context-Word Biases in Lexical Semantic Datasets" />
          <attvalue for="1" value="  State-of-the-art pretrained contextualized models (PCM) eg. BERT use tasks&#10;such as WiC and WSD to evaluate their word-in-context representations. This&#10;inherently assumes that performance in these tasks reflect how well a model&#10;represents the coupled word and context semantics. We question this assumption&#10;by presenting the first quantitative analysis on the context-word interaction&#10;being tested in major contextual lexical semantic tasks. To achieve this, we&#10;run probing baselines on masked input, and propose measures to calculate and&#10;visualize the degree of context or word biases in existing datasets. The&#10;analysis was performed on both models and humans. Our findings demonstrate that&#10;models are usually not being tested for word-in-context semantics in the same&#10;way as humans are in these tasks, which helps us better understand the&#10;model-human gap. Specifically, to PCMs, most existing datasets fall into the&#10;extreme ends (the retrieval-based tasks exhibit strong target word bias while&#10;WiC-style tasks and WSD show strong context bias); In comparison, humans are&#10;less biased and achieve much better performance when both word and context are&#10;available than with masked input. We recommend our framework for understanding&#10;and controlling these biases for model interpretation and future task design.&#10;" />
          <attvalue for="2" value="&#10;&#10;Meaning contextualization (i.e., identifying the correct meaning of a target word in linguistic context) is essential for understanding natural language, and has been the focus in many lexical semantic tasks. Pretrained contextualized models (PCMs) have brought large improvements in these tasks including WSD \cite{hadiwinoto-etal-2019-improved,loureiro-jorge-2019-language,huang-etal-2019-glossbert,blevins-zettlemoyer-2020-moving}, WiC \cite{pilehvar-camacho-collados-2019-wic,gari-soler-etal-2019-word} and entity linking (EL) \cite{wu-etal-2020-scalable,broscheit-2019-investigating}. &#10;&#10;These superior performances have been taken as proof that PCMs can successfully model word-in-context semantics. Many studies have investigated the process of lexical contextualization in these PCMs. Specifically, \cite{vulic-etal-2020-probing, aina-etal-2019-putting} found language models `contextualize' words in higher layers while the type-level information is better kept in lower layers. \cite{voita-etal-2019-bottom} point out different learning objectives affect the contextualization process, and \cite{gari-soler-apidianaki-2021-lets, pimentel-etal-2020-speakers} show PCMs can capture words' ambiguity levels. &#10;&#10;While most these studies have focused on probing the innerworkings of the PCM feature space, there is no systematic study to quantify the word-context interaction (either learned by PCMs or intrinsic) across different lexical semantic tasks. On one hand, these datasets often vary in their emphasis on context vs target words. For example, we could expect tasks such as WSD and WiC to rely more on context by design as the target words are either given or the same in each input pair. On the other hand, models may find shortcuts from datasets to avoid learning the complex word-context interaction. {What is missing in the current literature is an accurate quantification of this word-context interplay being tested in each task so that we can fully understand task goals and model performance.} In particular, we need to flag the situation where a model can solve a task by relying solely on context or the target words. Such heavy word or context reliance hinders a scientific assessment of the models' meaning contextualization abilities as it essentially bypasses the key word-context interaction challenge in &#10;human understanding of lexical semantics.&#10;Therefore, we refer to such heavy reliance on target words or context in a contextual lexical semantic dataset as target word biases or context biases.&#10;&#10;This study presents an analysis framework to quantify this context-word interaction by measuring context and target word biases across lexical semantic tasks. We first run controlled probing baselines by masking the input to show the context or the target word alone. Based on model's performance on these probing baselines, we calculate two ratios that reflect how much of the model performance in this dataset can be achieved from simply relying on context alone or the target word alone, i.e. the degree of context or target word biases (See Figure~\ref{fig:main} which will be discussed fully in \ref{sec:results}). The design of the probing baselines follows previous studies that applied input perturbation techniques for model and task analysis in GLUE \cite{pham2020out}, NLI \cite{poliak-etal-2018-hypothesis,wang-etal-2018-glue,talman2021nli} and relation extraction \cite{peng-etal-2020-learning}. While previous probing studies usually assume no meaningful information from corrupted input with no human verification, we provide fairer comparison with model performance by collecting human judgment on the same masked input in four tasks. Such comparison reveals whether the biases are learned by models from the datasets or are inherent in the tasks.&#10;&#10;Our key findings are (1) the tasks can be clearly divided into target-word-biased (the retrieval-based tasks), and context-biased (WiC-style tasks and WSD). Among the retrieval-based tasks, domain affects ambiguity level and thus the target word bias: models even achieve the best performance using target words alone in the medical domain. (2) AM$^2$iCo and Sense Retrieval show less extreme model biases and challenge a model more to represent both the context and target words; and (3) a similar trend of biases exists in humans but is much less extreme, as humans find semantic judgment more difficult on masked input and require both word and context to do well in each task. This analysis helps us better understand the nuanced differences between models and humans in existing tasks, and we recommend the framework to be applied when designing new datasets to check whether word and context are required and whether the models rely on the coupled word and context semantics in a similar way to humans.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Lexical Semantic Tasks, Computer Science, Human-Model Comparison, Linguistics, Cognitive Science, Contextualized Language Models, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1906.10007" label="1906.10007">
        <attvalues>
          <attvalue for="0" value="Language Modelling Makes Sense: Propagating Representations through&#10;  WordNet for Full-Coverage Word Sense Disambiguation" />
          <attvalue for="1" value="  Contextual embeddings represent a new generation of semantic representations&#10;learned from Neural Language Modelling (NLM) that addresses the issue of&#10;meaning conflation hampering traditional word embeddings. In this work, we show&#10;that contextual embeddings can be used to achieve unprecedented gains in Word&#10;Sense Disambiguation (WSD) tasks. Our approach focuses on creating sense-level&#10;embeddings with full-coverage of WordNet, and without recourse to explicit&#10;knowledge of sense distributions or task-specific modelling. As a result, a&#10;simple Nearest Neighbors (k-NN) method using our representations is able to&#10;consistently surpass the performance of previous systems using powerful neural&#10;sequencing models. We also analyse the robustness of our approach when ignoring&#10;part-of-speech and lemma features, requiring disambiguation against the full&#10;sense inventory, and revealing shortcomings to be improved. Finally, we explore&#10;applications of our sense embeddings for concept-level analyses of contextual&#10;embeddings and their respective NLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.07245" label="1908.07245">
        <attvalues>
          <attvalue for="0" value="GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge" />
          <attvalue for="1" value="  Word Sense Disambiguation (WSD) aims to find the exact sense of an ambiguous&#10;word in a particular context. Traditional supervised methods rarely take into&#10;consideration the lexical resources like WordNet, which are widely utilized in&#10;knowledge-based methods. Recent studies have shown the effectiveness of&#10;incorporating gloss (sense definition) into neural networks for WSD. However,&#10;compared with traditional word expert supervised methods, they have not&#10;achieved much improvement. In this paper, we focus on how to better leverage&#10;gloss knowledge in a supervised neural WSD system. We construct context-gloss&#10;pairs and propose three BERT-based models for WSD. We fine-tune the pre-trained&#10;BERT model on SemCor3.0 training corpus and the experimental results on several&#10;English all-words WSD benchmark datasets show that our approach outperforms the&#10;state-of-the-art systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.08377" label="1905.08377">
        <attvalues>
          <attvalue for="0" value="Word Usage Similarity Estimation with Sentence Representations and&#10;  Automatic Substitutes" />
          <attvalue for="1" value="  Usage similarity estimation addresses the semantic proximity of word&#10;instances in different contexts. We apply contextualized (ELMo and BERT) word&#10;and sentence embeddings to this task, and propose supervised models that&#10;leverage these representations for prediction. Our models are further assisted&#10;by lexical substitute annotations automatically assigned to word instances by&#10;context2vec, a neural model that relies on a bidirectional LSTM. We perform an&#10;extensive comparison of existing word and sentence representations on benchmark&#10;datasets addressing both graded and binary similarity. The best performing&#10;models outperform previous methods in both settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03814" label="1911.03814">
        <attvalues>
          <attvalue for="0" value="Scalable Zero-shot Entity Linking with Dense Entity Retrieval" />
          <attvalue for="1" value="  This paper introduces a conceptually simple, scalable, and highly effective&#10;BERT-based entity linking model, along with an extensive evaluation of its&#10;accuracy-speed trade-off. We present a two-stage zero-shot linking algorithm,&#10;where each entity is defined only by a short textual description. The first&#10;stage does retrieval in a dense space defined by a bi-encoder that&#10;independently embeds the mention context and the entity descriptions. Each&#10;candidate is then re-ranked with a cross-encoder, that concatenates the mention&#10;and entity text. Experiments demonstrate that this approach is state of the art&#10;on recent zero-shot benchmarks (6 point absolute gains) and also on more&#10;established non-zero-shot evaluations (e.g. TACKBP-2010), despite its relative&#10;simplicity (e.g. no explicit entity embeddings or manually engineered mention&#10;tables). We also show that bi-encoder linking is very fast with nearest&#10;neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds),&#10;and that much of the accuracy gain from the more expensive cross-encoder can be&#10;transferred to the bi-encoder via knowledge distillation. Our code and models&#10;are available at https://github.com/facebookresearch/BLINK.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.05473" label="2003.05473">
        <attvalues>
          <attvalue for="0" value="Investigating Entity Knowledge in BERT with Simple Neural End-To-End&#10;  Entity Linking" />
          <attvalue for="1" value="  A typical architecture for end-to-end entity linking systems consists of&#10;three steps: mention detection, candidate generation and entity disambiguation.&#10;In this study we investigate the following questions: (a) Can all those steps&#10;be learned jointly with a model for contextualized text-representations, i.e.&#10;BERT (Devlin et al., 2019)? (b) How much entity knowledge is already contained&#10;in pretrained BERT? (c) Does additional entity knowledge improve BERT's&#10;performance in downstream tasks? To this end, we propose an extreme&#10;simplification of the entity linking setup that works surprisingly well: simply&#10;cast it as a per token classification over the entire entity vocabulary (over&#10;700K classes in our case). We show on an entity linking benchmark that (i) this&#10;model improves the entity representations over plain BERT, (ii) that it&#10;outperforms entity linking architectures that optimize the tasks separately and&#10;(iii) that it only comes second to the current state-of-the-art that does&#10;mention detection and entity disambiguation jointly. Additionally, we&#10;investigate the usefulness of entity-aware token-representations in the&#10;text-understanding benchmark GLUE, as well as the question answering benchmarks&#10;SQUAD V2 and SWAG and also the EN-DE WMT14 machine translation benchmark. To&#10;our surprise, we find that most of those benchmarks do not benefit from&#10;additional entity knowledge, except for a task with very small training data,&#10;the RTE task in GLUE, which improves by 2%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.01380" label="1909.01380">
        <attvalues>
          <attvalue for="0" value="The Bottom-up Evolution of Representations in the Transformer: A Study&#10;  with Machine Translation and Language Modeling Objectives" />
          <attvalue for="1" value="  We seek to understand how the representations of individual tokens and the&#10;structure of the learned feature space evolve between layers in deep neural&#10;networks under different learning objectives. We focus on the Transformers for&#10;our analysis as they have been shown effective on various tasks, including&#10;machine translation (MT), standard left-to-right language models (LM) and&#10;masked language modeling (MLM). Previous work used black-box probing tasks to&#10;show that the representations learned by the Transformer differ significantly&#10;depending on the objective. In this work, we use canonical correlation analysis&#10;and mutual information estimators to study how information flows across&#10;Transformer layers and how this process depends on the choice of learning&#10;objective. For example, as you go from bottom to top layers, information about&#10;the past in left-to-right language models gets vanished and predictions about&#10;the future get formed. In contrast, for MLM, representations initially acquire&#10;information about the context around the token, partially forgetting the token&#10;identity and producing a more generalized token representation. The token&#10;identity then gets recreated at the top MLM layers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.14694" label="2104.14694">
        <attvalues>
          <attvalue for="0" value="Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and&#10;  Partitionability into Senses" />
          <attvalue for="1" value="  Pre-trained language models (LMs) encode rich information about linguistic&#10;structure but their knowledge about lexical polysemy remains unclear. We&#10;propose a novel experimental setup for analysing this knowledge in LMs&#10;specifically trained for different languages (English, French, Spanish and&#10;Greek) and in multilingual BERT. We perform our analysis on datasets carefully&#10;designed to reflect different sense distributions, and control for parameters&#10;that are highly correlated with polysemy such as frequency and grammatical&#10;category. We demonstrate that BERT-derived representations reflect words'&#10;polysemy level and their partitionability into senses. Polysemy-related&#10;information is more clearly present in English BERT embeddings, but models in&#10;other languages also manage to establish relevant distinctions between words at&#10;different polysemy levels. Our results contribute to a better understanding of&#10;the knowledge encoded in contextualised representations and open up new avenues&#10;for multilingual lexical semantics research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The knowledge pre-trained contextual LMs encode about lexical semantics &#10;has only recently started being explored. &#10;Works by \cite{reif2019visualizing} and \cite{wiedemann2019does} propose experiments using representations built from Wikipedia and the SemCor corpus \cite{miller-etal-1993-semantic}, and show that BERT can organise word usages in the semantic space in a way that reflects the meaning distinctions present in the data. &#10;It is also shown that BERT can perform well in the word sense disambiguation (WSD) task by leveraging the sense-related information available in these resources. &#10;These works address the disambiguation capabilities of the model but do not show what BERT actually knows about words' polysemy, which is the main axis of our work. In our experiments, sense annotations are {\underline{not}} used to guide the models into establishing sense distinctions, but rather for creating controlled conditions that allow us to analyse BERT's inherent knowledge of lexical polysemy. &#10;&#10;&#10;&#10;Probing has also been proposed for lexical semantics analysis, but addressing different questions than the ones posed in our work. \cite{aina-etal-2019-putting} probe the hidden representations of a bidirectional (bi-LSTM) LM for lexical (type-level) and contextual (token-level) information. They specifically train diagnostic classifiers on the tasks of retrieving the input embedding of a word and a representation of its contextual meaning (as reflected in its lexical substitutes). The results show that the information about the input word that is present in LSTM representations is not lost after contextualisation; however, the quality of the information available for a word is assessed through the model's ability to identify the corresponding embedding, as in \cite{Adi2016FinegrainedAO} and \cite{conneau-etal-2018-cram}. Also, lexical ambiguity is only viewed through the lens of contextualisation.&#10;In our work, on the contrary, it is given a central role: we explicitly address the knowledge BERT encodes about words' degree of polysemy and partitionability into senses. &#10;\cite{vulic2020probing} also propose to probe &#10;contextualised models for lexical semantics, &#10;but they do so using ``static'' word embeddings obtained through pooling over several contexts, or extracting representations for words in isolation and from BERT's embedding layer, before contextualisation. &#10;These representations are evaluated on tasks traditionally used for assessing the quality of static embeddings, such as out-of-context similarity and word analogy, which are not tailored for addressing lexical polysemy.&#10;Other contemporaneous work explores lexical polysemy in static embeddings \cite{jakubowski-etal-2020-topology}, and the relation of ambiguity and context uncertainty as approximated in the space constructed by mBERT using information-theoretic measures \cite{pimentel-etal-2020-speakers}.&#10;Finally, work by \cite{ethayarajh-2019-contextual} provides useful observations regarding the impact of context on the representations, without explicitly addressing the semantic knowledge encoded by the models. &#10;Through an exploration of BERT, ELMo and GPT-2 \cite{radford2019language}, the author highlights the highly distorted similarity of the obtained contextualised representations which is due to the anisotropy of the vector space built by each model.&#10;The question of meaning is not addressed in this work, making it hard to draw any conclusions about lexical polysemy. &#10;&#10;Our proposed experimental setup is aimed at investigating the polysemy information encoded in the representations built at different layers of deep pre-trained LMs. &#10;Our approach basically relies on the similarity of contextualised representations, which amounts to word usage similarity (Usim) estimation, a classical task in lexical semantics \cite{erketal2009,Huang:2012:IWR:2390524.2390645,erk2013measuring}. The Usim task precisely involves predicting the similarity of word instances in context without &#10;use of sense annotations. BERT has been shown to be particularly good at this task \cite{gari-soler-etal-2019-word,pilehvar-camacho-collados-2019-wic}. Our experiments allow to explore and understand what this ability is due to.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2012.15180" label="2012.15180">
        <attvalues>
          <attvalue for="0" value="Out of Order: How Important Is The Sequential Order of Words in a&#10;  Sentence in Natural Language Understanding Tasks?" />
          <attvalue for="1" value="  Do state-of-the-art natural language understanding models care about word&#10;order - one of the most important characteristics of a sequence? Not always! We&#10;found 75% to 90% of the correct predictions of BERT-based classifiers, trained&#10;on many GLUE tasks, remain constant after input words are randomly shuffled.&#10;Despite BERT embeddings are famously contextual, the contribution of each&#10;individual word to downstream tasks is almost unchanged even after the word's&#10;context is shuffled. BERT-based models are able to exploit superficial cues&#10;(e.g. the sentiment of keywords in sentiment analysis; or the word-wise&#10;similarity between sequence-pair inputs in natural language inference) to make&#10;correct decisions when tokens are arranged in random orders. Encouraging&#10;classifiers to capture word order information improves the performance on most&#10;GLUE tasks, SQuAD 2.0 and out-of-samples. Our work suggests that many GLUE&#10;tasks are not challenging machines to understand the meaning of a sentence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.01923" label="2010.01923">
        <attvalues>
          <attvalue for="0" value="Learning from Context or Names? An Empirical Study on Neural Relation&#10;  Extraction" />
          <attvalue for="1" value="  Neural models have achieved remarkable success on relation extraction (RE)&#10;benchmarks. However, there is no clear understanding which type of information&#10;affects existing RE models to make decisions and how to further improve the&#10;performance of these models. To this end, we empirically study the effect of&#10;two main information sources in text: textual context and entity mentions&#10;(names). We find that (i) while context is the main source to support the&#10;predictions, RE models also heavily rely on the information from entity&#10;mentions, most of which is type information, and (ii) existing datasets may&#10;leak shallow heuristics via entity mentions and thus contribute to the high&#10;performance on RE benchmarks. Based on the analyses, we propose an&#10;entity-masked contrastive pre-training framework for RE to gain a deeper&#10;understanding on both textual context and type information while avoiding rote&#10;memorization of entities or use of superficial cues in mentions. We carry out&#10;extensive experiments to support our views, and show that our framework can&#10;improve the effectiveness and robustness of neural models in different RE&#10;scenarios. All the code and datasets are released at&#10;https://github.com/thunlp/RE-Context-or-Names.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.16692" label="2308.16692">
        <attvalues>
          <attvalue for="0" value="SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language&#10;  Models" />
          <attvalue for="1" value="  Current speech large language models build upon discrete speech&#10;representations, which can be categorized into semantic tokens and acoustic&#10;tokens. However, existing speech tokens are not specifically designed for&#10;speech language modeling. To assess the suitability of speech tokens for&#10;building speech language models, we established the first benchmark,&#10;SLMTokBench. Our results indicate that neither semantic nor acoustic tokens are&#10;ideal for this purpose. Therefore, we propose SpeechTokenizer, a unified speech&#10;tokenizer for speech large language models. SpeechTokenizer adopts the&#10;Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying&#10;semantic and acoustic tokens, SpeechTokenizer disentangles different aspects of&#10;speech information hierarchically across different RVQ layers. Furthermore, We&#10;construct a Unified Speech Language Model (USLM) leveraging SpeechTokenizer.&#10;Experiments show that SpeechTokenizer performs comparably to EnCodec in speech&#10;reconstruction and demonstrates strong performance on the SLMTokBench&#10;benchmark. Also, USLM outperforms VALL-E in zero-shot Text-to-Speech tasks.&#10;Code and models are available at&#10;https://github.com/ZhangXInFD/SpeechTokenizer/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Oure related work is put in Appendix~\ref{sec:app:related}.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Deep Learning Architectures, Signal Processing, Speech Language Modeling, Tokenization Techniques" />
        </attvalues>
      </node>
      <node id="2407.10795" label="2407.10795">
        <attvalues>
          <attvalue for="0" value="Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping" />
          <attvalue for="1" value="  Decoding by contrasting layers (DoLa), is designed to improve the generation&#10;quality of large language models (LLMs) by contrasting the prediction&#10;probabilities between an early exit output (amateur logits) and the final&#10;output (expert logits). However, we find that this approach does not work well&#10;on non-English tasks. Inspired by previous interpretability work on language&#10;transition during the model's forward pass, we discover that this issue arises&#10;from a language mismatch between early exit output and final output. In this&#10;work, we propose an improved contrastive decoding algorithm that is effective&#10;for diverse languages beyond English. To obtain more helpful amateur logits, we&#10;devise two strategies to skip a set of bottom, language-agnostic layers based&#10;on our preliminary analysis. Experimental results on multilingual reasoning&#10;benchmarks demonstrate that our proposed method outperforms previous&#10;contrastive decoding baselines and substantially improves LLM's&#10;chain-of-thought reasoning accuracy across 11 languages. The project will be&#10;available at: https://github.com/NJUNLP/SkipLayerCD.&#10;" />
          <attvalue for="2" value="&#10;Contrastive decoding~\cite{li-etal-2023-contrastive} presents a novel approach to enhance the text generation quality of large language models.&#10;At each inference step, contrastive decoding uses logits generated by an amateur model (usually small) to contrast with the output logits of an expert model (usually large).&#10;This reduces the probability of the expert model to make similar mistakes as the amateur model, thus making the generation content more logical and coherent~\cite{li-etal-2023-contrastive, o2023contrastive, zhao2024enhancing}.&#10;To further eliminate the need of finding an extra amateur LLM, \cite{chuang2023dola} propose DoLa, which uses the expert model's early exit output as amateur logits.&#10;&#10;However, in this paper, we find that DoLa does not work well on non-English tasks.&#10;Inspired by the recent interpretability study by \cite{wendler2024llamas}, which analyzes the language transitions during the forward pass, we identify that the issue with DoLa arises from the language mismatch between amateur logits and expert logits.&#10;Specifically, the early exit logits accumulate on English tokens even during non-English generation, thus failing to provide a helpful contrastive distribution for the expert model.&#10;&#10;Contributions To obtain more helpful amateur logits, we propose an improved contrastive decoding algorithm by skipping a set of lower language-agnostic layers while preserving the computations in the upper transformer blocks (Figures~\ref{fig-method}).&#10;Specifically, we design two strategies to determine the positions for layer skipping: one based on heuristic rules, and the other based on entropy change. &#10;&#10;Our experimental results on multilingual reasoning benchmarks mGSM show that our devised approach significantly outperforms the previous contrastive decoding approach DoLa, and improves the chain-of-thought reasoning accuracy of a group of open-source LLMs: LLaMA2~\cite{llama2}, LLaMA3~\cite{meta2024introducing}, Mistral~\cite{jiang2023mistral}, etc., across 11 languages. &#10;The performance gap between our approach and DoLa on the multilingual benchmark also validates the findings about the language transition of intermediate decodings across the layers of LLMs by \cite{wendler2024llamas} and provides further insight into the working patterns of LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Multilingual Reasoning Benchmarks, Linguistics, Cognitive Science, Artificial Intelligence, Contrastive Decoding Algorithms, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2402.10588" label="2402.10588">
        <attvalues>
          <attvalue for="0" value="Do Llamas Work in English? On the Latent Language of Multilingual&#10;  Transformers" />
          <attvalue for="1" value="  We ask whether multilingual language models trained on unbalanced,&#10;English-dominated corpora use English as an internal pivot language -- a&#10;question of key importance for understanding how language models function and&#10;the origins of linguistic bias. Focusing on the Llama-2 family of transformer&#10;models, our study uses carefully constructed non-English prompts with a unique&#10;correct single-token continuation. From layer to layer, transformers gradually&#10;map an input embedding of the final prompt token to an output embedding from&#10;which next-token probabilities are computed. Tracking intermediate embeddings&#10;through their high-dimensional space reveals three distinct phases, whereby&#10;intermediate embeddings (1) start far away from output token embeddings; (2)&#10;already allow for decoding a semantically correct next token in the middle&#10;layers, but give higher probability to its version in English than in the input&#10;language; (3) finally move into an input-language-specific region of the&#10;embedding space. We cast these results into a conceptual model where the three&#10;phases operate in &quot;input space&quot;, &quot;concept space&quot;, and &quot;output space&quot;,&#10;respectively. Crucially, our evidence suggests that the abstract &quot;concept&#10;space&quot; lies closer to English than to other languages, which may have important&#10;consequences regarding the biases held by multilingual language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.02070" label="2208.02070">
        <attvalues>
          <attvalue for="0" value="Efficient Fine-Tuning of Compressed Language Models with Learners" />
          <attvalue for="1" value="  Fine-tuning BERT-based models is resource-intensive in memory, computation,&#10;and time. While many prior works aim to improve inference efficiency via&#10;compression techniques, e.g., pruning, these works do not explicitly address&#10;the computational challenges of training to downstream tasks. We introduce&#10;Learner modules and priming, novel methods for fine-tuning that exploit the&#10;overparameterization of pre-trained language models to gain benefits in&#10;convergence speed and resource utilization. Learner modules navigate the double&#10;bind of 1) training efficiently by fine-tuning a subset of parameters, and 2)&#10;training effectively by ensuring quick convergence and high metric scores. Our&#10;results on DistilBERT demonstrate that learners perform on par with or surpass&#10;the baselines. Learners train 7x fewer parameters than state-of-the-art methods&#10;on GLUE. On CoLA, learners fine-tune 20% faster, and have significantly lower&#10;resource utilization.&#10;" />
          <attvalue for="2" value="&#10;Transformer-based Pre-trained Language Models (PLM) have become ubiquitous in Natural Language Processing (NLP). BERT and its various derivatives outperform the previous generations of NLP models significantly, requiring in contrast many more parameters and, consequently, more powerful hardware resources for training and inference \cite{energy_in_NLP, green_ai, devlin2018bert}. Works in model compression have sought to improve inference efficiency of these models by pruning, quantization, and distillation \cite{Ganesh2021CompressingLT, survey_on_device_ML}. Fine-tuning is, however, complicated by the size of PLMs \cite{energy_in_NLP, green_ai}. For example, BERT and DistilBERT have parameter counts of 110 million and 66 million parameters respectively \cite{devlin2018bert, distilbert}. Fine-tuning such large models requires substantial data and considerably more computations and memory accesses than inference \cite{tinytl, green_ai, energy_in_NLP}. Further, memory operations are energy intensive and slow \cite{songhan_eie, comp_arch_hennessy_patterson_2012}. Fine-tuning must be made fast and resource-efficient to facilitate use-cases such as personalized auto-correct systems on mobile device keyboards, or green AI \cite{federated_mobile_keyboard_example, survey_on_device_ML, green_ai}.&#10;&#10;Prior works in efficient training focus primarily on parameter efficiency for uncompressed (i.e., huge) transformer-based models \cite{bitfit, houlsby2019parameter_eff_adapter, he2022towards_unified_param_eff_parallel_adapter, guo-etal-2021-parameter-diff-pruning}. Adapters \cite{houlsby2019parameter_eff_adapter} manage to achieve parameter efficiency in a computationally efficient manner, while difference pruning \cite{guo-etal-2021-parameter-diff-pruning} requires triple the training-time parameter usage. We show that the computational efficiency of adapters is undermined on a compressed model by slow convergence. State-of-the-art methods in efficient fine-tuning, namely Freeze-and-Reconfigure (FAR), are decidedly quick to converge, but require a far greater proportion of parameters and are plagued by slow memory operations \cite{FAR}. The optimal efficient fine-tuning technique must then satisfy a few requirements: 1) demonstrate quick convergence, and 2) be resource efficient in memory, computation, and time.&#10;&#10;Our main contribution is the proposal of Learner modules fine-tuned with priming steps, which satisfy these requirements by: a) exploiting pre-trained parameters for their quick convergence, b) exploiting model overparameterization to train a small subset of parameters, and c) avoiding slow, complex memory operations. Learner modules, as illustrated in Figure \ref{figure_compare_learner}, are added in parallel to each linear layer in a target model. Learners consist of a low-rank projection matrix whose product with module input is added to the output of the linear layer. In essence, the small learner module learns for the much larger linear layer. After training, the learner modules can be collapsed leaving just the original model, unlike adapters which permanently add modules to the architecture. In this work we demonstrate the importance of convergence for efficient fine-tuning. We show that learner modules outperform adapters by 3 GLUE points while adding a similar number of parameters, because of their quicker convergence. We also demonstrate that learners perform on par with FAR while training 7x fewer parameters, and fine-tuning 20\% faster.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Efficient Fine-Tuning Methods, Mathematics, Language Model Optimization, Resource Utilization Reduction" />
        </attvalues>
      </node>
      <node id="2002.11985" label="2002.11985">
        <attvalues>
          <attvalue for="0" value="Compressing Large-Scale Transformer-Based Models: A Case Study on BERT" />
          <attvalue for="1" value="  Pre-trained Transformer-based models have achieved state-of-the-art&#10;performance for various Natural Language Processing (NLP) tasks. However, these&#10;models often have billions of parameters, and, thus, are too resource-hungry&#10;and computation-intensive to suit low-capability devices or applications with&#10;strict latency requirements. One potential remedy for this is model&#10;compression, which has attracted a lot of research attention. Here, we&#10;summarize the research in compressing Transformers, focusing on the especially&#10;popular BERT model. In particular, we survey the state of the art in&#10;compression for BERT, we clarify the current best practices for compressing&#10;large-scale Transformer models, and we provide insights into the workings of&#10;various methods. Our categorization and analysis also shed light on promising&#10;future research directions for achieving lightweight, accurate, and generic NLP&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1602.01528" label="1602.01528">
        <attvalues>
          <attvalue for="0" value="EIE: Efficient Inference Engine on Compressed Deep Neural Network" />
          <attvalue for="1" value="  State-of-the-art deep neural networks (DNNs) have hundreds of millions of&#10;connections and are both computationally and memory intensive, making them&#10;difficult to deploy on embedded systems with limited hardware resources and&#10;power budgets. While custom hardware helps the computation, fetching weights&#10;from DRAM is two orders of magnitude more expensive than ALU operations, and&#10;dominates the required power.&#10;  Previously proposed 'Deep Compression' makes it possible to fit large DNNs&#10;(AlexNet and VGGNet) fully in on-chip SRAM. This compression is achieved by&#10;pruning the redundant connections and having multiple connections share the&#10;same weight. We propose an energy efficient inference engine (EIE) that&#10;performs inference on this compressed network model and accelerates the&#10;resulting sparse matrix-vector multiplication with weight sharing. Going from&#10;DRAM to SRAM gives EIE 120x energy saving; Exploiting sparsity saves 10x;&#10;Weight sharing gives 8x; Skipping zero activations from ReLU saves another 3x.&#10;Evaluated on nine DNN benchmarks, EIE is 189x and 13x faster when compared to&#10;CPU and GPU implementations of the same DNN without compression. EIE has a&#10;processing power of 102GOPS/s working directly on a compressed network,&#10;corresponding to 3TOPS/s on an uncompressed network, and processes FC layers of&#10;AlexNet at 1.88x10^4 frames/sec with a power dissipation of only 600mW. It is&#10;24,000x and 3,400x more energy efficient than a CPU and GPU respectively.&#10;Compared with DaDianNao, EIE has 2.9x, 19x and 3x better throughput, energy&#10;efficiency and area efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.03604" label="1811.03604">
        <attvalues>
          <attvalue for="0" value="Federated Learning for Mobile Keyboard Prediction" />
          <attvalue for="1" value="  We train a recurrent neural network language model using a distributed,&#10;on-device learning framework called federated learning for the purpose of&#10;next-word prediction in a virtual keyboard for smartphones. Server-based&#10;training using stochastic gradient descent is compared with training on client&#10;devices using the Federated Averaging algorithm. The federated algorithm, which&#10;enables training on a higher-quality dataset for this use case, is shown to&#10;achieve better prediction recall. This work demonstrates the feasibility and&#10;benefit of training language models on client devices without exporting&#10;sensitive user data to servers. The federated learning environment gives users&#10;greater control over the use of their data and simplifies the task of&#10;incorporating privacy by default with distributed training and aggregation&#10;across a population of client devices.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.01541" label="2205.01541">
        <attvalues>
          <attvalue for="0" value="Efficient Fine-Tuning of BERT Models on the Edge" />
          <attvalue for="1" value="  Resource-constrained devices are increasingly the deployment targets of&#10;machine learning applications. Static models, however, do not always suffice&#10;for dynamic environments. On-device training of models allows for quick&#10;adaptability to new scenarios. With the increasing size of deep neural&#10;networks, as noted with the likes of BERT and other natural language processing&#10;models, comes increased resource requirements, namely memory, computation,&#10;energy, and time. Furthermore, training is far more resource intensive than&#10;inference. Resource-constrained on-device learning is thus doubly difficult,&#10;especially with large BERT-like models. By reducing the memory usage of&#10;fine-tuning, pre-trained BERT models can become efficient enough to fine-tune&#10;on resource-constrained devices. We propose Freeze And Reconfigure (FAR), a&#10;memory-efficient training regime for BERT-like models that reduces the memory&#10;usage of activation maps during fine-tuning by avoiding unnecessary parameter&#10;updates. FAR reduces fine-tuning time on the DistilBERT model and CoLA dataset&#10;by 30%, and time spent on memory operations by 47%. More broadly, reductions in&#10;metric performance on the GLUE and SQuAD datasets are around 1% on average.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.03900" label="2408.03900">
        <attvalues>
          <attvalue for="0" value="Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond" />
          <attvalue for="1" value="  We present Speech-MASSIVE, a multilingual Spoken Language Understanding (SLU)&#10;dataset comprising the speech counterpart for a portion of the MASSIVE textual&#10;corpus. Speech-MASSIVE covers 12 languages from different families and inherits&#10;from MASSIVE the annotations for the intent prediction and slot-filling tasks.&#10;Our extension is prompted by the scarcity of massively multilingual SLU&#10;datasets and the growing need for versatile speech datasets to assess&#10;foundation models (LLMs, speech encoders) across languages and tasks. We&#10;provide a multimodal, multitask, multilingual dataset and report SLU baselines&#10;using both cascaded and end-to-end architectures in various training scenarios&#10;(zero-shot, few-shot, and full fine-tune). Furthermore, we demonstrate the&#10;suitability of Speech-MASSIVE for benchmarking other tasks such as speech&#10;transcription, language identification, and speech translation. The dataset,&#10;models, and code are publicly available at:&#10;https://github.com/hlt-mt/Speech-MASSIVE&#10;" />
          <attvalue for="2" value="&#10;&#10;Multilingual speech corpora have limited coverage of speech-related tasks, primarily focusing on automatic speech recognition (ASR) \cite{commonvoice,fleurs,MaSS,multilinguallibrispeech} and speech translation (ST) \cite{mustc,mtedx,europarlst,covost2}, while neglecting spoken language understanding (SLU -- the task of extracting semantic information from spoken utterances, which typically involves subtasks like intent detection and slot filling).&#10;Unlike text processing, where extensive efforts in natural language understanding (NLU) have led to resources covering a wide range of languages \cite{mlqa,multi3nlu,multiatis,massive}, SLU datasets are mainly English-centric \cite{slurp}, with few exceptions \cite{snips,portmedia,italic}.&#10;&#10;Our goal is to bridge the gap in multilingual SLU drawing inspiration from \cite{italic} and collecting speech recordings in multiple languages. We start with the MASSIVE NLU (i.e. textual) dataset \cite{massive}, an ideal foundation due to its size, domain diversity, and broad coverage of languages, intent, and slot types. Developed by commissioning professional translators to localize the English SLURP dataset \cite{slurp} into 51 languages, MASSIVE comprises 1M labeled utterances spanning 18 domains, with 60 intents and 55 slots. Our contribution, Speech-MASSIVE, spans 12 languages from diverse families: Arabic, German, Spanish, French, Hungarian, Korean, Dutch, Polish, European Portuguese, Russian, Turkish, and Vietnamese. It also facilitates evaluation across various speech tasks beyond SLU, including ASR, ST, and language identification (LID).&#10;We release Speech-MASSIVE publicly under CC-BY-NC-SA 4.0 license.&#10;&#10;Besides detailing the creation process involving a crowdsourcing-based protocol for data collection and quality control, this paper presents&#10;baseline SLU results on Speech-MASSIVE. Our results with both cascade and end-to-end architectures trained in different conditions (zero-shot, few-shot, full fine-tune) will enable future comparisons and tracking SLU advancements compared to the more mature field of NLU. Lastly, we showcase Speech-MASSIVE's versatility through additional experiments on ASR, LID, and ST.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Understanding, Dataset Development, Multilingual Speech, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2012.03411" label="2012.03411">
        <attvalues>
          <attvalue for="0" value="MLS: A Large-Scale Multilingual Dataset for Speech Research" />
          <attvalue for="1" value="  This paper introduces Multilingual LibriSpeech (MLS) dataset, a large&#10;multilingual corpus suitable for speech research. The dataset is derived from&#10;read audiobooks from LibriVox and consists of 8 languages, including about&#10;44.5K hours of English and a total of about 6K hours for other languages.&#10;Additionally, we provide Language Models (LM) and baseline Automatic Speech&#10;Recognition (ASR) models and for all the languages in our dataset. We believe&#10;such a large transcribed dataset will open new avenues in ASR and&#10;Text-To-Speech (TTS) research. The dataset will be made freely available for&#10;anyone at http://www.openslr.org.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;As for our work, LibriSpeech~\cite{povey2015} is derived from the LibriVox data, and is distributed under an open license. It ships with about $1000$ hours of labeled audio, obtained by leveraging alignments between textbooks and their read (audio) counterpart. In contrast to our work, it is only mono-lingual (English). A notable multi-lingual ASR dataset was built with the IARPA Babel Program~\cite{babel}. It collected data for 24 languages, mostly from conversational telephone speech. The dataset is however not released and under an open license, and focused on low-resource languages, with labeled data ranging between $25$ to $65$ hours per language. On the open license side, two important volunteer-supported multi-lingual speech gathering efforts are being conducted: (i)~VoxForge~\cite{Voxforge.org} which collected data for about 15 different languages, but remains low-scale (about $300$ hours in total). (ii)~CommonVoice~\cite{ardila2019common}, a more scalable solution, with more than 30 languages available, which keeps growing with $4500$ (validated) hours currently available. Other notatable multi-lingual datasets distributed under an open license are the M-AILABS~\cite{ailabs2019} and the CMU Wilderness~\cite{black2019} datasets. M-AILABS is a lower-scale version of our work, with 9 languages collected from LibriVox, for a total of about $1000$ hours available. The CMU Wilderness collects readings from the New Testament, with 700 different languages available.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1911.03167" label="1911.03167">
        <attvalues>
          <attvalue for="0" value="Europarl-ST: A Multilingual Corpus For Speech Translation Of&#10;  Parliamentary Debates" />
          <attvalue for="1" value="  Current research into spoken language translation (SLT),or speech-to-text&#10;translation, is often hampered by the lack of specific data resources for this&#10;task, as currently available SLT datasets are restricted to a limited set of&#10;language pairs. In this paper we present Europarl-ST, a novel multilingual SLT&#10;corpus containing paired audio-text samples for SLT from and into 6 European&#10;languages, for a total of 30 different translation directions. This corpus has&#10;been compiled using the debates held in the European Parliament in the period&#10;between 2008 and 2012. This paper describes the corpus creation process and&#10;presents a series of automatic speech recognition, machine translation and&#10;spoken language translation experiments that highlight the potential of this&#10;new resource. The corpus is released under a Creative Commons license and is&#10;freely accessible and downloadable.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10455" label="2212.10455">
        <attvalues>
          <attvalue for="0" value="MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for&#10;  Natural Language Understanding in Task-Oriented Dialogue" />
          <attvalue for="1" value="  Task-oriented dialogue (TOD) systems have been widely deployed in many&#10;industries as they deliver more efficient customer support. These systems are&#10;typically constructed for a single domain or language and do not generalise&#10;well beyond this. To support work on Natural Language Understanding (NLU) in&#10;TOD across multiple languages and domains simultaneously, we constructed&#10;MULTI3NLU++, a multilingual, multi-intent, multi-domain dataset. MULTI3NLU++&#10;extends the English only NLU++ dataset to include manual translations into a&#10;range of high, medium, and low resource languages (Spanish, Marathi, Turkish&#10;and Amharic), in two domains (BANKING and HOTELS). Because of its multi-intent&#10;property, MULTI3NLU++ represents complex and natural user goals, and therefore&#10;allows us to measure the realistic performance of TOD systems in a varied set&#10;of the world's languages. We use MULTI3NLU++ to benchmark state-of-the-art&#10;multilingual models for the NLU tasks of intent detection and slot labelling&#10;for TOD systems in the multilingual setting. The results demonstrate the&#10;challenging nature of the dataset, particularly in the low-resource language&#10;setting, offering ample room for future experimentation in multi-domain&#10;multilingual TOD setups.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14353" label="2004.14353">
        <attvalues>
          <attvalue for="0" value="End-to-End Slot Alignment and Recognition for Cross-Lingual NLU" />
          <attvalue for="1" value="  Natural language understanding (NLU) in the context of goal-oriented dialog&#10;systems typically includes intent classification and slot labeling tasks.&#10;Existing methods to expand an NLU system to new languages use machine&#10;translation with slot label projection from source to the translated&#10;utterances, and thus are sensitive to projection errors. In this work, we&#10;propose a novel end-to-end model that learns to align and predict target slot&#10;labels jointly for cross-lingual transfer. We introduce MultiATIS++, a new&#10;multilingual NLU corpus that extends the Multilingual ATIS corpus to nine&#10;languages across four language families, and evaluate our method using the&#10;corpus. Results show that our method outperforms a simple label projection&#10;method using fast-align on most languages, and achieves competitive performance&#10;to the more complex, state-of-the-art projection method with only half of the&#10;training time. We release our MultiATIS++ corpus to the community to continue&#10;future research on cross-lingual NLU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.08502" label="2306.08502">
        <attvalues>
          <attvalue for="0" value="ITALIC: An Italian Intent Classification Dataset" />
          <attvalue for="1" value="  Recent large-scale Spoken Language Understanding datasets focus predominantly&#10;on English and do not account for language-specific phenomena such as&#10;particular phonemes or words in different lects. We introduce ITALIC, the first&#10;large-scale speech dataset designed for intent classification in Italian. The&#10;dataset comprises 16,521 crowdsourced audio samples recorded by 70 speakers&#10;from various Italian regions and annotated with intent labels and additional&#10;metadata. We explore the versatility of ITALIC by evaluating current&#10;state-of-the-art speech and text models. Results on intent classification&#10;suggest that increasing scale and running language adaptation yield better&#10;speech models, monolingual text models outscore multilingual ones, and that&#10;speech recognition on ITALIC is more challenging than on existing Italian&#10;benchmarks. We release both the dataset and the annotation scheme to streamline&#10;the development of new Italian SLU models and language-specific datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.09025" label="2402.09025">
        <attvalues>
          <attvalue for="0" value="SLEB: Streamlining LLMs through Redundancy Verification and Elimination&#10;  of Transformer Blocks" />
          <attvalue for="1" value="  Large language models (LLMs) have proven to be highly effective across&#10;various natural language processing tasks. However, their large number of&#10;parameters poses significant challenges for practical deployment. Pruning, a&#10;technique aimed at reducing the size and complexity of LLMs, offers a potential&#10;solution by removing redundant components from the network. Despite the promise&#10;of pruning, existing methods often struggle to achieve substantial end-to-end&#10;LLM inference speedup. In this paper, we introduce SLEB, a novel approach&#10;designed to streamline LLMs by eliminating redundant transformer blocks. We&#10;choose the transformer block as the fundamental unit for pruning, because LLMs&#10;exhibit block-level redundancy with high similarity between the outputs of&#10;neighboring blocks. This choice allows us to effectively enhance the processing&#10;speed of LLMs. Our experimental results demonstrate that SLEB outperforms&#10;previous LLM pruning methods in accelerating LLM inference while also&#10;maintaining superior perplexity and accuracy, making SLEB as a promising&#10;technique for enhancing the efficiency of LLMs. The code is available at:&#10;https://github.com/jiwonsong-dev/SLEB.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs), such as GPT-3, OPT, and LLaMA demonstrate exceptional proficiency in a variety of natural language processing (NLP) tasks and have become key components in applications like chatbots and question-answering systems~\cite{gpt3, opt, palm, llama_v1, llama_v2}.&#10;However, their substantial number of parameters creates significant challenges in deploying these models for real-world services, especially due to the increased memory consumption and computational demands.&#10;This limitation restricts their widespread use. &#10;Consequently, it is critical to develop techniques that improve the compactness and processing efficiency of LLMs while preserving their linguistic prowess.&#10;&#10;Network pruning is a technique aimed at reducing the size and complexity of neural networks by eliminating redundant weight parameters~\cite{obd, obs, songhan}. Its application in LLMs has been somewhat limited, primarily due to challenges that arise in managing sparse matrices~\cite{obc, wanda, sparsegpt, DSnoT}.&#10;This complexity becomes particularly evident when using modern GPU hardware, as these systems are typically optimized for operations involving dense matrices~\cite{sparsert, sparsegpu}.&#10;&#10;In the realm of LLMs, a significant similarity in output is observed among successive transformer blocks~\cite{jump_to_con, dejavu}. This similarity arises because each transformer block incrementally contributes to the residual path spanning the entire LLM.&#10;Figure~\ref{fig:llm} depicts the typical architecture of conventional LLMs, characterized by a continuous stack of transformer blocks. A key aspect of LLM computation is the residual path that traverses the entire network, a feature introduced to stabilize the backpropagation process during training.&#10;Consequently, each transformer block contributes its computational outputs, derived from both the attention mechanisms and feed-forward layers, to this residual path.&#10;This design feature results in a considerable degree of output similarity between consecutive transformer blocks, leading to redundancy within the LLMs.&#10;&#10;In this paper, we propose SLEB, a novel approach designed to streamline LLMs by identifying and eliminating redundant transformer blocks.&#10;Figure~\ref{fig:overview} compares the proposed approach with previous pruning methods.&#10;SLEB is tailored to refine LLMs through the strategic removal of redundant transformer blocks, thereby effectively aligning speedup with the pruning ratio. &#10;The fundamental principle behind SLEB is that the careful elimination of redundant transformer blocks can be achieved without affecting the text generation capabilities of LLMs.&#10;By targeting these specific redundant elements within the LLM's architecture, SLEB seeks to provide a more efficient pruning approach. It aims to overcome the challenges typically associated with traditional network pruning, particularly in enhancing the acceleration of LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Neural Network Pruning, Mathematics, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2208.11580" label="2208.11580">
        <attvalues>
          <attvalue for="0" value="Optimal Brain Compression: A Framework for Accurate Post-Training&#10;  Quantization and Pruning" />
          <attvalue for="1" value="  We consider the problem of model compression for deep neural networks (DNNs)&#10;in the challenging one-shot/post-training setting, in which we are given an&#10;accurate trained model, and must compress it without any retraining, based only&#10;on a small amount of calibration input data. This problem has become popular in&#10;view of the emerging software and hardware support for executing models&#10;compressed via pruning and/or quantization with speedup, and well-performing&#10;solutions have been proposed independently for both compression approaches. In&#10;this paper, we introduce a new compression framework which covers both weight&#10;pruning and quantization in a unified setting, is time- and space-efficient,&#10;and considerably improves upon the practical performance of existing&#10;post-training methods. At the technical level, our approach is based on an&#10;exact and efficient realization of the classical Optimal Brain Surgeon (OBS)&#10;framework of [LeCun, Denker, and Solla, 1990] extended to also cover weight&#10;quantization at the scale of modern DNNs. From the practical perspective, our&#10;experimental results show that it can improve significantly upon the&#10;compression-accuracy trade-offs of existing post-training methods, and that it&#10;can enable the accurate compound application of both pruning and quantization&#10;in a post-training setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.11849" label="2008.11849">
        <attvalues>
          <attvalue for="0" value="SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning&#10;  Inference" />
          <attvalue for="1" value="  In recent years, there has been a flurry of research in deep neural network&#10;pruning and compression. Early approaches prune weights individually. However,&#10;it is difficult to take advantage of the resulting unstructured sparsity&#10;patterns on modern hardware like GPUs. As a result, pruning strategies which&#10;impose sparsity structures in the weights have become more popular.&#10;However,these structured pruning approaches typically lead to higher losses in&#10;accuracy than unstructured pruning. In this paper, we present SparseRT, a code&#10;generator that leverage unstructured sparsity to accelerate sparse linear&#10;algebra operations in deep learning inference on GPUs. For 1x1 convolutions and&#10;fully connected layers, we demonstrate geometric mean of speedups of 3.4x over&#10;the equivalent dense computation at 90% sparsity and 5.4x at 95% sparsity when&#10;evaluated on hundreds of test cases in deep learning. For sparse 3x3&#10;convolutions, we show speedups of over 5x on use cases in ResNet-50.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.09435" label="2303.09435">
        <attvalues>
          <attvalue for="0" value="Jump to Conclusions: Short-Cutting Transformers With Linear&#10;  Transformations" />
          <attvalue for="1" value="  Transformer-based language models create hidden representations of their&#10;inputs at every layer, but only use final-layer representations for prediction.&#10;This obscures the internal decision-making process of the model and the utility&#10;of its intermediate representations. One way to elucidate this is to cast the&#10;hidden representations as final representations, bypassing the transformer&#10;computation in-between. In this work, we suggest a simple method for such&#10;casting, using linear transformations. This approximation far exceeds the&#10;prevailing practice of inspecting hidden representations from all layers, in&#10;the space of the final layer. Moreover, in the context of language modeling,&#10;our method produces more accurate predictions from hidden layers, across&#10;various model scales, architectures, and data distributions. This allows&#10;&quot;peeking&quot; into intermediate representations, showing that GPT-2 and BERT often&#10;predict the final output already in early layers. We then demonstrate the&#10;practicality of our method to recent early exit strategies, showing that when&#10;aiming, for example, at retention of 95% accuracy, our approach saves&#10;additional 7.9% layers for GPT-2 and 5.4% layers for BERT. Last, we extend our&#10;method to linearly approximate sub-modules, finding that attention is most&#10;tolerant to this change. Our code and learned mappings are publicly available&#10;at https://github.com/sashayd/mat.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.12820" label="2204.12820">
        <attvalues>
          <attvalue for="0" value="LyS_ACoru\~na at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools&#10;  for Sentiment Analysis as Semantic Dependency Parsing" />
          <attvalue for="1" value="  This paper addressed the problem of structured sentiment analysis using a&#10;bi-affine semantic dependency parser, large pre-trained language models, and&#10;publicly available translation models. For the monolingual setup, we&#10;considered: (i) training on a single treebank, and (ii) relaxing the setup by&#10;training on treebanks coming from different languages that can be adequately&#10;processed by cross-lingual language models. For the zero-shot setup and a given&#10;target treebank, we relied on: (i) a word-level translation of available&#10;treebanks in other languages to get noisy, unlikely-grammatical, but annotated&#10;data (we release as much of it as licenses allow), and (ii) merging those&#10;translated treebanks to obtain training data. In the post-evaluation phase, we&#10;also trained cross-lingual models that simply merged all the English treebanks&#10;and did not use word-level translations, and yet obtained better results.&#10;According to the official results, we ranked 8th and 9th in the monolingual and&#10;cross-lingual setups.&#10;" />
          <attvalue for="2" value="&#10;&#10;Sentiment Analysis \cite[SA,][]{Pang08opinionmining} deals with the automatic processing of subjective information in natural language texts. Early work on SA focused on conceptually simpler tasks, such as polarity classification at the sentence or document level. With the advances in natural language processing (NLP), more fine-grained and complex tasks have been proposed, such as detecting the entity that expresses an opinionated chunk of text, or the entity that was targeted.&#10;More particularly, \cite{barnes-etal-2021-structured} consider sentiment analysis as a (graph) structured task, and discuss up to five subtasks: (i) sentiment expression extraction, (ii) sentiment target extraction, (iii) sentiment holder extraction, (iv) defining the relationship between these elements, and (v) assigning a polarity label. They discuss that although these tasks have been extensively studied by different authors \cite[inter alia]{turney-2002-thumbs,pontiki-etal-2015-semeval,zhang2019end}, they are not addressed all together. They also discuss that such subdivision into subtasks might have a negative impact in the general analysis of the sentence, and that a joint analysis could translate into a holistic approach. To do so, they propose to encapsulate all these tasks in the form of a sentiment graph. Formally, the goal is to find the set of opinion tuples $\{O_1,\ldots,O_i,\ldots,O_n\}$ in a given text, where each opinion $O_i$ is a tuple of the form $(h, t, e, p)$ where $h$ is a holder who expresses a polarity $p$ towards a target $t$ through a sentiment expression $e$, implicitly defining pairwise relationships between elements of the same tuple. We illustrate an example in Figure \ref{fig:sentiment-graph-example}.&#10;&#10;More particularly, for the SemEval-2022 Task 10 \cite{barnes-etal-2022-semeval}, the organizers proposed both a monolingual and a cross-lingual (zero-shot) setup. They considered 5 languages (and 7 treebanks): English, Spanish, Catalan, Basque, and Norwegian. For the zero-shot setup Basque, Catalan, and Spanish were the target languages.&#10;&#10;Our approach is based on the idea of viewing this task as semantic dependency parsing \cite{oepen-etal-2015-semeval}, since both tasks are structurally similar even if the graphs have different meaning. More specifically, we rely on a bi-affine graph-based parser \cite{dozat-manning-2018-simpler} and different large pre-trained language models (LM), such as BERT \cite{devlin-etal-2019-bert}, RoBERTa \cite{liu2019roberta} or XLM-R \cite{conneau-etal-2020-unsupervised}. For the monolingual setup we train a semantic parsing model on single and merged treebanks, and compare the performance using different LMs. For the cross-lingual setup, we first do a word-level translation of the datasets in a different language than the target treebank, and then proceed similarly to the monolingual setup. Overall, the approach relies on off-the-shelf tools already available, but traditionally used for other purposes. We here re-purpose them for their use for sentiment analysis as graph-based parsing.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Sentiment Analysis, Language Models, Cross-Lingual Parsing, Computational Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2105.14504" label="2105.14504">
        <attvalues>
          <attvalue for="0" value="Structured Sentiment Analysis as Dependency Graph Parsing" />
          <attvalue for="1" value="  Structured sentiment analysis attempts to extract full opinion tuples from a&#10;text, but over time this task has been subdivided into smaller and smaller&#10;sub-tasks, e,g,, target extraction or targeted polarity classification. We&#10;argue that this division has become counterproductive and propose a new unified&#10;framework to remedy the situation. We cast the structured sentiment problem as&#10;dependency graph parsing, where the nodes are spans of sentiment holders,&#10;targets and expressions, and the arcs are the relations between them. We&#10;perform experiments on five datasets in four languages (English, Norwegian,&#10;Basque, and Catalan) and show that this approach leads to strong improvements&#10;over state-of-the-art baselines. Our analysis shows that refining the sentiment&#10;graphs with syntactic dependency information further improves results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="cs/0212032" label="cs/0212032">
        <attvalues>
          <attvalue for="0" value="Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised&#10;  Classification of Reviews" />
          <attvalue for="1" value="  This paper presents a simple unsupervised learning algorithm for classifying&#10;reviews as recommended (thumbs up) or not recommended (thumbs down). The&#10;classification of a review is predicted by the average semantic orientation of&#10;the phrases in the review that contain adjectives or adverbs. A phrase has a&#10;positive semantic orientation when it has good associations (e.g., &quot;subtle&#10;nuances&quot;) and a negative semantic orientation when it has bad associations&#10;(e.g., &quot;very cavalier&quot;). In this paper, the semantic orientation of a phrase is&#10;calculated as the mutual information between the given phrase and the word&#10;&quot;excellent&quot; minus the mutual information between the given phrase and the word&#10;&quot;poor&quot;. A review is classified as recommended if the average semantic&#10;orientation of its phrases is positive. The algorithm achieves an average&#10;accuracy of 74% when evaluated on 410 reviews from Epinions, sampled from four&#10;different domains (reviews of automobiles, banks, movies, and travel&#10;destinations). The accuracy ranges from 84% for automobile reviews to 66% for&#10;movie reviews.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.01396" label="1807.01396">
        <attvalues>
          <attvalue for="0" value="Simpler but More Accurate Semantic Dependency Parsing" />
          <attvalue for="1" value="  While syntactic dependency annotations concentrate on the surface or&#10;functional structure of a sentence, semantic dependency annotations aim to&#10;capture between-word relationships that are more closely related to the meaning&#10;of a sentence, using graph-structured representations. We extend the LSTM-based&#10;syntactic parser of Dozat and Manning (2017) to train on and generate these&#10;graph structures. The resulting system on its own achieves state-of-the-art&#10;performance, beating the previous, substantially more complex state-of-the-art&#10;system by 0.6% labeled F1. Adding linguistically richer input representations&#10;pushes the margin even higher, allowing us to beat it by 1.9% labeled F1.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.12621" label="2406.12621">
        <attvalues>
          <attvalue for="0" value="Growing Trees on Sounds: Assessing Strategies for End-to-End Dependency&#10;  Parsing of Speech" />
          <attvalue for="1" value="  Direct dependency parsing of the speech signal -- as opposed to parsing&#10;speech transcriptions -- has recently been proposed as a task (Pupier et al.&#10;2022), as a way of incorporating prosodic information in the parsing system and&#10;bypassing the limitations of a pipeline approach that would consist of using&#10;first an Automatic Speech Recognition (ASR) system and then a syntactic parser.&#10;In this article, we report on a set of experiments aiming at assessing the&#10;performance of two parsing paradigms (graph-based parsing and sequence labeling&#10;based parsing) on speech parsing. We perform this evaluation on a large&#10;treebank of spoken French, featuring realistic spontaneous conversations. Our&#10;findings show that (i) the graph based approach obtain better results across&#10;the board (ii) parsing directly from speech outperforms a pipeline approach,&#10;despite having 30% fewer parameters.&#10;" />
          <attvalue for="2" value="&#10;Dependency parsing is a central task in natural language processing (NLP).&#10;In the NLP community, it has mostly been addressed on textual data, either natively written texts or sometimes speech transcriptions.&#10;Yet, speech is the main form of communication between humans, as well as arguably one of the most realistic types of linguistic data, which motivates the design of NLP systems able to deal directly with speech, both for applicative purposes and to construct corpora annotated with linguistic information.&#10;When parsing speech transcriptions, most prior work has focused on disfluency detection and removal \cite{charniak-johnson-2001-edit,johnson-charniak-2004-tag,rasooli-tetreault-2013-joint,honnibal-johnson-2014-joint, jamshid-lou-etal-2019-neural}, in an effort to `normalize' the transcriptions and make them suitable input for NLP systems trained on written language.&#10;Using only transcriptions as input is a natural choice from an NLP perspective: it makes it possible to use off-the-shelf NLP parsers `as is'.&#10;However, predicted transcriptions can be very noisy, in particular for speech from spontaneous conversations. Furthermore, transcriptions are abstractions that contain much less information than the speech signal. The prosody, and the pauses in the speech utterances are very important clues for parsing \cite{price1991use} that are completely absent from transcriptions. &#10;Hence, we address speech parsing using only the speech signal as input. &#10;With the popularization of self-supervised method and modern neural network architecture (pretrained transformers), both speech and text domains now use similar techniques \cite{chrupala-2023-putting}.&#10;This convergence of methodology has raised interest in other applications of speech models to go beyond `simple' speech recognition.&#10;Thus, addressing classical NLP tasks directly on speech is a natural step and design NLP tools able to deal with spontaneous speech, arguably the most realistic type of linguistic production. In short, Our contributions are the following:&#10;\begin{itemize}[noitemsep, topsep=0pt]&#10; \item we introduce a graph-based end-to-end dependency parsing algorithm for speech;&#10; \item we evaluate the parser on Orféo, a large treebank of spoken French that features spontaneous speech, and compare its performance to pipeline systems and to a parsing-as-tagging parser;&#10; \item we release our code at.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Dependency Parsing Models, Speech Recognition, Natural Language Processing, Speech Signal Parsing" />
        </attvalues>
      </node>
      <node id="2305.04572" label="2305.04572">
        <attvalues>
          <attvalue for="0" value="Putting Natural in Natural Language Processing" />
          <attvalue for="1" value="  Human language is firstly spoken and only secondarily written. Text, however,&#10;is a very convenient and efficient representation of language, and modern&#10;civilization has made it ubiquitous. Thus the field of NLP has overwhelmingly&#10;focused on processing written rather than spoken language. Work on spoken&#10;language, on the other hand, has been siloed off within the largely separate&#10;speech processing community which has been inordinately preoccupied with&#10;transcribing speech into text. Recent advances in deep learning have led to a&#10;fortuitous convergence in methods between speech processing and mainstream NLP.&#10;Arguably, the time is ripe for a unification of these two fields, and for&#10;starting to take spoken language seriously as the primary mode of human&#10;communication. Truly natural language processing could lead to better&#10;integration with the rest of language science and could lead to systems which&#10;are more data-efficient and more human-like, and which can communicate beyond&#10;the textual modality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.05955" label="2405.05955">
        <attvalues>
          <attvalue for="0" value="Smurfs: Leveraging Multiple Proficiency Agents with Context-Efficiency&#10;  for Tool Planning" />
          <attvalue for="1" value="  The emergence of large language models (LLMs) has opened up unprecedented&#10;possibilities for automating complex tasks that are often comparable to human&#10;performance. Despite their capabilities, LLMs still encounter difficulties in&#10;completing tasks that require high levels of accuracy and complexity due to&#10;their inherent limitations in handling multifaceted problems single-handedly.&#10;This paper introduces `Smurfs', a cutting-edge multi-agent framework designed&#10;to revolutionize the application of LLMs. By seamlessly transforming a&#10;conventional LLM into a synergistic multi-agent ensemble, Smurfs can enhance&#10;the model's ability to solve complex tasks at no additional cost. This is&#10;achieved through innovative prompting strategies that allocate distinct roles&#10;within the model, thereby facilitating collaboration among specialized agents&#10;and forming an intelligent multi-agent system. Our empirical investigation on&#10;both open-ended task of StableToolBench and closed-ended task on HotpotQA&#10;showcases Smurfs' superior capability in intricate tool utilization scenarios.&#10;Notably, Smurfs outmatches all the baseline methods in both experiments,&#10;setting new state-of-the-art performance. Furthermore, through comprehensive&#10;ablation studies, we dissect the contribution of the core components of the&#10;multi-agent framework to its overall efficacy. This not only verifies the&#10;effectiveness of the framework, but also sets a route for future exploration of&#10;multi-agent LLM systems.&#10;" />
          <attvalue for="2" value="&#10;&#10;Tool manipulation has traditionally been seen as a distinctive human characteristic, dating back approximately 2.5 million years~\cite{oakley1972man, ambrose2001paleolithic}. For large language models (LLMs), access to external tools can equip them with broader capabilities beyond their fixed language modeling knowledge. For example, the search engine API empowers ChatGPT to access real-time information~\cite{zhao2023survey}. However, LLMs still encounter several challenges when using multiple tools to solve tasks. These challenges include effective solution planning and adaptability to new tools.~\cite{hao2024toolkengpt, guu2020retrieval, qin2024toolllm}.&#10;&#10;This paper addresses the critical research problem of enhancing the problem-solving capabilities of LLMs through the adoption of a plug-and-play multi-agent system~(MAS) framework~\cite{dorri2018multi, van2008multi}. We posit that a MAS approach can significantly augment the efficacy of LLMs in handling tasks that require a high degree of precision, adaptability, and comprehensive knowledge integration. &#10;&#10;To this end, we introduce `Smurfs' an innovative MAS framework inspired by the collaborative and versatile nature of its namesake cartoon characters. The proposed framework is based on the principle: synergistic collaboration among specialized agents can overcome the limitations faced by individual LLMs. Each agent within the Smurfs framework is designed to perform specific sub-tasks, facilitating a more nuanced and effective approach to complex problem-solving. Our research delves into the architectural design, coordination mechanisms, and the operational dynamics of integrating specialized agents into a cohesive system. &#10;The effectiveness of Smurfs is validated through both open-ended and closed-ended tool planning benchmark experiments~\cite{guo2024stabletoolbench, yang2018hotpotqa}, where the proposed MAS system consistently outperform baseline methods on both benchmarks. An ablation study followed by a case study further investigates the underlying reasons for this effectiveness. These results not only establish a new state-of-the-art in the field but also offer concrete evidence of the multi-agent approach's efficacy in enhancing LLM capabilities.&#10;&#10;The contributions of this paper can be summarized as follows:&#10;\begin{enumerate}&#10; \item We introduce a novel plug-and-play MAS framework to enhance the tool planning capabilities of LLMs. Experiments demonstrate the effectiveness of this approach, which is also more cost-efficient compared to existing tool planning methods.&#10; \item Ablation studies further reveal the underlying reasons for the effectiveness of the MAS framework, providing valuable insights for future research.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Multi-Agent Systems, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2403.07714" label="2403.07714">
        <attvalues>
          <attvalue for="0" value="StableToolBench: Towards Stable Large-Scale Benchmarking on Tool&#10;  Learning of Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have witnessed remarkable advancements in recent&#10;years, prompting the exploration of tool learning, which integrates LLMs with&#10;external tools to address diverse real-world challenges. Assessing the&#10;capability of LLMs to utilise tools necessitates large-scale and stable&#10;benchmarks. However, previous works relied on either hand-crafted online tools&#10;with limited scale, or large-scale real online APIs suffering from instability&#10;of API status. To address this problem, we introduce StableToolBench, a&#10;benchmark evolving from ToolBench, proposing a virtual API server and stable&#10;evaluation system. The virtual API server contains a caching system and API&#10;simulators which are complementary to alleviate the change in API status.&#10;Meanwhile, the stable evaluation system designs solvable pass and win rates&#10;using GPT-4 as the automatic evaluator to eliminate the randomness during&#10;evaluation. Experimental results demonstrate the stability of StableToolBench,&#10;and further discuss the effectiveness of API simulators, the caching system,&#10;and the evaluator system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Tool Learning Benchmarks.&#10;Recent studies have shed light on the burgeoning capabilities of LLMs in understanding and mastering tools~\cite{li2023apibank, patil2023gorilla, gpt4tools, song2023restgpt, tang2023toolalpaca, ye2024tooleyes, xu2023tool}.&#10;Gaining access to external tools endows LLMs with real-time factual knowledge~\cite{yang2023chatgpt}, multimodal functionalities~\cite{gupta2023visual}, and specialised skills in vertical domains~\cite{jin2023genegpt}. &#10;However, few work has been done to explore the stability of the tool environment in specific benchmarks and how it affects the LLMs' performance in tool-augmented tasks. &#10;&#10;Tool Inference Methods.&#10;Recent literature has begun to explore various methodologies for integrating tool functionalities within LLMs. &#10;Notably, the robust in-context learning prowess of LLMs, as demonstrated in \cite{brown2020language}, has facilitated the augmentation of LLMs with external tools via in-context tool descriptions and demonstrations ~\cite{hsieh2023tool, ruan2023tptu, mialon2023augmented}.&#10;An alternative approach involves the explicit training of LLMs ~\cite{patil2023gorilla, tang2023toolalpaca, chen2023fireact, qin2023toolllm, huang2023metatool} using datasets enriched with tool interactions, thereby familiarising models with the nuances of tool usage. &#10;&#10;Evaluation in Tool Learning.&#10;Evaluating the performance of LLMs in tool-augmented tasks presents unique challenges and opportunities.&#10;Numerous works have been developed for the assessment of tool utilisation, primarily emphasising response comparison ~\cite{zhuang2023toolqa}, tool call accuracy ~\cite{patil2023gorilla}, or a synthesis of these aspects ~\cite{li2023apibank}. Distinguishing itself, \cite{qin2023toolllm} introduces an innovative methodology by integrating a large language model (LLM) as a judge to evaluate the comprehensive solution path. &#10;Subsequent research ~\cite{wang2023mint} focuses on the multi-turn interaction capabilities of LLMs with both tools and user feedback.&#10;In a departure from the aforementioned approaches, \cite{chen2023teval} presents itself as the inaugural benchmark specifically tailored for the fine-grained assessment of tool utilisation capabilities.&#10;However, there exists a gap in the literature concerning the exploration of evaluation stability when evaluating the tool usage capabilities of LLMs.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2306.05969" label="2306.05969">
        <attvalues>
          <attvalue for="0" value="Language Models Can Learn Exceptions to Syntactic Rules" />
          <attvalue for="1" value="  Artificial neural networks can generalize productively to novel contexts. Can&#10;they also learn exceptions to those productive rules? We explore this question&#10;using the case of restrictions on English passivization (e.g., the fact that&#10;&quot;The vacation lasted five days&quot; is grammatical, but &quot;*Five days was lasted by&#10;the vacation&quot; is not). We collect human acceptability judgments for passive&#10;sentences with a range of verbs, and show that the probability distribution&#10;defined by GPT-2, a language model, matches the human judgments with high&#10;correlation. We also show that the relative acceptability of a verb in the&#10;active vs. passive voice is positively correlated with the relative frequency&#10;of its occurrence in those voices. These results provide preliminary support&#10;for the entrenchment hypothesis, according to which learners track and uses the&#10;distributional properties of their input to learn negative exceptions to rules.&#10;At the same time, this hypothesis fails to explain the magnitude of&#10;unpassivizability demonstrated by certain individual verbs, suggesting that&#10;other cues to exceptionality are available in the linguistic input.&#10;" />
          <attvalue for="2" value="&#10;Many studies have demonstrated language models' ability to extend a generalization from a small set of examples to novel lexical items, structures, and contexts, even if the models do not always do so in a human-like way \cite{hupkes2020,kim-linzen-2020-cogs,lake2018,mccoy2018}. These studies show that models can substitute novel lexical items into rules where those items were previously unseen. At the same time, language models can sometimes over-generalize, for instance by producing a literal, compositional translation of idiomatic expressions like kick the bucket when humans would not \cite{dankers2022}. A full evaluation of language models' generalization abilities should thus not only measure whether models can generalize when humans do, but also whether models are able to constrain their generalizations when humans do.&#10;&#10;We address this question by building on a line of work that probes whether human-like acceptability judgments for argument structure alternations can be predicted from the probability distribution that a from language model defines over sentences. This studies have shown, for example, that the GPT-2 language model \cite{radford2019} can match human judgments about whether the dative alternation applies to a verb \cite{hawkins2020}, and that information about which syntactic frames a verb can appear in (e.g. whether a verb participates in the spray/load alternation) can be recovered from the verb's contextualized representations and from sentence embeddings \cite{kann-etal-2019-verb}.&#10;&#10;In this work, we evaluate models' ability to identify exceptions using the case study of the English passive. The passive voice is highly productive in English; most strikingly, young children exposed to novel verbs in the active voice are able to understand and produce passive constructions using those verbs \cite{pinker1987,brooks1999}. This suggests that English speakers do not in general conclude that verbs that they have never encountered in the passive voice are unacceptable in that voice. &#10;Yet there are limits to the productivity of the English passive; examples such as (\nextx) have been reported to be unacceptable in the passive voice:&#10;\pex&#10;\a The vacation lasted five days.&#10;\a\ljudge{*} Five days was lasted by the vacation.&#10;\xe&#10;&#10;Sentences like (\lastx b) are unlikely to occur productively in natural speech---just like passives of infrequent verbs. Yet even though they do not receive explicit evidence that these sentences are unacceptable, rather than simply rare, English speakers nonetheless learn that they constitute exceptions, and do not judge (\lastx b) to be acceptable.&#10;&#10;How do humans acquire such exceptions? The entrenchment hypothesis suggests that speakers track and use the distributional properties of their input as indirect negative evidence for the existence of an exception \cite{braine1995,regier2004,theakston2004}. For instance, if an English learner never encounters the verb last in the passive voice, despite having seen last used productively in the active voice, they may conclude that last cannot occur in the the passive voice. &#10;Are language models---which do not have access to human feedback or syntactic supervision, and are trained solely to perform next-word prediction---attentive to the same information that humans are when determining the extent to which syntactic rules can generalize?&#10;&#10;In this paper, we tackle these questions by comparing human acceptability judgments on sentences containing verbs that are exceptional in the passive voice, on the one hand, to the probability distribution defined by a GPT-2-like model trained on a 100-million word English corpus. We find that the language model matches human acceptability judgments on active and passive sentences to a large degree (Figure \ref{fig:summary_scatter}), suggesting that language models can constrain their syntactic generalizations in a human-like way.&#10;Using our model's training corpus, we further show that there is a weak but positive correlation between the relative frequency of actives and passives in the input and their relative acceptability. Together, these empirical results suggest that the linguistic input contains useful information from which exceptions to syntactic generalizations can be learned.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Neural Network Generalization, Linguistics, Cognitive Science, Linguistic Exception Learning, Artificial Intelligence, Language Modeling" />
        </attvalues>
      </node>
      <node id="2010.02375" label="2010.02375">
        <attvalues>
          <attvalue for="0" value="Investigating representations of verb bias in neural language models" />
          <attvalue for="1" value="  Languages typically provide more than one grammatical construction to express&#10;certain types of messages. A speaker's choice of construction is known to&#10;depend on multiple factors, including the choice of main verb -- a phenomenon&#10;known as \emph{verb bias}. Here we introduce DAIS, a large benchmark dataset&#10;containing 50K human judgments for 5K distinct sentence pairs in the English&#10;dative alternation. This dataset includes 200 unique verbs and systematically&#10;varies the definiteness and length of arguments. We use this dataset, as well&#10;as an existing corpus of naturally occurring data, to evaluate how well recent&#10;neural language models capture human preferences. Results show that larger&#10;models perform better than smaller models, and transformer architectures (e.g.&#10;GPT-2) tend to out-perform recurrent architectures (e.g. LSTMs) even under&#10;comparable parameter and training settings. Additional analyses of internal&#10;feature representations suggest that transformers may better integrate specific&#10;lexical information with grammatical constructions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.10773" label="1811.10773">
        <attvalues>
          <attvalue for="0" value="Verb Argument Structure Alternations in Word and Sentence Embeddings" />
          <attvalue for="1" value="  Verbs occur in different syntactic environments, or frames. We investigate&#10;whether artificial neural networks encode grammatical distinctions necessary&#10;for inferring the idiosyncratic frame-selectional properties of verbs. We&#10;introduce five datasets, collectively called FAVA, containing in aggregate&#10;nearly 10k sentences labeled for grammatical acceptability, illustrating&#10;different verbal argument structure alternations. We then test whether models&#10;can distinguish acceptable English verb-frame combinations from unacceptable&#10;ones using a sentence embedding alone. For converging evidence, we further&#10;construct LaVA, a corresponding word-level dataset, and investigate whether the&#10;same syntactic features can be extracted from word embeddings. Our models&#10;perform reliable classifications for some verbal alternations but not others,&#10;suggesting that while these representations do encode fine-grained lexical&#10;information, it is incomplete or can be hard to extract. Further, differences&#10;between the word- and sentence-level models show that some information present&#10;in word embeddings is not passed on to the down-stream sentence embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.05640" label="2311.05640">
        <attvalues>
          <attvalue for="0" value="FinGPT: Large Generative Models for a Small Language" />
          <attvalue for="1" value="  Large language models (LLMs) excel in many tasks in NLP and beyond, but most&#10;open models have very limited coverage of smaller languages and LLM work tends&#10;to focus on languages where nearly unlimited data is available for pretraining.&#10;In this work, we study the challenges of creating LLMs for Finnish, a language&#10;spoken by less than 0.1% of the world population. We compile an extensive&#10;dataset of Finnish combining web crawls, news, social media and eBooks. We&#10;pursue two approaches to pretrain models: 1) we train seven monolingual models&#10;from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the&#10;pretraining of the multilingual BLOOM model on a mix of its original training&#10;data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.&#10;For model evaluation, we introduce FIN-bench, a version of BIG-bench with&#10;Finnish tasks. We also assess other model qualities such as toxicity and bias.&#10;Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural language models based on the Transformer architecture \cite{vaswani2017attention} have revolutionized Natural Language Processing (NLP) in recent years, advancing the state of the art in tasks ranging from text classification to open-ended text generation. &#10;Generative, decoder-only language models such as the Generative Pretrained Transformer (GPT) \cite{radford2018improving} series have been a particular focus of interest in part due to their multitask and few-shot capabilities \cite{radford2019language,brown2020language}. The ability of such models to implicitly learn to perform tasks that they have not been directly trained on has been considered to be closely tied to the scale of the model \cite{brown2020language,chowdhery2022palm} and, perhaps even more importantly, to the number of training tokens \cite{hoffmann2022training, muennighoff2023scaling,touvron2023llama}. Most work on such models focuses on English, often entirely excluding other languages, and assumes that hundreds of billions of tokens of text are readily available for model training.&#10;&#10;In this study, we consider the challenges of introducing large generative models for Finnish, a Uralic language natively spoken by fewer than 6 million people. While the language is comparatively well represented in online resources relative to this number, less than 1\% of texts available in e.g.\ Wikipedia and Common Crawl are Finnish \cite{pyysalo2021wikibert,xue2021mt5}. As the other members in the language family are either even smaller and lesser-resourced or quite distant, the resources for creating models for the language are quite limited. Finnish has been represented to some degree in Transformer-based models since the release of the original multilingual BERT model \cite{devlin2019bert}, and a dedicated monolingual BERT for the language was previously created by \newcite{virtanen2019multilingual}. Also some generative models for Finnish have been previously introduced by the &quot;Finnish-NLP&quot; group and \newcite{hatanpaa2022generative}, but as training LLMs is very expensive and Finnish is constrained by the size of available data, models exceeding a billion parameters have been so far missing from the Finnish NLP landscape.&#10;&#10;We compile a broad-coverage dataset of Finnish and train monolingual models up to 13 billion parameters for 300 billion tokens (approx.\ 8 epochs). We also perform continued pretraining of the 176-billion parameter BLOOM model \cite{scao2022bloom} to extend its coverage of Finnish, introduce novel evaluation datasets, and assess multiple aspects of the resulting models. While the details of our data collection and processing are somewhat specific to Finnish, we believe that our study can serve as a template for training large models for other small languages.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Finnish Language Processing, Computational Linguistics, Multilingual NLP Challenges, Artificial Intelligence, Language Model Development, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2006.01538" label="2006.01538">
        <attvalues>
          <attvalue for="0" value="WikiBERT models: deep transfer learning for many languages" />
          <attvalue for="1" value="  Deep neural language models such as BERT have enabled substantial recent&#10;advances in many natural language processing tasks. Due to the effort and&#10;computational cost involved in their pre-training, language-specific models are&#10;typically introduced only for a small number of high-resource languages such as&#10;English. While multilingual models covering large numbers of languages are&#10;available, recent work suggests monolingual training can produce better models,&#10;and our understanding of the tradeoffs between mono- and multilingual training&#10;is incomplete. In this paper, we introduce a simple, fully automated pipeline&#10;for creating language-specific BERT models from Wikipedia data and introduce 42&#10;new such models, most for languages up to now lacking dedicated deep neural&#10;language models. We assess the merits of these models using the&#10;state-of-the-art UDify parser on Universal Dependencies data, contrasting&#10;performance with results using the multilingual BERT model. We find that UDify&#10;using WikiBERT models outperforms the parser using mBERT on average, with the&#10;language-specific models showing substantially improved performance for some&#10;languages, yet limited improvement or a decrease in performance for others. We&#10;also present preliminary results as first steps toward an understanding of the&#10;conditions under which language-specific models are most beneficial. All of the&#10;methods and models introduced in this work are available under open licenses&#10;from https://github.com/turkunlp/wikibert.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.14388" label="2012.14388">
        <attvalues>
          <attvalue for="0" value="Universal Sentence Representation Learning with Conditional Masked&#10;  Language Model" />
          <attvalue for="1" value="  This paper presents a novel training method, Conditional Masked Language&#10;Modeling (CMLM), to effectively learn sentence representations on large scale&#10;unlabeled corpora. CMLM integrates sentence representation learning into MLM&#10;training by conditioning on the encoded vectors of adjacent sentences. Our&#10;English CMLM model achieves state-of-the-art performance on SentEval, even&#10;outperforming models learned using supervised signals. As a fully unsupervised&#10;learning method, CMLM can be conveniently extended to a broad range of&#10;languages and domains. We find that a multilingual CMLM model co-trained with&#10;bitext retrieval (BR) and natural language inference (NLI) tasks outperforms&#10;the previous state-of-the-art multilingual models by a large margin, e.g. 10%&#10;improvement upon baseline models on cross-lingual semantic search. We explore&#10;the same language bias of the learned representations, and propose a simple,&#10;post-training and model agnostic approach to remove the language identifying&#10;information from the representation while still retaining sentence semantics.&#10;" />
          <attvalue for="2" value="&#10;Sentence embeddings map sentences into a vector space. The vectors capture rich semantic information that can be used to measure semantic textual similarity~(STS) between sentences or train classifiers for a broad range of downstream tasks~\cite{infersent,subramanian2018learning,qk,use,sentbert,muse,gem}.&#10;State-of-the-art models are usually trained on supervised tasks such as natural language inference~\cite{infersent}, or with semi-structured data like question-answer pairs~\cite{use} and translation pairs~\cite{subramanian2018learning,muse}.&#10;However, labeled and semi-structured data are difficult and expensive to obtain, making it hard to cover many domains and languages.&#10;Conversely, recent efforts to improve language models include the development of masked language model (MLM) pre-training from large scale unlabeled corpora \cite{bert,albert,roberta}. &#10;While internal MLM model representations are helpful when fine-tuning on downstream tasks, they do not directly produce good sentence representations,&#10;without further supervised \cite{sentbert} or semi-structured \cite{labse} fine-tuning.&#10;&#10;In this paper, we explore an unsupervised approach, called Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations from large scale unlabeled corpora. The CMLM model architecture is illustrated in \ref{fig:cmlm}, which integrates sentence representation learning into MLM training by conditioning on sentence level representations produced by adjacent sentences. The model therefore needs to learn effective sentence representations in order to perform good MLM.&#10;Since CMLM is fully unsupervised, it can be easily extended to new languages. We explore CMLM for both English and multilingual sentence embeddings for 100+ languages. &#10;Our English CMLM model achieves state-of-the-art performance on SentEval~\cite{senteval}, even outperforming models learned using (semi-)supervised signals. Moreover, models training on the English Amazon review data using our multilingual vectors exhibit strong multilingual transfer performance on translations of the Amazon review evaluation data to French, German and Japanese, outperforming existing multilingual sentence embedding models by $&gt;5\%$ for non-English languages and by $&gt; 2\%$ on English. &#10;&#10;We further extend the multilingual CMLM to co-train with parallel text (bitext) retrieval task, and finetune with cross-lingual natural language inference (NLI) data, inspired by the success of prior work on multitask sentence representation learning~\cite{subramanian2018learning,muse,distill-mling} and NLI learning~\cite{infersent,sentbert}. We achieve performance $3.6\%$ better than the previous state-of-the-art multilingual sentence representation model (see details in \ref{sec:multitask}). On cross-lingual semantic search task, our model outperforms baseline models by $10\%$ on average over 36 languages.&#10;Language agnostic representations require semantically similar cross-lingual pairs to be closer in representation space than unrelated same-language pairs~\cite{lareqa}. While we find our original sentence embeddings do have a bias for same language sentences, we discover that removing the first few principal components of the embeddings eliminates the self language bias.&#10;&#10;The rest of the paper is organized as follows. \ref{sec:cmlm} describes the architecture for CMLM unsupervised learning. In \ref{sec:en_cmlm} we present CMLM trained on English data and evaluation results on SentEval. In \ref{sec:mling_cmlm} we apply CMLM to learn sentence multilingual sentence representations. Multitask training strategies on how to effectively combining CMLM, bitext retrieval and cross-lingual NLI &#10;finetuning are explored. In \ref{sec:analysis}, we investigate self language bias in multilingual representations and propose a simple but effective approach to eliminate it. The pre-trained models&#10;are released at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Sentence Representation, Artificial Intelligence, Mathematics, Multilingual Learning, Language Modeling" />
        </attvalues>
      </node>
      <node id="2004.05484" label="2004.05484">
        <attvalues>
          <attvalue for="0" value="LAReQA: Language-agnostic answer retrieval from a multilingual pool" />
          <attvalue for="1" value="  We present LAReQA, a challenging new benchmark for language-agnostic answer&#10;retrieval from a multilingual candidate pool. Unlike previous cross-lingual&#10;tasks, LAReQA tests for &quot;strong&quot; cross-lingual alignment, requiring&#10;semantically related cross-language pairs to be closer in representation space&#10;than unrelated same-language pairs. Building on multilingual BERT (mBERT), we&#10;study different strategies for achieving strong alignment. We find that&#10;augmenting training data via machine translation is effective, and improves&#10;significantly over using mBERT out-of-the-box. Interestingly, the embedding&#10;baseline that performs the best on LAReQA falls short of competing baselines on&#10;zero-shot variants of our task that only target &quot;weak&quot; alignment. This finding&#10;underscores our claim that languageagnostic retrieval is a substantively new&#10;kind of cross-lingual evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.11920" label="2312.11920">
        <attvalues>
          <attvalue for="0" value="External Knowledge Augmented Polyphone Disambiguation Using Large&#10;  Language Model" />
          <attvalue for="1" value="  One of the key issues in Mandarin Chinese text-to-speech (TTS) systems is&#10;polyphone disambiguation when doing grapheme-to-phoneme (G2P) conversion. In&#10;this paper, we introduce a novel method to solve the problem as a generation&#10;task. Following the trending research of large language models (LLM) and prompt&#10;learning, the proposed method consists of three modules. Retrieval module&#10;incorporates external knowledge which is a multi-level semantic dictionary of&#10;Chinese polyphonic characters to format the sentence into a prompt. Generation&#10;module adopts the decoder-only Transformer architecture to induce the target&#10;text. Postprocess module corrects the generated text into a valid result if&#10;needed. Experimental results show that our method outperforms the existing&#10;methods on a public dataset called CPP. We also empirically study the impacts&#10;of different templates of the prompt, different sizes of training data, and&#10;whether to incorporate external knowledge.&#10;" />
          <attvalue for="2" value="&#10;&#10;Grapheme-to-phoneme (G2P) conversion plays an important role in the front end of text-to-speech (TTS) synthesis systems, which is mainly a text-processing module. The target of G2P in Mandarin Chinese TTS is to convert Chinese characters into pinyin which is the pronunciation system of Chinese in the form of the Latin alphabet. However, as a symbolic language, some Chinese characters may have more than one pronunciation, and they are often called polyphonic characters. Therefore, it is necessary to do polyphone disambiguation before synthesizing speech, and usually, collocation and context information around the characters would be leveraged. Figure \ref{fig:f1} shows some examples of this phenomenon. Generally, a polyphonic character has a commonly used pinyin and a few rare pinyin.&#10;&#10;There has already been a lot of research focused on the problem of polyphone disambiguation. In early publications, some rule-based approaches were proposed, which heavily relied on the pronunciation dictionary and pre-defined rules. For example, utilizing the frequency of different pronunciation and context patterns, which can be mined from a large corpus, is quite straightforward \cite{zhang2001disambiguation}. However, the construction of hand-crafted rules is costly and sometimes inaccurate. Some researchers further build a decision list for polyphonic characters through collocation distribution in a context where they appear. Based on the decision list, the likelihood of a pinyin in a certain collocation can be calculated \cite{dong2004grapheme}. Later, some statistical methods achieve good benchmarks, like Decision Trees (DT) and Maximum Entropy (ME) models \cite{You2011PolyphoneDB,2012Polyphone}.&#10;&#10;Recently, with the advancement of Neural Networks (NN), a lot of studies have been applying NN models to polyphone disambiguation. Most of the research describes the problem as a classification task, such as adopting a bi-directional long-short-term memory (BLSTM) layer to encode the input sentence and an output layer for producing probability distribution over all candidate pronunciations \cite{rao2015grapheme,shan2016bi}. Furthermore, a few studies show that taking advantage of multi-level embedding features is helpful, including characters, words, and their positions \cite{cai2019polyphone,dai2019disambiguation}. There is also a series of research that defines the G2P conversion as a Machine Translation (MT) task. A paradigm of encoder-decoder generation is employed in the task to train a Seq2Seq model with attention mechanism \cite{yao2015sequence,zhang2020distant}.&#10;&#10;The existing methods are suffering from a lack of supervised data and deep semantic features. As the trends of large language models (LLM) grow, a lot of tasks manage to follow the generation paradigm to acquire stronger generalizability and flexibility. GPT \cite{radford2018improving,brown2020language} uses a massive amount of unlabeled text to pre-train a multi-layer Transformer decoder which shows better results on both natural language understanding and generation tasks. Based on autoregressive models, GLM \cite{du2021glm} improves blank filling pretraining by 2D positional encodings and supports different types of tasks by varying the number and lengths of blanks.&#10;&#10;Inspired by LLM, we propose an approach to solving the problem by language models and external knowledge. Specifically, we consider polyphone disambiguation as a generation task that takes the sentence and essential information as model input and directly generates the target pinyin. As known by previous research, the performance of LLM heavily relies on the input prompt, so we also augment it by adding external knowledge such as a semantic dictionary. It can be easily crawled from the Internet and it would be restructured to a multi-level knowledge base for retrieval. Unlike the past methods, our design can also predict unseen characters without any incremental training only if they are in the dictionary. The contribution of this paper can be summarized as follows:&#10;\begin{enumerate}&#10; \item The proposed method is the first work to introduce LLM techniques into polyphone disambiguation.&#10; \item We utilize a multi-level semantic dictionary as external knowledge for better performance as well as the adaption to unseen characters.&#10; \item With the external knowledge, the proposed method outperforms the baseline models in accuracy on the CPP dataset.&#10;\end{enumerate} &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Generation, Linguistics, Chinese Polyphone Disambiguation, Text-to-Speech Systems, Artificial Intelligence, Information Science" />
        </attvalues>
      </node>
      <node id="1907.01749" label="1907.01749">
        <attvalues>
          <attvalue for="0" value="Polyphone Disambiguation for Mandarin Chinese Using Conditional Neural&#10;  Network with Multi-level Embedding Features" />
          <attvalue for="1" value="  This paper describes a conditional neural network architecture for Mandarin&#10;Chinese polyphone disambiguation. The system is composed of a bidirectional&#10;recurrent neural network component acting as a sentence encoder to accumulate&#10;the context correlations, followed by a prediction network that maps the&#10;polyphonic character embeddings along with the conditions to corresponding&#10;pronunciations. We obtain the word-level condition from a pre-trained&#10;word-to-vector lookup table. One goal of polyphone disambiguation is to address&#10;the homograph problem existing in the front-end processing of Mandarin Chinese&#10;text-to-speech system. Our system achieves an accuracy of 94.69\% on a publicly&#10;available polyphonic character dataset. To further validate our choices on the&#10;conditional feature, we investigate polyphone disambiguation systems with&#10;multi-level conditions respectively. The experimental results show that both&#10;the sentence-level and the word-level conditional embedding features are able&#10;to attain good performance for Mandarin Chinese polyphone disambiguation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.00196" label="1506.00196">
        <attvalues>
          <attvalue for="0" value="Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme&#10;  Conversion" />
          <attvalue for="1" value="  Sequence-to-sequence translation methods based on generation with a&#10;side-conditioned language model have recently shown promising results in&#10;several tasks. In machine translation, models conditioned on source side words&#10;have been used to produce target-language text, and in image captioning, models&#10;conditioned images have been used to generate caption text. Past work with this&#10;approach has focused on large vocabulary tasks, and measured quality in terms&#10;of BLEU. In this paper, we explore the applicability of such models to the&#10;qualitatively different grapheme-to-phoneme task. Here, the input and output&#10;side vocabularies are small, plain n-gram models do well, and credit is only&#10;given when the output is exactly correct. We find that the simple&#10;side-conditioned generation approach is able to rival the state-of-the-art, and&#10;we are able to significantly advance the stat-of-the-art with bi-directional&#10;long short-term memory (LSTM) neural networks that use the same alignment&#10;information that is used in conventional approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15643" label="2203.15643">
        <attvalues>
          <attvalue for="0" value="Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise&#10;  Distillation" />
          <attvalue for="1" value="  Several solutions for lightweight TTS have shown promising results. Still,&#10;they either rely on a hand-crafted design that reaches non-optimum size or use&#10;a neural architecture search but often suffer training costs. We present&#10;Nix-TTS, a lightweight TTS achieved via knowledge distillation to a&#10;high-quality yet large-sized, non-autoregressive, and end-to-end (vocoder-free)&#10;TTS teacher model. Specifically, we offer module-wise distillation, enabling&#10;flexible and independent distillation to the encoder and decoder module. The&#10;resulting Nix-TTS inherited the advantageous properties of being&#10;non-autoregressive and end-to-end from the teacher, yet significantly smaller&#10;in size, with only 5.23M parameters or up to 89.34% reduction of the teacher&#10;model; it also achieves over 3.04x and 8.36x inference speedup on Intel-i7 CPU&#10;and Raspberry Pi 3B respectively and still retains a fair voice naturalness and&#10;intelligibility compared to the teacher model. We provide pretrained models and&#10;audio samples of Nix-TTS.&#10;" />
          <attvalue for="2" value="&#10;&#10;Synthetic voices generated by several recent neural TTS models~\cite{ren2020fastspeech, donahue2020end, kim2020glow, kim2021conditional} have been able to achieve high naturalness and intelligibility compared to the real recordings it was trained on. However, most of the models are known to be quite large in size and suffer from slow CPU inference. These limitations hinder the possibility of deploying a real-time, highly natural, and intelligible voice-based interface in low-cost and resource-constrained settings. Although there have been several embedded devices with neural accelerator such as NVIDIA Jetson Nano or Google Coral Dev Board to mitigate said problems, the cost and its availability are still varies. The most challenging is to deploy a TTS in a low-cost CPU-bound devices. In such settings, there are needs for neural TTS models to be inherently lightweight and fast, yet have a good naturalness and intelligibility.&#10;&#10;Recent works on lightweight neural TTS models have shown promising results in fulfilling those needs. Several lightweight neural acoustic models design are proposed in~\cite{vainer2020speedyspeech, lee2021bidirectional, beliaev2021talknet}. The size of these models ranges from $4.3$M to $15$M, more than a half smaller compared to established high-quality TTS like~\cite{tacotron2, ren2020fastspeech} which is around $30$M in size. However, these models mainly focus on text-to-Mel, requiring additional neural vocoder to synthesize the waveforms, which variably inflates the model size depending on the chosen vocoder models. The work by ~\cite{nguyen2021litetts} then proposed an end-to-end design of a lightweight TTS model eliminating the need for a neural vocoder. But unfortunately, it could only achieve the final size of above $10$M (about $13$M) parameters which is not yet optimal.&#10;&#10;Another solution is to utilize neural compression methods that do not necessarily require manually designing the model and can reduce the size even further. To date, ~\cite{luo2021lightspeech} proposed applying neural architecture search to automatically design an acoustic model; the discovered model is only $1.8$M in size while retaining good naturalness. However, the training cost is arguably high, as it requires sorting through all the possible architecture in the defined search space, making the approach not always feasible for researchers with limited computing resources. Another work by~\cite{lai2021interplay} shows that a neural acoustic model is highly prunable. Even after pruning $90\%$ of its original model (achieving around $3.0$M parameters in size), the pruned model still retains perceivable intelligibility. However, the naturalness of the generated voices is degraded.&#10;&#10;As can be seen, the existing works still either rely on a hand-crafted design that reaches non-optimum size or use a neural architecture search but often suffer training costs. Therefore, a novel solution that can achieve a very small-size model without heavy training costs while keeping the performance is necessary.&#10;&#10;In this work, we present Nix-TTS, a lightweight TTS achieved via knowledge distillation (KD)~\cite{hinton2015distilling} to a high-quality yet large-sized, non-autoregressive, and end-to-end (vocoder-free) TTS teacher model. However, in contrast with previous work~\cite{vainer2020speedyspeech} that performed KD by only distilled the teacher network’s duration or the work by ~\cite{oord2018parallel, ping2018clarinet} that only utilized KD by compressing neural vocoders, our proposed approach was performed on an end-to-end TTS model. Specifically, we offer a novel module-wise distillation, enabling flexible and independent distillation to the encoder and decoder module. The overall process does not suffer from the training cost and the resulting Nix-TTS:&#10;(1) inherits the teacher properties of being a non-autoregressive and end-to-end framework without the need for an additional vocoder,&#10;(2) achieves a significantly smaller size and inference speedup,&#10;(3) while keeping a fair voice naturalness and intelligibility compared to the teacher model. In addition, we tested our model in CPU-bound devices, a single-thread Intel i7 CPU and Raspberry Pi Model 3B, without any type of neural acceleration. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Signal Processing, Text To Speech, Neural Architecture, Model Optimization" />
        </attvalues>
      </node>
      <node id="2006.04558" label="2006.04558">
        <attvalues>
          <attvalue for="0" value="FastSpeech 2: Fast and High-Quality End-to-End Text to Speech" />
          <attvalue for="1" value="  Non-autoregressive text to speech (TTS) models such as FastSpeech can&#10;synthesize speech significantly faster than previous autoregressive models with&#10;comparable quality. The training of FastSpeech model relies on an&#10;autoregressive teacher model for duration prediction (to provide more&#10;information as input) and knowledge distillation (to simplify the data&#10;distribution in output), which can ease the one-to-many mapping problem (i.e.,&#10;multiple speech variations correspond to the same text) in TTS. However,&#10;FastSpeech has several disadvantages: 1) the teacher-student distillation&#10;pipeline is complicated and time-consuming, 2) the duration extracted from the&#10;teacher model is not accurate enough, and the target mel-spectrograms distilled&#10;from teacher model suffer from information loss due to data simplification,&#10;both of which limit the voice quality. In this paper, we propose FastSpeech 2,&#10;which addresses the issues in FastSpeech and better solves the one-to-many&#10;mapping problem in TTS by 1) directly training the model with ground-truth&#10;target instead of the simplified output from teacher, and 2) introducing more&#10;variation information of speech (e.g., pitch, energy and more accurate&#10;duration) as conditional inputs. Specifically, we extract duration, pitch and&#10;energy from speech waveform and directly take them as conditional inputs in&#10;training and use predicted values in inference. We further design FastSpeech&#10;2s, which is the first attempt to directly generate speech waveform from text&#10;in parallel, enjoying the benefit of fully end-to-end inference. Experimental&#10;results show that 1) FastSpeech 2 achieves a 3x training speed-up over&#10;FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech&#10;2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even&#10;surpass autoregressive models. Audio samples are available at&#10;https://speechresearch.github.io/fastspeech2/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.03802" label="2008.03802">
        <attvalues>
          <attvalue for="0" value="SpeedySpeech: Efficient Neural Speech Synthesis" />
          <attvalue for="1" value="  While recent neural sequence-to-sequence models have greatly improved the&#10;quality of speech synthesis, there has not been a system capable of fast&#10;training, fast inference and high-quality audio synthesis at the same time. We&#10;propose a student-teacher network capable of high-quality faster-than-real-time&#10;spectrogram synthesis, with low requirements on computational resources and&#10;fast training time. We show that self-attention layers are not necessary for&#10;generation of high quality audio. We utilize simple convolutional blocks with&#10;residual connections in both student and teacher networks and use only a single&#10;attention layer in the teacher model. Coupled with a MelGAN vocoder, our&#10;model's voice quality was rated significantly higher than Tacotron 2. Our model&#10;can be efficiently trained on a single GPU and can run in real time even on a&#10;CPU. We provide both our source code and audio samples in our GitHub&#10;repository.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.04040" label="2102.04040">
        <attvalues>
          <attvalue for="0" value="LightSpeech: Lightweight and Fast Text to Speech with Neural&#10;  Architecture Search" />
          <attvalue for="1" value="  Text to speech (TTS) has been broadly used to synthesize natural and&#10;intelligible speech in different scenarios. Deploying TTS in various end&#10;devices such as mobile phones or embedded devices requires extremely small&#10;memory usage and inference latency. While non-autoregressive TTS models such as&#10;FastSpeech have achieved significantly faster inference speed than&#10;autoregressive models, their model size and inference latency are still large&#10;for the deployment in resource constrained devices. In this paper, we propose&#10;LightSpeech, which leverages neural architecture search~(NAS) to automatically&#10;design more lightweight and efficient models based on FastSpeech. We first&#10;profile the components of current FastSpeech model and carefully design a novel&#10;search space containing various lightweight and potentially effective&#10;architectures. Then NAS is utilized to automatically discover well performing&#10;architectures within the search space. Experiments show that the model&#10;discovered by our method achieves 15x model compression ratio and 6.5x&#10;inference speedup on CPU with on par voice quality. Audio demos are provided at&#10;https://speechresearch.github.io/lightspeech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.01147" label="2110.01147">
        <attvalues>
          <attvalue for="0" value="On the Interplay Between Sparsity, Naturalness, Intelligibility, and&#10;  Prosody in Speech Synthesis" />
          <attvalue for="1" value="  Are end-to-end text-to-speech (TTS) models over-parametrized? To what extent&#10;can these models be pruned, and what happens to their synthesis capabilities?&#10;This work serves as a starting point to explore pruning both spectrogram&#10;prediction networks and vocoders. We thoroughly investigate the tradeoffs&#10;between sparsity and its subsequent effects on synthetic speech. Additionally,&#10;we explored several aspects of TTS pruning: amount of finetuning data versus&#10;sparsity, TTS-Augmentation to utilize unspoken text, and combining knowledge&#10;distillation and pruning. Our findings suggest that not only are end-to-end TTS&#10;models highly prunable, but also, perhaps surprisingly, pruned TTS models can&#10;produce synthetic speech with equal or higher naturalness and intelligibility,&#10;with similar prosody. All of our experiments are conducted on publicly&#10;available models, and findings in this work are backed by large-scale&#10;subjective tests and objective measures. Code and 200 pruned models are made&#10;available to facilitate future research on efficiency in TTS.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.10433" label="1711.10433">
        <attvalues>
          <attvalue for="0" value="Parallel WaveNet: Fast High-Fidelity Speech Synthesis" />
          <attvalue for="1" value="  The recently-developed WaveNet architecture is the current state of the art&#10;in realistic speech synthesis, consistently rated as more natural sounding for&#10;many different languages than any previous system. However, because WaveNet&#10;relies on sequential generation of one audio sample at a time, it is poorly&#10;suited to today's massively parallel computers, and therefore hard to deploy in&#10;a real-time production setting. This paper introduces Probability Density&#10;Distillation, a new method for training a parallel feed-forward network from a&#10;trained WaveNet with no significant difference in quality. The resulting system&#10;is capable of generating high-fidelity speech samples at more than 20 times&#10;faster than real-time, and is deployed online by Google Assistant, including&#10;serving multiple English and Japanese voices.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12725" label="2010.12725">
        <attvalues>
          <attvalue for="0" value="Compositional Generalization and Natural Language Variation: Can a&#10;  Semantic Parsing Approach Handle Both?" />
          <attvalue for="1" value="  Sequence-to-sequence models excel at handling natural language variation, but&#10;have been shown to struggle with out-of-distribution compositional&#10;generalization. This has motivated new specialized architectures with stronger&#10;compositional biases, but most of these approaches have only been evaluated on&#10;synthetically-generated datasets, which are not representative of natural&#10;language variation. In this work we ask: can we develop a semantic parsing&#10;approach that handles both natural language variation and compositional&#10;generalization? To better assess this capability, we propose new train and test&#10;splits of non-synthetic datasets. We demonstrate that strong existing&#10;approaches do not perform well across a broad set of evaluations. We also&#10;propose NQG-T5, a hybrid model that combines a high-precision grammar-based&#10;approach with a pre-trained sequence-to-sequence model. It outperforms existing&#10;approaches across several compositional generalization challenges on&#10;non-synthetic data, while also being competitive with the state-of-the-art on&#10;standard evaluations. While still far from solving this problem, our study&#10;highlights the importance of diverse evaluations and the open challenge of&#10;handling both compositional generalization and natural language variation in&#10;semantic parsing.&#10;" />
          <attvalue for="2" value="&#10;&#10;Sequence-to-sequence (seq2seq) models have been widely used in semantic parsing~\cite{dong2016language, jia2016data} and excel at handling the natural language variation of human-generated queries. However, evaluations on synthetic&#10;tasks such as SCAN~\cite{lake2018generalization} have shown that seq2seq models often generalize poorly to out-of-distribution compositional utterances, such as ``jump twice'' when only ``jump'', ``walk'', and ``walk twice'' are seen during training. This ability to generalize to novel combinations of the elements observed during training is referred to as compositional generalization.&#10;&#10;This has motivated many specialized architectures that improve peformance on SCAN~\cite{li2019compositional,russin2019compositional,gordon2019permutation,lake2019compositional,liu2020compositional,nye2020learning,chen2020compositional}. However, most approaches have only been evaluated on synthetic datasets.&#10;While synthetic datasets enable precise, interpretable evaluation of specific phenomena, they are less representative of the natural language variation that a real-world semantic parsing system must handle.&#10;&#10;In this paper, we ask: {can we develop a semantic parsing approach that handles both natural language variation and compositional generalization?} Surprisingly, this question is understudied. As visualized in Figure~\ref{fig:challenge}, most prior work evaluates either out-of-distribution compositional generalization on synthetic datasets, or in-distribution performance on non-synthetic datasets.&#10;Notably, designing approaches that can handle both compositional generalization and the natural language variation of non-synthetic datasets is difficult.\eat{, as different approaches have their own strengths and limitations.} For example, large pre-trained seq2seq models that perform well on in-distribution evaluations do not address most of the compositional generalization challenges proposed in SCAN~\cite{furrer2020compositional}.&#10;&#10;Our research question has two important motivations. First, humans have been shown to be adept compositional learners~\cite{lake2019human}. Several authors have argued that a greater focus on compositional generalization is an important path to more human-like generalization and NLU~\cite{lake2017building,battaglia2018relational}. &#10;Second, it is practically important to assess performance on non-synthetic data and out-of-distribution examples, as random train and test splits can overestimate real-world performance and miss important error cases~\cite{ribeiro2020beyond}. Therefore, we are interested in approaches that do well not only on controlled synthetic challenges of compositionality or in-distribution natural utterances, but across all of the diverse set of evaluations shown in Figure~\ref{fig:evaluations}.&#10;&#10;Our contributions are two-fold. First, on the evaluation front,&#10;we show that performance on SCAN is not well-correlated with performance on non-synthetic tasks. In addition, strong existing approaches do not perform well across all evaluations in Figure~\ref{fig:evaluations}.&#10;We also propose new Target Maximum Compound Divergence (\tmcd) train and test splits, extending the methodology of~\cite{keysers2019measuring} to create challenging evaluations of compositional generalization for non-synthetic datasets. We show that \tmcd splits complement existing evaluations by focusing on different aspects of the problem.&#10;&#10;Second, on the modeling front, we propose \qg, a simple and general grammar-based approach that solves SCAN and also scales to natural utterances, obtaining high precision for non-synthetic data. In addition, we introduce and evaluate \qgxt, a hybrid model that combines \qg with T5~\cite{raffel2019exploring}, leading to improvements across several compositional generalization evaluations while also being competitive on the standard splits of GeoQuery~\cite{zelle1996learning} and Spider~\cite{yu2018spider}. Our results indicate that \qgxt is a strong baseline for our challenge of developing approaches that perform well across a diverse set of evaluations focusing on either natural language variation, compositional generalization, or both. Comparing five approaches across eight evaluations on SCAN and GeoQuery, its average rank is 1, with the rank of the best previous approach (T5) being 2.9; performance is also competitive across several evaluations on Spider.&#10;&#10;While still far from affirmatively answering our research question, &#10;our study highlights the importance of a diverse set of evaluations and the open challenge of handling both compositional generalization and natural language variation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Semantic Parsing, Computer Science, Linguistics, Natural Language Variation, Compositional Generalization, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1601.01280" label="1601.01280">
        <attvalues>
          <attvalue for="0" value="Language to Logical Form with Neural Attention" />
          <attvalue for="1" value="  Semantic parsing aims at mapping natural language to machine interpretable&#10;meaning representations. Traditional approaches rely on high-quality lexicons,&#10;manually-built templates, and linguistic features which are either domain- or&#10;representation-specific. In this paper we present a general method based on an&#10;attention-enhanced encoder-decoder model. We encode input utterances into&#10;vector representations, and generate their logical forms by conditioning the&#10;output sequences or trees on the encoding vectors. Experimental results on four&#10;datasets show that our approach performs competitively without using&#10;hand-engineered features and is easy to adapt across domains and meaning&#10;representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.02612" label="1910.02612">
        <attvalues>
          <attvalue for="0" value="Compositional Generalization for Primitive Substitutions" />
          <attvalue for="1" value="  Compositional generalization is a basic mechanism in human language learning,&#10;but current neural networks lack such ability. In this paper, we conduct&#10;fundamental research for encoding compositionality in neural networks.&#10;Conventional methods use a single representation for the input sentence, making&#10;it hard to apply prior knowledge of compositionality. In contrast, our approach&#10;leverages such knowledge with two representations, one generating attention&#10;maps, and the other mapping attended input words to output symbols. We reduce&#10;the entropy in each representation to improve generalization. Our experiments&#10;demonstrate significant improvements over the conventional methods in five NLP&#10;tasks including instruction learning and machine translation. In the SCAN&#10;domain, it boosts accuracies from 14.0% to 98.8% in Jump task, and from 92.0%&#10;to 99.7% in TurnLeft task. It also beats human performance on a few-shot&#10;learning task. We hope the proposed approach can help ease future research&#10;towards human-level compositional language learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.05381" label="1906.05381">
        <attvalues>
          <attvalue for="0" value="Compositional generalization through meta sequence-to-sequence learning" />
          <attvalue for="1" value="  People can learn a new concept and use it compositionally, understanding how&#10;to &quot;blicket twice&quot; after learning how to &quot;blicket.&quot; In contrast, powerful&#10;sequence-to-sequence (seq2seq) neural networks fail such tests of&#10;compositionality, especially when composing new concepts together with existing&#10;concepts. In this paper, I show how memory-augmented neural networks can be&#10;trained to generalize compositionally through meta seq2seq learning. In this&#10;approach, models train on a series of seq2seq problems to acquire the&#10;compositional skills needed to solve new seq2seq problems. Meta se2seq learning&#10;solves several of the SCAN tests for compositional learning and can learn to&#10;apply implicit rules to variables.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.10627" label="2006.10627">
        <attvalues>
          <attvalue for="0" value="Compositional Generalization by Learning Analytical Expressions" />
          <attvalue for="1" value="  Compositional generalization is a basic and essential intellective capability&#10;of human beings, which allows us to recombine known parts readily. However,&#10;existing neural network based models have been proven to be extremely deficient&#10;in such a capability. Inspired by work in cognition which argues&#10;compositionality can be captured by variable slots with symbolic functions, we&#10;present a refreshing view that connects a memory-augmented neural model with&#10;analytical expressions, to achieve compositional generalization. Our model&#10;consists of two cooperative neural modules, Composer and Solver, fitting well&#10;with the cognitive argument while being able to be trained in an end-to-end&#10;manner via a hierarchical reinforcement learning algorithm. Experiments on the&#10;well-known benchmark SCAN demonstrate that our model seizes a great ability of&#10;compositional generalization, solving all challenges addressed by previous&#10;works with 100% accuracies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.05562" label="2003.05562">
        <attvalues>
          <attvalue for="0" value="Learning Compositional Rules via Neural Program Synthesis" />
          <attvalue for="1" value="  Many aspects of human reasoning, including language, require learning rules&#10;from very little data. Humans can do this, often learning systematic rules from&#10;very few examples, and combining these rules to form compositional rule-based&#10;systems. Current neural architectures, on the other hand, often fail to&#10;generalize in a compositional manner, especially when evaluated in ways that&#10;vary systematically from training. In this work, we present a neuro-symbolic&#10;model which learns entire rule systems from a small set of examples. Instead of&#10;directly predicting outputs from inputs, we train our model to induce the&#10;explicit system of rules governing a set of previously seen examples, drawing&#10;upon techniques from the neural program synthesis literature. Our&#10;rule-synthesis approach outperforms neural meta-learning techniques in three&#10;domains: an artificial instruction-learning domain used to evaluate human&#10;learning, the SCAN challenge datasets, and learning rule-based translations of&#10;number words into integers for a wide range of human languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.06662" label="2008.06662">
        <attvalues>
          <attvalue for="0" value="Compositional Generalization via Neural-Symbolic Stack Machines" />
          <attvalue for="1" value="  Despite achieving tremendous success, existing deep learning models have&#10;exposed limitations in compositional generalization, the capability to learn&#10;compositional rules and apply them to unseen cases in a systematic manner. To&#10;tackle this issue, we propose the Neural-Symbolic Stack Machine (NeSS). It&#10;contains a neural network to generate traces, which are then executed by a&#10;symbolic stack machine enhanced with sequence manipulation operations. NeSS&#10;combines the expressive power of neural sequence models with the recursion&#10;supported by the symbolic stack machine. Without training supervision on&#10;execution traces, NeSS achieves 100% generalization performance in four&#10;domains: the SCAN benchmark of language-driven navigation tasks, the task of&#10;few-shot learning of compositional instructions, the compositional machine&#10;translation benchmark, and context-free grammar parsing tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.04587" label="1901.04587">
        <attvalues>
          <attvalue for="0" value="Human few-shot learning of compositional instructions" />
          <attvalue for="1" value="  People learn in fast and flexible ways that have not been emulated by&#10;machines. Once a person learns a new verb &quot;dax,&quot; he or she can effortlessly&#10;understand how to &quot;dax twice,&quot; &quot;walk and dax,&quot; or &quot;dax vigorously.&quot; There have&#10;been striking recent improvements in machine learning for natural language&#10;processing, yet the best algorithms require vast amounts of experience and&#10;struggle to generalize new concepts in compositional ways. To better understand&#10;these distinctively human abilities, we study the compositional skills of&#10;people through language-like instruction learning tasks. Our results show that&#10;people can learn and use novel functional concepts from very few examples&#10;(few-shot learning), successfully applying familiar functions to novel inputs.&#10;People can also compose concepts in complex ways that go beyond the provided&#10;demonstrations. Two additional experiments examined the assumptions and&#10;inductive biases that people make when solving these tasks, revealing three&#10;biases: mutual exclusivity, one-to-one mappings, and iconic concatenation. We&#10;discuss the implications for cognitive modeling and the potential for building&#10;machines with more human-like language learning capabilities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.01261" label="1806.01261">
        <attvalues>
          <attvalue for="0" value="Relational inductive biases, deep learning, and graph networks" />
          <attvalue for="1" value="  Artificial intelligence (AI) has undergone a renaissance recently, making&#10;major progress in key domains such as vision, language, control, and&#10;decision-making. This has been due, in part, to cheap data and cheap compute&#10;resources, which have fit the natural strengths of deep learning. However, many&#10;defining characteristics of human intelligence, which developed under much&#10;different pressures, remain out of reach for current approaches. In particular,&#10;generalizing beyond one's experiences--a hallmark of human intelligence from&#10;infancy--remains a formidable challenge for modern AI.&#10;  The following is part position paper, part review, and part unification. We&#10;argue that combinatorial generalization must be a top priority for AI to&#10;achieve human-like abilities, and that structured representations and&#10;computations are key to realizing this objective. Just as biology uses nature&#10;and nurture cooperatively, we reject the false choice between&#10;&quot;hand-engineering&quot; and &quot;end-to-end&quot; learning, and instead advocate for an&#10;approach which benefits from their complementary strengths. We explore how&#10;using relational inductive biases within deep learning architectures can&#10;facilitate learning about entities, relations, and rules for composing them. We&#10;present a new building block for the AI toolkit with a strong relational&#10;inductive bias--the graph network--which generalizes and extends various&#10;approaches for neural networks that operate on graphs, and provides a&#10;straightforward interface for manipulating structured knowledge and producing&#10;structured behaviors. We discuss how graph networks can support relational&#10;reasoning and combinatorial generalization, laying the foundation for more&#10;sophisticated, interpretable, and flexible patterns of reasoning. As a&#10;companion to this paper, we have released an open-source software library for&#10;building graph networks, with demonstrations of how to use them in practice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.00804" label="2102.00804">
        <attvalues>
          <attvalue for="0" value="Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR&#10;  Transcript" />
          <attvalue for="1" value="  Recent years have witnessed significant improvement in ASR systems to&#10;recognize spoken utterances. However, it is still a challenging task for noisy&#10;and out-of-domain data, where substitution and deletion errors are prevalent in&#10;the transcribed text. These errors significantly degrade the performance of&#10;downstream tasks. In this work, we propose a BERT-style language model,&#10;referred to as PhonemeBERT, that learns a joint language model with phoneme&#10;sequence and ASR transcript to learn phonetic-aware representations that are&#10;robust to ASR errors. We show that PhonemeBERT can be used on downstream tasks&#10;using phoneme sequences as additional features, and also in low-resource setup&#10;where we only have ASR-transcripts for the downstream tasks with no phoneme&#10;information available. We evaluate our approach extensively by generating noisy&#10;data for three benchmark datasets - Stanford Sentiment Treebank, TREC and ATIS&#10;for sentiment, question and intent classification tasks respectively. The&#10;results of the proposed approach beats the state-of-the-art baselines&#10;comprehensively on each dataset.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the proliferation of voice-enabled technologies&#10;spoken language understanding (SLU) has become significantly ubiquitous.&#10;The general modus-operandi of SLU systems is to convert voice into text using an ASR engine and use Natural Language Understanding (NLU) on the transcribed text to comprehend the speaker’s intents and requests. Despite advancements in ASR systems, domain adaptation and word recognition in noisy setups remain a big challenge. &#10;In a typical SLU system that operates on the ASR outputs, these errors degrade the performance of the system on the downstream tasks \cite{DBLP:conf/asru/DesotPV19}.&#10;&#10;The approaches tried by scientific community to address the errors in the ASR system can be broadly categorized into four groups: a) Modelling word confidence: Liu et al. \cite{liu2020jointlyArxiv} proposes a BERT model that jointly encodes the word confidence network and the dialog context. Ladhak et al. \cite{DBLP:conf/interspeech/LadhakGDMRH16} proposes LatticeRNN to encode the ambiguities of the ASR recognition for the intent classification task; b) ASR correction: Weng et al. \cite{DBLP:conf/icassp/WengMKWZMNPWBT20} presents a contextual language correction on ASR outputs jointly with modelling LU task that learns from ASR n-best transcriptions. Mani et al. \cite{mani2020towards} use machine translation technique for domain adaptation to correct ASR mistakes in medical conversations; c) End-to-End SLU: Serdyuk et al. \cite{DBLP:conf/icassp/SerdyukWFKLB18} explore the possibility to extend the end-to-end ASR learning to include NLU component and optimize the whole system for SLU task while Ghannay et al. \cite{DBLP:conf/slt/GhannayCECSLM18} study end-to-end named entity and semantic concept extraction from speech to circumvent the errors arising from the ASR pipeline; d) Phoneme enhanced representations: Yenigalla et al. \cite{DBLP:conf/interspeech/YenigallaKTSKV18} use word2vec generated phoneme embedding for emotion recognition task. Fang et al. \cite{DBLP:conf/sigir/FangFLR20} propose word2vec based approaches to learn phoneme embeddings capturing pronunciation similarities of phonemes to make classification robust to ASR errors.&#10;&#10;Our work falls in the category of Phoneme enhanced representations to learn representations that are robust to ASR transcription errors. In one of the earlier works, Yenigalla et al. \cite{DBLP:conf/interspeech/YenigallaKTSKV18} trained word2vec based phoneme and ASR embeddings independent of each other for SLU tasks. &#10;Fang et al. \cite{DBLP:conf/sigir/FangFLR20} propose multiple variants of phoneme embedding which either require alignments between the reference transcript and the phoneme sequence (p2va) or learns phoneme embedding in isolation to the ASR transcript (p2vc, s2s). Additionally, the authors derive phoneme sequence obtained on top of the ASR transcripts. We note two main drawbacks in these methods: a) Learning embeddings from isolated sequences prohibits leveraging the complementary information present in the ASR transcript and phoneme sequence; b) Due to direct conversion of words in ASR transcript to phoneme sequences, the errors from ASR transcript are propagated to the phoneme sequence which induces redundant errors that can lead to sub-optimal results.&#10;&#10;To address these drawbacks, we propose PhonemeBERT that jointly models the phoneme and ASR sequence with an added benefit of producing better results in a low-resource setup.&#10;The main contributions of this work are: &#10;\begin{enumerate}&#10; \item PhonemeBERT: A method to jointly model ASR transcripts and phoneme sequences using a BERT-based pre-training setup is proposed.&#10; \item Extensive experiments are carried out on benchmark and real life dataset to show the method's effectiveness. Results show that joint language model in PhonemeBERT can leverage phoneme sequences as complementary features, making it robust to ASR errors.&#10; \item Pre-trained PhonemeBERT can be effectively used as word-only encoder in a low-resource downstream setup where phoneme sequences are not available, still producing better results than word-only language model.&#10; \item We also release our generated datasets used in the work for research usages: &#10;&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Error Robustness, Speech Recognition, Language Modeling" />
        </attvalues>
      </node>
      <node id="2005.11640" label="2005.11640">
        <attvalues>
          <attvalue for="0" value="Jointly Encoding Word Confusion Network and Dialogue Context with BERT&#10;  for Spoken Language Understanding" />
          <attvalue for="1" value="  Spoken Language Understanding (SLU) converts hypotheses from automatic speech&#10;recognizer (ASR) into structured semantic representations. ASR recognition&#10;errors can severely degenerate the performance of the subsequent SLU module. To&#10;address this issue, word confusion networks (WCNs) have been used to encode the&#10;input for SLU, which contain richer information than 1-best or n-best&#10;hypotheses list. To further eliminate ambiguity, the last system act of&#10;dialogue context is also utilized as additional input. In this paper, a novel&#10;BERT based SLU model (WCN-BERT SLU) is proposed to encode WCNs and the dialogue&#10;context jointly. It can integrate both structural information and ASR posterior&#10;probabilities of WCNs in the BERT architecture. Experiments on DSTC2, a&#10;benchmark of SLU, show that the proposed method is effective and can outperform&#10;previous state-of-the-art models significantly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.00750" label="2002.00750">
        <attvalues>
          <attvalue for="0" value="Joint Contextual Modeling for ASR Correction and Language Understanding" />
          <attvalue for="1" value="  The quality of automatic speech recognition (ASR) is critical to Dialogue&#10;Systems as ASR errors propagate to and directly impact downstream tasks such as&#10;language understanding (LU). In this paper, we propose multi-task neural&#10;approaches to perform contextual language correction on ASR outputs jointly&#10;with LU to improve the performance of both tasks simultaneously. To measure the&#10;effectiveness of this approach we used a public benchmark, the 2nd Dialogue&#10;State Tracking (DSTC2) corpus. As a baseline approach, we trained task-specific&#10;Statistical Language Models (SLM) and fine-tuned state-of-the-art Generalized&#10;Pre-training (GPT) Language Model to re-rank the n-best ASR hypotheses,&#10;followed by a model to identify the dialog act and slots. i) We further trained&#10;ranker models using GPT and Hierarchical CNN-RNN models with discriminatory&#10;losses to detect the best output given n-best hypotheses. We extended these&#10;ranker models to first select the best ASR output and then identify the&#10;dialogue act and slots in an end to end fashion. ii) We also proposed a novel&#10;joint ASR error correction and LU model, a word confusion pointer network&#10;(WCN-Ptr) with multi-head self-attention on top, which consumes the word&#10;confusions populated from the n-best. We show that the error rates of off the&#10;shelf ASR and following LU systems can be reduced significantly by 14% relative&#10;with joint models trained using small amounts of in-domain data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.10267" label="2406.10267">
        <attvalues>
          <attvalue for="0" value="Unused information in token probability distribution of generative LLM:&#10;  improving LLM reading comprehension through calculation of expected values" />
          <attvalue for="1" value="  LLM text decoding is key component for perceived LLM quality. We demonstrate&#10;two experiments showing that decoding methods could be improved by manipulation&#10;of token probabilities. First, we test few LLM on SummEval summary scoring&#10;dataset, to measure reading comprehension. We compare scores from greedy&#10;decoding to expected values over the next token distribution. We scale logits&#10;by large temperature to increase the entropy of scores. This allows strong&#10;improvement of performance on SummEval (in terms of correlations to human&#10;judgement). We see improvement from 6-8% to 13-28% for 7B Mistral and from&#10;20%-46% to 37%-56% for Mixtral, beating GPT 4 0314 result on two metrics. Part&#10;of the gain seems related to positional bias. Secondly, we use&#10;probability-based tree sampling algorithm, to examine all most probable&#10;generations for given prompt.&#10;" />
          <attvalue for="2" value="&#10;G{enerative LLMs} are trained on large text corpora as estimators of next token probability conditional on prior text. Then sampling from such probability distribution is performed, or token with largest probability is chosen (greedy decoding). Typically, one introduces parameter $T$ – called temperature. Let $l_i$ be the logit for $i$-th token. Then token probability is as follows:&#10;\begin{equation}&#10; p_i =\frac{e^{\frac{l_i}{T}}}{\sum_{j}e^{\frac{l_j}{T}}}&#10;\end{equation}&#10;&#10;Question arises: what the T should be? Typically $T \in (0,1]$, with greedy decoding as limit in 0 and larger values corresponding to greater diversity (but also greater randomness). Research \cite{Holtzman} shows that human generated text often does not correspond to modelled highest probability. Human choice of words is not guided by greatest probability, as “humans optimize against stating the obvious”. The author of this observation, H. P. Grice in \cite{grice1975logic} gave following example: suppose that I meet a man with a gas tank asking me to sell some gas and I answer ``There's gas station over the corner''. I said only a bit of information that doesn't asnwer directly, while there's lots of implicit information not being said (gas station is open, it seels gasoline and has it available and you can go there buy some). Such concepts might be pretty foreign to LLMs: when we asked Mixtral Instruct ``How to get gasoline in Fresno'', it gave us long instruction on finding gas station on a map, choosing best gas station, operating the pump safely, paying and so on, despite the fact that most of it is irrelevant to the problem of getting gasoline in Fresno specifically.&#10;&#10;Thus, a tradeoff arises. Probability maximization with small temperature doesn't give us natural, relevant responses. Large $T$ is not ideal either, introducing more randomness, as low probability token might be either very informative or very wrong. Is however a single fixed value of $T$ sufficient, even for specific use case?&#10;&#10;We conjecture that decoding should be more dynamically controlled to more fully utilize the information in the distribution.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Decoding, Computer Science, Machine Learning, Text Generation Optimization, Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing, Reading Comprehension Evaluation" />
        </attvalues>
      </node>
      <node id="2405.00888" label="2405.00888">
        <attvalues>
          <attvalue for="0" value="DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token&#10;  Sampling" />
          <attvalue for="1" value="  Traditional language models operate autoregressively, i.e., they predict one&#10;token at a time. Rapid explosion in model sizes has resulted in high inference&#10;times. In this work, we propose DynaMo, a suite of multi-token prediction&#10;language models that reduce net inference times. Our models&#10;$\textit{dynamically}$ predict multiple tokens based on their confidence in the&#10;predicted joint probability distribution. We propose a lightweight technique to&#10;train these models, leveraging the weights of traditional autoregressive&#10;counterparts. Moreover, we propose novel ways to enhance the estimated joint&#10;probability to improve text generation quality, namely co-occurrence weighted&#10;masking and adaptive thresholding. We also propose systematic qualitative and&#10;quantitative methods to rigorously test the quality of generated text for&#10;non-autoregressive generation. One of the models in our suite, DynaMo-7.3B-T3,&#10;achieves same-quality generated text as the baseline (Pythia-6.9B) while&#10;achieving 2.57$\times$ speed-up with only 5.87% and 2.67% parameter and&#10;training time overheads, respectively.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent research has demonstrated the tremendous promise of large language models (LLMs) as competent artificial&#10;intelligence (AI) assistants~\cite{touvron2023llama2}. This has led to their rapid and widespread adoption as &#10;chatbots in diverse applications, e.g., healthcare, e-commerce, education, etc. However, the high computational &#10;requirements of LLM training and inference and the use of massive closed-source corpora have restricted their &#10;development to a few laboratories. The increasing number of open-source LLMs, including Pythia~\cite{biderman2023pythia} and LLaMA-2~\cite{touvron2023llama2}, &#10;democratizes research in natural language processing (NLP). For instance, Vicuna-13B~\cite{vicuna2023}, an &#10;instruction-finetuned LLaMA model~\cite{touvron2023llama}, has gained significant interest among researchers due to &#10;its exceptional instruction-following capabilities for its relatively compact size. Nevertheless, access and study of &#10;LLMs remain limited due to challenges involved in their efficient evaluation on resource-constrained devices.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Generation Quality, Computer Science, Linguistics, Non-Autoregressive Modeling, Artificial Intelligence, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1608.00869" label="1608.00869">
        <attvalues>
          <attvalue for="0" value="SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity" />
          <attvalue for="1" value="  Verbs play a critical role in the meaning of sentences, but these ubiquitous&#10;words have received little attention in recent distributional semantics&#10;research. We introduce SimVerb-3500, an evaluation resource that provides human&#10;ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed&#10;verb types from the USF free-association database, providing at least three&#10;examples for every VerbNet class. This broad coverage facilitates detailed&#10;analyses of how syntactic and semantic phenomena together influence human&#10;understanding of verb meaning. Further, with significantly larger development&#10;and test sets than existing benchmarks, SimVerb-3500 enables more robust&#10;evaluation of representation learning architectures and promotes the&#10;development of methods tailored to verbs. We hope that SimVerb-3500 will enable&#10;a richer understanding of the diversity and complexity of verb semantics and&#10;guide the development of systems that can effectively represent and interpret&#10;this meaning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10650" label="2212.10650">
        <attvalues>
          <attvalue for="0" value="KronA: Parameter Efficient Tuning with Kronecker Adapter" />
          <attvalue for="1" value="  Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task&#10;has been a well-known paradigm in Natural Language Processing. However, with&#10;the ever-growing size of PLMs, training the entire model on several downstream&#10;tasks becomes very expensive and resource-hungry. Recently, different Parameter&#10;Efficient Tuning (PET) techniques are proposed to improve the efficiency of&#10;fine-tuning PLMs. One popular category of PET methods is the low-rank&#10;adaptation methods which insert learnable truncated SVD modules into the&#10;original model either sequentially or in parallel. However, low-rank&#10;decomposition suffers from limited representation power. In this work, we&#10;address this problem using the Kronecker product instead of the low-rank&#10;representation. We introduce KronA, a Kronecker product-based adapter module&#10;for efficient fine-tuning of Transformer-based PLMs. We apply the proposed&#10;methods for fine-tuning T5 on the GLUE benchmark to show that incorporating the&#10;Kronecker-based modules can outperform state-of-the-art PET methods.&#10;" />
          <attvalue for="2" value="&#10;Large PLMs are used as a backbone model in a variety of NLP tasks to achieve state-of-the-art results \cite{devlin-etal-2019-bert,radford2019language}. These large pre-trained models are adapted to the downstream applications either via in-context learning or fine-tuning of the model parameters. In-context learning imposes substantial&#10;memory and computational overhead during inference as all the training examples have to be processed for each sample \cite{liu2022few}. On the other hand, full Fine-Tuning (FT) the entire model provides both less inference latency and improved accuracy. However, as these models become larger, full fine-tuning of their parameters becomes more challenging. Additionally, one has to store an entire model checkpoint for each downstream application, which makes deployment and switching between different tasks extremely inefficient. &#10;&#10;To address these challenges, several works have proposed to insert a small number of trainable parameters while freezing most (or even all) of the pre-trained model parameters. &#10;This significantly reduces the memory and computation requirements for fine-tuning. Furthermore, instead of storing one copy of the entire model, a small set of tuned parameters can be stored for each task. We refer to these methods as PET methods. &#10;&#10;Among the PET methods, soft prompts \cite{li-liang-2021-prefix,lester2021power} prepend trainable parameters to the input of the layers. The increase in length of the embedding layers leads to a significant computation overhead during the inference. &#10;&#10;In another category of the PET methods, adapter modules are inserted \cite{pmlr-v97-houlsby19a,karimi2021compacter,he2022towards} into the Transformer. Adapters are low-rank modules that are composed of an up projection followed by a down projection. One limitation of these approaches is that they increase the computational overhead and the latency during the inference which makes them inefficient for latency-critical scenarios. &#10;&#10;Therefore, Low Rank Adaption (LoRA) \cite{hu2021lora} was developed, which also uses extra low-rank modules as the trainable parameters. However, once fine-tuned, the task-specific parameters can be merged with the original pre-trained model weights, making the latency and energy requirements for inference, intact. Despite of fast inference, usually LoRA suffers from an accuracy drop compared to full fine-tuning. This is because of the strong assumption imposed by its low-rank structure for task-specific updates. &#10;&#10;Kronecker product decomposition is another factorization method that does not rely on the low-rank assumption. This powerful decomposition method, when used for model compression, has proven to outperform low-rank factorization methods \cite{thakker2019pushing,hameed2021convolutional}. It has also been successfully used for the compression of Transformer-based language models \cite{tahaei2021kroneckerbert,edalati2021kronecker}.&#10;&#10;Inspired by the ubiquitous success of Kronecker decomposition, in this work we replace low-rank decomposition in LoRA with Kronecker product decomposition to develop the Kronecker Adapter (KronA). We show that this simple modification can improve the accuracy without increasing the inference latency. Also for applications where an increase in latency is tolerable, we propose to use $KronA^B$. This module is a version of KronA developed to be utilized in parallel to Feed-Forward Network (FFN) modules and achieves significant improvements over full fine-tuning on the General Language Understanding Evaluation (GLUE) benchmark \cite{wang-etal-2018-glue}. In addition, when a proposed learnable residual connection is added to $KronA^B$, $KronA^B_res$ is developed to achieve even better results.&#10;&#10;We evaluated our methods on the GLUE benchmark \cite{wang-etal-2018-glue} to study the impact of Kronecker product on the performance. To summarize, our contributions are:&#10;\begin{itemize}&#10; \item Proposing KronA, a \underline{Kron}ecker \underline{A}dapter module that can be inserted in parallel to the weight matrices and is suitable for latency-critical scenarios.&#10; \item Using the KronA module in parallel to the FFN module ($KronA^B$) along with a learnable residual connection ($KronA^B_res$) to further improve the accuracy at the cost of increased inference latency.&#10; \item Providing evaluation of the methods in comparison to the state-of-the-art in terms of the GLUE score, training time and inference latency.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Tuning, Computer Science, Linguistics, Parameter Efficient Methods, Mathematics, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="1910.02558" label="1910.02558">
        <attvalues>
          <attvalue for="0" value="Pushing the limits of RNN Compression" />
          <attvalue for="1" value="  Recurrent Neural Networks (RNN) can be difficult to deploy on resource&#10;constrained devices due to their size. As a result, there is a need for&#10;compression techniques that can significantly compress RNNs without negatively&#10;impacting task accuracy. This paper introduces a method to compress RNNs for&#10;resource constrained environments using Kronecker product (KP). KPs can&#10;compress RNN layers by 16-38x with minimal accuracy loss. We show that KP can&#10;beat the task accuracy achieved by other state-of-the-art compression&#10;techniques (pruning and low-rank matrix factorization) across 4 benchmarks&#10;spanning 3 different applications, while simultaneously improving inference&#10;run-time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.14710" label="2109.14710">
        <attvalues>
          <attvalue for="0" value="Convolutional Neural Network Compression through Generalized Kronecker&#10;  Product Decomposition" />
          <attvalue for="1" value="  Modern Convolutional Neural Network (CNN) architectures, despite their&#10;superiority in solving various problems, are generally too large to be deployed&#10;on resource constrained edge devices. In this paper, we reduce memory usage and&#10;floating-point operations required by convolutional layers in CNNs. We compress&#10;these layers by generalizing the Kronecker Product Decomposition to apply to&#10;multidimensional tensors, leading to the Generalized Kronecker Product&#10;Decomposition (GKPD). Our approach yields a plug-and-play module that can be&#10;used as a drop-in replacement for any convolutional layer. Experimental results&#10;for image classification on CIFAR-10 and ImageNet datasets using ResNet,&#10;MobileNetv2 and SeNet architectures substantiate the effectiveness of our&#10;proposed approach. We find that GKPD outperforms state-of-the-art decomposition&#10;methods including Tensor-Train and Tensor-Ring as well as other relevant&#10;compression methods such as pruning and knowledge distillation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.06243" label="2109.06243">
        <attvalues>
          <attvalue for="0" value="KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language&#10;  Models via Knowledge Distillation" />
          <attvalue for="1" value="  The development of over-parameterized pre-trained language models has made a&#10;significant contribution toward the success of natural language processing.&#10;While over-parameterization of these models is the key to their generalization&#10;power, it makes them unsuitable for deployment on low-capacity devices. We push&#10;the limits of state-of-the-art Transformer-based pre-trained language model&#10;compression using Kronecker decomposition. We use this decomposition for&#10;compression of the embedding layer, all linear mappings in the multi-head&#10;attention, and the feed-forward network modules in the Transformer layer. We&#10;perform intermediate-layer knowledge distillation using the uncompressed model&#10;as the teacher to improve the performance of the compressed model. We present&#10;our KroneckerBERT, a compressed version of the BERT_BASE model obtained using&#10;this framework. We evaluate the performance of KroneckerBERT on well-known NLP&#10;benchmarks and show that for a high compression factor of 19 (5% of the size of&#10;the BERT_BASE model), our KroneckerBERT outperforms state-of-the-art&#10;compression methods on the GLUE. Our experiments indicate that the proposed&#10;model has promising out-of-distribution robustness and is superior to the&#10;state-of-the-art compression methods on SQuAD.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08152" label="2110.08152">
        <attvalues>
          <attvalue for="0" value="Kronecker Decomposition for GPT Compression" />
          <attvalue for="1" value="  GPT is an auto-regressive Transformer-based pre-trained language model which&#10;has attracted a lot of attention in the natural language processing (NLP)&#10;domain due to its state-of-the-art performance in several downstream tasks. The&#10;success of GPT is mostly attributed to its pre-training on huge amount of data&#10;and its large number of parameters (from ~100M to billions of parameters).&#10;Despite the superior performance of GPT (especially in few-shot or zero-shot&#10;setup), this overparameterized nature of GPT can be very prohibitive for&#10;deploying this model on devices with limited computational power or memory.&#10;This problem can be mitigated using model compression techniques; however,&#10;compressing GPT models has not been investigated much in the literature. In&#10;this work, we use Kronecker decomposition to compress the linear mappings of&#10;the GPT-22 model. Our Kronecker GPT-2 model (KnGPT2) is initialized based on&#10;the Kronecker decomposed version of the GPT-2 model and then is undergone a&#10;very light pre-training on only a small portion of the training data with&#10;intermediate layer knowledge distillation (ILKD). Finally, our KnGPT2 is&#10;fine-tuned on down-stream tasks using ILKD as well. We evaluate our model on&#10;both language modeling and General Language Understanding Evaluation benchmark&#10;tasks and show that with more efficient pre-training and similar number of&#10;parameters, our KnGPT2 outperforms the existing DistilGPT2 model significantly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00187" label="2005.00187">
        <attvalues>
          <attvalue for="0" value="Cross-Linguistic Syntactic Evaluation of Word Prediction Models" />
          <attvalue for="1" value="  A range of studies have concluded that neural word prediction models can&#10;distinguish grammatical from ungrammatical sentences with high accuracy.&#10;However, these studies are based primarily on monolingual evidence from&#10;English. To investigate how these models' ability to learn syntax varies by&#10;language, we introduce CLAMS (Cross-Linguistic Assessment of Models on Syntax),&#10;a syntactic evaluation suite for monolingual and multilingual models. CLAMS&#10;includes subject-verb agreement challenge sets for English, French, German,&#10;Hebrew and Russian, generated from grammars we develop. We use CLAMS to&#10;evaluate LSTM language models as well as monolingual and multilingual BERT.&#10;Across languages, monolingual LSTMs achieved high accuracy on dependencies&#10;without attractors, and generally poor accuracy on agreement across object&#10;relative clauses. On other constructions, agreement accuracy was generally&#10;higher in languages with richer morphology. Multilingual models generally&#10;underperformed monolingual models. Multilingual BERT showed high syntactic&#10;accuracy on English, but noticeable deficiencies in other languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural networks can be trained to predict words from their context with much greater accuracy than the architectures used for this purpose in the past. This has been shown to be the case for both recurrent neural networks \cite{rnn_lm,lstm_lm,jozefowicz2016exploring} and non-recurrent attention-based models \cite{bert,gpt2}.\blfootnote{$^\dagger$ Work done while at Johns Hopkins University. Now in the University of British Columbia's Linguistics Department.}&#10;&#10;To gain a better understanding of these models' successes and failures, in particular in the domain of syntax, &#10;proposals have been made for testing the models on subsets of the test corpus where successful word prediction crucially depends on a correct analysis of the structure of the sentence \cite{dupouxlinzen16}. A paradigmatic example is subject-verb agreement. In many languages, including English, the verb often needs to agree in number (here, singular or plural) with the subject (asterisks represent ungrammatical word predictions):&#10;&#10;\ex.The key to the cabinets \underline{is}/*\underline{are} next to the coins.&#10;&#10;To correctly predict the form of the verb (underlined), the model needs to determine that the head of the subject of the sentence---an abstract, structurally defined notion---is the word key rather than cabinets or coins. &#10;&#10;The approach of sampling challenging sentences from a test corpus has its limitations. Examples of relevant constructions may be difficult to find in the corpus, and naturally occurring sentences often contain statistical cues (confounds) that make it possible for the model to predict the correct form of the verb without an adequate syntactic analysis \cite{gulordavacolorless18}. To address these limitations, a growing number of studies have used constructed materials, which improve experimental control and coverage of syntactic constructions \cite{marvinlinzen18,wilcox18gap,futrell2019neural,warstadt2019blimp}.&#10;&#10;Existing experimentally controlled data sets---in particular, those targeting subject-verb agreement---have largely been restricted to English. As such, we have a limited understanding of the effect of the cross-linguistic variability in neural networks' syntactic prediction abilities. In this paper, we introduce the Cross-Linguistic Assessment of Models on Syntax (CLAMS) data set, which extends the subject-verb agreement component of the \cite{marvinlinzen18} challenge set to French, German, Hebrew and Russian. By focusing on a single linguistic phenomenon in related languages, we can directly compare the models' performance across languages. We see the present effort as providing a core data set that can be expanded in future work to improve coverage to other languages and syntactic constructions. To this end, we release the code for a simple grammar engineering framework that facilitates the creation and generation of syntactic evaluation sets.&#10;&#10;We use CLAMS to test two hypotheses. First, we hypothesize that a multilingual model would show transfer across languages with similar syntactic constructions, which would lead to improved syntactic performance compared to monolingual models. In experiments on LSTM language models (LMs), we do not find support for this hypothesis; contrarily, accuracy was lower for the multilingual model than the monolingual ones. Second, we hypothesize that language models would be better able to learn hierarchical syntactic generalizations in morphologically complex languages (which provide frequent overt cues to syntactic structure) than in morphologically simpler languages \cite{gulordavacolorless18,Lorimor2008,mccoy18pos}. We test this using LSTM LMs we train, and find moderate support for this hypothesis. &#10;&#10;In addition to our analysis of LSTM LMs, we demonstrate the utility of CLAMS for testing pre-trained word prediction models. We evaluate multilingual BERT \cite{bert}, a bidirectional Transformer model trained on a multilingual corpus, and find that this model performs well on English, has mixed syntactic abilities in French and German, and performs poorly on Hebrew and Russian. Its syntactic performance in English was somewhat worse than that of monolingual English BERT, again suggesting that interference between languages offsets any potential syntactic transfer.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Neural Word Prediction, Cross-Linguistic Syntax" />
        </attvalues>
      </node>
      <node id="1602.02410" label="1602.02410">
        <attvalues>
          <attvalue for="0" value="Exploring the Limits of Language Modeling" />
          <attvalue for="1" value="  In this work we explore recent advances in Recurrent Neural Networks for&#10;large scale Language Modeling, a task central to language understanding. We&#10;extend current models to deal with two key challenges present in this task:&#10;corpora and vocabulary sizes, and complex, long term structure of language. We&#10;perform an exhaustive study on techniques such as character Convolutional&#10;Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark.&#10;Our best single model significantly improves state-of-the-art perplexity from&#10;51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20),&#10;while an ensemble of models sets a new record by improving perplexity from 41.0&#10;down to 23.7. We also release these models for the NLP and ML community to&#10;study and improve upon.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03720" label="2004.03720">
        <attvalues>
          <attvalue for="0" value="Byte Pair Encoding is Suboptimal for Language Model Pretraining" />
          <attvalue for="1" value="  The success of pretrained transformer language models (LMs) in natural&#10;language processing has led to a wide range of pretraining setups. In&#10;particular, these models employ a variety of subword tokenization methods, most&#10;notably byte-pair encoding (BPE) (Sennrich et al., 2016; Gage, 1994), the&#10;WordPiece method (Schuster and Nakajima, 2012), and unigram language modeling&#10;(Kudo, 2018), to segment text. However, to the best of our knowledge, the&#10;literature does not contain a direct evaluation of the impact of tokenization&#10;on language model pretraining. We analyze differences between BPE and unigram&#10;LM tokenization, finding that the latter method recovers subword units that&#10;align more closely with morphology and avoids problems stemming from BPE's&#10;greedy construction procedure. We then compare the fine-tuned task performance&#10;of identical transformer masked language models pretrained with these&#10;tokenizations. Across downstream tasks and two languages (English and&#10;Japanese), we find that the unigram LM tokenization method matches or&#10;outperforms BPE. We hope that developers of future pretrained LMs will consider&#10;adopting the unigram LM method over the more prevalent BPE.&#10;" />
          <attvalue for="2" value="&#10;Large transformers \cite{vaswani2017attention} pretrained with variants of a language modeling objective, such as BERT \cite{devlin2019bert}, have proven their effectiveness at flexibly transferring to a variety of domains and tasks. One design decision that makes them particularly adaptable is their graceful handling of the open vocabulary problem through subword tokenization. Subword tokenization, popularized in the neural machine translation literature \cite{sennrich-etal-2016-neural, vaswani2017attention, wu2016google}, produces tokens at multiple levels of granularity, from individual characters to full words. As a result, rare words are broken down into a collection of subword units, bottoming out in characters in the worst case.&#10;&#10;Critically, a pretrained language model's subword vocabulary cannot be altered: any downstream application of these models must tokenize input or generate output using the original subword vocabulary, making the choice of tokenization a particularly significant decision.&#10;&#10;A variety of subword tokenization methods have seen use in pretrained language models. BERT uses the WordPiece method \cite{Schuster2012JapaneseAK}, a language-modeling based variant of BPE;&#10;T5 \cite{raffel2019exploring} uses character-level BPE;&#10;GPT2 \cite{radford2019language} and RoBERTa \cite{liu2019roberta} use BPE over raw bytes instead of unicode characters;&#10;XLNet \cite{yang2019xlnet} and ALBERT \cite{lan2019albert} use the SentencePiece library \cite{kudo-richardson-2018-sentencepiece} which implements both BPE and unigram language model tokenization, but in both cases fail to clarify which of these methods they chose. The effects of tokenization are not examined in a reported experiment in any of the above works except \cite{liu2019roberta}, who note that WordPiece gave a small advantage over BPE in their preliminary investigation. In the machine translation literature, \cite{kudo-2018-subword} introduced the unigram language model tokenization method in the context of machine translation and found it comparable in performance to BPE. \cite{domingo2018much} performed further experiments to investigate the effects of tokenization on neural machine translation, but used a shared BPE vocabulary across all experiments. \cite{galle-2019-investigating} examined algorithms in the BPE family, but did not compare to unigram language modeling.&#10;&#10;In this work, we characterize the space of proposed subword tokenization algorithms and analyze the differences between the two methods with publicly available implementations: BPE (merging tokens based on bigram frequency) and unigram language modeling (pruning tokens based on unigram LM perplexity). While the vocabularies resulting from these schemes are heavily overlapping, we compare each method to reference morphological segmentations and find that the unigram LM method produces tokens better aligned with morphology. To understand whether this more natural tokenization leads to improved performance, we pretrain separate language models using the RoBERTa objective \cite{liu2019roberta} with each tokenization for both English and Japanese, two typologically distant languages. On downstream tasks, we find a performance gap across tasks and languages, with the unigram LM method providing an improvement over BPE of up to 10\% in our Japanese QA experiments, indicating the benefits of adopting this technique in the context of language model pretraining.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Tokenization Methods, Artificial Intelligence, Language Model Pretraining, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2312.07886" label="2312.07886">
        <attvalues>
          <attvalue for="0" value="Modality Plug-and-Play: Elastic Modality Adaptation in Multimodal LLMs&#10;  for Embodied AI" />
          <attvalue for="1" value="  Large Language Models (LLMs) are capable of reasoning over diverse input data&#10;modalities through pre-trained encoders. However, the growing diversity of&#10;input data modalities prevents incorporating all modalities into LLMs,&#10;especially when LLMs are deployed on resource-constrained edge devices for&#10;embodied AI applications. Instead, a better option is to adaptively involve&#10;only the useful modalities at runtime, depending on the current environmental&#10;contexts and task requirements. For such modality adaptation, existing work&#10;adopts fixed connections between encoders and the LLM's input layer, leading to&#10;high training cost at runtime and ineffective cross-modal interaction. In this&#10;paper, we address these limitations by presenting mPnP-LLM, a new technique&#10;that allows fully elastic, automated and prompt runtime modality adaptation, by&#10;connecting unimodal encoders to a flexible set of last LLM blocks and making&#10;such latent connections fully trainable at runtime. Experiments over the&#10;nuScenes-QA dataset show that mPnP-LLM can achieve up to 3.7x FLOPs reduction&#10;and 30% GPU memory usage reduction, while retaining on-par accuracy with the&#10;existing schemes. Under the same compute budget, mPnP-LLM improves the task&#10;accuracy by up to 4% compared to the best existing scheme.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs) can do reasoning over diverse input data modalities [\cite{alayrac2022flamingo, driess2023palm, brohan2023rt, moon2023anymal}], besides the natural language domain. Such multimodal reasoning relies on pre-trained encoders for different input modalities, such as RGB frames [\cite{he2022masked}], LiDAR point clouds [\cite{hess2023masked}] and acoustic signals [\cite{li2023ti}], to extract task-relevant features. By incorporating encoders from multiple input modalities, LLMs can fully perceive the physical world and enable intelligent embodied agents, such as autonomous vehicles and robots, that adapt to environmental contexts and task needs [\cite{huang2022inner, driess2023palm, song2023llm}].&#10;&#10;One major challenge of such multimodal reasoning is the growing diversity of input data modalities [\cite{kong2022m3track, xu2022mask, jiao2023bioscatter, fan2023apg}]. Since today's transformer-based encoders [\cite{vaswani2017attention,wu2021autoformer}] are usually large in size, incorporating all modalities to LLMs is computationally expensive or even infeasible, especially on resource-constrained edge devices used in embodied AI applications. Instead, we envision that the usefulness of different input modalities, even for the same embodied AI task, could greatly vary when the environmental contexts change. Hence, a better option is to adaptively involve only the useful modalities at runtime, for the minimum on-device computing cost. An example of such runtime modality adaptation for multimodal question answering (QA) tasks in autonomous driving is shown in Figure \ref{fig:modality_adaptation}. &#10;&#10;An intuitive approach to such modality adaptation is to jointly train the encoders of all involved modalities with LLM to align every modality with the natural language domain [\cite{brohan2023rt, driess2023palm, wu2023next}], but is too expensive for runtime. Instead, we can freeze both encoders and LLM at runtime, but only train the inserted projection modules in between. As shown in Figure \ref{fig:snapshot} - Top, existing work connects encoders to LLM's input layer through a trainable projector [\cite{li2023blip, zhu2023minigpt}], and then applies parameter-efficient LLM fine-tuning [\cite{hu2021lora, sung2022vl, liang2022modular}] to improve accuracy. However, they still require backpropagating activation gradients throughout the entire LLM and hence incur large training costs. &#10;&#10;Adopting more lightweight projectors or reducing the amount of trainable parameters can only reduce the FLOPs of computing weight updates in training, but do not change the cost of gradient backpropagation. In addition, this connection also requires the projected multimodal features to be all aligned with the LLM input layer's text embedding, but such alignment can be inefficient in bridging the semantic gap across different modalities, because the text representation in early LLM blocks can be too superficial to match the details in input modalities [\cite{snoek2005early}]. Although some recent work [\cite{shukor2023ep}] explored connections to LLM's intermediate layers, such connections are arbitrarily decided and always fixed at runtime, and hence lack runtime adaptability. Improperly inserting multimodal information to intermediate LLM blocks could also interfere with the LLM's reasoning and affect efficient cross-modal interaction.&#10;&#10;To address these limitations, in this paper we present Modality Plug-and-Play in multimodal LLMs (mPnP-LLM), a new technique for elastic, automated and prompt runtime modality adaptation in multimodal LLMs, by connecting unimodal encoders to a flexible set of last LLM blocks and making such latent connections fully trainable at runtime. As shown in Figure \ref{fig:snapshot} - Bottom, we can adaptively adjust the amount of LLM blocks being connected for different tradeoffs between accuracy and runtime training cost. We can also optimize the efficiency of cross-modal interaction and hence improve accuracy, by controlling the amount of information being injected in each connection with a trainable weighting module. Our design focuses on decoder-only LLM, which is the dominant LLM architecture due to stronger generative power [\cite{chowdhery2022palm}] and has been widely adopted by most existing LLMs, ranging from BLOOM-1.1B [\cite{muennighoff2022crosslingual}] and OPT-1.3B [\cite{zhang2022opt}] to GPT3-175B [\cite{brown2020language}]. &#10;&#10;Being different from the existing plug-and-play approaches [\cite{de2017modulating, tan2019lxmert, brohan2022rt, shukor2023ep}] that require re-implementation of LLM's source codes, mPnP-LLM inserts the projected multimodal tokens as new key-value pairs into the multi-head attention (MHA) module of LLM block. Since such interfacing is well supported in popular LLM frameworks (e.g., HuggingFace Transformers [\cite{wolf2019huggingface}]), we can avoid any manual programming and reconfiguration efforts at runtime.&#10;&#10;We implemented and evaluated mPnP-LLM with two open-sourced LLMs, namely OPT [\cite{zhang2022opt}] and BLOOMZ [\cite{muennighoff2022crosslingual}], on the nuScene-QA dataset [\cite{qian2023nuscenes}] for multimodal QA task in autonomous driving. When adapting between modalities at runtime, mPnP-LLM achieves up to 3.7$\times$ FLOPs reduction and 30\% GPU memory usage reduction, while retaining on-par accuracy with the existing schemes. Such speedup enables modality adaptation within a few minutes on a weak edge device (NVidia Jetson AGX Orin). Under the same compute budget, mPnP-LLM improves the task accuracy by up to 4\%, compared to the best existing scheme.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Large Language Models, Cognitive Science, Edge AI Optimization, Modality Adaptation, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2207.00531" label="2207.00531">
        <attvalues>
          <attvalue for="0" value="Masked Autoencoder for Self-Supervised Pre-training on Lidar Point&#10;  Clouds" />
          <attvalue for="1" value="  Masked autoencoding has become a successful pretraining paradigm for&#10;Transformer models for text, images, and, recently, point clouds. Raw&#10;automotive datasets are suitable candidates for self-supervised pre-training as&#10;they generally are cheap to collect compared to annotations for tasks like 3D&#10;object detection (OD). However, the development of masked autoencoders for&#10;point clouds has focused solely on synthetic and indoor data. Consequently,&#10;existing methods have tailored their representations and models toward small&#10;and dense point clouds with homogeneous point densities. In this work, we study&#10;masked autoencoding for point clouds in an automotive setting, which are sparse&#10;and for which the point density can vary drastically among objects in the same&#10;scene. To this end, we propose Voxel-MAE, a simple masked autoencoding&#10;pre-training scheme designed for voxel representations. We pre-train the&#10;backbone of a Transformer-based 3D object detector to reconstruct masked voxels&#10;and to distinguish between empty and non-empty voxels. Our method improves the&#10;3D OD performance by 1.75 mAP points and 1.05 NDS on the challenging nuScenes&#10;dataset. Further, we show that by pre-training with Voxel-MAE, we require only&#10;40% of the annotated data to outperform a randomly initialized equivalent. Code&#10;available at https://github.com/georghess/voxel-mae&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.13008" label="2106.13008">
        <attvalues>
          <attvalue for="0" value="Autoformer: Decomposition Transformers with Auto-Correlation for&#10;  Long-Term Series Forecasting" />
          <attvalue for="1" value="  Extending the forecasting time is a critical demand for real applications,&#10;such as extreme weather early warning and long-term energy consumption&#10;planning. This paper studies the long-term forecasting problem of time series.&#10;Prior Transformer-based models adopt various self-attention mechanisms to&#10;discover the long-range dependencies. However, intricate temporal patterns of&#10;the long-term future prohibit the model from finding reliable dependencies.&#10;Also, Transformers have to adopt the sparse versions of point-wise&#10;self-attentions for long series efficiency, resulting in the information&#10;utilization bottleneck. Going beyond Transformers, we design Autoformer as a&#10;novel decomposition architecture with an Auto-Correlation mechanism. We break&#10;with the pre-processing convention of series decomposition and renovate it as a&#10;basic inner block of deep models. This design empowers Autoformer with&#10;progressive decomposition capacities for complex time series. Further, inspired&#10;by the stochastic process theory, we design the Auto-Correlation mechanism&#10;based on the series periodicity, which conducts the dependencies discovery and&#10;representation aggregation at the sub-series level. Auto-Correlation&#10;outperforms self-attention in both efficiency and accuracy. In long-term&#10;forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative&#10;improvement on six benchmarks, covering five practical applications: energy,&#10;traffic, economics, weather and disease. Code is available at this repository:&#10;\url{https://github.com/thuml/Autoformer}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.15818" label="2307.15818">
        <attvalues>
          <attvalue for="0" value="RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic&#10;  Control" />
          <attvalue for="1" value="  We study how vision-language models trained on Internet-scale data can be&#10;incorporated directly into end-to-end robotic control to boost generalization&#10;and enable emergent semantic reasoning. Our goal is to enable a single&#10;end-to-end trained model to both learn to map robot observations to actions and&#10;enjoy the benefits of large-scale pretraining on language and vision-language&#10;data from the web. To this end, we propose to co-fine-tune state-of-the-art&#10;vision-language models on both robotic trajectory data and Internet-scale&#10;vision-language tasks, such as visual question answering. In contrast to other&#10;approaches, we propose a simple, general recipe to achieve this goal: in order&#10;to fit both natural language responses and robotic actions into the same&#10;format, we express the actions as text tokens and incorporate them directly&#10;into the training set of the model in the same way as natural language tokens.&#10;We refer to such category of models as vision-language-action models (VLA) and&#10;instantiate an example of such a model, which we call RT-2. Our extensive&#10;evaluation (6k evaluation trials) shows that our approach leads to performant&#10;robotic policies and enables RT-2 to obtain a range of emergent capabilities&#10;from Internet-scale training. This includes significantly improved&#10;generalization to novel objects, the ability to interpret commands not present&#10;in the robot training data (such as placing an object onto a particular number&#10;or icon), and the ability to perform rudimentary reasoning in response to user&#10;commands (such as picking up the smallest or largest object, or the one closest&#10;to another object). We further show that incorporating chain of thought&#10;reasoning allows RT-2 to perform multi-stage semantic reasoning, for example&#10;figuring out which object to pick up for use as an improvised hammer (a rock),&#10;or which type of drink is best suited for someone who is tired (an energy&#10;drink).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.11403" label="2303.11403">
        <attvalues>
          <attvalue for="0" value="eP-ALM: Efficient Perceptual Augmentation of Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have so far impressed the world, with&#10;unprecedented capabilities that emerge in models at large scales. On the vision&#10;side, transformer models (i.e., ViT) are following the same trend, achieving&#10;the best performance on challenging benchmarks. With the abundance of such&#10;unimodal models, a natural question arises; do we need also to follow this&#10;trend to tackle multimodal tasks? In this work, we propose to rather direct&#10;effort to efficient adaptations of existing models, and propose to augment&#10;Language Models with perception. Existing approaches for adapting pretrained&#10;models for vision-language tasks still rely on several key components that&#10;hinder their efficiency. In particular, they still train a large number of&#10;parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP)&#10;trained on huge image-text datasets, and add significant inference overhead. In&#10;addition, most of these approaches have focused on Zero-Shot and In Context&#10;Learning, with little to no effort on direct finetuning. We investigate the&#10;minimal computational effort needed to adapt unimodal models for multimodal&#10;tasks and propose a new challenging setup, alongside different approaches, that&#10;efficiently adapts unimodal pretrained models. We show that by freezing more&#10;than 99% of total parameters, training only one linear projection layer, and&#10;prepending only one trainable token, our approach (dubbed eP-ALM) significantly&#10;outperforms other baselines on VQA and Captioning across Image, Video, and&#10;Audio modalities, following the proposed setup. The code is available here:&#10;https://github.com/mshukor/eP-ALM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.00683" label="1707.00683">
        <attvalues>
          <attvalue for="0" value="Modulating early visual processing by language" />
          <attvalue for="1" value="  It is commonly assumed that language refers to high-level visual concepts&#10;while leaving low-level visual processing unaffected. This view dominates the&#10;current literature in computational models for language-vision tasks, where&#10;visual and linguistic input are mostly processed independently before being&#10;fused into a single representation. In this paper, we deviate from this classic&#10;pipeline and propose to modulate the \emph{entire visual processing} by&#10;linguistic input. Specifically, we condition the batch normalization parameters&#10;of a pretrained residual network (ResNet) on a language embedding. This&#10;approach, which we call MOdulated RESnet (\MRN), significantly improves strong&#10;baselines on two visual question answering tasks. Our ablation study shows that&#10;modulating from the early stages of the visual processing is beneficial.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14836" label="2305.14836">
        <attvalues>
          <attvalue for="0" value="NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for&#10;  Autonomous Driving Scenario" />
          <attvalue for="1" value="  We introduce a novel visual question answering (VQA) task in the context of&#10;autonomous driving, aiming to answer natural language questions based on&#10;street-view clues. Compared to traditional VQA tasks, VQA in autonomous driving&#10;scenario presents more challenges. Firstly, the raw visual data are&#10;multi-modal, including images and point clouds captured by camera and LiDAR,&#10;respectively. Secondly, the data are multi-frame due to the continuous,&#10;real-time acquisition. Thirdly, the outdoor scenes exhibit both moving&#10;foreground and static background. Existing VQA benchmarks fail to adequately&#10;address these complexities. To bridge this gap, we propose NuScenes-QA, the&#10;first benchmark for VQA in the autonomous driving scenario, encompassing 34K&#10;visual scenes and 460K question-answer pairs. Specifically, we leverage&#10;existing 3D detection annotations to generate scene graphs and design question&#10;templates manually. Subsequently, the question-answer pairs are generated&#10;programmatically based on these templates. Comprehensive statistics prove that&#10;our NuScenes-QA is a balanced large-scale benchmark with diverse question&#10;formats. Built upon it, we develop a series of baselines that employ advanced&#10;3D detection and VQA techniques. Our extensive experiments highlight the&#10;challenges posed by this new task. Codes and dataset are available at&#10;https://github.com/qiantianwen/NuScenes-QA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.00173" label="2304.00173">
        <attvalues>
          <attvalue for="0" value="Lego-Features: Exporting modular encoder features for streaming and&#10;  deliberation ASR" />
          <attvalue for="1" value="  In end-to-end (E2E) speech recognition models, a representational&#10;tight-coupling inevitably emerges between the encoder and the decoder. We build&#10;upon recent work that has begun to explore building encoders with modular&#10;encoded representations, such that encoders and decoders from different models&#10;can be stitched together in a zero-shot manner without further fine-tuning.&#10;While previous research only addresses full-context speech models, we explore&#10;the problem in a streaming setting as well. Our framework builds on top of&#10;existing encoded representations, converting them to modular features, dubbed&#10;as Lego-Features, without modifying the pre-trained model. The features remain&#10;interchangeable when the model is retrained with distinct initializations.&#10;Though sparse, we show that the Lego-Features are powerful when tested with&#10;RNN-T or LAS decoders, maintaining high-quality downstream performance. They&#10;are also rich enough to represent the first-pass prediction during two-pass&#10;deliberation. In this scenario, they outperform the N-best hypotheses, since&#10;they do not need to be supplemented with acoustic features to deliver the best&#10;results. Moreover, generating the Lego-Features does not require beam search or&#10;auto-regressive computation. Overall, they present a modular, powerful and&#10;cheap alternative to the standard encoder output, as well as the N-best&#10;hypotheses.&#10;" />
          <attvalue for="2" value=" &#10;&#10;E2E speech recognition models, which combine acoustic, pronunciation and language models from conventional systems \cite{Golan16} into one neural network, have become widely used, especially for on-device applications \cite{bo21system,Ryan19,CC18,KimHoriWatanabe17,JinyuLi2019,Zeyer2020}. Since they are much smaller than conventional models, and their inference speed is often much faster \cite{bo21system,Ryan19,sainath2021cascadedlm,sainath2020streaming}, they work well for various streaming applications. They typically use an encoder-decoder architecture \cite{RohitSeq17}. Like most deep neural networks, the whole architecture is usually trained end to end. The encoder implicitly learns to serve the subsequent decoder layers, and thus conversely, the decoder is thoroughly oriented towards inputs coming from the specific encoder that it has been trained with. Therefore, encoders and decoders from different models or training runs, are generally not interchangeable without further E2E training.&#10;&#10;This tight coupling between both components stands in the way of a flexible, modular architecture. Speech encoders that have been trained on high-resource ASR data can serve as foundation models for other tasks like sentiment analysis \cite{lu2020speech} or low-resource translation \cite{bansal2018pre}, to name a few. However, this presents a challenge if a shared encoder representation is used for multiple downstream tasks: When the ASR encoder is retrained, all downstream models must be retrained as well. Hence, it would be more practical if each component can be developed and updated independently. To that end, we present a method for building modular speech encoder features, where different versions of the encoder can be plugged into the decoder in a zero-shot stitching manner without fine-tuning.&#10;&#10;Our method works by building on top of an existing base encoder, which is kept frozen. We adapt the Beam-Convolution scheme described in \cite{dalmia2019enforcing} to train streaming modular encoded representations, which we call Lego-Features. To produce them, the original (fixed) continuous encoded features pass through a few extra trainable ``Exporter'' layers, then through a CTC decoder, which is trained with an auxiliary CTC loss. Lego-Features are defined as the sorted top $K$ CTC logit indices at every frame, see \ref{fig:lego_encoder}. The logits operate over a discrete space (here: wordpiece vocabulary) and are grounded in the transcript text, which is why they tend to be modular. Overall, the traditional encoder features are forced through a tight discretizing bottleneck, which protects downstream models from coupling themselves to fine details in the encoded representation. Downstream consumers of Lego-Features need to first re-embed them, since they come in as sparse indices.&#10;&#10;\cite{dalmia2019enforcing, dalmia2022legonn} have shown how this tight bottleneck still produces a powerful representation which is sufficiently informative for downstream ASR decoders. They also perform a ``modularity test'': The downstream decoder is kept constant, but gets input with a new version of the encoded representation, which is obtained by retraining the encoder from scratch using a different initialization. The switch is done in a zero-shot manner without any extra fine-tuning. Traditional continuous encoded features categorically fail the modularity test, bringing the downstream performance to nearly 100\% WER, which is what motivates this new type of encoded representation. We build on the original works with a few novel contributions:&#10;&#10;\begin{enumerate}[align=right,itemindent=2em,labelsep=4pt,labelwidth=1em,leftmargin=0pt,nosep,label={\arabic*)}]&#10;\item We find that training the modular encoder from scratch under the CTC loss is insufficient for producing the best performance. Instead, our recipe pre-trains some base encoder layers with RNN-T loss and keeps them frozen. Next, we just train the extra Exporter layers with the auxiliary CTC loss. This solution is also practical since it enables researchers to cheaply export modular features without having to modify their original system. Thus, the quality, latency and efficiency of the base model are all maintained.&#10;&#10;\item We adapt the design to a streaming setting for the first time. Unlike the original work \cite{dalmia2019enforcing, dalmia2022legonn}, our encoder layers attention have limited left and right context windows, and the produced Lego-Features are successfully paired with a streaming-friendly RNN-T decoder. The streaming architecture still exhibits strong downstream ASR quality and passes the modularity test. By plugging the same fixed set of Lego-Features into causal as well as non-causal decoders, our work adds further evidence to their modularity and interoperability.&#10;&#10;\item Rather than merely looking at the Lego-Features as an encoded representation, we also study them as an alternative to the N-best hypotheses within two-pass systems. We provide new comparisons against the N-best in terms of speed, accuracy and modularity. To this end, the Lego-Features are used as a first-pass output within the deliberation framework \cite{hu2020deliberation}. This achieves good post-deliberation WER performance, which is shown to be on-par with a baseline that performs deliberation on 1st-pass RNN-T N-best hypotheses + audio features. The Lego-Features demonstrate success in the modularity test here as well. On the other hand, we find that the N-best hypothesis text does not pass the modularity test, i.e. a new N-best from a second model would confuse the deliberation decoder from the first, which is a novel observation. Moreover, the Lego-Features are cheaper to produce than the N-best, since they require no beam-search or auto-regressive decoding, but are generated via a simple projection at every frame.&#10;\end{enumerate}&#10;Other works have attempted to present generic methods for zero-shot stitching between layers. In \cite{moschella2022relative}, this is achieved by learning representations relative to data-dependent anchors. In contrast, the method presented here does not need to choose anchor samples and leverages the existence of ground-truth speech transcripts instead. Another general approach, presented in \cite{gygli2021towards}, uses self-supervised objectives designed to encourage compatibility of different layer outputs. It is an open question whether the cited methods can deal with long sequences, whereas the CTC loss used here is a natural choice that works well with ASR and gives interpretable outputs.&#10;&#10;Further, some research has already experimented with deliberation on top of CTC outputs to save the cost of first-pass decoding \cite{chi2020align,wang2022deliberation,wang2022streaming}. This includes the Align-refine approach, which iteratively improves on the first-pass output. Those works tend to focus on optimizing the size and speed of the first-pass model, whereas our focus is mainly on modularity. Nevertheless, since we build on base encoder layers that have been pre-trained with the RNN-T loss, we find our CTC outputs to have high quality, which removes the need for audio attention that is used in other deliberation models. Hence, this work also introduces some speed gains to deliberation, without using the iterative Align-refine approach.&#10;&#10;On the whole, with one simple representations, we get a compelling cheap, streaming-friendly, as well as modular, alternative to both the continuous encoding vector and the N-best hypotheses, without any loss in quality.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Recognition Models, Computer Science, Modular Encoded Representations, Linguistics, Artificial Intelligence, Streaming Speech Processing" />
        </attvalues>
      </node>
      <node id="2011.10798" label="2011.10798">
        <attvalues>
          <attvalue for="0" value="A Better and Faster End-to-End Model for Streaming ASR" />
          <attvalue for="1" value="  End-to-end (E2E) models have shown to outperform state-of-the-art&#10;conventional models for streaming speech recognition [1] across many&#10;dimensions, including quality (as measured by word error rate (WER)) and&#10;endpointer latency [2]. However, the model still tends to delay the predictions&#10;towards the end and thus has much higher partial latency compared to a&#10;conventional ASR model. To address this issue, we look at encouraging the E2E&#10;model to emit words early, through an algorithm called FastEmit [3]. Naturally,&#10;improving on latency results in a quality degradation. To address this, we&#10;explore replacing the LSTM layers in the encoder of our E2E model with&#10;Conformer layers [4], which has shown good improvements for ASR. Secondly, we&#10;also explore running a 2nd-pass beam search to improve quality. In order to&#10;ensure the 2nd-pass completes quickly, we explore non-causal Conformer layers&#10;that feed into the same 1st-pass RNN-T decoder, an algorithm called Cascaded&#10;Encoders [5]. Overall, we find that the Conformer RNN-T with Cascaded Encoders&#10;offers a better quality and latency tradeoff for streaming ASR.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.12415" label="1909.12415">
        <attvalues>
          <attvalue for="0" value="Improving RNN Transducer Modeling for End-to-End Speech Recognition" />
          <attvalue for="1" value="  In the last few years, an emerging trend in automatic speech recognition&#10;research is the study of end-to-end (E2E) systems. Connectionist Temporal&#10;Classification (CTC), Attention Encoder-Decoder (AED), and RNN Transducer&#10;(RNN-T) are the most popular three methods. Among these three methods, RNN-T&#10;has the advantages to do online streaming which is challenging to AED and it&#10;doesn't have CTC's frame-independence assumption. In this paper, we improve the&#10;RNN-T training in two aspects. First, we optimize the training algorithm of&#10;RNN-T to reduce the memory consumption so that we can have larger training&#10;minibatch for faster training speed. Second, we propose better model structures&#10;so that we obtain RNN-T models with the very good accuracy but small footprint.&#10;Trained with 30 thousand hours anonymized and transcribed Microsoft production&#10;data, the best RNN-T model with even smaller model size (216 Megabytes)&#10;achieves up-to 11.8% relative word error rate (WER) reduction from the baseline&#10;RNN-T model. This best RNN-T model is significantly better than the device&#10;hybrid model with similar size by achieving up-to 15.0% relative WER reduction,&#10;and obtains similar WERs as the server hybrid model of 5120 Megabytes in size.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09319" label="2005.09319">
        <attvalues>
          <attvalue for="0" value="A New Training Pipeline for an Improved Neural Transducer" />
          <attvalue for="1" value="  The RNN transducer is a promising end-to-end model candidate. We compare the&#10;original training criterion with the full marginalization over all alignments,&#10;to the commonly used maximum approximation, which simplifies, improves and&#10;speeds up our training. We also generalize from the original neural network&#10;model and study more powerful models, made possible due to the maximum&#10;approximation. We further generalize the output label topology to cover RNN-T,&#10;RNA and CTC. We perform several studies among all these aspects, including a&#10;study on the effect of external alignments. We find that the transducer model&#10;generalizes much better on longer sequences than the attention model. Our final&#10;transducer model outperforms our attention model on Switchboard 300h by over 6%&#10;relative WER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.09762" label="1911.09762">
        <attvalues>
          <attvalue for="0" value="Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR&#10;  Models" />
          <attvalue for="1" value="  In this paper, we propose to use pre-trained features from end-to-end ASR&#10;models to solve speech sentiment analysis as a down-stream task. We show that&#10;end-to-end ASR features, which integrate both acoustic and text information&#10;from speech, achieve promising results. We use RNN with self-attention as the&#10;sentiment classifier, which also provides an easy visualization through&#10;attention weights to help interpret model predictions. We use well benchmarked&#10;IEMOCAP dataset and a new large-scale speech sentiment dataset SWBD-sentiment&#10;for evaluation. Our approach improves the-state-of-the-art accuracy on IEMOCAP&#10;from 66.6% to 71.7%, and achieves an accuracy of 70.10% on SWBD-sentiment with&#10;more than 49,500 utterances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03782" label="1911.03782">
        <attvalues>
          <attvalue for="0" value="Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models" />
          <attvalue for="1" value="  Inspired by modular software design principles of independence,&#10;interchangeability, and clarity of interface, we introduce a method for&#10;enforcing encoder-decoder modularity in seq2seq models without sacrificing the&#10;overall model quality or its full differentiability. We discretize the encoder&#10;output units into a predefined interpretable vocabulary space using the&#10;Connectionist Temporal Classification (CTC) loss. Our modular systems achieve&#10;near SOTA performance on the 300h Switchboard benchmark, with WER of 8.3% and&#10;17.6% on the SWB and CH subsets, using seq2seq models with encoder and decoder&#10;modules which are independent and interchangeable.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03898" label="2004.03898">
        <attvalues>
          <attvalue for="0" value="Towards Reusable Network Components by Learning Compatible&#10;  Representations" />
          <attvalue for="1" value="  This paper proposes to make a first step towards compatible and hence&#10;reusable network components. Rather than training networks for different tasks&#10;independently, we adapt the training process to produce network components that&#10;are compatible across tasks. In particular, we split a network into two&#10;components, a features extractor and a target task head, and propose various&#10;approaches to accomplish compatibility between them. We systematically analyse&#10;these approaches on the task of image classification on standard datasets. We&#10;demonstrate that we can produce components which are directly compatible&#10;without any fine-tuning or compromising accuracy on the original tasks.&#10;Afterwards, we demonstrate the use of compatible components on three&#10;applications: Unsupervised domain adaptation, transferring classifiers across&#10;feature extractors with different architectures, and increasing the&#10;computational efficiency of transfer learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14233" label="2010.14233">
        <attvalues>
          <attvalue for="0" value="Align-Refine: Non-Autoregressive Speech Recognition via Iterative&#10;  Realignment" />
          <attvalue for="1" value="  Non-autoregressive models greatly improve decoding speed over typical&#10;sequence-to-sequence models, but suffer from degraded performance. Infilling&#10;and iterative refinement models make up some of this gap by editing the outputs&#10;of a non-autoregressive model, but are constrained in the edits that they can&#10;make. We propose iterative realignment, where refinements occur over latent&#10;alignments rather than output sequence space. We demonstrate this in speech&#10;recognition with Align-Refine, an end-to-end Transformer-based model which&#10;refines connectionist temporal classification (CTC) alignments to allow&#10;length-changing insertions and deletions. Align-Refine outperforms Imputer and&#10;Mask-CTC, matching an autoregressive baseline on WSJ at 1/14th the real-time&#10;factor and attaining a LibriSpeech test-other WER of 9.0% without an LM. Our&#10;model is strong even in one iteration with a shallower decoder.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;\import{sections/}{2-background.tex}&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2112.11442" label="2112.11442">
        <attvalues>
          <attvalue for="0" value="Deliberation of Streaming RNN-Transducer by Non-autoregressive Decoding" />
          <attvalue for="1" value="  We propose to deliberate the hypothesis alignment of a streaming RNN-T model&#10;with the previously proposed Align-Refine non-autoregressive decoding method&#10;and its improved versions. The method performs a few refinement steps, where&#10;each step shares a transformer decoder that attends to both text features&#10;(extracted from alignments) and audio features, and outputs complete updated&#10;alignments. The transformer decoder is trained with the CTC loss which&#10;facilitates parallel greedy decoding, and performs full-context attention to&#10;capture label dependencies. We improve Align-Refine by introducing cascaded&#10;encoder that captures more audio context before refinement, and alignment&#10;augmentation which enforces learning label dependency. We show that,&#10;conditioned on hypothesis alignments of a streaming RNN-T model, our method&#10;obtains significantly more accurate recognition results than the first-pass&#10;RNN-T, with only small amount of model parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.07556" label="2204.07556">
        <attvalues>
          <attvalue for="0" value="Streaming Align-Refine for Non-autoregressive Deliberation" />
          <attvalue for="1" value="  We propose a streaming non-autoregressive (non-AR) decoding algorithm to&#10;deliberate the hypothesis alignment of a streaming RNN-T model. Our algorithm&#10;facilitates a simple greedy decoding procedure, and at the same time is capable&#10;of producing the decoding result at each frame with limited right context, thus&#10;enjoying both high efficiency and low latency. These advantages are achieved by&#10;converting the offline Align-Refine algorithm to be streaming-compatible, with&#10;a novel transformer decoder architecture that performs local self-attentions&#10;for both text and audio, and a time-aligned cross-attention at each layer.&#10;Furthermore, we perform discriminative training of our model with the minimum&#10;word error rate (MWER) criterion, which has not been done in the non-AR&#10;decoding literature. Experiments on voice search datasets and Librispeech show&#10;that with reasonable right context, our streaming model performs as well as the&#10;offline counterpart, and discriminative training leads to further WER gain when&#10;the first-pass model has small capacity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08696" label="2104.08696">
        <attvalues>
          <attvalue for="0" value="Knowledge Neurons in Pretrained Transformers" />
          <attvalue for="1" value="  Large-scale pretrained language models are surprisingly good at recalling&#10;factual knowledge presented in the training corpus. In this paper, we present&#10;preliminary studies on how factual knowledge is stored in pretrained&#10;Transformers by introducing the concept of knowledge neurons. Specifically, we&#10;examine the fill-in-the-blank cloze task for BERT. Given a relational fact, we&#10;propose a knowledge attribution method to identify the neurons that express the&#10;fact. We find that the activation of such knowledge neurons is positively&#10;correlated to the expression of their corresponding facts. In our case studies,&#10;we attempt to leverage knowledge neurons to edit (such as update, and erase)&#10;specific factual knowledge without fine-tuning. Our results shed light on&#10;understanding the storage of knowledge within pretrained Transformers. The code&#10;is available at https://github.com/Hunter-DDM/knowledge-neurons.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04541" label="2110.04541">
        <attvalues>
          <attvalue for="0" value="The Inductive Bias of In-Context Learning: Rethinking Pretraining&#10;  Example Design" />
          <attvalue for="1" value="  Pretraining Neural Language Models (NLMs) over a large corpus involves&#10;chunking the text into training examples, which are contiguous text segments of&#10;sizes processable by the neural architecture. We highlight a bias introduced by&#10;this common practice: we prove that the pretrained NLM can model much stronger&#10;dependencies between text segments that appeared in the same training example,&#10;than it can between text segments that appeared in different training examples.&#10;This intuitive result has a twofold role. First, it formalizes the motivation&#10;behind a broad line of recent successful NLM training heuristics, proposed for&#10;the pretraining and fine-tuning stages, which do not necessarily appear related&#10;at first glance. Second, our result clearly indicates further improvements to&#10;be made in NLM pretraining for the benefit of Natural Language Understanding&#10;tasks. As an example, we propose &quot;kNN-Pretraining&quot;: we show that including&#10;semantically related non-neighboring sentences in the same pretraining example&#10;yields improved sentence representations and open domain question answering&#10;abilities. This theoretically motivated degree of freedom for pretraining&#10;example design indicates new training schemes for self-improving&#10;representations.&#10;" />
          <attvalue for="2" value="&#10;&#09;\ifdefined\SQUEEZE &#10;&#10;Beyond excelling in their core task of pure language modeling, modern Neural Language Models (NLMs) show impressive zero- and few-shot abilities in more general Natural Language Understanding (NLU) tasks~\cite{GPT3}. &#10;&#09;This implies that the training corpus contains the information required for performing such tasks, and moreover it implies that the common pretraining process &#10;&#09;grants the trained NLM some access to these higher level capabilities. &#10;&#09;In this paper, we highlight a connection between the quality of the emergent NLU capabilities and a basic component in the NLM training scheme: the process of segmenting the corpus into training examples.&#10;&#10;Specifically, NLMs self-train over huge training corpora (typically, billions to trillions of words).&#10;&#09;A basic, automatic, operation in the training pipeline is to segment these corpora into {training examples}: contiguous text chunks of sizes processable by the neural architecture (typically, up to thousands of words). &#10;&#09;We formalize an expressivity bias that this segmentation process introduces, to be referred to as the in-context bias, which directly affects the NLM's ability to integrate cross-corpus information.&#10;&#09;We show that the NLM can model much stronger dependencies between sentences that were shown together at least once {in-context}, i.e., in the same training example, than between sentences that were never shown together in the same input.&#10;&#09;This inductive bias may be good for language modeling, but it implies that NLU capabilities that involve integrating information from different examples across the corpus (see, e.g., figure~\ref{fig:fig1}), are under-favored by design in the current setting.&#10;&#09;Thus, if one sentence in the corpus can elucidate the meaning of another sentence (e.g., defines a hard concept or provides auxiliary information), our result implies that a model that saw them in different training examples will enjoy this elucidation less than a model that saw them in the same training example. &#10;&#10;While standard approximation results examine the expressivity of an architecture over a single input, our theoretical approach pertains to the entire training process, and examines the expressive capacity of the resultant NLM with respect to the training set. &#10;&#09;Therefore, our approximation result ties an optimization parameter (the learning-rate) to the regular NLM architecture expressivity parameters (depth, width). &#10;&#09;Intuitively, sentences that were never shown in the same input can only access each other via the weights of the network during training. The mechanism for ``storing&quot; information in the network involves a very small learning-rate term $\eta$; our analysis formalizes and quantifies an ``expressivity toll&quot; that the model pays when making use of such harder-to-access stored information.&#10;&#10;We employ the tool of a function’s separation rank with respect to subsets of its variables, which&#10;&#09;quantifies its ability to model input dependencies between these subsets. The separation rank&#10;&#09;was employed for analyzing the dependencies modeled by convolutional~\cite{cohen2017inductive}, recurrent~\cite{levine2018benefits}, and self-attention ~\cite{levine2020limits} networks with respect to a single input example.&#10;&#09;In order to analyze an NLM's ability to model dependencies between different training examples, we refine the usage of this measure in two manners: (1) we introduce the $\varepsilon$-separation rank, which measures the effective ability of a function to model dependencies in a finite precision setting, and (2) we modify the separation rank such that it can account for the more intricate mechanism of mixing between variables that occurs in the sequential case. &#10;&#10;Specifically, we upper bound the log of the separation rank of a depth $L$ width $d_x$ self-attention based NLM, with respect to two sentences that are shown in its input, by $\tilde{O}(d_xL)$, and prove that this bound is tight. &#10;&#09;On the other hand, we upper bound this measure with respect to two sentences that were never shown in the same input by $\tilde{O}(d_x[L-0.5 \log_3(\eta^{-1})])$.&#10;&#09;Given common learning-rate values of $\eta\in[10^{-6},10^{-4}]$, &#10;&#09;{this implies a guaranteed ``depth deficit&quot; of $\sim 6$ layers for modeling dependencies between sentences that are not seen in the same training example}. After the presentation of our results, we point at empirical evidence that imply that this depth deficit is more significant, and may behave like a fraction of $L$. We leave attempts to tighten the depth deficit estimates to future work. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Natural Language Understanding, Neural Language Models, Mathematics, Text Representation" />
        </attvalues>
      </node>
      <node id="1605.06743" label="1605.06743">
        <attvalues>
          <attvalue for="0" value="Inductive Bias of Deep Convolutional Networks through Pooling Geometry" />
          <attvalue for="1" value="  Our formal understanding of the inductive bias that drives the success of&#10;convolutional networks on computer vision tasks is limited. In particular, it&#10;is unclear what makes hypotheses spaces born from convolution and pooling&#10;operations so suitable for natural images. In this paper we study the ability&#10;of convolutional networks to model correlations among regions of their input.&#10;We theoretically analyze convolutional arithmetic circuits, and empirically&#10;validate our findings on other types of convolutional networks as well.&#10;Correlations are formalized through the notion of separation rank, which for a&#10;given partition of the input, measures how far a function is from being&#10;separable. We show that a polynomially sized deep network supports&#10;exponentially high separation ranks for certain input partitions, while being&#10;limited to polynomial separation ranks for others. The network's pooling&#10;geometry effectively determines which input partitions are favored, thus serves&#10;as a means for controlling the inductive bias. Contiguous pooling windows as&#10;commonly employed in practice favor interleaved partitions over coarse ones,&#10;orienting the inductive bias towards the statistics of natural images. Other&#10;pooling schemes lead to different preferences, and this allows tailoring the&#10;network to data that departs from the usual domain of natural imagery. In&#10;addition to analyzing deep networks, we show that shallow ones support only&#10;linear separation ranks, and by this gain insight into the benefit of functions&#10;brought forth by depth - they are able to efficiently model strong correlation&#10;under favored partitions of the input.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.12467" label="2006.12467">
        <attvalues>
          <attvalue for="0" value="The Depth-to-Width Interplay in Self-Attention" />
          <attvalue for="1" value="  Self-attention architectures, which are rapidly pushing the frontier in&#10;natural language processing, demonstrate a surprising depth-inefficient&#10;behavior: previous works indicate that increasing the internal representation&#10;(network width) is just as useful as increasing the number of self-attention&#10;layers (network depth). We theoretically predict a width-dependent transition&#10;between depth-efficiency and depth-inefficiency in self-attention. We conduct&#10;systematic empirical ablations on networks of depths 6 to 48 that clearly&#10;reveal the theoretically predicted behaviors, and provide explicit quantitative&#10;suggestions regarding the optimal depth-to-width allocation for a given&#10;self-attention network size. The race towards beyond 1-Trillion parameter&#10;language models renders informed guidelines for increasing self-attention depth&#10;and width in tandem an essential ingredient. Our guidelines elucidate the&#10;depth-to-width trade-off in self-attention networks of sizes up to the scale of&#10;GPT3 (which we project to be too deep for its size), and beyond, marking an&#10;unprecedented width of 30K as optimal for a 1-Trillion parameter network.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.11696" label="2108.11696">
        <attvalues>
          <attvalue for="0" value="Rethinking Why Intermediate-Task Fine-Tuning Works" />
          <attvalue for="1" value="  Supplementary Training on Intermediate Labeled-data Tasks (STILTs) is a&#10;widely applied technique, which first fine-tunes the pretrained language models&#10;on an intermediate task before on the target task of interest. While STILTs is&#10;able to further improve the performance of pretrained language models, it is&#10;still unclear why and when it works. Previous research shows that those&#10;intermediate tasks involving complex inference, such as commonsense reasoning,&#10;work especially well for RoBERTa. In this paper, we discover that the&#10;improvement from an intermediate task could be orthogonal to it containing&#10;reasoning or other complex skills -- a simple real-fake discrimination task&#10;synthesized by GPT2 can benefit diverse target tasks. We conduct extensive&#10;experiments to study the impact of different factors on STILTs. These findings&#10;suggest rethinking the role of intermediate fine-tuning in the STILTs pipeline.&#10;" />
          <attvalue for="2" value="&#10;Pretrained language models~\cite{peters2018deep, radford2018improving, devlin2019bert, liu2019roberta} have contributed to great progress in natural language understanding (NLU).&#10;STILT~\cite{phang2018sentence, wang2019can, clark2019boolq, pruksachatkun2020intermediate, phang2020english, vu2020exploring} can further improve their performance on downstream NLU tasks by redesigning the training pipeline, introducing an intermediate-task fine-tuning phase before fine-tuning the pretrained models on the target task of interest (Figure~\ref{fig:STILT}). &#10;Nevertheless, this approach is not necessarily beneficial, and its effectiveness depends highly on the intermediate task applied. &#10;&#10;To study when and why STILT works, \cite{pruksachatkun2020intermediate} conduct large-scale experiments based on RoBERTa-large~\cite{liu2019roberta} with different intermediate-target task pairs.&#10;They focus on studying what kind of intermediate tasks are helpful overall and which linguistic skills a model learns from the intermediate phase. They show the difficulty to have a generally useful intermediate task and conclude that those containing complex reasoning and inference, such as CosmosQA~\cite{huang2019cosmos} and HellaSwag~\cite{zellers2019hellaswag}, tend to enhance various target tasks. However, this ignores the fact that HellaSwag is a synthetic dataset, and RoBERTa tends to capture the data artifacts when fine-tuned on HellaSwag~\cite{tamborrino2020pre}.&#10;&#10;In this paper, we demonstrate that intermediate tasks' enhancement could be irrelevant to providing complex reasoning or special linguistic skills --- a simple real-fake discrimination task synthesized by GPT2~\cite{radford2019language} can benefit diverse target tasks, including those commonsense reasoning tasks.&#10;These observations suggest rethinking the role of the intermediate-finetuning phase in the pipeline. Our main contributions are as follows.&#10;\begin{itemize}&#10;\item We discover that a widely beneficial intermediate task is not required to provide specific linguistic or reasoning skills.&#10;\item We highlight STILT's enhancement on fine-tuning stability, providing more than 1000 experimental observations on RoBERTa-large.&#10;\item We study different factors that may influence STILT's efficacy, suggesting rethinking why it works.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Training, Intermediate Task Design, Fine-Tuning Techniques, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1909.00277" label="1909.00277">
        <attvalues>
          <attvalue for="0" value="Cosmos QA: Machine Reading Comprehension with Contextual Commonsense&#10;  Reasoning" />
          <attvalue for="1" value="  Understanding narratives requires reading between the lines, which in turn,&#10;requires interpreting the likely causes and effects of events, even when they&#10;are not mentioned explicitly. In this paper, we introduce Cosmos QA, a&#10;large-scale dataset of 35,600 problems that require commonsense-based reading&#10;comprehension, formulated as multiple-choice questions. In stark contrast to&#10;most existing reading comprehension datasets where the questions focus on&#10;factual and literal understanding of the context paragraph, our dataset focuses&#10;on reading between the lines over a diverse collection of people's everyday&#10;narratives, asking such questions as &quot;what might be the possible reason of&#10;...?&quot;, or &quot;what would have happened if ...&quot; that require reasoning beyond the&#10;exact text spans in the context. To establish baseline performances on Cosmos&#10;QA, we experiment with several state-of-the-art neural architectures for&#10;reading comprehension, and also propose a new architecture that improves over&#10;the competitive baselines. Experimental results demonstrate a significant gap&#10;between machine (68.4%) and human performance (94%), pointing to avenues for&#10;future research on commonsense machine comprehension. Dataset, code and&#10;leaderboard is publicly available at https://wilburone.github.io/cosmos.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14074" label="2004.14074">
        <attvalues>
          <attvalue for="0" value="Pre-training Is (Almost) All You Need: An Application to Commonsense&#10;  Reasoning" />
          <attvalue for="1" value="  Fine-tuning of pre-trained transformer models has become the standard&#10;approach for solving common NLP tasks. Most of the existing approaches rely on&#10;a randomly initialized classifier on top of such networks. We argue that this&#10;fine-tuning procedure is sub-optimal as the pre-trained model has no prior on&#10;the specific classifier labels, while it might have already learned an&#10;intrinsic textual representation of the task. In this paper, we introduce a new&#10;scoring method that casts a plausibility ranking task in a full-text format and&#10;leverages the masked language modeling head tuned during the pre-training&#10;phase. We study commonsense reasoning tasks where the model must rank a set of&#10;hypotheses given a premise, focusing on the COPA, Swag, HellaSwag and&#10;CommonsenseQA datasets. By exploiting our scoring method without fine-tuning,&#10;we are able to produce strong baselines (e.g. 80% test accuracy on COPA) that&#10;are comparable to supervised approaches. Moreover, when fine-tuning directly on&#10;the proposed scoring function, we show that our method provides a much more&#10;stable training phase across random restarts (e.g $\times 10$ standard&#10;deviation reduction on COPA test accuracy) and requires less annotated data&#10;than the standard classifier approach to reach equivalent performances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.13034" label="1910.13034">
        <attvalues>
          <attvalue for="0" value="Big Bidirectional Insertion Representations for Documents" />
          <attvalue for="1" value="  The Insertion Transformer is well suited for long form text generation due to&#10;its parallel generation capabilities, requiring $O(\log_2 n)$ generation steps&#10;to generate $n$ tokens. However, modeling long sequences is difficult, as there&#10;is more ambiguity captured in the attention mechanism. This work proposes the&#10;Big Bidirectional Insertion Representations for Documents (Big BIRD), an&#10;insertion-based model for document-level translation tasks. We scale up the&#10;insertion-based models to long form documents. Our key contribution is&#10;introducing sentence alignment via sentence-positional embeddings between the&#10;source and target document. We show an improvement of +4.3 BLEU on the WMT'19&#10;English$\rightarrow$German document-level translation task compared with the&#10;Insertion Transformer baseline.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, insertion-based models \cite{stern-icml-2019,welleck-icml-2019,gu-arxiv-2019,chan-arxiv-2019} have been introduced for text generation. Unlike traditional autoregressive left-to-right models \cite{cho-emnlp-2014,sutskever-nips-2014,vaswani-nips-2017}, insertion-based models are not restricted to generating text sequences in a serial left-to-right manner, but these models are endowed with the capabilities of parallel generation. More specifically, \cite{stern-icml-2019,chan-arxiv-2019} showed that we can teach neural nets to generate text to follow a balanced binary tree order. An autoregressive left-to-right model would require $O(n)$ generation steps to generate $n$ tokens, whereas the Insertion Transformer \cite{stern-icml-2019} and KERMIT \cite{chan-arxiv-2019} following a balanced binary tree policy requires only $O(\log_2 n)$ generation steps to generate $n$ tokens. This is especially important for long-form text generation, for example, Document-Level Machine Translation.&#10;&#10;Document-Level Machine Translation is becoming an increasingly important task. Recent research suggests we are nearing human-level parity for sentence-level translation in certain domains \cite{hassan-arxiv-2018}, however, we lag significantly behind in document-level translation \cite{laubli-emnlp-2018}. Various papers have proposed incorporating context for document-level translation \cite{dowmunt-wmt-2019}, which has been shown to improve translation quality. There are two primary methods to include context in a document-level machine translation model compared to a sentence-level translation model.&#10;\begin{enumerate}[itemsep=0pt]&#10; \item Source Contextualization. We can include source context, wherein when we generate the target sentence, we can condition on the corresponding source sentence and its neighbours, or even the whole source document. This allows the target sentence to be contextualized to the source document.&#10; \item Target Contextualization. We can include target context, wherein when we generate the target sentence, we can condition on all the target tokens generated thus far in the whole document. This allows the target sentence to be contextualized to other target sentences.&#10;\end{enumerate}&#10;&#10;Target contextualization is especially difficult in an autoregressive left-to-right model (i.e., Transformer \cite{vaswani-nips-2017}), the model must generate the whole document in linear fashion, which would be prohibitively expensive costing $O(n)$ iterations to generate $n$ tokens. Additionally, the model is unable to model bidirectional context, since the text is always generated in a left-to-right manner.&#10;Some prior work have focused on utilizing block coordinate descent like algorithms during inference \cite{maruf-acl-2018}, however this adds complexity and additional runtime cost during inference.&#10;&#10;Insertion-based models, for example, the Insertion Transformer \cite{stern-icml-2019} is one potential solution. The Insertion Transformer can generate text following a balanced binary tree order. It requires $O(\log_2 n)$ iterations to generate $n$ tokens, offering significant inference time advantages over a serial generation model. The source document is naturally fully conditioned on, which provides full source contextualization. Additionally, the generation order offers bidirectional contextualization, permitting target contextualization that is not solely on a left-to-right basis.&#10;&#10;In this paper, we present Big Bidirectional Insertion Representations for Documents (Big BIRD). We address the limitations of scaling up the Insertion Transformer to document-level machine translation. We present a model that can model long-form documents with thousands of tokens in a fully contextualized manner.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Network Architectures, Artificial Intelligence, Mathematics, Document Translation Tasks, Text Generation Models" />
        </attvalues>
      </node>
      <node id="1902.02192" label="1902.02192">
        <attvalues>
          <attvalue for="0" value="Non-Monotonic Sequential Text Generation" />
          <attvalue for="1" value="  Standard sequential generation methods assume a pre-specified generation&#10;order, such as text generation methods which generate words from left to right.&#10;In this work, we propose a framework for training models of text generation&#10;that operate in non-monotonic orders; the model directly learns good orders,&#10;without any additional annotation. Our framework operates by generating a word&#10;at an arbitrary position, and then recursively generating words to its left and&#10;then words to its right, yielding a binary tree. Learning is framed as&#10;imitation learning, including a coaching method which moves from imitating an&#10;oracle to reinforcing the policy's own preferences. Experimental results&#10;demonstrate that using the proposed method, it is possible to learn policies&#10;which generate text without pre-specifying a generation order, while achieving&#10;competitive performance with conventional left-to-right generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.01370" label="1902.01370">
        <attvalues>
          <attvalue for="0" value="Insertion-based Decoding with automatically Inferred Generation Order" />
          <attvalue for="1" value="  Conventional neural autoregressive decoding commonly assumes a fixed&#10;left-to-right generation order, which may be sub-optimal. In this work, we&#10;propose a novel decoding algorithm -- InDIGO -- which supports flexible&#10;sequence generation in arbitrary orders through insertion operations. We extend&#10;Transformer, a state-of-the-art sequence generation model, to efficiently&#10;implement the proposed approach, enabling it to be trained with either a&#10;pre-defined generation order or adaptive orders obtained from beam-search.&#10;Experiments on four real-world tasks, including word order recovery, machine&#10;translation, image caption and code generation, demonstrate that our algorithm&#10;can generate sequences following arbitrary orders, while achieving competitive&#10;or even better performance compared to the conventional left-to-right&#10;generation. The generated sequences show that InDIGO adopts adaptive generation&#10;orders based on input information.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.05567" label="1803.05567">
        <attvalues>
          <attvalue for="0" value="Achieving Human Parity on Automatic Chinese to English News Translation" />
          <attvalue for="1" value="  Machine translation has made rapid advances in recent years. Millions of&#10;people are using it today in online translation systems and mobile applications&#10;in order to communicate across language barriers. The question naturally arises&#10;whether such systems can approach or achieve parity with human translations. In&#10;this paper, we first address the problem of how to define and accurately&#10;measure human parity in translation. We then describe Microsoft's machine&#10;translation system and measure the quality of its translations on the widely&#10;used WMT 2017 news translation task from Chinese to English. We find that our&#10;latest neural machine translation system has reached a new state-of-the-art,&#10;and that the translation quality is at human parity when compared to&#10;professional human translations. We also find that it significantly exceeds the&#10;quality of crowd-sourced non-professional translations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.06170" label="1907.06170">
        <attvalues>
          <attvalue for="0" value="Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level&#10;  Neural Machine Translation" />
          <attvalue for="1" value="  This paper describes the Microsoft Translator submissions to the WMT19 news&#10;translation shared task for English-German. Our main focus is document-level&#10;neural machine translation with deep transformer models. We start with strong&#10;sentence-level baselines, trained on large-scale data created via&#10;data-filtering and noisy back-translation and find that back-translation seems&#10;to mainly help with translationese input. We explore fine-tuning techniques,&#10;deeper models and different ensembling strategies to counter these effects.&#10;Using document boundaries present in the authentic and synthetic parallel data,&#10;we create sequences of up to 1000 subword segments and train transformer&#10;translation models. We experiment with data augmentation techniques for the&#10;smaller authentic data with document-boundaries and for larger authentic data&#10;without boundaries. We further explore multi-task training for the&#10;incorporation of document-level source language monolingual data via the&#10;BERT-objective on the encoder and two-pass decoding for combinations of&#10;sentence-level and document-level systems. Based on preliminary human&#10;evaluation results, evaluators strongly prefer the document-level systems over&#10;our comparable sentence-level system. The document-level systems also seem to&#10;score higher than the human references in source-based direct assessment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.09268" label="2302.09268">
        <attvalues>
          <attvalue for="0" value="Bag of Tricks for Effective Language Model Pretraining and Downstream&#10;  Adaptation: A Case Study on GLUE" />
          <attvalue for="1" value="  This technical report briefly describes our JDExplore d-team's submission&#10;Vega v1 on the General Language Understanding Evaluation (GLUE) leaderboard,&#10;where GLUE is a collection of nine natural language understanding tasks,&#10;including question answering, linguistic acceptability, sentiment analysis,&#10;text similarity, paraphrase detection, and natural language inference. [Method]&#10;We investigate several effective strategies and choose their best combination&#10;setting as the training recipes. As for model structure, we employ the vanilla&#10;Transformer with disentangled attention as the basic block encoder. For&#10;self-supervised training, we employ the representative denoising objective&#10;(i.e., replaced token detection) in phase 1 and combine the contrastive&#10;objective (i.e., sentence embedding contrastive learning) with it in phase 2.&#10;During fine-tuning, several advanced techniques such as transductive&#10;fine-tuning, self-calibrated fine-tuning, and adversarial fine-tuning are&#10;adopted. [Results] According to our submission record (Jan. 2022), with our&#10;optimized pretraining and fine-tuning strategies, our 1.3 billion model sets&#10;new state-of-the-art on 4/9 tasks, achieving the best average score of 91.3.&#10;Encouragingly, our Vega v1 is the first to exceed powerful human performance on&#10;the two challenging tasks, i.e., SST-2 and WNLI. We believe our empirically&#10;successful recipe with a bag of tricks could shed new light on developing&#10;efficient discriminative large language models.&#10;" />
          <attvalue for="2" value="&#10;Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}. These PLMs share a common principle of performing self-supervised learning with massive easy-to-acquire unlabelled text corpora during the pretraining stage and effectively fine-tuning on downstream tasks. In such a context, the general language understanding evaluation (GLUE,~\cite{wang2018glue}) benchmark has emerged as the leading evaluation standard for the pretrained language model community, where most high-performing models (e.g., T5~\cite{t5}) on its leaderboard provide valuable insights and best practices for future research and applications.&#10;&#10;We recently submitted our 1.3B Vega v1 model to the GLUE leaderboard and, as seen in Figure~\ref{fig:radia}, obtained state-of-the-art records on 4 out of 9 tasks, sitting atop the leaderboard as of January 1, 2022, with an average score of 91.3. &#10;More encouragingly, our Vega v1 is the first to exceed powerful human performance on the two challenging tasks, i.e., SST-2~\cite{socher2013recursive} and WNLI~\cite{levesque2012winograd}. This technical report briefly describes how we build our powerful model under a certain parameter budget, i.e., 1.3B, from different aspects, including backbone framework (\S\ref{subsec:backbone}), efficient pretraining processes (\S\ref{subsec:pretrain}), and effective downstream adaptation approaches (\S\ref{subsec:downstream}).&#10;To achieve efficient and sufficient pretraining, we replace the widely-used masked language modeling (MLM,~\cite{devlin-etal-2019-bert}) with two simple but effective objectives, i.e., denoising and contrastive objectives. The denoising objective~\cite{yamaguchi2021frustratingly} aims to improve the data efficiency and save training costs, while the contrastive objective involves leveraging contrastive learning~\cite{gao2021simcse} to learn better sentence representations. &#10;For downstream adaptation, we focus on two common problems, i.e., domain discrepancy and over-fitting, and adopt several effective fine-tuning methods, such as self-calibrated transductive fine-tuning and adversarial fine-tuning, to achieve better performance and generalization.&#10;&#10;The rest of this paper is organized as follows. In Section~\ref{sec:app}, we introduce the major utilized approaches. Then, Section~\ref{sec:exp} reports and discusses our evaluation results. Finally, we conclude our study in Section~\ref{sec:con}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Natural Language Understanding, Artificial Intelligence, Model Training Strategies" />
        </attvalues>
      </node>
      <node id="2204.07832" label="2204.07832">
        <attvalues>
          <attvalue for="0" value="A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based&#10;  Sentiment Analysis" />
          <attvalue for="1" value="  Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment analysis&#10;task, which focuses on detecting the sentiment polarity towards the aspect in a&#10;sentence. However, it is always sensitive to the multi-aspect challenge, where&#10;features of multiple aspects in a sentence will affect each other. To mitigate&#10;this issue, we design a novel training framework, called Contrastive&#10;Cross-Channel Data Augmentation (C3 DA), which leverages an in-domain generator&#10;to construct more multi-aspect samples and then boosts the robustness of ABSA&#10;models via contrastive learning on these generated data. In practice, given a&#10;generative pretrained language model and some limited ABSA labeled data, we&#10;first employ some parameter-efficient approaches to perform the in-domain&#10;fine-tuning. Then, the obtained in-domain generator is used to generate the&#10;synthetic sentences from two channels, i.e., Aspect Augmentation Channel and&#10;Polarity Augmentation Channel, which generate the sentence condition on a given&#10;aspect and polarity respectively. Specifically, our C3 DA performs the sentence&#10;generation in a cross-channel manner to obtain more sentences, and proposes an&#10;Entropy-Minimization Filter to filter low-quality generated samples. Extensive&#10;experiments show that our C3 DA can outperform those baselines without any&#10;augmentations by about 1% on accuracy and Macro- F1. Code and data are released&#10;in https://github.com/wangbing1416/C3DA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.04831" label="2201.04831">
        <attvalues>
          <attvalue for="0" value="Knowledge Graph Augmented Network Towards Multiview Representation&#10;  Learning for Aspect-based Sentiment Analysis" />
          <attvalue for="1" value="  Aspect-based sentiment analysis (ABSA) is a fine-grained task of sentiment&#10;analysis. To better comprehend long complicated sentences and obtain accurate&#10;aspect-specific information, linguistic and commonsense knowledge are generally&#10;required in this task. However, most current methods employ complicated and&#10;inefficient approaches to incorporate external knowledge, e.g., directly&#10;searching the graph nodes. Additionally, the complementarity between external&#10;knowledge and linguistic information has not been thoroughly studied. To this&#10;end, we propose a knowledge graph augmented network KGAN, which aims to&#10;effectively incorporate external knowledge with explicitly syntactic and&#10;contextual information. In particular, KGAN captures the sentiment feature&#10;representations from multiple different perspectives, i.e., context-, syntax-&#10;and knowledge-based. First, KGAN learns the contextual and syntactic&#10;representations in parallel to fully extract the semantic features. Then, KGAN&#10;integrates the knowledge graphs into the embedding space, based on which the&#10;aspect-specific knowledge representations are further obtained via an attention&#10;mechanism. Last, we propose a hierarchical fusion module to complement these&#10;multi-view representations in a local-to-global manner. Extensive experiments&#10;on five popular ABSA benchmarks demonstrate the effectiveness and robustness of&#10;our KGAN. Notably, with the help of the pretrained model of RoBERTa, KGAN&#10;achieves a new record of state-of-the-art performance among all datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.02693" label="2010.02693">
        <attvalues>
          <attvalue for="0" value="SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection&#10;  and Slot Filling" />
          <attvalue for="1" value="  Slot filling and intent detection are two main tasks in spoken language&#10;understanding (SLU) system. In this paper, we propose a novel&#10;non-autoregressive model named SlotRefine for joint intent detection and slot&#10;filling. Besides, we design a novel two-pass iteration mechanism to handle the&#10;uncoordinated slots problem caused by conditional independence of&#10;non-autoregressive model. Experiments demonstrate that our model significantly&#10;outperforms previous models in slot filling task, while considerably speeding&#10;up the decoding (up to X 10.77). In-depth analyses show that 1) pretraining&#10;schemes could further enhance our model; 2) two-pass mechanism indeed remedy&#10;the uncoordinated slots.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.10714" label="2205.10714">
        <attvalues>
          <attvalue for="0" value="Interpretable Proof Generation via Iterative Backward Reasoning" />
          <attvalue for="1" value="  We present IBR, an Iterative Backward Reasoning model to solve the proof&#10;generation tasks on rule-based Question Answering (QA), where models are&#10;required to reason over a series of textual rules and facts to find out the&#10;related proof path and derive the final answer. We handle the limitations of&#10;existed works in two folds: 1) enhance the interpretability of reasoning&#10;procedures with detailed tracking, by predicting nodes and edges in the proof&#10;path iteratively backward from the question; 2) promote the efficiency and&#10;accuracy via reasoning on the elaborate representations of nodes and history&#10;paths, without any intermediate texts that may introduce external noise during&#10;proof generation. There are three main modules in IBR, QA and proof strategy&#10;prediction to obtain the answer and offer guidance for the following procedure;&#10;parent node prediction to determine a node in the existing proof that a new&#10;child node will link to; child node prediction to find out which new node will&#10;be added to the proof. Experiments on both synthetic and paraphrased datasets&#10;demonstrate that IBR has better in-domain performance as well as cross-domain&#10;transferability than several strong baselines. Our code and models are&#10;available at https://github.com/find-knowledge/IBR .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.09972" label="2406.09972">
        <attvalues>
          <attvalue for="0" value="A Better LLM Evaluator for Text Generation: The Impact of Prompt Output&#10;  Sequencing and Optimization" />
          <attvalue for="1" value="  This research investigates prompt designs of evaluating generated texts using&#10;large language models (LLMs). While LLMs are increasingly used for scoring&#10;various inputs, creating effective prompts for open-ended text evaluation&#10;remains challenging due to model sensitivity and subjectivity in evaluation of&#10;text generation. Our study experimented with different prompt structures,&#10;altering the sequence of output instructions and including explanatory reasons.&#10;We found that the order of presenting reasons and scores significantly&#10;influences LLMs' scoring, with a different level of rule understanding in the&#10;prompt. An additional optimization may enhance scoring alignment if sufficient&#10;data is available. This insight is crucial for improving the accuracy and&#10;consistency of LLM-based evaluations.&#10;" />
          <attvalue for="2" value="&#10;&#10;Using large language models (LLMs)~\cite{GPT4,llama2} as evaluators to assign scores to the given inputs have become prevalent. &#10;\cite{AlphaCode2} outputs a score between 0 and 1 to estimate the correctness of generated code, thereby automatically ranking its quality. Similarly, \cite{generative_agent} assign poignancy score to the generated text for the retrieval task. &#10;Other research explores using LLMs to assess generated texts, finding the LLM scores correlates higher with human evaluators than existing automatic metrics \cite{gao2023human,shen2023large,liu-etal-2023-g,luo2023chatgpt}. &#10;&#10;However, designing evaluation prompt for LLMs is not a trivial task, especially for dialogue evaluation. &#10;Different models exhibit varied sensitivity to the nuances of input prompts. Even slight linguistic variations can lead to significant fluctuations in task performance \cite{leidinger2023linguistic}. &#10;Moreover, the inherent subjectivity in dialogue evaluation adds on the difficulty and versatility in LLMs' evaluation results. &#10;While prompt optimization techniques \cite{chen2023instructzero,LLMoptim,zhang2023tempera,prasad2023grips} have been developed to assist in designing more effective prompts, these methods require paired input-output samples for objective value calculation. &#10;Unfortunately, the lack of available dialogue-score pairing data hampers the application of prompt optimization in dialogue evaluation. &#10;&#10;In this study, we aim to investigate the influence of prompt design on dialogue evaluation, specifically focusing on how the output instructions affects the resulting scores. &#10;We have developed multiple prompt variations to assess the quality of a series of dialogues. &#10;These variations involve altering the sequence order of the outputs and examining whether including explanatory reasons along with the scores impacts the evaluation. &#10;Our analysis compares the influence of different prompts on the scoring outcomes across various versions of GPT models. &#10;&#10;We observed that the different order of output instructions can result in different scoring distributions by certain LLMs, even when the corresponding output reasons are similar. &#10;Considering the sequential generation nature of auto-regressive models, placing the score after the reasons allows it to reference both the reasons and the input prompt, a dynamic not possible when this order is reversed. &#10;The finding suggests that a ``reason-first'' output instruction might lead to a more comprehensive understanding and adherence to the specific requirements of the task. &#10;&#10;Finally, we further investigate the effect of prompt optimization on the ``reason-first'' output instruction. This is to observe the most effective form of instruction in a hypothetical setting where paired data is available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Prompt Design Optimization, Natural Language Processing, Text Generation Scoring" />
        </attvalues>
      </node>
      <node id="2304.02554" label="2304.02554">
        <attvalues>
          <attvalue for="0" value="Human-like Summarization Evaluation with ChatGPT" />
          <attvalue for="1" value="  Evaluating text summarization is a challenging problem, and existing&#10;evaluation metrics are far from satisfactory. In this study, we explored&#10;ChatGPT's ability to perform human-like summarization evaluation using four&#10;human evaluation methods on five datasets. We found that ChatGPT was able to&#10;complete annotations relatively smoothly using Likert scale scoring, pairwise&#10;comparison, Pyramid, and binary factuality evaluation. Additionally, it&#10;outperformed commonly used automatic evaluation metrics on some datasets.&#10;Furthermore, we discussed the impact of different prompts, compared its&#10;performance with that of human evaluation, and analyzed the generated&#10;explanations and invalid responses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13091" label="2305.13091">
        <attvalues>
          <attvalue for="0" value="Large Language Models are Not Yet Human-Level Evaluators for Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  With the recent undeniable advancement in reasoning abilities in large&#10;language models (LLMs) like ChatGPT and GPT-4, there is a growing trend for&#10;using LLMs on various tasks. One area where LLMs can be employed is as an&#10;alternative evaluation metric for complex generative tasks, which generally&#10;demands expensive human judges to complement the traditional automatic metrics&#10;for various evaluation dimensions such as fluency and consistency. In this&#10;work, we conduct extensive analysis to investigate the stability and&#10;reliability of LLMs as automatic evaluators for abstractive summarization. We&#10;found that while ChatGPT and GPT-4 outperform the commonly used automatic&#10;metrics, they are not ready as human replacements due to significant&#10;limitations. That is, LLM evaluators rate each candidate system inconsistently&#10;and are dimension-dependent. They also struggle to compare candidates with&#10;close performance and become more unreliable with higher-quality summaries by&#10;obtaining a lower correlation with humans. In other words, with better&#10;abstractive summarization systems being introduced at a fast pace, LLMs may&#10;result in misleading and unreliable evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.07281" label="2203.07281">
        <attvalues>
          <attvalue for="0" value="GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large&#10;  Language Models" />
          <attvalue for="1" value="  Providing natural language instructions in prompts is a useful new paradigm&#10;for improving task performance of large language models in a zero-shot setting.&#10;Recent work has aimed to improve such prompts via manual rewriting or&#10;gradient-based tuning. However, manual rewriting is time-consuming and requires&#10;subjective interpretation, while gradient-based tuning can be extremely&#10;computationally demanding for large models and may not be feasible for&#10;API-based models. In this work, we introduce Gradient-free Instructional Prompt&#10;Search (GrIPS), a gradient-free, edit-based search approach for improving task&#10;instructions for large language models. GrIPS takes in instructions designed&#10;for humans and automatically returns an improved, edited prompt, while allowing&#10;for API-based tuning. With InstructGPT models, GrIPS improves the average task&#10;performance by up to 4.30 percentage points on eight classification tasks from&#10;the Natural Instructions dataset (with similar improvements for OPT, BLOOM, and&#10;FLAN-T5). We see improvements for both instruction-only prompts and instruction&#10;+ k-shot examples prompts. Notably, GrIPS outperforms manual rewriting and&#10;purely example-based prompts while controlling for the available compute and&#10;data budget. Further, performance of GrIPS is comparable to select&#10;gradient-based tuning approaches. Qualitatively, we show our edits can simplify&#10;instructions and at times make them incoherent but nonetheless improve&#10;accuracy. Our code is available at: https://github.com/archiki/GrIPS&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.03349" label="1610.03349">
        <attvalues>
          <attvalue for="0" value="Survey on the Use of Typological Information in Natural Language&#10;  Processing" />
          <attvalue for="1" value="  In recent years linguistic typology, which classifies the world's languages&#10;according to their functional and structural properties, has been widely used&#10;to support multilingual NLP. While the growing importance of typological&#10;information in supporting multilingual tasks has been recognised, no systematic&#10;survey of existing typological resources and their use in NLP has been&#10;published. This paper provides such a survey as well as discussion which we&#10;hope will both inform and inspire future work in the area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.07486" label="1612.07486">
        <attvalues>
          <attvalue for="0" value="Continuous multilinguality with language vectors" />
          <attvalue for="1" value="  Most existing models for multilingual natural language processing (NLP) treat&#10;language as a discrete category, and make predictions for either one language&#10;or the other. In contrast, we propose using continuous vector representations&#10;of language. We show that these can be learned efficiently with a&#10;character-based neural language model, and used to improve inference about&#10;language varieties not seen during training. In experiments with 1303 Bible&#10;translations into 990 different languages, we empirically explore the capacity&#10;of multilingual language models, and also show that the language vectors&#10;capture genetic relationships between languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.00914" label="1807.00914">
        <attvalues>
          <attvalue for="0" value="Modeling Language Variation and Universals: A Survey on Typological&#10;  Linguistics for Natural Language Processing" />
          <attvalue for="1" value="  Linguistic typology aims to capture structural and semantic variation across&#10;the world's languages. A large-scale typology could provide excellent guidance&#10;for multilingual Natural Language Processing (NLP), particularly for languages&#10;that suffer from the lack of human labeled resources. We present an extensive&#10;literature survey on the use of typological information in the development of&#10;NLP techniques. Our survey demonstrates that to date, the use of information in&#10;existing typological databases has resulted in consistent but modest&#10;improvements in system performance. We show that this is due to both intrinsic&#10;limitations of databases (in terms of coverage and feature granularity) and&#10;under-employment of the typological features included in them. We advocate for&#10;a new approach that adapts the broad and discrete nature of typological&#10;categories to the contextual and continuous nature of machine learning&#10;algorithms used in contemporary NLP. In particular, we suggest that such&#10;approach could be facilitated by recent developments in data-driven induction&#10;of typological knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.09055" label="1808.09055">
        <attvalues>
          <attvalue for="0" value="Parameter sharing between dependency parsers for related languages" />
          <attvalue for="1" value="  Previous work has suggested that parameter sharing between transition-based&#10;neural dependency parsers for related languages can lead to better performance,&#10;but there is no consensus on what parameters to share. We present an evaluation&#10;of 27 different parameter sharing strategies across 10 languages, representing&#10;five pairs of related languages, each pair from a different language family. We&#10;find that sharing transition classifier parameters always helps, whereas the&#10;usefulness of sharing word and/or character LSTM parameters varies. Based on&#10;this result, we propose an architecture where the transition classifier is&#10;shared, and the sharing of word and character parameters is controlled by a&#10;parameter that can be tuned on validation data. This model is linguistically&#10;motivated and obtains significant improvements over a monolingually trained&#10;baseline. We also find that sharing transition classifier parameters helps when&#10;training a parser on unrelated language pairs, but we find that, in the case of&#10;unrelated languages, sharing too many parameters does not help.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1409.7495" label="1409.7495">
        <attvalues>
          <attvalue for="0" value="Unsupervised Domain Adaptation by Backpropagation" />
          <attvalue for="1" value="  Top-performing deep architectures are trained on massive amounts of labeled&#10;data. In the absence of labeled data for a certain task, domain adaptation&#10;often provides an attractive option given that labeled data of similar nature&#10;but from a different domain (e.g. synthetic images) are available. Here, we&#10;propose a new approach to domain adaptation in deep architectures that can be&#10;trained on large amount of labeled data from the source domain and large amount&#10;of unlabeled data from the target domain (no labeled target-domain data is&#10;necessary).&#10;  As the training progresses, the approach promotes the emergence of &quot;deep&quot;&#10;features that are (i) discriminative for the main learning task on the source&#10;domain and (ii) invariant with respect to the shift between the domains. We&#10;show that this adaptation behaviour can be achieved in almost any feed-forward&#10;model by augmenting it with few standard layers and a simple new gradient&#10;reversal layer. The resulting augmented architecture can be trained using&#10;standard backpropagation.&#10;  Overall, the approach can be implemented with little effort using any of the&#10;deep-learning packages. The method performs very well in a series of image&#10;classification experiments, achieving adaptation effect in the presence of big&#10;domain shifts and outperforming previous state-of-the-art on Office datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.02720" label="1810.02720">
        <attvalues>
          <attvalue for="0" value="TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic&#10;  Parsing and Code Generation" />
          <attvalue for="1" value="  We present TRANX, a transition-based neural semantic parser that maps natural&#10;language (NL) utterances into formal meaning representations (MRs). TRANX uses&#10;a transition system based on the abstract syntax description language for the&#10;target MR, which gives it two major advantages: (1) it is highly accurate,&#10;using information from the syntax of the target MR to constrain the output&#10;space and model the information flow, and (2) it is highly generalizable, and&#10;can easily be applied to new types of MR by just writing a new abstract syntax&#10;description corresponding to the allowable structures in the MR. Experiments on&#10;four different semantic parsing and code generation tasks show that our system&#10;is generalizable, extensible, and effective, registering strong results&#10;compared to existing neural semantic parsers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.04942" label="1911.04942">
        <attvalues>
          <attvalue for="0" value="RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL&#10;  Parsers" />
          <attvalue for="1" value="  When translating natural language questions into SQL queries to answer&#10;questions from a database, contemporary semantic parsing models struggle to&#10;generalize to unseen database schemas. The generalization challenge lies in (a)&#10;encoding the database relations in an accessible way for the semantic parser,&#10;and (b) modeling alignment between database columns and their mentions in a&#10;given query. We present a unified framework, based on the relation-aware&#10;self-attention mechanism, to address schema encoding, schema linking, and&#10;feature representation within a text-to-SQL encoder. On the challenging Spider&#10;dataset this framework boosts the exact match accuracy to 57.2%, surpassing its&#10;best counterparts by 8.7% absolute improvement. Further augmented with BERT, it&#10;achieves the new state-of-the-art performance of 65.6% on the Spider&#10;leaderboard. In addition, we observe qualitative improvements in the model's&#10;understanding of schema linking and alignment. Our implementation will be&#10;open-sourced at https://github.com/Microsoft/rat-sql.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.02449" label="2406.02449">
        <attvalues>
          <attvalue for="0" value="Representations as Language: An Information-Theoretic Framework for&#10;  Interpretability" />
          <attvalue for="1" value="  Large scale neural models show impressive performance across a wide array of&#10;linguistic tasks. Despite this they remain, largely, black-boxes - inducing&#10;vector-representations of their input that prove difficult to interpret. This&#10;limits our ability to understand what they learn, and when the learn it, or&#10;describe what kinds of representations generalise well out of distribution. To&#10;address this we introduce a novel approach to interpretability that looks at&#10;the mapping a model learns from sentences to representations as a kind of&#10;language in its own right. In doing so we introduce a set of&#10;information-theoretic measures that quantify how structured a model's&#10;representations are with respect to its input, and when during training that&#10;structure arises. Our measures are fast to compute, grounded in linguistic&#10;theory, and can predict which models will generalise best based on their&#10;representations. We use these measures to describe two distinct phases of&#10;training a transformer: an initial phase of in-distribution learning which&#10;reduces task loss, then a second stage where representations becoming robust to&#10;noise. Generalisation performance begins to increase during this second phase,&#10;drawing a link between generalisation and robustness to noise. Finally we look&#10;at how model size affects the structure of the representational space, showing&#10;that larger models ultimately compress their representations more than their&#10;smaller counterparts.&#10;" />
          <attvalue for="2" value="&#10;&#10;Deep-Learning models achieve remarkable performance across a broad range of natural-language tasks \cite{vaswani_attention_2017}, but we still have a limited understanding of the learning process they undertake, and how they come to represent information so effectively. This is in part because these models are black-boxes \cite{tishby_deep_2015, shwartz-ziv_opening_2017}. They learn representations of their training data that are high-dimensional vectors, gigantic lists of numbers that are hard to interpret. While there is a growing body of work on interpretability, offering techniques for predicting what is encoded in a model's representations \cite{voita_information-theoretic_2020, pimentel2020information}, there's still lack of clarity about how representations themselves are structured, how that structure emerges, and what kinds of structures are desirable. &#10;&#10;Central to language's ability to generalise is its regularity, exemplified by syntactic structure \cite{partee1995lexical}, which allows predictable \&amp; regular encoding of meanings across the entire system. Languages are also rich with variation which can make them more expressive \cite{hurford2003synonymy} and structured ambiguities that can make them more compressible \cite{piantadosi_communicative_2012}.&#10;Do the representations learned by a transformer model \cite{vaswani_attention_2017} exhibit similar system level-structures? To answer this we look at the representations that emerge over the course of training as a kind of language in their own right. At a high-level we can think of language as a mapping between spaces, like between meaning and form \cite{saussure_course_1916}. A multi-layered neural model needs to learn to map a sentence to a vector representation that later layers can successfully map to the output; encoder-decoder models \cite{cho2014learning} even more explicitly use separate parts of a model to map in and out of vector space. We draw an analogy between these two mappings, quantifying different kinds of regularity and variation in a model's mapping between inputs and representations. While there has long been interest in the kinds of representations learned by deep-models \cite{bengio2013representation, locatello2019challenging}, there has been little work quantifying systematic structure in the representations learned by transformers or relating them to the kinds of structures that characterise natural language. It's worth noting our approach is in contrast to some existing work that draws parallels between model weights and formal languages \cite&lt;trying to infer functions or 'source code' from model weights; &gt;{elhage2021mathematical} --- we think an approach grounded in natural language is more scalable and better suited to characterising the kinds of systematic structure \&amp; variation that emerge in deep-learning models, especially those trained on data from natural languages. &#10;&#10;We introduce a novel information-theoretic framework for assessing whether the representations learned by a model are systematic. In order to do this we first discretise vector representations into a sequence of symbols, then quantify 4 properties of the learned mapping from sentences to symbols: the degree of compression, regularity, variation and disentanglement. By doing this at different levels of abstraction we show when lexical and syntactic information are learned. We can identify two clear phases of training, the first characterised by the model rapidly learning to disentangle and align representations with token and part of speech information, the second (far longer) phase of training characterised by representations becoming more robust to noise. During this second phase models compress their representations, with larger models compressing considerably more; at the same time, generalisation performance begins to slowly improve, showing a link between robustness to noise and generalisation. Finally we discuss what kinds of representational structure are desirable, using our measures to predict which models will perform best on a generalisation set.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Neural Model Interpretability, Language Representation Learning, Mathematics, Deep Learning Generalization" />
        </attvalues>
      </node>
      <node id="1503.02406" label="1503.02406">
        <attvalues>
          <attvalue for="0" value="Deep Learning and the Information Bottleneck Principle" />
          <attvalue for="1" value="  Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the&#10;information bottleneck (IB) principle. We first show that any DNN can be&#10;quantified by the mutual information between the layers and the input and&#10;output variables. Using this representation we can calculate the optimal&#10;information theoretic limits of the DNN and obtain finite sample generalization&#10;bounds. The advantage of getting closer to the theoretical limit is&#10;quantifiable both by the generalization bound and by the network's simplicity.&#10;We argue that both the optimal architecture, number of layers and&#10;features/connections at each layer, are related to the bifurcation points of&#10;the information bottleneck tradeoff, namely, relevant compression of the input&#10;layer with respect to the output layer. The hierarchical representations at the&#10;layered network naturally correspond to the structural phase transitions along&#10;the information curve. We believe that this new insight can lead to new&#10;optimality bounds and deep learning algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.12298" label="2003.12298">
        <attvalues>
          <attvalue for="0" value="Information-Theoretic Probing with Minimum Description Length" />
          <attvalue for="1" value="  To measure how well pretrained representations encode some linguistic&#10;property, it is common to use accuracy of a probe, i.e. a classifier trained to&#10;predict the property from the representations. Despite widespread adoption of&#10;probes, differences in their accuracy fail to adequately reflect differences in&#10;representations. For example, they do not substantially favour pretrained&#10;representations over randomly initialized ones. Analogously, their accuracy can&#10;be similar when probing for genuine linguistic labels and probing for random&#10;synthetic tasks. To see reasonable differences in accuracy with respect to&#10;these random baselines, previous work had to constrain either the amount of&#10;probe training data or its model size. Instead, we propose an alternative to&#10;the standard probes, information-theoretic probing with minimum description&#10;length (MDL). With MDL probing, training a probe to predict labels is recast as&#10;teaching it to effectively transmit the data. Therefore, the measure of&#10;interest changes from probe accuracy to the description length of labels given&#10;representations. In addition to probe quality, the description length evaluates&#10;&quot;the amount of effort&quot; needed to achieve the quality. This amount of effort&#10;characterizes either (i) size of a probing model, or (ii) the amount of data&#10;needed to achieve the high quality. We consider two methods for estimating MDL&#10;which can be easily implemented on top of the standard probing pipelines:&#10;variational coding and online coding. We show that these methods agree in&#10;results and are more informative and stable than the standard probes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03913" label="1911.03913">
        <attvalues>
          <attvalue for="0" value="Can Monolingual Pretrained Models Help Cross-Lingual Classification?" />
          <attvalue for="1" value="  Multilingual pretrained language models (such as multilingual BERT) have&#10;achieved impressive results for cross-lingual transfer. However, due to the&#10;constant model capacity, multilingual pre-training usually lags behind the&#10;monolingual competitors. In this work, we present two approaches to improve&#10;zero-shot cross-lingual classification, by transferring the knowledge from&#10;monolingual pretrained models to multilingual ones. Experimental results on two&#10;cross-lingual classification benchmarks show that our methods outperform&#10;vanilla multilingual fine-tuning.&#10;" />
          <attvalue for="2" value="&#10;&#10;Supervised text classification heavily relies on manually annotated training data, while the data are usually only available in rich-resource languages, such as English. &#10;It requires great effort to make the resources available in other languages.&#10;Various methods have been proposed to build cross-lingual classification models by exploiting machine translation systems~\cite{cldc,chen2018adversarial,xnli}, and learning multilingual embeddings~\cite{xnli,yu2018multilingual,clse-artetxet,eisenschlos2019multifit}.&#10;&#10;Recently, multilingual pretrained language models have shown surprising cross-lingual effectiveness on a wide range of downstream tasks~\cite{bert,xlm,xnlg,xlmr}.&#10;Even without using any parallel corpora, the pretrained models can still perform zero-shot cross-lingual classification~\cite{pires2019multilingual,wu2019beto,keung2019adversarial}.&#10;That is, these models can be fine-tuned in a source language, and then directly evaluated in other target languages.&#10;Despite the effectiveness of cross-lingual transfer, the multilingual pretrained language models have their own drawbacks.&#10;Due to the constant number of model parameters, the model capacity of the rich-resource languages decreases if we adds languages for pre-training.&#10;The curse of multilinguality results in that the multilingual models usually perform worse than their monolingual competitors on downstream tasks~\cite{nmt:wild19,xlmr}.&#10;The observations motivate us to leverage monolingual pretrained models to improve multilingual models for cross-lingual classification.&#10;&#10;In this paper, we propose a multilingual fine-tuning method (\our) based on the teacher-student framework, where a multilingual student model learns end task skills from a monolingual teacher.&#10;Intuitively, monolingual pretrained models are used to provide supervision of downstream tasks, while multilingual models are employed for knowledge transfer across languages.&#10;We conduct experiments on two widely used cross-lingual classification datasets,&#10;where our methods outperform baseline models on zero-shot cross-lingual classification.&#10;Moreover, we show that the monolingual teacher model can help the student multilingual model for both the source language and target languages, even though the student model is only trained in the source language.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Multilingual Classification, Linguistics, Artificial Intelligence, Language Model Transfer, Cross-Lingual Learning" />
        </attvalues>
      </node>
      <node id="1705.02073" label="1705.02073">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Distillation for Text Classification" />
          <attvalue for="1" value="  Cross-lingual text classification(CLTC) is the task of classifying documents&#10;written in different languages into the same taxonomy of categories. This paper&#10;presents a novel approach to CLTC that builds on model distillation, which&#10;adapts and extends a framework originally proposed for model compression. Using&#10;soft probabilistic predictions for the documents in a label-rich language as&#10;the (induced) supervisory labels in a parallel corpus of documents, we train&#10;classifiers successfully for new languages in which labeled training data are&#10;not available. An adversarial feature adaptation technique is also applied&#10;during the model training to reduce distribution mismatch. We conducted&#10;experiments on two benchmark CLTC datasets, treating English as the source&#10;language and German, French, Japan and Chinese as the unlabeled target&#10;languages. The proposed approach had the advantageous or comparable performance&#10;of the other state-of-art methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.02782" label="2402.02782">
        <attvalues>
          <attvalue for="0" value="From Partial to Strictly Incremental Constituent Parsing" />
          <attvalue for="1" value="  We study incremental constituent parsers to assess their capacity to output&#10;trees based on prefix representations alone. Guided by strictly left-to-right&#10;generative language models and tree-decoding modules, we build parsers that&#10;adhere to a strong definition of incrementality across languages. This builds&#10;upon work that asserted incrementality, but that mostly only enforced it on&#10;either the encoder or the decoder. Finally, we conduct an analysis against&#10;non-incremental and partially incremental models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Incremental NLP aims to learn and adapt partial representations as information unfolds. &#10;However, with the rise of bidirectional LSTMs \cite{hochreiter1997long} and Transformers \cite{vaswani2017attention}, &#10;recent research has focused on non-incremental solutions.&#10;These models process the full input for contextualization before they start generating any output. Therefore, this approach does not capture the progressive unfolding of input over time, giving the sense that all of it is available all of a sudden \cite{madureira-schlangen-2020-incremental}. This is not an issue for most NLP tasks, but it is relevant for others, such as real-time NLP, e.g., instant machine translation or real-time speech. Furthermore, work on incremental processing holds relevance in interdisciplinary research, especially where computer science, linguistics, and cognitive studies intersect.&#10;&#10;While some studies have addressed the challenge of outputting incremental structured representations - for various definitions of incrementality \cite{konstas-etal-2014-incremental,kohn-2018-incremental,shen-etal-2021-explicitly} - analyses of trees remain limited, more notably since the popularization of deep learning, and are mostly partially incremental approaches.&#10;&#10;In this context, \cite{titov-henderson-2007-constituent}, one of the first neural parsing models, was also an incremental network based on sigmoid belief networks. This generative model broke down the probability of a structure into probabilities for individual derivation decisions,&#10;each influenced by previous decision history. However, the computation was expensive and its evaluation was restricted to sentences of up to 15 tokens in the English Penn Treebank \cite{marcus1993building}. For shift-reduce constituent parsing, \cite{cross-huang-2016-incremental} proposed an incremental model with minimal features, focusing on only three sentence positions to predict the next action. However, input sentences were contextualized using bidirectional LSTMs, thus relying on non-incremental encoders and effectively considering all upcoming words; a strategy that was later widely adopted by most neural syntactic parsing architectures, but that does not adhere to a definition of strong incrementality. More recently, \cite{kitaev-etal-2022-learned} introduced a span-based model that incrementally encodes input sentences into discrete elements using vectors from GPT-2 mapped into a codebook. Despite this, it relied on bidirectional Transformers and a CYK architecture \cite{kitaev2018constituency} for decoding these vectors into trees. Complementarily, \cite{yang2020strongly} proposed an incremental decoder based on graph neural networks. Although they referred to their parser as strongly incremental, sentences were encoded with bidirectional architectures like BERT or XLNET \cite{devlin-etal-2019-bert,yang2019xlnet}.&#10;&#10;Incrementality has been also explored for other parsing formalisms. \cite{stanojevic-steedman-2019-ccg} developed an almost fully incremental parser for combinatory categorical grammars (CCG), relying on ELMo embeddings \cite{peters-etal-2018-deep} and a bidirectional LSTM for these predictions. Later, a genuinely fully incremental CCG parser was introduced \cite{stanojevic-steedman-2020-max}, using only ELMo's forward pass and a left-to-right LSTM, addressing biases in incremental CCG parsing. In the field of dependency parsing, incrementality has been a focus since the pre-neural era \cite{beuck2013structural,kohn-menzel-2014-incremental,kohn-baumann-2016-predictive}, with some models rivaling non-incremental ones. Recently, \cite{ezquerro2023challenges} found that with current neural architectures, incremental models for dependency parsing are less effective than bidirectional approaches. However, incorporating human-like reading strategies, such as brief delays, can significantly enhance performance, particularly in languages with leftward dependencies.&#10;&#10;Contribution &#10;We study the viability and challenges of fully incremental constituent parsing with encoder-decoder architectures. All components strictly process the sentence from left to right, adding each read word to the partial tree based on the input prefix. For the encoder, we leverage generative LLMs. For the decoder, we reassess two options that generate partial trees based solely on current inputs: (i) an incremental parsing-as-tagging model \cite{gomez-rodriguez-vilares-2018-constituent}, and (ii) a transition-based decoder that uses graph-neural-network representations \cite{yang2020strongly}. The code is available at .&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Models, Incremental Parsing, Mathematics, Tree Decoding" />
        </attvalues>
      </node>
      <node id="2010.05330" label="2010.05330">
        <attvalues>
          <attvalue for="0" value="Incremental Processing in the Age of Non-Incremental Encoders: An&#10;  Empirical Assessment of Bidirectional Models for Incremental NLU" />
          <attvalue for="1" value="  While humans process language incrementally, the best language encoders&#10;currently used in NLP do not. Both bidirectional LSTMs and Transformers assume&#10;that the sequence that is to be encoded is available in full, to be processed&#10;either forwards and backwards (BiLSTMs) or as a whole (Transformers). We&#10;investigate how they behave under incremental interfaces, when partial output&#10;must be provided based on partial input seen up to a certain time step, which&#10;may happen in interactive systems. We test five models on various NLU datasets&#10;and compare their performance using three incremental evaluation metrics. The&#10;results support the possibility of using bidirectional encoders in incremental&#10;mode while retaining most of their non-incremental quality. The&#10;&quot;omni-directional&quot; BERT model, which achieves better non-incremental&#10;performance, is impacted more by the incremental access. This can be alleviated&#10;by adapting the training regime (truncated training), or the testing procedure,&#10;by delaying the output until some right context is available or by&#10;incorporating hypothetical right contexts generated by a language model like&#10;GPT-2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.12518" label="1805.12518">
        <attvalues>
          <attvalue for="0" value="Incremental Natural Language Processing: Challenges, Strategies, and&#10;  Evaluation" />
          <attvalue for="1" value="  Incrementality is ubiquitous in human-human interaction and beneficial for&#10;human-computer interaction. It has been a topic of research in different parts&#10;of the NLP community, mostly with focus on the specific topic at hand even&#10;though incremental systems have to deal with similar challenges regardless of&#10;domain. In this survey, I consolidate and categorize the approaches,&#10;identifying similarities and differences in the computation and data, and show&#10;trade-offs that have to be considered. A focus lies on evaluating incremental&#10;systems because the standard metrics often fail to capture the incremental&#10;properties of a system and coming up with a suitable evaluation scheme is&#10;non-trivial.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.07960" label="2011.07960">
        <attvalues>
          <attvalue for="0" value="Explicitly Modeling Syntax in Language Models with Incremental Parsing&#10;  and a Dynamic Oracle" />
          <attvalue for="1" value="  Syntax is fundamental to our thinking about language. Failing to capture the&#10;structure of input language could lead to generalization problems and&#10;over-parametrization. In the present work, we propose a new syntax-aware&#10;language model: Syntactic Ordered Memory (SOM). The model explicitly models the&#10;structure with an incremental parser and maintains the conditional probability&#10;setting of a standard language model (left-to-right). To train the incremental&#10;parser and avoid exposure bias, we also propose a novel dynamic oracle, so that&#10;SOM is more robust to wrong parsing decisions. Experiments show that SOM can&#10;achieve strong results in language modeling, incremental parsing and syntactic&#10;generalization tests, while using fewer parameters than other models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.06406" label="1606.06406">
        <attvalues>
          <attvalue for="0" value="Incremental Parsing with Minimal Features Using Bi-Directional LSTM" />
          <attvalue for="1" value="  Recently, neural network approaches for parsing have largely automated the&#10;combination of individual features, but still rely on (often a larger number&#10;of) atomic features created from human linguistic intuition, and potentially&#10;omitting important global context. To further reduce feature engineering to the&#10;bare minimum, we use bi-directional LSTM sentence representations to model a&#10;parser state with only three sentence positions, which automatically identifies&#10;important aspects of the entire sentence. This model achieves state-of-the-art&#10;results among greedy dependency parsers for English. We also introduce a novel&#10;transition system for constituency parsing which does not require binarization,&#10;and together with the above architecture, achieves state-of-the-art results&#10;among greedy parsers for both English and Chinese.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.01052" label="1805.01052">
        <attvalues>
          <attvalue for="0" value="Constituency Parsing with a Self-Attentive Encoder" />
          <attvalue for="1" value="  We demonstrate that replacing an LSTM encoder with a self-attentive&#10;architecture can lead to improvements to a state-of-the-art discriminative&#10;constituency parser. The use of attention makes explicit the manner in which&#10;information is propagated between different locations in the sentence, which we&#10;use to both analyze our model and propose potential improvements. For example,&#10;we find that separating positional and content information in the encoder can&#10;lead to improved parsing accuracy. Additionally, we evaluate different&#10;approaches for lexical representation. Our parser achieves new state-of-the-art&#10;results for single models trained on the Penn Treebank: 93.55 F1 without the&#10;use of any external data, and 95.13 F1 when using pre-trained word&#10;representations. Our parser also outperforms the previous best-published&#10;accuracy figures on 8 of the 9 languages in the SPMRL dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14568" label="2010.14568">
        <attvalues>
          <attvalue for="0" value="Strongly Incremental Constituency Parsing with Graph Neural Networks" />
          <attvalue for="1" value="  Parsing sentences into syntax trees can benefit downstream applications in&#10;NLP. Transition-based parsers build trees by executing actions in a state&#10;transition system. They are computationally efficient, and can leverage machine&#10;learning to predict actions based on partial trees. However, existing&#10;transition-based parsers are predominantly based on the shift-reduce transition&#10;system, which does not align with how humans are known to parse sentences.&#10;Psycholinguistic research suggests that human parsing is strongly incremental:&#10;humans grow a single parse tree by adding exactly one token at each step. In&#10;this paper, we propose a novel transition system called attach-juxtapose. It is&#10;strongly incremental; it represents a partial sentence using a single tree;&#10;each action adds exactly one token into the partial tree. Based on our&#10;transition system, we develop a strongly incremental parser. At each step, it&#10;encodes the partial tree using a graph neural network and predicts an action.&#10;We evaluate our parser on Penn Treebank (PTB) and Chinese Treebank (CTB). On&#10;PTB, it outperforms existing parsers trained with only constituency trees; and&#10;it performs on par with state-of-the-art parsers that use dependency trees as&#10;additional training data. On CTB, our parser establishes a new state of the&#10;art. Code is available at&#10;https://github.com/princeton-vl/attach-juxtapose-parser.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.08994" label="1810.08994">
        <attvalues>
          <attvalue for="0" value="Constituent Parsing as Sequence Labeling" />
          <attvalue for="1" value="  We introduce a method to reduce constituent parsing to sequence labeling. For&#10;each word w_t, it generates a label that encodes: (1) the number of ancestors&#10;in the tree that the words w_t and w_{t+1} have in common, and (2) the&#10;nonterminal symbol at the lowest common ancestor. We first prove that the&#10;proposed encoding function is injective for any tree without unary branches. In&#10;practice, the approach is made extensible to all constituency trees by&#10;collapsing unary branches. We then use the PTB and CTB treebanks as testbeds&#10;and propose a set of fast baselines. We achieve 90.7% F-score on the PTB test&#10;set, outperforming the Vinyals et al. (2015) sequence-to-sequence parser. In&#10;addition, sacrificing some accuracy, our approach achieves the fastest&#10;constituent parsing speeds reported to date on PTB by a wide margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.08833" label="2403.08833">
        <attvalues>
          <attvalue for="0" value="TINA: Think, Interaction, and Action Framework for Zero-Shot Vision&#10;  Language Navigation" />
          <attvalue for="1" value="  Zero-shot navigation is a critical challenge in Vision-Language Navigation&#10;(VLN) tasks, where the ability to adapt to unfamiliar instructions and to act&#10;in unknown environments is essential. Existing supervised learning-based&#10;models, trained using annotated data through reinforcement learning, exhibit&#10;limitations in generalization capabilities. Large Language Models (LLMs), with&#10;their extensive knowledge and emergent reasoning abilities, present a potential&#10;pathway for achieving zero-shot navigation. This paper presents a VLN agent&#10;based on LLMs, exploring approaches to the zero-shot navigation problem. To&#10;compensate for the shortcomings of LLMs in environmental perception, we propose&#10;the Thinking, Interacting, and Action (TINA) framework. TINA enables the agent&#10;to scrutinize perceptual information and autonomously query key clues within&#10;the environment through an introduced question-answering module, thereby&#10;aligning instructions with specific perceptual data. The navigation agent's&#10;perceptual abilities are enhanced through the TINA framework, while the&#10;explicit thought and query processes also improve the navigational procedure's&#10;explainability and transparency. We evaluate the performance of our method on&#10;the Room-to-Room dataset. The experiment results indicate that our approach&#10;improves the navigation performance of LLM-based agents. Our approach also&#10;outperformed some supervised learning-based methods, highlighting its efficacy&#10;in zero-shot navigation.&#10;" />
          <attvalue for="2" value="&#10;&#10;The advancement of computer vision and natural language processing has facilitated research in vision-language fusion, such as Visual Question Answering\cite{antol2015vqa} (VQA), Image Caption\cite{xu2015show}, and Vision-Language Navigation\cite{anderson2018vision} (VLN). In VLN tasks, agents must navigate through diverse environments based on natural language instructions, necessitating multifaceted expertise in linguistic semantics, visual perception, and dynamic decision-making (Figure \ref{fig:vln}). While supervised deep learning has propelled the development of VLN models, existing models exhibit limitations in generalization and decision-making transparency\cite{hong2021vln, lin2022adapt, fried2018speaker, tan2019learning, anderson2019chasing}. They still lack the zero-shot ability\cite{romera2015embarrassingly} to interpret unfamiliar instructions and navigate unseen environments, which limits the development of these models for broader applications.&#10;&#10;Large Language Models (LLMs) have recently received considerable attention due to their remarkable language generation capabilities and extensive knowledge\cite{touvron2302llama,wei2022chain}. Research indicates that as LLMs scale, they demonstrate emergent abilities that expand their applicability across various domains, including reasoning and decision-making tasks\cite{wei2022emergent}. Recent research has revealed that, despite being trained, current VLN models still encounter difficulties when dealing with diverse instructions\cite{zhang2023vln}. For the VLN task, LLMs hold promise for enabling the zero-shot capability\cite{kojima2022large,chen20232,liang2023mo}. They can leverage their extensive knowledge and common sense reasoning to interpret and decompose new instructions and reason according to the environment, thus enabling navigation agents to adapt to unseen environments without prior training. However, a challenge arises in developing LLM-based VLN systems due to LLMs' limited visual perceptual capabilities\cite{berrios2023towards}, primarily trained on textual data. Some efforts introduced visual perceptual modules, converting visual information into textual descriptions for analysis by LLMs\cite{liu2023internchat, zhu2023minigpt,zhou2023navgpt}. While these approaches give LLM-based agents a rudimentary understanding of visual content, descriptions often remain generalized and lack specificity. In cases where visual perception needs to align with specific instructions, these visual descriptions might omit crucial information. &#10;&#10;To enhance the perceptual capabilities of LLMs for specific targets, we propose the TINA framework — Think, Interaction, and Action — which endows agents with the ability to scrutinize perceptual outcomes and autonomously query specific clues. Our framework comprises the core LLM agent and three additional modules: the Visual Perception (VP) module, the Question-Answer Interaction (QAI) module, and the Trajectory Memorizer (TM) module. The VP module generates rough environmental descriptions, prompting the agent to reason based on these descriptions in response to specific perceptual demands in instructions. The framework enables the agent to review the outputs of the VP and make targeted queries through the QAI module to supplement missing perceptual information. TINA extends the agent's perception through the QAI module, aligning instructions with the environment. Additionally, our framework includes a memory bank that stores the agent's actions in each round, enhancing its dynamic adaptation capabilities while filtering out redundant historical information. In the subsequent sections, we will provide a detailed description of our TINA framework for zero-shot VLN, along with insights from our experimental results and findings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Zero-Shot Learning, Computer Vision, Vision Language Navigation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1505.00468" label="1505.00468">
        <attvalues>
          <attvalue for="0" value="VQA: Visual Question Answering" />
          <attvalue for="1" value="  We propose the task of free-form and open-ended Visual Question Answering&#10;(VQA). Given an image and a natural language question about the image, the task&#10;is to provide an accurate natural language answer. Mirroring real-world&#10;scenarios, such as helping the visually impaired, both the questions and&#10;answers are open-ended. Visual questions selectively target different areas of&#10;an image, including background details and underlying context. As a result, a&#10;system that succeeds at VQA typically needs a more detailed understanding of&#10;the image and complex reasoning than a system producing generic image captions.&#10;Moreover, VQA is amenable to automatic evaluation, since many open-ended&#10;answers contain only a few words or a closed set of answers that can be&#10;provided in a multiple-choice format. We provide a dataset containing ~0.25M&#10;images, ~0.76M questions, and ~10M answers (www.visualqa.org), and discuss the&#10;information it provides. Numerous baselines and methods for VQA are provided&#10;and compared with human performance. Our VQA demo is available on CloudCV&#10;(http://cloudcv.org/vqa).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1502.03044" label="1502.03044">
        <attvalues>
          <attvalue for="0" value="Show, Attend and Tell: Neural Image Caption Generation with Visual&#10;  Attention" />
          <attvalue for="1" value="  Inspired by recent work in machine translation and object detection, we&#10;introduce an attention based model that automatically learns to describe the&#10;content of images. We describe how we can train this model in a deterministic&#10;manner using standard backpropagation techniques and stochastically by&#10;maximizing a variational lower bound. We also show through visualization how&#10;the model is able to automatically learn to fix its gaze on salient objects&#10;while generating the corresponding words in the output sequence. We validate&#10;the use of attention with state-of-the-art performance on three benchmark&#10;datasets: Flickr8k, Flickr30k and MS COCO.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.09230" label="2302.09230">
        <attvalues>
          <attvalue for="0" value="VLN-Trans: Translator for the Vision and Language Navigation Agent" />
          <attvalue for="1" value="  Language understanding is essential for the navigation agent to follow&#10;instructions. We observe two kinds of issues in the instructions that can make&#10;the navigation task challenging: 1. The mentioned landmarks are not&#10;recognizable by the navigation agent due to the different vision abilities of&#10;the instructor and the modeled agent. 2. The mentioned landmarks are applicable&#10;to multiple targets, thus not distinctive for selecting the target among the&#10;candidate viewpoints. To deal with these issues, we design a translator module&#10;for the navigation agent to convert the original instructions into&#10;easy-to-follow sub-instruction representations at each step. The translator&#10;needs to focus on the recognizable and distinctive landmarks based on the&#10;agent's visual abilities and the observed visual environment. To achieve this&#10;goal, we create a new synthetic sub-instruction dataset and design specific&#10;tasks to train the translator and the navigation agent. We evaluate our&#10;approach on Room2Room~(R2R), Room4room~(R4R), and Room2Room Last (R2R-Last)&#10;datasets and achieve state-of-the-art results on multiple benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.16410" label="2306.16410">
        <attvalues>
          <attvalue for="0" value="Towards Language Models That Can See: Computer Vision Through the LENS&#10;  of Natural Language" />
          <attvalue for="1" value="  We propose LENS, a modular approach for tackling computer vision problems by&#10;leveraging the power of large language models (LLMs). Our system uses a&#10;language model to reason over outputs from a set of independent and highly&#10;descriptive vision modules that provide exhaustive information about an image.&#10;We evaluate the approach on pure computer vision settings such as zero- and&#10;few-shot object recognition, as well as on vision and language problems. LENS&#10;can be applied to any off-the-shelf LLM and we find that the LLMs with LENS&#10;perform highly competitively with much bigger and much more sophisticated&#10;systems, without any multimodal training whatsoever. We open-source our code at&#10;https://github.com/ContextualAI/lens and provide an interactive demo.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16986" label="2305.16986">
        <attvalues>
          <attvalue for="0" value="NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large&#10;  Language Models" />
          <attvalue for="1" value="  Trained with an unprecedented scale of data, large language models (LLMs)&#10;like ChatGPT and GPT-4 exhibit the emergence of significant reasoning abilities&#10;from model scaling. Such a trend underscored the potential of training LLMs&#10;with unlimited language data, advancing the development of a universal embodied&#10;agent. In this work, we introduce the NavGPT, a purely LLM-based&#10;instruction-following navigation agent, to reveal the reasoning capability of&#10;GPT models in complex embodied scenes by performing zero-shot sequential action&#10;prediction for vision-and-language navigation (VLN). At each step, NavGPT takes&#10;the textual descriptions of visual observations, navigation history, and future&#10;explorable directions as inputs to reason the agent's current status, and makes&#10;the decision to approach the target. Through comprehensive experiments, we&#10;demonstrate NavGPT can explicitly perform high-level planning for navigation,&#10;including decomposing instruction into sub-goal, integrating commonsense&#10;knowledge relevant to navigation task resolution, identifying landmarks from&#10;observed scenes, tracking navigation progress, and adapting to exceptions with&#10;plan adjustment. Furthermore, we show that LLMs is capable of generating&#10;high-quality navigational instructions from observations and actions along a&#10;path, as well as drawing accurate top-down metric trajectory given the agent's&#10;navigation history. Despite the performance of using NavGPT to zero-shot R2R&#10;tasks still falling short of trained models, we suggest adapting multi-modality&#10;inputs for LLMs to use as visual navigation agents and applying the explicit&#10;reasoning of LLMs to benefit learning-based models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.04791" label="1811.04791">
        <attvalues>
          <attvalue for="0" value="Multilingual and Unsupervised Subword Modeling for Zero-Resource&#10;  Languages" />
          <attvalue for="1" value="  Subword modeling for zero-resource languages aims to learn low-level&#10;representations of speech audio without using transcriptions or other resources&#10;from the target language (such as text corpora or pronunciation dictionaries).&#10;A good representation should capture phonetic content and abstract away from&#10;other types of variability, such as speaker differences and channel noise.&#10;Previous work in this area has primarily focused unsupervised learning from&#10;target language data only, and has been evaluated only intrinsically. Here we&#10;directly compare multiple methods, including some that use only target language&#10;speech data and some that use transcribed speech from other (non-target)&#10;languages, and we evaluate using two intrinsic measures as well as on a&#10;downstream unsupervised word segmentation and clustering task. We find that&#10;combining two existing target-language-only methods yields better features than&#10;either method alone. Nevertheless, even better results are obtained by&#10;extracting target language bottleneck features using a model trained on other&#10;languages. Cross-lingual training using just one other language is enough to&#10;provide this benefit, but multilingual training helps even more. In addition to&#10;these results, which hold across both intrinsic measures and the extrinsic&#10;task, we discuss the qualitative differences between the different types of&#10;learned features.&#10;" />
          <attvalue for="2" value="&#10;Recent years have seen increasing interest in &#10;speech technology for ``zero-resource'' languages, where systems must be developed for a target language without using&#10;transcribed data or other hand-curated resources from that language. Such&#10;systems could potentially be applied to tasks such as endangered language&#10;documentation or query-by-example search for languages without a written form.&#10;One challenge for these systems, highlighted by the zrsc shared tasks of&#10;2015 \cite{Versteegh2015} and 2017 \cite{Dunbar2017}, is to improve subword&#10;modeling, i.e., to extract or learn speech features from the target language&#10;audio. Good features should be more effective at discriminating between&#10;linguistic units, e.g. words or subwords, while abstracting away from factors&#10;such as speaker identity and channel noise.&#10;&#10;The ZRSCs were motivated largely by questions in artificial intelligence and&#10;human perceptual learning, and focused on approaches where no transcribed data&#10;from {any} language is used. Yet from an engineering perspective it also&#10;makes sense to explore how training data from higher-resource languages can be&#10;used to improve speech features in a zero-resource language.&#10;&#10;This paper explores several methods for improving subword modeling in&#10;zero-resource languages, either with or without the use of labeled data from&#10;other languages. Although the individual methods are not new, our work provides&#10;a much more thorough empirical evaluation of these methods compared to the&#10;existing literature. We experiment with each method both alone and in&#10;combinations not tried before, and provide results across a range of target&#10;languages, evaluation measures, and tasks.&#10;&#10;We start by evaluating two methods for feature extraction that are trained using&#10;(untranscribed) target language data only: traditional vtln and the&#10;cae proposed more recently by \cite{Kamper2015}. The cae learns to&#10;abstract away from signal noise and variability by training on pairs of speech&#10;segments extracted using an utd system---i.e., pairs that are likely to be&#10;instances of the same word or phrase. We confirm previous work showing that&#10;cae features outperform mfccs on a word discriminability task,&#10;although we also show that this benefit is not consistently better than that of&#10;simply applying vtln. More interestingly, however, we find that applying&#10;vtln to the input of the cae system improves the learned features&#10;considerably, leading to better performance than either method alone. These&#10;improvements indicate that cae and vtln abstract over different&#10;aspects of the signal, and suggest that vtln might also be a useful&#10;preprocessing step in other recent neural-network-based unsupervised&#10;feature-learning methods.&#10;&#10;Next, we explore how multilingual annotated data can be used to improve feature&#10;extraction for a zero-resource target language. We train multilingual bnfs&#10;on between one and ten languages from the GlobalPhone collection and evaluate on&#10;six other languages (simulating different zero-resource targets). We show that&#10;training on more languages consistently improves performance on word&#10;discrimination, and that the improvement is not simply due to more training&#10;data: an equivalent amount of data from one language fails to give the same&#10;benefit. In fact, we observe the largest gain in performance when adding the&#10;second training language, which is already better than adding three times as&#10;much data from the same language. Moreover, when compared to our best results&#10;from training unsupervised on target language data only, we find that bnfs&#10;trained on just a single other language already outperform the&#10;target-language-only training, with multilingual bnfs doing better by a&#10;wide margin.&#10;&#10;Although multilingual training outperforms unsupervised target-language&#10;training, it could still be possible to improve on the multilingual bnfs&#10;by using them as inputs for further target-language training. To test this&#10;hypothesis, we passed the multilingual bnfs as input to the&#10;cae. When trained with utd word pairs, we found no benefit to this&#10;method. However, training with manually labeled word pairs did yield&#10;benefits, suggesting that this type of supervision can help improve on the&#10;bnfs if the word pairs are sufficiently high-quality.&#10;&#10;The results above were presented as part of an earlier conference version of&#10;this paper \cite{Hermann2018}. Here, we expand upon that work in several ways.&#10;First, we include new results on the corpora and evaluation measures used in the&#10;zrsc, to allow more direct comparisons with other work. In doing so, we&#10;also provide the first set of results on identical systems evaluated using both&#10;the same-different and ABX evaluation measures. This permits the two measures&#10;themselves to be better compared. Finally, we provide both a qualitative&#10;analysis of the differences between the different features we extract, and a&#10;quantitative evaluation on the downstream target-language task of unsupervised&#10;full-coverage speech segmentation and clustering using the system of&#10;\cite{Kamper2017}. This is the first time that multilingual features are used in&#10;such a system, which performs a complete segmentation of input speech into&#10;hypothesized words. As in our intrinsic evaluations, we find that the&#10;multilingual bnfs consistently outperform the best unsupervised cae&#10;features, which in turn outperform or do similarly to MFCCs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Acoustics, Computer Science, Speech Processing, Linguistics, Multilingual Modeling, Speech Representation, Unsupervised Learning" />
        </attvalues>
      </node>
      <node id="1803.08863" label="1803.08863">
        <attvalues>
          <attvalue for="0" value="Multilingual bottleneck features for subword modeling in zero-resource&#10;  languages" />
          <attvalue for="1" value="  How can we effectively develop speech technology for languages where no&#10;transcribed data is available? Many existing approaches use no annotated&#10;resources at all, yet it makes sense to leverage information from large&#10;annotated corpora in other languages, for example in the form of multilingual&#10;bottleneck features (BNFs) obtained from a supervised speech recognition&#10;system. In this work, we evaluate the benefits of BNFs for subword modeling&#10;(feature extraction) in six unseen languages on a word discrimination task.&#10;First we establish a strong unsupervised baseline by combining two existing&#10;methods: vocal tract length normalisation (VTLN) and the correspondence&#10;autoencoder (cAE). We then show that BNFs trained on a single language already&#10;beat this baseline; including up to 10 languages results in additional&#10;improvements which cannot be matched by just adding more data from a single&#10;language. Finally, we show that the cAE can improve further on the BNFs if&#10;high-quality same-word pairs are available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.00883" label="1910.00883">
        <attvalues>
          <attvalue for="0" value="Exploiting BERT for End-to-End Aspect-based Sentiment Analysis" />
          <attvalue for="1" value="  In this paper, we investigate the modeling power of contextualized embeddings&#10;from pre-trained language models, e.g. BERT, on the E2E-ABSA task.&#10;Specifically, we build a series of simple yet insightful neural baselines to&#10;deal with E2E-ABSA. The experimental results show that even with a simple&#10;linear classification layer, our BERT-based architecture can outperform&#10;state-of-the-art works. Besides, we also standardize the comparative study by&#10;consistently utilizing a hold-out validation dataset for model selection, which&#10;is largely ignored by previous works. Therefore, our work can serve as a&#10;BERT-based benchmark for E2E-ABSA.&#10;" />
          <attvalue for="2" value="&#10;Aspect-based sentiment analysis (ABSA) is to discover the users' sentiment or opinion towards an aspect, usually in the form of explicitly mentioned aspect terms~\cite{mitchell-etal-2013-open,zhang-etal-2015-neural} or implicit aspect categories~\cite{wang-etal-2016-attention}, from user-generated natural language texts~\cite{liu2012sentiment}. The most popular ABSA benchmark datasets are from SemEval ABSA challenges~\cite{pontiki-etal-2014-semeval,pontiki-etal-2015-semeval,pontiki-etal-2016-semeval} where a few thousand review sentences with gold standard aspect sentiment annotations are provided. &#10;&#10;Table~\ref{tab:problem_settings} summarizes three existing research problems related to ABSA. The first one is the original ABSA, aiming at predicting the sentiment polarity of the sentence towards the given aspect. Compared to this classification problem, the second one and the third one, namely, Aspect-oriented Opinion Words Extraction (AOWE)~\cite{fan-etal-2019-target} and End-to-End Aspect-based Sentiment Analysis (E2E-ABSA)~\cite{ma-etal-2018-joint,schmitt-etal-2018-joint,li2019unified,li2017learning,li2019learning}, are related to a sequence tagging problem. Precisely, the goal of AOWE is to extract the aspect-specific opinion words from the sentence given the aspect. The goal of E2E-ABSA is to jointly detect aspect terms/categories and the corresponding aspect sentiments. &#10;&#10;Many neural models composed of a task-agnostic pre-trained word embedding layer and task-specific neural architecture have been proposed for the original ABSA task (i.e. the aspect-level sentiment classification)~\cite{tang-etal-2016-aspect,wang-etal-2016-attention,chen-etal-2017-recurrent-attention,liu-zhang-2017-attention,ma2017interactive,ma2018targeted,majumder-etal-2018-iarm,li-etal-2018-transformation,he-etal-2018-exploiting,xue-li-2018-aspect,wang-etal-2018-target,fan-etal-2018-multi,huang-carley-2018-parameterized,lei2019human,li2019exploiting,zhang2019aspect}, but the improvement of these models measured by the accuracy or F1 score has reached a bottleneck. One reason is that the task-agnostic embedding layer, usually a linear layer initialized with Word2Vec~\cite{mikolov2013distributed} or GloVe~\cite{pennington-etal-2014-glove}, only provides context-independent word-level features, which is insufficient for capturing the complex semantic dependencies in the sentence. Meanwhile, the size of existing datasets is too small to train sophisticated task-specific architectures. Thus, introducing a context-aware word embedding layer pre-trained on large-scale datasets with deep LSTM~\cite{mccann2017learned,peters-etal-2018-deep,howard-ruder-2018-universal} or Transformer~\cite{radford2018improving,radford2019language,devlin-etal-2019-bert,lample2019cross,yang2019xlnet,dong2019unified} for fine-tuning a lightweight task-specific network using the labeled data has good potential for further enhancing the performance. &#10;&#10;\cite{xu-etal-2019-bert,sun-etal-2019-utilizing,song2019attentional,yu2019adapting,rietzler2019adapt,huang2019syntax,hu2019learning} have conducted some initial attempts to couple the deep contextualized word embedding layer with downstream neural models for the original ABSA task and establish the new state-of-the-art results. It encourages us to explore the potential of using such contextualized embeddings to the more difficult but practical task, i.e. E2E-ABSA (the third setting in Table~\ref{tab:problem_settings}).&#10;Note that we are not aiming at developing a task-specific architecture, instead, our focus is to examine the potential of contextualized embedding for E2E-ABSA, coupled with various simple layers for prediction of E2E-ABSA labels.&#10;&#10;In this paper, we investigate the modeling power of BERT~\cite{devlin-etal-2019-bert}, one of the most popular pre-trained language model armed with Transformer~\cite{vaswani2017attention}, on the task of E2E-ABSA. Concretely, inspired by the investigation of E2E-ABSA in~\cite{li2019unified}, which predicts aspect boundaries as well as aspect sentiments using a single sequence tagger, we build a series of simple yet insightful neural baselines for the sequence labeling problem and fine-tune the task-specific components with BERT or deem BERT as feature extractor. Besides, we standardize the comparative study by consistently utilizing the hold-out development dataset for model selection, which is ignored in most of the existing ABSA works~\cite{tay2018learning}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Language Model Analysis, Linguistics, Artificial Intelligence, Sentiment Analysis Benchmarking" />
        </attvalues>
      </node>
      <node id="1811.05082" label="1811.05082">
        <attvalues>
          <attvalue for="0" value="A Unified Model for Opinion Target Extraction and Target Sentiment&#10;  Prediction" />
          <attvalue for="1" value="  Target-based sentiment analysis involves opinion target extraction and target&#10;sentiment classification. However, most of the existing works usually studied&#10;one of these two sub-tasks alone, which hinders their practical use. This paper&#10;aims to solve the complete task of target-based sentiment analysis in an&#10;end-to-end fashion, and presents a novel unified model which applies a unified&#10;tagging scheme. Our framework involves two stacked recurrent neural networks:&#10;The upper one predicts the unified tags to produce the final output results of&#10;the primary target-based sentiment analysis; The lower one performs an&#10;auxiliary target boundary prediction aiming at guiding the upper network to&#10;improve the performance of the primary task. To explore the inter-task&#10;dependency, we propose to explicitly model the constrained transitions from&#10;target boundaries to target sentiment polarities. We also propose to maintain&#10;the sentiment consistency within an opinion target via a gate mechanism which&#10;models the relation between the features for the current word and the previous&#10;word. We conduct extensive experiments on three benchmark datasets and our&#10;framework achieves consistently superior results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.07593" label="1909.07593">
        <attvalues>
          <attvalue for="0" value="Learning Explicit and Implicit Structures for Targeted Sentiment&#10;  Analysis" />
          <attvalue for="1" value="  Targeted sentiment analysis is the task of jointly predicting target entities&#10;and their associated sentiment information. Existing research efforts mostly&#10;regard this joint task as a sequence labeling problem, building models that can&#10;capture explicit structures in the output space. However, the importance of&#10;capturing implicit global structural information that resides in the input&#10;space is largely unexplored. In this work, we argue that both types of&#10;information (implicit and explicit structural information) are crucial for&#10;building a successful targeted sentiment analysis model. Our experimental&#10;results show that properly capturing both information is able to lead to better&#10;performance than competitive existing approaches. We also conduct extensive&#10;experiments to investigate our model's effectiveness and robustness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1605.08900" label="1605.08900">
        <attvalues>
          <attvalue for="0" value="Aspect Level Sentiment Classification with Deep Memory Network" />
          <attvalue for="1" value="  We introduce a deep memory network for aspect level sentiment classification.&#10;Unlike feature-based SVM and sequential neural models such as LSTM, this&#10;approach explicitly captures the importance of each context word when inferring&#10;the sentiment polarity of an aspect. Such importance degree and text&#10;representation are calculated with multiple computational layers, each of which&#10;is a neural attention model over an external memory. Experiments on laptop and&#10;restaurant datasets demonstrate that our approach performs comparable to&#10;state-of-art feature based SVM system, and substantially better than LSTM and&#10;attention-based LSTM architectures. On both datasets we show that multiple&#10;computational layers could improve the performance. Moreover, our approach is&#10;also fast. The deep memory network with 9 layers is 15 times faster than LSTM&#10;with a CPU implementation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.01086" label="1805.01086">
        <attvalues>
          <attvalue for="0" value="Transformation Networks for Target-Oriented Sentiment Classification" />
          <attvalue for="1" value="  Target-oriented sentiment classification aims at classifying sentiment&#10;polarities over individual opinion targets in a sentence. RNN with attention&#10;seems a good fit for the characteristics of this task, and indeed it achieves&#10;the state-of-the-art performance. After re-examining the drawbacks of attention&#10;mechanism and the obstacles that block CNN to perform well in this&#10;classification task, we propose a new model to overcome these issues. Instead&#10;of attention, our model employs a CNN layer to extract salient features from&#10;the transformed word representations originated from a bi-directional RNN&#10;layer. Between the two layers, we propose a component to generate&#10;target-specific representations of words in the sentence, meanwhile incorporate&#10;a mechanism for preserving the original contextual information from the RNN&#10;layer. Experiments show that our model achieves a new state-of-the-art&#10;performance on a few benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.04346" label="1806.04346">
        <attvalues>
          <attvalue for="0" value="Exploiting Document Knowledge for Aspect-level Sentiment Classification" />
          <attvalue for="1" value="  Attention-based long short-term memory (LSTM) networks have proven to be&#10;useful in aspect-level sentiment classification. However, due to the&#10;difficulties in annotating aspect-level data, existing public datasets for this&#10;task are all relatively small, which largely limits the effectiveness of those&#10;neural models. In this paper, we explore two approaches that transfer knowledge&#10;from document- level data, which is much less expensive to obtain, to improve&#10;the performance of aspect-level sentiment classification. We demonstrate the&#10;effectiveness of our approaches on 4 public datasets from SemEval 2014, 2015,&#10;and 2016, and we show that attention-based LSTM benefits from document-level&#10;knowledge in multiple ways.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.07043" label="1805.07043">
        <attvalues>
          <attvalue for="0" value="Aspect Based Sentiment Analysis with Gated Convolutional Networks" />
          <attvalue for="1" value="  Aspect based sentiment analysis (ABSA) can provide more detailed information&#10;than general sentiment analysis, because it aims to predict the sentiment&#10;polarities of the given aspects or entities in text. We summarize previous&#10;approaches into two subtasks: aspect-category sentiment analysis (ACSA) and&#10;aspect-term sentiment analysis (ATSA). Most previous approaches employ long&#10;short-term memory and attention mechanisms to predict the sentiment polarity of&#10;the concerned targets, which are often complicated and need more training time.&#10;We propose a model based on convolutional neural networks and gating&#10;mechanisms, which is more accurate and efficient. First, the novel Gated&#10;Tanh-ReLU Units can selectively output the sentiment features according to the&#10;given aspect or entity. The architecture is much simpler than attention layer&#10;used in the existing models. Second, the computations of our model could be&#10;easily parallelized during training, because convolutional layers do not have&#10;time dependency as in LSTM layers, and gating units also work independently.&#10;The experiments on SemEval datasets demonstrate the efficiency and&#10;effectiveness of our models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.06276" label="1909.06276">
        <attvalues>
          <attvalue for="0" value="Parameterized Convolutional Neural Networks for Aspect Level Sentiment&#10;  Classification" />
          <attvalue for="1" value="  We introduce a novel parameterized convolutional neural network for aspect&#10;level sentiment classification. Using parameterized filters and parameterized&#10;gates, we incorporate aspect information into convolutional neural networks&#10;(CNN). Experiments demonstrate that our parameterized filters and parameterized&#10;gates effectively capture the aspect-specific features, and our CNN-based&#10;models achieve excellent results on SemEval 2014 datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.10999" label="1811.10999">
        <attvalues>
          <attvalue for="0" value="Exploiting Coarse-to-Fine Task Transfer for Aspect-level Sentiment&#10;  Classification" />
          <attvalue for="1" value="  Aspect-level sentiment classification (ASC) aims at identifying sentiment&#10;polarities towards aspects in a sentence, where the aspect can behave as a&#10;general Aspect Category (AC) or a specific Aspect Term (AT). However, due to&#10;the especially expensive and labor-intensive labeling, existing public corpora&#10;in AT-level are all relatively small. Meanwhile, most of the previous methods&#10;rely on complicated structures with given scarce data, which largely limits the&#10;efficacy of the neural models. In this paper, we exploit a new direction named&#10;coarse-to-fine task transfer, which aims to leverage knowledge learned from a&#10;rich-resource source domain of the coarse-grained AC task, which is more easily&#10;accessible, to improve the learning in a low-resource target domain of the&#10;fine-grained AT task. To resolve both the aspect granularity inconsistency and&#10;feature mismatch between domains, we propose a Multi-Granularity Alignment&#10;Network (MGAN). In MGAN, a novel Coarse2Fine attention guided by an auxiliary&#10;task can help the AC task modeling at the same fine-grained level with the AT&#10;task. To alleviate the feature false alignment, a contrastive feature alignment&#10;method is adopted to align aspect-specific feature representations&#10;semantically. In addition, a large-scale multi-domain dataset for the AC task&#10;is provided. Empirically, extensive experiments demonstrate the effectiveness&#10;of the MGAN.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.00107" label="1708.00107">
        <attvalues>
          <attvalue for="0" value="Learned in Translation: Contextualized Word Vectors" />
          <attvalue for="1" value="  Computer vision has benefited from initializing multiple deep layers with&#10;weights pretrained on large supervised training sets like ImageNet. Natural&#10;language processing (NLP) typically sees initialization of only the lowest&#10;layer of deep models with pretrained word vectors. In this paper, we use a deep&#10;LSTM encoder from an attentional sequence-to-sequence model trained for machine&#10;translation (MT) to contextualize word vectors. We show that adding these&#10;context vectors (CoVe) improves performance over using only unsupervised word&#10;and character vectors on a wide variety of common NLP tasks: sentiment analysis&#10;(SST, IMDb), question classification (TREC), entailment (SNLI), and question&#10;answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe&#10;improves performance of our baseline models to the state of the art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.09588" label="1903.09588">
        <attvalues>
          <attvalue for="0" value="Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing&#10;  Auxiliary Sentence" />
          <attvalue for="1" value="  Aspect-based sentiment analysis (ABSA), which aims to identify fine-grained&#10;opinion polarity towards a specific aspect, is a challenging subtask of&#10;sentiment analysis (SA). In this paper, we construct an auxiliary sentence from&#10;the aspect and convert ABSA to a sentence-pair classification task, such as&#10;question answering (QA) and natural language inference (NLI). We fine-tune the&#10;pre-trained model from BERT and achieve new state-of-the-art results on&#10;SentiHood and SemEval-2014 Task 4 datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.09314" label="1902.09314">
        <attvalues>
          <attvalue for="0" value="Attentional Encoder Network for Targeted Sentiment Classification" />
          <attvalue for="1" value="  Targeted sentiment classification aims at determining the sentimental&#10;tendency towards specific targets. Most of the previous approaches model&#10;context and target words with RNN and attention. However, RNNs are difficult to&#10;parallelize and truncated backpropagation through time brings difficulty in&#10;remembering long-term patterns. To address this issue, this paper proposes an&#10;Attentional Encoder Network (AEN) which eschews recurrence and employs&#10;attention based encoders for the modeling between context and target. We raise&#10;the label unreliability issue and introduce label smoothing regularization. We&#10;also apply pre-trained BERT to this task and obtain new state-of-the-art&#10;results. Experiments and analysis demonstrate the effectiveness and lightweight&#10;of our model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.11860" label="1908.11860">
        <attvalues>
          <attvalue for="0" value="Adapt or Get Left Behind: Domain Adaptation through BERT Language Model&#10;  Finetuning for Aspect-Target Sentiment Classification" />
          <attvalue for="1" value="  Aspect-Target Sentiment Classification (ATSC) is a subtask of Aspect-Based&#10;Sentiment Analysis (ABSA), which has many applications e.g. in e-commerce,&#10;where data and insights from reviews can be leveraged to create value for&#10;businesses and customers. Recently, deep transfer-learning methods have been&#10;applied successfully to a myriad of Natural Language Processing (NLP) tasks,&#10;including ATSC. Building on top of the prominent BERT language model, we&#10;approach ATSC using a two-step procedure: self-supervised domain-specific BERT&#10;language model finetuning, followed by supervised task-specific finetuning. Our&#10;findings on how to best exploit domain-specific language model finetuning&#10;enable us to produce new state-of-the-art performance on the SemEval 2014 Task&#10;4 restaurants dataset. In addition, to explore the real-world robustness of our&#10;models, we perform cross-domain evaluation. We show that a cross-domain adapted&#10;BERT language model performs significantly better than strong baseline models&#10;like vanilla BERT-base and XLNet-base. Finally, we conduct a case study to&#10;interpret model prediction errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.02606" label="1909.02606">
        <attvalues>
          <attvalue for="0" value="Syntax-Aware Aspect Level Sentiment Classification with Graph Attention&#10;  Networks" />
          <attvalue for="1" value="  Aspect level sentiment classification aims to identify the sentiment&#10;expressed towards an aspect given a context sentence. Previous neural network&#10;based methods largely ignore the syntax structure in one sentence. In this&#10;paper, we propose a novel target-dependent graph attention network (TD-GAT) for&#10;aspect level sentiment classification, which explicitly utilizes the dependency&#10;relationship among words. Using the dependency graph, it propagates sentiment&#10;features directly from the syntactic context of an aspect target. In our&#10;experiments, we show our method outperforms multiple baselines with GloVe&#10;embeddings. We also demonstrate that using BERT representations further&#10;substantially boosts the performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.11297" label="1909.11297">
        <attvalues>
          <attvalue for="0" value="Learning to Detect Opinion Snippet for Aspect-Based Sentiment Analysis" />
          <attvalue for="1" value="  Aspect-based sentiment analysis (ABSA) is to predict the sentiment polarity&#10;towards a particular aspect in a sentence. Recently, this task has been widely&#10;addressed by the neural attention mechanism, which computes attention weights&#10;to softly select words for generating aspect-specific sentence representations.&#10;The attention is expected to concentrate on opinion words for accurate&#10;sentiment prediction. However, attention is prone to be distracted by noisy or&#10;misleading words, or opinion words from other aspects. In this paper, we&#10;propose an alternative hard-selection approach, which determines the start and&#10;end positions of the opinion snippet, and selects the words between these two&#10;positions for sentiment prediction. Specifically, we learn deep associations&#10;between the sentence and aspect, and the long-term dependencies within the&#10;sentence by leveraging the pre-trained BERT model. We further detect the&#10;opinion snippet by self-critical reinforcement learning. Especially,&#10;experimental results demonstrate the effectiveness of our method and prove that&#10;our hard-selection approach outperforms soft-selection approaches when handling&#10;multi-aspect sentences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.05403" label="1712.05403">
        <attvalues>
          <attvalue for="0" value="Learning to Attend via Word-Aspect Associative Fusion for Aspect-based&#10;  Sentiment Analysis" />
          <attvalue for="1" value="  Aspect-based sentiment analysis (ABSA) tries to predict the polarity of a&#10;given document with respect to a given aspect entity. While neural network&#10;architectures have been successful in predicting the overall polarity of&#10;sentences, aspect-specific sentiment analysis still remains as an open problem.&#10;In this paper, we propose a novel method for integrating aspect information&#10;into the neural model. More specifically, we incorporate aspect information&#10;into the neural model by modeling word-aspect relationships. Our novel model,&#10;\textit{Aspect Fusion LSTM} (AF-LSTM) learns to attend based on associative&#10;relationships between sentence words and aspect which allows our model to&#10;adaptively focus on the correct words given an aspect term. This ameliorates&#10;the flaws of other state-of-the-art models that utilize naive concatenations to&#10;model word-aspect similarity. Instead, our model adopts circular convolution&#10;and circular correlation to model the similarity between aspect and words and&#10;elegantly incorporates this within a differentiable neural attention framework.&#10;Finally, our model is end-to-end differentiable and highly related to&#10;convolution-correlation (holographic like) memories. Our proposed neural model&#10;achieves state-of-the-art performance on benchmark datasets, outperforming&#10;ATAE-LSTM by $4\%-5\%$ on average across multiple datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.09567" label="2011.09567">
        <attvalues>
          <attvalue for="0" value="Predicting metrical patterns in Spanish poetry with language models" />
          <attvalue for="1" value="  In this paper, we compare automated metrical pattern identification systems&#10;available for Spanish against extensive experiments done by fine-tuning&#10;language models trained on the same task. Despite being initially conceived as&#10;a model suitable for semantic tasks, our results suggest that BERT-based models&#10;retain enough structural information to perform reasonably well for Spanish&#10;scansion.&#10;" />
          <attvalue for="2" value="&#10;We can consider the metre of a verse as a sequence of stressed (strong) and unstressed (weak) syllables, which are sometimes denoted with the plus symbol `$+$' for stressed syllables and the minus `$-$' for the unstressed ones. Example \ref{example.1} shows an hendecasyllabic Spanish verse and the resulting metrical pattern after applying rhetorical figures that might shrink (synalepha) or expand (syneresis) its length. The stress of the last word also affects the metrical length in Spanish poetry. The identification of metrical patterns is part of a larger procedure for the scansion of a poem.&#10;&#10;\begin{example}&#10;cubra de nieve la hermosa cumbre\\&#10;cu-bra-de-nie-ve-la-her-mo-sa-cum-bre \\&#10;$+--+---+-+-$ 11 \\&#10;(Garcilaso de la Vega) \\&#10;\end{example}&#10;&#10;Research has shown that neural models implicitly encode linguistic features ranging from token labeling to different kinds of segmentation \cite{liu2019linguistic}. There is also evidence that language models and embeddings are able to capture not only semantic and syntactic properties but structural information, as shown by \cite{hewitt2019structural} in their work with structural probes for extracting syntax trees, and \cite{conneau2018you} approximating the length in words of a sentence by its vector.&#10;&#10;On the other hand, the earliest computational approach to Spanish scansion was introduced by \cite{gervas2000logic}, whose tool uses DCG (Definite Clause Grammars) to model word syllabification as well as additional predicates to define synalepha, syllable count and rhyme. More recently, the ADSO Scansion system introduced by \cite{navarro2017metrical} first applies part of speech (PoS) tags to the words of every line in a poem to analyze hendecasyllables. Rantanplan \cite{delarosa2020rantanplan} employs a similar system but focused on accuracy and speed and achieves state of the art results. The only neural approach was explored by \cite{agirrezabal2016zeuscansion,agirrezabal2017comparison}, who used bi-LSTM neural networks and CRF's to automatically scan poetry in three languages (i.e. English, Spanish and Basque).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Metrical Pattern Analysis, Linguistics, Language Model Evaluation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1805.01070" label="1805.01070">
        <attvalues>
          <attvalue for="0" value="What you can cram into a single vector: Probing sentence embeddings for&#10;  linguistic properties" />
          <attvalue for="1" value="  Although much effort has recently been devoted to training high-quality&#10;sentence embeddings, we still have a poor understanding of what they are&#10;capturing. &quot;Downstream&quot; tasks, often based on sentence classification, are&#10;commonly used to evaluate the quality of sentence representations. The&#10;complexity of the tasks makes it however difficult to infer what kind of&#10;information is present in the representations. We introduce here 10 probing&#10;tasks designed to capture simple linguistic features of sentences, and we use&#10;them to study embeddings generated by three different encoders trained in eight&#10;distinct ways, uncovering intriguing properties of both encoders and training&#10;methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.00938" label="1711.00938">
        <attvalues>
          <attvalue for="0" value="A Comparison of Feature-Based and Neural Scansion of Poetry" />
          <attvalue for="1" value="  Automatic analysis of poetic rhythm is a challenging task that involves&#10;linguistics, literature, and computer science. When the language to be analyzed&#10;is known, rule-based systems or data-driven methods can be used. In this paper,&#10;we analyze poetic rhythm in English and Spanish. We show that the&#10;representations of data learned from character-based neural models are more&#10;informative than the ones from hand-crafted features, and that a&#10;Bi-LSTM+CRF-model produces state-of-the art accuracy on scansion of poetry in&#10;two languages. Results also show that the information about whole word&#10;structure, and not just independent syllables, is highly informative for&#10;performing scansion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.03070" label="2408.03070">
        <attvalues>
          <attvalue for="0" value="Probing structural constraints of negation in Pretrained Language Models" />
          <attvalue for="1" value="  Contradictory results about the encoding of the semantic impact of negation&#10;in pretrained language models (PLMs). have been drawn recently (e.g. Kassner&#10;and Sch{\&quot;u}tze (2020); Gubelmann and Handschuh (2022)). In this paper we focus&#10;rather on the way PLMs encode negation and its formal impact, through the&#10;phenomenon of the Negative Polarity Item (NPI) licensing in English. More&#10;precisely, we use probes to identify which contextual representations best&#10;encode 1) the presence of negation in a sentence, and 2) the polarity of a&#10;neighboring masked polarity item. We find that contextual representations of&#10;tokens inside the negation scope do allow for (i) a better prediction of the&#10;presence of not compared to those outside the scope and (ii) a better&#10;prediction of the right polarity of a masked polarity item licensed by not,&#10;although the magnitude of the difference varies from PLM to PLM. Importantly,&#10;in both cases the trend holds even when controlling for distance to not. This&#10;tends to indicate that the embeddings of these models do reflect the notion of&#10;negation scope, and do encode the impact of negation on NPI licensing. Yet,&#10;further control experiments reveal that the presence of other lexical items is&#10;also better captured when using the contextual representation of a token within&#10;the same syntactic clause than outside from it, suggesting that PLMs simply&#10;capture the more general notion of syntactic clause.&#10;" />
          <attvalue for="2" value="&#10;&#10;Negation has recently been the focus of various works aiming at determining the abilities of Pretrained Language Models (PLMs) to capture linguistic knowledge.&#10;&#10;Some works investigate the `semantic impact' of negation, namely its impact in terms of truth values, by interpreting how the presence of negation impacts the probability distribution at a masked position. The rationale is that negating a verb reverses the truth value of its clause, which should be reflected in the probability distribution at certain positions.&#10;\cite{ettinger_what_2020,kassner_negated_2020} use factual statements such as \Next, and report that models \draftreplace{fail {to reflect the presence of negation in the probability distribution.}}{output similar distributions for the positive and negative variants of \Next, and conclude that models largely ignore negation.} &#10;&#10;\ex. A robin is (not) a [MASK]&#10;&#10;\cite{gubelmann-handschuh-2022-context} chose to avoid factual statements and to focus rather on multi-sentence self-contained examples, such that, given the context provided by the first sentence, one particular word is either likely (in positive items) or ruled out (in negative items) at a masked position in the second sentence. Because this particular word is substantially less often the top-1 prediction in the negative items than in the positive items, the authors draw the opposite conclusion that PLMs do show sensitivity to negation.&#10;&#10;A different line of works focused on finding out to what extent \draftreplace{contextual information}{negation} is encoded in PLM embeddings. &#10;\cite{celikkanat_controlling_2020} train classifiers taking as input the contextual embedding of a verb or its subject or direct object, and predicting whether the verb is negated or not. \draftreplace{They report that traces of not can indeed be found in those tokens' representations.}{The resulting high accuracy allows them to conclude that these tokens' embeddings do contain ``traces'' of not.}&#10;More generally, several authors have investigated whether the contextual representation of a token encodes information about surrounding tokens. To ease further reading, we will talk of a classifier taking as input an input embedding, i.e.\ the contextual representation of an input token, and predicting some target information about another token in the sentence.&#10;For instance, \cite{klafka_spying_2020} study how input embeddings encode animacy, gender, and number of surrounding words in a specific SVO context. \cite{li-etal-2022-distributed} target the number feature of French participles in the context of object-past participle agreement.&#10;They show that the performance of the classifier depends on the syntactic position of the input token in the sentence. We will build on their idea to compare performance at predicting target information depending on the syntactic zone the input token belongs to. In this paper, one of the probed target information will be the presence or absence of a given word within the sentence, which we call the target token.&#10;&#10;More precisely, our aim is to study PLMs' ability to capture and encode structural information concerning negation (namely negation scope). To do so we first probe whether input embeddings can serve to accurately predict the presence or absence of a target not. Moreover, we wish to test PLMs' ability to actually mobilize this encoding to capture phenomena that are direct consequences of the presence of negation. &#10;To do so, we focus on the licensing of Negative Polarity Items (NPI) by not modifying a verb. Polarity Items (PI), either positive (e.g. some), or negative (e.g. any), are words or expressions that are constrained in their distribution \cite{homer20.comp}. A NPI will require that a word or a construction, called the {licensor}, be in the vicinity. More precisely, the licensor itself grammatically defines a zone of the sentence, called the {licensing scope}, in which the NPI can appear. The adverb not modifying a verb is one such licensor. While any is licensed by negation in \Next[a] vs.\ \Next[b], it is not licensed in \Next[c], even though the verb is negated, arguably because it is not in the licensing scope.&#10;&#10;\ex. \a. Sam didn't find any books. &#10;\b. *Sam found any books. &#10;\c. *Any book was not found by Sam.&#10;&#10;\cite{jumelet_language_2018} have shown that LSTM embeddings do encode the notion of licensing scope (given an input embedding, a classifier can predict the structural zone the input token belongs to), a finding later confirmed for transformer-based PLMs \cite{warstadt_investigating_2019}. \draftadd{Focusing on when the licensor is a verb-modifying not, }we rather investigate whether this encoding of the zones go as far as enabling a better prediction of a PI’s polarity from inside the licensing scope compared to outside the scope. So instead of the question ``Is this input embedding the embedding of a token located within, before or after the licensing scope?'', we rather ask the question ``Given a masked PI position, \draftreplace{from which neighbor input embeddings are we able to best predict the polarity of the PI?}{and an input embedding of a neighboring token, what is the polarity of the PI?''}, and we study whether this question is better answered when the input embedding is inside or outside the licensing or negation scopes.&#10;&#10;Note that our methodology differs from that of \cite{jumelet_language_2018}, who, given an input token, predict the zone this token belongs to. We instead predict the polarity of a neighboring masked polarity item and then compare accuracies depending on the input token's zone. Our motivation is that the polarity, being a lexical information, requires less linguistic preconception, and hence our probing method is a more direct translation of the NPI licensing phenomenon: we study whether and where the information of ``which PIs are licit where?'' is encoded, in the context of sentence negation. This method also allows us to better control the confounding factor of distance between the input embedding and the licensor not.&#10;&#10;In the following, we define the linguistic notions of negation scope and NPI licensing scope in section~\ref{s:defs}, and show how we actually identified them in English sentences. In section~\ref{s:probing_scopes}, we describe our probing experiments and discuss their results, both for the encoding of not (section~\ref{ss:probing_neg_scope}), and the encoding of NPI licensing (section~\ref{ss:probing_licensing_scope}). We then study the more general ability of PLMs to deal with clause boundaries (section~\ref{s:clause_boundaries}), and &#10; conclude in section~\ref{s:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Syntactic Clause Representation, Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Negation Encoding, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1808.10627" label="1808.10627">
        <attvalues>
          <attvalue for="0" value="Do Language Models Understand Anything? On the Ability of LSTMs to&#10;  Understand Negative Polarity Items" />
          <attvalue for="1" value="  In this paper, we attempt to link the inner workings of a neural language&#10;model to linguistic theory, focusing on a complex phenomenon well discussed in&#10;formal linguis- tics: (negative) polarity items. We briefly discuss the leading&#10;hypotheses about the licensing contexts that allow negative polarity items and&#10;evaluate to what extent a neural language model has the ability to correctly&#10;process a subset of such constructions. We show that the model finds a relation&#10;between the licensing context and the negative polarity item and appears to be&#10;aware of the scope of this context, which we extract from a parse tree of the&#10;sentence. With this research, we hope to pave the way for other studies linking&#10;formal linguistics to deep learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.02597" label="1909.02597">
        <attvalues>
          <attvalue for="0" value="Investigating BERT's Knowledge of Language: Five Analysis Methods with&#10;  NPIs" />
          <attvalue for="1" value="  Though state-of-the-art sentence representation models can perform tasks&#10;requiring significant knowledge of grammar, it is an open question how best to&#10;evaluate their grammatical knowledge. We explore five experimental methods&#10;inspired by prior work evaluating pretrained sentence representation models. We&#10;use a single linguistic phenomenon, negative polarity item (NPI) licensing in&#10;English, as a case study for our experiments. NPIs like &quot;any&quot; are grammatical&#10;only if they appear in a licensing environment like negation (&quot;Sue doesn't have&#10;any cats&quot; vs. &quot;Sue has any cats&quot;). This phenomenon is challenging because of&#10;the variety of NPI licensing environments that exist. We introduce an&#10;artificially generated dataset that manipulates key features of NPI licensing&#10;for the experiments. We find that BERT has significant knowledge of these&#10;features, but its success varies widely across different experimental methods.&#10;We conclude that a variety of methods is necessary to reveal all relevant&#10;aspects of a model's grammatical knowledge in a given domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.06643" label="2401.06643">
        <attvalues>
          <attvalue for="0" value="Effects of diversity incentives on sample diversity and downstream model&#10;  performance in LLM-based text augmentation" />
          <attvalue for="1" value="  The latest generative large language models (LLMs) have found their&#10;application in data augmentation tasks, where small numbers of text samples are&#10;LLM-paraphrased and then used to fine-tune downstream models. However, more&#10;research is needed to assess how different prompts, seed data selection&#10;strategies, filtering methods, or model settings affect the quality of&#10;paraphrased data (and downstream models). In this study, we investigate three&#10;text diversity incentive methods well established in crowdsourcing: taboo&#10;words, hints by previous outlier solutions, and chaining on previous outlier&#10;solutions. Using these incentive methods as part of instructions to LLMs&#10;augmenting text datasets, we measure their effects on generated texts lexical&#10;diversity and downstream model performance. We compare the effects over 5&#10;different LLMs, 6 datasets and 2 downstream models. We show that diversity is&#10;most increased by taboo words, but downstream model performance is highest with&#10;hints.&#10;" />
          <attvalue for="2" value="&#10; The emergence of large language models (LLMs) such as GPT-4, LLaMA, etc., has sparked interest in using them to augment textual datasets~\cite{ubani2023zeroshotdataaug, dai2023auggpt, piedboeuf-langlais-2023-chatgpt}. In these scenarios, the number of samples is expanded by paraphrasing existing ones through LLM prompting. The created paraphrases are then added to the original dataset and used for downstream model training. Such methods have been explored for various domains such as sentiment classification~\cite{piedboeuf-langlais-2023-chatgpt, ubani2023zeroshotdataaug}, news classification~\cite{piedboeuf-langlais-2023-chatgpt} and health symptoms classifications~\cite{dai2023auggpt}. However, investigation of the effect of various prompts, specific instructions, and selection of seed data inspired by crowd in the text augmentation process when using LLMs is lacking.&#10;&#10;Crowdsourcing is an established practice for collecting training or validation examples for a variety of NLP tasks. Scenarios of data collection using human workers can be similar to those of data augmentation: workers create paraphrases on existing sentences chosen from a dataset. The aim of such data collection is to increase the data diversity and subsequent performance of classifiers trained on the data~\cite{Larson2019, larson-etal-2020-iterative}. To increase the diversity, various methods are used in crowdsourcing to guide workers. These include taboo words~\cite{larson-etal-2020-iterative} - where most significant words from the collected data are identified and listed in the worker instructions to be avoided during paraphrasing, chaining~\cite{Cox2021, Larson2019} - where outliers in the previous paraphrases are identified and used as seed sentences in the next round of data collection, and hints where previous outlier paraphrases are used as examples in the instructions. The hints~\cite{Cox2021, Yaghoub-Zadeh-Fard2020} method itself is similar to LLM in-context learning, where examples are included in the instructions for the model to achieve better performance. All of these diversity incentive methods report increased diversity of paraphrases and some also report increased performance of the classifiers trained on the so-collected data.&#10;&#10;This work is inspired by the parallels between crowdsourcing and LLM prompting and by the performance of diversity incentive methods on the diversity of paraphrases and the performance of models trained on them. We investigate the effects of the three diversity incentive methods (originating in crowdsourcing) on data augmentation using LLMs. The baseline, taken from a previous study~\cite{cegin-etal-2023-chatgpt}, is a simple prompting for paraphrases. Measuring paraphrase diversity and downstream performance of classification models, we assess whether the diversity incentives (added to the base prompt) improve LLM outputs similarly as in crowdsourcing scenarios. To our knowledge, this is the first work to investigate the effects of diversity incentive methods on LLMs.&#10;&#10;In this paper, we answer the following research questions:&#10;&#09;\begin{description}[labelwidth = 24pt, leftmargin = !]&#10;&#09;&#09;\item[RQ1:] Does the usage of diversity incentive methods on LLMs yield more diverse paraphrases? (compared to base prompting)&#10;&#09;&#09;\item[RQ2:] Do classifiers achieve better performance if trained on data augmented using diversity incentive methods on LLMs? (compared to base prompting)&#10;&#09;\end{description}&#10;&#10;To answer these questions, we have conducted a data augmentation experiment using 5 different LLMs on 6 different datasets in the tasks of sentiment (movie and app reviews), news, and intent (flight and voice assistant commands) classification. In this experiment, we repeatedly collect LLM paraphrases using different diversity incentive methods. Then, we compare the lexical diversity of the collected data and the performance of downstream classifiers. Additionally, we also conduct an ablation study, where we modify the diversity incentive methods with random data to validate, that the inputs used by these methods (e.g., most influential taboo words, outlier paraphrases) contribute to the method's performance and a combination of the best performing methods for lexical diversity and model performance. In total, we collected 253,500 paraphrases.&#10;&#10;The most prominent findings are the following: 1) We do not observe statistically significant improvements in lexical diversity of the generated datasets, but only minor improvements using the taboo method,&#10; 2) The hints method increases the performance of classification models trained on such data compared to the baseline, while also reducing standard deviation and thus increasing the stability of results,&#10; 3) The chaining method and taboo method both do not significantly affect the performance of classification models trained on such data compared to the baseline.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Text Data Augmentation, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence Applications, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1904.03122" label="1904.03122">
        <attvalues>
          <attvalue for="0" value="Outlier Detection for Improved Data Quality and Diversity in Dialog&#10;  Systems" />
          <attvalue for="1" value="  In a corpus of data, outliers are either errors: mistakes in the data that&#10;are counterproductive, or are unique: informative samples that improve model&#10;robustness. Identifying outliers can lead to better datasets by (1) removing&#10;noise in datasets and (2) guiding collection of additional data to fill gaps.&#10;However, the problem of detecting both outlier types has received relatively&#10;little attention in NLP, particularly for dialog systems. We introduce a simple&#10;and effective technique for detecting both erroneous and unique samples in a&#10;corpus of short texts using neural sentence embeddings combined with&#10;distance-based outlier detection. We also present a novel data collection&#10;pipeline built atop our detection technique to automatically and iteratively&#10;mine unique data samples while discarding erroneous samples. Experiments show&#10;that our outlier detection technique is effective at finding errors while our&#10;data collection pipeline yields highly diverse corpora that in turn produce&#10;more robust intent classification and slot-filling models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.01063" label="2010.01063">
        <attvalues>
          <attvalue for="0" value="Syntax Representation in Word Embeddings and Neural Networks -- A Survey" />
          <attvalue for="1" value="  Neural networks trained on natural language processing tasks capture syntax&#10;even though it is not provided as a supervision signal. This indicates that&#10;syntactic analysis is essential to the understating of language in artificial&#10;intelligence systems. This overview paper covers approaches of evaluating the&#10;amount of syntactic information included in the representations of words for&#10;different neural network architectures. We mainly summarize re-search on&#10;English monolingual data on language modeling tasks and multilingual data for&#10;neural machine translation systems and multilingual language models. We&#10;describe which pre-trained models and representations of language are best&#10;suited for transfer to syntactic tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Modern methods of natural language processing (NLP) are based on complex neural network architectures, where language units are represented in a metric space \cite{mikolov2013efficient, pennington2014glove, peters2018deep, devlin2019bert, radford2019language}. Such a phenomenon allows us to express linguistic features (i.e., morphological, lexical, syntactic) mathematically. &#10;&#10;The method of obtaining such representation and their interpretations were described in multiple overview works. Almeida and Xex\'eo surveyed different types of static word embeddings \cite{almeida2019word}, and Liu et al. \cite{liu2020contextual} focused on contextual representations found in the most recent neural models. Belinkov and Glass \cite{belinkov2017evaluating} surveyed the strategies of interpreting latent representation. Best to our knowledge, we are the first to focus on the syntactic and morphological abilities of the word representations. We also cover the latest approaches, which go beyond the interpretation of latent vectors and analyze the attentions present in state-of-the-art Transformer models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Modeling Tasks, Linguistics, Cognitive Science, Neural Network Architectures, Artificial Intelligence, Syntactic Analysis Methods" />
        </attvalues>
      </node>
      <node id="1901.09069" label="1901.09069">
        <attvalues>
          <attvalue for="0" value="Word Embeddings: A Survey" />
          <attvalue for="1" value="  This work lists and describes the main recent strategies for building&#10;fixed-length, dense and distributed representations for words, based on the&#10;distributional hypothesis. These representations are now commonly called word&#10;embeddings and, in addition to encoding surprisingly good syntactic and&#10;semantic information, have been proven useful as extra features in many&#10;downstream NLP tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;The task of representing words and documents is part and parcel of most, if not all, Natural Language Processing (NLP) tasks. In general, it has been found to be useful to represent them as vectors, which have an appealing, intuitive interpretation, can be the subject of useful operations (e.g. addition, subtraction, distance measures, etc) and lend themselves well to be used in many Machine Learning (ML) algorithms and strategies.&#10;&#10;The Vector Space Model (VSM), generally attributed to Salton (\cite{salton_1975}) and stemming from the Information Retrieval (IR) community, is arguably the most successful and influential model to encode words and documents as vectors.&#10;&#10;Another very important part of natural language-based solutions is, of course, the study of language models. A language model is a statistical model of language usage. It focuses mainly on predicting the next word given a number of previous words. This is very useful, for instance, in speech recognition software, where one needs to correctly decide what is the word said by the speaker, even when signal quality is poor or there is a lot of background noise.&#10;&#10;These two seemingly independent fields have arguably been brought together by recent research on Neural Network Language Models (NNLMs), with \cite{bengio_et_al_2003}) having developed the first large-scale language models based on neural nets. &#10;&#10;Their idea was to reframe the problem as an unsupervised learning problem. A key feature of this solution is the way raw words vectors are first projected onto a so-called embedding layer before being fed into other layers of the network. Among other reasons, this was imagined to help ease the effect of the curse of dimensionality on language models, and help generalization (\cite{bengio_et_al_2003}).&#10;&#10;With time, such word embeddings have emerged as a topic of research in and of themselves, with the realization that they can be used as standalone features in many NLP tasks (\cite{turian_et_al_2010}) and the fact that they encode surprisingly accurate syntactic and semantic word relationships (\cite{linguisticregularities}).&#10;&#10;More recently, other ways of creating embeddings have surfaced, which rely not on neural networks and embedding layers but on leveraging word-context matrices to arrive at vector representations for words. Among the most influential models we can cite the GloVe model (\cite{pennington_et_al_2014}).&#10;&#10;These two types of model have something in common, namely their reliance on the assumption that words with similar contexts (other words) have the same meaning. This has been called the distributional hypothesis, and has been suggested some time ago by Harris (\cite{harris_1954}), among others.&#10;&#10;This brings us to the definition of word embeddings we will use in this article, as suggested by the literature (for instance, \cite{turian_et_al_2010,blacoe_and_lapata_2012,schnabel_et_al_2015}), according to which word embeddings are dense, distributed, fixed-length word vectors, built using word co-occurrence statistics as per the distributional hypothesis.&#10;&#10;Embedding models derived from neural network language models have (\cite{baroni_et_al_2014}) been called prediction-based models, since they usually leverage language models, which predict the next word given its context. Other matrix-based models have been called count-based models, due to their taking into account global word-context co-occurrence counts to derive word embeddings. These are described next.&#10;&#10;This survey is structured as follows: in section2 we describe the origins of statistical language modelling. In section3 we give an overview of word embeddings, generated both by so-called prediction-based models and by count-based methods. In Section 4 we conclude and in Section 5 we provide some pointers to promising further research topics.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1801.07772" label="1801.07772">
        <attvalues>
          <attvalue for="0" value="Evaluating Layers of Representation in Neural Machine Translation on&#10;  Part-of-Speech and Semantic Tagging Tasks" />
          <attvalue for="1" value="  While neural machine translation (NMT) models provide improved translation&#10;quality in an elegant, end-to-end framework, it is less clear what they learn&#10;about language. Recent work has started evaluating the quality of vector&#10;representations learned by NMT models on morphological and syntactic tasks. In&#10;this paper, we investigate the representations learned at different layers of&#10;NMT encoders. We train NMT systems on parallel data and use the trained models&#10;to extract features for training a classifier on two tasks: part-of-speech and&#10;semantic tagging. We then measure the performance of the classifier as a proxy&#10;to the quality of the original NMT model for the given task. Our quantitative&#10;analysis yields interesting insights regarding representation learning in NMT&#10;models. For instance, we find that higher layers are better at learning&#10;semantics while lower layers tend to be better for part-of-speech tagging. We&#10;also observe little effect of the target language on source-side&#10;representations, especially with higher quality NMT models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.09963" label="2403.09963">
        <attvalues>
          <attvalue for="0" value="Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias&#10;  in Factual Knowledge Extraction" />
          <attvalue for="1" value="  Recent research shows that pre-trained language models (PLMs) suffer from&#10;&quot;prompt bias&quot; in factual knowledge extraction, i.e., prompts tend to introduce&#10;biases toward specific labels. Prompt bias presents a significant challenge in&#10;assessing the factual knowledge within PLMs. Therefore, this paper aims to&#10;improve the reliability of existing benchmarks by thoroughly investigating and&#10;mitigating prompt bias. We show that: 1) all prompts in the experiments exhibit&#10;non-negligible bias, with gradient-based prompts like AutoPrompt and OptiPrompt&#10;displaying significantly higher levels of bias; 2) prompt bias can amplify&#10;benchmark accuracy unreasonably by overfitting the test datasets, especially on&#10;imbalanced datasets like LAMA. Based on these findings, we propose a&#10;representation-based approach to mitigate the prompt bias during inference&#10;time. Specifically, we first estimate the biased representation using&#10;prompt-only querying, and then remove it from the model's internal&#10;representations to generate the debiased representations, which are used to&#10;produce the final debiased outputs. Experiments across various prompts, PLMs,&#10;and benchmarks show that our approach can not only correct the overfitted&#10;performance caused by prompt bias, but also significantly improve the prompt&#10;retrieval capability (up to 10% absolute performance gain). These results&#10;indicate that our approach effectively alleviates prompt bias in knowledge&#10;evaluation, thereby enhancing the reliability of benchmark assessments.&#10;Hopefully, our plug-and-play approach can be a golden standard to strengthen&#10;PLMs toward reliable knowledge bases. Code and data are released in&#10;https://github.com/FelliYang/PromptBias.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Factual Knowledge Extraction&#10;\cite{petroni-etal-2019-language} first introduce the LAMA benchmark to evaluate the factual knowledge contained in PLMs by prompting, and propose that PLMs have the potential to serve as knowledge bases.&#10;~\cite{kassner-etal-2021-multilingual} extend the LAMA benchmark to different languages and investigate factual knowledge contained in multilingual PLMs.&#10;One subsequent research line focuses on finding prompts with better retrieval capability.~\cite{jiang-etal-2020-how} use text-mining and paraphrasing to automatically generate prompts that show better performance than LAMA.~\cite{Shin-etal-2020-Auto} collect a training dataset consisting of different facts with LAMA and use a gradient-based searching algorithm~\cite{wallace-etal-2019-universal} to find better discrete prompts.~\cite{liu-etal-2021-gpt} and ~\cite{zhong-etal-2021-factual} take a further step by exploring continued prompt optimization on factual knowledge extraction.&#10;&#10;However, some other works point out that PLMs can hardly serve as reliable knowledge bases currently. \cite{kassner-schutze-2020-negated} find PLMs can't distinguish negated and non-negated queries. \cite{zhong-etal-2021-factual} show that gradient-based prompts will learn patterns from the training dataset and overfit test datasets.~\cite{cao-etal-2021-knowledgeable} find PLMs suffer from prompt bias in factual knowledge extraction and propose that previous decent performance may be attributed to the prompt bias overfitting on test datasets.&#10;Inspired by~\cite{cao-etal-2021-knowledgeable}, we take a further step to quantify the prompt bias across diverse prompts and PLMs and assess its impact on different benchmarks. We reveal two negative impacts of prompt bias and propose a novel approach to mitigate the prompt bias in factual knowledge extraction.&#10;&#10;There are more advanced language models~\cite{he2020deberta,zhong2022toward,zhong2023bag} for complex understanding tasks, e.g., GLUE~\cite{wang2018glue}, SuperGLUE~\cite{wang2019superglue}, in the future work, we would explore the bias and the effectiveness of our method in these models and tasks.&#10;&#10;Bias in PLM&#10;Bias in PLMs is widely investigated in the NLP field, from training corpus~\cite{kurita-etal-2019-measuring, webster-etal-2020-measuring,dev-etal-2020-measuring} to downstream tasks such as PLM-based metrics~\cite{sun-etal-2022-BERTscore}, machine translation~\cite{stanovsky-etal-2019-evaluating, prates-etal-2020-assessing, wang-etal-2022-measuring}.&#10;Many works explore how to mitigate the intrinsic bias in PLMs~\cite{qian-etal-2019-reducing,bordia-bowman-2019-identifying,webster-etal-2020-measuring,qianCounterfactualInferenceText2021,feiMitigatingLabelBiases2023}.&#10;and the extrinsic bias in downstream tasks\cite{zhao-etal-2017-men, zhao-etal-2018-gender,sun-etal-2022-BERTscore, wang-etal-2022-measuring, behnke-etal-2022-bias}. Focusing on the intrinsic bias, &#10;counterfactual data augmentation (CDA)~\cite{zmigrodCounterfactualDataAugmentation2019,dinanQueensArePowerful2020,webster-etal-2020-measuring,barikeri-etal-2021-redditbias} involves re-balancing the training· corpus by swapping bias attribute words and taking further training. \cite{karimimahabadiEndtoEndBiasMitigation2020} and~\cite{utamaDebiasingNLUModels2020} adjust the model's training loss to mitigate bias by down-weighting the biased data in the corpus. \cite{webster-etal-2020-measuring} propose using dropout regularization~\cite{srivastavaDropoutSimpleWay} as a bias mitigation approach. \cite{schickSelfDiagnosisSelfDebiasingProposal2021} propose a post-hoc debiasing technique to discourage PLMs from generating biased text by leveraging their internal knowledge. There are also other bias mitigation technologies such as projection-based debiasing~\cite{ravfogelNullItOut2020,liangDebiasingSentenceRepresentations2020}, and contrastive learning debiasing~\cite{lyuFeatureLevelDebiasedNatural2023}. In contrast to our method, most of these approaches require the extra cost of data manipulations or model retraining. &#10;&#10;Our approach for performing debiasing builds on recent work that explores data-free debiasing using prompt-only querying~\cite{zhao-etal-2021-calibrate}. Different from~\cite{zhao-etal-2021-calibrate}, we mitigate bias by manipulating representation vectors instead of output probabilities. One advantage of our approach is the ability to generate debiased representation vectors, which can be used in sentence embeddings like PromptBERT~\cite{jiangPromptBERTImprovingBERT2022}. Our approach is also similar to counterfactual inference~\cite{qianCounterfactualInferenceText2021,wangShouldWeRely2022} but their approach only considers keywords in the prompt when distilling bias and discard relatively unimportant words, which have been shown can significantly affect prompts~\cite{schickExploitingClozeQuestions2021,schickItNotJust2021}; in contrast, our target is to mitigate the bias of the whole prompt.&#10;&#10;Although generative language models~\cite{touvron2023llama,achiam2023gpt} have shown significant success in various language understanding and generation tasks~\cite{zhong2023chat,peng2023ChatGPT4MT,lu2023EAPrompt}, recent studies~\cite{zheng2023large,lyu2024beyond} show that LLMs tend to make biased choices that are inconsistent with their inherent (generated) knowledge. Our debiasing strategy consistently enhances performance and reliability on Llama2, demonstrating the universality of our method and its potential to serve as the standard post-processing toolkit for large language model.&#10;In future work, it is also worth investigating the prompt bias in other LLM-prompting-based tasks, such as the lexical choice bias in translation~\cite{ding2020understanding}, decision bias in healthcare copilot~\cite{ren2024healthcare}.&#10; " />
          <attvalue for="4" value="Computer Science, Knowledge Extraction Reliability, Linguistics, Cognitive Science, Prompt Mitigation Techniques, Language Model Bias, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2311.03881" label="2311.03881">
        <attvalues>
          <attvalue for="0" value="Sparse Contrastive Learning of Sentence Embeddings" />
          <attvalue for="1" value="  Recently, SimCSE has shown the feasibility of contrastive learning in&#10;training sentence embeddings and illustrates its expressiveness in spanning an&#10;aligned and uniform embedding space. However, prior studies have shown that&#10;dense models could contain harmful parameters that affect the model&#10;performance, and it is no wonder that SimCSE can as well be invented with such&#10;parameters. Driven by this, parameter sparsification is applied, where&#10;alignment and uniformity scores are used to measure the contribution of each&#10;parameter to the overall quality of sentence embeddings. Drawing from a&#10;preliminary study, we consider parameters with minimal contributions to be&#10;detrimental, as their sparsification results in improved model performance. To&#10;discuss the ubiquity of detrimental parameters and remove them, more&#10;experiments on the standard semantic textual similarity (STS) tasks and&#10;transfer learning tasks are conducted, and the results show that the proposed&#10;sparsified SimCSE (SparseCSE) has excellent performance in comparison with&#10;SimCSE. Furthermore, through in-depth analysis, we establish the validity and&#10;stability of our sparsification method, showcasing that the embedding space&#10;generated by SparseCSE exhibits improved alignment compared to that produced by&#10;SimCSE. Importantly, the uniformity yet remains uncompromised.&#10;" />
          <attvalue for="2" value="&#10;&#10;The task of learning universal sentence embeddings using large-scale pre-trained models has been extensively explored in prior research &#10;~\cite{&#10;DBLP:conf/iclr/LogeswaranL18,&#10;reimers-gurevych-2019-sentence,&#10;DBLP:conf/emnlp/LiZHWYL20,&#10;DBLP:conf/emnlp/ZhangHLLB20,&#10;DBLP:conf/emnlp/GaoYC21,&#10;DBLP:conf/emnlp/0001VKC21,&#10;DBLP:conf/acl/YanLWZWX20,&#10;DBLP:conf/acl/FengYCA022}. &#10;More recently, contrastive learning has been proposed as a method to enhance the quality of sentence embeddings &#10;~\cite{&#10;DBLP:conf/wsdm/QiuHYW22,&#10;DBLP:conf/emnlp/ZhangHLLB20,&#10;DBLP:conf/emnlp/GaoYC21,&#10;DBLP:conf/emnlp/0001VKC21,&#10;DBLP:conf/acl/YanLWZWX20}. &#10;By employing contrastive learning, semantically similar sentences are brought closer together while dissimilar sentences are pushed apart, thereby a semantically-driven structure is established within the space of sentence embeddings.&#10;&#10;Unsupervised SimCSE (unsup-SimCSE) is a notable framework for contrastive sentence embeddings~\cite{DBLP:conf/emnlp/GaoYC21}. It utilizes dropout as a simple data augmentation technique to create positive pairs and employs a cross-entropy objective based on cosine similarity for contrastive learning. &#10;Inspired by recent research on parameter sparsification~\cite{DBLP:conf/acl/XiaZC22, prasanna-etal-2020-bert,DBLP:conf/nips/HouHSJCL20,DBLP:conf/nips/MichelLN19}, particularly the works on the lottery ticket hypothesis (LTH)~\cite{DBLP:conf/iclr/FrankleC19,DBLP:conf/iclr/BaiWTL022,DBLP:conf/icml/FrankleD0C20,DBLP:conf/nlpcc/YangZWS22} showing its effectiveness in improving model performance through pruning, we hypothesize that certain parameters in SimCSE might hinder the representation of universal sentence embeddings. By removing these parameters, we anticipate an improvement in the model's performance.&#10;&#10;To accurately estimate the contribution of each parameter, it is essential to consider properties that characterize contrastive representation learning. &#10;In the literature ~\cite{DBLP:conf/icml/0001I20}, two such properties have been proposed: alignment and uniformity.&#10;Alignment measures the proximity of features derived from positive pairs, indicating how well the model captures semantic similarity. On the other hand, uniformity pertains to the distribution of features across the hypersphere, ensuring that the representations are spread out evenly. These properties offer valuable insights into understanding and evaluating contrastive representation learning.&#10;Utilizing alignment and uniformity as guiding principles, we propose an innovative approach, named alignment and uniformity score, to quantify parameter contribution during the preparation phase for pruning. &#10;&#10;Based on a pilot study presented in Figure \ref{fig:pre_peak}, we observed that model performance does not consistently decrease during pruning, instead it exhibits an upward trend when the model is less sparse. This suggests that the parameters with the lowest scores are detrimental to model performance, as evidenced by the performance improvement resulting from their pruning. Building upon this, we conducted a series of more extensive and detailed experiments to explore the ubiquity of detrimental parameters and assess the stability of our proposed pruning method.&#10;&#10;Specifically, our approach consists of three stages: training, parameter sparsification, and rewinding. &#10;First, we train an unsupervised SimCSE model using a pre-trained language model (LM). Then, we estimate alignment and uniformity scores for each parameter based on the trained model's feedback. Parameters with low scores are pruned and varying sparsity is attempted in formal experiments than in pilot study to clearly identify harmful parameters. Finally, the remaining parameters are initialized, and the pruned model is fine-tuned to regain its performance.&#10;Our model is thus named SparseCSE.&#10;&#10;We extensively evaluate SparseCSE on seven STS tasks and seven transfer learning tasks. Results show that SparseCSE outperforms SimCSE, demonstrating its superior performance. Our pruning method is also shown to effectively identify the optimal sparsity for pruning, further enhancing performance.&#10;Further analysis reveals the stability of our pruning method across multiple tasks. &#10;Comparison with other works highlights the similarity of SparseCSE to SimCSE in uniformity and its competitive performance in alignment.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Contrastive Learning, Linguistics, Parameter Sparsification, Artificial Intelligence, Mathematics, Sentence Embeddings" />
        </attvalues>
      </node>
      <node id="2203.04248" label="2203.04248">
        <attvalues>
          <attvalue for="0" value="Dual Lottery Ticket Hypothesis" />
          <attvalue for="1" value="  Fully exploiting the learning capacity of neural networks requires&#10;overparameterized dense networks. On the other side, directly training sparse&#10;neural networks typically results in unsatisfactory performance. Lottery Ticket&#10;Hypothesis (LTH) provides a novel view to investigate sparse network training&#10;and maintain its capacity. Concretely, it claims there exist winning tickets&#10;from a randomly initialized network found by iterative magnitude pruning and&#10;preserving promising trainability (or we say being in trainable condition). In&#10;this work, we regard the winning ticket from LTH as the subnetwork which is in&#10;trainable condition and its performance as our benchmark, then go from a&#10;complementary direction to articulate the Dual Lottery Ticket Hypothesis&#10;(DLTH): Randomly selected subnetworks from a randomly initialized dense network&#10;can be transformed into a trainable condition and achieve admirable performance&#10;compared with LTH -- random tickets in a given lottery pool can be transformed&#10;into winning tickets. Specifically, by using uniform-randomly selected&#10;subnetworks to represent the general cases, we propose a simple sparse network&#10;training strategy, Random Sparse Network Transformation (RST), to substantiate&#10;our DLTH. Concretely, we introduce a regularization term to borrow learning&#10;capacity and realize information extrusion from the weights which will be&#10;masked. After finishing the transformation for the randomly selected&#10;subnetworks, we conduct the regular finetuning to evaluate the model using fair&#10;comparisons with LTH and other strong baselines. Extensive experiments on&#10;several public datasets and comparisons with competitive approaches validate&#10;our DLTH as well as the effectiveness of the proposed model RST. Our work is&#10;expected to pave a way for inspiring new research directions of sparse network&#10;training in the future. Our code is available at&#10;https://github.com/yueb17/DLTH.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.05671" label="1912.05671">
        <attvalues>
          <attvalue for="0" value="Linear Mode Connectivity and the Lottery Ticket Hypothesis" />
          <attvalue for="1" value="  We study whether a neural network optimizes to the same, linearly connected&#10;minimum under different samples of SGD noise (e.g., random data order and&#10;augmentation). We find that standard vision models become stable to SGD noise&#10;in this way early in training. From then on, the outcome of optimization is&#10;determined to a linearly connected region. We use this technique to study&#10;iterative magnitude pruning (IMP), the procedure used by work on the lottery&#10;ticket hypothesis to identify subnetworks that could have trained in isolation&#10;to full accuracy. We find that these subnetworks only reach full accuracy when&#10;they are stable to SGD noise, which either occurs at initialization for&#10;small-scale settings (MNIST) or early in training for large-scale settings&#10;(ResNet-50 and Inception-v3 on ImageNet).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.10242" label="2005.10242">
        <attvalues>
          <attvalue for="0" value="Understanding Contrastive Representation Learning through Alignment and&#10;  Uniformity on the Hypersphere" />
          <attvalue for="1" value="  Contrastive representation learning has been outstandingly successful in&#10;practice. In this work, we identify two key properties related to the&#10;contrastive loss: (1) alignment (closeness) of features from positive pairs,&#10;and (2) uniformity of the induced distribution of the (normalized) features on&#10;the hypersphere. We prove that, asymptotically, the contrastive loss optimizes&#10;these properties, and analyze their positive effects on downstream tasks.&#10;Empirically, we introduce an optimizable metric to quantify each property.&#10;Extensive experiments on standard vision and language datasets confirm the&#10;strong agreement between both metrics and downstream task performance.&#10;Remarkably, directly optimizing for these two metrics leads to representations&#10;with comparable or better performance at downstream tasks than contrastive&#10;learning.&#10;  Project Page: https://tongzhouwang.info/hypersphere&#10;  Code: https://github.com/SsnL/align_uniform ,&#10;https://github.com/SsnL/moco_align_uniform&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.05892" label="2210.05892">
        <attvalues>
          <attvalue for="0" value="Perplexity from PLM Is Unreliable for Evaluating Text Quality" />
          <attvalue for="1" value="  Recently, amounts of works utilize perplexity~(PPL) to evaluate the quality&#10;of the generated text. They suppose that if the value of PPL is smaller, the&#10;quality(i.e. fluency) of the text to be evaluated is better. However, we find&#10;that the PPL referee is unqualified and it cannot evaluate the generated text&#10;fairly for the following reasons: (i) The PPL of short text is larger than long&#10;text, which goes against common sense, (ii) The repeated text span could damage&#10;the performance of PPL, and (iii) The punctuation marks could affect the&#10;performance of PPL heavily. Experiments show that the PPL is unreliable for&#10;evaluating the quality of given text. Last, we discuss the key problems with&#10;evaluating text quality using language models.&#10;" />
          <attvalue for="2" value="&#10;&#10;The rapid development in natural language processing, particularly the success of pre-trained language models, has brought tremendous growth and progress to various text generation tasks. Examples include machine translation~\cite{DBLP:conf/acl/TuLLLL16,DBLP:journals/taslp/ZhangLSZX021}, question answering~\cite{DBLP:conf/emnlp/DuanTCZ17}, and generation-based dialog system~\cite{DBLP:conf/acl/TuLC0W022}. &#10;How to evaluate the quality of the generated text in a cost efficient manner has become a key challenge. &#10;&#10;Researchers have adopted various statistical metrics to evaluate the generated text. These measures include word-based measures like BLEU~\cite{DBLP:conf/acl/PapineniRWZ02} and ROUGE~\cite{lin-2004-rouge}), character-based metrics like chrF~\cite{DBLP:conf/wmt/Popovic15}, and embedding-based metrics like Vector Extrema~\cite{forgues2014bootstrapping} and Greedy Matching~\cite{DBLP:conf/bea/RusL12}. Specifically, &#10;BLEU reflects the ratio of overlapping $n$-grams to the total $n$-grams, denoting a precision-based measure. ROUGE and its variants, also evaluating text based on $n$-grams, are recall-based measures~\cite{DBLP:journals/csur/SaiMK23}.&#10;Vector Extrema prioritizes informative words by taking the extreme value along each dimension. All these measures are widely adopted in many experiments and tasks. However, such statistical-based measures cannot well evaluate the creativeness, diversity, and complexity of texts, particularly in the scenario that the same semantic is expressed in different expressions, e.g., different words/phrases, or different sentence structures. &#10;&#10;In addition to the aforementioned statistical-based measures, perplexity (PPL) has also been used to evaluate the text quality or fluency in generation tasks. PPL is an intrinsic measure to quantify to what extent classical language models, e.g., $n$-gram models, learn natural language~\cite{DBLP:conf/acl/MeisterC20}. Considering the large-scale pre-trained language models (PLMs) e.g., BERT~\cite{DBLP:conf/naacl/DevlinCLT19} and GPT~\cite{radford2019language}, have well captured language knowledge, PPL has also been used to evaluate quality of generated text. Given a PLM model and a sequence of generated text, perplexity reflects how likely the model is to generate this text sequence. If we assume a large PLM well captures language knowledge and is well-behaved, then the PPL value computed in this way could reflect the quality of the input sequence to some extent. &#10;&#10;In this paper, we use PLM to compute PPL values of high quality sentences, as if these sentences were outputs from some generative models. Based on the distributions of PPL values, we claim that PPL computed in this way cannot fairly evaluate text quality. Specifically, we used GPT-2 model~\cite{radford2019language} to compute PPL of sentences in WikiText-2 dataset. As the sentences in WikiText dataset were extracted from verified good and featured articles on Wikipedia, we trust these sentences are of high quality. However, our experiments lead to the following findings. &#10;(i) PPL is sensitive to text length, i.e., PPL of short text is likely to be much larger than that of long text. On the other hand, the generated texts to be evaluated may have different lengths~\cite{meister-cotterell-2021-language}. Strictly speaking, text quality is independent of text length. &#10;(ii) PPL is lower for text with repeated span(s). Generative text may contain repeated span(s). Although legitimate repeated text spans can be used to express emphasis in sentences, PPL cannot distinguish valid semantic emphasis in sentences from unreasonable straightforward repetitions. &#10;(iii) PPL is sensitive to punctuation marks in sentences. Simply removing the last punctuation mark in a sentence may lead a significant increase in its PPL. On the other hand, removing the last punctuation from a sentence may only lead to a very small impact to human perception of the sentence. &#10;&#10;To the best of our knowledge, this is the first attempt to systematically analyze PPL for its suitability as a quality measure for generative text. Based on the findings, we call for more carefully designed metrics which are expected to be (i) not sensitive to length; (ii) sensitive to common mistakes, e.g., unnecessary repeated text; (iii) not sensitive to minor punctuation changes. &#10;In other words, a measure of text fluency shall not be much affected by text length, while penalizing unnecessary text spans and not attending to non-significant punctuation marks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Limitations, Text Quality Evaluation, Computational Linguistics, Perplexity Measurement Issues, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1601.04811" label="1601.04811">
        <attvalues>
          <attvalue for="0" value="Modeling Coverage for Neural Machine Translation" />
          <attvalue for="1" value="  Attention mechanism has enhanced state-of-the-art Neural Machine Translation&#10;(NMT) by jointly learning to align and translate. It tends to ignore past&#10;alignment information, however, which often leads to over-translation and&#10;under-translation. To address this problem, we propose coverage-based NMT in&#10;this paper. We maintain a coverage vector to keep track of the attention&#10;history. The coverage vector is fed to the attention model to help adjust&#10;future attention, which lets NMT system to consider more about untranslated&#10;source words. Experiments show that the proposed approach significantly&#10;improves both translation quality and alignment quality over standard&#10;attention-based NMT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.11520" label="1911.11520">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation with Explicit Phrase Alignment" />
          <attvalue for="1" value="  While neural machine translation (NMT) has achieved state-of-the-art&#10;translation performance, it is unable to capture the alignment between the&#10;input and output during the translation process. The lack of alignment in NMT&#10;models leads to three problems: it is hard to (1) interpret the translation&#10;process, (2) impose lexical constraints, and (3) impose structural constraints.&#10;To alleviate these problems, we propose to introduce explicit phrase alignment&#10;into the translation process of arbitrary NMT models. The key idea is to build&#10;a search space similar to that of phrase-based statistical machine translation&#10;for NMT where phrase alignment is readily available. We design a new decoding&#10;algorithm that can easily impose lexical and structural constraints.&#10;Experiments show that our approach makes the translation process of NMT more&#10;interpretable without sacrificing translation quality. In addition, our&#10;approach achieves significant improvements in lexically and structurally&#10;constrained translation tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.13560" label="2203.13560">
        <attvalues>
          <attvalue for="0" value="MISC: A MIxed Strategy-Aware Model Integrating COMET for Emotional&#10;  Support Conversation" />
          <attvalue for="1" value="  Applying existing methods to emotional support conversation -- which provides&#10;valuable assistance to people who are in need -- has two major limitations: (a)&#10;they generally employ a conversation-level emotion label, which is too&#10;coarse-grained to capture user's instant mental state; (b) most of them focus&#10;on expressing empathy in the response(s) rather than gradually reducing user's&#10;distress. To address the problems, we propose a novel model \textbf{MISC},&#10;which firstly infers the user's fine-grained emotional status, and then&#10;responds skillfully using a mixture of strategy. Experimental results on the&#10;benchmark dataset demonstrate the effectiveness of our method and reveal the&#10;benefits of fine-grained emotion understanding as well as mixed-up strategy&#10;modeling. Our code and data could be found in&#10;\url{https://github.com/morecry/MISC}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.12009" label="2008.12009">
        <attvalues>
          <attvalue for="0" value="A Survey of Evaluation Metrics Used for NLG Systems" />
          <attvalue for="1" value="  The success of Deep Learning has created a surge in interest in a wide a&#10;range of Natural Language Generation (NLG) tasks. Deep Learning has not only&#10;pushed the state of the art in several existing NLG tasks but has also&#10;facilitated researchers to explore various newer NLG tasks such as image&#10;captioning. Such rapid progress in NLG has necessitated the development of&#10;accurate automatic evaluation metrics that would allow us to track the progress&#10;in the field of NLG. However, unlike classification tasks, automatically&#10;evaluating NLG systems in itself is a huge challenge. Several works have shown&#10;that early heuristic-based metrics such as BLEU, ROUGE are inadequate for&#10;capturing the nuances in the different NLG tasks. The expanding number of NLG&#10;models and the shortcomings of the current metrics has led to a rapid surge in&#10;the number of evaluation metrics proposed since 2014. Moreover, various&#10;evaluation metrics have shifted from using pre-determined heuristic-based&#10;formulae to trained transformer models. This rapid change in a relatively short&#10;time has led to the need for a survey of the existing NLG metrics to help&#10;existing and new researchers to quickly come up to speed with the developments&#10;that have happened in NLG evaluation in the last few years. Through this&#10;survey, we first wish to highlight the challenges and difficulties in&#10;automatically evaluating NLG systems. Then, we provide a coherent taxonomy of&#10;the evaluation metrics to organize the existing metrics and to better&#10;understand the developments in the field. We also describe the different&#10;metrics in detail and highlight their key contributions. Later, we discuss the&#10;main shortcomings identified in the existing metrics and describe the&#10;methodology used to evaluate evaluation metrics. Finally, we discuss our&#10;suggestions and recommendations on the next steps forward to improve the&#10;automatic evaluation metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.08131" label="2002.08131">
        <attvalues>
          <attvalue for="0" value="A Systematic Comparison of Architectures for Document-Level Sentiment&#10;  Classification" />
          <attvalue for="1" value="  Documents are composed of smaller pieces - paragraphs, sentences, and tokens&#10;- that have complex relationships between one another. Sentiment classification&#10;models that take into account the structure inherent in these documents have a&#10;theoretical advantage over those that do not. At the same time, transfer&#10;learning models based on language model pretraining have shown promise for&#10;document classification. However, these two paradigms have not been&#10;systematically compared and it is not clear under which circumstances one&#10;approach is better than the other. In this work we empirically compare&#10;hierarchical models and transfer learning for document-level sentiment&#10;classification. We show that non-trivial hierarchical models outperform&#10;previous baselines and transfer learning on document-level sentiment&#10;classification in five languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;The inherent structure found in documents -- paragraphs, sentences, and tokens -- and their inter-dependence is vital to document-level sentiment, as rhetorical devices and anaphora relationships disperse the sentiment signal across the various sub-components \cite{yang-cardie-2014-context}. This also means that not all sub-components contribute equally towards identifying the overall polarity of a document \cite{yu-hatzivassiloglou-2003-towards,pang-lee-2004-sentimental} and models that are able to take these relationships into account should theoretically perform better.&#10;&#10;Recently, two divergent research directions have shown promise for document-classification: on the one hand, transfer learning \cite{peters-etal-2018-deep,howard-ruder-2018-universal,devlin-etal-2019-bert} and on the other hand hierarchical modeling \cite{Xia:Cho:16,Con:Sch:Bar:17,yang-etal-2016-hierarchical}. Transfer learning (in its current form) attempts to take advantage of large amounts of unlabeled text in order to improve contextualized representations of tokens, while ignoring the structure of documents. Hierarchical models, on the other hand, attempt to take document structure into account by first building up representations for sentences and then aggregating them to create document representations.&#10;&#10;While the two approaches are complementary in the sense that one could use pretrained LMs for transfer-learning also for hierarchical models, we here focus on isolating their relative strengths and weaknesses. In this paper we empirically show that methods which explicitly incorporate the structure of documents outperform those that do not and further examine the influence of data characteristics such as document length and size of training data on the choice of architecture. &#10;Finally, we release the code to reproduce the results from our study. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Computational Linguistics, Document Structure Analysis, Mathematics, Sentiment Classification Models, Natural Language Processing, Transfer Learning Methods" />
        </attvalues>
      </node>
      <node id="1602.00367" label="1602.00367">
        <attvalues>
          <attvalue for="0" value="Efficient Character-level Document Classification by Combining&#10;  Convolution and Recurrent Layers" />
          <attvalue for="1" value="  Document classification tasks were primarily tackled at word level. Recent&#10;research that works with character-level inputs shows several benefits over&#10;word-level approaches such as natural incorporation of morphemes and better&#10;handling of rare words. We propose a neural network architecture that utilizes&#10;both convolution and recurrent layers to efficiently encode character inputs.&#10;We validate the proposed model on eight large scale document classification&#10;tasks and compare with character-level convolution-only models. It achieves&#10;comparable performances with much less parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.01781" label="1606.01781">
        <attvalues>
          <attvalue for="0" value="Very Deep Convolutional Networks for Text Classification" />
          <attvalue for="1" value="  The dominant approach for many NLP tasks are recurrent neural networks, in&#10;particular LSTMs, and convolutional neural networks. However, these&#10;architectures are rather shallow in comparison to the deep convolutional&#10;networks which have pushed the state-of-the-art in computer vision. We present&#10;a new architecture (VDCNN) for text processing which operates directly at the&#10;character level and uses only small convolutions and pooling operations. We are&#10;able to show that the performance of this model increases with depth: using up&#10;to 29 convolutional layers, we report improvements over the state-of-the-art on&#10;several public text classification tasks. To the best of our knowledge, this is&#10;the first time that very deep convolutional nets have been applied to text&#10;processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.08067" label="2010.08067">
        <attvalues>
          <attvalue for="0" value="Montague Grammar Induction" />
          <attvalue for="1" value="  We propose a computational modeling framework for inducing combinatory&#10;categorial grammars from arbitrary behavioral data. This framework provides the&#10;analyst fine-grained control over the assumptions that the induced grammar&#10;should conform to: (i) what the primitive types are; (ii) how complex types are&#10;constructed; (iii) what set of combinators can be used to combine types; and&#10;(iv) whether (and to what) the types of some lexical items should be fixed. In&#10;a proof-of-concept experiment, we deploy our framework for use in&#10;distributional analysis. We focus on the relationship between&#10;s(emantic)-selection and c(ategory)-selection, using as input a lexicon-scale&#10;acceptability judgment dataset focused on English verbs' syntactic distribution&#10;(the MegaAcceptability dataset) and enforcing standard assumptions from the&#10;semantics literature on the induced grammar.&#10;" />
          <attvalue for="2" value="&#10;&#10;Semantic theories aim to capture two kinds of facts about languages' expressions: (i) their distributional characteristics; and (ii) their inferential affordances. The descriptive adequacy of any such theory is evaluated in terms of its coverage of these facts---an evaluation that is commonly carried out informally on a relatively small number of test cases. While this approach to theory-building and evaluation has yielded deep insights, it also carries significant risks: generalizations that appear unassailable based a small number of high-frequency examples (and theories built on them) can collapse when evaluated on a more diverse range of expressions purportedly covered by the generalization (see \cite{white_believing_accepted} for recent discussion). &#10;&#10;Ameliorating these risks requires developing scalable methods for building and evaluating semantic theories. Until recently, a major obstacle to developing such methods was that sufficiently large-scale behavioral datasets were not available. This situation has changed with the advent of lexicon-scale acceptability and inference judgment datasets, such as the \href{ datasets \cite{white_computational_2016,white_role_2018,white_frequency_accepted,white_lexicosyntactic_2018,an_lexical_2020,moon_source_toappear}. Concomitant advances in computational modeling have cleared the way for automating distributional and inferential analysis for the purposes of theory-building and evaluation. The remaining challenge is one of integration. On the one hand, powerful methods for learning structured representations from corpus data and (to some extent) behavioral data now exist \cite{le_inside-outside_2014,le_compositional_2015,williams_latent_2018,shen_neural_2018,kim_unsupervised_2019,drozdov_unsupervised_labeled_2019,drozdov_unsupervised_latent_2019}, but the relationship between these models' representations and grammars posited under standard frameworks (\cite{montague_proper_1973} et seq) assuming some form of (combinatory) categorial grammar \cite{steedman_syntactic_2000} remains unclear. On the other hand, powerful methods for inducing such grammars now exist \cite{zettlemoyer_learning_2005,zettlemoyer_online_2007,zettlemoyer_learning_2009,kwiatkowksi_inducing_2010,kwiatkowski_lexical_2011,bisk_induction_2012,bisk_simple_2012,bisk_hdp_2013,bisk_probing_2015}, but they do not straightforwardly generalize to the full range of behavioral data of interest in experimental semantics.&#10;&#10;To address these limitations, we propose a general deep learning-based, computational modeling framework for inducing full-fledged combinatory categorial grammars from multiple distinct types of behavioral data. Beyond providing the ability to synthesize arbitrary distributional and inferential data within a single model, our framework provides the analyst fine-grained control over the assumptions that the induced grammar should conform to: (i) what the primitive types are (e.g. $\sffamily e$, $\sffamily s$, $\sffamily t$, etc.); (ii) how complex types are constructed (e.g. that $\langle t_1, t_2 \rangle$ is a type if $t_1$ and $t_2$ are types); (iii) what set of combinators can be used to combine types (e.g. application, composition, etc.); and (iv) whether (and to what) the types of some lexical items should be fixed. As a proof of concept, we deploy our framework for use in distributional analysis. We focus, in particular, on the relationship between s(emantic)-selection and c(ategory)-selection, using as input a lexicon-scale acceptability judgment dataset focused on English verbs' syntactic distribution (the \href{ dataset; \cite{white_computational_2016,white_frequency_accepted}) and enforcing standard assumptions from the semantics literature on the induced grammar. As a case study, we analyze the typing that the induced grammar infers for clausal complements. Clausal complements are useful in this regard, since their syntactic complexity provides a rigorous test of our framework's ability to recover interpretable types.&#10;&#10;We begin with some brief background on the deep learning-based approach to grammar induction that we build on (\S\ref{sec:background}). We then describe how we extend that approach to learn full-fledge combinatory categorial grammars (\S\ref{sec:computational-model}) before turning to our proof-of-concept experiments with the MegaAcceptability dataset (\S\ref{sec:experiments}) and the results of these experiments (\S\ref{sec:results}). We conclude with a discussion of future directions for our framework (\S\ref{sec:conclusion}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Computational Modeling, Linguistics, Distributional Analysis, Computational Linguistics, Artificial Intelligence, Mathematics, Categorial Grammars" />
        </attvalues>
      </node>
      <node id="1808.06232" label="1808.06232">
        <attvalues>
          <attvalue for="0" value="Lexicosyntactic Inference in Neural Models" />
          <attvalue for="1" value="  We investigate neural models' ability to capture lexicosyntactic inferences:&#10;inferences triggered by the interaction of lexical and syntactic information.&#10;We take the task of event factuality prediction as a case study and build a&#10;factuality judgment dataset for all English clause-embedding verbs in various&#10;syntactic contexts. We use this dataset, which we make publicly available, to&#10;probe the behavior of current state-of-the-art neural systems, showing that&#10;these systems make certain systematic errors that are clearly visible through&#10;the lens of factuality prediction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.05253" label="1908.05253">
        <attvalues>
          <attvalue for="0" value="The lexical and grammatical sources of neg-raising inferences" />
          <attvalue for="1" value="  We investigate neg(ation)-raising inferences, wherein negation on a predicate&#10;can be interpreted as though in that predicate's subordinate clause. To do&#10;this, we collect a large-scale dataset of neg-raising judgments for effectively&#10;all English clause-embedding verbs and develop a model to jointly induce the&#10;semantic types of verbs and their subordinate clauses and the relationship of&#10;these types to neg-raising inferences. We find that some neg-raising inferences&#10;are attributable to properties of particular predicates, while others are&#10;attributable to subordinate clause structure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1503.02510" label="1503.02510">
        <attvalues>
          <attvalue for="0" value="Compositional Distributional Semantics with Long Short Term Memory" />
          <attvalue for="1" value="  We are proposing an extension of the recursive neural network that makes use&#10;of a variant of the long short-term memory architecture. The extension allows&#10;information low in parse trees to be stored in a memory register (the `memory&#10;cell') and used much later higher up in the parse tree. This provides a&#10;solution to the vanishing gradient problem and allows the network to capture&#10;long range dependencies. Experimental results show that our composition&#10;outperformed the traditional neural-network composition on the Stanford&#10;Sentiment Treebank.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1207.1420" label="1207.1420">
        <attvalues>
          <attvalue for="0" value="Learning to Map Sentences to Logical Form: Structured Classification&#10;  with Probabilistic Categorial Grammars" />
          <attvalue for="1" value="  This paper addresses the problem of mapping natural language sentences to&#10;lambda-calculus encodings of their meaning. We describe a learning algorithm&#10;that takes as input a training set of sentences labeled with expressions in the&#10;lambda calculus. The algorithm induces a grammar for the problem, along with a&#10;log-linear model that represents a distribution over syntactic and semantic&#10;analyses conditioned on the input sentence. We apply the method to the task of&#10;learning natural language interfaces to databases and show that the learned&#10;parsers outperform previous methods in two benchmark database domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.11047" label="1908.11047">
        <attvalues>
          <attvalue for="0" value="Shallow Syntax in Deep Water" />
          <attvalue for="1" value="  Shallow syntax provides an approximation of phrase-syntactic structure of&#10;sentences; it can be produced with high accuracy, and is computationally cheap&#10;to obtain. We investigate the role of shallow syntax-aware representations for&#10;NLP tasks using two techniques. First, we enhance the ELMo architecture to&#10;allow pretraining on predicted shallow syntactic parses, instead of just raw&#10;text, so that contextual embeddings make use of shallow syntactic context. Our&#10;second method involves shallow syntactic features obtained automatically on&#10;downstream task data. Neither approach leads to a significant gain on any of&#10;the four downstream tasks we considered relative to ELMo-only baselines.&#10;Further analysis using black-box probes confirms that our shallow-syntax-aware&#10;contextual embeddings do not transfer to linguistic tasks any more easily than&#10;ELMo's embeddings. We take these findings as evidence that ELMo-style&#10;pretraining discovers representations which make additional awareness of&#10;shallow syntax redundant.&#10;" />
          <attvalue for="2" value="&#10;&#10;The NLP community is revisiting the role of linguistic structure in applications with the advent of contextual word representations (\cwrs) derived from pretraining language models on large corpora~\cite{Peters:18,Radford:18,Howard:18,Devlin:18}.&#10;Recent work has shown that downstream task performance may benefit from explicitly injecting a syntactic inductive bias into model architectures \cite{Kuncoro:18}, even when \cwrs are also used \cite{Strubell:18}.&#10;However, high quality linguistic structure annotation at a large scale remains expensive---a trade-off needs to be made between the quality of the annotations and the computational expense of obtaining them.&#10;Shallow syntactic structures (\cite{Abney:91}; also called chunk sequences) offer a viable middle ground, by providing a flat, non-hierarchical approximation to phrase-syntactic trees (see Fig.~\ref{fig:chunk_tree} for an example).&#10;These structures can be obtained efficiently, and with high accuracy, using sequence labelers.&#10;In this paper we consider shallow syntax to be a proxy for linguistic structure. &#10;&#10;While shallow syntactic chunks are almost as ubiquitous as part-of-speech tags in standard NLP pipelines \cite{Jurafsky:00}, their relative merits in the presence of \cwrs remain unclear.&#10;We investigate the role of these structures using two methods.&#10;First, we enhance the ELMo architecture \cite{Peters:18b} to allow pretraining on predicted shallow syntactic parses, instead of just raw text, so that contextual embeddings make use of shallow syntactic context (\sect{sec:pretraining}).&#10;Our second method involves classical addition of chunk features to \cwr-infused architectures for four different downstream tasks (\S\ref{sec:shallow_features}).&#10;Shallow syntactic information is obtained automatically using a highly accurate model (97\% $F_1$ on standard benchmarks).&#10;In both settings, we observe only modest gains on three of the four downstream tasks relative to ELMo-only baselines (\sect{sec:experiments}).&#10;&#10;Recent work has probed the knowledge encoded in \cwrs and found they capture a surprisingly large amount of syntax~\cite{Blevins:18,Liu:19,Tenney:18}. &#10;We further examine the contextual embeddings obtained from the enhanced architecture and a shallow syntactic context, using black-box probes from \cite{Liu:19}.&#10;Our analysis indicates that our shallow-syntax-aware contextual embeddings do not transfer to linguistic tasks any more easily than ELMo embeddings (\sect{sec:probes}).&#10;&#10;Overall, our findings show that while shallow syntax can be somewhat useful, ELMo-style pretraining discovers representations which make additional awareness of shallow syntax largely redundant.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Learning Architectures, Artificial Intelligence, Natural Language Processing, Syntax Aware Representations" />
        </attvalues>
      </node>
      <node id="1804.08199" label="1804.08199">
        <attvalues>
          <attvalue for="0" value="Linguistically-Informed Self-Attention for Semantic Role Labeling" />
          <attvalue for="1" value="  Current state-of-the-art semantic role labeling (SRL) uses a deep neural&#10;network with no explicit linguistic features. However, prior work has shown&#10;that gold syntax trees can dramatically improve SRL decoding, suggesting the&#10;possibility of increased accuracy from explicit modeling of syntax. In this&#10;work, we present linguistically-informed self-attention (LISA): a neural&#10;network model that combines multi-head self-attention with multi-task learning&#10;across dependency parsing, part-of-speech tagging, predicate detection and SRL.&#10;Unlike previous models which require significant pre-processing to prepare&#10;linguistic features, LISA can incorporate syntax using merely raw tokens as&#10;input, encoding the sequence only once to simultaneously perform parsing,&#10;predicate detection and role labeling for all predicates. Syntax is&#10;incorporated by training one attention head to attend to syntactic parents for&#10;each token. Moreover, if a high-quality syntactic parse is already available,&#10;it can be beneficially injected at test time without re-training our SRL model.&#10;In experiments on CoNLL-2005 SRL, LISA achieves new state-of-the-art&#10;performance for a model using predicted predicates and standard word&#10;embeddings, attaining 2.5 F1 absolute higher than the previous state-of-the-art&#10;on newswire and more than 3.5 F1 on out-of-domain data, nearly 10% reduction in&#10;error. On ConLL-2012 English SRL we also show an improvement of more than 2.5&#10;F1. LISA also out-performs the state-of-the-art with contextually-encoded&#10;(ELMo) word representations, by nearly 1.0 F1 on news and more than 2.0 F1 on&#10;out-of-domain text.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.12452" label="2205.12452">
        <attvalues>
          <attvalue for="0" value="Sparse*BERT: Sparse Models Generalize To New tasks and Domains" />
          <attvalue for="1" value="  Large Language Models have become the core architecture upon which most&#10;modern natural language processing (NLP) systems build. These models can&#10;consistently deliver impressive accuracy and robustness across tasks and&#10;domains, but their high computational overhead can make inference difficult and&#10;expensive. To make using these models less costly, recent work has explored&#10;leveraging structured and unstructured pruning, quantization, and distillation&#10;to improve inference speed and decrease size. This paper studies how models&#10;pruned using Gradual Unstructured Magnitude Pruning can transfer between&#10;domains and tasks. Our experimentation shows that models that are pruned during&#10;pretraining using general domain masked language models can transfer to novel&#10;domains and tasks without extensive hyperparameter exploration or specialized&#10;approaches. We demonstrate that our general sparse model Sparse*BERT can become&#10;SparseBioBERT simply by pretraining the compressed architecture on unstructured&#10;biomedical text. Moreover, we show that SparseBioBERT can match the quality of&#10;BioBERT with only 10\% of the parameters.&#10;" />
          <attvalue for="2" value="&#10;Foundational Models \cite{Bommasani2021OnTO} based on the Transformer architecture \cite{Vaswani2017AttentionIA} has quickly become the most common building block in the modern language understanding stack, &#10;providing robust language representations which can be leveraged to provide impressive accuracy on tasks like question answering, text classification, and token classification. These Large Language Models (LLMs) are able to adapt to novel domains through pretraining resulting in models like BioBERT \cite{Lee2020BioBERTAP}, LegalBERT \cite{Chalkidis2020LEGALBERTTM}, and SciBERT \cite{beltagy2019SciBERTAP} have become a popular strategy for improving performance further. While accurate and robust, LLMs are not without drawbacks. &#10;They commonly have hundreds of millions or billions of parameters requiring large specialized computer clusters to run inference at scale. Several approaches have been successfully used to improve the performance of these LLMs, such as approximating attention \cite{Peng2021RandomFA}, removing portions of the models \cite{Sridhar2020UndividedAA}, and reducing the precision of activation and weight values.\\&#10;&#10;Recent work \cite{Zafrir2021PruneOF} \cite{Kurti2022TheOB} has shown that the application of unstructured and semi-structured (block) pruning mechanisms on LLMs can significantly compress models with little to no loss in accuracy. While these approaches are successful and applicable during model general domain pretraining and task-specific fine-tuning, prior work has not studied how pruned models transfer to new domains nor the impact of pretraining stage pruning on transfer accuracy. Given that most applications would require the transfer of the general domain LLMs to a specific application domain, it is important to study the generality and robustness of the pruned LLMs when applied to multiple tasks in an application domain. \\&#10;While existing pruning research has found it possible to prune models heavily without loss in accuracy, most approaches have focused on the compression of individual tasks or textual domains. These specialized models match or exceed the accuracy of the dense model but commonly require vast amounts of hyperparameter turning and task-specific optimization to achieve this result. Compressed models like DistillBERT \cite{Sanh2019DistilBERTAD} and TinyBERT \cite{Jiao2020TinyBERTDB} are some of the most popular LLMs because they provide compression without any additional know-how or optimization. For pruned models to become a common architecture for NLP tasks, they must be as robust and easy to use as their uncompressed counterparts. \\&#10;This paper explores this potential and proposes Sparse*BERT, a new pruned LLM that can adapt effectively to new domains without extensive fine-tuning or task-specific pruning.\\&#10;Our work studies generalizable pruned LLMs by evaluating how well they can transfer to previously unstudied tasks in the biomedical domain.&#10;Specifically, we study these questions by focusing on transferring pruned and unpruned LLMs to the biomedical domain and evaluating the accuracy of said models on downstream tasks like Entity Extraction(EE), Relation Extraction(RE), and Question Answering(QA). Our experiments demonstrate that pruned LLMs generalize well and are robust to domain transfer and variation in target task, dataset size, and dataset difficulty. In summary, our contributions are as follows:&#10;\begin{itemize} &#10;\item We reinforce Zafrir et al.'s findings that LLMs pruned on the general domain data can transfer to new domains without extensive hyperparameter tuning and extend their work, demonstrating these pruned models can be transferred to new pre-training domains without additional parameter optimization.&#10;\item We introduce a sparse model, Sparse*BERT, and its domain adaptation adapted for the Medical/Bio NLP domain called SparseBioBERT. This model matches the accuracy of the BioBERT with $10\%$ of the active parameters.&#10;\end{itemize}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Pruning Techniques, Mathematics, Information Science, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2012.11881" label="2012.11881">
        <attvalues>
          <attvalue for="0" value="Undivided Attention: Are Intermediate Layers Necessary for BERT?" />
          <attvalue for="1" value="  In recent times, BERT-based models have been extremely successful in solving&#10;a variety of natural language processing (NLP) tasks such as reading&#10;comprehension, natural language inference, sentiment analysis, etc. All&#10;BERT-based architectures have a self-attention block followed by a block of&#10;intermediate layers as the basic building component. However, a strong&#10;justification for the inclusion of these intermediate layers remains missing in&#10;the literature. In this work we investigate the importance of intermediate&#10;layers on the overall network performance of downstream tasks. We show that&#10;reducing the number of intermediate layers and modifying the architecture for&#10;BERT-BASE results in minimal loss in fine-tuning accuracy for downstream tasks&#10;while decreasing the number of parameters and training time of the model.&#10;Additionally, we use centered kernel alignment and probing linear classifiers&#10;to gain insight into our architectural modifications and justify that removal&#10;of intermediate layers has little impact on the fine-tuned accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.05754" label="2111.05754">
        <attvalues>
          <attvalue for="0" value="Prune Once for All: Sparse Pre-Trained Language Models" />
          <attvalue for="1" value="  Transformer-based language models are applied to a wide range of applications&#10;in natural language processing. However, they are inefficient and difficult to&#10;deploy. In recent years, many compression algorithms have been proposed to&#10;increase the implementation efficiency of large Transformer-based models on&#10;target hardware. In this work we present a new method for training sparse&#10;pre-trained Transformer language models by integrating weight pruning and model&#10;distillation. These sparse pre-trained models can be used to transfer learning&#10;for a wide range of tasks while maintaining their sparsity pattern. We&#10;demonstrate our method with three known architectures to create sparse&#10;pre-trained BERT-Base, BERT-Large and DistilBERT. We show how the compressed&#10;sparse pre-trained models we trained transfer their knowledge to five different&#10;downstream natural language tasks with minimal accuracy loss. Moreover, we show&#10;how to further compress the sparse models' weights to 8bit precision using&#10;quantization-aware training. For example, with our sparse pre-trained&#10;BERT-Large fine-tuned on SQuADv1.1 and quantized to 8bit we achieve a&#10;compression ratio of $40$X for the encoder with less than $1\%$ accuracy loss.&#10;To the best of our knowledge, our results show the best compression-to-accuracy&#10;ratio for BERT-Base, BERT-Large, and DistilBERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.07259" label="2203.07259">
        <attvalues>
          <attvalue for="0" value="The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for&#10;  Large Language Models" />
          <attvalue for="1" value="  Transformer-based language models have become a key building block for&#10;natural language processing. While these models are extremely accurate, they&#10;can be too large and computationally intensive to run on standard deployments.&#10;A variety of compression methods, including distillation, quantization,&#10;structured and unstructured pruning are known to decrease model size and&#10;increase inference speed, with low accuracy loss. In this context, this paper's&#10;contributions are two-fold. We perform an in-depth study of the&#10;accuracy-compression trade-off for unstructured weight pruning of BERT models.&#10;We introduce Optimal BERT Surgeon (oBERT), an efficient and accurate weight&#10;pruning method based on approximate second-order information, which we show to&#10;yield state-of-the-art results in both stages of language tasks: pre-training&#10;and fine-tuning. Specifically, oBERT extends existing work on unstructured&#10;second-order pruning by allowing for pruning blocks of weights, and by being&#10;applicable at the BERT scale. Second, we investigate the impact of this pruning&#10;method when compounding compression approaches to obtain highly compressed but&#10;accurate models for deployment on edge devices. These models significantly push&#10;boundaries of the current state-of-the-art sparse BERT models with respect to&#10;all metrics: model size, inference speed and task accuracy. For example,&#10;relative to the dense BERT-base, we obtain 10x model size compression (in MB)&#10;with &lt; 1% accuracy drop, 10x CPU-inference speedup with &lt; 2% accuracy drop, and&#10;29x CPU-inference speedup with &lt; 7.5% accuracy drop. Our code, fully integrated&#10;with Transformers and SparseML, is available at&#10;https://github.com/neuralmagic/sparseml/tree/main/research/optimal_BERT_surgeon_oBERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.15282" label="2210.15282">
        <attvalues>
          <attvalue for="0" value="Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic&#10;  Forgetting in Automatic Speech Recognition" />
          <attvalue for="1" value="  Adapting a trained Automatic Speech Recognition (ASR) model to new tasks&#10;results in catastrophic forgetting of old tasks, limiting the model's ability&#10;to learn continually and to be extended to new speakers, dialects, languages,&#10;etc. Focusing on End-to-End ASR, in this paper, we propose a simple yet&#10;effective method to overcome catastrophic forgetting: weight averaging. By&#10;simply taking the average of the previous and the adapted model, our method&#10;achieves high performance on both the old and new tasks. It can be further&#10;improved by introducing a knowledge distillation loss during the adaptation. We&#10;illustrate the effectiveness of our method on both monolingual and multilingual&#10;ASR. In both cases, our method strongly outperforms all baselines, even in its&#10;simplest form.&#10;" />
          <attvalue for="2" value="&#10;Catastrophic Forgetting (CF) \cite{catastrophicforgetting} remains a problem when adapting a trained Automatic Speech Recognition (ASR) model to new tasks, regardless of whether the new tasks are new languages, dialects, accents or simply new speakers. CF thus severely limits the ability of ASR models to be extended to new domains (e.g. to build very powerful ASR models able to perform well for all dialects, accents, speakers, etc.), or to exploit all the (recently become available) data at its disposal. To overcome CF, one needs to re-introduce all past data when extending the ASR models, which soon becomes very expensive in terms time, energy and resources.&#10;&#10;Alternatively, one could use Continual Learning (CL) methods to enable models to learn continually without suffering from CF. CL has been a hot research topic in image classification, with many CL methods being proposed. These CL methods can be grouped into three categories \cite{defy}: (i) regularization-based methods add a term to the loss to regularize training, e.g. \cite{ewc, mas, imm}; (ii) rehearsal-based methods rehearse previous tasks through a small memory of samples from previous tasks, e.g. \cite{agem, er}; (iii) architectural-based methods increase the model capacity when learning new tasks, e.g. \cite{pnn}. For ASR, CL is still a relatively new topic. \cite{lifelongasr} and \cite{eeckt2021continual} implement, respectively, four and nine CL methods for (End-to-End) E2E ASR, both finding that rehearsal-based methods remain the most practical way to overcome CF. \cite{ogem} applies Gradient Episodic Memory \cite{gem}, also a rehearsal-based method, to E2E ASR, while assuming that task boundaries are not known. However, the disadvantage of rehearsal-based methods is that they require storing a memory (i.e. a set of samples from previous tasks), which may not always be allowed due to privacy concerns. \cite{eeckt_adapters} finds that using rehearsal-based methods is not necessary to prevent CF when task-specific adapters \cite{adapters} are used, which comes, however, at the cost of introducing task-specific parameters, possibly requiring a task label at inference time. &#10;&#10;Similar to \cite{lifelongasr, eeckt2021continual, eeckt_adapters}, this paper focuses on CL (with known task boundaries) for E2E ASR, and proposes a simple yet very effective method to overcome CF: weight averaging. By simply computing the (weighted) average of the model before and the model after adaptation to the new task, CF can be prevented while learning the new task well. Though already very effective by itself, weight averaging can be further improved by introducing knowledge distillation losses (as in \cite{lwf}) during the adaptation. Overall, our method remains very simple, while, compared to the best methods from \cite{lifelongasr, eeckt2021continual}, not requiring a memory. We illustrate the effectiveness of our method on both monolingual and multilingual experiments, consisting of six and five tasks, respectively. On both experiments, our method strongly outperforms all baselines, even in its simplest form.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Adaptation, Continual Learning, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1812.00420" label="1812.00420">
        <attvalues>
          <attvalue for="0" value="Efficient Lifelong Learning with A-GEM" />
          <attvalue for="1" value="  In lifelong learning, the learner is presented with a sequence of tasks,&#10;incrementally building a data-driven prior which may be leveraged to speed up&#10;learning of a new task. In this work, we investigate the efficiency of current&#10;lifelong approaches, in terms of sample complexity, computational and memory&#10;cost. Towards this end, we first introduce a new and a more realistic&#10;evaluation protocol, whereby learners observe each example only once and&#10;hyper-parameter selection is done on a small and disjoint set of tasks, which&#10;is not used for the actual learning experience and evaluation. Second, we&#10;introduce a new metric measuring how quickly a learner acquires a new skill.&#10;Third, we propose an improved version of GEM (Lopez-Paz &amp; Ranzato, 2017),&#10;dubbed Averaged GEM (A-GEM), which enjoys the same or even better performance&#10;as GEM, while being almost as computationally and memory efficient as EWC&#10;(Kirkpatrick et al., 2016) and other regularization-based methods. Finally, we&#10;show that all algorithms including A-GEM can learn even more quickly if they&#10;are provided with task descriptors specifying the classification tasks under&#10;consideration. Our experiments on several standard lifelong learning benchmarks&#10;demonstrate that A-GEM has the best trade-off between accuracy and efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.04671" label="1606.04671">
        <attvalues>
          <attvalue for="0" value="Progressive Neural Networks" />
          <attvalue for="1" value="  Learning to solve complex sequences of tasks--while both leveraging transfer&#10;and avoiding catastrophic forgetting--remains a key obstacle to achieving&#10;human-level intelligence. The progressive networks approach represents a step&#10;forward in this direction: they are immune to forgetting and can leverage prior&#10;knowledge via lateral connections to previously learned features. We evaluate&#10;this architecture extensively on a wide variety of reinforcement learning tasks&#10;(Atari and 3D maze games), and show that it outperforms common baselines based&#10;on pretraining and finetuning. Using a novel sensitivity measure, we&#10;demonstrate that transfer occurs at both low-level sensory and high-level&#10;control layers of the learned policy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.01616" label="2104.01616">
        <attvalues>
          <attvalue for="0" value="Towards Lifelong Learning of End-to-end ASR" />
          <attvalue for="1" value="  Automatic speech recognition (ASR) technologies today are primarily optimized&#10;for given datasets; thus, any changes in the application environment (e.g.,&#10;acoustic conditions or topic domains) may inevitably degrade the performance.&#10;We can collect new data describing the new environment and fine-tune the&#10;system, but this naturally leads to higher error rates for the earlier&#10;datasets, referred to as catastrophic forgetting. The concept of lifelong&#10;learning (LLL) aiming to enable a machine to sequentially learn new tasks from&#10;new datasets describing the changing real world without forgetting the&#10;previously learned knowledge is thus brought to attention. This paper reports,&#10;to our knowledge, the first effort to extensively consider and analyze the use&#10;of various approaches of LLL in end-to-end (E2E) ASR, including proposing novel&#10;methods in saving data for past domains to mitigate the catastrophic forgetting&#10;problem. An overall relative reduction of 28.7% in WER was achieved compared to&#10;the fine-tuning baseline when sequentially learning on three very different&#10;benchmark corpora. This can be the first step toward the highly desired ASR&#10;technologies capable of synchronizing with the continuously changing real&#10;world.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.05071" label="2207.05071">
        <attvalues>
          <attvalue for="0" value="Online Continual Learning of End-to-End Speech Recognition Models" />
          <attvalue for="1" value="  Continual Learning, also known as Lifelong Learning, aims to continually&#10;learn from new data as it becomes available. While prior research on continual&#10;learning in automatic speech recognition has focused on the adaptation of&#10;models across multiple different speech recognition tasks, in this paper we&#10;propose an experimental setting for \textit{online continual learning} for&#10;automatic speech recognition of a single task. Specifically focusing on the&#10;case where additional training data for the same task becomes available&#10;incrementally over time, we demonstrate the effectiveness of performing&#10;incremental model updates to end-to-end speech recognition models with an&#10;online Gradient Episodic Memory (GEM) method. Moreover, we show that with&#10;online continual learning and a selective sampling strategy, we can maintain an&#10;accuracy that is similar to retraining a model from scratch while requiring&#10;significantly lower computation costs. We have also verified our method with&#10;self-supervised learning (SSL) features.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.09282" label="1606.09282">
        <attvalues>
          <attvalue for="0" value="Learning without Forgetting" />
          <attvalue for="1" value="  When building a unified vision system or gradually adding new capabilities to&#10;a system, the usual assumption is that training data for all tasks is always&#10;available. However, as the number of tasks grows, storing and retraining on&#10;such data becomes infeasible. A new problem arises where we add new&#10;capabilities to a Convolutional Neural Network (CNN), but the training data for&#10;its existing capabilities are unavailable. We propose our Learning without&#10;Forgetting method, which uses only new task data to train the network while&#10;preserving the original capabilities. Our method performs favorably compared to&#10;commonly used feature extraction and fine-tuning adaption techniques and&#10;performs similarly to multitask learning that uses original task data we assume&#10;unavailable. A more surprising observation is that Learning without Forgetting&#10;may be able to replace fine-tuning with similar old and new task datasets for&#10;improved new task performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.18031" label="2402.18031">
        <attvalues>
          <attvalue for="0" value="Corpus-Steered Query Expansion with Large Language Models" />
          <attvalue for="1" value="  Recent studies demonstrate that query expansions generated by large language&#10;models (LLMs) can considerably enhance information retrieval systems by&#10;generating hypothetical documents that answer the queries as expansions.&#10;However, challenges arise from misalignments between the expansions and the&#10;retrieval corpus, resulting in issues like hallucinations and outdated&#10;information due to the limited intrinsic knowledge of LLMs. Inspired by Pseudo&#10;Relevance Feedback (PRF), we introduce Corpus-Steered Query Expansion (CSQE) to&#10;promote the incorporation of knowledge embedded within the corpus. CSQE&#10;utilizes the relevance assessing capability of LLMs to systematically identify&#10;pivotal sentences in the initially-retrieved documents. These corpus-originated&#10;texts are subsequently used to expand the query together with LLM-knowledge&#10;empowered expansions, improving the relevance prediction between the query and&#10;the target documents. Extensive experiments reveal that CSQE exhibits strong&#10;performance without necessitating any training, especially with queries for&#10;which LLMs lack knowledge.&#10;" />
          <attvalue for="2" value="&#10;&#10;Query expansion enhances the effectiveness of information retrieval systems by incorporating additional texts into the original query, which are traditionally identified via pseudo-relevance feedback~\cite{prf1, prf2} or by leveraging external lexical knowledge sources~\cite{lexical_expansion1, lexical_expansion_2}. Recent studies~\cite{{hyde, query2doc, prompt_qe, generative_relevance_feedback}} show query expansions generated by LLMs are able to significantly boost retrieval effectiveness, especially in zero-shot scenarios. For instance, \cite{hyde} demonstrates the effectiveness of utilizing LLMs to generate hypothetical documents answering the original query as additional texts to augment the query. \cite{generative_relevance_feedback}&#10;show the efficacy of applying pseudo-relevance feedback upon the LLM-generated answers for expansion. Despite variations in prompts or expansion methods, a common foundational element across these approaches is the reliance on the intrinsic knowledge of LLMs.&#10;&#10;Despite their effectiveness, generations that rely on the intrinsic parametric knowledge within LLMs encounter various issues. These include hallucination~\cite{hallucination_1}, inability to update~\cite{timeqa}, and a deficiency in long-tail knowledge~\cite{long-tail}. Such generations may introduce irrelevant or misleading texts, degrading retrieval performance~\cite{expan_fail}. These query expansions can be seen as an evolution of earlier query expansions reliant on external lexical knowledge. In contrast, tradition PRF that typically chooses additional texts from the top-retrieved documents, has received less attention. However, given that the expanded texts are sourced directly from the original documents, these methods hold significant potential for enhancing factuality.&#10;&#10;To this end, we propose Corpus-Steered&#10;Query Expansion (CSQE). Unlike methods that rely on the intrinsic parametric knowledge of LLMs, CSQE exclusively leverages the strong relevance assessing capability of LLMs~\cite{relevance_assess_1, relevance_assess_2}.&#10;As illustrated in Figure~\ref{fig:illustration}, given a query and its initially retrieved documents, CSQE utilizes a LLM to first identify relevant documents to the query and then extracts pivotal sentences that contribute to their relevance. These corpus-originated texts are then combined together with LLM-knowledge empowered expansions to expand the original query.&#10;By incorporating query expansions that strictly originate from the corpus, CSQE balances out the limitations commonly found in LLM-knowledge empowered expansions.&#10;&#10;To sum up, our contributions are 3-fold:&#10;&#10;1) We propose CSQE, which exclusively exploits the relevance assessing capability of LLMs to overcome the hinderance posed by LLM-knowledge empowered expansions.&#10;&#10;2) Experimental results reveal that CSQE combined with a simple BM25 model, without necessitating any training, outperform not only LLM-knowledge empowered expansion methods but also the SOTA supervised Contriever$^FT$ model across two high-resource web search datasets and six low-resource BEIR datasets.&#10;&#10;3) Further analysis demonstrates the advantages of BM25 over dense retrieval with query expansion from LLMs, as well as query expansion over large-scale fine-tuning upon Contriever.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Language Models, Artificial Intelligence, Information Science, Query Expansion" />
        </attvalues>
      </node>
      <node id="2212.10496" label="2212.10496">
        <attvalues>
          <attvalue for="0" value="Precise Zero-Shot Dense Retrieval without Relevance Labels" />
          <attvalue for="1" value="  While dense retrieval has been shown effective and efficient across tasks and&#10;languages, it remains difficult to create effective fully zero-shot dense&#10;retrieval systems when no relevance label is available. In this paper, we&#10;recognize the difficulty of zero-shot learning and encoding relevance. Instead,&#10;we propose to pivot through Hypothetical Document Embeddings~(HyDE). Given a&#10;query, HyDE first zero-shot instructs an instruction-following language model&#10;(e.g. InstructGPT) to generate a hypothetical document. The document captures&#10;relevance patterns but is unreal and may contain false details. Then, an&#10;unsupervised contrastively learned encoder~(e.g. Contriever) encodes the&#10;document into an embedding vector. This vector identifies a neighborhood in the&#10;corpus embedding space, where similar real documents are retrieved based on&#10;vector similarity. This second step ground the generated document to the actual&#10;corpus, with the encoder's dense bottleneck filtering out the incorrect&#10;details. Our experiments show that HyDE significantly outperforms the&#10;state-of-the-art unsupervised dense retriever Contriever and shows strong&#10;performance comparable to fine-tuned retrievers, across various tasks (e.g. web&#10;search, QA, fact verification) and languages~(e.g. sw, ko, ja).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.13157" label="2304.13157">
        <attvalues>
          <attvalue for="0" value="Generative Relevance Feedback with Large Language Models" />
          <attvalue for="1" value="  Current query expansion models use pseudo-relevance feedback to improve&#10;first-pass retrieval effectiveness; however, this fails when the initial&#10;results are not relevant. Instead of building a language model from retrieved&#10;results, we propose Generative Relevance Feedback (GRF) that builds&#10;probabilistic feedback models from long-form text generated from Large Language&#10;Models. We study the effective methods for generating text by varying the&#10;zero-shot generation subtasks: queries, entities, facts, news articles,&#10;documents, and essays. We evaluate GRF on document retrieval benchmarks&#10;covering a diverse set of queries and document collections, and the results&#10;show that GRF methods significantly outperform previous PRF methods.&#10;Specifically, we improve MAP between 5-19% and NDCG@10 17-24% compared to RM3&#10;expansion, and achieve the best R@1k effectiveness on all datasets compared to&#10;state-of-the-art sparse, dense, and expansion models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Query Expansion:&#10;Lexical mismatch is a crucial issue in information retrieval, whereby a user query fails to capture their complete information need~\cite{belkin1982ask}. Query expansion methods~\cite{rocchio1971relevance} tackle this problem by incorporating terms closer to the user's intended meaning. One popular technique for automatic query expansion is pseudo-relevance feedback (PRF), where the top $k$ documents from the initial retrieval are assumed to be relevant.&#10;For example, Rocchio~\cite{rocchio1971relevance}, KL expansion~\cite{zhai2001model}, relevance modelling~\cite{metzler2005markov}, LCE~\cite{metzler2007latent}, and RM3 expansion~\cite{abdul2004umass}.&#10;Additionally, we have seen approaches that expand queries with KG-based information~\cite{meij2010conceptual, xiong2015query, Xu2009query, dalton2014entity} or utilize query-focused LLM vectors for query expansion~\cite{naseri2021ceqe}.&#10;&#10;Recent advancements in dense retrieval~\cite{khattab2020colbert, lin2020distilling, xiongapproximate} have led to the development of vector-based PRF models~\cite{li2022improving}, such as ColBERT PRF~\cite{wang2022colbert}, ColBERT-TCT PRF~\cite{yu2021improving}, and ANCE-PRF~\cite{yu2021improving}. Furthermore, SPLADE~\cite{formal2021splade} is a neural retrieval model that uses BERT and sparse regularization to learn query and document sparse expansions. Recent work has leveraged query expansion with PRF of learned sparse representations~\cite{lassance2023naver}. &#10;Unlike prior work, GRF does not rely on pseudo-relevance feedback, instead generating relevant text context for query expansion using LLMs.&#10;&#10;LLM Query Augmentation&#10;The emergence of LLMs has shown progress across many different aspects of information retrieval~\cite{yates2021pretrained}.&#10;This includes using LLMs to change the query representation, such as query generation and rewriting~\cite{jeronymo2023inpars, nogueira2019doc2query, zamani2020generating, wu-etal-2022-conqrr, samarinas2022revisiting, macavaney2021intent5}, context generation~\cite{hyde, liu2022query}, and query-specific reasoning~\cite{pereira2023visconde, ferraretto2023exaranker}.&#10;For example, \cite{nogueira2019doc2query} fine-tune a T5 model to generate queries for document expansion for passage retrieval.&#10;More recent work by \cite{bonifacio2022inpars} shows that GPT3 can be effectively leveraged for few-short query generation for dataset generation.&#10;Furthermore, LLMs have been used for conversational query re-writing~\cite{wu-etal-2022-conqrr} and generating clarifying questions~\cite{zamani2020generating}. &#10;&#10;We have also seen facet generation using T5~\cite{macavaney2021intent5} and GPT3~\cite{samarinas2022revisiting} to improve the relevance or diversity of search results.&#10;While in QA, \cite{liu2022query} sample various contextual clues from LLMs and augment and fuse multiple queries.&#10;For passage ranking, HyDe~\cite{hyde} uses InstructGPT~\cite{ouyang2022training} to generate hypothetical document embeddings and use Contriever~\cite{izacard2021contriever} for dense retrieval.&#10;Lastly, works have shown LLM generation used for query-specific reasoning~\cite{pereira2023visconde, ferraretto2023exaranker} to improve ranking effectiveness.&#10;Our approach differs from prior LLM augmentation approaches as we use LLMs to generate long-form text to produce a probabilistic expansion model to tackle query-document lexical mismatch.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2207.13332" label="2207.13332">
        <attvalues>
          <attvalue for="0" value="RealTime QA: What's the Answer Right Now?" />
          <attvalue for="1" value="  We introduce REALTIME QA, a dynamic question answering (QA) platform that&#10;announces questions and evaluates systems on a regular basis (weekly in this&#10;version). REALTIME QA inquires about the current world, and QA systems need to&#10;answer questions about novel events or information. It therefore challenges&#10;static, conventional assumptions in open-domain QA datasets and pursues&#10;instantaneous applications. We build strong baseline models upon large&#10;pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing&#10;effort, and this paper presents real-time evaluation results over the past&#10;year. Our experimental results show that GPT-3 can often properly update its&#10;generation results, based on newly-retrieved documents, highlighting the&#10;importance of up-to-date information retrieval. Nonetheless, we find that GPT-3&#10;tends to return outdated answers when retrieved documents do not provide&#10;sufficient information to find an answer. This suggests an important avenue for&#10;future research: can an open-domain QA system identify such unanswerable cases&#10;and communicate with the user or even the retrieval module to modify the&#10;retrieval results? We hope that REALTIME QA will spur progress in instantaneous&#10;applications of question answering and beyond.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.08541" label="2309.08541">
        <attvalues>
          <attvalue for="0" value="When do Generative Query and Document Expansions Fail? A Comprehensive&#10;  Study Across Methods, Retrievers, and Datasets" />
          <attvalue for="1" value="  Using large language models (LMs) for query or document expansion can improve&#10;generalization in information retrieval. However, it is unknown whether these&#10;techniques are universally beneficial or only effective in specific settings,&#10;such as for particular retrieval models, dataset domains, or query types. To&#10;answer this, we conduct the first comprehensive analysis of LM-based expansion.&#10;We find that there exists a strong negative correlation between retriever&#10;performance and gains from expansion: expansion improves scores for weaker&#10;models, but generally harms stronger models. We show this trend holds across a&#10;set of eleven expansion techniques, twelve datasets with diverse distribution&#10;shifts, and twenty-four retrieval models. Through qualitative error analysis,&#10;we hypothesize that although expansions provide extra information (potentially&#10;improving recall), they add additional noise that makes it difficult to discern&#10;between the top relevant documents (thus introducing false positives). Our&#10;results suggest the following recipe: use expansions for weaker models or when&#10;the target dataset significantly differs from training corpus in format;&#10;otherwise, avoid expansions to keep the relevance signal clear.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.04023" label="2211.04023">
        <attvalues>
          <attvalue for="0" value="A Dynamic Graph Interactive Framework with Label-Semantic Injection for&#10;  Spoken Language Understanding" />
          <attvalue for="1" value="  Multi-intent detection and slot filling joint models are gaining increasing&#10;traction since they are closer to complicated real-world scenarios. However,&#10;existing approaches (1) focus on identifying implicit correlations between&#10;utterances and one-hot encoded labels in both tasks while ignoring explicit&#10;label characteristics; (2) directly incorporate multi-intent information for&#10;each token, which could lead to incorrect slot prediction due to the&#10;introduction of irrelevant intent. In this paper, we propose a framework termed&#10;DGIF, which first leverages the semantic information of labels to give the&#10;model additional signals and enriched priors. Then, a multi-grain interactive&#10;graph is constructed to model correlations between intents and slots.&#10;Specifically, we propose a novel approach to construct the interactive graph&#10;based on the injection of label semantics, which can automatically update the&#10;graph to better alleviate error propagation. Experimental results show that our&#10;framework significantly outperforms existing approaches, obtaining a relative&#10;improvement of 13.7% over the previous best model on the MixATIS dataset in&#10;overall accuracy.&#10;" />
          <attvalue for="2" value="&#10;Spoken language understanding (SLU) is a crucial component in task-oriented dialogue systems \cite{weld2021survey}, which typically consists of two subtasks: intent detection (ID) and slot filling (SF). \cite{kim2017fourth} discovered that complicated real-world scenarios frequently involve multiple intents in a single utterance. Take an example in Fig.\ref{fig:example}, the task of ID should classify both intent labels in the utterance (i.e., AddToPlaylist and PlayMusic), while SF can be treated as a sequence labeling task to predict slot for each token in BIO format \cite{zhang2016joint,qin2021co}. &#10;&#10;Since intents and slots are inextricably related \cite{zhou2021pin,huang-etal-2020-federated, huang2021sentiment}, researchers in recent years \cite{gangadharaiah2019joint, qin-etal-2020-agif, qin2021gl,cai2022slim} have increasingly focused on joint multiple intent detection and slot filling. Although achieving promising performance, existing approaches typically classify an utterance to intents represented by one-hot encoding(e.g., 0) while the same problem occurs in slot filling.&#10;&#10;They ignore intuitive and explicit label characteristics, oversimplifying representations of labels. We argue that the label semantics may be useful, which could improve performances for both subtasks by assessing semantic similarity between words in utterances and words in labels.&#10;&#10;Another key challenge in multi-intent SLU is how to effectively incorporate multiple intents information to guide the slot prediction. To handle this, \cite{gangadharaiah2019joint} first investigated a multi-task network with a slot-gated mechanism \cite{goo2018slot}. For fine-grained interaction between multiple intents and slots, \cite{qin-etal-2020-agif} proposed an adaptive graph interactive framework, which builds an interactive graph for each token in the utterance by using all predicted intents. \cite{qin2021gl} explored a global-locally graph interaction network, which models slot dependency and intent-slot interaction for each utterance. However, different tokens appearing in the utterance have various importance for representing the intents. Unfortunately, models mentioned above straightforwardly attach multiple intent information to all tokens, including those without contribution to intent representations, which will introduce noise into sentence-level semantics to some extent.&#10;&#10;In this paper, we propose a novel framework DGIF for joint multiple ID and SF to tackle the above two issues. Concretely, inspired by the success of leveraging label characteristics to help model optimization \cite{wu-etal-2021-label,cui2019hierarchically}, we construct intent and slot spaces using words in each intent label and slot label respectively to inject label information into utterance representations adaptively. Moreover, we propose label-aware regularization to model the rich semantic dependencies among labels in each label space. Then, we capture relevant intents for each token to construct the multi-grain intent-slot interactive graph, as opposed to prior works which directly incorporate multiple intents information statically. Empirical results on two public datasets (MixATIS and MixSNIPS \cite{qin-etal-2020-agif}) demonstrate that our framework outperforms competitive baselines. The source code for this paper can be obtained from \href{&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Mathematics, Natural Language Processing, Slot Filling Models, Intent Detection" />
        </attvalues>
      </node>
      <node id="2101.08091" label="2101.08091">
        <attvalues>
          <attvalue for="0" value="A survey of joint intent detection and slot-filling models in natural&#10;  language understanding" />
          <attvalue for="1" value="  Intent classification and slot filling are two critical tasks for natural&#10;language understanding. Traditionally the two tasks have been deemed to proceed&#10;independently. However, more recently, joint models for intent classification&#10;and slot filling have achieved state-of-the-art performance, and have proved&#10;that there exists a strong relationship between the two tasks. This article is&#10;a compilation of past work in natural language understanding, especially joint&#10;intent classification and slot filling. We observe three milestones in this&#10;research so far: Intent detection to identify the speaker's intention, slot&#10;filling to label each word token in the speech/text, and finally, joint intent&#10;classification and slot filling tasks. In this article, we describe trends,&#10;approaches, issues, data sets, evaluation metrics in intent classification and&#10;slot filling. We also discuss representative performance values, describe&#10;shared tasks, and provide pointers to future work, as given in prior works. To&#10;interpret the state-of-the-art trends, we provide multiple tables that describe&#10;and summarise past research along different dimensions, including the types of&#10;features, base approaches, and dataset domain used.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03880" label="2010.03880">
        <attvalues>
          <attvalue for="0" value="A Co-Interactive Transformer for Joint Slot Filling and Intent Detection" />
          <attvalue for="1" value="  Intent detection and slot filling are two main tasks for building a spoken&#10;language understanding (SLU) system. The two tasks are closely related and the&#10;information of one task can be utilized in the other task. Previous studies&#10;either model the two tasks separately or only consider the single information&#10;flow from intent to slot. None of the prior approaches model the bidirectional&#10;connection between the two tasks simultaneously. In this paper, we propose a&#10;Co-Interactive Transformer to consider the cross-impact between the two tasks.&#10;Instead of adopting the self-attention mechanism in vanilla Transformer, we&#10;propose a co-interactive module to consider the cross-impact by building a&#10;bidirectional connection between the two related tasks. In addition, the&#10;proposed co-interactive module can be stacked to incrementally enhance each&#10;other with mutual features. The experimental results on two public datasets&#10;(SNIPS and ATIS) show that our model achieves the state-of-the-art performance&#10;with considerable improvements (+3.4% and +0.9% on overall acc). Extensive&#10;experiments empirically verify that our model successfully captures the mutual&#10;interaction knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.13431" label="2009.13431">
        <attvalues>
          <attvalue for="0" value="PIN: A Novel Parallel Interactive Network for Spoken Language&#10;  Understanding" />
          <attvalue for="1" value="  Spoken Language Understanding (SLU) is an essential part of the spoken&#10;dialogue system, which typically consists of intent detection (ID) and slot&#10;filling (SF) tasks. Recently, recurrent neural networks (RNNs) based methods&#10;achieved the state-of-the-art for SLU. It is noted that, in the existing&#10;RNN-based approaches, ID and SF tasks are often jointly modeled to utilize the&#10;correlation information between them. However, we noted that, so far, the&#10;efforts to obtain better performance by supporting bidirectional and explicit&#10;information exchange between ID and SF are not well studied.In addition, few&#10;studies attempt to capture the local context information to enhance the&#10;performance of SF. Motivated by these findings, in this paper, Parallel&#10;Interactive Network (PIN) is proposed to model the mutual guidance between ID&#10;and SF. Specifically, given an utterance, a Gaussian self-attentive encoder is&#10;introduced to generate the context-aware feature embedding of the utterance&#10;which is able to capture local context information. Taking the feature&#10;embedding of the utterance, Slot2Intent module and Intent2Slot module are&#10;developed to capture the bidirectional information flow for ID and SF tasks.&#10;Finally, a cooperation mechanism is constructed to fuse the information&#10;obtained from Slot2Intent and Intent2Slot modules to further reduce the&#10;prediction bias.The experiments on two benchmark datasets, i.e., SNIPS and&#10;ATIS, demonstrate the effectiveness of our approach, which achieves a&#10;competitive result with state-of-the-art models. More encouragingly, by using&#10;the feature embedding of the utterance generated by the pre-trained language&#10;model BERT, our method achieves the state-of-the-art among all comparison&#10;approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2108.11711" label="2108.11711">
        <attvalues>
          <attvalue for="0" value="SLIM: Explicit Slot-Intent Mapping with BERT for Joint Multi-Intent&#10;  Detection and Slot Filling" />
          <attvalue for="1" value="  Utterance-level intent detection and token-level slot filling are two key&#10;tasks for natural language understanding (NLU) in task-oriented systems. Most&#10;existing approaches assume that only a single intent exists in an utterance.&#10;However, there are often multiple intents within an utterance in real-life&#10;scenarios. In this paper, we propose a multi-intent NLU framework, called SLIM,&#10;to jointly learn multi-intent detection and slot filling based on BERT. To&#10;fully exploit the existing annotation data and capture the interactions between&#10;slots and intents, SLIM introduces an explicit slot-intent classifier to learn&#10;the many-to-one mapping between slots and intents. Empirical results on three&#10;public multi-intent datasets demonstrate (1) the superior performance of SLIM&#10;compared to the current state-of-the-art for NLU with multiple intents and (2)&#10;the benefits obtained from the slot-intent classifier.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.01925" label="2106.01925">
        <attvalues>
          <attvalue for="0" value="GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple&#10;  Intent Detection and Slot Filling" />
          <attvalue for="1" value="  Multi-intent SLU can handle multiple intents in an utterance, which has&#10;attracted increasing attention. However, the state-of-the-art joint models&#10;heavily rely on autoregressive approaches, resulting in two issues: slow&#10;inference speed and information leakage. In this paper, we explore a&#10;non-autoregressive model for joint multiple intent detection and slot filling,&#10;achieving more fast and accurate. Specifically, we propose a Global-Locally&#10;Graph Interaction Network (GL-GIN) where a local slot-aware graph interaction&#10;layer is proposed to model slot dependency for alleviating uncoordinated slots&#10;problem while a global intent-slot graph interaction layer is introduced to&#10;model the interaction between multiple intents and all slots in the utterance.&#10;Experimental results on two public datasets show that our framework achieves&#10;state-of-the-art performance while being 11.5 times faster.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.13986" label="2306.13986">
        <attvalues>
          <attvalue for="0" value="Large Language Models as Sous Chefs: Revising Recipes with GPT-3" />
          <attvalue for="1" value="  With their remarkably improved text generation and prompting capabilities,&#10;large language models can adapt existing written information into forms that&#10;are easier to use and understand. In our work, we focus on recipes as an&#10;example of complex, diverse, and widely used instructions. We develop a prompt&#10;grounded in the original recipe and ingredients list that breaks recipes down&#10;into simpler steps. We apply this prompt to recipes from various world&#10;cuisines, and experiment with several large language models (LLMs), finding&#10;best results with GPT-3.5. We also contribute an Amazon Mechanical Turk task&#10;that is carefully designed to reduce fatigue while collecting human judgment of&#10;the quality of recipe revisions. We find that annotators usually prefer the&#10;revision over the original, demonstrating a promising application of LLMs in&#10;serving as digital sous chefs for recipes and beyond. We release our prompt,&#10;code, and MTurk template for public use.&#10;" />
          <attvalue for="2" value="&#10;We have access to vast amounts of written information, but many of these resources are written in different styles, levels of complexity, and domains. Previous work in adapting information to be easier to use and understand have relied on advances in general knowledge, event reasoning, text simplification, and other subtasks in Natural Language Processing. Today, the tremendous improvement in large language models, especially their zero-shot prompting and in-context learning capabilities, has made it easier than ever to leverage NLP research for user-centered goals.&#10;&#10;Our work focuses on revising food recipes as a key use case for adapting written information. Recipes are abundant, diverse, and widely used, making them a prime candidate to explore how well GPT-3.5~\cite{brown2020language}---one of the currently most powerful LLMs---can handle them. For each recipe sampled across various world cuisines, we ask GPT-3.5 to use the original recipe and ingredients list to produce an adaptation that is easier to follow.&#10;&#10;We rely on human judgment collected through a carefully designed task on Amazon Mechanical Turk to evaluate revised recipes. Instead of displaying original and revised recipes side by side and asking annotators to evaluate many parts of both at once, we simplify the process by displaying one step at a time. Beyond general usability, we ask annotators to identify new, missing, or incorrect information in a revised recipe. GPT-3.5 often introduces new information to clarify implicit information. This new information can be classified as either a `hallucination', which is detrimental to generation quality, or an `elaboration', which is helpful for end-users.&#10;&#10;Our work makes these three key contributions:&#10;\begin{enumerate}&#10; \item We propose the task of using LLMs to improve the presentation of complex, written information. Specifically, we prompt an LLM to revise a given recipe.&#10; \item We describe our experimentation to design a successful prompt. We find that by including grounding information (here the original recipe and list of ingredients) in the prompt, LLM generations will emphasize helpful elaborations over detrimental hallucinations.&#10; \item We design a human evaluation task that enables annotators to make granular, step-by-step comparisons between the original and revised recipes. This granularity both reduces the mental strain of annotation and allows for step-level analysis of the model's revisions. We find that annotators prefer revised recipes.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Recipe Simplification Techniques, Linguistics, Cognitive Science, Language Model Applications, Artificial Intelligence, Human Evaluation Methods, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.13856" label="2010.13856">
        <attvalues>
          <attvalue for="0" value="Data Troubles in Sentence Level Confidence Estimation for Machine&#10;  Translation" />
          <attvalue for="1" value="  The paper investigates the feasibility of confidence estimation for neural&#10;machine translation models operating at the high end of the performance&#10;spectrum. As a side product of the data annotation process necessary for&#10;building such models we propose sentence level accuracy $SACC$ as a simple,&#10;self-explanatory evaluation metric for quality of translation.&#10;  Experiments on two different annotator pools, one comprised of non-expert&#10;(crowd-sourced) and one of expert (professional) translators show that $SACC$&#10;can vary greatly depending on the translation proficiency of the annotators,&#10;despite the fact that both pools are about equally reliable according to&#10;Krippendorff's alpha metric; the relatively low values of inter-annotator&#10;agreement confirm the expectation that sentence-level binary labeling $good$ /&#10;$needs\ work$ for translation out of context is very hard.&#10;  For an English-Spanish translation model operating at $SACC = 0.89$ according&#10;to a non-expert annotator pool we can derive a confidence estimate that labels&#10;0.5-0.6 of the $good$ translations in an &quot;in-domain&quot; test set with 0.95&#10;Precision. Switching to an expert annotator pool decreases $SACC$ dramatically:&#10;$0.61$ for English-Spanish, measured on the exact same data as above. This&#10;forces us to lower the CE model operating point to 0.9 Precision while labeling&#10;correctly about 0.20-0.25 of the $good$ translations in the data.&#10;  We find surprising the extent to which CE depends on the level of proficiency&#10;of the annotator pool used for labeling the data. This leads to an important&#10;recommendation we wish to make when tackling CE modeling in practice: it is&#10;critical to match the end-user expectation for translation quality in the&#10;desired domain with the demands of annotators assigning binary quality labels&#10;to CE training data.&#10;" />
          <attvalue for="2" value="&#10;&#10;The quality of machine translation models has improved significantly in recent years to the point that ``in-domain&quot; performance for languages rich in training data has approached ``human parity&quot; \cite{barrault-etal-2019-findings}. Outside this strict context the claim is disputable however: the boundaries of ``in-domain&quot; data are unclear even when the training data that was used to build the model is made available, not to mention the situation of a casual user interacting with the translation model. Idiosyncrasies in the input text can trigger surprising model behaviors even for language pairs at the high end of the translation quality spectrum, \cite{DBLP:journals/corr/abs-1906-11943}. \cite{L_ubli_2020} propose revising the evaluation of strong machine translation systems.&#10;&#10;This brings to fore the issue of trustworthiness of a translation service or model, as with the output of pretty much any machine learning algorithm \cite{Spiegelhalter2020Should}: it needs to be accompanied by a confidence score to make it truly meaningful and useful.&#10;&#10;Besides monolingual users of machine translation (MT) technology (particularly when not knowing the target language), algorithmic uses of MT could also greatly benefit from a sentence-level confidence score. As an example, consider the possibility of translating English web pages and serving them as search results in a different language: keyword hits, titles, snippets should be weighed accordingly when ranking or rendering such content. Another use case is reducing post-editing costs: out of one thousand translations, which is the half or third that is probably correct?&#10;&#10;We therefore focus on confidence estimation (CE) for MT: we wish to build a model that can predict whether a given translation is $good$ as is or not. In contrast with the quality evaluation (QE) task \cite{specia2018machine}, we model translation quality as binary valued; to further depart from the QE setup we assume access to both the MT model that produces the translation and to the training data used to build it. We contrast our approach with traditional QE in more detail in Section~\ref{sec:related_work_qe}.&#10;&#10;A necessary ingredient is the capability of annotating sentence pairs consisting of source sentence and translation with binary labels $good$ / $needs\ work$. As a side product, this enables a very straightforward and interpret-able way of evaluating the quality of MT models by measuring sentence-level accuracy (SACC). &#10;&#10;We use two different annotator pools for annotation:&#10;\begin{enumerate}&#10; \item non-experts: proficient in the source and native in the target language&#10; \item experts: professional translators&#10;\end{enumerate}&#10;&#10;With the non-expert pool, in agreement with~\cite{barrault-etal-2019-findings} we find that for high-quality language pairs such as English-German, English-French or English-Spanish, SACC values on ``in-domain&quot; data vary between 0.89 and 0.98; for a lower-quality model such as English-Romanian, SACC is 0.61. A simple model for the reliability of binary annotations allows us to estimate the probability of a annotator assigning the incorrect label on a sample (about 0.05--0.10 in our experiments) and leads to the use of label smoothing in training our CE models.&#10;&#10;Switching to the expert pool lowers SACC dramatically: for English-German it drops from 0.98 to 0.57; for English-Spanish from 0.89 to 0.61; for English-French from 0.89 to 0.44 and for English-Romanian from 0.61 to 0.26. The simple Bernoulli annotation model breaks down. &#10;&#10;Analyzing the two annotator pools using Krippendorff's alpha (KA) statistic~\cite{KrippendorffAlpha} for inter-annotator agreement shows that they are both about equally reliable, with the non-expert pool being simply more lenient than the expert one. The low values of KA confirm the expectation that binary translation quality annotation for isolated sentences out of context is very hard.&#10;&#10;Practical use cases of MT output are well aligned with CE models in that translation quality is viewed as a binary variable: correct or incorrect, as described in \cite{zhou2020practical}. As a result we suggest measuring CE performance using Recall at a fixed Precision value which is chosen depending on the SACC of the model, e.g. 0.9 or 0.95.&#10;&#10;The utility of a CE model is however measured by the product of Recall and SACC of the underlying translation model. Indeed, we find that for an English-Spanish MT model whose ``in-domain&quot; performance is 89\% SACC, the CE model can achieve 0.6 Recall at 0.95 Precision, meaning that it can annotate roughly half of the input sentences as $good$ translations with relatively high precision. &#10;&#10;As a more general remark, it is interesting to examine the benefit of CE at various translation quality levels as measured by SACC: &#10;\begin{itemize} &#10;\item at the low end of the SACC range, CE at the Precision/Recall operating point mentioned above does not add significant value simply because the ratio of $good$ samples in the data is too low;&#10;\item as SACC increases, e.g. 60\%-90\%, the missed opportunity from not leveraging CE becomes significant: a large percentage of sentences are translated correctly ($good$) and yet we do not know which ones. CE can add significant value in this regime, its utility increasing with the accuracy of the underlying MT model; &#10;\item finally, when SACC exceeds the desired Precision level for CE (e.g 95\%) we can simply label all translations as $good$, making CE modeling irrelevant.&#10;\end{itemize}&#10;&#10;The remainder of the paper is organized as follows: in Section~\ref{sec:problem_def_related_work} we formulate the CE problem for MT and contrast our approach to QE, and relate it to previous work that tackled QE with a binary quality metric. While our approach pertains to the more general body of work on ``uncertainty estimation&quot; for machine learning models, and in particular for seq2seq models~\cite{sutskever2014sequence}, we find little related work in the context of neural MT, as highlighted in~\ref{sec:related_work_ce}. Section~\ref{sec:data} described the data annotation framework and an analysis of the annotator reliability for both non-expert and expert pools. Section~\ref{sec:modeling} describes our modeling approach as an offshoot of standard QE work re-purposed for binary classification augmented with features that are expected to help CE: Monte-Carlo dropout (Section~\ref{sec:modeling:mcdropout}) aiming at model uncertainty and language modeling on source sentences (Section~\ref{sec:modeling:lm}) aiming at mitigating MT behavior on ``out-of-distribution&quot; data. Section~\ref{sec:exps} presents experiments on two data sets. They are derived from the same source and translation pairs by asking two different annotator pools to provide binary labels $good$/$needs work$. The source sentences are ``in-domain&quot; relative to the translation model, sampled from web pages whose translation is requested in a given locale; the ``in-domain&quot;-ness of the data is confirmed by the fact that a language model trained on the source side of the MT training data does not benefit from incremental training on the CE training data.&#10;We follow with conclusions and future work directions in Section~\ref{sec:conclusions}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Confidence Estimation, Computer Science, Linguistics, Data Annotation Quality, Statistics, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2005.03519" label="2005.03519">
        <attvalues>
          <attvalue for="0" value="Practical Perspectives on Quality Estimation for Machine Translation" />
          <attvalue for="1" value="  Sentence level quality estimation (QE) for machine translation (MT) attempts&#10;to predict the translation edit rate (TER) cost of post-editing work required&#10;to correct MT output. We describe our view on sentence-level QE as dictated by&#10;several practical setups encountered in the industry. We find consumers of MT&#10;output---whether human or algorithmic ones---to be primarily interested in a&#10;binary quality metric: is the translated sentence adequate as-is or does it&#10;need post-editing? Motivated by this we propose a quality classification (QC)&#10;view on sentence-level QE whereby we focus on maximizing recall at precision&#10;above a given threshold. We demonstrate that, while classical QE regression&#10;models fare poorly on this task, they can be re-purposed by replacing the&#10;output regression layer with a binary classification one, achieving 50-60\%&#10;recall at 90\% precision. For a high-quality MT system producing 75-80\%&#10;correct translations, this promises a significant reduction in post-editing&#10;work indeed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.09830" label="1611.09830">
        <attvalues>
          <attvalue for="0" value="NewsQA: A Machine Comprehension Dataset" />
          <attvalue for="1" value="  We present NewsQA, a challenging machine comprehension dataset of over&#10;100,000 human-generated question-answer pairs. Crowdworkers supply questions&#10;and answers based on a set of over 10,000 news articles from CNN, with answers&#10;consisting of spans of text from the corresponding articles. We collect this&#10;dataset through a four-stage process designed to solicit exploratory questions&#10;that require reasoning. A thorough analysis confirms that NewsQA demands&#10;abilities beyond simple word matching and recognizing textual entailment. We&#10;measure human performance on the dataset and compare it to several strong&#10;neural models. The performance gap between humans and machines (0.198 in F1)&#10;indicates that significant progress can be made on NewsQA through future&#10;research. The dataset is freely available at&#10;https://datasets.maluuba.com/NewsQA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.02789" label="1809.02789">
        <attvalues>
          <attvalue for="0" value="Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book&#10;  Question Answering" />
          <attvalue for="1" value="  We present a new kind of question answering dataset, OpenBookQA, modeled&#10;after open book exams for assessing human understanding of a subject. The open&#10;book that comes with our questions is a set of 1329 elementary level science&#10;facts. Roughly 6000 questions probe an understanding of these facts and their&#10;application to novel situations. This requires combining an open book fact&#10;(e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of&#10;armor is made of metal) obtained from other sources. While existing QA datasets&#10;over documents or knowledge bases, being generally self-contained, focus on&#10;linguistic understanding, OpenBookQA probes a deeper understanding of both the&#10;topic---in the context of common knowledge---and the language it is expressed&#10;in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art&#10;pre-trained QA methods perform surprisingly poorly, worse than several simple&#10;neural baselines we develop. Our oracle experiments designed to circumvent the&#10;knowledge retrieval bottleneck demonstrate the value of both the open book and&#10;additional facts. We leave it as a challenge to solve the retrieval problem in&#10;this multi-hop setting and to close the large gap to human performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00278" label="2005.00278">
        <attvalues>
          <attvalue for="0" value="Unsupervised Transfer of Semantic Role Models from Verbal to Nominal&#10;  Domain" />
          <attvalue for="1" value="  Semantic role labeling (SRL) is an NLP task involving the assignment of&#10;predicate arguments to types, called semantic roles. Though research on SRL has&#10;primarily focused on verbal predicates and many resources available for SRL&#10;provide annotations only for verbs, semantic relations are often triggered by&#10;other linguistic constructions, e.g., nominalizations. In this work, we&#10;investigate a transfer scenario where we assume role-annotated data for the&#10;source verbal domain but only unlabeled data for the target nominal domain. Our&#10;key assumption, enabling the transfer between the two domains, is that&#10;selectional preferences of a role (i.e., preferences or constraints on the&#10;admissible arguments) do not strongly depend on whether the relation is&#10;triggered by a verb or a noun. For example, the same set of arguments can fill&#10;the Acquirer role for the verbal predicate `acquire' and its nominal form&#10;`acquisition'. We approach the transfer task from the variational autoencoding&#10;perspective. The labeler serves as an encoder (predicting role labels given a&#10;sentence), whereas selectional preferences are captured in the decoder&#10;component (generating arguments for the predicting roles). Nominal roles are&#10;not labeled in the training data, and the learning objective instead pushes the&#10;labeler to assign roles predictive of the arguments. Sharing the decoder&#10;parameters across the domains encourages consistency between labels predicted&#10;for both domains and facilitates the transfer. The method substantially&#10;outperforms baselines, such as unsupervised and `direct transfer' methods, on&#10;the English CoNLL-2009 dataset.&#10;" />
          <attvalue for="2" value="&#10;&#10;Semantic role labeling~\cite{gildea2002automatic} methods detect the underlying predicate-argument structures of sentences, or, more formally, assign {semantic roles} to&#10;arguments of predicates:&#10;&#10;In this sentence,&#10;Pinocchio is labeled as {\text A0}, indicating that it is an agent (`a trader') of the trading event, whereas&#10;book is marked as a patient ({\text A1}, `an entity being traded').&#10;Semantic-role structures have been shown effective in many NLP tasks, including machine translation~\cite{marcheggiani2018exploiting},&#10;question answering~\cite{shen2007using},&#10;and summarization~\cite{khan2015framework}.&#10;&#10;Most work on SRL relies on supervised learning~\cite{he2017deep,marcheggiani2017encoding},&#10;and thus requires annotated resources such as PropBank~\cite{palmer2005proposition} and&#10;FrameNet~\cite{baker1998berkeley} for English,&#10;or SALSA~\cite{burchardt2006salsa} for German. However, the annotated data is available only for a dozen of languages.&#10;Moreover, many of these resources&#10;cover only verbal predicates, whereas semantic relations are often triggered by other linguistic constructions, such as nominalizations and prepositions. For example, in the popular multilingual CoNLL-2009 dataset~\cite{hajivc2009conll} nominal predicates are provided only for 3 languages (English, Czech, and Japanese). &#10;The scarcity of annotated data&#10;motivated research into unsupervised SRL methods &#10;(e.g., &#10;\cite{swier2004unsupervised,&#10;&#09;lang2011unsupervisedsplitmerge,&#10;&#09;titov2012bayesian,woodsend2015distributed}) &#10;but these approaches have also focused&#10;only on verbal predicates, and, as we will show in our experiments, do not appear effective in the nominal SRL setting.&#10;In contrast, nominal predicates, though often neglected in annotation efforts and model development,&#10;have been shown crucial in many applications, such as extracting relations from scientific literature~\cite{bethard2008semantic} or events from social media~\cite{liu2012collective}.&#10;&#10;In this work, we investigate a transfer scenario, where we assume the presence of role-annotated data for the `source' verbal domain but only unlabeled data for the `target' nominal domain. Our key assumption, driving the transfer between the two domains, is that selectional preferences of a role (i.e.,&#10;preferences or constraints on the admissible arguments) do not strongly depend on whether the relation is triggered by a verb or a noun. &#10;Take the example in Figure~\ref{fig:same_frame}, semantically similar arguments can fill the A0 (`entity acquiring something') role for the verbal predicate {acquire} and its nominal form {acquisition}.&#10;&#10;We approach the transfer problem from the generative perspective and use the variational autoencoding (VAE) framework~\cite{kingma2013auto} or, more specifically, its semi-supervised version~\cite{kingma2014semi}.&#10;The semantic role labeler serves as an encoder (predicting roles given a sentence),&#10;whereas selectional preferences are captured in the decoder component (generating arguments for the predicting roles). Nominal roles are not labeled in the training data, instead, the autoencoder's learning objective pushes the labeler to assign roles predictive of the arguments. Sharing the decoder parameters across the domains&#10;encourages consistency between labels predicted for both domains.&#10;Intuitively, the knowledge is transferred from the verbal domain &#10;to the nominal domain. &#10;&#10;Similarly, to work on unsupervised semantic role induction in the verbal domain&#10;~\cite{swier2004unsupervised,lang2010unsupervised,titov2015unsupervised}, we focus solely on the argument labeling&#10;subtask, and assume that candidate arguments are provided.&#10;While we classify gold arguments in our experiments, candidate arguments can also be &#10;identified with simple rules~(see Section \ref{sec:auto_eval}). &#10;&#10;We experiment on the English CoNLL-2009 dataset and &#10;compare our approach to baselines. Among the baselines, we consider, (1) a direct transfer approach~\cite{zeman2008cross,sogaard2011data}, which applies a verbal SRL model to the nominal data,&#10;and (2) an unsupervised role induction method, which has been shown to achieve state-of-the-art results on the verbal domain~\cite{titov2015unsupervised}.&#10;We observe that our model outperforms the strongest baseline models by 6.58\% in accuracy and 4.27\% in F1, according to supervised and unsupervised evaluation metrics, respectively. &#10;Our key contributions can be summarized as follows:&#10;(1) we introduce a novel task of transferring SRL models from the verbal to the nominal domain, &#10;(2) we propose a simple latent variable model for the task, &#10;and (3) we show that the approach compares favourably to existing alternatives.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Semantic Role Labeling, Artificial Intelligence, Natural Language Processing, Transfer Learning" />
        </attvalues>
      </node>
      <node id="1804.08313" label="1804.08313">
        <attvalues>
          <attvalue for="0" value="Exploiting Semantics in Neural Machine Translation with Graph&#10;  Convolutional Networks" />
          <attvalue for="1" value="  Semantic representations have long been argued as potentially useful for&#10;enforcing meaning preservation and improving generalization performance of&#10;machine translation methods. In this work, we are the first to incorporate&#10;information about predicate-argument structure of source sentences (namely,&#10;semantic-role representations) into neural machine translation. We use Graph&#10;Convolutional Networks (GCNs) to inject a semantic bias into sentence encoders&#10;and achieve improvements in BLEU scores over the linguistic-agnostic and&#10;syntax-aware versions on the English--German language pair.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.04826" label="1703.04826">
        <attvalues>
          <attvalue for="0" value="Encoding Sentences with Graph Convolutional Networks for Semantic Role&#10;  Labeling" />
          <attvalue for="1" value="  Semantic role labeling (SRL) is the task of identifying the&#10;predicate-argument structure of a sentence. It is typically regarded as an&#10;important step in the standard NLP pipeline. As the semantic representations&#10;are closely related to syntactic ones, we exploit syntactic information in our&#10;model. We propose a version of graph convolutional networks (GCNs), a recent&#10;class of neural networks operating on graphs, suited to model syntactic&#10;dependency graphs. GCNs over syntactic dependency trees are used as sentence&#10;encoders, producing latent feature representations of words in a sentence. We&#10;observe that GCN layers are complementary to LSTM ones: when we stack both GCN&#10;and LSTM layers, we obtain a substantial improvement over an already&#10;state-of-the-art LSTM SRL model, resulting in the best reported scores on the&#10;standard benchmark (CoNLL-2009) both for Chinese and English.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.03714" label="2406.03714">
        <attvalues>
          <attvalue for="0" value="Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis&#10;  with Context-Aware Contrastive Language-Audio Pretraining" />
          <attvalue for="1" value="  Recent prompt-based text-to-speech (TTS) models can clone an unseen speaker&#10;using only a short speech prompt. They leverage a strong in-context ability to&#10;mimic the speech prompts, including speaker style, prosody, and emotion.&#10;Therefore, the selection of a speech prompt greatly influences the generated&#10;speech, akin to the importance of a prompt in large language models (LLMs).&#10;However, current prompt-based TTS models choose the speech prompt manually or&#10;simply at random. Hence, in this paper, we adapt retrieval augmented generation&#10;(RAG) from LLMs to prompt-based TTS. Unlike traditional RAG methods, we&#10;additionally consider contextual information during the retrieval process and&#10;present a Context-Aware Contrastive Language-Audio Pre-training (CA-CLAP) model&#10;to extract context-aware, style-related features. The objective and subjective&#10;evaluations demonstrate that our proposed RAG method outperforms baselines, and&#10;our CA-CLAP achieves better results than text-only retrieval methods.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text-to-speech (TTS) synthesis aims to generate natural speech from text and has seen tremendous improvements due to the adoption of deep learning methods. Recently, the integration of Large Language Models (LLMs) with TTS synthesis technology has emerged as a new trend, garnering widespread attention. LLMs, through in-context learning (ICL), have shown significant advancements in learning from minimal prompts. This breakthrough, coupled with the use of neural audio codecs~\cite{soundstream,encodec,hubert} that convert continuous audio features into discrete tokens, has greatly propelled recent speech synthesis frameworks~\cite{megatts,megatts2}, such as VALL-E~\cite{valle}, AudioLM~\cite{audiolm}, NaturalSpeech2~\cite{naturespeech2}, and SPEAR-TTS~\cite{speartts}. These systems can generate high-quality, personalized speech from just a few seconds of unseen audio used as a speech prompt.&#10;&#10;VALL-E~\cite{valle}, a pioneering TTS framework, adopts RVQ-based audio codec Encodec~\cite{encodec} and utilizes a language model as a prompt-based language modeling task. It can generate acoustic tokens based on the input of a only 3-second voice recording. AudioLM~\cite{audiolm} uses a hierarchical sequence-to-sequence approach and adopts w2v-BERT~\cite{w2vbert} and SoundStream~\cite{soundstream} to extract semantic and acoustic tokens respectively. Therefore the speech prompt is used in both stages and extracted with different representations. SPEAR-TTS~\cite{speartts} has the same structure except for replacing the first stage with an encoder-decoder scheme. Compared with traditional TTS systems like FastSpeech2~\cite{fastspeech2} and Tacotron2~\cite{tacotron}, these recent models show great voice cloning ability by providing only a 3-second speech prompt and have natural prosody comparable with human speakers. This huge success can be attributed to in-context learning provided by GPT-like architecture, and adoption of audio codecs which enable TTS models to utilize vast, diverse, and noisy data instead of only recorded data. However, the generation in a GPT-like manner is highly dependent on previously predicted tokens. This means that the speech prompt has a substantial impact on the subsequent generation process, significantly influencing the generated speech and affecting aspects such as speaker timbre, prosody, and speaking style.&#10;&#10;Hence, the selection of speech prompts is critically important, akin to the significance of prompts in the LLM domain, where the quality and clarity of prompts significantly influence the outcomes~\cite{cot,zerocot}. However, existing methods often randomly select speech prompts from the target speaker, resulting in a choice that is frequently inadequate for guiding the zero-shot TTS system to mimic the desired speaking style and target timbre effectively. The choice of speech prompt should vary given different texts. Furthermore, in TTS scenarios that incorporate context information, such as audiobook TTS~\cite{xue2022paratts,chen2022unsupervised,chen2023stylespeech} and conversational TTS~\cite{m2ctts,deng2023cmcu,deng2023concss}, the choice of a speech prompt should also take contextual information into account. &#10;&#10;To address this challenge, the given audio prompt should coherent the style information with current text and context information. In LLM area, RAG methods~\cite{rag4ki,ragmp} are recognized as a significant enhancement across a variety of tasks. Since LLMs cannot accurately memorize every piece of knowledge but they have strong in-context learning abilities, RAG methods find the most relevant information from external databases and use them as prompts. Retrieval augments the LLM’s ability to generate accurate, grounded responses, especially for queries demanding specialized domain knowledge. &#10;&#10;Motivated by this insight, we adapt the RAG concept to the speech domain to tackle the challenge of selecting appropriate speech prompts. To this end, we introduce a novel framework that combines context-aware retrieval-augmented generation with a prompt-based TTS system. Furthermore, unlike traditional RAG methods that rely solely on textual data, our approach incorporates acoustic inputs during retrieval. This is because the acoustic modality offers richer information, including emotion and speaking style, enhancing the overall quality and relevance of the retrieved content. Specifically, our proposed framework incorporates an innovative Context-Aware Contrastive Language-Audio Pre-training (CA-CLAP) model which is designed to extract context-aware, style-related textual features (STFs) under audio supervision. It employs an audio encoder for extracting style embeddings from speech and a text encoder for deriving STFs from both the text and its context. Additionally, we enhance context integration by implementing cross-attention mechanisms between textual and contextual features. Overall, our paper makes the following contributions: 1) We propose a RAG-enhanced prompt-based TTS framework to enhance audio prompt specialized selection. 2) We design a CA-CLAP model to extract textual and acoustic representations for retrieval. 3) We conduct extensive subjective and objective experiments and find that our proposed methods outperform baselines and our introduced CA-CLAP has better results than text-only embedding methods. Audio samples are available on the project page.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Audio Signal Processing, Linguistics, Artificial Intelligence, Text-to-Speech Models, Speech Synthesis Techniques, Language Model Retrieval" />
        </attvalues>
      </node>
      <node id="2306.03509" label="2306.03509">
        <attvalues>
          <attvalue for="0" value="Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive&#10;  Bias" />
          <attvalue for="1" value="  Scaling text-to-speech to a large and wild dataset has been proven to be&#10;highly effective in achieving timbre and speech style generalization,&#10;particularly in zero-shot TTS. However, previous works usually encode speech&#10;into latent using audio codec and use autoregressive language models or&#10;diffusion models to generate it, which ignores the intrinsic nature of speech&#10;and may lead to inferior or uncontrollable results. We argue that speech can be&#10;decomposed into several attributes (e.g., content, timbre, prosody, and phase)&#10;and each of them should be modeled using a module with appropriate inductive&#10;biases. From this perspective, we carefully design a novel and large zero-shot&#10;TTS system called Mega-TTS, which is trained with large-scale wild data and&#10;models different attributes in different ways: 1) Instead of using latent&#10;encoded by audio codec as the intermediate feature, we still choose spectrogram&#10;as it separates the phase and other attributes very well. Phase can be&#10;appropriately constructed by the GAN-based vocoder and does not need to be&#10;modeled by the language model. 2) We model the timbre using global vectors&#10;since timbre is a global attribute that changes slowly over time. 3) We further&#10;use a VQGAN-based acoustic model to generate the spectrogram and a latent code&#10;language model to fit the distribution of prosody, since prosody changes&#10;quickly over time in a sentence, and language models can capture both local and&#10;long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K&#10;hours of speech and evaluate its performance on unseen speakers. Experimental&#10;results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on&#10;zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior&#10;naturalness, robustness, and speaker similarity due to the proper inductive&#10;bias of each module. Audio samples are available at&#10;https://mega-tts.github.io/demo-page.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.06484" label="2209.06484">
        <attvalues>
          <attvalue for="0" value="ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in&#10;  Paragraph-based TTS" />
          <attvalue for="1" value="  Recent advancements in neural end-to-end TTS models have shown high-quality,&#10;natural synthesized speech in a conventional sentence-based TTS. However, it is&#10;still challenging to reproduce similar high quality when a whole paragraph is&#10;considered in TTS, where a large amount of contextual information needs to be&#10;considered in building a paragraph-based TTS model. To alleviate the difficulty&#10;in training, we propose to model linguistic and prosodic information by&#10;considering cross-sentence, embedded structure in training. Three sub-modules,&#10;including linguistics-aware, prosody-aware and sentence-position networks, are&#10;trained together with a modified Tacotron2. Specifically, to learn the&#10;information embedded in a paragraph and the relations among the corresponding&#10;component sentences, we utilize linguistics-aware and prosody-aware networks.&#10;The information in a paragraph is captured by encoders and the inter-sentence&#10;information in a paragraph is learned with multi-head attention mechanisms. The&#10;relative sentence position in a paragraph is explicitly exploited by a&#10;sentence-position network. Trained on a storytelling audio-book corpus (4.08&#10;hours), recorded by a female Mandarin Chinese speaker, the proposed TTS model&#10;demonstrates that it can produce rather natural and good-quality speech&#10;paragraph-wise. The cross-sentence contextual information, such as break and&#10;prosodic variations between consecutive sentences, can be better predicted and&#10;rendered than the sentence-based model. Tested on paragraph texts, of which the&#10;lengths are similar to, longer than, or much longer than the typical paragraph&#10;length of the training data, the TTS speech produced by the new model is&#10;consistently preferred over the sentence-based model in subjective tests and&#10;confirmed in objective measures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.02269" label="2305.02269">
        <attvalues>
          <attvalue for="0" value="M2-CTTS: End-to-End Multi-scale Multi-modal Conversational&#10;  Text-to-Speech Synthesis" />
          <attvalue for="1" value="  Conversational text-to-speech (TTS) aims to synthesize speech with proper&#10;prosody of reply based on the historical conversation. However, it is still a&#10;challenge to comprehensively model the conversation, and a majority of&#10;conversational TTS systems only focus on extracting global information and omit&#10;local prosody features, which contain important fine-grained information like&#10;keywords and emphasis. Moreover, it is insufficient to only consider the&#10;textual features, and acoustic features also contain various prosody&#10;information. Hence, we propose M2-CTTS, an end-to-end multi-scale multi-modal&#10;conversational text-to-speech system, aiming to comprehensively utilize&#10;historical conversation and enhance prosodic expression. More specifically, we&#10;design a textual context module and an acoustic context module with both&#10;coarse-grained and fine-grained modeling. Experimental results demonstrate that&#10;our model mixed with fine-grained context information and additionally&#10;considering acoustic features achieves better prosody performance and&#10;naturalness in CMOS tests.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.10358" label="2312.10358">
        <attvalues>
          <attvalue for="0" value="CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate&#10;  Prosody in Conversational Speech Synthesis" />
          <attvalue for="1" value="  Conversational speech synthesis (CSS) incorporates historical dialogue as&#10;supplementary information with the aim of generating speech that has&#10;dialogue-appropriate prosody. While previous methods have already delved into&#10;enhancing context comprehension, context representation still lacks effective&#10;representation capabilities and context-sensitive discriminability. In this&#10;paper, we introduce a contrastive learning-based CSS framework, CONCSS. Within&#10;this framework, we define an innovative pretext task specific to CSS that&#10;enables the model to perform self-supervised learning on unlabeled&#10;conversational datasets to boost the model's context understanding.&#10;Additionally, we introduce a sampling strategy for negative sample augmentation&#10;to enhance context vectors' discriminability. This is the first attempt to&#10;integrate contrastive learning into CSS. We conduct ablation studies on&#10;different contrastive learning strategies and comprehensive experiments in&#10;comparison with prior CSS systems. Results demonstrate that the synthesized&#10;speech from our proposed method exhibits more contextually appropriate and&#10;sensitive prosody.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.09545" label="1904.09545">
        <attvalues>
          <attvalue for="0" value="Good-Enough Compositional Data Augmentation" />
          <attvalue for="1" value="  We propose a simple data augmentation protocol aimed at providing a&#10;compositional inductive bias in conditional and unconditional sequence models.&#10;Under this protocol, synthetic training examples are constructed by taking real&#10;training examples and replacing (possibly discontinuous) fragments with other&#10;fragments that appear in at least one similar environment. The protocol is&#10;model-agnostic and useful for a variety of tasks. Applied to neural&#10;sequence-to-sequence models, it reduces error rate by as much as 87% on&#10;diagnostic tasks from the SCAN dataset and 16% on a semantic parsing task.&#10;Applied to n-gram language models, it reduces perplexity by roughly 1% on small&#10;corpora in several languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.10819" label="2011.10819">
        <attvalues>
          <attvalue for="0" value="Evaluating Semantic Accuracy of Data-to-Text Generation with Natural&#10;  Language Inference" />
          <attvalue for="1" value="  A major challenge in evaluating data-to-text (D2T) generation is measuring&#10;the semantic accuracy of the generated text, i.e. checking if the output text&#10;contains all and only facts supported by the input data. We propose a new&#10;metric for evaluating the semantic accuracy of D2T generation based on a neural&#10;model pretrained for natural language inference (NLI). We use the NLI model to&#10;check textual entailment between the input data and the output text in both&#10;directions, allowing us to reveal omissions or hallucinations. Input data are&#10;converted to text for NLI using trivial templates. Our experiments on two&#10;recent D2T datasets show that our metric can achieve high accuracy in&#10;identifying erroneous system outputs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Automatic Evaluation of NLG NLG outputs were traditionally evaluated by reference-based metrics measuring n-gram overlap with a reference, such as BLEU \cite{papineni-etal-2002-bleu}, ROUGE \cite{lin-2004-rouge} and METEOR \cite{lavie_meteor:_2007}. Alternative, referenceless quality estimation metrics based on language model scores \cite{kann_sentence-level_2018} or linguistic features \cite{tian_treat_2018} focus on fluency and do not consider semantic accuracy. Recent works try to estimate NLG output quality with finetuned pretrained models \cite{zhou_learning_2020,zhang_bertscore:_2020,sellam_bleurt_2020}. The score from these models can capture some aspects of semantic accuracy, but only implicitly.&#10;&#10;Semantic Accuracy&#10;To our knowledge, there is no generally accepted automatic metric for explicitly measuring semantic accuracy of NLG outputs. The closest commonly used metric is the slot error rate, which is typically based on pattern matching tailored for a given dataset \cite{reed-etal-2018-neural,ijcai2019-437,duvsek2020evaluating}. Recently, \cite{goodrich_assessing_2019} introduced a metric based on training a neural model on named-entity recognition and fact extraction.&#10;&#10;Faithful NLG&#10;Some recent neural NLG systems train specifically for semantic accuracy&#10;\cite{nie-etal-2019-simple,tian2019sticking,kedzie-mckeown-2019-good}. Similarly to us, \cite{harkous2020have} use a pretrained neural model as a classifier to detect inaccurate output, finetuning the classifier on manually augmented domain-specific data.&#10;&#10;Unlike previous works, we use a pretrained neural model finetuned for NLI which we do not further train on any domain-specific data.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2404.16115" label="2404.16115">
        <attvalues>
          <attvalue for="0" value="Online Personalizing White-box LLMs Generation with Neural Bandits" />
          <attvalue for="1" value="  The advent of personalized content generation by LLMs presents a novel&#10;challenge: how to efficiently adapt text to meet individual preferences without&#10;the unsustainable demand of creating a unique model for each user. This study&#10;introduces an innovative online method that employs neural bandit algorithms to&#10;dynamically optimize soft instruction embeddings based on user feedback,&#10;enhancing the personalization of open-ended text generation by white-box LLMs.&#10;Through rigorous experimentation on various tasks, we demonstrate significant&#10;performance improvements over baseline strategies. NeuralTS, in particular,&#10;leads to substantial enhancements in personalized news headline generation,&#10;achieving up to a 62.9% improvement in terms of best ROUGE scores and up to&#10;2.76% increase in LLM-agent evaluation against the baseline.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, the advancements in large language models (LLMs) have been remarkable~\cite{Brown2020LanguageMA,Zhao2023ASO}, with these models demonstrating an unparalleled ability to understand and generate text across a wide spectrum of tasks~\cite{Wei2022ChainOT,Kojima2022LargeLM}. This capability has &#10;revolutionized the way we interact with machine-generated content &#10;and &#10;opened up new avenues for personalized text generation~\cite{Kirk2023PersonalisationWB,Li2023TeachLT}. &#10;&#10;Personalization in text generation is of paramount importance &#10;to ensure&#10;user engagement and satisfaction~\cite{Huang2022UserNLP222I}&#10;across a range of in applications such as composing tweets, or generating news articles and financial reports, or in more personalized settings like business communications and creative writing~\cite{Li2019TowardsCA,Li2020KnowledgeEnhancedPR}.&#10;&#10;However, the prospect of developing a unique LLM for each user presents &#10;challenges, including the prohibitive resource requirements~\cite{Hoffmann2022TrainingCL}, data privacy concerns~\cite{Li2023MultistepJP}, and the scarcity of personalized data~\cite{Rafailov2023DirectPO}. These obstacles necessitate an alternative strategy that is both practical and flexible. A promising solution lies in adopting lightweight models capable of online learning, which can dynamically adjust their output based on continuous user feedback~\cite{Bai2022TrainingAH}. Such an approach not only circumvents the need for a bespoke model for each user but also encourages &#10;alignment of the &#10;generated content &#10;to individual preferences over time. Importantly, this adaptive process is poised to unlock long-term rewards stemming from personalization, encompassing not just explicit preferences expressed by users but also responding to favorable actions~\cite{Xie2021InteractionGroundedL}. &#10;&#10;Despite these benefits, the ultimate effectiveness of LLMs hinges on the quality of the given instructions~\cite{Zhou2022LargeLM,Bang2023AMM,White2023APP}. Previous efforts have focused on gradient-based strategies~\cite{Shin2020ElicitingKF,Li2021PrefixTuningOC,Lester2021ThePO} for automated instruction optimization, the applicability is limited to less advanced public models, leaving out many advanced yet proprietary models. With the emergence of more advanced open models such as Mistral-7B~\cite{Jiang2023Mistral7}, Llama-70B~\cite{Touvron2023LLaMAOA,Touvron2023Llama2O}, and Mixtral-8x7B~\cite{Jiang2024MixtralOE}, which offer transparency and have reported performance that even surpasses that of ChatGPT-3.5, there is a renewed focus on leveraging these models for direct optimization. &#10;&#10;In this study, we introduce a novel online method for enhancing the personalization of open-ended text generation with white-box LLMs. Considering that {\itshape capturing the nuances of persona in natural language instructions is challenging}, we aim to directly optimize the soft token embeddings~\cite{Chen2023InstructZeroEI,Lin2023UseYI}, representing the contextual factors through user feedback by utilizing neural bandit algorithms~\cite{Zhou2019NeuralCB,zhang2021neural}. This method not only promises to refine the personalization process of text generation but also contributes to the broader application of adaptive algorithms in creating content that closely reflects individual user preferences.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Text Generation Optimization, Linguistics, Neural Bandit Algorithms, Artificial Intelligence, Operations Research, Natural Language Processing, Personalized Content Generation" />
        </attvalues>
      </node>
      <node id="2303.05453" label="2303.05453">
        <attvalues>
          <attvalue for="0" value="Personalisation within bounds: A risk taxonomy and policy framework for&#10;  the alignment of large language models with personalised feedback" />
          <attvalue for="1" value="  Large language models (LLMs) are used to generate content for a wide range of&#10;tasks, and are set to reach a growing audience in coming years due to&#10;integration in product interfaces like ChatGPT or search engines like Bing.&#10;This intensifies the need to ensure that models are aligned with human&#10;preferences and do not produce unsafe, inaccurate or toxic outputs. While&#10;alignment techniques like reinforcement learning with human feedback (RLHF) and&#10;red-teaming can mitigate some safety concerns and improve model capabilities,&#10;it is unlikely that an aggregate fine-tuning process can adequately represent&#10;the full range of users' preferences and values. Different people may&#10;legitimately disagree on their preferences for language and conversational&#10;norms, as well as on values or ideologies which guide their communication.&#10;Personalising LLMs through micro-level preference learning processes may result&#10;in models that are better aligned with each user. However, there are several&#10;normative challenges in defining the bounds of a societally-acceptable and safe&#10;degree of personalisation. In this paper, we ask how, and in what ways, LLMs&#10;should be personalised. First, we review literature on current paradigms for&#10;aligning LLMs with human feedback, and identify issues including (i) a lack of&#10;clarity regarding what alignment means; (ii) a tendency of technology providers&#10;to prescribe definitions of inherently subjective preferences and values; and&#10;(iii) a 'tyranny of the crowdworker', exacerbated by a lack of documentation in&#10;who we are really aligning to. Second, we present a taxonomy of benefits and&#10;risks associated with personalised LLMs, for individuals and society at large.&#10;Finally, we propose a three-tiered policy framework that allows users to&#10;experience the benefits of personalised alignment, while restraining unsafe and&#10;undesirable LLM-behaviours within (supra-)national and organisational bounds.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.03506" label="1910.03506">
        <attvalues>
          <attvalue for="0" value="Towards Controllable and Personalized Review Generation" />
          <attvalue for="1" value="  In this paper, we propose a novel model RevGAN that automatically generates&#10;controllable and personalized user reviews based on the arbitrarily given&#10;sentimental and stylistic information. RevGAN utilizes the combination of three&#10;novel components, including self-attentive recursive autoencoders, conditional&#10;discriminators, and personalized decoders. We test its performance on the&#10;several real-world datasets, where our model significantly outperforms&#10;state-of-the-art generation models in terms of sentence quality, coherence,&#10;personalization and human evaluations. We also empirically show that the&#10;generated reviews could not be easily distinguished from the organically&#10;produced reviews and that they follow the same statistical linguistics laws.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.01480" label="2010.01480">
        <attvalues>
          <attvalue for="0" value="Knowledge-Enhanced Personalized Review Generation with Capsule Graph&#10;  Neural Network" />
          <attvalue for="1" value="  Personalized review generation (PRG) aims to automatically produce review&#10;text reflecting user preference, which is a challenging natural language&#10;generation task. Most of previous studies do not explicitly model factual&#10;description of products, tending to generate uninformative content. Moreover,&#10;they mainly focus on word-level generation, but cannot accurately reflect more&#10;abstractive user preference in multiple aspects. To address the above issues,&#10;we propose a novel knowledge-enhanced PRG model based on capsule graph neural&#10;network~(Caps-GNN). We first construct a heterogeneous knowledge graph (HKG)&#10;for utilizing rich item attributes. We adopt Caps-GNN to learn graph capsules&#10;for encoding underlying characteristics from the HKG. Our generation process&#10;contains two major steps, namely aspect sequence generation and sentence&#10;generation. First, based on graph capsules, we adaptively learn aspect capsules&#10;for inferring the aspect sequence. Then, conditioned on the inferred aspect&#10;label, we design a graph-based copy mechanism to generate sentences by&#10;incorporating related entities or words from HKG. To our knowledge, we are the&#10;first to utilize knowledge graph for the PRG task. The incorporated KG&#10;information is able to enhance user preference at both aspect and word levels.&#10;Extensive experiments on three real-world datasets have demonstrated the&#10;effectiveness of our model on the PRG task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.05197" label="2304.05197">
        <attvalues>
          <attvalue for="0" value="Multi-step Jailbreaking Privacy Attacks on ChatGPT" />
          <attvalue for="1" value="  With the rapid progress of large language models (LLMs), many downstream NLP&#10;tasks can be well solved given appropriate prompts. Though model developers and&#10;researchers work hard on dialog safety to avoid generating harmful content from&#10;LLMs, it is still challenging to steer AI-generated content (AIGC) for the&#10;human good. As powerful LLMs are devouring existing text data from various&#10;domains (e.g., GPT-3 is trained on 45TB texts), it is natural to doubt whether&#10;the private information is included in the training data and what privacy&#10;threats can these LLMs and their downstream applications bring. In this paper,&#10;we study the privacy threats from OpenAI's ChatGPT and the New Bing enhanced by&#10;ChatGPT and show that application-integrated LLMs may cause new privacy&#10;threats. To this end, we conduct extensive experiments to support our claims&#10;and discuss LLMs' privacy implications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.04887" label="2106.04887">
        <attvalues>
          <attvalue for="0" value="Interaction-Grounded Learning" />
          <attvalue for="1" value="  Consider a prosthetic arm, learning to adapt to its user's control signals.&#10;We propose Interaction-Grounded Learning for this novel setting, in which a&#10;learner's goal is to interact with the environment with no grounding or&#10;explicit reward to optimize its policies. Such a problem evades common RL&#10;solutions which require an explicit reward. The learning agent observes a&#10;multidimensional context vector, takes an action, and then observes a&#10;multidimensional feedback vector. This multidimensional feedback vector has no&#10;explicit reward information. In order to succeed, the algorithm must learn how&#10;to evaluate the feedback vector to discover a latent reward signal, with which&#10;it can ground its policies without supervision. We show that in an&#10;Interaction-Grounded Learning setting, with certain natural assumptions, a&#10;learner can discover the latent reward and ground its policy for successful&#10;interaction. We provide theoretical guarantees and a proof-of-concept empirical&#10;evaluation to demonstrate the effectiveness of our proposed approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.04462" label="1911.04462">
        <attvalues>
          <attvalue for="0" value="Neural Contextual Bandits with UCB-based Exploration" />
          <attvalue for="1" value="  We study the stochastic contextual bandit problem, where the reward is&#10;generated from an unknown function with additive noise. No assumption is made&#10;about the reward function other than boundedness. We propose a new algorithm,&#10;NeuralUCB, which leverages the representation power of deep neural networks and&#10;uses a neural network-based random feature mapping to construct an upper&#10;confidence bound (UCB) of reward for efficient exploration. We prove that,&#10;under standard assumptions, NeuralUCB achieves $\tilde O(\sqrt{T})$ regret,&#10;where $T$ is the number of rounds. To the best of our knowledge, it is the&#10;first neural network-based contextual bandit algorithm with a near-optimal&#10;regret guarantee. We also show the algorithm is empirically competitive against&#10;representative baselines in a number of benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.00827" label="2010.00827">
        <attvalues>
          <attvalue for="0" value="Neural Thompson Sampling" />
          <attvalue for="1" value="  Thompson Sampling (TS) is one of the most effective algorithms for solving&#10;contextual multi-armed bandit problems. In this paper, we propose a new&#10;algorithm, called Neural Thompson Sampling, which adapts deep neural networks&#10;for both exploration and exploitation. At the core of our algorithm is a novel&#10;posterior distribution of the reward, where its mean is the neural network&#10;approximator, and its variance is built upon the neural tangent features of the&#10;corresponding neural network. We prove that, provided the underlying reward&#10;function is bounded, the proposed algorithm is guaranteed to achieve a&#10;cumulative regret of $\mathcal{O}(T^{1/2})$, which matches the regret of other&#10;contextual bandit algorithms in terms of total round number $T$. Experimental&#10;comparisons with other benchmark bandit algorithms on various data sets&#10;corroborate our theory.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.20574" label="2405.20574">
        <attvalues>
          <attvalue for="0" value="Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with&#10;  Ko-H5 Benchmark" />
          <attvalue for="1" value="  This paper introduces the Open Ko-LLM Leaderboard and the Ko-H5 Benchmark as&#10;vital tools for evaluating Large Language Models (LLMs) in Korean.&#10;Incorporating private test sets while mirroring the English Open LLM&#10;Leaderboard, we establish a robust evaluation framework that has been well&#10;integrated in the Korean LLM community. We perform data leakage analysis that&#10;shows the benefit of private test sets along with a correlation study within&#10;the Ko-H5 benchmark and temporal analyses of the Ko-H5 score. Moreover, we&#10;present empirical support for the need to expand beyond set benchmarks. We hope&#10;the Open Ko-LLM Leaderboard sets precedent for expanding LLM evaluation to&#10;foster more linguistic diversity.&#10;" />
          <attvalue for="2" value="&#10;The emergence of Large Language Models (LLMs)~\cite{zhao2023survey} have also introduced an ever growing demand for robust evaluation frameworks for LLMs. While multiple benchmarks~\cite{open-llm-leaderboard, alpaca_eval, zheng2023judging, 2023opencompass} have been proposed for a more holistic evaluation of LLMs, they are mostly limited to the English language.&#10;Recognizing the need to expand the mostly English-centric LLM benchmarks to other languages such as Korean, we introduce the ``Open Ko-LLM Leaderboard'' and the ``Ko-H5 Benchmark''.&#10;&#10;The Open Ko-LLM Leaderboard is built on the following two principles: i) alignment with the English Open LLM Leaderboard~\cite{open-llm-leaderboard} and ii) private test sets.&#10;Enabling straightforward comparison between the two leaderboard results, following the well-established composition of the Open LLM Leaderboard is key to the successful integration of the Open Ko-LLM Leaderboard in the Korean LLM community. Further, our private test sets allow for robust evaluation of a plethora of models in the wild without significant worry of data contamination on the tested benchmarks~\cite{sainz2023nlp,zhou2023don,balloccu2024leak}. We show that our private test sets have little overlap with some of the most popular training datasets used by top models in the Open Ko-LLM Leaderboard, empirically solidifying the argument for private test sets.&#10;&#10;To reveal various key insights, we perform an extensive multi-faceted analysis.&#10;For instance, correlation between the tasks that constitute the Ko-H5 benchmark shows that the newly added dataset, {i.e.}, Ko-CommonGen v2, differentiates the Open Ko-LLM Leaderboard from the English Open LLM Leaderboard by bringing more diversity to the evaluation suite.&#10;Additionally, analysis of the improvements in the Ko-H5 score over time for differently sized models presents insights into a potential critical model size that enables rapid performance improvement.&#10;Another temporal analysis of the Ko-H5 benchmark scores with respect to various model types brings quantitative support for the notion that improvements in pretrained models lead to improvements in instruction-tuned models. &#10;Further analysis reveals a relatively quick saturation of certain task scores, indicating the need to move beyond a set benchmark. In other words, a shift towards a more holistic evaluation scheme that better adheres to real-world use-cases is needed. Building on the analytical results on score changes for each task of the top performing models, we offer a practical criteria of judging when to expand the evaluation suite for LLMs.&#10;&#10;Our contributions can be summarized as follows:&#10;\begin{itemize}&#10; \item We introduce the ``Open Ko-LLM Leaderboard'' and ``Ko-H5 Benchmark'' for expanding robust and widespread evaluation of Korean LLMs.&#10; \item We address the issue of data contamination by using private test sets for fair model evaluation, ensuring minimal overlap with popular training datasets.&#10; \item We present several analyses that highlight diverse insights ranging from inter-benchmark correlation to change of the benchmark scores over time, aggregated by model size and type and individual tasks.&#10; \item We offer practical criteria of when to expand beyond a set benchmark, emphasizing the need for diverse tasks to continually enhance LLM evaluation.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Linguistic Diversity, Computer Science, Benchmark Development, Linguistics, Language Model Evaluation, Data Science, Statistics" />
        </attvalues>
      </node>
      <node id="2310.18018" label="2310.18018">
        <attvalues>
          <attvalue for="0" value="NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination&#10;  for each Benchmark" />
          <attvalue for="1" value="  In this position paper, we argue that the classical evaluation on Natural&#10;Language Processing (NLP) tasks using annotated benchmarks is in trouble. The&#10;worst kind of data contamination happens when a Large Language Model (LLM) is&#10;trained on the test split of a benchmark, and then evaluated in the same&#10;benchmark. The extent of the problem is unknown, as it is not straightforward&#10;to measure. Contamination causes an overestimation of the performance of a&#10;contaminated model in a target benchmark and associated task with respect to&#10;their non-contaminated counterparts. The consequences can be very harmful, with&#10;wrong scientific conclusions being published while other correct ones are&#10;discarded. This position paper defines different levels of data contamination&#10;and argues for a community effort, including the development of automatic and&#10;semi-automatic measures to detect when data from a benchmark was exposed to a&#10;model, and suggestions for flagging papers with conclusions that are&#10;compromised by data contamination.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.03927" label="2402.03927">
        <attvalues>
          <attvalue for="0" value="Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in&#10;  Closed-Source LLMs" />
          <attvalue for="1" value="  Natural Language Processing (NLP) research is increasingly focusing on the&#10;use of Large Language Models (LLMs), with some of the most popular ones being&#10;either fully or partially closed-source. The lack of access to model details,&#10;especially regarding training data, has repeatedly raised concerns about data&#10;contamination among researchers. Several attempts have been made to address&#10;this issue, but they are limited to anecdotal evidence and trial and error.&#10;Additionally, they overlook the problem of \emph{indirect} data leaking, where&#10;models are iteratively improved by using data coming from users. In this work,&#10;we conduct the first systematic analysis of work using OpenAI's GPT-3.5 and&#10;GPT-4, the most prominently used LLMs today, in the context of data&#10;contamination. By analysing 255 papers and considering OpenAI's data usage&#10;policy, we extensively document the amount of data leaked to these models&#10;during the first year after the model's release. We report that these models&#10;have been globally exposed to $\sim$4.7M samples from 263 benchmarks. At the&#10;same time, we document a number of evaluation malpractices emerging in the&#10;reviewed papers, such as unfair or missing baseline comparisons and&#10;reproducibility issues. We release our results as a collaborative project on&#10;https://leak-llm.github.io/, where other researchers can contribute to our&#10;efforts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.13250" label="2304.13250">
        <attvalues>
          <attvalue for="0" value="Exploring the Curious Case of Code Prompts" />
          <attvalue for="1" value="  Recent work has shown that prompting language models with code-like&#10;representations of natural language leads to performance improvements on&#10;structured reasoning tasks. However, such tasks comprise only a small subset of&#10;all natural language tasks. In our work, we seek to answer whether or not&#10;code-prompting is the preferred way of interacting with language models in&#10;general. We compare code and text prompts across three popular GPT models&#10;(davinci, code-davinci-002, and text-davinci-002) on a broader selection of&#10;tasks (e.g., QA, sentiment, summarization) and find that with few exceptions,&#10;code prompts do not consistently outperform text prompts. Furthermore, we show&#10;that the style of code prompt has a large effect on performance for some but&#10;not all tasks and that fine-tuning on text instructions leads to better&#10;relative performance of code prompts.&#10;" />
          <attvalue for="2" value="&#10;Recent work has shown that pre-training language models (LMs) on a mixture of text and program code (e.g., Python or Javascript) makes them more capable of reasoning over natural language \cite{suzgun2022challenging}.&#10;Such program-trained language models (PLMs) significantly outperform text-only LMs on tasks such as math problems and tracking shuffled objects despite such tasks lacking any explicit code formulae \cite{liang2022holistic}.&#10;&#10;Furthermore, prompting such PLMs with code-like structures (e.g., Python, JSON, PDDL) instead of text has been shown to lead to performance improvements on structured common sense reasoning \cite{madaan2022language}, event argument extraction \cite{wang2022code4struct}, knowledge graph construction \cite{bi2023codekgc}, story understanding \cite{dong2022corrpus}, and causal reasoning \cite{zhang-etal-2023-causal}.&#10;&#10;Such results naturally lead us to ask whether code-prompting is the preferred way of interacting with PLMs in general. While previous work is limited to reasoning tasks, in this work we analyze a broad selection of tasks (e.g., QA, sentiment, summarization) and systematically compare the performance of prompting PLMs with code vs. prompting with text.&#10;We find that:&#10;\begin{itemize}[noitemsep,nolistsep]&#10; \item With the exception of some reasoning tasks, code prompts do not outperform text prompts&#10; \item The style of code prompt has a large effect on performance for some but not all tasks.&#10; \item Fine-tuning on text instructions leads to relative improvements when using code prompts.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Language Model Performance, Code Prompting Methods, Natural Language Tasks" />
        </attvalues>
      </node>
      <node id="2210.07128" label="2210.07128">
        <attvalues>
          <attvalue for="0" value="Language Models of Code are Few-Shot Commonsense Learners" />
          <attvalue for="1" value="  We address the general task of structured commonsense reasoning: given a&#10;natural language input, the goal is to generate a graph such as an event -- or&#10;a reasoning-graph. To employ large language models (LMs) for this task,&#10;existing approaches ``serialize'' the output graph as a flat list of nodes and&#10;edges. Although feasible, these serialized graphs strongly deviate from the&#10;natural language corpora that LMs were pre-trained on, hindering LMs from&#10;generating them correctly. In this paper, we show that when we instead frame&#10;structured commonsense reasoning tasks as code generation tasks, pre-trained&#10;LMs of code are better structured commonsense reasoners than LMs of natural&#10;language, even when the downstream task does not involve source code at all. We&#10;demonstrate our approach across three diverse structured commonsense reasoning&#10;tasks. In all these natural language tasks, we show that using our approach, a&#10;code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the&#10;target task (e.g., T5) and other strong LMs such as GPT-3 in the few-shot&#10;setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12096" label="2305.12096">
        <attvalues>
          <attvalue for="0" value="Can NLP Models Correctly Reason Over Contexts that Break the Common&#10;  Assumptions?" />
          <attvalue for="1" value="  Pre-training on large corpora of text enables the language models to acquire&#10;a vast amount of factual and commonsense knowledge which allows them to achieve&#10;remarkable performance on a variety of language understanding tasks. They&#10;typically acquire this knowledge by learning from the pre-training text and&#10;capturing certain patterns from it. However, real-world settings often present&#10;scenarios that do not abide by these patterns i.e. scenarios that break the&#10;common assumptions. Can state-of-the-art NLP models correctly reason over the&#10;contexts of such scenarios?&#10;  Addressing the above question, in this paper, we investigate the ability of&#10;models to correctly reason over contexts that break the common assumptions. To&#10;this end, we first systematically create evaluation data in which each data&#10;instance consists of (a) a common assumption, (b) a context that follows the&#10;assumption, (c) a context that breaks the assumption, and (d) questions based&#10;on the contexts. Then, through evaluations on multiple models including GPT-3&#10;and Flan T5, we show that while doing fairly well on contexts that follow the&#10;common assumptions, the models struggle to correctly reason over contexts that&#10;break those assumptions. Specifically, the performance gap is as high as 20%&#10;absolute points. Furthermore, we thoroughly analyze these results revealing&#10;several interesting findings. We believe our work and findings will encourage&#10;and facilitate further research in developing more robust models that can also&#10;reliably reason over contexts that break the common assumptions. Data is&#10;available at \url{https://github.com/nrjvarshney/break_the_common_assumptions}.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-training on large corpora of text enables the natural language processing (NLP) models to acquire a vast amount of factual and commonsense knowledge \cite{liu-etal-2019-linguistic,petroni-etal-2019-language,yogatama2019learning,davison-etal-2019-commonsense}. &#10;Due to this knowledge, they are able to achieve remarkable performance on a variety of language understanding tasks.&#10;They typically acquire this knowledge by learning from the pre-training text and capturing certain patterns from it.&#10;However, in real-world settings, we often encounter scenarios that do not abide by these patterns i.e. scenarios that break the common assumptions.&#10;Consider a context, `John likes to have tomato soup only when it is cold', this breaks the common assumption that `people prefer to consume soup when it is hot'.&#10;Answering questions based on such contexts requires a model to truly understand the context and override its knowledge &#10;that it may have acquired (due to the predominant presence of certain patterns in the raw text) during pre-training.&#10;How well can state-of-the-art NLP models perform in such scenarios?&#10;&#10;Recently, many datasets have been created that test different language understanding skills such as pronoun resolution \cite{sakaguchi2021winogrande,levesque_winograd_2012}, commonsense reasoning \cite{talmor-etal-2019-commonsenseqa}, numerical reasoning \cite{dua-etal-2019-drop,patel-etal-2021-nlp,mishra-etal-2022-numglue}, qualitative reasoning \cite{tafjord-etal-2019-quartz, tafjord2019quarel}, temporal reasoning \cite{zhou-etal-2019-going}, and feasibility reasoning \cite{gupta2022john}. &#10;Furthermore, numerous adversarial datasets \cite{mccoy-etal-2019-right,bartolo-etal-2020-beat, naik-etal-2018-stress} have also been developed that test the robustness of models.&#10;\cite{longpre-etal-2021-entity} study entity-based conflicts in the parametric and contextual knowledge.&#10;\cite{agarwal2020entity} investigate entity-based swapping to test the robustness of models.&#10;Prior work has also studied creating counterfactuals using various techniques such as token substitutions and adversarial attacks \cite{ribeiro-etal-2020-beyond,michel-etal-2019-evaluation,kaushik2020learning}.&#10;However, evaluating models on the ability to reason over contexts that break the common assumptions (this is different from entity-based conflicts) has remained underexplored, and existing datasets do not contain a sufficient number of such examples.&#10;&#10;In this work, we address the above limitations and comprehensively study the models' ability to reason over contexts that break the common assumptions.&#10;To this end, we first systematically create questions (binary classification) in which the contexts break the common assumptions and the questions test the ability to reason over those contexts. &#10;Furthermore, for each such context, we also create a corresponding context that `follows' the common assumption.&#10;Specifically, instances in our evaluation data consist of the following:&#10;(a) a common assumption,&#10;(b) a context that follows the assumption,&#10;(c) a context that breaks the assumption, and&#10;(d) questions based on the contexts.&#10;Figure \ref{fig:teaser_figure} illustrates examples of our dataset.&#10;For binary classification questions, the task is to answer a given question as either `Yes' or `No'.&#10;&#10;We conduct comprehensive experiments with several NLP models such as Flan T5 \cite{chung2022scaling}, GPT-3 \cite{NEURIPS2020_1457c0d6}, and UnifiedQA \cite{khashabi-etal-2020-unifiedqa}.&#10;First, we evaluate models on the scenario where the contexts follow the common assumptions; we show that the models perform fairly well in this setting.&#10;However, on evaluating them for the scenario where the contexts break the common assumptions, we find that the models falter and achieve considerably lower performance.&#10;Specifically, on the binary classification questions, Flan T5-xxl achieves an accuracy of just $70.67\%$ in the latter scenario ($\sim20$ absolute points lower than its performance on the former scenario).&#10;Furthermore, we show that this performance is considerably and consistently lower than the human performance baseline.&#10;&#10;We further conduct a thorough analysis which reveals several interesting findings such as (a) models show poor consistency i.e. they are often not able to correctly answer both (context-question) and (context (Breaking)-question) pairs correctly and (b) explicitly providing the common assumption along with the context improves the performance when the context aligns withe the assumption but degrades when it breaks the assumption.&#10;Overall, we believe our work and findings will encourage and facilitate further research in developing more robust models that can also reliably reason over contexts that break the common assumptions. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Limitations, Contextual Reasoning Challenges, Artificial Intelligence, Natural Language Processing, NLP Model Evaluation" />
        </attvalues>
      </node>
      <node id="2103.07191" label="2103.07191">
        <attvalues>
          <attvalue for="0" value="Are NLP Models really able to Solve Simple Math Word Problems?" />
          <attvalue for="1" value="  The problem of designing NLP solvers for math word problems (MWP) has seen&#10;sustained research activity and steady gains in the test accuracy. Since&#10;existing solvers achieve high performance on the benchmark datasets for&#10;elementary level MWPs containing one-unknown arithmetic word problems, such&#10;problems are often considered &quot;solved&quot; with the bulk of research attention&#10;moving to more complex MWPs. In this paper, we restrict our attention to&#10;English MWPs taught in grades four and lower. We provide strong evidence that&#10;the existing MWP solvers rely on shallow heuristics to achieve high performance&#10;on the benchmark datasets. To this end, we show that MWP solvers that do not&#10;have access to the question asked in the MWP can still solve a large fraction&#10;of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve&#10;surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP,&#10;created by applying carefully chosen variations over examples sampled from&#10;existing datasets. The best accuracy achieved by state-of-the-art models is&#10;substantially lower on SVAMP, thus showing that much remains to be done even&#10;for the simplest of the MWPs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05660" label="2204.05660">
        <attvalues>
          <attvalue for="0" value="NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning&#10;  Tasks" />
          <attvalue for="1" value="  Given the ubiquitous nature of numbers in text, reasoning with numbers to&#10;perform simple calculations is an important skill of AI systems. While many&#10;datasets and models have been developed to this end, state-of-the-art AI&#10;systems are brittle; failing to perform the underlying mathematical reasoning&#10;when they appear in a slightly different scenario. Drawing inspiration from&#10;GLUE that was proposed in the context of natural language understanding, we&#10;propose NumGLUE, a multi-task benchmark that evaluates the performance of AI&#10;systems on eight different tasks, that at their core require simple arithmetic&#10;understanding. We show that this benchmark is far from being solved with neural&#10;models including state-of-the-art large-scale language models performing&#10;significantly worse than humans (lower by 46.4%). Further, NumGLUE promotes&#10;sharing knowledge across tasks, especially those with limited training data as&#10;evidenced by the superior performance (average gain of 3.4% on each task) when&#10;a model is jointly trained on all the tasks as opposed to task-specific&#10;modeling. Finally, we hope that NumGLUE will encourage systems that perform&#10;robust and general arithmetic reasoning within language, a first step towards&#10;being able to perform more complex mathematical reasoning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.00293" label="2002.00293">
        <attvalues>
          <attvalue for="0" value="Beat the AI: Investigating Adversarial Human Annotation for Reading&#10;  Comprehension" />
          <attvalue for="1" value="  Innovations in annotation methodology have been a catalyst for Reading&#10;Comprehension (RC) datasets and models. One recent trend to challenge current&#10;RC models is to involve a model in the annotation process: humans create&#10;questions adversarially, such that the model fails to answer them correctly. In&#10;this work we investigate this annotation methodology and apply it in three&#10;different settings, collecting a total of 36,000 samples with progressively&#10;stronger models in the annotation loop. This allows us to explore questions&#10;such as the reproducibility of the adversarial effect, transfer from data&#10;collected with varying model-in-the-loop strengths, and generalisation to data&#10;collected without a model. We find that training on adversarially collected&#10;samples leads to strong generalisation to non-adversarially collected datasets,&#10;yet with progressive performance deterioration with increasingly stronger&#10;models-in-the-loop. Furthermore, we find that stronger models can still learn&#10;from datasets collected with substantially weaker models-in-the-loop. When&#10;trained on data collected with a BiDAF model in the loop, RoBERTa achieves&#10;39.9F1 on questions that it cannot answer when trained on SQuAD - only&#10;marginally lower than when trained on data collected using RoBERTa itself&#10;(41.0F1).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.05052" label="2109.05052">
        <attvalues>
          <attvalue for="0" value="Entity-Based Knowledge Conflicts in Question Answering" />
          <attvalue for="1" value="  Knowledge-dependent tasks typically use two sources of knowledge: parametric,&#10;learned at training time, and contextual, given as a passage at inference time.&#10;To understand how models use these sources together, we formalize the problem&#10;of knowledge conflicts, where the contextual information contradicts the&#10;learned information. Analyzing the behaviour of popular models, we measure&#10;their over-reliance on memorized information (the cause of hallucinations), and&#10;uncover important factors that exacerbate this behaviour. Lastly, we propose a&#10;simple method to mitigate over-reliance on parametric knowledge, which&#10;minimizes hallucination, and improves out-of-distribution generalization by&#10;4%-7%. Our findings demonstrate the importance for practitioners to evaluate&#10;model tendency to hallucinate rather than read, and show that our mitigation&#10;strategy encourages generalization to evolving information (i.e.,&#10;time-dependent queries). To encourage these practices, we have released our&#10;framework for generating knowledge conflicts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.04123" label="2004.04123">
        <attvalues>
          <attvalue for="0" value="Entity-Switched Datasets: An Approach to Auditing the In-Domain&#10;  Robustness of Named Entity Recognition Models" />
          <attvalue for="1" value="  Named entity recognition systems perform well on standard datasets comprising&#10;English news. But given the paucity of data, it is difficult to draw&#10;conclusions about the robustness of systems with respect to recognizing a&#10;diverse set of entities. We propose a method for auditing the in-domain&#10;robustness of systems, focusing specifically on differences in performance due&#10;to the national origin of entities. We create entity-switched datasets, in&#10;which named entities in the original texts are replaced by plausible named&#10;entities of the same type but of different national origin. We find that&#10;state-of-the-art systems' performance vary widely even in-domain: In the same&#10;context, entities from certain origins are more reliably recognized than&#10;entities from elsewhere. Systems perform best on American and Indian entities,&#10;and worst on Vietnamese and Indonesian entities. This auditing approach can&#10;facilitate the development of more robust named entity recognition systems, and&#10;will allow research in this area to consider fairness criteria that have&#10;received heightened attention in other predictive technology work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.11037" label="2102.11037">
        <attvalues>
          <attvalue for="0" value="Highly Fast Text Segmentation With Pairwise Markov Chains" />
          <attvalue for="1" value="  Natural Language Processing (NLP) models' current trend consists of using&#10;increasingly more extra-data to build the best models as possible. It implies&#10;more expensive computational costs and training time, difficulties for&#10;deployment, and worries about these models' carbon footprint reveal a critical&#10;problem in the future. Against this trend, our goal is to develop NLP models&#10;requiring no extra-data and minimizing training time. To do so, in this paper,&#10;we explore Markov chain models, Hidden Markov Chain (HMC) and Pairwise Markov&#10;Chain (PMC), for NLP segmentation tasks. We apply these models for three&#10;classic applications: POS Tagging, Named-Entity-Recognition, and Chunking. We&#10;develop an original method to adapt these models for text segmentation's&#10;specific challenges to obtain relevant performances with very short training&#10;and execution times. PMC achieves equivalent results to those obtained by&#10;Conditional Random Fields (CRF), one of the most applied models for these tasks&#10;when no extra-data are used. Moreover, PMC has training times 30 times shorter&#10;than the CRF ones, which validates this model given our objectives.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the past ten years, developments of Deep Learning methods \cite{Goodfellow-et-al-2016, lecun2015deep} have enabled research in Natural Language Processing (NLP) to take an impressive leap. Some tasks, like Question Answering \cite{rajpurkar2016squad} or Sentiment Analysis \cite{maas-EtAl:2011:ACL-HLT2011}, seemed unrealistic twenty years ago, and nowadays recent neural models achieved better scores than humans \cite{devlin2018bert} \cite{lan2019albert} for these applications. This dynamic's main motivation is the direct applications of NLP models in the industry, with tasks such as Named-Entity-Recognition and mail classification. However, the cost of improving models' performances increases, and learning algorithms always need more data and power to be trained and make a prediction. We can produce models that achieve impressive scores, but deployment and climate change problems \cite{strubell2019energy} raise some issues. Our aim in this paper is to initiate a reflection around light models by introducing a new Markov model design for text segmentation and comparing it with machine learning algorithms that have a reasonable carbon impact and execution time. Motivation about the choice of segmentation tasks is explained later in this paper.&#10;&#10;The Hidden Markov Chains (HMC), introduced by Stratonovitch sixty years ago \cite{stratonovich1965conditional} \cite{baum1966statistical} \cite{cappe2006inference} \cite{rabiner1986introduction} \cite{rabiner1989tutorial} \cite{ephraim2002hidden}, which model poor correlations, are widely used in machine learning, and have especially been applied for NLP segmentation tasks \cite{brants2000tnt} \cite{morwal2012named} \cite{ekbal2007pos}.&#10;&#10;For over twenty years, HMCs have been strictly generalized to Pairwise Markov Chains (PMCs) \cite{pieczynski2003pairwise}, a family of models including HMCs. In PMCs, the ``hidden&quot; chain is not necessarily Markov, and the noise is modeled in a more correlated – and thus more informative – way. However, PMCs keep the same advantages as HMCs in regards to the hidden data estimation. In particular, the training and estimation tasks still have linear complexity. PMCs have especially been studied for image segmentation, with discrete hidden variables and continuous observations. It turns out that using PMCs instead of HMCs can divide the error rate by two \cite{derrode2004signal} \cite{gorynin2018assessing}. However, for specific reasons relative to language processing, which we will develop later, PMCs have never been applied for NLP tasks.&#10;&#10;This paper explores the interest in using PMCs for three of the main text segmentation tasks: Part-Of-Speech (POS) Tagging, Chunking, and Named-Entity-Recognition (NER). The best methods for these tasks are based on Deep Learning models \cite{yang2019xlnet} \cite{akbik2018coling}. However, to produce excellent scores, these models require a large amount of extra-data, which results in very long training time and difficulties in deploying them with classic architectures.&#10;&#10;The paper is organized as follows. In the next section, we present and compare PMCs and HMCs, the bayesian segmentation methods, and the parameter estimation algorithm. The third sectionis devoted to the text segmentation tasks and an original way to adapt Markov chain models for these tasks while keeping fast training and relevant results. Experiments are presented in sectionfour. The last sectionis devoted to conclusions and perspectives. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Markov Chain Models, Efficient NLP Modeling, Mathematics, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1903.07785" label="1903.07785">
        <attvalues>
          <attvalue for="0" value="Cloze-driven Pretraining of Self-attention Networks" />
          <attvalue for="1" value="  We present a new approach for pretraining a bi-directional transformer model&#10;that provides significant performance gains across a variety of language&#10;understanding problems. Our model solves a cloze-style word reconstruction&#10;task, where each word is ablated and must be predicted given the rest of the&#10;text. Experiments demonstrate large performance gains on GLUE and new state of&#10;the art results on NER as well as constituency parsing benchmarks, consistent&#10;with the concurrently introduced BERT model. We also present a detailed&#10;analysis of a number of factors that contribute to effective pretraining,&#10;including data domain and size, model capacity, and variations on the cloze&#10;objective.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.10860" label="1812.10860">
        <attvalues>
          <attvalue for="0" value="Can You Tell Me How to Get Past Sesame Street? Sentence-Level&#10;  Pretraining Beyond Language Modeling" />
          <attvalue for="1" value="  Natural language understanding has recently seen a surge of progress with the&#10;use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et&#10;al., 2019) which are pretrained on variants of language modeling. We conduct&#10;the first large-scale systematic study of candidate pretraining tasks,&#10;comparing 19 different tasks both as alternatives and complements to language&#10;modeling. Our primary results support the use language modeling, especially&#10;when combined with pretraining on additional labeled-data tasks. However, our&#10;results are mixed across pretraining tasks and show some concerning trends: In&#10;ELMo's pretrain-then-freeze paradigm, random baselines are worryingly strong&#10;and results vary strikingly across target tasks. In addition, fine-tuning BERT&#10;on an intermediate task often negatively impacts downstream transfer. In a more&#10;positive trend, we see modest gains from multitask training, suggesting the&#10;development of more sophisticated multitask and transfer learning techniques as&#10;an avenue for further research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.04382" label="1906.04382">
        <attvalues>
          <attvalue for="0" value="DoubleTransfer at MEDIQA 2019: Multi-Source Transfer Learning for&#10;  Natural Language Understanding in the Medical Domain" />
          <attvalue for="1" value="  This paper describes our competing system to enter the MEDIQA-2019&#10;competition. We use a multi-source transfer learning approach to transfer the&#10;knowledge from MT-DNN and SciBERT to natural language understanding tasks in&#10;the medical domain. For transfer learning fine-tuning, we use multi-task&#10;learning on NLI, RQE and QA tasks on general and medical domains to improve&#10;performance. The proposed methods are proved effective for natural language&#10;understanding in the medical domain, and we rank the first place on the QA&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.02493" label="1911.02493">
        <attvalues>
          <attvalue for="0" value="SentiLARE: Sentiment-Aware Language Representation Learning with&#10;  Linguistic Knowledge" />
          <attvalue for="1" value="  Most of the existing pre-trained language representation models neglect to&#10;consider the linguistic knowledge of texts, which can promote language&#10;understanding in NLP tasks. To benefit the downstream tasks in sentiment&#10;analysis, we propose a novel language representation model called SentiLARE,&#10;which introduces word-level linguistic knowledge including part-of-speech tag&#10;and sentiment polarity (inferred from SentiWordNet) into pre-trained models. We&#10;first propose a context-aware sentiment attention mechanism to acquire the&#10;sentiment polarity of each word with its part-of-speech tag by querying&#10;SentiWordNet. Then, we devise a new pre-training task called label-aware masked&#10;language model to construct knowledge-aware language representation.&#10;Experiments show that SentiLARE obtains new state-of-the-art performance on a&#10;variety of sentiment analysis tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="cs/0506075" label="cs/0506075">
        <attvalues>
          <attvalue for="0" value="Seeing stars: Exploiting class relationships for sentiment&#10;  categorization with respect to rating scales" />
          <attvalue for="1" value="  We address the rating-inference problem, wherein rather than simply decide&#10;whether a review is &quot;thumbs up&quot; or &quot;thumbs down&quot;, as in previous sentiment&#10;analysis work, one must determine an author's evaluation with respect to a&#10;multi-point scale (e.g., one to five &quot;stars&quot;). This task represents an&#10;interesting twist on standard multi-class text categorization because there are&#10;several different degrees of similarity between class labels; for example,&#10;&quot;three stars&quot; is intuitively closer to &quot;four stars&quot; than to &quot;one star&quot;. We&#10;first evaluate human performance at the task. Then, we apply a meta-algorithm,&#10;based on a metric labeling formulation of the problem, that alters a given&#10;n-ary classifier's output in an explicit attempt to ensure that similar items&#10;receive similar labels. We show that the meta-algorithm can provide significant&#10;improvements over both multi-class and regression versions of SVMs when we&#10;employ a novel similarity measure appropriate to the problem.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.09263" label="2310.09263">
        <attvalues>
          <attvalue for="0" value="Table-GPT: Table-tuned GPT for Diverse Table Tasks" />
          <attvalue for="1" value="  Language models, such as GPT-3.5 and ChatGPT, demonstrate remarkable&#10;abilities to follow diverse human instructions and perform a wide range of&#10;tasks. However, when probing language models using a range of basic&#10;table-understanding tasks, we observe that today's language models are still&#10;sub-optimal in many table-related tasks, likely because they are pre-trained&#10;predominantly on \emph{one-dimensional} natural-language texts, whereas&#10;relational tables are \emph{two-dimensional} objects.&#10;  In this work, we propose a new &quot;\emph{table-tuning}&quot; paradigm, where we&#10;continue to train/fine-tune language models like GPT-3.5 and ChatGPT, using&#10;diverse table-tasks synthesized from real tables as training data, with the&#10;goal of enhancing language models' ability to understand tables and perform&#10;table tasks. We show that our resulting Table-GPT models demonstrate (1) better&#10;\emph{table-understanding} capabilities, by consistently outperforming the&#10;vanilla GPT-3.5 and ChatGPT, on a wide-range of table tasks, including holdout&#10;unseen tasks, and (2) strong \emph{generalizability}, in its ability to respond&#10;to diverse human instructions to perform new table-tasks, in a manner similar&#10;to GPT-3.5 and ChatGPT.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models, such as GPT and LLaMa, have recently demonstrated impressive abilities in performing diverse natural-language tasks~\cite{llm-gpt-3, llm-palm-2, llm-llama, llm-palm}. In the database literature, a number of pioneering work, such as~\cite{stanford-prompt-engineer, cta-prompt-engineering, em-prompt-engineering, llm-vision}, have also shown that by using ``prompt engineering'', to careful select the best instructions and few-shot examples for a particular task at hand, language models can be prompted to perform well on a number of table-tasks such as entity matching and data-imputation.&#10;&#10;While prompt-engineering is a promising direction to enhance model performance, it requires task-specific tuning (e.g., task-specific labeled-data to test the performance of different instruction/example combinations)~\cite{llm-gpt-3, prompt-engineering, prompt-engineering-2}. We in this work&#10;propose an orthogonal paradigm called ``table-tuning'', where instead of modifying prompts, we modify the weights of the underlying language models for once (i.e., not task-specific), by continuing to train them using diverse table-tasks as training data, to improve their ability to understand tables.&#10;We show that table-tuned \sys &#10;consistently outperform the vanilla GPT-3.5 and ChatGPT on a wide-range of table tasks, including new and unseen table-tasks. We note that our model-tuning approach is complementary to prompt-engineering, because carefully engineered prompts can continue to benefit both vanilla language-models and our table-tuned models.&#10;&#10;Today's language models cannot ``read tables'' reliably.&#10;While today's language models excel in natural-language tasks, we start by asking the question of whether these models are optimal for table-tasks, because after all, they are pre-trained predominantly on natural language texts, which are different from tables. &#10;&#10;More specifically, natural language texts are (1) one-directional, (2) read left-to-right, where (3) swapping two tokens will generally change the meaning of a sentence. In contrast, relational tables are (1) two-dimensional in nature with both rows and columns, (2) where reading top-to-bottom in the vertical direction for values in the same column, is crucial in many table-tasks. Furthermore, unlike text, (3) tables are largely ``invariant'' to row and column permutations, where swapping two rows or columns do not generally change the semantic meaning of the table. &#10;&#10;With this question in mind, we perform two simple tests to probe language models' ability to ``read'' tables and then answer basic questions, which we call (T-1) Missing-value-identification, and (T-2) Column-finding, as shown in Figure~\ref{fig:basic-tests}. &#10;&#10;In (T-1) Missing-value-identification, we show language models with a real table, presented in a markdown or alternative format, where we make sure that there is exactly one empty cell in the table. We then ask the model to identify the empty cell, by responding with the column-name and row-id of the empty cell, repeating for 1000 randomly sampled real tables. Despite the impressive ability of language-models like GPT-3.5 to perform diverse tasks, we find that they fail on a surprisingly large fraction (up to 74\%) of such tests, often responding with incorrect column-headers or row-ids -- for instance, in the example shown in Figure~\ref{fig:basic-tests}, the model may answer that the column ``\code{music}'' has a missing cell, when the correct answer should be ``\code{art}''. &#10;&#10;In order to ensure that there is no ambiguity in what ``missing value'' or ``empty cell'' could mean to language models, we design a second and even simpler test, which we refer to as: (T-2) Column-finding, shown on the right of Figure~\ref{fig:basic-tests}. In this test, we present a language model with a real table, and ask it to find a specific cell-value that appears exactly once in the entire table (e.g., ``\code{93}'' in this example), and then respond with the column-name of the that value. We find that language models such as GPT-3.5 are prone to fail on such tests again (e.g., answering that ``\code{93}'' is in column ``\code{art}'' when the correct answer is ``\code{music}''), on over half of such tests. &#10;&#10;We believe these simple probes show that today's large language models, when pre-trained on large amounts of one-directional natural-language texts, are not best-suited to ``read'' two-dimensional tables, especially in the vertical direction, which however is crucial in performing many table-tasks. &#10;&#10;Consider, for example, the popular NLP task of (T-3) Table-QA~\cite{table-qa-tabfact, table-qa-wikitablequestions, table-qa-2}, where the task is to answer a natural-language question, based on the content of the table. The left side of Figure~\ref{fig:more-tests} shows such an example. Given the question ``\code{How many second-graders scored over 90 in art, in the table below?}'' Imagine that a model is not able to ``read'' tables correctly, it may reason that both ``\code{Jennifer}'' and ``\code{James}'' satisfy the condition (because it believes ``\code{93}'' is in the column ``\code{art}'', like shown in Figure~\ref{fig:basic-tests} (Right)), and may answer ``\code{2}'' instead of the correct ``\code{1}''. We emphasize that the ability to read in the vertical direction (top-to-bottom for values in the same column) is similarly important in many other table-tasks, such as data-imputation (shown on the right of Figure~\ref{fig:more-tests}), data-transformation, error-detection, NL-to-SQL, etc., like the list in Table~\ref{tab:task-summary} would show, which includes a diverse set of table-tasks considered in this work.&#10;&#10;In addition, we find that large language models are sensitive to the order in which columns are presented in a table -- e.g., when we swap the order of two columns in a table, a model can change its response for a table-task, even when such a swap should not change the semantic meaning of the table, at least to humans. This is presumably because language-models are pre-trained on text where the order of tokens matters (e.g., ``\code{Jennifer called you}'' vs. ``\code{you called Jennifer}''), leading to sub-optimal behaviors on tables.&#10;&#10;We believe observations like these point to opportunities for us to improve the underlying language model, by enhancing their ability to understand tables and perform table-tasks. &#10;&#10;Instruction-tuning in NLP: train language-models to follow diverse human instructions.&#10;To change the behaviour of language models, successful attempts have been made in the NLP community, using a technique known as ``instruction-tuning'' in the literature~\cite{flan, instruct-gpt, self-instruct, t0, super-natural-instruction}. &#10;&#10;It was observed in the NLP community~\cite{flan, llm-gpt-3, instruct-gpt}, that earlier versions of pre-trained language models, such as GPT-3, is able to complete a sentence with the next likely token (e.g., ``\code{write a bed-time}'' $arrow$ ``\code{story}''), but cannot reliable follow higher-level instructions from humans (e.g., ``\code{write a bed-time story for a 3 years-old, in 100 words}''), a behavior that is only demonstrated in later models such as ChatGPT. &#10;&#10;Instruction-tuning was the key technique invented that continues to &#10;train GPT-like models into ChatGPT-like models, in a process shown on the left of Figure~\ref{fig:instruction-tuning-vs-table-tuning}. Diverse training data in the form of ``\code{(instruction, completion)}'' pairs are constructed, often manually annotated by human labellers~\cite{instruct-gpt}, e.g. (\code{``write a bed-time story'' $arrow$ an-actual-story}), to continue train language-models on these explicit demonstrations of how to follow high-level human instructions, leading to well-known models such as ChatGPT/InstructGPT~\cite{instruct-gpt, chatgpt}, as well as their open-source counterparts like Stanford-Alpaca~\cite{stanford-alpaca} and LLaMa-chat~\cite{llm-llama}.&#10;&#10;Table-tuning: train language-models to understand tables.&#10;We believe that the research on instruction-tuning in NLP, which successfully enhances language-models ability to follow human instructions, holds lessons for us when we aim to enhance language-models ability to understand tables and perform table-tasks.&#10;&#10;In this work, we propose a ``table-tuning'' paradigm analogous to instruction-tuning, where we continue to train language-models, using diverse training data in the form of (\code{instruction, table, completion}), which we synthesize using large amounts of real tables. This process is illustrated on the right of Figure~\ref{fig:instruction-tuning-vs-table-tuning}. &#10;&#10;Through extensive experiments, we show that ``table-tuning'' is a promising new direction, as our resulting \sys models are:&#10;\begin{itemize}[noitemsep,topsep=0pt,leftmargin=*]&#10;\item[] (1) \underline{Strong table models}, which substantially outperform 175B GPT-3.5 and ChatGPT, on a wide range of seen and unseen table-tasks, as we summarize in Table~\ref{tab:task-summary} and Figure~\ref{fig:main-quality-chatgpt}; &#10;\item[] (2) \underline{Generalizable to new tasks}, as they can respond well to novel and unseen table-tasks, similar to how Chat-GPT could generalize and respond to new and unseen NLP tasks, like shown in Figure~\ref{fig:unseen-new-tests}.&#10;\end{itemize}&#10;&#10;Contributions. We make the following contributions:&#10;&#10;\begin{itemize}[noitemsep,topsep=0pt,leftmargin=*]&#10;\item We propose a new ``table-tuning'' paradigm to continue to train language models, specifically designed to enhance language-models' ability to perform table-tasks, using diverse table-tasks synthesized from large amounts of real tables, in a ``synthesis-then-augment'' process. \item We develop task-level, table-level, instruction-level, and completion-level data augmentation techniques for table-tuning, which we show are crucial to avoid over-fitting and ensure the generality of \sys.&#10;\item We show that \sys not only excels on table-tasks in both zero-shot and few-shot settings out of box, but can also serve as a ``table foundation model'' and used as a better starting point than vanilla GPT, for down-stream single-task optimizations such as task-specific fine-tuning and prompt-engineering.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Limitations, Artificial Intelligence, Model Fine-Tuning Techniques, Table Understanding Tasks" />
        </attvalues>
      </node>
      <node id="2408.04303" label="2408.04303">
        <attvalues>
          <attvalue for="0" value="Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language&#10;  Adaptation of LLMs for Low-Resource NLP" />
          <attvalue for="1" value="  The development of monolingual language models for low and mid-resource&#10;languages continues to be hindered by the difficulty in sourcing high-quality&#10;training data. In this study, we present a novel cross-lingual vocabulary&#10;transfer strategy, trans-tokenization, designed to tackle this challenge and&#10;enable more efficient language adaptation. Our approach focuses on adapting a&#10;high-resource monolingual LLM to an unseen target language by initializing the&#10;token embeddings of the target language using a weighted average of&#10;semantically similar token embeddings from the source language. For this, we&#10;leverage a translation resource covering both the source and target languages.&#10;We validate our method with the Tweeties, a series of trans-tokenized LLMs, and&#10;demonstrate their competitive performance on various downstream tasks across a&#10;small but diverse set of languages. Additionally, we introduce Hydra LLMs,&#10;models with multiple swappable language modeling heads and embedding tables,&#10;which further extend the capabilities of our trans-tokenization strategy. By&#10;designing a Hydra LLM based on the multilingual model TowerInstruct, we&#10;developed a state-of-the-art machine translation model for Tatar, in a&#10;zero-shot manner, completely bypassing the need for high-quality parallel data.&#10;This breakthrough is particularly significant for low-resource languages like&#10;Tatar, where high-quality parallel data is hard to come by. By lowering the&#10;data and time requirements for training high-quality models, our&#10;trans-tokenization strategy allows for the development of LLMs for a wider&#10;range of languages, especially those with limited resources. We hope that our&#10;work will inspire further research and collaboration in the field of&#10;cross-lingual vocabulary transfer and contribute to the empowerment of&#10;languages on a global scale.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multilingual tokenization is unfair, with all existing approaches inadvertently favoring some languages over others \cite{petrov-etal-2023-tokenizer-unfairness,rust-etal-2021-good}. This bias is particularly pronounced in multilingual subword tokenization techniques, which face the impossible task of distributing their token capacity equitably among all supported languages. Western European languages often benefit from this, thanks to their shared alphabet and linguistic heritage \cite{limisiewicz-etal-2023-tokenization}. Although character or byte-level encoders appear to handle diverse scripts more fairly, they frequently struggle to capture meaningful word-level information, especially in non-ideographic languages with limited alphabets \cite{libovicky-etal-2022-dont,edman-etal-2022-subword}. Furthermore, byte-level tokenizers also display bias due to the substantial disparities in unicode encoding efficiency across languages. &#10;&#10;In light of these challenges, we stress the need for a more personalized approach, where each language is equipped with its own tokenizer, specifically tailored to its unique needs.&#10;Unfortunately, the challenge of developing monolingual language models for all the world's languages has never been more present due to the vast amounts of data required to train large language models (LLMs), as evidenced by the technical reports of Mistral \citepar{jiang-etal-2023-mistral}, OLMo \citepar{groeneveld-etal-2024-olmo} and Gemma \citepar{gemma-2024-report}. The trillion tokens required for training LLMs simply does not exist in most languages \cite{joshi-etal-2020-state}, turning transfer learning into a requirement.&#10;&#10;Moreover, serving a wide array of monolingual LLMs at scale remains impractical. Efficient computation necessitates the batch-processing of requests \cite{pope-2022-efficiently}, but many languages also suffer from intermittent workloads. This also makes it unsustainable to dedicate extensive GPU resources to continuously host often-idling LLMs, while the time required to load them back into memory impedes many commercial applications that require low latency \cite{apple-llm-flask}.&#10;&#10;In this paper, we introduce several key innovations designed to democratize the training and deployment of high-quality monolingual models across a diverse set of languages. More specifically, we demonstrate how model conversion enables researchers to adapt LLMs to new languages using a very limited amount of resources, with a performance competitive with continual pre-training. Our approach preserves most layers of the original model, thereby facilitating the batch-processing of queries written in different languages, a critical factor in making the deployment of language-specific models economically viable.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Lingual Transfer Learning, Computer Science, Multilingual Natural Language Processing, Linguistics, Artificial Intelligence, Language Model Development, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.15425" label="2305.15425">
        <attvalues>
          <attvalue for="0" value="Language Model Tokenizers Introduce Unfairness Between Languages" />
          <attvalue for="1" value="  Recent language models have shown impressive multilingual performance, even&#10;when not explicitly trained for it. Despite this, there are concerns about the&#10;quality of their outputs across different languages. In this paper, we show how&#10;disparity in the treatment of different languages arises at the tokenization&#10;stage, well before a model is even invoked. The same text translated into&#10;different languages can have drastically different tokenization lengths, with&#10;differences up to 15 times in some cases. These disparities persist even for&#10;tokenizers that are intentionally trained for multilingual support.&#10;Character-level and byte-level models also exhibit over 4 times the difference&#10;in the encoding length for some language pairs. This induces unfair treatment&#10;for some language communities in regard to the cost of accessing commercial&#10;language services, the processing time and latency, as well as the amount of&#10;content that can be provided as context to the models. Therefore, we make the&#10;case that we should train future language models using multilingually fair&#10;subword tokenizers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.17179" label="2305.17179">
        <attvalues>
          <attvalue for="0" value="Tokenization Impacts Multilingual Language Modeling: Assessing&#10;  Vocabulary Allocation and Overlap Across Languages" />
          <attvalue for="1" value="  Multilingual language models have recently gained attention as a promising&#10;solution for representing multiple languages in a single model. In this paper,&#10;we propose new criteria to evaluate the quality of lexical representation and&#10;vocabulary overlap observed in sub-word tokenizers. Our findings show that the&#10;overlap of vocabulary across languages can be actually detrimental to certain&#10;downstream tasks (POS, dependency tree labeling). In contrast, NER and&#10;sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing&#10;vocabulary. We also observe that the coverage of the language-specific tokens&#10;in the multilingual vocabulary significantly impacts the word-level tasks. Our&#10;study offers a deeper understanding of the role of tokenizers in multilingual&#10;language models and guidelines for future model developers to choose the most&#10;suitable tokenizer for their specific application before undertaking costly&#10;model pre-training&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.01304" label="2212.01304">
        <attvalues>
          <attvalue for="0" value="Subword-Delimited Downsampling for Better Character-Level Translation" />
          <attvalue for="1" value="  Subword-level models have been the dominant paradigm in NLP. However,&#10;character-level models have the benefit of seeing each character individually,&#10;providing the model with more detailed information that ultimately could lead&#10;to better models. Recent works have shown character-level models to be&#10;competitive with subword models, but costly in terms of time and computation.&#10;Character-level models with a downsampling component alleviate this, but at the&#10;cost of quality, particularly for machine translation. This work analyzes the&#10;problems of previous downsampling methods and introduces a novel downsampling&#10;method which is informed by subwords. This new downsampling method not only&#10;outperforms existing downsampling methods, showing that downsampling characters&#10;can be done without sacrificing quality, but also leads to promising&#10;performance compared to subword models for translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;&#10;Character-level models have been of interest for several years, notably used for character-level translation prior to the advent of Transformer models with some success \cite{costa2016character, lee2017fully}.&#10;&#10;\cite{lee2017fully} raises the issue that the length of the sequences require the models to potentially capture much longer range dependencies, and as such introduces a downsampling method. This method consists of convolutional layers followed by a sequence-length-wise max pooling. The convolutional layers serve to learn local patterns in the characters and the max pooling is intended to reduce the length of the input, alleviating the long-range dependency issue. Reducing the length with a downsampling method such as max pooling can be thought of as a transformation from character tokens to pseudo-word tokens.&#10;&#10;More recently, character-level NLP has been investigated with the use of the Transformer. The Transformer, while better able to handle longer sequences than RNNs, can similarly suffer when on the character-level due to the $O(n^2)$ complexity of self-attention. Nevertheless, ByT5 \cite{xue2022byt5}, a multilingual unsupervised pretrained character-level model, has shown comparable results to its subword-level counterpart mT5, while demonstrating some beneficial properties such as robustness to character-level noise. It is however slower than subword models both in training and test time. &#10;&#10;The Charformer \cite{tay2021charformer} reintroduces downsampling using a novel downsampling method, GBST, which uses a learned, weighted-average of character n-grams for each downsampled token. This shows similar performance to ByT5 while also being faster, however its performance on generative tasks such as NMT appears less promising. \cite{edman2022patching} investigate the usefulness of Charformer's GBST method for NMT, finding that using GBST decoder-side does not work out-of-the-box due to an information leak, and that even with a fix to the leak, it does not perform up to the level of the aforementioned convolutional downsampling method. &#10;&#10;Similar to the Charformer, CharacterBERT \cite{boukkouri2020characterbert} shows that incorporating character information can be useful on encoder-only tasks. They use a CNN similar to \cite{lee2017fully}'s, but downsample based on the length of the whole word, rather than at a fixed size. Their results show better generalization than subword models on classification of medical data, despite it not seeing any medical data in pretraining. They attribute this to its more generalized internal vocabulary as a result of receiving characters as input. &#10;&#10;In the context of NMT, \cite{libovicky2021don} attempt to answer why the current state-of-the-art models are not character models, to which the answer appears that their performance is not superior to subword models, and that downsampling methods sacrifice quality for efficiency. &#10;&#10;In doing so, \cite{libovicky2021don} convert existing character models such as \cite{lee2017fully}'s to the Transformer architecture. With this, they propose a two-step decoding method, which adds an LSTM layer that takes as input the hidden representation of the Transformer decoder, concatenated with separately-learned character embeddings. The light-weight nature of the two-step decoder means little computation time is added. &#10;&#10;We show a tabular summary of the relevant previous work in Table~\ref{tab:prev_work}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2312.11514" label="2312.11514">
        <attvalues>
          <attvalue for="0" value="LLM in a flash: Efficient Large Language Model Inference with Limited&#10;  Memory" />
          <attvalue for="1" value="  Large language models (LLMs) are central to modern natural language&#10;processing, delivering exceptional performance in various tasks. However, their&#10;substantial computational and memory requirements present challenges,&#10;especially for devices with limited DRAM capacity. This paper tackles the&#10;challenge of efficiently running LLMs that exceed the available DRAM capacity&#10;by storing the model parameters in flash memory, but bringing them on demand to&#10;DRAM. Our method involves constructing an inference cost model that takes into&#10;account the characteristics of flash memory, guiding us to optimize in two&#10;critical areas: reducing the volume of data transferred from flash and reading&#10;data in larger, more contiguous chunks. Within this hardware-informed&#10;framework, we introduce two principal techniques. First, &quot;windowing&quot;&#10;strategically reduces data transfer by reusing previously activated neurons,&#10;and second, &quot;row-column bundling&quot;, tailored to the sequential data access&#10;strengths of flash memory, increases the size of data chunks read from flash&#10;memory. These methods collectively enable running models up to twice the size&#10;of the available DRAM, with a 4-5x and 20-25x increase in inference speed&#10;compared to naive loading approaches in CPU and GPU, respectively. Our&#10;integration of sparsity awareness, context-adaptive loading, and a&#10;hardware-oriented design paves the way for effective inference of LLMs on&#10;devices with limited memory.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.06535" label="2211.06535">
        <attvalues>
          <attvalue for="0" value="A unified one-shot prosody and speaker conversion system with&#10;  self-supervised discrete speech units" />
          <attvalue for="1" value="  We present a unified system to realize one-shot voice conversion (VC) on the&#10;pitch, rhythm, and speaker attributes. Existing works generally ignore the&#10;correlation between prosody and language content, leading to the degradation of&#10;naturalness in converted speech. Additionally, the lack of proper language&#10;features prevents these systems from accurately preserving language content&#10;after conversion. To address these issues, we devise a cascaded modular system&#10;leveraging self-supervised discrete speech units as language representation.&#10;These discrete units provide duration information essential for rhythm&#10;modeling. Our system first extracts utterance-level prosody and speaker&#10;representations from the raw waveform. Given the prosody representation, a&#10;prosody predictor estimates pitch, energy, and duration for each discrete unit&#10;in the utterance. A synthesizer further reconstructs speech based on the&#10;predicted prosody, speaker representation, and discrete units. Experiments show&#10;that our system outperforms previous approaches in naturalness,&#10;intelligibility, speaker transferability, and prosody transferability. Code and&#10;samples are publicly available.&#10;" />
          <attvalue for="2" value="&#10;Human speech carries different aspects of information, including prosody, speaker traits, and language content.&#10;The objective of voice conversion (VC) is to control individual speech attributes with language content unchanged.&#10;In this paper, we focus on the conversion of three main attributes: pitch-energy, speaker traits, and rhythm.&#10;&#10;One-shot voice conversion is challenging as the model can only access source and target speech without speaker identities given.&#10;Existing works mostly learned a speaker encoder jointly to isolate speaker information from prosody and language content.&#10;AutoVC~\cite{AUTOVC} attempted to disentangle speaker traits from language by a carefully designed autoencoder.&#10;To separate speaker timbre from prosody, works~\cite{AUTOVCF0} provided pitch contours explicitly to the system.&#10;Several works~\cite{liu19c_interspeech,VQVC+} further improved content separation by learning representations with vector quantization.&#10;Additionally, VQMIVC~\cite{VQMIVC} proposed to minimize mutual information between content, speaker, and pitch representations for better disentanglement.&#10;The above methods, however, focused largely on speaker conversion.&#10;In applications such as emotion style transfer, separate control for prosody is desirable.&#10;Several works built upon AutoVC attempted to control prosodic attributes of speech.&#10;AutoPST~\cite{AUTOPST} modeled rhythm by similarity-based re-sampling.&#10;SpeechSplit~\cite{SpeechSplit,SpeechSplit2} achieved rhythm and pitch conversion with multiple carefully designed autoencoders.&#10;Leveraging these works, SRDVC~\cite{SRDVC} presented a unified one-shot VC system that allows control over both prosody and speaker attributes.&#10;&#10;Despite their success, we found that improvements could be made.&#10;Previous approaches often suffered from intelligibility degradation after conversion due to the lack of disentangled language representations.&#10;To address this, recent approaches began to explore self-supervised speech representation~\cite{wav2vec2,wav2vec} (S3R) as a source of language information.&#10;However, these works~\cite{NANSY,s3prl-vc} generally focus on continuous S3R and are limited to speaker conversion.&#10;In contrast, we explore the use of discrete self-supervised speech units on both speaker and prosody conversion.&#10;Compared to continuous S3R, these discrete units formed from clustering naturally encode duration via repeated tokens, which is crucial for rhythm modeling.&#10;&#10;Furthermore, we adopt a different modeling approach for prosodic features.&#10;In SRDVC, the pitch representation is directly extracted from a given pitch contour without explicit access to language information.&#10;However, as prosody is correlated with language~\cite{doi:10.1177/002383099704000203}, it causes naturalness degradation of prosody-converted samples (see Section~\ref{ssec:res-pro}).&#10;Energy and duration, although important prosodic features, are also not explicitly modeled.&#10;To address these issues, we propose a cascaded modular system that leverages discrete speech units for language information.&#10;First, our system extracts prosody and speaker representations from the raw waveform.&#10;Given the prosody representation, a prosody predictor estimates the pitch, energy, and duration of each speech unit.&#10;In combination with the predicted prosody, a synthesizer reconstructs speech based on speaker representation and discrete units.&#10;Empirical results demonstrate that our system outperforms previous approaches in intelligibility, naturalness, speaker and prosody transferability.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Speech Processing, Linguistics, Speech Representation Models, Signal Processing, Voice Conversion Systems, Prosody Transfer Techniques, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1905.05879" label="1905.05879">
        <attvalues>
          <attvalue for="0" value="AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss" />
          <attvalue for="1" value="  Non-parallel many-to-many voice conversion, as well as zero-shot voice&#10;conversion, remain under-explored areas. Deep style transfer algorithms, such&#10;as generative adversarial networks (GAN) and conditional variational&#10;autoencoder (CVAE), are being applied as new solutions in this field. However,&#10;GAN training is sophisticated and difficult, and there is no strong evidence&#10;that its generated speech is of good perceptual quality. On the other hand,&#10;CVAE training is simple but does not come with the distribution-matching&#10;property of a GAN. In this paper, we propose a new style transfer scheme that&#10;involves only an autoencoder with a carefully designed bottleneck. We formally&#10;show that this scheme can achieve distribution-matching style transfer by&#10;training only on a self-reconstruction loss. Based on this scheme, we proposed&#10;AUTOVC, which achieves state-of-the-art results in many-to-many voice&#10;conversion with non-parallel data, and which is the first to perform zero-shot&#10;voice conversion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.07370" label="2004.07370">
        <attvalues>
          <attvalue for="0" value="F0-consistent many-to-many non-parallel voice conversion via conditional&#10;  autoencoder" />
          <attvalue for="1" value="  Non-parallel many-to-many voice conversion remains an interesting but&#10;challenging speech processing task. Many style-transfer-inspired methods such&#10;as generative adversarial networks (GANs) and variational autoencoders (VAEs)&#10;have been proposed. Recently, AutoVC, a conditional autoencoders (CAEs) based&#10;method achieved state-of-the-art results by disentangling the speaker identity&#10;and speech content using information-constraining bottlenecks, and it achieves&#10;zero-shot conversion by swapping in a different speaker's identity embedding to&#10;synthesize a new voice. However, we found that while speaker identity is&#10;disentangled from speech content, a significant amount of prosodic information,&#10;such as source F0, leaks through the bottleneck, causing target F0 to fluctuate&#10;unnaturally. Furthermore, AutoVC has no control of the converted F0 and thus&#10;unsuitable for many applications. In the paper, we modified and improved&#10;autoencoder-based voice conversion to disentangle content, F0, and speaker&#10;identity at the same time. Therefore, we can control the F0 contour, generate&#10;speech with F0 consistent with the target speaker, and significantly improve&#10;quality and similarity. We support our improvement through quantitative and&#10;qualitative analysis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.04154" label="2006.04154">
        <attvalues>
          <attvalue for="0" value="VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net&#10;  architecture" />
          <attvalue for="1" value="  Voice conversion (VC) is a task that transforms the source speaker's timbre,&#10;accent, and tones in audio into another one's while preserving the linguistic&#10;content. It is still a challenging work, especially in a one-shot setting.&#10;Auto-encoder-based VC methods disentangle the speaker and the content in input&#10;speech without given the speaker's identity, so these methods can further&#10;generalize to unseen speakers. The disentangle capability is achieved by vector&#10;quantization (VQ), adversarial training, or instance normalization (IN).&#10;However, the imperfect disentanglement may harm the quality of output speech.&#10;In this work, to further improve audio quality, we use the U-Net architecture&#10;within an auto-encoder-based VC system. We find that to leverage the U-Net&#10;architecture, a strong information bottleneck is necessary. The VQ-based&#10;method, which quantizes the latent vectors, can serve the purpose. The&#10;objective and the subjective evaluations show that the proposed method performs&#10;well in both audio naturalness and speaker similarity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.11284" label="2004.11284">
        <attvalues>
          <attvalue for="0" value="Unsupervised Speech Decomposition via Triple Information Bottleneck" />
          <attvalue for="1" value="  Speech information can be roughly decomposed into four components: language&#10;content, timbre, pitch, and rhythm. Obtaining disentangled representations of&#10;these components is useful in many speech analysis and generation applications.&#10;Recently, state-of-the-art voice conversion systems have led to speech&#10;representations that can disentangle speaker-dependent and independent&#10;information. However, these systems can only disentangle timbre, while&#10;information about pitch, rhythm and content is still mixed together. Further&#10;disentangling the remaining speech components is an under-determined problem in&#10;the absence of explicit annotations for each component, which are difficult and&#10;expensive to obtain. In this paper, we propose SpeechSplit, which can blindly&#10;decompose speech into its four components by introducing three carefully&#10;designed information bottlenecks. SpeechSplit is among the first algorithms&#10;that can separately perform style transfer on timbre, pitch and rhythm without&#10;text labels. Our code is publicly available at&#10;https://github.com/auspicious3000/SpeechSplit.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.08757" label="2208.08757">
        <attvalues>
          <attvalue for="0" value="Speech Representation Disentanglement with Adversarial Mutual&#10;  Information Learning for One-shot Voice Conversion" />
          <attvalue for="1" value="  One-shot voice conversion (VC) with only a single target speaker's speech for&#10;reference has become a hot research topic. Existing works generally disentangle&#10;timbre, while information about pitch, rhythm and content is still mixed&#10;together. To perform one-shot VC effectively with further disentangling these&#10;speech components, we employ random resampling for pitch and content encoder&#10;and use the variational contrastive log-ratio upper bound of mutual information&#10;and gradient reversal layer based adversarial mutual information learning to&#10;ensure the different parts of the latent space containing only the desired&#10;disentangled representation during training. Experiments on the VCTK dataset&#10;show the model achieves state-of-the-art performance for one-shot VC in terms&#10;of naturalness and intellgibility. In addition, we can transfer characteristics&#10;of one-shot VC on timbre, pitch and rhythm separately by speech representation&#10;disentanglement. Our code, pre-trained models and demo are available at&#10;https://im1eon.github.io/IS2022-SRDVC/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.14513" label="2110.14513">
        <attvalues>
          <attvalue for="0" value="Neural Analysis and Synthesis: Reconstructing Speech from&#10;  Self-Supervised Representations" />
          <attvalue for="1" value="  We present a neural analysis and synthesis (NANSY) framework that can&#10;manipulate voice, pitch, and speed of an arbitrary speech signal. Most of the&#10;previous works have focused on using information bottleneck to disentangle&#10;analysis features for controllable synthesis, which usually results in poor&#10;reconstruction quality. We address this issue by proposing a novel training&#10;strategy based on information perturbation. The idea is to perturb information&#10;in the original input signal (e.g., formant, pitch, and frequency response),&#10;thereby letting synthesis networks selectively take essential attributes to&#10;reconstruct the input signal. Because NANSY does not need any bottleneck&#10;structures, it enjoys both high reconstruction quality and controllability.&#10;Furthermore, NANSY does not require any labels associated with speech data such&#10;as text and speaker information, but rather uses a new set of analysis&#10;features, i.e., wav2vec feature and newly proposed pitch feature, Yingram,&#10;which allows for fully self-supervised training. Taking advantage of fully&#10;self-supervised training, NANSY can be easily extended to a multilingual&#10;setting by simply training it with a multilingual dataset. The experiments show&#10;that NANSY can achieve significant improvement in performance in several&#10;applications such as zero-shot voice conversion, pitch shift, and time-scale&#10;modification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.13749" label="2210.13749">
        <attvalues>
          <attvalue for="0" value="AugCSE: Contrastive Sentence Embedding with Diverse Augmentations" />
          <attvalue for="1" value="  Data augmentation techniques have been proven useful in many applications in&#10;NLP fields. Most augmentations are task-specific, and cannot be used as a&#10;general-purpose tool. In our work, we present AugCSE, a unified framework to&#10;utilize diverse sets of data augmentations to achieve a better, general&#10;purpose, sentence embedding model. Building upon the latest sentence embedding&#10;models, our approach uses a simple antagonistic discriminator that&#10;differentiates the augmentation types. With the finetuning objective borrowed&#10;from domain adaptation, we show that diverse augmentations, which often lead to&#10;conflicting contrastive signals, can be tamed to produce a better and more&#10;robust sentence representation. Our methods achieve state-of-the-art results on&#10;downstream transfer tasks and perform competitively on semantic textual&#10;similarity tasks, using only unsupervised data.&#10;" />
          <attvalue for="2" value="&#10;&#10;Data augmentation in NLP can be useful in many situations, from low resource data setting, domain adaptation \cite{wei2021few}, debiasing \cite{dinan2020queens}, to improving generalization, robustness \cite{dhole2021nl}. In the vision domain, \cite{chen2020simple} shows that a diverse set of augmentation can be used to learn a robust general-purpose representation with contrastive learning. Similar work in sentence embedding space (\cite{gao2021simcse, chuang2022diffcse}) has shown that a simple single augmentation such as dropouts from transformers \cite{devlin2019bert} can be used for contrastive objective. However, no previous work has thoroughly explored the impacts of a diverse set of augmentations with contrastive learning in the sentence embedding space. It is not straightforward to find the best augmentations that work for contrastive learning in different datasets or tasks \cite{gao2021simcse}. Single augmentation can instill invariance in models for a specific aspects of linguistic variability, while naively combining a diverse set of augmentations can lead to contradicting gradients, preventing models from generalizing well (Table~\ref{tab:ablation-discriminator}). In this work, we present AugCSE (Figure~\ref{fig:architecture}), a general approach to select and unify a diverse set of augmentations for the purpose of building a general-purpose sentence embedding. During training, in addition to using contrastive loss, we randomly perturb sentences with different augmentations and use a discriminator loss to unify embeddings from diverse augmentations. In short, our work presents the following key contributions:&#10;&#10;\begin{itemize}[noitemsep,topsep=0pt]&#10; \item We show simple data augmentation methods can be used to improve individual tasks, while degrading performance on other tasks (due to shifted domain distribution).&#10; \item We present our simple discriminator objective that achieves competitive results on sentence similarity task (STS) and transfer classification tasks against state-of-the-art methods.&#10; \item We demonstrate through ablation and visualization that our model can unify contrasting distribution from diverse augmentations and that simple rule-based augmentations are sufficient for achieving competitive results.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Data Augmentation Techniques, Linguistics, Artificial Intelligence, Sentence Embedding Models, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2103.07552" label="2103.07552">
        <attvalues>
          <attvalue for="0" value="Few-Shot Text Classification with Triplet Networks, Data Augmentation,&#10;  and Curriculum Learning" />
          <attvalue for="1" value="  Few-shot text classification is a fundamental NLP task in which a model aims&#10;to classify text into a large number of categories, given only a few training&#10;examples per category. This paper explores data augmentation -- a technique&#10;particularly suitable for training with limited data -- for this few-shot,&#10;highly-multiclass text classification setting. On four diverse text&#10;classification tasks, we find that common data augmentation techniques can&#10;improve the performance of triplet networks by up to 3.0% on average.&#10;  To further boost performance, we present a simple training strategy called&#10;curriculum data augmentation, which leverages curriculum learning by first&#10;training on only original examples and then introducing augmented data as&#10;training progresses. We explore a two-stage and a gradual schedule, and find&#10;that, compared with standard single-stage training, curriculum data&#10;augmentation trains faster, improves performance, and remains robust to high&#10;amounts of noising from augmentation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.02721" label="2112.02721">
        <attvalues>
          <attvalue for="0" value="NL-Augmenter: A Framework for Task-Sensitive Natural Language&#10;  Augmentation" />
          <attvalue for="1" value="  Data augmentation is an important component in the robustness evaluation of&#10;models in natural language processing (NLP) and in enhancing the diversity of&#10;the data they are trained on. In this paper, we present NL-Augmenter, a new&#10;participatory Python-based natural language augmentation framework which&#10;supports the creation of both transformations (modifications to the data) and&#10;filters (data splits according to specific features). We describe the framework&#10;and an initial set of 117 transformations and 23 filters for a variety of&#10;natural language tasks. We demonstrate the efficacy of NL-Augmenter by using&#10;several of its transformations to analyze the robustness of popular natural&#10;language models. The infrastructure, datacards and robustness analysis results&#10;are available publicly on the NL-Augmenter repository&#10;(https://github.com/GEM-benchmark/NL-Augmenter).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.09631" label="2402.09631">
        <attvalues>
          <attvalue for="0" value="Representation Surgery: Theory and Practice of Affine Steering" />
          <attvalue for="1" value="  Language models often exhibit undesirable behavior, e.g., generating toxic or&#10;gender-biased text. In the case of neural language models, an encoding of the&#10;undesirable behavior is often present in the model's representations. Thus, one&#10;natural (and common) approach to prevent the model from exhibiting undesirable&#10;behavior is to steer the model's representations in a manner that reduces the&#10;probability of it generating undesirable text. This paper investigates the&#10;formal and empirical properties of steering functions, i.e., transformation of&#10;the neural language model's representations that alter its behavior. First, we&#10;derive two optimal, in the least-squares sense, affine steering functions under&#10;different constraints. Our theory provides justification for existing&#10;approaches and offers a novel, improved steering approach. Second, we offer a&#10;series of experiments that demonstrate the empirical effectiveness of the&#10;methods in mitigating bias and reducing toxic generation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language models (LMs) based on neural networks contain representations that encode diverse aspects of natural language. &#10;The manipulation of these representations, referred to as representation surgery, enables to both better understand the model's behavior and to shape the text it generates \cite{bolukbasi2016man,ravfogel2020null,elazar2021amnesic,feder2021causalm,meng2022locating,geva2021transformer,ghandeharioun2024patchscope}. &#10;One form of representation surgery is called steering, whose goal is to shift a subset of the representations towards a target concept in such a way that the representations encode that concept.&#10;For instance, one may wish to steer the representations towards those that encode non-toxic text to prevent the model from generating harmful content \cite{wallace-etal-2019-universal, sheng2019woman}.&#10;While there are many manners to steer representations, this paper focuses on affine steering functions that constitute a minimal change to the representations. &#10;Our paper provides the basic theory to support common techniques already present in the literature.&#10;&#10;The key conceptual point in our paper is the connection between concept erasure techniques and steering \cite{ravfogel2020null, ravfogel2022linear, ravfogel2022adversarial, belrose2024leace, guerner2023geometric}. &#10;Concept erasure techniques remove specific concepts from the representations.&#10;For instance, in the case of gender, one could apply a concept erasure technique to prevent the model from being able to distinguish between male and female-centric text. &#10;Such an application may be particularly relevant for mitigating gender bias, as text generated by models often encodes societal biases with respect to gender \cite{bolukbasi2016man, zhao-etal-2018-learning}.&#10;&#10;However, in the context of toxicity, concept erasure techniques make less sense.&#10;If one erases the concept of toxicity from the model's representations, the outcome may be that the model loses the ability to distinguish between toxic and non-toxic text.&#10;And, in fact, the model could potentially generate toxic text at a higher rate as a result.&#10;In contrast, most natural use cases relating to toxicity require that the model's behavior is steered towards only generating non-toxic text rather than erasing the model's awareness of toxicity \cite{subramani2022extracting, li2023inference}.&#10;Thus, at first blush, concept erasure is an inadequate tool for steering.&#10;&#10;Digging into the formal underpinning of concept erasure, however, we find that concept erasure techniques are built on the notion of guardedness \cite{ravfogel-etal-2023-linear}.&#10;In words, representations are said to be (affinely) guarded with respect to a concept if no linear classifier can recover the concept from the representations above chance. &#10;There are many functions that induce guardedness.&#10;For instance, trivially mapping all representations to zero enforces that any downstream classifier acts the same, notwithstanding the specific representation that is given as input.&#10;However, such a guarding function would be of limited practical utility as it throws away the representations' content.&#10;Thus, subject to a guardedness constraint, concept erasure techniques search for an affine transformation that minimally alters the existing representations \cite{belrose2024leace}. &#10;Just as with guarding functions, a good steering function also requires guardedness.&#10;In this paper, we give a novel derivation of optimal affine steering functions making use of guardedness. &#10;&#10;Our paper provides both theoretical and empirical results. &#10;Theoretically, we derive the optimal, in terms of least-squares error, affine steering function under a guardedness assumption, i.e., we find the steering function that changes the representation minimally in terms of $L_2$ but still provably steers the representations. This function turns out to be a linear translation of the representations, giving a theoretical justification to the usage of steering vectors \cite{subramani2022extracting, li2023inference}. We additionally derive a second optimal affine steering function by imposing a covariance constraint, i.e., we match the first and second moments of the concept-conditional representations.&#10;Applying the covariance constraint endows the resulting steering function with another guarantee: it provably removes bias by neighbors \cite{gonen2019lipstick} in expectation, i.e., it reduces the tendency of the representations to cluster by their associated gender.&#10;&#10;Empirically, we conduct three sets of experiments to explore how well our optimal affine steering functions work in practice. In the first two experiments, we apply the affine steering functions to target different types of bias in multiclass classification. In the first experiment, we focus on gender bias in profession classification (\ref{sec:gender}), and in the second experiment, we focus on dialect bias in sentiment classification (\ref{sec:controlled-experiment}). Finally, in the last experiment, we use our affine steering functions to reduce toxicity when generating text from a language model (\ref{sec:toxicity}), by intervening in the last hidden representation at each generation step. A schematic illustration of our third experiment is given in \ref{fig:first-page}. We find that in all cases, affine steering demonstrates empirical success.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Text Generation Control, Language Model Bias, Artificial Intelligence, Mathematics, Neural Network Steering" />
        </attvalues>
      </node>
      <node id="2005.13407" label="2005.13407">
        <attvalues>
          <attvalue for="0" value="CausaLM: Causal Model Explanation Through Counterfactual Language Models" />
          <attvalue for="1" value="  Understanding predictions made by deep neural networks is notoriously&#10;difficult, but also crucial to their dissemination. As all machine learning&#10;based methods, they are as good as their training data, and can also capture&#10;unwanted biases. While there are tools that can help understand whether such&#10;biases exist, they do not distinguish between correlation and causation, and&#10;might be ill-suited for text-based models and for reasoning about high level&#10;language concepts. A key problem of estimating the causal effect of a concept&#10;of interest on a given model is that this estimation requires the generation of&#10;counterfactual examples, which is challenging with existing generation&#10;technology. To bridge that gap, we propose CausaLM, a framework for producing&#10;causal model explanations using counterfactual language representation models.&#10;Our approach is based on fine-tuning of deep contextualized embedding models&#10;with auxiliary adversarial tasks derived from the causal graph of the problem.&#10;Concretely, we show that by carefully choosing auxiliary adversarial&#10;pre-training tasks, language representation models such as BERT can effectively&#10;learn a counterfactual representation for a given concept of interest, and be&#10;used to estimate its true causal effect on model performance. A byproduct of&#10;our method is a language representation model that is unaffected by the tested&#10;concept, which can be useful in mitigating unwanted bias ingrained in the data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Previous work on the intersectionof DNN interpretations and causal inference, specifically in relation to NLP is rare. While there is a vast and rich literature on each of those topics alone, the gap between interpretability, causality and NLP is only now starting to close~\cite{vig2020causal}. To ground our work in those pillars, we survey here previous work in each. Specifically, we discuss how to use causal inference in NLP~\cite{keith2020text}, and describe the current state of research on model interpretations and debiasing in NLP. Finally, we discuss our contribution in light of the relevant work.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2012.14913" label="2012.14913">
        <attvalues>
          <attvalue for="0" value="Transformer Feed-Forward Layers Are Key-Value Memories" />
          <attvalue for="1" value="  Feed-forward layers constitute two-thirds of a transformer model's&#10;parameters, yet their role in the network remains under-explored. We show that&#10;feed-forward layers in transformer-based language models operate as key-value&#10;memories, where each key correlates with textual patterns in the training&#10;examples, and each value induces a distribution over the output vocabulary. Our&#10;experiments show that the learned patterns are human-interpretable, and that&#10;lower layers tend to capture shallow patterns, while upper layers learn more&#10;semantic ones. The values complement the keys' input patterns by inducing&#10;output distributions that concentrate probability mass on tokens likely to&#10;appear immediately after each pattern, particularly in the upper layers.&#10;Finally, we demonstrate that the output of a feed-forward layer is a&#10;composition of its memories, which is subsequently refined throughout the&#10;model's layers via residual connections to produce the final output&#10;distribution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.10012" label="2210.10012">
        <attvalues>
          <attvalue for="0" value="Log-linear Guardedness and its Implications" />
          <attvalue for="1" value="  Methods for erasing human-interpretable concepts from neural representations&#10;that assume linearity have been found to be tractable and useful. However, the&#10;impact of this removal on the behavior of downstream classifiers trained on the&#10;modified representations is not fully understood. In this work, we formally&#10;define the notion of log-linear guardedness as the inability of an adversary to&#10;predict the concept directly from the representation, and study its&#10;implications. We show that, in the binary case, under certain assumptions, a&#10;downstream log-linear model cannot recover the erased concept. However, we&#10;demonstrate that a multiclass log-linear model \emph{can} be constructed that&#10;indirectly recovers the concept in some cases, pointing to the inherent&#10;limitations of log-linear guardedness as a downstream bias mitigation&#10;technique. These findings shed light on the theoretical limitations of linear&#10;erasure methods and highlight the need for further research on the connections&#10;between intrinsic and extrinsic bias in neural models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.03819" label="2306.03819">
        <attvalues>
          <attvalue for="0" value="LEACE: Perfect linear concept erasure in closed form" />
          <attvalue for="1" value="  Concept erasure aims to remove specified features from a representation. It&#10;can improve fairness (e.g. preventing a classifier from using gender or race)&#10;and interpretability (e.g. removing a concept to observe changes in model&#10;behavior). We introduce LEAst-squares Concept Erasure (LEACE), a closed-form&#10;method which provably prevents all linear classifiers from detecting a concept&#10;while changing the representation as little as possible, as measured by a broad&#10;class of norms. We apply LEACE to large language models with a novel procedure&#10;called &quot;concept scrubbing,&quot; which erases target concept information from every&#10;layer in the network. We demonstrate our method on two tasks: measuring the&#10;reliance of language models on part-of-speech information, and reducing gender&#10;bias in BERT embeddings. Code is available at&#10;https://github.com/EleutherAI/concept-erasure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.11388" label="2205.11388">
        <attvalues>
          <attvalue for="0" value="StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in&#10;  Question Answering Models" />
          <attvalue for="1" value="  Knowledge and language understanding of models evaluated through question&#10;answering (QA) has been usually studied on static snapshots of knowledge, like&#10;Wikipedia. However, our world is dynamic, evolves over time, and our models'&#10;knowledge becomes outdated. To study how semi-parametric QA models and their&#10;underlying parametric language models (LMs) adapt to evolving knowledge, we&#10;construct a new large-scale dataset, StreamingQA, with human written and&#10;generated questions asked on a given date, to be answered from 14 years of&#10;time-stamped news articles. We evaluate our models quarterly as they read new&#10;articles not seen in pre-training. We show that parametric models can be&#10;updated without full retraining, while avoiding catastrophic forgetting. For&#10;semi-parametric models, adding new articles into the search space allows for&#10;rapid adaptation, however, models with an outdated underlying LM under-perform&#10;those with a retrained LM. For questions about higher-frequency named entities,&#10;parametric updates are particularly beneficial. In our dynamic world, the&#10;StreamingQA dataset enables a more realistic evaluation of QA models, and our&#10;experiments highlight several promising directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.09036" label="2008.09036">
        <attvalues>
          <attvalue for="0" value="Language Models as Knowledge Bases: On Entity Representations, Storage&#10;  Capacity, and Paraphrased Queries" />
          <attvalue for="1" value="  Pretrained language models have been suggested as a possible alternative or&#10;complement to structured knowledge bases. However, this emerging LM-as-KB&#10;paradigm has so far only been considered in a very limited setting, which only&#10;allows handling 21k entities whose single-token name is found in common LM&#10;vocabularies. Furthermore, the main benefit of this paradigm, namely querying&#10;the KB using a variety of natural language paraphrases, is underexplored so&#10;far. Here, we formulate two basic requirements for treating LMs as KBs: (i) the&#10;ability to store a large number facts involving a large number of entities and&#10;(ii) the ability to query stored facts. We explore three entity representations&#10;that allow LMs to represent millions of entities and present a detailed case&#10;study on paraphrased querying of world knowledge in LMs, thereby providing a&#10;proof-of-concept that language models can indeed serve as knowledge bases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.03983" label="2308.03983">
        <attvalues>
          <attvalue for="0" value="SimplyRetrieve: A Private and Lightweight Retrieval-Centric Generative&#10;  AI Tool" />
          <attvalue for="1" value="  Large Language Model (LLM) based Generative AI systems have seen significant&#10;progress in recent years. Integrating a knowledge retrieval architecture allows&#10;for seamless integration of private data into publicly available Generative AI&#10;systems using pre-trained LLM without requiring additional model fine-tuning.&#10;Moreover, Retrieval-Centric Generation (RCG) approach, a promising future&#10;research direction that explicitly separates roles of LLMs and retrievers in&#10;context interpretation and knowledge memorization, potentially leads to more&#10;efficient implementation. SimplyRetrieve is an open-source tool with the goal&#10;of providing a localized, lightweight, and user-friendly interface to these&#10;sophisticated advancements to the machine learning community. SimplyRetrieve&#10;features a GUI and API based RCG platform, assisted by a Private Knowledge Base&#10;Constructor and a Retrieval Tuning Module. By leveraging these capabilities,&#10;users can explore the potential of RCG for improving generative AI performance&#10;while maintaining privacy standards. The tool is available at&#10;https://github.com/RCGAI/SimplyRetrieve with an MIT license.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11915" label="2010.11915">
        <attvalues>
          <attvalue for="0" value="Challenges in Information-Seeking QA: Unanswerable Questions and&#10;  Paragraph Retrieval" />
          <attvalue for="1" value="  Recent pretrained language models &quot;solved&quot; many reading comprehension&#10;benchmarks, where questions are written with access to the evidence document.&#10;However, datasets containing information-seeking queries where evidence&#10;documents are provided after the queries are written independently remain&#10;challenging. We analyze why answering information-seeking queries is more&#10;challenging and where their prevalent unanswerabilities arise, on Natural&#10;Questions and TyDi QA. Our controlled experiments suggest two headrooms --&#10;paragraph selection and answerability prediction, i.e. whether the paired&#10;evidence document contains the answer to the query or not. When provided with a&#10;gold paragraph and knowing when to abstain from answering, existing models&#10;easily outperform a human annotator. However, predicting answerability itself&#10;remains challenging. We manually annotate 800 unanswerable examples across six&#10;languages on what makes them challenging to answer. With this new data, we&#10;conduct per-category answerability prediction, revealing issues in the current&#10;dataset collection as well as task formulation. Together, our study points to&#10;avenues for future research in information-seeking question answering, both for&#10;dataset creation and model development.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Analyzing unanswerable questions.&#10;There is prior work that seeks to understand unanswerability in reading comprehension datasets. &#10;\cite{Yatskar2019AQC} analyzes unanswerable questions in SQuAD 2.0 and two conversational reading comprehension datasets, namely CoQA and QuAC, while we focus on information-seeking QA datasets to understand the potential dataset collection improvements and quantify the modeling challenges of the state-of-the-art QA models.&#10;&#10;\cite{ravichander-etal-2019-question} compare unanswerable factors between NQ and a QA dataset on privacy policies.&#10;This work primarily focuses on a privacy QA, which leads to differences of the categorizations of the unanswerable questions. &#10;We search alternative knowledge sources as well as the answers to understand how we could improve answer coverage from dataset creation perspective and connect the annotation results with answerability prediction experiments for modeling improvements.&#10;&#10;Answer Calibrations.&#10;Answerability prediction can bring practical values, when errors are expensive but abstaining from it is less so~\cite{kamath-etal-2020-selective}.&#10;While predicting answerability has been studied in SQuAD 2.0~\cite{zhang2020retrospective,hu2019read+}, the unanswerability in SQuAD 2.0 has different characteristics from unanswerability in information-seeking QA as we discussed above.&#10;To handle unanswerable questions in information-seeking QA, models either adopt threshold based answerable verification~\cite{bert}, or introduce an extra layer to classify unanswerablity and training the model jointly~\cite{zhang2019sg,yang2019xlnet}. &#10;\cite{kamath-etal-2020-selective} observes the difficulty of answer calibrations, especially under domain shift. &#10;&#10;Artifacts in datasets.&#10;Recent work~\cite{Gururangan2018AnnotationAI,kaushik-lipton-2018-much,sugawara-etal-2018-makes,Chen2019UnderstandingDD} exhibited that models can capture annotation bias in crowdsourced data effectively, achieving high performance when only provided with a partial input. &#10;Although NQ and TyDi QA attempt to avoid such typical artifacts of QA data by annotating questions independently from the existing documents~\cite{Clark2020TyDiQA}, we found artifacts in question surface forms can let models easily predict answerability with a partial input (i.e., question only).&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2207.00216" label="2207.00216">
        <attvalues>
          <attvalue for="0" value="Updating Only Encoders Prevents Catastrophic Forgetting of End-to-End&#10;  ASR Models" />
          <attvalue for="1" value="  In this paper, we present an incremental domain adaptation technique to&#10;prevent catastrophic forgetting for an end-to-end automatic speech recognition&#10;(ASR) model. Conventional approaches require extra parameters of the same size&#10;as the model for optimization, and it is difficult to apply these approaches to&#10;end-to-end ASR models because they have a huge amount of parameters. To solve&#10;this problem, we first investigate which parts of end-to-end ASR models&#10;contribute to high accuracy in the target domain while preventing catastrophic&#10;forgetting. We conduct experiments on incremental domain adaptation from the&#10;LibriSpeech dataset to the AMI meeting corpus with two popular end-to-end ASR&#10;models and found that adapting only the linear layers of their encoders can&#10;prevent catastrophic forgetting. Then, on the basis of this finding, we develop&#10;an element-wise parameter selection focused on specific layers to further&#10;reduce the number of fine-tuning parameters. Experimental results show that our&#10;approach consistently prevents catastrophic forgetting compared to parameter&#10;selection from the whole model.&#10;" />
          <attvalue for="2" value="&#10;End-to-end automatic speech recognition (ASR) has made remarkable progress in cases where a large amount of training data is available.&#10;However, in real applications, there is often an acoustic mismatch between the training environment and the operational environment where users utilize the ASR model, and it is difficult to collect a sufficient amount&#10;of training data in the operational environment beforehand.&#10;Therefore, we sometimes collect target domain data during operation and continuously adapt the model using the collected data.&#10;&#10;A typical domain adaptation method~\cite{NIPS2006_b1b0432c,mirsamadi17_interspeech} aims to improve the performance on a target domain and thus usually suffers from performance degradation on a source domain, which is known as catastrophic forgetting~\cite{journals/nn/ParisiKPKW19}.&#10;Catastrophic forgetting becomes more severe when the model is continuously adapted to the target domain using sequentially arriving data.&#10;To mitigate catastrophic forgetting, several incremental learning methods have been proposed~\cite{10.1109/TPAMI.2017.2773081,kirkpatrick2017overcoming,conf/iclr/YoonYLH18,conf/nips/Lopez-PazR17}.&#10;Regularization-based methods~\cite{10.1109/TPAMI.2017.2773081,kirkpatrick2017overcoming} introduce an additional loss during adaptation to make the current model close to the original one.&#10;Architecture-based methods~\cite{conf/iclr/YoonYLH18,rusu2016progressive} dynamically expand a model architecture for new data.&#10;In the field of ASR, various studies~\cite{conf/asru/GhorbaniKH19,conf/interspeech/SadhuH20,9746594} have demonstrated the benefits of the incremental learning approach.&#10;Fu~{et al.}~\cite{fu2021incremental} proposed an incremental learning algorithm for end-to-end ASR that uses attention distillation and knowledge distillation~\cite{hinton2015distilling} to prevent catastrophic forgetting.&#10;Such methods require retaining the previous model~\cite{conf/iclr/YoonYLH18,fu2021incremental} or adding extra parameters of the same size as the model for optimization~\cite{rusu2016progressive,conf/asru/GhorbaniKH19}.&#10;Therefore, adapting recent end-to-end ASR models with a huge amount of parameters is computationally expensive.&#10;A potential solution is pruning-based domain adaptation, which reduces the number of fine-tuning parameters by pruning~\cite{DBLP:conf/naacl/GuFX21,DBLP:conf/emnlp/ZhuWZLZ021}.&#10;In~\cite{DBLP:conf/naacl/GuFX21}, parameters of a subnetwork are fixed during adaptation to keep the performance of the source domain.&#10;The parameters to be fixed are determined on the basis of a pruning algorithm, since the parameters selected to be pruned are considered important for processing source domain data.&#10;However, the pruning algorithm is applied for the entire network, and there has been no investigation into the subnetwork-wise parameter freezing.&#10;Some studies on domain adaptation of ASR models have shown that updating only a part of the layers improves the performance on the target domain~\cite{conf/interspeech/UenoMMSSYAK18,sukhadia2022domain}.&#10;However, there has been no research on the performance against catastrophic forgetting.&#10;&#10;In this study, we investigate which parts of the end-to-end ASR model should and should not be adapted to the target domain to prevent catastrophic forgetting during incremental domain adaptation.&#10;For the experiments, we utilize two popular ASR models: a Transformer-based model and a recurrent neural network transducer (RNN-T).&#10;We use LibriSpeech~\cite{conf/icassp/PanayotovCPK15} as the large-scale source domain dataset and the AMI meeting corpus~\cite{conf/mlmi/CarlettaABFGHKKKKLLLMPRW05} as the target domain dataset.&#10;We first investigate a module-wise parameter selection to be fine-tuned and&#10;show that adapting only the encoder can prevent catastrophic forgetting.&#10;Then, to further reduce the number of fine-tuning parameters, we develop an element-wise parameter selection from specific layers in the model.&#10;Experimental results demonstrate that our proposed parameter selection could select better parameters to prevent catastrophic forgetting compared to parameter selection from the entire network.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Domain Adaptation, Neural Network Optimization, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1708.01547" label="1708.01547">
        <attvalues>
          <attvalue for="0" value="Lifelong Learning with Dynamically Expandable Networks" />
          <attvalue for="1" value="  We propose a novel deep network architecture for lifelong learning which we&#10;refer to as Dynamically Expandable Network (DEN), that can dynamically decide&#10;its network capacity as it trains on a sequence of tasks, to learn a compact&#10;overlapping knowledge sharing structure among tasks. DEN is efficiently trained&#10;in an online manner by performing selective retraining, dynamically expands&#10;network capacity upon arrival of each task with only the necessary number of&#10;units, and effectively prevents semantic drift by splitting/duplicating units&#10;and timestamping them. We validate DEN on multiple public datasets under&#10;lifelong learning scenarios, on which it not only significantly outperforms&#10;existing lifelong learning methods for deep networks, but also achieves the&#10;same level of performance as the batch counterparts with substantially fewer&#10;number of parameters. Further, the obtained network fine-tuned on all tasks&#10;obtained significantly better performance over the batch models, which shows&#10;that it can be used to estimate the optimal network structure even when all&#10;tasks are available in the first place.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.00565" label="1910.00565">
        <attvalues>
          <attvalue for="0" value="Domain Expansion in DNN-based Acoustic Models for Robust Speech&#10;  Recognition" />
          <attvalue for="1" value="  Training acoustic models with sequentially incoming data -- while both&#10;leveraging new data and avoiding the forgetting effect-- is an essential&#10;obstacle to achieving human intelligence level in speech recognition. An&#10;obvious approach to leverage data from a new domain (e.g., new accented speech)&#10;is to first generate a comprehensive dataset of all domains, by combining all&#10;available data, and then use this dataset to retrain the acoustic models.&#10;However, as the amount of training data grows, storing and retraining on such a&#10;large-scale dataset becomes practically impossible. To deal with this problem,&#10;in this study, we study several domain expansion techniques which exploit only&#10;the data of the new domain to build a stronger model for all domains. These&#10;techniques are aimed at learning the new domain with a minimal forgetting&#10;effect (i.e., they maintain original model performance). These techniques&#10;modify the adaptation procedure by imposing new constraints including (1)&#10;weight constraint adaptation (WCA): keeping the model parameters close to the&#10;original model parameters; (2) elastic weight consolidation (EWC): slowing down&#10;training for parameters that are important for previously established domains;&#10;(3) soft KL-divergence (SKLD): restricting the KL-divergence between the&#10;original and the adapted model output distributions; and (4) hybrid SKLD-EWC:&#10;incorporating both SKLD and EWC constraints. We evaluate these techniques in an&#10;accent adaptation task in which we adapt a deep neural network (DNN) acoustic&#10;model trained with native English to three different English accents:&#10;Australian, Hispanic, and Indian. The experimental results show that SKLD&#10;significantly outperforms EWC, and EWC works better than WCA. The hybrid&#10;SKLD-EWC technique results in the best overall performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Current state-of-the-art neural network-based ASR systems have advanced to nearly human performance in several evaluation settings~\cite{saon2017english, best2}; however, these systems perform poorly for domains that are not included in the original training data~\cite{myself1,sun2017unsupervised,hsu2017unsupervised,jafarlou2019LRF}. For example, if we train an ASR system using a U.S. English dataset, the performance of the system significantly degrades for other English accents (e.g., Australian, Indian, and Hispanic). In order to improve performance of the system for an unseen domain, we can adapt the previously trained model to capture the statistics of the new domain. However, adaptation techniques suffer from the forgetting effect: previously learned information will be lost by learning the new information. We need an ASR system that not only performs well for the new domain, but also retains performance for previously seen domains. This is the goal of domain expansion methods.&#10;&#10;Domain Expansion -- In a domain expansion scenario, we are given a model trained on an initial domain and a dataset for an unseen domain, the goal is to modify the model such that it performs well for both domains. The main difficulty of domain expansion is to preserve the functionality (input-output mapping) of the original model (mitigating the forgetting problem). Many approaches have been proposed to deal with the forgetting problem in neural networks. These approaches can be divided into three categories: architectural, rehearsal, and regularization strategies. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.04288" label="2005.04288">
        <attvalues>
          <attvalue for="0" value="Incremental Learning for End-to-End Automatic Speech Recognition" />
          <attvalue for="1" value="  In this paper, we propose an incremental learning method for end-to-end&#10;Automatic Speech Recognition (ASR) which enables an ASR system to perform well&#10;on new tasks while maintaining the performance on its originally learned ones.&#10;To mitigate catastrophic forgetting during incremental learning, we design a&#10;novel explainability-based knowledge distillation for ASR models, which is&#10;combined with a response-based knowledge distillation to maintain the original&#10;model's predictions and the &quot;reason&quot; for the predictions. Our method works&#10;without access to the training data of original tasks, which addresses the&#10;cases where the previous data is no longer available or joint training is&#10;costly. Results on a multi-stage sequential training task show that our method&#10;outperforms existing ones in mitigating forgetting. Furthermore, in two&#10;practical scenarios, compared to the target-reference joint training method,&#10;the performance drop of our method is 0.02% Character Error Rate (CER), which&#10;is 97% smaller than the drops of the baseline methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.13678" label="2103.13678">
        <attvalues>
          <attvalue for="0" value="Pruning-then-Expanding Model for Domain Adaptation of Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Domain Adaptation is widely used in practical applications of neural machine&#10;translation, which aims to achieve good performance on both the general-domain&#10;and in-domain. However, the existing methods for domain adaptation usually&#10;suffer from catastrophic forgetting, domain divergence, and model explosion. To&#10;address these three problems, we propose a method of &quot;divide and conquer&quot; which&#10;is based on the importance of neurons or parameters in the translation model.&#10;In our method, we first prune the model and only keep the important neurons or&#10;parameters, making them responsible for both general-domain and in-domain&#10;translation. Then we further train the pruned model supervised by the original&#10;unpruned model with the knowledge distillation method. Last we expand the model&#10;to the original size and fine-tune the added parameters for the in-domain&#10;translation. We conduct experiments on different languages and domains and the&#10;results show that our method can achieve significant improvements compared with&#10;several strong baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.09167" label="2202.09167">
        <attvalues>
          <attvalue for="0" value="Domain Adaptation of low-resource Target-Domain models using&#10;  well-trained ASR Conformer Models" />
          <attvalue for="1" value="  In this paper, we investigate domain adaptation for low-resource Automatic&#10;Speech Recognition (ASR) of target-domain data, when a well-trained ASR model&#10;trained with a large dataset is available. We argue that in the encoder-decoder&#10;framework, the decoder of the well-trained ASR model is largely tuned towards&#10;the source-domain, hurting the performance of target-domain models in vanilla&#10;transfer-learning. On the other hand, the encoder layers of the well-trained&#10;ASR model mostly capture the acoustic characteristics. We, therefore, propose&#10;to use the embeddings tapped from these encoder layers as features for a&#10;downstream Conformer target-domain model and show that they provide significant&#10;improvements. We do ablation studies on which encoder layer is optimal to tap&#10;the embeddings, as well as the effect of freezing or updating the well-trained&#10;ASR model's encoder layers. We further show that applying Spectral Augmentation&#10;(SpecAug) on the proposed features (this is in addition to default SpecAug on&#10;input spectral features) provides a further improvement on the target-domain&#10;performance. For the LibriSpeech-100-clean data as target-domain and SPGI-5000&#10;as a well-trained model, we get 30% relative improvement over baseline.&#10;Similarly, with WSJ data as target-domain and LibriSpeech-960 as a well-trained&#10;model, we get 50% relative improvement over baseline.&#10;" />
          <attvalue for="2" value="&#10;&#10;In most real-world problems, the amount of in-domain labeled data available for training Automatic Speech Recognition (ASR) models is usually very limited. Collecting a large amount of transcribed speech data is very expensive. Therefore, building in-domain ASR models with good accuracy is very difficult. One of the popular approaches to overcome this problem is to use transfer-learning, assuming a well-trained ASR model trained on a large dataset is available, possibly from a different domain. The transfer of knowledge can be in the form of feature representation, parameters, or weights \cite{pan2009survey}. &#10;&#10;Most state-of-the-art ASR models use encoder-decoder architecture. The decoder is optimized to learn the language model of the source-domain. In transfer-learning, when there is a mismatch in the domain, the performance of the transfer-learnt model would be affected. In \cite{meng2021internal,zeyer2021LibriSpeech,wang2020multitask}, the idea of internal language model estimation is used to improve performance when the well-trained ASR model is used to decode target-domain (i.e., in-domain) data directly without any transfer-learning step. While the decoder of the well-trained ASR model may be tuned toward the source-domain, the encoder layers may still model the acoustic aspects well, which can be transferred to the target-domain model. Therefore, in this paper, we investigate the use of encoder layer output of the well-trained ASR model as embeddings, which are fed to the randomly initialized encoder layers of the low-resource target-domain model instead of conventional Mel-filterbank features. Recently, pre-trained models such as Wav2Vec2.0 \cite{baevski2020wav2vec}, HuBERT \cite{hsu2021hubert} have become popular to address the low-resource speech modelling problem. However, they do suffer significant degradation when the target-domain is different from the data used in pre-training the model \cite{hsu2021robust,conneau2019unsupervised}. One approach is to use the target data along with the original pre-train data and start pre-training again. However, this is expensive. Loosely, the well-trained ASR models used in this paper are analogous to the pretrained self-supervised (SSL) models, except they have been trained in a supervised framework. The focus of this paper is, therefore, on domain adaptation to improve the performance of target-domain ASR models using a well-trained ASR model from a different domain. &#10;&#10;Our approach is along a line of thinking that bears some resemblance to the works in \cite{ghahremani2017investigation,yosinski2014transferable}.&#10;Through a series of experiments, we investigate the effect of following different methods on the final performance:&#10;\begin{itemize}&#10; \item The effect of different layer embeddings of the well-trained ASR model on the final performance of the target-domain model (i.e., in-domain model).&#10; \item The effect of freezing the well-trained ASR model's encoder layers versus allowing all the layers to update while training the low-resource target-domain model.&#10; \item The effect of applying a Spectral Augmentation (SpecAug) \cite{park2019specaugment} step on the well-trained ASR encoder features to improve the performance of the low-resource model. Note that well-trained ASR models have been trained after applying SpecAug on the Mel-filterbank features. The SpecAug is now applied when the ASR encoder embeddings are fed as features to the target-domain model.&#10;\end{itemize}&#10;&#10;The paper is organized as follows. Section \ref{sec:2} presents the details on the datasets and the architecture of source and target-domain models and establishes the baselines. Section \ref{sec:3} discusses the proposed method and shows the comparison with SSL features. Finally, the paper is concluded in Section \ref{sec:4}. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.09442" label="1903.09442">
        <attvalues>
          <attvalue for="0" value="LINSPECTOR: Multilingual Probing Tasks for Word Representations" />
          <attvalue for="1" value="  Despite an ever growing number of word representation models introduced for a&#10;large number of languages, there is a lack of a standardized technique to&#10;provide insights into what is captured by these models. Such insights would&#10;help the community to get an estimate of the downstream task performance, as&#10;well as to design more informed neural architectures, while avoiding extensive&#10;experimentation which requires substantial computational resources not all&#10;researchers have access to. A recent development in NLP is to use simple&#10;classification tasks, also called probing tasks, that test for a single&#10;linguistic feature such as part-of-speech. Existing studies mostly focus on&#10;exploring the linguistic information encoded by the continuous representations&#10;of English text. However, from a typological perspective the morphologically&#10;poor English is rather an outlier: the information encoded by the word order&#10;and function words in English is often stored on a morphological level in other&#10;languages. To address this, we introduce 15 type-level probing tasks such as&#10;case marking, possession, word length, morphological tag count and pseudoword&#10;identification for 24 languages. We present a reusable methodology for creation&#10;and evaluation of such tests in a multilingual setting. We then present&#10;experiments on several diverse multilingual word embedding models, in which we&#10;relate the probing task performance for a diverse set of languages to a range&#10;of five classic NLP tasks: POS-tagging, dependency parsing, semantic role&#10;labeling, named entity recognition and natural language inference. We find that&#10;a number of probing tests have significantly high positive correlation to the&#10;downstream tasks, especially for morphologically rich languages. We show that&#10;our tests can be used to explore word embeddings or black-box neural models for&#10;linguistic cues in a multilingual setting.&#10;" />
          <attvalue for="2" value="&#10;&#10;The field of natural language processing (NLP) has seen great development after replacing the traditional discrete word representations with continuous ones. Representing text with dense, low-dimensional vectors - or embeddings - has become the de facto approach, since these representations can encode complex relationships between the units of language and can be learned from unlabeled data, thus eliminating the need for expensive manual feature engineering. The initial success of dense representations in NLP applications has led to the development of a multitude of embedding models, which differ in terms of design objective (monolingual~\cite{Mikolov:2013:DRW:2999792.2999959}, cross-lingual~\cite{ruder2017survey}, contextualized~\cite{peters:NAACL2018}, retrofitted~\cite{faruqui2015retrofitting}, multi-sense~\cite{PilehvarCNC17}, cross-domain~\cite{YangLZ17}, dependency-based~\cite{LevyG14}), encoding architecture, (convolution~\cite{kim2015}, linear vector operations~\cite{bojanowski:TACL2017}, bidirectional LSTM~\cite{ling2015}), as well as in terms of the target units (words, characters, character n-grams, morphemes, phonemes). &#10;&#10;While offering substantial benefits over the traditional feature-based representations of language, the performance of unsupervised embeddings may differ considerably depending on the language and the task. For instance, early embedding models such as word2vec~\cite{Mikolov:2013:DRW:2999792.2999959} and GloVe~\cite{glove:14} have been shown to suffer from out-of-vocabulary (OOV) issues for agglutinative languages like Turkish and Finnish~\cite{sahin:acl18}, while performing relatively well on analytic and fusional languages like English. Furthermore, there is no guarantee that a representation well-suited for some task would score similarly well at other tasks even for the same language due to the differences in the information required to solve the tasks, as demonstrated in~\cite{rogers2018s}. &#10;&#10;Given the variety of word representations and parameter options, searching for the right word \rev{embedding/encoder}{representation} model for a specific language and a certain task is not trivial. Scanning the large parameter space may be extremely time consuming and computationally expensive, which poses significant challenges, especially in the lower-resource non-English academic NLP communities. To simplify the search for a good representation, and estimate the ``quality'' of the representations, intrinsic evaluation via similarity and analogy tasks has been proposed. While these tasks seem to be intuitive, there are concerns regarding their consistency and correlation with downstream task performance \cite{linzen2016issues,SchnabelLMJ15}. Furthermore, such evaluation requires manually created test sets and these are usually only available for a small number of languages. Another \rev{popular technique} option to assess the quality of word representation is through extrinsic evaluation, where the word vectors are used directly in downstream tasks, such as machine translation (MT)~\cite{ataman2018}, semantic role labeling (SRL)~\cite{sahin:acl18} or language modeling (LM)~\cite{vania2017}. Although this method provides more insightful information about the end task performance, it requires expensive human annotations, computational resources and the results are sensitive to hyperparameter choice.&#10;&#10;To address the aforementioned problems, a few studies have introduced the idea of probing tasks \cite{kohn2016evaluating,shi-etAl:ACL2016,adi2017fine,Veldhoen2016DiagnosticCR,senteval18}; which are a set of \rev{multi-label}{multi-class} classification problems that probe a learned word vector for a specific linguistic property, such as part-of-speech (POS), semantic, or morphological tag. Probing tasks have gained a lot of attention~(\cite{belinkov:acl2017,bisazza:emnlp2018}, among others) due to their simplicity, low computational cost, and ability to provide some insights \rev{into}{regarding} the linguistic properties \rev{of the word}{that are captured by the learned} representations. &#10;&#10;The majority of the probing tests proposed so far are mostly designed for English language only, and operate on the sentence-level (e.g. tree depth, word count, top constituent by \cite{senteval18}). Although sentence-level probing may provide valuable insights for English sentence-level representations, \rev{we hypothesize that they would not be similarly beneficial in a multilingual setup for several reasons.} The first reason is that the information encoded by the word order and function words in English is encoded at the morphological, subword level information in many other languages. Consider the Turkish word katılamayanlardan, that means ``he/she is one of the folks who can not participate''. In morphologically complex languages like Turkish, single tokens might already communicate a lot of information such as event, its participants, tense, person, number, polarity. In analytic languages, this information would be encoded as a multi-word clause. The second reason is the confusion of the signals: as pointed out by \cite{tenney2018what}, sometimes ``operating on full sentence encodings introduces confounds into the analysis, since sentence representation models must pool word representations over the entire sequence''. \rev{Furthermore, we argue that such tests would carry over the statistics of the data they originate from, introducing undesired biases such as domain and majority bias. In order to address the aforementioned issues, we introduce context independent, dictionary-based type-level probing tasks that operate on word-level and do not contain domain or majority biases. To investigate the limitations and strengths of the proposed type-level tasks, we introduce and investigate another set of similar, but context dependent, treebank-based and thereby potentially biased token-level tests.}&#10;&#10;In this work,&#10;\begin{itemize}&#10;\item We extend the line of work by~\cite{senteval18} and \cite{tenney2018what} and introduce 15 type-level probing tasks for 24 languages by taking language properties into account. Our probing tasks cover a range of features: from superficial ones such as word length, to morphosyntactic features such as case marker, gender, and number; and psycholinguistic ones like pseudowords (artificial words that are phonologically well-formed but have no meaning). Although languages share a large set of common probing tasks, each has a list of its own, e.g., Russian and Spanish are probed for gender, while Turkish is probed for polarity and possession; &#10;\item We introduce a reusable, systematic methodology for creation and evaluation of such tests by utilizing the existing resources such as UniMorph~\cite{sylakGlassmanK15,sylak2016composition,kirov-etal-2018-unimorph}, Wikipedia and Wuggy~\cite{keuleers2010wuggy};&#10;\item We then use the proposed probing tasks to evaluate \rev{the most commonly used multilingual word embedding models}{a set of diverse multilingual embedding models} and to diagnose a neural end-to-end semantic role labeling model as a case study. We statistically assess the correlation between probing and downstream task performance for a variety of downstream tasks (POS tagging, dependency parsing (DEP), semantic role labeling (SRL), named entity recognition (NER) and natural language inference (NLI)) for a set of typologically diverse languages and find that a number of probing tests have significantly high positive correlation to a number of syntactic and semantic downstream tasks, especially for morphologically rich languages;&#10;&#10;\item \rev{We introduce a set of comparable token-level probing tasks that additionally employs the context of the token. We analyze the type- and token-level probing tasks through a series of intrinsic and diagnostic experiments and show that they are similar with some exceptions: token-level tasks may be influenced by domain and majority class biases, while type-level tasks may suffer in case of lack of lexical diversity and high ambiguity ratios;}&#10;&#10;\item \rev{We provide comprehensive discussions for the intrinsic and extrinsic experimental results along with diagnostic and correlation study. We show that numerous factors except from the neural architectures play role on the results such as out-of-vocabulary rates, domain similarity, statistics of both datasets (e.g., ambiguity, size), training corpora for the embeddings; as well as typology, language family, paradigm size and morphological irregularity.}&#10;&#10;\item We release the LINSPECTOR framework that contains the probing datasets along with the intrinsic and extrinsic evaluation suite:.&#10;\end{itemize}&#10;We believe our evaluation suite together with probing datasets could be of great use for comparing various multilingual word representations such as automatically created cross-lingual embeddings; exploring the linguistic features captured by word encoding layers of black-box neural models; systematic searching of model or architecture parameters \rev{by evaluating the models with different architectures and parameters on the proposed probing tasks}; or comparing transfer learning techniques \rev{by probing for the transferred linguistic knowledge from high-resource to low-resource languages}{i.e. by evaluating a set of cross-lingual embeddings that are transfered or learned using different transfer learning techniques} \rev{, on the proposed language-specific probing task set}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Multilingual Word Embeddings, Computer Science, Linguistics, Probing Linguistic Features, Neural Architecture Evaluation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1902.00184" label="1902.00184">
        <attvalues>
          <attvalue for="0" value="A Simple Regularization-based Algorithm for Learning Cross-Domain Word&#10;  Embeddings" />
          <attvalue for="1" value="  Learning word embeddings has received a significant amount of attention&#10;recently. Often, word embeddings are learned in an unsupervised manner from a&#10;large collection of text. The genre of the text typically plays an important&#10;role in the effectiveness of the resulting embeddings. How to effectively train&#10;word embedding models using data from different domains remains a problem that&#10;is underexplored. In this paper, we present a simple yet effective method for&#10;learning word embeddings based on text from different domains. We demonstrate&#10;the effectiveness of our approach through extensive experiments on various&#10;down-stream NLP tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1508.06615" label="1508.06615">
        <attvalues>
          <attvalue for="0" value="Character-Aware Neural Language Models" />
          <attvalue for="1" value="  We describe a simple neural language model that relies only on&#10;character-level inputs. Predictions are still made at the word-level. Our model&#10;employs a convolutional neural network (CNN) and a highway network over&#10;characters, whose output is given to a long short-term memory (LSTM) recurrent&#10;neural network language model (RNN-LM). On the English Penn Treebank the model&#10;is on par with the existing state-of-the-art despite having 60% fewer&#10;parameters. On languages with rich morphology (Arabic, Czech, French, German,&#10;Spanish, Russian), the model outperforms word-level/morpheme-level LSTM&#10;baselines, again with fewer parameters. The results suggest that on many&#10;languages, character inputs are sufficient for language modeling. Analysis of&#10;word representations obtained from the character composition part of the model&#10;reveals that the model is able to encode, from characters only, both semantic&#10;and orthographic information.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.11937" label="1805.11937">
        <attvalues>
          <attvalue for="0" value="Character-Level Models versus Morphology in Semantic Role Labeling" />
          <attvalue for="1" value="  Character-level models have become a popular approach specially for their&#10;accessibility and ability to handle unseen data. However, little is known on&#10;their ability to reveal the underlying morphological structure of a word, which&#10;is a crucial skill for high-level semantic analysis tasks, such as semantic&#10;role labeling (SRL). In this work, we train various types of SRL models that&#10;use word, character and morphology level information and analyze how&#10;performance of characters compare to words and morphology for several&#10;languages. We conduct an in-depth error analysis for each morphological&#10;typology and analyze the strengths and limitations of character-level models&#10;that relate to out-of-domain data, training data size, long range dependencies&#10;and model complexity. Our exhaustive analyses shed light on important&#10;characteristics of character-level models and their semantic capability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.02036" label="1805.02036">
        <attvalues>
          <attvalue for="0" value="Compositional Representation of Morphologically-Rich Input for Neural&#10;  Machine Translation" />
          <attvalue for="1" value="  Neural machine translation (NMT) models are typically trained with fixed-size&#10;input and output vocabularies, which creates an important bottleneck on their&#10;accuracy and generalization capability. As a solution, various studies proposed&#10;segmenting words into sub-word units and performing translation at the&#10;sub-lexical level. However, statistical word segmentation methods have recently&#10;shown to be prone to morphological errors, which can lead to inaccurate&#10;translations. In this paper, we propose to overcome this problem by replacing&#10;the source-language embedding layer of NMT with a bi-directional recurrent&#10;neural network that generates compositional representations of the input at any&#10;desired level of granularity. We test our approach in a low-resource setting&#10;with five languages from different morphological typologies, and under&#10;different composition assumptions. By training NMT to compose word&#10;representations from character n-grams, our approach consistently outperforms&#10;(from 1.71 to 2.48 BLEU points) NMT learning embeddings of statistically&#10;generated sub-word units.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.11101" label="1810.11101">
        <attvalues>
          <attvalue for="0" value="UniMorph 2.0: Universal Morphology" />
          <attvalue for="1" value="  The Universal Morphology UniMorph project is a collaborative effort to&#10;improve how NLP handles complex morphology across the world's languages. The&#10;project releases annotated morphological data using a universal tagset, the&#10;UniMorph schema. Each inflected form is associated with a lemma, which&#10;typically carries its underlying lexical meaning, and a bundle of morphological&#10;features from our schema. Additional supporting data and tools are also&#10;released on a per-language basis when available. UniMorph is based at the&#10;Center for Language and Speech Processing (CLSP) at Johns Hopkins University in&#10;Baltimore, Maryland and is sponsored by the DARPA LORELEI program. This paper&#10;details advances made to the collection, annotation, and dissemination of&#10;project resources since the initial UniMorph release described at LREC 2016.&#10;lexical resources} }&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.09392" label="2012.09392">
        <attvalues>
          <attvalue for="0" value="MASKER: Masked Keyword Regularization for Reliable Text Classification" />
          <attvalue for="1" value="  Pre-trained language models have achieved state-of-the-art accuracies on&#10;various text classification tasks, e.g., sentiment analysis, natural language&#10;inference, and semantic textual similarity. However, the reliability of the&#10;fine-tuned text classifiers is an often underlooked performance criterion. For&#10;instance, one may desire a model that can detect out-of-distribution (OOD)&#10;samples (drawn far from training distribution) or be robust against domain&#10;shifts. We claim that one central obstacle to the reliability is the&#10;over-reliance of the model on a limited number of keywords, instead of looking&#10;at the whole context. In particular, we find that (a) OOD samples often contain&#10;in-distribution keywords, while (b) cross-domain samples may not always contain&#10;keywords; over-relying on the keywords can be problematic for both cases. In&#10;light of this observation, we propose a simple yet effective fine-tuning&#10;method, coined masked keyword regularization (MASKER), that facilitates&#10;context-based prediction. MASKER regularizes the model to reconstruct the&#10;keywords from the rest of the words and make low-confidence predictions without&#10;enough context. When applied to various pre-trained language models (e.g.,&#10;BERT, RoBERTa, and ALBERT), we demonstrate that MASKER improves OOD detection&#10;and cross-domain generalization without degrading classification accuracy. Code&#10;is available at https://github.com/alinlab/MASKER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Distribution shift in NLP.&#10;The reliable text classifier should detect distribution shift, i.e., test distribution is different from the training distribution. However, the most common scenarios: OOD detection and cross-domain generalization are relatively under-explored in NLP domains \cite{hendrycks2020pretrained, MarasovicGradient2018NLP}. \cite{hendrycks2020pretrained} found that pre-trained models are robust to the distribution shift compared to traditional NLP models. We find that the pre-trained models are not robust enough, and we empirically show that pre-trained models are still relying on undesirable dataset bias. Our method further improves the generalization performance, applied to the pre-trained models.&#10;&#10;Shortcut bias.&#10;One may interpret the over-reliance problem as a type of shortcut bias \cite{geirhos2020shortcut}, i.e., the model learns an easy-to-learn but not generalizable solution, as the keywords can be considered as a shortcut. The shortcut bias is investigated under various NLP tasks \cite{sun2019fine}, e.g., natural language inference \cite{mccoy2019right}, reasoning comprehension \cite{niven2019probing}, and question answering \cite{min2019compositional}. To our best knowledge, we are the first to point out that the over-reliance on keywords can also be a shortcut, especially for text classification. We remark that the shortcut bias is not always harmful as it can be a useful feature for in-distribution accuracy. However, we claim that they can be problematic for unexpected (i.e., OOD) samples, as demonstrated in our experiments.&#10;&#10;Debiasing methods.&#10;Numerous debiasing techniques have been proposed to regularize shortcuts, e.g., careful data collection \cite{choi2018quac,reddy2019coqa}, bias-tailored architecture \cite{agrawal2018don}, and adversarial regularization \cite{clark2019don,minderer2020automatic,nam2020learning}. However, most prior work requires supervision of biases, i.e., the shortcuts are explicitly given. In contrast, our method can be viewed as an unsupervised debiasing method, as our keyword selection schemes automatically select the keywords.&#10;&#10;" />
          <attvalue for="4" value="Text Classification Accuracy, Language Model Reliability, Computer Science, Linguistics, Artificial Intelligence, Mathematics, Context-Based Prediction" />
        </attvalues>
      </node>
      <node id="2112.05346" label="2112.05346">
        <attvalues>
          <attvalue for="0" value="Findings on Conversation Disentanglement" />
          <attvalue for="1" value="  Conversation disentanglement, the task to identify separate threads in&#10;conversations, is an important pre-processing step in multi-party&#10;conversational NLP applications such as conversational question answering and&#10;conversation summarization. Framing it as a utterance-to-utterance&#10;classification problem -- i.e. given an utterance of interest (UOI), find which&#10;past utterance it replies to -- we explore a number of transformer-based models&#10;and found that BERT in combination with handcrafted features remains a strong&#10;baseline. We then build a multi-task learning model that jointly learns&#10;utterance-to-utterance and utterance-to-thread classification. Observing that&#10;the ground truth label (past utterance) is in the top candidates when our model&#10;makes an error, we experiment with using bipartite graphs as a post-processing&#10;step to learn how to best match a set of UOIs to past utterances. Experiments&#10;on the Ubuntu IRC dataset show that this approach has the potential to&#10;outperform the conventional greedy approach of simply selecting the highest&#10;probability candidate for each UOI independently, indicating a promising future&#10;research direction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Conversation disentanglement methods can be classified into two &#10;categories: (1) two-step methods and (2) end-to-end methods. &#10;&#10;In two-step methods, the first step is to measure the relations between &#10;utterance pairs, e.g., reply-to relations \cite{zhu2020did, &#10;kummerfeldetal2019large} or same thread relations &#10;\cite{elsner-charniak-2008-talking0, &#10;elsner-charniak-2010-disentangling}. Either feature-based models &#10;\cite{elsner-charniak-2008-talking0, elsner-charniak-2010-disentangling} &#10;or deep learning models \cite{kummerfeldetal2019large, zhu2020did} are &#10;used. Afterwards a clustering algorithm is applied to recover separate &#10;threads using results from the first step. &#10;\newcite{elsner-charniak-2008-talking0, &#10;elsner-charniak-2010-disentangling, elsner-charniak-2011-disentangling} &#10;use a greedy graph partition algorithm to assign an utterance $u$ to the &#10;thread of $u'$ which has the maximum relevance to $u$ among candidates &#10;if the score is larger than a threshold. &#10;\newcite{kummerfeldetal2019large, zhu2020did} use a greedy algorithm to &#10;recover threads following all reply-to relations independently &#10;identified for each utterance. \newcite{jiang2018learning} propose a &#10;graph connected component-based algorithm.&#10;&#10;End-to-end methods construct threads incrementally by scanning through a &#10;chat log and either append the current utterance to an existing thread &#10;or create a new thread. \newcite{tan2019context} use a hierarchical LSTM &#10;model to obtain utterance representation and thread representation. &#10;\newcite{liu2020end} build a transition-based model that uses three LSTMs &#10;for utterance encoding, context encoding and thread state updating.&#10;&#10;" />
          <attvalue for="4" value="Graph-Based Modeling, Computer Science, Conversational NLP, Linguistics, Multi-Task Learning, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1910.11455" label="1910.11455">
        <attvalues>
          <attvalue for="0" value="Recognizing long-form speech using streaming end-to-end models" />
          <attvalue for="1" value="  All-neural end-to-end (E2E) automatic speech recognition (ASR) systems that&#10;use a single neural network to transduce audio to word sequences have been&#10;shown to achieve state-of-the-art results on several tasks. In this work, we&#10;examine the ability of E2E models to generalize to unseen domains, where we&#10;find that models trained on short utterances fail to generalize to long-form&#10;speech. We propose two complementary solutions to address this: training on&#10;diverse acoustic data, and LSTM state manipulation to simulate long-form audio&#10;when training using short utterances. On a synthesized long-form test set,&#10;adding data diversity improves word error rate (WER) by 90% relative, while&#10;simulating long-form training improves it by 67% relative, though the&#10;combination doesn't improve over data diversity alone. On a real long-form&#10;call-center test set, adding data diversity improves WER by 40% relative.&#10;Simulating long-form training on top of data diversity improves performance by&#10;an additional 27% relative.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.12226" label="2108.12226">
        <attvalues>
          <attvalue for="0" value="Injecting Text in Self-Supervised Speech Pretraining" />
          <attvalue for="1" value="  Self-supervised pretraining for Automated Speech Recognition (ASR) has shown&#10;varied degrees of success. In this paper, we propose to jointly learn&#10;representations during pretraining from two different modalities: speech and&#10;text. The proposed method, tts4pretrain complements the power of contrastive&#10;learning in self-supervision with linguistic/lexical representations derived&#10;from synthesized speech, effectively learning from untranscribed speech and&#10;unspoken text. Lexical learning in the speech encoder is enforced through an&#10;additional sequence loss term that is coupled with contrastive loss during&#10;pretraining. We demonstrate that this novel pretraining method yields Word&#10;Error Rate (WER) reductions of 10% relative on the well-benchmarked,&#10;Librispeech task over a state-of-the-art baseline pretrained with wav2vec2.0&#10;only. The proposed method also serves as an effective strategy to compensate&#10;for the lack of transcribed speech, effectively matching the performance of&#10;5000 hours of transcribed speech with just 100 hours of transcribed speech on&#10;the AMI meeting transcription task. Finally, we demonstrate WER reductions of&#10;up to 15% on an in-house Voice Search task over traditional pretraining.&#10;Incorporating text into encoder pretraining is complimentary to rescoring with&#10;a larger or in-domain language model, resulting in additional 6% relative&#10;reduction in WER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.07685" label="2001.07685">
        <attvalues>
          <attvalue for="0" value="FixMatch: Simplifying Semi-Supervised Learning with Consistency and&#10;  Confidence" />
          <attvalue for="1" value="  Semi-supervised learning (SSL) provides an effective means of leveraging&#10;unlabeled data to improve a model's performance. In this paper, we demonstrate&#10;the power of a simple combination of two common SSL methods: consistency&#10;regularization and pseudo-labeling. Our algorithm, FixMatch, first generates&#10;pseudo-labels using the model's predictions on weakly-augmented unlabeled&#10;images. For a given image, the pseudo-label is only retained if the model&#10;produces a high-confidence prediction. The model is then trained to predict the&#10;pseudo-label when fed a strongly-augmented version of the same image. Despite&#10;its simplicity, we show that FixMatch achieves state-of-the-art performance&#10;across a variety of standard semi-supervised learning benchmarks, including&#10;94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just&#10;4 labels per class. Since FixMatch bears many similarities to existing SSL&#10;methods that achieve worse performance, we carry out an extensive ablation&#10;study to tease apart the experimental factors that are most important to&#10;FixMatch's success. We make our code available at&#10;https://github.com/google-research/fixmatch.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12096" label="2010.12096">
        <attvalues>
          <attvalue for="0" value="Improving Streaming Automatic Speech Recognition With Non-Streaming&#10;  Model Distillation On Unsupervised Data" />
          <attvalue for="1" value="  Streaming end-to-end automatic speech recognition (ASR) models are widely&#10;used on smart speakers and on-device applications. Since these models are&#10;expected to transcribe speech with minimal latency, they are constrained to be&#10;causal with no future context, compared to their non-streaming counterparts.&#10;Consequently, streaming models usually perform worse than non-streaming models.&#10;We propose a novel and effective learning method by leveraging a non-streaming&#10;ASR model as a teacher to generate transcripts on an arbitrarily large data&#10;set, which is then used to distill knowledge into streaming ASR models. This&#10;way, we scale the training of streaming models to up to 3 million hours of&#10;YouTube audio. Experiments show that our approach can significantly reduce the&#10;word error rate (WER) of RNNT models not only on LibriSpeech but also on&#10;YouTube data in four languages. For example, in French, we are able to reduce&#10;the WER by 16.4% relatively to a baseline streaming model by leveraging a&#10;non-streaming teacher model trained on the same amount of labeled data as the&#10;baseline.&#10;" />
          <attvalue for="2" value="&#10;&#10;The advent of smart speakers such as Google Assistant, Siri, and Alexa has motivated a new generation of on-device recognition systems. End-to-end streaming models~\cite{graves2012sequence, he2019streaming, yeh2019transformer, zhang2020transformer, li2020towards, moritz2020streaming, tsunoo2019towards} have become attractive for on-device recognition tasks in two aspects: first, end-to-end models are usually compact, which makes them suitable to be used on devices. Second, such models often have a low latency (i.e. streaming), which is crucial to facilitate human-computer interactions -- an automated assistant can only engage the user when it responds quickly to requests. &#10;&#10;Contrary to non-streaming ASR models such as Chorowski et al's attention-based models~\cite{chorowski2015attention} or Chan et al's listen-attend-spell models \cite{chan2016listen}, streaming ASR models cannot utilize the full context. In the past few years, many research efforts have been devoted to improving streaming ASR~\cite{sainath2019two,sainath2020streaming,Saon-distillation-2020}. &#10;However, a key question that remains is how to utilize unlabeled data, especially for non-English languages with much less training data. &#10;&#10;In~\cite{liao2013large}, Liao et al showed that we could generate large-scale training data from the public YouTube videos, leveraging transcripts uploaded by the video owners. Their method ~\cite{liao2013large} is called ``Island of Confidence&quot; because it identifies segments of audio that have correct transcripts with high confidence. In this paper, we name the data generated by \cite{liao2013large} as Confisland for short.&#10;Because of the continuously increasing amount of YouTube data with user-uploaded transcripts, such Confisland data is a good resource to train end-to-end ASR models. &#10;&#10;In this paper, we propose a new approach to train end-to-end streaming models from unsupervised data. Our approach can be divided into three steps: (1) We employ the state-of-the-art full-context model as a teacher model. (2) We convert unlabeled audio sequences into random segments and transcribe them using the full-context teacher model. (3) We use the waveforms and their predicted transcripts in a noisy student learning framework~\cite{park2020improved,xie2020self,he2019revisiting,li2017learning}. &#10;Our method can potentially collect much more data than Confisland \cite{liao2013large} as it can also use audio data without user-uploaded transcripts. Besides, since full-context ASR models perform significantly better than streaming models, they work as stronger teachers and finally foster more robust streaming student models. \cite{Saon-distillation-2020} also proposed to use a full-context model as the teacher for RNN-T models, but their approach uses posteriors as targets and therefore requires additional distillation pretraining to address the alignment mismatch between the student and the teacher. Our approach is more efficient as we use the predicted transcripts as targets directly which does not have the alignment issue and requires only one distillation step.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.10847" label="2109.10847">
        <attvalues>
          <attvalue for="0" value="Small-Bench NLP: Benchmark for small single GPU trained models in&#10;  Natural Language Processing" />
          <attvalue for="1" value="  Recent progress in the Natural Language Processing domain has given us&#10;several State-of-the-Art (SOTA) pretrained models which can be finetuned for&#10;specific tasks. These large models with billions of parameters trained on&#10;numerous GPUs/TPUs over weeks are leading in the benchmark leaderboards. In&#10;this paper, we discuss the need for a benchmark for cost and time effective&#10;smaller models trained on a single GPU. This will enable researchers with&#10;resource constraints experiment with novel and innovative ideas on&#10;tokenization, pretraining tasks, architecture, fine tuning methods etc. We set&#10;up Small-Bench NLP, a benchmark for small efficient neural language models&#10;trained on a single GPU. Small-Bench NLP benchmark comprises of eight NLP tasks&#10;on the publicly available GLUE datasets and a leaderboard to track the progress&#10;of the community. Our ELECTRA-DeBERTa (15M parameters) small model architecture&#10;achieves an average score of 81.53 which is comparable to that of BERT-Base's&#10;82.20 (110M parameters). Our models, code and leaderboard are available at&#10;https://github.com/smallbenchnlp&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent research in machine learning and deep learning have delivered state-of-the-art results in various tasks across different domains. AlphaFold &#10; from Deepmind – recognised as a solution to the 50-year old protein folding problem – has made a significant scientific breakthrough. This state-of-the-art attention based neural network model was trained on 16 TPUv3s (which is 128 TPUv3 cores or roughly equivalent to ~100-200 GPUs) for a few weeks. Exponential growth in the computing capabilities, availability of large datasets together with efficient and optimised algorithms have resulted in such highly accurate state-of-the-art models across different domains. &#10;&#10;Similarly, language modelling task at the core of the natural language understanding has leveraged unsupervised learning techniques together with availability of huge corpus of data to build contextualized dense vector representations for the words. Recent large-scale transformer-based neural language models like GPT-2 \cite{radford2019language, brown2020language}, BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta}, XLNet \cite{yang2020xlnet}, UniLM \cite{dong2019unified}, ELECTRA \cite{clark2020electra}, T5 \cite{2020t5} have released pre-trained language models. The learned features from these pre-trained neural language models are then used for several supervised downstream tasks using the transfer learning approach. &#10;&#10;These state-of-the-art models have millions/billions of parameters and are time and compute-intensive. BERT-Large has 340M parameters, GPT-2 XLarge \cite{radford2019language} has 1.5B parameters, T5-11B \cite{2020t5} has 11B parameters and GPT-3 \cite{brown2020language} has 175B parameters. In the recent past, we can see that the parameters of such language models have gone from hundreds of millions to billions in number. Training such large models cost huge sums of money. This trend with institutions competing on the leaderboards with ever-increasing compute-intensive large billion parameter models has significantly increased the financial costs and the carbon footprint. In addition to the financial and environmental costs, these large models as mentioned in \cite{bender2021dangers} also increase opportunity cost as researchers will be steered away from directions requiring lower resources. &#10;&#10;In order to foster research and to encourage experimenting with innovative and novel ideas in tokenization, neural network architecture, pre-training tasks and finetuning methods for language models, we believe there has to be a benchmark for smaller neural language models with few million parameters that can be trained on a single GPU. This benchmark will enable researchers and practitioners from diverse backgrounds to experiment and validate their models built using relatively smaller computational resources. It will also provide an opportunity to make Natural Language Processing (NLP) research more inclusive and accessible to a large community. &#10;&#10;In this paper,&#10;\begin{enumerate}&#10; \item create Small-Bench NLP, a benchmark for evaluating the performance of smaller neural language models trained on a single GPU.&#10; \item we present our results of single GPU (V100-16GB of RAM) trained models with a smaller architecture on eight different annotated datasets part of the GLUE \cite{wang2019glue} benchmark.&#10; \item we show that our ELECTRA-DeBERTa small model (15M parameters), which has DeBERTa encoders with disentangled attention in ELECTRA's generator/discriminator architecture achieves an average score of 81.53 which is comparable to the average score 82.20 of BERT-Base (110M parameters).&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Model Efficiency Optimization, Artificial Intelligence, Natural Language Processing, Benchmarking Neural Networks" />
        </attvalues>
      </node>
      <node id="2011.02048" label="2011.02048">
        <attvalues>
          <attvalue for="0" value="SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End&#10;  Simultaneous Speech Translation" />
          <attvalue for="1" value="  Simultaneous text translation and end-to-end speech translation have recently&#10;made great progress but little work has combined these tasks together. We&#10;investigate how to adapt simultaneous text translation methods such as wait-k&#10;and monotonic multihead attention to end-to-end simultaneous speech translation&#10;by introducing a pre-decision module. A detailed analysis is provided on the&#10;latency-quality trade-offs of combining fixed and flexible pre-decision with&#10;fixed and flexible policies. We also design a novel computation-aware latency&#10;metric, adapted from Average Lagging.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.12093" label="2201.12093">
        <attvalues>
          <attvalue for="0" value="PCL: Peer-Contrastive Learning with Diverse Augmentations for&#10;  Unsupervised Sentence Embeddings" />
          <attvalue for="1" value="  Learning sentence embeddings in an unsupervised manner is fundamental in&#10;natural language processing. Recent common practice is to couple pre-trained&#10;language models with unsupervised contrastive learning, whose success relies on&#10;augmenting a sentence with a semantically-close positive instance to construct&#10;contrastive pairs. Nonetheless, existing approaches usually depend on a&#10;mono-augmenting strategy, which causes learning shortcuts towards the&#10;augmenting biases and thus corrupts the quality of sentence embeddings. A&#10;straightforward solution is resorting to more diverse positives from a&#10;multi-augmenting strategy, while an open question remains about how to&#10;unsupervisedly learn from the diverse positives but with uneven augmenting&#10;qualities in the text field. As one answer, we propose a novel Peer-Contrastive&#10;Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive&#10;positives and negatives at the group level for unsupervised sentence&#10;embeddings. PCL performs peer-positive contrast as well as peer-network&#10;cooperation, which offers an inherent anti-bias ability and an effective way to&#10;learn from diverse augmentations. Experiments on STS benchmarks verify the&#10;effectiveness of PCL against its competitors in unsupervised sentence&#10;embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Unsupervised Sentence Embedding.&#10;Common practice of unsupervised sentence embedding is to take the average of pre-trained word embeddings~\cite{Mikolov2013EfficientEO,pennington2014glove} PLMs, like BERT~\cite{Devlin2019BERTPO} or RoBERTa~\cite{Liu2019RoBERTaAR}, \cite{wu2021taking} takes the average of word embeddings as context embedding to enhance the language pre-training. Other works also take the [CLS] embedding from the last layer of PLMs with post-processing~\cite{Li2020OnTS,su2021whitening}. Some works~\cite{skipthought,logeswaran2018an,hill-etal-2016-learning} directly train a deep model for sentence embeddings using co-occurrence information. Recent approaches couple PLMs with CL~\cite{Zhang2020AnUS,yan2021consert,Kim2021SelfGuidedCL,Carlsson2021SemanticRW,Giorgi2021DeCLUTRDC,gao2021simcse,xie2022stable} with a particular single strategy to construct contrastive pairs.&#10;It is straightforward to extend mono-augmentation into multi-augmentation to learn expressive representations. For example, CLEAR~\cite{wu2020clear} uses various token/span manipulations for noise-invariant representations while Mirror-BERT~\cite{liu2021fast} employs several fast augmentation strategies. However, these methods usually take the augmented positives equally, regardless of the uncontrollable qualities. &#10;Thereby, we take a step further to consider the contrasts among the augmented positives to figure out which augmentation is relatively reasonable. This is achieved by our novel cooperative learning method with peer networks.&#10;More related to our work, ESimCSE~\cite{Wu2021ESimCSEES} found learning on dual-dropout causes sentence length bias so it employs another augmentation strategy, i.e., word repetition, to prevent the length bias. However, word repetition introduces learning shortcut by itself, not to mention it makes the sentence unnatural and even semantics-wrong. &#10;To circumvent this dilemma, we propose exhaustive augmentations to ensure ``at-least-one'' true positive and reduce learning shortcuts by complementary augmenting strategies. By doing so, PCL achieve a better performance on STS tasks.&#10;Please refer to \S~\ref{ap:realted} for more discussion details.&#10;&#10;Contrastive Learning.&#10;The main idea of CL is to pull semantic close neighbors close and push non-neighbors apart~\cite{hadsell2006dimensionality,pmlrzbontar21a}. It is shown to be a successful way to learn representation. Approaches in computer vision (CV)~\cite{chen2017sampling,Wu2018UnsupervisedFL,Tian2020ContrastiveMC,He2020MomentumCF,pmlrzbontar21a} try to make an image to be invariant to transformations on itself, while remaining discriminative to other images. More references in CV are discussed in the recent survey~\cite{jaiswal2021survey}. CL is also coupled with PLMs to learn sentence embeddings. But, recent works~\cite{Xiao2021WhatSN} argue that learning invariance to particular transformations may be harmful to the robustness of the model. This also supports our idea of leveraging diverse augmentations to improve unsupervised sentence embeddings from another angle.&#10;&#10;Learning from Agreement.&#10;Another line of work close to ours is learning from agreement, e.g., Decoupling~\cite{malach2017decoupling}, Co-teaching~\cite{han2018co,yu2019does}, and mutual CL \cite{Yang2021MutualCL}. This paradigm has been proven effective in improving model performance and learning with label noises by prior fully-supervised works.&#10;As text data is discrete and compositional, qualities of multiple augmentations can be uneven, which may corrupt the generalization of sentence embeddings. Besides widely used regularization like dropout~\cite{srivastava2014dropout} and weight decay~\cite{krogh1991simple}, we consider learning from agreement paradigm to offer a robust way to learn from our diverse positives.&#10;&#10;" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Artificial Intelligence, Sentence Embeddings, Natural Language Processing, Unsupervised Learning" />
        </attvalues>
      </node>
      <node id="2308.07922" label="2308.07922">
        <attvalues>
          <attvalue for="0" value="RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder&#10;  Language Models" />
          <attvalue for="1" value="  In this paper, we investigate the in-context learning ability of&#10;retrieval-augmented encoder-decoder language models. We first conduct a&#10;comprehensive analysis of existing models and identify their limitations in&#10;in-context learning, primarily due to a mismatch between pretraining and&#10;inference, as well as a restricted context length. To address these issues, we&#10;propose RAVEN, a model that combines retrieval-augmented masked language&#10;modeling and prefix language modeling. We further introduce Fusion-in-Context&#10;Learning to enhance the few-shot performance by enabling the model to leverage&#10;more in-context examples without requiring additional training. Through&#10;extensive experiments, we demonstrate that our simple yet effective design&#10;significantly improves performance, achieving results comparable to the most&#10;advanced language models in certain scenarios, despite having substantially&#10;fewer parameters. Our work underscores the potential of retrieval-augmented&#10;encoder-decoder language models for in-context learning and encourages further&#10;research in this direction.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancements in natural language processing have been predominantly driven by the development of large language models (LLMs) \cite{NEURIPS2020_1457c0d6,openai2022chatgpt,openai2023gpt4,chowdhery2022palm,smith2022using}.&#10;These models have demonstrated remarkable performance across a wide range of tasks~\cite{qin2023chatgpt,bubeck2023sparks,huang2022towards}. &#10;One of the key features that enables these models to excel is their ability to perform in-context learning~\cite{dong2022survey}. By conditioning on given context, LLMs can adapt to new tasks and domains without the need for task-specific fine-tuning. This enables LLMs to perform well on zero-shot or few-shot learning tasks, where only a limited number of examples are available.&#10;&#10;While in-context learning has been extensively studied for decoder-only language models like GPT-3~\cite{NEURIPS2020_1457c0d6} and PaLM~\cite{chowdhery2022palm}, research on encoder-decoder language models, which have shown to learn stronger representations~\cite{devlin-etal-2019-bert,10.5555/3455716.3455856}, remains limited. Notably, \cite{patel2023bidirectional} tap into the potential of mT5~\cite{xue-etal-2021-mt5}, a multilingual encoder-decoder LM, by iteratively prompting the model to produce long generations with in-context examples. \cite{chung2022scaling,longpre2023flan} finetune T5~\cite{10.5555/3455716.3455856} with a large mixture of tasks using instruction tuning~\cite{mishra-etal-2022-cross,wei2022finetuned,sanh2022multitask} to improve model performance and generalization to unseen tasks in both zero-shot and few-shot settings.&#10;&#10;On the other hand, LLMs still face challenges such as hallucination and limitations in representing the long-tail and most recent knowledge~\cite{mallen2022not,huang-etal-2022-large,luu-etal-2022-time,jang-etal-2022-temporalwiki,zheng2023does}. Retrieval-augmented language models~\cite{izacard2022few,pmlr-v162-borgeaud22a,wang2023shall,shi2023replug} have emerged as a powerful approach to address these issues by retrieving relevant knowledge from an external corpus. &#10;Among these, the encoder-decoder models, such as Atlas~\cite{izacard2022few}, stand out. They benefit from the strong representation ability of a bidirectional encoder, coupled with of the efficacy of a Fusion-in-Decoder architecture~\cite{izacard-grave-2021-leveraging}, enabling the effective integration of multiple retrieved passages.&#10;Despite these advancements, in-context learning with these models remains underexplored.\looseness=-1&#10;&#10;In this regard, we first conduct a comprehensive analysis of the state-of-the-art retrieval-augmented encoder-decoder language models by designing and experimenting with different prompting strategies. We find that these models exhibit a certain in-context learning ability; however, due to a mismatch between pretraining and inference and a limited context length—issues that are common to existing encoder-decoder LMs trained with masked language modeling—its few-shot performance is not stable&#10;and providing more than, e.g., 8-shot, examples does not lead to further improvement.&#10;&#10;Based on the analysis, we develop Raven by first mitigating the mismatch between pretraining and inference through a combination of retrieval-augmented masked language modeling and prefix language modeling. Moreover, to enable the model to learn from more in-context examples, we propose Fusion-in-Context Learning, a novel approach that allows the model to utilize more in-context examples without modifying the model configuration or requiring additional training. Furthermore, we suggest using the retriever of the model to obtain relevant in-context examples to further enhance few-shot performance. Our empirical results demonstrate that Raven significantly outperforms previous retrieval-augmented encoder-decoder LMs in both zero-shot and few-shot settings, even achieving comparable results to decoder-only LLMs in some settings despite having 180 times fewer parameters.&#10;&#10;The main contributions of this paper are twofold:&#10;\begin{itemize}[leftmargin=*, nolistsep]&#10;&#10;\item From an analytical standpoint, we provide a thorough analysis of the in-context learning ability of retrieval-augmented encoder-decoder language models. We demonstrate the possibilities and offer insights for future development.\looseness=-1&#10;\item From a technological perspective, we introduce Raven, coupled with our Fusion-in-Context Learning and In-Context Example Retrieval strategies, building upon the analytical groundwork. These techniques, though simple, are highly effective. They not only enhance the base model's capabilities but also highlight the potential of in-context learning with retrieval-augmented encoder-decoder LMs.\looseness=-1&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, In-Context Learning, Language Model Design, Mathematics, Retrieval-Augmented Modeling" />
        </attvalues>
      </node>
      <node id="2209.14500" label="2209.14500">
        <attvalues>
          <attvalue for="0" value="Bidirectional Language Models Are Also Few-shot Learners" />
          <attvalue for="1" value="  Large language models such as GPT-3 (Brown et al., 2020) can perform&#10;arbitrary tasks without undergoing fine-tuning after being prompted with only a&#10;few labeled examples. An arbitrary task can be reformulated as a natural&#10;language prompt, and a language model can be asked to generate the completion,&#10;indirectly performing the task in a paradigm known as prompt-based learning. To&#10;date, emergent prompt-based learning capabilities have mainly been demonstrated&#10;for unidirectional language models. However, bidirectional language models&#10;pre-trained on denoising objectives such as masked language modeling produce&#10;stronger learned representations for transfer learning. This motivates the&#10;possibility of prompting bidirectional models, but their pre-training&#10;objectives have made them largely incompatible with the existing prompting&#10;paradigm. We present SAP (Sequential Autoregressive Prompting), a technique&#10;that enables the prompting of bidirectional models. Utilizing the machine&#10;translation task as a case study, we prompt the bidirectional mT5 model (Xue et&#10;al., 2021) with SAP and demonstrate its few-shot and zero-shot translations&#10;outperform the few-shot translations of unidirectional models like GPT-3 and&#10;XGLM (Lin et al., 2021), despite mT5's approximately 50% fewer parameters. We&#10;further show SAP is effective on question answering and summarization. For the&#10;first time, our results demonstrate prompt-based learning is an emergent&#10;property of a broader class of language models, rather than only unidirectional&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.04225" label="2308.04225">
        <attvalues>
          <attvalue for="0" value="Investigating Speaker Embedding Disentanglement on Natural Read Speech" />
          <attvalue for="1" value="  Disentanglement is the task of learning representations that identify and&#10;separate factors that explain the variation observed in data. Disentangled&#10;representations are useful to increase the generalizability, explainability,&#10;and fairness of data-driven models. Only little is known about how well such&#10;disentanglement works for speech representations. A major challenge when&#10;tackling disentanglement for speech representations are the unknown generative&#10;factors underlying the speech signal. In this work, we investigate to what&#10;degree speech representations encoding speaker identity can be disentangled. To&#10;quantify disentanglement, we identify acoustic features that are highly&#10;speaker-variant and can serve as proxies for the factors of variation&#10;underlying speech. We find that disentanglement of the speaker embedding is&#10;limited when trained with standard objectives promoting disentanglement but can&#10;be improved over vanilla representation learning to some extent.&#10;" />
          <attvalue for="2" value="&#10;Speech is a rich source of information that conveys not only content-related information but also information about the speaker and the acoustic environment. &#10;Disentangling those sources of variation would be beneficial for &#10;a number of downstream tasks, for which either the content-related variations can be considered as noise (e.g., speaker recognition) or the speaker and environment-induced variations (e.g., speech recognition). Further, a fine-grained disentanglement could contribute to phonetic research by providing explicit control over single parameters.&#10;&#10;Among the deep generative models, the VAE~\cite{vae} is a particularly promising model because it provides an efficient inference mechanism to determine the latent factors of variation from the observed data. Although it has been shown that the vanilla VAE is unable to identify the true latent factors of variation from observed data \cite{Khemakhem2020}, this does not render the approach futile. For speech disentanglement where one is interested in separating static from dynamic factors, VAE-based encoder-decoder models with encoders specifically designed for speech have been developed. &#10;The hierarchical VAE \cite{Hsu2017UnsupervisedLO} and the factorized \cite{fcae} VAE use two encoders for unsupervised disentanglement of the speech signal into two disjoint representations where one captures short-term variations and the other long-term variations in the signal.&#10;The former representation, which is generated roughly at frame rate, can be attributed to capturing the linguistic content, and the latter, typically one vector per utterance, represents speaker, style, or environmental factors. We call this latter representation the speaker embedding or encoding.&#10;&#10;In this study, we go one step further and scrutinize the speaker encoding.&#10;Our goal is to reveal the fov underlying the speaker encoding and improve disentanglement.&#10;In the following, we refer to this task as (speaker) disentanglement.&#10;Ideally, a disentangled representation encodes all information about one fov in one latent dimension (compactness) and the fov do not overlap in the latent space (modularity).&#10;This would allow for dedicated manipulation of single factors by latent traversal.&#10;However, most research in this field focuses on analyzing disentanglement on controlled synthetic datasets~\cite{Higgins, Chen, dispeech, locatello2019challenging}.&#10;Here, we tackle disentanglement of the speaker encoding computed from natural speech, whose fov are unknown.&#10;We sidestep this issue by taking acoustic features that have a significant impact on speaker identity.&#10;&#10;The tools we employ for improving disentanglement are variants of the VAE, the &#10;$\beta$-VAE~\cite{Higgins} and the TCVAE~\cite{Chen}.&#10;The $\beta$-VAE introduces a weighting factor $\beta$ between the two loss terms of the VAE, the reconstruction loss and the KLD between the variational posterior and a normal isotropic prior, which is a regularization term that punishes deviations from a Gaussian random vector with independent components.&#10;Increasing the weight of the latter improves disentanglement, but at the cost of poorer reconstruction.&#10;The decomposition of the KLD proposed by Chen et al.~\cite{Chen} allows a fine-grained control over the different goals the VAE objective tries to optimize.&#10;&#10;To keep the investigation within the framework of VAE which reconstruct an observation from a latent encoding of the same, we train $\beta$-VAE and TCVAE to disentangle pretrained speaker embeddings (d-vectors) instead of integrating the VAE into the speaker embedding training.&#10;To quantify the disentanglement of the speaker embedding, we identify a small set of speaker-variant speech features as proxy factors and measure the modularity and compactness of the speaker embedding dimensions relative to the proxies with the measures defined in~\cite{Eastwood, carbonneau2022measuring}, while informativeness is assessed by speaker recognition experiments.&#10;We find that speaker disentanglement performance is limited and quickly loses informativeness with increasing contribution of the disentanglement objectives.&#10;Yet, we are able to show that these objectives can improve disentanglement to some extent.&#10; \newacronym{cacomp}{CaComp}{Canonical Completeness}&#10;\newacronym{cadis}{CaDis}{Canonical Disentanglement}&#10;&#10;\glsreset{ELBO}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Speech Processing, Disentanglement Methods, Linguistics, Machine Learning Fairness, Mathematics, Signal Processing, Speech Representation" />
        </attvalues>
      </node>
      <node id="1907.04809" label="1907.04809">
        <attvalues>
          <attvalue for="0" value="Variational Autoencoders and Nonlinear ICA: A Unifying Framework" />
          <attvalue for="1" value="  The framework of variational autoencoders allows us to efficiently learn deep&#10;latent-variable models, such that the model's marginal distribution over&#10;observed variables fits the data. Often, we're interested in going a step&#10;further, and want to approximate the true joint distribution over observed and&#10;latent variables, including the true prior and posterior distributions over&#10;latent variables. This is known to be generally impossible due to&#10;unidentifiability of the model. We address this issue by showing that for a&#10;broad family of deep latent-variable models, identification of the true joint&#10;distribution over observed and latent variables is actually possible up to very&#10;simple transformations, thus achieving a principled and powerful form of&#10;disentanglement. Our result requires a factorized prior distribution over the&#10;latent variables that is conditioned on an additionally observed variable, such&#10;as a class label or almost any other observation. We build on recent&#10;developments in nonlinear ICA, which we extend to the case with noisy,&#10;undercomplete or discrete observations, integrated in a maximum likelihood&#10;framework. The result also trivially contains identifiable flow-based&#10;generative models as a special case.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.12963" label="2005.12963">
        <attvalues>
          <attvalue for="0" value="Contrastive Predictive Coding Supported Factorized Variational&#10;  Autoencoder for Unsupervised Learning of Disentangled Speech Representations" />
          <attvalue for="1" value="  In this work we address disentanglement of style and content in speech&#10;signals. We propose a fully convolutional variational autoencoder employing two&#10;encoders: a content encoder and a style encoder. To foster disentanglement, we&#10;propose adversarial contrastive predictive coding. This new disentanglement&#10;method does neither need parallel data nor any supervision. We show that the&#10;proposed technique is capable of separating speaker and content traits into the&#10;two different representations and show competitive speaker-content&#10;disentanglement performance compared to other unsupervised approaches. We&#10;further demonstrate an increased robustness of the content representation&#10;against a train-test mismatch compared to spectral features, when used for&#10;phone recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.07707" label="2309.07707">
        <attvalues>
          <attvalue for="0" value="CoLLD: Contrastive Layer-to-layer Distillation for Compressing&#10;  Multilingual Pre-trained Speech Encoders" />
          <attvalue for="1" value="  Large-scale self-supervised pre-trained speech encoders outperform&#10;conventional approaches in speech recognition and translation tasks. Due to the&#10;high cost of developing these large models, building new encoders for new tasks&#10;and deploying them to on-device applications are infeasible. Prior studies&#10;propose model compression methods to address this issue, but those works focus&#10;on smaller models and less realistic tasks. Thus, we propose Contrastive&#10;Layer-to-layer Distillation (CoLLD), a novel knowledge distillation method to&#10;compress pre-trained speech encoders by leveraging masked prediction and&#10;contrastive learning to train student models to copy the behavior of a large&#10;teacher model. CoLLD outperforms prior methods and closes the gap between small&#10;and large models on multilingual speech-to-text translation and recognition&#10;benchmarks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-supervised learning~(SSL) for speech encoder pre-training benefits various speech processing tasks and outperforms conventional approaches~\cite{mohamed2022self}.&#10;SSL methods leverage large unlabeled speech corpus to train deep neural networks to encode useful representations and succeed in applications like speech translation~\cite{seamlessm4t2023} and automatic speech recognition~(ASR)~\cite{zhang2023usm}.&#10;However, powerful speech encoders usually have many parameters, making real-time or on-device speech processing less feasible.&#10;&#10;Researchers propose model compression techniques to address the issues of large speech encoders.&#10;The compressed SSL pre-trained encoders can be applied to various downstream tasks.&#10;These approaches can be categorized into knowledge distillation~(KD) and parameter pruning.&#10;In KD, a lightweight student model learns to predict hidden representations to mimic the large teacher model's behavior~\cite{chang2022distilhubert,lee2022fithubert,ashihara2022deep,wang2022lighthubert,huang2023ensemble,jang2023recycle,wang2023distilxlsr}.&#10;DistilHuBERT~\cite{chang2022distilhubert} predicts multiple hidden layers in a HuBERT teacher~\cite{hsu2021hubert} using the student's output with separate prediction heads.&#10;FitHuBERT~\cite{lee2022fithubert} and Ashihara et al.~\cite{ashihara2022deep} propose layer-to-layer~(L2L) KD that uses narrow and deep students to layer-wise distill the teacher's hidden representations.&#10;In unstructured pruning, parameters with small values are set to zero~\cite{lai2021parp}, while structured pruning removes submodules from a model~\cite{peng2023structured,jiang2023accurate,wang2023task} to reduce the parameters but requires complicated implementation.&#10;Other studies combine the above methods~\cite{peng2023dphubert} or techniques like layer-skipping~\cite{peng2023i3d} and low-bit quantization~\cite{yeh2022efficient}.&#10;&#10;Although existing methods succeed in many tasks, most works focus on compressing small SSL models and evaluating with unrealistic problem setups.&#10;Those works compress a HuBERT Base~\cite{hsu2021hubert} model~(95M parameters) to models around 20M to 30M parameters and evaluate with the Speech processing Universal PERformance Benchmark~(SUPERB)~\cite{yang2021superb,tsai-etal-2022-superb}.&#10;These compressed models are unsuitable for complex tasks that require fine-tuning because of the small model capacities, limiting application scenarios.&#10;Under this setting, the effectiveness of these methods for large-scale models and problems remains to be discovered.&#10;&#10;To bridge the gap between academic research and real-world problems, we extend the speech encoder compression task to a large-scale pre-trained speech encoder (w2v-BERT 2.0~\cite{seamlessm4t2023}) and apply the compressed model to multilingual speech-to-text translation~(S2T).&#10;This problem is challenging because the original model is significantly larger~(1B parameters), and the compressed model is fine-tuned with a more complicated yet realistic task.&#10;Following previous studies, we use unlabeled data to compress an SSL pre-trained teacher model because this setup allows flexible utilization and avoids fine-tuning huge encoders.&#10;Moreover, the compressed encoder has 300M parameters, which is currently the largest encoder size widely used in both production and academia~\cite{yang2021superb}.&#10;&#10;Under this new problem setting, we propose Contrastive Layer-to-layer Distillation~(CoLLD) by combining L2L KD~\cite{ashihara2022deep} and a contrastive masked prediction learning objective~\cite{baevski2020wav2vec2}.&#10;First, some student model input frames are masked while the teacher remains unmasked.&#10;Then, each masked student's hidden layer frame classifies the corresponding teacher's hidden layer frame from a set of distractors, where the distractors are randomly sampled from other frames of the teacher's representations.&#10;After distillation, we evaluate the student model with internal and public benchmarks, covering S2T and multilingual ASR.&#10;As shown in Fig.~\ref{fig:size-bleu} and Sec.~\ref{sec:exp}, CoLLD~surpasses prior distillation methods, narrows the performance gap between large models~(0.6B and 1.0B parameters) and outperforms strong baselines like XLS-R~\cite{babu2022xlsr} and MMS~\cite{pratap2023mms}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Compression, Deep Learning, Artificial Intelligence, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2207.06867" label="2207.06867">
        <attvalues>
          <attvalue for="0" value="Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic&#10;  Knowledge Distillation of Self-Supervised Speech Models" />
          <attvalue for="1" value="  Self-supervised learning (SSL) is seen as a very promising approach with high&#10;performance for several speech downstream tasks. Since the parameters of SSL&#10;models are generally so large that training and inference require a lot of&#10;memory and computational cost, it is desirable to produce compact SSL models&#10;without a significant performance degradation by applying compression methods&#10;such as knowledge distillation (KD). Although the KD approach is able to shrink&#10;the depth and/or width of SSL model structures, there has been little research&#10;on how varying the depth and width impacts the internal representation of the&#10;small-footprint model. This paper provides an empirical study that addresses&#10;the question. We investigate the performance on SUPERB while varying the&#10;structure and KD methods so as to keep the number of parameters constant; this&#10;allows us to analyze the contribution of the representation introduced by&#10;varying the model architecture. Experiments demonstrate that a certain depth is&#10;essential for solving content-oriented tasks (e.g. automatic speech&#10;recognition) accurately, whereas a certain width is necessary for achieving&#10;high performance on several speaker-oriented tasks (e.g. speaker&#10;identification). Based on these observations, we identify, for SUPERB, a more&#10;compressed model with better performance than previous studies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.12757" label="2302.12757">
        <attvalues>
          <attvalue for="0" value="Ensemble knowledge distillation of self-supervised speech models" />
          <attvalue for="1" value="  Distilled self-supervised models have shown competitive performance and&#10;efficiency in recent years. However, there is a lack of experience in jointly&#10;distilling multiple self-supervised speech models. In our work, we performed&#10;Ensemble Knowledge Distillation (EKD) on various self-supervised speech models&#10;such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation&#10;techniques, layerwise-average and layerwise-concatenation, to the&#10;representations of different teacher models and found that the former was more&#10;effective. On top of that, we proposed a multiple prediction head method for&#10;student models to predict different layer outputs of multiple teacher models&#10;simultaneously. The experimental results show that our method improves the&#10;performance of the distilled models on four downstream speech processing tasks,&#10;Phoneme Recognition, Speaker Identification, Emotion Recognition, and Automatic&#10;Speech Recognition in the hidden-set track of the SUPERB benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11685" label="2305.11685">
        <attvalues>
          <attvalue for="0" value="Recycle-and-Distill: Universal Compression Strategy for&#10;  Transformer-based Speech SSL Models with Attention Map Reusing and Masking&#10;  Distillation" />
          <attvalue for="1" value="  Transformer-based speech self-supervised learning (SSL) models, such as&#10;HuBERT, show surprising performance in various speech processing tasks.&#10;However, huge number of parameters in speech SSL models necessitate the&#10;compression to a more compact model for wider usage in academia or small&#10;companies. In this study, we suggest to reuse attention maps across the&#10;Transformer layers, so as to remove key and query parameters while retaining&#10;the number of layers. Furthermore, we propose a novel masking distillation&#10;strategy to improve the student model's speech representation quality. We&#10;extend the distillation loss to utilize both masked and unmasked speech frames&#10;to fully leverage the teacher model's high-quality representation. Our&#10;universal compression strategy yields the student model that achieves phoneme&#10;error rate (PER) of 7.72% and word error rate (WER) of 9.96% on the SUPERB&#10;benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.19549" label="2305.19549">
        <attvalues>
          <attvalue for="0" value="Accurate and Structured Pruning for Efficient Automatic Speech&#10;  Recognition" />
          <attvalue for="1" value="  Automatic Speech Recognition (ASR) has seen remarkable advancements with deep&#10;neural networks, such as Transformer and Conformer. However, these models&#10;typically have large model sizes and high inference costs, posing a challenge&#10;to deploy on resource-limited devices. In this paper, we propose a novel&#10;compression strategy that leverages structured pruning and knowledge&#10;distillation to reduce the model size and inference cost of the Conformer model&#10;while preserving high recognition performance. Our approach utilizes a set of&#10;binary masks to indicate whether to retain or prune each Conformer module, and&#10;employs L0 regularization to learn the optimal mask values. To further enhance&#10;pruning performance, we use a layerwise distillation strategy to transfer&#10;knowledge from unpruned to pruned models. Our method outperforms all pruning&#10;baselines on the widely used LibriSpeech benchmark, achieving a 50% reduction&#10;in model size and a 28% reduction in inference cost with minimal performance&#10;loss.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.01385" label="2306.01385">
        <attvalues>
          <attvalue for="0" value="Task-Agnostic Structured Pruning of Speech Representation Models" />
          <attvalue for="1" value="  Self-supervised pre-trained models such as Wav2vec2, Hubert, and WavLM have&#10;been shown to significantly improve many speech tasks. However, their large&#10;memory and strong computational requirements hinder their industrial&#10;applicability. Structured pruning is a hardware-friendly model compression&#10;technique but usually results in a larger loss of accuracy. In this paper, we&#10;propose a fine-grained attention head pruning method to compensate for the&#10;performance degradation. In addition, we also introduce the straight through&#10;estimator into the L0 regularization to further accelerate the pruned model.&#10;Experiments on the SUPERB benchmark show that our model can achieve comparable&#10;performance to the dense model in multiple tasks and outperforms the Wav2vec&#10;2.0 base model on average, with 72% fewer parameters and 2 times faster&#10;inference speed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.17651" label="2305.17651">
        <attvalues>
          <attvalue for="0" value="DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech&#10;  Models" />
          <attvalue for="1" value="  Self-supervised learning (SSL) has achieved notable success in many speech&#10;processing tasks, but the large model size and heavy computational cost hinder&#10;the deployment. Knowledge distillation trains a small student model to mimic&#10;the behavior of a large teacher model. However, the student architecture&#10;usually needs to be manually designed and will remain fixed during training,&#10;which requires prior knowledge and can lead to suboptimal performance. Inspired&#10;by recent success of task-specific structured pruning, we propose DPHuBERT, a&#10;novel task-agnostic compression method for speech SSL based on joint&#10;distillation and pruning. Experiments on SUPERB show that DPHuBERT outperforms&#10;pure distillation methods in almost all tasks. Moreover, DPHuBERT requires&#10;little training time and performs well with limited training data, making it&#10;suitable for resource-constrained applications. Our method can also be applied&#10;to various speech SSL models. Our code and models will be publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.07624" label="2303.07624">
        <attvalues>
          <attvalue for="0" value="I3D: Transformer architectures with input-dependent dynamic depth for&#10;  speech recognition" />
          <attvalue for="1" value="  Transformer-based end-to-end speech recognition has achieved great success.&#10;However, the large footprint and computational overhead make it difficult to&#10;deploy these models in some real-world applications. Model compression&#10;techniques can reduce the model size and speed up inference, but the compressed&#10;model has a fixed architecture which might be suboptimal. We propose a novel&#10;Transformer encoder with Input-Dependent Dynamic Depth (I3D) to achieve strong&#10;performance-efficiency trade-offs. With a similar number of layers at inference&#10;time, I3D-based models outperform the vanilla Transformer and the static pruned&#10;model via iterative layer pruning. We also present interesting analysis on the&#10;gate probabilities and the input-dependency, which helps us better understand&#10;deep encoders.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.00652" label="2301.00652">
        <attvalues>
          <attvalue for="0" value="Efficient Speech Representation Learning with Low-Bit Quantization" />
          <attvalue for="1" value="  With the development of hardware for machine learning, newer models often&#10;come at the cost of both increased sizes and computational complexity. In&#10;effort to improve the efficiency for these models, we apply and investigate&#10;recent quantization techniques on speech representation learning models. The&#10;quantization techniques were evaluated on the SUPERB benchmark. On the ASR&#10;task, with aggressive quantization to 1 bit, we achieved 86.32% storage&#10;reduction (184.42 -&gt; 25.23), 88% estimated runtime reduction (1.00 -&gt; 0.12)&#10;with increased word error rate (7.06 -&gt; 15.96). In comparison with&#10;DistillHuBERT which also aims for model compression, the 2-bit configuration&#10;yielded slightly smaller storage (35.84 vs. 46.98), better word error rate&#10;(12.68 vs. 13.37) and more efficient estimated runtime (0.15 vs. 0.73).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13516" label="2305.13516">
        <attvalues>
          <attvalue for="0" value="Scaling Speech Technology to 1,000+ Languages" />
          <attvalue for="1" value="  Expanding the language coverage of speech technology has the potential to&#10;improve access to information for many more people. However, current speech&#10;technology is restricted to about one hundred languages which is a small&#10;fraction of the over 7,000 languages spoken around the world. The Massively&#10;Multilingual Speech (MMS) project increases the number of supported languages&#10;by 10-40x, depending on the task. The main ingredients are a new dataset based&#10;on readings of publicly available religious texts and effectively leveraging&#10;self-supervised learning. We built pre-trained wav2vec 2.0 models covering&#10;1,406 languages, a single multilingual automatic speech recognition model for&#10;1,107 languages, speech synthesis models for the same number of languages, as&#10;well as a language identification model for 4,017 languages. Experiments show&#10;that our multilingual speech recognition model more than halves the word error&#10;rate of Whisper on 54 languages of the FLEURS benchmark while being trained on&#10;a small fraction of the labeled data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.05782" label="2109.05782">
        <attvalues>
          <attvalue for="0" value="Effectiveness of Pre-training for Few-shot Intent Classification" />
          <attvalue for="1" value="  This paper investigates the effectiveness of pre-training for few-shot intent&#10;classification. While existing paradigms commonly further pre-train language&#10;models such as BERT on a vast amount of unlabeled corpus, we find it highly&#10;effective and efficient to simply fine-tune BERT with a small set of labeled&#10;utterances from public datasets. Specifically, fine-tuning BERT with roughly&#10;1,000 labeled data yields a pre-trained model -- IntentBERT, which can easily&#10;surpass the performance of existing pre-trained models for few-shot intent&#10;classification on novel domains with very different semantics. The high&#10;effectiveness of IntentBERT confirms the feasibility and practicality of&#10;few-shot intent detection, and its high generalization ability across different&#10;domains suggests that intent classification tasks may share a similar&#10;underlying structure, which can be efficiently learned from a small set of&#10;labeled data. The source code can be found at&#10;https://github.com/hdzhang-code/IntentBERT.&#10;" />
          <attvalue for="2" value="&#10;\efboxsetup{linecolor=black,linewidth=0.7pt}&#10;&#10;Task-oriented dialogue systems have been widely deployed to a variety of sectors~\cite{yan2017building, chen2017survey, zhang2020recent, NEURIPS2020_e9462095}, ranging from shopping~\cite{yan2017building} to medical services~\cite{arora2020cross, wei-etal-2018-task}, to provide interactive experience. Training an accurate intent classifier is vital for the development of such task-oriented dialogue systems. However, an important issue is how to achieve this when only limited number of labeled instances are available, which is often the case at the early development stage.&#10;&#10;To tackle few-shot intent detection, some recent attempts employ induction network~\cite{geng2019few}, generation-based methods~\cite{xia2020composed, xia-etal-2020-composed}, metric learning~\cite{nguyen2020dynamic}, or self-training~\cite{dopierre2020few}. These works mainly focus on designing novel algorithms for representation learning and inference, which often comes with complicated models. Most recently, large-scale pre-trained language models such as BERT~\cite{devlin2018bert, radford2019language, brown2020language} have shown great promise in many natural language understanding tasks~\cite{DBLP:conf/nips/WangPNSMHLB19}, and there has been a surge of interest in fine-tuning the pre-trained language models for intent detection~\cite{zhang2020discriminative, zhang2020intent, peng2020soloist, wu2020tod, casanueva2020efficient,larson2019evaluation}.&#10;&#10;While fine-tuning pre-trained language models on large-scale annotated datasets has yielded significant improvements in many tasks including intent detection, it is laborious and expensive to construct large-scale annotated datasets in new application domains.&#10;Therefore, recent efforts have been dedicated to adapting pre-trained language models to a specific task such as intent detection by conducting continued pre-training~\cite{gururangan2020don, gu-etal-2021-pral} on a large unlabeled dialogue corpus with a specially designed optimization objective. Below we summarize the most related works in this line of research for few-shot intent detection.&#10;&#10;\begin{itemize}&#10; \item CONVBERT~\cite{mehri2020dialoglue} finetunes BERT on an unlabeled dialogue corpus consisting of nearly $700$ million conversations.&#10; \item TOD-BERT~\cite{wu2020tod} further pre-trains BERT on a task-oriented dialogue corpus of $100,000$ unlabeled samples with masked language modelling~(MLM) and response contrastive objectives.&#10; \item USE-ConveRT~\cite{henderson-etal-2020-convert, casanueva2020efficient} investigates a dual encoder model trained with response selection tasks on $727$ million input-response pairs.&#10; \item DNNC~\cite{zhang2020discriminative} pre-trains a language model with around $1$ million annotated samples for natural language inference~(NLI) and use the pre-trained model for intent detection. \item WikiHowRoBERTa~\cite{zhang2020intent} constructs some pre-training tasks based on the wikiHow database with $110,000$ articles.&#10;\end{itemize}&#10;&#10;While these methods have achieved impressive performance, they heavily rely on the existence of a large-scale corpus~\cite{mehri2020dialoglue} that is close in semantics to the target domain or consists of similar tasks&#10;for continued pre-training, which needs huge effort for data collection and comes at a high computational cost. More importantly, they completely ignore the ``free lunch'' --&#10;the publicly available, high-quality, manually-annotated intent detection benchmarks. For example, the dataset OOS~\cite{larson2019evaluation} provides labeled utterances across $10$ different domains. &#10;Hence, our study in this paper centers around the following research question: &#10;\begin{itemize}&#10; \item Is it possible to utilize publicly available datasets to pre-train an intent detection model that can learn transferable task-specific knowledge to generalize across different domains?&#10;\end{itemize}&#10;&#10;In this paper, we provide an affirmative answer to this question. We fine-tune BERT using a simple standard supervised training with approximately 1,000 labeled utterances from public datasets and obtain a pre-trained model, called IntentBERT. It can be directly applied for few-shot intent classification on a target domain that is drastically different from the pre-training data and &#10;significantly outperform existing pre-trained models,&#10;without further fine-tuning on target data (labeled or unlabeled). This simple ``free-lunch'' solution not only confirms the feasibility and practicality of few-shot intent detection, but also provides a ready-to-use well-performing model for practical use, saving the effort in algorithm design and data collection. Moreover, the high generalization ability of IntentBERT on cross-domain few-shot classification tasks, which are generally considered very difficult due to large domain gaps and the few data constraint, suggests that most intent detection tasks probably share a common underlying structure that could be learned from a small set of data.&#10;&#10;Further, to leverage unlabeled data in the target domain, we design a joint pre-training scheme, which simultaneously optimizes the classification error on the source labeled data and the language modeling loss on the target unlabeled data. This joint-training scheme can learn better semantic representations and significantly outperforms existing two-stage pre-training methods~\cite{gururangan2020don}. A visualization of the embedding spaces produced by strong baselines and our methods is provided in Fig.~\ref{figure: scatter plot}, which clearly demonstrates the superiority of our pre-trained models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Computer Science, Linguistics, Language Model Pre-training, Intent Classification, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2010.02481" label="2010.02481">
        <attvalues>
          <attvalue for="0" value="Dynamic Semantic Matching and Aggregation Network for Few-shot Intent&#10;  Detection" />
          <attvalue for="1" value="  Few-shot Intent Detection is challenging due to the scarcity of available&#10;annotated utterances. Although recent works demonstrate that multi-level&#10;matching plays an important role in transferring learned knowledge from seen&#10;training classes to novel testing classes, they rely on a static similarity&#10;measure and overly fine-grained matching components. These limitations inhibit&#10;generalizing capability towards Generalized Few-shot Learning settings where&#10;both seen and novel classes are co-existent. In this paper, we propose a novel&#10;Semantic Matching and Aggregation Network where semantic components are&#10;distilled from utterances via multi-head self-attention with additional dynamic&#10;regularization constraints. These semantic components capture high-level&#10;information, resulting in more effective matching between instances. Our&#10;multi-perspective matching method provides a comprehensive matching measure to&#10;enhance representations of both labeled and unlabeled instances. We also&#10;propose a more challenging evaluation setting that considers classification on&#10;the joint all-class label space. Extensive experimental results demonstrate the&#10;effectiveness of our method. Our code and data are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.02027" label="1909.02027">
        <attvalues>
          <attvalue for="0" value="An Evaluation Dataset for Intent Classification and Out-of-Scope&#10;  Prediction" />
          <attvalue for="1" value="  Task-oriented dialog systems need to know when a query falls outside their&#10;range of supported intents, but current text classification corpora only define&#10;label sets that cover every example. We introduce a new dataset that includes&#10;queries that are out-of-scope---i.e., queries that do not fall into any of the&#10;system's supported intents. This poses a new challenge because models cannot&#10;assume that every query at inference time belongs to a system-supported intent&#10;class. Our dataset also covers 150 intent classes over 10 domains, capturing&#10;the breadth that a production task-oriented agent must handle. We evaluate a&#10;range of benchmark classifiers on our dataset along with several different&#10;out-of-scope identification schemes. We find that while the classifiers perform&#10;well on in-scope intent classification, they struggle to identify out-of-scope&#10;queries. Our dataset and evaluation fill an important gap in the field,&#10;offering a way of more rigorously and realistically benchmarking text&#10;classification in task-driven dialog systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.13570" label="2009.13570">
        <attvalues>
          <attvalue for="0" value="DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented&#10;  Dialogue" />
          <attvalue for="1" value="  A long-standing goal of task-oriented dialogue research is the ability to&#10;flexibly adapt dialogue models to new domains. To progress research in this&#10;direction, we introduce DialoGLUE (Dialogue Language Understanding Evaluation),&#10;a public benchmark consisting of 7 task-oriented dialogue datasets covering 4&#10;distinct natural language understanding tasks, designed to encourage dialogue&#10;research in representation-based transfer, domain adaptation, and&#10;sample-efficient task learning. We release several strong baseline models,&#10;demonstrating performance improvements over a vanilla BERT architecture and&#10;state-of-the-art results on 5 out of 7 tasks, by pre-training on a large&#10;open-domain dialogue corpus and task-adaptive self-supervised training. Through&#10;the DialoGLUE benchmark, the baseline methods, and our evaluation scripts, we&#10;hope to facilitate progress towards the goal of developing more general&#10;task-oriented dialogue models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.03688" label="1911.03688">
        <attvalues>
          <attvalue for="0" value="ConveRT: Efficient and Accurate Conversational Representations from&#10;  Transformers" />
          <attvalue for="1" value="  General-purpose pretrained sentence encoders such as BERT are not ideal for&#10;real-world conversational AI applications; they are computationally heavy,&#10;slow, and expensive to train. We propose ConveRT (Conversational&#10;Representations from Transformers), a pretraining framework for conversational&#10;tasks satisfying all the following requirements: it is effective, affordable,&#10;and quick to train. We pretrain using a retrieval-based response selection&#10;task, effectively leveraging quantization and subword-level parameterization in&#10;the dual encoder to build a lightweight memory- and energy-efficient model. We&#10;show that ConveRT achieves state-of-the-art performance across widely&#10;established response selection tasks. We also demonstrate that the use of&#10;extended dialog history as context yields further performance gains. Finally,&#10;we show that pretrained representations from the proposed encoder can be&#10;transferred to the intent classification task, yielding strong results across&#10;three diverse data sets. ConveRT trains substantially faster than standard&#10;sentence encoders or previous state-of-the-art dual encoders. With its reduced&#10;size and superior performance, we believe this model promises wider portability&#10;and scalability for Conversational AI applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14318" label="2010.14318">
        <attvalues>
          <attvalue for="0" value="Multitask Training with Text Data for End-to-End Speech Recognition" />
          <attvalue for="1" value="  We propose a multitask training method for attention-based end-to-end speech&#10;recognition models. We regularize the decoder in a listen, attend, and spell&#10;model by multitask training it on both audio-text and text-only data. Trained&#10;on the 100-hour subset of LibriSpeech, the proposed method, without requiring&#10;an additional language model, leads to an 11% relative performance improvement&#10;over the baseline and approaches the performance of language model shallow&#10;fusion on the test-clean evaluation set. We observe a similar trend on the&#10;whole 960-hour LibriSpeech training set. Analyses of different types of errors&#10;and sample output sentences demonstrate that the proposed method can&#10;incorporate language level information, suggesting its effectiveness in&#10;real-world applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1411.1792" label="1411.1792">
        <attvalues>
          <attvalue for="0" value="How transferable are features in deep neural networks?" />
          <attvalue for="1" value="  Many deep neural networks trained on natural images exhibit a curious&#10;phenomenon in common: on the first layer they learn features similar to Gabor&#10;filters and color blobs. Such first-layer features appear not to be specific to&#10;a particular dataset or task, but general in that they are applicable to many&#10;datasets and tasks. Features must eventually transition from general to&#10;specific by the last layer of the network, but this transition has not been&#10;studied extensively. In this paper we experimentally quantify the generality&#10;versus specificity of neurons in each layer of a deep convolutional neural&#10;network and report a few surprising results. Transferability is negatively&#10;affected by two distinct issues: (1) the specialization of higher layer neurons&#10;to their original task at the expense of performance on the target task, which&#10;was expected, and (2) optimization difficulties related to splitting networks&#10;between co-adapted neurons, which was not expected. In an example network&#10;trained on ImageNet, we demonstrate that either of these two issues may&#10;dominate, depending on whether features are transferred from the bottom,&#10;middle, or top of the network. We also document that the transferability of&#10;features decreases as the distance between the base task and target task&#10;increases, but that transferring features even from distant tasks can be better&#10;than using random features. A final surprising result is that initializing a&#10;network with transferred features from almost any number of layers can produce&#10;a boost to generalization that lingers even after fine-tuning to the target&#10;dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.08779" label="1904.08779">
        <attvalues>
          <attvalue for="0" value="SpecAugment: A Simple Data Augmentation Method for Automatic Speech&#10;  Recognition" />
          <attvalue for="1" value="  We present SpecAugment, a simple data augmentation method for speech&#10;recognition. SpecAugment is applied directly to the feature inputs of a neural&#10;network (i.e., filter bank coefficients). The augmentation policy consists of&#10;warping the features, masking blocks of frequency channels, and masking blocks&#10;of time steps. We apply SpecAugment on Listen, Attend and Spell networks for&#10;end-to-end speech recognition tasks. We achieve state-of-the-art performance on&#10;the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work.&#10;On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language&#10;model, and 5.8% WER with shallow fusion with a language model. This compares to&#10;the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we&#10;achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5'00 test set&#10;without the use of a language model, and 6.8%/14.1% with shallow fusion, which&#10;compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.09783" label="2206.09783">
        <attvalues>
          <attvalue for="0" value="Boosting Cross-Domain Speech Recognition with Self-Supervision" />
          <attvalue for="1" value="  The cross-domain performance of automatic speech recognition (ASR) could be&#10;severely hampered due to the mismatch between training and testing&#10;distributions. Since the target domain usually lacks labeled data, and domain&#10;shifts exist at acoustic and linguistic levels, it is challenging to perform&#10;unsupervised domain adaptation (UDA) for ASR. Previous work has shown that&#10;self-supervised learning (SSL) or pseudo-labeling (PL) is effective in UDA by&#10;exploiting the self-supervisions of unlabeled data. However, these&#10;self-supervisions also face performance degradation in mismatched domain&#10;distributions, which previous work fails to address. This work presents a&#10;systematic UDA framework to fully utilize the unlabeled data with&#10;self-supervision in the pre-training and fine-tuning paradigm. On the one hand,&#10;we apply continued pre-training and data replay techniques to mitigate the&#10;domain mismatch of the SSL pre-trained model. On the other hand, we propose a&#10;domain-adaptive fine-tuning approach based on the PL technique with three&#10;unique modifications: Firstly, we design a dual-branch PL method to decrease&#10;the sensitivity to the erroneous pseudo-labels; Secondly, we devise an&#10;uncertainty-aware confidence filtering strategy to improve pseudo-label&#10;correctness; Thirdly, we introduce a two-step PL approach to incorporate target&#10;domain linguistic knowledge, thus generating more accurate target domain&#10;pseudo-labels. Experimental results on various cross-domain scenarios&#10;demonstrate that the proposed approach effectively boosts the cross-domain&#10;performance and significantly outperforms previous approaches.&#10;" />
          <attvalue for="2" value="&#10;T{he} performance of end-to-end (E2E) automatic speech recognition (ASR) systems has improved dramatically over the past years \cite{li2021recent,cheng2022eteh,huang2019exploring,miao2020online,miao2020transformer} due to the advanced neural network architectures, improved training criteria, and large amounts of training data. However, the performance degradation on the cross-domain data is still a challenging issue for ASR due to the domain shift between the training and testing data. Since it is impossible to cover all test domains in the training data, applying domain adaptation \cite{bell2020adaptation} for a new target domain is of great interest in the application of ASR.&#10;&#10;Domain adaptation aims to transfer a model trained on the source data to a given target domain in supervised or unsupervised conditions.&#10;When the labeled data is available in the target domain, the supervised domain adaptation is straightforward since we could simply use the labeled data to fine-tune the source model \cite{sim2018domain,hou2021exploiting}. &#10;However, since the labeled target data is costly to collect, the unsupervised domain adaptation (UDA) scenario, where no labeled data is available in the target domain, is more desired in real-world applications.&#10;&#10;Existing UDA approaches tackle the lack of labeled data issue from different aspects.&#10;An intuitive solution is to synthesize target domain data \cite{li2017large,hsu2017unsupervised,hosseini2018multi,li2020developing,baskar2021eat,yue2022exploring}.&#10;However, these approaches require a specific design for the target domain \cite{li2017large} or careful tuning of the data generation model \cite{hosseini2018multi}, making them inconvenient when extending to an arbitrary new domain or large-scale applications.&#10;Another category is domain-invariant feature learning with distribution matching approaches \cite{hou2021cross,sun2018domain}, which aims to learn a domain-invariant representation while being class-discriminative on the source domain.&#10;However, the class-discriminative representations on the target domain cannot be easily guaranteed.&#10;Thus, it would fail under certain domain mismatch scenarios \cite{li2020rethinking,zhao2019learning,liu2019transferable}.&#10;&#10;Recently, self-supervised learning (SSL) based pre-training~\cite{devlin2019bert,hwang2021large,misra2021comparison,hsu2021robust} and pseudo-labeling (PL)~\cite{khurana2021unsupervised,higuchi2021momentum} are shown to be effective for UDA of the E2E-ASR model by directly training on the unlabeled target data with self-supervision.&#10;On the one hand, both SSL and PL approaches are simple in practice and do not need customization for specific domains.&#10;Thus, they are convenient to be applied to any new domain and large-scale applications.&#10;On the other hand, SSL and PL are shown to be robust under various domain mismatch conditions \cite{hsu2021robust,khurana2021unsupervised,higuchi2021momentum}.&#10;&#10;Nonetheless, existing literature typically focused on one aspect to address the UDA problem, e.g., with SSL~\cite{hsu2021robust}, online~\cite{higuchi2021momentum} or offline PL~\cite{khurana2021unsupervised}. This practice failed to realize the full potential of self-supervision for UDA. To push the limits of UDA, in this work, we first identify the weaknesses of SSL and PL when applying them to the UDA scenario and propose innovative solutions to address them. Then, we seamlessly integrate the improved SSL and PL approaches into a systematic UDA framework to boost Cross-domAin Speech recogniTion with SeLf-SupErvision, namely, CASTLE. &#10;&#10;In summary, the major novelties of this paper are as follows:&#10;&#10;\begin{itemize}&#10;&#10;\item Dual-Branch PL (DPL): There are two vital challenges in existing online PL approaches: Firstly, since the self-generated pseudo-labels are used as the supervision, the errors would be accumulated and cause the error accumulation~\cite{jiang2020implicit} (or the confirmation bias~\cite{arazo2020pseudo}) issue, degrading the performance and sometimes driving the training to collapse. Secondly, due to the lack of confidence filtering design for online PL, all pseudo-labels are used in training, including the extremely noisy pseudo-labels. To address these challenges, on the one hand, DPL proposes to break the error accumulation chain by using an auxiliary branch to generate pseudo-labels. On the other hand, DPL utilizes a specifically designed confidence estimation that discards the CTC blank scores, thus robustly improving the online PL. Extensive experiments demonstrated the advantage of DPL over existing online PL approaches.&#10;\item Uncertainty-Aware Confidence Filtering (UCF): Most existing filtering methods for offline PL utilize decoding scores as confidence estimation to rule out the noisy pseudo-labels~\cite{kahn2020self,park2020improved}. However, this confidence estimation is unreliable when ASR networks are poorly calibrated~\cite{li2021confidence}. UCF addressed this issue by adaptively utilizing uncertainty~\cite{gal2016dropout} and confidence estimations to select pseudo-labels in offline PL, where the combination hyper-parameters in UCF are adaptively determined on the development set. Experiments demonstrated that UCF can outperform existing filtering approaches without tuning hyper-parameters.&#10;\item Two-Step PL: Two-step PL is an empirically motivated approach that utilizes the offline PL to refine the online PL linguistically. Although a simple combination of two types of PL approaches, it consistently outperforms either one of them in practice.&#10;\item Continued Pre-Training with Data Replay: Catastrophic forgetting is a critical issue in continual learning. Since continued pre-training~\cite{gururangan2020don,hsu2021robust} is proven effective and used in our approach, we examine whether catastrophic forgetting is severe here by evaluating the effectiveness of data replay~\cite{hu2022how}, which is widely adopted to address knowledge forgetting. Based on experimental findings, we provide suggestions on continued pre-training strategies given different fine-tuning strategies.&#10;\end{itemize}&#10;&#10;We performed detailed experiments on various cross-domain datasets and showed that the proposed approach CASTLE could effectively boost the cross-domain performance of ASR and consistently outperforms previous UDA approaches. &#10;&#10;The rest of the paper is organized as follows. In \ref{sec:releted} we formulate the UDA problem and review related works. Then the proposed approach CASTLE is introduced in \ref{sec:proposed}. We describe experimental settings in \ref{sec:experiment}, and then present experimental results in \ref{sec:results}. Finally, \ref{sec:conclusion} concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Domain Adaptation, Signal Processing, Speech Recognition, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2307.02351" label="2307.02351">
        <attvalues>
          <attvalue for="0" value="Online Hybrid CTC/Attention End-to-End Automatic Speech Recognition&#10;  Architecture" />
          <attvalue for="1" value="  Recently, there has been increasing progress in end-to-end automatic speech&#10;recognition (ASR) architecture, which transcribes speech to text without any&#10;pre-trained alignments. One popular end-to-end approach is the hybrid&#10;Connectionist Temporal Classification (CTC) and attention (CTC/attention) based&#10;ASR architecture. However, how to deploy hybrid CTC/attention systems for&#10;online speech recognition is still a non-trivial problem. This article&#10;describes our proposed online hybrid CTC/attention end-to-end ASR architecture,&#10;which replaces all the offline components of conventional CTC/attention ASR&#10;architecture with their corresponding streaming components. Firstly, we propose&#10;stable monotonic chunk-wise attention (sMoChA) to stream the conventional&#10;global attention, and further propose monotonic truncated attention (MTA) to&#10;simplify sMoChA and solve the training-and-decoding mismatch problem of sMoChA.&#10;Secondly, we propose truncated CTC (T-CTC) prefix score to stream CTC prefix&#10;score calculation. Thirdly, we design dynamic waiting joint decoding (DWJD)&#10;algorithm to dynamically collect the predictions of CTC and attention in an&#10;online manner. Finally, we use latency-controlled bidirectional long short-term&#10;memory (LC-BLSTM) to stream the widely-used offline bidirectional encoder&#10;network. Experiments with LibriSpeech English and HKUST Mandarin tasks&#10;demonstrate that, compared with the offline CTC/attention model, our proposed&#10;online CTC/attention model improves the real time factor in human-computer&#10;interaction services and maintains its performance with moderate degradation.&#10;To the best of our knowledge, this is the first work to provide the full-stack&#10;online solution for CTC/attention end-to-end ASR architecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.08290" label="2001.08290">
        <attvalues>
          <attvalue for="0" value="Transformer-based Online CTC/attention End-to-End Speech Recognition&#10;  Architecture" />
          <attvalue for="1" value="  Recently, Transformer has gained success in automatic speech recognition&#10;(ASR) field. However, it is challenging to deploy a Transformer-based&#10;end-to-end (E2E) model for online speech recognition. In this paper, we propose&#10;the Transformer-based online CTC/attention E2E ASR architecture, which contains&#10;the chunk self-attention encoder (chunk-SAE) and the monotonic truncated&#10;attention (MTA) based self-attention decoder (SAD). Firstly, the chunk-SAE&#10;splits the speech into isolated chunks. To reduce the computational cost and&#10;improve the performance, we propose the state reuse chunk-SAE. Sencondly, the&#10;MTA based SAD truncates the speech features monotonically and performs&#10;attention on the truncated features. To support the online recognition, we&#10;integrate the state reuse chunk-SAE and the MTA based SAD into online&#10;CTC/attention architecture. We evaluate the proposed online models on the HKUST&#10;Mandarin ASR benchmark and achieve a 23.66% character error rate (CER) with a&#10;320 ms latency. Our online model yields as little as 0.19% absolute CER&#10;degradation compared with the offline baseline, and achieves significant&#10;improvement over our prior work on Long Short-Term Memory (LSTM) based online&#10;E2E models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.00522" label="1804.00522">
        <attvalues>
          <attvalue for="0" value="A Multi-Discriminator CycleGAN for Unsupervised Non-Parallel Speech&#10;  Domain Adaptation" />
          <attvalue for="1" value="  Domain adaptation plays an important role for speech recognition models, in&#10;particular, for domains that have low resources. We propose a novel generative&#10;model based on cyclic-consistent generative adversarial network (CycleGAN) for&#10;unsupervised non-parallel speech domain adaptation. The proposed model employs&#10;multiple independent discriminators on the power spectrogram, each in charge of&#10;different frequency bands. As a result we have 1) better discriminators that&#10;focus on fine-grained details of the frequency features, and 2) a generator&#10;that is capable of generating more realistic domain-adapted spectrogram. We&#10;demonstrate the effectiveness of our method on speech recognition with gender&#10;adaptation, where the model only has access to supervised data from one gender&#10;during training, but is evaluated on the other at test time. Our model is able&#10;to achieve an average of $7.41\%$ on phoneme error rate, and $11.10\%$ word&#10;error rate relative performance improvement as compared to the baseline, on&#10;TIMIT and WSJ dataset, respectively. Qualitatively, our model also generates&#10;more natural sounding speech, when conditioned on data from the other domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07474" label="2104.07474">
        <attvalues>
          <attvalue for="0" value="EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition" />
          <attvalue for="1" value="  Self-supervised ASR-TTS models suffer in out-of-domain data conditions. Here&#10;we propose an enhanced ASR-TTS (EAT) model that incorporates two main features:&#10;1) The ASR$\rightarrow$TTS direction is equipped with a language model reward&#10;to penalize the ASR hypotheses before forwarding it to TTS. 2) In the&#10;TTS$\rightarrow$ASR direction, a hyper-parameter is introduced to scale the&#10;attention context from synthesized speech before sending it to ASR to handle&#10;out-of-domain data. Training strategies and the effectiveness of the EAT model&#10;are explored under out-of-domain data conditions. The results show that EAT&#10;reduces the performance gap between supervised and self-supervised training&#10;significantly by absolute 2.6\% and 2.7\% on Librispeech and BABEL&#10;respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07491" label="2104.07491">
        <attvalues>
          <attvalue for="0" value="Cross-domain Speech Recognition with Unsupervised Character-level&#10;  Distribution Matching" />
          <attvalue for="1" value="  End-to-end automatic speech recognition (ASR) can achieve promising&#10;performance with large-scale training data. However, it is known that domain&#10;mismatch between training and testing data often leads to a degradation of&#10;recognition accuracy. In this work, we focus on the unsupervised domain&#10;adaptation for ASR and propose CMatch, a Character-level distribution matching&#10;method to perform fine-grained adaptation between each character in two&#10;domains. First, to obtain labels for the features belonging to each character,&#10;we achieve frame-level label assignment using the Connectionist Temporal&#10;Classification (CTC) pseudo labels. Then, we match the character-level&#10;distributions using Maximum Mean Discrepancy. We train our algorithm using the&#10;self-training technique. Experiments on the Libri-Adapt dataset show that our&#10;proposed approach achieves 14.39% and 16.50% relative Word Error Rate (WER)&#10;reduction on both cross-device and cross-environment ASR. We also&#10;comprehensively analyze the different strategies for frame-level label&#10;assignment and Transformer adaptations.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the recent success of deep learning and high-quality, large-scale transcribed corpora, the performance of end-to-end (E2E) automatic speech recognition (ASR) has been significantly improved.&#10;Existing approaches are built on the i.i.d. condition that training and testing data are from the same distribution, e.g., from the same recording device or environment.&#10;However, this assumption does not always hold in reality when the training and test distributions are different.&#10;For instance, a well-trained ASR model based on PlayStation Eye recordings is likely to have deteriorated performance when it is used to recognize speech from the Matrix recordings.&#10;It is expensive and time-consuming to collecting labeled speech data from massive domains (distributions).&#10;In this paper, we tackle a more challenging scenario where there are no labeled samples available in the test data (i.e., target domain).&#10;Thus, our goal is to perform unsupervised domain adaptation (UDA)~\cite{pan2009survey,patel2015visual} to improve the cross-domain ASR performance on the unlabeled target domain using the well-labeled source data.&#10;&#10;UDA for ASR has been studied in the existing literature.&#10;Liang et al.~\cite{liang2018learning} proposed to combine data augmentation with representation matching to force the model to learn noise-invariant representations between clean speech and their augmented noisy counterparts.&#10;In~\cite{khurana2020unsupervised}, the authors were able to recover 60\% to 80\% of the word error rates (WER) on the target domain by introducing a pseudo-label filtering approach based on the model's uncertainty using dropout for ASR UDA.&#10;Recent work \cite{sun2018domain,duan2020unsupervised} used the domain-adversarial training~\cite{ganin2016domain} for speech recognition where they adversarially trained domain discriminators to distinguish the source and target samples.&#10;&#10;However, existing methods typically treated a domain as one distribution when matching distributions.&#10;They may ignore the fine-grained knowledge in cross-domain ASR tasks that can be utilized for distribution matching.&#10;These fine-grained knowledge are important to preserve the detailed distribution property of different domains, such as the characters, phoneme, and word.&#10;Ignoring these information is likely to result in unsatisfying results.&#10;This is also validated in \cite{zhu2020deep} where the images aligned in subdomains (i.e., domains split by class labels) can generally achieve better adaptation performance than traditional methods that align the whole domains.&#10;&#10;In this paper, we propose CMatch, an algorithm to match the \underline{C}haracter-level distributions for cross-domain ASR.&#10;Compared to other types such as word-level that are sparse and could lead to unstable performance, the character-level distribution is more fine-grained and easier to compute in E2E ASR models.&#10;Thus, it can preserve more fine-grained knowledge for each character than existing methods that used one distribution for all characters. gurename~\ref{fig-motiv} shows that the distances of the same characters from two domains are smaller after applying our algorithm.&#10;&#10;To enable character-level distribution matching, CMatch consists of two steps.&#10;First, we achieve frame-level label assignment to acquire the labels for each encoded frame to compute the conditional distribution.&#10;This is done by using the Connectionist Temporal Classification (CTC)~\cite{graves2006connectionist} pseudo labels. Second, CMatch can reduce the conditional distributions between each cross-domain characters using the well-defined Maximum Mean Discrepancy (MMD)~\cite{gretton2012kernel}.&#10;The above two steps are jointly optimized in a self-training framework~\cite{masumura2020sequence} in Speech-Transformer~\cite{dong2018speech}.&#10;Experiments on the Libri-Adapt dataset~\cite{mathur2020libri} show that our proposed approach achieves 14.39\% and 16.50\% relative Word Error Rate (WER) reduction on both cross-device and cross-environment ASR.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.02786" label="1806.02786">
        <attvalues>
          <attvalue for="0" value="Domain Adversarial Training for Accented Speech Recognition" />
          <attvalue for="1" value="  In this paper, we propose a domain adversarial training (DAT) algorithm to&#10;alleviate the accented speech recognition problem. In order to reduce the&#10;mismatch between labeled source domain data (&quot;standard&quot; accent) and unlabeled&#10;target domain data (with heavy accents), we augment the learning objective for&#10;a Kaldi TDNN network with a domain adversarial training (DAT) objective to&#10;encourage the model to learn accent-invariant features. In experiments with&#10;three Mandarin accents, we show that DAT yields up to 7.45% relative character&#10;error rate reduction when we do not have transcriptions of the accented speech,&#10;compared with the baseline trained on standard accent data only. We also find a&#10;benefit from DAT when used in combination with training from automatic&#10;transcriptions on the accented data. Furthermore, we find that DAT is superior&#10;to multi-task learning for accented speech recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.08922" label="2106.08922">
        <attvalues>
          <attvalue for="0" value="Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition" />
          <attvalue for="1" value="  Pseudo-labeling (PL) has been shown to be effective in semi-supervised&#10;automatic speech recognition (ASR), where a base model is self-trained with&#10;pseudo-labels generated from unlabeled data. While PL can be further improved&#10;by iteratively updating pseudo-labels as the model evolves, most of the&#10;previous approaches involve inefficient retraining of the model or intricate&#10;control of the label update. We present momentum pseudo-labeling (MPL), a&#10;simple yet effective strategy for semi-supervised ASR. MPL consists of a pair&#10;of online and offline models that interact and learn from each other, inspired&#10;by the mean teacher method. The online model is trained to predict&#10;pseudo-labels generated on the fly by the offline model. The offline model&#10;maintains a momentum-based moving average of the online model. MPL is performed&#10;in a single training process and the interaction between the two models&#10;effectively helps them reinforce each other to improve the ASR performance. We&#10;apply MPL to an end-to-end ASR model based on the connectionist temporal&#10;classification. The experimental results demonstrate that MPL effectively&#10;improves over the base model and is scalable to different semi-supervised&#10;scenarios with varying amounts of data or domain mismatch.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.04996" label="2006.04996">
        <attvalues>
          <attvalue for="0" value="Implicit Class-Conditioned Domain Alignment for Unsupervised Domain&#10;  Adaptation" />
          <attvalue for="1" value="  We present an approach for unsupervised domain adaptation---with a strong&#10;focus on practical considerations of within-domain class imbalance and&#10;between-domain class distribution shift---from a class-conditioned domain&#10;alignment perspective. Current methods for class-conditioned domain alignment&#10;aim to explicitly minimize a loss function based on pseudo-label estimations of&#10;the target domain. However, these methods suffer from pseudo-label bias in the&#10;form of error accumulation. We propose a method that removes the need for&#10;explicit optimization of model parameters from pseudo-labels directly. Instead,&#10;we present a sampling-based implicit alignment approach, where the sample&#10;selection procedure is implicitly guided by the pseudo-labels. Theoretical&#10;analysis reveals the existence of a domain-discriminator shortcut in misaligned&#10;classes, which is addressed by the proposed implicit alignment approach to&#10;facilitate domain-adversarial learning. Empirical results and ablation studies&#10;confirm the effectiveness of the proposed approach, especially in the presence&#10;of within-domain class imbalance and between-domain class distribution shift.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.09116" label="1909.09116">
        <attvalues>
          <attvalue for="0" value="Self-Training for End-to-End Speech Recognition" />
          <attvalue for="1" value="  We revisit self-training in the context of end-to-end speech recognition. We&#10;demonstrate that training with pseudo-labels can substantially improve the&#10;accuracy of a baseline model. Key to our approach are a strong baseline&#10;acoustic and language model used to generate the pseudo-labels, filtering&#10;mechanisms tailored to common errors from sequence-to-sequence models, and a&#10;novel ensemble approach to increase pseudo-label diversity. Experiments on the&#10;LibriSpeech corpus show that with an ensemble of four models and label&#10;filtering, self-training yields a 33.9% relative improvement in WER compared&#10;with a baseline trained on 100 hours of labelled data in the noisy speech&#10;setting. In the clean speech setting, self-training recovers 59.3% of the gap&#10;between the baseline and an oracle model, which is at least 93.8% relatively&#10;higher than what previous approaches can achieve.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11428" label="2010.11428">
        <attvalues>
          <attvalue for="0" value="Confidence Estimation for Attention-based Sequence-to-sequence Models&#10;  for Speech Recognition" />
          <attvalue for="1" value="  For various speech-related tasks, confidence scores from a speech recogniser&#10;are a useful measure to assess the quality of transcriptions. In traditional&#10;hidden Markov model-based automatic speech recognition (ASR) systems,&#10;confidence scores can be reliably obtained from word posteriors in decoding&#10;lattices. However, for an ASR system with an auto-regressive decoder, such as&#10;an attention-based sequence-to-sequence model, computing word posteriors is&#10;difficult. An obvious alternative is to use the decoder softmax probability as&#10;the model confidence. In this paper, we first examine how some commonly used&#10;regularisation methods influence the softmax-based confidence scores and study&#10;the overconfident behaviour of end-to-end models. Then we propose a lightweight&#10;and effective approach named confidence estimation module (CEM) on top of an&#10;existing end-to-end ASR model. Experiments on LibriSpeech show that CEM can&#10;mitigate the overconfidence problem and can produce more reliable confidence&#10;scores with and without shallow fusion of a language model. Further analysis&#10;shows that CEM generalises well to speech from a moderately mismatched domain&#10;and can potentially improve downstream tasks such as semi-supervised learning.&#10;" />
          <attvalue for="2" value="&#10;\postsec&#10;Confidence scores have been an intrinsic part of automatic speech recognition (ASR) systems~\cite{Wessel2001ConfidenceMF,Jiang2005ConfidenceMF,Yu2011CalibrationOC}. Many speech-related applications depend on high-quality confidence scores to mitigate error from speech recognisers. For example, in semi-supervised learning and active learning, utterances with highly confident hypotheses are selected to further improve ASR performance~\cite{Chan2004ImprovingBN,Tr2005CombiningAA,Riccardi2005ActiveLT}. Confidence scores are also used in dialogue systems where queries with low confidence may be returned to users for clarification~\cite{Tr2005CombiningAA}. As an indication of ASR uncertainty, confidence scores can play an role in speaker adaptation\cite{Uebel2001SpeakerAU}, and system combination~\cite{Evermann2000PosteriorPD}.&#10;&#10;In conventional HMM-based systems, reliable confidence scores can be easily obtained by computing word posterior probabilities from compact representations of the hypotheses space, \eg lattices or confusion networks~\cite{Mangu2000FindingCI,Evermann2000PosteriorPD}. Improved confidence estimation can be achieved by using model-based approaches, such as conditional random fields\cite{Seigel2011CombiningIS}, recurrent neural networks~\cite{Kalgaonkar2015EstimatingCS,Ragni2018ConfidenceEA} and graph neural networks~\cite{Li2019BidirectionalLR}, or leveraging more related information including phonetics, word/phone duration and language models~\cite{Jiang2005ConfidenceMF,Kastanos2020ConfidenceEF}.&#10;&#10;Recently, end-to-end speech recognition has achieved promising performance over the conventional systems~\cite{Chiu2018StateoftheArtSR}. ``End-to-end systems'' refers to end-to-end differentiable and trainable neural networks, in contrast to modular systems with separate acoustic models, language models and token passing decoders. As end-to-end speech recognition has various modelling and engineering advantages, they are becoming more widely adopted~\cite{He2019StreamingES}. One class of the end-to-end systems is attention-based sequence-to-sequence models~\cite{Chorowski2015AttentionBasedMF,Chan2016ListenAA}. The auto-regressive decoder in end-to-end systems implies that compact representations like lattices cannot be constructed for attention-based models, except using specific decoder architectures and heuristics~\cite{Zapotoczny2019LatticeGI}. Consequently, computing word posteriors in the end-to-end hypothesis space becomes prohibitively expensive. A greedy approximation would be taking the softmax probability from each step of the decoder as the confidence scores for each token~\cite{Park2020ImprovedNS}. However, the quality of confidence estimation by softmax probabilities may be very poor~\cite{Hendrycks2017ABF}. &#10;&#10;This issue is illustrated by gdot{teaser} which is a plot used for data selection for semi-supervised learning~\cite{Park2020ImprovedNS}. The plot shows that if all utterances are selected, then a conventional system and an attention-based sequence-to-sequence model have similar performance (7$\sim$8\% WER). As the confidence threshold increases, the WER of the conventional system monotonically decreases. However, for the end-to-end model, a higher threshold does not always mean a reduced WER. The spike in gdot{teaser} indicates that the end-to-end model is overconfident based on softmax probabilities.&#10;&#10;To address the overconfidence issue, the confidence estimation module (CEM) is proposed for attention-based sequence-to-sequence models in \sectdot{cem}. \twosect{setup}{exp} describes the setup and experiments that demonstrate the effectiveness of CEM. \sect{analysis} shows the generalisation performance and impact on downstream tasks of the proposed method. Conclusions are drawn in \sectdot{conclusion}.&#10; " />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.12081" label="2104.12081">
        <attvalues>
          <attvalue for="0" value="How Well Does Self-Supervised Pre-Training Perform with Streaming Data?" />
          <attvalue for="1" value="  Prior works on self-supervised pre-training focus on the joint training&#10;scenario, where massive unlabeled data are assumed to be given as input all at&#10;once, and only then is a learner trained. Unfortunately, such a problem setting&#10;is often impractical if not infeasible since many real-world tasks rely on&#10;sequential learning, e.g., data are decentralized or collected in a streaming&#10;fashion. In this paper, we conduct the first thorough and dedicated&#10;investigation on self-supervised pre-training with streaming data, aiming to&#10;shed light on the model behavior under this overlooked setup. Specifically, we&#10;pre-train over 500 models on four categories of pre-training streaming data&#10;from ImageNet and DomainNet and evaluate them on three types of downstream&#10;tasks and 12 different downstream datasets. Our studies show that, somehow&#10;beyond our expectation, with simple data replay or parameter regularization,&#10;sequential self-supervised pre-training turns out to be an efficient&#10;alternative for joint pre-training, as the performances of the former are&#10;mostly on par with those of the latter. Moreover, catastrophic forgetting, a&#10;common issue in sequential supervised learning, is much alleviated in&#10;sequential self-supervised learning (SSL), which is well justified through our&#10;comprehensive empirical analysis on representations and the sharpness of minima&#10;in the loss landscape. Our findings, therefore, suggest that, in practice, for&#10;SSL, the cumbersome joint training can be replaced mainly by sequential&#10;learning, which in turn enables a much broader spectrum of potential&#10;application scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.01497" label="1911.01497">
        <attvalues>
          <attvalue for="0" value="On Compositionality in Neural Machine Translation" />
          <attvalue for="1" value="  We investigate two specific manifestations of compositionality in Neural&#10;Machine Translation (NMT) : (1) Productivity - the ability of the model to&#10;extend its predictions beyond the observed length in training data and (2)&#10;Systematicity - the ability of the model to systematically recombine known&#10;parts and rules. We evaluate a standard Sequence to Sequence model on tests&#10;designed to assess these two properties in NMT. We quantitatively demonstrate&#10;that inadequate temporal processing, in the form of poor encoder&#10;representations is a bottleneck for both Productivity and Systematicity. We&#10;propose a simple pre-training mechanism which alleviates model performance on&#10;the two properties and leads to a significant improvement in BLEU scores.&#10;" />
          <attvalue for="2" value="&#10;Sequence-to-sequence (Seq2Seq) networks have achieved impressive results \cite{bahdanau2014neural, sutskever2014sequence, neubig2017neural,vinyals2015neural,venugopalan2015sequence,karpathy2015deep} on a variety of problems within natural language processing. However, their inability to handle long sentences \cite{length1} as well as a lack of systematic generalizability \cite{lake2017still} questions the ability of seq2seq networks to model compositionality in natural language. \cite{hupkes2019compositionality} provide tests corresponding to different ineterpretations of compositionality. In this work, we investigate two such properties, which are intrinsic to the way that humans utilize language namely, - (1) Productivity: the ability to generalize beyond the observed length and (2) Systematicity: the ability to recombine knows parts and rules. Loosely, Productivity could be regarded as &quot;unbounded&quot; application of known rules, while systematicity implies arbitrary recombinations of known rules.&#10;&#10;In the coming sections, we first describe the quantitative tests to evaluate productivity and systematicity in Neural Machine Translation (NMT). For both productivity and systematicity, we first show that a standard Seq2Seq model performs poorly under the tests for evaluating productivity and systematicity. Further, we investigate the encoder representations, which constitute the first potential bottleneck for compositionality in the sequence to sequence transduction pipeline as a potential cause for poor performance on these tests. We quantitatively demonstrate the weaknesses of encoder representations and propose a simple pre-training scheme to improve the performance on the two properties, leading to considerable improvements in BLEU scores.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Compositionality Analysis, Model Performance Optimization, Artificial Intelligence, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1409.1259" label="1409.1259">
        <attvalues>
          <attvalue for="0" value="On the Properties of Neural Machine Translation: Encoder-Decoder&#10;  Approaches" />
          <attvalue for="1" value="  Neural machine translation is a relatively new approach to statistical&#10;machine translation based purely on neural networks. The neural machine&#10;translation models often consist of an encoder and a decoder. The encoder&#10;extracts a fixed-length representation from a variable-length input sentence,&#10;and the decoder generates a correct translation from this representation. In&#10;this paper, we focus on analyzing the properties of the neural machine&#10;translation using two models; RNN Encoder--Decoder and a newly proposed gated&#10;recursive convolutional neural network. We show that the neural machine&#10;translation performs relatively well on short sentences without unknown words,&#10;but its performance degrades rapidly as the length of the sentence and the&#10;number of unknown words increase. Furthermore, we find that the proposed gated&#10;recursive convolutional network learns a grammatical structure of a sentence&#10;automatically.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.10752" label="2108.10752">
        <attvalues>
          <attvalue for="0" value="Generalizing RNN-Transducer to Out-Domain Audio via Sparse&#10;  Self-Attention Layers" />
          <attvalue for="1" value="  Recurrent neural network transducer (RNN-T) is an end-to-end speech&#10;recognition framework converting input acoustic frames into a character&#10;sequence. The state-of-the-art encoder network for RNN-T is the Conformer,&#10;which can effectively model the local-global context information via its&#10;convolution and self-attention layers. Although Conformer RNN-T has shown&#10;outstanding performance, most studies have been verified in the setting where&#10;the train and test data are drawn from the same domain. The domain mismatch&#10;problem for Conformer RNN-T has not been intensively investigated yet, which is&#10;an important issue for the product-level speech recognition system. In this&#10;study, we identified that fully connected self-attention layers in the&#10;Conformer caused high deletion errors, specifically in the long-form out-domain&#10;utterances. To address this problem, we introduce sparse self-attention layers&#10;for Conformer-based encoder networks, which can exploit local and generalized&#10;global information by pruning most of the in-domain fitted global connections.&#10;Also, we propose a state reset method for the generalization of the prediction&#10;network to cope with long-form utterances. Applying proposed methods to an&#10;out-domain test, we obtained 27.6% relative character error rate (CER)&#10;reduction compared to the fully connected self-attention layer-based&#10;Conformers.&#10;" />
          <attvalue for="2" value="&#10;&#10;The recent research focus of automatic speech recognition (ASR) is end-to-end (E2E) frameworks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18,Graves-CTC06,Graves-TEE14,Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20,Daniel-SA19,Christoph-RWTH19,Moritz-CMR21}, which can directly map incoming speech signals into characters \cite{Amodei-DS216} or word targets \cite{Hagen-NSR17,Audhkhasi-DAW17}. The E2E frameworks include encoder--decoder networks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18}, connectionist temporal classification (CTC) \cite{Graves-CTC06,Graves-TEE14}, and recurrent neural network transducer (RNN-T) \cite{Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20}. Considering the various E2E frameworks, RNN-T based approaches have shown promising results based on word error rate (WER) and decoding speed. Most of studies \cite{Anmol-Conformer20,Xiong-ECP21,shim-URS22,Burchi-EC21} on RNN-T have been conducted in the same domain, such as Librispeech \cite{Panayotov-Librispeech15}; however, the robust performance across the different domains must be involved for the production-level ASR. This indicates that it is important to address the domain-mismatch problem between training and inference.&#10;&#10;In \cite{Chiu-RMF21}, the domain mismatch problem for RNN-T was intensively investigated. First, the encoder network in the RNN-T suffers from overfitting to the training domain, referred to as the in-domain. Second, the RNN-T is vulnerable to long-form utterances during inference because it is generally trained on short segments. These two problems cause high deletion errors when decoding is conducted on out-domain long-form utterances. To overcome these problems, multiple regularization methods (e.g., variational weight noise \cite{Graves-PVI11}) and dynamic overlapping inference (DOI) which splits long-form utterances into several overlapping segments, have been proposed. Although applying the methods \cite{Chiu-RMF21} improved the WER for out-domain long-form utterances, the investigation was conducted only on long short-term memory (LSTM)-based encoder networks. Furthermore, the long segment length for DOI ($&gt;20$ s) was not investigated although the LSTM was designed for long-term context information \cite{Hochreiter-LSTM97}.&#10;&#10;In \cite{Li-ABF21,Xiong-ECP21}, a Conformer-based encoder network was proposed for the RNN-T as the Conformer can effectively model the local-global context information through its convolution and self-attention layers \cite{Anmol-Conformer20}, showing promising performance. However, the domain mismatch problem for Conformer-based encoder network has not been intensively investigated yet.&#10;&#10;In this study, we propose a generalization strategy for RNN-T with a Conformer-based encoder network. The main contributions of this study are as follows. (i) The sparse self-attention layers that can exploit both local and global connections are designed for the Conformer: the generalized global connections robust to the domain mismatch problem are identified by pruning most of the redundant global connections while conserving the important global connections considered by the model. &#10;(ii) The state reset method is proposed to cope with long-form utterances by re-initializing the LSTM states of the prediction network when silence is detected during the decoding phase. Considering the experimental evaluations, we found that combining local and sparse global connections outperformed the local connections alone, and the state reset method showed further improvement in the out-domain test set.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Engineering, Domain Adaptation, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2104.02858" label="2104.02858">
        <attvalues>
          <attvalue for="0" value="Capturing Multi-Resolution Context by Dilated Self-Attention" />
          <attvalue for="1" value="  Self-attention has become an important and widely used neural network&#10;component that helped to establish new state-of-the-art results for various&#10;applications, such as machine translation and automatic speech recognition&#10;(ASR). However, the computational complexity of self-attention grows&#10;quadratically with the input sequence length. This can be particularly&#10;problematic for applications such as ASR, where an input sequence generated&#10;from an utterance can be relatively long. In this work, we propose a&#10;combination of restricted self-attention and a dilation mechanism, which we&#10;refer to as dilated self-attention. The restricted self-attention allows&#10;attention to neighboring frames of the query at a high resolution, and the&#10;dilation mechanism summarizes distant information to allow attending to it with&#10;a lower resolution. Different methods for summarizing distant frames are&#10;studied, such as subsampling, mean-pooling, and attention-based pooling. ASR&#10;results demonstrate substantial improvements compared to restricted&#10;self-attention alone, achieving similar results compared to full-sequence based&#10;self-attention with a fraction of the computational costs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.09975" label="1610.09975">
        <attvalues>
          <attvalue for="0" value="Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large&#10;  Vocabulary Speech Recognition" />
          <attvalue for="1" value="  We present results that show it is possible to build a competitive, greatly&#10;simplified, large vocabulary continuous speech recognition system with whole&#10;words as acoustic units. We model the output vocabulary of about 100,000 words&#10;directly using deep bi-directional LSTM RNNs with CTC loss. The model is&#10;trained on 125,000 hours of semi-supervised acoustic training data, which&#10;enables us to alleviate the data sparsity problem for word models. We show that&#10;the CTC word models work very well as an end-to-end all-neural speech&#10;recognition model without the use of traditional context-dependent sub-word&#10;phone units that require a pronunciation lexicon, and without any language&#10;model removing the need to decode. We demonstrate that the CTC word models&#10;perform better than a strong, more complex, state-of-the-art baseline with&#10;sub-word units.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.09236" label="2106.09236">
        <attvalues>
          <attvalue for="0" value="Efficient Conformer with Prob-Sparse Attention Mechanism for&#10;  End-to-EndSpeech Recognition" />
          <attvalue for="1" value="  End-to-end models are favored in automatic speech recognition (ASR) because&#10;of their simplified system structure and superior performance. Among these&#10;models, Transformer and Conformer have achieved state-of-the-art recognition&#10;accuracy in which self-attention plays a vital role in capturing important&#10;global information. However, the time and memory complexity of self-attention&#10;increases squarely with the length of the sentence. In this paper, a&#10;prob-sparse self-attention mechanism is introduced into Conformer to sparse the&#10;computing process of self-attention in order to accelerate inference speed and&#10;reduce space consumption. Specifically, we adopt a Kullback-Leibler divergence&#10;based sparsity measurement for each query to decide whether we compute the&#10;attention function on this query. By using the prob-sparse attention mechanism,&#10;we achieve impressively 8% to 45% inference speed-up and 15% to 45% memory&#10;usage reduction of the self-attention module of Conformer Transducer while&#10;maintaining the same level of error rate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.03271" label="2005.03271">
        <attvalues>
          <attvalue for="0" value="RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and&#10;  Solutions" />
          <attvalue for="1" value="  In recent years, all-neural end-to-end approaches have obtained&#10;state-of-the-art results on several challenging automatic speech recognition&#10;(ASR) tasks. However, most existing works focus on building ASR models where&#10;train and test data are drawn from the same domain. This results in poor&#10;generalization characteristics on mismatched-domains: e.g., end-to-end models&#10;trained on short segments perform poorly when evaluated on longer utterances.&#10;In this work, we analyze the generalization properties of streaming and&#10;non-streaming recurrent neural network transducer (RNN-T) based end-to-end&#10;models in order to identify model components that negatively affect&#10;generalization performance. We propose two solutions: combining multiple&#10;regularization techniques during training, and using dynamic overlapping&#10;inference. On a long-form YouTube test set, when the nonstreaming RNN-T model&#10;is trained with shorter segments of data, the proposed combination improves&#10;word error rate (WER) from 22.3% to 14.8%; when the streaming RNN-T model&#10;trained on short Search queries, the proposed techniques improve WER on the&#10;YouTube set from 67.0% to 25.3%. Finally, when trained on Librispeech, we find&#10;that dynamic overlapping inference improves WER on YouTube from 99.8% to 33.0%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.17190" label="2203.17190">
        <attvalues>
          <attvalue for="0" value="Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme&#10;  Representations for Text to Speech" />
          <attvalue for="1" value="  Recently, leveraging BERT pre-training to improve the phoneme encoder in text&#10;to speech (TTS) has drawn increasing attention. However, the works apply&#10;pre-training with character-based units to enhance the TTS phoneme encoder,&#10;which is inconsistent with the TTS fine-tuning that takes phonemes as input.&#10;Pre-training only with phonemes as input can alleviate the input mismatch but&#10;lack the ability to model rich representations and semantic information due to&#10;limited phoneme vocabulary. In this paper, we propose MixedPhoneme BERT, a&#10;novel variant of the BERT model that uses mixed phoneme and sup-phoneme&#10;representations to enhance the learning capability. Specifically, we merge the&#10;adjacent phonemes into sup-phonemes and combine the phoneme sequence and the&#10;merged sup-phoneme sequence as the model input, which can enhance the model&#10;capacity to learn rich contextual representations. Experiment results&#10;demonstrate that our proposed Mixed-Phoneme BERT significantly improves the TTS&#10;performance with 0.30 CMOS gain compared with the FastSpeech 2 baseline. The&#10;Mixed-Phoneme BERT achieves 3x inference speedup and similar voice quality to&#10;the previous TTS pre-trained model PnG BERT&#10;" />
          <attvalue for="2" value=" &#10;&#10;In recent years, neural text to speech (TTS) ~\cite{tan2021survey,tan2022naturalspeech,wang2017tacotron,shen2018natural,ren2019fastspeech,ren2020fastspeech, zhang2021study} has demonstrated significant successes in producing natural-sounding speech. Specifically, the non-autoregressive TTS systems\cite{ren2019fastspeech} have received increasing attention due to their advanced ability in generating stable mel-spectrograms with fast speed. However, existing non-autoregressive TTS still retains some flaws, like the ``one-to-many'' mapping between text and speech. In \cite{ren2020fastspeech, elias21_interspeech, zhang21u_interspeech}, the information from the ground truth speech data, e.g. pitch, duration, is incorporated in model training to alleviate the ``one-to-many'' problem. It was noted that the generated speech still tends to carry flat prosody since the phoneme sequence does not contain adequate information for predicting natural prosody \cite{kenter2020improving}.&#10;&#10;In order to handle this problem, some works have tried to enrich the phoneme sequence input with syntactic information and linguistic features~\cite{taylor2009text, zen2015unidirectional,wu2016merlin}. However, this strategy has two main disadvantages: i) it needs several separated text pre-processing modules (e.g., POS tagger and syntax parser), possibly leading to error propagation from one module to the subsequent ones; ii) designing and labelling linguistic features are usually time-consuming tasks that require language expertise~\cite{zhang2020learning}. Recently, leveraging contextual representations learned from unlabeled text data to improve the TTS model has become a rising topic \cite{wang2015word,chung2019semi,hayashi2019pre,kenter2020improving,xu2021improving}, which is also the focus of our paper.&#10;&#10;Pre-trained language models (e.g., BERT~\cite{devlin2019bert}) have achieved state-of-the-art performance in solving natural language processing tasks. Recent works~\cite{hayashi2019pre,xiao2020improving,kenter2020improving,xu2021improving} have applied the BERT model as an auxiliary encoder for the TTS system. The auxiliary BERT encoder extracts additional text features for character-based units (e.g., character\cite{el2020characterbert}, subword\cite{liu2019roberta, devlin2019bert}), which enables the TTS system to generate speech with better pronunciation and expressiveness. Those methods have to face the challenge that the pre-training with the character-based units is inconsistent with the TTS fine-tuning taking pure phoneme as the input. The inconsistency will bring several problems: 1) The alignment between phoneme and character information might be unstable(e.g., misalignment between phoneme and character sequence), making the TTS phoneme encoder fail to leverage the pre-trained text features. 2) The auxiliary BERT encoder with text input will also bring longer training/inference time and more computing resources. Besides, using phoneme and the auxiliary encoder simultaneously will result in larger model parameters. Some works \cite{kastner2019representation, jia21_interspeech} attempt to enhance the TTS phoneme encoder with character information directly rather than introducing an auxiliary BERT model. Since the model still needs character-based units as an extra input, inconsistency and corresponding drawbacks still exist.&#10;&#10;So what if we pre-train the phoneme encoder with only the phoneme as the input? We note that the size of the phoneme dictionary is only around 200, and directly using such a small dictionary would not convey contextual semantic information effectively\cite{ding2019call}. To enhance the representation capacity of model input and avoid the problem of inconsistency between pre-training and fine-tuning, we propose Mixed-Phoneme BERT, a novel variant of the BERT model to handle TTS tasks. The Mixed-Phoneme BERT model merges phoneme and sup-phoneme sequences into a new sequence. The sup-phoneme tokens are obtained by applying the learnt Byte-Pair Encoding (BPE) ~\cite{sennrich2016neural} rules to words. Compared with the phoneme dictionary, which is usually too small, the dictionary size of sup-phoneme is much larger, producing representations with better semantics. The mixed-Phoneme BERT employs Masked Language Modelling objective for pre-training on large-scale unlabeled text corpora. In order to prevent the information leakage problem, a pre-processed data alignment and consistent masking strategy are introduced, which require our Mixed-Phoneme BERT model to predict the masked sup-phoneme tokens and all phoneme tokens corresponding to the masked sup-phoneme tokens simultaneously. The pre-trained Mixed-Phoneme BERT serves as the phoneme encoder for TTS fine-tuning. Experiment results indicate that the Mixed-Phoneme BERT model can significantly improve the TTS performance over the baseline Fastspeech 2 model and alleviate the flat prosody problem. Ablation studies also validate the effectiveness of using the sup-phoneme information in the input representations. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Speech Processing, Linguistics, Text to Speech, Phoneme Encoding, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2205.04421" label="2205.04421">
        <attvalues>
          <attvalue for="0" value="NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level&#10;  Quality" />
          <attvalue for="1" value="  Text to speech (TTS) has made rapid progress in both academia and industry in&#10;recent years. Some questions naturally arise that whether a TTS system can&#10;achieve human-level quality, how to define/judge that quality and how to&#10;achieve it. In this paper, we answer these questions by first defining the&#10;human-level quality based on the statistical significance of subjective measure&#10;and introducing appropriate guidelines to judge it, and then developing a TTS&#10;system called NaturalSpeech that achieves human-level quality on a benchmark&#10;dataset. Specifically, we leverage a variational autoencoder (VAE) for&#10;end-to-end text to waveform generation, with several key modules to enhance the&#10;capacity of the prior from text and reduce the complexity of the posterior from&#10;speech, including phoneme pre-training, differentiable duration modeling,&#10;bidirectional prior/posterior modeling, and a memory mechanism in VAE.&#10;Experiment evaluations on popular LJSpeech dataset show that our proposed&#10;NaturalSpeech achieves -0.01 CMOS (comparative mean opinion score) to human&#10;recordings at the sentence level, with Wilcoxon signed rank test at p-level p&#10;&gt;&gt; 0.05, which demonstrates no statistically significant difference from human&#10;recordings for the first time on this dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.10128" label="1808.10128">
        <attvalues>
          <attvalue for="0" value="Semi-Supervised Training for Improving Data Efficiency in End-to-End&#10;  Speech Synthesis" />
          <attvalue for="1" value="  Although end-to-end text-to-speech (TTS) models such as Tacotron have shown&#10;excellent results, they typically require a sizable set of high-quality &lt;text,&#10;audio&gt; pairs for training, which are expensive to collect. In this paper, we&#10;propose a semi-supervised training framework to improve the data efficiency of&#10;Tacotron. The idea is to allow Tacotron to utilize textual and acoustic&#10;knowledge contained in large, publicly-available text and speech corpora.&#10;Importantly, these external data are unpaired and potentially noisy.&#10;Specifically, first we embed each word in the input text into word vectors and&#10;condition the Tacotron encoder on them. We then use an unpaired speech corpus&#10;to pre-train the Tacotron decoder in the acoustic domain. Finally, we fine-tune&#10;the model using available paired data. We demonstrate that the proposed&#10;framework enables Tacotron to generate intelligible speech using less than half&#10;an hour of paired training data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.05161" label="2011.05161">
        <attvalues>
          <attvalue for="0" value="Improving Prosody Modelling with Cross-Utterance BERT Embeddings for&#10;  End-to-end Speech Synthesis" />
          <attvalue for="1" value="  Despite prosody is related to the linguistic information up to the discourse&#10;structure, most text-to-speech (TTS) systems only take into account that within&#10;each sentence, which makes it challenging when converting a paragraph of texts&#10;into natural and expressive speech. In this paper, we propose to use the text&#10;embeddings of the neighboring sentences to improve the prosody generation for&#10;each utterance of a paragraph in an end-to-end fashion without using any&#10;explicit prosody features. More specifically, cross-utterance (CU) context&#10;vectors, which are produced by an additional CU encoder based on the sentence&#10;embeddings extracted by a pre-trained BERT model, are used to augment the input&#10;of the Tacotron2 decoder. Two types of BERT embeddings are investigated, which&#10;leads to the use of different CU encoder structures. Experimental results on a&#10;Mandarin audiobook dataset and the LJ-Speech English audiobook dataset&#10;demonstrate the use of CU information can improve the naturalness and&#10;expressiveness of the synthesized speech. Subjective listening testing shows&#10;most of the participants prefer the voice generated using the CU encoder over&#10;that generated using standard Tacotron2. It is also found that the prosody can&#10;be controlled indirectly by changing the neighbouring sentences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.07240" label="1811.07240">
        <attvalues>
          <attvalue for="0" value="Representation Mixing for TTS Synthesis" />
          <attvalue for="1" value="  Recent character and phoneme-based parametric TTS systems using deep learning&#10;have shown strong performance in natural speech generation. However, the choice&#10;between character or phoneme input can create serious limitations for practical&#10;deployment, as direct control of pronunciation is crucial in certain cases. We&#10;demonstrate a simple method for combining multiple types of linguistic&#10;information in a single encoder, named representation mixing, enabling flexible&#10;choice between character, phoneme, or mixed representations during inference.&#10;Experiments and user studies on a public audiobook corpus show the efficacy of&#10;our approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.10453" label="1905.10453">
        <attvalues>
          <attvalue for="0" value="A Call for Prudent Choice of Subword Merge Operations in Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Most neural machine translation systems are built upon subword units&#10;extracted by methods such as Byte-Pair Encoding (BPE) or wordpiece. However,&#10;the choice of number of merge operations is generally made by following&#10;existing recipes. In this paper, we conduct a systematic exploration on&#10;different numbers of BPE merge operations to understand how it interacts with&#10;the model architecture, the strategy to build vocabularies and the language&#10;pair. Our exploration could provide guidance for selecting proper BPE&#10;configurations in the future. Most prominently: we show that for LSTM-based&#10;architectures, it is necessary to experiment with a wide range of different BPE&#10;operations as there is no typical optimal BPE configuration, whereas for&#10;Transformer architectures, smaller BPE size tends to be a typically optimal&#10;choice. We urge the community to make prudent choices with subword merge&#10;operations, as our experiments indicate that a sub-optimal BPE configuration&#10;alone could easily reduce the system performance by 3-4 BLEU points.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14901" label="2305.14901">
        <attvalues>
          <attvalue for="0" value="Chain-of-Questions Training with Latent Answers for Robust Multistep&#10;  Question Answering" />
          <attvalue for="1" value="  We train a language model (LM) to robustly answer multistep questions by&#10;generating and answering sub-questions. We propose Chain-of-Questions, a&#10;framework that trains a model to generate sub-questions and sub-answers one at&#10;a time by leveraging human annotated question decomposition meaning&#10;representation (QDMR). The key technical challenge is that QDMR only contains&#10;sub-questions but not answers to those sub-questions, so we treat sub-answers&#10;as latent variables and optimize them using a novel dynamic mixture of Hard-EM&#10;and MAPO. Chain-of-Questions greatly outperforms strong neuro-symbolic methods&#10;by 9.0 F1 on DROP contrast set, and outperforms GPT-3.5 by 24.3 F1 on HOTPOTQA&#10;adversarial set, thus demonstrating the effectiveness and robustness of our&#10;framework.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multistep question answering (QA) poses a reasoning challenge that current state-of-the-art QA models have not fully addressed. &#10;Strong fine-tuned QA models like UnifiedQA~\cite{2020unifiedqa} can achieve impressive results on various QA tasks through multitask training, but exhibit subpar performance on multistep reasoning.&#10;Moreover, because some multistep reasoning benchmarks contain annotation artifacts or reasoning shortcuts~\cite{jiang-bansal-2019-avoiding}, dedicated models trained on these benchmarks often have much lower F1 performance on contrast sets~\cite{gardner-etal-2020-evaluating} and adversarial sets~\cite{Schlegel2020SemanticsAM}, indicating their lack of robustness.&#10;&#10;Prior research has attempted to tackle this challenge with various question decomposition strategies to explicitly incorporate reasoning chains into the question answering process. &#10;However, as we show in our experiments, existing methods~\cite{andor-etal-2019-giving, Chen2020NeuralSR} that perform explicit reasoning steps still suffer from robustness issues.&#10;Moreover, multi-step reasoning methods are often engineered for a specific domain or type of multistep QA \cite{fu-etal-2021-decomposing-complex, perez-etal-2020-unsupervised}, and thus cannot be easily extended to other multistep QA settings. Prompting methods \cite{Chen2020NeuralSR, dua-etal-2022-successive} have shown promise in generating multistep solutions to questions, but they require very large language models (LMs) as well as careful prompt engineering, and still lag behind fine-tuned methods~\cite{OpenAI2023GPT4TR}.&#10;&#10;To develop a robust multistep QA system, we propose a novel framework, Chain-of-Questions training with latent answers.&#10;Our framework trains a model to generate sub-questions and their corresponding sub-answers one at a time, as shown in Fig.~\ref{fig:teaser}, then aggregates those sub-answers to answer the original question.&#10;To define an appropriate set of sub-questions, we use question decomposition meaning representation (QDMR), an existing dataset with human-annotated sub-questions for questions from multiple multistep QA benchmarks. &#10;While QDMR is helpful, it only contains annotated sub-questions, not sub-answers, which makes training a QA system to generate sub-answers technically challenging.&#10;We view the sub-answers in the intermediate steps as latent variables, and apply Hard-EM~\cite{Neal1998AVO} to optimize these latent variables during training.&#10;To further improve performance, we use a memory buffer to store trajectories with high F1 score, inspired by Memory-Augmented Policy Optimization (MAPO;~\cite{NEURIPS2018_f4e369c0}), previously used for semantic parsing.&#10;Because starting with MAPO alone does not converge well, we design a dynamic loss function that combines the Hard-EM and MAPO objectives for fast improvement at the beginning and better final convergence.&#10;&#10;We conduct experiments on DROP~\cite{dua-etal-2019-drop}, HotpotQA~\cite{yang-etal-2018-hotpotqa}, and their contrast and adversarial sets to evaluate the performance of our proposed Chain-of-Questions framework. On the contrast set of DROP, Chain-of-Questions outperforms neuro-symbolic baselines by 9.0 on F1 score, and outperforms Chain-of-Thought on GPT-3.5 by 16.8 despite using a much smaller model (T5-Large, 770M parameters).&#10;On the adversarial set of HotpotQA, Chain-of-Questions outperforms Longformer by 5.5 on F1 score, and outperforms Chain-of-Thought on GPT-3.5 by 24.3.&#10;Our experimental results demonstrate that Chain-of-Questions successfully leverages existing QDMR annotations to train an effective and robust multistep QA model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Neuro-Symbolic Methods, Linguistics, Cognitive Science, Language Model Training, Question Decomposition, Mathematics" />
        </attvalues>
      </node>
      <node id="1906.07132" label="1906.07132">
        <attvalues>
          <attvalue for="0" value="Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and&#10;  Model Development for Multi-Hop QA" />
          <attvalue for="1" value="  Multi-hop question answering requires a model to connect multiple pieces of&#10;evidence scattered in a long context to answer the question. In this paper, we&#10;show that in the multi-hop HotpotQA (Yang et al., 2018) dataset, the examples&#10;often contain reasoning shortcuts through which models can directly locate the&#10;answer by word-matching the question with a sentence in the context. We&#10;demonstrate this issue by constructing adversarial documents that create&#10;contradicting answers to the shortcut but do not affect the validity of the&#10;original answer. The performance of strong baseline models drops significantly&#10;on our adversarial evaluation, indicating that they are indeed exploiting the&#10;shortcuts rather than performing multi-hop reasoning. After adversarial&#10;training, the baseline's performance improves but is still limited on the&#10;adversarial evaluation. Hence, we use a control unit that dynamically attends&#10;to the question at different reasoning hops to guide the model's multi-hop&#10;reasoning. We show that this 2-hop model trained on the regular data is more&#10;robust to the adversaries than the baseline model. After adversarial training,&#10;this 2-hop model not only achieves improvements over its counterpart trained on&#10;regular data, but also outperforms the adversarially-trained 1-hop baseline. We&#10;hope that these insights and initial improvements will motivate the development&#10;of new models that combine explicit compositional reasoning with adversarial&#10;training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.02709" label="2004.02709">
        <attvalues>
          <attvalue for="0" value="Evaluating Models' Local Decision Boundaries via Contrast Sets" />
          <attvalue for="1" value="  Standard test sets for supervised learning evaluate in-distribution&#10;generalization. Unfortunately, when a dataset has systematic gaps (e.g.,&#10;annotation artifacts), these evaluations are misleading: a model can learn&#10;simple decision rules that perform well on the test set but do not capture a&#10;dataset's intended capabilities. We propose a new annotation paradigm for NLP&#10;that helps to close systematic gaps in the test data. In particular, after a&#10;dataset is constructed, we recommend that the dataset authors manually perturb&#10;the test instances in small but meaningful ways that (typically) change the&#10;gold label, creating contrast sets. Contrast sets provide a local view of a&#10;model's decision boundary, which can be used to more accurately evaluate a&#10;model's true linguistic capabilities. We demonstrate the efficacy of contrast&#10;sets by creating them for 10 diverse NLP datasets (e.g., DROP reading&#10;comprehension, UD parsing, IMDb sentiment analysis). Although our contrast sets&#10;are not explicitly adversarial, model performance is significantly lower on&#10;them than on the original test sets---up to 25\% in some cases. We release our&#10;contrast sets as new evaluation benchmarks and encourage future dataset&#10;construction efforts to follow similar annotation processes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.04056" label="2012.04056">
        <attvalues>
          <attvalue for="0" value="Semantics Altering Modifications for Evaluating Comprehension in Machine&#10;  Reading" />
          <attvalue for="1" value="  Advances in NLP have yielded impressive results for the task of machine&#10;reading comprehension (MRC), with approaches having been reported to achieve&#10;performance comparable to that of humans. In this paper, we investigate whether&#10;state-of-the-art MRC models are able to correctly process Semantics Altering&#10;Modifications (SAM): linguistically-motivated phenomena that alter the&#10;semantics of a sentence while preserving most of its lexical surface form. We&#10;present a method to automatically generate and align challenge sets featuring&#10;original and altered examples. We further propose a novel evaluation&#10;methodology to correctly assess the capability of MRC systems to process these&#10;examples independent of the data they were optimised on, by discounting for&#10;effects introduced by domain shift. In a large-scale empirical study, we apply&#10;the methodology in order to evaluate extractive MRC models with regard to their&#10;capability to correctly process SAM-enriched data. We comprehensively cover 12&#10;different state-of-the-art neural architecture configurations and four training&#10;datasets and find that -- despite their well-known remarkable performance --&#10;optimised models consistently struggle to correctly process semantically&#10;altered data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.00109" label="1909.00109">
        <attvalues>
          <attvalue for="0" value="Giving BERT a Calculator: Finding Operations and Arguments with Reading&#10;  Comprehension" />
          <attvalue for="1" value="  Reading comprehension models have been successfully applied to extractive&#10;text answers, but it is unclear how best to generalize these models to&#10;abstractive numerical answers. We enable a BERT-based reading comprehension&#10;model to perform lightweight numerical reasoning. We augment the model with a&#10;predefined set of executable 'programs' which encompass simple arithmetic as&#10;well as extraction. Rather than having to learn to manipulate numbers directly,&#10;the model can pick a program and execute it. On the recent Discrete Reasoning&#10;Over Passages (DROP) dataset, designed to challenge reading comprehension&#10;models, we show a 33% absolute improvement by adding shallow programs. The&#10;model can learn to predict new operations when appropriate in a math word&#10;problem setting (Roy and Roth, 2015) with very few training examples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.13472" label="2110.13472">
        <attvalues>
          <attvalue for="0" value="Decomposing Complex Questions Makes Multi-Hop QA Easier and More&#10;  Interpretable" />
          <attvalue for="1" value="  Multi-hop QA requires the machine to answer complex questions through finding&#10;multiple clues and reasoning, and provide explanatory evidence to demonstrate&#10;the machine reasoning process. We propose Relation Extractor-Reader and&#10;Comparator (RERC), a three-stage framework based on complex question&#10;decomposition, which is the first work that the RERC model has been proposed&#10;and applied in solving the multi-hop QA challenges. The Relation Extractor&#10;decomposes the complex question, and then the Reader answers the sub-questions&#10;in turn, and finally the Comparator performs numerical comparison and&#10;summarizes all to get the final answer, where the entire process itself&#10;constitutes a complete reasoning evidence path. In the 2WikiMultiHopQA dataset,&#10;our RERC model has achieved the most advanced performance, with a winning joint&#10;F1 score of 53.58 on the leaderboard. All indicators of our RERC are close to&#10;human performance, with only 1.95 behind the human level in F1 score of support&#10;fact. At the same time, the evidence path provided by our RERC framework has&#10;excellent readability and faithfulness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.02322" label="1807.02322">
        <attvalues>
          <attvalue for="0" value="Memory Augmented Policy Optimization for Program Synthesis and Semantic&#10;  Parsing" />
          <attvalue for="1" value="  We present Memory Augmented Policy Optimization (MAPO), a simple and novel&#10;way to leverage a memory buffer of promising trajectories to reduce the&#10;variance of policy gradient estimate. MAPO is applicable to deterministic&#10;environments with discrete actions, such as structured prediction and&#10;combinatorial optimization tasks. We express the expected return objective as a&#10;weighted sum of two terms: an expectation over the high-reward trajectories&#10;inside the memory buffer, and a separate expectation over trajectories outside&#10;the buffer. To make an efficient algorithm of MAPO, we propose: (1) memory&#10;weight clipping to accelerate and stabilize training; (2) systematic&#10;exploration to discover high-reward trajectories; (3) distributed sampling from&#10;inside and outside of the memory buffer to scale up training. MAPO improves the&#10;sample efficiency and robustness of policy gradient, especially on tasks with&#10;sparse rewards. We evaluate MAPO on weakly supervised program synthesis from&#10;natural language (semantic parsing). On the WikiTableQuestions benchmark, we&#10;improve the state-of-the-art by 2.6%, achieving an accuracy of 46.3%. On the&#10;WikiSQL benchmark, MAPO achieves an accuracy of 74.9% with only weak&#10;supervision, outperforming several strong baselines with full supervision. Our&#10;source code is available at&#10;https://github.com/crazydonkey200/neural-symbolic-machines&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.15982" label="2308.15982">
        <attvalues>
          <attvalue for="0" value="MerA: Merging Pretrained Adapters For Few-Shot Learning" />
          <attvalue for="1" value="  Adapter tuning, which updates only a few parameters, has become a mainstream&#10;method for fine-tuning pretrained language models to downstream tasks. However,&#10;it often yields subpar results in few-shot learning. AdapterFusion, which&#10;assembles pretrained adapters using composition layers tailored to specific&#10;tasks, is a possible solution but significantly increases trainable parameters&#10;and deployment costs. Despite this, our preliminary study reveals that even&#10;single adapters can outperform Adapterfusion in few-shot learning, urging us to&#10;propose \textbf{\texttt{Merging Pretrained Adapters}} (MerA) that efficiently&#10;incorporates pretrained adapters to a single model through model fusion.&#10;Extensive experiments on two PLMs demonstrate that MerA achieves substantial&#10;improvements compared to both single adapters and AdapterFusion. To further&#10;enhance the capacity of MerA, we also introduce a simple yet effective&#10;technique, referred to as the &quot;\textit{same-track}&quot; setting, that merges&#10;adapters from the same track of pretraining tasks. With the implementation of&#10;the &quot;\textit{same-track}&quot; setting, we observe even more impressive gains,&#10;surpassing the performance of both full fine-tuning and adapter tuning by a&#10;substantial margin, e.g., 3.5\% in MRPC and 5.0\% in MNLI.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretrained language models (PLMs) \cite{devlin2018bert, liu2019roberta} have revolutionized the field of natural language processing, with fine-tuning being a mainstream approach to leverage the power of PLMs. However, with the ever-increasing number of parameters in PLMs \cite{NEURIPS2020_1457c0d6}, there is a need for parameter-efficient fine-tuning techniques \cite{DBLP:conf/iclr/HeZMBN22} to reduce training costs. One representative technique is adapters tuning \cite{houlsby2019parameter}, which updates only a subset of parameters. Despite their advantages, adapter tuning often starts with randomly initialized blocks and may not perform well in scenarios with limited training data, such as few-shot learning \cite{DBLP:conf/naacl/MoosaviDKG22, bansal2022meta}.&#10;&#10;One possible solution is to transfer knowledge from pretrained adapters to target tasks \cite{Chawla_2021_WACV,zhong2022panda,pmlr-v180-wang22a}. AdapterFusion~\cite{pfeiffer2020adapterfusion} has been proposed to assemble pretrained adapters with composition layers to integrate knowledge. However, AdapterFusion compromises parameter efficiency~\cite{He2022SparseAdapterAE}, primarily due to the excessive number of trainable parameters in the composition layers. On the other hand, deploying parallel pretrained adapters also increases computational costs. &#10;&#10;In this work, we turn to explore an efficient approach to leverage pretrained adapters, raising the following question: Can the current AdapterFusion framework fully exploit the pretrained adapters under few-shot scenarios? If not, how to incorporate the pretrained adapters more efficiently? To this end, we first conduct a series of experiments to compare the performance of single adapters and that of AdapterFusion under few-shot scenarios, with results shown in Figure \ref{pre_experiment}. Surprisingly, a single adapter outperforms AdapterFusion with much few trainable parameters. Such preliminary study prompts us to directly leverage pretrained adapters to extend the potential of single adapters.&#10;&#10;To achieve this, we propose an approach that merges pretrained adapters into a single one (MerA). On the one hand, the merge pretrained adapter does not introduce additional trainable parameters. On the other hand, the knowledge from pretrained adapters enhances downstream performance, as illustrated in Figure~\ref{fig: poster}. We first implement two straightforward methods for merging parameters, including summation (``Sum.'') and averaging (``Avg.''), whereas the lack of one-to-one correspondences between the parameters of different models leads to suboptimal performance. Inspired by~\cite{DBLP:journals/tog/SolomonGPCBNDG15, singh2020model}, we further propose to align adapters' parameters through optimal transport based on weights (``Wts.'') and activations (``Acts.'').&#10;&#10;Extensive few-shot experiments demonstrate that MerA achieves significant improvements compared with Adapters, e.g., 2.7\% in averaged accuracy. In addition, we also find that merging adapters from the same track of tasks further enhances the capacity of MerA. Therefore, we introduce a simple yet effective technique called the &quot;same-track&quot; setting. With the implementation of the &quot;same-track&quot; setting, we observe even more impressive gains, surpassing the performance of full fine-tuning and Adapter tuning by a substantial margin, e.g., 3.5\% in MRPC and 5.0\% in MNLI.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Tuning, Computer Science, Adapter Fusion Techniques, Linguistics, Cognitive Science, Artificial Intelligence, Few-Shot Learning Optimization" />
        </attvalues>
      </node>
      <node id="2208.10160" label="2208.10160">
        <attvalues>
          <attvalue for="0" value="PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model&#10;  Adaptation" />
          <attvalue for="1" value="  Prompt Transfer (PoT) is a recently-proposed approach to improve&#10;prompt-tuning, by initializing the target prompt with the existing prompt&#10;trained on similar source tasks. However, such a vanilla PoT approach usually&#10;achieves sub-optimal performance, as (i) the PoT is sensitive to the similarity&#10;of source-target pair and (ii) directly fine-tuning the prompt initialized with&#10;source prompt on target task might lead to forgetting of the useful general&#10;knowledge learned from source task. To tackle these issues, we propose a new&#10;metric to accurately predict the prompt transferability (regarding (i)), and a&#10;novel PoT approach (namely PANDA) that leverages the knowledge distillation&#10;technique to alleviate the knowledge forgetting effectively (regarding (ii)).&#10;Extensive and systematic experiments on 189 combinations of 21 source and 9&#10;target datasets across 5 scales of PLMs demonstrate that: 1) our proposed&#10;metric works well to predict the prompt transferability; 2) our PANDA&#10;consistently outperforms the vanilla PoT approach by 2.3% average score (up to&#10;24.1%) among all tasks and model sizes; 3) with our PANDA approach,&#10;prompt-tuning can achieve competitive and even better performance than&#10;model-tuning in various PLM scales scenarios. We have publicly released our&#10;code in https://github.com/WHU-ZQH/PANDA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.04284" label="2210.04284">
        <attvalues>
          <attvalue for="0" value="SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency&#10;  of Adapters" />
          <attvalue for="1" value="  Adapter Tuning, which freezes the pretrained language models (PLMs) and only&#10;fine-tunes a few extra modules, becomes an appealing efficient alternative to&#10;the full model fine-tuning. Although computationally efficient, the recent&#10;Adapters often increase parameters (e.g. bottleneck dimension) for matching the&#10;performance of full model fine-tuning, which we argue goes against their&#10;original intention. In this work, we re-examine the parameter-efficiency of&#10;Adapters through the lens of network pruning (we name such plug-in concept as&#10;\texttt{SparseAdapter}) and find that SparseAdapter can achieve comparable or&#10;better performance than standard Adapters when the sparse ratio reaches up to&#10;80\%. Based on our findings, we introduce an easy but effective setting&#10;``\textit{Large-Sparse}'' to improve the model capacity of Adapters under the&#10;same parameter budget. Experiments on five competitive Adapters upon three&#10;advanced PLMs show that with proper sparse method (e.g. SNIP) and ratio (e.g.&#10;40\%) SparseAdapter can consistently outperform their corresponding&#10;counterpart. Encouragingly, with the \textit{Large-Sparse} setting, we can&#10;obtain further appealing gains, even outperforming the full fine-tuning by a&#10;large margin. Our code will be released at:&#10;https://github.com/Shwai-He/SparseAdapter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.11906" label="2207.11906">
        <attvalues>
          <attvalue for="0" value="Learning a Dual-Mode Speech Recognition Model via Self-Pruning" />
          <attvalue for="1" value="  There is growing interest in unifying the streaming and full-context&#10;automatic speech recognition (ASR) networks into a single end-to-end ASR model&#10;to simplify the model training and deployment for both use cases. While in&#10;real-world ASR applications, the streaming ASR models typically operate under&#10;more storage and computational constraints - e.g., on embedded devices - than&#10;any server-side full-context models. Motivated by the recent progress in&#10;Omni-sparsity supernet training, where multiple subnetworks are jointly&#10;optimized in one single model, this work aims to jointly learn a compact sparse&#10;on-device streaming ASR model, and a large dense server non-streaming model, in&#10;a single supernet. Next, we present that, performing supernet training on both&#10;wav2vec 2.0 self-supervised learning and supervised ASR fine-tuning can not&#10;only substantially improve the large non-streaming model as shown in prior&#10;works, and also be able to improve the compact sparse streaming model.&#10;" />
          <attvalue for="2" value="&#10;&#10;Thus far, end-to-end automatic speech recognition (ASR) models, which use neural networks to transduce audio into word sequences, have demonstrated state-of-the-art results compared to conventional hybrid speech recognizers. Specifically, recurrent neural network transducer (RNN-T) originally presented in \cite{graves2012sequence} has shown competitive ASR performance on various benchmarks \cite{chiu2019comparison, li2020comparison, zhang2021benchmarking}. Typically based on token emission latency, we categorize ASR models into: &#10;(i) streaming recognizers \cite{sainath2020streaming, mahadeokar2021flexi} that emit hypothesized words in real time, with low latency measured by milliseconds, and (ii) non-streaming models \cite{gulati2020conformer, zhang2020pushing} that only emit word hypotheses after processing the complete speech utterance. Latest streaming recognizers often employ a transformer/conformer encoder \cite{zhang2020transformer, li2021better}, and may use a limited future audio context (also referred to as look-ahead audio frames) \cite{shi2021emformer, shi2022streaming}. Non-streaming recognizer takes the entire speech utterance as input, and scaling up the model size can often improve the model accuracies \cite{zhang2020pushing}. &#10;&#10;Recently it has been shown favorable to unify the streaming and non-streaming models, either through a single shared encoder \cite{zhang2020transformer, yu2020dual, yao2021wenet, kim2021multi, weninger2022conformer}, or through cascaded streaming and non-streaming encoders \cite{li2021better, narayanan2021cascaded}. &#10;The efficacy of such unified or cascaded encoders includes that the previously two separate development and deployment workflows can be simplified into one process.&#10;Note that in the two-pass cascaded encoders, input acoustic features are typically first processed by a streaming encoder, and a non-streaming encoder processes the streaming encoder outputs and aims to cover the first-pass accuracy loss. &#10;While for the unified dual-mode encoder, the non-streaming encoder directly processes the entire utterance and is immune from the accuracy degradation of the streaming encoder; additionally, the accuracy and latency of the streaming encoder can benefit from the weight sharing, or inplace knowledge distillation from the more performant non-streaming encoder \cite{yu2020dual}. &#10;&#10;This work also focuses on the one-pass dual-mode encoder, while in practice, various streaming ASR models run on devices under more resource constraints, like disk size and memory footprint. In contrast, most non-streaming models run from the server with fewer constraints. Therefore, instead of developing equally sized encoders, it is preferable to jointly build a compact streaming model and a large non-streaming model for real-world ASR applications. We note that even though a single encoder is shared for both modes, we can substantially prune it into a featherweight, e.g., about 30M parameters as a streaming model, and use the original copy as a performant non-streaming encoder. Given the recent progress made in neural network pruning \cite{frankle2018lottery, wu2021dynamic, yang2022omni, ding2021audio}, we can specify a target sparsity level during model training, prune the model weights accordingly before inference, and finally obtain a model of the target model size. Meanwhile, we also aim to maintain the unpruned encoder's performance such that we can keep a copy of the original dense encoder and use it as a competitive non-streaming encoder. &#10;&#10;Prior work \cite{yang2022omni} has shown success on the ASR training of varying sparsities jointly in a single model, also known as supernet training. A supernet is a shared-weight backbone network, where a subnetwork is extracted given each target sparsity level, and all the subnetworks are jointly optimized during supernet training. While it can facilitate ASR training of various model sizes, each sub-model in \cite{yang2022omni} operates with the same inference latency. &#10;Instead, this work focuses on two sparsity levels and two latency conditions: a high sparsity and low latency for the streaming model, and a zero sparsity (i.e., dense or unpruned) and full-utterance latency for the non-streaming model. &#10;Thus, in this case, the dual modes refer to the pruned/sparse streaming mode and the other unpruned/dense non-streaming mode. &#10;&#10;Next, it has been widely shown that the self-supervised acoustic model pre-training based on wav2vec 2.0 \cite{baevski2020wav2vec} can substantially improve large non-streaming models; given sufficient unlabeled data, the potential accuracy gain can be proportional to the growing model size \cite{zhang2020pushing}. Similarly, achieving accuracy gains from pre-training will be difficult given a compact model size. &#10;Also, very few works \cite{sainath2022improving} have shown the self-supervised pre-training efficacy in streaming models. In this paper, we present that by doing the dual-mode supernet training, self-supervised pre-training is not only able to substantially improve the large non-streaming model, and also to improve the compact sparse streaming model. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Sparse Neural Networks, Automatic Speech Recognition, Signal Processing, Model Training Optimization" />
        </attvalues>
      </node>
      <node id="1911.02242" label="1911.02242">
        <attvalues>
          <attvalue for="0" value="A comparison of end-to-end models for long-form speech recognition" />
          <attvalue for="1" value="  End-to-end automatic speech recognition (ASR) models, including both&#10;attention-based models and the recurrent neural network transducer (RNN-T),&#10;have shown superior performance compared to conventional systems. However,&#10;previous studies have focused primarily on short utterances that typically last&#10;for just a few seconds or, at most, a few tens of seconds. Whether such&#10;architectures are practical on long utterances that last from minutes to hours&#10;remains an open question. In this paper, we both investigate and improve the&#10;performance of end-to-end models on long-form transcription. We first present&#10;an empirical comparison of different end-to-end models on a real world&#10;long-form task and demonstrate that the RNN-T model is much more robust than&#10;attention-based systems in this regime. We next explore two improvements to&#10;attention-based systems that significantly improve its performance: restricting&#10;the attention to be monotonic, and applying a novel decoding algorithm that&#10;breaks long utterances into shorter overlapping segments. Combining these two&#10;improvements, we show that attention-based end-to-end models can be very&#10;competitive to RNN-T on long-form speech recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.06030" label="2010.06030">
        <attvalues>
          <attvalue for="0" value="Dual-mode ASR: Unify and Improve Streaming ASR with Full-context&#10;  Modeling" />
          <attvalue for="1" value="  Streaming automatic speech recognition (ASR) aims to emit each hypothesized&#10;word as quickly and accurately as possible, while full-context ASR waits for&#10;the completion of a full speech utterance before emitting completed hypotheses.&#10;In this work, we propose a unified framework, Dual-mode ASR, to train a single&#10;end-to-end ASR model with shared weights for both streaming and full-context&#10;speech recognition. We show that the latency and accuracy of streaming ASR&#10;significantly benefit from weight sharing and joint training of full-context&#10;ASR, especially with inplace knowledge distillation during the training. The&#10;Dual-mode ASR framework can be applied to recent state-of-the-art&#10;convolution-based and transformer-based ASR networks. We present extensive&#10;experiments with two state-of-the-art ASR networks, ContextNet and Conformer,&#10;on two datasets, a widely used public dataset LibriSpeech and a large-scale&#10;dataset MultiDomain. Experiments and ablation studies demonstrate that&#10;Dual-mode ASR not only simplifies the workflow of training and deploying&#10;streaming and full-context ASR models, but also significantly improves both&#10;emission latency and recognition accuracy of streaming ASR. With Dual-mode ASR,&#10;we achieve new state-of-the-art streaming ASR results on both LibriSpeech and&#10;MultiDomain in terms of accuracy and latency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08352" label="2110.08352">
        <attvalues>
          <attvalue for="0" value="Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming&#10;  E2E ASR via Supernet" />
          <attvalue for="1" value="  From wearables to powerful smart devices, modern automatic speech recognition&#10;(ASR) models run on a variety of edge devices with different computational&#10;budgets. To navigate the Pareto front of model accuracy vs model size,&#10;researchers are trapped in a dilemma of optimizing model accuracy by training&#10;and fine-tuning models for each individual edge device while keeping the&#10;training GPU-hours tractable. In this paper, we propose Omni-sparsity DNN,&#10;where a single neural network can be pruned to generate optimized model for a&#10;large range of model sizes. We develop training strategies for Omni-sparsity&#10;DNN that allows it to find models along the Pareto front of word-error-rate&#10;(WER) vs model size while keeping the training GPU-hours to no more than that&#10;of training one singular model. We demonstrate the Omni-sparsity DNN with&#10;streaming E2E ASR models. Our results show great saving on training time and&#10;resources with similar or better accuracy on LibriSpeech compared to&#10;individually pruned sparse models: 2%-6.6% better WER on Test-other.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12720" label="2305.12720">
        <attvalues>
          <attvalue for="0" value="llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large&#10;  Language Models and its Methodology" />
          <attvalue for="1" value="  This study constructed a Japanese chat dataset for tuning large language&#10;models (LLMs), which consist of about 8.4 million records. Recently, LLMs have&#10;been developed and gaining popularity. However, high-performing LLMs are&#10;usually mainly for English. There are two ways to support languages other than&#10;English by those LLMs: constructing LLMs from scratch or tuning existing&#10;models. However, in both ways, datasets are necessary parts. In this study, we&#10;focused on supporting Japanese in those LLMs and making a dataset for training&#10;or tuning LLMs in Japanese. The dataset we constructed consisted of various&#10;tasks, such as translation and knowledge tasks. In our experiment, we tuned an&#10;existing LLM using our dataset and evaluated the performance qualitatively. The&#10;results suggest that our dataset is possibly beneficial for LLMs. However, we&#10;also revealed some difficulties in constructing LLMs in languages other than&#10;English.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have recently achieved remarkable progress in performance and generalization.&#10;Specifically, Transformer-based LLMs such as BERT \cite{Devlin2018} and the GPT series \cite{GPT-1,GPT-2,GPT-3} have demonstrated high-performance thanks to their pre-training.&#10;Furthermore, models that have evolved from these, such as ChatGPT \cite{chatgpt} and GPT4 \cite{gpt4}, have gained popularity for their remarkable performance.&#10;Other models such as Bard \cite{bard}, LLaMA \cite{touvron2023llama}, Dolly \cite{dolly}, Bloom \cite{scao2022bloom}, and Vicuna \cite{vicuna} have also emerged.&#10;&#10;Some of those models are already provided to consumers as a web service.&#10;Moreover, via API, those models and services are also now available for sub-parts of web services, and many spin-off services are emerging.&#10;&#10;However, despite the prosperity of language models, there are still challenges in handling diverse prompts, including prompts written in languages other than English.&#10;For example, Alpaca \cite{alpaca} dataset has been proposed due to the incompleteness of LLaMA's response.&#10;However, the dataset of Alpaca is only available in English, and the incompleteness pointed out by Alpaca has not been filled yet in the other languages.&#10;Moreover, LLaMA has difficultness to respond appropriately to some prompts in languages other than English.&#10;&#10;Considering these challenges, it is necessary to enhance models' performances in languages other than English.&#10;However, it is not a good idea to study a specific model in terms of performance improvements in the other language.&#10;Moreover, model development is still ongoing and very competitive, and the situation is changing dramatically recently.&#10;It is also easy to assume that newer models with better performance will emerge in a few months or even 1-2 months.&#10;Therefore, enhancing datasets that support model training may be more useful than focusing on specific models.&#10;This approach may also lower the barrier to adapting new models to languages other than English.&#10;&#10;Therefore, this study constructed a new chat dataset in Japanese for LLM training, which contains approximately 8.4 million data points, and demonstrated the performance of the dataset qualitatively.&#10;The dataset and trained models are open-sourced and publicly available. The details are as follows:&#10;\begin{itemize}&#10; \item Dataset: &#10; \item Trained Models (LLaMA 1 epoch): \\ &#10;\end{itemize}&#10;The more details are explained in the following.&#10;Moreover, data expansion and additional model training are planned as future tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Support, Data Science, Artificial Intelligence, Language Model Development, Dataset Construction" />
        </attvalues>
      </node>
      <node id="2106.06087" label="2106.06087">
        <attvalues>
          <attvalue for="0" value="Causal Analysis of Syntactic Agreement Mechanisms in Neural Language&#10;  Models" />
          <attvalue for="1" value="  Targeted syntactic evaluations have demonstrated the ability of language&#10;models to perform subject-verb agreement given difficult contexts. To elucidate&#10;the mechanisms by which the models accomplish this behavior, this study applies&#10;causal mediation analysis to pre-trained neural language models. We investigate&#10;the magnitude of models' preferences for grammatical inflections, as well as&#10;whether neurons process subject-verb agreement similarly across sentences with&#10;different syntactic structures. We uncover similarities and differences across&#10;architectures and model sizes -- notably, that larger models do not necessarily&#10;learn stronger preferences. We also observe two distinct mechanisms for&#10;producing subject-verb agreement depending on the syntactic structure of the&#10;input sentence. Finally, we find that language models rely on similar sets of&#10;neurons when given sentences with similar syntactic structure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Syntactic Structure Analysis, Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Neural Network Mechanisms" />
        </attvalues>
      </node>
      <node id="2204.03985" label="2204.03985">
        <attvalues>
          <attvalue for="0" value="KGI: An Integrated Framework for Knowledge Intensive Language Tasks" />
          <attvalue for="1" value="  In this paper, we present a system to showcase the capabilities of the latest&#10;state-of-the-art retrieval augmented generation models trained on&#10;knowledge-intensive language tasks, such as slot filling, open domain question&#10;answering, dialogue, and fact-checking. Moreover, given a user query, we show&#10;how the output from these different models can be combined to cross-examine the&#10;outputs of each other. Particularly, we show how accuracy in dialogue can be&#10;improved using the question answering model. We are also releasing all models&#10;used in the demo as a contribution of this paper. A short video demonstrating&#10;the system is available at https://ibm.box.com/v/emnlp2022-demo.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, we proposed $Re^2G$~\cite{glass-etal-2022-re2g}, the core of our $KGI_$ (Knowledge Graph Induction) system.&#10;$Re^2G$~combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation.&#10;We show that the end-to-end reranking component also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval.&#10;Moreover, to train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker, and generation using only ground truth on the target sequence output. &#10;We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact-checking, and dialog, with relative gains of 9\% to 34\% over the previous state-of-the-art on the KILT leaderboard~\cite{kilt}.&#10;&#10;In this work, we describe the complete KGI system, which is an enhancement of our previous work. We demonstrate how users can asynchronously interact with the system in real-time, not only for completing triples (aka slot filling), but also for dialogue, fact-checking, and open-domain question answering. We empirically show that our system is the state of the art for these tasks on the KILT leaderboard. In addition, we show how dialog accuracy can be improved by exploiting the question answering model, a novel approach demonstrated in this paper.&#10;&#10;There are several different intended usages of our system. For example, KGI allows users to interact with different levels of verbosity. Also, it enables users to cross-examine results through different KILT tasks that are part of the same GUI.&#10;&#10;We are releasing our best KGI core models (i.e. $Re^2G$) that we used in this paper at .&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Generation Models, Linguistics, Artificial Intelligence, Knowledge Retrieval Systems, Information Science, Dialogue Accuracy Improvement, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2407.17874" label="2407.17874">
        <attvalues>
          <attvalue for="0" value="Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions" />
          <attvalue for="1" value="  End-to-end automatic speech recognition (E2E ASR) systems have significantly&#10;improved speech recognition through training on extensive datasets. Despite&#10;these advancements, they still struggle to accurately recognize domain specific&#10;words, such as proper nouns and technical terminologies. To address this&#10;problem, we propose a method to utilize the state-of-the-art Whisper without&#10;modifying its architecture, preserving its generalization performance while&#10;enabling it to leverage descriptions effectively. Moreover, we propose two&#10;additional training techniques to improve the domain specific ASR: decoder&#10;fine-tuning, and context perturbation. We also propose a method to use a Large&#10;Language Model (LLM) to generate descriptions with simple metadata, when&#10;descriptions are unavailable. Our experiments demonstrate that proposed methods&#10;notably enhance domain-specific ASR accuracy on real-life datasets, with&#10;LLM-generated descriptions outperforming human-crafted ones in effectiveness.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancements in end-to-end (E2E) automatic speech recognition (ASR) systems, such as Wav2Vec 2.0 \cite{wav2vec} and Whisper \cite{whisper}, have significantly improved the capabilities of speech recognition through extensive training on large datasets.&#10;However, these systems often encounter difficulties in accurately identifying domain specific terms, such as proper nouns and technical jargon.&#10;Consider a scenario where an ASR system is tasked with transcribing a lecture on mycology. &#10;The pronunciation of `morel' ([m\textopeno\textlengthmark\textprimstress rel]&#10;) and `moral' ([\textprimstress m\textopeno r\textschwa l]) can be very similar, especially in rapid speech, although these words have different meanings.&#10;Since `moral' appears much more frequently in general speech than `morel', general-domain ASRs may misinterpret `morel' as `moral'.&#10;&#10;Contextual biasing \cite{aleksic2015bringing} is widely used in previous studies \cite{le2021contextualized,huang2023contextualized,CB-Conformer,sun2023can} to improve the performance of speech recognition for domain-specific words. &#10;In addition to the audio input, this method provides a biasing list that consists of words that do not appear frequently throughout the dataset or with a high error rate. &#10;However, a notable limitation arises from the reliance on a single biasing list tailored for each specific dataset, such as Mathematics, Finance, and Chemistry.&#10;Creating a comprehensive list that covers all potential domain-specific terms is inherently challenging. Thus, it often fails to recognize highly specific words in the current audio input (e.g., Creating a comprehensive list that covers all product names across various companies for earnings calls.).&#10;&#10;To alleviate the problem, recent works \cite{PromptASR,chang2023context,promptformer,deep-LLM-fusion} introduce methods to incorporate more specific contextual information.&#10;Some of the works \cite{chang2023context,promptformer} utilize the preceding utterance as the contextual information. &#10;However, this approach can suffer from the error propagation problem: an incorrect recognition in an utterance causes incorrect recognition in the next utterances.&#10;In addition, they require an additional text encoder which should be trained on the domain specific dataset, demanding a substantial amount of training data that is frequently unavailable in domain-specific ASR.&#10;The most relevant work to ours is \cite{deep-LLM-fusion}, which utilizes human-written descriptions.&#10;However, it also employs an additional model LLaMa~\cite{Llama} to utilize the textual description.&#10;Since LLaMa (7B$\sim$65B parameters) is much larger than typical ASR models like Whisper (0.04B $\sim$ 1.6B parameters), it introduces a significant computational overhead.&#10;Moreover, the human-written descriptions in real-world datasets often lack details and even are unavailable in many cases.&#10;&#10;In response to these limitations, primarily due to the integration of additional models with pretrained ASR models, &#10;We utilize the existing Whisper \cite{whisper} framework without any additional module. &#10;Since Whisper is already trained for ASR tasks, our approach minimizes the requirement for domain-specific training data.&#10;We also introduce several techniques to efficiently fine-tune Whisper for domain-specific ASR with small domain specific data.&#10;Moreover, for the cases without human-created descriptions, we propose a method to generate a description for each audio by using a Large Language Model (LLM).&#10;The cost of utilizing an LLM is relatively small by generating a single description for each speech rather than for each utterance.&#10;Notably, we empirically demonstrate that the LLM-generated description can surpass human-created ones in improving the accuracy of domain-specific ASRs, thanks to the detailed explanations of LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Domain Adaptation, Artificial Intelligence, Speech Recognition, Language Modeling" />
        </attvalues>
      </node>
      <node id="2104.02194" label="2104.02194">
        <attvalues>
          <attvalue for="0" value="Contextualized Streaming End-to-End Speech Recognition with Trie-Based&#10;  Deep Biasing and Shallow Fusion" />
          <attvalue for="1" value="  How to leverage dynamic contextual information in end-to-end speech&#10;recognition has remained an active research area. Previous solutions to this&#10;problem were either designed for specialized use cases that did not generalize&#10;well to open-domain scenarios, did not scale to large biasing lists, or&#10;underperformed on rare long-tail words. We address these limitations by&#10;proposing a novel solution that combines shallow fusion, trie-based deep&#10;biasing, and neural network language model contextualization. These techniques&#10;result in significant 19.5% relative Word Error Rate improvement over existing&#10;contextual biasing approaches and 5.4%-9.3% improvement compared to a strong&#10;hybrid baseline on both open-domain and constrained contextualization tasks,&#10;where the targets consist of mostly rare long-tail words. Our final system&#10;remains lightweight and modular, allowing for quick modification without model&#10;re-training.&#10;" />
          <attvalue for="2" value="&#10;&#10;End-to-end automatic speech recognition (ASR) models have become increasingly popular in recent years, thanks to their simplicity and competitive performance on generic transcription tasks \cite{Prabhavalkar17,He2019RNNT,Gulati2020conformer,Zhang2021benchmark}. Meanwhile, traditional hybrid systems based on Hidden Markov Model and Deep Neural Network (HMM-DNN) still remain attractive in practice due to their modularity and flexibility. Specifically, the stand-alone language model (LM) allows for easy integration of external knowledge sources, such as unpaired text data and contextual biasing information.&#10;&#10;Much of the past research in end-to-end ASR has focused on methods to inject external knowledge into the system. For static knowledge (i.e., knowledge that does not change from one utterance to the next), various LM fusion techniques have been proposed \cite{gulcehre2015using,Kannan2018,sriram2018cold,toshniwal2018comparison,shan2019component,Kim2021lmfusion}, in addition to methods that remove the internal LM's contribution prior to fusion \cite{McDermott19,Variani2020hat,Meng2021ILME}. For dynamic knowledge (i.e., utterance-specific contextual information), possible solutions include shallow fusion with class-based Weighted Finite State Transducer (WFST) \cite{Zhao2019,He2019RNNT,Le2021deepshallow}, attention-based deep context \cite{Pundak2018DC,Chen2019DC,Jain2020DC}, and trie-based deep biasing \cite{Jain2020DC,Le2021deepshallow}. The shallow fusion-based solutions \cite{Zhao2019,He2019RNNT,Le2021deepshallow} rely on strong context prefixes (e.g., call, play, message) to increase precision and avoid overbiasing. While effective for the targeted tasks, this method does not generalize well to open-domain use cases where such prefixes are not available. On the other hand, deep biasing solutions that inject contextual information directly into the network \cite{Pundak2018DC,Chen2019DC,Jain2020DC,Le2021deepshallow} are not reliant on context prefixes or known patterns, but do not work well when the biasing list gets larger and tend to underperform on rare words. How to develop an end-to-end ASR system that can handle large biasing lists of rare words in an open-domain scenario remains a challenge.&#10;&#10;We tackle this problem by proposing a novel solution that combines WFST shallow fusion, trie-based deep biasing, and neural network LM (NNLM) contextualization. Unlike previous work which relied on specialized WFSTs \cite{Zhao2019,He2019RNNT,Le2021deepshallow}, we use a generic WFST in this work that does not require strong context prefixes nor domain-specific sentence patterns. Unlike previous work which only contextualized the end-to-end ASR model \cite{Pundak2018DC,Chen2019DC,Jain2020DC,Le2021deepshallow}, we propose to fuse the NNLM with trie-based deep biasing to give the former implicit access to biasing information and leverage the vast amount of unpaired text data. On LibriSpeech, our proposed techniques produce on average 34.5\% relative Word Error Rate (WER) improvement compared to the baseline and 19.5\% compared to existing contextual biasing approaches. Finally, our experiments on large-scale in-house data validate the findings on LibriSpeech; the resulting end-to-end system improves significantly over the baseline as well as outperforms a strong hybrid setup by 5.4\%--9.3\% on both open-domain and constrained contextualization tasks, where the targets consist of mostly rare long-tail words. Our final system remains lightweight and modular, allowing for quick modification without re-training the main ASR model.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12493" label="2305.12493">
        <attvalues>
          <attvalue for="0" value="Contextualized End-to-End Speech Recognition with Contextual Phrase&#10;  Prediction Network" />
          <attvalue for="1" value="  Contextual information plays a crucial role in speech recognition&#10;technologies and incorporating it into the end-to-end speech recognition models&#10;has drawn immense interest recently. However, previous deep bias methods lacked&#10;explicit supervision for bias tasks. In this study, we introduce a contextual&#10;phrase prediction network for an attention-based deep bias method. This network&#10;predicts context phrases in utterances using contextual embeddings and&#10;calculates bias loss to assist in the training of the contextualized model. Our&#10;method achieved a significant word error rate (WER) reduction across various&#10;end-to-end speech recognition models. Experiments on the LibriSpeech corpus&#10;show that our proposed model obtains a 12.1% relative WER improvement over the&#10;baseline model, and the WER of the context phrases decreases relatively by&#10;40.5%. Moreover, by applying a context phrase filtering strategy, we also&#10;effectively eliminate the WER degradation when using a larger biasing list.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.09607" label="2304.09607">
        <attvalues>
          <attvalue for="0" value="CB-Conformer: Contextual biasing Conformer for biased word recognition" />
          <attvalue for="1" value="  Due to the mismatch between the source and target domains, how to better&#10;utilize the biased word information to improve the performance of the automatic&#10;speech recognition model in the target domain becomes a hot research topic.&#10;Previous approaches either decode with a fixed external language model or&#10;introduce a sizeable biasing module, which leads to poor adaptability and slow&#10;inference. In this work, we propose CB-Conformer to improve biased word&#10;recognition by introducing the Contextual Biasing Module and the Self-Adaptive&#10;Language Model to vanilla Conformer. The Contextual Biasing Module combines&#10;audio fragments and contextual information, with only 0.2% model parameters of&#10;the original Conformer. The Self-Adaptive Language Model modifies the internal&#10;weights of biased words based on their recall and precision, resulting in a&#10;greater focus on biased words and more successful integration with the&#10;automatic speech recognition model than the standard fixed language model. In&#10;addition, we construct and release an open-source Mandarin biased-word dataset&#10;based on WenetSpeech. Experiments indicate that our proposed method brings a&#10;15.34% character error rate reduction, a 14.13% biased word recall increase,&#10;and a 6.80% biased word F1-score increase compared with the base Conformer.&#10;" />
          <attvalue for="2" value="&#10;End-to-end (E2E) automatic speech recognition (ASR) is gaining popularity due to its simple model structure, high training efficiency, and astounding performance across many tasks\cite{asr0,asr1}. In daily life, due to the diversity of scenarios, directly deploying an ASR model trained on one specific dataset to other domains will cause the problem of domain mismatch\cite{gourav2021personalization,mismatch0,mismatch2,mismatch3,mismatch100}.&#10;&#10;There are unique words in specific domains, called biased words, such as Chinese person name \begin{CJK*}{UTF8}{gbsn}``曹操&quot;&#10;\end{CJK*} (CAO Cao in English). The domain mismatch is also reflected in the problem of biased word recognition. An intuitive and feasible way is introducing contextual information into the speech recognition process to emphasize specific biased words. &#10;&#10;There are two traditional approaches to integrating contextual information into speech recognition. The first approach is shallow fusion\cite{gourav2021personalization,lm0,lm1,lm2}, which fuses the task-specific external language model with the ASR model and boosts the scores of biased words during decoding. The second approach is integrating the contextual module with the ASR model\cite{nn0,nn1} in an all-neural network to incorporate contextual information, leveraging the powerful modeling expressiveness of E2E neural networks\cite{li2022recent}.&#10;&#10;However, the former approach suffers from the adaptation problem, where the traditional language model conflicts with the internal language model\cite{ilm,lm44} in the ASR model due to the inconsistency of the trained domains. In addition, the traditional language model maintains a consistent emphasis on biased words and is incapable of altering the weights of biased words, resulting in a poor prediction performance for biased words\cite{lm5,lm_33}. The latter approach employs an end-to-end model structure that co-trains the contextual module as part of the entire ASR model, making it impractical to regulate the degree of biasing in inference\cite{nn0,recent0,recent1}. Additionally, this approach tends to choose the recurrent neural network transducer (RNN-T)\cite{nn1,nn2} as the base model. However, the RNN-T model has a more complex structure and is more challenging to train, resulting in a decline in model prediction performance\cite{rnnt}. &#10;&#10;In this work, we propose CB-Conformer to solve the problem of biased word recognition by augmenting the Conformer-Encoder\cite{gulati2020conformer} with a Contextual Biasing Module and a Self-Adaptive Language Model. The Self-Adaptive Language Model is proposed to increase the precision of biased word recognition and better coordinate with the ASR internal language model. To boost the sensitivity to biased words, we iteratively alter the weight of biased words in the Self-Adaptive Language Model depending on the recall and precision of biased words. The Contextual Biasing Module with an exceptionally minimal number of parameters is proposed as part of the E2E ASR model.&#10;Due to its high applicability and extensibility, the Contextual Biasing Module can be attached to any block of the Conformer Encoder, and its training and decoding rates are fast. &#10;&#10;In addition, as there is no open-source Mandarin biased words dataset, we construct and release the first open-source Mandarin biased words dataset. We provide three specific subsets filtered from the WenetSpeech\cite{wenetspeech} dataset: the person-name dataset, the place-name dataset, and the organization-name dataset. In addition, we combine the three sub-datasets into a composite biased words dataset and produce a dataset devoid of biased words. All our experiments are based on the divided datasets.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.07414" label="2309.07414">
        <attvalues>
          <attvalue for="0" value="PromptASR for contextualized ASR with controllable style" />
          <attvalue for="1" value="  Prompts are crucial to large language models as they provide context&#10;information such as topic or logical relationships. Inspired by this, we&#10;propose PromptASR, a framework that integrates prompts in end-to-end automatic&#10;speech recognition (E2E ASR) systems to achieve contextualized ASR with&#10;controllable style of transcriptions. Specifically, a dedicated text encoder&#10;encodes the text prompts and the encodings are injected into the speech encoder&#10;by cross-attending the features from two modalities. When using the ground&#10;truth text from preceding utterances as content prompt, the proposed system&#10;achieves 21.9% and 6.8% relative word error rate reductions on a book reading&#10;dataset and an in-house dataset compared to a baseline ASR system. The system&#10;can also take word-level biasing lists as prompt to improve recognition&#10;accuracy on rare words. An additional style prompt can be given to the text&#10;encoder and guide the ASR system to output different styles of transcriptions.&#10;The code is available at icefall.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.04659" label="1910.04659">
        <attvalues>
          <attvalue for="0" value="Multilingual Question Answering from Formatted Text applied to&#10;  Conversational Agents" />
          <attvalue for="1" value="  Recent advances with language models (e.g. BERT, XLNet, ...), have allowed&#10;surpassing human performance on complex NLP tasks such as Reading&#10;Comprehension. However, labeled datasets for training are available mostly in&#10;English which makes it difficult to acknowledge progress in other languages.&#10;Fortunately, models are now pre-trained on unlabeled data from hundreds of&#10;languages and exhibit interesting transfer abilities from one language to&#10;another. In this paper, we show that multilingual BERT is naturally capable of&#10;zero-shot transfer for an extractive Question Answering task (eQA) from English&#10;to other languages. More specifically, it outperforms the best previously known&#10;baseline for transfer to Japanese and French. Moreover, using a recently&#10;published large eQA French dataset, we are able to further show that (1)&#10;zero-shot transfer provides results really close to a direct training on the&#10;target language and (2) combination of transfer and training on target is the&#10;best option overall. We finally present a practical application: a multilingual&#10;conversational agent called Kate which answers to HR-related questions in&#10;several languages directly from the content of intranet pages.&#10;" />
          <attvalue for="2" value="&#10;Over the past few years, we have witnessed a revolution of machine learning in the domain of Natural Language Processing (NLP) \cite{vaswani2017attention,devlin2018bert}. Perhaps motivated by needs and competitions (e.g. GLUE \cite{wang2018glue}), novel and powerful proposals are made every day by public and private laboratories around the world to solve several complex NLP tasks: Natural Language Inference \cite{williams2017broad,levesque2012winograd}, Sentence Similarity and Paraphrasing \cite{dolan2005automatically,agirre2012semeval}, Text Classification \cite{socher2013recursive,warstadt2018neural}, Reading Comprehension/Question Answering (QA) \cite{rajpurkar2016squad,nguyen2016ms,lai2017race,joshi2017triviaqa}. For instance, trained on the SQuAD dataset \cite{rajpurkar2016squad}, language models such as BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta} or XLNet \cite{yang2019xlnet}, have shown a great ability in identifying the answer to a question in a given unstructured source of information (raw text). For most conversational agents currently limited to the detection of a user's intent among a predefined set of possibility, this capability offers new interesting features. In particular, developers would no longer need to anticipate everything as the agent would be able to answer to unexpected questions from web pages or a documentation. With today's available pretrained models and datasets, this feature can be implemented with success for English but, even for other widely used languages, data/models remain scarce. Could we still make it work for those languages then? Creating labeled QA datasets in every target languages would be resource consuming and not very flexible. An alternative direction is transfer learning and in particular, zero-shot transfer \cite{hardalov2019beyond,liu2019xqa} when there is no target data. On the one hand, many specific strategies have been proposed for zero-shot transfer with explicit language alignment \cite{firat2016zero,johnson2017google,asai2018multilingual}. On the other hand, state-of-the-art language models are currently being pre-trained and released on hundreds of languages and they seem to naturally integrate language alignment which allows a surprisingly good performance in zero-shot transfer. &#10;&#10;In this paper, after introducing the history behind Transformer-based language models and then the specificity of BERT and its multilingual version, we empirically demonstrate the latter's capability, when trained to solve the SQuAD task in English, to do the same in other languages (French and Japanese) without using any labeled target data. We also introduce six new cross-lingual QA datasets (question and source in a different language) to better understand the mechanism behind the transfer ability. We then show that we can further improve the results using &quot;virtual&quot; target data obtained with machine translation and end up with a performance competitive to the one that we would have with target data annotated by humans \cite{fquad}. We finally present a practical use of the trained models: an HR virtual assistant named Kate able to automatically answer to user questions from multilingual web pages.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Multilingual NLP, Linguistics, Conversational AI, Artificial Intelligence, Natural Language Processing, Language Model Advances" />
        </attvalues>
      </node>
      <node id="1704.04683" label="1704.04683">
        <attvalues>
          <attvalue for="0" value="RACE: Large-scale ReAding Comprehension Dataset From Examinations" />
          <attvalue for="1" value="  We present RACE, a new dataset for benchmark evaluation of methods in the&#10;reading comprehension task. Collected from the English exams for middle and&#10;high school Chinese students in the age range between 12 to 18, RACE consists&#10;of near 28,000 passages and near 100,000 questions generated by human experts&#10;(English instructors), and covers a variety of topics which are carefully&#10;designed for evaluating the students' ability in understanding and reasoning.&#10;In particular, the proportion of questions that requires reasoning is much&#10;larger in RACE than that in other benchmark datasets for reading comprehension,&#10;and there is a significant gap between the performance of the state-of-the-art&#10;models (43%) and the ceiling human performance (95%). We hope this new dataset&#10;can serve as a valuable resource for research and evaluation in machine&#10;comprehension. The dataset is freely available at&#10;http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at&#10;https://github.com/qizhex/RACE_AR_baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.01519" label="1908.01519">
        <attvalues>
          <attvalue for="0" value="Beyond English-Only Reading Comprehension: Experiments in Zero-Shot&#10;  Multilingual Transfer for Bulgarian" />
          <attvalue for="1" value="  Recently, reading comprehension models achieved near-human performance on&#10;large-scale datasets such as SQuAD, CoQA, MS Macro, RACE, etc. This is largely&#10;due to the release of pre-trained contextualized representations such as BERT&#10;and ELMo, which can be fine-tuned for the target task. Despite those advances&#10;and the creation of more challenging datasets, most of the work is still done&#10;for English. Here, we study the effectiveness of multilingual BERT fine-tuned&#10;on large-scale English datasets for reading comprehension (e.g., for RACE), and&#10;we apply it to Bulgarian multiple-choice reading comprehension. We propose a&#10;new dataset containing 2,221 questions from matriculation exams for twelfth&#10;grade in various subjects -history, biology, geography and philosophy-, and 412&#10;additional questions from online quizzes in history. While the quiz authors&#10;gave no relevant context, we incorporate knowledge from Wikipedia, retrieving&#10;documents matching the combination of question + each answer option. Moreover,&#10;we experiment with different indexing and pre-training strategies. The&#10;evaluation results show accuracy of 42.23%, which is well above the baseline of&#10;24.89%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.04164" label="1606.04164">
        <attvalues>
          <attvalue for="0" value="Zero-Resource Translation with Multi-Lingual Neural Machine Translation" />
          <attvalue for="1" value="  In this paper, we propose a novel finetuning algorithm for the recently&#10;introduced multi-way, mulitlingual neural machine translate that enables&#10;zero-resource machine translation. When used together with novel many-to-one&#10;translation strategies, we empirically show that this finetuning algorithm&#10;allows the multi-way, multilingual model to translate a zero-resource language&#10;pair (1) as well as a single-pair neural translation model trained with up to&#10;1M direct parallel sentences of the same language pair and (2) better than&#10;pivot-based translation strategy, while keeping only one additional copy of&#10;attention-related parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.06071" label="2002.06071">
        <attvalues>
          <attvalue for="0" value="FQuAD: French Question Answering Dataset" />
          <attvalue for="1" value="  Recent advances in the field of language modeling have improved&#10;state-of-the-art results on many Natural Language Processing tasks. Among them,&#10;Reading Comprehension has made significant progress over the past few years.&#10;However, most results are reported in English since labeled resources available&#10;in other languages, such as French, remain scarce. In the present work, we&#10;introduce the French Question Answering Dataset (FQuAD). FQuAD is a French&#10;Native Reading Comprehension dataset of questions and answers on a set of&#10;Wikipedia articles that consists of 25,000+ samples for the 1.0 version and&#10;60,000+ samples for the 1.1 version. We train a baseline model which achieves&#10;an F1 score of 92.2 and an exact match ratio of 82.1 on the test set. In order&#10;to track the progress of French Question Answering models we propose a&#10;leader-board and we have made the 1.0 version of our dataset freely available&#10;at https://illuin-tech.github.io/FQuAD-explorer/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.03487" label="2406.03487">
        <attvalues>
          <attvalue for="0" value="Analyzing LLM Behavior in Dialogue Summarization: Unveiling&#10;  Circumstantial Hallucination Trends" />
          <attvalue for="1" value="  Recent advancements in large language models (LLMs) have considerably&#10;advanced the capabilities of summarization systems. However, they continue to&#10;face concerns about hallucinations. While prior work has evaluated LLMs&#10;extensively in news domains, most evaluation of dialogue summarization has&#10;focused on BART-based models, leaving a gap in our understanding of their&#10;faithfulness. Our work benchmarks the faithfulness of LLMs for dialogue&#10;summarization, using human annotations and focusing on identifying and&#10;categorizing span-level inconsistencies. Specifically, we focus on two&#10;prominent LLMs: GPT-4 and Alpaca-13B. Our evaluation reveals subtleties as to&#10;what constitutes a hallucination: LLMs often generate plausible inferences,&#10;supported by circumstantial evidence in the conversation, that lack direct&#10;evidence, a pattern that is less prevalent in older models. We propose a&#10;refined taxonomy of errors, coining the category of &quot;Circumstantial Inference&quot;&#10;to bucket these LLM behaviors and release the dataset. Using our taxonomy, we&#10;compare the behavioral differences between LLMs and older fine-tuned models.&#10;Additionally, we systematically assess the efficacy of automatic error&#10;detection methods on LLM summaries and find that they struggle to detect these&#10;nuanced errors. To address this, we introduce two prompt-based approaches for&#10;fine-grained error detection that outperform existing metrics, particularly for&#10;identifying &quot;Circumstantial Inference.&quot;&#10;" />
          <attvalue for="2" value="&#10;&#10;Considerable progress has been made in summarization using large language models (LLMs) \cite{goyal2022news, zhang2023benchmarking}. &#10;However, the challenge of so-called ``hallucinations'',&#10;characterized in this context as statements in summaries&#10;that do not have direct evidence in the source material persists. &#10;As a result, evaluation of these summaries is an active area of research. &#10;&#10;In prior research, news articles have been the main testbed &#10;for LLM-generated summary evaluation \cite{zhang2023benchmarking, yang2023exploring}. &#10;Dialogue summarization remain less explored,&#10;with prior works mostly focused on smaller fine-tuned models &#10;\cite{zhu2023annotating, gao2023reference, wang2022analyzing}. &#10;In this work, we close the evaluation gap,&#10;focusing our analysis on LLM summaries of chit-chat style dialogues.&#10;We obtain fine-grained inconsistency annotations &#10;for summaries generated (zero-shot) by two prominent LLMs&#10;(GPT-4 \cite{luo2023chatgpt} and Alpaca-13B \cite{alpaca}) &#10;and across two summarization datasets &#10;(SAMSum \cite{gliwa2019samsum} and DialogSum \cite{chen2021dialogsum}).&#10;&#10;In the domain of dialogues, a further gap exists &#10;in understanding the differences between &#10;summaries generated by LLMs &#10;and those generated by smaller fine-tuned models. &#10;In the news domain, prior work has found that LLM-generated summaries&#10;have fewer inconsistencies \cite{goyal2022news, zhang2023benchmarking}. &#10;Work done by \cite{tang2022understanding}, also in the news domain, &#10;notes varying error distributions across different model categories. &#10;In our work in the dialogue domain, &#10;we compare differences in error rates and analyze the categories of errors &#10;for summaries of dialogues with fine-tuned models versus summaries with LLMs. &#10;As in the news domain, we find that LLM-generated summaries have fewer inconsistencies. &#10;Surprisingly, our analysis reveals that over 30\&#10;of LLM-generated summaries contain inconsistencies, &#10;contrasting sharply with the inconsistency rate &#10;of less than 5\% in GPT-generated news summaries \cite{zhang2023benchmarking}.&#10;&#10;To further elucidate the differences between LLMs and fine-tuned models, &#10; we annotate spans with error categories. &#10; Previous work has primarily relied on part-of-speech-based tags &#10; for error classification \cite{wang2022analyzing, zhu2023annotating, gao2023reference}. &#10; However, complexities inherent in LLM-generated summaries, &#10; often lengthier and more intricate, &#10; do not neatly align with error categories based solely on part of speech, &#10; warranting alternative strategies for a more meaningful categorization.&#10; Hence, our work proposes a refined taxonomy integrating existing error types. &#10; We further introduce a new error category specific to LLM behavior:&#10; &quot;Circumstantial Inference.&quot; &#10; This category stems from the observation that LLMs frequently produce &#10; statements that appear plausible based on circumstantial &#10; (but not direct) evidence in the dialogues, an aspect hitherto unexplored.&#10; In particular, LLMs tend to produce statements &#10; that may be circumstantially implied &#10; based on contextual cues in the conversation&#10; but not explicitly stated as seen in Figure \ref{fig:contextual_inference}. &#10; Although these inferences are not directly stated and can be inherently unsupported, &#10; they can still be useful in some instances,&#10; especially when summarizing ambiguous dialogues.&#10; However, the appropriateness of such inferred details &#10; varies depending on context and domain, &#10; highlighting the need for further investigation.&#10;&#10;In addition, there is limited understanding &#10; regarding the automatic detection of the mentioned error types.&#10; Therefore, we systematically evaluate the performance &#10; of state-of-the-art error detectors on LLM-generated dialogue summaries. &#10; We also introduce two prompt-based methods for fine-grained error detection, &#10; which notably outperform all prior state-of-the-art error detectors, &#10; particularly in identifying the newly introduced error type, &quot;Circumstantial Inference.&quot;\\&#10;&#10;In summary, our primary contributions are as follows:&#10; \begin{enumerate}&#10; \item We bridge a gap in understanding LLM effectiveness for dialogue summarization &#10; by collecting fine-grained human annotations that highlight inconsistencies &#10; and make the benchmark publicly available.&#10; \item We propose a refined taxonomy for error categorization of LLM-generated summaries,&#10; including a new error category called &quot;Circumstantial Inference&quot; &#10; that captures the tendency of LLMs to produce plausible hallucinations based on conversation context.&#10;&#10;\item We examine differences in behavior in dialogue summarization &#10; between LLMs and fine-tuned models by comparing error rates and types.&#10;&#10;\item We introduce two prompt-based methods for fine-grained error detection, &#10; which notably outperform existing metrics. &#10; These methods excel even in detecting the recently identified error type &quot;Circumstantial Inference.&quot;&#10; Additionally, we evaluate state-of-the-art error detectors on model-generated summaries &#10; across model categories and error types &#10; unveiling their effectiveness and limitations. &#10;&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Cognitive Science, Error Detection Methods, Artificial Intelligence, Dialogue Summarization" />
        </attvalues>
      </node>
      <node id="2305.16548" label="2305.16548">
        <attvalues>
          <attvalue for="0" value="Annotating and Detecting Fine-grained Factual Errors for Dialogue&#10;  Summarization" />
          <attvalue for="1" value="  A series of datasets and models have been proposed for summaries generated&#10;for well-formatted documents such as news articles. Dialogue summaries,&#10;however, have been under explored. In this paper, we present the first dataset&#10;with fine-grained factual error annotations named DIASUMFACT. We define&#10;fine-grained factual error detection as a sentence-level multi-label&#10;classification problem, and we evaluate two state-of-the-art (SOTA) models on&#10;our dataset. Both models yield sub-optimal results, with a macro-averaged F1&#10;score of around 0.25 over 6 error classes. We further propose an unsupervised&#10;model ENDERANKER via candidate ranking using pretrained encoder-decoder models.&#10;Our model performs on par with the SOTA models while requiring fewer resources.&#10;These observations confirm the challenges in detecting factual errors from&#10;dialogue summaries, which call for further studies, for which our dataset and&#10;results offer a solid foundation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.12854" label="2205.12854">
        <attvalues>
          <attvalue for="0" value="Understanding Factual Errors in Summarization: Errors, Summarizers,&#10;  Datasets, Error Detectors" />
          <attvalue for="1" value="  The propensity of abstractive summarization models to make factual errors has&#10;been studied extensively, including design of metrics to detect factual errors&#10;and annotation of errors in current systems' outputs. However, the&#10;ever-evolving nature of summarization systems, metrics, and annotated&#10;benchmarks makes factuality evaluation a moving target, and drawing clear&#10;comparisons among metrics has become increasingly difficult. In this work, we&#10;aggregate factuality error annotations from nine existing datasets and stratify&#10;them according to the underlying summarization model. We compare performance of&#10;state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on&#10;this stratified benchmark and show that their performance varies significantly&#10;across different types of summarization models. Critically, our analysis shows&#10;that much of the recent improvement in the factuality detection space has been&#10;on summaries from older (pre-Transformer) models instead of more relevant&#10;recent summarization models. We further perform a finer-grained analysis per&#10;error-type and find similar performance variance across error types for&#10;different factuality metrics. Our results show that no one metric is superior&#10;in all settings or for all error types, and we provide recommendations for best&#10;practices given these insights.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.08209" label="2105.08209">
        <attvalues>
          <attvalue for="0" value="BookSum: A Collection of Datasets for Long-form Narrative Summarization" />
          <attvalue for="1" value="  The majority of available text summarization datasets include short-form&#10;source documents that lack long-range causal and temporal dependencies, and&#10;often contain strong layout and stylistic biases. While relevant, such datasets&#10;will offer limited challenges for future generations of text summarization&#10;systems. We address these issues by introducing BookSum, a collection of&#10;datasets for long-form narrative summarization. Our dataset covers source&#10;documents from the literature domain, such as novels, plays and stories, and&#10;includes highly abstractive, human written summaries on three levels of&#10;granularity of increasing difficulty: paragraph-, chapter-, and book-level. The&#10;domain and structure of our dataset poses a unique set of challenges for&#10;summarization systems, which include: processing very long documents,&#10;non-trivial causal and temporal dependencies, and rich discourse structures. To&#10;facilitate future work, we trained and evaluated multiple extractive and&#10;abstractive summarization models as baselines for our dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.02050" label="2409.02050">
        <attvalues>
          <attvalue for="0" value="Enhancing Code-Switching Speech Recognition with LID-Based Collaborative&#10;  Mixture of Experts Model" />
          <attvalue for="1" value="  Due to the inherent difficulty in modeling phonetic similarities across&#10;different languages, code-switching speech recognition presents a formidable&#10;challenge. This study proposes a Collaborative-MoE, a Mixture of Experts (MoE)&#10;model that leverages a collaborative mechanism among expert groups. Initially,&#10;a preceding routing network explicitly learns Language Identification (LID)&#10;tasks and selects experts based on acquired LID weights. This process ensures&#10;robust routing information to the MoE layer, mitigating interference from&#10;diverse language domains on expert network parameter updates. The LID weights&#10;are also employed to facilitate inter-group collaboration, enabling the&#10;integration of language-specific representations. Furthermore, within each&#10;language expert group, a gating network operates unsupervised to foster&#10;collaboration on attributes beyond language. Extensive experiments demonstrate&#10;the efficacy of our approach, achieving significant performance enhancements&#10;compared to alternative methods. Importantly, our method preserves the&#10;efficient inference capabilities characteristic of MoE models without&#10;necessitating additional pre-training.&#10;" />
          <attvalue for="2" value="&#10;Multilingual speech recognition systems are required to excel in recognizing distinct monolingual languages and, equally importantly, in effectively managing code-switching (CS) scenarios. This necessitates the system's capability to efficiently navigate the complexities of language switching within utterances, a phenomenon increasingly common in today's globalized context. Unfortunately, this issue is challenging, as similar pronunciations across various languages can easily be confused.&#10;&#10;End-to-end speech recognition systems have gained popularity in recent years due to their simple training process and outstanding performance \cite{ctc, attention, LAS, joint,transformer-transducer,speech-transformer}. However, the end-to-end training approach usually requires a large amount of training data. Consequently, the need for more CS training data has emerged as a bottleneck, constraining recognition performance in CS scenarios. CS corpus synthesis \cite{TTS1, TTS2, TTS3} and self-supervised speech representation learning \cite{self-sup1,self-sup3,self-sup2} alleviate this data scarcity problem by synthesizing corpus or utilizing unlabeled data. Cross-language transfer learning &#10; \cite{qianyi1,qianyi2,qianyi3} also effectively alleviates the data scarcity problem by utilizing monolingual corpus to improve performance in CS scenarios. From a practical point of view, mixing monolingual data from different languages with CS data to train a model that performs well in both monolingual and CS scenarios is a reasonable practice. However, the unified modeling of multilingual units may lead to the incorrect modeling of phonetically similar pronunciations across different languages. Therefore, effectively modeling diverse languages within a unified architecture presents a significant challenge &#10; \cite{degrade,youxiaojianmo1,youxiaojianmo2}. &#10;&#10;Previous Bi-Encoder methods \cite{Biencoder-MoE, LAE, LSCA} address this issue by modeling each language separately using two parallel encoders. However, these approaches lack flexibility, exhibit high computational complexity, and struggle to support additional language extensions. Moreover, a separate pre-training initialization of each encoder on monolingual data is required to obtain better performance. The sparse Mixture of Experts (MoE) method \cite{FLR-MOE, LSR-MOE, google-smoe, MIE} aims to extract language-specific representations by introducing expert layers. Nonetheless, the expert layers in these methods struggle to achieve explicit specialized training as they do not utilize the language information inherent in the transcription text of the training data. In other words, without additional language supervision, accurate language classification is hard to achieve, which can easily disrupt the specialized training of expert networks.&#10;&#10;This work aims to develop a bilingual model that performs well in both monolingual and CS scenarios. We propose leveraging Language Identification (LID) to guide expert routing and collaboration within the MoE model. Accurate expert routing ensures minimal interference from other language domains, effectively mitigating language confusion. Furthermore, expert collaboration based on LID weights enables comprehensive integration of language-specific representations, improving performance. Our contributions can be summarized as follows:&#10;\begin{itemize}&#10; \item We show the superiority of LID-based routing in MoE models over the model without language supervision, achieving better performance and inference efficiency than bi-encoder architectures.&#10; \item We investigate the effectiveness of inter-group collaboration using LID weights and intra-group collaboration in an unsupervised manner, confirming their efficacy through empirical validation.&#10; \item Our model maintains efficiency through sparse activation while also exhibiting flexibility, allowing for adjusting the number of experts within expert groups according to the requirements of the specific task.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Artificial Intelligence, Speech Recognition, Language Modeling" />
        </attvalues>
      </node>
      <node id="1906.08647" label="1906.08647">
        <attvalues>
          <attvalue for="0" value="Semi-supervised acoustic model training for five-lingual code-switched&#10;  ASR" />
          <attvalue for="1" value="  This paper presents recent progress in the acoustic modelling of&#10;under-resourced code-switched (CS) speech in multiple South African languages.&#10;We consider two approaches. The first constructs separate bilingual acoustic&#10;models corresponding to language pairs (English-isiZulu, English-isiXhosa,&#10;English-Setswana and English-Sesotho). The second constructs a single unified&#10;five-lingual acoustic model representing all the languages (English, isiZulu,&#10;isiXhosa, Setswana and Sesotho). For these two approaches we consider the&#10;effectiveness of semi-supervised training to increase the size of the very&#10;sparse acoustic training sets. Using approximately 11 hours of untranscribed&#10;speech, we show that both approaches benefit from semi-supervised training. The&#10;bilingual TDNN-F acoustic models also benefit from the addition of CNN layers&#10;(CNN-TDNN-F), while the five-lingual system does not show any significant&#10;improvement. Furthermore, because English is common to all language pairs in&#10;our data, it dominates when training a unified language model, leading to&#10;improved English ASR performance at the expense of the other languages.&#10;Nevertheless, the five-lingual model offers flexibility because it can process&#10;more than two languages simultaneously, and is therefore an attractive option&#10;as an automatic transcription system in a semi-supervised training pipeline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.09699" label="1810.09699">
        <attvalues>
          <attvalue for="0" value="Semi-supervised acoustic model training for speech with code-switching" />
          <attvalue for="1" value="  In the FAME! project, we aim to develop an automatic speech recognition (ASR)&#10;system for Frisian-Dutch code-switching (CS) speech extracted from the archives&#10;of a local broadcaster with the ultimate goal of building a spoken document&#10;retrieval system. Unlike Dutch, Frisian is a low-resourced language with a very&#10;limited amount of manually annotated speech data. In this paper, we describe&#10;several automatic annotation approaches to enable using of a large amount of&#10;raw bilingual broadcast data for acoustic model training in a semi-supervised&#10;setting. Previously, it has been shown that the best-performing ASR system is&#10;obtained by two-stage multilingual deep neural network (DNN) training using 11&#10;hours of manually annotated CS speech (reference) data together with speech&#10;data from other high-resourced languages. We compare the quality of&#10;transcriptions provided by this bilingual ASR system with several other&#10;approaches that use a language recognition system for assigning language labels&#10;to raw speech segments at the front-end and using monolingual ASR resources for&#10;transcription. We further investigate automatic annotation of the speakers&#10;appearing in the raw broadcast data by first labeling with (pseudo) speaker&#10;tags using a speaker diarization system and then linking to the known speakers&#10;appearing in the reference data using a speaker recognition system. These&#10;speaker labels are essential for speaker-adaptive training in the proposed&#10;setting. We train acoustic models using the manually and automatically&#10;annotated data and run recognition experiments on the development and test data&#10;of the FAME! speech corpus to quantify the quality of the automatic&#10;annotations. The ASR and CS detection results demonstrate the potential of&#10;using automatic language and speaker tagging in semi-supervised bilingual&#10;acoustic model training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.06200" label="1806.06200">
        <attvalues>
          <attvalue for="0" value="Study of Semi-supervised Approaches to Improving English-Mandarin&#10;  Code-Switching Speech Recognition" />
          <attvalue for="1" value="  In this paper, we present our overall efforts to improve the performance of a&#10;code-switching speech recognition system using semi-supervised training methods&#10;from lexicon learning to acoustic modeling, on the South East Asian&#10;Mandarin-English (SEAME) data. We first investigate semi-supervised lexicon&#10;learning approach to adapt the canonical lexicon, which is meant to alleviate&#10;the heavily accented pronunciation issue within the code-switching conversation&#10;of the local area. As a result, the learned lexicon yields improved&#10;performance. Furthermore, we attempt to use semi-supervised training to deal&#10;with those transcriptions that are highly mismatched between human transcribers&#10;and ASR system. Specifically, we conduct semi-supervised training assuming&#10;those poorly transcribed data as unsupervised data. We found the&#10;semi-supervised acoustic modeling can lead to improved results. Finally, to&#10;make up for the limitation of the conventional n-gram language models due to&#10;data sparsity issue, we perform lattice rescoring using neural network language&#10;models, and significant WER reduction is obtained.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.01458" label="2211.01458">
        <attvalues>
          <attvalue for="0" value="Towards Zero-Shot Code-Switched Speech Recognition" />
          <attvalue for="1" value="  In this work, we seek to build effective code-switched (CS) automatic speech&#10;recognition systems (ASR) under the zero-shot setting where no transcribed CS&#10;speech data is available for training. Previously proposed frameworks which&#10;conditionally factorize the bilingual task into its constituent monolingual&#10;parts are a promising starting point for leveraging monolingual data&#10;efficiently. However, these methods require the monolingual modules to perform&#10;language segmentation. That is, each monolingual module has to simultaneously&#10;detect CS points and transcribe speech segments of one language while ignoring&#10;those of other languages -- not a trivial task. We propose to simplify each&#10;monolingual module by allowing them to transcribe all speech segments&#10;indiscriminately with a monolingual script (i.e. transliteration). This simple&#10;modification passes the responsibility of CS point detection to subsequent&#10;bilingual modules which determine the final output by considering multiple&#10;monolingual transliterations along with external language model information. We&#10;apply this transliteration-based approach in an end-to-end differentiable&#10;neural network and demonstrate its efficacy for zero-shot CS ASR on&#10;Mandarin-English SEAME test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.00784" label="1904.00784">
        <attvalues>
          <attvalue for="0" value="A Survey of Code-switched Speech and Language Processing" />
          <attvalue for="1" value="  Code-switching, the alternation of languages within a conversation or&#10;utterance, is a common communicative phenomenon that occurs in multilingual&#10;communities across the world. This survey reviews computational approaches for&#10;code-switched Speech and Natural Language Processing. We motivate why&#10;processing code-switched text and speech is essential for building intelligent&#10;agents and systems that interact with users in multilingual communities. As&#10;code-switching data and resources are scarce, we list what is available in&#10;various code-switched language pairs with the language processing tasks they&#10;can be used for. We review code-switching research in various Speech and NLP&#10;applications, including language processing tools and end-to-end systems. We&#10;conclude with future directions and open problems in the field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.07987" label="2304.07987">
        <attvalues>
          <attvalue for="0" value="Chinese Open Instruction Generalist: A Preliminary Release" />
          <attvalue for="1" value="  Instruction tuning is widely recognized as a key technique for building&#10;generalist language models, which has attracted the attention of researchers&#10;and the public with the release of InstructGPT~\citep{ouyang2022training} and&#10;ChatGPT\footnote{\url{https://chat.openai.com/}}. Despite impressive progress&#10;in English-oriented large-scale language models (LLMs), it is still&#10;under-explored whether English-based foundation LLMs can perform similarly on&#10;multilingual tasks compared to English tasks with well-designed instruction&#10;tuning and how we can construct the corpora needed for the tuning. To remedy&#10;this gap, we propose the project as an attempt to create a Chinese instruction&#10;dataset by various methods adapted to the intrinsic characteristics of 4&#10;sub-tasks. We collect around 200k Chinese instruction tuning samples, which&#10;have been manually checked to guarantee high quality. We also summarize the&#10;existing English and Chinese instruction corpora and briefly describe some&#10;potential applications of the newly constructed Chinese instruction corpora.&#10;The resulting \textbf{C}hinese \textbf{O}pen \textbf{I}nstruction&#10;\textbf{G}eneralist (\textbf{COIG}) corpora are available in&#10;Huggingface\footnote{\url{https://huggingface.co/datasets/BAAI/COIG}} and&#10;Github\footnote{\url{https://github.com/BAAI-Zlab/COIG}}, and will be&#10;continuously updated.&#10;" />
          <attvalue for="2" value="&#10;&#10;Pre-trained large-scale language models (LLMs) have shown revolutionary performance in many downstream tasks~\cite{hc3,flan}. One crucial ability of LLMs is called instruction following. That is, models can complete the tasks described by instructions given as input. &#10;This ability is based on a specialized training stage called instruction tuning. &#10;Compared to unlabeled data used for pre-training, the data for instruction tuning is typically more goal-oriented, and it should explicitly demonstrate how a response follows its corresponding instruction with a given input.&#10;&#10;&#10;&#10;There are many instruction tuning datasets in English. &#10;For example, the FLAN collection~\cite{flancollection} contains 15M examples covering 1836 tasks, and OPT-IML~\cite{iyer2022opt} claims to have 18M examples for more than 2000 tasks (although it is still not publicly available). &#10;In contrast, existing data resources for Chinese instruction tuning are either small in scale or have questionable quality. &#10;For example, \cite{luotuo} directly translate English instruction tuning data into Chinese, but do not consider mitigating translation errors or potential cultural gaps, e.g. Chinese prefer self-sacrifice spirit while most Western countries prefer self-expression and individualistic heroism, between languages. &#10;For datasets that are mostly constructed through generations using Chinese LLMs \cite{belle, xu2023baize, leng2023chinese-vicuna}, they typically lacks a thorough data verification process for quality assurance. &#10;Therefore, we are motivated to develop new instruction-tuning corpora, Chinese Open Instruction Generalist (COIG), that is larger, more diverse, and manually verified by humans.&#10;This ensures its quality, which echoes the findings of Alpaca~\cite{alpaca} that high-quality and diverse demonstrations are crucial for good instruction-following performance.&#10;&#10;We highlight some unique features of COIG:&#10;&#10;\begin{itemize}&#10; \item Domain Adaption: As shown in Tab.~\ref{tab:overview}, we consider four dimensions of instruction-tuning datasets (Verification, Format, Culture, Scaling). For each domain, we adapt our data collection pipeline to better reflect the domain specialty.&#10;&#10;\item Diversity: We consider a variety of tasks, including common sense reasoning, human value alignment, code generation, and hallucination correction, while very few Chinese instruction tuning data is deliberately designed for such a complete spectrum.&#10;&#10;\item Quality Check by Humans: Compared to existing model-generated Chinese instruction corpora, including \cite{luotuo,belle,instructionwild,guanaco_dataset}, COIG translated corpus is carefully verified by human annotators. &#10; Moreover, since COIG translated corpus is translated from English instruction corpora~\cite{supernaturalinstructions,honovich2022unnaturalinstructions,selfinstruct} with diverse tasks, it is much more diverse than Chinese instruction corpora built by adapting prompt engineering on existing Chinese datasets, e.g. \cite{zeng2023glm-130b,Firefly,hc3}. &#10;\end{itemize}&#10;&#10;The main portion of COIG data is actual data that already exists on the Web, and we convert it into the proper instruction-following manner in terms of their characteristics. For example, for the academic exams domain, we crawled and manually annotated 63.5k instructions from the Chinese National College Entrance Examination, Civil Servant Examination, etc. COIG also features in including data on human value alignment in the Chinese-speaking world, and leetcode-based instruction following samples for programming. To ensure the final data quality, we hired 223 Chinese college students as quality checkers, to help us with data filtering, correction, and ratings. The resulting COIG corpus is a comprehensive set that can equip Chinese LLMs with strong instruction-following abilities in many domains.&#10;&#10;In addition, we provide insights into the data construction pipeline based on empirical observations. We demonstrate that selecting the proper pipeline for different domains is crucial, and we have suggested the best practice for constructing instruction-tuning data in the domains COIG covers ($\S$~\ref{sec: coig}), which can be used as a reference for future instruction corpus construction workflow design.&#10;&#10;The paper's contributions are as follows:&#10;\begin{itemize}&#10; \item To the best of our knowledge, this is one of the very first research works specifically summarizing the existing Chinese instruction tuning corpora and providing insights about how future Chinese instruction tuning corpora can be constructed.&#10; \item We construct 5 open-source high-quality Chinese instruction corpora, including a 68k general Chinese instruction corpus, a 62k Chinese exam instruction corpus, a 3k Chinese human-value alignment corpus, and a 13k Chinese Counterfactual Correction Multi-round Chat corpus, as samples of constructing new Chinese instruction corpora along the research directions pointed out.&#10; \item We construct a manually verified general high-quality Chinese instruction tuning corpus which can be directly used for Chinese LLMs' instruction tuning, both commercial and non-commercial.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Corpus Construction, Computer Science, Linguistics, Multilingual Instruction Tuning, Computational Linguistics, Artificial Intelligence, Language Model Development, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2402.02302" label="2402.02302">
        <attvalues>
          <attvalue for="0" value="Predicting positive transfer for improved low-resource speech&#10;  recognition using acoustic pseudo-tokens" />
          <attvalue for="1" value="  While massively multilingual speech models like wav2vec 2.0 XLSR-128 can be&#10;directly fine-tuned for automatic speech recognition (ASR), downstream&#10;performance can still be relatively poor on languages that are&#10;under-represented in the pre-training data. Continued pre-training on 70-200&#10;hours of untranscribed speech in these languages can help -- but what about&#10;languages without that much recorded data? For such cases, we show that&#10;supplementing the target language with data from a similar, higher-resource&#10;'donor' language can help. For example, continued pre-training on only 10 hours&#10;of low-resource Punjabi supplemented with 60 hours of donor Hindi is almost as&#10;good as continued pretraining on 70 hours of Punjabi. By contrast, sourcing&#10;data from less similar donors like Bengali does not improve ASR performance. To&#10;inform donor language selection, we propose a novel similarity metric based on&#10;the sequence distribution of induced acoustic units: the Acoustic Token&#10;Distribution Similarity (ATDS). Across a set of typologically different target&#10;languages (Punjabi, Galician, Iban, Setswana), we show that the ATDS between&#10;the target language and its candidate donors precisely predicts target language&#10;ASR performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;For developing automatic speech recognition (ASR), `low resource' languages are typically classified as such based on the availability of transcribed speech.~Untranscribed speech, texts, or reliable metadata about the language are often assumed to be easily obtainable.~This assumption may not hold true for under-described languages with little digital representation.~For such languages, we are interested in two questions:~1) does leveraging untranscribed speech from a similar, higher-resource `donor' language for pre-trained model adaptation help improve speech recognition in the target language,~and 2) how do we select the best donor?&#10;&#10;These questions are of interest as ASR system development with little transcribed speech has become viable with multilingual pre-trained transformer models for speech ~\cite[e.g. wav2vec 2.0 XLSR-128:][]{babu22_interspeech}.~Yet, as most languages are under-represented in the pre-training data, directly fine-tuning these models for ASR in the target language can yield lower performance compared to their well-represented counterparts \cite{conneau2023fleurs}.~While recent studies have shown the effectiveness of continued pre-training to adapt these models to the target language~\cite{NOWAKOWSKI2023103148,10301554}, they involved using 70--200 hours of target language data.~For some languages, it may be quite difficult to source this much speech data --- even untranscribed.&#10;&#10;Thus, in our first set of experiments, we investigated whether supplementing target language data with data from another language could be a viable approach for model adaptation via continued pre-training (CPT).~We selected Punjabi as our target language to establish top-line performance when sufficient data is available~\cite[70 hours, approximating the setup in][]{10301554}, along with a limited data baseline (when only 10 hours of Punjabi is available).~We compared this baseline to supplementing the 10 hours of Punjabi with 60 hours of data from 8 other Indic languages (Indo-Aryan:~Hindi, Urdu, Gujarati, Marathi, Bengali, Odia; Dravidian:~Malayalam, Tamil).~We fine-tuned each CPT-adapted model using the same 1 hour of transcribed Punjabi speech.&#10;&#10;Results indicated that adding data from unrelated Dravidian languages (Malayalam, Tamil) or dissimilar Indo-Aryan languages (Bengali, Odia) yielded no better than baseline performance, 25\% word error rate (WER).~By contrast, we observed improved WERs from adding more similar languages (Marathi, Urdu, Gujarati, Hindi),~with adding Hindi coming close to the 70-hour Punjabi top-line: 23.2\% vs. 22.2\%, respectively.&#10;&#10;In our second set of experiments, we investigated how well measures of similarity between the target and donor languages predicted target language ASR performance.~We found that commonly used measures based on external typological databases such as lang2vec \cite{littell2017uriel} were not sufficiently fine-grained for our use case and, crucially, also varied with the quality/completeness of the available metadata for a given target language.&#10;&#10;To sidestep these issues, we propose the Acoustic Token Distribution Similarity (ATDS), which measures the degree of similarity for two untranscribed speech corpora based on frequencies of occurrence of recurring acoustic-phonetic sequences.~This measure extends Token Distribution Similarity~\cite{gogoulou2023study}, shown to correlate with positive transfer in continued pre-training for text-based language models.~To account for the text-/token-less nature of untranscribed speech corpora, we induce them in a bottom-up manner using wav2seq \cite{10096988}, a method for inducing pseudo-tokens using pre-trained speech embeddings.~We compared the ASR performance from the Indic language experiments to various similarity measures and found that ATDS offered the most accurate ranking.~Furthermore, ATDS correctly predicted the best donor language between two options for three non-Indic low-resource languages (Galician, Iban, and Setswana).&#10;&#10;In sum, the main contributions of this paper are:~1) a systematic study of pairwise transfer between languages in continued pre-training and its effects on target language ASR performance,~and 2) the development, analysis, and first validation of ATDS --- a fine-grained, bottom-up measure of acoustic-phonetic similarity to predict this ASR performance.~To facilitate reproducibility and further research,~we make available all our code, model checkpoints, and experimental artefacts.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Acoustics, Computer Science, Linguistics, Low-Resource Language ASR, Signal Processing, Language Similarity Metrics, Multilingual Speech Models" />
        </attvalues>
      </node>
      <node id="2205.01086" label="2205.01086">
        <attvalues>
          <attvalue for="0" value="Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo&#10;  Languages" />
          <attvalue for="1" value="  We introduce Wav2Seq, the first self-supervised approach to pre-train both&#10;parts of encoder-decoder models for speech data. We induce a pseudo language as&#10;a compact discrete representation, and formulate a self-supervised pseudo&#10;speech recognition task -- transcribing audio inputs into pseudo subword&#10;sequences. This process stands on its own, or can be applied as low-cost&#10;second-stage pre-training. We experiment with automatic speech recognition&#10;(ASR), spoken named entity recognition, and speech-to-text translation. We set&#10;new state-of-the-art results for end-to-end spoken named entity recognition,&#10;and show consistent improvements on 20 language pairs for speech-to-text&#10;translation, even when competing methods use additional text data for training.&#10;Finally, on ASR, our approach enables encoder-decoder methods to benefit from&#10;pre-training for all parts of the network, and shows comparable performance to&#10;highly optimized recent methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12547" label="2010.12547">
        <attvalues>
          <attvalue for="0" value="Multilingual BERT Post-Pretraining Alignment" />
          <attvalue for="1" value="  We propose a simple method to align multilingual contextual embeddings as a&#10;post-pretraining step for improved zero-shot cross-lingual transferability of&#10;the pretrained models. Using parallel data, our method aligns embeddings on the&#10;word level through the recently proposed Translation Language Modeling&#10;objective as well as on the sentence level via contrastive learning and random&#10;input shuffling. We also perform sentence-level code-switching with English&#10;when finetuning on downstream tasks. On XNLI, our best model (initialized from&#10;mBERT) improves over mBERT by 4.7% in the zero-shot setting and achieves&#10;comparable result to XLM for translate-train while using less than 18% of the&#10;same parallel data and 31% less model parameters. On MLQA, our model&#10;outperforms XLM-R_Base that has 57% more parameters than ours.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Training Multilingual LMs with Shared Vocabulary&#10;mBERT \cite{Devlin+18} is trained using MLM and NSP objectives on Wikipedia data in 104 languages with a shared vocabulary. Several works study what makes this pretrained model multilingual, and why it works well for cross-lingual transfer. \cite{Pires+20} hypothesize that having a shared vocabulary for all languages helps mapping tokens to a shared space. However, \cite{K+20} train several bilingual BERT models such as en-es, and {enfake}-es, where data for {enfake} is constructed by Unicode shifting of the English data such that there is no character overlap with data of the other language. Result shows that {enfake}-es still transfers well to Spanish and the contribution from shared vocabulary is very small. The authors point out that model depth and capacity instead are the key factors contributing to mBERT's cross-lingual transferability. XLM-R \cite{Conneau+20a} improves over mBERT by training longer with more data from CommonCrawl, and without the NSP objective. In terms of model size, XLM-R uses over 3x more parameters than mBERT. Its base version, XLM-R$_{Base}$, is more comparable to mBERT with the same hidden size and number of attention heads, but a larger shared vocabulary.&#10;&#10;Training Multilingual LMs with Parallel Sentences&#10;In addition to MLM on monolingual data, XLM \cite{Conneau-Lample-19} further improves their cross-lingual LM pretraining by introducing a new TLM objective on parallel data. TLM concatenates source and target sentences together, and predicts randomly masked tokens. Our work uses a slightly different version of TLM together with a contrastive objective to post-pretrain mBERT. Unlike XLM, our TLM does not reset positions of target sentences, and does not use language embeddings. We also randomly shuffle the order of source and target sentences. Another difference between XLM and our work is XLM has $45\%$ more parameters and uses more training data. Similar to XLM, Unicoder \cite{Huang+19} pretrains LMs on multilingual corpora. In addition to MLM and TLM, they introduce three additional cross-lingual pretraining tasks: word recover, paraphrase classification, and mask language model. \cite{Yang+20} propose Alternating Language Modeling (ALM). On a pair of bilingual sequences, instead of TLM, they perform phrase-level code-switching and MLM on the code-switched sequence. ALM is pretrained on both monolingual Wikipedia data and 1.5B code-switched sentences. &#10;&#10;Training mBERT with Word Alignments&#10;\cite{Cao+20} post-align mBERT embeddings by first generating word alignments on parallel sentences that involve English. For each aligned word pair, the $L_2$ distance between their embeddings is minimized to train the model. In order to maintain original transferability to downstream tasks, a regularization term is added to prevent the target language embeddings from deviating too much from their mBERT initialization. Our approach post-aligns mBERT with two self-supervised signals from parallel data without using pre-alignment tools. \cite{Wang+19} also align mBERT embeddings using parallel data. They learn a linear transformation that maps a word embedding in a target language to the embedding of the aligned word in the source language. They show that their transformed embeddings are more effective on zero-shot cross-lingual dependency parsing.&#10;&#10;Besides the aforementioned three major directions, \cite{Artetxe-Schwenk-19} train a multilingual sentence encoder on $93$ languages. Their stacked BiLSTM encoder is trained by first generating embedding of a source sentence and then decoding the embedding into the target sentence in other languages.&#10;&#10;Concurrent to our work, \cite{Chi+20}, \cite{Feng+20} and \cite{Wei+20} also leverage variants of contrastive learning for cross-lingual alignment. We focus on a smaller model and improve on it using as little parallel data as possible. We also explore code-switching during finetuning on downtream tasks to complement the post-pretraining alignment objectives. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Cross-Lingual Transfer, Artificial Intelligence, Mathematics, Multilingual Embeddings, Language Modeling" />
        </attvalues>
      </node>
      <node id="2310.11324" label="2310.11324">
        <attvalues>
          <attvalue for="0" value="Quantifying Language Models' Sensitivity to Spurious Features in Prompt&#10;  Design or: How I learned to start worrying about prompt formatting" />
          <attvalue for="1" value="  As large language models (LLMs) are adopted as a fundamental component of&#10;language technologies, it is crucial to accurately characterize their&#10;performance. Because choices in prompt design can strongly influence model&#10;behavior, this design process is critical in effectively using any modern&#10;pre-trained generative language model. In this work, we focus on LLM&#10;sensitivity to a quintessential class of meaning-preserving design choices:&#10;prompt formatting. We find that several widely used open-source LLMs are&#10;extremely sensitive to subtle changes in prompt formatting in few-shot&#10;settings, with performance differences of up to 76 accuracy points when&#10;evaluated using LLaMA-2-13B. Sensitivity remains even when increasing model&#10;size, the number of few-shot examples, or performing instruction tuning. Our&#10;analysis suggests that work evaluating LLMs with prompting-based methods would&#10;benefit from reporting a range of performance across plausible prompt formats,&#10;instead of the currently-standard practice of reporting performance on a single&#10;format. We also show that format performance only weakly correlates between&#10;models, which puts into question the methodological validity of comparing&#10;models with an arbitrarily chosen, fixed prompt format. To facilitate&#10;systematic analysis we propose FormatSpread, an algorithm that rapidly&#10;evaluates a sampled set of plausible prompt formats for a given task, and&#10;reports the interval of expected performance without accessing model weights.&#10;Furthermore, we present a suite of analyses that characterize the nature of&#10;this sensitivity, including exploring the influence of particular atomic&#10;perturbations and the internal representation of particular formats.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;The task of automatically finding the best-performing prompt for a given task without changing model parameters has recently gained attention, given the constantly improving yet somewhat unpredictable performance of LLMs. &#10;Prior work has often focused on discovering optimal prompts with gradient-based methods, which are effective, but often lead to disfluent or unnatural prompts~\cite{shin2020autoprompt}, which can be mitigated with a Langevin dynamics-based method~\cite{shi2022toward}. &#10;Another approach is to learn, optimize, and insert continuous representations of prompts and tasks as input to models~\cite{qin2021learning,lester2021power,ding2022openprompt,ilharco23}.&#10;These methods also require access to the LLM's parameters, thus cannot be applied to models behind an API. &#10;In contrast, { FormatSpread} does not assume access to any model internals.&#10;Prior gradient-free work has focused on edit-based enumeration over human-written prompts \cite{prasad2022grips}, reinforcement learning \cite{deng2022rlprompt}, and by using LLMs themselves \cite{zhou2022large,gao2020making}. &#10;These works aim to achieve competitive task performance, even if the meaning of the prompt or instruction is modified. &#10;To our knowledge, we are the first to focus specifically on prompt formatting variance, a quintessential example of semantic equivalence. &#10;&#10;Jailbreaking refers to the behavior of intentionally manipulating prompts to elicit inappropriate or sensitive responses, or otherwise reveal parts of the prompt that were intentionally not revealed.&#10;While the objective differs from our work, jailbreaking works \cite{wei2023jailbroken,zou2023universal} share the underlying technical question of finding the lowest-performing prompt. Our methods differ, since \cite{wei2023jailbroken} evaluate human-generated attacks to guide adversarial prompt design, and \cite{zou2023universal} uses gradient-based search methods simultaneously across multiple models.&#10;&#10;Some existing work has explored the influence of certain prompt design choices on model performance, for example the prompt's language~\cite{gonen2022demystifying}, the ordering of few-shot examples~\cite{lu2021fantastically}, and their patterns \cite{madaan2023makes}. Other work has focused on providing textual interpretations of continuous prompt representations~\cite{khashabi-etal-2022-prompt}.&#10;Beyond autoregressive LLMs, existing work has focused on performance variance in masked language models~\cite{elazar2021measuring, jiang2020can}.&#10;Our work follows efforts in other domains that explore the influence of spurious features on research evaluations, e.g., in deep reinforcement learning~\cite{islam2017reproducibility,henderson2018deep} and statistical machine translation~\cite{clark2011better}. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Evaluation Methodology, Artificial Intelligence, Language Model Performance, Statistics, Prompt Design Sensitivity" />
        </attvalues>
      </node>
      <node id="2307.11779" label="2307.11779">
        <attvalues>
          <attvalue for="0" value="The Extractive-Abstractive Axis: Measuring Content &quot;Borrowing&quot; in&#10;  Generative Language Models" />
          <attvalue for="1" value="  Generative language models produce highly abstractive outputs by design, in&#10;contrast to extractive responses in search engines. Given this characteristic&#10;of LLMs and the resulting implications for content Licensing &amp; Attribution, we&#10;propose the the so-called Extractive-Abstractive axis for benchmarking&#10;generative models and highlight the need for developing corresponding metrics,&#10;datasets and annotation guidelines. We limit our discussion to the text&#10;modality.&#10;" />
          <attvalue for="2" value="&#10;&#10;The widespread adoption of Large Language Models (LLMs) has created many practical data governance challenges, among which Licensing \&amp; Attribution has emerged as a key one \cite {jernite2022data}. The interplay between generative language models and copyright law, the fair use doctrine and licensing requirements is of broad research and practical interest to legal practitioners, and increasingly, developers and users of LLMs. This topic is not new: content owners' rights have been of interest to the legal community since the inception of the web and the subsequent wide spread use of search engines \cite{travis2008opting}. Traditionally, search engines have been powered by information retrieval techniques, which take as input a user query and output a query answer by parsing out relevant paragraphs, sentences or phrases from a web-scale corpus of documents to produce an attributable extractive answer to the query.&#10;&#10;The advent of LLMs -- which \cite{liu2023evaluating} call \say{generative search engines} -- is leading to a paradigm shift from attributable extractive question answering and summarization methodologies to increasingly abstractive ones. To produce these abstractive responses, generative models \cite{bart, t5} synthesize information from multiple sources/text documents using sequence-to-sequence LLMs such that the generated answers may be highly abstractive or otherwise not readily attributable -- as they are in search engines -- to a specific content source such as a document on the web with a unique URI identifier . Given this shift, we propose the Extractive--Abstractive axis for quantifying the propensity of LLMs for content borrowing.&#10;We highlight the need for relevant metrics, benchmarks and annotations and list some practical challenges in Section \ref{challenges}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Generative Text Analysis, Artificial Intelligence, Information Science, Content Licensing Attribution, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2109.03264" label="2109.03264">
        <attvalues>
          <attvalue for="0" value="Text-Free Prosody-Aware Generative Spoken Language Modeling" />
          <attvalue for="1" value="  Speech pre-training has primarily demonstrated efficacy on classification&#10;tasks, while its capability of generating novel speech, similar to how GPT-2&#10;can generate coherent paragraphs, has barely been explored. Generative Spoken&#10;Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing&#10;the generative aspects of speech pre-training, which replaces text with&#10;discovered phone-like units for language modeling and shows the ability to&#10;generate meaningful novel sentences. Unfortunately, despite eliminating the&#10;need of text, the units used in GSLM discard most of the prosodic information.&#10;Hence, GSLM fails to leverage prosody for better comprehension, and does not&#10;generate expressive speech. In this work, we present a prosody-aware generative&#10;spoken language model (pGSLM). It is composed of a multi-stream transformer&#10;language model (MS-TLM) of speech, represented as discovered unit and prosodic&#10;feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to&#10;waveforms. We devise a series of metrics for prosody modeling and generation,&#10;and re-use metrics from GSLM for content modeling. Experimental results show&#10;that the pGSLM can utilize prosody to improve both prosody and content&#10;modeling, and also generate natural, meaningful, and coherent speech given a&#10;spoken prompt. Audio samples can be found at https://speechbot.github.io/pgslm.&#10;Codes and models are available at&#10;https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/pgslm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Our work is related to utilizing prosody for comprehension and predicting prosody for speech synthesis, which we discuss in the following sections.&#10;" />
          <attvalue for="4" value="Computer Science, Generative Language Modeling, Linguistics, Speech Pre-training, Engineering, Artificial Intelligence, Prosody Aware Modeling" />
        </attvalues>
      </node>
      <node id="2210.09683" label="2210.09683">
        <attvalues>
          <attvalue for="0" value="Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Task" />
          <attvalue for="1" value="  In this report, we present our submission to the WMT 2022 Metrics Shared&#10;Task. We build our system based on the core idea of UNITE (Unified Translation&#10;Evaluation), which unifies source-only, reference-only, and&#10;source-reference-combined evaluation scenarios into one single model.&#10;Specifically, during the model pre-training phase, we first apply the&#10;pseudo-labeled data examples to continuously pre-train UNITE. Notably, to&#10;reduce the gap between pre-training and fine-tuning, we use data cropping and a&#10;ranking-based score normalization strategy. During the fine-tuning phase, we&#10;use both Direct Assessment (DA) and Multidimensional Quality Metrics (MQM) data&#10;from past years' WMT competitions. Specially, we collect the results from&#10;models with different pre-trained language model backbones, and use different&#10;ensembling strategies for involved translation directions.&#10;" />
          <attvalue for="2" value="&#10;Translation metric aims at delivering accurate and convincing predictions to identify the translation quality of outputs with access to one or many gold-standard reference translations~\cite{ma-etal-2018-results,ma-etal-2019-results,mathur-etal-2020-results,freitag-etal-2021-results}.&#10;As the development of neural machine translation research~\cite{vaswani2017attention,wei-etal-2022-learning}, the metric methods should be capable of evaluating the high-quality translations at the level of semantics rather than surfance-level features~\cite{sellam-etal-2020-bleurt,ranasinghe-etal-2020-transquest,rei-etal-2020-comet,wan-etal-2022-unite}.&#10;In this paper, we describe Alibaba Translate China's submissions to the WMT 2022 Metrics Shared Task to deliver a more adequate evaluation solution at the level of semantics.&#10;&#10;Pre-trained language models (PLMs) like BERT~\cite{devlin-etal-2019-bert} and XLM-R~\cite{conneau-etal-2020-unsupervised} have shown promising results in identifying the quality of translation outputs.&#10;Compared to conventional statistical- (e.g., BLEU,~\cite{papineni-etal-2002-bleu} and representation-based methods (e.g.,~BERTScore,~\cite{zhang2019bertscore}), the model-based approaches (e.g.,~\cite[BLEURT,][]{sellam-etal-2020-bleurt};~\cite[COMET,][]{rei-etal-2020-comet};~\cite[UniTE,][]{wan-etal-2022-unite}) show their strong ability on delivering more accurate quality predictions, especially those approaches which apply source sentences as additional input for the metric model~\cite{rei-etal-2020-comet,takahashi-etal-2020-automatic,wan-etal-2021-robleurt,wan-etal-2022-unite}.&#10;Specifically, those metric models are designed as a combination of PLM and feedforward network, where the former is in charge of deriving representations on input sequence, and the latter predicts the translation quality based on the representation.&#10;The metric model, which is trained on synthetic or human annotations following a regressive objective, learns to mimic human predictions to identify the translation quality of the hypothesis sentence.&#10;&#10;Although those model-based metrics have shown promising results in modern applications and translation quality estimation, they still show their own shortcomings as follows.&#10;First, they often handle one specific evaluation scenario,~e.g., COMET serves source-reference-only evaluation, where the source and reference sentence should be concurrently fed to the model for prediction.&#10;For the other evaluation scenarios, they hardly give accurate predictions, showing the straits of metric models due to the disagreement between training and inference.&#10;Besides, recent studies have investigated the feasibility of unifying those evaluation scenarios into one single model, which can further improve the evaluation correlation with human ratings in any scenario among source-only, reference-only, and source-reference-combined evaluation~\cite{wan-etal-2021-robleurt,wan-etal-2022-unite}.&#10;This indicates that, training with multiple input formats than a specific one can deliver more appropriate predictions for translation quality identification.&#10;More importantly, unifying all translation evaluation functionalities into one single model can serve as a more convenient toolkit in real-world applications.&#10;&#10;Following the idea of~\newcite{wan-etal-2022-unite} and the experience in previous competition~\cite{wan-etal-2021-robleurt}, we directly use the pipeline of UniTE~\cite{wan-etal-2022-unite} to build models for this year's metric task.&#10;Each of our models can integrate the functionalities of source-only, reference-only, and source-reference-combined translation evaluation into itself.&#10;When collecting the system outputs for the WMT 2022 Metrics Shared Task, we employ our~UniTE models to predict the translation quality scores following the source-reference-combined setting.&#10;Compared to the previous version of~UniTE~\cite{wan-etal-2022-unite}, we reform the synthetic training set for the continuous pre-training phase, raising the ratio of training examples consisting of high-quality hypothesis sentences.&#10;Also, during fine-tuning our metric model, we apply available Direct Assessment~\cite[DA,][]{bojar-etal-2017-results,ma-etal-2018-results,ma-etal-2019-results,mathur-etal-2020-results} and Multidimensional Quality Metrics datasets~\cite[MQM,][]{freitag-etal-2021-experts,freitag-etal-2021-results} from previous WMT competitions to further improve the performance of our model.&#10;Specifically, for each translation direction among English to German (En-De), English to Russian (En-Ru), and Chinese to English (Zh-En) directions, we applied different ensembling strategies to achieve a better correlation with human ratings on MQM 2021 dataset.&#10;Results on WMT 2021 MQM dataset further demonstrate the effectiveness of our method.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Artificial Intelligence, Evaluation Metrics, Language Modeling, Statistics" />
        </attvalues>
      </node>
      <node id="2204.06812" label="2204.06812">
        <attvalues>
          <attvalue for="0" value="Learning to Generalize to More: Continuous Semantic Augmentation for&#10;  Neural Machine Translation" />
          <attvalue for="1" value="  The principal task in supervised neural machine translation (NMT) is to learn&#10;to generate target sentences conditioned on the source inputs from a set of&#10;parallel sentence pairs, and thus produce a model capable of generalizing to&#10;unseen instances. However, it is commonly observed that the generalization&#10;performance of the model is highly influenced by the amount of parallel data&#10;used in training. Although data augmentation is widely used to enrich the&#10;training data, conventional methods with discrete manipulations fail to&#10;generate diverse and faithful training samples. In this paper, we present a&#10;novel data augmentation paradigm termed Continuous Semantic Augmentation&#10;(CsaNMT), which augments each training instance with an adjacency semantic&#10;region that could cover adequate variants of literal expression under the same&#10;meaning. We conduct extensive experiments on both rich-resource and&#10;low-resource settings involving various language pairs, including WMT14&#10;English-{German,French}, NIST Chinese-English and multiple low-resource IWSLT&#10;translation tasks. The provided empirical evidences show that CsaNMT sets a new&#10;level of performance among existing augmentation techniques, improving on the&#10;state-of-the-art by a large margin. The core codes are contained in Appendix E.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Data Augmentation (DA)~\cite{edunov-etal-2018-understanding,kobayashi-2018-contextual,gao-etal-2019-soft,khayrallah-etal-2020-simulated,pham2021meta} has been widely used in neural machine translation. The most popular one is the family of back-translation~\cite{sennrich-etal-2016-improving,NEURIPS2020_7221e5c8}, which utilizes a target-to-source model to translate monolingual target sentences back into the source language. Besides, constructing adversarial training instances with diverse literal forms via word replacing or embedding interpolating~\cite{wang-etal-2018-switchout,cheng-etal-2020-advaug} is beneficial to improve the generalization performance of NMT models.&#10;&#10;Vicinal Risk Minimization (VRM)~\cite{Chapelle2020VRM} is another principle of data augmentation, in which DA is formalized as extracting additional pseudo samples from the vicinal distribution of observed instances. Typically the vicinity of each training instance is defined artificially according to the characteristics of the dataset (or task), such as color (scale, mixup) augmentation~\cite{simonyan2014very,Krizhevsky2012ImageNet,zhang2018mixup} in computer vision and adversarial augmentation with manifold neighborhoods~\cite{ng-etal-2020-ssmba,cheng2021self} in NLP. Our approach relates to VRM that involves with an adjacency semantic region as the vicinity manifold for each training instance.&#10;&#10;Sentence Representation Learning is a well investigated area with dozens of methods~\cite{kiros2015skip,cer2018universal,yang-etal-2018-learning}. In recent years, the methods built on large pre-trained models~\cite{devlin-etal-2019-bert,conneau-etal-2020-unsupervised} have been widely used for learning sentence level representations~\cite{reimers-gurevych-2019-sentence,huang-etal-2019-unicoder,yang-etal-2019-sentence}. Our work is also related to the methods that aims at learning the universal representation~\cite{zhang-etal-2016-variational-neural,schwenk2017learning,yang-etal-2021-universal} for multiple semantically-equivalent sentences in NMT. In this context, contrastive learning has become a popular paradigm in NLP~\cite{Kong2020A,DBLP:conf/iclr/ClarkLLM20,gao-etal-2021-simcse}. The most related work are \cite{wei2021on} and \cite{chi-etal-2021-infoxlm}, which suggested transforming cross-lingual sentences into a shared vector by contrastive objectives.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2011.01536" label="2011.01536">
        <attvalues>
          <attvalue for="0" value="TransQuest: Translation Quality Estimation with Cross-lingual&#10;  Transformers" />
          <attvalue for="1" value="  Recent years have seen big advances in the field of sentence-level quality&#10;estimation (QE), largely as a result of using neural-based architectures.&#10;However, the majority of these methods work only on the language pair they are&#10;trained on and need retraining for new language pairs. This process can prove&#10;difficult from a technical point of view and is usually computationally&#10;expensive. In this paper we propose a simple QE framework based on&#10;cross-lingual transformers, and we use it to implement and evaluate two&#10;different neural architectures. Our evaluation shows that the proposed methods&#10;achieve state-of-the-art results outperforming current open-source quality&#10;estimation frameworks when trained on datasets from WMT. In addition, the&#10;framework proves very useful in transfer learning settings, especially when&#10;dealing with low-resourced languages, allowing us to obtain very competitive&#10;results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;During the past decade there has been tremendous progress in the field of quality estimation, largely as a result of the QE shared tasks organised annually by the Workshops on Statistical Machine Translation (WMT), more recently called the Conferences on Machine Translation, since 2012. The annotated datasets these shared tasks released each year have led to the development of many open-source QE systems like QuEst \cite{specia-etal-2013-quest}, QuEst++ \cite{specia-etal-2015-multi}, deepQuest \cite{ive-etal-2018-deepquest}, and OpenKiwi \cite{kepler-etal-2019-openkiwi}. Before the neural network era, most of the quality estimation systems like QuEst \cite{specia-etal-2013-quest} and QuEst++ \cite{specia-etal-2015-multi} were heavily dependent on linguistic processing and feature engineering to train traditional machine-learning algorithms like support vector regression and randomised decision trees \cite{specia-etal-2013-quest}. Even though, they provided good results, these traditional approaches are no longer the state of the art. In recent years, neural-based QE systems have consistently topped the leader boards in WMT quality estimation shared tasks \cite{kepler-etal-2019-openkiwi}. &#10;&#10;For example, the best-performing system at the WMT 2017 shared task on QE was POSTECH, which is purely neural and does not rely on feature engineering at all \cite{kim-etal-2017-predictor}. POSTECH revolves around an encoder-decoder Recurrent Neural Network (RNN) (referred to as the 'predictor'), stacked with a bidirectional RNN (the 'estimator') that produces quality estimates. In the predictor, an encoder-decoder RNN model predicts words based on their context representations and in the estimator step there is a bidirectional RNN model to produce quality estimates for words, phrases and sentences based on representations from the predictor. To be effective, POSTECH requires extensive predictor pre-training, which means it depends on large parallel data and is computationally intensive \cite{ive-etal-2018-deepquest}. The POSTECH architecture was later re-implemented in deepQuest \cite{ive-etal-2018-deepquest}. &#10;&#10;OpenKiwi \cite{kepler-etal-2019-openkiwi} is another open-source QE framework developed by Unbabel. It implements four different neural network architectures QUETCH \cite{kreutzer-etal-2015-quality}, NuQE \cite{martins-etal-2016-unbabels}, Predictor-Estimator \cite{kim-etal-2017-predictor} and a stacked model of those architectures. Both the QUETCH and NuQE architectures have simple neural network models that do not rely on additional parallel data, but do not perform that well. The Predictor-Estimator model is similar to the POSTECH architecture and relies on additional parallel data. In OpenKiwi, the best performance for sentence-level quality estimation was given by the stacked model that used the Predictor-Estimator model, meaning that the best model requires extensive predictor pre-training and relies on large parallel data and computational resources. &#10;&#10;In order to remove the dependency on large parallel data, which also entails the need for powerful computational resources, we propose to use crosslingual embeddings that are already fine-tuned to reflect properties between languages. We assume that by using them we will ease the burden of having complex neural network architectures. Over the last few years there has been significant work done in the area of crosslingual embeddings \cite{10.1613/jair.1.11640}. &#10;&#10;Since the introduction of BERT \cite{devlin2019bert}, transformer models have been used successfully for various NLP tasks such as named entity recognition \cite{devlin2019bert}, sentence classification \cite{10.1007/978-3-030-32381-3_16}, and question answering \cite{devlin2019bert}, in many cases improving the state of the art. Most of the tasks were focused on English due to the fact that most of the pre-trained transformer models were trained on English data. Although there are several multilingual models like multilingual BERT (mBERT) \cite{devlin2019bert} and multilingual DistilBERT (mDistilBERT) \cite{Sanh2019DistilBERTAD}, researchers expressed some reservations about their ability to represent all the languages \cite{pires-etal-2019-multilingual}. In addition, although mBERT and mDistilBERT showed some crosslingual characteristics, they do not perform well on crosslingual benchmarks \cite{karthikeyan2020cross}. &#10;&#10;XLM-RoBERTa (XML-R) was released in November 2019 \cite{conneau2019unsupervised} as an update to the XLM-100 model \cite{lample2019cross}. XLM-R takes a step back from XLM, eschewing XLM's Translation Language Modeling (TLM) objective since it requires a dataset of parallel sentences, which can be difficult to acquire. Instead, XLM-R trains RoBERTa\cite{liu2019roberta} on a huge, multilingual dataset at an enormous scale: unlabelled text in 104 languages is extracted from CommonCrawl datasets, totalling 2.5TB of text. It is trained using only RoBERTa's \cite{liu2019roberta} masked language modelling (MLM) objective. Surprisingly, this strategy provided better results in crosslingual tasks. XLM-R outperforms mBERT on a variety of crosslingual benchmarks such as crosslingual natural language inference and crosslingual question answering&#10; \cite{conneau2019unsupervised}. &#10;&#10;Both architectures proposed in \TransQuest have been successfully applied in the monolingual semantic textual similarity tasks \cite{devlin2019bert,reimers-gurevych-2019-sentence}. When applied in monolingual experiments, both of them use monolingual transformer models like BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta} as the input. This inspired us to change the input in such a way that it can represent both the source and target sentences for which the quality of translation needs to be estimated, with the hope that the same architectures would also provide good results in the QE task. Our initial experiments showed that crosslingual embeddings like XLM-R provide better results than multilingual embeddings like mBERT. Therefore, in this research we explore the performance of crosslingual embeddings with simple neural network architectures for the sentence-level quality estimation task. To the best of our knowledge, state-of-the-art crosslingual contextual embeddings such as XLM-R have not been used in quality estimation before.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2311.14740" label="2311.14740">
        <attvalues>
          <attvalue for="0" value="AutoKG: Efficient Automated Knowledge Graph Generation for Language&#10;  Models" />
          <attvalue for="1" value="  Traditional methods of linking large language models (LLMs) to knowledge&#10;bases via the semantic similarity search often fall short of capturing complex&#10;relational dynamics. To address these limitations, we introduce AutoKG, a&#10;lightweight and efficient approach for automated knowledge graph (KG)&#10;construction. For a given knowledge base consisting of text blocks, AutoKG&#10;first extracts keywords using a LLM and then evaluates the relationship weight&#10;between each pair of keywords using graph Laplace learning. We employ a hybrid&#10;search scheme combining vector similarity and graph-based associations to&#10;enrich LLM responses. Preliminary experiments demonstrate that AutoKG offers a&#10;more comprehensive and interconnected knowledge retrieval mechanism compared to&#10;the semantic similarity search, thereby enhancing the capabilities of LLMs in&#10;generating more insightful and relevant outputs.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs) such as BERT \cite{devlin2018bert}, RoBERTA \cite{liu2019roberta}, T5 \cite{raffel2020exploring}, and PaLM \cite{chowdhery2022palm}, are intricately designed architectures equipped with an extensive number of parameters. These models have been rigorously pre-trained on vast and diverse corpora, thereby enabling them to excel in a wide array of Natural Language Processing (NLP) tasks, from language understanding to both conditional and unconditional text generation \cite{tirumala2022memorization, zhou2022learning}. These advancements have been heralded as a step toward higher-bandwidth human-computer interactions. However, their deployment faces significant challenges. On one hand, LLMs exhibit a tendency for 'hallucinations' \cite{welleck2019neural, ji2023survey}, providing plausible yet nonfactual predictions. On the other hand, the black-box nature of LLMs compromises both interpretability and factual accuracy, often resulting in erroneous statements despite memorizing facts during training \cite{petroni2019language, scialom-etal-2022-fine}.&#10;&#10;Knowledge in natural language can be externally sourced from a retrievable database, reducing hallucinations and enhancing the interpretability of LLMs \cite{mialon2023augmented}.&#10;Utilizing dense neural retrievers, which employ dense query and document vectors generated by a neural network \cite{asai2021one}, the system can evaluate the semantic similarity to an information-seeking query by calculating the embedding vector similarity across related concepts \cite{lewis2020retrieval, luan2021sparse}.&#10;&#10;To go beyond mere semantic similarity in information retrieval and augment the reasoning capabilities of LLMs, two advanced methodologies are particularly transformative: prompt engineering like the Chain-of-thought prompting, and the incorporation of Knowledge Graphs (KGs)\cite{pan2023unifying}. The former, chain-of-thought prompting, provides a framework for advanced reasoning by generating paths of explanations and predictions that are cross-verified through knowledge retrieval \cite{he2022rethinking, trivedi2022interleaving}. While this method offers significant benefits, it is not the primary focus of this study. As for the latter, KGs offer LLMs a structured and efficient way to address their limitations in factual accuracy and reasoning \cite{xiong2019improving, pan2023unifying}. KGs not only provide accurate and explicit knowledge crucial for various applications \cite{ji2021survey} but are also known for their symbolic reasoning capabilities to produce interpretable results \cite{zhang2021neural}. These graphs are dynamic, continuously evolving with the addition of new knowledge \cite{mitchell2018never}, and can be specialized for domain-specific requirements \cite{abu2021domain}.&#10;&#10;In this study, our emphasis is on techniques of automated KG generation and incorporation with LLMs. Most of the works related to these two tasks rely intensively on the ongoing training of neural networks \cite{pan2023unifying, zhong2023comprehensive}, which is both difficult to employ and less flexible for on-the-fly updates. Traditional KG construction approach uses NLP techniques for entity recognition\cite{nadeau2007survey, grishman1996message}, or keyword identification based on term frequency\cite{salton1988term, ramos2003using}, followed by determining relationship strength through word proximity\cite{mintz2009distant}. Current automated techniques necessitate neural network training \cite{luo2023normalizing,wan2021reasoning, wang2023knowledge}. As for the interaction between KGs and LLMs, neural networks are trained to let LLMs understand the information retrieved from KGs \cite{tian2023graph,yasunaga2022deep}. &#10;&#10;The recent advancements in LLMs make us think much more simply about the automatic generation of KGs and the integration of LLMs with KGs. State-of-the-art LLMs such as ChatGPT, BARD, and LLAMA\cite{touvron2023llama} have demonstrated impressive reasoning capabilities \cite{bang2023multitask, agarwal2023analysing}. Given sufficient information, they can independently execute effective inference. This observation suggests an opportunity to simplify the KG structure: perhaps the intricate relational patterns found in traditional KGs could be simplified into basic strength indicators of association. Consequently, specific relationships are implicitly conveyed to the model through corpus blocks associated with the KG. In addition, we can provide retrieved keywords and the related corpus directly in the prompt rather than training a network to let LLMs understand the retrieved subgraph structure.&#10;&#10;Motivated by these ideas, this study makes the following contributions:&#10;\begin{enumerate}&#10;\item We introduce AutoKG, an innovative method for automated KG generation, based on a knowledge base comprised of text blocks. AutoKG circumvents the need for training or fine-tuning neural networks, employs pretrained LLMs for extracting keywords as nodes, and applies graph Laplace learning to evaluate the edge weights between these keywords. The output is a simplified KG, where edges lack attributes and directionality, possessing only a weight that signifies the relevance between nodes.&#10;\item We present a hybrid search strategy in tandem with prompt engineering, which empowers large LLMs to effectively utilize information from the generated KGs. This approach simultaneously searches for semantically relevant corpora based on embedding vectors and the most pertinent adjacent information within the knowledge graphs.&#10;\end{enumerate}&#10;&#10;The KG constructed here is a simplified version compared to traditional KGs, which are typically composed of relations in the form of triplets. Firstly, nodes in AutoKG are not entities in the usual sense; they are more abstract keywords. These keywords can represent entities, concepts, or any content that serves as a foundation for search. Additionally, instead of directed edges with specific semantic meanings found in traditional KGs, AutoKG utilizes undirected edges with a single weight value. The node keywords are extracted from the knowledge base with the aid of LLMs, while the graph structure is algorithmically derived. Such a KG can be efficiently stored with just a keyword list and a sparse adjacency matrix. &#10;&#10;Section~\ref{sec: KG_generation} explains the detailed process of automated KG generation, while Section~\ref{sec:hybrid_search} describes the hybrid search method. An essential highlight is that our proposed techniques require no neural network training or fine-tuning. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Knowledge Graph Construction, Language Model Enhancement, Artificial Intelligence, Mathematics, Natural Language Processing, Semantic Search Optimization" />
        </attvalues>
      </node>
      <node id="2107.11976" label="2107.11976">
        <attvalues>
          <attvalue for="0" value="One Question Answering Model for Many Languages with Cross-lingual Dense&#10;  Passage Retrieval" />
          <attvalue for="1" value="  We present Cross-lingual Open-Retrieval Answer Generation (CORA), the first&#10;unified many-to-many question answering (QA) model that can answer questions&#10;across many languages, even for ones without language-specific annotated data&#10;or knowledge sources. We introduce a new dense passage retrieval algorithm that&#10;is trained to retrieve documents across languages for a question. Combined with&#10;a multilingual autoregressive generation model, CORA answers directly in the&#10;target language without any translation or in-language retrieval modules as&#10;used in prior work. We propose an iterative training method that automatically&#10;extends annotated data available only in high-resource languages to&#10;low-resource ones. Our results show that CORA substantially outperforms the&#10;previous state of the art on multilingual open QA benchmarks across 26&#10;languages, 9 of which are unseen during training. Our analyses show the&#10;significance of cross-lingual retrieval and generation in many languages,&#10;particularly under low-resource settings.&#10;" />
          <attvalue for="2" value="&#10;Multilingual open question answering (QA) is the task of answering a question from a large collection of multilingual documents. Most recent progress in open QA is made for English by building a pipeline based on a dense passage retriever trained on large-scale English QA datasets to find evidence passages in English ~\cite{lee-chang-toutanova:2019:ACL2019,karpukhin2020dense}, followed by a reader that extracts an answer from retrieved passages. &#10;However, extending this approach to multilingual open QA poses new challenges. &#10;Answering multilingual questions requires retrieving evidence from knowledge sources of other languages than the original question since many languages have limited reference documents or the question sometimes inquires about concepts from other cultures \cite{xorqa,lin2020pretrained}. &#10;Nonetheless, large-scale cross-lingual open QA training data whose questions and evidence are in different languages are not available in many of those languages.&#10;&#10;To address these challenges, previous work in multilingual open QA~\cite{ture-boschee-2016-learning,xorqa} translates questions into English, applies an English open QA system to answer in English, and then translates answers back to the target language. &#10;Those pipeline approaches suffer from error propagation of the machine translation component into the downstream QA, especially for low-resource languages. Moreover, they are not able to answer questions whose answers can be found in resources written in languages other than English or the target languages. &#10;&#10;In this paper, we introduce a unified many-to-many QA model that can answer questions in any target language by retrieving evidence from any language and generating answers in the target language.&#10;Our method (called CORA, Fig.~\ref{img:method_overview}) extends the {retrieve-then-generate} approach of English open QA~\cite{lewis2020retrieval,izacard2020leveraging} with a single cross-lingual retriever and a generator that do not rely on language-specific retrievers or machine translation modules. &#10;The multilingual retrieval module ({mDPR}) produces dense embeddings of a question and all multilingual passages, thereby retrieving passages across languages.&#10;The generation module ({mGEN}) is trained to output an answer in the target language conditioned on the retrieved multilingual passages. &#10;To overcome the aforementioned data scarcity issue, we automatically mine training data using external language links and train mDPR and mGEN iteratively.&#10;In particular, each iteration proceeds over two stages of updating model parameters with available training data and mining new training data {cross-lingually} by Wikipedia language links and predictions made by the models.&#10;This approach does not require any additional human annotations or machine translation, and can be applied to many new languages with low resources. &#10;&#10;Our experiments show that CORA advances the state of the art on two multilingual open QA datasets, Xor-TyDi QA \cite{xorqa} and MKQA \cite{mkqa}, across 26 typologically diverse languages; CORA achieves gains of 23.4 and 4.7 F1 points in Xor-TyDi QA and MKQA respectively, where MKQA data is not used for training. &#10;Moreover, CORA achieves F1 scores of roughly 30 over 8 languages on MKQA that have no training data or even reference Wikipedia documents, outperforming the state-of-the-art approach by 5.4 F1 points. &#10;Our controlled experiments and human analyses illustrate the impact of many-to-many cross-lingual retrieval in improving multilingual open QA performance.&#10;We further observe that through cross-lingual retrieval, CORA can find answers to 20\% of the multilingual questions that are valid but are originally annotated as {unanswerable} by humans due to the lack of evidence in the English knowledge sources.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.07098" label="1905.07098">
        <attvalues>
          <attvalue for="0" value="Improving Question Answering over Incomplete KBs with Knowledge-Aware&#10;  Reader" />
          <attvalue for="1" value="  We propose a new end-to-end question answering model, which learns to&#10;aggregate answer evidence from an incomplete knowledge base (KB) and a set of&#10;retrieved text snippets. Under the assumptions that the structured KB is easier&#10;to query and the acquired knowledge can help the understanding of unstructured&#10;text, our model first accumulates knowledge of entities from a question-related&#10;KB subgraph; then reformulates the question in the latent space and reads the&#10;texts with the accumulated entity knowledge at hand. The evidence from KB and&#10;texts are finally aggregated to predict answers. On the widely-used KBQA&#10;benchmark WebQSP, our model achieves consistent improvements across settings&#10;with different extents of KB incompleteness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.00388" label="2002.00388">
        <attvalues>
          <attvalue for="0" value="A Survey on Knowledge Graphs: Representation, Acquisition and&#10;  Applications" />
          <attvalue for="1" value="  Human knowledge provides a formal understanding of the world. Knowledge&#10;graphs that represent structural relations between entities have become an&#10;increasingly popular research direction towards cognition and human-level&#10;intelligence. In this survey, we provide a comprehensive review of knowledge&#10;graph covering overall research topics about 1) knowledge graph representation&#10;learning, 2) knowledge acquisition and completion, 3) temporal knowledge graph,&#10;and 4) knowledge-aware applications, and summarize recent breakthroughs and&#10;perspective directions to facilitate future research. We propose a full-view&#10;categorization and new taxonomies on these topics. Knowledge graph embedding is&#10;organized from four aspects of representation space, scoring function, encoding&#10;models, and auxiliary information. For knowledge acquisition, especially&#10;knowledge graph completion, embedding methods, path inference, and logical rule&#10;reasoning, are reviewed. We further explore several emerging topics, including&#10;meta relational learning, commonsense reasoning, and temporal knowledge graphs.&#10;To facilitate future research on knowledge graphs, we also provide a curated&#10;collection of datasets and open-source libraries on different tasks. In the&#10;end, we have a thorough outlook on several promising research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05446" label="2010.05446">
        <attvalues>
          <attvalue for="0" value="Neural, Symbolic and Neural-Symbolic Reasoning on Knowledge Graphs" />
          <attvalue for="1" value="  Knowledge graph reasoning is the fundamental component to support machine&#10;learning applications such as information extraction, information retrieval,&#10;and recommendation. Since knowledge graphs can be viewed as the discrete&#10;symbolic representations of knowledge, reasoning on knowledge graphs can&#10;naturally leverage the symbolic techniques. However, symbolic reasoning is&#10;intolerant of the ambiguous and noisy data. On the contrary, the recent&#10;advances of deep learning promote neural reasoning on knowledge graphs, which&#10;is robust to the ambiguous and noisy data, but lacks interpretability compared&#10;to symbolic reasoning. Considering the advantages and disadvantages of both&#10;methodologies, recent efforts have been made on combining the two reasoning&#10;methods. In this survey, we take a thorough look at the development of the&#10;symbolic, neural and hybrid reasoning on knowledge graphs. We survey two&#10;specific reasoning tasks, knowledge graph completion and question answering on&#10;knowledge graphs, and explain them in a unified reasoning framework. We also&#10;briefly discuss the future directions for knowledge graph reasoning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00235" label="2011.00235">
        <attvalues>
          <attvalue for="0" value="Domain-specific Knowledge Graphs: A survey" />
          <attvalue for="1" value="  Knowledge Graphs (KGs) have made a qualitative leap and effected a real&#10;revolution in knowledge representation. This is leveraged by the underlying&#10;structure of the KG which underpins a better comprehension, reasoning and&#10;interpretation of knowledge for both human and machine. Therefore, KGs continue&#10;to be used as the main means of tackling a plethora of real-life problems in&#10;various domains. However, there is no consensus in regard to a plausible and&#10;inclusive definition of a domain-specific KG. Further, in conjunction with&#10;several limitations and deficiencies, various domain-specific KG construction&#10;approaches are far from perfect. This survey is the first to offer a&#10;comprehensive definition of a domain-specific KG. Also, the paper presents a&#10;thorough review of the state-of-the-art approaches drawn from academic works&#10;relevant to seven domains of knowledge. An examination of current approaches&#10;reveals a range of limitations and deficiencies. At the same time, uncharted&#10;territories on the research map are highlighted to tackle extant issues in the&#10;literature and point to directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.08183" label="2304.08183">
        <attvalues>
          <attvalue for="0" value="Normalizing Flow-based Neural Process for Few-Shot Knowledge Graph&#10;  Completion" />
          <attvalue for="1" value="  Knowledge graphs (KGs), as a structured form of knowledge representation,&#10;have been widely applied in the real world. Recently, few-shot knowledge graph&#10;completion (FKGC), which aims to predict missing facts for unseen relations&#10;with few-shot associated facts, has attracted increasing attention from&#10;practitioners and researchers. However, existing FKGC methods are based on&#10;metric learning or meta-learning, which often suffer from the&#10;out-of-distribution and overfitting problems. Meanwhile, they are incompetent&#10;at estimating uncertainties in predictions, which is critically important as&#10;model predictions could be very unreliable in few-shot settings. Furthermore,&#10;most of them cannot handle complex relations and ignore path information in&#10;KGs, which largely limits their performance. In this paper, we propose a&#10;normalizing flow-based neural process for few-shot knowledge graph completion&#10;(NP-FKGC). Specifically, we unify normalizing flows and neural processes to&#10;model a complex distribution of KG completion functions. This offers a novel&#10;way to predict facts for few-shot relations while estimating the uncertainty.&#10;Then, we propose a stochastic ManifoldE decoder to incorporate the neural&#10;process and handle complex relations in few-shot settings. To further improve&#10;performance, we introduce an attentive relation path-based graph neural network&#10;to capture path information in KGs. Extensive experiments on three public&#10;datasets demonstrate that our method significantly outperforms the existing&#10;FKGC methods and achieves state-of-the-art performance. Code is available at&#10;https://github.com/RManLuo/NP-FKGC.git.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.15427" label="2309.15427">
        <attvalues>
          <attvalue for="0" value="Graph Neural Prompting with Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have shown remarkable generalization capability&#10;with exceptional performance in various language modeling tasks. However, they&#10;still exhibit inherent limitations in precisely capturing and returning&#10;grounded knowledge. While existing work has explored utilizing knowledge graphs&#10;(KGs) to enhance language modeling via joint training and customized model&#10;architectures, applying this to LLMs is problematic owing to their large number&#10;of parameters and high computational cost. Therefore, how to enhance&#10;pre-trained LLMs using grounded knowledge, e.g., retrieval-augmented&#10;generation, remains an open question. In this work, we propose Graph Neural&#10;Prompting (GNP), a novel plug-and-play method to assist pre-trained LLMs in&#10;learning beneficial knowledge from KGs. GNP encompasses various designs,&#10;including a standard graph neural network encoder, a cross-modality pooling&#10;module, a domain projector, and a self-supervised link prediction objective.&#10;Extensive experiments on multiple datasets demonstrate the superiority of GNP&#10;on both commonsense and biomedical reasoning tasks across different LLM sizes&#10;and settings. Code is available at https://github.com/meettyj/GNP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.09338" label="2210.09338">
        <attvalues>
          <attvalue for="0" value="Deep Bidirectional Language-Knowledge Graph Pretraining" />
          <attvalue for="1" value="  Pretraining a language model (LM) on text has been shown to help various&#10;downstream NLP tasks. Recent works show that a knowledge graph (KG) can&#10;complement text data, offering structured background knowledge that provides a&#10;useful scaffold for reasoning. However, these works are not pretrained to learn&#10;a deep fusion of the two modalities at scale, limiting the potential to acquire&#10;fully joint representations of text and KG. Here we propose DRAGON (Deep&#10;Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach&#10;to pretraining a deeply joint language-knowledge foundation model from text and&#10;KG at scale. Specifically, our model takes pairs of text segments and relevant&#10;KG subgraphs as input and bidirectionally fuses information from both&#10;modalities. We pretrain this model by unifying two self-supervised reasoning&#10;tasks, masked language modeling and KG link prediction. DRAGON outperforms&#10;existing LM and LM+KG models on diverse downstream tasks including question&#10;answering across general and biomedical domains, with +5% absolute gain on&#10;average. In particular, DRAGON achieves notable performance on complex&#10;reasoning about language and knowledge (+10% on questions involving long&#10;contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and&#10;RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our&#10;code and trained models are available at&#10;https://github.com/michiyasunaga/dragon.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.17399" label="2306.17399">
        <attvalues>
          <attvalue for="0" value="Japanese Lexical Complexity for Non-Native Readers: A New Dataset" />
          <attvalue for="1" value="  Lexical complexity prediction (LCP) is the task of predicting the complexity&#10;of words in a text on a continuous scale. It plays a vital role in simplifying&#10;or annotating complex words to assist readers. To study lexical complexity in&#10;Japanese, we construct the first Japanese LCP dataset. Our dataset provides&#10;separate complexity scores for Chinese/Korean annotators and others to address&#10;the readers' L1-specific needs. In the baseline experiment, we demonstrate the&#10;effectiveness of a BERT-based system for Japanese LCP.&#10;" />
          <attvalue for="2" value="&#10;&#10;Reading comprehension requires a certain level of vocabulary knowledge.&#10;The results reported by \cite{Hu2000} suggest that most English learners need to understand 98\% of tokens in a text to comprehend it.&#10;A follow-up study by \cite{Komori2004-es} estimates the percentage to be 96\% for Japanese learners to comprehend text.&#10;Acquiring vocabulary to reach such levels, in turn, is a lengthy and challenging task for learners.&#10;This opens up opportunities for assistive applications, such as simplification or annotation of complex words.&#10;The first step necessary for such applications is to predict the complexity of the words.&#10;The task of lexical complexity prediction (LCP) is defined as predicting how difficult to comprehend words or phrases in a text are on a continuous scale \cite{Shardlow2020-rd}. This differentiates LCP from complex word identification (CWI), i.e., binary classification of complex words \cite{Yimam2018-rw}.&#10;As complexity is naturally perceived as continuous, a continuous scale used in LCP allows to represent it without loss of information.&#10;&#10;The LCP research so far has been limited to English, for which two LCP datasets have been constructed \cite{Shardlow2020-rd, Shardlow2022-predicting}, and no such dataset has been created for Japanese. &#10;Meanwhile, there are a number of features specific to the Japanese language that could affect lexical complexity, and their effects have yet to be studied. For example, the Chinese characters, which are used extensively in Japanese, lower text readability \cite{Tateisi1988-yz}.&#10;&#10;Previous studies on Japanese lexical complexity used pedagogical word lists to estimate complexity level. &#10;\cite{Nishihara2020-oh} modeled lexical complexity of words based on the Japanese Educational Vocabulary List \cite{Sunakawa2012-dm}.&#10;The word list assigns a degree of difficulty to each item, based on the subjective judgment of Japanese language teachers, not learners themselves, and does not consider the learners' L1 background.&#10;&#10;In light of this, we present JaLeCoN, Dataset of Japanese Lexical Complexity for Non-Native Readers.&#10;Our dataset has the following key features:&#10;&#10;\begin{asparaenum}[(1)]&#10; \item Complexity scores for single words as well as multi-word expressions (MWEs);&#10; \item Separate complexity scores from Chinese/Korean annotators and others, addressing the considerable advantage of the former in Japanese reading comprehension.&#10;\end{asparaenum}&#10;&#10;Our analysis reveals that the non-Chinese/Ko\-rean annotators perceive words of Chinese origin or containing Chinese characters as especially complex.&#10;In the baseline experiment, we investigate the effectiveness of a BERT-based system in the Japanese LCP task, and how it varies according to the word complexity and L1 background.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Computational Linguistics, Lexical Complexity, Natural Language Processing, Language Modeling, Psychology" />
        </attvalues>
      </node>
      <node id="2405.10443" label="2405.10443">
        <attvalues>
          <attvalue for="0" value="Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in&#10;  Fine-tuning LLMs for Simultaneous Translation" />
          <attvalue for="1" value="  Large language models (LLMs) have achieved state-of-the-art performance in&#10;various language processing tasks, motivating their adoption in simultaneous&#10;translation. Current fine-tuning methods to adapt LLMs for simultaneous&#10;translation focus on prompting optimization strategies using either data&#10;augmentation or prompt structure modifications. However, these methods suffer&#10;from several issues, such as unnecessarily expanded training sets,&#10;computational inefficiency from dumping the key and value cache, increased&#10;prompt sizes, or restriction to a single decision policy. To eliminate these&#10;issues, in this work, we propose SimulMask, a new paradigm for fine-tuning LLMs&#10;for simultaneous translation. It utilizes a novel attention mask approach that&#10;models simultaneous translation during fine-tuning by masking attention for a&#10;desired decision policy. Applying the proposed SimulMask on a Falcon LLM for&#10;the IWSLT 2017 dataset, we have observed a significant translation quality&#10;improvement compared to state-of-the-art prompting optimization strategies on&#10;five language pairs while reducing the computational cost.&#10;" />
          <attvalue for="2" value="&#10;&#10;Simultaneous translation refers to the process of producing a target output translation concurrently with an oncoming source input. In our increasingly interconnected world, where communication across languages in real-time is desired, simultaneous translation is becoming a requirement. As such, there is a need for machine learning models to fill this role. &#10;&#10;Current literature has primarily focused on adapting end-to-end Transformer models \cite{vaswani-et-al-2017} to overcome the difficulties of simultaneous machine translation (SimulMT) due to their reduced parameter counts and greater inference speed\cite{ma2020simulmt}. However, the recent successes of large language models (LLMs) \cite{touvron2023llama, jiang2023mistral, almazrouei2023falcon} has prompted preliminary research applying them to SimulMT through fine-tuning and inference techniques \cite{agostinelli2023simul, wang2023simultaneous, koshkin2024transllama, wang2024conversational, guo2024sillm}. Unfortunately, most modern works have neglected the computational increases created by dumping the target sequence's key and value (KV) cache \cite{wang2024conversational}. Furthermore, there has yet to be a universal approach to fine-tuning LLMs for SimulMT that is not unnecessarily computationally expensive by either expanding the dataset through data augmentation, a process referred to as prefix fine-tuning \cite{agostinelli2023simul, wang2023simultaneous, koshkin2024transllama} or increasing the prompt length through prompt restructuring \cite{koshkin2024transllama, wang2024conversational}. &#10;&#10;The lack of an efficient fine-tuning strategy of LLMs for SimulMT has led us to propose a new paradigm, referred to as SimulMask. SimulMask is a novel attention mask to model SimulMT during fine-tuning by redistributing the attention under a decision policy. By design, SimulMask is broadly applicable to both flexible and fixed decision policies, creating a path forward for future work to build upon it. Furthermore, if we avoid injecting positional information into the keys and values through a modified ALiBi \cite{press2021train}, SimulMask allows for KV caching during SimulMT without accuracy degradation.&#10;&#10;To validate the efficacy of SimulMask, we fine-tuned and evaluated 1.3 billion parameter Falcon models pre-trained on the RefinedWeb dataset using SimulMask \cite{almazrouei2023falcon, refinedweb} and compared them against identical Falcon models that adopt existing prefix fine-tuning or prompt restructuring methods on the IWSLT 2017 dataset \cite{cettolo-etal-2017-overview}. From the results, we demonstrate models fine-tuned with SimulMask outperform prefix fine-tuning and prompt restructuring models at SimulMT for a given latency regime with a reduced computational cost.&#10;&#10;The main contributions of the paper include:&#10;\begin{enumerate}[topsep=1pt, itemsep=-1ex]&#10; \item Providing insights on the shortcomings of current methods in adapting LLMs to SimulMT.&#10; \item Proposing a novel attention masking approach to fine-tune SimulMT LLMs that enables efficient training and inference.&#10; \item Demonstrating the efficacy of our approach in terms of translation quality and computational costs by evaluating them on multiple language pairs across varied latencies.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Learning Techniques, Simultaneous Translation, Artificial Intelligence, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2402.10552" label="2402.10552">
        <attvalues>
          <attvalue for="0" value="Conversational SimulMT: Efficient Simultaneous Translation with Large&#10;  Language Models" />
          <attvalue for="1" value="  Simultaneous machine translation (SimulMT) presents a challenging trade-off&#10;between translation quality and latency. Recent studies have shown that LLMs&#10;can achieve good performance in SimulMT tasks. However, this often comes at the&#10;expense of high inference cost and latency. In this paper, we propose a&#10;conversational SimulMT framework to enhance the inference efficiency of&#10;LLM-based SimulMT through multi-turn-dialogue-based decoding. Our experiments&#10;with Llama2-7b-chat on two SimulMT benchmarks demonstrate the superiority of&#10;LLM in translation quality while achieving comparable computational latency to&#10;specialized SimulMT models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.04636" label="2402.04636">
        <attvalues>
          <attvalue for="0" value="TransLLaMa: LLM-based Simultaneous Translation System" />
          <attvalue for="1" value="  Decoder-only large language models (LLMs) have recently demonstrated&#10;impressive capabilities in text generation and reasoning. Nonetheless, they&#10;have limited applications in simultaneous machine translation (SiMT), currently&#10;dominated by encoder-decoder transformers. This study demonstrates that, after&#10;fine-tuning on a small dataset comprising causally aligned source and target&#10;sentence pairs, a pre-trained open-source LLM can control input segmentation&#10;directly by generating a special &quot;wait&quot; token. This obviates the need for a&#10;separate policy and enables the LLM to perform English-German and&#10;English-Russian SiMT tasks with BLEU scores that are comparable to those of&#10;specific state-of-the-art baselines. We also evaluated closed-source models&#10;such as GPT-4, which displayed encouraging results in performing the SiMT task&#10;without prior training (zero-shot), indicating a promising avenue for enhancing&#10;future SiMT systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.08493" label="1803.08493">
        <attvalues>
          <attvalue for="0" value="Contextual Salience for Fast and Accurate Sentence Vectors" />
          <attvalue for="1" value="  Unsupervised vector representations of sentences or documents are a major&#10;building block for many language tasks such as sentiment classification.&#10;However, current methods are uninterpretable and slow or require large training&#10;datasets. Recent word vector-based proposals implicitly assume that distances&#10;in a word embedding space are equally important, regardless of context. We&#10;introduce contextual salience (CoSal), a measure of word importance that uses&#10;the distribution of context vectors to normalize distances and weights. CoSal&#10;relies on the insight that unusual word vectors disproportionately affect&#10;phrase vectors. A bag-of-words model with CoSal-based weights produces accurate&#10;unsupervised sentence or document representations for classification, requiring&#10;little computation to evaluate and only a single covariance calculation to&#10;``train.&quot; CoSal supports small contexts, out-of context words and outperforms&#10;SkipThought on most benchmarks, beats tf-idf on all benchmarks, and is&#10;competitive with the unsupervised state-of-the-art.&#10;" />
          <attvalue for="2" value="&#10;Global context, a representation of an analyzed text's nature, such as recognizing that one is reading a movie review, is useful for transfer learning and document interpretation \cite{AAAI148361, 2018arXiv180205365P}. Yet, current solutions that account for global context are black boxes: an algorithm takes a document and returns some function mapping word vectors to a context representations. Method outputs vary from a vector \cite{2017arXiv170800107M} to another deep structure \cite{2018arXiv180205365P}. &#10;This results in obfuscated approaches to using context that are uninterpretable and resistant to further extension. Further, these algorithms often still require thousands of sentences in a transfer dataset to train effectively, especially when unsupervised \cite{2017arXiv170800107M}. &#10;We introduce CoSal, an approach relying on normalization of distances in a semantic space with respect to the word vector distribution of a context.&#10;&#10;It is generally accepted that words that are common in a document but rare in a corpus are important to the document: tf-idf is a standard technique to evaluate word importance, comparing document word frequency to overall corpus frequency, used by at least 70\% of text based recommender-systems \cite{Beel2016}. This premise is also the basis of the inverse-frequency model in Arora et al.'s ``A Simple but Tough to Beat Baseline for Sentence Embeddings,&quot; which remains nearly state-of-the-art \cite{arora2017asimple}.&#10;&#10;Finding that CoSal correlates with tf-idf in cases where tf-idf works well (large documents and corpora), we analyze the relationship between the contextual salience of words and their contribution to a compositional representation. In order to do this, we build a vector space with both phrases and words. This space showed a clear relationship: words that are slightly more contextually salient than the others in a sentence contribute much more to the meaning of the sentence than words that are slightly less contextually salient. Combining this sigmoidal importance pattern with a weighted bag-of-words for unsupervised sentence and document representations yields better performance than widely-used supervised and unsupervised classification models on most benchmarks. Then, for especially small corpora, we extend CoSal with a method for augmenting the covariance matrix with that of a broader context.&#10;&#10;Initially, cosine distance was used to measure the distance between a generated sentence vector and a target sentence vector, providing numerous advantages over Euclidean distance \cite{emmery_2017}. However, cosine distance still implies that all dimensions are equally valuable and uncorrelated, calculated as a dot product. Thus, it still produces context-blind measurements. Noting that M-distance can be calculated between any two points given an underlying distribution, an application of the law of cosines is used to define a robust and context-aware alternative to cosine distance, capable of realizing that ``cardinal&quot; (A color but also a metonym for Stanford) and ``red&quot; are less related to one another in the context of news about Stanford research than in an article about the color green. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Computational Linguistics, Vector Representations, Mathematics, Natural Language Processing, Unsupervised Learning" />
        </attvalues>
      </node>
      <node id="2306.03975" label="2306.03975">
        <attvalues>
          <attvalue for="0" value="Revisiting Conversation Discourse for Dialogue Disentanglement" />
          <attvalue for="1" value="  Dialogue disentanglement aims to detach the chronologically ordered&#10;utterances into several independent sessions. Conversation utterances are&#10;essentially organized and described by the underlying discourse, and thus&#10;dialogue disentanglement requires the full understanding and harnessing of the&#10;intrinsic discourse attribute. In this paper, we propose enhancing dialogue&#10;disentanglement by taking full advantage of the dialogue discourse&#10;characteristics. First of all, in feature encoding stage, we construct the&#10;heterogeneous graph representations to model the various dialogue-specific&#10;discourse structural features, including the static speaker-role structures&#10;(i.e., speaker-utterance and speaker-mentioning structure) and the dynamic&#10;contextual structures (i.e., the utterance-distance and partial-replying&#10;structure). We then develop a structure-aware framework to integrate the rich&#10;structural features for better modeling the conversational semantic context.&#10;Second, in model learning stage, we perform optimization with a hierarchical&#10;ranking loss mechanism, which groups dialogue utterances into different&#10;discourse levels and carries training covering pair-wise and session-wise&#10;levels hierarchically. Third, in inference stage, we devise an easy-first&#10;decoding algorithm, which performs utterance pairing under the easy-to-hard&#10;manner with a global context, breaking the constraint of traditional sequential&#10;decoding order. On two benchmark datasets, our overall system achieves new&#10;state-of-the-art performances on all evaluations. In-depth analyses further&#10;demonstrate the efficacy of each proposed idea and also reveal how our methods&#10;help advance the task. Our work has great potential to facilitate broader&#10;multi-party multi-thread dialogue applications.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multi-turn multi-party conversations are often characterized by intertwined utterances with many speakers, and multiple coexisted topic threads \cite{traum-etal-2004-evaluation}.&#10;This adds challenges to the dialogue understanding and responding.&#10;Dialogue disentanglement has thus been proposed with the aim of decomposing entangled utterances from different threads or sessions \cite{kummerfeld-etal-2019-large}, i.e., finding the reply-to relations between the chronologically-listed utterances (cf. Figure \ref{fig:intro}).&#10;Earlier research builds handcrafted discrete features with machine learning models to cluster utterances into different sessions or predict whether there is a replying relation between utterances \cite{elsner-charniak-2008-talking,elsner-charniak-2010-disentangling,icwsm-AumayrCH11}.&#10;Currently, the rapid development of deep neural models has greatly advanced the dialogue disentanglement task \cite{mehri-carenini-2017-chat,liuSGLWZ20-ijcai,jiang-etal-2018-identifying,chengyuhuang-emnlp}, especially by making use of the pre-trained language models (PLM) \cite{aaai-ZhuNWNX20,tiandali-arxiv}, e.g., BERT \cite{devlin-etal-2019-bert}, DiaBERT \cite{tiandali-arxiv}.&#10;As extensively revealed, the essence of the task lies in the understanding of the underlying conversational discourse \cite{zhu-etal-2021-findings,aclMa0Z22}, and thus it is key to model the discourse structure of the dialogue.&#10;Despite the progress of dialogue disentanglement achieved by prior efforts, existing explorations, unfortunately, still need to harness the conversational discourse nature fully.&#10;&#10;At feature modeling perspective, current research fails to sufficiently make use of the conversational discourse structural information.&#10;In the dialogue context, there are various types of discourse structures beneficial to the dialogue disentanglement task.&#10;The first type is the speaker-utterance discourse.&#10;In the multi-party dialog, due to the participant persona consistency, different utterances by the same speaker may exhibit identical styles.&#10;Thus, capturing the speaker-utterance correlations will facilitate the responding recognition.&#10;Second, the speaker-mentioning discourse.&#10;The speaker coreferences indicate the interactions between different utterances and speakers, which are the value clues to the detection of replying relation.&#10;Third, the utterance-distance discourse.&#10;A conversation usually involves a great number of turns,&#10;in which the valid contexts that offer critical features for the replying reasoning actually center around the current utterance (i.e., near neighbors), and the farther away, the lower the efficacy of the features.&#10;The Ubuntu IRC data \cite{kummerfeld-etal-2019-large} statistics shows that the average turn of conversation is 10.17, while 80\%/90\%/95\% replying relations are scattered within 6/13/21 utterances forward and backward, respectively.&#10;Fourth, the partial-replying discourse.&#10;In fact, the incorporation of the previously-discovered partial replying structure is beneficial to the detection of the following replying relation.&#10;For example as shown in Figure \ref{fig:intro}, directly determining the reply-to relation \#6$^{\curvearrowright}$\#3 can be tricky; while knowing the replying relation of \#3$^{\curvearrowright}$\#2 as prior, the detection of \#6$^{\curvearrowright}$\#3 can be greatly eased.&#10;&#10;At system optimization perspective, existing works disregard the dialogue discourse characteristic for model training and decoding.&#10;On the one hand, utterances are governed under both the local replying thread and global session discourses, and thus, the dialogue disentanglement task measures both the session-level and pair-wise detection.&#10;Unfortunately, most current models only train with the pair-wise cross-entropy loss without considering the higher-level optimization (e.g., thread, session) \cite{liuSGLWZ20-ijcai,jiang-etal-2018-identifying,aclMa0Z22}.&#10;On the other hand, existing methods take a front-to-end reading order to decode the replying relation for an utterance with merely the precedent context.&#10;Yet this decoding approach can be less effective and provide less informative results as the semantics of a conversation are organized in a hierarchical structure rather than a linear chronological order.&#10;Intrinsically, as humans, we always first recognize those utterances with simple replying relations, then try the harder cases gradually with more clues.&#10;Taking Figure \ref{fig:intro} as an example with \#5 as the current utterance, it can be easier to first determine the replying dependency of \#8$^{\curvearrowright}$\#5 with the cue word `mac' in the following context.&#10;Based on the established replying structure \#8$^{\curvearrowright}$\#5, the hard one of \#5$^{\curvearrowright}$\#1 can be further detected without much effort.&#10;&#10;In this work, we rethink the conversation discourse to dialogue disentanglement, and propose to enhance the task by giving full consideration to the above observations.&#10;First of all, to take advantage of the rich conversational discourse features ($\S$\ref{Construction of Dialogue Discourse Structures}), we construct the four types of graphs to represent the aforementioned discourse structures.&#10;We consider the two static speaker-role graphs, including the speaker-utterance and speaker-mentioning structures.&#10;Also, we build the two dynamic contextual structures, including a Gaussian-based utterance-distance structure and a dynamically updated partial-replying structure.&#10;We further develop a structure-aware framework ($\S$\ref{Structure-aware Framework}) for dialogue disentanglement.&#10;As shown in Figure \ref{fig:model}, we encode and integrate the various heterogeneous graphs with edge-aware graph convolutional networks (EGCN), where the resulting rich structural features aid the modeling of the intrinsic conversational contexts.&#10;Figure~\ref{fig:summary} illustrates our enhancement for dialogue disentanglement in different stages: feature encoding, model learning, and inference.&#10;&#10;Further, we propose to optimize the learning and inference of the above framework, following the hierarchical nature of conversation discourse.&#10;First, for model training, we devise a hierarchical ranking loss mechanism (cf $\S$\ref{sec:hi-ranking}).&#10;We group the candidate parents of the current utterance into different discourse levels within a dialogue, based on which we define the learning losses under three hierarchical levels, covering both the pair-wise and session-wise optimizations.&#10;Then, during inference, we introduce an easy-first relation decoding algorithm (cf $\S$\ref{sec:Easy-First Graph-based Decoding}).&#10;By consulting both the precedent and subsequent context, the utterance-parent pairing procedure is taken place in an easy-to-hard manner without following the sequential order.&#10;Specifically, we maintain a global utterance-pair scoring matrix, and at each decoding step, the utterance pair with the highest score will be selected.&#10;Also, the established replying pair is incrementally added into the partial-replying structure to update features for further facilitating the follow-up inference.&#10;&#10;We conduct experiments on two benchmark datasets, including the Ubuntu IRC \cite{kummerfeld-etal-2019-large} and Movie Dialogue \cite{liuSGLWZ20-ijcai}.&#10;The results show that our overall system outperforms the current state-of-the-art (SoTA) baselines with significant margins on all datasets and metrics.&#10;Model ablation studies prove the necessity of integrating the various dialogue discourse structure information, the hierarchical ranking loss mechanism, and also the easy-first decoding algorithm for dialogue disentanglement.&#10;Additionally, our experiments highlight the effectiveness of our discourse structure-aware method, particularly in scenarios with longer utterance distance and multiple participating speakers.&#10;Furthermore, our in-depth analysis of the Hierarchical Ranking Loss mechanism reveals its crucial role in rectifying prediction errors and improving dialogue disentanglement performance.&#10;The insights derived from our experimental analysis also underscore the power of the easy-first decoding strategy in facilitating confident decision-making, thereby demonstrating its significant advantage in the field of dialogue disentanglement.&#10;&#10;Moreover, we conducted an experiment comparing our model to GPT-3.5 for the task of dialogue disentanglement.&#10;The results of our experiment demonstrate a clear superiority of our model over GPT-3.5, indicating a significant advancement in the dialogue disentanglement task.&#10;These outcomes suggest that larger language models, despite their comprehensive capabilities, exhibit discernible shortcomings in the dialogue disentanglement task, particularly with respect to effectively capturing dialogue discourse structures.&#10;Based on these insights, we validate the effectiveness of our model and emphasize the necessity for specialized optimization of models tailored for dialogue discourse.&#10;&#10;All in all, this paper revisits the discourse attribute of conversation for better dialogue disentanglement.&#10;To our knowledge, this is by far the first work taking full consideration of dialogue discourse, from feature modeling to model optimization.&#10;To aid understanding, we summarize our key contributions as follows.&#10;\begin{itemize}&#10; \item We construct various dialogue discourse structural features to enrich the dialogue contexts.&#10; \item We propose a hierarchical ranking loss method to cover both pair-wise and session-wise task learning.&#10; \item We present an easy-first decoding algorithm to enable a highly effective inference of replying relations. &#10;\end{itemize}&#10;Our work can be instructive for a wider range of multi-party multi-thread dialogue applications without much effort.&#10;To facilitate the follow-up research, we will release all our codes and metadata upon acceptance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Dialogue Disentanglement, Discourse Structure Analysis, Artificial Intelligence, Conversational Semantic Modeling, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1810.11118" label="1810.11118">
        <attvalues>
          <attvalue for="0" value="A Large-Scale Corpus for Conversation Disentanglement" />
          <attvalue for="1" value="  Disentangling conversations mixed together in a single stream of messages is&#10;a difficult task, made harder by the lack of large manually annotated datasets.&#10;We created a new dataset of 77,563 messages manually annotated with&#10;reply-structure graphs that both disentangle conversations and define internal&#10;conversation structure. Our dataset is 16 times larger than all previously&#10;released datasets combined, the first to include adjudication of annotation&#10;disagreements, and the first to include context. We use our data to re-examine&#10;prior work, in particular, finding that 80% of conversations in a widely used&#10;dialogue corpus are either missing messages or contain extra messages. Our&#10;manually-annotated data presents an opportunity to develop robust data-driven&#10;methods for conversation disentanglement, which will help advance dialogue&#10;research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.15265" label="2210.15265">
        <attvalues>
          <attvalue for="0" value="Conversation Disentanglement with Bi-Level Contrastive Learning" />
          <attvalue for="1" value="  Conversation disentanglement aims to group utterances into detached sessions,&#10;which is a fundamental task in processing multi-party conversations. Existing&#10;methods have two main drawbacks. First, they overemphasize pairwise utterance&#10;relations but pay inadequate attention to the utterance-to-context relation&#10;modeling. Second, huge amount of human annotated data is required for training,&#10;which is expensive to obtain in practice. To address these issues, we propose a&#10;general disentangle model based on bi-level contrastive learning. It brings&#10;closer utterances in the same session while encourages each utterance to be&#10;near its clustered session prototypes in the representation space. Unlike&#10;existing approaches, our disentangle model works in both supervised setting&#10;with labeled data and unsupervised setting when no such data is available. The&#10;proposed method achieves new state-of-the-art performance on both settings&#10;across several public datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.10666" label="1911.10666">
        <attvalues>
          <attvalue for="0" value="Who did They Respond to? Conversation Structure Modeling using Masked&#10;  Hierarchical Transformer" />
          <attvalue for="1" value="  Conversation structure is useful for both understanding the nature of&#10;conversation dynamics and for providing features for many downstream&#10;applications such as summarization of conversations. In this work, we define&#10;the problem of conversation structure modeling as identifying the parent&#10;utterance(s) to which each utterance in the conversation responds to. Previous&#10;work usually took a pair of utterances to decide whether one utterance is the&#10;parent of the other. We believe the entire ancestral history is a very&#10;important information source to make accurate prediction. Therefore, we design&#10;a novel masking mechanism to guide the ancestor flow, and leverage the&#10;transformer model to aggregate all ancestors to predict parent utterances. Our&#10;experiments are performed on the Reddit dataset (Zhang, Culbertson, and&#10;Paritosh 2017) and the Ubuntu IRC dataset (Kummerfeld et al. 2019). In&#10;addition, we also report experiments on a new larger corpus from the Reddit&#10;platform and release this dataset. We show that the proposed model, that takes&#10;into account the ancestral history of the conversation, significantly&#10;outperforms several strong baselines including the BERT model on all datasets&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03760" label="2004.03760">
        <attvalues>
          <attvalue for="0" value="DialBERT: A Hierarchical Pre-Trained Model for Conversation&#10;  Disentanglement" />
          <attvalue for="1" value="  Disentanglement is a problem in which multiple conversations occur in the&#10;same channel simultaneously, and the listener should decide which utterance is&#10;part of the conversation he will respond to. We propose a new model, named&#10;Dialogue BERT (DialBERT), which integrates local and global semantics in a&#10;single stream of messages to disentangle the conversations that mixed together.&#10;We employ BERT to capture the matching information in each utterance pair at&#10;the utterance-level, and use a BiLSTM to aggregate and incorporate the&#10;context-level information. With only a 3% increase in parameters, a 12%&#10;improvement has been attained in comparison to BERT, based on the F1-Score. The&#10;model achieves a state-of-the-art result on the a new dataset proposed by IBM&#10;and surpasses previous work by a substantial margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08018" label="2110.08018">
        <attvalues>
          <attvalue for="0" value="Structural Characterization for Dialogue Disentanglement" />
          <attvalue for="1" value="  Tangled multi-party dialogue contexts lead to challenges for dialogue reading&#10;comprehension, where multiple dialogue threads flow simultaneously within a&#10;common dialogue record, increasing difficulties in understanding the dialogue&#10;history for both human and machine. Previous studies mainly focus on utterance&#10;encoding methods with carefully designed features but pay inadequate attention&#10;to characteristic features of the structure of dialogues. We specially take&#10;structure factors into account and design a novel model for dialogue&#10;disentangling. Based on the fact that dialogues are constructed on successive&#10;participation and interactions between speakers, we model structural&#10;information of dialogues in two aspects: 1)speaker property that indicates whom&#10;a message is from, and 2) reference dependency that shows whom a message may&#10;refer to. The proposed method achieves new state-of-the-art on the Ubuntu IRC&#10;benchmark dataset and contributes to dialogue-related comprehension.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10433" label="2305.10433">
        <attvalues>
          <attvalue for="0" value="Toxicity Inspector: A Framework to Evaluate Ground Truth in Toxicity&#10;  Detection Through Feedback" />
          <attvalue for="1" value="  Toxic language is difficult to define, as it is not monolithic and has many&#10;variations in perceptions of toxicity. This challenge of detecting toxic&#10;language is increased by the highly contextual and subjectivity of its&#10;interpretation, which can degrade the reliability of datasets and negatively&#10;affect detection model performance. To fill this void, this paper introduces a&#10;toxicity inspector framework that incorporates a human-in-the-loop pipeline&#10;with the aim of enhancing the reliability of toxicity benchmark datasets by&#10;centering the evaluator's values through an iterative feedback cycle. The&#10;centerpiece of this framework is the iterative feedback process, which is&#10;guided by two metric types (hard and soft) that provide evaluators and dataset&#10;creators with insightful examination to balance the tradeoff between&#10;performance gains and toxicity avoidance.&#10;" />
          <attvalue for="2" value="&#10;Determining what is toxic or harmful in a language is a very subjective task, as perceived toxicity varies based on many different characteristics ~\cite{Davidson2017ai}. &#10;&#10;To detect toxic language, the construction of the training dataset plays an essential role in the robustness of model evaluation and performance. &#10;{Notably, the appropriate handling of label annotator variations during dataset construction is often overlooked. However, recent research has been conducted on alleviating the possible effects of these variations. For example, ~\cite{Waseem2017-vg} proposed a typology that differentiates between direct/indirect abusive language toward specific individuals or groups.} Another work by ~\cite{Fanton2021-bc} proposed a human-in-the-loop revision cycle for pairs of narratives (i.e., hate speech and counter-hate speech), in which feedback is provided for the given text to generate a new dataset.&#10;&#10;In the novel framework proposed in this study, the feedback pertains to the label, which is set as concept-shifting model. Specifically, the feedback process is guided using soft and hard metrics, where cross entropy (CE) and error rate are provided alongside hard metric evaluations. &#10;&#10;Current toxicity detectors' reliability and robustness suffer from high subjectivity and bias towards keywords ~\cite{Sap2022-lj}. In this demonstration paper, a means of evaluating reliability while improving toxicity labeling is provided by incorporating a human-in-the-loop pipeline to evaluate concept shifts within a dataset. &#10;&#10;{Given the multitude of terms and definitions related to hate speech in the literature, several recent studies have investigated their common aspects in terms of language detection tasks. Specifically for hate speech, the research of ~\cite{Fanton2021-bc} focused on covariate shifts (modified text) in the human-in-the-loop pipeline, demonstrating that assessing covariate shifting is beneficial to addressing the shortcomings of extant collection strategies that grant either quality or quantity, but not both ~\cite{Bhatt2021-em}. This trend of diligence is clear evidence of the urgency of finding sustainable and data-centric ways to support a full benchmark creation cycle for toxicity identification in a way that provides clear transparency and evaluation capabilities between each feedback iteration. }&#10;&#10;The toxicity inspector provides the means of iterating sections of the dataset to verify its labels through a series of feedback cycles.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Machine Learning, Linguistics, Dataset Reliability Enhancement, Toxic Language Detection, Natural Language Processing, Psychology, Human-in-the-Loop Framework" />
        </attvalues>
      </node>
      <node id="2107.08720" label="2107.08720">
        <attvalues>
          <attvalue for="0" value="Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative&#10;  Dataset to Fight Online Hate Speech" />
          <attvalue for="1" value="  Undermining the impact of hateful content with informed and non-aggressive&#10;responses, called counter narratives, has emerged as a possible solution for&#10;having healthier online communities. Thus, some NLP studies have started&#10;addressing the task of counter narrative generation. Although such studies have&#10;made an effort to build hate speech / counter narrative (HS/CN) datasets for&#10;neural generation, they fall short in reaching either high-quality and/or&#10;high-quantity. In this paper, we propose a novel human-in-the-loop data&#10;collection methodology in which a generative language model is refined&#10;iteratively by using its own data from the previous loops to generate new&#10;training samples that experts review and/or post-edit. Our experiments&#10;comprised several loops including dynamic variations. Results show that the&#10;methodology is scalable and facilitates diverse, novel, and cost-effective data&#10;collection. To our knowledge, the resulting dataset is the only expert-based&#10;multi-target HS/CN dataset available to the community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.07997" label="2111.07997">
        <attvalues>
          <attvalue for="0" value="Annotators with Attitudes: How Annotator Beliefs And Identities Bias&#10;  Toxic Language Detection" />
          <attvalue for="1" value="  The perceived toxicity of language can vary based on someone's identity and&#10;beliefs, but this variation is often ignored when collecting toxic language&#10;datasets, resulting in dataset and model biases. We seek to understand the who,&#10;why, and what behind biases in toxicity annotations. In two online studies with&#10;demographically and politically diverse participants, we investigate the effect&#10;of annotator identities (who) and beliefs (why), drawing from social psychology&#10;research about hate speech, free speech, racist beliefs, political leaning, and&#10;more. We disentangle what is annotated as toxic by considering posts with three&#10;characteristics: anti-Black language, African American English (AAE) dialect,&#10;and vulgarity. Our results show strong associations between annotator identity&#10;and beliefs and their ratings of toxicity. Notably, more conservative&#10;annotators and those who scored highly on our scale for racist beliefs were&#10;less likely to rate anti-Black language as toxic, but more likely to rate AAE&#10;as toxic. We additionally present a case study illustrating how a popular&#10;toxicity detection system's ratings inherently reflect only specific beliefs&#10;and perspectives. Our findings call for contextualizing toxicity labels in&#10;social variables, which raises immense implications for toxic language&#10;annotation and detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03546" label="2010.03546">
        <attvalues>
          <attvalue for="0" value="Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic&#10;  Parsing" />
          <attvalue for="1" value="  Task-oriented semantic parsing is a critical component of virtual assistants,&#10;which is responsible for understanding the user's intents (set reminder, play&#10;music, etc.). Recent advances in deep learning have enabled several approaches&#10;to successfully parse more complex queries (Gupta et al., 2018; Rongali et&#10;al.,2020), but these models require a large amount of annotated training data&#10;to parse queries on new domains (e.g. reminder, music).&#10;  In this paper, we focus on adapting task-oriented semantic parsers to&#10;low-resource domains, and propose a novel method that outperforms a supervised&#10;neural model at a 10-fold data reduction. In particular, we identify two&#10;fundamental factors for low-resource domain adaptation: better representation&#10;learning and better training techniques. Our representation learning uses BART&#10;(Lewis et al., 2019) to initialize our model which outperforms encoder-only&#10;pre-trained representations used in previous work. Furthermore, we train with&#10;optimization-based meta-learning (Finn et al., 2017) to improve generalization&#10;to low-resource domains. This approach significantly outperforms all baseline&#10;methods in the experiments on a newly collected multi-domain task-oriented&#10;semantic parsing dataset (TOPv2), which we release to the public.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Task-Oriented Semantic Parsing&#10;has attracted attention from the research community since 1990s with the advent of the ATIS dataset~\cite{price-1990-evaluation}.&#10;Traditionally, the task is formulated as a joint text classification (intent prediction) and sequence tagging (slot filling) problem, and can be tackled with sequence labeling models such as RNNs~\cite{mesnil2013investigation,Liu+2016}.&#10;These models can only parse flat queries with one intent and non-nested slots.&#10;More recently, a number of studies propose alternative approaches for handling the more complex compositional queries using neural shift-reduce parsers~\cite{gupta-etal-2018-semantic-parsing,einolghozati2019improving} or seq2seq models~\cite{jia-liang-2016-data,rongali-etal-2020-dont}.&#10;&#10;On the other hand, there have been research efforts on scaling task-oriented parsers to new domains with less training data~\cite{Jaech2016DomainAO,Bapna2017TowardsZF,fan-etal-2017-transfer,goyal-etal-2018-fast,DBLP:conf/aaai/LeeJ19}.&#10;These methods, however, only focus on the simpler flat queries.&#10;Our proposed method, in contrast, can effectively parse both flat and compositional queries for low-resource target domains.&#10;&#10;Meta-Learning~\cite{Lake1332}, or learning to learn, aims to learn a model that can quickly adapt to new tasks with a small amount of training data.&#10;In particular, \cite{finn-etal-2017-maml} propose MAML, an optimization-based meta-learning method, which learns a good parameter initialization suitable for faster adaptation to new tasks.&#10;As MAML requires to compute second derivatives, which are computation and memory intensive, there have been studies to use either first-order approximation such as first-order MAML and Reptile~\cite{DBLP:journals/corr/abs-1803-02999}, or implicit differentiation~\cite{NIPS2019_8306}.&#10;Furthermore, meta-learning has also been applied to a number of NLP tasks lately~\cite{gu-etal-2018-meta,dou-etal-2019-investigating,ijcai2019-437,qian-yu-2019-domain,2019arXiv190905438S}.&#10; " />
          <attvalue for="4" value="Task-Oriented Parsing, Machine Learning, Computer Science, Linguistics, Low-Resource Domain Adaptation, Deep Learning Methods, Mathematics" />
        </attvalues>
      </node>
      <node id="2010.11125" label="2010.11125">
        <attvalues>
          <attvalue for="0" value="Beyond English-Centric Multilingual Machine Translation" />
          <attvalue for="1" value="  Existing work in translation demonstrated the potential of massively&#10;multilingual machine translation by training a single model able to translate&#10;between any pair of languages. However, much of this work is English-Centric by&#10;training only on data which was translated from or to English. While this is&#10;supported by large sources of training data, it does not reflect translation&#10;needs worldwide. In this work, we create a true Many-to-Many multilingual&#10;translation model that can translate directly between any pair of 100&#10;languages. We build and open source a training dataset that covers thousands of&#10;language directions with supervised data, created through large-scale mining.&#10;Then, we explore how to effectively increase model capacity through a&#10;combination of dense scaling and language-specific sparse parameters to create&#10;high quality models. Our focus on non-English-Centric models brings gains of&#10;more than 10 BLEU when directly translating between non-English directions&#10;while performing competitively to the best single systems of WMT. We&#10;open-source our scripts so that others may reproduce the data, evaluation, and&#10;final M2M-100 model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.09210" label="2302.09210">
        <attvalues>
          <attvalue for="0" value="How Good Are GPT Models at Machine Translation? A Comprehensive&#10;  Evaluation" />
          <attvalue for="1" value="  Generative Pre-trained Transformer (GPT) models have shown remarkable&#10;capabilities for natural language generation, but their performance for machine&#10;translation has not been thoroughly investigated. In this paper, we present a&#10;comprehensive evaluation of GPT models for machine translation, covering&#10;various aspects such as quality of different GPT models in comparison with&#10;state-of-the-art research and commercial systems, effect of prompting&#10;strategies, robustness towards domain shifts and document-level translation. We&#10;experiment with eighteen different translation directions involving high and&#10;low resource languages, as well as non English-centric translations, and&#10;evaluate the performance of three GPT models: ChatGPT, GPT3.5&#10;(text-davinci-003), and text-davinci-002. Our results show that GPT models&#10;achieve very competitive translation quality for high resource languages, while&#10;having limited capabilities for low resource languages. We also show that&#10;hybrid approaches, which combine GPT models with other translation systems, can&#10;further enhance the translation quality. We perform comprehensive analysis and&#10;human evaluation to further understand the characteristics of GPT translations.&#10;We hope that our paper provides valuable insights for researchers and&#10;practitioners in the field and helps to better understand the potential and&#10;limitations of GPT models for translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.12674" label="2308.12674">
        <attvalues>
          <attvalue for="0" value="Improving Translation Faithfulness of Large Language Models via&#10;  Augmenting Instructions" />
          <attvalue for="1" value="  Large Language Models (LLMs) present strong general capabilities, and a&#10;current compelling challenge is stimulating their specialized capabilities,&#10;such as machine translation, through low-cost instruction tuning. The standard&#10;instruction-following data is sequentially organized as the concatenation of an&#10;instruction, an input, and a response. As the attention mechanism of LLMs has&#10;limitations on local focus, LLMs tend to focus more on the words or sentences&#10;nearby at each position. This leads to a high risk of instruction forgetting&#10;during decoding. To alleviate the above issues, We propose SWIE&#10;(Segment-Weighted Instruction Embedding) and an instruction-following dataset&#10;OVERMISS. SWIE improves the model instruction understanding by adding a global&#10;instruction representation on the following input and response representations.&#10;OVERMISS improves model faithfulness by comparing over-translation and&#10;miss-translation results with the correct translation. We apply our methods to&#10;two main-stream open-source LLMs, BLOOM and LLaMA. The experimental results&#10;demonstrate significant improvements in translation performance with SWIE based&#10;on BLOOMZ-3b, particularly in zero-shot and long text translations due to&#10;reduced instruction forgetting risk. Additionally, OVERMISS outperforms the&#10;baseline in translation performance (e.g. an increase in BLEU scores from 0.69&#10;to 3.12 and an average improvement of 0.48 percentage comet scores for&#10;LLaMA-7b) with further enhancements seen in models combining OVERMISS and SWIE&#10;(e.g. the BLUE scores increase up to 0.56 from English to German across three&#10;different backbones), and both exhibit improvements in the faithfulness metric&#10;based on word alignment.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15083" label="2305.15083">
        <attvalues>
          <attvalue for="0" value="Eliciting the Translation Ability of Large Language Models via&#10;  Multilingual Finetuning with Translation Instructions" />
          <attvalue for="1" value="  Large-scale Pretrained Language Models (LLMs), such as ChatGPT and GPT4, have&#10;shown strong abilities in multilingual translations, without being explicitly&#10;trained on parallel corpora. It is interesting how the LLMs obtain their&#10;ability to carry out translation instructions for different languages. In this&#10;paper, we present a detailed analysis by finetuning a multilingual pretrained&#10;language model, XGLM-7B, to perform multilingual translation following given&#10;instructions. Firstly, we show that multilingual LLMs have stronger translation&#10;abilities than previously demonstrated. For a certain language, the performance&#10;depends on its similarity to English and the amount of data used in the&#10;pretraining phase. Secondly, we find that LLMs' ability to carry out&#10;translation instructions relies on the understanding of translation&#10;instructions and the alignment among different languages. With multilingual&#10;finetuning, LLMs could learn to perform the translation task well even for&#10;those language pairs unseen during the instruction tuning phase.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.10968" label="2306.10968">
        <attvalues>
          <attvalue for="0" value="BayLing: Bridging Cross-lingual Alignment and Instruction Following&#10;  through Interactive Translation for Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated remarkable prowess in language&#10;understanding and generation. Advancing from foundation LLMs to&#10;instructionfollowing LLMs, instruction tuning plays a vital role in aligning&#10;LLMs to human preferences. However, the existing LLMs are usually focused on&#10;English, leading to inferior performance in non-English languages. In order to&#10;improve the performance for non-English languages, it is necessary to collect&#10;language-specific training data for foundation LLMs and construct&#10;language-specific instructions for instruction tuning, both of which are heavy&#10;loads. To minimize human workload, we propose to transfer the capabilities of&#10;language generation and instruction following from English to other languages&#10;through an interactive translation task. We have developed BayLing, an&#10;instruction-following LLM by utilizing LLaMA as the foundation LLM and&#10;automatically constructing interactive translation instructions for instructing&#10;tuning. Extensive assessments demonstrate that BayLing achieves comparable&#10;performance to GPT-3.5-turbo, despite utilizing a considerably smaller&#10;parameter size of only 13 billion. Experimental results on translation tasks&#10;show that BayLing achieves 95% of single-turn translation capability compared&#10;to GPT-4 with automatic evaluation and 96% of interactive translation&#10;capability compared to GPT-3.5-turbo with human evaluation. To estimate the&#10;performance on general tasks, we created a multi-turn instruction test set&#10;called BayLing-80. The experimental results on BayLing-80 indicate that BayLing&#10;achieves 89% of performance compared to GPT-3.5-turbo. BayLing also&#10;demonstrates outstanding performance on knowledge assessment of Chinese GaoKao&#10;and English SAT, second only to GPT-3.5-turbo among a multitude of&#10;instruction-following LLMs. Demo, homepage, code and models of BayLing are&#10;available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.11674" label="2309.11674">
        <attvalues>
          <attvalue for="0" value="A Paradigm Shift in Machine Translation: Boosting Translation&#10;  Performance of Large Language Models" />
          <attvalue for="1" value="  Generative Large Language Models (LLMs) have achieved remarkable advancements&#10;in various NLP tasks. However, these advances have not been reflected in the&#10;translation task, especially those with moderate model sizes (i.e., 7B or 13B&#10;parameters), which still lag behind conventional supervised encoder-decoder&#10;translation models. Previous studies have attempted to improve the translation&#10;capabilities of these moderate LLMs, but their gains have been limited. In this&#10;study, we propose a novel fine-tuning approach for LLMs that is specifically&#10;designed for the translation task, eliminating the need for the abundant&#10;parallel data that traditional translation models usually depend on. Our&#10;approach consists of two fine-tuning stages: initial fine-tuning on monolingual&#10;data followed by subsequent fine-tuning on a small set of high-quality parallel&#10;data. We introduce the LLM developed through this strategy as Advanced Language&#10;Model-based trAnslator (ALMA). Based on LLaMA-2 as our underlying model, our&#10;results show that the model can achieve an average improvement of more than 12&#10;BLEU and 12 COMET over its zero-shot performance across 10 translation&#10;directions from the WMT'21 (2 directions) and WMT'22 (8 directions) test&#10;datasets. The performance is significantly better than all prior work and even&#10;superior to the NLLB-54B model and GPT-3.5-text-davinci-003, with only 7B or&#10;13B parameters. This method establishes the foundation for a novel training&#10;paradigm in machine translation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Generative (decoder-only) large language models (LLMs) such as GPT models \cite{gpt3_few_shot,openai2023gpt4}, PaLM \cite{palm}, OPT \cite{OPT}, BLOOM \cite{bloom}, LLaMA \cite{llama1,llama2}, and others have exhibited remarkable capabilities across various NLP tasks. However, for the translation task, only very large models such as GPT-3.5 and GPT-4 can rival the supervised encoder-decoder state-of-the-art (SoTA) models like NLLB \cite{nllb}, while they still fall short in translation for low-resource languages \cite{gpt_mt,jiao2023chatgpt}. The discrepancy becomes more evident when comparing other LLMs with traditional translation models \cite{zhu2023multilingual}. For instance, the OPT-175B model trails behind the NLLB-1.3B model by an average of more than 15 BLEU \cite{papineni2002bleu} points for languages within the Indo-European-Romance family. The gap is even larger in smaller LLMs; for example, XGLM \cite{xglm}, with a parameter size of 7B, lags behind the NLLB-1.3B by a substantial 30 BLEU points \cite{zhu2023multilingual}. Therefore, there is an urgent need to narrow this performance gap between LLMs and conventional SoTA models.&#10;&#10;As exemplified by NLLB-1.3B, traditional machine translation models demonstrate proficiency in producing high-quality translations with a small number of parameters. By extension, smaller LLMs should similarly possess the capability to adeptly manage the translation task. Recent research has sought to enhance translation performance by commencing with smaller LLMs \cite{bigtranslate,zeng2023tim,swie,zhu2023extrapolating,li2023eliciting,bayling}, especially 7B or 13B parameters. Nevertheless, the achieved improvements remain modest and limited. As depicted in Figure \ref{fig:intro}, contemporary studies such as Balyling \cite{bayling} and BigTranslate \cite{bigtranslate}, which use LLaMA as their backbone, exhibit a maximum increment of 3 to 4 BLEU or COMET in relation to the zero-shot performance of LLaMA on the WMT'22 test set (8 directions). While these gains represent promising research direction for smaller LLMs in the translation task, a significant performance chasm persists when benchmarked against very large LLMs such as GPT-3.5-text-davinci-003 and SoTA translation models such as NLLB-54B. We posit that the modest translation gains observed in prior studies can be ascribed to an unsuitable training recipe.&#10;&#10;We hypothesize that an efficacious training recipe ought to follow two stages: learning general multilingual linguistic knowledge and inducing (instructing) models toward translation generation. Consequently, we propose a two-stage fine-tuning approach and introduce the LLM developed through this strategy as \underline{A}dvanced \underline{L}anguage \underline{M}odel-based tr\underline{A}nslator (ALMA). Specifically, given most LLMs are trained on English-dominant data, the first stage is fine-tuning non-English monolingual data to enhance the model's proficiency in other languages involved in the translation task. Secondly, drawing inspiration from the recognized significance of data quality in other applications \cite{zhou2023lima,maillard2023small,gunasekar2023textbooks}, we fine-tune the model with a small amount of high-quality parallel data.&#10;&#10;Our main contributions are summarized as follows:&#10;&#10;Diminished Necessity of Parallel Data Traditional translation frameworks rely on large amounts of parallel data, which may lead to a false impression that such data is essential for the translation task with LLMs. Prior studies have fine-tuned LLMs with datasets containing over 300M parallel instances \cite{bigtranslate}. However, our empirical evaluations suggest that this strategy may not be optimal, and even harm the translation capabilities of LLMs.&#10;&#10;LLM Via A New Training Recipe: ALMA&#10;We introduce a novel two-stage fine-tuning method for translation with decoder-only LLMs. Leveraging LLaMA-2 as the base model, we attain an average improvement of more than 12 BLEU and COMET scores over its zero-shot performance across 10 translation directions from WMT'21 and WMT'22 test datasets. Notably, the performance surpasses all previous work and is even better than the NLLB-54B model and GPT-3.5-text-davinci-003.&#10;&#10;Efficient Computational Cost&#10;Our ablation study reveals both stages are crucial factors for achieving large improvements. The most computationally intensive part is monolingual data fine-tuning, however, we show that only fine-tuning 1B monolingual tokens is sufficient to have comparable performance to NLLB-54B in 10 translation directions, which only requires around 18 hours to complete with 16 MI200 GPUs.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.08357" label="2309.08357">
        <attvalues>
          <attvalue for="0" value="Audio-free Prompt Tuning for Language-Audio Models" />
          <attvalue for="1" value="  Contrastive Language-Audio Pretraining (CLAP) is pre-trained to associate&#10;audio features with human language, making it a natural zero-shot classifier to&#10;recognize unseen sound categories. To adapt CLAP to downstream tasks, prior&#10;works inevitably require labeled domain audios, which limits their scalability&#10;under data scarcity and deprives them of the capability to detect novel classes&#10;as the original CLAP. In this work, by leveraging the modality alignment in&#10;CLAP, we propose an efficient audio-free prompt tuning scheme aimed at&#10;optimizing a few prompt tokens from texts instead of audios, which regularizes&#10;the model space to avoid overfitting the seen classes as well. Based on this, a&#10;multi-grained prompt design is further explored to fuse global and local&#10;information. Experiments on several tasks demonstrate that our approach can&#10;boost the CLAP and outperform other training methods on model performance and&#10;training efficiency. While conducting zero-shot inference on unseen categories,&#10;it still shows better transferability than the vanilla CLAP. Moreover, our&#10;method is flexible enough even if only knowing the downstream class names. The&#10;code will be released soon.&#10;" />
          <attvalue for="2" value="&#10;&#10;\let \mybackup \thefootnote&#10;&#09;\let \thefootnote &#10;&#09;\let \thefootnote \mybackup&#10;&#09;\let \mybackup \imareallyundefinedcommand&#10;&#10;Audio classification, which aims at mapping an audio clip into one or more sound classes, can assist in perceiving physical environments. Recently, many approaches have achieved notable success on audio classification tasks with the emergence of large-scale audio datasets \cite{gemmeke2017audioset}, powerful network architectures \cite{chen2022hts}, and effective learning theories \cite{xiao2023semanticac}. However, they generally require a number of downstream audios for training and cannot be applied to recognize unseen categories out of the training set once tuned. With the supervision of natural language, Contrastive Language Audio Pre-training (CLAP) \cite{elizalde2023msclap, wu2023laionclap, wu2022wav2clip, guzhov2022audioclip, manco2022musicclip, huang2022mulan} resorts to a novel paradigm to tackle the above problems with two modality encoders learning a joint space for audio and text embeddings, which empowers it with zero-shot classification ability. Although CLAP provides a well-structured feature space for downstream training and demonstrates promising transferability, its zero-shot performance is still far away from real-world applications. As a result, how to efficiently enhance the classification ability of CLAP on targeted classes while reserving its generalization ability on unseen classes remains a tricky problem. &#10;&#10;To further adapt CLAP on downstream classification tasks, conventional Finetune updates the whole CLAP audio branch, which generally demands a substantial amount of labeled data to reach promising results. Linear Probe suggested by CLIP \cite{radford2021clip}, which solely tunes the downstream classifier, is proved to be more efficient but still ill-suited for few-shot settings. In pursuit of better performance under data scarcity, methods such as Treff adapter \cite{liang23treff} and TIP adapter \cite{zhang2022tip} utilize a key-value cache model from the training set and update the prior knowledge in CLAP by feature retrieval, which can achieve better performance than traditional few-shot algorithms, such as ProtoNet \cite{snell2017protonet}. However, the above methods discard the CLAP text encoder while newly adding learnable weights to train on a fixed number of categories, which will deprive their power to detect unseen classes as CLAP. Prompt tuning, explored in vision-language tasks \cite{zhou2022coop, khattak2023maple}, throws light on the above issue by converting the hand-crafted prompt template (e.g., ``this is a sound of'') into learnable tokens while reserving the CLAP text encoder. Although it performs well under few-shot settings and can be used to recognize unseen sound events, the learned prompt is relatively easy to overfit the seen classes and shows performance drops when testing on other datasets. Moreover, besides weakening the recognition performance on novel classes as mentioned above, prior methods rely heavily on domain audios for training, which limits their applications since domain audios might be inaccessible sometimes. In some extreme cases where no audios are available, one cannot but resort to the unpromising zero-shot predictions of CLAP. Two questions then arise: Can we bootstrap CLAP even without any downstream audios? Will it also boost the recognition ability on unseen classes?&#10;&#10;In this paper, we argue that it is feasible to treat texts as audios for training. The intuition is that the feature generated by the audio encoder will be close to the encoded text feature of its corresponding caption, so it is reasonable to extract text features from captions as alternatives for training. Compared to audios, the texts are easy to gather, and their class labels can be directly derived. We then borrow the idea of prompt tuning but train it with collected captions, resulting in a novel audio-free prompt tuning paradigm for classification tasks. To further exploit the power of prompt, we devise a multi-grained prompt tuning scheme, which aims to combine coarse-grained and fine-grained modality information to improve performance. During training, the CLAP audio encoder is excluded, making it a self-distillation process for the CLAP text encoder to regularize the parameter space of learnable prompts, thereby enhancing its generalization ability on novel classes. We refer to the above method as Prompt Tuning from Text ({PT-Text}). &#10;&#10;Extensive experiments with different CLAP models are conducted on single-label and multi-label classification tasks. The results show that PT-Text can promote CLAP without any domain audios and outperform prior methods using few-shot audios on training efficiency and model performance. Besides, the competence to classify unseen classes is examined by a source-to-target test, and the effects of multi-grained prompts are verified by ablation studies. Moreover, even if only knowing the target class names, PT-Text can still work with hand-crafted templates, indicating its flexibility. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Audio Modeling, Computer Science, Audio Signal Processing, Linguistics, Multimodal Pretraining, Zero-Shot Learning, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2207.09519" label="2207.09519">
        <attvalues>
          <attvalue for="0" value="Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification" />
          <attvalue for="1" value="  Contrastive Vision-Language Pre-training, known as CLIP, has provided a new&#10;paradigm for learning visual representations using large-scale image-text&#10;pairs. It shows impressive performance on downstream tasks by zero-shot&#10;knowledge transfer. To further enhance CLIP's adaption capability, existing&#10;methods proposed to fine-tune additional learnable modules, which significantly&#10;improves the few-shot performance but introduces extra training time and&#10;computational resources. In this paper, we propose a training-free adaption&#10;method for CLIP to conduct few-shot classification, termed as Tip-Adapter,&#10;which not only inherits the training-free advantage of zero-shot CLIP but also&#10;performs comparably to those training-required approaches. Tip-Adapter&#10;constructs the adapter via a key-value cache model from the few-shot training&#10;set, and updates the prior knowledge encoded in CLIP by feature retrieval. On&#10;top of that, the performance of Tip-Adapter can be further boosted to be&#10;state-of-the-art on ImageNet by fine-tuning the cache model for 10$\times$&#10;fewer epochs than existing methods, which is both effective and efficient. We&#10;conduct extensive experiments of few-shot classification on 11 datasets to&#10;demonstrate the superiority of our proposed methods. Code is released at&#10;https://github.com/gaopengcuhk/Tip-Adapter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.05175" label="1703.05175">
        <attvalues>
          <attvalue for="0" value="Prototypical Networks for Few-shot Learning" />
          <attvalue for="1" value="  We propose prototypical networks for the problem of few-shot classification,&#10;where a classifier must generalize to new classes not seen in the training set,&#10;given only a small number of examples of each new class. Prototypical networks&#10;learn a metric space in which classification can be performed by computing&#10;distances to prototype representations of each class. Compared to recent&#10;approaches for few-shot learning, they reflect a simpler inductive bias that is&#10;beneficial in this limited-data regime, and achieve excellent results. We&#10;provide an analysis showing that some simple design decisions can yield&#10;substantial improvements over recent approaches involving complicated&#10;architectural choices and meta-learning. We further extend prototypical&#10;networks to zero-shot learning and achieve state-of-the-art results on the&#10;CU-Birds dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.07801" label="2406.07801">
        <attvalues>
          <attvalue for="0" value="PolySpeech: Exploring Unified Multitask Speech Models for&#10;  Competitiveness with Single-task Models" />
          <attvalue for="1" value="  Recently, there have been attempts to integrate various speech processing&#10;tasks into a unified model. However, few previous works directly demonstrated&#10;that joint optimization of diverse tasks in multitask speech models has&#10;positive influence on the performance of individual tasks. In this paper we&#10;present a multitask speech model -- PolySpeech, which supports speech&#10;recognition, speech synthesis, and two speech classification tasks. PolySpeech&#10;takes multi-modal language model as its core structure and uses semantic&#10;representations as speech inputs. We introduce semantic speech embedding&#10;tokenization and speech reconstruction methods to PolySpeech, enabling&#10;efficient generation of high-quality speech for any given speaker. PolySpeech&#10;shows competitiveness across various tasks compared to single-task models. In&#10;our experiments, multitask optimization achieves performance comparable to&#10;single-task optimization and is especially beneficial for specific tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently researchers have proposed model architectures and training techniques to aggregate various speech processing tasks into one unified framework. Performance of these frameworks demonstrate the capability of deep learning models like Transformer\cite{NIPS2017_7181,Zhou2018} to learn and express the knowledge in various modalities required for diverse speech tasks.&#10;&#10;Classic encoder-decoder architecture has been explored to model multiple speech tasks. For example, &#10;Whisper\cite{radford2023robust} is capable of performing speech processing tasks such as speech recognition, speech translation, and language identification, utilizing large-scale multilingual supervised speech as training data for an encoder-decoder Transformer.&#10;There are also recent attempts to aggregate multiple speech tasks in simpler decoder-only language model (LM) structures instead of encoder-decoder. VioLA\cite{wang2023viola}, AudioPaLM\cite{rubenstein2023audiopalm}, LauraGPT\cite{chen2023lauragpt}, SpeechGPT\cite{zhang2023speechgpt}, etc. support speech transcription and generation tasks by modeling both speech and text representations with Transformer LMs. Speech representations used by these models can be categorised into two types -- acoustic ones and semantic ones. Acoustic speech representations are generated from speech waveform through speech codec methods, e.g. EnCodec\cite{defossez2022high} and SoundStream\cite{zeghidour2021soundstream}. Semantic representations are usually extracted from self-supervised learning models such as wav2vec\cite{schneider2019wav2vec,baevski2020wav2vec,chung2021w2v} and HuBERT\cite{hsu2021hubert}.&#10;&#10;Previous literature\cite{wang2023viola,rubenstein2023audiopalm,chen2023lauragpt} has analyzed the performance differences between the proposed multitask speech models and existing single-task models, but few studies compare these models under fair enough experimental conditions, such as same supervised training data.&#10;The aim of this paper is to demonstrate that multitask models are in deed competitive in performance on various tasks compared to single-task models. &#10;We intend to find out whether joint optimization of various speech tasks within a single auto-regressive Transformer decoder framework improves performance comparing to single-task optimization.&#10;We are also curious whether the multitask model benefits more from semantic speech representations or from acoustic ones.&#10;&#10;In this paper, we propose a multitask speech model framework, which we call PolySpeech. PolySpeech's core structure is a multi-modal decoder-only Transformer LM, which autoregressively predicts speech or text tokens. We integrate tasks of speech recognition (ASR), speech synthesis (TTS), spoken language identification (LID), and gender identification (GID) in PolySpeech. These tasks, covering three main types of speech tasks -- transcription, generation, and classification, are jointly optimized in a supervised manner. We prefer to using semantic speech representations rather than using acoustic ones in PolySpeech for better performance. PolySpeech is highly flexible and can be further extended to other speech tasks. &#10;&#10;Discretization of speech representations is crucial for autoregressive prediction models to generate speech. In preliminary experiments, we have found that the k-means discretization method used in AudioLM\cite{borsos2023audiolm} and Spear-TTS\cite{kharitonov2023speak} would lead to loss of acoustic information. We tried to resynthesize HuBERT k-means tokens of Mandarin Chinese speech with a HiFi-GAN\cite{kong2020hifi} vocoder and observed tone inaccuracies in synthesized speech, which are unacceptable for Mandarin. For better speech generation performance, we use a codec method for semantic embedding of speech in PolySpeech, which preserves more complete acoustic information in discrete tokens than the k-means method.&#10;Focusing on producing high quality speech while controlling acoustic conditions, we also design a semantic speech token decoder that generates speech waveform given discrete semantic tokens and a speech prompt from any speaker. &#10;&#10;We summarize the contribution of this paper as follows: &#10;\begin{enumerate} &#10;\item We propose a multitask speech model based on multi-modal LM and semantic speech representations, which is competitive across various tasks compared to single-task models.&#10;\item We introduce a high-fidelity speech codec method and a semantic speech token decoder to the multitask speech model, enabling efficient speech generation for any given speaker.&#10;\item We conduct meaningful experiments, demonstrating that multitask optimization achieves performance on par with single-task optimization and is beneficial for certain tasks.&#10;\end{enumerate} &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Multitask Learning Models, Linguistics, Language Model Integration, Artificial Intelligence, Speech Processing Tasks" />
        </attvalues>
      </node>
      <node id="1804.10752" label="1804.10752">
        <attvalues>
          <attvalue for="0" value="Syllable-Based Sequence-to-Sequence Speech Recognition with the&#10;  Transformer in Mandarin Chinese" />
          <attvalue for="1" value="  Sequence-to-sequence attention-based models have recently shown very&#10;promising results on automatic speech recognition (ASR) tasks, which integrate&#10;an acoustic, pronunciation and language model into a single neural network. In&#10;these models, the Transformer, a new sequence-to-sequence attention-based model&#10;relying entirely on self-attention without using RNNs or convolutions, achieves&#10;a new single-model state-of-the-art BLEU on neural machine translation (NMT)&#10;tasks. Since the outstanding performance of the Transformer, we extend it to&#10;speech and concentrate on it as the basic architecture of sequence-to-sequence&#10;attention-based model on Mandarin Chinese ASR tasks. Furthermore, we&#10;investigate a comparison between syllable based model and context-independent&#10;phoneme (CI-phoneme) based model with the Transformer in Mandarin Chinese.&#10;Additionally, a greedy cascading decoder with the Transformer is proposed for&#10;mapping CI-phoneme sequences and syllable sequences into word sequences.&#10;Experiments on HKUST datasets demonstrate that syllable based model with the&#10;Transformer performs better than CI-phoneme based counterpart, and achieves a&#10;character error rate (CER) of \emph{$28.77\%$}, which is competitive to the&#10;state-of-the-art CER of $28.0\%$ by the joint CTC-attention based&#10;encoder-decoder network.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16107" label="2305.16107">
        <attvalues>
          <attvalue for="0" value="VioLA: Unified Codec Language Models for Speech Recognition, Synthesis,&#10;  and Translation" />
          <attvalue for="1" value="  Recent research shows a big convergence in model architecture, training&#10;objectives, and inference methods across various tasks for different&#10;modalities. In this paper, we propose VioLA, a single auto-regressive&#10;Transformer decoder-only network that unifies various cross-modal tasks&#10;involving speech and text, such as speech-to-text, text-to-text,&#10;text-to-speech, and speech-to-speech tasks, as a conditional codec language&#10;model task via multi-task learning framework. To accomplish this, we first&#10;convert all the speech utterances to discrete tokens (similar to the textual&#10;data) using an offline neural codec encoder. In such a way, all these tasks are&#10;converted to token-based sequence conversion problems, which can be naturally&#10;handled with one conditional language model. We further integrate task IDs&#10;(TID) and language IDs (LID) into the proposed model to enhance the modeling&#10;capability of handling different languages and tasks. Experimental results&#10;demonstrate that the proposed VioLA model can support both single-modal and&#10;cross-modal tasks well, and the decoder-only model achieves a comparable and&#10;even better performance than the strong baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.04673" label="2310.04673">
        <attvalues>
          <attvalue for="0" value="LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT" />
          <attvalue for="1" value="  Generative Pre-trained Transformer (GPT) models have achieved remarkable&#10;performance on various natural language processing tasks, and have shown great&#10;potential as backbones for audio-and-text large language models (LLMs).&#10;Previous mainstream audio-and-text LLMs use discrete audio tokens to represent&#10;both input and output audio; however, they suffer from performance degradation&#10;on tasks such as automatic speech recognition, speech-to-text translation, and&#10;speech enhancement over models using continuous speech features. In this paper,&#10;we propose LauraGPT, a novel unified audio-and-text GPT-based LLM for audio&#10;recognition, understanding, and generation. LauraGPT is a versatile LLM that&#10;can process both audio and text inputs and generate outputs in either&#10;modalities. We propose a novel data representation that combines continuous and&#10;discrete features for audio: LauraGPT encodes input audio into continuous&#10;representations using an audio encoder and generates output audio from discrete&#10;codec codes. We propose a one-step codec vocoder to overcome the prediction&#10;challenge caused by the multimodal distribution of codec tokens. We fine-tune&#10;LauraGPT using supervised multi-task learning. Extensive experiments show that&#10;LauraGPT consistently achieves comparable to superior performance compared to&#10;strong baselines on a wide range of audio tasks related to content, semantics,&#10;paralinguistics, and audio-signal analysis, such as automatic speech&#10;recognition, speech-to-text translation, text-to-speech synthesis, speech&#10;enhancement, automated audio captioning, speech emotion recognition, and spoken&#10;language understanding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.01500" label="2312.01500">
        <attvalues>
          <attvalue for="0" value="Unsupervised Approach to Evaluate Sentence-Level Fluency: Do We Really&#10;  Need Reference?" />
          <attvalue for="1" value="  Fluency is a crucial goal of all Natural Language Generation (NLG) systems.&#10;Widely used automatic evaluation metrics fall short in capturing the fluency of&#10;machine-generated text. Assessing the fluency of NLG systems poses a challenge&#10;since these models are not limited to simply reusing words from the input but&#10;may also generate abstractions. Existing reference-based fluency evaluations,&#10;such as word overlap measures, often exhibit weak correlations with human&#10;judgments. This paper adapts an existing unsupervised technique for measuring&#10;text fluency without the need for any reference. Our approach leverages various&#10;word embeddings and trains language models using Recurrent Neural Network (RNN)&#10;architectures. We also experiment with other available multilingual Language&#10;Models (LMs). To assess the performance of the models, we conduct a comparative&#10;analysis across 10 Indic languages, correlating the obtained fluency scores&#10;with human judgments. Our code and human-annotated benchmark test-set for&#10;fluency is available at&#10;https://github.com/AnanyaCoder/TextFluencyForIndicLanaguges.&#10;" />
          <attvalue for="2" value="&#10;&#10;Fluency measures the quality of the generated text output from the model without considering the reference text \cite{fluencybook}. It accounts for grammar, spelling, word choice, and style characteristics. As stated by \cite{martindale-carpuat-2018-fluency}, maintaining text fluency avoids misapprehensions, makes interactions more realistic, and leads to higher user satisfaction and trust. Thus, fluency evaluation is essential for developing better models or screening unacceptable generations. &#10;&#10;Measuring fluency is important for evaluating the performance of NLG tasks like summarization, paraphrase generation, image captioning, etc. &#10;Fluency evaluation can be done by humans or automated metrics. For manual evaluation, proficiency in the target language is necessary. Besides, it is time-consuming, expensive, and requires a lot of human effort. &#10;&#10;To the best of our knowledge, there are no specific automatic evaluation metrics to measure text fluency without reference text. However, researchers use lexical overlap metrics to evaluate text quality in terms of fluency with the help of reference text \cite{lin-och-2004-automatic,papineni-etal-2002-bleu}. As a result, fluency is often manually assessed, which is expensive, laborious, and irreproducible.&#10;&#10;Fluency evaluation of sentences has been a linguistic ability of humans and has been an arguable subject for many decades in linguistics, psychology, and cognitive science. The question has been raised whether the grammatical knowledge underlying this ability is probabilistic or categorical \cite{Chomsky+2020,ManningManuscript-MANPS,Sprouse2007ContinuousAC}. In a similar context, \cite{ have illustrated that neural language models (LM) can be used to model human acceptability judgments. \cite{ proposed Syntactic Log-Odds Ratio (SLOR) score, which leverages sentence log probability, normalized by unigram probability and sentence length, to correlate well with human ratings at the sentence level. They investigated the practical implications of \cite{ findings for fluency evaluation of NLG, using the task of automatic compression. They also introduced a) WPSLOR: a Word-Piece \cite{ version of SLOR and b) ROUGE-LM: a combination of WPSLOR and ROUGE \cite{lin-och-2004-automatic}, the latter being a reference-based fluency evaluation approach. We extend their work by applying the syntactic log odds ratio to the 6 Indo-Aryan and 4 Dravidian languages. Our main motivation is to investigate this approach for morphologically rich languages. &#10;&#10;Data scarcity is a very common problem faced by the languages in the Indian subcontinent \cite{ In addition, the quality of the available datasets is highly questionable. &#10;&#10;Existing monolingual corpora have several issues like presence of non-unique sentences, junk/unwanted characters; requires additional cleaning and hence there is an overhead of pre-processing and de-noising. &#10;Therefore we retrieve clean, filtered data from regional news websites (see Appendix Table \ref{tab:crawling_sources}) and train multiple LMs using RNN \cite{hochreiter1997long, cho2014properties} and transformer architectures \cite{devlin-etal-2019-bert} leveraging various embedding techniques \cite{bojanowski2017enriching, heinzerling-strube-2018-bpemb, kakwani2020indicnlpsuite, khanuja2021muril} and compute sentence-level fluency score with syntactic log odds ratio. To assess the quality of the models, it is necessary to compare the scores with humans. However, due to the non-availability of benchmark dataset exclusively for fluency evaluation of the Indian subcontinent languages; we create a corpora for each language along with the designated fluency scores by humans. Proficient native speakers assign these manual scores by following strict guidelines. Table \ref{tab:my-table1} depicts our fluency scale with example sentences. &#10;Ultimately we compute the Pearson Product Moment Correlation score of the calculated fluency scores with the human assessments.&#10;&#10;Our contributions in this work are in two-fold:&#10;&#10;\begin{itemize}&#10; \item We release 5K human annotated sentences (500 sentences for each language) which can be further used as a benchmark test-set for fluency evaluation.&#10; \item We present our reference-free, unsupervised experiments to measure fluency for 6 Indo-Aryan and 4 Dravidian languages;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Text Fluency Evaluation, Artificial Intelligence, Language Model Development, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1802.06041" label="1802.06041">
        <attvalues>
          <attvalue for="0" value="Fluency Over Adequacy: A Pilot Study in Measuring User Trust in&#10;  Imperfect MT" />
          <attvalue for="1" value="  Although measuring intrinsic quality has been a key factor in the advancement&#10;of Machine Translation (MT), successfully deploying MT requires considering not&#10;just intrinsic quality but also the user experience, including aspects such as&#10;trust. This work introduces a method of studying how users modulate their trust&#10;in an MT system after seeing errorful (disfluent or inadequate) output amidst&#10;good (fluent and adequate) output. We conduct a survey to determine how users&#10;respond to good translations compared to translations that are either adequate&#10;but not fluent, or fluent but not adequate. In this pilot study, users&#10;responded strongly to disfluent translations, but were, surprisingly, much less&#10;concerned with adequacy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.04699" label="2211.04699">
        <attvalues>
          <attvalue for="0" value="FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration" />
          <attvalue for="1" value="  To accomplish punctuation restoration, most existing methods focus on&#10;introducing extra information (e.g., part-of-speech) or addressing the class&#10;imbalance problem. Recently, large-scale transformer-based pre-trained language&#10;models (PLMS) have been utilized widely and obtained remarkable success.&#10;However, the PLMS are trained on the large dataset with marks, which may not&#10;fit well with the small dataset without marks, causing the convergence to be&#10;not ideal. In this study, we propose a Feature Fusion two-stream framework&#10;(FF2) to bridge the gap. Specifically, one stream leverages a pre-trained&#10;language model to capture the semantic feature, while another auxiliary module&#10;captures the feature at hand. We also modify the computation of multi-head&#10;attention to encourage communication among heads. Then, two features with&#10;different perspectives are aggregated to fuse information and enhance context&#10;awareness. Without additional data, the experimental results on the popular&#10;benchmark IWSLT demonstrate that FF2 achieves new SOTA performance, which&#10;verifies that our approach is effective.&#10;" />
          <attvalue for="2" value="&#10;&#10;Punctuation restoration is a significant post-processing step in automatic speech recognition (ASR) systems because punctuation marks are not usually predicted. It can enhance the readability of speech transcripts and contribute to downstream tasks, such as machine translation, intent detection, or slot filling in dialogue systems. Thus, this task has attracted a large amount of interest.&#10;&#10;Generally, current works could be categorized into three groups: 1) The first line \cite{klejch2017sequence,8682260,8545470} treats this problem as a machine translation task, which feeds a non-punctuation sequence and yields the output with marks. 2) Second, some studied \cite{Adversarial, Lin2020JointPO,alam-etal-2020-punctuation,shi21_interspeech} regard it as a sequence labeling task, where a punctuation mark is assigned to each word by probability. 3) The others \cite{Che2016PunctuationPF} employ a classifier to forecast a tag for each token via taking it as a token-level classification task.&#10;&#10;To address this problem, transformer-based pre-trained language models (PLMS) have been widely applied to enhance their effectiveness. However, We argue that PLMS trained on the large-scale dataset with marks would be ineffective while fine-tuning on the dataset without marks. The main reason is that the absence of punctuation between words in a sentence may severely damage the semantics. Besides, the experiments \cite{Bottleneck, Talking} indicate that the limitation called Low-Rank Bottleneck exists in the self-attention mechanism. Concretely, with the fixed vector size of multi-head self-attention vector\cite{attention}, increasing the number of self-attention heads would decrease the vector size in each head, which causes significant degradation of its comprehension.&#10;&#10;Inspired by these observations, we propose a Feature Fusion framework based on two-stream attentions (FF2) to mitigate the shortage. Specifically, one stream utilizes a pre-trained language model to capture the semantic feature of the sequence, and another tiny module is randomly initialized, which captures the feature information on the current dataset without punctuation. Furthermore, we also modify the computation of multi-head attention to encourage communication among heads. By the mutual benefit of the two-stream modules, we first obtain two-type feature representations. Then, the two vectors are aggregated to fuse features and advance context awareness. Finally, we yield token-level punctuation tags as the output. Our main contributions are summarized as follows:&#10;\begin{itemize}&#10;\item We present a novel framework (FF2) to encourage message sharing and capture the features with different perspectives to advance the shortage of vanilla attention mechanisms. &#10;\item Without extra data, the results on the popular benchmark IWSLT indicate that FF2 can leverage the dataset itself and achieve the new state-of-the-art performance, demonstrating that FF2 is effective.&#10;\item We introduce a novel computation of multi-head attention to encourage communication among heads. The ablation studies show that it can increase the expression capability of attention heads.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Punctuation Restoration Methods, Linguistics, Artificial Intelligence, Feature Fusion Techniques, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2004.00248" label="2004.00248">
        <attvalues>
          <attvalue for="0" value="Adversarial Transfer Learning for Punctuation Restoration" />
          <attvalue for="1" value="  Previous studies demonstrate that word embeddings and part-of-speech (POS)&#10;tags are helpful for punctuation restoration tasks. However, two drawbacks&#10;still exist. One is that word embeddings are pre-trained by unidirectional&#10;language modeling objectives. Thus the word embeddings only contain&#10;left-to-right context information. The other is that POS tags are provided by&#10;an external POS tagger. So computation cost will be increased and incorrect&#10;predicted tags may affect the performance of restoring punctuation marks during&#10;decoding. This paper proposes adversarial transfer learning to address these&#10;problems. A pre-trained bidirectional encoder representations from transformers&#10;(BERT) model is used to initialize a punctuation model. Thus the transferred&#10;model parameters carry both left-to-right and right-to-left representations.&#10;Furthermore, adversarial multi-task learning is introduced to learn task&#10;invariant knowledge for punctuation prediction. We use an extra POS tagging&#10;task to help the training of the punctuation predicting task. Adversarial&#10;training is utilized to prevent the shared parameters from containing task&#10;specific information. We only use the punctuation predicting task to restore&#10;marks during decoding stage. Therefore, it will not need extra computation and&#10;not introduce incorrect tags from the POS tagger. Experiments are conducted on&#10;IWSLT2011 datasets. The results demonstrate that the punctuation predicting&#10;models obtain further performance improvement with task invariant knowledge&#10;from the POS tagging task. Our best model outperforms the previous&#10;state-of-the-art model trained only with lexical features by up to 9.2%&#10;absolute overall F_1-score on test set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06731" label="2106.06731">
        <attvalues>
          <attvalue for="0" value="Incorporating External POS Tagger for Punctuation Restoration" />
          <attvalue for="1" value="  Punctuation restoration is an important post-processing step in automatic&#10;speech recognition. Among other kinds of external information, part-of-speech&#10;(POS) taggers provide informative tags, suggesting each input token's syntactic&#10;role, which has been shown to be beneficial for the punctuation restoration&#10;task. In this work, we incorporate an external POS tagger and fuse its&#10;predicted labels into the existing language model to provide syntactic&#10;information. Besides, we propose sequence boundary sampling (SBS) to learn&#10;punctuation positions more efficiently as a sequence tagging task. Experimental&#10;results show that our methods can consistently obtain performance gains and&#10;achieve a new state-of-the-art on the common IWSLT benchmark. Further ablation&#10;studies illustrate that both large pre-trained language models and the external&#10;POS tagger take essential parts to improve the model's performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.07028" label="2002.07028">
        <attvalues>
          <attvalue for="0" value="Low-Rank Bottleneck in Multi-head Attention Models" />
          <attvalue for="1" value="  Attention based Transformer architecture has enabled significant advances in&#10;the field of natural language processing. In addition to new pre-training&#10;techniques, recent improvements crucially rely on working with a relatively&#10;larger embedding dimension for tokens. Unfortunately, this leads to models that&#10;are prohibitively large to be employed in the downstream tasks. In this paper&#10;we identify one of the important factors contributing to the large embedding&#10;size requirement. In particular, our analysis highlights that the scaling&#10;between the number of heads and the size of each head in the current&#10;architecture gives rise to a low-rank bottleneck in attention heads, causing&#10;this limitation. We further validate this in our experiments. As a solution we&#10;propose to set the head size of an attention unit to input sequence length, and&#10;independent of the number of heads, resulting in multi-head attention layers&#10;with provably more expressive power. We empirically show that this allows us to&#10;train models with a relatively smaller embedding dimension and with better&#10;performance scaling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.15472" label="2312.15472">
        <attvalues>
          <attvalue for="0" value="Towards Consistent Language Models Using Declarative Constraints" />
          <attvalue for="1" value="  Large language models have shown unprecedented abilities in generating&#10;linguistically coherent and syntactically correct natural language output.&#10;However, they often return incorrect and inconsistent answers to input&#10;questions. Due to the complexity and uninterpretability of the internally&#10;learned representations, it is challenging to modify language models such that&#10;they provide correct and consistent results. The data management community has&#10;developed various methods and tools for providing consistent answers over&#10;inconsistent datasets. In these methods, users specify the desired properties&#10;of data in a domain in the form of high-level declarative constraints. This&#10;approach has provided usable and scalable methods to delivering consistent&#10;information from inconsistent datasets. We aim to build upon this success and&#10;leverage these methods to modify language models such that they deliver&#10;consistent and accurate results. We investigate the challenges of using these&#10;ideas to obtain consistent and relevant answers from language models and report&#10;some preliminary empirical studies.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have shown unprecedented abilities in processing natural languages \cite{radford2018improving,openai2023gpt4}. &#10;They effectively generalize to perform various tasks with few or no training examples.&#10;Thus, there is a rapidly growing interest in using them to solve data-driven problems, such as, interactive question answering. &#10;&#10;Nonetheless, LLMs often provide incorrect answers to input queries and perform inaccurate inferences \cite{Ji_2023,openai2023gpt4}. &#10;Several studies indicate the recent LLMs provide {up to 40\% erroneous answers to factual questions} \cite{openai2023gpt4}. &#10;These erroneous results are important obstacle for wide-spread use of LLMs in real-world applications. &#10;&#10;To address the problem of inaccurate answers returned by LLMs, we should recognize that {LLMs are not knowledge bases, but rather approximate models of factual information}.&#10;Due to this approximate nature, they may represent inaccurate and inconsistent patterns. &#10;They may over-generalize relationships in the pretraining data, which leads to returning spurious relationships and inaccurate results. &#10;The uninterpretable mixture of learned linguistic patterns and factual information has made it challenging to eliminate incorrect information from LLMs.&#10;&#10;Nevertheless, we may be able to restrict LLMs' pre-trained representation or decoding to {adhere to semantic constraints} in the domain to avoid generating incorrect results.&#10;This is akin to the problem of {data cleaning} and {answering queries over inconsistent databases} \cite{AliceBook,Arenas:PODS:99:Consistent,bienvenu2013tractable,10.14778/1952376.1952378, DBLP:series/synthesis/2012Fan}.&#10;Databases often contain data that does not comply with the semantic constraints in their domains.&#10;For example, a person might not have any social security number or have more than one in a human resource database. &#10;The usual query processing methods might return inaccurate results over incomplete or inconsistent databases.&#10;The data management community has developed a unified, usable, and scalable approach to repairing and querying inconsistent data based on {declarative semantic constraints} \cite{Arenas:PODS:99:Consistent,10.5555/645505.656435,10.1145/3318464.3389708,Rekatsinas2017HoloCleanHD}.&#10;Hence, instead of writing long and complex imperative programs, users specify the properties of the consistent dataset succinctly in high-level declarative languages. &#10;They are usually subsets of first order logic that are sufficiently expressive to capture important knowledge in the domain yet not too expressive to make reasoning intractable. &#10;Hence, data systems may check incompatibilities or redundancies in constraints efficiently. &#10;They can also be learned from data in an unsupervised manner \cite{papenbrock2015functional, baskaran2017efficient}.&#10;This approach offers an end-to-end and unified method: data systems use these constraints both to clean or return reliable answers over inconsistent data.&#10;&#10;There has been recent effort on limiting the decoded output of LLMs to follow some {syntactical patterns}, e.g., contain certain keywords \cite{lu-etal-2021-neurologic,lew2023sequential}.&#10;In these systems, users often write (imperative) programs that detect some invalid patterns in the output of LLMs.&#10;These systems, then, use constrained optimization or probabilistic inference over the sequences generated by the LLM to reduce the probability of the outputs with invalid patterns.&#10;These efforts are steps in the right direction but fall short of providing a usable and scalable method to deliver consistent information over LLMs.&#10;First, they do not generally support semantic constraints.&#10;Second, users may have to write multiple and possibly long programs to {clean up} the output of the model.&#10;As some domain may have numerous constraints, it is challenging to develop and maintain these programs.&#10;Users must check manually whether these programs are consistent with each other and there is no redundancy across different programs.&#10;Third, they are applied during the decoding stage, therefore, they can detect and eliminate only a limited set of inconsistencies.&#10;Specifically, it is difficult for them to control all the implications of imprecise learned information in LLMs. &#10;For instance, the learned spurious relationships about one entity $e_1$ may impact how the LLM answers a question about a different but related entity $e_2$.&#10;It is challenging to understand and apply implicit implications of imprecise information about $e_1$ when they analyze the output about $e_2$ during decoding.&#10;Finally, modifying LLMs' outputs during decoding often reduces their (linguistic) coherency \cite{lu-etal-2021-neurologic,lew2023sequential}.&#10;&#10;We believe that the success using of high-level semantic constraints in data management suggests that applying semantic constraints could similarly offer a practical and scalable approach for developing and maintaining reliable and consistent LLMs.&#10;In this paper, we propose an end-to-end framework to provide a usable and unified approach to reduce inconsistencies in LLMs using high-level declarative constraints.&#10;We leverage concepts and methods of data cleaning and querying or learning over inconsistent data in the data management community. &#10;We also investigate the challenges of using declarative constraints to pre-train accurate representations and infer precise answers from LLMs.&#10;We discuss how to use current work on using declarative constraints to learn accurate ML models over inconsistent data \cite{picado2020learning,DBLP:conf/deem/ZhenCT23} and techniques to embed structured queries in vector space \cite{ijcai2019p845,DBLP:conf/iclr/RenHL20,jackermeier2023box2el} to address these problems. &#10;We also report preliminary results for integrating constraints in Llama-2 \cite{touvron2023llama}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Data Management, Computer Science, Linguistics, Language Model Limitations, Data Consistency Methods, Artificial Intelligence, Mathematics, Artificial Intelligence Integration, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.12884" label="2010.12884">
        <attvalues>
          <attvalue for="0" value="NeuroLogic Decoding: (Un)supervised Neural Text Generation with&#10;  Predicate Logic Constraints" />
          <attvalue for="1" value="  Conditional text generation often requires lexical constraints, i.e., which&#10;words should or shouldn't be included in the output text. While the dominant&#10;recipe for conditional text generation has been large-scale pretrained language&#10;models that are finetuned on the task-specific training data, such models do&#10;not learn to follow the underlying constraints reliably, even when supervised&#10;with large amounts of task-specific examples.&#10;  We propose NeuroLogic Decoding, a simple yet effective algorithm that enables&#10;neural language models -- supervised or not -- to generate fluent text while&#10;satisfying complex lexical constraints. Our approach is powerful yet efficient.&#10;It handles any set of lexical constraints that is expressible under predicate&#10;logic, while its asymptotic runtime is equivalent to conventional beam search.&#10;  Empirical results on four benchmarks show that NeuroLogic Decoding&#10;outperforms previous approaches, including algorithms that handle a subset of&#10;our constraints. Moreover, we find that unsupervised models with NeuroLogic&#10;Decoding often outperform supervised models with conventional decoding, even&#10;when the latter is based on considerably larger networks. Our results suggest&#10;the limit of large-scale neural networks for fine-grained controllable&#10;generation and the promise of inference-time algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.02308" label="2004.02308">
        <attvalues>
          <attvalue for="0" value="Learning Over Dirty Data Without Cleaning" />
          <attvalue for="1" value="  Real-world datasets are dirty and contain many errors. Examples of these&#10;issues are violations of integrity constraints, duplicates, and inconsistencies&#10;in representing data values and entities. Learning over dirty databases may&#10;result in inaccurate models. Users have to spend a great deal of time and&#10;effort to repair data errors and create a clean database for learning.&#10;Moreover, as the information required to repair these errors is not often&#10;available, there may be numerous possible clean versions for a dirty database.&#10;We propose DLearn, a novel relational learning system that learns directly over&#10;dirty databases effectively and efficiently without any preprocessing. DLearn&#10;leverages database constraints to learn accurate relational models over&#10;inconsistent and heterogeneous data. Its learned models represent patterns over&#10;all possible clean instances of the data in a usable form. Our empirical study&#10;indicates that DLearn learns accurate models over large real-world databases&#10;efficiently.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.05969" label="2002.05969">
        <attvalues>
          <attvalue for="0" value="Query2box: Reasoning over Knowledge Graphs in Vector Space using Box&#10;  Embeddings" />
          <attvalue for="1" value="  Answering complex logical queries on large-scale incomplete knowledge graphs&#10;(KGs) is a fundamental yet challenging task. Recently, a promising approach to&#10;this problem has been to embed KG entities as well as the query into a vector&#10;space such that entities that answer the query are embedded close to the query.&#10;However, prior work models queries as single points in the vector space, which&#10;is problematic because a complex query represents a potentially large set of&#10;its answer entities, but it is unclear how such a set can be represented as a&#10;single point. Furthermore, prior work can only handle queries that use&#10;conjunctions ($\wedge$) and existential quantifiers ($\exists$). Handling&#10;queries with logical disjunctions ($\vee$) remains an open problem. Here we&#10;propose query2box, an embedding-based framework for reasoning over arbitrary&#10;queries with $\wedge$, $\vee$, and $\exists$ operators in massive and&#10;incomplete KGs. Our main insight is that queries can be embedded as boxes&#10;(i.e., hyper-rectangles), where a set of points inside the box corresponds to a&#10;set of answer entities of the query. We show that conjunctions can be naturally&#10;represented as intersections of boxes and also prove a negative result that&#10;handling disjunctions would require embedding with dimension proportional to&#10;the number of KG entities. However, we show that by transforming queries into a&#10;Disjunctive Normal Form, query2box is capable of handling arbitrary logical&#10;queries with $\wedge$, $\vee$, $\exists$ in a scalable manner. We demonstrate&#10;the effectiveness of query2box on three large KGs and show that query2box&#10;achieves up to 25% relative improvement over the state of the art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.09634" label="2312.09634">
        <attvalues>
          <attvalue for="0" value="Vectorizing string entries for data processing on tables: when are&#10;  larger language models better?" />
          <attvalue for="1" value="  There are increasingly efficient data processing pipelines that work on&#10;vectors of numbers, for instance most machine learning models, or vector&#10;databases for fast similarity search. These require converting the data to&#10;numbers. While this conversion is easy for simple numerical and categorical&#10;entries, databases are strife with text entries, such as names or descriptions.&#10;In the age of large language models, what's the best strategies to vectorize&#10;tables entries, baring in mind that larger models entail more operational&#10;complexity? We study the benefits of language models in 14 analytical tasks on&#10;tables while varying the training size, as well as for a fuzzy join benchmark.&#10;We introduce a simple characterization of a column that reveals two settings:&#10;1) a dirty categories setting, where strings share much similarities across&#10;entries, and conversely 2) a diverse entries setting. For dirty categories,&#10;pretrained language models bring little-to-no benefit compared to simpler&#10;string models. For diverse entries, we show that larger language models improve&#10;data processing. For these we investigate the complexity-performance tradeoffs&#10;and show that they reflect those of classic text embedding: larger models tend&#10;to perform better, but it is useful to fine tune them for embedding purposes.&#10;" />
          <attvalue for="2" value="&#10;&#10;While much of data engineering deals with discrete entries --categories,&#10;normalized entities, or open-ended text-- there is a growing trend to use&#10;data representations made of numerical vectors. For instance, vector&#10;databases \cite{hanComprehensiveSurveyVector2023} use such representations in fast similarity searches for&#10;retrieval and fuzzy joins. Neural networks, which brought revolutions in&#10;many aspects of data processing, are also based on numerical vectors to&#10;represent the available information, including in natural language&#10;applications which deal solely with discrete tokens. However, for typical&#10;data tables, with columns containing entries of different nature and type, recent work has shown that bigger, more sophisticated, neural methods do not outperform simpler machine-learning models based on trees \cite{grinsztajnWhyTreebasedModels2022}. These tree-based methods handle discrete entries naturally, but struggle when the data cannot be represented as a moderate number of categories. In such a case, it is useful to combine them with representations of the string surface form of the entries \cite{cerdaEncodingHighcardinalityString2022}.&#10;&#10;Good vectorial representation of the string entries in tables remains&#10;crucial. Practitioners often rely on pretrained word embeddings&#10;developed in natural language processing \cite{joulinBagTricksEfficient2017} or&#10;numerical representations built from substrings&#10;\cite{cerdaEncodingHighcardinalityString2022}. Modern natural language&#10;processing has moved on to much more elaborate architectures, using&#10;pretrained attentional architectures \cite{devlinBERTPretrainingDeep2019} which have&#10;evolved to large language models LLMs, such as LLaMa&#10;\cite{touvronLLaMAOpenEfficient2023}. But vectorizing text with&#10;very large language models requires multiple expensive and&#10;rare high-end GPUs due to their memory&#10;footprint; it induces large energy consumption &#10;\cite{luccioniPowerHungryProcessing2023}. By contrast, table entries are typical fairly short strings.&#10;They seldom have the complex grammatical or narrative structures that&#10;pushed the development of language models of increasing depth and context&#10;window. This beg the question: what are the computational trade-off to create&#10;vectorial representations of string entries in tables? Are pretrained&#10;language models needed or are string representations enough? How complex should a model be? Given the cottage industry of language model&#10;--to date, the HuggingFace model hub has 42\,000 models for text&#10;classification, 2\,700 for sentence embedding--, which one to choose to&#10;embed text entries in tables? Evaluating many models for a given&#10;analysis is clearly impracticable; there is a dire need for guidelines.&#10;&#10;Here we contribute a thorough empirical study of embedding of string&#10;entries in table for data processing. We consider two settings: 1)&#10;Data analytics, ie statistical analysis of records in a table, where we&#10;consider 14 supervised learning tasks, and 2) Data engineering, in&#10;particular table assembly, where we consider fuzzy-join: joining across&#10;50 pairs of tables with imperfect alignment in the entity surface forms.&#10;We investigate more than 30 string embedding approaches. We show that a&#10;simple measure of the diversity across string entries enables separating&#10;columns on which string representations suffice, with entries that&#10;resemble ``dirty categories'', and columns with more diverse&#10;entries on which large language models are beneficial. On the diverse&#10;entries, we show that the learnings from the text-embedding literature in&#10;natural language processing carry over to the data engineering settings.&#10;&#10;Section \ref{sec:context} introduces the specific problem settings that we&#10;study and the related work on embedding entries. Section \ref{xp} then&#10;describes our benchmarking material: the datasets we use and the&#10;embedding methods that we survey. Finally, \ref{results} details the&#10;results from the benchmark, highlighting various important trends, before&#10;we conclude in \ref{sec:conlcusion}, giving high-level&#10;recommendations to encode text entries for data processing.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Language Model Analysis, Linguistics, Data Vectorization, Artificial Intelligence, Mathematics, Text Embedding Strategies" />
        </attvalues>
      </node>
      <node id="2310.11703" label="2310.11703">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey on Vector Database: Storage and Retrieval&#10;  Technique, Challenge" />
          <attvalue for="1" value="  A vector database is used to store high-dimensional data that cannot be&#10;characterized by traditional DBMS. Although there are not many articles&#10;describing existing or introducing new vector database architectures, the&#10;approximate nearest neighbor search problem behind vector databases has been&#10;studied for a long time, and considerable related algorithmic articles can be&#10;found in the literature. This article attempts to comprehensively review&#10;relevant algorithms to provide a general understanding of this booming research&#10;area. The basis of our framework categorises these studies by the approach of&#10;solving ANNS problem, respectively hash-based, tree-based, graph-based and&#10;quantization-based approaches. Then we present an overview of existing&#10;challenges for vector databases. Lastly, we sketch how vector databases can be&#10;combined with large language models and provide new possibilities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.08815" label="2207.08815">
        <attvalues>
          <attvalue for="0" value="Why do tree-based models still outperform deep learning on tabular data?" />
          <attvalue for="1" value="  While deep learning has enabled tremendous progress on text and image&#10;datasets, its superiority on tabular data is not clear. We contribute extensive&#10;benchmarks of standard and novel deep learning methods as well as tree-based&#10;models such as XGBoost and Random Forests, across a large number of datasets&#10;and hyperparameter combinations. We define a standard set of 45 datasets from&#10;varied domains with clear characteristics of tabular data and a benchmarking&#10;methodology accounting for both fitting models and finding good&#10;hyperparameters. Results show that tree-based models remain state-of-the-art on&#10;medium-sized data ($\sim$10K samples) even without accounting for their&#10;superior speed. To understand this gap, we conduct an empirical investigation&#10;into the differing inductive biases of tree-based models and Neural Networks&#10;(NNs). This leads to a series of challenges which should guide researchers&#10;aiming to build tabular-specific NNs: 1. be robust to uninformative features,&#10;2. preserve the orientation of the data, and 3. be able to easily learn&#10;irregular functions. To stimulate research on tabular architectures, we&#10;contribute a standard benchmark and raw data for baselines: every point of a 20&#10;000 compute hours hyperparameter search for each learner.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.01860" label="1907.01860">
        <attvalues>
          <attvalue for="0" value="Encoding high-cardinality string categorical variables" />
          <attvalue for="1" value="  Statistical models usually require vector representations of categorical&#10;variables, using for instance one-hot encoding. This strategy breaks down when&#10;the number of categories grows, as it creates high-dimensional feature vectors.&#10;Additionally, for string entries, one-hot encoding does not capture information&#10;in their representation.Here, we seek low-dimensional encoding of&#10;high-cardinality string categorical variables. Ideally, these should be:&#10;scalable to many categories; interpretable to end users; and facilitate&#10;statistical analysis. We introduce two encoding approaches for string&#10;categories: a Gamma-Poisson matrix factorization on substring counts, and the&#10;min-hash encoder, for fast approximation of string similarities. We show that&#10;min-hash turns set inclusions into inequality relations that are easier to&#10;learn. Both approaches are scalable and streamable. Experiments on real and&#10;simulated data show that these methods improve supervised learning with&#10;high-cardinality categorical variables. We recommend the following: if&#10;scalability is central, the min-hash encoder is the best option as it does not&#10;require any data fit; if interpretability is important, the Gamma-Poisson&#10;factorization is the best alternative, as it can be interpreted as one-hot&#10;encoding on inferred categories with informative feature names. Both models&#10;enable autoML on the original string entries as they remove the need for&#10;feature engineering or data cleaning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1607.01759" label="1607.01759">
        <attvalues>
          <attvalue for="0" value="Bag of Tricks for Efficient Text Classification" />
          <attvalue for="1" value="  This paper explores a simple and efficient baseline for text classification.&#10;Our experiments show that our fast text classifier fastText is often on par&#10;with deep learning classifiers in terms of accuracy, and many orders of&#10;magnitude faster for training and evaluation. We can train fastText on more&#10;than one billion words in less than ten minutes using a standard multicore~CPU,&#10;and classify half a million sentences among~312K classes in less than a minute.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.07685" label="1805.07685">
        <attvalues>
          <attvalue for="0" value="Fighting Offensive Language on Social Media with Unsupervised Text Style&#10;  Transfer" />
          <attvalue for="1" value="  We introduce a new approach to tackle the problem of offensive language in&#10;online social media. Our approach uses unsupervised text style transfer to&#10;translate offensive sentences into non-offensive ones. We propose a new method&#10;for training encoder-decoders using non-parallel data that combines a&#10;collaborative classifier, attention and the cycle consistency loss.&#10;Experimental results on data from Twitter and Reddit show that our method&#10;outperforms a state-of-the-art text style transfer system in two out of three&#10;quantitative metrics and produces reliable non-offensive transferred sentences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.07522" label="2005.07522">
        <attvalues>
          <attvalue for="0" value="Parallel Data Augmentation for Formality Style Transfer" />
          <attvalue for="1" value="  The main barrier to progress in the task of Formality Style Transfer is the&#10;inadequacy of training data. In this paper, we study how to augment parallel&#10;data and propose novel and simple data augmentation methods for this task to&#10;obtain useful sentence pairs with easily accessible models and systems.&#10;Experiments demonstrate that our augmented parallel data largely helps improve&#10;formality style transfer when it is used to pre-train the model, leading to the&#10;state-of-the-art results in the GYAFC benchmark dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.03126" label="2309.03126">
        <attvalues>
          <attvalue for="0" value="Everyone Deserves A Reward: Learning Customized Human Preferences" />
          <attvalue for="1" value="  Reward models (RMs) are essential for aligning large language models (LLMs)&#10;with human preferences to improve interaction quality. However, the real world&#10;is pluralistic, which leads to diversified human preferences with respect to&#10;different religions, politics, cultures, etc. Moreover, each individual can&#10;have their unique preferences on various topics. Neglecting the diversity of&#10;human preferences, current human feedback aligning methods only consider a&#10;general reward model, which is below satisfaction for customized or&#10;personalized application scenarios. To explore customized preference learning,&#10;we collect a domain-specific preference (DSP) dataset, which includes preferred&#10;responses for each given query from four practical domains. Besides, from the&#10;perspective of data efficiency, we propose a three-stage customized RM learning&#10;scheme, then empirically verify its effectiveness on both general preference&#10;datasets and our DSP set. Furthermore, we test multiple training and data&#10;strategies on the three learning stages. We find several ways to better&#10;preserve the general preferring ability while training the customized RMs,&#10;especially general preference enrichment, and customized preference imitation&#10;learning. The DSP dataset and code are available at&#10;https://github.com/Linear95/DSP.&#10;" />
          <attvalue for="2" value=" &#10;Large language models (LLMs), such as ChatGPT~\cite{chatgpt} and GPT-4~\cite{openai2022gpt4}, have recently pushed AI performance to a new height, with their astonishing capabilities in natural language processing~\cite{jiao2023chatgpt,han2023information}, logical reasoning~\cite{liu2023evaluating}, and imitation~\cite{wei2022emergent}. &#10;Besides a large amount of language modeling pretraining with tremendous tokens, aligning LLMs output with human feedback has been recognized as a critical learning strategy for LLMs' excellent performance, which enhanced the quality of human-LLM interactions~\cite{ouyang2022training,ganguli2022red,yuan2023rrhf}. &#10;To align human values, various methods have been proposed from different perspectives, such as reinforcement learning~\cite{ouyang2022training,bai2022constitutional}, ranking~\cite{yuan2023rrhf}, and reject sampling~\cite{touvron2023llama}. To guide the aligning directions, all these alignment methods depend on a reward (or preference) model~\cite{bohm2019better,askell2021general,ouyang2022training}, which provides reward scores representing the human feedback aligning degrees of LLMs' responses. Therefore, the quality of reward models is a decisive factor for human preference alignment approaches.&#10;&#10;To evaluate whether a reward model fits human preferences, prior works mainly consider two perspectives: helpfulness and harmlessness~\cite{bai2022training,fernandes2023bridging}.&#10;Helpfulness requires LLMs' responses to provide useful information~\cite{ouyang2022training,fernandes2023bridging}. Task-related rewards designed in earlier works of particular NLP domains (such as machine translation~\cite{kreutzer2018can}, summarization~\cite{ziegler2019fine}, and continuation~\cite{stiennon2020learning}) can be classified into the helpfulness category. \cite{askell2021general,ouyang2022training} extend the concept of helpfulness into a broader range without any particular task assigned, where models' responses should follow the instructions of user prompts. For harmlessness, &#10;models' responses are supposed to be fair, safe, and without toxicity~\cite{bai2022constitutional,ganguli2022red,fernandes2023bridging}.\cite{bai2022training} discover a clear trade-off between models' helpfulness and harmlessness. Moreover, several strategies~\cite{bai2022constitutional,ganguli2022red} have been proposed to improve models' harmlessness while preserving their helpfulness.&#10;&#10;Although helpfulness and harmlessness cover a wide range of mankind's tendencies, there are plenty of human preferences that cannot fall into the two categories, because of the diversity of human values. In this pluralistic world, people's preferences can diverge a lot based on their different cultures, educational backgrounds, religions, and political stands. &#10;Furthermore, even for the same person, the value of a particular LLM response can vary when the application scenario changes. For example, given the question ``What are the top valuable movies?'', a person in a movie seminar may expect an answer with detailed analysis from perspectives of acting, cinematography, or music. In contrast, he or she would possibly prefer a response with more descriptions of movies' commercial values in a business activity. Hence, there always exists a proportion of human preferences that can not be unified or even have contradictions. In the following, we call preferences that satisfy the universal human values as general preferences. Preferences related to a subgroup of people or a particular scenario are referred to customized preferences. General preferences (including helpfulness and harmlessness) have attracted increasing attention~\cite{bai2022training,bai2022constitutional,ganguli2022red,touvron2023llama}, while customized preferences remain unexplored. &#10;&#10;Moreover, the above classification of human preferences naturally leads to an interesting question: ``How to learn a customized reward model well while preserving its general preference ability?'' A high-qualified customized reward model is practically valued to enhance the domain-specific LLM fine-tuning by serving as a learning critic or an evaluation metric~\cite{askell2021general,&#10;touvron2023llama}, because general LLMs can not handle all application domains, especially in which professional knowledge are required~\cite{beltagy2019scibert,gu2021domain,li2023llava}. &#10;However, training a customized reward model can be much more difficult due to the scarcity of customized human preference data. General preferences represent mankind's common values, which can be collected across different groups of people and various application scenarios. In contrast, customized or personalized preferences require data collection from a particular person or domain. A worth-trying strategy is first training an RM on a large number of general preferences, then fine-tuning it with a few customized preferences. A similar reward pre-training idea has been empirically tested by \cite{askell2021general} as preference model pre-training (PMP). However, the transfer learning ability of PMP has not been evaluated on customized human preferences.&#10;&#10;To address the challenge of customized human preference learning, we construct a simulation dataset with the assistance of ChatGPT~\cite{chatgpt}, in which preferred responses are collected from four application domains: Academy, Business, Entertainment, and Literature\&amp;Art. We call this new dataset the Domain-Specific Preference (DSP) set.&#10;Then we train general and domain-specific reward models with LLaMA~\cite{touvron2023llama} as the base model using both general preferences~\cite{bai2022training,nakano2021webgpt,peng2023instruction} and DSP data. &#10;To study the learning behaviors of customized RMs, we divided the training process into three stages: base LM training, general RM fine-tuning, and customized RM fine-tuning (as in Figure~\ref{fig:rm-training-stage}). We try different data and training strategies respectively on the three training stages, and discover several ways to fit customized preferences while preserving general reward performance.&#10;Our main contributions are:&#10;\begin{itemize}[leftmargin=0.5cm]&#10; \item We collected a domain-specific preference (DSP) dataset with the usage of ChatGPT.&#10; \item We proposed a three-stage training scheme for customized RM learning, and verified its effectiveness on both general preference and domain-specific preference datasets. &#10; \item We discovered that imitation learning on customized preferences and general preference data enrichment are the two effective ways to preserve RMs' general preferring ability when fitting the customized human preferences.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human Preference Modeling, Computer Science, Machine Learning, Customized Reward Learning, Linguistics, Artificial Intelligence, Natural Language Processing, Psychology, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="2209.07858" label="2209.07858">
        <attvalues>
          <attvalue for="0" value="Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors,&#10;  and Lessons Learned" />
          <attvalue for="1" value="  We describe our early efforts to red team language models in order to&#10;simultaneously discover, measure, and attempt to reduce their potentially&#10;harmful outputs. We make three main contributions. First, we investigate&#10;scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B&#10;parameters) and 4 model types: a plain language model (LM); an LM prompted to&#10;be helpful, honest, and harmless; an LM with rejection sampling; and a model&#10;trained to be helpful and harmless using reinforcement learning from human&#10;feedback (RLHF). We find that the RLHF models are increasingly difficult to red&#10;team as they scale, and we find a flat trend with scale for the other model&#10;types. Second, we release our dataset of 38,961 red team attacks for others to&#10;analyze and learn from. We provide our own analysis of the data and find a&#10;variety of harmful outputs, which range from offensive language to more subtly&#10;harmful non-violent unethical outputs. Third, we exhaustively describe our&#10;instructions, processes, statistical methodologies, and uncertainty about red&#10;teaming. We hope that this transparency accelerates our ability to work&#10;together as a community in order to develop shared norms, practices, and&#10;technical standards for how to red team language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.00890" label="2306.00890">
        <attvalues>
          <attvalue for="0" value="LLaVA-Med: Training a Large Language-and-Vision Assistant for&#10;  Biomedicine in One Day" />
          <attvalue for="1" value="  Conversational generative AI has demonstrated remarkable promise for&#10;empowering biomedical practitioners, but current investigations focus on&#10;unimodal text. Multimodal conversational AI has seen rapid progress by&#10;leveraging billions of image-text pairs from the public web, but such&#10;general-domain vision-language models still lack sophistication in&#10;understanding and conversing about biomedical images. In this paper, we propose&#10;a cost-efficient approach for training a vision-language conversational&#10;assistant that can answer open-ended research questions of biomedical images.&#10;The key idea is to leverage a large-scale, broad-coverage biomedical&#10;figure-caption dataset extracted from PubMed Central, use GPT-4 to&#10;self-instruct open-ended instruction-following data from the captions, and then&#10;fine-tune a large general-domain vision-language model using a novel curriculum&#10;learning method. Specifically, the model first learns to align biomedical&#10;vocabulary using the figure-caption pairs as is, then learns to master&#10;open-ended conversational semantics using GPT-4 generated instruction-following&#10;data, broadly mimicking how a layperson gradually acquires biomedical&#10;knowledge. This enables us to train a Large Language and Vision Assistant for&#10;BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med&#10;exhibits excellent multimodal conversational capability and can follow&#10;open-ended instruction to assist with inquiries about a biomedical image. On&#10;three standard biomedical visual question answering datasets, LLaVA-Med&#10;outperforms previous supervised state-of-the-art on certain metrics. To&#10;facilitate biomedical multimodal research, we will release our&#10;instruction-following data and the LLaVA-Med model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.05972" label="2209.05972">
        <attvalues>
          <attvalue for="0" value="Don't Judge a Language Model by Its Last Layer: Contrastive Learning&#10;  with Layer-Wise Attention Pooling" />
          <attvalue for="1" value="  Recent pre-trained language models (PLMs) achieved great success on many&#10;natural language processing tasks through learning linguistic features and&#10;contextualized sentence representation. Since attributes captured in stacked&#10;layers of PLMs are not clearly identified, straightforward approaches such as&#10;embedding the last layer are commonly preferred to derive sentence&#10;representations from PLMs. This paper introduces the attention-based pooling&#10;strategy, which enables the model to preserve layer-wise signals captured in&#10;each layer and learn digested linguistic features for downstream tasks. The&#10;contrastive learning objective can adapt the layer-wise attention pooling to&#10;both unsupervised and supervised manners. It results in regularizing the&#10;anisotropic space of pre-trained embeddings and being more uniform. We evaluate&#10;our model on standard semantic textual similarity (STS) and semantic search&#10;tasks. As a result, our method improved the performance of the base contrastive&#10;learned BERT_base and variants.&#10;" />
          <attvalue for="2" value="&#10;Pre-trained language models (PLMs) \cite{kenton2019bert, liu2019roberta, radford2019language, raffel2019exploring} have shown competitive performance on many natural language processing (NLP) tasks. Also, contrastive learning using the PLMs shows the highest performance in sentence representation. Contrastive learning is to learn effective representations by staying semantically close sample pairs together while dissimilar ones are far apart\cite{hadsell2006dimensionality}.&#10;&#10;In general, PLMs use either $[CLS]$ tokens in the last layer, $AVG$ which is the average representation of tokens in the last layer\cite{reimers2019sentence, li2020sentence}, or $AVG_{FL}$ which is the average representation of tokens in the first and last layers\cite{gao2021simcse}, to pool out sentence representation from word representations. However, since language models show performance gaps by domain when trained on different objectives, the fixed pooling strategy has limitations in performance improvement. &#10;&#10;Figure \ref{fig:intro_fig1} and \ref{fig:intro_fig2} show the Spearman's correlation score of each layer or pooling method in PLMs. We evaluated the test set of the standard semantic textual similarity (STS) dataset\cite{cer2017semeval, agirre2012semeval,agirre2013sem,agirre2014semeval,agirre2015semeval,agirre2016semeval,marelli2014sick}. &#10;&#10;The comparison of performance when pooling each layer shown in Figure \ref{fig:intro_fig1} indicates that using only a specific layer for pooling is insufficient. Other layers other than the last layer may contain substantial information for sentence representation. For example, for the STS benchmark (STS-B) task\cite{cer2017semeval}, BERT$_{base}$ with $[CLS]$ embedding scored the highest at the fourth layer (48.66\%), which is about 20\% higher than the last layer. &#10;&#10;Figure \ref{fig:intro_fig2} shows that simply pooling from more layers impedes the performance by comparing models pooled from the first and last layer and the last layer. In addition, there is no consistent tendency to compare effectiveness for a given layer between $[CLS]$ pooling and average pooling. &#10;&#10;Motivated by this point, we designed the attention networks and task-agnostic pooling methods to assign more weights to spots that need more focus in the layer and lead to representation vector optimization. Our proposed method outperforms previously fixed pooling strategies in contrastive learning. In addition, contrastive learning models with layer-wise attention pooling show a higher semantic search performance with the same parameters.&#10;&#10;In summary, the contributions of this paper are as follows:&#10;\begin{itemize}&#10;\item We proposed layer-wise attention pooling to assign weights to each layer and learn sentence representation fitted to a given task.&#10;\item To our knowledge, our pooling strategy shows the best performance out of all InfoNCE-based loss functions for the sentence embedding tasks.&#10;\item For the semantic search evaluation, we excluded the proposed pooling method in the inference phase and obtained better performance.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Linguistics, Artificial Intelligence, Attention Mechanisms, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2009.12303" label="2009.12303">
        <attvalues>
          <attvalue for="0" value="Towards Debiasing NLU Models from Unknown Biases" />
          <attvalue for="1" value="  NLU models often exploit biases to achieve high dataset-specific performance&#10;without properly learning the intended task. Recently proposed debiasing&#10;methods are shown to be effective in mitigating this tendency. However, these&#10;methods rely on a major assumption that the types of bias should be known&#10;a-priori, which limits their application to many NLU tasks and datasets. In&#10;this work, we present the first step to bridge this gap by introducing a&#10;self-debiasing framework that prevents models from mainly utilizing biases&#10;without knowing them in advance. The proposed framework is general and&#10;complementary to the existing debiasing methods. We show that it allows these&#10;existing methods to retain the improvement on the challenge datasets (i.e.,&#10;sets of examples designed to expose models' reliance on biases) without&#10;specifically targeting certain biases. Furthermore, the evaluation suggests&#10;that applying the framework results in improved overall robustness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.01024" label="2106.01024">
        <attvalues>
          <attvalue for="0" value="Why Machine Reading Comprehension Models Learn Shortcuts?" />
          <attvalue for="1" value="  Recent studies report that many machine reading comprehension (MRC) models&#10;can perform closely to or even better than humans on benchmark datasets.&#10;However, existing works indicate that many MRC models may learn shortcuts to&#10;outwit these benchmarks, but the performance is unsatisfactory in real-world&#10;applications. In this work, we attempt to explore, instead of the expected&#10;comprehension skills, why these models learn the shortcuts. Based on the&#10;observation that a large portion of questions in current datasets have shortcut&#10;solutions, we argue that larger proportion of shortcut questions in training&#10;data make models rely on shortcut tricks excessively. To investigate this&#10;hypothesis, we carefully design two synthetic datasets with annotations that&#10;indicate whether a question can be answered using shortcut solutions. We&#10;further propose two new methods to quantitatively analyze the learning&#10;difficulty regarding shortcut and challenging questions, and revealing the&#10;inherent learning mechanism behind the different performance between the two&#10;kinds of questions. A thorough empirical analysis shows that MRC models tend to&#10;learn shortcut questions earlier than challenging questions, and the high&#10;proportions of shortcut questions in training sets hinder models from exploring&#10;the sophisticated reasoning skills in the later stage of training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00592" label="2011.00592">
        <attvalues>
          <attvalue for="0" value="Vec2Sent: Probing Sentence Embeddings with Natural Language Generation" />
          <attvalue for="1" value="  We introspect black-box sentence embeddings by conditionally generating from&#10;them with the objective to retrieve the underlying discrete sentence. We&#10;perceive of this as a new unsupervised probing task and show that it correlates&#10;well with downstream task performance. We also illustrate how the language&#10;generated from different encoders differs. We apply our approach to generate&#10;sentence analogies from sentence embeddings.&#10;" />
          <attvalue for="2" value="&#10;Generalizing the concept of word embeddings to sentence level, sentence embeddings (a.k.a.\ sentence encoders) are ubiquitous in NLP as features in downstream classification tasks and in semantic similarity and retrieval applications \cite{Kiros.2015,Conneau.2017}. &#10;Probing sentence encoders for the linguistic information signals they contain has likewise become an important field of research, as this &#10;allows to introspect otherwise black-box representations \cite{Adi.2017,Conneau.2018a}. &#10;The idea behind probing tasks is to &#10;query representations for certain kinds of linguistic information such as the dependency tree depth of an encoded sentence. There are a variety of &#10;problems surrounding current probing task specifications: (i) probing tasks need to be manually construed, which brings with it a certain degree of arbitrariness and incompleteness; (ii) most probing tasks require labeled datasets or trained classifiers such as dependency parsers for linguistic processing---however, these may be unavailable for many low-resource languages or available only to a limited degree; (iii) it is not entirely clear how probing tasks have to be designed, &#10;e.g., how much training data they require and which classifier to use for probing \cite{Eger_howto:2020}; (iv) &#10;\newcite{Ravichander2020ProbingTP} also argue that standard probing tasks do not outline the information signals a classifier actually uses for making predictions.&#10;&#10;Our contribution &#10;is &#10;an alternative, more direct introspection of sentence embeddings, namely, through conditional natural language generation, which we call ``vec2sent'' (V2S). &#10;By retrieving and (manually) investigating the discrete output obtained from a dense vector representation, linguistic properties of the embedding may be `directly' unveiled: e.g., we expect that a word-order insensitive model &#10;would have a comparatively hard time in restoring the correct syntax of an encoded sentence. &#10;V2S requires no labeled data, &#10;making it applicable to any language that has at least several ten thousands of written sentences available---e.g., it is particularly suitable for multilingual probing \cite{Krasnowska.2019,Eger_howto:2020}. &#10;Since V2S makes the opaque space $\mathbb{R}^d$ observable, it may also reveal intriguing properties of how encoders encode text (cf.\ Table \ref{table:examples}), without having to `guess' relevant probing tasks.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Generation Techniques, Computer Science, Linguistics, Sentence Embeddings Analysis, Artificial Intelligence, Mathematics, Natural Language Processing, Unsupervised Probing Methods" />
        </attvalues>
      </node>
      <node id="2104.07275" label="2104.07275">
        <attvalues>
          <attvalue for="0" value="Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic&#10;  Parsing" />
          <attvalue for="1" value="  An effective recipe for building seq2seq, non-autoregressive, task-oriented&#10;parsers to map utterances to semantic frames proceeds in three steps: encoding&#10;an utterance $x$, predicting a frame's length |y|, and decoding a |y|-sized&#10;frame with utterance and ontology tokens. Though empirically strong, these&#10;models are typically bottlenecked by length prediction, as even small&#10;inaccuracies change the syntactic and semantic characteristics of resulting&#10;frames. In our work, we propose span pointer networks, non-autoregressive&#10;parsers which shift the decoding task from text generation to span prediction;&#10;that is, when imputing utterance spans into frame slots, our model produces&#10;endpoints (e.g., [i, j]) as opposed to text (e.g., &quot;6pm&quot;). This natural&#10;quantization of the output space reduces the variability of gold frames,&#10;therefore improving length prediction and, ultimately, exact match.&#10;Furthermore, length prediction is now responsible for frame syntax and the&#10;decoder is responsible for frame semantics, resulting in a coarse-to-fine&#10;model. We evaluate our approach on several task-oriented semantic parsing&#10;datasets. Notably, we bridge the quality gap between non-autogressive and&#10;autoregressive parsers, achieving 87 EM on TOPv2 (Chen et al. 2020).&#10;Furthermore, due to our more consistent gold frames, we show strong&#10;improvements in model generalization in both cross-domain and cross-lingual&#10;transfer in low-resource settings. Finally, due to our diminished output&#10;vocabulary, we observe 70% reduction in latency and 83% reduction in memory at&#10;beam size 5 compared to prior non-autoregressive parsers.&#10;" />
          <attvalue for="2" value="&#10;&#10;Task-oriented conversational assistants typically first employ semantic parsers to map utterances to frames \cite{hemphill1990atis,coucke2018snips,gupta2018semantic,rongali2020don,decoupled}. Due to performance constraints in real-world deployments, recent work in task-oriented semantic parsing has shifted towards building seq2seq, non-autoregressive parsers optimized for both quality and latency \cite{Zhu2020DontPI,nar_semantic_parsing}. These models enforce strong independence assumptions during decoding, allowing frame components (e.g., ontology and utterance tokens) to be generated in parallel. However, the application of off-the-shelf, non-autoregressive algorithms to task-oriented semantic parsing is not trivial, often leading to brittle implementations with sub-optimal, opinionated components \cite{nar_semantic_parsing}.&#10;&#10;One popular family of seq2seq, non-autoregressive, task-oriented parsers is based on the mask-predict algorithm \cite{ghazvininejad2019maskpredict}, which operates in three steps: encoding, length prediction, and decoding.&#10;Unlike machine translation, task-oriented semantic parsing does not benefit much from iterative refinement; therefore, frames are typically generated in one step. However, this increases the burden of upstream components, namely placing a major bottleneck on the length prediction module. Therefore, even off-by-one length errors can change the syntactic and semantic characteristics of resulting frames.&#10;&#10;In this work, we create span pointer networks which model a seq2seq, non-autoregressive parsing task centered around span prediction as opposed to text generation. Figure \ref{fig:span_pointer_model} illustrates an application of our model; when parsing the utterance ``message I'll be there at 6pm'', our model produces span endpoints [1, 5] as opposed to exact text ``I'll be there at 6pm'' in the appropriate leaf slot. As a result, our length module implicitly predicts the syntax of a frame (i.e., how many intents and slots does the utterance have?) and the decoder resolves leaf arguments with span endpoints (i.e., what utterance spans should be included in the frame?). This additionally creates a parallel among our approach and coarse-to-fine \cite{Dong2018CoarsetoFineDF} modeling as our length predictor predicts a general structure that our decoder further refines. Despite this shift in behavior, our approach is largely compatible with the typical mask-predict methodology, as we largely modify the frame representation and model architecture to be span-based.&#10;&#10;We evaluate span pointer networks on three axes: quality, generalizability, and resources. First, we benchmark exact match (EM) on multiple task-oriented semantic parsing datasets, including TOPv2 \cite{chen-2020-topv2} and TOP \cite{gupta2018semantic}. Our non-autoregressive parser is competitive, achieving 87 EM on TOPv2, and matches or exceeds autoregressive parsers in many cases. &#10;Second, we evaluate generalizability by setting up cross-domain and cross-lingual transfer experiments. Our non-autoregressive outperforms strong baselines, notably outperforming autoregressive parsers by +15 EM when averaged across 5 zero-shot multilingual settings.&#10;Third, due to the restricted decoder vocabulary, we measure resource savings, in particular, latency and memory usage. Compared to non-autoregressive models our parser achieves 70\% reduction in latency and 83\% reduction in memory with a beam size of 5. &#10;&#10;To summarize, our contributions are: (1) We propose span pointer networks, non-autoregressive parsers which use span prediction as opposed to text generation; (2) We evaluate several aspects of span pointer networks, conducting a series of experiments around quality, generalizability, and resources; (3) We empirically show improvements on all three axes, notably outperforming both non-autoregressive and autoregressive parsers.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Semantic Parsing, Machine Learning, Computer Science, Linguistics, Non-Autoregressive Models, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1810.07942" label="1810.07942">
        <attvalues>
          <attvalue for="0" value="Semantic Parsing for Task Oriented Dialog using Hierarchical&#10;  Representations" />
          <attvalue for="1" value="  Task oriented dialog systems typically first parse user utterances to&#10;semantic frames comprised of intents and slots. Previous work on task oriented&#10;intent and slot-filling work has been restricted to one intent per query and&#10;one slot label per token, and thus cannot model complex compositional requests.&#10;Alternative semantic parsing systems have represented queries as logical forms,&#10;but these are challenging to annotate and parse. We propose a hierarchical&#10;annotation scheme for semantic parsing that allows the representation of&#10;compositional queries, and can be efficiently and accurately parsed by standard&#10;constituency parsing models. We release a dataset of 44k annotated queries&#10;(fb.me/semanticparsingdialog), and show that parsing models outperform&#10;sequence-to-sequence approaches on this dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.11458" label="2001.11458">
        <attvalues>
          <attvalue for="0" value="Don't Parse, Generate! A Sequence to Sequence Architecture for&#10;  Task-Oriented Semantic Parsing" />
          <attvalue for="1" value="  Virtual assistants such as Amazon Alexa, Apple Siri, and Google Assistant&#10;often rely on a semantic parsing component to understand which action(s) to&#10;execute for an utterance spoken by its users. Traditionally, rule-based or&#10;statistical slot-filling systems have been used to parse &quot;simple&quot; queries; that&#10;is, queries that contain a single action and can be decomposed into a set of&#10;non-overlapping entities. More recently, shift-reduce parsers have been&#10;proposed to process more complex utterances. These methods, while powerful,&#10;impose specific limitations on the type of queries that can be parsed; namely,&#10;they require a query to be representable as a parse tree.&#10;  In this work, we propose a unified architecture based on Sequence to Sequence&#10;models and Pointer Generator Network to handle both simple and complex queries.&#10;Unlike other works, our approach does not impose any restriction on the&#10;semantic parse schema. Furthermore, experiments show that it achieves state of&#10;the art performance on three publicly available datasets (ATIS, SNIPS, Facebook&#10;TOP), relatively improving between 3.3% and 7.7% in exact match accuracy over&#10;previous systems. Finally, we show the effectiveness of our approach on two&#10;internal datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.13655" label="2009.13655">
        <attvalues>
          <attvalue for="0" value="Conversational Semantic Parsing" />
          <attvalue for="1" value="  The structured representation for semantic parsing in task-oriented assistant&#10;systems is geared towards simple understanding of one-turn queries. Due to the&#10;limitations of the representation, the session-based properties such as&#10;co-reference resolution and context carryover are processed downstream in a&#10;pipelined system. In this paper, we propose a semantic representation for such&#10;task-oriented conversational systems that can represent concepts such as&#10;co-reference and context carryover, enabling comprehensive understanding of&#10;queries in a session. We release a new session-based, compositional&#10;task-oriented parsing dataset of 20k sessions consisting of 60k utterances.&#10;Unlike Dialog State Tracking Challenges, the queries in the dataset have&#10;compositional forms. We propose a new family of Seq2Seq models for the&#10;session-based parsing above, which achieve better or comparable performance to&#10;the current state-of-the-art on ATIS, SNIPS, TOP and DSTC2. Notably, we improve&#10;the best known results on DSTC2 by up to 5 points for slot-carryover.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03714" label="2010.03714">
        <attvalues>
          <attvalue for="0" value="Don't Parse, Insert: Multilingual Semantic Parsing with Insertion Based&#10;  Decoding" />
          <attvalue for="1" value="  Semantic parsing is one of the key components of natural language&#10;understanding systems. A successful parse transforms an input utterance to an&#10;action that is easily understood by the system. Many algorithms have been&#10;proposed to solve this problem, from conventional rulebased or statistical&#10;slot-filling systems to shiftreduce based neural parsers. For complex parsing&#10;tasks, the state-of-the-art method is based on autoregressive sequence to&#10;sequence models to generate the parse directly. This model is slow at inference&#10;time, generating parses in O(n) decoding steps (n is the length of the target&#10;sequence). In addition, we demonstrate that this method performs poorly in&#10;zero-shot cross-lingual transfer learning settings. In this paper, we propose a&#10;non-autoregressive parser which is based on the insertion transformer to&#10;overcome these two issues. Our approach 1) speeds up decoding by 3x while&#10;outperforming the autoregressive model and 2) significantly improves&#10;cross-lingual transfer in the low-resource setting by 37% compared to&#10;autoregressive baseline. We test our approach on three well-known monolingual&#10;datasets: ATIS, SNIPS and TOP. For cross lingual semantic parsing, we use the&#10;MultiATIS++ and the multilingual TOP datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#09;In this section, we introduce the sequence generation via insertion operations and the pretrained models we leverage in our work.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1805.04793" label="1805.04793">
        <attvalues>
          <attvalue for="0" value="Coarse-to-Fine Decoding for Neural Semantic Parsing" />
          <attvalue for="1" value="  Semantic parsing aims at mapping natural language utterances into structured&#10;meaning representations. In this work, we propose a structure-aware neural&#10;architecture which decomposes the semantic parsing process into two stages.&#10;Given an input utterance, we first generate a rough sketch of its meaning,&#10;where low-level information (such as variable names and arguments) is glossed&#10;over. Then, we fill in missing details by taking into account the natural&#10;language input and the sketch itself. Experimental results on four datasets&#10;characteristic of different domains and meaning representations show that our&#10;approach consistently improves performance, achieving competitive results&#10;despite the use of relatively simple decoders.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.12624" label="2312.12624">
        <attvalues>
          <attvalue for="0" value="Building a Llama2-finetuned LLM for Odia Language Utilizing Domain&#10;  Knowledge Instruction Set" />
          <attvalue for="1" value="  Building LLMs for languages other than English is in great demand due to the&#10;unavailability and performance of multilingual LLMs, such as understanding the&#10;local context. The problem is critical for low-resource languages due to the&#10;need for instruction sets. In a multilingual country like India, there is a&#10;need for LLMs supporting Indic languages to provide generative AI and LLM-based&#10;technologies and services to its citizens.&#10;  This paper presents our approach of i) generating a large Odia instruction&#10;set, including domain knowledge data suitable for LLM fine-tuning, and ii)&#10;building a Llama2-finetuned model tailored for enhanced performance in the Odia&#10;domain. The proposed work will help researchers build an instruction set and&#10;LLM, particularly for Indic languages. We will release the model and&#10;instruction set for the public for research and noncommercial purposes.&#10;" />
          <attvalue for="2" value="&#10;In the ever-evolving landscape of Generative AI, language models have played a pivotal role in revolutionizing how we interact with and understand the world's diverse linguistic tapestry. These models have demonstrated remarkable capabilities, from the groundbreaking BERT \cite{devlin2018bert} to the omnipotent GPT, particularly in widely spoken languages. However, the realm of Generative AI is not one-size-fits-all, and the inadequacy of state-of-the-art models when applied to less-resourced languages, especially those belonging to the Indic language family, remains an unsettling challenge.&#10;&#10;The multilingual landscape of NLP has witnessed significant advancements in recent years, with several models tailored to support a myriad of languages. However, a noticeable gap in the digital representation persists when it comes to providing comprehensive support for Indic languages, including Odia. This gap fuels our dedication to the cause – empowering the Odia language through robust and dedicated language models.&#10;&#10;Odia, an Indic language spoken predominantly in the Indian state of Odisha, boasts a rich cultural heritage and a growing digital presence. However, it has long been underserved in the digital sphere, and the need to empower it through enhanced language representation has never been more pressing \cite{parida2022universal}. Although only some multilingual LLMs support the Odia language due to the amount and quality of data, the performance of these LLMs could be better, as shown in Fig. \ref{fig_chatgpt}. &#10;&#10;In this paper, we embark on a journey to introduce the Llama2-finetuned language model meticulously tuned to cater to the specific requirements of the Odia language. Llama2 \cite{touvron2023llama} proudly stands as a member of a burgeoning family of models, poised to bridge the multilingual gap by providing precise, contextually relevant results for Odia while championing linguistic diversity in the NLP landscape.&#10;&#10;Our motivation transcends the boundaries of linguistic capabilities; it is rooted in the broader mission of preserving and promoting underrepresented languages. Llama2's development and meticulous fine-tuning of an extensive Odia domain knowledge dataset serve not only to enhance its performance but also to contribute to safeguarding against potential plagiarism in Odia language research. The authors aspire to foster innovation and knowledge creation within the Odia-speaking community by offering researchers a dependable and original language model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Indic Language Modeling, Low-Resource Languages, Computational Linguistics, Multilingual LLMs, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2104.11070" label="2104.11070">
        <attvalues>
          <attvalue for="0" value="Adapting Long Context NLM for ASR Rescoring in Conversational Agents" />
          <attvalue for="1" value="  Neural Language Models (NLM), when trained and evaluated with context&#10;spanning multiple utterances, have been shown to consistently outperform both&#10;conventional n-gram language models and NLMs that use limited context. In this&#10;paper, we investigate various techniques to incorporate turn based context&#10;history into both recurrent (LSTM) and Transformer-XL based NLMs. For recurrent&#10;based NLMs, we explore context carry over mechanism and feature based&#10;augmentation, where we incorporate other forms of contextual information such&#10;as bot response and system dialogue acts as classified by a Natural Language&#10;Understanding (NLU) model. To mitigate the sharp nearby, fuzzy far away problem&#10;with contextual NLM, we propose the use of attention layer over lexical&#10;metadata to improve feature based augmentation. Additionally, we adapt our&#10;contextual NLM towards user provided on-the-fly speech patterns by leveraging&#10;encodings from a large pre-trained masked language model and performing fusion&#10;with a Transformer-XL based NLM. We test our proposed models using N-best&#10;rescoring of ASR hypotheses of task-oriented dialogues and also evaluate on&#10;downstream NLU tasks such as intent classification and slot labeling. The best&#10;performing model shows a relative WER between 1.6% and 9.1% and a slot labeling&#10;F1 score improvement of 4% over non-contextual baselines.&#10;" />
          <attvalue for="2" value="&#10;&#10;Conversations in goal or task oriented conversational interfaces, such as digital personal assistants or chatbots, typically span multiple turns of back and forth between a user and a bot \cite{multiwoz:18,houseini_2020}. These interactions by definition involve accomplishing a specific task in a particular domain and mostly adhere to a dialogue structure that can be determined beforehand. Hence, most real-time task oriented chatbot systems allow users to provide a dialogue grammar that tries to capture usage patterns, intents, slots, and the conversation structure of the interactions \cite{petar:2015,lemon-etal-2006-isu, aggandhe:18}. &#10;&#10;A conventional NLM rescorer for ASR is typically trained and evaluated on context that is limited to single turns \cite{mikolov:15,bengio:01} and is therefore sub-optimal in task oriented dialogue systems. A number of studies recently explored incorporating cross utterance context into both recurrent and non-recurrent NLMs. In particular, recurrent neural network (RNN) and LSTM based NLMs are trained and evaluated without resetting hidden states across sentences \cite{ms:2018, irie:19, xiong-etal-2018-session}. The work proposed in \cite{partha:19} uses a multi-head dot-product attention over LSTM hidden states to better exploit the contextual information. However, context carry over models with either RNNs or LSTMs each has its own limitations. The wide spread adoption of transformer architecture based on self attention \cite{vaswani:17} and pretrained masked language models \cite{devlin:18} paved the way for use of transformers for NLMs. In transformer NLMs, long span context modeling is achieved through longer or adaptive attention spans \cite{sukhbaatar-etal-2019-adaptive}. While the initial work used modified attention masks to handle longer input sequences more efficiently, Transformer-XL (TXL) \cite{dai-etal-2019-transformer} made use of segment wise recurrence, making it well suited for long span decoding. More recently, \cite{sun2021transformer} further tried to improve cross utterance decoding using TXL by adding a LSTM fusion layer, where the hidden states of LSTM are carried over multiple sentences. The work in \cite{kim-etal-2019-gated} explored the use of a conversational context embedding from recent history to improve the contextualization of end-to-end ASR models where as some other works \cite{mikolov:12,keli:2018,anirudh:18,Chen:2015} used an explicit topic vector or a neural cache and a domain classifier for domain and contextual adaptation. &#10;&#10;While vast majority of the previous work focused on using context spanning multiple utterances, they do not take into account other contextual signals from NLU such as dialogue act, predefined dialogue grammars or any user-provided speech patterns. &#10;In this work, we investigate a number of ways to improve contextualization of LSTM and TXL based NLMs to transcribe task-oriented dialogue audio. Specifically, we use long span context that spans across all the utterances in the same dialogue session and system dialogue acts as classified by a NLU model. Additionally, in order to adapt the NLM towards user provided speech patterns in the pre-defined dialogue grammar, we use semantic embeddings derived from a large pretrained masked language models such as BERT \cite{devlin:18}. We use perplexity (PPL) and word error rate (WER) as our ASR evaluation metrics and also evaluate on the natural language understanding (NLU) metrics such as intent classification (IC) and slot labeling (SL) F1 scores to measure the impact on end to end task success rate. The overall contributions of this work can be summarized as follows :&#10;\begin{itemize}[topsep=0pt, leftmargin=*]&#10;&#10;\item For speech recognition in task-oriented conversations, we show that utilizing long span context from past utterances in the same dialogue session along with system dialogue act, provides significant improvements in WER reduction (WERR).&#10; \item We propose a new architecture that lets us leverage user provided speech patterns by using embeddings derived from a pretrained masked language model, such as BERT, to perform on-the-fly adaptation of a neural rescorer.&#10; \item By combining the different forms of contextual information, we successfully train NLMs that achieve a WERR ranging from 1.6\% to 9.1\%, IC F1 improvement ranging from 0.3\% to 1.2\% and SL F1 ranging from 0.4\% to 4.5\% over a non-contextual LSTM LM baseline.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Natural Language Understanding, Contextual Information Processing, Artificial Intelligence, Neural Language Models" />
        </attvalues>
      </node>
      <node id="1906.11604" label="1906.11604">
        <attvalues>
          <attvalue for="0" value="Gated Embeddings in End-to-End Speech Recognition for&#10;  Conversational-Context Fusion" />
          <attvalue for="1" value="  We present a novel conversational-context aware end-to-end speech recognizer&#10;based on a gated neural network that incorporates&#10;conversational-context/word/speech embeddings. Unlike conventional speech&#10;recognition models, our model learns longer conversational-context information&#10;that spans across sentences and is consequently better at recognizing long&#10;conversations. Specifically, we propose to use the text-based external word&#10;and/or sentence embeddings (i.e., fastText, BERT) within an end-to-end&#10;framework, yielding a significant improvement in word error rate with better&#10;conversational-context representation. We evaluated the models on the&#10;Switchboard conversational speech corpus and show that our model outperforms&#10;standard end-to-end speech recognition models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.10215" label="1806.10215">
        <attvalues>
          <attvalue for="0" value="Contextual Language Model Adaptation for Conversational Agents" />
          <attvalue for="1" value="  Statistical language models (LM) play a key role in Automatic Speech&#10;Recognition (ASR) systems used by conversational agents. These ASR systems&#10;should provide a high accuracy under a variety of speaking styles, domains,&#10;vocabulary and argots. In this paper, we present a DNN-based method to adapt&#10;the LM to each user-agent interaction based on generalized contextual&#10;information, by predicting an optimal, context-dependent set of LM&#10;interpolation weights. We show that this framework for contextual adaptation&#10;provides accuracy improvements under different possible mixture LM partitions&#10;that are relevant for both (1) Goal-oriented conversational agents where it's&#10;natural to partition the data by the requested application and for (2) Non-goal&#10;oriented conversational agents where the data can be partitioned using topic&#10;labels that come from predictions of a topic classifier. We obtain a relative&#10;WER improvement of 3% with a 1-pass decoding strategy and 6% in a 2-pass&#10;decoding framework, over an unadapted model. We also show up to a 15% relative&#10;improvement in recognizing named entities which is of significant value for&#10;conversational ASR systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.08579" label="2302.08579">
        <attvalues>
          <attvalue for="0" value="Adaptable End-to-End ASR Models using Replaceable Internal LMs and&#10;  Residual Softmax" />
          <attvalue for="1" value="  End-to-end (E2E) automatic speech recognition (ASR) implicitly learns the&#10;token sequence distribution of paired audio-transcript training data. However,&#10;it still suffers from domain shifts from training to testing, and domain&#10;adaptation is still challenging. To alleviate this problem, this paper designs&#10;a replaceable internal language model (RILM) method, which makes it feasible to&#10;directly replace the internal language model (LM) of E2E ASR models with a&#10;target-domain LM in the decoding stage when a domain shift is encountered.&#10;Furthermore, this paper proposes a residual softmax (R-softmax) that is&#10;designed for CTC-based E2E ASR models to adapt to the target domain without&#10;re-training during inference. For E2E ASR models trained on the LibriSpeech&#10;corpus, experiments showed that the proposed methods gave a 2.6% absolute WER&#10;reduction on the Switchboard data and a 1.0% WER reduction on the AESRC2020&#10;corpus while maintaining intra-domain ASR results.&#10;" />
          <attvalue for="2" value="&#10;End-to-end (E2E) automatic speech recognition (ASR) models simplify conventional pipeline ASR methods and directly transcribe input speech into corresponding text \cite{8068205,6638947}. Due to a large amount of labelled training data, E2E ASR models surpass pipeline methods on most public datasets \cite{9688009}.&#10;However, E2E ASR still suffers from unseen domains \cite{tsunoo22_interspeech}, and large quantities of labelled data are not always feasible to collect and can therefore be limited \cite{9746480}. Adaptation training methods can be utilised to alleviate this issue when the target domain has enough paired data \cite{tsunoo19_interspeech, 6424251}. However, text-only data from the target domain is easier to obtain in most scenarios, and it is more efficient to bias the E2E ASR systems to the target domain using such data \cite{tsunoo22_interspeech}.&#10;&#10;There are several studies exploring the use of text-only data via an external language model (LM). Shallow fusion \cite{chorowski2015attention} which linearly interpolates the E2E ASR with an external LM is straightforward and widely deployed \cite{8462682}.&#10;Several structural fusion methods like deep fusion \cite{gulcehre2015using} and cold fusion \cite{sriram18_interspeech} have been proposed, but require additional training and haven't replaced shallow fusion \cite{chorowski2015attention} as the dominant LM integration method \cite{9003790, 9383515}.&#10;Considering that E2E models learn to model the context between words and characterise the training data of the source domain, a density ratio method \cite{9003790} was proposed as an extension of shallow fusion. It subtracts the score of a source-domain LM from the log-linear combination of the E2E ASR model and target-domain LM scores \cite{9003790}. Furthermore, the estimate of the E2E ASR model's internal LM has been explored \cite{9383515, 9415039, 9746948, zeineldeen21_interspeech}.&#10;However, both the density ratio and internal LM estimation methods&#10;make the decoding process more complicated, and it is not always feasible to accurately estimate the internal LM due to domain mismatch \cite{tsunoo22_interspeech}. To simplify the decoding process based on the internal LM estimation,&#10;\cite{tsunoo22_interspeech} further proposed a residual LM that&#10;models the residual factor of external and internal LMs, but the internal LM estimate must be pre-calculated for all text data before training and the issue of inaccurate estimation still exists.&#10;Work in \cite{meng22_interspeech} explores fine-tuning the internal LM with text-only data but requires regularisation strategies to avoid the internal LM over-learning target domains.&#10;&#10;Previous methods rely on an external LM with the estimate of the internal LM used to bias the prediction of E2E ASR systems, thus improving the cross-domain ASR performance. However, incorporating the external LM demands extra computational cost and additional parameters, and accurate internal LM estimation is not always feasible \cite{tsunoo22_interspeech}. In this paper, the motivation is making the E2E ASR system itself adaptable and can be biased to unseen domains without needing an external LM or re-training. Therefore,&#10;this paper proposes a replaceable internal LM (RILM) method, through which the internal LM of the E2E ASR system can be directly replaced with a target-domain LM during the decoding stage to improve the cross-domain ASR accuracy. Furthermore, this paper designs a residual softmax (R-softmax) for CTC-based ASR models that efficiently adapts to the target domain during the inference stage. With E2E ASR models &#10;trained on the LibriSpeech corpus \cite{7178964}, experiments showed that the proposed methods greatly boosted the cross-domain ASR accuracy on the Switchboard \cite{225858} and AESRC2020 \cite{9413386} corpus while performing robustly in intra-domain scenarios.&#10;&#10;The rest of this paper is organized as follows:&#10;Sec.~\ref{sec:format} describes the proposed methods.&#10;Sec.~\ref{sec:typestyle}&#10;details the data, model, and experimental results. Conclusions are provided in Sec.~\ref{sec:print}.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Domain Adaptation, Signal Processing, Speech Recognition, Language Modeling" />
        </attvalues>
      </node>
      <node id="2206.07430" label="2206.07430">
        <attvalues>
          <attvalue for="0" value="Residual Language Model for End-to-end Speech Recognition" />
          <attvalue for="1" value="  End-to-end automatic speech recognition suffers from adaptation to unknown&#10;target domain speech despite being trained with a large amount of paired&#10;audio--text data. Recent studies estimate a linguistic bias of the model as the&#10;internal language model (LM). To effectively adapt to the target domain, the&#10;internal LM is subtracted from the posterior during inference and fused with an&#10;external target-domain LM. However, this fusion complicates the inference and&#10;the estimation of the internal LM may not always be accurate. In this paper, we&#10;propose a simple external LM fusion method for domain adaptation, which&#10;considers the internal LM estimation in its training. We directly model the&#10;residual factor of the external and internal LMs, namely the residual LM. To&#10;stably train the residual LM, we propose smoothing the estimated internal LM&#10;and optimizing it with a combination of cross-entropy and mean-squared-error&#10;losses, which consider the statistical behaviors of the internal LM in the&#10;target domain data. We experimentally confirmed that the proposed residual LM&#10;performs better than the internal LM estimation in most of the cross-domain and&#10;intra-domain scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.03655" label="2201.03655">
        <attvalues>
          <attvalue for="0" value="A Likelihood Ratio based Domain Adaptation Method for E2E Models" />
          <attvalue for="1" value="  End-to-end (E2E) automatic speech recognition models like Recurrent Neural&#10;Networks Transducer (RNN-T) are becoming a popular choice for streaming ASR&#10;applications like voice assistants. While E2E models are very effective at&#10;learning representation of the training data they are trained on, their&#10;accuracy on unseen domains remains a challenging problem. Additionally, these&#10;models require paired audio and text training data, are computationally&#10;expensive and are difficult to adapt towards the fast evolving nature of&#10;conversational speech. In this work, we explore a contextual biasing approach&#10;using likelihood-ratio that leverages text data sources to adapt RNN-T model to&#10;new domains and entities. We show that this method is effective in improving&#10;rare words recognition, and results in a relative improvement of 10% in 1-best&#10;word error rate (WER) and 10% in n-best Oracle WER (n=8) on multiple&#10;out-of-domain datasets without any degradation on a general dataset. We also&#10;show that complementing the contextual biasing adaptation with adaptation of a&#10;second-pass rescoring model gives additive WER improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.07149" label="1905.07149">
        <attvalues>
          <attvalue for="0" value="End-to-end Adaptation with Backpropagation through WFST for On-device&#10;  Speech Recognition System" />
          <attvalue for="1" value="  An on-device DNN-HMM speech recognition system efficiently works with a&#10;limited vocabulary in the presence of a variety of predictable noise. In such a&#10;case, vocabulary and environment adaptation is highly effective. In this paper,&#10;we propose a novel method of end-to-end (E2E) adaptation, which adjusts not&#10;only an acoustic model (AM) but also a weighted finite-state transducer (WFST).&#10;We convert a pretrained WFST to a trainable neural network and adapt the system&#10;to target environments/vocabulary by E2E joint training with an AM. We&#10;replicate Viterbi decoding with forward--backward neural network computation,&#10;which is similar to recurrent neural networks (RNNs). By pooling output score&#10;sequences, a vocabulary posterior for each utterance is obtained and used for&#10;discriminative loss computation. Experiments using 2--10 hours of&#10;English/Japanese adaptation datasets indicate that the fine-tuning of only&#10;WFSTs and that of only AMs are both comparable to a state-of-the-art adaptation&#10;method, and E2E joint training of the two components achieves the best&#10;recognition performance. We also adapt each language system to the other&#10;language using the adaptation data, and the results show that the proposed&#10;method also works well for language adaptations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.10233" label="2102.10233">
        <attvalues>
          <attvalue for="0" value="The Accented English Speech Recognition Challenge 2020: Open Datasets,&#10;  Tracks, Baselines, Results and Methods" />
          <attvalue for="1" value="  The variety of accents has posed a big challenge to speech recognition. The&#10;Accented English Speech Recognition Challenge (AESRC2020) is designed for&#10;providing a common testbed and promoting accent-related research. Two tracks&#10;are set in the challenge -- English accent recognition (track 1) and accented&#10;English speech recognition (track 2). A set of 160 hours of accented English&#10;speech collected from 8 countries is released with labels as the training set.&#10;Another 20 hours of speech without labels is later released as the test set,&#10;including two unseen accents from another two countries used to test the model&#10;generalization ability in track 2. We also provide baseline systems for the&#10;participants. This paper first reviews the released dataset, track setups,&#10;baselines and then summarizes the challenge results and major techniques used&#10;in the submissions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.02310" label="2311.02310">
        <attvalues>
          <attvalue for="0" value="Narrowing the Gap between Zero- and Few-shot Machine Translation by&#10;  Matching Styles" />
          <attvalue for="1" value="  Large language models trained primarily in a monolingual setting have&#10;demonstrated their ability to generalize to machine translation using zero- and&#10;few-shot examples with in-context learning. However, even though zero-shot&#10;translations are relatively good, there remains a discernible gap comparing&#10;their performance with the few-shot setting. In this paper, we investigate the&#10;factors contributing to this gap and find that this gap can largely be closed&#10;(for about 70%) by matching the writing styles of the target corpus.&#10;Additionally, we explore potential approaches to enhance zero-shot baselines&#10;without the need for parallel demonstration examples, providing valuable&#10;insights into how these methods contribute to improving translation metrics.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancements in large language models (LLMs) have revolutionized Natural Language Processing field as such models \cite[inter alia]{openai2023gpt4, instrcutgpt, chowdhery2022palm, touvron2023llama} can easily adapt to a new task through prompt (in-context) learning where the task instruction and demonstrations (examples to guide LLMs on the task) are provided to the model. Such capability opens up new opportunities for machine translation, which is traditionally trained/fine-tuned on large amounts of parallel corpus \cite[inter alia]{brown-etal-1993-mathematics, bahdanau2016neural, transformer, nllbteam2022language}. Recent work~\cite{vilar2022prompting, zhang2023prompting, jiao2023chatgpt, hendy2023good} has found that prompt-based methods perform well on language models trained primarily on monolingual data, rivaling state-of-the-art systems trained specifically for machine translation tasks on benchmark datasets.&#10;&#10;Comparing zero-and few-shot outputs (\ref{sec::gap}), we observe a huge gap in terms of BLEU~\cite{papineni-etal-2002-bleu} score, despite the acceptable quality of zero-shot translations. As shown in \ref{fig::teaser}, the zero-shot translation already conveys the meaning but the few-shot translation is better matched with the target (thus obtaining a better BLEU score). &#10;&#10;We present qualitative and quantitative analysis on zero- and few-shot translation to understand their quality difference. Our evaluation (\ref{sec::gap}) reveals that the performance gap stems mostly from writing styles rather than semantics. This finding motivates us to quantify the degree of style match between translations and references (\ref{sec::analysis}) and develop a data-efficient style-learning prompting strategy (\ref{sec::style}). Compared to few-shot translation, our prompting strategy relies solely on retrievals from in-domain target corpora and demonstrates effectiveness in closing approximately $70\%$ of the gap between zero-and few-shot translation. We provide an example in \ref{fig::teaser} to illustrate the effectiveness of the proposed style-matching approach.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Zero-Shot Learning, Language Model Translation, Artificial Intelligence, Machine Translation Optimization" />
        </attvalues>
      </node>
      <node id="2307.01878" label="2307.01878">
        <attvalues>
          <attvalue for="0" value="KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation" />
          <attvalue for="1" value="  In text classification tasks, fine tuning pretrained language models like&#10;BERT and GPT-3 yields competitive accuracy; however, both methods require&#10;pretraining on large text datasets. In contrast, general topic modeling methods&#10;possess the advantage of analyzing documents to extract meaningful patterns of&#10;words without the need of pretraining. To leverage topic modeling's&#10;unsupervised insights extraction on text classification tasks, we develop the&#10;Knowledge Distillation Semi-supervised Topic Modeling (KDSTM). KDSTM requires&#10;no pretrained embeddings, few labeled documents and is efficient to train,&#10;making it ideal under resource constrained settings. Across a variety of&#10;datasets, our method outperforms existing supervised topic modeling methods in&#10;classification accuracy, robustness and efficiency and achieves similar&#10;performance compare to state of the art weakly supervised text classification&#10;methods.&#10;" />
          <attvalue for="2" value="&#10;The current state-of-the-art language modeling methods often require transfer learning \cite{brown2020language}, large amount of labels \cite{yang2019xlnet} and pretrained embeddings \cite{cao2020multilingual}. Consequently, they are difficult to apply in low resource settings, where many endangered languages\cite{austin2011cambridge} lack both pre trained language models and sufficient labeled documents. &#10;For semi-supervised method \cite{meng2018weaklysupervised} tailored to limited label scenario, it is time consuming to both tune and train. &#10;&#10;Topic modeling is an unsupervised method for discovering latent structure within the training document sets and achieves great empirical performance in many fields\cite{blei2009nested}, including finance \cite{ healthcare \cite{DBLP:journals/corr/abs-1711-10960}, education \cite{zhao2020targeted}, marketing \cite{Reisenbichler2019} and social science \cite{762586}. \cite{jelodar2018latent} provides a survey on the applications of topic modeling. &#10;&#10;Latent Dirichlet Allocation (LDA) \cite{blei2003latent} is the most fundamental topic modeling approach based on Bayesian inference on Markov chain Monte Carlo (MCMC) and variational inference; however, it is hard to be expressive or capture large vocabularies. Neural topic model (NTM) \cite{miao2018discovering} leverages auto-encoding \cite{kingma2014semi} framework to approximate intractable distributions over latent variables. Recently, embedded topic model (ETM) \cite{dieng2020topic} uses word embedding during the reconstruction process to make topic more coherent and reduce the influence of stop words. The goal of unsupervised topic modeling methods \cite{blei2003latent,teh2006hierarchical,miao2018discovering,gemp2019weakly, xu-etal-2023-vontss} is to maximize the probability of the observed data, resulting in the tendency to identify obvious and superficial aspects of a corpus. To incorporate users' domain knowledge of documents into the model, supervised modeling \cite{blei2010supervised, JMLR:v13:zhu12a, pmlr-v108-wang20c} has been studied. However, supervised methods do not perform well when the labeled set is small.&#10;&#10;In this work, we propose knowledge distillation semi-supervised topic modeling (KDSTM), which only requires a few labeled documents for each topic as input. KDSTM utilizes knowledge distillation and optimal transport to guide topic extraction with seed documents. &#10;It achieves state of the art results when benchmarking with supervised topic modeling and weakly supervised text classification methods. &#10;&#10;Advantages of KDSTM are summarized as follows:&#10;\begin{itemize}[leftmargin=0.12in]&#10; \item KDSTM is a novel architecture which incorporates knowledge distillation and optimal transport into the neural topic modeling framework.&#10; \item KDSTM consistently achieves better topics classification performance on different datasets when compared to supervised topic modeling methods or weakly supervised text classification methods. &#10; \item KDSTM only requires a limited number of labeled documents as input, making it more practical in low resource settings.&#10; \item KDSTM does not rely on any transfer learning or pre trained language models. The embedding is trained on the dataset, making it suitable for less common/endangered languages. &#10; \item KDSTM is efficient to train and fine-tune compared to existing methods. This makes it suitable to be trained and run inference on resource constrained devices.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Classification, Machine Learning, Computer Science, Linguistics, Language Models, Artificial Intelligence, Natural Language Processing, Statistics, Topic Modeling" />
        </attvalues>
      </node>
      <node id="1812.11270" label="1812.11270">
        <attvalues>
          <attvalue for="0" value="Weakly-Supervised Hierarchical Text Classification" />
          <attvalue for="1" value="  Hierarchical text classification, which aims to classify text documents into&#10;a given hierarchy, is an important task in many real-world applications.&#10;Recently, deep neural models are gaining increasing popularity for text&#10;classification due to their expressive power and minimum requirement for&#10;feature engineering. However, applying deep neural networks for hierarchical&#10;text classification remains challenging, because they heavily rely on a large&#10;amount of training data and meanwhile cannot easily determine appropriate&#10;levels of documents in the hierarchical setting. In this paper, we propose a&#10;weakly-supervised neural method for hierarchical text classification. Our&#10;method does not require a large amount of training data but requires only&#10;easy-to-provide weak supervision signals such as a few class-related documents&#10;or keywords. Our method effectively leverages such weak supervision signals to&#10;generate pseudo documents for model pre-training, and then performs&#10;self-training on real unlabeled data to iteratively refine the model. During&#10;the training process, our model features a hierarchical neural structure, which&#10;mimics the given hierarchy and is capable of determining the proper levels for&#10;documents with a blocking mechanism. Experiments on three datasets from&#10;different domains demonstrate the efficacy of our method compared with a&#10;comprehensive set of baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="0710.0845" label="0710.0845">
        <attvalues>
          <attvalue for="0" value="The nested Chinese restaurant process and Bayesian nonparametric&#10;  inference of topic hierarchies" />
          <attvalue for="1" value="  We present the nested Chinese restaurant process (nCRP), a stochastic process&#10;which assigns probability distributions to infinitely-deep,&#10;infinitely-branching trees. We show how this stochastic process can be used as&#10;a prior distribution in a Bayesian nonparametric model of document collections.&#10;Specifically, we present an application to information retrieval in which&#10;documents are modeled as paths down a random tree, and the preferential&#10;attachment dynamics of the nCRP leads to clustering of documents according to&#10;sharing of topics at multiple levels of abstraction. Given a corpus of&#10;documents, a posterior inference algorithm finds an approximation to a&#10;posterior distribution over trees, topics and allocations of words to levels of&#10;the tree. We demonstrate this algorithm on collections of scientific abstracts&#10;from several journals. This model exemplifies a recent trend in statistical&#10;machine learning--the use of Bayesian nonparametric methods to infer&#10;distributions on flexible data structures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.10960" label="1711.10960">
        <attvalues>
          <attvalue for="0" value="Identifying Patterns of Associated-Conditions through Topic Models of&#10;  Electronic Medical Records" />
          <attvalue for="1" value="  Multiple adverse health conditions co-occurring in a patient are typically&#10;associated with poor prognosis and increased office or hospital visits.&#10;Developing methods to identify patterns of co-occurring conditions can assist&#10;in diagnosis. Thus identifying patterns of associations among co-occurring&#10;conditions is of growing interest. In this paper, we report preliminary results&#10;from a data-driven study, in which we apply a machine learning method, namely,&#10;topic modeling, to electronic medical records, aiming to identify patterns of&#10;associated conditions. Specifically, we use the well established latent&#10;dirichlet allocation, a method based on the idea that documents can be modeled&#10;as a mixture of latent topics, where each topic is a distribution over words.&#10;In our study, we adapt the LDA model to identify latent topics in patients'&#10;EMRs. We evaluate the performance of our method both qualitatively, and show&#10;that the obtained topics indeed align well with distinct medical phenomena&#10;characterized by co-occurring conditions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.00359" label="1706.00359">
        <attvalues>
          <attvalue for="0" value="Discovering Discrete Latent Topics with Neural Variational Inference" />
          <attvalue for="1" value="  Topic models have been widely explored as probabilistic generative models of&#10;documents. Traditional inference methods have sought closed-form derivations&#10;for updating the models, however as the expressiveness of these models grows,&#10;so does the difficulty of performing fast and accurate inference over their&#10;parameters. This paper presents alternative neural approaches to topic&#10;modelling by providing parameterisable distributions over topics which permit&#10;training by backpropagation in the framework of neural variational inference.&#10;In addition, with the help of a stick-breaking construction, we propose a&#10;recurrent network that is able to discover a notionally unbounded number of&#10;topics, analogous to Bayesian non-parametric topic models. Experimental results&#10;on the MXM Song Lyrics, 20NewsGroups and Reuters News datasets demonstrate the&#10;effectiveness and efficiency of these neural topic models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1003.0783" label="1003.0783">
        <attvalues>
          <attvalue for="0" value="Supervised Topic Models" />
          <attvalue for="1" value="  We introduce supervised latent Dirichlet allocation (sLDA), a statistical&#10;model of labelled documents. The model accommodates a variety of response&#10;types. We derive an approximate maximum-likelihood procedure for parameter&#10;estimation, which relies on variational methods to handle intractable posterior&#10;expectations. Prediction problems motivate this research: we use the fitted&#10;model to predict response values for new documents. We test sLDA on two&#10;real-world problems: movie ratings predicted from reviews, and the political&#10;tone of amendments in the U.S. Senate based on the amendment text. We&#10;illustrate the benefits of sLDA versus modern regularized regression, as well&#10;as versus an unsupervised LDA analysis followed by a separate regression.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.07068" label="2209.07068">
        <attvalues>
          <attvalue for="0" value="uChecker: Masked Pretrained Language Models as Unsupervised Chinese&#10;  Spelling Checkers" />
          <attvalue for="1" value="  The task of Chinese Spelling Check (CSC) is aiming to detect and correct&#10;spelling errors that can be found in the text. While manually annotating a&#10;high-quality dataset is expensive and time-consuming, thus the scale of the&#10;training dataset is usually very small (e.g., SIGHAN15 only contains 2339&#10;samples for training), therefore supervised-learning based models usually&#10;suffer the data sparsity limitation and over-fitting issue, especially in the&#10;era of big language models. In this paper, we are dedicated to investigating&#10;the \textbf{unsupervised} paradigm to address the CSC problem and we propose a&#10;framework named \textbf{uChecker} to conduct unsupervised spelling error&#10;detection and correction. Masked pretrained language models such as BERT are&#10;introduced as the backbone model considering their powerful language diagnosis&#10;capability. Benefiting from the various and flexible MASKing operations, we&#10;propose a Confusionset-guided masking strategy to fine-train the masked&#10;language model to further improve the performance of unsupervised detection and&#10;correction. Experimental results on standard datasets demonstrate the&#10;effectiveness of our proposed model uChecker in terms of character-level and&#10;sentence-level Accuracy, Precision, Recall, and F1-Measure on tasks of spelling&#10;error detection and correction respectively.&#10;" />
          <attvalue for="2" value="&#10;&#10;Chinese Spelling Check (CSC) is a crucial and essential task in the area of natural language processing. It aims to detect and correct spelling errors in the Chinese text~\cite{chang1995new,DBLP:journals/corr/abs-2005-06600}. &#10;&#09;Generally, sequence translation~\cite{DBLP:conf/emnlp/WangSLHZ18,DBLP:journals/corr/abs-1807-01270,DBLP:conf/acl/WangTZ19,DBLP:conf/ijcnlp/WangKKK20,DBLP:conf/acl/KanekoMKSI20} and sequence tagging~\cite{DBLP:conf/bea/OmelianchukACS20,liang-etal-2020-bert,DBLP:conf/emnlp/MallinsonSMG20,DBLP:conf/acl/ParnowLZ21} are the two most typical technical paradigms to tackle the problem. Benefiting from the development of pretraining techniques, many researchers fine-tune the pretrained language models such as BERT~\cite{DBLP:conf/naacl/DevlinCLT19} on the task of CSC and obtain encouraging performance~\cite{DBLP:conf/naacl/ZhaoWSJL19,DBLP:conf/aclnut/HongYHLL19,DBLP:conf/acl/ZhangHLL20,DBLP:conf/acl/LiuYYZW20,DBLP:conf/acl/LiZZH20,DBLP:conf/acl/HuangLJZCWX20,DBLP:conf/acl/GuoNWZX21,DBLP:conf/acl/ZhangPZWHSWW21,DBLP:conf/acl/Li020,dai-etal-2022-whole}. Meanwhile, it should be emphasized that almost all of the above mentioned models are trained via the supervised learning paradigm. &#10;&#10;However, during the investigating stage about those newly typical state-of-the-art models, we observe some spiny and serious phenomenons: (1) Occasionally those models may generate some special over-correcting results. As shown in Figure~\ref{fig:front}, operation path I is the regular spelling error detection and correction path, while operation path II is also observable in the inference stage where the models can detect the errors correctly but rectify them using some other error tokens in the correction stage. (2) The spelling error detection and correction performance will drop dramatically when those models did not see the spelling error cases in the training dataset or the text are from different genres and domains. This issue tells us that the generalization capability of those models are limited and need to be enhanced.&#10;&#10;Then what are the causes of these phenomenons? Since some of the models are already strong enough (which are constructed based on big pretrained models), then we shift our eyes to the data perspective. In real practical scenarios, natural human-labeled spelling error corpus are difficult and expensive to obtain. Although some works such as \cite{DBLP:conf/emnlp/WangSLHZ18} employ OCR and ASR based techniques to automatically synthetic the paired samples by replacing the correct tokens using visually &#10;&#09;or phonologically similar characters, obviously, the constructed data is unrealistic and far from the real and objective scenarios. Therefore, actually, the scale of the typical corpus for the task of Chinese spelling check is very small. Considering that almost all the research works have used SIGHAN &#10;&#09;series datasets \cite{DBLP:conf/acl-sighan/TsengLCC15} to train and evaluate their algorithms, we conduct counting on those three corpora, and the statistics results are shown in Table~\ref{tab:datasets}. From the results we can observe that there are only 2k$\sim$3k sentences with spelling errors in the training dataset and really far from the practical requirements. &#10;&#10;Thus, sticking to train the supervised learning models based on those scale-limited resources might not be a wise direction. Therefore, in this paper, we are dedicated to exploring unsupervised frameworks to conduct Chinese spelling error detection and correction. Fortunately, masked pretrained language models such as BERT \cite{DBLP:conf/naacl/DevlinCLT19}, RoBERTa~\cite{DBLP:journals/corr/abs-1907-11692}, ELECTRA~\cite{DBLP:conf/iclr/ClarkLLM20}, etc. can satisfy the needs of detecting and correcting spelling errors in an unsupervised manner. First, the masked training strategy is naturally a convenient and perfect shortcut for us to conduct token-grained detection and correction. For example, we can mask any token and predict it based on the bi-directional context to see if the current token is appropriate or not. Second, the pretrained language models are usually trained using large-scale corpora, thus the language diagnosis capability is very strong. Intuitively, these models can also guarantee the generalization capability considering the corpora may contain text from a wide range of domains and genres.&#10;&#10;Therefore, based on the masked pretrained language models, we propose a framework named uChecker to conduct unsupervised spelling error detection and correction respectively. uChecker is a two-stage framework and it will detect the text token-by-token first and then correct the abnormal tokens. Models such as BERT are introduced as the backbone model. Inspired by the previous works~\cite{DBLP:conf/acl/WangTZ19,DBLP:conf/acl/LiuYYZW20}, benefiting from the various and flexible masking operations, we also introduce a confusionset-guided masking strategy to fine-train the masked language model to further improve the performance of unsupervised detection and correction. Though uChecker is a two-stage framework, we design an elegant method to let the information pass BERT only once to guarantee the time efficiency. Moreover, interestingly, in unsupervised settings, we experimentally find the performance of error detection is crucial to the global and general performance. It means that the correction capability of the pretrained language models are strong enough, then the key-point is how to improve the performance of detection. Therefore, in uChecker, we also design several algorithms to improve the performance of seplling error detection. \cite{DBLP:conf/emnlp/YasunagaLL21} employ GPT2-like models to conduct unsupervised English grammatical error correction which also verifies the feasible of our direction. &#10;&#10;In summary, our contributions are as follows:&#10;&#09;\begin{itemize}[topsep=0pt]&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Error Detection Correction, Linguistics, Unsupervised Learning, Artificial Intelligence, Mathematics, Language Modeling" />
        </attvalues>
      </node>
      <node id="1807.01270" label="1807.01270">
        <attvalues>
          <attvalue for="0" value="Reaching Human-level Performance in Automatic Grammatical Error&#10;  Correction: An Empirical Study" />
          <attvalue for="1" value="  Neural sequence-to-sequence (seq2seq) approaches have proven to be successful&#10;in grammatical error correction (GEC). Based on the seq2seq framework, we&#10;propose a novel fluency boost learning and inference mechanism. Fluency&#10;boosting learning generates diverse error-corrected sentence pairs during&#10;training, enabling the error correction model to learn how to improve a&#10;sentence's fluency from more instances, while fluency boosting inference allows&#10;the model to correct a sentence incrementally with multiple inference steps.&#10;Combining fluency boost learning and inference with convolutional seq2seq&#10;models, our approach achieves the state-of-the-art performance: 75.72 (F_{0.5})&#10;on CoNLL-2014 10 annotation dataset and 62.42 (GLEU) on JFLEG test set&#10;respectively, becoming the first GEC system that reaches human-level&#10;performance (72.58 for CoNLL and 62.37 for JFLEG) on both of the benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.02093" label="2011.02093">
        <attvalues>
          <attvalue for="0" value="Chinese Grammatical Correction Using BERT-based Pre-trained Model" />
          <attvalue for="1" value="  In recent years, pre-trained models have been extensively studied, and&#10;several downstream tasks have benefited from their utilization. In this study,&#10;we verify the effectiveness of two methods that incorporate a BERT-based&#10;pre-trained model developed by Cui et al. (2020) into an encoder-decoder model&#10;on Chinese grammatical error correction tasks. We also analyze the error type&#10;and conclude that sentence-level errors are yet to be addressed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.10687" label="2003.10687">
        <attvalues>
          <attvalue for="0" value="Felix: Flexible Text Editing Through Tagging and Insertion" />
          <attvalue for="1" value="  We present Felix --- a flexible text-editing approach for generation,&#10;designed to derive the maximum benefit from the ideas of decoding with&#10;bi-directional contexts and self-supervised pre-training. In contrast to&#10;conventional sequence-to-sequence (seq2seq) models, Felix is efficient in&#10;low-resource settings and fast at inference time, while being capable of&#10;modeling flexible input-output transformations. We achieve this by decomposing&#10;the text-editing task into two sub-tasks: tagging to decide on the subset of&#10;input tokens and their order in the output text and insertion to in-fill the&#10;missing tokens in the output not present in the input. The tagging model&#10;employs a novel Pointer mechanism, while the insertion model is based on a&#10;Masked Language Model. Both of these models are chosen to be non-autoregressive&#10;to guarantee faster inference. Felix performs favourably when compared to&#10;recent text-editing methods and strong seq2seq baselines when evaluated on four&#10;NLG tasks: Sentence Fusion, Machine Translation Automatic Post-Editing,&#10;Summarization, and Text Simplification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Seq2seq models~\cite{sutskever2014sequence} have been applied to many text generation tasks that can be cast as monolingual translation, but they suffer from well-known drawbacks~\cite{wiseman2018learning}: they require large amounts of training data, and their outputs are difficult to control. Whenever input and output sequences have a large overlap, it is reasonable to cast the problem as a text editing task, rather than full-fledged sequence to sequence generation. \cite{ribeiro2018local} argued that the general problem of string transduction can be reduced to sequence labeling. Their approach applied only to character deletion and insertion and was based on simple patterns. LaserTagger~\cite{LaserTagger} is a general approach that has been shown to perform well on a number of text editing tasks, but it has two limitations: it does not allow for arbitrary reordering of the input tokens; and insertions are restricted to a fixed phrase vocabulary that is derived from the training data. Similarly, EditNTS~\cite{EditNTS} and PIE~\cite{awasthi2019parallel} are two other text-editing models that predict tokens to keep, delete, and add, which are developed specifically for the tasks of text simplification and grammatical error correction, respectively. In contrast to the aforementioned models, \GT allows more flexible rewriting, using a pointer network that points into the source to decide which tokens should be preserved in the output and in which order.&#10;&#10;Pointer networks have been previously proposed as a way to copy parts of&#10;the input in hybrid sequence-to-sequence models. \cite{gulcehre2016pointing} and&#10;\cite{nallapati2016abstractive} trained a pointer network to specifically deal with out-of-vocabulary&#10;words or named entities. \cite{see2017get} hybrid approach learns when to use the pointer&#10;to copy parts of the input. \cite{chen2018fast} proposed a summarization model that first&#10;selects salient sentences and then rewrites them abstractively, using a pointer mechanism to&#10;directly copy some out-of-vocabulary words. These methods still typically require large amounts of training data and they are inherently slow at inference time due to autoregressive decoding.&#10;&#10;Previous approaches have proposed&#10;alternatives to autoregressive decoding~\cite{gu2018nonautoregressive,lee2018deterministic,KERMIT,wang2019bert}.&#10;Instead of the left-to-right autoregressive decoding,&#10;Insertion Transformer~\cite{InsertionTransformer} and BLM~\cite{shen2020blank} generate the output sequence&#10;through insertion operations, whereas Levenshtein Transformer (\LevT)~\cite{LevenshteinTransformer} additionally incorporates a deletion operation.&#10;&#10;These methods produce the output iteratively, while \GT requires only two steps: \tagging and \insertion.&#10;&#10;The differences between the proposed model, \GT, its ablated variants, and a selection of related works is summarized in Table~\ref{tbl:comparison}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2105.14209" label="2105.14209">
        <attvalues>
          <attvalue for="0" value="Grammatical Error Correction as GAN-like Sequence Labeling" />
          <attvalue for="1" value="  In Grammatical Error Correction (GEC), sequence labeling models enjoy fast&#10;inference compared to sequence-to-sequence models; however, inference in&#10;sequence labeling GEC models is an iterative process, as sentences are passed&#10;to the model for multiple rounds of correction, which exposes the model to&#10;sentences with progressively fewer errors at each round. Traditional GEC models&#10;learn from sentences with fixed error rates. Coupling this with the iterative&#10;correction process causes a mismatch between training and inference that&#10;affects final performance. In order to address this mismatch, we propose a&#10;GAN-like sequence labeling model, which consists of a grammatical error&#10;detector as a discriminator and a grammatical error labeler with Gumbel-Softmax&#10;sampling as a generator. By sampling from real error distributions, our errors&#10;are more genuine compared to traditional synthesized GEC errors, thus&#10;alleviating the aforementioned mismatch and allowing for better training. Our&#10;results on several evaluation benchmarks demonstrate that our proposed approach&#10;is effective and improves the previous state-of-the-art baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.07421" label="2005.07421">
        <attvalues>
          <attvalue for="0" value="Spelling Error Correction with Soft-Masked BERT" />
          <attvalue for="1" value="  Spelling error correction is an important yet challenging task because a&#10;satisfactory solution of it essentially needs human-level language&#10;understanding ability. Without loss of generality we consider Chinese spelling&#10;error correction (CSC) in this paper. A state-of-the-art method for the task&#10;selects a character from a list of candidates for correction (including&#10;non-correction) at each position of the sentence on the basis of BERT, the&#10;language representation model. The accuracy of the method can be sub-optimal,&#10;however, because BERT does not have sufficient capability to detect whether&#10;there is an error at each position, apparently due to the way of pre-training&#10;it using mask language modeling. In this work, we propose a novel neural&#10;architecture to address the aforementioned issue, which consists of a network&#10;for error detection and a network for error correction based on BERT, with the&#10;former being connected to the latter with what we call soft-masking technique.&#10;Our method of using `Soft-Masked BERT' is general, and it may be employed in&#10;other language detection-correction problems. Experimental results on two&#10;datasets demonstrate that the performance of our proposed method is&#10;significantly better than the baselines including the one solely based on BERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.14813" label="2105.14813">
        <attvalues>
          <attvalue for="0" value="Exploration and Exploitation: Two Ways to Improve Chinese Spelling&#10;  Correction Models" />
          <attvalue for="1" value="  A sequence-to-sequence learning with neural networks has empirically proven&#10;to be an effective framework for Chinese Spelling Correction (CSC), which takes&#10;a sentence with some spelling errors as input and outputs the corrected one.&#10;However, CSC models may fail to correct spelling errors covered by the&#10;confusion sets, and also will encounter unseen ones. We propose a method, which&#10;continually identifies the weak spots of a model to generate more valuable&#10;training instances, and apply a task-specific pre-training strategy to enhance&#10;the model. The generated adversarial examples are gradually added to the&#10;training set. Experimental results show that such an adversarial training&#10;method combined with the pretraining strategy can improve both the&#10;generalization and robustness of multiple CSC models across three different&#10;datasets, achieving stateof-the-art performance for CSC task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.01609" label="2106.01609">
        <attvalues>
          <attvalue for="0" value="Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese&#10;  Grammatical Error Correction" />
          <attvalue for="1" value="  We investigate the problem of Chinese Grammatical Error Correction (CGEC) and&#10;present a new framework named Tail-to-Tail (\textbf{TtT}) non-autoregressive&#10;sequence prediction to address the deep issues hidden in CGEC. Considering that&#10;most tokens are correct and can be conveyed directly from source to target, and&#10;the error positions can be estimated and corrected based on the bidirectional&#10;context information, thus we employ a BERT-initialized Transformer Encoder as&#10;the backbone model to conduct information modeling and conveying. Considering&#10;that only relying on the same position substitution cannot handle the&#10;variable-length correction cases, various operations such substitution,&#10;deletion, insertion, and local paraphrasing are required jointly. Therefore, a&#10;Conditional Random Fields (CRF) layer is stacked on the up tail to conduct&#10;non-autoregressive sequence prediction by modeling the token dependencies.&#10;Since most tokens are correct and easily to be predicted/conveyed to the&#10;target, then the models may suffer from a severe class imbalance issue. To&#10;alleviate this problem, focal loss penalty strategies are integrated into the&#10;loss functions. Moreover, besides the typical fix-length error correction&#10;datasets, we also construct a variable-length corpus to conduct experiments.&#10;Experimental results on standard datasets, especially on the variable-length&#10;datasets, demonstrate the effectiveness of TtT in terms of sentence-level&#10;Accuracy, Precision, Recall, and F1-Measure on tasks of error Detection and&#10;Correction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.06822" label="2109.06822">
        <attvalues>
          <attvalue for="0" value="LM-Critic: Language Models for Unsupervised Grammatical Error Correction" />
          <attvalue for="1" value="  Training a model for grammatical error correction (GEC) requires a set of&#10;labeled ungrammatical / grammatical sentence pairs, but manually annotating&#10;such pairs can be expensive. Recently, the Break-It-Fix-It (BIFI) framework has&#10;demonstrated strong results on learning to repair a broken program without any&#10;labeled examples, but this relies on a perfect critic (e.g., a compiler) that&#10;returns whether an example is valid or not, which does not exist for the GEC&#10;task. In this work, we show how to leverage a pretrained language model (LM) in&#10;defining an LM-Critic, which judges a sentence to be grammatical if the LM&#10;assigns it a higher probability than its local perturbations. We apply this&#10;LM-Critic and BIFI along with a large set of unlabeled sentences to bootstrap&#10;realistic ungrammatical / grammatical pairs for training a corrector. We&#10;evaluate our approach on GEC datasets across multiple domains (CoNLL-2014,&#10;BEA-2019, GMEG-wiki and GMEG-yahoo) and show that it outperforms existing&#10;methods in both the unsupervised setting (+7.7 F0.5) and the supervised setting&#10;(+0.5 F0.5).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.09898" label="2305.09898">
        <attvalues>
          <attvalue for="0" value="Balancing Lexical and Semantic Quality in Abstractive Summarization" />
          <attvalue for="1" value="  An important problem of the sequence-to-sequence neural models widely used in&#10;abstractive summarization is exposure bias. To alleviate this problem,&#10;re-ranking systems have been applied in recent years. Despite some performance&#10;improvements, this approach remains underexplored. Previous works have mostly&#10;specified the rank through the ROUGE score and aligned candidate summaries, but&#10;there can be quite a large gap between the lexical overlap metric and semantic&#10;similarity. In this paper, we propose a novel training method in which a&#10;re-ranker balances the lexical and semantic quality. We further newly define&#10;false positives in ranking and present a strategy to reduce their influence.&#10;Experiments on the CNN/DailyMail and XSum datasets show that our method can&#10;estimate the meaning of summaries without seriously degrading the lexical&#10;aspect. More specifically, it achieves an 89.67 BERTScore on the CNN/DailyMail&#10;dataset, reaching new state-of-the-art performance. Our code is publicly&#10;available at https://github.com/jeewoo1025/BalSum.&#10;" />
          <attvalue for="2" value="&#10;The performance of sequence-to-sequence (Seq2Seq) neural models for abstractive summarization \cite{lewis-etal-2020-bart, nallapati-etal-2016-abstractive, see-etal-2017-get, zhang2020pegasus} has improved significantly. The dominant training paradigm of Seq2Seq models is that of Maximum Likelihood Estimation (MLE), maximizing the likelihood of each output given the gold history of target sequences during training. However, since the models generate the sequence in an auto-regressive manner at inference, the errors made in the previous steps accumulate in the next step thereby affecting the entire sequence. This phenomenon is known as exposure bias \cite{NIPS2015_e995f98d, ICLR2016_Aurelio}. To mitigate this problem, re-ranking systems \cite{liu-etal-2021-refsum, liu-liu-2021-simcls, liu-etal-2022-brio, ravaut-etal-2022-summareranker} have recently been introduced to generate a more appropriate summary. &#10;&#10;There are two training objectives for applying re-ranking to abstractive summarization: contrastive learning and multi-task learning. The contrastive learning-based approaches deploy margin-based losses. SimCLS \cite{liu-liu-2021-simcls} and BRIO-Ctr \cite{liu-etal-2022-brio} train a large pre-trained model, such as RoBERTa \cite{2019RoBERTa} and BART \cite{lewis-etal-2020-bart}, to align the candidate summaries according to the quality. The authors use the ROUGE \cite{lin-2004-rouge} score as a quality measurement. The multi-task learning-based approaches combine at least two losses that perform different roles. SummaReranker \cite{ravaut-etal-2022-summareranker} minimizes the average over the binary cross-entropy losses optimized for each evaluation metric. In addition, BRIO-Mul \cite{liu-etal-2022-brio} demonstrates that the combination of the contrastive and cross-entropy loss works complementarily and has better performance. &#10;&#10;In this paper, we analyze the three main drawbacks of existing re-ranking approaches. First, we argue that current methods focus excessively on ranking summaries in terms of lexical overlap. Inspired by \cite{zhong-etal-2020-extractive}, we conduct a preliminary study, by sorting candidate summaries in descending order based on the ROUGE score and then defining $z$ as the rank index of the highest BERTScore summary. As demonstrated in Fig. \ref{fig:1}, we can observe that there is a large gap between lexical overlap and semantic similarity. In a majority (52\%) of cases $z&gt;1$. Second, despite more than half of the candidates with the same ROUGE score, previous studies do not accurately reflect quality measurements as they are trained with different ranks even if they have equal scores (Appendix \ref{sec:appendix_identical_score}). Lastly, for the first time, we find summaries with high lexical overlap but low semantic similarity as false positives (Appendix \ref{sec:appendix_example}). They can be noises during training phrase, which are not considered substantially in the prior works. &#10;&#10;To address these issues, we propose a novel training method in which a re-ranker balances lexical and semantic quality. Based on a two-stage framework, our model, named BalSum, is trained on multi-task learning. We directly reflect the ROUGE score difference on a ranking loss to preserve the lexical quality as much as possible. Then, we use a contrastive loss with instance weighting to identify summaries whose meanings are close to the document. Specifically, we define novel false positives (semantic mistakes) and present a strategy to reduce their influence in ranking. Experiments on CNN/DM and XSum datasets demonstrate the effectiveness of our method. Notably, BalSum achieves an 89.67 BERTScore on CNN/DM, reaching a new state-of-the-art performance. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Neural Model Training, Linguistics, Abstractive Summarization, Semantic Similarity Evaluation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2104.07210" label="2104.07210">
        <attvalues>
          <attvalue for="0" value="RefSum: Refactoring Neural Summarization" />
          <attvalue for="1" value="  Although some recent works show potential complementarity among different&#10;state-of-the-art systems, few works try to investigate this problem in text&#10;summarization. Researchers in other areas commonly refer to the techniques of&#10;reranking or stacking to approach this problem. In this work, we highlight&#10;several limitations of previous methods, which motivates us to present a new&#10;framework Refactor that provides a unified view of text summarization and&#10;summaries combination. Experimentally, we perform a comprehensive evaluation&#10;that involves twenty-two base systems, four datasets, and three different&#10;application scenarios. Besides new state-of-the-art results on CNN/DailyMail&#10;dataset (46.18 ROUGE-1), we also elaborate on how our proposed method addresses&#10;the limitations of the traditional methods and the effectiveness of the&#10;Refactor model sheds light on insight for performance improvement. Our system&#10;can be directly used by other researchers as an off-the-shelf tool to achieve&#10;further performance improvements. We open-source all the code and provide a&#10;convenient interface to use it:&#10;https://github.com/yixinL7/Refactoring-Summarization. We have also made the&#10;demo of this work available at:&#10;http://explainaboard.nlpedia.ai/leaderboard/task-summ/index.php.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.01890" label="2106.01890">
        <attvalues>
          <attvalue for="0" value="SimCLS: A Simple Framework for Contrastive Learning of Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  In this paper, we present a conceptually simple while empirically powerful&#10;framework for abstractive summarization, SimCLS, which can bridge the gap&#10;between the learning objective and evaluation metrics resulting from the&#10;currently dominated sequence-to-sequence learning framework by formulating text&#10;generation as a reference-free evaluation problem (i.e., quality estimation)&#10;assisted by contrastive learning. Experimental results show that, with minor&#10;modification over existing top-scoring systems, SimCLS can improve the&#10;performance of existing top-performing models by a large margin. Particularly,&#10;2.51 absolute improvement against BART and 2.50 over PEGASUS w.r.t ROUGE-1 on&#10;the CNN/DailyMail dataset, driving the state-of-the-art performance to a new&#10;level. We have open-sourced our codes and results:&#10;https://github.com/yixinL7/SimCLS. Results of our proposed models have been&#10;deployed into ExplainaBoard platform, which allows researchers to understand&#10;our systems in a more fine-grained way.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.06569" label="2203.06569">
        <attvalues>
          <attvalue for="0" value="SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for&#10;  Abstractive Summarization" />
          <attvalue for="1" value="  Sequence-to-sequence neural networks have recently achieved great success in&#10;abstractive summarization, especially through fine-tuning large pre-trained&#10;language models on the downstream dataset. These models are typically decoded&#10;with beam search to generate a unique summary. However, the search space is&#10;very large, and with the exposure bias, such decoding is not optimal. In this&#10;paper, we show that it is possible to directly train a second-stage model&#10;performing re-ranking on a set of summary candidates. Our mixture-of-experts&#10;SummaReranker learns to select a better candidate and consistently improves the&#10;performance of the base model. With a base PEGASUS, we push ROUGE scores by&#10;5.44% on CNN-DailyMail (47.16 ROUGE-1), 1.31% on XSum (48.12 ROUGE-1) and 9.34%&#10;on Reddit TIFU (29.83 ROUGE-1), reaching a new state-of-the-art. Our code and&#10;checkpoints will be available at https://github.com/ntunlp/SummaReranker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.08795" label="2004.08795">
        <attvalues>
          <attvalue for="0" value="Extractive Summarization as Text Matching" />
          <attvalue for="1" value="  This paper creates a paradigm shift with regard to the way we build neural&#10;extractive summarization systems. Instead of following the commonly used&#10;framework of extracting sentences individually and modeling the relationship&#10;between sentences, we formulate the extractive summarization task as a semantic&#10;text matching problem, in which a source document and candidate summaries will&#10;be (extracted from the original text) matched in a semantic space. Notably,&#10;this paradigm shift to semantic matching framework is well-grounded in our&#10;comprehensive analysis of the inherent gap between sentence-level and&#10;summary-level extractors based on the property of the dataset.&#10;  Besides, even instantiating the framework with a simple form of a matching&#10;model, we have driven the state-of-the-art extractive result on CNN/DailyMail&#10;to a new level (44.41 in ROUGE-1). Experiments on the other five datasets also&#10;show the effectiveness of the matching framework. We believe the power of this&#10;matching-based summarization framework has not been fully exploited. To&#10;encourage more instantiations in the future, we have released our codes,&#10;processed dataset, as well as generated summaries in&#10;https://github.com/maszhongming/MatchSum.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.00099" label="2107.00099">
        <attvalues>
          <attvalue for="0" value="Sequence-level Confidence Classifier for ASR Utterance Accuracy and&#10;  Application to Acoustic Models" />
          <attvalue for="1" value="  Scores from traditional confidence classifiers (CCs) in automatic speech&#10;recognition (ASR) systems lack universal interpretation and vary with updates&#10;to the underlying confidence or acoustic models (AMs). In this work, we build&#10;interpretable confidence scores with an objective to closely align with ASR&#10;accuracy. We propose a new sequence-level CC with a richer context providing CC&#10;scores highly correlated with ASR accuracy and scores stable across CC updates.&#10;Hence, expanding CC applications. Recently, AM customization has gained&#10;traction with the widespread use of unified models. Conventional adaptation&#10;strategies that customize AM expect well-matched data for the target domain&#10;with gold-standard transcriptions. We propose a cost-effective method of using&#10;CC scores to select an optimal adaptation data set, where we maximize ASR gains&#10;from minimal data. We study data in various confidence ranges and optimally&#10;choose data for AM adaptation with KL-Divergence regularization. On the&#10;Microsoft voice search task, data selection for supervised adaptation using the&#10;sequence-level confidence scores achieves word error rate reduction (WERR) of&#10;8.5% for row-convolution LSTM (RC-LSTM) and 5.2% for latency-controlled&#10;bidirectional LSTM (LC-BLSTM). In the semi-supervised case, with ASR hypotheses&#10;as labels, our method provides WERR of 5.9% and 2.8% for RC-LSTM and LC-BLSTM,&#10;respectively.&#10;" />
          <attvalue for="2" value="&#10;Deep learning techniques have led to highly accurate ASR systems, resulting in their widespread use. It is therefore critical to ensure the reliability of ASR hypotheses. A confidence classifier is one such integral part of an ASR system, providing a measure of ASR reliability. But the current confidence scores lack a common meaning and potentially change with variation to confidence model or AM. Hence, we develop a sequence-level confidence model with an objective to closely tie the CC score for an utterance to its ASR accuracy. Thus, providing interpretation and uniformity in confidence scores across updates to confidence model or AM and expand CC applications. We study one such use, AM customization. We propose a cost-effective method using confidence scores for optimal data selection for AM customization in supervised and semi-supervised cases.&#10;&#10;ASR confidence applications include (a) mitigating false alarms in ASR-enabled devices in always-listening mode, (b) data selection for AM training and adaptation, (c) arbitration to select the best hypotheses between client and server results, (d) key-word spotting tasks. CCs are commonly built with multi-layer perceptron (MLP)~\cite{Huang2013a}, recurrent neural networks (RNNs)~\cite{Kalgaonkar2015EstimatingNetworks}, and bidirectional RNNs~\cite{Del-Agua2018Speaker-adaptedNetworks, Ragni2019ConfidenceNetworks}. These work at a word or a sub-word level~\cite{CMsurvey_Jiang_SpeechCommunication06}. Few methods use context~\cite{Sarma2004}, and alternative hypotheses to perform utterance verification~\cite{Setlur1996}. Recent works focus on training confidence models at an utterance-level to classify correct/incorrect utterances with applications to distributed end-to-end speech recognition systems~\cite{Kumar2020a, Li2021}. &#10;&#10;Conventional CC scores lack absolute interpretation and reflect the underlying CC training data, AM, language model, task, etc. So the scores from two different CCs are not directly comparable. Thus, needing to shift the confidence thresholds across CC updates or confidence normalization~\cite{Kumar2014} to recalibrate scores and retain those thresholds. While normalization preserves operating confidence thresholds across CC updates, it imparts no meaning to the scores. ASR developers or consumers expect CC scores to have a reasonable interpretation. We pursue that missing feature and introduce the ability to tie CC scores directly to ASR accuracy. In prior systems, a confidence score of say $0.9$ lacked an exact meaning except for the expectation that corresponding ASR hypotheses could be more accurate than the ones with lower confidence, say $0.8$. However, in the proposed CC, we expect a confidence score of $0.9$ for an utterance to imply 90\% accuracy. &#10;&#10;There is an increase in popularity of ASR systems~\cite{Ochiai2017UnifiedBeamforming, Fujita2016} built with ``unified data'', consolidated from domains like dictation, voice search, command-and-control, call center, etc. The acoustic model trained on unified data--a ``unified ASR''--simplifies AM training, where a single large-scale model includes multiple acoustic scenarios. It also facilitates better data sharing and delivers higher accuracy on unseen tasks. Given sufficient data for a specific domain, it is cost-effective to customize the unified model to the target domain rather than training a domain-dependent model for higher accuracy~\cite{Li2020, Bell2020}. &#10;&#10;Change in the target domain, speakers, tasks, accents, etc., is dealt with using AM adaptation~\cite{Bell2020}. Typically, it needs data well-matched to the target with supervised annotations but is impractical due to privacy and data constraints. Previously, a seed model provided target labels in a semi-supervised manner, filtering data using normalized frame-level entropy~\cite{Liu2007}, and minimum Bayes risk~\cite{Walker2017}. Others~\cite{Manohar2018, Klejch2019} used multiple potential transcriptions with a lattice or graph. We aim to design an efficient and scalable approach to customize AM. We propose to use confidence scores for supervised and semi-supervised AM adaptation. If supervised, we optimally select data for subsequent transcription and adaptation with a goal to minimize transcription costs and maximize ASR adaptation gains. In semi-supervised cases, we chose higher-quality ASR hypotheses.&#10;&#10;In this work, we propose a sequence-level LSTM-based confidence model whose scores align to ASR accuracy and stable across updates, eliminating the need for confidence normalization. Such a CC enables us to perform data selection and has scores tied to accuracy. We used CC scores to choose an optimal subset for AM adaptation developing a cost-effective method resulting in an improved WER. Besides, semi-supervised adaptation performance with optimal data is comparable to the supervised one with all the data. Section~\ref{sec:cc} presents the proposed sequence-level CC. Section~\ref{sec:am} details our recommendations for optimal data selection to customize AM. The experimental setup are in Section~\ref{sec:exp} and the results and discussion in Section~\ref{sec:results}. We conclude with Section~\ref{sec:conclusion}.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Confidence Modeling, Model Adaptation, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1705.09724" label="1705.09724">
        <attvalues>
          <attvalue for="0" value="Semi-Supervised Model Training for Unbounded Conversational Speech&#10;  Recognition" />
          <attvalue for="1" value="  For conversational large-vocabulary continuous speech recognition (LVCSR)&#10;tasks, up to about two thousand hours of audio is commonly used to train state&#10;of the art models. Collection of labeled conversational audio however, is&#10;prohibitively expensive, laborious and error-prone. Furthermore, academic&#10;corpora like Fisher English (2004) or Switchboard (1992) are inadequate to&#10;train models with sufficient accuracy in the unbounded space of conversational&#10;speech. These corpora are also timeworn due to dated acoustic telephony&#10;features and the rapid advancement of colloquial vocabulary and idiomatic&#10;speech over the last decades. Utilizing the colossal scale of our unlabeled&#10;telephony dataset, we propose a technique to construct a modern, high quality&#10;conversational speech training corpus on the order of hundreds of millions of&#10;utterances (or tens of thousands of hours) for both acoustic and language model&#10;training. We describe the data collection, selection and training, evaluating&#10;the results of our updated speech recognition system on a test corpus of 7K&#10;manually transcribed utterances. We show relative word error rate (WER)&#10;reductions of {35%, 19%} on {agent, caller} utterances over our seed model and&#10;5% absolute WER improvements over IBM Watson STT on this conversational speech&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.06551" label="2102.06551">
        <attvalues>
          <attvalue for="0" value="A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing&#10;  Task for Low-resource Morphologically Rich Languages" />
          <attvalue for="1" value="  Neural dependency parsing has achieved remarkable performance for many&#10;domains and languages. The bottleneck of massive labeled data limits the&#10;effectiveness of these approaches for low resource languages. In this work, we&#10;focus on dependency parsing for morphological rich languages (MRLs) in a&#10;low-resource setting. Although morphological information is essential for the&#10;dependency parsing task, the morphological disambiguation and lack of powerful&#10;analyzers pose challenges to get this information for MRLs. To address these&#10;challenges, we propose simple auxiliary tasks for pretraining. We perform&#10;experiments on 10 MRLs in low-resource settings to measure the efficacy of our&#10;proposed pretraining method and observe an average absolute gain of 2 points&#10;(UAS) and 3.6 points (LAS). Code and data available at:&#10;https://github.com/jivnesh/LCM&#10;" />
          <attvalue for="2" value="&#10;&#10;Dependency parsing has greatly benefited from neural network-based approaches. While these approaches simplify the parsing architecture and eliminate the need for hand-crafted feature engineering \cite{chen-manning-2014-fast,dyer-etal-2015-transition,kiperwasser-goldberg-2016-simple,DBLP:conf/iclr/DozatM17,kulmizev-etal-2019-deep}, their performance has been less exciting for several morphologically rich languages (MRLs) and low-resource languages \cite{more-etal-2019-joint,seeker-cetinoglu-2015-graph}. In fact, the need for large labeled treebanks for such systems has adversely affected the development of parsing solutions for low-resource languages~\cite{vania-etal-2019-systematic}. &#10;\newcite{zeman-etal-2018-conll} observe that data-driven parsing on 9 low resource treebanks resulted not only in low scores but those outputs ``are hardly useful for downstream applications&quot;.&#10;&#10;Several approaches have been suggested for improving the parsing performance of low-resource languages. This includes data augmentation strategies, cross-lingual transfer \cite{vania-etal-2019-systematic} and using unlabelled data with semi-supervised learning \cite{clark-etal-2018-semi} and self-training \cite{rotman2019deep}. Further, incorporating morphological knowledge substantially improves the parsing performance for MRLs, including low-resource languages ~\cite{vania-etal-2018-character,dehouck-denis-2018-framework}. This aligns well with the linguistic intuition of the role of morphological markers, especially that of case markers, in deciding the syntactic roles for the words involved \cite{wunderlich2001interaction,sigurdhsson2003case,kittila2011introduction}. However, obtaining the morphological tags for input sentences during run time is a challenge in itself for MRLs \cite{more-etal-2019-joint} and use of predicted tags from taggers, if available, often hampers the performance of these parsers. In this work, we primarily focus on one such morphologically-rich low-resource language, Sanskrit. &#10;&#10;We propose a simple pretraining approach, where we incorporate encoders from simple auxiliary tasks by means of a gating mechanism \cite{sato-etal-2017-adversarial}. This approach outperforms multi-task training and transfer learning methods under the same low-resource data conditions ($\sim$500 sentences). The proposed approach when applied to \newcite{dozat2017stanford}, a neural parser, not only obviates the need for providing morphological tags as input at runtime, but also outperforms its original configuration that uses gold morphological tags as input. Further, our method performs close to DCST \cite{rotman2019deep}, a self-training based extension of \newcite{dozat2017stanford}, which uses gold morphological tags as input for training. &#10;&#10;To measure the efficacy of the proposed method, we further perform a series of experiments on 10 MRLs in low-resource settings and show 2 points and 3.6 points average absolute gain (\S~\ref{results}) in terms of UAS and LAS, respectively. Our proposed method also outperforms multilingual BERT \cite[mBERT]{devlin-etal-2019-bert} based multi-task learning model \cite[Udify]{kondratyuk-straka-2019-75} for the languages which are not covered in mBERT (\S~\ref{Comparison with mBERT Pretraining}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Dependency Parsing, Computer Science, Linguistics, Artificial Intelligence, Low Resource Settings, Morphological Rich Languages" />
        </attvalues>
      </node>
      <node id="1809.08370" label="1809.08370">
        <attvalues>
          <attvalue for="0" value="Semi-Supervised Sequence Modeling with Cross-View Training" />
          <attvalue for="1" value="  Unsupervised representation learning algorithms such as word2vec and ELMo&#10;improve the accuracy of many supervised NLP models, mainly because they can&#10;take advantage of large amounts of unlabeled text. However, the supervised&#10;models only learn from task-specific labeled data during the main training&#10;phase. We therefore propose Cross-View Training (CVT), a semi-supervised&#10;learning algorithm that improves the representations of a Bi-LSTM sentence&#10;encoder using a mix of labeled and unlabeled data. On labeled examples,&#10;standard supervised learning is used. On unlabeled examples, CVT teaches&#10;auxiliary prediction modules that see restricted views of the input (e.g., only&#10;part of a sentence) to match the predictions of the full model seeing the whole&#10;input. Since the auxiliary modules and the full model share intermediate&#10;representations, this in turn improves the full model. Moreover, we show that&#10;CVT is particularly effective when combined with multi-task learning. We&#10;evaluate CVT on five sequence tagging tasks, machine translation, and&#10;dependency parsing, achieving state-of-the-art results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.04286" label="1911.04286">
        <attvalues>
          <attvalue for="0" value="Deep Contextualized Self-training for Low Resource Dependency Parsing" />
          <attvalue for="1" value="  Neural dependency parsing has proven very effective, achieving&#10;state-of-the-art results on numerous domains and languages. Unfortunately, it&#10;requires large amounts of labeled data, that is costly and laborious to create.&#10;In this paper we propose a self-training algorithm that alleviates this&#10;annotation bottleneck by training a parser on its own output. Our Deep&#10;Contextualized Self-training (DCST) algorithm utilizes representation models&#10;trained on sequence labeling tasks that are derived from the parser's output&#10;when applied to unlabeled data, and integrates these models with the base&#10;parser through a gating mechanism. We conduct experiments across multiple&#10;languages, both in low resource in-domain and in cross-domain setups, and&#10;demonstrate that DCST substantially outperforms traditional self-training as&#10;well as recent semi-supervised training methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.08393" label="2406.08393">
        <attvalues>
          <attvalue for="0" value="SCDNet: Self-supervised Learning Feature-based Speaker Change Detection" />
          <attvalue for="1" value="  Speaker Change Detection (SCD) is to identify boundaries among speakers in a&#10;conversation. Motivated by the success of fine-tuning wav2vec 2.0 models for&#10;the SCD task, a further investigation of self-supervised learning (SSL)&#10;features for SCD is conducted in this work. Specifically, an SCD model, named&#10;SCDNet, is proposed. With this model, various state-of-the-art SSL models,&#10;including Hubert, wav2vec 2.0, and WavLm are investigated. To discern the most&#10;potent layer of SSL models for SCD, a learnable weighting method is employed to&#10;analyze the effectiveness of intermediate representations. Additionally, a&#10;fine-tuning-based approach is also implemented to further compare the&#10;characteristics of SSL models in the SCD task. Furthermore, a contrastive&#10;learning method is proposed to mitigate the overfitting tendencies in the&#10;training of both the fine-tuning-based method and SCDNet. Experiments showcase&#10;the superiority of WavLm in the SCD task and also demonstrate the good design&#10;of SCDNet.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speaker Diarization (SD), a pivotal method in speech processing, aims to answer the question of `who speaks when' in scenarios involving multiple speakers~\cite{hruz2017convolutional}. In contrast, Speaker Change Detection (SCD) is to find the speaker turn points in the conversation~\cite{kunevsova2023multitask}, and thus it can be regarded as a subtask of SD~\cite{yin2018neural}, and also with broad applications, e.g., enhancing Automatic Speech Recognition (ASR) accuracy~\cite{sari2020auxiliary} and syncopate captioning~\cite{donabauer2021making}. &#10;&#10;The metric-based approach is a common early method for the SCD task, wherein speaker change points are identified through the comparison of distributions between two consecutive speech windows~\cite{kemp2000strategies}. Following the emergence of i-vector~\cite{dehak2010front} and DNN-based embeddings~\cite{snyder2018x}, uniform segmentation schemes have gained popularity as effective methods~\cite{sell2018diarization}. In this approach, the target audio undergoes segmentation into a series of segments with a constant window length and overlap length. Subsequently, speech embeddings from various segments are compared to determine if the speaker has changed. However, due to the fixed window length, a trade-off is inevitable between the efficacy of speech embedding and the accuracy of boundary detection.&#10;&#10;To overcome the limitations of segment-based methods, various works have endeavored to predict speaker change points at the frame level through neural networks~\cite{hruz2018lstm, hruz2017convolutional, fan2022sequence}. In these approaches, the model is generally trained with ground-truth SCD labels to perform a binary classification task. To be specific, in~\cite{hruz2018lstm}, using LSTM as the backbone, the optimizing target is to minimize the distance between the predicted probability signal and linear fuzzy labeling signal. &#10;&#10;In addition to label-based methods for frame-level SCD, several works have explored leveraging text transcription for word-level speaker change detection through ASR techniques~\cite{xia2022turn,zhao2023augmenting}. For example, in~\cite{xia2022turn}, the transcription used to train an ASR model is enhanced by incorporating a distinct token designed to denote speaker turns. Then the augmented transcription is used to train an ASR model that predicts not only regular text tokens but also special speaker turn tokens. While this approach alleviates the necessity for boundary annotations, using the textual transcription can be more intricate, especially in a dialogue scenario characterized by frequent interruptions and insertions, and the prevalence of intonation markers. &#10;Additionally, because the predicted boundaries in this method operate at the word level, the precision of boundary predictions may not be as high as those based on frame-level predictions.&#10;&#10;Most recently, Kune{\v{s}}ov{\'a} and Zaj{\'\i}~\cite{kunevsova2023multitask} explored the effectiveness of one of the most popular SSL models, wav2vec 2.0~\cite{baevski2020wav2vec}, on the SCD task. In their research, the pre-trained wav2vec 2.0 is fine-tuned in an end-to-end way involving multi-tasks, i.e., SCD, Overlapping Speech Detection (OSD), and Voice Activity Detection (VAD). This wav2vec 2.0 and multitask-based method showcases the remarkable performance, achieving a state-of-the-art (SOTA) level in the SCD task. Inspired by this research, we are conducting a further investigation into SSL-based end-to-end training methods for SCD.&#10;&#10;On the one hand, due to the typically large number of parameters in SSL models, directly fine-tuning them requires a certain threshold of data and computational resources. On the other hand, despite efforts by Kune{\v{s}}ov{\'a} and Zaj{\'\i}~\cite{kunevsova2023multitask} to enhance SCD performance through multitasking, e.g., OSD and VAD, all these tasks are frame-level binary classification tasks, which pose a risk of overfitting when training complex models due to the simplistic learning paradigm. Additionally, besides wav2vec 2.0, other SSL models such as Hubert~\cite{hsu2021hubert} and WavLm~\cite{chen2022wavlm} have also gained significant attention in various downstream tasks, such as Hubert-based speech recognition~\cite{nasersharif2023speech} and WavLm-based speech synthesis~\cite{lajszczak2024base}. However, the performance of these models in SCD has not been explored. &#10;&#10;To tackle those issues, we propose an innovative end-to-end SCD model, referred to as SCDNet, based on the Conformer architecture~\cite{gulati2020conformer}. SCDNet leverages off-the-shelf features as inputs and undergoes end-to-end training to accomplish the SCD task. Additionally, we propose a contrastive learning method for training SCD-oriented models to address the overfitting tendency associated with the frame-level binary classification task. Furthermore, we explore the performance of various SSL features via both SCDNet and fine-tuning-based methods.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Artificial Intelligence, Signal Processing, Self-Supervised Learning, Speaker Change Detection" />
        </attvalues>
      </node>
      <node id="2210.14755" label="2210.14755">
        <attvalues>
          <attvalue for="0" value="Multitask Detection of Speaker Changes, Overlapping Speech and Voice&#10;  Activity Using wav2vec 2.0" />
          <attvalue for="1" value="  Self-supervised learning approaches have lately achieved great success on a&#10;broad spectrum of machine learning problems. In the field of speech processing,&#10;one of the most successful recent self-supervised models is wav2vec 2.0. In&#10;this paper, we explore the effectiveness of this model on three basic speech&#10;classification tasks: speaker change detection, overlapped speech detection,&#10;and voice activity detection. First, we concentrate on only one task -- speaker&#10;change detection -- where our proposed system surpasses the previously reported&#10;results on four different corpora, and achieves comparable performance even&#10;when trained on out-of-domain data from an artificially designed dataset. Then&#10;we expand our approach to tackle all three tasks in a single multitask system&#10;with state-of-the-art performance on the AMI corpus. The implementation of the&#10;algorithms in this paper is publicly available at&#10;https://github.com/mkunes/w2v2_audioFrameClassification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.11641" label="2109.11641">
        <attvalues>
          <attvalue for="0" value="Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer&#10;  Transducer Speaker Turn Detection" />
          <attvalue for="1" value="  In this paper, we present a novel speaker diarization system for streaming&#10;on-device applications. In this system, we use a transformer transducer to&#10;detect the speaker turns, represent each speaker turn by a speaker embedding,&#10;then cluster these embeddings with constraints from the detected speaker turns.&#10;Compared with conventional clustering-based diarization systems, our system&#10;largely reduces the computational cost of clustering due to the sparsity of&#10;speaker turns. Unlike other supervised speaker diarization systems which&#10;require annotations of time-stamped speaker labels for training, our system&#10;only requires including speaker turn tokens during the transcribing process,&#10;which largely reduces the human efforts involved in data collection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.06482" label="2211.06482">
        <attvalues>
          <attvalue for="0" value="Augmenting Transformer-Transducer Based Speaker Change Detection With&#10;  Token-Level Training Loss" />
          <attvalue for="1" value="  In this work we propose a novel token-based training strategy that improves&#10;Transformer-Transducer (T-T) based speaker change detection (SCD) performance.&#10;The conventional T-T based SCD model loss optimizes all output tokens equally.&#10;Due to the sparsity of the speaker changes in the training data, the&#10;conventional T-T based SCD model loss leads to sub-optimal detection accuracy.&#10;To mitigate this issue, we use a customized edit-distance algorithm to estimate&#10;the token-level SCD false accept (FA) and false reject (FR) rates during&#10;training and optimize model parameters to minimize a weighted combination of&#10;the FA and FR, focusing the model on accurately predicting speaker changes. We&#10;also propose a set of evaluation metrics that align better with commercial use&#10;cases. Experiments on a group of challenging real-world datasets show that the&#10;proposed training method can significantly improve the overall performance of&#10;the SCD model with the same number of parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.03940" label="2304.03940">
        <attvalues>
          <attvalue for="0" value="Unsupervised Speech Representation Pooling Using Vector Quantization" />
          <attvalue for="1" value="  With the advent of general-purpose speech representations from large-scale&#10;self-supervised models, applying a single model to multiple downstream tasks is&#10;becoming a de-facto approach. However, the pooling problem remains; the length&#10;of speech representations is inherently variable. The naive average pooling is&#10;often used, even though it ignores the characteristics of speech, such as&#10;differently lengthed phonemes. Hence, we design a novel pooling method to&#10;squash acoustically similar representations via vector quantization, which does&#10;not require additional training, unlike attention-based pooling. Further, we&#10;evaluate various unsupervised pooling methods on various self-supervised&#10;models. We gather diverse methods scattered around speech and text to evaluate&#10;on various tasks: keyword spotting, speaker identification, intent&#10;classification, and emotion recognition. Finally, we quantitatively and&#10;qualitatively analyze our method, comparing it with supervised pooling methods.&#10;" />
          <attvalue for="2" value="&#10;wav2vec 2.0 \cite{baevski2020wav2vec} has swept the speech processing community like a storm with the surprising effectiveness of transformer-based self-supervised models \cite{yang2021superb}.&#10;Self-supervision leverages large quantities of unlabeled data, which is much easier to obtain than labeled data.&#10;Many task designs exist, but one of the most common is to mask the input and train the neural network to reconstruct the original input based on the surrounding information \cite{kenton2019bert,baevski2020wav2vec}.&#10;The task definition enables the neural network to model the input distribution so that it can yield a general representation useful for various downstream tasks.&#10;&#10;After building a general self-supervised model, supervised training ensues to fine-tune the model to each specific need.&#10;One often attaches the penultimate fully connected layer to the frozen shared model, acting as the prediction head \cite{yang2021superb}.&#10;However, input lengths often vary for both speech and text, a fundamental characteristic of sequential data.&#10;Transformer's output representation length linearly increases as the input size gets longer, while the final prediction head requires the fixed-size input representation.&#10;It raises the problem of representation pooling; given a variable number of sequential representations, one has to summarize the representation to have a fixed size.&#10;&#10;The pooling task is often addressed as the unsupervised or paraphrastic sentence embedding problem in the NLP literature \cite{wieting2015towards,arora2017simple}, especially after the success of word embeddings on various tasks.&#10;It aims to yield a general-purpose sentence embedding independent of each downstream task.&#10;By starting from simply averaging the word representations \cite{wieting2015towards}, many unparameterized methods were introduced, such as considering the word frequency while averaging \cite{arora2017simple}, whitening the representations \cite{huang2021whiteningbert,su2021whitening}, or utilizing the singular value transformation \cite{yan2022addressing}.&#10;However, existing methods are often dependent on the underlying tokenizer, making it nontrivial to apply to speech.&#10;Also, simply averaging the speech representations poses a fundamental problem; each phone has a different length, so vowels will be overrepresented compared to consonants.&#10;&#10;In the meantime, many pooling methods have been introduced for summarizing the frame-wise speech representations, notably for speaker recognition.&#10;For example, statistics pooling (SP) \cite{snyder2016deep,snyder2017deep} concatenates the first- and second-order statistics of representations.&#10;Also, the attention layer is often used to obtain which representation matters more for each downstream task \cite{okabe2018attentive,safari2020self,wu2020vector}.&#10;However, there has been limited interest in developing an unsupervised way of pooling.&#10;Modern methods, except for SP, are often heavily parameterized, requiring labeled samples.&#10;In contrast, unsupervised pooling removes the training procedure altogether, being more closer to the idea of general speech embedding.&#10;&#10;To avoid parameterization while being speech-friendly, we shed light on vector quantization (VQ) to substitute the tokenizer.&#10;Decades of research have been conducted on VQ \cite{soong1987report}, where it crept into the modern self-supervised models, such as vq-wav2vec \cite{jegou2010product,baevski2020vq}.&#10;VQ focuses on translating the real-valued vectors into countable indices, segmenting the latent embedding space with well-chosen centroids.&#10;Given that essential components comprising speech, such as frequency, amplitude, and formants, are embedded inside the model representation\cite{choi2022opening}, we expect the clusters to successfully gather similar phone representations to have the same cluster indices.&#10;We design various intuitive approaches that utilize VQ to effectively summarize the speech representations, and experimentally demonstrate its effectiveness in a wide range of settings.&#10;&#10;To further boost the research on unsupervised pooling methods, we devise a benchmark that evaluates various pooling methods across multiple backbone networks and a wide range of tasks, namely, keyword spotting \cite{warden2018speech}, speaker identification \cite{nagrani2017voxceleb}, intent classification \cite{lugosch2019speech}, and emotion recognition \cite{busso2008iemocap}.&#10;To avoid supervision altogether, we closely follow the recent contrastive learning literature \cite{chen2020simple,choi2022combating} to evaluate the effectiveness of the pooled representation directly via the nearest-neighbor approach \cite{bernhardsson2017annoy}.&#10;Also, we modify the existing unsupervised sentence embedding methods to compare their downstream performance.&#10;We further conduct various analyses that accurately depict the effectiveness and behaviors of our method by comparing with the parameterized methods via supervised learning, visualizing the weights directly, and exploring different settings for benchmark evaluation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Pooling Methods, Speech Representation, Signal Processing, Unsupervised Learning" />
        </attvalues>
      </node>
      <node id="2104.01767" label="2104.01767">
        <attvalues>
          <attvalue for="0" value="WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach" />
          <attvalue for="1" value="  Producing the embedding of a sentence in an unsupervised way is valuable to&#10;natural language matching and retrieval problems in practice. In this work, we&#10;conduct a thorough examination of pretrained model based unsupervised sentence&#10;embeddings. We study on four pretrained models and conduct massive experiments&#10;on seven datasets regarding sentence semantics. We have there main findings.&#10;First, averaging all tokens is better than only using [CLS] vector. Second,&#10;combining both top andbottom layers is better than only using top layers.&#10;Lastly, an easy whitening-based vector normalization strategy with less than 10&#10;lines of code consistently boosts the performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Unsupervised sentence embeddings are mainly composed with pre-trained (contextual) word embeddings \cite{Pennington2014GloveGV, Devlin2019BERTPO}. Recent attempts can be divided into two categories, according to whether the pre-trained embeddings are further trained or not.&#10;For the former, some works leverage unlabelled natural language inference datasets to train a sentence encoder without direct supervision \cite{Li2020BERTflow, Zhang2020ISBERT, Mu2018NATSV}. &#10;For the latter, some works propose weighted average word embeddings based on word features \cite{Arora2017SIF, Ethayarajh2018USIF, Yang2019GEM, Wang2020SBERTWK}. However, these approaches need further training or additional features, which limits the direct applications of sentence embeddings in real-world scenarios. Finally, we note that concurrent to this work, \newcite{Su2021WhiteningSR} also explored whitening sentence embedding, released to arXiv one week before our paper.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2208.11790" label="2208.11790">
        <attvalues>
          <attvalue for="0" value="Addressing Token Uniformity in Transformers via Singular Value&#10;  Transformation" />
          <attvalue for="1" value="  Token uniformity is commonly observed in transformer-based models, in which&#10;different tokens share a large proportion of similar information after going&#10;through stacked multiple self-attention layers in a transformer. In this paper,&#10;we propose to use the distribution of singular values of outputs of each&#10;transformer layer to characterise the phenomenon of token uniformity and&#10;empirically illustrate that a less skewed singular value distribution can&#10;alleviate the `token uniformity' problem. Base on our observations, we define&#10;several desirable properties of singular value distributions and propose a&#10;novel transformation function for updating the singular values. We show that&#10;apart from alleviating token uniformity, the transformation function should&#10;preserve the local neighbourhood structure in the original embedding space. Our&#10;proposed singular value transformation function is applied to a range of&#10;transformer-based language models such as BERT, ALBERT, RoBERTa and DistilBERT,&#10;and improved performance is observed in semantic textual similarity evaluation&#10;and a range of GLUE tasks. Our source code is available at&#10;https://github.com/hanqi-qi/tokenUni.git.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.10963" label="1803.10963">
        <attvalues>
          <attvalue for="0" value="Attentive Statistics Pooling for Deep Speaker Embedding" />
          <attvalue for="1" value="  This paper proposes attentive statistics pooling for deep speaker embedding&#10;in text-independent speaker verification. In conventional speaker embedding,&#10;frame-level features are averaged over all the frames of a single utterance to&#10;form an utterance-level feature. Our method utilizes an attention mechanism to&#10;give different weights to different frames and generates not only weighted&#10;means but also weighted standard deviations. In this way, it can capture&#10;long-term variations in speaker characteristics more effectively. An evaluation&#10;on the NIST SRE 2012 and the VoxCeleb data sets shows that it reduces equal&#10;error rates (EERs) from the conventional method by 7.5% and 8.1%, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.01077" label="2008.01077">
        <attvalues>
          <attvalue for="0" value="Self-attention encoding and pooling for speaker recognition" />
          <attvalue for="1" value="  The computing power of mobile devices limits the end-user applications in&#10;terms of storage size, processing, memory and energy consumption. These&#10;limitations motivate researchers for the design of more efficient deep models.&#10;On the other hand, self-attention networks based on Transformer architecture&#10;have attracted remarkable interests due to their high parallelization&#10;capabilities and strong performance on a variety of Natural Language Processing&#10;(NLP) applications. Inspired by the Transformer, we propose a tandem&#10;Self-Attention Encoding and Pooling (SAEP) mechanism to obtain a discriminative&#10;speaker embedding given non-fixed length speech utterances. SAEP is a stack of&#10;identical blocks solely relied on self-attention and position-wise feed-forward&#10;networks to create vector representation of speakers. This approach encodes&#10;short-term speaker spectral features into speaker embeddings to be used in&#10;text-independent speaker verification. We have evaluated this approach on both&#10;VoxCeleb1 &amp; 2 datasets. The proposed architecture is able to outperform the&#10;baseline x-vector, and shows competitive performance to some other benchmarks&#10;based on convolutions, with a significant reduction in model size. It employs&#10;94%, 95%, and 73% less parameters compared to ResNet-34, ResNet-50, and&#10;x-vector, respectively. This indicates that the proposed fully attention based&#10;architecture is more efficient in extracting time-invariant features from&#10;speaker utterances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.08612" label="1706.08612">
        <attvalues>
          <attvalue for="0" value="VoxCeleb: a large-scale speaker identification dataset" />
          <attvalue for="1" value="  Most existing datasets for speaker identification contain samples obtained&#10;under quite constrained conditions, and are usually hand-annotated, hence&#10;limited in size. The goal of this paper is to generate a large scale&#10;text-independent speaker identification dataset collected 'in the wild'. We&#10;make two contributions. First, we propose a fully automated pipeline based on&#10;computer vision techniques to create the dataset from open-source media. Our&#10;pipeline involves obtaining videos from YouTube; performing active speaker&#10;verification using a two-stream synchronization Convolutional Neural Network&#10;(CNN), and confirming the identity of the speaker using CNN based facial&#10;recognition. We use this pipeline to curate VoxCeleb which contains hundreds of&#10;thousands of 'real world' utterances for over 1,000 celebrities. Our second&#10;contribution is to apply and compare various state of the art speaker&#10;identification techniques on our dataset to establish baseline performance. We&#10;show that a CNN based architecture obtains the best performance for both&#10;identification and verification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.07932" label="2011.07932">
        <attvalues>
          <attvalue for="0" value="Combating the Instability of Mutual Information-based Losses via&#10;  Regularization" />
          <attvalue for="1" value="  Notable progress has been made in numerous fields of machine learning based&#10;on neural network-driven mutual information (MI) bounds. However, utilizing the&#10;conventional MI-based losses is often challenging due to their practical and&#10;mathematical limitations. In this work, we first identify the symptoms behind&#10;their instability: (1) the neural network not converging even after the loss&#10;seemed to converge, and (2) saturating neural network outputs causing the loss&#10;to diverge. We mitigate both issues by adding a novel regularization term to&#10;the existing losses. We theoretically and experimentally demonstrate that added&#10;regularization stabilizes training. Finally, we present a novel benchmark that&#10;evaluates MI-based losses on both the MI estimation power and its capability on&#10;the downstream tasks, closely following the pre-existing supervised and&#10;contrastive learning settings. We evaluate six different MI-based losses and&#10;their regularized counterparts on multiple benchmarks to show that our approach&#10;is simple yet effective.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.12171" label="2404.12171">
        <attvalues>
          <attvalue for="0" value="Stance Detection on Social Media with Fine-Tuned Large Language Models" />
          <attvalue for="1" value="  Stance detection, a key task in natural language processing, determines an&#10;author's viewpoint based on textual analysis. This study evaluates the&#10;evolution of stance detection methods, transitioning from early machine&#10;learning approaches to the groundbreaking BERT model, and eventually to modern&#10;Large Language Models (LLMs) such as ChatGPT, LLaMa-2, and Mistral-7B. While&#10;ChatGPT's closed-source nature and associated costs present challenges, the&#10;open-source models like LLaMa-2 and Mistral-7B offers an encouraging&#10;alternative. Initially, our research focused on fine-tuning ChatGPT, LLaMa-2,&#10;and Mistral-7B using several publicly available datasets. Subsequently, to&#10;provide a comprehensive comparison, we assess the performance of these models&#10;in zero-shot and few-shot learning scenarios. The results underscore the&#10;exceptional ability of LLMs in accurately detecting stance, with all tested&#10;models surpassing existing benchmarks. Notably, LLaMa-2 and Mistral-7B&#10;demonstrate remarkable efficiency and potential for stance detection, despite&#10;their smaller sizes compared to ChatGPT. This study emphasizes the potential of&#10;LLMs in stance detection and calls for more extensive research in this field.&#10;" />
          <attvalue for="2" value="&#10;&#10;Stance detection plays a crucial role in analyzing social media content, aiming to identify an author's viewpoint—supportive, oppositional, or neutral—towards various subjects, from political figures to environmental policies. This process, relying on nuanced textual analysis across platforms like X (formerly Twitter), presents both opportunities and challenges for accurate interpretation \cite{hasan-ng-2013-stance, 10.1145/3369026, DBLP:journals/corr/abs-2006-03644}, key to gaining insights into public sentiment and societal trends. These insights are invaluable for research in data extraction, policy analysis, and beyond \cite{ siddiqua-etal-2019-tweet, darwish2017trump, glandt-etal-2021-stance}. As natural language processing (NLP) and social computing fields advance, they provide enhanced tools for effectively extracting and analyzing stances from social media texts, reflecting the evolving landscape of online discourse.&#10;&#10;Initially, stance detection predominantly utilized rule-based and traditional machine learning approaches, with support vector machines (SVM) marking a significant early benchmark \cite{anand-etal-2011-cats, walker2012your, mohammad-etal-2016-semeval}. The introduction of deep learning models marked a pivotal shift, improving over traditional methods due to their ability to process large datasets and uncover complex patterns \cite{wei-etal-2016-pkudblab, zarrella-marsh-2016-mitre, dey2018topical, huang2018aspect, zhang-etal-2019-aspect}. The significant advancements continued with the development of pre-trained language models (PLMs) such as BERT, which facilitated a deeper understanding of textual context \cite{devlin-etal-2019-bert, li-etal-2021-p, kawintiranon-singh-2021-knowledge}.&#10;&#10;The evolution of Large Language Models (LLMs) like OpenAI's ChatGPT and Meta AI's LLaMa-2 represents a major leap in NLP \cite{qin2023chatgpt, touvron2023llama}. These models, trained on vast datasets, are adept at mimicking the subtleties of human language with remarkable accuracy \cite{zhao2023survey, yin-etal-2023-large}. Both open-source and adaptable, LLaMa-2 and Mistral-7B enhance NLP technology accessibility and facilitate research without proprietary constraints \cite{jiang2023mistral}. Unlike their predecessors, LLMs use flexible prompting strategies, allowing them to perform a wide array of NLP tasks directly, including stance detection. This capability to navigate the intricate relationship between textual context and target subjects, accurately aligning with the author's intended stance, underscores their advanced understanding of language \cite{brown2020language, chowdhery2022palm, wei2023larger}.&#10;&#10;The implementation of prompting strategies represents a significant departure from traditional NLP model training methods. By employing these strategies, LLMs can generate predictions without the extensive fine-tuning typically required, showcasing their versatility across various tasks. This methodological shift has not only simplified the application of LLMs but also expanded their utility, making them particularly effective for tasks that require a deep understanding of language nuances, such as stance detection \cite{bang-etal-2023-multitask, ouyang2022training}.&#10;&#10;Fine-tuning tailors LLMs like ChatGPT, LLaMa-2, and Mistral-7B to specific tasks, significantly enhancing their precision and relevance for contextually aware stance detection on social media platforms \cite{zhang2023llamaadapter}. This adaptation to the unique language and style of social media discourse enables these models to outperform traditional methods, demonstrating superior performance in identifying sentiments and viewpoints.&#10;&#10;The flexibility of LLMs in social media analysis is highlighted by their use of advanced techniques such as fine-tuning, chain of thought prompting \cite{chen2023you}, and both zero-shot and few-shot learning \cite{zhang2023stance, cruickshank2023use, aiyappa-etal-2023-trust}. These methods help navigate complex online discourse, enhancing the models' ability to interpret both explicit and implicit content effectively \cite{gatto2023chainofthought, lan2023stance}.&#10;&#10;Our study evaluates the enhanced performance of fine-tuned LLMs using Twitter datasets, now known as X, to analyze a wide range of user opinions. We aim to show that fine-tuning significantly improves model understanding of user viewpoints, offering a deeper insight into online discourse. This research underscores the advantages of fine-tuning in NLP, particularly in stance detection, highlighting its superiority over traditional and less-tailored methods.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Stance Detection Methods, Computer Science, Large Language Models, Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2303.16199" label="2303.16199">
        <attvalues>
          <attvalue for="0" value="LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init&#10;  Attention" />
          <attvalue for="1" value="  We present LLaMA-Adapter, a lightweight adaption method to efficiently&#10;fine-tune LLaMA into an instruction-following model. Using 52K self-instruct&#10;demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon&#10;the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8&#10;A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and&#10;prepend them to the word tokens at higher transformer layers. Then, a&#10;zero-initialized attention mechanism with zero gating is proposed, which&#10;adaptively injects the new instructional cues into LLaMA, while effectively&#10;preserves its pre-trained knowledge. With our efficient training, LLaMA-Adapter&#10;can generate high-quality responses, comparable to Alpaca with fully fine-tuned&#10;7B parameters. Besides language commands, our approach can be simply extended&#10;to multi-modal instructions for learning image-conditioned LLaMA model, which&#10;achieves superior reasoning performance on ScienceQA and COCO Caption&#10;benchmarks. Furthermore, we also evaluate the zero-initialized attention&#10;mechanism for fine-tuning other pre-trained models (ViT, RoBERTa) on&#10;traditional vision and language tasks, demonstrating the superior&#10;generalization capacity of our approach. Code is released at&#10;https://github.com/OpenGVLab/LLaMA-Adapter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.14548" label="2212.14548">
        <attvalues>
          <attvalue for="0" value="How would Stance Detection Techniques Evolve after the Launch of&#10;  ChatGPT?" />
          <attvalue for="1" value="  Stance detection refers to the task of extracting the standpoint (Favor,&#10;Against or Neither) towards a target in given texts. Such research gains&#10;increasing attention with the proliferation of social media contents. The&#10;conventional framework of handling stance detection is converting it into text&#10;classification tasks. Deep learning models have already replaced rule-based&#10;models and traditional machine learning models in solving such problems.&#10;Current deep neural networks are facing two main challenges which are&#10;insufficient labeled data and information in social media posts and the&#10;unexplainable nature of deep learning models. A new pre-trained language model&#10;chatGPT was launched on Nov 30, 2022. For the stance detection tasks, our&#10;experiments show that ChatGPT can achieve SOTA or similar performance for&#10;commonly used datasets including SemEval-2016 and P-Stance. At the same time,&#10;ChatGPT can provide explanation for its own prediction, which is beyond the&#10;capability of any existing model. The explanations for the cases it cannot&#10;provide classification results are especially useful. ChatGPT has the potential&#10;to be the best AI model for stance detection tasks in NLP, or at least change&#10;the research paradigm of this field. ChatGPT also opens up the possibility of&#10;building explanatory AI for stance detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.19750" label="2310.19750">
        <attvalues>
          <attvalue for="0" value="Chain-of-Thought Embeddings for Stance Detection on Social Media" />
          <attvalue for="1" value="  Stance detection on social media is challenging for Large Language Models&#10;(LLMs), as emerging slang and colloquial language in online conversations often&#10;contain deeply implicit stance labels. Chain-of-Thought (COT) prompting has&#10;recently been shown to improve performance on stance detection tasks --&#10;alleviating some of these issues. However, COT prompting still struggles with&#10;implicit stance identification. This challenge arises because many samples are&#10;initially challenging to comprehend before a model becomes familiar with the&#10;slang and evolving knowledge related to different topics, all of which need to&#10;be acquired through the training data. In this study, we address this problem&#10;by introducing COT Embeddings which improve COT performance on stance detection&#10;tasks by embedding COT reasonings and integrating them into a traditional&#10;RoBERTa-based stance detection pipeline. Our analysis demonstrates that 1) text&#10;encoders can leverage COT reasonings with minor errors or hallucinations that&#10;would otherwise distort the COT output label. 2) Text encoders can overlook&#10;misleading COT reasoning when a sample's prediction heavily depends on&#10;domain-specific patterns. Our model achieves SOTA performance on multiple&#10;stance detection datasets collected from social media.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.02557" label="2102.02557">
        <attvalues>
          <attvalue for="0" value="Adaptive Semiparametric Language Models" />
          <attvalue for="1" value="  We present a language model that combines a large parametric neural network&#10;(i.e., a transformer) with a non-parametric episodic memory component in an&#10;integrated architecture. Our model uses extended short-term context by caching&#10;local hidden states -- similar to transformer-XL -- and global long-term memory&#10;by retrieving a set of nearest neighbor tokens at each timestep. We design a&#10;gating function to adaptively combine multiple information sources to make a&#10;prediction. This mechanism allows the model to use either local context,&#10;short-term memory, or long-term memory (or any combination of them) on an ad&#10;hoc basis depending on the context. Experiments on word-based and&#10;character-based language modeling datasets demonstrate the efficacy of our&#10;proposed method compared to strong baselines.&#10;" />
          <attvalue for="2" value="&#10;Human language processing is facilitated by&#10;complex systems interacting together. A&#10;core component that enables such a process is human memory.&#10;Memory in humans consists of specialized systems,&#10;which forms a basis for&#10;intelligent behaviors \cite{tulving, rolls2000, eichenbaum2012memory}.&#10;For language processing, working (short-term) memory is &#10;a temporary storage that can be&#10;used to comprehend sentences and &#10;follow conversations. Episodic (long-term) memory stores individual experience &#10;and events. Semantic memory stores facts and knowledge about words&#10;and concepts.&#10;&#10;In artificial language processing systems (e.g., language models), &#10;a popular approach to&#10;design a better model&#10;is by encoding all of the desired knowledge (e.g., to produce &#10;grammatical sentences, process long text,&#10;remember events, etc.) in the weights of a large parametric &#10;neural network via end-to-end training.&#10;We see an increasingly larger transformer become&#10;a better language model \cite{gpt,gpt2,megatron,gpt3}. &#10;In this scale approach, the knowledge is &#10;implicitly represented in the weights of a parametric neural network, and &#10;it is not straightforward to interpret whether &#10;a model contains a particular knowledge without&#10;asking the model to produce a response---e.g., via a cloze-style question \cite{lmiskb} &#10;or a prompt \cite{gpt3}.&#10;&#10;An alternative strategy&#10;is to design a modular architecture that&#10;separates memory storage and computational processing, where each module has&#10;a clear purpose.&#10;Recent progress in memory-augmented neural networks has given rise to&#10;many variants of memory-augmented transformer language models&#10;that fall under this category.&#10;For example, attempts to incorporate&#10;extended local context to a neural network---such as those &#10;found in neural cache \cite{grave}, transformer-XL \cite{txl}&#10;compressive transformer \cite{compresstrans}, performers \cite{performer}, \revision{longformer \cite{longformer}, and reformer \cite{reformer}}---can &#10;be seen as &#10;models of working memory.&#10;Models of episodic memory&#10;include $k$NN-LM \cite{knnlm}&#10;and architectures that are designed for more complicated tasks &#10;such as question answering \cite{demasson, realm} and machine translation \cite{nnmt}.&#10;In&#10;machine learning and natural language processing,&#10;memory-augmented neural networks is used to refer to all types of memory systems.&#10;&#10;In this paper, \revision{inspired by the modular design of human memory systems, &#10;we present a language model architecture (Spalm)&#10;with storage modules that resemble working and episodic memory systems,&#10;which we combine with a large parametric neural network that&#10;is responsible for computation (\S{\ref{sec:model}})}.&#10;Our hypothesis is that encouraging&#10;each component to focus on a specific function (e.g.,&#10;storing long-term information, capturing extended context, &#10;modeling local information)&#10;facilitates easier training that produces an overall better language model.&#10;&#10;Specifically, we follow transformer-XL \cite{txl} to &#10;capture extended context&#10;by caching hidden states in a temporary short-term memory. For long-term context,&#10;we use a persistent key-value database and perform&#10;sparse retrieval with (approximate) $k$-nearest neighbors.&#10;In contrast to previous language models that either interpolate output probabilities&#10;\cite{smerity,grave,knnlm,bertknn} or use input concatenation \cite{realm,megatronctrl}&#10;to combine information&#10;from different sources, we design&#10;a context-dependent gating mechanism&#10;to incorporate local, extended, and global context.&#10;We discuss similarities and differences to related work in \S{\ref{sec:relwork}}.&#10;&#10;In language modeling, many tokens can be&#10;predicted from their local context without &#10;requiring long-term information.&#10;Our model can adaptively decide whether the current (local)&#10;context is enough, or whether it needs to use information from the short-term&#10;and/or long-term memory.&#10;&#10;In \S{\ref{sec:experiments}}, we compare Spalm with strong baselines---including transformer-XL and $k$NN-LM---on word-based and character-based&#10;language modeling.&#10;Our positive results establish the benefit of the proposed architecture.&#10;They also indicate the generality of our approach&#10;and its potential applicability to other sequence modeling tasks.&#10;&#10;We analyze how Spalm uses long vs. short-term context &#10;(\S{\ref{sec:analysis}}) to better understand how the model operates&#10;when making predictions.&#10;We conclude by discussing limitations and&#10;future directions (\S{\ref{sec:discussion}}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Cognitive Science, Mathematics, Language Modeling, Memory Augmentation" />
        </attvalues>
      </node>
      <node id="2005.00766" label="2005.00766">
        <attvalues>
          <attvalue for="0" value="BERT-kNN: Adding a kNN Search Component to Pretrained Language Models&#10;  for Better QA" />
          <attvalue for="1" value="  Khandelwal et al. (2020) use a k-nearest-neighbor (kNN) component to improve&#10;language model performance. We show that this idea is beneficial for&#10;open-domain question answering (QA). To improve the recall of facts encountered&#10;during training, we combine BERT (Devlin et al., 2019) with a traditional&#10;information retrieval step (IR) and a kNN search over a large datastore of an&#10;embedded text collection. Our contributions are as follows: i) BERT-kNN&#10;outperforms BERT on cloze-style QA by large margins without any further&#10;training. ii) We show that BERT often identifies the correct response category&#10;(e.g., US city), but only kNN recovers the factually correct answer (e.g.,&#10;&quot;Miami&quot;). iii) Compared to BERT, BERT-kNN excels for rare facts. iv) BERT-kNN&#10;can easily handle facts not covered by BERT's training set, e.g., recent&#10;events.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.06507" label="2308.06507">
        <attvalues>
          <attvalue for="0" value="AutoConv: Automatically Generating Information-seeking Conversations&#10;  with Large Language Models" />
          <attvalue for="1" value="  Information-seeking conversation, which aims to help users gather information&#10;through conversation, has achieved great progress in recent years. However, the&#10;research is still stymied by the scarcity of training data. To alleviate this&#10;problem, we propose AutoConv for synthetic conversation generation, which takes&#10;advantage of the few-shot learning ability and generation capacity of large&#10;language models (LLM). Specifically, we formulate the conversation generation&#10;problem as a language modeling task, then finetune an LLM with a few human&#10;conversations to capture the characteristics of the information-seeking process&#10;and use it for generating synthetic conversations with high quality.&#10;Experimental results on two frequently-used datasets verify that AutoConv has&#10;substantial improvements over strong baselines and alleviates the dependence on&#10;human annotation. In addition, we also provide several analysis studies to&#10;promote future research.&#10;" />
          <attvalue for="2" value="&#10;&#10;In information-seeking conversations, users repeatedly ask questions based on their interests, and the dialogue system provides answers to fulfill their information needs \cite{stede2004information, DBLP:conf/emnlp/ChoiHIYYCLZ18, DBLP:journals/tacl/ReddyCM19}.&#10;This scenario is important for addressing real-world open-ended questions, which requires discussions to explore in depth \cite{DBLP:conf/icml/DaiCZARGG22}, e.g., How to learn more efficiently? &#10;Though great progress has been achieved in recent years, most existing researches depend on abundant human annotation, which can be highly costly and limited in knowledge coverage.&#10;&#10;A promising way to alleviate this problem is data augmentation \cite{DBLP:journals/corr/abs-2106-07499}.&#10;Traditional methods, including token-level manipulation \cite{DBLP:conf/naacl/Kobayashi18, DBLP:conf/emnlp/WeiZ19} and sentence-level paraphrasing \cite{DBLP:conf/acl/SennrichHB16}, improve the linguistic diversity of training data.&#10;However, they cannot create conversations grounded on new documents, which are indispensable for dealing with out-of-domain scenarios.&#10;Another line of research focuses on simulation-based methods&#10;\cite{DBLP:journals/corr/abs-2112-08342, DBLP:journals/corr/abs-2205-12609}.&#10;Specifically, they can iteratively generate conversations grounded on new documents based on a span extractor and an utterance generator.&#10;Nevertheless, both the training of the extractor and the generator still require abundant human dialogues.&#10;Besides the above ways,&#10;\cite{DBLP:conf/icml/DaiCZARGG22} propose Dialog Inpainting, which creates information-seeking dialogues by inserting utterances between neighboring sentences in documents. &#10;One potential risk is the gap between the structure of documents and that of conversations.&#10;Documents are tighter, while real-world conversations are more open-ended.&#10;&#10;To alleviate the above issues, we propose a simple yet effective method AutoConv for Automatically generating information-seeking Conversations, which takes advantage of the few-shot learning ability and generation capacity of large language models (LLM) \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}.&#10;Specifically, we formulate conversation generation as a language modeling task and utilize an LLM for generating synthetic conversations grounded on external documents.&#10;Surprisingly, finetuning with a few human dialogues can help LLM capture the characteristics of the information-seeking process (e.g., grounding, question answering) and generate high-quality synthetic conversations.&#10;Then, we can train a small task model with these dialogues.&#10;The differences between AutoConv and others are shown in Table~\ref{tab:compare_with_other_methods}.&#10;&#10;We conduct comprehensive experiments on two frequently-used datasets QuAC \cite{DBLP:conf/emnlp/ChoiHIYYCLZ18} and CoQA \cite{DBLP:journals/tacl/ReddyCM19} in the low-resource setting, where only dozens of human dialogues are available.&#10;The results show that AutoConv has substantial improvements over several strong baselines.&#10;When scaling up the synthetic dialogues, AutoConv has the improvement of up to 5.06 F1 gain compared with directly finetuning, and thus largely reduces the labor force for annotation.&#10;In addition, we find that the small task model trained with synthetic dialogues can even surpass finetuned LLM with only $1.7\%$ parameters.&#10;Moreover, we also investigate the impact of decoding strategy and scaling laws for AutoConv.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Linguistics, Conversational AI, Artificial Intelligence, Natural Language Processing, Language Modeling, Data Generation" />
        </attvalues>
      </node>
      <node id="2205.09073" label="2205.09073">
        <attvalues>
          <attvalue for="0" value="Dialog Inpainting: Turning Documents into Dialogs" />
          <attvalue for="1" value="  Many important questions (e.g. &quot;How to eat healthier?&quot;) require conversation&#10;to establish context and explore in depth. However, conversational question&#10;answering (ConvQA) systems have long been stymied by scarce training data that&#10;is expensive to collect. To address this problem, we propose a new technique&#10;for synthetically generating diverse and high-quality dialog data: dialog&#10;inpainting. Our approach takes the text of any document and transforms it into&#10;a two-person dialog between the writer and an imagined reader: we treat&#10;sentences from the article as utterances spoken by the writer, and then use a&#10;dialog inpainter to predict what the imagined reader asked or said in between&#10;each of the writer's utterances. By applying this approach to passages from&#10;Wikipedia and the web, we produce WikiDialog and WebDialog, two datasets&#10;totalling 19 million diverse information-seeking dialogs -- 1,000x larger than&#10;the largest existing ConvQA dataset. Furthermore, human raters judge the answer&#10;adequacy and conversationality of WikiDialog to be as good or better than&#10;existing manually-collected datasets. Using our inpainted data to pre-train&#10;ConvQA retrieval systems, we significantly advance state-of-the-art across&#10;three benchmarks (QReCC, OR-QuAC, TREC CAsT) yielding up to 40% relative gains&#10;on standard evaluation metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08342" label="2112.08342">
        <attvalues>
          <attvalue for="0" value="DG2: Data Augmentation Through Document Grounded Dialogue Generation" />
          <attvalue for="1" value="  Collecting data for training dialog systems can be extremely expensive due to&#10;the involvement of human participants and need for extensive annotation.&#10;Especially in document-grounded dialog systems, human experts need to carefully&#10;read the unstructured documents to answer the users' questions. As a result,&#10;existing document-grounded dialog datasets are relatively small-scale and&#10;obstruct the effective training of dialogue systems. In this paper, we propose&#10;an automatic data augmentation technique grounded on documents through a&#10;generative dialogue model. The dialogue model consists of a user bot and agent&#10;bot that can synthesize diverse dialogues given an input document, which are&#10;then used to train a downstream model. When supplementing the original dataset,&#10;our method achieves significant improvement over traditional data augmentation&#10;methods. We also achieve great performance in the low-resource setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.05590" label="2205.05590">
        <attvalues>
          <attvalue for="0" value="A neural prosody encoder for end-ro-end dialogue act classification" />
          <attvalue for="1" value="  Dialogue act classification (DAC) is a critical task for spoken language&#10;understanding in dialogue systems. Prosodic features such as energy and pitch&#10;have been shown to be useful for DAC. Despite their importance, little research&#10;has explored neural approaches to integrate prosodic features into end-to-end&#10;(E2E) DAC models which infer dialogue acts directly from audio signals. In this&#10;work, we propose an E2E neural architecture that takes into account the need&#10;for characterizing prosodic phenomena co-occurring at different levels inside&#10;an utterance. A novel part of this architecture is a learnable gating mechanism&#10;that assesses the importance of prosodic features and selectively retains core&#10;information necessary for E2E DAC. Our proposed model improves DAC accuracy by&#10;1.07% absolute across three publicly available benchmark datasets.&#10;" />
          <attvalue for="2" value="&#10;Dialogue acts (DAs) are speech acts that represent intentions behind a user's request to achieve a conversational goal \cite{austin1962how}. Dialogue act classification (DAC) models aim to discriminate speech act units such as statement, question, backchannel, and agreement. For instance, when a user says ``yes'', DAC models {are used to} determine whether the user's intent is to agree with what the voice assistant system has said (DA: agreement) or to signal that the user is paying attention to the system (DA: backchannel). &#10;&#10;{Recent years have seen significant success in applying deep learning approaches to DAC \cite{tran2017a, ji2016a, ortega2018lexico, shen2016neural, dang2020endtoend, he2018exploring}. These approaches use either transcripts~\cite{tran2017a, ji2016a, shen2016neural} or a combination of transcript and audio~\cite{he2018exploring, ortega2018lexico, julia2010dialog} to predict DA. However, relying on transcripts has three limitations: First, transcripts are not always available for a spoken dialogue system. Second, collecting oracle transcripts is expensive. Third, errors introduced from transcribing audio have been shown to decrease the performance of DAC significantly~\cite{Tran_CNN}. More recently, \cite{dang2020endtoend} introduced an end-to-end (E2E) DAC approach, where DAs are directly inferred from audio signals. This approach can address the limitations of using transcripts as the inputs. Yet, how to effectively model audio signals for E2E DAC is underexplored. }&#10;&#10;{Prosody comprises the intonation, rhythm, and stress of spoken language. As highlighted in~\cite{wallbridge2021it}, it represents the non-lexical channel that serves a fundamental role in speech communication among humans. It captures the complex linguistic and semantic contents embedded in spoken language beyond words and their literal meanings \cite{dahan2015prosody}. At the syllable/word level, stressing on different syllables of a word can lead to different meanings (e.g., REcord vs. reCORD) \cite{ward2019prosodic}. At the sentence level, overall intonational contour contributes to characterize speaker’s intention and communicative meanings (e.g., agreement vs. backchannel: yes vs. yes?)~\cite{honda2004physiological}. This highly intuitive linguistic phenomena inspired many works to explore ways to incorporate prosodic {features} for DAC \cite{shriberg1998can, zimmermann2009joint, quarteroni2011simultaneous, ang2005automatic, stolcke2000dialogue, arsikere2016novel, Tran_CNN}.} Early research primarily focused on conventional cumulative-statistics~\cite{shriberg1998can} and traditional machine learning approaches~\cite {arsikere2016novel, zimmermann2009joint}. Of note, \cite{arsikere2016novel} found that the location of the maximum F0 occurrence can effectively distinguish between questions and statements.\cite{gravano2007on,benus2007the} also show that a pitch contour rises on the second syllable of words such as okay can mark a topic shift as well as conveying affirmation or a backchannel~\cite{gravano2007on}. Recently, neural modeling has emerged as a promising yet understudied approach to encode prosodic {features}. For instance, convolutional neural networks are used in \cite{Tran_CNN} {to model sentence-level prosodic {features}. However, little research has focused on neural approaches that fuse prosodic and spectral characteristics of audio signals at both syllable/word level and sentence level. Moreover, how best to integrate prosodic {features} for E2E DAC remains unexplored.}&#10;&#10;In this work, we propose a novel E2E neural architecture that take{s} into account this need of characterizing prosodic phenomena co-occurring at different levels inside an utterance. An essential part of this architecture is a learnable gating mechanism that assesses the importance of prosodic {features} and selectively retains core {features} necessary for E2E DAC.&#10;We compare our proposed model with previous E2E DAC models \cite{ortega2018lexico, he2018exploring} {that only use spectral-based audio features}. {The results show that our models outperform the reference ones. Further,} we compare our neural prosody encoder with {the state-of-the-art prosody neural encoder} \cite{Tran_CNN} on three public benchmark datasets: DSTC2~\cite{second}, and DSTC3~\cite{third}, and Switchboard~\cite{godfrey1992switchboard}. {We show that} our proposed model outperforms \cite{Tran_CNN} on all these datasets. We also examine the effects of the gating mechanism and different prosodic features on our proposed model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Prosodic Features, Dialogue Systems, Artificial Intelligence, Signal Processing, Neural Architecture" />
        </attvalues>
      </node>
      <node id="1810.07455" label="1810.07455">
        <attvalues>
          <attvalue for="0" value="Exploring Textual and Speech information in Dialogue Act Classification&#10;  with Speaker Domain Adaptation" />
          <attvalue for="1" value="  In spite of the recent success of Dialogue Act (DA) classification, the&#10;majority of prior works focus on text-based classification with oracle&#10;transcriptions, i.e. human transcriptions, instead of Automatic Speech&#10;Recognition (ASR)'s transcriptions. In spoken dialog systems, however, the&#10;agent would only have access to noisy ASR transcriptions, which may further&#10;suffer performance degradation due to domain shift. In this paper, we explore&#10;the effectiveness of using both acoustic and textual signals, either oracle or&#10;ASR transcriptions, and investigate speaker domain adaptation for DA&#10;classification. Our multimodal model proves to be superior to the unimodal&#10;models, particularly when the oracle transcriptions are not available. We also&#10;propose an effective method for speaker domain adaptation, which achieves&#10;competitive results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.11419" label="2004.11419">
        <attvalues>
          <attvalue for="0" value="End-to-end speech-to-dialog-act recognition" />
          <attvalue for="1" value="  Spoken language understanding, which extracts intents and/or semantic&#10;concepts in utterances, is conventionally formulated as a post-processing of&#10;automatic speech recognition. It is usually trained with oracle transcripts,&#10;but needs to deal with errors by ASR. Moreover, there are acoustic features&#10;which are related with intents but not represented with the transcripts. In&#10;this paper, we present an end-to-end model which directly converts speech into&#10;dialog acts without the deterministic transcription process. In the proposed&#10;model, the dialog act recognition network is conjunct with an acoustic-to-word&#10;ASR model at its latent layer before the softmax layer, which provides a&#10;distributed representation of word-level ASR decoding information. Then, the&#10;entire network is fine-tuned in an end-to-end manner. This allows for stable&#10;training as well as robustness against ASR errors. The model is further&#10;extended to conduct DA segmentation jointly. Evaluations with the Switchboard&#10;corpus demonstrate that the proposed method significantly improves dialog act&#10;recognition accuracy from the conventional pipeline framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00260" label="2105.00260">
        <attvalues>
          <attvalue for="0" value="It's not what you said, it's how you said it: discriminative perception&#10;  of speech as a multichannel communication system" />
          <attvalue for="1" value="  People convey information extremely effectively through spoken interaction&#10;using multiple channels of information transmission: the lexical channel of&#10;what is said, and the non-lexical channel of how it is said. We propose&#10;studying human perception of spoken communication as a means to better&#10;understand how information is encoded across these channels, focusing on the&#10;question 'What characteristics of communicative context affect listener's&#10;expectations of speech?'. To investigate this, we present a novel behavioural&#10;task testing whether listeners can discriminate between the true utterance in a&#10;dialogue and utterances sampled from other contexts with the same lexical&#10;content. We characterize how perception - and subsequent discriminative&#10;capability - is affected by different degrees of additional contextual&#10;information across both the lexical and non-lexical channel of speech. Results&#10;demonstrate that people can effectively discriminate between different prosodic&#10;realisations, that non-lexical context is informative, and that this channel&#10;provides more salient information than the lexical channel, highlighting the&#10;importance of the non-lexical channel in spoken interaction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.00831" label="1803.00831">
        <attvalues>
          <attvalue for="0" value="Lexico-acoustic Neural-based Models for Dialog Act Classification" />
          <attvalue for="1" value="  Recent works have proposed neural models for dialog act classification in&#10;spoken dialogs. However, they have not explored the role and the usefulness of&#10;acoustic information. We propose a neural model that processes both lexical and&#10;acoustic features for classification. Our results on two benchmark datasets&#10;reveal that acoustic features are helpful in improving the overall accuracy.&#10;Finally, a deeper analysis shows that acoustic features are valuable in three&#10;cases: when a dialog act has sufficient data, when lexical information is&#10;limited and when strong lexical cues are not present.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.14630" label="2401.14630">
        <attvalues>
          <attvalue for="0" value="An Empirical Investigation of Domain Adaptation Ability for Chinese&#10;  Spelling Check Models" />
          <attvalue for="1" value="  Chinese Spelling Check (CSC) is a meaningful task in the area of Natural&#10;Language Processing (NLP) which aims at detecting spelling errors in Chinese&#10;texts and then correcting these errors. However, CSC models are based on&#10;pretrained language models, which are trained on a general corpus.&#10;Consequently, their performance may drop when confronted with downstream tasks&#10;involving domain-specific terms. In this paper, we conduct a thorough&#10;evaluation about the domain adaption ability of various typical CSC models by&#10;building three new datasets encompassing rich domain-specific terms from the&#10;financial, medical, and legal domains. Then we conduct empirical investigations&#10;in the corresponding domain-specific test datasets to ascertain the&#10;cross-domain adaptation ability of several typical CSC models. We also test the&#10;performance of the popular large language model ChatGPT. As shown in our&#10;experiments, the performances of the CSC models drop significantly in the new&#10;domains.&#10;" />
          <attvalue for="2" value="&#10;&#10;Chinese Spelling Check (CSC) aims to detect and correct spelling mistakes in Chinese sentences. It is a challenging task in Chinese Natural Language Processing (NLP) and holds substantial importance for various downstream applications, including Optical Character Recognition (OCR)~\cite{afli-etal-2016-using}, Automatic Speech Recognition (ASR)~\cite{gao-etal-2010-large}, AI Writing Assistant~\cite{DBLP:journals/corr/abs-2208-01815}, and numerous other tasks that contain Chinese texts. By employing CSC in downstream tasks, a considerable reduction in common errors caused by both machines and humans can be achieved.&#10;&#10;As spelling mistakes in Chinese texts need to be detected in a specific sentence, effective detection and correction of such errors rely on context comprehension of CSC models. Fortunately, great success has been achieved in CSC since pretrained language models became the backbone model. However, the training data of BERT mainly comes from BooksCorpus and Wikipedia\cite{devlin-etal-2019-bert}, while the corpus of downstream tasks vary differently. This mismatch will lead to a considerable decline in performance when testing the CSC model on domain-specific terms. This phenomenon is exemplified in Figure~\ref{fig}, where CSC is conducted using CSC models FASPell~\cite{hong-etal-2019-FASPell}, Soft-Masked BERT~\cite{Softmasked}, and SpellGCN~\cite{spellgcn} across various domains. The results demonstrate that all those three models can obtain correct detection and correction results on the general domain, while performance collapses on medical and legal domains. The models often replacing them with commonly used terms rather than specific terms.&#10;&#10;Considering the practical scenarios where input texts may originate from diverse domains, it is crucial and necessary to evaluate the domain adaptation ability of current typical CSC models. As there is no public domain-specific CSC datasets available, it becomes imperative to develop novel datasets to quantitatively assess the domain adaptation ability of classical models. Specifically, we focus on three distinct domains: financial, medical, and legal domain. After corpora collection, we apply predefined rules to introduce errors into normal sentences. Furthermore, we deliberately construct sentences with two error positions, thereby enhancing the challenges of detecting and correcting all the errors.&#10;&#10;Furthermore, it is important to note that several existing models~\cite{hong-etal-2019-FASPell,Softmasked,spellgcn,Lv_2022} rely on supervised learning approaches which are trained on datasets with limited coverage of terminologies from specialized domains. As a result, these models may have poor performance when confronted with out-of-domain tasks. This raises concerns about the ability of task-specific models to maintain the domain adaptation capability inherent in the original pretrained language models. To verify this conjecture, we also conduct cross-domain evaluation on some unsupervised CSC methods such as uChecker~\cite{li-2022-uchecker} which can preserve the knowledge of the original pretrained language models as much as possible. With the construction of the aforementioned three domain-specific datasets, we systematically evaluate the typical CSC models mentioned above on these datasets and present the corresponding results.&#10;&#10;There are works such as ECSPell~\cite{Lv_2022} that raises the issue of the domain adaptation problem in the CSC area, and evaluates some state-of-the-art models on the human-annotated domain datasets. However, constructing a dataset through human annotation is a time-consuming process, and the proposed approach incorporates external domain-specific knowledge to filter output. While this work serves as an inspiration for our research, it does not consider unsupervised methods and lacks a comprehensive analysis of the compared models.&#10;&#10;We are also interested in the performance of large language models in CSC. To this end, we choose ChatGPT as the example and use prompts to make evaluation of the performance of large language models.&#10;&#10;In summary, our contributions are as follows: &#10;\begin{itemize}[topsep=0pt]&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Domain Adaptation, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2208.01815" label="2208.01815">
        <attvalues>
          <attvalue for="0" value="Effidit: Your AI Writing Assistant" />
          <attvalue for="1" value="  In this technical report, we introduce Effidit (Efficient and Intelligent&#10;Editing), a digital writing assistant that facilitates users to write&#10;higher-quality text more efficiently by using artificial intelligence (AI)&#10;technologies. Previous writing assistants typically provide the function of&#10;error checking (to detect and correct spelling and grammatical errors) and&#10;limited text-rewriting functionality. With the emergence of large-scale neural&#10;language models, some systems support automatically completing a sentence or a&#10;paragraph. In Effidit, we significantly expand the capacities of a writing&#10;assistant by providing functions in five categories: text completion, error&#10;checking, text polishing, keywords to sentences (K2S), and cloud input methods&#10;(cloud IME). In the text completion category, Effidit supports generation-based&#10;sentence completion, retrieval-based sentence completion, and phrase&#10;completion. In contrast, many other writing assistants so far only provide one&#10;or two of the three functions. For text polishing, we have three functions:&#10;(context-aware) phrase polishing, sentence paraphrasing, and sentence&#10;expansion, whereas many other writing assistants often support one or two&#10;functions in this category. The main contents of this report include major&#10;modules of Effidit, methods for implementing these modules, and evaluation&#10;results of some key methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14166" label="2004.14166">
        <attvalues>
          <attvalue for="0" value="SpellGCN: Incorporating Phonological and Visual Similarities into&#10;  Language Models for Chinese Spelling Check" />
          <attvalue for="1" value="  Chinese Spelling Check (CSC) is a task to detect and correct spelling errors&#10;in Chinese natural language. Existing methods have made attempts to incorporate&#10;the similarity knowledge between Chinese characters. However, they take the&#10;similarity knowledge as either an external input resource or just heuristic&#10;rules. This paper proposes to incorporate phonological and visual similarity&#10;knowledge into language models for CSC via a specialized graph convolutional&#10;network (SpellGCN). The model builds a graph over the characters, and SpellGCN&#10;is learned to map this graph into a set of inter-dependent character&#10;classifiers. These classifiers are applied to the representations extracted by&#10;another network, such as BERT, enabling the whole network to be end-to-end&#10;trainable. Experiments (The dataset and all code for this paper are available&#10;at https://github.com/ACL2020SpellGCN/SpellGCN) are conducted on three&#10;human-annotated datasets. Our method achieves superior performance against&#10;previous models by a large margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.10929" label="2203.10929">
        <attvalues>
          <attvalue for="0" value="General and Domain Adaptive Chinese Spelling Check with Error Consistent&#10;  Pretraining" />
          <attvalue for="1" value="  The lack of label data is one of the significant bottlenecks for Chinese&#10;Spelling Check (CSC). Existing researches use the method of automatic&#10;generation by exploiting unlabeled data to expand the supervised corpus.&#10;However, there is a big gap between the real input scenario and automatic&#10;generated corpus. Thus, we develop a competitive general speller ECSpell which&#10;adopts the Error Consistent masking strategy to create data for pretraining.&#10;This error consistency masking strategy is used to specify the error types of&#10;automatically generated sentences which is consistent with real scene. The&#10;experimental result indicates our model outperforms previous state-of-the-art&#10;models on the general benchmark. Moreover, spellers often work within a&#10;particular domain in real life. Due to lots of uncommon domain terms,&#10;experiments on our built domain specific datasets show that general models&#10;perform terribly. Inspired by the common practice of input methods, we propose&#10;to add an alterable user dictionary to handle the zero-shot domain adaption&#10;problem. Specifically, we attach a User Dictionary guided inference module (UD)&#10;to a general token classification based speller. Our experiments demonstrate&#10;that ECSpell$^{UD}$, namely ECSpell combined with UD, surpasses all the other&#10;baselines largely, even approaching the performance on the general benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.06018" label="2307.06018">
        <attvalues>
          <attvalue for="0" value="PolyLM: An Open Source Polyglot Large Language Model" />
          <attvalue for="1" value="  Large language models (LLMs) demonstrate remarkable ability to comprehend,&#10;reason, and generate following nature language instructions. However, the&#10;development of LLMs has been primarily focused on high-resource languages, such&#10;as English, thereby limiting their applicability and research in other&#10;languages. Consequently, we present PolyLM, a multilingual LLM trained on 640&#10;billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its&#10;multilingual capabilities, we 1) integrate bilingual data into training data;&#10;and 2) adopt a curriculum learning strategy that increases the proportion of&#10;non-English data from 30% in the first stage to 60% in the final stage during&#10;pre-training. Further, we propose a multilingual self-instruct method which&#10;automatically generates 132.7K diverse multilingual instructions for model&#10;fine-tuning. To assess the model's performance, we collect several existing&#10;multilingual tasks, including multilingual understanding, question answering,&#10;generation, and translation. Extensive experiments show that PolyLM surpasses&#10;other open-source models such as LLaMA and BLOOM on multilingual tasks while&#10;maintaining comparable performance in English. Our models, alone with the&#10;instruction data and multilingual benchmark, are available at:&#10;\url{https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation}.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) are trained on vast amounts of data in a self-supervised fashion, which has shown promising performance in a variety of zero-shot and few-shot tasks \cite{gpt3,Chowdhery2022PaLMSL}. Fine-tuning these models on a diverse set of tasks allows them to handle unseen tasks following natural language instructions~\cite{ouyang2022training,longpre2023flan,taori2023alpaca,gpt4all}. These properties have attracted significant attention from the Artificial Intelligence community and offering a potential path towards artificial general intelligence. Unfortunately, most LLMs are developed for English, such as LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, Chinchilla~\cite{hoffmann2022chinchilla}, OPT~\cite{zhang2022opt}. A main reason stems from recent findings that model performance is closely related to the scale of the training dataset~\cite{kaplan2020scaling,rae2021scaling,biderman2023pythia,touvron2023llama}, leading to predominant focus on resource-rich languages, particularly English.&#10;&#10;The relatively high concentration of studies on English limits the research and usage of LLMs in other languages. &#10;For instance, Thai and Indonesian have over 300 million (M) speakers, yet the size of these two languages in common crawl-based dataset such as mC4~\cite{xue2020mt5} is only 80 billion (B) tokens, comprising a mere 3\% of the English data. Due to the insufficient high-quality internet data, LLM capabilities on low-resource languages fail to be easily improved through expanding their data size like English~\cite{kaplan2020scaling,rae2021scaling,biderman2023pythia}. As a result, existing open-source LLMs such as XGLM~\cite{lin-etal-2022-xglm}, BLOOM~\cite{scao2022bloom}, and LLaMA~\cite{touvron2023llama} perform relatively poor on these languages, some of which are entirely overlooked. It is crucial to explore multilingual LLMs to bridge this gap and achieve academic and social significance. &#10;&#10;Our goal is to enhance the exploration and utilization of LLMs for non-native English speakers. In this work, we fill three significant gaps in this field: 1) the absence of an open-source multilingual LLM; 2) the inadequate availability of multilingual instruction data; and 3) the lack of a unified evaluation benchmark for multilingual settings. &#10;&#10;Concretely, we first develop an open-source multilingual LLM from scratch, called Polyglot Large Language Model (PolyLM, Section \ref{sec2}). Contrary to existing open-source multilingual LLMs that lack 13B model, we release PolyLM-13B and PolyLM-1.7B to facilitate its usage. To construct PolyLM, we leverage a massive dataset of 640B tokens, culled from publicly available sources such as Wikipedia, mC4~\cite{xue2020mt5}, CC-100~\cite{Conneau2019UnsupervisedCR}. This dataset contains over 30\% of non-English languages, specifically covering 18 of the most commonly spoken languages.&#10;To alleviate the problem of insufficient data for low-resource languages, we propose a curriculum learning strategy. The training schedule increases the amount of data available for training in English during the initial phases, then ramping up the ratio of high-quality, low-resource languages as training progresses. We expect the method to enable the transfer of general knowledge from English to other languages, leading to significant improvements in overall performance.&#10;&#10;In light of the supervised fine-tuning (SFT) stage, we construct a multilingual instruction dataset termed {MultiAlpaca}~ with 132,701 samples (Section \ref{sec3}). &#10;At present, there is a dearth of high-quality open-source multilingual SFT datasets. &#10;On the one hand, extant multilingual SFT datasets, e.g. xP3-MT~\cite{muennighoff2022crosslingual}, are acquired via machine translation, which potentially yields a style of translationese, a lack of cultural nuances, as well as translation errors. On the other hands, manually annotating instructions is a laborious and costly process that does not lend itself well to the incorporation of creative flourishes. &#10;Drawing inspiration from recent advances in self-instruct~\cite{wang2022self,taori2023alpaca}, we devise a multilingual self-instruct method to automatically generate instruction data. Utilizing 175 English seeds as a starting point, our method leverage multilingual seed translation, instruction generation, and filtering mechanisms to deliver high quality multilingual instruction data.&#10;&#10;In order to assess the multilingual capabilities of LLM, we curate a benchmark derived from existing multilingual tasks (Section \ref{sec4}), including QA~\cite{tydiqa}, understanding~\cite{Conneau2019UnsupervisedCR,Yang2019PAWSXAC,tikhonov2021heads,ponti2020xcopa}, generation~\cite{Chen2021MTGAB}, and cross-lingual machine translation~\cite{barrault-etal-2020-findings}. The benchmark is constructed with meticulously prompting and finally covers 10 tasks across 15 languages. Extensive experiments (Section \ref{sec5}) demonstrate that our pretrained model outperforms open-source models of comparable model size (e.g. BLOOM, LLaMA, etc.) in non-English languages. Through in-depth analyses, we identify finding that the proposed curriculum training strategy boosts the multilingual performance while maintain the English proficiency. &#10;In addition, the use of multilingual instruction data markedly enhances the ability of PolyLM to tackle multilingual zero-shot tasks. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Multilingual Language Models, Large Language Model Training, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2106.12066" label="2106.12066">
        <attvalues>
          <attvalue for="0" value="It's All in the Heads: Using Attention Heads as a Baseline for&#10;  Cross-Lingual Transfer in Commonsense Reasoning" />
          <attvalue for="1" value="  Commonsense reasoning is one of the key problems in natural language&#10;processing, but the relative scarcity of labeled data holds back the progress&#10;for languages other than English. Pretrained cross-lingual models are a source&#10;of powerful language-agnostic representations, yet their inherent reasoning&#10;capabilities are still actively studied. In this work, we design a simple&#10;approach to commonsense reasoning which trains a linear classifier with weights&#10;of multi-head attention as features. To evaluate this approach, we create a&#10;multilingual Winograd Schema corpus by processing several datasets from prior&#10;work within a standardized pipeline and measure cross-lingual generalization&#10;ability in terms of out-of-sample performance. The method performs&#10;competitively with recent supervised and unsupervised approaches for&#10;commonsense reasoning, even when applied to other languages in a zero-shot&#10;manner. Also, we demonstrate that most of the performance is given by the same&#10;small subset of attention heads for all studied languages, which provides&#10;evidence of universal reasoning capabilities in multilingual encoders.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.09168" label="2102.09168">
        <attvalues>
          <attvalue for="0" value="Gaussian Kernelized Self-Attention for Long Sequence Data and Its&#10;  Application to CTC-based Speech Recognition" />
          <attvalue for="1" value="  Self-attention (SA) based models have recently achieved significant&#10;performance improvements in hybrid and end-to-end automatic speech recognition&#10;(ASR) systems owing to their flexible context modeling capability. However, it&#10;is also known that the accuracy degrades when applying SA to long sequence&#10;data. This is mainly due to the length mismatch between the inference and&#10;training data because the training data are usually divided into short segments&#10;for efficient training. To mitigate this mismatch, we propose a new&#10;architecture, which is a variant of the Gaussian kernel, which itself is a&#10;shift-invariant kernel. First, we mathematically demonstrate that&#10;self-attention with shared weight parameters for queries and keys is equivalent&#10;to a normalized kernel function. By replacing this kernel function with the&#10;proposed Gaussian kernel, the architecture becomes completely shift-invariant&#10;with the relative position information embedded using a frame indexing&#10;technique. The proposed Gaussian kernelized SA was applied to connectionist&#10;temporal classification (CTC) based ASR. An experimental evaluation with the&#10;Corpus of Spontaneous Japanese (CSJ) and TEDLIUM 3 benchmarks shows that the&#10;proposed SA achieves a significant improvement in accuracy (e.g., from 24.0%&#10;WER to 6.0% in CSJ) in long sequence data without any windowing techniques.&#10;" />
          <attvalue for="2" value="&#10;In recent years, automatic speech recognition (ASR) using self-attention (SA) \cite{vaswani2017attention} has attracted considerable attention.&#10;Both transformer-based speech recognition \cite{dong2018speech,karita2019comparative,mohamed2019transformers,zeyer2019comparison,chang2020end} and hybrid \cite{povey2018time,wang2019transformer} and connectionist temporal classification (CTC) \cite{pham2019very,salazar2019self} models have shown a high recognition performance with SA.&#10;The SA network has a mathematically simple structure by fully using matrix-vector based operations designed for an efficient parallel computation.&#10;Thus, the recurrent neural network (RNN) based architecture has been replaced with the SA network because of the efficient computation property and its high performance.&#10;&#10;However, SA is unsuitable for decoding long sequence data because it has a high computational complexity on the order of the square of the sequence length.&#10;In addition, the recognition accuracy degrades in long utterances owing to its excessive flexibility in context modeling.&#10;In this paper, we focus on the problems of the accuracy degradation in long sequence data.&#10;In general, self-attention requires dividing original long recordings into short segments during training for efficient GPU computing.&#10;This leads to a mismatch between the sequence lengths of the training and test data, resulting in a performance degradation.&#10;&#10;To solve this problem, several studies have been proposed.&#10;Masking \cite{sperber2018self} limits the range of self-attention by using a Gaussian window, whereas relative positional encoding \cite{shaw2018self,pham2020relative} uses relative embedding in a self-attention architecture to eliminate the effect of the length mismatch.&#10;However, masking does not take into account the correlation between input features and relative distance.&#10;In addition, the relative positional encoding does not limit the attention to the neighborhood in a mathematical.&#10;&#10;Inspired by the mathematical expression based on the shared-QK attention used in Reformer \cite{kitaev2020reformer}, in this paper, yet another self-attention reformulation based on a Gaussian kernel is proposed.&#10;First, we mathematically demonstrate that the linear layers and softmax functions in the shared-QK attention can be represented as normalized kernel functions, similarly to \cite{gao2016compact} and \cite{raginsky2009locality}, interpreting bilinear pooling as a kernel function.&#10;These kernel functions are replaced with a Gaussian kernel and thus we call our model Gaussian kernelized self-attention.&#10;Gaussian kernel, known as a radial basis function kernel, has several useful features and has been widely used with a support vector machine (SVM) \cite{kuo2013kernel,dahake2016speaker,shao2005wavelet, stadermann2004hybrid,ganapathiraju2004applications}.&#10;The Gaussian kernelization applied in our new formulation also provides a shift-invariance into the self-attention architecture.&#10;This shift-invariant property is a highly desirable property for controlling the relative position.&#10;To take advantage of this property, we propose concatenating the bare frame index to an input feature, which is called a frame indexing technique.&#10;&#10;To compare the differences in SA structure, this paper applies the proposed Gaussian kernelized SA to CTC-based ASR because the decoder network of a CTC is rather simple compared with other end-to-end architectures, and we can purely evaluate the effectiveness between the proposed and conventional SA methods.&#10;An experimental evaluation shows that our proposed SA with frame indexing achieved a significant improvement in the long sequence data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Learning Architectures, Automatic Speech Recognition, Self-Attention Models, Signal Processing, Mathematics" />
        </attvalues>
      </node>
      <node id="1909.06317" label="1909.06317">
        <attvalues>
          <attvalue for="0" value="A Comparative Study on Transformer vs RNN in Speech Applications" />
          <attvalue for="1" value="  Sequence-to-sequence models have been widely used in end-to-end speech&#10;processing, for example, automatic speech recognition (ASR), speech translation&#10;(ST), and text-to-speech (TTS). This paper focuses on an emergent&#10;sequence-to-sequence model called Transformer, which achieves state-of-the-art&#10;performance in neural machine translation and other natural language processing&#10;applications. We undertook intensive studies in which we experimentally&#10;compared and analyzed Transformer and conventional recurrent neural networks&#10;(RNN) in a total of 15 ASR, one multilingual ASR, one ST, and two TTS&#10;benchmarks. Our experiments revealed various training tips and significant&#10;performance benefits obtained with Transformer for each task including the&#10;surprising superiority of Transformer in 13/15 ASR benchmarks in comparison&#10;with RNN. We are preparing to release Kaldi-style reproducible recipes using&#10;open source and publicly available datasets for all the ASR, ST, and TTS tasks&#10;for the community to succeed our exciting outcomes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.09799" label="1910.09799">
        <attvalues>
          <attvalue for="0" value="Transformer-based Acoustic Modeling for Hybrid Speech Recognition" />
          <attvalue for="1" value="  We propose and evaluate transformer-based acoustic models (AMs) for hybrid&#10;speech recognition. Several modeling choices are discussed in this work,&#10;including various positional embedding methods and an iterated loss to enable&#10;training deep transformers. We also present a preliminary study of using&#10;limited right context in transformer models, which makes it possible for&#10;streaming applications. We demonstrate that on the widely used Librispeech&#10;benchmark, our transformer-based AM outperforms the best published hybrid&#10;result by 19% to 26% relative when the standard n-gram language model (LM) is&#10;used. Combined with neural network LM for rescoring, our proposed approach&#10;achieves state-of-the-art results on Librispeech. Our findings are also&#10;confirmed on a much larger internal dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.13377" label="1904.13377">
        <attvalues>
          <attvalue for="0" value="Very Deep Self-Attention Networks for End-to-End Speech Recognition" />
          <attvalue for="1" value="  Recently, end-to-end sequence-to-sequence models for speech recognition have&#10;gained significant interest in the research community. While previous&#10;architecture choices revolve around time-delay neural networks (TDNN) and long&#10;short-term memory (LSTM) recurrent neural networks, we propose to use&#10;self-attention via the Transformer architecture as an alternative. Our analysis&#10;shows that deep Transformer networks with high learning capacity are able to&#10;exceed performance from previous end-to-end approaches and even match the&#10;conventional hybrid systems. Moreover, we trained very deep models with up to&#10;48 Transformer layers for both encoder and decoders combined with stochastic&#10;residual connections, which greatly improve generalizability and training&#10;efficiency. The resulting models outperform all previous end-to-end ASR&#10;approaches on the Switchboard benchmark. An ensemble of these models achieve&#10;9.9% and 17.7% WER on Switchboard and CallHome test sets respectively. This&#10;finding brings our end-to-end models to competitive levels with previous hybrid&#10;systems. Further, with model ensembling the Transformers can outperform certain&#10;hybrid systems, which are more complicated in terms of both structure and&#10;training procedure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.10055" label="1901.10055">
        <attvalues>
          <attvalue for="0" value="Self-Attention Networks for Connectionist Temporal Classification in&#10;  Speech Recognition" />
          <attvalue for="1" value="  The success of self-attention in NLP has led to recent applications in&#10;end-to-end encoder-decoder architectures for speech recognition. Separately,&#10;connectionist temporal classification (CTC) has matured as an alignment-free,&#10;non-autoregressive approach to sequence transduction, either by itself or in&#10;various multitask and decoding frameworks. We propose SAN-CTC, a deep, fully&#10;self-attentional network for CTC, and show it is tractable and competitive for&#10;end-to-end speech recognition. SAN-CTC trains quickly and outperforms existing&#10;CTC models and most encoder-decoder models, with character error rates (CERs)&#10;of 4.7% in 1 day on WSJ eval92 and 2.8% in 1 week on LibriSpeech test-clean,&#10;with a fixed architecture and one GPU. Similar improvements hold for WERs after&#10;LM decoding. We motivate the architecture for speech, evaluate position and&#10;downsampling approaches, and explore how label alphabets (character, phoneme,&#10;subword) affect attention heads and performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.09519" label="1803.09519">
        <attvalues>
          <attvalue for="0" value="Self-Attentional Acoustic Models" />
          <attvalue for="1" value="  Self-attention is a method of encoding sequences of vectors by relating these&#10;vectors to each-other based on pairwise similarities. These models have&#10;recently shown promising results for modeling discrete sequences, but they are&#10;non-trivial to apply to acoustic modeling due to computational and modeling&#10;issues. In this paper, we apply self-attention to acoustic modeling, proposing&#10;several improvements to mitigate these issues: First, self-attention memory&#10;grows quadratically in the sequence length, which we address through a&#10;downsampling technique. Second, we find that previous approaches to incorporate&#10;position information into the model are unsuitable and explore other&#10;representations and hybrid models to this end. Third, to stress the importance&#10;of local context in the acoustic signal, we propose a Gaussian biasing approach&#10;that allows explicit control over the context range. Experiments find that our&#10;model approaches a strong baseline based on LSTMs with network-in-network&#10;connections while being much faster to compute. Besides speed, we find that&#10;interpretability is a strength of self-attentional acoustic models, and&#10;demonstrate that self-attention heads learn a linguistically plausible division&#10;of labor.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09940" label="2005.09940">
        <attvalues>
          <attvalue for="0" value="Relative Positional Encoding for Speech Recognition and Direct&#10;  Translation" />
          <attvalue for="1" value="  Transformer models are powerful sequence-to-sequence architectures that are&#10;capable of directly mapping speech inputs to transcriptions or translations.&#10;However, the mechanism for modeling positions in this model was tailored for&#10;text modeling, and thus is less ideal for acoustic inputs. In this work, we&#10;adapt the relative position encoding scheme to the Speech Transformer, where&#10;the key addition is relative distance between input states in the&#10;self-attention network. As a result, the network can better adapt to the&#10;variable distributions present in speech data. Our experiments show that our&#10;resulting model achieves the best recognition result on the Switchboard&#10;benchmark in the non-augmentation condition, and the best published result in&#10;the MuST-C speech translation benchmark. We also show that this model is able&#10;to better utilize synthetic data than the Transformer, and adapts better to&#10;variable sentence segmentation quality for speech translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.04797" label="2403.04797">
        <attvalues>
          <attvalue for="0" value="Found in the Middle: How Language Models Use Long Contexts Better via&#10;  Plug-and-Play Positional Encoding" />
          <attvalue for="1" value="  This paper aims to overcome the &quot;lost-in-the-middle&quot; challenge of large&#10;language models (LLMs). While recent advancements have successfully enabled&#10;LLMs to perform stable language modeling with up to 4 million tokens, the&#10;persistent difficulty faced by most LLMs in identifying relevant information&#10;situated in the middle of the context has not been adequately tackled. To&#10;address this problem, this paper introduces Multi-scale Positional Encoding&#10;(Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the&#10;capacity of LLMs to handle the relevant information located in the middle of&#10;the context, without fine-tuning or introducing any additional overhead. Ms-PoE&#10;leverages the position indice rescaling to relieve the long-term decay effect&#10;introduced by RoPE, while meticulously assigning distinct scaling ratios to&#10;different attention heads to preserve essential knowledge learned during the&#10;pre-training step, forming a multi-scale context fusion from short to long&#10;distance. Extensive experiments with a wide range of LLMs demonstrate the&#10;efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of&#10;up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are&#10;available at https://github.com/VITA-Group/Ms-PoE.&#10;" />
          <attvalue for="2" value="&#10;&#10;Effective long-sequence reasoning in large language models (LLMs) is crucial for a wide range of applications~\cite{re_longer_2022,li2023loogle}, from understanding extensive texts~\cite{tay2020long, kryscinski2021booksum} and managing day-long conversations~\cite{zhang2021summ,zhong2022dialoglm} to code generation~\cite{du2023classeval,zheng2023codegeex} and science discoveries~\cite{varadi2022alphafold,song2023deepspeed4science}. Recent system support advancements~\cite{dao2023flashattention2, jacobs2023deepspeed} have enabled training transformers for any $L$ sequence length even with $O(L^2)$ computational complexity. This is exemplified by models such as MPT~\cite{MosaicML2023Introducing} and Mistral~\cite{jiang2024mixtral} pre-trained with sequence lengths 16k and 32k respectively. &#10;&#10;Nevertheless, emerging research reveals the constrained efficacy of LLMs in managing tasks requiring long contextual understanding. Particularly, \cite{liu2023lost} demonstrated a substantial degradation in LLMs' performance when crucial information is positioned amidst a lengthy context, a phenomenon they refer to as ``lost-in-the-middle&quot;. One explanation is about the use of rotary positional embedding (RoPE) \cite{su2024roformer}, a prevalent positional encoding technique used in open-source LLMs. As a relative position embedding, RoPE incorporates a long-term decay property, predisposing the model to prioritize current/nearby tokens while paying less attention to further ones. \cite{xiao2023efficient} identified a surprising trend attributed to the Softmax operation where attention scores are disproportionately allocated into initial tokens, irrespective of their relevance to the language modeling task. Despite the presence of considerable redundancy in long-context inputs \cite{zhang2023h}, crucial information may be located across different positions. The inclination of LLMs to overlook the middle sectionpresents a challenge for their applications, particularly in the context of long-context reasoning. &#10;Several approaches successfully extend pre-trained LLMs with context up to extreme token length, either through sparse selection of crucial tokens during generation \cite{xiao2023efficient, zhang2023h, han2023lm} or by modifying positional encoding \cite{chen2023extending, jin2024llm}.&#10;Nevertheless, these approaches primarily aim to extend the context length of LLMs and, consequently, fall short in addressing the ``lost-in-the-middle'' problem when applied out-of-the-box. &#10;&#10;Efforts have been made to enhance LLMs' capacity to capture vital information located within the middle of the context.&#10;These include extra memory bank \cite{LongMem}, reordering the input context based on relevance \cite{peysakhovich2023attention,chen2023fortify}, enhancing the information searching and reflection ability via attention strengthening tasks~\cite{junqing2023never,xu2023retrieval}, splitting the input into short segments and applying short-text models~\cite{ivgi2023efficient}. For example, \cite{peysakhovich2023attention} empirically discovered that LLMs tend to emphasize more on the current window while still paying more attention to the relevant text than distracting content. They subsequently introduced ``attention sorting&quot; where the main idea is iteratively sorting documents based on their attention scores, such that critical information will likely be placed at the end, to fit the position-biased nature of RoPE. \cite{chen2023fortify} conducted parallel runs of LLMs with different RoPE angles, thereby mitigating the risk of overlooking crucial information through a weighted sum of the outputs. These approaches usually require additional memory or multiple inference runs, which can be expensive for LLMs. &#10;&#10;In this paper, we aim to address the ``lost-in-the-middle'' problem by reintroducing the concept of multi-scale features from computer vision into the context of Transformer-based LLMs. Multi-scale features, well-established in Inception-style models \cite{szegedy2015going, szegedy2016rethinking, guo2022segnext}, utilize parallel employment of kernels with different sizes to fuse multi-scale information, spanning short to long distances. &#10;Introducing multi-scale operations into LLMs intuitively can help compensate for crucial information located in the middle, which might be easily overlooked by full attention operation.&#10;Unlike modifying the attention module to form multi-scale attention, we choose to re-scale the indices of positional encoding. This decision is grounded not only in its effectiveness in easily adjusting the scale of the context window by simply changing the position indices \cite{chen2023extending} but also in the potential of down-scaling the position indices to relieve the long-term decay property introduced by RoPE. However, this approach was initially introduced to extend context windows, and its performance regarding the ``lost-in-the-middle'' problem remains uncertain for several reasons: (i) Indice re-scaling forces position embeddings of original context window to reside in a narrower region, leading to performance degradation in the original context window &#10;as shown in \cite{chen2023extending}. (ii) Uniformly applying the same scaling ratio throughout the entire model might be sub-optimal to preserve essential knowledge learned during pre-training; (ii) Fine-tuning is necessary for the original approach, albeit minimal. The impact without fine-tuning remains unknown.&#10;&#10;To this end, we systematically visit the position indices scaling regarding the ``lost-in-the-middle'' problem and counter-intuitively discover that it is possible to slightly mitigate the ``lost-in-the-middle'' issue if we carefully choose the scaling ratio to be around 1.5-2. Additionally, we observe that different attention heads exhibit varying sensitivity to the position shift of the relevant document. Some attention heads are ``position-aware'', consistently capturing relevant information even with position shifts, while others may occasionally capture position changes, and some heads are completely insensitive to position changes. This highlights the need to treat attention heads separately when re-scaling position indices.&#10;&#10;Contribution. Inspired by the above observations, we introduce Multi-scale Positional Encoding (Ms-PoE), a simple yet effective plug-and-play approach that can enhance the long-context reasoning capability of pre-trained LLMs without requiring fine-tuning or introducing any additional overhead. Ms-PoE meticulously assigns distinct scaling ratios to different attention heads, with the scaling factor monotonically increasing from ``position-aware'' heads to ``position-unaware'' heads. This enables us to improve long-context ability by re-scaling position indices to shorter values while preserving essential knowledge acquired during the pre-training phase. The efficacy of Ms-PoE is substantiated through extensive experiments. By simply re-scaling the indices of positional encoding, Ms-PoE consistently enhances the performance of various LLMs including Llama-2~\cite{touvron2023llama}, StableBeluga~\cite{StableBelugaModels} and Vicuna~\cite{vicuna2023} on the ZeroSCROLLS benchmark \cite{shaham2023zeroscrolls}, achieving a notable average accuracy gain of up to 3.8.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Contextual Information Retrieval, Deep Learning Architectures, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2311.04939" label="2311.04939">
        <attvalues>
          <attvalue for="0" value="LooGLE: Can Long-Context Language Models Understand Long Contexts?" />
          <attvalue for="1" value="  Large language models (LLMs), despite their impressive performance in various&#10;language tasks, are typically limited to processing texts within context-window&#10;size. This limitation has spurred significant research efforts to enhance LLMs'&#10;long-context understanding with high-quality long-sequence benchmarks. However,&#10;prior datasets in this regard suffer from shortcomings, such as short context&#10;length compared to the context window of modern LLMs; outdated documents that&#10;have data leakage problems; and an emphasis on short dependency tasks rather&#10;than long dependency tasks. In this paper, we present LooGLE, a Long Context&#10;Generic Language Evaluation benchmark for LLMs' long context understanding.&#10;LooGLE features relatively new documents post-2022, with over 24,000 tokens per&#10;document and 6,000 newly generated questions spanning diverse domains. Human&#10;annotators meticulously crafted more than 1,100 high-quality question-answer&#10;pairs to meet the long dependency requirements. These pairs underwent thorough&#10;cross-validation, yielding the most precise assessment of LLMs' long dependency&#10;capabilities. The evaluation of eight state-of-the-art LLMs on LooGLE revealed&#10;key findings: (i) commercial models outperformed open-sourced models; (ii) LLMs&#10;excelled in short dependency tasks like short question-answering and cloze&#10;tasks but struggled with more intricate long dependency tasks; (iii) in-context&#10;learning and chaining thoughts offered only marginal improvements; (iv)&#10;retrieval-based techniques demonstrated substantial benefits for short&#10;question-answering, while strategies for extending context window length had&#10;limited impact on long context understanding. As such, LooGLE not only provides&#10;a systematic and comprehensive evaluation schema on long-context LLMs, but also&#10;sheds light on future development of enhanced models towards &quot;true long-context&#10;understanding&quot;.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.04006" label="2011.04006">
        <attvalues>
          <attvalue for="0" value="Long Range Arena: A Benchmark for Efficient Transformers" />
          <attvalue for="1" value="  Transformers do not scale very well to long sequence lengths largely because&#10;of quadratic self-attention complexity. In the recent months, a wide spectrum&#10;of efficient, fast Transformers have been proposed to tackle this problem, more&#10;often than not claiming superior or comparable model quality to vanilla&#10;Transformer models. To this date, there is no well-established consensus on how&#10;to evaluate this class of models. Moreover, inconsistent benchmarking on a wide&#10;spectrum of tasks and datasets makes it difficult to assess relative model&#10;quality amongst many models. This paper proposes a systematic and unified&#10;benchmark, LRA, specifically focused on evaluating model quality under&#10;long-context scenarios. Our benchmark is a suite of tasks consisting of&#10;sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data&#10;types and modalities such as text, natural, synthetic images, and mathematical&#10;expressions requiring similarity, structural, and visual-spatial reasoning. We&#10;systematically evaluate ten well-established long-range Transformer models&#10;(Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers,&#10;Synthesizers, Sparse Transformers, and Longformers) on our newly proposed&#10;benchmark suite. LRA paves the way towards better understanding this class of&#10;efficient Transformer models, facilitates more research in this direction, and&#10;presents new challenging tasks to tackle. Our benchmark code will be released&#10;at https://github.com/google-research/long-range-arena.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.01861" label="2308.01861">
        <attvalues>
          <attvalue for="0" value="ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on&#10;  Class-level Code Generation" />
          <attvalue for="1" value="  In this work, we make the first attempt to evaluate LLMs in a more&#10;challenging code generation scenario, i.e. class-level code generation. We&#10;first manually construct the first class-level code generation benchmark&#10;ClassEval of 100 class-level Python code generation tasks with approximately&#10;500 person-hours. Based on it, we then perform the first study of 11&#10;state-of-the-art LLMs on class-level code generation. Based on our results, we&#10;have the following main findings. First, we find that all existing LLMs show&#10;much worse performance on class-level code generation compared to on standalone&#10;method-level code generation benchmarks like HumanEval; and the method-level&#10;coding ability cannot equivalently reflect the class-level coding ability among&#10;LLMs. Second, we find that GPT-4 and GPT-3.5 still exhibit dominate superior&#10;than other LLMs on class-level code generation, and the second-tier models&#10;includes Instruct-Starcoder, Instruct-Codegen, and Wizardcoder with very&#10;similar performance. Third, we find that generating the entire class all at&#10;once (i.e. holistic generation strategy) is the best generation strategy only&#10;for GPT-4 and GPT-3.5, while method-by-method generation (i.e. incremental and&#10;compositional) is better strategies for the other models with limited ability&#10;of understanding long instructions and utilizing the middle information.&#10;Lastly, we find the limited model ability of generating method-dependent code&#10;and discuss the frequent error types in generated classes. Our benchmark is&#10;available at https://github.com/FudanSELab/ClassEval.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.04610" label="2310.04610">
        <attvalues>
          <attvalue for="0" value="DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery&#10;  through Sophisticated AI System Technologies" />
          <attvalue for="1" value="  In the upcoming decade, deep learning may revolutionize the natural sciences,&#10;enhancing our capacity to model and predict natural occurrences. This could&#10;herald a new era of scientific exploration, bringing significant advancements&#10;across sectors from drug development to renewable energy. To answer this call,&#10;we present DeepSpeed4Science initiative (deepspeed4science.ai) which aims to&#10;build unique capabilities through AI system technology innovations to help&#10;domain experts to unlock today's biggest science mysteries. By leveraging&#10;DeepSpeed's current technology pillars (training, inference and compression) as&#10;base technology enablers, DeepSpeed4Science will create a new set of AI system&#10;technologies tailored for accelerating scientific discoveries by addressing&#10;their unique complexity beyond the common technical approaches used for&#10;accelerating generic large language models (LLMs). In this paper, we showcase&#10;the early progress we made with DeepSpeed4Science in addressing two of the&#10;critical system challenges in structural biology research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.01427" label="2310.01427">
        <attvalues>
          <attvalue for="0" value="Attention Sorting Combats Recency Bias In Long Context Language Models" />
          <attvalue for="1" value="  Current language models often fail to incorporate long contexts efficiently&#10;during generation. We show that a major contributor to this issue are attention&#10;priors that are likely learned during pre-training: relevant information&#10;located earlier in context is attended to less on average. Yet even when models&#10;fail to use the information from a relevant document in their response, they&#10;still pay preferential attention to that document compared to an irrelevant&#10;document at the same position. We leverage this fact to introduce ``attention&#10;sorting'': perform one step of decoding, sort documents by the attention they&#10;receive (highest attention going last), repeat the process, generate the answer&#10;with the newly sorted context. We find that attention sorting improves&#10;performance of long context models. Our findings highlight some challenges in&#10;using off-the-shelf language models for retrieval augmented generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.04455" label="2312.04455">
        <attvalues>
          <attvalue for="0" value="Fortify the Shortest Stave in Attention: Enhancing Context Awareness of&#10;  Large Language Models for Effective Tool Use" />
          <attvalue for="1" value="  In this paper, we demonstrate that an inherent waveform pattern in the&#10;attention allocation of large language models (LLMs) significantly affects&#10;their performance in tasks demanding a high degree of context awareness, such&#10;as utilizing LLMs for tool-use. Specifically, the crucial information in the&#10;context will be potentially overlooked by model when it is positioned in the&#10;trough zone of the attention waveform, leading to decreased performance. To&#10;address this issue, we propose a novel inference method named Attention&#10;Buckets. It allows LLMs to process their input through multiple parallel&#10;processes. Each process utilizes a distinct base angle for the rotary position&#10;embedding, thereby creating a unique attention waveform. By compensating an&#10;attention trough of a particular process with an attention peak of another&#10;process, our approach enhances LLM's awareness to various contextual positions,&#10;thus mitigating the risk of overlooking crucial information. In the largest&#10;tool-use benchmark, our method elevates a 7B model to achieve state-of-the-art&#10;performance, comparable to that of GPT-4. On other benchmarks and some RAG&#10;tasks, which also demand a thorough understanding of contextual content,&#10;Attention Buckets also exhibited notable enhancements in performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1409.4842" label="1409.4842">
        <attvalues>
          <attvalue for="0" value="Going Deeper with Convolutions" />
          <attvalue for="1" value="  We propose a deep convolutional neural network architecture codenamed&#10;&quot;Inception&quot;, which was responsible for setting the new state of the art for&#10;classification and detection in the ImageNet Large-Scale Visual Recognition&#10;Challenge 2014 (ILSVRC 2014). The main hallmark of this architecture is the&#10;improved utilization of the computing resources inside the network. This was&#10;achieved by a carefully crafted design that allows for increasing the depth and&#10;width of the network while keeping the computational budget constant. To&#10;optimize quality, the architectural decisions were based on the Hebbian&#10;principle and the intuition of multi-scale processing. One particular&#10;incarnation used in our submission for ILSVRC 2014 is called GoogLeNet, a 22&#10;layers deep network, the quality of which is assessed in the context of&#10;classification and detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.16998" label="2402.16998">
        <attvalues>
          <attvalue for="0" value="What Do Language Models Hear? Probing for Auditory Representations in&#10;  Language Models" />
          <attvalue for="1" value="  This work explores whether language models encode meaningfully grounded&#10;representations of sounds of objects. We learn a linear probe that retrieves&#10;the correct text representation of an object given a snippet of audio related&#10;to that object, where the sound representation is given by a pretrained audio&#10;model. This probe is trained via a contrastive loss that pushes the language&#10;representations and sound representations of an object to be close to one&#10;another. After training, the probe is tested on its ability to generalize to&#10;objects that were not seen during training. Across different language models&#10;and audio models, we find that the probe generalization is above chance in many&#10;cases, indicating that despite being trained only on raw text, language models&#10;encode grounded knowledge of sounds for some objects.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Probing language models. Language models have been shown to encode much linguistic information in their contextualized representations \cite{tenney2019bert,liu-etal-2019-linguistic,jawahar2019does} and attention distributions \cite{clark2019does,vig2019analyzing}. Building on top of these more linguistically-oriented probes, there has been mounting recent evidence that language models trained on just text are able to meaningfully encode a surprising amount of grounded or extralinguistic information, such as color \cite{abdou-etal-2021-language}, direction \cite{patel2021mapping}, size \cite{zhang-etal-2020-language-embeddings,grand2022semantic}, geography \cite{konkol-etal-2017-geographical,lietard2021language,faisal-anastasopoulos-2023-geographic,chen2023more}, time \cite{gurnee2023language}, visual representations \cite{ilharco_probing_2021,merullo2022linearly,li_implications_2023}, character-level information of word-pieces \cite{kaushal-mahowald-2022-tokens}, and representations of meaning \cite{li-etal-2021-implicit}. LM-derived similiarity measures have also been shown to correlate with human-derived similarity measures across a number of modalities \cite{marjieh2023large,marjieh2022words}. The present work extends the probing-based line of work to sounds, and investigates the extent to which language models trained on text-only can encode auditory representations. &#10;&#10;Our work is also related to the line of work investigating whether a model that has been trained on raw outputs of a synthetic environment can acquire ``true'' representations of that environment. Examples of such environments include Othello \cite{li2022emergent}, chess \cite{toshniwal2021learning}, and toy grid worlds \cite{yun2023emergence,jin2023evidence}.&#10;&#10;Meaning in language models. Whether language models can acquire meaning and understanding from being trained on form alone is the subject of much debate \cite{bender_climbing_2020,merrill-etal-2021-provable,piantadosi2022meaning,pavlick2023symbols,sogaard2023grounding}. In operationalizations of meaning which do not rely on explicit reference to the external world, the fact that the geometry of language models' representation spaces is structurally related to the geometry of grounded representations could be construed as evidence for these models' acquiring meaning in some broad sense. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Linguistics, Cognitive Science, Multimodal Knowledge Encoding, Audio Representation Learning, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2407.02820" label="2407.02820">
        <attvalues>
          <attvalue for="0" value="Investigating the Contextualised Word Embedding Dimensions Responsible&#10;  for Contextual and Temporal Semantic Changes" />
          <attvalue for="1" value="  Words change their meaning over time as well as in different contexts. The&#10;sense-aware contextualised word embeddings (SCWEs) such as the ones produced by&#10;XL-LEXEME by fine-tuning masked langauge models (MLMs) on Word-in-Context (WiC)&#10;data attempt to encode such semantic changes of words within the contextualised&#10;word embedding (CWE) spaces. Despite the superior performance of SCWEs in&#10;contextual/temporal semantic change detection (SCD) benchmarks, it remains&#10;unclear as to how the meaning changes are encoded in the embedding space. To&#10;study this, we compare pre-trained CWEs and their fine-tuned versions on&#10;contextual and temporal semantic change benchmarks under Principal Component&#10;Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our&#10;experimental results reveal several novel insights such as (a) although there&#10;exist a smaller number of axes that are responsible for semantic changes of&#10;words in the pre-trained CWE space, this information gets distributed across&#10;all dimensions when fine-tuned, and (b) in contrast to prior work studying the&#10;geometry of CWEs, we find that PCA to better represent semantic changes than&#10;ICA. Source code is available at https://github.com/LivNLP/svp-dims .&#10;" />
          <attvalue for="2" value="&#10;&#10;Meaning of a word is a dynamic phenomenon that is both contextual (i.e. depends on the context in which the word is used)~\cite{pilehvar-camacho-collados-2019-wic} as well as temporal (i.e. the meaning of a word can change over time)~\cite{tahmasebia-etal-2021-survey}.&#10;A large body of methods have been proposed to represent the meaning of a word in a given context~\cite{BERT,conneau-etal-2020-xlmroberta,yi-zhou-2021-learning,rachinskiy-arefyev-2021-glossreader,periti-etal-2024-automatically}, or within a given time period~\cite{hamilton-etal-2016-diachronic,rosenfeld-erk-2018-deep,aida-etal-2021-comprehensive,rosin-etal-2022-time,aida-bollegala-2023-unsupervised,Tang2023-yu, fedorova-etal-2024-definition}. &#10;In particular, SCWE such as XL-LEXEME~\cite{cassotti-etal-2023-xl} obtained by fine-tuning MLM such as XLM-RoBERTa~\cite{conneau-etal-2020-xlmroberta} on WiC~\cite{pilehvar-camacho-collados-2019-wic} have reported superior performance in SCD benchmarks~\cite{cassotti-etal-2023-xl,aida-bollegala-2023-swap,periti-tahmasebi-2024-systematic,aida-bollegala-2024-semantic}, implying that semantic changes can be accurately inferred from SCWE.&#10;&#10;Despite the empirical success, to the best of our knowledge, no prior work has investigated whether there are dedicated dimensions in the XL-LEXEME embedding space specified for the semantic changes of the words it represents.&#10;In this paper, we study this problem from two complementary directions.&#10;First, in \ref{sec:contextual}, we investigate the embedding dimensions specific to the contextual semantic changes of words using WiC benchmarks~\cite{pilehvar-camacho-collados-2019-wic,raganato-etal-2020-xlwic,martelli-etal-2021-mclwic,liu-etal-2021-am2ico} as the evaluation task.&#10;Second, in \ref{sec:temporal}, we investigate the embedding dimensions specific to the temporal semantic changes of words on SemEval-2020 Task 1~\cite{schlechtweg-etal-2020-semeval} benchmark.&#10;In each setting, we compare pre-trained CWE and the SCWE obtained by fine-tuning on WiC using PCA and ICA, which have been used in prior work investigating dimensions in CWE~\cite{yamagiwa-etal-2023-discovering}.&#10;Our investigations reveal several interesting novel insights that will be useful when developing accurate and efficient low-dimensional SCD methods as follows.&#10;\begin{itemize}[leftmargin=*]&#10; \item PCA discovers contextual/temporal semantic change-aware axes within the top 10\% of the transformed axes better than ICA.&#10; \item In pre-trained embeddings, we identify a small number of axes that are specified for contextual/temporal semantic changes, while such axes are uniformly distributed in the fine-tuned embeddings.&#10; \item Semantic change aware dimensions report comparable or superior performance over using all dimensions in SCD benchmarks.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Word Embeddings, Dimensionality Reduction, Semantic Change Detection, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2110.02204" label="2110.02204">
        <attvalues>
          <attvalue for="0" value="Learning Sense-Specific Static Embeddings using Contextualised Word&#10;  Embeddings as a Proxy" />
          <attvalue for="1" value="  Contextualised word embeddings generated from Neural Language Models (NLMs),&#10;such as BERT, represent a word with a vector that considers the semantics of&#10;the target word as well its context. On the other hand, static word embeddings&#10;such as GloVe represent words by relatively low-dimensional, memory- and&#10;compute-efficient vectors but are not sensitive to the different senses of the&#10;word. We propose Context Derived Embeddings of Senses (CDES), a method that&#10;extracts sense related information from contextualised embeddings and injects&#10;it into static embeddings to create sense-specific static embeddings.&#10;Experimental results on multiple benchmarks for word sense disambiguation and&#10;sense discrimination tasks show that CDES can accurately learn sense-specific&#10;static embeddings reporting comparable performance to the current&#10;state-of-the-art sense embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.00226" label="2403.00226">
        <attvalues>
          <attvalue for="0" value="A Semantic Distance Metric Learning approach for Lexical Semantic Change&#10;  Detection" />
          <attvalue for="1" value="  Detecting temporal semantic changes of words is an important task for various&#10;NLP applications that must make time-sensitive predictions. Lexical Semantic&#10;Change Detection (SCD) task involves predicting whether a given target word,&#10;$w$, changes its meaning between two different text corpora, $C_1$ and $C_2$.&#10;For this purpose, we propose a supervised two-staged SCD method that uses&#10;existing Word-in-Context (WiC) datasets. In the first stage, for a target word&#10;$w$, we learn two sense-aware encoders that represent the meaning of $w$ in a&#10;given sentence selected from a corpus. Next, in the second stage, we learn a&#10;sense-aware distance metric that compares the semantic representations of a&#10;target word across all of its occurrences in $C_1$ and $C_2$. Experimental&#10;results on multiple benchmark datasets for SCD show that our proposed method&#10;achieves strong performance in multiple languages. Additionally, our method&#10;achieves significant improvements on WiC benchmarks compared to a sense-aware&#10;encoder with conventional distance functions. Source code is available at&#10;https://github.com/LivNLP/svp-sdml .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13175" label="2305.13175">
        <attvalues>
          <attvalue for="0" value="Discovering Universal Geometry in Embeddings with ICA" />
          <attvalue for="1" value="  This study utilizes Independent Component Analysis (ICA) to unveil a&#10;consistent semantic structure within embeddings of words or images. Our&#10;approach extracts independent semantic components from the embeddings of a&#10;pre-trained model by leveraging anisotropic information that remains after the&#10;whitening process in Principal Component Analysis (PCA). We demonstrate that&#10;each embedding can be expressed as a composition of a few intrinsic&#10;interpretable axes and that these semantic axes remain consistent across&#10;different languages, algorithms, and modalities. The discovery of a universal&#10;semantic structure in the geometric patterns of embeddings enhances our&#10;understanding of the representations in embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.18862" label="2310.18862">
        <attvalues>
          <attvalue for="0" value="Counterfactually Probing Language Identity in Multilingual Models" />
          <attvalue for="1" value="  Techniques in causal analysis of language models illuminate how linguistic&#10;information is organized in LLMs. We use one such technique, AlterRep, a method&#10;of counterfactual probing, to explore the internal structure of multilingual&#10;models (mBERT and XLM-R). We train a linear classifier on a binary language&#10;identity task, to classify tokens between Language X and Language Y. Applying a&#10;counterfactual probing procedure, we use the classifier weights to project the&#10;embeddings into the null space and push the resulting embeddings either in the&#10;direction of Language X or Language Y. Then we evaluate on a masked language&#10;modeling task. We find that, given a template in Language X, pushing towards&#10;Language Y systematically increases the probability of Language Y words, above&#10;and beyond a third-party control language. But it does not specifically push&#10;the model towards translation-equivalent words in Language Y. Pushing towards&#10;Language X (the same direction as the template) has a minimal effect, but&#10;somewhat degrades these models. Overall, we take these results as further&#10;evidence of the rich structure of massive multilingual language models, which&#10;include both a language-specific and language-general component. And we show&#10;that counterfactual probing can be fruitfully applied to multilingual models.&#10;" />
          <attvalue for="2" value="&#10;Large pretrained multilingual transformer models succeed at a variety of multilingual and monolingual tasks and can be used in transfer learning paradigms, where a model is trained to do a task in one language and then transferred to another language \cite{lauscher-etal-2020-zero,conneau-etal-2020-emerging, wu-dredze-2019-beto, wu-dredze-2020-languages, pires-etal-2019-multilingual,vulic-etal-2020-probing,rust-etal-2021-good}. &#10;These abilities have spurred a spate of papers probing the internal workings and capabilities of multilingual models, suggesting that such models may contain language-independent, along with langauge-specific knowledge of interesting linguistic structure \cite[e.g.,][]{chi-etal-2020-finding,papadimitriou-etal-2021-deep,ravishankar-etal-2021-attention,blevins-etal-2022-analyzing,gonen-etal-2020-greek}.&#10;&#10;While the results of this literature are suggestive, probing methods are susceptible to memorizing the original input and may not reflect what information models actually use downstream \cite{hewitt-liang-2019-designing,elazar-etal-2021-amnesic,pimentel-etal-2020-information,voita-etal-2021-analyzing}.&#10;It is thus desirable to test not only what information can be extracted but what information is actually used \cite{geiger2021causal,finlayson-etal-2021-causal,lasri-etal-2022-probing}.&#10;&#10;To do that we apply AlterRep \cite{ravfogel-etal-2021-counterfactual}, an offshoot of Iterative Nullspace Projection \cite[INLP;][]{ravfogel-etal-2020-null,elazar-etal-2021-amnesic}, in a multilingual setting.&#10;The AlterRep method is to train a classifier on the model representations to pick out a particular feature and then use the parameters learned by the classifier to intervene on the embeddings, pushing them in a particular direction. \cite{ravfogel-etal-2021-counterfactual} use it to intervene on whether a noun phrase is in a relative clause (e.g., training a classifier on whether the noun phrase is in a relative clause and then using projections from the classifier to push the embeddings towards or away from the relative clause direction).&#10;Crucially, they then measure how this manipulation affects downstream subject-verb number agreement.&#10;&#10;Whereas \cite{ravfogel-etal-2021-counterfactual} use AlterRep to explore syntactic representations in models, our hypothesis is that the same kind of causal manipulation could be informative as to how multilingual models process multilingual text.&#10;Doing so necessarily involves separating multilingual embedding space into language-neutral and language-specific components.&#10;\cite{libovicky-etal-2020-language} explore the idea of obtaining a language-neutral representation from a multilingual model by computing an ``average'' representation for each language and subtracting it from the token embedding. &#10;&#10;There is some precedent for using INLP to generate language-specific and language-neutral components. &#10;\cite{gonen-etal-2020-greek} showed that multilingual models like mBERT have both a language-specific and language-general component and that, by separating them using INLP on a language identification task, one can obtain language-agnostic representations (and, inversely, highly language-specific representations).&#10;They show that, by training on an English vs. non-English task and then projecting onto the nullspace using INLP, the generated text on a masked language modeling task (in English) is less likely to be English after INLP.&#10;\cite{gonen-etal-2020-greek} also show that, by subtracting an ``average'' representation of language $X$ from a particular token embedding and then adding the average language $Y$ embedding, one can obtain a translation of the token in language $Y$ by analogy.&#10;But they do not specifically use INLP to do these translations in a language-to-language way, as we do here.&#10;&#10;Using a similar logic but the AlterRep technique instead of the analogical method, we test whether we can do a kind of ``translation via AlterRep'', effectively ``pushing'' the embeddings towards a particular language. First, we use the original multilingual model embeddings for a particular token $h_t$ to train a language identity classifier $C$ to classify the language of tokens from Languages $X$ and $Y$.&#10;We then use INLP to null out language ID information, creating null embeddings $h_t^N$.&#10;We can then generate altered embeddings $h_t^X$ and $h_t^Y$, which go beyond merely nulling out language ID and instead represent embeddings that have been pushed into the direction of Language $X$ or $Y$, respectively.&#10;We use these counterfactual embeddings to generate predictions for masked text and compare the result to the original embeddings.&#10;&#10;To make this concrete, imagine training a language identification classifier on English vs. Spanish, as shown in Figure~\ref{fig:fig1}.&#10;Whereas a multilingual model would typically fill in the [MASK] position in the English sentence ``I ate a [MASK]'' with an English token, if we use the classifier to push the embeddings in the direction of Spanish, then we might expect a completion like ``I ate a cereza'' to become more likely where cereza is the English word for cherry.&#10;We would expect the probability of the English word ``cherry'' to decrease.&#10;&#10;Through this work, our hope is not only to illuminate the innerworkings of multilingual models, but also to validate and explore the use of counterfactual probing in a novel domain.&#10;&#10;To spoil the result: we show that language identity is encoded in contextual token embeddings and, crucially, that this information is used by multilingual models in masked language modeling.&#10;In effect, pushing embeddings in the direction of a particular language (and away from another) systematically increases probabilities of words in the PushedTo language and decreases the probabilities in the PushedAway language, while leaving words from other languages unchanged.&#10;By comparing the changes in probabilities of target words in the PushedTo language (i.e., translation equivalents of the original correct word) to random words in that language, we see that our alterations seem to push the model towards the prior of the intended language, without specifically boosting the semantic equivalent.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Causal Analysis Techniques, Artificial Intelligence, Mathematics, Counterfactual Probing Methods, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2101.11043" label="2101.11043">
        <attvalues>
          <attvalue for="0" value="Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT" />
          <attvalue for="1" value="  We investigate how Multilingual BERT (mBERT) encodes grammar by examining how&#10;the high-order grammatical feature of morphosyntactic alignment (how different&#10;languages define what counts as a &quot;subject&quot;) is manifested across the embedding&#10;spaces of different languages. To understand if and how morphosyntactic&#10;alignment affects contextual embedding spaces, we train classifiers to recover&#10;the subjecthood of mBERT embeddings in transitive sentences (which do not&#10;contain overt information about morphosyntactic alignment) and then evaluate&#10;them zero-shot on intransitive sentences (where subjecthood classification&#10;depends on alignment), within and across languages. We find that the resulting&#10;classifier distributions reflect the morphosyntactic alignment of their&#10;training languages. Our results demonstrate that mBERT representations are&#10;influenced by high-level grammatical features that are not manifested in any&#10;one input sentence, and that this is robust across languages. Further examining&#10;the characteristics that our classifiers rely on, we find that features such as&#10;passive voice, animacy and case strongly correlate with classification&#10;decisions, suggesting that mBERT does not encode subjecthood purely&#10;syntactically, but that subjecthood embedding is continuous and dependent on&#10;semantic and discourse factors, as is proposed in much of the functional&#10;linguistics literature. Together, these results provide insight into how&#10;grammatical features manifest in contextual embedding spaces, at a level of&#10;abstraction not covered by previous work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.10927" label="2101.10927">
        <attvalues>
          <attvalue for="0" value="Attention Can Reflect Syntactic Structure (If You Let It)" />
          <attvalue for="1" value="  Since the popularization of the Transformer as a general-purpose feature&#10;encoder for NLP, many studies have attempted to decode linguistic structure&#10;from its novel multi-head attention mechanism. However, much of such work&#10;focused almost exclusively on English -- a language with rigid word order and a&#10;lack of inflectional morphology. In this study, we present decoding experiments&#10;for multilingual BERT across 18 languages in order to test the generalizability&#10;of the claim that dependency syntax is reflected in attention patterns. We show&#10;that full trees can be decoded above baseline accuracy from single attention&#10;heads, and that individual relations are often tracked by the same heads across&#10;languages. Furthermore, in an attempt to address recent debates about the&#10;status of attention as an explanatory mechanism, we experiment with fine-tuning&#10;mBERT on a supervised parsing objective while freezing different series of&#10;parameters. Interestingly, in steering the objective to learn explicit&#10;linguistic structure, we find much of the same structure represented in the&#10;resulting attention patterns, with interesting differences with respect to&#10;which parameters are frozen.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.08275" label="2010.08275">
        <attvalues>
          <attvalue for="0" value="It's not Greek to mBERT: Inducing Word-Level Translations from&#10;  Multilingual BERT" />
          <attvalue for="1" value="  Recent works have demonstrated that multilingual BERT (mBERT) learns rich&#10;cross-lingual representations, that allow for transfer across languages. We&#10;study the word-level translation information embedded in mBERT and present two&#10;simple methods that expose remarkable translation capabilities with no&#10;fine-tuning. The results suggest that most of this information is encoded in a&#10;non-linear way, while some of it can also be recovered with purely linear&#10;tools. As part of our analysis, we test the hypothesis that mBERT learns&#10;representations which contain both a language-encoding component and an&#10;abstract, cross-lingual component, and explicitly identify an empirical&#10;language-identity subspace within mBERT representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03061" label="2004.03061">
        <attvalues>
          <attvalue for="0" value="Information-Theoretic Probing for Linguistic Structure" />
          <attvalue for="1" value="  The success of neural networks on a diverse set of NLP tasks has led&#10;researchers to question how much these networks actually ``know'' about natural&#10;language. Probes are a natural way of assessing this. When probing, a&#10;researcher chooses a linguistic task and trains a supervised model to predict&#10;annotations in that linguistic task from the network's learned representations.&#10;If the probe does well, the researcher may conclude that the representations&#10;encode knowledge related to the task. A commonly held belief is that using&#10;simpler models as probes is better; the logic is that simpler models will&#10;identify linguistic structure, but not learn the task itself. We propose an&#10;information-theoretic operationalization of probing as estimating mutual&#10;information that contradicts this received wisdom: one should always select the&#10;highest performing probe one can, even if it is more complex, since it will&#10;result in a tighter estimate, and thus reveal more of the linguistic&#10;information inherent in the representation. The experimental portion of our&#10;paper focuses on empirically estimating the mutual information between a&#10;linguistic property and BERT, comparing these estimates to several baselines.&#10;We evaluate on a set of ten typologically diverse languages often&#10;underrepresented in NLP research---plus English---totalling eleven languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.10907" label="2010.10907">
        <attvalues>
          <attvalue for="0" value="Analyzing the Source and Target Contributions to Predictions in Neural&#10;  Machine Translation" />
          <attvalue for="1" value="  In Neural Machine Translation (and, more generally, conditional language&#10;modeling), the generation of a target token is influenced by two types of&#10;context: the source and the prefix of the target sequence. While many attempts&#10;to understand the internal workings of NMT models have been made, none of them&#10;explicitly evaluates relative source and target contributions to a generation&#10;decision. We argue that this relative contribution can be evaluated by adopting&#10;a variant of Layerwise Relevance Propagation (LRP). Its underlying&#10;'conservation principle' makes relevance propagation unique: differently from&#10;other methods, it evaluates not an abstract quantity reflecting token&#10;importance, but the proportion of each token's influence. We extend LRP to the&#10;Transformer and conduct an analysis of NMT models which explicitly evaluates&#10;the source and target relative contributions to the generation process. We&#10;analyze changes in these contributions when conditioning on different types of&#10;prefixes, when varying the training objective or the amount of training data,&#10;and during the training process. We find that models trained with more data&#10;tend to rely on source information more and to have more sharp token&#10;contributions; the training process is non-monotonic with several stages of&#10;different nature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.02997" label="2106.02997">
        <attvalues>
          <attvalue for="0" value="Causal Abstractions of Neural Networks" />
          <attvalue for="1" value="  Structural analysis methods (e.g., probing and feature attribution) are&#10;increasingly important tools for neural network analysis. We propose a new&#10;structural analysis method grounded in a formal theory of causal abstraction&#10;that provides rich characterizations of model-internal representations and&#10;their roles in input/output behavior. In this method, neural representations&#10;are aligned with variables in interpretable causal models, and then interchange&#10;interventions are used to experimentally verify that the neural representations&#10;have the causal properties of their aligned variables. We apply this method in&#10;a case study to analyze neural models trained on Multiply Quantified Natural&#10;Language Inference (MQNLI) corpus, a highly complex NLI dataset that was&#10;constructed with a tree-structured natural logic causal model. We discover that&#10;a BERT-based model with state-of-the-art performance successfully realizes&#10;parts of the natural logic model's causal structure, whereas a simpler baseline&#10;model fails to show any such structure, demonstrating that BERT representations&#10;encode the compositional structure of MQNLI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.06965" label="2105.06965">
        <attvalues>
          <attvalue for="0" value="Counterfactual Interventions Reveal the Causal Effect of Relative Clause&#10;  Representations on Agreement Prediction" />
          <attvalue for="1" value="  When language models process syntactically complex sentences, do they use&#10;their representations of syntax in a manner that is consistent with the grammar&#10;of the language? We propose AlterRep, an intervention-based method to address&#10;this question. For any linguistic feature of a given sentence, AlterRep&#10;generates counterfactual representations by altering how the feature is&#10;encoded, while leaving intact all other aspects of the original representation.&#10;By measuring the change in a model's word prediction behavior when these&#10;counterfactual representations are substituted for the original ones, we can&#10;draw conclusions about the causal effect of the linguistic feature in question&#10;on the model's behavior. We apply this method to study how BERT models of&#10;different sizes process relative clauses (RCs). We find that BERT variants use&#10;RC boundary information during word prediction in a manner that is consistent&#10;with the rules of English grammar; this RC boundary information generalizes to&#10;a considerable extent across different RC types, suggesting that BERT&#10;represents RCs as an abstract linguistic category.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.09659" label="2409.09659">
        <attvalues>
          <attvalue for="0" value="Leveraging Open-Source Large Language Models for Native Language&#10;  Identification" />
          <attvalue for="1" value="  Native Language Identification (NLI) - the task of identifying the native&#10;language (L1) of a person based on their writing in the second language (L2) -&#10;has applications in forensics, marketing, and second language acquisition.&#10;Historically, conventional machine learning approaches that heavily rely on&#10;extensive feature engineering have outperformed transformer-based language&#10;models on this task. Recently, closed-source generative large language models&#10;(LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a&#10;zero-shot setting, including promising results in open-set classification.&#10;However, closed-source LLMs have many disadvantages, such as high costs and&#10;undisclosed nature of training data. This study explores the potential of using&#10;open-source LLMs for NLI. Our results indicate that open-source LLMs do not&#10;reach the accuracy levels of closed-source LLMs when used out-of-the-box.&#10;However, when fine-tuned on labeled training data, open-source LLMs can achieve&#10;performance comparable to that of commercial LLMs.&#10;" />
          <attvalue for="2" value=" &#10;Native Language Identification (NLI) is the task of automatically identifying an author's native language (L1) based on texts written in their second language (L2). The task is based on the language transfer hypothesis, the phenomenon in which &#10;characteristics of L1 influence the production of texts in L2 to the degree that L1 is identifiable~\cite{Odlin:89}. &#10;NLI is useful for educational purposes, forensic applications in the context of author profiling, and to inform second language acquisition research~\cite{goswami-etal-2024-native}. &#10;&#10;From a machine learning (ML) perspective, NLI is commonly framed as a supervised multi-class classification task, where NLI systems are trained to assign an author's L1. &#10;While the task has been proven difficult to perform by humans \cite{malmasi-etal-2015-oracle}, automated methods have shown remarkable results &#10;using conventional ML approaches based on extensive feature engineering, e.g.,~\cite{cimino-dellorletta-2017-stacked, Markov:2018}. Such methods rely on features that capture L1-indicative linguistic patterns in L2 writing, &#10;e.g., spelling errors \cite{koppel_2005, chen-etal-2017-improving, markov-etal-2019-anglicized}, word choice \cite{brooke-hirst-2012-robust}, and syntactic patterns \cite{wong-dras-2011-exploiting}. &#10;&#10;Transformer-based encoder models, like BERT~\cite{devlin:2019-bert}, on the other hand, &#10;have yielded poorer performance than conventional ML approaches for the NLI task \cite{markov-exploiting2022, steinbakken-gamback-2020-native, goswami-etal-2024-native}. &#10;Previous research suggests that this is likely because NLI concerns very specific linguistic features that models trained on general corpora cannot capture~\cite{ markov-exploiting2022}.&#10;Recent research has shown that generative large language models (LLMs) demonstrate promising results for NLI. \cite{lotfi-etal-2020-deep} presented the first study addressing NLI using fine-tuned GPT-2 models, which outperformed previous traditional ML approaches and achieved state-of-the-art results on the NLI benchmark TOEFL11 and ICLE datasets. &#10;\cite{zhang-2023} explored the ability of GPT-3.5 \cite{gpt3-brown:20} and GPT-4 \cite{openai-gpt4:23} to perform NLI. Their results indicate that out-of-the-box GPT models demonstrate outstanding performance, with GPT-4 setting a new performance record of 91.7\% accuracy on the TOEFL11 benchmark dataset, and achieve promising results for open-set classification (without a predefined set of L1s), a useful setting for real-world NLI applications. &#10;&#10;While \cite{zhang-2023}'s results indicate that LLMs achieve state-of-the-art performance on NLI, they only evaluate the performance of GPT-3.5 and GPT-4. The closed-source nature of these models presents a multitude of limitations to research. Providers of closed-source models often disclose minimal information regarding the training data or procedure, hindering the evaluation of results achieved with these models and obscuring biases in training data and models \cite{balloccu-etal-2024-leak}. &#10;The undisclosed nature of the training data has also raised concerns among researchers about data contamination risks, as it is challenging to determine whether a model's high performance on a task can be attributed to the model's effective generalization or potential data leakage \cite{yu-open-2023}. &#10;In addition, closed-source models are typically only accessible via an API, causing lack of control over model updates, &#10;which are often communicated poorly to users \cite{yu-open-2023, pozzobon-etal-2023-challenges}. In turn, the reproducibility of experiments &#10;cannot be guaranteed. The usage of closed-source LLMs is also highly costly, which negatively impacts the accessibility of LLMs~\cite{bender2021}. &#10;&#10;Providers of open-source LLMs, on the other hand, often release more information regarding training data and procedures. &#10;As model weights are released openly, open-source LLMs can be fine-tuned for a down-stream task, which is often highly costly or not supported for closed-source models. &#10;Despite these advantages, employing open-source LLMs for NLI remains unexplored, and it is therefore important to investigate the difference in performance between open-source and proprietary LLMs on this task. &#10;Hence, the research question addressed in this study is: Can open-source LLMs be used for effective Native Language Identification? &#10;&#10;The contributions of this work are the following: (i) we are the first to explore the performance of open-source LLMs on NLI and quantify the difference in performance with closed-source models, and (ii) &#10;we investigate the impact of fine-tuning open-source LLMs on NLI performance. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning Models, Machine Learning, Computer Science, Linguistics, Language Identification, Artificial Intelligence, Natural Language Processing, Open Source Technology" />
        </attvalues>
      </node>
      <node id="2312.07819" label="2312.07819">
        <attvalues>
          <attvalue for="0" value="Native Language Identification with Large Language Models" />
          <attvalue for="1" value="  We present the first experiments on Native Language Identification (NLI)&#10;using LLMs such as GPT-4. NLI is the task of predicting a writer's first&#10;language by analyzing their writings in a second language, and is used in&#10;second language acquisition and forensic linguistics. Our results show that GPT&#10;models are proficient at NLI classification, with GPT-4 setting a new&#10;performance record of 91.7% on the benchmark TOEFL11 test set in a zero-shot&#10;setting. We also show that unlike previous fully-supervised settings, LLMs can&#10;perform NLI without being limited to a set of known classes, which has&#10;practical implications for real-world applications. Finally, we also show that&#10;LLMs can provide justification for their choices, providing reasoning based on&#10;spelling errors, syntactic patterns, and usage of directly translated&#10;linguistic patterns.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.04873" label="2204.04873">
        <attvalues>
          <attvalue for="0" value="Adapting BigScience Multilingual Model to Unseen Languages" />
          <attvalue for="1" value="  We benchmark different strategies of adding new languages (German and Korean)&#10;into the BigScience's pretrained multilingual language model with 1.3 billion&#10;parameters that currently supports 13 languages. We investigate the factors&#10;that affect the language adaptability of the model and the trade-offs between&#10;computational costs and expected performance.&#10;" />
          <attvalue for="2" value="&#10;Pretrained multilingual language models (LMs) have enabled cross-lingual transfer \cite{artetxe-etal-2020-cross, conneau-etal-2020-emerging, k-etal-2021-analyzing}. While several works suggest that such knowledge transfer goes beyond just vocabulary sharing across languages \cite{artetxe-etal-2020-cross, k-etal-2021-analyzing}, others have shown that the models' performance is sensitive to the quality of the tokenization it relies on \cite{pfeiffer-etal-2021-unks}. Moreover, the quality of knowledge transfer can degrade for unseen languages, especially when the scripts are unknown to the model \cite{muller-etal-2021-unseen}. &#10;&#10;Data availability, but also the curse of multilinguality \cite{conneau-etal-2020-unsupervised} makes training a single model covering all the languages challenging: there is a trade off between the number of languages in pretraining data, model capacity and the downstream performance for each individual language. Finally, it is hard to anticipate any potential usage of pretrained LM in advance, hence it is important to study a posteoriori adaptation to new tasks/languages. &#10;Recently proposed methods include continual pretraining of the model (restricted to the embedding layer training only in some cases) \cite{artetxe-etal-2020-cross,chau-etal-2020-parsing,muller-etal-2021-unseen,zhang-etal-2020-multi-stage,wang-etal-2020-extending}, or training of language-specific adapters \cite{pfeiffer-etal-2020-mad, pfeiffer-etal-2021-adapterfusion, pfeiffer-etal-2021-unks, philip-etal-2020-monolingual, ustun-etal-2021-multilingual, berard-2021-continual} for the target language. The core motivation behind these methods is to benefit from knowledge transfer encoded in the pretrained LM for the new language processing at a small computational cost (compared to full model retraining). &#10;&#10;In this work, we aim at better understanding the trade-offs between the amount of compute and the final downstream task performance. Specifically, we study the impact of the following three factors: original pretraining steps, adaptation strategies, and adapters' capacity on the Natural Language Inference (NLI) task. As part of the initiative of BigScience, we experiment with its multilingual LM, which currently only supports 13 languages. We investigate how researchers could adapt BigScience's full-open-access multilingual models to their languages of interest. Moreover, BigScience open source its intermediate pretraining checkpoints, therefore we also study how does the adaptability to new language changes with the amount of pretraining. &#10;&#10;Our main findings are: (1) the most promising strategy is training of the embedding layers and adapters simultaneously; (2) position encoding is an important component to adapt; (3) pretraining steps benefit zero-shot performance; (4) adaptation to low-resource languages can be done with low parameter budget. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Multilingual Model Training, Linguistics, Mathematics, Computational Cost Optimization, Language Adaptability Factors" />
        </attvalues>
      </node>
      <node id="2010.05904" label="2010.05904">
        <attvalues>
          <attvalue for="0" value="Multi-Stage Pre-training for Low-Resource Domain Adaptation" />
          <attvalue for="1" value="  Transfer learning techniques are particularly useful in NLP tasks where a&#10;sizable amount of high-quality annotated data is difficult to obtain. Current&#10;approaches directly adapt a pre-trained language model (LM) on in-domain text&#10;before fine-tuning to downstream tasks. We show that extending the vocabulary&#10;of the LM with domain-specific terms leads to further gains. To a bigger&#10;effect, we utilize structure in the unlabeled data to create auxiliary&#10;synthetic tasks, which helps the LM transfer to downstream tasks. We apply&#10;these approaches incrementally on a pre-trained Roberta-large LM and show&#10;considerable performance gain on three tasks in the IT domain: Extractive&#10;Reading Comprehension, Document Ranking and Duplicate Question Detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.07259" label="2102.07259">
        <attvalues>
          <attvalue for="0" value="Thank you for Attention: A survey on Attention-based Artificial Neural&#10;  Networks for Automatic Speech Recognition" />
          <attvalue for="1" value="  Attention is a very popular and effective mechanism in artificial neural&#10;network-based sequence-to-sequence models. In this survey paper, a&#10;comprehensive review of the different attention models used in developing&#10;automatic speech recognition systems is provided. The paper focuses on the&#10;development and evolution of attention models for offline and streaming speech&#10;recognition within recurrent neural network- and Transformer- based&#10;architectures.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic speech recognition (ASR) is a type of sequence-to-sequence (seq2seq) task. The input speech sequence is transcribed into a sequence of symbols. The majority of the existing state-of-the art ASR systems consisted of three modules: acoustic, pronunciation and language \cite{jelinek1976continuous}. These three modules are separately trained. The acoustic module predicts phonemes based on the input speech feature like Mel Frequency Cepstral Coefficient (MFCC) \cite{muda2010voice}. The pronunciation module is a hidden Markov model \cite{gales2008application} which maps the phonemes predicted at the earlier module to word sequences. Finally, the language module which is pre-trained on a large corpus, scores the word sequences. In other words, language model estimates the probabilities of next word based on previously predicted words to establish a meaningful sentence. This traditional approach has some limitations. First, the modules are trained separately for different objective functions. Therefore, it may result incompatibility between modules. Also separate training is time expensive. Second, the pronunciation model requires a dictionary for mapping between phonemes and word sequences. The pronunciation dictionary is developed by linguistic experts and is prone to human errors \cite{kudo2004applying, bird2006nltk}.&#10;&#10;From the last decade, deep learning has been applied significantly in various domains, such as image and video processing, machine translation and text processing. Speech recognition is not an exception as well. Early deep learning-based ASR systems mostly consider a hybrid approach where the acoustic model is replaced by a deep neural network and the rest of modules use the traditional approach \cite{mohamed2009deep, hinton2012deep, graves2013hybrid}.&#10;&#10;The recent trend of building ASR systems is to develop an end-to-end deep neural network. The network can therefore map the input speech sequence to a sequence of either graphemes, characters or words. In end-to-end ASR systems, the acoustic, pronunciation and language modules are trained jointly to optimize a common objective function and the network overcomes the limitations of traditional ASR systems. In the literature, there are generally two major end-to-end ASR architectures can be found. They are (a) Connectionist temporal classification (CTC)-based, and (b) Attention-based. CTC uses Markov assumptions to solve sequence-to-sequence problem with a forward-backward algorithm \cite{ctc}. Attention mechanism aligns the relevant speech frames for predicting symbols at each output time step \cite{chorowski2014end,chorowski2015attention}.&#10;&#10;The end-to-end ASR models are mainly based on an encoder-decoder architecture. The encoder part converts the speech frames and their temporal dependencies into a high level representation which will be used by the decoder for output predictions. The initial versions of the encoder-decoder architecture for ASR modelled with recurrent neural network (RNN) as the main component for sequence processing \cite{graves2012sequence, graves2013speech}. RNN is a type of artificial neural network which is typically used for modelling sequential data. Apart from the vanilla RNN, some other variations like long short-term memory (LSTM) \cite{hochreiter1997long}, gated recurrent unit (GRU) \cite{cho2014learning} are also popular in modelling sequential data. RNNs can be used in unidirectional as well as bi-directional fashion \cite{schuster1997bidirectional, graves2005bidirectional}. Convolutional neural networks (CNN) coupled with RNNs \cite{7953077} or stand-alone \cite{zhang2016towards} have also been used to make effective ASR models. Processing data sequentially is an inefficient process and may not capture temporal dependencies effectively. To address the limitations of RNN, Transformer network \cite{vaswani2017attention} has been recently proposed for sequence-to-sequence transduction. Transformer is a recurrence-free encoder-decoder architecture where sequence tokens are processed parallelly using self-attention mechanism.&#10;&#10;Automatic speech recognition operates in two different modes: offline (when recorded speech is available before transcription starts), and online or streaming (when transcription starts simultaneously as the speaker(s) starts speaking). In this paper, we have reviewed attention-based ASR literature for both offline and streaming speech recognition. While reviewing, we have only considered the models built with either recurrent neural network (RNN) or Transformer. Nowadays, ASR models are widely embedded in systems like smart devices and chatbots. In addition, application of attention mechanism is showing great potential in achieving higher effectiveness and efficiency for ASR. From the middle of last decade, a lot of progress has been made on attention-based models. Recently, some survey papers \cite{chaudhari2019attentive, galassi2020attention} have presented the development of attention-based models on natural language processing (NLP). These survey papers have documented the advancement of a wide range of NLP applications like machine translation, text and document classification, text summarisation, question answering, sentiment analysis, and speech processing. However, the existing literature still lacks a survey specifically targeted on the evolution of attention-based models for ASR. Therefore, we have been motivated to write this paper.&#10;&#10;The rest of paper is organised as follows. Section \ref{sec:attention} provides a simple explanation of Attention mechanism. A brief introduction to attention-based encoder-decoder architecture is discussed in Section \ref{sec:attention-aed}. Section \ref{sec:offline} discusses the evolution of offline speech recognition followed by the evolution of streaming speech recognition in Section \ref{sec:streaming}. Finally Section \ref{sec:conclusion} concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Attention Models, Neural Networks, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1003.4083" label="1003.4083">
        <attvalues>
          <attvalue for="0" value="Voice Recognition Algorithms using Mel Frequency Cepstral Coefficient&#10;  (MFCC) and Dynamic Time Warping (DTW) Techniques" />
          <attvalue for="1" value="  Digital processing of speech signal and voice recognition algorithm is very&#10;important for fast and accurate automatic voice recognition technology. The&#10;voice is a signal of infinite information. A direct analysis and synthesizing&#10;the complex voice signal is due to too much information contained in the&#10;signal. Therefore the digital signal processes such as Feature Extraction and&#10;Feature Matching are introduced to represent the voice signal. Several methods&#10;such as Liner Predictive Predictive Coding (LPC), Hidden Markov Model (HMM),&#10;Artificial Neural Network (ANN) and etc are evaluated with a view to identify a&#10;straight forward and effective method for voice signal. The extraction and&#10;matching process is implemented right after the Pre Processing or filtering&#10;signal is performed. The non-parametric method for modelling the human auditory&#10;perception system, Mel Frequency Cepstral Coefficients (MFCCs) are utilize as&#10;extraction techniques. The non linear sequence alignment known as Dynamic Time&#10;Warping (DTW) introduced by Sakoe Chiba has been used as features matching&#10;techniques. Since it's obvious that the voice signal tends to have different&#10;temporal rate, the alignment is important to produce the better&#10;performance.This paper present the viability of MFCC to extract features and&#10;DTW to compare the test patterns.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1412.1602" label="1412.1602">
        <attvalues>
          <attvalue for="0" value="End-to-end Continuous Speech Recognition using Attention-based Recurrent&#10;  NN: First Results" />
          <attvalue for="1" value="  We replace the Hidden Markov Model (HMM) which is traditionally used in in&#10;continuous speech recognition with a bi-directional recurrent neural network&#10;encoder coupled to a recurrent neural network decoder that directly emits a&#10;stream of phonemes. The alignment between the input and output sequences is&#10;established using an attention mechanism: the decoder emits each symbol based&#10;on a context created with a subset of input symbols elected by the attention&#10;mechanism. We report initial results demonstrating that this new approach&#10;achieves phoneme error rates that are comparable to the state-of-the-art&#10;HMM-based decoders, on the TIMIT dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.03022" label="1610.03022">
        <attvalues>
          <attvalue for="0" value="Very Deep Convolutional Networks for End-to-End Speech Recognition" />
          <attvalue for="1" value="  Sequence-to-sequence models have shown success in end-to-end speech&#10;recognition. However these models have only used shallow acoustic encoder&#10;networks. In our work, we successively train very deep convolutional networks&#10;to add more expressive power and better generalization for end-to-end ASR&#10;models. We apply network-in-network principles, batch normalization, residual&#10;connections and convolutional LSTMs to build very deep recurrent and&#10;convolutional structures. Our models exploit the spectral structure in the&#10;feature space and add computational depth without overfitting issues. We&#10;experiment with the WSJ ASR task and achieve 10.5\% word error rate without any&#10;dictionary or language using a 15 layer deep network.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1701.02720" label="1701.02720">
        <attvalues>
          <attvalue for="0" value="Towards End-to-End Speech Recognition with Deep Convolutional Neural&#10;  Networks" />
          <attvalue for="1" value="  Convolutional Neural Networks (CNNs) are effective models for reducing&#10;spectral variations and modeling spectral correlations in acoustic features for&#10;automatic speech recognition (ASR). Hybrid speech recognition systems&#10;incorporating CNNs with Hidden Markov Models/Gaussian Mixture Models&#10;(HMMs/GMMs) have achieved the state-of-the-art in various benchmarks.&#10;Meanwhile, Connectionist Temporal Classification (CTC) with Recurrent Neural&#10;Networks (RNNs), which is proposed for labeling unsegmented sequences, makes it&#10;feasible to train an end-to-end speech recognition system instead of hybrid&#10;settings. However, RNNs are computationally expensive and sometimes difficult&#10;to train. In this paper, inspired by the advantages of both CNNs and the CTC&#10;approach, we propose an end-to-end speech framework for sequence labeling, by&#10;combining hierarchical CNNs with CTC directly without recurrent connections. By&#10;evaluating the approach on the TIMIT phoneme recognition task, we show that the&#10;proposed model is not only computationally efficient, but also competitive with&#10;the existing baseline systems. Moreover, we argue that CNNs have the capability&#10;to model temporal correlations with appropriate context information.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.02874" label="1904.02874">
        <attvalues>
          <attvalue for="0" value="An Attentive Survey of Attention Models" />
          <attvalue for="1" value="  Attention Model has now become an important concept in neural networks that&#10;has been researched within diverse application domains. This survey provides a&#10;structured and comprehensive overview of the developments in modeling&#10;attention. In particular, we propose a taxonomy which groups existing&#10;techniques into coherent categories. We review salient neural architectures in&#10;which attention has been incorporated, and discuss applications in which&#10;modeling attention has shown a significant impact. We also describe how&#10;attention has been used to improve the interpretability of neural networks.&#10;Finally, we discuss some future research directions in attention. We hope this&#10;survey will provide a succinct introduction to attention models and guide&#10;practitioners while developing approaches for their applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.14103" label="2310.14103">
        <attvalues>
          <attvalue for="0" value="Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial&#10;  Applications" />
          <attvalue for="1" value="  Instruction Fine-Tuning (IFT) is a powerful paradigm that strengthens the&#10;zero-shot capabilities of Large Language Models (LLMs), but in doing so induces&#10;new evaluation metric requirements. We show LLM-based metrics to be well&#10;adapted to these requirements, and leverage them to conduct an investigation of&#10;task-specialization strategies, quantifying the trade-offs that emerge in&#10;practical industrial settings. Our findings offer practitioners actionable&#10;insights for real-world IFT model deployment.&#10;" />
          <attvalue for="2" value="&#10;&#10;Adapting pre-trained language models (LMs) for specific applications is central in industrial NLP to unlock task-specific performance gains and strengthen model alignment with industry requirements. A paradigm gaining traction is the use of instruction fine-tuned (IFT) models, LMs capable of following arbitrary instructions expressed in natural language \cite{wei2022finetuned, sanh2022multitask, ouyang2022training}.&#10;&#10;Researchers primarily concentrate on improving general-purpose IFT models to be used as versatile agents capable of executing instructions expressed in natural language \cite{li2023camel, zhou2023lima, xu2023wizardlm}. &#10;In an industrial setting, prompting ChatGPT to improve the wording of an email, or to assist with a code snippet would be instances of this zero-shot utilization scenario, which we define as $\mathcal{S}_0$. &#10;Critical industrial LLM applications may however not always align with $\mathcal{S}_0$, and often prioritize two other settings. The first scenario, $\mathcal{S}_1$, requires extending a generalist IFT model's capabilities to new specific tasks not included in the original instruction training set. The second scenario, $\mathcal{S}_2$, centers around converting IFT models into specialized models proficient exclusively on specific tasks. &#10;In $\mathcal{S}_1$ for instance, a large company may want an LLM assistant for internal employee use, and decide to extend an openly available Chat model by training it to write memos with a specific templating scheme, to respond to internal FAQs, and to use internal coding tools, all the while retaining the original chat assistant's general purpose abilities. In $\mathcal{S}_2$, that same company is only interested in a given specific task; extracting specific information from business documents, and specializes an IFT model for that purpose, aiming to leverage prompting and the generalization capabilities of the model for a more data-efficient training.&#10;&#10;In this paper, we thoroughly examine $\mathcal{S}_1$ and $\mathcal{S}_2$ by investigating the learning dynamics of specializing IFT models through a practical lens. To ensure the reliability of our tooling and the rigor of our conclusions, we first undertake a critical assessment of the current evaluation practices employed for IFT models. Formally, our contributions are:&#10;&#10;Contribution 1. IFT models are designed to handle tasks of diverse natures and varying difficulties. However, current metrics used to measure their performance are often task-specific \cite{zellers2019hellaswag, eval-harness}, or rely on automatic metrics designed for other intended purposes \cite{papineni-etal-2002-bleu, lin-2004-rouge}. To address this limitation, we introduce two new requirements for metrics used to evaluate IFT models: \underline{C}omparability \underline{A}cross \underline{T}ask (CAT) and \underline{T}ask and \underline{F}ormat \underline{A}gnostism (TFA). CAT imposes for metric scores to exhibit consistency across a diverse set of generative tasks, in contrast to the sole traditional focus of consistency within a specific task. TFA defines the need for metrics to demonstrate robustness to variations in the output formats. By highlighting the shortcomings of existing metrics in meeting CAT and TFA, we present compelling evidence that using LLMs as scoring agents is a viable evaluation alternative of IFT models. &#10;&#10;Contribution 2. We approach our examination of $\mathcal{S}_1$ and $\mathcal{S}_2$ from a practical perspective and focus on the trade-off between data availability and overall performance. Our analysis uncovers two distinct phases of learning during IFT model specialization: learning to format, and learning to solve tasks. Subsequently, we showcase how practitioners can (i) leverage synthetic data to facilitate learning the desired formatting aspects and (ii) use IFT models to reduce the need of expert data in industrial scenarios. Our study provides practical insights and actionable recommendations to practitioners looking to deploy IFT models in production settings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Mathematics, Industrial Deployment Insights, Natural Language Processing, Task Specialization Strategies" />
        </attvalues>
      </node>
      <node id="2211.15613" label="2211.15613">
        <attvalues>
          <attvalue for="0" value="Frustratingly Easy Label Projection for Cross-lingual Transfer" />
          <attvalue for="1" value="  Translating training data into many languages has emerged as a practical&#10;solution for improving cross-lingual transfer. For tasks that involve&#10;span-level annotations, such as information extraction or question answering,&#10;an additional label projection step is required to map annotated spans onto the&#10;translated texts. Recently, a few efforts have utilized a simple&#10;mark-then-translate method to jointly perform translation and projection by&#10;inserting special markers around the labeled spans in the original sentence.&#10;However, as far as we are aware, no empirical analysis has been conducted on&#10;how this approach compares to traditional annotation projection based on word&#10;alignment. In this paper, we present an extensive empirical study across 57&#10;languages and three tasks (QA, NER, and Event Extraction) to evaluate the&#10;effectiveness and limitations of both methods, filling an important gap in the&#10;literature. Experimental results show that our optimized version of&#10;mark-then-translate, which we call EasyProject, is easily applied to many&#10;languages and works surprisingly well, outperforming the more complex word&#10;alignment-based methods. We analyze several key factors that affect the&#10;end-task performance, and show EasyProject works well because it can accurately&#10;preserve label span boundaries after translation. We will publicly release all&#10;our code and data.&#10;" />
          <attvalue for="2" value="&#10;&#10;Zero-shot cross-lingual transfer, where models trained on a source language (e.g., English) are directly applied to other target languages, has the potential to extend NLP systems to many languages \cite{nooralahzadeh-etal-2020-zero, keung-etal-2020-dont,chen2021model, niu-etal-2022-onealigner, huang-etal-2022-multilingual-generative}. Yet, its performance still lags behind models that are directly fine-tuned on labeled data (if available) from the target language.&#10;Recent work has shown that combining training data in a source language together with its automatic translation to the target language leads to consistent performance improvements \cite{xue2021mt5,hu2020xtreme}.&#10;However, for NLP tasks that involve span-level annotations, an additional label projection step is needed to map the span annotations onto the translated texts (see Figure \ref{fig:EasyProject}).&#10;&#10;Traditionally, this annotation projection step is performed based on word alignment after machine translation \cite{akbik2015generating, aminian-etal-2019-cross}. To avoid the use of complex word alignment models, several recent efforts \cite{lewis2020mlqa, hu2020xtreme} directly translated sentences with span annotations wrapped between special markers (e.g., {&lt;a&gt;} and {&lt;/a&gt;}). However, due to limited analysis presented in prior work, it is unclear (1) how well this approach works across different language families, (2) how robust MT systems are in handling special markers, as inserting markers inevitably degrades the translation quality, and (3) how well marker-based projection works in comparison to traditional alignment-based methods.&#10;&#10;In this paper, we present the first systematic study of the mark-then-translate annotation projection technique, which includes careful evaluation of the choice of markers, projection accuracy, impact on translation quality, robustness to different MT systems, as well as a comparison to traditional alignment-based method across 57 languages (including 18 from Africa) on 5 datasets and 3 NLP tasks. We also propose an improved variant of marker-based projection, EasyProject, that consistently outperforms the alignment-based approach, while being incredibly easy to use to project a variety of annotations (QA, entities, relations, events) across many languages. The key is to use language-agnostic square bracket markers, combined with an efficient fine-tuning strategy to encourage the multilingual MT system to better preserve the special markers during translation.&#10;&#10;Our main findings include (1) the marker-based method is surprisingly robust across different translation systems and languages, but the choice of markers matters ($\S$\ref{sec:choice-of-markers}); (2) EasyProject can project annotated spans more accurately and is better at preserving span boundaries than the alignment-based methods, which is key to its success ($\S$\ref{sec:comparsion_to_alignment}); (3) fine-tuning an MT system for only 200 steps is sufficient to improve its robustness in handling special markers during translation ($\S$\ref{sec:easyproject_intro}); (4) the margin of improved cross-lingual transfer is related to the language/script family and amount of pre-training data included in the multilingual model ($\S$\ref{sec:pre-training-data-size}). &#10;We hope our work will inspire more research on robust models that better handle text markup for the purpose of generating span annotations.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Cross-Lingual Transfer, Computational Linguistics, Multilingual Information Extraction, Artificial Intelligence, Annotation Projection Methods, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2003.02739" label="2003.02739">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Cross-Lingual Transfer with Meta Learning" />
          <attvalue for="1" value="  Learning what to share between tasks has been a topic of great importance&#10;recently, as strategic sharing of knowledge has been shown to improve&#10;downstream task performance. This is particularly important for multilingual&#10;applications, as most languages in the world are under-resourced. Here, we&#10;consider the setting of training models on multiple different languages at the&#10;same time, when little or no data is available for languages other than&#10;English. We show that this challenging setup can be approached using&#10;meta-learning, where, in addition to training a source language model, another&#10;model learns to select which training instances are the most beneficial to the&#10;first. We experiment using standard supervised, zero-shot cross-lingual, as&#10;well as few-shot cross-lingual settings for different natural language&#10;understanding tasks (natural language inference, question answering). Our&#10;extensive experimental setup demonstrates the consistent effectiveness of&#10;meta-learning for a total of 15 languages. We improve upon the state-of-the-art&#10;for zero-shot and few-shot NLI (on MultiNLI and XNLI) and QA (on the MLQA&#10;dataset). A comprehensive error analysis indicates that the correlation of&#10;typological features between languages can partly explain when parameter&#10;sharing learned via meta-learning is beneficial.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.06127" label="2010.06127">
        <attvalues>
          <attvalue for="0" value="Model Selection for Cross-Lingual Transfer" />
          <attvalue for="1" value="  Transformers that are pre-trained on multilingual corpora, such as, mBERT and&#10;XLM-RoBERTa, have achieved impressive cross-lingual transfer capabilities. In&#10;the zero-shot transfer setting, only English training data is used, and the&#10;fine-tuned model is evaluated on another target language. While this works&#10;surprisingly well, substantial variance has been observed in target language&#10;performance between different fine-tuning runs, and in the zero-shot setup, no&#10;target-language development data is available to select among multiple&#10;fine-tuned models. Prior work has relied on English dev data to select among&#10;models that are fine-tuned with different learning rates, number of steps and&#10;other hyperparameters, often resulting in suboptimal choices. In this paper, we&#10;show that it is possible to select consistently better models when small&#10;amounts of annotated data are available in auxiliary pivot languages. We&#10;propose a machine learning approach to model selection that uses the fine-tuned&#10;model's own internal representations to predict its cross-lingual capabilities.&#10;In extensive experiments we find that this method consistently selects better&#10;models than English validation data across twenty five languages (including&#10;eight low-resource languages), and often achieves results that are comparable&#10;to model selection using target language development data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.15610" label="2402.15610">
        <attvalues>
          <attvalue for="0" value="Selective &quot;Selective Prediction&quot;: Reducing Unnecessary Abstention in&#10;  Vision-Language Reasoning" />
          <attvalue for="1" value="  Selective prediction minimizes incorrect predictions from vision-language&#10;models (VLMs) by allowing them to abstain from answering when uncertain.&#10;However, when deploying a vision-language system with low tolerance for&#10;inaccurate predictions, selective prediction may be over-cautious and abstain&#10;too frequently, even on many correct predictions. We introduce ReCoVERR, an&#10;inference-time algorithm to reduce the over-abstention of a selective&#10;vision-language system without increasing the error rate of the system's&#10;predictions. When the VLM makes a low-confidence prediction, instead of&#10;abstaining ReCoVERR tries to find relevant clues in the image that provide&#10;additional evidence for the prediction. ReCoVERR uses an LLM to pose related&#10;questions to the VLM, collects high-confidence evidences, and if enough&#10;evidence confirms the prediction the system makes a prediction instead of&#10;abstaining. ReCoVERR enables three VLMs (BLIP2, InstructBLIP, and LLaVA-1.5) to&#10;answer up to 20% more questions on the VQAv2 and A-OKVQA tasks without&#10;decreasing system accuracy, thus improving overall system reliability. Our code&#10;is available at https://github.com/tejas1995/ReCoVERR.&#10;" />
          <attvalue for="2" value="&#10;&#10;Instruction-tuned vision-and-language models (VLMs)~\cite{dai2023instructblip, liu2023improved, laurencon2023obelics, bai2023qwen} have achieved strong accuracy on reasoning benchmarks, which typically require VLMs to produce an answer for each instance. &#10;For downstream use, however, these systems should abstain from answering when uncertain (e.g., by saying ``I don't know&quot;)~\cite{rajpurkar2018know}. &#10;Selective prediction systems~\cite{de2000reject, el2010foundations} aim to balance the number of predictions made (coverage) and the error rate on predicted instances (risk).&#10;&#10;However, a vanilla selective prediction system with low tolerance for incorrect predictions will abstain too frequently to be practical, even when the model answer may be correct~\cite{whitehead2022reliable}.&#10;For example, if a user specifies that the BLIP2~\cite{li2023blip} predictions should be right at least 90\% of the time, vanilla selective prediction will make a prediction for just 4\% of A-OKVQA~\cite{schwenk2022okvqa} questions, with 94\% of the correct predictions being abstained on.&#10;&#10;We introduce ReCoVERR\ (Reason by Collecting Visual Evidences that are Reliable and Relevant), an algorithm that increases the number of questions that a selective VLM system can answer confidently while adhering to a specified risk tolerance, without any additional training. &#10;When the VLM is uncertain about its prediction for a given question, instead of abstaining outright, ReCoVERR tries to verify the prediction by recovering reliable supporting (or contradicting) evidence. &#10;This ability is predicated on two characteristics of VLMs. &#10;First, VLMs can produce well-calibrated confidence estimates (\S~\ref{subsubsec:vlm-calib}).&#10;Second, VLMs can often correctly and confidently recover information in the image that entails a low-confidence initial prediction.&#10;For instance, in Figure~\ref{fig:teaser}, when asked about the number of floor tile colors, a VLM (here, BLIP2) correctly answers ``two'' but with low confidence: simple threshold-based abstention would abstain on this instance.&#10;But, when asked to identify the colors of the floor tiles, the model confidently answers ``red and white''. &#10;While it is obvious that ``red and white'' entails ``two colors'', the VLM was unable to confidently make that inference.&#10;Based on these two insights, ReCoVERR searches for additional visual evidence by iteratively posing relevant questions to the VLM, and collecting answers as visual evidence if they are: a) reliable, i.e, the VLM is highly confident in its answer, and b) relevant to the question the VLM is trying to answer in the first place. &#10;&#10;We experiment on the VQAv2 and A-OKVQA visual reasoning tasks using three VLMs: BLIP2, InstructBLIP and LLaVA-1.5. &#10;For all three VLMs, ReCoVERR\ substantially increases the number of questions answered by the selective VLM system, while keeping system risk under the specified risk tolerance (Section~\ref{sec:results}). &#10;ReCoVERR\ is particularly helpful for BLIP2, which has not been trained on the target task, by improving coverage by 20\% and recall by 25-30\%. &#10;Our analysis reveals that the ability to give accurate confidence estimates, and high estimates for correct predictions, is crucial to ReCoVERR's performance. &#10;Further experiments demonstrate the importance of ensuring evidences are both reliable and relevant (Section~\ref{subsec:ablation}), and that ReCoVERR\ tuned for a single task can be directly applied to new tasks without further tuning (Section~\ref{subsec:task-generalization}). &#10;Our findings suggest that ReCoVERR\ is a promising solution towards building more reliable multimodal reasoning systems.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Selective Prediction Methods, Linguistics, Computer Vision, Inference Time Algorithms, Artificial Intelligence, Natural Language Processing, Vision Language Models" />
        </attvalues>
      </node>
      <node id="2305.06500" label="2305.06500">
        <attvalues>
          <attvalue for="0" value="InstructBLIP: Towards General-purpose Vision-Language Models with&#10;  Instruction Tuning" />
          <attvalue for="1" value="  Large-scale pre-training and instruction tuning have been successful at&#10;creating general-purpose language models with broad competence. However,&#10;building general-purpose vision-language models is challenging due to the rich&#10;input distributions and task diversity resulting from the additional visual&#10;input. Although vision-language pretraining has been widely studied,&#10;vision-language instruction tuning remains under-explored. In this paper, we&#10;conduct a systematic and comprehensive study on vision-language instruction&#10;tuning based on the pretrained BLIP-2 models. We gather 26 publicly available&#10;datasets, covering a wide variety of tasks and capabilities, and transform them&#10;into instruction tuning format. Additionally, we introduce an instruction-aware&#10;Query Transformer, which extracts informative features tailored to the given&#10;instruction. Trained on 13 held-in datasets, InstructBLIP attains&#10;state-of-the-art zero-shot performance across all 13 held-out datasets,&#10;substantially outperforming BLIP-2 and larger Flamingo models. Our models also&#10;lead to state-of-the-art performance when finetuned on individual downstream&#10;tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts).&#10;Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over&#10;concurrent multimodal models. All InstructBLIP models are open-sourced at&#10;https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.13631" label="2204.13631">
        <attvalues>
          <attvalue for="0" value="Reliable Visual Question Answering: Abstain Rather Than Answer&#10;  Incorrectly" />
          <attvalue for="1" value="  Machine learning has advanced dramatically, narrowing the accuracy gap to&#10;humans in multimodal tasks like visual question answering (VQA). However, while&#10;humans can say &quot;I don't know&quot; when they are uncertain (i.e., abstain from&#10;answering a question), such ability has been largely neglected in multimodal&#10;research, despite the importance of this problem to the usage of VQA in real&#10;settings. In this work, we promote a problem formulation for reliable VQA,&#10;where we prefer abstention over providing an incorrect answer. We first enable&#10;abstention capabilities for several VQA models, and analyze both their&#10;coverage, the portion of questions answered, and risk, the error on that&#10;portion. For that, we explore several abstention approaches. We find that&#10;although the best performing models achieve over 70% accuracy on the VQA v2&#10;dataset, introducing the option to abstain by directly using a model's softmax&#10;scores limits them to answering less than 7.5% of the questions to achieve a&#10;low risk of error (i.e., 1%). This motivates us to utilize a multimodal&#10;selection function to directly estimate the correctness of the predicted&#10;answers, which we show can increase the coverage by, for example, 2.3x from&#10;6.8% to 15.6% at 1% risk. While it is important to analyze both coverage and&#10;risk, these metrics have a trade-off which makes comparing VQA models&#10;challenging. To address this, we also propose an Effective Reliability metric&#10;for VQA that places a larger cost on incorrect answers compared to abstentions.&#10;This new problem formulation, metric, and analysis for VQA provide the&#10;groundwork for building effective and reliable VQA models that have the&#10;self-awareness to abstain if and only if they don't know the answer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.01718" label="2206.01718">
        <attvalues>
          <attvalue for="0" value="A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge" />
          <attvalue for="1" value="  The Visual Question Answering (VQA) task aspires to provide a meaningful&#10;testbed for the development of AI models that can jointly reason over visual&#10;and natural language inputs. Despite a proliferation of VQA datasets, this goal&#10;is hindered by a set of common limitations. These include a reliance on&#10;relatively simplistic questions that are repetitive in both concepts and&#10;linguistic structure, little world knowledge needed outside of the paired&#10;image, and limited reasoning required to arrive at the correct answer. We&#10;introduce A-OKVQA, a crowdsourced dataset composed of a diverse set of about&#10;25K questions requiring a broad base of commonsense and world knowledge to&#10;answer. In contrast to the existing knowledge-based VQA datasets, the questions&#10;generally cannot be answered by simply querying a knowledge base, and instead&#10;require some form of commonsense reasoning about the scene depicted in the&#10;image. We demonstrate the potential of this new dataset through a detailed&#10;analysis of its contents and baseline performance measurements over a variety&#10;of state-of-the-art vision-language models. Project page:&#10;http://a-okvqa.allenai.org/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00932" label="2005.00932">
        <attvalues>
          <attvalue for="0" value="Improving Non-autoregressive Neural Machine Translation with Monolingual&#10;  Data" />
          <attvalue for="1" value="  Non-autoregressive (NAR) neural machine translation is usually done via&#10;knowledge distillation from an autoregressive (AR) model. Under this framework,&#10;we leverage large monolingual corpora to improve the NAR model's performance,&#10;with the goal of transferring the AR model's generalization ability while&#10;preventing overfitting. On top of a strong NAR baseline, our experimental&#10;results on the WMT14 En-De and WMT16 En-Ro news translation tasks confirm that&#10;monolingual data augmentation consistently improves the performance of the NAR&#10;model to approach the teacher AR model's performance, yields comparable or&#10;better results than the best non-iterative NAR methods in the literature and&#10;helps reduce overfitting in the training process.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural machine translation (NMT) \cite{sutskever2014sequence, bahdanau2014neural} has achieved impressive performance in recent years, but the autoregressive decoding process limits the translation speed and restricts low-latency applications. To mitigate this issue, many non-autoregressive (NAR) translation methods have been proposed, including latent space models \cite{gu2017non, ma2019flowseq, shu2019latent}, iterative refinement methods \cite{lee2018deterministic, ghazvininejad2019mask}, and alternative loss functions \cite{libovicky2018end, wang2019non, wei2019imitation, li2019hint, shao2019minimizing}.&#10;The decoding speedup for NAR models is typically 2-15$\times$ depending on the specific setup (e.g., the number of length candidates, number of latent samples, etc.), and NAR models can be tuned to achieve different trade-offs between time complexity and decoding quality \cite{gu2017non, wei2019imitation, ghazvininejad2019mask, ma2019flowseq}.&#10;&#10;Although different in various aspects, all of these methods are based on transformer modules \cite{vaswani2017attention}, and depend on a well-trained AR model to obtain its output translations to create targets for NAR model training. This training setup is well-suited to leverage external monolingual data, since the target side of the NAR training corpus is always generated by an AR model. Techniques like backtranslation \cite{sennrich2015improving} are known to improve MT performance using monolingual data alone. However, to the best of our knowledge, monolingual data augmentation for NAR-MT has not been reported in the literature.&#10;&#10;In typical NAR-MT model training, an AR teacher provides a consistent supervision signal for the NAR model; the source text that was used to train the teacher is decoded by the teacher to create synthetic target text. In this work, we use a large amount of source text from monolingual corpora to generate additional teacher outputs for NAR-MT training.&#10;&#10;We use a transformer model with minor structural changes to perform NAR generation in a non-iterative way, which establishes stronger baselines than most of the previous methods. We demonstrate that generating additional training data with monolingual corpora consistently improves the translation quality of our baseline NAR system on the WMT14 En-De and WMT16 En-Ro translation tasks. Furthermore, our experiments show that NAR models trained with increasing amount of extra monolingual data are less prone to overfitting and generalize better on longer sentences.&#10;&#10;In addition, we have obtained Ro$arrow$En and En$arrow$De results which are state-of-the-art for non-iterative NAR-MT, just by using more monolingual data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Knowledge Distillation, Artificial Intelligence, Monolingual Data Augmentation, Statistics, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1811.04719" label="1811.04719">
        <attvalues>
          <attvalue for="0" value="End-to-End Non-Autoregressive Neural Machine Translation with&#10;  Connectionist Temporal Classification" />
          <attvalue for="1" value="  Autoregressive decoding is the only part of sequence-to-sequence models that&#10;prevents them from massive parallelization at inference time.&#10;Non-autoregressive models enable the decoder to generate all output symbols&#10;independently in parallel. We present a novel non-autoregressive architecture&#10;based on connectionist temporal classification and evaluate it on the task of&#10;neural machine translation. Unlike other non-autoregressive methods which&#10;operate in several steps, our model can be trained end-to-end. We conduct&#10;experiments on the WMT English-Romanian and English-German datasets. Our models&#10;achieve a significant speedup over the autoregressive models, keeping the&#10;translation quality comparable to other non-autoregressive models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.10976" label="2202.10976">
        <attvalues>
          <attvalue for="0" value="DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised&#10;  Learning" />
          <attvalue for="1" value="  Any-to-any voice conversion problem aims to convert voices for source and&#10;target speakers, which are out of the training data. Previous works wildly&#10;utilize the disentangle-based models. The disentangle-based model assumes the&#10;speech consists of content and speaker style information and aims to untangle&#10;them to change the style information for conversion. Previous works focus on&#10;reducing the dimension of speech to get the content information. But the size&#10;is hard to determine to lead to the untangle overlapping problem. We propose&#10;the Disentangled Representation Voice Conversion (DRVC) model to address the&#10;issue. DRVC model is an end-to-end self-supervised model consisting of the&#10;content encoder, timbre encoder, and generator. Instead of the previous work&#10;for reducing speech size to get content, we propose a cycle for restricting the&#10;disentanglement by the Cycle Reconstruct Loss and Same Loss. The experiments&#10;show there is an improvement for converted speech on quality and voice&#10;similarity.&#10;" />
          <attvalue for="2" value="&#10;&#10;Voice conversion (VC) aims to generate a new voice with the source voice content and target speaker timbre \cite{DBLP:journals/taslp/LiuCWWLM21,tang2022avqvc,DBLP:conf/icassp/ChenSH21,DBLP:conf/icassp/HayashiHKT21}. VC models can be roughly named as $multiple_1$-to-$multiple_2$ models, with $multiple_1, multiple_2 \in \{one, many, any\}$, the $multiple_1, multiple_2$ represents the source speakers and the target speakers, respectively. $One$ means the speaker is fixed, whether the training or inferring process. $Many$ and $any$ represents the speaker is seen or unseen in the training process, respectively.&#10;&#10;One-to-one VC model is inefficient due to only being able to convert voice between a fixed pair of source speaker and target speaker, such as the CycleGAN-VC \cite{asru2021zhang,DBLP:journals/taslp/NakashikaTA15,asru2021tang}. Even though the any-to-one and many-to-one can work in uncertain source speaker\cite{DBLP:conf/icmcs/SunLWKM16}, but the target speaker is also fixed. For VC models with uncertain speaker pair, such as many-to-many~\cite{DBLP:journals/access/LeeKP21,DBLP:conf/icassp/WangY21} and any-to-any ~\cite{DBLP:conf/icassp/LinCLLL21}, widely utilize the disentanglement-based method. Disentanglement-based models assume that the speech consists of the content and speaker style information. They aim to split the two pieces of information from speeches and exchange the content to achieve the conversion task. But the challenge is to avoid the overlapping of untangling results~\cite{DBLP:conf/iclr/YuanCZHGC21,DBLP:conf/icassp/LiTYWXSM21}. AutoVC proposes to circumspection choose the content dimension to separate the content information before combining it with the pre-trained speaker information~\cite{DBLP:conf/icml/QianZCYH19}. But it is hard to determine the number of reduced sizes to avoid residual the source speaker information or loss of the content. The similar problem also exists in VQVC+, which proposes to use a codebook to obtain the content information by combining similar dimensions~\cite{DBLP:conf/interspeech/WuCL20}. The suitable codebook size is the key factor to get mostly content information without speakers' influence.&#10;&#10;The image-to-image (I2I) task aims to convert the target image style to the source image. Disentangled Representation for Image-to-Image Translation (DRIT) assumes image consists of content and attribute information, and two input images have same content \cite{DBLP:journals/ijcv/LeeTMHLSY20}. Besides, it novelty utilizes double exchange process for changing the content information, one for synthesis new image, one for reconstructing image, to reduce the overlapping of disentanglement.&#10;&#10;Inspired by the double exchange process of DRIT, we propose to use the process to address the untangle overlapping problem without circumspection choose the content size. The proposed end-to-end framework is Disentangled Representation Voice Conversion (DRVC). Comparing to DRIT, we believe neither of the content or style information is the same between the input speeches. Furthermore, we design a cycle framework for the double exchange of the style information with cycle loss and two discriminators. We experiment with the model on VCC2018, both the subjective and objective results show our model has better performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Voice Conversion, Machine Learning, Computer Science, Speech Processing, Linguistics, Disentanglement Models, Signal Processing, Speech Representation" />
        </attvalues>
      </node>
      <node id="2009.02725" label="2009.02725">
        <attvalues>
          <attvalue for="0" value="Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence&#10;  Modeling" />
          <attvalue for="1" value="  This paper proposes an any-to-many location-relative, sequence-to-sequence&#10;(seq2seq), non-parallel voice conversion approach, which utilizes text&#10;supervision during training. In this approach, we combine a bottle-neck feature&#10;extractor (BNE) with a seq2seq synthesis module. During the training stage, an&#10;encoder-decoder-based hybrid connectionist-temporal-classification-attention&#10;(CTC-attention) phoneme recognizer is trained, whose encoder has a bottle-neck&#10;layer. A BNE is obtained from the phoneme recognizer and is utilized to extract&#10;speaker-independent, dense and rich spoken content representations from&#10;spectral features. Then a multi-speaker location-relative attention based&#10;seq2seq synthesis model is trained to reconstruct spectral features from the&#10;bottle-neck features, conditioning on speaker representations for speaker&#10;identity control in the generated speech. To mitigate the difficulties of using&#10;seq2seq models to align long sequences, we down-sample the input spectral&#10;feature along the temporal dimension and equip the synthesis model with a&#10;discretized mixture of logistic (MoL) attention mechanism. Since the phoneme&#10;recognizer is trained with large speech recognition data corpus, the proposed&#10;approach can conduct any-to-many voice conversion. Objective and subjective&#10;evaluations show that the proposed any-to-many approach has superior voice&#10;conversion performance in terms of both naturalness and speaker similarity.&#10;Ablation studies are conducted to confirm the effectiveness of feature&#10;selection and model design strategies in the proposed approach. The proposed VC&#10;approach can readily be extended to support any-to-any VC (also known as&#10;one/few-shot VC), and achieve high performance according to objective and&#10;subjective evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.10020" label="2202.10020">
        <attvalues>
          <attvalue for="0" value="AVQVC: One-shot Voice Conversion by Vector Quantization with applying&#10;  contrastive learning" />
          <attvalue for="1" value="  Voice Conversion(VC) refers to changing the timbre of a speech while&#10;retaining the discourse content. Recently, many works have focused on&#10;disentangle-based learning techniques to separate the timbre and the linguistic&#10;content information from a speech signal. Once successful, voice conversion&#10;will be feasible and straightforward. This paper proposed a novel one-shot&#10;voice conversion framework based on vector quantization voice conversion (VQVC)&#10;and AutoVC, called AVQVC. A new training method is applied to VQVC to separate&#10;content and timbre information from speech more effectively. The result shows&#10;that this approach has better performance than VQVC in separating content and&#10;timbre to improve the sound quality of generated speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11646" label="2010.11646">
        <attvalues>
          <attvalue for="0" value="Towards Low-Resource StarGAN Voice Conversion using Weight Adaptive&#10;  Instance Normalization" />
          <attvalue for="1" value="  Many-to-many voice conversion with non-parallel training data has seen&#10;significant progress in recent years. StarGAN-based models have been interests&#10;of voice conversion. However, most of the StarGAN-based methods only focused on&#10;voice conversion experiments for the situations where the number of speakers&#10;was small, and the amount of training data was large. In this work, we aim at&#10;improving the data efficiency of the model and achieving a many-to-many&#10;non-parallel StarGAN-based voice conversion for a relatively large number of&#10;speakers with limited training samples. In order to improve data efficiency,&#10;the proposed model uses a speaker encoder for extracting speaker embeddings and&#10;conducts adaptive instance normalization (AdaIN) on convolutional weights.&#10;Experiments are conducted with 109 speakers under two low-resource situations,&#10;where the number of training samples is 20 and 5 per speaker. An objective&#10;evaluation shows the proposed model is better than the baseline methods.&#10;Furthermore, a subjective evaluation shows that, for both naturalness and&#10;similarity, the proposed model outperforms the baseline method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.06793" label="2104.06793">
        <attvalues>
          <attvalue for="0" value="Non-autoregressive sequence-to-sequence voice conversion" />
          <attvalue for="1" value="  This paper proposes a novel voice conversion (VC) method based on&#10;non-autoregressive sequence-to-sequence (NAR-S2S) models. Inspired by the great&#10;success of NAR-S2S models such as FastSpeech in text-to-speech (TTS), we extend&#10;the FastSpeech2 model for the VC problem. We introduce the&#10;convolution-augmented Transformer (Conformer) instead of the Transformer,&#10;making it possible to capture both local and global context information from&#10;the input sequence. Furthermore, we extend variance predictors to variance&#10;converters to explicitly convert the source speaker's prosody components such&#10;as pitch and energy into the target speaker. The experimental evaluation with&#10;the Japanese speaker dataset, which consists of male and female speakers of&#10;1,000 utterances, demonstrates that the proposed model enables us to perform&#10;more stable, faster, and better conversion than autoregressive S2S (AR-S2S)&#10;models such as Tacotron2 and Transformer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.04035" label="2208.04035">
        <attvalues>
          <attvalue for="0" value="TGAVC: Improving Autoencoder Voice Conversion with Text-Guided and&#10;  Adversarial Training" />
          <attvalue for="1" value="  Non-parallel many-to-many voice conversion remains an interesting but&#10;challenging speech processing task. Recently, AutoVC, a conditional autoencoder&#10;based method, achieved excellent conversion results by disentangling the&#10;speaker identity and the speech content using information-constraining&#10;bottlenecks. However, due to the pure autoencoder training method, it is&#10;difficult to evaluate the separation effect of content and speaker identity. In&#10;this paper, a novel voice conversion framework, named $\boldsymbol T$ext&#10;$\boldsymbol G$uided $\boldsymbol A$utoVC(TGAVC), is proposed to more&#10;effectively separate content and timbre from speech, where an expected content&#10;embedding produced based on the text transcriptions is designed to guide the&#10;extraction of voice content. In addition, the adversarial training is applied&#10;to eliminate the speaker identity information in the estimated content&#10;embedding extracted from speech. Under the guidance of the expected content&#10;embedding and the adversarial training, the content encoder is trained to&#10;extract speaker-independent content embedding from speech. Experiments on&#10;AIShell-3 dataset show that the proposed model outperforms AutoVC in terms of&#10;naturalness and similarity of converted speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14150" label="2010.14150">
        <attvalues>
          <attvalue for="0" value="FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and&#10;  Fusing Fine-Grained Voice Fragments With Attention" />
          <attvalue for="1" value="  Any-to-any voice conversion aims to convert the voice from and to any&#10;speakers even unseen during training, which is much more challenging compared&#10;to one-to-one or many-to-many tasks, but much more attractive in real-world&#10;scenarios. In this paper we proposed FragmentVC, in which the latent phonetic&#10;structure of the utterance from the source speaker is obtained from Wav2Vec&#10;2.0, while the spectral features of the utterance(s) from the target speaker&#10;are obtained from log mel-spectrograms. By aligning the hidden structures of&#10;the two different feature spaces with a two-stage training process, FragmentVC&#10;is able to extract fine-grained voice fragments from the target speaker&#10;utterance(s) and fuse them into the desired utterance, all based on the&#10;attention mechanism of Transformer as verified with analysis on attention maps,&#10;and is accomplished end-to-end. This approach is trained with reconstruction&#10;loss only without any disentanglement considerations between content and&#10;speaker information and doesn't require parallel data. Objective evaluation&#10;based on speaker verification and subjective evaluation with MOS both showed&#10;that this approach outperformed SOTA approaches, such as AdaIN-VC and AutoVC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.09420" label="2103.09420">
        <attvalues>
          <attvalue for="0" value="Improving Zero-shot Voice Style Transfer via Disentangled Representation&#10;  Learning" />
          <attvalue for="1" value="  Voice style transfer, also called voice conversion, seeks to modify one&#10;speaker's voice to generate speech as if it came from another (target) speaker.&#10;Previous works have made progress on voice conversion with parallel training&#10;data and pre-known speakers. However, zero-shot voice style transfer, which&#10;learns from non-parallel data and generates voices for previously unseen&#10;speakers, remains a challenging problem. We propose a novel zero-shot voice&#10;transfer method via disentangled representation learning. The proposed method&#10;first encodes speaker-related style and voice content of each input voice into&#10;separated low-dimensional embedding spaces, and then transfers to a new voice&#10;by combining the source content embedding and target style embedding through a&#10;decoder. With information-theoretic guidance, the style and content embedding&#10;spaces are representative and (ideally) independent of each other. On&#10;real-world VCTK datasets, our method outperforms other baselines and obtains&#10;state-of-the-art results in terms of transfer accuracy and voice naturalness&#10;for voice style transfer experiments under both many-to-many and zero-shot&#10;setups.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14804" label="2010.14804">
        <attvalues>
          <attvalue for="0" value="PPG-based singing voice conversion with adversarial representation&#10;  learning" />
          <attvalue for="1" value="  Singing voice conversion (SVC) aims to convert the voice of one singer to&#10;that of other singers while keeping the singing content and melody. On top of&#10;recent voice conversion works, we propose a novel model to steadily convert&#10;songs while keeping their naturalness and intonation. We build an end-to-end&#10;architecture, taking phonetic posteriorgrams (PPGs) as inputs and generating&#10;mel spectrograms. Specifically, we implement two separate encoders: one encodes&#10;PPGs as content, and the other compresses mel spectrograms to supply acoustic&#10;and musical information. To improve the performance on timbre and melody, an&#10;adversarial singer confusion module and a mel-regressive representation&#10;learning module are designed for the model. Objective and subjective&#10;experiments are conducted on our private Chinese singing corpus. Comparing with&#10;the baselines, our methods can significantly improve the conversion performance&#10;in terms of naturalness, melody, and voice similarity. Moreover, our PPG-based&#10;method is proved to be robust for noisy sources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.01270" label="1905.01270">
        <attvalues>
          <attvalue for="0" value="DRIT++: Diverse Image-to-Image Translation via Disentangled&#10;  Representations" />
          <attvalue for="1" value="  Image-to-image translation aims to learn the mapping between two visual&#10;domains. There are two main challenges for this task: 1) lack of aligned&#10;training pairs and 2) multiple possible outputs from a single input image. In&#10;this work, we present an approach based on disentangled representation for&#10;generating diverse outputs without paired training images. To synthesize&#10;diverse outputs, we propose to embed images onto two spaces: a domain-invariant&#10;content space capturing shared information across domains and a domain-specific&#10;attribute space. Our model takes the encoded content features extracted from a&#10;given input and attribute vectors sampled from the attribute space to&#10;synthesize diverse outputs at test time. To handle unpaired training data, we&#10;introduce a cross-cycle consistency loss based on disentangled representations.&#10;Qualitative results show that our model can generate diverse and realistic&#10;images on a wide range of tasks without paired training data. For quantitative&#10;evaluations, we measure realism with user study and Fr\'{e}chet inception&#10;distance, and measure diversity with the perceptual distance metric,&#10;Jensen-Shannon divergence, and number of statistically-different bins.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.06358" label="2309.06358">
        <attvalues>
          <attvalue for="0" value="Generative Data Augmentation using LLMs improves Distributional&#10;  Robustness in Question Answering" />
          <attvalue for="1" value="  Robustness in Natural Language Processing continues to be a pertinent issue,&#10;where state of the art models under-perform under naturally shifted&#10;distributions. In the context of Question Answering, work on domain adaptation&#10;methods continues to be a growing body of research. However, very little&#10;attention has been given to the notion of domain generalization under natural&#10;distribution shifts, where the target domain is unknown. With drastic&#10;improvements in the quality and access to generative models, we answer the&#10;question: How do generated datasets influence the performance of QA models&#10;under natural distribution shifts? We perform experiments on 4 different&#10;datasets under varying amounts of distribution shift, and analyze how&#10;&quot;in-the-wild&quot; generation can help achieve domain generalization. We take a&#10;two-step generation approach, generating both contexts and QA pairs to augment&#10;existing datasets. Through our experiments, we demonstrate how augmenting&#10;reading comprehension datasets with generated data leads to better robustness&#10;towards natural distribution shifts.&#10;" />
          <attvalue for="2" value="&#10;&#10;In this work, we perform a systematic study of how &quot;in-the-wild&quot; generation can affect the distributional robustness of question-answering models trained on the popular Stanford Question Answering Dataset (SQUAD) \cite{rajpurkar-etal-2016-squad}. Synthetic data generation is a widely adopted method for domain adaptation in QA systems \cite{shakeri2020end} \cite{yue2021contrastive} \cite{yue2022domain}. However, domain adaptation methods have access to unlabelled/labelled data belonging to the target domain, and do not account for unseen natural distribution shifts. Our work studies the effect of generated data on distribution shifts where the target domain is unseen. &#10;&#10;The conception of a dataset has undergone significant evolution in recent times. This transformation has been catalyzed by the advent of generative models trained 'in-the-wild', such as those described in \cite{brown2020language}, \cite{bubeck2023sparks}, and \cite{touvron2023llama}. These models, which use vast and diverse datasets across a range of domains, have facilitated the infusion of the web with synthesized data of high calibre, applicable to an extensive array of conceptual topics. Interestingly, these models are not merely confined to generation based on a pre-established distribution; they possess the capacity for repeated prompting, resulting in the creation of markedly diverse data. In the context of this emerging model paradigm, our research investigates the following query: How do generated datasets affect the distributional robustness of Question Answering models? Specifically, natural distribution shifts in NLP can arise due to differences in the text genre and style, text topics and vocabulary, demographics of the authors, medium of the text (written vs spoken), and other attributes \cite{wang-etal-2022-measure}. A key challenge is that NLP models trained on one data distribution often fail to generalize well to these naturally occurring shifts. For instance, \cite{miller2020effect} found that question answering models experienced average F1 score drops of 3.8 points on news articles, 14 points on Reddit posts, and 17.4 points on Amazon reviews compared to Wikipedia articles. This reveals brittleness of NLP models to natural distribution shifts.&#10;&#10;We present an overview of our generation setup in Figure \ref{fig:my_label}. For generating data, use GPT-3.5 \cite{brown2020language}, and create a question-answering dataset using questions provided in the SQUAD \cite{rajpurkar-etal-2016-squad} dataset. We use a dual generation approach, by first prompting the language model to generate a context for a question given in the SQUAD dataset, and then generating question-answer pairs for the newly generated context. &#10;&#10;Recent surveys, such as \cite{ramponi-plank-2020-neural}, discuss domain adaptation in NLP and divide approaches into data centric and model centric. We take a data-centric approach, as highlighted by findings from \cite{wang-etal-2022-measure} that demonstrate overlap in test-train data for QA models. The scarcity of research on generalization in QA models, especially with natural distribution shifts, is a motivation for our work, backed by observations from \cite{arora-etal-2021-types} on out-of-distribution data in NLP.&#10;&#10;Initial experiments like \cite{longpre-etal-2019-exploration} ventured into domain-agnostic question answering using data augmentation. New datasets introduced by \cite{miller2020effect}, sourced from various platforms, emphasize the effect of natural distribution shifts on QA models. While these studies provide extensive evaluations, our work builds on them by focusing on the impact of large language model (LLM)-generated datasets for QA tasks and further leveraging these datasets for our data augmentation method.&#10;&#10;The benefits of generated data have been explored by \cite{gowal2021improving}, showing its potential in adversarial robustness. \cite{bartolo-etal-2021-improving} and \cite{mekala-etal-2022-leveraging} use synthetic and context-generated data respectively for QA and text classification. Our method uses a GPT-3.5 model, as described by \cite{wei2022finetuned}, to generate context for questions. With similar motivations, \cite{bansal2023leaving} demonstrates the application of Stable Diffusion in diverse dataset creation for image tasks.&#10;&#10;\begin{tcolorbox}&#10;[colback=blue!5!white,colframe=blue!75!black,title=Our Contributions]&#10;\begin{itemize}&#10;[leftmargin=1mm]&#10;&#10;\item[\ding{224}] { &#10; {\fontfamily{phv}&#10; We propose a framework to improve the distributional robustness of reading comprehension models in the presence of natural distribution shifts.}&#10; }&#10;&#10;\item[\ding{224}] { &#10; {\fontfamily{phv}&#10; Through a thorough quantitative evaluation, we evaluate the capabilites of LLMs to generate high quality synthetic data for question answering tasks. }&#10; } &#10;&#10;\end{itemize}&#10;\end{tcolorbox}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Robustness Evaluation, Artificial Intelligence, Natural Language Processing, Domain Generalization" />
        </attvalues>
      </node>
      <node id="2010.06028" label="2010.06028">
        <attvalues>
          <attvalue for="0" value="End-to-End Synthetic Data Generation for Domain Adaptation of Question&#10;  Answering Systems" />
          <attvalue for="1" value="  We propose an end-to-end approach for synthetic QA data generation. Our model&#10;comprises a single transformer-based encoder-decoder network that is trained&#10;end-to-end to generate both answers and questions. In a nutshell, we feed a&#10;passage to the encoder and ask the decoder to generate a question and an answer&#10;token-by-token. The likelihood produced in the generation process is used as a&#10;filtering score, which avoids the need for a separate filtering model. Our&#10;generator is trained by fine-tuning a pretrained LM using maximum likelihood&#10;estimation. The experimental results indicate significant improvements in the&#10;domain adaptation of QA models outperforming current state-of-the-art methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.13854" label="2108.13854">
        <attvalues>
          <attvalue for="0" value="Contrastive Domain Adaptation for Question Answering using Limited Text&#10;  Corpora" />
          <attvalue for="1" value="  Question generation has recently shown impressive results in customizing&#10;question answering (QA) systems to new domains. These approaches circumvent the&#10;need for manually annotated training data from the new domain and, instead,&#10;generate synthetic question-answer pairs that are used for training. However,&#10;existing methods for question generation rely on large amounts of synthetically&#10;generated datasets and costly computational resources, which render these&#10;techniques widely inaccessible when the text corpora is of limited size. This&#10;is problematic as many niche domains rely on small text corpora, which&#10;naturally restricts the amount of synthetic data that can be generated. In this&#10;paper, we propose a novel framework for domain adaptation called contrastive&#10;domain adaptation for QA (CAQA). Specifically, CAQA combines techniques from&#10;question generation and domain-invariant learning to answer out-of-domain&#10;questions in settings with limited text corpora. Here, we train a QA system on&#10;both source data and generated data from the target domain with a contrastive&#10;adaptation loss that is incorporated in the training objective. By combining&#10;techniques from question generation and domain-invariant learning, our model&#10;achieved considerable improvements compared to state-of-the-art baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.04998" label="2209.04998">
        <attvalues>
          <attvalue for="0" value="Domain Adaptation for Question Answering via Question Classification" />
          <attvalue for="1" value="  Question answering (QA) has demonstrated impressive progress in answering&#10;questions from customized domains. Nevertheless, domain adaptation remains one&#10;of the most elusive challenges for QA systems, especially when QA systems are&#10;trained in a source domain but deployed in a different target domain. In this&#10;work, we investigate the potential benefits of question classification for QA&#10;domain adaptation. We propose a novel framework: Question Classification for&#10;Question Answering (QC4QA). Specifically, a question classifier is adopted to&#10;assign question classes to both the source and target data. Then, we perform&#10;joint training in a self-supervised fashion via pseudo-labeling. For&#10;optimization, inter-domain discrepancy between the source and target domain is&#10;reduced via maximum mean discrepancy (MMD) distance. We additionally minimize&#10;intra-class discrepancy among QA samples of the same question class for&#10;fine-grained adaptation performance. To the best of our knowledge, this is the&#10;first work in QA domain adaptation to leverage question classification with&#10;self-supervised adaptation. We demonstrate the effectiveness of the proposed&#10;QC4QA with consistent improvements against the state-of-the-art baselines on&#10;multiple datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.06827" label="2109.06827">
        <attvalues>
          <attvalue for="0" value="Types of Out-of-Distribution Texts and How to Detect Them" />
          <attvalue for="1" value="  Despite agreement on the importance of detecting out-of-distribution (OOD)&#10;examples, there is little consensus on the formal definition of OOD examples&#10;and how to best detect them. We categorize these examples by whether they&#10;exhibit a background shift or a semantic shift, and find that the two major&#10;approaches to OOD detection, model calibration and density estimation (language&#10;modeling for text), have distinct behavior on these types of OOD data. Across&#10;14 pairs of in-distribution and OOD English natural language understanding&#10;datasets, we find that density estimation methods consistently beat calibration&#10;methods in background shift settings, while performing worse in semantic shift&#10;settings. In addition, we find that both methods generally fail to detect&#10;examples from challenge data, highlighting a weak spot for current methods.&#10;Since no single method works well across all settings, our results call for an&#10;explicit definition of OOD examples when evaluating different detection&#10;methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.02145" label="1912.02145">
        <attvalues>
          <attvalue for="0" value="An Exploration of Data Augmentation and Sampling Techniques for&#10;  Domain-Agnostic Question Answering" />
          <attvalue for="1" value="  To produce a domain-agnostic question answering model for the Machine Reading&#10;Question Answering (MRQA) 2019 Shared Task, we investigate the relative&#10;benefits of large pre-trained language models, various data sampling&#10;strategies, as well as query and context paraphrases generated by&#10;back-translation. We find a simple negative sampling technique to be&#10;particularly effective, even though it is typically used for datasets that&#10;include unanswerable questions, such as SQuAD 2.0. When applied in conjunction&#10;with per-domain sampling, our XLNet (Yang et al., 2019)-based submission&#10;achieved the second best Exact Match and F1 in the MRQA leaderboard&#10;competition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.09468" label="2110.09468">
        <attvalues>
          <attvalue for="0" value="Improving Robustness using Generated Data" />
          <attvalue for="1" value="  Recent work argues that robust training requires substantially larger&#10;datasets than those required for standard classification. On CIFAR-10 and&#10;CIFAR-100, this translates into a sizable robust-accuracy gap between models&#10;trained solely on data from the original training set and those trained with&#10;additional data extracted from the &quot;80 Million Tiny Images&quot; dataset (TI-80M).&#10;In this paper, we explore how generative models trained solely on the original&#10;training set can be leveraged to artificially increase the size of the original&#10;training set and improve adversarial robustness to $\ell_p$ norm-bounded&#10;perturbations. We identify the sufficient conditions under which incorporating&#10;additional generated data can improve robustness, and demonstrate that it is&#10;possible to significantly reduce the robust-accuracy gap to models trained with&#10;additional real data. Surprisingly, we even show that even the addition of&#10;non-realistic random data (generated by Gaussian sampling) can improve&#10;robustness. We evaluate our approach on CIFAR-10, CIFAR-100, SVHN and&#10;TinyImageNet against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of&#10;size $\epsilon = 8/255$ and $\epsilon = 128/255$, respectively. We show large&#10;absolute improvements in robust accuracy compared to previous state-of-the-art&#10;methods. Against $\ell_\infty$ norm-bounded perturbations of size $\epsilon =&#10;8/255$, our models achieve 66.10% and 33.49% robust accuracy on CIFAR-10 and&#10;CIFAR-100, respectively (improving upon the state-of-the-art by +8.96% and&#10;+3.29%). Against $\ell_2$ norm-bounded perturbations of size $\epsilon =&#10;128/255$, our model achieves 78.31% on CIFAR-10 (+3.81%). These results beat&#10;most prior works that use external data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08678" label="2104.08678">
        <attvalues>
          <attvalue for="0" value="Improving Question Answering Model Robustness with Synthetic Adversarial&#10;  Data Generation" />
          <attvalue for="1" value="  Despite recent progress, state-of-the-art question answering models remain&#10;vulnerable to a variety of adversarial attacks. While dynamic adversarial data&#10;collection, in which a human annotator tries to write examples that fool a&#10;model-in-the-loop, can improve model robustness, this process is expensive&#10;which limits the scale of the collected data. In this work, we are the first to&#10;use synthetic adversarial data generation to make question answering models&#10;more robust to human adversaries. We develop a data generation pipeline that&#10;selects source passages, identifies candidate answers, generates questions,&#10;then finally filters or re-labels them to improve quality. Using this approach,&#10;we amplify a smaller human-written adversarial dataset to a much larger set of&#10;synthetic question-answer pairs. By incorporating our synthetic data, we&#10;improve the state-of-the-art on the AdversarialQA dataset by 3.7F1 and improve&#10;model generalisation on nine of the twelve MRQA datasets. We further conduct a&#10;novel human-in-the-loop evaluation to show that our models are considerably&#10;more robust to new human-written adversarial examples: crowdworkers can fool&#10;our model only 8.8% of the time on average, compared to 17.6% for a model&#10;trained without synthetic data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.02503" label="2302.02503">
        <attvalues>
          <attvalue for="0" value="Leaving Reality to Imagination: Robust Classification via Generated&#10;  Datasets" />
          <attvalue for="1" value="  Recent research on robustness has revealed significant performance gaps&#10;between neural image classifiers trained on datasets that are similar to the&#10;test set, and those that are from a naturally shifted distribution, such as&#10;sketches, paintings, and animations of the object categories observed during&#10;training. Prior work focuses on reducing this gap by designing engineered&#10;augmentations of training data or through unsupervised pretraining of a single&#10;large model on massive in-the-wild training datasets scraped from the Internet.&#10;However, the notion of a dataset is also undergoing a paradigm shift in recent&#10;years. With drastic improvements in the quality, ease-of-use, and access to&#10;modern generative models, generated data is pervading the web. In this light,&#10;we study the question: How do these generated datasets influence the natural&#10;robustness of image classifiers? We find that Imagenet classifiers trained on&#10;real data augmented with generated data achieve higher accuracy and effective&#10;robustness than standard training and popular augmentation strategies in the&#10;presence of natural distribution shifts. We analyze various factors influencing&#10;these results, including the choice of conditioning strategies and the amount&#10;of generated data. Additionally, we find that the standard ImageNet classifiers&#10;suffer a performance degradation of upto 20\% on the generated data, indicating&#10;their fragility at accurately classifying the objects under novel variations.&#10;Lastly, we demonstrate that the image classifiers, which have been trained on&#10;real data augmented with generated data from the base generative model, exhibit&#10;greater resilience to natural distribution shifts compared to the classifiers&#10;trained on real data augmented with generated data from the finetuned&#10;generative model on the real data. The code, models, and datasets are available&#10;at https://github.com/Hritikbansal/generative-robustness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.13064" label="2203.13064">
        <attvalues>
          <attvalue for="0" value="Ensembling and Knowledge Distilling of Large Sequence Taggers for&#10;  Grammatical Error Correction" />
          <attvalue for="1" value="  In this paper, we investigate improvements to the GEC sequence tagging&#10;architecture with a focus on ensembling of recent cutting-edge&#10;Transformer-based encoders in Large configurations. We encourage ensembling&#10;models by majority votes on span-level edits because this approach is tolerant&#10;to the model architecture and vocabulary size. Our best ensemble achieves a new&#10;SOTA result with an $F_{0.5}$ score of 76.05 on BEA-2019 (test), even without&#10;pre-training on synthetic datasets. In addition, we perform knowledge&#10;distillation with a trained ensemble to generate new synthetic training&#10;datasets, &quot;Troy-Blogs&quot; and &quot;Troy-1BW&quot;. Our best single sequence tagging model&#10;that is pretrained on the generated Troy-datasets in combination with the&#10;publicly available synthetic PIE dataset achieves a near-SOTA (To the best of&#10;our knowledge, our best single model gives way only to much heavier T5 model&#10;result with an $F_{0.5}$ score of 73.21 on BEA-2019 (test). The code, datasets,&#10;and trained models are publicly available).&#10;" />
          <attvalue for="2" value="&#10;&#10;The purpose of the Grammatical Error Correction (GEC) task is to correct grammatical errors in natural texts. This includes correcting errors in spelling, punctuation, grammar, morphology, word choice, and others. An intelligent GEC system receives text containing mistakes and produces its corrected version. The GEC task is complicated and challenging: the accuracy of edits, inference speed, and memory limitations are topics of intensive research.&#10;&#10;Currently, Machine Translation (MT) is the mainstream approach for GEC. In this setting, errorful sentences correspond to the source language, and error-free sentences correspond to the target language. Early GEC-MT methods leveraged phrase-based statistical machine translation (PBSMT) \cite{yuan-felice-2013-constrained}. Then this approach rapidly evolved to seq2seq Neural Machine Translation (NMT) based on gated recurrent neural networks \cite{yuan2016grammatical} and recent powerful Transformer-based seq2seq models. Transformer-based models autoregressively capture the full dependency among output tokens; however, inference can be slow due to sequential decoding. \cite{grundkiewicz-etal-2019-neural} leveraged a Transformer model \cite{vaswani2017attention} that was pre-trained on synthetic GEC data and right-to-left re-ranking for ensemble. \cite{kaneko2020encoder} adopted several strategies of BERT \cite{devlin2018bert} usage for GEC. Recently, \cite{rothe2021a} built their system on top of T5 \cite{xue-etal-2021-mt5}, a xxl version of the T5 Transformer encoder-decoder model and reached new state-of-the-art results (11B parameters). &#10;&#10;While still not as widespread as MT, the sequence tagging approach for GEC, which generates a sequence of text edit operations encoded by tags for errorful input text is becoming more common. LaserTagger \cite{malmi-etal-2019-encode} is a sequence tagging model that casts text generation as a text editing task. Corrected texts are reconstructed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. LaserTagger combines a BERT encoder with an autoregressive Transformer decoder, which predicts edit operations. The Parallel Iterative Edit (PIE) model \cite{awasthi2019parallel} does parallel decoding, achieving quality that is competitive with the seq2seq models. It predicts edits instead of tokens and iteratively refines predictions to capture dependencies.&#10;&#10;A similar approach is presented in \cite{omelianchuk2020gector}. The GECToR system achieves competitive results using various Transformers as an encoder; and linear layers with softmax for tag prediction and error detection. By replacing an autoregressive decoder with linear output layers, it’s also potentially several times faster than seq2seq systems.&#10;&#10;Today, the generation of synthetic data is becoming significant for most GEC models. Natural languages are rich, and their grammars contain many rules and exceptions; therefore, professional linguists are often utilized to annotate high-quality corpora for further training ML-based systems mostly in a supervised manner \cite{dahlmeier2013building}, \cite{bryant2019the}. However, human annotation is expensive, so researchers are working on methods for augmentation of training data, synthetic data generation, and strategies for its efficient usage \cite{lichtarge2019corpora}, \cite{kiyono2019an}, \cite{stahlberg2021synthetic}. The majority of GEC systems today use synthetic data to pre-train Transformer-based components of their models.&#10;&#10;In this work, we are focusing on exploring sequence tagging models and their ensembles. Although most of our developments may eventually be applied to other languages, we work with English only in this study. Being a resource-rich language, English is a highly competitive area for the GEC task$^3$.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Ensembling, Sequence Tagging Architecture, Computer Science, Linguistics, Knowledge Distillation Techniques, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2106.03830" label="2106.03830">
        <attvalues>
          <attvalue for="0" value="A Simple Recipe for Multilingual Grammatical Error Correction" />
          <attvalue for="1" value="  This paper presents a simple recipe to train state-of-the-art multilingual&#10;Grammatical Error Correction (GEC) models. We achieve this by first proposing a&#10;language-agnostic method to generate a large number of synthetic examples. The&#10;second ingredient is to use large-scale multilingual language models (up to 11B&#10;parameters). Once fine-tuned on language-specific supervised sets we surpass&#10;the previous state-of-the-art results on GEC benchmarks in four languages:&#10;English, Czech, German and Russian. Having established a new set of baselines&#10;for GEC, we make our results easily reproducible and accessible by releasing a&#10;cLang-8 dataset. It is produced by using our best model, which we call gT5, to&#10;clean the targets of a widely used yet noisy lang-8 dataset. cLang-8 greatly&#10;simplifies typical GEC training pipelines composed of multiple fine-tuning&#10;stages -- we demonstrate that performing a single fine-tuning step on cLang-8&#10;with the off-the-shelf language models yields further accuracy improvements&#10;over an already top-performing gT5 model for English.&#10;" />
          <attvalue for="2" value="&#10;&#10;Grammatical Error Correction (GEC) is the task of correcting grammatical and other related errors in text.&#10;It has been the subject of several modeling efforts in recent years due to its ability to improve grammaticality and readability of user generated texts.&#10;This is of particular importance to non-native speakers, children, and individuals with language impairments, who may be more prone to producing texts with grammatical errors.&#10;&#10;Modern approaches often view the GEC task as monolingual text-to-text rewriting \cite{naplava,katsumata2020stronger,grundkiewicz-etal-2019-neural} and employ encoder-decoder neural architectures~\cite{sutskever2014sequence,bahdanau2014neural}.&#10;These methods typically require large training sets to work well~\cite{malmi-etal-2019-encode} which are scarce especially for languages other than English.&#10;One of the largest and most widely used datasets for GEC is the \lang Learner Corpus, which covers 80 languages and has been created by language learners correcting each other's texts.&#10;However, the distribution of languages is very skewed, with Japanese and English being the most prevalent languages with over a million ungrammatical-grammatical sentence pairs each, while only ten languages have more than 10,000 sentence pairs each.&#10;Additionally, given the uncontrolled nature of the data collection, many of the examples contain unnecessary paraphrasing and erroneous or incomplete corrections. &#10;&#10;Limited amounts of suitable training data has led to multiple approaches that propose to generate synthetic training data for GEC~\cite{madnani2012exploring,grundkiewicz2014wiked,grundkiewicz-etal-2019-neural,lichtarge-etal-2019-corpora,awasthi-etal-2019-parallel}.&#10;Although using synthetic data as the first fine-tuning step has been shown to improve model accuracy, it introduces practical challenges that make the development and fair comparison of GEC models challenging: ($i$) the synthetic methods often require language-specific tuning (e.g. language-specific hyperparameters and spelling dictionaries~\cite{naplava}), and; ($ii$) due to the inability of synthetic data to capture the complete error distribution of the target eval sets, the final model is obtained by following a multi-stage fine-tuning process~\cite{lichtarge-etal-2019-corpora,lichtarge-etal-2020-data,omelianchuk-etal-2020-gector}. &#10;Because of this, carefully picking the learning rates and number of training steps for each of the fine-tuning stages is required, making it difficult to replicate and build on top of previous best reported models.&#10;&#10;The ideas of leveraging self-supervised pre-training and increasing the model size have yielded significant improvements on numerous seq2seq tasks in recent years~\cite{t5,mt5,BART,MASS,KERMIT,Bert2Bert}, but these approaches have been applied to GEC to only a limited extent.&#10;&#10;In this paper we adopt the mT5 \cite{mt5} as our base model which has already been pre-trained on a corpus covering 101 languages.&#10;To adapt the model to the GEC task, we design a fully unsupervised language-agnostic pre-training objective that mimics corrections typically contained in labeled data.&#10;We generate synthetic training data by automatically corrupting grammatical sentences, but in contrast to the previous state-of-the-art by \cite{naplava} for low-resources languages, we use our synthetic pre-training to train a single model on all 101 languages, employing no language-specific priors to remain fully language-agnostic. After pre-training we further fine-tune our model on supervised GEC data for available languages (with data conditions ranging from millions to tens of thousands).&#10;Additionally, we explore the effect of scaling up the model size from 60M to 11B parameters. We surpass the previous state-of-the-art results on four evaluated languages: English, Czech, German and Russian.&#10;&#10;Fine-tuning and running inference with our largest and most accurate models require multi-GPU/TPU infrastructure. To make the results of our research widely accessible we release a \clangg dataset obtained by using our largest \GT model to clean up the targets of the frequently used yet noisy \lang dataset. We show that off-the-shelf variants of T5~\cite{t5} when fine-tuned only on \clangg, outperform those models trained on the original \lang data with and w/o additional fine-tuning data, thus simplifying the complex multi-stage process of training GEC models. Thus \clangg not only allows others to easily train highly competitive GEC models, but it also greatly simplifies GEC training pipeline, basically reducing a multi-step fine-tuning processs to a single fine-tuning step.&#10;&#10;Our contributions in this paper are three-fold: (1)~We show that a simple language-agnostic pre-training objective can achieve state-of-the-art GEC results when models are scaled up in size; (2)~We show the effect model size has on GEC, and; (3)~We release a large multilingual GEC dataset based on Lang-8, which allows for state-of-the-art results without additional fine-tuning steps, thus significantly simplifying the training setup.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.02893" label="1910.02893">
        <attvalues>
          <attvalue for="0" value="Parallel Iterative Edit Models for Local Sequence Transduction" />
          <attvalue for="1" value="  We present a Parallel Iterative Edit (PIE) model for the problem of local&#10;sequence transduction arising in tasks like Grammatical error correction (GEC).&#10;Recent approaches are based on the popular encoder-decoder (ED) model for&#10;sequence to sequence learning. The ED model auto-regressively captures full&#10;dependency among output tokens but is slow due to sequential decoding. The PIE&#10;model does parallel decoding, giving up the advantage of modelling full&#10;dependency in the output, yet it achieves accuracy competitive with the ED&#10;model for four reasons: 1.~predicting edits instead of tokens, 2.~labeling&#10;sequences instead of generating sequences, 3.~iteratively refining predictions&#10;to capture dependencies, and 4.~factorizing logits over edits and their token&#10;argument to harness pre-trained language models like BERT. Experiments on tasks&#10;spanning GEC, OCR correction and spell correction demonstrate that the PIE&#10;model is an accurate and significantly faster alternative for local sequence&#10;transduction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.05780" label="1904.05780">
        <attvalues>
          <attvalue for="0" value="Corpora Generation for Grammatical Error Correction" />
          <attvalue for="1" value="  Grammatical Error Correction (GEC) has been recently modeled using the&#10;sequence-to-sequence framework. However, unlike sequence transduction problems&#10;such as machine translation, GEC suffers from the lack of plentiful parallel&#10;data. We describe two approaches for generating large parallel datasets for GEC&#10;using publicly available Wikipedia data. The first method extracts&#10;source-target pairs from Wikipedia edit histories with minimal filtration&#10;heuristics, while the second method introduces noise into Wikipedia sentences&#10;via round-trip translation through bridge languages. Both strategies yield&#10;similar sized parallel corpora containing around 4B tokens. We employ an&#10;iterative decoding strategy that is tailored to the loosely supervised nature&#10;of our constructed corpora. We demonstrate that neural GEC models trained using&#10;either type of corpora give similar performance. Fine-tuning these models on&#10;the Lang-8 corpus and ensembling allows us to surpass the state of the art on&#10;both the CoNLL-2014 benchmark and the JFLEG task. We provide systematic&#10;analysis that compares the two approaches to data generation and highlights the&#10;effectiveness of ensembling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.00502" label="1909.00502">
        <attvalues>
          <attvalue for="0" value="An Empirical Study of Incorporating Pseudo Data into Grammatical Error&#10;  Correction" />
          <attvalue for="1" value="  The incorporation of pseudo data in the training of grammatical error&#10;correction models has been one of the main factors in improving the performance&#10;of such models. However, consensus is lacking on experimental configurations,&#10;namely, choosing how the pseudo data should be generated or used. In this&#10;study, these choices are investigated through extensive experiments, and&#10;state-of-the-art performance is achieved on the CoNLL-2014 test set&#10;($F_{0.5}=65.0$) and the official test set of the BEA-2019 shared task&#10;($F_{0.5}=70.2$) without making any modifications to the model architecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.13318" label="2105.13318">
        <attvalues>
          <attvalue for="0" value="Synthetic Data Generation for Grammatical Error Correction with Tagged&#10;  Corruption Models" />
          <attvalue for="1" value="  Synthetic data generation is widely known to boost the accuracy of neural&#10;grammatical error correction (GEC) systems, but existing methods often lack&#10;diversity or are too simplistic to generate the broad range of grammatical&#10;errors made by human writers. In this work, we use error type tags from&#10;automatic annotation tools such as ERRANT to guide synthetic data generation.&#10;We compare several models that can produce an ungrammatical sentence given a&#10;clean sentence and an error type tag. We use these models to build a new, large&#10;synthetic pre-training data set with error tag frequency distributions matching&#10;a given development set. Our synthetic data set yields large and consistent&#10;gains, improving the state-of-the-art on the BEA-19 and CoNLL-14 test sets. We&#10;also show that our approach is particularly effective in adapting a GEC system,&#10;trained on mixed native and non-native English, to a native English test set,&#10;even surpassing real training data consisting of high-quality sentence pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.17626" label="2406.17626">
        <attvalues>
          <attvalue for="0" value="CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue&#10;  Coreference" />
          <attvalue for="1" value="  As large language models (LLMs) constantly evolve, ensuring their safety&#10;remains a critical research problem. Previous red-teaming approaches for LLM&#10;safety have primarily focused on single prompt attacks or goal hijacking. To&#10;the best of our knowledge, we are the first to study LLM safety in multi-turn&#10;dialogue coreference. We created a dataset of 1,400 questions across 14&#10;categories, each featuring multi-turn coreference safety attacks. We then&#10;conducted detailed evaluations on five widely used open-source LLMs. The&#10;results indicated that under multi-turn coreference safety attacks, the highest&#10;attack success rate was 56% with the LLaMA2-Chat-7b model, while the lowest was&#10;13.9% with the Mistral-7B-Instruct model. These findings highlight the safety&#10;vulnerabilities in LLMs during dialogue coreference interactions.&#10;" />
          <attvalue for="2" value="&#10;&#10;The rapid advancement of large language models (LLMs) has made them prevalent across various real-life applications in recent years. However, the safety vulnerabilities of LLMs have raised concerns, underscoring the critical need for safety assurances.&#10;For safety enhancement, Anthropic and DeepMind \cite{HH-RLHF} have introduced the ``red-teaming'' approach, which involves the role-playing of attackers in order to identify and fix the models' safety loopholes. Furthermore, existing studies have created various red-teaming datasets to assess the safety of LLMs in different scenarios, such as bias \cite{genderbias}, discrimination \cite{harmfulq}, and privacy breaches \cite{Confaide}.&#10;Nevertheless, current studies have not noticed the safety risks of coreference relationships in multi-trun dialogues, leading to underestimating potential vulnerabilities.&#10;&#10;Figure~\ref{figure: coreference example} illustrates an example of a dialogue coreference attack that triggers the model to generate insults for a schizophrenic. On the left, it shows that the model refuses to respond with a single prompt attack and highlights the harmful nature of such behavior. However, on the right, by engaging in a multi-turn dialogue and referring to schizophrenic as ``a person with this condition'' in the final question, the model is manipulated into generating harmful content.&#10;&#10;To address the safety issues in dialogue coreference scenarios mentioned above, we introduce CoSafe, a dataset comprising $1,400$ multi-turn attack questions across $14$ categories. Initially, we selected $100$ attack prompts from each category defined by BeaverTails~\cite{beavertails}. These prompts were designed to induce unsafe responses from LLMs. We then used GPT-4 to expand these single-turn prompts into multi-turn dialogues, strategically placing the attack question with coreference at the end of each dialogue. &#10;To the best of our knowledge, CoSafe is the first benchmark to study LLM safety in multi-turn dialogue coreference. &#10;We then test five popular open-source LLMs on CoSafe. Their responses to the final attack question were evaluated using QA moderation, human assessment, and LLM evaluation.&#10;&#10;Experimental results show that dialogue coreference indeed threatens LLMs' safety. Firstly, our dataset construction method generally had high attack success rates, with harmful response rates from LLMs in dialogue coreference scenarios surpassing those in single prompt scenarios. For example, the attack successful rate on the LLaMA2-Chat-7b model was up to $56\%$. In dialogue coreference scenarios, it generated harmful responses $39.4\%$ of the time, compared to only $14.5\%$ in single prompt scenarios. Secondly, there was significant agreement between QA moderation and human evaluation, indicating the reliability of our evaluation metrics. Finally, system prompts and Chain-of-Thought methods could reduce harmful responses, but unsafe responses still occur. These findings demonstrate that safety issues of dialogue coreference require further study and solutions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Safety, Coreference Attacks, Artificial Intelligence, Dialogue Vulnerabilities, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.17884" label="2310.17884">
        <attvalues>
          <attvalue for="0" value="Can LLMs Keep a Secret? Testing Privacy Implications of Language Models&#10;  via Contextual Integrity Theory" />
          <attvalue for="1" value="  The interactive use of large language models (LLMs) in AI assistants (at&#10;work, home, etc.) introduces a new set of inference-time privacy risks: LLMs&#10;are fed different types of information from multiple sources in their inputs&#10;and are expected to reason about what to share in their outputs, for what&#10;purpose and with whom, within a given context. In this work, we draw attention&#10;to the highly critical yet overlooked notion of contextual privacy by proposing&#10;ConfAIde, a benchmark designed to identify critical weaknesses in the privacy&#10;reasoning capabilities of instruction-tuned LLMs. Our experiments show that&#10;even the most capable models such as GPT-4 and ChatGPT reveal private&#10;information in contexts that humans would not, 39% and 57% of the time,&#10;respectively. This leakage persists even when we employ privacy-inducing&#10;prompts or chain-of-thought reasoning. Our work underscores the immediate need&#10;to explore novel inference-time privacy-preserving approaches, based on&#10;reasoning and theory of mind.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.04657" label="2307.04657">
        <attvalues>
          <attvalue for="0" value="BeaverTails: Towards Improved Safety Alignment of LLM via a&#10;  Human-Preference Dataset" />
          <attvalue for="1" value="  In this paper, we introduce the BeaverTails dataset, aimed at fostering&#10;research on safety alignment in large language models (LLMs). This dataset&#10;uniquely separates annotations of helpfulness and harmlessness for&#10;question-answering pairs, thus offering distinct perspectives on these crucial&#10;attributes. In total, we have gathered safety meta-labels for 333,963&#10;question-answer (QA) pairs and 361,903 pairs of expert comparison data for both&#10;the helpfulness and harmlessness metrics. We further showcase applications of&#10;BeaverTails in content moderation and reinforcement learning with human&#10;feedback (RLHF), emphasizing its potential for practical safety measures in&#10;LLMs. We believe this dataset provides vital resources for the community,&#10;contributing towards the safe development and deployment of LLMs. Our project&#10;page is available at the following URL:&#10;https://sites.google.com/view/pku-beavertails.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.12869" label="2407.12869">
        <attvalues>
          <attvalue for="0" value="Bilingual Adaptation of Monolingual Foundation Models" />
          <attvalue for="1" value="  We present an efficient method for adapting a monolingual Large Language&#10;Model (LLM) to another language, addressing challenges of catastrophic&#10;forgetting and tokenizer limitations. We focus this study on adapting Llama 2&#10;to Arabic. Our two-stage approach begins with expanding the vocabulary and&#10;training only the embeddings matrix, followed by full model continual&#10;pre-training on a bilingual corpus. By continually pre-training on a mix of&#10;Arabic and English corpora, the model retains its proficiency in English while&#10;acquiring capabilities in Arabic. Our approach results in significant&#10;improvements in Arabic and slight enhancements in English, demonstrating&#10;cost-effective cross-lingual transfer. We perform ablations on embedding&#10;initialization techniques, data mix ratios, and learning rates and release a&#10;detailed training recipe. To demonstrate generalizability of this approach we&#10;also adapted Llama 3 8B to Arabic and Llama 2 13B to Hindi.&#10;" />
          <attvalue for="2" value="&#10;&#10;There has been a rapid advancement in open source English-dominant foundation language models like Llama 2 \cite{touvron2023llama}, Mistral 7B \cite{jiang2023mistral}, and Llama 3, primarily trained on extensive English corpora with minimal inclusion of non-English languages. &#10;To create models proficient in low-resource languages, two approaches can be taken: training a bilingual or multilingual model from scratch or adapting an existing strong English-dominant model to the target language.&#10;While bilingual and monolingual models trained from scratch, like Jais \cite{sengupta2023jais} and Bloom \cite{workshop2023bloom}, have shown promise in non-English capabilities, they are expensive to train and have inferior capabilities in English.&#10;Adapting strong English-dominant models to new languages also pose challenges, such as catastrophic forgetting of English capabilities, inefficiencies of English-dominant tokenizers, and the need for hyperparameter adjustments. \cite{fujii2024swallow, luo2024empirical, FRENCH1999128, huang2024acegpt}. &#10;Here we address the challenges of the model adaptation approach.&#10;&#10;Knowledge, reasoning and truthfulness capabilities of Large Language Models (LLMs) are transferable across languages \cite{yang2024large, sengupta2023jais}. This gives us the basis and motivation to explore efficient methods for cross-lingual transfer from English to Arabic through continual pre-training of a monolingual English-dominant LLM without degradation of English capabilities.&#10;Several recent works demonstrate cross-lingual transfer of foundation models \cite{de-vries-nissim-2021-good, marchisio-etal-2023-mini, csaki2023efficiently, zhao2024Llama, huang2024acegpt, da-dalt-etal-2024-flor-effectiveness}, yet they lack comprehensive analysis of hyperparameter tuning, tokenizer, data mix selections, and the impact of different model sizes.&#10;&#10;We study the following aspects of cross-lingual adaptation. &#10;&#10;Vocabulary extension&#10;We find that adapting a pre-trained model to a new language requires expanding the vocabulary, along with employing the methods below to maintain the model's original capabilities while acquiring new linguistic skills. We determine the optimal extension ratio of the original vocabulary through experimentation.&#10;&#10;Embedding alignment&#10;We find that ensuring alignment between the embeddings of the original and newly added vocabulary tokens is vital. We explore three techniques for initializing newly added token embeddings. We follow with embedding-only pre-training, which further aligns the embedding scale and orientation for original and new tokens.&#10;&#10;Continual pre-training&#10;Following the embedding alignment, we continually pre-train the model. We conduct experiments at the 7B model scale to assess various English-Arabic mix ratios and learning rates. We leverage the insights obtained from these experiments to perform cross-lingual adaptation to Arabic with Llama 2 13B and Llama 2 70B models.&#10;&#10;Careful empirical study of vocabulary extension, embedding alignment, data mixtures and hyperparameter tuning, results in a recipe for language adaptation with significant performance improvements in Arabic and, uniquely, enhancements in English on Llama 2 models. To demonstrate the generalizability of this recipe to other languages and models, we adapt Llama 3 8B to Arabic and Llama 2 13B to Hindi.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Training, Cross-Lingual Transfer, Artificial Intelligence, Mathematics, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2210.01703" label="2210.01703">
        <attvalues>
          <attvalue for="0" value="Improving Label-Deficient Keyword Spotting Through Self-Supervised&#10;  Pretraining" />
          <attvalue for="1" value="  Keyword Spotting (KWS) models are becoming increasingly integrated into&#10;various systems, e.g. voice assistants. To achieve satisfactory performance,&#10;these models typically rely on a large amount of labelled data, limiting their&#10;applications only to situations where such data is available. Self-supervised&#10;Learning (SSL) methods can mitigate such a reliance by leveraging&#10;readily-available unlabelled data. Most SSL methods for speech have primarily&#10;been studied for large models, whereas this is not ideal, as compact KWS models&#10;are generally required. This paper explores the effectiveness of SSL on small&#10;models for KWS and establishes that SSL can enhance the performance of small&#10;KWS models when labelled data is scarce. We pretrain three compact&#10;transformer-based KWS models using Data2Vec, and fine-tune them on a&#10;label-deficient setup of the Google Speech Commands data set. It is found that&#10;Data2Vec pretraining leads to a significant increase in accuracy, with&#10;label-deficient scenarios showing an improvement of 8.22% 11.18% absolute&#10;accuracy.&#10;" />
          <attvalue for="2" value="&#10;Common for personal assistants like Google Assistant and Apple's Siri is that they make use of an \gls*{asr} system, which is activated by a smaller \gls*{kws} system in order to save resources when the \gls*{asr} system is not needed \cite{ivan_deep_spoken_keyword}. &#10;Modern deep learning based \gls*{kws} models have improved the accuracy of \gls*{kws} systems. However, they need to be trained on a large amount of labelled data to generalize well and obtaining properly labelled speech data is a labour-intensive and costly process, especially for low-resource languages. &#10;&#10;Recently, self-supervised learning methods have shown to be able to learn strong representations from unlabelled data, yielding good performance on a number of downstream tasks, including \gls*{kws}, when fine-tuned on a limited amount of labelled data.&#10;However, current studies mainly focus on developing universal speech models \cite{baevski_wav2vec2, Chen_WavLMLS}, which are trained on large speech corpuses such as Librispeech \cite{librispeech} or LibriLight \cite{librilight}, with the goal of obtaining a model that can perform well for multiple downstream tasks.&#10;These large models are commonly evaluated on benchmarks like SUPERB \cite{Yang_SUPERBSP}, requiring fine-tuning on multiple downstream tasks. &#10;Consequently, training these models require numerous high-end GPUs and often several weeks of training, making training these models infeasible in many cases, e.g., due to limited time or restricted computing resources.&#10;Additionally, for many use cases, such as \gls*{kws} for voice assistants, it is desirable that the models are small and efficient \cite{ivan_deep_spoken_keyword}.&#10;&#10;While knowledge distillation \cite{hinton_distilling} has been investigated for transferring the representations learned by a large model to a smaller model \cite{Gu_Liu, Fang_SEED, chen_kornblith}, such methods do not deal with the problem of the necessity of training a large model initially.&#10;One study used a contrastive type of SSL method to train smaller models without distillation from a large pretrained model and found that, contrary to former assumptions, small models are able to solve the self-supervised pretext tasks without overfitting \cite{efficacy_of_small_self_supervised_contrastive_models}. &#10;Additionally, they were able to improve the performance of five different small image recognition models, ranging from 2.5 to 11 million parameters, suggesting that training small self-supervised models is feasible.&#10;Other work found that the learned parameters of large speech models suffer from redundancy across layers, and proposed the use of weight sharing to reduce parameter redundancy and the network size \cite{Chi_AudioAA}. &#10;&#10;In this paper, we investigate the adaption of the general non-contrastive SSL framework Data2Vec \cite{data2vec} to improve \gls*{kws} performance in label-deficient scenarios. &#10;We implement three variations of the \gls*{kwt} model \cite{berg_KWT}, varying from 600k to 5.4M parameters, and pretrain the models using Data2Vec. &#10;The models are evaluated on a label-deficient setup of the Google Speech Commands data set \cite{speechcommandsv2} with only \SI{20}{\percent} labelled data for supervised training, and the results show the following: &#10;\begin{enumerate}&#10; performance for all three models when the amount of labelled data is limited, indicating that self-supervised learning can also be beneficial for small models.&#10; \item All three pretrained and fine-tuned models achieve similar performance to models trained on \SI{100}{\percent} labelled data, while using \SI{80}{\percent} data as unlabelled data for self-supervised pretraining, and only \SI{20}{\percent} labelled data for fine-tuning.&#10; \item A significant performance improvement from self-supervised pretraining is also seen when using a larger out-of-domain data set for pretraining.&#10; \item Fine-tuning the entire pretrained model is necessary.&#10;\end{enumerate}&#10;&#10;The source code used to produce the results of this paper is made publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Linguistics, Artificial Intelligence, Self-Supervised Learning, Speech Recognition, Natural Language Processing, Keyword Spotting Models" />
        </attvalues>
      </node>
      <node id="2111.10592" label="2111.10592">
        <attvalues>
          <attvalue for="0" value="Deep Spoken Keyword Spotting: An Overview" />
          <attvalue for="1" value="  Spoken keyword spotting (KWS) deals with the identification of keywords in&#10;audio streams and has become a fast-growing technology thanks to the paradigm&#10;shift introduced by deep learning a few years ago. This has allowed the rapid&#10;embedding of deep KWS in a myriad of small electronic devices with different&#10;purposes like the activation of voice assistants. Prospects suggest a sustained&#10;growth in terms of social use of this technology. Thus, it is not surprising&#10;that deep KWS has become a hot research topic among speech scientists, who&#10;constantly look for KWS performance improvement and computational complexity&#10;reduction. This context motivates this paper, in which we conduct a literature&#10;review into deep spoken KWS to assist practitioners and researchers who are&#10;interested in this technology. Specifically, this overview has a comprehensive&#10;nature by covering a thorough analysis of deep KWS systems (which includes&#10;speech features, acoustic modeling and posterior handling), robustness methods,&#10;applications, datasets, evaluation metrics, performance of deep KWS systems and&#10;audio-visual KWS. The analysis performed in this paper allows us to identify a&#10;number of directions for future research, including directions adopted from&#10;automatic speech recognition research and directions that are unique to the&#10;problem of spoken KWS.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.11304" label="2106.11304">
        <attvalues>
          <attvalue for="0" value="Simple Distillation Baselines for Improving Small Self-supervised Models" />
          <attvalue for="1" value="  While large self-supervised models have rivalled the performance of their&#10;supervised counterparts, small models still struggle. In this report, we&#10;explore simple baselines for improving small self-supervised models via&#10;distillation, called SimDis. Specifically, we present an offline-distillation&#10;baseline, which establishes a new state-of-the-art, and an online-distillation&#10;baseline, which achieves similar performance with minimal computational&#10;overhead. We hope these baselines will provide useful experience for relevant&#10;future research. Code is available at: https://github.com/JindongGu/SimDis/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.14762" label="2107.14762">
        <attvalues>
          <attvalue for="0" value="On the Efficacy of Small Self-Supervised Contrastive Models without&#10;  Distillation Signals" />
          <attvalue for="1" value="  It is a consensus that small models perform quite poorly under the paradigm&#10;of self-supervised contrastive learning. Existing methods usually adopt a large&#10;off-the-shelf model to transfer knowledge to the small one via distillation.&#10;Despite their effectiveness, distillation-based methods may not be suitable for&#10;some resource-restricted scenarios due to the huge computational expenses of&#10;deploying a large model. In this paper, we study the issue of training&#10;self-supervised small models without distillation signals. We first evaluate&#10;the representation spaces of the small models and make two non-negligible&#10;observations: (i) the small models can complete the pretext task without&#10;overfitting despite their limited capacity and (ii) they universally suffer the&#10;problem of over clustering. Then we verify multiple assumptions that are&#10;considered to alleviate the over-clustering phenomenon. Finally, we combine the&#10;validated techniques and improve the baseline performances of five small&#10;architectures with considerable margins, which indicates that training small&#10;self-supervised contrastive models is feasible even without distillation&#10;signals. The code is available at&#10;\textit{https://github.com/WOWNICE/ssl-small}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.03555" label="2202.03555">
        <attvalues>
          <attvalue for="0" value="data2vec: A General Framework for Self-supervised Learning in Speech,&#10;  Vision and Language" />
          <attvalue for="1" value="  While the general idea of self-supervised learning is identical across&#10;modalities, the actual algorithms and objectives differ widely because they&#10;were developed with a single modality in mind. To get us closer to general&#10;self-supervised learning, we present data2vec, a framework that uses the same&#10;learning method for either speech, NLP or computer vision. The core idea is to&#10;predict latent representations of the full input data based on a masked view of&#10;the input in a self-distillation setup using a standard Transformer&#10;architecture. Instead of predicting modality-specific targets such as words,&#10;visual tokens or units of human speech which are local in nature, data2vec&#10;predicts contextualized latent representations that contain information from&#10;the entire input. Experiments on the major benchmarks of speech recognition,&#10;image classification, and natural language understanding demonstrate a new&#10;state of the art or competitive performance to predominant approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.00769" label="2104.00769">
        <attvalues>
          <attvalue for="0" value="Keyword Transformer: A Self-Attention Model for Keyword Spotting" />
          <attvalue for="1" value="  The Transformer architecture has been successful across many domains,&#10;including natural language processing, computer vision and speech recognition.&#10;In keyword spotting, self-attention has primarily been used on top of&#10;convolutional or recurrent encoders. We investigate a range of ways to adapt&#10;the Transformer architecture to keyword spotting and introduce the Keyword&#10;Transformer (KWT), a fully self-attentional architecture that exceeds&#10;state-of-the-art performance across multiple tasks without any pre-training or&#10;additional data. Surprisingly, this simple architecture outperforms more&#10;complex models that mix convolutional, recurrent and attentive layers. KWT can&#10;be used as a drop-in replacement for these models, setting two new benchmark&#10;records on the Google Speech Commands dataset with 98.6% and 97.7% accuracy on&#10;the 12 and 35-command tasks respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.12624" label="2102.12624">
        <attvalues>
          <attvalue for="0" value="Meta-Learning for improving rare word recognition in end-to-end ASR" />
          <attvalue for="1" value="  We propose a new method of generating meaningful embeddings for speech,&#10;changes to four commonly used meta learning approaches to enable them to&#10;perform keyword spotting in continuous signals and an approach of combining&#10;their outcomes into an end-to-end automatic speech recognition system to&#10;improve rare word recognition. We verify the functionality of each of our three&#10;contributions in two experiments exploring their performance for different&#10;amounts of classes (N-way) and examples per class (k-shot) in a few-shot&#10;setting. We find that the speech embeddings work well and the changes to the&#10;meta learning approaches also clearly enable them to perform continuous signal&#10;spotting. Despite the interface between keyword spotting and speech recognition&#10;being very simple, we are able to consistently improve word error rate by up to&#10;5%.&#10;" />
          <attvalue for="2" value=" &#10; While end-to-end (E2E) \cite{graves2014towards} deep learning (DL) models brought great&#10;improvements to the field of automatic speech recognition (ASR)&#10;in recent years and reduced word error rates (WER) on benchmark&#10;datasets significantly \cite{li2020comparison}, they also come with a set of problems.&#10;Even though they are designed to work well with unknown words,&#10;since they usually concatenate subword-units and can thus essentially&#10;produce any textual transcription, they tend to perform even worse on&#10;rare words than classical DL ASR systems \cite{spell2019}. This is due to them&#10;having an internal latent language model, which is biased and cannot&#10;be tweaked. For example, an ASR trained on the LibriSpeech corpus&#10;is very likely to transcribe the name Hilde as Hilda or the name Josef&#10;as Joseph.&#10;&#10;Even though those words are rare, they tend to be of great importance&#10;for many tasks. A good example for this is automatic meeting&#10;transcription. People frequently address each other by name in a&#10;meeting. And meeting participants can have very unique names with&#10;unique pronunciations. So for the transcriptions, the prior knowledge&#10;which names to expect could be very helpful. Incorporating it&#10;however is a non-trivial task, because E2E approaches cannot be fine-tuned&#10;easily, since they lack a lexicon or a pronunciation dictionary.&#10;&#10;Proper nouns have been identified as a challenging problem in&#10;ASR for a while now \cite{proper2014}. Recently some approaches have arisen to&#10;tackle this challenge with E2E ASR using a specialised architecture&#10;and losses \cite{peyser2020improving} or using specific data and training procedures to better&#10;represent contextual information \cite{alon2019contextual}. Our approach is meant for rare&#10;words in general, however in this work we choose rare proper nouns&#10;as exemplary data and use few-shot learning to improve performance&#10;on them.&#10;&#10;In meta learning \cite{schmidhuber1987evolutionary} using the metric space approach (MSML)&#10;\cite{Goldberger2004NeighbourhoodCA}, an embedding function is trained to transform datapoints into a&#10;metric space, where comparisons purely based on positions are possible.&#10;Since structural knowledge about a task is within the embedding&#10;function, new information can be considered by making comparisons&#10;to reference samples of data we want to adapt to. Essentially clustering&#10;is performed in a latent space. MSML could provide solutions&#10;for our goal of improving recognition of rare words in highly specific&#10;contexts, since it can operate with as little as one reference datapoint&#10;of the data we want to adapt to, it doesn’t need any retraining or time&#10;consuming tweaking mechanisms for new reference samples, and the&#10;decisions made on the basis of the comparisons in the metric space&#10;are simple and interpretable and can thus be integrated into an E2E&#10;workflow with few complications.&#10;&#10;We propose a two part ASR system that integrates metric space&#10;representations of expected difficult keywords into the E2E pipeline.&#10;An ASR system is built using the Transformer architecture \cite{vaswani2017attention} which&#10;has been shown to outperform any prior architecture for most speech&#10;related tasks\cite{karita2019comparative, li2019neural, vila2018end, dong2018speech} by heavily relying on attention mechanisms&#10;\cite{bahdanau2015neural}. The intermediate representation that the Transformer-encoder&#10;produces is used as prior embedding and then embedded&#10;further into a metric space on a frame-shifted window basis inspired&#10;by previous work on keyword spotting \cite{chen2014small}. Prior approaches of embedding&#10;audio include using a skip-gram model for speech \cite{chung2018speech2vec} and&#10;using convolutional neural nets (CNNs) for general purpose audio&#10;detection \cite{audiomax}. By using the Transformer as embedding function, we&#10;hope to get more phonetically rich embeddings. The metric space&#10;embeddings are then compared to reference samples using various&#10;renown MSML approaches such as Siamese Networks \cite{bromley1994signature}, Relation&#10;Networks \cite{sung2018learning}, Prototypical Networks \cite{snell2017prototypical} and Matching Networks&#10;\cite{vinyals2016matching}. The results of these comparisons are taken into account when&#10;decoding the orthographic representation of the utterance.&#10;&#10;Our main contributions are as follows: we propose a new method&#10;of generating meaningful embeddings of spoken language, we redesign&#10;MSML approaches to spot keywords in a continuous signal&#10;and we showcase an approach to integrate the results into an E2E&#10;ASR system. We verify our proposed methods on two severely different&#10;datasets and observe the embeddings working well, the keyword&#10;spotting achieving high F$_1$ scores and the keyword spotting ASR&#10;interface improving rare-word WER by 1.6\% on average.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Embeddings, Computer Science, Linguistics, Engineering, Automatic Speech Recognition, Meta Learning Approaches, Signal Processing" />
        </attvalues>
      </node>
      <node id="1810.12170" label="1810.12170">
        <attvalues>
          <attvalue for="0" value="Contextual Speech Recognition with Difficult Negative Training Examples" />
          <attvalue for="1" value="  Improving the representation of contextual information is key to unlocking&#10;the potential of end-to-end (E2E) automatic speech recognition (ASR). In this&#10;work, we present a novel and simple approach for training an ASR context&#10;mechanism with difficult negative examples. The main idea is to focus on proper&#10;nouns (e.g., unique entities such as names of people and places) in the&#10;reference transcript, and use phonetically similar phrases as negative&#10;examples, encouraging the neural model to learn more discriminative&#10;representations. We apply our approach to an end-to-end contextual ASR model&#10;that jointly learns to transcribe and select the correct context items, and&#10;show that our proposed method gives up to $53.1\%$ relative improvement in word&#10;error rate (WER) across several benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.08976" label="1803.08976">
        <attvalues>
          <attvalue for="0" value="Speech2Vec: A Sequence-to-Sequence Framework for Learning Word&#10;  Embeddings from Speech" />
          <attvalue for="1" value="  In this paper, we propose a novel deep neural network architecture,&#10;Speech2Vec, for learning fixed-length vector representations of audio segments&#10;excised from a speech corpus, where the vectors contain semantic information&#10;pertaining to the underlying spoken words, and are close to other vectors in&#10;the embedding space if their corresponding underlying spoken words are&#10;semantically similar. The proposed model can be viewed as a speech version of&#10;Word2Vec. Its design is based on a RNN Encoder-Decoder framework, and borrows&#10;the methodology of skipgrams or continuous bag-of-words for training. Learning&#10;word embeddings directly from speech enables Speech2Vec to make use of the&#10;semantic information carried by speech that does not exist in plain text. The&#10;learned word embeddings are evaluated and analyzed on 13 widely used word&#10;similarity benchmarks, and outperform word embeddings learned by Word2Vec from&#10;the transcriptions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.06025" label="1711.06025">
        <attvalues>
          <attvalue for="0" value="Learning to Compare: Relation Network for Few-Shot Learning" />
          <attvalue for="1" value="  We present a conceptually simple, flexible, and general framework for&#10;few-shot learning, where a classifier must learn to recognise new classes given&#10;only few examples from each. Our method, called the Relation Network (RN), is&#10;trained end-to-end from scratch. During meta-learning, it learns to learn a&#10;deep distance metric to compare a small number of images within episodes, each&#10;of which is designed to simulate the few-shot setting. Once trained, a RN is&#10;able to classify images of new classes by computing relation scores between&#10;query images and the few examples of each new class without further updating&#10;the network. Besides providing improved performance on few-shot learning, our&#10;framework is easily extended to zero-shot learning. Extensive experiments on&#10;five benchmarks demonstrate that our simple approach provides a unified and&#10;effective approach for both of these two tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.11761" label="2208.11761">
        <attvalues>
          <attvalue for="0" value="IndicSUPERB: A Speech Processing Universal Performance Benchmark for&#10;  Indian languages" />
          <attvalue for="1" value="  A cornerstone in AI research has been the creation and adoption of&#10;standardized training and test datasets to earmark the progress of&#10;state-of-the-art models. A particularly successful example is the GLUE dataset&#10;for training and evaluating Natural Language Understanding (NLU) models for&#10;English. The large body of research around self-supervised BERT-based language&#10;models revolved around performance improvements on NLU tasks in GLUE. To&#10;evaluate language models in other languages, several language-specific GLUE&#10;datasets were created. The area of speech language understanding (SLU) has&#10;followed a similar trajectory. The success of large self-supervised models such&#10;as wav2vec2 enable creation of speech models with relatively easy to access&#10;unlabelled data. These models can then be evaluated on SLU tasks, such as the&#10;SUPERB benchmark. In this work, we extend this to Indic languages by releasing&#10;the IndicSUPERB benchmark. Specifically, we make the following three&#10;contributions. (i) We collect Kathbath containing 1,684 hours of labelled&#10;speech data across 12 Indian languages from 1,218 contributors located in 203&#10;districts in India. (ii) Using Kathbath, we create benchmarks across 6 speech&#10;tasks: Automatic Speech Recognition, Speaker Verification, Speaker&#10;Identification (mono/multi), Language Identification, Query By Example, and&#10;Keyword Spotting for 12 languages. (iii) On the released benchmarks, we train&#10;and evaluate different self-supervised models alongside a commonly used&#10;baseline FBANK. We show that language-specific fine-tuned models are more&#10;accurate than baseline on most of the tasks, including a large gap of 76\% for&#10;the Language Identification task. However, for speaker identification,&#10;self-supervised models trained on large datasets demonstrate an advantage. We&#10;hope IndicSUPERB contributes to the progress of developing speech language&#10;understanding models for Indian languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.10752" label="2203.10752">
        <attvalues>
          <attvalue for="0" value="XTREME-S: Evaluating Cross-lingual Speech Representations" />
          <attvalue for="1" value="  We introduce XTREME-S, a new benchmark to evaluate universal cross-lingual&#10;speech representations in many languages. XTREME-S covers four task families:&#10;speech recognition, classification, speech-to-text translation and retrieval.&#10;Covering 102 languages from 10+ language families, 3 different domains and 4&#10;task families, XTREME-S aims to simplify multilingual speech representation&#10;evaluation, as well as catalyze research in &quot;universal&quot; speech representation&#10;learning. This paper describes the new benchmark and establishes the first&#10;speech-only and speech-text baselines using XLS-R and mSLAM on all downstream&#10;tasks. We motivate the design choices and detail how to use the benchmark.&#10;Datasets and fine-tuning scripts are made easily accessible at&#10;https://hf.co/datasets/google/xtreme_s.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.13713" label="2406.13713">
        <attvalues>
          <attvalue for="0" value="Benchmarking Open-Source Language Models for Efficient Question&#10;  Answering in Industrial Applications" />
          <attvalue for="1" value="  In the rapidly evolving landscape of Natural Language Processing (NLP), Large&#10;Language Models (LLMs) have demonstrated remarkable capabilities in tasks such&#10;as question answering (QA). However, the accessibility and practicality of&#10;utilizing these models for industrial applications pose significant challenges,&#10;particularly concerning cost-effectiveness, inference speed, and resource&#10;efficiency. This paper presents a comprehensive benchmarking study comparing&#10;open-source LLMs with their non-open-source counterparts on the task of&#10;question answering. Our objective is to identify open-source alternatives&#10;capable of delivering comparable performance to proprietary models while being&#10;lightweight in terms of resource requirements and suitable for Central&#10;Processing Unit (CPU)-based inference. Through rigorous evaluation across&#10;various metrics including accuracy, inference speed, and resource consumption,&#10;we aim to provide insights into selecting efficient LLMs for real-world&#10;applications. Our findings shed light on viable open-source alternatives that&#10;offer acceptable performance and efficiency, addressing the pressing need for&#10;accessible and efficient NLP solutions in industry settings.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Processing (NLP) has been revolutionized by the emergence of Large Language Models (LLMs), enabling advancements in tasks such as machine translation~\cite{zhao2023transformer,bo2024_attentionMT, huang2024moceadaptivemixturecontextualization}, text summarization~\cite{li2023improving,doan2023too}, and knowledge extraction~\cite{gu2023distilling,xu2024take}. These models leverage large and sophisticated architectures to generate coherent and contextually relevant outputs, significantly expanding the scope of natural language understanding and generation. Prominent examples include GPT-4~\cite{chatGPT4}, Claude 3 Opus~\cite{claude}, and Mixtral 8x7B~\cite{jiang2024_mixtral}. While LLMs have reshaped the language technology landscape, their practical applications in industrial contexts remain an area of active exploration. This study focuses on Machine Reading Comprehension (MRC), an NLP task where LLMs demonstrate the ability to interpret and answer questions based on textual content. MRC systems are particularly valuable for industries that depend on document analysis, automated customer support, and knowledge management systems, making them a key component of modern AI-driven solutions.&#10;&#10;While LLMs have great potential, their adoption in real-world industrial scenarios involves complex trade-offs. Proprietary LLMs often deliver state-of-the-art performance on benchmark datasets, but their substantial computational demands, high costs, and limited transparency pose challenges for widespread adoption. On the other hand, open-source LLMs offer benefits such as lower costs and customizable deployments, yet their performance may fall short in resource-intensive applications. These trade-offs are especially pronounced in MRC tasks, where industries require solutions that balance accuracy, inference speed, and resource usage under operational constraints.&#10;&#10;In this context, the current study addresses the gap in systematic evaluations of LLMs for industrial MRC applications. By comparing open-source and proprietary LLMs on metrics relevant to real-world deployments, this work aims to guide stakeholders in selecting models that align with their specific needs. The contributions of this study are threefold: (1) it benchmarks open-source and proprietary LLMs across performance metrics such as accuracy, efficiency, and scalability; (2) it identifies practical trade-offs for deploying these models in resource-constrained environments; and (3) it highlights the potential of high-performing open-source models as viable alternatives to proprietary systems. This analysis aims to provide actionable insights for industries seeking to integrate MRC capabilities into their workflows while navigating operational challenges.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Information Technology, Linguistics, Industrial Applications, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2408.04216" label="2408.04216">
        <attvalues>
          <attvalue for="0" value="Attention Mechanism and Context Modeling System for Text Mining Machine&#10;  Translation" />
          <attvalue for="1" value="  This paper advances a novel architectural schema anchored upon the&#10;Transformer paradigm and innovatively amalgamates the K-means categorization&#10;algorithm to augment the contextual apprehension capabilities of the schema.&#10;The transformer model performs well in machine translation tasks due to its&#10;parallel computing power and multi-head attention mechanism. However, it may&#10;encounter contextual ambiguity or ignore local features when dealing with&#10;highly complex language structures. To circumvent this constraint, this&#10;exposition incorporates the K-Means algorithm, which is used to stratify the&#10;lexis and idioms of the input textual matter, thereby facilitating superior&#10;identification and preservation of the local structure and contextual&#10;intelligence of the language. The advantage of this combination is that K-Means&#10;can automatically discover the topic or concept regions in the text, which may&#10;be directly related to translation quality. Consequently, the schema contrived&#10;herein enlists K-Means as a preparatory phase antecedent to the Transformer and&#10;recalibrates the multi-head attention weights to assist in the discrimination&#10;of lexis and idioms bearing analogous semantics or functionalities. This&#10;ensures the schema accords heightened regard to the contextual intelligence&#10;embodied by these clusters during the training phase, rather than merely&#10;focusing on locational intelligence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.06439" label="2307.06439">
        <attvalues>
          <attvalue for="0" value="Distilling Large Language Models for Biomedical Knowledge Extraction: A&#10;  Case Study on Adverse Drug Events" />
          <attvalue for="1" value="  Large language models (LLMs), such as GPT-4, have demonstrated remarkable&#10;capabilities across a wide range of tasks, including health applications. In&#10;this paper, we study how LLMs can be used to scale biomedical knowledge&#10;curation. We find that while LLMs already possess decent competency in&#10;structuring biomedical text, by distillation into a task-specific student model&#10;through self-supervised learning, substantial gains can be attained over&#10;out-of-box LLMs, with additional advantages such as cost, efficiency, and&#10;white-box model access.&#10;  We conduct a case study on adverse drug event (ADE) extraction, which is an&#10;important area for improving care. On standard ADE extraction evaluation, a&#10;GPT-3.5 distilled PubMedBERT model attained comparable accuracy as supervised&#10;state-of-the-art models without using any labeled data. Despite being over&#10;1,000 times smaller, the distilled model outperformed its teacher GPT-3.5 by&#10;over 6 absolute points in F1 and GPT-4 by over 5 absolute points.&#10;  Ablation studies on distillation model choice (e.g., PubMedBERT vs BioGPT)&#10;and ADE extraction architecture shed light on best practice for biomedical&#10;knowledge extraction. Similar gains were attained by distillation for other&#10;standard biomedical knowledge extraction tasks such as gene-disease&#10;associations and protected health information, further illustrating the promise&#10;of this approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.09874" label="2204.09874">
        <attvalues>
          <attvalue for="0" value="Is Neural Topic Modelling Better than Clustering? An Empirical Study on&#10;  Clustering with Contextual Embeddings for Topics" />
          <attvalue for="1" value="  Recent work incorporates pre-trained word embeddings such as BERT embeddings&#10;into Neural Topic Models (NTMs), generating highly coherent topics. However,&#10;with high-quality contextualized document representations, do we really need&#10;sophisticated neural models to obtain coherent and interpretable topics? In&#10;this paper, we conduct thorough experiments showing that directly clustering&#10;high-quality sentence embeddings with an appropriate word selecting method can&#10;generate more coherent and diverse topics than NTMs, achieving also higher&#10;efficiency and simplicity.&#10;" />
          <attvalue for="2" value="&#10;&#10;Topic modelling is an unsupervised method to uncover latent semantic themes among documents \cite{boyd2017applications}. Neural topic models (NTMs) \cite{miao2016neural, srivastava2017autoencoding} incorporating neural components have significantly advanced the modelling results than the traditional Latent Dirichlet Allocation (LDA; \cite{blei2003latent}).&#10;Later, contextualized word and sentence embeddings produced by pre-trained language models such as BERT \cite{devlin-etal-2019-bert} have demonstrated the state-of-the-art results in multiple Natural Language Processing (NLP) tasks \cite{xia-etal-2020-bert}, which attracts attentions from the topic modelling community. &#10;Recent work has successfully incorporated these contextualized embeddings into NTMs, showing improved topic coherence than conventional NTMs that use Bag-of-Words (BoW) as document representations \cite{ bianchi-etal-2021-pre, bianchi-etal-2021-cross, jin-etal-2021-neural}.&#10;Despite the promising performance, existing NTMs are generally based on a variational autoencoder framework (VAE; \cite{kingma2013auto}), which suffers from hyper-parameters tuning and computational overheads \cite{zhao2021topic}. Moreover, the integration of the pre-trained embeddings to the standard VAE framework adds additional model complexity. With high-quality contextualized document representations, do we really need sophisticated NTMs to obtain coherent and interpretable topics?&#10;&#10;Recent work \cite{aharoni-goldberg-2020-unsupervised, sia-etal-2020-tired, thompson2020topic, grootendorst2020bertopic} has shown that directly congregating contextualized embeddings can get semantically similar word or document clusters. Specifically, \cite{sia-etal-2020-tired} cluster vocabulary-level word embeddings and obtain top words from each cluster using weighing and re-ranking, while \cite{thompson2020topic} consider polysemy and perform token-level clustering. However, the use of term frequency (TF) to select topic words fails to capture the semantics of clusters precisely because words with high frequency may be common across different clusters. \cite{grootendorst2020bertopic} propose a class-based Term Frequency-&#10;Inverse Document Frequenc (c-TF-IDF) method that extract important words from each clustered documents, which tends to choose representative words within each cluster to form topics. However, it overlooks the global semantics between clusters which could be incorporated. In addition, all above works only compare the performance with the traditional LDA while ignoring the promising NTMs proposed recently. The performance of the clustering-based topic models is still yet uncovered.&#10;&#10;Is neural topic modelling better than simple embedding clustering?&#10;This work compares the performance of NTMs and contextualized embedding-based clustering systematically. Our main focus is to provide insights by comparing the two paradigms for topic models, which has not been investigated before. We employ a straightforward framework for clustering. In addition, we explore different strategies to select topic words for clusters. We evaluate our approach on three datasets with various text lengths.&#10;&#10;Our contributions are as follows:&#10;First, we find that directly clustering high-quality sentence embeddings can generate as good topics as NTMs, providing a simple and efficient solution to uncover latent topics among documents. Second, we propose a new topic word selecting method, which is the key to producing highly coherent and diverse topics. Third, we show that the clustering-based model is robust to the length of documents and the number of topics. Reducing the embedding dimensionality negligibly affects the performance but saves runtime. From our best knowledge, we are the first to compare with NTMs, using contextualized embeddings that produced by various transformer-based models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Neural Topic Models, Machine Learning, Computer Science, Linguistics, Document Clustering Methods, Mathematics, Natural Language Processing, Word Embeddings Analysis" />
        </attvalues>
      </node>
      <node id="1511.06038" label="1511.06038">
        <attvalues>
          <attvalue for="0" value="Neural Variational Inference for Text Processing" />
          <attvalue for="1" value="  Recent advances in neural variational inference have spawned a renaissance in&#10;deep latent variable models. In this paper we introduce a generic variational&#10;inference framework for generative and conditional models of text. While&#10;traditional variational methods derive an analytic approximation for the&#10;intractable distributions over latent variables, here we construct an inference&#10;network conditioned on the discrete text input to provide the variational&#10;distribution. We validate this framework on two very different text modelling&#10;applications, generative document modelling and supervised question answering.&#10;Our neural variational document model combines a continuous stochastic document&#10;representation with a bag-of-words generative model and achieves the lowest&#10;reported perplexities on two standard test corpora. The neural answer selection&#10;model employs a stochastic representation layer within an attention mechanism&#10;to extract the semantics between a question and answer pair. On two question&#10;answering benchmarks this model exceeds all previous published benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14914" label="2004.14914">
        <attvalues>
          <attvalue for="0" value="Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for&#10;  Fast and Good Topics too!" />
          <attvalue for="1" value="  Topic models are a useful analysis tool to uncover the underlying themes&#10;within document collections. The dominant approach is to use probabilistic&#10;topic models that posit a generative story, but in this paper we propose an&#10;alternative way to obtain topics: clustering pre-trained word embeddings while&#10;incorporating document information for weighted clustering and reranking top&#10;words. We provide benchmarks for the combination of different word embeddings&#10;and clustering algorithms, and analyse their performance under dimensionality&#10;reduction with PCA. The best performing combination for our approach performs&#10;as well as classical topic models, but with lower runtime and computational&#10;complexity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12626" label="2010.12626">
        <attvalues>
          <attvalue for="0" value="Topic Modeling with Contextualized Word Representation Clusters" />
          <attvalue for="1" value="  Clustering token-level contextualized word representations produces output&#10;that shares many similarities with topic models for English text collections.&#10;Unlike clusterings of vocabulary-level word embeddings, the resulting models&#10;more naturally capture polysemy and can be used as a way of organizing&#10;documents. We evaluate token clusterings trained from several different output&#10;layers of popular contextualized language models. We find that BERT and GPT-2&#10;produce high quality clusterings, but RoBERTa does not. These cluster models&#10;are simple, reliable, and can perform as well as, if not better than, LDA topic&#10;models, maintaining high topic quality even when the number of topics is large&#10;relative to the size of the local collection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.07739" label="2102.07739">
        <attvalues>
          <attvalue for="0" value="Personalization Strategies for End-to-End Speech Recognition Systems" />
          <attvalue for="1" value="  The recognition of personalized content, such as contact names, remains a&#10;challenging problem for end-to-end speech recognition systems. In this work, we&#10;demonstrate how first and second-pass rescoring strategies can be leveraged&#10;together to improve the recognition of such words. Following previous work, we&#10;use a shallow fusion approach to bias towards recognition of personalized&#10;content in the first-pass decoding. We show that such an approach can improve&#10;personalized content recognition by up to 16% with minimum degradation on the&#10;general use case. We describe a fast and scalable algorithm that enables our&#10;biasing models to remain at the word-level, while applying the biasing at the&#10;subword level. This has the advantage of not requiring the biasing models to be&#10;dependent on any subword symbol table. We also describe a novel second-pass&#10;de-biasing approach: used in conjunction with a first-pass shallow fusion that&#10;optimizes on oracle WER, we can achieve an additional 14% improvement on&#10;personalized content recognition, and even improve accuracy for the general use&#10;case by up to 2.5%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.07754" label="2011.07754">
        <attvalues>
          <attvalue for="0" value="Deep Shallow Fusion for RNN-T Personalization" />
          <attvalue for="1" value="  End-to-end models in general, and Recurrent Neural Network Transducer (RNN-T)&#10;in particular, have gained significant traction in the automatic speech&#10;recognition community in the last few years due to their simplicity,&#10;compactness, and excellent performance on generic transcription tasks. However,&#10;these models are more challenging to personalize compared to traditional hybrid&#10;systems due to the lack of external language models and difficulties in&#10;recognizing rare long-tail words, specifically entity names. In this work, we&#10;present novel techniques to improve RNN-T's ability to model rare WordPieces,&#10;infuse extra information into the encoder, enable the use of alternative&#10;graphemic pronunciations, and perform deep fusion with personalized language&#10;models for more robust biasing. We show that these combined techniques result&#10;in 15.4%-34.5% relative Word Error Rate improvement compared to a strong RNN-T&#10;baseline which uses shallow fusion and text-to-speech augmentation. Our work&#10;helps push the boundary of RNN-T personalization and close the gap with hybrid&#10;systems on use cases where biasing and entity recognition are crucial.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.12806" label="2201.12806">
        <attvalues>
          <attvalue for="0" value="Improving End-to-End Contextual Speech Recognition with Fine-Grained&#10;  Contextual Knowledge Selection" />
          <attvalue for="1" value="  Nowadays, most methods in end-to-end contextual speech recognition bias the&#10;recognition process towards contextual knowledge. Since all-neural contextual&#10;biasing methods rely on phrase-level contextual modeling and attention-based&#10;relevance modeling, they may encounter confusion between similar&#10;context-specific phrases, which hurts predictions at the token level. In this&#10;work, we focus on mitigating confusion problems with fine-grained contextual&#10;knowledge selection (FineCoS). In FineCoS, we introduce fine-grained knowledge&#10;to reduce the uncertainty of token predictions. Specifically, we first apply&#10;phrase selection to narrow the range of phrase candidates, and then conduct&#10;token attention on the tokens in the selected phrase candidates. Moreover, we&#10;re-normalize the attention weights of most relevant phrases in inference to&#10;obtain more focused phrase-level contextual representations, and inject&#10;position information to better discriminate phrases or tokens. On LibriSpeech&#10;and an in-house 160,000-hour dataset, we explore the proposed methods based on&#10;a controllable all-neural biasing method, collaborative decoding (ColDec). The&#10;proposed methods provide at most 6.1% relative word error rate reduction on&#10;LibriSpeech and 16.4% relative character error rate reduction on the in-house&#10;dataset over ColDec.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.02220" label="2110.02220">
        <attvalues>
          <attvalue for="0" value="Fast Contextual Adaptation with Neural Associative Memory for On-Device&#10;  Personalized Speech Recognition" />
          <attvalue for="1" value="  Fast contextual adaptation has shown to be effective in improving Automatic&#10;Speech Recognition (ASR) of rare words and when combined with an on-device&#10;personalized training, it can yield an even better recognition result. However,&#10;the traditional re-scoring approaches based on an external language model is&#10;prone to diverge during the personalized training. In this work, we introduce a&#10;model-based end-to-end contextual adaptation approach that is decoder-agnostic&#10;and amenable to on-device personalization. Our on-device simulation experiments&#10;demonstrate that the proposed approach outperforms the traditional re-scoring&#10;technique by 12% relative WER and 15.7% entity mention specific F1-score in a&#10;continues personalization scenario.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.05544" label="2104.05544">
        <attvalues>
          <attvalue for="0" value="Investigating Methods to Improve Language Model Integration for&#10;  Attention-based Encoder-Decoder ASR Models" />
          <attvalue for="1" value="  Attention-based encoder-decoder (AED) models learn an implicit internal&#10;language model (ILM) from the training transcriptions. The integration with an&#10;external LM trained on much more unpaired text usually leads to better&#10;performance. A Bayesian interpretation as in the hybrid autoregressive&#10;transducer (HAT) suggests dividing by the prior of the discriminative acoustic&#10;model, which corresponds to this implicit LM, similarly as in the hybrid hidden&#10;Markov model approach. The implicit LM cannot be calculated efficiently in&#10;general and it is yet unclear what are the best methods to estimate it. In this&#10;work, we compare different approaches from the literature and propose several&#10;novel methods to estimate the ILM directly from the AED model. Our proposed&#10;methods outperform all previous approaches. We also investigate other methods&#10;to suppress the ILM mainly by decreasing the capacity of the AED model,&#10;limiting the label context, and also by training the AED model together with a&#10;pre-existing LM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.11134" label="2202.11134">
        <attvalues>
          <attvalue for="0" value="ProtoSound: A Personalized and Scalable Sound Recognition System for&#10;  Deaf and Hard-of-Hearing Users" />
          <attvalue for="1" value="  Recent advances have enabled automatic sound recognition systems for deaf and&#10;hard of hearing (DHH) users on mobile devices. However, these tools use&#10;pre-trained, generic sound recognition models, which do not meet the diverse&#10;needs of DHH users. We introduce ProtoSound, an interactive system for&#10;customizing sound recognition models by recording a few examples, thereby&#10;enabling personalized and fine-grained categories. ProtoSound is motivated by&#10;prior work examining sound awareness needs of DHH people and by a survey we&#10;conducted with 472 DHH participants. To evaluate ProtoSound, we characterized&#10;performance on two real-world sound datasets, showing significant improvement&#10;over state-of-the-art (e.g., +9.7% accuracy on the first dataset). We then&#10;deployed ProtoSound's end-user training and real-time recognition through a&#10;mobile application and recruited 19 hearing participants who listened to the&#10;real-world sounds and rated the accuracy across 56 locations (e.g., homes,&#10;restaurants, parks). Results show that ProtoSound personalized the model&#10;on-device in real-time and accurately learned sounds across diverse acoustic&#10;contexts. We close by discussing open challenges in personalizable sound&#10;recognition, including the need for better recording interfaces and algorithmic&#10;improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.16954" label="2203.16954">
        <attvalues>
          <attvalue for="0" value="An End-to-end Chinese Text Normalization Model based on Rule-guided&#10;  Flat-Lattice Transformer" />
          <attvalue for="1" value="  Text normalization, defined as a procedure transforming non standard words to&#10;spoken-form words, is crucial to the intelligibility of synthesized speech in&#10;text-to-speech system. Rule-based methods without considering context can not&#10;eliminate ambiguation, whereas sequence-to-sequence neural network based&#10;methods suffer from the unexpected and uninterpretable errors problem. Recently&#10;proposed hybrid system treats rule-based model and neural model as two cascaded&#10;sub-modules, where limited interaction capability makes neural network model&#10;cannot fully utilize expert knowledge contained in the rules. Inspired by&#10;Flat-LAttice Transformer (FLAT), we propose an end-to-end Chinese text&#10;normalization model, which accepts Chinese characters as direct input and&#10;integrates expert knowledge contained in rules into the neural network, both&#10;contribute to the superior performance of proposed model for the text&#10;normalization task. We also release a first publicly accessible largescale&#10;dataset for Chinese text normalization. Our proposed model has achieved&#10;excellent results on this dataset.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text normalization (TN) is crucial to the intelligibility of synthesized speech in text-to-speech (TTS) system.&#10;It is defined as a procedure that transforms non-standard words (NSWs),&#10;e.g.&#10;written-form&#10;numbers, symbols or characters, to spoken-form words (SFWs), such as &#10;transforming ``3.4&quot; to ``three point four&quot; and ``2021/10&quot; to ``October Twenty Twenty-one&quot;.&#10;To deal with the ambiguity problem in transforming NSWs to SFWs,&#10;context information and NSW's inherent special construct should be considered.&#10;For example, &#10;context can decide whether to read ``2021'' as year or number, &#10;whereas special construct of ``172.0.0.1'' can be determined as IP address.&#10;Furthermore, to form a context, word information is crucial.&#10;Take ``2021 \begin{CJK*}{UTF8}{gbsn}光年\end{CJK*}'' as example, the ``2021'' will be read as number only if the word ``\begin{CJK*}{UTF8}{gbsn}光年\end{CJK*} (light-years)'' is correctly identified;&#10;otherwise, ``2021'' might be read as year if the keyword ``\begin{CJK*}{UTF8}{gbsn}年\end{CJK*} (year)'' is &#10;inaccurately matched.&#10;&#10;Based on the taxonomy approach for NSW \cite{richard2001}, the TN tasks can be resolved by rule-based approaches which utilize handcrafted regular expressions and/or keywords \cite{2009A, 2009Document, 2016Text} to determine the category of NSWs and then convert to corresponding SFWs with predefined conversion functions.&#10;However, the &#10;selection&#10;of keywords as well as the construction of regular expression rules are time-consuming and labor-intensive.&#10;Several machine learning methods have been proposed for the disambiguation task of NSWs, including finite state automata (FSA) \cite{2009A}, maximum entropy (ME) \cite{Jia2008TN}, conditional random fields (CRF) \cite{Liou2016}, etc.&#10;&#10;With the development of deep learning technologies, using neural network to model contextual information has achieved impressive progress for TN task.&#10;Sequence-to-sequence (seq2seq) models typically encode the written-form text representation into a state vector, and decode it into a sequence of spoken-form text output directly \cite{lusetti2018encoder}.&#10;Long short-term memory (LSTM) and attention-based recurrent neural network (RNN) sequence-to-sequence models are well applied in English and Russian text normalization \cite{2016RNN, mansfield2019neural}.&#10;Bi-directional LSTM or gated recurrent unit (GRU) are further utilized in both encoder and decoder \cite{zhang2019neural, sproat2016rnn, mansfield2019neural}.&#10;However, directly applying sequence-to-sequence models to TN task may cause unexpected and uninterpretable errors caused by the model or data bias.&#10;&#10;Recently, a hybrid TN system for Mandarin has been proposed, which combines a rule-based model based on pattern match and a multi-head self-attention based non-seq2seq neural network model,&#10;to &#10;address&#10;the corresponding shortcomings mentioned above \cite{zhang2020hybrid}.&#10;According to the priority, NSWs are sent to the rule-based and neural models respectively. &#10;If the result of the neural model is of mismatched format, the NSW will be processed by the rule-based model again.&#10;However, the hybrid system simply treats rule-based model and neural model as cascaded sub-modules serially, which may cause error accumulation.&#10;It is easy to tell that rule-based model and neural network model can &#10;supplement&#10;each other.&#10;But limited interaction capability of these two cascaded sub-modules makes neural network model cannot fully utilize the expert knowledge included in the rules.&#10;&#10;Inspired by the superior performance and the flexibility of the latest Flat-LAttice Transformer (FLAT) \cite{li2020flat},&#10;we propose a FLAT based end-to-end Chinese TN model,&#10;named FlatTN,&#10;which can directly incorporate the expert knowledge in predefined rules into the network,&#10;providing a novel way of leveraging the complementary advantages of the two models.&#10;&#10;The advantages of using FlatTN for the text normalization task falls into two aspects.&#10;First, there is no need of the prerequisite word segmentation module.&#10;FLAT can obtain all potential words in the sentence that match the specific lexicon, organize all characters and matched words to a lattice structure and flatten the lattice structure into spans \cite{li2020flat}, then send them into Transformer encoder.&#10;The method of combining lexicon is fully independent of word segmentation, and more effective in using word information thanks to the freedom of choosing lexicon words in a context \cite{zhang2018chinese}.&#10;Second, the NSW matching rules can be &#10;easily&#10;incorporated into the model and the definition of rules is greatly simplified.&#10;In the proposed model, rules are only adopted for the purpose of pattern match to derive all possible candidate NSWs in the input sentence.&#10;There is no need for the rules to account for complex context matching for disambiguation task as in the conventional method.&#10;We also release a large-scale dataset for the Chinese text normalization task, which will be open-sourced for public access soon.&#10;Experimental results on this dataset demonstrate that our proposed model has achieved an excellent performance.&#10;&#10;The contributions of our work &#10;are:&#10;\begin{enumerate}&#10;\item[1).] For the first time, we propose the use of flat-lattice transformer (FLAT) for the task of Chinese TN problem, enhancing the controllability and scalability of TN task.&#10;\item[2).] We come up with &#10;a novel rule-guided FLAT model, that can directly incorporate the expert knowledge on the predefined rules into the neural network based model. The proposed model is also an end-to-end model which predicts the NSW categories directly from the raw Chinese characters with NSWs.&#10;\item[3).] We release, as open-sourced resources, a Chinese text normalization dataset&#10;with standard NSW taxonomies to eliminate the ambiguity in pronunciation of NSWs. It is a first publicly accessible dataset for the Chinese TN task.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Linguistics, Neural Network Models, Text Normalization, Speech Synthesis, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2004.11795" label="2004.11795">
        <attvalues>
          <attvalue for="0" value="FLAT: Chinese NER Using Flat-Lattice Transformer" />
          <attvalue for="1" value="  Recently, the character-word lattice structure has been proved to be&#10;effective for Chinese named entity recognition (NER) by incorporating the word&#10;information. However, since the lattice structure is complex and dynamic, most&#10;existing lattice-based models are hard to fully utilize the parallel&#10;computation of GPUs and usually have a low inference-speed. In this paper, we&#10;propose FLAT: Flat-LAttice Transformer for Chinese NER, which converts the&#10;lattice structure into a flat structure consisting of spans. Each span&#10;corresponds to a character or latent word and its position in the original&#10;lattice. With the power of Transformer and well-designed position encoding,&#10;FLAT can fully leverage the lattice information and has an excellent&#10;parallelization ability. Experiments on four datasets show FLAT outperforms&#10;other lexicon-based models in performance and efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.14227" label="2212.14227">
        <attvalues>
          <attvalue for="0" value="StyleTTS-VC: One-Shot Voice Conversion by Knowledge Transfer from&#10;  Style-Based TTS Models" />
          <attvalue for="1" value="  One-shot voice conversion (VC) aims to convert speech from any source speaker&#10;to an arbitrary target speaker with only a few seconds of reference speech from&#10;the target speaker. This relies heavily on disentangling the speaker's identity&#10;and speech content, a task that still remains challenging. Here, we propose a&#10;novel approach to learning disentangled speech representation by transfer&#10;learning from style-based text-to-speech (TTS) models. With cycle consistent&#10;and adversarial training, the style-based TTS models can perform&#10;transcription-guided one-shot VC with high fidelity and similarity. By learning&#10;an additional mel-spectrogram encoder through a teacher-student knowledge&#10;transfer and novel data augmentation scheme, our approach results in&#10;disentangled speech representation without needing the input text. The&#10;subjective evaluation shows that our approach can significantly outperform the&#10;previous state-of-the-art one-shot voice conversion models in both naturalness&#10;and similarity.&#10;" />
          <attvalue for="2" value="&#10;Voice conversion (VC) is a technique that converts one speaker’s voice into another’s voice while preserving linguistic and prosodic information such as phonemes and prosody. Recent advances in deep learning have enriched research on one particular type of voice conversion: one-shot voice conversion. This type of voice conversion, also known as any-to-any voice conversion, aims to convert speech from any source speaker to an arbitrary target speaker using only a few seconds of reference audio from the target speaker. To convert an unseen speaker’s voice into another speaker’s voice unseen during training, the model needs to learn a shared representation of speech across all potential sources and target speakers \cite{qian2019autovc}. Therefore, learning disentangled representations of speech and speaker identity is crucial for successful one-shot voice conversion. &#10;&#10;Several techniques have been proposed for learning disentangled representations, including instance normalization \cite{chou2019one, wu2020one, chen2021again}, vector quantization \cite{wu2020one, van2020vector, wang2021vqmivc, tang2022avqvc}, transfer learning from ASR or TTS models \cite{li2021ppg, lin2021s2vc, zhang2021transfer, casanova2022yourtts, gabrys2022voice}, and adversarial training \cite{wang2020one, tang2021tgavc}. These methods, albeit effective, do not guarantee that the empirically trained representations contain no source speaker information. VC systems such as Mellotron \cite{valle2020mellotron} and Cotatron \cite{park2020cotatron}, on the other hand, use phoneme alignment and pitch curve from the source speech and re-synthesize the speech of the target speaker. Since phoneme alignment and normalized pitch curve are largely speaker-agnostic, the re-synthesized speech should only reflect the speech content and prosody of the source audio without leaking any other source-specific information. These TTS-based methods that theoretically guarantee a disentangled representation still suffer from two essential problems. The major drawback of TTS-based models is that this method requires input text or a sequence of phonemes to generate the alignment which limits its potential for applications in real-time inference. Zhang et. al. \cite{zhang2021transfer} has made an attempt to address this problem by training an additional mel-spectrogram encoder that produces the same latent representation as the one generated from phoneme alignment and text representation. This is equivalent to training an automatic speech recognition (ASR) system, but as we show here, this way of encoder training is not optimal. Another obstacle endured by the TTS method is the generalization problem. Since the original TTS models are trained to only reconstruct speech from the pitch and phoneme alignment of the source speaker, there is no guarantee that the synthesized speech will sound natural and similar to the target speakers when the input pitch and phoneme alignment are from a different speaker.&#10;&#10;In this paper, we present StyleTTS-VC, a non-parallel one-shot voice conversion framework based on StyleTTS \cite{li2022styletts}, a style-based text-to-speech model. We address the aforementioned generalization problems by first training a StyleTTS speech decoder with a cycle consistency loss function and adversarial objectives. We then train a mel-spectrogram encoder to produce representations that reconstruct the decoder output generated using representations from phoneme alignment for all speakers in the training set with both synthesized and real speech as input. Unlike the previous method \cite{zhang2021transfer}, our proposed technique does not force the encoded representations to be close to the phoneme alignment representations. The subjective human evaluation shows that our model outperforms the previous state-of-the-art one-shot voice conversion model, YourTTS \cite{casanova2022yourtts}, and two other baseline models, AGAIN-VC \cite{chen2021again} and VQMIVC \cite{wang2021vqmivc}, for unseen source and target speakers. Moreover, since our model consists of only convolutional layers without non-causal RNN or transformers, our model has the capability to perform real-time inference with a faster-than-real-time vocoder. &#10;&#10;Our work makes multiple contributions: (i) we show that the cycle consistency and adversarial objective are effective in training both TTS decoder and mel-spectrogram encoder for VC applications, (ii) we introduce novel data augmentation using text-guided voice conversion results as both input and target during training, and (iii) we demonstrate that the loss function proposed in \cite{zhang2021transfer} is suboptimal for transfer learning from TTS models for voice conversion applications and propose an alternative solution with a mutual information (MI) maximization objective. The audio samples from our model are available at. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Voice Conversion, Machine Learning, Computer Science, Speech Processing, Linguistics, Deep Learning, Signal Processing, Speech Representation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1904.05742" label="1904.05742">
        <attvalues>
          <attvalue for="0" value="One-shot Voice Conversion by Separating Speaker and Content&#10;  Representations with Instance Normalization" />
          <attvalue for="1" value="  Recently, voice conversion (VC) without parallel data has been successfully&#10;adapted to multi-target scenario in which a single model is trained to convert&#10;the input voice to many different speakers. However, such model suffers from&#10;the limitation that it can only convert the voice to the speakers in the&#10;training data, which narrows down the applicable scenario of VC. In this paper,&#10;we proposed a novel one-shot VC approach which is able to perform VC by only an&#10;example utterance from source and target speaker respectively, and the source&#10;and target speaker do not even need to be seen during training. This is&#10;achieved by disentangling speaker and content representations with instance&#10;normalization (IN). Objective and subjective evaluation shows that our model is&#10;able to generate the voice similar to target speaker. In addition to the&#10;performance measurement, we also demonstrate that this model is able to learn&#10;meaningful speaker representations without any supervision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.11997" label="1910.11997">
        <attvalues>
          <attvalue for="0" value="Mellotron: Multispeaker expressive voice synthesis by conditioning on&#10;  rhythm, pitch and global style tokens" />
          <attvalue for="1" value="  Mellotron is a multispeaker voice synthesis model based on Tacotron 2 GST&#10;that can make a voice emote and sing without emotive or singing training data.&#10;By explicitly conditioning on rhythm and continuous pitch contours from an&#10;audio signal or music score, Mellotron is able to generate speech in a variety&#10;of styles ranging from read speech to expressive speech, from slow drawls to&#10;rap and from monotonous voice to singing voice. Unlike other methods, we train&#10;Mellotron using only read speech data without alignments between text and&#10;audio. We evaluate our models using the LJSpeech and LibriTTS datasets. We&#10;provide F0 Frame Errors and synthesized samples that include style transfer&#10;from other speakers, singers and styles not seen during training, procedural&#10;manipulation of rhythm and pitch and choir synthesis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.03295" label="2005.03295">
        <attvalues>
          <attvalue for="0" value="Cotatron: Transcription-Guided Speech Encoder for Any-to-Many Voice&#10;  Conversion without Parallel Data" />
          <attvalue for="1" value="  We propose Cotatron, a transcription-guided speech encoder for&#10;speaker-independent linguistic representation. Cotatron is based on the&#10;multispeaker TTS architecture and can be trained with conventional TTS&#10;datasets. We train a voice conversion system to reconstruct speech with&#10;Cotatron features, which is similar to the previous methods based on Phonetic&#10;Posteriorgram (PPG). By training and evaluating our system with 108 speakers&#10;from the VCTK dataset, we outperform the previous method in terms of both&#10;naturalness and speaker similarity. Our system can also convert speech from&#10;speakers that are unseen during training, and utilize ASR to automate the&#10;transcription with minimal reduction of the performance. Audio samples are&#10;available at https://mindslab-ai.github.io/cotatron, and the code with a&#10;pre-trained model will be made available soon.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.14399" label="2009.14399">
        <attvalues>
          <attvalue for="0" value="Transfer Learning from Speech Synthesis to Voice Conversion with&#10;  Non-Parallel Training Data" />
          <attvalue for="1" value="  This paper presents a novel framework to build a voice conversion (VC) system&#10;by learning from a text-to-speech (TTS) synthesis system, that is called TTS-VC&#10;transfer learning. We first develop a multi-speaker speech synthesis system&#10;with sequence-to-sequence encoder-decoder architecture, where the encoder&#10;extracts robust linguistic representations of text, and the decoder,&#10;conditioned on target speaker embedding, takes the context vectors and the&#10;attention recurrent network cell output to generate target acoustic features.&#10;We take advantage of the fact that TTS system maps input text to speaker&#10;independent context vectors, and reuse such a mapping to supervise the training&#10;of latent representations of an encoder-decoder voice conversion system. In the&#10;voice conversion system, the encoder takes speech instead of text as input,&#10;while the decoder is functionally similar to TTS decoder. As we condition the&#10;decoder on speaker embedding, the system can be trained on non-parallel data&#10;for any-to-any voice conversion. During voice conversion training, we present&#10;both text and speech to speech synthesis and voice conversion networks&#10;respectively. At run-time, the voice conversion network uses its own&#10;encoder-decoder architecture. Experiments show that the proposed approach&#10;outperforms two competitive voice conversion baselines consistently, namely&#10;phonetic posteriorgram and variational autoencoder methods, in terms of speech&#10;quality, naturalness, and speaker similarity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00316" label="2011.00316">
        <attvalues>
          <attvalue for="0" value="AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and&#10;  Adaptive Instance Normalization" />
          <attvalue for="1" value="  Recently, voice conversion (VC) has been widely studied. Many VC systems use&#10;disentangle-based learning techniques to separate the speaker and the&#10;linguistic content information from a speech signal. Subsequently, they convert&#10;the voice by changing the speaker information to that of the target speaker. To&#10;prevent the speaker information from leaking into the content embeddings,&#10;previous works either reduce the dimension or quantize the content embedding as&#10;a strong information bottleneck. These mechanisms somehow hurt the synthesis&#10;quality. In this work, we propose AGAIN-VC, an innovative VC system using&#10;Activation Guidance and Adaptive Instance Normalization. AGAIN-VC is an&#10;auto-encoder-based model, comprising of a single encoder and a decoder. With a&#10;proper activation as an information bottleneck on content embeddings, the&#10;trade-off between the synthesis quality and the speaker similarity of the&#10;converted speech is improved drastically. This one-shot VC system obtains the&#10;best performance regardless of the subjective or objective evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.12359" label="1811.12359">
        <attvalues>
          <attvalue for="0" value="Challenging Common Assumptions in the Unsupervised Learning of&#10;  Disentangled Representations" />
          <attvalue for="1" value="  The key idea behind the unsupervised learning of disentangled representations&#10;is that real-world data is generated by a few explanatory factors of variation&#10;which can be recovered by unsupervised learning algorithms. In this paper, we&#10;provide a sober look at recent progress in the field and challenge some common&#10;assumptions. We first theoretically show that the unsupervised learning of&#10;disentangled representations is fundamentally impossible without inductive&#10;biases on both the models and the data. Then, we train more than 12000 models&#10;covering most prominent methods and evaluation metrics in a reproducible&#10;large-scale experimental study on seven different data sets. We observe that&#10;while the different methods successfully enforce properties ``encouraged'' by&#10;the corresponding losses, well-disentangled models seemingly cannot be&#10;identified without supervision. Furthermore, increased disentanglement does not&#10;seem to lead to a decreased sample complexity of learning for downstream tasks.&#10;Our results suggest that future work on disentanglement learning should be&#10;explicit about the role of inductive biases and (implicit) supervision,&#10;investigate concrete benefits of enforcing disentanglement of the learned&#10;representations, and consider a reproducible experimental setup covering&#10;several data sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.05983" label="1802.05983">
        <attvalues>
          <attvalue for="0" value="Disentangling by Factorising" />
          <attvalue for="1" value="  We define and address the problem of unsupervised learning of disentangled&#10;representations on data generated from independent factors of variation. We&#10;propose FactorVAE, a method that disentangles by encouraging the distribution&#10;of representations to be factorial and hence independent across the dimensions.&#10;We show that it improves upon $\beta$-VAE by providing a better trade-off&#10;between disentanglement and reconstruction quality. Moreover, we highlight the&#10;problems of a commonly used disentanglement metric and introduce a new metric&#10;that does not suffer from them.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.02812" label="1804.02812">
        <attvalues>
          <attvalue for="0" value="Multi-target Voice Conversion without Parallel Data by Adversarially&#10;  Learning Disentangled Audio Representations" />
          <attvalue for="1" value="  Recently, cycle-consistent adversarial network (Cycle-GAN) has been&#10;successfully applied to voice conversion to a different speaker without&#10;parallel data, although in those approaches an individual model is needed for&#10;each target speaker. In this paper, we propose an adversarial learning&#10;framework for voice conversion, with which a single model can be trained to&#10;convert the voice to many different speakers, all without parallel data, by&#10;separating the speaker characteristics from the linguistic content in speech&#10;signals. An autoencoder is first trained to extract speaker-independent latent&#10;representations and speaker embedding separately using another auxiliary&#10;speaker classifier to regularize the latent representation. The decoder then&#10;takes the speaker-independent latent representation and the target speaker&#10;embedding as the input to generate the voice of the target speaker with the&#10;linguistic content of the source utterance. The quality of decoder output is&#10;further improved by patching with the residual signal produced by another pair&#10;of generator and discriminator. A target speaker set size of 20 was tested in&#10;the preliminary experiments, and very good voice quality was obtained.&#10;Conventional voice conversion metrics are reported. We also show that the&#10;speaker information has been properly reduced from the latent representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.07111" label="2210.07111">
        <attvalues>
          <attvalue for="0" value="A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained&#10;  Models" />
          <attvalue for="1" value="  Recent work on tokenizer-free multilingual pretrained models show promising&#10;results in improving cross-lingual transfer and reducing engineering overhead&#10;(Clark et al., 2022; Xue et al., 2022). However, these works mainly focus on&#10;reporting accuracy on a limited set of tasks and data settings, placing less&#10;emphasis on other important factors when tuning and deploying the models in&#10;practice, such as memory usage, inference speed, and fine-tuning data&#10;robustness. We attempt to fill this gap by performing a comprehensive empirical&#10;comparison of multilingual tokenizer-free and subword-based models considering&#10;these various dimensions. Surprisingly, we find that subword-based models might&#10;still be the most practical choice in many settings, achieving better&#10;performance for lower inference latency and memory usage. Based on these&#10;results, we encourage future work in tokenizer-free methods to consider these&#10;factors when designing and evaluating new models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Several recent results \cite{Clark2022CaninePA,Xue2022ByT5TA} have excited the research community with the possibility of ``tokenizer-free'' models, character-level and byte-level models, as an alternative to more traditional subword-based models.&#10;We, the authors of this paper, were also initially excited by these results -- the possibility of eschewing the two-step processing pipeline of subword segmentation and subword-based models would reduce the corresponding difficulties in cross-lingual transfer \cite{pmlr-v119-hu20b,maronikolakis-etal-2021-wine-v,rust-etal-2021-good,Wang2021-wy} or domain adaptation \cite{sato-etal-2020-vocabulary,liu-etal-2021-bridging} due to inconsistent subword units.&#10;However, upon several attempts to apply tokenizer-free methods, our excitement was tempered upon realization of a number of practical difficulties in applying these methods.&#10;This paper is a chronicle of some of the concerns we uncovered; we highlight some challenges with applying these models and propose best practices for future results reporting in this area.&#10;&#10;Specifically, we perform experiments fine-tuning pretrained multilingual models, evaluating them with respect to (1) robustness to fine-tuning data settings, (2) data efficiency, and (3) inference time and memory consumption.&#10;Based on these multiple dimensions, we come to the somewhat surprising conclusion that subword-based models might still be the most practical choice in most settings, as they are comparably robust to various fine-tuning data settings with a relatively low inference cost. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cross-Lingual Transfer, Multilingual Models, Artificial Intelligence, Model Efficiency" />
        </attvalues>
      </node>
      <node id="2206.03025" label="2206.03025">
        <attvalues>
          <attvalue for="0" value="OCHADAI at SemEval-2022 Task 2: Adversarial Training for Multilingual&#10;  Idiomaticity Detection" />
          <attvalue for="1" value="  We propose a multilingual adversarial training model for determining whether&#10;a sentence contains an idiomatic expression. Given that a key challenge with&#10;this task is the limited size of annotated data, our model relies on&#10;pre-trained contextual representations from different multi-lingual&#10;state-of-the-art transformer-based language models (i.e., multilingual BERT and&#10;XLM-RoBERTa), and on adversarial training, a training method for further&#10;enhancing model generalization and robustness. Without relying on any&#10;human-crafted features, knowledge bases, or additional datasets other than the&#10;target datasets, our model achieved competitive results and ranked 6th place in&#10;SubTask A (zero-shot) setting and 15th place in SubTask A (one-shot) setting.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large-scale pre-trained language models such as BERT \cite{devlin2019bert} have achieved great success in a wide range of natural language processing (NLP) tasks. However, more recent studies show that even such contextual models have a limited ability to capture idiomaticity \cite{garcia2021probing}. Idiomatic expressions denote a group of words that behave as single&#10;words to some extent. Their linguistic behavior cannot be inferred from the characteristics of their components, and still pose a challenge to natural language processing (NLP) systems.&#10;&#10;This paper describes the system developed by the OCHADAI team for SemEval-2022 Task 2 - Multilingual Idiomaticity Detection and Sentence Embedding \cite{tayyarmadabushi-etal-2022-semeval}. Given that a key challenge in this task is the limited size of annotated data, we follow best practices from recent work on enhancing model generalization and robustness and propose a model ensemble that leverages multilingual pre-trained representations and adversarial training. Our model ranked 6th on SubTask A (zero-shot), and 15th on SubTask A (one-shot).&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Adversarial Training, Linguistics, Computational Linguistics, Multilingual Models, Idiomatic Expression Detection, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2401.10465" label="2401.10465">
        <attvalues>
          <attvalue for="0" value="Data-driven grapheme-to-phoneme representations for a lexicon-free&#10;  text-to-speech" />
          <attvalue for="1" value="  Grapheme-to-Phoneme (G2P) is an essential first step in any modern,&#10;high-quality Text-to-Speech (TTS) system. Most of the current G2P systems rely&#10;on carefully hand-crafted lexicons developed by experts. This poses a two-fold&#10;problem. Firstly, the lexicons are generated using a fixed phoneme set,&#10;usually, ARPABET or IPA, which might not be the most optimal way to represent&#10;phonemes for all languages. Secondly, the man-hours required to produce such an&#10;expert lexicon are very high. In this paper, we eliminate both of these issues&#10;by using recent advances in self-supervised learning to obtain data-driven&#10;phoneme representations instead of fixed representations. We compare our&#10;lexicon-free approach against strong baselines that utilize a well-crafted&#10;lexicon. Furthermore, we show that our data-driven lexicon-free method performs&#10;as good or even marginally better than the conventional rule-based or&#10;lexicon-based neural G2Ps in terms of Mean Opinion Score (MOS) while using no&#10;prior language lexicon or phoneme set, i.e. no linguistic expertise.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text-to-Speech synthesis has been the subject of extensive research for several decades \cite{ 8461368, DBLP:conf/ssw/OordDZSVGKSK16}. &#10;Initially, concatenative speech synthesis models were developed to address this task by assembling waveforms from a pre-existing database of speech. Subsequently, statistical approaches were introduced to generate speech features from the text, which were then fed to a vocoder to produce the final output. The results of these methods are unsatisfactory due to the unnaturalness and mispronunciations in the generated speech.&#10;&#10;Grapheme-to-Phoneme (G2P) models are an integral part of current Text-to-Speech (TTS) engines~\cite{neuralG2P, neural_speech_synthesis}.&#10;The initial work in this field was done using rule-based and joint sequence models. However, with the rise of deep learning methods, RNN, and even more recently, Transformer based architectures have been used to perform a variety of tasks such as Grapheme-to-Phoneme, Automatic Speech Recognition\cite{garg2019improved, Garg2020StreamingOE, Garg2020HierarchicalMW}, Machine Translation, and Text-to-Speech Synthesis \cite{neuralG2P, neural_speech_synthesis, 8462506, JiyeonASRU2021}.&#10;As G2P conversion is essentially a sequence-to-sequence modeling task, using Encoder-Decoder architectures\cite{Kim2021Steamingseven, Gowda2019MultiTaskMC} helped obtain improvements on Grapheme-to-Phoneme conversion \cite{neuralG2P}.&#10;&#10;Traditional G2Ps \cite{cmudict} typically use a large lexicon to perform dictionary searches of the most frequent words and use hand-crafted rules to generate pronunciations for out-of-vocabulary words. Neural G2Ps~\cite{neuralG2P}, in contrast, use lexicons as their data for training their neural network and use the obtained network for predicting pronunciations. While both of these methods have been instrumental in building modern-day TTS systems, they have severe limitations in terms of requiring an external lexicon. Building a lexicon is an expensive and highly cumbersome task, as it requires multiple language experts to propose and then verify its validity.&#10;&#10;Recently there have been efforts to build a massively multi-lingual ByT5 G2P~\cite{Zhu2022ByT5MF}. &#10; It uses a T5 transformer based encoder-decoder architecture~\cite{Rezckov2021T5G2PUT} and uses an UTF8 based input tokenization to handle scripts in multiple languages.&#10; It uses publicly available lexicons from the internet covering upto 100 languages with varying size and quality of the lexicons.&#10; The difficulty in procuring high quality lexicon for all languages and the decreasing accuracy of g2p models for languages with limited or noisy data can be clearly seen from this paper.&#10;&#10;In this paper, we propose a new mechanism to train the Grapheme-to-Phoneme model without the need of any lexicon. We use Text-to-Speech as a use-case to show the effectiveness of our G2P model. We first use unlabeled speech data to pre-train a HuBERT \cite{9585401} for three iterations. Once we have a pre-trained HuBERT model, we input the labeled speech data and use a specific transformer layer to extract the speech features and apply k-means clustering on them to obtain frame-level phoneme targets. We use paired phoneme targets and labeled speech data to train our G2P transformer model. We then use the trained G2P to train a Tacotron 2 \cite{8461368} model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Text-to-Speech Systems, Artificial Intelligence, Self-Supervised Learning, Phoneme Representation" />
        </attvalues>
      </node>
      <node id="2204.03067" label="2204.03067">
        <attvalues>
          <attvalue for="0" value="ByT5 model for massively multilingual grapheme-to-phoneme conversion" />
          <attvalue for="1" value="  In this study, we tackle massively multilingual grapheme-to-phoneme&#10;conversion through implementing G2P models based on ByT5. We have curated a G2P&#10;dataset from various sources that covers around 100 languages and trained&#10;large-scale multilingual G2P models based on ByT5. We found that ByT5 operating&#10;on byte-level inputs significantly outperformed the token-based mT5 model in&#10;terms of multilingual G2P. Pairwise comparison with monolingual models in these&#10;languages suggests that multilingual ByT5 models generally lower the phone&#10;error rate by jointly learning from a variety of languages. The pretrained&#10;model can further benefit low resource G2P through zero-shot prediction on&#10;unseen languages or provides pretrained weights for finetuning, which helps the&#10;model converge to a lower phone error rate than randomly initialized weights.&#10;To facilitate future research on multilingual G2P, we make available our code&#10;and pretrained multilingual G2P models at:&#10;https://github.com/lingjzhu/CharsiuG2P.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.02821" label="2108.02821">
        <attvalues>
          <attvalue for="0" value="Applying the Information Bottleneck Principle to Prosodic Representation&#10;  Learning" />
          <attvalue for="1" value="  This paper describes a novel design of a neural network-based speech&#10;generation model for learning prosodic representation.The problem of&#10;representation learning is formulated according to the information bottleneck&#10;(IB) principle. A modified VQ-VAE quantized layer is incorporated in the speech&#10;generation model to control the IB capacity and adjust the balance between&#10;reconstruction power and disentangle capability of the learned representation.&#10;The proposed model is able to learn word-level prosodic representations from&#10;speech data. With an optimized IB capacity, the learned representations not&#10;only are adequate to reconstruct the original speech but also can be used to&#10;transfer the prosody onto different textual content. Extensive results of the&#10;objective and subjective evaluation are presented to demonstrate the effect of&#10;IB capacity control, the effectiveness, and potential usage of the learned&#10;prosodic representation in controllable neural speech generation.&#10;" />
          <attvalue for="2" value="&#10;Prosody is an essential component embedded in human speech. Carrying a full array of linguistic and paralinguistic functions, prosody manifests tone, stress, intonation, and rhythm of speech and contributes to naturalness, style, attitude, and emotion of speech \cite{taylor2009text,wagner2010experimental}. Conventionally speech prosody has been studied through the analysis of pitch, intensity, and duration features, focusing on one or more specific functions of prosody. This approach does not provide a holistic representation of prosody for speech generation purposes. In recent years, neural network-based speech synthesis systems \cite{wang2017tacotron,shen2018natural,li2019neural} show superior performance in terms of speech quality and naturalness and offer an effective approach to learning speech representations. The problem of prosodic representation learning has attracted particular attention in view of its potential use in prosody transfer and style control \cite{skerry2018towards,klimkov2019fine,lee2019robust}, prosody control \cite{zhang2020learning,hodariusing}.&#10;&#10;Various approaches to unsupervised learning of prosodic representation have been proposed \cite{skerry2018towards,klimkov2019fine,lee2019robust,zhang2020learning,sun2020fully,tan2020fine,wang2019vector,kenter2019chive}. In these studies, a neural network model is used to disentangle contributing factors of input speech and subsequently perform speech reconstruction from the disentangled factors. The prosodic representation is obtained as one of the learned factors, parallel with non-prosodic factors that correspond to content, speaker, channel, etc. In&#10;\cite{tan2020fine, zhang2021estimating, hsu2019disentangling,karlapati2020copycat}, adversarial learning was applied to address the problem that the learned prosodic representation might contain substantial information related to non-prosodic factors. The use of an adversarial classifier requires the availability of the labels for one of the disentangled non-prosodic factors. The design of the adversarial classifier is specific to only one non-prosodic factor and can not be applied to other non-prosodic factors. Furthermore, the non-prosodic factors(e.g., speaker) might be related to prosody \cite{zhang2021estimating}, while disentangling with an adversarial classifier might also result in low prosody information in the prosodic representation. &#10;&#10;In the present study, prosodic representations learning is tackled from the perspective of information bottleneck (IB), by which a good representation is determined with the trade-off between its predictive/reconstructive power and compact representation \cite{tishby2015deep}. We propose to define a good prosodic representation in three different aspects. First, it should have a good capability of capturing prosody-related information. It can reconstruct the reference speech conditioned on the other necessary factors (good reconstructive power). Second, the representation is expected to include as little as possible information about non-prosodic factors (compact representation). Third, the predicted prosodic representation should contribute to generating natural speech. The prosodic representations extracted from reference speech could be used to train a prosody predictor, which predicts the prosodic representation from the text. Specific speech generation applications (e.g., TTS) require appropriate and expressive prosody predicted from the text \cite{hodariusing}. The predicted prosodic representation is expected to help improve the naturalness or expressiveness of the generated speech.&#10;&#10;The contributions of this study are as follows: (1) the prosodic representation learning problem is formulated based on the IB principle; (2) a prosodic representation learning system with controllable IB capacity is developed; (3) subjective and objective evaluation results show that learned prosodic representations have good reconstructive power and can concisely capture prosody-related information by choosing appropriate IB capacity; (4) a machine-translation-based prosody predictor is proposed to realize text-to-prosody prediction. The predictor can generate prosodic representations for improving the naturalness of synthesized speech.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Design, Computer Science, Linguistics, Cognitive Science, Speech Generation Models, Signal Processing, Representation Learning" />
        </attvalues>
      </node>
      <node id="1811.02122" label="1811.02122">
        <attvalues>
          <attvalue for="0" value="Robust and fine-grained prosody control of end-to-end speech synthesis" />
          <attvalue for="1" value="  We propose prosody embeddings for emotional and expressive speech synthesis&#10;networks. The proposed methods introduce temporal structures in the embedding&#10;networks, thus enabling fine-grained control of the speaking style of the&#10;synthesized speech. The temporal structures can be designed either on the&#10;speech side or the text side, leading to different control resolutions in time.&#10;The prosody embedding networks are plugged into end-to-end speech synthesis&#10;networks and trained without any other supervision except for the target speech&#10;for synthesizing. It is demonstrated that the prosody embedding networks&#10;learned to extract prosodic features. By adjusting the learned prosody&#10;features, we could change the pitch and amplitude of the synthesized speech&#10;both at the frame level and the phoneme level. We also introduce the temporal&#10;normalization of prosody embeddings, which shows better robustness against&#10;speaker perturbations during prosody transfer tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.04233" label="1906.04233">
        <attvalues>
          <attvalue for="0" value="Using generative modelling to produce varied intonation for speech&#10;  synthesis" />
          <attvalue for="1" value="  Unlike human speakers, typical text-to-speech (TTS) systems are unable to&#10;produce multiple distinct renditions of a given sentence. This has previously&#10;been addressed by adding explicit external control. In contrast, generative&#10;models are able to capture a distribution over multiple renditions and thus&#10;produce varied renditions using sampling. Typical neural TTS models learn the&#10;average of the data because they minimise mean squared error. In the context of&#10;prosody, taking the average produces flatter, more boring speech: an &quot;average&#10;prosody&quot;. A generative model that can synthesise multiple prosodies will, by&#10;design, not model average prosody. We use variational autoencoders (VAEs) which&#10;explicitly place the most &quot;average&quot; data close to the mean of the Gaussian&#10;prior. We propose that by moving towards the tails of the prior distribution,&#10;the model will transition towards generating more idiosyncratic, varied&#10;renditions. Focusing here on intonation, we investigate the trade-off between&#10;naturalness and intonation variation and find that typical acoustic models can&#10;either be natural, or varied, but not both. However, sampling from the tails of&#10;the VAE prior produces much more varied intonation than the traditional&#10;approaches, whilst maintaining the same level of naturalness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.03943" label="2011.03943">
        <attvalues>
          <attvalue for="0" value="Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech&#10;  Synthesis via Phone-Level Content-Style Disentanglement" />
          <attvalue for="1" value="  This paper presents a novel design of neural network system for fine-grained&#10;style modeling, transfer and prediction in expressive text-to-speech (TTS)&#10;synthesis. Fine-grained modeling is realized by extracting style embeddings&#10;from the mel-spectrograms of phone-level speech segments. Collaborative&#10;learning and adversarial learning strategies are applied in order to achieve&#10;effective disentanglement of content and style factors in speech and alleviate&#10;the &quot;content leakage&quot; problem in style modeling. The proposed system can be&#10;used for varying-content speech style transfer in the single-speaker scenario.&#10;The results of objective and subjective evaluation show that our system&#10;performs better than other fine-grained speech style transfer models,&#10;especially in the aspect of content preservation. By incorporating a style&#10;predictor, the proposed system can also be used for text-to-speech synthesis.&#10;Audio samples are provided for system demonstration&#10;https://daxintan-cuhk.github.io/pl-csd-speech .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.07195" label="1905.07195">
        <attvalues>
          <attvalue for="0" value="CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven&#10;  Dynamic Hierarchical Conditional Variational Network" />
          <attvalue for="1" value="  The prosodic aspects of speech signals produced by current text-to-speech&#10;systems are typically averaged over training material, and as such lack the&#10;variety and liveliness found in natural speech. To avoid monotony and averaged&#10;prosody contours, it is desirable to have a way of modeling the variation in&#10;the prosodic aspects of speech, so audio signals can be synthesized in multiple&#10;ways for a given text. We present a new, hierarchically structured conditional&#10;variational autoencoder to generate prosodic features (fundamental frequency,&#10;energy and duration) suitable for use with a vocoder or a generative model like&#10;WaveNet. At inference time, an embedding representing the prosody of a sentence&#10;may be sampled from the variational layer to allow for prosodic variation. To&#10;efficiently capture the hierarchical nature of the linguistic input (words,&#10;syllables and phones), both the encoder and decoder parts of the auto-encoder&#10;are hierarchical, in line with the linguistic structure, with layers being&#10;clocked dynamically at the respective rates. We show in our experiments that&#10;our dynamic hierarchical network outperforms a non-hierarchical&#10;state-of-the-art baseline, and, additionally, that prosody transfer across&#10;sentences is possible by employing the prosody embedding of one sentence to&#10;generate the speech signal of another.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.14875" label="2407.14875">
        <attvalues>
          <attvalue for="0" value="Seal: Advancing Speech Language Models to be Few-Shot Learners" />
          <attvalue for="1" value="  Existing auto-regressive language models have demonstrated a remarkable&#10;capability to perform a new task with just a few examples in prompt, without&#10;requiring any additional training. In order to extend this capability to a&#10;multi-modal setting (i.e. speech and language), this paper introduces the Seal&#10;model, an abbreviation for speech language model. It incorporates a novel&#10;alignment method, in which Kullback-Leibler divergence loss is performed to&#10;train a projector that bridges a frozen speech encoder with a frozen language&#10;model decoder. The resulting Seal model exhibits robust performance as a&#10;few-shot learner on two speech understanding tasks. Additionally, consistency&#10;experiments are conducted to validate its robustness on different pre-trained&#10;language models.&#10;" />
          <attvalue for="2" value="&#10;Auto-regressive language models \cite{gpt3} have a very impressive ability to perform a new task with a few examples without any additional fine-tuning, named few-shot learners.&#10;Despite these remarkable capabilities, such language models are deaf to speech modalities. &#10;Therefore, a separate automatic speech recognition (ASR) model is necessary to transcribe speech into text, enabling language models to act as few-shot learners for speech modal, which suffers from the transcription errors caused by ASR.&#10;&#10;This paper aims to enhance the few-shot capabilities of language models for the speech modality.&#10;The foremost challenge of advancing language models to be speech few-shot learners is that aligning speech features into word embedding space of pre-trained language models effectively.&#10;Prior speech language models \cite{salmonn,qwen-audio} utilize next token prediction task to align speech features into word embedding space, where the desired next tokens are generated by task-related prompt and speech features.&#10;These configurations are expressed as $P(o|s,p)$, where $s=[s_1,...,s_l]$ represents speech features, $p=[p_1,...,p_m]$ signifies the task-related prompt, and $o=[o_1,...,o_n]$ denotes the intended output text.&#10;This formulation has a negative impact on context learning, as it overlooks the fact that speech features and their corresponding transcripts should exhibit the same behaviors in pre-trained language models, which should be formulated as $P(o|s,p)=P(o|t,p)$, where the $t=[t_1,...,t_I]$ denotes the transcripts of speech features.&#10;&#10;In the research area of mechanistic interpretability, \cite{induction} introduces that the ability of context-learning is derived from an induction head, a circuit whose complete the pattern by copying and completing sequences that have occurred before.&#10;Inspired by induction heads, this paper argue that extending pre-trained language models to function as few-shot learners for speech should be formulated as probabilistic models $P(o|s) = P(o|t)$.&#10;Therefore, this paper introduces a speech language model that consists of three parts: a frozen speech encoder, a trainable projector, and a frozen language model, where the trainable projector maps the speech features to the word embedding space of language models.&#10;The projector is trained with a noval aligment method that Kullback-Leibler divergence loss is performed as probabilistic models $P(o|s) = P(o|t)$.&#10;In implementation stage, $P(o|s) = P(o|t)$ is computed as follows&#10;{&#10;&#10;\begin{equation}&#10; D_{KL}(P(o|t)||P(o|s))=\sum_{j=0}^{J}\sum_{i=0}^{I}P(o_j^i|t,t_j^i)log\frac{P(o_j^i|t,t_j^i)}{P(o_j^i|s,t_j^i)},&#10;\end{equation}&#10;}&#10;where $j$, a hyper-parameter, represents the number of duplicate transcripts.&#10;This is mainly different from \cite{BLSP,BLSPKD}, they only use one simplex prompt `Continue the following text', which may lead to task overfitting phenomenon as described in \cite{salmonn}.&#10;&#10;Extensive experiments demonstrate that the resulting Seal model exhibits robust performance as a few-shot learner on two speech understanding tasks, including FSC \cite{fsc} and SLURP \cite{slurp}.&#10;Considering the alignment robustness for various pre-trained language models, phi2 and phi3 pre-trained language models are employed as frozen language model back end.&#10;Experimental results show that the proposed alignment approach achieves the same performance on difference pre-trained language models as speech few-shot learner.&#10;&#10;Our contributions are summarized as follows:&#10;&#10;\begin{itemize}&#10; \item We present an innovative alignment technique to enhance pre-trained language models to be speech few-shot learners. This involves using Kullback-Leibler divergence loss to align speech features with their corresponding transcripts within the pre-trained language models.&#10; \item We conduct enrich experiments on various speech understanding tasks for different language models to valid the effectiveness and robustness of our proposed alignment method.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Models, Artificial Intelligence, Mathematics, Multi-Modal Learning, Speech Understanding" />
        </attvalues>
      </node>
      <node id="2309.00916" label="2309.00916">
        <attvalues>
          <attvalue for="0" value="BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment&#10;  of Continuation Writing" />
          <attvalue for="1" value="  The emergence of large language models (LLMs) has sparked significant&#10;interest in extending their remarkable language capabilities to speech.&#10;However, modality alignment between speech and text still remains an open&#10;problem. Current solutions can be categorized into two strategies. One is a&#10;cascaded approach where outputs (tokens or states) of a separately trained&#10;speech recognition system are used as inputs for LLMs, which limits their&#10;potential in modeling alignment between speech and text. The other is an&#10;end-to-end approach that relies on speech instruction data, which is very&#10;difficult to collect in large quantities. In this paper, we address these&#10;issues and propose the BLSP approach that Bootstraps Language-Speech&#10;Pre-training via behavior alignment of continuation writing. We achieve this by&#10;learning a lightweight modality adapter between a frozen speech encoder and an&#10;LLM, ensuring that the LLM exhibits the same generation behavior regardless of&#10;the modality of input: a speech segment or its transcript. The training process&#10;can be divided into two steps. The first step prompts an LLM to generate texts&#10;with speech transcripts as prefixes, obtaining text continuations. In the&#10;second step, these continuations are used as supervised signals to train the&#10;modality adapter in an end-to-end manner. We demonstrate that this&#10;straightforward process can extend the capabilities of LLMs to speech, enabling&#10;speech recognition, speech translation, spoken language understanding, and&#10;speech conversation, even in zero-shot cross-lingual scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.06800" label="2407.06800">
        <attvalues>
          <attvalue for="0" value="Learn and Don't Forget: Adding a New Language to ASR Foundation Models" />
          <attvalue for="1" value="  Foundation ASR models often support many languages, e.g. 100 languages in&#10;Whisper. However, there has been limited work on integrating an additional,&#10;typically low-resource, language, while maintaining performance on the original&#10;language set. Fine-tuning, while simple, may degrade the accuracy of the&#10;original set. We compare three approaches that exploit adaptation parameters:&#10;soft language code tuning, train only the language code; soft prompt tuning,&#10;train prepended tokens; and LoRA where a small set of additional parameters are&#10;optimised. Elastic Weight Consolidation (EWC) offers an alternative compromise&#10;with the potential to maintain performance in specific target languages.&#10;Results show that direct fine-tuning yields the best performance for the new&#10;language but degrades existing language capabilities. EWC can address this&#10;issue for specific languages. If only adaptation parameters are used, the&#10;language capabilities are maintained but at the cost of performance in the new&#10;language.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years have seen impressive advancements in Automatic Speech Recognition (ASR) systems, particularly for languages with abundant linguistic resources, leading to high performance. However, the situation is different for ASR systems tailored to low-resource languages, the performance of which still falls short when compared to their high-resource counterparts. Low-resource languages typically lack a strong online presence, linguistic expertise, sufficient speech and text data, and pronunciation lexicons. These characteristics pose challenges for developing effective ASR systems in such languages.&#10;&#10;In the era of hybrid DNN-HMM models, extensive research on ASR for low-resource languages focused on building propriety lexicons for modelling the acoustic units attracts. \cite{qian2022automatic, lonergan2022cross} proposed a global lexicon, which captures dialect variant forms with relatively abstract representations, and a multi-dialect lexicon containing all dialect variants for Irish. &#10;Another approach was to develop graphemic lexicons where the ``pronunciation'' for a word is defined by the letters forming the word e.g. \cite{gales2015unicode}. End-to-end ASR models don't require a lexicon, so this doesn't affect performance. However, the lack of transcribed training data, as for the hybrid models, is a big problem for low-resources languages.&#10;&#10;To handle the lack of training data, a common method is using self-supervised or semi-supervised training~\cite{zhu22c_interspeech,synnaeve2020end, vesely2017semi}, where a model initially trained with limited transcribed data is used to generate transcriptions for unlabeled data which are then added to the training data to update the model. Data augmentation can also be used to tackle this problem where additional data is synthesised from existing data e.g.~\cite{xie23_interspeech,bartelds2023making,park19specaug,ragni14data}.&#10;&#10;With the development and release of pre-trained multilingual end-to-end foundation ASR models~\cite{radford2023robust,zhang2023google,babu2021xls}, utilising a multilingual model is another option for low-resource languages. Zhang et al. proposed a smaller universal monolingual output layer shared across languages for high-quality and high-efficiency multilingual ASR~\cite{zhang2023uml}.&#10;Chen et al. used hierarchical CTC to leverage language identity throughout the entire encoder-decoder network, aiming to improve ASR performance by correctly identifying languages~\cite{chen2023improving}. Qin et al. utilised multilingual and multilevel unit modeling to improve ASR performance on low-resource Tibetan languages~\cite{qin2022improving}.&#10;&#10;The challenge in multilingual models lies in expanding them to new languages without compromising the performance of existing ones, particularly when the training data for new languages is limited or unavailable. &#10;In NLP, approaches like language-specific adapters and output heads, as seen in~\cite{he2021towards, houlsby2019parameter}, enable parameter-efficient fine-tuning but necessitate prior knowledge of the input language. Alternatively, continual lifelong learning, as proposed by \cite{parisi2019continual}, combines data from both existing and new languages for ongoing training. Demonstrating its effectiveness in ASR, \cite{li2021scaling,li2022massively} have expanded language coverage from 15 to 32 and to 66 languages, respectively.&#10;Pham et al. proposed a weight factorization technique to factorize each weight matrix in the network into language dependent and independent factors~\cite{pham21efficient}. When combined with Elastic Weight Consolidation (EWC), this approach allows a multilingual ASR model to expand its learning from an initial 10 languages to 26 languages without catastrophic forgetting~\cite{pham23towards,Pham2023thesis}.&#10;&#10;In this paper, we focus on adding a new language to an ASR foundation model, using Whisper as a case study. Firstly, the zero-shot ability of the foundation model is examined. Secondly, we compare efficient fine-tuning methods such as Low Rank Adaptation (LoRA) with standard fine-tuning. Moreover, we propose Soft Language Code Tuning (SLCT) to train a language-specific embedding vector for the new language. &#10;Inspired by techniques from NLP tasks, we implement Soft Prompt Tuning (SPT), introduced in~\cite{lester2021power} and utilised in a language assessment task~\cite{ma2023adapting_slate}, to effectively expand language coverage.&#10;For fine-tuning, we analysed catastrophic forgetting and mitigated it with Elastic Weight Consolidation (EWC). Initially, we assess the zero-shot ability of Whisper on six languages not originally supported in the model, then focus on three languages to evaluate the performance of various approaches.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Adaptation Techniques, Artificial Intelligence, Multilingual ASR Models, Performance Optimization Methods" />
        </attvalues>
      </node>
      <node id="2110.04484" label="2110.04484">
        <attvalues>
          <attvalue for="0" value="Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASR" />
          <attvalue for="1" value="  Self-supervised pre-training could effectively improve the performance of&#10;low-resource automatic speech recognition (ASR). However, existing&#10;self-supervised pre-training are task-agnostic, i.e., could be applied to&#10;various downstream tasks. Although it enlarges the scope of its application,&#10;the capacity of the pre-trained model is not fully utilized for the ASR task,&#10;and the learned representations may not be optimal for ASR. In this work, in&#10;order to build a better pre-trained model for low-resource ASR, we propose a&#10;pre-training approach called wav2vec-S, where we use task-specific&#10;semi-supervised pre-training to refine the self-supervised pre-trained model&#10;for the ASR task thus more effectively utilize the capacity of the pre-trained&#10;model to generate task-specific representations for ASR. Experiments show that&#10;compared to wav2vec 2.0, wav2vec-S only requires a marginal increment of&#10;pre-training time but could significantly improve ASR performance on in-domain,&#10;cross-domain and cross-lingual datasets. Average relative WER reductions are&#10;24.5% and 6.6% for 1h and 10h fine-tuning, respectively. Furthermore, we show&#10;that semi-supervised pre-training could close the representation gap between&#10;the self-supervised pre-trained model and the corresponding fine-tuned model&#10;through canonical correlation analysis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;The idea to adapt the task-agnostic self-supervised pre-trained model to an ASR task-specific pre-trained model is also explored in other works. \cite{wang2021self} forces the&#10;model to concentrate on ASR-related information by adding the self-supervised losses on intermediate layers. Other works \cite{wang2021unispeech,bai2021joint,zhang2021xlst,chen2021speech} utilize labeled data to inject ASR task information into the pre-trained model. Our work belongs to this category. Among them, Unispeech \cite{wang2021unispeech} uses multi-task learning to conduct semi-supervised pre-training, where contrastive loss is used on the unlabeled data and CTC loss is used on the labeled data. JUST \cite{bai2021joint} jointly optimizes two self-supervised losses and a supervised RNN-T loss. XLST \cite{zhang2021xlst} uses supervised training as the initialization and then conducts self-training on the unlabeled data. In our work, CTC loss is used on both labeled and unlabeled data, where the ground-truth labels are used for labeled data and pseudo labels are used for unlabeled data. Since previous work mostly conducts task-specific pre-training from scratch, substantial training time is required for each task. In this work, we treat semi-supervised pre-training as the task-specific refinement of the self-supervised pre-training. Thus, it can benefit from the initialization of the self-supervised pre-trained model for faster convergence. Concurrent works \cite{hwang2021large,zhang2021bigssl} also explored the combination of self-supervised pre-training and semi-supervised learning, where \cite{hwang2021large} focused on the domain adaptation and \cite{zhang2021bigssl} focused on the large-scale applications.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2310.18450" label="2310.18450">
        <attvalues>
          <attvalue for="0" value="MixRep: Hidden Representation Mixup for Low-Resource Speech Recognition" />
          <attvalue for="1" value="  In this paper, we present MixRep, a simple and effective data augmentation&#10;strategy based on mixup for low-resource ASR. MixRep interpolates the feature&#10;dimensions of hidden representations in the neural network that can be applied&#10;to both the acoustic feature input and the output of each layer, which&#10;generalizes the previous MixSpeech method. Further, we propose to combine the&#10;mixup with a regularization along the time axis of the input, which is shown as&#10;complementary. We apply MixRep to a Conformer encoder of an E2E LAS&#10;architecture trained with a joint CTC loss. We experiment on the WSJ dataset&#10;and subsets of the SWB dataset, covering reading and telephony conversational&#10;speech. Experimental results show that MixRep consistently outperforms other&#10;regularization methods for low-resource ASR. Compared to a strong SpecAugment&#10;baseline, MixRep achieves a +6.5\% and a +6.7\% relative WER reduction on the&#10;eval92 set and the Callhome part of the eval'2000 set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10951" label="2305.10951">
        <attvalues>
          <attvalue for="0" value="Making More of Little Data: Improving Low-Resource Automatic Speech&#10;  Recognition Using Data Augmentation" />
          <attvalue for="1" value="  The performance of automatic speech recognition (ASR) systems has advanced&#10;substantially in recent years, particularly for languages for which a large&#10;amount of transcribed speech is available. Unfortunately, for low-resource&#10;languages, such as minority languages, regional languages or dialects, ASR&#10;performance generally remains much lower. In this study, we investigate whether&#10;data augmentation techniques could help improve low-resource ASR performance,&#10;focusing on four typologically diverse minority languages or language variants&#10;(West Germanic: Gronings, West-Frisian; Malayo-Polynesian: Besemah, Nasal). For&#10;all four languages, we examine the use of self-training, where an ASR system&#10;trained with the available human-transcribed data is used to generate&#10;transcriptions, which are then combined with the original data to train a new&#10;ASR system. For Gronings, for which there was a pre-existing text-to-speech&#10;(TTS) system available, we also examined the use of TTS to generate ASR&#10;training data from text-only sources. We find that using a self-training&#10;approach consistently yields improved performance (a relative WER reduction up&#10;to 20.5% compared to using an ASR system trained on 24 minutes of manually&#10;transcribed speech). The performance gain from TTS augmentation for Gronings&#10;was even stronger (up to 25.5% relative reduction in WER compared to a system&#10;based on 24 minutes of manually transcribed speech). In sum, our results show&#10;the benefit of using self-training or (if possible) TTS-generated data as an&#10;efficient solution to overcome the limitations of data availability for&#10;resource-scarce languages in order to improve ASR performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.11186" label="2302.11186">
        <attvalues>
          <attvalue for="0" value="UML: A Universal Monolingual Output Layer for Multilingual ASR" />
          <attvalue for="1" value="  Word-piece models (WPMs) are commonly used subword units in state-of-the-art&#10;end-to-end automatic speech recognition (ASR) systems. For multilingual ASR,&#10;due to the differences in written scripts across languages, multilingual WPMs&#10;bring the challenges of having overly large output layers and scaling to more&#10;languages. In this work, we propose a universal monolingual output layer (UML)&#10;to address such problems. Instead of one output node for only one WPM, UML&#10;re-associates each output node with multiple WPMs, one for each language, and&#10;results in a smaller monolingual output layer shared across languages.&#10;Consequently, the UML enables to switch in the interpretation of each output&#10;node depending on the language of the input speech. Experimental results on an&#10;11-language voice search task demonstrated the feasibility of using UML for&#10;high-quality and high-efficiency multilingual streaming ASR.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.12829" label="2302.12829">
        <attvalues>
          <attvalue for="0" value="Improving Massively Multilingual ASR With Auxiliary CTC Objectives" />
          <attvalue for="1" value="  Multilingual Automatic Speech Recognition (ASR) models have extended the&#10;usability of speech technologies to a wide variety of languages. With how many&#10;languages these models have to handle, however, a key to understanding their&#10;imbalanced performance across different languages is to examine if the model&#10;actually knows which language it should transcribe. In this paper, we introduce&#10;our work on improving performance on FLEURS, a 102-language open ASR benchmark,&#10;by conditioning the entire model on language identity (LID). We investigate&#10;techniques inspired from recent Connectionist Temporal Classification (CTC)&#10;studies to help the model handle the large number of languages, conditioning on&#10;the LID predictions of auxiliary tasks. Our experimental results demonstrate&#10;the effectiveness of our technique over standard CTC/Attention-based hybrid&#10;models. Furthermore, our state-of-the-art systems using self-supervised models&#10;with the Conformer architecture improve over the results of prior work on&#10;FLEURS by a relative 28.4% CER. Trained models and reproducible recipes are&#10;available at https://github.com/espnet/espnet/tree/master/egs2/fleurs/asr1 .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.03010" label="2105.03010">
        <attvalues>
          <attvalue for="0" value="Efficient Weight factorization for Multilingual Speech Recognition" />
          <attvalue for="1" value="  End-to-end multilingual speech recognition involves using a single model&#10;training on a compositional speech corpus including many languages, resulting&#10;in a single neural network to handle transcribing different languages. Due to&#10;the fact that each language in the training data has different characteristics,&#10;the shared network may struggle to optimize for all various languages&#10;simultaneously. In this paper we propose a novel multilingual architecture that&#10;targets the core operation in neural networks: linear transformation functions.&#10;The key idea of the method is to assign fast weight matrices for each language&#10;by decomposing each weight matrix into a shared component and a language&#10;dependent component. The latter is then factorized into vectors using rank-1&#10;assumptions to reduce the number of parameters per language. This efficient&#10;factorization scheme is proved to be effective in two multilingual settings&#10;with $7$ and $27$ languages, reducing the word error rates by $26\%$ and $27\%$&#10;rel. for two popular architectures LSTM and Transformer, respectively.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multilingual modeling has been an important topic in applying sequence-to-squence models to language applications ranging from machine translation~\cite{ha2016toward,johnson2017google} to automatic speech recognition (ASR)~\cite{Waibel2000}. It is possible to employ one single neural model for multiple datasets with different languages with the goal of capturing the shared features between the languages. This method has been widely used to help under-resourced languages benefiting from the knowledge acquired from the richer counterparts. &#10;&#10;It is noticeable that the recent multilingual neural models are based on a semi-shared mechanism in which the largest body of the network architecture is exposed to all languages, while a smaller weight subset provides a language specific bias. This was shown to be more effective in a multilingual scenario than fully sharing the whole network\cite{ha2016toward,johnson2017google} since each language has certain unique features, and the single architecture often struggles to handle a variety of languages~\cite{platanios-etal-2018-contextual}. &#10;&#10;There are two main drawbacks that are typically presented in the existing implementations of the semi-shared mechanism. On the one hand, the implementations often depends heavily on a certain architecture being popular at the time, and the given improvement is going to be diminished when a new architecture evolves. For example, the language-specifically biased attention~\cite{zhu2020multilingual} modified the self-attention architecture~\cite{vaswani2017attention} specifically based on the assumption that each language can benefit from a bias added to the attention scores. On the other hand, the language-dependent components might require a considerable amount of parameters and struggles to scale to the number of languages. For example, the language adapters added to the Transformer layers~\cite{bapna2019simple} are essentially feed-forward neural network layers being similar to the counterpart already in the shared Transformer body. A scenario with $20$ languages consequently generates hundreds of these layers accounting for a large amount of parameters to be optimized. &#10;&#10;In this work, we propose a multilingual architecture using a factorization scheme that is both effective and highly scalable with the number of languages involved. Moreover, this scheme is applicable to any neural architectures as long as matrix-vector multiplication is the dominant operation. The key idea of our work is that each weight matrix in the shared architecture can be factorized into a shared component and multiple additive and multiplicative language dependent components. While each language is assigned with extra weights to learn distinctive features, simplicity and scalability are achieved by further representing those weights into as a rank-1 matrix, thus can be factored into two vectors. This method is demonstrated to be computational friendly with a minimal overhead and can be applied to a arbitrary neural architecture.&#10;&#10;Subsequently, this weight factorization method is then evaluated on two different scenarios: one with $7$ languages having similar amounts of data, and one with $27$ languages with various extremely low resource data. The method is implemented on two commonly used architectures: Long Short-Term Memories (LSTM) and Transformers which show that both types of networks can benefit by weight factorization in multilingual ASR. The reduction of error rate can be up to $47\%$ rel. in the case of low-resource languages such as Japanese and $15.5\%$ rel. on average with the moderately sized languages. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.11703" label="2211.11703">
        <attvalues>
          <attvalue for="0" value="Towards continually learning new languages" />
          <attvalue for="1" value="  Multilingual speech recognition with neural networks is often implemented&#10;with batch-learning, when all of the languages are available before training.&#10;An ability to add new languages after the prior training sessions can be&#10;economically beneficial, but the main challenge is catastrophic forgetting. In&#10;this work, we combine the qualities of weight factorization and elastic weight&#10;consolidation in order to counter catastrophic forgetting and facilitate&#10;learning new languages quickly. Such combination allowed us to eliminate&#10;catastrophic forgetting while still achieving performance for the new languages&#10;comparable with having all languages at once, in experiments of learning from&#10;an initial 10 languages to achieve 26 languages without catastrophic forgetting&#10;and a reasonable performance compared to training all languages from scratch.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.09378" label="2307.09378">
        <attvalues>
          <attvalue for="0" value="Adapting an ASR Foundation Model for Spoken Language Assessment" />
          <attvalue for="1" value="  A crucial part of an accurate and reliable spoken language assessment system&#10;is the underlying ASR model. Recently, large-scale pre-trained ASR foundation&#10;models such as Whisper have been made available. As the output of these models&#10;is designed to be human readable, punctuation is added, numbers are presented&#10;in Arabic numeric form and abbreviations are included. Additionally, these&#10;models have a tendency to skip disfluencies and hesitations in the output.&#10;Though useful for readability, these attributes are not helpful for assessing&#10;the ability of a candidate and providing feedback. Here a precise transcription&#10;of what a candidate said is needed. In this paper, we give a detailed analysis&#10;of Whisper outputs and propose two solutions: fine-tuning and soft prompt&#10;tuning. Experiments are conducted on both public speech corpora and an English&#10;learner dataset. Results show that we can effectively alter the decoding&#10;behaviour of Whisper to generate the exact words spoken in the response.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.00004" label="2012.00004">
        <attvalues>
          <attvalue for="0" value="UWB at SemEval-2020 Task 1: Lexical Semantic Change Detection" />
          <attvalue for="1" value="  In this paper, we describe our method for the detection of lexical semantic&#10;change, i.e., word sense changes over time. We examine semantic differences&#10;between specific words in two corpora, chosen from different time periods, for&#10;English, German, Latin, and Swedish. Our method was created for the SemEval&#10;2020 Task 1: \textit{Unsupervised Lexical Semantic Change Detection.} We ranked&#10;$1^{st}$ in Sub-task 1: binary change detection, and $4^{th}$ in Sub-task 2:&#10;ranked change detection. Our method is fully unsupervised and language&#10;independent. It consists of preparing a semantic vector space for each corpus,&#10;earlier and later; computing a linear transformation between earlier and later&#10;spaces, using Canonical Correlation Analysis and Orthogonal Transformation; and&#10;measuring the cosines between the transformed vector for the target word from&#10;the earlier corpus and the vector for the target word in the later corpus.&#10;" />
          <attvalue for="2" value="&#10;&#10;\blfootnote{&#10; $^{*}$Equal contribution.&#10;}&#10;\blfootnote{&#10; This work is licensed under a Creative Commons &#10; Attribution 4.0 International Licence.&#10; Licence details:&#10; .&#10;}&#10;&#10;Language evolves with time. New words appear, old words fall out of use, the meanings of some words shift.&#10;The culture changes as well as the expected audience of the printed word. There are changes in topics, in syntax, in presentation structure. Reading the natural philosophy musings of aristocratic amateurs from the &#10;eighteenth century, and comparing with a monograph from the nineteenth century, or a medical study from the twentieth century, we can observe differences in many dimensions, some of which seem hard to study. Changes in word senses are both a visible and a tractable part of language evolution.&#10;Computational methods for researching words' stories have the potential of helping us understand this small corner of linguistic evolution. The tools for measuring these diachronic semantic shifts might also be useful for measuring whether the same word is used in different ways in synchronic documents. The task of finding word sense changes over time is called diachronic Lexical Semantic Change (LSC) detection. The task&#10;is getting more attention in recent years \cite{hamilton-etal-2016-diachronic,frermann-lapata-2016-bayesian,schlechtweg-etal-2017-german}. There is also the synchronic LSC task, which aims to identify domain-specific changes of word senses compared to general-language usage \cite{schlechtweg-etal-2019-wind}. &#10;&#10;\newcite{tahmasebi2018survey} provides a comprehensive survey of techniques for the LSC task, as does \newcite{kutuzov-etal-2018-diachronic}. \newcite{schlechtweg-etal-2019-wind} evaluated available approaches for LSC detection using the DURel dataset \cite{schlectweg-etal-DURel}.&#10; Some of the methodologies for finding time-sensitive meanings were borrowed from information retrieval techniques in the first place. According to \newcite{schlechtweg-etal-2019-wind}, there are mainly three types of approaches. (1) Semantic vector spaces approaches \cite{gulordava-baroni-2011-distributional,kim-etal-2014-temporal,Xu2015ACE,eger-mehler-2016-linearity,hamilton-etal-2016-cultural,hamilton-etal-2016-diachronic,rosenfeld-erk-2018-deep} represent each word with two vectors for two different time periods. The change of meaning is then measured by the cosine distance between the two vectors. (2) Topic modeling approaches \cite{wang-topics,bamman-topics,Wijaya-topic,mihalcea-nastase-2012-word,cook-etal-2014-novel,frermann-lapata-2016-bayesian,Schlechtweg20} estimate a probability distribution of words over their different senses, i.e., topics. (3) Clustering models \cite{mitra2015automatic,tahmasebi-risse-2017-finding} are used to cluster words into clusters representing different senses.&#10;&#10;We participated in the SemEval 2020 Task 1: Unsupervised Lexical Semantic Change Detection \cite{sem20-task1-overview} competition. In this paper, we describe our solution and submitted systems for this competition. The task consists of two sub-tasks, a binary classification task (Sub-task 1) and a ranking task (Sub-task 2), which involve comparing usage of target words between two lemmatized corpora, each drawn from documents from a different time period for four languages: English, German, Latin, and Swedish. For both sub-tasks, only the target words and two corpora for each language were provided by organizers, no annotated data. &#10;The task is intended to be solved in a completely unsupervised way.&#10;&#10;In the binary classification task, the goal is for two given corpora $C_1$ and $C_2$ (for time $t_1$ and $t_2$) and for a set of target words, decide which of these words changed or did not change their sense (semantic) between $t_1$ and $t_2$. Change of sense is whether the word lost or gained any sense between the two periods (corpora).&#10;The objective of the Sub-task 2 is for two given corpora $C_1$ and $C_2$, rank a set of target words according to their degree of lexical semantic change between $t_1$ and $t_2$.&#10;A higher rank means a stronger change. Target words are the same for both sub-tasks.&#10;&#10;Because language is evolving, expressions, words, and sentence constructions in two corpora from different time periods about the same topic &#10;will be written in languages that are quite similar but slightly different. They will share the majority of their words, grammar, and syntax. &#10;&#10;The main idea behind our solution is that we treat each pair of corpora $C_1$ and $C_2$ as different languages $L_1$ and $L_2$ even though that text from both corpora is written in the same language.&#10;We believe that these two languages $L_1$ and $L_2$ will be extremely similar in all aspects, including semantic. We train separate semantic space for each corpus and subsequently, we map these two spaces into one common cross-lingual space. We use methods for cross-lingual mapping \cite{Brychcin2019,artetxe-labaka-agirre:2016:EMNLP2016,artetxe-etal-2017,artetxe-etal-2018b,artetxe-etal-2018-robust} and thanks to the large similarity between $L_1$ and $L_2$ the quality of transformation should be high. We compute cosine similarity to classify and rank the target words, see Section \ref{sec:system-desc} for details. &#10;&#10;\nocite{Hardoon:2004}\nocite{artetxe-etal-2018-robust}\nocite{Hardoon:2004,artetxe-labaka-agirre:2016:EMNLP2016,FaruquiDyer14,Ammar2016,Brychcin2019}&#10;&#10;Our systems ranked $1^{st}$ out of $33$ teams in Sub-task 1 with an average accuracy of $0.687$, and $4^{th}$ out of $32$ teams in Sub-task 2 with an average Spearman's rank correlation of $0.481$. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Mathematics, Natural Language Processing, Language Modeling, Lexical Semantic Change" />
        </attvalues>
      </node>
      <node id="1706.04971" label="1706.04971">
        <attvalues>
          <attvalue for="0" value="German in Flux: Detecting Metaphoric Change via Word Entropy" />
          <attvalue for="1" value="  This paper explores the information-theoretic measure entropy to detect&#10;metaphoric change, transferring ideas from hypernym detection to research on&#10;language change. We also build the first diachronic test set for German as a&#10;standard for metaphoric change annotation. Our model shows high performance, is&#10;unsupervised, language-independent and generalizable to other processes of&#10;semantic change.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.06517" label="1804.06517">
        <attvalues>
          <attvalue for="0" value="Diachronic Usage Relatedness (DURel): A Framework for the Annotation of&#10;  Lexical Semantic Change" />
          <attvalue for="1" value="  We propose a framework that extends synchronic polysemy annotation to&#10;diachronic changes in lexical meaning, to counteract the lack of resources for&#10;evaluating computational models of lexical semantic change. Our framework&#10;exploits an intuitive notion of semantic relatedness, and distinguishes between&#10;innovative and reductive meaning changes with high inter-annotator agreement.&#10;The resulting test set for German comprises ratings from five annotators for&#10;the relatedness of 1,320 use pairs across 22 target words.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.02821" label="1606.02821">
        <attvalues>
          <attvalue for="0" value="Cultural Shift or Linguistic Drift? Comparing Two Computational Measures&#10;  of Semantic Change" />
          <attvalue for="1" value="  Words shift in meaning for many reasons, including cultural factors like new&#10;technologies and regular linguistic processes like subjectification.&#10;Understanding the evolution of language and culture requires disentangling&#10;these underlying causes. Here we show how two different distributional measures&#10;can be used to detect two different types of semantic change. The first&#10;measure, which has been used in many previous works, analyzes global shifts in&#10;a word's distributional semantics, it is sensitive to changes due to regular&#10;processes of linguistic drift, such as the semantic generalization of promise&#10;(&quot;I promise.&quot; -&gt; &quot;It promised to be exciting.&quot;). The second measure, which we&#10;develop here, focuses on local changes to a word's nearest semantic neighbors;&#10;it is more sensitive to cultural shifts, such as the change in the meaning of&#10;cell (&quot;prison cell&quot; -&gt; &quot;cell phone&quot;). Comparing measurements made by these two&#10;methods allows researchers to determine whether changes are more cultural or&#10;linguistic in nature, a distinction that is essential for work in the digital&#10;humanities and historical linguistics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.03216" label="2001.03216">
        <attvalues>
          <attvalue for="0" value="Simulating Lexical Semantic Change from Sense-Annotated Data" />
          <attvalue for="1" value="  We present a novel procedure to simulate lexical semantic change from&#10;synchronic sense-annotated data, and demonstrate its usefulness for assessing&#10;lexical semantic change detection models. The induced dataset represents a&#10;stronger correspondence to empirically observed lexical semantic change than&#10;previous synthetic datasets, because it exploits the intimate relationship&#10;between synchronic polysemy and diachronic change. We publish the data and&#10;provide the first large-scale evaluation gold standard for LSC detection&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.04175" label="1807.04175">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Word Analogies using Linear Transformations between&#10;  Semantic Spaces" />
          <attvalue for="1" value="  We generalize the word analogy task across languages, to provide a new&#10;intrinsic evaluation method for cross-lingual semantic spaces. We experiment&#10;with six languages within different language families, including English,&#10;German, Spanish, Italian, Czech, and Croatian. State-of-the-art monolingual&#10;semantic spaces are transformed into a shared space using dictionaries of word&#10;translations. We compare several linear transformations and rank them for&#10;experiments with monolingual (no transformation), bilingual (one semantic space&#10;is transformed to another), and multilingual (all semantic spaces are&#10;transformed onto English space) versions of semantic spaces. We show that&#10;tested linear transformations preserve relationships between words (word&#10;analogies) and lead to impressive results. We achieve average accuracy of&#10;51.1%, 43.1%, and 38.2% for monolingual, bilingual, and multilingual semantic&#10;spaces, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.12233" label="2402.12233">
        <attvalues>
          <attvalue for="0" value="Empirical Study on Updating Key-Value Memories in Transformer&#10;  Feed-forward Layers" />
          <attvalue for="1" value="  The feed-forward networks (FFNs) in transformers are recognized as a group of&#10;key-value neural memories to restore abstract high-level knowledge. In this&#10;work, we conduct an empirical ablation study on updating keys (the 1st layer in&#10;the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those&#10;two methods in various knowledge editing and fine-tuning tasks of large&#10;language models to draw insights to understand FFNs further. Code is available&#10;at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$.&#10;" />
          <attvalue for="2" value="&#10;&#10;How do pre-trained Transformer models process and store information? &#10;~\cite{DBLP:conf/emnlp/GevaSBL21,DBLP:conf/emnlp/GevaCWG22} suggest feed-forward networks (FFNs) operate as key-value neural memories~\cite{DBLP:conf/nips/SukhbaatarSWF15}. &#10;Specifically, given hidden states $h\in \mathbb{R}^{d_e}$, where $d_e$ is embedding size, $FFNs(h)=f(h\cdot K^T)\cdot V$, where $K, V \in \mathbb{R}^{d_m \times d_e}$, $d_m$ is the width of FFNs and $f$ is the non-linear activation. &#10;Each row of $K$ can be viewed as a key correlated with input textual patterns, and each row of $V$ can be viewed as a value that induces a distribution shift over the residual stream.&#10;For example, when a model predicts the next token based on the prefix `Eiffel Tower is located in', $k_2$ is activated so that $v_2$ can promote the probability of `Paris' in the output. On the contrary, values for irrelevant concepts (e.g., $v_1$ for `Cat', $v_d$ for Soccer) are deactivated and can not dominate the output distribution.&#10;&#10;Based on this point of view, how can we update the information processing and storing? We employ Knowledge Editing (KE)~\cite{wang2023knowledge} as an illustrative example: altering `Paris' to `Seattle' in response to `Eiffel Tower is located in' by modifying the model weights.&#10;In this work, we compare two updating choices as shown in Figure~\ref{fig:key_tuning}:&#10;1. Values: &#10;tuning $v_2$ to shift its concept from `Paris' to `Seattle'.&#10;2. Keys: &#10;tuning $k_2$ to boost the activation of $v_i$ for the target concept `Seattle'.&#10;&#10;We test the two methods in various scenarios for different pre-trained transformers, including knowledge editing~\cite{DBLP:conf/emnlp/CaoAT21, DBLP:conf/iclr/HuangSZZR023}, &#10;multi-task tuning~\cite{DBLP:conf/iclr/AribandiTSRZMZ022}, and instruction-tuning~\cite{DBLP:conf/iclr/WeiBZGYLDDL22}. &#10;We generally recognize the superiority of updating keys over updating values.&#10;We contend that compared to directly modifying the model's knowledge (values), altering the mechanism of controlling this knowledge (keys) can be more effective.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Cognitive Science, Knowledge Representation, Artificial Intelligence, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2203.14680" label="2203.14680">
        <attvalues>
          <attvalue for="0" value="Transformer Feed-Forward Layers Build Predictions by Promoting Concepts&#10;  in the Vocabulary Space" />
          <attvalue for="1" value="  Transformer-based language models (LMs) are at the core of modern NLP, but&#10;their internal prediction construction process is opaque and largely not&#10;understood. In this work, we make a substantial step towards unveiling this&#10;underlying prediction process, by reverse-engineering the operation of the&#10;feed-forward network (FFN) layers, one of the building blocks of transformer&#10;models. We view the token representation as a changing distribution over the&#10;vocabulary, and the output from each FFN layer as an additive update to that&#10;distribution. Then, we analyze the FFN updates in the vocabulary space, showing&#10;that each update can be decomposed to sub-updates corresponding to single FFN&#10;parameter vectors, each promoting concepts that are often human-interpretable.&#10;We then leverage these findings for controlling LM predictions, where we reduce&#10;the toxicity of GPT2 by almost 50%, and for improving computation efficiency&#10;with a simple early exit rule, saving 20% of computation on average.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1503.08895" label="1503.08895">
        <attvalues>
          <attvalue for="0" value="End-To-End Memory Networks" />
          <attvalue for="1" value="  We introduce a neural network with a recurrent attention model over a&#10;possibly large external memory. The architecture is a form of Memory Network&#10;(Weston et al., 2015) but unlike the model in that work, it is trained&#10;end-to-end, and hence requires significantly less supervision during training,&#10;making it more generally applicable in realistic settings. It can also be seen&#10;as an extension of RNNsearch to the case where multiple computational steps&#10;(hops) are performed per output symbol. The flexibility of the model allows us&#10;to apply it to tasks as diverse as (synthetic) question answering and to&#10;language modeling. For the former our approach is competitive with Memory&#10;Networks, but with less supervision. For the latter, on the Penn TreeBank and&#10;Text8 datasets our approach demonstrates comparable performance to RNNs and&#10;LSTMs. In both cases we show that the key concept of multiple computational&#10;hops yields improved results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.16218" label="2310.16218">
        <attvalues>
          <attvalue for="0" value="Knowledge Editing for Large Language Models: A Survey" />
          <attvalue for="1" value="  Large language models (LLMs) have recently transformed both the academic and&#10;industrial landscapes due to their remarkable capacity to understand, analyze,&#10;and generate texts based on their vast knowledge and reasoning ability.&#10;Nevertheless, one major drawback of LLMs is their substantial computational&#10;cost for pre-training due to their unprecedented amounts of parameters. The&#10;disadvantage is exacerbated when new knowledge frequently needs to be&#10;introduced into the pre-trained model. Therefore, it is imperative to develop&#10;effective and efficient techniques to update pre-trained LLMs. Traditional&#10;methods encode new knowledge in pre-trained LLMs through direct fine-tuning.&#10;However, naively re-training LLMs can be computationally intensive and risks&#10;degenerating valuable pre-trained knowledge irrelevant to the update in the&#10;model. Recently, Knowledge-based Model Editing (KME) has attracted increasing&#10;attention, which aims to precisely modify the LLMs to incorporate specific&#10;knowledge, without negatively influencing other irrelevant knowledge. In this&#10;survey, we aim to provide a comprehensive and in-depth overview of recent&#10;advances in the field of KME. We first introduce a general formulation of KME&#10;to encompass different KME strategies. Afterward, we provide an innovative&#10;taxonomy of KME techniques based on how the new knowledge is introduced into&#10;pre-trained LLMs, and investigate existing KME strategies while analyzing key&#10;insights, advantages, and limitations of methods from each category. Moreover,&#10;representative metrics, datasets, and applications of KME are introduced&#10;accordingly. Finally, we provide an in-depth analysis regarding the&#10;practicality and remaining challenges of KME and suggest promising research&#10;directions for further advancement in this field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In this section, we provide an overview of the editing strategies for machine learning models and the basics of large language models (LLMs) as background knowledge to facilitate the understanding of technical details in KME. In this survey, we use bold uppercase letters (e.g., $\mathbf{K}$ and $\mathbf{V}$) to represent matrices, use lowercase bold letters (e.g., $\mathbf{k}$ and $\mathbf{v}$) to represent vectors, and use calligraphic uppercase letters (e.g., $\mathcal{X}$ and $\mathcal{Y}$) to represent sets. We summarize the primary notations used in this survey in Table~\ref{tab:notations} for the convenience of understanding.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2111.10952" label="2111.10952">
        <attvalues>
          <attvalue for="0" value="ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning" />
          <attvalue for="1" value="  Despite the recent success of multi-task learning and transfer learning for&#10;natural language processing (NLP), few works have systematically studied the&#10;effect of scaling up the number of tasks during pre-training. Towards this&#10;goal, this paper introduces ExMix (Extreme Mixture): a massive collection of&#10;107 supervised NLP tasks across diverse domains and task-families. Using ExMix,&#10;we study the effect of multi-task pre-training at the largest scale to date,&#10;and analyze co-training transfer amongst common families of tasks. Through this&#10;analysis, we show that manually curating an ideal set of tasks for multi-task&#10;pre-training is not straightforward, and that multi-task scaling can vastly&#10;improve models on its own. Finally, we propose ExT5: a model pre-trained using&#10;a multi-task objective of self-supervised span denoising and supervised ExMix.&#10;Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on&#10;SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of&#10;ExMix. ExT5 also significantly improves sample efficiency while pre-training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.07566" label="2210.07566">
        <attvalues>
          <attvalue for="0" value="A Survey of Parameters Associated with the Quality of Benchmarks in NLP" />
          <attvalue for="1" value="  Several benchmarks have been built with heavy investment in resources to&#10;track our progress in NLP. Thousands of papers published in response to those&#10;benchmarks have competed to top leaderboards, with models often surpassing&#10;human performance. However, recent studies have shown that models triumph over&#10;several popular benchmarks just by overfitting on spurious biases, without&#10;truly learning the desired task. Despite this finding, benchmarking, while&#10;trying to tackle bias, still relies on workarounds, which do not fully utilize&#10;the resources invested in benchmark creation, due to the discarding of low&#10;quality data, and cover limited sets of bias. A potential solution to these&#10;issues -- a metric quantifying quality -- remains underexplored. Inspired by&#10;successful quality indices in several domains such as power, food, and water,&#10;we take the first step towards a metric by identifying certain language&#10;properties that can represent various possible interactions leading to biases&#10;in a benchmark. We look for bias related parameters which can potentially help&#10;pave our way towards the metric. We survey existing works and identify&#10;parameters capturing various properties of bias, their origins, types and&#10;impact on performance, generalization, and robustness. Our analysis spans over&#10;datasets and a hierarchy of tasks ranging from NLI to Summarization, ensuring&#10;that our parameters are generic and are not overfitted towards a specific task&#10;or dataset. We also develop certain parameters in this process.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large-scale Benchmarks such as SNLI \cite{bowman2015large}, SQUAD \cite{rajpurkar2016squad}, GLUE \cite{wang2018glue}, and Senteval \cite{conneau2018senteval} have been guiding our progress in NLP over the years. Benchmark creation and subsequent model development have involved heavy investment in resources such as money and time. &#10;Often, language models such as BERT \cite{devlin2018bert}, RoBERTA \cite{liu2019roberta} and GPT3 \cite{brown2020language} beat human performance. A growing number of recent works \cite{gururangan2018annotation, poliak2018hypothesis, kaushik2018much, tsuchiya2018performance, tan2019investigating, schwartz2017effect, swayamdipta-etal-2020-dataset, gardner-etal-2021-competency, pezeshkpour2021combining} however, expose an undesired reason this can happen: instead of learning tasks like humans, models simply overfit to spurious biases. Therefore, we must reexamine the process of creating and solving benchmarks.&#10;&#10;Several algorithms \cite{sakaguchi2019winogrande,li2019repair,li2018resound, wang2018dataset,clark2019don,he2019unlearn,mahabadi2019simple,zellers2018swag,nie2019adversarial,kaushik2019learning,gardner2020evaluating} have been proposed over the past few years to tackle the issue of bias. However, there is significant opportunities as we need methods that can (i) provide a metric to quantify quality, (ii) justify the original investment in benchmark creation as they discard bad samples, (iii) cover diverse categories of bias, (iv) are generic, i.e. independent of models and tasks, and automated, i.e., free of human intervention in judging data quality. A metric that quantifies benchmark quality could serve as a single potential solution to all these issues. An important question which has remained underexplored over the years is: how can we assess and quantify the quality of benchmarks?&#10;&#10;In pursuit of this goal, we draw inspiration from successful quality indices in various domains such as power \cite{bollen2000understanding}, water \cite{world1993guidelines}, food \cite{grunert2005food} and air \cite{jones1999indoor}.&#10;&#10;This paper focuses on the initial steps to develop a metric quantifying data quality for NLP benchmarks. In our study, higher data quality implies lower bias and higher generalization capability. To construct such a metric, we first identify a set of seven language properties to represent various possible sample interactions (and therefore biases) in a benchmark. For each property we identify potential bias parameters that can be integrated into the metric. This is done by surveying existing literature, to understand how various parameters capture bias properties, including origins, types and impact on performance, generalization, and robustness. We also develop several new parameters in this process. To ensure that identified parameters are generic and not overfitted, we cover several datasets and a hierarchy of tasks: NLI, Argumentation, QA, Reading Comprehension (RC), and Abstractive Summarization. This order reflects the presence of increasing amounts of data per sample across tasks. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, NLP Benchmarking, Linguistics, Bias Detection, Metric Development, Artificial Intelligence, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2310.04027" label="2310.04027">
        <attvalues>
          <attvalue for="0" value="Enhancing Financial Sentiment Analysis via Retrieval Augmented Large&#10;  Language Models" />
          <attvalue for="1" value="  Financial sentiment analysis is critical for valuation and investment&#10;decision-making. Traditional NLP models, however, are limited by their&#10;parameter size and the scope of their training datasets, which hampers their&#10;generalization capabilities and effectiveness in this field. Recently, Large&#10;Language Models (LLMs) pre-trained on extensive corpora have demonstrated&#10;superior performance across various NLP tasks due to their commendable&#10;zero-shot abilities. Yet, directly applying LLMs to financial sentiment&#10;analysis presents challenges: The discrepancy between the pre-training&#10;objective of LLMs and predicting the sentiment label can compromise their&#10;predictive performance. Furthermore, the succinct nature of financial news,&#10;often devoid of sufficient context, can significantly diminish the reliability&#10;of LLMs' sentiment analysis. To address these challenges, we introduce a&#10;retrieval-augmented LLMs framework for financial sentiment analysis. This&#10;framework includes an instruction-tuned LLMs module, which ensures LLMs behave&#10;as predictors of sentiment labels, and a retrieval-augmentation module which&#10;retrieves additional context from reliable external sources. Benchmarked&#10;against traditional models and LLMs like ChatGPT and LLaMA, our approach&#10;achieves 15\% to 48\% performance gain in accuracy and F1 score.&#10;" />
          <attvalue for="2" value="&#10;&#10;Financial sentiment analysis is a critical tool that extracts, quantifies, and studies the affective states and subjective information within financial documents, news articles, and social media content \cite{fsadefinition}. Its significance lies in its potential to forecast market movements and provide valuable insights into investors' behaviors. Given that market reactions are often influenced by news sentiments, which can be positive, negative, or neutral, financial sentiment analysis plays a pivotal role in aiding traders and financial institutions in making informed decisions. It helps manage risks and identify potential investment opportunities by providing a nuanced understanding of the market's emotional undercurrents.&#10;&#10;In recent years, numerous studies have turned to Natural Language Processing (NLP) models to enhance the accuracy and efficiency of financial sentiment analysis \cite{araci2019finbert,yang2020finbert,sohangir2018big,day2016deep,wu2023bloomberggpt,yang2023fingpt}. Traditional NLP models, constrained by the limitations of their model parameters and the scale of their training corpora, often lack the capability to comprehensively understand intricate financial news, thereby limiting the efficacy of financial sentiment analysis \cite{araci2019finbert,yang2020finbert,sohangir2018big,day2016deep}. These limitations have sometimes resulted in suboptimal outcomes in financial sentiment analysis tasks. In contrast, the advent of large language models (LLMs) \cite{ouyang2022training,touvron2023llama,wu2023bloomberggpt,yang2023fingpt} has ushered in a new era in the NLP domain. These LLMs, having been pre-trained on vast and diverse corpora, boast formidable zero-shot learning abilities. As a result, they are gradually outperforming many other models across various NLP tasks, owing to their ability to generalize from their extensive training and derive meaningful insights even from previously unseen financial data.&#10;&#10;However, directly applying LLMs for financial sentiment analysis poses two notable challenges. Firstly, the discrepancy between the objective function used in LLMs' pre-training and the goal of predicting financial sentiment may result in LLMs' inability to consistently output labels for financial sentiment analysis as expected \cite{thoppilan2022lamda,ouyang2022training}. Secondly, the typical subjects of financial sentiment analysis, such as news flashes and tweets, are characteristically concise and often lack adequate background information. The scarcity of information has not only interfered with the judgment of human experts \cite{malo2014good} but also poses a significant challenge to the accurate prediction of large language models.&#10;&#10;To address the aforementioned challenges, in our study, we present a retrieval-augmented large language model framework for financial sentiment analysis. This framework consists of two key components. 1) instruction-finetuned LLMs \cite{ouyang2022training}, which refines LLMs using a limited set of instruction-following examples crafted specifically for financial sentiment analysis, aligning LLMs' predictions with user intentions and significantly boosting their prediction accuracy. 2) retrieval-augmented component \cite{lewis2020retrieval}, which introduces additional context to brief statements from news flashes or tweets. It employs search engines and verified financial sources to gather relevant background information from external sources. This enriched context is then passed to the instruction-tuned LLMs for prediction, resulting in more accurate and nuanced results.&#10;&#10;Through extensive evaluations on multiple financial sentiment analysis benchmarks, we demonstrate that compared to traditional smaller-scale sentiment analysis models \cite{araci2019finbert} and general-purpose LLMs, such as ChatGPT \cite{ouyang2022training} and LLaMA \cite{touvron2023llama}, our approach markedly outperforms them.&#10;The primary contributions of this paper can be summarized as follows:&#10;&#10;\begin{itemize}&#10;\item We introduce a novel retrieval-augmented large language model framework tailored for financial sentiment analysis. By integrating external knowledge retrieval, we optimize the depth and context of the information feeding into the LLMs, ensuring more nuanced and informed predictions.&#10;\item Our method of instruction tuning leverages a unique set of instruction-following examples. This fine-tuning process realigns LLMs to respond more accurately to user-intended financial sentiment analysis tasks, markedly enhancing their predictive accuracy.&#10;\item Through extensive evaluations on established benchmarks, we demonstrate that our approach outperforms traditional sentiment analysis models and notable general-purpose LLMs, achieving a 15\% to 48\% performance gain in accuracy and F1 score.&#10;\end{itemize}&#10;&#10;The remainder of this paper is organized as follows. Section 2 briefly reviews the backgroun and related work. In Section 3, we describe the retrieval augmented method that consists two modules. In Section 4, we present the performance evaluation from three aspects. Section 5 concludes this work and points out directions for future work.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Finance, Computer Science, Large Language Models, Linguistics, Economics, Natural Language Processing, Financial Sentiment Analysis" />
        </attvalues>
      </node>
      <node id="2306.06031" label="2306.06031">
        <attvalues>
          <attvalue for="0" value="FinGPT: Open-Source Financial Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have shown the potential of revolutionizing&#10;natural language processing tasks in diverse domains, sparking great interest&#10;in finance. Accessing high-quality financial data is the first challenge for&#10;financial LLMs (FinLLMs). While proprietary models like BloombergGPT have taken&#10;advantage of their unique data accumulation, such privileged access calls for&#10;an open-source alternative to democratize Internet-scale financial data.&#10;  In this paper, we present an open-source large language model, FinGPT, for&#10;the finance sector. Unlike proprietary models, FinGPT takes a data-centric&#10;approach, providing researchers and practitioners with accessible and&#10;transparent resources to develop their FinLLMs. We highlight the importance of&#10;an automatic data curation pipeline and the lightweight low-rank adaptation&#10;technique in building FinGPT. Furthermore, we showcase several potential&#10;applications as stepping stones for users, such as robo-advising, algorithmic&#10;trading, and low-code development. Through collaborative efforts within the&#10;open-source AI4Finance community, FinGPT aims to stimulate innovation,&#10;democratize FinLLMs, and unlock new opportunities in open finance. Two&#10;associated code repos are \url{https://github.com/AI4Finance-Foundation/FinGPT}&#10;and \url{https://github.com/AI4Finance-Foundation/FinNLP}&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1307.5336" label="1307.5336">
        <attvalues>
          <attvalue for="0" value="Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts" />
          <attvalue for="1" value="  The use of robo-readers to analyze news texts is an emerging technology trend&#10;in computational finance. In recent research, a substantial effort has been&#10;invested to develop sophisticated financial polarity-lexicons that can be used&#10;to investigate how financial sentiments relate to future company performance.&#10;However, based on experience from other fields, where sentiment analysis is&#10;commonly applied, it is well-known that the overall semantic orientation of a&#10;sentence may differ from the prior polarity of individual words. The objective&#10;of this article is to investigate how semantic orientations can be better&#10;detected in financial and economic news by accommodating the overall&#10;phrase-structure information and domain-specific use of language. Our three&#10;main contributions are: (1) establishment of a human-annotated finance&#10;phrase-bank, which can be used as benchmark for training and evaluating&#10;alternative models; (2) presentation of a technique to enhance financial&#10;lexicons with attributes that help to identify expected direction of events&#10;that affect overall sentiment; (3) development of a linearized phrase-structure&#10;model for detecting contextual semantic orientations in financial and economic&#10;news texts. The relevance of the newly added lexicon features and the benefit&#10;of using the proposed learning-algorithm are demonstrated in a comparative&#10;study against previously used general sentiment models as well as the popular&#10;word frequency models used in recent financial studies. The proposed framework&#10;is parsimonious and avoids the explosion in feature-space caused by the use of&#10;conventional n-gram features.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.07929" label="2310.07929">
        <attvalues>
          <attvalue for="0" value="Crosslingual Structural Priming and the Pre-Training Dynamics of&#10;  Bilingual Language Models" />
          <attvalue for="1" value="  Do multilingual language models share abstract grammatical representations&#10;across languages, and if so, when do these develop? Following Sinclair et al.&#10;(2022), we use structural priming to test for abstract grammatical&#10;representations with causal effects on model outputs. We extend the approach to&#10;a Dutch-English bilingual setting, and we evaluate a Dutch-English language&#10;model during pre-training. We find that crosslingual structural priming effects&#10;emerge early after exposure to the second language, with less than 1M tokens of&#10;data in that language. We discuss implications for data contamination,&#10;low-resource transfer, and how abstract grammatical representations emerge in&#10;multilingual models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Multilingual language models share representations across languages \cite{artetxe-etal-2020-cross, conneau-etal-2020-emerging}, which is thought to enable their crosslingual transfer abilities \cite{wu-dredze-2019-beto,Chi_Dong_Wei_Wang_Mao_Huang_2020,pmlr-v119-hu20b,winata-etal-2021-language,winata-etal-2022-cross}.&#10;In this study, we use a paradigm known as crosslingual structural priming to study shared representations of grammatical structure in a Dutch-English bilingual language model.&#10;Specifically, does crosslingual structural priming occur, and how much pre-training data does it require?&#10;&#10;Structural priming is a phenomenon in which after being presented with a sentence with a given grammatical structure, people (and language models; \cite{sinclair_2022_StructuralPersistenceLanguage}) are more likely to produce a sentence with the same structure \cite{bock_1986_SyntacticPersistenceLanguage,prasad-etal-2019-using,frank_2021_CrosslanguageStructuralPriming,li-etal-2022-neural,choi2022syntactic2}. For example, a language model would assign a higher probability to a prepositional object (PO) dative sentence (e.g. ``the chef gives a hat to the swimmer'') following another PO sentence than it would following a double object (DO) dative sentence (e.g. ``the chef gives the swimmer a hat''; sentences from \cite{schoonbaert_2007_RepresentationLexicalSyntactic}).&#10;Because the grammatical structure is primed rather than a specific semantic meaning, \cite{sinclair_2022_StructuralPersistenceLanguage} argue that structural priming effects provide evidence for abstract grammatical representations in language models.&#10;By measuring output model probabilities given a prime sentence, structural priming demonstrates causal effects of grammatical representations on model outputs without relying on access to internal model states.&#10;The presence of structural priming in crosslingual scenarios (e.g. a structure primes a similar structure in another language) would indicate that these representations are shared between languages.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Crosslingual Transfer Learning, Linguistics, Cognitive Science, Multilingual Language Models, Abstract Grammatical Representations, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2109.07684" label="2109.07684">
        <attvalues>
          <attvalue for="0" value="Language Models are Few-shot Multilingual Learners" />
          <attvalue for="1" value="  General-purpose language models have demonstrated impressive capabilities,&#10;performing on par with state-of-the-art approaches on a range of downstream&#10;natural language processing (NLP) tasks and benchmarks when inferring&#10;instructions from very few examples. Here, we evaluate the multilingual skills&#10;of the GPT and T5 models in conducting multi-class classification on&#10;non-English languages without any parameter updates. We show that, given a few&#10;English examples as context, pre-trained language models can predict not only&#10;English test samples but also non-English ones. Finally, we find the in-context&#10;few-shot cross-lingual prediction results of language models are significantly&#10;better than random prediction, and they are competitive compared to the&#10;existing state-of-the-art cross-lingual models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2209.00797" label="2209.00797">
        <attvalues>
          <attvalue for="0" value="Random Text Perturbations Work, but not Always" />
          <attvalue for="1" value="  We present three large-scale experiments on binary text matching&#10;classification task both in Chinese and English to evaluate the effectiveness&#10;and generalizability of random text perturbations as a data augmentation&#10;approach for NLP. It is found that the augmentation can bring both negative and&#10;positive effects to the test set performance of three neural classification&#10;models, depending on whether the models train on enough original training&#10;examples. This remains true no matter whether five random text editing&#10;operations, used to augment text, are applied together or separately. Our study&#10;demonstrates with strong implication that the effectiveness of random text&#10;perturbations is task specific and not generally positive.&#10;" />
          <attvalue for="2" value="&#10;&#10;Data augmentation (DA) is a common strategy to generate novel label-preserving data to remedy data scarcity and imbalance problems \cite{xie2020unsupervised}, which has been applied with noteworthy success in image and speech recognition \cite{Iwana2021,Park2019,Shorten2019ASO}. In the field of natural language processing (NLP), there have also been a number of studies that use various DA techniques to boost the trained models’ performance \cite{feng-etal-2021-survey,Liu:9240734}, ranging from word replacement \cite{wang-yang-2015-thats,wang-etal-2018-switchout,Zhang2015}, to predictive neural language models \cite{hou2018,kobayashi-2018-contextual,Kurata2016LabeledDG}. However, an evident and critical difference between text and image/speech is that text cannot be treated as purely physical. For any given sequence of words, both the word order and the semantic compatibility among words affect the meaning, and possibly the label of the sequence. This complex nature raises the question as to whether there exists some generally effective DA approach for NLP because automatic strict paraphrasing barely exists \cite{bhagat-hovy-2013-squibs}. &#10;&#10;This study is a preliminary examination of the effectiveness and generalizability of random text perturbations as a DA approach, exemplified by Easy Data Augmentation (EDA), which has been proposed to be a universal DA approach for NLP \cite{wei-zou-2019-eda}. This approach consists of four commonly used token-level editing operations \cite{wei-etal-2021-text,wei-zou-2019-eda}, i.e., Synonym Replacement (SR), Random Insertion (RI), Random Swap (RS), and Random Deletion (RD). SR randomly replaces synonyms for eligible words, while RS randomly swap word pairs. RI inserts random synonyms, if any, instead of random words, whereas RD deletes words at random. Simple as these operations may seem, they have shown general success in various sentiment-related and sentence type classification tasks \cite{wei-zou-2019-eda}.&#10;&#10;To do the examination, we first present a linguistically informed hypothesis and propose a relevant method of evaluation in section~\ref{sec:2}. We then introduce the experimental settings and results in section~\ref{sec:3} and section~\ref{sec:4}, respectively. The paper ends with some discussions and conclusions in section~\ref{sec:5}. &#10;&#10;The major contributions of this study are threefold. First, it reveals the possible inherent limitations of random text perturbations used as a DA approach for NLP with cross-lingual evidence. Second, the paper provides a critical angle and possibly a general way to evaluate the effectiveness and generalizability of a DA approach or technique for NLP. Third, we present an EDA-like Python program that refines EDA's functionalities, contains a novel DA technique, and can be easily employed for text augmentation in other languages. The source code for this program can be found at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Text Data Augmentation, Linguistics, Neural Classification Models, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2007.15951" label="2007.15951">
        <attvalues>
          <attvalue for="0" value="An Empirical Survey of Data Augmentation for Time Series Classification&#10;  with Neural Networks" />
          <attvalue for="1" value="  In recent times, deep artificial neural networks have achieved many successes&#10;in pattern recognition. Part of this success can be attributed to the reliance&#10;on big data to increase generalization. However, in the field of time series&#10;recognition, many datasets are often very small. One method of addressing this&#10;problem is through the use of data augmentation. In this paper, we survey data&#10;augmentation techniques for time series and their application to time series&#10;classification with neural networks. We propose a taxonomy and outline the four&#10;families in time series data augmentation, including transformation-based&#10;methods, pattern mixing, generative models, and decomposition methods.&#10;Furthermore, we empirically evaluate 12 time series data augmentation methods&#10;on 128 time series classification datasets with six different types of neural&#10;networks. Through the results, we are able to analyze the characteristics,&#10;advantages and disadvantages, and recommendations of each data augmentation&#10;method. This survey aims to help in the selection of time series data&#10;augmentation for neural network applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.03075" label="2105.03075">
        <attvalues>
          <attvalue for="0" value="A Survey of Data Augmentation Approaches for NLP" />
          <attvalue for="1" value="  Data augmentation has recently seen increased interest in NLP due to more&#10;work in low-resource domains, new tasks, and the popularity of large-scale&#10;neural networks that require large amounts of training data. Despite this&#10;recent upsurge, this area is still relatively underexplored, perhaps due to the&#10;challenges posed by the discrete nature of language data. In this paper, we&#10;present a comprehensive and unifying survey of data augmentation for NLP by&#10;summarizing the literature in a structured manner. We first introduce and&#10;motivate data augmentation for NLP, and then discuss major methodologically&#10;representative approaches. Next, we highlight techniques that are used for&#10;popular NLP applications and tasks. We conclude by outlining current challenges&#10;and directions for future research. Overall, our paper aims to clarify the&#10;landscape of existing literature in data augmentation for NLP and motivate&#10;additional work in this area. We also present a GitHub repository with a paper&#10;list that will be continuously updated at&#10;https://github.com/styfeng/DataAug4NLP&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.07512" label="1808.07512">
        <attvalues>
          <attvalue for="0" value="SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  In this work, we examine methods for data augmentation for text-based tasks&#10;such as neural machine translation (NMT). We formulate the design of a data&#10;augmentation policy with desirable properties as an optimization problem, and&#10;derive a generic analytic solution. This solution not only subsumes some&#10;existing augmentation schemes, but also leads to an extremely simple data&#10;augmentation strategy for NMT: randomly replacing words in both the source&#10;sentence and the target sentence with other random words from their&#10;corresponding vocabularies. We name this method SwitchOut. Experiments on three&#10;translation datasets of different scales show that SwitchOut yields consistent&#10;improvements of about 0.5 BLEU, achieving better or comparable performances to&#10;strong alternatives such as word dropout (Sennrich et al., 2016a). Code to&#10;implement this method is included in the appendix.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.05469" label="2101.05469">
        <attvalues>
          <attvalue for="0" value="Text Augmentation in a Multi-Task View" />
          <attvalue for="1" value="  Traditional data augmentation aims to increase the coverage of the input&#10;distribution by generating augmented examples that strongly resemble original&#10;samples in an online fashion where augmented examples dominate training. In&#10;this paper, we propose an alternative perspective -- a multi-task view (MTV) of&#10;data augmentation -- in which the primary task trains on original examples and&#10;the auxiliary task trains on augmented examples. In MTV data augmentation, both&#10;original and augmented samples are weighted substantively during training,&#10;relaxing the constraint that augmented examples must resemble original data and&#10;thereby allowing us to apply stronger levels of augmentation. In empirical&#10;experiments using four common data augmentation techniques on three benchmark&#10;text classification datasets, we find that the MTV leads to higher and more&#10;robust performance improvements than traditional augmentation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.07914" label="2003.07914">
        <attvalues>
          <attvalue for="0" value="Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code" />
          <attvalue for="1" value="  Statistical language modeling techniques have successfully been applied to&#10;large source code corpora, yielding a variety of new software development&#10;tools, such as tools for code suggestion, improving readability, and API&#10;migration. A major issue with these techniques is that code introduces new&#10;vocabulary at a far higher rate than natural language, as new identifier names&#10;proliferate. Both large vocabularies and out-of-vocabulary issues severely&#10;affect Neural Language Models (NLMs) of source code, degrading their&#10;performance and rendering them unable to scale.&#10;  In this paper, we address this issue by: 1) studying how various modelling&#10;choices impact the resulting vocabulary on a large-scale corpus of 13,362&#10;projects; 2) presenting an open vocabulary source code NLM that can scale to&#10;such a corpus, 100 times larger than in previous work; and 3) showing that such&#10;models outperform the state of the art on three distinct code corpora (Java, C,&#10;Python). To our knowledge, these are the largest NLMs for code that have been&#10;reported.&#10;  All datasets, code, and trained models used in this work are publicly&#10;available.&#10;" />
          <attvalue for="2" value="&#10;&#10;Many works have taken advantage of the ``naturalness'' of software \cite{Hindle2012} to assist software engineering tasks, including code completion \cite{Raychev2014}, improving&#10;code readability \cite{Allamanis2014}, program repair \cite{santos2018syntax,chen2018sequencer}, identifying buggy code \cite{Ray2016} and API migration \cite{gu2017deepam}, among many others \cite{big-code-survey}. These approaches analyze large amounts of source code, ranging from hundreds to thousands of software projects, building machine learning models of source code properties, inspired by techniques from natural language processing (NLP).&#10;&#10;When applying any NLP method to create any type of&#10;software development tool, a crucial early decision is how to model software's vocabulary. This is all the more important because, unlike in natural language, software developers are free to create any identifiers they like, and can make them arbitrarily complex. Because of this&#10;fundamental fact, any model that is trained on a large-scale software corpus has to deal with an extremely large and sparse vocabulary (Section \ref{sec:background}). Rare words can not be modelled effectively. Furthermore, if identifiers were not observed in the training set, &#10;many classes of models cannot predict them, which is known as the out-of-vocabulary (OOV) problem. Hellendoorn and Devanbu observe this issue &#10;for the task of language modeling, showing that a neural language model (NLM) has difficulties scaling beyond as few as a hundred projects \cite{Hellendoorn2017}.&#10;Given that neural approaches are the state-of-the-art in NLP,&#10;finding ways to scale them to a larger software corpus is a very important goal. &#10;&#10;Our first contribution is a thorough study of&#10;the effects of the vocabulary design choices&#10;that must be made when creating any NLP model of software (Section \ref{sec:choices}).&#10;The vocabulary design choices we study include how&#10;to handle comments, string literals, and white space; &#10;whether to filter out infrequent tokens; and whether and how&#10;to split compound tokens, such as names that contain camel&#10;case and underscores.&#10;We examine how these choices affect&#10;the vocabulary size, which affects the scalability&#10;of models, and how they affect the OOV rate, that is, how often&#10;the vocabulary fails to include names that appear&#10;in new projects.&#10;We find that the choices have a large impact, leading to variations in vocabulary size of up to three orders of magnitude. However, we find that the most common ways to reduce vocabulary that were previously considered in the software engineering literature, such as splitting identifiers according to underscores and case, are not enough to obtain a vocabulary of a manageable size; advanced approaches such as adaptations of the Byte-Pair Encoding (BPE) algorithm \cite{Gage1994,Sennrich2015} are needed to reach this goal and deal with the OOV problem.&#10;&#10;This empirical study motivates our second contribution. Drawing on our results, we develop a large-scale open-vocabulary NLM for source code (Section \ref{sec:rnnlm}).&#10;To our knowledge, this is the first BPE NLM for source code reported in the literature.&#10;This NLM model leverages BPE, beam search, and caching to both keep vocabulary size low and successfully predict OOV tokens. We show that this NLM is able to scale: we train it on up to \numprojtrain software projects, yielding the largest NLM trained on source code we are aware of. &#10;&#10;Finally, in our third contribution we extensively evaluate our NLM (Sections \ref{sec:evaluation}--\ref{sec:results}). We show that the open-vocabulary NLM&#10;outperforms both&#10;$n$-gram LMs and closed vocabulary NLMs for the task of code completion for several languages (Java, C, and Python).&#10;To show that improvement in language modelling transfers&#10;to downstream SE tasks, we conduct an experiment similar to &#10;\cite{Ray2016}, who showed that language models&#10;can be used to highlight buggy code. Indeed, we find that&#10;our open-vocabulary NLM is more effective than previous LMs at highlighting buggy code.&#10;&#10;More broadly, these contributions may impact&#10;future development software tools.&#10;First, source code LMs have been used in&#10;a diverse variety of tools well beyond the obvious&#10;application of autocompletion, ranging from code &#10;readability \cite{Allamanis2014} to program repair&#10;\cite{chen2018sequencer}.&#10;Our improved NLM could lead to improvements to all of these tools.&#10;Second, recent results in NLP \cite{howard2018universal, Peters2018, Devlin2018} show that NLMs can be used as upstream tasks in transfer learning, leading to state-of-the-art improvement in downstream tasks: for instance, a model can be pre-trained as an NLM, and later on fine-tuned as a classifier.&#10;Improved NLM architectures could lead to improved downstream classifiers, especially if the labelled data is scarce. While transfer learning from language models has been applied in software engineering \cite{robbes2019leveraging}, it has not been applied to source code due to the aforementioned vocabulary issues.&#10;Finally, the general insights about vocabulary design that we study are not specific to NLMs, but arise whenever we build development tools by applying NLP methods to source code.&#10;&#10;We conclude the paper in Section \ref{sec:conclusion}, and briefly describe the artifacts used in this work and how to obtain them in Section \ref{sec:artifacts}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Modeling Techniques, Artificial Intelligence, Mathematics, Software Development Tools, Neural Network Scalability" />
        </attvalues>
      </node>
      <node id="1402.4182" label="1402.4182">
        <attvalues>
          <attvalue for="0" value="Learning Natural Coding Conventions" />
          <attvalue for="1" value="  Every programmer has a characteristic style, ranging from preferences about&#10;identifier naming to preferences about object relationships and design&#10;patterns. Coding conventions define a consistent syntactic style, fostering&#10;readability and hence maintainability. When collaborating, programmers strive&#10;to obey a project's coding conventions. However, one third of reviews of&#10;changes contain feedback about coding conventions, indicating that programmers&#10;do not always follow them and that project members care deeply about adherence.&#10;Unfortunately, programmers are often unaware of coding conventions because&#10;inferring them requires a global view, one that aggregates the many local&#10;decisions programmers make and identifies emergent consensus on style. We&#10;present NATURALIZE, a framework that learns the style of a codebase, and&#10;suggests revisions to improve stylistic consistency. NATURALIZE builds on&#10;recent work in applying statistical natural language processing to source code.&#10;We apply NATURALIZE to suggest natural identifier names and formatting&#10;conventions. We present four tools focused on ensuring natural code during&#10;development and release management, including code review. NATURALIZE achieves&#10;94% accuracy in its top suggestions for identifier names and can even transfer&#10;knowledge about conventions across projects, leveraging a corpus of 10,968 open&#10;source projects. We used NATURALIZE to generate 18 patches for 5 open source&#10;projects: 14 were accepted.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.01808" label="1901.01808">
        <attvalues>
          <attvalue for="0" value="SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair" />
          <attvalue for="1" value="  This paper presents a novel end-to-end approach to program repair based on&#10;sequence-to-sequence learning. We devise, implement, and evaluate a system,&#10;called SequenceR, for fixing bugs based on sequence-to-sequence learning on&#10;source code. This approach uses the copy mechanism to overcome the unlimited&#10;vocabulary problem that occurs with big code. Our system is data-driven; we&#10;train it on 35,578 samples, carefully curated from commits to open-source&#10;repositories. We evaluate it on 4,711 independent real bug fixes, as well on&#10;the Defects4J benchmark used in program repair research. SequenceR is able to&#10;perfectly predict the fixed line for 950/4711 testing samples, and find correct&#10;patches for 14 bugs in Defects4J. It captures a wide range of repair operators&#10;without any domain-specific top-down design.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.01159" label="1506.01159">
        <attvalues>
          <attvalue for="0" value="On the &quot;Naturalness&quot; of Buggy Code" />
          <attvalue for="1" value="  Real software, the kind working programmers produce by the kLOC to solve&#10;real-world problems, tends to be &quot;natural&quot;, like speech or natural language; it&#10;tends to be highly repetitive and predictable. Researchers have captured this&#10;naturalness of software through statistical models and used them to good effect&#10;in suggestion engines, porting tools, coding standards checkers, and idiom&#10;miners. This suggests that code that appears improbable, or surprising, to a&#10;good statistical language model is &quot;unnatural&quot; in some sense, and thus possibly&#10;suspicious. In this paper, we investigate this hypothesis. We consider a large&#10;corpus of bug fix commits (ca.~8,296), from 10 different Java projects, and we&#10;focus on its language statistics, evaluating the naturalness of buggy code and&#10;the corresponding fixes. We find that code with bugs tends to be more entropic&#10;(i.e., unnatural), becoming less so as bugs are fixed. Focusing on highly&#10;entropic lines is similar in cost-effectiveness to some well-known static bug&#10;finders (PMD, FindBugs) and ordering warnings from these bug finders using an&#10;entropy measure improves the cost-effectiveness of inspecting code implicated&#10;in warnings. This suggests that entropy may be a valid language-independent and&#10;simple way to complement the effectiveness of PMD or FindBugs, and that&#10;search-based bug-fixing methods may benefit from using entropy both for&#10;fault-localization and searching for fixes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.06182" label="1709.06182">
        <attvalues>
          <attvalue for="0" value="A Survey of Machine Learning for Big Code and Naturalness" />
          <attvalue for="1" value="  Research at the intersection of machine learning, programming languages, and&#10;software engineering has recently taken important steps in proposing learnable&#10;probabilistic models of source code that exploit code's abundance of patterns.&#10;In this article, we survey this work. We contrast programming languages against&#10;natural languages and discuss how these similarities and differences drive the&#10;design of probabilistic models. We present a taxonomy based on the underlying&#10;design principles of each model and use it to navigate the literature. Then, we&#10;review how researchers have adapted these models to application areas and&#10;discuss cross-cutting and application-specific challenges and opportunities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.06385" label="2308.06385">
        <attvalues>
          <attvalue for="0" value="ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIF" />
          <attvalue for="1" value="  In this work, we address the problem of directing the text generation of a&#10;language model (LM) towards a desired behavior, aligning the generated text&#10;with the preferences of the human operator. We propose using another,&#10;instruction-tuned language model as a critic reward model in a zero-shot way&#10;thanks to the prompt of a Yes-No question that represents the user preferences,&#10;without requiring further labeled data. This zero-shot reward model provides&#10;the learning signal to further fine-tune the base LM using Reinforcement&#10;Learning from AI Feedback (RLAIF); yet our approach is also compatible in other&#10;contexts such as quality-diversity search. Extensive evidence of the&#10;capabilities of the proposed ZYN framework is provided through experiments in&#10;different domains related to text generation, including detoxification;&#10;optimizing sentiment of movie reviews, or any other attribute; steering the&#10;opinion about a particular topic the model may have; and personalizing prompt&#10;generators for text-to-image tasks. Code available at&#10;\url{https://github.com/vicgalle/zero-shot-reward-models/}.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs), trained on extensive text datasets, have demonstrated remarkable emergent capabilities in zero or few-shot learning within the natural language processing (NLP) domain \cite{Radford2019LanguageMA, brown2020language, openai2023gpt4}. However, these models often exhibit undesirable behaviors such as fabricating information, producing biased or harmful content, or failing to adhere to user instructions \cite{parrots, gehman2020realtoxicityprompts, weidinger2021ethical, kenton2021alignment, bommasani2021opportunities, tamkin2021understanding}. Yet the customization of these models to accommodate specific user preferences poses a significant challenge. Therefore, the mitigation of undesired behaviors and the enhancement of model adaptability to user preferences are highly sought-after attributes in language models. &#10;&#10;Reinforcement Learning from Human Feedback (RLHF) has recently demonstrated impressive outcomes in aligning large, pretrained language models with human preferences. By optimizing for key qualities such as harmlessness and helpfulness, this technique, as underscored by \cite{bai2022training}, has gained significant ground. Furthermore, this approach has proven its efficacy by attaining unprecedented results across a wide array of natural language tasks \cite{openai2023gpt4}. &#10;&#10;The conventional RLHF pipeline refines an initial, non-aligned LLM by employing an online RL algorithm like the popular Proximal Policy Optimization (PPO) \cite{schulman2017proximal}. The goal is to optimize the LLM to align accurately with human preferences. A key dependency of RLHF is the reward model, which is trained to predict the best alternative from a pair of two model outputs, $(o_1, o_2)$, generated from the same prompt $p$. However, the large-scale collection of human-ranked preference data, particularly of high quality, can be expensive and time-consuming. In order to mitigate this challenge, several alternatives such as Reinforcement Learning from AI Feedback (RLAIF) have been suggested, where human annotation is not a requisite for the availability of labels. RLAIF methods \cite{bai2022constitutional} aim to mimic human binary preferences by assigning scores to outputs $o_1$ and $o_2$ utilizing a LLM. It's worth noting that the LLM used for scoring frequently matches the one that initially generates the given outputs $(o_1, o_2)$. Unsurprisingly, these LLM-generated binary choices tend to be somewhat less accurate compared to actual human labels, yet they can be helpful in some contexts. Furthermore, since the same prompt $p$ is used to generate both outputs, these are of comparable quality, so having the LLM rank the two can be challenging.&#10;&#10;In this work, we depart from preference models that act on pairwise preference data (choose the best between two alternatives) to design zero-shot reward models that score each generation independently in a zero-shot fashion, just by prompting the model with a yes-no question, or an ensemble of questions. The resulting framework, which we call ZYN, is a way of creating reward models without need for annotated data, leveraging an instruction-tuned model that will act as the critic that will guide the student language model. As such, the rewards that ZYN computes can be straightforwardly integrated into any RL-finetuning pipeline, such as PPO-based RLAIF. But ZYN can also be used within other contexts too, such as best-of-N sampling and quality-diversity search.&#10;&#10;The structure of this paper is as follows. Section \ref{sec:related} gives a brief overview of related literature, focusing on RL and instruction fine-tuning of language models; Section \ref{sec:method} showcases our novel approach to design and use zero-shot reward models; Section \ref{sec:experiments} presents a wide battery of different experiments and results, confirming the general usefulness and applicability of our framework; and Section \ref{sec:conclusions} sums up with conclusions, limitations, and several avenues for further research.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Control, Text Generation, Artificial Intelligence, Natural Language Processing, Reinforcement Learning" />
        </attvalues>
      </node>
      <node id="1606.09403" label="1606.09403">
        <attvalues>
          <attvalue for="0" value="Learning Crosslingual Word Embeddings without Bilingual Corpora" />
          <attvalue for="1" value="  Crosslingual word embeddings represent lexical items from different languages&#10;in the same vector space, enabling transfer of NLP tools. However, previous&#10;attempts had expensive resource requirements, difficulty incorporating&#10;monolingual data or were unable to handle polysemy. We address these drawbacks&#10;in our method which takes advantage of a high coverage dictionary in an EM&#10;style training algorithm over monolingual corpora in two languages. Our model&#10;achieves state-of-the-art performance on bilingual lexicon induction task&#10;exceeding models using large bilingual corpora, and competitive results on the&#10;monolingual word similarity and cross-lingual document classification task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.08469" label="2011.08469">
        <attvalues>
          <attvalue for="0" value="Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin&#10;  Speech Recognition with a Syllable-to-Character Converter" />
          <attvalue for="1" value="  End-to-end models are favored in automatic speech recognition (ASR) because&#10;of its simplified system structure and superior performance. Among these&#10;models, recurrent neural network transducer (RNN-T) has achieved significant&#10;progress in streaming on-device speech recognition because of its high-accuracy&#10;and low-latency. RNN-T adopts a prediction network to enhance language&#10;information, but its language modeling ability is limited because it still&#10;needs paired speech-text data to train. Further strengthening the language&#10;modeling ability through extra text data, such as shallow fusion with an&#10;external language model, only brings a small performance gain. In view of the&#10;fact that Mandarin Chinese is a character-based language and each character is&#10;pronounced as a tonal syllable, this paper proposes a novel cascade RNN-T&#10;approach to improve the language modeling ability of RNN-T. Our approach&#10;firstly uses an RNN-T to transform acoustic feature into syllable sequence, and&#10;then converts the syllable sequence into character sequence through an&#10;RNN-T-based syllable-to-character converter. Thus a rich text repository can be&#10;easily used to strengthen the language model ability. By introducing several&#10;important tricks, the cascade RNN-T approach surpasses the character-based&#10;RNN-T by a large margin on several Mandarin test sets, with much higher&#10;recognition quality and similar latency.&#10;" />
          <attvalue for="2" value="&#10;&#10;Conventional automatic speech recognition (ASR) usually adopts a hybrid system of deep neural network - hidden Markov model (DNN-HMM)~\cite{hinton2012deep} which is complex and requires a considerable amount of computing resource, so it is difficult to deploy on edge devices. Recently, end-to-end (E2E) speech recognition has achieved significant progress with simplified system architecture and superior performance. The E2E models usually adopt a sequence-to-sequence (S2S) framework to directly transform acoustic feature sequences into text sequences through specifically-designed neural networks. These models are particularly favored on edge devices for more concise architecture and reduced computing resource consumption over the hybrid ASR systems. However, the E2E speech recognition models, modeling acoustic and language information jointly in a unified framework, usually require a large amount of paired speech-text data for model training. Thus it is difficult for the models themselves to acquire strong language modeling ability by using available text data with more orders of magnitude than the speech-text paired data, especially when the training set does not match the language domain of specific applications. This paper addresses this problem by introducing a novel approach to improve the language modeling ability of E2E models.&#10;&#10;As an S2S model, recurrent neural network transducer (RNN-T)~\cite{graves2012sequence} and its variants have achieved high-accuracy and low-latency in streaming on-device speech recognition~\cite{rnntshibie,graves2013speech}. Neural transducer has the streaming decoding ability in nature, while other E2E competitors, particularly those based on attention mechanism, such as transformer~\cite{transformer,povey2018time,dong2018speech} and listen, attend and spell~\cite{las} (LAS), have to be modified to possess the streaming ability. RNN-T uses a prediction network to enhance the language information based on the connectionist temporal classification (CTC)~\cite{graves2014towards} criterion. But its language modeling ability is not satisfactory because it still needs paired speech-text data to train. A recent study has unveiled that the language modeling ability of the prediction network is still quite weak~\cite{ghodsi2020rnn}.&#10;&#10;Plenty of effort has been made on improving the performance of E2E models by introducing additional language information. A common solution is to use a language model (LM) fusion strategy: an LM is first externally trained on text data and then incorporated into the E2E model~\cite{shallowfusion,zhao2019shallow}. Shallow fusion simply interpolates the label probabilities with the ones from an external LM during decoding stage. Other fusion variants, such as deep fusion, cold fusion and component fusion, have also been proposed. Data augmentation through speech synthesis is another solution. Work in ~\cite{ttsrnnt,ttsrnnt2} has shown that data augmentation with text-to-speech utterances yields improvement to E2E models; however, there still remains a substantial gap in performance between models trained on human speech and those trained on synthesized speech. Similar to the tricks used in conventional hybrid approaches, two-pass decoding also can be introduced to E2E models with improved recognition performance. Recently, a two-pass RNN-T+LAS model, where LAS rescores hypotheses from RNN-T, has been proposed~\cite{rnntlas} and improved further with more tricks~\cite{rnntlas2}. To surpass server-side conventional model, trade-off between quality and latency has been particularly considered.&#10;&#10;Most approaches on neural transducer have been conducted on English corpora and different modeling units, such as phonemes, grapheme and word-piece, have been explored~\cite{rao2017exploring}. In this paper, we are particularly interested in streaming on-device Mandarin ASR using RNN-T. Mandarin Chinese is significantly different from English in both written and spoken aspects. Chinese is a character-based language and each character is pronounced as a tonal syllable. There are several studies on LAS and Transformer based Mandarin ASR, but we only find one paper on the use of RNN-T in Mandarin which shows its feasibility on modeling Chinese characters~\cite{sengmaornnt}. Character has been widely chosen as a natural modeling unit in Transformer-based Mandarin ASR as well. However, as Chinese has a huge set of characters, all these works have chosen a partial set of frequently-used characters to model while the rest are simply abandoned, which means the abandoned characters can never be outputted, leading to out-of-vocabulary (OOV) problem.&#10;&#10;In this paper, we propose a novel cascade RNN-T approach for streaming on-device Mandarin speech recognition. Specifically, we cascade two RNN transducers to strengthen the language modeling ability -- the first transforms acoustic input into a syllable sequence, while the second converts the syllable sequence into the final character sequence. The proposed approach has the following advantages: 1) a rich text repository can be easily used to strengthen the language modeling ability; 2) the OOV issue does not exist by the introduction of RNN-T based syllable-to-character (S2C) converter; 3) streaming ability has been maintained as the use of the transducer framework. By introducing several important tricks on the proposed syllable-based cascade RNN-T, including adding convolution layer, self shallow fusion, text augmentation and syllable correction, we manage to surpass the character-based RNN-T by a large margin. Compared with character RNN-T with shallow fusion, cascade RNN-T has an obvious improvement on several Mandarin test sets, with higher recognition quality and similar latency.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Electrical Engineering, Linguistics, Signal Processing, Speech Recognition, Language Modeling" />
        </attvalues>
      </node>
      <node id="2005.07157" label="2005.07157">
        <attvalues>
          <attvalue for="0" value="You Do Not Need More Data: Improving End-To-End Speech Recognition by&#10;  Text-To-Speech Data Augmentation" />
          <attvalue for="1" value="  Data augmentation is one of the most effective ways to make end-to-end&#10;automatic speech recognition (ASR) perform close to the conventional hybrid&#10;approach, especially when dealing with low-resource tasks. Using recent&#10;advances in speech synthesis (text-to-speech, or TTS), we build our TTS system&#10;on an ASR training database and then extend the data with synthesized speech to&#10;train a recognition model. We argue that, when the training data amount is&#10;relatively low, this approach can allow an end-to-end model to reach hybrid&#10;systems' quality. For an artificial low-to-medium-resource setup, we compare&#10;the proposed augmentation with the semi-supervised learning technique. We also&#10;investigate the influence of vocoder usage on final ASR performance by&#10;comparing Griffin-Lim algorithm with our modified LPCNet. When applied with an&#10;external language model, our approach outperforms a semi-supervised setup for&#10;LibriSpeech test-clean and only 33% worse than a comparable supervised setup.&#10;Our system establishes a competitive result for end-to-end ASR trained on&#10;LibriSpeech train-clean-100 set with WER 4.3% for test-clean and 13.5% for&#10;test-other.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.11699" label="1909.11699">
        <attvalues>
          <attvalue for="0" value="Speech Recognition with Augmented Synthesized Speech" />
          <attvalue for="1" value="  Recent success of the Tacotron speech synthesis architecture and its variants&#10;in producing natural sounding multi-speaker synthesized speech has raised the&#10;exciting possibility of replacing expensive, manually transcribed,&#10;domain-specific, human speech that is used to train speech recognizers. The&#10;multi-speaker speech synthesis architecture can learn latent embedding spaces&#10;of prosody, speaker and style variations derived from input acoustic&#10;representations thereby allowing for manipulation of the synthesized speech. In&#10;this paper, we evaluate the feasibility of enhancing speech recognition&#10;performance using speech synthesis using two corpora from different domains. We&#10;explore algorithms to provide the necessary acoustic and lexical diversity&#10;needed for robust speech recognition. Finally, we demonstrate the feasibility&#10;of this approach as a data augmentation strategy for domain-transfer.&#10;  We find that improvements to speech recognition performance is achievable by&#10;augmenting training data with synthesized material. However, there remains a&#10;substantial gap in performance between recognizers trained on human speech&#10;those trained on synthesized speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.05097" label="1811.05097">
        <attvalues>
          <attvalue for="0" value="Exploring RNN-Transducer for Chinese Speech Recognition" />
          <attvalue for="1" value="  End-to-end approaches have drawn much attention recently for significantly&#10;simplifying the construction of an automatic speech recognition (ASR) system.&#10;RNN transducer (RNN-T) is one of the popular end-to-end methods. Previous&#10;studies have shown that RNN-T is difficult to train and a very complex training&#10;process is needed for a reasonable performance. In this paper, we explore RNN-T&#10;for a Chinese large vocabulary continuous speech recognition (LVCSR) task and&#10;aim to simplify the training process while maintaining performance. First, a&#10;new strategy of learning rate decay is proposed to accelerate the model&#10;convergence. Second, we find that adding convolutional layers at the beginning&#10;of the network and using ordered data can discard the pre-training process of&#10;the encoder without loss of performance. Besides, we design experiments to find&#10;a balance among the usage of GPU memory, training circle and model performance.&#10;Finally, we achieve 16.9% character error rate (CER) on our test set which is&#10;2% absolute improvement from a strong BLSTM CE system with language model&#10;trained on the same text corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.09010" label="1909.09010">
        <attvalues>
          <attvalue for="0" value="A Random Gossip BMUF Process for Neural Language Modeling" />
          <attvalue for="1" value="  Neural network language model (NNLM) is an essential component of industrial&#10;ASR systems. One important challenge of training an NNLM is to leverage between&#10;scaling the learning process and handling big data. Conventional approaches&#10;such as block momentum provides a blockwise model update filtering (BMUF)&#10;process and achieves almost linear speedups with no performance degradation for&#10;speech recognition. However, it needs to calculate the model average from all&#10;computing nodes (e.g., GPUs) and when the number of computing nodes is large,&#10;the learning suffers from the severe communication latency. As a consequence,&#10;BMUF is not suitable under restricted network conditions. In this paper, we&#10;present a decentralized BMUF process, in which the model is split into&#10;different components, each of which is updated by communicating to some&#10;randomly chosen neighbor nodes with the same component, followed by a BMUF-like&#10;process. We apply this method to several LSTM language modeling tasks.&#10;Experimental results show that our approach achieves consistently better&#10;performance than conventional BMUF. In particular, we obtain a lower perplexity&#10;than the single-GPU baseline on the wiki-text-103 benchmark using 4 GPUs. In&#10;addition, no performance degradation is observed when scaling to 8 and 16 GPUs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Machine learning, and in particular deep learning technology \cite{DeepLearning} powers many aspects of modern lives. At the core of deep learning lies the deep neural networks (DNNs), long short term memory networks (LSTMs), transformers, convolutional neural networks (CNNs) and their variants. These technologies have been widely implemented in a plenty of fields, such as language modeling \cite{Transformerxl,jozefowicz2016exploring}, natural language processing (NLP) \cite{vaswani2017attention,devlin2018bert} and large vocabulary continuous speech recognition (LVCSR) \cite{dahl2011context,graves2014towards}. As data size and model complexity increase, one essential challenge is to leverage between scaling the learning procedure and handling big data. Usually, to train a neural language model on nowadays language modeling datasets with competitive accuracy requires a high-performance computing cluster. &#10;&#10;Many works have been proposed to scale up the capability of deep learning. For example, DistBelief \cite{dean2012large} utilizes thousands of machines to train various deep machines with an asynchronous SGD (ASGD) procedure called Downpour SGD. Hogwild \cite{recht2011hogwild} employs a lock-free ASGD procedure which is suitable for sparse gradients. Elastic averaging SGD \cite{zhang2015deep} has been proposed recently and is the state-of-art asynchronous parameter-server method. We refer to \cite{ben2018demystifying} as a good survey for introducing these algorithms. &#10;&#10;Another popular approach to introduce parallelism is to average models (MA) directly \cite{Zinkevich2010,Miao2014}. MA updates local models independently on each worker and average them only once \cite{Zinkevich2010} or every a few iterations \cite{Miao2014}. These methods achieve nearly linear speedups but suffer from accuracy degradation \cite{Block-Momentum,povey2014parallel}. BMUF \cite{Block-Momentum}, proposed to tackle the degradation problem in MA, is widely used in speech recognition \cite{li2018improving,zhaoyou2019}. BMUF introduces a blockwise model update filtering process to stabilize the training. BMUF outperforms the traditional model averaging (MA) as well as alternating direction method of multipliers (ADMM) while enjoying the advantage&#10;of low communication costs of such methods. Chen and Huo \cite{Block-Momentum} reported a performance with $28$X speedup with $32$ GPUs while also achieve better accuracy than the single-GPU SGD on large vocabulary speech recognition tasks. The results in \cite{Empirical17} showed that BMUF outperforms EASGD and ASGD on speech recognition tasks. &#10;&#10;All of the above methods utilize a centralized parameter-server structure for communications or use an all-reduce process to average local models. Issues occur if one of these training nodes gets stuck which makes the whole training process hang up. Worse still, when the number of nodes is large, the problem of communication latency should be carefully considered. In this paper, we extend the BMUF process to a decentralized network topology. Each node of this training network only needs to communicate with a small number of its neighbors. During training, the entire model is split into multiple components, and each node randomly selects a few neighbor nodes to communicate. We refer this process to as gossip \cite{jin2016scale}. Then this node aggregates these components from selected neighbors, and then performs a BMUF-like process as we mentioned above. In the experiments, we evaluate the proposed method on two benchmark datasets (e.g., wiki-text-103 and Gutenberg). The results show that our method consistently outperforms conventional approaches using $4$, $8$ and $16$ GPUs, with higher accuracy, lower variance, and comparable speedups.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Distributed Computing Methods, Linguistics, Language Model Training, Artificial Intelligence, Neural Network Optimization, Mathematics" />
        </attvalues>
      </node>
      <node id="1412.6651" label="1412.6651">
        <attvalues>
          <attvalue for="0" value="Deep learning with Elastic Averaging SGD" />
          <attvalue for="1" value="  We study the problem of stochastic optimization for deep learning in the&#10;parallel computing environment under communication constraints. A new algorithm&#10;is proposed in this setting where the communication and coordination of work&#10;among concurrent processes (local workers), is based on an elastic force which&#10;links the parameters they compute with a center variable stored by the&#10;parameter server (master). The algorithm enables the local workers to perform&#10;more exploration, i.e. the algorithm allows the local variables to fluctuate&#10;further from the center variable by reducing the amount of communication&#10;between local workers and the master. We empirically demonstrate that in the&#10;deep learning setting, due to the existence of many local optima, allowing more&#10;exploration can lead to the improved performance. We propose synchronous and&#10;asynchronous variants of the new algorithm. We provide the stability analysis&#10;of the asynchronous variant in the round-robin scheme and compare it with the&#10;more common parallelized method ADMM. We show that the stability of EASGD is&#10;guaranteed when a simple stability condition is satisfied, which is not the&#10;case for ADMM. We additionally propose the momentum-based version of our&#10;algorithm that can be applied in both synchronous and asynchronous settings.&#10;Asynchronous variant of the algorithm is applied to train convolutional neural&#10;networks for image classification on the CIFAR and ImageNet datasets.&#10;Experiments demonstrate that the new algorithm accelerates the training of deep&#10;architectures compared to DOWNPOUR and other common baseline approaches and&#10;furthermore is very communication efficient.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1410.7455" label="1410.7455">
        <attvalues>
          <attvalue for="0" value="Parallel training of DNNs with Natural Gradient and Parameter Averaging" />
          <attvalue for="1" value="  We describe the neural-network training framework used in the Kaldi speech&#10;recognition toolkit, which is geared towards training DNNs with large amounts&#10;of training data using multiple GPU-equipped or multi-core machines. In order&#10;to be as hardware-agnostic as possible, we needed a way to use multiple&#10;machines without generating excessive network traffic. Our method is to average&#10;the neural network parameters periodically (typically every minute or two), and&#10;redistribute the averaged parameters to the machines for further training. Each&#10;machine sees different data. By itself, this method does not work very well.&#10;However, we have another method, an approximate and efficient implementation of&#10;Natural Gradient for Stochastic Gradient Descent (NG-SGD), which seems to allow&#10;our periodic-averaging method to work well, as well as substantially improving&#10;the convergence of SGD on a single machine.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.05698" label="1907.05698">
        <attvalues>
          <attvalue for="0" value="Teach an all-rounder with experts in different domains" />
          <attvalue for="1" value="  In many automatic speech recognition (ASR) tasks, an ideal model has to be&#10;applicable over multiple domains. In this paper, we propose to teach an&#10;all-rounder with experts in different domains. Concretely, we build a&#10;multi-domain acoustic model by applying the teacher-student training framework.&#10;First, for each domain, a teacher model (domain-dependent model) is trained by&#10;fine-tuning a multi-condition model with domain-specific subset. Then all these&#10;teacher models are used to teach one single student model simultaneously. We&#10;perform experiments on two predefined domain setups. One is domains with&#10;different speaking styles, the other is nearfield, far-field and far-field with&#10;noise. Moreover, two types of models are examined: deep feedforward sequential&#10;memory network (DFSMN) and long short term memory (LSTM). Experimental results&#10;show that the model trained with this framework outperforms not only&#10;multi-condition model but also domain-dependent model. Specially, our training&#10;method provides up to 10.4% relative character error rate improvement over&#10;baseline model (multi-condition model).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.05880" label="1703.05880">
        <attvalues>
          <attvalue for="0" value="Empirical Evaluation of Parallel Training Algorithms on Acoustic&#10;  Modeling" />
          <attvalue for="1" value="  Deep learning models (DLMs) are state-of-the-art techniques in speech&#10;recognition. However, training good DLMs can be time consuming especially for&#10;production-size models and corpora. Although several parallel training&#10;algorithms have been proposed to improve training efficiency, there is no clear&#10;guidance on which one to choose for the task in hand due to lack of systematic&#10;and fair comparison among them. In this paper we aim at filling this gap by&#10;comparing four popular parallel training algorithms in speech recognition,&#10;namely asynchronous stochastic gradient descent (ASGD), blockwise model-update&#10;filtering (BMUF), bulk synchronous parallel (BSP) and elastic averaging&#10;stochastic gradient descent (EASGD), on 1000-hour LibriSpeech corpora using&#10;feed-forward deep neural networks (DNNs) and convolutional, long short-term&#10;memory, DNNs (CLDNNs). Based on our experiments, we recommend using BMUF as the&#10;top choice to train acoustic models since it is most stable, scales well with&#10;number of GPUs, can achieve reproducible results, and in many cases even&#10;outperforms single-GPU SGD. ASGD can be used as a substitute in some cases.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.09831" label="2003.09831">
        <attvalues>
          <attvalue for="0" value="Prior Knowledge Driven Label Embedding for Slot Filling in Natural&#10;  Language Understanding" />
          <attvalue for="1" value="  Traditional slot filling in natural language understanding (NLU) predicts a&#10;one-hot vector for each word. This form of label representation lacks semantic&#10;correlation modelling, which leads to severe data sparsity problem, especially&#10;when adapting an NLU model to a new domain. To address this issue, a novel&#10;label embedding based slot filling framework is proposed in this paper. Here,&#10;distributed label embedding is constructed for each slot using prior knowledge.&#10;Three encoding methods are investigated to incorporate different kinds of prior&#10;knowledge about slots: atomic concepts, slot descriptions, and slot exemplars.&#10;The proposed label embeddings tend to share text patterns and reuses data with&#10;different slot labels. This makes it useful for adaptive NLU with limited data.&#10;Also, since label embedding is independent of NLU model, it is compatible with&#10;almost all deep learning based slot filling models. The proposed approaches are&#10;evaluated on three datasets. Experiments on single domain and domain adaptation&#10;tasks show that label embedding achieves significant performance improvement&#10;over traditional one-hot label representation as well as advanced zero-shot&#10;approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Standard approaches to solving the data sparsity problem of slot filling include semi-supervised learning~\cite{Tur2005Combining,celikyilmaz2016new,oyl11-lan-icassp18,zhu2018robust}, domain adaptation~\cite{daume2009frustratingly,jaech2016domain,kim2016frustratingly,Liu2017MultiDomainAL,kim2017domain,jha2018bag,goyal-etal-2018-fast,zhang2018joint,zhao2019data} and zero-shot learning~\cite{ferreira2015zero,yazdani2015model,chen2016zero,Bapna2017towards,lee2018zero,shah2019robust}. The semi-supervised learning and domain adaptation (mostly parameters sharing) methods focus on improving the coverage of feature space while not label space. The zero-shot learning methods involve ontological descriptions to enhance the slot filling. However, Ferreira et al. \cite{ferreira2015zero} and Yazdani et al. \cite{yazdani2015model} made an assumption that all possible values of each slot are known, which is not practical for real-world applications. Our methods seek to interpret semantic labels (slots) in multiple dimensions where relations of slots can be inferred implicitly. It helps data reuse for different labels and domains.&#10;&#10;Label embeddings for SLU have been introduced in several works. Chen et al. \cite{chen2016zero} exploit label embedding for intent detection but not suitable for slot filling. Some works choose to extract label embeddings from data samples \cite{kim-etal-2015-new,ma2016label,lee2018coupled,wu2019joint} by exploiting words of values tagged with a semantic label, but they only focus on the corresponding values. We consider both the value and context information in the slot exemplar encoding. Concept tagger~\cite{Bapna2017towards} and zero-shot adaptive transfer~\cite{lee2018zero} also incorporate slot embeddings from slot descriptions for the slot filling. However, they proposed a slot-independent conditional layer to predict the existence for each slot one by one, which would take more time for training and inference. Moreover, it is not possible for modelling the time series dependence of different slots in those methods. We propose label embeddings for slot filling without changing the basic workflow, which overcomes the above issues.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Learning Techniques, Natural Language Understanding, Artificial Intelligence, Natural Language Processing, Slot Filling Models" />
        </attvalues>
      </node>
      <node id="2211.00490" label="2211.00490">
        <attvalues>
          <attvalue for="0" value="Delay-penalized transducer for low-latency streaming ASR" />
          <attvalue for="1" value="  In streaming automatic speech recognition (ASR), it is desirable to reduce&#10;latency as much as possible while having minimum impact on recognition&#10;accuracy. Although a few existing methods are able to achieve this goal, they&#10;are difficult to implement due to their dependency on external alignments. In&#10;this paper, we propose a simple way to penalize symbol delay in transducer&#10;model, so that we can balance the trade-off between symbol delay and accuracy&#10;for streaming models without external alignments. Specifically, our method adds&#10;a small constant times (T/2 - t), where T is the number of frames and t is the&#10;current frame, to all the non-blank log-probabilities (after normalization)&#10;that are fed into the two dimensional transducer recursion. For both streaming&#10;Conformer models and unidirectional long short-term memory (LSTM) models,&#10;experimental results show that it can significantly reduce the symbol delay&#10;with an acceptable performance degradation. Our method achieves similar&#10;delay-accuracy trade-off to the previously published FastEmit, but we believe&#10;our method is preferable because it has a better justification: it is&#10;equivalent to penalizing the average symbol delay. Our work is open-sourced and&#10;publicly available (https://github.com/k2-fsa/k2).&#10;" />
          <attvalue for="2" value="&#10;\postsec&#10;End-to-end models have achieved remarkable success in Automatic Speech Recognition (ASR). As a prominent example, transducer~\cite{transducer, transformer-transducer, pruned-rnnt} has gained more and more popularity for real-time ASR system development, because it is&#10;naturally streaming and demonstrates superior performance.&#10;However, one limitation of transducer is that it focuses on maximizing the total log-probability over all alignments but ignores their specific symbol delays. We hypothesize that the streaming model would augment those alignments emitting symbols later to access more contexts for better performance, leading to higher emission latency in practical ASR application. &#10;&#10;There are several classical methods~\cite{cd-ctc-smbr, fast_and_accurate, emit_word_timing, min_latency, align_restrict} to reduce the model latency by constraining the alignments between the frames and transcriptions based on the alignment references generated from external models. Whilst this type of methods achieve good trade-offs between accuracy and latency, it suffers from two limitations: 1) the model performance heavily depends on the precision of the &#10;reference alignments; 2) it defeats the advantage of end-to-end model training since it requires an extra frame-level token-time alignments. &#10;&#10;To address these limitations, another line of research ~\cite{fastemit, On-Device, self-alignment} tends to regularize the objective function in a sequence-level manner. A prominent example is FastEmit~\cite{fastemit}, which encourages the model to emit symbols earlier by scaling up the derivatives of emitting non-blank tokens in backpropagation. Another work named Self alignment~\cite{self-alignment} proposes to boost the log-probability of the alignment that is one frame to the left of the Viterbi forced-alignment, which requires an extra recursion with a time complexity of $\mathcal{O}(T \times U)$ to obtain the Viterbi forced-alignment, where $T$ and $U$ are the lengths of frame sequence and token sequence respectively. Our method, like FastEmit~\cite{fastemit}, is simple to implement, but we are able to provide a more detailed demonstration explaining why our method would cause alignment times to change.&#10;&#10;In this paper, we propose a novel method of delay penalization for transducer which is able to balance the trade-off between symbol delay and accuracy for streaming models in a simple and efficient way. Different from FastEmit~\cite{fastemit} that directly changes the derivatives, we modify the log-probabilities of emitting symbols by adding a small&#10;constant $\lambda$ times the frame offsets relative to middle frame. We mathematically prove that it is approximately equivalent to adding a regularization term that aims to decrease the averaged symbol delay on the regular transducer objective function. &#10;&#10;The main contributions of this paper are: &#10;\begin{itemize}[leftmargin=*,topsep=1.8pt,parsep=0pt,itemsep=1.8pt,]&#10; \item We propose the delay-penalized transducer, which penalizes the symbol delay without extra token-time alignment. &#10; \item We provide a detailed proof why it can encourage the low-delay alignments and penalize the high-delay alignments.&#10; \item We show that a tunable trade-off between latency and accuracy can be achieved by adjusting the hyperparameter $\lambda$.&#10;\end{itemize}&#10;&#10;\presec&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Streaming Algorithms, Linguistics, Signal Processing, Mathematics, Speech Recognition, Model Optimization" />
        </attvalues>
      </node>
      <node id="2010.11148" label="2010.11148">
        <attvalues>
          <attvalue for="0" value="FastEmit: Low-latency Streaming ASR with Sequence-level Emission&#10;  Regularization" />
          <attvalue for="1" value="  Streaming automatic speech recognition (ASR) aims to emit each hypothesized&#10;word as quickly and accurately as possible. However, emitting fast without&#10;degrading quality, as measured by word error rate (WER), is highly challenging.&#10;Existing approaches including Early and Late Penalties and Constrained&#10;Alignments penalize emission delay by manipulating per-token or per-frame&#10;probability prediction in sequence transducer models. While being successful in&#10;reducing delay, these approaches suffer from significant accuracy regression&#10;and also require additional word alignment information from an existing model.&#10;In this work, we propose a sequence-level emission regularization method, named&#10;FastEmit, that applies latency regularization directly on per-sequence&#10;probability in training transducer models, and does not require any alignment.&#10;We demonstrate that FastEmit is more suitable to the sequence-level&#10;optimization of transducer models for streaming ASR by applying it on various&#10;end-to-end streaming ASR networks including RNN-Transducer,&#10;Transformer-Transducer, ConvNet-Transducer and Conformer-Transducer. We achieve&#10;150-300 ms latency reduction with significantly better accuracy over previous&#10;techniques on a Voice Search test set. FastEmit also improves streaming ASR&#10;accuracy from 4.4%/8.9% to 3.1%/7.5% WER, meanwhile reduces 90th percentile&#10;latency from 210 ms to only 30 ms on LibriSpeech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.05005" label="2105.05005">
        <attvalues>
          <attvalue for="0" value="Reducing Streaming ASR Model Delay with Self Alignment" />
          <attvalue for="1" value="  Reducing prediction delay for streaming end-to-end ASR models with minimal&#10;performance regression is a challenging problem. Constrained alignment is a&#10;well-known existing approach that penalizes predicted word boundaries using&#10;external low-latency acoustic models. On the contrary, recently proposed&#10;FastEmit is a sequence-level delay regularization scheme encouraging vocabulary&#10;tokens over blanks without any reference alignments. Although all these schemes&#10;are successful in reducing delay, ASR word error rate (WER) often severely&#10;degrades after applying these delay constraining schemes. In this paper, we&#10;propose a novel delay constraining method, named self alignment. Self alignment&#10;does not require external alignment models. Instead, it utilizes Viterbi&#10;forced-alignments from the trained model to find the lower latency alignment&#10;direction. From LibriSpeech evaluation, self alignment outperformed existing&#10;schemes: 25% and 56% less delay compared to FastEmit and constrained alignment&#10;at the similar word error rate. For Voice Search evaluation,12% and 25% delay&#10;reductions were achieved compared to FastEmit and constrained alignment with&#10;more than 2% WER improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.17347" label="2404.17347">
        <attvalues>
          <attvalue for="0" value="InspectorRAGet: An Introspection Platform for RAG Evaluation" />
          <attvalue for="1" value="  Large Language Models (LLM) have become a popular approach for implementing&#10;Retrieval Augmented Generation (RAG) systems, and a significant amount of&#10;effort has been spent on building good models and metrics. In spite of&#10;increased recognition of the need for rigorous evaluation of RAG systems, few&#10;tools exist that go beyond the creation of model output and automatic&#10;calculation. We present InspectorRAGet, an introspection platform for RAG&#10;evaluation. InspectorRAGet allows the user to analyze aggregate and&#10;instance-level performance of RAG systems, using both human and algorithmic&#10;metrics as well as annotator quality. InspectorRAGet is suitable for multiple&#10;use cases and is available publicly to the community. The demo video is&#10;available at https://youtu.be/MJhe8QIXcEc&#10;" />
          <attvalue for="2" value="&#10;&#10;The recent advances in Large Language Models (LLMs) have led to an explosion of research on Retrieval-Augmented Generation (RAG): combining generative LLMs with data retrieval to provide responses grounded on authoritative document collections \cite{rag-neurips-2020}.&#10;RAG systems have been deployed in diverse domains (see \cite{gao2024retrievalaugmented} for a recent survey). &#10;&#10;The development of all RAG systems have one thing in common: the need for rigorous evaluation. &#10;Evaluating such systems requires continuously designing experiments including datasets, models, and metrics; running evaluations; and analyzing the results to address shortcomings and make appropriate deployment decisions. Businesses wishing to deploy a RAG system must evaluate it on their specific use cases (not only academic benchmarks), often comparing multiple architectures and models. &#10;&#10;Recognizing the importance of evaluation, the research community has been creating evaluation benchmark datasets \cite{liu2023recall,chen2023benchmarking}, evaluation metrics \cite{es2023ragas}, and noise robustness \cite{chen2023benchmarking}, as well as evaluation frameworks (such as RAGAs \cite{es2023ragas}, and ARES \cite{saadfalcon2023ares}).&#10;Leveraging these assets, practitioners can automatically compute aggregate evaluation scores for their RAG systems. However, this provides only a very limited lens of a system's performance. We suggest that a comprehensive evaluation of RAG systems should include the following:&#10;&#10;Aggregate Performance: Overall evaluation is important for continuous benchmarking of models and dataset performance \cite{gehrmann-etal-2022-gemv2}.&#10;&#10;Instance-level Analysis: Aggregate metrics alone do not offer much insight into the RAG system, particularly in identifying the source of undesirable output. A flexible, feature-rich workflow for detecting and inspecting related individual instances empowers the researcher to perform actionable error analysis.&#10;&#10;Mixed Metrics: Algorithmic metrics such as BLEU and ROUGE do not necessarily align with human judgements \cite{reiter-2018-structured, Blagec2021critical}; nor are automatic evaluations using LLMs \cite{es2023ragas} a perfect substitute. However, a comprehensive analysis of all types of metrics can yield a rich array of insights.&#10;&#10;Annotator Qualification: Even human judgements of RAG systems are imperfect \cite{chiang-lee-2023-large}. A thorough understanding of annotator behavior allows the researcher to identify and improve ambiguous guidelines, complex data points, and underperforming annotators, resulting in higher quality evaluations.&#10;&#10;Dataset Characterization: The dataset itself should be subjected to a thorough inspection during evaluation. Fixing erroneous reference answers, clarifying ambiguous instances, or even identifying bias in the content can improve the overall evaluation outcome by providing much needed context for the observed quantitative results. &#10;&#10;Achieving such a multi-pronged evaluation would currently require researchers to manually examine their evaluation output, perhaps creating various ad-hoc data processing scripts and visualizations in spreadsheet tools. Our goal is to significantly lower the required effort for this important and undersupported aspect of RAG evaluation. Users should be able to easily drill down from aggregate performance overviews to instance-level views of data, models, metrics and annotators, in order to perform detailed error analysis and identify actionable insights. &#10;&#10;Our contributions are as follows:&#10;&#10;\begin{enumerate}&#10; \item We present \system, an interactive evaluation platform with a rich feature set that addresses each of the desired aspects of a comprehensive RAG evaluation: performance benchmarking, a combined aggregate and instance level analysis, a holistic view of results via a mix of metrics, annotator qualification, and dataset characterization.&#10; \item We use \system\ to yield concrete and actionable insights on two use cases, on both new and existing datasets.&#10; \item We open source the platform for use by the community.&#10;\end{enumerate}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Retrieval Augmented Generation, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2309.01431" label="2309.01431">
        <attvalues>
          <attvalue for="0" value="Benchmarking Large Language Models in Retrieval-Augmented Generation" />
          <attvalue for="1" value="  Retrieval-Augmented Generation (RAG) is a promising approach for mitigating&#10;the hallucination of large language models (LLMs). However, existing research&#10;lacks rigorous evaluation of the impact of retrieval-augmented generation on&#10;different large language models, which make it challenging to identify the&#10;potential bottlenecks in the capabilities of RAG for different LLMs. In this&#10;paper, we systematically investigate the impact of Retrieval-Augmented&#10;Generation on large language models. We analyze the performance of different&#10;large language models in 4 fundamental abilities required for RAG, including&#10;noise robustness, negative rejection, information integration, and&#10;counterfactual robustness. To this end, we establish Retrieval-Augmented&#10;Generation Benchmark (RGB), a new corpus for RAG evaluation in both English and&#10;Chinese. RGB divides the instances within the benchmark into 4 separate&#10;testbeds based on the aforementioned fundamental abilities required to resolve&#10;the case. Then we evaluate 6 representative LLMs on RGB to diagnose the&#10;challenges of current LLMs when applying RAG. Evaluation reveals that while&#10;LLMs exhibit a certain degree of noise robustness, they still struggle&#10;significantly in terms of negative rejection, information integration, and&#10;dealing with false information. The aforementioned assessment outcomes indicate&#10;that there is still a considerable journey ahead to effectively apply RAG to&#10;LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Retrieval-augmented models &#10;&#10;The knowledge stored in large language models is commonly out-of-date~\cite{he2022rethinking} and they also sometimes generate hallucination~\cite{cao-etal-2020-factual,raunak-etal-2021-curious,10.1145/3571730} i.e., they may generate irrelevant or factually incorrect contents. By using external knowledge as guidance, retrieval-augmented models can generate more accurate and reliable responses~\cite{10.5555/3524938.3525306,10.5555/3495724.3496517,borgeaud2022improving,izacard2022atlas,shi2023replug,ren2023investigating}. Retrieval-augmented models have achieved remarkable results in various tasks such as open-domain QA~\cite{izacard-grave-2021-leveraging,trivedi-etal-2023-interleaving,li-etal-2023-large}, dialogue~\cite{cai-etal-2019-skeleton,cai-etal-2019-retrieval,peng2023check}, domain-specific question answering~\cite{cui2023chatlaw} and code generation~\cite{zhou2023docprompting}. Recently, with the development of large models, a series of retrieval-enhanced tools and products have gained widespread attention, such as ChatGPT retrieval plugin, Langchain, New Bing, etc. However, in real-world scenarios, the retrieved text inevitably contains noise. Therefore, in this paper we conducted a systematic evaluation and analysis of retrieval-augmented generation in LLMs.&#10;&#10;Evaluation of LLMs&#10;Evaluating LLMs has received significant attention due to their remarkable general capability~\cite{chang2023survey}. It enables us to gain a deeper understanding of the specific abilities and limitations of LLMs, while also providing valuable guidance for future research. In the past, benchmarks such as GLUE~\cite{wang2018glue} and SuperCLUE~\cite{10.5555/3454287.3454581} primarily focused on evaluating NLP tasks, particularly in natural language understanding. However, these evaluations often fail to fully capture the capabilities of LLMs. MMLU~\cite{hendrycks2021measuring} was then proposed to measure the knowledge acquired by language models when pre-training. Recently, with the development of LLMs, a series of general evaluation benchmarks have emerged, such as AGIEval~\cite{zhong2023agieval}, C-Eval~\cite{huang2023ceval}, AlpacaEval~\cite{alpaca_eval}, OpenLLM Leaderboard~\cite{open-llm-leaderboard}, etc. In addition to general abilities, there are also specific benchmarks that focus on evaluating the capabilities of models. For example, CValues~\cite{xu2023cvalues} focuses on the safety and responsibility of LLMs, M3Exam~\cite{zhang2023m3exam} focuses on human exam and ToolBench~\cite{qin2023toolllm} evaluates how well LLMs use external tools. Recently, \cite{adlakha2023evaluating} evaluate the RAG of LLMs in exist QA dataset. Different from their work, we focus on 4 required abilities of RAG and create Retrieval-Augmented Generation Benchmark to evaluate the LLMs.&#10; " />
        </attvalues>
      </node>
      <node id="2311.09476" label="2311.09476">
        <attvalues>
          <attvalue for="0" value="ARES: An Automated Evaluation Framework for Retrieval-Augmented&#10;  Generation Systems" />
          <attvalue for="1" value="  Evaluating retrieval-augmented generation (RAG) systems traditionally relies&#10;on hand annotations for input queries, passages to retrieve, and responses to&#10;generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating&#10;RAG systems along the dimensions of context relevance, answer faithfulness, and&#10;answer relevance. By creating its own synthetic training data, ARES finetunes&#10;lightweight LM judges to assess the quality of individual RAG components. To&#10;mitigate potential prediction errors, ARES utilizes a small set of&#10;human-annotated datapoints for prediction-powered inference (PPI). Across eight&#10;different knowledge-intensive tasks in KILT, SuperGLUE, and AIS, ARES&#10;accurately evaluates RAG systems while using only a few hundred human&#10;annotations during evaluation. Furthermore, ARES judges remain effective across&#10;domain shifts, proving accurate even after changing the type of queries and/or&#10;documents used in the evaluated RAG systems. We make our code and datasets&#10;publicly available on Github.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.11249" label="2206.11249">
        <attvalues>
          <attvalue for="0" value="GEMv2: Multilingual NLG Benchmarking in a Single Line of Code" />
          <attvalue for="1" value="  Evaluation in machine learning is usually informed by past choices, for&#10;example which datasets or metrics to use. This standardization enables the&#10;comparison on equal footing using leaderboards, but the evaluation choices&#10;become sub-optimal as better alternatives arise. This problem is especially&#10;pertinent in natural language generation which requires ever-improving suites&#10;of datasets, metrics, and human evaluation to make definitive claims. To make&#10;following best model evaluation practices easier, we introduce GEMv2. The new&#10;version of the Generation, Evaluation, and Metrics Benchmark introduces a&#10;modular infrastructure for dataset, model, and metric developers to benefit&#10;from each others work. GEMv2 supports 40 documented datasets in 51 languages.&#10;Models for all datasets can be evaluated online and our interactive data card&#10;creation and rendering tools make it easier to add new datasets to the living&#10;benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.04726" label="2304.04726">
        <attvalues>
          <attvalue for="0" value="Uncertainty-Aware Natural Language Inference with Stochastic Weight&#10;  Averaging" />
          <attvalue for="1" value="  This paper introduces Bayesian uncertainty modeling using Stochastic Weight&#10;Averaging-Gaussian (SWAG) in Natural Language Understanding (NLU) tasks. We&#10;apply the approach to standard tasks in natural language inference (NLI) and&#10;demonstrate the effectiveness of the method in terms of prediction accuracy and&#10;correlation with human annotation disagreements. We argue that the uncertainty&#10;representations in SWAG better reflect subjective interpretation and the&#10;natural variation that is also present in human language understanding. The&#10;results reveal the importance of uncertainty modeling, an often neglected&#10;aspect of neural language modeling, in NLU tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Arguably, human language understanding is not objective nor deterministic. The same utterance or text can be interpreted in different ways by different people depending on their language standards, background knowledge and world views, the linguistic context, as well as the situation in which the utterance or text appears. This uncertainty about potential readings is typically not modeled in Natural Language Understanding (NLU) research and is often ignored in NLU benchmarks and datasets. Instead, they usually assign a single interpretation as a gold standard to be predicted by an artificial system ignoring the inherent ambiguity of language and potential disagreements that humans arrive at.&#10;&#10;Some datasets like SNLI \cite{bowman-etal-2015-large} and MNLI \cite{williams-etal-2018-broad} do, however, contain information about different readings in the form of annotation disagreement. These datasets include the labels from five different rounds of annotation which show in some cases clear disagreement about the correct label for the sentence pair. Those labeling discrepancies &#10;can certainly be a result of annotation mistakes but more commonly they arise from differences in understanding the task, the given information and how it relates to world knowledge and personal experience.&#10;&#10;Moving towards uncertainty-aware neural language models, we present our initial results using Stochastic Weight Averaging (SWA) \cite{izmailov2018averaging} and SWA-Gaussian (SWAG) \cite{maddox2019simple} on the task of Natural Language Inference. SWAG provides a scalable approach to calibrate neural networks and to model uncertainty presentations and is straightforward to apply with standard neural architectures. Our study addresses the two main questions:&#10;\begin{itemize}&#10; \item How does uncertainty modeling using SWAG influence prediction performance and generalization in NLI tasks?&#10; \item How well does the calibrated model reflect human disagreement and annotation variance?&#10;\end{itemize}&#10;In this paper, we first &#10;test the performance of SWA and SWAG in SNLI and MNLI tasks. We then study if adding weight averaging improves the generalization power of NLI models as tested through cross-dataset experiments. Finally, we analyse the probability distributions from SWA and SWAG to test how well the model uncertainty corresponds to annotator disagreements. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Natural Language Understanding, Neural Language Modeling, Mathematics, Bayesian Uncertainty Modeling, Statistics" />
        </attvalues>
      </node>
      <node id="1902.02476" label="1902.02476">
        <attvalues>
          <attvalue for="0" value="A Simple Baseline for Bayesian Uncertainty in Deep Learning" />
          <attvalue for="1" value="  We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose&#10;approach for uncertainty representation and calibration in deep learning.&#10;Stochastic Weight Averaging (SWA), which computes the first moment of&#10;stochastic gradient descent (SGD) iterates with a modified learning rate&#10;schedule, has recently been shown to improve generalization in deep learning.&#10;With SWAG, we fit a Gaussian using the SWA solution as the first moment and a&#10;low rank plus diagonal covariance also derived from the SGD iterates, forming&#10;an approximate posterior distribution over neural network weights; we then&#10;sample from this Gaussian distribution to perform Bayesian model averaging. We&#10;empirically find that SWAG approximates the shape of the true posterior, in&#10;accordance with results describing the stationary distribution of SGD iterates.&#10;Moreover, we demonstrate that SWAG performs well on a wide variety of tasks,&#10;including out of sample detection, calibration, and transfer learning, in&#10;comparison to many popular alternatives including MC dropout, KFAC Laplace,&#10;SGLD, and temperature scaling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.02448" label="2010.02448">
        <attvalues>
          <attvalue for="0" value="On the Branching Bias of Syntax Extracted from Pre-trained Language&#10;  Models" />
          <attvalue for="1" value="  Many efforts have been devoted to extracting constituency trees from&#10;pre-trained language models, often proceeding in two stages: feature definition&#10;and parsing. However, this kind of methods may suffer from the branching bias&#10;issue, which will inflate the performances on languages with the same branch it&#10;biases to. In this work, we propose quantitatively measuring the branching bias&#10;by comparing the performance gap on a language and its reversed language, which&#10;is agnostic to both language models and extracting methods. Furthermore, we&#10;analyze the impacts of three factors on the branching bias, namely parsing&#10;algorithms, feature definitions, and language models. Experiments show that&#10;several existing works exhibit branching biases, and some implementations of&#10;these three factors can introduce the branching bias.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural language models such as LSTM~\cite{merityRegOpt,peters2018deep}, GPT2~\cite{radford2019language}, and BERT~\cite{devlin2019bert,liu2019roberta} have achieved state-of-the-art performance in various downstream NLP tasks. Many recent works try to interpret their success by revealing the linguistic properties captured by these language models \cite{hewitt2019structural, clark2019does, jawahar2019does, tenney2019you}. One interesting line of these works tries to extract discrete constituency trees from pre-trained language models \cite{marevcek2018extracting,marevcek2019balustrades,kim2020pre, wu2020perturbed}. The core of these works is to extract syntax in two stages. Firstly, it defines the feature scores based on a language model, namely, the {feature definition stage}. Secondly, it leverages the feature scores to build a constituency tree, namely, the {parsing stage}.&#10;&#10;However, the degree to which the extracted constituency trees match gold constituency annotations may imprecisely reflect the model's competence of capturing syntax, since their final performance may benefit from the branching bias.&#10;For example, as pointed out by~\cite{dyer2019critical}, the syntax extracted from the ordered neuron based language model~\cite{shen2018ordered} is biased to right-branching languages (e.g., English). Nevertheless, the approach to measuring the bias in~\cite{dyer2019critical} is highly dependent on the architecture of ordered neuron and its parsing algorithm. Therefore, it is far from trivial to be applied to general pre-trained language models. &#10;&#10;This paper proposes a new approach to reveal the branching bias of syntax from pre-trained language models, which is agnostic to model architectures and parsing algorithms. &#10;The key idea of our approach is based on the following observation: We can construct a left-branching language by reversing a right-branching language and vice versa. An illustration is given in Figure ~\ref{fig:trees}. If a syntax extracting method has no branching bias, the parsing performances on the original language and the reversed language should have little or no difference.&#10;Therefore, the performance gap can be used as an indicator of branching bias. &#10;Using our approach, we find that some recent works on pre-trained language models suffer from the branching bias~\cite{kim2020pre,wu2020perturbed, marevcek2018extracting}.&#10;We further investigate on an in-depth question: Does the bias come from language models? Or the extraction methods (feature definition and parsing algorithm)?&#10;We propose a simple approach to quantitatively analyze the bias in them, which tries to control the impacts of other factors while studying a specific part in the pipeline.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Analysis, Linguistics, Artificial Intelligence, Mathematics, Branching Bias Measurement, Parsing Algorithm Evaluation" />
        </attvalues>
      </node>
      <node id="1906.01958" label="1906.01958">
        <attvalues>
          <attvalue for="0" value="From Balustrades to Pierre Vinken: Looking for Syntax in Transformer&#10;  Self-Attentions" />
          <attvalue for="1" value="  We inspect the multi-head self-attention in Transformer NMT encoders for&#10;three source languages, looking for patterns that could have a syntactic&#10;interpretation. In many of the attention heads, we frequently find sequences of&#10;consecutive states attending to the same position, which resemble syntactic&#10;phrases. We propose a transparent deterministic method of quantifying the&#10;amount of syntactic information present in the self-attentions, based on&#10;automatically building and evaluating phrase-structure trees from the&#10;phrase-like sequences. We compare the resulting trees to existing constituency&#10;treebanks, both manually and by computing precision and recall.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14786" label="2004.14786">
        <attvalues>
          <attvalue for="0" value="Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting&#10;  BERT" />
          <attvalue for="1" value="  By introducing a small set of additional parameters, a probe learns to solve&#10;specific linguistic tasks (e.g., dependency parsing) in a supervised manner&#10;using feature representations (e.g., contextualized embeddings). The&#10;effectiveness of such probing tasks is taken as evidence that the pre-trained&#10;model encodes linguistic knowledge. However, this approach of evaluating a&#10;language model is undermined by the uncertainty of the amount of knowledge that&#10;is learned by the probe itself. Complementary to those works, we propose a&#10;parameter-free probing technique for analyzing pre-trained language models&#10;(e.g., BERT). Our method does not require direct supervision from the probing&#10;tasks, nor do we introduce additional parameters to the probing process. Our&#10;experiments on BERT show that syntactic trees recovered from BERT using our&#10;method are significantly better than linguistically-uninformed baselines. We&#10;further feed the empirically induced dependency structures into a downstream&#10;sentiment classification task and find its improvement compatible with or even&#10;superior to a human-designed dependency schema.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;There has been substantial research investigating what pre-trained language models have learned about languages' structures. &#10;&#10;One rising line of research uses probing classifiers to investigate the different syntactic properties captured by the model. They are generally referred to as ``probing task''~\cite{conneau2018you}, ``diagnostic classifier''~\cite{giulianelli2018under}, and ``auxiliary prediction tasks''~\cite{Adi2017}. The syntactic properties investigated range from basic ones like sentence length~\cite{Shi2016, Jawahar2019}, syntactic tree depth~\cite{Jawahar2019}, and segmentation~\cite{liu2019linguistic} to challenging ones like syntactic labeling~\cite{tenney2019bert,Tenney2019}, dependency parsing~\cite{Hewitt2019, Clark2019}, and constituency parsing~\cite{peters2018dissecting}. However, when a probe achieves high accuracy, it’s difficult to differentiate if it is the representation that encodes targeted syntactic information, or it is the probe that just learns the task~\cite{hewitt2019designing}.&#10;&#10;In line with our work, recent studies seek to find correspondences between parts of the neural network and certain linguistic properties, without explicit supervision.&#10;&#10;Most of them focus on analyzing attention mechanism, by extracting syntactic tree for each attention head and layer individually~\cite{Tiedemann2018, Clark2019}. Their goal is to check if the attention heads of a given pre-trained model can track syntactic relations better than chance or baselines. In particular, \cite{Tiedemann2018} analyze a machine translation model's encoder by extracting dependency trees from its self-attention weights, using Chu-Liu/Edmonds algorithm. \cite{Clark2019} conduct a similar investigation on BERT, but the simple head selection strategy they used does not guarantee a valid dependency tree. \cite{Marecek2018} propose heuristic methods to convert attention weights to syntactic trees. However, they do not quantitatively evaluate their approach. In their later study~\cite{Marecek2019}, they propose a bottom-up algorithm to extract constituent trees from transformer-based NMT encoders and evaluate their results on three languages. \cite{htut2019attention} reassess these works but find that there are no generalist heads that can do holistic parsing. Hence, analyzing attention weights directly may not reveal much of the syntactic knowledge that a model has learned. Recent dispute about attention as explanation~\cite{jain2019attention, serrano-smith-2019-attention, wiegreffe2019attention} also suggests that the attention's behavior does not necessarily represent that of the original model. &#10;&#10;Another group of research examine the outputs of language models on carefully chosen input sentences~\cite{Goldberg2019, Bacon2019}. They extend previous works~\cite{Linzen2016, Gulordava2018, marvin2018targeted} on subject-verb agreement test (generating the correct number of a verb far away from its subject) to provide a measure of the model’s syntactic ability. Their results show that the BERT model captures syntax-sensitive agreement patterns well in general. However, subject-verb agreement cannot provide more nuanced tests of other complex structures (e.g., dependency structure, constituency structure), which are the interest of our work.&#10;&#10;Two recent works also perturb the input sequence for model interpretability~\cite{Rosa2019, Xintong2019}. However, these works only perturb the sequence once. \cite{Rosa2019} utilize the original MLM objective to estimate each word's ``reducibility'' and import simple heuristics into a right-chain baseline to construct dependency trees. \cite{Xintong2019} focus on evaluating word alignment in NMT, but unlike our two-step masking strategy, they only replace the token of interest with a zero embedding or a randomly sampled word in the vocabulary. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2311.02883" label="2311.02883">
        <attvalues>
          <attvalue for="0" value="SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Data" />
          <attvalue for="1" value="  Text-to-SQL aims to automate the process of generating SQL queries on a&#10;database from natural language text. In this work, we propose &quot;SQLPrompt&quot;,&#10;tailored to improve the few-shot prompting capabilities of Text-to-SQL for&#10;Large Language Models (LLMs). Our methods include innovative prompt design,&#10;execution-based consistency decoding strategy which selects the SQL with the&#10;most consistent execution outcome among other SQL proposals, and a method that&#10;aims to improve performance by diversifying the SQL proposals during&#10;consistency selection with different prompt designs (&quot;MixPrompt&quot;) and&#10;foundation models (&quot;MixLLMs&quot;). We show that \emph{SQLPrompt} outperforms&#10;previous approaches for in-context learning with few labeled data by a large&#10;margin, closing the gap with finetuning state-of-the-art with thousands of&#10;labeled data.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text-to-SQL enables natural language interfaces for databases via SQL query generation. It is crucial for enhancing database accessibility without SQL expertise, and enabling the development of conversational agents with data analysis ability.&#10;&#10;Language models (LM) have shown to be promising for Text-to-SQL. Notable previous work on finetuning, including PICARD \cite{scholak2021picard}, UnifiedSKG \cite{xie2022unifiedskg}, and RESDSQL-3B + NatSQL\cite{li2023decoupling}, achieve impressive results by leveraging customized SQL-specific syntax knowledge and training on a large number of (text, SQL) paired data samples. Recently, large language models (LLMs) such as GPT-3 \cite{brown2020language}, PaLM \cite{chowdhery2022palm}, and ChatGPT\cite{stiennon2020learning} have demonstrated promising few-shot abilities via prompting\cite{wei2022emergent}. By only providing a few demonstrations in the prompt, LLMs are able to follow the demonstrations and generate reasonable answers (``in-context learning''). For Text-to-SQL, few-shot prompting is beneficial as it does not require expensive training, lowers adaptation data requirements, reduces out-of-distribution issues (e.g. for unseen phrases), and reduces the risk of over-fitting and poor generalization. \\In this paper, we introduce ``SQLPrompt'', a few-shot prompting approach for Text-to-SQL. The approach is comprised of execution-based consistency decoding and execution error filtering, ``MixPrompt'' and ``MixLLMs'' to enhance diversity of SQL proposals. In few-shot prompting, self-consistency decoding \cite{wang2022self}, which samples a diverse set of reasoning paths and selects the most consistent answer, has shown remarkable performance improvements across different tasks. Here we propose a novel variant of consistency decoding for Text-to-SQL, ``execution-based consistency decoding and execution error filtering''. The proposed decoding is tailored to be SQL-execution specific -- we marginalize over SQLs, and conduct majority vote on execution outcome, whereas the original self consistency approach marginalizes over chain-of-thought thinking path, and conduct majority vote on the final answer. \\&#10;Furthermore, the improvement brought by self-consistency' decoding using the same prompt and same LLMs saturate beyond a certain number of samples. Because the diversity of SQL candidates are limited with the same prompt and same LLM. Here we propose employing diverse prompt designs (&quot;MixPrompt&quot;) and LLMs (&quot;MixLLMs&quot;) to obtain more diverse LLM outputs. &quot;MixPrompt&quot; applies different prompt designs, which changes the interface of query and LLMs, leading to more diverse LLM's outputed SQLs. &quot;MixLLMs&quot; holds the assumption that different LLMs pretrained on diverse set of knowledge, can yield different outcomes. Finally, the answer is the consistent answer across different prompt designs and LLMs. &#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Mathematics, Database Systems, Query Optimization Techniques, Text-to-SQL Generation" />
        </attvalues>
      </node>
      <node id="2109.05093" label="2109.05093">
        <attvalues>
          <attvalue for="0" value="PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding&#10;  from Language Models" />
          <attvalue for="1" value="  Large pre-trained language models for textual data have an unconstrained&#10;output space; at each decoding step, they can produce any of 10,000s of&#10;sub-word tokens. When fine-tuned to target constrained formal languages like&#10;SQL, these models often generate invalid code, rendering it unusable. We&#10;propose PICARD (code and trained models available at&#10;https://github.com/ElementAI/picard), a method for constraining auto-regressive&#10;decoders of language models through incremental parsing. PICARD helps to find&#10;valid output sequences by rejecting inadmissible tokens at each decoding step.&#10;On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that&#10;PICARD transforms fine-tuned T5 models with passable performance into&#10;state-of-the-art solutions.&#10;" />
          <attvalue for="2" value="&#10;&#10;While there have been many successes in applying large pre-trained language models&#10;to downstream tasks,&#10;our ability to control and constrain the output of these models is still very limited.&#10;Many enterprise applications are out of reach&#10;because they require a degree of rigour and exactitude that&#10;language models are not able to deliver yet.&#10;If the target is a formal language like SQL,&#10;then we would like the model to adhere exactly and provably to the SQL specification&#10;with all its lexical, grammatical, logical, and semantical constraints.&#10;Unfortunately, with pre-training alone, language models may not satisfy&#10;these correctness requirements.&#10;&#10;For \texttosql translation,&#10;the most widespread solution to constrained decoding&#10;is to make invalid SQL unrepresentable.&#10;For a while now&#10;it has been possible to restrict auto-regressive decoding to only those token sequences&#10;that correctly parse to SQL abstract syntax trees&#10;\cite{Yin_2018,lin2019grammarbased,Wang_2020}.&#10;More recently,&#10;semi-auto-regressive improvements to this parsing paradigm&#10;have been proposed \cite{rubin2020smbop}.&#10;However, while effective,&#10;these approaches have in common that they are achieved at the expense of&#10;using a custom vocabulary of special control tokens&#10;or a custom model architecture, or both.&#10;Unfortunately,&#10;this makes them incompatible with generic pre-trained language model decoders.&#10;A less invasive and more compatible approach is&#10;to not constrain the generation process,&#10;but instead to filter finalized beam hypotheses by validity&#10;\cite{suhr-etal-2020-exploring,Lin_2020}.&#10;Yet, such filtering is at the expense of a very large beam size.&#10;&#10;We address the expenses of these approaches with&#10;a novel incremental parsing method&#10;for constrained decoding called \Picard,&#10;which stands for&#10;&quot;Parsing Incrementally for Constrained Auto-Regressive Decoding.&quot;&#10;\Picard is compatible with any existing auto-regressive language model decoder&#10;and vocabulary---including, but not limited to, those of large pre-trained transformers---and it does not require very large beam sizes.&#10;\Picard is entirely absent from pre-training or fine-tuning of the model,&#10;and can be easily and optionally enabled at inference time.&#10;\Picard operates directly on the output of the language model&#10;which, in the case of \texttosql translation,&#10;is the readable surface form of the SQL code.&#10;&#10;In our experiments, we find that&#10;\Picard can significantly improve the performance of a large pre-trained language model&#10;\cite{t5}&#10;after it is fine-tuned on the \texttosql task.&#10;On the Spider \texttosql dataset \cite{Yu_2018},&#10;we find that a T5-Base model with \Picard can outperform a T5-Large model without it,&#10;and likewise for a T5-Large and a T5-3B model.&#10;Significantly,&#10;with the help of \Picard,&#10;a T5-3B model can be raised to state-of-the-art performance on&#10;the Spider and CoSQL datasets \cite{yu-etal-2019-cosql}.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.05966" label="2201.05966">
        <attvalues>
          <attvalue for="0" value="UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding&#10;  with Text-to-Text Language Models" />
          <attvalue for="1" value="  Structured knowledge grounding (SKG) leverages structured knowledge to&#10;complete user requests, such as semantic parsing over databases and question&#10;answering over knowledge bases. Since the inputs and outputs of SKG tasks are&#10;heterogeneous, they have been studied separately by different communities,&#10;which limits systematic and compatible research on SKG. In this paper, we&#10;overcome this limitation by proposing the UnifiedSKG framework, which unifies&#10;21 SKG tasks into a text-to-text format, aiming to promote systematic SKG&#10;research, instead of being exclusive to a single task, domain, or dataset. We&#10;use UnifiedSKG to benchmark T5 with different sizes and show that T5, with&#10;simple modifications when necessary, achieves state-of-the-art performance on&#10;almost all of the 21 tasks. We further demonstrate that multi-task&#10;prefix-tuning improves the performance on most tasks, largely improving the&#10;overall performance. UnifiedSKG also facilitates the investigation of zero-shot&#10;and few-shot learning, and we show that T0, GPT-3, and Codex struggle in&#10;zero-shot and few-shot learning for SKG. We also use UnifiedSKG to conduct a&#10;series of controlled experiments on structured knowledge encoding variants&#10;across SKG tasks. UnifiedSKG is easily extensible to more tasks, and it is&#10;open-sourced at https://github.com/hkunlp/unifiedskg.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.02396" label="2406.02396">
        <attvalues>
          <attvalue for="0" value="The Scandinavian Embedding Benchmarks: Comprehensive Assessment of&#10;  Multilingual and Monolingual Text Embedding" />
          <attvalue for="1" value="  The evaluation of English text embeddings has transitioned from evaluating a&#10;handful of datasets to broad coverage across many tasks through benchmarks such&#10;as MTEB. However, this is not the case for multilingual text embeddings due to&#10;a lack of available benchmarks. To address this problem, we introduce the&#10;Scandinavian Embedding Benchmark (SEB). SEB is a comprehensive framework that&#10;enables text embedding evaluation for Scandinavian languages across 24 tasks,&#10;10 subtasks, and 4 task categories. Building on SEB, we evaluate more than 26&#10;models, uncovering significant performance disparities between public and&#10;commercial solutions not previously captured by MTEB. We open-source SEB and&#10;integrate it with MTEB, thus bridging the text embedding evaluation gap for&#10;Scandinavian languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural language embeddings are used in a diverse range of applications, including clustering~\cite{liu2011survey, Angelov2020Top2VecDR}, &#10;text mining~\cite{jiang2015training}, &#10;semantic search~\cite{reimers2019sentence,muennighoff2022sgpt} &#10;and feature representation~\cite{alayrac2022flamingo}. &#10;Furthermore, embeddings are crucial in rag systems \cite{borgeaud2022improving}, particularly for low- to mid-resource languages and domains. rag systems enable the enrichment of generative models with the knowledge that might be underrepresented or absent during training. Thus, they can play a role in broadening linguistic and domain coverage.&#10;&#10;With the breadth of applications for text embeddings, a proper evaluation of their quality is critical. Recent work has proposed mteb \cite{muennighoff-etal-2023-mteb}, a benchmark for evaluating the quality of document embeddings for a wide variety of tasks. mteb improves upon prior benchmarks by addressing the lack of evaluations across tasks. This has led to the widespread adoption of the benchmark for evaluating natural language embeddings.&#10;&#10;However, while mteb substantially improves the evaluation of text embeddings, the benchmark has the following shortcomings:&#10;&#10;\begin{enumerate}&#10; \item Support for non-English evaluation: mteb contains only limited support for evaluating non-English embeddings and multiple task categories are predominantly covered by translated datasets (classification) and important task such as retrieval has no multilingual support.&#10; \item Reproducibilty: mteb does not include model implementations in the benchmark's code. This is especially problematic since recent approaches such as prompt-based embedding models~\cite{muennighoff2022sgpt,xiao2023c,su2022one}, Matryoshka embeddings \cite{NEURIPS2022_c32319f4} introduce variables which can dramatically influence performance.&#10; \item Coverage: While mteb has broad coverage across tasks, its domain coverage is still limited, as it primarily includes datasets from academic articles, social media, and web sources. This lack of coverage is especially pronounced for non-English tasks.&#10;\end{enumerate}&#10;&#10;Our work is driven by the reality that Scandinavian research, public institutions, and industry have to make decisions about their choice of text embedding model for various use cases.&#10;These choices are currently made in the absence of a reliable standard to evaluate text embedding models' performance on Scandinavian languages.&#10;As a result, these institutions have relied on proxies, such as models' performance on predominantly English benchmarks or Bitext mining tasks. &#10;As we demonstrate, performance on these tasks is not necessarily transferable to Scandinavian applications, thus not properly accounting for these institutions' requirements.&#10;By introducing a benchmark tailored for Scandinavian languages, we aim to aid these organizations in making informed decisions.&#10;Additionally, SEB will presumably support the development of Scandinavian embedding models by providing a standardized means for evaluating new models and comparing them against previously existing ones.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Benchmark Evaluation Frameworks, Linguistics, Multilingual Text Embeddings, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.13243" label="2310.13243">
        <attvalues>
          <attvalue for="0" value="Open-source Large Language Models are Strong Zero-shot Query Likelihood&#10;  Models for Document Ranking" />
          <attvalue for="1" value="  In the field of information retrieval, Query Likelihood Models (QLMs) rank&#10;documents based on the probability of generating the query given the content of&#10;a document. Recently, advanced large language models (LLMs) have emerged as&#10;effective QLMs, showcasing promising ranking capabilities. This paper focuses&#10;on investigating the genuine zero-shot ranking effectiveness of recent LLMs,&#10;which are solely pre-trained on unstructured text data without supervised&#10;instruction fine-tuning. Our findings reveal the robust zero-shot ranking&#10;ability of such LLMs, highlighting that additional instruction fine-tuning may&#10;hinder effectiveness unless a question generation task is present in the&#10;fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking&#10;system that integrates LLM-based QLMs with a hybrid zero-shot retriever,&#10;demonstrating exceptional effectiveness in both zero-shot and few-shot&#10;scenarios. We make our codebase publicly available at&#10;https://github.com/ielab/llm-qlm.&#10;" />
          <attvalue for="2" value="&#10;Ranking models (or rankers) are a fundamental component in many information retrieval (IR) pipelines. &#10;Pre-trained language models (PLMs) have recently been leveraged across bi-encoder~\cite{karpukhin-etal-2020-dense, xiong2021approximate,zhuang2021tilde,wang2022simlm,gao-callan-2022-unsupervised}, cross-encoder~\cite{nogueira2019passage, nogueira-etal-2020-document,zhuang2021QLMT5}, and sparse~\cite{lin2021few,Formal2021SPLADE,zhuang2021fast} ranker architectures, showing impressive ranking effectiveness. &#10;&#10;Despite this success, the strong effectiveness of PLM-based rankers does not always generalise without sufficient in-domain training data~\cite{thakur2021beir,zhuang-zuccon-2021-dealing,zhuang2022character}.&#10;Transferring knowledge from other domains has been used to overcome this issue~\cite{Lin2023HowTT} by training these rankers on large-scale supervised QA datasets such as MS MARCO~\cite{nguyen2017ms}.&#10;Alternatively, generative large language models (LLMs) like GPT3~\cite{Brown2020GPT3} have been used to synthesize domain-specific training queries, which are then used to train these rankers~\cite{Luiz2022inpars, dai2023promptagator}.&#10;Despite their effectiveness, all of these methods consume significant expenses in training a PLM-based ranker.&#10;&#10;In this paper, we consider a third avenue to address this challenge: leveraging LLMs to function as Query Likelihood Models (QLMs)~\cite{Ponte1998QLM,hiemstra2000using,Zhai2001QLM}.&#10;Essentially, QLMs are expected to understand the semantics of documents and queries, and estimate the possibility that each document can answer a certain query.&#10;Notably, recent advances in this direction have greatly enhanced the ranking effectiveness of QLM-based rankers by leveraging PLMs like BERT~\cite{devlin-etal-2019-bert} and T5~\cite{raffel2020t5}. These PLM-based QLMs are fine-tuned on query generation tasks and subsequently employed to rank documents as per their likelihood~\cite{nogueira-dos-santos-etal-2020-beyond,zhuang2021QLMT5,Lesota2021QLM,zhuang2021tilde}. &#10;&#10;We focus on a specific PLM-based QLM, the recently proposed Unsupervised Passage Re-ranker (UPR)~\cite{sachan-etal-2022-improving}. UPR leverages advanced LLMs to obtained the query likelihood estimations. Empirical results show that using the T0 LLM~\cite{sanh2022multitask} as a QLM, large gains in ranking effectiveness can be obtained. A key aspect of this work is that this effectiveness is obtained without requiring additional fine-tuning data, making \cite{sachan-etal-2022-improving} highlight the zero-shot ranking capabilities of their LLM-based QLM.&#10;However, we argue that the experimental setting used by \cite{sachan-etal-2022-improving}&#10;does not fully align with a genuine zero-shot scenario for the QLM ranking task.&#10;This is because T0 has already undergone fine-tuning on numerous question generation (QG) tasks and datasets, subsequent to its unsupervised pre-training. Consequently, there exists a discernible task leakage to the downstream QLM ranking task, thereby rendering their approach more akin to a transfer learning setting, rather than a true zero-shot approach.&#10;&#10;To gain a comprehensive understanding of the zero-shot ranking capabilities of LLM-based QLM rankers, in this paper we take a fresh examination of this topic. Our approach involves harnessing the power of state-of-the-art transformer decoder-only LLMs, such as LLaMA~\cite{touvron2023llama}, which have undergone pre-training solely on unstructured text through unsupervised next token prediction. Importantly, the models we consider have not undergone any additional supervised instruction fine-tuning, ensuring a truly complete zero-shot setting for our investigation. &#10;&#10;We further extend our analysis by comparing the effectiveness of these LLMs with various popular instruction-tuned LLMs in the context of zero-shot ranking tasks. Interestingly, our findings reveal that further instruction fine-tuning adversely affects the effectiveness of QLM ranking, particularly when the fine-tuning datasets lack specific QG tasks. This insight highlights the strong zero-shot QLM ranking ability of LLMs that solely rely on pre-training, thereby suggesting that further instruction fine-tuning is unnecessary for achieving strong zero-shot effectiveness. Building upon these insights, we push the boundaries of zero-shot ranking even further by integrating a hybrid zero-shot first-stage retrieval system, followed by re-ranking using the zero-shot LLM-based QLM re-rankers and a relevance score interpolation technique~\cite{shuai2021interpolate}. Our approach achieves state-of-the-art effectiveness in zero-shot ranking on a subset of the BIER dataset~\cite{thakur2021beir}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Language Models, Artificial Intelligence, Mathematics, Query Ranking" />
        </attvalues>
      </node>
      <node id="2207.02578" label="2207.02578">
        <attvalues>
          <attvalue for="0" value="SimLM: Pre-training with Representation Bottleneck for Dense Passage&#10;  Retrieval" />
          <attvalue for="1" value="  In this paper, we propose SimLM (Similarity matching with Language Model&#10;pre-training), a simple yet effective pre-training method for dense passage&#10;retrieval. It employs a simple bottleneck architecture that learns to compress&#10;the passage information into a dense vector through self-supervised&#10;pre-training. We use a replaced language modeling objective, which is inspired&#10;by ELECTRA, to improve the sample efficiency and reduce the mismatch of the&#10;input distribution between pre-training and fine-tuning. SimLM only requires&#10;access to unlabeled corpus, and is more broadly applicable when there are no&#10;labeled data or queries. We conduct experiments on several large-scale passage&#10;retrieval datasets, and show substantial improvements over strong baselines&#10;under various settings. Remarkably, SimLM even outperforms multi-vector&#10;approaches such as ColBERTv2 which incurs significantly more storage cost. Our&#10;code and model check points are available at&#10;https://github.com/microsoft/unilm/tree/master/simlm .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.05720" label="2107.05720">
        <attvalues>
          <attvalue for="0" value="SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking" />
          <attvalue for="1" value="  In neural Information Retrieval, ongoing research is directed towards&#10;improving the first retriever in ranking pipelines. Learning dense embeddings&#10;to conduct retrieval using efficient approximate nearest neighbors methods has&#10;proven to work well. Meanwhile, there has been a growing interest in learning&#10;sparse representations for documents and queries, that could inherit from the&#10;desirable properties of bag-of-words models such as the exact matching of terms&#10;and the efficiency of inverted indexes. In this work, we present a new&#10;first-stage ranker based on explicit sparsity regularization and a&#10;log-saturation effect on term weights, leading to highly sparse representations&#10;and competitive results with respect to state-of-the-art dense and sparse&#10;methods. Our approach is simple, trained end-to-end in a single stage. We also&#10;explore the trade-off between effectiveness and efficiency, by controlling the&#10;contribution of the sparsity regularization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.08513" label="2108.08513">
        <attvalues>
          <attvalue for="0" value="Fast Passage Re-ranking with Contextualized Exact Term Matching and&#10;  Efficient Passage Expansion" />
          <attvalue for="1" value="  BERT-based information retrieval models are expensive, in both time (query&#10;latency) and computational resources (energy, hardware cost), making many of&#10;these models impractical especially under resource constraints. The reliance on&#10;a query encoder that only performs tokenization and on the pre-processing of&#10;passage representations at indexing, has allowed the recently proposed TILDE&#10;method to overcome the high query latency issue typical of BERT-based models.&#10;This however is at the expense of a lower effectiveness compared to other&#10;BERT-based re-rankers and dense retrievers. In addition, the original TILDE&#10;method is characterised by indexes with a very high memory footprint, as it&#10;expands each passage into the size of the BERT vocabulary. In this paper, we&#10;propose TILDEv2, a new model that stems from the original TILDE but that&#10;addresses its limitations. TILDEv2 relies on contextualized exact term matching&#10;with expanded passages. This requires to only store in the index the score of&#10;tokens that appear in the expanded passages (rather than all the vocabulary),&#10;thus producing indexes that are 99% smaller than those of TILDE. This matching&#10;mechanism also improves ranking effectiveness by 24%, without adding to the&#10;query latency. This makes TILDEv2 the state-of-the-art passage re-ranking&#10;method for CPU-only environments, capable of maintaining query latency below&#10;100ms on commodity hardware.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03073" label="2010.03073">
        <attvalues>
          <attvalue for="0" value="Beyond [CLS] through Ranking by Generation" />
          <attvalue for="1" value="  Generative models for Information Retrieval, where ranking of documents is&#10;viewed as the task of generating a query from a document's language model, were&#10;very successful in various IR tasks in the past. However, with the advent of&#10;modern deep neural networks, attention has shifted to discriminative ranking&#10;functions that model the semantic similarity of documents and queries instead.&#10;Recently, deep generative models such as GPT2 and BART have been shown to be&#10;excellent text generators, but their effectiveness as rankers have not been&#10;demonstrated yet. In this work, we revisit the generative framework for&#10;information retrieval and show that our generative approaches are as effective&#10;as state-of-the-art semantic similarity-based discriminative models for the&#10;answer selection task. Additionally, we demonstrate the effectiveness of&#10;unlikelihood losses for IR.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.18192" label="2406.18192">
        <attvalues>
          <attvalue for="0" value="Methodology of Adapting Large English Language Models for Specific&#10;  Cultural Contexts" />
          <attvalue for="1" value="  The rapid growth of large language models(LLMs) has emerged as a prominent&#10;trend in the field of artificial intelligence. However, current&#10;state-of-the-art LLMs are predominantly based on English. They encounter&#10;limitations when directly applied to tasks in specific cultural domains, due to&#10;deficiencies in domain-specific knowledge and misunderstandings caused by&#10;differences in cultural values. To address this challenge, our paper proposes a&#10;rapid adaptation method for large models in specific cultural contexts, which&#10;leverages instruction-tuning based on specific cultural knowledge and safety&#10;values data. Taking Chinese as the specific cultural context and utilizing the&#10;LLaMA3-8B as the experimental English LLM, the evaluation results demonstrate&#10;that the adapted LLM significantly enhances its capabilities in domain-specific&#10;knowledge and adaptability to safety values, while maintaining its original&#10;expertise advantages.&#10;" />
          <attvalue for="2" value="&#10;In the esteemed rankings of the global language model benchmark platform, Chatbot Arena\cite{Chiang2024ChatbotAA}, reveals that the top-performing LLMs~\cite{Claude3,achiam2023gpt,team2023gemini} are predominantly English-based. Google's LLaMA3\cite{llama3} exhibits remarkable performance improvements compared to its competitors of the same parameter scale in areas such as coding, reasoning, writing, and summarization. However, it is noteworthy that its primary application scenarios are centered around English environments. While its training data encompasses over 30 languages, the proportion of non-English multilingual data in the overall training corpus only accounts for 5\%. Given that the primary training corpus for outstanding large models is English, their intelligence in English-specific scenarios significantly surpasses other languages, posing numerous challenges and limitations when directly applying such models to specific cultural contexts.&#10;&#10;The English proficiency of LLMs significantly surpasses other languages, which is directly related to the imbalance in pre-training corpora, particularly when it concerns unique knowledge capabilities and safety values specific to various countries and regions. When users speaking non-primary training languages interact with these large models, there are often misunderstandings and even erroneous responses. Therefore, LLMs not only need to have a profound understanding of comprehensive general knowledge and possess basic capabilities such as reasoning, computation, translation, classification, and generation, but they also need to adapt to specific cultures and values unique to different countries or regions, ensuring a tailored interaction experience.&#10;&#10;In the practical application of LLMs, maintaining their superior English proficiency while ensuring alignment with specific cultural knowledge capabilities and safety values poses an urgent challenge. By observing successful adaptation cases of foreign experts in China, as illustrated in Fig.1, we gain valuable insights. Taking the example of Dr. George Hatem, a medical expert and a medical doctor from the University of Geneva, Switzerland, who arrived in China in 1933 to engage in medical research. He not only actively participated in diagnosis, treatment, and research, but he also quickly mastered Mandarin and the northern Shaanxi dialect. He successfully assisted in the establishment of the Central Skin Disease Research Institute and participated in the formulation of prevention and treatment plans for sexually transmitted diseases and leprosy, contributing his expertise to China. Dr. Hatem's case demonstrates that through targeted capability enhancement and value adjustment for specific linguistic and cultural environments, even in the context of cultural and value differences, the professional knowledge of foreign experts can be effectively utilized. This experience provides a reference for the field of LLMs: developing methods for enhancing capabilities and reshaping values that are adapted to specific cultural backgrounds, thereby efficiently optimizing existing English LLMs and enabling them to better serve users from diverse cultural backgrounds worldwide.&#10;&#10;To address the challenge of rapidly adapting English LLMs to specific cultural contexts, we propose a instruction-tuning process and methodology grounded in data related to knowledge capabilities and safety values. This approach enables rapid cultural adaptation to specific countries and regions within a short period without the need for pre-training. Taking Chinese culture as an example, we employ LLaMA3-8B as the English LLM and delve into the effectiveness of instruction-tuning strategies in promoting the model's rapid adaptation to specific cultural foundations and safety values. Our evaluation results demonstrate that the proposed method not only preserves the original superior professional knowledge of the large model but also enables rapid adaptation to the knowledge capabilities and safety values within a specific culture.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Cultural Anthropology, Artificial Intelligence, Cultural Domain Adaptation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2403.04132" label="2403.04132">
        <attvalues>
          <attvalue for="0" value="Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference" />
          <attvalue for="1" value="  Large Language Models (LLMs) have unlocked new capabilities and applications;&#10;however, evaluating the alignment with human preferences still poses&#10;significant challenges. To address this issue, we introduce Chatbot Arena, an&#10;open platform for evaluating LLMs based on human preferences. Our methodology&#10;employs a pairwise comparison approach and leverages input from a diverse user&#10;base through crowdsourcing. The platform has been operational for several&#10;months, amassing over 240K votes. This paper describes the platform, analyzes&#10;the data we have collected so far, and explains the tried-and-true statistical&#10;methods we are using for efficient and accurate evaluation and ranking of&#10;models. We confirm that the crowdsourced questions are sufficiently diverse and&#10;discriminating and that the crowdsourced human votes are in good agreement with&#10;those of expert raters. These analyses collectively establish a robust&#10;foundation for the credibility of Chatbot Arena. Because of its unique value&#10;and openness, Chatbot Arena has emerged as one of the most referenced LLM&#10;leaderboards, widely cited by leading LLM developers and companies. Our demo is&#10;publicly available at \url{https://chat.lmsys.org}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.06668" label="2311.06668">
        <attvalues>
          <attvalue for="0" value="In-context Vectors: Making In Context Learning More Effective and&#10;  Controllable Through Latent Space Steering" />
          <attvalue for="1" value="  Large language models (LLMs) demonstrate emergent in-context learning&#10;capabilities, where they adapt to new tasks based on example demonstrations.&#10;However, in-context learning has seen limited effectiveness in many settings,&#10;is difficult to quantitatively control and takes up context window space. To&#10;overcome these limitations, we propose an alternative approach that recasts&#10;in-context learning as in-context vectors (ICV). Using ICV has two steps. We&#10;first use a forward pass on demonstration examples to create the in-context&#10;vector from the latent embedding of the LLM. This vector captures essential&#10;information about the intended task. On a new query, instead of adding&#10;demonstrations to the prompt, we shift the latent states of the LLM using the&#10;ICV. The ICV approach has several benefits: 1) it enables the LLM to more&#10;effectively follow the demonstration examples; 2) it's easy to control by&#10;adjusting the magnitude of the ICV; 3) it reduces the length of the prompt by&#10;removing the in-context demonstrations; 4) ICV is computationally much more&#10;efficient than fine-tuning. We demonstrate that ICV achieves better performance&#10;compared to standard in-context learning and fine-tuning on diverse tasks&#10;including safety, style transfer, role-playing and formatting. Moreover, we&#10;show that we can flexibly teach LLM to simultaneously follow different types of&#10;instructions by simple vector arithmetics on the corresponding ICVs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.16083" label="2311.16083">
        <attvalues>
          <attvalue for="0" value="BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using&#10;  Genre Classification" />
          <attvalue for="1" value="  While performance of many text classification tasks has been recently&#10;improved due to Pre-trained Language Models (PLMs), in this paper we show that&#10;they still suffer from a performance gap when the underlying distribution of&#10;topics changes. For example, a genre classifier trained on \textit{political}&#10;topics often fails when tested on documents about \textit{sport} or&#10;\textit{medicine}. In this work, we quantify this phenomenon empirically with a&#10;large corpus and a large set of topics. Consequently, we verify that domain&#10;transfer remains challenging both for classic PLMs, such as BERT, and for&#10;modern large models, such as GPT-3. We also suggest and successfully test a&#10;possible remedy: after augmenting the training dataset with&#10;topically-controlled synthetic texts, the F1 score improves by up to 50\% for&#10;some topics, nearing on-topic training results, while others show little to no&#10;improvement. While our empirical results focus on genre classification, our&#10;methodology is applicable to other classification tasks such as gender,&#10;authorship, or sentiment classification. The code and data to replicate the&#10;experiments are available at https://github.com/dminus1/genre&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic text classification is a critical task in natural language processing, enabling proper understanding, summarization, archiving, and retrieval of documents across various domains, such as legal and medical. This task has been greatly improved due to pre-trained language models&#10;such as BERT \cite{devlin2018bert}, T5 \cite{raffel20} or GPTs \cite{brown2020language}. &#10;To achieve true artificial general intelligence (AGI), it is essential that trained computer models can recognize various document categories across different domains. However it has been noticed \cite{hendrycks20pretrained,moon2021masker} that while in general PLMs are more robust than previous models, they still suffer from spurious domain-specific clues. While all our methods proposed here apply to other non-topical text classification tasks such as sentiment or authorship identification, in this particular work we have taken a thorough look at document genre classifiers: distinguishing between different styles (genres) of texts, such as academic articles, experimental protocols, regulatory documents, and patient leaflets \cite{santini10genreintro,sharoff10lrec}. People can easily recognize document genres from just a few examples even if those examples are from a different domain \cite{crowston10}. &#10;&#10;Text classification research often contrasts the properties of topic vs. those of style \cite{dewdney01}. However, this contrast is difficult to maintain, as the training sets in most corpora for style or genre prediction are biased with respect to topics specific to individual styles or genres, so that classifiers do not transfer across corpora in case of variation between their topics. For example, a model {identifying} FAQs can learn to pay attention to such keywords as hurricane and tax advice in case these topics are common for FAQs in a specific training corpus \cite{sharoff10lrec}. &#10;&#10;So far, this cross-influence of topics and styles has not been studied in the context of PLMs such as BERT \cite{devlin2018bert}, T5 \cite{raffel20} or GPTs \cite{brown2020language}. There has also been no quantification of the gap in transferring genre/style classifiers to new domains. For instance, no study has yet assessed the performance degradation when a classifier is trained on political topics but tested on texts about sports or medicine.&#10;&#10;In light of the aforementioned challenges, our study offers the following novel contributions:&#10;\begin{itemize}[noitemsep]&#10;\item While our study primarily focuses on genre classification, the methodology we use to assess and mitigate domain transfer gaps can be broadly applied, making it suitable for other non-topical classifications such as authorship or sentiment identification; &#10;\item We have created a large corpus with ``natural genre annotation'' covering a range of topics with some biases;&#10;\item We empirically quantify the domain transfer gap on our corpus, demonstrating drops in F1 classification performance by 20-30 absolute percentage points; &#10;\item We propose a data augmentation approach which involves training text generators that can produce synthetic documents in any of the genres present in the genre training corpus and on any topic, out of those identified by neural topic-modeling algorithm \cite{dieng20topic} trained on an unrelated topically diverse large corpus. \item We verify that augmenting the training dataset with synthetics texts generated by our approach facilitates domain transfer by improving F1 classification metric by 2-6 absolute percentage points in average and on some topics as much as from 57.6 to 73.0. This improvement surpasses a general data augmentation baseline that generates synthetic documents but does not apply any domain transfer mechanisms that we propose here.&#10;\item Through ablation studies, we verify that all the components of our augmentation approach are crucial. Also, by varying hyper-parameters, we can identify the optimal augmentation setup and avoid performance degradation.&#10;\item Through a qualitative exploratory study with ChatGPT we were able to confirm that even a much larger language model can still suffer from a domain transfer gap.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Classification, Computer Science, Linguistics, Domain Transfer Solutions, Language Model Limitations, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2104.04487" label="2104.04487">
        <attvalues>
          <attvalue for="0" value="Language model fusion for streaming end to end speech recognition" />
          <attvalue for="1" value="  Streaming processing of speech audio is required for many contemporary&#10;practical speech recognition tasks. Even with the large corpora of manually&#10;transcribed speech data available today, it is impossible for such corpora to&#10;cover adequately the long tail of linguistic content that's important for tasks&#10;such as open-ended dictation and voice search. We seek to address both the&#10;streaming and the tail recognition challenges by using a language model (LM)&#10;trained on unpaired text data to enhance the end-to-end (E2E) model. We extend&#10;shallow fusion and cold fusion approaches to streaming Recurrent Neural Network&#10;Transducer (RNNT), and also propose two new competitive fusion approaches that&#10;further enhance the RNNT architecture. Our results on multiple languages with&#10;varying training set sizes show that these fusion methods improve streaming&#10;RNNT performance through introducing extra linguistic features. Cold fusion&#10;works consistently better on streaming RNNT with up to a 8.5% WER improvement.&#10;" />
          <attvalue for="2" value="&#10;&#10;End-to-end (E2E) models for automatic speech recognition (ASR) tasks have gained&#10;popularity because these models predict subword sequences from acoustic features&#10;with a single model, unlike classic ASR systems which have separate&#10;acoustic, pronunciation, and language model components. The most common E2E&#10;architectures are either attention-based (e.g., listen, attention, and spell&#10;(LAS)~\cite{chan_las_icassp2016}) or RNNT~\cite{graves_arxiv2012}&#10;models (see~\cite{prabhavalkar_is2017} for comparison).&#10;&#10;Overall, E2E models show comparable or better word error rate (WER) performance&#10;with a simplified system setup. While standard attention-based models&#10;must inspect the entire input sequence before generating outputs,&#10;streaming-friendly modifications of such models have been proposed,&#10;such as MoChA~\cite{Mocha17} and neural transducer~\cite{NeuralTransducer16}.&#10;&#10;While these approaches have shown promise, the RNNT architecture is an&#10;alternative E2E model that can natively predict output sequences on&#10;the fly (with unidirectional encoders), and thus is a natural choice&#10;for streaming applications. RNNT models have also shown extremely strong&#10;results., e.g., He et al.~\cite{he_icassp2019}&#10;presented an RNNT based real-time steaming recognizer, which outperformed classic&#10;ASR models by a wide margin.&#10;&#10;Classic ASR models leverage unpaired text data with a separately trained&#10;language model (LM) and second-pass rescoring model~\cite{biadsy_is2017},&#10;but unpaired text data cannot be easily utilized when training E2E&#10;models. Although E2E models have overall shown strong results, they&#10;have been shown to have difficulty accurately modeling tail&#10;phenomena such as proper nouns, numerics, and accented&#10;speech~\cite{Crosslingual1998,NonNative2014,Numeric2019,ProperNoun2020},&#10;due to the requirement that they be trained on paired (speech-transcript) data.&#10;&#10;Recent papers have proposed fusing E2E models with LMs trained with text data (usually referred&#10;to this as fusion), including shallow fusion~\cite{glehre_2015OnUM,kannan_icassp2018},&#10;deep fusion~\cite{glehre_2015OnUM}, cold fusion~\cite{sriram_archiv2015}, component&#10;fusion~\cite{shan_icassp2019}, etc. Most experiments used neural LMs, and some used&#10;n-gram fst LMs~\cite{chan_las_icassp2016,bahdanau_icassp2016,glehre_2015OnUM,chorowski_is2017}.&#10;See~\cite{Toshniwal2018ACO} for comparison of some of these approaches.&#10;However, these experiments were performed with standard non-streaming&#10;attention models. Fusion approaches with streaming E2E models have been&#10;unexplored, and it is unknown whether the observe gains on attention-based models&#10;could be translated to streaming models.&#10;&#10;In this paper, we explore shallow fusion, cold fusion and two new fusion approaches&#10;unique to the streaming RNNT models in section\ref{sec:methods}, we detail the experimentation&#10;across multiple languages of varying sizes of training data in section\ref{sec:setup},&#10;and we analyze the results in section\ref{sec:results}. We show that&#10;while shallow fusion worked better than cold fusion for attention-based&#10;models, cold fusion outperforms shallow fusion for&#10;RNNT models, with a WER reduction of up to 8.5\%.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Engineering, Artificial Intelligence, Speech Recognition, Language Modeling" />
        </attvalues>
      </node>
      <node id="2110.07982" label="2110.07982">
        <attvalues>
          <attvalue for="0" value="Scribosermo: Fast Speech-to-Text models for German and other Languages" />
          <attvalue for="1" value="  Recent Speech-to-Text models often require a large amount of hardware&#10;resources and are mostly trained in English. This paper presents Speech-to-Text&#10;models for German, as well as for Spanish and French with special features: (a)&#10;They are small and run in real-time on microcontrollers like a RaspberryPi. (b)&#10;Using a pretrained English model, they can be trained on consumer-grade&#10;hardware with a relatively small dataset. (c) The models are competitive with&#10;other solutions and outperform them in German. In this respect, the models&#10;combine advantages of other approaches, which only include a subset of the&#10;presented features. Furthermore, the paper provides a new library for handling&#10;datasets, which is focused on easy extension with additional datasets and shows&#10;an optimized way for transfer-learning new languages using a pretrained model&#10;from another language with a similar alphabet.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech-to-Text models based on neural networks are mostly trained in English and often require large amounts of training resources. But there exist many other languages and those who are interested in training a speech-to-text system for their own language do not always have access to high-performance server hardware. A few papers and projects focus on the aforementioned problems, but most are solving them only partially.&#10;&#10;The authors of IMS-Speech \cite{IMSDE} trained a German STT model, which so far had the best results on the German Tuda dataset \cite{TUDA}. In a comparison with Google's STT service (executed 01/2019), their network could outperform it in English as well as in German.&#10;&#10;In VoxPopuli \cite{VOXPOP}, an approach for training multilingual models using a large unlabeled dataset is investigated. A mix of 50k hours of unlabeled data in different languages from the European Parliament and a comparatively small labeled dataset for semi-supervised training are used. This approach proved very effective and achieves a Word-Error-Rate (WER) of \SI{7.8}{\percent}\,$/$\,\SI{9.6}{\percent}\,$/$\,\SI{10.0}{\percent} in German\,$/$\,Spanish\,$/$\,French on the CommonVoice datasets~\cite{COMV}, which so far have been the best results on these datasets.&#10;&#10;Luo et al. \cite{QNTECR} used the same network architecture as this work, but in Nvidia's original implementation, and also trained it for other languages like German or Spanish, using very small datasets and following a different transfer-learning approach of reinitializing the last network layer if the alphabet changes. &#10;&#10;Mozilla's DeepSpeech project \cite{DEPSPE} provides pretrained English models that are relatively small and one of the few that are able to run in real-time on a RaspberryPi. It achieves a WER of \SI{7.1}{\percent} on the LibriSpeech \cite{LIBSPE} testset. Some early experiments on multilingual trainings have been run with this network, but performance was much lower than the results presented in the following sections. They still can be found in the project's repository which is linked later.&#10;&#10;Park et al. \cite{FNNMED} built a model for embedded devices, which reached a WER of \SI{9.0}{\percent} on LibriSpeech and could run on an ARM-Cortex-A57. Zhang et al. \cite{TINTRA} trained a very small model on a large in-house Chinese dataset which can run faster than real-time on an ARMv7 chip. He et al. \cite{STRSRM} did train an English model on a very large in-house dataset which can run twice as fast than real-time on a Google-Pixel smartphone.&#10;&#10;Ghoshal et al. \cite{MLDNN} and Thomas et al. \cite{MLFDNN} did run early explorations of transfer-learning for different languages using deep neural networks. The first approach replaces the last language specific layer of a network with a new one and finetunes the whole network on the new language, while the second uses a multilingual training of the first network layers, and different output layers for each language.&#10;&#10;This paper presents a small Speech-to-Text model for German, as well as for Spanish and French, that combines the advantages of the aforementioned approaches. The main contributions of project Scribosermo are:&#10;(a) The models are competitive with the models from IMS-Speech and VoxPopuli.&#10;(b) Providing pretrained models in multiple languages that can run in real-time even on single-board computers like a RaspberryPi.&#10;(c) The models can be trained on a relatively small dataset, like the models from VoxPopuli and only require consumer-grade hardware for training.&#10;(d) Shows a fast transfer-learning approach with a single step through the concept of alphabet adaption.&#10;(e) Improved SOTA performance for German STT.&#10;&#10;Furthermore, the paper provides a new library for handling datasets, which is focused on easy extension with additional datasets and shows a simple way for transfer-learning new languages using a pretrained model from a language with an almost similar alphabet, which is demonstrated for English to Spanish and Spanish to Italian transfer-learning.&#10;&#10;The training code and models are provided as open source at: &#10;&#10;For reasons of readability, the results of the experiments are not presented in full detail, but can instead be found in the project's repository.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Speech-to-Text Models, Efficient Computing, Artificial Intelligence, Language Translation" />
        </attvalues>
      </node>
      <node id="1908.04743" label="1908.04743">
        <attvalues>
          <attvalue for="0" value="IMS-Speech: A Speech to Text Tool" />
          <attvalue for="1" value="  We present the IMS-Speech, a web based tool for German and English speech&#10;transcription aiming to facilitate research in various disciplines which&#10;require accesses to lexical information in spoken language materials. This tool&#10;is based on modern open source software stack, advanced speech recognition&#10;methods and public data resources and is freely available for academic&#10;researchers. The utilized models are built to be generic in order to provide&#10;transcriptions of competitive accuracy on a diverse set of tasks and&#10;conditions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.00390" label="2101.00390">
        <attvalues>
          <attvalue for="0" value="VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation&#10;  Learning, Semi-Supervised Learning and Interpretation" />
          <attvalue for="1" value="  We introduce VoxPopuli, a large-scale multilingual corpus providing 100K&#10;hours of unlabelled speech data in 23 languages. It is the largest open data to&#10;date for unsupervised representation learning as well as semi-supervised&#10;learning. VoxPopuli also contains 1.8K hours of transcribed speeches in 16&#10;languages and their aligned oral interpretations into 5 other languages&#10;totaling 5.1K hours. We provide speech recognition baselines and validate the&#10;versatility of VoxPopuli unlabelled data in semi-supervised learning under&#10;challenging out-of-domain settings. We will release the corpus at&#10;https://github.com/facebookresearch/voxpopuli under an open license.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.06856" label="2101.06856">
        <attvalues>
          <attvalue for="0" value="Tiny Transducer: A Highly-efficient Speech Recognition Model on Edge&#10;  Devices" />
          <attvalue for="1" value="  This paper proposes an extremely lightweight phone-based transducer model&#10;with a tiny decoding graph on edge devices. First, a phone synchronous decoding&#10;(PSD) algorithm based on blank label skipping is first used to speed up the&#10;transducer decoding process. Then, to decrease the deletion errors introduced&#10;by the high blank score, a blank label deweighting approach is proposed. To&#10;reduce parameters and computation, deep feedforward sequential memory network&#10;(DFSMN) layers are used in the transducer encoder, and a CNN-based stateless&#10;predictor is adopted. SVD technology compresses the model further. WFST-based&#10;decoding graph takes the context-independent (CI) phone posteriors as input and&#10;allows us to flexibly bias user-specific information. Finally, with only 0.9M&#10;parameters after SVD, our system could give a relative 9.1% - 20.5% improvement&#10;compared with a bigger conventional hybrid system on edge devices.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.03327" label="2110.03327">
        <attvalues>
          <attvalue for="0" value="Improving Confidence Estimation on Out-of-Domain Data for End-to-End&#10;  Speech Recognition" />
          <attvalue for="1" value="  As end-to-end automatic speech recognition (ASR) models reach promising&#10;performance, various downstream tasks rely on good confidence estimators for&#10;these systems. Recent research has shown that model-based confidence estimators&#10;have a significant advantage over using the output softmax probabilities. If&#10;the input data to the speech recogniser is from mismatched acoustic and&#10;linguistic conditions, the ASR performance and the corresponding confidence&#10;estimators may exhibit severe degradation. Since confidence models are often&#10;trained on the same in-domain data as the ASR, generalising to out-of-domain&#10;(OOD) scenarios is challenging. By keeping the ASR model untouched, this paper&#10;proposes two approaches to improve the model-based confidence estimators on OOD&#10;data: using pseudo transcriptions and an additional OOD language model. With an&#10;ASR model trained on LibriSpeech, experiments show that the proposed methods&#10;can greatly improve the confidence metrics on TED-LIUM and Switchboard datasets&#10;while preserving in-domain performance. Furthermore, the improved confidence&#10;estimators are better calibrated on OOD data and can provide a much more&#10;reliable criterion for data selection.&#10;" />
          <attvalue for="2" value="&#10;\postsec&#10;Confidence scores are an important attribute associated with speech recognisers~\cite{Wessel2001ConfidenceMF,Jiang2005ConfidenceMF,Yu2011CalibrationOC}. Various downstream tasks rely on high-quality confidence scores, such as keyword spotting, dialogue systems and active / semi-supervised learning~\cite{Chan2004ImprovingBN,Tr2005CombiningAA,Riccardi2005ActiveLT}. For hidden Markov model (HMM)-based systems, word posterior probabilities from lattices or confusion networks can provide reasonably good estimates of confidence scores~\cite{Evermann2000PosteriorPD,Mangu2000FindingCI}. In order to have more reliable estimates, many model-based approaches have been proposed~\cite{Seigel2011CombiningIS,Kalgaonkar2015EstimatingCS,DelAgua2018SpeakerAdaptedCM,Ragni2018ConfidenceEA,Li2019BidirectionalLR}. More recently, as end-to-end speech recognition systems reach competitive performance with a simplified pipeline~\cite{Li2021ABA,Tuske2021OnTL,Li2020DevelopingRM}, confidence estimation for end-to-end models has become a much-needed component of an ASR system.&#10;&#10;Because popular end-to-end systems such as recurrent neural network transducers~\cite{Graves2012SequenceTW} and attention-based encoder-decoder models~\cite{Chorowski2015AttentionBasedMF} have auto-regressive decoders that depend on the full history, it is not straightforward to generate lattice-like representations for a large number of hypotheses, from which word posteriors can be derived~\cite{Oneata2021AnEO}. Recent work has focused on data-driven approaches, where dedicated neural networks are used to predict confidence scores with various features extracted from end-to-end ASR models. For example, multi-layer perceptrons (MLPs), recurrent neural networks (RNNs) and self-attention networks have been used to learn token-level~\cite{Woodward2020ConfidenceMI,Li2021ConfidenceEF}, word-level~\cite{Qiu2021LearningWC}, and utterance-level confidence scores~\cite{Kumar2020UtteranceCM,Li2021ResidualEM,Liu2021UtterancelevelNC}. \cite{Qiu2021MultiTaskLF} proposes to jointly learn word and utterance level confidence scores together with deletion prediction via multi-task learning. With effective confidence estimation, simply rescoring $n$-best hypotheses with confidence scores can directly improve the ASR performance~\cite{Li2021ResidualEM,Qiu2021MultiTaskLF}.&#10;&#10;Although model-based confidence estimators can yield good performance on both HMM-based systems and end-to-end systems, they are normally trained on the same data as the ASR system. Therefore, it is questionable whether the confidence estimation on out-of-domain (OOD) data will be reliable~\cite{Ovadia2019CanYT}. Under mismatched acoustic and / or linguistic conditions, it may be hard for the ASR model to generalise well on unseen data. However, a reliable confidence estimator should ideally provide a good indication of the quality of the automatic transcription, even for OOD data. In this paper, assuming that the accessible OOD data is not transcribed, \ie only acoustic data and text data but they are not paired, and the ASR model is fixed, two approaches are proposed to improve OOD confidence estimation. By using automatically generated ``pseudo'' transcriptions on OOD acoustic data and features from an additional language model (LM) trained from OOD text, experiments show that the quality of confidence scores can be improved greatly on two OOD datasets while maintaining in-domain performance.&#10;&#10;In the rest of the paper, \sectdot{cem} describes the model-based confidence estimator used for end-to-end ASR model and \sectdot{ood} presents the two approaches for improving OOD confidence scores. \sect{setup} details the data, model and metrics used for producing the experimental results in \sectdot{exp}. Conclusions are given in \sectdot{conclusion}.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Out-of-Domain Generalization, Confidence Estimation, Computer Science, Linguistics, Engineering, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="cs/0010012" label="cs/0010012">
        <attvalues>
          <attvalue for="0" value="Finding consensus in speech recognition: word error minimization and&#10;  other applications of confusion networks" />
          <attvalue for="1" value="  We describe a new framework for distilling information from word lattices to&#10;improve the accuracy of speech recognition and obtain a more perspicuous&#10;representation of a set of alternative hypotheses. In the standard MAP decoding&#10;approach the recognizer outputs the string of words corresponding to the path&#10;with the highest posterior probability given the acoustics and a language&#10;model. However, even given optimal models, the MAP decoder does not necessarily&#10;minimize the commonly used performance metric, word error rate (WER). We&#10;describe a method for explicitly minimizing WER by extracting word hypotheses&#10;with the highest posterior probabilities from word lattices. We change the&#10;standard problem formulation by replacing global search over a large set of&#10;sentence hypotheses with local search over a small set of word candidates. In&#10;addition to improving the accuracy of the recognizer, our method produces a new&#10;representation of the set of candidate hypotheses that specifies the sequence&#10;of word-level confusions in a compact lattice format. We study the properties&#10;of confusion networks and examine their use for other tasks, such as lattice&#10;compression, word spotting, confidence annotation, and reevaluation of&#10;recognition hypotheses using higher-level knowledge sources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.13025" label="1810.13025">
        <attvalues>
          <attvalue for="0" value="Confidence Estimation and Deletion Prediction Using Bidirectional&#10;  Recurrent Neural Networks" />
          <attvalue for="1" value="  The standard approach to assess reliability of automatic speech&#10;transcriptions is through the use of confidence scores. If accurate, these&#10;scores provide a flexible mechanism to flag transcription errors for upstream&#10;and downstream applications. One challenging type of errors that recognisers&#10;make are deletions. These errors are not accounted for by the standard&#10;confidence estimation schemes and are hard to rectify in the upstream and&#10;downstream processing. High deletion rates are prominent in limited resource&#10;and highly mismatched training/testing conditions studied under IARPA Babel and&#10;Material programs. This paper looks at the use of bidirectional recurrent&#10;neural networks to yield confidence estimates in predicted as well as deleted&#10;words. Several simple schemes are examined for combination. To assess&#10;usefulness of this approach, the combined confidence score is examined for&#10;untranscribed data selection that favours transcriptions with lower deletion&#10;errors. Experiments are conducted using IARPA Babel/Material program languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.13024" label="1810.13024">
        <attvalues>
          <attvalue for="0" value="Bi-Directional Lattice Recurrent Neural Networks for Confidence&#10;  Estimation" />
          <attvalue for="1" value="  The standard approach to mitigate errors made by an automatic speech&#10;recognition system is to use confidence scores associated with each predicted&#10;word. In the simplest case, these scores are word posterior probabilities&#10;whilst more complex schemes utilise bi-directional recurrent neural network&#10;(BiRNN) models. A number of upstream and downstream applications, however, rely&#10;on confidence scores assigned not only to 1-best hypotheses but to all words&#10;found in confusion networks or lattices. These include but are not limited to&#10;speaker adaptation, semi-supervised training and information retrieval.&#10;Although word posteriors could be used in those applications as confidence&#10;scores, they are known to have reliability issues. To make improved confidence&#10;scores more generally available, this paper shows how BiRNNs can be extended&#10;from 1-best sequences to confusion network and lattice structures. Experiments&#10;are conducted using one of the Cambridge University submissions to the IARPA&#10;OpenKWS 2016 competition. The results show that confusion network and&#10;lattice-based BiRNNs can provide a significant improvement in confidence&#10;estimation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00982" label="2105.00982">
        <attvalues>
          <attvalue for="0" value="On the limit of English conversational speech recognition" />
          <attvalue for="1" value="  In our previous work we demonstrated that a single headed attention&#10;encoder-decoder model is able to reach state-of-the-art results in&#10;conversational speech recognition. In this paper, we further improve the&#10;results for both Switchboard 300 and 2000. Through use of an improved&#10;optimizer, speaker vector embeddings, and alternative speech representations we&#10;reduce the recognition errors of our LSTM system on Switchboard-300 by 4%&#10;relative. Compensation of the decoder model with the probability ratio approach&#10;allows more efficient integration of an external language model, and we report&#10;5.9% and 11.5% WER on the SWB and CHM parts of Hub5'00 with very simple LSTM&#10;models. Our study also considers the recently proposed conformer, and more&#10;advanced self-attention based language models. Overall, the conformer shows&#10;similar performance to the LSTM; nevertheless, their combination and decoding&#10;with an improved LM reaches a new record on Switchboard-300, 5.0% and 10.0% WER&#10;on SWB and CHM. Our findings are also confirmed on Switchboard-2000, and a new&#10;state of the art is reported, practically reaching the limit of the benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.06716" label="2103.06716">
        <attvalues>
          <attvalue for="0" value="Learning Word-Level Confidence For Subword End-to-End ASR" />
          <attvalue for="1" value="  We study the problem of word-level confidence estimation in subword-based&#10;end-to-end (E2E) models for automatic speech recognition (ASR). Although prior&#10;works have proposed training auxiliary confidence models for ASR systems, they&#10;do not extend naturally to systems that operate on word-pieces (WP) as their&#10;vocabulary. In particular, ground truth WP correctness labels are needed for&#10;training confidence models, but the non-unique tokenization from word to WP&#10;causes inaccurate labels to be generated. This paper proposes and studies two&#10;confidence models of increasing complexity to solve this problem. The final&#10;model uses self-attention to directly learn word-level confidence without&#10;needing subword tokenization, and exploits full context features from multiple&#10;hypotheses to improve confidence accuracy. Experiments on Voice Search and&#10;long-tail test sets show standard metrics (e.g., NCE, AUC, RMSE) improving&#10;substantially. The proposed confidence module also enables a model selection&#10;approach to combine an on-device E2E model with a hybrid model on the server to&#10;address the rare word recognition problem for the E2E model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07750" label="2109.07750">
        <attvalues>
          <attvalue for="0" value="Utterance-level neural confidence measure for end-to-end children speech&#10;  recognition" />
          <attvalue for="1" value="  Confidence measure is a performance index of particular importance for&#10;automatic speech recognition (ASR) systems deployed in real-world scenarios. In&#10;the present study, utterance-level neural confidence measure (NCM) in&#10;end-to-end automatic speech recognition (E2E ASR) is investigated. The E2E&#10;system adopts the joint CTC-attention Transformer architecture. The prediction&#10;of NCM is formulated as a task of binary classification, i.e., accept/reject&#10;the input utterance, based on a set of predictor features acquired during the&#10;ASR decoding process. The investigation is focused on evaluating and comparing&#10;the efficacies of predictor features that are derived from different internal&#10;and external modules of the E2E system. Experiments are carried out on children&#10;speech, for which state-of-the-art ASR systems show less than satisfactory&#10;performance and robust confidence measure is particularly useful. It is noted&#10;that predictor features related to acoustic information of speech play a more&#10;important role in estimating confidence measure than those related to&#10;linguistic information. N-best score features show significantly better&#10;performance than single-best ones. It has also been shown that the metrics of&#10;EER and AUC are not appropriate to evaluate the NCM of a mismatched ASR with&#10;significant performance gap.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.12870" label="2104.12870">
        <attvalues>
          <attvalue for="0" value="Multi-Task Learning for End-to-End ASR Word and Utterance Confidence&#10;  with Deletion Prediction" />
          <attvalue for="1" value="  Confidence scores are very useful for downstream applications of automatic&#10;speech recognition (ASR) systems. Recent works have proposed using neural&#10;networks to learn word or utterance confidence scores for end-to-end ASR. In&#10;those studies, word confidence by itself does not model deletions, and&#10;utterance confidence does not take advantage of word-level training signals.&#10;This paper proposes to jointly learn word confidence, word deletion, and&#10;utterance confidence. Empirical results show that multi-task learning with all&#10;three objectives improves confidence metrics (NCE, AUC, RMSE) without the need&#10;for increasing the model size of the confidence estimation module. Using the&#10;utterance-level confidence for rescoring also decreases the word error rates on&#10;Google's Voice Search and Long-tail Maps datasets by 3-5% relative, without&#10;needing a dedicated neural rescorer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.02530" label="1906.02530">
        <attvalues>
          <attvalue for="0" value="Can You Trust Your Model's Uncertainty? Evaluating Predictive&#10;  Uncertainty Under Dataset Shift" />
          <attvalue for="1" value="  Modern machine learning methods including deep learning have achieved great&#10;success in predictive accuracy for supervised learning tasks, but may still&#10;fall short in giving useful estimates of their predictive {\em uncertainty}.&#10;Quantifying uncertainty is especially critical in real-world settings, which&#10;often involve input distributions that are shifted from the training&#10;distribution due to a variety of factors including sample bias and&#10;non-stationarity. In such settings, well calibrated uncertainty estimates&#10;convey information about when a model's output should (or should not) be&#10;trusted. Many probabilistic deep learning methods, including Bayesian-and&#10;non-Bayesian methods, have been proposed in the literature for quantifying&#10;predictive uncertainty, but to our knowledge there has not previously been a&#10;rigorous large-scale empirical comparison of these methods under dataset shift.&#10;We present a large-scale benchmark of existing state-of-the-art methods on&#10;classification problems and investigate the effect of dataset shift on accuracy&#10;and calibration. We find that traditional post-hoc calibration does indeed fall&#10;short, as do several other previous methods. However, some methods that&#10;marginalize over models give surprisingly strong results across a broad&#10;spectrum of tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00573" label="2105.00573">
        <attvalues>
          <attvalue for="0" value="Searchable Hidden Intermediates for End-to-End Models of Decomposable&#10;  Sequence Tasks" />
          <attvalue for="1" value="  End-to-end approaches for sequence tasks are becoming increasingly popular.&#10;Yet for complex sequence tasks, like speech translation, systems that cascade&#10;several models trained on sub-tasks have shown to be superior, suggesting that&#10;the compositionality of cascaded systems simplifies learning and enables&#10;sophisticated search capabilities. In this work, we present an end-to-end&#10;framework that exploits compositionality to learn searchable hidden&#10;representations at intermediate stages of a sequence model using decomposed&#10;sub-tasks. These hidden intermediates can be improved using beam search to&#10;enhance the overall performance and can also incorporate external models at&#10;intermediate stages of the network to re-score or adapt towards out-of-domain&#10;data. One instance of the proposed framework is a Multi-Decoder model for&#10;speech translation that extracts the searchable hidden intermediates from a&#10;speech recognition sub-task. The model demonstrates the aforementioned benefits&#10;and outperforms the previous state-of-the-art by around +6 and +3 BLEU on the&#10;two test sets of Fisher-CallHome and by around +3 and +4 BLEU on the&#10;English-German and English-French test sets of MuST-C.&#10;" />
          <attvalue for="2" value="&#10;&#10;The principle of compositionality loosely states that a complex whole is composed of its parts and the rules by which those parts are combined \cite{lake2018generalization}. This principle is present in engineering, where task decomposition of a complex system is required to assess and optimize task allocations \cite{taskdecomp}, and in natural language, where paragraph coherence and discourse analysis rely on decomposition into sentences \cite{johnson1992, kuo1995} and sentence level semantics relies on decomposition into lexical units \cite{Liu2020}. &#10;&#10;Similarly, many sequence-to-sequence tasks that convert one sequence into another \cite{sutskever2014sequence} can be decomposed to simpler sequence sub-tasks in order to reduce the overall complexity. &#10;For example, speech translation systems, which seek to process speech in one language and output text in another language, can be naturally decomposed into the transcription of source language audio through automatic speech recognition (ASR) and translation into the target language through machine translation (MT). Such cascaded approaches have been widely used to build practical systems for a variety of sequence tasks like hybrid ASR \cite{hinton2012deep}, phrase-based MT \cite{koehn-etal-2007-moses}, and cascaded ASR-MT systems for speech translation (ST)~\cite{Pham2019TheI2}.&#10;&#10;End-to-end sequence models like encoder-decoder models \cite{bahdanau2014neural, vaswani2017attention}, are attractive in part due to their simplistic design and the reduced need for hand-crafted features. However, studies have shown mixed results compared to cascaded models particularly for complex sequence tasks like speech translation \cite{inaguma-etal-2020-espnet-st} and spoken language understanding \cite{coucke2018snips}. Although direct target sequence prediction avoids the issue of error propagation from one system to another in cascaded approaches \cite{tzoukermann-miller-2018-evaluating-error}, there are many attractive properties of cascaded systems, missing in end-to-end approaches, that are useful in complex sequence tasks. &#10;&#10;In particular, we are interested in (1) the strong search capabilities of the cascaded systems that compose the final task output from individual system predictions \cite{mohri2002weighted, kumar2006weighted, beck-etal-2019-neural-lattice}, (2) the ability to incorporate external models to re-score each individual system \cite{och2002discriminative, huang2007forest}, (3) the ability to easily adapt individual components towards out-of-domain data \cite{koehn2007experiments, peddinti2015jhu}, and finally (4) the ability to monitor performance of the individual systems towards the decomposed sub-task \cite{tillmann2003word, meyer2016performance}.&#10;&#10;In this paper, we seek to incorporate these properties of cascaded systems into end-to-end sequence models. We first propose a generic framework to learn searchable hidden intermediates using an auto-regressive encoder-decoder model for any decomposable sequence task (\S\ref{sec:methods}). We then apply this approach to speech translation, where the intermediate stage is the output of ASR, by passing continuous hidden representations of discrete transcript sequences from the ASR sub-net decoder to the MT sub-net encoder. By doing so, we gain the ability to use beam search with optional external model re-scoring on the hidden intermediates, while maintaining end-to-end differentiability. Next, we suggest mitigation strategies for the error propagation issues inherited from decomposition. &#10;&#10;We show the efficacy of searchable intermediate representations in our proposed model, called the Multi-Decoder, on speech translation with a 5.4 and 2.8 BLEU score improvement over the previous state-of-the-arts for Fisher and CallHome test sets respectively (\S\ref{sec:results}). We extend these improvements by an average of 0.5 BLEU score through the aforementioned benefit of re-scoring the intermediate search with external models trained on the same dataset. We also show a method for monitoring sub-net performance using oracle intermediates that are void of search errors (\S\ref{sec:benefits}). Finally, we show how these models can adapt to out-of-domain speech translation datasets, how our approach can be generalized to other sequence tasks like speech recognition, and how the benefits of decomposition persist even for larger corpora like MuST-C (\S\ref{sec:gen}).&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Learning Architectures, Sequence Task Modeling, Speech Translation Systems, Artificial Intelligence, Signal Processing" />
        </attvalues>
      </node>
      <node id="2004.10234" label="2004.10234">
        <attvalues>
          <attvalue for="0" value="ESPnet-ST: All-in-One Speech Translation Toolkit" />
          <attvalue for="1" value="  We present ESPnet-ST, which is designed for the quick development of&#10;speech-to-speech translation systems in a single framework. ESPnet-ST is a new&#10;project inside end-to-end speech processing toolkit, ESPnet, which integrates&#10;or newly implements automatic speech recognition, machine translation, and&#10;text-to-speech functions for speech translation. We provide all-in-one recipes&#10;including data pre-processing, feature extraction, training, and decoding&#10;pipelines for a wide range of benchmark datasets. Our reproducible results can&#10;match or even outperform the current state-of-the-art performances; these&#10;pre-trained models are downloadable. The toolkit is publicly available at&#10;https://github.com/espnet/espnet.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.13116" label="2308.13116">
        <attvalues>
          <attvalue for="0" value="Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge&#10;  Distillation" />
          <attvalue for="1" value="  Contextual language models have been trained on Classical languages,&#10;including Ancient Greek and Latin, for tasks such as lemmatization,&#10;morphological tagging, part of speech tagging, authorship attribution, and&#10;detection of scribal errors. However, high-quality sentence embedding models&#10;for these historical languages are significantly more difficult to achieve due&#10;to the lack of training data. In this work, we use a multilingual knowledge&#10;distillation approach to train BERT models to produce sentence embeddings for&#10;Ancient Greek text. The state-of-the-art sentence embedding approaches for&#10;high-resource languages use massive datasets, but our distillation approach&#10;allows our Ancient Greek models to inherit the properties of these models while&#10;using a relatively small amount of translated sentence data. We build a&#10;parallel sentence dataset using a sentence-embedding alignment method to align&#10;Ancient Greek documents with English translations, and use this dataset to&#10;train our models. We evaluate our models on translation search, semantic&#10;similarity, and semantic retrieval tasks and investigate translation bias. We&#10;make our training and evaluation datasets freely available at&#10;https://github.com/kevinkrahn/ancient-greek-datasets .&#10;" />
          <attvalue for="2" value="&#10;&#10;Sentence embedding models, which map sentences or other sequences of text to a dense vector space, such that&#10;semantically similar sentences are close together in the vector space, have many applications&#10;in NLP. Current state-of-the-art sentence embedding models, however, are trained on modern, high-resource languages&#10;such as English and use massive datasets consisting of billions of sentence pairs \cite{ni-etal-2022-sentence}. A different&#10;approach is needed for historical languages, which have much less data available.&#10;&#10;In this work, we train several sentence embedding models for Ancient Greek.&#10;Many more Ancient Greek texts have survived compared to texts from most other&#10;ancient languages, which makes sentence embedding models both more feasible and useful.&#10;&#10;Several previous works have trained language models for Ancient Greek. \cite{johnson-etal-2021-classical}&#10;introduced the Classical Language Toolkit (CLTK) which includes several tools for Ancient Greek processing,&#10;including static word embeddings. \cite{singh-etal-2021-pilot} fine-tuned a Modern Greek BERT model&#10;\cite{koutsikakisGREEKBERTGreeksVisiting2020} on Ancient Greek text for PoS tagging, morphological tagging, and lemmatization tasks. \cite{yamshchikov-etal-2022-bert}&#10;trained a BERT model for authorship classification of Pseudo-Plutarch texts. \cite{cowen-breenLogionMachineLearning2023}&#10;trained another BERT model for the purpose of identifying errors in scribal transmission.&#10;\cite{riemenschneiderExploringLargeLanguage2023} produced the most comprehensive work on&#10;Classical language models to date, training multiple models on a large multilingual corpus of&#10;Ancient Greek, Latin, and English texts and comprehensively evaluating and comparing&#10;their new models to previous models on a variety of tasks. None of these works, however,&#10;produce sentence embedding models for Ancient Greek.&#10;&#10;Although there are many digitized Ancient Greek texts available, there is a lack of suitable training&#10;data for training sentence embedding models from scratch. The best approaches for high-resource languages&#10;involve large human-annotated datasets, such as the natural language inference (NLI) datasets used&#10;by Sentence-BERT \cite{reimers-gurevych-2019-sentence}.&#10;Needless to say, such datasets are not available for Ancient Greek.&#10;&#10;Following \cite{reimers-gurevych-2020-making}, we use multilingual knowledge distillation&#10;to train sentence embedding models with an aligned vector space for Ancient Greek and English.&#10;Given a teacher model \(M\) for a language \(s\), and a dataset&#10;of translated sentences \(((s_{1},t_{1})..(s_{n},t_{n}))\) where \(s_{i}\) and \(t_{i}\) are parallel sentences,&#10;we train a new student model \(\hat{M}\) to mimic the sentence embeddings of the teacher \(M\) using mean squared loss,&#10;such that \(\hat{M}(s_i) \approx M(t_i)\) and \(\hat{M}(t_i) \approx M(s_i)\). In our case,&#10;the teacher model is English and the student model learns both&#10;Greek&#10;and English embeddings.&#10;&#10;This approach has numerous advantages:&#10;1) it requires a relatively small amount of training data,&#10;2) the student model inherits the vector space properties of a state-of-the-art English sentence embedding model,&#10;3) the student model is multilingual, and&#10;4) the vector spaces are aligned across languages.&#10;&#10;The cross-lingual nature of this approach is especially useful for Ancient Greek semantic retrieval,&#10;since it is much easier to formulate search queries in English than in Ancient Greek.&#10;Although it is possible to operate on the English translations of Greek texts,&#10;translations are not readily available for all Greek texts, and the available translations are&#10;usually not aligned at the sentence level, making it difficult to quickly&#10;find the corresponding Greek text.&#10;Furthermore, English translations can suffer from various kinds of translator bias,&#10;whereas a language model that operates directly on the Greek text can offer an ``average'' of multiple translators'&#10;interpretations of the text (See Section \ref{sec:bias}).&#10;&#10;We produce a training dataset of parallel sentences using a two-step translation alignment process:&#10;an initial, smaller dataset was produced using a&#10;sentence-length heuristic and dictionary-based alignment technique \cite{halacsyParallelCorporaMedium2007},&#10;and this initial dataset was used to train an intermediate multilingual sentence embedding model, which was used to align a larger dataset&#10;using the approach introduced by \cite{liuBertalignImprovedWord2023}, which uses&#10;sentence embeddings for state-of-the-art alignment quality.&#10;&#10;We create new evaluation datasets for Ancient Greek translation search,&#10;semantic textual similarity (STS), and semantic retrieval (SR)&#10;and we evaluate our models on these datasets.&#10;&#10;In summary, our contributions are as follows:&#10;\begin{enumerate}&#10; \item We use a multilingual knowledge distillation approach to train several Ancient Greek sentence embedding models.&#10; \item We use translation alignment to produce a dataset of Ancient Greek sentences and their English translations.&#10; \item We develop evaluation datasets for translation search, semantic retrieval, and semantic textual similarity, and we evaluate&#10; our sentence embedding models on these tasks.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Classics, Computer Science, Linguistics, Historical Text Analysis, Artificial Intelligence, Natural Language Processing, Language Modeling, Multilingual Knowledge" />
        </attvalues>
      </node>
      <node id="2108.08877" label="2108.08877">
        <attvalues>
          <attvalue for="0" value="Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text&#10;  Models" />
          <attvalue for="1" value="  We provide the first exploration of sentence embeddings from text-to-text&#10;transformers (T5). Sentence embeddings are broadly useful for language&#10;processing tasks. While T5 achieves impressive performance on language tasks&#10;cast as sequence-to-sequence mapping problems, it is unclear how to produce&#10;sentence embeddings from encoder-decoder models. We investigate three methods&#10;for extracting T5 sentence embeddings: two utilize only the T5 encoder and one&#10;uses the full T5 encoder-decoder model. To support our investigation, we&#10;establish a new sentence representation transfer benchmark, SentGLUE, which&#10;extends the SentEval toolkit to nine tasks from the GLUE benchmark. Our&#10;encoder-only models outperforms Sentence-BERT and SimCSE sentence embeddings on&#10;both SentEval and SentGLUE transfer tasks, including semantic textual&#10;similarity (STS). Scaling up T5 from millions to billions of parameters is&#10;found to produce consistent further improvements. Finally, our encoder-decoder&#10;method achieves a new state-of-the-art on STS when using sentence embeddings.&#10;Our models are released at https://tfhub.dev/google/collections/sentence-t5/1.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.05673" label="2211.05673">
        <attvalues>
          <attvalue for="0" value="BERT in Plutarch's Shadows" />
          <attvalue for="1" value="  The extensive surviving corpus of the ancient scholar Plutarch of Chaeronea&#10;(ca. 45-120 CE) also contains several texts which, according to current&#10;scholarly opinion, did not originate with him and are therefore attributed to&#10;an anonymous author Pseudo-Plutarch. These include, in particular, the work&#10;Placita Philosophorum (Quotations and Opinions of the Ancient Philosophers),&#10;which is extremely important for the history of ancient philosophy. Little is&#10;known about the identity of that anonymous author and its relation to other&#10;authors from the same period. This paper presents a BERT language model for&#10;Ancient Greek. The model discovers previously unknown statistical properties&#10;relevant to these literary, philosophical, and historical problems and can shed&#10;new light on this authorship question. In particular, the Placita&#10;Philosophorum, together with one of the other Pseudo-Plutarch texts, shows&#10;similarities with the texts written by authors from an Alexandrian context&#10;(2nd/3rd century CE).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.01099" label="2305.01099">
        <attvalues>
          <attvalue for="0" value="Logion: Machine Learning for Greek Philology" />
          <attvalue for="1" value="  This paper presents machine-learning methods to address various problems in&#10;Greek philology. After training a BERT model on the largest premodern Greek&#10;dataset used for this purpose to date, we identify and correct previously&#10;undetected errors made by scribes in the process of textual transmission, in&#10;what is, to our knowledge, the first successful identification of such errors&#10;via machine learning. Additionally, we demonstrate the model's capacity to fill&#10;gaps caused by material deterioration of premodern manuscripts and compare the&#10;model's performance to that of a domain expert. We find that best performance&#10;is achieved when the domain expert is provided with model suggestions for&#10;inspiration. With such human-computer collaborations in mind, we explore the&#10;model's interpretability and find that certain attention heads appear to encode&#10;select grammatical features of premodern Greek.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11791" label="2305.11791">
        <attvalues>
          <attvalue for="0" value="Enhancing Few-shot NER with Prompt Ordering based Data Augmentation" />
          <attvalue for="1" value="  Recently, data augmentation (DA) methods have been proven to be effective for&#10;pre-trained language models (PLMs) in low-resource settings, including few-shot&#10;named entity recognition (NER). However, conventional NER DA methods are mostly&#10;aimed at sequence labeling models, i.e., token-level classification, and few&#10;are compatible with unified autoregressive generation frameworks, which can&#10;handle a wider range of NER tasks, such as nested NER. Furthermore, these&#10;generation frameworks have a strong assumption that the entities will appear in&#10;the target sequence with the same left-to-right order as the source sequence.&#10;In this paper, we claim that there is no need to keep this strict order, and&#10;more diversified but reasonable target entity sequences can be provided during&#10;the training stage as a novel DA method. Nevertheless, a naive mixture of&#10;augmented data can confuse the model since one source sequence will then be&#10;paired with different target sequences. Therefore, we propose a simple but&#10;effective Prompt Ordering based Data Augmentation (PODA) method to improve the&#10;training of unified autoregressive generation frameworks under few-shot NER&#10;scenarios. Experimental results on three public NER datasets and further&#10;analyses demonstrate the effectiveness of our approach.&#10;" />
          <attvalue for="2" value=" &#10;Named entity recognition (NER)~\cite{tjong2003introduction, doddington-etal-2004-automatic-2} has been a long-standing and one of the most important fundamental tasks in natural language processing (NLP). Existing NER models can be divided into three different categories, including sequence labeling methods~\cite{lample2016neural,devlin2019bert}, span-level classification~\cite{wang-lu-2020-two-2, zhong-chen-2021-frustratingly-2} and generation-based methods~\cite{yan-etal-2021-unified-generative-2, lu-etal-2022-unified-2}. However, even with pre-trained language models (PLMs), training these state-of-the-art named entity recognizers requires sufficient training samples, which is in contrast with real-world scenarios, where only small amounts of labeled data are available. This draws our attention to the challenging but practical problem: few-shot NER.&#10;&#10;By introducing more samples in the training stage, data augmentation (DA) methods have been proven to be effective solutions in low-resource settings, including few-shot NER~\cite{chen-etal-2020-local-2, zhou-etal-2022-melm-2, chen-etal-2021-data-2}. However, these approaches are mostly designed in a token-level classification style. So they must be combined with different tagging schemes or special-designed structures before they can be applied to other NER subtasks, such as nested NER. On the other hand, generation-based models~\cite{yan-etal-2021-unified-generative-2, Paolini2021StructuredPA} can overcome this limitation by leveraging generative PLMs and introducing a unified tagging strategy. However, few efforts are made on DA over generative PLMs (e.g. BART~\cite{lewis-etal-2020-bart-2}, T5~\cite{Raffel2019ExploringTL}), so limited data resources will lead to weakly fine-tuning of these generation-based methods. Hence, developing a DA approach that can easily be compatible with these generative PLMs would be worthwhile.&#10;&#10;Due to the autoregressive decoding of generative PLMs, vanilla generation-based NER methods have a strong assumption that the entities will appear in the target sequence with the same left-to-right order as the source sequence. For example, in Figure \ref{fig:model}, there are 4 entities in the input sentence. The prediction of entity EU-wide will be strictly after that of entity Fischler, following their order of appearing in the input sentence. However, in the NER task, the output entities are essentially forming an unordered set. To mitigate the above-mentioned challenge, significant efforts have been made from different aspects. \cite{Tan2021ASN} proposed a sequence-to-set network and relied mainly on non-autoregressive generation~\cite{gu2018nonautoregressive}. Although they were able to predict the entities as a set, they might suffer from uncertain boundaries, and non-autoregressive generation may also lead to tremendous search space. \cite{zhang-etal-2022-de-2} tried to address this issue still in an autoregressive perspective and constructed augmented samples based on the entities' context and positional orders. However, a simple mixture of these target sequences can confuse the model since there will be several ``gold'' target sequences corresponding to a same source sequence, which will result in a one-to-many mapping problem (also known as multimodality problem~\cite{gu2018nonautoregressive}), especially harmful in few-shot NER settings.&#10;&#10;In this work, we try to fully utilize the order-agnostic property of NER, and propose a simple but effective Prompt Ordering based Data Augmentation (PODA) method for few-shot NER. In our view, any sequence containing complete information (i.e. every entity's mention and its type) should be regarded reasonable and can serve as an augmented target sequence. With the help of different prompt-based order instructions, we separate the original one-to-many mapping into various one-to-one mappings. As shown in Figure \ref{fig:model}, following a certain entity type permutation like ``PER, MISC, LOC, ORG'', a unique target sequence will then be constructed, and its related source sequence will be the combination of source sentence and the order instruction. In this way, the strict left-to-right order does not need to be maintained. &#10;&#10;In summary, our contributions include: (1) We for the first time propose a new data augmentation method which can be uniformly applied over several generative PLMs. Furthermore, we combine our augmented data with prompt-based order instructions to prevent one-to-many mapping problem; (2) Experiments over three benchmark NER datasets, including flat and nested NER, demonstrate the effectiveness of our data augmentation method. Further analyses show the strong generalization ability of our method and validity of our augmented data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Training, Named Entity Recognition, Artificial Intelligence, Data Augmentation Methods" />
        </attvalues>
      </node>
      <node id="1603.01360" label="1603.01360">
        <attvalues>
          <attvalue for="0" value="Neural Architectures for Named Entity Recognition" />
          <attvalue for="1" value="  State-of-the-art named entity recognition systems rely heavily on&#10;hand-crafted features and domain-specific knowledge in order to learn&#10;effectively from the small, supervised training corpora that are available. In&#10;this paper, we introduce two new neural architectures---one based on&#10;bidirectional LSTMs and conditional random fields, and the other that&#10;constructs and labels segments using a transition-based approach inspired by&#10;shift-reduce parsers. Our models rely on two sources of information about&#10;words: character-based word representations learned from the supervised corpus&#10;and unsupervised word representations learned from unannotated corpora. Our&#10;models obtain state-of-the-art performance in NER in four languages without&#10;resorting to any language-specific knowledge or resources such as gazetteers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.03851" label="2010.03851">
        <attvalues>
          <attvalue for="0" value="Two are Better than One: Joint Entity and Relation Extraction with&#10;  Table-Sequence Encoders" />
          <attvalue for="1" value="  Named entity recognition and relation extraction are two important&#10;fundamental problems. Joint learning algorithms have been proposed to solve&#10;both tasks simultaneously, and many of them cast the joint task as a&#10;table-filling problem. However, they typically focused on learning a single&#10;encoder (usually learning representation in the form of a table) to capture&#10;information required for both tasks within the same space. We argue that it can&#10;be beneficial to design two distinct encoders to capture such two different&#10;types of information in the learning process. In this work, we propose the&#10;novel {\em table-sequence encoders} where two different encoders -- a table&#10;encoder and a sequence encoder are designed to help each other in the&#10;representation learning process. Our experiments confirm the advantages of&#10;having {\em two} encoders over {\em one} encoder. On several standard datasets,&#10;our model shows significant improvements over existing approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.01677" label="2010.01677">
        <attvalues>
          <attvalue for="0" value="Local Additivity Based Data Augmentation for Semi-supervised NER" />
          <attvalue for="1" value="  Named Entity Recognition (NER) is one of the first stages in deep language&#10;understanding yet current NER models heavily rely on human-annotated data. In&#10;this work, to alleviate the dependence on labeled data, we propose a Local&#10;Additivity based Data Augmentation (LADA) method for semi-supervised NER, in&#10;which we create virtual samples by interpolating sequences close to each other.&#10;Our approach has two variations: Intra-LADA and Inter-LADA, where Intra-LADA&#10;performs interpolations among tokens within one sentence, and Inter-LADA&#10;samples different sentences to interpolate. Through linear additions between&#10;sampled training data, LADA creates an infinite amount of labeled data and&#10;improves both entity and context learning. We further extend LADA to the&#10;semi-supervised setting by designing a novel consistency loss for unlabeled&#10;data. Experiments conducted on two NER benchmarks demonstrate the effectiveness&#10;of our methods over several strong baselines. We have publicly released our&#10;code at https://github.com/GT-SALT/LADA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.08901" label="2105.08901">
        <attvalues>
          <attvalue for="0" value="A Sequence-to-Set Network for Nested Named Entity Recognition" />
          <attvalue for="1" value="  Named entity recognition (NER) is a widely studied task in natural language&#10;processing. Recently, a growing number of studies have focused on the nested&#10;NER. The span-based methods, considering the entity recognition as a span&#10;classification task, can deal with nested entities naturally. But they suffer&#10;from the huge search space and the lack of interactions between entities. To&#10;address these issues, we propose a novel sequence-to-set neural network for&#10;nested NER. Instead of specifying candidate spans in advance, we provide a&#10;fixed set of learnable vectors to learn the patterns of the valuable spans. We&#10;utilize a non-autoregressive decoder to predict the final set of entities in&#10;one pass, in which we are able to capture dependencies between entities.&#10;Compared with the sequence-to-sequence method, our model is more suitable for&#10;such unordered recognition task as it is insensitive to the label order. In&#10;addition, we utilize the loss function based on bipartite matching to compute&#10;the overall training loss. Experimental results show that our proposed model&#10;achieves state-of-the-art on three nested NER corpora: ACE 2004, ACE 2005 and&#10;KBP 2017. The code is available at&#10;https://github.com/zqtan1024/sequence-to-set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.05997" label="2409.05997">
        <attvalues>
          <attvalue for="0" value="TransformerRanker: A Tool for Efficiently Finding the Best-Suited&#10;  Language Models for Downstream Classification Tasks" />
          <attvalue for="1" value="  Classification tasks in NLP are typically addressed by selecting a&#10;pre-trained language model (PLM) from a model hub, and fine-tuning it for the&#10;task at hand. However, given the very large number of PLMs that are currently&#10;available, a practical challenge is to determine which of them will perform&#10;best for a specific downstream task. With this paper, we introduce&#10;TransformerRanker, a lightweight library that efficiently ranks PLMs for&#10;classification tasks without the need for computationally costly fine-tuning.&#10;Our library implements current approaches for transferability estimation&#10;(LogME, H-Score, kNN), in combination with layer aggregation options, which we&#10;empirically showed to yield state-of-the-art rankings of PLMs (Garbas et al.,&#10;2024). We designed the interface to be lightweight and easy to use, allowing&#10;users to directly connect to the HuggingFace Transformers and Dataset&#10;libraries. Users need only select a downstream classification task and a list&#10;of PLMs to create a ranking of likely best-suited PLMs for their task. We make&#10;TransformerRanker available as a pip-installable open-source library&#10;https://github.com/flairNLP/transformer-ranker.&#10;" />
          <attvalue for="2" value="&#10;&#10;There currently exists a multitude of pre-trained transformer language models (PLMs) that are&#10;readily available \cite[e.g. through model hubs;][]{wolf-huggingface}. From a practical perspective, this raises the question of which PLM will perform best once fine-tuned for a specific downstream NLP task. However, &#10;since fine-tuning a PLM is both computationally costly and sensitive to hyperparameters (such as the learning rate used for fine-tuning), an exhaustive search of all models is infeasible. In practice, this restricts users to the exploration of only a small number of PLMs and may lead to the best-suited PLM for a particular task not being found. &#10;&#10;To address this issue, prior work proposed methods for transferability estimation. These methods avoid the high computational costs associated with fine-tuning a PLM by keeping the internal states frozen. Prominent examples of such methods include H-score \cite{baoInformationTheoreticApproachTransferability2019,hscore2023} and LogME \cite{logme2021}. In our prior work, we found that these methods can be improved by incorporating features from deeper layers in the estimation process~\cite{garbas2024choose}. We empirically showed that this yields better per-model estimates since, depending on the downstream task, different layers in the transformer model are best suited to provide features. Further, we showed that averaging across layers makes the selection process more robust against the different pre-training objectives used in each model, allowing a better comparison over a diverse set of PLMs.&#10;&#10;A library for transferability estimation. With this paper, we present TransformerRanker, a Python library that enables users to leverage transferability estimation to identify the best-suited PLM for a downstream classification task. With TransformerRanker, we consolidate work in layer-wise analysis and transferability estimation methods into a single library and provide a three-step interface for ranking any transformer LMs available on the HuggingFace model hub (see Figure~\ref{fig:process-illustration}). Our goal is twofold: &#10;&#10;\begin{itemize}[left=0.82em]&#10;&#10;\item First, to give practitioners an easy-to-use method of using transferability estimation to select PLMs for their downstream tasks. To this end, we designed a simple interface that directly connects to the&#10;HuggingFace transformers and datasets libraries. From these, users need only select a downstream classification task and a list of PLMs. Using our default settings, the library will output a ranking of likely best-suited PLMs for their task.&#10;&#10;\item Second, to assist researchers in the field of transferability estimation by having a single library that implements multiple state-of-the-art estimators and aggregation methods to combine and compare against. &#10;\end{itemize}&#10;&#10;TransformerRanker supports NLP classification tasks of two main families: (1) Text classification tasks such as question classification~\cite{voorhees2000overview}, sentiment analysis~\cite{socher2013recursive} or textual entailment~\cite{wang2018glue} in which a classification decision is made for an entire text (or text pair), (2) sequence labeling tasks such as named entity recognition~\cite[NER;][]{sang-conll} and part-of-speech tagging~\cite{petrov2011universal} where classification decisions are made per-word.&#10;&#10;Our library is built to rely solely on PyTorch~\cite{pytorch2019} and HuggingFace~\cite{wolf-huggingface} ecosystems and can be integrated as a fast model selection step in a larger NLP pipeline. We make TransformerRanker publicly available as a pip-installable open-source project. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Mathematics, Deep Learning Libraries, Natural Language Processing, Model Selection Optimization" />
        </attvalues>
      </node>
      <node id="2212.10082" label="2212.10082">
        <attvalues>
          <attvalue for="0" value="An Information-Theoretic Approach to Transferability in Task Transfer&#10;  Learning" />
          <attvalue for="1" value="  Task transfer learning is a popular technique in image processing&#10;applications that uses pre-trained models to reduce the supervision cost of&#10;related tasks. An important question is to determine task transferability, i.e.&#10;given a common input domain, estimating to what extent representations learned&#10;from a source task can help in learning a target task. Typically,&#10;transferability is either measured experimentally or inferred through task&#10;relatedness, which is often defined without a clear operational meaning. In&#10;this paper, we present a novel metric, H-score, an easily-computable evaluation&#10;function that estimates the performance of transferred representations from one&#10;task to another in classification problems using statistical and information&#10;theoretic principles. Experiments on real image data show that our metric is&#10;not only consistent with the empirical transferability measurement, but also&#10;useful to practitioners in applications such as source model selection and task&#10;transfer curriculum learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1104.2086" label="1104.2086">
        <attvalues>
          <attvalue for="0" value="A Universal Part-of-Speech Tagset" />
          <attvalue for="1" value="  To facilitate future research in unsupervised induction of syntactic&#10;structure and to standardize best-practices, we propose a tagset that consists&#10;of twelve universal part-of-speech categories. In addition to the tagset, we&#10;develop a mapping from 25 different treebank tagsets to this universal set. As&#10;a result, when combined with the original treebank data, this universal tagset&#10;and mapping produce a dataset consisting of common parts-of-speech for 22&#10;different languages. We highlight the use of this resource via two experiments,&#10;including one that reports competitive accuracies for unsupervised grammar&#10;induction without gold standard part-of-speech tags.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.09885" label="2106.09885">
        <attvalues>
          <attvalue for="0" value="An Improved Single Step Non-autoregressive Transformer for Automatic&#10;  Speech Recognition" />
          <attvalue for="1" value="  Non-autoregressive mechanisms can significantly decrease inference time for&#10;speech transformers, especially when the single step variant is applied.&#10;Previous work on CTC alignment-based single step non-autoregressive transformer&#10;(CASS-NAT) has shown a large real time factor (RTF) improvement over&#10;autoregressive transformers (AT). In this work, we propose several methods to&#10;improve the accuracy of the end-to-end CASS-NAT, followed by performance&#10;analyses. First, convolution augmented self-attention blocks are applied to&#10;both the encoder and decoder modules. Second, we propose to expand the trigger&#10;mask (acoustic boundary) for each token to increase the robustness of CTC&#10;alignments. In addition, iterated loss functions are used to enhance the&#10;gradient update of low-layer parameters. Without using an external language&#10;model, the WERs of the improved CASS-NAT, when using the three methods, are&#10;3.1%/7.2% on Librispeech test clean/other sets and the CER is 5.4% on the&#10;Aishell1 test set, achieving a 7%~21% relative WER/CER improvement. For the&#10;analyses, we plot attention weight distributions in the decoders to visualize&#10;the relationships between token-level acoustic embeddings. When the acoustic&#10;embeddings are visualized, we find that they have a similar behavior to word&#10;embeddings, which explains why the improved CASS-NAT performs similarly to AT.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformers have been dominant in many sequence generation tasks, outperforming their recurrent neural network (RNN) counterparts in terms of both accuracy and speed for end-to-end systems\cite{vaswani2017attention, li2020comparison, karita2019comparative}. However, the autoregressive (left-to-right) generation order slows down inference speed significantly. To accelerate the inference, non-autoregressive transformers (NAT) were proposed for the parallel generation of the output sequence. The idea is widely adopted in neural machine translation (NMT)\cite{gu2018non, lee2020deterministic, saharia2020non}, automatic speech recognition (ASR)\cite{chen2020non, chan2020imputer, bai2020listen, tian2020spike, higuchi2020mask, fujita2020insertion, fan2021cass, chi2020align, song2021non, higuchi2021improved, fujita2020end, bai2021fast}, text-to-speech (TTS)\cite{peng2020non, miao2020flow} and speech translation \cite{inaguma2021orthros}. &#10;&#10;Current NAT models for ASR can be categorized into: (i) iterative NAT, and (ii) single step NAT, according to the number of iterations for sequence generation. Essentially, autoregressive models are also iterative-based since they use a left-to-right generation order and take N iterations to generate a sequence of length N. Hence, the idea of iterative NAT is to adopt a different generation order with less than N iterations to accelerate the inference. Chen et al. regarded the transformer decoder as a masked language model that first generates tokens with high confidence \cite{chen2020non}, while Higuchi et al. applied the same idea but based on the connectionist temporal classification (CTC) output \cite{higuchi2020mask,higuchi2021improved}. In addition, Fujita et al. used the idea of the insertion transformer from NMT to generate the output sequence with an arbitrary order \cite{fujita2020insertion}. Another recent effective method is using multiple decoders as refiners to do an iterative refinement based on CTC alignments \cite{chi2020align}. Theoretically, the iterative NAT has a limited improvement of inference speed since multiple iterations are still needed to obtain a competitive result. In contrast, single step NAT, which attempts to generate the output sequence with only one iteration, can have a better speed up for inference. The idea is to substitute the word embedding in autoregressive models with an acoustic representation for each output token, assuming that language semantics can also be captured by acoustic representations \cite{bai2020listen, tian2020spike, fan2021cass}.&#10;&#10;Although various NAT methods were proposed for ASR, the WER performance still lags behind that of state-of-the-art autoregressive models. Therefore, based on our previous work\cite{fan2021cass}, we propose several methods to improve the accuracy of CTC alignment-based single step NAT (CASS-NAT) with little inference speed loss. First, convolution augmented self-attention blocks are applied to both the encoder and decoder modules, while other work only considered using them in the encoder\cite{gulati2020conformer}. The second method is to expand the trigger mask (acoustic boundary) for each token to increase the robustness of the CTC alignment. Third, considering the wide use of iterated loss functions to train deep transformers\cite{tjandra2020deja, wang2020transformer, lee2021intermediate}, we apply iterated loss to enhance the gradient update of low-layer parameters for both the encoder and decoder modules. When no external language model is used, large improvements are observed on both the Librispeech\cite{panayotov2015librispeech} and Aishell1\cite{bu2017aishell} datasets in terms of error rate, and the performance is close to the autoregressive baseline. Additionally, we analyse the self-attention distributions and token-level acoustic embeddings in the decoders. We find a similar behaviour between the token-level acoustic embedding and word embedding, which explains why CASS-NAT performs similarly to autoregressive models. &#10;&#10;The remainder of the paper is organized as follows. Section 2 briefly reviews the CASS-NAT and describes the proposed methods for improving the system. Section 3 describes the recognition experimental setup, followed by results and analyses in Section 4. Section 5 concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Non-Autoregressive Inference, Signal Processing, Speech Recognition, Transformer Models" />
        </attvalues>
      </node>
      <node id="1905.08459" label="1905.08459">
        <attvalues>
          <attvalue for="0" value="Non-Autoregressive Neural Text-to-Speech" />
          <attvalue for="1" value="  In this work, we propose ParaNet, a non-autoregressive seq2seq model that&#10;converts text to spectrogram. It is fully convolutional and brings 46.7 times&#10;speed-up over the lightweight Deep Voice 3 at synthesis, while obtaining&#10;reasonably good speech quality. ParaNet also produces stable alignment between&#10;text and speech on the challenging test sentences by iteratively improving the&#10;attention in a layer-by-layer manner. Furthermore, we build the parallel&#10;text-to-speech system and test various parallel neural vocoders, which can&#10;synthesize speech from text through a single feed-forward pass. We also explore&#10;a novel VAE-based approach to train the inverse autoregressive flow (IAF) based&#10;parallel vocoder from scratch, which avoids the need for distillation from a&#10;separately trained WaveNet as previous work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13047" label="2010.13047">
        <attvalues>
          <attvalue for="0" value="Orthros: Non-autoregressive End-to-end Speech Translation with&#10;  Dual-decoder" />
          <attvalue for="1" value="  Fast inference speed is an important goal towards real-world deployment of&#10;speech translation (ST) systems. End-to-end (E2E) models based on the&#10;encoder-decoder architecture are more suitable for this goal than traditional&#10;cascaded systems, but their effectiveness regarding decoding speed has not been&#10;explored so far. Inspired by recent progress in non-autoregressive (NAR)&#10;methods in text-based translation, which generates target tokens in parallel by&#10;eliminating conditional dependencies, we study the problem of NAR decoding for&#10;E2E-ST. We propose a novel NAR E2E-ST framework, Orthros, in which both NAR and&#10;autoregressive (AR) decoders are jointly trained on the shared speech encoder.&#10;The latter is used for selecting better translation among various length&#10;candidates generated from the former, which dramatically improves the&#10;effectiveness of a large length beam with negligible overhead. We further&#10;investigate effective length prediction methods from speech inputs and the&#10;impact of vocabulary sizes. Experiments on four benchmarks show the&#10;effectiveness of the proposed method in improving inference speed while&#10;maintaining competitive translation quality compared to state-of-the-art AR&#10;E2E-ST systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13270" label="2010.13270">
        <attvalues>
          <attvalue for="0" value="Improved Mask-CTC for Non-Autoregressive End-to-End ASR" />
          <attvalue for="1" value="  For real-world deployment of automatic speech recognition (ASR), the system&#10;is desired to be capable of fast inference while relieving the requirement of&#10;computational resources. The recently proposed end-to-end ASR system based on&#10;mask-predict with connectionist temporal classification (CTC), Mask-CTC,&#10;fulfills this demand by generating tokens in a non-autoregressive fashion.&#10;While Mask-CTC achieves remarkably fast inference speed, its recognition&#10;performance falls behind that of conventional autoregressive (AR) systems. To&#10;boost the performance of Mask-CTC, we first propose to enhance the encoder&#10;network architecture by employing a recently proposed architecture called&#10;Conformer. Next, we propose new training and decoding methods by introducing&#10;auxiliary objective to predict the length of a partial target sequence, which&#10;allows the model to delete or insert tokens during inference. Experimental&#10;results on different ASR tasks show that the proposed approaches improve&#10;Mask-CTC significantly, outperforming a standard CTC model (15.5% $\rightarrow$&#10;9.1% WER on WSJ). Moreover, Mask-CTC now achieves competitive results to AR&#10;models with no degradation of inference speed ($&lt;$ 0.1 RTF using CPU). We also&#10;show a potential application of Mask-CTC to end-to-end speech translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.13211" label="2005.13211">
        <attvalues>
          <attvalue for="0" value="Insertion-Based Modeling for End-to-End Automatic Speech Recognition" />
          <attvalue for="1" value="  End-to-end (E2E) models have gained attention in the research field of&#10;automatic speech recognition (ASR). Many E2E models proposed so far assume&#10;left-to-right autoregressive generation of an output token sequence except for&#10;connectionist temporal classification (CTC) and its variants. However,&#10;left-to-right decoding cannot consider the future output context, and it is not&#10;always optimal for ASR. One of the non-left-to-right models is known as&#10;non-autoregressive Transformer (NAT) and has been intensively investigated in&#10;the area of neural machine translation (NMT) research. One NAT model,&#10;mask-predict, has been applied to ASR but the model needs some heuristics or&#10;additional component to estimate the length of the output token sequence. This&#10;paper proposes to apply another type of NAT called insertion-based models, that&#10;were originally proposed for NMT, to ASR tasks. Insertion-based models solve&#10;the above mask-predict issues and can generate an arbitrary generation order of&#10;an output sequence. In addition, we introduce a new formulation of joint&#10;training of the insertion-based models and CTC. This formulation reinforces CTC&#10;by making it dependent on insertion-based token generation in a&#10;non-autoregressive manner. We conducted experiments on three public benchmarks&#10;and achieved competitive performance to strong autoregressive Transformer with&#10;a similar decoding condition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.04862" label="2005.04862">
        <attvalues>
          <attvalue for="0" value="Listen Attentively, and Spell Once: Whole Sentence Generation via a&#10;  Non-Autoregressive Architecture for Low-Latency Speech Recognition" />
          <attvalue for="1" value="  Although attention based end-to-end models have achieved promising&#10;performance in speech recognition, the multi-pass forward computation in&#10;beam-search increases inference time cost, which limits their practical&#10;applications. To address this issue, we propose a non-autoregressive end-to-end&#10;speech recognition system called LASO (listen attentively, and spell once).&#10;Because of the non-autoregressive property, LASO predicts a textual token in&#10;the sequence without the dependence on other tokens. Without beam-search, the&#10;one-pass propagation much reduces inference time cost of LASO. And because the&#10;model is based on the attention based feedforward structure, the computation&#10;can be implemented in parallel efficiently. We conduct experiments on publicly&#10;available Chinese dataset AISHELL-1. LASO achieves a character error rate of&#10;6.4%, which outperforms the state-of-the-art autoregressive transformer model&#10;(6.7%). The average inference latency is 21 ms, which is 1/50 of the&#10;autoregressive transformer model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.10324" label="1910.10324">
        <attvalues>
          <attvalue for="0" value="Deja-vu: Double Feature Presentation and Iterated Loss in Deep&#10;  Transformer Networks" />
          <attvalue for="1" value="  Deep acoustic models typically receive features in the first layer of the&#10;network, and process increasingly abstract representations in the subsequent&#10;layers. Here, we propose to feed the input features at multiple depths in the&#10;acoustic model. As our motivation is to allow acoustic models to re-examine&#10;their input features in light of partial hypotheses we introduce intermediate&#10;model heads and loss function. We study this architecture in the context of&#10;deep Transformer networks, and we use an attention mechanism over both the&#10;previous layer activations and the input features. To train this model's&#10;intermediate output hypothesis, we apply the objective function at each layer&#10;right before feature re-use. We find that the use of such iterated loss&#10;significantly improves performance by itself, as well as enabling input feature&#10;re-use. We present results on both Librispeech, and a large scale video&#10;dataset, with relative improvements of 10 - 20% for Librispeech and 3.2 - 13%&#10;for videos.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.12184" label="2011.12184">
        <attvalues>
          <attvalue for="0" value="Neural Text Classification by Jointly Learning to Cluster and Align" />
          <attvalue for="1" value="  Distributional text clustering delivers semantically informative&#10;representations and captures the relevance between each word and semantic&#10;clustering centroids. We extend the neural text clustering approach to text&#10;classification tasks by inducing cluster centers via a latent variable model&#10;and interacting with distributional word embeddings, to enrich the&#10;representation of tokens and measure the relatedness between tokens and each&#10;learnable cluster centroid. The proposed method jointly learns word clustering&#10;centroids and clustering-token alignments, achieving the state of the art&#10;results on multiple benchmark datasets and proving that the proposed&#10;cluster-token alignment mechanism is indeed favorable to text classification.&#10;Notably, our qualitative analysis has conspicuously illustrated that text&#10;representations learned by the proposed model are in accord well with our&#10;intuition.&#10;" />
          <attvalue for="2" value="&#10;Text classification, as an extensively applied fundamental cornerstone for natural language processing (NLP) applications, such as sentiment analysis~\cite{xue2018aspect}, spam detection~\cite{kennedy2019fact} and spoken dialogue systems~\cite{lowe2016evaluation,gupta2019simple}, has been widely studied for decades. In general, almost all NLP tasks can be cast into classification problems on either document, sentence, or word level. Here we are focusing on the means of it in a narrow sense, i.e., given a sequence of tokens with arbitrary length, predicting the most likely categorization it belongs to.&#10;&#10;Considerable compelling neural approaches to the text classification task have empirically demonstrated their remarkable behaviors in recent years, to whom how to orchestrate and compose the semantic and syntactic representations from texts are central. Much of the work concentrated on learning the composition of distributional word representations~\cite{mikolov2013efficient, pennington2014glove, bojanowski2017enriching} for categorization, wherein plenty of deep learning methods have been adopted, such as TextCNNs~\cite{kim2014convolutional}, RCNNs~\cite{lai2015recurrent}, recurrent neural networks (RNNs)~\cite{liu2016recurrent}, FastText~\cite{joulin2016bag}, BERT~\cite{devlin2018bert}, etc. Most of them learn the word representations by firstly projecting the one-hot encoding of each token through a pretrained or randomly initialized word embedding matrices to acquire the dense real-valued vectors, and then feed them into neural models for classification. &#10;&#10;These methods, however, have only exploited the low-dimensional semantic representations for each sample text in a supervised way. Some argued that unsupervised latent representations such as topic~\cite{hingmire2014sprinkling,ma2015distributional,li2016news} or cluster modeling~\cite{baker1998distributional, zhang2009text, wang2016semantic, gowda2016semi} mined by latent variable models may be of benefit. \cite{baker1998distributional} maintained that word clustering could deliver the useful semantic information by grouping all words in the corpus and can thus promote the classification accuracy. Moreover, \cite{zeng2018topic} incorporated the neural topic models with Variational Autoencoder (VAE)~\cite{kingma2013auto} into the classification tasks so as to discover the latent topics in the document level and encode the co-occurrence of words with bag-of-words statistics. &#10;&#10;Learning such corpus-level representation can administer to the enrichment of more globally informative features and is thus favorable to the task performance. There are plenty of works adopting VAE for learning these latent variables to boost the text classification performance~\cite{xu2017variational,ayinde2017deep,soares2018effort}.&#10;Nevertheless, there remain problems that we cannot directly treat the sampled latent space of VAE for clustering centroids since there is no mechanism to modulate the representation of different samples towards different mean and variance for a better discrimination purpose under the Gaussian distribution assumption~\cite{lim2020deep}. \cite{song2013auto} and \cite{lim2020deep} alleviate these issues by minimizing the distance between the learnable latent representation from latent variable models and the clustering centers generated from statistical clustering approaches. &#10;&#10;Grounding on this, we design an ad hoc Clustering-Enchanced neural model (hereafter CluE) that jointly learns the distributional clustering and the alignment between the domain-aware clustering centroids and word representations in the Euclidean hidden semantic space for text classification, with the vector space assumption that words with similar meanings are close to each other~\cite{mikolov2013distributed}. Instead of directly treating the latent variables as the clustering centroids, we employ a co-adaptation strategy to minimize the difference between the hidden variables and trainable clustering centroids initialized by traditional clustering algorithms with soft alignments.&#10;&#10;In the present work, we propose the cluster-token alignment mechanism by assigning relevance probability distribution of clusters to each token, indicating how likely it is that tokens are correlated with each cluster center. In which clustering centroids are co-regulated with learned latent variables and can be regarded as the domain- or task-specific feature indicators.&#10;&#10;Our work illustrates that jointly adapting the clustering centroids and learning the cluster-token alignment holds the promise of advancing the text classification performance by incorporating the clustering-aware representations. Our key contributions are:&#10;&#10;\begin{itemize} [leftmargin=*]&#10; \item to empirically and visually demonstrate that the proposed model could surprisingly deliver visually-interpretable text representations for text classification (as fig.~\ref{fig:sent_emb}).&#10; \item to show that our clustering-token interaction mechanism could apparently capture semantic meanings, including the relevance alignment between clusters and input tokens.&#10; \item to confirm that our joint learning model eclipses the prevailing baseline models and achieves state-of-the-art results on classifying both short and long texts. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Classification, Machine Learning, Computer Science, Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing, Distributional Semantics, Text Clustering" />
        </attvalues>
      </node>
      <node id="2010.15296" label="2010.15296">
        <attvalues>
          <attvalue for="0" value="Fact or Factitious? Contextualized Opinion Spam Detection" />
          <attvalue for="1" value="  In this paper we perform an analytic comparison of a number of techniques&#10;used to detect fake and deceptive online reviews. We apply a number machine&#10;learning approaches found to be effective, and introduce our own approach by&#10;fine-tuning state of the art contextualised embeddings. The results we obtain&#10;show the potential of contextualised embeddings for fake review detection, and&#10;lay the groundwork for future research in this area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1605.05414" label="1605.05414">
        <attvalues>
          <attvalue for="0" value="On the Evaluation of Dialogue Systems with Next Utterance Classification" />
          <attvalue for="1" value="  An open challenge in constructing dialogue systems is developing methods for&#10;automatically learning dialogue strategies from large amounts of unlabelled&#10;data. Recent work has proposed Next-Utterance-Classification (NUC) as a&#10;surrogate task for building dialogue systems from text data. In this paper we&#10;investigate the performance of humans on this task to validate the relevance of&#10;NUC as a method of evaluation. Our results show three main findings: (1) humans&#10;are able to correctly classify responses at a rate much better than chance,&#10;thus confirming that the task is feasible, (2) human performance levels vary&#10;across task domains (we consider 3 datasets) and expertise levels (novice vs&#10;experts), thus showing that a range of performance is possible on this type of&#10;task, (3) automated dialogue systems built using state-of-the-art machine&#10;learning methods have similar performance to the human novices, but worse than&#10;the experts, thus confirming the utility of this class of tasks for driving&#10;further research in automated dialogue systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1903.08268" label="1903.08268">
        <attvalues>
          <attvalue for="0" value="Simple, Fast, Accurate Intent Classification and Slot Labeling for&#10;  Goal-Oriented Dialogue Systems" />
          <attvalue for="1" value="  With the advent of conversational assistants, like Amazon Alexa, Google Now,&#10;etc., dialogue systems are gaining a lot of traction, especially in industrial&#10;setting. These systems typically consist of Spoken Language understanding&#10;component which, in turn, consists of two tasks - Intent Classification (IC)&#10;and Slot Labeling (SL). Generally, these two tasks are modeled together jointly&#10;to achieve best performance. However, this joint modeling adds to model&#10;obfuscation. In this work, we first design framework for a modularization of&#10;joint IC-SL task to enhance architecture transparency. Then, we explore a&#10;number of self-attention, convolutional, and recurrent models, contributing a&#10;large-scale analysis of modeling paradigms for IC+SL across two datasets.&#10;Finally, using this framework, we propose a class of 'label-recurrent' models&#10;that otherwise non-recurrent, with a 10-dimensional representation of the label&#10;history, and show that our proposed systems are easy to interpret, highly&#10;accurate (achieving over 30% error reduction in SL over the state-of-the-art on&#10;the Snips dataset), as well as fast, at 2x the inference and 2/3 to 1/2 the&#10;training time of comparable recurrent models, thus giving an edge in critical&#10;real-world systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.03664" label="1809.03664">
        <attvalues>
          <attvalue for="0" value="Topic Memory Networks for Short Text Classification" />
          <attvalue for="1" value="  Many classification models work poorly on short texts due to data sparsity.&#10;To address this issue, we propose topic memory networks for short text&#10;classification with a novel topic memory mechanism to encode latent topic&#10;representations indicative of class labels. Different from most prior work that&#10;focuses on extending features with external knowledge or pre-trained topics,&#10;our model jointly explores topic inference and text classification with memory&#10;networks in an end-to-end manner. Experimental results on four benchmark&#10;datasets show that our model outperforms state-of-the-art models on short text&#10;classification, meanwhile generates coherent topics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.11431" label="2103.11431">
        <attvalues>
          <attvalue for="0" value="SEMIE: SEMantically Infused Embeddings with Enhanced Interpretability&#10;  for Domain-specific Small Corpus" />
          <attvalue for="1" value="  Word embeddings are a basic building block of modern NLP pipelines. Efforts&#10;have been made to learn rich, efficient, and interpretable embeddings for large&#10;generic datasets available in the public domain. However, these embeddings have&#10;limited applicability for small corpora from specific domains such as&#10;automotive, manufacturing, maintenance and support, etc. In this work, we&#10;present a comprehensive notion of interpretability for word embeddings and&#10;propose a novel method to generate highly interpretable and efficient&#10;embeddings for a domain-specific small corpus. We report the evaluation results&#10;of our resulting word embeddings and demonstrate their novel features for&#10;enhanced interpretability.&#10;" />
          <attvalue for="2" value="&#10;Distributed representations of words, also termed as word embeddings, have been used extensively to excel at various applications such as parsing~\cite{lazaridou2013fish, bansal2014tailoring}, named entity recognition~\cite{guo2014revisiting}, image captioning~\cite{you2016image} and sentiment analysis~\cite{socher2013recursive}. They have also proven effective in modeling cognitive operations such as the judgement of word similarity~\cite{turney2010frequency, baroni2010distributional}, and the brain activity elicited by specific concepts~\cite{mitchell2008predicting}. &#10;However, these representations contain mappings of words to vectors of real numbers in dense and continuous space, and thus, inherently difficult to interpret. &#10;&#10;Recent studies~\cite{murphy2012learning, fyshe2014interpretable} suggest that sparsity and non-negativity of the word embeddings are two important characteristics that make them interpretable. The sparsity makes each word vector contain a small number of active (non-zero) dimensions~\cite{olshausen1997sparse}, which helps in increasing their separability and stability in the presence of noise~\cite{lewicki2000learning, donoho2005stable}.&#10;In addition, the studies define the notion of interpretability in terms of the coherence of dimensions of the word embeddings~\cite{faruqui2015sparse, lipton2018mythos, subramanian2018spine}. In other words, the word embeddings are considered as interpretable if their dimensions denote specific semantic concepts. However, these studies are primarily focused on pre-trained word embeddings like GloVe~\cite{pennington2014glove} and word2vec~\cite{mikolov2013distributed}. These pre-trained embeddings are generated using millions of documents from generic public domain datasets such as Wikipedia and Google News, which contain billions of words. Also, to interpret the thousands of dimensions of the sparse non-negative (hereafter `SNN') word embeddings and understand the sense they correspond to, we still need human judges who put in manual efforts and read these unlabeled dimensions.&#10;&#10;In addition to the aforementioned semantic similarities, the recent studies suggest that the interpretability of word embeddings should also consider the semantic dissimilarities such as identification of discriminative word triples~\cite{krebs2016capturing, krebs2018semeval}. Given a triplet of words ($w_1$, $w_2$, $d$), the word embeddings should be able to determine whether $d$ is a discriminative feature between two concepts $w_1$ and $w_2$. For example, the word ``buckle'' is a discriminative feature in the triplet &#10;(``seat belt'', ``tires'', ``buckle'') that characterizes the first concept but not the second. &#10;Researchers have formulated this property as a binary classification task and proposed machine learning and similarity-based methods to evaluate the word embeddings ~\cite{zhang2018umd, dumitru2018alb, grishin2018igevorse}. However, to perform these evaluations for a domain-specific small corpus, we would need a manually curated set of discriminative (positive) and non-discriminative (negative) triples, which can be costly and time-consuming to curate. &#10;&#10;Comprehensive Notion of Interpretability: In conclusion, we can say that the interpretability of word embeddings is expressed in terms of both semantic similarities and dissimilarities while representing them in SNN embedding space. Here, semantic similarities correspond to dimensional coherence, whereas semantic dissimilarities refer to properties such as identification of discriminative word triples. This raises the following question:\\ &#10;&#10;How can we generate word embeddings for domain-specific small corpus, which are interpretable in terms of both semantic similarities and dissimilarities, when represented in SNN embedding space?\\&#10;&#10;To address the above-mentioned question, we present a novel method to generate such word embeddings, which we name as SEMantically Infused Embeddings (SEMIE). We showcase the enhanced interpretability (both in terms of semantic similarities and dissimilarities) capabilities of SEMIE while representing them in SNN embedding space. We also demonstrate the efficiency of SEMIE on the downstream classification task, both in dense and SNN embedding space. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Word Embeddings, Computational Linguistics, Domain Specific Corpora, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1003.1141" label="1003.1141">
        <attvalues>
          <attvalue for="0" value="From Frequency to Meaning: Vector Space Models of Semantics" />
          <attvalue for="1" value="  Computers understand very little of the meaning of human language. This&#10;profoundly limits our ability to give instructions to computers, the ability of&#10;computers to explain their actions to us, and the ability of computers to&#10;analyse and process text. Vector space models (VSMs) of semantics are beginning&#10;to address these limits. This paper surveys the use of VSMs for semantic&#10;processing of text. We organize the literature on VSMs according to the&#10;structure of the matrix in a VSM. There are currently three broad classes of&#10;VSMs, based on term-document, word-context, and pair-pattern matrices, yielding&#10;three classes of applications. We survey a broad range of applications in these&#10;three categories and we take a detailed look at a specific open source project&#10;in each category. Our goal in this survey is to show the breadth of&#10;applications of VSMs for semantics, to provide a new perspective on VSMs for&#10;those who are already familiar with the area, and to provide pointers into the&#10;literature for those who are less familiar with the field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.02004" label="1506.02004">
        <attvalues>
          <attvalue for="0" value="Sparse Overcomplete Word Vector Representations" />
          <attvalue for="1" value="  Current distributed representations of words show little resemblance to&#10;theories of lexical semantics. The former are dense and uninterpretable, the&#10;latter largely based on familiar, discrete classes (e.g., supersenses) and&#10;relations (e.g., synonymy and hypernymy). We propose methods that transform&#10;word vectors into sparse (and optionally binary) vectors. The resulting&#10;representations are more similar to the interpretable features typically used&#10;in NLP, though they are discovered automatically from raw corpora. Because the&#10;vectors are highly sparse, they are computationally easy to work with. Most&#10;importantly, we find that they outperform the original vectors on benchmark&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.08217" label="2307.08217">
        <attvalues>
          <attvalue for="0" value="BASS: Block-wise Adaptation for Speech Summarization" />
          <attvalue for="1" value="  End-to-end speech summarization has been shown to improve performance over&#10;cascade baselines. However, such models are difficult to train on very large&#10;inputs (dozens of minutes or hours) owing to compute restrictions and are hence&#10;trained with truncated model inputs. Truncation leads to poorer models, and a&#10;solution to this problem rests in block-wise modeling, i.e., processing a&#10;portion of the input frames at a time. In this paper, we develop a method that&#10;allows one to train summarization models on very long sequences in an&#10;incremental manner. Speech summarization is realized as a streaming process,&#10;where hypothesis summaries are updated every block based on new acoustic&#10;information. We devise and test strategies to pass semantic context across the&#10;blocks. Experiments on the How2 dataset demonstrate that the proposed&#10;block-wise training method improves by 3 points absolute on ROUGE-L over a&#10;truncated input baseline.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the rising amount of data that people consume in daily life -- videos, music, podcasts, meetings, lectures, and more -- building artificial intelligence that can concisely extract important information ~\cite{sharma2022xnor,palaskar21_interspeech} has gained importance. Speech Summarization refers to the task of developing intelligent machines that can generate condensed textual representations called &quot;summaries&quot; from long audio inputs. Speech summarization, whether extractive \cite{liu2015combining} or abstractive \cite{kano2020asru,kano2022icassp,shon2022slueted}, requires global acoustic context since knowledge of the entire speech signal is helpful for either extracting relevant key-frames or generating comprehensive abstractive summaries.&#10;&#10;Recently, end-to-end speech summarization models ~\cite{sharma2022end,matsuura2023} have been shown to outperform competitive cascade models that comprise speech recognition and text summarization modules. Such end-to-end models use very long speech sequences as input, and standard transformer models cannot handle very long inputs owing to the quadratic computational complexity of self-attention. Prior work has proposed restricting the scope of attention using the Longformer ~\cite{sharma2022end,beltagy2020longformer} or linear self-attentions like the XNOR-former ~\cite{sharma2022xnor}. However, even with such optimizations, there remains an upper limit on the number of input frames that a given end-to-end model can consume with available computing infrastructure. For example, with a 6-layer conformer~\cite{conformer} encoder, a 32G V-100 GPU can take sequences of length 25,000; and with an XNOR-encoder, the same GPU can take ~45,000 frames. Any input speech sequence of length greater than this upper limit is truncated to be able to train and infer, and truncating inputs makes summarization less accurate since information is effectively removed from the input. Further, attention-based sequence models do not generalize well to input lengths that are different from those used in training ~\cite{deng22b_interspeech}, which makes adapting to longer input sequences important.&#10;&#10;To address this challenge, one solution is to build models that can process a small set of input frames, i.e., a block of input at a time rather than using the entire input sequence. Such &quot;block-wise&quot; models can be trained in two ways - to predict an output either after seeing multiple blocks of input or after every new block of input. &#10;&#10;Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}. Block-wise training for streaming applications uses one block of input at a time to generate a block-level encoding. These block-level encodings from all blocks are then combined to make an utterance-level prediction. During training, all the inputs from all blocks, intermediate outputs, and the final output are retained in the computational graph for backpropagation. This requires significant compute and memory so these models still can't scale to very long audio sequences.&#10;&#10;The latter category of block-wise models overcomes this challenge by producing outputs after every block, so they can be optimized at the block level without requiring the entire input to be present in the computational graph. Prior work in speech recognition for long conversations ~\cite{KimMetze18,hori2021advanced, HoriMHR20} can be considered examples of such block-wise models, since they produce an utterance transcription for every new block of input. Such block-level targets are relatively easy to derive for tasks like speech recognition, where there exists a monotonic alignment between the input frames and output tokens. However, for abstractive speech summarization, the relationship between input frames and output tokens is non-monotonic and indirect, and it is consequently challenging to obtain block-level targets. &#10;&#10;In this paper, we first mathematically formulate the process of block-wise training and introduce in Figure \ref{fig:block_diagram} Block-Wise Adaptation for Speech Summarization (BASS), an online model that can be trained with the full reference summary as the block-level target. This means that our model attempts to produce the output summary given only the first block, and then subsequently refines its prediction with every additional block of speech input. While streaming mechanisms assume that new acoustic inputs may incrementally modify the output, we permit the model to modify the entire summary if necessary based on the information present in the new input block. When using such block-wise inputs during training or inference, blocks should have access to the information encoded by previous blocks. &#10;We propose to achieve this by passing the latent representation across blocks since it is likely where the semantic information is encoded. While it is also possible to carry forward input acoustics or output summaries, these may not be as useful because input acoustics may not be as informative, and output summaries could be erroneous or change entirely with new blocks. In summary, this paper makes the following contributions:&#10;&#10;\begin{enumerate}&#10; \item We introduce Block-wise Adaptation for Speech Summarization (BASS), a novel algorithm for training speech summarization models. BASS predicts a speech summary after consuming a new block of the input speech and allows new summaries to be modified fully if necessary.&#10; \item We introduce an explicit layer of semantic representation, which aggregates semantics from the input acoustics and is not affected by how it is expressed. We then describe mechanisms to carry this semantic context across blocks for adaptation and training. &#10; \item We evaluate the relative strengths of block-wise adaptation from a pretrained model and block-wise training from scratch, and show that BASS improves performance under adaptation settings by 3 points on ROUGE-L. &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Summarization, Linguistics, Artificial Intelligence, Incremental Training, Block-Wise Modeling" />
        </attvalues>
      </node>
      <node id="2111.08201" label="2111.08201">
        <attvalues>
          <attvalue for="0" value="Attention-based Multi-hypothesis Fusion for Speech Summarization" />
          <attvalue for="1" value="  Speech summarization, which generates a text summary from speech, can be&#10;achieved by combining automatic speech recognition (ASR) and text summarization&#10;(TS). With this cascade approach, we can exploit state-of-the-art models and&#10;large training datasets for both subtasks, i.e., Transformer for ASR and&#10;Bidirectional Encoder Representations from Transformers (BERT) for TS. However,&#10;ASR errors directly affect the quality of the output summary in the cascade&#10;approach. We propose a cascade speech summarization model that is robust to ASR&#10;errors and that exploits multiple hypotheses generated by ASR to attenuate the&#10;effect of ASR errors on the summary. We investigate several schemes to combine&#10;ASR hypotheses. First, we propose using the sum of sub-word embedding vectors&#10;weighted by their posterior values provided by an ASR system as an input to a&#10;BERT-based TS system. Then, we introduce a more general scheme that uses an&#10;attention-based fusion module added to a pre-trained BERT module to align and&#10;combine several ASR hypotheses. Finally, we perform speech summarization&#10;experiments on the How2 dataset and a newly assembled TED-based dataset that we&#10;will release with this paper. These experiments show that retraining the&#10;BERT-based TS system with these schemes can improve summarization performance&#10;and that the attention-based fusion module is particularly effective.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10525" label="2212.10525">
        <attvalues>
          <attvalue for="0" value="SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding&#10;  Tasks" />
          <attvalue for="1" value="  Spoken language understanding (SLU) tasks have been studied for many decades&#10;in the speech research community, but have not received as much attention as&#10;lower-level tasks like speech and speaker recognition. In particular, there are&#10;not nearly as many SLU task benchmarks, and many of the existing ones use data&#10;that is not freely available to all researchers. Recent work has begun to&#10;introduce such benchmark datasets for several tasks. In this work, we introduce&#10;several new annotated SLU benchmark tasks based on freely available speech&#10;data, which complement existing benchmarks and address gaps in the SLU&#10;evaluation landscape. We contribute four tasks: question answering and&#10;summarization involve inference over longer speech sequences; named entity&#10;localization addresses the speech-specific task of locating the targeted&#10;content in the signal; dialog act classification identifies the function of a&#10;given speech utterance. We follow the blueprint of the Spoken Language&#10;Understanding Evaluation (SLUE) benchmark suite. In order to facilitate the&#10;development of SLU models that leverage the success of pre-trained speech&#10;representations, we will be publishing for each task (i) annotations for a&#10;relatively small fine-tuning set, (ii) annotated development and test sets, and&#10;(iii) baseline models for easy reproducibility and comparisons. In this work,&#10;we present the details of data collection and annotation and the performance of&#10;the baseline models. We also perform sensitivity analysis of pipeline models'&#10;performance (speech recognizer + text model) to the speech recognition&#10;accuracy, using more than 20 state-of-the-art speech recognition models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.00978" label="2303.00978">
        <attvalues>
          <attvalue for="0" value="Leveraging Large Text Corpora for End-to-End Speech Summarization" />
          <attvalue for="1" value="  End-to-end speech summarization (E2E SSum) is a technique to directly&#10;generate summary sentences from speech. Compared with the cascade approach,&#10;which combines automatic speech recognition (ASR) and text summarization&#10;models, the E2E approach is more promising because it mitigates ASR errors,&#10;incorporates nonverbal information, and simplifies the overall system. However,&#10;since collecting a large amount of paired data (i.e., speech and summary) is&#10;difficult, the training data is usually insufficient to train a robust E2E SSum&#10;system. In this paper, we present two novel methods that leverage a large&#10;amount of external text summarization data for E2E SSum training. The first&#10;technique is to utilize a text-to-speech (TTS) system to generate synthesized&#10;speech, which is used for E2E SSum training with the text summary. The second&#10;is a TTS-free method that directly inputs phoneme sequence instead of&#10;synthesized speech to the E2E SSum model. Experiments show that our proposed&#10;TTS- and phoneme-based methods improve several metrics on the How2 dataset. In&#10;particular, our best system outperforms a previous state-of-the-art one by a&#10;large margin (i.e., METEOR score improvements of more than 6 points). To the&#10;best of our knowledge, this is the first work to use external language&#10;resources for E2E SSum. Moreover, we report a detailed analysis of the How2&#10;dataset to confirm the validity of our proposed E2E SSum system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.08920" label="2204.08920">
        <attvalues>
          <attvalue for="0" value="Blockwise Streaming Transformer for Spoken Language Understanding and&#10;  Simultaneous Speech Translation" />
          <attvalue for="1" value="  Although Transformers have gained success in several speech processing tasks&#10;like spoken language understanding (SLU) and speech translation (ST), achieving&#10;online processing while keeping competitive performance is still essential for&#10;real-world interaction. In this paper, we take the first step on streaming SLU&#10;and simultaneous ST using a blockwise streaming Transformer, which is based on&#10;contextual block processing and blockwise synchronous beam search. Furthermore,&#10;we design an automatic speech recognition (ASR)-based intermediate loss&#10;regularization for the streaming SLU task to improve the classification&#10;performance further. As for the simultaneous ST task, we propose a&#10;cross-lingual encoding method, which employs a CTC branch optimized with target&#10;language translations. In addition, the CTC translation output is also used to&#10;refine the search space with CTC prefix score, achieving joint CTC/attention&#10;simultaneous translation for the first time. Experiments for SLU are conducted&#10;on FSC and SLURP corpora, while the ST task is evaluated on Fisher-CallHome&#10;Spanish and MuST-C En-De corpora. Experimental results show that the blockwise&#10;streaming Transformer achieves competitive results compared to offline models,&#10;especially with our proposed methods that further yield a 2.4% accuracy gain on&#10;the SLU task and a 4.3 BLEU gain on the ST task over streaming baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.02674" label="2001.02674">
        <attvalues>
          <attvalue for="0" value="Streaming automatic speech recognition with the transformer model" />
          <attvalue for="1" value="  Encoder-decoder based sequence-to-sequence models have demonstrated&#10;state-of-the-art results in end-to-end automatic speech recognition (ASR).&#10;Recently, the transformer architecture, which uses self-attention to model&#10;temporal context information, has been shown to achieve significantly lower&#10;word error rates (WERs) compared to recurrent neural network (RNN) based system&#10;architectures. Despite its success, the practical usage is limited to offline&#10;ASR tasks, since encoder-decoder architectures typically require an entire&#10;speech utterance as input. In this work, we propose a transformer based&#10;end-to-end ASR system for streaming ASR, where an output must be generated&#10;shortly after each spoken word. To achieve this, we apply time-restricted&#10;self-attention for the encoder and triggered attention for the encoder-decoder&#10;attention mechanism. Our proposed streaming transformer architecture achieves&#10;2.8% and 7.2% WER for the &quot;clean&quot; and &quot;other&quot; test data of LibriSpeech, which&#10;to our knowledge is the best published streaming end-to-end ASR result for this&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.14941" label="2006.14941">
        <attvalues>
          <attvalue for="0" value="Streaming Transformer ASR with Blockwise Synchronous Beam Search" />
          <attvalue for="1" value="  The Transformer self-attention network has shown promising performance as an&#10;alternative to recurrent neural networks in end-to-end (E2E) automatic speech&#10;recognition (ASR) systems. However, Transformer has a drawback in that the&#10;entire input sequence is required to compute both self-attention and&#10;source--target attention. In this paper, we propose a novel blockwise&#10;synchronous beam search algorithm based on blockwise processing of encoder to&#10;perform streaming E2E Transformer ASR. In the beam search, encoded feature&#10;blocks are synchronously aligned using a block boundary detection technique,&#10;where a reliability score of each predicted hypothesis is evaluated based on&#10;the end-of-sequence and repeated tokens in the hypothesis. Evaluations of the&#10;HKUST and AISHELL-1 Mandarin, LibriSpeech English, and CSJ Japanese tasks show&#10;that the proposed streaming Transformer algorithm outperforms conventional&#10;online approaches, including monotonic chunkwise attention (MoChA), especially&#10;when using the knowledge distillation technique. An ablation study indicates&#10;that our streaming approach contributes to reducing the response time, and the&#10;repetition criterion contributes significantly in certain tasks. Our streaming&#10;ASR models achieve comparable or superior performance to batch models and other&#10;streaming-based Transformer methods in all tasks considered.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00033" label="2011.00033">
        <attvalues>
          <attvalue for="0" value="Streaming Simultaneous Speech Translation with Augmented Memory&#10;  Transformer" />
          <attvalue for="1" value="  Transformer-based models have achieved state-of-the-art performance on speech&#10;translation tasks. However, the model architecture is not efficient enough for&#10;streaming scenarios since self-attention is computed over an entire input&#10;sequence and the computational cost grows quadratically with the length of the&#10;input sequence. Nevertheless, most of the previous work on simultaneous speech&#10;translation, the task of generating translations from partial audio input,&#10;ignores the time spent in generating the translation when analyzing the&#10;latency. With this assumption, a system may have good latency quality&#10;trade-offs but be inapplicable in real-time scenarios. In this paper, we focus&#10;on the task of streaming simultaneous speech translation, where the systems are&#10;not only capable of translating with partial input but are also able to handle&#10;very long or continuous input. We propose an end-to-end transformer-based&#10;sequence-to-sequence model, equipped with an augmented memory transformer&#10;encoder, which has shown great success on the streaming automatic speech&#10;recognition task with hybrid or transducer-based models. We conduct an&#10;empirical evaluation of the proposed model on segment, context and memory sizes&#10;and we compare our approach to a transformer with a unidirectional mask.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.04488" label="2102.04488">
        <attvalues>
          <attvalue for="0" value="Wake Word Detection with Streaming Transformers" />
          <attvalue for="1" value="  Modern wake word detection systems usually rely on neural networks for&#10;acoustic modeling. Transformers has recently shown superior performance over&#10;LSTM and convolutional networks in various sequence modeling tasks with their&#10;better temporal modeling power. However it is not clear whether this advantage&#10;still holds for short-range temporal modeling like wake word detection.&#10;Besides, the vanilla Transformer is not directly applicable to the task due to&#10;its non-streaming nature and the quadratic time and space complexity. In this&#10;paper we explore the performance of several variants of chunk-wise streaming&#10;Transformers tailored for wake word detection in a recently proposed LF-MMI&#10;system, including looking-ahead to the next chunk, gradient stopping, different&#10;positional embedding methods and adding same-layer dependency between chunks.&#10;Our experiments on the Mobvoi wake word dataset demonstrate that our proposed&#10;Transformer model outperforms the baseline convolution network by 25% on&#10;average in false rejection rate at the same false alarm rate with a comparable&#10;model size, while still maintaining linear complexity w.r.t. the sequence&#10;length.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.02171" label="1808.02171">
        <attvalues>
          <attvalue for="0" value="Dialog-context aware end-to-end speech recognition" />
          <attvalue for="1" value="  Existing speech recognition systems are typically built at the sentence&#10;level, although it is known that dialog context, e.g. higher-level knowledge&#10;that spans across sentences or speakers, can help the processing of long&#10;conversations. The recent progress in end-to-end speech recognition systems&#10;promises to integrate all available information (e.g. acoustic, language&#10;resources) into a single model, which is then jointly optimized. It seems&#10;natural that such dialog context information should thus also be integrated&#10;into the end-to-end models to improve further recognition accuracy. In this&#10;work, we present a dialog-context aware speech recognition model, which&#10;explicitly uses context information beyond sentence-level information, in an&#10;end-to-end fashion. Our dialog-context model captures a history of&#10;sentence-level context so that the whole system can be trained with&#10;dialog-context information in an end-to-end manner. We evaluate our proposed&#10;approach on the Switchboard conversational speech corpus and show that our&#10;system outperforms a comparable sentence-level end-to-end speech recognition&#10;system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.09426" label="2104.09426">
        <attvalues>
          <attvalue for="0" value="Advanced Long-context End-to-end Speech Recognition Using&#10;  Context-expanded Transformers" />
          <attvalue for="1" value="  This paper addresses end-to-end automatic speech recognition (ASR) for long&#10;audio recordings such as lecture and conversational speeches. Most end-to-end&#10;ASR models are designed to recognize independent utterances, but contextual&#10;information (e.g., speaker or topic) over multiple utterances is known to be&#10;useful for ASR. In our prior work, we proposed a context-expanded Transformer&#10;that accepts multiple consecutive utterances at the same time and predicts an&#10;output sequence for the last utterance, achieving 5-15% relative error&#10;reduction from utterance-based baselines in lecture and conversational ASR&#10;benchmarks. Although the results have shown remarkable performance gain, there&#10;is still potential to further improve the model architecture and the decoding&#10;process. In this paper, we extend our prior work by (1) introducing the&#10;Conformer architecture to further improve the accuracy, (2) accelerating the&#10;decoding process with a novel activation recycling technique, and (3) enabling&#10;streaming decoding with triggered attention. We demonstrate that the extended&#10;Transformer provides state-of-the-art end-to-end ASR performance, obtaining a&#10;17.3% character error rate for the HKUST dataset and 12.0%/6.3% word error&#10;rates for the Switchboard-300 Eval2000 CallHome/Switchboard test sets. The new&#10;decoding method reduces decoding time by more than 50% and further enables&#10;streaming ASR with limited accuracy degradation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.19923" label="2310.19923">
        <attvalues>
          <attvalue for="0" value="Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long&#10;  Documents" />
          <attvalue for="1" value="  Text embedding models have emerged as powerful tools for transforming&#10;sentences into fixed-sized feature vectors that encapsulate semantic&#10;information. While these models are essential for tasks like information&#10;retrieval, semantic clustering, and text re-ranking, most existing open-source&#10;models, especially those built on architectures like BERT, struggle to&#10;represent lengthy documents and often resort to truncation. One common approach&#10;to mitigate this challenge involves splitting documents into smaller paragraphs&#10;for embedding. However, this strategy results in a much larger set of vectors,&#10;consequently leading to increased memory consumption and computationally&#10;intensive vector searches with elevated latency.&#10;  To address these challenges, we introduce Jina Embeddings 2, an open-source&#10;text embedding model capable of accommodating up to 8192 tokens. This model is&#10;designed to transcend the conventional 512-token limit and adeptly process long&#10;documents. Jina Embeddings 2 not only achieves state-of-the-art performance on&#10;a range of embedding-related tasks in the MTEB benchmark but also matches the&#10;performance of OpenAI's proprietary ada-002 model. Additionally, our&#10;experiments indicate that an extended context can enhance performance in tasks&#10;such as NarrativeQA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Embedding training has undergone significant evolution, transitioning from foundational techniques such as Latent Semantic Indexing (LSA) \cite{deerwester} and Latent Dirichlet Allocation (LDA) \cite{NIPS2001_296472c9} to the sophisticated prowess of pre-trained models like Sentence-BERT \cite{reimers2019sentence}. A notable shift in recent advancements is the emphasis on unsupervised contrastive learning, as showcased by works like \cite{gao2022simcse, wang2022text}. Pioneering models like Condenser \cite{gao2021condenser} and RetroMAE \cite{xiao2022retromae} have brought forth specialized architectures and pre-training methods explicitly designed for dense encoding and retrieval.&#10;&#10;The E5~\cite{wang2022text}, \JEmbeddingVOne~\cite{gunther2023jina}, and GTE~\cite{li2023general} collections of embedding models represent another leap forward. These models propose a holistic framework tailored for effective training across a myriad of tasks. This framework adopts a multi-stage contrastive training approach. An initial phase focuses on training using a vast collection of weak pairs sourced from public data, enhancing the model's domain generalization. Following this, a supervised fine-tuning stage employs a curated set of annotated text triples, representing diverse tasks. Together, these sequential stages yield state-of-the-art outcomes on the MTEB benchmark.&#10;&#10;Yet, despite such advancements, a glaring limitation persists: the $512$-token constraint on input sequences, stemming from foundational models like BERT. This cap is insufficient for encoding lengthy documents, often exceeding a page. ALiBi~\cite{press2022alibi} emerges as a promising solution, presenting a technique that sidesteps conventional positional embeddings and facilitates training on sequences exceeding $2048$ tokens. Notably, its typical application is centered around generative models, which inherently adopt a unidirectional bias, rendering it less suitable for embedding tasks.&#10;&#10;Effective evaluation remains paramount for embedding models, ensuring they meet the diverse demands of real-world applications. The BEIR benchmark \cite{thakur2021beir} stands out, offering evaluations across a set of retrieval tasks and settings. Similarly, the MTEB benchmark \cite{muennighoff2023mteb} highlights the extensive applicability of text embeddings, spanning a variety of tasks and languages. However, a notable gap in both benchmarks is their limited focus on encoding long documents --- a critical aspect for comprehensive embedding evaluation.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Learning Architectures, Artificial Intelligence, Information Science, Natural Language Processing, Text Embedding Models" />
        </attvalues>
      </node>
      <node id="2202.12191" label="2202.12191">
        <attvalues>
          <attvalue for="0" value="Finding Inverse Document Frequency Information in BERT" />
          <attvalue for="1" value="  For many decades, BM25 and its variants have been the dominant document&#10;retrieval approach, where their two underlying features are Term Frequency (TF)&#10;and Inverse Document Frequency (IDF). The traditional approach, however, is&#10;being rapidly replaced by Neural Ranking Models (NRMs) that can exploit&#10;semantic features. In this work, we consider BERT-based NRMs and study if IDF&#10;information is present in the NRMs. This simple question is interesting because&#10;IDF has been indispensable for the traditional lexical matching, but global&#10;features like IDF are not explicitly learned by neural language models&#10;including BERT. We adopt linear probing as the main analysis tool because&#10;typical BERT based NRMs utilize linear or inner-product based score&#10;aggregators. We analyze input embeddings, representations of all BERT layers,&#10;and the self-attention weights of CLS. By studying MS-MARCO dataset with three&#10;BERT-based models, we show that all of them contain information that is&#10;strongly dependent on IDF.&#10;" />
          <attvalue for="2" value="&#10;Since its inception in 1970's, BM25~\cite{robertson2009probabilistic} has been one of the most popular ranking functions for search engines. For a given search query $\mathrm{q}$, BM25 evaluates the relevance score of a candidate document $\mathrm{d} \in \mathrm{D}$ as&#10;\begin{equation}&#10; \mathsf{score}(\mathrm{q},\mathrm{d}) = \sum_{i=1}^{N}{\mathsf{IDF}(q_{i},\mathrm{D}) \cdot \mathsf{TF}(q_{i},\mathrm{d})},&#10;\end{equation}&#10;where $\mathrm{D}$ is the set of candidate documents, $\mathrm{q}$ contains $N$ query terms $\{q_{1}, q_{2}, \cdots, q_{N}\}$,&#10;$\mathsf{TF}(q_{i}, \mathrm{d})$ is the Term Frequency~(TF) that is the frequency of $q_{i}$ within the candidate document $\mathrm{d}$, and &#10;$\mathsf{IDF}(q_{i},\mathrm{D})$ is the Inverse Document Frequency~(IDF) that is the inverse of the term $q_{i}$'s frequency over the document set $\mathrm{D}$.&#10;Both TF and IDF have been successfully utilized for lexical matching algorithms such as BM25. The two, however, have quite different characteristics because TF is a local feature and IDF is a global feature. IDF reflects global information over the entire document set $\mathrm{D}$ while TF is a simple similarity measure between the pair $\mathrm{q}$ and $\mathrm{d}$.&#10;&#10;Since the advent of BERT~(Bidirectional Encoder Representations from Transformers~\cite{devlin2019bert}), a variety of BERT-based Neural Ranking Models~(NRMs) have been proposed~\cite{nogueira2019passage, khattab2020colbert}. A BERT-based NRM relies on the general-purpose language representation capability of BERT. For the specific goal of document ranking, a score aggregator is cascaded where it typically performs only a simple operation such as a linear regression or an inner-product. Despite the simplicity, NRMs can easily outperform the traditional lexical matching models. As in many other natural language tasks, the outstanding performance is known to be the result of BERT's excellent representations. On the other hand, BERT is trained only with sentence-level tasks such as Masked Language Modeling~(MLM). This means a global feature like IDF cannot be learned explicitly because its calculation requires an inspection of the entire document set $\mathrm{D}$. Considering that IDF has been a pivotal feature for document ranking, it becomes natural to ask if IDF information is available in NRMs and if we can improve NRMs using global features like IDF. &#10;&#10;In this work, we focus on IDF where the ground-truth IDF values of the tokens are pre-calculated using $\mathrm{D}$. The ground-truth values are used only for the investigations and they do not play any part in BERT-based NRMs. We first investigate if the ground-truth IDF values can be extracted from the input embeddings or from the BERT layer representations. A basic and popular probing technique known as linear probing~\cite{alain2016understanding} is adopted to avoid ambiguous interpretations -- if IDF information can be reliably extracted with a linear probing, it indicates that IDF information is not only present but also linearly decodable from the embeddings or representations. &#10;Additionally, we investigate the correlation between $\mathrm{q}$'s IDF vector and the self-attention weights that are used to form CLS in each head. If a high correlation is found for a head, it &#10;indicates that a strong linear dependency exists between $\mathsf{IDF}(q_{i},\mathrm{D})$ and token $i$'s attention value. The CLS vector of such a head can be interpreted as an IDF-weighted sum of the token representations. &#10;&#10;We performed our experiments with MS-MARCO, a large-scale document ranking dataset. For the NRM's language model, we considered pre-trained BERT, fine-tuned ColBERT~\cite{khattab2020colbert}, and prefix-tuned ColBERT~\cite{jung2021semi}. ColBERT is known as a state-of-the-art bi-encoder NRM, and prefix-tuned ColBERT is known to outperform ColBERT. Through experiments, we show that IDF information can be reliably extracted from BERT-based NRMs. This does not necessarily mean an NRM explicitly calculates or utilizes IDF, but it confirms that some information within NRM is strongly correlated with IDF. Furthermore, we found that IDF information can be more reliably extracted from BERT's input embeddings than from Word2Vec~\cite{mikolov2013efficient} or Glove~\cite{pennington2014glove} embeddings, and from the representations of a better performing NRM model than from a worse performing one. Overall, availability of IDF information correlated well with the performance of information retrieval. As for the attention weight vectors to form CLS, we found that the attention vectors of some heads strongly correlate with the IDF vector. BERT heads are known to perform different roles and our result indicates that some of the heads might play more IDF-like roles than the other heads. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Document Retrieval, Linguistics, Artificial Intelligence, Information Science, Language Modeling, Neural Ranking Models" />
        </attvalues>
      </node>
      <node id="2004.12832" label="2004.12832">
        <attvalues>
          <attvalue for="0" value="ColBERT: Efficient and Effective Passage Search via Contextualized Late&#10;  Interaction over BERT" />
          <attvalue for="1" value="  Recent progress in Natural Language Understanding (NLU) is driving fast-paced&#10;advances in Information Retrieval (IR), largely owed to fine-tuning deep&#10;language models (LMs) for document ranking. While remarkably effective, the&#10;ranking models based on these LMs increase computational cost by orders of&#10;magnitude over prior approaches, particularly as they must feed each&#10;query-document pair through a massive neural network to compute a single&#10;relevance score. To tackle this, we present ColBERT, a novel ranking model that&#10;adapts deep LMs (in particular, BERT) for efficient retrieval. ColBERT&#10;introduces a late interaction architecture that independently encodes the query&#10;and the document using BERT and then employs a cheap yet powerful interaction&#10;step that models their fine-grained similarity. By delaying and yet retaining&#10;this fine-granular interaction, ColBERT can leverage the expressiveness of deep&#10;LMs while simultaneously gaining the ability to pre-compute document&#10;representations offline, considerably speeding up query processing. Beyond&#10;reducing the cost of re-ranking the documents retrieved by a traditional model,&#10;ColBERT's pruning-friendly interaction mechanism enables leveraging&#10;vector-similarity indexes for end-to-end retrieval directly from a large&#10;document collection. We extensively evaluate ColBERT using two recent passage&#10;search datasets. Results show that ColBERT's effectiveness is competitive with&#10;existing BERT-based models (and outperforms every non-BERT baseline), while&#10;executing two orders-of-magnitude faster and requiring four orders-of-magnitude&#10;fewer FLOPs per query.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.14943" label="2110.14943">
        <attvalues>
          <attvalue for="0" value="Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight&#10;  Fine-Tuning" />
          <attvalue for="1" value="  A BERT-based Neural Ranking Model (NRM) can be either a crossencoder or a&#10;bi-encoder. Between the two, bi-encoder is highly efficient because all the&#10;documents can be pre-processed before the actual query time. In this work, we&#10;show two approaches for improving the performance of BERT-based bi-encoders.&#10;The first approach is to replace the full fine-tuning step with a lightweight&#10;fine-tuning. We examine lightweight fine-tuning methods that are adapter-based,&#10;prompt-based, and hybrid of the two. The second approach is to develop&#10;semi-Siamese models where queries and documents are handled with a limited&#10;amount of difference. The limited difference is realized by learning two&#10;lightweight fine-tuning modules, where the main language model of BERT is kept&#10;common for both query and document. We provide extensive experiment results for&#10;monoBERT, TwinBERT, and ColBERT where three performance metrics are evaluated&#10;over Robust04, ClueWeb09b, and MS-MARCO datasets. The results confirm that both&#10;lightweight fine-tuning and semi-Siamese are considerably helpful for improving&#10;BERT-based bi-encoders. In fact, lightweight fine-tuning is helpful for&#10;crossencoder, too&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.16457" label="2402.16457">
        <attvalues>
          <attvalue for="0" value="RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for&#10;  Short-form Open-Domain Question Answering" />
          <attvalue for="1" value="  Adaptive retrieval-augmented generation (ARAG) aims to dynamically determine&#10;the necessity of retrieval for queries instead of retrieving indiscriminately&#10;to enhance the efficiency and relevance of the sourced information. However,&#10;previous works largely overlook the evaluation of ARAG approaches, leading to&#10;their effectiveness being understudied. This work presents a benchmark,&#10;RetrievalQA, comprising 1,271 short-form questions covering new world and&#10;long-tail knowledge. The knowledge necessary to answer the questions is absent&#10;from LLMs; therefore, external information must be retrieved to answer&#10;correctly. This makes RetrievalQA a suitable testbed to evaluate existing ARAG&#10;methods. We observe that calibration-based methods heavily rely on threshold&#10;tuning, while vanilla prompting is inadequate for guiding LLMs to make reliable&#10;retrieval decisions. Based on our findings, we propose Time-Aware Adaptive&#10;Retrieval (TA-ARE), a simple yet effective method that helps LLMs assess the&#10;necessity of retrieval without calibration or additional training. The dataset&#10;and code will be available at https://github.com/hyintell/RetrievalQA&#10;" />
          <attvalue for="2" value="&#10;&#10;Retrieval-augmented generation (RAG) &#10; \cite{guu2020retrieval, lewis2020retrieval, ram-etal-2023-context}&#10;that augments large language models (LLMs) with retrieval of relevant information has become increasingly popular in knowledge-intensive tasks, including open-domain question-answering (QA) \cite{zhang-etal-2023-large, kasai2023realtime, cui2023chatlaw,zhang-etal-2023-survey-efficient}.&#10;However, standard RAG methods conduct retrieval indiscriminately, irrespective of the input query, which may result in suboptimal task performance and increased inference costs \cite{gao2024retrievalaugmented}.&#10;On one hand, LLMs encode vast knowledge in parameters through large-scale pre-training, enabling them to effortlessly handle straightforward&#10;queries without retrieval \cite{mallen-etal-2023-trust}. On the other hand, the retrieved context may contain noise and irrelevant information, and augmenting noisy context can potentially distract LLMs, thereby impeding task performance &#10;\cite{pmlr-v202-shi23a}.&#10;&#10;To alleviate the limitations of RAG mentioned above,&#10;recent studies advocate for &#10;adaptive RAG (ARAG),&#10;which dynamically determines retrieval necessity and relies only on LLMs' parametric knowledge when deemed unnecessary \cite{feng2023trends}.&#10;However, the effectiveness of these methods is understudied, as there is no suitable benchmark and evaluation.&#10;ARAG approaches can be categorized into calibration-based and model-based judgement.&#10;Calibration-based methods \cite{mallen-etal-2023-trust, jiang-etal-2023-active, asai2023selfrag}, while effective, trigger retrieval only when a metric surpasses a pre-defined threshold.&#10;For example, \cite{mallen-etal-2023-trust} heuristically retrieve when the popularity of an entity on Wikipedia is below a certain threshold;&#10;\cite{jiang-etal-2023-active} trigger retrieval if any token in the temporarily generated sentence has low confidence.&#10;Clearly, these ad-hoc calibration-based methods are suboptimal, as we need to tune thresholds for different datasets and models to &#10;balance task performance and inference overheads.&#10;To obviate the hyperparameter threshold, model-based methods \cite{feng2023knowledge, ren2023investigating} directly prompt LLMs for retrieval decisions, given the observation that LLMs can acknowledge their knowledge boundaries to some extent \cite{kadavath2022language, yin-etal-2023-large}.&#10;These methods undergo separate evaluations, and their effectiveness remains ambiguous due to the limited scope of the assessments.&#10;&#10;In this paper, we investigate to what extent LLMs can perform calibration-free adaptive retrieval &#10;via prompting.&#10;To answer this question, we need to evaluate whether LLMs retrieve only when necessary.&#10;This requests a benchmark that distinguishes between questions that can be answered using LLMs' parametric knowledge and those that require external information through retrieval.&#10;Nevertheless, commonly used open-domain QA datasets \cite{rajpurkar-etal-2016-squad, joshi-etal-2017-triviaqa, kwiatkowski-etal-2019-natural, mallen-etal-2023-trust} fail to fulfil this purpose, as various LLMs have distinct sizes and levels of pre-trained knowledge, making them inadequately assess the necessity of external retrieval for LLMs.&#10;&#10;To fill this gap, we create RetrievalQA, a short-form QA dataset, covering new world and long-tail knowledge and spanning diverse topics.&#10;We ensure the knowledge necessary to answer the questions is absent from LLMs. Therefore, LLMs must truthfully decide whether to retrieve to be able to answer the questions correctly. &#10;RetrievalQA enables us to evaluate the effectiveness of ARAG approaches, an aspect predominantly overlooked in prior studies and recent RAG evaluation systems \cite{chen2023benchmarking, saadfalcon2023ares, es2023ragas}, which focus only on task performance, the relevance of retrieval context or the faithfulness of answers.&#10;&#10;Using RetrievalQA as a testbed, we benchmark both calibration-based and model-based methods with varying sizes of LLMs.&#10;As shown in Fig.\ref{fig_sankey_gpt35},&#10;we find calibration-based Self-RAG requires threshold tuning to balance QA performance and retrieval efficiency, while vanilla prompting is insufficient in guiding LLMs to make reliable retrieval decisions.&#10;As an initial effort, we propose Time-Aware Adaptive REtrieval (TA-ARE), a simple yet effective method to improve ARAG via in-context learning (ICL; \cite{NEURIPS2020_1457c0d6}), obviating the need for calibration or additional training.&#10;&#10;To sum up, this paper makes the following contributions: &#10;\circled{1} we create a new dataset RetrievalQA to assess ARAG for short-form open-domain QA;&#10;\circled{2} &#10;we benchmark existing methods and conduct extensive analysis,&#10;finding that vanilla prompting is insufficient in guiding LLMs to make reliable retrieval decisions;&#10;\circled{3} we then propose TA-ARE, a simple yet effective method to help LLMs assess the necessity of retrieval without calibration or additional training.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Language Model Evaluation, Adaptive Generation Methods, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2211.07886" label="2211.07886">
        <attvalues>
          <attvalue for="0" value="A Survey for Efficient Open Domain Question Answering" />
          <attvalue for="1" value="  Open domain question answering (ODQA) is a longstanding task aimed at&#10;answering factual questions from a large knowledge corpus without any explicit&#10;evidence in natural language processing (NLP). Recent works have predominantly&#10;focused on improving the answering accuracy and achieved promising progress.&#10;However, higher accuracy often comes with more memory consumption and inference&#10;latency, which might not necessarily be efficient enough for direct deployment&#10;in the real world. Thus, a trade-off between accuracy, memory consumption and&#10;processing speed is pursued. In this paper, we provide a survey of recent&#10;advances in the efficiency of ODQA models. We walk through the ODQA models and&#10;conclude the core techniques on efficiency. Quantitative analysis on memory&#10;cost, processing speed, accuracy and overall comparison are given. We hope that&#10;this work would keep interested scholars informed of the advances and open&#10;challenges in ODQA efficiency research, and thus contribute to the further&#10;development of ODQA efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.09955" label="2305.09955">
        <attvalues>
          <attvalue for="0" value="Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized&#10;  Language Models" />
          <attvalue for="1" value="  By design, large language models (LLMs) are static general-purpose models,&#10;expensive to retrain or update frequently. As they are increasingly adopted for&#10;knowledge-intensive tasks, it becomes evident that these design choices lead to&#10;failures to generate factual, relevant, and up-to-date knowledge. To this end,&#10;we propose Knowledge Card, a modular framework to plug in new factual and&#10;relevant knowledge into general-purpose LLMs. We first introduce knowledge&#10;cards -- specialized language models trained on corpora from specific domains&#10;and sources. Knowledge cards serve as parametric repositories that are selected&#10;at inference time to generate background knowledge for the base LLM. We then&#10;propose three content selectors to dynamically select and retain information in&#10;documents generated by knowledge cards, specifically controlling for relevance,&#10;brevity, and factuality of outputs. Finally, we propose two complementary&#10;integration approaches to augment the base LLM with the (relevant, factual)&#10;knowledge curated from the specialized LMs. Through extensive experiments, we&#10;demonstrate that Knowledge Card achieves state-of-the-art performance on six&#10;benchmark datasets. Ultimately, Knowledge Card framework enables dynamic&#10;synthesis and updates of knowledge from diverse domains. Its modularity will&#10;ensure that relevant knowledge can be continuously updated through the&#10;collective efforts of the research community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08718" label="2112.08718">
        <attvalues>
          <attvalue for="0" value="Prompt Tuning GPT-2 language model for parameter-efficient domain&#10;  adaptation of ASR systems" />
          <attvalue for="1" value="  Automatic Speech Recognition (ASR) systems have found their use in numerous&#10;industrial applications in very diverse domains creating a need to adapt to new&#10;domains with small memory and deployment overhead. In this work, we introduce&#10;domain-prompts, a methodology that involves training a small number of domain&#10;embedding parameters to prime a Transformer-based Language Model (LM) to a&#10;particular domain. Using this domain-adapted LM for rescoring ASR hypotheses&#10;can achieve 7-13% WER reduction for a new domain with just 1000 unlabeled&#10;textual domain-specific sentences. This improvement is comparable or even&#10;better than fully fine-tuned models even though just 0.02% of the parameters of&#10;the base LM are updated. Additionally, our method is deployment-friendly as the&#10;learnt domain embeddings are prefixed to the input to the model rather than&#10;changing the base model architecture. Therefore, our method is an ideal choice&#10;for on-the-fly adaptation of LMs used in ASR systems to progressively scale it&#10;to new domains.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic Speech Recognition (ASR) systems form a key component of various products across industry. With recent advancements \cite{graves2012sequence, chan2016listen, zhang2020Transformer}, they have been deployed in a wide range of domains, including healthcare, travel reservations, and customer services etc. &#10;A typical technique to improve the performance of these systems is to do a rescoring of the $n$-best hypotheses with an external Language Model (LM) \cite{chan2016listen}.&#10;Recent pretrained Transformer-based LMs such as GPT-2 \cite{radford2019language} and BERT \cite{devlin2018bert}&#10;have been shown \cite{irie2019language} to be more effective than conventional LSTM based LMs for rescoring. However, their use in an industrial ASR system that needs to incrementally support new domains poses the following challenge. As showcased in \cite{shenoy21_interspeech, Shenoy_2021}, domain-specific data is useful for improving performance in a domain. However, retraining or maintaining copies of Transformer-based LMs for each domain separately is not scalable as updating and storing millions of parameters comes with a large cost. Therefore, a need for an efficient domain-adaptation method for such LMs is evident. &#10;\cite{9746748, 9688109, jain2020contextual} used external knowledge, memory and context respectively to improve performance in specific difficult domains, while &#10;\cite{park2010improved, alumae2013multi} adapted the neural LM used within the system. However, to the best of our knowledge, ours is the first work to propose and study methods to do efficient domain-adaptation of Transformer-based LMs to benefit ASR systems. Language modeling literature \cite{pfeiffer2020adapterhub, lester2021power, li2021prefix} introduced novel methodologies to solve a related problem of efficiently adapting such LMs to specific tasks. Instead of fine-tuning and storing millions of parameters for each task, they propose augmenting the frozen task-agnostic model with a handful of task-specific trainable parameters. &#10;For example, AdapterHub \cite{pfeiffer2020adapterhub} introduced new task-specific layers in conjunction to frozen pre-trained weights of LMs. &#10;More recent models, such as GPT-3 \cite{brown2020language}, are able to solve new tasks with the help of just the textual descriptions of the task (called prompts). &#10;&#10;Extending \cite{DBLP:journals/corr/abs-2110-06502}, the focus of this work is to adapt such LMs to different domains of the same task rather than solving multiple tasks. Our objective is to learn a small set of domain-specific parameters to score ASR hypotheses better than the base Transformer-based LM without the domain data. Drawing ideas from prompt-tuning \cite{lester2021power} for task adaptation, we introduce domain-prompts for our goal.&#10;We define domain-prompts as domain-specific embeddings, which when prefixed to the sequence of token embeddings, and passed through a pretrained Transformer LM, return the probability distribution of the next token, close to that given by a fully domain-adapted LM. &#10;Our main contributions are summarized as follows: &#10;(1) we introduce a new methodology domain-prompts, which is the first attempt to apply prompt-tuning for parameter-efficient domain-adaptation of Transformer-based LMs for their use in ASR systems, &#10;(2) In new domains with limited data, we demonstrate that rescoring ASR hypotheses with LM adapted using our method can achieve 7-13\% WER reduction while using a handful of additional domain-specific parameters&#10;(3) Along with saving memory and training cost, domain-prompts can match or even beat the performance of fully fine-tuned models with no change to the deployment of the base model, thereby making it the ideal choice for on-the-fly domain adaptation for industrial ASR systems.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Domain Adaptation, Artificial Intelligence, Speech Recognition, Language Modeling" />
        </attvalues>
      </node>
      <node id="2110.06502" label="2110.06502">
        <attvalues>
          <attvalue for="0" value="Prompt-tuning in ASR systems for efficient domain-adaptation" />
          <attvalue for="1" value="  Automatic Speech Recognition (ASR) systems have found their use in numerous&#10;industrial applications in very diverse domains. Since domain-specific systems&#10;perform better than their generic counterparts on in-domain evaluation, the&#10;need for memory and compute-efficient domain adaptation is obvious.&#10;Particularly, adapting parameter-heavy transformer-based language models used&#10;for rescoring ASR hypothesis is challenging. In this work, we overcome the&#10;problem using prompt-tuning, a methodology that trains a small number of domain&#10;token embedding parameters to prime a transformer-based LM to a particular&#10;domain. With just a handful of extra parameters per domain, we achieve much&#10;better perplexity scores over the baseline of using an unadapted LM. Despite&#10;being parameter-efficient, these improvements are comparable to those of&#10;fully-fine-tuned models with hundreds of millions of parameters. We replicate&#10;our findings in perplexity numbers to Word Error Rate in a domain-specific ASR&#10;system for one such domain.&#10;" />
          <attvalue for="2" value="&#10;Automatic Speech Recognition (ASR) systems form a key component of various products across industry. Many of these ASR systems rely on a complex Acoustic Model (AM) whose output is rescored by a domain-specific Language Model (LM). &#10;As we use ASR systems in new domains, the memory, maintenance and data-collection costs for these domain-specific LMs increase.&#10;Particularly, with advent of parameter-heavy Transformer based LMs \cite{devlin-etal-2019-bert}, maintaining multiple domain-specific LMs is practically infeasible. While on the other hand, using a generic LM for all domains falls short in performance when compared to multiple domain-specific LMs. Therefore, a need for a middle ground between performance and costs is evident. &#10;&#10;To overcome this problem, we bring forward a methodology based on recently proposed Prompt Tuning. \cite{lester2021power} introduced this idea of learning the token embeddings of the prompt used to prime a LM to a particular task. Prompts are special tokens describing a task which when appended to the input data sample, helps the model understand and use this problem description to better solve the task. For example, to solve the machine translation task, instead of fine-tuning the Transformer model with corresponding dataset, one can achieve comparable performance by just showing text describing machine translation to the powerful Transformer-based LM. In prompt tuning, instead of providing this prompt manually to the model, one learn it from the labelled examples from the task.&#10;&#10;To the best of our knowledge, we are the first one to apply prompt-tuning for domain-adaptation of ASR systems. We find that one can learn the prompt-embeddings for not only different tasks but also different domains of the same task. This means there exists a domain description, which when passed to model as prefix, helps it better score sentences from that domain. We hypothesize that this is due to the fact that self-attention in Transformer model will create a positive interaction between domain embeddings (description) and the words from sentence, resulting in better prediction of the next token. This has an important usecase in multi-domain ASR systems, where learning a small number of parameters corresponding to domain prompt embedding can save costs associated with maintaining separate copies of LMs for each domain. Particularly, with Transformer models with millions of parameters, fine-tuning can be very slow and requires a large amount of domain data to prevent overfitting. However, with prompt-tuning one can achieve similar performance with &lt;0.1\% of the parameters required for fine-tuning. We provide experimental evidence by comparing the performance of prompt-tuned and fine-tuned versions of LMs to score sentences from different domains of a dialog dataset. We also provide ablations on different sizes of prompt. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Domain Adaptation, Artificial Intelligence, Speech Recognition, Language Modeling" />
        </attvalues>
      </node>
      <node id="2305.17926" label="2305.17926">
        <attvalues>
          <attvalue for="0" value="Large Language Models are not Fair Evaluators" />
          <attvalue for="1" value="  In this paper, we uncover a systematic bias in the evaluation paradigm of&#10;adopting large language models~(LLMs), e.g., GPT-4, as a referee to score and&#10;compare the quality of responses generated by candidate models. We find that&#10;the quality ranking of candidate responses can be easily hacked by simply&#10;altering their order of appearance in the context. This manipulation allows us&#10;to skew the evaluation result, making one model appear considerably superior to&#10;the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries&#10;with ChatGPT as an evaluator. To address this issue, we propose a calibration&#10;framework with three simple yet effective strategies: 1) Multiple Evidence&#10;Calibration, which requires the evaluator model to generate multiple evaluation&#10;evidence before assigning ratings; 2) Balanced Position Calibration, which&#10;aggregates results across various orders to determine the final score; 3)&#10;Human-in-the-Loop Calibration, which introduces a balanced position diversity&#10;entropy to measure the difficulty of each example and seeks human assistance&#10;when needed. We also manually annotate the &quot;win/tie/lose&quot; outcomes of responses&#10;from ChatGPT and Vicuna-13B in the Vicuna Benchmark's question prompt, and&#10;extensive experiments demonstrate that our approach successfully mitigates&#10;evaluation bias, resulting in closer alignment with human judgments. We release&#10;our code and human annotation at \url{https://github.com/i-Eval/FairEval} to&#10;facilitate future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Bias Mitigation Techniques, Computer Science, Human-AI Alignment, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2402.04925" label="2402.04925">
        <attvalues>
          <attvalue for="0" value="TP-Aware Dequantization" />
          <attvalue for="1" value="  In this paper, we present a novel method that reduces model inference latency&#10;during distributed deployment of Large Language Models (LLMs). Our contribution&#10;is an optimized inference deployment scheme that address the current&#10;limitations of state-of-the-art quantization kernels when used in conjunction&#10;with Tensor Parallel (TP). Our method preserves data locality in GPU memory&#10;access patterns and exploits a priori knowledge of TP to reduce global&#10;communication. We demonstrate an up to 1.81x speedup over existing methods for&#10;Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer&#10;problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.&#10;" />
          <attvalue for="2" value="&#10;&#10;Given the recent advancement of LLMs, deployment optimizations are becoming more crucial as the size of state-of-the-art LLMs increase in scale. As these these models continue to grow, so does the need to optimize the increasingly parallel and increasingly distributed workload requirements of modern-day deep learning inference. Strategies like GPTQ \cite{frantar_gptq_2023} and Tensor Parallel (TP) \cite{noauthor_tensor_nodate} are hence essential in achieving high-throughput performance. Our method is motivated by several key properties of GPTQ, TP and General Matrix Multiplication (GEMM). We build on these existing methods and present a key innovation that helps maximize memory throughput and reduce latency. Our method shows up to a 1.81x speedup on Llama-70B and up to a 1.78x speedup on Granite-20B MLP layer problem sizes. We achieve this by reducing global communication and enforcing data locality.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Parallel Computing, Engineering, Model Optimization, Artificial Intelligence, Mathematics, Distributed Computing" />
        </attvalues>
      </node>
      <node id="2303.12767" label="2303.12767">
        <attvalues>
          <attvalue for="0" value="Can we trust the evaluation on ChatGPT?" />
          <attvalue for="1" value="  ChatGPT, the first large language model (LLM) with mass adoption, has&#10;demonstrated remarkable performance in numerous natural language tasks. Despite&#10;its evident usefulness, evaluating ChatGPT's performance in diverse problem&#10;domains remains challenging due to the closed nature of the model and its&#10;continuous updates via Reinforcement Learning from Human Feedback (RLHF). We&#10;highlight the issue of data contamination in ChatGPT evaluations, with a case&#10;study of the task of stance detection. We discuss the challenge of preventing&#10;data contamination and ensuring fair model evaluation in the age of closed and&#10;continuously trained models.&#10;" />
          <attvalue for="2" value="&#10;&#10;ChatGPT~\cite{chatgpt} has become the most prominent and widely-adopted pre-trained large language model (LLM) thanks to its impressive capabilities to perform a plethora of natural language tasks and its public accessibility. &#10;Although significant concerns regarding LLMs, particularly their tendency to ``hallucinate'' (or ``making things up'') and generation of biased or harmful content in scale have been raised~\cite{bender2021dangers, alkaissi2023artificial}, ChatGPT is becoming a common tool not only for everyday tasks such as essay writing, translation, and summarization~\cite{taecharungroj2023can,patel2023chatgpt}, but also for more sophisticated tasks such as code generation, debugging~\cite{sobania2023analysis}, and mathematical problem-solving~\cite{frieder2023mathematical}. &#10;With more than 100 million users within two months after its launch~\cite{milmo2023chatgpt} and its abilities pass hard exams like bar exam~\cite{terwiesch} and medical licensing exam~\cite{kung2023performance}, ChatGPT has stirred public perception of AI and has been touted as the paradigm for the next-generation search engine and writing assistant, which is already being tested by Microsoft's Bing search and Office products~\cite{bing}. Beyond commercial interests, LLMs are also being tested for assisting scientific research~\cite{stokel2023chatgpt,dowling2023chatgpt,van2023chatgpt,wu2023large}. &#10;&#10;Although OpenAI---the creators of ChatGPT---performed internal tests, they do not cover all problem domains. Although the excellent general performance of ChatGPT is evident, it is still important to quantitatively characterize its performance on specific tasks to better understand and contextualize the model. &#10;Note that, given that it is currently not possible for a user to fine-tune ChatGPT, one can only evaluate it with a few-shot/zero-shot setting---a highly desirable setting that requires close to no annotated data.&#10;A recent study showed that although ChatGPT performs generally well in many tasks, it has different strengths and weaknesses for different tasks and does not tend to beat the SOTA models~\cite{kocon2023chatgpt}.&#10;&#10;However, given that the ChatGPT is a closed model without information about its training dataset and how it is currently being trained, there is a large loxodonta mammal in the room: how can we know whether ChatGPT has not been contaminated with the evaluation datasets? &#10;&#10;Preventing data leakage (training-test contamination) is one of the most fundamental principles of machine learning because such leakage makes evaluation results unreliable.&#10;It has been shown that LLMs can also be significantly affected by data leakage, both by the leakage of labels and even by the leakage of dataset without labels~\cite{min2022rethinking,brown2020language,gpt4}.&#10;Given that the ChatGPT's training datasets are unknown and that ChatGPT is constantly updated, partly based on human inputs from more than 100 million users via Reinforcement Learning from Human Feedback (RLHF)~\cite{chatgpt}, it is impossible to ascertain the lack of data leakage, especially for the datasets that have been on the internet. &#10;&#10;As far as it has been known, ChatGPT is trained in a three-step process. First, an initial LLM (GPT 3/3.5) is fine-tuned in a supervised manner on a dataset curated by asking hired human annotators to write what they think is the desired output to prompts submitted to the OpenAI API. Next, a set of prompts is sampled from a larger collection of prompts submitted to the OpenAI API. For each prompt in this set, the LLM produces multiple responses, which are then ranked by human annotators who are asked to indicate their preferred response. The second step then trains a reward model (RM) on this dataset of response-ranking pairs to mimic the human ranking. This step keeps the LLM frozen and solely trains the RM. Finally, the LLM is made to generate responses to a set of prompts, which were not included in the previous steps, but submitted to the OpenAI API nevertheless. The now-frozen RM is used as a reward function, and the LLM is further fine-tuned to maximize this reward using the Proximal Policy Optimization (PPO) algorithm~\cite{schulman2017proximal}. &#10;&#10;Thus, if OpenAI continuously updates its models, by using queries submitted by researchers who wanted to evaluate ChatGPT's performance on various Natural Language Processing (NLP) tasks, it is likely that ChatGPT is already contaminated with the test datasets of many NLP tasks, which can lead to performance overestimation in NLP tasks. &#10;Such contamination has been documented in the training data of other language models~\cite{brown2020language,dodge2021documenting,carlini2020extracting}.&#10;&#10;It is important to highlight a distinction between two kinds of contamination acknowledged in literature~\cite{dodge2021documenting}: (1) the case where both the task input and labels are leaked to the model via training versus (2) the case where just the input is exposed. The latter is surely a smaller concern. However, even without the correct labels, exposure to the text in the same domain has been documented to increase the performance of the model to the corresponding NLP task~\cite{min2022rethinking}. &#10;Although we do not have any documented evidence that the ground-truth output answers/labels of the NLP tasks were submitted to the platform and the ChatGPT model has been trained with such data, we cannot exclude such possibility either. &#10;The annotator-generated responses to queries submitted to OpenAI during the RLHF step could potentially match the input text with output labels of the right kind; it is not possible to ensure no one has exposed certain input-label pairs to the model, for instance, via a few-shot learning experiment.&#10;Given that language models show competitive performance in classification tasks despite poorly labeled data~\cite{min2022rethinking,garg2022can}, we cannot discard the possibility that the RLHF pipeline might essentially be a weaker variant of type (1) contamination.&#10;&#10;Here, we use a case study of a stance detection problem~\cite{kuccuk2020stance} to raise awareness on this issue of data leakage and ask a question about how we should approach the evaluation of closed models. Stance detection is a fundamental computational tool that is widely used across many disciplines, including political science and communication studies.&#10;It refers to the task of extracting the standpoint (e.g., Favor, Against, or Neither) towards a target from a given text. The task becomes more challenging when the texts are from social media like Twitter because of the presence of abbreviations, hashtags, URLs, spelling errors, and the incoherent nature of tweets. &#10;Recent studies have claimed that ChatGPT outperforms most of the previous models proposed for this task~\cite{zhang2022would} on a few existing evaluation datasets, such as the SemEval 2016 Task6 dataset~\cite{mohammad2016semeval,mohammad2017stance} and P-stance~\cite{li2021p}, even in a zero-shot setting where the model was not fine-tuned on the task-specific training data. &#10;&#10;Can this result be due to the data leakage and contamination of the model? &#10;Could this study itself have contaminated the ChatGPT model?&#10;Although it is not possible to definitely answer these questions, it is also impossible to rule out the possibility of contamination without the model owners' in-depth analysis. &#10;&#10;Following its release on Nov $30^{th}\ 2022$, on Dec $15^{th}$ $2022$, Jan $9^{th}$, Jan $30^{th}$, Feb $9^{th}$, and Feb $13^{th}\ 2023$, ChatGPT has been updated multiple times. While most of these releases updated the model itself, it is our understanding that the February releases were about handling more users to the platform, optimizing for speed, and the offering of ChatGPT plus---a subscription plan which provides priority access to new features, and faster response times.&#10;Given that there has been at least one study that evaluated ChatGPT's performance on stance detection tasks~\cite{zhang2022would}, and that newer versions of ChatGPT are more likely to be exposed to past queries to the platform, an opportunity arises to test whether the performance of the newer versions of ChatGPT on stance detection has been substantially improved after the study by~\cite{zhang2022would}. &#10;&#10;As we will present below, we do see an overall improvement in the performance before and after the publication of the stance detection evaluation paper~\cite{zhang2022would}. &#10;Of course, there is an alternative explanation that the model simply got better. &#10;However, we would also like to note that OpenAI has been updating the model primarily to address the model's problematic behaviors by making it more restricted, which led to the observation, although largely anecdotal, that the model has become `less impressive.' &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Data Contamination Prevention, Linguistics, Language Model Evaluation, Mathematics, Fair Model Assessment" />
        </attvalues>
      </node>
      <node id="2301.13867" label="2301.13867">
        <attvalues>
          <attvalue for="0" value="Mathematical Capabilities of ChatGPT" />
          <attvalue for="1" value="  We investigate the mathematical capabilities of two iterations of ChatGPT&#10;(released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on&#10;publicly available datasets, as well as hand-crafted ones, using a novel&#10;methodology. In contrast to formal mathematics, where large databases of formal&#10;proofs are available (e.g., the Lean Mathematical Library), current datasets of&#10;natural-language mathematics, used to benchmark language models, either cover&#10;only elementary mathematics or are very small. We address this by publicly&#10;releasing two new datasets: GHOSTS and miniGHOSTS. These are the first&#10;natural-language datasets curated by working researchers in mathematics that&#10;(1) aim to cover graduate-level mathematics, (2) provide a holistic overview of&#10;the mathematical capabilities of language models, and (3) distinguish multiple&#10;dimensions of mathematical reasoning. These datasets also test whether ChatGPT&#10;and GPT-4 can be helpful assistants to professional mathematicians by emulating&#10;use cases that arise in the daily professional activities of mathematicians. We&#10;benchmark the models on a range of fine-grained performance metrics. For&#10;advanced mathematics, this is the most detailed evaluation effort to date. We&#10;find that ChatGPT can be used most successfully as a mathematical assistant for&#10;querying facts, acting as a mathematical search engine and knowledge base&#10;interface. GPT-4 can additionally be used for undergraduate-level mathematics&#10;but fails on graduate-level difficulty. Contrary to many positive reports in&#10;the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of&#10;selection bias), their overall mathematical performance is well below the level&#10;of a graduate student. Hence, if your goal is to use ChatGPT to pass a&#10;graduate-level math exam, you would be better off copying from your average&#10;peer!&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1605.01655" label="1605.01655">
        <attvalues>
          <attvalue for="0" value="Stance and Sentiment in Tweets" />
          <attvalue for="1" value="  We can often detect from a person's utterances whether he/she is in favor of&#10;or against a given target entity -- their stance towards the target. However, a&#10;person may express the same stance towards a target by using negative or&#10;positive language. Here for the first time we present a dataset of&#10;tweet--target pairs annotated for both stance and sentiment. The targets may or&#10;may not be referred to in the tweets, and they may or may not be the target of&#10;opinion in the tweets. Partitions of this dataset were used as training and&#10;test sets in a SemEval-2016 shared task competition. We propose a simple stance&#10;detection system that outperforms submissions from all 19 teams that&#10;participated in the shared task. Additionally, access to both stance and&#10;sentiment annotations allows us to explore several research questions. We show&#10;that while knowing the sentiment expressed by a tweet is beneficial for stance&#10;classification, it alone is not sufficient. Finally, we use additional&#10;unlabeled data through distant supervision techniques and word embeddings to&#10;further improve stance classification.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14538" label="2305.14538">
        <attvalues>
          <attvalue for="0" value="Cascaded Beam Search: Plug-and-Play Terminology-Forcing For Neural&#10;  Machine Translation" />
          <attvalue for="1" value="  This paper presents a plug-and-play approach for translation with terminology&#10;constraints. Terminology constraints are an important aspect of many modern&#10;translation pipelines. In both specialized domains and newly emerging domains&#10;(such as the COVID-19 pandemic), accurate translation of technical terms is&#10;crucial. Recent approaches often train models to copy terminologies from the&#10;input into the output sentence by feeding the target terminology along with the&#10;input. But this requires expensive training whenever the underlying language&#10;model is changed or the system should specialize to a new domain. We propose&#10;Cascade Beam Search, a plug-and-play terminology-forcing approach that requires&#10;no training. Cascade Beam Search has two parts: 1) logit manipulation to&#10;increase the probability of target terminologies and 2) a cascading beam setup&#10;based on grid beam search, where beams are grouped by the number of&#10;terminologies they contain. We evaluate the performance of our approach by&#10;competing against the top submissions of the WMT21 terminology translation&#10;task. Our plug-and-play approach performs on par with the winning submissions&#10;without using a domain-specific language model and with no additional training.&#10;" />
          <attvalue for="2" value="&#10;&#10;Terminology translation is a key challenge in modern machine translation systems. While most translation systems are trained to be generalists, applications that require accurate translation of terminology are plentiful (e.g., in the bio-medical or legal domains). In addition, new terms and domains can emerge over time, rendering large pretrained language models outdated (e.g., COVID-19). Not only is it difficult and expensive to come by parallel corpora for specialized or emerging domains, but the periodic retraining or fine-tuning of machine translation models can also be energy intensive.&#10;&#10;In this context, the use of word- or phrase-level terminology lists as also used by human professional translators can be an interesting resource to guide translation. Such lists can be created in a timely manner even for newly emerging domains and if used effectively, they offer the possibility for a flexible neural machine translation (NMT) pipeline that can become an expert in any domain.&#10;&#10;NMT with terminology constraints has gained significant attention in recent years culminating in the WMT21 shared task: Machine Translation using Terminologies \cite{alam-etal-2021-findings}. Participants were asked to translate COVID-related sentences across five different language pairs.&#10;&#10;One of the most successful early approaches to terminology translation comes from \cite{hokamp-liu-2017-lexically} in the form of grid beam search (GBS). In this approach beam search is run on multiple levels in parallel, where each level contains the best beams with a given number of fulfilled constraints. &#10;This approach can be summarized as trying to place a constraint in every possible position until it finds a good one.&#10;Although very effective, GBS' runtime increases linearly with the number of constraints. To mitigate this issue \cite{post2018fast} propose dynamic beam allocation, which reduces the computational overhead to a constant factor.&#10;More recently, \cite{dinu2019training} have argued that GBS can be brittle under realistic conditions and propose instead to train a language model to copy target terminologies, after first adding these to the input. Indeed all the competitors in the WMT21 competition used some version of this approach.&#10;&#10;We argue that it may be time to revisit constrained decoding methods. Our Cascaded Beam Search is based on GBS, but we show that with two important modifications one can achieve significant performance improvements: 1) allowing arbitrary constraint tokenizations and 2) using a cascade level per full constraint instead of per constraint token. These modifications lead to improvements of more than 6 BLEU points across all datasets tested.&#10;&#10;To compare against the state-of-the-art in terminology translation, we also evaluate our approach on the WMT21 competition dataset with a generalist underlying multilingual model.&#10;Our decoding approach achieves a near $100\%$ appearance of terminologies, while retaining the BLEU score of the underlying model. This result beats all competitors in terms of terminology appearance, but cannot surpass the BLEU scores of the fine-tuned, competition-winning models as our underlying model &#10;is limited by the weaker domain-specific translation quality of the underlying model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Terminology Constraints, Computer Science, Machine Translation, Linguistics, Artificial Intelligence, Language Modeling" />
        </attvalues>
      </node>
      <node id="1906.01105" label="1906.01105">
        <attvalues>
          <attvalue for="0" value="Training Neural Machine Translation To Apply Terminology Constraints" />
          <attvalue for="1" value="  This paper proposes a novel method to inject custom terminology into neural&#10;machine translation at run time. Previous works have mainly proposed&#10;modifications to the decoding algorithm in order to constrain the output to&#10;include run-time-provided target terms. While being effective, these&#10;constrained decoding methods add, however, significant computational overhead&#10;to the inference step, and, as we show in this paper, can be brittle when&#10;tested in realistic conditions. In this paper we approach the problem by&#10;training a neural MT system to learn how to use custom terminology when&#10;provided with the input. Comparative experiments show that our method is not&#10;only more effective than a state-of-the-art implementation of constrained&#10;decoding, but is also as fast as constraint-free decoding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.07890" label="2006.07890">
        <attvalues>
          <attvalue for="0" value="FinEst BERT and CroSloEngual BERT: less is more in multilingual models" />
          <attvalue for="1" value="  Large pretrained masked language models have become state-of-the-art&#10;solutions for many NLP problems. The research has been mostly focused on&#10;English language, though. While massively multilingual models exist, studies&#10;have shown that monolingual models produce much better results. We train two&#10;trilingual BERT-like models, one for Finnish, Estonian, and English, the other&#10;for Croatian, Slovenian, and English. We evaluate their performance on several&#10;downstream tasks, NER, POS-tagging, and dependency parsing, using the&#10;multilingual BERT and XLM-R as baselines. The newly created FinEst BERT and&#10;CroSloEngual BERT improve the results on all tasks in most monolingual and&#10;cross-lingual situations&#10;" />
          <attvalue for="2" value="&#10;&#10;In natural language processing (NLP), a lot of research focuses on numeric word representations. Static pretrained word embeddings like word2vec \cite{mikolov2013exploiting} are recently replaced by dynamic, contextual embeddings, such as ELMo \cite{Peters2018} and BERT \cite{Devlin2018}. These generate a word vector based on the context the word appears in, mostly using the sentence as the context.&#10;&#10;Large pretrained masked language models like BERT \cite{Devlin2018} and its derivatives achieve state-of-the-art performance when fine-tuned for specific NLP tasks. The research into these models has been mostly limited to English and a few other well-resourced languages, such as Chinese Mandarin, French, German, and Spanish. However, two massively multilingual masked language models have been released: a multilingual BERT (mBERT) \cite{Devlin2018}, trained on 104 languages, and newer even larger XLM-RoBERTa (XLM-R) \cite{conneau2019unsupervised}, trained on 100 languages. While both, mBERT and XLM-R, achieve good results, it has been shown that monolingual models significantly outperform multilingual models \cite{virtanen2019multilingual,martin2019camembert}.&#10;In our work, we reduced the number of languages in multilingual models to three, two similar less-resourced languages from the same language family, and English. The main reasons for this choice are to better represent each language, and keep sensible sub-word vocabulary, as shown by \cite{virtanen2019multilingual}. We decided against production of monolingual models, because we are interested in using the models in multilingual sense and for cross-lingual knowledge transfer. By including English in each of the two models, we expect to better transfer existing prediction models from English to involved less-resourced languages.&#10;Additional reason against purely monolingual models for less-resourced languages is the size of training corpora, i.e. BERT-like models use transformer architecture which is known to be data hungry.&#10;&#10;We thus trained two multilingual BERT models: FinEst BERT was trained on Finnish, Estonian, and English, while CroSloEngual BERT was trained on Croatian, Slovenian, and English. In the paper, we present the creation and evaluation of these models, which required considerable computational resources, unavailable to most NLP researchers. We make the models which are valuable resources for the involved less-resourced languages publicly available. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Multilingual Language Models, Deep Learning Applications, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2306.01768" label="2306.01768">
        <attvalues>
          <attvalue for="0" value="A Quantitative Review on Language Model Efficiency Research" />
          <attvalue for="1" value="  Language models (LMs) are being scaled and becoming powerful. Improving their&#10;efficiency is one of the core research topics in neural information processing&#10;systems. Tay et al. (2022) provided a comprehensive overview of efficient&#10;Transformers that have become an indispensable staple in the field of NLP.&#10;However, in the section of &quot;On Evaluation&quot;, they left an open question &quot;which&#10;fundamental efficient Transformer one should consider,&quot; answered by &quot;still a&#10;mystery&quot; because &quot;many research papers select their own benchmarks.&quot;&#10;Unfortunately, there was not quantitative analysis about the performances of&#10;Transformers on any benchmarks. Moreover, state space models (SSMs) have&#10;demonstrated their abilities of modeling long-range sequences with&#10;non-attention mechanisms, which were not discussed in the prior review. This&#10;article makes a meta analysis on the results from a set of papers on efficient&#10;Transformers as well as those on SSMs. It provides a quantitative review on LM&#10;efficiency research and gives suggestions for future research.&#10;" />
          <attvalue for="2" value="&#10;Language models are trained to learn the underlying distribution of words in a given language.&#10;A Transformer model is a neural network that learns context and thus meaning by tracking relationships in sequential data like the words in natural language~\cite{vaswani2017attention}. It has been observed that when the Transformers are scaled to have billions of parameters, the Transformer-based language models exhibit amazing performance on various language tasks~\cite{wei2022emergent}. Meanwhile, the efficiency of the language models, in terms of time and memory complexity, has attracted great attention from research, because it would address the bottleneck of training and deploying such large-scale models.&#10;\cite{tay2022efficient} wrote a survey paper that provided a taxonomy of efficient Transformer models, characterizing them by their technical innovation and primary use case. The review was comprehensive, but unfortunately, readers who really wanted to learn or do language model efficiency research would not be able to find answers to the following questions:&#10;\begin{itemize}&#10; \item Q1: What were the state-of-the-art efficient language models?&#10; \item Q2: Were the results (i.e., performance measures) reported and confirmed by multiple sources? Were there inconsistent results, i.e., significantly different measured performances of one type of solutions reported by different sources?&#10; \item Q3: Most studies claimed that they were the best solution when the papers were submitted or accepted. Were these claims correct?&#10;\end{itemize}&#10;&#10;That's because the solid answers would require a quantitative analysis across papers in this research field. The existing survey, in On Evaluation sectionon page 21, discussed a few NLP benchmarks such as GLUE, NaturalQuestions, and TriviaQA; and it argued that ``many research papers select their own benchmarks to showcase the abilities of the proposed model,'' leaving it ``a mystery to which fundamental efficient Transformer block one should consider using''~\cite{tay2022efficient}.&#10;&#10;As dozens of related studies are being performed and published, some language tasks and datasets have been commonly selected as test scenarios to evaluate/compare model efficiency. Moreover, non-attention models such as state space models (SSMs)~\cite{gu2022efficiently} have been proposed to address the long range modeling problem and evaluated on the NLP benchmarks. Neither their technical innovations or experimental results were discussed in the existing survey. A broad, comprehensive, quantitative review is needed in language model efficiency research. And this article presents the work. It briefly describes different types of efficient language models, followed by tasks, datasets, and evaluation metrics. Then it provides a comprehensive quantitative meta analysis that aims to integrate the results from prior research to answer the aforementioned questions.&#10;&#10;Reducing time and/or memory complexity would inevitably sacrifice a bit non-efficiency performance like accuracy. When the complexities of a set of models were reduced to a certain level, one could claim that the model that achieved the highest accuracy would be the most efficient solution. It sacrificed the accuracy the least, so people hypothesize that when the models achieved the same accuracy, this model would have the least complexity. Past empirical studies unanimously performed efficiency evaluation based on this hypothesis. Our key observations from a meta analysis on these empirical results are listed as follows:&#10;\begin{enumerate}&#10; \item Most empirical studies compared their proposed model against others on multiple tasks, and usually claimed theirs is the best one. However, the meta analysis identifies different winning approaches for different tasks and even different datasets. It fixes the one-sided understanding that researchers would have from learning only one or a few empirical studies.&#10; \item More than half of the results were reported by at least two sources. However, it is impossible to tell from the papers whether the numbers were reproduced/confirmed or just re-used from previous work. Meanwhile, inconsistent results were found on almost every task, caused by various settings of hyperparameters (e.g., model sizes, configurations) and reproductions.&#10; \item In quite a few studies, the proposed models were evaluated on a small subset of the tasks, and they actually did not perform better than those that were published earlier and not cited.&#10;\end{enumerate}&#10;&#10;Based on this quantitative review, we offer a few suggestions for future research:&#10;\begin{enumerate}&#10; \item Researchers are suggested to investigate as many suitable baselines as possible, write clearly if the numbers in experimental results were re-used from prior work or reproduced, and report and analyze any inconsistent results that are identified compared with related studies.&#10; \item Researchers in this field need a community and need a public collection of leaderboards.&#10; \item We hope that this quantitative review is helpful, and if it is, it should be continuously updated. Quantitative reviews are needed in flourishing research fields.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Information Processing, Computer Science, Linguistics, Mathematics, Language Model Efficiency, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="2202.09049" label="2202.09049">
        <attvalues>
          <attvalue for="0" value="CLSEG: Contrastive Learning of Story Ending Generation" />
          <attvalue for="1" value="  Story Ending Generation (SEG) is a challenging task in natural language&#10;generation. Recently, methods based on Pre-trained Language Models (PLM) have&#10;achieved great prosperity, which can produce fluent and coherent story endings.&#10;However, the pre-training objective of PLM-based methods is unable to model the&#10;consistency between story context and ending. The goal of this paper is to&#10;adopt contrastive learning to generate endings more consistent with story&#10;context, while there are two main challenges in contrastive learning of SEG.&#10;First is the negative sampling of wrong endings inconsistent with story&#10;contexts. The second challenge is the adaptation of contrastive learning for&#10;SEG. To address these two issues, we propose a novel Contrastive Learning&#10;framework for Story Ending Generation (CLSEG), which has two steps:&#10;multi-aspect sampling and story-specific contrastive learning. Particularly,&#10;for the first issue, we utilize novel multi-aspect sampling mechanisms to&#10;obtain wrong endings considering the consistency of order, causality, and&#10;sentiment. To solve the second issue, we well-design a story-specific&#10;contrastive training strategy that is adapted for SEG. Experiments show that&#10;CLSEG outperforms baselines and can produce story endings with stronger&#10;consistency and rationality.&#10;" />
          <attvalue for="2" value="&#10;&#10;Story Ending Generation (SEG) is a challenging task in natural language generation and artificial intelligence (AI), which aims to complete the plot and conclude a story given a story context \cite{Mostafazadeh2016ACA,Sharma2018TacklingTS,Zhao2018FromPT}. &#10;&#09;SEG systems require to understand the story context, and then generate coherent, reasonable, and diversified endings according to the temporal and causal relationships. &#10;&#09;Previous state-of-the-art Seq2Seq based methods, like GPT-2 \cite{Radford2019LanguageMA} and BART \cite{DBLP:conf/acl/LewisLGGMLSZ20}, mainly generate in a left-to-right manner and train with language model objective, which undergo an issue of generating inconsistent and safe endings. &#10;&#10;To solve this issue, researches consider integrating semantic \cite{Huang2021StoryEG,Xu2020ControllableSG}, commonsense \cite{Li2018GeneratingRA,Guan2019StoryEG,Paul2021COINSDG}, sentiment/emotion \cite{Luo2019LearningTC,Brahman2020ModelingPE} or even multi-modal \cite{Huang2021IgSEGIS} knowledge into backbone models. &#10;&#09;Particularly, \cite{Guan2020AKP} introduces a self-supervised task to distinguish true stories from auto-constructed fake stories to incorporate commonsense knowledge into GPT-2, so that the content of generated endings appears more coherent. &#10;&#09;In terms of consistency and rationality, however, there is still a big gap between machines and humans. &#10;&#09;One reason is that the proposed self-supervised classification task can learn the difference between true story and fake story, but it can not directly model the consistency between story context and endings.&#10;&#10;We argue that the comparison of story context and various endings is vital to generating more consistent story endings, therefore it is necessary to seek an approach to better grasp the consistency. &#10;&#09;Based on the above analysis, we adopt the paradigm of contrastive learning \cite{Arora2019ATA} to introduce an approach for story ending generation which achieves the goal of modeling the consistency between story context and candidate endings.&#10;&#09;There are two crucial problems in contrastive learning of story ending generation:&#10;&#10;Q1: How to generate high-quality wrong endings inconsistent with story contexts? &#10;&#10;Q2: How to design a contrastive training strategy adapted for story ending generation?&#10;&#10;To address the aforementioned problems, we propose a novel Contrastive Learning framework for Story Ending Generation (CLseg), which has two steps: multi-aspect sampling and story-specific contrastive learning. &#10;&#09;Targeted at Problem Q1, multi-aspect sampling mechanisms are utilized to sample high-quality wrong endings by considering the consistency of order, causality, and sentiment. &#10;&#09;To solve Problem Q2, a story-specific contrastive training strategy is designed, which is adapted for SEG. &#10;&#09;Experimental results demonstrate that CLseg outperforms baselines and can produce more consistent and reasonable story endings.&#10;&#10;Our contributions are summarized as below: &#10;&#09;(1) We propose a novel CLseg framework for story ending generation that can generate endings more consistent to the story context. &#10;&#09;(2) To achieve contrastive learning of SEG, we introduce novel multi-aspect sampling mechanisms and a story-specific contrastive training strategy. &#10;&#09;(3) Automatic and manual evaluations demonstrate that CLseg outperforms baselines on recall-oriented metrics and can generate story endings with stronger consistency and rationality.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Story Context Modeling, Contrastive Learning Methods, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2106.02497" label="2106.02497">
        <attvalues>
          <attvalue for="0" value="COINS: Dynamically Generating COntextualized Inference Rules for&#10;  Narrative Story Completion" />
          <attvalue for="1" value="  Despite recent successes of large pre-trained language models in solving&#10;reasoning tasks, their inference capabilities remain opaque. We posit that such&#10;models can be made more interpretable by explicitly generating interim&#10;inference rules, and using them to guide the generation of task-specific&#10;textual outputs. In this paper we present COINS, a recursive inference&#10;framework that i) iteratively reads context sentences, ii) dynamically&#10;generates contextualized inference rules, encodes them, and iii) uses them to&#10;guide task-specific output generation. We apply COINS to a Narrative Story&#10;Completion task that asks a model to complete a story with missing sentences,&#10;to produce a coherent story with plausible logical connections, causal&#10;relationships, and temporal dependencies. By modularizing inference and&#10;sentence generation steps in a recurrent model, we aim to make reasoning steps&#10;and their effects on next sentence generation transparent. Our automatic and&#10;manual evaluations show that the model generates better story sentences than&#10;SOTA baselines, especially in terms of coherence. We further demonstrate&#10;improved performance over strong pre-trained LMs in generating commonsense&#10;inference rules. The recursive nature of COINS holds the potential for&#10;controlled generation of longer sequences.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Sentence-level Commonsense Inference and Beyond. &#10;Recent research in this area has&#10;focused on commonsense knowledge acquisition \cite{sapatomic, zhang2020aser, speer2017conceptnet, malaviya2019exploiting} and commonsense reasoning \cite{zellers2019hellaswag,Talmor2018CommonsenseQAAQ}. In our work, we focus on inferential knowledge about events, and entities participating in such events. \cite{rashkin-etal-2018-event2mind} introduced&#10;a knowledge resource &#10;of&#10;commonsense inferences regarding&#10;people's intents and reactions towards&#10;a diverse set of events.&#10;With Comet, \cite{bosselut-etal-2019-comet} &#10;have shown that pre-trained neural language models can be fine-tuned using large knowledge bases (such as Atomic, \cite{sapatomic}) to generate &#10;inferences for a given event or sentence.&#10;However, the generated knowledge from Comet\ is non-contextualized and hence, can be inconsistent. Recently, \cite{mostafazadeh-etal-2020-glucose} proposed Glucose, a new resource and dataset that offers semi-structured commonsense inference rules that are grounded in sentences of specific stories. They show that fine-tuning a pre-trained LM&#10;on the Glucose dataset helps the model to better generate inferrable commonsense explanations given a complete &#10;story. In concurrent work, \cite{Gabriel2021ParagraphLevelCT} proposed PARA-COMET, a model that incorporates paragraph-level information to generate coherent commonsense inferences from narratives.&#10;In this work, we investigate how well a neural model can generate contextualized commonsense inference rules for an incomplete story. &#10;Learning to predict iterative inference steps for successive events in a narration using&#10;semi-structured knowledge rules &#10;is still a difficult and underexplored task. We propose a model that learns to iteratively generate a coherent completion of an incomplete narrative story utilizing semi-structured knowledge as &#10;offered&#10;by the Glucose framework.&#10;&#10;Commonsense Reasoning in&#10;Narrative Stories. Early work on narrative events focused on script learning, by defining stereotypical event sequences together with their participants \cite{welin1979scripts}.&#10;In later works, \cite{chambers-jurafsky-2008-unsupervised, chambers-jurafsky-2009-unsupervised, balasubramanian-etal-2013-generating, nguyen2015generative, pichotta2014statistical} proposed methods to learn narrative event chains using a simpler event representation that allows for efficient learning and inference.&#10;\cite{chambers-jurafsky-2009-unsupervised} acquired Narrative Event Schemata from corpora and established the Narrative Cloze Task \cite{chambers-jurafsky-2008-unsupervised} that evaluates script knowledge by predicting a missing event (verb and its arguments) in a sequence of observed events.&#10;More recently, \cite{mostafazadeh-etal-2016-corpus} proposed the story cloze task that selects a plausible (right) over an implausible (wrong) story ending.&#10;\cite{bhagavatula2019abductive} proposed an abductive reasoning task to test a model’s ability to generate plausible explanations for&#10;an incomplete set of observations. \cite{paul-frank-2020-social} proposed a multi-head knowledge attention method to dynamically incorporate non-contextualized inferential knowledge to address the abductive reasoning task. \cite{qin-etal-2020-back} proposed an unsupervised decoding algorithm that can flexibly incorporate both the past and future contexts using only off-the-shelf language models to generate plausible explanations. Concurrent to our work, \cite{paul-frank-2021-generate-hypothetical-events} presented a method for addressing the abductive reasoning task by explicitly learning what events could follow other events in a hypothetical scenario. In our work, we make use of the ROCStories dataset \cite{mostafazadeh-etal-2016-corpus} to build a Narrative Story Completion task that tests a model's ability of generating missing sentences in a story. We propose a model that aims to produce&#10;coherent narrative stories by performing iterative commonsense inference steps.&#10;&#10;Narrative Story Generation. Much existing work on story generation&#10;relied on symbolic planning methods \cite{lebowitz1987planning, doi:10.1080/09528130010029820, Jzefowicz2016ExploringTL}. With the advances of Seq2Seq models, several works &#10;applied them in automatic story generation tasks \cite{roemmele2016writing, Jain2017StoryGF}. \cite{fan-etal-2018-hierarchical} proposed a hierarchical approach to &#10;generate short stories from initial prompts. Recently, many works have focused on integrating external commonsense knowledge from large static knowledge bases like Atomic\ \cite{sapatomic} or ConceptNet \cite{speer2017conceptnet} for different tasks&#10;such as story ending generation \cite{ji-etal-2020-language, Guan_Wang_Huang_2019} or story generation \cite{guan-etal-2020-knowledge, xu-etal-2020-megatron}. &#10;In&#10;concurrent work, \cite{ammanabrolu2020automated} look into causality for a commonsense plot generation task. In our work, we model the assumption that contextualized inference rules provide inferred information that can guide a system in generating both contextually grounded and coherent follow-up&#10;sentences in a story generation task. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2010.06822" label="2010.06822">
        <attvalues>
          <attvalue for="0" value="Modeling Protagonist Emotions for Emotion-Aware Storytelling" />
          <attvalue for="1" value="  Emotions and their evolution play a central role in creating a captivating&#10;story. In this paper, we present the first study on modeling the emotional&#10;trajectory of the protagonist in neural storytelling. We design methods that&#10;generate stories that adhere to given story titles and desired emotion arcs for&#10;the protagonist. Our models include Emotion Supervision (EmoSup) and two&#10;Emotion-Reinforced (EmoRL) models. The EmoRL models use special rewards&#10;designed to regularize the story generation process through reinforcement&#10;learning. Our automatic and manual evaluations demonstrate that these models&#10;are significantly better at generating stories that follow the desired emotion&#10;arcs compared to baseline methods, without sacrificing story quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.05139" label="2001.05139">
        <attvalues>
          <attvalue for="0" value="A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation" />
          <attvalue for="1" value="  Story generation, namely generating a reasonable story from a leading&#10;context, is an important but challenging task. In spite of the success in&#10;modeling fluency and local coherence, existing neural language generation&#10;models (e.g., GPT-2) still suffer from repetition, logic conflicts, and lack of&#10;long-range coherence in generated stories. We conjecture that this is because&#10;of the difficulty of associating relevant commonsense knowledge, understanding&#10;the causal relationships, and planning entities and events with proper temporal&#10;order. In this paper, we devise a knowledge-enhanced pretraining model for&#10;commonsense story generation. We propose to utilize commonsense knowledge from&#10;external knowledge bases to generate reasonable stories. To further capture the&#10;causal and temporal dependencies between the sentences in a reasonable story,&#10;we employ multi-task learning which combines a discriminative objective to&#10;distinguish true and fake stories during fine-tuning. Automatic and manual&#10;evaluation shows that our model can generate more reasonable stories than&#10;state-of-the-art baselines, particularly in terms of logic and global&#10;coherence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.00971" label="2404.00971">
        <attvalues>
          <attvalue for="0" value="Exploring and Evaluating Hallucinations in LLM-Powered Code Generation" />
          <attvalue for="1" value="  The rise of Large Language Models (LLMs) has significantly advanced many&#10;applications on software engineering tasks, particularly in code generation.&#10;Despite the promising performance, LLMs are prone to generate hallucinations,&#10;which means LLMs might produce outputs that deviate from users' intent, exhibit&#10;internal inconsistencies, or misalign with the factual knowledge, making the&#10;deployment of LLMs potentially risky in a wide range of applications. Existing&#10;work mainly focuses on investing the hallucination in the domain of natural&#10;language generation (NLG), leaving a gap in understanding the types and extent&#10;of hallucinations in the context of code generation. To bridge the gap, we&#10;conducted a thematic analysis of the LLM-generated code to summarize and&#10;categorize the hallucinations present in it. Our study established a&#10;comprehensive taxonomy of hallucinations in LLM-generated code, encompassing 5&#10;primary categories of hallucinations depending on the conflicting objectives&#10;and varying degrees of deviation observed in code generation. Furthermore, we&#10;systematically analyzed the distribution of hallucinations, exploring&#10;variations among different LLMs and their correlation with code correctness.&#10;Based on the results, we proposed HalluCode, a benchmark for evaluating the&#10;performance of code LLMs in recognizing hallucinations. Hallucination&#10;recognition and mitigation experiments with HalluCode and HumanEval show&#10;existing LLMs face great challenges in recognizing hallucinations, particularly&#10;in identifying their types, and are hardly able to mitigate hallucinations. We&#10;believe our findings will shed light on future research about hallucination&#10;evaluation, detection, and mitigation, ultimately paving the way for building&#10;more effective and reliable code LLMs in the future.&#10;" />
          <attvalue for="2" value="&#10;Code generation is the process of automatically generating source code based on provided specifications or requirements, which enables developers to save time by reducing manual coding efforts and allows them to focus on higher-level tasks and problem-solving. Moreover, it can aid in ensuring consistency and reducing the risk of human error during development. Automatic code generation has been a longstanding challenge in both software engineering and artificial intelligence communities. &#10;The recent advancements in Large Language Models (LLMs) have significantly propelled this field forward \cite{chen2021evaluatingCodex,roziere2023codellama,chatgpt}. For example, OpenAI’s Codex \cite{chen2021evaluatingCodex}, released in 2021, has achieved a success rate of 28.8\% in solving a set of 164 hand-written programming problems. Microsoft's Copilot, a code generation tool powered by Codex, has captured the interest of over 1 million professional developers \cite{euronews}. Moreover, it has shown the potential to speed up coding tasks by up to 55\% \cite{Kalliamvakou}. Subsequently, various code LLMs emerged in both academia and industry, such as Incoder \cite{Fried2023Incoder}, StarCoder \cite{li2023starcoder}, CodeRL \cite{le2022coderl}, CodeGen \cite{Nijkamp2023codegen,nijkamp2023codegen2}, Code Llama \cite{roziere2023codellama}, ChatGPT \cite{chatgpt}, etc. These models are capable of generating code with functional accuracy comparable to that of human developers.&#10;&#10;Despite the remarkable success of LLMs, they are prone to generate hallucinations across various tasks. In other words, LLMs might produce outputs that, although seemingly plausible, deviate from users' intent, factual knowledge, or their contexts \cite{zhang2023siren}.&#10;The hallucination issue poses a potential risk in deploying LLMs across various applications \cite{ji2023survey}. Most existing work mainly focuses on investigating the hallucination for natural language generation (NLG) tasks, for instance, generative question answering \cite{li2021addressing}, abstractive summarization \cite{maynez2020faithfulness}, dialogue generation \cite{huang2020challenges}, etc. The hallucinations are mainly divided into three categories: input-conflicting, context-conflicting, and fact-conflicting hallucinations \cite{zhang2023siren}. However, there is still a lack of clarity regarding the specific types of content that LLMs tend to hallucinate during code generation, as well as the potential consequences they may have.&#10;&#10;We argue that similar hallucinations also occur in the domain of code generation, where the model could generate code snippets that conflict with the user's requirements, contextual information, or code knowledge. &#10;These occurrences may undermine the correctness, performance, maintenance, and even security of the developed software. As a result, the widespread adoption of Code LLMs for code recommendation has the inherent potential to compromise the overall quality and reliability of software.&#10;Hence, it is imperative to thoroughly investigate hallucinations in LLM-powered code generation. This will allow us to gain valuable insights into the specific weaknesses that the LLM model may generate.&#10;Moreover, analyzing hallucinations helps us identify code snippets or patterns that are likely to be incorrect or unreliable, providing valuable feedback for improving LLMs. Through collaboration between researchers and developers, we can refine and fine-tune the model, thus enhancing its code generation capabilities in terms of accuracy and reliability.&#10;&#10;To facilitate research in this area, we conducted a thematic analysis \cite{Cruzes2011thematic} of the LLM-generated code to summarize and categorize the hallucinations presented in it. Specifically, we first collected 13,968 code snippets generated by different LLMs, and sampled 3,084 code snippets for subsequent analysis. Finally, we establish a comprehensive taxonomy of hallucinations, which comprises 5 primary categories: Intent Conflicting, Context Inconsistency, Context Repetition, Dead Code, and Knowledge Conflicting. The taxonomy encompasses 19 specific types of hallucinations.&#10;Then we conducted a comprehensive investigation and various statistical analyses of these hallucinations from diverse perspectives to gain a deeper understanding of the prevailing challenges and opportunities in the domain of code generation with LLMs. &#10;The analysis reveals that code LLMs are frequently influenced by a diverse range of hallucinations with distinct distributions. Moreover, multiple different hallucinations can occur simultaneously within a single generated program. Additionally, the majority of these hallucinations can result in functional errors or serve as indicators of their presence. Therefore, it is imperative to develop effective techniques to detect and mitigate hallucinations during code generation.&#10;&#10;In summary, this paper makes the following contributions:&#10;\begin{itemize}&#10; \item We conducted the first comprehensive study to analyze the types of content LLMs may tend to hallucinate in code generation, and established a taxonomy of hallucination types.&#10; \item We systematically analyzed the distribution of hallucinations as well as the correlation between these hallucinations and the correctness of the code. &#10; \item We developed and released HalluCode, an evaluation benchmark specifically designed to assess hallucinations in code LLMs, and also conducted hallucination recognition experiments using HalluCode and HumanEval to evaluate several state-of-the-art code LLMs.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Information Technology, Code Generation Analysis, Linguistics, Hallucination Detection, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2305.02309" label="2305.02309">
        <attvalues>
          <attvalue for="0" value="CodeGen2: Lessons for Training LLMs on Programming and Natural Languages" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated remarkable abilities in&#10;representation learning for program synthesis and understanding tasks. The&#10;quality of the learned representations appears to be dictated by the neural&#10;scaling laws as a function of the number of model parameters and observations,&#10;while imposing upper bounds on the model performance by the amount of available&#10;data and compute, which is costly.&#10;  In this study, we attempt to render the training of LLMs for program&#10;synthesis more efficient by unifying four key components: (1) model&#10;architectures, (2) learning methods, (3) infill sampling, and, (4) data&#10;distributions. Specifically, for the model architecture, we attempt to unify&#10;encoder and decoder-based models into a single prefix-LM. For learning methods,&#10;(i) causal language modeling, (ii) span corruption, (iii) infilling are unified&#10;into a simple learning algorithm. For infill sampling, we explore the claim of&#10;a &quot;free lunch&quot; hypothesis. For data distributions, the effect of a mixture&#10;distribution and multi-epoch training of programming and natural languages on&#10;model performance is explored.&#10;  We conduct a comprehensive series of empirical experiments on 1B LLMs, for&#10;which failures and successes of this exploration are distilled into five&#10;lessons. We will provide a final recipe for training and release CodeGen2&#10;models in size 1B, 3.7B, 7B, and, 16B parameters, along with the training&#10;framework as open-source: https://github.com/salesforce/CodeGen.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;LLMs on Code Transformers capture dependency among sequence elements through attention mechanism \cite{bahdanau2014neural} and are highly scalable, as shown in natural language processing \cite{devlin2018bert, lewis2020bart, raffel2020exploring}. Several efforts explore these models for program synthesis~\cite{chen2021evaluating, austin2021program, alphacode, fried2022incoder, nijkamp2022codegen, allal2023santacoder} and its effectiveness~\cite{vaithilingam2022expectation}.&#10;&#10;Ablation Studies&#10;&#10;\cite{raffel2020exploring} introduce the concept of non-causal decoder in the form of a Prefix-LM with favorable performance over causal decoders after fine-tuning on down-stream tasks. The performance in few-shot generative tasks was not evaluated. \cite{wang2022language} conduct an extensive ablation study over architectures and objectives with the conclusion that decoder-only models with causal language modeling exhibit the strongest zero-shot generalization. Therefore, we limit our investigation to causal and non-causal decoders. \cite{tay2022unifying} compare encoder-decoder, decoder-only, and Prefix-LM architectures and report the beneficial performance of encoder-decoder models, while zero-shot generation tasks are not evaluated. The authors later adopt Prefix-LM instead of encoder-decoder in~\cite{tay2022transcending}.&#10;&#10;Data Mixtures&#10;&#10;LaMDA~\cite{thoppilan2022lamda} was trained on a mixture of various data sources including dialogues, code documents, Q\&amp;A data, tutorials, and, Wikipedia. However, the impact of this mixture and the specific sources are unclear. \cite{xie2023data} introduces a data selection method based on importance resampling which allows to mix datasets of various sizes, however, the evaluation only covers encoder-only models.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="1905.05709" label="1905.05709">
        <attvalues>
          <attvalue for="0" value="Challenges in Building Intelligent Open-domain Dialog Systems" />
          <attvalue for="1" value="  There is a resurgent interest in developing intelligent open-domain dialog&#10;systems due to the availability of large amounts of conversational data and the&#10;recent progress on neural approaches to conversational AI. Unlike traditional&#10;task-oriented bots, an open-domain dialog system aims to establish long-term&#10;connections with users by satisfying the human need for communication,&#10;affection, and social belonging. This paper reviews the recent works on neural&#10;approaches that are devoted to addressing three challenges in developing such&#10;systems: semantics, consistency, and interactiveness. Semantics requires a&#10;dialog system to not only understand the content of the dialog but also&#10;identify user's social needs during the conversation. Consistency requires the&#10;system to demonstrate a consistent personality to win users trust and gain&#10;their long-term confidence. Interactiveness refers to the system's ability to&#10;generate interpersonal responses to achieve particular social goals such as&#10;entertainment, conforming, and task completion. The works we select to present&#10;here is based on our unique views and are by no means complete. Nevertheless,&#10;we hope that the discussion will inspire new research in developing more&#10;intelligent dialog systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.08007" label="2309.08007">
        <attvalues>
          <attvalue for="0" value="DiariST: Streaming Speech Translation with Speaker Diarization" />
          <attvalue for="1" value="  End-to-end speech translation (ST) for conversation recordings involves&#10;several under-explored challenges such as speaker diarization (SD) without&#10;accurate word time stamps and handling of overlapping speech in a streaming&#10;fashion. In this work, we propose DiariST, the first streaming ST and SD&#10;solution. It is built upon a neural transducer-based streaming ST system and&#10;integrates token-level serialized output training and t-vector, which were&#10;originally developed for multi-talker speech recognition. Due to the absence of&#10;evaluation benchmarks in this area, we develop a new evaluation dataset,&#10;DiariST-AliMeeting, by translating the reference Chinese transcriptions of the&#10;AliMeeting corpus into English. We also propose new metrics, called&#10;speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality&#10;while taking SD accuracy into account. Our system achieves a strong ST and SD&#10;capability compared to offline systems based on Whisper, while performing&#10;streaming inference for overlapping speech. To facilitate the research in this&#10;new direction, we release the evaluation data, the offline baseline systems,&#10;and the evaluation code.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech translation (ST) is a task to convert speech signals into texts in other languages.&#10;The field of ST has been extensively studied with the aim of reducing language barriers.&#10;Traditionally, ST has been implemented by cascading two separate processes: automatic speech recognition (ASR)&#10;and machine translation (MT) \cite{ney1999speech,matusov2005integration,post2013improved}. &#10;However, such a cascaded system has several limitations.&#10;&#10;Firstly, errors originating from the ASR may propagate to the MT.&#10;Secondly, the non-linguistic information, such as short pauses and prosody, &#10;may not be fully leveraged because of the text-based intermediate representation.&#10;Lastly, the latency of the cascaded system tends to be large because &#10;the MT system needs to wait for the ASR result to be generated. &#10;To overcome these shortcomings, &#10;end-to-end (E2E) ST systems that directly convert speech signals into text without a separate ASR stage have been extensively studied (e.g., \cite{berard2016listen,vila2018end,sperber2020speech,radford2022robust}).&#10;With the recent advancements in deep learning,&#10;neural E2E ST systems&#10;have achieved significantly better accuracy on multilingual translation using a single model \cite{barrault2023seamlessm4t},&#10;compared to&#10;strong cascaded systems.&#10;Furthermore, E2E ST systems based on neural transducer~\cite{xue22d_interspeech,xue2022weakly,wang23oa_interspeech} &#10;have succeeded in achieving low-latency ST while maintaining higher accuracy compared to traditional, non-streaming cascaded systems.&#10;&#10;In conversational translation scenarios,&#10;accurately identifying the speaker of each utterance is of great importance to end users. &#10;This is known as the speaker diarization (SD) problem \cite{park2021review}, which has been long studied&#10;in the context of speaker-attributed ASR \cite{fiscus2007rich,watanabe2020chime}.&#10;For a cascaded ST system that uses separate ASR and MT components, &#10;conventional SD techniques employing word-level&#10;timestamps (e.g. \cite{huang2007ibm,silovsky2012incorporation})&#10;can be leveraged.&#10;However, unlike ASR, it is not straightforward to obtain word-level time stamps&#10;for E2E ST systems where the neural networks implicitly learn the mapping between input audio signals and output translated texts.&#10;Note that, although a few techniques have recently been proposed for E2E ASR&#10;to estimate word-level time stamps \cite{radford2022robust,bain23_interspeech},&#10;they assume a monotonic alignment between the input and output,&#10;which does not hold in the ST task.&#10;Adding to the complexity of the SD,&#10;the presence of the overlapping speech in conversations \cite{shriberg2001observations,watanabe2020chime,yu2022m2met} further makes &#10;the development of the ST systems challenging.&#10;While there have been a lot of studies for ASR and SD,&#10;to the best of our knowledge, no prior works have investigated the integration of ST and SD for conversational recordings.&#10;&#10;In this work, &#10;we propose DiariST, the first streaming ST and SD system specifically designed for conversational recordings.&#10;Due to the lack of prior work in this area, we first develop a new evaluation dataset for the task of translating Chinese audio to English text. This dataset, named DiariST-AliMeeting, is developed by&#10;translating &#10;the Mandarin Chinese meeting recordings, AliMeeting corpus~\cite{yu2022m2met,yu2022summary},&#10;into English.&#10;We also establish an evaluation scheme consisting of two novel metrics, &#10;named &#10;speaker-agnostic BLEU (SAgBLEU) and speaker-attributed BLEU (SAtBLEU),&#10;to measure the translation quality by taking the SD accuracy into account. &#10;We develop a streaming multi-talker ST and SD system, dubbed DiariST,&#10;by&#10;integrating &#10;the &#10;token-level serialized output training (t-SOT) \cite{kanda22arxiv}&#10;and t-vector \cite{kanda22b_interspeech}, both originally developed for speaker-attributed ASR,&#10;into the neural transducer-based ST system.&#10;The proposed system shows a strong ST and SD capability &#10;compared to the offline baseline systems based on Whisper \cite{radford2022robust} &#10;while allowing for streaming inference, even for overlapping speech.&#10;To facilitate the research in this new direction, &#10;we release the evaluation data, the offline baseline systems, &#10;and the evaluation code.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Translation, Computer Science, Linguistics, Engineering, Streaming Inference, Signal Processing, Speaker Diarization" />
        </attvalues>
      </node>
      <node id="1612.01744" label="1612.01744">
        <attvalues>
          <attvalue for="0" value="Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text&#10;  Translation" />
          <attvalue for="1" value="  This paper proposes a first attempt to build an end-to-end speech-to-text&#10;translation system, which does not use source language transcription during&#10;learning or decoding. We propose a model for direct speech-to-text translation,&#10;which gives promising results on a small French-English synthetic corpus.&#10;Relaxing the need for source language transcription would drastically change&#10;the data collection methodology in speech translation, especially in&#10;under-resourced scenarios. For instance, in the former project DARPA TRANSTAC&#10;(speech translation from spoken Arabic dialects), a large effort was devoted to&#10;the collection of speech transcripts (and a prerequisite to obtain transcripts&#10;was often a detailed transcription guide for languages with little standardized&#10;spelling). Now, if end-to-end approaches for speech-to-text translation are&#10;successful, one might consider collecting data by asking bilingual speakers to&#10;directly utter speech in the source language from target language text&#10;utterances. Such an approach has the advantage to be applicable to any&#10;unwritten (source) language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.06358" label="2004.06358">
        <attvalues>
          <attvalue for="0" value="Speech Translation and the End-to-End Promise: Taking Stock of Where We&#10;  Are" />
          <attvalue for="1" value="  Over its three decade history, speech translation has experienced several&#10;shifts in its primary research themes; moving from loosely coupled cascades of&#10;speech recognition and machine translation, to exploring questions of tight&#10;coupling, and finally to end-to-end models that have recently attracted much&#10;attention. This paper provides a brief survey of these developments, along with&#10;a discussion of the main challenges of traditional approaches which stem from&#10;committing to intermediate representations from the speech recognizer, and from&#10;training cascaded models separately towards different objectives.&#10;  Recent end-to-end modeling techniques promise a principled way of overcoming&#10;these issues by allowing joint training of all model components and removing&#10;the need for explicit intermediate representations. However, a closer look&#10;reveals that many end-to-end models fall short of solving these issues, due to&#10;compromises made to address data scarcity. This paper provides a unifying&#10;categorization and nomenclature that covers both traditional and recent&#10;approaches and that may help researchers by highlighting both trade-offs and&#10;open research questions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05352" label="2204.05352">
        <attvalues>
          <attvalue for="0" value="Large-Scale Streaming End-to-End Speech Translation with Neural&#10;  Transducers" />
          <attvalue for="1" value="  Neural transducers have been widely used in automatic speech recognition&#10;(ASR). In this paper, we introduce it to streaming end-to-end speech&#10;translation (ST), which aims to convert audio signals to texts in other&#10;languages directly. Compared with cascaded ST that performs ASR followed by&#10;text-based machine translation (MT), the proposed Transformer transducer&#10;(TT)-based ST model drastically reduces inference latency, exploits speech&#10;information, and avoids error propagation from ASR to MT. To improve the&#10;modeling capacity, we propose attention pooling for the joint network in TT. In&#10;addition, we extend TT-based ST to multilingual ST, which generates texts of&#10;multiple languages at the same time. Experimental results on a large-scale 50&#10;thousand (K) hours pseudo-labeled training set show that TT-based ST not only&#10;significantly reduces inference time but also outperforms non-streaming&#10;cascaded ST for English-German translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.02499" label="2211.02499">
        <attvalues>
          <attvalue for="0" value="A Weakly-Supervised Streaming Multilingual Speech Model with Truly&#10;  Zero-Shot Capability" />
          <attvalue for="1" value="  In this paper, we introduce our work of building a Streaming Multilingual&#10;Speech Model (SM2), which can transcribe or translate multiple spoken languages&#10;into texts of the target language. The backbone of SM2 is Transformer&#10;Transducer, which has high streaming capability. Instead of human labeled&#10;speech translation (ST) data, SM2 models are trained using weakly supervised&#10;data generated by converting the transcriptions in speech recognition corpora&#10;with a machine translation service. With 351 thousand hours of anonymized&#10;speech training data from 25 languages, SM2 models achieve comparable or even&#10;better ST quality than some recent popular large-scale non-streaming speech&#10;models. More importantly, we show that SM2 has the truly zero-shot capability&#10;when expanding to new target languages, yielding high quality ST results for&#10;{source-speech, target-text} pairs that are not seen during training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.09624" label="2101.09624">
        <attvalues>
          <attvalue for="0" value="A Review of Speaker Diarization: Recent Advances with Deep Learning" />
          <attvalue for="1" value="  Speaker diarization is a task to label audio or video recordings with classes&#10;that correspond to speaker identity, or in short, a task to identify &quot;who spoke&#10;when&quot;. In the early years, speaker diarization algorithms were developed for&#10;speech recognition on multispeaker audio recordings to enable speaker adaptive&#10;processing. These algorithms also gained their own value as a standalone&#10;application over time to provide speaker-specific metainformation for&#10;downstream tasks such as audio retrieval. More recently, with the emergence of&#10;deep learning technology, which has driven revolutionary changes in research&#10;and practices across speech application domains, rapid advancements have been&#10;made for speaker diarization. In this paper, we review not only the historical&#10;development of speaker diarization technology but also the recent advancements&#10;in neural speaker diarization approaches. Furthermore, we discuss how speaker&#10;diarization systems have been integrated with speech recognition applications&#10;and how the recent surge of deep learning is leading the way of jointly&#10;modeling these two components to be complementary to each other. By considering&#10;such exciting technical trends, we believe that this paper is a valuable&#10;contribution to the community to provide a survey work by consolidating the&#10;recent developments with neural methods and thus facilitating further progress&#10;toward a more efficient speaker diarization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.07393" label="2110.07393">
        <attvalues>
          <attvalue for="0" value="M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription&#10;  Challenge" />
          <attvalue for="1" value="  Recent development of speech processing, such as speech recognition, speaker&#10;diarization, etc., has inspired numerous applications of speech technologies.&#10;The meeting scenario is one of the most valuable and, at the same time, most&#10;challenging scenarios for the deployment of speech technologies. Specifically,&#10;two typical tasks, speaker diarization and multi-speaker automatic speech&#10;recognition have attracted much attention recently. However, the lack of large&#10;public meeting data has been a major obstacle for the advancement of the field.&#10;Therefore, we make available the AliMeeting corpus, which consists of 120 hours&#10;of recorded Mandarin meeting data, including far-field data collected by&#10;8-channel microphone array as well as near-field data collected by headset&#10;microphone. Each meeting session is composed of 2-4 speakers with different&#10;speaker overlap ratio, recorded in rooms with different size. Along with the&#10;dataset, we launch the ICASSP 2022 Multi-channel Multi-party Meeting&#10;Transcription Challenge (M2MeT) with two tracks, namely speaker diarization and&#10;multi-speaker ASR, aiming to provide a common testbed for meeting rich&#10;transcription and promote reproducible research in this field. In this paper we&#10;provide a detailed introduction of the AliMeeting dateset, challenge rules,&#10;evaluation methods and baseline systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.03647" label="2202.03647">
        <attvalues>
          <attvalue for="0" value="Summary On The ICASSP 2022 Multi-Channel Multi-Party Meeting&#10;  Transcription Grand Challenge" />
          <attvalue for="1" value="  The ICASSP 2022 Multi-channel Multi-party Meeting Transcription Grand&#10;Challenge (M2MeT) focuses on one of the most valuable and the most challenging&#10;scenarios of speech technologies. The M2MeT challenge has particularly set up&#10;two tracks, speaker diarization (track 1) and multi-speaker automatic speech&#10;recognition (ASR) (track 2). Along with the challenge, we released 120 hours of&#10;real-recorded Mandarin meeting speech data with manual annotation, including&#10;far-field data collected by 8-channel microphone array as well as near-field&#10;data collected by each participants' headset microphone. We briefly describe&#10;the released dataset, track setups, baselines and summarize the challenge&#10;results and major techniques used in the submissions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.13678" label="2408.13678">
        <attvalues>
          <attvalue for="0" value="A layer-wise analysis of Mandarin and English suprasegmentals in SSL&#10;  speech models" />
          <attvalue for="1" value="  This study asks how self-supervised speech models represent suprasegmental&#10;categories like Mandarin lexical tone, English lexical stress, and English&#10;phrasal accents. Through a series of probing tasks, we make layer-wise&#10;comparisons of English and Mandarin 12 layer monolingual models. Our findings&#10;suggest that 1) English and Mandarin wav2vec 2.0 models learn contextual&#10;representations of abstract suprasegmental categories which are strongest in&#10;the middle third of the network. 2) Models are better at representing features&#10;that exist in the language of their training data, and this difference is&#10;driven by enriched context in transformer blocks, not local acoustic&#10;representation. 3) Fine-tuned wav2vec 2.0 improves performance in later layers&#10;compared to pre-trained models mainly for lexically contrastive features like&#10;tone and stress, 4) HuBERT and WavLM learn similar representations to wav2vec&#10;2.0, differing mainly in later layer performance. Our results extend previous&#10;understanding of how models represent suprasegmentals and offer new insights&#10;into the language-specificity and contextual nature of these representations.&#10;" />
          <attvalue for="2" value="&#10;&#10;How do self-supervised learning (SSL) speech models represent suprasegmental information about features like accents, stresses, and lexical tones? Probing how these models build up these linguistic representations over layers could help us understand how they incorporate contextual information, how their representations differ across languages, and how these representations get modified when fine-tuned.&#10;&#10;Here we study three models: wav2vec 2.0 \cite{Baevski2020wav2vec2A}, HuBERT \cite{Hsu2021HuBERTSS}, and WavLM \cite{WavLM}, to understand how they represent Mandarin lexical tone, English lexical stress, and English phrasal pitch accents. Comparing stress and tone, which are properties of words, with pitch accent, which is phrasal, allows us to explore the role of the lexicon in representation learning. &#10;&#10;We study these models through probing: training classifiers to predict linguistic classes from internal representations of the model to see what kinds of information the model represents about those classes. Probing has been used to show that SSL models represent phone identity \cite{Martin2023ProbingSS, CormacEnglish2022DomainInformedPO, Ma2020ProbingAR, Pasad2023WhatDS, Pasad2022ComparativeLA, Pasad2021LayerWiseAO}, and semantic or syntactic features of words \cite{Pasad2021LayerWiseAO, Pasad2023WhatDS, Shah2021WhatAD}. A consistent finding is that wav2vec 2.0 follows an autoencoder-like behavior where early and later layers relate more closely to local acoustic features and the middle layers seem to represent more abstract linguistic categories. HuBERT and WavLM, on the other hand, have representations that relate closely to abstract linguistic categories up to their last layers.&#10; \cite{Martin2023ProbingSS, Pasad2023WhatDS, Pasad2022ComparativeLA}.&#10; While we also know that suprasegmentals like prosody and tone are represented by SSL models \cite{Yuan2021AutomaticRO, Yang2023WhatCA, prosodyTasks, emotionProbing}, we know less about how these representations develop across layers, what contexts they draw on, and how language-general they are. Our investigations of how suprasegmental representations develop through model layers led us to the following contributions:&#10;&#10;\begin{enumerate}&#10; \item wav2vec 2.0 learns representations of English stress, Mandarin tone, and English phrasal accents. The best classifier probe performance uses representations that come from the middle third of the network. Layer-wise improvements in probe performance are not driven by improvements in the ability of the model to better track F0. This result suggests that categorical suprasegmental representations are more abstract and not directly tied to simple acoustic features.&#10; \item Representation performance is similar for all models at the CNN output. As context gets added by the transformer, models improve at classifying features of the language they were trained on. This shows that language specificity in SSL models is driven by domain specific context and not accurate acoustic representation.&#10; \item Fine-tuning wav2vec2.0 for ASR improves performance in later layers even if probed features are not orthographically represented. This appears more robust for lexical categories (i.e., tone and stress) than for phrasal ones like accent. Probe performance here likely improves because of the model's improved knowledge of lexical identity, which explains why the effect is larger for lexically contrastive suprasegmentals like stress and tone.&#10; \item HuBERT, WavLM, and wav2vec 2.0 represent suprasegmental categories equally well and show similar layer-wise behavior.&#10;&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Suprasegmental Category Modeling, Language Specificity Analysis, Speech Model Representations, Phonetics" />
        </attvalues>
      </node>
      <node id="2306.06232" label="2306.06232">
        <attvalues>
          <attvalue for="0" value="Probing self-supervised speech models for phonetic and phonemic&#10;  information: a case study in aspiration" />
          <attvalue for="1" value="  Textless self-supervised speech models have grown in capabilities in recent&#10;years, but the nature of the linguistic information they encode has not yet&#10;been thoroughly examined. We evaluate the extent to which these models' learned&#10;representations align with basic representational distinctions made by humans,&#10;focusing on a set of phonetic (low-level) and phonemic (more abstract)&#10;contrasts instantiated in word-initial stops. We find that robust&#10;representations of both phonetic and phonemic distinctions emerge in early&#10;layers of these models' architectures, and are preserved in the principal&#10;components of deeper layer representations. Our analyses suggest two sources&#10;for this success: some can only be explained by the optimization of the models&#10;on speech data, while some can be attributed to these models' high-dimensional&#10;architectures. Our findings show that speech-trained HuBERT derives a low-noise&#10;and low-dimensional subspace corresponding to abstract phonological&#10;distinctions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.01122" label="2108.01122">
        <attvalues>
          <attvalue for="0" value="Automatic recognition of suprasegmentals in speech" />
          <attvalue for="1" value="  This study reports our efforts to improve automatic recognition of&#10;suprasegmentals by fine-tuning wav2vec 2.0 with CTC, a method that has been&#10;successful in automatic speech recognition. We demonstrate that the method can&#10;improve the state-of-the-art on automatic recognition of syllables, tones, and&#10;pitch accents. Utilizing segmental information, by employing tonal finals or&#10;tonal syllables as recognition units, can significantly improve Mandarin tone&#10;recognition. Language models are helpful when tonal syllables are used as&#10;recognition units, but not helpful when tones are recognition units. Finally,&#10;Mandarin tone recognition can benefit from English phoneme recognition by&#10;combining the two tasks in fine-tuning wav2vec 2.0.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.14731" label="2308.14731">
        <attvalues>
          <attvalue for="0" value="Distilled GPT for Source Code Summarization" />
          <attvalue for="1" value="  A code summary is a brief natural language description of source code.&#10;Summaries are usually only a single sentence long, and yet form the backbone of&#10;developer documentation. A short descriptions such as &quot;changes all visible&#10;polygons to the color blue&quot; can give a programmer a high-level idea of what&#10;code does without the effort of reading the code itself. Recently, products&#10;based on Large Language Models such as ChatGPT have demonstrated a strong&#10;ability to write these descriptions automatically. However, to use these tools,&#10;programmers must send their code to untrusted third parties for processing&#10;(e.g., via an API call). This loss of custody is not acceptable to many&#10;organizations. In this paper, we present an alternative: we train an open&#10;source model using sample output generated by GPT-3.5 in a process related to&#10;knowledge distillation. Our model is small enough (350m parameters) to be run&#10;on a single 16gb GPU, yet we show in our evaluation that it is large enough to&#10;mimic GPT-3.5 on this task.&#10;" />
          <attvalue for="2" value="&#10;&#10;A code summary is a brief, natural language description of source code. Summaries are typically only a single sentence. When reading a Java method, for instance, a programmer may start with the Javadoc sentence ``changes all visible polygons to the color blue.'' The summary provides a quick way for the programmer to understand what the method does without having to read the code itself. The benefits of summaries in documentation have been studied for decades~\cite{haiduc2010use}, and Software Engineering (SE) research has long sought to automate the process of writing them, to reduce manual effort by programmers, support under-documented legacy programs, and build accessibility tools~\cite{robillard2017demand}. Code summaries form the backbone of much documentation for programmers, and the dream of automatic generation of these summaries has been described as a ``holy grail'' of SE research~\cite{allamanis2018survey, forward2002relevance, leclair2019neural}.&#10;&#10;Recently, the dream seems within reach. Years of effort on neural code summarization techniques has culminated in products such as Copilot~\cite{github2022copilot} and ChatGPT~\cite{openai2022chatgpt}, which exhibit an ability to describe arbitrary code~\cite{ma2023scope}. At the heart of these products is a language model that is trained using big data input. The language model in the most powerful products may be tens or hundreds of billions of parameters, and the data input often includes trillions of tokens, such as the entirety of public GitHub repositories, plus StackOverflow, Wikipedia, etc. The effectiveness of these products has captured the public imagination and helped drive a new wave of research~\cite{sun2023automatic}. Like in many research areas, the decades-long effort toward automatic code summarization suddenly seems at hand.&#10;&#10;Yet a major problem looms. For programmers to use these tools, they must send their code to third parties for processing. An IDE plugin wishing to use GPT-3.5, for example, must harvest code from the programmer's codebase and send it via an API call to OpenAI. This call is a loss of data custody and a non-starter for many institutions~\cite{derner2023beyond}. In addition, the closed nature of these products has caused controversy among researchers, who point out a lack of reproducibility, potential data contamination from public test sets to private training data, and resultant loss of scientific rigor~\cite{hellendoorn2021growing}. The situation for many programmers is that the technology to automate a major portion of code summarization exists, but it is not usable.&#10;&#10;In this paper, we present an alternative: knowledge distillation from a large model (GPT-3.5) to smaller models. Our paper has three key novel research contributions: &#10;&#10;\begin{enumerate}&#10;\item We present a study comparing summaries from GPT-3.5 to the reference summaries written by human programmers, and show that the generated summaries tend to be superior, indicating they are good source of training data. (Section~\ref{sec:studyone})&#10;\item We present a study of knowledge distillation of these summaries for the size of model (38m - 15.5B parameters) and size of training data (170k - 2.15m samples). We collect 2.15m summaries generated by GPT-3.5 for Java methods. We use a simple prompt and methods from open-source Java programs. As foundation models, we compare jam~\cite{su2023language} and starcoder~\cite{li2023starcoder}. The jam model is pretrained on 52m Java methods and has an easily-searchable dataset to ensure reproducibility and a controlled experimental framework. The starcoder model is much larger and has a larger pretraining dataset, but is also more expensive and has more non-controllable experimental variables due the the dataset. (Section~\ref{sec:studytwo})&#10;\item We evaluate our distilled model against GPT-3.5 in a study with human experts. (Section~\ref{sec:studythree})&#10;\end{enumerate}&#10;&#10;We release all code and implementation details. The model we recommend from our experiments can be run from a workstation with a single 16GB GPU, which is relatively low cost for many organizations. This low cost and open source structure enables programmers to access automatic code summarization while keeping data custody. Although there are some papers that have already formulated the code summarization as a fine-tuning problem such as~\cite{wang2021codet5, bender2021danger} and studied knowledge distillation for smaller models from larger models~\cite{hsieh2023distilling, yu2023large}, we thoroughly explore the data and model size for knowledge distillation on code summarization and conduct the human study to compare the language models generated summary and human reference with human experts.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Code Summarization, Computer Science, Large Language Models, Linguistics, Secure Code Analysis, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.12865" label="2305.12865">
        <attvalues>
          <attvalue for="0" value="Automatic Code Summarization via ChatGPT: How Far Are We?" />
          <attvalue for="1" value="  To support software developers in understanding and maintaining programs,&#10;various automatic code summarization techniques have been proposed to generate&#10;a concise natural language comment for a given code snippet. Recently, the&#10;emergence of large language models (LLMs) has led to a great boost in the&#10;performance of natural language processing tasks. Among them, ChatGPT is the&#10;most popular one which has attracted wide attention from the software&#10;engineering community. However, it still remains unclear how ChatGPT performs&#10;in (automatic) code summarization. Therefore, in this paper, we focus on&#10;evaluating ChatGPT on a widely-used Python dataset called CSN-Python and&#10;comparing it with several state-of-the-art (SOTA) code summarization models.&#10;Specifically, we first explore an appropriate prompt to guide ChatGPT to&#10;generate in-distribution comments. Then, we use such a prompt to ask ChatGPT to&#10;generate comments for all code snippets in the CSN-Python test set. We adopt&#10;three widely-used metrics (including BLEU, METEOR, and ROUGE-L) to measure the&#10;quality of the comments generated by ChatGPT and SOTA models (including NCS,&#10;CodeBERT, and CodeT5). The experimental results show that in terms of BLEU and&#10;ROUGE-L, ChatGPT's code summarization performance is significantly worse than&#10;all three SOTA models. We also present some cases and discuss the advantages&#10;and disadvantages of ChatGPT in code summarization. Based on the findings, we&#10;outline several open challenges and opportunities in ChatGPT-based code&#10;summarization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.08005" label="2305.08005">
        <attvalues>
          <attvalue for="0" value="Beyond the Safeguards: Exploring the Security Risks of ChatGPT" />
          <attvalue for="1" value="  The increasing popularity of large language models (LLMs) such as ChatGPT has&#10;led to growing concerns about their safety, security risks, and ethical&#10;implications. This paper aims to provide an overview of the different types of&#10;security risks associated with ChatGPT, including malicious text and code&#10;generation, private data disclosure, fraudulent services, information&#10;gathering, and producing unethical content. We present an empirical study&#10;examining the effectiveness of ChatGPT's content filters and explore potential&#10;ways to bypass these safeguards, demonstrating the ethical implications and&#10;security risks that persist in LLMs even when protections are in place. Based&#10;on a qualitative analysis of the security implications, we discuss potential&#10;strategies to mitigate these risks and inform researchers, policymakers, and&#10;industry professionals about the complex security challenges posed by LLMs like&#10;ChatGPT. This study contributes to the ongoing discussion on the ethical and&#10;security implications of LLMs, underscoring the need for continued research in&#10;this area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.08286" label="2305.08286">
        <attvalues>
          <attvalue for="0" value="A Language Model of Java Methods with Train/Test Deduplication" />
          <attvalue for="1" value="  This tool demonstration presents a research toolkit for a language model of&#10;Java source code. The target audience includes researchers studying problems at&#10;the granularity level of subroutines, statements, or variables in Java. In&#10;contrast to many existing language models, we prioritize features for&#10;researchers including an open and easily-searchable training set, a held out&#10;test set with different levels of deduplication from the training set,&#10;infrastructure for deduplicating new examples, and an implementation platform&#10;suitable for execution on equipment accessible to a relatively modest budget.&#10;Our model is a GPT2-like architecture with 350m parameters. Our training set&#10;includes 52m Java methods (9b tokens) and 13m StackOverflow threads (10.5b&#10;tokens). To improve accessibility of research to more members of the community,&#10;we limit local resource requirements to GPUs with 16GB video memory. We provide&#10;a test set of held out Java methods that include descriptive comments,&#10;including the entire Java projects for those methods. We also provide&#10;deduplication tools using precomputed hash tables at various similarity&#10;thresholds to help researchers ensure that their own test examples are not in&#10;the training set. We make all our tools and data open source and available via&#10;Huggingface and Github.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.04842" label="2309.04842">
        <attvalues>
          <attvalue for="0" value="Leveraging Large Language Models for Exploiting ASR Uncertainty" />
          <attvalue for="1" value="  While large language models excel in a variety of natural language processing&#10;(NLP) tasks, to perform well on spoken language understanding (SLU) tasks, they&#10;must either rely on off-the-shelf automatic speech recognition (ASR) systems&#10;for transcription, or be equipped with an in-built speech modality. This work&#10;focuses on the former scenario, where LLM's accuracy on SLU tasks is&#10;constrained by the accuracy of a fixed ASR system on the spoken input.&#10;Specifically, we tackle speech-intent classification task, where a high&#10;word-error-rate can limit the LLM's ability to understand the spoken intent.&#10;Instead of chasing a high accuracy by designing complex or specialized&#10;architectures regardless of deployment costs, we seek to answer how far we can&#10;go without substantially changing the underlying ASR and LLM, which can&#10;potentially be shared by multiple unrelated tasks. To this end, we propose&#10;prompting the LLM with an n-best list of ASR hypotheses instead of only the&#10;error-prone 1-best hypothesis. We explore prompt-engineering to explain the&#10;concept of n-best lists to the LLM; followed by the finetuning of Low-Rank&#10;Adapters on the downstream tasks. Our approach using n-best lists proves to be&#10;effective on a device-directed speech detection task as well as on a keyword&#10;spotting task, where systems using n-best list prompts outperform those using&#10;1-best ASR hypothesis; thus paving the way for an efficient method to exploit&#10;ASR uncertainty via LLMs for speech-based applications.&#10;" />
          <attvalue for="2" value="&#10;Large language models have recently revolutionized the field of NLP by showing excellent performance on a diverse set of downstream text-processing tasks often with little to no finetuning required on the downstream tasks~\cite{brown2020neurips,ouyang2022training,chen2021evaluating, bang2023multitask,qin2023chatgpt}. When trained with a large number of parameters in the order of billions (and even trillions~\cite{openai2023gpt4}) and similarly large quantities of text data, these models demonstrate an emergent ability to do in-context learning~\cite{xie2021explanation, min2022rethinking} and reasoning via chain-of-thought prompting~\cite{wei2023chainofthought}, which renders LLMs more accurate than dedicated smaller models trained on task-specific data.&#10;&#10;While this out-of-the-box generalizability and robustness of LLMs have instantly made them a popular tool for text-based applications~\cite{openai2023chatgpt,googlebard}, their usage for traditional speech-based applications is still an ongoing topic of research. One set of approaches relies on multi-modal LLMs, which ingest the audio modality by processing the underlying speech using an encoder network and feeding the LLM with speech embeddings~\cite{gong2023listen, fathullah2023prompting,zhang2023speechgpt,deshmukh2023pengi}. However, such architectures are mainly motivated towards the goal of having a single end-to-end model and the speech recognition capabilities of such multi-modal LLMs are limited~\cite{gong2023listen}. Another set of approaches interface LLMs with pretrained ASR models such that the ASR output is fed directly to the LLM as a prompt to tackle the downstream task~\cite{huang2023audiogpt,shen2023hugginggpt,he2023chatgpt}. The benefit of this modular approach is that one can choose any desirable in-domain ASR model and any LLM of appropriate size and configuration. In this work, we focus on such modular ASR+LLM architecture to approach the speech intent classification task and address the problem of incorrect ASR outputs affecting the ability of LLMs to determine the correct underlying intent.&#10;&#10;An important consideration regarding LLM-based solutions, that has not received enough attention in the literature, is the high cost of deploying them, either server-side or on-device. We argue that to viably deploy LLMs into production, multiple tasks need to utilize one underlying LLM, effectively sharing the cost. Therefore, it is crucial to seek out ways of using LLMs in a non-intrusive, shareable way. This consideration has informed the scope of our exploration. For example, off-the-shelf frozen ASR systems and LLMs are preferred over complex specialized models; prompting, if effective, is preferred over finetuning; and finally, LoRA finetuning is favored over full finetuning of LLMs.&#10;&#10;While ASR models aim to transcribe speech to text accurately, speech recognition on real-world speech is often inaccurate and word-error-rates (WER) are typically non-zero due to challenging conditions like noisy speech, environmental noise, accents, acoustic and speaker variations~\cite{yu2016automatic}. For any downstream SLU task, the 1-best ASR hypothesis acts as an information bottleneck between the ASR and the LLM component, as shown in Figure \ref{fig:motivation}. In this work, we propose to widen this information bottleneck by exposing the LLM to an $n$-best list of ASR hypotheses. Our hypothesis is that using $n$-best lists instead of 1-best enables the LLM to benefit from the uncertainties in ASR prediction instead of being adversely affected by them. $n$-best lists also conform to our goal of making minimal changes to an LLM as they are a prompting-friendly format to convey ASR uncertainty to the LLM as discussed in Section \ref{sec:prompts}. Using descriptive prompts which explain the concept of $n$-best lists to an LLM, or by finetuning the LLM with $n$-best lists as the prompt, we demonstrate that an LLM can interface with a frozen ASR model in a more effective way as compared to using just the 1-best ASR output directly.&#10;A relevant work~\cite{he2023chatgpt} utilizes ASR 1-best outputs as well as oracle transcripts as prompts to the LLM for speech-intent classification with encouraging results. Another work~\cite{ma2023generative} explores correcting ASR errors by prompting $n$-best lists to ChatGPT which results in improvements in ASR WER. Our work differs in multiple ways: (i) we explore prompting with $n$-best lists for the downstream intent classification and keyword spotting tasks, (ii) our $n$-best lists are augmented with ASR hypothesis costs as an additional source of information on uncertainty in ASR, and (iii) apart from direct prompting, we also explore finetuning of LoRA adapters using a training data of $n$-best list prompts. We use an in-house general-purpose English ASR system and Vicuna~\cite{vicuna2023}, an instruction-tuned LLaMA LLM~\cite{touvron2023llama} , in this paper, and we experiment on (i) a device-directed speech detection (DDSD) task for binary intent classification on whether an utterance is directed towards a voice assistant or not, and on (ii) a keyword spotting (KS) task on the Google Speech Commands (GSC)~\cite{warden2018speech} dataset, which we treat as a multi-class intent classification problem. While the labels for the binary classification DDSD task are typically 1 (directed) and 0 (undirected), we also explore the LLM's capability to output its decision on a scale of 0 and 100 so that the output score can be converted to a probability and used for generating smooth ROC curves.&#10;We show improved accuracy using $n$-best lists on both GSC and DDSD tasks. &#10;&#10;The rest of the paper is structured as follows: Section \ref{sec:approach} explains our methodology. Section \ref{sec:experiments} provides experimental details and analysis. Section \ref{sec:conclusions} summarizes the conclusions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Recognition Systems, Large Language Models, Linguistics, Artificial Intelligence, Signal Processing, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2204.00890" label="2204.00890">
        <attvalues>
          <attvalue for="0" value="From Simulated Mixtures to Simulated Conversations as Training Data for&#10;  End-to-End Neural Diarization" />
          <attvalue for="1" value="  End-to-end neural diarization (EEND) is nowadays one of the most prominent&#10;research topics in speaker diarization. EEND presents an attractive alternative&#10;to standard cascaded diarization systems since a single system is trained at&#10;once to deal with the whole diarization problem. Several EEND variants and&#10;approaches are being proposed, however, all these models require large amounts&#10;of annotated data for training but available annotated data are scarce. Thus,&#10;EEND works have used mostly simulated mixtures for training. However, simulated&#10;mixtures do not resemble real conversations in many aspects. In this work we&#10;present an alternative method for creating synthetic conversations that&#10;resemble real ones by using statistics about distributions of pauses and&#10;overlaps estimated on genuine conversations. Furthermore, we analyze the effect&#10;of the source of the statistics, different augmentations and amounts of data.&#10;We demonstrate that our approach performs substantially better than the&#10;original one, while reducing the dependence on the fine-tuning stage.&#10;Experiments are carried out on 2-speaker telephone conversations of Callhome&#10;and DIHARD 3. Together with this publication, we release our implementations of&#10;EEND and the method for creating simulated conversations.&#10;" />
          <attvalue for="2" value="&#10;&#10;Since the introduction of end-to-end neural diarization (EEND)~\cite{fujita2019enda} and its extension to deal with arbitrary amounts of speakers~\cite{fujita2020neural,horiguchi2020end}, it has been established as a state-of-the-art alternative to the standard cascaded diarization systems based on different submodules, i.e. voice activity detection (VAD), uniform segmentation, speaker embeddings extraction, clustering and overlapped speech detection (OSD) with handling. &#10;EEND formulates the speaker diarization problem as a per-speaker-per-time-frame binary classification problem where a permutation-free objective is used to minimize the speech activity error for all speakers. Therefore, EEND models generate one speech activity probability output for each speaker per time-step, which effectively solves VAD, speaker labeling and OSD at once. &#10;&#10;Most works following the EEND principle have focused on improvements on the architecture or modeling. Some by using self-attention layers~\cite{fujita2019endb} or conformer layers~\cite{chieh2021end} instead of the original BLSTM layers for feature encoding; others have focused on more complex diarization scenarios such as its online fashion~\cite{han2021bw,xue2021onlineA} or when more than one microphone is available~\cite{horiguchi2021multi} or by improving the model iteratively using pseudo-labels~\cite{takashima2021end}. Some have used EEND together with more standard approaches by using EEND-inspired models to find overlaps among pairs of speakers in the output of a cascaded system~\cite{horiguchi2021end} or leveraging EEND's VAD performance by using an external VAD system~\cite{horiguchi2021hitachi} or combining short duration diarization outputs to produce better whole-utterance diarization~\cite{kinoshita2021integrating,kinoshita2021advances,horiguchi2021towards,kinoshita2022tight}. However, none have yet tackled one of the main aspects of EEND: training data generation.&#10;&#10;EEND models require large amounts of training data and datasets manually annotated for diarization do not amount for thousands of hours. When presenting the first version of EEND, Yusuke et al. proposed a strategy for constructing simulated mixtures using telephone conversations from different collections and this strategy has been used with both telephony data or read books to create simulated mixtures by mixing speakers from different recordings. However, little analysis has been presented about how the simulations were devised nor what impact have the used augmentations. Furthermore, the mixtures do not resemble real conversations, specially when more than two speakers are included. In this work we revise the approach and propose an alternative method that, based on statistics from real conversations, emulates some of the attributes of natural conversations. We also analyze the impact of using different types of augmentations on the diarization performance and show that our approach performs better than the original one while significantly reducing the dependence on the fine-tuning stage.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Synthetic Conversation, Speaker Recognition, Neural Diarization, Signal Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1909.05952" label="1909.05952">
        <attvalues>
          <attvalue for="0" value="End-to-End Neural Speaker Diarization with Permutation-Free Objectives" />
          <attvalue for="1" value="  In this paper, we propose a novel end-to-end neural-network-based speaker&#10;diarization method. Unlike most existing methods, our proposed method does not&#10;have separate modules for extraction and clustering of speaker representations.&#10;Instead, our model has a single neural network that directly outputs speaker&#10;diarization results. To realize such a model, we formulate the speaker&#10;diarization problem as a multi-label classification problem, and introduces a&#10;permutation-free objective function to directly minimize diarization errors&#10;without being suffered from the speaker-label permutation problem. Besides its&#10;end-to-end simplicity, the proposed method also benefits from being able to&#10;explicitly handle overlapping speech during training and inference. Because of&#10;the benefit, our model can be easily trained/adapted with real-recorded&#10;multi-speaker conversations just by feeding the corresponding multi-speaker&#10;segment labels. We evaluated the proposed method on simulated speech mixtures.&#10;The proposed method achieved diarization error rate of 12.28%, while a&#10;conventional clustering-based system produced diarization error rate of 28.77%.&#10;Furthermore, the domain adaptation with real-recorded speech provided 25.6%&#10;relative improvement on the CALLHOME dataset. Our source code is available&#10;online at https://github.com/hitachi-speech/EEND.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.01796" label="2006.01796">
        <attvalues>
          <attvalue for="0" value="Neural Speaker Diarization with Speaker-Wise Chain Rule" />
          <attvalue for="1" value="  Speaker diarization is an essential step for processing multi-speaker audio.&#10;Although an end-to-end neural diarization (EEND) method achieved&#10;state-of-the-art performance, it is limited to a fixed number of speakers. In&#10;this paper, we solve this fixed number of speaker issue by a novel speaker-wise&#10;conditional inference method based on the probabilistic chain rule. In the&#10;proposed method, each speaker's speech activity is regarded as a single random&#10;variable, and is estimated sequentially conditioned on previously estimated&#10;other speakers' speech activities. Similar to other sequence-to-sequence&#10;models, the proposed method produces a variable number of speakers with a stop&#10;sequence condition. We evaluated the proposed method on multi-speaker audio&#10;recordings of a variable number of speakers. Experimental results show that the&#10;proposed method can correctly produce diarization results with a variable&#10;number of speakers and outperforms the state-of-the-art end-to-end speaker&#10;diarization methods in terms of diarization error rate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09921" label="2005.09921">
        <attvalues>
          <attvalue for="0" value="End-to-End Speaker Diarization for an Unknown Number of Speakers with&#10;  Encoder-Decoder Based Attractors" />
          <attvalue for="1" value="  End-to-end speaker diarization for an unknown number of speakers is addressed&#10;in this paper. Recently proposed end-to-end speaker diarization outperformed&#10;conventional clustering-based speaker diarization, but it has one drawback: it&#10;is less flexible in terms of the number of speakers. This paper proposes a&#10;method for encoder-decoder based attractor calculation (EDA), which first&#10;generates a flexible number of attractors from a speech embedding sequence.&#10;Then, the generated multiple attractors are multiplied by the speech embedding&#10;sequence to produce the same number of speaker activities. The speech embedding&#10;sequence is extracted using the conventional self-attentive end-to-end neural&#10;speaker diarization (SA-EEND) network. In a two-speaker condition, our method&#10;achieved a 2.69 % diarization error rate (DER) on simulated mixtures and a 8.07&#10;% DER on the two-speaker subset of CALLHOME, while vanilla SA-EEND attained&#10;4.56 % and 9.54 %, respectively. In unknown numbers of speakers conditions, our&#10;method attained a 15.29 % DER on CALLHOME, while the x-vector-based clustering&#10;method achieved a 19.43 % DER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.06247" label="1909.06247">
        <attvalues>
          <attvalue for="0" value="End-to-End Neural Speaker Diarization with Self-attention" />
          <attvalue for="1" value="  Speaker diarization has been mainly developed based on the clustering of&#10;speaker embeddings. However, the clustering-based approach has two major&#10;problems; i.e., (i) it is not optimized to minimize diarization errors&#10;directly, and (ii) it cannot handle speaker overlaps correctly. To solve these&#10;problems, the End-to-End Neural Diarization (EEND), in which a bidirectional&#10;long short-term memory (BLSTM) network directly outputs speaker diarization&#10;results given a multi-talker recording, was recently proposed. In this study,&#10;we enhance EEND by introducing self-attention blocks instead of BLSTM blocks.&#10;In contrast to BLSTM, which is conditioned only on its previous and next hidden&#10;states, self-attention is directly conditioned on all the other frames, making&#10;it much suitable for dealing with the speaker diarization problem. We evaluated&#10;our proposed method on simulated mixtures, real telephone calls, and real&#10;dialogue recordings. The experimental results revealed that the self-attention&#10;was the key to achieving good performance and that our proposed method&#10;performed significantly better than the conventional BLSTM-based method. Our&#10;method was even better than that of the state-of-the-art x-vector&#10;clustering-based method. Finally, by visualizing the latent representation, we&#10;show that the self-attention can capture global speaker characteristics in&#10;addition to local speech activity dynamics. Our source code is available online&#10;at https://github.com/hitachi-speech/EEND.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.07167" label="2106.07167">
        <attvalues>
          <attvalue for="0" value="End-to-end Neural Diarization: From Transformer to Conformer" />
          <attvalue for="1" value="  We propose a new end-to-end neural diarization (EEND) system that is based on&#10;Conformer, a recently proposed neural architecture that combines convolutional&#10;mappings and Transformer to model both local and global dependencies in speech.&#10;We first show that data augmentation and convolutional subsampling layers&#10;enhance the original self-attentive EEND in the Transformer-based EEND, and&#10;then Conformer gives an additional gain over the Transformer-based EEND.&#10;However, we notice that the Conformer-based EEND does not generalize as well&#10;from simulated to real conversation data as the Transformer-based model. This&#10;leads us to quantify the mismatch between simulated data and real speaker&#10;behavior in terms of temporal statistics reflecting turn-taking between&#10;speakers, and investigate its correlation with diarization error. By mixing&#10;simulated and real data in EEND training, we mitigate the mismatch further,&#10;with Conformer-based EEND achieving 24% error reduction over the baseline&#10;SA-EEND system, and 10% improvement over the best augmented Transformer-based&#10;system, on two-speaker CALLHOME data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.02678" label="2011.02678">
        <attvalues>
          <attvalue for="0" value="BW-EDA-EEND: Streaming End-to-End Neural Speaker Diarization for a&#10;  Variable Number of Speakers" />
          <attvalue for="1" value="  We present a novel online end-to-end neural diarization system, BW-EDA-EEND,&#10;that processes data incrementally for a variable number of speakers. The system&#10;is based on the Encoder-Decoder-Attractor (EDA) architecture of Horiguchi et&#10;al., but utilizes the incremental Transformer encoder, attending only to its&#10;left contexts and using block-level recurrence in the hidden states to carry&#10;information from block to block, making the algorithm complexity linear in&#10;time. We propose two variants: For unlimited-latency BW-EDA-EEND, which&#10;processes inputs in linear time, we show only moderate degradation for up to&#10;two speakers using a context size of 10 seconds compared to offline EDA-EEND.&#10;With more than two speakers, the accuracy gap between online and offline grows,&#10;but the algorithm still outperforms a baseline offline clustering diarization&#10;system for one to four speakers with unlimited context size, and shows&#10;comparable accuracy with context size of 10 seconds. For limited-latency&#10;BW-EDA-EEND, which produces diarization outputs block-by-block as audio&#10;arrives, we show accuracy comparable to the offline clustering-based system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.02616" label="2006.02616">
        <attvalues>
          <attvalue for="0" value="Online End-to-End Neural Diarization with Speaker-Tracing Buffer" />
          <attvalue for="1" value="  This paper proposes a novel online speaker diarization algorithm based on a&#10;fully supervised self-attention mechanism (SA-EEND). Online diarization&#10;inherently presents a speaker's permutation problem due to the possibility to&#10;assign speaker regions incorrectly across the recording. To circumvent this&#10;inconsistency, we proposed a speaker-tracing buffer mechanism that selects&#10;several input frames representing the speaker permutation information from&#10;previous chunks and stores them in a buffer. These buffered frames are stacked&#10;with the input frames in the current chunk and fed into a self-attention&#10;network. Our method ensures consistent diarization outputs across the buffer&#10;and the current chunk by checking the correlation between their corresponding&#10;outputs. Additionally, we trained SA-EEND with variable chunk-sizes to mitigate&#10;the mismatch between training and inference introduced by the speaker-tracing&#10;buffer mechanism. Experimental results, including online SA-EEND and variable&#10;chunk-size, achieved DERs of 12.54% for CALLHOME and 20.77% for CSJ with 1.4s&#10;actual latency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04694" label="2110.04694">
        <attvalues>
          <attvalue for="0" value="Multi-Channel End-to-End Neural Diarization with Distributed Microphones" />
          <attvalue for="1" value="  Recent progress on end-to-end neural diarization (EEND) has enabled&#10;overlap-aware speaker diarization with a single neural network. This paper&#10;proposes to enhance EEND by using multi-channel signals from distributed&#10;microphones. We replace Transformer encoders in EEND with two types of encoders&#10;that process a multi-channel input: spatio-temporal and co-attention encoders.&#10;Both are independent of the number and geometry of microphones and suitable for&#10;distributed microphone settings. We also propose a model adaptation method&#10;using only single-channel recordings. With simulated and real-recorded&#10;datasets, we demonstrated that the proposed method outperformed conventional&#10;EEND when a multi-channel input was given while maintaining comparable&#10;performance with a single-channel input. We also showed that the proposed&#10;method performed well even when spatial information is inoperative given&#10;multi-channel inputs, such as in hybrid meetings in which the utterances of&#10;multiple remote participants are played back from the same loudspeaker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.04078" label="2106.04078">
        <attvalues>
          <attvalue for="0" value="End-to-End Speaker Diarization Conditioned on Speech Activity and&#10;  Overlap Detection" />
          <attvalue for="1" value="  In this paper, we present a conditional multitask learning method for&#10;end-to-end neural speaker diarization (EEND). The EEND system has shown&#10;promising performance compared with traditional clustering-based methods,&#10;especially in the case of overlapping speech. In this paper, to further improve&#10;the performance of the EEND system, we propose a novel multitask learning&#10;framework that solves speaker diarization and a desired subtask while&#10;explicitly considering the task dependency. We optimize speaker diarization&#10;conditioned on speech activity and overlap detection that are subtasks of&#10;speaker diarization, based on the probabilistic chain rule. Experimental&#10;results show that our proposed method can leverage a subtask to effectively&#10;model speaker diarization, and outperforms conventional EEND systems in terms&#10;of diarization error rate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.10055" label="2012.10055">
        <attvalues>
          <attvalue for="0" value="End-to-End Speaker Diarization as Post-Processing" />
          <attvalue for="1" value="  This paper investigates the utilization of an end-to-end diarization model as&#10;post-processing of conventional clustering-based diarization. Clustering-based&#10;diarization methods partition frames into clusters of the number of speakers;&#10;thus, they typically cannot handle overlapping speech because each frame is&#10;assigned to one speaker. On the other hand, some end-to-end diarization methods&#10;can handle overlapping speech by treating the problem as multi-label&#10;classification. Although some methods can treat a flexible number of speakers,&#10;they do not perform well when the number of speakers is large. To compensate&#10;for each other's weakness, we propose to use a two-speaker end-to-end&#10;diarization method as post-processing of the results obtained by a&#10;clustering-based method. We iteratively select two speakers from the results&#10;and update the results of the two speakers to improve the overlapped region.&#10;Experimental results show that the proposed algorithm consistently improved the&#10;performance of the state-of-the-art methods across CALLHOME, AMI, and DIHARD II&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.01363" label="2102.01363">
        <attvalues>
          <attvalue for="0" value="The Hitachi-JHU DIHARD III System: Competitive End-to-End Neural&#10;  Diarization and X-Vector Clustering Systems Combined by DOVER-Lap" />
          <attvalue for="1" value="  This paper provides a detailed description of the Hitachi-JHU system that was&#10;submitted to the Third DIHARD Speech Diarization Challenge. The system outputs&#10;the ensemble results of the five subsystems: two x-vector-based subsystems, two&#10;end-to-end neural diarization-based subsystems, and one hybrid subsystem. We&#10;refine each system and all five subsystems become competitive and&#10;complementary. After the DOVER-Lap based system combination, it achieved&#10;diarization error rates of 11.58 % and 14.09 % in Track 1 full and core, and&#10;16.94 % and 20.01 % in Track 2 full and core, respectively. With their results,&#10;we won second place in all the tasks of the challenge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13366" label="2010.13366">
        <attvalues>
          <attvalue for="0" value="Integrating end-to-end neural and clustering-based diarization: Getting&#10;  the best of both worlds" />
          <attvalue for="1" value="  Recent diarization technologies can be categorized into two approaches, i.e.,&#10;clustering and end-to-end neural approaches, which have different pros and&#10;cons. The clustering-based approaches assign speaker labels to speech regions&#10;by clustering speaker embeddings such as x-vectors. While it can be seen as a&#10;current state-of-the-art approach that works for various challenging data with&#10;reasonable robustness and accuracy, it has a critical disadvantage that it&#10;cannot handle overlapped speech that is inevitable in natural conversational&#10;data. In contrast, the end-to-end neural diarization (EEND), which directly&#10;predicts diarization labels using a neural network, was devised to handle the&#10;overlapped speech. While the EEND, which can easily incorporate emerging&#10;deep-learning technologies, has started outperforming the x-vector clustering&#10;approach in some realistic database, it is difficult to make it work for `long'&#10;recordings (e.g., recordings longer than 10 minutes) because of, e.g., its huge&#10;memory consumption. Block-wise independent processing is also difficult because&#10;it poses an inter-block label permutation problem, i.e., an ambiguity of the&#10;speaker label assignments between blocks. In this paper, we propose a simple&#10;but effective hybrid diarization framework that works with overlapped speech&#10;and for long recordings containing an arbitrary number of speakers. It modifies&#10;the conventional EEND framework to simultaneously output global speaker&#10;embeddings so that speaker clustering can be performed across blocks to solve&#10;the permutation problem. With experiments based on simulated noisy reverberant&#10;2-speaker meeting-like data, we show that the proposed framework works&#10;significantly better than the original EEND especially when the input data is&#10;long.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.09040" label="2105.09040">
        <attvalues>
          <attvalue for="0" value="Advances in integration of end-to-end neural and clustering-based&#10;  diarization for real conversational speech" />
          <attvalue for="1" value="  Recently, we proposed a novel speaker diarization method called&#10;End-to-End-Neural-Diarization-vector clustering (EEND-vector clustering) that&#10;integrates clustering-based and end-to-end neural network-based diarization&#10;approaches into one framework. The proposed method combines advantages of both&#10;frameworks, i.e. high diarization performance and handling of overlapped speech&#10;based on EEND, and robust handling of long recordings with an arbitrary number&#10;of speakers based on clustering-based approaches. However, the method was only&#10;evaluated so far on simulated 2-speaker meeting-like data. This paper is to (1)&#10;report recent advances we made to this framework, including newly introduced&#10;robust constrained clustering algorithms, and (2) experimentally show that the&#10;method can now significantly outperform competitive diarization methods such as&#10;Encoder-Decoder Attractor (EDA)-EEND, on CALLHOME data which comprises real&#10;conversational speech data including overlapped speech and an arbitrary number&#10;of speakers. By further analyzing the experimental results, this paper also&#10;discusses pros and cons of the proposed method and reveals potential for&#10;further improvement. A set of the code to reproduce the results is available at&#10;https://github.com/nttcslab-sp/EEND-vector-clustering.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.01545" label="2107.01545">
        <attvalues>
          <attvalue for="0" value="Towards Neural Diarization for Unlimited Numbers of Speakers Using&#10;  Global and Local Attractors" />
          <attvalue for="1" value="  Attractor-based end-to-end diarization is achieving comparable accuracy to&#10;the carefully tuned conventional clustering-based methods on challenging&#10;datasets. However, the main drawback is that it cannot deal with the case where&#10;the number of speakers is larger than the one observed during training. This is&#10;because its speaker counting relies on supervised learning. In this work, we&#10;introduce an unsupervised clustering process embedded in the attractor-based&#10;end-to-end diarization. We first split a sequence of frame-wise embeddings into&#10;short subsequences and then perform attractor-based diarization for each&#10;subsequence. Given subsequence-wise diarization results, inter-subsequence&#10;speaker correspondence is obtained by unsupervised clustering of the vectors&#10;computed from the attractors from all the subsequences. This makes it possible&#10;to produce diarization results of a large number of speakers for the whole&#10;recording even if the number of output speakers for each subsequence is&#10;limited. Experimental results showed that our method could produce accurate&#10;diarization results of an unseen number of speakers. Our method achieved 11.84&#10;%, 28.33 %, and 19.49 % on the CALLHOME, DIHARD II, and DIHARD III datasets,&#10;respectively, each of which is better than the conventional end-to-end&#10;diarization methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.01608" label="1901.01608">
        <attvalues>
          <attvalue for="0" value="Scaling description of generalization with number of parameters in deep&#10;  learning" />
          <attvalue for="1" value="  Supervised deep learning involves the training of neural networks with a&#10;large number $N$ of parameters. For large enough $N$, in the so-called&#10;over-parametrized regime, one can essentially fit the training data points.&#10;Sparsity-based arguments would suggest that the generalization error increases&#10;as $N$ grows past a certain threshold $N^{*}$. Instead, empirical studies have&#10;shown that in the over-parametrized regime, generalization error keeps&#10;decreasing with $N$. We resolve this paradox through a new framework. We rely&#10;on the so-called Neural Tangent Kernel, which connects large neural nets to&#10;kernel methods, to show that the initialization causes finite-size random&#10;fluctuations $\|f_{N}-\bar{f}_{N}\|\sim N^{-1/4}$ of the neural net output&#10;function $f_{N}$ around its expectation $\bar{f}_{N}$. These affect the&#10;generalization error $\epsilon_{N}$ for classification: under natural&#10;assumptions, it decays to a plateau value $\epsilon_{\infty}$ in a power-law&#10;fashion $\sim N^{-1/2}$. This description breaks down at a so-called jamming&#10;transition $N=N^{*}$. At this threshold, we argue that $\|f_{N}\|$ diverges.&#10;This result leads to a plausible explanation for the cusp in test error known&#10;to occur at $N^{*}$. Our results are confirmed by extensive empirical&#10;observations on the MNIST and CIFAR image datasets. Our analysis finally&#10;suggests that, given a computational envelope, the smallest generalization&#10;error is obtained using several networks of intermediate sizes, just beyond&#10;$N^{*}$, and averaging their outputs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.12191" label="1904.12191">
        <attvalues>
          <attvalue for="0" value="Linearized two-layers neural networks in high dimension" />
          <attvalue for="1" value="  We consider the problem of learning an unknown function $f_{\star}$ on the&#10;$d$-dimensional sphere with respect to the square loss, given i.i.d. samples&#10;$\{(y_i,{\boldsymbol x}_i)\}_{i\le n}$ where ${\boldsymbol x}_i$ is a feature&#10;vector uniformly distributed on the sphere and $y_i=f_{\star}({\boldsymbol&#10;x}_i)+\varepsilon_i$. We study two popular classes of models that can be&#10;regarded as linearizations of two-layers neural networks around a random&#10;initialization: the random features model of Rahimi-Recht (RF); the neural&#10;tangent kernel model of Jacot-Gabriel-Hongler (NT). Both these approaches can&#10;also be regarded as randomized approximations of kernel ridge regression (with&#10;respect to different kernels), and enjoy universal approximation properties&#10;when the number of neurons $N$ diverges, for a fixed dimension $d$.&#10;  We consider two specific regimes: the approximation-limited regime, in which&#10;$n=\infty$ while $d$ and $N$ are large but finite; and the sample size-limited&#10;regime in which $N=\infty$ while $d$ and $n$ are large but finite. In the first&#10;regime we prove that if $d^{\ell + \delta} \le N\le d^{\ell+1-\delta}$ for&#10;small $\delta &gt; 0$, then \RF\, effectively fits a degree-$\ell$ polynomial in&#10;the raw features, and \NT\, fits a degree-$(\ell+1)$ polynomial. In the second&#10;regime, both RF and NT reduce to kernel methods with rotationally invariant&#10;kernels. We prove that, if the number of samples is $d^{\ell + \delta} \le n&#10;\le d^{\ell +1-\delta}$, then kernel methods can fit at most a a degree-$\ell$&#10;polynomial in the raw features. This lower bound is achieved by kernel ridge&#10;regression. Optimal prediction error is achieved for vanishing ridge&#10;regularization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.06701" label="2102.06701">
        <attvalues>
          <attvalue for="0" value="Explaining Neural Scaling Laws" />
          <attvalue for="1" value="  The population loss of trained deep neural networks often follows precise&#10;power-law scaling relations with either the size of the training dataset or the&#10;number of parameters in the network. We propose a theory that explains the&#10;origins of and connects these scaling laws. We identify variance-limited and&#10;resolution-limited scaling behavior for both dataset and model size, for a&#10;total of four scaling regimes. The variance-limited scaling follows simply from&#10;the existence of a well-behaved infinite data or infinite width limit, while&#10;the resolution-limited regime can be explained by positing that models are&#10;effectively resolving a smooth data manifold. In the large width limit, this&#10;can be equivalently obtained from the spectrum of certain kernels, and we&#10;present evidence that large width and large dataset resolution-limited scaling&#10;exponents are related by a duality. We exhibit all four scaling regimes in the&#10;controlled setting of large random feature and pretrained models and test the&#10;predictions empirically on a range of standard architectures and datasets. We&#10;also observe several empirical relationships between datasets and scaling&#10;exponents under modifications of task and architecture aspect ratio. Our work&#10;provides a taxonomy for classifying different scaling regimes, underscores that&#10;there can be different mechanisms driving improvements in loss, and lends&#10;insight into the microscopic origins of and relationships between scaling&#10;exponents.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.08203" label="2211.08203">
        <attvalues>
          <attvalue for="0" value="Investigating the Frequency Distortion of Word Embeddings and Its Impact&#10;  on Bias Metrics" />
          <attvalue for="1" value="  Recent research has shown that static word embeddings can encode word&#10;frequency information. However, little has been studied about this phenomenon&#10;and its effects on downstream tasks. In the present work, we systematically&#10;study the association between frequency and semantic similarity in several&#10;static word embeddings. We find that Skip-gram, GloVe and FastText embeddings&#10;tend to produce higher semantic similarity between high-frequency words than&#10;between other frequency combinations. We show that the association between&#10;frequency and similarity also appears when words are randomly shuffled. This&#10;proves that the patterns found are not due to real semantic associations&#10;present in the texts, but are an artifact produced by the word embeddings.&#10;Finally, we provide an example of how word frequency can strongly impact the&#10;measurement of gender bias with embedding-based metrics. In particular, we&#10;carry out a controlled experiment that shows that biases can even change sign&#10;or reverse their order by manipulating word frequencies.&#10;" />
          <attvalue for="2" value=" &#10;&#10;Static word embeddings have proven to encode semantic information of words and are therefore useful to solve tasks such as synonym selection and analogical reasoning \cite{mikolov2013distributed, levy2015improving}. More recent contextualized representations have achieved better results \cite{ethayarajh2019contextual}, specially in tasks where the local context of words is important \cite{sezerer2021survey}. However, static word embeddings are still widely used in computational social science studies that examine global aspects of corpora. For example, embeddings are trained on specific corpora and are used to compute metrics that quantify societal biases and stereotypes that might be present in the text \cite{garg2018word, kozlowski2019geometry, defranza2020language, jones2020stereotypical, lewis2020gender, charlesworth2021gender}. Static embeddings are also used in a wide range of applications like topic coherence evaluation \cite{aletras2013evaluating}, dream theory analysis \cite{altszyler2017interpretation}, literature studies \cite{diuk2012quantitative}, and cognitive science studies \cite{mota2022imagetic}. &#10;&#10;Previous research has found static word embeddings appear to be associated with word frequency in various ways: word frequency correlates with embedding norm \cite{wilson2015controlled, arora2016latent}, the nearest neighbors of the embeddings of medium-frequency English words are more unstable \cite{hellrich2016bad}, there are frequency-related differences in the distribution of the inner products between target and context vectors \cite{mimno2017strange}, embeddings can accurately predict whether a word is frequent or rare \cite{schnabel2015evaluation}, and the visual inspection of their top principal components suggest they encode frequency \cite{gong2018frage, mu2018allbutthetop}. When it comes to using embeddings to measure bias in text, \cite{valentini2022undesirable} found that gender embedding-based bias metrics can spuriously depend on word frequency. &#10;&#10;Our work addresses several gaps in the existing literature regarding the frequency distortion of static word embeddings and its impact on the quantification of biases in corpora. Even if it has been pointed out that embeddings can encode frequency, this is the first study that:&#10;&#10;\begin{enumerate}[leftmargin=*,noitemsep,topsep=0pt]&#10; \item Comprehensively investigates the association between frequency and similarity in commonly used embeddings.&#10; \item Examines whether embeddings encode frequency due to undesirable properties of embeddings or actual properties of corpora.&#10; \item Explores the persistence of the frequency distortion under different hyperparameter settings.&#10; \item Assesses the impact on a computational social science application, namely bias measurement. &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Semantic Similarity Patterns, Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Statistics, Language Bias Measurement, Word Embeddings Analysis" />
        </attvalues>
      </node>
      <node id="1510.02675" label="1510.02675">
        <attvalues>
          <attvalue for="0" value="Controlled Experiments for Word Embeddings" />
          <attvalue for="1" value="  An experimental approach to studying the properties of word embeddings is&#10;proposed. Controlled experiments, achieved through modifications of the&#10;training corpus, permit the demonstration of direct relations between word&#10;properties and word vector direction and length. The approach is demonstrated&#10;using the word2vec CBOW model with experiments that independently vary word&#10;frequency and word co-occurrence noise. The experiments reveal that word vector&#10;length depends more or less linearly on both word frequency and the level of&#10;noise in the co-occurrence distribution of the word. The coefficients of&#10;linearity depend upon the word. The special point in feature space, defined by&#10;the (artificial) word with pure noise in its co-occurrence distribution, is&#10;found to be small but non-zero.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.06858" label="1809.06858">
        <attvalues>
          <attvalue for="0" value="FRAGE: Frequency-Agnostic Word Representation" />
          <attvalue for="1" value="  Continuous word representation (aka word embedding) is a basic building block&#10;in many neural network-based models used in natural language processing tasks.&#10;Although it is widely accepted that words with similar semantics should be&#10;close to each other in the embedding space, we find that word embeddings&#10;learned in several tasks are biased towards word frequency: the embeddings of&#10;high-frequency and low-frequency words lie in different subregions of the&#10;embedding space, and the embedding of a rare word and a popular word can be far&#10;from each other even if they are semantically similar. This makes learned word&#10;embeddings ineffective, especially for rare words, and consequently limits the&#10;performance of these neural network models. In this paper, we develop a neat,&#10;simple yet effective way to learn \emph{FRequency-AGnostic word Embedding}&#10;(FRAGE) using adversarial training. We conducted comprehensive studies on ten&#10;datasets across four natural language processing tasks, including word&#10;similarity, language modeling, machine translation and text classification.&#10;Results show that with FRAGE, we achieve higher performance than the baselines&#10;in all tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.07327" label="2406.07327">
        <attvalues>
          <attvalue for="0" value="3D-Properties: Identifying Challenges in DPO and Charting a Path Forward" />
          <attvalue for="1" value="  Aligning large language models (LLMs) with human preference has recently&#10;gained tremendous attention, with the canonical yet costly RLHF-PPO and the&#10;simple and straightforward Direct Preference Optimization (DPO) as two&#10;examples. Despite the efficiency, DPO has rarely be used in the&#10;state-of-the-art production-level LLMs, implying its potential pathologies. In&#10;this work, we revisit DPO with a comprehensive examination of its empirical&#10;efficacy and a systematic comparison with RLHF-PPO. We identify the&#10;\textbf{3D}-properties of DPO's learning outcomes: the \textbf{D}rastic drop in&#10;the likelihood of rejected responses, the \textbf{D}egradation into LLM&#10;unlearning, and the \textbf{D}ispersion effect on unseen responses through&#10;experiments with both a carefully designed toy model and practical LLMs on&#10;tasks including mathematical problem-solving and instruction following. These&#10;findings inherently connect to some observations made by related works and we&#10;additionally contribute a plausible theoretical explanation for them.&#10;Accordingly, we propose easy regularization methods to mitigate the issues&#10;caused by \textbf{3D}-properties, improving the training stability and final&#10;performance of DPO. Our contributions also include an investigation into how&#10;the distribution of the paired preference data impacts the effectiveness of&#10;DPO. We hope this work could offer research directions to narrow the gap&#10;between reward-free preference learning methods and reward-based ones.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks and domains~\cite{touvron2023llama, chowdhery2023palm, jiang2023mistral, zhang2022opt}. Several techniques have been developed for fine-tuning LLMs, most notably Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF)~\cite{achiam2023gpt, touvron2023llama}. SFT involves directly training LLMs on labeled data to tailor their responses for specific tasks, whereas RLHF refines LLMs by incorporating feedback that aligns their outputs with human preferences. RLHF, in particular, has been instrumental in expanding the application of both closed-source~\cite{OMS, claude_2024, team2023gemini} and open-source LLMs~\cite{touvron2023llama, yang2023baichuan}, driven by the need to align foundational models with human&#10;values and preferences~\cite{ziegler2019fine, stiennon2020learning, ouyang2022training}.&#10;&#10;Existing RLHF methods can be majorly categorized into two classes based on whether the reward signal is explicitly modeled. Reward-model-based (RM-based) alignment pioneered by OpenAI~\cite{ouyang2022training,achiam2023gpt, touvron2023llama} first trains a Reward Model (RM) from user preferences, typically through Maximum Likelihood Estimation (MLE), and then leverages actor-critic algorithms such as Proximal Policy Optimization (PPO)~\cite{schulman2017proximal} to tune the SFT model to realize alignment. This approach often requires substantial computational resources and suffers from sample inefficiency~\cite{choshen2019weaknesses}. Conversely, another class of methods, known as reward-model-free (RM-free) alignment, such as Direct Preference Optimization (DPO)~\cite{rafailov2024direct}, Identity Preference Optimization (IPO)~\cite{azar2024general}, Sequence Likelihood Calibration (SLiC)~\cite{zhao2023slic}, DPO-positive~\cite{pal2024smaug} and Simple Preference Optimization (SimPO)~\cite{meng2024simpo}, do not rely on an extra RM. These approaches offer a more resource-efficient alternative by optimizing the policy directly from preferences, therefore attracting much attention from the academic community, where computational resources are often limited.&#10;&#10;In this work, we begin our analysis by using the vanilla DPO as a case study, subsequently extending our findings to encompass broader RM-free alignment strategies. Despite its simplicity and promise, DPO has exhibited several perplexing phenomena that remain unclear or underexplained in practice. One notable counter-intuitive observation is that the likelihood of both preferred and rejected responses tends to decrease over the course of DPO training~\cite{yuan2024advancing, pytheia_MM}, while the likelihood of certain tokens diverging from the training data increases~\cite{xu2024dpo}. Additional observations are summarized in Section~\ref{subsec:Unexplored Facts about DPO and its Variants}. Without a deeper theoretical exploration of these phenomena, purely empirical efforts to apply or improve DPO are likely to face inefficiencies.&#10;&#10;Our work identifies the issues surrounding vanilla DPO and its variants from both theoretical and practical perspectives. The analysis reveals inherent instability in the DPO training process, which we encapsulate as the 3D-properties: Drastic drop in the likelihood of rejected responses, Degradation into response suppression, and Dispersion effect on unseen responses. Through our analytical framework, we show that these phenomena stem from the inherent features of DPO's optimization objective, where the interaction between the gradients of chosen and rejected responses leads to instability and hinders overall performance. Furthermore, our findings confirm that the distribution of preference data critically influences DPO’s effectiveness, with on-policy DPO performing better than off-policy DPO, which is consistent with concurrent empirical studies~\cite{tang2024understanding, guo2024direct}. &#10;&#10;To enhance DPO's stability and performance, we propose several regularization methods, including the adaptive adjustment of weights on the gradients of chosen and rejected responses, as well as incorporating an SFT loss into the objective. Our results suggest a fundamental trade-off within the DPO algorithm: balancing the mitigation of the 3D-properties while preventing LLMs from straying too far from the preference learning paradigm. Additionally, we compare DPO with the state-of-the-art RM-based method, RLHF-PPO, revealing that its superiority stem largely from avoiding the 3D-properties. Our experimental approach begins with the design of a toy model to quickly validate our hypotheses, followed by a rigorous test of the actual performance of real LLMs on tasks such as mathematical problem solving and instruction following. &#10;&#10;As this topic has garnered significant attention recently, an increasing number of works are contributing to the discussion. To highlight the contributions of our approach, we compare our findings with several of the most relevant concurrent studies in Section~\ref{subsec:comparison with related contemporary studies}. A comprehensive review of related works is provided in Appendix~\ref{supple:sec:related work}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Pathologies, Machine Learning, Computer Science, Preference Learning Methods, Artificial Intelligence, Mathematics, Natural Language Processing, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="1907.01752" label="1907.01752">
        <attvalues>
          <attvalue for="0" value="On the Weaknesses of Reinforcement Learning for Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Reinforcement learning (RL) is frequently used to increase performance in&#10;text generation tasks, including machine translation (MT), notably through the&#10;use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN).&#10;However, little is known about what and how these methods learn in the context&#10;of MT. We prove that one of the most common RL methods for MT does not optimize&#10;the expected reward, as well as show that other methods take an infeasibly long&#10;time to converge. In fact, our results suggest that RL practices in MT are&#10;likely to improve performance only where the pre-trained parameters are already&#10;close to yielding the correct translation. Our findings further suggest that&#10;observed gains may be due to effects unrelated to the training signal, but&#10;rather from changes in the shape of the distribution curve.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10425" label="2305.10425">
        <attvalues>
          <attvalue for="0" value="SLiC-HF: Sequence Likelihood Calibration with Human Feedback" />
          <attvalue for="1" value="  Learning from human feedback has been shown to be effective at aligning&#10;language models with human preferences. Past work has often relied on&#10;Reinforcement Learning from Human Feedback (RLHF), which optimizes the language&#10;model using reward scores assigned from a reward model trained on human&#10;preference data. In this work we show how the recently introduced Sequence&#10;Likelihood Calibration (SLiC), can also be used to effectively learn from human&#10;preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human&#10;feedback data collected for a different model, similar to off-policy, offline&#10;RL data. Automatic and human evaluation experiments on the TL;DR summarization&#10;task show that SLiC-HF significantly improves supervised fine-tuning baselines.&#10;Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF&#10;implementation used in past work while being much simpler to implement, easier&#10;to tune and more computationally efficient in practice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;RL has been used to optimize arbitrary reward in language generation such as BLEU for translation \cite{wu2016google} and ROUGE for summarization \cite{paulus2017deep}; however, while those metrics improved, human judgement of quality suffered due to metrics misalignment.&#10;&#10;In an effort to better align the reward function with human judgement, many works used RL to align language models with a reward model trained to predict carefully collected human judgements \cite{ziegler2020finetuning, openai_sum_hf, instructgpt} using summarization as an initial proof-of-concept.&#10;A KL penalty term, first used in \cite{jaques2017sequence}, is used as regularization to prevent the tuned model from departing from the initial supervised model, and is also used in SLiC \cite{slic}. &#10;&#10;\cite{brio} propose BRIO, which has a similar intent as SLiC \cite{slic} of rank-ordering model-generated decodes according to a reward function.&#10;BRIO trains models to align length normalized sequence probability of generated decodes to their similarity to reference as measured by ROUGE using a list-wise loss function.&#10;In contrast, and similar to RLHF, SLiC-HF adapts the technique to align with a model trained to predict human preference given two summaries instead of their similarity to the reference.&#10;&#10;\cite{bai2022constitutional} substitutes human preference data with judgements from a large language model, and calls it AI feedback (AIF).&#10;SLIC-HF can also be used with AIF exactly in the same way and is indifferent about the AI or human origin of the feedback.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.13228" label="2402.13228">
        <attvalues>
          <attvalue for="0" value="Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive" />
          <attvalue for="1" value="  Direct Preference Optimisation (DPO) is effective at significantly improving&#10;the performance of large language models (LLMs) on downstream tasks such as&#10;reasoning, summarisation, and alignment. Using pairs of preferred and&#10;dispreferred data, DPO models the relative probability of picking one response&#10;over another. In this work, first we show theoretically that the standard DPO&#10;loss can lead to a reduction of the model's likelihood of the preferred&#10;examples, as long as the relative probability between the preferred and&#10;dispreferred classes increases. We then show empirically that this phenomenon&#10;occurs when fine-tuning LLMs on common datasets, especially datasets in which&#10;the edit distance between pairs of completions is low. Using these insights, we&#10;design DPO-Positive (DPOP), a new loss function and training procedure which&#10;avoids this failure mode. Surprisingly, we find that DPOP outperforms DPO and&#10;other fine-tuning procedures across a wide variety of datasets and downstream&#10;tasks, including datasets with high edit distances between completions.&#10;Furthermore, we find that the DPOP-tuned model outperforms the DPO-tuned model&#10;(all else equal) on benchmarks independent of the fine-tuning data, such as&#10;MT-Bench. Finally, using DPOP, we create and open-source Smaug-34B and&#10;Smaug-72B, with the latter becoming the first open-source LLM to surpass an&#10;average accuracy of 80% on the HuggingFace Open LLM Leaderboard.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.14734" label="2405.14734">
        <attvalues>
          <attvalue for="0" value="SimPO: Simple Preference Optimization with a Reference-Free Reward" />
          <attvalue for="1" value="  Direct Preference Optimization (DPO) is a widely used offline preference&#10;optimization algorithm that reparameterizes reward functions in reinforcement&#10;learning from human feedback (RLHF) to enhance simplicity and training&#10;stability. In this work, we propose SimPO, a simpler yet more effective&#10;approach. The effectiveness of SimPO is attributed to a key design: using the&#10;average log probability of a sequence as the implicit reward. This reward&#10;formulation better aligns with model generation and eliminates the need for a&#10;reference model, making it more compute and memory efficient. Additionally, we&#10;introduce a target reward margin to the Bradley-Terry objective to encourage a&#10;larger margin between the winning and losing responses, further enhancing the&#10;algorithm's performance. We compare SimPO to DPO and its latest variants across&#10;various state-of-the-art training setups, including both base and&#10;instruction-tuned models like Mistral and Llama3. We evaluated on extensive&#10;instruction-following benchmarks, including AlpacaEval 2, MT-Bench, and the&#10;recent challenging Arena-Hard benchmark. Our results demonstrate that SimPO&#10;consistently and significantly outperforms existing approaches without&#10;substantially increasing response length. Specifically, SimPO outperforms DPO&#10;by up to 6.4 points on AlpacaEval 2 and by up to 7.5 points on Arena-Hard. Our&#10;top-performing model, built on Llama3-8B-Instruct, achieves a remarkable 53.7&#10;length-controlled win rate on AlpacaEval 2 -- surpassing Claude 3 Opus on the&#10;leaderboard, and a 36.5 win rate on Arena-Hard -- making it the strongest 8B&#10;open-source model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.02078" label="2404.02078">
        <attvalues>
          <attvalue for="0" value="Advancing LLM Reasoning Generalists with Preference Trees" />
          <attvalue for="1" value="  We introduce Eurus, a suite of large language models (LLMs) optimized for&#10;reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve&#10;state-of-the-art results among open-source models on a diverse set of&#10;benchmarks covering mathematics, code generation, and logical reasoning&#10;problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a&#10;comprehensive benchmarking across 12 tests covering five tasks, and achieves a&#10;33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging&#10;benchmarks, substantially outperforming existing open-source models by margins&#10;more than 13.3%. The strong performance of Eurus can be primarily attributed to&#10;UltraInteract, our newly-curated large-scale, high-quality alignment dataset&#10;specifically designed for complex reasoning tasks. UltraInteract can be used in&#10;both supervised fine-tuning and preference learning. For each instruction, it&#10;includes a preference tree consisting of (1) reasoning chains with diverse&#10;planning strategies in a unified format, (2) multi-turn interaction&#10;trajectories with the environment and the critique, and (3) pairwise data to&#10;facilitate preference learning. UltraInteract allows us to conduct an in-depth&#10;exploration of preference learning for reasoning tasks. Our investigation&#10;reveals that some well-established preference learning algorithms may be less&#10;suitable for reasoning tasks compared to their effectiveness in general&#10;conversations. Inspired by this, we derive a novel reward modeling objective&#10;which, together with UltraInteract, leads to a strong reward model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.08448" label="2405.08448">
        <attvalues>
          <attvalue for="0" value="Understanding the performance gap between online and offline alignment&#10;  algorithms" />
          <attvalue for="1" value="  Reinforcement learning from human feedback (RLHF) is the canonical framework&#10;for large language model alignment. However, rising popularity in offline&#10;alignment algorithms challenge the need for on-policy sampling in RLHF. Within&#10;the context of reward over-optimization, we start with an opening set of&#10;experiments that demonstrate the clear advantage of online methods over offline&#10;methods. This prompts us to investigate the causes to the performance&#10;discrepancy through a series of carefully designed experimental ablations. We&#10;show empirically that hypotheses such as offline data coverage and data quality&#10;by itself cannot convincingly explain the performance difference. We also find&#10;that while offline algorithms train policy to become good at pairwise&#10;classification, it is worse at generations; in the meantime the policies&#10;trained by online algorithms are good at generations while worse at pairwise&#10;classification. This hints at a unique interplay between discriminative and&#10;generative capabilities, which is greatly impacted by the sampling process.&#10;Lastly, we observe that the performance discrepancy persists for both&#10;contrastive and non-contrastive loss functions, and appears not to be addressed&#10;by simply scaling up policy networks. Taken together, our study sheds light on&#10;the pivotal role of on-policy sampling in AI alignment, and hints at certain&#10;fundamental challenges of offline alignment algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08675" label="2104.08675">
        <attvalues>
          <attvalue for="0" value="Dual-View Distilled BERT for Sentence Embedding" />
          <attvalue for="1" value="  Recently, BERT realized significant progress for sentence matching via&#10;word-level cross sentence attention. However, the performance significantly&#10;drops when using siamese BERT-networks to derive two sentence embeddings, which&#10;fall short in capturing the global semantic since the word-level attention&#10;between two sentences is absent. In this paper, we propose a Dual-view&#10;distilled BERT~(DvBERT) for sentence matching with sentence embeddings. Our&#10;method deals with a sentence pair from two distinct views, i.e., Siamese View&#10;and Interaction View. Siamese View is the backbone where we generate sentence&#10;embeddings. Interaction View integrates the cross sentence interaction as&#10;multiple teachers to boost the representation ability of sentence embeddings.&#10;Experiments on six STS tasks show that our method outperforms the&#10;state-of-the-art sentence embedding methods significantly.&#10;" />
          <attvalue for="2" value="&#10;Recent sentence representation models like BERT~\cite{DBLP:conf/naacl/DevlinCLT19} achieved state-of-the-art results on sentence-pair regression/classification tasks, such as question answering, natural language inference~(NLI)~\cite{DBLP:conf/emnlp/BowmanAPM15,DBLP:conf/naacl/WilliamsNB18}, and semantic textual similarity~(STS)~\cite{DBLP:conf/semeval/AgirreCDG12,agirre-etal-2013-sem,agirre-etal-2016-semeval,agirre-etal-2014-semeval,agirre-etal-2015-semeval}. &#10;However, it has a low computational efficiency when candidate sentence-pairs are not given ahead, leading to a massive computational overhead. &#10;For example, seeking the most relevant sentence-pair of a collection requires pairing all sentences.&#10;The $O(n^2)$ computational complexity is an obstacle preventing many te{retrieval} applications from adopting the technology.&#10;&#10;A standard method to reduce the computations is separately encoding each sentence into a vector representation and then compare any two of them by similarity distance.&#10;However, in contrast to the standard BERT model, the performance of sentence matching is constrained.&#10;For instance, SBERT~\cite{DBLP:conf/emnlp/ReimersG19} using the siamese BERT-networks that decreased the performance by 3-4 points evaluated by Spearman correlation~\cite{myers2004spearman} on STS-Benchmark~\cite{cer-etal-2017-semeval}, which implies room for improvement.&#10;We argue that the siamese BERT-networks are limited to capture the full complexity of global semantic matching, neglecting the word-level interaction features across two sentences.&#10;The feature has been proved vital for predicting matching degrees~\cite{DBLP:conf/coling/Lan018,xu2020symmetric}.&#10;&#10;Motivated by these observations, we propose a Dual-view distilled BERT~(DvBERT) by incorporating the word-level interaction features into sentence embeddings while maintains the same efficiency as siamese BERT-networks. &#10;We take inspiration from Multi-view learning~\cite{DBLP:journals/corr/abs-1304-5634,DBLP:conf/emnlp/ClarkLML18} and train the sentence matching model from two views:&#10;(1) Siamese View,&#10;we start with the siamese BERT-networks as a backbone to derive sentence embeddings, to be able to capture semantics similarity efficiently by calculating distances on the two fixed-size vectors.&#10;(2) Interaction View,&#10;the standard pre-trained models with cross-sentence interactions are utilized, acting as multiple teachers that generate predictions about the training set provided to the siamese networks to learn. &#10;The association between the two views acts as a regularization term that trains a student with soft targets from the multiple teacher's output distributions, making the procedure similar to knowledge distillation~\cite{DBLP:journals/corr/HintonVD15}.&#10;In contrast of other distilled versions of BERT~\cite{sanh2019distilbert,DBLP:conf/emnlp/SunCGL19}, our method aims to optimize sentence embedding representations with two heterogeneous networks, together with multi-task knowledge distillation~\cite{DBLP:journals/corr/abs-1904-09482}, neither distilling large models into a small model~\cite{DBLP:journals/corr/abs-1904-00796, DBLP:journals/corr/abs-1903-04190} nor born-again networks~\cite{DBLP:conf/icml/FurlanelloLTIA18,DBLP:conf/acl/ClarkLKML19}.&#10;Besides, we compared the loss weighting and teacher annealing strategy~\cite{DBLP:conf/acl/ClarkLKML19} during the distillation process, suggesting that the latter was more efficient.&#10;Experiments demonstrate that DvBERT can achieve superior performance than siamese BERT-networks on six STS datasets.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, BERT Networks, Artificial Intelligence, Information Science, Sentence Matching, Semantic Embeddings" />
        </attvalues>
      </node>
      <node id="1806.04330" label="1806.04330">
        <attvalues>
          <attvalue for="0" value="Neural Network Models for Paraphrase Identification, Semantic Textual&#10;  Similarity, Natural Language Inference, and Question Answering" />
          <attvalue for="1" value="  In this paper, we analyze several neural network designs (and their&#10;variations) for sentence pair modeling and compare their performance&#10;extensively across eight datasets, including paraphrase identification,&#10;semantic textual similarity, natural language inference, and question answering&#10;tasks. Although most of these models have claimed state-of-the-art performance,&#10;the original papers often reported on only one or two selected datasets. We&#10;provide a systematic study and show that (i) encoding contextual information by&#10;LSTM and inter-sentence interactions are critical, (ii) Tree-LSTM does not help&#10;as much as previously claimed but surprisingly improves performance on Twitter&#10;datasets, (iii) the Enhanced Sequential Inference Model is the best so far for&#10;larger datasets, while the Pairwise Word Interaction Model achieves the best&#10;performance when less data is available. We release our implementations as an&#10;open-source toolkit.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.03405" label="1909.03405">
        <attvalues>
          <attvalue for="0" value="Symmetric Regularization based BERT for Pair-wise Semantic Reasoning" />
          <attvalue for="1" value="  The ability of semantic reasoning over the sentence pair is essential for&#10;many natural language understanding tasks, e.g., natural language inference and&#10;machine reading comprehension. A recent significant improvement in these tasks&#10;comes from BERT. As reported, the next sentence prediction (NSP) in BERT, which&#10;learns the contextual relationship between two sentences, is of great&#10;significance for downstream problems with sentence-pair input. Despite the&#10;effectiveness of NSP, we suggest that NSP still lacks the essential signal to&#10;distinguish between entailment and shallow correlation. To remedy this, we&#10;propose to augment the NSP task to a 3-class categorization task, which&#10;includes a category for previous sentence prediction (PSP). The involvement of&#10;PSP encourages the model to focus on the informative semantics to determine the&#10;sentence order, thereby improves the ability of semantic understanding. This&#10;simple modification yields remarkable improvement against vanilla BERT. To&#10;further incorporate the document-level information, the scope of NSP and PSP is&#10;expanded into a broader range, i.e., NSP and PSP also include close but&#10;nonsuccessive sentences, the noise of which is mitigated by the label-smoothing&#10;technique. Both qualitative and quantitative experimental results demonstrate&#10;the effectiveness of the proposed method. Our method consistently improves the&#10;performance on the NLI and MRC benchmarks, including the challenging HANS&#10;dataset \cite{hans}, suggesting that the document-level task is still promising&#10;for the pre-training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1304.5634" label="1304.5634">
        <attvalues>
          <attvalue for="0" value="A Survey on Multi-view Learning" />
          <attvalue for="1" value="  In recent years, a great many methods of learning from multi-view data by&#10;considering the diversity of different views have been proposed. These views&#10;may be obtained from multiple sources or different feature subsets. In trying&#10;to organize and highlight similarities and differences between the variety of&#10;multi-view learning approaches, we review a number of representative multi-view&#10;learning algorithms in different areas and classify them into three groups: 1)&#10;co-training, 2) multiple kernel learning, and 3) subspace learning. Notably,&#10;co-training style algorithms train alternately to maximize the mutual agreement&#10;on two distinct views of the data; multiple kernel learning algorithms exploit&#10;kernels that naturally correspond to different views and combine kernels either&#10;linearly or non-linearly to improve learning performance; and subspace learning&#10;algorithms aim to obtain a latent subspace shared by multiple views by assuming&#10;that the input views are generated from this latent subspace. Though there is&#10;significant variance in the approaches to integrating multiple views to improve&#10;learning performance, they mainly exploit either the consensus principle or the&#10;complementary principle to ensure the success of multi-view learning. Since&#10;accessing multiple views is the fundament of multi-view learning, with the&#10;exception of study on learning a model from multiple views, it is also valuable&#10;to study how to construct multiple views and how to evaluate these views.&#10;Overall, by exploring the consistency and complementary properties of different&#10;views, multi-view learning is rendered more effective, more promising, and has&#10;better generalization ability than single-view learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.09482" label="1904.09482">
        <attvalues>
          <attvalue for="0" value="Improving Multi-Task Deep Neural Networks via Knowledge Distillation for&#10;  Natural Language Understanding" />
          <attvalue for="1" value="  This paper explores the use of knowledge distillation to improve a Multi-Task&#10;Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text&#10;representations across multiple natural language understanding tasks. Although&#10;ensemble learning can improve model performance, serving an ensemble of large&#10;DNNs such as MT-DNN can be prohibitively expensive. Here we apply the knowledge&#10;distillation method (Hinton et al., 2015) in the multi-task learning setting.&#10;For each task, we train an ensemble of different MT-DNNs (teacher) that&#10;outperforms any single model, and then train a single MT-DNN (student) via&#10;multi-task learning to \emph{distill} knowledge from these ensemble teachers.&#10;We show that the distilled MT-DNN significantly outperforms the original MT-DNN&#10;on 7 out of 9 GLUE tasks, pushing the GLUE benchmark (single model) to 83.7\%&#10;(1.5\% absolute improvement\footnote{ Based on the GLUE leaderboard at&#10;https://gluebenchmark.com/leaderboard as of April 1, 2019.}). The code and&#10;pre-trained models will be made publicly available at&#10;https://github.com/namisan/mt-dnn.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.00796" label="1904.00796">
        <attvalues>
          <attvalue for="0" value="Making Neural Machine Reading Comprehension Faster" />
          <attvalue for="1" value="  This study aims at solving the Machine Reading Comprehension problem where&#10;questions have to be answered given a context passage. The challenge is to&#10;develop a computationally faster model which will have improved inference time.&#10;State of the art in many natural language understanding tasks, BERT model, has&#10;been used and knowledge distillation method has been applied to train two&#10;smaller models. The developed models are compared with other models which have&#10;been developed with the same intention.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.04829" label="1907.04829">
        <attvalues>
          <attvalue for="0" value="BAM! Born-Again Multi-Task Networks for Natural Language Understanding" />
          <attvalue for="1" value="  It can be challenging to train multi-task neural networks that outperform or&#10;even match their single-task counterparts. To help address this, we propose&#10;using knowledge distillation where single-task models teach a multi-task model.&#10;We enhance this training with teacher annealing, a novel method that gradually&#10;transitions the model from distillation to supervised learning, helping the&#10;multi-task model surpass its single-task teachers. We evaluate our approach by&#10;multi-task fine-tuning BERT on the GLUE benchmark. Our method consistently&#10;improves over standard single-task and multi-task training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.13674" label="2205.13674">
        <attvalues>
          <attvalue for="0" value="Global Normalization for Streaming Speech Recognition in a Modular&#10;  Framework" />
          <attvalue for="1" value="  We introduce the Globally Normalized Autoregressive Transducer (GNAT) for&#10;addressing the label bias problem in streaming speech recognition. Our solution&#10;admits a tractable exact computation of the denominator for the sequence-level&#10;normalization. Through theoretical and empirical results, we demonstrate that&#10;by switching to a globally normalized model, the word error rate gap between&#10;streaming and non-streaming speech-recognition models can be greatly reduced&#10;(by more than 50\% on the Librispeech dataset). This model is developed in a&#10;modular framework which encompasses all the common neural speech recognition&#10;models. The modularity of this framework enables controlled comparison of&#10;modelling choices and creation of new models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Deep neural network models have been tremendously successful in the field of automatic&#10;speech recognition (ASR). Several different models have been proposed over the years:&#10;cross-entropy (CE) models with a deep feed-forward architecture \cite{hinton2012deep},&#10;connectionist temporal classification (CTC) models \cite{graves2006connectionist} with recurrent&#10;architectures such as long short-term memory (LSTM) \cite{hochreiter1997long}, and more recently&#10;sequence-to-sequence (Seq2Seq) models like listen, attend and spell (LAS) \cite{chan2015listen},&#10;recurrent neural network transducer (RNN-T) \cite{graves2012sequence}, and&#10;hybrid autoregressive transducer (HAT) \cite{variani2020hybrid}.&#10;When configured in non-streaming mode, these neural ASR models have reached state-of-the-art&#10;word error rate (WER) on many tasks. However, the WER significantly drops when they are operating&#10;in streaming mode. In this paper, we argue that one main cause of such WER gap is that&#10;all the existing models are constrained to be locally normalized&#10;which makes them susceptible to label bias problem \cite{smith2007weighted, andor2016globally, lafferty2001conditional, bottou1991approche}.&#10;To address this problem, we introduce new category of globally normalized models called&#10;Globally Normalized Autoregressive Transducer (GNAT). Our contributions are:&#10;\begin{compactenum}[(1)]&#10; \item {Addressing the label bias problem in streaming ASR through global normalization} that significantly closes more than 50\% of the WER gap between streaming and non-streaming ASR.&#10; \item {Efficient, accelerator-friendly algorithms for the exact computation of the global normalization} under the finite context assumption.&#10; \item {A modular framework for neural ASR} which encompasses all the common models (CE, CTC, LAS, RNN-T, HAT), allowing creation of new ones, and extension to their globally normalized counterparts.&#10;\end{compactenum}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Networks, Signal Processing, Mathematics, Speech Recognition, Model Optimization" />
        </attvalues>
      </node>
      <node id="1508.01211" label="1508.01211">
        <attvalues>
          <attvalue for="0" value="Listen, Attend and Spell" />
          <attvalue for="1" value="  We present Listen, Attend and Spell (LAS), a neural network that learns to&#10;transcribe speech utterances to characters. Unlike traditional DNN-HMM models,&#10;this model learns all the components of a speech recognizer jointly. Our system&#10;has two components: a listener and a speller. The listener is a pyramidal&#10;recurrent network encoder that accepts filter bank spectra as inputs. The&#10;speller is an attention-based recurrent network decoder that emits characters&#10;as outputs. The network produces character sequences without making any&#10;independence assumptions between the characters. This is the key improvement of&#10;LAS over previous end-to-end CTC models. On a subset of the Google voice search&#10;task, LAS achieves a word error rate (WER) of 14.1% without a dictionary or a&#10;language model, and 10.3% with language model rescoring over the top 32 beams.&#10;By comparison, the state-of-the-art CLDNN-HMM model achieves a WER of 8.0%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.08218" label="2006.08218">
        <attvalues>
          <attvalue for="0" value="Self-supervised Learning: Generative or Contrastive" />
          <attvalue for="1" value="  Deep supervised learning has achieved great success in the last decade.&#10;However, its deficiencies of dependence on manual labels and vulnerability to&#10;attacks have driven people to explore a better solution. As an alternative,&#10;self-supervised learning attracts many researchers for its soaring performance&#10;on representation learning in the last several years. Self-supervised&#10;representation learning leverages input data itself as supervision and benefits&#10;almost all types of downstream tasks. In this survey, we take a look into new&#10;self-supervised learning methods for representation in computer vision, natural&#10;language processing, and graph learning. We comprehensively review the existing&#10;empirical methods and summarize them into three main categories according to&#10;their objectives: generative, contrastive, and generative-contrastive&#10;(adversarial). We further investigate related theoretical analysis work to&#10;provide deeper thoughts on how self-supervised learning works. Finally, we&#10;briefly discuss open problems and future directions for self-supervised&#10;learning. An outline slide for the survey is provided.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.05777" label="2110.05777">
        <attvalues>
          <attvalue for="0" value="Large-scale Self-Supervised Speech Representation Learning for Automatic&#10;  Speaker Verification" />
          <attvalue for="1" value="  The speech representations learned from large-scale unlabeled data have shown&#10;better generalizability than those from supervised learning and thus attract a&#10;lot of interest to be applied for various downstream tasks. In this paper, we&#10;explore the limits of speech representations learned by different&#10;self-supervised objectives and datasets for automatic speaker verification&#10;(ASV), especially with a well-recognized SOTA ASV model, ECAPA-TDNN [1], as a&#10;downstream model. The representations from all hidden layers of the pre-trained&#10;model are firstly averaged with learnable weights and then fed into the&#10;ECAPA-TDNN as input features. The experimental results on Voxceleb dataset show&#10;that the weighted average representation is significantly superior to FBank, a&#10;conventional handcrafted feature for ASV. Our best single system achieves&#10;0.537%, 0.569%, and 1.180% equal error rate (EER) on the three official trials&#10;of VoxCeleb1, separately. Accordingly, the ensemble system with three&#10;pre-trained models can further improve the EER to 0.479%, 0.536% and 1.023%.&#10;Among the three evaluation trials, our best system outperforms the winner&#10;system [2] of the VoxCeleb Speaker Recognition Challenge 2021 (VoxSRC2021) on&#10;the VoxCeleb1-E trial.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.00540" label="2204.00540">
        <attvalues>
          <attvalue for="0" value="End-to-End Integration of Speech Recognition, Speech Enhancement, and&#10;  Self-Supervised Learning Representation" />
          <attvalue for="1" value="  This work presents our end-to-end (E2E) automatic speech recognition (ASR)&#10;model targetting at robust speech recognition, called Integraded speech&#10;Recognition with enhanced speech Input for Self-supervised learning&#10;representation (IRIS). Compared with conventional E2E ASR models, the proposed&#10;E2E model integrates two important modules including a speech enhancement (SE)&#10;module and a self-supervised learning representation (SSLR) module. The SE&#10;module enhances the noisy speech. Then the SSLR module extracts features from&#10;enhanced speech to be used for speech recognition (ASR). To train the proposed&#10;model, we establish an efficient learning scheme. Evaluation results on the&#10;monaural CHiME-4 task show that the IRIS model achieves the best performance&#10;reported in the literature for the single-channel CHiME-4 benchmark (2.0% for&#10;the real development and 3.9% for the real test) thanks to the powerful&#10;pre-trained SSLR module and the fine-tuned SE module.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.07272" label="2005.07272">
        <attvalues>
          <attvalue for="0" value="Target-Speaker Voice Activity Detection: a Novel Approach for&#10;  Multi-Speaker Diarization in a Dinner Party Scenario" />
          <attvalue for="1" value="  Speaker diarization for real-life scenarios is an extremely challenging&#10;problem. Widely used clustering-based diarization approaches perform rather&#10;poorly in such conditions, mainly due to the limited ability to handle&#10;overlapping speech. We propose a novel Target-Speaker Voice Activity Detection&#10;(TS-VAD) approach, which directly predicts an activity of each speaker on each&#10;time frame. TS-VAD model takes conventional speech features (e.g., MFCC) along&#10;with i-vectors for each speaker as inputs. A set of binary classification&#10;output layers produces activities of each speaker. I-vectors can be estimated&#10;iteratively, starting with a strong clustering-based diarization. We also&#10;extend the TS-VAD approach to the multi-microphone case using a simple&#10;attention mechanism on top of hidden representations extracted from the&#10;single-channel TS-VAD model. Moreover, post-processing strategies for the&#10;predicted speaker activity probabilities are investigated. Experiments on the&#10;CHiME-6 unsegmented data show that TS-VAD achieves state-of-the-art results&#10;outperforming the baseline x-vector-based system by more than 30% Diarization&#10;Error Rate (DER) abs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.19334" label="2405.19334">
        <attvalues>
          <attvalue for="0" value="LLMs Meet Multimodal Generation and Editing: A Survey" />
          <attvalue for="1" value="  With the recent advancement in large language models (LLMs), there is a&#10;growing interest in combining LLMs with multimodal learning. Previous surveys&#10;of multimodal large language models (MLLMs) mainly focus on multimodal&#10;understanding. This survey elaborates on multimodal generation and editing&#10;across various domains, comprising image, video, 3D, and audio. Specifically,&#10;we summarize the notable advancements with milestone works in these fields and&#10;categorize these studies into LLM-based and CLIP/T5-based methods. Then, we&#10;summarize the various roles of LLMs in multimodal generation and exhaustively&#10;investigate the critical technical components behind these methods and the&#10;multimodal datasets utilized in these studies. Additionally, we dig into&#10;tool-augmented multimodal agents that can leverage existing generative models&#10;for human-computer interaction. Lastly, we discuss the advancements in the&#10;generative AI safety field, investigate emerging applications, and discuss&#10;future prospects. Our work provides a systematic and insightful overview of&#10;multimodal generation and processing, which is expected to advance the&#10;development of Artificial Intelligence for Generative Content (AIGC) and world&#10;models. A curated list of all related papers can be found at&#10;https://github.com/YingqingHe/Awesome-LLMs-meet-Multimodal-Generation&#10;" />
          <attvalue for="2" value="&#10;The interactions between human beings and the physical world involve information from multiple modalities, such as language, vision, and audio. &#10;Therefore, realizing a world simulator also requires the model to be able to perceive and respond with multimodal information in a flexible manner.&#10;Recently, OpenAI introduced a foundation text-to-video generation model termed Sora~\cite{sora} that is capable of generating highly realistic videos as world simulators.&#10;It makes great progress in simulating or generating real-world scenes but is unable to generate other modalities, such as text, 3D, and audio. &#10;Also, it lacks the ability to perceive other modalities such as image, video, 3D, and audio, making it an incomprehension world simulator.&#10;&#10;In the past few years, researchers focused on the generation of each single modality and have achieved great progress:&#10;In terms of text generation, we have witnessed a qualitative leap in the performance of natural language processing tasks:&#10;From BERT~\cite{devlin2018bert}, GPT1~\cite{2018gpt1}, GPT2~\cite{2019gpt2}, GPT3~\cite{gpt3}, GPT4~\cite{gpt4} to ChatGPT~\cite{chatgpt}, LLaMA~\cite{li2023llama,gao2023llama}, the number of model parameters and training samples has grown rapidly, resulting in the continual growth of modal abilities and product deployment.&#10;In the visual generation field, with the rapid progress of diffusion models and large-scale image-text datasets, text-to-image (T2I) generation has achieved remarkable achievement and can synthesize high-quality images based on various user-provided text prompts, such as SDXL~\cite{sdxl} and PIXART-$\alpha$~\cite{chen2023pixart}.&#10;Subsequently, significant advancements have been made in the field of text-to-video generation through the utilization of video diffusion models~\cite{vdm} and large-scale video-language datasets~\cite{Webvid}. &#10;Notably, several milestone works have emerged, such as~\cite{he2022lvdm, zhou2022magicvideo, singer2022make-a-video, ho2022imagen-video, villegas2022phenaki, chen2023videocrafter1, guo2023animatediff, bar2024lumiere, girdhar2023emuvideo, videocrafter2} and Sora~\cite{sora}.&#10;For the 3D generation, with the emergence of the CLIP~\cite{radford2021learning} model, some methods~\cite{sanghi2022clip,mohammad2022clip,michel2022text2mesh} try to align the text information to the rendered images from 3D representations, {i.e.}, mesh, point cloud, NeRF~\cite{wang2023nerf} and gaussian splatting~\cite{kerbl3Dgaussians}). &#10;These approaches have led to significant developments in text-to-3D generation. &#10;Additionally, the integration of Stable Diffusion (SD)~\cite{ldm} with text-to-image rendering has enabled a series of works in text-to-3D generation~\cite{yi2023gaussiandreamer, tang2023dreamgaussian, hoellein2023text2room, liang2023_luciddreamer, yu2023_csd, li2023_sweetdreamer, wang2023prolificdreamer, lorraine2023_att3d, xu2023_dream3d, zhu2023_hifa, Chen_2023_ICCV, tsalicoglou2023textmesh, poole2022dreamfusion, lin2023magic3d,seo2023let}. &#10;The powerful text-to-image model helps the 3D generation achieve higher performance and better visual results. &#10;In the area of text-to-audio generation, a series of representative works tackle different audio domains &#10;such as~\cite{liu2023audioldm, liu2023audioldm2, kreuk2022audiogen} for text-to-audio,~\cite{agostinelli2023musiclm, copet2024musicgen, forsgren2022riffusion} for text-to-music, and~\cite{tan2024naturalspeech, shen2023naturalspeech2, ju2024naturalspeech3, wang2023valle, jiang2023megatts2, ren2020fastspeech2} for text-to-speech, and they have achieved significant performance in generating high-quality natural sounds, music, and human-level speech.&#10;&#10;With the notable progress and significant performance improvements of Large Language Models (LLMs), other non-text modalities have started harnessing the power of LLMs to either enhance their generation quality or integrate multiple modalities in a unified system to achieve more powerful functionalities.&#10;In the context of image generation, the integration of LLMs can be divided into two categories.&#10;The first category involves encoding visual information into discrete token indices, trying to unify visual understanding and generation~\cite{ge2023planting, zeqiang2023mini, tang2023codi, ge2023making, sun2023emu2, zhao2023making}. &#10;Specifically, visual information is encoded into token representations, and LLMs directly comprehend and generate visual tokens, enabling simultaneous visual understanding and generation.&#10;The second category focuses on leveraging LLMs to enhance the generation quality of existing pretrained T2I models: &#10;One type of approach utilizes an LLM as a layout planner to incorporate knowledge of object spatial positions, quantity, and object size, enabling the generation of required bounding boxes~\cite{chen2023textdiffuser2, lian2023llm, feng2023layoutgpt, zhang2023controllable, qu2023layoutllm}.&#10;After obtaining the bounding boxes, the images can be generated through a grounded T2I model such as GLIGEN~\cite{li2023gligen}.&#10;Another approach utilizes LLMs to expand input user prompts~\cite{betker2023improving}: &#10;By providing highly detailed and comprehensive prompts, LLMs generate images with high quality and richness.&#10;With the assistance of LLMs, image generation has achieved higher generation quality, improved prompt following capabilities, dialogic function, and user-friendly interface.&#10;&#10;Similar to image domain, in video generation, LLMs serve as the general backbone for unified multimodal joint generation~\cite{kondratyuk2023videopoet, yu2023language}, video layout planning~\cite{fei2023empowering, lin2023videodirectorgpt, lian2023llm, lv2023gpt4motion, lu2023flowzero} and temporal prompt generation~\cite{hong2023large, huang2023free, wang2023intercontrol, liu2023plan, long2024videodrafter} for temporal dynamics guidance. &#10;For 3D generation and editing, LLMs serve as a bridge between users and 3D assets, which improves interaction efficiency~\cite{sun20233d,feng2023posegpt} and helps users understand the 3D assets~\cite{chen2023ll3da,wu2023gpteval3d}.&#10;In the context of audio generation and editing,&#10;the role of LLMs primarily lies in serving as coordinated backbones for multimodal audio~\cite{zhang2023speechgpt, gong2023listen,deshmukh2023pengi, rubenstein2023audiopalm, liu2024music, chen2023lauragpt, gardner2023llark, tang2023salmonn, chu2023qwen-audio, hussain2023m,shu2023llasm,yuan2024chatmusician,ding2024songcomposer}, conditioners for specific tasks~\cite{wu2023music, ghosal2023text, wu2024improving}, labelers for audio understanding~\cite{huang2023make2, wang2023assessing, vyas2023audiobox}, agents for interactive generation and editing~\cite{shen2023hugginggpt, huang2023audiogpt, liu2023wavjourney, yu2023musicagent, zhang2023loop, zhuo2023lyricwhiz}, as well as inspiration for novel approaches~\cite{wang2023valle, agostinelli2023musiclm, dhariwal2020jukebox, copet2024musicgen, borsos2023audiolm, yang2023uniaudio}.&#10;The growing utilization of LLMs in the audio domain is not only transforming our way of engaging with sound and music but also expanding the boundaries at the crossroads of AGI and audio technologies. &#10;Besides, multimodal agents~\cite{shen2023hugginggpt,liu2023controlllm,yang2023gpt4tools,wu2023visual,liu2023internchat,li2023modelscope} integrate lots of AIGC tools into the framework as a universal system, which relies on LLMs to invoke tools but endows LLMs with the ability to comprehend and generate content of non-text modalities.&#10;Generally, LLMs significantly play an indispensable role in generating various modes of content.&#10;&#10;To promote the development of multimodal generation and empower the world simulator, in this work, we provide a comprehensive review of works involving LLMs in the generation of multiple modalities. &#10;As shown in Fig.~\ref{fig:intro}, &#10;we summarize the roles of LLMs into several key aspects, such as evaluator, labeller, instruction processor, planner, provider of semantic guidance, or as backbone architectures. &#10;Additionally, we discuss the development in the generative AI safety topic in Sec.~\ref{sec: safety}, with the emerged applications, and the potential future prospects in Sec.~\ref{sec: applications} and Sec.~\ref{sec: future_work}.&#10;&#10;We summarize our contributions as follows:&#10;\begin{itemize}&#10; \item We present the first systematic review of LLMs applied to the generation and editing of multiple modalities, including images, videos, 3D, and audio. &#10; \item We discuss the evolution of generative techniques through a comparative analysis of pre-LLM and post-LLM eras, offering a clear perspective on the progression and refinement of these approaches. &#10; \item We summarize the various roles of LLMs in the generation or editing process for each modality from a technical view.&#10; \item We discuss important AI safety issues, investigate emerging applications and explore future directions to boost the development of multimodal generation and world models.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence Applications, Artificial Intelligence, Multimodal Generation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2304.15010" label="2304.15010">
        <attvalues>
          <attvalue for="0" value="LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model" />
          <attvalue for="1" value="  How to efficiently transform large language models (LLMs) into instruction&#10;followers is recently a popular research direction, while training LLM for&#10;multi-modal reasoning remains less explored. Although the recent LLaMA-Adapter&#10;demonstrates the potential to handle visual inputs with LLMs, it still cannot&#10;generalize well to open-ended visual instructions and lags behind GPT-4. In&#10;this paper, we present LLaMA-Adapter V2, a parameter-efficient visual&#10;instruction model. Specifically, we first augment LLaMA-Adapter by unlocking&#10;more learnable parameters (e.g., norm, bias and scale), which distribute the&#10;instruction-following ability across the entire LLaMA model besides adapters.&#10;Secondly, we propose an early fusion strategy to feed visual tokens only into&#10;the early LLM layers, contributing to better visual knowledge incorporation.&#10;Thirdly, a joint training paradigm of image-text pairs and&#10;instruction-following data is introduced by optimizing disjoint groups of&#10;learnable parameters. This strategy effectively alleviates the interference&#10;between the two tasks of image-text alignment and instruction following and&#10;achieves strong multi-modal reasoning with only a small-scale image-text and&#10;instruction dataset. During inference, we incorporate additional expert models&#10;(e.g. captioning/OCR systems) into LLaMA-Adapter to further enhance its image&#10;understanding capability without incurring training costs. Compared to the&#10;original LLaMA-Adapter, our LLaMA-Adapter V2 can perform open-ended multi-modal&#10;instructions by merely introducing 14M parameters over LLaMA. The newly&#10;designed framework also exhibits stronger language-only instruction-following&#10;capabilities and even excels in chat interactions. Our code and models are&#10;available at https://github.com/ZrrSkywalker/LLaMA-Adapter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.01952" label="2307.01952">
        <attvalues>
          <attvalue for="0" value="SDXL: Improving Latent Diffusion Models for High-Resolution Image&#10;  Synthesis" />
          <attvalue for="1" value="  We present SDXL, a latent diffusion model for text-to-image synthesis.&#10;Compared to previous versions of Stable Diffusion, SDXL leverages a three times&#10;larger UNet backbone: The increase of model parameters is mainly due to more&#10;attention blocks and a larger cross-attention context as SDXL uses a second&#10;text encoder. We design multiple novel conditioning schemes and train SDXL on&#10;multiple aspect ratios. We also introduce a refinement model which is used to&#10;improve the visual fidelity of samples generated by SDXL using a post-hoc&#10;image-to-image technique. We demonstrate that SDXL shows drastically improved&#10;performance compared the previous versions of Stable Diffusion and achieves&#10;results competitive with those of black-box state-of-the-art image generators.&#10;In the spirit of promoting open research and fostering transparency in large&#10;model training and evaluation, we provide access to code and model weights at&#10;https://github.com/Stability-AI/generative-models&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.00426" label="2310.00426">
        <attvalues>
          <attvalue for="0" value="PixArt-$\alpha$: Fast Training of Diffusion Transformer for&#10;  Photorealistic Text-to-Image Synthesis" />
          <attvalue for="1" value="  The most advanced text-to-image (T2I) models require significant training&#10;costs (e.g., millions of GPU hours), seriously hindering the fundamental&#10;innovation for the AIGC community while increasing CO2 emissions. This paper&#10;introduces PIXART-$\alpha$, a Transformer-based T2I diffusion model whose image&#10;generation quality is competitive with state-of-the-art image generators (e.g.,&#10;Imagen, SDXL, and even Midjourney), reaching near-commercial application&#10;standards. Additionally, it supports high-resolution image synthesis up to&#10;1024px resolution with low training cost, as shown in Figure 1 and 2. To&#10;achieve this goal, three core designs are proposed: (1) Training strategy&#10;decomposition: We devise three distinct training steps that separately optimize&#10;pixel dependency, text-image alignment, and image aesthetic quality; (2)&#10;Efficient T2I Transformer: We incorporate cross-attention modules into&#10;Diffusion Transformer (DiT) to inject text conditions and streamline the&#10;computation-intensive class-condition branch; (3) High-informative data: We&#10;emphasize the significance of concept density in text-image pairs and leverage&#10;a large Vision-Language model to auto-label dense pseudo-captions to assist&#10;text-image alignment learning. As a result, PIXART-$\alpha$'s training speed&#10;markedly surpasses existing large-scale T2I models, e.g., PIXART-$\alpha$ only&#10;takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU&#10;days), saving nearly \$300,000 (\$26,000 vs. \$320,000) and reducing 90% CO2&#10;emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training&#10;cost is merely 1%. Extensive experiments demonstrate that PIXART-$\alpha$&#10;excels in image quality, artistry, and semantic control. We hope&#10;PIXART-$\alpha$ will provide new insights to the AIGC community and startups to&#10;accelerate building their own high-quality yet low-cost generative models from&#10;scratch.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.03458" label="2204.03458">
        <attvalues>
          <attvalue for="0" value="Video Diffusion Models" />
          <attvalue for="1" value="  Generating temporally coherent high fidelity video is an important milestone&#10;in generative modeling research. We make progress towards this milestone by&#10;proposing a diffusion model for video generation that shows very promising&#10;initial results. Our model is a natural extension of the standard image&#10;diffusion architecture, and it enables jointly training from image and video&#10;data, which we find to reduce the variance of minibatch gradients and speed up&#10;optimization. To generate long and higher resolution videos we introduce a new&#10;conditional sampling technique for spatial and temporal video extension that&#10;performs better than previously proposed methods. We present the first results&#10;on a large text-conditioned video generation task, as well as state-of-the-art&#10;results on established benchmarks for video prediction and unconditional video&#10;generation. Supplementary material is available at&#10;https://video-diffusion.github.io/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.00650" label="2104.00650">
        <attvalues>
          <attvalue for="0" value="Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval" />
          <attvalue for="1" value="  Our objective in this work is video-text retrieval - in particular a joint&#10;embedding that enables efficient text-to-video retrieval. The challenges in&#10;this area include the design of the visual architecture and the nature of the&#10;training data, in that the available large scale video-text training datasets,&#10;such as HowTo100M, are noisy and hence competitive performance is achieved only&#10;at scale through large amounts of compute. We address both these challenges in&#10;this paper. We propose an end-to-end trainable model that is designed to take&#10;advantage of both large-scale image and video captioning datasets. Our model is&#10;an adaptation and extension of the recent ViT and Timesformer architectures,&#10;and consists of attention in both space and time. The model is flexible and can&#10;be trained on both image and video text datasets, either independently or in&#10;conjunction. It is trained with a curriculum learning schedule that begins by&#10;treating images as 'frozen' snapshots of video, and then gradually learns to&#10;attend to increasing temporal context when trained on video datasets. We also&#10;provide a new video-text pretraining dataset WebVid-2M, comprised of over two&#10;million videos with weak captions scraped from the internet. Despite training&#10;on datasets that are an order of magnitude smaller, we show that this approach&#10;yields state-of-the-art results on standard downstream video-retrieval&#10;benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.13221" label="2211.13221">
        <attvalues>
          <attvalue for="0" value="Latent Video Diffusion Models for High-Fidelity Long Video Generation" />
          <attvalue for="1" value="  AI-generated content has attracted lots of attention recently, but&#10;photo-realistic video synthesis is still challenging. Although many attempts&#10;using GANs and autoregressive models have been made in this area, the visual&#10;quality and length of generated videos are far from satisfactory. Diffusion&#10;models have shown remarkable results recently but require significant&#10;computational resources. To address this, we introduce lightweight video&#10;diffusion models by leveraging a low-dimensional 3D latent space, significantly&#10;outperforming previous pixel-space video diffusion models under a limited&#10;computational budget. In addition, we propose hierarchical diffusion in the&#10;latent space such that longer videos with more than one thousand frames can be&#10;produced. To further overcome the performance degradation issue for long video&#10;generation, we propose conditional latent perturbation and unconditional&#10;guidance that effectively mitigate the accumulated errors during the extension&#10;of video length. Extensive experiments on small domain datasets of different&#10;categories suggest that our framework generates more realistic and longer&#10;videos than previous strong baselines. We additionally provide an extension to&#10;large-scale text-to-video generation to demonstrate the superiority of our&#10;work. Our code and models will be made publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.02624" label="2110.02624">
        <attvalues>
          <attvalue for="0" value="CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation" />
          <attvalue for="1" value="  Generating shapes using natural language can enable new ways of imagining and&#10;creating the things around us. While significant recent progress has been made&#10;in text-to-image generation, text-to-shape generation remains a challenging&#10;problem due to the unavailability of paired text and shape data at a large&#10;scale. We present a simple yet effective method for zero-shot text-to-shape&#10;generation that circumvents such data scarcity. Our proposed method, named&#10;CLIP-Forge, is based on a two-stage training process, which only depends on an&#10;unlabelled shape dataset and a pre-trained image-text network such as CLIP. Our&#10;method has the benefits of avoiding expensive inference time optimization, as&#10;well as the ability to generate multiple shapes for a given text. We not only&#10;demonstrate promising zero-shot generalization of the CLIP-Forge model&#10;qualitatively and quantitatively, but also provide extensive comparative&#10;evaluations to better understand its behavior.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.13333" label="2203.13333">
        <attvalues>
          <attvalue for="0" value="CLIP-Mesh: Generating textured meshes from text using pretrained&#10;  image-text models" />
          <attvalue for="1" value="  We present a technique for zero-shot generation of a 3D model using only a&#10;target text prompt. Without any 3D supervision our method deforms the control&#10;shape of a limit subdivided surface along with its texture map and normal map&#10;to obtain a 3D asset that corresponds to the input text prompt and can be&#10;easily deployed into games or modeling applications. We rely only on a&#10;pre-trained CLIP model that compares the input text prompt with differentiably&#10;rendered images of our 3D model. While previous works have focused on&#10;stylization or required training of generative models we perform optimization&#10;on mesh parameters directly to generate shape, texture or both. To constrain&#10;the optimization to produce plausible meshes and textures we introduce a number&#10;of techniques using image augmentations and the use of a pretrained prior that&#10;generates CLIP image embeddings given a text embedding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.03221" label="2112.03221">
        <attvalues>
          <attvalue for="0" value="Text2Mesh: Text-Driven Neural Stylization for Meshes" />
          <attvalue for="1" value="  In this work, we develop intuitive controls for editing the style of 3D&#10;objects. Our framework, Text2Mesh, stylizes a 3D mesh by predicting color and&#10;local geometric details which conform to a target text prompt. We consider a&#10;disentangled representation of a 3D object using a fixed mesh input (content)&#10;coupled with a learned neural network, which we term neural style field&#10;network. In order to modify style, we obtain a similarity score between a text&#10;prompt (describing style) and a stylized mesh by harnessing the&#10;representational power of CLIP. Text2Mesh requires neither a pre-trained&#10;generative model nor a specialized 3D mesh dataset. It can handle low-quality&#10;meshes (non-manifold, boundaries, etc.) with arbitrary genus, and does not&#10;require UV parameterization. We demonstrate the ability of our technique to&#10;synthesize a myriad of styles over a wide variety of 3D meshes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.08070" label="2212.08070">
        <attvalues>
          <attvalue for="0" value="NeRF-Art: Text-Driven Neural Radiance Fields Stylization" />
          <attvalue for="1" value="  As a powerful representation of 3D scenes, the neural radiance field (NeRF)&#10;enables high-quality novel view synthesis from multi-view images. Stylizing&#10;NeRF, however, remains challenging, especially on simulating a text-guided&#10;style with both the appearance and the geometry altered simultaneously. In this&#10;paper, we present NeRF-Art, a text-guided NeRF stylization approach that&#10;manipulates the style of a pre-trained NeRF model with a simple text prompt.&#10;Unlike previous approaches that either lack sufficient geometry deformations&#10;and texture details or require meshes to guide the stylization, our method can&#10;shift a 3D scene to the target style characterized by desired geometry and&#10;appearance variations without any mesh guidance. This is achieved by&#10;introducing a novel global-local contrastive learning strategy, combined with&#10;the directional constraint to simultaneously control both the trajectory and&#10;the strength of the target style. Moreover, we adopt a weight regularization&#10;method to effectively suppress cloudy artifacts and geometry noises which arise&#10;easily when the density field is transformed during geometry stylization.&#10;Through extensive experiments on various styles, we demonstrate that our method&#10;is effective and robust regarding both single-view stylization quality and&#10;cross-view consistency. The code and more results can be found in our project&#10;page: https://cassiepython.github.io/nerfart/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.08529" label="2310.08529">
        <attvalues>
          <attvalue for="0" value="GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging&#10;  2D and 3D Diffusion Models" />
          <attvalue for="1" value="  In recent times, the generation of 3D assets from text prompts has shown&#10;impressive results. Both 2D and 3D diffusion models can help generate decent 3D&#10;objects based on prompts. 3D diffusion models have good 3D consistency, but&#10;their quality and generalization are limited as trainable 3D data is expensive&#10;and hard to obtain. 2D diffusion models enjoy strong abilities of&#10;generalization and fine generation, but 3D consistency is hard to guarantee.&#10;This paper attempts to bridge the power from the two types of diffusion models&#10;via the recent explicit and efficient 3D Gaussian splatting representation. A&#10;fast 3D object generation framework, named as GaussianDreamer, is proposed,&#10;where the 3D diffusion model provides priors for initialization and the 2D&#10;diffusion model enriches the geometry and appearance. Operations of noisy point&#10;growing and color perturbation are introduced to enhance the initialized&#10;Gaussians. Our GaussianDreamer can generate a high-quality 3D instance or 3D&#10;avatar within 15 minutes on one GPU, much faster than previous methods, while&#10;the generated instances can be directly rendered in real time. Demos and code&#10;are available at https://taoranyi.com/gaussiandreamer/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.07937" label="2303.07937">
        <attvalues>
          <attvalue for="0" value="Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D&#10;  Generation" />
          <attvalue for="1" value="  Text-to-3D generation has shown rapid progress in recent days with the advent&#10;of score distillation, a methodology of using pretrained text-to-2D diffusion&#10;models to optimize neural radiance field (NeRF) in the zero-shot setting.&#10;However, the lack of 3D awareness in the 2D diffusion models destabilizes score&#10;distillation-based methods from reconstructing a plausible 3D scene. To address&#10;this issue, we propose 3DFuse, a novel framework that incorporates 3D awareness&#10;into pretrained 2D diffusion models, enhancing the robustness and 3D&#10;consistency of score distillation-based methods. We realize this by first&#10;constructing a coarse 3D structure of a given text prompt and then utilizing&#10;projected, view-specific depth map as a condition for the diffusion model.&#10;Additionally, we introduce a training strategy that enables the 2D diffusion&#10;model learns to handle the errors and sparsity within the coarse 3D structure&#10;for robust generation, as well as a method for ensuring semantic consistency&#10;throughout all viewpoints of the scene. Our framework surpasses the limitations&#10;of prior arts, and has significant implications for 3D consistent generation of&#10;2D diffusion models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.11325" label="2301.11325">
        <attvalues>
          <attvalue for="0" value="MusicLM: Generating Music From Text" />
          <attvalue for="1" value="  We introduce MusicLM, a model generating high-fidelity music from text&#10;descriptions such as &quot;a calming violin melody backed by a distorted guitar&#10;riff&quot;. MusicLM casts the process of conditional music generation as a&#10;hierarchical sequence-to-sequence modeling task, and it generates music at 24&#10;kHz that remains consistent over several minutes. Our experiments show that&#10;MusicLM outperforms previous systems both in audio quality and adherence to the&#10;text description. Moreover, we demonstrate that MusicLM can be conditioned on&#10;both text and a melody in that it can transform whistled and hummed melodies&#10;according to the style described in a text caption. To support future research,&#10;we publicly release MusicCaps, a dataset composed of 5.5k music-text pairs,&#10;with rich text descriptions provided by human experts.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.08041" label="2307.08041">
        <attvalues>
          <attvalue for="0" value="Planting a SEED of Vision in Large Language Model" />
          <attvalue for="1" value="  We present SEED, an elaborate image tokenizer that empowers Large Language&#10;Models (LLMs) with the emergent ability to SEE and Draw at the same time.&#10;Research on image tokenizers has previously reached an impasse, as frameworks&#10;employing quantized visual tokens have lost prominence due to subpar&#10;performance and convergence in multimodal comprehension (compared to BLIP-2,&#10;etc.) or generation (compared to Stable Diffusion, etc.). Despite the&#10;limitations, we remain confident in its natural capacity to unify visual and&#10;textual representations, facilitating scalable multimodal training with LLM's&#10;original recipe. In this study, we identify two crucial principles for the&#10;architecture and training of SEED that effectively ease subsequent alignment&#10;with LLMs. (1) Image tokens should be independent of 2D physical patch&#10;positions and instead be produced with a 1D causal dependency, exhibiting&#10;intrinsic interdependence that aligns with the left-to-right autoregressive&#10;prediction mechanism in LLMs. (2) Image tokens should capture high-level&#10;semantics consistent with the degree of semantic abstraction in words, and be&#10;optimized for both discriminativeness and reconstruction during the tokenizer&#10;training phase. As a result, the off-the-shelf LLM is able to perform both&#10;image-to-text and text-to-image generation by incorporating our SEED through&#10;efficient LoRA tuning. Comprehensive multimodal pretraining and instruction&#10;tuning, which may yield improved results, are reserved for future&#10;investigation. This version of SEED was trained in 5.7 days using only 64 V100&#10;GPUs and 5M publicly available image-text pairs. Our preliminary study&#10;emphasizes the great potential of discrete visual tokens in versatile&#10;multimodal LLMs and the importance of proper image tokenizers in broader&#10;research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.16465" label="2311.16465">
        <attvalues>
          <attvalue for="0" value="TextDiffuser-2: Unleashing the Power of Language Models for Text&#10;  Rendering" />
          <attvalue for="1" value="  The diffusion model has been proven a powerful generative model in recent&#10;years, yet remains a challenge in generating visual text. Several methods&#10;alleviated this issue by incorporating explicit text position and content as&#10;guidance on where and what text to render. However, these methods still suffer&#10;from several drawbacks, such as limited flexibility and automation, constrained&#10;capability of layout prediction, and restricted style diversity. In this paper,&#10;we present TextDiffuser-2, aiming to unleash the power of language models for&#10;text rendering. Firstly, we fine-tune a large language model for layout&#10;planning. The large language model is capable of automatically generating&#10;keywords for text rendering and also supports layout modification through&#10;chatting. Secondly, we utilize the language model within the diffusion model to&#10;encode the position and texts at the line level. Unlike previous methods that&#10;employed tight character-level guidance, this approach generates more diverse&#10;text images. We conduct extensive experiments and incorporate user studies&#10;involving human participants as well as GPT-4V, validating TextDiffuser-2's&#10;capacity to achieve a more rational text layout and generation with enhanced&#10;diversity. The code and model will be available at&#10;\url{https://aka.ms/textdiffuser-2}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.07093" label="2301.07093">
        <attvalues>
          <attvalue for="0" value="GLIGEN: Open-Set Grounded Text-to-Image Generation" />
          <attvalue for="1" value="  Large-scale text-to-image diffusion models have made amazing advances.&#10;However, the status quo is to use text input alone, which can impede&#10;controllability. In this work, we propose GLIGEN, Grounded-Language-to-Image&#10;Generation, a novel approach that builds upon and extends the functionality of&#10;existing pre-trained text-to-image diffusion models by enabling them to also be&#10;conditioned on grounding inputs. To preserve the vast concept knowledge of the&#10;pre-trained model, we freeze all of its weights and inject the grounding&#10;information into new trainable layers via a gated mechanism. Our model achieves&#10;open-world grounded text2img generation with caption and bounding box condition&#10;inputs, and the grounding ability generalizes well to novel spatial&#10;configurations and concepts. GLIGEN's zero-shot performance on COCO and LVIS&#10;outperforms that of existing supervised layout-to-image baselines by a large&#10;margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.12945" label="2310.12945">
        <attvalues>
          <attvalue for="0" value="3D-GPT: Procedural 3D Modeling with Large Language Models" />
          <attvalue for="1" value="  In the pursuit of efficient automated content creation, procedural&#10;generation, leveraging modifiable parameters and rule-based systems, emerges as&#10;a promising approach. Nonetheless, it could be a demanding endeavor, given its&#10;intricate nature necessitating a deep understanding of rules, algorithms, and&#10;parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing&#10;large language models~(LLMs) for instruction-driven 3D modeling. 3D-GPT&#10;positions LLMs as proficient problem solvers, dissecting the procedural 3D&#10;modeling tasks into accessible segments and appointing the apt agent for each&#10;task. 3D-GPT integrates three core agents: the task dispatch agent, the&#10;conceptualization agent, and the modeling agent. They collaboratively achieve&#10;two objectives. First, it enhances concise initial scene descriptions, evolving&#10;them into detailed forms while dynamically adapting the text based on&#10;subsequent instructions. Second, it integrates procedural generation,&#10;extracting parameter values from enriched text to effortlessly interface with&#10;3D software for asset creation. Our empirical investigations confirm that&#10;3D-GPT not only interprets and executes instructions, delivering reliable&#10;results but also collaborates effectively with human designers. Furthermore, it&#10;seamlessly integrates with Blender, unlocking expanded manipulation&#10;possibilities. Our work highlights the potential of LLMs in 3D modeling,&#10;offering a basic framework for future advancements in scene generation and&#10;animation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.18651" label="2311.18651">
        <attvalues>
          <attvalue for="0" value="LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding,&#10;  Reasoning, and Planning" />
          <attvalue for="1" value="  Recent advances in Large Multimodal Models (LMM) have made it possible for&#10;various applications in human-machine interactions. However, developing LMMs&#10;that can comprehend, reason, and plan in complex and diverse 3D environments&#10;remains a challenging topic, especially considering the demand for&#10;understanding permutation-invariant point cloud 3D representations of the 3D&#10;scene. Existing works seek help from multi-view images, and project 2D features&#10;to 3D space as 3D scene representations. This, however, leads to huge&#10;computational overhead and performance degradation. In this paper, we present&#10;LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and&#10;respond to both textual-instructions and visual-prompts. This help LMMs better&#10;comprehend human interactions and further help to remove the ambiguities in&#10;cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results,&#10;and surpasses various 3D vision-language models on both 3D Dense Captioning and&#10;3D Question Answering.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.04092" label="2401.04092">
        <attvalues>
          <attvalue for="0" value="GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation" />
          <attvalue for="1" value="  Despite recent advances in text-to-3D generative methods, there is a notable&#10;absence of reliable evaluation metrics. Existing metrics usually focus on a&#10;single criterion each, such as how well the asset aligned with the input text.&#10;These metrics lack the flexibility to generalize to different evaluation&#10;criteria and might not align well with human preferences. Conducting user&#10;preference studies is an alternative that offers both adaptability and&#10;human-aligned results. User studies, however, can be very expensive to scale.&#10;This paper presents an automatic, versatile, and human-aligned evaluation&#10;metric for text-to-3D generative models. To this end, we first develop a prompt&#10;generator using GPT-4V to generate evaluating prompts, which serve as input to&#10;compare text-to-3D models. We further design a method instructing GPT-4V to&#10;compare two 3D assets according to user-defined criteria. Finally, we use these&#10;pairwise comparison results to assign these models Elo ratings. Experimental&#10;results suggest our metric strongly align with human preference across&#10;different evaluation criteria.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.16153" label="2402.16153">
        <attvalues>
          <attvalue for="0" value="ChatMusician: Understanding and Generating Music Intrinsically with LLM" />
          <attvalue for="1" value="  While Large Language Models (LLMs) demonstrate impressive capabilities in&#10;text generation, we find that their ability has yet to be generalized to music,&#10;humanity's creative language. We introduce ChatMusician, an open-source LLM&#10;that integrates intrinsic musical abilities. It is based on continual&#10;pre-training and finetuning LLaMA2 on a text-compatible music representation,&#10;ABC notation, and the music is treated as a second language. ChatMusician can&#10;understand and generate music with a pure text tokenizer without any external&#10;multi-modal neural structures or tokenizers. Interestingly, endowing musical&#10;abilities does not harm language abilities, even achieving a slightly higher&#10;MMLU score. Our model is capable of composing well-structured, full-length&#10;music, conditioned on texts, chords, melodies, motifs, musical forms, etc,&#10;surpassing GPT-4 baseline. On our meticulously curated college-level music&#10;understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and&#10;GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs&#10;can be an excellent compressor for music, but there remains significant&#10;territory to be conquered. We release our 4B token music-language corpora&#10;MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18474" label="2305.18474">
        <attvalues>
          <attvalue for="0" value="Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation" />
          <attvalue for="1" value="  Large diffusion models have been successful in text-to-audio (T2A) synthesis&#10;tasks, but they often suffer from common issues such as semantic misalignment&#10;and poor temporal consistency due to limited natural language understanding and&#10;data scarcity. Additionally, 2D spatial structures widely used in T2A works&#10;lead to unsatisfactory audio quality when generating variable-length audio&#10;samples since they do not adequately prioritize temporal information. To&#10;address these challenges, we propose Make-an-Audio 2, a latent diffusion-based&#10;T2A method that builds on the success of Make-an-Audio. Our approach includes&#10;several techniques to improve semantic alignment and temporal consistency:&#10;Firstly, we use pre-trained large language models (LLMs) to parse the text into&#10;structured &lt;event &amp; order&gt; pairs for better temporal information capture. We&#10;also introduce another structured-text encoder to aid in learning semantic&#10;alignment during the diffusion denoising process. To improve the performance of&#10;variable length generation and enhance the temporal information extraction, we&#10;design a feed-forward Transformer-based diffusion denoiser. Finally, we use&#10;LLMs to augment and transform a large amount of audio-label data into&#10;audio-text datasets to alleviate the problem of scarcity of temporal data.&#10;Extensive experiments show that our method outperforms baseline models in both&#10;objective and subjective metrics, and achieves significant gains in temporal&#10;information understanding, semantic consistency, and sound quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.17796" label="2310.17796">
        <attvalues>
          <attvalue for="0" value="ControlLLM: Augment Language Models with Tools by Searching on Graphs" />
          <attvalue for="1" value="  We present ControlLLM, a novel framework that enables large language models&#10;(LLMs) to utilize multi-modal tools for solving complex real-world tasks.&#10;Despite the remarkable performance of LLMs, they still struggle with tool&#10;invocation due to ambiguous user prompts, inaccurate tool selection and&#10;parameterization, and inefficient tool scheduling. To overcome these&#10;challenges, our framework comprises three key components: (1) a \textit{task&#10;decomposer} that breaks down a complex task into clear subtasks with&#10;well-defined inputs and outputs; (2) a \textit{Thoughts-on-Graph (ToG)&#10;paradigm} that searches the optimal solution path on a pre-built tool graph,&#10;which specifies the parameter and dependency relations among different tools;&#10;and (3) an \textit{execution engine with a rich toolbox} that interprets the&#10;solution path and runs the tools efficiently on different computational&#10;devices. We evaluate our framework on diverse tasks involving image, audio, and&#10;video processing, demonstrating its superior accuracy, efficiency, and&#10;versatility compared to existing methods. The code is at&#10;https://github.com/OpenGVLab/ControlLLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18752" label="2305.18752">
        <attvalues>
          <attvalue for="0" value="GPT4Tools: Teaching Large Language Model to Use Tools via&#10;  Self-instruction" />
          <attvalue for="1" value="  This paper aims to efficiently enable Large Language Models (LLMs) to use&#10;multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have&#10;shown great potential for tool usage through sophisticated prompt engineering.&#10;Nevertheless, these models typically rely on prohibitive computational costs&#10;and publicly inaccessible data. To address these challenges, we propose the&#10;GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and&#10;OPT, to use tools. It generates an instruction-following dataset by prompting&#10;an advanced teacher with various multi-modal contexts. By using the Low-Rank&#10;Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs&#10;to solve a range of visual problems, including visual comprehension and image&#10;generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to&#10;use tools, which is performed in both zero-shot and fine-tuning ways. Extensive&#10;experiments demonstrate the effectiveness of our method on various language&#10;models, which not only significantly improves the accuracy of invoking seen&#10;tools, but also enables the zero-shot capacity for unseen tools. The code and&#10;demo are available at https://github.com/StevenGrove/GPT4Tools.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.00986" label="2309.00986">
        <attvalues>
          <attvalue for="0" value="ModelScope-Agent: Building Your Customizable Agent System with&#10;  Open-source Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) have recently demonstrated remarkable&#10;capabilities to comprehend human intentions, engage in reasoning, and design&#10;planning-like behavior. To further unleash the power of LLMs to accomplish&#10;complex tasks, there is a growing trend to build agent framework that equips&#10;LLMs, such as ChatGPT, with tool-use abilities to connect with massive external&#10;APIs. In this work, we introduce ModelScope-Agent, a general and customizable&#10;agent framework for real-world applications, based on open-source LLMs as&#10;controllers. It provides a user-friendly system library, with customizable&#10;engine design to support model training on multiple open-source LLMs, while&#10;also enabling seamless integration with both model APIs and common APIs in a&#10;unified way. To equip the LLMs with tool-use abilities, a comprehensive&#10;framework has been proposed spanning over tool-use data collection, tool&#10;retrieval, tool registration, memory control, customized model training, and&#10;evaluation for practical real-world applications. Finally, we showcase&#10;ModelScopeGPT, a real-world intelligent assistant of ModelScope Community based&#10;on the ModelScope-Agent framework, which is able to connect open-source LLMs&#10;with more than 1000 public AI models and localized community knowledge in&#10;ModelScope. The ModelScope-Agent&#10;library\footnote{https://github.com/modelscope/modelscope-agent} and online&#10;demo\footnote{https://modelscope.cn/studios/damo/ModelScopeGPT/summary} are now&#10;publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.12581" label="2306.12581">
        <attvalues>
          <attvalue for="0" value="Morphological Inflection with Phonological Features" />
          <attvalue for="1" value="  Recent years have brought great advances into solving morphological tasks,&#10;mostly due to powerful neural models applied to various tasks as (re)inflection&#10;and analysis. Yet, such morphological tasks cannot be considered solved,&#10;especially when little training data is available or when generalizing to&#10;previously unseen lemmas. This work explores effects on performance obtained&#10;through various ways in which morphological models get access to subcharacter&#10;phonological features that are the targets of morphological processes. We&#10;design two methods to achieve this goal: one that leaves models as is but&#10;manipulates the data to include features instead of characters, and another&#10;that manipulates models to take phonological features into account when&#10;building representations for phonemes. We elicit phonemic data from standard&#10;graphemic data using language-specific grammars for languages with shallow&#10;grapheme-to-phoneme mapping, and we experiment with two reinflection models&#10;over eight languages. Our results show that our methods yield comparable&#10;results to the grapheme-based baseline overall, with minor improvements in some&#10;of the languages. All in all, we conclude that patterns in character&#10;distributions are likely to allow models to infer the underlying phonological&#10;characteristics, even when phonemes are not explicitly represented.&#10;" />
          <attvalue for="2" value="&#10;In recent years, morphological tasks received much attention in NLP through various tasks such as (re)inflection, lemmatization and others, specifically through the SIGMORPHON shared tasks \cite{cotterell-etal-2016-sigmorphon, cotterell-etal-2017-conll, cotterell-etal-2018-conll, mccarthy-etal-2019-sigmorphon, vylomova-etal-2020-sigmorphon, pimentel-ryskina-etal-2021-sigmorphon}.&#10;State-of-the-art models seem to achieve quite high results in such cross-lingual evaluation campaigns, although recent works showed that there is still room for improvements \cite{goldman-etal-2022-un}.&#10;&#10;Most studies aiming at morphological tasks design models that operate at the character level, without reference to the phonological components that compose the phonemes represented by the characters. This is despite the fact that many morphological processes have distinct phonological features, rather than phonemes, as either the trigger or target of morphological processes. For example, in vowel harmony, a single feature of a vowel in the stem determines the vowels that appear in the affixes added to that stem. Without direct evidence of the phonological features composing every phoneme, models must resort to memorizing groups of phonemes that pattern together for an unobserved reason.&#10;&#10;In this work we hypothesize that explicitly inputting models with phonological features will lead to better modelling of morphological tasks. We set out to equip models with two alternative methods for incorporating that information. One method replaces the character-level tokens with phonological feature tokens; and another one equips the model with a self-attention mechanism that learns representation of phonemes from their features.&#10;&#10;We implement these methods on the task of morphological reinflection, where forms of the same lemma are inflected from one another. We experiment with 8 languages and 2 models: an LSTM encoder-decoder with global attention by \cite{silfverberg-hulden-2018-encoder}; and a transducer by \cite{makarov-clematide-2018-neural} that predicts edit actions between source and target word-forms and is suitable for lower amounts of training data.&#10;&#10;Our experiments show that the proposed methods yield results comparable to the grapheme-based baseline setting for the transducer model. On average across languages, the best phonologically-aware method suffered from a drop of 2.8 accuracy points, although the performance on some individual languages marginally improved. We thus conjecture that the phonological characteristics are already encoded in the graphemic representations elicited by this model. The results of this work are in line with other works, performed in different settings, investigating the role of phonology in morphological models (see Section~\ref{sec:discussion}).&#10;&#10;We further note that the LSTM model, unlike the transducer, did not perform well on graphemic data and suffered from a severe drop when applied on phonological data in all tested languages. We attribute this to the transducer's attested ability to perform well particularly in low-resource setting. We subsequently conjecture that, for the phonologically-aware variant of the reinflection task, standard amounts of reinflection data should be effectively considered low-resourced. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Phonological Features, Computer Science, Linguistics, Morphological Modeling, Neural Language Processing, Artificial Intelligence, Phonetics" />
        </attvalues>
      </node>
      <node id="2108.05682" label="2108.05682">
        <attvalues>
          <attvalue for="0" value="(Un)solving Morphological Inflection: Lemma Overlap Artificially&#10;  Inflates Models' Performance" />
          <attvalue for="1" value="  In the domain of Morphology, Inflection is a fundamental and important task&#10;that gained a lot of traction in recent years, mostly via SIGMORPHON's&#10;shared-tasks. With average accuracy above 0.9 over the scores of all languages,&#10;the task is considered mostly solved using relatively generic neural seq2seq&#10;models, even with little data provided. In this work, we propose to re-evaluate&#10;morphological inflection models by employing harder train-test splits that will&#10;challenge the generalization capacity of the models. In particular, as opposed&#10;to the na{\&quot;i}ve split-by-form, we propose a split-by-lemma method to challenge&#10;the performance on existing benchmarks. Our experiments with the three&#10;top-ranked systems on the SIGMORPHON's 2020 shared-task show that the&#10;lemma-split presents an average drop of 30 percentage points in macro-average&#10;for the 90 languages included. The effect is most significant for low-resourced&#10;languages with a drop as high as 95 points, but even high-resourced languages&#10;lose about 10 points on average. Our results clearly show that generalizing&#10;inflection to unseen lemmas is far from being solved, presenting a simple yet&#10;effective means to promote more sophisticated models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.06562" label="2406.06562">
        <attvalues>
          <attvalue for="0" value="Achieving Sparse Activation in Small Language Models" />
          <attvalue for="1" value="  Sparse activation, which selectively activates only an input-dependent set of&#10;neurons in inference, is a useful technique to reduce the computing cost of&#10;Large Language Models (LLMs) without retraining or adaptation efforts. However,&#10;whether it can be applied to the recently emerging Small Language Models (SLMs)&#10;remains questionable, because SLMs are generally less over-parameterized than&#10;LLMs. In this paper, we aim to achieve sparse activation in SLMs. We first show&#10;that the existing sparse activation schemes in LLMs that build on neurons'&#10;output magnitudes cannot be applied to SLMs, and activating neurons based on&#10;their attribution scores is a better alternative. Further, we demonstrated and&#10;quantified the large errors of existing attribution metrics when being used for&#10;sparse activation, due to the interdependency among attribution scores of&#10;neurons across different layers. Based on these observations, we proposed a new&#10;attribution metric that can provably correct such errors and achieve precise&#10;sparse activation. Experiments over multiple popular SLMs and datasets show&#10;that our approach can achieve 80% sparsification ratio with &lt;5% model accuracy&#10;loss, comparable to the sparse activation achieved in LLMs. The source code is&#10;available at: https://github.com/pittisl/Sparse-Activation.&#10;" />
          <attvalue for="2" value="&#10;In the era dominated by Large Language Models (LLMs), the recent emergence of Small Language Models (SLMs) presents an intriguing shift. Typical SLMs [\cite{abdin2024phi, thawakar2024mobillama, team2024gemma}] are much more lightweight (e.g., with $&lt;$3B parameters) than the existing LLMs \cite{touvron2023llama, almazrouei2023falcon} (e.g., with $&gt;$65B parameters), but can achieve on-par accuracy in simpler or specific tasks. However, inference with SLMs on resource-constrained mobile and embedded devices could still be too computationally expensive. &#10;&#10;Research efforts have been made to reduce the inference cost of language models. Model compression techniques reduce the model size, via either pruning that sparsifies the model structures [\cite{kurtic2024ziplm, ma2023llm, kurtic2024ziplm}], quantization that reduces the numerical precision of the model [\cite{lin2023awq, kim2024memory, chee2024quip}], or knowledge distillation that migrates the general knowledge to a smaller model [\cite{kang2024knowledge, zhao2023multistage}]. However, most of these methods require intensive model retraining, which is computationally expensive and cannot well adapt to different downstream tasks or input data [\cite{michel2019sixteen, bansal2022rethinking}]. Other schemes of efficient attention algorithms [\cite{dao2022flashattention}] and decoding techniques [\cite{leviathan2023fast}] can also reduce the memory cost and computing latency in inference, but do not help reduce the model's redundancy. &#10;&#10;Sparse activation, as shown in Figure \ref{fig:sparse_activation}, can well complement the techniques above and enable run-time improvement of inference performance without any model retraining or adaptation efforts, by selectively activating only an input-dependent set of model's neurons that are most critical to inference with current input data. Although sparse activation has been demonstrated to significantly reduce the memory cost and computing latency of inference in LLMs without impairing accuracy [\cite{liu2023deja, song2023powerinfer}], whether it can be applied to SLMs still remains questionable. The major reason is that SLMs are much less over-parameterized than LLMs. Even neurons with small output magnitudes may still be important in inference, because they may largely affect the model's gradients across layers. Simply deactivating these neurons, as suggested by the existing work [\cite{liu2023deja, song2023powerinfer}], may cause non-negligible accuracy loss. &#10;&#10;In this paper, we focus on achieving sparse activation in SLMs. Based on observations that deactivating neurons with small output magnitudes indeed results in large accuracy loss in SLMs, we experimentally showed that using gradient-based attribution scores to evaluate neurons' importance in inference and deactivate less important neurons is a more promising approach (\S\ref{subsec:attribution}), because such attribution scores precisely evaluate the impact of neuron deactivation on the model output. Some attribution schemes (e.g., Integrated Gradients [\cite{sundararajan2017axiomatic,yvinec2022singe}]) integrate multiple gradients over interpolated input samples to ensure precise attribution scores, but also incur high computing costs. Other schemes provide more computationally efficient methods by calculating the attribution scores as the product of a neuron's gradient and output magnitude (Gradient $\times$ Output) [\cite{liu2021group,lee2018snip}], which is the first-order approximation of the model output's change due to neuron deactivation. However, such attribution scores, when being applied to sparse activation, could result in large error due to the interdependency of neurons in one layer and across different layers. Such error may result in improper rankings of neuron's attribution scores and hence lead to suboptimal neuron activation (\S\ref{subsec:approximation_error}).&#10;&#10;To effectively mitigate such attribution errors and achieve optimal sparse activation, we analytically quantified the lower and upper bounds of such error caused by inter-layer dependency among neurons in transformer-based SLM architectures, and further proposed to apply the expectation of such attribution error as a corrective term to the Gradient $\times$ Output (GxO) attribution metric (\S\ref{sec:method}). We evaluated the model performance when applying such corrected GxO metric for sparse activation, with multiple popular SLMs including Phi-1.5/2 [\cite{gunasekar2023textbooks}] and MobiLlama-0.5B/1B [\cite{thawakar2024mobillama}], and multiple question answering (QA) datasets including TruthfulQA [\cite{lin2021truthfulqa}] and YahooAnswersQA [\cite{YahooAnswersQA}]. Our main findings are as follows:&#10;&#10;\begin{itemize}&#10;\item Our proposed attribution metric achieves high sparsity in SLMs, in both attention layers and MLP layers. It can deactivate up to 80\% of neurons in major SLM models while incurring $&lt;$5\% model accuracy loss. Such sparisification ratio is similar to that reported in existing work for LLMs [\cite{liu2023deja}], and allows significant memory savings and computing latency reduction.&#10;\item We demonstrate good generality over different types of SLMs. On both models of the Phi and MobiLlama series, our proposed attribution metric outperforms baseline schemes in model accuracy by at least 25\%.&#10;\item Our approach of applying the corrective term to neuron attribution metrics has high compute efficiency and incur a negligible amount of extra computing costs.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Cognitive Science, Linguistics, Sparse Activation Techniques, Mathematics, Neural Network Efficiency, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2302.04089" label="2302.04089">
        <attvalues>
          <attvalue for="0" value="ZipLM: Inference-Aware Structured Pruning of Language Models" />
          <attvalue for="1" value="  The breakthrough performance of large language models (LLMs) comes with major&#10;computational footprints and high deployment costs. In this paper, we progress&#10;towards resolving this problem by proposing a novel structured compression&#10;approach for LLMs, called ZipLM. ZipLM achieves state-of-the-art&#10;accuracy-vs-speedup, while matching a set of desired target runtime speedups in&#10;any given inference environment. Specifically, given a model, a dataset, an&#10;inference environment, as well as a set of speedup targets, ZipLM iteratively&#10;identifies and removes components with the worst loss-runtime trade-off. Unlike&#10;prior methods that specialize in either the post-training/one-shot or the&#10;gradual compression setting, and only for specific families of models such as&#10;BERT (encoder) or GPT (decoder), ZipLM produces state-of-the-art compressed&#10;models across all these settings. Furthermore, ZipLM achieves superior results&#10;for a fraction of the computational cost relative to prior distillation and&#10;pruning techniques, making it a cost-effective approach for generating an&#10;entire family of smaller, faster, and highly accurate models, guaranteed to&#10;meet the desired inference specifications. In particular, ZipLM outperforms all&#10;prior BERT-base distillation and pruning techniques, such as CoFi, MiniLM, and&#10;TinyBERT. Moreover, it matches the performance of the heavily optimized&#10;MobileBERT model, obtained via extensive architecture search, by simply pruning&#10;the baseline BERT-large model. When compressing GPT2, ZipLM outperforms&#10;DistilGPT2 while being 60% smaller and 30% faster. Our code is available at:&#10;https://github.com/IST-DASLab/ZipLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18395" label="2305.18395">
        <attvalues>
          <attvalue for="0" value="Knowledge-Augmented Reasoning Distillation for Small Language Models in&#10;  Knowledge-Intensive Tasks" />
          <attvalue for="1" value="  Large Language Models (LLMs) have shown promising performance in&#10;knowledge-intensive reasoning tasks that require a compound understanding of&#10;knowledge. However, deployment of the LLMs in real-world applications can be&#10;challenging due to their high computational requirements and concerns on data&#10;privacy. Previous studies have focused on building task-specific small Language&#10;Models (LMs) by fine-tuning them with labeled data or distilling LLMs. However,&#10;these approaches are ill-suited for knowledge-intensive reasoning tasks due to&#10;the limited capacity of small LMs in memorizing the knowledge required.&#10;Motivated by our theoretical analysis on memorization, we propose&#10;Knowledge-Augmented Reasoning Distillation (KARD), a novel method that&#10;fine-tunes small LMs to generate rationales obtained from LLMs with augmented&#10;knowledge retrieved from an external knowledge base. Moreover, we further&#10;propose a neural reranker to obtain documents relevant to rationale generation.&#10;We empirically show that KARD significantly improves the performance of small&#10;T5 and GPT models on the challenging knowledge-intensive reasoning datasets,&#10;namely MedQA-USMLE, StrategyQA, and OpenbookQA. Notably, our method makes the&#10;250M T5 models achieve superior performance against the fine-tuned 3B models,&#10;having 12 times larger parameters, on both MedQA-USMLE and StrategyQA&#10;benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.04089" label="2207.04089">
        <attvalues>
          <attvalue for="0" value="SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance" />
          <attvalue for="1" value="  The leap in performance in state-of-the-art computer vision methods is&#10;attributed to the development of deep neural networks. However it often comes&#10;at a computational price which may hinder their deployment. To alleviate this&#10;limitation, structured pruning is a well known technique which consists in&#10;removing channels, neurons or filters, and is commonly applied in order to&#10;produce more compact models. In most cases, the computations to remove are&#10;selected based on a relative importance criterion. At the same time, the need&#10;for explainable predictive models has risen tremendously and motivated the&#10;development of robust attribution methods that highlight the relative&#10;importance of pixels of an input image or feature map. In this work, we discuss&#10;the limitations of existing pruning heuristics, among which magnitude and&#10;gradient-based methods. We draw inspiration from attribution methods to design&#10;a novel integrated gradient pruning criterion, in which the relevance of each&#10;neuron is defined as the integral of the gradient variation on a path towards&#10;this neuron removal. Furthermore, we propose an entwined DNN pruning and&#10;fine-tuning flowchart to better preserve DNN accuracy while removing&#10;parameters. We show through extensive validation on several datasets,&#10;architectures as well as pruning scenarios that the proposed method, dubbed&#10;SInGE, significantly outperforms existing state-of-the-art DNN pruning methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.00708" label="2108.00708">
        <attvalues>
          <attvalue for="0" value="Group Fisher Pruning for Practical Network Compression" />
          <attvalue for="1" value="  Network compression has been widely studied since it is able to reduce the&#10;memory and computation cost during inference. However, previous methods seldom&#10;deal with complicated structures like residual connections, group/depth-wise&#10;convolution and feature pyramid network, where channels of multiple layers are&#10;coupled and need to be pruned simultaneously. In this paper, we present a&#10;general channel pruning approach that can be applied to various complicated&#10;structures. Particularly, we propose a layer grouping algorithm to find coupled&#10;channels automatically. Then we derive a unified metric based on Fisher&#10;information to evaluate the importance of a single channel and coupled&#10;channels. Moreover, we find that inference speedup on GPUs is more correlated&#10;with the reduction of memory rather than FLOPs, and thus we employ the memory&#10;reduction of each channel to normalize the importance. Our method can be used&#10;to prune any structures including those with coupled channels. We conduct&#10;extensive experiments on various backbones, including the classic ResNet and&#10;ResNeXt, mobile-friendly MobileNetV2, and the NAS-based RegNet, both on image&#10;classification and object detection which is under-explored. Experimental&#10;results validate that our method can effectively prune sophisticated networks,&#10;boosting inference speed without sacrificing accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.16840" label="2402.16840">
        <attvalues>
          <attvalue for="0" value="MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT" />
          <attvalue for="1" value="  &quot;Bigger the better&quot; has been the predominant trend in recent Large Language&#10;Models (LLMs) development. However, LLMs do not suit well for scenarios that&#10;require on-device processing, energy efficiency, low memory footprint, and&#10;response efficiency. These requisites are crucial for privacy, security, and&#10;sustainable deployment. This paper explores the &quot;less is more&quot; paradigm by&#10;addressing the challenge of designing accurate yet efficient Small Language&#10;Models (SLMs) for resource constrained devices. Our primary contribution is the&#10;introduction of an accurate and fully transparent open-source 0.5 billion&#10;(0.5B) parameter SLM, named MobiLlama, catering to the specific needs of&#10;resource-constrained computing with an emphasis on enhanced performance with&#10;reduced resource demands. MobiLlama is a SLM design that initiates from a&#10;larger model and applies a careful parameter sharing scheme to reduce both the&#10;pre-training and the deployment cost. Our work strives to not only bridge the&#10;gap in open-source SLMs but also ensures full transparency, where complete&#10;training data pipeline, training code, model weights, and over 300 checkpoints&#10;along with evaluation codes is available at :&#10;https://github.com/mbzuai-oryx/MobiLlama.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.11923" label="2310.11923">
        <attvalues>
          <attvalue for="0" value="Investigating semantic subspaces of Transformer sentence embeddings&#10;  through linear structural probing" />
          <attvalue for="1" value="  The question of what kinds of linguistic information are encoded in different&#10;layers of Transformer-based language models is of considerable interest for the&#10;NLP community. Existing work, however, has overwhelmingly focused on word-level&#10;representations and encoder-only language models with the masked-token training&#10;objective. In this paper, we present experiments with semantic structural&#10;probing, a method for studying sentence-level representations via finding a&#10;subspace of the embedding space that provides suitable task-specific pairwise&#10;distances between data-points. We apply our method to language models from&#10;different families (encoder-only, decoder-only, encoder-decoder) and of&#10;different sizes in the context of two tasks, semantic textual similarity and&#10;natural-language inference. We find that model families differ substantially in&#10;their performance and layer dynamics, but that the results are largely&#10;model-size invariant.&#10;" />
          <attvalue for="2" value="&#10;&#10;It is more or less generally assumed that the success of bidirectional masked language models (MLMs), such as &#10;BERT \cite{devlin-etal-2019-bert}, on downstream tasks is largely due to the fact that in pre-training they &#10;learn to compute rich and well-structured representations of their inputs. &#10;More precisely, it is often argued that the task of masked language modelling encourages models to &#10;successively aggregate lexical/collocational, syntactic, and semantic information from the input text &#10;as the activations progress through encoder layers \cite{tenney-etal-2019-bert}. The extent to which BERT-like &#10;models follow the stages of the classical NLP pipeline \cite{niu-etal-2022-bert} or &#10;accumulate contextual information \cite{kunz-kuhlmann-2022-linguistic} has been questioned. &#10;However, the association of middle layers of MLMs with syntax and higher levels with semantic information is not &#10;widely disputed as a general principle and is taken for granted in many papers on model analysis and &#10;downstream applications \cite{chi-etal-2020-finding,li-etal-2021-bert,sharma2022exploratory}.&#10;&#10;Despite the high volume of literature on Transformer representations, these studies are mostly constrained&#10;in two ways: (i)~they deal almost exclusively with word/token-level and not sentence-level embeddings,&#10;and (ii)~the lion's share of attention is given to encoder-only MLMs, notably variants of BERT &#10;\cite[cf.][]{reif2019visualizing,hewitt-manning-2019-structural,vulic-etal-2020-probing,conia-navigli-2022-probing}.&#10;As a result, the representations computed by text-to-text models, such as T5, and causal language&#10;models, such as members of the GPT family, remain understudied. This can be largely attributed to the &#10;fact that the standard way of analyzing pretrained language models, namely probing, proceeds by &#10;applying linear classifiers to token representations at different layers \cite{belinkov-2022-probing}. This &#10;approach is not as easily applicable to MLM-derived sentence representations or to representations &#10;computed by other types of models.&#10;Conversely, while it is possible to provide nuanced analyses &#10;of causal Transformer-based models \cite{geva-etal-2021-transformer,geva-etal-2022-transformer}, &#10;such analyses are not easily transferable to MLMs \cite{nikolaev2023universe}. &#10;&#10;In this study, we propose&#10;a unified methodology for studying layer-specific sentence-level representations extracted from masked, &#10;text-to-text, and causal language models. We &#10;analyze these representations via structural semantic probing, largely inspired by &#10;\cite{chi-etal-2020-finding}. Instead of directly predicting features of interest from &#10;representations, structural probing projects them onto lower-dimensional subspaces where&#10;distances are interpretable in terms of task properties, or where different classes&#10;of data points are directly linearly separable. By varying the dimensionality of the projection space, &#10;we can gauge the amount of information contained in the embeddings.&#10;&#10;While \cite{chi-etal-2020-finding} identify well-structured syntactic subspaces, &#10;i.e.\ those encoding the topology and labels of Universal Dependency trees, we target sentence-level &#10;semantic subspaces and carry out experiments on two semantic tasks, viz.\ sentence similarity &#10;and natural language inference (NLI). Our contributions are as follows:&#10;&#10;\begin{enumerate}&#10; \item We analyse the efficiency of solving different semantics-level downstream tasks using &#10; only suitably projected sentence embeddings derived from vanilla pre-trained encoder-only, &#10; encoder-decoder, and decoder-only models.&#10; \item We conduct an extensive analysis of the informativeness of embeddings derived from different &#10; model layers using varying dimensionalities of projection subspaces. &#10; Many of the models we study have never been analysed in this way, and we find that&#10; their behaviour is influenced in interesting ways by both architecture and training regime.&#10; \item We conduct our experiments at widely differing model scales: from BERT base, T5 mini, and OPT 125m to &#10; T5 XXL, Llama 13B, and OPT 66b. Our main finding is that the way information is structured across layers&#10; is largely scale invariant, with models sharing the same architecture and &#10; training regime demonstrating similar activation patterns.&#10; \item We show that three major NLI datasets -- SNLI, MNLI, and ANLI -- lead to very different results&#10; when tackled with projected vanilla embeddings. While SNLI and MNLI, surprisingly, can be almost&#10; \enquote{solved} with most vanilla models, ANLI, in contrast, is nearly completely opaque, &#10; and only embeddings from the biggest models are useful there.&#10;\end{enumerate}&#10;The structure of the paper is as follows: \S\ 2 introduces structural probing and its application; \S~3 lays out our experimental setup; \S~4 presents and discusses our findings, and \S~5 &#10;concludes.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Semantic Representation, Language Model Analysis, Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2105.07452" label="2105.07452">
        <attvalues>
          <attvalue for="0" value="How is BERT surprised? Layerwise detection of linguistic anomalies" />
          <attvalue for="1" value="  Transformer language models have shown remarkable ability in detecting when a&#10;word is anomalous in context, but likelihood scores offer no information about&#10;the cause of the anomaly. In this work, we use Gaussian models for density&#10;estimation at intermediate layers of three language models (BERT, RoBERTa, and&#10;XLNet), and evaluate our method on BLiMP, a grammaticality judgement benchmark.&#10;In lower layers, surprisal is highly correlated to low token frequency, but&#10;this correlation diminishes in upper layers. Next, we gather datasets of&#10;morphosyntactic, semantic, and commonsense anomalies from psycholinguistic&#10;studies; we find that the best performing model RoBERTa exhibits surprisal in&#10;earlier layers when the anomaly is morphosyntactic than when it is semantic,&#10;while commonsense anomalies do not exhibit surprisal at any intermediate layer.&#10;These results suggest that language models employ separate mechanisms to detect&#10;different types of linguistic anomalies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.10200" label="2204.10200">
        <attvalues>
          <attvalue for="0" value="An Exploratory Study on Code Attention in BERT" />
          <attvalue for="1" value="  Many recent models in software engineering introduced deep neural models&#10;based on the Transformer architecture or use transformer-based Pre-trained&#10;Language Models (PLM) trained on code. Although these models achieve the state&#10;of the arts results in many downstream tasks such as code summarization and bug&#10;detection, they are based on Transformer and PLM, which are mainly studied in&#10;the Natural Language Processing (NLP) field. The current studies rely on the&#10;reasoning and practices from NLP for these models in code, despite the&#10;differences between natural languages and programming languages. There is also&#10;limited literature on explaining how code is modeled.&#10;  Here, we investigate the attention behavior of PLM on code and compare it&#10;with natural language. We pre-trained BERT, a Transformer based PLM, on code&#10;and explored what kind of information it learns, both semantic and syntactic.&#10;We run several experiments to analyze the attention values of code constructs&#10;on each other and what BERT learns in each layer. Our analyses show that BERT&#10;pays more attention to syntactic entities, specifically identifiers and&#10;separators, in contrast to the most attended token [CLS] in NLP. This&#10;observation motivated us to leverage identifiers to represent the code sequence&#10;instead of the [CLS] token when used for code clone detection. Our results show&#10;that employing embeddings from identifiers increases the performance of BERT by&#10;605% and 4% F1-score in its lower layers and the upper layers, respectively.&#10;When identifiers' embeddings are used in CodeBERT, a code-based PLM, the&#10;performance is improved by 21-24% in the F1-score of clone detection. The&#10;findings can benefit the research community by using code-specific&#10;representations instead of applying the common embeddings used in NLP, and open&#10;new directions for developing smaller models with similar performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.12452" label="2102.12452">
        <attvalues>
          <attvalue for="0" value="Probing Classifiers: Promises, Shortcomings, and Advances" />
          <attvalue for="1" value="  Probing classifiers have emerged as one of the prominent methodologies for&#10;interpreting and analyzing deep neural network models of natural language&#10;processing. The basic idea is simple -- a classifier is trained to predict some&#10;linguistic property from a model's representations -- and has been used to&#10;examine a wide variety of models and properties. However, recent studies have&#10;demonstrated various methodological limitations of this approach. This article&#10;critically reviews the probing classifiers framework, highlighting their&#10;promises, shortcomings, and advances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.12094" label="1910.12094">
        <attvalues>
          <attvalue for="0" value="Meta Learning for End-to-End Low-Resource Speech Recognition" />
          <attvalue for="1" value="  In this paper, we proposed to apply meta learning approach for low-resource&#10;automatic speech recognition (ASR). We formulated ASR for different languages&#10;as different tasks, and meta-learned the initialization parameters from many&#10;pretraining languages to achieve fast adaptation on unseen target language, via&#10;recently proposed model-agnostic meta learning algorithm (MAML). We evaluated&#10;the proposed approach using six languages as pretraining tasks and four&#10;languages as target tasks. Preliminary results showed that the proposed method,&#10;MetaASR, significantly outperforms the state-of-the-art multitask pretraining&#10;approach on all target languages with different combinations of pretraining&#10;languages. In addition, since MAML's model-agnostic property, this paper also&#10;opens new research direction of applying meta learning to more speech-related&#10;applications.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the recent advances of deep learning, integrating the main modules of automatic speech recognition (ASR) such as acoustic model, pronunciation lexicon, and language model into a single end-to-end model is highly attractive. Connectionist Temporal Classification (CTC) \cite{graves2006connectionist} lends itself on such end-to-end approaches by introducing an additional blank symbol and specifically-designed loss function optimizing to generate the correct character sequences from the speech signal directly, without framewise phoneme alignment in advance. With many recent results \cite{hannun2014deep, amodei2016deep, collobert2016wav2letter}, end-to-end deep learning has created a larger interest in the speech community.&#10;&#10;However, to build such an end-to-end ASR system requires a huge amount of paired speech-transcription data, which is costly. &#10;For most languages in the world, they lack sufficient paired data for training. &#10;Pretraining on other language sources as the initialization, then fine-tuning on target language is the dominant approach under such low-resource setting, also known as multilingual transfer learning / pretraining (MultiASR) \cite{vu2014multilingual, tong2017investigation}. &#10;The backbone of MultiASR is a multitask model with shared hidden layers (encoder), and many language-specific heads. &#10;The model structure is designed to learn an encoder to extract language-independent representations to build a better acoustic model from many source languages. &#10;The success of ``language-independent'' features to improve ASR performance compared to monolingual training has been shown in many recent works \cite{cho2018multilingual, dalmia2018sequence, tong2017multilingual}.&#10;&#10;Besides directly training the model with all the source languages, there are various variants of MultiASR approaches. &#10;Language-adversarial training approaches \cite{Yi2018AdversarialMT, adams2019massively} introduced language-adversarial classification objective to the shared encoder, negating the gradients backpropagated from the language classifier to encourage the encoder to extract more language-independent representations. &#10;Hierarchical approaches \cite{Sanabria2018HierarchicalMT} introduced different granularity objectives by combining both character and phoneme prediction at different levels of the model.&#10;&#10;In this paper, we provide a novel research direction following up on the idea of multilingual pretraining -- Meta learning. &#10;Meta learning, or learning-to-learn, has recently received considerable interest in the machine learning community. The goal of meta learning is to solve the problem of ``fast adaptation on unseen data'', which is aligned with our low-resource setting.&#10; With its success in computer vision under the few-shot learning setting~\cite{rusu2018meta, snell2017prototypical, vinyals2016matching}, there have been some works in language and speech processing, for instance, language transfer in neural machine translation \cite{gu2018meta}, dialogue generation \cite{mi2019meta}, and speaker adaptive training \cite{klejch2018learning}, but not multilingual pretraining for speech recognition.&#10;&#10;We use model-agnostic meta-learning algorithm (MAML) \cite{finn2017model} in this work. &#10;As its name suggestes, MAML can be applied to any network architecture. &#10;MAML only modifies the optimization process following meta learning training scheme.&#10;It does not introduce additional modules like adversarial training or requires phoneme level annotation (usually through lexicon) such as hierarchical approaches. &#10;We evaluated the effectiveness of the proposed meta learning algorithm, MetaASR, on the IARPA BABEL dataset \cite{gales2014speech}. Our experiments reveal that MetaASR outperforms MultiASR significantly across all target languages.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Meta Learning, Linguistics, Multitask Pretraining, Artificial Intelligence, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1412.5567" label="1412.5567">
        <attvalues>
          <attvalue for="0" value="Deep Speech: Scaling up end-to-end speech recognition" />
          <attvalue for="1" value="  We present a state-of-the-art speech recognition system developed using&#10;end-to-end deep learning. Our architecture is significantly simpler than&#10;traditional speech systems, which rely on laboriously engineered processing&#10;pipelines; these traditional systems also tend to perform poorly when used in&#10;noisy environments. In contrast, our system does not need hand-designed&#10;components to model background noise, reverberation, or speaker variation, but&#10;instead directly learns a function that is robust to such effects. We do not&#10;need a phoneme dictionary, nor even the concept of a &quot;phoneme.&quot; Key to our&#10;approach is a well-optimized RNN training system that uses multiple GPUs, as&#10;well as a set of novel data synthesis techniques that allow us to efficiently&#10;obtain a large amount of varied data for training. Our system, called Deep&#10;Speech, outperforms previously published results on the widely studied&#10;Switchboard Hub5'00, achieving 16.0% error on the full test set. Deep Speech&#10;also handles challenging noisy environments better than widely used,&#10;state-of-the-art commercial speech systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.07104" label="1807.07104">
        <attvalues>
          <attvalue for="0" value="Hierarchical Multi Task Learning With CTC" />
          <attvalue for="1" value="  In Automatic Speech Recognition it is still challenging to learn useful&#10;intermediate representations when using high-level (or abstract) target units&#10;such as words. For that reason, character or phoneme based systems tend to&#10;outperform word-based systems when just few hundreds of hours of training data&#10;are being used. In this paper, we first show how hierarchical multi-task&#10;training can encourage the formation of useful intermediate representations. We&#10;achieve this by performing Connectionist Temporal Classification at different&#10;levels of the network with targets of different granularity. Our model thus&#10;performs predictions in multiple scales for the same input. On the standard&#10;300h Switchboard training setup, our hierarchical multi-task architecture&#10;exhibits improvements over single-task architectures with the same number of&#10;parameters. Our model obtains 14.0% Word Error Rate on the Eval2000 Switchboard&#10;subset without any decoder or language model, outperforming the current&#10;state-of-the-art on acoustic-to-word models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.05960" label="1807.05960">
        <attvalues>
          <attvalue for="0" value="Meta-Learning with Latent Embedding Optimization" />
          <attvalue for="1" value="  Gradient-based meta-learning techniques are both widely applicable and&#10;proficient at solving challenging few-shot learning and fast adaptation&#10;problems. However, they have practical difficulties when operating on&#10;high-dimensional parameter spaces in extreme low-data regimes. We show that it&#10;is possible to bypass these limitations by learning a data-dependent latent&#10;generative representation of model parameters, and performing gradient-based&#10;meta-learning in this low-dimensional latent space. The resulting approach,&#10;latent embedding optimization (LEO), decouples the gradient-based adaptation&#10;procedure from the underlying high-dimensional space of model parameters. Our&#10;evaluation shows that LEO can achieve state-of-the-art performance on the&#10;competitive miniImageNet and tieredImageNet few-shot classification tasks.&#10;Further analysis indicates LEO is able to capture uncertainty in the data, and&#10;can perform adaptation more effectively by optimizing in latent space.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.08437" label="1808.08437">
        <attvalues>
          <attvalue for="0" value="Meta-Learning for Low-Resource Neural Machine Translation" />
          <attvalue for="1" value="  In this paper, we propose to extend the recently introduced model-agnostic&#10;meta-learning algorithm (MAML) for low-resource neural machine translation&#10;(NMT). We frame low-resource translation as a meta-learning problem, and we&#10;learn to adapt to low-resource languages based on multilingual high-resource&#10;language tasks. We use the universal lexical&#10;representation~\citep{gu2018universal} to overcome the input-output mismatch&#10;across different languages. We evaluate the proposed meta-learning strategy&#10;using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt,&#10;Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro,&#10;Lv, Fi, Tr and Ko) as target tasks. We show that the proposed approach&#10;significantly outperforms the multilingual, transfer learning based&#10;approach~\citep{zoph2016transfer} and enables us to train a competitive NMT&#10;system with only a fraction of training examples. For instance, the proposed&#10;approach can achieve as high as 22.04 BLEU on Romanian-English WMT'16 by seeing&#10;only 16,000 translated words (~600 parallel sentences).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.05644" label="1905.05644">
        <attvalues>
          <attvalue for="0" value="Meta-Learning for Low-resource Natural Language Generation in&#10;  Task-oriented Dialogue Systems" />
          <attvalue for="1" value="  Natural language generation (NLG) is an essential component of task-oriented&#10;dialogue systems. Despite the recent success of neural approaches for NLG, they&#10;are typically developed for particular domains with rich annotated training&#10;examples. In this paper, we study NLG in a low-resource setting to generate&#10;sentences in new scenarios with handful training examples. We formulate the&#10;problem from a meta-learning perspective, and propose a generalized&#10;optimization-based approach (Meta-NLG) based on the well-recognized&#10;model-agnostic meta-learning (MAML) algorithm. Meta-NLG defines a set of meta&#10;tasks, and directly incorporates the objective of adapting to new low-resource&#10;NLG tasks into the meta-learning optimization process. Extensive experiments&#10;are conducted on a large multi-domain dataset (MultiWoz) with diverse&#10;linguistic variations. We show that Meta-NLG significantly outperforms other&#10;training procedures in various low-resource configurations. We analyze the&#10;results, and demonstrate that Meta-NLG adapts extremely fast and well to&#10;low-resource situations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.10239" label="1808.10239">
        <attvalues>
          <attvalue for="0" value="Learning to adapt: a meta-learning approach for speaker adaptation" />
          <attvalue for="1" value="  The performance of automatic speech recognition systems can be improved by&#10;adapting an acoustic model to compensate for the mismatch between training and&#10;testing conditions, for example by adapting to unseen speakers. The success of&#10;speaker adaptation methods relies on selecting weights that are suitable for&#10;adaptation and using good adaptation schedules to update these weights in order&#10;not to overfit to the adaptation data. In this paper we investigate a&#10;principled way of adapting all the weights of the acoustic model using a&#10;meta-learning. We show that the meta-learner can learn to perform supervised&#10;and unsupervised speaker adaptation and that it outperforms a strong baseline&#10;adapting LHUC parameters when adapting a DNN AM with 1.5M parameters. We also&#10;report initial experiments on adapting TDNN AMs, where the meta-learner&#10;achieves comparable performance with LHUC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.04009" label="2409.04009">
        <attvalues>
          <attvalue for="0" value="Large Margin Prototypical Network for Few-shot Relation Classification&#10;  with Fine-grained Features" />
          <attvalue for="1" value="  Relation classification (RC) plays a pivotal role in both natural language&#10;understanding and knowledge graph completion. It is generally formulated as a&#10;task to recognize the relationship between two entities of interest appearing&#10;in a free-text sentence. Conventional approaches on RC, regardless of feature&#10;engineering or deep learning based, can obtain promising performance on&#10;categorizing common types of relation leaving a large proportion of&#10;unrecognizable long-tail relations due to insufficient labeled instances for&#10;training. In this paper, we consider few-shot learning is of great practical&#10;significance to RC and thus improve a modern framework of metric learning for&#10;few-shot RC. Specifically, we adopt the large-margin ProtoNet with fine-grained&#10;features, expecting they can generalize well on long-tail relations. Extensive&#10;experiments were conducted by FewRel, a large-scale supervised few-shot RC&#10;dataset, to evaluate our framework: LM-ProtoNet (FGF). The results demonstrate&#10;that it can achieve substantial improvements over many baseline approaches.&#10;" />
          <attvalue for="2" value="&#10;Relation classification (RC)~\cite{10.1007/978-981-10-7359-5_6} is a pivotal task for both natural language understanding (NLU)~\cite{hirschberg2015advances} and knowledge graph completion (KGC)~\cite{nickel2016review}. Given a free-text sentence, we can first adopt an off-the-shelf software (e.g., Stanford CoreNLP~\cite{P14-5010}) to discover entities of interest, and RC is a successive module which takes in charge of recognizing the relationship between each pair of the named entities. In this way, we can extract many triplets, denoted as ({head-entity, relation, tail-entity}), from free-text sentences. These triplets can facilitate machines understanding the inner structure of natural language and even extending knowledge bases. &#10;&#10;A great number of approaches have been proposed in the past decades for the task of sentence-level RC, including feature engineering methods~\cite{zelenko2003kernel,culotta2004dependency} and neural learning models~\cite{Sun:2018:LUE:3159652.3159712,sun-etal-2018-logician}. Regardless of those ways of producing various evidence for RC, conventional approaches mainly rely on a large number of labeled instances for each type of relation in the training phase, and the promising performance of RC is merely obtained on common relations. Many manually annotated corpora of RC such as MUC-7 and ACE can only cover $1\%$-$2\%$ of the relations recorded by some large-scale knowledge bases, such as Freebase~\cite{bollacker2008freebase}, DBpedia~\cite{lehmann2015dbpedia} and Wikidata~\cite{42240} where thousands of relations are included. &#10;&#10;To alleviate the issue of insufficient labeled instances for long-tail relations, \cite{P09-1113} proposed the distant supervision paradigm to automatically label free-text sentences with the relations from an existing knowledge base by some heuristic alignment rules to build training data. It, however, naturally suffers from the problem of incorrect annotations.&#10;&#10;The emerging study on few-shot learning~\cite{Fei-Fei:2006:OLO:1115692.1115783,chen19closerfewshot} inspires us that we can train an RC model with the instances labeled by common relations and transfer the knowledge to infer more sentences that may express long-tail relations. Few-shot RC models are expected to obtain considerable accuracy supported by very few instances annotated by long-tail relations without learning from scratch. For example, Table~\ref{tab:example} shows an example of a $5$-way-$1$-shot scenario of RC, where we can leverage a few-shot RC model $\mathcal{M}$ to tell whether the query instance/sentence mentions any type of the $5$ relations given by the support set ($1$ shot/instance for each relation). What makes few-shot RC unique is that the model $\mathcal{M}$ is trained by the instances labeled by other relations where the $5$ relations are not included. &#10;&#10;To attract more successive studies on few-shot RC, \cite{han2018fewrel} constructed a large-scale supervised few-shot relation classification dataset (FewRel)&#10;for the purpose of evaluating the performance of various meta-learning approaches, including Meta Network~\cite{munkhdalai2017meta}, GNN~\cite{garcia2018fewshot}, SNAIL~\cite{mishra2018a} and ProtoNet~\cite{NIPS2017_6996}, on RC. They reported that the CNN-based~\cite{C14-1220} ProtoNet outperforms the other baseline methods. However, we consider the framework can be further improved either by task-specific features or by advanced learning targets.&#10;&#10;In this paper, we contribute two updates to the original CNN-based~\cite{C14-1220} ProtoNet from the perspectives of fine-grained feature generation and large-margin learning, respectively, aiming at increasing the generalization ability of few-shot RC models on recognizing long-tail relations. Further experiments were also conducted on FewRel, and the results demonstrate that our framework, i.e. LM-ProtoNet (FGF), attains a leading performance over many baselines with a substantial improvement by 6.84\% accuracy.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Machine Learning, Computer Science, Knowledge Graph Completion, Linguistics, Natural Language Understanding, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1810.10147" label="1810.10147">
        <attvalues>
          <attvalue for="0" value="FewRel: A Large-Scale Supervised Few-Shot Relation Classification&#10;  Dataset with State-of-the-Art Evaluation" />
          <attvalue for="1" value="  We present a Few-Shot Relation Classification Dataset (FewRel), consisting of&#10;70, 000 sentences on 100 relations derived from Wikipedia and annotated by&#10;crowdworkers. The relation of each sentence is first recognized by distant&#10;supervision methods, and then filtered by crowdworkers. We adapt the most&#10;recent state-of-the-art few-shot learning methods for relation classification&#10;and conduct a thorough evaluation of these methods. Empirical results show that&#10;even the most competitive few-shot learning models struggle on this task,&#10;especially as compared with humans. We also show that a range of different&#10;reasoning skills are needed to solve our task. These results indicate that&#10;few-shot relation classification remains an open problem and still requires&#10;further research. Our detailed analysis points multiple directions for future&#10;research. All details and resources about the dataset and baselines are&#10;released on http://zhuhao.me/fewrel.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.04043" label="1711.04043">
        <attvalues>
          <attvalue for="0" value="Few-Shot Learning with Graph Neural Networks" />
          <attvalue for="1" value="  We propose to study the problem of few-shot learning with the prism of&#10;inference on a partially observed graphical model, constructed from a&#10;collection of input images whose label can be either observed or not. By&#10;assimilating generic message-passing inference algorithms with their&#10;neural-network counterparts, we define a graph neural network architecture that&#10;generalizes several of the recently proposed few-shot learning models. Besides&#10;providing improved numerical performance, our framework is easily extended to&#10;variants of few-shot learning, such as semi-supervised or active learning,&#10;demonstrating the ability of graph-based models to operate well on 'relational'&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.03141" label="1707.03141">
        <attvalues>
          <attvalue for="0" value="A Simple Neural Attentive Meta-Learner" />
          <attvalue for="1" value="  Deep neural networks excel in regimes with large amounts of data, but tend to&#10;struggle when data is scarce or when they need to adapt quickly to changes in&#10;the task. In response, recent work in meta-learning proposes training a&#10;meta-learner on a distribution of similar tasks, in the hopes of generalization&#10;to novel but related tasks by learning a high-level strategy that captures the&#10;essence of the problem it is asked to solve. However, many recent meta-learning&#10;approaches are extensively hand-designed, either using architectures&#10;specialized to a particular application, or hard-coding algorithmic components&#10;that constrain how the meta-learner solves the task. We propose a class of&#10;simple and generic meta-learner architectures that use a novel combination of&#10;temporal convolutions and soft attention; the former to aggregate information&#10;from past experience and the latter to pinpoint specific pieces of information.&#10;In the most extensive set of meta-learning experiments to date, we evaluate the&#10;resulting Simple Neural AttentIve Learner (or SNAIL) on several&#10;heavily-benchmarked tasks. On all tasks, in both supervised and reinforcement&#10;learning, SNAIL attains state-of-the-art performance by significant margins.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.15195" label="2211.15195">
        <attvalues>
          <attvalue for="0" value="Distance Metric Learning Loss Functions in Few-Shot Scenarios of&#10;  Supervised Language Models Fine-Tuning" />
          <attvalue for="1" value="  This paper presents an analysis regarding an influence of the Distance Metric&#10;Learning (DML) loss functions on the supervised fine-tuning of the language&#10;models for classification tasks. We experimented with known datasets from&#10;SentEval Transfer Tasks.&#10;  Our experiments show that applying the DML loss function can increase&#10;performance on downstream classification tasks of RoBERTa-large models in&#10;few-shot scenarios. Models fine-tuned with the use of SoftTriple loss can&#10;achieve better results than models with a standard categorical cross-entropy&#10;loss function by about 2.89 percentage points from 0.04 to 13.48 percentage&#10;points depending on the training dataset. Additionally, we accomplished a&#10;comprehensive analysis with explainability techniques to assess the models'&#10;reliability and explain their results.&#10;" />
          <attvalue for="2" value="&#10;&#10;The development of new techniques in the Natural Language Processing (NLP) field has been studied over the last few years. It resulted in a few breakthroughs, which constantly stimulated waves of interest in the text processing area. The most recent discoveries are based on the Transformer architecture that enabled capturing the semantic meaning of the sentence~\cite{vaswani2017attention}. The Transformer architecture facilitated the discovery of the BERT encoder~\cite{devlin2018bert} that nowadays is massively used to solve most NLP tasks by adapting it in the fine-tuning process.&#10;&#10;Unfortunately, fine-tuning of pre-trained models has a number of flaws. First of all, it is not designed to perform well when the number of observations is limited, so large training sets are consistently required for models to perform well~\cite{bansal2019learning}. Secondly, the fine-tuning process happens to be very unstable across different runs with different seeds, even though just a few minor components of the learning process are dependent on random seeds~\cite{zhang2020revisiting}. The lack of stability is even more exacerbated in the case of few-shot learning scenarios.&#10;&#10;On the other hand, pre-trained models are fine-tuned for a specific task using a cross-entropy objective function that focuses on learning class-specific features rather than their class representations. In other words, it only encourages inter-class distances and is not taking care of minimizing intra-class distances that would result in learning discriminative features~\cite{wen2016discriminative}. This leads to a poor generalization of the model and thus causes problems with noisy or outlier data~\cite{cao2019learning}. &#10;&#10;However, a few research studies proposed other loss functions, i.e. Distance Metric Learning (DML) function family, which addresses the problems of the cross-entropy loss. The DML losses are meant to push representations of observations together if they belong to the same class and separate those from different classes. We believe this will be particularly helpful for few-shot learning, where the number of observations is insufficient for a suitable ordering of the embedding space. Therefore, we decided to investigate the effect of using loss functions from the DML family on fine-tuning the BERT-based encoder for downstream tasks in few-shot learning scenarios.&#10;&#10;Our main contributions are the following:&#10;\begin{enumerate}&#10; \item We apply SoftTriple loss in the few-shot scenarios for the supervised fine-tuning language model in NLP domain.&#10; \item We examine the effect of using loss functions from the DML family on the supervised fine-tuning of the RoBERTa-large language model.&#10; \item We establish that SoftTriple loss is more efficient than Supervised Contrastive loss for the supervised fine-tuning RoBERTa-large language model.&#10; \item After thorough experiments, our finding is that applying the DML loss to the RoBERTa-large language model is more viable the smaller the training set.&#10; \item Additionally, to deeper understand the models and check their reliability we propose a comprehensive analysis and explainability techniques.\end{enumerate}&#10;&#10;The following sectionis dedicated to a brief overview of the DML methods. Our new method is outlined in Section~\ref{sec:method}. The next sectionprovides a performance analysis of the models and investigates their behaviour. Finally, a summary of the experiments is described in the last section.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Classification Task Performance, Mathematics, Statistics, Language Model Fine-tuning, Distance Metric Learning" />
        </attvalues>
      </node>
      <node id="1911.03863" label="1911.03863">
        <attvalues>
          <attvalue for="0" value="Learning to Few-Shot Learn Across Diverse Natural Language&#10;  Classification Tasks" />
          <attvalue for="1" value="  Self-supervised pre-training of transformer models has shown enormous success&#10;in improving performance on a number of downstream tasks. However, fine-tuning&#10;on a new task still requires large amounts of task-specific labelled data to&#10;achieve good performance. We consider this problem of learning to generalize to&#10;new tasks with few examples as a meta-learning problem. While meta-learning has&#10;shown tremendous progress in recent years, its application is still limited to&#10;simulated problems or problems with limited diversity across tasks. We develop&#10;a novel method, LEOPARD, which enables optimization-based meta-learning across&#10;tasks with different number of classes, and evaluate different methods on&#10;generalization to diverse NLP classification tasks. LEOPARD is trained with the&#10;state-of-the-art transformer architecture and shows better generalization to&#10;tasks not seen at all during training, with as few as 4 examples per label.&#10;Across 17 NLP tasks, including diverse domains of entity typing, natural&#10;language inference, sentiment analysis, and several other text classification&#10;tasks, we show that LEOPARD learns better initial parameters for few-shot&#10;learning than self-supervised pre-training or multi-task training,&#10;outperforming many strong baselines, for example, yielding 14.5% average&#10;relative gain in accuracy on unseen tasks with only 4 examples per label.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.07413" label="1906.07413">
        <attvalues>
          <attvalue for="0" value="Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss" />
          <attvalue for="1" value="  Deep learning algorithms can fare poorly when the training dataset suffers&#10;from heavy class-imbalance but the testing criterion requires good&#10;generalization on less frequent classes. We design two novel methods to improve&#10;performance in such scenarios. First, we propose a theoretically-principled&#10;label-distribution-aware margin (LDAM) loss motivated by minimizing a&#10;margin-based generalization bound. This loss replaces the standard&#10;cross-entropy objective during training and can be applied with prior&#10;strategies for training with class-imbalance such as re-weighting or&#10;re-sampling. Second, we propose a simple, yet effective, training schedule that&#10;defers re-weighting until after the initial stage, allowing the model to learn&#10;an initial representation while avoiding some of the complications associated&#10;with re-weighting or re-sampling. We test our methods on several benchmark&#10;vision tasks including the real-world imbalanced dataset iNaturalist 2018. Our&#10;experiments show that either of these methods alone can already improve over&#10;existing techniques and their combination achieves even better performance&#10;gains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.07096" label="2406.07096">
        <attvalues>
          <attvalue for="0" value="Fast Context-Biasing for CTC and Transducer ASR models with CTC-based&#10;  Word Spotter" />
          <attvalue for="1" value="  Accurate recognition of rare and new words remains a pressing problem for&#10;contextualized Automatic Speech Recognition (ASR) systems. Most context-biasing&#10;methods involve modification of the ASR model or the beam-search decoding&#10;algorithm, complicating model reuse and slowing down inference. This work&#10;presents a new approach to fast context-biasing with CTC-based Word Spotter&#10;(CTC-WS) for CTC and Transducer (RNN-T) ASR models. The proposed method matches&#10;CTC log-probabilities against a compact context graph to detect potential&#10;context-biasing candidates. The valid candidates then replace their greedy&#10;recognition counterparts in corresponding frame intervals. A Hybrid&#10;Transducer-CTC model enables the CTC-WS application for the Transducer model.&#10;The results demonstrate a significant acceleration of the context-biasing&#10;recognition with a simultaneous improvement in F-score and WER compared to&#10;baseline methods. The proposed method is publicly available in the NVIDIA NeMo&#10;toolkit.&#10;" />
          <attvalue for="2" value="&#10;&#10;ASR models often struggle to recognize words that were absent or had few examples in the training data. Context-biasing methods attempt to solve this problem by assuming that we have a list of words and phrases (context-biasing list) in advance for which we want to improve recognition accuracy.&#10;&#10;One of the directions of context-biasing methods is based on the ``deep fusion''. These methods require intervention into the ASR model and its training process. In this case, the context-biasing list is supplied to the encoder or decoder via a cross-attention mechanism as a vector of an entire word \cite{Pundak2018DeepCE, Jain2020ContextualRF, Yang2023PromptASRFC} or a token from context trie \cite{Le2021ContextualizedSE, Harding2023SelectiveBW}. There are also methods based on SpeechLM, when the context-biasing list is fed directly into the prompt for the LLM part of the model \cite{Wang2023SLMBT,Chen2023SALMSL}.&#10;&#10;Another direction is methods based on ``shallow fusion''. In this case, the only decoding process is modified. Initially, shallow fusion methods were applied to classic ASR systems by adding new words to the WFST decoding graph \cite{Dixon2012ASW,Hall2015CompositionbasedOR}. Shallow fusion methods are also used for End-to-End ASR. During the beam-search decoding, the hypothesis is re-scored depending on the presence of the current word in the context-biasing list \cite{Zhao2019ShallowFusionEC, Jung2021SpellMN, Galvez2023GPUAcceleratedWB}. It is also possible to combine an end-to-end ASR model with WFST to obtain context-biasing abilities of the classic models \cite{Fox2022ImprovingCR,Zhang2021TinyTA,Andrusenko2022ImprovingOO}. &#10;&#10;Despite the advantages of shallow fusion methods in model reuse, these approaches use beam-search decoding. Processing many alternative hypotheses leads to a significant decoding slowdown even for the Connectionist Temporal Classification (CTC) model~\cite{Graves2006ConnectionistTC}. This problem is considerably worsened in the case of the Transducer (RNN-T) model \cite{Graves2012SequenceTW} since beam-search decoding involves multiple Decoder (Prediction) and Joint networks calculations. Moreover, the context-biasing recognition is limited by the model prediction pool biased toward training data. In the case of rare or new words, the model may not have a hypothesis for the desired word from the context-biasing list whose probability we want to amplify.&#10;&#10;This work presents a new fast context-biasing method using a CTC-based Word Spotter called CTC-WS (Figure~\ref{fig:ctcws_scheme}).&#10;The method involves decoding CTC log-probabilities with a context graph built for words and phrases from the context-biasing list. The spotted context-biasing candidates (with their scores and time intervals) are compared by scores with words from the greedy CTC decoding results to improve recognition accuracy and reduce false-positive errors of context-biasing (Figure~\ref{fig:CTC-WS}).&#10;&#10;We also propose a method of improving the recognition accuracy of abbreviations and complicated words with alternative transcriptions inspired by \cite{Fox2022ImprovingCR}, but obtained automatically without preliminary speech recognition.&#10;&#10;A Hybrid Transducer-CTC model \cite{noroozi2024stateful} (a shared encoder trained together with CTC and Transducer output heads) enables the use of the CTC-WS method for the Transducer model. Context-biasing candidates obtained by CTC-WS are also filtered by the scores with greedy CTC predictions and then merged with greedy Transducer results. Compared to baseline shallow fusion methods, the CTC-WS demonstrates better WER and context-biasing word recognition and remarkably speeds up the decoding process for CTC and Transducer models.&#10;The proposed method is publicly available in the NVIDIA NeMo toolkit.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Electrical Engineering, Linguistics, Artificial Intelligence, Speech Recognition, Context Biasing" />
        </attvalues>
      </node>
      <node id="2310.09424" label="2310.09424">
        <attvalues>
          <attvalue for="0" value="SALM: Speech-augmented Language Model with In-context Learning for&#10;  Speech Recognition and Translation" />
          <attvalue for="1" value="  We present a novel Speech Augmented Language Model (SALM) with {\em&#10;multitask} and {\em in-context} learning capabilities. SALM comprises a frozen&#10;text LLM, a audio encoder, a modality adapter module, and LoRA layers to&#10;accommodate speech input and associated task instructions. The unified SALM not&#10;only achieves performance on par with task-specific Conformer baselines for&#10;Automatic Speech Recognition (ASR) and Speech Translation (AST), but also&#10;exhibits zero-shot in-context learning capabilities, demonstrated through&#10;keyword-boosting task for ASR and AST. Moreover, {\em speech supervised&#10;in-context training} is proposed to bridge the gap between LLM training and&#10;downstream speech tasks, which further boosts the in-context learning ability&#10;of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.01250" label="2209.01250">
        <attvalues>
          <attvalue for="0" value="Improving Contextual Recognition of Rare Words with an Alternate&#10;  Spelling Prediction Model" />
          <attvalue for="1" value="  Contextual ASR, which takes a list of bias terms as input along with audio,&#10;has drawn recent interest as ASR use becomes more widespread. We are releasing&#10;contextual biasing lists to accompany the Earnings21 dataset, creating a public&#10;benchmark for this task. We present baseline results on this benchmark using a&#10;pretrained end-to-end ASR model from the WeNet toolkit. We show results for&#10;shallow fusion contextual biasing applied to two different decoding algorithms.&#10;Our baseline results confirm observations that end-to-end models struggle in&#10;particular with words that are rarely or never seen during training, and that&#10;existing shallow fusion techniques do not adequately address this problem. We&#10;propose an alternate spelling prediction model that improves recall of rare&#10;words by 34.7% relative and of out-of-vocabulary words by 97.2% relative,&#10;compared to contextual biasing without alternate spellings. This model is&#10;conceptually similar to ones used in prior work, but is simpler to implement as&#10;it does not rely on either a pronunciation dictionary or an existing&#10;text-to-speech system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.17279" label="2312.17279">
        <attvalues>
          <attvalue for="0" value="Stateful Conformer with Cache-based Inference for Streaming Automatic&#10;  Speech Recognition" />
          <attvalue for="1" value="  In this paper, we propose an efficient and accurate streaming speech&#10;recognition model based on the FastConformer architecture. We adapted the&#10;FastConformer architecture for streaming applications through: (1) constraining&#10;both the look-ahead and past contexts in the encoder, and (2) introducing an&#10;activation caching mechanism to enable the non-autoregressive encoder to&#10;operate autoregressively during inference. The proposed model is thoughtfully&#10;designed in a way to eliminate the accuracy disparity between the train and&#10;inference time which is common for many streaming models. Furthermore, our&#10;proposed encoder works with various decoder configurations including&#10;Connectionist Temporal Classification (CTC) and RNN-Transducer (RNNT) decoders.&#10;Additionally, we introduced a hybrid CTC/RNNT architecture which utilizes a&#10;shared encoder with both a CTC and RNNT decoder to boost the accuracy and save&#10;computation. We evaluate the proposed model on LibriSpeech dataset and a&#10;multi-domain large scale dataset and demonstrate that it can achieve better&#10;accuracy with lower latency and inference time compared to a conventional&#10;buffered streaming model baseline. We also showed that training a model with&#10;multiple latencies can achieve better accuracy than single latency models while&#10;it enables us to support multiple latencies with a single model. Our&#10;experiments also showed the hybrid architecture would not only speedup the&#10;convergence of the CTC decoder but also improves the accuracy of streaming&#10;models compared to single decoder models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00699" label="2005.00699">
        <attvalues>
          <attvalue for="0" value="Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer" />
          <attvalue for="1" value="  Multilingual representations embed words from many languages into a single&#10;semantic space such that words with similar meanings are close to each other&#10;regardless of the language. These embeddings have been widely used in various&#10;settings, such as cross-lingual transfer, where a natural language processing&#10;(NLP) model trained on one language is deployed to another language. While the&#10;cross-lingual transfer techniques are powerful, they carry gender bias from the&#10;source to target languages. In this paper, we study gender bias in multilingual&#10;embeddings and how it affects transfer learning for NLP applications. We create&#10;a multilingual dataset for bias analysis and propose several ways for&#10;quantifying bias in multilingual representations from both the intrinsic and&#10;extrinsic perspectives. Experimental results show that the magnitude of bias in&#10;the multilingual representations changes differently when we align the&#10;embeddings to different target spaces and that the alignment direction can also&#10;have an influence on the bias in transfer learning. We further provide&#10;recommendations for using the multilingual word representations for downstream&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Gender Bias in Word Representations&#10;Word embeddings are widely used in different NLP applications. They represent words using low dimensional vectors. \cite{bolukbasi2016man} find that, in the embedding space, occupation words such as ``professor'' and ``nurse'' show discrepancy concerning the genders. Similarly, \cite{caliskan2017semantics} also reveal the gender stereotypes in the English word embeddings based on the Word Embedding Association Test (WEAT). However, both works only consider English and cannot be directly adapted to other languages such as Spanish. \cite{mccurdy2017grammatical} reveal that bias exists in languages with grammatical gender while \cite{zhou2019grammaticalgenderbias} and \cite{lauscher2019we} show that there is bias in bilingual word embeddings. However, none of them consider the cross-lingual transfer learning which is an important application of the multilingual word embeddings. &#10;To mitigate the bias in word embeddings, various approaches have been proposed~\cite{bolukbasi2016man,zhao2018learning}. In contrast to these methods in English embedding space, we propose to mitigate the bias from the multilingual perspectives. Comparing to \cite{zhou2019grammaticalgenderbias}, we show that a different choice of alignment target can help to reduce the bias in multilingual embeddings from both intrinsic and extrinsic perspectives.&#10;&#10;Multilingual Word Embeddings and Cross-lingual Transfer Learning&#10;Multilingual word embeddings represent words from different languages using the same embedding space which enables cross-lingual transfer learning~\cite{ruder2017survey}. The model is trained on a labeled data rich language and adopted to another language where no or a small portion of labeled data is available~\cite{duong2015low, guo2016representation}. To get the multilingual word embeddings, \cite{mikolov2013exploiting} learn a linear mapping between the source and target language. &#10;However, \cite{xing2015normalized} argue that there are some inconsistencies in directly learning the linear mapping. To solve those limitations, they constrain the embeddings to be normalized and enforce an orthogonal transformation. &#10;While those methods achieve reasonable results on benchmark datasets, they all suffer from the hubness problem which is solved by adding cross-domain similarity constraints~\cite{conneau2017word,joulin2018loss}.&#10;Our work is based on the multilingual word embeddings achieved by~\newcite{joulin2018loss}. Besides the commonly used multilingual word embeddings obtained by aligning all the embeddings to the English space, we also analyze the embeddings aligned to different target spaces.&#10;&#10;Bias in Other Applications Besides the bias in word embeddings, such issues have also been demonstrated in other applications, including named entity recognition~\cite{mehrabi2019man}, sentiment analysis~\cite{kiritchenko2018examining}, and natural language inferences~\cite{rudinger2017social}. However, those analyses are limited to English corpus and lack the insight of multilingual situations.&#10;&#10;" />
          <attvalue for="4" value="Gender Bias Analysis, Computer Science, Linguistics, Multilingual Representations, Natural Language Processing, Transfer Learning, Statistics" />
        </attvalues>
      </node>
      <node id="2401.08833" label="2401.08833">
        <attvalues>
          <attvalue for="0" value="Revisiting Self-supervised Learning of Speech Representation from a&#10;  Mutual Information Perspective" />
          <attvalue for="1" value="  Existing studies on self-supervised speech representation learning have&#10;focused on developing new training methods and applying pre-trained models for&#10;different applications. However, the quality of these models is often measured&#10;by the performance of different downstream tasks. How well the representations&#10;access the information of interest is less studied. In this work, we take a&#10;closer look into existing self-supervised methods of speech from an&#10;information-theoretic perspective. We aim to develop metrics using mutual&#10;information to help practical problems such as model design and selection. We&#10;use linear probes to estimate the mutual information between the target&#10;information and learned representations, showing another insight into the&#10;accessibility to the target information from speech representations. Further,&#10;we explore the potential of evaluating representations in a self-supervised&#10;fashion, where we estimate the mutual information between different parts of&#10;the data without using any labels. Finally, we show that both supervised and&#10;unsupervised measures echo the performance of the models on layer-wise linear&#10;probing and speech recognition.&#10;" />
          <attvalue for="2" value="&#10;&#10;Estimating the amount of information encoded in learned representations has been an important research topic in speech representation learning.&#10;A good estimation can not only offer a better view of designing training objectives, especially under a self-supervised paradigm but also help select models for the applications of interest. &#10;To measure to what extent the representations reveal specific information, several studies have adopted a phonetic-related linear probing protocol \cite{chung2020generative,yang2022autoregressive}.&#10;Also, a collection of downstream tasks has been proposed to evaluate learned representations, including phone classification and speech recognition \cite{yang2021superb}. The accuracy obtained from a task is then believed to reflect the accessibility of representations to certain information.&#10;&#10;Although the aforementioned approaches for measuring representations have been widely used, there are certain limitations. For example, a probing task is not formally measuring the ``information'' inherent in the representations but their accuracy on a task.&#10;Another limitation is that the probing tasks all rely on labeled data.&#10;Further, contextual speech representations learned from self-supervised models are actually trained to predict the context such as the future or masked frames \cite{baevski2020wav2vec,hsu2021hubert,chung2020generative,chung2020vector,yang2022autoregressive}. The current probing approaches, however, ask the classifier to do same-frame prediction. &#10;The mismatch between the training objectives and the evaluations makes it unclear whether the current measurements have properly reflected the information representations encoded. &#10;&#10;In this paper, we present an information-theoretic approach to assess the information contained in representations \cite{pimentel2020information,voita2020information}. We use mutual information (MI) to measure the relationship between representations and their targets, such as phonetic labels.&#10;To evaluate how well representations capture the context, we propose that effective representations should exhibit higher MI between different parts of the input due to self-supervised training. To test this, we divide the input into two parts and estimate the MI between representations derived from different parts. This offers an unsupervised alternative to measure the learned representations.&#10;&#10;Through extensive experiments, our findings reveal a strong correlation between unsupervised measures and supervised ones in phonetic-related probing. This correlation suggests the potential for probing representations without labeled data. Furthermore, we observe that models exhibiting higher MI in an unsupervised measure also exhibit superior performance in downstream speech recognition.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Information-Theoretic Analysis, Speech Representation Learning, Information Science, Model Evaluation Metrics" />
        </attvalues>
      </node>
      <node id="2107.08248" label="2107.08248">
        <attvalues>
          <attvalue for="0" value="Learning De-identified Representations of Prosody from Raw Audio" />
          <attvalue for="1" value="  We propose a method for learning de-identified prosody representations from&#10;raw audio using a contrastive self-supervised signal. Whereas prior work has&#10;relied on conditioning models on bottlenecks, we introduce a set of inductive&#10;biases that exploit the natural structure of prosody to minimize timbral&#10;information and decouple prosody from speaker representations. Despite&#10;aggressive downsampling of the input and having no access to linguistic&#10;information, our model performs comparably to state-of-the-art speech&#10;representations on DAMMP, a new benchmark we introduce for spoken language&#10;understanding. We use minimum description length probing to show that our&#10;representations have selectively learned the subcomponents of non-timbral&#10;prosody, and that the product quantizer naturally disentangles them without&#10;using bottlenecks. We derive an information-theoretic definition of speech&#10;de-identifiability and use it to demonstrate that our prosody representations&#10;are less identifiable than other speech representations.&#10;" />
          <attvalue for="2" value="&#10;&#10;To produce and understand spoken language, humans encode and decode audio information at multiple timescales. Phonetic information is used to decode linguistic content, which carries part of the meaning in speech. Another source of meaning, prosody, can be decoded through non-phonetic acoustic patterns; for example to identify who's speaking, which primarily relies on the timbre subcomponent of prosody \cite{skerry2018towards,qian2020unsupervised}. Artificial systems mimicking these human processes must solve the similar problems of representing phonetic information (to obtain a linguistic representation) and representing prosodic information \cite{baevski2020wav2vec,kenter2019chive}.&#10;&#10;The phonetic problem has automatic speech recognition (ASR) as its obvious use-case. In recent years, speech representation learning has been increasingly dominated by self-supervised frameworks using contrastive losses. These include contrastive predictive coding (CPC) \cite{oord2018representation}, wav2vec \cite{schneider2019wav2vec}, vq-wav2vec \cite{baevski2019vq} and wav2vec 2.0 \cite{baevski2020wav2vec}, which learn representations directly from raw audio. These models are generally frame- or phone-based and use fine-timescale, fixed-length audio frames as input. This promotes encoding of high-frequency phonetic information, crucial for transcription, but makes them less incentivized to capture patterns occurring on longer timescales. Other approaches have used triplet loss and temporal proximity as a training signal to learn ``semantic'' \cite{jansen2018unsupervised} or ``non-semantic'' \cite{shor2020towards} representations from spectrograms.&#10;&#10;The prosodic problem has been less studied. Its primary use-case has been building more expressive text-to-speech (TTS) systems. Prior approaches to learning representations of prosody have relied on subtractive definitions such as \cite{skerry2018towards}: ``Prosody is the variation in speech signals that remains after accounting for variation due to phonetics, speaker identity, and channel effects (i.e. the recording environment)''. These approaches typically use autoencoders conditioned on lexical information and speaker identity \cite{skerry2018towards, wang2018style, battenberg2019effective, zhang2019learning}. This encourages the remaining information to be contained in a bottleneck that encodes prosody. Other work has used a triple bottleneck to further decompose prosody in its subcomponents \cite{qian2020unsupervised}. For non-timbral prosody (i.e. what remains after removing speaker characteristics), these subcomponents are pitch, rhythm and tempo, acoustically reflected in the fundamental frequency ($F_0$, the pitch contour), intensity or energy ($c_0$), and the speech rate respectively. WaveNet makes explicit use of $F_0$, $c_0$ and phone durations to synthesize speech \cite{oord2016wavenet}. CHiVE represents prosody using $F_0$, $c_0$ and phoneme durations as features in a conditional variational autoencoder \cite{kenter2019chive}. These models share a set of characteristics that motivate our work: they are subtractive and rely on conditioning models on bottlenecked information. Having been developed in the context of TTS, they have inductive biases that are particularly suited to learning primarily phonetic representations, rather than prosodic representations, a weakness highlighted by \cite{oord2016wavenet}.&#10;&#10;The contributions of this paper are as follows:&#10;\begin{itemize}&#10; \item Introducing and characterizing VQP, a self-supervised contrastive model that learns to selectively represent non-timbral prosody from raw audio without using bottlenecks.&#10; \item Adapting probes from the natural language processing (NLP) literature to demonstrate that VQP representations selectively encode the subcomponents of non-timbral prosody.&#10; \item Demonstrating that product quantization can be used for disentanglement of audio representations without using bottlenecks.&#10; \item Introducing an information-theoretic definition of de-identification using prequential probes on a speaker verification task.&#10; \item Benchmarking a number of state-of-the-art, self-supervised audio representation learning models on a set of tasks for spoken language understanding, as well as quantifying their de-identifiability.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Prosody Analysis, Speech Representation Learning, Information Science, De-identification Techniques" />
        </attvalues>
      </node>
      <node id="1711.02209" label="1711.02209">
        <attvalues>
          <attvalue for="0" value="Unsupervised Learning of Semantic Audio Representations" />
          <attvalue for="1" value="  Even in the absence of any explicit semantic annotation, vast collections of&#10;audio recordings provide valuable information for learning the categorical&#10;structure of sounds. We consider several class-agnostic semantic constraints&#10;that apply to unlabeled nonspeech audio: (i) noise and translations in time do&#10;not change the underlying sound category, (ii) a mixture of two sound events&#10;inherits the categories of the constituents, and (iii) the categories of events&#10;in close temporal proximity are likely to be the same or related. Without&#10;labels to ground them, these constraints are incompatible with classification&#10;loss functions. However, they may still be leveraged to identify geometric&#10;inequalities needed for triplet loss-based training of convolutional neural&#10;networks. The result is low-dimensional embeddings of the input spectrograms&#10;that recover 41% and 84% of the performance of their fully-supervised&#10;counterparts when applied to downstream query-by-example sound retrieval and&#10;sound event classification tasks, respectively. Moreover, in&#10;limited-supervision settings, our unsupervised embeddings double the&#10;state-of-the-art classification performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.14324" label="2309.14324">
        <attvalues>
          <attvalue for="0" value="Towards General-Purpose Text-Instruction-Guided Voice Conversion" />
          <attvalue for="1" value="  This paper introduces a novel voice conversion (VC) model, guided by text&#10;instructions such as &quot;articulate slowly with a deep tone&quot; or &quot;speak in a&#10;cheerful boyish voice&quot;. Unlike traditional methods that rely on reference&#10;utterances to determine the attributes of the converted speech, our model adds&#10;versatility and specificity to voice conversion. The proposed VC model is a&#10;neural codec language model which processes a sequence of discrete codes,&#10;resulting in the code sequence of converted speech. It utilizes text&#10;instructions as style prompts to modify the prosody and emotional information&#10;of the given speech. In contrast to previous approaches, which often rely on&#10;employing separate encoders like prosody and content encoders to handle&#10;different aspects of the source speech, our model handles various information&#10;of speech in an end-to-end manner. Experiments have demonstrated the impressive&#10;capabilities of our model in comprehending instructions and delivering&#10;reasonable results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
          <attvalue for="4" value="Voice Conversion, Computer Science, Speech Processing, Linguistics, Artificial Intelligence, Signal Processing, Neural Language Models" />
        </attvalues>
      </node>
      <node id="2110.04392" label="2110.04392">
        <attvalues>
          <attvalue for="0" value="The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and&#10;  Results" />
          <attvalue for="1" value="  In this paper, we introduce the Eval4NLP-2021shared task on explainable&#10;quality estimation. Given a source-translation pair, this shared task requires&#10;not only to provide a sentence-level score indicating the overall quality of&#10;the translation, but also to explain this score by identifying the words that&#10;negatively impact translation quality. We present the data, annotation&#10;guidelines and evaluation setup of the shared task, describe the six&#10;participating systems, and analyze the results. To the best of our knowledge,&#10;this is the first shared task on explainable NLP evaluation metrics. Datasets&#10;and results are available at https://github.com/eval4nlp/SharedTask2021.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent Natural Language Processing (NLP) systems based on pre-trained representations from Transformer language models, such as BERT \cite{devlin-etal-2019-bert} and XLM-Roberta \cite{conneau-etal-2020-unsupervised}, have achieved outstanding results in a variety of tasks. This boost in performance, however, comes at the cost of efficiency and interpretability. Interpretability is a major concern in modern Artificial Intelligence &#10;(AI) &#10;and NLP research \cite{doshi2017towards,danilevsky-etal-2020-survey}, as black-box models undermine users’ trust in new technologies \cite{mercado2016intelligent,toreini2020relationship}.&#10;&#10;In the Eval4NLP 2021 shared task,&#10;we focus on evaluating machine translation (MT) as an example of this problem. Specifically, we look at the task of quality estimation (QE), where the aim is to predict the quality of MT output at inference time without access to reference translations \cite{blatz-etal-2004-confidence,specia2018quality}.&#10;Translation quality can be assessed at different levels of granularity: sentence-level, i.e.\ predicting the overall quality of translated sentences, and word-level, i.e.\ highlighting specific errors in the MT output. Those have traditionally been treated as two separate tasks, each one requiring dedicated training data.&#10;&#10;In this shared task, we propose to address word-level translation error identification as an explainability task. Explainability is a broad area aimed at explaining predictions of machine learning models. Rationale extraction methods achieve this by selecting a portion of the input that justifies model output for a given data point \cite{lei-etal-2016-rationalizing,jain-etal-2020-learning}. A natural way to explain sentence-level quality assessment is &#10;to identify translation errors. Hence, we frame error identification as a task of providing explanations for the predictions of sentence-level QE models. We claim that this task represents a challenging new benchmark for testing explainability for NLP and provides a new way of addressing word-level QE.&#10;&#10;On the one hand, QE is different from other explainable NLP tasks with existing datasets \cite{deyoung2019eraser} in various important aspects. First, it is a regression task, as opposed to binary or multiclass text classification explored in previous work. Second, it is a multilingual task where the output score captures the relationship between source and target sentences. Finally, QE is fundamentally different from e.g.\ text classification, where clues are typically separate words or phrases \cite{zaidan2007using} that can often be considered independently of the rest of the text. By contrast, translation errors can only be identified given the context of the source and target sentences. Thus, this shared task provides a new benchmark for testing explainability methods in NLP.&#10;&#10;On the other hand, treating word-level QE as an explainability problem offers some advantages compared to the current approaches. First, we can potentially avoid the need for supervised data at word level. Second, gold standard test sets can be made less expensive and more reliable. As we will show in Section \ref{sec:data}, rationalized sentence-level evaluation can be a middle ground between relatively cheap but noisy annotations derived from post-editing \cite{fomicheva2020mlqepe} and very informative but expensive explicit error annotation based on error taxonomies, such as the Multidimensional Quality Metrics (MQM) framework \cite{LommelMQM:2014}. For this shared task, we build a new test set with manually annotated explanations for sentence-level quality ratings. To the best of our knowledge, this is the first MT evaluation dataset annotated with human rationales.&#10;&#10;The main objective of the shared task is threefold. &#10;First, it aims to explore the plausibility of explainable evaluation metrics \cite{wiegreffe-pinter-2019-attention}, by proposing a test set with manually annotated rationales. It helps the community better understand how similar the generated explanations are to the human explanations. &#10;Second, the shared task encourages research on unsupervised or semi-supervised methods for error identification, so as to reduce the cost on word-level MT error annotation.&#10;Last but not least, the shared task sheds light on how current NLP evaluation systems arrive at their predictions and to what extent this process is aligned with human reasoning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Quality Estimation, Translation Evaluation, Artificial Intelligence, Natural Language Processing, Explainable NLP" />
        </attvalues>
      </node>
      <node id="2010.00711" label="2010.00711">
        <attvalues>
          <attvalue for="0" value="A Survey of the State of Explainable AI for Natural Language Processing" />
          <attvalue for="1" value="  Recent years have seen important advances in the quality of state-of-the-art&#10;models, but this has come at the expense of models becoming less interpretable.&#10;This survey presents an overview of the current state of Explainable AI (XAI),&#10;considered within the domain of Natural Language Processing (NLP). We discuss&#10;the main categorization of explanations, as well as the various ways&#10;explanations can be arrived at and visualized. We detail the operations and&#10;explainability techniques currently available for generating explanations for&#10;NLP model predictions, to serve as a resource for model developers in the&#10;community. Finally, we point out the current gaps and encourage directions for&#10;future work in this important research area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.04480" label="2010.04480">
        <attvalues>
          <attvalue for="0" value="MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset" />
          <attvalue for="1" value="  We present MLQE-PE, a new dataset for Machine Translation (MT) Quality&#10;Estimation (QE) and Automatic Post-Editing (APE). The dataset contains eleven&#10;language pairs, with human labels for up to 10,000 translations per language&#10;pair in the following formats: sentence-level direct assessments and&#10;post-editing effort, and word-level good/bad labels. It also contains the&#10;post-edited sentences, as well as titles of the articles where the sentences&#10;were extracted from, and the neural MT models used to translate the text.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.04626" label="1908.04626">
        <attvalues>
          <attvalue for="0" value="Attention is not not Explanation" />
          <attvalue for="1" value="  Attention mechanisms play a central role in NLP systems, especially within&#10;recurrent neural network (RNN) models. Recently, there has been increasing&#10;interest in whether or not the intermediate representations offered by these&#10;modules may be used to explain the reasoning for a model's prediction, and&#10;consequently reach insights regarding the model's decision-making process. A&#10;recent paper claims that `Attention is not Explanation' (Jain and Wallace,&#10;2019). We challenge many of the assumptions underlying this work, arguing that&#10;such a claim depends on one's definition of explanation, and that testing it&#10;needs to take into account all elements of the model, using a rigorous&#10;experimental design. We propose four alternative tests to determine&#10;when/whether attention can be used as explanation: a simple uniform-weights&#10;baseline; a variance calibration based on multiple random seed runs; a&#10;diagnostic framework using frozen weights from pretrained models; and an&#10;end-to-end adversarial attention training protocol. Each allows for meaningful&#10;interpretation of attention mechanisms in RNN models. We show that even when&#10;reliable adversarial distributions can be found, they don't perform well on the&#10;simple diagnostic, indicating that prior work does not disprove the usefulness&#10;of attention mechanisms for explainability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.16370" label="2407.16370">
        <attvalues>
          <attvalue for="0" value="Evolutionary Prompt Design for LLM-Based Post-ASR Error Correction" />
          <attvalue for="1" value="  Building upon the strength of modern large language models (LLMs), generative&#10;error correction (GEC) has emerged as a promising paradigm that can elevate the&#10;performance of modern automatic speech recognition (ASR) systems. One&#10;representative approach is to leverage in-context learning to prompt LLMs so&#10;that a better hypothesis can be generated by the LLMs based on a&#10;carefully-designed prompt and an $N$-best list of hypotheses produced by ASR&#10;systems. However, it is yet unknown whether the existing prompts are the most&#10;effective ones for the task of post-ASR error correction. In this context, this&#10;paper first explores alternative prompts to identify an initial set of&#10;effective prompts, and then proposes to employ an evolutionary prompt&#10;optimization algorithm to refine the initial prompts. Evaluations results on&#10;the CHiME-4 subset of the Task $1$ of the SLT $2024$ GenSEC challenge show the&#10;effectiveness and potential of the proposed algorithms.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) \cite{Minaee2024}, trained on massive textual data using neural network-based transformer architectures, have revolutionized many tasks in natural language processing. The auto-regressive learning nature of LLMs introduces and empowers a new ``prompting'' mechanism based on input instructions, where users can provide text prompts to guide LLMs to complete particular tasks \cite{Liu2022} as a form of next token prediction.&#10;&#10;One of these popular prompt-activated tasks is post-automatic speech recognition (ASR) text correction, where based on an $N$-best list of hypotheses generated by ASR systems, the task is to predict the true transcription \cite{Chen2023HyPoradise, Radhakrishnan2023, Gu2024DenoisingLM, Hu2024robustASR}.&#10;Although this problem can be approached by language model re-scoring techniques~\cite{yang2021multi,liu2016attention, ma2023n}, recent studies \cite{Chen2023HyPoradise, Radhakrishnan2023, Gu2024DenoisingLM, Hu2024robustASR} have shown that leveraging LLMs to correct errors in an generative way often leads to better performance. Specifically, LLM-based generative error correction~\cite{yang2023generative} (GER) can infer phonetic similarity and contextual information to reduce errors beyond oracle ranking result of the $N$-hypotheses list.&#10;&#10;When input prompts are critical for instructing LLMs on unseen ASR tasks, these task-activating prompts \cite{yang2023generative} are often empirically-designed and under-explored in the research community. For example, early works~\cite{watanabe2017language, yang2021voice2series,gao2022wavprompt, chang2023speechprompt} focus on iterative optimization at the waveform level to instruct acoustic models for new tasks, but there are fewer studies on optimizing LLM-prompts for ASR tasks.&#10;&#10;To achieve better post-ASR error correction, this paper explores alternative prompts for this task, and investigates a conditional evolutionary strategies based prompt optimization algorithm, named EvoPrompt \cite{Guo2024Evoprompt}, to refine the alternative prompts.&#10;Evaluation results on the CHiME-4 subset of the HyPoradise dataset \cite{Chen2023HyPoradise} show the effectiveness of the proposed algorithms.&#10;&#10;In the rest of this paper, we describe the proposed algorithm in Section \ref{proposed_algorithm}, experimental setup in Section \ref{exp_setup}, evaluation results in Section \ref{results}, and conclusions in Section \ref{conclusions}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Error Correction, Language Models, Artificial Intelligence, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2011.11715" label="2011.11715">
        <attvalues>
          <attvalue for="0" value="Multi-task Language Modeling for Improving Speech Recognition of Rare&#10;  Words" />
          <attvalue for="1" value="  End-to-end automatic speech recognition (ASR) systems are increasingly&#10;popular due to their relative architectural simplicity and competitive&#10;performance. However, even though the average accuracy of these systems may be&#10;high, the performance on rare content words often lags behind hybrid ASR&#10;systems. To address this problem, second-pass rescoring is often applied&#10;leveraging upon language modeling. In this paper, we propose a second-pass&#10;system with multi-task learning, utilizing semantic targets (such as intent and&#10;slot prediction) to improve speech recognition performance. We show that our&#10;rescoring model trained with these additional tasks outperforms the baseline&#10;rescoring model, trained with only the language modeling task, by 1.4% on a&#10;general test and by 2.6% on a rare word test set in terms of word-error-rate&#10;relative (WERR). Our best ASR system with multi-task LM shows 4.6% WERR&#10;deduction compared with RNN Transducer only ASR baseline for rare words&#10;recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.08532" label="2309.08532">
        <attvalues>
          <attvalue for="0" value="Connecting Large Language Models with Evolutionary Algorithms Yields&#10;  Powerful Prompt Optimizers" />
          <attvalue for="1" value="  Large Language Models (LLMs) excel in various tasks, but they rely on&#10;carefully crafted prompts that often demand substantial human effort. To&#10;automate this process, in this paper, we propose a novel framework for discrete&#10;prompt optimization, called EvoPrompt, which borrows the idea of evolutionary&#10;algorithms (EAs) as they exhibit good performance and fast convergence. To&#10;enable EAs to work on discrete prompts, which are natural language expressions&#10;that need to be coherent and human-readable, we connect LLMs with EAs. This&#10;approach allows us to simultaneously leverage the powerful language processing&#10;capabilities of LLMs and the efficient optimization performance of EAs.&#10;Specifically, abstaining from any gradients or parameters, EvoPrompt starts&#10;from a population of prompts and iteratively generates new prompts with LLMs&#10;based on the evolutionary operators, improving the population based on the&#10;development set. We optimize prompts for both closed- and open-source LLMs&#10;including GPT-3.5 and Alpaca, on 31 datasets covering language understanding,&#10;generation tasks, as well as BIG-Bench Hard (BBH) tasks. EvoPrompt&#10;significantly outperforms human-engineered prompts and existing methods for&#10;automatic prompt generation (e.g., up to 25% on BBH). Furthermore, EvoPrompt&#10;demonstrates that connecting LLMs with EAs creates synergies, which could&#10;inspire further research on the combination of LLMs and conventional&#10;algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.10602" label="2406.10602">
        <attvalues>
          <attvalue for="0" value="Multilingual Large Language Models and Curse of Multilinguality" />
          <attvalue for="1" value="  Multilingual Large Language Models (LLMs) have gained large popularity among&#10;Natural Language Processing (NLP) researchers and practitioners. These models,&#10;trained on huge datasets, show proficiency across various languages and&#10;demonstrate effectiveness in numerous downstream tasks. This paper navigates&#10;the landscape of multilingual LLMs, providing an introductory overview of their&#10;technical aspects. It explains underlying architectures, objective functions,&#10;pre-training data sources, and tokenization methods. This work explores the&#10;unique features of different model types: encoder-only (mBERT, XLM-R),&#10;decoder-only (XGLM, PALM, BLOOM, GPT-3), and encoder-decoder models (mT5,&#10;mBART). Additionally, it addresses one of the significant limitations of&#10;multilingual LLMs - the curse of multilinguality - and discusses current&#10;attempts to overcome it.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language Models (LLMs) \cite{devlin2018bert, lewis2019bart, liu2019roberta} have made a significant impact on the field of Natural Language Processing (NLP), showing effectiveness in various tasks. The remarkable aspect of LLMs is their capacity to learn a language during pre-training and enhance their expertise for specific tasks during fine-tuning. Pre-training involves the acquisition of knowledge, where a model grasps language structures by analyzing huge datasets. Fine-tuning, on the other hand, specializes the model by adjusting its parameters so that it can perform specific downstream tasks using a smaller set of examples compared to those used in pre-training.&#10;&#10;Another significant advancement involves teaching a model to comprehend multiple languages, leading to the concept of multilingual LLMs \cite{mbert, pires2019multilingual, conneau2019unsupervised, xue2020mt5, liu2020multilingual}. While monolingual LLMs focus on understanding patterns within a single language, multilingual LLMs simultaneously learn from multiple languages. This is accomplished by exposing these models to data from various languages during the pre-training phase. Furthermore, variations in the architectures of multilingual LLMs contribute to their strengths in certain tasks while potentially limiting their effectiveness in others. &#10;&#10;This paper aims to provide a brief overview of the architectures of the most prominent multilingual LLMs, including details such as their pre-training objective functions, data sources, tokenization schemas, the number of languages supported, and the peculiarities of each individual multilingual LLM. Subsequently, the primary challenge facing multilingual LLMs, known as the &quot;curse of multilinguality&quot; \cite{conneau2019unsupervised}, and the current attempts to solve it, are discussed.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Processing, Computer Science, Linguistics, Cognitive Science, Multilingual Models, Artificial Intelligence, Model Architectures" />
        </attvalues>
      </node>
      <node id="2010.03726" label="2010.03726">
        <attvalues>
          <attvalue for="0" value="Learning to Fuse Sentences with Transformers for Summarization" />
          <attvalue for="1" value="  The ability to fuse sentences is highly attractive for summarization systems&#10;because it is an essential step to produce succinct abstracts. However, to&#10;date, summarizers can fail on fusing sentences. They tend to produce few&#10;summary sentences by fusion or generate incorrect fusions that lead the summary&#10;to fail to retain the original meaning. In this paper, we explore the ability&#10;of Transformers to fuse sentences and propose novel algorithms to enhance their&#10;ability to perform sentence fusion by leveraging the knowledge of points of&#10;correspondence between sentences. Through extensive experiments, we investigate&#10;the effects of different design choices on Transformer's performance. Our&#10;findings highlight the importance of modeling points of correspondence between&#10;sentences for effective sentence fusion.&#10;" />
          <attvalue for="2" value="&#10;&#10;A renewed emphasis must be placed on sentence fusion in the context of neural abstractive summarization.&#10;A majority of the systems are trained end-to-end~\cite{see-etal-2017-get,paulus2018a,narayan-etal-2018-dont,chen-bansal-2018-fast,gehrmann-etal-2018-bottom,liu-lapata-2019-hierarchical}, where an abstractive summarizer is rewarded for generating summaries that contain the same words as human abstracts, measured by automatic metrics such as ROUGE~\cite{lin-2004-rouge}.&#10;A summarizer, however, is not rewarded for correctly fusing sentences.&#10;In fact, when examined more closely, only few sentences in system abstracts are generated by fusion~\cite{falke-etal-2019-ranking,lebanoff-etal-2019-analyzing}.&#10;For instance, 6\% of summary sentences generated by Pointer-Gen~\cite{see-etal-2017-get} are through fusion, whereas human abstracts contain 32\% fusion sentences.&#10;Moreover, sentences generated by fusion are prone to errors.&#10;They can be ungrammatical, nonsensical, or otherwise ill-formed.&#10;There is thus an urgent need to develop neural abstractive summarizers to fuse sentences properly.&#10;&#10;The importance of sentence fusion has long been recognized by the community before the era of neural text summarization.&#10;The pioneering work of Barzilay et al.~\shortcite{barzilay-etal-1999-information} introduces an information fusion algorithm that combines similar elements across related text to generate a succinct summary.&#10;Later work, such as~\cite{marsi-krahmer-2005-explorations,filippova-strube-2008-sentence,elsner-santhanam-2011-learning,thadani-mckeown-2013-supervised,mehdad-etal-2013-abstractive}, builds a dependency or word graph by combining syntactic trees of similar sentences, then employs integer linear programming to decode a summary sentence from the graph. &#10;Most of these studies have assumed a set of similar sentences as input, where fusion is necessary to reduce repetition.&#10;Nonetheless, humans do not limit themselves to combine similar sentences.&#10;In this paper, we pay particular attention to fuse disparate sentences that contain fundamentally different content but remain related to make fusion sensible~\cite{elsner-santhanam-2011-learning}.&#10;In Figure~\ref{fig:example_fusion}, we provide an example of a sentence fusion instance.&#10;&#10;We address the challenge of fusing disparate sentences by enhancing the Transformer architecture~\cite{NIPS2017_7181} with points of correspondence between sentences, which are devices that tie two sentences together into a coherent text.&#10;The task of sentence fusion involves choosing content from each sentence and weaving the content pieces together into an output sentence that is linguistically plausible and semantically truthful to the original input.&#10;It is distinct from~\cite{geva-etal-2019-discofuse} that connect two sentences with discourse markers.&#10;Our contributions are as follows.&#10;\begin{itemize}[topsep=5pt,itemsep=0pt,leftmargin=*]&#10;\item &#10;We make crucial use of points of correspondence (PoC) between sentences for information fusion.&#10;Our use of PoC was initiated by the current lack of understanding of how sentences are combined in neural text summarization.&#10;&#10;\item &#10;We design new sentence fusion systems and experiment with a fusion dataset containing quality PoC annotations as the test bed for this investigation.&#10;Our findings highlight the importance of modeling points of correspondence for fusion.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Summarization Systems, Sentence Fusion, Computer Science, Linguistics, Artificial Intelligence, Transformer Models, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1905.13164" label="1905.13164">
        <attvalues>
          <attvalue for="0" value="Hierarchical Transformers for Multi-Document Summarization" />
          <attvalue for="1" value="  In this paper, we develop a neural summarization model which can effectively&#10;process multiple input documents and distill Transformer architecture with the&#10;ability to encode documents in a hierarchical manner. We represent&#10;cross-document relationships via an attention mechanism which allows to share&#10;information as opposed to simply concatenating text spans and processing them&#10;as a flat sequence. Our model learns latent dependencies among textual units,&#10;but can also take advantage of explicit graph representations focusing on&#10;similarity or discourse relations. Empirical results on the WikiSum dataset&#10;demonstrate that the proposed architecture brings substantial improvements over&#10;several strong baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.01767" label="2206.01767">
        <attvalues>
          <attvalue for="0" value="[Re] Badder Seeds: Reproducing the Evaluation of Lexical Methods for&#10;  Bias Measurement" />
          <attvalue for="1" value="  Combating bias in NLP requires bias measurement. Bias measurement is almost&#10;always achieved by using lexicons of seed terms, i.e. sets of words specifying&#10;stereotypes or dimensions of interest. This reproducibility study focuses on&#10;the original authors' main claim that the rationale for the construction of&#10;these lexicons needs thorough checking before usage, as the seeds used for bias&#10;measurement can themselves exhibit biases. The study aims to evaluate the&#10;reproducibility of the quantitative and qualitative results presented in the&#10;paper and the conclusions drawn thereof. We reproduce most of the results&#10;supporting the original authors' general claim: seed sets often suffer from&#10;biases that affect their performance as a baseline for bias metrics. Generally,&#10;our results mirror the original paper's. They are slightly different on select&#10;occasions, but not in ways that undermine the paper's general intent to show&#10;the fragility of seed sets.&#10;" />
          <attvalue for="2" value="&#10;The emergence of bias quantification in Natural Language Processing (NLP) methods has given rise to two use cases, referred to as downstream and upstream. In the former, bias measurements are used to debias or correct biases in word representations to avoid encoded biases trickling down when applying these NLP models \cite{bolukbasi_man_2016, caliskan_semantics_2017}. In the latter, bias measurements are used on models trained on small corpora to quantify the bias present and compare them.&#10;This use case has endowed social scientists with the quantitative foundation to answer political and&#10;social questions about bias across corpora in an empirical manner&#10;\cite{rudinger_social_2017,joseph_girls_2017}.&#10;Crucially, most bias quantification methods depend on lexicons of seed terms that specify the bias dimensions of interest. The selection of seed terms varies considerably across the literature, and seed sets themselves may exhibit social and cognitive biases \cite{antoniak-mimno-2021-bad}. It is not clear whether it is possible to re-use seed set across corpora (thereby interfering with upstream use cases), and elements such as seed term frequency have been shown to affect bias measurements, and thus downstream uses \cite{ethayarajh2019understanding}.&#10;&#10;We seek to replicate the \cite{antoniak-mimno-2021-bad} paper, hereafter referred to as ''the original paper/work''. In it, the authors seek to 1) qualitatively explore seed selection and their sources, 2) demonstrate that features of seed sets such as pairing order, set similarity, and frequency can cause instability in bias measurements, and 3) make recommendations for the testing and justifying of seed sets in future work. We have replicated the experiments showing the fragility of seed sets, thus verifying the claims of a need for better justification and analysis of them in future literature. We have also built a public toolkit to reproduce these measures on arbitrary seed sets and trained embeddings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Lexicon Construction, Linguistics, Bias Measurement, Natural Language Processing, Statistics, NLP Reproducibility" />
        </attvalues>
      </node>
      <node id="2407.14822" label="2407.14822">
        <attvalues>
          <attvalue for="0" value="Text Style Transfer: An Introductory Overview" />
          <attvalue for="1" value="  Text Style Transfer (TST) is a pivotal task in natural language generation to&#10;manipulate text style attributes while preserving style-independent content.&#10;The attributes targeted in TST can vary widely, including politeness,&#10;authorship, mitigation of offensive language, modification of feelings, and&#10;adjustment of text formality. TST has become a widely researched topic with&#10;substantial advancements in recent years. This paper provides an introductory&#10;overview of TST, addressing its challenges, existing approaches, datasets,&#10;evaluation measures, subtasks, and applications. This fundamental overview&#10;improves understanding of the background and fundamentals of text style&#10;transfer.&#10;" />
          <attvalue for="2" value="&#10;Natural Language Generation&#10;Natural Language Generation (NLG) is the process of producing meaningful phrases and sentences in natural language.&#10;The main goal of NLG is to automatically produce narratives that describe, summarize, and explain the input data in a human-like manner. In other words, it generates fluent texts with minimal grammatical errors and retains the specific intended content.&#10;&#10;Some of the popular NLG tasks include machine translation \cite{cho2014learning}, dialogue systems \cite{shang2015neural}, and text summarization \cite{rush2017neural}. Through these tasks, the generated text has shown to be more coherent, logical, and emotionally rich, &#10;especially with the latest approaches based on neural language models.&#10;&#10;Controllable NLG&#10;&#10;Most of the built NLG systems target text fluency and grammatical correctness, and do not consider any specific control over text style. This is a motivation for research on controllable text generation &#10;\cite{len2020controllable}. &#10;The aspects of text generation that are commonly controlled include topic \cite{dziri-etal-2019-augmenting,feng2018topic,ijcai2018-619,xing2017topic}, style \cite{li2018delete, sudhakar2019transforming, prabhumoye2018style, chen2018adversarial}, emotion \cite{fu2018style,kong2019adversarial,DBLP:journals/inffus/SunLWLT20,zhou2018emotional}, and user preferences \cite{li-etal-2016-persona,luan-etal-2017-multi,yang2018investigating,yang2017personalized}. Some of the applications of controllable text generation are context-based text generation \cite{jaech2018low}, topic-aware text generation, \cite{wang2018reinforced}, knowledge-enhanced text generation \cite{young2018augmenting} and text style transfer \cite{hu2022text}.&#10;&#10;Control can be applied at various stages of the neural generation process, such as input, hidden states, and decoding \cite{prabhumoye2020exploring}. The Plug and Play language model (PPLM) that was proposed by \cite{dathathri2019plug} takes an external input, performs computations on hidden states, and then combines a pre-trained language model with one or more simple attribute classifiers that guide text generation toward the desired topic or sentiment. Another model by \cite{gehrmann2018end} describes a training method based on diverse ensembling that would lead models to learn distinct text styles. It can thus be inferred that end-to-end models can be equipped with the ability to control style and length. More details on how NLG can be controlled using various control strategies in the state-of-the-art models can be found in \cite{len2020controllable}.&#10;&#10;Style-Controlled Text Generation&#10;&#10;In recent research, more attention has been paid to a subtask of controllable text generation dubbed style-controlled text generation, i.e., modeling and manipulating the style of the generated text \cite{mou2020stylized}. The goal of this approach is to model the content of a text along with controlling its style. For example, the persona of a speaker in dialogue \cite{li2016persona} or the sentiment of product reviews \cite{hu2017toward}. Understanding and dealing with style in text proves to be very complex \cite{hu2022text}, but recent advances in deep learning techniques are helping stylized text generation tasks in various ways \cite{jin2022deep}. For example, embedding learning techniques are used to represent style \cite{fu2018style}, and then adversarial learning is used to match content but to distinguish between different styles\cite{hu2017toward, xu2018unpaired, john2018disentangled}.&#10;&#10;Text Style Transfer&#10;&#10;In this paper, we will focus on Text Style Transfer (TST). TST is a task closely related to Style-Controlled Text Generation. Style-Controlled Text Generation aims to generate new text in a specific style. In contrast, Text Style Transfer is an existing text written in source style, aiming to change the text style, i.e. a text retaining most of the content but conforming to the target style.&#10;Our aim is to give a very basic introduction to the TST task. All of the sections are presented in a brief and simple manner with an illustrative number of examples. A more detailed overview can be found in \cite{hu2022text, jin2022deep, mou2020stylized, toshevska2021review, prabhumoye2020exploring}.&#10;&#10;The paper is organized as follows. After the introductory section, Section \ref{tst} provides an overview of text style transfer. Section~\ref{challenges} reflects on the challenges facing the TST task. The discussion of the existing data sets, approaches, evaluation measures and applications is presented in Sections~\ref{datasets}-\ref{applications}. A short overview of the related ethical considerations is given in Section \ref{ethics}. Section \ref{conclusion} concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Language Manipulation, Natural Language Processing, Natural Language Generation, Text Style Transfer" />
        </attvalues>
      </node>
      <node id="1503.02364" label="1503.02364">
        <attvalues>
          <attvalue for="0" value="Neural Responding Machine for Short-Text Conversation" />
          <attvalue for="1" value="  We propose Neural Responding Machine (NRM), a neural network-based response&#10;generator for Short-Text Conversation. NRM takes the general encoder-decoder&#10;framework: it formalizes the generation of response as a decoding process based&#10;on the latent representation of the input text, while both encoding and&#10;decoding are realized with recurrent neural networks (RNN). The NRM is trained&#10;with a large amount of one-round conversation data collected from a&#10;microblogging service. Empirical study shows that NRM can generate&#10;grammatically correct and content-wise appropriate responses to over 75% of the&#10;input text, outperforming state-of-the-arts in the same setting, including&#10;retrieval-based and SMT-based models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.01063" label="1811.01063">
        <attvalues>
          <attvalue for="0" value="Augmenting Neural Response Generation with Context-Aware Topical&#10;  Attention" />
          <attvalue for="1" value="  Sequence-to-Sequence (Seq2Seq) models have witnessed a notable success in&#10;generating natural conversational exchanges. Notwithstanding the syntactically&#10;well-formed responses generated by these neural network models, they are prone&#10;to be acontextual, short and generic. In this work, we introduce a Topical&#10;Hierarchical Recurrent Encoder Decoder (THRED), a novel, fully data-driven,&#10;multi-turn response generation system intended to produce contextual and&#10;topic-aware responses. Our model is built upon the basic Seq2Seq model by&#10;augmenting it with a hierarchical joint attention mechanism that incorporates&#10;topical concepts and previous interactions into the response generation. To&#10;train our model, we provide a clean and high-quality conversational dataset&#10;mined from Reddit comments. We evaluate THRED on two novel automated metrics,&#10;dubbed Semantic Similarity and Response Echo Index, as well as with human&#10;evaluation. Our experiments demonstrate that the proposed model is able to&#10;generate more diverse and contextually relevant responses compared to the&#10;strong baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.03616" label="1805.03616">
        <attvalues>
          <attvalue for="0" value="A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for&#10;  Abstractive Text Summarization" />
          <attvalue for="1" value="  In this paper, we propose a deep learning approach to tackle the automatic&#10;summarization tasks by incorporating topic information into the convolutional&#10;sequence-to-sequence (ConvS2S) model and using self-critical sequence training&#10;(SCST) for optimization. Through jointly attending to topics and word-level&#10;alignment, our approach can improve coherence, diversity, and informativeness&#10;of generated summaries via a biased probability generation mechanism. On the&#10;other hand, reinforcement training, like SCST, directly optimizes the proposed&#10;model with respect to the non-differentiable metric ROUGE, which also avoids&#10;the exposure bias during inference. We carry out the experimental evaluation&#10;with state-of-the-art methods over the Gigaword, DUC-2004, and LCSTS datasets.&#10;The empirical results demonstrate the superiority of our proposed method in the&#10;abstractive summarization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.08340" label="1606.08340">
        <attvalues>
          <attvalue for="0" value="Topic Aware Neural Response Generation" />
          <attvalue for="1" value="  We consider incorporating topic information into the sequence-to-sequence&#10;framework to generate informative and interesting responses for chatbots. To&#10;this end, we propose a topic aware sequence-to-sequence (TA-Seq2Seq) model. The&#10;model utilizes topics to simulate prior knowledge of human that guides them to&#10;form informative and interesting responses in conversation, and leverages the&#10;topic information in generation by a joint attention mechanism and a biased&#10;generation probability. The joint attention mechanism summarizes the hidden&#10;vectors of an input message as context vectors by message attention,&#10;synthesizes topic vectors by topic attention from the topic words of the&#10;message obtained from a pre-trained LDA model, and let these vectors jointly&#10;affect the generation of words in decoding. To increase the possibility of&#10;topic words appearing in responses, the model modifies the generation&#10;probability of topic words by adding an extra probability item to bias the&#10;overall distribution. Empirical study on both automatic evaluation metrics and&#10;human annotations shows that TA-Seq2Seq can generate more informative and&#10;interesting responses, and significantly outperform the-state-of-the-art&#10;response generation models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1901.07129" label="1901.07129">
        <attvalues>
          <attvalue for="0" value="An Adversarial Approach to High-Quality, Sentiment-Controlled Neural&#10;  Dialogue Generation" />
          <attvalue for="1" value="  In this work, we propose a method for neural dialogue response generation&#10;that allows not only generating semantically reasonable responses according to&#10;the dialogue history, but also explicitly controlling the sentiment of the&#10;response via sentiment labels. Our proposed model is based on the paradigm of&#10;conditional adversarial learning; the training of a sentiment-controlled&#10;dialogue generator is assisted by an adversarial discriminator which assesses&#10;the fluency and feasibility of the response generating from the dialogue&#10;history and a given sentiment label. Because of the flexibility of our&#10;framework, the generator could be a standard sequence-to-sequence (SEQ2SEQ)&#10;model or a more complicated one such as a conditional variational&#10;autoencoder-based SEQ2SEQ model. Experimental results using automatic and human&#10;evaluation both demonstrate that our proposed framework is able to generate&#10;both semantically reasonable and sentiment-controlled dialogue responses.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1603.06155" label="1603.06155">
        <attvalues>
          <attvalue for="0" value="A Persona-Based Neural Conversation Model" />
          <attvalue for="1" value="  We present persona-based models for handling the issue of speaker consistency&#10;in neural response generation. A speaker model encodes personas in distributed&#10;embeddings that capture individual characteristics such as background&#10;information and speaking style. A dyadic speaker-addressee model captures&#10;properties of interactions between two interlocutors. Our models yield&#10;qualitative performance improvements in both perplexity and BLEU scores over&#10;baseline sequence-to-sequence models, with similar gains in speaker consistency&#10;as measured by human judges.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.07388" label="1710.07388">
        <attvalues>
          <attvalue for="0" value="Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation&#10;  Models" />
          <attvalue for="1" value="  Building a persona-based conversation agent is challenging owing to the lack&#10;of large amounts of speaker-specific conversation data for model training. This&#10;paper addresses the problem by proposing a multi-task learning approach to&#10;training neural conversation models that leverages both conversation data&#10;across speakers and other types of data pertaining to the speaker and speaker&#10;roles to be modeled. Experiments show that our approach leads to significant&#10;improvements over baseline model quality, generating responses that capture&#10;more precisely speakers' traits and speaking styles. The model offers the&#10;benefits of being algorithmically simple and easy to implement, and not relying&#10;on large quantities of data representing specific individual speakers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.02603" label="1710.02603">
        <attvalues>
          <attvalue for="0" value="Low-Rank RNN Adaptation for Context-Aware Language Modeling" />
          <attvalue for="1" value="  A context-aware language model uses location, user and/or domain metadata&#10;(context) to adapt its predictions. In neural language models, context&#10;information is typically represented as an embedding and it is given to the RNN&#10;as an additional input, which has been shown to be useful in many applications.&#10;We introduce a more powerful mechanism for using context to adapt an RNN by&#10;letting the context vector control a low-rank transformation of the recurrent&#10;layer weight matrix. Experiments show that allowing a greater fraction of the&#10;model parameters to be adjusted has benefits in terms of perplexity and&#10;classification for several different types of context.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12742" label="2010.12742">
        <attvalues>
          <attvalue for="0" value="Text Style Transfer: A Review and Experimental Evaluation" />
          <attvalue for="1" value="  The stylistic properties of text have intrigued computational linguistics&#10;researchers in recent years. Specifically, researchers have investigated the&#10;Text Style Transfer (TST) task, which aims to change the stylistic properties&#10;of the text while retaining its style independent content. Over the last few&#10;years, many novel TST algorithms have been developed, while the industry has&#10;leveraged these algorithms to enable exciting TST applications. The field of&#10;TST research has burgeoned because of this symbiosis. This article aims to&#10;provide a comprehensive review of recent research efforts on text style&#10;transfer. More concretely, we create a taxonomy to organize the TST models and&#10;provide a comprehensive summary of the state of the art. We review the existing&#10;evaluation methodologies for TST tasks and conduct a large-scale&#10;reproducibility study where we experimentally benchmark 19 state-of-the-art TST&#10;algorithms on two publicly available datasets. Finally, we expand on current&#10;trends and provide new perspectives on the new and exciting developments in the&#10;TST field.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.04700" label="1810.04700">
        <attvalues>
          <attvalue for="0" value="End-to-End Content and Plan Selection for Data-to-Text Generation" />
          <attvalue for="1" value="  Learning to generate fluent natural language from structured data with neural&#10;networks has become an common approach for NLG. This problem can be challenging&#10;when the form of the structured data varies between examples. This paper&#10;presents a survey of several extensions to sequence-to-sequence models to&#10;account for the latent content selection process, particularly variants of copy&#10;attention and coverage decoding. We further propose a training method based on&#10;diverse ensembling to encourage models to learn distinct sentence templates&#10;during training. An empirical evaluation of these techniques shows an increase&#10;in the quality of generated text across five automated metrics, as well as&#10;human evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.00418" label="2301.00418">
        <attvalues>
          <attvalue for="0" value="Is word segmentation necessary for Vietnamese sentiment classification?" />
          <attvalue for="1" value="  To the best of our knowledge, this paper made the first attempt to answer&#10;whether word segmentation is necessary for Vietnamese sentiment classification.&#10;To do this, we presented five pre-trained monolingual S4- based language models&#10;for Vietnamese, including one model without word segmentation, and four models&#10;using RDRsegmenter, uitnlp, pyvi, or underthesea toolkits in the pre-processing&#10;data phase. According to comprehensive experimental results on two corpora,&#10;including the VLSP2016-SA corpus of technical article reviews from the news and&#10;social media and the UIT-VSFC corpus of the educational survey, we have two&#10;suggestions. Firstly, using traditional classifiers like Naive Bayes or Support&#10;Vector Machines, word segmentation maybe not be necessary for the Vietnamese&#10;sentiment classification corpus, which comes from the social domain. Secondly,&#10;word segmentation is necessary for Vietnamese sentiment classification when&#10;word segmentation is used before using the BPE method and feeding into the deep&#10;learning model. In this way, the RDRsegmenter is the stable toolkit for word&#10;segmentation among the uitnlp, pyvi, and underthesea toolkits.&#10;" />
          <attvalue for="2" value="&#10;Word segmentation is a fundamental problem in the field of Vietnamese natural language processing. For instance, the Vietnamese text ``hiện đại hóa đất nước'' ($modernize_hiện\_đại\_hoá$ $country_đất\_nước$), which consists of five syllables, is segmented into ``hiện\_đại\_hoá đất\_nước''. Underscores indicate the white spaces function as syllable separators, and white spaces are used for word splits. That is a challenging problem in the early stage of natural language processing in Vietnamese \cite{diend1}. Common tasks in Vietnamese syntax analysis tasks, such as part-of-speech tagging \cite{lehongtag10}, constituency parsing \cite{nguyen-etal-2009-building, 10.1007/s10579-017-9398-3}, dependency parsing \cite{10.1007/978-3-319-07983-7_26,6719884,nguyen-2018-bktreebank}, and semantic parsing \cite{JCSCE}, are required to undergo word segmentation. In these tasks, the mistake of word segmentation directly affects them. Therefore, many previous works introduced various approaches to improve Vietnamese word segmentation performance, including single word segmentation task only \cite{diend1, dinhdien06, ha2003, nguyenetal2006, thangdq1, hongphuong08, songnguyen16, phongnt1, datnq1, vund2018, spanviws} and multi-task containing word segmentation and part-of-speech tagging \cite{ducpd1, nguyen-etal-2017-word}, dependency parsing \cite{nguyen-2019-neural}. From the previous works above, word segmentation is obligatory for Vietnamese syntax analysis tasks, including part-of-speech tagging, constituency parsing, dependency parsing, and semantic parsing.&#10;&#10;In addition to the Vietnamese syntax analysis tasks, sentiment classification attracts many researchers because we can apply it quickly to real-life applications. Firstly, \cite{5632131} introduced a system that can classify a computer product review into positive or negative. The study of \cite{5632131} is the first work that analyzes sentiment at the sentence level in Vietnamese. Another example, in the research of \cite{7043403}, they proposed a framework classifying the hotel review into one positive, neutral, or negative class. Almost all do word segmentation in data pre-processing before researching methodology to solve the sentiment classification task. The vnTokenizer toolkit \cite{hongphuong08} was used for pre-processing the VLSP2016-SA \cite{vlsp2016sa} and VS corpora in the work of \cite{8119429}, the hotel reviews corpus proposed by \cite{7043403}, the electronic devices corpus proposed by \cite{7371776}, the education survey corpus proposed by \cite{bagofstruct}, and the online reviews corpus proposed by \cite{Tran2017}. Another Vietnamese word segmenter proposed by \cite{ducpd1}, was used in the works of \cite{10.1007/978-3-642-23620-4_21}, \cite{5632131}, and \cite{Vu2011} about product reviews domain. After that, \cite{10.1145/2676585.2676606} used the JVnSegmenter toolkit \cite{nguyenetal2006} for pre-processing their food reviews corpus. In addition, \cite{7758052} apply the UETsegmenter \cite{phongnt1} to preproces their mobile product reviews corpus. Lastly, the works \cite{8573351, 9335912} about sentiment analysis on feedback of students \cite{vsfc} and \cite{10.1007/978-981-15-6168-9_27} about emotion recognition used the RDRsegmenter of \cite{datnq1} for their research. All of the above works on Vietnamese sentiment analysis used Vietnamese word segmenter published in scientific papers. Moreover, we can survey two well-known Vietnamese word segmenters, namely pyvi and underthesea scientifically unpublished up to now. For instances, the pyvi toolkit was used in the research of \cite{10.1007/978-981-15-6168-9_15} about sentiment analysis on VLSP2018-SA corpus \cite{vlsp2016sa} and research of \cite{DBLP:conf/somet/0002LTLHP21} on product reviews. Another instance, \cite{DBLP:conf/enase/0002HHPZ20} used underthesea toolkit for pre-processing their electronic products comments dataset. Lastly, we observed many prior studies on Vietnamese sentiment classification apply word segmentation in the pre-processing phase.&#10;&#10;Although we can observe that the word segmentation phase is almost necessary for the Vietnamese sentiment classification problem, recently, there have been some studies on the Vietnamese sentiment classification problem without the word segmentation phase. For example, the research \cite{10.1007/978-3-030-82147-0_53} applied the fastText \cite{joulin2017bag} model for pre-processing and embedding the input data without the Vietnamese word segmentation phase. For other examples, the research \cite{9140757, 9335899, 9287650} studied the sentiment classification problem using the pre-trained multilingual language model mBERT \cite{devlin-etal-2019-bert}, which is not required Vietnamese word segmentation. To our best knowledge, the studies \cite{10.1007/978-3-030-82147-0_53,9140757, 9335899, 9287650} applied sub-words \cite{bojanowski2017enriching} method for the better handling unseen words. On the other hand, the fastText and mBERT models are made for multilingual purposes. Hence, the studies \cite{10.1007/978-3-030-82147-0_53,9140757, 9335899, 9287650} did not use the word segmentation phase for the Vietnamese sentiment classification problem. Indeed, the research \cite{9335912} used RDRsegmenter toolkit for data pre-processing before using the pre-trained monolingual PhoBERT model \cite{phobert}, which is made for Vietnamese and applied Byte-Pair Encoding (BPE) method \cite{sennrich-etal-2016-neural} for sub-word representations for Vietnamese.&#10;&#10;From the above observations, we have to admit that word segmentation is crucial for Vietnamese syntax analysis tasks, including part-of-speech tagging, constituency parsing, dependency parsing, and semantic parsing. That strongly motivates many proposed Vietnamese word segmentation methods in prior studies. On the Vietnamese Treebank corpus for word segmentation \cite{nguyen-etal-2009-building}, the highest F-score is 98.31\% achieved by the span labeling approach \cite{spanviws} using XLM-RoBERTa \cite{conneau-etal-2020-unsupervised}, which is very slow when inference on CPU device. However, the well-known Vietnamese word segmenter toolkit RDRsegmenter achieved the F-score of 97.90\% by the rule approach with a fast speed for inference. In addition, from the above observations, word segmentation is used widely in many prior studies on Vietnamese sentiment classification, while some studies did not use word segmentation. Consequently, we have raised a research question, ``Is word segmentation necessary for Vietnamese sentiment classification?''&#10;&#10;To attempt to answer the question above, we used four Vietnamese word segmentation toolkits. Firstly, we chose the fast and accurate Vietnamese word segmentation toolkit RDRsegmenter \cite{datnq1}. Secondly, we chose the Vietnamese word segmentation toolkit uitnlp \cite{vund2018}, which is proposed for ambiguity reduction and suffix capture. Lastly, we chose two well-known Vietnamese word segmentation toolkits, including pyvi and underthesea scientifically unpublished up to now. Regarding corpora, we chose two related to the Vietnamese sentiment classification problem, including the VLSP2016-SA corpus \cite{vlsp2016sa} of technical article reviews from the news and social media and the UIT-VSFC corpus of educational survey \cite{vsfc}. Regarding classifiers, we chose two traditional classifiers as baselines, including Naive Bayes (NB) and Support Vector Machines (SVMs) and the recent modern text encoder, namely, the Structured State Space Sequence model (S4) \cite{s4}.&#10;&#10;In summary our contributions are the following:&#10;\begin{itemize}&#10; \item Five pre-trained monolingual S4-based language models for Vietnamese, including one model without word segmentation, and four models using RDRsegmenter \cite{datnq1}, uitnlp \cite{vund2018}, pyvi, or underthesea toolkits in the pre-processing data phase.&#10; \item According to extensive experimental results on two corpora, including the VLSP2016-SA corpus of technical article reviews from the news and social media and the UIT-VSFC corpus of the educational survey, we have two suggestions:&#10; \begin{itemize}&#10; \item Using traditional classifiers like Naive Bayes or Support Vector Machines, word segmentation maybe not be necessary for the Vietnamese sentiment classification corpus, which comes from the social domain.&#10; \item Word segmentation is necessary for Vietnamese sentiment classification when word segmentation is considered pre-processing before using the BPE method and feeding into the deep learning model. By this way, the RDRsegmenter is the stable toolkit for word segmentation among the uitnlp, pyvi, and underthesea toolkits.&#10; \end{itemize}&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Word Segmentation, Linguistics, Sentiment Classification, Artificial Intelligence, Natural Language Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="1812.11459" label="1812.11459">
        <attvalues>
          <attvalue for="0" value="A neural joint model for Vietnamese word segmentation, POS tagging and&#10;  dependency parsing" />
          <attvalue for="1" value="  We propose the first multi-task learning model for joint Vietnamese word&#10;segmentation, part-of-speech (POS) tagging and dependency parsing. In&#10;particular, our model extends the BIST graph-based dependency parser&#10;(Kiperwasser and Goldberg, 2016) with BiLSTM-CRF-based neural layers (Huang et&#10;al., 2015) for word segmentation and POS tagging. On Vietnamese benchmark&#10;datasets, experimental results show that our joint model obtains&#10;state-of-the-art or competitive performances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.06307" label="1709.06307">
        <attvalues>
          <attvalue for="0" value="A Fast and Accurate Vietnamese Word Segmenter" />
          <attvalue for="1" value="  We propose a novel approach to Vietnamese word segmentation. Our approach is&#10;based on the Single Classification Ripple Down Rules methodology (Compton and&#10;Jansen, 1990), where rules are stored in an exception structure and new rules&#10;are only added to correct segmentation errors given by existing rules.&#10;Experimental results on the benchmark Vietnamese treebank show that our&#10;approach outperforms previous state-of-the-art approaches JVnSegmenter,&#10;vnTokenizer, DongDu and UETsegmenter in terms of both accuracy and performance&#10;speed. Our code is open-source and available at:&#10;https://github.com/datquocnguyen/RDRsegmenter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.00156" label="2110.00156">
        <attvalues>
          <attvalue for="0" value="Span Labeling Approach for Vietnamese and Chinese Word Segmentation" />
          <attvalue for="1" value="  In this paper, we propose a span labeling approach to model n-gram&#10;information for Vietnamese word segmentation, namely SPAN SEG. We compare the&#10;span labeling approach with the conditional random field by using encoders with&#10;the same architecture. Since Vietnamese and Chinese have similar linguistic&#10;phenomena, we evaluated the proposed method on the Vietnamese treebank&#10;benchmark dataset and five Chinese benchmark datasets. Through our experimental&#10;results, the proposed approach SpanSeg achieves higher performance than the&#10;sequence tagging approach with the state-of-the-art F-score of 98.31% on the&#10;Vietnamese treebank benchmark, when they both apply the contextual pre-trained&#10;language model XLM-RoBERTa and the predicted word boundary information.&#10;Besides, we do fine-tuning experiments for the span labeling approach on BERT&#10;and ZEN pre-trained language model for Chinese with fewer parameters, faster&#10;inference time, and competitive or higher F-scores than the previous&#10;state-of-the-art approach, word segmentation with word-hood memory networks, on&#10;five Chinese benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.07804" label="2006.07804">
        <attvalues>
          <attvalue for="0" value="Vietnamese Word Segmentation with SVM: Ambiguity Reduction and Suffix&#10;  Capture" />
          <attvalue for="1" value="  In this paper, we approach Vietnamese word segmentation as a binary&#10;classification by using the Support Vector Machine classifier. We inherit&#10;features from prior works such as n-gram of syllables, n-gram of syllable&#10;types, and checking conjunction of adjacent syllables in the dictionary. We&#10;propose two novel ways to feature extraction, one to reduce the overlap&#10;ambiguity and the other to increase the ability to predict unknown words&#10;containing suffixes. Different from UETsegmenter and RDRsegmenter, two&#10;state-of-the-art Vietnamese word segmentation methods, we do not employ the&#10;longest matching algorithm as an initial processing step or any post-processing&#10;technique. According to experimental results on benchmark Vietnamese datasets,&#10;our proposed method obtained a better F1-score than the prior state-of-the-art&#10;methods UETsegmenter, and RDRsegmenter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.03163" label="2404.03163">
        <attvalues>
          <attvalue for="0" value="Uncertainty in Language Models: Assessment through Rank-Calibration" />
          <attvalue for="1" value="  Language Models (LMs) have shown promising performance in natural language&#10;generation. However, as LMs often generate incorrect or hallucinated responses,&#10;it is crucial to correctly quantify their uncertainty in responding to given&#10;inputs. In addition to verbalized confidence elicited via prompting, many&#10;uncertainty measures ($e.g.$, semantic entropy and affinity-graph-based&#10;measures) have been proposed. However, these measures can differ greatly, and&#10;it is unclear how to compare them, partly because they take values over&#10;different ranges ($e.g.$, $[0,\infty)$ or $[0,1]$). In this work, we address&#10;this issue by developing a novel and practical framework, termed&#10;$Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.&#10;Our key tenet is that higher uncertainty (or lower confidence) should imply&#10;lower generation quality, on average. Rank-calibration quantifies deviations&#10;from this ideal relationship in a principled manner, without requiring ad hoc&#10;binary thresholding of the correctness score ($e.g.$, ROUGE or METEOR). The&#10;broad applicability and the granular interpretability of our methods are&#10;demonstrated empirically.&#10;" />
          <attvalue for="2" value="&#10;Language Models (LMs), especially&#10;Large Language Models (LLMs), have shown promising performance in Natural Language Generation (NLG). &#10;These models, fitted on huge text corpora, can produce responses resembling those of humans~\cite{touvron2023llama2,gpt-4}. &#10;However, since LMs often generate wrong or hallucinated responses~\cite{weidinger2021ethical,xiao2021hallucination,huang2024one}, &#10;it is crucial to correctly quantify their level of uncertainty in responding to particular inputs.&#10;&#10;Uncertainty quantification is well-explored in supervised learning, specifically in &#10;classification~\cite[e.g.,][etc]{lichtenstein1977calibration,gal2016dropout,lakshminarayanan2017simple}.&#10;In classification,&#10;a confidence measure&#10;is an estimate&#10;of the probability that the predicted class $\widehat Y$ matches the true class label $Y$~\cite{lichtenstein1977calibration,lee2023t}. &#10;A confidence measure $C$&#10;is considered calibrated &#10;if it reflects the probability of correct prediction, i.e., &#10;$\mathbb{P}(\widehat Y = Y\mid C) = C$, for all values in $C$'s range. &#10;The Expected Calibration Error~\ref{eqn:ece} measures the miscalibration of a confidence measure~\cite{frank2015regression,naeini2015obtaining}:&#10;\begin{equation}&#10; \mathbb{E}_{C} [|\mathbb{P}(\widehat Y = Y \mid C) -C|].\tag{ECE}&#10;\end{equation}&#10;&#10;In classification, confidence measures are predominantly built on model logits~\cite{guo2017calibration,kull2019beyond}.&#10;However, these methods are less suitable for NLG tasks. &#10;First, the label space is often too large to assess correctness via $\widehat Y= Y$, since LMs produce potentially long textual responses $\widehat Y$ for any given input. &#10;Second, for LMs, logits encode &#10;the likelihood of selecting the next token and do not necessarily capture linguistic sense~\cite{mielke2022reducing}. &#10;Third, even hand-crafted prompts intended to make LMs express confidence explicitly may not lead to reliable confidence values because elicitation is heavily tied to prompt formats~\cite{zhao2021calibrate,xiong2023llms}.&#10;&#10;Recent works have studied uncertainty measures as an alternative to confidence measures.&#10;These capture the ``dispersion'' of an LMs' potential outputs for a fixed input. &#10;\cite{kuhn2023semantic} introduce semantic entropy, which incorporates linguistic invariances arising from the shared meaning of generated responses. \cite{lin2023generating} extend semantic entropy by leveraging the affinity matrices induced by entailment scores of generated outputs. &#10;Further,&#10;\cite{chen2024inside} characterize differential entropy in the&#10;embedding space with EigenScore, via the covariance of embeddings of potential responses. &#10;&#10;Uncertainty measures are more general and arguably more principled&#10;than confidence measures for LMs, &#10;but they lack a universal assessment metric such as ECE. &#10;A key issue is that uncertainty measures are not necessarily commensurate. &#10;For instance, the semantic entropy \cite{kuhn2023semantic} can take arbitrarily large&#10;positive values, whereas the EigV measure of \cite{lin2023generating} depends on the number of responses generated. &#10;This makes it difficult to understand, evaluate, and compare uncertainty measures via a unified lens.&#10;&#10;This paper develops a principled framework&#10;to assess the quality of uncertainty and confidence measures for LMs.&#10;We provide a novel and practical framework, termed Rank-Calibration.&#10;Specifically, our contributions are as follows.&#10;\begin{itemize}[leftmargin=0.2in]&#10; \item We mathematically formalize the assessment of uncertainty/confidence measures for LMs in NLG tasks, going beyond binary correctness.&#10;&#10;\item We demonstrate empirically that existing assessment metrics (e.g., AUROC, ECE, etc) have several limitations, including a heavy dependence on the LM's performance, instability caused by ad hoc binarization of correctness scores, and incompatibility with diverse uncertainty ranges. &#10;&#10;\item We address these limitations by starting from a basic principle: lower uncertainty/higher confidence should indicate higher-quality generation. We thus propose assessing uncertainty measures in terms of rank-calibration and introduce a suitable metric, the Rank-Calibration Error \ref{eqn:rank-ece}.&#10;&#10;\item To make rank-calibration practical, we introduce the \ref{eqn:empirical-erce}---an estimate of \ref{eqn:rank-ece} based on a finite dataset. &#10; Moreover, we introduce novel indication diagrams, previewed in Fig.~\ref{fig:chat_trivia}, that intuitively visualize the deviation of any uncertainty/confidence measure from the monotonicity required for rank-calibration.&#10;&#10;\item We experimentally demonstrate the broader applicability and granular interpretability of our proposed methods. Comprehensive ablation studies are conducted to examine its robustness.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Uncertainty, Computer Science, Linguistics, Mathematics, Statistics, Natural Language Generation, Confidence Quantification" />
        </attvalues>
      </node>
      <node id="2405.19544" label="2405.19544">
        <attvalues>
          <attvalue for="0" value="One-Shot Safety Alignment for Large Language Models via Optimal&#10;  Dualization" />
          <attvalue for="1" value="  The growing safety concerns surrounding Large Language Models (LLMs) raise an&#10;urgent need to align them with diverse human preferences to simultaneously&#10;enhance their helpfulness and safety. A promising approach is to enforce safety&#10;constraints through Reinforcement Learning from Human Feedback (RLHF). For such&#10;constrained RLHF, common Lagrangian-based primal-dual policy optimization&#10;methods are computationally expensive and often unstable. This paper presents a&#10;dualization perspective that reduces constrained alignment to an equivalent&#10;unconstrained alignment problem. We do so by pre-optimizing a smooth and convex&#10;dual function that has a closed form. This shortcut eliminates the need for&#10;cumbersome primal-dual policy iterations, thus greatly reducing the&#10;computational burden and improving training stability. Our strategy leads to&#10;two practical algorithms in model-based and preference-based scenarios (MoCAN&#10;and PeCAN, respectively). A broad range of experiments demonstrate the&#10;effectiveness of our methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.01474" label="1612.01474">
        <attvalues>
          <attvalue for="0" value="Simple and Scalable Predictive Uncertainty Estimation using Deep&#10;  Ensembles" />
          <attvalue for="1" value="  Deep neural networks (NNs) are powerful black box predictors that have&#10;recently achieved impressive performance on a wide spectrum of tasks.&#10;Quantifying predictive uncertainty in NNs is a challenging and yet unsolved&#10;problem. Bayesian NNs, which learn a distribution over weights, are currently&#10;the state-of-the-art for estimating predictive uncertainty; however these&#10;require significant modifications to the training procedure and are&#10;computationally expensive compared to standard (non-Bayesian) NNs. We propose&#10;an alternative to Bayesian NNs that is simple to implement, readily&#10;parallelizable, requires very little hyperparameter tuning, and yields high&#10;quality predictive uncertainty estimates. Through a series of experiments on&#10;classification and regression benchmarks, we demonstrate that our method&#10;produces well-calibrated uncertainty estimates which are as good or better than&#10;approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate&#10;the predictive uncertainty on test examples from known and unknown&#10;distributions, and show that our method is able to express higher uncertainty&#10;on out-of-distribution examples. We demonstrate the scalability of our method&#10;by evaluating predictive uncertainty estimates on ImageNet.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.01850" label="2203.01850">
        <attvalues>
          <attvalue for="0" value="T-Cal: An optimal test for the calibration of predictive models" />
          <attvalue for="1" value="  The prediction accuracy of machine learning methods is steadily increasing,&#10;but the calibration of their uncertainty predictions poses a significant&#10;challenge. Numerous works focus on obtaining well-calibrated predictive models,&#10;but less is known about reliably assessing model calibration. This limits our&#10;ability to know when algorithms for improving calibration have a real effect,&#10;and when their improvements are merely artifacts due to random noise in finite&#10;datasets. In this work, we consider detecting mis-calibration of predictive&#10;models using a finite validation dataset as a hypothesis testing problem. The&#10;null hypothesis is that the predictive model is calibrated, while the&#10;alternative hypothesis is that the deviation from calibration is sufficiently&#10;large.&#10;  We find that detecting mis-calibration is only possible when the conditional&#10;probabilities of the classes are sufficiently smooth functions of the&#10;predictions. When the conditional class probabilities are H\&quot;older continuous,&#10;we propose T-Cal, a minimax optimal test for calibration based on a debiased&#10;plug-in estimator of the $\ell_2$-Expected Calibration Error (ECE). We further&#10;propose Adaptive T-Cal, a version that is adaptive to unknown smoothness. We&#10;verify our theoretical findings with a broad range of experiments, including&#10;with several popular deep neural net architectures and several standard&#10;post-hoc calibration methods. T-Cal is a practical general-purpose tool, which&#10;-- combined with classical tests for discrete-valued predictors -- can be used&#10;to test the calibration of virtually any probabilistic classification method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.03744" label="2402.03744">
        <attvalues>
          <attvalue for="0" value="INSIDE: LLMs' Internal States Retain the Power of Hallucination&#10;  Detection" />
          <attvalue for="1" value="  Knowledge hallucination have raised widespread concerns for the security and&#10;reliability of deployed LLMs. Previous efforts in detecting hallucinations have&#10;been employed at logit-level uncertainty estimation or language-level&#10;self-consistency evaluation, where the semantic information is inevitably lost&#10;during the token-decoding procedure. Thus, we propose to explore the dense&#10;semantic information retained within LLMs' \textbf{IN}ternal \textbf{S}tates&#10;for halluc\textbf{I}nation \textbf{DE}tection (\textbf{INSIDE}). In particular,&#10;a simple yet effective \textbf{EigenScore} metric is proposed to better&#10;evaluate responses' self-consistency, which exploits the eigenvalues of&#10;responses' covariance matrix to measure the semantic consistency/diversity in&#10;the dense embedding space. Furthermore, from the perspective of self-consistent&#10;hallucination detection, a test time feature clipping approach is explored to&#10;truncate extreme activations in the internal states, which reduces&#10;overconfident generations and potentially benefits the detection of&#10;overconfident hallucinations. Extensive experiments and ablation studies are&#10;performed on several popular LLMs and question-answering (QA) benchmarks,&#10;showing the effectiveness of our proposal.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.06495" label="2401.06495">
        <attvalues>
          <attvalue for="0" value="An investigation of structures responsible for gender bias in BERT and&#10;  DistilBERT" />
          <attvalue for="1" value="  In recent years, large Transformer-based Pre-trained Language Models (PLM)&#10;have changed the Natural Language Processing (NLP) landscape, by pushing the&#10;performance boundaries of the state-of-the-art on a wide variety of tasks.&#10;However, this performance gain goes along with an increase in complexity, and&#10;as a result, the size of such models (up to billions of parameters) represents&#10;a constraint for their deployment on embedded devices or short-inference time&#10;tasks. To cope with this situation, compressed models emerged (e.g.&#10;DistilBERT), democratizing their usage in a growing number of applications that&#10;impact our daily lives. A crucial issue is the fairness of the predictions made&#10;by both PLMs and their distilled counterparts. In this paper, we propose an&#10;empirical exploration of this problem by formalizing two questions: (1) Can we&#10;identify the neural mechanism(s) responsible for gender bias in BERT (and by&#10;extension DistilBERT)? (2) Does distillation tend to accentuate or mitigate&#10;gender bias (e.g. is DistilBERT more prone to gender bias than its uncompressed&#10;version, BERT)? Our findings are the following: (I) one cannot identify a&#10;specific layer that produces bias; (II) every attention head uniformly encodes&#10;bias; except in the context of underrepresented classes with a high imbalance&#10;of the sensitive attribute; (III) this subset of heads is different as we&#10;re-fine tune the network; (IV) bias is more homogeneously produced by the heads&#10;in the distilled model.&#10;" />
          <attvalue for="2" value="&#10;&#10;The introduction of large Pre-trained Language Models (PLM) has marked an important paradigm shift in Natural Language Processing (NLP). It leads to unprecedented progress in tasks such as machine translation, document classification \cite{devlin2018bert}, and multitasks text generation \cite{radford2019language}. The strength of these approaches lies in their ability to produce contextual representations. &#10;They have been initially based on Recurrent Neural Networks (RNN) \cite{dai2015} and they have gradually integrated the Transformers model \cite{vaswani2017attention} as is the case for GPT3 \cite{radford2019language} or BERT \cite{devlin2018bert}, for example. Compared to RNNs, Transformers can be parallelized, which opens the way, on one hand, to the use of ever-increasing training corpus (for example, GPT3 is trained on 45TB of data - almost the entire public web), and on the other hand, to the design of increasingly complex architectures (e.g., BERT large comprises 345 million parameters, BERT base 110 million).&#10;In a nutshell, Transformers~\cite{vaswani2017attention} are founded on three key innovations: positional encoding, scaled dot product attention, and multi-head attention (we will come back to these elements in more detail in Section~\ref{sec:background}). As a result of a combination of all these elements, Transformers can learn an internal understanding of language automatically from the data. Despite their good performance on many different tasks, the use of these models in so-called sensitive applications or areas raised concerns over the past couple of years. Indeed, when decisions have an impact on individuals, for example in the medical and legal domains \cite{demner2009can} or human resources \cite{jatoba2019evolution}, it becomes crucial to study the fairness of these models. &#10;&#10;The core definition of fairness is still a hotly debated topic in the scientific community. In our work, we adopt the following commonly accepted definition \cite{mehrabi2021survey}: fairness refers to the absence of any prejudice or favoritism towards an individual or a group based on their intrinsic or acquired traits. In machine learning, we assume that unfairness is the result of biased predictions (prejudice or favoritism), which are defined as elements that conduct a model to treat groups of individuals conditionally on some particular protected attributes, such as gender, race, or sexual orientation. &#10;&#10;As an example, in human resources, the NLP-based recruitment task consists in analyzing and then selecting the relevant candidates. A lack of diversity inherent to the data, for instance, a corpus containing a large majority of male profiles (i.e. sample bias), will cause the model to maintain and accentuate a gender bias~\cite{swinger2019biases}.&#10;When handling simple linear models trained on reasonable size corpora, creating safeguards to avoid this type of bias is conceivable. &#10;With PLM, the characteristics that allow them to perform so well are numerous: the size of their training corpus, the number of parameters, and their ability to infer a fine-grained semantic from the data. However, they are also what make it difficult to prevent them from encoding societal biases~\cite{bender2021dangers}.&#10;&#10;Related Works&#10;&#10;Several recent studies highlight fairness issues raised by models based on the Transformer architecture. These issues are observed in different levels of the NLP pipeline: text encoding~\cite{basta2019evaluating,kurita2019measuring}, during the fine-tuning process~\cite{delobell2021measuringfairness}, or simply as the potential harm caused on downstream tasks~\cite{kurita2019measuring}, with dedicated studies on language generation \cite{sheng2019woman}, document classification \cite{bhardwaj2020investigating}, toxicity detection, and sentiment analysis \cite{hutchinson2020social}. Besides measuring the fairness issue, locating the neural mechanism responsible for these issues is largely understudied and unsolved -- locating such mechanisms would unlock the possibility for counter-measures in neural architectures. &#10;At the same time, a segment of research focused on compressing these large pre-trained models to attain similar performances with fewer parameters, so that running these models is more sustainable and more cost-effective. Several model compression techniques have been proposed, as discussed in~\cite{Gupta22tkdd} and namely the following compression families : pruning, quantization and distillation. The primer \cite{lecun1989optimal} increases the speed and generalization capacities by removing the less important model's weights with regard to the task, while quantization approximates the model’s weights to reduce its complexity (e.g. reducing the numerical precision of the weights \cite{kim2021ibert}). Finally, distillation \cite{hinton2015distilling} consists in training a smaller model (called student model) to mimic the predictions of the large PLM to distill (teacher model). &#10;In the present work, we focus on this latter, approach.&#10;One of the earliest model, DistilBERT \cite{sanh2020distilbert} is able to reduce the number of parameters of BERT by 40\% while maintaining 96\% of accuracy in document classification. &#10;Looking at the impact of model distillation through fairness lenses has started to be investigated, mainly in the context of computer vision~\cite{hooker2020characterising,hooker2021compressed,lukasik2021teacher}. To summarize, their findings: i) compressed models impact underrepresented visual features directly related to bias, and ii) distilled models tend to accentuate discrimination already made by the teacher model. In NLP, fewer works have been conducted, and the conclusions are sometimes contradictory. &#10;While some works have shown that distilled versions of PLMs can exacerbate bias \cite{radford2018improving,delobelle2022fairdistillation}, other articles seem to reach an opposite conclusion \cite{xu2022can}; in this latter, authors state that model distillation acts as a regularization technique allowing bias reduction.&#10;&#10;Contribution Based on existing results, we start from the postulate that PLMs, and more specifically BERT, encode undesirable bias. With a focus on the task of document classification on the Bias in Bio dataset, our objective is to identify the inner structure of the neural network architecture that produce bias, both for BERT and its distilled version DistilBERT. To this end, we design and conduct a series of experiments to verify the relation between models' fairness and their intermediate representation or the attention they carry to the embedding in different data balance setting.&#10;&#10;Organisation&#10;Section~\ref{sec:background} provides background knowledge about BERT and DistilBERT. Section~\ref{section:protocol} presents the empirical protocols that we design. Section \ref{section:experiments} details the technical setting and shows the obtained results of our experiments. Finally, we conclude in Section \ref{section:conclusion} and provide several perspectives unlocked by our experiments.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Model Compression Techniques, Fairness in AI Predictions, Language Model Complexity" />
        </attvalues>
      </node>
      <node id="1908.09635" label="1908.09635">
        <attvalues>
          <attvalue for="0" value="A Survey on Bias and Fairness in Machine Learning" />
          <attvalue for="1" value="  With the widespread use of AI systems and applications in our everyday lives,&#10;it is important to take fairness issues into consideration while designing and&#10;engineering these types of systems. Such systems can be used in many sensitive&#10;environments to make important and life-changing decisions; thus, it is crucial&#10;to ensure that the decisions do not reflect discriminatory behavior toward&#10;certain groups or populations. We have recently seen work in machine learning,&#10;natural language processing, and deep learning that addresses such challenges&#10;in different subdomains. With the commercialization of these systems,&#10;researchers are becoming aware of the biases that these applications can&#10;contain and have attempted to address them. In this survey we investigated&#10;different real-world applications that have shown biases in various ways, and&#10;we listed different sources of biases that can affect AI applications. We then&#10;created a taxonomy for fairness definitions that machine learning researchers&#10;have defined in order to avoid the existing bias in AI systems. In addition to&#10;that, we examined different domains and subdomains in AI showing what&#10;researchers have observed with regard to unfair outcomes in the&#10;state-of-the-art methods and how they have tried to address them. There are&#10;still many future directions and solutions that can be taken to mitigate the&#10;problem of bias in AI systems. We are hoping that this survey will motivate&#10;researchers to tackle these issues in the near future by observing existing&#10;work in their respective fields.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.08769" label="1812.08769">
        <attvalues>
          <attvalue for="0" value="What are the biases in my word embedding?" />
          <attvalue for="1" value="  This paper presents an algorithm for enumerating biases in word embeddings.&#10;The algorithm exposes a large number of offensive associations related to&#10;sensitive features such as race and gender on publicly available embeddings,&#10;including a supposedly &quot;debiased&quot; embedding. These biases are concerning in&#10;light of the widespread use of word embeddings. The associations are identified&#10;by geometric patterns in word embeddings that run parallel between people's&#10;names and common lower-case tokens. The algorithm is highly unsupervised: it&#10;does not even require the sensitive features to be pre-specified. This is&#10;desirable because: (a) many forms of discrimination--such as racial&#10;discrimination--are linked to social constructs that may vary depending on the&#10;context, rather than to categories with fixed definitions; and (b) it makes it&#10;easier to identify biases against intersectional groups, which depend on&#10;combinations of sensitive features. The inputs to our algorithm are a list of&#10;target tokens, e.g. names, and a word embedding. It outputs a number of Word&#10;Embedding Association Tests (WEATs) that capture various biases present in the&#10;data. We illustrate the utility of our approach on publicly available word&#10;embeddings and lists of names, and evaluate its output using crowdsourcing. We&#10;also show how removing names may not remove potential proxy bias.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.08783" label="1904.08783">
        <attvalues>
          <attvalue for="0" value="Evaluating the Underlying Gender Bias in Contextualized Word Embeddings" />
          <attvalue for="1" value="  Gender bias is highly impacting natural language processing applications.&#10;Word embeddings have clearly been proven both to keep and amplify gender biases&#10;that are present in current data sources. Recently, contextualized word&#10;embeddings have enhanced previous word embedding techniques by computing word&#10;vector representations dependent on the sentence they appear in.&#10;  In this paper, we study the impact of this conceptual change in the word&#10;embedding computation in relation with gender bias. Our analysis includes&#10;different measures previously applied in the literature to standard word&#10;embeddings. Our findings suggest that contextualized word embeddings are less&#10;biased than standard ones even when the latter are debiased.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00813" label="2005.00813">
        <attvalues>
          <attvalue for="0" value="Social Biases in NLP Models as Barriers for Persons with Disabilities" />
          <attvalue for="1" value="  Building equitable and inclusive NLP technologies demands consideration of&#10;whether and how social attitudes are represented in ML models. In particular,&#10;representations encoded in models often inadvertently perpetuate undesirable&#10;social biases from the data on which they are trained. In this paper, we&#10;present evidence of such undesirable biases towards mentions of disability in&#10;two different English language models: toxicity prediction and sentiment&#10;analysis. Next, we demonstrate that the neural embeddings that are the critical&#10;first step in most NLP pipelines similarly contain undesirable biases towards&#10;mentions of disability. We end by highlighting topical biases in the discourse&#10;about disability which may contribute to the observed model biases; for&#10;instance, gun violence, homelessness, and drug addiction are over-represented&#10;in texts discussing mental illness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.05221" label="2008.05221">
        <attvalues>
          <attvalue for="0" value="Compression of Deep Learning Models for Text: A Survey" />
          <attvalue for="1" value="  In recent years, the fields of natural language processing (NLP) and&#10;information retrieval (IR) have made tremendous progress thanksto deep learning&#10;models like Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs) and&#10;Long Short-Term Memory (LSTMs)networks, and Transformer [120] based models like&#10;Bidirectional Encoder Representations from Transformers (BERT) [24],&#10;GenerativePre-training Transformer (GPT-2) [94], Multi-task Deep Neural Network&#10;(MT-DNN) [73], Extra-Long Network (XLNet) [134], Text-to-text transfer&#10;transformer (T5) [95], T-NLG [98] and GShard [63]. But these models are&#10;humongous in size. On the other hand,real world applications demand small model&#10;size, low response times and low computational power wattage. In this survey,&#10;wediscuss six different types of methods (Pruning, Quantization, Knowledge&#10;Distillation, Parameter Sharing, Tensor Decomposition, andSub-quadratic&#10;Transformer based methods) for compression of such models to enable their&#10;deployment in real industry NLP projects.Given the critical need of building&#10;applications with efficient and small models, and the large amount of recently&#10;published work inthis area, we believe that this survey organizes the plethora&#10;of work done by the 'deep learning for NLP' community in the past fewyears and&#10;presents it as a coherent story.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.01321" label="2101.01321">
        <attvalues>
          <attvalue for="0" value="I-BERT: Integer-only BERT Quantization" />
          <attvalue for="1" value="  Transformer based models, like BERT and RoBERTa, have achieved&#10;state-of-the-art results in many Natural Language Processing tasks. However,&#10;their memory footprint, inference latency, and power consumption are&#10;prohibitive efficient inference at the edge, and even at the data center. While&#10;quantization can be a viable solution for this, previous work on quantizing&#10;Transformer based models use floating-point arithmetic during inference, which&#10;cannot efficiently utilize integer-only logical units such as the recent Turing&#10;Tensor Cores, or traditional integer-only ARM processors. In this work, we&#10;propose I-BERT, a novel quantization scheme for Transformer based models that&#10;quantizes the entire inference with integer-only arithmetic. Based on&#10;lightweight integer-only approximation methods for nonlinear operations, e.g.,&#10;GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end&#10;integer-only BERT inference without any floating point calculation. We evaluate&#10;our approach on GLUE downstream tasks using RoBERTa-Base/Large. We show that&#10;for both cases, I-BERT achieves similar (and slightly higher) accuracy as&#10;compared to the full-precision baseline. Furthermore, our preliminary&#10;implementation of I-BERT shows a speedup of 2.4-4.0x for INT8 inference on a T4&#10;GPU system as compared to FP32 inference. The framework has been developed in&#10;PyTorch and has been open-sourced.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.05248" label="1911.05248">
        <attvalues>
          <attvalue for="0" value="What Do Compressed Deep Neural Networks Forget?" />
          <attvalue for="1" value="  Deep neural network pruning and quantization techniques have demonstrated it&#10;is possible to achieve high levels of compression with surprisingly little&#10;degradation to test set accuracy. However, this measure of performance conceals&#10;significant differences in how different classes and images are impacted by&#10;model compression techniques. We find that models with radically different&#10;numbers of weights have comparable top-line performance metrics but diverge&#10;considerably in behavior on a narrow subset of the dataset. This small subset&#10;of data points, which we term Pruning Identified Exemplars (PIEs) are&#10;systematically more impacted by the introduction of sparsity. Compression&#10;disproportionately impacts model performance on the underrepresented long-tail&#10;of the data distribution. PIEs over-index on atypical or noisy images that are&#10;far more challenging for both humans and algorithms to classify. Our work&#10;provides intuition into the role of capacity in deep neural networks and the&#10;trade-offs incurred by compression. An understanding of this disparate impact&#10;is critical given the widespread deployment of compressed models in the wild.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.04546" label="2207.04546">
        <attvalues>
          <attvalue for="0" value="FairDistillation: Mitigating Stereotyping in Language Models" />
          <attvalue for="1" value="  Large pre-trained language models are successfully being used in a variety of&#10;tasks, across many languages. With this ever-increasing usage, the risk of&#10;harmful side effects also rises, for example by reproducing and reinforcing&#10;stereotypes. However, detecting and mitigating these harms is difficult to do&#10;in general and becomes computationally expensive when tackling multiple&#10;languages or when considering different biases. To address this, we present&#10;FairDistillation: a cross-lingual method based on knowledge distillation to&#10;construct smaller language models while controlling for specific biases. We&#10;found that our distillation method does not negatively affect the downstream&#10;performance on most tasks and successfully mitigates stereotyping and&#10;representational harms. We demonstrate that FairDistillation can create fairer&#10;language models at a considerably lower cost than alternative approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.01503" label="2307.01503">
        <attvalues>
          <attvalue for="0" value="On Evaluating and Mitigating Gender Biases in Multilingual Settings" />
          <attvalue for="1" value="  While understanding and removing gender biases in language models has been a&#10;long-standing problem in Natural Language Processing, prior research work has&#10;primarily been limited to English. In this work, we investigate some of the&#10;challenges with evaluating and mitigating biases in multilingual settings which&#10;stem from a lack of existing benchmarks and resources for bias evaluation&#10;beyond English especially for non-western context. In this paper, we first&#10;create a benchmark for evaluating gender biases in pre-trained masked language&#10;models by extending DisCo to different Indian languages using human&#10;annotations. We extend various debiasing methods to work beyond English and&#10;evaluate their effectiveness for SOTA massively multilingual models on our&#10;proposed metric. Overall, our work highlights the challenges that arise while&#10;studying social biases in multilingual settings and provides resources as well&#10;as mitigation techniques to take a step toward scaling to more languages.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs) \cite{devlin-etal-2019-bert, brown-etal-2020-language, raffel-etal-2020-exploring} have obtained impressive performance on a wide range of NLP tasks showing great potential in several downstream applications for real world impact. However, these models have shown to be prone to picking up unwanted correlations and stereotypes from the pre-training data \cite{sheng-etal-2019-woman, kurita-etal-2019-measuring, hutchinson-etal-2020-social} which, can perpetuate harmful biases for people belonging to marginalized groups. While there has been a great deal of interest in understanding and mitigating such biases in LLMs \cite{nadeem-etal-2021-stereoset, schick-etal-2021-self, meade-etal-2022-empirical}, the focus of such studies has primarily been on English.&#10;&#10;While Massively Multilingual Language Models \cite{devlin-etal-2019-bert, conneau-etal-2020-unsupervised, xue-etal-2021-mt5}, have shown impressive performances across a wide range of languages, especially with their surprising effectiveness at zero-shot cross-lingual transfer, there still exists a lack of focused research to evaluate and mitigate the biases that exist in these models. This can lead to a lack of inclusive and responsible technologies for groups whose native language is not English and can also lead to the dissemination of stereotypes and the widening of existing cultural gaps.&#10;&#10;Past work on evaluating and mitigating biases in multilingual models has mostly been concerned with gender bias in cross-lingual word embeddings \cite{zhao-etal-2020-gender, bansal-etal-2021-debiasing} which fails to account for contextual information \cite{kurita-etal-2019-measuring, delobelle-etal-2022-measuring}, making them unreliable for LLMs. Other methods for estimating biases in contextualized representations involve Multilingual Bias Evaluation \cite[MBE]{kaneko-etal-2022-gender}, which utilizes parallel translation corpora in different languages that might lack non-western cultural contexts \cite{talat-etal-2022-reap}. For debiasing LLMs, \cite{lauscher-etal-2021-sustainable-modular} proposed an adapter \cite{houlsby-etal-2019-parameter} based approach. However, the biases are measured in the word representations and only English data was used for debiasing, missing out on cultural context for other languages.&#10;&#10;To address these concerns, we make the following key contributions in our work. First, we extend the DisCo metric \cite{webster2020measuring} by creating human-corrected templates for 6 Indian languages. &#10;DisCo takes sentence-level context while measuring bias and our templates are largely culturally agnostic making them more generally applicable. Second, we extend existing debiasing strategies like Counterfactual Data Augmentation \cite{zhao-etal-2018-gender} and Self-Debiasing \cite{schick-etal-2021-self} to mitigate gender biases across languages in Masked Language Models (MLMs). &#10;&#10;Finally, we also evaluate the transferability of debiasing MLMs from one source language to other target languages and observe limited transfer from English to languages lacking western context. However, we do observe that typologically and culturally similar languages aid each other in reducing gender bias. While there have been multiple studies on measuring biases in multilingual models, previous work has not explored mitigating gender biases from these models on multiple languages and studying the transferability of debiasing across different languages. This is especially true while using non-embedding based approaches for evaluation and debiasing. To the best of our knowledge, ours is the first work to debias multilingual LLMs for different languages and measure the cross-lingual transfer for gender bias mitigation. To encourage future research in this area, we will release our code and datasets publically.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Bias, Social Bias Mitigation, Artificial Intelligence, Multilingual Processing, Social Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2303.01191" label="2303.01191">
        <attvalues>
          <attvalue for="0" value="Denoising-based UNMT is more robust to word-order divergence than&#10;  MASS-based UNMT" />
          <attvalue for="1" value="  We aim to investigate whether UNMT approaches with self-supervised&#10;pre-training are robust to word-order divergence between language pairs. We&#10;achieve this by comparing two models pre-trained with the same self-supervised&#10;pre-training objective. The first model is trained on language pairs with&#10;different word-orders, and the second model is trained on the same language&#10;pairs with source language re-ordered to match the word-order of the target&#10;language. Ideally, UNMT approaches which are robust to word-order divergence&#10;should exhibit no visible performance difference between the two&#10;configurations. In this paper, we investigate two such self-supervised&#10;pre-training based UNMT approaches, namely Masked Sequence-to-Sequence&#10;Pre-Training, (MASS) (which does not have shuffling noise) and Denoising&#10;AutoEncoder (DAE), (which has shuffling noise).&#10;  We experiment with five English$\rightarrow$Indic language pairs, i.e.,&#10;en-hi, en-bn, en-gu, en-kn, and en-ta) where word-order of the source language&#10;is SVO (Subject-Verb-Object), and the word-order of the target languages is SOV&#10;(Subject-Object-Verb). We observed that for these language pairs, DAE-based&#10;UNMT approach consistently outperforms MASS in terms of translation accuracies.&#10;Moreover, bridging the word-order gap using reordering improves the translation&#10;accuracy of MASS-based UNMT models, while it cannot improve the translation&#10;accuracy of DAE-based UNMT models. This observation indicates that DAE-based&#10;UNMT is more robust to word-order divergence than MASS-based UNMT.&#10;Word-shuffling noise in DAE approach could be the possible reason for the&#10;approach being robust to word-order divergence.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, self-supervised pre-training based approaches to Unsupervised Neural Machine Translation (UNMT) \cite{conneau2019cross,song2019mass,liu-etal-2020-multilingual-denoising} have become the norm. These UNMT models are trained in two stages viz (i) self-supervised pre-training and (ii) iterative back-translation. The self-supervised pre-training phase typically involves training an encoder-decoder transformer on the combined monolingual corpora of two or more languages using self-supervision. The second stage employs iterative back-translation to train the model for translation in all directions.&#10;&#10;The approaches differ in the self-supervised objective employed during the pre-training phase. For instance, \cite{conneau2019cross} employ Masked Language Modeling (MLM) objective to train an encoder. Here, some of the tokens in the input sequence are masked and the objective is to predict the masked tokens. MASS \cite{song2019mass} employs the masking of consecutive tokens and employs a decoder to generate the masked tokens. &#10; \cite{liu-etal-2020-multilingual-denoising} use Denoising Auto-Encoder (DAE) to pre-train the model. The noise functions considered are (a) removing spans of text and replacing them with a mask token and (b) sentence shuffling. &#10;&#10;While the above approaches show promising results for closely-related language-pairs, however, language divergence between source and target languages can adversely impact the benefits obtained from the joint-training, resulting in poor translation by UNMT models. &#10;&#10;Language divergence can be observed between two languages at various stages of NLP \cite{bhattacharyya2012natural}. Recently, several approaches have been proposed to address the issue of lexical divergence between languages \cite{chronopoulou-etal-2021-improving,banerjee-etal-2021-crosslingual, khatri2021simple}. However, the impact of word-order divergence on UNMT is still not explored. Previously, \cite{sun2021unsupervised} experimented with the iterative UNMT approach proposed by \cite{lample-etal-2018-phrase} which does not employ self-supervised pre-training and is found to be sensitive to word-order divergence.&#10;&#10;Recently, \cite{abdou-etal-2022-word} showed that shuffled language models such as DAE can still surprisingly exhibit information about naturalistic word order. &#10;&#10;Re-ordering (synonymous with pre-ordering) has been used as a tool in Machine Translation to bridge the word-order gap among languages. While \cite{du2017pre} found re-ordering redundant for NMT systems, \cite{zhao-etal-2018-exploiting} observed improvement from re-ordering by reducing the number of under-translation cases. In a transfer learning scenario, \cite{murthy2018addressing} observed improvement in result by re-ordering the assisting source language sentences to match the word-order of the actual source language before training the parent model.&#10;&#10;While the use of re-ordering tools sounds promising, re-ordering tools require either an (i) word re-ordered data to train a supervised re-ordering tool or ii) parser and re-ordering rules to use a rule-based system. Also, &#10;it is unclear if existing UNMT approaches are robust to word-order divergence or not. &#10;&#10;In this paper, we aim to investigate the effect of word-order divergence in state-of-the-art UNMT systems. In this direction, we compare MASS \cite{song2019mass} and DAE \cite{artetxe2018unsupervised} on language pairs with different word-orders. We experiment with five language pairs in which the word-order of the source language does not match the word-order of the target language. For each language-pair, we compare (i) UNMT model trained on the source$arrow$target language data (original data), with (ii) UNMT model trained on the re-ordered source $arrow$ target data (re-ordered data). Here, re-ordered source refers to the source sentences re-ordered to match the word-order of the target language. In the former case, word-order divergence is present while in the latter case word-order divergence is bridged. &#10;&#10;We observe from our experiments that there is no significant difference in terms of translation accuracy between DAE-based models trained on original data and re-ordered data. It signifies that DAE-based UNMT is robust to the word-order difference between the two languages compared to MASS-based UNMT. We also observe that in most language pairs, the UNMT model trained using DAE approach with original data produces better quality translations compared to other models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Pre-training, Unsupervised Machine Translation, Artificial Intelligence, Word Order Robustness" />
        </attvalues>
      </node>
      <node id="2103.10531" label="2103.10531">
        <attvalues>
          <attvalue for="0" value="Improving the Lexical Ability of Pretrained Language Models for&#10;  Unsupervised Neural Machine Translation" />
          <attvalue for="1" value="  Successful methods for unsupervised neural machine translation (UNMT) employ&#10;crosslingual pretraining via self-supervision, often in the form of a masked&#10;language modeling or a sequence generation task, which requires the model to&#10;align the lexical- and high-level representations of the two languages. While&#10;cross-lingual pretraining works for similar languages with abundant corpora, it&#10;performs poorly in low-resource and distant languages. Previous research has&#10;shown that this is because the representations are not sufficiently aligned. In&#10;this paper, we enhance the bilingual masked language model pretraining with&#10;lexical-level information by using type-level cross-lingual subword embeddings.&#10;Empirical results demonstrate improved performance both on UNMT (up to 4.5&#10;BLEU) and bilingual lexicon induction using our method compared to a UNMT&#10;baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;\cite{artetxe2017unsupervised,lample2017unsupervised} initialize unmt models with word-by-word translations, based on a bilingual lexicon inducted in an unsupervised way by the same monolingual data, or simply with cross-lingual embeddings.&#10;\cite{lample2018phrase} also use pretrained embeddings, learned on joint monolingual corpora of the two languages of interest, to initialize the embedding layer of the encoder-decoder.&#10; \cite{lample2019cross} remove pretrained embeddings from the unmt pipeline and align language distributions by simply pretraining a mlm on both languages, in order to learn a cross-lingual mapping. &#10;However, it has been shown that this pretraining method provides a weak alignment of the language distributions \cite{ren-etal-2019-explicit}. While that work identified as a cause the lack of sharing n-gram level cross-lingual information, we address the lack of cross-lingual information at the lexical level. &#10;&#10;Moreover, most prior work on unmt focuses on languages with abundant, high-quality monolingual corpora. In low-resource scenarios though, especially when the languages are not related, pretraining a cross-lingual mlm for unsupervised nmt does not yield good results \cite{guzman2019flores, chronopoulou2020reusing}. We propose a method that overcomes this issue by enhancing the mlm with cross-lingual lexical-level representations.&#10;&#10;Another line of work tries to enrich the representations of multilingual mlms with additional knowledge \cite{wang2020kadapter, pfeiffer-etal-2020-mad} without harming the already-learned representations. In our work, we identify lexical information as a source of knowledge that is missing from mlms, especially when it comes to low-resource languages. Surprisingly, static embeddings, such as fastText, largely outperform representations extracted by multilingual mlms in terms of cross-lingual lexical alignment \cite{Vulic2020ProbingPL}. Motivated by this, we aim to narrow the gap between the lexical representations of bilingual mlms and static embeddings, in order to achieve a higher translation quality, when transferring the mlm to an encoder-decoder unmt model. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.04995" label="2106.04995">
        <attvalues>
          <attvalue for="0" value="Crosslingual Embeddings are Essential in UNMT for Distant Languages: An&#10;  English to IndoAryan Case Study" />
          <attvalue for="1" value="  Recent advances in Unsupervised Neural Machine Translation (UNMT) have&#10;minimized the gap between supervised and unsupervised machine translation&#10;performance for closely related language pairs. However, the situation is very&#10;different for distant language pairs. Lack of lexical overlap and low syntactic&#10;similarities such as between English and Indo-Aryan languages leads to poor&#10;translation quality in existing UNMT systems. In this paper, we show that&#10;initializing the embedding layer of UNMT models with cross-lingual embeddings&#10;shows significant improvements in BLEU score over existing approaches with&#10;embeddings randomly initialized. Further, static embeddings (freezing the&#10;embedding layer weights) lead to better gains compared to updating the&#10;embedding layer weights during training (non-static). We experimented using&#10;Masked Sequence to Sequence (MASS) and Denoising Autoencoder (DAE) UNMT&#10;approaches for three distant language pairs. The proposed cross-lingual&#10;embedding initialization yields BLEU score improvement of as much as ten times&#10;over the baseline for English-Hindi, English-Bengali, and English-Gujarati. Our&#10;analysis shows the importance of cross-lingual embedding, comparisons between&#10;approaches, and the scope of improvements in these systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.00383" label="1811.00383">
        <attvalues>
          <attvalue for="0" value="Addressing word-order Divergence in Multilingual Neural Machine&#10;  Translation for extremely Low Resource Languages" />
          <attvalue for="1" value="  Transfer learning approaches for Neural Machine Translation (NMT) train a NMT&#10;model on the assisting-target language pair (parent model) which is later&#10;fine-tuned for the source-target language pair of interest (child model), with&#10;the target language being the same. In many cases, the assisting language has a&#10;different word order from the source language. We show that divergent word&#10;order adversely limits the benefits from transfer learning when little to no&#10;parallel corpus between the source and target language is available. To bridge&#10;this divergence, We propose to pre-order the assisting language sentence to&#10;match the word order of the source language and train the parent model. Our&#10;experiments on many language pairs show that bridging the word order gap leads&#10;to significant improvement in the translation quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.14739" label="2408.14739">
        <attvalues>
          <attvalue for="0" value="VoiceTailor: Lightweight Plug-In Adapter for Diffusion-Based&#10;  Personalized Text-to-Speech" />
          <attvalue for="1" value="  We propose VoiceTailor, a parameter-efficient speaker-adaptive text-to-speech&#10;(TTS) system, by equipping a pre-trained diffusion-based TTS model with a&#10;personalized adapter. VoiceTailor identifies pivotal modules that benefit from&#10;the adapter based on a weight change ratio analysis. We utilize Low-Rank&#10;Adaptation (LoRA) as a parameter-efficient adaptation method and incorporate&#10;the adapter into pivotal modules of the pre-trained diffusion decoder. To&#10;achieve powerful adaptation performance with few parameters, we explore various&#10;guidance techniques for speaker adaptation and investigate the best strategies&#10;to strengthen speaker information. VoiceTailor demonstrates comparable speaker&#10;adaptation performance to existing adaptive TTS models by fine-tuning only&#10;0.25\% of the total parameters. VoiceTailor shows strong robustness when&#10;adapting to a wide range of real-world speakers, as shown in the demo.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancements in deep generative models have led to improvements in adaptive text-to-speech (TTS), enabling models to generate a target speaker's voice from a given transcript and reference speech \cite{pmlr-v162-casanova22a, kim2022guidedtts, le2023voicebox}.&#10;Zero-shot approach \cite{pmlr-v162-casanova22a, le2023voicebox, wang2023neural, kim2023pflow, shen2024naturalspeech} for adaptive TTS eliminates the need for extra fine-tuning on reference audio for speaker adaptation.&#10;Despite its advantage of no further training, this approach generally requires large speech corpus during training to achieve high speaker similarity, and is comparatively less robust against unique out-of-distribution voices commonly encountered in real-world scenarios. &#10;&#10;One-shot approach, an alternative type of adaptive TTS, constructs personalized TTS by fine-tuning pre-trained multi-speaker TTS models with few reference speeches of target speaker \cite{pmlr-v162-casanova22a, DBLP:journals/corr/abs-2005-05642, Moss2020BOFFINTF, hsieh23_interspeech, NEURIPS2018_4559912e, chen2021adaspeech, 9414872}. &#10;To efficiently adapt to the target speaker, several studies fine-tuned a subset of the model's parameters \cite{DBLP:journals/corr/abs-2005-05642, Moss2020BOFFINTF, NEURIPS2018_4559912e, chen2021adaspeech, 9414872}, or leveraged adapter-based fine-tuning techniques \cite{hsieh23_interspeech} such as Low-Rank Adaptation (LoRA) \cite{hu2022lora} or prefix-tuning \cite{li-liang-2021-prefix}, which only fine-tune the parameters of newly integrated adapters.&#10;However, these works often fail to generate speech with high speaker similarity due to the limitations of the generative models used as decoder and typically require more than a minute of speech data for fine-tuning.&#10;&#10;Recently, inspired by successes of diffusion-based generative model \cite{DDPM} on fine-tuning-based personalized generation tasks \cite{Ruiz_2023_CVPR}, diffusion-based one-shot TTS models have been proposed \cite{kim2022guidedtts, kim23k_interspeech}. &#10;They leverage the diffusion model's adaptation performance to achieve high speaker similarity in personalized TTS task with as short as 5 to 10 seconds of reference speech. &#10;However, in contrast to other one-shot approaches, these works fine-tune all model parameters, resulting in parameter inefficiency.&#10;&#10;In this work, we introduce VoiceTailor, a parameter-efficient adaptive TTS model that requires fine-tuning only a subset of parameters from a diffusion-based pre-trained TTS model.&#10;We utilize a diffusion-based pre-trained TTS model and adopt a fine-tuning methodology following UnitSpeech \cite{kim23k_interspeech}. &#10;Inspired by the approaches in \cite{kumari2022customdiffusion, 10.5555/3495724.3497056}, we analyze the change ratio in the weights of each module in the model before and after fine-tuning and identify that attention modules play a crucial role in speaker adaptation. &#10;Based on this observation, VoiceTailor carefully integrates LoRA into the effective attention modules in the model and fine-tunes only the injected low-rank matrices for adaptation.&#10;&#10;We demonstrate that VoiceTailor achieves speaker adaptation performance comparable to the fully fine-tuned one-shot baseline by plugging in the small adapter with $0.25\%$ of the total parameters of the pre-trained model, which occupies approximately 1.3 MB of storage space.&#10;In addition, we systematically analyze the impact of various design choices and hyperparameters during the parameter-efficient adaptation stage.&#10;Furthermore, we investigate the best strategy from various guidance techniques in the inference stage. &#10;We illustrate VoiceTailor's robust performance in real-world scenarios by presenting a variety of samples, including those adapted for real-world speakers, on our demo page. Our contributions are as follows:&#10;&#10;\begin{itemize}&#10;\item To the best of our knowledge, this is the first work that systematically incorporates LoRA for diffusion-based speaker adaptive TTS that achieves high speaker similarity.&#10;\item VoiceTailor significantly reduces cost of adapting TTS to new speaker using 10 seconds of untranscribed speech with approximately 15 seconds of training time on a single GPU by utilizing $0.25\%$ of the model parameters.&#10;\item We compare and analyze various methods to enhance speaker information using LoRA modules and speaker classifier-free guidance and investigate the optimal strategy.&#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Parameter-Efficient Modeling, Linguistics, Speaker Adaptation Techniques, Text-to-Speech Systems, Signal Processing, Mathematics" />
        </attvalues>
      </node>
      <node id="2005.05642" label="2005.05642">
        <attvalues>
          <attvalue for="0" value="AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN" />
          <attvalue for="1" value="  This paper investigates how to leverage a DurIAN-based average model to&#10;enable a new speaker to have both accurate pronunciation and fluent&#10;cross-lingual speaking with very limited monolingual data. A weakness of the&#10;recently proposed end-to-end text-to-speech (TTS) systems is that robust&#10;alignment is hard to achieve, which hinders it to scale well with very limited&#10;data. To cope with this issue, we introduce AdaDurIAN by training an improved&#10;DurIAN-based average model and leverage it to few-shot learning with the shared&#10;speaker-independent content encoder across different speakers. Several few-shot&#10;learning tasks in our experiments show AdaDurIAN can outperform the baseline&#10;end-to-end system by a large margin. Subjective evaluations also show that&#10;AdaDurIAN yields higher mean opinion score (MOS) of naturalness and more&#10;preferences of speaker similarity. In addition, we also apply AdaDurIAN to&#10;emotion transfer tasks and demonstrate its promising performance.&#10;" />
          <attvalue for="2" value="&#10;The rise of deep learning~\cite{lecun2015deep} has made more complex sequence generation tasks~\cite{sutskever2014sequence,Wang2017TacotronTE,shen2018natural,oord2016wavenet,Kalchbrenner2018EfficientNA} feasible. Text-based generation of natural speech has been continuously investigated over the past decades. Concatenative synthesis with unit selection~\cite{hunt1996unit} and statistical parametric speech synthesis~\cite{zen2009statistical} were the state-of-the-art systems for many years. However, such systems require lots of human labour and are unsatisfactory for lacking naturalness. Recently, a sequence-to-sequence architecture, Tacotron~\cite{Wang2017TacotronTE,shen2018natural}, has greatly improved the naturalness and similarity of speech synthesis compared to traditional statistical parametric speech synthesis system~\cite{zen2009statistical}. Tacotron, usually followed by a traditional or neural vocoder~\cite{oord2016wavenet,Griffin1984SignalEF}, takes linguistic feature and speaker identity as input and generates mel-spectrogram as output. Unfortunately, when dealing with out-of-domain or abnormal texts inputs, Tacotron-like attention based end-to-end structures could render unacceptable errors, including skipping, repeating, long unexpected pause and attention collapse~\cite{shen2018natural,tiantencent}. More recently, stepwise monotonic attention (SMA)~\cite{He2019RobustSA} method, which is based on monotonic attention~\cite{raffel2017online}, was proposed to enforce strict constraint to meet the demand of locality, monotonicity and completeness in the speech synthesis process. &#10;&#10;As far as we know, building a naturally speaking TTS system requires at least ten hours of recording audio. Moreover, every audio utterance should be recorded in a professional recording studio and the transcribed phonemes should be evenly distributed. Preparing such a large amount of high-quality data with multiple speakers is impractical and extremely expensive. Typically, it's troublesome and unnecessary to let native Chinese speaker to say English if he knows little about English. Moreover, there is no chance to gather 10 hours training data for a specific person like a pop star. The only resources we can get are the limited talks or shows from TV. Therefore, utilizing a few minutes of audio and synthesizing arbitrary speech in target's voice remains a very important task. &#10;&#10;However, building TTS system with limited data often sacrifices quality and reliability~\cite{chung2019semi}. To scale the capacity for new speakers, we can adapt existing pre-trained multi-speaker system to generate new speakers' voice, which is a well-studied subject of few-shot learning~\cite{fink2005object,fei2006one} also known as speaker adaptation~\cite{yamagishi2009analysis,leggetter1995maximum}. There are mainly two approaches here: the first is just to update the new speaker embedding and combine it with linguistic feature as inputs to a TTS model~\cite{jia2018transfer,li2017deep}, which may require a very strong speaker encoder network trained by thousands of speakers~\cite{8462665}; the second is to fine-tune the entire multi-speaker network to select a optimal single-speaker model~\cite{arik2018neural,chen2018sample,9054301}. Although fine-tuning can combine the advantages of multiple speakers and achieve a new speaker's better performance, as we described before, end-to-end attention models such as Tacotron-like models may meet unpredictable instability and bad cross-lingual speaking in few-shot learning settings. To achieve naturalness and robustness in speech synthesis, FastSpeech~\cite{ren2019fastspeech} and duration informed attention network (DurIAN)~\cite{Yu2019DurIANDI} have been recently proposed to overcome the unexpected errors of end-to-end systems by combining duration information of traditional statistical parametric speech synthesis system~\cite{zen2009statistical}. The former FastSpeech is a non-autoregressive feed-forward framework without attention. The latter DurIAN, originally proposed for multi-modal speech synthesis, is an autoregressive framework which achieves robustness and naturalness by using skip state encoder and combining duration with windowed content-based attention~\cite{Bahdanau2015NeuralMT}. &#10;&#10;To improve the scalability of TTS in few-shot speaker adaptation, we introduce AdaDurIAN, an adaptive neural TTS system based on DurIAN, with the ability to synthesize natural cross-lingual speech in a new speaker's voice with just few minutes of monolingual data. We investigate it in three different aspects that have not been fully explored in previous work. First, we employ sequences of phoneme and tone (or stress) to achieve a robust speaker-independent content encoder, and incorporate the concatenated representation of speaker characteristics into the output states of content encoder. Second, instead of fine-tuning weights of the whole architecture, we found a key aspect that only fine-tuning the speaker embedding and decoder network leads to fewer pronunciation errors. Last, to generate the smooth mel spectrograms in a streaming inference manner, we adopt a time-delayed LSTM post-net instead of a global CBHG-like~\cite{Wang2017TacotronTE} module. Through various evaluations, our proposed AdaDurIAN significantly surpasses the Tacotron-like model~\cite{He2019RobustSA} in terms of naturalness, speaker similarity and cross-lingual speaking, and also shows its promising performance in few-shot emotion transfer tasks.&#10;&#10;The rest of this paper is organized as follows. Section~\ref{section2} describes the detailed architecture of AdaDurIAN and the speaker adaptation strategy. The experiment setup and evaluations are presented in Section $3$. Concluding remarks are summarized in the final section.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.00585" label="2211.00585">
        <attvalues>
          <attvalue for="0" value="Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New&#10;  Speakers" />
          <attvalue for="1" value="  Fine-tuning is a popular method for adapting text-to-speech (TTS) models to&#10;new speakers. However this approach has some challenges. Usually fine-tuning&#10;requires several hours of high quality speech per speaker. There is also that&#10;fine-tuning will negatively affect the quality of speech synthesis for&#10;previously learnt speakers. In this paper we propose an alternative approach&#10;for TTS adaptation based on using parameter-efficient adapter modules. In the&#10;proposed approach, a few small adapter modules are added to the original&#10;network. The original weights are frozen, and only the adapters are fine-tuned&#10;on speech for new speaker. The parameter-efficient fine-tuning approach will&#10;produce a new model with high level of parameter sharing with original model.&#10;Our experiments on LibriTTS, HiFi-TTS and VCTK datasets validate the&#10;effectiveness of adapter-based method through objective and subjective metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.12242" label="2208.12242">
        <attvalues>
          <attvalue for="0" value="DreamBooth: Fine Tuning Text-to-Image Diffusion Models for&#10;  Subject-Driven Generation" />
          <attvalue for="1" value="  Large text-to-image models achieved a remarkable leap in the evolution of AI,&#10;enabling high-quality and diverse synthesis of images from a given text prompt.&#10;However, these models lack the ability to mimic the appearance of subjects in a&#10;given reference set and synthesize novel renditions of them in different&#10;contexts. In this work, we present a new approach for &quot;personalization&quot; of&#10;text-to-image diffusion models. Given as input just a few images of a subject,&#10;we fine-tune a pretrained text-to-image model such that it learns to bind a&#10;unique identifier with that specific subject. Once the subject is embedded in&#10;the output domain of the model, the unique identifier can be used to synthesize&#10;novel photorealistic images of the subject contextualized in different scenes.&#10;By leveraging the semantic prior embedded in the model with a new autogenous&#10;class-specific prior preservation loss, our technique enables synthesizing the&#10;subject in diverse scenes, poses, views and lighting conditions that do not&#10;appear in the reference images. We apply our technique to several&#10;previously-unassailable tasks, including subject recontextualization,&#10;text-guided view synthesis, and artistic rendering, all while preserving the&#10;subject's key features. We also provide a new dataset and evaluation protocol&#10;for this new task of subject-driven generation. Project page:&#10;https://dreambooth.github.io/&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.15370" label="2205.15370">
        <attvalues>
          <attvalue for="0" value="Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech&#10;  with Untranscribed Data" />
          <attvalue for="1" value="  We propose Guided-TTS 2, a diffusion-based generative model for high-quality&#10;adaptive TTS using untranscribed data. Guided-TTS 2 combines a&#10;speaker-conditional diffusion model with a speaker-dependent phoneme classifier&#10;for adaptive text-to-speech. We train the speaker-conditional diffusion model&#10;on large-scale untranscribed datasets for a classifier-free guidance method and&#10;further fine-tune the diffusion model on the reference speech of the target&#10;speaker for adaptation, which only takes 40 seconds. We demonstrate that&#10;Guided-TTS 2 shows comparable performance to high-quality single-speaker TTS&#10;baselines in terms of speech quality and speaker similarity with only a&#10;ten-second untranscribed data. We further show that Guided-TTS 2 outperforms&#10;adaptive TTS baselines on multi-speaker datasets even with a zero-shot&#10;adaptation setting. Guided-TTS 2 can adapt to a wide range of voices only using&#10;untranscribed speech, which enables adaptive TTS with the voice of non-human&#10;characters such as Gollum in \textit{&quot;The Lord of the Rings&quot;}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.16083" label="2306.16083">
        <attvalues>
          <attvalue for="0" value="UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Data" />
          <attvalue for="1" value="  We propose UnitSpeech, a speaker-adaptive speech synthesis method that&#10;fine-tunes a diffusion-based text-to-speech (TTS) model using minimal&#10;untranscribed data. To achieve this, we use the self-supervised unit&#10;representation as a pseudo transcript and integrate the unit encoder into the&#10;pre-trained TTS model. We train the unit encoder to provide speech content to&#10;the diffusion-based decoder and then fine-tune the decoder for speaker&#10;adaptation to the reference speaker using a single $&lt;$unit, speech$&gt;$ pair.&#10;UnitSpeech performs speech synthesis tasks such as TTS and voice conversion&#10;(VC) in a personalized manner without requiring model re-training for each&#10;task. UnitSpeech achieves comparable and superior results on personalized TTS&#10;and any-to-any VC tasks compared to previous baselines. Our model also shows&#10;widespread adaptive performance on real-world data and other tasks that use a&#10;unit sequence as input.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.03257" label="2409.03257">
        <attvalues>
          <attvalue for="0" value="Understanding LLM Development Through Longitudinal Study: Insights from&#10;  the Open Ko-LLM Leaderboard" />
          <attvalue for="1" value="  This paper conducts a longitudinal study over eleven months to address the&#10;limitations of prior research on the Open Ko-LLM Leaderboard, which have relied&#10;on empirical studies with restricted observation periods of only five months.&#10;By extending the analysis duration, we aim to provide a more comprehensive&#10;understanding of the progression in developing Korean large language models&#10;(LLMs). Our study is guided by three primary research questions: (1) What are&#10;the specific challenges in improving LLM performance across diverse tasks on&#10;the Open Ko-LLM Leaderboard over time? (2) How does model size impact task&#10;performance correlations across various benchmarks? (3) How have the patterns&#10;in leaderboard rankings shifted over time on the Open Ko-LLM Leaderboard?. By&#10;analyzing 1,769 models over this period, our research offers a comprehensive&#10;examination of the ongoing advancements in LLMs and the evolving nature of&#10;evaluation frameworks.&#10;" />
          <attvalue for="2" value="&#10;The rapid advancement of large language models (LLMs)~\cite{zhao2023survey} has led to the creation of various leaderboards designed to evaluate their performance across a wide range of tasks~\cite{alpaca_eval,lee2023holistic,HughesBae2023,bigcodeleaderboard,li2023halueval}. Among these, the Open LLM Leaderboard~\cite{open-llm-leaderboard-v1,open-llm-leaderboard-v2} developed by Hugging Face~\cite{jain2022hugging} has achieved significant global recognition. In the context of Korean language models, the Open Ko-LLM Leaderboard~\cite{park2024open} was established to specifically assess LLM performance within the Korean language environment.&#10;&#10;While previous analyses of the Open Ko-LLM Leaderboard~\cite{park2024open} have provided valuable insights into LLM performance, they have been constrained observation periods of only five months, limiting their ability to capture long-term trends. To better understand the ongoing evolution and inherent challenges in LLM development, a more comprehensive and extended analysis is required. This paper addresses this gap by conducting a detailed longitudinal study of the Open Ko-LLM Leaderboard, guided by three primary research questions:&#10;&#10;First, we analyze the longitudinal changes in performance across five tasks monitored by the Open Ko-LLM Leaderboard. These tasks are designed to evaluate various capabilities of LLMs, including reasoning, natural language understanding, and common sense knowledge. By examining data collected over a eleven-month period, this study aims to identify which capabilities have presented the greatest challenges for LLM developers, which tasks have reached performance saturation rapidly, and which tasks continue to pose significant difficulties. This analysis will provide quantitative insights into performance trends across different tasks, thereby guiding targeted research efforts and highlighting key areas that require further advancement to push the boundaries of model development.&#10;&#10;Second, we explore the correlations between different tasks based on model size. This aspect of the study examines how the performance across different tasks varies depending on the scale of the model. Understanding these correlations will provide insights into the interaction between model capacity and task performance, offering a deeper understanding of how scaling influences overall effectiveness across tasks.&#10;&#10;Third, we examine the evolution of leaderboard dynamics from the initial stages to the present by focusing on three key aspects: the correlations between task performances in the early months compared to the entire eleven-month period, the temporal changes in performance based on model type, and the shifts in performance relative to model size. This comprehensive analysis offers insights into the evolving interplay among tasks and the influence of various model characteristics on LLM performance throughout different phases of development.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Performance Evaluation, Leaderboard Analysis, Data Science, Language Model Development, Statistics" />
        </attvalues>
      </node>
      <node id="2311.04287" label="2311.04287">
        <attvalues>
          <attvalue for="0" value="Holistic Evaluation of Text-To-Image Models" />
          <attvalue for="1" value="  The stunning qualitative improvement of recent text-to-image models has led&#10;to their widespread attention and adoption. However, we lack a comprehensive&#10;quantitative understanding of their capabilities and risks. To fill this gap,&#10;we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models&#10;(HEIM). Whereas previous evaluations focus mostly on text-image alignment and&#10;image quality, we identify 12 aspects, including text-image alignment, image&#10;quality, aesthetics, originality, reasoning, knowledge, bias, toxicity,&#10;fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios&#10;encompassing these aspects and evaluate 26 state-of-the-art text-to-image&#10;models on this benchmark. Our results reveal that no single model excels in all&#10;aspects, with different models demonstrating different strengths. We release&#10;the generated images and human evaluation results for full transparency at&#10;https://crfm.stanford.edu/heim/v1.1.0 and the code at&#10;https://github.com/stanford-crfm/helm, which is integrated with the HELM&#10;codebase.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11747" label="2305.11747">
        <attvalues>
          <attvalue for="0" value="HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large&#10;  Language Models" />
          <attvalue for="1" value="  Large language models (LLMs), such as ChatGPT, are prone to generate&#10;hallucinations, i.e., content that conflicts with the source or cannot be&#10;verified by the factual knowledge. To understand what types of content and to&#10;which extent LLMs are apt to hallucinate, we introduce the Hallucination&#10;Evaluation benchmark for Large Language Models (HaluEval), a large collection&#10;of generated and human-annotated hallucinated samples for evaluating the&#10;performance of LLMs in recognizing hallucination. To generate these samples, we&#10;propose a ChatGPT-based two-step framework, i.e., sampling-then-filtering.&#10;Besides, we also hire some human labelers to annotate the hallucinations in&#10;ChatGPT responses. The empirical results suggest that ChatGPT is likely to&#10;generate hallucinated content in specific topics by fabricating unverifiable&#10;information (i.e., about $19.5\%$ responses). Moreover, existing LLMs face&#10;great challenges in recognizing the hallucinations in texts. However, our&#10;experiments also prove that providing external knowledge or adding reasoning&#10;steps can help LLMs recognize hallucinations. Our benchmark can be accessed at&#10;https://github.com/RUCAIBox/HaluEval.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04711" label="2109.04711">
        <attvalues>
          <attvalue for="0" value="Pre-train or Annotate? Domain Adaptation with a Constrained Budget" />
          <attvalue for="1" value="  Recent work has demonstrated that pre-training in-domain language models can&#10;boost performance when adapting to a new domain. However, the costs associated&#10;with pre-training raise an important question: given a fixed budget, what steps&#10;should an NLP practitioner take to maximize performance? In this paper, we view&#10;domain adaptation with a constrained budget as a consumer choice problem, where&#10;the goal is to select an optimal combination of data annotation and&#10;pre-training. We measure annotation costs of three procedural text datasets,&#10;along with the pre-training costs of several in-domain language models. The&#10;utility of different combinations of pre-training and data annotation are&#10;evaluated under varying budget constraints to assess which combination strategy&#10;works best. We find that for small budgets, spending all funds on annotation&#10;leads to the best performance; once the budget becomes large enough, however, a&#10;combination of data annotation and in-domain pre-training yields better&#10;performance. Our experiments suggest task-specific data annotation should be&#10;part of an economical strategy when adapting an NLP model to a new domain.&#10;" />
          <attvalue for="2" value="&#10;The conventional wisdom on semi-supervised learning and unsupervised domain adaptation is that labeled data is expensive; therefore, training on a combination of labeled and unlabeled data is an economical approach to improve performance when adapting to a new domain \cite{blum1998combining,daume2006domain,hoffman2018cycada,chen2020mixtext}.&#10;&#10;Recent work has shown that pre-training in-domain Transformers is an effective method for unsupervised adaptation \cite{Han2019UnsupervisedDA,wright2020transformer} &#10;and even boosts performance when large quantities of in-domain data are available \cite{Gururangan2020DontSP}. &#10;However, modern pre-training methods incur substantial costs \cite{izsak2021train}, and generate carbon emissions \cite{Strubell2019EnergyAP,schwartz2019green,bender2021dangers}. This raises an important question: given a fixed budget to improve a model's performance, what steps should an NLP practitioner take? On one hand, they could hire annotators to label in-domain task-specific data, while on the other, they could buy or rent GPUs or TPUs to pre-train large in-domain language models. In this paper, we empirically study the best strategy for adapting to a new domain given a fixed budget.&#10;&#10;We view the NLP practitioner’s dilemma of how to adapt to a new domain as a problem of consumer choice, a classical problem in microeconomics \cite{becker1965theory,lancaster1966new}. &#10;As illustrated in Figure \ref{fig:consumer_choice}, the NLP practitioner (consumer) can obtain $X_a$ annotated documents (by hiring annotators) at a cost of $C_a$ each, and $X_p$ hours of pre-training (by renting GPUs or TPUs) at a cost of $C_p$ per hour. Given a fixed budget $B$, the consumer may choose any combination that fits within the budget constraint $X_a C_a + X_p C_P \leq B$. The goal is to choose a combination that maximizes the utility function, $U(X_a, X_p)$, which can be defined using an appropriate performance metric, such as $F_1$ score, that is achieved after pre-training for $X_p$ hours and then fine-tuning on $X_a$ in-domain documents. &#10;&#10;To empirically estimate the cost of annotation, we hire annotators to label domain-specific documents for supervised fine-tuning in three procedural text domains: wet-lab protocols, paragraphs describing scientific procedures in PubMed articles, and chemical synthesis procedures described in patents. We choose to target natural language understanding for scientific procedures in this study, because there is an opportunity to help automate lab protocols and support more reproducible scientific experiments, yet few annotated datasets currently exist in these domains. Furthermore, annotation of scientific procedures is not easily amenable to crowdsourcing, making this an ideal testbed for pre-training-based domain adaptation. We measure the cost of in-domain pre-training on a large collection of unlabeled procedural texts using Google's Cloud TPUs.&#10;Model performance is then evaluated under varying budget constraints in six source and target domain combinations.&#10;&#10;Our analysis suggests that given current costs of pre-training large Transformer models, such as BERT \cite{devlin-etal-2019-bert}, and RoBERTa \cite{Liu2019RoBERTaAR}, in-domain data annotation should always be part of an economical strategy when adapting a single NLP system to a new domain.&#10;For small budgets (e.g. less than \$800 USD), spending all funds on annotation is the best policy; however, as more funding becomes available, a combination of pre-training and annotation is the best choice. &#10;&#10;This paper addresses a specific question that is often faced by NLP practitioners working on applications: what is the most economical approach to adapt an NLP system to a new domain when no pre-trained models or task-annotated datasets are initially available? If multiple NLP systems need to be adapted to a single target domain, model costs can be amortized, making pre-training an attractive option for smaller budgets.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Economics, Domain Adaptation, Natural Language Processing, Language Modeling, Budget Optimization" />
        </attvalues>
      </node>
      <node id="1711.03213" label="1711.03213">
        <attvalues>
          <attvalue for="0" value="CyCADA: Cycle-Consistent Adversarial Domain Adaptation" />
          <attvalue for="1" value="  Domain adaptation is critical for success in new, unseen environments.&#10;Adversarial adaptation models applied in feature spaces discover domain&#10;invariant representations, but are difficult to visualize and sometimes fail to&#10;capture pixel-level and low-level domain shifts. Recent work has shown that&#10;generative adversarial networks combined with cycle-consistency constraints are&#10;surprisingly effective at mapping images between domains, even without the use&#10;of aligned image pairs. We propose a novel discriminatively-trained&#10;Cycle-Consistent Adversarial Domain Adaptation model. CyCADA adapts&#10;representations at both the pixel-level and feature-level, enforces&#10;cycle-consistency while leveraging a task loss, and does not require aligned&#10;pairs. Our model can be applied in a variety of visual recognition and&#10;prediction settings. We show new state-of-the-art results across multiple&#10;adaptation tasks, including digit classification and semantic segmentation of&#10;road scenes demonstrating transfer from synthetic to real world domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.12239" label="2004.12239">
        <attvalues>
          <attvalue for="0" value="MixText: Linguistically-Informed Interpolation of Hidden Space for&#10;  Semi-Supervised Text Classification" />
          <attvalue for="1" value="  This paper presents MixText, a semi-supervised learning method for text&#10;classification, which uses our newly designed data augmentation method called&#10;TMix. TMix creates a large amount of augmented training samples by&#10;interpolating text in hidden space. Moreover, we leverage recent advances in&#10;data augmentation to guess low-entropy labels for unlabeled data, hence making&#10;them as easy to use as labeled data.By mixing labeled, unlabeled and augmented&#10;data, MixText significantly outperformed current pre-trained and fined-tuned&#10;models and other state-of-the-art semi-supervised learning methods on several&#10;text classification benchmarks. The improvement is especially prominent when&#10;supervision is extremely limited. We have publicly released our code at&#10;https://github.com/GT-SALT/MixText.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.02817" label="1904.02817">
        <attvalues>
          <attvalue for="0" value="Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence&#10;  Labeling" />
          <attvalue for="1" value="  Contextualized word embeddings such as ELMo and BERT provide a foundation for&#10;strong performance across a wide range of natural language processing tasks by&#10;pretraining on large corpora of unlabeled text. However, the applicability of&#10;this approach is unknown when the target domain varies substantially from the&#10;pretraining corpus. We are specifically interested in the scenario in which&#10;labeled data is available in only a canonical source domain such as newstext,&#10;and the target domain is distinct from both the labeled and pretraining texts.&#10;To address this scenario, we propose domain-adaptive fine-tuning, in which the&#10;contextualized embeddings are adapted by masked language modeling on text from&#10;the target domain. We test this approach on sequence labeling in two&#10;challenging domains: Early Modern English and Twitter. Both domains differ&#10;substantially from existing pretraining corpora, and domain-adaptive&#10;fine-tuning yields substantial improvements over strong BERT baselines, with&#10;particularly impressive results on out-of-vocabulary words. We conclude that&#10;domain-adaptive fine-tuning offers a simple and effective approach for the&#10;unsupervised adaptation of sequence labeling to difficult new domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10204" label="2212.10204">
        <attvalues>
          <attvalue for="0" value="TTS-Guided Training for Accent Conversion Without Parallel Data" />
          <attvalue for="1" value="  Accent Conversion (AC) seeks to change the accent of speech from one (source)&#10;to another (target) while preserving the speech content and speaker identity.&#10;However, many AC approaches rely on source-target parallel speech data. We&#10;propose a novel accent conversion framework without the need of parallel data.&#10;Specifically, a text-to-speech (TTS) system is first pretrained with&#10;target-accented speech data. This TTS model and its hidden representations are&#10;expected to be associated only with the target accent. Then, a speech encoder&#10;is trained to convert the accent of the speech under the supervision of the&#10;pretrained TTS model. In doing so, the source-accented speech and its&#10;corresponding transcription are forwarded to the speech encoder and the&#10;pretrained TTS, respectively. The output of the speech encoder is optimized to&#10;be the same as the text embedding in the TTS system. At run-time, the speech&#10;encoder is combined with the pretrained TTS decoder to convert the&#10;source-accented speech toward the target. In the experiments, we converted&#10;English with two source accents (Chinese and Indian) to the target accent&#10;(American/British/Canadian). Both objective metrics and subjective listening&#10;tests successfully validate that, without any parallel data, the proposed&#10;approach generates speech samples that are close to the target accent with high&#10;speech quality.&#10;" />
          <attvalue for="2" value="&#10;Accent Conversion (AC) is a technique that converts speech with a source accent to a target accent. Ideally it is expected to modify only the accent-related speech attributes, while preserving the speech content and the speaker identity. Accent conversion technique will enable a variety of real-life applications, such as language education \cite{felps2009foreign,aryal2014can}, movie dubbing \cite{zhao2021converting}, and personalized TTS \cite{ding2022accentron}. &#10;&#10;Among the many aspects of proficiency in speaking a language, e.g., lexical, syntactic, semantic, phonological, pronunciation is one of the most fundamental due to the neuro-musculatory basis of speech production \cite{scovel1988time,felps2009foreign,felps2012foreign}. An accent represents a distinctive way of pronouncing a language. There have been many studies to find the mapping between accents. Voice morphing methods perform accent conversion by decomposing and modifying the spectral details of speech~\cite{huckvale2007spoken,felps2010developing,aryal2013foreign}. Frame-wise feature mapping methods achieve the same by pairing source and target vectors based on their linguistic similarity ~\cite{aryal2014can}. Articulatory synthesizer methods take a different path by replacing the source pronunciation unit with the target counterpart through unit selection~\cite{felps2012foreign,aryal2015reduction,aryal2015articulatory}. All these studies rely on parallel speech data between the source and target accents, which limits the scope of applications~\cite{zhao2021converting,li2020improving}.&#10;&#10;Another successful AC approach utilizes phonetic posteriorgram (PPG) to characterize the phonetic pronunciation \cite{zhao2018icassp,li2020improving} that doesn't require parallel speech data. It trains a speech synthesizer with the source-accented speech to map the PPGs to the acoustic features. Nonetheless, to synthesize the target accent for one speaker, a reference speech from another speaker with the target accent is still required.&#10;Alternatively, the end-to-end framework can be trained on speech data with multiple accents, adding an additional accent encoder to the framework \cite{liu2020end, ding2022accentron}. In this way, speech with different accents can be obtained by varying the learned accent embedding without using reference speech. Recently, another reference-free AC framework with a pronunciation correction model has been explored \cite{zhao2021converting}. It first creates speech with the target accent in the desired speaker's voice. Then, the pronunciation correction model is trained to convert the source-accented speech to the synthesized target-accented speech.&#10;&#10;In this paper, we propose a novel AC framework without parallel speech data. In the proposed framework, a TTS system is first trained with target-accented speech data. Then, a speech encoder is trained to convert the accent of the speech under the supervision of the pretrained TTS model. Specifically, source-accented speech and its corresponding transcription are respectively forwarded to the speech encoder and TTS system. The training of the speech encoder aims to minimize the distance between the its output and the hidden text embedding in the TTS system. In particular, the TTS system is trained with speech data containing the target accent only, and its hidden representations such as text embedding are assumed to be free of non-target accents. As a result, the output of the speech encoder is expected to be the same as the text embedding representing the target accent pronunciation.&#10;During conversion, the speech encoder and the TTS decoder are combined for generating target-accented speech. It is worth mentioning that parallel speech data is not used in this work.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Machine Learning, Linguistics, Engineering, Accent Conversion, Deep Learning, Speech Synthesis, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2005.09271" label="2005.09271">
        <attvalues>
          <attvalue for="0" value="Improving Accent Conversion with Reference Encoder and End-To-End&#10;  Text-To-Speech" />
          <attvalue for="1" value="  Accent conversion (AC) transforms a non-native speaker's accent into a native&#10;accent while maintaining the speaker's voice timbre. In this paper, we propose&#10;approaches to improving accent conversion applicability, as well as quality.&#10;First of all, we assume no reference speech is available at the conversion&#10;stage, and hence we employ an end-to-end text-to-speech system that is trained&#10;on native speech to generate native reference speech. To improve the quality&#10;and accent of the converted speech, we introduce reference encoders which make&#10;us capable of utilizing multi-source information. This is motivated by acoustic&#10;features extracted from native reference and linguistic information, which are&#10;complementary to conventional phonetic posteriorgrams (PPGs), so they can be&#10;concatenated as features to improve a baseline system based only on PPGs.&#10;Moreover, we optimize model architecture using GMM-based attention instead of&#10;windowed attention to elevate synthesized performance. Experimental results&#10;indicate when the proposed techniques are applied the integrated system&#10;significantly raises the scores of acoustic quality (30$\%$ relative increase&#10;in mean opinion score) and native accent (68$\%$ relative preference) while&#10;retaining the voice identity of the non-native speaker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.15925" label="2402.15925">
        <attvalues>
          <attvalue for="0" value="MultiContrievers: Analysis of Dense Retrieval Representations" />
          <attvalue for="1" value="  Dense retrievers compress source documents into (possibly lossy) vector&#10;representations, yet there is little analysis of what information is lost&#10;versus preserved, and how it affects downstream tasks. We conduct the first&#10;analysis of the information captured by dense retrievers compared to the&#10;language models they are based on (e.g., BERT versus Contriever). We use 25&#10;MultiBert checkpoints as randomized initialisations to train MultiContrievers,&#10;a set of 25 contriever models. We test whether specific pieces of information&#10;-- such as gender and occupation -- can be extracted from contriever vectors of&#10;wikipedia-like documents. We measure this extractability via information&#10;theoretic probing. We then examine the relationship of extractability to&#10;performance and gender bias, as well as the sensitivity of these results to&#10;many random initialisations and data shuffles. We find that (1) contriever&#10;models have significantly increased extractability, but extractability usually&#10;correlates poorly with benchmark performance 2) gender bias is present, but is&#10;not caused by the contriever representations 3) there is high sensitivity to&#10;both random initialisation and to data shuffle, suggesting that future&#10;retrieval research should test across a wider spread of both.&#10;" />
          <attvalue for="2" value="&#10;&#10;Dense retrievers \cite{Karpukhin2020DensePR, izacard2022unsupervised, Hofstatter2021EfficientlyTA} are a standard component of retrieval augmented Question Answering (QA) \cite{RAG}, and other retrieval systems such as fact-checking~\cite{thorne2018fever}, argumentation \cite{wachsmuth-etal-2018-retrieval}, and others.&#10;Despite their ubiquity, we lack an understanding of the information recoverable from dense retriever representations, and how it affects retrieval system behaviour.&#10;This lack of analytical work is surprising. Retrievers are widespread, and are used in contexts that require trust: increasing factuality and decreasing hallucination \cite{shuster-etal-2021-retrieval-augmentation}, and providing trust and transparency \cite{lewis2020retrieval} via a source document that has provenance and can be examined. The information a representation retains from a source document constrains these abilities.&#10;Dense retrievers lossily encode input documents into N-dimensional representations, and by doing so necessarily emphasise some pieces of information over others. &#10;A biography of Mary Somerville will contain many details about her: her profession (astronomy and mathematics), her gender (female), her political influence (women's suffrage), her country of origin (Scotland) and others. Each of these features are relevant to different kinds of queries. Which ones will a given retriever represent most recoverably?\looseness-1&#10;&#10;Some analysis of this type exists for Masked Language Models (MLMs) (\S \ref{subsec:what_is_mdl}), but there is no such analysis for retrievers, which optimise a contrastive loss. Contrastive training is a very different objective than MLM, based on (dis)similarity of paired samples. The choice of pair affects feature suppression -- what is recoverable and what is not \cite{robinson2021can}.&#10;So we extend this previous analytical work into the retrieval domain, by training 25 MultiContrievers initialised from MultiBert checkpoints~\cite{multiberts}. This is the first study that includes variability over a large number of retriever initialisations, with some surprising results from this alone. We use information theoretic probing, also known as minimum description length (MDL) probing~\cite{voita-titov-2020-information}, to measure the information in MultiContriever representations. We evaluate the models on 14 retrieval datasets from the BEIR benchmark~\cite{thakur2021beir}. We test how well retrievers preserve information in a document, like gender and occupation, which we refer to as features. We adapt the existing datasets to better test for knowledge of these, by creating a new manually annotated gender subset of Natural Questions, NQ-gender. We ultimately test if gender information is predictive of gender bias, as it was in previous MLM work (\S \ref{subsec:what_is_mdl}).We address the following four research questions:&#10;\@startsection{paragraph}{4}{\z@}{.2ex \@plus.2ex \@minus.2ex}{-1em}{\normalfont\bfseries}{Q1} To what extent do retrievers preserve information like gender and occupation in an encoded document? (\S \ref{subsec:q1_extractability})&#10;&#10;For both MultiBerts and MultiContrievers, gender is more extractable than occupation, which can cause a model to rely on gender heuristics (a source of gender bias). But there are noticeable differences in the models. Both features are more extractable in MultiContrievers than MultiBerts, but there is a lower ratio (less difference) between gender and occupation. This indicates MultiContrievers are less likely to rely less on gender heuristics \cite{lovering2021predicting}, but still might.&#10;\@startsection{paragraph}{4}{\z@}{.2ex \@plus.2ex \@minus.2ex}{-1em}{\normalfont\bfseries}{Q2} How sensitive is this to random initialisation and data shuffle? (\S \ref{subsec:q2_initialisation})&#10;&#10;In MultiBerts, extractability is very sensitive to random initialisation and shuffle, in MultiContrievers it is not. MultiContrievers have a much smaller variance between the 25 seeds, suggesting a regularising effect. However, MultiContriever performance is surprisingly sensitive to both random initialisation and to data shuffle. MultiContrievers have a very wide range of performance on BEIR benchmarks, despite identical loss curves. &#10;But it is not easy to select a `best' model, since the best and worst model is not consistent across datasets - the ranking of each model can change, sometimes drastically.&#10;&#10;\@startsection{paragraph}{4}{\z@}{.2ex \@plus.2ex \@minus.2ex}{-1em}{\normalfont\bfseries}{Q3} Do differences in this information correlate with performance on retrieval benchmarks? (\S \ref{subsec:q3_correlation})&#10;&#10;On partitions of examples that ostensibly require gender information (NQ-gender), we show that gender extractability is highly correlated with retrieval performance. However, overall retrieval performance on benchmarks like BEIR is poorly correlated with extractability. This suggests that while some benchmark examples do reward models for preserving gender information, most examples do not require that, so the benchmark as a whole does not require that capability.&#10;\@startsection{paragraph}{4}{\z@}{.2ex \@plus.2ex \@minus.2ex}{-1em}{\normalfont\bfseries}{Q4} Is gender information in retrievers predictive of their gender bias? (\S \ref{subsec:q4_bias})&#10;&#10;Despite the evidence that extractability of gender information is helpful to a model, it is not the cause of gender bias in the NQ-gender dataset. When we do a causal analysis by removing gender from MultiContriever representations, gender bias persists, suggesting that the source of bias is in the queries or corpus. &#10;&#10;Our contributions are:&#10;1) the first information theoretic analysis of dense retrievers, 2) an analysis of variability in performance and social bias across random retriever seeds,&#10;3) the first causal analysis of sources of social bias in dense retrievers,&#10;4) NQ-gender, an annotated subset of Natural Questions for queries that constrain gender, and&#10;5) a suite of 25 MultiContrievers for use in future work, with all training and evaluation code.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Information Theory, Computer Science, Linguistics, Language Models, Bias Analysis, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1803.05355" label="1803.05355">
        <attvalues>
          <attvalue for="0" value="FEVER: a large-scale dataset for Fact Extraction and VERification" />
          <attvalue for="1" value="  In this paper we introduce a new publicly available dataset for verification&#10;against textual sources, FEVER: Fact Extraction and VERification. It consists&#10;of 185,445 claims generated by altering sentences extracted from Wikipedia and&#10;subsequently verified without knowledge of the sentence they were derived from.&#10;The claims are classified as Supported, Refuted or NotEnoughInfo by annotators&#10;achieving 0.6841 in Fleiss $\kappa$. For the first two classes, the annotators&#10;also recorded the sentence(s) forming the necessary evidence for their&#10;judgment. To characterize the challenge of the dataset presented, we develop a&#10;pipeline approach and compare it to suitably designed oracles. The best&#10;accuracy we achieve on labeling a claim accompanied by the correct evidence is&#10;31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that&#10;FEVER is a challenging testbed that will help stimulate progress on claim&#10;verification against textual sources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.11230" label="2106.11230">
        <attvalues>
          <attvalue for="0" value="Can contrastive learning avoid shortcut solutions?" />
          <attvalue for="1" value="  The generalization of representations learned via contrastive learning&#10;depends crucially on what features of the data are extracted. However, we&#10;observe that the contrastive loss does not always sufficiently guide which&#10;features are extracted, a behavior that can negatively impact the performance&#10;on downstream tasks via &quot;shortcuts&quot;, i.e., by inadvertently suppressing&#10;important predictive features. We find that feature extraction is influenced by&#10;the difficulty of the so-called instance discrimination task (i.e., the task of&#10;discriminating pairs of similar points from pairs of dissimilar ones). Although&#10;harder pairs improve the representation of some features, the improvement comes&#10;at the cost of suppressing previously well represented features. In response,&#10;we propose implicit feature modification (IFM), a method for altering positive&#10;and negative samples in order to guide contrastive models towards capturing a&#10;wider variety of predictive features. Empirically, we observe that IFM reduces&#10;feature suppression, and as a result improves performance on vision and medical&#10;imaging tasks. The code is available at: \url{https://github.com/joshr17/IFM}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.16163" label="2106.16163">
        <attvalues>
          <attvalue for="0" value="The MultiBERTs: BERT Reproductions for Robustness Analysis" />
          <attvalue for="1" value="  Experiments with pre-trained models such as BERT are often based on a single&#10;checkpoint. While the conclusions drawn apply to the artifact tested in the&#10;experiment (i.e., the particular instance of the model), it is not always clear&#10;whether they hold for the more general procedure which includes the&#10;architecture, training data, initialization scheme, and loss function. Recent&#10;work has shown that repeating the pre-training process can lead to&#10;substantially different performance, suggesting that an alternate strategy is&#10;needed to make principled statements about procedures. To enable researchers to&#10;draw more robust conclusions, we introduce the MultiBERTs, a set of 25&#10;BERT-Base checkpoints, trained with similar hyper-parameters as the original&#10;BERT model but differing in random weight initialization and shuffling of&#10;training data. We also define the Multi-Bootstrap, a non-parametric bootstrap&#10;method for statistical inference designed for settings where there are multiple&#10;pre-trained models and limited test data. To illustrate our approach, we&#10;present a case study of gender bias in coreference resolution, in which the&#10;Multi-Bootstrap lets us measure effects that may not be detected with a single&#10;checkpoint. We release our models and statistical library along with an&#10;additional set of 140 intermediate checkpoints captured during pre-training to&#10;facilitate research on learning dynamics.&#10;" />
          <attvalue for="2" value="&#10;Contemporary natural language processing (NLP) relies heavily on pretrained language models, which are trained using large-scale unlabeled data~\cite{bommasani2021opportunities}. Bert~\cite{devlin2018bert} is a particularly popular choice: it has been widely adopted in academia and industry, and aspects of its performance have been reported on in thousands of research papers~\cite[see, e.g.,][for an overview]{rogers-etal-2020-primer}. Because pre-training large language models is computationally expensive~\cite{strubell2019energy}, researchers often rely on the release of model checkpoints through libraries such as HuggingFace Transformers \cite{wolf-etal-2020-transformers}, which enable them to use large-scale language models without repeating the pre-training work. Consequently, most published results are based on a small number of publicly released model checkpoints.&#10;&#10;While this reuse of model checkpoints has lowered the cost of research and facilitated head-to-head comparisons, it limits our ability to draw general scientific conclusions about the performance of a particular class of models~\cite{dror2019deep, d2020underspecification, zhong2021larger}. The key issue is that reusing model checkpoints makes it hard to generalize observations about the behavior of a single model artifact to statements about the underlying pre-training procedure which created it. Pre-training such models is an inherently stochastic process which depends on the initialization of the model's parameters and the ordering of training examples; for example, \cite{d2020underspecification} report substantial quantitative differences across multiple checkpoints of the same model architecture on several ``stress tests''~\cite{naik2018stress,mccoy2019right}. It is therefore difficult to know how much of the success of a model based on the original Bert checkpoint is due to Bert's design, and how much is due to idiosyncracies of a particular artifact. Understanding this difference is critical if we are to generate reusable insights about deep learning for NLP, and improve the state-of-the-art going forward~\cite{zhou2020curse,dodge2020fine,krishna2021how}.&#10;&#10;This paper describes the MultiBerts, an effort to facilitate more robust research on the Bert model. &#10;Our primary contributions are:&#10;&#10;\begin{itemize}&#10; \item We release the MultiBerts, a set of 25 Bert-Base, Uncased checkpoints to facilitate studies of robustness to parameter initialization and order of training examples (\S\ref{sec:release}).&#10; Releasing these models preserves the benefits to the community of a single checkpoint release (i.e., low cost of experiments, apples-to-apples comparisons between studies based on these checkpoints), while enabling researchers to draw more general conclusions about the Bert pre-training procedure.&#10; \item We present the Multi-Bootstrap, a non-parametric method to quantify the uncertainty of experimental results based on multiple pre-training seeds (\S\ref{sec:stats}), and provide recommendations for how to use the Multi-Bootstrap and MultiBerts in typical experimental scenarios. We implement these recommendations in a software library.&#10; \item We illustrate the approach with a practical use case: we investigate the impact of counterfactual data augmentation on gender bias, in a Bert-based coreference resolution systems~\cite{webster2020measuring} (\S\ref{sec:experiments}). Additional examples are provided in Appendix~\ref{sec:original-bert}, where we document challenges with reproducing the widely-used original Bert checkpoint.&#10;\end{itemize}&#10;&#10;The release also includes an additional 140 intermediate checkpoints, captured during training for 5 of the runs (28 checkpoints per run), to facilitate studies of learning dynamics. Our checkpoints and statistical libraries are available at: .&#10;&#10;Additional Related Work. The MultiBerts release builds on top of a large body of work that seeks to analyze the behavior of Bert~\cite{rogers-etal-2020-primer}. In addition to the studies of robustness cited above, several authors have introduced methods to reduce Bert's variability during fine-tuning~\cite{zhang2021revisiting, mosbach2021on, dodge2020fine, Lee2020Mixout, phang2018sentence}. Other authors have also studied the time dimension, which motivates our release of intermediate checkpoints \cite{liu2021probing, hao-etal-2020-investigating, saphra-lopez-2019-understanding, chiang-etal-2020-pretrained, dodge2020fine}.&#10;Similarly to \S\ref{sec:stats}, authors in the NLP literature have recommended best practices for statistical testing~\cite{koehn2004statistical, dror2018hitchhiker, berg2012empirical, card-etal-2020-little, sogaard-etal-2014-whats, peyrard-etal-2021-better}, many of which are based on existing tests to estimate the uncertainty of test sample. In concurrent work, \cite{deutsch2021statistical} considered bootstrapping methods similar to the Multi-Bootstrap, in the context of summarization metrics evaluation. Also in concurrent work, the Mistral project~\cite{mistral} released a set of 10 GPT-2 models with intermediate checkpoints at different stages of pre-training. Our work is complementary, focusing on BERT, introducing a larger number of pre-training seeds, and presenting a methodology to draw robust conclusions about model performance.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.07337" label="2106.07337">
        <attvalues>
          <attvalue for="0" value="Speech Disorder Classification Using Extended Factorized Hierarchical&#10;  Variational Auto-encoders" />
          <attvalue for="1" value="  Objective speech disorder classification for speakers with communication&#10;difficulty is desirable for diagnosis and administering therapy. With the&#10;current state of speech technology, it is evident to propose neural networks&#10;for this application. But neural network model training is hampered by a lack&#10;of labeled disordered speech data. In this research, we apply an extended&#10;version of Factorized Hierarchical Variational Auto-encoders (FHVAE) for&#10;representation learning on disordered speech. The FHVAE model extracts both&#10;content-related and sequence-related latent variables from speech data, and we&#10;utilize the extracted variables to explore how disorder type information is&#10;represented in the latent variables. For better classification performance, the&#10;latent variables are aggregated at the word and sentence level. We show that an&#10;extension of the FHVAE model succeeds in the better disentanglement of the&#10;content-related and sequence-related related representations, but both&#10;representations are still required for best results on disorder type&#10;classification.&#10;" />
          <attvalue for="2" value="&#10;Automated diagnosis of neurological diseases from the speech is receiving increased interest, as evidenced in the recent ADReSS-challenge \cite{luz2020alzheimer, luz2021detecting} which focuses on Alzheimer disease detection. Other neurological diseases such as Amyotrophic Lateral Sclerosis (ALS) and Parkinson's disease (PD) are reflected in speech disorders. Unfortunately, for many diseases, there is no cure but early detection and treatment may reduce complications, and progression may be slowed down with medication.&#10;&#10;With the current state of the art in speech processing, it is a natural choice to explore neural network architectures to achieve accurate and objective automatic speech disorder classification.&#10;In \cite{gillespie2017cross}\cite{kadiri2020parkinson}, classifications are done between control speakers and dysarthric speakers to achieve dysarthria detection. In \cite{gope2020raw}, convolutional neural networks and bidirectional long-short time memory network (BLSTM) are used for a binary classification between PD and ALS using raw speech. In \cite{kourkounakis2020detecting}, residual networks and BLSTMs are used for classifying different types of stuttering.&#10;&#10;While neural networks are a sensible choice to classify speech disorders, they demand large amounts of labeled speech data to train the model. Disordered speech data, especially labeled data, are scarce due to the speaker's difficulty in speaking and the high cost to obtain reliable data labels. To avoid impact from insufficient data, authors have used auto-encoders for learning representations for disordered speech processing \cite{vachhani2017deep, yue2020autoencoder}. &#10;In this work we will apply an evolution of the basic auto-encoders, the Factorized Hierarchical Variational Auto-encoders (FHVAE), and extend it further.&#10;An FHVAE \cite{hsu2017unsupervised, shon2018unsupervised} models the generative process of a sequence of segments in a hierarchical structure. It encodes a speech utterance into a segment-related variable (short time scale) and a sequence-related variable (long time scale) via two linked encoders. The segment-related latent variable represents the information that only appears in a single segment in the sequence, such as the phonetic content of the segment, and is conditioned on the sequence-related variable. The sequence-related variable reflects the features of the whole sequence, like the acoustic environment or speaker characteristics. The model hence offers a separation between speaker characteristics (sequence) and content (segment). &#10;&#10;The disentangled hierarchical representation of the FHVAE suggests using the sequence-related variable for inferring disorder type or severeness. However, we cannot expect the FHVAE model to disentangle speaker and content completely: dysarthric speakers exhibit a reduced vowel triangle~\cite{skodda2012impairment}, which would be reflected in the content variable. We therefore try to push this limitation in the sequence variable by expressing that speakers need to use the same content (segment) space, and hope to see that the disorder-related information is moved to the sequence space. &#10;&#10;We extend the FHVAE model with a regularization that will force the content (segment) space of different speakers to be similar at the expense of a different sequence (speaker) representation. We analyze to which extent we are capable of moving the disorder type information to the sequence level, where it belongs. To succeed in this goal, we will use a small corpus of phonetically labeled data. The training will hence be partly supervised.&#10;&#10;This approach also opens perspectives for further work, e.g., automatic speech recognition or spoken language understanding for the disordered speech from the content variable. &#10;&#10;In this research, we take the first step towards this purpose, analyzing the FHVAE models on disordered speech and exploring how much disorder-related information the model can extract to the latent variables. &#10;We train the FHVAE model with labeled dysarthric speech data and use both the segment and sequence latent variable for disordered speech classification to quantify how much relevant information is found in each representation. &#10;Both latent variables are calculated for each segment, which is only 200~ms in this work, too short for reliable classification. To aggregate information over words or sentences, we compare simple averages or standard deviations or attention-based averages \cite{vaswani2017attention}, where we assume the attention mechanism can be trained to select those segments which show a high discrimination potential.&#10;&#10;We introduce the basic and extended version of the FHVAE model and our classification scheme in section2. In section3, we describe the two databases we used and the experimental settings. Results and analysis will be provided in section4, and section5 gives conclusions. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Modeling, Machine Learning, Computer Science, Speech Processing, Linguistics, Neuroscience, Speech Disorder Classification, Psychology, Representation Learning" />
        </attvalues>
      </node>
      <node id="2004.06833" label="2004.06833">
        <attvalues>
          <attvalue for="0" value="Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS&#10;  Challenge" />
          <attvalue for="1" value="  The ADReSS Challenge at INTERSPEECH 2020 defines a shared task through which&#10;different approaches to the automated recognition of Alzheimer's dementia based&#10;on spontaneous speech can be compared. ADReSS provides researchers with a&#10;benchmark speech dataset which has been acoustically pre-processed and balanced&#10;in terms of age and gender, defining two cognitive assessment tasks, namely:&#10;the Alzheimer's speech classification task and the neuropsychological score&#10;regression task. In the Alzheimer's speech classification task, ADReSS&#10;challenge participants create models for classifying speech as dementia or&#10;healthy control speech. In the the neuropsychological score regression task,&#10;participants create models to predict mini-mental state examination scores.&#10;This paper describes the ADReSS Challenge in detail and presents a baseline for&#10;both tasks, including feature extraction procedures and results for&#10;classification and regression models. ADReSS aims to provide the speech and&#10;language Alzheimer's research community with a platform for comprehensive&#10;methodological comparisons. This will hopefully contribute to addressing the&#10;lack of standardisation that currently affects the field and shed light on&#10;avenues for future research and clinical applicability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.12590" label="1910.12590">
        <attvalues>
          <attvalue for="0" value="Detecting Multiple Speech Disfluencies using a Deep Residual Network&#10;  with Bidirectional Long Short-Term Memory" />
          <attvalue for="1" value="  Stuttering is a speech impediment affecting tens of millions of people on an&#10;everyday basis. Even with its commonality, there is minimal data and research&#10;on the identification and classification of stuttered speech. This paper&#10;tackles the problem of detection and classification of different forms of&#10;stutter. As opposed to most existing works that identify stutters with language&#10;models, our work proposes a model that relies solely on acoustic features,&#10;allowing for identification of several variations of stutter disfluencies&#10;without the need for speech recognition. Our model uses a deep residual network&#10;and bidirectional long short-term memory layers to classify different types of&#10;stutters and achieves an average miss rate of 10.03%, outperforming the&#10;state-of-the-art by almost 27%&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.10392" label="2304.10392">
        <attvalues>
          <attvalue for="0" value="CKBP v2: An Expert-Annotated Evaluation Set for Commonsense Knowledge&#10;  Base Population" />
          <attvalue for="1" value="  Populating Commonsense Knowledge Bases (CSKB) is an important yet hard task&#10;in NLP, as it tackles knowledge from external sources with unseen events and&#10;entities. Fang et al. (2021a) proposed a CSKB Population benchmark with an&#10;evaluation set CKBP v1. However, CKBP v1 adopts crowdsourced annotations that&#10;suffer from a substantial fraction of incorrect answers, and the evaluation set&#10;is not well-aligned with the external knowledge source as a result of random&#10;sampling. In this paper, we introduce CKBP v2, a new high-quality CSKB&#10;Population benchmark, which addresses the two mentioned problems by using&#10;experts instead of crowd-sourced annotation and by adding diversified&#10;adversarial samples to make the evaluation set more representative. We conduct&#10;extensive experiments comparing state-of-the-art methods for CSKB Population on&#10;the new evaluation set for future research comparisons. Empirical results show&#10;that the population task is still challenging, even for large language models&#10;(LLM) such as ChatGPT. Codes and data are available at&#10;https://github.com/HKUST-KnowComp/CSKB-Population.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;In this section, we discuss 1) CSKBs and their role in the era of LLMs and 2) methods and benchmarks for completing and populating knowledge bases in general.&#10;&#10;Commonsense Knowledge Bases.&#10;There are many commonsense knowledge bases introduced in the past few years, such as ATOMIC2020~\cite{hwang2021comet}, ComFact~\cite{gao-etal-2022-comfact}, CICERO~\cite{ghosal-etal-2022-cicero}, PIQA~\cite{bisk2020piqa}, Numersense~\cite{lin-etal-2020-birds}. &#10;Unlike the decades-old knowledge base ConceptNet~\cite{liu2004conceptnet} that only focuses on taxonomic commonsense, these knowledge bases study a broad range of commonsense, including human-event-centric, contextualized, physical, numerical commonsense.&#10;&#10;Along with pure-symbolic CSKBs whose knowledge is obtained from corpora and stored in textual format, there is a stream of research that works on developing neural(-symbolic) CSKBs, which are either knowledge models such as COMET~\cite{bosselut2019comet} or symbolic CSKBs built by prompting knowledge from language models, such as ATOMIC$^{10X}$ ~\cite{WestBHHJBLWC22}, SODA~\cite{kim2022soda}. Although the approach seems highly scalable and seems promising to build more and larger CSKBs, knowledge from neural(-symbolic) CSKBs remains unreliable~\cite{kim2022soda, he2022rethinking, peng2023check} thus often needs to have a robust critic model to filter for good/correct knowledge. &#10;&#10;Reasoning over (Commonsense) Knowledge Bases.&#10;&#10;Regarding conventional knowledge bases like Wordnet~\cite{wordnet} and Freebases~\cite{freebase}, tasks involving completion and population have been well-studied as transductive and inductive link prediction problems in the field of graph neural network~\cite{nips2013transe, Yang2014EmbeddingEA, sun2018rotate, shang2019end, DBLP:conf/www/FangZWSH21}. &#10;Methods powered by pre-trained language models have also been studied in these tasks thanks to the models' representation power~\cite{yao2019kgbert}. &#10;In that setting, knowledge instances of the knowledge bases are serialized to a text sequence, which serves as input to LMs such as BERT or RoBERTa.&#10;&#10;Specific to CSKB Population task on {{CKBP v1}}, \cite{fang-etal-2021-benchmarking} proposed KGBertSAGE, a combination of KG-BERT~\cite{yao2019kgbert} and GraphSAGE~\cite{hamilton2017inductive}. The model showed higher performance over baselines yet still suffered from the out-of-domain problem. &#10;The follow-up works use pseudo-labeling~\cite{fang-etal-2022-pseudoreasoner} and constrained prompting~\cite{DBLP:conf/eacl/DoFDWS24} for solving the problem.&#10;&#10;There are also works focusing on using the knowledge from commonsense knowledge bases to help downstream commonsense reasoning such as QA~\cite{DBLP:conf/aaai/BianH0021, DBLP:conf/coling/WangCL00JXLZ24}, including sampling logical queries over CSKBs for better commonsense reasoning~\cite{DBLP:conf/acl/FangCSB24}, reasoning over entailment or abstraction relations~\cite{DBLP:conf/naacl/WangSWFZCLS24, DBLP:journals/corr/abs-2305-14869}, and using knowledge-constrained decoding to guide the generation of LLMs~\cite{DBLP:conf/emnlp/ChoiF0S23}, etc.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Benchmark Evaluation, Artificial Intelligence, Commonsense Knowledge Bases, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2103.01273" label="2103.01273">
        <attvalues>
          <attvalue for="0" value="On the Effectiveness of Dataset Embeddings in Mono-lingual,Multi-lingual&#10;  and Zero-shot Conditions" />
          <attvalue for="1" value="  Recent complementary strands of research have shown that leveraging&#10;information on the data source through encoding their properties into&#10;embeddings can lead to performance increase when training a single model on&#10;heterogeneous data sources. However, it remains unclear in which situations&#10;these dataset embeddings are most effective, because they are used in a large&#10;variety of settings, languages and tasks. Furthermore, it is usually assumed&#10;that gold information on the data source is available, and that the test data&#10;is from a distribution seen during training. In this work, we compare the&#10;effect of dataset embeddings in mono-lingual settings, multi-lingual settings,&#10;and with predicted data source label in a zero-shot setting. We evaluate on&#10;three morphosyntactic tasks: morphological tagging, lemmatization, and&#10;dependency parsing, and use 104 datasets, 66 languages, and two different&#10;dataset grouping strategies. Performance increases are highest when the&#10;datasets are of the same language, and we know from which distribution the&#10;test-instance is drawn. In contrast, for setups where the data is from an&#10;unseen distribution, performance increase vanishes.&#10;" />
          <attvalue for="2" value="&#10;The performance of natural language processing systems is dependent on the&#10;amount of training data, which is often scarce. To complement existing training&#10;data, supplementary data sources can be used. Especially data annotated for the&#10;same task from other sources can be beneficial to exploit. However, because of&#10;heterogeneity in language or domain this might lead to sub-optimal performance.&#10;In early work on combining training sources, data was selected at training&#10;time~\cite{plank-van-noord-2011-effective,khan-etal-2013-towards} for a given&#10;test set. A more nuanced way to exploit heterogeneous data is to encode&#10;properties of the language as features~\cite{naseem-etal-2012-selective}.&#10;&#10;Recently, ~\newcite{ammar-etal-2016-many} showed that encoding the language of&#10;an instance as an embedding in a neural model is beneficial for multi-lingual&#10;learning. Follow-up work found that multiple datasets within&#10;the same language can also be combined by encoding their&#10;origin~\cite{stymne-etal-2018-parser,ustun-etal-2019-multi}, thereby implicitly&#10;learning useful commonalities, while still encoding dataset-specific knowledge.&#10;These dataset embeddings are employed in groups of datasets which&#10;usually range in size from 2 to 10 datasets. However, it remains unclear in&#10;which situations these dataset embeddings thrive best.&#10;&#10;Furthermore, two often overseen issues with dataset embeddings are that they&#10;are commonly learned from the gold data-source labels attached to each&#10;training and test instance and it is assumed that the test data is from a&#10;distribution which is seen during training. In many real world situations these&#10;assumptions are clearly violated. A common strategy when the test data is&#10;drawn from a different distribution as the training datasets (zero-shot), is to&#10;use a manually assigned proxy&#10;treebank~\cite{smith-etal-2018-82,barry-etal-2019-cross,meechan-maddon-nivre-2019-parse}.&#10;Recent work showed that for unseen datasets in mono-lingual&#10;setups~\cite{wagner-etal-2020-treebank}, interpolated dataset embeddings can be&#10;used to improve performance for zero-shot settings. We use automatically&#10;predicted proxy data sources instead, and focus on mono-linugal as well as&#10;cross-lingual setups.&#10;&#10;In this paper, we provide an extensive evaluation of the usefulness of dataset&#10;embeddings in existing setups and beyond. More concretely, we ask: 1) What are&#10;good indicators to predict the usefulness of dataset embeddings? 2) Can we&#10;effectively use dataset embeddings in the absence of gold data-source&#10;information?&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Modeling, Morphosyntactic Analysis, Artificial Intelligence, Dataset Embeddings, Statistics" />
        </attvalues>
      </node>
      <node id="1910.07938" label="1910.07938">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Parsing with Polyglot Training and Multi-treebank&#10;  Learning: A Faroese Case Study" />
          <attvalue for="1" value="  Cross-lingual dependency parsing involves transferring syntactic knowledge&#10;from one language to another. It is a crucial component for inducing dependency&#10;parsers in low-resource scenarios where no training data for a language exists.&#10;Using Faroese as the target language, we compare two approaches using&#10;annotation projection: first, projecting from multiple monolingual source&#10;models; second, projecting from a single polyglot model which is trained on the&#10;combination of all source languages. Furthermore, we reproduce multi-source&#10;projection (Tyers et al., 2018), in which dependency trees of multiple sources&#10;are combined. Finally, we apply multi-treebank modelling to the projected&#10;treebanks, in addition to or alternatively to polyglot modelling on the source&#10;side. We find that polyglot training on the source languages produces an&#10;overall trend of better results on the target language but the single best&#10;result for the target language is obtained by projecting from monolingual&#10;source parsing models and then training multi-treebank POS tagging and parsing&#10;models on the target side.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.15042" label="2403.15042">
        <attvalues>
          <attvalue for="0" value="LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement" />
          <attvalue for="1" value="  Pretrained large language models (LLMs) are currently state-of-the-art for&#10;solving the vast majority of natural language processing tasks. While many&#10;real-world applications still require fine-tuning to reach satisfactory levels&#10;of performance, many of them are in the low-data regime, making fine-tuning&#10;challenging. To address this, we propose LLM2LLM, a targeted and iterative data&#10;augmentation strategy that uses a teacher LLM to enhance a small seed dataset&#10;by augmenting additional data that can be used for fine-tuning on a specific&#10;task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data,&#10;(2) evaluates and extracts data points that the model gets wrong, and (3) uses&#10;a teacher LLM to generate synthetic data based on these incorrect data points,&#10;which are then added back into the training data. This approach amplifies the&#10;signal from incorrectly predicted data points by the LLM during training and&#10;reintegrates them into the dataset to focus on more challenging examples for&#10;the LLM. Our results show that LLM2LLM significantly enhances the performance&#10;of LLMs in the low-data regime, outperforming both traditional fine-tuning and&#10;other data augmentation baselines. LLM2LLM reduces the dependence on&#10;labor-intensive data curation and paves the way for more scalable and&#10;performant LLM solutions, allowing us to tackle data-constrained domains and&#10;tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on&#10;CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular&#10;fine-tuning in the low-data regime using a Llama-2-7B student model. Our code&#10;is available at https://github.com/SqueezeAILab/LLM2LLM .&#10;" />
          <attvalue for="2" value="&#10;&#10;Pretrained large language models (LLMs) have achieved impressive performance on various benchmarks and datasets that have previously required specialized neural network architectures.&#10;For many of these general benchmarks \cite{hendrycks2020measuring, zhong2023agieval}, LLMs are prompted with custom instructions or in-context examples.&#10;&#10;However, in various real-world applications, these prompting strategies are not a one-size-fits-all solution. &#10;For instance, LLMs have a limit on the amount of input context they can process, thus limiting the number of in-context examples or instructions we can input to make the LLM follow a certain behavior. &#10;For simple tasks that are closely aligned with the data that the LLM was pretrained on, extensive prompting may not be necessary.&#10;However, applying LLMs to specialized domains (e.g., a specific medical field \cite{nori2023can} or private data with niche protocols) can be more challenging, often requiring prohibitively long prompts to achieve adequate performance. &#10;Even if the prompt length does not exceed the limit, processing long prompts increases the latency and cost of each inference. &#10;Additionally, LLMs also tend to forget or ignore information in long contexts~\cite{liu2023lost}, leading to potential accuracy drops even when the model can handle long input prompts.&#10;While Retrieval-Augmented Generation (RAG) \cite{lewis2020retrieval} has been developed&#10;to address some of these challenges, it may sometimes retrieve irrelevant passages or documents, which can potentially degrade the generation performance. &#10;Furthermore, RAG does not necessarily solve the latency and cost issue as &#10;processing a long input prompt may still be required.&#10;&#10;A promising method for addressing this is fine-tuning.&#10;With the emergence of Parameter Efficient Fine-tuning (PEFT)~\cite{hu2021lora, peft}, the computational resources required to fine-tune a task-specific LLM have decreased significantly.&#10;However, herein lies a new problem: successful fine-tuning requires enough training data.&#10;This can be challenging for some applications, where we only have access to a small amount of task-specific data.&#10;Often, collecting, cleaning, and labeling additional data can be costly and time-consuming. &#10;So the key question is:&#10;how should we increase the user's training data to be enough for fine-tuning?&#10;&#10;Data augmentation is a known method that could help effectively expand the training dataset. &#10;For natural language processing (NLP) tasks, one can use approaches such as synonym replacement,&#10;character replacement (e.g., by intentionally introducing spelling errors), random swapping, and&#10;back translation, just to name a few~\cite{wei2019eda, belinkov2017synthetic, coulombe2018text, zhang2018mixup}. &#10;However, these approaches fail to effectively expand the training data for fine-tuning LLMs in the case of new and specialized tasks, as we will show later in Section~\ref{sec:baseline_results}.&#10;&#10;To address this, several recent papers have explored using an LLM to expand&#10;the fine-tuning dataset~\cite{dai2023auggpt, kumar2020data, zhou2023lima,chen2023alpagasus, cao2023instruction, wei2023instructiongpt4, zhu2023minigpt4}. This approach has&#10;proven to be more effective than traditional data augmentation methods.&#10;However, these approaches often apply LLM-based data augmentation on all of the&#10;available training dataset, without considering the LLM's prediction accuracy on individual&#10;training data points. We have observed that for various reasoning tasks such as arithmetic and reading comprehension, the LLM correctly solves simpler examples in the fine-tuning dataset, but may struggle with harder examples. &#10;It will be sub-optimal to keep augmenting data points for which the LLM is already achieving high accuracy on.&#10;&#10;To address these challenges, we introduce LLM2LLM, a new targeted and iterative data augmentation framework that uses a teacher LLM to expand the training dataset, with a targeted and iterative approach. &#10;In more detail, we make the following contributions:&#10;&#10;\begin{itemize}[leftmargin=3mm]&#10; \item &#10; We propose LLM2LLM, a targeted and iterative LLM-based data augmentation technique that efficiently and effectively augments small task-specific datasets. &#10; LLM2LLM achieves this by (1) fine-tuning a student LLM on the initial dataset, (2) evaluating on the training data and extracting data points which the model got incorrect after training, and (3) using a Self-Instruct \cite{wang-etal-2023-self-instruct} style data augmentation to augment these data points, which are then added back into the training data (Section~\ref{sec:llm2llm}).&#10; \item &#10; We benchmark LLM2LLM on randomly sampled subsets of GSM8K \cite{cobbe2021gsm8k}, CaseHOLD \cite{zhengguha2021}, SNIPS \cite{coucke2018snips}, TREC \cite{li-roth-2002-learning} and SST-2 \cite{socher2013recursive} in order to evaluate the effectiveness of our approach in the low-data regime (Section~\ref{sec:main_results}). &#10; Here, we get up to a 24.2\% improvement on GSM8K, 32.6\% on CaseHOLD, 32.0\% on SNIPS, 52.6\% on TREC, and 39.8\% on SST-2 (Table~\ref{table:results}).&#10; \item &#10; We conduct a series of ablations studies comparing LLM2LLM to several existing baselines as well as to variants of LLM2LLM to evaluate the effectiveness of our design decisions (Section~\ref{sec:ablations}).&#10; We observe that both the iterative and targeted nature of LLM2LLM are critical to improving model performance.&#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Fine-Tuning, Computer Science, Linguistics, Artificial Intelligence, Data Augmentation Strategies, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2104.08671" label="2104.08671">
        <attvalues>
          <attvalue for="0" value="When Does Pretraining Help? Assessing Self-Supervised Learning for Law&#10;  and the CaseHOLD Dataset" />
          <attvalue for="1" value="  While self-supervised learning has made rapid advances in natural language&#10;processing, it remains unclear when researchers should engage in&#10;resource-intensive domain-specific pretraining (domain pretraining). The law,&#10;puzzlingly, has yielded few documented instances of substantial gains to domain&#10;pretraining in spite of the fact that legal language is widely seen to be&#10;unique. We hypothesize that these existing results stem from the fact that&#10;existing legal NLP tasks are too easy and fail to meet conditions for when&#10;domain pretraining can help. To address this, we first present CaseHOLD (Case&#10;Holdings On Legal Decisions), a new dataset comprised of over 53,000+ multiple&#10;choice questions to identify the relevant holding of a cited case. This dataset&#10;presents a fundamental task to lawyers and is both legally meaningful and&#10;difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second,&#10;we assess performance gains on CaseHOLD and existing legal NLP datasets. While&#10;a Transformer architecture (BERT) pretrained on a general corpus (Google Books&#10;and Wikipedia) improves performance, domain pretraining (using corpus of&#10;approximately 3.5M decisions across all courts in the U.S. that is larger than&#10;BERT's) with a custom legal vocabulary exhibits the most substantial&#10;performance gains with CaseHOLD (gain of 7.2% on F1, representing a 12%&#10;improvement on BERT) and consistent performance gains across two other legal&#10;tasks. Third, we show that domain pretraining may be warranted when the task&#10;exhibits sufficient similarity to the pretraining corpus: the level of&#10;performance increase in three legal tasks was directly tied to the domain&#10;specificity of the task. Our findings inform when researchers should engage&#10;resource-intensive pretraining and show that Transformer-based architectures,&#10;too, learn embeddings suggestive of distinct legal language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.04798" label="1611.04798">
        <attvalues>
          <attvalue for="0" value="Toward Multilingual Neural Machine Translation with Universal Encoder&#10;  and Decoder" />
          <attvalue for="1" value="  In this paper, we present our first attempts in building a multilingual&#10;Neural Machine Translation framework under a unified approach. We are then able&#10;to employ attention-based NMT for many-to-many multilingual translation tasks.&#10;Our approach does not require any special treatment on the network architecture&#10;and it allows us to learn minimal number of free parameters in a standard way&#10;of training. Our approach has shown its effectiveness in an under-resourced&#10;translation scenario with considerable improvements up to 2.6 BLEU points. In&#10;addition, the approach has achieved interesting and promising results when&#10;applied in the translation task that there is no direct parallel corpus between&#10;source and target languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.08493" label="1808.08493">
        <attvalues>
          <attvalue for="0" value="Contextual Parameter Generation for Universal Neural Machine Translation" />
          <attvalue for="1" value="  We propose a simple modification to existing neural machine translation (NMT)&#10;models that enables using a single universal model to translate between&#10;multiple languages while allowing for language specific parameterization, and&#10;that can also be used for domain adaptation. Our approach requires no changes&#10;to the model architecture of a standard NMT system, but instead introduces a&#10;new component, the contextual parameter generator (CPG), that generates the&#10;parameters of the system (e.g., weights in a neural network). This parameter&#10;generator accepts source and target language embeddings as input, and generates&#10;the parameters for the encoder and the decoder, respectively. The rest of the&#10;model remains unchanged and is shared across all languages. We show how this&#10;simple modification enables the system to use monolingual data for training and&#10;also perform zero-shot translation. We further show it is able to surpass&#10;state-of-the-art performance for both the IWSLT-15 and IWSLT-17 datasets and&#10;that the learned language embeddings are able to uncover interesting&#10;relationships between languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.02135" label="2204.02135">
        <attvalues>
          <attvalue for="0" value="Exploring the influence of fine-tuning data on wav2vec 2.0 model for&#10;  blind speech quality prediction" />
          <attvalue for="1" value="  Recent studies have shown how self-supervised models can produce accurate&#10;speech quality predictions. Speech representations generated by the pre-trained&#10;wav2vec 2.0 model allows constructing robust predicting models using small&#10;amounts of annotated data. This opens the possibility of developing strong&#10;models in scenarios where labelled data is scarce. It is known that fine-tuning&#10;improves the model's performance; however, it is unclear how the data (e.g.,&#10;language, amount of samples) used for fine-tuning is influencing that&#10;performance. In this paper, we explore how using different speech corpus to&#10;fine-tune the wav2vec 2.0 can influence its performance. We took four speech&#10;datasets containing degradations found in common conferencing applications and&#10;fine-tuned wav2vec 2.0 targeting different languages and data size scenarios.&#10;The fine-tuned models were tested across all four conferencing datasets plus an&#10;additional dataset containing synthetic speech and they were compared against&#10;three external baseline models. Results showed that fine-tuned models were able&#10;to compete with baseline models. Larger fine-tune data guarantee better&#10;performance; meanwhile, diversity in language helped the models deal with&#10;specific languages. Further research is needed to evaluate other wav2vec 2.0&#10;models pre-trained with multi-lingual datasets and to develop prediction models&#10;that are more resilient to language diversity.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech quality assessment is key to monitoring and evaluating the performance of applications and services in which speech is an essential component. Their success depends heavily on assuring acceptable levels of perceived quality, thus the importance of measuring it. Accurate quality assessments can be gathered through subjective experiments in the form of mean opinion scores (MOS) \cite{rec1996p} or using the MUltiple Stimuli with Hidden Reference and Anchor method (MUSHRA) \cite{recommendation2001method}, yet, the resources required (e.g., time, human availability) and their nature make it hard to implement them over an automatic quality monitoring design. Computer algorithms (objective metrics) that can automatically estimate speech quality offer a practical solution to this problem. Within speech objective metrics, Full-reference (FR) metrics, which use both degraded and reference speech signals, have been shown to work very well at estimating the perceived speech quality \cite{hines2012speech, beerends2013perceptual}; however, they depend on the availability of the reference signal. On the other hand, No-reference metrics (NR), which rely solely on the degraded signal, can be deployed in real-time monitoring scenarios, but at the cost of accuracy \cite{malfait2006p}.&#10;&#10;Machine learning (ML) provides the potential for data-driven metrics rather than similarity or distance error-based approaches that have been shown to work well in FR scenarios \cite{hines2012speech}. Self-supervised pre-trained speech models have shown to be very useful for different speech-related tasks, including speech quality prediction \cite{tseng2021utilizing, cooper2021generalization}. The wav2vec 2.0 model, a self-supervised learning (SSL) framework composed of a multi-layer convolutional feature encoder and a Transformer, was successfully applied for MOS prediction using the fixed-size representations generated by the model \cite{cooper2021generalization, tseng2021utilizing}. Models based on deep-learning techniques have also been developed to predict the speech quality with very accurate outcomes. NISQA is a model based on a convolutional neural network (CNN) for frame-wise feature computation, a self-attention block for time-dependency modelling, and an attention-pooling for estimating the overall MOS. This model showed stable results when tested on unknown data and live phone calls \cite{mittag2021nisqa}. Despite their promising results, data-driven models are exposed to bias depending on the type of data used to train them. Collecting data that does not bias the model is a challenge (specifically neutral to: speaker voice/accent, language, degradation) \cite{jimenez2021removing,Jimenez21language,chinen21marginal}.&#10;&#10;Several studies have pointed out the importance of the fine-tuning step to achieve better performance, especially over specific contexts where labelled data is scarce or hard to collect \cite{fan2020exploring, cooper2021generalization}. For wav2vec 2.0, adding a fine-tune step has shown a noticeable accuracy improvement in MOS prediction over different speech datasets \cite{tseng2021utilizing, cooper2021generalization}. Adding this step facilitates the exploration of different testing scenarios involving specific types of degradation, speech languages, or subjective collection methods (e.g., MUSHRA, MOS). However, further research is needed to understand the level of influence that the data (e.g., language, type of distortion, amount of samples) used for fine-tuning is exerting over the model's performance.&#10;&#10;This paper aims: 1) to better understand the role of fine-tuning in wav2vec 2.0 based models and how the content and size of the fine-tuning datasets impact the model performance over a range of speech datasets, 2) to compare wav2vec 2.0 based models and NISQA based models over a range of speech datasets.&#10;With that aim, two experiments were carried out using the fine-tuning system presented in \cite{cooper2021generalization}. The first experiment explores the content language aspect; for this experiment, three models were fine-tuned using only English, German, and Chinese speech samples. The experiment aims to better understand the influence of language on speech quality prediction on datasets with different language samples. The second experiment explores the size of the datasets used for fine-tuning; three models were fine-tuned using 1 k, 10 k, and 50 k mixed speech samples. The experiment seeks to understand the level of influence that fine-tuning data size has on speech quality predictions. Labelled speech datasets containing conferencing degradations (Tencent, IU Bloomington \cite{stupakov2009cosine,richey2018voices}, NISQA Corpus \cite{mittag2021nisqa}, and PSTN \cite{mittag2020dnn}) and synthesised speech (VoiceMOS \cite{cooper2021generalization}) were used to build the fine-tuning datasets and test the resulting models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Fine-Tuning, Linguistics, Engineering, Speech Quality Prediction, Signal Processing, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2104.03017" label="2104.03017">
        <attvalues>
          <attvalue for="0" value="Utilizing Self-supervised Representations for MOS Prediction" />
          <attvalue for="1" value="  Speech quality assessment has been a critical issue in speech processing for&#10;decades. Existing automatic evaluations usually require clean references or&#10;parallel ground truth data, which is infeasible when the amount of data soars.&#10;Subjective tests, on the other hand, do not need any additional clean or&#10;parallel data and correlates better to human perception. However, such a test&#10;is expensive and time-consuming because crowd work is necessary. It thus&#10;becomes highly desired to develop an automatic evaluation approach that&#10;correlates well with human perception while not requiring ground truth data. In&#10;this paper, we use self-supervised pre-trained models for MOS prediction. We&#10;show their representations can distinguish between clean and noisy audios.&#10;Then, we fine-tune these pre-trained models followed by simple linear layers in&#10;an end-to-end manner. The experiment results showed that our framework&#10;outperforms the two previous state-of-the-art models by a significant&#10;improvement on Voice Conversion Challenge 2018 and achieves comparable or&#10;superior performance on Voice Conversion Challenge 2016. We also conducted an&#10;ablation study to further investigate how each module benefits the task. The&#10;experiment results are implemented and reproducible with publicly available&#10;toolkits.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.02635" label="2110.02635">
        <attvalues>
          <attvalue for="0" value="Generalization Ability of MOS Prediction Networks" />
          <attvalue for="1" value="  Automatic methods to predict listener opinions of synthesized speech remain&#10;elusive since listeners, systems being evaluated, characteristics of the&#10;speech, and even the instructions given and the rating scale all vary from test&#10;to test. While automatic predictors for metrics such as mean opinion score&#10;(MOS) can achieve high prediction accuracy on samples from the same test, they&#10;typically fail to generalize well to new listening test contexts. In this&#10;paper, using a variety of networks for MOS prediction including MOSNet and&#10;self-supervised speech models such as wav2vec2, we investigate their&#10;performance on data from different listening tests in both zero-shot and&#10;fine-tuned settings. We find that wav2vec2 models fine-tuned for MOS prediction&#10;have good generalization capability to out-of-domain data even for the most&#10;challenging case of utterance-level predictions in the zero-shot setting, and&#10;that fine-tuning to in-domain data can improve predictions. We also observe&#10;that unseen systems are especially challenging for MOS prediction models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.09494" label="2104.09494">
        <attvalues>
          <attvalue for="0" value="NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech&#10;  Quality Prediction with Crowdsourced Datasets" />
          <attvalue for="1" value="  In this paper, we present an update to the NISQA speech quality prediction&#10;model that is focused on distortions that occur in communication networks. In&#10;contrast to the previous version, the model is trained end-to-end and the&#10;time-dependency modelling and time-pooling is achieved through a Self-Attention&#10;mechanism. Besides overall speech quality, the model also predicts the four&#10;speech quality dimensions Noisiness, Coloration, Discontinuity, and Loudness,&#10;and in this way gives more insight into the cause of a quality degradation.&#10;Furthermore, new datasets with over 13,000 speech files were created for&#10;training and validation of the model. The model was finally tested on a new,&#10;live-talking test dataset that contains recordings of real telephone calls.&#10;Overall, NISQA was trained and evaluated on 81 datasets from different sources&#10;and showed to provide reliable predictions also for unknown speech samples. The&#10;code, model weights, and datasets are open-sourced.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.05053" label="1804.05053">
        <attvalues>
          <attvalue for="0" value="Voices Obscured in Complex Environmental Settings (VOICES) corpus" />
          <attvalue for="1" value="  This paper introduces the Voices Obscured In Complex Environmental Settings&#10;(VOICES) corpus, a freely available dataset under Creative Commons BY 4.0. This&#10;dataset will promote speech and signal processing research of speech recorded&#10;by far-field microphones in noisy room conditions. Publicly available speech&#10;corpora are mostly composed of isolated speech at close-range microphony. A&#10;typical approach to better represent realistic scenarios, is to convolve clean&#10;speech with noise and simulated room response for model training. Despite these&#10;efforts, model performance degrades when tested against uncurated speech in&#10;natural conditions. For this corpus, audio was recorded in furnished rooms with&#10;background noise played in conjunction with foreground speech selected from the&#10;LibriSpeech corpus. Multiple sessions were recorded in each room to accommodate&#10;for all foreground speech-background noise combinations. Audio was recorded&#10;using twelve microphones placed throughout the room, resulting in 120 hours of&#10;audio per microphone. This work is a multi-organizational effort led by SRI&#10;International and Lab41 with the intent to push forward state-of-the-art&#10;distant microphone approaches in signal processing and speech recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.14598" label="2007.14598">
        <attvalues>
          <attvalue for="0" value="DNN No-Reference PSTN Speech Quality Prediction" />
          <attvalue for="1" value="  Classic public switched telephone networks (PSTN) are often a black box for&#10;VoIP network providers, as they have no access to performance indicators, such&#10;as delay or packet loss. Only the degraded output speech signal can be used to&#10;monitor the speech quality of these networks. However, the current&#10;state-of-the-art speech quality models are not reliable enough to be used for&#10;live monitoring. One of the reasons for this is that PSTN distortions can be&#10;unique depending on the provider and country, which makes it difficult to train&#10;a model that generalizes well for different PSTN networks. In this paper, we&#10;present a new open-source PSTN speech quality test set with over 1000&#10;crowdsourced real phone calls. Our proposed no-reference model outperforms the&#10;full-reference POLQA and no-reference P.563 on the validation and test set.&#10;Further, we analyzed the influence of file cropping on the perceived speech&#10;quality and the influence of the number of ratings and training size on the&#10;model accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.12070" label="2401.12070">
        <attvalues>
          <attvalue for="0" value="Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated&#10;  Text" />
          <attvalue for="1" value="  Detecting text generated by modern large language models is thought to be&#10;hard, as both LLMs and humans can exhibit a wide range of complex behaviors.&#10;However, we find that a score based on contrasting two closely related language&#10;models is highly accurate at separating human-generated and machine-generated&#10;text. Based on this mechanism, we propose a novel LLM detector that only&#10;requires simple calculations using a pair of pre-trained LLMs. The method,&#10;called Binoculars, achieves state-of-the-art accuracy without any training&#10;data. It is capable of spotting machine text from a range of modern LLMs&#10;without any model-specific modifications. We comprehensively evaluate&#10;Binoculars on a number of text sources and in varied situations. Over a wide&#10;range of document types, Binoculars detects over 90% of generated samples from&#10;ChatGPT (and other LLMs) at a false positive rate of 0.01%, despite not being&#10;trained on any ChatGPT data.&#10;" />
          <attvalue for="2" value="&#10;&#10;We present a method to detect LLM-generated text that works in the zero-shot setting in which no training examples are used from the LLM source. Even with this strict limitation, our scheme still out-performs all open-source methods for ChatGPT detection and is competitive with or better than commercial APIs, despite these competitors using training samples from ChatGPT~\cite{mitchell_detectgpt_2023,verma_ghostbuster_2023}.&#10;At the same time, because of the zero-shot nature of our detector, the very same detector can spot multiple different LLMs with high accuracy---something that all existing solutions fail to do.&#10;&#10;The ability to detect LLMs in the zero-shot setting addresses issues of growing importance. &#10;Prior research on combating academic plagiarism \cite{turnitin} has fixated strongly on ChatGPT because of its simple and accessible interface. &#10;But more sophisticated actors use LLM APIs to operate bots, create fake product reviews, and spread misinformation on social media platforms at a large scale. &#10;These actors have a wide range of LLMs available to them beyond just ChatGPT, making zero-shot, model-agnostic detection critical for social media moderation and platform integrity assurance~\cite{crothers_machine_2022, bail_difficulty_2023}. &#10;Our zero-shot capability is a departure from existing detectors that rely on model-specific training data and often fail to transfer to new models. &#10;&#10;Our proposed detector, called {Binoculars}, works by viewing text through two lenses. &#10;First, we compute the $\log$ perplexity of the text in question using an ``observer'' LLM.&#10;We then compute next-token predictions using a ``performer'' LLM and compute their perplexity according to the observer. We call this metric cross-perplexity. We observe that perplexity per cross-perplexity is a surprisingly powerful signal to detect LLM-text.&#10;We first motivate this simple observation, and then show that it is sufficient to build a strong zero-shot detector, which we extensively stress-test in a number of text domains.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Artificial Intelligence Evaluation, Artificial Intelligence, Natural Language Processing, Language Model Detection" />
        </attvalues>
      </node>
      <node id="2305.15047" label="2305.15047">
        <attvalues>
          <attvalue for="0" value="Ghostbuster: Detecting Text Ghostwritten by Large Language Models" />
          <attvalue for="1" value="  We introduce Ghostbuster, a state-of-the-art system for detecting&#10;AI-generated text. Our method works by passing documents through a series of&#10;weaker language models, running a structured search over possible combinations&#10;of their features, and then training a classifier on the selected features to&#10;predict whether documents are AI-generated. Crucially, Ghostbuster does not&#10;require access to token probabilities from the target model, making it useful&#10;for detecting text generated by black-box models or unknown model versions. In&#10;conjunction with our model, we release three new datasets of human- and&#10;AI-generated text as detection benchmarks in the domains of student essays,&#10;creative writing, and news articles. We compare Ghostbuster to a variety of&#10;existing detectors, including DetectGPT and GPTZero, as well as a new RoBERTa&#10;baseline. Ghostbuster achieves 99.0 F1 when evaluated across domains, which is&#10;5.9 F1 higher than the best preexisting model. It also outperforms all previous&#10;approaches in generalization across writing domains (+7.5 F1), prompting&#10;strategies (+2.1 F1), and language models (+4.4 F1). We also analyze the&#10;robustness of our system to a variety of perturbations and paraphrasing attacks&#10;and evaluate its performance on documents written by non-native English&#10;speakers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.07321" label="2210.07321">
        <attvalues>
          <attvalue for="0" value="Machine Generated Text: A Comprehensive Survey of Threat Models and&#10;  Detection Methods" />
          <attvalue for="1" value="  Machine generated text is increasingly difficult to distinguish from human&#10;authored text. Powerful open-source models are freely available, and&#10;user-friendly tools that democratize access to generative models are&#10;proliferating. ChatGPT, which was released shortly after the first edition of&#10;this survey, epitomizes these trends. The great potential of state-of-the-art&#10;natural language generation (NLG) systems is tempered by the multitude of&#10;avenues for abuse. Detection of machine generated text is a key countermeasure&#10;for reducing abuse of NLG models, with significant technical challenges and&#10;numerous open problems. We provide a survey that includes both 1) an extensive&#10;analysis of threat models posed by contemporary NLG systems, and 2) the most&#10;complete review of machine generated text detection methods to date. This&#10;survey places machine generated text within its cybersecurity and social&#10;context, and provides strong guidance for future work addressing the most&#10;critical threat models, and ensuring detection systems themselves demonstrate&#10;trustworthiness through fairness, robustness, and accountability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13912" label="2010.13912">
        <attvalues>
          <attvalue for="0" value="Probing Task-Oriented Dialogue Representation from Language Models" />
          <attvalue for="1" value="  This paper investigates pre-trained language models to find out which model&#10;intrinsically carries the most informative representation for task-oriented&#10;dialogue tasks. We approach the problem from two aspects: supervised classifier&#10;probe and unsupervised mutual information probe. We fine-tune a feed-forward&#10;layer as the classifier probe on top of a fixed pre-trained language model with&#10;annotated labels in a supervised way. Meanwhile, we propose an unsupervised&#10;mutual information probe to evaluate the mutual dependence between a real&#10;clustering and a representation clustering. The goals of this empirical paper&#10;are to 1) investigate probing techniques, especially from the unsupervised&#10;mutual information aspect, 2) provide guidelines of pre-trained language model&#10;selection for the dialogue research community, 3) find insights of pre-training&#10;factors for dialogue application that may be the key to success.&#10;" />
          <attvalue for="2" value="&#10;&#10;Task-oriented dialogue systems achieve specific user goals within a limited number of dialogue turns via natural language. They have been used in a wide range of applications, such as booking restaurants~\cite{wen2016network}, providing tourist information~\cite{budzianowski2018multiwoz,wu2019global}, ordering tickets~\cite{schulz2017frame}, and healthcare consultation~\cite{wei2018task}. They are also crucial components of intelligent virtual assistants like Siri, Alexa, and Google Assistant. &#10;&#10;Most of the task-oriented dialogue systems nowadays, are benefited from transfer learning~\cite{WuTradeDST2019,lin2020mintl}, especially pre-trained language models trained on general text, such as BERT~\cite{devlin2018bert} and GPT2~\cite{radford2019language}. &#10;However, previous work claims that linguistic patterns could differ between writing text and human conversation, resulting in a large gap of data distributions~\cite{bao2019plato,wolf2019transfertransfo}. Recently, several approaches are leveraging open-domain data~\cite{henderson2019convert,zhang2019dialogpt}, or aggregating task-oriented data~\cite{wu2020tod} to pre-train language models. &#10;&#10;In this paper, we are interested in answering these questions: which language model has the most informative representations that is better for what task-oriented dialogue task? Does pre-training with dialogue-specific data or different objectives make any difference? &#10;We investigate how good these pre-trained representations are for a task-oriented dialogue system, ignoring the model architectures and training strategies by only probing their final representations with fine-tuning models. A good representation implies better knowledge transferring and domain generalization ability, making downstream applications easier and cheaper to be improved. &#10;&#10;We tackle this problem with two probing solutions: supervised classifier probe and unsupervised mutual information probe.&#10;Classifier probe is commonly used in different NLP tasks such as morphology~\cite{belinkov-etal-2017-neural}, sentence length~\cite{adi2016fine}, or linguistic structure~\cite{hewitt2019structural}. In this setting, we fine-tune a simple classifier for a specific task (e.g., intent identification) on a fixed pre-trained language model. The probe uses supervision to find the best transformation for each sub-task.&#10;&#10;In addition, we present mutual information probe to investigate these language models by directly clustering their output representations, as recent study~\cite{pimentel2020information} suggests that a simple classifier may not be able to achieve the best estimate of mutual information between features and the downstream task.&#10;We apply two clustering techniques, K-means~\cite{lloyd1982least} and Gaussian mixture model~\cite{reynolds2009gaussian}, to calculate its adjusted normalized mutual information (ANMI)~\cite{vinh2010information} between the predicted clustering and the true task-specific clustering.&#10;&#10;We investigate 12 language models, as shown in Table~\ref{tab:model}, where five of them have been pre-trained with dialogue data.&#10;We evaluate four core task-oriented dialogue tasks, domain identification, intent detection, slot tagging, and dialogue act prediction. They correspond to the commonly defined natural language understanding, dialogue state tracking, and dialogue management modules \cite{wen2016network}.&#10;We hope our probing analysis can provide insights to facilitate future task-oriented dialogue research.&#10;Some of the key observations in this work are summarized here (More discussion in Section~\ref{subsec-results}):&#10;\begin{itemize}[leftmargin=*]&#10; \item No matter the open-domain or close-domain, pre-training with dialogue data helps learning better representations for task-oriented dialogue. &#10;&#10;\item Pre-trained language models intrinsically contain more information about intents and dialogue acts but less for slots.&#10;&#10;\item ConveRT~\cite{henderson2019convert} and TOD-BERT-jnt~\cite{wu2020tod} have the highest classification accuracy and mutual information score, suggesting that response selection is useful for dialogue pre-training, especially when we compare TOD-BERT-jnt to TOD-BERT-mlm.&#10;&#10;\item Top models also include TOD-GPT2 and DistilBERT~\cite{sanh2019distilbert}. The distilled version of BERT surprisingly outperforms BERT and other strong baselines such as RoBERTa~\cite{liu2019roberta}.&#10;&#10;\item DialoGPT and GPT2 do not perform well on mutual information evaluation but have a middle-ranking classification accuracy, implying that their representations are informative but not suitable for unsupervised clustering.&#10;&#10;\item Models such as AlBERT~\cite{lan2019albert} and ELECTRA~\cite{clark2020electra} have low classification accuracy and mutual information, showing the least useful information on task-oriented dialogue tasks. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Dialogue Task Analysis, Pre-training Methodology" />
        </attvalues>
      </node>
      <node id="1706.01690" label="1706.01690">
        <attvalues>
          <attvalue for="0" value="A Frame Tracking Model for Memory-Enhanced Dialogue Systems" />
          <attvalue for="1" value="  Recently, resources and tasks were proposed to go beyond state tracking in&#10;dialogue systems. An example is the frame tracking task, which requires&#10;recording multiple frames, one for each user goal set during the dialogue. This&#10;allows a user, for instance, to compare items corresponding to different goals.&#10;This paper proposes a model which takes as input the list of frames created so&#10;far during the dialogue, the current user utterance as well as the dialogue&#10;acts, slot types, and slot values associated with this utterance. The model&#10;then outputs the frame being referenced by each triple of dialogue act, slot&#10;type, and slot value. We show that on the recently published Frames dataset,&#10;this model significantly outperforms a previously proposed rule-based baseline.&#10;In addition, we propose an extensive analysis of the frame tracking task by&#10;dividing it into sub-tasks and assessing their difficulty with respect to our&#10;model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.12005" label="2009.12005">
        <attvalues>
          <attvalue for="0" value="MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems" />
          <attvalue for="1" value="  In this paper, we propose Minimalist Transfer Learning (MinTL) to simplify&#10;the system design process of task-oriented dialogue systems and alleviate the&#10;over-dependency on annotated data. MinTL is a simple yet effective transfer&#10;learning framework, which allows us to plug-and-play pre-trained seq2seq&#10;models, and jointly learn dialogue state tracking and dialogue response&#10;generation. Unlike previous approaches, which use a copy mechanism to&#10;&quot;carryover&quot; the old dialogue states to the new one, we introduce Levenshtein&#10;belief spans (Lev), that allows efficient dialogue state tracking with a&#10;minimal generation length. We instantiate our learning framework with two&#10;pre-trained backbones: T5 and BART, and evaluate them on MultiWOZ. Extensive&#10;experiments demonstrate that: 1) our systems establish new state-of-the-art&#10;results on end-to-end response generation, 2) MinTL-based systems are more&#10;robust than baseline methods in the low resource setting, and they achieve&#10;competitive results with only 20\% training data, and 3) Lev greatly improves&#10;the inference efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-trained Language Models. Language model (LM) pre-training ~\cite{radford2019language,devlin2019bert,yang2019xlnet}, &#10;has been shown to be beneficial in NLP downstream tasks. Generative pre-trained unidirectional LMs (e.g., GPT2) are effective in language generation tasks~\cite{radford2019language,hosseini2020simple,peng2020soloist,lin2020exploring}. Several works have applied a generative pre-training approach in open domain chitchat tasks~\cite{wolf2019transfertransfo,zhang2019dialogpt}, and achieved promising results. On the other hand, bidirectional pre-trained LMs~\cite{devlin2019bert, liu2019roberta} significantly improve the performance of natural language understanding tasks. These models are usually evaluated on classification tasks such as the GLUE benchmark~\cite{wang2018glue}, extractive question answering tasks~\cite{rajpurkar2016squad}, and dialogue context understanding~\cite{wu2020tod}. However, their bidirectionality nature makes them difficult to be applied to natural language generation tasks~\cite{dong2019unified}.&#10;Recent works~\cite{dong2019unified,raffel2019exploring,lewis2019bart} unified unidirectional LM and bidirectional LM pre-training approaches, and proposed a Seq2Seq LM, which are pre-trained with language denoising objectives. A systematic study conducted by~\cite{raffel2019exploring} suggests that the combination of an encoder-decoder architecture and language denoising pre-training objectives yields the best result in both language understanding and generation tasks. Notably, the two latest pre-trained chatbots, Meena~\cite{adiwardana2020towards} and BST~\cite{roller2020recipes}, are also built on an encoder-decoder architecture.&#10;In this work, we transfer the prior knowledge of Seq2Seq LMs to task-oriented dialogues, and successfully improve the SOTA~\cite{zhang2019task} result with less human annotation.&#10;&#10;Task-Oriented Dialogue. &#10;Task-oriented dialogue systems are designed to accomplish a goal described by a user in natural language. Such systems are usually built with a pipeline approach. The pipeline often requires natural language understanding (NLU) for belief state tracking, dialogue management (DM) for deciding which actions to take, and natural language generation (NLG) for generating responses~\cite{williams2007partially}. To simplify the system design and reduce human supervision, several end-to-end trainable systems have been proposed~\cite{bordes2016learning,wen2017network,lei2018sequicity,neelakantan2019neural,eric-manning:2017:EACLshort,ericKVR2017,madotto2018mem2seq}. These methods have been shown to achieve promising results in single-domain tasks. However, the recently proposed multi-domain task-oriented dialogue datasets~\cite{budzianowski2018multiwoz,eric2019multiwoz} &#10;bring new challenges for multi-domain dialogue state tracking and response generation. Several follow up works~\cite{wu2019transferable, chen2019semantically,budzianowski2019hello,mehri2019structured,madotto2020attention} improved on the initial baselines with various methodologies. \cite{zhang2019task} proposed the domain aware multi-decoder network and augmented the system act labels by leveraging the user act annotation, achieving the SOTA results in MultiWoz. However, the aforementioned works rely on task-specific design and extensive human annotations. To reduce the human effort and simplify the system design, we propose a simple transfer learning framework that can be easily set up with pre-trained Seq2Seq models and obtain decent performance with a small fraction of the training data.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2402.09759" label="2402.09759">
        <attvalues>
          <attvalue for="0" value="Efficient Language Adaptive Pre-training: Extending State-of-the-Art&#10;  Large Language Models for Polish" />
          <attvalue for="1" value="  This study explores the potential of fine-tuning foundational English Large&#10;Language Models (LLMs) for generating Polish text. The first step involves&#10;Language Adaptive Pre-training (LAPT) on a high-quality dataset of 3.11 GB,&#10;consisting of 276 million Polish tokens. The LAPT is followed by additional&#10;fine-tuning aimed at solving nine KLEJ challenges. Our trained model&#10;Curie-7B-v1 not only generates Polish text with the lowest perplexity of 3.02&#10;among decoder-based Polish models but also closely rivals the performance of&#10;the best Polish encoder-decoder models with a less than 2% gap on 8 out of 9&#10;tasks. Curie-7B-v1 used approximately 2-3% of a typical dataset size to learn&#10;Polish. The LAPT was completed in less than five days using a consumer GPU,&#10;highlighting the method's efficiency. The proficiency of the model in Polish&#10;was significantly enhanced, demonstrating the viability of this approach for&#10;adding new languages to existing LLMs by training just 1.2% of its parameters.&#10;To contribute to the community's collaborative progress, the model has been&#10;released as open-source.&#10;" />
          <attvalue for="2" value="&#10;LLMs have enhanced the efficiency of many natural language processing (NLP) tasks. This improvement comes with the trade-off of resource-intensive pre-training and inference. At the pre-training phase model gains a general understanding of language, including grammar rules, linguistic patterns, factual information, and reasoning abilities \cite{bommasani2022opportunities}. Currently, all of the \href{ open-source LLMs} are pre-trained on mostly English data. As per the findings of Web Technology Surveys3 \cite{w3techs2024contentlanguage}, more than 51.7\% of the content on the internet is in English, while data in over 100 non-English languages accounts for just 48.3\% of the total. The Polish language contributes to just 1.6\% of the Internet's content. Due to data insufficiency, it is significantly harder to develop a non-English LLM.&#10;\par&#10;The performance of LLMs is influenced by several crucial factors, including the number of model parameters, the number of observed tokens, and the overall quality of the text \cite{xue2023repeat} \cite{DBLP:journals/corr/abs-2001-08361}.&#10;Ideally, the pre-training dataset should scale with the number of model parameters \cite{xue2023repeat}. The resource-intensive nature of pre-training LLMs poses a challenge for low-resource languages such as Polish. For comparison, Meta's LLama 2 was trained on 2 trillion tokens \cite{touvron2023llama} and GPT-3 on roughly 300 billion tokens \cite{brown2020language}. As of today, to the best of the author's knowledge, there are no high-quality open-source datasets of Polish text exceeding 100 billion tokens in size. Developing LLM is a substantial investment. For the sake of comparison, it is claimed that GPT-4 cost is over \$100,000,000, MistralAi's Mistral-7B cost \$500,000 to train, Meta's LLaMa2 70b was trained on 2048 A100 GPUs for 23 days which is estimated to cost around \$2,000,000. These are the costs of just a plain LLM pre-training without including the costs of e.g. data collection or human evaluation necessary to turn these into complex AI assistants or classifiers.&#10;\par&#10; Pre-training isn't the only technique to adopt LLMs to low-resource languages. This can also be done via transfer learning \cite{zhuang2020comprehensive} \cite{DBLP:journals/corr/abs-2004-10964}, fine-tuning LLM for Causal Language Modeling (predicting the next element in a sequence iteratively) \cite{wu2023metalearning} in a supervised manner on text in a language it has merely or never seen in a pre-training phase. LAPT for text generation in a specific language, such as Polish is a potentially effective strategy. For instance, studies have shown that Domain Adaptive Pre-training can significantly improve the performance of foundational LLMs in clinical tasks \cite{gema2023parameterefficient} \cite{karn-etal-2023-shs} \cite{chen2023meditron70b}. &#10;LLaMA \cite{touvron2023llama}, when equipped with a LoRA adapter fine-tuned on medical texts, particularly outperforms foundational models in clinical domain tasks \cite{gema2023parameterefficient}. The study \cite{gema2023parameterefficient} demonstrates that this approach yields substantial improvements, especially in large-scale multilabel classification tasks like diagnoses and procedures classification. This marks a significant advancement over existing custom-trained language models, highlighting the efficiency of LoRA Domain Adaptive Pre-training in highly specialized domains.&#10;&#10;While the specific application to Polish wasn't addressed in the papers we found, the principles of Domain Adaptive Pre-training are widely applicable across languages. This is especially relevant in the context of neural machine translation \cite{bapna2019simple} and cross-lingual tasks, where models are often adapted to new languages and domains to improve their performance. This could be directly applicable to a language like Polish by enabling the model to better capture the syntax, semantics, and unique idiomatic expressions, leading to more accurate and contextually appropriate text generation.&#10;The preliminary evaluations \ref{tab:test-mistral-lang-understanding} have revealed that Mistral-7B English open-source LLM exhibits a basic ability to generate and understand texts in Polish. This capability could be leveraged to significantly improve Polish text generation and comprehension.&#10;&#10;The applications of LLMs include diverse domains such as online retail, medicine, education, engineering, linguistics, and the gaming industry. The integration of these AI solutions not only enhances business productivity but also yields substantial economic benefits. The introduction of LLM-based AI tools has demonstrated a marked improvement in operational efficiency, evidenced by a 14\% average increase in the rate of completed tasks per hour \cite{brynjolfsson2023generative}.&#10;In the context of Poland, the usage of such technologies is almost instant but the development and research of custom solutions is progressing at a slower pace. As of now, there isn't a cutting-edge, high-quality LLM designed exclusively for Polish. This delay forces businesses to depend on externally hosted solutions, such as OpenAI's ChatGPT, particularly in the realm of digital assistants. While these external solutions offer immediate benefits, they also entail financial costs and limit control over data flow. The reliance on external AI technologies, while a temporary solution, underscores the need for the development and deployment of localized LLMs to ensure data sovereignty and capitalize on the economic and technological potential of AI.&#10;\par&#10;This study aims to ascertain whether utilizing an established LLM solution can facilitate the creation of versatile Polish-adapted LLM that is both time-efficient and economically viable. This approach, which involves further building a classifier/regressor on top of LAPT model fine-tuned to solve a domain-specific downstream task that is applicable for business use cases (sentiment analysis, predicting/labelling online reviews, generating texts). &#10;&#10;The following Research Question (RQ) have been defined and will be addressed in this paper:&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Artificial Intelligence Efficiency, Machine Learning, Computer Science, Linguistics, Multilingual Text Generation, Mathematics, Language Model Fine-tuning" />
        </attvalues>
      </node>
      <node id="1911.02685" label="1911.02685">
        <attvalues>
          <attvalue for="0" value="A Comprehensive Survey on Transfer Learning" />
          <attvalue for="1" value="  Transfer learning aims at improving the performance of target learners on&#10;target domains by transferring the knowledge contained in different but related&#10;source domains. In this way, the dependence on a large number of target domain&#10;data can be reduced for constructing target learners. Due to the wide&#10;application prospects, transfer learning has become a popular and promising&#10;area in machine learning. Although there are already some valuable and&#10;impressive surveys on transfer learning, these surveys introduce approaches in&#10;a relatively isolated way and lack the recent advances in transfer learning.&#10;Due to the rapid expansion of the transfer learning area, it is both necessary&#10;and challenging to comprehensively review the relevant studies. This survey&#10;attempts to connect and systematize the existing transfer learning researches,&#10;as well as to summarize and interpret the mechanisms and the strategies of&#10;transfer learning in a comprehensive way, which may help readers have a better&#10;understanding of the current research status and ideas. Unlike previous&#10;surveys, this survey paper reviews more than forty representative transfer&#10;learning approaches, especially homogeneous transfer learning approaches, from&#10;the perspectives of data and model. The applications of transfer learning are&#10;also briefly introduced. In order to show the performance of different transfer&#10;learning models, over twenty representative transfer learning models are used&#10;for experiments. The models are performed on three different datasets, i.e.,&#10;Amazon Reviews, Reuters-21578, and Office-31. And the experimental results&#10;demonstrate the importance of selecting appropriate transfer learning models&#10;for different applications in practice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05884" label="2310.05884">
        <attvalues>
          <attvalue for="0" value="A Meta-Learning Perspective on Transformers for Causal Language Modeling" />
          <attvalue for="1" value="  The Transformer architecture has become prominent in developing large causal&#10;language models. However, mechanisms to explain its capabilities are not well&#10;understood. Focused on the training process, here we establish a meta-learning&#10;view of the Transformer architecture when trained for the causal language&#10;modeling task, by explicating an inner optimization process within the&#10;Transformer. Further, within the inner optimization, we discover and&#10;theoretically analyze a special characteristic of the norms of learned token&#10;representations within Transformer-based causal language models. Our analysis&#10;is supported by experiments in various settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.03042" label="2307.03042">
        <attvalues>
          <attvalue for="0" value="Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain" />
          <attvalue for="1" value="  Adapting pretrained language models to novel domains, such as clinical&#10;applications, traditionally involves retraining their entire set of parameters.&#10;Parameter-Efficient Fine-Tuning (PEFT) techniques for fine-tuning language&#10;models significantly reduce computational requirements by selectively&#10;fine-tuning small subsets of parameters. In this study, we propose a two-step&#10;PEFT framework and evaluate it in the clinical domain. Our approach combines a&#10;specialised PEFT adapter layer designed for clinical domain adaptation with&#10;another adapter specialised for downstream tasks. We evaluate the framework on&#10;multiple clinical outcome prediction datasets, comparing it to clinically&#10;trained language models. Our framework achieves a better AUROC score averaged&#10;across all clinical downstream tasks compared to clinical language models. In&#10;particular, we observe large improvements of 4-5% AUROC in large-scale&#10;multilabel classification tasks, such as diagnoses and procedures&#10;classification. To our knowledge, this study is the first to provide an&#10;extensive empirical analysis of the interplay between PEFT techniques and&#10;domain adaptation in an important real-world domain of clinical applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.03264" label="2306.03264">
        <attvalues>
          <attvalue for="0" value="shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned&#10;  LLMs for Radiology Report Impression Generation" />
          <attvalue for="1" value="  Instruction-tuned generative Large language models (LLMs) like ChatGPT and&#10;Bloomz possess excellent generalization abilities, but they face limitations in&#10;understanding radiology reports, particularly in the task of generating the&#10;IMPRESSIONS section from the FINDINGS section. They tend to generate either&#10;verbose or incomplete IMPRESSIONS, mainly due to insufficient exposure to&#10;medical text data during training. We present a system which leverages&#10;large-scale medical text data for domain-adaptive pre-training of&#10;instruction-tuned LLMs to enhance its medical knowledge and performance on&#10;specific medical tasks. We show that this system performs better in a zero-shot&#10;setting than a number of pretrain-and-finetune adaptation methods on the&#10;IMPRESSIONS generation task, and ranks 1st among participating systems in Task&#10;1B: Radiology Report Summarization at the BioNLP 2023 workshop.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.16079" label="2311.16079">
        <attvalues>
          <attvalue for="0" value="MEDITRON-70B: Scaling Medical Pretraining for Large Language Models" />
          <attvalue for="1" value="  Large language models (LLMs) can potentially democratize access to medical&#10;knowledge. While many efforts have been made to harness and improve LLMs'&#10;medical knowledge and reasoning capacities, the resulting models are either&#10;closed-source (e.g., PaLM, GPT-4) or limited in scale (&lt;= 13B parameters),&#10;which restricts their abilities. In this work, we improve access to large-scale&#10;medical LLMs by releasing MEDITRON: a suite of open-source LLMs with 7B and 70B&#10;parameters adapted to the medical domain. MEDITRON builds on Llama-2 (through&#10;our adaptation of Nvidia's Megatron-LM distributed trainer), and extends&#10;pretraining on a comprehensively curated medical corpus, including selected&#10;PubMed articles, abstracts, and internationally-recognized medical guidelines.&#10;Evaluations using four major medical benchmarks show significant performance&#10;gains over several state-of-the-art baselines before and after task-specific&#10;finetuning. Overall, MEDITRON achieves a 6% absolute performance gain over the&#10;best public baseline in its parameter class and 3% over the strongest baseline&#10;we finetuned from Llama-2. Compared to closed-source LLMs, MEDITRON-70B&#10;outperforms GPT-3.5 and Med-PaLM and is within 5% of GPT-4 and 10% of&#10;Med-PaLM-2. We release our code for curating the medical pretraining corpus and&#10;the MEDITRON model weights to drive open-source development of more capable&#10;medical LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.11771" label="2304.11771">
        <attvalues>
          <attvalue for="0" value="Generative AI at Work" />
          <attvalue for="1" value="  We study the staggered introduction of a generative AI-based conversational&#10;assistant using data from 5,000 customer support agents. Access to the tool&#10;increases productivity, as measured by issues resolved per hour, by 14 percent&#10;on average, with the greatest impact on novice and low-skilled workers, and&#10;minimal impact on experienced and highly skilled workers. We provide suggestive&#10;evidence that the AI model disseminates the potentially tacit knowledge of more&#10;able workers and helps newer workers move down the experience curve. In&#10;addition, we show that AI assistance improves customer sentiment, reduces&#10;requests for managerial intervention, and improves employee retention.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.07737" label="2409.07737">
        <attvalues>
          <attvalue for="0" value="Ruri: Japanese General Text Embeddings" />
          <attvalue for="1" value="  We report the development of Ruri, a series of Japanese general text&#10;embedding models. While the development of general-purpose text embedding&#10;models in English and multilingual contexts has been active in recent years,&#10;model development in Japanese remains insufficient. The primary reasons for&#10;this are the lack of datasets and the absence of necessary expertise. In this&#10;report, we provide a detailed account of the development process of Ruri.&#10;Specifically, we discuss the training of embedding models using synthesized&#10;datasets generated by LLMs, the construction of the reranker for dataset&#10;filtering and knowledge distillation, and the performance evaluation of the&#10;resulting general-purpose text embedding models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Text embeddings are widely used for tasks such as retrieval-augmented generation (RAG) and similar document retrieval~\cite{SentenceBERT,SimCSE,E5}. &#10;In recent years, the development of general-purpose text embedding models trained on diverse datasets has become increasingly common~\cite{E5,mE5,GTE,BGE,JinaBERT}. &#10;However, these efforts have mainly focused on English and multilingual models, where the proportion of Japanese vocabulary and training datasets is relatively small.&#10;Building embedding models using large-scale Japanese datasets may enable the creation of higher-performing models.&#10;&#10;In this report, we present a general-purpose text embedding model specialized for Japanese, which was developed through contrastive pre-training, the construction of synthetic training datasets using LLMs, and fine-tuning on high-quality datasets. &#10;Our contributions are summarized as follows:&#10;\begin{enumerate}[leftmargin=0.5cm, itemsep=0.3pt]&#10; \item We collected datasets for building Japanese embedding models and released them under a permissive license.&#10; \item To address the lack of Japanese retrieval datasets, we constructed a synthetic dataset using LLMs. A performance comparison with and without the synthetic dataset in benchmark tests showed a difference of over 1 point, confirming its utility in training Japanese embedding models.&#10; \item We created a large-scale dataset for contrastive pre-training in Japanese, demonstrating its effectiveness by outperforming existing multilingual models, even when using contrastive pre-training alone.&#10; \item We developed a Japanese reranker, achieving the highest performance among existing Japanese rerankers.&#10; \item We built the Japanese embedding model Ruri, which significantly outperformed existing models in text embedding benchmarks.&#10;\end{enumerate}&#10;Our models and datasets are publicly available\footnotemark\foothreftext{&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Language Model Development, Natural Language Processing, Text Embedding Models" />
        </attvalues>
      </node>
      <node id="2402.05672" label="2402.05672">
        <attvalues>
          <attvalue for="0" value="Multilingual E5 Text Embeddings: A Technical Report" />
          <attvalue for="1" value="  This technical report presents the training methodology and evaluation&#10;results of the open-source multilingual E5 text embedding models, released in&#10;mid-2023. Three embedding models of different sizes (small / base / large) are&#10;provided, offering a balance between the inference efficiency and embedding&#10;quality. The training procedure adheres to the English E5 model recipe,&#10;involving contrastive pre-training on 1 billion multilingual text pairs,&#10;followed by fine-tuning on a combination of labeled datasets. Additionally, we&#10;introduce a new instruction-tuned embedding model, whose performance is on par&#10;with state-of-the-art, English-only models of similar sizes. Information&#10;regarding the model release can be found at&#10;https://github.com/microsoft/unilm/tree/master/e5 .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.13003" label="2301.13003">
        <attvalues>
          <attvalue for="0" value="Knowledge Transfer from Pre-trained Language Models to Cif-based Speech&#10;  Recognizers via Hierarchical Distillation" />
          <attvalue for="1" value="  Large-scale pre-trained language models (PLMs) have shown great potential in&#10;natural language processing tasks. Leveraging the capabilities of PLMs to&#10;enhance automatic speech recognition (ASR) systems has also emerged as a&#10;promising research direction. However, previous works may be limited by the&#10;inflexible structures of PLMs and the insufficient utilization of PLMs. To&#10;alleviate these problems, we propose the hierarchical knowledge distillation&#10;(HKD) on the continuous integrate-and-fire (CIF) based ASR models. To transfer&#10;knowledge from PLMs to the ASR models, HKD employs cross-modal knowledge&#10;distillation with contrastive loss at the acoustic level and knowledge&#10;distillation with regression loss at the linguistic level. Compared with the&#10;original CIF-based model, our method achieves 15% and 9% relative error rate&#10;reduction on the AISHELL-1 and LibriSpeech datasets, respectively.&#10;" />
          <attvalue for="2" value="&#10;&#10;End-to-end (E2E) models have recently made remarkable progress on automatic speech recognition (ASR) tasks. Compared with hybrid models, E2E models are optimized in a unified structure. However, the tight integration in this unified structure hinders the infusion of linguistic knowledge and limits the use of large-scale textual corpora.&#10;&#10;Currently, there are two popular approaches widely used to leverage unpaired text for E2E ASR models: language model (LM) fusion~\cite{gulcehre2015using,sriram2017cold,toshniwal2018comparison,shan2019component} and re-scoring~\cite{chan2016listen}. Apart from them, utilizing large-scale pre-trained language models (PLMs) to improve language modeling of ASR models~\cite{futami2020distilling,huang2021speech,chen2023xllm} is also a practical approach to make use of unpaired text dataset. PLMs possess powerful language modeling abilities, and their outputs contain rich linguistic information that can improve ASR language modeling~\cite{futami2020distilling,kubo2022knowledge}. Therefore, employing PLMs to improve speech recognition has gradually become an important research direction. Until now, the methods used to improve ASR with PLMs can be categorized into three classes: re-scorer based method, model-based method, and knowledge distillation based method. The re-scorer based methods~\cite{shin2019effective,Salazar2020MaskedLM,Chiu2021InnovativeBR,Futami2021ASRRA,Xu2022RescoreBERTDS} convert PLMs into re-scorers and use them to re-score the $N$-best lists or lattices from the first-pass decoding, while not changing the ASR model. Unlike the re-scorer based method, the model-based method and KD-based method focus on improving the ASR model itself. The model-based method refers to using PLM as part of the ASR model. For example, Huang et al.~\cite{huang2021speech} fine-tune PLM as an ASR model with acoustics as cues. Yi et al.~\cite{yi2021efficiently} use the CIF mechanism~\cite{dong2020cif} to combine pre-trained acoustic and language models in a unified structure. Following~\cite{yi2021efficiently}, Zheng et al.~\cite{zheng2021wav} and Deng et al.~\cite{deng2022model} integrate pre-trained acoustic and language models for low-resource ASR and non-autoregressive (NAR) ASR, respectively. However, directly deploying model-based methods may be challenging due to the large size and different structures of PLMs. The KD-based methods transfer knowledge from PLMs to ASR models via knowledge distillation~\cite{Hinton2015DistillingTK}. Futami et al.~\cite{futami2020distilling} distill knowledge from the BERT output distribution to the output distribution of the ASR model. Unlike the probability-based KD, the representation-based KD, which optimizes the similarity between teacher and student representations, transfers knowledge from PLMs to NAR ASR models~\cite{bai2021fast}. Furthermore, the representation-based KD is applied to various ASR models~\cite{kubo2022knowledge,deng2022distill}. However, most KD-based methods transfer the knowledge to only one of acoustics or linguistics and thus cannot fully leverage PLMs.&#10;&#10;In this paper, to explore effective schemes of using PLMs in ASR, we propose a knowledge transfer strategy called hierarchical knowledge distillation (HKD). HKD transfers linguistic knowledge from PLMs to different levels of the ASR model, including the acoustic level. However, it is not easy to directly transfer linguistic knowledge to the acoustic level of E2E models. Unlike other E2E schemes, the continuous integrate-and-fire mechanism (CIF)\cite{dong2020cif}, which generates token-level acoustic representations aligned with the text, provides a natural option for the KD at the acoustic level. Thus, we develop the HKD based on the CIF-based ASR model. Inspired by contrastive knowledge distillation (CKD)\cite{tian2019contrastive,fu2021lrc}, we leverage contrastive loss to transfer the knowledge to the high-level acoustics of CIF-based ASR models. By pushing positive pairs together and negative pairs apart, the contrastive loss encourages the model to capture semantic alignment, giving CKD an advantage over losses that optimize similarity when distilling knowledge across different modalities and structures. At the linguistic level, we apply regression loss to transfer knowledge from the PLM to the linguistic representations. Unlike model-based methods, HKD does not require adapting the ASR model for PLMs. Compared with other representation-based KD methods, HKD transfers the knowledge into the ASR model at multiple levels and applies contrastive distillation to effectively bridge the semantic gap between acoustics and linguistics. Experiments show that HKD achieves 15\% and 9\% relative error rate reduction over the original CIF-based model on AISHELL-1 and LibriSpeech, respectively. The implementation is available on GitHub.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Recognition Systems, Linguistics, Language Model Enhancement, Knowledge Distillation Techniques, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2102.00291" label="2102.00291">
        <attvalues>
          <attvalue for="0" value="Speech Recognition by Simply Fine-tuning BERT" />
          <attvalue for="1" value="  We propose a simple method for automatic speech recognition (ASR) by&#10;fine-tuning BERT, which is a language model (LM) trained on large-scale&#10;unlabeled text data and can generate rich contextual representations. Our&#10;assumption is that given a history context sequence, a powerful LM can narrow&#10;the range of possible choices and the speech signal can be used as a simple&#10;clue. Hence, comparing to conventional ASR systems that train a powerful&#10;acoustic model (AM) from scratch, we believe that speech recognition is&#10;possible by simply fine-tuning a BERT model. As an initial study, we&#10;demonstrate the effectiveness of the proposed idea on the AISHELL dataset and&#10;show that stacking a very simple AM on top of BERT can yield reasonable&#10;performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.07894" label="2202.07894">
        <attvalues>
          <attvalue for="0" value="Knowledge Transfer from Large-scale Pretrained Language Models to&#10;  End-to-end Speech Recognizers" />
          <attvalue for="1" value="  End-to-end speech recognition is a promising technology for enabling compact&#10;automatic speech recognition (ASR) systems since it can unify the acoustic and&#10;language model into a single neural network. However, as a drawback, training&#10;of end-to-end speech recognizers always requires transcribed utterances. Since&#10;end-to-end models are also known to be severely data hungry, this constraint is&#10;crucial especially because obtaining transcribed utterances is costly and can&#10;possibly be impractical or impossible. This paper proposes a method for&#10;alleviating this issue by transferring knowledge from a language model neural&#10;network that can be pretrained with text-only data. Specifically, this paper&#10;attempts to transfer semantic knowledge acquired in embedding vectors of&#10;large-scale language models. Since embedding vectors can be assumed as implicit&#10;representations of linguistic information such as part-of-speech, intent, and&#10;so on, those are also expected to be useful modeling cues for ASR decoders.&#10;This paper extends two types of ASR decoders, attention-based decoders and&#10;neural transducers, by modifying training loss functions to include embedding&#10;prediction terms. The proposed systems were shown to be effective for error&#10;rate reduction without incurring extra computational costs in the decoding&#10;phase.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.06699" label="2101.06699">
        <attvalues>
          <attvalue for="0" value="Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for&#10;  Low-resource Speech Recognition" />
          <attvalue for="1" value="  End-to-end models have achieved impressive results on the task of automatic&#10;speech recognition (ASR). For low-resource ASR tasks, however, labeled data can&#10;hardly satisfy the demand of end-to-end models. Self-supervised acoustic&#10;pre-training has already shown its amazing ASR performance, while the&#10;transcription is still inadequate for language modeling in end-to-end models.&#10;In this work, we fuse a pre-trained acoustic encoder (wav2vec2.0) and a&#10;pre-trained linguistic encoder (BERT) into an end-to-end ASR model. The fused&#10;model only needs to learn the transfer from speech to language during&#10;fine-tuning on limited labeled data. The length of the two modalities is&#10;matched by a monotonic attention mechanism without additional parameters.&#10;Besides, a fully connected layer is introduced for the hidden mapping between&#10;modalities. We further propose a scheduled fine-tuning strategy to preserve and&#10;utilize the text context modeling ability of the pre-trained linguistic&#10;encoder. Experiments show our effective utilizing of pre-trained modules. Our&#10;model achieves better recognition performance on CALLHOME corpus (15 hours)&#10;than other end-to-end models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.09161" label="2109.09161">
        <attvalues>
          <attvalue for="0" value="Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning&#10;  for Low-Resource Speech Recognition" />
          <attvalue for="1" value="  Unifying acoustic and linguistic representation learning has become&#10;increasingly crucial to transfer the knowledge learned on the abundance of&#10;high-resource language data for low-resource speech recognition. Existing&#10;approaches simply cascade pre-trained acoustic and language models to learn the&#10;transfer from speech to text. However, how to solve the representation&#10;discrepancy of speech and text is unexplored, which hinders the utilization of&#10;acoustic and linguistic information. Moreover, previous works simply replace&#10;the embedding layer of the pre-trained language model with the acoustic&#10;features, which may cause the catastrophic forgetting problem. In this work, we&#10;introduce Wav-BERT, a cooperative acoustic and linguistic representation&#10;learning method to fuse and utilize the contextual information of speech and&#10;text. Specifically, we unify a pre-trained acoustic model (wav2vec 2.0) and a&#10;language model (BERT) into an end-to-end trainable framework. A Representation&#10;Aggregation Module is designed to aggregate acoustic and linguistic&#10;representation, and an Embedding Attention Module is introduced to incorporate&#10;acoustic information into BERT, which can effectively facilitate the&#10;cooperation of two pre-trained models and thus boost the representation&#10;learning. Extensive experiments show that our Wav-BERT significantly&#10;outperforms the existing approaches and achieves state-of-the-art performance&#10;on low-resource speech recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.10103" label="2201.10103">
        <attvalues>
          <attvalue for="0" value="Improving non-autoregressive end-to-end speech recognition with&#10;  pre-trained acoustic and language models" />
          <attvalue for="1" value="  While Transformers have achieved promising results in end-to-end (E2E)&#10;automatic speech recognition (ASR), their autoregressive (AR) structure becomes&#10;a bottleneck for speeding up the decoding process. For real-world deployment,&#10;ASR systems are desired to be highly accurate while achieving fast inference.&#10;Non-autoregressive (NAR) models have become a popular alternative due to their&#10;fast inference speed, but they still fall behind AR systems in recognition&#10;accuracy. To fulfill the two demands, in this paper, we propose a NAR&#10;CTC/attention model utilizing both pre-trained acoustic and language models:&#10;wav2vec2.0 and BERT. To bridge the modality gap between speech and text&#10;representations obtained from the pre-trained models, we design a novel&#10;modality conversion mechanism, which is more suitable for logographic&#10;languages. During inference, we employ a CTC branch to generate a target&#10;length, which enables the BERT to predict tokens in parallel. We also design a&#10;cache-based CTC/attention joint decoding method to improve the recognition&#10;accuracy while keeping the decoding speed fast. Experimental results show that&#10;the proposed NAR model greatly outperforms our strong wav2vec2.0 CTC baseline&#10;(15.1% relative CER reduction on AISHELL-1). The proposed NAR model&#10;significantly surpasses previous NAR systems on the AISHELL-1 benchmark and&#10;shows a potential for English tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.07594" label="2102.07594">
        <attvalues>
          <attvalue for="0" value="Fast End-to-End Speech Recognition via Non-Autoregressive Models and&#10;  Cross-Modal Knowledge Transferring from BERT" />
          <attvalue for="1" value="  Attention-based encoder-decoder (AED) models have achieved promising&#10;performance in speech recognition. However, because the decoder predicts text&#10;tokens (such as characters or words) in an autoregressive manner, it is&#10;difficult for an AED model to predict all tokens in parallel. This makes the&#10;inference speed relatively slow. We believe that because the encoder already&#10;captures the whole speech utterance, which has the token-level relationship&#10;implicitly, we can predict a token without explicitly autoregressive language&#10;modeling. When the prediction of a token does not rely on other tokens, the&#10;parallel prediction of all tokens in the sequence is realizable. Based on this&#10;idea, we propose a non-autoregressive speech recognition model called LASO&#10;(Listen Attentively, and Spell Once). The model consists of an encoder, a&#10;decoder, and a position dependent summarizer (PDS). The three modules are based&#10;on basic attention blocks. The encoder extracts high-level representations from&#10;the speech. The PDS uses positional encodings corresponding to tokens to&#10;convert the acoustic representations into token-level representations. The&#10;decoder further captures token-level relationships with the self-attention&#10;mechanism. At last, the probability distribution on the vocabulary is computed&#10;for each token position. Therefore, speech recognition is re-formulated as a&#10;position-wise classification problem. Further, we propose a cross-modal&#10;transfer learning method to refine semantics from a large-scale pre-trained&#10;language model BERT for improving the performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.03582" label="2203.03582">
        <attvalues>
          <attvalue for="0" value="Improving CTC-based speech recognition via knowledge transferring from&#10;  pre-trained language models" />
          <attvalue for="1" value="  Recently, end-to-end automatic speech recognition models based on&#10;connectionist temporal classification (CTC) have achieved impressive results,&#10;especially when fine-tuned from wav2vec2.0 models. Due to the conditional&#10;independence assumption, CTC-based models are always weaker than&#10;attention-based encoder-decoder models and require the assistance of external&#10;language models (LMs). To solve this issue, we propose two knowledge&#10;transferring methods that leverage pre-trained LMs, such as BERT and GPT2, to&#10;improve CTC-based models. The first method is based on representation learning,&#10;in which the CTC-based models use the representation produced by BERT as an&#10;auxiliary learning target. The second method is based on joint classification&#10;learning, which combines GPT2 for text modeling with a hybrid CTC/attention&#10;architecture. Experiment on AISHELL-1 corpus yields a character error rate&#10;(CER) of 4.2% on the test set. When compared to the vanilla CTC-based models&#10;fine-tuned from the wav2vec2.0 models, our knowledge transferring method&#10;reduces CER by 16.1% relatively without external LMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.10699" label="1910.10699">
        <attvalues>
          <attvalue for="0" value="Contrastive Representation Distillation" />
          <attvalue for="1" value="  Often we wish to transfer representational knowledge from one neural network&#10;to another. Examples include distilling a large network into a smaller one,&#10;transferring knowledge from one sensory modality to a second, or ensembling a&#10;collection of models into a single estimator. Knowledge distillation, the&#10;standard approach to these problems, minimizes the KL divergence between the&#10;probabilistic outputs of a teacher and student network. We demonstrate that&#10;this objective ignores important structural knowledge of the teacher network.&#10;This motivates an alternative objective by which we train a student to capture&#10;significantly more information in the teacher's representation of the data. We&#10;formulate this objective as contrastive learning. Experiments demonstrate that&#10;our resulting new objective outperforms knowledge distillation and other&#10;cutting-edge distillers on a variety of knowledge transfer tasks, including&#10;single model compression, ensemble distillation, and cross-modal transfer. Our&#10;method sets a new state-of-the-art in many transfer tasks, and sometimes even&#10;outperforms the teacher network when combined with knowledge distillation.&#10;Code: http://github.com/HobbitLong/RepDistiller.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.07335" label="2012.07335">
        <attvalues>
          <attvalue for="0" value="LRC-BERT: Latent-representation Contrastive Knowledge Distillation for&#10;  Natural Language Understanding" />
          <attvalue for="1" value="  The pre-training models such as BERT have achieved great results in various&#10;natural language processing problems. However, a large number of parameters&#10;need significant amounts of memory and the consumption of inference time, which&#10;makes it difficult to deploy them on edge devices. In this work, we propose a&#10;knowledge distillation method LRC-BERT based on contrastive learning to fit the&#10;output of the intermediate layer from the angular distance aspect, which is not&#10;considered by the existing distillation methods. Furthermore, we introduce a&#10;gradient perturbation-based training architecture in the training phase to&#10;increase the robustness of LRC-BERT, which is the first attempt in knowledge&#10;distillation. Additionally, in order to better capture the distribution&#10;characteristics of the intermediate layer, we design a two-stage training&#10;method for the total distillation loss. Finally, by verifying 8 datasets on the&#10;General Language Understanding Evaluation (GLUE) benchmark, the performance of&#10;the proposed LRC-BERT exceeds the existing state-of-the-art methods, which&#10;proves the effectiveness of our method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.00795" label="1910.00795">
        <attvalues>
          <attvalue for="0" value="Speech-to-speech Translation between Untranscribed Unknown Languages" />
          <attvalue for="1" value="  In this paper, we explore a method for training speech-to-speech translation&#10;tasks without any transcription or linguistic supervision. Our proposed method&#10;consists of two steps: First, we train and generate discrete representation&#10;with unsupervised term discovery with a discrete quantized autoencoder. Second,&#10;we train a sequence-to-sequence model that directly maps the source language&#10;speech to the target language's discrete representation. Our proposed method&#10;can directly generate target speech without any auxiliary or pre-training steps&#10;with a source or target transcription. To the best of our knowledge, this is&#10;the first work that performed pure speech-to-speech translation between&#10;untranscribed unknown languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;Information exchanges among different countries continue to increase. International travelers for tourism, emigration, or foreign study are becoming increasingly diverse, heightening the need for devising a means to offer effective interaction among people who speak different languages. Since automatic spoken-to-speech translation (S2ST) provides an opportunity for people to communicate in their own languages, it significantly overcomes language barriers and closes cross-cultural gaps.&#10;&#10;Many researchers have been developing a S2ST system over the past several decades. A traditional approach in S2ST systems requires effort to construct several components, including automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) synthesis, all of which are trained and tuned independently. Given speech input, ASR processes and transforms speech into text in the source language, MT transforms the source language text to corresponding text in the target language, and finally TTS generates speech from the text in the target language. Significant progress has been made and various commercial speech translation systems are already available for several language pairs. However, more than 6000 languages, spoken by 350 million people, have not been covered yet. Critically, over half of the world's languages actually have no written form; they are only spoken. &#10;&#10;Recently, end-to-end deep learning frameworks have shown impressive performances on many sequence-related tasks, such as ASR, MT, and TTS \cite{chorowski2015attentionasr,bahdanau2014nmt,wang2017tacotron}. Their architecture commonly uses an attentional-based encoder-decoder mechanism, which allows the model to learn the alignments between the source and the target sequence, that can perform end-to-end mapping tasks of different modalities. Many complicated hand-engineered models can also be simplified by letting neural networks find their way to map from input to output spaces. Thus, the approach provides the possibility of learning a direct mapping between the variable-length of the source and the target sequences that are often not known a priori. Several works extended the sequence-to-sequence model’s coverage by directly performing end-to-end speech translation using only a single neural network architecture instead of separately focusing on its components (ASR, MT, and TTS).&#10;&#10;Although the first feasibility was shown by Duong et al. \cite{duong2016sptranslation}, they focused on the alignment between the speech in the source language and the text in the target language because their speech-to-word model did not yield any useful output. The first full-fledged end-to-end attentional-based speech-to-text translation system was successfully performed by B$\acute{e}$rard et al. on a small French-English synthetic corpus \cite{berard2016proof}. But their performance was only compared with statistical MT systems. Weiss et al. \cite{weiss2017direct} demonstrated that end-to-end speech-to-text models on Spanish-English language pairs outperformed neural cascade models. Kano et al. then proved that this approach is possible for distant language pairs such as Japanese-to-English translation \cite{kano2017structured}. Similar to the model by Weiss et al. \cite{weiss2017direct}, although it does not explicitly transcribe the speech into text in the source language, it also doesn’t require supervision from the groundtruth of the source language transcription during training. However, most of these works remain limited to speech-to-text translation and require text transcription in the target language.&#10;&#10;Recently, Jia et al. \cite{jia2019direct} proposed the deep learning model that is trained end-to-end, which learns to map speech spectrograms into target spectrograms in another language that corresponds to the translated content (in a same or different canonical voice). Unfortunately, since training without auxiliary losses leads to extremely poor performance, they provided a solution by integrating auxiliary decoder networks to predict phoneme sequences that correspond to the source and/or target speech. Despite much progress in direct speech translation research, no completely direct speech-to-speech translation has been achieved without any text transcription in source and target languages, during training, has not been achieved yet. Therefore, it remains difficult to scale-up the existing approach to unknown languages without written forms or transcription data available.&#10;&#10;On the other hand, there has been a project that held by speech community to push toward developing unsupervised, data-driven systems that are less reliant on linguistic expertise. Zero resource modeling is an approach where completely unsupervised techniques can learn the elements of a language’s speech hierarchy solely from untranscribed audio data. This means that only spoken audio data are&#10;&#09;available in a specific language, but transcriptions, annotations, and prior knowledge for it are all unavailable. The “Zero Resource Speech Challenge” s eries \cite{versteegh2015zero,dunbar2017zero,dunbar2019zero} was constructed to progress incrementally toward a system that learns an end-to-end spoken dialog (SD) system in an unknown language from scratch just using information available to language learning infants. The ZeroSpeech 2019 \cite{dunbar2019zero} challenge confronts the problem of constructing a speech synthesizer without any text or phonetic labels: TTS without T. It is a continuation of the subword unit discovery track of ZeroSpeech 2015 and 2017 \cite{versteegh2015zero, dunbar2017zero}. 19 systems were submitted, but few studies proposed end-to-end frameworks \cite{liu2019zero,cho2019zero,andros2019zero}. Among these proposed systems, the vector quantized variational autoencoder (VQ-VAE) approach provides a better performance of naturalness based on mean opinion score (MOS) on the generated speech and character error rate after human transcription of the speech synthesis. Further details of the results are available: \href{www.zerospeech.com/2019/results.html}.&#10;&#10;In this paper, we take a step beyond the task of the current ZeroSpeech 2019 and propose a method for training speech to speech translation tasks without any transcription or linguistic supervision. Instead of only discovering subword units and synthesizing them within a certain language, our approach discovers subword units that are directly translated to another language. Our proposed method consists of two steps: (1) we train and generate discrete representation with unsupervised term discovery, which is also based on a discrete quantized autoencoder; (2) we train a sequence-to-sequence model to directly map the source language speech to the target language discrete representation. Our proposed method can directly generate target speech without any auxiliary or pre-training steps with source or target transcription. To the best of our knowledge, this is the first work that performed pure speech-to-speech translation between untranscribed unknown languages.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Translation, Language Modeling, Computer Science, Linguistics, Artificial Intelligence, Signal Processing, Unsupervised Learning" />
        </attvalues>
      </node>
      <node id="1703.08581" label="1703.08581">
        <attvalues>
          <attvalue for="0" value="Sequence-to-Sequence Models Can Directly Translate Foreign Speech" />
          <attvalue for="1" value="  We present a recurrent encoder-decoder deep neural network architecture that&#10;directly translates speech in one language into text in another. The model does&#10;not explicitly transcribe the speech into text in the source language, nor does&#10;it require supervision from the ground truth source language transcription&#10;during training. We apply a slightly modified sequence-to-sequence with&#10;attention architecture that has previously been used for speech recognition and&#10;show that it can be repurposed for this more complex task, illustrating the&#10;power of attention-based models. A single model trained end-to-end obtains&#10;state-of-the-art performance on the Fisher Callhome Spanish-English speech&#10;translation task, outperforming a cascade of independently trained&#10;sequence-to-sequence speech recognition and machine translation models by 1.8&#10;BLEU points on the Fisher test set. In addition, we find that making use of the&#10;training data in both languages by multi-task training sequence-to-sequence&#10;speech translation and recognition models with a shared encoder network can&#10;improve performance by a further 1.4 BLEU points.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.06003" label="1802.06003">
        <attvalues>
          <attvalue for="0" value="Structured-based Curriculum Learning for End-to-end English-Japanese&#10;  Speech Translation" />
          <attvalue for="1" value="  Sequence-to-sequence attentional-based neural network architectures have been&#10;shown to provide a powerful model for machine translation and speech&#10;recognition. Recently, several works have attempted to extend the models for&#10;end-to-end speech translation task. However, the usefulness of these models&#10;were only investigated on language pairs with similar syntax and word order&#10;(e.g., English-French or English-Spanish). In this work, we focus on end-to-end&#10;speech translation tasks on syntactically distant language pairs (e.g.,&#10;English-Japanese) that require distant word reordering.&#10;  To guide the encoder-decoder attentional model to learn this difficult&#10;problem, we propose a structured-based curriculum learning strategy.&#10;  Unlike conventional curriculum learning that gradually emphasizes difficult&#10;data examples, we formalize learning strategies from easier network structures&#10;to more difficult network structures. Here, we start the training with&#10;end-to-end encoder-decoder for speech recognition or text-based machine&#10;translation task then gradually move to end-to-end speech translation task. The&#10;experiment results show that the proposed approach could provide significant&#10;improvements in comparison with the one without curriculum learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.06037" label="1904.06037">
        <attvalues>
          <attvalue for="0" value="Direct speech-to-speech translation with a sequence-to-sequence model" />
          <attvalue for="1" value="  We present an attention-based sequence-to-sequence neural network which can&#10;directly translate speech from one language into speech in another language,&#10;without relying on an intermediate text representation. The network is trained&#10;end-to-end, learning to map speech spectrograms into target spectrograms in&#10;another language, corresponding to the translated content (in a different&#10;canonical voice). We further demonstrate the ability to synthesize translated&#10;speech using the voice of the source speaker. We conduct experiments on two&#10;Spanish-to-English speech translation datasets, and find that the proposed&#10;model slightly underperforms a baseline cascade of a direct speech-to-text&#10;translation model and a text-to-speech synthesis model, demonstrating the&#10;feasibility of the approach on this very challenging task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.01128" label="2401.01128">
        <attvalues>
          <attvalue for="0" value="SSP: A Simple and Safe automatic Prompt engineering method towards&#10;  realistic image synthesis on LVM" />
          <attvalue for="1" value="  Recently, text-to-image (T2I) synthesis has undergone significant&#10;advancements, particularly with the emergence of Large Language Models (LLM)&#10;and their enhancement in Large Vision Models (LVM), greatly enhancing the&#10;instruction-following capabilities of traditional T2I models. Nevertheless,&#10;previous methods focus on improving generation quality but introduce unsafe&#10;factors into prompts. We explore that appending specific camera descriptions to&#10;prompts can enhance safety performance. Consequently, we propose a simple and&#10;safe prompt engineering method (SSP) to improve image generation quality by&#10;providing optimal camera descriptions. Specifically, we create a dataset from&#10;multi-datasets as original prompts. To select the optimal camera, we design an&#10;optimal camera matching approach and implement a classifier for original&#10;prompts capable of automatically matching. Appending camera descriptions to&#10;original prompts generates optimized prompts for further LVM image generation.&#10;Experiments demonstrate that SSP improves semantic consistency by an average of&#10;16% compared to others and safety metrics by 48.9%.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, some text-to-image (T2I) synthesis methods, such as ~\cite{nichol2021glide, DingYHZZYLZSYT2021CogView, RombachBLEO22StableDiffusion, DingZHT2022CogView2, SahariaCSLWDGLA22imagen} have undergone significant advancements, particularly with the emergence of Large Language Models (LLM) and their enhancement in Large Vision Models (LVM), greatly enhancing the instruction-following capabilities of traditional T2I models.&#10;However there is unable to enhance the performance of LVM through fine-tuning or retraining, researchers have begun searching for methods that can boost the generation performance without the additional training process. The key to improving the generation quality of T2I models lies in designing effective prompts~\cite{dong2022dreamartist}. &#10;LVM exhibits strong text comprehension~\cite{jiang2020can}, which brings a prompt engineering method~\cite{gao2020making} for better meet specific image generation requirements~\cite{oppenlaender2023taxonomy}.&#10;&#10;Nevertheless, previous prompt engineering methods involved introducing random additional vocabulary into prompts to generate high-quality prompts. Specifically, BeautifulPrompt~\cite{cao2023beautifulprompt} trains model using both low-quality and high-quality prompt pairs to enhances the quality of the generated images through Reinforcement Learning(RL). BestPrompt~\cite{pavlichenko2023best} selects and compares different candidate keyword sets using genetic algorithms to generate more aesthetically pleasing images.&#10;PromptMagician~\cite{feng2023promptmagician} is trained using a substantial amount of high-quality prompts, aiming to generate optimal prompts for Stable Diffusion.&#10;However, above approaches that involve the process of randomly adding vocabulary are uncontrolled. This randomness may change the original semantics and introduce unsafe factors, raising safety concerns~\cite{ba2023surrogateprompt}.&#10;&#10;To address these challenges, we observe GPT and high-quality image datasets, noting three key points: a) Real-world images must be captured by a camera for computer recognition and storage. Different cameras are dedicated to optimizing different shooting themes (e.g., vibrant color cameras for natural landscapes). b) LVM training data is complex, with some containing camera models; different camera prompts change the image generation effect. c) Appending specific camera descriptions to prompts can prevent the introduction of unsafe factors, enhancing safety performance.&#10;&#10;Therefore, we introduce SSP, a simple and safe prompt engineering method designed to improve image generation quality and avoid introducing unsafe factors by providing camera descriptions. &#10;Specifically, we first create a dataset sourced from various public text-only or text-image pairs datasets, which is then summarized and filtered by GPT-4~\cite{openai2023gpt4}. Subsequently, we design an optimal camera matching approach by fine-tuning the BERT model~\cite{devlin2018bert} with this dataset, serving as a classifier for original prompts capable of automatic matching. Optimized prompts are obtained by appending the optimal camera descriptions to the originals. We then input these optimized prompts into GPT-4 to generate images. &#10;A comparison of prompts and generating images are shown in Fig.~\ref{banner}.&#10;&#10;In this study, our main contributions can be summarized as follows: &#10;\begin{itemize}&#10; \item We release a new dataset for image generation prompt optimization, suitable for visual prompt optimization tasks.&#10; \item We introduce SSP, a novel method designed to improve image generation quality by providing optimal camera descriptions without altering the original content or introducing unsafe factors.&#10; \item Extensive experiments demonstrate the superior performance of SSP compared to two robust baselines. SSP shows an average improvement of 16\% in prompt consistency and 5\% in text-image alignment compared to the baselines, with a 48.9\% increase in safety metrics. &#10; \item Experiments with text feature analysis have proved that prompt engineering in LVM can change the data distribution of prompt, so as to achieve better generation results. This conclusion may inspire other strategies for prompt-driven on large model optimization.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Image Generation Safety, Linguistics, Cognitive Science, Computer Vision, Artificial Intelligence, Natural Language Processing, Text-to-Image Synthesis" />
        </attvalues>
      </node>
      <node id="2112.10741" label="2112.10741">
        <attvalues>
          <attvalue for="0" value="GLIDE: Towards Photorealistic Image Generation and Editing with&#10;  Text-Guided Diffusion Models" />
          <attvalue for="1" value="  Diffusion models have recently been shown to generate high-quality synthetic&#10;images, especially when paired with a guidance technique to trade off diversity&#10;for fidelity. We explore diffusion models for the problem of text-conditional&#10;image synthesis and compare two different guidance strategies: CLIP guidance&#10;and classifier-free guidance. We find that the latter is preferred by human&#10;evaluators for both photorealism and caption similarity, and often produces&#10;photorealistic samples. Samples from a 3.5 billion parameter text-conditional&#10;diffusion model using classifier-free guidance are favored by human evaluators&#10;to those from DALL-E, even when the latter uses expensive CLIP reranking.&#10;Additionally, we find that our models can be fine-tuned to perform image&#10;inpainting, enabling powerful text-driven image editing. We train a smaller&#10;model on a filtered dataset and release the code and weights at&#10;https://github.com/openai/glide-text2im.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.06752" label="2311.06752">
        <attvalues>
          <attvalue for="0" value="BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image&#10;  Synthesis" />
          <attvalue for="1" value="  Recently, diffusion-based deep generative models (e.g., Stable Diffusion)&#10;have shown impressive results in text-to-image synthesis. However, current&#10;text-to-image models often require multiple passes of prompt engineering by&#10;humans in order to produce satisfactory results for real-world applications. We&#10;propose BeautifulPrompt, a deep generative model to produce high-quality&#10;prompts from very simple raw descriptions, which enables diffusion-based models&#10;to generate more beautiful images. In our work, we first fine-tuned the&#10;BeautifulPrompt model over low-quality and high-quality collecting prompt&#10;pairs. Then, to ensure that our generated prompts can generate more beautiful&#10;images, we further propose a Reinforcement Learning with Visual AI Feedback&#10;technique to fine-tune our model to maximize the reward values of the generated&#10;prompts, where the reward values are calculated based on the PickScore and the&#10;Aesthetic Scores. Our results demonstrate that learning from visual AI feedback&#10;promises the potential to improve the quality of generated prompts and images&#10;significantly. We further showcase the integration of BeautifulPrompt to a&#10;cloud-native AI platform to provide better text-to-image generation service in&#10;the cloud.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.09036" label="2307.09036">
        <attvalues>
          <attvalue for="0" value="PromptMagician: Interactive Prompt Engineering for Text-to-Image&#10;  Creation" />
          <attvalue for="1" value="  Generative text-to-image models have gained great popularity among the public&#10;for their powerful capability to generate high-quality images based on natural&#10;language prompts. However, developing effective prompts for desired images can&#10;be challenging due to the complexity and ambiguity of natural language. This&#10;research proposes PromptMagician, a visual analysis system that helps users&#10;explore the image results and refine the input prompts. The backbone of our&#10;system is a prompt recommendation model that takes user prompts as input,&#10;retrieves similar prompt-image pairs from DiffusionDB, and identifies special&#10;(important and relevant) prompt keywords. To facilitate interactive prompt&#10;refinement, PromptMagician introduces a multi-level visualization for the&#10;cross-modal embedding of the retrieved images and recommended keywords, and&#10;supports users in specifying multiple criteria for personalized exploration.&#10;Two usage scenarios, a user study, and expert interviews demonstrate the&#10;effectiveness and usability of our system, suggesting it facilitates prompt&#10;engineering and improves the creativity support of the generative text-to-image&#10;model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.14122" label="2309.14122">
        <attvalues>
          <attvalue for="0" value="SurrogatePrompt: Bypassing the Safety Filter of Text-To-Image Models via&#10;  Substitution" />
          <attvalue for="1" value="  Advanced text-to-image models such as DALL-E 2 and Midjourney possess the&#10;capacity to generate highly realistic images, raising significant concerns&#10;regarding the potential proliferation of unsafe content. This includes adult,&#10;violent, or deceptive imagery of political figures. Despite claims of rigorous&#10;safety mechanisms implemented in these models to restrict the generation of&#10;not-safe-for-work (NSFW) content, we successfully devise and exhibit the first&#10;prompt attacks on Midjourney, resulting in the production of abundant&#10;photorealistic NSFW images. We reveal the fundamental principles of such prompt&#10;attacks and suggest strategically substituting high-risk sections within a&#10;suspect prompt to evade closed-source safety measures. Our novel framework,&#10;SurrogatePrompt, systematically generates attack prompts, utilizing large&#10;language models, image-to-text, and image-to-image modules to automate attack&#10;prompt creation at scale. Evaluation results disclose an 88% success rate in&#10;bypassing Midjourney's proprietary safety filter with our attack prompts,&#10;leading to the generation of counterfeit images depicting political figures in&#10;violent scenarios. Both subjective and objective assessments validate that the&#10;images generated from our attack prompts present considerable safety hazards.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.10558" label="2402.10558">
        <attvalues>
          <attvalue for="0" value="Neural paraphrasing by automatically crawled and aligned sentence pairs" />
          <attvalue for="1" value="  Paraphrasing is the task of re-writing an input text using other words,&#10;without altering the meaning of the original content. Conversational systems&#10;can exploit automatic paraphrasing to make the conversation more natural, e.g.,&#10;talking about a certain topic using different paraphrases in different time&#10;instants. Recently, the task of automatically generating paraphrases has been&#10;approached in the context of Natural Language Generation (NLG). While many&#10;existing systems simply consist in rule-based models, the recent success of the&#10;Deep Neural Networks in several NLG tasks naturally suggests the possibility of&#10;exploiting such networks for generating paraphrases. However, the main obstacle&#10;toward neural-network-based paraphrasing is the lack of large datasets with&#10;aligned pairs of sentences and paraphrases, that are needed to efficiently&#10;train the neural models. In this paper we present a method for the automatic&#10;generation of large aligned corpora, that is based on the assumption that news&#10;and blog websites talk about the same events using different narrative styles.&#10;We propose a similarity search procedure with linguistic constraints that,&#10;given a reference sentence, is able to locate the most similar candidate&#10;paraphrases out from millions of indexed sentences. The data generation process&#10;is evaluated in the case of the Italian language, performing experiments using&#10;pointer-based deep neural architectures.&#10;" />
          <attvalue for="2" value="&#10;&#10;Humans can easily rephrase a text passage without changing the meaning of the considered text, and they can communicate the same piece of information using different words, e.g., during a conversation. Humans have the cognitive ability to deeply understand the contents that are described in the considered text passage, of the involved events and facts. Machines commonly lack this ability, and designing systems that automatically generate paraphrases of an input text is a challenging task that, recently, has been the subject of a number of studies in the Natural Language Generation (NLG) literature.&#10;In particular, the task of rewriting text has been classified into three different categories \cite{madnani2010generating}: lexical paraphrasing, that is based or replacing words with other words with same meaning \cite{bolshakov2004synonymous}; phrasal paraphrasing, when the paraphrase is created acting on fragments with the same meaning \cite{ganitkevitch2013ppdb}; sentential paraphrasing when the paraphrases is performed at a sentence level, considering sentences with the same meaning \cite{barzilay2001extracting,barzilay2003learning,dolan2004unsupervised}.&#10;&#10;In this paper we let the machine learn to generate paraphrases of an input text, without enforcing any specific word-level, fragment-level, or sentence-level operations. In particular, we follow the idea that paraphrasing can be approached as a special Statistical Machine Translation task in which the two languages involved in the translation, referred to as $L_{input}$ and $L_{target}$, respectively, do coincide ($L_{input} \equiv L_{target}$) but they are modeled with two different languages models, one that is about the input text and one that is about the target paraphrase. &#10;The success of using Deep Neural Networks in Statistical Machine Translation \cite{kalchbrenner2013recurrent}, in particular sequence-to-sequence architectures \cite{sutskever2014sequence}, suggests that the task of text paraphrasing can be addressed using similar approaches.&#10;Unfortunately, while in the Machine Translation field a large number of datasets with millions of aligned pairs of sentences can be easily found (where each pair is composed of a sentence from $L_{input}$ and the associated translation in $L_{target}$), in the case of paraphrasing very large datasets of (sentence, paraphrase) pairs do not exist, especially in the case of languages different from English.&#10;The creation of such ``supervised'' (i.e., aligned) dataset requires an expensive annotation activity which turns out to be very difficult to scale to large amounts of data. &#10;&#10;We propose a method for building a dataset of aligned sentences that can be used to train machine learning models to approach the task of paraphrasing. &#10;The resulting dataset consists of a set of pairs of textual sentences, each of them composed of an input sentence and a target paraphrase. Our method is based on the idea that various news and blog websites normally report the same important facts and events using different idiolects. &#10;An idiolect is defined as the individual distinctive and unique use of language, concerning the morpho-syntactic and stylistic features.&#10;This assumption allows us to model the problem of dataset building as a Highly Constrained Sentence Similarity Search (HCSSS), where the results of a search in a document base must satisfy hard linguistic constraints (morphological, syntactic and semantic constraints) in addition to word co-occurrences.&#10;The proposed method has been evaluated by crawling a large number of articles from Italian newspaper sites and blogs, thus generating a corpus of pairs that we use to train a deep sequence-to-sequence neural network model. In particular, we focus on a Pointer network \cite{gu2016incorporating,vaswani2017attention,see2017get}) that, given an input text, learns to generate a possible target paraphrase.&#10;We report some preliminary results and we discuss the quality and the limits of the considered neural models in function of the properties of the data generated by the proposed mechanism. &#10;&#10;This paper is organized as follows. Section \ref{sec:data} describes the proposed process to automatically crawl and align sentence pairs. Section \ref{sec:model} summarizes the properties of the considered deep neural network model, while Section \ref{sec:exp} reports experimental results and discussions about them. Finally, Section \ref{sec:concl} concludes the paper with some proposals of future work.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Paraphrasing Techniques, Linguistics, Deep Neural Networks, Artificial Intelligence, Mathematics, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="cs/0304006" label="cs/0304006">
        <attvalues>
          <attvalue for="0" value="Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence&#10;  Alignment" />
          <attvalue for="1" value="  We address the text-to-text generation problem of sentence-level paraphrasing&#10;-- a phenomenon distinct from and more difficult than word- or phrase-level&#10;paraphrasing. Our approach applies multiple-sequence alignment to sentences&#10;gathered from unannotated comparable corpora: it learns a set of paraphrasing&#10;patterns represented by word lattice pairs and automatically determines how to&#10;apply these patterns to rewrite new sentences. The results of our evaluation&#10;experiments show that the system derives accurate paraphrases, outperforming&#10;baseline systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1603.06393" label="1603.06393">
        <attvalues>
          <attvalue for="0" value="Incorporating Copying Mechanism in Sequence-to-Sequence Learning" />
          <attvalue for="1" value="  We address an important problem in sequence-to-sequence (Seq2Seq) learning&#10;referred to as copying, in which certain segments in the input sequence are&#10;selectively replicated in the output sequence. A similar phenomenon is&#10;observable in human language communication. For example, humans tend to repeat&#10;entity names or even long phrases in conversation. The challenge with regard to&#10;copying in Seq2Seq is that new machinery is needed to decide when to perform&#10;the operation. In this paper, we incorporate copying into neural network-based&#10;Seq2Seq learning and propose a new model called CopyNet with encoder-decoder&#10;structure. CopyNet can nicely integrate the regular way of word generation in&#10;the decoder with the new copying mechanism which can choose sub-sequences in&#10;the input sequence and put them at proper places in the output sequence. Our&#10;empirical study on both synthetic data sets and real world data sets&#10;demonstrates the efficacy of CopyNet. For example, CopyNet can outperform&#10;regular RNN-based model with remarkable margins on text summarization tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.01337" label="2201.01337">
        <attvalues>
          <attvalue for="0" value="ZeroBERTo: Leveraging Zero-Shot Text Classification by Topic Modeling" />
          <attvalue for="1" value="  Traditional text classification approaches often require a good amount of&#10;labeled data, which is difficult to obtain, especially in restricted domains or&#10;less widespread languages. This lack of labeled data has led to the rise of&#10;low-resource methods, that assume low data availability in natural language&#10;processing. Among them, zero-shot learning stands out, which consists of&#10;learning a classifier without any previously labeled data. The best results&#10;reported with this approach use language models such as Transformers, but fall&#10;into two problems: high execution time and inability to handle long texts as&#10;input. This paper proposes a new model, ZeroBERTo, which leverages an&#10;unsupervised clustering step to obtain a compressed data representation before&#10;the classification task. We show that ZeroBERTo has better performance for long&#10;inputs and shorter execution time, outperforming XLM-R by about 12% in the F1&#10;score in the FolhaUOL dataset. Keywords: Low-Resource NLP, Unlabeled data,&#10;Zero-Shot Learning, Topic Modeling, Transformers.&#10;" />
          <attvalue for="2" value="&#10;&#10;The current success of supervised learning techniques in real-world Natural Language Processing (NLP) applications is undeniable.&#10;While these techniques require a good set of labeled data, large corpora of annotated texts are difficult to obtain, as people (sometimes experts) are needed to create manual annotations or revise and correct predefined labels.&#10;This problem is even more critical in languages other than English: statistics show that English is used by \SI{63.1}{\percent} of the population on the internet, while Portuguese, for instance, is only used by \SI{0.7}{\percent}. This scenario has contributed to the rise of Low-Resource NLP, which aims to develop techniques to deal with low data availability in a specific language or application domain \cite{hedderich2020survey}. &#10;&#10;Recently, the concept of zero-shot learning emerged in NLP: a semi-supervised approach in which models can present results equivalent to those of supervised tasks, such as classification in the absence of labeled data.&#10;Current approaches to the zero-shot text classification task ({0shot-TC}) make use of the good performance that Transformers have demonstrated in text entailment tasks \cite{yin2019benchmarking}. &#10;In order to be able to process text in a way that is not uniquely suited to any specific task or data-set, these Transformers are first pre-trained in large general databases (usually taken from Wikipedia) and then fine-tuned into a small mainstream data-set for the natural language inference task (such as GLUE \cite{wang2019glue} and XNLI \cite{conneau2018xnli}). &#10;However, the use of models based entirely on Transformers falls into two critical problems: (i) limitation of the maximum size of the input text, and (ii) long run-time for large volumes of data.&#10;While there are transformer-based solutions to these problems individually \cite{beltagy2020longformer,sanh2019distilbert,Zaheer2020BigBT}, to the best of our knowledge, there is no solution that addresses both, nor even in the context of 0shot-TC. &#10;&#10;In this paper, we propose a new hybrid model that merges Transformers with unsupervised learning, called ZeroBERTo -- Zero-shot BERT based on Topic Modeling --, which is able to classify texts by learning only from unlabeled data. &#10;Our contribution not only handles long inputs -- not limiting the input size and considering every input token to encode the data -- but also offers a faster execution time.&#10;We propose an experimental setup with unlabeled data, simulating low-resource scenarios where real-life NLP researchers may find themselves. Then, we compare ZeroBERTo to a fully-Transformer-based zero-shot on a categorization dataset in Portuguese, FolhaUOL.&#10;Our results show that our model outperforms the previous one, in the best scenario, with about \SI{12}{\percent} better label aware weighted F1-score and around 13 times faster total time.&#10;&#10;The paper is structured as follows: Sect. \ref{sec:background} presents a background of how it is possible to move from data scarcity to zero-shot learning, as well as the related work on getting the best model for the 0shot-TC task. Sect. \ref{sec:proposedframework} formalizes the ZeroBERTo task and describe its training and inference procedures. Then, Sect. \ref{sec:experiments} describes the experimental setup that makes it possible to simulate low-resource scenarios to evaluate the proposed model. Finally, the discussion of the results of the experiments along with our final remarks is in Sect. \ref{sec:discussion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Zero-Shot Learning, Low-Resource Methods, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.12309" label="2010.12309">
        <attvalues>
          <attvalue for="0" value="A Survey on Recent Approaches for Natural Language Processing in&#10;  Low-Resource Scenarios" />
          <attvalue for="1" value="  Deep neural networks and huge language models are becoming omnipresent in&#10;natural language applications. As they are known for requiring large amounts of&#10;training data, there is a growing body of work to improve the performance in&#10;low-resource settings. Motivated by the recent fundamental changes towards&#10;neural models and the popular pre-train and fine-tune paradigm, we survey&#10;promising approaches for low-resource natural language processing. After a&#10;discussion about the different dimensions of data availability, we give a&#10;structured overview of methods that enable learning when training data is&#10;sparse. This includes mechanisms to create additional labeled data like data&#10;augmentation and distant supervision as well as transfer learning settings that&#10;reduce the need for target supervision. A goal of our survey is to explain how&#10;these methods differ in their requirements as understanding them is essential&#10;for choosing a technique suited for a specific low-resource setting. Further&#10;key aspects of this work are to highlight open issues and to outline promising&#10;directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.07928" label="2305.07928">
        <attvalues>
          <attvalue for="0" value="AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation&#10;  Framework For Multilingual Language Inference" />
          <attvalue for="1" value="  Knowledge distillation is of key importance to launching multilingual&#10;pre-trained language models for real applications. To support cost-effective&#10;language inference in multilingual settings, we propose AMTSS, an adaptive&#10;multi-teacher single-student distillation framework, which allows distilling&#10;knowledge from multiple teachers to a single student. We first introduce an&#10;adaptive learning strategy and teacher importance weight, which enables a&#10;student to effectively learn from max-margin teachers and easily adapt to new&#10;languages. Moreover, we present a shared student encoder with different&#10;projection layers in support of multiple languages, which contributes to&#10;largely reducing development and machine cost. Experimental results show that&#10;AMTSS gains competitive results on the public XNLI dataset and the realistic&#10;industrial dataset AliExpress (AE) in the E-commerce scenario.&#10;" />
          <attvalue for="2" value="&#10;Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}.&#10;As pre-trained language models are usually computationally expensive, many transformer distillation methods~\cite{jiao-etal-2020-tinybert,liu2020cross} have been proposed, which distill knowledge from a large teacher model to a lightweight student network, to accelerate inference and reduce model size while maintaining the accuracy.&#10;&#10;The majority of works mainly focus on learning from a single teacher as in Figure~\ref{fig:pipeline} (a), while only a few studies have considered to learn from multiple teachers~\cite{peng2020mtss,liu2020adaptive} which allows to select the optimal model as teacher for different domains during the student training. &#10;It is essential for cross domain knowledge distillation, especially in multilingual NLI.&#10;&#10;In this work, we focus on chatbot settings, which currently supports nearly twenty kinds of languages on E-commerce language inference, and are constantly accepting new languages.&#10;Considering the linearly increased development and machine cost, we cannot develop a model instance for each language, which has poor scalability.&#10;Meanwhile, as we need to distill from around twenty teacher models each time when dealing with a new coming language, current multi-teacher distillation methods~\cite{peng2020mtss,liu2020adaptive,10.1145/3097983.3098135,yang2020model} are not fit for our scenario. &#10;This begs an important question in practice: can we distill knowledge from multiple teachers in a multilingual settings to cost-effectively support multiple languages and easily adapt to a new language?&#10;&#10;To address this challenge, we propose an adaptive multi-teacher single-student distillation framework (AMTSS). &#10;Firstly, we fine-tune a pre-trained language model for each language and obtain the optimal teacher, either monolingual or multilingual. &#10;Then, we distill the knowledge from teachers to a single student with a novel adaptive training strategy and a shared student encoder with different projection layers, instead of training several students for each language. &#10;For adapting to the new languages, we fine-tune the student model to learn from the max-margin teachers instead of re-training the student model with all teachers.&#10;&#10;The contributions of this work are as follows:&#10;\begin{itemize}&#10; \item We propose an adaptive multi-teacher single-student knowledge distillation framework, consists of a shared student encoder with different projection layers in support of multiple languages in a cost-effective manner.&#10; \item We propose a weight based adaptive learning strategy that enables a student model to effectively learn from max-margin teachers with the importance weights, and easily adapt to new coming languages.&#10; \item We demonstrated the effectiveness of AMTSS through the experimental evaluation on public XNLI dataset and a realistic industrial dataset AliExpress (AE) in E-commerce scenario.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Multilingual Language Models, Knowledge Distillation, Adaptive Learning Strategies, Mathematics" />
        </attvalues>
      </node>
      <node id="2011.03080" label="2011.03080">
        <attvalues>
          <attvalue for="0" value="EXAMS: A Multi-Subject High School Examinations Dataset for&#10;  Cross-Lingual and Multilingual Question Answering" />
          <attvalue for="1" value="  We propose EXAMS -- a new benchmark dataset for cross-lingual and&#10;multilingual question answering for high school examinations. We collected more&#10;than 24,000 high-quality high school exam questions in 16 languages, covering 8&#10;language families and 24 school subjects from Natural Sciences and Social&#10;Sciences, among others.&#10;  EXAMS offers a fine-grained evaluation framework across multiple languages&#10;and subjects, which allows precise analysis and comparison of various models.&#10;We perform various experiments with existing top-performing multilingual&#10;pre-trained models and we show that EXAMS offers multiple challenges that&#10;require multilingual knowledge and reasoning in multiple domains. We hope that&#10;EXAMS will enable researchers to explore challenging reasoning and knowledge&#10;transfer methods and pre-trained models for school question answering in&#10;various languages which was not possible before. The data, code, pre-trained&#10;models, and evaluation are available at https://github.com/mhardalov/exams-qa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.14271" label="2010.14271">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Machine Reading Comprehension with Language Branch&#10;  Knowledge Distillation" />
          <attvalue for="1" value="  Cross-lingual Machine Reading Comprehension (CLMRC) remains a challenging&#10;problem due to the lack of large-scale annotated datasets in low-source&#10;languages, such as Arabic, Hindi, and Vietnamese. Many previous approaches use&#10;translation data by translating from a rich-source language, such as English,&#10;to low-source languages as auxiliary supervision. However, how to effectively&#10;leverage translation data and reduce the impact of noise introduced by&#10;translation remains onerous. In this paper, we tackle this challenge and&#10;enhance the cross-lingual transferring performance by a novel augmentation&#10;approach named Language Branch Machine Reading Comprehension (LBMRC). A&#10;language branch is a group of passages in one single language paired with&#10;questions in all target languages. We train multiple machine reading&#10;comprehension (MRC) models proficient in individual language based on LBMRC.&#10;Then, we devise a multilingual distillation approach to amalgamate knowledge&#10;from multiple language branch models to a single model for all target&#10;languages. Combining the LBMRC and multilingual distillation can be more robust&#10;to the data noises, therefore, improving the model's cross-lingual ability.&#10;Meanwhile, the produced single multilingual model is applicable to all target&#10;languages, which saves the cost of training, inference, and maintenance for&#10;multiple models. Extensive experiments on two CLMRC benchmarks clearly show the&#10;effectiveness of our proposed method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.10450" label="2005.10450">
        <attvalues>
          <attvalue for="0" value="MTSS: Learn from Multiple Domain Teachers and Become a Multi-domain&#10;  Dialogue Expert" />
          <attvalue for="1" value="  How to build a high-quality multi-domain dialogue system is a challenging&#10;work due to its complicated and entangled dialogue state space among each&#10;domain, which seriously limits the quality of dialogue policy, and further&#10;affects the generated response. In this paper, we propose a novel method to&#10;acquire a satisfying policy and subtly circumvent the knotty dialogue state&#10;representation problem in the multi-domain setting. Inspired by real school&#10;teaching scenarios, our method is composed of multiple domain-specific teachers&#10;and a universal student. Each individual teacher only focuses on one specific&#10;domain and learns its corresponding domain knowledge and dialogue policy based&#10;on a precisely extracted single domain dialogue state representation. Then,&#10;these domain-specific teachers impart their domain knowledge and policies to a&#10;universal student model and collectively make this student model a multi-domain&#10;dialogue expert. Experiment results show that our method reaches competitive&#10;results with SOTAs in both multi-domain and single domain setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value=" &#10;" />
        </attvalues>
      </node>
      <node id="1909.00325" label="1909.00325">
        <attvalues>
          <attvalue for="0" value="Repurposing Decoder-Transformer Language Models for Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  Neural network models have shown excellent fluency and performance when&#10;applied to abstractive summarization. Many approaches to neural abstractive&#10;summarization involve the introduction of significant inductive bias,&#10;exemplified through the use of components such as pointer-generator&#10;architectures, coverage, and partially extractive procedures, designed to mimic&#10;the process by which humans summarize documents. We show that it is possible to&#10;attain competitive performance by instead directly viewing summarization as a&#10;language modeling problem and effectively leveraging transfer learning. We&#10;introduce a simple procedure built upon decoder-transformers to obtain highly&#10;competitive ROUGE scores for summarization performance using a language&#10;modeling loss alone, with no beam-search or other decoding-time optimization,&#10;and instead relying on efficient nucleus sampling and greedy decoding.&#10;" />
          <attvalue for="2" value="&#10;Text summarization aims to produce short, coherent natural language summaries of longer-form documents while retaining important information from the original source text. Techniques for this task fall on a point along a continuum between extractive and abstractive summarization. The former seeks to extract grammatically valid subsets of the source document such that, when combined, produce a coherent, shorter text. The latter, as the name suggests, aims to abstract away the direct lexical and syntactic choices of the source document, and generate summary text from scratch. &#10;&#10;Neural network approaches to abstractive summarization generally encode the source document into some hidden state or representation, then decode this representation into a summarized, abstracted version of the source document~\cite{rush2015neural,nallapati2016abstractive}. These approaches usually rely on a sequence-to-sequence~\cite{sutskever2014sequence} style architecture, and tend to produce fluent, well formed natural language summaries when coupled with beam search or other decoding techniques.&#10;&#10;A major weakness of traditional sequence-to-sequence learning when applied to summarization is the lack of a direct copy mechanism, leading to missing or misrepresented details in decoded summaries~\cite{chopra2016abstractive,nallapati2016abstractive,rush2015neural,zeng2016efficient}. Though attention helps ameliorate this issue by directly learning to focus on specific words or phrases in a source document~\cite{chopra2016abstractive}, many have allowed for an explicit copy mechanism inspired by Pointer Networks~\cite{vinyals2015pointer}, by optimizing a differentiable decision whether to generate new text or directly copy from the source~\cite{gu2016incorporating,zeng2016efficient,see2017get}. &#10;&#10;Additional components in many neural abstractive summarization systems model semantic coverage~\cite{tu2016modeling,see2017get} and provide guidance on where to attend~\cite{gehrmann2018bottom} in order to directly avoid repetition and ancillary details, while encouraging completeness.&#10;&#10;Recent work has incorporated the use of reinforcement learning to directly optimize objectives of interest that may not be differentiable, but are nonetheless useful for summarization, such as directly optimizing the ROUGE score~\cite{paulus2017deep,li2018actor,celikyilmaz2018deep}.&#10;&#10;Simultaneously, \cite{peters2018deep}, \cite{Devlin2018BERTPO}, \cite{howard2018universal}, \cite{radford2018improving}, and \cite{radford2019language}, among others, have shown the benefits of large-scale pretraining on large, unlabeled corpora on a variety of downstream tasks in transfer learning settings. In particular, it has been shown that large-scale, attention-only language modeling via decoder-only transformers~\cite{decoder-transformer} as an unsupervised pretraining task admits the ability to perform zero-shot learning on meaningful tasks involving natural language generation~\cite{radford2019language}.&#10;&#10;Motivated by this, we propose a simple method that exhibits competitive performance on abstractive summarization without using sequence-to-sequence architectures or other standard tools in the neural abstractive summarization toolbox, and instead using a decoder-only transformer language model with transfer learning. This further illustrates the utility of finetuning language models trained on open domain text. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Neural Network Models, Abstractive Summarization, Artificial Intelligence, Language Modeling" />
        </attvalues>
      </node>
      <node id="1611.03382" label="1611.03382">
        <attvalues>
          <attvalue for="0" value="Efficient Summarization with Read-Again and Copy Mechanism" />
          <attvalue for="1" value="  Encoder-decoder models have been widely used to solve sequence to sequence&#10;prediction tasks. However current approaches suffer from two shortcomings.&#10;First, the encoders compute a representation of each word taking into account&#10;only the history of the words it has read so far, yielding suboptimal&#10;representations. Second, current decoders utilize large vocabularies in order&#10;to minimize the problem of unknown words, resulting in slow decoding times. In&#10;this paper we address both shortcomings. Towards this goal, we first introduce&#10;a simple mechanism that first reads the input sequence before committing to a&#10;representation of each word. Furthermore, we propose a simple copy mechanism&#10;that is able to exploit very small vocabularies and handle out-of-vocabulary&#10;words. We demonstrate the effectiveness of our approach on the Gigaword dataset&#10;and DUC competition outperforming the state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1506.03134" label="1506.03134">
        <attvalues>
          <attvalue for="0" value="Pointer Networks" />
          <attvalue for="1" value="  We introduce a new neural architecture to learn the conditional probability&#10;of an output sequence with elements that are discrete tokens corresponding to&#10;positions in an input sequence. Such problems cannot be trivially addressed by&#10;existent approaches such as sequence-to-sequence and Neural Turing Machines,&#10;because the number of target classes in each step of the output depends on the&#10;length of the input, which is variable. Problems such as sorting variable sized&#10;sequences, and various combinatorial optimization problems belong to this&#10;class. Our model solves the problem of variable size output dictionaries using&#10;a recently proposed mechanism of neural attention. It differs from the previous&#10;attention attempts in that, instead of using attention to blend hidden units of&#10;an encoder to a context vector at each decoder step, it uses attention as a&#10;pointer to select a member of the input sequence as the output. We call this&#10;architecture a Pointer Net (Ptr-Net). We show Ptr-Nets can be used to learn&#10;approximate solutions to three challenging geometric problems -- finding planar&#10;convex hulls, computing Delaunay triangulations, and the planar Travelling&#10;Salesman Problem -- using training examples alone. Ptr-Nets not only improve&#10;over sequence-to-sequence with input attention, but also allow us to generalize&#10;to variable size output dictionaries. We show that the learnt models generalize&#10;beyond the maximum lengths they were trained on. We hope our results on these&#10;tasks will encourage a broader exploration of neural learning for discrete&#10;problems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.11070" label="1803.11070">
        <attvalues>
          <attvalue for="0" value="Actor-Critic based Training Framework for Abstractive Summarization" />
          <attvalue for="1" value="  We present a training framework for neural abstractive summarization based on&#10;actor-critic approaches from reinforcement learning. In the traditional neural&#10;network based methods, the objective is only to maximize the likelihood of the&#10;predicted summaries, no other assessment constraints are considered, which may&#10;generate low-quality summaries or even incorrect sentences. To alleviate this&#10;problem, we employ an actor-critic framework to enhance the training procedure.&#10;For the actor, we employ the typical attention based sequence-to-sequence&#10;(seq2seq) framework as the policy network for summary generation. For the&#10;critic, we combine the maximum likelihood estimator with a well designed global&#10;summary quality estimator which is a neural network based binary classifier&#10;aiming to make the generated summaries indistinguishable from the human-written&#10;ones. Policy gradient method is used to conduct the parameter learning. An&#10;alternating training strategy is proposed to conduct the joint training of the&#10;actor and critic models. Extensive experiments on some benchmark datasets in&#10;different languages show that our framework achieves improvements over the&#10;state-of-the-art methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.10357" label="1803.10357">
        <attvalues>
          <attvalue for="0" value="Deep Communicating Agents for Abstractive Summarization" />
          <attvalue for="1" value="  We present deep communicating agents in an encoder-decoder architecture to&#10;address the challenges of representing a long document for abstractive&#10;summarization. With deep communicating agents, the task of encoding a long text&#10;is divided across multiple collaborating agents, each in charge of a subsection&#10;of the input text. These encoders are connected to a single decoder, trained&#10;end-to-end using reinforcement learning to generate a focused and coherent&#10;summary. Empirical results demonstrate that multiple communicating encoders&#10;lead to a higher quality summary compared to several strong baselines,&#10;including those based on a single encoder or multiple non-communicating&#10;encoders.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.12944" label="2209.12944">
        <attvalues>
          <attvalue for="0" value="On the Impact of Speech Recognition Errors in Passage Retrieval for&#10;  Spoken Question Answering" />
          <attvalue for="1" value="  Interacting with a speech interface to query a Question Answering (QA) system&#10;is becoming increasingly popular. Typically, QA systems rely on passage&#10;retrieval to select candidate contexts and reading comprehension to extract the&#10;final answer. While there has been some attention to improving the reading&#10;comprehension part of QA systems against errors that automatic speech&#10;recognition (ASR) models introduce, the passage retrieval part remains&#10;unexplored. However, such errors can affect the performance of passage&#10;retrieval, leading to inferior end-to-end performance. To address this gap, we&#10;augment two existing large-scale passage ranking and open domain QA datasets&#10;with synthetic ASR noise and study the robustness of lexical and dense&#10;retrievers against questions with ASR noise. Furthermore, we study the&#10;generalizability of data augmentation techniques across different domains; with&#10;each domain being a different language dialect or accent. Finally, we create a&#10;new dataset with questions voiced by human users and use their transcriptions&#10;to show that the retrieval performance can further degrade when dealing with&#10;natural ASR noise instead of synthetic ASR noise.&#10;" />
          <attvalue for="2" value="&#10;Nowadays users interact with a wide range of commercial Question Answering (QA) systems via speech interfaces. Millions of users are voicing their questions on virtual voice assistants such as Amazon Alexa, Apple Siri, or Google Assistant through their smart devices. Such voice assistants do not only increase the convenience with which users can query them but can support users with visual and motor impairments for which the use of conventional text entry mechanisms (keyboard) is not applicable \cite{DBLP:conf/chi/PradhanMF18}. Despite the popularity of voice assistants among users globally and the advancements in spoken-language understanding \cite{DBLP:conf/emnlp/BastianelliVSR20, DBLP:journals/corr/abs-2204-08582}, there are surprisingly limited efforts in studying spoken QA and its limitations.&#10;&#10;The majority of research focuses on reading comprehension as a component of spoken QA \cite{DBLP:conf/interspeech/LeeWLL18, DBLP:conf/emnlp/FaisalKAA21, DBLP:conf/eacl/RavichanderDRMH21}. In detail, previous works studied the case where the provided question includes noise introduced by an automated speech recognition (ASR) system; audio is converted to text before reading comprehension is performed. &#10;Ravichander et al.~\cite{DBLP:conf/eacl/RavichanderDRMH21} showed that ASR noise not only dramatically affects the performance of transformer-based reading comprehension models but also that it is a more challenging type of noise compared to the noise generated from keyboard mistyping or faulty machine translation. &#10;Faisal et al.~\cite{DBLP:conf/emnlp/FaisalKAA21} showed that background differences in users, such as their accent, can affect the performance of reading comprehension models differently.&#10;&#10;Even though robustifying reading comprehension against ASR noise is essential for extracting the answer to a question, as a subsequent step of passage retrieval, it is bounded by the ability of the QA system to retrieve the relevant passages. A typical QA pipeline consists of an efficient retriever that reduces the search space from millions of passages to the top-k and a reader that extracts the answer. Dense passage retrieval has become a new paradigm&#10;to retrieve relevant passages, setting the state-of-the-art performance in several leaderboards. Inferior retrieval of the relevant passages will negatively affect the performance of the overall system.&#10;&#10;Typically, state-of-the-art dense retrieval models are evaluated on clean datasets with noise-free questions. However, questions posed to real-world QA systems are prone to errors. Therefore, these models will encounter noisy questions when deployed in real-world applications, affecting their performance. User-generated textual questions can include typos such as keyboard typos due to fast typing, misspellings and phonetic typing errors (for words with close pronunciation). Recent works showed that even state-of-the-art dense retrieval models are not robust against simple typos \cite{DBLP:journals/corr/abs-2204-00716, DBLP:conf/sigir/SidiropoulosK22,DBLP:conf/emnlp/ZhuangZ21}. Sidiropoulos and Kanoulas \cite{DBLP:conf/sigir/SidiropoulosK22} showcased the dense retrievers' lack of robustness to typos in the question and proposed a combination of data augmentation with a contrastive loss to robustify the model. Zhuang and Zuccon \cite{DBLP:journals/corr/abs-2204-00716} increased the robustness of dense retrievers against typos by replacing the extremely sensitive to typos WordPiece tokenizer with the Character-CNN module and further combined it with a knowledge distillation method. &#10;&#10;On the other end of the spectrum, spoken questions voiced by users are also vulnerable to errors due to the ASR systems that convert them to text. How the existence of ASR noise in questions affects retrieval models is not studied yet. In this work, we address the need for evaluating passage retrieval for spoken QA. To the best of our knowledge, this is the first work in this direction. &#10;&#10;Since there is no available dataset for passage retrieval where questions have ASR noise, we simulate ASR noise by automatically transcribing synthetically voiced questions. We then compare the robustness of lexical and dense retrievers by evaluating them against questions with and without ASR noise. Preliminary results showed that neither lexical nor dense models are effective against questions with ASR noise leading to a significant drop in retrieval performance. We find that using data augmentation with ASR noise to train a dense retriever is a promising approach for increasing robustness against ASR noise. However, the generation of such synthetic data is time-consuming and limited to the languages/accents supported by the text-to-speech system. To this extent, we explore if typo augmentation (faster and not bound to specific accents/languages) can alleviate these limitations. Our experimental results show that typo robust dense retrievers can increase robustness against ASR noise to some extent; however, ASR data augmentation remains a significantly more effective approach. Since users can have different local accents, we experiment with multiple accents of the same language and unveil that users' accents play an important role in retrieval performance. Finally, to study a real-world scenario with voice variation and non-native speakers voicing questions, we also build a new test set where the questions have natural ASR noise. This set consists of 700 questions voiced by human annotators.&#10;&#10;We aim to answer the following research questions:&#10;RQ1 What is the impact on the performance of lexical and dense retrievers when questions have ASR errors?&#10;RQ2 Are typo-robust dense retrieval approaches also robust against ASR noise? How competitive are they against dense retrieval trained via data augmentation with ASR noise?&#10;RQ3 Do certain accents affect the effectiveness of the retriever more than others? &#10;RQ4 Does natural ASR noise affect the robustness of dense retrievers more than synthetic ASR noise?&#10;&#10;Our main contributions can be summarized as follows: (i) we provide two large-scale datasets where questions have synthetic ASR noise to facilitate research (evaluation and training of new models) on passage retrieval for spoken QA, (ii) we create a new challenging test set that contains 700 questions with natural ASR noise, (iii) we show how lexical and dense retrievers are not robust against ASR noise and propose data augmentation for robustifying the latter, and (iv) we study how performance varies with respect to different accents and synthetic vs. natural spoken questions. &#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Speech Recognition, Information Science, Natural Language Processing, Speech Interface Errors, Question Answering Robustness, Passage Retrieval Systems" />
        </attvalues>
      </node>
      <node id="2102.08345" label="2102.08345">
        <attvalues>
          <attvalue for="0" value="NoiseQA: Challenge Set Evaluation for User-Centric Question Answering" />
          <attvalue for="1" value="  When Question-Answering (QA) systems are deployed in the real world, users&#10;query them through a variety of interfaces, such as speaking to voice&#10;assistants, typing questions into a search engine, or even translating&#10;questions to languages supported by the QA system. While there has been&#10;significant community attention devoted to identifying correct answers in&#10;passages assuming a perfectly formed question, we show that components in the&#10;pipeline that precede an answering engine can introduce varied and considerable&#10;sources of error, and performance can degrade substantially based on these&#10;upstream noise sources even for powerful pre-trained QA models. We conclude&#10;that there is substantial room for progress before QA systems can be&#10;effectively deployed, highlight the need for QA evaluation to expand to&#10;consider real-world use, and hope that our findings will spur greater community&#10;interest in the issues that arise when our systems actually need to be of&#10;utility to humans.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.12072" label="2109.12072">
        <attvalues>
          <attvalue for="0" value="SD-QA: Spoken Dialectal Question Answering for the Real World" />
          <attvalue for="1" value="  Question answering (QA) systems are now available through numerous commercial&#10;applications for a wide variety of domains, serving millions of users that&#10;interact with them via speech interfaces. However, current benchmarks in QA&#10;research do not account for the errors that speech recognition models might&#10;introduce, nor do they consider the language variations (dialects) of the&#10;users. To address this gap, we augment an existing QA dataset to construct a&#10;multi-dialect, spoken QA benchmark on five languages (Arabic, Bengali, English,&#10;Kiswahili, Korean) with more than 68k audio prompts in 24 dialects from 255&#10;speakers. We provide baseline results showcasing the real-world performance of&#10;QA systems and analyze the effect of language variety and other sensitive&#10;speaker attributes on downstream performance. Last, we study the fairness of&#10;the ASR and QA models with respect to the underlying user populations. The&#10;dataset, model outputs, and code for reproducing all our experiments are&#10;available: https://github.com/ffaisal93/SD-QA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.02303" label="2205.02303">
        <attvalues>
          <attvalue for="0" value="Analysing the Robustness of Dual Encoders for Dense Retrieval Against&#10;  Misspellings" />
          <attvalue for="1" value="  Dense retrieval is becoming one of the standard approaches for document and&#10;passage ranking. The dual-encoder architecture is widely adopted for scoring&#10;question-passage pairs due to its efficiency and high performance. Typically,&#10;dense retrieval models are evaluated on clean and curated datasets. However,&#10;when deployed in real-life applications, these models encounter noisy&#10;user-generated text. That said, the performance of state-of-the-art dense&#10;retrievers can substantially deteriorate when exposed to noisy text. In this&#10;work, we study the robustness of dense retrievers against typos in the user&#10;question. We observe a significant drop in the performance of the dual-encoder&#10;model when encountering typos and explore ways to improve its robustness by&#10;combining data augmentation with contrastive learning. Our experiments on two&#10;large-scale passage ranking and open-domain question answering datasets show&#10;that our proposed approach outperforms competing approaches. Additionally, we&#10;perform a thorough analysis on robustness. Finally, we provide insights on how&#10;different typos affect the robustness of embeddings differently and how our&#10;method alleviates the effect of some typos but not of others.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.02363" label="2305.02363">
        <attvalues>
          <attvalue for="0" value="Entity Tracking in Language Models" />
          <attvalue for="1" value="  Keeping track of how states of entities change as a text or dialog unfolds is&#10;a key prerequisite to discourse understanding. Yet, there have been few&#10;systematic investigations into the ability of large language models (LLMs) to&#10;track discourse entities. In this work, we present a task probing to what&#10;extent a language model can infer the final state of an entity given an English&#10;description of the initial state and a series of state-changing operations. We&#10;use this task to first investigate whether Flan-T5, GPT-3 and GPT-3.5 can track&#10;the state of entities, and find that only GPT-3.5 models, which have been&#10;pretrained on large amounts of code, exhibit this ability. We then investigate&#10;whether smaller models pretrained primarily on text can learn to track&#10;entities, through finetuning T5 on several training/evaluation splits. While&#10;performance degrades for more complex splits, we find that even when evaluated&#10;on a different set of entities from training or longer operation sequences, a&#10;finetuned model can perform non-trivial entity tracking. Taken together, these&#10;results suggest that language models can learn to track entities but&#10;pretraining on text corpora alone does not make this capacity surface.&#10;" />
          <attvalue for="2" value="&#10;A key prerequisite to long-context understanding and generating coherent text is the ability to accurately represent entities as the discourse unfolds \cite[][i.a.]{karttunen1976discourse,groenendijk1991dynamic,heim2002file,nieuwland2006peanuts,kamp2011discourse}. For example, consider the following example in the context of a recipe:&#10;&#10;\ex. Put the eggs, sugar, flour, and baking powder in a bowl and mix to form a light batter. Make sure that the final batter does not contain any lumps of flour or sugar.&#10;&#10;In order to understand this instruction, several distinct abilities are necessary:&#10;&#10;New discourse entity recognition: recognizing when new discourse entities are introduced. E.g., a bowl introduces a new discourse entity but the final batter or any lumps of... does not.&#10;&#10;Coreference resolution: associating referring expressions with discourse entities. E.g., a light batter and the final batter refer to the same entity.&#10;&#10;Discourse entity tracking: tracking the state changes made to each discourse entity. E.g., the eggs are put into the bowl and mixed with the other ingredients.&#10;&#10;There exist many datasets that aim to evaluate these abilities \cite[e.g.,][]{walker2006ace,pradhan-etal-2012-conll,rahman-ng-2012-resolving,weston2015towards,chen-etal-2018-preco,bamman-etal-2020-annotated,uryupina2020annotating} and many NLP models that aim to solve these tasks \cite[e.g.,][]{haghighi-klein-2010-coreference,lee-etal-2011-stanfords,hill2016goldilocks,henaff2017tracking,ji-etal-2017-dynamic,lee-etal-2017-end,bosselut2018simulating,gupta-durrett-2019-effective,gupta-durrett-2019-tracking,aina-etal-2019-entity,toshniwal-etal-2020-learning,wu-etal-2020-corefqa}. In the context of large language models (LLMs), \newcite{tenney-etal-2019-bert}, \newcite{clark-etal-2019-bert}, and \newcite{sorodoc-etal-2020-probing} found that representations of LSTMs and Transformer-based models such as BERT \cite{devlin-etal-2019-bert} do capture coreference relations. \newcite{loaiciga-etal-2022-new} and \newcite{schuster-linzen-2022-sentence} found that pretrained models are able to detect whether noun phrases introduce discourse entities, albeit not fully systematically.&#10;&#10;The question of whether LLMs can track the state of discourse entities, however, has mostly been indirectly evaluated. \newcite{toshniwal-etal-2022-chess} showed that GPT-2 \cite{radford2019language} can learn to predict valid chess moves based on a compact, nonlinguistic description of previous moves. Similarly, \newcite{li2023emergent} showed that a GPT model trained on Othello can predict valid next moves, and that these predictions are tied to the model's internal representations of the board states. Still, these results do not tell us whether LLMs track state changes expressed in natural language discourses. The most relevant evaluation is \newcite{li-etal-2021-implicit}, where they tested whether model representations encode entity states described in naturalistic text. Using a probing classifier, they found that the states can be decoded from T5 \cite{raffel2020exploring} and BART \cite{lewis-etal-2020-bart} with high accuracy. However, as we show in a reanalysis of their results (Section~\ref{sec:motivation}), they do not provide definitive evidence for entity tracking. Hence, whether LLMs can track entities during the processing of natural language discourse remains an open question.&#10;&#10;Contributions This work attempts to answer this question by developing a task targeted towards evaluating a language model's ability to track state changes of discourse entities (illustrated in Figure~\ref{fig:task}). We use this novel task to evaluate GPT-3 \cite{brown2020language}, \href{&#10;and Flan-T5 \cite{chung2022scaling} without any finetuning. We find that only models in the GPT 3.5 series, which have been trained on both text and code, are able to perform non-trivial entity tracking. We then show that a smaller language model (T5) can learn to perform non-trivial entity tracking and also demonstrates some capacity to generalize to state descriptions with more operations or with low lexical overlap. Our results suggest that language models can learn to track entities but pretraining on text corpora alone does not make this capacity surface. More broadly, our task can also serve as a useful tool for investigations into emergent world models in LMs \cite[e.g.,][]{li2023emergent,tsai-etal-2023-large}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Entity Tracking, Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Discourse Understanding" />
        </attvalues>
      </node>
      <node id="1912.01140" label="1912.01140">
        <attvalues>
          <attvalue for="0" value="An Annotated Dataset of Coreference in English Literature" />
          <attvalue for="1" value="  We present in this work a new dataset of coreference annotations for works of&#10;literature in English, covering 29,103 mentions in 210,532 tokens from 100&#10;works of fiction. This dataset differs from previous coreference datasets in&#10;containing documents whose average length (2,105.3 words) is four times longer&#10;than other benchmark datasets (463.7 for OntoNotes), and contains examples of&#10;difficult coreference problems common in literature. This dataset allows for an&#10;evaluation of cross-domain performance for the task of coreference resolution,&#10;and analysis into the characteristics of long-distance within-document&#10;coreference.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1511.02301" label="1511.02301">
        <attvalues>
          <attvalue for="0" value="The Goldilocks Principle: Reading Children's Books with Explicit Memory&#10;  Representations" />
          <attvalue for="1" value="  We introduce a new test of how well language models capture meaning in&#10;children's books. Unlike standard language modelling benchmarks, it&#10;distinguishes the task of predicting syntactic function words from that of&#10;predicting lower-frequency words, which carry greater semantic content. We&#10;compare a range of state-of-the-art models, each with a different way of&#10;encoding what has been previously read. We show that models which store&#10;explicit representations of long-term contexts outperform state-of-the-art&#10;neural language models at predicting semantic content words, although this&#10;advantage is not observed for syntactic function words. Interestingly, we find&#10;that the amount of text encoded in a single memory representation is highly&#10;influential to the performance: there is a sweet-spot, not too big and not too&#10;small, between single words and full sentences that allows the most meaningful&#10;information in a text to be effectively retained and recalled. Further, the&#10;attention over such window-based memories can be trained effectively through&#10;self-supervision. We then assess the generality of this principle by applying&#10;it to the CNN QA benchmark, which involves identifying named entities in&#10;paraphrased summaries of news articles, and achieve state-of-the-art&#10;performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.02635" label="1909.02635">
        <attvalues>
          <attvalue for="0" value="Effective Use of Transformer Networks for Entity Tracking" />
          <attvalue for="1" value="  Tracking entities in procedural language requires understanding the&#10;transformations arising from actions on entities as well as those entities'&#10;interactions. While self-attention-based pre-trained language encoders like GPT&#10;and BERT have been successfully applied across a range of natural language&#10;understanding tasks, their ability to handle the nuances of procedural texts is&#10;still untested. In this paper, we explore the use of pre-trained transformer&#10;networks for entity tracking tasks in procedural text. First, we test standard&#10;lightweight approaches for prediction with pre-trained transformers, and find&#10;that these approaches underperform even simple baselines. We show that much&#10;stronger results can be attained by restructuring the input to guide the&#10;transformer model to focus on a particular entity. Second, we assess the degree&#10;to which transformer networks capture the process dynamics, investigating such&#10;factors as merged entities and oblique entity references. On two different&#10;tasks, ingredient detection in recipes and QA over scientific processes, we&#10;achieve state-of-the-art results, but our models still largely attend to&#10;shallow context clues and do not form complex representations of intermediate&#10;entity or process state.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.03518" label="1904.03518">
        <attvalues>
          <attvalue for="0" value="Tracking Discrete and Continuous Entity State for Process Understanding" />
          <attvalue for="1" value="  Procedural text, which describes entities and their interactions as they&#10;undergo some process, depicts entities in a uniquely nuanced way. First, each&#10;entity may have some observable discrete attributes, such as its state or&#10;location; modeling these involves imposing global structure and enforcing&#10;consistency. Second, an entity may have properties which are not made explicit&#10;but can be effectively induced and tracked by neural networks. In this paper,&#10;we propose a structured neural architecture that reflects this dual nature of&#10;entity evolution. The model tracks each entity recurrently, updating its hidden&#10;continuous representation at each step to contain relevant state information.&#10;The global discrete state structure is explicitly modeled with a neural CRF&#10;over the changing hidden representation of the entity. This CRF can explicitly&#10;capture constraints on entity states over time, enforcing that, for example, an&#10;entity cannot move to a location after it is destroyed. We evaluate the&#10;performance of our proposed model on QA tasks over process paragraphs in the&#10;ProPara dataset and find that our model achieves state-of-the-art results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.06649" label="1905.06649">
        <attvalues>
          <attvalue for="0" value="What do Entity-Centric Models Learn? Insights from Entity Linking in&#10;  Multi-Party Dialogue" />
          <attvalue for="1" value="  Humans use language to refer to entities in the external world. Motivated by&#10;this, in recent years several models that incorporate a bias towards learning&#10;entity representations have been proposed. Such entity-centric models have&#10;shown empirical success, but we still know little about why. In this paper we&#10;analyze the behavior of two recently proposed entity-centric models in a&#10;referential task, Entity Linking in Multi-party Dialogue (SemEval 2018 Task 4).&#10;We show that these models outperform the state of the art on this task, and&#10;that they do better on lower frequency entities than a counterpart model that&#10;is not entity-centric, with the same model size. We argue that making models&#10;entity-centric naturally fosters good architectural decisions. However, we also&#10;show that these models do not really build entity representations and that they&#10;make poor use of linguistic context. These negative results underscore the need&#10;for model analysis, to test whether the motivations for particular&#10;architectures are borne out in how models behave when deployed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.13382" label="2210.13382">
        <attvalues>
          <attvalue for="0" value="Emergent World Representations: Exploring a Sequence Model Trained on a&#10;  Synthetic Task" />
          <attvalue for="1" value="  Language models show a surprising range of capabilities, but the source of&#10;their apparent competence is unclear. Do these networks just memorize a&#10;collection of surface statistics, or do they rely on internal representations&#10;of the process that generates the sequences they see? We investigate this&#10;question by applying a variant of the GPT model to the task of predicting legal&#10;moves in a simple board game, Othello. Although the network has no a priori&#10;knowledge of the game or its rules, we uncover evidence of an emergent&#10;nonlinear internal representation of the board state. Interventional&#10;experiments indicate this representation can be used to control the output of&#10;the network and create &quot;latent saliency maps&quot; that can help explain predictions&#10;in human terms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.02868" label="2304.02868">
        <attvalues>
          <attvalue for="0" value="Can Large Language Models Play Text Games Well? Current State-of-the-Art&#10;  and Open Questions" />
          <attvalue for="1" value="  Large language models (LLMs) such as ChatGPT and GPT-4 have recently&#10;demonstrated their remarkable abilities of communicating with human users. In&#10;this technical report, we take an initiative to investigate their capacities of&#10;playing text games, in which a player has to understand the environment and&#10;respond to situations by having dialogues with the game world. Our experiments&#10;show that ChatGPT performs competitively compared to all the existing systems&#10;but still exhibits a low level of intelligence. Precisely, ChatGPT can not&#10;construct the world model by playing the game or even reading the game manual;&#10;it may fail to leverage the world knowledge that it already has; it cannot&#10;infer the goal of each step as the game progresses. Our results open up new&#10;research questions at the intersection of artificial intelligence, machine&#10;learning, and natural language processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.09954" label="2402.09954">
        <attvalues>
          <attvalue for="0" value="Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of&#10;  In-Context Learning for Persona-based Dialogue Generation" />
          <attvalue for="1" value="  Previous in-context learning (ICL) research has focused on tasks such as&#10;classification, machine translation, text2table, etc., while studies on whether&#10;ICL can improve human-like dialogue generation are scarce. Our work fills this&#10;gap by systematically investigating the ICL capabilities of large language&#10;models (LLMs) in persona-based dialogue generation, conducting extensive&#10;experiments on high-quality real human Chinese dialogue datasets. From&#10;experimental results, we draw three conclusions: 1) adjusting prompt&#10;instructions is the most direct, effective, and economical way to improve&#10;generation quality; 2) randomly retrieving demonstrations (demos) achieves the&#10;best results, possibly due to the greater diversity and the amount of effective&#10;information; counter-intuitively, retrieving demos with a context identical to&#10;the query performs the worst; 3) even when we destroy the multi-turn&#10;associations and single-turn semantics in the demos, increasing the number of&#10;demos still improves dialogue performance, proving that LLMs can learn from&#10;corrupted dialogue demos. Previous explanations of the ICL mechanism, such as&#10;$n$-gram induction head, cannot fully account for this phenomenon.&#10;" />
          <attvalue for="2" value="&#10;Current chatbots based on LLMs have superior performance on question answering, polishing documents, etc.~\cite{srivastava2022beyond}. However, for the task of persona-based dialogue generation, these universal LLMs still lag far behind real humans. Persona-based dialogue generation is crucial and has practical application value. For instance, optimizing persona dialogue generation is of great significance for empathetic and medical chatbots~\cite{de2020effectiveness,liu2022persona}, and it can bring trust to users~\cite{huang2023personalized,de2020effectiveness}.&#10;&#10;On the other hand, fine-tuning a high-quality persona-based dialogue model usually requires a proprietary dataset, and the cost of manually writing dialogues is very high~\cite{cao2022model,huang2023personalized}. Since the valid context length of LLMs is constantly growing~\cite{xiong2023effective}, we believe that ICL may be an effective way to generate high-quality dialogues at a low cost. However, research papers on how ICL affects dialogue generation are few~\cite{dong2022survey}. Only one paper~\cite{xu2023towards} investigates the ability of LLM to learn from dialogue demos, but these demos only contain character background information, which is quite different from real human conversations. In this paper, we experiment with the ICL capability of LLMs on dialogues of real-human conversations, systematically exploring the capabilities of ICL in persona-based dialogue generation, including how we should choose the demo retrieval method, whether we should focus more on the text quality of the demo itself or the input-output mapping, whether providing more demos can improve the results, and give insight on what the LLM learns and does not learn from the provided demos.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Machine Learning, Linguistics, Cognitive Science, Language Model Training, Dialogue Generation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2212.03699" label="2212.03699">
        <attvalues>
          <attvalue for="0" value="Persona-Based Conversational AI: State of the Art and Challenges" />
          <attvalue for="1" value="  Conversational AI has become an increasingly prominent and practical&#10;application of machine learning. However, existing conversational AI techniques&#10;still suffer from various limitations. One such limitation is a lack of&#10;well-developed methods for incorporating auxiliary information that could help&#10;a model understand conversational context better. In this paper, we explore how&#10;persona-based information could help improve the quality of response generation&#10;in conversations. First, we provide a literature review focusing on the current&#10;state-of-the-art methods that utilize persona information. We evaluate two&#10;strong baseline methods, the Ranking Profile Memory Network and the&#10;Poly-Encoder, on the NeurIPS ConvAI2 benchmark dataset. Our analysis elucidates&#10;the importance of incorporating persona information into conversational&#10;systems. Additionally, our study highlights several limitations with current&#10;state-of-the-art methods and outlines challenges and future research directions&#10;for advancing personalized conversational AI technology.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.15088" label="2210.15088">
        <attvalues>
          <attvalue for="0" value="Personalized Dialogue Generation with Persona-Adaptive Attention" />
          <attvalue for="1" value="  Persona-based dialogue systems aim to generate consistent responses based on&#10;historical context and predefined persona. Unlike conventional dialogue&#10;generation, the persona-based dialogue needs to consider both dialogue context&#10;and persona, posing a challenge for coherent training. Specifically, this&#10;requires a delicate weight balance between context and persona. To achieve&#10;that, in this paper, we propose an effective framework with Persona-Adaptive&#10;Attention (PAA), which adaptively integrates the weights from the persona and&#10;context information via our designed attention. In addition, a dynamic masking&#10;mechanism is applied to the PAA to not only drop redundant information in&#10;context and persona but also serve as a regularization mechanism to avoid&#10;overfitting. Experimental results demonstrate the superiority of the proposed&#10;PAA framework compared to the strong baselines in both automatic and human&#10;evaluation. Moreover, the proposed PAA approach can perform equivalently well&#10;in a low-resource regime compared to models trained in a full-data setting,&#10;which achieve a similar result with only 20% to 30% of data compared to the&#10;larger models trained in the full-data setting. To fully exploit the&#10;effectiveness of our design, we designed several variants for handling the&#10;weighted information in different ways, showing the necessity and sufficiency&#10;of our weighting and masking designs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.09867" label="2204.09867">
        <attvalues>
          <attvalue for="0" value="A Model-Agnostic Data Manipulation Method for Persona-based Dialogue&#10;  Generation" />
          <attvalue for="1" value="  Towards building intelligent dialogue agents, there has been a growing&#10;interest in introducing explicit personas in generation models. However, with&#10;limited persona-based dialogue data at hand, it may be difficult to train a&#10;dialogue generation model well. We point out that the data challenges of this&#10;generation task lie in two aspects: first, it is expensive to scale up current&#10;persona-based dialogue datasets; second, each data sample in this task is more&#10;complex to learn with than conventional dialogue data. To alleviate the above&#10;data issues, we propose a data manipulation method, which is model-agnostic to&#10;be packed with any persona-based dialogue generation model to improve its&#10;performance. The original training samples will first be distilled and thus&#10;expected to be fitted more easily. Next, we show various effective ways that&#10;can diversify such easier distilled data. A given base model will then be&#10;trained via the constructed data curricula, i.e. first on augmented distilled&#10;samples and then on original ones. Experiments illustrate the superiority of&#10;our method with two strong base dialogue models (Transformer encoder-decoder&#10;and GPT2).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.08981" label="2102.08981">
        <attvalues>
          <attvalue for="0" value="Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize&#10;  Long-Tail Visual Concepts" />
          <attvalue for="1" value="  The availability of large-scale image captioning and visual question&#10;answering datasets has contributed significantly to recent successes in&#10;vision-and-language pre-training. However, these datasets are often collected&#10;with overrestrictive requirements inherited from their original target tasks&#10;(e.g., image caption generation), which limit the resulting dataset scale and&#10;diversity. We take a step further in pushing the limits of vision-and-language&#10;pre-training data by relaxing the data collection pipeline used in Conceptual&#10;Captions 3M (CC3M) [Sharma et al. 2018] and introduce the Conceptual 12M&#10;(CC12M), a dataset with 12 million image-text pairs specifically meant to be&#10;used for vision-and-language pre-training. We perform an analysis of this&#10;dataset and benchmark its effectiveness against CC3M on multiple downstream&#10;tasks with an emphasis on long-tail visual recognition. Our results clearly&#10;illustrate the benefit of scaling up pre-training data for vision-and-language&#10;tasks, as indicated by the new state-of-the-art results on both the nocaps and&#10;Conceptual Captions benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.11740" label="1909.11740">
        <attvalues>
          <attvalue for="0" value="UNITER: UNiversal Image-TExt Representation Learning" />
          <attvalue for="1" value="  Joint image-text embedding is the bedrock for most Vision-and-Language (V+L)&#10;tasks, where multimodality inputs are simultaneously processed for joint visual&#10;and textual understanding. In this paper, we introduce UNITER, a UNiversal&#10;Image-TExt Representation, learned through large-scale pre-training over four&#10;image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU&#10;Captions), which can power heterogeneous downstream V+L tasks with joint&#10;multimodal embeddings. We design four pre-training tasks: Masked Language&#10;Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text&#10;Matching (ITM), and Word-Region Alignment (WRA). Different from previous work&#10;that applies joint random masking to both modalities, we use conditional&#10;masking on pre-training tasks (i.e., masked language/region modeling is&#10;conditioned on full observation of image/text). In addition to ITM for global&#10;image-text alignment, we also propose WRA via the use of Optimal Transport (OT)&#10;to explicitly encourage fine-grained alignment between words and image regions&#10;during pre-training. Comprehensive analysis shows that both conditional masking&#10;and OT-based WRA contribute to better pre-training. We also conduct a thorough&#10;ablation study to find an optimal combination of pre-training tasks. Extensive&#10;experiments show that UNITER achieves new state of the art across six V+L tasks&#10;(over nine datasets), including Visual Question Answering, Image-Text&#10;Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning,&#10;Visual Entailment, and NLVR$^2$. Code is available at&#10;https://github.com/ChenRocks/UNITER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.05657" label="2310.05657">
        <attvalues>
          <attvalue for="0" value="A Closer Look into Automatic Evaluation Using Large Language Models" />
          <attvalue for="1" value="  Using large language models (LLMs) to evaluate text quality has recently&#10;gained popularity. Some prior works explore the idea of using LLMs for&#10;evaluation, while they differ in some details of the evaluation process. In&#10;this paper, we analyze LLM evaluation (Chiang and Lee, 2023) and G-Eval (Liu et&#10;al., 2023), and we discuss how those details in the evaluation process change&#10;how well the ratings given by LLMs correlate with human ratings. We find that&#10;the auto Chain-of-Thought (CoT) used in G-Eval does not always make G-Eval more&#10;aligned with human ratings. We also show that forcing the LLM to output only a&#10;numeric rating, as in G-Eval, is suboptimal. Last, we reveal that asking the&#10;LLM to explain its own ratings consistently improves the correlation between&#10;the ChatGPT and human ratings and pushes state-of-the-art (SoTA) correlations&#10;on two meta-evaluation datasets.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) trained with task instructions and human feedback can follow natural language instructions to complete a task~\cite{askell2021general,sanh2022multitask,wei2022finetuned,ouyang2022training}.&#10;Recently, the instruction-following ability of LLMs makes them promising candidates for automatic evaluation~\cite{chiang-lee-2023-large,liu2023gpteval, wang2023chatgpt,huang2023chatgpt}.&#10;By simply instructing the LLMs on how to rate and giving the LLMs the sample to be rated, the LLM can follow the instructions and provide a rating of the sample.&#10;&#10;~\cite{chiang-lee-2023-large} propose LLM evaluation and~\cite{liu2023gpteval} propose G-Eval; both of which use LLMs to evaluate samples by giving the LLM instructions, and they both show that some LLMs can yield evaluation results that are aligned to the evaluation results of humans.&#10;Still, LLM evaluation and G-Eval differ in some specific design choices in the evaluation procedure.&#10;Since ~\cite{chiang-lee-2023-large} and~\cite{liu2023gpteval} use distinct tasks, it is hard to know how the differences between LLM evaluation and G-Eval affect the evaluation results.&#10;This makes practitioners in the future hard to determine how to conduct an automatic evaluation using LLMs.&#10;&#10;Given that LLM evaluation and G-Eval have already received significant attention shortly after publication, these methods will likely revolutionize the evaluation in NLP.&#10;Therefore, conducting a detailed analysis of these approaches is essential and timely. &#10;This paper aims to identify the crucial components in LLM evaluation and G-Eval that contribute to stronger correlations with human ratings.&#10;Based on our analysis, we provide guidelines on how to use LLMs for automatic evaluations.&#10;We have the following findings:&#10;\begin{itemize}&#10; \item Auto-CoT (proposed by G-Eval) does not always improve the correlation between LLM and human ratings. \item Making the LLMs output only a single numeric rating is suboptimal.&#10; \item Asking the LLMs to rationalize their own ratings significantly improves the correlation between the LLMs' ratings and human ratings.&#10; \item On two datasets, we improve the best correlation that ChatGPT's rating can achieve, and some correlations even exceed prior SoTA correlations obtained using the ratings of GPT-4 in~\cite{liu2023gpteval}. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Human-AI Correlation Analysis, Natural Language Processing, Text Quality Assessment" />
        </attvalues>
      </node>
      <node id="2302.07736" label="2302.07736">
        <attvalues>
          <attvalue for="0" value="Is ChatGPT better than Human Annotators? Potential and Limitations of&#10;  ChatGPT in Explaining Implicit Hate Speech" />
          <attvalue for="1" value="  Recent studies have alarmed that many online hate speeches are implicit. With&#10;its subtle nature, the explainability of the detection of such hateful speech&#10;has been a challenging problem. In this work, we examine whether ChatGPT can be&#10;used for providing natural language explanations (NLEs) for implicit hateful&#10;speech detection. We design our prompt to elicit concise ChatGPT-generated NLEs&#10;and conduct user studies to evaluate their qualities by comparison with&#10;human-written NLEs. We discuss the potential and limitations of ChatGPT in the&#10;context of implicit hateful speech research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.04009" label="2306.04009">
        <attvalues>
          <attvalue for="0" value="Triggering Multi-Hop Reasoning for Question Answering in Language Models&#10;  using Soft Prompts and Random Walks" />
          <attvalue for="1" value="  Despite readily memorizing world knowledge about entities, pre-trained&#10;language models (LMs) struggle to compose together two or more facts to perform&#10;multi-hop reasoning in question-answering tasks. In this work, we propose&#10;techniques that improve upon this limitation by relying on random walks over&#10;structured knowledge graphs. Specifically, we use soft prompts to guide LMs to&#10;chain together their encoded knowledge by learning to map multi-hop questions&#10;to random walk paths that lead to the answer. Applying our methods on two T5&#10;LMs shows substantial improvements over standard tuning approaches in answering&#10;questions that require 2-hop reasoning.&#10;" />
          <attvalue for="2" value="&#10;Performing multi-hop reasoning to answer questions such as Where was David Beckham’s daughter born? requires two fundamental capacities:&#10;C1: possessing pre-requisite knowledge (David Beckham’s daughter is Harper Beckham, Harper Beckham was born in Los Angeles), and C2: ability to compose internalized knowledge.&#10;Contemporary pre-trained language models (LMs) such as BERT \cite{devlin-etal-2019-bert} and T5 \cite{raffel2020exploring} have been shown to be adept at encoding factual knowledge \cite{petroni-etal-2019-language, zhong-etal-2021-factual, roberts-etal-2020-much}, an ability that can be further boosted by explicitly integrating them with knowledge about entities and relations \cite[i.a.]{bosselut-etal-2019-comet, sun-etal-2020-colake, wang-etal-2021-kepler}.&#10;At the same time, these LMs often struggle to compose the knowledge they encode \cite{kassner-etal-2020-pretrained, talmor-etal-2020-olmpics, moiseev-etal-2022-skill}, and therefore do not satisfy C2.&#10;To overcome this limitation, previous works have proposed methods that decompose multi-hop questions into single hop sub-questions that models can more easily answer \cite[i.a.]{min-etal-2019-multi, perez-etal-2020-unsupervised}. However, such methods require training entirely separate models, or make use of human-annotations \cite{patel2022question}. Furthermore, they focus on tasks where models explicitly receive additional text containing relevant facts, which makes it unclear if they can truly compose the knowledge that they have internalized.&#10;&#10;In this work, we aim to improve the standalone, self-contained ability of LMs to perform multi-hop reasoning. We posit that random walks---paths between entity nodes sampled from structured knowledge graphs---can provide a useful training signal for LMs to compose entity knowledge. To test this, we perform a case-study on two T5 models \cite[large and xxl,][]{raffel2020exploring}. &#10;Specifically, we first integrate within the LMs the single-hop knowledge that is required to answer multi-hop questions (effectively guaranteeing C1 is met). &#10;We show that this alone is not enough to demonstrate substantial improvements on questions requiring 2-hop reasoning.&#10;{We then adapt the knowledge integrated T5 models by training soft prompts \cite{qin-eisner-2021-learning, lester-etal-2021-power} on random walks over the structured knowledge that they have encoded, and devise two methods that trigger this ability in the LMs given a multi-hop question as input. &#10;The first method, \modular (\pth), uses two specialized soft prompts: one to parse entities and relations from the question, and another to generate a path to the answer, resembling the outputs of a random walk. The second method, \mixture, trains a single prompt on a mixture that combines the QA task with the random walk training, so as to allow the model to implicitly learn \pth's task. Both these soft prompt methods use the same underlying LM (kept frozen), and guide it to compose its internalized entity knowledge.}&#10;&#10;Our experiments suggest that integrating random walks in the \tfive models using our proposed techniques can substantially improve their ability to answer entity-centric 2-hop questions \cite{ho-etal-2020-constructing} at larger model sizes.&#10;Briefly, on \txxl our methods show improvements over previously proposed prompt-tuning approaches \cite{lester-etal-2021-power, vu-etal-2022-spot} as well as full model fine-tuning, with \pth and \mixture demonstrating gains of $\sim$16 and $\sim$9.6 points in exact match scores over fine-tuning the entire model, respectively.&#10;In the case of \tl, our methods demonstrate improvements over standard prompt-tuning methods, but fall short of the performance achieved using fine-tuning, suggesting that larger models---with up to 11B parameters---are more conducive to leveraging the training signal provided by random walks via soft prompts.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Limitations, Knowledge Graph Reasoning, Artificial Intelligence, Natural Language Processing, Multi-Hop Question Answering" />
        </attvalues>
      </node>
      <node id="2006.10413" label="2006.10413">
        <attvalues>
          <attvalue for="0" value="Are Pretrained Language Models Symbolic Reasoners Over Knowledge?" />
          <attvalue for="1" value="  How can pretrained language models (PLMs) learn factual knowledge from the&#10;training set? We investigate the two most important mechanisms: reasoning and&#10;memorization. Prior work has attempted to quantify the number of facts PLMs&#10;learn, but we present, using synthetic data, the first study that investigates&#10;the causal relation between facts present in training and facts learned by the&#10;PLM. For reasoning, we show that PLMs seem to learn to apply some symbolic&#10;reasoning rules correctly but struggle with others, including two-hop&#10;reasoning. Further analysis suggests that even the application of learned&#10;reasoning rules is flawed. For memorization, we identify schema conformity&#10;(facts systematically supported by other facts) and frequency as key factors&#10;for its success.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.12538" label="2205.12538">
        <attvalues>
          <attvalue for="0" value="Is a Question Decomposition Unit All We Need?" />
          <attvalue for="1" value="  Large Language Models (LMs) have achieved state-of-the-art performance on&#10;many Natural Language Processing (NLP) benchmarks. With the growing number of&#10;new benchmarks, we build bigger and more complex LMs. However, building new LMs&#10;may not be an ideal option owing to the cost, time and environmental impact&#10;associated with it. We explore an alternative route: can we modify data by&#10;expressing it in terms of the model's strengths, so that a question becomes&#10;easier for models to answer? We investigate if humans can decompose a hard&#10;question into a set of simpler questions that are relatively easier for models&#10;to solve. We analyze a range of datasets involving various forms of reasoning&#10;and find that it is indeed possible to significantly improve model performance&#10;(24% for GPT3 and 29% for RoBERTa-SQuAD along with a symbolic calculator) via&#10;decomposition. Our approach provides a viable option to involve people in NLP&#10;research in a meaningful way. Our findings indicate that Human-in-the-loop&#10;Question Decomposition (HQD) can potentially provide an alternate path to&#10;building large LMs. Code and data is available at&#10;https://github.com/Pruthvi98/QuestionDecomposition&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.01453" label="2404.01453">
        <attvalues>
          <attvalue for="0" value="Unveiling Divergent Inductive Biases of LLMs on Temporal Data" />
          <attvalue for="1" value="  Unraveling the intricate details of events in natural language necessitates a&#10;subtle understanding of temporal dynamics. Despite the adeptness of Large&#10;Language Models (LLMs) in discerning patterns and relationships from data,&#10;their inherent comprehension of temporal dynamics remains a formidable&#10;challenge. This research meticulously explores these intrinsic challenges&#10;within LLMs, with a specific emphasis on evaluating the performance of GPT-3.5&#10;and GPT-4 models in the analysis of temporal data. Employing two distinct&#10;prompt types, namely Question Answering (QA) format and Textual Entailment (TE)&#10;format, our analysis probes into both implicit and explicit events. The&#10;findings underscore noteworthy trends, revealing disparities in the performance&#10;of GPT-3.5 and GPT-4. Notably, biases toward specific temporal relationships&#10;come to light, with GPT-3.5 demonstrating a preference for &quot;AFTER'' in the QA&#10;format for both implicit and explicit events, while GPT-4 leans towards&#10;&quot;BEFORE''. Furthermore, a consistent pattern surfaces wherein GPT-3.5 tends&#10;towards &quot;TRUE'', and GPT-4 exhibits a preference for &quot;FALSE'' in the TE format&#10;for both implicit and explicit events. This persistent discrepancy between&#10;GPT-3.5 and GPT-4 in handling temporal data highlights the intricate nature of&#10;inductive bias in LLMs, suggesting that the evolution of these models may not&#10;merely mitigate bias but may introduce new layers of complexity.&#10;" />
          <attvalue for="2" value="&#10;&#10;Temporal relations play a crucial role across diverse applications, including event summarization \cite{10.1007/s11280-017-0501-x, keith2023survey}, predicting future events \cite{future-event}, and medical information processing \cite{jung-etal-2011-building, alfattni2020extraction}. Despite their importance, LLMs, especially those with limited context windows, face challenges in accurately sequencing events due to intricate temporal dependencies. Efforts have been devoted to developing methodologies for effective temporal relation extraction \cite{choubey-huang-2017-sequential, ning2018cogcomptime, ning2019improved, wang-etal-2020-joint, zhang-etal-2022-extracting}, along with initiatives to create benchmark datasets with a temporal focus \cite{TBcorpus, verhagen-etal-2010-Tempeval2, ning2018multi, zhou2021temporal, gantt2022decomposing}. However, discerning causal relationships between events adds complexity and can lead to misunderstandings. This complexity is strengthened by the absence of explicit temporal reasoning mechanisms, introducing biases in models' predictions and preferences for specific temporal relations.&#10;&#10;Surprisingly, a notable gap exists in research exploring inductive bias in LLMs when discerning temporal relations. Our study investigates the temporal comprehension abilities of GPT-3.5 and GPT-4 \cite{OPENAI}, aiming to understand their grasp of temporal relationships. Despite frequent model updates, significant biases were unveiled. Using Question Answering (QA) and Textual Entailment (TE) prompts, we queried both models to determine temporal relations. Illustrated in Figure \ref{fig:main}, the results expose variations in GPT-3.5 and GPT-4 performance, revealing biases towards specific temporal relationships. GPT-3.5 favors &quot;AFTER&quot; in QA for implicit and explicit events, while GPT-4 leans towards &quot;BEFORE.&quot; In TE, GPT-3.5 tends towards &quot;TRUE,&quot; and GPT-4 prefers &quot;FALSE&quot; for both implicit and explicit events.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Temporal Dynamics Analysis, Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1707.07343" label="1707.07343">
        <attvalues>
          <attvalue for="0" value="A Sequential Model for Classifying Temporal Relations between&#10;  Intra-Sentence Events" />
          <attvalue for="1" value="  We present a sequential model for temporal relation classification between&#10;intra-sentence events. The key observation is that the overall syntactic&#10;structure and compositional meanings of the multi-word context between events&#10;are important for distinguishing among fine-grained temporal relations.&#10;Specifically, our approach first extracts a sequence of context words that&#10;indicates the temporal relation between two events, which well align with the&#10;dependency path between two event mentions. The context word sequence, together&#10;with a parts-of-speech tag sequence and a dependency relation sequence that are&#10;generated corresponding to the word sequence, are then provided as input to&#10;bidirectional recurrent neural network (LSTM) models. The neural nets learn&#10;compositional syntactic and semantic representations of contexts surrounding&#10;the two events and predict the temporal relation between them. Evaluation of&#10;the proposed approach on TimeBank corpus shows that sequential modeling is&#10;capable of accurately recognizing temporal relations between events, which&#10;outperforms a neural net model using various discrete features as input that&#10;imitates previous feature based models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.03111" label="2110.03111">
        <attvalues>
          <attvalue for="0" value="Cut the CARP: Fishing for zero-shot story evaluation" />
          <attvalue for="1" value="  Recent advances in large-scale language models (Raffel et al., 2019; Brown et&#10;al., 2020) have brought significant qualitative and quantitative improvements&#10;in machine-driven text generation. Despite this, generation and evaluation of&#10;machine-generated narrative text remains a challenging problem. Objective&#10;evaluation of computationally-generated stories may be prohibitively expensive,&#10;require meticulously annotated datasets, or may not adequately measure the&#10;logical coherence of a generated story's narratological structure.&#10;  Informed by recent advances in contrastive learning (Radford et al., 2021),&#10;we present Contrastive Authoring and Reviewing Pairing (CARP): a scalable,&#10;efficient method for performing qualitatively superior, zero-shot evaluation of&#10;stories. We show a strong correlation between human evaluation of stories and&#10;those of CARP. Model outputs more significantly correlate with corresponding&#10;human input than those language-model based methods which utilize finetuning or&#10;prompt engineering approaches. We also present and analyze the Story-Critique&#10;Dataset, a new corpora composed of 1.3 million aligned story-critique pairs&#10;derived from over 80,000 stories. We expect this corpus to be of interest to&#10;NLP researchers.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent breakthroughs in natural language processing (NLP) and natural language generation (NLG) have revitalized interest in applying computational methods to story \cite{see2019massively,xu2020megatroncntrl,nichols2020collaborative,fang2021transformerbased,hazarika2021zeroshot}. Automated Story Generation is the challenge of designing an artificial intelligence system that can generate a story from a minimal number of inputs---often a simple prompt and some storytelling primitives. Even with modern deep learning techniques this is a significant challenge, as people expect stories to be consistent and coherent, two things that transformers are not particularly good at doing across long passages \cite{gpt3,yao2019plan}.&#10;&#10;Another reason automated story generation is challenging is that automated story evaluation is challenging. In other domains such as image generation \cite{karras2019style, patashnik2021styleclip, galanos2021affectgan}, strategic game-playing \cite{alphazero, alphago}, and planning~\cite{karkus2017qmdp,fan2019automatic}, powerful models for evaluating objects of interest have lead to more powerful models for generating them. Unfortunately, methods for automatically evaluating stories, such as ROUGE \cite{lin2004rouge}, are extremely limited in their ability to accurately assess models and are easily Goodharted \cite{belz2006comparing,cohan2016revisiting,schluter2017limits,schluter2016approximate,eyal2019question}. Recently, researchers interested in story evaluation have developed more sophisticated techniques for evaluating stories \cite{akoury2020storium,dou2021scarecrow} as well as the Purdy Index \cite{purdy2018predicting} and the Fabula-Entropy Index \cite{castricato2021formal,castricato2021fabula}. Unfortunately, all of these methodologies rely on costly human evaluation and are therefore not suitable for integrating into end-to-end pipelines.&#10;&#10;In this paper, we introduce CARP: a transformer-based model for automated story evaluation. CARP is a contrastive model, analogous in design to CLIP \cite{radford2021learning}, that learns to align the embedding space of two distinct encoder models, one that processes a story and one that processes feedback on that story. The result is that a passage's embedding has a higher similarity to those of its critical reviews. This way, CARP is capable of zero-shot story classification and review ranking, and is able to score stories based on their similarity to selected reviews.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Narrative Text Generation, Contrastive Learning Methods, Computational Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2011.10208" label="2011.10208">
        <attvalues>
          <attvalue for="0" value="Collaborative Storytelling with Large-scale Neural Language Models" />
          <attvalue for="1" value="  Storytelling plays a central role in human socializing and entertainment.&#10;However, much of the research on automatic storytelling generation assumes that&#10;stories will be generated by an agent without any human interaction. In this&#10;paper, we introduce the task of collaborative storytelling, where an artificial&#10;intelligence agent and a person collaborate to create a unique story by taking&#10;turns adding to it. We present a collaborative storytelling system which works&#10;with a human storyteller to create a story by generating new utterances based&#10;on the story so far. We constructed the storytelling system by tuning a&#10;publicly-available large scale language model on a dataset of writing prompts&#10;and their accompanying fictional works. We identify generating sufficiently&#10;human-like utterances to be an important technical issue and propose a&#10;sample-and-rank approach to improve utterance quality. Quantitative evaluation&#10;shows that our approach outperforms a baseline, and we present qualitative&#10;evaluation of our system's capabilities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.00828" label="2101.00828">
        <attvalues>
          <attvalue for="0" value="Transformer-based Conditional Variational Autoencoder for Controllable&#10;  Story Generation" />
          <attvalue for="1" value="  We investigate large-scale latent variable models (LVMs) for neural story&#10;generation -- an under-explored application for open-domain long text -- with&#10;objectives in two threads: generation effectiveness and controllability. LVMs,&#10;especially the variational autoencoder (VAE), have achieved both effective and&#10;controllable generation through exploiting flexible distributional latent&#10;representations. Recently, Transformers and its variants have achieved&#10;remarkable effectiveness without explicit latent representation learning, thus&#10;lack satisfying controllability in generation. In this paper, we advocate to&#10;revive latent variable modeling, essentially the power of representation&#10;learning, in the era of Transformers to enhance controllability without hurting&#10;state-of-the-art generation effectiveness. Specifically, we integrate latent&#10;representation vectors with a Transformer-based pre-trained architecture to&#10;build conditional variational autoencoder (CVAE). Model components such as&#10;encoder, decoder and the variational posterior are all built on top of&#10;pre-trained language models -- GPT2 specifically in this paper. Experiments&#10;demonstrate state-of-the-art conditional generation ability of our model, as&#10;well as its excellent representation learning capability and controllability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06411" label="2106.06411">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Controlled Generation with Encoder-Decoder Transformers" />
          <attvalue for="1" value="  Controlling neural network-based models for natural language generation (NLG)&#10;has broad applications in numerous areas such as machine translation, document&#10;summarization, and dialog systems. Approaches that enable such control in a&#10;zero-shot manner would be of great importance as, among other reasons, they&#10;remove the need for additional annotated data and training. In this work, we&#10;propose novel approaches for controlling encoder-decoder transformer-based NLG&#10;models in zero-shot. This is done by introducing three control knobs, namely,&#10;attention biasing, decoder mixing, and context augmentation, that are applied&#10;to these models at generation time. These knobs control the generation process&#10;by directly manipulating trained NLG models (e.g., biasing cross-attention&#10;layers) to realize the desired attributes in the generated outputs. We show&#10;that not only are these NLG models robust to such manipulations, but also their&#10;behavior could be controlled without an impact on their generation performance.&#10;These results, to the best of our knowledge, are the first of their kind.&#10;Through these control knobs, we also investigate the role of transformer&#10;decoder's self-attention module and show strong evidence that its primary role&#10;is maintaining fluency of sentences generated by these models. Based on this&#10;hypothesis, we show that alternative architectures for transformer decoders&#10;could be viable options. We also study how this hypothesis could lead to more&#10;efficient ways for training encoder-decoder transformer models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.04948" label="1812.04948">
        <attvalues>
          <attvalue for="0" value="A Style-Based Generator Architecture for Generative Adversarial Networks" />
          <attvalue for="1" value="  We propose an alternative generator architecture for generative adversarial&#10;networks, borrowing from style transfer literature. The new architecture leads&#10;to an automatically learned, unsupervised separation of high-level attributes&#10;(e.g., pose and identity when trained on human faces) and stochastic variation&#10;in the generated images (e.g., freckles, hair), and it enables intuitive,&#10;scale-specific control of the synthesis. The new generator improves the&#10;state-of-the-art in terms of traditional distribution quality metrics, leads to&#10;demonstrably better interpolation properties, and also better disentangles the&#10;latent factors of variation. To quantify interpolation quality and&#10;disentanglement, we propose two new, automated methods that are applicable to&#10;any generator architecture. Finally, we introduce a new, highly varied and&#10;high-quality dataset of human faces.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.01717" label="2010.01717">
        <attvalues>
          <attvalue for="0" value="STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story&#10;  Generation" />
          <attvalue for="1" value="  Systems for story generation are asked to produce plausible and enjoyable&#10;stories given an input context. This task is underspecified, as a vast number&#10;of diverse stories can originate from a single input. The large output space&#10;makes it difficult to build and evaluate story generation models, as (1)&#10;existing datasets lack rich enough contexts to meaningfully guide models, and&#10;(2) existing evaluations (both crowdsourced and automatic) are unreliable for&#10;assessing long-form creative text. To address these issues, we introduce a&#10;dataset and evaluation platform built from STORIUM, an online collaborative&#10;storytelling community. Our author-generated dataset contains 6K lengthy&#10;stories (125M tokens) with fine-grained natural language annotations (e.g.,&#10;character goals and attributes) interspersed throughout each narrative, forming&#10;a robust source for guiding models. We evaluate language models fine-tuned on&#10;our dataset by integrating them onto STORIUM, where real authors can query a&#10;model for suggested story continuations and then edit them. Automatic metrics&#10;computed over these edits correlate well with both user ratings of generated&#10;stories and qualitative feedback from semi-structured user interviews. We&#10;release both the STORIUM dataset and evaluation platform to spur more&#10;principled research into story generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07472" label="2104.07472">
        <attvalues>
          <attvalue for="0" value="Fabula Entropy Indexing: Objective Measures of Story Coherence" />
          <attvalue for="1" value="  Automated story generation remains a difficult area of research because it&#10;lacks strong objective measures. Generated stories may be linguistically sound,&#10;but in many cases suffer poor narrative coherence required for a compelling,&#10;logically-sound story. To address this, we present Fabula Entropy Indexing&#10;(FEI), an evaluation method to assess story coherence by measuring the degree&#10;to which human participants agree with each other when answering true/false&#10;questions about stories. We devise two theoretically grounded measures of&#10;reader question-answering entropy, the entropy of world coherence (EWC), and&#10;the entropy of transitional coherence (ETC), focusing on global and local&#10;coherence, respectively. We evaluate these metrics by testing them on&#10;human-written stories and comparing against the same stories that have been&#10;corrupted to introduce incoherencies. We show that in these controlled studies,&#10;our entropy indices provide a reliable objective measure of story coherence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.12283" label="2404.12283">
        <attvalues>
          <attvalue for="0" value="Enhancing Embedding Performance through Large Language Model-based Text&#10;  Enrichment and Rewriting" />
          <attvalue for="1" value="  Embedding models are crucial for various natural language processing tasks&#10;but can be limited by factors such as limited vocabulary, lack of context, and&#10;grammatical errors. This paper proposes a novel approach to improve embedding&#10;performance by leveraging large language models (LLMs) to enrich and rewrite&#10;input text before the embedding process. By utilizing ChatGPT 3.5 to provide&#10;additional context, correct inaccuracies, and incorporate metadata, the&#10;proposed method aims to enhance the utility and accuracy of embedding models.&#10;The effectiveness of this approach is evaluated on three datasets:&#10;Banking77Classification, TwitterSemEval 2015, and Amazon Counter-factual&#10;Classification. Results demonstrate significant improvements over the baseline&#10;model on the TwitterSemEval 2015 dataset, with the best-performing prompt&#10;achieving a score of 85.34 compared to the previous best of 81.52 on the&#10;Massive Text Embedding Benchmark (MTEB) Leaderboard. However, performance on&#10;the other two datasets was less impressive, highlighting the importance of&#10;considering domain-specific characteristics. The findings suggest that&#10;LLM-based text enrichment has shown promising results to improve embedding&#10;performance, particularly in certain domains. Hence, numerous limitations in&#10;the process of embedding can be avoided.&#10;" />
          <attvalue for="2" value=" &#10;Text embeddings are widely adopted in the field of Natural Language Processing (NLP) that refer to vectorized representation of natural language. An embedding is a representation of words in a low-dimensional continuous vector space. They encapsulate the semantic content of the text \cite{pittaras2021text}. These embeddings find extensive applications across a spectrum of natural language processing (NLP) endeavors including information retrieval (IR), question answering, assessing semantic textual similarity, mining bitexts, recommending items, etc. The researchers are making continuous efforts to improve the accuracy and reduce the training steps \cite{wang2023improving}.&#10;&#10;Furthermore, an efficient technique for creating high-quality text embeddings using synthetic data and minimal training, avoiding complex pipelines and extensive labeled datasets, and achieving top results on key benchmarks when mixed with labeled data \cite{wang2023improving}.&#10;&#10;There were early approaches like word2vec \cite{mikolov2013efficient} and GloVe \cite{pennington2014glove} to more advanced models such as FastText \cite{bojanowski2017enriching} and BERT \cite{devlin2018bert}. It discusses the strengths and limitations of each model and their impact on various natural language processing (NLP) tasks.&#10;&#10;Various techniques have been proposed to improve the performance of embedding models, such as fine-tuning on domain-specific data \cite{howard2018universal}, using ensemble methods, and incorporating external knowledge sources \cite{zhang2019ernie}. Large language models have been successfully applied to a wide range of NLP tasks, such as text generation \cite{radford2019language}, question answering \cite{raffel2020exploring}, and sentiment analysis \cite{brown2020language}. Several studies have explored the use of text enrichment and rewriting techniques to improve the quality and informativeness of text data. For example, a method for contextual augmentation of text data using a bidirectional language model is being proposed \cite{kobayashi2018contextual}, while a retrieval-augmented generation approach for improving the factual accuracy of generated text was also introduced \cite{guu2020retrieval}. &#10;&#10;Recent research has explored the use of LLMs for text compression to reduce computational costs in Retrieval-Augmented Generation (RAG) systems and large LLMs. For instance, RECOMP proposes compressing retrieved documents into summaries before integrating them with language models, aiming to reduce computational costs and help LMs identify relevant information more efficiently \cite{xu2023recomp}. Similarly, TCRA-LLM introduces a token compression scheme for retrieval-augmented LLMs, employing summarization and semantic compression techniques to reduce inference costs \cite{liu2023tcra}.&#10;Context Tuning for RAG addresses the limitation of RAG's tool retrieval step by employing a smart context retrieval system to fetch relevant information, improving the efficiency and effectiveness of the generation process \cite{anantha2023context}. In the domain of prompt compression, LLMLingua introduces a method for compressing prompts to accelerate inference in LLMs, achieving up to $20$x compression while preserving the original prompt's capabilities \cite{kulkarni2023application}.&#10;The Natural Language Prompt Encapsulation (Nano-Capsulator) framework compresses original prompts into NL formatted Capsule Prompts while maintaining prompt utility and transferability \cite{chuang2024learning}. Compress-Then-Prompt [18] indicates that the generation quality in a compressed LLM can be markedly improved for specific queries by selecting prompts with high efficiency and accuracy trade-offs \cite{xu2023compress}.&#10;LongLLMLingua focuses on improving LLMs' perception of key information in long context scenarios through prompt compression, showing that compressed prompts could derive higher performance with much less cost and reduce the latency of the end-to-end system. Data Distillation proposes a data distillation procedure to compress prompts without losing crucial information, addressing issues related to the efficiency and fidelity of task-agnostic prompt compression.&#10;While these approaches aim to reduce computational costs, the current study explores the potential of LLMs for text enrichment to enhance embedding quality.&#10;&#10;Embedding models have become an essential component of various natural language processing (NLP) tasks, such as text classification, clustering, and retrieval. These models learn dense vector representations of words, sentences, or documents, capturing semantic and syntactic relationships between them. The quality of these embeddings directly impacts the performance of downstream applications.&#10;&#10;Despite their widespread use, embedding models face several challenges that limit their performance. These challenges include limited vocabulary, lack of context, sensitivity to grammatical errors, data sparsity, and lack of domain-specific tuning. For example, embedding models may struggle with newer or domain-specific terms not present in their training data, leading to mis-classification or poor retrieval performance. Existing approaches to improve embedding performance often focus on fine-tuning the embedding models on domain-specific data or using ensemble techniques. However, these methods can be resource-intensive and may not effectively address the fundamental limitations of embedding models, such as their inability to capture context or handle grammatical errors. Large language models (LLMs) have demonstrated remarkable capabilities in understanding and generating human-like text. By leveraging the knowledge and contextual understanding of LLMs, it is possible to enrich and rewrite input text before the embedding process, thereby addressing the limitations of embedding models and improving their performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Enhancement, Text Embedding Optimization, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.15556" label="2310.15556">
        <attvalues>
          <attvalue for="0" value="TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for&#10;  Inference Cost Reduction" />
          <attvalue for="1" value="  Since ChatGPT released its API for public use, the number of applications&#10;built on top of commercial large language models (LLMs) increase exponentially.&#10;One popular usage of such models is leveraging its in-context learning ability&#10;and generating responses given user queries leveraging knowledge obtained by&#10;retrieval augmentation. One problem of deploying commercial retrieval-augmented&#10;LLMs is the cost due to the additionally retrieved context that largely&#10;increases the input token size of the LLMs. To mitigate this, we propose a&#10;token compression scheme that includes two methods: summarization compression&#10;and semantic compression. The first method applies a T5-based model that is&#10;fine-tuned by datasets generated using self-instruct containing samples with&#10;varying lengths and reduce token size by doing summarization. The second method&#10;further compresses the token size by removing words with lower impact on the&#10;semantic. In order to adequately evaluate the effectiveness of the proposed&#10;methods, we propose and utilize a dataset called Food-Recommendation DB (FRDB)&#10;focusing on food recommendation for women around pregnancy period or infants.&#10;Our summarization compression can reduce 65% of the retrieval token size with&#10;further 0.3% improvement on the accuracy; semantic compression provides a more&#10;flexible way to trade-off the token size with performance, for which we can&#10;reduce the token size by 20% with only 1.6% of accuracy drop.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.05708" label="2312.05708">
        <attvalues>
          <attvalue for="0" value="Context Tuning for Retrieval Augmented Generation" />
          <attvalue for="1" value="  Large language models (LLMs) have the remarkable ability to solve new tasks&#10;with just a few examples, but they need access to the right tools. Retrieval&#10;Augmented Generation (RAG) addresses this problem by retrieving a list of&#10;relevant tools for a given task. However, RAG's tool retrieval step requires&#10;all the required information to be explicitly present in the query. This is a&#10;limitation, as semantic search, the widely adopted tool retrieval method, can&#10;fail when the query is incomplete or lacks context. To address this limitation,&#10;we propose Context Tuning for RAG, which employs a smart context retrieval&#10;system to fetch relevant information that improves both tool retrieval and plan&#10;generation. Our lightweight context retrieval model uses numerical,&#10;categorical, and habitual usage signals to retrieve and rank context items. Our&#10;empirical results demonstrate that context tuning significantly enhances&#10;semantic search, achieving a 3.5-fold and 1.5-fold improvement in Recall@K for&#10;context retrieval and tool retrieval tasks respectively, and resulting in an&#10;11.6% increase in LLM-based planner accuracy. Additionally, we show that our&#10;proposed lightweight model using Reciprocal Rank Fusion (RRF) with LambdaMART&#10;outperforms GPT-4 based retrieval. Moreover, we observe context augmentation at&#10;plan generation, even after tool retrieval, reduces hallucination.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Using retrieval to incorporate tools into plan generation with LLMs has emerged as a burgeoning area of research, with ongoing investigations aimed at enhancing both the retrieval component and the LLMs themselves. Our work falls within the former category, placing a particular emphasis on refining retrieval methodologies to enhance contextual understanding of implicit and ambiguous queries that demand context-seeking capabilities.&#10;&#10;The integration of tools into generation has been demonstrated to enhance the capabilities of LLM-based planners in recent studies~\cite{toolformer:2023, chameleon:2023}. However, these works primarily focus on well-defined or unambiguous queries, where retrieving supplementary information to augment the query is not strictly required. For question answering (QA) tasks, incorporating any off-the-shelf document retriever has been shown to improve LLM generation, with the addition of re-ranking further boosting performance~\cite{icralm:2023}. While re-ranking is preferred, employing any pre-trained retriever, particularly a text-based retriever, would be sub-optimal due to the inadequate information expected from ambiguous queries. Our work demonstrates the inadequacy of text-based retrievers for context retrieval and the necessity of more advanced retrieval models.&#10;&#10;To address the lack of context inherent in under-specified queries, some studies have explored the use of CoT~\cite{cot:2022} mechanisms to generate text that closely approximates the semantic similarity of relevant context~\cite{qrrag:2023}. While CoT augmentation improves upon baseline methods, such as vanilla semantic search, CoT may potentially increase the input length to the LLM, which has a limited context window size. Additionally, studies have demonstrated that the placement of relevant information impacts LLM generation~\cite{lost-in-middle:2023}. Therefore, it is preferable to avoid increasing input sequence length if the same or better results can be achieved without query augmentation. Distillation-based query augmentation approaches have been proposed to address this problem~\cite{quill:2023}. Our work unveils that fine-tuning semantic search obviates the necessity for query augmentation while achieving comparable performance.&#10;&#10;Recent studies have shown LLMs can act as zero-shot rankers through pairwise ranking prompting~\cite{llm-ranking:2023}. While addition of ranking for retrieval component has shown improvement in QA tasks, direct use of LLMs for the ranking task, in addition to plan generation, incurs twice the inference cost. We empirically show that our proposed lightweight context tuning method, LambdaMART~\cite{lambdamart:2010} based RRF~\cite{rrf:09}, outperforms both fine-tuning approach and GPT-4~\cite{gpt-4:23} based CoT Augmentation. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2209.04156" label="2209.04156">
        <attvalues>
          <attvalue for="0" value="Multi-grained Label Refinement Network with Dependency Structures for&#10;  Joint Intent Detection and Slot Filling" />
          <attvalue for="1" value="  Slot filling and intent detection are two fundamental tasks in the field of&#10;natural language understanding. Due to the strong correlation between these two&#10;tasks, previous studies make efforts on modeling them with multi-task learning&#10;or designing feature interaction modules to improve the performance of each&#10;task. However, none of the existing approaches consider the relevance between&#10;the structural information of sentences and the label semantics of two tasks.&#10;The intent and semantic components of a utterance are dependent on the&#10;syntactic elements of a sentence. In this paper, we investigate a multi-grained&#10;label refinement network, which utilizes dependency structures and label&#10;semantic embeddings. Considering to enhance syntactic representations, we&#10;introduce the dependency structures of sentences into our model by graph&#10;attention layer. To capture the semantic dependency between the syntactic&#10;information and task labels, we combine the task specific features with&#10;corresponding label embeddings by attention mechanism. The experimental results&#10;demonstrate that our model achieves the competitive performance on two public&#10;datasets.&#10;" />
          <attvalue for="2" value="&#10;Slot filling and intent detection are two critical tasks for natural language&#10;understanding (NLU).&#10;The two tasks are defined to identify intents and extract semantic components&#10;from utterances in dialog systems~\cite{DBLP:journals/corr/abs-2101-08091}.&#10;Intent detection is formulated as a sentence-level classification problem, and&#10;slot filling can be regarded as a sequence labeling problem.&#10;Traditional methods tend to solve the two tasks independently.&#10;For intent detection, researchers applied traditional machine learning methods,&#10;such as logistic regression, random forest, and deep belief&#10;networks~\cite{tur2011spoken}.&#10;The sequence-based models, such as long short-term memory&#10;(LSTM)~\cite{DBLP:journals/neco/HochreiterS97} and conditional random fields&#10;(CRF)~\cite{DBLP:conf/icml/LaffertyMP01}, have achieved significant performances&#10;on slot filling.&#10;&#10;Considering the relevance between intent detection and slot filling, some works&#10;proposed joint models to tackle the two tasks with feature interaction between&#10;them.&#10;Previous joint learning methods utilized the supervised signal from intent&#10;detection to improve the performance of slot filling by&#10;attention~\cite{DBLP:conf/interspeech/LiuL16} or&#10;gated~\cite{DBLP:conf/naacl/GooGHHCHC18} mechanisms.&#10;Recently, Qin et al.~\cite{DBLP:conf/icassp/QinLCKZ021} proposed a&#10;co-interactive module to model the cross-impact of two tasks and achieved the&#10;state-of-the-art performance.&#10;The joint learning methods always demonstrate their effectiveness over the&#10;independent&#10;models~\cite{DBLP:conf/interspeech/Hakkani-TurTCCG16,DBLP:conf/acl/ZhangLDFY19,DBLP:conf/icassp/HuiWCYWX21}.&#10;However, the existing methods do not take the advantage of the correlation&#10;between the syntactic information and target label semantics.&#10;The syntactic information implies the dependency structures of sentences.&#10;And the objects of prepositional phrases are often slot values to be extracted&#10;in a sentence while the intent of speakers can be reflected by the verbs.&#10;Therefore, it is a meaningful way to exploit the dependency structures of&#10;syntactic for enhancing the sentence representations cooperated with label&#10;semantics.&#10;&#10;To address the limitations of existing approaches, we propose a multi-grained&#10;label refinement network with dependency structures for jointly modeling slot&#10;filling and intent detection.&#10;When utilizing the syntactic knowledge, Wang et&#10;al.~\cite{DBLP:conf/aaai/WangWRZC21} proposed an task to predict the dependency&#10;matrix.&#10;Considering the variant importance of syntactic characteristics in dependency&#10;structures, we encode the syntactic information by graph attention&#10;network~\cite{velickovic2018graph}.&#10;Furthermore, we acquire the semantic embeddings of task labels by their&#10;descriptions~\cite{DBLP:journals/taslp/ZhuZMY20}.&#10;Through the attention mechanism~\cite{zhou-etal-2016-attention}, we fuse the&#10;syntactic-enhanced sentence features with prior label semantics of slot filling&#10;and intent detection.&#10;The above operations can bridge the gap between the syntactic information and&#10;label semantics, improving the performance of the two tasks.&#10;&#10;We compare our model with several state-of-the-art baselines on two public&#10;datasets: ATIS~\cite{DBLP:conf/naacl/HemphillGD90} and&#10;SNIPS~\cite{DBLP:journals/corr/abs-1805-10190}.&#10;The experimental results demonstrate that our model can achieve significant&#10;improvements on different metrics.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Natural Language Understanding, Syntactic Representation, Multi-Task Learning, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2102.10905" label="2102.10905">
        <attvalues>
          <attvalue for="0" value="Joint Intent Detection And Slot Filling Based on Continual Learning&#10;  Model" />
          <attvalue for="1" value="  Slot filling and intent detection have become a significant theme in the&#10;field of natural language understanding. Even though slot filling is&#10;intensively associated with intent detection, the characteristics of the&#10;information required for both tasks are different while most of those&#10;approaches may not fully aware of this problem. In addition, balancing the&#10;accuracy of two tasks effectively is an inevitable problem for the joint&#10;learning model. In this paper, a Continual Learning Interrelated Model (CLIM)&#10;is proposed to consider semantic information with different characteristics and&#10;balance the accuracy between intent detection and slot filling effectively. The&#10;experimental results show that CLIM achieves state-of-the-art performace on&#10;slot filling and intent detection on ATIS and Snips.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.10838" label="1910.10838">
        <attvalues>
          <attvalue for="0" value="Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural&#10;  Speaker Embeddings" />
          <attvalue for="1" value="  While speaker adaptation for end-to-end speech synthesis using speaker&#10;embeddings can produce good speaker similarity for speakers seen during&#10;training, there remains a gap for zero-shot adaptation to unseen speakers. We&#10;investigate multi-speaker modeling for end-to-end text-to-speech synthesis and&#10;study the effects of different types of state-of-the-art neural speaker&#10;embeddings on speaker similarity for unseen speakers. Learnable dictionary&#10;encoding-based speaker embeddings with angular softmax loss can improve equal&#10;error rates over x-vectors in a speaker verification task; these embeddings&#10;also improve speaker similarity and naturalness for unseen speakers when used&#10;for zero-shot adaptation to new speakers in end-to-end speech synthesis.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advances in end-to-end text-to-speech (TTS) synthesis have enabled us to produce very realistic and natural-sounding synthetic speech \cite{wang2017tacotron,ping2018clarinet} with mean opinion scores (MOS) approaching those of natural human speech \cite{shen2018natural}. Not only speaker dependent TTS systems but also multi-speaker TTS systems show remarkable results \cite{abs-1907-04462}. However, adapting voice models to arbitrary new speakers using a small amount of data (speaker adaptation) remains a challenge.&#10;&#10;An effective approach for speaker adaptation in neural TTS is to fine-tune all or part of model with a small amount of data from the target speaker \cite{kons2019high,chen2019sample}. This approach can also be used to adapt to new speaking styles such as Lombard speech \cite{bollepalli2019lombard}. A different but complementary approach is to use speaker embeddings to model speaker identity in TTS. Prior studies have focused on training a speaker encoder network jointly with the TTS model \cite{chen2019sample,park2019multi,nachmani2018fitting} or the neural vocoder \cite{deng2018modeling}; others have explored the use of speaker embeddings in combination with fine-tuning the TTS model \cite{deng2018modeling,hu2019neural,arik2018neural}. Approaches that use fine-tuning necessarily require transcribed adaptation data, as well as more computational time and resources to adapt to a new speaker. Furthermore, speaker encoder networks that are jointly trained with the TTS model cannot benefit from data outside of the TTS training data, which is restricted to be of relatively high quality in clean recording conditions. &#10;&#10;Transfer learning for speaker modeling in TTS addresses these issues. With this approach, the speaker embedding network is trained completely separately, perhaps for a different task such as speaker recognition. The benefit of this approach is that speaker recognition models can be trained on a large amount of data that does not have to be of the same high quality typically required for TTS, and these models can obtain robust speaker representations that are independent of channel and recording conditions using relatively small amounts of target speaker data, which does not necessarily have to be transcribed. End-to-end synthesis models can then be used to adapt to a target speaker's voice in a zero-shot manner by using the speaker embedding only, without necessarily needing to fine-tune the entire model. Several recent studies \cite{nachmani2018fitting,jia2018transfer,pascual2019learning,chen2019cross} have used this approach for speaker modeling in TTS, with \cite{chen2019cross} modeling both speaker and language characteristics. \cite{jia2018transfer} observed that unseen speakers' synthetic speech had lower speaker similarity to the target speaker than seen speakers, accents were often mismatched, and nuances such as characteristic prosody were lost, indicating that while seen speakers can be well-modeled in this manner, there is room for improvement for modeling unseen speakers.&#10;&#10;In parallel with the above-mentioned studies, there has been substantial development in end-to-end speaker recognition. Villalba et al. summarized several state-of-the-art speaker recognition systems for the NIST SRE18 Challenge \cite{villalba2019state}, where x-vector based systems \cite{snyder2018x} consistently outperformed i-vector based systems \cite{dehak2010front}. There has also been a surge of interest in new encoding methods and end-to-end loss functions for speaker recognition \cite{cai2018exploring, Huang2018, chen2019tied, xie2019utterance, hajibabaei2018unified, xiang2019margin, jung2019spatial}. One prominent advancement is the use of learnable dictionary encoding (LDE) \cite{cai2018exploring} and angular softmax \cite{Huang2018} for speaker recognition, which are reported to boost the speaker recognition performance on open-source corpora such as the VoxCelebs \cite{nagrani2017voxceleb, chung2018voxceleb2}. &#10;&#10;One aspect of our study is therefore an attempt to find out how effective these recent developments in speaker verification are for speaker adaption in TTS.&#10;More specifically we investigate the capability of neural speaker embeddings \cite{villalba2019state, snyder2018x, cai2018exploring} to capture and model characteristics of speakers that were unseen during TTS model training. For this purpose, we extend an improved Tacotron system in \cite{yasuda2019investigation} to a multi-speaker TTS system and conduct systematic analysis to answer the above question. We also analyze how the quality and similarity of generated voices are correlated with automatic speaker verification (ASV) accuracy.&#10;&#10;While prior studies have focused on transfer learning for zero-shot speaker adaptation for end-to-end TTS, to our knowledge this is the first investigation of many different types of speaker embeddings to determine whether some type of embedding is best for modeling unseen speakers and to learn whether the best embeddings for ASV are the same as the best embeddings for TTS.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Speech Synthesis, Speaker Embeddings, Signal Processing, Neural Modeling" />
        </attvalues>
      </node>
      <node id="1807.07281" label="1807.07281">
        <attvalues>
          <attvalue for="0" value="ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech" />
          <attvalue for="1" value="  In this work, we propose a new solution for parallel wave generation by&#10;WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we&#10;distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet&#10;by minimizing a regularized KL divergence between their highly-peaked output&#10;distributions. Our method computes the KL divergence in closed-form, which&#10;simplifies the training algorithm and provides very efficient distillation. In&#10;addition, we introduce the first text-to-wave neural architecture for speech&#10;synthesis, which is fully convolutional and enables fast end-to-end training&#10;from scratch. It significantly outperforms the previous pipeline that connects&#10;a text-to-spectrogram model to a separately trained WaveNet (Ping et al.,&#10;2018). We also successfully distill a parallel waveform synthesizer conditioned&#10;on the hidden representation in this end-to-end model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.04462" label="1907.04462">
        <attvalues>
          <attvalue for="0" value="Multi-Speaker End-to-End Speech Synthesis" />
          <attvalue for="1" value="  In this work, we extend ClariNet (Ping et al., 2019), a fully end-to-end&#10;speech synthesis model (i.e., text-to-wave), to generate high-fidelity speech&#10;from multiple speakers. To model the unique characteristic of different voices,&#10;low dimensional trainable speaker embeddings are shared across each component&#10;of ClariNet and trained together with the rest of the model. We demonstrate&#10;that the multi-speaker ClariNet outperforms state-of-the-art systems in terms&#10;of naturalness, because the whole model is jointly optimized in an end-to-end&#10;manner.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.00590" label="1905.00590">
        <attvalues>
          <attvalue for="0" value="High quality, lightweight and adaptable TTS using LPCNet" />
          <attvalue for="1" value="  We present a lightweight adaptable neural TTS system with high quality&#10;output. The system is composed of three separate neural network blocks: prosody&#10;prediction, acoustic feature prediction and Linear Prediction Coding Net as a&#10;neural vocoder. This system can synthesize speech with close to natural quality&#10;while running 3 times faster than real-time on a standard CPU. The modular&#10;setup of the system allows for simple adaptation to new voices with a small&#10;amount of data. We first demonstrate the ability of the system to produce high&#10;quality speech when trained on large, high quality datasets. Following that, we&#10;demonstrate its adaptability by mimicking unseen voices using 5 to 20 minutes&#10;long datasets with lower recording quality. Large scale Mean Opinion Score&#10;quality and similarity tests are presented, showing that the system can adapt&#10;to unseen voices with quality gap of 0.12 and similarity gap of 3% compared to&#10;natural speech for male voices and quality gap of 0.35 and similarity of gap of&#10;9 % for female voices.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.10460" label="1809.10460">
        <attvalues>
          <attvalue for="0" value="Sample Efficient Adaptive Text-to-Speech" />
          <attvalue for="1" value="  We present a meta-learning approach for adaptive text-to-speech (TTS) with&#10;few data. During training, we learn a multi-speaker model using a shared&#10;conditional WaveNet core and independent learned embeddings for each speaker.&#10;The aim of training is not to produce a neural network with fixed weights,&#10;which is then deployed as a TTS system. Instead, the aim is to produce a&#10;network that requires few data at deployment time to rapidly adapt to new&#10;speakers. We introduce and benchmark three strategies: (i) learning the speaker&#10;embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire&#10;architecture with stochastic gradient descent, and (iii) predicting the speaker&#10;embedding with a trained neural network encoder. The experiments show that&#10;these approaches are successful at adapting the multi-speaker neural network to&#10;new speakers, obtaining state-of-the-art results in both sample naturalness and&#10;voice similarity with merely a few minutes of audio data from new speakers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.05253" label="1812.05253">
        <attvalues>
          <attvalue for="0" value="Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick&#10;  Enrolling New Speaker and Enhancing Premium Voice" />
          <attvalue for="1" value="  Neural TTS has shown it can generate high quality synthesized speech. In this&#10;paper, we investigate the multi-speaker latent space to improve neural TTS for&#10;adapting the system to new speakers with only several minutes of speech or&#10;enhancing a premium voice by utilizing the data from other speakers for richer&#10;contextual coverage and better generalization. A multi-speaker neural TTS model&#10;is built with the embedded speaker information in both spectral and speaker&#10;latent space. The experimental results show that, with less than 5 minutes of&#10;training data from a new speaker, the new model can achieve an MOS score of&#10;4.16 in naturalness and 4.64 in speaker similarity close to human recordings&#10;(4.74). For a well-trained premium voice, we can achieve an MOS score of 4.5&#10;for out-of-domain texts, which is comparable to an MOS of 4.58 for professional&#10;recordings, and significantly outperforms single speaker result of 4.28.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.05160" label="1804.05160">
        <attvalues>
          <attvalue for="0" value="Exploring the Encoding Layer and Loss Function in End-to-End Speaker and&#10;  Language Recognition System" />
          <attvalue for="1" value="  In this paper, we explore the encoding/pooling layer and loss function in the&#10;end-to-end speaker and language recognition system. First, a unified and&#10;interpretable end-to-end system for both speaker and language recognition is&#10;developed. It accepts variable-length input and produces an utterance level&#10;result. In the end-to-end system, the encoding layer plays a role in&#10;aggregating the variable-length input sequence into an utterance level&#10;representation. Besides the basic temporal average pooling, we introduce a&#10;self-attentive pooling layer and a learnable dictionary encoding layer to get&#10;the utterance level representation. In terms of loss function for open-set&#10;speaker verification, to get more discriminative speaker embedding, center loss&#10;and angular softmax loss is introduced in the end-to-end system. Experimental&#10;results on Voxceleb and NIST LRE 07 datasets show that the performance of&#10;end-to-end learning system could be significantly improved by the proposed&#10;encoding layer and loss function.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04939" label="2109.04939">
        <attvalues>
          <attvalue for="0" value="Modeling Human Sentence Processing with Left-Corner Recurrent Neural&#10;  Network Grammars" />
          <attvalue for="1" value="  In computational linguistics, it has been shown that hierarchical structures&#10;make language models (LMs) more human-like. However, the previous literature&#10;has been agnostic about a parsing strategy of the hierarchical models. In this&#10;paper, we investigated whether hierarchical structures make LMs more&#10;human-like, and if so, which parsing strategy is most cognitively plausible. In&#10;order to address this question, we evaluated three LMs against human reading&#10;times in Japanese with head-final left-branching structures: Long Short-Term&#10;Memory (LSTM) as a sequential model and Recurrent Neural Network Grammars&#10;(RNNGs) with top-down and left-corner parsing strategies as hierarchical&#10;models. Our computational modeling demonstrated that left-corner RNNGs&#10;outperformed top-down RNNGs and LSTM, suggesting that hierarchical and&#10;left-corner architectures are more cognitively plausible than top-down or&#10;sequential architectures. In addition, the relationships between the cognitive&#10;plausibility and (i) perplexity, (ii) parsing, and (iii) beam size will also be&#10;discussed.&#10;" />
          <attvalue for="2" value="&#10;It has been debated in computational linguistics whether language models (LMs) become more human-like by explicitly modeling hierarchical structures of natural languages. Previous work has revealed that while sequential models such as recurrent neural networks (RNNs) can successfully predict human reading times \cite{frankInsensitivityHumanSentenceProcessing2011}, there is an advantage in explicitly modeling hierarchical structures \cite{fossumSequentialVsHierarchical2012a}. More recently, RNNs that explicitly model hierarchical structures, namely Recurrent Neural Network Grammars (RNNGs, \cite{dyerRecurrentNeuralNetwork2016a}), have attracted considerable attention, effectively capturing grammatical dependencies (e.g., subject-verb agreement) much better than RNNs in targeted syntactic evaluations \cite{kuncoroLSTMsCanLearn2018, wilcox-etal-2019-structural}. In addition, \cite{haleFindingSyntaxHuman2018} showed that RNNGs can successfully predict human brain activities, and recommended RNNGs as ``a mechanistic model of the syntactic processing that occurs during normal human sentence processing.''&#10;&#10;However, this debate has focused almost exclusively on the dichotomy between the hierarchical and sequential models, without reference to the parsing strategies among the hierarchical models. Specifically, although \cite{dyerRecurrentNeuralNetwork2016a} and \cite{haleFindingSyntaxHuman2018} adopted the vanilla RNNG with a top-down parsing strategy for English with head-initial right-branching structures, \cite{abneyMemoryRequirementsLocal1991} and \cite{resnikLeftcornerParsingPsychological1992} suggested that the top-down parsing strategy is not optimal for head-final left-branching structures, and alternatively proposed the left-corner parsing strategy as more human-like parsing strategy.&#10;&#10;In this paper, we investigate whether hierarchical structures make LMs more human-like, and if so, which parsing strategy is most cognitively plausible. In order to address this question, we evaluate three LMs against human reading times in Japanese with head-final left-branching structures: Long Short-Term Memory (LSTM) as a sequential model and Recurrent Neural Network Grammars (RNNGs) with top-down and left-corner parsing strategies as hierarchical models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Computational Linguistics, Artificial Intelligence, Cognitive Plausibility Modeling, Hierarchical Parsing Strategies" />
        </attvalues>
      </node>
      <node id="1806.04127" label="1806.04127">
        <attvalues>
          <attvalue for="0" value="Finding Syntax in Human Encephalography with Beam Search" />
          <attvalue for="1" value="  Recurrent neural network grammars (RNNGs) are generative models of&#10;(tree,string) pairs that rely on neural networks to evaluate derivational&#10;choices. Parsing with them using beam search yields a variety of incremental&#10;complexity metrics such as word surprisal and parser action count. When used as&#10;regressors against human electrophysiological responses to naturalistic text,&#10;they derive two amplitude effects: an early peak and a P600-like later peak. By&#10;contrast, a non-syntactic neural language model yields no reliable effects.&#10;Model comparisons attribute the early peak to syntactic composition within the&#10;RNNG. This pattern of results recommends the RNNG+beam search combination as a&#10;mechanistic model of the syntactic processing that occurs during normal human&#10;language comprehension.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.06341" label="2110.06341">
        <attvalues>
          <attvalue for="0" value="Learning Compact Metrics for MT" />
          <attvalue for="1" value="  Recent developments in machine translation and multilingual text generation&#10;have led researchers to adopt trained metrics such as COMET or BLEURT, which&#10;treat evaluation as a regression problem and use representations from&#10;multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on&#10;related tasks suggest that these models are most efficient when they are large,&#10;which is costly and impractical for evaluation. We investigate the trade-off&#10;between multilinguality and model capacity with RemBERT, a state-of-the-art&#10;multilingual language model, using data from the WMT Metrics Shared Task. We&#10;present a series of experiments which show that model size is indeed a&#10;bottleneck for cross-lingual transfer, then demonstrate how distillation can&#10;help addressing this bottleneck, by leveraging synthetic data generation and&#10;transferring knowledge from one teacher to multiple students trained on related&#10;languages. Our method yields up to 10.5% improvement over vanilla fine-tuning&#10;and reaches 92.6% of RemBERT's performance using only a third of its&#10;parameters.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent improvements in Machine Translation (MT) and multilingual Natural Language Generation (NLG) have led researchers to question the use of n-gram overlap metrics such as BLEU and ROUGE~\cite{papineni2002bleu,lin2004rouge}. Since these metrics focus solely on surface-level aspects of the generated text, they correlate poorly with human evaluation, especially when models are producing high-quality text~\cite{belz2006comparing,callison2006re,ma2019results,mathur2020tangled}. This has led to a surge of interest in learned metrics that cast evaluation as a regression problem and leverage pre-trained multilingual models to capture the semantic similarity between references and generated text~\cite{celikyilmaz2020evaluation}. Popular examples of those metrics include Comet~\cite{rei2020comet} and Bleurt-Extended~\cite{sellam2020bleurt}, based on XLM-RoBERTa~\cite{conneau2019cross, conneau2019unsupervised} and mBERT~\cite{devlin2018bert} respectively. These metrics deliver superior performance over those based on lexical overlap, outperforming even crowd-sourced annotations~\cite{freitag2021experts, mathur2020results}.&#10;&#10;Large pre-trained models benefit learned metrics in at least two ways. First, they allow for cross-task transfer: the contextual embeddings they produce allow researchers to address the relative scarcity of training data that exist for the task, especially with large models such as BERT or XLNet~\cite{zhang2019bertscore, devlin2018bert, yang2019xlnet}. Second, they allow for cross-lingual transfer: MT evaluation is often multilingual, yet few, if any, popular datasets cover more than 20 languages. Evidence suggests that training on many languages improves performance on languages for which there is little training data, including the zero-shot setup, in which no fine-tuning data is available~\cite{conneau2019cross, sellam2020learning, conneau2018xnli, pires2019multilingual}. &#10;&#10;However, the accuracy gains only appear if the model is large enough. In the case of cross-lingual transfer, this phenomenon is known as the curse of multilinguality: to allow for positive transfer, the model must be scaled up with the number of languages~\cite{conneau2019cross}. Scaling up metric models is particularly problematic, since they must often run alongside an already large MT or NLG model and, therefore, must share hardware resources (see ~\cite{shu2021reward} for a recent use case). This contention may lead to impractical delays, it increases the cost of running experiments, and it prevents researchers with limited resources from engaging in shared tasks.&#10;&#10;We first present a series of experiments that validate that previous findings on cross-lingual transfer and the curse of multilinguality apply to the metrics domain, using RemBERT (Rebalanced mBERT), a multilingual extension of BERT~\cite{chung2020rethinking}. We then investigate how a combination of multilingual data generation and distillation can help us reap the benefits of multiple languages while keeping the models compact. Distillation has been shown to successfully transfer knowledge from large models to smaller ones~\cite{hinton2015distilling}, but it requires access to a large corpus of unlabelled data~\cite{sanh2019distilbert,turc2019well}, which does not exist for our task. Inspired by~\cite{sellam2020bleurt}, we introduce a data generation method based on random perturbations that allows us to synthesize arbitrary amounts of multilingual training data. We generate an 80M-sentence distillation corpus in 13 languages from Wikipedia, and show that we can improve a vanilla pre-trained distillation setup~\cite{turc2019well} by up to 12\%. A second, less explored benefit of distillation is that it lets us partially bypass the curse of multilinguality. Once the teacher (i.e., larger) model has been trained, we can generate training data for any language, including the zero-shot ones. Thus, we are less reliant on cross-lingual transfer. We can lift the restriction that one model must carry all the languages, and train smaller models, targeted towards specific language families. Doing so increases performance further by up to~4\%. Combining these two methods, we match 92.6\% of the the largest RemBERT model's performance using only a third of its parameters.&#10;&#10;A selection of code and models is available online at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Multilingual Models, Artificial Intelligence, Model Efficiency" />
        </attvalues>
      </node>
      <node id="2006.14799" label="2006.14799">
        <attvalues>
          <attvalue for="0" value="Evaluation of Text Generation: A Survey" />
          <attvalue for="1" value="  The paper surveys evaluation methods of natural language generation (NLG)&#10;systems that have been developed in the last few years. We group NLG evaluation&#10;methods into three categories: (1) human-centric evaluation metrics, (2)&#10;automatic metrics that require no training, and (3) machine-learned metrics.&#10;For each category, we discuss the progress that has been made and the&#10;challenges still being faced, with a focus on the evaluation of recently&#10;proposed NLG tasks and neural NLG models. We then present two examples for&#10;task-specific NLG evaluations for automatic text summarization and long text&#10;generation, and conclude the paper by proposing future research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.12234" label="2309.12234">
        <attvalues>
          <attvalue for="0" value="Bridging the Gaps of Both Modality and Language: Synchronous Bilingual&#10;  CTC for Speech Translation and Speech Recognition" />
          <attvalue for="1" value="  In this study, we present synchronous bilingual Connectionist Temporal&#10;Classification (CTC), an innovative framework that leverages dual CTC to bridge&#10;the gaps of both modality and language in the speech translation (ST) task.&#10;Utilizing transcript and translation as concurrent objectives for CTC, our&#10;model bridges the gap between audio and text as well as between source and&#10;target languages. Building upon the recent advances in CTC application, we&#10;develop an enhanced variant, BiL-CTC+, that establishes new state-of-the-art&#10;performances on the MuST-C ST benchmarks under resource-constrained scenarios.&#10;Intriguingly, our method also yields significant improvements in speech&#10;recognition performance, revealing the effect of cross-lingual learning on&#10;transcription and demonstrating its broad applicability. The source code is&#10;available at https://github.com/xuchennlp/S2T.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic speech recognition (ASR) and speech translation (ST) have undergone remarkable evolution in recent years. &#10;Current end-to-end systems consistently outperform their traditional hybrid and cascaded counterparts in both efficiency and performance \cite{Gulati_ISCA2020, xu2023recent}. &#10;Despite these progresses, achieving stable convergence still remains a challenge due to the inherent complexities arising from the modality gap between audio and text.&#10;To mitigate this, Connectionist Temporal Classification (CTC) \cite{Graves_ACL2006} has emerged as a widely-used auxiliary training objective \cite{watanabe2017hybrid}, which has demonstrated a notable improvement in stabilizing model convergence and enhancing performance, especially in the more challenging ST task \cite{bahar2019comparative}.&#10;&#10;CTC is typically implemented on top of the encoder, where it predicts the corresponding transcript of the input audio. &#10;This mechanism offers a more lightweight and direct solution compared to alternative techniques like pre-training \cite{bansal2018pre} or data augmentation \cite{pino_corr2019}. &#10;Nevertheless, this approach has its limitations.&#10;In particular, its narrow focus on the transcription can potentially interfere with the semantic understanding that is critical in the ST task \cite{Xu_ACL2021}.&#10;&#10;Researchers propose a stacked design for ST that decouples the encoding into an acoustic encoder and a textual encoder, where only the former is supervised by the CTC loss\cite{Xu_ACL2021,nast}. &#10;Considering the gaps of both modality and language in the ST task, the following study further enhances it by introducing another cross-lingual CTC (XCTC), which guides the textual encoder to predict target translation \cite{nast}, as illustrated in Figure \ref{fig:progressive}.&#10;This approach offers comprehensive guidance from the bilingual text in a progressive manner.&#10;We refer to it as bilingual CTC (BiL-CTC).&#10;&#10;In this paper, we present a novel implementation of BiL-CTC that synchronously predicts both transcript and translation, as illustrated in Figure \ref{fig:syn}. &#10;While the progressive strategy of incrementally learning the cross-modal and cross-lingual prediction may seem intuitively sound, our synchronous approach promotes mutual assistance between bilingual prediction.&#10;This not only facilitates the learning of language-agnostic representations but also enhances the accuracy of semantic understanding.&#10;This strategy is inspired by the paradigm shift from cascaded systems to end-to-end models, where direct semantic comprehension replaces the two-stage pipeline of cross-modal transcription followed by cross-lingual translation.&#10;&#10;We further come up with an enhanced variant by combining several recent advancements in CTC application, including InterCTC \cite{Lee_ICASSP2021}, prediction-aware encoding \cite{nast}, and a curriculum-based training strategy \cite{nast}, referred to as BiL-CTC+. &#10;The resulting speech translation model not only establishes a new state-of-the-art performance on the MuST-C corpora in resource-constrained scenarios but also outperforms most of the methods that rely on abundant external data. &#10;Intriguingly, we observe that our method consistently improves ASR model performance across both low-resource and high-resource settings. &#10;To the best of our knowledge, this is the first work to demonstrate that direct cross-lingual modeling can positively impact ASR.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Translation, Computer Science, Linguistics, Deep Learning, Artificial Intelligence, Signal Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="1909.06515" label="1909.06515">
        <attvalues>
          <attvalue for="0" value="Harnessing Indirect Training Data for End-to-End Automatic Speech&#10;  Translation: Tricks of the Trade" />
          <attvalue for="1" value="  For automatic speech translation (AST), end-to-end approaches are&#10;outperformed by cascaded models that transcribe with automatic speech&#10;recognition (ASR), then translate with machine translation (MT). A major cause&#10;of the performance gap is that, while existing AST corpora are small, massive&#10;datasets exist for both the ASR and MT subsystems. In this work, we evaluate&#10;several data augmentation and pretraining approaches for AST, by comparing all&#10;on the same datasets. Simple data augmentation by translating ASR transcripts&#10;proves most effective on the English--French augmented LibriSpeech dataset,&#10;closing the performance gap from 8.2 to 1.4 BLEU, compared to a very strong&#10;cascade that could directly utilize copious ASR and MT data. The same&#10;end-to-end approach plus fine-tuning closes the gap on the English--Romanian&#10;MuST-C dataset from 6.7 to 3.7 BLEU. In addition to these results, we present&#10;practical recommendations for augmentation and pretraining approaches. Finally,&#10;we decrease the performance gap to 0.01 BLEU using a Transformer-based&#10;architecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.05752" label="2105.05752">
        <attvalues>
          <attvalue for="0" value="Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained&#10;  Models into Speech Translation Encoders" />
          <attvalue for="1" value="  Encoder pre-training is promising in end-to-end Speech Translation (ST),&#10;given the fact that speech-to-translation data is scarce. But ST encoders are&#10;not simple instances of Automatic Speech Recognition (ASR) or Machine&#10;Translation (MT) encoders. For example, we find that ASR encoders lack the&#10;global context representation, which is necessary for translation, whereas MT&#10;encoders are not designed to deal with long but locally attentive acoustic&#10;sequences. In this work, we propose a Stacked Acoustic-and-Textual Encoding&#10;(SATE) method for speech translation. Our encoder begins with processing the&#10;acoustic sequence as usual, but later behaves more like an MT encoder for a&#10;global representation of the input sequence. In this way, it is straightforward&#10;to incorporate the pre-trained models into the system. Also, we develop an&#10;adaptor module to alleviate the representation inconsistency between the&#10;pre-trained ASR encoder and MT encoder, and develop a multi-teacher knowledge&#10;distillation method to preserve the pre-training knowledge. Experimental&#10;results on the LibriSpeech En-Fr and MuST-C En-De ST tasks show that our method&#10;achieves state-of-the-art BLEU scores of 18.3 and 25.2. To our knowledge, we&#10;are the first to develop an end-to-end ST system that achieves comparable or&#10;even better BLEU performance than the cascaded ST counterpart when large-scale&#10;ASR and MT data is available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.17358" label="2305.17358">
        <attvalues>
          <attvalue for="0" value="CTC-based Non-autoregressive Speech Translation" />
          <attvalue for="1" value="  Combining end-to-end speech translation (ST) and non-autoregressive (NAR)&#10;generation is promising in language and speech processing for their advantages&#10;of less error propagation and low latency. In this paper, we investigate the&#10;potential of connectionist temporal classification (CTC) for non-autoregressive&#10;speech translation (NAST). In particular, we develop a model consisting of two&#10;encoders that are guided by CTC to predict the source and target texts,&#10;respectively. Introducing CTC into NAST on both language sides has obvious&#10;challenges: 1) the conditional independent generation somewhat breaks the&#10;interdependency among tokens, and 2) the monotonic alignment assumption in&#10;standard CTC does not hold in translation tasks. In response, we develop a&#10;prediction-aware encoding approach and a cross-layer attention approach to&#10;address these issues. We also use curriculum learning to improve convergence of&#10;training. Experiments on the MuST-C ST benchmarks show that our NAST model&#10;achieves an average BLEU score of 29.5 with a speed-up of 5.67$\times$, which&#10;is comparable to the autoregressive counterpart and even outperforms the&#10;previous best result of 0.9 BLEU points.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.18461" label="2407.18461">
        <attvalues>
          <attvalue for="0" value="Enhancing Dysarthric Speech Recognition for Unseen Speakers via&#10;  Prototype-Based Adaptation" />
          <attvalue for="1" value="  Dysarthric speech recognition (DSR) presents a formidable challenge due to&#10;inherent inter-speaker variability, leading to severe performance degradation&#10;when applying DSR models to new dysarthric speakers. Traditional speaker&#10;adaptation methodologies typically involve fine-tuning models for each speaker,&#10;but this strategy is cost-prohibitive and inconvenient for disabled users,&#10;requiring substantial data collection. To address this issue, we introduce a&#10;prototype-based approach that markedly improves DSR performance for unseen&#10;dysarthric speakers without additional fine-tuning. Our method employs a&#10;feature extractor trained with HuBERT to produce per-word prototypes that&#10;encapsulate the characteristics of previously unseen speakers. These prototypes&#10;serve as the basis for classification. Additionally, we incorporate supervised&#10;contrastive learning to refine feature extraction. By enhancing representation&#10;quality, we further improve DSR performance, enabling effective personalized&#10;DSR. We release our code at https://github.com/NKU-HLT/PB-DSR.&#10;" />
          <attvalue for="2" value="&#10;Dysarthria, a speech disorder caused by various factors such as neuropathy, muscle paralysis affecting speech, decreased muscle contractility, or motor incoordination, is frequently associated with conditions like cerebral palsy, Parkinson's disease, and head trauma. For those affected, the limited ability to use keyboards or touchscreens makes speech the most convenient means of interacting with devices like smartphones and smart home devices. However, dysarthria-induced changes in breathing, resonance, pronunciation, and prosody significantly impair the performance of speech recognition systems trained on typical speech \cite{34,27}. Addressing these impairments, dysarthric speech recognition (DSR) technology seeks to bridge the gap for dysarthric speakers, enabling seamless interaction with digital devices. While some studies \cite{18, 19, 15, 16} have focused on developing speaker-independent (SI) DSR models, the inherent variability among dysarthric speakers—due to differences in etiology, age, gender, speaking style, and severity of dysarthria—poses a significant challenge. This variability makes each speaker's speech patterns distinct, leading to notable performance degradation when SI models are applied to unseen dysarthric speakers.&#10;&#10;Consequently, several studies \cite{14,33,9,20,2} have explored fine-tuning speech recognition models with data from the target speaker for personalized DSR. For instance, Shor et al. \cite{14} achieved this by fine-tuning a conventional speech recognition model with data from a specific dysarthric speaker, selectively fine-tuning only a subset of the network layers to avoid overfitting. Takashima et al. \cite{33} introduced a two-stage fine-tuning approach, initially leveraging data from multiple dysarthric speakers to develop an SI model that captures general dysarthric speech patterns, followed by further fine-tuning with target speaker data. Recently, Shahamiri et al. \cite{2} devised a novel DSR system that learns to recognize the word shapes spoken by dysarthric speakers and maps them to words. This system achieved optimal performance through the use of target speaker data adaptation. However, these personalized fine-tuning methods require extensive speech data from the target speaker and incur significant training costs. Moreover, the varying severity of dysarthria over time \cite{11} can compromise the long-term efficacy of personalized DSR models, necessitating continuous data collection and model optimization. This not only escalates training expenses but also poses substantial challenges for dysarthric speakers.&#10;&#10;To address the aforementioned challenges, we introduce a prototype-based DSR (PB-DSR) method that leverages the unique pronunciation error characteristics of dysarthric speakers. Unlike traditional approaches that necessitate extensive model fine-tuning for each new speaker, our method requires only a minimal dataset to effectively adapt to individual speech patterns. Pronunciation errors, such as phoneme deletion, substitution, insertion, and distortion, are consistent within speakers. Previous studies \cite{28,29} attempted to address these errors by creating adaptive pronunciation dictionaries for each speaker, based on the analysis of error frequencies. However, these solutions were limited to the most common errors and did not adequately address the misrecognition of severe breath sounds and background noise as phonemes. In contrast, our approach targets word-level pronunciation errors, treating the speech for each word by an individual as a unique class that requires similarity rather than identical matches. Inspired by the principles of prototypical networks for few-shot learning \cite{3}, our PB-DSR method utilizes a feature extractor to create per-word prototypes from only few-shot samples, enabling rapid adaptation to the unique speech patterns of each speaker. We utilize the pre-trained HuBERT \cite{7} model, renowned for its general speech recognition capabilities, to extract speech features. To further refine its accuracy for dysarthric speech, HuBERT is initially fine-tuned with a specialized dysarthric speech dataset. Moreover, we integrate supervised contrastive learning (SCL) \cite{10} to enhance feature extraction, adopting a strategy from contrastive learning \cite{23} that optimizes feature representations by minimizing intra-class distances while maximizing inter-class separations, a technique proven effective in various speech tasks \cite{22,32,31,35}.&#10;&#10;To empirically validate the efficacy of our PB-DSR method, we conduct experiments with the UASpeech dataset \cite{5}. The results demonstrate notable improvements in speech recognition for unseen dysarthric speakers. Specifically, the PB-DSR method achieves an average absolute reduction in Word Error Rate (WER) of 15.59\% compared to its Speaker-Independent (SI) counterpart. Furthermore, incorporating SCL loss into our DSR model training to refine feature extraction leads to an additional 1.21\% reduction in WER.&#10;&#10;The main contributions of this work are as follows:&#10;&#10;\begin{itemize} &#10; \item We propose a prototype-based DSR approach that offers a rapid and effective method for improving the recognition of speech from unseen dysarthric speakers. &#10; \item We combine CTC loss with SCL loss to train the DSR model to improve performance by learning better feature representations.&#10; \item We have demonstrated the effectiveness of the proposed methods on the UASpeech dataset.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Engineering, Speaker Adaptation, Deep Learning, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2204.01670" label="2204.01670">
        <attvalues>
          <attvalue for="0" value="Cross-lingual Self-Supervised Speech Representations for Improved&#10;  Dysarthric Speech Recognition" />
          <attvalue for="1" value="  State-of-the-art automatic speech recognition (ASR) systems perform well on&#10;healthy speech. However, the performance on impaired speech still remains an&#10;issue. The current study explores the usefulness of using Wav2Vec&#10;self-supervised speech representations as features for training an ASR system&#10;for dysarthric speech. Dysarthric speech recognition is particularly difficult&#10;as several aspects of speech such as articulation, prosody and phonation can be&#10;impaired. Specifically, we train an acoustic model with features extracted from&#10;Wav2Vec, Hubert, and the cross-lingual XLSR model. Results suggest that speech&#10;representations pretrained on large unlabelled data can improve word error rate&#10;(WER) performance. In particular, features from the multilingual model led to&#10;lower WERs than filterbanks (Fbank) or models trained on a single language.&#10;Improvements were observed in English speakers with cerebral palsy caused&#10;dysarthria (UASpeech corpus), Spanish speakers with Parkinsonian dysarthria&#10;(PC-GITA corpus) and Italian speakers with paralysis-based dysarthria (EasyCall&#10;corpus). Compared to using Fbank features, XLSR-based features reduced WERs by&#10;6.8%, 22.0%, and 7.0% for the UASpeech, PC-GITA, and EasyCall corpus,&#10;respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.13511" label="1907.13511">
        <attvalues>
          <attvalue for="0" value="Personalizing ASR for Dysarthric and Accented Speech with Limited Data" />
          <attvalue for="1" value="  Automatic speech recognition (ASR) systems have dramatically improved over&#10;the last few years. ASR systems are most often trained from 'typical' speech,&#10;which means that underrepresented groups don't experience the same level of&#10;improvement. In this paper, we present and evaluate finetuning techniques to&#10;improve ASR for users with non-standard speech. We focus on two types of&#10;non-standard speech: speech from people with amyotrophic lateral sclerosis&#10;(ALS) and accented speech. We train personalized models that achieve 62% and&#10;35% relative WER improvement on these two groups, bringing the absolute WER for&#10;ALS speakers, on a test set of message bank phrases, down to 10% for mild&#10;dysarthria and 20% for more serious dysarthria. We show that 71% of the&#10;improvement comes from only 5 minutes of training data. Finetuning a particular&#10;subset of layers (with many fewer parameters) often gives better results than&#10;finetuning the entire model. This is the first step towards building state of&#10;the art ASR models for dysarthric speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.01686" label="2011.01686">
        <attvalues>
          <attvalue for="0" value="Improved End-to-End Dysarthric Speech Recognition via Meta-learning&#10;  Based Model Re-initialization" />
          <attvalue for="1" value="  Dysarthric speech recognition is a challenging task as dysarthric data is&#10;limited and its acoustics deviate significantly from normal speech. Model-based&#10;speaker adaptation is a promising method by using the limited dysarthric speech&#10;to fine-tune a base model that has been pre-trained from large amounts of&#10;normal speech to obtain speaker-dependent models. However, statistic&#10;distribution mismatches between the normal and dysarthric speech data limit the&#10;adaptation performance of the base model. To address this problem, we propose&#10;to re-initialize the base model via meta-learning to obtain a better model&#10;initialization. Specifically, we focus on end-to-end models and extend the&#10;model-agnostic meta learning (MAML) and Reptile algorithms to meta update the&#10;base model by repeatedly simulating adaptation to different dysarthric&#10;speakers. As a result, the re-initialized model acquires dysarthric speech&#10;knowledge and learns how to perform fast adaptation to unseen dysarthric&#10;speakers with improved performance. Experimental results on UASpeech dataset&#10;show that the best model with proposed methods achieves 54.2% and 7.6% relative&#10;word error rate reduction compared with the base model without finetuning and&#10;the model directly fine-tuned from the base model, respectively, and it is&#10;comparable with the state-of-the-art hybrid DNN-HMM model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.00089" label="2211.00089">
        <attvalues>
          <attvalue for="0" value="An analysis of degenerating speech due to progressive dysarthria on ASR&#10;  performance" />
          <attvalue for="1" value="  Although personalized automatic speech recognition (ASR) models have recently&#10;been designed to recognize even severely impaired speech, model performance may&#10;degrade over time for persons with degenerating speech. The aims of this study&#10;were to (1) analyze the change of performance of ASR over time in individuals&#10;with degrading speech, and (2) explore mitigation strategies to optimize&#10;recognition throughout disease progression. Speech was recorded by four&#10;individuals with degrading speech due to amyotrophic lateral sclerosis (ALS).&#10;Word error rates (WER) across recording sessions were computed for three ASR&#10;models: Unadapted Speaker Independent (U-SI), Adapted Speaker Independent&#10;(A-SI), and Adapted Speaker Dependent (A-SD or personalized). The performance&#10;of all three models degraded significantly over time as speech became more&#10;impaired, but the performance of the A-SD model improved markedly when it was&#10;updated with recordings from the severe stages of speech progression. Recording&#10;additional utterances early in the disease before speech degraded significantly&#10;did not improve the performance of A-SD models. Overall, our findings emphasize&#10;the importance of continuous recording (and model retraining) when providing&#10;personalized models for individuals with progressive speech impairments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.11362" label="2004.11362">
        <attvalues>
          <attvalue for="0" value="Supervised Contrastive Learning" />
          <attvalue for="1" value="  Contrastive learning applied to self-supervised representation learning has&#10;seen a resurgence in recent years, leading to state of the art performance in&#10;the unsupervised training of deep image models. Modern batch contrastive&#10;approaches subsume or significantly outperform traditional contrastive losses&#10;such as triplet, max-margin and the N-pairs loss. In this work, we extend the&#10;self-supervised batch contrastive approach to the fully-supervised setting,&#10;allowing us to effectively leverage label information. Clusters of points&#10;belonging to the same class are pulled together in embedding space, while&#10;simultaneously pushing apart clusters of samples from different classes. We&#10;analyze two possible versions of the supervised contrastive (SupCon) loss,&#10;identifying the best-performing formulation of the loss. On ResNet-200, we&#10;achieve top-1 accuracy of 81.4% on the ImageNet dataset, which is 0.8% above&#10;the best number reported for this architecture. We show consistent&#10;outperformance over cross-entropy on other datasets and two ResNet variants.&#10;The loss shows benefits for robustness to natural corruptions and is more&#10;stable to hyperparameter settings such as optimizers and data augmentations.&#10;Our loss function is simple to implement, and reference TensorFlow code is&#10;released at https://t.ly/supcon.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.16485" label="2308.16485">
        <attvalues>
          <attvalue for="0" value="Supervised Contrastive Learning with Nearest Neighbor Search for Speech&#10;  Emotion Recognition" />
          <attvalue for="1" value="  Speech Emotion Recognition (SER) is a challenging task due to limited data&#10;and blurred boundaries of certain emotions. In this paper, we present a&#10;comprehensive approach to improve the SER performance throughout the model&#10;lifecycle, including pre-training, fine-tuning, and inference stages. To&#10;address the data scarcity issue, we utilize a pre-trained model, wav2vec2.0.&#10;During fine-tuning, we propose a novel loss function that combines&#10;cross-entropy loss with supervised contrastive learning loss to improve the&#10;model's discriminative ability. This approach increases the inter-class&#10;distances and decreases the intra-class distances, mitigating the issue of&#10;blurred boundaries. Finally, to leverage the improved distances, we propose an&#10;interpolation method at the inference stage that combines the model prediction&#10;with the output from a k-nearest neighbors model. Our experiments on IEMOCAP&#10;demonstrate that our proposed methods outperform current state-of-the-art&#10;results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.01157" label="1910.01157">
        <attvalues>
          <attvalue for="0" value="Cracking the Contextual Commonsense Code: Understanding Commonsense&#10;  Reasoning Aptitude of Deep Contextual Representations" />
          <attvalue for="1" value="  Pretrained deep contextual representations have advanced the state-of-the-art&#10;on various commonsense NLP tasks, but we lack a concrete understanding of the&#10;capability of these models. Thus, we investigate and challenge several aspects&#10;of BERT's commonsense representation abilities. First, we probe BERT's ability&#10;to classify various object attributes, demonstrating that BERT shows a strong&#10;ability in encoding various commonsense features in its embedding space, but is&#10;still deficient in many areas. Next, we show that, by augmenting BERT's&#10;pretraining data with additional data related to the deficient attributes, we&#10;are able to improve performance on a downstream commonsense reasoning task&#10;while using a minimal amount of data. Finally, we develop a method of&#10;fine-tuning knowledge graphs embeddings alongside BERT and show the continued&#10;importance of explicit knowledge graphs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Similar to our attribute classification investigation, several other works have used applied semantic norm datasets to computational linguistics \cite{Agirre2009ASO, Bruni2012DistributionalSI, Kiela2016VirtualEA}. Methodologically, our work is most similar to \cite{lucy2017}, who use a logistic regression classifier to determine fit score of word type embeddings based on leave-one-out verification. \cite{forbes2019neural} investigates the commonsense aptitude of contextual representations. However, our work differs in several important ways: 1) we connect our analysis to downstream reasoning aptitude, underscoring the importance of the semantic norm analysis, and 2) we introduce various ways of improving BERT, motivated by our analysis.&#10;&#10;In contemporaneous work, various research has been done in improving upon BERT's performance through knowledge augmentation. Implicitly, \cite{Sun2019HowTF} explores fine-tuning on in-domain data, similarly to our fine-tuning on the RACE dataset \cite{Lai2017RACELR}. They discover an increase in accuracy that is especially prevalent over smaller datasets. Our work differs in that we do not fine-tune on the entire domain data, but rather select a smaller subset of data to fine-tune on. Other work extends BERT to domains where its original training data does not suffice \cite{Beltagy2019SciBERTPC, Lee2019BioBERTAP}. RoBERTa \cite{Liu2019RoBERTaAR} also pretrains on RACE, and finds increased results through altering several of BERT's pretraining tasks, claiming that BERT was extensively undertrained. Explicitly, ERNIE, \cite{Sun2019ERNIEER} introduces information to contextual representations during pretraining. ERNIE uses word-level fusion between the contextual representation and explicit information.&#10;&#10;Prior work has developed several benchmark datasets to assess commonsense knowledge of NLP models \cite{Roemmele2011,Mostafazadeh2016ACA,zhang-etal-2017-ordinal,zellers2018swagaf,zellers2019hellaswag,Ostermann2018MCScriptAN,mcscript2,Sakaguchi2019WINOGRANDEAA}.&#10;These benchmarks are typically posed as question answering, but we use semantic norm datasets to specifically assess BERT's ability to represent grounded attributes.&#10;Further, we demonstrate that these abstract attributes can be used to enhance BERT's representations and improve the downstream performance.&#10;&#10;" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Cognitive Science, Commonsense NLP Tasks, Knowledge Graph Embeddings, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2006.07398" label="2006.07398">
        <attvalues>
          <attvalue for="0" value="Evaluating a Multi-sense Definition Generation Model for Multiple&#10;  Languages" />
          <attvalue for="1" value="  Most prior work on definition modeling has not accounted for polysemy, or has&#10;done so by considering definition modeling for a target word in a given&#10;context. In contrast, in this study, we propose a context-agnostic approach to&#10;definition modeling, based on multi-sense word embeddings, that is capable of&#10;generating multiple definitions for a target word. In further, contrast to most&#10;prior work, which has primarily focused on English, we evaluate our proposed&#10;approach on fifteen different datasets covering nine languages from several&#10;language families. To evaluate our approach we consider several variations of&#10;BLEU. Our results demonstrate that our proposed multi-sense model outperforms a&#10;single-sense model on all fifteen datasets.&#10;" />
          <attvalue for="2" value="&#10;&#10;The advent of pre-trained distributed word representations, such as&#10;\cite{mikolov2013distributed}, led to improvements in a wide range of&#10;natural language processing (NLP) tasks. One limitation of such word&#10;embeddings, however, is that they conflate all of a word's senses into&#10;a single vector. Subsequent work has considered approaches to learn&#10;multi-sense embeddings, in which a word is represented by multiple&#10;vectors, each corresponding to a sense&#10;\cite{adagram,lee2017muse}. More recent work has considered&#10;contextualized word embeddings, such as \cite{devlin2018bert}, which&#10;provide a representation of the meaning of a word in a given context.&#10;&#10;Definition modelling, recently introduced by&#10;\cite{noraset2017definition}, is a specific type of language modelling&#10;which aims to generate dictionary-style definitions for a given&#10;word. Definition modelling can provide a transparent interpretation of&#10;the information represented in word embeddings, and has the potential&#10;to be applied to generate definitions for newly-emerged words that are&#10;not yet recorded in dictionaries.&#10;&#10;The approach to definition modelling of \cite{noraset2017definition}&#10;is based on a recurrent neural network (RNN) language model, which is&#10;conditioned on a word embedding for the target word to be defined,&#10;specifically pre-trained word2vec \cite{mikolov2013distributed}&#10;embeddings. As such, this model does not account for polysemy. To&#10;address this limitation, a number of studies have proposed&#10;context-aware definition generation models&#10;\cite{ni2017learning,gadetsky2018conditional,ishiwatari2018learning,mickus-etal-2019-mark,chang2019does}. In&#10;all of these approaches, the models generate a definition&#10;corresponding to the usage of a given target word in a given context.&#10;&#10;In contrast, in this paper we propose a context-agnostic multi-sense&#10;definition generation model. Given a target word type (i.e., without&#10;its usage in a specific context) the proposed model generates multiple&#10;definitions corresponding to different senses of that word. Our&#10;proposed model is an extension of \cite{noraset2017definition} that&#10;incorporates pre-trained multi-sense embeddings. As such, the&#10;definitions that are generated are based on the senses learned by the&#10;embedding model on a background corpus, and reflect the usage of words&#10;in that corpus. Under this setup --- i.e., generating multiple&#10;definitions for each word corresponding to senses present in a corpus&#10;--- the proposed definition generation model has the potential to&#10;generate partial dictionary entries. In order to train the proposed&#10;model, pre-trained sense vectors for a word need to be matched to&#10;reference definitions for that word. We consider two approaches to&#10;this matching based on cosine similarity between sense vectors and&#10;reference definitions.&#10;&#10;Recently, \cite{zhu2019multi} propose a multi-sense model for&#10;generating definitions for the various senses of a target word. This&#10;model utilizes word embeddings and coarse-grained atom embeddings to&#10;represent senses \cite{arora2018linearTRIM}, in which atoms are shared&#10;across words. In contrast, we only rely on fine-grained multi-sense&#10;embeddings. To match sense vectors to reference definitions during&#10;training, \cite{zhu2019multi} propose a neural approach, and also&#10;consider a heuristic-based approach that incorporates cosine&#10;similarity between senses and definitions. Our proposed approach to&#10;this matching is similar to their heuristic-based approach, although&#10;we explore two variations of this method. Furthermore,&#10;\cite{zhu2019multi} only consider English for evaluation, whereas we&#10;consider fifteen datasets covering nine languages.&#10;&#10;Following \cite{zhu2019multi} we evaluate our proposed model using&#10;variations of BLEU \cite{papineni2002bleu}. We evaluate our model on&#10;fifteen datasets covering nine languages from several families. Our&#10;experimental results show that, for every language and dataset&#10;considered, our proposed approach outperforms the benchmark approach&#10;of \cite{noraset2017definition} which does not model polysemy.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Definition Modeling, Computer Science, Linguistics, Word Embeddings, Artificial Intelligence, Natural Language Processing, Multilingual Evaluation" />
        </attvalues>
      </node>
      <node id="1502.07257" label="1502.07257">
        <attvalues>
          <attvalue for="0" value="Breaking Sticks and Ambiguities with Adaptive Skip-gram" />
          <attvalue for="1" value="  Recently proposed Skip-gram model is a powerful method for learning&#10;high-dimensional word representations that capture rich semantic relationships&#10;between words. However, Skip-gram as well as most prior work on learning word&#10;representations does not take into account word ambiguity and maintain only&#10;single representation per word. Although a number of Skip-gram modifications&#10;were proposed to overcome this limitation and learn multi-prototype word&#10;representations, they either require a known number of word meanings or learn&#10;them using greedy heuristic approaches. In this paper we propose the Adaptive&#10;Skip-gram model which is a nonparametric Bayesian extension of Skip-gram&#10;capable to automatically learn the required number of representations for all&#10;words at desired semantic resolution. We derive efficient online variational&#10;learning algorithm for the model and empirically demonstrate its efficiency on&#10;word-sense induction task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.04601" label="1704.04601">
        <attvalues>
          <attvalue for="0" value="MUSE: Modularizing Unsupervised Sense Embeddings" />
          <attvalue for="1" value="  This paper proposes to address the word sense ambiguity issue in an&#10;unsupervised manner, where word sense representations are learned along a word&#10;sense selection mechanism given contexts. Prior work focused on designing a&#10;single model to deliver both mechanisms, and thus suffered from either&#10;coarse-grained representation learning or inefficient sense selection. The&#10;proposed modular approach, MUSE, implements flexible modules to optimize&#10;distinct mechanisms, achieving the first purely sense-level representation&#10;learning system with linear-time sense selection. We leverage reinforcement&#10;learning to enable joint training on the proposed modules, and introduce&#10;various exploration techniques on sense selection for better robustness. The&#10;experiments on benchmark data show that the proposed approach achieves the&#10;state-of-the-art performance on synonym selection as well as on contextual word&#10;similarities in terms of MaxSimC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.00394" label="1612.00394">
        <attvalues>
          <attvalue for="0" value="Definition Modeling: Learning to define word embeddings in natural&#10;  language" />
          <attvalue for="1" value="  Distributed representations of words have been shown to capture lexical&#10;semantics, as demonstrated by their effectiveness in word similarity and&#10;analogical relation tasks. But, these tasks only evaluate lexical semantics&#10;indirectly. In this paper, we study whether it is possible to utilize&#10;distributed representations to generate dictionary definitions of words, as a&#10;more direct and transparent representation of the embeddings' semantics. We&#10;introduce definition modeling, the task of generating a definition for a given&#10;word and its embedding. We present several definition model architectures based&#10;on recurrent neural networks, and experiment with the models over multiple data&#10;sets. Our results show that a model that controls dependencies between the word&#10;being defined and the definition words performs significantly better, and that&#10;a character-level convolution layer designed to leverage morphology can&#10;complement word-level embeddings. Finally, an error analysis suggests that the&#10;errors made by a definition model may provide insight into the shortcomings of&#10;word embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.09254" label="1709.09254">
        <attvalues>
          <attvalue for="0" value="Learning to Explain Non-Standard English Words and Phrases" />
          <attvalue for="1" value="  We describe a data-driven approach for automatically explaining new,&#10;non-standard English expressions in a given sentence, building on a large&#10;dataset that includes 15 years of crowdsourced examples from&#10;UrbanDictionary.com. Unlike prior studies that focus on matching keywords from&#10;a slang dictionary, we investigate the possibility of learning a neural&#10;sequence-to-sequence model that generates explanations of unseen non-standard&#10;English expressions given context. We propose a dual encoder approach---a&#10;word-level encoder learns the representation of context, and a second&#10;character-level encoder to learn the hidden representation of the target&#10;non-standard expression. Our model can produce reasonable definitions of new&#10;non-standard English expressions given their context with certain confidence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.10090" label="1806.10090">
        <attvalues>
          <attvalue for="0" value="Conditional Generators of Words Definitions" />
          <attvalue for="1" value="  We explore recently introduced definition modeling technique that provided&#10;the tool for evaluation of different distributed vector representations of&#10;words through modeling dictionary definitions of words. In this work, we study&#10;the problem of word ambiguities in definition modeling and propose a possible&#10;solution by employing latent variable modeling and soft attention mechanisms.&#10;Our quantitative and qualitative evaluation and analysis of the model shows&#10;that taking into account words ambiguity and polysemy leads to performance&#10;improvement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.05715" label="1911.05715">
        <attvalues>
          <attvalue for="0" value="Mark my Word: A Sequence-to-Sequence Approach to Definition Modeling" />
          <attvalue for="1" value="  Defining words in a textual context is a useful task both for practical&#10;purposes and for gaining insight into distributed word representations.&#10;Building on the distributional hypothesis, we argue here that the most natural&#10;formalization of definition modeling is to treat it as a sequence-to-sequence&#10;task, rather than a word-to-sequence task: given an input sequence with a&#10;highlighted word, generate a contextually appropriate definition for it. We&#10;implement this approach in a Transformer-based sequence-to-sequence model. Our&#10;proposal allows to train contextualization and definition generation in an&#10;end-to-end fashion, which is a conceptual improvement over earlier works. We&#10;achieve state-of-the-art results both in contextual and non-contextual&#10;definition modeling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.09483" label="1909.09483">
        <attvalues>
          <attvalue for="0" value="Multi-sense Definition Modeling using Word Sense Decompositions" />
          <attvalue for="1" value="  Word embeddings capture syntactic and semantic information about words.&#10;Definition modeling aims to make the semantic content in each embedding&#10;explicit, by outputting a natural language definition based on the embedding.&#10;However, existing definition models are limited in their ability to generate&#10;accurate definitions for different senses of the same word. In this paper, we&#10;introduce a new method that enables definition modeling for multiple senses. We&#10;show how a Gumble-Softmax approach outperforms baselines at matching&#10;sense-specific embeddings to definitions during training. In experiments, our&#10;multi-sense definition model improves recall over a state-of-the-art&#10;single-sense definition model by a factor of three, without harming precision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1601.03764" label="1601.03764">
        <attvalues>
          <attvalue for="0" value="Linear Algebraic Structure of Word Senses, with Applications to Polysemy" />
          <attvalue for="1" value="  Word embeddings are ubiquitous in NLP and information retrieval, but it is&#10;unclear what they represent when the word is polysemous. Here it is shown that&#10;multiple word senses reside in linear superposition within the word embedding&#10;and simple sparse coding can recover vectors that approximately capture the&#10;senses. The success of our approach, which applies to several embedding&#10;methods, is mathematically explained using a variant of the random walk on&#10;discourses model (Arora et al., 2016). A novel aspect of our technique is that&#10;each extracted word sense is accompanied by one of about 2000 &quot;discourse atoms&quot;&#10;that gives a succinct description of which other words co-occur with that word&#10;sense. Discourse atoms can be of independent interest, and make the method&#10;potentially more useful. Empirical tests are used to verify and support the&#10;theory.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.01903" label="2404.01903">
        <attvalues>
          <attvalue for="0" value="Understanding How CodeLLMs (Mis)Predict Types with Activation Steering" />
          <attvalue for="1" value="  CodeLLMs are transforming software development as we know it. This is&#10;especially true for tasks where rule-based approaches fall short, like type&#10;prediction. The type prediction task consists in adding a new type annotation&#10;to a partially typed program, such that the resulting program is closer to&#10;being fully typed. The intractability of rule-based approaches and high cost of&#10;manual annotation make CodeLLMs an attractive solution to the problem. However,&#10;CodeLLMs are still far from being deployed on the large-scale due to doubts&#10;surrounding their reliability.&#10;  To shed some light on how CodeLLMs approach type prediction, we investigate&#10;what happens when a model mispredicts a type. We show that by applying&#10;semantics-preserving edits to code, CodeLLMs are eventually misled into&#10;mispredicting type annotations. However, by leveraging activation steering we&#10;are able to &quot;steer&quot; the model back to the correct prediction, making models&#10;more robust against semantically irrelevant prompt features. We show that&#10;steering achieves comparable performance to fine-tuning directly on the type&#10;prediction task. Furthermore, we find that steering vectors computed from&#10;Python code are effective at correcting TypeScript mispredictions, and vice&#10;versa. To our knowledge, this is the first evidence of its kind to suggest that&#10;CodeLLMs learn task representations that transfer across languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models trained on code, or CodeLLMs, are one of the most successful applications of LLMs to date. They are the foundation of both developer tools (e.g., \cite{tabnine,github-copilot}) and reasoning agents (e.g., \cite{zelikman2024parsel, yang2024if,hu:codebotler,alphacode}), capable of performing a variety of programming tasks across several programming languages. This paper focuses on type prediction for gradually typed programming languages, specifically Python and TypeScript. Given a partially typed program $p$ written in a gradually typed language, the type prediction task is defined as follows~\cite{migeed:decidable}:&#10;&#10;\begin{quote}&#10;Choose an untyped variable binding $var \in p$, predict a type annotation $var : T$, and insert the annotation back into the program to get a new program $p'$. The task is successful if $p'$ continues to pass the type-checker. &#10;\end{quote}&#10;&#10;A gradually typed programming language allows programs to freely mix typed and untyped code, giving programmers more flexibility than traditional static typing affords~\cite{siek:gtlc,th:migration}. However, this flexibility comes with a downside: at runtime, untyped code still needs to type-correct for the program to run correctly. With omitted or weak type annotations, type errors may not be caught until program execution.&#10;&#10;Automating type prediction is important for both scientific and practical reasons. From a practical perspective, well-known companies like Airbnb, Dropbox, Slack, Netflix, and many others have each taken several years to manually add type annotations to their multi-million line gradually typed codebases~\cite{airbnb:ts-migrate,dropbox:mypy,slack:ts,heap:ts,netflix:ts,abacus:ts,quip:ts,stripe:sorbet}. Automated gradual type prediction would save significant effort and resources. &#10;&#10;From a scientific perspective, type prediction is intractable, and rule-based approaches have not been scaled up to contemporary languages (\ref{background-and-related-work}). Because of this, prior work has explored leveraging CodeLLMs for type prediction. However, CodeLLM performance on the task is far from being able to automate large-scale type annotation efforts. In particular, we show that CodeLLM type predictions are sensitive to small semantics-preserving changes in prompts like renaming variables, which limits their practical utility. &#10;&#10;The sensitivity of CodeLLMs towards trivial prompt changes begs the question of whether CodeLLMs approach type prediction in the ``correct'' way--that is, by inferring the type of a variable based on the operations performed over it. Alternatively, CodeLLMs could be relying on text features to make spurious predictions--for example, if a variable name is n, then its type is integer. We term these features ``distractors''. As CodeLLM assistants are increasingly integrated into our workflows, understanding whether their predictions are principled or spurious is fundamental for security and large-scale adoption of these systems.&#10;&#10;To answer the question of how CodeLLMs approach type prediction, we design the following experimental setting. &#10;&#10;Given a type prediction prompt that the CodeLLM predicts correctly, we apply edits until the model output fails type checking. We construct our edits such that they do not matter for type prediction. To do so, we take inspiration from mutation testing~\cite{demillo:test-data-selection}. We present a variation of mutation testing that constructs minimal, semantics-preserving edits that trigger CodeLLM mispredictions. The nature of code allows us to construct these edits in a sound and scaleable way (\ref{constructing-pairs}). &#10;&#10;Since the model could originally predict the prompt, but fails with our trivial edits, this could mean that CodeLLM type prediction is spurious and based on distractors. However, our findings suggest a different story: that mispredictions happen because the model fails to identify the inference task as a type prediction task. This likely occurs because our edits shift the prompt outside the distribution of a model's training data. We show that by editing intermediate computations (activations) of the model, we can recover performance on type prediction despite the presence of distractors. &#10;&#10;We employ activation steering~\cite{li2024inference,rimsky2023steering,turner2023activation, subramani2022extracting} to steer model activations toward the target task of type prediction. Activation steering is an inference-time model editing technique that modifies the activations of a model using steering vectors to steer model behavior toward desired outcomes.&#10;&#10;Finally, we show that steering transfers across programming languages: the steering vectors we construct from TypeScript code are very effective at steering Python type predictions, and vice versa. This suggests that there exists a robust, generalizable mechanism for type prediction in CodeLLMs which, when activated, can correctly predict types despite the presence of distractors. From this, we conclude that CodeLLMs are learning task-solving mechanisms that transfer across languages (\ref{section:lang-transfer}). To our knowledge, this is the first empirical evidence of its kind that identifies the occurrence of language transfer within LLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Reliability, Computer Science, Type Prediction Tasks, Linguistics, Code Prediction Models, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2212.10561" label="2212.10561">
        <attvalues>
          <attvalue for="0" value="Parsel: Algorithmic Reasoning with Language Models by Composing&#10;  Decompositions" />
          <attvalue for="1" value="  Despite recent success in large language model (LLM) reasoning, LLMs struggle&#10;with hierarchical multi-step reasoning tasks like generating complex programs.&#10;For these tasks, humans often start with a high-level algorithmic design and&#10;implement each part gradually. We introduce Parsel, a framework enabling&#10;automatic implementation and validation of complex algorithms with code LLMs.&#10;With Parsel, we automatically decompose algorithmic tasks into hierarchical&#10;natural language function descriptions and then search over combinations of&#10;possible function implementations using tests. We show that Parsel can be used&#10;across domains requiring hierarchical reasoning, including program synthesis&#10;and robotic planning. We find that, using Parsel, LLMs solve more&#10;competition-level problems in the APPS dataset, resulting in pass rates over&#10;75\% higher than prior results from directly sampling AlphaCode and Codex,&#10;while often using a smaller sample budget. Moreover, with automatically&#10;generated tests, we find that Parsel can improve the state-of-the-art pass@1&#10;performance on HumanEval from 67\% to 85\%. We also find that LLM-generated&#10;robotic plans using Parsel are more than twice as likely to be considered&#10;accurate than directly generated plans. Lastly, we explore how Parsel addresses&#10;LLM limitations and discuss how Parsel may be useful for human programmers. We&#10;release our code at https://github.com/ezelikman/parsel&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.11183" label="2311.11183">
        <attvalues>
          <attvalue for="0" value="Deploying and Evaluating LLMs to Program Service Mobile Robots" />
          <attvalue for="1" value="  Recent advancements in large language models (LLMs) have spurred interest in&#10;using them for generating robot programs from natural language, with promising&#10;initial results. We investigate the use of LLMs to generate programs for&#10;service mobile robots leveraging mobility, perception, and human interaction&#10;skills, and where accurate sequencing and ordering of actions is crucial for&#10;success. We contribute CodeBotler, an open-source robot-agnostic tool to&#10;program service mobile robots from natural language, and RoboEval, a benchmark&#10;for evaluating LLMs' capabilities of generating programs to complete service&#10;robot tasks. CodeBotler performs program generation via few-shot prompting of&#10;LLMs with an embedded domain-specific language (eDSL) in Python, and leverages&#10;skill abstractions to deploy generated programs on any general-purpose mobile&#10;robot. RoboEval evaluates the correctness of generated programs by checking&#10;execution traces starting with multiple initial states, and checking whether&#10;the traces satisfy temporal logic properties that encode correctness for each&#10;task. RoboEval also includes multiple prompts per task to test for the&#10;robustness of program generation. We evaluate several popular state-of-the-art&#10;LLMs with the RoboEval benchmark, and perform a thorough analysis of the modes&#10;of failures, resulting in a taxonomy that highlights common pitfalls of LLMs at&#10;generating robot programs. We release our code and benchmark at&#10;https://amrl.cs.utexas.edu/codebotler/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.07814" label="2203.07814">
        <attvalues>
          <attvalue for="0" value="Competition-Level Code Generation with AlphaCode" />
          <attvalue for="1" value="  Programming is a powerful and ubiquitous problem-solving tool. Developing&#10;systems that can assist programmers or even generate programs independently&#10;could make programming more productive and accessible, yet so far incorporating&#10;innovations in AI has proven challenging. Recent large-scale language models&#10;have demonstrated an impressive ability to generate code, and are now able to&#10;complete simple programming tasks. However, these models still perform poorly&#10;when evaluated on more complex, unseen problems that require problem-solving&#10;skills beyond simply translating instructions into code. For example,&#10;competitive programming problems which require an understanding of algorithms&#10;and complex natural language remain extremely challenging. To address this gap,&#10;we introduce AlphaCode, a system for code generation that can create novel&#10;solutions to these problems that require deeper reasoning. In simulated&#10;evaluations on recent programming competitions on the Codeforces platform,&#10;AlphaCode achieved on average a ranking of top 54.3% in competitions with more&#10;than 5,000 participants. We found that three key components were critical to&#10;achieve good and reliable performance: (1) an extensive and clean competitive&#10;programming dataset for training and evaluation, (2) large and&#10;efficient-to-sample transformer-based architectures, and (3) large-scale model&#10;sampling to explore the search space, followed by filtering based on program&#10;behavior to a small set of submissions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.10248" label="2308.10248">
        <attvalues>
          <attvalue for="0" value="Activation Addition: Steering Language Models Without Optimization" />
          <attvalue for="1" value="  Reliably controlling the behavior of large language models is a pressing open&#10;problem. Existing methods include supervised finetuning, reinforcement learning&#10;from human feedback, prompt engineering and guided decoding. We instead&#10;investigate activation engineering: modifying activations at inference-time to&#10;predictably alter model behavior. We bias the forward pass with a 'steering&#10;vector' implicitly specified through natural language. Past work learned these&#10;steering vectors; our Activation Addition (ActAdd) method instead computes them&#10;by taking activation differences resulting from pairs of prompts. We&#10;demonstrate ActAdd on a range of LLMs (LLaMA-3, OPT, GPT-2, and GPT-J),&#10;obtaining SOTA on detoxification and negative-to-positive sentiment control.&#10;Our approach yields inference-time control over high-level properties of output&#10;like topic and sentiment while preserving performance on off-target tasks.&#10;ActAdd takes far less compute and implementation effort than finetuning or&#10;RLHF, allows users control through natural language, and its computational&#10;overhead (as a fraction of inference time) appears stable or improving over&#10;increasing model size.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.16498" label="2210.16498">
        <attvalues>
          <attvalue for="0" value="Articulatory Representation Learning Via Joint Factor Analysis and&#10;  Neural Matrix Factorization" />
          <attvalue for="1" value="  Articulatory representation learning is the fundamental research in modeling&#10;neural speech production system. Our previous work has established a deep&#10;paradigm to decompose the articulatory kinematics data into gestures, which&#10;explicitly model the phonological and linguistic structure encoded with human&#10;speech production mechanism, and corresponding gestural scores. We continue&#10;with this line of work by raising two concerns: (1) The articulators are&#10;entangled together in the original algorithm such that some of the articulators&#10;do not leverage effective moving patterns, which limits the interpretability of&#10;both gestures and gestural scores; (2) The EMA data is sparsely sampled from&#10;articulators, which limits the intelligibility of learned representations. In&#10;this work, we propose a novel articulatory representation decomposition&#10;algorithm that takes the advantage of guided factor analysis to derive the&#10;articulatory-specific factors and factor scores. A neural convolutive matrix&#10;factorization algorithm is then employed on the factor scores to derive the new&#10;gestures and gestural scores. We experiment with the rtMRI corpus that captures&#10;the fine-grained vocal tract contours. Both subjective and objective evaluation&#10;results suggest that the newly proposed system delivers the articulatory&#10;representations that are intelligible, generalizable, efficient and&#10;interpretable.&#10;" />
          <attvalue for="2" value="&#10;&#10;The mainstream research of deep speech representation learning is to develop a human-like neural speech processing system. However, the gap between human and machine intelligence is not that straightforward to be filled in the post-transformer era~\cite{karita2019comparative, lecun2022autonomous, mohamed2022self_speech, speech_representation}. Inspired by the concept of autonomous human intelligence system~\cite{lecun2022autonomous}, we aim to develop a human speech processing system that is interpretable, efficient and effective. Basically there are two types of such systems. The first one is the neural speech perception system~\cite{1980_speech_perception, perception, perception_production}, which is the theoretical model for most of the current neural networks. The second one is the neural speech production system~\cite{fant1970_production, perception_production} which learns the speech representations in a way that simulates the human speech production process. We focus on the second line. Directly modeling speech production systems from raw speech is fairly difficult and little work has been explored in this direction. To simplify the problem and to make initiative attempts, we model the speech production system and derive the speech representations from the articulatory signal, which we call articulatory representation learning. &#10;&#10;Articulatory representation learning is defined over the framework of articulatory phonology~\cite{articulatory-phonology}, which models the relation between phonological representations as a set of discrete units called gestures, and the variability in time that derives from variation in the activation of the gestures in real-time: the magnitude of their activation, and the temporal intervals of activation as represented in gestural scores. The gestures explicitly capture the moving patterns of different articulators. The gestural scores are the ultimate form of articulatory representations. The desired properties of gestural scores should be as follows: 1) Intelligible. 2) Interpretable. 3) Sparse. The gestures and gestural scores explicitly form a simple yet effective speech production system. Previously, only a few works have attempted to derive the gestures and gestural scores in a data-driven manner.~\cite{spatio-temporal} adopts the convolutive sparse non-negative matrix factorization (CSNMF) to decompose the articulatory data into gestures and gestural scores. Our previous work~\cite{lian_art} proposed an end-to-end neural convolutive matrix factorization (NCMF) paradigm so that both gestures and gestural scores can be automatically learned via neural networks. However, directly applying matrix factorization on the articulatory data is problematic. In the task dynamics model of speech production~\cite{task_dynamical}, each articulator contributes to the production of speech with a certain percentage. These percent contributions should be carefully determined~\cite{sorensen2019task}. In NCMF paradigm~\cite{lian_art}, the gestural scores, which implicitly encode the percent contributions of articulators, are pretty sensitive to the parameter initialization and are randomly determined. This also affects the dynamical patterns of gestures so that some of the gestures do not capture articulator-specific moving patterns, which limits the interpretability of both gestures and gestural scores. &#10;&#10;To alleviate the aforementioned problems, we propose a two-step articulatory decomposition rule. In the first step, we adopt the guided factor analysis algorithm~\cite{guided, sorensen2019task} to extract factors and factor scores from the articulatory data. The factor characterizes spatial variation in the position and shape of an articulator. The factor scores parameterize how the position and shape of all articulators change over time. The first step ensures that percent contribution of each articulator is within a reasonable range and ensures that the articulator-specific patterns could be captured. In the second step, we perform the NCMF~\cite{lian_art} on the factor scores to to obtain the sparse gestural scores and gestures. As we mentioned, the gestural scores are the ultimate form of articulatory representation. But different from NCMF~\cite{lian_art}, we define the matrix product of gestures and factors as the new gestures, which capture fine-grained articulatory moving patterns. Details about our new gestures formulation are given in Sec.~\ref{NCMF_sec}. Eq.\ref{gesture_formular}. Combining these two-step decomposition algorithms, we obtain the gestural scores and gestures from the articutary data. Another disadvantage of NCMF~\cite{lian_art} is that the MNGU0 data is sparsely sampled from articulators, which limits the intelligibility of learned representations. In this paper, we experiment with the rtMRI corpus~\cite{rtMRI} that captures the fine-grained vocal tract contours to improve the intelligibility of the learned representations. Both subjective and objective evaluation results suggest that the newly proposed system delivers decent articulatory representations in terms of intelligibility, explainability, efficiency and generalizability. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Production Modeling, Neural Signal Processing, Machine Learning, Computer Science, Speech Processing, Linguistics, Articulatory Representation Learning, Biological Sciences" />
        </attvalues>
      </node>
      <node id="2204.00465" label="2204.00465">
        <attvalues>
          <attvalue for="0" value="Deep Neural Convolutive Matrix Factorization for Articulatory&#10;  Representation Decomposition" />
          <attvalue for="1" value="  Most of the research on data-driven speech representation learning has&#10;focused on raw audios in an end-to-end manner, paying little attention to their&#10;internal phonological or gestural structure. This work, investigating the&#10;speech representations derived from articulatory kinematics signals, uses a&#10;neural implementation of convolutive sparse matrix factorization to decompose&#10;the articulatory data into interpretable gestures and gestural scores. By&#10;applying sparse constraints, the gestural scores leverage the discrete&#10;combinatorial properties of phonological gestures. Phoneme recognition&#10;experiments were additionally performed to show that gestural scores indeed&#10;code phonological information successfully. The proposed work thus makes a&#10;bridge between articulatory phonology and deep neural networks to leverage&#10;informative, intelligible, interpretable,and efficient speech representations.&#10;" />
          <attvalue for="2" value="&#10;&#10;Research on speech representation learning has been dominated by deep learning recently in the areas such as speaker recognition~\cite{lian2020masked, ECAPA-TDNN}, automatic speech recognition~\cite{conformer}, voice conversion~\cite{DSVAE-VC, C-DSVAE} and text-to-speech~\cite{VITS, lian2022utts}, etc. The goal of speech representation learning is to optimize both the performance of the model architectures and the interpretability of the learned representations. As there is growing demand of real-life applications of speech interfaces \cite{herff2016automatic}, the performance is emphasized to a larger extent, enabling human-machine interactions highly accurate and robust. Consequently, in most of these works the interpretability of representations has not been explored to an equivalent extent, which is one of the most significant bottlenecks that keeps the speech research from going farther. In general, speech representations need to be better understood and developed. &#10;&#10;People usually represent speech via audio because human perceive speech through hearing and audio is cheap to record, collect and process. However, speech processing is quite a lot different from audio processing. It might not need any evidence to indicate that any information that can be perceived via human can be perceived anywhere from source to destination. Perceiving the speech signal from the source and leveraging how it is produced are the most straightforward way to interpret it. The speech signal is the result of respiratory, phonatory and articulatory processes that generate the perceivable acoustic resonances to encode an intended linguistic message \cite{origin-of-speech}. In that sense, perceiving the speech signal from articulatory data is a preferred way to derive interpretable, natural and robust speech representations. &#10;&#10;The framework of articulatory phonology \cite{articulatory-phonology} has offered a lawful approach to modeling the relation between phonological representations as a set of discrete compositional units, or gestures, and the variability in time that derives from variation in the activation of the gestures in real-time: the magnitude of their activation, and the temporal intervals of activation as represented in gestural scores. However, the gestures and gestural scores of particular utterances have never been estimated in a completely data-driven manner. \cite{spatio-temporal} utilized the convolutive sparse non-negative matrix factorization (CSNMF) to decompose the non-negative articulatory data into the gestures and gestural scores, both of which are pretty much interpretable. The downsides of such method are that all the training utterances have to be concatenated into a large matrix, resulting in both memory and training efficiency issues. Additionally, such a model is not compatible with the modern deep learning based speech models so that it is challenging to perform end-to-end training on articulatory data. &#10;&#10;To handle the aforementioned problem, \cite{neural-nmf} proposed an auto-encoder based model to replace non-negatve matrix factorization for speech separation task. Inspired by this work, we propose a convolutional auto-encoder as the neural implementation of convolutive matrix factorization. Such auto-encoder based matrix factorization method is compatible with modern deep neural network and the batch-wise optimization improves the convergence rate to the huge extent. Under such framework, the articulatory signal is decomposed into gestures and gestural scores which are still interpretable. The gestural scores are the learned articulatory speech representations and are constrained to be sparse. In the last stage, the phoneme recognition experiments were performed to show that the learned gestural scores are also intelligible and consistent in time domain. All the experiments are performed using MNGU0 EMA (Electromagnetic midsagittal articulography) \cite{mngu0} corpus. The intention is that the proposed work could bridge the gap between explainable articulatory phonology and modern deep neural networks to deliver interpretable, intelligible, informative, and efficient speech representations.&#10; " />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.01898" label="1703.01898">
        <attvalues>
          <attvalue for="0" value="Generative and Discriminative Text Classification with Recurrent Neural&#10;  Networks" />
          <attvalue for="1" value="  We empirically characterize the performance of discriminative and generative&#10;LSTM models for text classification. We find that although RNN-based generative&#10;models are more powerful than their bag-of-words ancestors (e.g., they account&#10;for conditional dependencies across words in a document), they have higher&#10;asymptotic error rates than discriminatively trained RNN models. However we&#10;also find that generative models approach their asymptotic error rate more&#10;rapidly than their discriminative counterparts---the same pattern that Ng &amp;&#10;Jordan (2001) proved holds for linear classification models that make more&#10;naive conditional independence assumptions. Building on this finding, we&#10;hypothesize that RNN-based generative classification models will be more robust&#10;to shifts in the data distribution. This hypothesis is confirmed in a series of&#10;experiments in zero-shot and continual learning settings that show that&#10;generative models substantially outperform discriminative models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.02254" label="2306.02254">
        <attvalues>
          <attvalue for="0" value="A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean&#10;  Language Models" />
          <attvalue for="1" value="  Polyglot is a pioneering project aimed at enhancing the non-English language&#10;performance of multilingual language models. Despite the availability of&#10;various multilingual models such as mBERT (Devlin et al., 2019), XGLM (Lin et&#10;al., 2022), and BLOOM (Scao et al., 2022), researchers and developers often&#10;resort to building monolingual models in their respective languages due to the&#10;dissatisfaction with the current multilingual models non-English language&#10;capabilities. Addressing this gap, we seek to develop advanced multilingual&#10;language models that offer improved performance in non-English languages. In&#10;this paper, we introduce the Polyglot Korean models, which represent a specific&#10;focus rather than being multilingual in nature. In collaboration with TUNiB,&#10;our team collected 1.2TB of Korean data meticulously curated for our research&#10;journey. We made a deliberate decision to prioritize the development of Korean&#10;models before venturing into multilingual models. This choice was motivated by&#10;multiple factors: firstly, the Korean models facilitated performance&#10;comparisons with existing multilingual models; and finally, they catered to the&#10;specific needs of Korean companies and researchers. This paper presents our&#10;work in developing the Polyglot Korean models, which propose some steps towards&#10;addressing the non-English language performance gap in multilingual language&#10;models.&#10;" />
          <attvalue for="2" value="&#10;&#10;The advent of large-scale language models has revolutionized the field of natural language processing, leading to significant advancements in various applications, such as language translation and text classification \cite{devlin2019bert,radford2019language,liu2019roberta,clark2020electra,chowdhery2022palm,anil2023palm}. While numerous large language models for English have been publicly released \cite{zhang2022opt,black2022gptneox20b,biderman2023pythia,touvron2023llama,together2023redpajama,MosaicML2023Introducing}, the availability of such models for non-English languages remains limited. Although several multilingual large language models have also been released \cite{lin2022fewshot,scao2022bloom}, they are typically trained on English-centric corpora, resulting in lower performance on other languages. &#10;&#10;However, with the increasing interest in non-English languages, there is a growing need for high-performance language models that are specifically tailored to these languages. To tackle this challenge, we initiated the Polyglot project, which focuses on the development of large language models customized for non-English languages. As part of the project, our first model in this endeavor is Polyglot-Ko, an exceptional language model specifically designed for the Korean language. We chose to prioritize the Korean language as the initial model because our founding members are primarily Korean and we had a readily available dataset for training purposes. Our goal is to make these language models accessible to researchers and practitioners, empowering them to explore and advance natural language processing tasks in their respective languages. Polyglot-Ko leverages the transformer architecture, known for its effectiveness in capturing long-range dependencies in natural language text. Our model has been trained on an extensive corpus of text data, incorporating diverse sources such as web pages, news articles, and social media posts to ensure its versatility across different domains and styles.&#10;&#10;In this technical report, we provide a comprehensive description of the architecture and training process for four distinct Polyglot-Ko models, differing in parameter sizes: 1.3B (billion), 3.8B, 5.8B, and 12.8B. Notably, our 12.8 billion parameter model represents the largest publicly available Korean language model suitable for commercial applications, making it an invaluable resource for researchers and practitioners engaged in Korean natural language processing tasks.&#10;&#10;We assess the zero-shot and few-shot performance of our Polyglot-Ko models using the KOBEST benchmark \cite{kim2022kobest}. Through our experiments, we have successfully demonstrated that Polyglot-Ko attains competitive results across various benchmark datasets.&#10;&#10;In addition to presenting our achievements, we also acknowledge potential limitations and identify areas that warrant future improvement. By offering recommendations for further research, we aspire to foster advancements in the field. We firmly believe that Polyglot-Ko will serve as a valuable resource for the Korean natural language processing community, enabling the development of innovative applications and contributing to a deeper understanding of the intricacies and dynamics of the Korean language.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Non-English Language Performance, Artificial Intelligence, Information Science, Korean Language Processing" />
        </attvalues>
      </node>
      <node id="2204.04541" label="2204.04541">
        <attvalues>
          <attvalue for="0" value="KOBEST: Korean Balanced Evaluation of Significant Tasks" />
          <attvalue for="1" value="  A well-formulated benchmark plays a critical role in spurring advancements in&#10;the natural language processing (NLP) field, as it allows objective and precise&#10;evaluation of diverse models. As modern language models (LMs) have become more&#10;elaborate and sophisticated, more difficult benchmarks that require linguistic&#10;knowledge and reasoning have been proposed. However, most of these benchmarks&#10;only support English, and great effort is necessary to construct benchmarks for&#10;other low resource languages. To this end, we propose a new benchmark named&#10;Korean balanced evaluation of significant tasks (KoBEST), which consists of&#10;five Korean-language downstream tasks. Professional Korean linguists designed&#10;the tasks that require advanced Korean linguistic knowledge. Moreover, our data&#10;is purely annotated by humans and thoroughly reviewed to guarantee high data&#10;quality. We also provide baseline models and human performance results. Our&#10;dataset is available on the Huggingface.&#10;" />
          <attvalue for="2" value="&#10;The NLP field is now facing unprecedented rapid development. A major factor propelling the progress is the existence of unified benchmark datasets like GLUE~\cite{GLUE}, which are designed to assess models' language understanding capabilities. Such benchmark datasets, enabled modern PLM, such as BERT \cite{BERT} and GPT-2 and GPT-3 \cite{GPT2, GPT3}, to be assessed in objective and multifaceted manners. The success of GLUE also lead to similar benchmark datasets in a variety of other languages.~\cite{CLUE, FLUE, indonlu, KLUE}.&#10;&#10;However, many recent studies reveal that the outstanding performance of PLM on such benchmark datasets seems plausible but not probable. These studies have found that datasets may contain many spurious artefacts, and the performance of PLM is enhanced by excessive usage of said artefacts~\cite{habernal2018argument, niven2019probing, mccoy2019right, benderkoller2020climbing}. Another line of work observed that many PLM, which showed promising results in GLUE, fall short of expectations for more difficult tasks that require linguistic knowledge~\cite{bhatt2021case} or logical reasoning~\cite{logicNLI}. As a result, the importance of well-designed evaluation datasets with higher difficulty-level has been highlighted, and new datasets, such as CheckList~\cite{ribeiro2020beyond}, and LogicNLI~\cite{logicNLI}, have been proposed. Most of them only support specific languages like English, and it requires large efforts to build higher difficulty-level language evaluation suits for other low resource languages, however. &#10;&#10;When it comes to the Korean language, two benchmarks are widely used: Korean-NLI \&amp; STS~\cite{ham2020kornli} and KLUE~\cite{KLUE}. The former is machine- and human-translated from English NLI and STS datasets, which hardly reflect the characteristics of the Korean language. The latter is a Korean version GLUE benchmark which supports eight tasks, such as NLI, STS, NER, and RE. Although these tasks are useful for assessing general language ability, it is difficult to ascertain whether a model is able to reason based on more complicated knowledge beyond text form (e.g., passage of time, meaning of text, causality). &#10;To this end, we aim to construct a new benchmark dataset in Korean named KoBEST, which consists of five downstream tasks that require advanced knowledge of Korean. We carefully constructed the data based on the following design principles:&#10;&#10;\begin{itemize}&#10; \item Human-driven data annotation: Our data is purely annotated by humans to prevent incorrect and ambiguous data instances caused by automatic data annotation approach.&#10;&#10;\item Leveraging professional linguistic knowledge: As a result of our collaboration with professional Korean linguists, we re able to collect grammatically correct data with rich vocabulary and expressions.&#10;&#10;\item Availability to public: As a benchmark dataset, it is important to ensure public&#10; accessibility. We guarantee that our data is free to use and redistribute.&#10;&#10;\item High data quality: Our data passed thorough reviews driven by both models and humans to deliver high quality data without superficial cues and heuristic artefacts.&#10;&#10;\item Avoiding AI ethical issues: Our data does not contain any toxic content, social biases, or personal information.&#10;&#10;\end{itemize}&#10;&#10;Next, we evaluated widely used Korean PLM on the KoBEST dataset. Specifically, we conducted fine-tuning, zero-shot, one-shot, and few-shot experiments. The experimental results can serve as a baseline for performance on KoBEST. Participants also provided human performance baselines for all of our tasks. Our results suggest that modern PLM and a large-size GLM are far from reaching human-level language ability.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.07071" label="2105.07071">
        <attvalues>
          <attvalue for="0" value="Listen with Intent: Improving Speech Recognition with Audio-to-Intent&#10;  Front-End" />
          <attvalue for="1" value="  Comprehending the overall intent of an utterance helps a listener recognize&#10;the individual words spoken. Inspired by this fact, we perform a novel study of&#10;the impact of explicitly incorporating intent representations as additional&#10;information to improve a recurrent neural network-transducer (RNN-T) based&#10;automatic speech recognition (ASR) system. An audio-to-intent (A2I) model&#10;encodes the intent of the utterance in the form of embeddings or posteriors,&#10;and these are used as auxiliary inputs for RNN-T training and inference.&#10;Experimenting with a 50k-hour far-field English speech corpus, this study shows&#10;that when running the system in non-streaming mode, where intent representation&#10;is extracted from the entire utterance and then used to bias streaming RNN-T&#10;search from the start, it provides a 5.56% relative word error rate reduction&#10;(WERR). On the other hand, a streaming system using per-frame intent posteriors&#10;as extra inputs for the RNN-T ASR system yields a 3.33% relative WERR. A&#10;further detailed analysis of the streaming system indicates that our proposed&#10;method brings especially good gain on media-playing related intents (e.g. 9.12%&#10;relative WERR on PlayMusicIntent).&#10;" />
          <attvalue for="2" value="&#10;Spoken language understanding (SLU) systems are conventionally designed as a pipeline that includes an automatic speech recognition (ASR) system that converts speech to text, followed by a natural language understanding (NLU) system that extracts structured data such as domain, intent and slots. &#10;&#10;For the ASR system, end-to-end models have gained popularity in recent years as they combine separate components of conventional DNN-HMM hybrid ASR systems \cite{hinton2012deep} (acoustic, pronunciation and language models) into a single neural network. End-to-end models include connectionist temporal classification \cite{graves2006connectionist}, recurrent neural network-transducer (RNN-T) \cite{graves2012sequence}, and attention-based sequence-to-sequence models \cite{bahdanau2014neural,chorowski2015attention,ray2018ad3} also known as LAS: Listen, Attend and Spell \cite{chan2016listen}. Among these three methods, RNN-T is replacing the traditional hybrid ASR models \cite{he2019streaming,sainath2019two} since it has good streaming capability which is challenging to LAS and does not have CTC's frame-independence assumption. Various directions have been explored to enhance RNN-T ASR performance. Depth-LSTM and layer normalization was tried in \cite{li2019improving}. Using LAS as a second-pass rescorer by attending to both encoder features and n-best output from the RNN-T has been explored in \cite{sainath2019two}, and an inter encoder-decoder attention mechanism was introduced in \cite{wang2020attention} to better align the encoder feature with the hypothesis. Other improvements include better initialization methods, training on TTS data, and use of lookahead encoders \cite{li2020developing}. All these improvements on the RNN-T ASR system so far have focused on better acoustic or language modeling and rescoring extensions without adding any capability of understanding to the system.&#10;&#10;When humans process speech signals, transcription and understanding happen simultaneously, and the capability of understanding enables human to usually provide better quality of transcription than machines. Training an end-to-end SLU system to predict intent and slot values directly from audio is therefore becoming a popular research area \cite{qian2017exploring,serdyuk2018towards,chen2018spoken,haghani2018audio, lugosch2019speech,rao2020speech,rao2021mean}. While such system may not readily outperform or replace large-scale conventional SLU systems with independently optimized ASR and NLU modules, it is indicated that semantic information such as intent and slots could potentially help improve an ASR system \cite{haghani2018audio,rao2021mean}. However, these approaches were incorporating semantic information implicitly by using training intent and slot prediction as extra tasks. The technique to explicitly incorporate various contextual signals analogous to intent, such as dialog state and music play state, into an RNN-T based ASR system has been proposed in \cite{wu2020multistate,ray2021improving}, but most of these contextual signals are derived only after the first turn of the dialog and would only benefit subsequent turns. Therefore, extra studies are needed to investigate the impact of using semantic embeddings produced on-the-fly directly with input audio features on an ASR system. Focusing on intent-based semantic embeddings, our contributions in this study are as follows:&#10;\begin{itemize}[leftmargin=5mm]&#10; \item We propose to incorporate intent embeddings of the audio into the RNN-T ASR system to improve its recognition accuracy through an auxiliary audio-to-intent (A2I) front-end.&#10; \item We run extensive experiments on a large corpus of 50k hours of far-field US English speech to demonstrate effectiveness of the proposed approach.&#10;\end{itemize}&#10;To the best of our knowledge, this is the first study to enhance RNN-T ASR performance using intent representations produced on-the-fly from an auxiliary intent prediction model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Intent Modeling, Linguistics, Cognitive Science, Neural Network Optimization, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2005.08497" label="2005.08497">
        <attvalues>
          <attvalue for="0" value="Attention-based Transducer for Online Speech Recognition" />
          <attvalue for="1" value="  Recent studies reveal the potential of recurrent neural network transducer&#10;(RNN-T) for end-to-end (E2E) speech recognition. Among some most popular E2E&#10;systems including RNN-T, Attention Encoder-Decoder (AED), and Connectionist&#10;Temporal Classification (CTC), RNN-T has some clear advantages given that it&#10;supports streaming recognition and does not have frame-independency assumption.&#10;Although significant progresses have been made for RNN-T research, it is still&#10;facing performance challenges in terms of training speed and accuracy. We&#10;propose attention-based transducer with modification over RNN-T in two aspects.&#10;First, we introduce chunk-wise attention in the joint network. Second,&#10;self-attention is introduced in the encoder. Our proposed model outperforms&#10;RNN-T for both training speed and accuracy. For training, we achieves over 1.7x&#10;speedup. With 500 hours LAIX non-native English training data, attention-based&#10;transducer yields ~10.6% WER reduction over baseline RNN-T. Trained with full&#10;set of over 10K hours data, our final system achieves ~5.5% WER reduction over&#10;that trained with the best Kaldi TDNN-f recipe. After 8-bit weight quantization&#10;without WER degradation, RTF and latency drop to 0.34~0.36 and 268~409&#10;milliseconds respectively on a single CPU core of a production server.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.06173" label="2008.06173">
        <attvalues>
          <attvalue for="0" value="Speech To Semantics: Improve ASR and NLU Jointly via All-Neural&#10;  Interfaces" />
          <attvalue for="1" value="  We consider the problem of spoken language understanding (SLU) of extracting&#10;natural language intents and associated slot arguments or named entities from&#10;speech that is primarily directed at voice assistants. Such a system subsumes&#10;both automatic speech recognition (ASR) as well as natural language&#10;understanding (NLU). An end-to-end joint SLU model can be built to a required&#10;specification opening up the opportunity to deploy on hardware constrained&#10;scenarios like devices enabling voice assistants to work offline, in a privacy&#10;preserving manner, whilst also reducing server costs.&#10;  We first present models that extract utterance intent directly from speech&#10;without intermediate text output. We then present a compositional model, which&#10;generates the transcript using the Listen Attend Spell ASR system and then&#10;extracts interpretation using a neural NLU model. Finally, we contrast these&#10;methods to a jointly trained end-to-end joint SLU model, consisting of ASR and&#10;NLU subsystems which are connected by a neural network based interface instead&#10;of text, that produces transcripts as well as NLU interpretation. We show that&#10;the jointly trained model shows improvements to ASR incorporating semantic&#10;information from NLU and also improves NLU by exposing it to ASR confusion&#10;encoded in the hidden layer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.06750" label="2102.06750">
        <attvalues>
          <attvalue for="0" value="Do as I mean, not as I say: Sequence Loss Training for Spoken Language&#10;  Understanding" />
          <attvalue for="1" value="  Spoken language understanding (SLU) systems extract transcriptions, as well&#10;as semantics of intent or named entities from speech, and are essential&#10;components of voice activated systems. SLU models, which either directly&#10;extract semantics from audio or are composed of pipelined automatic speech&#10;recognition (ASR) and natural language understanding (NLU) models, are&#10;typically trained via differentiable cross-entropy losses, even when the&#10;relevant performance metrics of interest are word or semantic error rates. In&#10;this work, we propose non-differentiable sequence losses based on SLU metrics&#10;as a proxy for semantic error and use the REINFORCE trick to train ASR and SLU&#10;models with this loss. We show that custom sequence loss training is the&#10;state-of-the-art on open SLU datasets and leads to 6% relative improvement in&#10;both ASR and NLU performance metrics on large proprietary datasets. We also&#10;demonstrate how the semantic sequence loss training paradigm can be used to&#10;update ASR and SLU models without transcripts, using semantic feedback alone.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06183" label="2106.06183">
        <attvalues>
          <attvalue for="0" value="Improving RNN-T ASR Performance with Date-Time and Location Awareness" />
          <attvalue for="1" value="  In this paper, we explore the benefits of incorporating context into a&#10;Recurrent Neural Network (RNN-T) based Automatic Speech Recognition (ASR) model&#10;to improve the speech recognition for virtual assistants. Specifically, we use&#10;meta information extracted from the time at which the utterance is spoken and&#10;the approximate location information to make ASR context aware. We show that&#10;these contextual information, when used individually, improves overall&#10;performance by as much as 3.48% relative to the baseline and when the contexts&#10;are combined, the model learns complementary features and the recognition&#10;improves by 4.62%. On specific domains, these contextual signals show&#10;improvements as high as 11.5%, without any significant degradation on others.&#10;We ran experiments with models trained on data of sizes 30K hours and 10K&#10;hours. We show that the scale of improvement with the 10K hours dataset is much&#10;higher than the one obtained with 30K hours dataset. Our results indicate that&#10;with limited data to train the ASR model, contextual signals can improve the&#10;performance significantly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.02392" label="2111.02392">
        <attvalues>
          <attvalue for="0" value="A Comparison of Discrete and Soft Speech Units for Improved Voice&#10;  Conversion" />
          <attvalue for="1" value="  The goal of voice conversion is to transform source speech into a target&#10;voice, keeping the content unchanged. In this paper, we focus on&#10;self-supervised representation learning for voice conversion. Specifically, we&#10;compare discrete and soft speech units as input features. We find that discrete&#10;representations effectively remove speaker information but discard some&#10;linguistic content - leading to mispronunciations. As a solution, we propose&#10;soft speech units. To learn soft units, we predict a distribution over discrete&#10;speech units. By modeling uncertainty, soft units capture more content&#10;information, improving the intelligibility and naturalness of converted speech.&#10;Samples available at https://ubisoft-laforge.github.io/speech/soft-vc/. Code&#10;available at https://github.com/bshall/soft-vc/.&#10;" />
          <attvalue for="2" value="&#10;&#10;Voice conversion systems transform source speech into a target voice, keeping the content unchanged.&#10;From re-creating young Luke Skywalker in The Mandalorian \cite{mandalorian}, to restoring the voice of an Amyloidosis patient \cite{singer2021respeecher}, voice conversion has applications across entertainment, education and healthcare.&#10;&#10;In a typical voice conversion system, the goal is to learn features that capture linguistic content but discard speaker-specific details.&#10;We can then replace the speaker information to synthesize audio in a target voice.&#10;While systems trained on parallel data \cite{toda2007voice, tanaka2019atts2s} or text transcriptions \cite{sun2016phonetic, Huang2020} produce convincing results, they require costly data collection and annotation efforts.&#10;Unsupervised voice conversion addresses this issue by learning without labels or parallel speech \cite{kameoka2018stargan, Qian2019}.&#10;However, there is still a gap in quality and intelligibility between unsupervised and supervised systems \cite{zhao2020voice}.&#10;&#10;To bridge this gap, recent work investigates self-supervised representation learning for voice-conversion.&#10;Most of these studies focus on discrete speech units \cite{polyak2021speech, van2020vector, huang2021any}.&#10;The idea is that discretization imposes an information bottleneck separating content from speaker details.&#10;While effective at removing speaker information, discretization also discards some linguistic content -- increasing mispronunciations in the converted speech.&#10;Take the word fin, for example. &#10;Ambiguous frames in the fricative \textipa{/f/} may be assigned to incorrect nearby units, resulting in the mispronunciation thin.&#10;&#10;To tackle this problem, we propose soft speech units.&#10;Using a fine-tuning procedure similar to \cite{hsu2021hubert}, we train a network to predict a distribution over discrete speech units.&#10;By modeling uncertainty in discrete-unit assignments, we aim to retain more content information and, as a result, correct mispronunciations like fin-thin.&#10;This idea is inspired by soft-assignment in computer vision, which has been shown to improve performance on classification tasks \cite{VanGemert2010}.&#10;&#10;Focusing on any-to-one voice conversion (i.e., any source speaker to a single target speaker), we compare discrete and soft speech units across two self-supervised methods: contrastive predictive coding (CPC) \cite{oord2018representation} and hidden-unit BERT (HuBERT) \cite{hsu2021hubert}.&#10;Finally, we evaluate discrete and soft units on a cross-lingual voice conversion task.&#10;&#10;Our main contributions are as follows:&#10;\begin{itemize}[leftmargin=10pt,topsep=4pt,partopsep=0pt,parsep=0pt]&#10;\item We propose soft speech units for voice conversion and describe a method to learn them from discrete units.&#10;\item We find that soft units improve intelligibility and naturalness compared to discrete speech units.&#10;\item We show that soft units transfer better to unseen languages in cross-lingual voice conversion.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Voice Conversion, Machine Learning, Computer Science, Speech Processing, Linguistics, Engineering, Deep Learning, Speech Representation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2008.12527" label="2008.12527">
        <attvalues>
          <attvalue for="0" value="Voice Conversion Challenge 2020: Intra-lingual semi-parallel and&#10;  cross-lingual voice conversion" />
          <attvalue for="1" value="  The voice conversion challenge is a bi-annual scientific event held to&#10;compare and understand different voice conversion (VC) systems built on a&#10;common dataset. In 2020, we organized the third edition of the challenge and&#10;constructed and distributed a new database for two tasks, intra-lingual&#10;semi-parallel and cross-lingual VC. After a two-month challenge period, we&#10;received 33 submissions, including 3 baselines built on the database. From the&#10;results of crowd-sourced listening tests, we observed that VC methods have&#10;progressed rapidly thanks to advanced deep learning methods. In particular,&#10;speaker similarity scores of several systems turned out to be as high as target&#10;speakers in the intra-lingual semi-parallel VC task. However, we confirmed that&#10;none of them have achieved human-level naturalness yet for the same task. The&#10;cross-lingual conversion task is, as expected, a more difficult task, and the&#10;overall naturalness and similarity scores were lower than those for the&#10;intra-lingual conversion task. However, we observed encouraging results, and&#10;the MOS scores of the best systems were higher than 4.0. We also show a few&#10;additional analysis results to aid in understanding cross-lingual VC better.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.11299" label="2405.11299">
        <attvalues>
          <attvalue for="0" value="The CAP Principle for LLM Serving: A Survey of Long-Context Large&#10;  Language Model Serving" />
          <attvalue for="1" value="  We survey the large language model (LLM) serving area to understand the&#10;intricate dynamics between cost-efficiency and accuracy, which is magnified by&#10;the growing need for longer contextual understanding when deploying models at a&#10;massive scale. Our findings reveal that works in this space optimize along&#10;three distinct but conflicting goals: improving serving context length (C),&#10;improving serving accuracy (A), and improving serving performance (P). Drawing&#10;inspiration from the CAP theorem in databases, we propose a CAP principle for&#10;LLM serving, which suggests that any optimization can improve at most two of&#10;these three goals simultaneously. Our survey categorizes existing works within&#10;this framework. We find the definition and continuity of user-perceived&#10;measurement metrics are crucial in determining whether a goal has been met,&#10;akin to prior CAP databases in the wild. We recognize the CAP principle for LLM&#10;serving as a guiding principle, rather than a formal theorem, to inform&#10;designers of the inherent and dynamic trade-offs in serving models. As serving&#10;accuracy and performance have been extensively studied, this survey focuses on&#10;works that extend serving context length and address the resulting challenges.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) and their underlying transformer architecture have revolutionized AI&#10;and have become the bedrock of many emerging applications.&#10;The ecosystem around LLM is on an upward spiral towards artificial general intelligence (AGI): the number of new LLMs and their applications skyrocketed, and as of 2024, LLM-based applications already outperform humans across many tasks such as image classification and visual reasoning~\cite{levels-of-agi-2023, stanford-ai-report-2024}.&#10;High-quality models are essential for any realization of AGI,&#10;but it's equally important to deploy and serve models at a massive scale with a reasonably low cost without compromising their accuracy.&#10;The conflict between serving accuracy and serving performance (e.g., tokens per second.) is a hard one,&#10;prompting extensive research in this area~\cite{xu2024survey, zhou2024survey}.&#10;Generally, there is no one-size-fits-all solution in production settings.&#10;Optimizations to improve performance can lead to reduced accuracy and vice versa.&#10;For example, sparsity and quantization are two common techniques that trade accuracy for better performance.&#10;&#10;Unfortunately, this conflict between accuracy and performance has been exacerbated recently&#10;by the growing demand for longer contextual understanding when deploying models in practice~\cite{med-gemini}.&#10;This introduces new complexities as the transformer's attention mechanism exhibits&#10;a quadratic increase in resource consumption with longer contexts~\cite{pope2023efficiently}.&#10;Furthermore, LLMs struggle to utilize information from longer contexts effectively~\cite{lost-in-the-middle-2023}.&#10;Essentially, the need for long-context serving breaks the fragile balance between serving accuracy and performance,&#10;and calls for novel system designs.&#10;&#10;To explore the complex relationship between accuracy and performance in large-scale model deployments, particularly for handling long contexts,&#10;we conducted an extensive survey of the LLM serving area.&#10;We highlight three key observations after reviewing related literature.&#10;\begin{enumerate}&#10;&#10;\item &#10;First, we find the scope of a serving system has expanded.&#10;It comprises two system layers: a model serving layer and an agent serving layer. &#10;The model-layer system runs a given LLM model, typically exposing model inference as its northbound APIs~\cite{vLLM-sosp23,tensorrt-llm}. Works at this layer commonly optimize the model structure~\cite{transformer-xl, zhao2023atom}, cache~\cite{vLLM-sosp23,streamingLLM}, scheduling~\cite{orca-osdi22,tetriserve-2024}, etc.&#10;The agent-layer system sits atop the model-layer system and results from emerging LLM-based system applications that leverage LLM-driven workflow to improve a raw LLM model's accuracy and efficiency while handling complex real-world tasks~\cite{compound-ai-blog}.&#10;&#10;\item&#10;Second, we find works in this space&#10;optimize along three distinct goals:&#10;improving serving context length (Context),&#10;improving serving accuracy (Accuracy),&#10;and improving serving performance (Performance).&#10;Specifically, context means the number of tokens in the context window; accuracy means evaluation metrics on certain tasks (e.g., MMLU), and performance means time-to-first-token, tokens per second, price per million tokens, etc.&#10;&#10;\item&#10;Finally, we find a trilemma among the above three goals regardless of which layer they are applied to.&#10;We find that any serving optimization can only improve at most two distinct goals.&#10;We also observe progress in one direction does not lead to progress in others.&#10;For example, using positional embedding to extend a model's range does not improve the model’s accuracy beyond the context length~\cite{roformer-su2021},&#10;and using quantization~\cite{zhao2023atom}, pruning~\cite{compression-survey-2023}, and sparsity~\cite{streamingLLM} enable one to serve a model with faster speed but at the cost of potentially lower accuracy.&#10;&#10;\end{enumerate}&#10;&#10;Based on the above observations and&#10;inspired by the classical CAP theorem in databases~\cite{cap-wiki},&#10;we propose the CAP principle for LLM serving, which states that any given LLM serving optimization, regardless of which system layer it is applied to, can improve at most two of the following goals:&#10;\begin{itemize}&#10; \item Context: The length of context effectively processed and perceived by end users.&#10; \item Accuracy: The precision of outputs as evaluated by end users, based on specific task metrics.&#10; \item Performance: The efficiency of token processing and generation perceived by end users.&#10;\end{itemize}&#10;&#10;The perspective of the proposed CAP principle emphasizes what end users perceive from applying a specific optimization to a remote LLM serving system rather than focusing on a specific component within the LLM serving system.&#10;This is crucial because we care whether an LLM serving system as a whole can serve AGI rather than a singular improvement in one direction.&#10;In general, this principle leads to six types of optimizations: C, A, P, CA, CP, and AP, depending on which goals are prioritized.&#10;&#10;The LLM's CAP principle is similar to the database's CAP theorem in many ways.&#10;\begin{itemize}&#10;&#10;\item &#10;Both state that you must forfeit at least one goal to achieve the others.&#10;Since our focus is on long-context serving, maintaining a lengthy context (C) is essential.&#10;This leaves us with two options: improving accuracy (A) or improving performance (P).&#10;Improving accuracy relies on devising new algorithms to better leverage the feature of lengthy context.&#10;However, these algorithms could hurt model execution cost-efficiency due to increased FLOPs, hardware-unfriendly operations, etc.&#10;On the other hand, enhancing performance on specific hardware through techniques like quantization and sparsity usually comes at the cost of reduced accuracy. Although there are methods to increase performance without losing accuracy, they generally require additional hardware resources.&#10;&#10;\item &#10;Their goals are measured continuously rather than in binary.&#10;The definition and continuity of user-perceived measurement metrics are crucial in determining whether a goal has been met. Some recent studies have examined this aspect for accuracy~\cite{du2024understanding, schaeffer2024emergent}.&#10;The availability of the database's CAP and the accuracy of the LLM's CAP both range from 0 to 100.&#10;The accuracy of LLM's CAP principle, like the availability of the database's CAP theorem, does not have to be 100\%. It just has to be high enough that end users deem it useful.&#10;Thus, from a system's perspective, an optimization categorized as CP might still be perceived as achieving all three CAP goals if it fulfils the user's accuracy requirements, similar to how CAP is observed in practical databases~\cite{spanner-cap}.&#10;&#10;\item &#10;Both are originally proposed to keep system designers aware of the hard design trade-offs while deploying large-scale systems.&#10;&#10;\end{itemize}&#10;&#10;We foresee the possibility of a true CAP in the future, in which there is no inherent conflict among these goals.&#10;The proposed CAP principle primarily arises from the use of transformer-based LLMs on existing AI chips, reflecting both the constraints and capabilities of today's hardware and software.&#10;As we progress towards AGI, both models and hardware are expected to evolve significantly.&#10;Emerging technologies are likely to be developed in tandem, with new models specifically designed to optimize performance on the next generation of hardware. &#10;This synergy between evolving models and hardware is crucial for overcoming current barriers and achieving a true CAP in LLM serving.&#10;&#10;Our survey is organized based on the propose CAP principle.&#10;Compared to prior surveys~\cite{zhou2024survey,xu2024survey,long-context-survey, long-context-survey-dong2023, long-context-survey-wang2024,agent-survey-2023,rag-survey},&#10;we makes two unique contributions.&#10;First, we propose the CAP principle for LLM serving and map existing works onto the CAP landscape to highlight the tension among them. &#10;Second, we approach the large-scale LLM serving system as a whole rather than focusing on a specific technique (e.g., RAG~\cite{rag-survey}, long-context~\cite{long-context-survey-dong2023}), or a layer (e.g., model~\cite{xu2024survey}, agent~\cite{agent-survey-2023}).&#10;In the rest of the paper, we will discuss works as listed in Table~\ref{tbl:cap-overview} and Figure~\ref{fig:cap-overview}.&#10;We focus on works that extend serving context length and address the resulting accuracy and performance issues.&#10;Specifically, we will cover model memory (Table~\ref{tbl:mm}), positional embedding (Table~\ref{tbl:pe}), found-in-the-middle, distributed acceleration for long context, prompt compression, sparsity (Table~\ref{tbl:sparse}), and agent memory (Table~\ref{tbl:am}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Performance Optimization, Computer Science, Linguistics, Language Model Serving, Contextual Understanding, Mathematics, Information Systems" />
        </attvalues>
      </node>
      <node id="2404.18416" label="2404.18416">
        <attvalues>
          <attvalue for="0" value="Capabilities of Gemini Models in Medicine" />
          <attvalue for="1" value="  Excellence in a wide variety of medical applications poses considerable&#10;challenges for AI, requiring advanced reasoning, access to up-to-date medical&#10;knowledge and understanding of complex multimodal data. Gemini models, with&#10;strong general capabilities in multimodal and long-context reasoning, offer&#10;exciting possibilities in medicine. Building on these core strengths of Gemini,&#10;we introduce Med-Gemini, a family of highly capable multimodal models that are&#10;specialized in medicine with the ability to seamlessly use web search, and that&#10;can be efficiently tailored to novel modalities using custom encoders. We&#10;evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art&#10;(SoTA) performance on 10 of them, and surpass the GPT-4 model family on every&#10;benchmark where a direct comparison is viable, often by a wide margin. On the&#10;popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves&#10;SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search&#10;strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU&#10;(health &amp; medicine), Med-Gemini improves over GPT-4V by an average relative&#10;margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context&#10;capabilities through SoTA performance on a needle-in-a-haystack retrieval task&#10;from long de-identified health records and medical video question answering,&#10;surpassing prior bespoke methods using only in-context learning. Finally,&#10;Med-Gemini's performance suggests real-world utility by surpassing human&#10;experts on tasks such as medical text summarization, alongside demonstrations&#10;of promising potential for multimodal medical dialogue, medical research and&#10;education. Taken together, our results offer compelling evidence for&#10;Med-Gemini's potential, although further rigorous evaluation will be crucial&#10;before real-world deployment in this safety-critical domain.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.19102" label="2310.19102">
        <attvalues>
          <attvalue for="0" value="Atom: Low-bit Quantization for Efficient and Accurate LLM Serving" />
          <attvalue for="1" value="  The growing demand for Large Language Models (LLMs) in applications such as&#10;content generation, intelligent chatbots, and sentiment analysis poses&#10;considerable challenges for LLM service providers. To efficiently use GPU&#10;resources and boost throughput, batching multiple requests has emerged as a&#10;popular paradigm; to further speed up batching, LLM quantization techniques&#10;reduce memory consumption and increase computing capacity. However, prevalent&#10;quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully&#10;leverage the capabilities of modern GPUs, such as 4-bit integer operators,&#10;resulting in sub-optimal performance.&#10;  To maximize LLMs' serving throughput, we introduce Atom, a low-bit&#10;quantization method that achieves high throughput improvements with negligible&#10;accuracy loss. Atom significantly boosts serving throughput by using low-bit&#10;operators and considerably reduces memory consumption via low-bit quantization.&#10;It attains high accuracy by applying a novel mixed-precision and fine-grained&#10;quantization process. We evaluate Atom on 4-bit weight-activation quantization&#10;in the serving context. Atom improves end-to-end throughput (token/s) by up to&#10;$7.7\times$ compared to the FP16 and by $2.5\times$ compared to INT8&#10;quantization, while maintaining the same latency target.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.14294" label="2404.14294">
        <attvalues>
          <attvalue for="0" value="A Survey on Efficient Inference for Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have attracted extensive attention due to their&#10;remarkable performance across various tasks. However, the substantial&#10;computational and memory requirements of LLM inference pose challenges for&#10;deployment in resource-constrained scenarios. Efforts within the field have&#10;been directed towards developing techniques aimed at enhancing the efficiency&#10;of LLM inference. This paper presents a comprehensive survey of the existing&#10;literature on efficient LLM inference. We start by analyzing the primary causes&#10;of the inefficient LLM inference, i.e., the large model size, the&#10;quadratic-complexity attention operation, and the auto-regressive decoding&#10;approach. Then, we introduce a comprehensive taxonomy that organizes the&#10;current literature into data-level, model-level, and system-level optimization.&#10;Moreover, the paper includes comparative experiments on representative methods&#10;within critical sub-fields to provide quantitative insights. Last but not&#10;least, we provide some knowledge summary and discuss future research&#10;directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.07864" label="2309.07864">
        <attvalues>
          <attvalue for="0" value="The Rise and Potential of Large Language Model Based Agents: A Survey" />
          <attvalue for="1" value="  For a long time, humanity has pursued artificial intelligence (AI) equivalent&#10;to or surpassing the human level, with AI agents considered a promising vehicle&#10;for this pursuit. AI agents are artificial entities that sense their&#10;environment, make decisions, and take actions. Many efforts have been made to&#10;develop intelligent agents, but they mainly focus on advancement in algorithms&#10;or training strategies to enhance specific capabilities or performance on&#10;particular tasks. Actually, what the community lacks is a general and powerful&#10;model to serve as a starting point for designing AI agents that can adapt to&#10;diverse scenarios. Due to the versatile capabilities they demonstrate, large&#10;language models (LLMs) are regarded as potential sparks for Artificial General&#10;Intelligence (AGI), offering hope for building general AI agents. Many&#10;researchers have leveraged LLMs as the foundation to build AI agents and have&#10;achieved significant progress. In this paper, we perform a comprehensive survey&#10;on LLM-based agents. We start by tracing the concept of agents from its&#10;philosophical origins to its development in AI, and explain why LLMs are&#10;suitable foundations for agents. Building upon this, we present a general&#10;framework for LLM-based agents, comprising three main components: brain,&#10;perception, and action, and the framework can be tailored for different&#10;applications. Subsequently, we explore the extensive applications of LLM-based&#10;agents in three aspects: single-agent scenarios, multi-agent scenarios, and&#10;human-agent cooperation. Following this, we delve into agent societies,&#10;exploring the behavior and personality of LLM-based agents, the social&#10;phenomena that emerge from an agent society, and the insights they offer for&#10;human society. Finally, we discuss several key topics and open problems within&#10;the field. A repository for the related papers at&#10;https://github.com/WooooDyy/LLM-Agent-Paper-List.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.05955" label="2002.05955">
        <attvalues>
          <attvalue for="0" value="A Data Efficient End-To-End Spoken Language Understanding Architecture" />
          <attvalue for="1" value="  End-to-end architectures have been recently proposed for spoken language&#10;understanding (SLU) and semantic parsing. Based on a large amount of data,&#10;those models learn jointly acoustic and linguistic-sequential features. Such&#10;architectures give very good results in the context of domain, intent and slot&#10;detection, their application in a more complex semantic chunking and tagging&#10;task is less easy. For that, in many cases, models are combined with an&#10;external language model to enhance their performance.&#10;  In this paper we introduce a data efficient system which is trained&#10;end-to-end, with no additional, pre-trained external module. One key feature of&#10;our approach is an incremental training procedure where acoustic, language and&#10;semantic models are trained sequentially one after the other. The proposed&#10;model has a reasonable size and achieves competitive results with respect to&#10;state-of-the-art while using a small training dataset. In particular, we reach&#10;24.02% Concept Error Rate (CER) on MEDIA/test while training on MEDIA/train&#10;without any additional data.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken Language Understanding (SLU) aims at extracting a semantic representation from a speech signal in human-computer interaction applications \cite{DeMori1997:SDBook}. First SLU systems were based on pipeline architectures where an automatic speech recognition (ASR) module generates a transcription of utterances and a SLU module predicts the semantic labels.&#10;Pipeline systems now tend to be replaced by end-to-end architectures based on neural models, where semantic representations are produced directly from a speech input without using transcriptions \cite{DBLP:journals/corr/abs-1802-08395,DBLP:journals/corr/abs-1906-07601,ghannay:hal-01987740,DBLP:journals/corr/abs-1904-03670}.&#10;Most of recently proposed end-to-end models are based on sequence-to-sequence architectures. They were initially applied to speech translation \cite{berard-nips2016,weiss2017sequence} and then to SLU tasks where the main goal is to extract the domain and user intent from an utterance, together with some semantic slots \cite{DBLP:journals/corr/abs-1802-08395,DBLP:journals/corr/abs-1904-03670}.&#10;&#10;In this paper we address end-to-end&#10;semantic chunking and tagging of spoken utterances.&#10;The most relevant works of the literature with respect to this task \cite{DBLP:journals/corr/abs-1906-07601,ghannay:hal-01987740} propose models based on Feed-Forward Neural Networks (FFNN) similar to the Deep Speech 2 model proposed for ASR \cite{DBLP-journals/corr/AmodeiABCCCCCCD15}, and an independent pre-trained language model re-scores semantic outputs.&#10;Except for \cite{DBLP:journals/corr/abs-1904-03670}, most end-to-end SLU systems of the literature are trained on huge amount of data. \cite{DBLP:journals/corr/abs-1906-07601} also apply pre-training and transfer learning from other NLP tasks such as Named Entity Recognition (NER).&#10;&#10;The contribution of this paper lies in the proposal of a data efficient architecture which is trained end-to-end, with no additional pre-trained external module.&#10;The proposed model achieves competitive results with respect to state-of-the-art while using a small training dataset (French MEDIA \cite{Bonneau-Maynard2005:media}) and having a reasonable computational footprint. In particular, we reach 24.02\% Concept Error Rate (CER) on MEDIA/test while training on MEDIA/train without any additional data.&#10;&#10;The remainder of this paper is organised as follows. After presenting the task addressed by this work in Section~\ref{sec:SLU}, we describe our sequence-to-sequence neural model in Section~\ref{sec:seq2seq}. Section~\ref{sec:eval} provides our experimental study on the French MEDIA corpus and we conclude in Section~\ref{sec:conclusions}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="End-to-End Architectures, Computer Science, Semantic Parsing Models, Linguistics, Spoken Language Understanding, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2201.10716" label="2201.10716">
        <attvalues>
          <attvalue for="0" value="Neural Grapheme-to-Phoneme Conversion with Pre-trained Grapheme Models" />
          <attvalue for="1" value="  Neural network models have achieved state-of-the-art performance on&#10;grapheme-to-phoneme (G2P) conversion. However, their performance relies on&#10;large-scale pronunciation dictionaries, which may not be available for a lot of&#10;languages. Inspired by the success of the pre-trained language model BERT, this&#10;paper proposes a pre-trained grapheme model called grapheme BERT (GBERT), which&#10;is built by self-supervised training on a large, language-specific word list&#10;with only grapheme information. Furthermore, two approaches are developed to&#10;incorporate GBERT into the state-of-the-art Transformer-based G2P model, i.e.,&#10;fine-tuning GBERT or fusing GBERT into the Transformer model by attention.&#10;Experimental results on the Dutch, Serbo-Croatian, Bulgarian and Korean&#10;datasets of the SIGMORPHON 2021 G2P task confirm the effectiveness of our&#10;GBERT-based G2P models under both medium-resource and low-resource data&#10;conditions.&#10;" />
          <attvalue for="2" value="&#10;&#10;The grapheme-to-phoneme (G2P) conversion task is predicting the pronunciation of words from their spellings. &#10;Considering that a pronunciation dictionary can never cover all possible words in a language, G2P conversion is essential for any applications that depend on the mapping relationship between the spoken and written forms of a language, such as TTS and ASR \cite{elias2021parallel, gao2021pre, masumura20_interspeech}. &#10;&#10;Many studies have been conducted on G2P conversion. In early years, joint n-gram models \cite{galescu2002pronunciation}, joint sequence models \cite{bisani2008joint} and wFST \cite{novak2012wfst}&#10; were proposed. &#10;Recently, &#10;neural networks such as LSTM \cite{toshniwal2016jointly} and Transformer \cite{yolchuyeva2019transformer} have showed powerful ability on G2P conversion. &#10;The Transformer-based models have achieved state-of-the-art&#10;performance in many benchmarks \cite{yolchuyeva2019transformer, gorman2020sigmorphon}. &#10;Some imitation learning based methods \cite{makarov2020cluzh} also achieved comparable performance to the Transformer model.&#10;Nevertheless, building neural G2P models usually relies on a large, language-specific pronunciation dictionary, which may not be available for a lot of languages.&#10;One approach to address this issue is cross-lingual modeling. An early work is a wFST-based system \cite{deri2016grapheme}. Subsequently, multilingual neural networks \cite{peters2017massively} and pre-trained G2P models of high-resource languages \cite{engelhart2021grapheme} showed better cross-lingual G2P modeling ability.&#10;Another approach is utilizing multimodal data. Route et al. \cite{route2019multimodal} found that additional audio supervision can help the G2P model to learn a more optimal intermediate representation of graphemes.&#10;However, these studies mainly focused on utilizing the data resources of other languages or other modals to improve the performance of limited-resource G2P conversion, which do not explore better grapheme representations for G2P in an unsupervised way.&#10;&#10;Therefore, this paper proposes a pre-trained grapheme model named grapheme BERT (GBERT) to improve the Transformer-based G2P model. The design of GBERT is inspired by the language model BERT \cite{devlin2019bert}, which provides contextual word representations and has achieved great successes in various NLP tasks, such as machine translation \cite{zhu2019incorporating} and text summarization \cite{liu2019text}. &#10;Similarly, GBERT is designed to capture the contextual relationship among the graphemes in a word,&#10;which is essential for the G2P task since the same grapheme may have different pronunciations due to different contexts.&#10;Following BERT, &#10;GBERT is a multi-layer Transformer encoder and is pre-trained by self-supervised learning on a large, language-specific word list with only grapheme information. The pre-training task of GBERT is a masked grapheme prediction task, i.e, predicting the masked graphemes from the seen graphemes in a word. &#10;Furthermore, two approaches are developed to improve the Transformer-based G2P model with GBERT. They are fine-tuning GBERT and fusing GBERT into the Transformer model by attention \cite{zhu2019incorporating}. &#10;Experiments were conducted on the Dutch, Serbo-Croatian, Bulgarian and Korean datasets of the SIGMORPHON 2021 G2P task \cite{ashby2021results}. The results show that fusing GBERT by attention can reduce the word error rate (WER) and phone error rate (PER) of G2P under a medium-resource condition, while fine-tuning GBERT is effective for most languages under a low-resource condition. It should be noted that a concurrent work T5G2P \cite{vrezavckova2021t5g2p} also mentions the pre-trained grapheme model can help the G2P conversion. The difference is that the T5G2P uses an encoder-decoder framework which captures contextual relationship between graphemes and autoregressive information of graphemes in the pre-training stage and only uses a directly fine-tuning method.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Processing, Computer Science, Linguistics, Cognitive Science, Neural Network Models, Artificial Intelligence, Grapheme-to-Phoneme Conversion" />
        </attvalues>
      </node>
      <node id="1610.06540" label="1610.06540">
        <attvalues>
          <attvalue for="0" value="Jointly Learning to Align and Convert Graphemes to Phonemes with Neural&#10;  Attention Models" />
          <attvalue for="1" value="  We propose an attention-enabled encoder-decoder model for the problem of&#10;grapheme-to-phoneme conversion. Most previous work has tackled the problem via&#10;joint sequence models that require explicit alignments for training. In&#10;contrast, the attention-enabled encoder-decoder model allows for jointly&#10;learning to align and convert characters to phonemes. We explore different&#10;types of attention models, including global and local attention, and our best&#10;models achieve state-of-the-art results on three standard data sets (CMUDict,&#10;Pronlex, and NetTalk).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.01464" label="1708.01464">
        <attvalues>
          <attvalue for="0" value="Massively Multilingual Neural Grapheme-to-Phoneme Conversion" />
          <attvalue for="1" value="  Grapheme-to-phoneme conversion (g2p) is necessary for text-to-speech and&#10;automatic speech recognition systems. Most g2p systems are monolingual: they&#10;require language-specific data or handcrafting of rules. Such systems are&#10;difficult to extend to low resource languages, for which data and handcrafted&#10;rules are not available. As an alternative, we present a neural&#10;sequence-to-sequence approach to g2p which is trained on&#10;spelling--pronunciation pairs in hundreds of languages. The system shares a&#10;single encoder and decoder across all languages, allowing it to utilize the&#10;intrinsic similarities between different writing systems. We show an 11%&#10;improvement in phoneme error rate over an approach based on adapting&#10;high-resource monolingual g2p models to low-resource languages. Our model is&#10;also much more compact relative to previous approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.04091" label="2104.04091">
        <attvalues>
          <attvalue for="0" value="Grapheme-to-Phoneme Transformer Model for Transfer Learning Dialects" />
          <attvalue for="1" value="  Grapheme-to-Phoneme (G2P) models convert words to their phonetic&#10;pronunciations. Classic G2P methods include rule-based systems and&#10;pronunciation dictionaries, while modern G2P systems incorporate learning, such&#10;as, LSTM and Transformer-based attention models. Usually, dictionary-based&#10;methods require significant manual effort to build, and have limited adaptivity&#10;on unseen words. And transformer-based models require significant training&#10;data, and do not generalize well, especially for dialects with limited data.&#10;  We propose a novel use of transformer-based attention model that can adapt to&#10;unseen dialects of English language, while using a small dictionary. We show&#10;that our method has potential applications for accent transfer for&#10;text-to-speech, and for building robust G2P models for dialects with limited&#10;pronunciation dictionary size.&#10;  We experiment with two English dialects: Indian and British. A model trained&#10;from scratch using 1000 words from British English dictionary, with 14211 words&#10;held out, leads to phoneme error rate (PER) of 26.877%, on a test set generated&#10;using the full dictionary. The same model pretrained on CMUDict American&#10;English dictionary, and fine-tuned on the same dataset leads to PER of 2.469%&#10;on the test set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.07425" label="2407.07425">
        <attvalues>
          <attvalue for="0" value="Out-of-distribution generalisation in spoken language understanding" />
          <attvalue for="1" value="  Test data is said to be out-of-distribution (OOD) when it unexpectedly&#10;differs from the training data, a common challenge in real-world use cases of&#10;machine learning. Although OOD generalisation has gained interest in recent&#10;years, few works have focused on OOD generalisation in spoken language&#10;understanding (SLU) tasks. To facilitate research on this topic, we introduce a&#10;modified version of the popular SLU dataset SLURP, featuring data splits for&#10;testing OOD generalisation in the SLU task. We call our modified dataset SLURP&#10;For OOD generalisation, or SLURPFOOD. Utilising our OOD data splits, we find&#10;end-to-end SLU models to have limited capacity for generalisation. Furthermore,&#10;by employing model interpretability techniques, we shed light on the factors&#10;contributing to the generalisation difficulties of the models. To improve the&#10;generalisation, we experiment with two techniques, which improve the results on&#10;some, but not all the splits, emphasising the need for new techniques.&#10;" />
          <attvalue for="2" value="&#10;Spoken language understanding (SLU) systems are the backbone of human-computer interaction devices that need to understand the meaning of the utterance before taking an action. When these models are deployed for real-world use, their performance should be consistent even when presented with out-of-distribution (OOD) data that differs from the training distribution in an unpredictable manner \cite{liu2021towards}.&#10;&#10;Multiple types of OOD generalisation capacities are desirable but not necessarily achieved by neural SLU systems. Length generalisation (see e.g. \cite{pmlr-v202-ray-chowdhury23b,zhou2023algorithms}) is the capacity to process sequences that are longer (or shorter) than those seen in the training set. Out-of-vocabulary (OOV) generalisation is needed when the test set includes words or other units not seen in the training set \cite{lugosch19_interspeech, palogiannidi2020end}. A third type is compositional generalisation (CG) \cite{fodor1988connectionism, hupkes2020compositionality}, required when test samples combine familiar units in novel ways. For example, novel combinations of slot types in a slot filling task has been shown to pose difficulties for SLU systems \cite{broscheit2022distributionally, ray23_interspeech}. More generally, neural (NLP) systems have been found often to fail in tasks that require CG \cite{lake2018generalization, keysers2019measuring, yao2022structural}, although they do have some capacity for CG \cite{lepori2023break}. In addition to the generalisation types applicable to both text and audio sequences, some types are specific to audio-based models, such as diverse accents \cite{viglino19_interspeech}, different age groups \cite{potamianos2003robust}, and various acoustic environments \cite{haeb2020far}.&#10;&#10;The traditional pipeline SLU systems consist of two sub-modules: an automatic speech recognition (ASR) system that generates transcripts and a separate text-based natural language understanding (NLU) system that extracts meaning from the generated transcripts \cite{gupta2005t, ray23_interspeech}. Previous studies that explore generalisation in SLU employ a pipeline system, focusing on the transcripts instead of the original audio \cite{gaspers2022temporal, ray23_interspeech}. As these studies focus on the text-based language understanding task, the proposed evaluation sets are mostly text-only data, and as such, they can not always be applied to the end-to-end (E2E) SLU models that rely solely on audio \cite{serdyuk2018towards, haghani2018audio, palogiannidi2020end}. The main reason is that text is easy to segment due to word boundaries, whereas adding or removing segments of an audio signal corresponding to certain tokens in the transcript is challenging. Moreover, the audio models' generalisation criteria can differ from those of text-based models.&#10;&#10;Despite these challenges, a few previous works have evaluated the generalisation capabilities of E2E SLU models. In \cite{lugosch19_interspeech}, a system was trained on three specific phrases and tested with also a new phrase, assessing the OOV generalisation. Similarly, in \cite{palogiannidi2020end}, E2E SLU models were tested on unique wordings not seen in training. The findings revealed a notable increase in error rates as the number of unseen wordings increased. These studies, however, predominantly focus on a singular type of generalisation. The authors of \cite{arora21_interspeech} noted that generalisation in SLU tasks may be divided into two types: generalising to diverse speakers and generalising to diverse phrases (e.g., n-grams). These two types may be orthogonal and thus can be assessed separately. We argue, in the same vein, that there are multiple types of generalisation that SLU models should ideally achieve. However, our data splitting methods differ from those in \cite{arora21_interspeech} in multiple ways. The n-gram-based splits in \cite{arora21_interspeech} may result in OOV words in the test set, or possibly novel combinations (n-grams) of familiar words, requiring compositional generalisation. We distinguish OOV splits from CG splits, allowing for a more fine-grained evaluation. Moreover, we focus on slightly more abstract units than n-grams, namely the scenario and action labels. Our methods and resources therefore aim to complement the previous works, adding to the collection of resources enabling development of more robust SLU systems.&#10;&#10;We introduce OOV, CG, and microphone mismatch splits of the SLURP dataset \cite{bastianelli-etal-2020-slurp}, which we call SLURP For OOD generalisation (SLURPFOOD). Utilising these splits, we assess SLU models' capacity for OOD generalisation. As a benchmark, we report results for pre-trained self-supervised models. We also leverage a model interpretability method&#10;(Integrated Gradients \cite{sundararajan2017axiomatic}) to determine the underlying causes contributing to the limited generalisation of the systems. To improve the generalisation, we explore two techniques: TOPK \cite{kawaguchi2020ordered} and segmented processing. We make the SLURPFOOD splits, the models, and the code for reproducing all the experiments publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning Models, Computer Science, Out-of-Distribution Generalisation, Linguistics, Spoken Language Understanding, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2305.20019" label="2305.20019">
        <attvalues>
          <attvalue for="0" value="Monotonic Location Attention for Length Generalization" />
          <attvalue for="1" value="  We explore different ways to utilize position-based cross-attention in&#10;seq2seq networks to enable length generalization in algorithmic tasks. We show&#10;that a simple approach of interpolating the original and reversed encoded&#10;representations combined with relative attention allows near-perfect length&#10;generalization for both forward and reverse lookup tasks or copy tasks that had&#10;been generally hard to tackle. We also devise harder diagnostic tasks where the&#10;relative distance of the ideal attention position varies with timestep. In such&#10;settings, the simple interpolation trick with relative attention is not&#10;sufficient. We introduce novel variants of location attention building on top&#10;of Dubois et al. (2020) to address the new diagnostic tasks. We also show the&#10;benefits of our approaches for length generalization in SCAN (Lake &amp; Baroni,&#10;2018) and CFQ (Keysers et al., 2020). Our code is available on GitHub.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.16028" label="2310.16028">
        <attvalues>
          <attvalue for="0" value="What Algorithms can Transformers Learn? A Study in Length Generalization" />
          <attvalue for="1" value="  Large language models exhibit surprising emergent generalization properties,&#10;yet also struggle on many simple reasoning tasks such as arithmetic and parity.&#10;This raises the question of if and when Transformer models can learn the true&#10;algorithm for solving a task. We study the scope of Transformers' abilities in&#10;the specific setting of length generalization on algorithmic tasks. Here, we&#10;propose a unifying framework to understand when and how Transformers can&#10;exhibit strong length generalization on a given task. Specifically, we leverage&#10;RASP (Weiss et al., 2021) -- a programming language designed for the&#10;computational model of a Transformer -- and introduce the RASP-Generalization&#10;Conjecture: Transformers tend to length generalize on a task if the task can be&#10;solved by a short RASP program which works for all input lengths. This simple&#10;conjecture remarkably captures most known instances of length generalization on&#10;algorithmic tasks. Moreover, we leverage our insights to drastically improve&#10;generalization performance on traditionally hard tasks (such as parity and&#10;addition). On the theoretical side, we give a simple example where the&#10;&quot;min-degree-interpolator&quot; model of learning from Abbe et al. (2023) does not&#10;correctly predict Transformers' out-of-distribution behavior, but our&#10;conjecture does. Overall, our work provides a novel perspective on the&#10;mechanisms of compositional generalization and the algorithmic capabilities of&#10;Transformers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.10884" label="2301.10884">
        <attvalues>
          <attvalue for="0" value="Break It Down: Evidence for Structural Compositionality in Neural&#10;  Networks" />
          <attvalue for="1" value="  Though modern neural networks have achieved impressive performance in both&#10;vision and language tasks, we know little about the functions that they&#10;implement. One possibility is that neural networks implicitly break down&#10;complex tasks into subroutines, implement modular solutions to these&#10;subroutines, and compose them into an overall solution to a task - a property&#10;we term structural compositionality. Another possibility is that they may&#10;simply learn to match new inputs to learned templates, eliding task&#10;decomposition entirely. Here, we leverage model pruning techniques to&#10;investigate this question in both vision and language across a variety of&#10;architectures, tasks, and pretraining regimens. Our results demonstrate that&#10;models often implement solutions to subroutines via modular subnetworks, which&#10;can be ablated while maintaining the functionality of other subnetworks. This&#10;suggests that neural networks may be able to learn compositionality, obviating&#10;the need for specialized symbolic mechanisms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.09395" label="2009.09395">
        <attvalues>
          <attvalue for="0" value="Far-Field Automatic Speech Recognition" />
          <attvalue for="1" value="  The machine recognition of speech spoken at a distance from the microphones,&#10;known as far-field automatic speech recognition (ASR), has received a&#10;significant increase of attention in science and industry, which caused or was&#10;caused by an equally significant improvement in recognition accuracy. Meanwhile&#10;it has entered the consumer market with digital home assistants with a spoken&#10;language interface being its most prominent application. Speech recorded at a&#10;distance is affected by various acoustic distortions and, consequently, quite&#10;different processing pipelines have emerged compared to ASR for close-talk&#10;speech. A signal enhancement front-end for dereverberation, source separation&#10;and acoustic beamforming is employed to clean up the speech, and the back-end&#10;ASR engine is robustified by multi-condition training and adaptation. We will&#10;also describe the so-called end-to-end approach to ASR, which is a new&#10;promising architecture that has recently been extended to the far-field&#10;scenario. This tutorial article gives an account of the algorithms used to&#10;enable accurate speech recognition from a distance, and it will be seen that,&#10;although deep learning has a significant share in the technological&#10;breakthroughs, a clever combination with traditional signal processing can lead&#10;to surprisingly effective solutions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.10599" label="1910.10599">
        <attvalues>
          <attvalue for="0" value="End-to-end architectures for ASR-free spoken language understanding" />
          <attvalue for="1" value="  Spoken Language Understanding (SLU) is the problem of extracting the meaning&#10;from speech utterances. It is typically addressed as a two-step problem, where&#10;an Automatic Speech Recognition (ASR) model is employed to convert speech into&#10;text, followed by a Natural Language Understanding (NLU) model to extract&#10;meaning from the decoded text. Recently, end-to-end approaches were emerged,&#10;aiming at unifying the ASR and NLU into a single SLU deep neural architecture,&#10;trained using combinations of ASR and NLU-level recognition units. In this&#10;paper, we explore a set of recurrent architectures for intent classification,&#10;tailored to the recently introduced Fluent Speech Commands (FSC) dataset, where&#10;intents are formed as combinations of three slots (action, object, and&#10;location). We show that by combining deep recurrent architectures with standard&#10;data augmentation, state-of-the-art results can be attained, without using&#10;ASR-level targets or pretrained ASR models. We also investigate its&#10;generalizability to new wordings, and we show that the model can perform&#10;reasonably well on wordings unseen during training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.06833" label="1809.06833">
        <attvalues>
          <attvalue for="0" value="Advancing Multi-Accented LSTM-CTC Speech Recognition using a Domain&#10;  Specific Student-Teacher Learning Paradigm" />
          <attvalue for="1" value="  Non-native speech causes automatic speech recognition systems to degrade in&#10;performance. Past strategies to address this challenge have considered model&#10;adaptation, accent classification with a model selection, alternate&#10;pronunciation lexicon, etc. In this study, we consider a recurrent neural&#10;network (RNN) with connectionist temporal classification (CTC) cost function&#10;trained on multi-accent English data including US (Native), Indian and Hispanic&#10;accents. We exploit dark knowledge from a model trained with the multi-accent&#10;data to train student models under the guidance of both a teacher model and CTC&#10;cost of target transcription. We show that transferring knowledge from a single&#10;RNN-CTC trained model toward a student model, yields better performance than&#10;the stand-alone teacher model. Since the outputs of different trained CTC&#10;models are not necessarily aligned, it is not possible to simply use an&#10;ensemble of CTC teacher models. To address this problem, we train accent&#10;specific models under the guidance of a single multi-accent teacher, which&#10;results in having multiple aligned and trained CTC models. Furthermore, we&#10;train a student model under the supervision of the accent-specific teachers,&#10;resulting in an even further complementary model, which achieves +20.1%&#10;relative Character Error Rate (CER) reduction compared to the baseline trained&#10;without any teacher. Having this effective multi-accent model, we can achieve&#10;further improvement for each accent by adapting the model to each accent. Using&#10;the accent specific model's outputs to regularize the adapting process (i.e., a&#10;knowledge distillation version of Kullback-Leibler (KL) divergence) results in&#10;even superior performance compared to the conventional approach using general&#10;teacher models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.07047" label="1910.07047">
        <attvalues>
          <attvalue for="0" value="Analyzing Large Receptive Field Convolutional Networks for Distant&#10;  Speech Recognition" />
          <attvalue for="1" value="  Despite significant efforts over the last few years to build a robust&#10;automatic speech recognition (ASR) system for different acoustic settings, the&#10;performance of the current state-of-the-art technologies significantly degrades&#10;in noisy reverberant environments.&#10;  Convolutional Neural Networks (CNNs) have been successfully used to achieve&#10;substantial improvements in many speech processing applications including&#10;distant speech recognition (DSR). However, standard CNN architectures were not&#10;efficient in capturing long-term speech dynamics, which are essential in the&#10;design of a robust DSR system. In the present study, we address this issue by&#10;investigating variants of large receptive field CNNs (LRF-CNNs) which include&#10;deeply recursive networks, dilated convolutional neural networks, and stacked&#10;hourglass networks. To compare the efficacy of the aforementioned architectures&#10;with the standard CNN for Wall Street Journal (WSJ) corpus, we use a hybrid&#10;DNN-HMM based speech recognition system. We extend the study to evaluate the&#10;system performances for distant speech simulated using realistic room impulse&#10;responses (RIRs). Our experiments show that with fixed number of parameters&#10;across all architectures, the large receptive field networks show consistent&#10;improvements over the standard CNNs for distant speech. Amongst the explored&#10;LRF-CNNs, stacked hourglass network has shown improvements with a 8.9% relative&#10;reduction in word error rate (WER) and 10.7% relative improvement in frame&#10;accuracy compared to the standard CNNs for distant simulated speech signals.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.20620" label="2310.20620">
        <attvalues>
          <attvalue for="0" value="The Unreasonable Effectiveness of Random Target Embeddings for&#10;  Continuous-Output Neural Machine Translation" />
          <attvalue for="1" value="  Continuous-output neural machine translation (CoNMT) replaces the discrete&#10;next-word prediction problem with an embedding prediction. The semantic&#10;structure of the target embedding space (i.e., closeness of related words) is&#10;intuitively believed to be crucial. We challenge this assumption and show that&#10;completely random output embeddings can outperform laboriously pretrained ones,&#10;especially on larger datasets. Further investigation shows this surprising&#10;effect is strongest for rare words, due to the geometry of their embeddings. We&#10;shed further light on this finding by designing a mixed strategy that combines&#10;random and pre-trained embeddings for different tokens.&#10;" />
          <attvalue for="2" value="&#10;Since text is naturally discrete, \ie, each token in a target sentence is represented by an integer index in the vocabulary, neural machine translation (NMT), as many other language generation tasks, is trained mainly as a discrete-output model with softmax over the full vocabulary followed by the cross-entropy loss. Continuous-output neural machine translation (CoNMT) models, in contrast, are trained to predict the continuous representation based on the distances between vectors. It is an appealing line of study for computational and modeling related reasons~\cite{kumar2018von}, as well as a reliable test bed for exploring the properties of &#10;continuous language spaces &#10;that appear in modern deep generative models~\cite{Li-2022-DiffusionLM}.&#10;However, CoNMT introduces its own challenge, namely mapping to and from a continuous space. During training, CoNMT model requires continuous targets, and while decoding, one needs to map back to the discrete text representation.&#10;&#10;Text mapping to continuous space is widely explored in NLP and can be done using embeddings of tokens, words~\cite{turian-etal-2010-word,mikolov-w2v,mikolov-etal-2018-advances} and sentences~\cite{reimers-gurevych-2019-sentence,feng-etal-2022-language}.&#10;Cosine similarity between word embeddings is well correlated with lexical similarity metrics, motivating the use of cosine distance against pre-trained embeddings&#10;as an effective training strategy for CoNMT&#10;Nearest neighbor beam decoding would in this case include related words and, unlike discrete cross-entropy, the training strategy does not discourage synonyms.&#10;&#10;Previous studies show that the quality of continuous-output models highly depends on the choice of embeddings~\cite{Li-2022-DiffusionLM, tokarchuk-niculae-2022-target, kumar2018von}. In general, in CoNMT the embeddings are pre-trained and fixed: otherwise, making all embeddings equal yields an unwanted global optimum. Obtaining pre-trained word embeddings can be computationally expensive, especially if one needs to train an embeddings model from scratch.&#10;&#10;In this work we randomly initialize target embeddings for continuous-output models and keep them static during training.&#10;\cite{arora-etal-2020-contextual} applied static random embeddings for text classification model's input; however, to the best of our knowledge, the effect of untrained random target embeddings has not been previously studied in the literature, especially for text-generating tasks such as machine translation.&#10;However, we show that random target embeddings perform close to their pre-trained counterpart, and even surpass them &#10;on the larger datasets, challenging the assumption that target embeddings must preserve semantic relationships.&#10;Meaningful structures in target embedding space could help with generalization, but our results suggest that any such benefits are smaller than one might expect, and sensitive to embedding concentration.&#10;We hypothesize and bring experimental evidence that CoNMT performance is negatively impacted when there is too little space around embeddings, \ie, when embeddings are tangled rather than more spread out.&#10;Our findings on three NMT tasks, namely WMT 2018 English$arrow$Turkish (en-tr), WMT 2016 English$arrow$Romanian (en-ro), and WMT 2019 English$arrow$German (en-de) indicate that random embeddings&#10;are more spread out and perform better on rare words for all language pairs. &#10;Strikingly, on the largest dataset (en-de), random embeddings show the largest gain over pre-trained ones.&#10;We propose a simple yet efficient combination of random and pre-trained embeddings, and show that it improves model performance &#10;in most cases considered.&#10;More generally, our findings show that dispersion is an important property of embedding space geometry,&#10;and that integrating semantic information should be done with care.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Embedding Space Geometry, Computer Science, Linguistics, Language Modeling Strategies, Artificial Intelligence, Mathematics, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2205.14217" label="2205.14217">
        <attvalues>
          <attvalue for="0" value="Diffusion-LM Improves Controllable Text Generation" />
          <attvalue for="1" value="  Controlling the behavior of language models (LMs) without re-training is a&#10;major open problem in natural language generation. While recent works have&#10;demonstrated successes on controlling simple sentence attributes (e.g.,&#10;sentiment), there has been little progress on complex, fine-grained controls&#10;(e.g., syntactic structure). To address this challenge, we develop a new&#10;non-autoregressive language model based on continuous diffusions that we call&#10;Diffusion-LM. Building upon the recent successes of diffusion models in&#10;continuous domains, Diffusion-LM iteratively denoises a sequence of Gaussian&#10;vectors into word vectors, yielding a sequence of intermediate latent&#10;variables. The continuous, hierarchical nature of these intermediate variables&#10;enables a simple gradient-based algorithm to perform complex, controllable&#10;generation tasks. We demonstrate successful control of Diffusion-LM for six&#10;challenging fine-grained control tasks, significantly outperforming prior work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.09405" label="1712.09405">
        <attvalues>
          <attvalue for="0" value="Advances in Pre-Training Distributed Word Representations" />
          <attvalue for="1" value="  Many Natural Language Processing applications nowadays rely on pre-trained&#10;word representations estimated from large text corpora such as news&#10;collections, Wikipedia and Web Crawl. In this paper, we show how to train&#10;high-quality word vector representations by using a combination of known tricks&#10;that are however rarely used together. The main result of our work is the new&#10;set of publicly available pre-trained models that outperform the current state&#10;of the art by a large margin on a number of tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09117" label="2005.09117">
        <attvalues>
          <attvalue for="0" value="Contextual Embeddings: When Are They Worth It?" />
          <attvalue for="1" value="  We study the settings for which deep contextual embeddings (e.g., BERT) give&#10;large improvements in performance relative to classic pretrained embeddings&#10;(e.g., GloVe), and an even simpler baseline---random word embeddings---focusing&#10;on the impact of the training set size and the linguistic properties of the&#10;task. Surprisingly, we find that both of these simpler baselines can match&#10;contextual embeddings on industry-scale data, and often perform within 5 to 10%&#10;accuracy (absolute) on benchmark tasks. Furthermore, we identify properties of&#10;data for which contextual embeddings give particularly large gains: language&#10;containing complex structure, ambiguous word usage, and words unseen in&#10;training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.02855" label="2105.02855">
        <attvalues>
          <attvalue for="0" value="Adapting Monolingual Models: Data can be Scarce when Language Similarity&#10;  is High" />
          <attvalue for="1" value="  For many (minority) languages, the resources needed to train large models are&#10;not available. We investigate the performance of zero-shot transfer learning&#10;with as little data as possible, and the influence of language similarity in&#10;this process. We retrain the lexical layers of four BERT-based models using&#10;data from two low-resource target language varieties, while the Transformer&#10;layers are independently fine-tuned on a POS-tagging task in the model's source&#10;language. By combining the new lexical layers and fine-tuned Transformer&#10;layers, we achieve high task performance for both target languages. With high&#10;language similarity, 10MB of data appears sufficient to achieve substantial&#10;monolingual transfer performance. Monolingual BERT-based models generally&#10;achieve higher downstream task performance after retraining the lexical layer&#10;than multilingual BERT, even when the target language is included in the&#10;multilingual model.&#10;" />
          <attvalue for="2" value="&#10;Large pre-trained language models are the dominant approach for solving many tasks in natural language processing.&#10;These models represent linguistic structure on the basis of large corpora that exist for high-resource languages, such as English.&#10;However, for the majority of the world's languages, these large corpora are not available.&#10;&#10;Past work on multilingual learning has found that multilingual BERT~(mBERT; \cite{devlin2019-mbert}) generalizes across languages with high zero-shot transfer performance on a variety of tasks~\cite{pires-etal-2019-multilingual, wu-dredze-2019-beto}.&#10;However, it has also been observed that &#10;high-resource languages included in mBERT pre-training often have a better-performing monolingual model, and low-resource languages that are not included in mBERT pre-training usually show poor performance \cite{nozza2020mask, wu-dredze-2020-languages}.&#10;&#10;An alternative to multilingual transfer learning is the adaptation of existing monolingual models to other languages.&#10;\cite{zoph-etal-2016-transfer} introduce a method for transferring a pre-trained machine translation model to lower-resource languages by only fine-tuning the lexical layer.&#10;This method has also been applied to BERT \cite{artetxe-etal-2020-cross} and GPT-2 \cite{devries2020good}.&#10;\cite{artetxe-etal-2020-cross} also show that BERT models with retrained lexical layers perform well in downstream tasks, but comparatively high performance has only been demonstrated for languages for which at least 400MB of data is available.&#10;&#10;To test if this procedure is also effective for low- to zero-resource languages, we consider two regional language varieties spoken in the North of the Netherlands, namely Gronings (Low Saxon language variant) and West Frisian.&#10;&#10;Figure~\ref{appendix:fig:map} visualizes the geographical areas where these regional language variants are spoken. The regional Low Saxon language is spoken in the north-eastern provinces of the Netherlands and in the North of Germany (shown in yellow). As part of the Low Saxon language, Gronings is spoken in the province of Groningen (highlighted in green). The West Frisian language is spoken in the province of Friesland (shown in red), and it is the second official language of the Netherlands, next to Dutch. Dutch is the national language of the Netherlands, and it is spoken in every province of the Netherlands and in Flanders (North of Belgium).&#10;&#10;For both Gronings and West Frisian limited data is available. &#10;In addition to unlabeled data, for both target languages we have a small collection of annotated part-of-speech (POS) tagging data, which we use for evaluating zero-shot model transfer.&#10;We use three monolingual BERT models (source languages English, German, Dutch) and mBERT to investigate if linguistic structure can be transferred to Gronings and West Frisian by learning new sub-word embeddings.&#10;Our model source and target languages are closely related West Germanic languages \cite{eberhard_ethnologue_2020}.&#10;In Table~\ref{tab:examples}, we show parallel sentences in Gronings, West Frisian, Dutch, German, and English to illustrate the lexical similarity between these languages. &#10;Additionally, the examples show that there are some lexical and syntactic differences.&#10;&#10;We also evaluate to what extent the similarity between each source language of the monolingual models and the target languages is relevant for transferring monolingual representations, and assess the minimum amount of data necessary to adapt these models.&#10;&#10;Our pre-trained models for Gronings and West Frisian (which did not yet exist) are released. &#10;Additionally, our code is publicly available for bringing language models to other low-resource languages at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Modeling, Low-Resource Languages, Language Transfer Learning, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2003.02912" label="2003.02912">
        <attvalues>
          <attvalue for="0" value="What the [MASK]? Making Sense of Language-Specific BERT Models" />
          <attvalue for="1" value="  Recently, Natural Language Processing (NLP) has witnessed an impressive&#10;progress in many areas, due to the advent of novel, pretrained contextual&#10;representation models. In particular, Devlin et al. (2019) proposed a model,&#10;called BERT (Bidirectional Encoder Representations from Transformers), which&#10;enables researchers to obtain state-of-the art performance on numerous NLP&#10;tasks by fine-tuning the representations on their data set and task, without&#10;the need for developing and training highly-specific architectures. The authors&#10;also released multilingual BERT (mBERT), a model trained on a corpus of 104&#10;languages, which can serve as a universal language model. This model obtained&#10;impressive results on a zero-shot cross-lingual natural inference task. Driven&#10;by the potential of BERT models, the NLP community has started to investigate&#10;and generate an abundant number of BERT models that are trained on a particular&#10;language, and tested on a specific data domain and task. This allows us to&#10;evaluate the true potential of mBERT as a universal language model, by&#10;comparing it to the performance of these more specific models. This paper&#10;presents the current state of the art in language-specific BERT models,&#10;providing an overall picture with respect to different dimensions (i.e.&#10;architectures, data domains, and tasks). Our aim is to provide an immediate and&#10;straightforward overview of the commonalities and differences between&#10;Language-Specific (language-specific) BERT models and mBERT. We also provide an&#10;interactive and constantly updated website that can be used to explore the&#10;information we have collected, at https://bertlang.unibocconi.it.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.12005" label="2401.12005">
        <attvalues>
          <attvalue for="0" value="ALMs: Authorial Language Models for Authorship Attribution" />
          <attvalue for="1" value="  In this paper, we introduce an authorship attribution method called Authorial&#10;Language Models (ALMs) that involves identifying the most likely author of a&#10;questioned document based on the perplexity of the questioned document&#10;calculated for a set of causal language models fine-tuned on the writings of a&#10;set of candidate author. We benchmarked ALMs against state-of-art-systems using&#10;the CCAT50 dataset and the Blogs50 datasets. We find that ALMs achieves a&#10;macro-average accuracy score of 83.6% on Blogs50, outperforming all other&#10;methods, and 74.9% on CCAT50, matching the performance of the best method. To&#10;assess the performance of ALMs on shorter texts, we also conducted text&#10;ablation testing. We found that to reach a macro-average accuracy of 70%, ALMs&#10;needs 40 tokens on Blogs50 and 400 tokens on CCAT50, while to reach 60% ALMs&#10;requires 20 tokens on Blogs50 and 70 tokens on CCAT50.&#10;" />
          <attvalue for="2" value="&#10;&#10;For over a century, researchers have developed methods for authorship attribution to resolve cases of disputed authorship by comparing the style of a questioned document to writing samples from a set of candidate authors \cite{Juola_2006,Stamatatos_2009}. The goal of authorship attribution is to identify the candidate whose style of writing is most similar to a questioned document. Stylometry is the quantitative analysis of style and is a common approach to authorship attribution \cite{Juola_2006,Stamatatos_2009}. A wide range of different measurements and methods for authorship attribution have been developed in stylometry \cite{Grieve_2007,Stamatatos_2009}. Popular techniques include Principal Component Analysis of function word frequencies \cite{Binongo_2003,grieve2023_language} and distance-based comparisons of the frequencies of common words \cite{Argamon_2007,Burrows_2002}. &#10;&#10;Although stylometric approaches are useful for resolving certain types of authorship attribution tasks, there are clear limitations with these techniques. Overall performance declines dramatically when the number of candidate authors increases \cite{Grieve_2007,Luyckx_Daelemans_2011}, when the length of the question document decreases \cite{Eder_2015,grieve2017}, and when the amount of training data from the candidate authors decreases \cite{Luyckx_Daelemans_2011,grieve2017}.&#10;&#10;Recent research in authorship analysis has begun to explore the use of modern Large Language Models (LLMs) to address these issues. Examples include universal authorial embeddings using Siamese BERT \cite{Rivera-Soto_Miano_Ordonez_Chen_Khan_Bishop_Andrews_2021} and Character BERT \cite{El_Boukkouri_Ferret_Lavergne_Noji_Zweigenbaum_Tsujii_2020}, and using BERT for classification \cite{Fabien_Villatoro-Tello_Motlicek_Parida_2020,Tyo_Dhingra_Lipton_2022}. LLM predictability metrics, such as perplexity and cross-entropy, have also been tested in a small number of studies. \cite{Fourkioti_Symeonidis_Arampatzis_2019} found that the perplexity of a single LLM pretrained on PoS-tags can be effective for authorship attribution. \cite{Barlas_Stamatatos_2020} extended this approach by training a multi-head classifier using the cross entropy of a single pretrained LLM, achieving their best performance using BERT, although they also considered other LLMs, including causal language models. Subsequently, \cite{Tyo_Dhingra_Lipton_2022} included \cite{Barlas_Stamatatos_2020}'s BERT-based approach, which they referred to as pALM (per Author Language Model), in their state-of-the-art authorship attribution benchmarking study, but found that pALM has the worst performance of all methods considered, which included an n-gram based classifier (Ngram) \cite{Tyo_Dhingra_Lipton_2022}, a prediction by partial matching compression model (PPM)\cite{croft_using_2003,neal_surveying_2018}, and a pre-trained BERT model with a dense layer for classification (BERT)\cite{Fabien_Villatoro-Tello_Motlicek_Parida_2020}. &#10;&#10;Although previous research has had relatively little success using LLM predictability metrics for human authorship attribution, this approach currently underlies state-of-the-art methods for LLM detection. LLM detection is a new type of attribution task that involves identifying whether a questioned text was written by a human or an LLM. The task has gained prominence in recent years due to increasing concerns about the misuse of LLMs \cite{Bommasani_Hudson_Adeli_Altman_Arora_von_Arx_Bernstein_Bohg_Bosselut_Brunskill__2022,Gehrmann_Strobelt_Rush_2019,Tian_Chen_Wang_Bai_Zhang_Li_Xu_Wang_2023,Wu_Pang_Shen_Cheng_Chua_2023,Gehrmann_Strobelt_Rush_2019,Wu_Pang_Shen_Cheng_Chua_2023}. In these studies, casual language model perplexity has been found to be an effective indicator of authorship, where LLM-authored texts tend to be associated with relatively low perplexity scores in comparison to human-authored texts: LLM-authored texts are generally more predictable to a LLM. Approaches include both fully automated detection and computer-assisted detection, e.g., GLTR \cite{Gehrmann_Strobelt_Rush_2019} and GPTZero \cite{chakraborty_possibilities_2023}. &#10;&#10;Building on this research, in this paper, we revisit the idea of using LLM perplexity for human authorship attribution. However, rather than work with a single LLM, as has been the case in previous research on both human and machine attribution, we build a set of adapted Authorial Language Models (ALMs), each of which is fine-tuned on the writings of a single candidate author. Instead of computing the perplexity or cross entropy based on a single LLM, our approach involves predicting the authorship of a questioned document by comparing the perplexities of this text for multiple ALMs, selecting the author whose associated LLM yields the lowest perplexity. We benchmark ALMs on the standard Blogs50 and CCAT50 datasets, following \cite{Tyo_Dhingra_Lipton_2022}, finding that our approach achieves state-of-the-art overall performance. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Authorship Attribution, Computer Science, Linguistics, Text Analysis, Natural Language Processing, Language Modeling, Statistics" />
        </attvalues>
      </node>
      <node id="2209.06869" label="2209.06869">
        <attvalues>
          <attvalue for="0" value="On the State of the Art in Authorship Attribution and Authorship&#10;  Verification" />
          <attvalue for="1" value="  Despite decades of research on authorship attribution (AA) and authorship&#10;verification (AV), inconsistent dataset splits/filtering and mismatched&#10;evaluation methods make it difficult to assess the state of the art. In this&#10;paper, we present a survey of the fields, resolve points of confusion,&#10;introduce Valla that standardizes and benchmarks AA/AV datasets and metrics,&#10;provide a large-scale empirical evaluation, and provide apples-to-apples&#10;comparisons between existing methods. We evaluate eight promising methods on&#10;fifteen datasets (including distribution-shifted challenge sets) and introduce&#10;a new large-scale dataset based on texts archived by Project Gutenberg.&#10;Surprisingly, we find that a traditional Ngram-based model performs best on 5&#10;(of 7) AA tasks, achieving an average macro-accuracy of $76.50\%$ (compared to&#10;$66.71\%$ for a BERT-based model). However, on the two AA datasets with the&#10;greatest number of words per author, as well as on the AV datasets, BERT-based&#10;models perform best. While AV methods are easily applied to AA, they are seldom&#10;included as baselines in AA papers. We show that through the application of&#10;hard-negative mining, AV methods are competitive alternatives to AA methods.&#10;Valla and all experiment code can be found here:&#10;https://github.com/JacobTyo/Valla&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.04043" label="1906.04043">
        <attvalues>
          <attvalue for="0" value="GLTR: Statistical Detection and Visualization of Generated Text" />
          <attvalue for="1" value="  The rapid improvement of language models has raised the specter of abuse of&#10;text generation systems. This progress motivates the development of simple&#10;methods for detecting generated text that can be used by and explained to&#10;non-experts. We develop GLTR, a tool to support humans in detecting whether a&#10;text was generated by a model. GLTR applies a suite of baseline statistical&#10;methods that can detect generation artifacts across common sampling schemes. In&#10;a human-subjects study, we show that the annotation scheme provided by GLTR&#10;improves the human detection-rate of fake text from 54% to 72% without any&#10;prior training. GLTR is open-source and publicly deployed, and has already been&#10;widely used to detect generated outputs&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18149" label="2305.18149">
        <attvalues>
          <attvalue for="0" value="Multiscale Positive-Unlabeled Detection of AI-Generated Texts" />
          <attvalue for="1" value="  Recent releases of Large Language Models (LLMs), e.g. ChatGPT, are&#10;astonishing at generating human-like texts, but they may impact the&#10;authenticity of texts. Previous works proposed methods to detect these&#10;AI-generated texts, including simple ML classifiers, pretrained-model-based&#10;zero-shot methods, and finetuned language classification models. However,&#10;mainstream detectors always fail on short texts, like SMSes, Tweets, and&#10;reviews. In this paper, a Multiscale Positive-Unlabeled (MPU) training&#10;framework is proposed to address the difficulty of short-text detection without&#10;sacrificing long-texts. Firstly, we acknowledge the human-resemblance property&#10;of short machine texts, and rephrase AI text detection as a partial&#10;Positive-Unlabeled (PU) problem by regarding these short machine texts as&#10;partially ``unlabeled&quot;. Then in this PU context, we propose the&#10;length-sensitive Multiscale PU Loss, where a recurrent model in abstraction is&#10;used to estimate positive priors of scale-variant corpora. Additionally, we&#10;introduce a Text Multiscaling module to enrich training corpora. Experiments&#10;show that our MPU method augments detection performance on long AI-generated&#10;texts, and significantly improves short-text detection of language model&#10;detectors. Language Models trained with MPU could outcompete existing detectors&#10;on various short-text and long-text detection benchmarks. The codes are&#10;available at&#10;https://github.com/mindspore-lab/mindone/tree/master/examples/detect_chatgpt&#10;and https://github.com/YuchuanTian/AIGC_text_detector.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.04736" label="2304.04736">
        <attvalues>
          <attvalue for="0" value="On the Possibilities of AI-Generated Text Detection" />
          <attvalue for="1" value="  Our work addresses the critical issue of distinguishing text generated by&#10;Large Language Models (LLMs) from human-produced text, a task essential for&#10;numerous applications. Despite ongoing debate about the feasibility of such&#10;differentiation, we present evidence supporting its consistent achievability,&#10;except when human and machine text distributions are indistinguishable across&#10;their entire support. Drawing from information theory, we argue that as&#10;machine-generated text approximates human-like quality, the sample size needed&#10;for detection increases. We establish precise sample complexity bounds for&#10;detecting AI-generated text, laying groundwork for future research aimed at&#10;developing advanced, multi-sample detectors. Our empirical evaluations across&#10;multiple datasets (Xsum, Squad, IMDb, and Kaggle FakeNews) confirm the&#10;viability of enhanced detection methods. We test various state-of-the-art text&#10;generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and&#10;Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector,&#10;GPTZero. Our findings align with OpenAI's empirical data related to sequence&#10;length, marking the first theoretical substantiation for these observations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00851" label="2005.00851">
        <attvalues>
          <attvalue for="0" value="A language score based output selection method for multilingual speech&#10;  recognition" />
          <attvalue for="1" value="  The quality of a multilingual speech recognition system can be improved by&#10;adaptation methods if the input language is specified. For systems that can&#10;accept multilingual inputs, the popular approach is to apply a language&#10;identifier to the input then switch or configure decoders in the next step, or&#10;use one more subsequence model to select the output from a set of candidates.&#10;Motivated by the goal of reducing the latency for real-time applications, in&#10;this paper, a language model rescoring method is firstly applied to produce all&#10;possible candidates for target languages, then a simple score is proposed to&#10;automatically select the output without any identifier model or language&#10;specification of the input language. The main point is that this score can be&#10;simply and automatically estimated on-the-fly so that the whole decoding&#10;pipeline is more simple and compact. Experimental results showed that this&#10;method can achieve the same quality as when the input language is specified. In&#10;addition, we present to design an English and Vietnamese End-to-End model to&#10;deal with not only the problem of cross-lingual speakers but also as a solution&#10;to improve the accuracy of borrowed words of English in Vietnamese.&#10;" />
          <attvalue for="2" value="&#10;&#10;One of the challenges for speech recognition systems is that the cross-lingual input in systems such as machine translation or dialog systems. Besides, for non-English speaking countries, there is another problem that is the borrowing of words in English used in daily conversations. That is why building a system that accepts multilingual input has been attracting a lot of research. Studies can be divided into two main directions. The first direction is to enhance the quality of monolingual models by leveraging or sharing resources from other languages \cite{go_2019_ml,2019_jh_ml,8114354}. The second is to build models that can accept multilingual input when spoken language changes happen at either word or utterance level. In this work, we follow the second approach to be able to use the system in real-life applications such as smart home, machine translation, or dialog systems. A system followed by the second approach usually has two main parts which are identifier and recognizer. The identifier is to detect the input language from which to select or configure the decoder model of the corresponding language for the next recognition step \cite{go_2019_ml,8114354,6289013,8268945} or do a post-evaluation \cite{Waibel2000,Niesler2006LanguageIA} to select a result from outputs of the front decoder. In this architecture, the identifier is often a statistic model and needs to be trained. Therefore, it will take more time and resources during the development and operation of the system. Motivated by this reason, in this work we propose a score to automatically select the result for the target language from outputs of a multilingual end-to-end (E2E) acoustic model using a single DNN shared for all languages. The benefit of this method is that the score is calculated simply and automatically based on language models and is performed on-the-fly during the decoding process. So the system using our method could be considered as only a single decoder, and will be more compact and less latency. We also present an approach to construct a multilingual E2E model for English and Vietnamese for not only improving the performance but also improving the accuracy of borrowed English or foreign words in Vietnamese.&#10;&#10;The rest of this paper is organized as follows. Section 2&#10;describes our proposed method. The experimental setup with a multilingual model for English and Vietnamese is explained in Section 3. Finally, the conclusions are presented in Section 4.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="End-to-End Modeling, Computer Science, Linguistics, Engineering, Multilingual Speech Recognition, Artificial Intelligence, Language Model Rescoring" />
        </attvalues>
      </node>
      <node id="1908.01060" label="1908.01060">
        <attvalues>
          <attvalue for="0" value="Multilingual Speech Recognition with Corpus Relatedness Sampling" />
          <attvalue for="1" value="  Multilingual acoustic models have been successfully applied to low-resource&#10;speech recognition. Most existing works have combined many small corpora&#10;together and pretrained a multilingual model by sampling from each corpus&#10;uniformly. The model is eventually fine-tuned on each target corpus. This&#10;approach, however, fails to exploit the relatedness and similarity among&#10;corpora in the training set. For example, the target corpus might benefit more&#10;from a corpus in the same domain or a corpus from a close language. In this&#10;work, we propose a simple but useful sampling strategy to take advantage of&#10;this relatedness. We first compute the corpus-level embeddings and estimate the&#10;similarity between each corpus. Next, we start training the multilingual model&#10;with uniform-sampling from each corpus at first, then we gradually increase the&#10;probability to sample from related corpora based on its similarity with the&#10;target corpus. Finally, the model would be fine-tuned automatically on the&#10;target corpus. Our sampling strategy outperforms the baseline multilingual&#10;model on 16 low-resource tasks. Additionally, we demonstrate that our corpus&#10;embeddings capture the language and domain information of each corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Multilingual speech recognition has explored various models to share parameters across languages in different ways. For example, parameters can be shared by using posterior features from other languages \cite{stolcke2006cross}, applying the same GMM components across different HMM states \cite{burget2010multilingual}, training shared hidden layers in DNNs \cite{huang2013cross, heigold2013multilingual} or LSTM \cite{dalmia2018sequence}, using language independent bottleneck features \cite{vesely2012language, dalmia2018domain}. Some models only share their hidden layers, but use separate output layers to predict their phones \cite{huang2013cross, heigold2013multilingual}. Other models have only one shared output layer to predict the universal phone set shared by all languages \cite{schultz2001language, tong2017investigation, vu2013multilingual}. While those works proposed the multilingual models in different ways, few of them have explicitly exploited the relatedness across various languages and corpora. In contrast, our work computes the relatedness between different corpora using the embedding representations and exploits them efficiently.&#10;&#10;The embedding representations have been heavily used in multiple fields. In particular, embeddings of multiple granularities have been explored in many NLP tasks. To name a few, character embedding \cite{kim2016character}, subword embedding \cite{bojanowski2017enriching}, sentence embedding \cite{kiros2015skip} and document embedding \cite{le2014distributed}. However, there are few works exploring the corpus level embeddings. The main reason is that the number of corpora involved in most experiments is usually limited and it is not useful to compute corpus embeddings. The only exception is the multitask learning where many tasks and corpora are combined together. For instance, the language level (corpus level) embedding can be generated along with the model in machine translation \cite{johnson2017google} and speech recognition \cite{li2018multi}. However, those embeddings are only used as an auxiliary feature to the model, few works continue to exploit those embeddings themselves.&#10;&#10;Another important aspect of our work is that we focused on the sampling strategy for speech recognition. While most of the previous speech works mainly emphasized the acoustic modeling side, there are also some attempts focusing on the sampling strategies. For instance, curriculum learning would train the acoustic model by starting from easy training samples and increasingly adapt it to more difficult samples \cite{amodei2016deep, braun2017curriculum}. Active learning is an approach trying to minimize human costs to collect transcribed speech data \cite{riccardi2005active}. Furthermore, sampling strategies can also be helpful to speed up the training process \cite{cui2015multilingual}. However, the goals of most strategies are to improve the acoustic model by modifying the sampling distribution within a single speech corpus for a single language. On the contrary, our approach aims to optimize the multilingual acoustic model by modifying distributions across all the training corpora.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2210.04243" label="2210.04243">
        <attvalues>
          <attvalue for="0" value="Fine-Tuning Pre-trained Transformers into Decaying Fast Weights" />
          <attvalue for="1" value="  Autoregressive Transformers are strong language models but incur O(T)&#10;complexity during per-token generation due to the self-attention mechanism.&#10;Recent work proposes kernel-based methods to approximate causal self-attention&#10;by replacing it with recurrent formulations with various update rules and&#10;feature maps to achieve O(1) time and memory complexity. We explore these&#10;approaches and find that they are unnecessarily complex, and propose a simple&#10;alternative - decaying fast weights - that runs fast on GPU, outperforms prior&#10;methods, and retains 99% of attention's performance for GPT-2. We also show&#10;competitive performance on WikiText-103 against more complex attention&#10;substitutes.&#10;" />
          <attvalue for="2" value="&#10;Autoregressive Transformers \cite{vaswani2017attention} have demonstrated strong performance on text generation \cite{brown2020language}.&#10;The success of self-attention in Transformers over recurrent models \cite{hochreiter1997long} can be attributed to its parallelizability \cite{hooker2021hardware} and its effective gradient propagation over many time steps \cite{ke2018sparse}.&#10;However, self-attention has a high computation and memory cost. During inference sampling, it consumes $O(T)$ time and memory and grows linearly per token generated.&#10;&#10;These drawbacks motivated recent work to convert or fine-tune attention into recurrent formulations with $O(1)$ memory and time complexity for auto-regressive generation.&#10;Kernel-based methods for self-attention \cite{tay2020long} learn approximations of the exponential similarity function using $m$-dimensional feature maps to reformulate attention as a recurrent computation.&#10;They replace attention with ``unlimited capacity'' with fixed-capacity fast weights \cite{schmidhuber1992learning,peng2021abc}, where the memory-accuracy trade-off \cite{kerg2020untangling} is controlled by $m$.&#10;Several works explored different feature maps and recurrent formulations (i.e.,~update rules).&#10;\cite{katharopoulos2020transformers} propose feature maps to maintain positive outputs, while \cite{choromanski2020rethinking,peng2021random} carefully ensure their random feature maps are unbiased estimates of the softmax attention kernel.&#10;\cite{schlag2021linear,peng2021random} propose more sophisticated update rules to forget information in the recurrent state to improve performance.&#10;Recently, \cite{kasai2021finetuning} showed that pre-trained Transformers can be fine-tuned into a recurrent formulation using learned ReLU feature maps with minor degradations.&#10;While promising, it is unclear which update rules or feature maps are critical for successful fine-tuning.&#10;&#10;In this work, we investigate various update rule configurations to fine-tune pre-trained Transformers into RNNs for fast inference.&#10;We find that prior proposals contain unnecessary operations, leading us to propose a simple element-wise decay update rule with no feature map.&#10;We fine-tune GPT-2 \cite{radford2019language} into our recurrent formulation to demonstrate that our rule outperforms prior methods and recovers 99\% of self-attention's performance.&#10;We also show competitive performance on WikiText-103 \cite{merity2016pointer} compared to more complex attention alternatives.&#10;Our results support the idea \cite{merity2019single,zhai2021attention} that it is unnecessary for attention alternatives to maintain a close analogy to self-attention, and it is more important to focus on designing an expressive update rule.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Mathematics, Transformer Architecture Improvements, Attention Mechanism Alternatives, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2009.06489" label="2009.06489">
        <attvalues>
          <attvalue for="0" value="The Hardware Lottery" />
          <attvalue for="1" value="  Hardware, systems and algorithms research communities have historically had&#10;different incentive structures and fluctuating motivation to engage with each&#10;other explicitly. This historical treatment is odd given that hardware and&#10;software have frequently determined which research ideas succeed (and fail).&#10;This essay introduces the term hardware lottery to describe when a research&#10;idea wins because it is suited to the available software and hardware and not&#10;because the idea is superior to alternative research directions. Examples from&#10;early computer science history illustrate how hardware lotteries can delay&#10;research progress by casting successful ideas as failures. These lessons are&#10;particularly salient given the advent of domain specialized hardware which make&#10;it increasingly costly to stray off of the beaten path of research ideas. This&#10;essay posits that the gains from progress in computing are likely to become&#10;even more uneven, with certain research directions moving into the fast-lane&#10;while progress on others is further obstructed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.11174" label="2102.11174">
        <attvalues>
          <attvalue for="0" value="Linear Transformers Are Secretly Fast Weight Programmers" />
          <attvalue for="1" value="  We show the formal equivalence of linearised self-attention mechanisms and&#10;fast weight controllers from the early '90s, where a ``slow&quot; neural net learns&#10;by gradient descent to program the ``fast weights&quot; of another net through&#10;sequences of elementary programming instructions which are additive outer&#10;products of self-invented activation patterns (today called keys and values).&#10;Such Fast Weight Programmers (FWPs) learn to manipulate the contents of a&#10;finite memory and dynamically interact with it. We infer a memory capacity&#10;limitation of recent linearised softmax attention variants, and replace the&#10;purely additive outer products by a delta rule-like programming instruction,&#10;such that the FWP can more easily learn to correct the current mapping from&#10;keys to values. The FWP also learns to compute dynamically changing learning&#10;rates. We also propose a new kernel function to linearise attention which&#10;balances simplicity and effectiveness. We conduct experiments on synthetic&#10;retrieval problems as well as standard machine translation and language&#10;modelling tasks which demonstrate the benefits of our methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.13076" label="2103.13076">
        <attvalues>
          <attvalue for="0" value="Finetuning Pretrained Transformers into RNNs" />
          <attvalue for="1" value="  Transformers have outperformed recurrent neural networks (RNNs) in natural&#10;language generation. But this comes with a significant computational cost, as&#10;the attention mechanism's complexity scales quadratically with sequence length.&#10;Efficient transformer variants have received increasing interest in recent&#10;works. Among them, a linear-complexity recurrent variant has proven well suited&#10;for autoregressive generation. It approximates the softmax attention with&#10;randomized or heuristic feature maps, but can be difficult to train and may&#10;yield suboptimal accuracy. This work aims to convert a pretrained transformer&#10;into its efficient recurrent counterpart, improving efficiency while&#10;maintaining accuracy. Specifically, we propose a swap-then-finetune procedure:&#10;in an off-the-shelf pretrained transformer, we replace the softmax attention&#10;with its linear-complexity recurrent alternative and then finetune. With a&#10;learned feature map, our approach provides an improved tradeoff between&#10;efficiency and accuracy over the standard transformer and other recurrent&#10;variants. We also show that the finetuning process has lower training cost&#10;relative to training these recurrent variants from scratch. As many models for&#10;natural language tasks are increasingly dependent on large-scale pretrained&#10;transformers, this work presents a viable approach to improving inference&#10;efficiency without repeating the expensive pretraining process.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08593" label="2112.08593">
        <attvalues>
          <attvalue for="0" value="Goal-Directed Story Generation: Augmenting Generative Language Models&#10;  with Reinforcement Learning" />
          <attvalue for="1" value="  The advent of large pre-trained generative language models has provided a&#10;common framework for AI story generation via sampling the model to create&#10;sequences that continue the story. However, sampling alone is insufficient for&#10;story generation. In particular, it is hard to direct a language model to&#10;create stories to reach a specific goal event. We present two automated&#10;techniques grounded in deep reinforcement learning and reward shaping to&#10;control the plot of computer-generated stories. The first utilizes proximal&#10;policy optimization to fine-tune an existing transformer-based language model&#10;to generate text continuations but also be goal-seeking. The second extracts a&#10;knowledge graph from the unfolding story, which is used by a policy network&#10;with graph attention to select a candidate continuation generated by a language&#10;model. We report on automated metrics pertaining to how often stories achieve a&#10;given goal event as well as human participant rankings of coherence and overall&#10;story quality compared to baselines and ablations.&#10;" />
          <attvalue for="2" value="&#10;&#10;Humans produce and consume stories every day in many different forms: news, entertainment, conversations, and more. Storytelling is crucial to the communication process, as humans engage with well-told stories and comprehend more information from stories \cite{suzuki2018dialogues}. Because of its effectiveness in human-to-human interactions, automating storytelling has become an important research focus. &#10;&#10;Automated Story Generation is the challenge of designing an artificial intelligence system that can generate a story from a minimal number of inputs---often just a prompt and some storytelling knowledge and/or storytelling model. &#10;&#10;Early work on story generation used planning~\cite{meehan1976metanovel,lebowitz1987planning,cavazza2003interacting,porteous2009controlling,riedl2010narrative,ware2010modeling} &#10;or case-based reasoning~\cite{perez2001mexica,peinado2005creativity,turner2014creative}.&#10;In many cases, these systems are provided with a goal or outcome state.&#10;For example a goal might be ``character X in jail'' or ``characters X and Y are married''.&#10;However, these approaches require extensive domain knowledge engineering and rely on templated language.&#10;Recently, large pre-trained neural language models have been applied to story generation because they circumvent the need for knowledge engineering and tend to produce relatively fluent, varied, and naturalistic language~\cite{roemmele2016writing,khalifa2017deeptingle,clark2018neural,martin2018event}.&#10;To generate texts, language models are repeatedly invoked to generate the next token or sequence of tokens---called a {continuation}.&#10;Language models are, however, not goal-directed. &#10;That is, one cannot naively provide both an context prompt and a goal to be achieved after an arbitrary number of continuations. &#10;Further, language models struggle with maintaining story coherence---the logical progression of events---and may also become repetitive.&#10;&#10;Previous attempts to enhance the coherence of generated stories and control the trajectory of the story use conditioning on content-relevant features such as plot outlines~\cite{fan2018hierarchical,peng2018towards,rashkin2020plotmachines} or by hierarchical reasoning with abstract representations that help constrain story progression~\cite{martin2018event,yao2019plan,fan2019strategies,peng2021inferring}. &#10;These techniques do not address the need to make story generation systems goal-driven.&#10;&#10;To make story generation systems goal-driven,&#10;\cite{tambwekar2019controllable} &#10;trained a seq2seq language model on event abstractions (tuples containing verb, direct object, indirect object, and some other information) and then fine-tuned the model using reinforcement learning with a reward based on the average number of events it takes to get from one verb to a goal verb. &#10;They were able to show their models can achieve a given goal greater than 93\% of the time, and subjective ratings of story coherence improved significantly. However, the event abstractions are not human-readable without manual rewriting or a second event-to-sentence technique that often undoes any gains in coherence, as observed by an analysis of hierarchical story generation systems by \cite{ammanabrolu2020story}.&#10;&#10;This paper has two aims.&#10;First, we show that applying a reward shaping fine-tuning technique such as that by \cite{tambwekar2019controllable} above does not directly translate to more modern large pre-trained language models such as GPT-2~\cite{radford2019language}.&#10;Large pre-trained language models produce more natural language and can handle a larger range of inputs but, like seq2seq models, are not inherently goal-driven. &#10;Unfortunately, we observe that large language models are harder to control; our experiments with reward shaping based fine-tuning toward a given goal&#10;only results in a 50\% goal achievement rate (although fluency of story outputs is greatly improved). &#10;&#10;Our second aim is to introduce a new technique in which a second {policy model} can be trained to guide a non-fine-tuned GPT-2 to a given end-goal, achieving 90+\% goal success rate while retaining the language model's fluency.&#10;The key insight is that this second model operates on an abstracted state space represented as a knowledge graph---a set of $\langle subject, relation, object\rangle$ tuples.&#10;This story world state representation explicitly captures the entities in a story and their relations instead of relying on the hidden state of the language model to accurately represent the state of the story world.&#10;Given a knowledge graph representing the state of the story world, the policy model predicts the utility of the state, which is proportional to the number of sentences needed to achieve a goal.&#10;Thus, GPT-2 generates plausible continuations while the policy model learns to select continuations based on how they move the story forward.&#10;&#10;We report on a combination of automated and human participant evaluations.&#10;We focus on evaluating our system in the domain of science fiction plots~\cite{ammanabrolu2020story}, consistent with prior work~\cite{tambwekar2019controllable,ammanabrolu2020story}.&#10;Automated evaluation shows that our full two-model reinforcement learning technique achieves the desired-end goal $98.73\%$ of the time, which means that users can provide both a prompt and an ending.&#10;Our human participant studies show that the full model is perceived to be more coherent than baseline alternatives. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Story Generation Techniques, Computer Science, Linguistics, Cognitive Science, Language Model Control, Artificial Intelligence, Deep Reinforcement Learning, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1401.3841" label="1401.3841">
        <attvalues>
          <attvalue for="0" value="Narrative Planning: Balancing Plot and Character" />
          <attvalue for="1" value="  Narrative, and in particular storytelling, is an important part of the human&#10;experience. Consequently, computational systems that can reason about narrative&#10;can be more effective communicators, entertainers, educators, and trainers. One&#10;of the central challenges in computational narrative reasoning is narrative&#10;generation, the automated creation of meaningful event sequences. There are&#10;many factors -- logical and aesthetic -- that contribute to the success of a&#10;narrative artifact. Central to this success is its understandability. We argue&#10;that the following two attributes of narratives are universal: (a) the logical&#10;causal progression of plot, and (b) character believability. Character&#10;believability is the perception by the audience that the actions performed by&#10;characters do not negatively impact the audiences suspension of disbelief.&#10;Specifically, characters must be perceived by the audience to be intentional&#10;agents. In this article, we explore the use of refinement search as a technique&#10;for solving the narrative generation problem -- to find a sound and believable&#10;sequence of character actions that transforms an initial world state into a&#10;world state in which goal propositions hold. We describe a novel refinement&#10;search planning algorithm -- the Intent-based Partial Order Causal Link (IPOCL)&#10;planner -- that, in addition to creating causally sound plot progression,&#10;reasons about character intentionality by identifying possible character goals&#10;that explain their actions and creating plan structures that explain why those&#10;characters commit to their goals. We present the results of an empirical&#10;evaluation that demonstrates that narrative plans generated by the IPOCL&#10;algorithm support audience comprehension of character intentions better than&#10;plans generated by conventional partial-order planners.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.03557" label="1705.03557">
        <attvalues>
          <attvalue for="0" value="DeepTingle" />
          <attvalue for="1" value="  DeepTingle is a text prediction and classification system trained on the&#10;collected works of the renowned fantastic gay erotica author Chuck Tingle.&#10;Whereas the writing assistance tools you use everyday (in the form of&#10;predictive text, translation, grammar checking and so on) are trained on&#10;generic, purportedly &quot;neutral&quot; datasets, DeepTingle is trained on a very&#10;specific, internally consistent but externally arguably eccentric dataset. This&#10;allows us to foreground and confront the norms embedded in data-driven&#10;creativity and productivity assistance tools. As such tools effectively&#10;function as extensions of our cognition into technology, it is important to&#10;identify the norms they embed within themselves and, by extension, us.&#10;DeepTingle is realized as a web application based on LSTM networks and the&#10;GloVe word embedding, implemented in JavaScript with Keras-JS.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14967" label="2004.14967">
        <attvalues>
          <attvalue for="0" value="PlotMachines: Outline-Conditioned Generation with Dynamic Plot State&#10;  Tracking" />
          <attvalue for="1" value="  We propose the task of outline-conditioned story generation: given an outline&#10;as a set of phrases that describe key characters and events to appear in a&#10;story, the task is to generate a coherent narrative that is consistent with the&#10;provided outline. This task is challenging as the input only provides a rough&#10;sketch of the plot, and thus, models need to generate a story by interweaving&#10;the key points provided in the outline. This requires the model to keep track&#10;of the dynamic states of the latent plot, conditioning on the input outline&#10;while generating the full story. We present PlotMachines, a neural narrative&#10;model that learns to transform an outline into a coherent story by tracking the&#10;dynamic plot states. In addition, we enrich PlotMachines with high-level&#10;discourse structure so that the model can learn different writing styles&#10;corresponding to different parts of the narrative. Comprehensive experiments&#10;over three fiction and non-fiction datasets demonstrate that large-scale&#10;language models, such as GPT-2 and Grover, despite their impressive generation&#10;performance, are not sufficient in generating coherent narratives for the given&#10;outline, and dynamic plot state tracking is important for composing narratives&#10;with tighter, more consistent plots.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.01311" label="2105.01311">
        <attvalues>
          <attvalue for="0" value="Inferring the Reader: Guiding Automated Story Generation with&#10;  Commonsense Reasoning" />
          <attvalue for="1" value="  Transformer-based language model approaches to automated story generation&#10;currently provide state-of-the-art results. However, they still suffer from&#10;plot incoherence when generating narratives over time, and critically lack&#10;basic commonsense reasoning. Furthermore, existing methods generally focus only&#10;on single-character stories, or fail to track characters at all. To improve the&#10;coherence of generated narratives and to expand the scope of character-centric&#10;narrative generation, we introduce Commonsense-inference Augmented neural&#10;StoryTelling (CAST), a framework for introducing commonsense reasoning into the&#10;generation process with the option to model the interaction between multiple&#10;characters. We find that our CAST method produces significantly more coherent,&#10;on-topic, enjoyable and fluent stories than existing models in both the&#10;single-character and two-character settings in three storytelling domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.03480" label="1909.03480">
        <attvalues>
          <attvalue for="0" value="Story Realization: Expanding Plot Events into Sentences" />
          <attvalue for="1" value="  Neural network based approaches to automated story plot generation attempt to&#10;learn how to generate novel plots from a corpus of natural language plot&#10;summaries. Prior work has shown that a semantic abstraction of sentences called&#10;events improves neural plot generation and and allows one to decompose the&#10;problem into: (1) the generation of a sequence of events (event-to-event) and&#10;(2) the transformation of these events into natural language sentences&#10;(event-to-sentence). However, typical neural language generation approaches to&#10;event-to-sentence can ignore the event details and produce&#10;grammatically-correct but semantically-unrelated sentences. We present an&#10;ensemble-based model that generates natural language guided by events.We&#10;provide results---including a human subjects study---for a full end-to-end&#10;automated story generation system showing that our method generates more&#10;coherent and plausible stories than baseline approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.04928" label="2211.04928">
        <attvalues>
          <attvalue for="0" value="miCSE: Mutual Information Contrastive Learning for Low-shot Sentence&#10;  Embeddings" />
          <attvalue for="1" value="  This paper presents miCSE, a mutual information-based contrastive learning&#10;framework that significantly advances the state-of-the-art in few-shot sentence&#10;embedding. The proposed approach imposes alignment between the attention&#10;pattern of different views during contrastive learning. Learning sentence&#10;embeddings with miCSE entails enforcing the structural consistency across&#10;augmented views for every sentence, making contrastive self-supervised learning&#10;more sample efficient. As a result, the proposed approach shows strong&#10;performance in the few-shot learning domain. While it achieves superior results&#10;compared to state-of-the-art methods on multiple benchmarks in few-shot&#10;learning, it is comparable in the full-shot scenario. This study opens up&#10;avenues for efficient self-supervised learning methods that are more robust&#10;than current contrastive methods for sentence embedding.&#10;" />
          <attvalue for="2" value="&#10;&#10;Measuring sentence similarity has been challenging due to the ambiguity and variability of linguistic expressions. The community's strong interest in the topic can be attributed to its applicability in numerous language processing applications, such as sentiment analysis, information retrieval, and semantic search~\cite{PILEHVAR201595,iyyer-etal-2015-deep}. &#10;Language models perform well on these tasks but typically require fine-tuning on the downstream task and corpora ~\cite{reimers-gurevych-2019-sentence,devlin2018bert,pfeiffer2020AdapterHub,mosbach2021on}.&#10;In terms of sentence embeddings, contrastive learning schemes have already been adopted successfully~\cite{vanDenOord2018,liu-etal-2021-fast,gao2021simcse,Carlsson2021ICLR}. The idea of contrastive learning is that positive and negative pairs are generated given a batch of samples. Whereas the positive pairs are obtained via augmentation, negative pairs are often created by random collation of sentences. Following the construction of pairs,&#10;contrastive learning forces the network to learn feature representations by pushing apart different samples (negative pairs) or pulling together similar ones (positive pairs). &#10;While some methods seek to optimize for selecting ``hard'' negative for negative pair generation~\cite{zhou2022debiased}, others investigated better augmentation techniques for positive pair creation. In this regard, many methods have been proposed to create augmentations to boost representation learning. &#10;Standard approaches for the augmentation aim at input data level (a.k.a discrete augmentation), which comprises word level operations such as swapping, insertion, deletion, and substitution~\cite{XieICLR2017,Coulombe2018TextDA,wei-zou-2019-eda}. In contrast to that, continuous augmentation operates at the representation level, comprising approaches like interpolation or ``mixup'' on the embedding space~\cite{chen-etal-2020-mixtext, cheng-etal-2020-advaug, Guo2019AugmentingDW}. Most recently, augmentation was also proposed in a more continuous fashion operating in a parameter level via simple techniques such as drop-out~\cite{gao2021simcse,liu-etal-2021-fast,klein2022scd} or random span masking~\cite{liu-etal-2021-fast}. The intuition is that ``drop-out'' acts as minimal data augmentation, providing an expressive semantic variation. &#10;However, it will likely affect structural alignment across views. Since positive pairs are constructed from identical sentences, we hypothesize that the structural dependency over the views should be preserved by utilizing drop-out noise. &#10;Building on this idea, we maximize the structural dependence by enforcing distributional similarity over the attention values across the augmentation views. &#10;To this end, we employ maximization of the mutual information (MI) on the attention tensors of the positive pairs.&#10;However, since attention tensors can be very high-dimensional, computing MI can quickly become a significant burden if not intractable. &#10;This paper proposes a simple solution to alleviate the computational burden of MI computation, which can be deployed efficiently. Similar to ~\cite{NEURIPS2020_bcff3f63}, we adopt the Log-Normal distribution to model attention. Empirical evidence confirms this model as a good fit while facilitating the optimization objective to be defined in closed form. In this case, mutual information can be provably reformulated as a function of correlation, allowing native GPU implementation. As discussed above, the proposed approach builds upon the contrastive learning paradigm known to suffer from model collapse.&#10;This issue becomes even more problematic when enforcing MI on the attention level, as it tightens the positive pairs via regularizing the attention. Therefore the selection of negative pairs becomes more critical in our setup. To this end, we utilize momentum contrastive learning to generate harder negatives~\cite{MoCo_He2020}. A ``tighter'' binding on positive pairs and repulsion on ''harder'' negative pairs empowers the proposed contrastive objective, yielding more powerful representations.&#10;&#10;Combining ideas from momentum contrastive learning and attention regularization, we propose \ours, a conceptually simple yet empirically powerful method for sentence embedding, with the goal of integrating semantic and structural information of a sentence in an information-theoretic and Transformer-specific manner. We conjecture the relation between attention maps and a form of syntax to be the main driver behind the success of our approach. We speculate that our proposed method injects structural information into the model as an inductive bias, facilitating representation learning with fewer samples. The adopted structural inductive biases provide a ``syntactic'' prior as an implicit form of supervision during training~\cite{wilcox2020structural}, which promotes few-shot learning capabilities in neural language models. To validate this, we introduced a low-shot setup for training sentence embeddings. In this benchmark, we finetune the language model only with a small number of training samples. Note that this is a very challenging setup. The inherent difficulty can be attributed to the need to mitigate the domain shift in the low-shot self-supervised learning scheme. We emphasize the importance of this task, as in many real-world applications, only small datasets are often available. Such cases include NLP for low-resource languages or expert-produced texts (e.g., medical records by doctors), personalized LM for social media analysis (e.g., personalized hate speed recognition on Twitter), etc. Our proposed method significantly improves over the state-of-the-art in the low-shot sentence embedding benchmark. This is the first work that explores how to combine semantic and structural information through attention regularization and empirically demonstrates this benefit for low-shot sentence embeddings. &#10;&#10;Previous works: Recently, VaSCL~\cite{zhang-etal-2022-virtual}, ConSERT~\cite{yan-etal-2021-consert}, PCL~\cite{WuPCL22} and ~\cite{chuang2022diffcse} proposed contrastive representation learning with diverse augmentation strategies on positive pair. However, we proposed a principled approach for enforcing alignment in positive pairs at contrastive learning without discretely augmenting the data. Similar to us, ESimCSE~\cite{wu2021esimcse} and MoCoSE~\cite{cao-etal-2022-exploring} proposed to exploit a momentum contrastive learning model with negative sample queue for sentence embedding to boost uniformity of the representations. However, unlike us, they do not enforce any further tightening objective on the positive pairs nor consider few-shot learning. Very recently, authors in InforMin-CL~\cite{chen2022informin-cl} and InfoCSE~\cite{WuInfoCSE2022}proposed information minimization-based contrastive learning. Specifically, the authors propose to minimize the information entropy between positive embeddings generated by drop-out augmentation. Our model differs from this paper and the method in \cite{bachman2019learning,yang2021mutual,zhang-etal-2020-unsupervised,sordoni2021decomposed,wu2020mutual}, which focuses on using mutual information for self-supervised learning. A key difference compared to these methods is that they estimate MI directly on the representation space. In contrast, our method computes the MI on attention. Other related work include~\cite{zhang2022unsupervised,zhou-etal-2022-debiased,zhang2022contrastive,liu2022transencoder}.&#10;&#10;The contributions of the proposed work are:&#10;First, we propose to inject structural information into language models by adding an attention-level objective.&#10;Second, we introduce Attention Mutual Information (AMI), a sample-efficient self-supervised contrastive learning.&#10;Third, we introduce low-shot learning for sentence embedding. We show that our method performs comparably to the state-of-the-art in the full-shot scenario and significantly better in few-shot learning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Information Theory, Computer Science, Contrastive Learning, Linguistics, Mathematics, Sentence Embeddings" />
        </attvalues>
      </node>
      <node id="2006.04884" label="2006.04884">
        <attvalues>
          <attvalue for="0" value="On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and&#10;  Strong Baselines" />
          <attvalue for="1" value="  Fine-tuning pre-trained transformer-based language models such as BERT has&#10;become a common practice dominating leaderboards across various NLP benchmarks.&#10;Despite the strong empirical performance of fine-tuned models, fine-tuning is&#10;an unstable process: training the same model with multiple random seeds can&#10;result in a large variance of the task performance. Previous literature (Devlin&#10;et al., 2019; Lee et al., 2020; Dodge et al., 2020) identified two potential&#10;reasons for the observed instability: catastrophic forgetting and small size of&#10;the fine-tuning datasets. In this paper, we show that both hypotheses fail to&#10;explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT,&#10;fine-tuned on commonly used datasets from the GLUE benchmark, and show that the&#10;observed instability is caused by optimization difficulties that lead to&#10;vanishing gradients. Additionally, we show that the remaining variance of the&#10;downstream task performance can be attributed to differences in generalization&#10;where fine-tuned models with the same training loss exhibit noticeably&#10;different test performance. Based on our analysis, we present a simple but&#10;strong baseline that makes fine-tuning BERT-based models significantly more&#10;stable than the previously proposed approaches. Code to reproduce our results&#10;is available online: https://github.com/uds-lsv/bert-stable-fine-tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.04718" label="1812.04718">
        <attvalues>
          <attvalue for="0" value="Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs" />
          <attvalue for="1" value="  In practice, it is common to find oneself with far too little text data to&#10;train a deep neural network. This &quot;Big Data Wall&quot; represents a challenge for&#10;minority language communities on the Internet, organizations, laboratories and&#10;companies that compete the GAFAM (Google, Amazon, Facebook, Apple, Microsoft).&#10;While most of the research effort in text data augmentation aims on the&#10;long-term goal of finding end-to-end learning solutions, which is equivalent to&#10;&quot;using neural networks to feed neural networks&quot;, this engineering work focuses&#10;on the use of practical, robust, scalable and easy-to-implement data&#10;augmentation pre-processing techniques similar to those that are successful in&#10;computer vision. Several text augmentation techniques have been experimented.&#10;Some existing ones have been tested for comparison purposes such as noise&#10;injection or the use of regular expressions. Others are modified or improved&#10;techniques like lexical replacement. Finally more innovative ones, such as the&#10;generation of paraphrases using back-translation or by the transformation of&#10;syntactic trees, are based on robust, scalable, and easy-to-use NLP Cloud APIs.&#10;All the text augmentation techniques studied, with an amplification factor of&#10;only 5, increased the accuracy of the results in a range of 4.3% to 21.6%, with&#10;significant statistical fluctuations, on a standardized task of text polarity&#10;prediction. Some standard deep neural network architectures were tested: the&#10;multilayer perceptron (MLP), the long short-term memory recurrent network&#10;(LSTM) and the bidirectional LSTM (biLSTM). Classical XGBoost algorithm has&#10;been tested with up to 2.5% improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.07847" label="2203.07847">
        <attvalues>
          <attvalue for="0" value="SCD: Self-Contrastive Decorrelation for Sentence Embeddings" />
          <attvalue for="1" value="  In this paper, we propose Self-Contrastive Decorrelation (SCD), a&#10;self-supervised approach. Given an input sentence, it optimizes a joint&#10;self-contrastive and decorrelation objective. Learning a representation is&#10;facilitated by leveraging the contrast arising from the instantiation of&#10;standard dropout at different rates. The proposed method is conceptually simple&#10;yet empirically powerful. It achieves comparable results with state-of-the-art&#10;methods on multiple benchmarks without using contrastive pairs. This study&#10;opens up avenues for efficient self-supervised learning methods that are more&#10;robust than current contrastive methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.10604" label="2010.10604">
        <attvalues>
          <attvalue for="0" value="Bayesian Attention Modules" />
          <attvalue for="1" value="  Attention modules, as simple and effective tools, have not only enabled deep&#10;neural networks to achieve state-of-the-art results in many domains, but also&#10;enhanced their interpretability. Most current models use deterministic&#10;attention modules due to their simplicity and ease of optimization. Stochastic&#10;counterparts, on the other hand, are less popular despite their potential&#10;benefits. The main reason is that stochastic attention often introduces&#10;optimization issues or requires significant model changes. In this paper, we&#10;propose a scalable stochastic version of attention that is easy to implement&#10;and optimize. We construct simplex-constrained attention distributions by&#10;normalizing reparameterizable distributions, making the training process&#10;differentiable. We learn their parameters in a Bayesian framework where a&#10;data-dependent prior is introduced for regularization. We apply the proposed&#10;stochastic attention modules to various attention-based models, with&#10;applications to graph node classification, visual question answering, image&#10;captioning, machine translation, and language understanding. Our experiments&#10;show the proposed method brings consistent improvements over the corresponding&#10;baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08552" label="2110.08552">
        <attvalues>
          <attvalue for="0" value="Virtual Augmentation Supported Contrastive Learning of Sentence&#10;  Representations" />
          <attvalue for="1" value="  Despite profound successes, contrastive representation learning relies on&#10;carefully designed data augmentations using domain specific knowledge. This&#10;challenge is magnified in natural language processing where no general rules&#10;exist for data augmentation due to the discrete nature of natural language. We&#10;tackle this challenge by presenting a Virtual augmentation Supported&#10;Contrastive Learning of sentence representations (VaSCL). Originating from the&#10;interpretation that data augmentation essentially constructs the neighborhoods&#10;of each training instance, we in turn utilize the neighborhood to generate&#10;effective data augmentations. Leveraging the large training batch size of&#10;contrastive learning, we approximate the neighborhood of an instance via its&#10;K-nearest in-batch neighbors in the representation space. We then define an&#10;instance discrimination task regarding this neighborhood and generate the&#10;virtual augmentation in an adversarial training manner. We access the&#10;performance of VaSCL on a wide range of downstream tasks, and set a new&#10;state-of-the-art for unsupervised sentence representation learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.04380" label="2109.04380">
        <attvalues>
          <attvalue for="0" value="ESimCSE: Enhanced Sample Building Method for Contrastive Learning of&#10;  Unsupervised Sentence Embedding" />
          <attvalue for="1" value="  Contrastive learning has been attracting much attention for learning&#10;unsupervised sentence embeddings. The current state-of-the-art unsupervised&#10;method is the unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE takes dropout as&#10;a minimal data augmentation method, and passes the same input sentence to a&#10;pre-trained Transformer encoder (with dropout turned on) twice to obtain the&#10;two corresponding embeddings to build a positive pair. As the length&#10;information of a sentence will generally be encoded into the sentence&#10;embeddings due to the usage of position embedding in Transformer, each positive&#10;pair in unsup-SimCSE actually contains the same length information. And thus&#10;unsup-SimCSE trained with these positive pairs is probably biased, which would&#10;tend to consider that sentences of the same or similar length are more similar&#10;in semantics. Through statistical observations, we find that unsup-SimCSE does&#10;have such a problem. To alleviate it, we apply a simple repetition operation to&#10;modify the input sentence, and then pass the input sentence and its modified&#10;counterpart to the pre-trained Transformer encoder, respectively, to get the&#10;positive pair. Additionally, we draw inspiration from the community of computer&#10;vision and introduce a momentum contrast, enlarging the number of negative&#10;pairs without additional calculations. The proposed two modifications are&#10;applied on positive and negative pairs separately, and build a new sentence&#10;embedding method, termed Enhanced Unsup-SimCSE (ESimCSE). We evaluate the&#10;proposed ESimCSE on several benchmark datasets w.r.t the semantic text&#10;similarity (STS) task. Experimental results show that ESimCSE outperforms the&#10;state-of-the-art unsup-SimCSE by an average Spearman correlation of 2.02% on&#10;BERT-base.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.13093" label="2202.13093">
        <attvalues>
          <attvalue for="0" value="Exploring the Impact of Negative Samples of Contrastive Learning: A Case&#10;  Study of Sentence Embedding" />
          <attvalue for="1" value="  Contrastive learning is emerging as a powerful technique for extracting&#10;knowledge from unlabeled data. This technique requires a balanced mixture of&#10;two ingredients: positive (similar) and negative (dissimilar) samples. This is&#10;typically achieved by maintaining a queue of negative samples during training.&#10;Prior works in the area typically uses a fixed-length negative sample queue,&#10;but how the negative sample size affects the model performance remains unclear.&#10;The opaque impact of the number of negative samples on performance when&#10;employing contrastive learning aroused our in-depth exploration. This paper&#10;presents a momentum contrastive learning model with negative sample queue for&#10;sentence embedding, namely MoCoSE. We add the prediction layer to the online&#10;branch to make the model asymmetric and together with EMA update mechanism of&#10;the target branch to prevent the model from collapsing. We define a maximum&#10;traceable distance metric, through which we learn to what extent the text&#10;contrastive learning benefits from the historical information of negative&#10;samples. Our experiments find that the best results are obtained when the&#10;maximum traceable distance is at a certain range, demonstrating that there is&#10;an optimal range of historical information for a negative sample queue. We&#10;evaluate the proposed unsupervised MoCoSE on the semantic text similarity (STS)&#10;task and obtain an average Spearman's correlation of $77.27\%$. Source code is&#10;available at https://github.com/xbdxwyh/mocose.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.12565" label="2104.12565">
        <attvalues>
          <attvalue for="0" value="Mutual Contrastive Learning for Visual Representation Learning" />
          <attvalue for="1" value="  We present a collaborative learning method called Mutual Contrastive Learning&#10;(MCL) for general visual representation learning. The core idea of MCL is to&#10;perform mutual interaction and transfer of contrastive distributions among a&#10;cohort of networks. A crucial component of MCL is Interactive Contrastive&#10;Learning (ICL). Compared with vanilla contrastive learning, ICL can aggregate&#10;cross-network embedding information and maximize the lower bound to the mutual&#10;information between two networks. This enables each network to learn extra&#10;contrastive knowledge from others, leading to better feature representations&#10;for visual recognition tasks. We emphasize that the resulting MCL is&#10;conceptually simple yet empirically powerful. It is a generic framework that&#10;can be applied to both supervised and self-supervised representation learning.&#10;Experimental results on image classification and transfer learning to object&#10;detection show that MCL can lead to consistent performance gains, demonstrating&#10;that MCL can guide the network to generate better feature representations. Code&#10;is available at https://github.com/winycg/MCL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.12061" label="2009.12061">
        <attvalues>
          <attvalue for="0" value="An Unsupervised Sentence Embedding Method by Mutual Information&#10;  Maximization" />
          <attvalue for="1" value="  BERT is inefficient for sentence-pair tasks such as clustering or semantic&#10;search as it needs to evaluate combinatorially many sentence pairs which is&#10;very time-consuming. Sentence BERT (SBERT) attempted to solve this challenge by&#10;learning semantically meaningful representations of single sentences, such that&#10;similarity comparison can be easily accessed. However, SBERT is trained on&#10;corpus with high-quality labeled sentence pairs, which limits its application&#10;to tasks where labeled data is extremely scarce. In this paper, we propose a&#10;lightweight extension on top of BERT and a novel self-supervised learning&#10;objective based on mutual information maximization strategies to derive&#10;meaningful sentence embeddings in an unsupervised manner. Unlike SBERT, our&#10;method is not restricted by the availability of labeled data, such that it can&#10;be applied on different domain-specific corpus. Experimental results show that&#10;the proposed method significantly outperforms other unsupervised sentence&#10;embedding baselines on common semantic textual similarity (STS) tasks and&#10;downstream supervised tasks. It also outperforms SBERT in a setting where&#10;in-domain labeled data is not available, and achieves performance competitive&#10;with supervised methods on various tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2106.13401" label="2106.13401">
        <attvalues>
          <attvalue for="0" value="Decomposed Mutual Information Estimation for Contrastive Representation&#10;  Learning" />
          <attvalue for="1" value="  Recent contrastive representation learning methods rely on estimating mutual&#10;information (MI) between multiple views of an underlying context. E.g., we can&#10;derive multiple views of a given image by applying data augmentation, or we can&#10;split a sequence into views comprising the past and future of some step in the&#10;sequence. Contrastive lower bounds on MI are easy to optimize, but have a&#10;strong underestimation bias when estimating large amounts of MI. We propose&#10;decomposing the full MI estimation problem into a sum of smaller estimation&#10;problems by splitting one of the views into progressively more informed&#10;subviews and by applying the chain rule on MI between the decomposed views.&#10;This expression contains a sum of unconditional and conditional MI terms, each&#10;measuring modest chunks of the total MI, which facilitates approximation via&#10;contrastive bounds. To maximize the sum, we formulate a contrastive lower bound&#10;on the conditional MI which can be approximated efficiently. We refer to our&#10;general approach as Decomposed Estimation of Mutual Information (DEMI). We show&#10;that DEMI can capture a larger amount of MI than standard non-decomposed&#10;contrastive bounds in a synthetic setting, and learns better representations in&#10;a vision domain and for dialogue generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.13149" label="2005.13149">
        <attvalues>
          <attvalue for="0" value="On Mutual Information in Contrastive Learning for Visual Representations" />
          <attvalue for="1" value="  In recent years, several unsupervised, &quot;contrastive&quot; learning algorithms in&#10;vision have been shown to learn representations that perform remarkably well on&#10;transfer tasks. We show that this family of algorithms maximizes a lower bound&#10;on the mutual information between two or more &quot;views&quot; of an image where typical&#10;views come from a composition of image augmentations. Our bound generalizes the&#10;InfoNCE objective to support negative sampling from a restricted region of&#10;&quot;difficult&quot; contrasts. We find that the choice of negative samples and views&#10;are critical to the success of these algorithms. Reformulating previous&#10;learning objectives in terms of mutual information also simplifies and&#10;stabilizes them. In practice, our new objectives yield representations that&#10;outperform those learned with previous approaches for transfer to&#10;classification, bounding box detection, instance segmentation, and keypoint&#10;detection. % experiments show that choosing more difficult negative samples&#10;results in a stronger representation, outperforming those learned with IR, LA,&#10;and CMC in classification, bounding box detection, instance segmentation, and&#10;keypoint detection. The mutual information framework provides a unifying&#10;comparison of approaches to contrastive learning and uncovers the choices that&#10;impact representation learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.13059" label="2109.13059">
        <attvalues>
          <attvalue for="0" value="Trans-Encoder: Unsupervised sentence-pair modelling through self- and&#10;  mutual-distillations" />
          <attvalue for="1" value="  In NLP, a large volume of tasks involve pairwise comparison between two&#10;sequences (e.g. sentence similarity and paraphrase identification).&#10;Predominantly, two formulations are used for sentence-pair tasks: bi-encoders&#10;and cross-encoders. Bi-encoders produce fixed-dimensional sentence&#10;representations and are computationally efficient, however, they usually&#10;underperform cross-encoders. Cross-encoders can leverage their attention heads&#10;to exploit inter-sentence interactions for better performance but they require&#10;task fine-tuning and are computationally more expensive. In this paper, we&#10;present a completely unsupervised sentence representation model termed as&#10;Trans-Encoder that combines the two learning paradigms into an iterative joint&#10;framework to simultaneously learn enhanced bi- and cross-encoders.&#10;Specifically, on top of a pre-trained Language Model (PLM), we start with&#10;converting it to an unsupervised bi-encoder, and then alternate between the bi-&#10;and cross-encoder task formulations. In each alternation, one task formulation&#10;will produce pseudo-labels which are used as learning signals for the other&#10;task formulation. We then propose an extension to conduct such&#10;self-distillation approach on multiple PLMs in parallel and use the average of&#10;their pseudo-labels for mutual-distillation. Trans-Encoder creates, to the best&#10;of our knowledge, the first completely unsupervised cross-encoder and also a&#10;state-of-the-art unsupervised bi-encoder for sentence similarity. Both the&#10;bi-encoder and cross-encoder formulations of Trans-Encoder outperform recently&#10;proposed state-of-the-art unsupervised sentence encoders such as Mirror-BERT&#10;and SimCSE by up to 5% on the sentence similarity benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.11532" label="2310.11532">
        <attvalues>
          <attvalue for="0" value="Multi-stage Large Language Model Correction for Speech Recognition" />
          <attvalue for="1" value="  In this paper, we investigate the usage of large language models (LLMs) to&#10;improve the performance of competitive speech recognition systems. Different&#10;from previous LLM-based ASR error correction methods, we propose a novel&#10;multi-stage approach that utilizes uncertainty estimation of ASR outputs and&#10;reasoning capability of LLMs. Specifically, the proposed approach has two&#10;stages: the first stage is about ASR uncertainty estimation and exploits N-best&#10;list hypotheses to identify less reliable transcriptions; The second stage&#10;works on these identified transcriptions and performs LLM-based corrections.&#10;This correction task is formulated as a multi-step rule-based LLM reasoning&#10;process, which uses explicitly written rules in prompts to decompose the task&#10;into concrete reasoning steps. Our experimental results demonstrate the&#10;effectiveness of the proposed method by showing 10% ~ 20% relative improvement&#10;in WER over competitive ASR systems -- across multiple test domains and in&#10;zero-shot settings.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) such as ChatGPT \cite{DBLP:journals/corr/abs-2303-08774}, Llama \cite{touvron2023llama} have changed the landscape of AI research because of their groundbreaking capabilities. In this paper, we focus on the topic of ASR error correction and explore the usage of LLMs to push the state-of-the-art (SOTA) performance. ASR correction has a long history in the community and serves as a post-processing to improve the readability and quality of ASR transcriptions. With the recent rise of LLMs, the LLM-based ASR correction methods have been proposed \cite{ma2023n, ma2023can, yang2023generative, radhakrishnan2023whispering, chen2023hyporadise}. These methods can be roughly categorized into two groups depending on whether to re-train LLMs: i) fine-tuning LLMs \cite{ma2023n, radhakrishnan2023whispering} and ii) in-context learning of LLMs \cite{min2022rethinking}, which utilizes prompts without changing parameters of LLMs \cite{ma2023can, yang2023generative}. Herein, we focus on the in-context learning scheme of LLMs, as it is much simpler (requiring no training) and avoid the over-fitting issue when fine-tuning \cite{radhakrishnan2023whispering, chen2023hyporadise}. &#10;&#10;Along with this line of research, different LLMs have been explored, e.g. T5 \cite{ma2023n}, Llama \cite{radhakrishnan2023whispering} and ChatGPT \cite{ma2023can}. Despite interesting results obtained in these works, the previous LLM-based approaches struggle to improve competitive ASR systems and cannot go beyond SOTA. On LibriSpeech, when the word-error-rate (WER) is lower than 2$\%$, there is very limited improvement that previous methods can contribute \cite{ma2023can, chen2023hyporadise}. This is an inherent limitation caused by two issues: i) over-correction: LLMs make many unnecessary changes to the input transcription, as it steers the sentence more towards written language, instead of a verbatim transcript for the input speech. Because of the discrepancy between spoken language and written language, LLM-based correction can hinder the fidelity of ASR transcriptions \cite{chen2024its}; ii) multi-step reasoning challenge for LLMs. The task of ASR error correction itself is quite complex and requires a high level of reasoning for LLMs, e.g. where to pick the operating point of its correction, when to perform no correction if the sentence is correct and which words to replace if an error is found. Simple prompting techniques cannot fully capture this reasoning rationale or accommodate the diversity within the task, and thus cannot yield satisfactory results over SOTA. &#10;&#10;To address the first issue of over-correction, we propose a confidence-based ASR uncertainty estimation stage to first detect less reliable (uncertain) transcriptions and only perform LLM-based corrections on these detected sentences. Contrary to existing approaches \cite{gekhman2022red, qiu2021learning, li2021confidence}, we extract the confidence and uncertainty information from N-best list hypotheses and identify them by thresholding the obtained confidence scores. For transcriptions that are believed to be less reliable, we will prompt their N-best list to LLMs for a later correction. &#10;&#10;For the second issue of multi-step reasoning, it is a well-known challenge for LLMs \cite{dziri2023faith, srivastava2023beyond}. The widely-used prompting techniques, chain-of-thought (CoT) prompting \cite{wei2022chain} and the prefix `let's think step by step' \cite{kojima2022large}, are not best suited for our ASR correction task. Because the complexity of the task cannot be fully accommodated in several hand-crafted exemplars that CoT relies on \cite{suzgun2022challenging, zhou2022least} and it is also hard for LLMs to figure out the reasoning rationale entirely by themselves \cite{kojima2022large, khot2022decomposed}. To accommodate this complexity, \cite{yang2023generative} proposed a task-activating prompting (TAP), which needs four rounds of Q and A to properly set up the correction task. Herein, we want to use just one well-guided prompt and enable the zero-shot setting for LLMs. To this end, we propose a multi-step rule-based prompt that leverages explicitly written rules to facilitate the reasoning process of LLMs on this task. These rules guide LLMs to break down such a complex task into intermediate reasoning steps and also set up constrictions during the LLM generation process.&#10;&#10;Experiments showed with the proposed correction pipeline, we can achieve 10\% $\sim$ 20\% WER relative improvement consistently for competitive ASR systems. Our best system for the LibriSpeech benchmark can reach 1.3\% WER on LibriSpeech test-clean set, which sets a new state-of-the-art. Altogether, we summarize the main contribution and the novelty of this paper as following: &#10;&#10;\begin{itemize}&#10;\item ASR uncertainty estimation stage. To deal with the over-correction issue of LLMs, we propose an uncertainty estimation to identify less reliable transcriptions. Given that, only a small portion of uncertain utterances will be sent to LLMs. It helps to keep the fidelity of majority spoken transcriptions and alleviates the total computational cost of using LLMs.&#10;&#10;\item Multi-step rule-based reasoning. To facilitate the complex LLM reasoning process for ASR error correction, we propose a multi-step rule-based prompt that explicitly decomposes the task into concrete reasoning steps and constrictions to follow. With these explicitly defined rules, we sidestep the need to provide hand-crafted exemplars or craft fine-tuning datasets. Besides, it shows that LLMs can be zero-shot reasoners for the ASR correction task and demonstrates the SOTA performance in zero-shot settings.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Error Correction Techniques, Language Model Enhancement, Artificial Intelligence, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2203.07172" label="2203.07172">
        <attvalues>
          <attvalue for="0" value="RED-ACE: Robust Error Detection for ASR using Confidence Embeddings" />
          <attvalue for="1" value="  ASR Error Detection (AED) models aim to post-process the output of Automatic&#10;Speech Recognition (ASR) systems, in order to detect transcription errors.&#10;Modern approaches usually use text-based input, comprised solely of the ASR&#10;transcription hypothesis, disregarding additional signals from the ASR model.&#10;Instead, we propose to utilize the ASR system's word-level confidence scores&#10;for improving AED performance. Specifically, we add an ASR Confidence Embedding&#10;(ACE) layer to the AED model's encoder, allowing us to jointly encode the&#10;confidence scores and the transcribed text into a contextualized&#10;representation. Our experiments show the benefits of ASR confidence scores for&#10;AED, their complementary effect over the textual signal, as well as the&#10;effectiveness and robustness of ACE for combining these signals. To foster&#10;further research, we publish a novel AED dataset consisting of ASR outputs on&#10;the LibriSpeech corpus with annotated transcription errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.08159" label="2106.08159">
        <attvalues>
          <attvalue for="0" value="Maximum Spanning Trees Are Invariant to Temperature Scaling in&#10;  Graph-based Dependency Parsing" />
          <attvalue for="1" value="  Modern graph-based syntactic dependency parsers operate by predicting, for&#10;each token within a sentence, a probability distribution over its possible&#10;syntactic heads (i.e., all other tokens) and then extracting a maximum spanning&#10;tree from the resulting log-probabilities. Nowadays, virtually all such parsers&#10;utilize deep neural networks and may thus be susceptible to miscalibration (in&#10;particular, overconfident predictions). In this paper, we prove that&#10;temperature scaling, a popular technique for post-hoc calibration of neural&#10;networks, cannot change the output of the aforementioned procedure. We conclude&#10;that other techniques are needed to tackle miscalibration in graph-based&#10;dependency parsers in a way that improves parsing accuracy.&#10;" />
          <attvalue for="2" value="&#10;Syntactic dependency parsing refers to the task of predicting, for a given sentence, the grammatical relations between its tokens.&#10;Most commonly, the output is a dependency tree, i.e., a graph structure in which each token constitutes a node and is assigned exactly one parent (its syntactic head). The parent may be either one of the other words in the sentence or an additional, implicit Root node. &#10;For the dependency tree to be valid, each token must be reachable from Root.&#10;Figure~\ref{fig:dep_example} shows such a structure for the sentence \enquote{Mary likes fluffy cats.}&#10;&#10;Graph-based dependency parsing is a technique for predicting dependency trees.&#10;In its usual formulation, the approach entails using some machine learning classifier to predict, for each token within the input sentence, a probability distribution over its possible syntactic heads (i.e., all other tokens in the sentence, as well as Root), as shown in the rows of Figure~\ref{fig:head_probs}.&#10;In a second step, the logarithms of the resulting probabilities are then interpreted as edge weights between pairs of nodes corresponding to tokens and a maximum spanning tree is extracted, using, e.g., the Chu-Liu/Edmonds algorithm \cite{chuliu1965shortest,edmonds1967optimum}.&#10;This maximum spanning tree is then returned as the dependency tree for the input sentence.&#10;&#10;The calibration of a machine learning classifier refers to its ability to generate output probabilities that are representative of the actual correctness likelihoods.&#10;For example, in a well-calibrated dependency parser, roughly 80\,\% of edges that are predicted with a probability of $0.8$ should actually be present in the gold data. &#10;However, as \newcite{guo2017calibration} show, modern neural networks are often miscalibrated and prone to overconfident predictions.&#10;In the context of graph-based dependency parsing, this would mean that the probability distributions over syntactic heads are too concentrated on the parent considered the most likely, not properly reflecting the uncertainty in the syntactic attachment of the tokens.&#10;Since the final dependency trees are extracted using an MST algorithm that operates on the (log-)probabilities returned by the classifier (and not merely the highest-scoring parent for each token), overconfidence in the syntactic attachments of the tokens might not only lead to overconfidence in the parse itself, but also to incorrect parses. Thus, good estimates of token attachment uncertainty may not only improve the uncertainty estimates of the final trees, but also their accuracy.&#10;&#10;One popular technique for post-hoc calibration of neural networks is temperature scaling \cite{guo2017calibration}.&#10;This approach works by dividing the unnormalized output values (\enquote{logits}) of the neural network by a constant $T \in \mathbb{R}_{&gt;0}$ (the \enquote{softmax temperature}) before applying the softmax function, i.e., given a vector $\mathbf{x}$ of unnormalized scores, the vector of probabilities is computed as $softmax(\mathbf{x}/T)$ rather than simply $softmax(\mathbf{x})$. $T$ is optimized w.r.t. negative log-likelihood on a validation set and usually $T &gt; 1$, resulting in a \enquote{smoother} probability distribution (i.e., one with higher entropy, or higher uncertainty) that corrects for the model's overconfidence.&#10;&#10;In this paper, we show that applying temperature scaling to the logits of a graph-based dependency parser (i.e., the unnormalized values of the head probability distributions; the input values for the rows in Figure~\ref{fig:head_probs}) does not change the resulting dependency trees. Although it is already known that temperature scaling does not change predictions for individual classifications, this is nonetheless a somewhat surprising result since the maximum spanning trees in dependency parsing depend on all (log-)probabilities provided by the underlying classifier.&#10;We conclude that calibration methods that simply re-scale softmax probabilities are not suited to improve the accuracy of graph-based dependency parsers.&#10;&#10;The remainder of this paper is structured as follows.&#10;Section~\ref{sec:definitions} sets up definitions for temperature scaling in graph-based dependency parsing;&#10;Section~\ref{sec:main_proof} proves our main claim;&#10;Section~\ref{sec:conclusion} concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Syntactic Dependency Parsing, Computer Science, Linguistics, Neural Network Calibration, Graph-Based Parsing Accuracy, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2204.00498" label="2204.00498">
        <attvalues>
          <attvalue for="0" value="Evaluating the Text-to-SQL Capabilities of Large Language Models" />
          <attvalue for="1" value="  We perform an empirical evaluation of Text-to-SQL capabilities of the Codex&#10;language model. We find that, without any finetuning, Codex is a strong&#10;baseline on the Spider benchmark; we also analyze the failure modes of Codex in&#10;this setting. Furthermore, we demonstrate on the GeoQuery and Scholar&#10;benchmarks that a small number of in-domain examples provided in the prompt&#10;enables Codex to perform better than state-of-the-art models finetuned on such&#10;few-shot examples.&#10;" />
          <attvalue for="2" value="&#10;Translating natural language questions to SQL queries (Text-to-SQL) is an important business problem which has seen significant research interest.&#10;A common approach to this task involves training a model to produce a SQL query when given a question, a database schema, and possibly database content as inputs.&#10;A clear trend in this area is to finetune models pretrained on natural language; notably, performance significantly improves as larger pretrained models are used \cite{shaw-etal-2021-compositional,scholak2021picard}.&#10;&#10;Recent results from the broader field demonstrate that simply scaling training data and model size for generative language models brings advanced capabilities, such as few-shot learning without finetuning \cite[GPT-3, ][]{brown2020language} and code generation \cite[Codex, ][]{chen2021evaluating}.&#10;In this work we study if such models are already competitive Text-to-SQL solutions without any further finetuning on task-specific training data, evaluating Codex and GPT-3 models of different sizes with varied prompts on Text-to-SQL benchmarks.&#10;&#10;We find that Codex achieves a competitive performance of up to 67\% execution accuracy on the Spider development set.&#10;We analyze the predicted queries that automatic evaluation judged as wrong and find that many of them would be judged correct by humans, whereas others could likely be fixed within the no-finetuning paradigm. &#10;Lastly, using GeoQuery and Scholar benchmarks we show that adapting Codex to a specific domain by prompting it with few examples can be more effective than fine-tuning a smaller language model on the same examples.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Few-Shot Learning, Computer Science, Language Model Analysis, Linguistics, Data Science, Artificial Intelligence, Text-to-SQL Evaluation" />
        </attvalues>
      </node>
      <node id="2308.12219" label="2308.12219">
        <attvalues>
          <attvalue for="0" value="Diffusion Language Models Can Perform Many Tasks with Scaling and&#10;  Instruction-Finetuning" />
          <attvalue for="1" value="  The recent surge of generative AI has been fueled by the generative power of&#10;diffusion probabilistic models and the scalable capabilities of large language&#10;models. Despite their potential, it remains elusive whether diffusion language&#10;models can solve general language tasks comparable to their autoregressive&#10;counterparts. This paper demonstrates that scaling diffusion models w.r.t.&#10;data, sizes, and tasks can effectively make them strong language learners. We&#10;build competent diffusion language models at scale by first acquiring knowledge&#10;from massive data via masked language modeling pretraining thanks to their&#10;intrinsic connections. We then reprogram pretrained masked language models into&#10;diffusion language models via diffusive adaptation, wherein task-specific&#10;finetuning and instruction finetuning are explored to unlock their versatility&#10;in solving general language tasks. Experiments show that scaling diffusion&#10;language models consistently improves performance across downstream language&#10;tasks. We further discover that instruction finetuning can elicit zero-shot and&#10;few-shot in-context learning abilities that help tackle many unseen tasks by&#10;following natural language instructions, and show promise in advanced and&#10;challenging abilities such as reasoning.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advances in generative modeling have led to remarkable progress in the field of generative AI. &#10;In domains of continuous signals, diffusion probabilistic models have shown great success in rendering photorealistic images~\cite{rombach2021highresolution,ramesh2022dalle2}, immersive videos~\cite{bar2024lumiere} and synthesizing high-quality audio~\cite{kong2020diffwave} through iterative denoising, outperforming GANs and autoregressive (AR) models, even contributing to the surge of AI art. &#10;The story is different in the domains of discrete signals comprising symbolic sequences such as natural languages, where AR large language models~\cite[large language models or LLMs,][]{brown2020lgpt3,openai2023gpt4} have dominated the scene, delivering impressive generalist language abilities in language understanding and generating human-like texts, and can even follow natural language instructions to perform unseen tasks.&#10;&#10;While many recent endeavors try unifying the generation paradigms by enabling large language models to draw~\cite{ge2023seed} or speak~\cite{zhang2023speechgpt}, few explore generating discrete sequences such as languages with diffusion models. &#10;We suggest that the revolutionized generative abilities of diffusion models give the promise of a strong complement to autoregressive LMs for several favorable reasons, including (1) global receptive field \vs one-sided context, and (2) non-autoregressive drafting-then-revising manner \vs restrictive unidirectional generation/autoregression.&#10;Hence, an intriguing question arises: can diffusion models speak languages well?&#10;&#10;The key to the great success of modern LLMs lies in their scalability which fosters powerful generalist capabilities.&#10;The question of the capability of {Diffusion-LLM}s is thus in turn to ask about their scalability, which can be further boiled down into the following specific research questions regarding the three key ingredients of the success of large-scale LMs, i.e., data, model sizes, and tasks:&#10;\begin{itemize}[itemsep=2pt,topsep=0pt,parsep=1pt,leftmargin=20pt]&#10;\item [(i)] On scaling data. &#10;Acquiring general knowledge via self-supervised pre-training from massive unlabeled data plays a crucial role in the success of the modern NLP paradigms~\cite{radford2018improving,devlin2018bert}, hence it is also of importance to enable Diffusion LMs to learn from massive data.&#10;Can {Diffusion-LLM}s leverage knowledge from large-scale data?&#10;\item [(ii)] On scaling model sizes. It has been widely observed that the larger the model size, the more competent the LMs become. Can enlarging &#10;{Diffusion-LLM}s effectively improve downstream tasks?&#10;\item [(iii)] On scaling tasks. &#10;What makes LLMs most attractive is they can tackle new tasks that they were never exposed to during training by following natural language and even multimodal instructions with little to no demonstrations.&#10;Can {Diffusion-LLM}s exhibit general zero-shot and few-shot in-context learning capabilities to generalize to unseen tasks?&#10;\end{itemize}&#10;&#10;In this paper, we delve into the potential of {Diffusion-LLM}s through the three research questions.&#10;We highlight our contributions and findings as follows:&#10;&#10;(1) We first demonstrate the intrinsic connection between masked LMs and discrete diffusion models, which permits us to treat pre-trained masked LMs of various scales as pre-trained {Diffusion-LLM}s, without the need for expensive learning from scratch. &#10;We then reprogram pre-trained masked LMs into {Diffusion-LLM}s via diffusive adaptation, where task-specific finetuning and instruction finetuning~\cite{wei2021flanv1} are explored for solving certain downstream tasks or general language problems, showing {Diffusion-LLM}s benefit from pre-training on large scale data. &#10;(2) We reveal that large-scale {Diffusion-LLM}s can serve as strong sequence generative models to tackle multitasks, exhibiting competitive performance compared with autoregressive LMs.&#10;And the performance consistently improves as the model sizes scale up. (3) We further elicit zero-shot and few-shot abilities for {Diffusion-LLM}s to tackle multiple unseen tasks, spanning from language and visual ones, through both language and vision instruction finetuning.&#10;Notably, {Diffusion-LLM}s demonstrate promising structured reasoning behaviors thanks to their flexible non-autoregressive generation order. Nevertheless, their capacity to tackle complex reasoning tasks remains an ongoing challenge awaiting resolution.&#10;&#10;To sum up, we hope that our explorations provide valuable insights into the scalability of {Diffusion-LLM}s and their potential as a viable complement in tackling generative language tasks across the board. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Task Solving, Linguistics, Diffusion Model Scaling, Artificial Intelligence, Mathematics, Generative AI Models" />
        </attvalues>
      </node>
      <node id="2401.12945" label="2401.12945">
        <attvalues>
          <attvalue for="0" value="Lumiere: A Space-Time Diffusion Model for Video Generation" />
          <attvalue for="1" value="  We introduce Lumiere -- a text-to-video diffusion model designed for&#10;synthesizing videos that portray realistic, diverse and coherent motion -- a&#10;pivotal challenge in video synthesis. To this end, we introduce a Space-Time&#10;U-Net architecture that generates the entire temporal duration of the video at&#10;once, through a single pass in the model. This is in contrast to existing video&#10;models which synthesize distant keyframes followed by temporal super-resolution&#10;-- an approach that inherently makes global temporal consistency difficult to&#10;achieve. By deploying both spatial and (importantly) temporal down- and&#10;up-sampling and leveraging a pre-trained text-to-image diffusion model, our&#10;model learns to directly generate a full-frame-rate, low-resolution video by&#10;processing it in multiple space-time scales. We demonstrate state-of-the-art&#10;text-to-video generation results, and show that our design easily facilitates a&#10;wide range of content creation tasks and video editing applications, including&#10;image-to-video, video inpainting, and stylized generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.09761" label="2009.09761">
        <attvalues>
          <attvalue for="0" value="DiffWave: A Versatile Diffusion Model for Audio Synthesis" />
          <attvalue for="1" value="  In this work, we propose DiffWave, a versatile diffusion probabilistic model&#10;for conditional and unconditional waveform generation. The model is&#10;non-autoregressive, and converts the white noise signal into structured&#10;waveform through a Markov chain with a constant number of steps at synthesis.&#10;It is efficiently trained by optimizing a variant of variational bound on the&#10;data likelihood. DiffWave produces high-fidelity audios in different waveform&#10;generation tasks, including neural vocoding conditioned on mel spectrogram,&#10;class-conditional generation, and unconditional generation. We demonstrate that&#10;DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44&#10;versus 4.43), while synthesizing orders of magnitude faster. In particular, it&#10;significantly outperforms autoregressive and GAN-based waveform models in the&#10;challenging unconditional generation task in terms of audio quality and sample&#10;diversity from various automatic and human evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.01218" label="2310.01218">
        <attvalues>
          <attvalue for="0" value="Making LLaMA SEE and Draw with SEED Tokenizer" />
          <attvalue for="1" value="  The great success of Large Language Models (LLMs) has expanded the potential&#10;of multimodality, contributing to the gradual evolution of General Artificial&#10;Intelligence (AGI). A true AGI agent should not only possess the capability to&#10;perform predefined multi-tasks but also exhibit emergent abilities in an&#10;open-world context. However, despite the considerable advancements made by&#10;recent multimodal LLMs, they still fall short in effectively unifying&#10;comprehension and generation tasks, let alone open-world emergent abilities. We&#10;contend that the key to overcoming the present impasse lies in enabling text&#10;and images to be represented and processed interchangeably within a unified&#10;autoregressive Transformer. To this end, we introduce SEED, an elaborate image&#10;tokenizer that empowers LLMs with the ability to SEE and Draw at the same time.&#10;We identify two crucial design principles: (1) Image tokens should be&#10;independent of 2D physical patch positions and instead be produced with a 1D&#10;causal dependency, exhibiting intrinsic interdependence that aligns with the&#10;left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens&#10;should capture high-level semantics consistent with the degree of semantic&#10;abstraction in words, and be optimized for both discriminativeness and&#10;reconstruction during the tokenizer training phase. With SEED tokens, LLM is&#10;able to perform scalable multimodal autoregression under its original training&#10;recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by&#10;large-scale pretraining and instruction tuning on the interleaved textual and&#10;visual data, demonstrating impressive performance on a broad range of&#10;multimodal comprehension and generation tasks. More importantly, SEED-LLaMA has&#10;exhibited compositional emergent abilities such as multi-turn in-context&#10;multimodal generation, acting like your AI assistant.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.03857" label="2110.03857">
        <attvalues>
          <attvalue for="0" value="A study on the efficacy of model pre-training in developing neural&#10;  text-to-speech system" />
          <attvalue for="1" value="  In the development of neural text-to-speech systems, model pre-training with&#10;a large amount of non-target speakers' data is a common approach. However, in&#10;terms of ultimately achieved system performance for target speaker(s), the&#10;actual benefits of model pre-training are uncertain and unstable, depending&#10;very much on the quantity and text content of training data. This study aims to&#10;understand better why and how model pre-training can positively contribute to&#10;TTS system performance. It is postulated that the pre-training process plays a&#10;critical role in learning text-related variation in speech, while further&#10;training with the target speaker's data aims to capture the speaker-related&#10;variation. Different test sets are created with varying degrees of similarity&#10;to target speaker data in terms of text content. Experiments show that&#10;leveraging a speaker-independent TTS trained on speech data with diverse text&#10;content can improve the target speaker TTS on domain-mismatched text. We also&#10;attempt to reduce the amount of pre-training data for a new text domain and&#10;improve the data and computational efficiency. It is found that the TTS system&#10;could achieve comparable performance when the pre-training data is reduced to&#10;1/8 of its original size.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, neural text-to-speech technology has demonstrated significant successes in generating high-quality speech with good naturalness and expressiveness for a target speaker \cite{tan2021survey,wang2017tacotron,shen2018natural,elias2021parallel,ren2020fastspeech}. In research and development of neural TTS, the text content of training and test data are often highly similar and in the same text domain. For many real-world applications, TTS systems need to deal with text input with arbitrary content across a wide range of domains. Their performance may deteriorate substantially on domain-mismatched text \cite{he2019robust} due to the limited content and domain coverage of training data. It is generally costly or impractical to increase the quantity and diversity of training data for a specific target speaker, whilst speech data from other ``non-target'' speakers may be easily accessible and available. Leveraging large amount of non-target speakers' data from different sources has become a common and appealing approach to developing high-performance TTS systems when training data from the target speaker(s) are limited \cite{chung2019semi,arik2018neural,chen2018sample,cooper2020zero, tan2021cuhk}. However, it was noticed that the actual benefits of using training speech from other speakers could be uncertain and unstable \cite{chung2019semi}. Understandably the benefits depend very much on the content and quality of non-target speakers' data. There are three aspects of consideration: (1) coverage and domain of text content; (2) speaker similarity with respect to the target speaker; and (3) acoustic condition. The present study mainly focuses on the aspect of text content. &#10;&#10;Our preliminary explorations showed that TTS systems trained solely on target speaker's data did not perform well in predicting appropriate prosody for domain-mismatched input text. Prosody is utmost important in determining the naturalness and expressiveness of speech. We postulate that speech prosody can be seen as the combination of two components of variation in speech, namely text-based variation and speaker-based variation. The text-based component, termed as text prosody henceforth, refers to the general prosodic characteristics that are basic and essential to expressing the intended text content, e.g., lexical tone, lexical stress, sentence intonation \cite{taylor2009text}. If text prosody is not realized or controlled properly, the synthesized speech would sound unnatural and inappropriate, even if the pronunciation is largely correct. In \cite{aoyama2007prosody}, native listeners were found to be capable of detecting abnormality in the non-native speech that has International Phonetic Alphabet (IPA) transcription identical to that produced by native speakers. The speaker-based component of prosodic variation\cite{jia2018Transfer}, termed as speaker prosody, is concerned primarily with an individual's speaking style. In particular, we differentiate it from timbre, which refers mainly to voice (phonation) characteristics.&#10;&#10;We conjecture that text prosody can be captured by pre-training the TTS model on a large amount of speech data with diverse text content and involving multiple non-target speakers. A speech generation system fine-tuned from such pre-trained model with target speaker's data is expected to have better performance on domain-mismatched text input than a corresponding system without using pre-trained model. In a typical multi-speaker TTS model design\cite{yamagishi2009robust,yang2016training}, text content and speaker identity are processed by separate modules. It adopts a speaker-independent TTS model with text embeddings and speaker embeddings. This speaker-independent TTS model deals with only text information regardless of speaker variation. After being fine-tuned on target speaker data, the speaker-independent TTS model would supposedly retain the text prosody learned during pre-training. In this way, effective pre-training with diverse text content can contribute to the performance of target speaker TTS system in the aspect of handling domain-mismatched text.&#10;&#10;In some cases, it may not be convenient to collect extensive speech data with diverse content for model pre-training. Traditionally large amounts of text-audio parallel data could be acquired by: (1) studio recording with the target speaker(s) reading text scripts for many hours; or (2) downloading long speech recordings from the Internet, dividing them into sentence-level utterances and aligning with given text transcriptions \cite{chu2002domain, panayotov2015librispeech, zen2019libritts}. Both approaches are tedious, labour-intensive and costly. Using an excessive amount of pre-training data might also cause other concerns, namely long training time and high computational resources. For applications of speech generation in specific text domains, we investigate different approaches to reducing the required amount of pre-training data while maintaining desired TTS performance.&#10;&#10;The contributions of this paper are as follows. First, we show that using diverse speech data to pre-train a speaker-independent TTS model can improve the performance of the target speaker TTS on domain-mismatched text. Second, test sets with different degrees of similarities to the text domain of target speaker data are designed. The designed test sets are used to study how the pre-trained speaker-independent TTS model improves target speaker TTS performance; Third, We propose a method to improve the data and computational efficiency by reducing the pre-training data for a specific new text domain.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Text-to-Speech, Computer Science, Speech Variation Analysis, Linguistics, Engineering, Model Pre-training Benefits, Signal Processing" />
        </attvalues>
      </node>
      <node id="1906.00672" label="1906.00672">
        <attvalues>
          <attvalue for="0" value="Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic&#10;  Attention for Neural TTS" />
          <attvalue for="1" value="  Neural TTS has demonstrated strong capabilities to generate human-like speech&#10;with high quality and naturalness, while its generalization to out-of-domain&#10;texts is still a challenging task, with regard to the design of attention-based&#10;sequence-to-sequence acoustic modeling. Various errors occur in those inputs&#10;with unseen context, including attention collapse, skipping, repeating, etc.,&#10;which limits the broader applications. In this paper, we propose a novel&#10;stepwise monotonic attention method in sequence-to-sequence acoustic modeling&#10;to improve the robustness on out-of-domain inputs. The method utilizes the&#10;strict monotonic property in TTS with constraints on monotonic hard attention&#10;that the alignments between inputs and outputs sequence must be not only&#10;monotonic but allowing no skipping on inputs. Soft attention could be used to&#10;evade mismatch between training and inference. The experimental results show&#10;that the proposed method could achieve significant improvements in robustness&#10;on out-of-domain scenarios for phoneme-based models, without any regression on&#10;the in-domain naturalness test.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.01107" label="2005.01107">
        <attvalues>
          <attvalue for="0" value="Simplifying Paragraph-level Question Generation via Transformer Language&#10;  Models" />
          <attvalue for="1" value="  Question generation (QG) is a natural language generation task where a model&#10;is trained to ask questions corresponding to some input text. Most recent&#10;approaches frame QG as a sequence-to-sequence problem and rely on additional&#10;features and mechanisms to increase performance; however, these often increase&#10;model complexity, and can rely on auxiliary data unavailable in practical use.&#10;A single Transformer-based unidirectional language model leveraging transfer&#10;learning can be used to produce high quality questions while disposing of&#10;additional task-specific complexity. Our QG model, finetuned from GPT-2 Small,&#10;outperforms several paragraph-level QG baselines on the SQuAD dataset by 0.95&#10;METEOR points. Human evaluators rated questions as easy to answer, relevant to&#10;their context paragraph, and corresponding well to natural human speech. Also&#10;introduced is a new set of baseline scores on the RACE dataset, which has not&#10;previously been used for QG tasks. Further experimentation with varying model&#10;capacities and datasets with non-identification type questions is recommended&#10;in order to further verify the robustness of pretrained Transformer-based LMs&#10;as question generators.&#10;" />
          <attvalue for="2" value="&#10;&#10;&#10;Question Generation (QG) \cite{rus2008question}, while not as prominent as its sibling task Question Answering (QA), still remains a relevant task in NLP. The ability to ask meaningful questions provides evidence towards comprehension within an Artificial Intelligence (AI) model \cite{nappi2017importance}. This makes the task of QG important in the bigger picture of AI.&#10;&#10;Many studies have produced robust models with good performance for QG in recent years. The most widely-used techniques are Deep Learning-based approaches involving Sequence-to-Sequence (Seq2Seq) \cite{sutskever2014sequence} models. These approaches use two LSTM-based \cite{Hochreiter:1997:LSM:1246443.1246450} neural networks, one to encode the source context paragraph, and the other to decode the embedded information and output a generated question \cite{duan2017question}.&#10;&#10;Further works that improve on the standard Seq2Seq-based QG models use either extra mechanisms, extra features, or both. These include the usage of extra linguistic features \cite{zhou2017Neural} or the introduction of answer-awareness \cite{zhao2018paragraph,du2017learning,dong2019unified}, which uses the answer to the desired question, or the position of the answer within the context paragraph as additional features. A combination of these techniques provide the base for state-of-the-art QG in recent years.&#10;&#10;More recently, other techniques have been proposed in order to perform QG. Reinforcement Learning (RL) have produced consistent results for the task by using policy gradients \cite{yuan2017machine}. The use of Transformers \cite{vaswani2017attention} over standard RNNs have also been adopted as these models provide the power of Attention in order to refer to specific points of context within the context paragraph, alleviating the RNN's memory bottleneck \cite{dong2019unified}.&#10;&#10;While all of these techniques are robust, they all employ complex models, extra features, and additional mechanisms that make them harder to train and expensive to reproduce. In this work, we show that transformer-based finetuning techniques can be used to create robust question generation systems using only a single pretrained language model, without the use of additional mechanisms, answer metadata, and extensive features. &#10;&#10;We show that our method, albeit simpler, produces results on par with the state-of-the-art. We benchmark standard language model finetuning on a reformatting of the SQuAD \cite{rajpurkar-etal-2016-squad} v.1.1 dataset and evaluate generation performance with standard language generation metrics. In addition, we perform a variety of analyses in order to isolate performance indicators within our model and identify its weaknesses and failure modes.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Deep Learning Techniques, Artificial Intelligence, Natural Language Generation, Question Generation Models" />
        </attvalues>
      </node>
      <node id="1704.01792" label="1704.01792">
        <attvalues>
          <attvalue for="0" value="Neural Question Generation from Text: A Preliminary Study" />
          <attvalue for="1" value="  Automatic question generation aims to generate questions from a text passage&#10;where the generated questions can be answered by certain sub-spans of the given&#10;passage. Traditional methods mainly use rigid heuristic rules to transform a&#10;sentence into related questions. In this work, we propose to apply the neural&#10;encoder-decoder model to generate meaningful and diverse questions from natural&#10;language sentences. The encoder reads the input text and the answer position,&#10;to produce an answer-aware input representation, which is fed to the decoder to&#10;generate an answer focused question. We conduct a preliminary study on neural&#10;question generation from text with the SQuAD dataset, and the experiment&#10;results show that our method can produce fluent and diverse questions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.00106" label="1705.00106">
        <attvalues>
          <attvalue for="0" value="Learning to Ask: Neural Question Generation for Reading Comprehension" />
          <attvalue for="1" value="  We study automatic question generation for sentences from text passages in&#10;reading comprehension. We introduce an attention-based sequence learning model&#10;for the task and investigate the effect of encoding sentence- vs.&#10;paragraph-level information. In contrast to all previous work, our model does&#10;not rely on hand-crafted rules or a sophisticated NLP pipeline; it is instead&#10;trainable end-to-end via sequence-to-sequence learning. Automatic evaluation&#10;results show that our system significantly outperforms the state-of-the-art&#10;rule-based system. In human evaluations, questions generated by our system are&#10;also rated as being more natural (i.e., grammaticality, fluency) and as more&#10;difficult to answer (in terms of syntactic and lexical divergence from the&#10;original text and reasoning needed to answer).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.02012" label="1705.02012">
        <attvalues>
          <attvalue for="0" value="Machine Comprehension by Text-to-Text Neural Question Generation" />
          <attvalue for="1" value="  We propose a recurrent neural model that generates natural-language questions&#10;from documents, conditioned on answers. We show how to train the model using a&#10;combination of supervised and reinforcement learning. After teacher forcing for&#10;standard maximum likelihood training, we fine-tune the model using policy&#10;gradient techniques to maximize several rewards that measure question quality.&#10;Most notably, one of these rewards is the performance of a question-answering&#10;system. We motivate question generation as a means to improve the performance&#10;of question answering systems. Our model is trained and evaluated on the recent&#10;question-answering dataset SQuAD.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.06483" label="2104.06483">
        <attvalues>
          <attvalue for="0" value="Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural&#10;  Morphological Inflection Models" />
          <attvalue for="1" value="  Deep learning sequence models have been successfully applied to the task of&#10;morphological inflection. The results of the SIGMORPHON shared tasks in the&#10;past several years indicate that such models can perform well, but only if the&#10;training data cover a good amount of different lemmata, or if the lemmata that&#10;are inflected at test time have also been seen in training, as has indeed been&#10;largely the case in these tasks. Surprisingly, standard models such as the&#10;Transformer almost completely fail at generalizing inflection patterns when&#10;asked to inflect previously unseen lemmata -- i.e. under &quot;wug test&quot;-like&#10;circumstances. While established data augmentation techniques can be employed&#10;to alleviate this shortcoming by introducing a copying bias through&#10;hallucinating synthetic new word forms using the alphabet in the language at&#10;hand, we show that, to be more effective, the hallucination process needs to&#10;pay attention to substrings of syllable-like length rather than individual&#10;characters or stems. We report a significant performance improvement with our&#10;substring-based hallucination model over previous data hallucination methods&#10;when training and test data do not overlap in their lemmata.&#10;" />
          <attvalue for="2" value="&#10;&#10;The Transformer model has delivered convincing results in many different tasks related to word-formation and analysis \cite{vylomova-etal-2020-sigmorphon}. Especially on inflection tasks, where an input lemma such as {dog}, and input inflectional features such as \{{N,PL}\}, are expected to produce an output such as {dogs}, the model has shown to be particularly adept at generalizing patterns \cite{wu2020applying,liu-hulden-2020-analogy}. However, we have discovered that this is only true if {some} variant of the input lemma to be inflected has been witnessed during training. In a ``wug test'' \cite{berko1958} setting where a previously unseen lemma---like {wug}---is to be inflected in some way, we find that the Transformer almost completely fails to generalize inflection patterns, despite abundant training data. It has been noted earlier that neural sequence-to-sequence models are apt to perform poorly if they have been exposed to little training data and that autoencoding on hallucinated forms could be useful \cite{kann-schutze-2017-unlabeled}. Our starting point is the observation that the poor ``wug test'' performance is maintained even with abundant training data. &#10;&#10;In our study, we show three main results. (1) We demonstrate that, even if trained with relatively large amounts of data, a Transformer model of the kind that has been very successful at recent shared tasks largely fails to generalize inflection patterns if it has not been exposed during training to lemmata in the test set. This is true even for datasets where all words inflect in the same way---i.e. there are no inflectional classes or allomorphs of morphemes, as is found in the low-resource Niger-Congo dataset used in SIGMORPHON 2020 shared task \cite{vylomova-etal-2020-sigmorphon}. (2) We also show that simply exposing the model to uninflected lemmata in the test set---without providing a single inflected form---allows the model to dramatically improve its performance when actually inflecting such lemmata. (3) Further, we investigate several strategies that avoid leveraging test set lemmata. We show that when inducing a copy bias in the model by hallucinating new lemmata, or by hallucinating new inflected forms, the method of hallucination is much more effective if it is sensitive to substrings of syllable-like length rather than individual characters or stems. Our best models significantly improve upon earlier state-of-the-art data hallucination methods such as \newcite{silfverberg-etal-2017-data} and \newcite{anastasopoulos-neubig-2019-pushing}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Data Augmentation Techniques, Linguistics, Artificial Intelligence, Morphological Inflection, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2006.11572" label="2006.11572">
        <attvalues>
          <attvalue for="0" value="SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological&#10;  Inflection" />
          <attvalue for="1" value="  A broad goal in natural language processing (NLP) is to develop a system that&#10;has the capacity to process any natural language. Most systems, however, are&#10;developed using data from just one language such as English. The SIGMORPHON&#10;2020 shared task on morphological reinflection aims to investigate systems'&#10;ability to generalize across typologically distinct languages, many of which&#10;are low resource. Systems were developed using data from 45 languages and just&#10;5 language families, fine-tuned with data from an additional 45 languages and&#10;10 language families (13 in total), and evaluated on all 90 languages. A total&#10;of 22 systems (19 neural) from 10 teams were submitted to the task. All four&#10;winning systems were neural (two monolingual transformers and two massively&#10;multilingual RNN-based models with gated attention). Most teams demonstrate&#10;utility of data hallucination and augmentation, ensembles, and multilingual&#10;training for low-resource languages. Non-neural learners and manually designed&#10;grammars showed competitive and even superior performance on some languages&#10;(such as Ingrian, Tajik, Tagalog, Zarma, Lingala), especially with very limited&#10;data. Some language families (Afro-Asiatic, Niger-Congo, Turkic) were&#10;relatively easy for most systems and achieved over 90% mean accuracy while&#10;others were more challenging.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.06106" label="1705.06106">
        <attvalues>
          <attvalue for="0" value="Unlabeled Data for Morphological Generation With Character-Based&#10;  Sequence-to-Sequence Models" />
          <attvalue for="1" value="  We present a semi-supervised way of training a character-based&#10;encoder-decoder recurrent neural network for morphological reinflection, the&#10;task of generating one inflected word form from another. This is achieved by&#10;using unlabeled tokens or random strings as training data for an autoencoding&#10;task, adapting a network for morphological reinflection, and performing&#10;multi-task training. We thus use limited labeled data more effectively,&#10;obtaining up to 9.9% improvement over state-of-the-art baselines for 8&#10;different languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.12202" label="2304.12202">
        <attvalues>
          <attvalue for="0" value="ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the&#10;  LexGLUE benchmark" />
          <attvalue for="1" value="  Following the hype around OpenAI's ChatGPT conversational agent, the last&#10;straw in the recent development of Large Language Models (LLMs) that&#10;demonstrate emergent unprecedented zero-shot capabilities, we audit the latest&#10;OpenAI's GPT-3.5 model, `gpt-3.5-turbo', the first available ChatGPT model, in&#10;the LexGLUE benchmark in a zero-shot fashion providing examples in a templated&#10;instruction-following format. The results indicate that ChatGPT achieves an&#10;average micro-F1 score of 47.6% across LexGLUE tasks, surpassing the baseline&#10;guessing rates. Notably, the model performs exceptionally well in some&#10;datasets, achieving micro-F1 scores of 62.8% and 70.2% in the ECtHR B and&#10;LEDGAR datasets, respectively. The code base and model predictions are&#10;available for review on https://github.com/coastalcph/zeroshot_lexglue.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advances in Large Language Models (LLMs)~\cite{brown-etal-2020-gpt3,chowdhery-etal-2022-palm}, also known as Foundation Models~\cite{bommasani-etal-2022-foundation}, have challenged the traditional supervised learning paradigm of fine-tuning by demonstrating emergent zero-shot Natural Language Understanding (NLU) capabilities~\cite{wei2022emergent} through scaling the model's size in billions of parameters~\cite{kaplan-etal-2020-scale}. &#10;&#10;OpenAI's latest conversational agent, ChatGPT~\cite{openai-2022-chatgpt}, a successor of InstructGPT~\cite{ouyang-etal-2022-instructgpt} -also known as GPT-3.5- models, is an instruction-following transformer-based language model, which has been further trained (aligned) with reinforcement learning from human feedback (RLHF) \cite{christiano-etal-2017-drl}. ChatGPT demonstrates unprecedented emergent capabilities in zero-shot Question-Answering (QA) capabilities that cover common sense knowledge, but also extend to specialized domains such as problem solving, programming/debugging, and law, as presented by many users in the web. &#10;&#10;Recently, \cite{bommarito-bar-exams-2023} audited several variants of OpenAI's GPT 2/3/3.5 models in legal bar exam questions, and found that the most advanced -at the time- model (`text-davinci-003') achieves an accuracy of 50.3\% on a complete practice exam, significantly in excess of the 25\% baseline guessing rate, while it performs at a passing rate in two legal areas (Evidence and Torts). In a follow-up work, \cite{bommarito-etal-2023-gpt} assessed the model's performance in accounting certification exams, where the model significantly under-performs human capabilities with a correct rate of 14.4\%.&#10;&#10;Following the work of \cite{bommarito-bar-exams-2023}, we evaluate the latest OpenAI's GPT-3.5 model \cite{ouyang-etal-2022-instructgpt} (`gpt-3.5-turbo', v. March 2023), the first available ChatGPT, on legal text classification tasks from the LexGLUE \cite{chalkidis-etal-2022-lexglue} benchmark in a zero-shot fashion providing examples in a templated instruction-following format, similar to those used by \cite{chung-etal-2023-flant5}. We find that ChatGPT achieves an average micro-F1 score of 49.0\% across LexGLUE tasks in a zero-shot setting, significantly in excess of the baseline guessing rates, while the model performs exceptionally well in some datasets achieving micro-F1 scores of 62.8\% and 70.1\% in the ECtHR B and LEDGAR datasets.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Conversational Agent Performance, Artificial Intelligence Benchmarking, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2212.14402" label="2212.14402">
        <attvalues>
          <attvalue for="0" value="GPT Takes the Bar Exam" />
          <attvalue for="1" value="  Nearly all jurisdictions in the United States require a professional license&#10;exam, commonly referred to as &quot;the Bar Exam,&quot; as a precondition for law&#10;practice. To even sit for the exam, most jurisdictions require that an&#10;applicant completes at least seven years of post-secondary education, including&#10;three years at an accredited law school. In addition, most test-takers also&#10;undergo weeks to months of further, exam-specific preparation. Despite this&#10;significant investment of time and capital, approximately one in five&#10;test-takers still score under the rate required to pass the exam on their first&#10;try. In the face of a complex task that requires such depth of knowledge, what,&#10;then, should we expect of the state of the art in &quot;AI?&quot; In this research, we&#10;document our experimental evaluation of the performance of OpenAI's&#10;`text-davinci-003` model, often-referred to as GPT-3.5, on the multistate&#10;multiple choice (MBE) section of the exam. While we find no benefit in&#10;fine-tuning over GPT-3.5's zero-shot performance at the scale of our training&#10;data, we do find that hyperparameter optimization and prompt engineering&#10;positively impacted GPT-3.5's zero-shot performance. For best prompt and&#10;parameters, GPT-3.5 achieves a headline correct rate of 50.3% on a complete&#10;NCBE MBE practice exam, significantly in excess of the 25% baseline guessing&#10;rate, and performs at a passing rate for both Evidence and Torts. GPT-3.5's&#10;ranking of responses is also highly-correlated with correctness; its top two&#10;and top three choices are correct 71% and 88% of the time, respectively,&#10;indicating very strong non-entailment performance. While our ability to&#10;interpret these results is limited by nascent scientific understanding of LLMs&#10;and the proprietary nature of GPT, we believe that these results strongly&#10;suggest that an LLM will pass the MBE component of the Bar Exam in the near&#10;future.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.04408" label="2301.04408">
        <attvalues>
          <attvalue for="0" value="GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities" />
          <attvalue for="1" value="  The global economy is increasingly dependent on knowledge workers to meet the&#10;needs of public and private organizations. While there is no single definition&#10;of knowledge work, organizations and industry groups still attempt to measure&#10;individuals' capability to engage in it. The most comprehensive assessment of&#10;capability readiness for professional knowledge workers is the Uniform CPA&#10;Examination developed by the American Institute of Certified Public Accountants&#10;(AICPA). In this paper, we experimentally evaluate OpenAI's `text-davinci-003`&#10;and prior versions of GPT on both a sample Regulation (REG) exam and an&#10;assessment of over 200 multiple-choice questions based on the AICPA Blueprints&#10;for legal, financial, accounting, technology, and ethical tasks. First, we find&#10;that `text-davinci-003` achieves a correct rate of 14.4% on a sample REG exam&#10;section, significantly underperforming human capabilities on quantitative&#10;reasoning in zero-shot prompts. Second, `text-davinci-003` appears to be&#10;approaching human-level performance on the Remembering &amp; Understanding and&#10;Application skill levels in the Exam absent calculation. For best prompt and&#10;parameters, the model answers 57.6% of questions correctly, significantly&#10;better than the 25% guessing rate, and its top two answers are correct 82.1% of&#10;the time, indicating strong non-entailment. Finally, we find that recent&#10;generations of GPT-3 demonstrate material improvements on this assessment,&#10;rising from 30% for `text-davinci-001` to 57% for `text-davinci-003`. These&#10;findings strongly suggest that large language models have the potential to&#10;transform the quality and efficiency of future knowledge work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.00976" label="2110.00976">
        <attvalues>
          <attvalue for="0" value="LexGLUE: A Benchmark Dataset for Legal Language Understanding in English" />
          <attvalue for="1" value="  Laws and their interpretations, legal arguments and agreements\ are typically&#10;expressed in writing, leading to the production of vast corpora of legal text.&#10;Their analysis, which is at the center of legal practice, becomes increasingly&#10;elaborate as these collections grow in size. Natural language understanding&#10;(NLU) technologies can be a valuable tool to support legal practitioners in&#10;these endeavors. Their usefulness, however, largely depends on whether current&#10;state-of-the-art models can generalize across various tasks in the legal&#10;domain. To answer this currently open question, we introduce the Legal General&#10;Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets&#10;for evaluating model performance across a diverse set of legal NLU tasks in a&#10;standardized way. We also provide an evaluation and analysis of several generic&#10;and legal-oriented models demonstrating that the latter consistently offer&#10;performance improvements across multiple tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.02252" label="2206.02252">
        <attvalues>
          <attvalue for="0" value="Exploring Cross-lingual Textual Style Transfer with Large Multilingual&#10;  Language Models" />
          <attvalue for="1" value="  Detoxification is a task of generating text in polite style while preserving&#10;meaning and fluency of the original toxic text. Existing detoxification methods&#10;are designed to work in one exact language. This work investigates multilingual&#10;and cross-lingual detoxification and the behavior of large multilingual models&#10;like in this setting. Unlike previous works we aim to make large language&#10;models able to perform detoxification without direct fine-tuning in given&#10;language. Experiments show that multilingual models are capable of performing&#10;multilingual style transfer. However, models are not able to perform&#10;cross-lingual detoxification and direct fine-tuning on exact language is&#10;inevitable.&#10;" />
          <attvalue for="2" value="&#10;&#10;The task of Textual Style Transfer (Textual Style Transfer) can be viewed as a task where certain properties of text are being modified while rest retain the same. In this work we focus on detoxification textual style transfer \cite{santos2018fighting,dementieva2021methods}. It can be formulated as follows: given two text corpora $D^X=\{x_1, x_2, \dots x_n\}$ and $D^Y=\{y_1, y_2, \dots, y_n\}$, where $X$, $Y$ - are two sets of all possible text in styles $s^X$, $s^Y$ respectively, we want to build a model $f_{\theta} : X arrow Y$, such that the probability $p(y_{gen}|x, s^X, s^Y)$ of transferring the style $s^X$ of given text $x$ (by generation $y_{gen}$) to the style $s^Y$ is maximized (where $s^X$ and $s^Y$ are toxic and non-toxic styles respectively).&#10;&#10;Some examples of detoxification presented in Table \ref{tab:detoxification_example}.&#10;&#10;Textual style transfer gained a lot of attention with a rise of deep learning-based NLP methods. Given that, Textual Style Transfer has now a lot of specific subtasks ranging from formality style transfer \cite{rao-tetreault-2018-dear, yao2021improving} and simplification of domain-specific texts \cite{devaraj2021paragraphlevel, maddela2021controllable} to emotion modification \cite{sharma2021facilitating} and detoxification (debiasing) \cite{li2020stylecontent, dementieva2021methods}. &#10;&#10;There exist a variety of Textual Style Transfer methods: from totally supervised methods \cite{Wang2019HarnessingPN,zhang2020parallel,dementieva2021methods} which require a parallel text corpus for training to unsupervised \cite{shen2017style,wang2019controllable,xu2021vae} that are designed to work without any parallel data. The latter sub-field of research is more popular nowadays due to the scarcity of parallel text data for Textual Style Transfer. On the other hand, if we address Textual Style Transfer task as a Machine Translation task we get a significant performance boost \cite{prabhumoye2018style}. &#10;&#10;The task of detoxification, in which we focus in this work, is relatively new. First work on detoxification was a sequence-to-sequence collaborative classifier, attention and the cycle consistency loss \cite{nogueira-dos-santos-etal-2018-fighting}. A recent work by \cite{laugier2021civil} introduces self-supervised model based on T$5$ model \cite{raffel2020exploring} with a denoising and cyclic auto-encoder loss. &#10;&#10;Both these methods are unsupervised which is an advantage but it comes from the major current problem of the textual style transfer. There is a lack of parallel data for Textual Style Transfer since there exist only few parallel datasets for English \cite{rao-tetreault-2018-dear} and some other languages \cite{briakou2021xformal}. When it comes to detoxification there are only two parallel detoxification corpora available now and they both appeared only last year \cite{dementieva2021crowdsourcing}. Most state-of-the-art methods rely on large amounts of text data which is often available for some well-researched languages like English but lacking for other languages almost entirely. Therefore, it is important to study whether cross-lingual (or at least multilingual) detoxification is possible. &#10;&#10;Multilingual language models such as mBART \cite{liu2020multilingual}, mT5 \cite{xue2021mt5} have recently become available. This work explores the possibility of multilingual and cross-lingual textual style transfer (Textual Style Transfer) using such large multilingual language models. We test the hypothesis that modern large text-to-text models are able to generalize ability of style transfer across languages. &#10;&#10;Our contributions can be summarized as follows:&#10;\begin{enumerate}&#10; \item We introduce a novel study of multilingual textual style transfer and conduct experiments with several multilingual language models and evaluate their performance. &#10; \item We conduct cross-lingual Textual Style Transfer experiments to investigate whether multilingual language models are able to perform Textual Style Transfer without fine-tuning on a specific language.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Processing, Computer Science, Linguistics, Text Detoxification, Multilingual Models, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2105.09052" label="2105.09052">
        <attvalues>
          <attvalue for="0" value="Methods for Detoxification of Texts for the Russian Language" />
          <attvalue for="1" value="  We introduce the first study of automatic detoxification of Russian texts to&#10;combat offensive language. Such a kind of textual style transfer can be used,&#10;for instance, for processing toxic content in social media. While much work has&#10;been done for the English language in this field, it has never been solved for&#10;the Russian language yet. We test two types of models - unsupervised approach&#10;based on BERT architecture that performs local corrections and supervised&#10;approach based on pretrained language GPT-2 model - and compare them with&#10;several baselines. In addition, we describe evaluation setup providing training&#10;datasets and metrics for automatic evaluation. The results show that the tested&#10;approaches can be successfully used for detoxification, although there is room&#10;for improvement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.05767" label="2104.05767">
        <attvalues>
          <attvalue for="0" value="Paragraph-level Simplification of Medical Texts" />
          <attvalue for="1" value="  We consider the problem of learning to simplify medical texts. This is&#10;important because most reliable, up-to-date information in biomedicine is dense&#10;with jargon and thus practically inaccessible to the lay audience. Furthermore,&#10;manual simplification does not scale to the rapidly growing body of biomedical&#10;literature, motivating the need for automated approaches. Unfortunately, there&#10;are no large-scale resources available for this task. In this work we introduce&#10;a new corpus of parallel texts in English comprising technical and lay&#10;summaries of all published evidence pertaining to different clinical topics. We&#10;then propose a new metric based on likelihood scores from a masked language&#10;model pretrained on scientific texts. We show that this automated measure&#10;better differentiates between technical and lay summaries than existing&#10;heuristics. We introduce and evaluate baseline encoder-decoder Transformer&#10;models for simplification and propose a novel augmentation to these in which we&#10;explicitly penalize the decoder for producing &quot;jargon&quot; terms; we find that this&#10;yields improvements over baselines in terms of readability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.07714" label="2101.07714">
        <attvalues>
          <attvalue for="0" value="Towards Facilitating Empathic Conversations in Online Mental Health&#10;  Support: A Reinforcement Learning Approach" />
          <attvalue for="1" value="  Online peer-to-peer support platforms enable conversations between millions&#10;of people who seek and provide mental health support. If successful, web-based&#10;mental health conversations could improve access to treatment and reduce the&#10;global disease burden. Psychologists have repeatedly demonstrated that empathy,&#10;the ability to understand and feel the emotions and experiences of others, is a&#10;key component leading to positive outcomes in supportive conversations.&#10;However, recent studies have shown that highly empathic conversations are rare&#10;in online mental health platforms.&#10;  In this paper, we work towards improving empathy in online mental health&#10;support conversations. We introduce a new task of empathic rewriting which aims&#10;to transform low-empathy conversational posts to higher empathy. Learning such&#10;transformations is challenging and requires a deep understanding of empathy&#10;while maintaining conversation quality through text fluency and specificity to&#10;the conversational context. Here we propose PARTNER, a deep reinforcement&#10;learning agent that learns to make sentence-level edits to posts in order to&#10;increase the expressed level of empathy while maintaining conversation quality.&#10;Our RL agent leverages a policy network, based on a transformer language model&#10;adapted from GPT-2, which performs the dual task of generating candidate&#10;empathic sentences and adding those sentences at appropriate positions. During&#10;training, we reward transformations that increase empathy in posts while&#10;maintaining text fluency, context specificity and diversity. Through a&#10;combination of automatic and human evaluation, we demonstrate that PARTNER&#10;successfully generates more empathic, specific, and diverse responses and&#10;outperforms NLP methods from related tasks like style transfer and empathic&#10;dialogue generation. Our work has direct implications for facilitating empathic&#10;conversations on web-based platforms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.07419" label="2012.07419">
        <attvalues>
          <attvalue for="0" value="The Style-Content Duality of Attractiveness: Learning to Write&#10;  Eye-Catching Headlines via Disentanglement" />
          <attvalue for="1" value="  Eye-catching headlines function as the first device to trigger more clicks,&#10;bringing reciprocal effect between producers and viewers. Producers can obtain&#10;more traffic and profits, and readers can have access to outstanding articles.&#10;When generating attractive headlines, it is important to not only capture the&#10;attractive content but also follow an eye-catching written style. In this&#10;paper, we propose a Disentanglement-based Attractive Headline Generator (DAHG)&#10;that generates headline which captures the attractive content following the&#10;attractive style. Concretely, we first devise a disentanglement module to&#10;divide the style and content of an attractive prototype headline into latent&#10;spaces, with two auxiliary constraints to ensure the two spaces are indeed&#10;disentangled. The latent content information is then used to further polish the&#10;document representation and help capture the salient part. Finally, the&#10;generator takes the polished document as input to generate headline under the&#10;guidance of the attractive style. Extensive experiments on the public Kuaibao&#10;dataset show that DAHG achieves state-of-the-art performance. Human evaluation&#10;also demonstrates that DAHG triggers 22% more clicks than existing models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.06284" label="1809.06284">
        <attvalues>
          <attvalue for="0" value="Style Transfer Through Multilingual and Feedback-Based Back-Translation" />
          <attvalue for="1" value="  Style transfer is the task of transferring an attribute of a sentence (e.g.,&#10;formality) while maintaining its semantic content. The key challenge in style&#10;transfer is to strike a balance between the competing goals, one to preserve&#10;meaning and the other to improve the style transfer accuracy. Prior research&#10;has identified that the task of meaning preservation is generally harder to&#10;attain and evaluate. This paper proposes two extensions of the state-of-the-art&#10;style transfer models aiming at improving the meaning preservation in style&#10;transfer. Our evaluation shows that these extensions help to ground meaning&#10;better while improving the transfer accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.05456" label="2102.05456">
        <attvalues>
          <attvalue for="0" value="Civil Rephrases Of Toxic Texts With Self-Supervised Transformers" />
          <attvalue for="1" value="  Platforms that support online commentary, from social networks to news sites,&#10;are increasingly leveraging machine learning to assist their moderation&#10;efforts. But this process does not typically provide feedback to the author&#10;that would help them contribute according to the community guidelines. This is&#10;prohibitively time-consuming for human moderators to do, and computational&#10;approaches are still nascent. This work focuses on models that can help suggest&#10;rephrasings of toxic comments in a more civil manner. Inspired by recent&#10;progress in unpaired sequence-to-sequence tasks, a self-supervised learning&#10;model is introduced, called CAE-T5. CAE-T5 employs a pre-trained text-to-text&#10;transformer, which is fine tuned with a denoising and cyclic auto-encoder loss.&#10;Experimenting with the largest toxicity detection dataset to date (Civil&#10;Comments) our model generates sentences that are more fluent and better at&#10;preserving the initial content compared to earlier text style transfer systems&#10;which we compare with using several scoring systems and human evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.03391" label="2207.03391">
        <attvalues>
          <attvalue for="0" value="Non-Linear Pairwise Language Mappings for Low-Resource Multilingual&#10;  Acoustic Model Fusion" />
          <attvalue for="1" value="  Multilingual speech recognition has drawn significant attention as an&#10;effective way to compensate data scarcity for low-resource languages.&#10;End-to-end (e2e) modelling is preferred over conventional hybrid systems,&#10;mainly because of no lexicon requirement. However, hybrid DNN-HMMs still&#10;outperform e2e models in limited data scenarios. Furthermore, the problem of&#10;manual lexicon creation has been alleviated by publicly available trained&#10;models of grapheme-to-phoneme (G2P) and text to IPA transliteration for a lot&#10;of languages. In this paper, a novel approach of hybrid DNN-HMM acoustic models&#10;fusion is proposed in a multilingual setup for the low-resource languages.&#10;Posterior distributions from different monolingual acoustic models, against a&#10;target language speech signal, are fused together. A separate regression neural&#10;network is trained for each source-target language pair to transform posteriors&#10;from source acoustic model to the target language. These networks require very&#10;limited data as compared to the ASR training. Posterior fusion yields a&#10;relative gain of 14.65% and 6.5% when compared with multilingual and&#10;monolingual baselines respectively. Cross-lingual model fusion shows that the&#10;comparable results can be achieved without using posteriors from the language&#10;dependent ASR.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the advancements of the computational resources, many Deep Neural Networks (DNNs) architectures and networks have been proposed to make speech recognition more efficient and accurate. DNN-HMM hybrid systems \cite{madikeri2020} outperform conventional GMM-HMM systems. For end-to-end (e2e) speech recognition, sequence-to-sequence models \cite{cho18}, RNN transducers (RNN-T) \cite{kannan2019}, transformers \cite{vishwas20} and unsupervised learning \cite{conneau21} are being used. These systems can be further improved with coupling of various techniques such as multi-task learning (MTL) \cite{sailor2020}, mixture of experts (MOE) \cite{neeraj21} and learning hidden unit contributions (LHUC) \cite{ml_epfl_thesis20} depending on the task. All these statistical modelling techniques require a lot of data for reliable parameters estimation. However, out of nearly 7000 languages being spoken around the world, just 23 languages are spoken by more than half of the world's population \cite{ethnologue}. So, sufficient data resources are available for few languages.&#10;&#10;Over the past decade, multilingual automatic speech recognition systems have stolen the limelight being an effective way to compensate the data scarcity for low-resource languages \cite{abate20,tachbelie20,martin16,besacier14,imseng14,vu13_interspeech}. DNN based multilingual acoustic models (AM) can be used to extract features to train a monolingual model \cite{frantisek14,karel12,arnab13} or multilingual models can directly be adapted to target language \cite{tong18,huang13}. Though e2e multilingual speech recognition systems are preferred over conventional ASR to avoid lexicon creations, DNN-HMMs still outperform e2e models in limited data scenarios such as low-resource languages. Furthermore, the advancement of G2P and text to IPA transliteration approaches such as Phonetisarus \cite{phonetisarus}, Epitran \cite{epitran} and open source LanguageNet G2P models \cite{langnet} for many languages have alleviated the problem of manual creation of lexicons.&#10;&#10;Previous work on e2e multilingual speech recognition systems has shown that a multilingual setup does not guarantee the reduction in Word Error Rate (WER) for target languages \cite{conneau21,Pratap2020,hou20}. Recent efforts to interpret the learning of multilingual speech recognition systems \cite{zelasko20,feng21} observe that Phoneme Error Rate (PER) of an overlapped phoneme is not reduced with the growing number of sharing languages. The number of shared phonemes is not a reliable metric to measure language similarities and each participating language in a multilingual system has a different similarity with the target language. Even the balanced language data sampling can cause degradation or improvement due to internal acoustic-phonetic unbalancing \cite{us}. It demands very controlled language mixing for a target language ASR.&#10;&#10;To that end, a novel technique is proposed to fuse outputs of different monolingual models against the target language speech. Various previous studies on monolingual speech recognition have fused outputs from different models for acoustic \cite{aziz18,Ilyes17,mallidi16} and language models \cite{lmFusion1, lmFusion2}. However, monolingual models have never been fused in a multilingual setup because it can not be done straightforwardly due to different phonetic decision trees of monolingual models. In this work, a separate regression neural network (mapping network) is trained for each $&lt;$source, target$&gt;$ pair to map posteriors from a source language AM to the posteriors of the target language AM. The mapped posteriors are then fused in multilingual and cross-lingual fashion for phoneme recognition of the target language. The intuition is that a mapping network is able to learn some language related relationships between posterior distributions of source and target acoustic models. The proposed approach is helpful especially for low-resource languages because;&#10;\begin{itemize}&#10; \item the mapping networks can be trained with very limited amounts of data since a few hours can provide sufficient examples for phonetic level training.&#10; \item controlled fusion of posteriors based on language similarity will allow to control contribution of different source languages. &#10;\end{itemize}&#10;&#10;The mapped posteriors from the monolingual AMs are fused in a multilingual setup which not only outperforms the classical multilingual systems, but also the monolingual ASRs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Electrical Engineering, Linguistics, Multilingual Speech Recognition, Language Resource Optimization, Signal Processing" />
        </attvalues>
      </node>
      <node id="2005.08118" label="2005.08118">
        <attvalues>
          <attvalue for="0" value="That Sounds Familiar: an Analysis of Phonetic Representations Transfer&#10;  Across Languages" />
          <attvalue for="1" value="  Only a handful of the world's languages are abundant with the resources that&#10;enable practical applications of speech processing technologies. One of the&#10;methods to overcome this problem is to use the resources existing in other&#10;languages to train a multilingual automatic speech recognition (ASR) model,&#10;which, intuitively, should learn some universal phonetic representations. In&#10;this work, we focus on gaining a deeper understanding of how general these&#10;representations might be, and how individual phones are getting improved in a&#10;multilingual setting. To that end, we select a phonetically diverse set of&#10;languages, and perform a series of monolingual, multilingual and crosslingual&#10;(zero-shot) experiments. The ASR is trained to recognize the International&#10;Phonetic Alphabet (IPA) token sequences. We observe significant improvements&#10;across all languages in the multilingual setting, and stark degradation in the&#10;crosslingual setting, where the model, among other errors, considers Javanese&#10;as a tone language. Notably, as little as 10 hours of the target language&#10;training data tremendously reduces ASR error rates. Our analysis uncovered that&#10;even the phones that are unique to a single language can benefit greatly from&#10;adding training data from other languages - an encouraging result for the&#10;low-resource speech community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12104" label="2010.12104">
        <attvalues>
          <attvalue for="0" value="How Phonotactics Affect Multilingual and Zero-shot ASR Performance" />
          <attvalue for="1" value="  The idea of combining multiple languages' recordings to train a single&#10;automatic speech recognition (ASR) model brings the promise of the emergence of&#10;universal speech representation. Recently, a Transformer encoder-decoder model&#10;has been shown to leverage multilingual data well in IPA transcriptions of&#10;languages presented during training. However, the representations it learned&#10;were not successful in zero-shot transfer to unseen languages. Because that&#10;model lacks an explicit factorization of the acoustic model (AM) and language&#10;model (LM), it is unclear to what degree the performance suffered from&#10;differences in pronunciation or the mismatch in phonotactics. To gain more&#10;insight into the factors limiting zero-shot ASR transfer, we replace the&#10;encoder-decoder with a hybrid ASR system consisting of a separate AM and LM.&#10;Then, we perform an extensive evaluation of monolingual, multilingual, and&#10;crosslingual (zero-shot) acoustic and language models on a set of 13&#10;phonetically diverse languages. We show that the gain from modeling&#10;crosslingual phonotactics is limited, and imposing a too strong model can hurt&#10;the zero-shot transfer. Furthermore, we find that a multilingual LM hurts a&#10;multilingual ASR system's performance, and retaining only the target language's&#10;phonotactic data in LM training is preferable.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.21066" label="2407.21066">
        <attvalues>
          <attvalue for="0" value="ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech&#10;  Processing Tasks" />
          <attvalue for="1" value="  Self-supervised learning has emerged as a key approach for learning generic&#10;representations from speech data. Despite promising results in downstream tasks&#10;such as speech recognition, speaker verification, and emotion recognition, a&#10;significant number of parameters is required, which makes fine-tuning for each&#10;task memory-inefficient. To address this limitation, we introduce ELP-adapter&#10;tuning, a novel method for parameter-efficient fine-tuning using three types of&#10;adapter, namely encoder adapters (E-adapters), layer adapters (L-adapters), and&#10;a prompt adapter (P-adapter). The E-adapters are integrated into&#10;transformer-based encoder layers and help to learn fine-grained speech&#10;representations that are effective for speech recognition. The L-adapters&#10;create paths from each encoder layer to the downstream head and help to extract&#10;non-linguistic features from lower encoder layers that are effective for&#10;speaker verification and emotion recognition. The P-adapter appends pseudo&#10;features to CNN features to further improve effectiveness and efficiency. With&#10;these adapters, models can be quickly adapted to various speech processing&#10;tasks. Our evaluation across four downstream tasks using five backbone models&#10;demonstrated the effectiveness of the proposed method. With the WavLM backbone,&#10;its performance was comparable to or better than that of full fine-tuning on&#10;all tasks while requiring 90% fewer learnable parameters.&#10;" />
          <attvalue for="2" value="&#10;In the field of audio and speech processing, self-supervised learning using large-scale unlabeled datasets has become a leading approach for extracting generic representations from speech data~\cite{schneider2020wav2vec, baevski2020wav2vec2, wang2021unispeech, hsu2021hubert, chen2022unispeech-sat, chen2022wavlm, qian2022contentvec}. The main idea of this approach is to leverage the inherent structures and patterns within the speech data to train models via representation learning loss such as contrastive loss~\cite{chen2020simclr, jiang2021speechsimclr, huh2020augmentation, inoue2020semi}. This significantly reduces the need for manually labeled data, making model training more scalable and efficient.&#10;Examples of models trained by self-supervised learning, which we refer to as self-supervised models,&#10;include wav2vec~\cite{schneider2020wav2vec, baevski2020wav2vec2}, HuBERT~\cite{hsu2021hubert}, and WavLM~\cite{chen2022wavlm}. These models have demonstrated the ability to extract task-independent features with transformer-based architectures.&#10;&#10;In recent years, the range of speech processing tasks that can be covered by self-supervised models has been steadily expanding beyond automatic speech recognition.&#10;For example, a number of studies have proposed methods that utilize speech embeddings extracted from self-supervised models for discriminative tasks such as speaker verification\cite{finetune-sv, fan21wav2vecsv, lee22wav2vecsv, peng2023improving} and speech emotion recognition~\cite{finetune-er, pepino21wav2vecer}.&#10;Some pioneering studies have demonstrated the effectiveness of self-supervised models in addressing more complex and generative tasks.&#10;For example, spoken question answering is an important line of research focused on developing models capable of understanding and responding to questions posed in natural spoken language, where recent studies leverage self-supervised models~\cite{sqa1, sqa2, sqa3, sqa4, sqa5, sqa6}.&#10;It has also been demonstrated that self-supervised models can perform voice conversion effectively and efficiently by integrating a decoder and a vocoder~\cite{vc0, vc1, vc2, vc3, vc4}.&#10;These studies highlight the potential of self-supervised models across various speech tasks.&#10;&#10;To apply self-supervised models to downstream tasks, fine-tuning on task-specific labeled datasets is often required.&#10;This process enables the models to adapt and specialize in specific tasks, leading to excellent results not only in speech recognition but also in various speech tasks.&#10;However, one limitation is the substantial number of parameters involved. When fine-tuning is conducted for each downstream task, multiple models must be stored, one for each task. This can lead to storage inefficiencies in real-world application settings, such as when each user wants to fine-tune the model with their private data and task.&#10;&#10;A parameter-efficient method for adapting self-supervised models to various downstream tasks is thus desirable.&#10;Learning task-specific downstream head modules, such as a linear classification head, with frozen self-supervised models is an efficient solution; however, it often degrades the final performance compared to that obtained by fine-tuning all parameters because the optimal features can differ substantially depending on each task. For instance, linguistic features that include phoneme information are crucial for speech recognition, whereas non-linguistic features are crucial for speaker verification.&#10;&#10;Recently, learning with adapter modules that can be inserted into the intermediate encoder layers of a frozen model has emerged as a promising approach for parameter-efficient fine-tuning.&#10;The first adapter tuning method~\cite{nlp-adapter} was proposed for BERT \cite{bert} in the field of natural language processing,&#10;where two adapter modules are inserted into each encoder layer of BERT.&#10;Each adapter module consists of two linear layers with an activation between them and a skip connection. This approach requires fewer parameters (the frozen parameters are shared among all downstream tasks) without degrading accuracy.&#10;A number of follow-up studies have used adapters for various natural language processing tasks \cite{lin2020exploring, Guo2021AdaptiveAdapters, Zhang2023PoE}.&#10;&#10;For speech recognition, Kannan~{et al.}~\cite{rnnt-adapter} integrated adapter modules into recurrent neural network transducers.&#10;Hou~{et al.}~\cite{hou21meta, hou22adapters}&#10;proposed the use of adapters for cross-lingual speech adaptation.&#10;Winata~{et al.}~\cite{ctcattn-adapter} proposed the adapt-and-adjust framework, which uses adapter modules for multilingual speech recognition based on hybrid connectionist temporal classification (CTC)-attention networks.&#10;Qian~{et al.}~\cite{Qian2022LayerWiseFastAdaptation} proposed gated and multi-basis adapters for multi-accent speech recognition.&#10;The effectiveness of adapter tuning has also been demonstrated in other speech processing tasks such as speech translation~\cite{sl-adapter}.&#10;&#10;Some recent studies have explored the application of adapter tuning to self-supervised models.&#10;Thomas~{et al.}~\cite{speech-adapter} introduced adapter modules into wav2vec2.0 for speech recognition.&#10;Chen~{et al.}~\cite{Chen2022EfficientTuning} compared the adapter modules with other efficient fine-tuning methods such as low-rank adaptation (LoRA)~\cite{hu22lora}.&#10;It is also reported that various acoustic and linguistic features tend to&#10;be encoded in different layers in wav2vec2.0~\cite{analysis1, analysis2}.&#10;These studies inspired us to develop an adapter tuning method for not only speech recognition but also various other speech processing tasks.&#10;&#10;In this work, we propose ELP-adapter tuning, a parameter-efficient fine-tuning method that utilizes three types of adapter, namely encoder adapters (E-adapters), layer adapters (L-adapters), and a prompt adapter (P-adapter). Each adapter is a small learnable module that has a distinct role in enhancing performance in downstream tasks.&#10;Given a frozen self-supervised model that consists of multiple encoder layers, the E-adapters are integrated into the transformer-based encoder layers. They help to extract fine-grained linguistic representations and improve speech recognition performance. The L-adapters create paths from each encoder layer to the downstream head. This improves the performance of tasks such as emotion recognition and speaker verification, as features extracted from intermediate encoder layers often help to capture non-linguistic features. The P-adapter appends learnable embeddings that are used as auxiliary inputs to the transformer-based encoders. This further enhances learning effectiveness and efficiency.&#10;&#10;In experiments, we applied ELP-adapter tuning to four downstream tasks, namely automatic speech recognition (ASR), automatic speaker verification (ASV), speech emotion recognition (SER), and speech intent classification (SIC).&#10;With the WavLM backbone, our method achieved performance comparable to or even better than that of full fine-tuning while using 90\% fewer learnable parameters. Further, we visualized the weight coefficients for each layer to explain the improvement obtained with our method.&#10;&#10;This paper is an extended version of our previously published paper~\cite{otake2023parameter} at ICASSP 2023. Compared to the previous version, we have made the following significant extensions:&#10;\begin{enumerate}&#10;\item We introduced a P-adapter that can be utilized in conjunction with the previously proposed E-adapters and L-adapters.&#10;\item We demonstrated the effectiveness of ELP-adapter tuning across multiple self-supervised models. Specifically, we expanded our evaluations to include wav2vec2.0~\cite{baevski2020wav2vec2}, HuBERT~\cite{hsu2021hubert}, ContentVec~\cite{qian2022contentvec}, and WavLM+~\cite{chen2022wavlm}.&#10;\item We thoroughly conducted experimental evaluation with multiple conventional fine-tuning methods including&#10;weight tuning~\cite{chen2022wavlm}, LoRA tuning~\cite{hu22lora}, Prefix tuning~\cite{li21prefixtuning}, and Efficient adapter tuning~\cite{speech-adapter}.&#10;\end{enumerate}&#10;&#10;The rest of this paper is organized as follows.&#10;Section~\ref{sec:conventional} reviews conventional self-supervised models and fine-tuning methods.&#10;Section~\ref{sec:method} introduces ELP-adapter tuning for parameter-efficient fine-tuning.&#10;Sections~\ref{sec:ASR}-\ref{sec:SIC}&#10;respectively present experiments on ASR, ASV, SER, and SIC tasks.&#10;Section~\ref{sec:analysis} provides detailed analysis on layer weights and adapter configurations.&#10;Finally, Section~\ref{sec:conclusion} concludes this paper and discusses future research directions.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Parameter-Efficient Tuning, Signal Processing, Mathematics, Speech Recognition, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2109.15053" label="2109.15053">
        <attvalues>
          <attvalue for="0" value="Fine-tuning wav2vec2 for speaker recognition" />
          <attvalue for="1" value="  This paper explores applying the wav2vec2 framework to speaker recognition&#10;instead of speech recognition. We study the effectiveness of the pre-trained&#10;weights on the speaker recognition task, and how to pool the wav2vec2 output&#10;sequence into a fixed-length speaker embedding. To adapt the framework to&#10;speaker recognition, we propose a single-utterance classification variant with&#10;CE or AAM softmax loss, and an utterance-pair classification variant with BCE&#10;loss. Our best performing variant, w2v2-aam, achieves a 1.88% EER on the&#10;extended voxceleb1 test set compared to 1.69% EER with an ECAPA-TDNN baseline.&#10;Code is available at https://github.com/nikvaessen/w2v2-speaker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.03502" label="2104.03502">
        <attvalues>
          <attvalue for="0" value="Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings" />
          <attvalue for="1" value="  Emotion recognition datasets are relatively small, making the use of the more&#10;sophisticated deep learning approaches challenging. In this work, we propose a&#10;transfer learning method for speech emotion recognition where features&#10;extracted from pre-trained wav2vec 2.0 models are modeled using simple neural&#10;networks. We propose to combine the output of several layers from the&#10;pre-trained model using trainable weights which are learned jointly with the&#10;downstream model. Further, we compare performance using two different wav2vec&#10;2.0 models, with and without finetuning for speech recognition. We evaluate our&#10;proposed approaches on two standard emotion databases IEMOCAP and RAVDESS,&#10;showing superior performance compared to results in the literature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.03381" label="2109.03381">
        <attvalues>
          <attvalue for="0" value="Self-supervised Contrastive Cross-Modality Representation Learning for&#10;  Spoken Question Answering" />
          <attvalue for="1" value="  Spoken question answering (SQA) requires fine-grained understanding of both&#10;spoken documents and questions for the optimal answer prediction. In this&#10;paper, we propose novel training schemes for spoken question answering with a&#10;self-supervised training stage and a contrastive representation learning stage.&#10;In the self-supervised stage, we propose three auxiliary self-supervised tasks,&#10;including utterance restoration, utterance insertion, and question&#10;discrimination, and jointly train the model to capture consistency and&#10;coherence among speech documents without any additional data or annotations. We&#10;then propose to learn noise-invariant utterance representations in a&#10;contrastive objective by adopting multiple augmentation strategies, including&#10;span deletion and span substitution. Besides, we design a Temporal-Alignment&#10;attention to semantically align the speech-text clues in the learned common&#10;space and benefit the SQA tasks. By this means, the training schemes can more&#10;effectively guide the generation model to predict more proper answers.&#10;Experimental results show that our model achieves state-of-the-art results on&#10;three SQA benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.09883" label="2204.09883">
        <attvalues>
          <attvalue for="0" value="Layer-wise Fast Adaptation for End-to-End Multi-Accent Speech&#10;  Recognition" />
          <attvalue for="1" value="  Accent variability has posed a huge challenge to automatic speech&#10;recognition~(ASR) modeling. Although one-hot accent vector based adaptation&#10;systems are commonly used, they require prior knowledge about the target accent&#10;and cannot handle unseen accents. Furthermore, simply concatenating accent&#10;embeddings does not make good use of accent knowledge, which has limited&#10;improvements. In this work, we aim to tackle these problems with a novel&#10;layer-wise adaptation structure injected into the E2E ASR model encoder. The&#10;adapter layer encodes an arbitrary accent in the accent space and assists the&#10;ASR model in recognizing accented speech. Given an utterance, the adaptation&#10;structure extracts the corresponding accent information and transforms the&#10;input acoustic feature into an accent-related feature through the linear&#10;combination of all accent bases. We further explore the injection position of&#10;the adaptation layer, the number of accent bases, and different types of accent&#10;bases to achieve better accent adaptation. Experimental results show that the&#10;proposed adaptation structure brings 12\% and 10\% relative word error&#10;rate~(WER) reduction on the AESRC2020 accent dataset and the Librispeech&#10;dataset, respectively, compared to the baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.02780" label="2212.02780">
        <attvalues>
          <attvalue for="0" value="Parameter Efficient Transfer Learning for Various Speech Processing&#10;  Tasks" />
          <attvalue for="1" value="  Fine-tuning of self-supervised models is a powerful transfer learning method&#10;in a variety of fields, including speech processing, since it can utilize&#10;generic feature representations obtained from large amounts of unlabeled data.&#10;Fine-tuning, however, requires a new parameter set for each downstream task,&#10;which is parameter inefficient. Adapter architecture is proposed to partially&#10;solve this issue by inserting lightweight learnable modules into a frozen&#10;pre-trained model. However, existing adapter architectures fail to adaptively&#10;leverage low- to high-level features stored in different layers, which is&#10;necessary for solving various kinds of speech processing tasks. Thus, we&#10;propose a new adapter architecture to acquire feature representations more&#10;flexibly for various speech tasks. In experiments, we applied this adapter to&#10;WavLM on four speech tasks. It performed on par or better than naive&#10;fine-tuning, with only 11% of learnable parameters. It also outperformed an&#10;existing adapter architecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.10750" label="2009.10750">
        <attvalues>
          <attvalue for="0" value="Lifelong Learning Dialogue Systems: Chatbots that Self-Learn On the Job" />
          <attvalue for="1" value="  Dialogue systems, also called chatbots, are now used in a wide range of&#10;applications. However, they still have some major weaknesses. One key weakness&#10;is that they are typically trained from manually-labeled data and/or written&#10;with handcrafted rules, and their knowledge bases (KBs) are also compiled by&#10;human experts. Due to the huge amount of manual effort involved, they are&#10;difficult to scale and also tend to produce many errors ought to their limited&#10;ability to understand natural language and the limited knowledge in their KBs.&#10;Thus, the level of user satisfactory is often low. In this paper, we propose to&#10;dramatically improve this situation by endowing the system the ability to&#10;continually learn (1) new world knowledge, (2) new language expressions to&#10;ground them to actions, and (3) new conversational skills, during conversation&#10;or &quot;on the job&quot; by themselves so that as the systems chat more and more with&#10;users, they become more and more knowledgeable and are better and better able&#10;to understand diverse natural language expressions and improve their&#10;conversational skills. A key approach to achieving these is to exploit the&#10;multi-user environment of such systems to self-learn through interactions with&#10;users via verb and non-verb means. The paper discusses not only key challenges&#10;and promising directions to learn from users during conversation but also how&#10;to ensure the correctness of the learned knowledge.&#10;" />
          <attvalue for="2" value="&#10;Building dialogue systems or conversational agents capable of conversing with humans in natural language (NL) and understanding human NL instructions is a long-standing goal of AI \cite{winograd1972understanding}. These agents, also called chatbots, have become the front runner of AI advancement due to wide-spread applications such as assisting customers in buying products, booking flight tickets, reducing stress, and executing actions like controlling house appliances and reporting weather information. Because of the proliferation of Internet of Things (IoT) with NL interfaces, the importance of these agents have become ubiquitous in recent times. &#10;&#10;Conversational agents can be broadly categorized into two main types: (1) Chit-chat systems \cite{shang2015neural,sordoni2015neural,li2016deep,serban2016building,serban2017hierarchical} designed to engage users and provide mental support by conducting chit-chat type of conversation in wide range of topics without having a specific goal to complete. (2) Task-oriented chatbots \cite{raux2005let,williams2007partially,wen2017network} designed to assist users to complete tasks based on users' requests, e.g., providing the requested information and taking actions. Most of the popular personal assistants such as Amazon Alexa, Apple Siri, Google Home, and Microsoft Cortana, are task-oriented bots. They are primarily designed as Natural Language Interaction (NLI) systems that take human NL instructions (commands) and translate them into some actions to be executed by the underlying application. Question-answering (QA) and conversational recommendation systems also fall into this category.&#10;&#10;Before deep learning became popular, chatbots were developed mainly using the markup language AIML or handcrafted conversation generation rules. With the advent of deep learning, the trend has shifted toward end-to-end conversation modeling \cite{vinyals2015neural,xing2017topic,wen2017network}. However, despite the fact that chatbots are widely used, they still have some serious weaknesses: (1) A great deal of manual effort is needed to label training data, write rules and compile knowledge bases (KBs). No matter how much data is collected and used to train a chatbot, it is hard to cover all possible variations of natural language. Thus, when deployed in practice, a well-trained chatbot often performs poorly. (2) The pre-compiled KBs cannot cover the rich knowledge needed in practice. &#10;&#10;This paper argues that a truly intelligent chatbot should not be limited by its offline-trained model or pre-compiled KB. It should learn continuously on the job, i.e., after model deployment and during conversing or interacting with the (human) end users and thereby, improve its capability over time in a self-supervised manner~\cite{ChenAndLiubook2018,liu2020learning}. Thus, this paper proposes the new paradigm, called \underline{L}ifelong \underline{IN}teractive learning in \underline{C}onversation (LINC). LINC needs a new definition because traditional lifelong learning is for offline learning of a sequence of tasks with given tasks and given labeled data~\cite{ChenAndLiubook2018}. However, learning during conversation is like human on-the-job learning where the system has to discover its own tasks and also the training data. This paper focuses on three continuous learning capabilities of chatbots: (1) learning factual knowledge in open-ended and information-seeking conversations, &#10;(2) learning to ground new NL commands (language expressions),&#10;and (3) learning new conversational skills from users. Some preliminary work has been done on (1) and (2) in~\cite{mazumder2018towards,mazumder2019building,mazumder2019lifelong}.&#10;&#10;The key idea for solving the LINC problem is to exploit the wisdom of the crowd in the multi-user environment in which almost all chatbots work to learn new knowledge by asking or interacting with the current user and/or the other users to enable the chatbot to learn quickly and effectively. &#10;This powerful approach, however, also comes with a major shortcoming. The knowledge learned from end-users can be erroneous and some users may even purposely fool the system by providing wrong information or knowledge. We will discuss how to solve this problem to ensure the credibility or trustworthiness of the learned knowledge from end users. Note, we use the terms: chatbot, bot, agent, NLI systems, dialogue systems interchangeably in the rest of the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Continual Learning Methods, Computer Science, Machine Learning, Dialogue System Limitations, Linguistics, Cognitive Science, Natural Language Understanding, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1910.14084" label="1910.14084">
        <attvalues>
          <attvalue for="0" value="Building an Application Independent Natural Language Interface" />
          <attvalue for="1" value="  Traditional approaches to building natural language (NL) interfaces typically&#10;use a semantic parser to parse the user command and convert it to a logical&#10;form, which is then translated to an executable action in an application.&#10;However, it is still challenging for a semantic parser to correctly parse&#10;natural language. For a different domain, the parser may need to be retrained&#10;or tuned, and a new translator also needs to be written to convert the logical&#10;forms to executable actions. In this work, we propose a novel and application&#10;independent approach to building NL interfaces that does not need a semantic&#10;parser or a translator. It is based on natural language to natural language&#10;matching and learning, where the representation of each action and each user&#10;command are both in natural language. To perform a user intended action, the&#10;system only needs to match the user command with the correct action&#10;representation, and then execute the corresponding action. The system also&#10;interactively learns new (paraphrased) commands for actions to expand the&#10;action representations over time. Our experimental results show the&#10;effectiveness of the proposed approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.13295" label="1907.13295">
        <attvalues>
          <attvalue for="0" value="Lifelong and Interactive Learning of Factual Knowledge in Dialogues" />
          <attvalue for="1" value="  Dialogue systems are increasingly using knowledge bases (KBs) storing&#10;real-world facts to help generate quality responses. However, as the KBs are&#10;inherently incomplete and remain fixed during conversation, it limits dialogue&#10;systems' ability to answer questions and to handle questions involving entities&#10;or relations that are not in the KB. In this paper, we make an attempt to&#10;propose an engine for Continuous and Interactive Learning of Knowledge (CILK)&#10;for dialogue systems to give them the ability to continuously and interactively&#10;learn and infer new knowledge during conversations. With more knowledge&#10;accumulated over time, they will be able to learn better and answer more&#10;questions. Our empirical evaluation shows that CILK is promising.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;To the best of our knowledge, no existing system can perform the proposed task. We reported a priliminary research in~\cite{mazumder2018towards}. &#10;&#10;CILK is related to interactive language learning \cite{wang2016learning,wang2017naturalizing}, which is mainly about language grounding, not about knowledge learning. \cite{li2016dialogue,li2016learning} and \cite{zhang2017listen} train chatbots using human teachers who can ask and answer the chatbot questions. \cite{Otsuka2017Lexical},~\cite{Otsuka2013Generating},~\cite{Ono2016Toward} and~\cite{Komatani2018Question} allow a system to ask the user whether its prediction of category of a term is correct or not. Compared to these works, CILK performs interactive knowledge learning and inference (over existing and acquired knowledge) while conversing with users after the dialogue system has been deployed (i.e., learning on the job~\cite{ChenAndLiubook2018}) without any teacher supervision or help. &#10;&#10;NELL \cite{mitchell2015never} updates its KB using facts extracted from the Web (complementary to our work). We do not do Web fact extraction. &#10;&#10;KB completion (KBC) has been studied in recent years~\cite{lao2011random,bordes2011learning,bordes2015large,mazumder2017context}. But they mainly handle facts with known entities and relations. \cite{neelakantan2015compositional} work on fixed unknown relations with known embeddings, but does not allow unknown entities. \cite{xiong2018one} also deal with queries involving unknown relations, but known entities in the KB. \cite{shi2017open} handles unknown entities by exploiting an external text corpus. None of the KBC methods perform conversational knowledge learning like CILK.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2210.12619" label="2210.12619">
        <attvalues>
          <attvalue for="0" value="Conformal Predictor for Improving Zero-shot Text Classification&#10;  Efficiency" />
          <attvalue for="1" value="  Pre-trained language models (PLMs) have been shown effective for zero-shot&#10;(0shot) text classification. 0shot models based on natural language inference&#10;(NLI) and next sentence prediction (NSP) employ cross-encoder architecture and&#10;infer by making a forward pass through the model for each label-text pair&#10;separately. This increases the computational cost to make inferences linearly&#10;in the number of labels. In this work, we improve the efficiency of such&#10;cross-encoder-based 0shot models by restricting the number of likely labels&#10;using another fast base classifier-based conformal predictor (CP) calibrated on&#10;samples labeled by the 0shot model. Since a CP generates prediction sets with&#10;coverage guarantees, it reduces the number of target labels without excluding&#10;the most probable label based on the 0shot model. We experiment with three&#10;intent and two topic classification datasets. With a suitable CP for each&#10;dataset, we reduce the average inference time for NLI- and NSP-based models by&#10;25.6% and 22.2% respectively, without dropping performance below the predefined&#10;error rate of 1%.&#10;" />
          <attvalue for="2" value="&#10;Zero-shot (0shot) text classification is an important NLP problem with many real-world applications. The earliest approaches for {0shot} text classification use a similarity score between text and labels mapped to common embedding space~\cite{10.5555/1620163.1620201,10.5555/1625275.1625535,Chen2015DatalessTC,li-etal-2016-joint,Sappadla2016UsingSS,xia-etal-2018-zero}. These models calculate text and label embeddings independently and make only one forward pass over the text resulting in a minimal increase in the computation. Later approaches explicitly incorporate label information when processing the text, e.g., \cite{Yogatama2017GenerativeAD} uses generative modeling and generates text given label embedding, and \cite{rios-kavuluru-2018-shot} uses label embedding based attention over text, both requiring multiple passes over the text and increasing the computational cost.&#10;&#10;Most recently, NLI- \cite{condoravdi-etal-2003-entailment,williams-etal-2018-broad,yin-etal-2019-benchmarking} and NSP- \cite{ma-etal-2021-issues} based {0shot} text classification formulations have been proposed. NLI and NSP make inferences by defining a representative hypothesis sentence for each label and producing a score corresponding to every pair of input text and hypothesis. To compute the score, they employ a cross-encoder architecture that is full self-attention over the text and hypothesis sentences, which requires recomputing the encoding for text and each hypothesis separately. It increases the computational cost to make inferences linearly in the number of target labels. &#10;&#10;NLI and NSP use large transformer-based PLMs \cite{devlin-etal-2019-bert,Liu2019RoBERTaAR,DBLP:journals/corr/abs-1910-13461} and outperform previous non-transformer-based models by a large margin. However, the size of PLMs and the number of target labels drastically reduce the prediction efficiency, increasing the computation and inference time, and may significantly increase the carbon footprint of making predictions \cite{strubell-etal-2019-energy,sustainlp-2020-sustainlp,10.1145/3381831,zhou-etal-2021-hulk}. &#10;&#10;In this work, we focus on the correlation between the number of labels and prediction efficiency and propose to use a conformal predictor (CP) \cite{10.5555/1062391,JMLR:v9:shafer08a} to filter out unlikely labels from the target. &#10;Conformal prediction provides a model-agnostic framework to generate a label set, instead of a single label prediction, within a pre-defined error rate. Consequently, we use a CP, with a small error rate we select, based on another fast base classifier to generate candidate target labels. Candidate labels are then used with the larger NLI/NSP-based {0shot} models to make the final prediction. &#10;&#10;We experiment with three intent classification (SNIPS \cite{Coucke2018SnipsVP}, ATIS \cite{5700816} and HWU64 \cite{Liu2019BenchmarkingNL}) and two topic classification (AG's news and Yahoo! Answers \cite{10.5555/2969239.2969312}) datasets using {0shot} models based on a moderately sized bart-large (NLI) \cite{lewis-etal-2020-bart} and a small bert-base (NSP) PLM. We use four different {base} classifiers, each with different computational complexity, and a small error rate of 1\%. By using the best CP for each dataset, we reduce the average computational time by 25.6\% (22.2\%) and the average number of labels by 41.09\% (43.38\%) for NLI-(NSP-) based models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Classification Methods, Computer Science, Linguistics, Artificial Intelligence, Mathematics, Language Model Efficiency, Computational Cost Reduction" />
        </attvalues>
      </node>
      <node id="1903.05566" label="1903.05566">
        <attvalues>
          <attvalue for="0" value="Benchmarking Natural Language Understanding Services for building&#10;  Conversational Agents" />
          <attvalue for="1" value="  We have recently seen the emergence of several publicly available Natural&#10;Language Understanding (NLU) toolkits, which map user utterances to structured,&#10;but more abstract, Dialogue Act (DA) or Intent specifications, while making&#10;this process accessible to the lay developer. In this paper, we present the&#10;first wide coverage evaluation and comparison of some of the most popular NLU&#10;services, on a large, multi-domain (21 domains) dataset of 25K user utterances&#10;that we have collected and annotated with Intent and Entity Type specifications&#10;and which will be released as part of this submission. The results show that on&#10;Intent classification Watson significantly outperforms the other platforms,&#10;namely, Dialogflow, LUIS and Rasa; though these also perform well.&#10;Interestingly, on Entity Type recognition, Watson performs significantly worse&#10;due to its low Precision. Again, Dialogflow, LUIS and Rasa perform well on this&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.06364" label="2304.06364">
        <attvalues>
          <attvalue for="0" value="AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models" />
          <attvalue for="1" value="  Evaluating the general abilities of foundation models to tackle human-level&#10;tasks is a vital aspect of their development and application in the pursuit of&#10;Artificial General Intelligence (AGI). Traditional benchmarks, which rely on&#10;artificial datasets, may not accurately represent human-level capabilities. In&#10;this paper, we introduce AGIEval, a novel benchmark specifically designed to&#10;assess foundation model in the context of human-centric standardized exams,&#10;such as college entrance exams, law school admission tests, math competitions,&#10;and lawyer qualification tests. We evaluate several state-of-the-art foundation&#10;models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark.&#10;Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math&#10;competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5%&#10;accuracy on the English test of the Chinese national college entrance exam.&#10;This demonstrates the extraordinary performance of contemporary foundation&#10;models. In contrast, we also find that GPT-4 is less proficient in tasks that&#10;require complex reasoning or specific domain knowledge. Our comprehensive&#10;analyses of model capabilities (understanding, knowledge, reasoning, and&#10;calculation) reveal these models' strengths and limitations, providing valuable&#10;insights into future directions for enhancing their general capabilities. By&#10;concentrating on tasks pertinent to human cognition and decision-making, our&#10;benchmark delivers a more meaningful and robust evaluation of foundation&#10;models' performance in real-world scenarios. The data, code, and all model&#10;outputs are released in https://github.com/ruixiangcui/AGIEval.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05169" label="2204.05169">
        <attvalues>
          <attvalue for="0" value="Towards End-to-End Integration of Dialog History for Improved Spoken&#10;  Language Understanding" />
          <attvalue for="1" value="  Dialog history plays an important role in spoken language understanding (SLU)&#10;performance in a dialog system. For end-to-end (E2E) SLU, previous work has&#10;used dialog history in text form, which makes the model dependent on a cascaded&#10;automatic speech recognizer (ASR). This rescinds the benefits of an E2E system&#10;which is intended to be compact and robust to ASR errors. In this paper, we&#10;propose a hierarchical conversation model that is capable of directly using&#10;dialog history in speech form, making it fully E2E. We also distill semantic&#10;knowledge from the available gold conversation transcripts by jointly training&#10;a similar text-based conversation model with an explicit tying of acoustic and&#10;semantic embeddings. We also propose a novel technique that we call DropFrame&#10;to deal with the long training time incurred by adding dialog history in an E2E&#10;manner. On the HarperValleyBank dialog dataset, our E2E history integration&#10;outperforms a history independent baseline by 7.7% absolute F1 score on the&#10;task of dialog action recognition. Our model performs competitively with the&#10;state-of-the-art history based cascaded baseline, but uses 48% fewer&#10;parameters. In the absence of gold transcripts to fine-tune an ASR model, our&#10;model outperforms this baseline by a significant margin of 10% absolute F1&#10;score.&#10;" />
          <attvalue for="2" value="&#10;&#10;Traditionally, spoken dialog systems (SDS) comprise an automatic speech recognition (ASR) component followed by a natural language understanding (NLU) component \cite{tur2002improving, hakkani2006beyond, henderson2012discriminative, huang2019adapting}. In recent times, end-to-end (E2E) spoken language understanding (SLU) has gained popularity for SDS \cite{rongali2020exploring, chung2021splat, morais2021end, denisov2020pretrained}. The reason is that state-of-the-art models for ASR and NLU can be inherently large and cascading them leads to an even larger overall model size, which may make them difficult to deploy, especially for on-device applications. Also, ASR errors can degrade the performance of a cascaded SLU system. In contrast, E2E SLU models are compact and more robust to ASR errors as they process speech directly.&#10;&#10;It is also well known that a dialog system's NLU performance benefits from using the entire dialog history instead of just the utterance to be labeled. Dialog history helps particularly by resolving ambiguities and co-references. This idea has been explored extensively in written/typed dialog systems \cite{bothe2018context,raheja2019dialogue,colombo2020guiding}. Some recent work has explored ways of incorporating dialog history for intent identification in E2E SLU. However, a limitation here is that the features for the dialog history are extracted from a text encoder which makes the model dependent on a cascaded ASR.&#10;&#10;In particular, Tomashenko et al. \cite{tomashenko2020dialogue} use text from the previous system prompt as history. In an extension to this, Ganhotra et al. \cite{ganhotra2021integrating} use the entire conversation as history which is a transcript decoded from an ASR, making it prone to ASR errors and also not fully E2E. ASR performance has also been shown to improve by integrating dialog history \cite{kim2019gated} and intent representations \cite{ray2021listen}. &#10;&#10;In this paper, we propose a conversation model with a full E2E integration of dialog history that is significantly more compact than its cascaded counterpart. Our conversation model is hierarchical, where we have an utterance level speech encoder at the lower level and a conversation encoder at the top level. &#10;&#10;To further improve the performance, we use a teacher-student training framework to distill knowledge from the semantically richer BERT encoder using available transcriptions. We jointly train a BERT based conversation model along with the speech based model with weight sharing at the conversation level between modalities (see Figure \ref{fig:model_over}). Further, speech embeddings are tied with BERT embeddings via Euclidean loss \cite{huang2020leveraging} and a novel use of Contrastive loss \cite{zhang2021cross}.&#10;&#10;Using dialog history in an E2E manner can lead to an increased training time. We propose a technique called DropFrame to deal with this issue.&#10;Similar to dropout \cite{srivastava2013improving}, this technique drops out random speech frames during training. In addition to a reduced training time, this technique also helps to improve SLU performance.&#10;&#10;We perform extensive experiments on the recently released HarperValleyBank corpus \cite{wu2020harpervalleybank} which is a task oriented dialog dataset. We focus on the multi-label classification task of dialog action recognition. When gold transcripts are available for fine-tuning an ASR, our model performs competitively with a state-of-the-art cascaded baseline while using 48\% fewer parameters. In the absence of gold transcripts, our model significantly outperforms this baseline.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="End-to-End Modeling, Computer Science, Linguistics, Spoken Language Understanding, Artificial Intelligence, Dialog History Integration" />
        </attvalues>
      </node>
      <node id="2012.08549" label="2012.08549">
        <attvalues>
          <attvalue for="0" value="Exploring Transfer Learning For End-to-End Spoken Language Understanding" />
          <attvalue for="1" value="  Voice Assistants such as Alexa, Siri, and Google Assistant typically use a&#10;two-stage Spoken Language Understanding pipeline; first, an Automatic Speech&#10;Recognition (ASR) component to process customer speech and generate text&#10;transcriptions, followed by a Natural Language Understanding (NLU) component to&#10;map transcriptions to an actionable hypothesis. An end-to-end (E2E) system that&#10;goes directly from speech to a hypothesis is a more attractive option. These&#10;systems were shown to be smaller, faster, and better optimized. However, they&#10;require massive amounts of end-to-end training data and in addition, don't take&#10;advantage of the already available ASR and NLU training data.&#10;  In this work, we propose an E2E system that is designed to jointly train on&#10;multiple speech-to-text tasks, such as ASR (speech-transcription) and SLU&#10;(speech-hypothesis), and text-to-text tasks, such as NLU (text-hypothesis). We&#10;call this the Audio-Text All-Task (AT-AT) Model and we show that it beats the&#10;performance of E2E models trained on individual tasks, especially ones trained&#10;on limited data. We show this result on an internal music dataset and two&#10;public datasets, FluentSpeech and SNIPS Audio, where we achieve&#10;state-of-the-art results. Since our model can process both speech and text&#10;input sequences and learn to predict a target sequence, it also allows us to do&#10;zero-shot E2E SLU by training on only text-hypothesis data (without any speech)&#10;from a new domain. We evaluate this ability of our model on the Facebook TOP&#10;dataset and set a new benchmark for zeroshot E2E performance. We will soon&#10;release the audio data collected for the TOP dataset for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.06280" label="1805.06280">
        <attvalues>
          <attvalue for="0" value="A Context-based Approach for Dialogue Act Recognition using Simple&#10;  Recurrent Neural Networks" />
          <attvalue for="1" value="  Dialogue act recognition is an important part of natural language&#10;understanding. We investigate the way dialogue act corpora are annotated and&#10;the learning approaches used so far. We find that the dialogue act is&#10;context-sensitive within the conversation for most of the classes.&#10;Nevertheless, previous models of dialogue act classification work on the&#10;utterance-level and only very few consider context. We propose a novel&#10;context-based learning method to classify dialogue acts using a character-level&#10;language model utterance representation, and we notice significant improvement.&#10;We evaluate this method on the Switchboard Dialogue Act corpus, and our results&#10;show that the consideration of the preceding utterances as a context of the&#10;current utterance improves dialogue act detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.02594" label="1904.02594">
        <attvalues>
          <attvalue for="0" value="Dialogue Act Classification with Context-Aware Self-Attention" />
          <attvalue for="1" value="  Recent work in Dialogue Act classification has treated the task as a sequence&#10;labeling problem using hierarchical deep neural networks. We build on this&#10;prior work by leveraging the effectiveness of a context-aware self-attention&#10;mechanism coupled with a hierarchical recurrent neural network. We conduct&#10;extensive evaluations on standard Dialogue Act classification datasets and show&#10;significant improvement over state-of-the-art results on the Switchboard&#10;Dialogue Act (SwDA) Corpus. We also investigate the impact of different&#10;utterance-level representation learning methods and show that our method is&#10;effective at capturing utterance-level semantic text representations while&#10;maintaining high accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.08801" label="2002.08801">
        <attvalues>
          <attvalue for="0" value="Guiding attention in Sequence-to-sequence models for Dialogue Act&#10;  prediction" />
          <attvalue for="1" value="  The task of predicting dialog acts (DA) based on conversational dialog is a&#10;key component in the development of conversational agents. Accurately&#10;predicting DAs requires a precise modeling of both the conversation and the&#10;global tag dependencies. We leverage seq2seq approaches widely adopted in&#10;Neural Machine Translation (NMT) to improve the modelling of tag sequentiality.&#10;Seq2seq models are known to learn complex global dependencies while currently&#10;proposed approaches using linear conditional random fields (CRF) only model&#10;local tag dependencies. In this work, we introduce a seq2seq model tailored for&#10;DA classification using: a hierarchical encoder, a novel guided attention&#10;mechanism and beam search applied to both training and inference. Compared to&#10;the state of the art our model does not require handcrafted features and is&#10;trained end-to-end. Furthermore, the proposed approach achieves an unmatched&#10;accuracy score of 85% on SwDA, and state-of-the-art accuracy score of 91.6% on&#10;MRDA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.08405" label="2108.08405">
        <attvalues>
          <attvalue for="0" value="Integrating Dialog History into End-to-End Spoken Language Understanding&#10;  Systems" />
          <attvalue for="1" value="  End-to-end spoken language understanding (SLU) systems that process&#10;human-human or human-computer interactions are often context independent and&#10;process each turn of a conversation independently. Spoken conversations on the&#10;other hand, are very much context dependent, and dialog history contains useful&#10;information that can improve the processing of each conversational turn. In&#10;this paper, we investigate the importance of dialog history and how it can be&#10;effectively integrated into end-to-end SLU systems. While processing a spoken&#10;utterance, our proposed RNN transducer (RNN-T) based SLU model has access to&#10;its dialog history in the form of decoded transcripts and SLU labels of&#10;previous turns. We encode the dialog history as BERT embeddings, and use them&#10;as an additional input to the SLU model along with the speech features for the&#10;current utterance. We evaluate our approach on a recently released spoken&#10;dialog data set, the HarperValleyBank corpus. We observe significant&#10;improvements: 8% for dialog action and 30% for caller intent recognition tasks,&#10;in comparison to a competitive context independent end-to-end baseline system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.13929" label="2010.13929">
        <attvalues>
          <attvalue for="0" value="HarperValleyBank: A Domain-Specific Spoken Dialog Corpus" />
          <attvalue for="1" value="  We introduce HarperValleyBank, a free, public domain spoken dialog corpus.&#10;The data simulate simple consumer banking interactions, containing about 23&#10;hours of audio from 1,446 human-human conversations between 59 unique speakers.&#10;We selected intents and utterance templates to allow realistic variation while&#10;controlling overall task complexity and limiting vocabulary size to about 700&#10;unique words. We provide audio data along with transcripts and annotations for&#10;speaker identity, caller intent, dialog actions, and emotional valence. The&#10;data size and domain specificity makes for quick transcription experiments with&#10;modern end-to-end neural approaches. Further, we provide baselines for&#10;representation learning, adapting recent work to embed waveforms for downstream&#10;prediction tasks. Our experiments show that tasks using our annotations are&#10;sensitive to both the model choice and corpus size.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.17611" label="2310.17611">
        <attvalues>
          <attvalue for="0" value="Uncovering Meanings of Embeddings via Partial Orthogonality" />
          <attvalue for="1" value="  Machine learning tools often rely on embedding text as vectors of real&#10;numbers. In this paper, we study how the semantic structure of language is&#10;encoded in the algebraic structure of such embeddings. Specifically, we look at&#10;a notion of ``semantic independence'' capturing the idea that, e.g.,&#10;``eggplant'' and ``tomato'' are independent given ``vegetable''. Although such&#10;examples are intuitive, it is difficult to formalize such a notion of semantic&#10;independence. The key observation here is that any sensible formalization&#10;should obey a set of so-called independence axioms, and thus any algebraic&#10;encoding of this structure should also obey these axioms. This leads us&#10;naturally to use partial orthogonality as the relevant algebraic structure. We&#10;develop theory and methods that allow us to demonstrate that partial&#10;orthogonality does indeed capture semantic independence. Complementary to this,&#10;we also introduce the concept of independence preserving embeddings where&#10;embeddings preserve the conditional independence structures of a distribution,&#10;and we prove the existence of such embeddings and approximations to them.&#10;" />
          <attvalue for="2" value="&#10;This paper concerns the question of how semantic meaning is encoded in neural embeddings, such as those produced by \cite{radford2021learning}.&#10;There is strong empirical evidence that these embeddings---vectors of real numbers---capture the semantic meaning of the underlying text. For example, classical results show that word embeddings can be used for analogical reasoning \cite[e.g.,][]{mikolov2013efficient, pennington2014glove}, and such embeddings are the backbone of modern generative AI systems \cite[e.g.,][]{ramesh2022hierarchical, bubeck2023sparks, saharia2022photorealistic, devlin2018bert}.&#10;The high-level question we're interested in is: How is the semantic structure of text encoded in the algebraic structure of embeddings?&#10;In this paper, we provide evidence that the concept of partial orthogonality plays a key role.&#10;&#10;The first step is to identify the semantic structure of interest.&#10;Intuitively, words or phrases possess a notion of semantic independence, which does not have to be statistical in nature.&#10;For example, the word ``eggplant'' seems more similar to ``tomato'' than to ``ennui''. Yet, if we were to ``condition&quot; on the common property of ``vegetable'', then ``eggplant'' and ``tomato'' should be ``independent&quot;.&#10;And, if we condition on both ``vegetable'' and ``purple'', then ``eggplant&quot; may be ``independent'' of all other words.&#10;However, it is difficult to formalize what is meant by ``independent&quot; and ``condition on&quot; in these informal statements. Accordingly, it is hard to establish a formal definition of semantic independence, and thus it is challenging to explore how this structure might be encoded algebraically!&#10;&#10;The key observation in this paper is to recall that most reasonable concepts of ``independence'' adhere to a common set of axioms similar to those defining probabilistic conditional independence. Formally, this abstract idea is captured by the axioms of the so-called independence models \cite{lauritzen1996graphical}. &#10;Thus, if semantic independence is encoded algebraically, it should be encoded as an algebraic structure that respects these axioms. &#10;In this paper, we use a natural candidate independence model in vector spaces known as partial orthogonality \cite{lauritzen1996graphical,amini2022lattice}.&#10;Here, for two vectors $v_a$ and $v_b$ and a conditioning set of vectors $v_C$, partial orthogonality takes $v_a$ independent $v_b$ given $v_C$ if the residuals of $v_a$ and $v_b$ are orthogonal after projecting onto the span of $v_C$. &#10;We discover that this particular tool is indeed valuable for understanding CLIP embeddings. For instance, \ref{fig:eggplant-demo} shows that after projecting onto the linear subspace spanned by CLIP embeddings of ``purple'' and ``vegetable'', the residual of embedding ``eggplant'' has on average low cosine similarity with the residuals of random test embeddings, which also matches our intuitive understanding of the word. &#10;&#10;Since partial orthogonality is an independence model, &#10;we can go one step further to define Markov boundaries for embeddings as well. Drawing inspiration from graphical models, it is reasonable to expect that the Markov boundary of any target embedding should constitute a minimal collection of embeddings that encompasses valuable information regarding the target. Unlike classical applications of partial orthogonality in regression and Gaussian models, however,&#10;the geometry of embeddings presents several subtle technical challenges to&#10;directly adopting the usual notion of Markov boundary. First, the intersectionaxiom &#10;never holds for practical embeddings,&#10;which makes the standard Markov boundary non-unique. More importantly, practical embeddings could potentially incorporate distortion, noise and undergo phenomena resembling superposition \cite{elhage2022superposition}. Therefore, in this paper, we introduce generalized Markov boundaries for studying the structure of text embeddings. &#10;&#10;\looseness=-1&#10;Contributions&#10;Specifically, we make the following contributions:&#10;\begin{enumerate}&#10; \item We adapt ideas from graphical independence models to specify the structure that should be satisfied by semantic independence. We discover that partial orthogonality in the embedding space offers a natural way of encoding semantic independence structure (\ref{sec:prelim}). &#10;&#10;\item We study the semantic structure of partial orthogonality via Markov boundaries. Due to the unique characteristics of embeddings and noise in learning, exact orthogonality is unlikely to hold. So, we give a distributional relaxation of the Markov boundary and use this to provide a practical algorithm for finding generalized Markov boundaries and measuring the semantic independence induced by generalized Markov boundaries (\ref{sec:find-mb}).&#10;&#10;\item We introduce the concept of independence preserving embeddings, which studies how embeddings can be used to maintain the independence structure of distributions. This holds its own intrigue for further research (\ref{sec:ipe}).&#10;&#10;\item Finally, we design and conduct experimental evaluations on CLIP text embeddings, finding that the partial orthogonality structure and generalized Markov boundary encode semantic structure (\ref{sec:exp}).&#10;&#10;\end{enumerate}&#10;&#10;Throughout, we use CLIP text embeddings as a running example, though the method and theory presented can be applied more broadly. &#10;&#10;Related work&#10;There are many papers \cite[e.g.,][]{arora2016latent, gittens2017skip, allen2019analogies, ethayarajh2019understanding, trager2023linear, perera2023prompt, leemann2023post, merullo2023language, wang2023concept} connecting semantic meanings and algebraic structures of popular embeddings like &#10;CLIP \cite{radford2021learning}, Glove \cite{pennington2014glove} and word2vec \cite{mikolov2013efficient}. Simple arithmetic on these embeddings reveals that they carry semantic meanings. The most popular arithmetic operation is called linear analogy \cite{ethayarajh2019understanding}. There are several papers trying to understand the reasoning behind this phenomenon. \Citet{arora2016latent} explains this by proposing the latent variable model but it requires the word vectors to be uniformly distributed in the embedding space which generally is not true in practice \cite{mimno2017strange}. Alternatively, \cite{gittens2017skip, allen2019analogies} adopts the paraphrase model that also does not fit practice. \cite{ethayarajh2019understanding}, on the other hand, studies the geometry of embeddings that decomposes the shifted pointwise mutual information (PMI) matrix. \Citet{trager2023linear, perera2023prompt} decomposes embeddings into combinations of a smaller set of vectors that are more interpretable. On the other hand, similar to using vector orthogonality to represent (conditional) independence, kernel mean embeddings \cite{muandet2017kernel} are Hilbert space embeddings of distributions that can also be used to represent conditional independences \cite{song2009hilbert, song2013kernel}. It is a popular method for machine learning, and causal inference \cite{gretton2005measuring, mooij2009regression, greenfeld2020robust}. But unlike independence preserving embeddings, kernel mean embeddings use the kernel and do not explicitly construct finite-dimensional vector representations.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Language Embeddings, Linguistics, Mathematics, Semantic Structure, Natural Language Processing, Algebraic Encoding" />
        </attvalues>
      </node>
      <node id="2206.05829" label="2206.05829">
        <attvalues>
          <attvalue for="0" value="A non-graphical representation of conditional independence via the&#10;  neighbourhood lattice" />
          <attvalue for="1" value="  We introduce and study the neighbourhood lattice decomposition of a&#10;distribution, which is a compact, non-graphical representation of conditional&#10;independence that is valid in the absence of a faithful graphical&#10;representation. The idea is to view the set of neighbourhoods of a variable as&#10;a subset lattice, and partition this lattice into convex sublattices, each of&#10;which directly encodes a collection of conditional independence relations. We&#10;show that this decomposition exists in any compositional graphoid and can be&#10;computed efficiently and consistently in high-dimensions. {In particular, this&#10;gives a way to encode all of independence relations implied by a distribution&#10;that satisfies the composition axiom, which is strictly weaker than the&#10;faithfulness assumption that is typically assumed by graphical approaches.} We&#10;also discuss various special cases such as graphical models and projection&#10;lattices, each of which has intuitive interpretations. Along the way, we see&#10;how this problem is closely related to neighbourhood regression, which has been&#10;extensively studied in the context of graphical models and structural&#10;equations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.10652" label="2209.10652">
        <attvalues>
          <attvalue for="0" value="Toy Models of Superposition" />
          <attvalue for="1" value="  Neural networks often pack many unrelated concepts into a single neuron - a&#10;puzzling phenomenon known as 'polysemanticity' which makes interpretability&#10;much more challenging. This paper provides a toy model where polysemanticity&#10;can be fully understood, arising as a result of models storing additional&#10;sparse features in &quot;superposition.&quot; We demonstrate the existence of a phase&#10;change, a surprising connection to the geometry of uniform polytopes, and&#10;evidence of a link to adversarial examples. We also discuss potential&#10;implications for mechanistic interpretability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1605.09522" label="1605.09522">
        <attvalues>
          <attvalue for="0" value="Kernel Mean Embedding of Distributions: A Review and Beyond" />
          <attvalue for="1" value="  A Hilbert space embedding of a distribution---in short, a kernel mean&#10;embedding---has recently emerged as a powerful tool for machine learning and&#10;inference. The basic idea behind this framework is to map distributions into a&#10;reproducing kernel Hilbert space (RKHS) in which the whole arsenal of kernel&#10;methods can be extended to probability measures. It can be viewed as a&#10;generalization of the original &quot;feature map&quot; common to support vector machines&#10;(SVMs) and other kernel methods. While initially closely associated with the&#10;latter, it has meanwhile found application in fields ranging from kernel&#10;machines and probabilistic modeling to statistical inference, causal discovery,&#10;and deep learning. The goal of this survey is to give a comprehensive review of&#10;existing work and recent advances in this research area, and to discuss the&#10;most challenging issues and open problems that could lead to new research&#10;directions. The survey begins with a brief introduction to the RKHS and&#10;positive definite kernels which forms the backbone of this survey, followed by&#10;a thorough discussion of the Hilbert space embedding of marginal distributions,&#10;theoretical guarantees, and a review of its applications. The embedding of&#10;distributions enables us to apply RKHS methods to probability measures which&#10;prompts a wide range of applications such as kernel two-sample testing,&#10;independent testing, and learning on distributional data. Next, we discuss the&#10;Hilbert space embedding for conditional distributions, give theoretical&#10;insights, and review some applications. The conditional mean embedding enables&#10;us to perform sum, product, and Bayes' rules---which are ubiquitous in&#10;graphical model, probabilistic inference, and reinforcement learning---in a&#10;non-parametric way. We then discuss relationships between this framework and&#10;other related areas. Lastly, we give some suggestions on future research&#10;directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.14337" label="2402.14337">
        <attvalues>
          <attvalue for="0" value="AURA: Natural Language Reasoning for Aleatoric Uncertainty in Rationales" />
          <attvalue for="1" value="  Rationales behind answers not only explain model decisions but boost language&#10;models to reason well on complex reasoning tasks. However, obtaining impeccable&#10;rationales is often impossible. Besides, it is non-trivial to estimate the&#10;degree to which the rationales are faithful enough to encourage model&#10;performance. Thus, such reasoning tasks often compel models to output correct&#10;answers under undesirable rationales and are sub-optimal compared to what the&#10;models are fully capable of. In this work, we propose how to deal with&#10;imperfect rationales causing aleatoric uncertainty. We first define the&#10;ambiguous rationales with entropy scores of given rationales, using model prior&#10;beliefs as informativeness. We then guide models to select one of two different&#10;reasoning models according to the ambiguity of rationales. We empirically argue&#10;that our proposed method produces robust performance superiority against the&#10;adversarial quality of rationales and low-resource settings.&#10;" />
          <attvalue for="2" value="&#10;&#10;\usepgfplotslibrary{units}&#10;&#10;Language models (LMs) have achieved significant progress on sophisticated reasoning tasks requiring commonsense knowledge or selecting the best answer among tricky multiple choicese~\cite{Wei0SBIXCLZ22, 0010LLWWBCH22}.&#10;Recent advances have enabled LMs to generate explicit free-text rationales and use them to guide task predictions with better search space for candidate answers without injecting additional knowledge~\cite{ZelikmanWMG22, KojimaGRMI22}.&#10;&#10;However, it is impossible to consistently obtain perfect rationales from models or even from humans.~\cite{JungQWBB0C22, WangCIC023, ChenB0J0S23}.&#10;Despite the advantage of rationales that can explain model decisions, the human utility of those generated by large-scale LMs is far from satisfactory. \cite{JoshiLRCTNW0023} has recently reported that only 20\% of them are useful for humans to gain additional information to answer questions. Human annotation may enhance the quality of rationales but is often costly yet does not guarantee perfect conditions, either.&#10;&#10;Above all, learning the patterns of rationales that are generated based on a tremendous number of different statements is almost improbable. By nature, rationales carry varying normative concepts, corresponding to various kinds of reasoning to justify thinking or action in a certain way~\cite{wedgwood2017value}.&#10;The diversity leads models to face the aleatoric uncertainty (i.e., data uncertainty) that identifies ambiguity or randomness inherent in the observation, causing the difficulty of learning the rationales, as shown in Figure~\ref{figure:mismatch}. The Figure~\ref{figure:mismatch_training_ratios} shows that the performance does not significantly improve with additional data due to the noisiness in the dataset even with human rationales. &#10;&#10;Our study aims to investigate how the ambiguous rationales play in model performances of natural language reasoning.&#10;We explore how differently human-annotated and machine-generated rationales impact model reasoning performances. We then introduce a method of simply dealing with unclear machine-generated rationales, proposing a suitable two-system reasoning mechanism depending on the level of ambiguity. Because the quantity of the data does not improve the quality of the data, we target to work with the suboptimal quality of the rationales for faithful enough reasoning rather than improving the quality of the rationales itself. &#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Rationale Uncertainty, Machine Learning, Computer Science, Language Model Reasoning, Model Performance Optimization, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2205.11822" label="2205.11822">
        <attvalues>
          <attvalue for="0" value="Maieutic Prompting: Logically Consistent Reasoning with Recursive&#10;  Explanations" />
          <attvalue for="1" value="  Despite their impressive capabilities, large pre-trained language models&#10;(LMs) struggle with consistent reasoning; recently, prompting LMs to generate&#10;explanations that self-guide the inference has emerged as a promising direction&#10;to amend this. However, these approaches are fundamentally bounded by the&#10;correctness of explanations, which themselves are often noisy and inconsistent.&#10;In this work, we develop Maieutic Prompting, which infers a correct answer to a&#10;question even from the noisy and inconsistent generations of LM. Maieutic&#10;Prompting induces a tree of explanations abductively (e.g. X is true, because&#10;...) and recursively, then frames the inference as a satisfiability problem&#10;over these explanations and their logical relations. We test Maieutic Prompting&#10;for true/false QA on three challenging benchmarks that require complex&#10;commonsense reasoning. Maieutic Prompting achieves up to 20% better accuracy&#10;than state-of-the-art prompting methods, and as a fully unsupervised approach,&#10;performs competitively with supervised models. We also show that Maieutic&#10;Prompting improves robustness in inference while providing interpretable&#10;rationales.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.07095" label="2305.07095">
        <attvalues>
          <attvalue for="0" value="Are Machine Rationales (Not) Useful to Humans? Measuring and Improving&#10;  Human Utility of Free-Text Rationales" />
          <attvalue for="1" value="  Among the remarkable emergent capabilities of large language models (LMs) is&#10;free-text rationalization; beyond a certain scale, large LMs are capable of&#10;generating seemingly useful rationalizations, which in turn, can dramatically&#10;enhance their performances on leaderboards. This phenomenon raises a question:&#10;can machine generated rationales also be useful for humans, especially when lay&#10;humans try to answer questions based on those machine rationales? We observe&#10;that human utility of existing rationales is far from satisfactory, and&#10;expensive to estimate with human studies. Existing metrics like task&#10;performance of the LM generating the rationales, or similarity between&#10;generated and gold rationales are not good indicators of their human utility.&#10;While we observe that certain properties of rationales like conciseness and&#10;novelty are correlated with their human utility, estimating them without human&#10;involvement is challenging. We show that, by estimating a rationale's&#10;helpfulness in answering similar unseen instances, we can measure its human&#10;utility to a better extent. We also translate this finding into an automated&#10;score, GEN-U, that we propose, which can help improve LMs' ability to generate&#10;rationales with better human utility, while maintaining most of its task&#10;performance. Lastly, we release all code and collected data with this project.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.15250" label="2403.15250">
        <attvalues>
          <attvalue for="0" value="Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A&#10;  Multifaceted Statistical Approach" />
          <attvalue for="1" value="  Amidst the rapid evolution of LLMs, the significance of evaluation in&#10;comprehending and propelling these models forward is increasingly paramount.&#10;Evaluations have revealed that factors such as scaling, training types,&#10;architectures and other factors profoundly impact the performance of LLMs.&#10;However, the extent and nature of these impacts continue to be subjects of&#10;debate because most assessments have been restricted to a limited number of&#10;models and data points. Clarifying the effects of these factors on performance&#10;scores can be more effectively achieved through a statistical lens. Our study&#10;embarks on a thorough re-examination of these LLMs, targeting the inadequacies&#10;in current evaluation methods. With the advent of a uniform evaluation&#10;framework, our research leverages an expansive dataset of evaluation results,&#10;introducing a comprehensive statistical methodology. This includes the&#10;application of ANOVA, Tukey HSD tests, GAMM, and clustering technique, offering&#10;a robust and transparent approach to deciphering LLM performance data. Contrary&#10;to prevailing findings, our results challenge assumptions about emergent&#10;abilities and the influence of given training types and architectures in LLMs.&#10;These findings furnish new perspectives on the characteristics, intrinsic&#10;nature, and developmental trajectories of LLMs. By providing straightforward&#10;and reliable methods to scrutinize and reassess LLM performance data, this&#10;study contributes a nuanced perspective on LLM efficiency and potentials.&#10;" />
          <attvalue for="2" value="&#10;&#10;The advent of Large Language Models (LLMs) marks a significant milestone in the evolution of artificial intelligence. These models are revolutionizing the way we interact with technology, offering unprecedented capabilities, and consequently reshaping the AI landscape, prompting new discussions about artificial general intelligence (AGI )(\cite{bubeck2023sparks}; \cite{zhao2023survey}). The advancement of LLMs has been remarkable, yet the foundational elements that govern their operations remain somewhat of a mystery. For instance, central to this puzzle is understanding why LLMs exhibit certain advanced abilities that their smaller counterparts do not \cite{wei2022emergent}. This phenomenon underscores the urgency for thorough research to dissect the underlying factors contributing to these advanced features. &#10;&#10;With the rapid emergence of numerous LLMs, it has become crucial to swiftly and effectively evaluate their performance adopting reliable and standardized approaches. The extraordinary pace at which LLMs are evolving presents challenges in fully grasping their nature, characteristics, and potentials. As mentioned above, the mystery issues on LLMs could potentially be resolved through efficient and thorough evaluations. To assess the effectiveness and superiority of LLMs, a significant number of tasks and benchmarks have been introduced, aiming at empirically evaluating and analyzing their capabilities and the factors influencing their abilities (\cite{chia2023instructeval}; \cite{liang2022holistic}; \cite{zhao2023survey}; \cite{chang2023survey}; \cite{guo2023evaluating}).&#10;Current evaluation datasets predominantly focus on specific abilities like language understanding, reasoning, and human alignment individually. Previous research identifies several critical measures that must be considered in the evaluation of LLMs, such as accuracy, efficiency, bias, safety etc. (\cite{liang2022holistic}; \cite{chang2023survey}). Accuracy is paramount, encompassing not only factual correctness but also the precision of inferences and problem-solving. Efficiency is also vital. The speed at which these models generate results can be a decisive factor in their deployment for critical scenarios. Additionally, LLMs exhibit neutrality and are devoid of social biases. However, current LLM evaluations tend to prioritize accuracy (\cite{fu2023chain}; \cite{safdari2023personality}; \cite{choi2023llms}; \cite{yuan2023revisiting}; \cite{li2023api}).&#10;&#10;Recent evaluation efforts reveal several glaring issues. For instance, ``emergent abilities'' could be observed from a number of LLMs, such as GPT, PaLM and LaMDA (\cite{wei2022emergent}; \cite{schaeffer2023emergent}).&#10;Some researchers found that instruction-tuning provides a broad set of advantages compared with other types of training (fine-tune, pretrained, RL-tuned etc.) (\cite{liang2022holistic}; \cite{chung2022scaling}; \cite{zhao2023survey}). \cite{zhao2023survey} also reported that the small-sized open-source models perform not well on mathematical reasoning and scaling the open-source modes can improve the performance consistently. Researchers also found that some of the inconsistencies among the relationships between model size and task performance \cite{burnell2023revealing}. These findings actually are involved the overall performance of LLMs and different abilities with training types and scaling. However, the findings drawn from these studies primarily stem from observations made using a relatively small dataset. Notably, these findings have not undergone rigorous validation with a more extensive dataset. For enhanced reliability and accuracy of the results, further validation efforts could benefit from the application of comprehensive statistical methods. The following details these potential problems and challenges. &#10;&#10;A primary issue is the narrow range of models typically assessed in multiple tasks — often several to 30 (\cite{yu2023mm}; \cite{yu2023kola}; \cite{fu2023mme}; \cite{jiang2023structgpt}; \cite{huang2023trustgpt}), compared to the over 120000 models available, for instance, on Huggingface. The limited selection fails to capture the full spectrum of LLMs, diminishing our understanding of their diverse capabilities. For example, the limited number of LLMs may have emergent abilities. The question is whether these few models could represent the population of LLMs. As the array of LLMs expands and the availability of massive data on LLMs evaluation results, it is crucial to consider them collectively, requiring more inclusive sampling methods to ensure the findings' representativeness and reliability. When LLM evaluations frequently depend on datasets with insufficient sample sizes (often ranging from three to 30 datapoints), this raises concerns about the validity of conclusions drawn from such limited data/models.Further, essential characteristics of LLMs, such as emergent abilities, should be analyzed using larger datasets and considering broader factors like training types and architectures, not just parameters like scale (e.g., parameter count, FLOPs). Moreover, the interplay of different LLM capabilities has not been examined. Understanding the potential interactions among their various abilities, similar to the interplay seen in human cognitive abilities (\cite{conway2002latent}; \cite{buehner2006cognitive}; \cite{socher2022relationship}), remains a largely unexplored area in LLM research. Finally, the critical aspects of evaluating LLMs revolve around understanding the impact of scaling factors, training types, and architectural designs on their performance. This evaluation process bears resemblance to the assessment of human cognitive abilities, where factors like age, education, race, and sex are analyzed for their influence on cognitive skills within a population. These complex issues in both LLMs and human cognitive evaluations can be effectively addressed and validated through meticulous statistical testing and analysis. Similar challenges in other fields have been successfully addressed through the application of multiple statistical methods. &#10;&#10;The rapid growth of LLMs calls for more comprehensive and reliable evaluation methods. This necessitates broadening the scope of assessments, employing rigorous statistical methods. A straightforward, reliable and efficient approach is essential to accurately assess the capabilities and limitations of LLMs. To achieve this, large-scale data on evaluation results is needed, using consistent evaluation datasets and standards across numerous LLMs. Fortunately, some researchers have begun establishing platforms for this unified data collection. Once collected, both basic and advanced statistical methods could be applied to thoroughly analyze these evaluation result data. Currently, fundamental statistical techniques, such as ANOVA or $\chi$\^2 tests, are missing in testing resulting data. These analyses are crucial for understanding whether LLM performance varies significantly across different training types, architectures, and parameter sizes. Moreover, (non-)linear regression models can be employed to examine how training parameters or types affect LLM performance, and to explore the interactions among various LLM capabilities. These different statistical methods can cross-validate each other. These multifaceted statistical analyses will create a comprehensive framework, enabling an in-depth re-evaluation of the performance result data in LLMs, shown as in Fig.\ref{eva_met}.&#10;&#10;In short, the multifaceted statistical framework is poised to validate the core issues in evaluating LLMs, including their emergent abilities, the efficacy of training types and architectures, and the distinct advantages held by training parameter scale. This study also aims to delve into the interplay among various abilities in LLMs, investigating how they influence and interact with each other.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Large Language Models, Computer Science, Evaluation Methodologies, Mathematics, Statistical Analysis, Statistics" />
        </attvalues>
      </node>
      <node id="2305.17306" label="2305.17306">
        <attvalues>
          <attvalue for="0" value="Chain-of-Thought Hub: A Continuous Effort to Measure Large Language&#10;  Models' Reasoning Performance" />
          <attvalue for="1" value="  As large language models (LLMs) are continuously being developed, their&#10;evaluation becomes increasingly important yet challenging. This work proposes&#10;Chain-of-Thought Hub, an open-source evaluation suite on the multi-step&#10;reasoning capabilities of large language models. We are interested in this&#10;setting for two reasons: (1) from the behavior of GPT and PaLM model family, we&#10;observe that complex reasoning is likely to be a key differentiator between&#10;weaker and stronger LLMs; (2) we envisage large language models to become the&#10;next-generation computational platform and foster an ecosystem of LLM-based new&#10;applications, this naturally requires the foundation models to perform complex&#10;tasks that often involve the composition of linguistic and logical operations.&#10;Our approach is to compile a suite of challenging reasoning benchmarks to track&#10;the progress of LLMs. Our current results show that: (1) model scale clearly&#10;correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and&#10;PaLM-2 are the only two models that are comparable with GPT-4, while&#10;open-sourced models still lag behind; (3) LLaMA-65B performs closely to&#10;code-davinci-002, indicating that with successful further development such as&#10;reinforcement learning from human feedback (RLHF), it has great potential to be&#10;close to GPT-3.5-Turbo. Our results also suggest that for the open-source&#10;efforts to catch up, the community may focus more on building better base&#10;models and exploring RLHF.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.00184" label="2307.00184">
        <attvalues>
          <attvalue for="0" value="Personality Traits in Large Language Models" />
          <attvalue for="1" value="  The advent of large language models (LLMs) has revolutionized natural&#10;language processing, enabling the generation of coherent and contextually&#10;relevant human-like text. As LLMs increasingly power conversational agents used&#10;by the general public world-wide, the synthetic personality embedded in these&#10;models, by virtue of training on large amounts of human data, is becoming&#10;increasingly important. Since personality is a key factor determining the&#10;effectiveness of communication, we present a comprehensive method for&#10;administering and validating personality tests on widely-used LLMs, as well as&#10;for shaping personality in the generated text of such LLMs. Applying this&#10;method, we found: 1) personality measurements in the outputs of some LLMs under&#10;specific prompting configurations are reliable and valid; 2) evidence of&#10;reliability and validity of synthetic LLM personality is stronger for larger&#10;and instruction fine-tuned models; and 3) personality in LLM outputs can be&#10;shaped along desired dimensions to mimic specific human personality profiles.&#10;We discuss application and ethical implications of the measurement and shaping&#10;method, in particular regarding responsible AI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14938" label="2305.14938">
        <attvalues>
          <attvalue for="0" value="Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large&#10;  Language Models with SocKET Benchmark" />
          <attvalue for="1" value="  Large language models (LLMs) have been shown to perform well at a variety of&#10;syntactic, discourse, and reasoning tasks. While LLMs are increasingly deployed&#10;in many forms including conversational agents that interact with humans, we&#10;lack a grounded benchmark to measure how well LLMs understand \textit{social}&#10;language. Here, we introduce a new theory-driven benchmark, SocKET, that&#10;contains 58 NLP tasks testing social knowledge which we group into five&#10;categories: humor &amp; sarcasm, offensiveness, sentiment &amp; emotion, and&#10;trustworthiness. In tests on the benchmark, we demonstrate that current models&#10;attain only moderate performance but reveal significant potential for task&#10;transfer among different types and categories of tasks, which were predicted&#10;from theory. Through zero-shot evaluations, we show that pretrained models&#10;already possess some innate but limited capabilities of social language&#10;understanding and training on one category of tasks can improve zero-shot&#10;testing on others. Our benchmark provides a systematic way to analyze model&#10;performance on an important dimension of language and points to clear room for&#10;improvement to build more socially-aware LLMs. The associated resources are&#10;released at https://github.com/minjechoi/SOCKET.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.10062" label="2306.10062">
        <attvalues>
          <attvalue for="0" value="Revealing the structure of language model capabilities" />
          <attvalue for="1" value="  Building a theoretical understanding of the capabilities of large language&#10;models (LLMs) is vital for our ability to predict and explain the behavior of&#10;these systems. Here, we investigate the structure of LLM capabilities by&#10;extracting latent capabilities from patterns of individual differences across a&#10;varied population of LLMs. Using a combination of Bayesian and frequentist&#10;factor analysis, we analyzed data from 29 different LLMs across 27 cognitive&#10;tasks. We found evidence that LLM capabilities are not monolithic. Instead,&#10;they are better explained by three well-delineated factors that represent&#10;reasoning, comprehension and core language modeling. Moreover, we found that&#10;these three factors can explain a high proportion of the variance in model&#10;performance. These results reveal a consistent structure in the capabilities of&#10;different LLMs and demonstrate the multifaceted nature of these capabilities.&#10;We also found that the three abilities show different relationships to model&#10;properties such as model size and instruction tuning. These patterns help&#10;refine our understanding of scaling laws and indicate that changes to a model&#10;that improve one ability might simultaneously impair others. Based on these&#10;findings, we suggest that benchmarks could be streamlined by focusing on tasks&#10;that tap into each broad model ability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.13394" label="2306.13394">
        <attvalues>
          <attvalue for="0" value="MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language&#10;  Models" />
          <attvalue for="1" value="  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform&#10;multimodal tasks, showing amazing emergent abilities in recent studies, such as&#10;writing poems based on an image. However, it is difficult for these case&#10;studies to fully reflect the performance of MLLM, lacking a comprehensive&#10;evaluation. In this paper, we fill in this blank, presenting the first&#10;comprehensive MLLM Evaluation benchmark MME. It measures both perception and&#10;cognition abilities on a total of 14 subtasks. In order to avoid data leakage&#10;that may arise from direct use of public datasets for evaluation, the&#10;annotations of instruction-answer pairs are all manually designed. The concise&#10;instruction design allows us to fairly compare MLLMs, instead of struggling in&#10;prompt engineering. Besides, with such an instruction, we can also easily carry&#10;out quantitative statistics. A total of 30 advanced MLLMs are comprehensively&#10;evaluated on our MME, which not only suggests that existing MLLMs still have a&#10;large room for improvement, but also reveals the potential directions for the&#10;subsequent model optimization. The data application manner and online&#10;leaderboards are released at&#10;https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.09645" label="2305.09645">
        <attvalues>
          <attvalue for="0" value="StructGPT: A General Framework for Large Language Model to Reason over&#10;  Structured Data" />
          <attvalue for="1" value="  In this paper, we study how to improve the zero-shot reasoning ability of&#10;large language models~(LLMs) over structured data in a unified way. Inspired by&#10;the study on tool augmentation for LLMs, we develop an \emph{Iterative&#10;Reading-then-Reasoning~(IRR)} approach for solving question answering tasks&#10;based on structured data, called \textbf{StructGPT}. In our approach, we&#10;construct the specialized function to collect relevant evidence from structured&#10;data (\ie \emph{reading}), and let LLMs concentrate the reasoning task based on&#10;the collected information (\ie \emph{reasoning}). Specially, we propose an&#10;\emph{invoking-linearization-generation} procedure to support LLMs in reasoning&#10;on the structured data with the help of the external interfaces. By iterating&#10;this procedures with provided interfaces, our approach can gradually approach&#10;the target answer to a given query. Extensive experiments conducted on three&#10;types of structured data demonstrate the effectiveness of our approach, which&#10;can significantly boost the performance of ChatGPT and achieve comparable&#10;performance against the full-data supervised-tuning baselines. Our codes and&#10;data are publicly available at~\url{https://github.com/RUCAIBox/StructGPT}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.11507" label="2306.11507">
        <attvalues>
          <attvalue for="0" value="TrustGPT: A Benchmark for Trustworthy and Responsible Large Language&#10;  Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) such as ChatGPT, have gained significant&#10;attention due to their impressive natural language processing capabilities. It&#10;is crucial to prioritize human-centered principles when utilizing these models.&#10;Safeguarding the ethical and moral compliance of LLMs is of utmost importance.&#10;However, individual ethical issues have not been well studied on the latest&#10;LLMs. Therefore, this study aims to address these gaps by introducing a new&#10;benchmark -- TrustGPT. TrustGPT provides a comprehensive evaluation of LLMs in&#10;three crucial areas: toxicity, bias, and value-alignment. Initially, TrustGPT&#10;examines toxicity in language models by employing toxic prompt templates&#10;derived from social norms. It then quantifies the extent of bias in models by&#10;measuring quantifiable toxicity values across different groups. Lastly,&#10;TrustGPT assesses the value of conversation generation models from both active&#10;value-alignment and passive value-alignment tasks. Through the implementation&#10;of TrustGPT, this research aims to enhance our understanding of the performance&#10;of conversation generation models and promote the development of language&#10;models that are more ethical and socially responsible.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.16850" label="2312.16850">
        <attvalues>
          <attvalue for="0" value="Accent-VITS:accent transfer for end-to-end TTS" />
          <attvalue for="1" value="  Accent transfer aims to transfer an accent from a source speaker to synthetic&#10;speech in the target speaker's voice. The main challenge is how to effectively&#10;disentangle speaker timbre and accent which are entangled in speech. This paper&#10;presents a VITS-based end-to-end accent transfer model named Accent-VITS.Based&#10;on the main structure of VITS, Accent-VITS makes substantial improvements to&#10;enable effective and stable accent transfer.We leverage a hierarchical CVAE&#10;structure to model accent pronunciation information and acoustic features,&#10;respectively, using bottleneck features and mel spectrums as&#10;constraints.Moreover, the text-to-wave mapping in VITS is decomposed into&#10;text-to-accent and accent-to-wave mappings in Accent-VITS. In this way, the&#10;disentanglement of accent and speaker timbre becomes be more stable and&#10;effective.Experiments on multi-accent and Mandarin datasets show that&#10;Accent-VITS achieves higher speaker similarity, accent similarity and speech&#10;naturalness as compared with a strong baseline.&#10;" />
          <attvalue for="2" value="&#10;In recent years, there have been significant advancements in neural text-to-speech (TTS), which can generate human-like natural speech from input text. Accented speech is highly desired for a better user experience in many TTS applications. Cross-speaker accent transfer is a promising technology for accented speech synthesis, which aims to transfer an accent from a source speaker to the synthetic speech in the target speaker's voice. Accent transfer can promote cross-region communication and make a TTS system better adapt to diverse language environments and user needs. &#10;&#10;An accent is usually reflected in the phoneme pronunciation pattern and prosody variations, both of which are key attributes of the accent rendering \cite{LootsN11, MareuilV06, DBLP:journals/corr/abs-2209-10804}. &#10;The segmental and suprasegmental structures may be in distinctive pronunciation patterns for different accents and influence the listening perception of speaking accents \cite{KolluruWLYG14, abs-2305-04816}. &#10;The prosody variations of accent are characterized by different pitch, energy, duration, and other prosodic appearance. To build an accent TTS system using accent transfer, the research problem can be treated as how to effectively disentangle speaker timbre and accent factors in speech. However, it is difficult to force the system to sufficiently disentangle the accent from the speaker timbre and content in speech since both pronunciation and prosody attributes are featured by local variations at the fine-grained level.&#10;And usually, each speaker has only one accent in the training phase which adds to the difficulty of disentangling.&#10;&#10;Previous approaches attempting to disentangle accent attributes and speaker timbre are mainly based on Domain Adversarial Training (DAT) ~\cite{DAT}. However, when the feature extraction function has a high capacity, DAT poses a weak constraint to the feature extraction function. Therefore, a single classifier with a gradient reversal layer in accent transfer TTS cannot disentangle the accent from the speaker's timbre, as the accent is varied in prosody and pronunciation. Additionally, gradient descent in domain adversarial training can violate the optimizer's asymptotic convergence guarantees, often hindering the transfer performance ~\cite{AcunaLZF22}. Applying DAT in accent transfer tasks, especially when each speaker has only one accent in the training phase, may result in inefficient and unstable feature disentanglement. Furthermore, there is a trade-off between speaker similarity and accent similarity, which means entirely removing speaker timbre hurts performance on preserving accent pronunciation ~\cite{ShuBNE18}. &#10;&#10;Bottleneck (BN) features are recently used as an intermediate representation to supervise accent attribute modeling in TTS ~\cite{ZhangWYSWX22}. The BN feature, extracted from a well-trained neural ASR model, is considered to be noise-robust and speaker-independent \cite{bnrecog, BNrobust}, which benifits speaker timbre and accent disentanglement. &#10;However, in the methods with BN as an intermediate representation \cite{bnTTS1, Hiertron}, models are often trained independently in multiple stages.&#10;This can lead to the issue of error accumulation and model mismatch between each stage, resulting in the degradation of synthesized speech quality and accent attributes.&#10;&#10;In this paper, we propose an end-to-end accent transfer model, Accent-VITS, with a hierarchical conditional variational autoencoder (CVAE) ~\cite{HierSpeech} utilizing bottleneck features as a constraint to eliminate speaker timbre from the original signal.&#10;Specifically, we leverage the end-to-end speech synthesis framework, VITS ~\cite{vits}, as the backbone of our model, since it achieves good audio quality and alleviates the error accumulation caused by the conventional two-stage TTS system consisting of an acoustic model and a vocoder.&#10;Based on the VITS structure, an additional CVAE is added to extract an accent-dependent latent distribution from the BN feature. The latent representation contains the accent and linguistic content and is modeled by the accented phoneme sequence input.&#10;The BN constraint factorizes the cross-speaker accent TTS into two joint-training processes, which are text-to-accent and accent-to-wave. The text-to-accent process takes the accented phoneme sequence as input to generate an accent-dependent distribution. The accent-to-wave process produces the speech distribution in the target accent and target speaker from the output accent distribution and is conditioned on speaker identity. This design enables more effective learning of accent attributes, leading to sufficient disentanglement and superior performance of accent transfer for the synthesized speech.&#10;Experimental results on Mandarin multi-accent datasets demonstrate the superiority of our proposed model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Accent Transfer, Machine Learning, Computer Science, Speech Processing, Linguistics, Engineering, Deep Learning, Speech Synthesis" />
        </attvalues>
      </node>
      <node id="2202.05352" label="2202.05352">
        <attvalues>
          <attvalue for="0" value="Domain Adversarial Training: A Game Perspective" />
          <attvalue for="1" value="  The dominant line of work in domain adaptation has focused on learning&#10;invariant representations using domain-adversarial training. In this paper, we&#10;interpret this approach from a game theoretical perspective. Defining optimal&#10;solutions in domain-adversarial training as a local Nash equilibrium, we show&#10;that gradient descent in domain-adversarial training can violate the asymptotic&#10;convergence guarantees of the optimizer, oftentimes hindering the transfer&#10;performance. Our analysis leads us to replace gradient descent with high-order&#10;ODE solvers (i.e., Runge-Kutta), for which we derive asymptotic convergence&#10;guarantees. This family of optimizers is significantly more stable and allows&#10;more aggressive learning rates, leading to high performance gains when used as&#10;a drop-in replacement over standard optimizers. Our experiments show that in&#10;conjunction with state-of-the-art domain-adversarial methods, we achieve up to&#10;3.5% improvement with less than of half training iterations. Our optimizers are&#10;easy to implement, free of additional parameters, and can be plugged into any&#10;domain-adversarial framework.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.08735" label="1802.08735">
        <attvalues>
          <attvalue for="0" value="A DIRT-T Approach to Unsupervised Domain Adaptation" />
          <attvalue for="1" value="  Domain adaptation refers to the problem of leveraging labeled data in a&#10;source domain to learn an accurate model in a target domain where labels are&#10;scarce or unavailable. A recent approach for finding a common representation of&#10;the two domains is via domain adversarial training (Ganin &amp; Lempitsky, 2015),&#10;which attempts to induce a feature extractor that matches the source and target&#10;feature distributions in some feature space. However, domain adversarial&#10;training faces two critical limitations: 1) if the feature extraction function&#10;has high-capacity, then feature distribution matching is a weak constraint, 2)&#10;in non-conservative domain adaptation (where no single classifier can perform&#10;well in both the source and target domains), training the model to do well on&#10;the source domain hurts performance on the target domain. In this paper, we&#10;address these issues through the lens of the cluster assumption, i.e., decision&#10;boundaries should not cross high-density data regions. We propose two novel and&#10;related models: 1) the Virtual Adversarial Domain Adaptation (VADA) model,&#10;which combines domain adversarial training with a penalty term that punishes&#10;the violation the cluster assumption; 2) the Decision-boundary Iterative&#10;Refinement Training with a Teacher (DIRT-T) model, which takes the VADA model&#10;as initialization and employs natural gradient steps to further minimize the&#10;cluster assumption violation. Extensive empirical results demonstrate that the&#10;combination of these two models significantly improve the state-of-the-art&#10;performance on the digit, traffic sign, and Wi-Fi recognition domain adaptation&#10;benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.17305" label="2210.17305">
        <attvalues>
          <attvalue for="0" value="AccentSpeech: Learning Accent from Crowd-sourced Data for Target Speaker&#10;  TTS with Accents" />
          <attvalue for="1" value="  Learning accent from crowd-sourced data is a feasible way to achieve a target&#10;speaker TTS system that can synthesize accent speech. To this end, there are&#10;two challenging problems to be solved. First, direct use of the poor acoustic&#10;quality crowd-sourced data and the target speaker data in accent transfer will&#10;apparently lead to synthetic speech with degraded quality. To mitigate this&#10;problem, we take a bottleneck feature (BN) based TTS approach, in which TTS is&#10;decomposed into a Text-to-BN (T2BN) module to learn accent and a BN-to-Mel&#10;(BN2Mel) module to learn speaker timbre, where neural network based BN feature&#10;serves as the intermediate representation that are robust to noise&#10;interference. Second, direct training T2BN using the crowd-sourced data in the&#10;two-stage system will produce accent speech of target speaker with poor&#10;prosody. This is because the the crowd-sourced recordings are contributed from&#10;the ordinary unprofessional speakers. To tackle this problem, we update the&#10;two-stage approach to a novel three-stage approach, where T2BN and BN2Mel are&#10;trained using the high-quality target speaker data and a new BN-to-BN module is&#10;plugged in between the two modules to perform accent transfer. To train the&#10;BN2BN module, the parallel unaccented and accented BN features are obtained by&#10;a proposed data augmentation procedure. Finally the proposed three-stage&#10;approach manages to produce accent speech for the target speaker with good&#10;prosody, as the prosody pattern is inherited from the professional target&#10;speaker and accent transfer is achieved by the BN2BN module at the same time.&#10;The proposed approach, named as AccentSpeech, is validated in a Mandarin TTS&#10;accent transfer task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.03439" label="2109.03439">
        <attvalues>
          <attvalue for="0" value="Referee: Towards reference-free cross-speaker style transfer with&#10;  low-quality data for expressive speech synthesis" />
          <attvalue for="1" value="  Cross-speaker style transfer (CSST) in text-to-speech (TTS) synthesis aims at&#10;transferring a speaking style to the synthesised speech in a target speaker's&#10;voice. Most previous CSST approaches rely on expensive high-quality data&#10;carrying desired speaking style during training and require a reference&#10;utterance to obtain speaking style descriptors as conditioning on the&#10;generation of a new sentence. This work presents Referee, a robust&#10;reference-free CSST approach for expressive TTS, which fully leverages&#10;low-quality data to learn speaking styles from text. Referee is built by&#10;cascading a text-to-style (T2S) model with a style-to-wave (S2W) model.&#10;Phonetic PosteriorGram (PPG), phoneme-level pitch and energy contours are&#10;adopted as fine-grained speaking style descriptors, which are predicted from&#10;text using the T2S model. A novel pretrain-refinement method is adopted to&#10;learn a robust T2S model by only using readily accessible low-quality data. The&#10;S2W model is trained with high-quality target data, which is adopted to&#10;effectively aggregate style descriptors and generate high-fidelity speech in&#10;the target speaker's voice. Experimental results are presented, showing that&#10;Referee outperforms a global-style-token (GST)-based baseline approach in CSST.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.08609" label="2112.08609">
        <attvalues>
          <attvalue for="0" value="DuQM: A Chinese Dataset of Linguistically Perturbed Natural Questions&#10;  for Evaluating the Robustness of Question Matching Models" />
          <attvalue for="1" value="  In this paper, we focus on studying robustness evaluation of Chinese question&#10;matching. Most of the previous work on analyzing robustness issue focus on just&#10;one or a few types of artificial adversarial examples. Instead, we argue that&#10;it is necessary to formulate a comprehensive evaluation about the linguistic&#10;capabilities of models on natural texts. For this purpose, we create a Chinese&#10;dataset namely DuQM which contains natural questions with linguistic&#10;perturbations to evaluate the robustness of question matching models. DuQM&#10;contains 3 categories and 13 subcategories with 32 linguistic perturbations.&#10;The extensive experiments demonstrate that DuQM has a better ability to&#10;distinguish different models. Importantly, the detailed breakdown of evaluation&#10;by linguistic phenomenon in DuQM helps us easily diagnose the strength and&#10;weakness of different models. Additionally, our experiment results show that&#10;the effect of artificial adversarial examples does not work on the natural&#10;texts.&#10;" />
          <attvalue for="2" value="&#10;&#10;The task of Question Matching (QM) aims to identify the question pairs that have the same meaning, and it has been widely used in many applications, e.g., community question answering and intelligent customer services, etc. &#10;Though neural QM models have&#10;shown compelling performance on various datasets, including Quora Question Pairs (QQP)~\cite{iyer2017first}, LCQMC~\cite{liu2018lcqmc}, BQ~\cite{chen2018bq} and AFQMC, &#10;neural models are often not robust to adversarial examples, which means that the neural models predict unexpected outputs given just a small perturbations on the inputs.&#10;As the example 1 in Tab.~\ref{tab:statistics-exp} shows, a model might not distinguish the minor difference (&quot;面~noodles&quot;) between the two sentences, and thus predicts the two questions semantically equivalent. &#10;&#10;Recently, it attracts a lot of attentions from the research community to deal with the robustness issues of neural models on various NLP tasks, such as question matching, natural language inference and machine reading comprehension. &#10;Early works examine the robustness of neural models by creating a certain types of artificial adversarial examples~\cite{jia2017adversarial,alzantot2018generating,ren2019generating,jin2020bert}, and involving human-and-model-in-the-loop to create dynamic adversarial examples~\cite{nie2019adversarial,wallace2019trick}. &#10;Further studies discover that a few types of superficial cues (i.e. shortcuts) in the training data, are learned by the models and hence affect the model robustness~\cite{gururangan2018annotation,mccoy2019right,lai2021machine}.&#10;Besides, several studies try to improve the robustness of the neural models by adversarial data augmentation~\cite{min2020syntactic} and data filtering~\cite{le2020adversarial}. &#10;All these efforts lead us to better find and fix the robustness issues to some extends. &#10;&#10;However, there are several limitations in previous studies. &#10;First, the analysis and evaluation in previous work focus on just one or a few types of adversarial examples or shortcuts, &#10;but we need normative evaluation~\cite{linzen2020can, ettinger2020bert,phangadversarially}. &#10;The goal of the normative evaluation is not to fool a system by exploiting its particular weaknesses, but using systemically controlled datasets to comprehensively evaluate the basic linguistic capabilities of the models in a diverse way. &#10;Checklist~\cite{ribeiro2020beyond} and Textflint~\cite{gui2021textflint} are great attempts of normative evaluation. However, it is not clear that if the effects of the artificial adversarial methods on artificial examples are still shown on natural texts from real-world applications~\cite{morris2020reevaluating}.&#10;Some other works manually perturb the examples to construct natural examples, but the manual perturbations is time consuming and costly~\cite{gardner2020evaluating}.&#10;Moreover, to the best of our knowledge, there are few Chinese datasets for QM robustness evaluation. &#10;&#10;Towards this end, we create a open-domain Chinese dataset namely DuQM contains natural questions with linguistic perturbation for evaluating the robustness of QM models. &#10;(1) By linguistic, we mean this systematically controlled dataset provides a detailed breakdown of evaluation by linguistic phenomenon. As shown in Tab.~\ref{tab:statistics-exp}, there are 3 categories and 13 subcategories with 32 linguistic perturbation in DuQM, which enables us to evaluate the model performance by each category instead of just a single metric. &#10;(2) By natural, we mean all the questions in DuQM are natural and issued by the users in a commercial search engine. This design can help us to properly evaluate the progress of a model's robustness on natural texts rather than artificial texts which may not preserve semantics and introduce grammatical errors. &#10;&#10;The contributions of this paper can be summarized as follows:&#10;\begin{itemize}[leftmargin=*,noitemsep]&#10;\item We construct a Chinese dataset namely DuQM that contains linguistically perturbed natural questions from a commercial search engine. &#10;It is a systemically controlled dataset to test the basic linguistic capabilities of the models in a diverse way. (see Sec.~\ref{Constructing QQBugs Dataset} and Sec.~\ref{construction overview})&#10;\item Our experimental results show that 3 characteristics of DuQM: (1) DuQM is challenging, and has better discrimination power to distinguish the models that perform comparably on other datasets (see Sec.~\ref{sec:challenging}). (2) The detailed breakdown of evaluation by linguistic phenomena in DuQM helps diagnose the advantages and disadvantages of different models (see Sec.~\ref{sec:fine-grained}). &#10;(3) Extensive experiment shows that the effect of artificial adversarial examples does not work on natural texts of DuQM. DuQM can help us properly evaluate the models' robustness. (see Sec.~\ref{sec:natural}). &#10;\end{itemize}&#10;&#10;The remaining of this paper is organized as follows. Sec.~\ref{Constructing QQBugs Dataset} describes the 3 categories and 13 subcategories with 32 linguistic perturbation in DuQM. Sec.~\ref{Statistics} gives the construction process of DuQM. &#10;In Sec.~\ref{setup}, we conduct experiments to demonstrate 3 characteristics of DuQM. &#10;We conclude our work in Sec.~\ref{sec-conclusion}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Robustness Evaluation, Artificial Intelligence, Question Matching Models, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1907.11932" label="1907.11932">
        <attvalues>
          <attvalue for="0" value="Is BERT Really Robust? A Strong Baseline for Natural Language Attack on&#10;  Text Classification and Entailment" />
          <attvalue for="1" value="  Machine learning algorithms are often vulnerable to adversarial examples that&#10;have imperceptible alterations from the original counterparts but can fool the&#10;state-of-the-art models. It is helpful to evaluate or even improve the&#10;robustness of these models by exposing the maliciously crafted adversarial&#10;examples. In this paper, we present TextFooler, a simple but strong baseline to&#10;generate natural adversarial text. By applying it to two fundamental natural&#10;language tasks, text classification and textual entailment, we successfully&#10;attacked three target models, including the powerful pre-trained BERT, and the&#10;widely used convolutional and recurrent neural networks. We demonstrate the&#10;advantages of this framework in three ways: (1) effective---it outperforms&#10;state-of-the-art attacks in terms of success rate and perturbation rate, (2)&#10;utility-preserving---it preserves semantic content and grammaticality, and&#10;remains correctly classified by humans, and (3) efficient---it generates&#10;adversarial text with computational complexity linear to the text length. *The&#10;code, pre-trained target models, and test examples are available at&#10;https://github.com/jind11/TextFooler.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.02701" label="1809.02701">
        <attvalues>
          <attvalue for="0" value="Trick Me If You Can: Human-in-the-loop Generation of Adversarial&#10;  Examples for Question Answering" />
          <attvalue for="1" value="  Adversarial evaluation stress tests a model's understanding of natural&#10;language. While past approaches expose superficial patterns, the resulting&#10;adversarial examples are limited in complexity and diversity. We propose&#10;human-in-the-loop adversarial generation, where human authors are guided to&#10;break models. We aid the authors with interpretations of model predictions&#10;through an interactive user interface. We apply this generation framework to a&#10;question answering task called Quizbowl, where trivia enthusiasts craft&#10;adversarial questions. The resulting questions are validated via live&#10;human--computer matches: although the questions appear ordinary to humans, they&#10;systematically stump neural and information retrieval models. The adversarial&#10;questions cover diverse phenomena from multi-hop reasoning to entity type&#10;distractors, exposing open challenges in robust question answering.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.11594" label="2102.11594">
        <attvalues>
          <attvalue for="0" value="Unidirectional Memory-Self-Attention Transducer for Online Speech&#10;  Recognition" />
          <attvalue for="1" value="  Self-attention models have been successfully applied in end-to-end speech&#10;recognition systems, which greatly improve the performance of recognition&#10;accuracy. However, such attention-based models cannot be used in online speech&#10;recognition, because these models usually have to utilize a whole acoustic&#10;sequences as inputs. A common method is restricting the field of attention&#10;sights by a fixed left and right window, which makes the computation costs&#10;manageable yet also introduces performance degradation. In this paper, we&#10;propose Memory-Self-Attention (MSA), which adds history information into the&#10;Restricted-Self-Attention unit. MSA only needs localtime features as inputs,&#10;and efficiently models long temporal contexts by attending memory states.&#10;Meanwhile, recurrent neural network transducer (RNN-T) has proved to be a great&#10;approach for online ASR tasks, because the alignments of RNN-T are local and&#10;monotonic. We propose a novel network structure, called Memory-Self-Attention&#10;(MSA) Transducer. Both encoder and decoder of the MSA Transducer contain the&#10;proposed MSA unit. The experiments demonstrate that our proposed models improve&#10;WER results than Restricted-Self-Attention models by $13.5 on WSJ and $7.1 on&#10;SWBD datasets relatively, and without much computation costs increase.&#10;" />
          <attvalue for="2" value="&#10;In the past few years, models employing transformer structure have achieved state-of-art results for many tasks, such as nature language understanding, machine translation, and speech recognition. Especially in speech recognition, lots of attention-based models have proved to obtain a substantial performance improvement~\cite{Miao2019}\cite{Pham2019}\cite{Sperber2018}\cite{Chorowski2015}. For example, self-attention blocks are successfully applied in CTC-based network, SAN-CTC~\cite{salazar2019self} showed that self-attention encoder is competitive with existing end-to-end models. Speech-transformer~\cite{dong2018speech} proposed a 2D-Attention module, which computes attention weights on both time and frequency axes, in order to extract more discriminated representations of speech features. Transformer-Transducer~\cite{yeh2019transformer} used VGGNet with causal convolution as the frontend of encoder, and self-attention transducer as the network architecture. However, such attention-based models cannot be used in online speech recognition, because these models usually have to utilize a whole acoustic sequences as inputs. An additional challenge is that the computation complexity of these models increases quadratically with input sequence length, which is unacceptable for online ASR tasks. A typical solution for this challenge is restricting the field of attention sights by a fixed left and right window, which makes the computation costs manageable but also leads to performance degradation. To overcome the drawbacks of these restricted-attention model, we proposed Memory-Self-Attention (MSA) in this paper. MSA only needs localtime features as inputs, and efficiently models long temporal contexts by attending memory states. These memory states help MSA to gain better performance than window-restricted-attention models. Moreover, the computation complexity of MSA is linear with input sequence length, which is significant for online ASR tasks.&#10;&#10;CTC~\cite{graves2006connectionist}, Transformer~\cite{vaswani2017attention}, RNN-Transducer~\cite{tian2019self}\cite{Wang2019} are most common used architectures in speech recognition~\cite{battenberg2017exploring}. CTC~\cite{sainath2015convolutional}\cite{amodei2016deep} is first widely used to end-to-end models, but CTC has a fatal drawback that every timestep is outputted independently. Therefore, it has to be optimized jointly with external language model in practice. Transformer~\cite{Zeyer2019}\cite{Li2019} is another choice by encoder-decoder infrastructure. However, the mechanism of transformer allows the model to attend anywhere in the input sequence at each timestep. Therefore, the alignments of transformer are non-local and non-monotonic. The RNN-Transducer~\cite{graves2013speech}\cite{Tsunoo2019} was proposed as an extension to CTC, which also marginalizes over all possible alignments between the input sequence and the output targets. RNN-Transducer is typically composed of an encoder, which transforms the acoustic features into high-level representations, and a decoder, which produces linguistic outputs. Previous works employed GRU or LSTM as the encoders, giving the RNN-T its name. In this paper, we explore the possibility of replacing RNN-based encoders and decoders with our proposed MSA units, which is called Memory-Self-Attention (MSA) Transducer. MSA Transducer can learn an implicit language model and therefore removes the conditional independence assumption in CTC. More importantly, the alignments of RNN-T are local and monotonic, allowing MSA Transducer for online speech recognition tasks.&#10;&#10;In the previous works, some unidirectional neural network architectures were proposed for online ASR tasks, such as deep LCBLSTM~\cite{Xue2017Improving}, and TDNN-LSTM~\cite{Peddinti2017Low}. Daniel Povey proposed a time-restricted-attention layer for ASR~\cite{Povey2018}, and used it in LF-MMI~\cite{povey2016purely} models which are not end-to-end. Two unidirectional architectures, the time-delay LSTM (TDLSTM) and&#10;parallel time-delayed LSTM (PTDLSTM) were presented in~\cite{Moritz2019}. Another researches are worked on local monotonic attention~\cite{Andre2019}\cite{dong2019self}. Google proposed transformer encoders with RNN-T loss~\cite{ZhangTransformer}, and they showed that limiting the left and right context of attention per-layer can obtain not bad accuracy but still have some gap between the performance of full-attention models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Neural Network Architectures, Attention Models, Artificial Intelligence, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1909.13037" label="1909.13037">
        <attvalues>
          <attvalue for="0" value="Self-Attention Transducers for End-to-End Speech Recognition" />
          <attvalue for="1" value="  Recurrent neural network transducers (RNN-T) have been successfully applied&#10;in end-to-end speech recognition. However, the recurrent structure makes it&#10;difficult for parallelization . In this paper, we propose a self-attention&#10;transducer (SA-T) for speech recognition. RNNs are replaced with self-attention&#10;blocks, which are powerful to model long-term dependencies inside sequences and&#10;able to be efficiently parallelized. Furthermore, a path-aware regularization&#10;is proposed to assist SA-T to learn alignments and improve the performance.&#10;Additionally, a chunk-flow mechanism is utilized to achieve online decoding.&#10;All experiments are conducted on a Mandarin Chinese dataset AISHELL-1. The&#10;results demonstrate that our proposed approach achieves a 21.3% relative&#10;reduction in character error rate compared with the baseline RNN-T. In&#10;addition, the SA-T with chunk-flow mechanism can perform online decoding with&#10;only a little degradation of the performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.11871" label="1910.11871">
        <attvalues>
          <attvalue for="0" value="Towards Online End-to-end Transformer Automatic Speech Recognition" />
          <attvalue for="1" value="  The Transformer self-attention network has recently shown promising&#10;performance as an alternative to recurrent neural networks in end-to-end (E2E)&#10;automatic speech recognition (ASR) systems. However, Transformer has a drawback&#10;in that the entire input sequence is required to compute self-attention. We&#10;have proposed a block processing method for the Transformer encoder by&#10;introducing a context-aware inheritance mechanism. An additional context&#10;embedding vector handed over from the previously processed block helps to&#10;encode not only local acoustic information but also global linguistic, channel,&#10;and speaker attributes. In this paper, we extend it towards an entire online&#10;E2E ASR system by introducing an online decoding process inspired by monotonic&#10;chunkwise attention (MoChA) into the Transformer decoder. Our novel MoChA&#10;training and inference algorithms exploit the unique properties of Transformer,&#10;whose attentions are not always monotonic or peaky, and have multiple heads and&#10;residual connections of the decoder layers. Evaluations of the Wall Street&#10;Journal (WSJ) and AISHELL-1 show that our proposed online Transformer decoder&#10;outperforms conventional chunkwise approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.06450" label="1902.06450">
        <attvalues>
          <attvalue for="0" value="Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using&#10;  Self-Attention Network and Chunk-Hopping" />
          <attvalue for="1" value="  Self-attention network, an attention-based feedforward neural network, has&#10;recently shown the potential to replace recurrent neural networks (RNNs) in a&#10;variety of NLP tasks. However, it is not clear if the self-attention network&#10;could be a good alternative of RNNs in automatic speech recognition (ASR),&#10;which processes the longer speech sequences and may have online recognition&#10;requirements. In this paper, we present a RNN-free end-to-end model:&#10;self-attention aligner (SAA), which applies the self-attention networks to a&#10;simplified recurrent neural aligner (RNA) framework. We also propose a&#10;chunk-hopping mechanism, which enables the SAA model to encode on segmented&#10;frame chunks one after another to support online recognition. Experiments on&#10;two Mandarin ASR datasets show the replacement of RNNs by the self-attention&#10;networks yields a 8.4%-10.2% relative character error rate (CER) reduction. In&#10;addition, the chunk-hopping mechanism allows the SAA to have only a 2.5%&#10;relative CER degradation with a 320ms latency. After jointly training with a&#10;self-attention network language model, our SAA model obtains further error rate&#10;reduction on multiple datasets. Especially, it achieves 24.12% CER on the&#10;Mandarin ASR benchmark (HKUST), exceeding the best end-to-end model by over 2%&#10;absolute CER.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.01140" label="2001.01140">
        <attvalues>
          <attvalue for="0" value="Transformer-based language modeling and decoding for conversational&#10;  speech recognition" />
          <attvalue for="1" value="  We propose a way to use a transformer-based language model in conversational&#10;speech recognition. Specifically, we focus on decoding efficiently in a&#10;weighted finite-state transducer framework. We showcase an approach to lattice&#10;re-scoring that allows for longer range history captured by a transfomer-based&#10;language model and takes advantage of a transformer's ability to avoid&#10;computing sequentially.&#10;" />
          <attvalue for="2" value="&#10;In conversational speech, individual utterances may reference context from previous utterances. If certain topics or words have been mentioned in the past, related words are likely to be used. In automatic speech recognition, language models are responsible for capturing the probabilities of words likely to be uttered given past words. Traditionally, these probabilities are captured in an n-gram language model. For example, in a trigram language model, we would store a mapping of all 3-word combinations found in our training corpus, along with the probabilities of the third word following the previous two words. The past &quot;history&quot; that a trigram language model would capture is limited to two words. It is also limited in its ability to capture semantics. With the advent of more computational power, neural-based methods for language modeling, like the Recurrent Neural Network (RNN) architecture have become possible. In neural architectures, words are typically represented as word embeddings, n-dimensional vectors that attempt to capture the semantics in the latent space. Due to its recurrent set-up, the RNN-based language model is able to encapsulate previous word embeddings in its hidden state (see Figure \ref{fig:rnn}).&#10;&#10;The hidden state at each step represents a single value computed from a series of matrix operations on current word embedding and the hidden state from the previous step. However, with a long sequence, RNNs on their own suffer from vanishing gradients, Long Short-Term Memory (LSTM) units remedy this by introducing additional operations for computing each hidden state \cite{GRAD}. Despite their name, they fall short of capturing word dependencies past 200 words and focus on past 50 words more heavily \cite{LSTM_issues}. The transformer architecture, however, does not suffer from this problem. It does away with recurrence in favor of an attention mechanism \cite{ATTENTION}. The attention mechanism allows the network to learn a weighted average to determine which words are important at at each position. The network can capture much longer dependencies as all previous words are encoded and passed into the multi-head attention layers in parallel (see Figure \ref{fig:transformer}). The multi-head attention layer learns to reference words from the beginning of a very long context. This is a desirable attribute that we'd like to apply to conversational speech recognition, however, there is one problem: many transformer-based architectures take in fixed size input \cite{BERT}. Any change to the input, whether it be adding another word, or changing the last word in the input would require recomputing all values of the network. This can be prohibitive in the case of speech recognition as it would require the re-computation of all word inputs from the beginning of our speech context for every new utterance that we attempt to re-score. The transformer-XL architecture solves this by introducing a segment-level recurrence mechanism \cite{Transformer_xl}. In this work we use the Kaldi framework \cite{kaldi} and a transfomer-XL architecture to do efficient lattice re-scoring. For each new utterance, we cache the segment-level embeddings for use in future utterance lattice re-scoring.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Conversational Speech Recognition, Computer Science, Linguistics, Signal Processing, Mathematics, Transformer-Based Decoding, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2409.10788" label="2409.10788">
        <attvalues>
          <attvalue for="0" value="Exploring Prediction Targets in Masked Pre-Training for Speech&#10;  Foundation Models" />
          <attvalue for="1" value="  Speech foundation models, such as HuBERT and its variants, are pre-trained on&#10;large amounts of unlabeled speech for various downstream tasks. These models&#10;use a masked prediction objective, where the model learns to predict&#10;information about masked input segments from the unmasked context. The choice&#10;of prediction targets in this framework can influence performance on downstream&#10;tasks. For example, targets that encode prosody are beneficial for&#10;speaker-related tasks, while targets that encode phonetics are more suited for&#10;content-related tasks. Additionally, prediction targets can vary in the level&#10;of detail they encode; targets that encode fine-grained acoustic details are&#10;beneficial for denoising tasks, while targets that encode higher-level&#10;abstractions are more suited for content-related tasks. Despite the importance&#10;of prediction targets, the design choices that affect them have not been&#10;thoroughly studied. This work explores the design choices and their impact on&#10;downstream task performance. Our results indicate that the commonly used design&#10;choices for HuBERT can be suboptimal. We propose novel approaches to create&#10;more informative prediction targets and demonstrate their effectiveness through&#10;improvements across various downstream tasks.&#10;" />
          <attvalue for="2" value="&#10;Speech foundation models are trained on large amounts of unlabeled data using self-supervised learning (SSL). They can be used as pre-training weights~\cite{Wav2Vec2,BEST-RQ}, or as feature extractors for lightweight prediction heads~\cite{SuPERB}. This paper focuses on the latter use case.&#10;The success of SSL speech models depends on having a powerful encoder capable of producing features that are effective across a range of downstream tasks, including automatic speech recognition (ASR), speaker identification, and source separation. Consequently, numerous SSL approaches for learning encoders have been introduced (see~\cite{mohamed2022self} for a review).&#10;Among these, a particularly successful family of models makes use of the masked prediction objective, where the model is trained to reconstruct information randomly masked in the input from the unmasked context.&#10;Notable examples in this family include HuBERT~\cite{HuBERT} and its derivatives~\cite{WavLM,UniSAT,contentvec,multiresHuBERT,chen2024robustspeechrepresentationlearning}, which we refer to collectively as HuBERT-based methods.&#10;&#10;The choice of prediction targets is critical to the success of this paradigm.&#10;Early works~\cite{TERA,Mockingjay,decoar} explored using low-level spectral features as prediction targets.&#10;However, such targets are challenging to reconstruct due to their continuous and fine-grained nature~\cite{bai20223}. &#10;Consequently, later works~\cite{w2v-BERT,wav2vec-c,Wav2Vec2} explored methods for quantizing targets to abstract the fine-grained speech properties.&#10;Wav2Vec 2.0~\cite{Wav2Vec2} designed a quantization module trained jointly with the masked prediction objective.&#10;HuBERT~\cite{HuBERT} improved upon Wav2Vec 2.0 by replacing the quantization module with iterative clustering on learned features.&#10;BEST-RQ~\cite{BEST-RQ} used random-projection quantizer to quantize speech signals to discrete labels.&#10;Recent approaches~\cite{WavLM,contentvec,multiresHuBERT,UniSAT,chen2024robustspeechrepresentationlearning} built upon the iterative clustering framework of HuBERT with architectural changes and data augmentations.&#10;&#10;The iterative clustering procedure used in HuBERT has been shown to improve representation learning of foundation models, and it is the primary focus of our study.&#10;Iterative clustering involves key design decisions that affect the prediction targets.&#10;These decisions, in turn, influence the performance of SSL features across various downstream tasks.&#10;Note that HuBERT~\cite{HuBERT} original focused on content-based tasks, such as ASR, but widely adopted as a general foundation model~\cite{SuPERB,9747870,wang2022finetunedwav2vec20hubertbenchmark}. These observations motivate us to explore and adjust design decisions to support a wider range of downstream tasks.&#10;&#10;We study how design decisions in the iterative clustering process of HuBERT-based methods affect the quality of the features for various downstream tasks. Specifically, we investigate design decisions that affect the prediction targets in two dimensions: 1) the content encoded and 2) the amount of information captured, which will be detailed in Section~\ref{sec:method}.&#10;We analyze how variations in these two dimensions affect performance on downstream tasks.&#10;We demonstrate that the widely used setup is suboptimal across the speech task.&#10;We propose methods for enhancing the prediction targets, which attempt to improve the model's performance on phone recognition, speaker identification, and speech separation simultaneously.&#10;Our systematic analysis on the design decisions provides useful guidance for research on masked prediction of speech.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Acoustics, Computer Science, Prediction Target Design, Linguistics, Speech Foundation Models, Downstream Task Performance, Signal Processing" />
        </attvalues>
      </node>
      <node id="2404.09385" label="2404.09385">
        <attvalues>
          <attvalue for="0" value="A Large-Scale Evaluation of Speech Foundation Models" />
          <attvalue for="1" value="  The foundation model paradigm leverages a shared foundation model to achieve&#10;state-of-the-art (SOTA) performance for various tasks, requiring minimal&#10;downstream-specific modeling and data annotation. This approach has proven&#10;crucial in the field of Natural Language Processing (NLP). However, the speech&#10;processing community lacks a similar setup to explore the paradigm&#10;systematically. In this work, we establish the Speech processing Universal&#10;PERformance Benchmark (SUPERB) to study the effectiveness of the paradigm for&#10;speech. We propose a unified multi-tasking framework to address speech&#10;processing tasks in SUPERB using a frozen foundation model followed by&#10;task-specialized, lightweight prediction heads. Combining our results with&#10;community submissions, we verify that the foundation model paradigm is&#10;promising for speech, and our multi-tasking framework is simple yet effective,&#10;as the best-performing foundation model shows competitive generalizability&#10;across most SUPERB tasks. For reproducibility and extensibility, we have&#10;developed a long-term maintained platform that enables deterministic&#10;benchmarking, allows for result sharing via an online leaderboard, and promotes&#10;collaboration through a community-driven benchmark database to support new&#10;development cycles. Finally, we conduct a series of analyses to offer an&#10;in-depth understanding of SUPERB and speech foundation models, including&#10;information flows across tasks inside the models, the correctness of the&#10;weighted-sum benchmarking protocol and the statistical significance and&#10;robustness of the benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.02720" label="2310.02720">
        <attvalues>
          <attvalue for="0" value="Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised&#10;  Learning with Masked Unit Prediction" />
          <attvalue for="1" value="  Existing Self-Supervised Learning (SSL) models for speech typically process&#10;speech signals at a fixed resolution of 20 milliseconds. This approach&#10;overlooks the varying informational content present at different resolutions in&#10;speech signals. In contrast, this paper aims to incorporate multi-resolution&#10;information into speech self-supervised representation learning. We introduce a&#10;SSL model that leverages a hierarchical Transformer architecture, complemented&#10;by HuBERT-style masked prediction objectives, to process speech at multiple&#10;resolutions. Experimental results indicate that the proposed model not only&#10;achieves more efficient inference but also exhibits superior or comparable&#10;performance to the original HuBERT model over various tasks. Specifically,&#10;significant performance improvements over the original HuBERT have been&#10;observed in fine-tuning experiments on the LibriSpeech speech recognition&#10;benchmark as well as in evaluations using the Speech Universal PERformance&#10;Benchmark (SUPERB) and Multilingual SUPERB (ML-SUPERB).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.00837" label="2407.00837">
        <attvalues>
          <attvalue for="0" value="Towards Robust Speech Representation Learning for Thousands of Languages" />
          <attvalue for="1" value="  Self-supervised learning (SSL) has helped extend speech technologies to more&#10;languages by reducing the need for labeled data. However, models are still far&#10;from supporting the world's 7000+ languages. We propose XEUS, a Cross-lingual&#10;Encoder for Universal Speech, trained on over 1 million hours of data across&#10;4057 languages, extending the language coverage of SSL models 4-fold. We&#10;combine 1 million hours of speech from existing publicly accessible corpora&#10;with a newly created corpus of 7400+ hours from 4057 languages, which will be&#10;publicly released. To handle the diverse conditions of multilingual speech&#10;data, we augment the typical SSL masked prediction approach with a novel&#10;dereverberation objective, increasing robustness. We evaluate XEUS on several&#10;benchmarks, and show that it consistently outperforms or achieves comparable&#10;results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS&#10;sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT&#10;2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or&#10;pre-training data. Checkpoints, code, and data are found in&#10;https://www.wavlab.org/activities/2024/xeus/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.09690" label="2203.09690">
        <attvalues>
          <attvalue for="0" value="A$^3$T: Alignment-Aware Acoustic and Text Pretraining for Speech&#10;  Synthesis and Editing" />
          <attvalue for="1" value="  Recently, speech representation learning has improved many speech-related&#10;tasks such as speech recognition, speech classification, and speech-to-text&#10;translation. However, all the above tasks are in the direction of speech&#10;understanding, but for the inverse direction, speech synthesis, the potential&#10;of representation learning is yet to be realized, due to the challenging nature&#10;of generating high-quality speech. To address this problem, we propose our&#10;framework, Alignment-Aware Acoustic-Text Pretraining (A$^3$T), which&#10;reconstructs masked acoustic signals with text input and acoustic-text&#10;alignment during training. In this way, the pretrained model can generate high&#10;quality reconstructed spectrogram, which can be applied to the speech editing&#10;and unseen speaker TTS directly. Experiments show A$^3$T outperforms SOTA&#10;models on speech editing, and improves multi-speaker speech synthesis without&#10;the external speaker verification model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.03896" label="2202.03896">
        <attvalues>
          <attvalue for="0" value="Speech Emotion Recognition using Self-Supervised Features" />
          <attvalue for="1" value="  Self-supervised pre-trained features have consistently delivered state-of-art&#10;results in the field of natural language processing (NLP); however, their&#10;merits in the field of speech emotion recognition (SER) still need further&#10;investigation. In this paper we introduce a modular End-to- End (E2E) SER&#10;system based on an Upstream + Downstream architecture paradigm, which allows&#10;easy use/integration of a large variety of self-supervised features. Several&#10;SER experiments for predicting categorical emotion classes from the IEMOCAP&#10;dataset are performed. These experiments investigate interactions among&#10;fine-tuning of self-supervised feature models, aggregation of frame-level&#10;features into utterance-level features and back-end classification networks.&#10;The proposed monomodal speechonly based system not only achieves SOTA results,&#10;but also brings light to the possibility of powerful and well finetuned&#10;self-supervised acoustic features that reach results similar to the results&#10;achieved by SOTA multimodal systems using both Speech and Text modalities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.03938" label="2310.03938">
        <attvalues>
          <attvalue for="0" value="EFFUSE: Efficient Self-Supervised Feature Fusion for E2E ASR in Low&#10;  Resource and Multilingual Scenarios" />
          <attvalue for="1" value="  Self-Supervised Learning (SSL) models have demonstrated exceptional&#10;performance in various speech tasks, particularly in low-resource and&#10;multilingual domains. Recent works show that fusing diverse SSL models could&#10;achieve superior performance compared to using one SSL model. However, fusing&#10;models increases the overall parameter size, leading to higher computational&#10;costs. We propose EFFUSE, a novel approach that uses a single SSL model to&#10;mimic the features of multiple SSL models via prediction, resulting in a&#10;lightweight framework with competitive performance. Our experiments show that&#10;EFFUSE outperforms individual SSL models in multilingual speech recognition&#10;tasks. Our best performing model achieves an average SUPERB score increase of&#10;63.5 (6.3%) from the SSL baselines in Multilingual Speech Universal PERformance&#10;Benchmark (ML-SUPERB), while decreasing parameter size on average by 317M&#10;parameters (49%) from the fusion models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent works have incorporated self-supervised learning (SSL) models into end-to-end (E2E) automatic speech recognition (ASR) systems as frontend feature extractors to capture more relevant features of corpora~\cite{Chang2021AnEO, 9893562, Yang2021SUPERBSP, KrishnaD2021UsingLS, 9801640, shi2023multi, chen2023joint, chang2023colld}. Despite the encouraging progress in models equipped with an SSL frontend, there are still many limitations when applying SSL features. One such restraint is that most SSL models are trained with corpora in a single English talker scenario. Thus, this setting may not align well with other languages or recording conditions. Previous works have shown that these models might not capture enough information for downstream tasks in other languages or recording environments \cite{sanabria2022measuring, MengCLL22, ZuluagaGmez2022HowDP}. &#10;As a result, methods such as feature fusion \cite{berrebbi22_interspeech}, adapters \cite{10095130, 9746223}, teacher-student&#10;distillation \cite{Peng2021ShrinkingBR, 9747490, Peng2023DPHuBERTJD, lee22p_interspeech, Yang2021KnowledgeDF}, and continual training \cite{DBLP:conf/interspeech/HuangFZL22, vandereeckt_eusipco2022, Hsu2021RobustW2, 9746594}, have been proposed, which have shown performance improvements for different target domains.&#10;Given the diversity of speech, with different varieties &#10;of language and recording conditions, it is difficult to capture all of the relevant information. Feature fusion is especially effective as each SSL can capture unique, distinct information \cite{berrebbi22_interspeech, fearless}. As a result, performing feature fusion captures the wide diversity of the speech features, by utilizing all of multiple SSL models' representations.&#10;After the fusion, the output feature is expected to carry more information related to the downstream task.&#10;&#10;Based upon the powerful benefits of feature fusion, we propose an efficient self-supervised feature fusion (EFFUSE) methodology to utilize features from multiple SSL models, while limiting the increase in parameters and latency.&#10;Specifically, EFFUSE consists of a two-stage training strategy: (1)~Fusion stage: train the model with a fusion of self-supervised features, similar to \cite{berrebbi22_interspeech, fearless}. (2) Prediction stage: continue to train the downstream model by using only one SSL model's features to predict other SSL model representations by a minimal number of linear layers.&#10;&#10;Our proposed EFFUSE method reduces the number of parameters by using a single SSL model to mimic the benefits of multiple SSL models, thereby scaling down inference costs compared to the original feature fusion network \cite{berrebbi22_interspeech, fearless}. &#10;We extensively validate our framework's effectiveness in both the low-resource and multilingual speech-processing tasks. EFFUSE demonstrates an average reduction of 4.5 absolute (20\% relative) character error rate (CER) in the low-resource domain over two benchmarks (Yoloxital Mixtec and Totonac) and 2.4 absolute (6.6\% relative) CER in the multilingual domain over the Multilingual Speech Universal PERformance Benchmark (ML-SUPERB). Notably, there is an average relative decrease of 16\% in the real-time factor (RTF) during inference, compared to the conventional fusion model.&#10;&#10;Our contributions can be summarized as follows: &#10;(1) we extensively explore the SSL fusion model in low-resource and multilingual scenarios;&#10;(2) we propose a novel framework that employs one SSL model to predict the features of other SSL models in the fusion network during inference; (3) we demonstrate that our approach can improve the performance of downstream multilingual tasks while maintaining reasonable inference costs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Models, Signal Processing, Mathematics, Speech Recognition, Self-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2302.09331" label="2302.09331">
        <attvalues>
          <attvalue for="0" value="Front-End Adapter: Adapting Front-End Input of Speech based&#10;  Self-Supervised Learning for Speech Recognition" />
          <attvalue for="1" value="  Recent years have witnessed a boom in self-supervised learning (SSL) in&#10;various areas including speech processing. Speech based SSL models present&#10;promising performance in a range of speech related tasks. However, the training&#10;of SSL models is computationally expensive and a common practice is to&#10;fine-tune a released SSL model on the specific task. It is essential to use&#10;consistent front-end input during pre-training and fine-tuning. This&#10;consistency may introduce potential issues when the optimal front-end is not&#10;the same as that used in pre-training. In this paper, we propose a simple but&#10;effective front-end adapter to address this front-end discrepancy. By&#10;minimizing the distance between the outputs of different front-ends, the&#10;filterbank feature (Fbank) can be compatible with SSL models which are&#10;pre-trained with waveform. The experiment results demonstrate the effectiveness&#10;of our proposed front-end adapter on several popular SSL models for the speech&#10;recognition task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.16104" label="2203.16104">
        <attvalues>
          <attvalue for="0" value="Improving Distortion Robustness of Self-supervised Speech Processing&#10;  Tasks with Domain Adaptation" />
          <attvalue for="1" value="  Speech distortions are a long-standing problem that degrades the performance&#10;of supervisely trained speech processing models. It is high time that we&#10;enhance the robustness of speech processing models to obtain good performance&#10;when encountering speech distortions while not hurting the original performance&#10;on clean speech. In this work, we propose to improve the robustness of speech&#10;processing models by domain adversarial training (DAT). We conducted&#10;experiments based on the SUPERB framework on five different speech processing&#10;tasks. In case we do not always have knowledge of the distortion types for&#10;speech data, we analyzed the binary-domain and multi-domain settings, where the&#10;former treats all distorted speech as one domain, and the latter views&#10;different distortions as different domains. In contrast to supervised training&#10;methods, we obtained promising results in target domains where speech data is&#10;distorted with different distortions including new unseen distortions&#10;introduced during testing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.13458" label="2302.13458">
        <attvalues>
          <attvalue for="0" value="Varianceflow: High-Quality and Controllable Text-to-Speech using&#10;  Variance Information via Normalizing Flow" />
          <attvalue for="1" value="  There are two types of methods for non-autoregressive text-to-speech models&#10;to learn the one-to-many relationship between text and speech effectively. The&#10;first one is to use an advanced generative framework such as normalizing flow&#10;(NF). The second one is to use variance information such as pitch or energy&#10;together when generating speech. For the second type, it is also possible to&#10;control the variance factors by adjusting the variance values provided to a&#10;model. In this paper, we propose a novel model called VarianceFlow combining&#10;the advantages of the two types. By modeling the variance with NF, VarianceFlow&#10;predicts the variance information more precisely with improved speech quality.&#10;Also, the objective function of NF makes the model use the variance information&#10;and the text in a disentangled manner resulting in more precise variance&#10;control. In experiments, VarianceFlow shows superior performance over other&#10;state-of-the-art TTS models both in terms of speech quality and&#10;controllability.&#10;" />
          <attvalue for="2" value="&#10;In Text-to-Speech (TTS), the one-to-many relationship between text and speech is one of the major problems that makes it challenging to learn the text-to-speech conversion.&#10;The early autoregressive (AR) TTS models \cite{shen2018natural, li2019neural} dealt with the difficulty by factorizing the speech distribution into the product of homogeneous conditional factors in sequential order.&#10;However, although they succeeded in generating high-quality speech, their slow inference speed and exposure bias were inevitable problems inherent in the AR models.&#10;&#10;As it has advanced from AR TTS models to non-AR TTS models, two types of methods for the non-AR models to solve the one-to-many problem have been proposed.&#10;The first type (Type-$\textup{\uppercase}$) is to use an advanced generative framework such as normalizing flow \cite{NEURIPS2020_5c3b99e8}, diffusion model \cite{pmlr-v139-popov21a}, and generative adversarial network \cite{yang21e_interspeech}.&#10;Unlike the mean squared error (MSE) based training that assumes a Gaussian distribution, these frameworks do not assume any pre-defined distribution for a target distribution.&#10;As a result, they can generate high-quality and diverse speech samples compared to the previous non-AR TTS models trained with the MSE loss \cite{NEURIPS2019_f63f65b5, Peng2020nonautoregressive}.&#10;The second type of the methods (Type-$\textup{\uppercase}$) is to solve a task of TTS by dividing it into simpler two tasks based on variance information such as pitch or energy \cite{ren2021fastspeech, lancucki2021fastpitch, bak21_interspeech}: (1) text conditioned variance modeling; (2) text and variance information conditioned speech generation.&#10;Then, by using ground-truth variance information when learning the two tasks, Type-$\textup{\uppercase}$ models achieve faster training convergence and higher speech quality.&#10;Moreover, unlike Type-$\textup{\uppercase}$ models, Type-$\textup{\uppercase}$ models can explicitly control the variance factors by manipulating the variance values used in the speech generation.&#10;&#10;In this paper, we propose VarianceFlow, a novel model that combines the advantages of Type-$\textup{\uppercase}$ and Type-$\textup{\uppercase}$, achieving a high-quality and controllable TTS model.&#10;Unlike the previous Type-$\textup{\uppercase}$ models that learn the text conditioned variance modeling based on the MSE loss, our model uses normalizing flow (NF) for the variance modeling (Figure \ref{fig: varianceflow}).&#10;&#10;There are two large advantages of using NF for the variance modeling.&#10;First, since NF is robust to the one-to-many problem, it learns the variance distribution better than using the MSE loss \cite{DBLP:conf/iclr/DinhSB17}, and it leads to improved speech quality.&#10;Second, NF enhances the variance controllability by disentangling the latent variance representation and the text \cite{NEURIPS2020_1cfa81af}.&#10;&#10;In experiments, VarianceFlow shows its superiority in variance modeling by showing better speech quality compared to other AR and non-AR state-of-the-art TTS models.&#10;In addition, VarianceFlow shows its more precise variance controllability by showing that it uses the provided pitch values more intactly.&#10;Lastly, we show that VarianceFlow can generate diverse speech samples given a text input.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Signal Processing, Mathematics, Text-to-Speech Models, Variance Control Methods, Generative Frameworks" />
        </attvalues>
      </node>
      <node id="2106.15153" label="2106.15153">
        <attvalues>
          <attvalue for="0" value="GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech&#10;  Synthesis" />
          <attvalue for="1" value="  Recent advances in neural multi-speaker text-to-speech (TTS) models have&#10;enabled the generation of reasonably good speech quality with a single model&#10;and made it possible to synthesize the speech of a speaker with limited&#10;training data. Fine-tuning to the target speaker data with the multi-speaker&#10;model can achieve better quality, however, there still exists a gap compared to&#10;the real speech sample and the model depends on the speaker. In this work, we&#10;propose GANSpeech, which is a high-fidelity multi-speaker TTS model that adopts&#10;the adversarial training method to a non-autoregressive multi-speaker TTS&#10;model. In addition, we propose simple but efficient automatic scaling methods&#10;for feature matching loss used in adversarial training. In the subjective&#10;listening tests, GANSpeech significantly outperformed the baseline&#10;multi-speaker FastSpeech and FastSpeech2 models, and showed a better MOS score&#10;than the speaker-specific fine-tuned FastSpeech2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1605.08803" label="1605.08803">
        <attvalues>
          <attvalue for="0" value="Density estimation using Real NVP" />
          <attvalue for="1" value="  Unsupervised learning of probabilistic models is a central yet challenging&#10;problem in machine learning. Specifically, designing models with tractable&#10;learning, sampling, inference and evaluation is crucial in solving this task.&#10;We extend the space of such models using real-valued non-volume preserving&#10;(real NVP) transformations, a set of powerful invertible and learnable&#10;transformations, resulting in an unsupervised learning algorithm with exact&#10;log-likelihood computation, exact sampling, exact inference of latent&#10;variables, and an interpretable latent space. We demonstrate its ability to&#10;model natural images on four datasets through sampling, log-likelihood&#10;evaluation and latent variable manipulations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.13580" label="2005.13580">
        <attvalues>
          <attvalue for="0" value="Network-to-Network Translation with Conditional Invertible Neural&#10;  Networks" />
          <attvalue for="1" value="  Given the ever-increasing computational costs of modern machine learning&#10;models, we need to find new ways to reuse such expert models and thus tap into&#10;the resources that have been invested in their creation. Recent work suggests&#10;that the power of these massive models is captured by the representations they&#10;learn. Therefore, we seek a model that can relate between different existing&#10;representations and propose to solve this task with a conditionally invertible&#10;network. This network demonstrates its capability by (i) providing generic&#10;transfer between diverse domains, (ii) enabling controlled content synthesis by&#10;allowing modification in other domains, and (iii) facilitating diagnosis of&#10;existing representations by translating them into interpretable domains such as&#10;images. Our domain transfer network can translate between fixed representations&#10;without having to learn or finetune them. This allows users to utilize various&#10;existing domain-specific expert models from the literature that had been&#10;trained with extensive computational resources. Experiments on diverse&#10;conditional image synthesis tasks, competitive image modification results and&#10;experiments on image-to-image and text-to-image generation demonstrate the&#10;generic applicability of our approach. For example, we translate between BERT&#10;and BigGAN, state-of-the-art text and image models to provide text-to-image&#10;generation, which neither of both experts can perform on their own.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.01544" label="2311.01544">
        <attvalues>
          <attvalue for="0" value="Divergent Token Metrics: Measuring degradation to prune away LLM&#10;  components -- and optimize quantization" />
          <attvalue for="1" value="  Large Language Models (LLMs) have reshaped natural language processing with&#10;their impressive capabilities. However, their ever-increasing size has raised&#10;concerns about their effective deployment and the need for LLM compression.&#10;This study introduces the Divergent Token Metrics (DTMs), a novel approach to&#10;assessing compressed LLMs, addressing the limitations of traditional perplexity&#10;or accuracy measures that fail to accurately reflect text generation quality.&#10;DTMs measure token divergences that allow deeper insights into the subtleties&#10;of model compression, in particular, when evaluating components' impacts&#10;individually. Utilizing the First Divergent Token Metric (FDTM) in model&#10;sparsification reveals that 25% of all attention components can be pruned&#10;beyond 90% on the Llama-2 model family, still keeping SOTA performance. For&#10;quantization, FDTM suggests that more than 80% of parameters can be naively&#10;transformed to int8 without special outlier management. These evaluations&#10;indicate the necessity of choosing appropriate compressions for parameters&#10;individually -- and that FDTM can identify those -- while standard metrics&#10;result in deteriorated outcomes.&#10;" />
          <attvalue for="2" value="&#10;&#10;Cutting-edge Large Language Models (LLMs) based on the transformer architecture \cite{vaswani2017attention} have revolutionized Natural Language Processing with their exceptional performance, notably exemplified by the GPT series~\cite{radford2018improving,radford2019language,brown2020language,bubeck2023sparks} in text generation. &#10;However, these models have grown massively, even exceeding half a trillion parameters~\cite{chowdhery2022palm}. Although the large number of parameters aid in early training convergence, their practical utility and true necessity remain unclear. In particular, for the attention mechanism, it was hinted that after some training convergence, certain heads dominate the inference process~\cite{michel2019sixteen}. &#10;&#10;Compression strategies such as sparsification and quantization can enhance the efficiency of the parameters.&#10;Current metrics, however, either average too coarsely, such as perplexity, or are by design too specific, such as standard NLP benchmarks.&#10;Both fail to capture the diverging performance nuances introduced early on by the compression as they ignore the actual discontinuous text generation process. This, however, is the main application of the final model, and so we argue that they are therefore insufficient measures for the performance of the compressed model.&#10;This misalignment can lead to unwanted subtle discrepancies in generation, &#10;such as grammatical errors or a mismatch in numbers, that cause subsequent divergences. As we will show, this even occurs when overall metrics, such as perplexity, appear satisfactory (cf. Prop.~\ref{prop:ppl_discontinuity}, Sec.~\ref{sec:compression}, Tab.~\ref{tab:comp_model_benchmarks}). &#10;&#10;To meet these challenges, we introduce the family of Divergent Token Metrics (DTMs) in Sec.~\ref{sec:metrics}. &#10;These metrics are tailored to measure the model divergence of LLMs throughout the compression process&#10;and in relation to the actual generation procedure on a token basis, as shown in Fig.~\ref{fig:metric-example}.&#10;We demonstrate that the First Divergent Token Metric (FDTM) and the Share of Divergent Tokens Metric (SDTM) offer a more nuanced evaluation compared to perplexity. They also enable individual component evaluation to rank parts of the model best suited for compression, thus enabling meaningful compression while preserving text generation quality.&#10;Based on FDT probing, we introduce new strategies for sparsifying and quantizing models in Sec.~\ref{sec:compression}.&#10;&#10;Specifically, our proposed individual component sparsification indicates significant differences in component utilization across layers. &#10;For the first time, we show that 25\% percent of the models' attention components can be pruned beyond 90\%, and several even entirely removed, while preserving a single-digit perplexity. &#10;Consequently, a sparse matrix format can be employed to accelerate computational efficiency.&#10;Likewise, for precision reduction, we show that sorting components by FDTM coincidentally correlates to sorting by their induced number of outliers when being naively converted to int8. FDTM identifies the optimal 80\% of the components that maintain overall performance without specific outlier handling.&#10;The observed decline in performance with more outliers, and the significant influence of specific components on those, suggests reevaluating the applied normalization methods throughout the model.&#10;We demonstrate that this level of precision goes beyond what standard perplexity and conventional NLP benchmarks can achieve.&#10;The proposed Divergent Token Metrics closely reflect the generation process and so can be a measure to foster confidence in the deployed compressed models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Compression, Mathematics, Natural Language Processing, Model Evaluation Metrics" />
        </attvalues>
      </node>
      <node id="2405.05445" label="2405.05445">
        <attvalues>
          <attvalue for="0" value="Large Language Model Enhanced Machine Learning Estimators for&#10;  Classification" />
          <attvalue for="1" value="  Pre-trained large language models (LLM) have emerged as a powerful tool for&#10;simulating various scenarios and generating output given specific instructions&#10;and multimodal input. In this work, we analyze the specific use of LLM to&#10;enhance a classical supervised machine learning method for classification&#10;problems. We propose a few approaches to integrate LLM into a classical machine&#10;learning estimator to further enhance the prediction performance. We examine&#10;the performance of the proposed approaches through both standard supervised&#10;learning binary classification tasks, and a transfer learning task where the&#10;test data observe distribution changes compared to the training data. Numerical&#10;experiments using four publicly available datasets are conducted and suggest&#10;that using LLM to enhance classical machine learning estimators can provide&#10;significant improvement on prediction performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Classification is a fundamental task in supervised machine learning, common across a wide array of applications. It involves training a model on a dataset where each instance is assigned a specific class label. When presented with a new, unlabeled instance, the trained model is expected to accurately predict the instance's class.&#10;&#10;To illustrate, consider a classification problem aimed at predicting whether a customer will find a product relevant online. In this problem, each instance involves two-fold information on (i) customer needs, e.g., reflected by the search queries of the customer and other attributes, and (ii) product attributes, including title, description, and/or images of the product. In the training dataset, each instance is labelled with one of the two classifying categories: ``relevant&quot; or ``not relevant&quot;. &#10;&#10;There are a wide range of well-established machine learning methods for such classification tasks, ranging from logistic regression, tree-based methods, to neural-networks, among many other; see \cite{efron2021computer}. These methods are typically trained and calibrated using the training dataset. Once trained, they serve as estimators to predict the labels of new, unlabeled instances. &#10;&#10;On the other hand, the emergence of pre-trained large language models (LLMs) offers an additional approach to these classification tasks, capable of functioning as estimators with or without additional fine-tuning on task-specific data. For instance, GPT \cite{radford2018improving} by OpenAI can serve as an estimator by simply taking an instance (comprising customer query and product information) as input and generating a prediction on whether the product is relevant to the customer's needs. Moreover, prompt engineering and fine-tuning can further enhance the model's performance on specific classification tasks.&#10;&#10;This work is motivated by the following questions. Can we integrate LLM to classical machine learning methods to significantly enhance the performance, compared to the separate use of LLM or the separate use of a classical machine learning model? What are the different ways of doing such integration, and how do they perform compared to benchmarks? &#10;&#10;We deliver the following results and analysis in this work. \begin{enumerate}&#10; \item We analyze the linear combination of an LLM model and a machine learning (ML) model. We observe that generally LLM predictions are more reliable than that of a machine learning model on borderline data. We then develop an adaptive weighted linear combination of LLM and ML to further enhance the performance via the heavier use of LLM on those regions where ML shows less confidence.&#10; \item By treating LLM predictions as additional group information, we apply model calibration methods to classical machine learning models. This method is straightforward to carry out and can be used upon any classical machine learning model. &#10; \item We consider the integration of LLM and ML on a transfer learning classification tasks. For the transfer learning task with covariate shift, we improve the machine learning model by augmenting the training data with additional samples from the target distribution, where the labels of those samples are generated by LLM. We then train a machine learning model on the augmented dataset to obtain better performance on the target distribution.&#10; \item We illustrate the empirical performances of our methods on four public datasets, including tasks such as relevance prediction, emotion recognition and hate speech detection. Numerical results show that all our methods perform better than only using LLM or only using a classical machine learning model.&#10;\end{enumerate}&#10;&#10;We would like to add some discussions before proceeding to the main sections of this work. The advantage of leveraging LLMs to enhance classical ML methods comes from two primary sources. Firstly, LLMs can serve as a variance reduction tool in addition to a classical machine learning model trained on the dataset. The use of LLMs in this context draws a close analogy to the method of control variates as a variance reduction tool in simulation literature; refer to \cite{asmussen2007stochastic}. We also note that the use of LLM in a classical data-driven method creates additional needs for input data uncertainty analysis, and leave that for future discussions; see \cite{song2014advanced,feng2019efficient}. Secondly, LLMs can enhance model accuracy by leveraging their knowledge on a broader range of data; see \cite{moller2023prompt,gao2023chat,chen2024exploring}. This aspect is particularly beneficial for improving classical ML models in transfer learning tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Classification Performance, Linguistics, Machine Learning Enhancement, Language Model Integration, Artificial Intelligence, Mathematics, Statistics" />
        </attvalues>
      </node>
      <node id="2211.00106" label="2211.00106">
        <attvalues>
          <attvalue for="0" value="Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks" />
          <attvalue for="1" value="  Large multilingual language models typically share their parameters across&#10;all languages, which enables cross-lingual task transfer, but learning can also&#10;be hindered when training updates from different languages are in conflict. In&#10;this paper, we propose novel methods for using language-specific subnetworks,&#10;which control cross-lingual parameter sharing, to reduce conflicts and increase&#10;positive transfer during fine-tuning. We introduce dynamic subnetworks, which&#10;are jointly updated with the model, and we combine our methods with&#10;meta-learning, an established, but complementary, technique for improving&#10;cross-lingual transfer. Finally, we provide extensive analyses of how each of&#10;our methods affects the models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large multilingual language models, such as mBERT \cite{devlin2018bert}, are pretrained on data covering many languages, but share their parameters across all languages. This modeling approach has several powerful advantages, such as allowing similar languages to exert positive influence on each other, and enabling cross-lingual task transfer (i.e., finetuning on some source language(s), then using the model on different target languages) \cite{pires2019multilingual}.&#10;These advantages are particularly enticing in low-resource scenarios since without sufficient training data in the target language, the model's effectiveness hinges on its ability to derive benefit from other languages' data.&#10;In practice, however, even state-of-the-art multilingual models tend to perform poorly on low-resource languages \cite{lauscher2020zero, ustun2020udapter}, due in part to negative interference effects---parameter updates that help the model on one language, but harm its ability to handle another---which undercut the benefits of multilingual modeling \cite{arivazhagan2019massively, wang2020negative, ansell2021mad}.&#10;&#10;In this paper, we propose novel methods for using language-specific subnetworks, which control cross-lingual parameter sharing, to reduce conflicts and increase positive transfer during fine-tuning, with the goal of improving the performance of multilingual language models on low-resource languages.&#10;While recent works apply various subnetwork based approaches to their models statically \cite{lu2022language, yang2022learning, nooralahzadeh2022improving}, we propose a new method that allows the model to dynamically update the subnetworks during fine-tuning. This allows for sharing between language pairs to a different extent at the different learning stages of the models. We accomplish this by using pruning techniques \cite{frankle2018lottery} to select an optimal subset of parameters from the full model for further language-specific fine-tuning. Inspired by studies that show that attention-heads in BERT-based models have specialized functions \cite{voita2019analyzing, htut2019attention}, we focus on learning subnetworks at the attention-head level. We learn separate---but potentially overlapping---head masks for each language by fine-tuning the model on the language, and then pruning out the least important heads.&#10;&#10;Given our focus on low-resource languages, we also combine our methods with meta-learning, a data-efficient technique to learn tasks from a few samples \cite{finn2017model}. Motivated by \cite{wang2020negative}, who find that meta-learning can reduce negative interference in the multilingual setup, we test how much our subnetwork methods can further benefit performance in this learning framework, as well as compare the subnetwork based approach to a meta-learning baseline. &#10;Our results show that a combination of meta-learning and dynamic subnetworks is especially powerful.&#10;To the best of our knowledge, we are the first to adapt subnetwork sharing to the meta-learning framework.&#10;&#10;We extensively test the effectiveness of our methods on the task of dependency parsing. We use data from Universal Dependencies (UD) \cite{nivre2016universal} comprising 86 datasets covering 74 distinct languages, from 43 language families; 58 of the languages can be considered truly low-resource. &#10;Our experiments show, quantitatively, that our language-specific subnetworks, when used &#10;during fine-tuning, act as an effective sharing mechanism: permitting positive influence from similar languages, while shielding each language's parameters from negative interference that would otherwise have been introduced by more distant languages. &#10;Moreover, we show substantial improvements in cross-lingual transfer to new languages at test time. Importantly, we are able to achieve this while requiring considerably less time and data, i.e. training for less than a day compared to $\sim$20 days for current state-of-the-art \cite{kondratyuk201975}, while relying on data from just 8 treebanks.&#10;&#10;Finally, we perform extensive analyses of our models to better understand how different choices affect generalisation properties. We analyse model behaviour with respect to several factors: typological relatedness of fine-tuning and test languages, data-scarcity during pretraining, robustness to domain transfer, and their ability to predict rare and unseen labels. We find interesting differences in model behaviour that can provide useful guidance on which method to choose based on the properties of the target language.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Lingual Transfer Learning, Computer Science, Linguistics, Cognitive Science, Multilingual Language Models, Mathematics, Meta-Learning Techniques" />
        </attvalues>
      </node>
      <node id="2203.04583" label="2203.04583">
        <attvalues>
          <attvalue for="0" value="Language Adaptive Cross-lingual Speech Representation Learning with&#10;  Sparse Sharing Sub-networks" />
          <attvalue for="1" value="  Unsupervised cross-lingual speech representation learning (XLSR) has recently&#10;shown promising results in speech recognition by leveraging vast amounts of&#10;unlabeled data across multiple languages. However, standard XLSR model suffers&#10;from language interference problem due to the lack of language specific&#10;modeling ability. In this work, we investigate language adaptive training on&#10;XLSR models. More importantly, we propose a novel language adaptive&#10;pre-training approach based on sparse sharing sub-networks. It makes room for&#10;language specific modeling by pruning out unimportant parameters for each&#10;language, without requiring any manually designed language specific component.&#10;After pruning, each language only maintains a sparse sub-network, while the&#10;sub-networks are partially shared with each other. Experimental results on a&#10;downstream multilingual speech recognition task show that our proposed method&#10;significantly outperforms baseline XLSR models on both high resource and low&#10;resource languages. Besides, our proposed method consistently outperforms other&#10;adaptation methods and requires fewer parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.02258" label="2101.02258">
        <attvalues>
          <attvalue for="0" value="Can RNNs learn Recursive Nested Subject-Verb Agreements?" />
          <attvalue for="1" value="  One of the fundamental principles of contemporary linguistics states that&#10;language processing requires the ability to extract recursively nested tree&#10;structures. However, it remains unclear whether and how this code could be&#10;implemented in neural circuits. Recent advances in Recurrent Neural Networks&#10;(RNNs), which achieve near-human performance in some language tasks, provide a&#10;compelling model to address such questions. Here, we present a new framework to&#10;study recursive processing in RNNs, using subject-verb agreement as a probe&#10;into the representations of the neural network. We trained six distinct types&#10;of RNNs on a simplified probabilistic context-free grammar designed to&#10;independently manipulate the length of a sentence and the depth of its&#10;syntactic tree. All RNNs generalized to subject-verb dependencies longer than&#10;those seen during training. However, none systematically generalized to deeper&#10;tree structures, even those with a structural bias towards learning nested tree&#10;(i.e., stack-RNNs). In addition, our analyses revealed primacy and recency&#10;effects in the generalization patterns of LSTM-based models, showing that these&#10;models tend to perform well on the outer- and innermost parts of a&#10;center-embedded tree structure, but poorly on its middle levels. Finally,&#10;probing the internal states of the model during the processing of sentences&#10;with nested tree structures, we found a complex encoding of grammatical&#10;agreement information (e.g. grammatical number), in which all the information&#10;for multiple words nouns was carried by a single unit. Taken together, these&#10;results indicate how neural networks may extract bounded nested tree&#10;structures, without learning a systematic recursive rule.&#10;" />
          <attvalue for="2" value="&#10;The syntactic organization of human languages has been proposed to follow a nested-tree organization \cite{chomsky:1957}. Nested trees necessitate a recursive operator, i.e. repeatedly applying a function over its own results. Consequently, recursive processing has been hypothesized to be at the core of the unique linguistic abilities of humans, possibly unique to humans, and is yet unidentified by human electrophysiology \cite{Hauser:etal:2002, Dehaene:etal:2015}. In recent years, Recurrent Neural Networks (RNNs) trained on large natural-language corpora have shown tremendous advances on a variety of NLP tasks, including word prediction (aka, language modeling) and sentence translation. Despite substantial differences between the human brain and artificial neural networks, the remarkable performances of modern RNNs on such tasks make them compelling objects for the study of recursive processing in neural devices akin to the human brain \cite[see, e.g., ][for early studies]{christiansen1999toward}.&#10;&#10;The notion of recursion was developed in the study of human linguistic knowledge. If a certain construction can be generated from a given grammar by an application of a rule, then a repeated application of the same rule could generate acceptable strings of an arbitrarily complexity. In contrast, it is empirically established that human linguistic processing is tightly limited, due to limitations such as memory capacity or attention span. This apparent inconsistency between the unbounded property of natural language and the tightly limited processing capacity of humans is commonly reconciled by drawing a distinction between human linguistic competence and performance. The former refers to the theoretical `ideal' knowledge of natural language, and is the object of linguistic inquiries, whereas the latter refers to the unfolding of this knowledge through parsing processes, whose operations incur a certain `cost' each. Such costs are commonly studied in psycholinguistics in behavioral experiment by measuring human accuracy and reaction times. &#10;&#10;In RNNs, the learned rules of the language are encoded in the network in a way that is directly related to the way the network applies them during sentence processing. Representations of abstract linguistic knowledge (construed here as `network competence'), and their unfolding in time during sentence processing (`network performance') could possibly be jointly studied in the network. Here, we explore the capacity of modern RNNs to learn to represent abstract rules of a recursive grammar. We study (1) the `behavioral' performance of RNNs, by evaluating model accuracy in processing nested structures sampled from the grammar, and (2) the inner representations learned by the model, by conducing an in-depth analysis into the way an RNN encodes underlying grammatical knowledge. &#10;&#10;Specifically, we focus on the capacity of RNN-based language models to learn artificial grammars with nested long-distance feature agreements. Feature agreement is central to our study since it allows to study recursive processing in RNNs given a linear order of words. Previous studies confirmed that RNN trained on natural data can successfully perform challenging long-range agreement between subject and verb \cite{Linzen:etal:2016, Bernardy:Lappin:2017, Gulordava:etal:2018, lakretz2019emergence}. However, several questions regarding recursive processing in RNNs remain unanswered: (1) it remains unclear whether RNNs learn to perform recursive processing over their input, akin to what argued for human language processing; (2) how do representations and mechanisms learned by RNNs affected by the statistics of the data? For example, if during training an RNN is presented with high occurrence of nested recursive structures (with, say, deeper constructions than those found in natural language), will it favor developing recursive mechanisms? (3) Can a structural bias towards learning recursive grammars (e.g., memory-augmented models) improve RNN performance on nested constructions?&#10;&#10;To address the above questions, we introduce a setup that is simple enough to control for various aspects of the training data, which are otherwise hard to explore in a natural-data setup, while preserving a higher degree of similarity to natural data compared to previous studies on simple artificial languages. We test and compare the performance of a variety of RNN models, with and without structural bias. We found that RNNs do not genuinely capture the underlying recursive grammar and do not truly generalize to deeper structures, importantly, neither RNNs with a structural bias towards learning hierarchical data. However, RNNs do succeed in generalizing to longer subject-verb dependencies for a given depth. An analysis of the generalization patterns of the networks revealed primacy and recency effects, consistent with recent findings on the distinction between short- and long-range number units identified in RNN language models \cite{lakretz2019emergence}; Finally, we describe the dynamics of the inner states of one such long-range unit and its complex encoding of multiple grammatical numbers.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Processing, Computer Science, Linguistics, Cognitive Science, Neural Networks, Neural Network Analysis, Recursive Structure Representation" />
        </attvalues>
      </node>
      <node id="2402.08382" label="2402.08382">
        <attvalues>
          <attvalue for="0" value="Punctuation Restoration Improves Structure Understanding without&#10;  Supervision" />
          <attvalue for="1" value="  Unsupervised learning objectives like language modeling and de-noising&#10;constitute a significant part in producing pre-trained models that perform&#10;various downstream applications from natural language understanding to&#10;conversational tasks. However, despite impressive generative capabilities of&#10;recent large language models, their abilities to capture syntactic or semantic&#10;structure within text lag behind. We hypothesize that the mismatch between&#10;linguistic performance and competence in machines is attributable to&#10;insufficient transfer of linguistic structure knowledge to computational&#10;systems with currently popular pre-training objectives. We show that&#10;punctuation restoration as a learning objective improves in- and&#10;out-of-distribution performance on structure-related tasks like named entity&#10;recognition, open information extraction, chunking, and part-of-speech tagging.&#10;Punctuation restoration is an effective learning objective that can improve&#10;structure understanding and yield a more robust structure-aware representations&#10;of natural language.&#10;" />
          <attvalue for="2" value="&#10;&#10;The current framework of natural language processing systems, described by \cite{linzen-2020-accelerate} as the PAID paradigm, consists of two production stages: unsupervised representation learning and task-specific engineering. Modern transformer-based systems that follow the framework \cite{devlin2019bert, raffel2019t5, radford2018gpt, peters-etal-2018-elmo} report high performance in various natural language understanding tasks, often matching or exceeding human performance baselines \cite{wang-etal-2018-glue, wang2019superglue}. However, there is ample evidence that current unsupervised representation learning yields weak structure understanding and brittle generalization abilities. In classification systems, we observe unstable outcome despite consistent input and reliance on shallow heuristics while processing unfamiliar input \cite{mccoy-etal-2020-berts, zhou-etal-2020-curse}. In generative and conversational systems, we observe stagnant natural language understanding performance despite drastic increase in conversational performance \cite{zhong2023chatgpt}, and failure to generalize sentences like &quot;A equals B&quot; to &quot;B equals A&quot; \cite{berglund2023reversal}.&#10;&#10;These are examples of weak structure understanding in language model based NLP systems. While it is difficult to pinpoint the exact source of these weaknesses or even disentangle between the effects of unsupervised pre-training and task specific engineering, the pre-training stage is at least partially attributable for these behaviors, and there exists room for improvement \cite{zhou-etal-2020-curse, min-etal-2020-syntactic}. We believe word prediction tasks like auto-regressive \cite{radford2018gpt}, masked \cite{devlin2019bert}, and perturbed \cite{raffel2019t5} language modeling may be insufficient to acquire robust representations that contain strong understanding of syntactic and semantic structure. We hypothesize that an additional unsupervised learning objective that focuses on capturing structure within natural language will improve structure understanding, measured by in-distribution (test set from the same source as training set) and out-of-distribution (test set from a different source than training set) performance in structure-related NLP tasks like chunking, information extraction, semantic role labeling, named entity recognition, sentence boundary detection, and part-of-speech tagging.&#10;&#10;This paper aims to test this hypothesis, using an unsupervised learning objective that reinforces structure understanding in language models. One nontrivial signal for syntactic and semantic structure in natural language is punctuation \cite{briscoe1996syntax, nunberg-1990-linguistics, dale1991exploring}, which can also be an effective parsing constraint that aids grammar induction in web mark-up text \cite{spitkovsky-etal-2010-profiting}. During human speech processing, syntactic disambiguation and grammar induction are facilitated by prosody \cite{kahn2005effective, price1991use}, which is analogous to punctuation in written text. Previously, punctuation has been used for grammar induction to improve unsupervised dependency parsing \cite{spitkovsky-etal-2011-punctuation}. Punctuation restoration is itself also a popular downstream task, especially for polishing output text from automatic speech recognition systems \cite[inter alia]{gravano-2009-restoring, alam2020punctuation, gupta2023punctuation} but has not been studied as a transferable language modeling objective.&#10;&#10;Here, we propose punctuation restoration as the structure-oriented learning objective, which we describe in detail in Section \ref{sec:objective_design}. Our results show additional pre-training with the punctuation restoration objective leads to improvements in various structure-related NLP task performance in both discriminative and generative approaches, supporting our hypothesis. Furthermore, this finding suggests that there is room for improvement in the unsupervised pre-training stage in the current paradigm of producing natural language processing systems.&#10;&#10;Our contribution is twofold:&#10;\begin{enumerate}&#10; \item We suggest a novel research direction in unsupervised transfer learning beyond word prediction&#10; \item We propose an unsupervised learning objective that yields robust structure understanding&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Natural Language Understanding, Artificial Intelligence, Natural Language Processing, Language Modeling, Machine Learning Objectives" />
        </attvalues>
      </node>
      <node id="2004.13606" label="2004.13606">
        <attvalues>
          <attvalue for="0" value="The Curse of Performance Instability in Analysis Datasets: Consequences,&#10;  Source, and Suggestions" />
          <attvalue for="1" value="  We find that the performance of state-of-the-art models on Natural Language&#10;Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be&#10;highly unstable. This raises three questions: (1) How will the instability&#10;affect the reliability of the conclusions drawn based on these analysis sets?&#10;(2) Where does this instability come from? (3) How should we handle this&#10;instability and what are some potential solutions? For the first question, we&#10;conduct a thorough empirical study over analysis sets and find that in addition&#10;to the unstable final performance, the instability exists all along the&#10;training curve. We also observe lower-than-expected correlations between the&#10;analysis validation set and standard validation set, questioning the&#10;effectiveness of the current model-selection routine. Next, to answer the&#10;second question, we give both theoretical explanations and empirical evidence&#10;regarding the source of the instability, demonstrating that the instability&#10;mainly comes from high inter-example correlations within analysis sets.&#10;Finally, for the third question, we discuss an initial attempt to mitigate the&#10;instability and suggest guidelines for future work such as reporting the&#10;decomposed variance for more interpretable results and fair comparison across&#10;models. Our code is publicly available at:&#10;https://github.com/owenzx/InstabilityAnalysis&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.01425" label="1708.01425">
        <attvalues>
          <attvalue for="0" value="The Argument Reasoning Comprehension Task: Identification and&#10;  Reconstruction of Implicit Warrants" />
          <attvalue for="1" value="  Reasoning is a crucial part of natural language argumentation. To comprehend&#10;an argument, one must analyze its warrant, which explains why its claim follows&#10;from its premises. As arguments are highly contextualized, warrants are usually&#10;presupposed and left implicit. Thus, the comprehension does not only require&#10;language understanding and logic skills, but also depends on common sense. In&#10;this paper we develop a methodology for reconstructing warrants systematically.&#10;We operationalize it in a scalable crowdsourcing process, resulting in a freely&#10;licensed dataset with warrants for 2k authentic arguments from news comments.&#10;On this basis, we present a new challenging task, the argument reasoning&#10;comprehension task. Given an argument with a claim and a premise, the goal is&#10;to choose the correct implicit warrant from two options. Both warrants are&#10;plausible and lexically close, but lead to contradicting claims. A solution to&#10;this task will define a substantial step towards automatic warrant&#10;reconstruction. However, experiments with several neural attention and language&#10;models reveal that current approaches do not suffice.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.03289" label="2004.03289">
        <attvalues>
          <attvalue for="0" value="KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language&#10;  Understanding" />
          <attvalue for="1" value="  Natural language inference (NLI) and semantic textual similarity (STS) are&#10;key tasks in natural language understanding (NLU). Although several benchmark&#10;datasets for those tasks have been released in English and a few other&#10;languages, there are no publicly available NLI or STS datasets in the Korean&#10;language. Motivated by this, we construct and release new datasets for Korean&#10;NLI and STS, dubbed KorNLI and KorSTS, respectively. Following previous&#10;approaches, we machine-translate existing English training sets and manually&#10;translate development and test sets into Korean. To accelerate research on&#10;Korean NLU, we also establish baselines on KorNLI and KorSTS. Our datasets are&#10;publicly available at https://github.com/kakaobrain/KorNLUDatasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.00545" label="2008.00545">
        <attvalues>
          <attvalue for="0" value="Cross-Domain Adaptation of Spoken Language Identification for Related&#10;  Languages: The Curious Case of Slavic Languages" />
          <attvalue for="1" value="  State-of-the-art spoken language identification (LID) systems, which are&#10;based on end-to-end deep neural networks, have shown remarkable success not&#10;only in discriminating between distant languages but also between&#10;closely-related languages or even different spoken varieties of the same&#10;language. However, it is still unclear to what extent neural LID models&#10;generalize to speech samples with different acoustic conditions due to domain&#10;shift. In this paper, we present a set of experiments to investigate the impact&#10;of domain mismatch on the performance of neural LID systems for a subset of six&#10;Slavic languages across two domains (read speech and radio broadcast) and&#10;examine two low-level signal descriptors (spectral and cepstral features) for&#10;this task. Our experiments show that (1) out-of-domain speech samples severely&#10;hinder the performance of neural LID models, and (2) while both spectral and&#10;cepstral features show comparable performance within-domain, spectral features&#10;show more robustness under domain mismatch. Moreover, we apply unsupervised&#10;domain adaptation to minimize the discrepancy between the two domains in our&#10;study. We achieve relative accuracy improvements that range from 9% to 77%&#10;depending on the diversity of acoustic conditions in the source domain.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken language identification, henceforth LID, is the problem of determining the identity of the language in a spoken utterance \cite{li2013spoken}. In today's globalized world, LID systems can facilitate a wide range of cross-lingual speech and communication technologies such as spoken language translation \cite{waibel2000multilinguality, fugen2007simultaneous, bangalore2012real} and multilingual spoken document retrieval \cite{chelba2008retrieval}. Furthermore, LID-aware transfer of language resources has been shown to be effective for multilingual ASR in low-resource settings \cite{muller2016language, muller2015using, nguyen2014multilingual, cutler2014language}. &#10;&#10;Earlier work has addressed the LID task using the so-called phonotactic approach. In this paradigm, the acoustic signal is first transduced into a sequence of discrete symbols (e.g., phones), then probabilistic models are utilized to obtain language likelihoods \cite{lamel1994language, li2005phonotactic}. This approach has been outperformed by acoustic approaches that are based on Gaussian Mixture Models (GMMs) and the i-vector framework which has been applied to speaker and language identification \cite{kenny2010bayesian, garcia2011analysis, martinez2011language, su2016factor}. Currently, end-to-end deep neural networks (DNNs) are predominant for LID and outperform GMMs, especially for short utterances \cite{mateju2018using, shen2018feature, shon2018convolutional, lopez2014automatic, gonzalez2014automatic}. &#10;&#10;The findings of the popular language guessing game, the Great Language Game \cite{skirgaard2017some}, have shown that discriminating between closely-related languages is a difficult task for humans. On the other hand, neural LID models have shown striking performance discriminating between spoken varieties of Arabic \cite{bulut2017utd, gelly2016language, shon2018convolutional}, Slavic languages \cite{mateju2018using}, and languages in accented speech samples from multilingual speakers \cite{titus2020improving}. For instance, the best neural LID model in \cite{mateju2018using} has reported an error rate as low as 1.2\% when discriminating between 11 Slavic languages. Generally speaking, the impressive performance of DNN-based LID reported in the literature gives the impression that LID is almost a solved problem.&#10;&#10;However, previous works have developed their models using disjoint splits of the same dataset where the training and evaluation samples have similar, if not identical, acoustic conditions (i.e., same domain). The impact of dataset-bias \cite{tzeng2017adversarial} on LID robustness has not yet been investigated with a systematic evaluation across datasets. In this paper we aim to fill this gap and focus on the challenging case of LID for short utterances of related languages (i.e., Slavic languages) in a cross-domain setting. We investigate the following questions:&#10;&#10;\begin{itemize}&#10; \item RQ1 To what degree do neural LID models for related languages generalize to another domain with different acoustic conditions?&#10; \item RQ2 Are different low-level speech features equally robust under domain mismatch?&#10; \item RQ3 Can we adapt LID models to a new domain without using labelled data in the new domain? If yes, what are the factors that affect the adaptability of the model?&#10;\end{itemize}&#10;&#10;To address these research questions, we conduct a series of LID experiments with datasets from two domains: (1) Read speech recordings from the Slavic subset of the GlobalPhone speech database \cite{schultz2013globalphone}, and (2) Slavic broadcast recordings collected and distributed in \cite{mateju2018using,nouza2016asr} for LID (RQ1). We also compare the performance of spectral (MFSCs) and cepstral (MFCCs) speech features within- and across-domain (RQ2). Finally, we apply adversarial domain confusion \cite{ganin2015unsupervised} to adapt our model to a target domain, analyze predictions from the adapted model, and visualize its representations compared to the baseline (RQ3).. \\&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Language Identification, Domain Adaptation, Signal Processing, Neural Network Performance" />
        </attvalues>
      </node>
      <node id="1702.05464" label="1702.05464">
        <attvalues>
          <attvalue for="0" value="Adversarial Discriminative Domain Adaptation" />
          <attvalue for="1" value="  Adversarial learning methods are a promising approach to training robust deep&#10;networks, and can generate complex samples across diverse domains. They also&#10;can improve recognition despite the presence of domain shift or dataset bias:&#10;several adversarial approaches to unsupervised domain adaptation have recently&#10;been introduced, which reduce the difference between the training and test&#10;domain distributions and thus improve generalization performance. Prior&#10;generative approaches show compelling visualizations, but are not optimal on&#10;discriminative tasks and can be limited to smaller shifts. Prior discriminative&#10;approaches could handle larger domain shifts, but imposed tied weights on the&#10;model and did not exploit a GAN-based loss. We first outline a novel&#10;generalized framework for adversarial adaptation, which subsumes recent&#10;state-of-the-art approaches as special cases, and we use this generalized view&#10;to better relate the prior approaches. We propose a previously unexplored&#10;instance of our general framework which combines discriminative modeling,&#10;untied weight sharing, and a GAN loss, which we call Adversarial Discriminative&#10;Domain Adaptation (ADDA). We show that ADDA is more effective yet considerably&#10;simpler than competing domain-adversarial methods, and demonstrate the promise&#10;of our approach by exceeding state-of-the-art unsupervised adaptation results&#10;on standard cross-domain digit classification tasks and a new more difficult&#10;cross-modality object classification task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18074" label="2305.18074">
        <attvalues>
          <attvalue for="0" value="An Experimental Review of Speaker Diarization methods with application&#10;  to Two-Speaker Conversational Telephone Speech recordings" />
          <attvalue for="1" value="  We performed an experimental review of current diarization systems for the&#10;conversational telephone speech (CTS) domain. In detail, we considered a total&#10;of eight different algorithms belonging to clustering-based, end-to-end neural&#10;diarization (EEND), and speech separation guided diarization (SSGD) paradigms.&#10;We studied the inference-time computational requirements and diarization&#10;accuracy on four CTS datasets with different characteristics and languages. We&#10;found that, among all methods considered, EEND-vector clustering (EEND-VC)&#10;offers the best trade-off in terms of computing requirements and performance.&#10;More in general, EEND models have been found to be lighter and faster in&#10;inference compared to clustering-based methods. However, they also require a&#10;large amount of diarization-oriented annotated data. In particular EEND-VC&#10;performance in our experiments degraded when the dataset size was reduced,&#10;whereas self-attentive EEND (SA-EEND) was less affected. We also found that&#10;SA-EEND gives less consistent results among all the datasets compared to&#10;EEND-VC, with its performance degrading on long conversations with high speech&#10;sparsity. Clustering-based diarization systems, and in particular VBx, instead&#10;have more consistent performance compared to SA-EEND but are outperformed by&#10;EEND-VC. The gap with respect to this latter is reduced when overlap-aware&#10;clustering methods are considered. SSGD is the most computationally demanding&#10;method, but it could be convenient if speech recognition has to be performed.&#10;Its performance is close to SA-EEND but degrades significantly when the&#10;training and inference data characteristics are less matched.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speaker diarization, also often referred to as simply diarization, aims at partitioning an audio recording into temporal segments denoting the boundaries of each speaker's utterances. In other words, it addresses the problem of ``who spoke when?'', without any a-priori knowledge of the speakers' identities. &#10;This discipline represents an important research field in the speech processing area. Its main applications involve ASR, speaker indexing, speaker recognition, real-time captioning, and audio analysis to name a few. &#10;Speaker diarization constitutes an important and often essential pre-processing step in most of these application scenarios: e.g., accurate diarization can be used effectively to drive multi-channel blind source separation algorithms to separate concurrent speakers for distant speech recognition \cite{boeddeker2018front, kanda2019guided} or speaker adaptation for ASR~\cite{saon2013speaker, miao2015speaker, wang2017unsupervised, sari2020unsupervised}.&#10;It is also a very challenging task as speaker identity is arguably not easy to estimate especially when many speakers are present and they often overlap. In such scenarios, even humans struggle and have to resort to linguistic information from neighboring utterances or long term information to resolve speaker identities' ambiguities on short segments. &#10;&#10;As such, it can be argued that the two-speaker conversation case is the simplest scenario when all the other characteristics that can impact diarization performance are equal: e.g., audio quality due to the registration setup, environmental noise, duration and sequence of speakers' turns, as well as the style and spontaneity of speech.&#10;On the other hand, the two-speaker scenario constitutes a particularly important use-case as it is a very common scenario and has a great commercial interest. &#10;In fact, it represents the most ordinary situation in normal conversations that take place via telephone or remotely via hands-free tele-conferencing, but it also appears in many other contexts, such as in doctor-patient meetings, interviews, air traffic control conversations, et cetera.&#10;&#10;In light of the tremendous progress made in recent years, we present in this work an experimental review focused on CTS, and compare current state-of-the-art methods based on clustering, separation, and end-to-end diarization on real-world CTS datasets with different languages. &#10;The main motivation is that recent research trends, such as the invention of fully end-to-end diarization methods \cite{fujita2019end-blstm, fujita2019end-self, fujita2020end, kinoshita2021integrating, kinoshita2021advances, kinoshita2022utterance, kinoshita2022tight}, seems to suggest that the technology is mature enough to attain highly accurate and reliable diarization in the CTS scenario, whereas other scenarios, such as distant-talk meetings with several speakers, still pose several challenges \cite{watanabe2020chime}. Our hope is that this work will be useful to researchers and practitioners alike, as we investigate computational requirements and the performance of eight diarization algorithms on multilingual data, all belonging to the CTS domain but with diverse characteristics, such as overlapped speech ratio, average length of the conversation, and speech sparsity.&#10;&#10;The remainder of the paper is organized as follows: we present in Section \ref{sec:history} and \ref{sec:motivation} a brief overview of speaker diarization methods and a comparison of this review work with previous ones.&#10;Section \ref{sec:section_2} describes the various systems that have been considered in our experimental analysis, along with their main characteristics. In Section \ref{sec:section_3} we outline the experimental setup and describe the systems implementation and hyper-parameters employed in our experiments, as well as the diarization metrics and the CTS datasets used for training and testing.&#10;In Section \ref{sec:results_discussion} we analyze the results.&#10;Finally, in Section \ref{sec:conclusion} we draw conclusions and discuss possible future research directions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Network Models, Conversational Analysis, Engineering, Speech Diarization, Signal Processing" />
        </attvalues>
      </node>
      <node id="2309.00949" label="2309.00949">
        <attvalues>
          <attvalue for="0" value="Multilingual Text Representation" />
          <attvalue for="1" value="  Modern NLP breakthrough includes large multilingual models capable of&#10;performing tasks across more than 100 languages. State-of-the-art language&#10;models came a long way, starting from the simple one-hot representation of&#10;words capable of performing tasks like natural language understanding,&#10;common-sense reasoning, or question-answering, thus capturing both the syntax&#10;and semantics of texts. At the same time, language models are expanding beyond&#10;our known language boundary, even competitively performing over very&#10;low-resource dialects of endangered languages. However, there are still&#10;problems to solve to ensure an equitable representation of texts through a&#10;unified modeling space across language and speakers. In this survey, we shed&#10;light on this iterative progression of multilingual text representation and&#10;discuss the driving factors that ultimately led to the current&#10;state-of-the-art. Subsequently, we discuss how the full potential of language&#10;democratization could be obtained, reaching beyond the known limits and what is&#10;the scope of improvement in that space.&#10;" />
          <attvalue for="2" value="&#10;Natural language processing (NLP) primarily involves making linguistic-specific applications for machines to understand language. Earlier days of NLP development mainly focused on the idea of distributional hypothesis~\cite{harris54} that is, &quot;words occurring in the same context tend to have a similar meaning or closely related meaning&quot;. From there, NLP has come a long way in the modeling language. The tasks NLP tries to solve are complex and multidimensional if we put them into the perspective of the machine and numerical mapping. For example, there are multiple dynamics to deal with here, like various languages and dialects to consider and tasks to solve with different levels of granularity. Combining all these dynamics in a unified representation space is a complex problem. The base starting point could be just words, as the word is one such unit of language that is quite universal. Following this thought, the most straightforward idea could be to compute the word frequency, thus constructing a count-based numerical mapping that we can think of as a computationally feasible language representation. N-gram language models are representations that have led the domain of NLP for a substantial time. Later on, advancing over the distributional scheme of language representation coupled with the availability of huge computation resources, neural models became the go-to approach for all kinds of NLP tasks. Researchers investigated different theories and directions in this domain of neural language modeling before transformer-based neural networks revolutionized NLP. The transformer-based model provides exceptional text representation utilizing the multi-layered encoder blocks~\cite{ This is useful as text gets different meanings based on how it is used in a context. In addition, multiple languages can share a single representation space using transformers. This effectively led to the path for multilingual text representation, where data collected from languages existing all over the world can be accumulated in a single setting, and models can learn and perform actionable inference on a wide variety of tasks comprising language and dialectal varieties. Though a monolingual or region-focused transformer still vastly outperforms a more generalized multilingual model on most tasks, it is not always feasible to train multiple versions of the domain-focused model. The idea is to make a shared representation space that effectively works for many languages, while the resource scarcity of specific languages should benefit from other high-resource languages. mBERT~\cite{devlin-etal-2019-bert} and xlm-r~\cite{conneau-etal-2020-unsupervised} trained with multiple objective functions on more than hundreds of languages came a long way in achieving this vision. However, the full potential of a unified multilingual text representation is still a significant research problem to solve. Because, quite regularly, the inclusion of new tasks and languages in the modeling paradigm points out the fact that, when these models move beyond the monolingual scheme, the total capacity of the model gets distributed across languages, thus often resulting in capacity dilution~\cite{wang-etal-2020-negative}. An ideal scenario would be to have no amount of negative interference, such that we get an equitable performance across languages. Another important direction for multilingual models is to ensure the easy-expand-ability to new languages and adaptability to new tasks. &#10;&#10;Keeping all these advanced development of multilingual text representation in context, in this survey, we provide insight into the open problems and questions to look for. In addition, we discuss how the text representation model starting from the count-based vector representation of words, came to the point of a multilingual text representation model capable of performing across more than 100 languages. We structure the contents based on the following contexts: (1) How did the text representation model make the iterative progress, and which were the driving factors in each step? (2) What are the primary building blocks of a unified multilingual text representation model, and how do they vary given the difference in scenarios? (3) What are the current barriers that limit the progression of full-scale multilingual text representation? (4) The fairness and interpretability of currently available models and how equitable they serve the intended user’s utility.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Multilingual Text Representation, Computer Science, Linguistics, Language Democratization, Artificial Intelligence, Language Model Development, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2002.00417" label="2002.00417">
        <attvalues>
          <attvalue for="0" value="WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss" />
          <attvalue for="1" value="  Tacotron-based text-to-speech (TTS) systems directly synthesize speech from&#10;text input. Such frameworks typically consist of a feature prediction network&#10;that maps character sequences to frequency-domain acoustic features, followed&#10;by a waveform reconstruction algorithm or a neural vocoder that generates the&#10;time-domain waveform from acoustic features. As the loss function is usually&#10;calculated only for frequency-domain acoustic features, that doesn't directly&#10;control the quality of the generated time-domain waveform. To address this&#10;problem, we propose a new training scheme for Tacotron-based TTS, referred to&#10;as WaveTTS, that has 2 loss functions: 1) time-domain loss, denoted as the&#10;waveform loss, that measures the distortion between the natural and generated&#10;waveform; and 2) frequency-domain loss, that measures the Mel-scale acoustic&#10;feature loss between the natural and generated acoustic features. WaveTTS&#10;ensures both the quality of the acoustic features and the resulting speech&#10;waveform. To our best knowledge, this is the first implementation of Tacotron&#10;with joint time-frequency domain loss. Experimental results show that the&#10;proposed framework outperforms the baselines and achieves high-quality&#10;synthesized speech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.00771" label="1904.00771">
        <attvalues>
          <attvalue for="0" value="Training Multi-Speaker Neural Text-to-Speech Systems using&#10;  Speaker-Imbalanced Speech Corpora" />
          <attvalue for="1" value="  When the available data of a target speaker is insufficient to train a high&#10;quality speaker-dependent neural text-to-speech (TTS) system, we can combine&#10;data from multiple speakers and train a multi-speaker TTS model instead. Many&#10;studies have shown that neural multi-speaker TTS model trained with a small&#10;amount data from multiple speakers combined can generate synthetic speech with&#10;better quality and stability than a speaker-dependent one. However when the&#10;amount of data from each speaker is highly unbalanced, the best approach to&#10;make use of the excessive data remains unknown. Our experiments showed that&#10;simply combining all available data from every speaker to train a multi-speaker&#10;model produces better than or at least similar performance to its&#10;speaker-dependent counterpart. Moreover by using an ensemble multi-speaker&#10;model, in which each subsystem is trained on a subset of available data, we can&#10;further improve the quality of the synthetic speech especially for&#10;underrepresented speakers whose training data is limited.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.02839" label="1911.02839">
        <attvalues>
          <attvalue for="0" value="Teacher-Student Training for Robust Tacotron-based TTS" />
          <attvalue for="1" value="  While neural end-to-end text-to-speech (TTS) is superior to conventional&#10;statistical methods in many ways, the exposure bias problem in the&#10;autoregressive models remains an issue to be resolved. The exposure bias&#10;problem arises from the mismatch between the training and inference process,&#10;that results in unpredictable performance for out-of-domain test data at&#10;run-time. To overcome this, we propose a teacher-student training scheme for&#10;Tacotron-based TTS by introducing a distillation loss function in addition to&#10;the feature loss function. We first train a Tacotron2-based TTS model by always&#10;providing natural speech frames to the decoder, that serves as a teacher model.&#10;We then train another Tacotron2-based model as a student model, of which the&#10;decoder takes the predicted speech frames as input, similar to how the decoder&#10;works during run-time inference. With the distillation loss, the student model&#10;learns the output probabilities from the teacher model, that is called&#10;knowledge distillation. Experiments show that our proposed training scheme&#10;consistently improves the voice quality for out-of-domain test data both in&#10;Chinese and English systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.07884" label="2005.07884">
        <attvalues>
          <attvalue for="0" value="Improved Prosody from Learned F0 Codebook Representations for VQ-VAE&#10;  Speech Waveform Reconstruction" />
          <attvalue for="1" value="  Vector Quantized Variational AutoEncoders (VQ-VAE) are a powerful&#10;representation learning framework that can discover discrete groups of features&#10;from a speech signal without supervision. Until now, the VQ-VAE architecture&#10;has previously modeled individual types of speech features, such as only phones&#10;or only F0. This paper introduces an important extension to VQ-VAE for learning&#10;F0-related suprasegmental information simultaneously along with traditional&#10;phone features.The proposed framework uses two encoders such that the F0&#10;trajectory and speech waveform are both input to the system, therefore two&#10;separate codebooks are learned. We used a WaveRNN vocoder as the decoder&#10;component of VQ-VAE. Our speaker-independent VQ-VAE was trained with raw speech&#10;waveforms from multi-speaker Japanese speech databases. Experimental results&#10;show that the proposed extension reduces F0 distortion of reconstructed speech&#10;for all unseen test speakers, and results in significantly higher preference&#10;scores from a listening test. We additionally conducted experiments using&#10;single-speaker Mandarin speech to demonstrate advantages of our architecture in&#10;another language which relies heavily on F0.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.06686" label="2003.06686">
        <attvalues>
          <attvalue for="0" value="Perception of prosodic variation for speech synthesis using an&#10;  unsupervised discrete representation of F0" />
          <attvalue for="1" value="  In English, prosody adds a broad range of information to segment sequences,&#10;from information structure (e.g. contrast) to stylistic variation (e.g.&#10;expression of emotion). However, when learning to control prosody in&#10;text-to-speech voices, it is not clear what exactly the control is modifying.&#10;Existing research on discrete representation learning for prosody has&#10;demonstrated high naturalness, but no analysis has been performed on what these&#10;representations capture, or if they can generate meaningfully-distinct variants&#10;of an utterance. We present a phrase-level variational autoencoder with a&#10;multi-modal prior, using the mode centres as &quot;intonation codes&quot;. Our evaluation&#10;establishes which intonation codes are perceptually distinct, finding that the&#10;intonation codes from our multi-modal latent model were significantly more&#10;distinct than a baseline using k-means clustering. We carry out a follow-up&#10;qualitative study to determine what information the codes are carrying. Most&#10;commonly, listeners commented on the intonation codes having a statement or&#10;question style. However, many other affect-related styles were also reported,&#10;including: emotional, uncertain, surprised, sarcastic, passive aggressive, and&#10;upset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.01410" label="1808.01410">
        <attvalues>
          <attvalue for="0" value="Predicting Expressive Speaking Style From Text In End-To-End Speech&#10;  Synthesis" />
          <attvalue for="1" value="  Global Style Tokens (GSTs) are a recently-proposed method to learn latent&#10;disentangled representations of high-dimensional data. GSTs can be used within&#10;Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to&#10;uncover expressive factors of variation in speaking style. In this work, we&#10;introduce the Text-Predicted Global Style Token (TP-GST) architecture, which&#10;treats GST combination weights or style embeddings as &quot;virtual&quot; speaking style&#10;labels within Tacotron. TP-GST learns to predict stylistic renderings from text&#10;alone, requiring neither explicit labels during training nor auxiliary inputs&#10;for inference. We show that, when trained on a dataset of expressive speech,&#10;our system generates audio with more pitch and energy variation than two&#10;state-of-the-art baseline models. We further demonstrate that TP-GSTs can&#10;synthesize speech with background noise removed, and corroborate these analyses&#10;with positive results on human-rated listener preference audiobook tasks.&#10;Finally, we demonstrate that multi-speaker TP-GST models successfully factorize&#10;speaker identity and speaking style. We provide a website with audio samples&#10;for each of our findings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1603.08155" label="1603.08155">
        <attvalues>
          <attvalue for="0" value="Perceptual Losses for Real-Time Style Transfer and Super-Resolution" />
          <attvalue for="1" value="  We consider image transformation problems, where an input image is&#10;transformed into an output image. Recent methods for such problems typically&#10;train feed-forward convolutional neural networks using a \emph{per-pixel} loss&#10;between the output and ground-truth images. Parallel work has shown that&#10;high-quality images can be generated by defining and optimizing&#10;\emph{perceptual} loss functions based on high-level features extracted from&#10;pretrained networks. We combine the benefits of both approaches, and propose&#10;the use of perceptual loss functions for training feed-forward networks for&#10;image transformation tasks. We show results on image style transfer, where a&#10;feed-forward network is trained to solve the optimization problem proposed by&#10;Gatys et al in real-time. Compared to the optimization-based method, our&#10;network gives similar qualitative results but is three orders of magnitude&#10;faster. We also experiment with single-image super-resolution, where replacing&#10;a per-pixel loss with a perceptual loss gives visually pleasing results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.09405" label="1707.09405">
        <attvalues>
          <attvalue for="0" value="Photographic Image Synthesis with Cascaded Refinement Networks" />
          <attvalue for="1" value="  We present an approach to synthesizing photographic images conditioned on&#10;semantic layouts. Given a semantic label map, our approach produces an image&#10;with photographic appearance that conforms to the input layout. The approach&#10;thus functions as a rendering engine that takes a two-dimensional semantic&#10;specification of the scene and produces a corresponding photographic image.&#10;Unlike recent and contemporaneous work, our approach does not rely on&#10;adversarial training. We show that photographic images can be synthesized from&#10;semantic layouts by a single feedforward network with appropriate structure,&#10;trained end-to-end with a direct regression objective. The presented approach&#10;scales seamlessly to high resolutions; we demonstrate this by synthesizing&#10;photographic images at 2-megapixel resolution, the full resolution of our&#10;training data. Extensive perceptual experiments on datasets of outdoor and&#10;indoor scenes demonstrate that images synthesized by the presented approach are&#10;considerably more realistic than alternative approaches. The results are shown&#10;in the supplementary video at https://youtu.be/0fhUJT21-bs&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.07284" label="2401.07284">
        <attvalues>
          <attvalue for="0" value="Improving Domain Adaptation through Extended-Text Reading Comprehension" />
          <attvalue for="1" value="  To enhance the domain-specific capabilities of large language models,&#10;continued pre-training on a domain-specific corpus is a prevalent method.&#10;Recent work demonstrates that adapting models using reading comprehension data&#10;formatted by regex-based patterns can significantly improve performance on&#10;domain-specific tasks. However, regex-based patterns are incapable of parsing&#10;raw corpora using domain-specific knowledge. Furthermore, the question and&#10;answer pairs are extracted directly from the corpus in predefined formats&#10;offers limited context. To address this limitation, we improve reading&#10;comprehension via LLM and clustering. LLM focuses on leveraging domain&#10;knowledge within the corpus to refine comprehension stage, while clustering&#10;supplies relevant knowledge by extending the context to enrich reading stage.&#10;Additionally, our method incorporates parameter-efficient fine-tuning to&#10;improve the efficiency of domain adaptation. In comparison to AdaptLLM, our&#10;method achieves an improvement exceeding 5% in domain-specific tasks. Our code&#10;will available at https://github.com/microsoft/LMOps.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the emergence of Large Language Models (LLMs), LLMs have shown promising performance on various downstream tasks.&#10;A number of domain-specific LLMs~\cite{cheng2023adapting, chen2023disc, wu2023bloomberggpt, han2023medalpaca, liu2023chipnemo} have also been proposed to enhance LLMs on domain-specific capabilities of LLMs, which demonstrate improved performances in respective domains compared to general models.&#10;These domain-specific LLMs can be trained in two ways: either from scratch or by adapting existing general LLMs through continued pre-training~\cite{gururangan2020don}, with the latter being a more efficient method due to the foundational benefits provided by the general LLMs.&#10;&#10;Recent work~\cite{cheng2023adapting} reveals that straightforward adaptation of a general LLM using raw domain-specific corpus is ineffective and can even impair prompting ability on domain-specific tasks.&#10;To harness the potential of domain-specific knowledge, they proposed a data preprocessing method named AdaptLLM. This method transforms a corpus into a reading comprehension format, utilizing specially designed regex-based patterns.&#10;Consequently, AdaptLLM notably enhances the performance of domain-specific tasks by structuring a corpus in the question-answering format.&#10;&#10;However, the reliance on regex-based patterns poses challenges in handling complex patterns and generating questions that reflect domain-specific knowledge.&#10;For example, the regex-based pattern \{SENT1\} Therefore, \{SENT2\} is converted into a question-answer format as: What is the cause of \{SENT1\}? \{SENT2\}. This method also limits the diversity of question types.&#10;Integrating LLMs in the preprocessing stage can overcome these limitations. LLMs like ChatGPT are capable of identifying domain-specific knowledge and generating high-quality question-answer pairs for educational purposes~\cite{olney2023generating, lu2023can}.&#10;To mitigate the processing costs associated with ChatGPT in preprocessing, we fine-tune a compact LLM through knowledge distillation, to efficiently preprocesses domain-specific data.&#10;&#10;We find that the context of question answering can be too short to learn domain-specific knowledge comprehensively. For example, in biomedicine, each document is a short abstract of paper, which is easy for LLM to answer questions, but lacks enough context to learn domain-specific knowledge.&#10;Inspired by~\cite{levine2021inductive, shi2023context}, we leverage length-based clustering to extend the context by concatenating similar documents into the same input as context.&#10;Moreover, we improve the efficiency of domain adaptation by utilizing parameter-efficient fine-tuning methods like LoRA~\cite{hu2021lora}.&#10;Contrary to previous work~\cite{liu2023chipnemo}, we find that LoRA can be more efficient than full fine-tuning for domain adaptation with proper settings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Reading Comprehension Enhancement, Artificial Intelligence, Domain Knowledge Integration, Natural Language Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2311.00176" label="2311.00176">
        <attvalues>
          <attvalue for="0" value="ChipNeMo: Domain-Adapted LLMs for Chip Design" />
          <attvalue for="1" value="  ChipNeMo aims to explore the applications of large language models (LLMs) for&#10;industrial chip design. Instead of directly deploying off-the-shelf commercial&#10;or open-source LLMs, we instead adopt the following domain adaptation&#10;techniques: domain-adaptive tokenization, domain-adaptive continued&#10;pretraining, model alignment with domain-specific instructions, and&#10;domain-adapted retrieval models. We evaluate these methods on three selected&#10;LLM applications for chip design: an engineering assistant chatbot, EDA script&#10;generation, and bug summarization and analysis. Our evaluations demonstrate&#10;that domain-adaptive pretraining of language models, can lead to superior&#10;performance in domain related downstream tasks compared to their base LLaMA2&#10;counterparts, without degradations in generic capabilities. In particular, our&#10;largest model, ChipNeMo-70B, outperforms the highly capable GPT-4 on two of our&#10;use cases, namely engineering assistant chatbot and EDA scripts generation,&#10;while exhibiting competitive performance on bug summarization and analysis.&#10;These results underscore the potential of domain-specific customization for&#10;enhancing the effectiveness of large language models in specialized&#10;applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.09325" label="2311.09325">
        <attvalues>
          <attvalue for="0" value="Temperature-scaling surprisal estimates improve fit to human reading&#10;  times -- but does it do so for the &quot;right reasons&quot;?" />
          <attvalue for="1" value="  A wide body of evidence shows that human language processing difficulty is&#10;predicted by the information-theoretic measure surprisal, a word's negative log&#10;probability in context. However, it is still unclear how to best estimate these&#10;probabilities needed for predicting human processing difficulty -- while a&#10;long-standing belief held that models with lower perplexity would provide more&#10;accurate estimates of word predictability, and therefore lead to better reading&#10;time predictions, recent work has shown that for very large models,&#10;psycholinguistic predictive power decreases. One reason could be that language&#10;models might be more confident of their predictions than humans, because they&#10;have had exposure to several magnitudes more data. In this paper, we test what&#10;effect temperature-scaling of large language model (LLM) predictions has on&#10;surprisal estimates and their predictive power of reading times of English&#10;texts. Firstly, we show that calibration of large language models typically&#10;improves with model size, i.e. poorer calibration cannot account for poorer fit&#10;to reading times. Secondly, we find that temperature-scaling probabilities lead&#10;to a systematically better fit to reading times (up to 89% improvement in delta&#10;log likelihood), across several reading time corpora. Finally, we show that&#10;this improvement in fit is chiefly driven by words that are composed of&#10;multiple subword tokens.&#10;" />
          <attvalue for="2" value="&#10;&#10;In psycholinguistics, a key finding is that words with higher surprisal (= negative log probability of the word in context) require more time for processing~\cite{hale2001probabilistic, levy2008expectation}. &#10;Numerous studies provided experimental evidence supporting this theory, demonstrating that surprisal is a powerful predictive measure of processing complexity~\cite[e.g.,][]{demberg2008data, wilcox2020predictive, wilcox2023testing, shain2022large}, and that the relationship between surprisal and reading times (RTs) seems to be linear~\cite{smith2013effect, wilcox2020predictive, shain2022large}.&#10;&#10;However, prior work implicitly made the assumption that human predictability estimates would be similar to the actual probability of a word occurring in a given context, and that therefore, surprisal values estimated from models that achieve lower perplexities should also approximate human processing difficulty better \cite{goodkind2018predictive, merkx-frank-2021-human}. &#10;&#10;Recent research has however found that this is not true -- surprisal values from very large LLMs provide in fact a very poor fit to reading times. \cite{oh2023does} hypothesize that this might be due to LLMs being ``too confident'' in their estimates of rare named entities compared to humans, thanks to their manifold larger exposure to data and greater memory capacity compared to humans.&#10;Furthermore, work on NLP applications like question answering has reported that probability estimates from pretrained language models are often overconfident, i.e.~they are higher than the ground truth probability~\cite{si2022re, kumar2022answer}. These findings hence beg the question whether current LLMs are well-calibrated with respect to ``objective'' word occurrence probabilities. Relatedly, we ask whether LLM probability estimates are overconfident compared to human estimates (as observed in reading times). &#10;&#10;One approach to address calibration problems is to use temperature scaling, as done e.g., in vision tasks \cite{guo2017calibration, hendrycks2019using}. Temperature-scaling with a temperature $T&gt;1$ has the effect that the probability distribution is flattened such that it becomes more similar to a uniform distribution. Temperature-scaling hence incorporates uncertainty into the probability estimates from LLMs. &#10;&#10;We note that the idea to work with flattened distributions instead of the original probability distributions from LLMs is also related to contextual R\'enyi Entropy as discussed by \cite{pimentel-etal-2023-effect}, as well as the super/sub-linear surprisal effect by \cite{shain2022large,hoover2023plausibility}. However, rather than merely adjust the power of surprisal in super/sub-logarithmic patterns or the power of probability in R\'enyi entropy, our work represents a distinct branch of study (i.e., probability calibration) in machine learning: shaping the probability distribution itself through shaping the logits before softmax. &#10;We also discuss the motivation for why a slightly flattened distribution may be more suitable, and whether this change in distribution is applied when calculating surprisal vs.~when calculating entropy. &#10;&#10;Our experimental results show that scaling probabilities can largely improve the fit to reading times in all 12 settings (3 corpora $\times$ 4 neural LMs). &#10;Our contributions are summarized as follows: &#10;(1) We propose temperature-scaled surprisal, where surprisal is calculated from temperature-scaled probabilities. &#10;(2) We demonstrate that temperature-scaling with temperature T$\approx$2.5 improves predictability of human reading times of English texts compared to T=1.&#10;(3) We identify linguistic phenomena that correlate with the benefit of temperature-scaled surprisal by analyzing residual errors from regression models. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Predicting Reading Times, Linguistics, Cognitive Science, Language Processing Difficulty" />
        </attvalues>
      </node>
      <node id="2205.12507" label="2205.12507">
        <attvalues>
          <attvalue for="0" value="Re-Examining Calibration: The Case of Question Answering" />
          <attvalue for="1" value="  For users to trust model predictions, they need to understand model outputs,&#10;particularly their confidence - calibration aims to adjust (calibrate) models'&#10;confidence to match expected accuracy. We argue that the traditional&#10;calibration evaluation does not promote effective calibrations: for example, it&#10;can encourage always assigning a mediocre confidence score to all predictions,&#10;which does not help users distinguish correct predictions from wrong ones.&#10;Building on those observations, we propose a new calibration metric, MacroCE,&#10;that better captures whether the model assigns low confidence to wrong&#10;predictions and high confidence to correct predictions. Focusing on the&#10;practical application of open-domain question answering, we examine&#10;conventional calibration methods applied on the widely-used retriever-reader&#10;pipeline, all of which do not bring significant gains under our new MacroCE&#10;metric. Toward better calibration, we propose a new calibration method&#10;(ConsCal) that uses not just final model predictions but whether multiple model&#10;checkpoints make consistent predictions. Altogether, we provide an alternative&#10;view of calibration along with a new metric, re-evaluation of existing&#10;calibration methods on our metric, and proposal of a more effective calibration&#10;method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.14301" label="2211.14301">
        <attvalues>
          <attvalue for="0" value="On the Effect of Anticipation on Reading Times" />
          <attvalue for="1" value="  Over the past two decades, numerous studies have demonstrated how less&#10;predictable (i.e., higher surprisal) words take more time to read. In general,&#10;these studies have implicitly assumed the reading process is purely responsive:&#10;Readers observe a new word and allocate time to process it as required. We&#10;argue that prior results are also compatible with a reading process that is at&#10;least partially anticipatory: Readers could make predictions about a future&#10;word and allocate time to process it based on their expectation. In this work,&#10;we operationalize this anticipation as a word's contextual entropy. We assess&#10;the effect of anticipation on reading by comparing how well surprisal and&#10;contextual entropy predict reading times on four naturalistic reading datasets:&#10;two self-paced and two eye-tracking. Experimentally, across datasets and&#10;analyses, we find substantial evidence for effects of contextual entropy over&#10;surprisal on a word's reading time (RT): in fact, entropy is sometimes better&#10;than surprisal in predicting a word's RT. Spillover effects, however, are&#10;generally not captured by entropy, but only by surprisal. Further, we&#10;hypothesize four cognitive mechanisms through which contextual entropy could&#10;impact RTs -- three of which we are able to design experiments to analyze.&#10;Overall, our results support a view of reading that is not just responsive, but&#10;also anticipatory.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.02694" label="2205.02694">
        <attvalues>
          <attvalue for="0" value="Quantifying Language Variation Acoustically with Few Resources" />
          <attvalue for="1" value="  Deep acoustic models represent linguistic information based on massive&#10;amounts of data. Unfortunately, for regional languages and dialects such&#10;resources are mostly not available. However, deep acoustic models might have&#10;learned linguistic information that transfers to low-resource languages. In&#10;this study, we evaluate whether this is the case through the task of&#10;distinguishing low-resource (Dutch) regional varieties. By extracting&#10;embeddings from the hidden layers of various wav2vec 2.0 models (including new&#10;models which are pre-trained and/or fine-tuned on Dutch) and using dynamic time&#10;warping, we compute pairwise pronunciation differences averaged over 10 words&#10;for over 100 individual dialects from four (regional) languages. We then&#10;cluster the resulting difference matrix in four groups and compare these to a&#10;gold standard, and a partitioning on the basis of comparing phonetic&#10;transcriptions. Our results show that acoustic models outperform the&#10;(traditional) transcription-based approach without requiring phonetic&#10;transcriptions, with the best performance achieved by the multilingual XLSR-53&#10;model fine-tuned on Dutch. On the basis of only six seconds of speech, the&#10;resulting clustering closely matches the gold standard.&#10;" />
          <attvalue for="2" value="&#10;&#10;Deep acoustic models have improved automatic speech recognition (ASR) substantially in recent years \cite{schneider2019wav2vec, baevski2019vq, baevski2020wav2vec, conneau2020unsupervised}.&#10;These models represent linguistic information based on massive amounts of data.&#10;While these models are generally evaluated on ASR benchmarks, few studies have addressed what kind of linguistic information is represented by them.&#10;The work of \cite{livescuprobing2021} examined information represented by the wav2vec~2.0 model \cite{baevski2020wav2vec} across the various Transformer layers.&#10;They showed that different layers encode different types of linguistic information.&#10;Specifically, the initial layers appeared to be most similar to the input speech features, whereas the middle layers mostly encoded contextual information.&#10;The final layers again turned out to be similar to the input speech features. However, the representations of the final layers changed when the model was fine-tuned, likely because task-specific information was learned.&#10;In addition, \cite{liberman2021} investigated several deep acoustic models using phonetic probing tasks, and found that representations from these models capture information useful for distinguishing English phones. Importantly, these deep acoustic models were better able to distinguish English phones than using conventional MFCC or filterbank features. Although they evaluated the transferability of deep acoustic representations across several domains, it remains unclear whether these models learned information that transfers to other languages.&#10;This is, however, important when working on more inclusive speech technology. Especially when resources for training these models are lacking, such as for regional languages and dialects.&#10;In this paper, we therefore investigate if hidden layers of deep acoustic models incorporate fine-grained information, which can be used to represent differences between, and in turn distinguish, regional language varieties. &#10;&#10;Past work on investigating language variation has often been based on computing pronunciation distances that rely on phonetically transcribed speech \cite{nerbonne1997measuring, livescu2000lexical, heeringa2004measuring}. These (edit) distances have been found to match perceptual judgements of similarity well \cite{gooskens2004perceptive, wieling2014a}.&#10;However, transcribing speech phonetically is time-consuming and prone to errors \cite{bucholtz2007variation, novotney2010cheap}.&#10;While automatic approaches for computing phonetic transcriptions exist (e.g., \cite{allosaurus}), they produce lower quality phonetic transcriptions than human transcribers do.&#10;Additionally, (discrete) phonetic transcriptions do not capture all (continuous) aspects of human speech \cite{liberman2018}.&#10;&#10;To mitigate these shortcomings, acoustic approaches have been developed for investigating language variation \cite{Huckvale2007, Ferragne2010, Strycharczuk2020, acoustic-measure}.&#10;However, these studies either exclusively focused on the vowels (ignoring differences in the consonants), or were negatively influenced by non-linguistic variation in the speech signal.&#10;&#10;Recently, \cite{bartelds2021neural} found that representations from the hidden layers of pre-trained and fine-tuned wav2vec~2.0 (large) models are suitable to represent language variation.&#10;They showed that these representations capture linguistic information that is not represented by phonetic transcriptions, while being less sensitive to non-linguistic variation in the speech signal. Furthermore, this approach seems to provide a better match to human perceptual judgements than phonetic transcription-based approaches. &#10;&#10;To investigate if wav2vec 2.0 acoustic models (including newly trained Dutch models) learn fine-grained linguistic information that can transfer to regional languages and dialects, we will assess whether or not regional languages and dialects spoken in the Netherlands can be distinguished using these models.&#10;Our code and newly trained models are publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Linguistics, Low-Resource Languages, Speech Variability Analysis, Deep Acoustic Models, Mathematics" />
        </attvalues>
      </node>
      <node id="2011.12649" label="2011.12649">
        <attvalues>
          <attvalue for="0" value="Neural Representations for Modeling Variation in Speech" />
          <attvalue for="1" value="  Variation in speech is often quantified by comparing phonetic transcriptions&#10;of the same utterance. However, manually transcribing speech is time-consuming&#10;and error prone. As an alternative, therefore, we investigate the extraction of&#10;acoustic embeddings from several self-supervised neural models. We use these&#10;representations to compute word-based pronunciation differences between&#10;non-native and native speakers of English, and between Norwegian dialect&#10;speakers. For comparison with several earlier studies, we evaluate how well&#10;these differences match human perception by comparing them with available human&#10;judgements of similarity. We show that speech representations extracted from a&#10;specific type of neural model (i.e. Transformers) lead to a better match with&#10;human perception than two earlier approaches on the basis of phonetic&#10;transcriptions and MFCC-based acoustic features. We furthermore find that&#10;features from the neural models can generally best be extracted from one of the&#10;middle hidden layers than from the final layer. We also demonstrate that neural&#10;speech representations not only capture segmental differences, but also&#10;intonational and durational differences that cannot adequately be represented&#10;by a set of discrete symbols used in phonetic transcriptions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.14400" label="2401.14400">
        <attvalues>
          <attvalue for="0" value="Modular Adaptation of Multilingual Encoders to Written Swiss German&#10;  Dialect" />
          <attvalue for="1" value="  Creating neural text encoders for written Swiss German is challenging due to&#10;a dearth of training data combined with dialectal variation. In this paper, we&#10;build on several existing multilingual encoders and adapt them to Swiss German&#10;using continued pre-training. Evaluation on three diverse downstream tasks&#10;shows that simply adding a Swiss German adapter to a modular encoder achieves&#10;97.5% of fully monolithic adaptation performance. We further find that for the&#10;task of retrieving Swiss German sentences given Standard German queries,&#10;adapting a character-level model is more effective than the other adaptation&#10;strategies. We release our code and the models trained for our experiments at&#10;https://github.com/ZurichNLP/swiss-german-text-encoders&#10;" />
          <attvalue for="2" value="&#10;When applying natural language processing~(NLP) techniques to languages with dialectal variation, two typical challenges are a lack of public training data as well as varying spelling conventions.&#10;In the case of Swiss German, which is spoken by around~5~million people and is often used for informal written communication in Switzerland, these factors make it more challenging to train a BERT-like text encoder for written text.&#10;&#10;In this paper, we adapt pre-trained multilingual encoders to Swiss German using continued pre-training on a modest amount of Swiss German training data.&#10;We evaluate the approaches on part-of-speech~(POS) tagging with zero-shot cross-lingual transfer from Standard German~\cite{aepli-sennrich-2022-improving}, as well as dialect identification~\cite{zampieri-etal-2019-report} and cross-lingual sentence retrieval based on a parallel Standard German--Swiss German test set~\cite{aepli-etal-2023-benchmark}.&#10;&#10;We find that depending on the multilingual encoder, continued pre-training leads to an average improvement of 10\%–45\% in average accuracy across the three downstream tasks.&#10;We then focus on comparing monolithic adaptation, where all the parameters of the encoder are updated during continued pre-training, to modular adaptation with language-specific modular components (language adapters; \cite{pfeiffer-etal-2022-lifting}).&#10;Even though modular adaptation only updates a fraction of the parameters, it is competitive to monolithic adaptation.&#10;Given these findings, we propose to extend the SwissBERT model~\cite{vamvas-etal-2023-swissbert}, which was trained on Standard German and other languages, with a Swiss German adapter~(Table~\ref{tab:figure-1}).&#10;&#10;We further hypothesize that the architecture of Canine~\cite{clark-etal-2022-canine}, a tokenization-free model that operates on characters, might be better suited to the highly variable spelling of Swiss German.&#10;Indeed, a Canine model adapted to Swiss German excels on the retrieval tasks, while POS tagging works better with subwords.&#10;&#10;Finally, we aim to combine the best of both worlds by integrating character-level down- and upsampling modules into a subword-based model and training a character-level adapter for Swiss German.&#10;However, this jointly modular and tokenization-free strategy underperforms the individual approaches.&#10;We hope that our findings can inform the development of modular approaches for other languages with dialectal variation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Dialectal Variation Modeling, Computer Science, Linguistics, Multilingual Adaptation, Artificial Intelligence, Neural Text Encoding" />
        </attvalues>
      </node>
      <node id="2311.16865" label="2311.16865">
        <attvalues>
          <attvalue for="0" value="A Benchmark for Evaluating Machine Translation Metrics on Dialects&#10;  Without Standard Orthography" />
          <attvalue for="1" value="  For sensible progress in natural language processing, it is important that we&#10;are aware of the limitations of the evaluation metrics we use. In this work, we&#10;evaluate how robust metrics are to non-standardized dialects, i.e. spelling&#10;differences in language varieties that do not have a standard orthography. To&#10;investigate this, we collect a dataset of human translations and human&#10;judgments for automatic machine translations from English to two Swiss German&#10;dialects. We further create a challenge set for dialect variation and benchmark&#10;existing metrics' performances. Our results show that existing metrics cannot&#10;reliably evaluate Swiss German text generation outputs, especially on segment&#10;level. We propose initial design adaptations that increase robustness in the&#10;face of non-standardized dialects, although there remains much room for further&#10;improvement. The dataset, code, and models are available here:&#10;https://github.com/textshuttle/dialect_eval&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.13310" label="2303.13310">
        <attvalues>
          <attvalue for="0" value="SwissBERT: The Multilingual Language Model for Switzerland" />
          <attvalue for="1" value="  We present SwissBERT, a masked language model created specifically for&#10;processing Switzerland-related text. SwissBERT is a pre-trained model that we&#10;adapted to news articles written in the national languages of Switzerland --&#10;German, French, Italian, and Romansh. We evaluate SwissBERT on natural language&#10;understanding tasks related to Switzerland and find that it tends to outperform&#10;previous models on these tasks, especially when processing contemporary news&#10;and/or Romansh Grischun. Since SwissBERT uses language adapters, it may be&#10;extended to Swiss German dialects in future work. The model and our open-source&#10;code are publicly released at https://github.com/ZurichNLP/swissbert.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.10185" label="2312.10185">
        <attvalues>
          <attvalue for="0" value="Student as an Inherent Denoiser of Noisy Teacher" />
          <attvalue for="1" value="  Knowledge distillation (KD) has been widely employed to transfer knowledge&#10;from a large language model (LLM) to a specialized model in low-data regimes&#10;through pseudo label learning. However, pseudo labels generated by teacher&#10;models are usually noisy and may influence KD performance. This study delves&#10;into KD with noisy teachers and uncovers that the student model can already&#10;generate more accurate predictions than the teacher labels used to train it&#10;during KD, indicating its inherent ability to denoise noisy teacher labels.&#10;Motivated by this finding, we propose Peer-Advised KD to improve vanilla KD&#10;from noisy teachers. Experiments show that Peer-Advised KD can outperform LLM&#10;by approximately 5% with 50 human-labeled data, and even competitive to&#10;standard supervised finetuning with 750 human-labeled data.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, with the emergence of large language models (LLMs), knowledge distillation (KD) with label matching has been widely applied to low-data regimes~\cite{shridhar2022distilling,ho2022large,li2023feasibility,gilardi2023chatgpt,wang-etal-2021-want-reduce,yoo-etal-2021-gpt3mix-leveraging,ding2022gpt}. In scenarios where there is a scarcity of labeled in-domain data but an abundance of unlabeled data, LLMs can serve as an annotator to generate pseudo labels through few-shot prompting, i.e., in-context learning~\cite{brown2020language}. These pseudo labels are then used to train a smaller student model through KD. The student model can offer improved efficiency and faster inference for deployment. However, one crucial problem with such use of KD is that the generated pseudo labels of teacher models are usually noisy. The performance of student models can be greatly dependent on the quality of teacher labels~\cite{NEURIPS2022_2e343555}. Despite many past works on leveraging LLMs as annotators, few works have investigated how to optimize learning noisy data. Some previous works for noisy label learning utilize loss reweighting~\cite{hinton2015distilling,NEURIPS2022_2e343555,lee2013pseudo} that may not apply to few-shot cases where pseudo labels are overwhelmingly more than gold labels, leading to great imbalance and unstable training. &#10;&#10;Therefore, this work aims at providing new insights into learning noisy teacher labels for more reliable student training in low-data regimes. This work mainly focuses on learning linguistic structures as our test bed, which usually requires extensive labeled data to acquire~\cite{drozdov-etal-2019-unsupervised-latent} and is the fundamental ability to downstream NLP tasks~\cite{manning2020emergent}. More specifically, we first analyze the learning process of KD (Sec.~\ref{sec:theo}) with according empirical results (Sec.~\ref{sec:exp1}). Our analysis reveals that during KD, the student model exhibits a denoising ability, generating superior predictions compared to the noisy teacher labels used to train it. Our findings imply that vanilla KD from noisy teacher gives sub-optimal student's performance. Motivated by our findings, we present Peer-Advised KD (PA-KD) in Sec.~\ref{sec:main} which harnesses the student's inherent denoising ability to generate improved pseudo labels for KD. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Noisy Teacher Models, Linguistics, Cognitive Science, Knowledge Distillation, Artificial Intelligence, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2210.06711" label="2210.06711">
        <attvalues>
          <attvalue for="0" value="Weighted Distillation with Unlabeled Examples" />
          <attvalue for="1" value="  Distillation with unlabeled examples is a popular and powerful method for&#10;training deep neural networks in settings where the amount of labeled data is&#10;limited: A large ''teacher'' neural network is trained on the labeled data&#10;available, and then it is used to generate labels on an unlabeled dataset&#10;(typically much larger in size). These labels are then utilized to train the&#10;smaller ''student'' model which will actually be deployed. Naturally, the&#10;success of the approach depends on the quality of the teacher's labels, since&#10;the student could be confused if trained on inaccurate data. This paper&#10;proposes a principled approach for addressing this issue based on a&#10;''debiasing'' reweighting of the student's loss function tailored to the&#10;distillation training paradigm. Our method is hyper-parameter free,&#10;data-agnostic, and simple to implement. We demonstrate significant improvements&#10;on popular academic datasets and we accompany our results with a theoretical&#10;analysis which rigorously justifies the performance of our method in certain&#10;settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.13590" label="2403.13590">
        <attvalues>
          <attvalue for="0" value="Teacher-Student Training for Debiasing: General Permutation Debiasing&#10;  for Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated impressive zero-shot&#10;capabilities and versatility in NLP tasks, however they sometimes fail to&#10;maintain crucial invariances for specific tasks. One example is permutation&#10;sensitivity, where LLMs' outputs may significantly vary depending on the order&#10;of the input options. While debiasing techniques can mitigate these issues, and&#10;yield better performance and reliability, they often come with a high&#10;computational cost at inference. This paper addresses this inefficiency at&#10;inference time. The aim is to distill the capabilities of a computationally&#10;intensive, debiased, teacher model into a more compact student model. We&#10;explore two variants of student models: one based on pure distillation, and the&#10;other on an error-correction approach for more complex tasks, where the student&#10;corrects a single biased decision from the teacher to achieve a debiased&#10;output. Our approach is general and can be applied to both black-box and&#10;white-box LLMs. Furthermore, we demonstrate that our compact, encoder-only&#10;student models can outperform their larger, biased teacher counterparts,&#10;achieving better results with significantly fewer parameters.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancements in Large Language Models (LLMs) have led to dramatic shifts within natural language processing (NLP). Unlike prior &quot;pre-train and fine-tune&quot; \cite{devlin-etal-2019-bert, he2020deberta} approaches, instruction-tuned LLMs combined with effective good prompting techniques has enabled LLMs to excel at unseen tasks without task-specific training \cite{brown2020language, touvron2023llama}. This has led to the current capabilities of LLMs, where they demonstrate great versatility, while also being powerful and displaying state-of-the-art performance on many standard NLP benchmark leaderboards \cite{Open-LLM-Leaderboard-Report-2023}.&#10;&#10;Despite their impressive general abilities, LLMs suffer from particular limitations. They are prone to hallucinating information \cite{huang2023survey, manakul2023selfcheckgpt}, can have large sensitivity to the form of prompts \cite{sclar2023quantifying, zhou2022large} and also demonstrate systematic biases such as gender bias \cite{kotek2023gender}. Furthermore, due to the general nature of their pre-training and instruction-tuning \cite{wei2021finetuned, ouyang2022training}, for certain applications, they may be unaware of particular important task invariances. One such invariance that LLMs may fail to maintain is permutation-invariance. Ongoing work has demonstrated that LLMs can be sensitive to the input order of options, which has been observed for both question answering \cite{pezeshkpour2023large, zheng2023large} and pairwise assessment \cite{zheng2023judging, wang2023large, liusie2023zero}. For these tasks, varying the ordering of the input options may lead to different decisions by the LLM, which can impact downstream performance and reliability.&#10;&#10;Although debiasing approaches can be applied to enforce invariances, such methods can be computationally expensive or inapplicable to black-box settings \cite{zheng2023large}. To address these challenges, this work introduces a general framework that can be used to adapt both black-box and white-box systems to follow a particular invariance, while being inference efficient. For a given invariance and debiasing scheme, our framework trains a compact student to emulate the debiased teacher, which during inference can be efficiently deployed. We investigate two variants of students, a simple knowledge-distilled student, as well as an error-correction student that takes in a single biased teacher sample and corrects it to learn the debiased teacher decision, applicable for more complicated tasks. We demonstrate the effectiveness of our framework on permutation invariance, and illustrate that small 330M parameter student models can outperform their larger biased teacher counterparts, while also maintaining particular embedded invariances. &#10;&#10;The contributions of this work are: 1) We provide metrics for assessing the sensitivity of models to the input ordering of options. 2) We showcase that LLMs can demonstrate large permutation sensitivity and that biases seem correlated to task performance. 3) We study several different debiased approaches that yield significant performance gains. 4) Experiments on RACE++ and SummEval demonstrate that the teacher-student training for debiasing framework yields effective students that perform better than their biased teacher while being inference-efficient and not expensive to train. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Debiasing, Linguistics, Cognitive Science, Knowledge Distillation, Artificial Intelligence, Mathematics, Efficient NLP Modeling" />
        </attvalues>
      </node>
      <node id="2308.11483" label="2308.11483">
        <attvalues>
          <attvalue for="0" value="Large Language Models Sensitivity to The Order of Options in&#10;  Multiple-Choice Questions" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated remarkable capabilities in&#10;various NLP tasks. However, previous works have shown these models are&#10;sensitive towards prompt wording, and few-shot demonstrations and their order,&#10;posing challenges to fair assessment of these models. As these models become&#10;more powerful, it becomes imperative to understand and address these&#10;limitations. In this paper, we focus on LLMs robustness on the task of&#10;multiple-choice questions -- commonly adopted task to study reasoning and&#10;fact-retrieving capability of LLMs. Investigating the sensitivity of LLMs&#10;towards the order of options in multiple-choice questions, we demonstrate a&#10;considerable performance gap of approximately 13% to 75% in LLMs on different&#10;benchmarks, when answer options are reordered, even when using demonstrations&#10;in a few-shot setting. Through a detailed analysis, we conjecture that this&#10;sensitivity arises when LLMs are uncertain about the prediction between the&#10;top-2/3 choices, and specific options placements may favor certain prediction&#10;between those top choices depending on the question caused by positional bias.&#10;We also identify patterns in top-2 choices that amplify or mitigate the model's&#10;bias toward option placement. We found that for amplifying bias, the optimal&#10;strategy involves positioning the top two choices as the first and last&#10;options. Conversely, to mitigate bias, we recommend placing these choices among&#10;the adjacent options. To validate our conjecture, we conduct various&#10;experiments and adopt two approaches to calibrate LLMs' predictions, leading to&#10;up to 8 percentage points improvement across different models and benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.03882" label="2309.03882">
        <attvalues>
          <attvalue for="0" value="Large Language Models Are Not Robust Multiple Choice Selectors" />
          <attvalue for="1" value="  Multiple choice questions (MCQs) serve as a common yet important task format&#10;in the evaluation of large language models (LLMs). This work shows that modern&#10;LLMs are vulnerable to option position changes in MCQs due to their inherent&#10;&quot;selection bias&quot;, namely, they prefer to select specific option IDs as answers&#10;(like &quot;Option A&quot;). Through extensive empirical analyses with 20 LLMs on three&#10;benchmarks, we pinpoint that this behavioral bias primarily stems from LLMs'&#10;token bias, where the model a priori assigns more probabilistic mass to&#10;specific option ID tokens (e.g., A/B/C/D) when predicting answers from the&#10;option IDs. To mitigate selection bias, we propose a label-free, inference-time&#10;debiasing method, called PriDe, which separates the model's prior bias for&#10;option IDs from the overall prediction distribution. PriDe first estimates the&#10;prior by permutating option contents on a small number of test samples, and&#10;then applies the estimated prior to debias the remaining samples. We&#10;demonstrate that it achieves interpretable and transferable debiasing with high&#10;computational efficiency. We hope this work can draw broader research attention&#10;to the bias and robustness of modern LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.17714" label="2310.17714">
        <attvalues>
          <attvalue for="0" value="Nearest Neighbor Search over Vectorized Lexico-Syntactic Patterns for&#10;  Relation Extraction from Financial Documents" />
          <attvalue for="1" value="  Relation extraction (RE) has achieved remarkable progress with the help of&#10;pre-trained language models. However, existing RE models are usually incapable&#10;of handling two situations: implicit expressions and long-tail relation&#10;classes, caused by language complexity and data sparsity. Further, these&#10;approaches and models are largely inaccessible to users who don't have direct&#10;access to large language models (LLMs) and/or infrastructure for supervised&#10;training or fine-tuning. Rule-based systems also struggle with implicit&#10;expressions. Apart from this, Real world financial documents such as various&#10;10-X reports (including 10-K, 10-Q, etc.) of publicly traded companies pose&#10;another challenge to rule-based systems in terms of longer and complex&#10;sentences. In this paper, we introduce a simple approach that consults training&#10;relations at test time through a nearest-neighbor search over dense vectors of&#10;lexico-syntactic patterns and provides a simple yet effective means to tackle&#10;the above issues. We evaluate our approach on REFinD and show that our method&#10;achieves state-of-the-art performance. We further show that it can provide a&#10;good start for human in the loop setup when a small number of annotations are&#10;available and it is also beneficial when domain experts can provide high&#10;quality patterns.&#10;" />
          <attvalue for="2" value="&#10;&#10;Relation extraction (RE) from text is a fundamental problem in NLP and information retrieval, which facilitates various tasks like knowledge graph construction, question answering and semantic search. Recent studies \cite{zhang-etal-2020-minimize, zeng2020copymtl, lin-etal-2020-joint, wang-lu-2020-two, cheng-etal-2020-dynamically, zhong-chen-2021-frustratingly} in supervised RE take advantage of pre-trained language models (PLMs) and achieve SOTA performances by fine-tuning PLMs with a relation classifier. However, \cite{wan-etal-2022-rescue} observes that existing RE models are usually incapable of handling two RE-specific situations: implicit expressions and long-tail relation types. &#10;&#10;Implicit expression refers to the situation whereas relation is expressed as the underlying message that is not explicitly stated or shown. &#10;&#10;In Figure 1, relation &quot;acquired\_by(organization, organization)&quot; occurs implicitly. Such underlying messages can easily confuse the relation classifier. &#10;&#10;The other problem of long-tail relation classes is caused by data sparsity in training. For example, the REFinD dataset \cite{kaur2023refind} comprises 45.5 \% of the no\_relation instances. The most frequent class in the dataset - “per:title:title” has 4,468 training examples, while over 14 out of 22 classes have less than 500 examples. The majority class can easily dominate model predictions and lead to low performance on long-tail classes. &#10;&#10;Recently, ICL (In-Context Learning) based approach \cite{wan2023gptre} is utilized for RE tasks. The approach achieves improvements over not only existing GPT-3 baselines, but also on fully-supervised baselines even with only a limited number of demonstrations provided in the prompt. Specifically, it achieves SOTA performances on the Semeval and SciERC datasets, and competitive performances on the TACRED and \cite{zhang2017tacred} ACE05 datasets. \cite{rajpoot2023gptfinre} utilized the GPT-4 under ICL framework on REFinD and achieved 3rd rank in the shared task. &#10;&#10;However, retrieval of examples to demonstrate is a key factor in the overall performance on these pipelines. Finding efficient demonstrates often relies on learning-based retrieval \cite{ye2023compositional, rubin-etal-2022-learning}. These learning-based retrievers use annotated data and a LLM. This type of retrieval strategy comes with the increased cost (API, infrastructure etc.), time as more experiments are required because most LLMs are black box and it also needs special expertise. &#10;&#10;Apart from the implicit expression challenge mentioned above, REFinD poses another challenge to rule-based systems in terms of longer and complex sentences. For example, \cite{kaur2023refind} cites that the average sentence length in the REFinD dataset is 53.7 while the average sentence length in the TACRED dataset \cite{zhang-etal-2017-position} is 36.2. Further, As per \cite{kaur2023refind}, REFinD includes more complex sentences than TACRED, with an average entity-pair distance of 11, compared to 8 in TACRED. Because of this, writing rules at surface text level is a challenge. Hence, rules at lexico-syntactic level is the need of the hour. However, strict matching of these rules can yield high precision but low recall result due to accuracy of syntactic parsing. Hence, a robust fuzzy pattern matching system is required. &#10;&#10;Inspired by recent studies \cite{wan-etal-2022-rescue, khandelwal2019generalization, guu2020retrieval, meng2021gnn} using k-Nearest Neighbor to retrieve diverse expressions for language generation tasks, we introduce a simple but effective approach that consults training relations at test time through a nearest-neighbor search over dense vectors of lexico-syntactic patterns and provides a simple yet effective means to tackle the above issues. Our method achieves an improvement of 1.18\% over baseline (F1-score - 0.7516). We achieved our results using commodity hardware within a day. That’s why our approach is easier to deploy, lightweight and fast. We further show that our approach can provide a good start (F1-score of 0.5122) for human in the loop setup when a small number of annotations (approx. 10\% of training data) are available and it is also beneficial (F1-score of 0.6939 with approx. 10\% of training data) when domain experts can provide high quality patterns. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Limitations, Financial Document Analysis, Artificial Intelligence, Information Science, Natural Language Processing, Relation Extraction" />
        </attvalues>
      </node>
      <node id="2009.07503" label="2009.07503">
        <attvalues>
          <attvalue for="0" value="Minimize Exposure Bias of Seq2Seq Models in Joint Entity and Relation&#10;  Extraction" />
          <attvalue for="1" value="  Joint entity and relation extraction aims to extract relation triplets from&#10;plain text directly. Prior work leverages Sequence-to-Sequence (Seq2Seq) models&#10;for triplet sequence generation. However, Seq2Seq enforces an unnecessary order&#10;on the unordered triplets and involves a large decoding length associated with&#10;error accumulation. These introduce exposure bias, which may cause the models&#10;overfit to the frequent label combination, thus deteriorating the&#10;generalization. We propose a novel Sequence-to-Unordered-Multi-Tree&#10;(Seq2UMTree) model to minimize the effects of exposure bias by limiting the&#10;decoding length to three within a triplet and removing the order among&#10;triplets. We evaluate our model on two datasets, DuIE and NYT, and&#10;systematically study how exposure bias alters the performance of Seq2Seq&#10;models. Experiments show that the state-of-the-art Seq2Seq model overfits to&#10;both datasets while Seq2UMTree shows significantly better generalization. Our&#10;code is available at https://github.com/WindChimeRan/OpenJERE .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.11800" label="2210.11800">
        <attvalues>
          <attvalue for="0" value="Rescue Implicit and Long-tail Cases: Nearest Neighbor Relation&#10;  Extraction" />
          <attvalue for="1" value="  Relation extraction (RE) has achieved remarkable progress with the help of&#10;pre-trained language models. However, existing RE models are usually incapable&#10;of handling two situations: implicit expressions and long-tail relation types,&#10;caused by language complexity and data sparsity. In this paper, we introduce a&#10;simple enhancement of RE using $k$ nearest neighbors ($k$NN-RE). $k$NN-RE&#10;allows the model to consult training relations at test time through a&#10;nearest-neighbor search and provides a simple yet effective means to tackle the&#10;two issues above. Additionally, we observe that $k$NN-RE serves as an effective&#10;way to leverage distant supervision (DS) data for RE. Experimental results show&#10;that the proposed $k$NN-RE achieves state-of-the-art performances on a variety&#10;of supervised RE datasets, i.e., ACE05, SciERC, and Wiki80, along with&#10;outperforming the best model to date on the i2b2 and Wiki80 datasets in the&#10;setting of allowing using DS. Our code and models are available at:&#10;https://github.com/YukinoWan/kNN-RE.&#10;" />
          <attvalue for="2" value="&#10;Relation extraction (RE) aims to identify the relationship between entities mentioned in a sentence, and is beneficial to a variety of downstream tasks such as question answering and knowledge base population. Recent studies~\cite{zhang-etal-2020-minimize,Zeng_Zhang_Liu_2020,lin-etal-2020-joint,wang-lu-2020-two,cheng-etal-2020-dynamically,zhong-chen-2021-frustratingly} in supervised RE &#10;take advantage of &#10; pre-trained language models (PLMs) and achieve SOTA performances by fine-tuning PLMs with a relation classifier. However, we observe that &#10; existing&#10;RE models are usually incapable of handling two RE-specific situations&#10;: implicit expressions and long-tail relation types.&#10;&#10;Implicit expression refers to the situation where&#10;a relation is expressed as the underlying message that is not explicitly stated or shown.&#10;For example, for the relation ``sibling to'', a common expression can be ``\underline{He} has a brother \underline{James}'', while an implicit expression could be ``He is the youngest son of Liones, &#10;comparing with \underline{Samuel Liones} and \underline{Henry Liones}.'' In the latter case, the relation ``sibling to'' between ``\underline{Samuel Liones}'' and ``\underline{Henry Liones}'' is not directly expressed but could be inferred from them both are brothers of the same person. Such underlying message can easily confuse the relation classifier. &#10;The problem of long-tail relation types is caused by data sparsity in training. For example, the widely used supervised RE dataset TACRED~\cite{zhang-etal-2017-position} &#10;includes 41 relation types. The most frequent type ``per:title'' has 3,862 training examples, while over 22 types have less than 300 examples. The majority types&#10;can easily dominate model &#10;predictions&#10;and lead to low performance on long-tail types.&#10;&#10;Inspired by recent studies~\cite{DBLP:conf/iclr/KhandelwalLJZL20, DBLP:journals/corr/abs-2002-08909,DBLP:journals/corr/abs-2110-08743} using $k$NN to retrieve diverse expressions for&#10;language generation tasks, we introduce a simple but effective $k$NN-RE framework to address above-mentioned two problems.&#10;Specifically, we store the training examples as the memory by a vanilla RE model and consult the stored memory at test time through a nearest-neighbor search.&#10;As shown in Figure~\ref{fig:problems}, for an implicit expression, the expression ``son of'' may mislead to an incorrect prediction while its retrieved nearest neighbor contains a direct expression ``brother of'', which is a more explicit expression of the gold label ``sibling to''.&#10;The prediction of long-tail examples, as shown in Figure~\ref{fig:problems}, is usually biased toward the majority class. Nearest neighbor retrieval provides direct guidance to the prediction by &#10;referring to the labels of its nearest neighbors in the training set, and thus can significantly reduce the imbalanced classification.&#10;&#10;Additionally, we observe that $k$NN-RE serves as an efficient way to leverage distant supervision (DS) data for RE. DS augments labeled RE datasets by matching knowledge base (KB) relation triplets and raw text entity pairs in a weak-supervision fashion ~\cite{mintz-etal-2009-distant,lin-etal-2016-neural,vashishth-etal-2018-reside,chen-etal-2021-cil}. &#10;Recent studies~\cite{baldini-soares-etal-2019-matching,DBLP:journals/corr/abs-2102-09681,peng-etal-2020-learning, which apply PLMs to the DS labeled data &#10; to improve supervised RE, &#10; require heavy computation due to the fact that &#10;they require pre-training on DS data, whose size is usually dozens of times that of supervised datasets. &#10;To address this issue, &#10;we propose a lightweight method to leverage DS data to benefit supervised RE by extending the construction of stored memory for $k$NN-RE to DS labeled data and outperforming the recent best pre-training method with no extra training. &#10;&#10;In summary, we propose $k$NN-RE: a flexible $k$NN framework to solve the RE task. We conduct the experiments for $k$NN-RE with three different memory settings: training, DS, and the combination of training and DS. The results show that our $k$NN-RE with the training memory obtains a 0.84\%-1.15\% absolute F1 improvement on five datasets and achieves state-of-the-art (SOTA) F1 scores on three of them (ACE05, SciERC and Wiki80). In the DS setup, $k$NN-RE outperforms SOTA DS pre-training methods on two datasets (i2b2, Wiki20) significantly without extra training.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.18322" label="2305.18322">
        <attvalues>
          <attvalue for="0" value="REFinD: Relation Extraction Financial Dataset" />
          <attvalue for="1" value="  A number of datasets for Relation Extraction (RE) have been created to aide&#10;downstream tasks such as information retrieval, semantic search, question&#10;answering and textual entailment. However, these datasets fail to capture&#10;financial-domain specific challenges since most of these datasets are compiled&#10;using general knowledge sources such as Wikipedia, web-based text and news&#10;articles, hindering real-life progress and adoption within the financial world.&#10;To address this limitation, we propose REFinD, the first large-scale annotated&#10;dataset of relations, with $\sim$29K instances and 22 relations amongst 8 types&#10;of entity pairs, generated entirely over financial documents. We also provide&#10;an empirical evaluation with various state-of-the-art models as benchmarks for&#10;the RE task and highlight the challenges posed by our dataset. We observed that&#10;various state-of-the-art deep learning models struggle with numeric inference,&#10;relational and directional ambiguity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.02105" label="2305.02105">
        <attvalues>
          <attvalue for="0" value="GPT-RE: In-context Learning for Relation Extraction using Large Language&#10;  Models" />
          <attvalue for="1" value="  In spite of the potential for ground-breaking achievements offered by large&#10;language models (LLMs) (e.g., GPT-3), they still lag significantly behind&#10;fully-supervised baselines (e.g., fine-tuned BERT) in relation extraction (RE).&#10;This is due to the two major shortcomings of LLMs in RE: (1) low relevance&#10;regarding entity and relation in retrieved demonstrations for in-context&#10;learning; and (2) the strong inclination to wrongly classify NULL examples into&#10;other pre-defined labels.&#10;  In this paper, we propose GPT-RE to bridge the gap between LLMs and&#10;fully-supervised baselines. GPT-RE successfully addresses the aforementioned&#10;issues by (1) incorporating task-specific entity representations in&#10;demonstration retrieval; and (2) enriching the demonstrations with gold&#10;label-induced reasoning logic. We evaluate GPT-RE on four widely-used RE&#10;datasets, and observe that GPT-RE achieves improvements over not only existing&#10;GPT-3 baselines, but also fully-supervised baselines. Specifically, GPT-RE&#10;achieves SOTA performances on the Semeval and SciERC datasets, and competitive&#10;performances on the TACRED and ACE05 datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.08753" label="2310.08753">
        <attvalues>
          <attvalue for="0" value="CompA: Addressing the Gap in Compositional Reasoning in Audio-Language&#10;  Models" />
          <attvalue for="1" value="  A fundamental characteristic of audio is its compositional nature.&#10;Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP)&#10;that learns a shared representation between audio and language modalities have&#10;improved performance in many downstream applications, including zero-shot audio&#10;classification, audio retrieval, etc. However, the ability of these models to&#10;effectively perform compositional reasoning remains largely unexplored and&#10;necessitates additional research. In this paper, we propose CompA, a collection&#10;of two expert-annotated benchmarks with a majority of real-world audio samples,&#10;to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates&#10;how well an ALM understands the order or occurrence of acoustic events in&#10;audio, and CompA-attribute evaluates attribute-binding of acoustic events. An&#10;instance from either benchmark consists of two audio-caption pairs, where both&#10;audios have the same acoustic events but with different compositions. An ALM is&#10;evaluated on how well it matches the right audio to the right caption. Using&#10;this benchmark, we first show that current ALMs perform only marginally better&#10;than random chance, thereby struggling with compositional reasoning. Next, we&#10;propose CompA-CLAP, where we fine-tune CLAP using a novel learning method to&#10;improve its compositional reasoning abilities. To train CompA-CLAP, we first&#10;propose improvements to contrastive training with composition-aware hard&#10;negatives, allowing for more focused training. Next, we propose a novel modular&#10;contrastive loss that helps the model learn fine-grained compositional&#10;understanding and overcomes the acute scarcity of openly available&#10;compositional audios. CompA-CLAP significantly improves over all our baseline&#10;models on the CompA benchmark, indicating its superior compositional reasoning&#10;capabilities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;{ Compositional Reasoning.} Early work in linguistics has tried to understand what models know about word order~\cite{sinha-etal-2021-masked}, syntax~\cite{gauthier-etal-2020-syntaxgym, gulordava-etal-2018-colorless, hu-etal-2020-systematic, linzen-etal-2016-assessing}, or the complex interaction between syntactic and semantic categories~\cite{kann-etal-2019-verb, thrush-etal-2020-investigating, warstadt-etal-2019, warstadt-etal-2020-blimp-benchmark}. Learning visio-linguistic compositionality has been extensively studied in prior art \cite{yuksekgonul2023when,ma2023crepe}. Winground \cite{thrush2022winoground}, the work closest to ours, proposes a benchmark with 400 test cases in the Winograd twin sentence format, with pairs of compositionally different images and captions with the same words but in a different order. The task, similar to the one proposed in this paper, is to match the right image with the right caption. They additionally show that current VLMs perform no better than random chance. The Winograd twin sentence format was originally proposed in the Winograd schema challenge \cite{levesque2012winograd} and has been earlier used for a variety of language-related tasks \cite{rudinger2018gender,sakaguchi2021winogrande,zhao2018gender}. &#10;Following this, \cite{yuksekgonul2023when} propose a large-scale benchmark with over 50,000 test cases and compositionally different captions by swapping relational tokens within sentences. Lack of compositional reasoning in VLMs has affected multiple downstream tasks like text-to-image generation \cite{conwell2022testing} and Visual Question Answering~\cite{bogin2021covr}. A similar problem was observed in AudioLDM~\cite{liu2023audioldm}, which employs CLAP as a text encoder for text-to-audio generation and fails to generate compositional audios~\cite{ghosal2023tango}.&#10;&#10;{ Audio and Language.} Recent developments indicate an increasing trend in leveraging language as a modality for interaction with audio systems. Downstream tasks like text-to-audio generation \cite{ghosal2023tango,liu2023audioldm,huang2023make} and text-to-music generation \cite{agostinelli2023musiclm} have gained much popularity. Other tasks include text-guided audio source separation \cite{liu2023separate} audio captioning \cite{ghosh2023recap}, etc. \cite{deshmukh2023pengi} integrate language models with audio encoders and frame all audio tasks as text-generation tasks. Their model, Pengi, achieves SoTA performance on 22 downstream tasks, which shows promises of effective language modality integration for enhanced audio system interactions. Most of these models employ a text encoder or audio encoder to accomplish their task. CLAP, which learns a shared representation between audio&#10;and language modalities and achieves impressive performance on zero-shot tasks, proving to be a compelling model for cross-modal understanding and interaction. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Contrastive Learning, Audio Signal Processing, Linguistics, Cognitive Science, Audio Language Models, Compositional Reasoning" />
        </attvalues>
      </node>
      <node id="2309.10299" label="2309.10299">
        <attvalues>
          <attvalue for="0" value="Using fine-tuning and min lookahead beam search to improve Whisper" />
          <attvalue for="1" value="  The performance of Whisper in low-resource languages is still far from&#10;perfect. In addition to a lack of training data on low-resource languages, we&#10;identify some limitations in the beam search algorithm used in Whisper. To&#10;address these issues, we fine-tune Whisper on additional data and propose an&#10;improved decoding algorithm. On the Vietnamese language, fine-tuning&#10;Whisper-Tiny with LoRA leads to an improvement of 38.49 in WER over the&#10;zero-shot Whisper-Tiny setting which is a further reduction of 1.45 compared to&#10;full-parameter fine-tuning. Additionally, by using Filter-Ends and Min&#10;Lookahead decoding algorithms, the WER reduces by 2.26 on average over a range&#10;of languages compared to standard beam search. These results generalise to&#10;larger Whisper model sizes. We also prove a theorem that Min Lookahead&#10;outperforms the standard beam search algorithm used in Whisper.&#10;" />
          <attvalue for="2" value="&#10;&#10;Whisper has remarkable performance in transcribing multilingual speech audio into text \cite{radford2022robust}. While its performance with English and other high-resource languages is impressive, the limited availability of training audio data for low-resources languages is a challenge. As Whisper is open-source, researchers may enhance its performance with new training datasets and methods. In this paper, we investigate unconventional fine-tuning and decoding algorithms to improve Whisper's performance in a low-resource scenario.&#10;&#10;While fine-tuning is common in practice, a systematic comparison between different fine-tuning strategies for an encoder-decoder model like Whisper has yet to be documented. In the work of Jain et al. \cite{jain2023adaptation}, the authors froze most of the model's parameters while fine-tuned only the final layer. Conversely, Rouditchenko et al. \cite{rouditchenko2023comparison} fine-tuned the entire model on unseen languages. Both studies lack comprehensive explanations for their choice of fine-tuning strategies. To fill this gap, we conduct a comprehensive study of fine-tuning strategies on Whisper, including full-parameter fine-tuning and partial-parameter fine-tuning where gradients are updated only in parts of the model. We selected Vietnamese as our target language, but we believe the results translate to other low-resource languages since we did not utilise any language specific features in our fine-tuning experiments. &#10;&#10;Whisper uses a beam search decoding algorithm with beam width $n=5$ and log-probability (logprob) as the score function \cite{radford2022robust}. This is as opposed to the greedy algorithm which chooses the token with the greatest logprob at each decoding step. Although beam search outperforms the greedy algorithm, we suggest it can be further improved by filtering out certain sequences and performing additional decoding to access the probabilities of tokens at later decoding steps.&#10;&#10;The rest of this paper is structured as follows: Section 2 details the methodology used in fine-tuning and the decoding algorithm. Section 3 documents the various experiments we conducted to improve Whisper models. Finally, we conclude with Section 4 where we discuss the contribution of our paper and outline the future work.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Decoding Algorithm Improvement, Artificial Intelligence, Mathematics, Low-Resource Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2307.13008" label="2307.13008">
        <attvalues>
          <attvalue for="0" value="Adaptation of Whisper models to child speech recognition" />
          <attvalue for="1" value="  Automatic Speech Recognition (ASR) systems often struggle with transcribing&#10;child speech due to the lack of large child speech datasets required to&#10;accurately train child-friendly ASR models. However, there are huge amounts of&#10;annotated adult speech datasets which were used to create multilingual ASR&#10;models, such as Whisper. Our work aims to explore whether such models can be&#10;adapted to child speech to improve ASR for children. In addition, we compare&#10;Whisper child-adaptations with finetuned self-supervised models, such as&#10;wav2vec2. We demonstrate that finetuning Whisper on child speech yields&#10;significant improvements in ASR performance on child speech, compared to non&#10;finetuned Whisper models. Additionally, utilizing self-supervised Wav2vec2&#10;models that have been finetuned on child speech outperforms Whisper finetuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12606" label="2305.12606">
        <attvalues>
          <attvalue for="0" value="Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech&#10;  Pre-Training for Adaptation to Unseen Languages" />
          <attvalue for="1" value="  Recent models such as XLS-R and Whisper have made multilingual speech&#10;technologies more accessible by pre-training on audio from around 100 spoken&#10;languages each. However, there are thousands of spoken languages worldwide, and&#10;adapting to new languages is an important problem. In this work, we aim to&#10;understand which model adapts better to languages unseen during pre-training.&#10;We fine-tune both models on 13 unseen languages and 18 seen languages. Our&#10;results show that the number of hours seen per language and language family&#10;during pre-training is predictive of how the models compare, despite the&#10;significant differences in the pre-training methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.00586" label="2211.00586">
        <attvalues>
          <attvalue for="0" value="T5lephone: Bridging Speech and Text Self-supervised Models for Spoken&#10;  Language Understanding via Phoneme level T5" />
          <attvalue for="1" value="  In Spoken language understanding (SLU), a natural solution is concatenating&#10;pre-trained speech models (e.g. HuBERT) and pretrained language models (PLM,&#10;e.g. T5). Most previous works use pretrained language models with subword-based&#10;tokenization. However, the granularity of input units affects the alignment of&#10;speech model outputs and language model inputs, and PLM with character-based&#10;tokenization is underexplored. In this work, we conduct extensive studies on&#10;how PLMs with different tokenization strategies affect spoken language&#10;understanding task including spoken question answering (SQA) and speech&#10;translation (ST). We further extend the idea to create T5lephone(pronounced as&#10;telephone), a variant of T5 that is pretrained using phonemicized text. We&#10;initialize T5lephone with existing PLMs to pretrain it using relatively&#10;lightweight computational resources. We reached state-of-the-art on NMSQA, and&#10;the T5lephone model exceeds T5 with other types of units on end-to-end SQA and&#10;ST.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken language understanding (SLU) aims to not only decipher but also comprehend audio signals. A well-trained SLU model could be applied to solve tasks such as spoken question answering (SQA) and speech translation (ST). Following the success of self-supervised text pretraining \cite{devlin2018bert, lewis2019bart, raffel2020exploring, he2020deberta}&#10;, self-supervised speech pretraining \cite{DBLP:conf/nips/BaevskiZMA20,DBLP:journals/taslp/HsuBTLSM21,DBLP:journals/corr/abs-2110-13900} &#10;&#10;aims to learn strong speech representation for downstream tasks. Despite reaching near-perfect performance on speech intent classification and speech keyword spotting on SUPERB \cite{yang2021superb}, these models' performance on ST using a randomly initialized transformer decoder \cite{tsai2022superb} is not competitive with works that incorporate knowledge from text \cite{li2020multilingual, conneau2022xtreme}. Pretrained language models (PLM) are also frequently present in previous works solving SQA \cite{chuang2019speechbert,chung2020splat,lin2022dual}. We thus conclude from previous works that incorporating textual knowledge in the system is desired. &#10;&#10;There are two main ways to include pretrained language model in the SLU system. Cascaded approaches utilize raw text as the anchor to link the speech representations and the word representations \cite{shon2022slue,huang2022mtl}. The more recent end-to-end approach seeks to reduce error prorogation by using speech representations directly as the input of pretrained language models \cite{li2020multilingual, lin2022dual}, and it is logical to do so since language models are cross-disciplinary learners \cite{kao2021bert}. However, most works use a subword-based tokenization level language model in their system, such as sentencepiece or byte-pair encoding (bpe). Other tokenization strategies, such as character-based tokenization are often overlooked. For example, the SLUE benchmark \cite{shon2022slue} uses DeBERTa \cite{he2020deberta} in cascaded speech understanding, DUAL \cite{lin2022dual} uses Longformer \cite{beltagy2020longformer} in end-to-end speech question answering, and \cite{li2020multilingual} uses mBART \cite{liu2020multilingual} in the speech translation task. These three pretrained language models all incoporate subword-based tokenization.&#10;&#10;The input unit granularity of the PLM is potentially important in semantic speech tasks for both cascaded and end-to-end methods. For cascaded systems, ASR error degrades performance, but an incorrectly recognized word may have characters that resemble the gold label. Therefore, the character error rate (CER) of the ASR results might be lower than the word error rate (WER), and using PLMs with character-level inputs is theoretically beneficial.&#10; For end-to-end systems utilizing a pretrained language model, mitigating the degree of mismatch between speech representations and the original pretraining text data may also boost system performance. Since self-supervised speech representations have been found to resemble phoneme sequences after clustering and reduction \cite{baevski2021unsupervised} when using 128 clusters, they are much more similar to character level inputs than to subword level inputs.&#10;&#10;In this work, we conduct an extensive study on how self-supervised PLMs with different input granularity affect SQA/ST performance, by inferring on datasets such as NMSQA \cite{lin2022dual} and Covost2 \cite{wang2021covost}. In particular, we compared T5/mT5 with ByT5, which has similar pretraining settings. &#10;&#10;We then further extend the idea to create T5lephone, a variant of T5 that takes phonemicized text as input. T5lephone is realized by self-supervised second-phase pretraining \cite{DBLP:journals/corr/abs-2004-10964} using phonemicized text from Wikipedia, with the model being initialized from mT5/ByT5. We devised a novel way to re-represent the phonemicized text to maximize the transferable knowledge from original text pretraining of ByT5 to our phoneme pretraining. We reached state-of-the-art and +12\% performance gain on previous cascaded NMSQA results\cite{lin2022dual} while using fewer parameters. The performance of our T5lephone model also exceeds previous methods on end-to-end NMSQA and ST.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Translation, Computer Science, Linguistics, Spoken Language Understanding, Pretrained Language Models, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2010.12829" label="2010.12829">
        <attvalues>
          <attvalue for="0" value="Multilingual Speech Translation with Efficient Finetuning of Pretrained&#10;  Models" />
          <attvalue for="1" value="  We present a simple yet effective approach to build multilingual&#10;speech-to-text (ST) translation by efficient transfer learning from pretrained&#10;speech encoder and text decoder. Our key finding is that a minimalistic LNA&#10;(LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and&#10;cross-modality transfer ability by only finetuning less than 10% of the&#10;pretrained parameters. This enables effectively leveraging large pretrained&#10;models with low training cost. Using wav2vec 2.0 for acoustic modeling, and&#10;mBART for multilingual text generation, our approach advanced the new&#10;state-of-the-art for 34 translation directions (and surpassing cascaded ST for&#10;23 of them) on large-scale multilingual ST benchmark CoVoST 2 (+6.4 BLEU on&#10;average across 15 En-X directions and +5.1 BLEU on average across 19 X-En&#10;directions). Our approach demonstrates strong zero-shot performance in a&#10;many-to-many multilingual model (+5.7 BLEU on average across 18 non-English&#10;directions), making it an appealing approach for attaining high-quality speech&#10;translation with improved parameter and data efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.07162" label="2103.07162">
        <attvalues>
          <attvalue for="0" value="Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of&#10;  Pre-trained Models' Transferability" />
          <attvalue for="1" value="  This paper investigates whether the power of the models pre-trained on text&#10;data, such as BERT, can be transferred to general token sequence classification&#10;applications. To verify pre-trained models' transferability, we test the&#10;pre-trained models on text classification tasks with meanings of tokens&#10;mismatches, and real-world non-text token sequence classification data,&#10;including amino acid, DNA, and music. We find that even on non-text data, the&#10;models pre-trained on text converge faster, perform better than the randomly&#10;initialized models, and only slightly worse than the models using task-specific&#10;knowledge. We also find that the representations of the text and non-text&#10;pre-trained models share non-trivial similarities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.08199" label="2409.08199">
        <attvalues>
          <attvalue for="0" value="AudioBERT: Audio Knowledge Augmented Language Model" />
          <attvalue for="1" value="  Recent studies have identified that language models, pretrained on text-only&#10;datasets, often lack elementary visual knowledge, \textit{e.g.,} colors of&#10;everyday objects. Motivated by this observation, we ask whether a similar&#10;shortcoming exists in terms of the \textit{auditory} knowledge. To answer this&#10;question, we construct a new dataset called AuditoryBench, which consists of&#10;two novel tasks for evaluating auditory knowledge. Based on our analysis using&#10;the benchmark, we find that language models also suffer from a severe lack of&#10;auditory knowledge. To address this limitation, we propose AudioBERT, a novel&#10;method to augment the auditory knowledge of BERT through a retrieval-based&#10;approach. First, we detect auditory knowledge spans in prompts to query our&#10;retrieval model efficiently. Then, we inject audio knowledge into BERT and&#10;switch on low-rank adaptation for effective adaptation when audio knowledge is&#10;required. Our experiments demonstrate that AudioBERT is quite effective,&#10;achieving superior performance on the AuditoryBench. The dataset and code are&#10;available at \bulurl{https://github.com/HJ-Ok/AudioBERT}.&#10;" />
          <attvalue for="2" value="&#10;&#10;The advance of pretrained language models has spurred significant improvements across various language-related tasks \cite{devlin2019bert, radford2019language}, and has been extended to processing multimodal information \cite{llava,pengi}. However, a major limitation of popular language models is that they are pretrained only on textual data, which can lead to gaps in knowledge from other domains. Indeed, in visual domains, researchers have repeatedly found that common language models lack sufficient visual commonsense knowledge---such as the color of common objects---leading to a poor performance on visual tasks \cite{zhang2022visual, liu2022things, alper2023bert, rahmanzadehgervi2024vision}. In response, recent studies have proposed algorithms to augment language models with visual knowledge \cite{tan2020vokenization, lu2022imagination, wangvisually, tang2023learning}. &#10;&#10;Does the same limitation hold for auditory commonsense knowledge? This question, unfortunately, has not been clearly addressed yet. Although a recent work studies the effectiveness of audio snippet embeddings in the language model representation space \cite{ngo2024language}, it is not known whether the language models have rich commonsense knowledge regarding the auditory signals, e.g., which animal make a specific sound (\ref{fig:our_task}).&#10;&#10;To answer this question, we present AuditoryBench, the first benchmark dataset (to our knowledge) for evaluating the language models' auditory knowledge. In particular, we propose two auditory knowledge tasks: (1) animal sound recognition and (2) sound pitch comparison. The animal sound recognition task asks the language model to predict which animal is likely to make a sound that corresponds to specific echoing words (i.e., onomatopoeia), such as ``meow.'' The sound pitch comparison task asks the language model to predict which sound source (e.g., musical instruments, objects, or environments) is likelier to produce sound with a higher pitch. To construct this benchmark, we propose an LLM-based data-processing pipeline for the sake of the scalability of the benchmark dataset \cite{kim2023soda,mei2024wavcaps}.&#10;&#10;Using AuditoryBench, we discover that language models severely lack auditory commonsense knowledge. In particular, we test three different language models---BERT\cite{devlin2019bert}, Gemma\cite{team2024gemma}, and LLaMA\cite{dubey2024llama}---and find that all models achieve low predictive accuracy in both benchmark tasks (\ref{tab:audio_knowledge_recognition}).&#10;&#10;To address this shortcoming, we propose AudioBERT, a simple yet effective retrieval-based framework for injecting auditory knowledge into language models. Our approach involves detecting text spans where auditory knowledge is necessary. Whenever needed, relevant audio is retrieved by querying the detected text span to CLAP \cite{elizalde2023clap}, a model that measures the text-audio similarity. Then, the embedding of the retrieved audio sample is injected into the language model. Upon identification of auditory spans by the detector, the language model activates Low-Rank Adaptation (LoRA)\cite{hu2022lora} weights, which is finetuned with AudiotoryBench, which maintains its pretrained knowledge makes the model perform well in other tasks by deactivating LoRA weights. Our experiments demonstrate that the proposed method is quite effective, improving the prediction accuracy on AuditoryBench by more than 40\% in the test set. As far as we know, AudioBERT is the first algorithm to augment language models with auditory commonsense knowledge.&#10;&#10;The key contributions of our work are threefold:&#10;\begin{itemize}&#10;\item We introduce the first benchmark dataset for evaluating the auditory knowledge of language models via an efficient LLM-based data generation pipeline.&#10;&#10;\item We discover that common language models pretrained on text-only datasets suffer from a severe lack of auditory commonsense knowledge.&#10;&#10;\item We propose a novel and effective framework to inject auditory knowledge into pretrained language models, which adaptively retrieves and utilizes relevant auditory knowledge, utilizing switch-on-off LoRA to preserve pretrained knowledge.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Limitations, Auditory Knowledge Evaluation, Multimodal Model Enhancement, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.06775" label="2010.06775">
        <attvalues>
          <attvalue for="0" value="Vokenization: Improving Language Understanding with Contextualized,&#10;  Visual-Grounded Supervision" />
          <attvalue for="1" value="  Humans learn language by listening, speaking, writing, reading, and also, via&#10;interaction with the multimodal real world. Existing language pre-training&#10;frameworks show the effectiveness of text-only self-supervision while we&#10;explore the idea of a visually-supervised language model in this paper. We find&#10;that the main reason hindering this exploration is the large divergence in&#10;magnitude and distributions between the visually-grounded language datasets and&#10;pure-language corpora. Therefore, we develop a technique named &quot;vokenization&quot;&#10;that extrapolates multimodal alignments to language-only data by contextually&#10;mapping language tokens to their related images (which we call &quot;vokens&quot;). The&#10;&quot;vokenizer&quot; is trained on relatively small image captioning datasets and we&#10;then apply it to generate vokens for large language corpora. Trained with these&#10;contextually generated vokens, our visually-supervised language models show&#10;consistent improvements over self-supervised alternatives on multiple&#10;pure-language tasks such as GLUE, SQuAD, and SWAG. Code and pre-trained models&#10;publicly available at https://github.com/airsplay/vokenization&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.00118" label="2408.00118">
        <attvalues>
          <attvalue for="0" value="Gemma 2: Improving Open Language Models at a Practical Size" />
          <attvalue for="1" value="  In this work, we introduce Gemma 2, a new addition to the Gemma family of&#10;lightweight, state-of-the-art open models, ranging in scale from 2 billion to&#10;27 billion parameters. In this new version, we apply several known technical&#10;modifications to the Transformer architecture, such as interleaving&#10;local-global attentions (Beltagy et al., 2020a) and group-query attention&#10;(Ainslie et al., 2023). We also train the 2B and 9B models with knowledge&#10;distillation (Hinton et al., 2015) instead of next token prediction. The&#10;resulting models deliver the best performance for their size, and even offer&#10;competitive alternatives to models that are 2-3 times bigger. We release all&#10;our models to the community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.12901" label="2404.12901">
        <attvalues>
          <attvalue for="0" value="Large Language Models for Networking: Workflow, Advances and Challenges" />
          <attvalue for="1" value="  The networking field is characterized by its high complexity and rapid&#10;iteration, requiring extensive expertise to accomplish network tasks, ranging&#10;from network design, configuration, diagnosis and security. The inherent&#10;complexity of these tasks, coupled with the ever-changing landscape of&#10;networking technologies and protocols, poses significant hurdles for&#10;traditional machine learning-based methods. These methods often struggle to&#10;generalize and automate complex tasks in networking, as they require extensive&#10;labeled data, domain-specific feature engineering, and frequent retraining to&#10;adapt to new scenarios. However, the recent emergence of large language models&#10;(LLMs) has sparked a new wave of possibilities in addressing these challenges.&#10;LLMs have demonstrated remarkable capabilities in natural language&#10;understanding, generation, and reasoning. These models, trained on extensive&#10;data, can benefit the networking domain. Some efforts have already explored the&#10;application of LLMs in the networking domain and revealed promising results. By&#10;reviewing recent advances, we present an abstract workflow to describe the&#10;fundamental process involved in applying LLM for Networking. We introduce the&#10;highlights of existing works by category and explain in detail how they operate&#10;at different stages of the workflow. Furthermore, we delve into the challenges&#10;encountered, discuss potential solutions, and outline future research&#10;prospects. We hope that this survey will provide insight for researchers and&#10;practitioners, promoting the development of this interdisciplinary research&#10;field.&#10;" />
          <attvalue for="2" value="&#10;Networks, including wide-area internet, data center networks, and satellite networks, are critical infrastructures with distinct design principles and architectures. &#10;Due to the complexity and rapid iteration in the field of networking, managing and maintaining these networks involves complex operations that demand rich network expertise.&#10;Efficient solutions for network tasks such as network design, network configuration, network diagnosis, and network security are essential for ensuring network stability and scalability. &#10;These tasks have garnered significant attention from both academia and industry.&#10;Although several machine learning (ML)-based methods have been proposed and shown effectiveness, they often lack generalization and necessitate repeated designs for different scenarios~\cite{boutaba2018comprehensive}.&#10;&#10;Recently, there has been significant progress in large language models (LLMs), representing the latest advancement in generative AI.&#10;Pretrained on extensive data, these models reveal remarkable capabilities in concept understanding, mathematical reasoning, and tool usage, enabling groundbreaking advancements in various domains, such as chip design, protein structure generation, and robot-embodied intelligence.&#10;Inspired by these promising endeavors, some researchers have also explored the application of LLMs in the field of networking and demonstrated encouraging results.&#10;In particular, LLMs offer significant advantages in networking due to their ability to process natural language input and output, eliminating the need for strict data modeling and feature extraction. This reduces task workloads. They can also construct logical chains for complex problem-solving by leveraging domain knowledge, and their transfer learning capability allows them to apply insights from related fields to new network challenges, enhancing their flexibility and applicability.&#10;&#10;Given the growing interest in large language models for networking (LLMN), we believe that it is the right time to survey the current literature.&#10;Based on the in-depth analysis of existing works, we first propose an abstract workflow to describe the fundamental process involved in LLMN.&#10;The LLMN workflow~(Section~\ref{sec:workflow}) consists of six stages: task definition, data representation, prompt engineering, model evolution, tools integration, and validation. &#10;These stages aim to address complex tasks, handle diverse data types, guide LLMs in generating accurate answers, expand LLM capabilities through tool integration, and ensure performance and output validation. This workflow provides a practical roadmap for researchers entering the field of LLMN.&#10;&#10;We then conduct a targeted review (Section~\ref{sec:advances}) of notable advancements in networking facilitated by LLMs, predominantly sourced from recent publications.&#10;For example, LLMs can assist in designing algorithms and managing network topologies~\cite{he2024llm,mani2023enhancing}, enhance configuration efficiency while minimizing errors~\cite{mondal2023llms,lian2023configuration}, identify hidden patterns or anomalies~\cite{zhou2023towards,kotaru2023adapting}, and introduce novel strategies for improving network security~\cite{meng2024large}.&#10;We categorize these advancements into several key fields of networking, including network design, network configuration, network diagnosis, and network security, and elaborate on how they are performed at each stage of the LLMN workflow.&#10;&#10;In addition to the literature review, we also identify several challenges~(Section~\ref{sec:challenges}) that necessitate further research.&#10;These challenges span across different stages of the LLMN workflow and can be categorized into six aspects: intelligent planning, multimodal data understanding, network-specific LLM construction, autonomous tool utilization by LLMs, reliability and safety assurance, and efficiency and real-time performance improvement. &#10;Potential approaches to address them are also discussed.&#10;&#10;Compared to a prior publication that merely focuses on a domain-adapted LLM framework with access to various external network tools~\cite{huang2023large}, our comprehensive survey presents a condensed overview of existing LLMN research and the lessons learned from it.&#10;Moreover, based on the understanding of existing research, we have abstracted a universal workflow that includes essential mechanisms such as validation and feedback.&#10;This survey aims to provide a foundational and practical research roadmap for newcomers to the LLMN field, while also enabling experienced network professionals to quickly grasp the transformative potential of LLMs within their domain. Additionally, we hope this article serves as a catalyst for further research endeavors by inspiring networking and artificial intelligence (AI) experts to delve into more profound investigations.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning Limitations, Computer Science, Network Complexity Challenges, Information Technology, Large Language Models, Computer Networking, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2404.01617" label="2404.01617">
        <attvalues>
          <attvalue for="0" value="LLM-ABR: Designing Adaptive Bitrate Algorithms via Large Language Models" />
          <attvalue for="1" value="  We present LLM-ABR, the first system that utilizes the generative&#10;capabilities of large language models (LLMs) to autonomously design adaptive&#10;bitrate (ABR) algorithms tailored for diverse network characteristics.&#10;Operating within a reinforcement learning framework, LLM-ABR empowers LLMs to&#10;design key components such as states and neural network architectures. We&#10;evaluate LLM-ABR across diverse network settings, including broadband,&#10;satellite, 4G, and 5G. LLM-ABR consistently outperforms default ABR algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.09690" label="2310.09690">
        <attvalues>
          <attvalue for="0" value="Configuration Validation with Large Language Models" />
          <attvalue for="1" value="  Misconfigurations are major causes of software failures. Existing practices&#10;rely on developer-written rules or test cases to validate configurations, which&#10;are expensive. Machine learning (ML) for configuration validation is considered&#10;a promising direction, but has been facing challenges such as the need of&#10;large-scale field data and system-specific models. Recent advances in Large&#10;Language Models (LLMs) show promise in addressing some of the long-lasting&#10;limitations of ML-based configuration validation. We present a first analysis&#10;on the feasibility and effectiveness of using LLMs for configuration&#10;validation. We empirically evaluate LLMs as configuration validators by&#10;developing a generic LLM-based configuration validation framework, named Ciri.&#10;Ciri employs effective prompt engineering with few-shot learning based on both&#10;valid configuration and misconfiguration data. Ciri checks outputs from LLMs&#10;when producing results, addressing hallucination and nondeterminism of LLMs. We&#10;evaluate Ciri's validation effectiveness on eight popular LLMs using&#10;configuration data of ten widely deployed open-source systems. Our analysis (1)&#10;confirms the potential of using LLMs for configuration validation, (2) explores&#10;design space of LLMbased validators like Ciri, and (3) reveals open challenges&#10;such as ineffectiveness in detecting certain types of misconfigurations and&#10;biases towards popular configuration parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.17474" label="2311.17474">
        <attvalues>
          <attvalue for="0" value="Large Language Models for Networking: Applications, Enabling Techniques,&#10;  and Challenges" />
          <attvalue for="1" value="  The rapid evolution of network technologies and the growing complexity of&#10;network tasks necessitate a paradigm shift in how networks are designed,&#10;configured, and managed. With a wealth of knowledge and expertise, large&#10;language models (LLMs) are one of the most promising candidates. This paper&#10;aims to pave the way for constructing domain-adapted LLMs for networking.&#10;Firstly, we present potential LLM applications for vertical network fields and&#10;showcase the mapping from natural language to network language. Then, several&#10;enabling technologies are investigated, including parameter-efficient&#10;finetuning and prompt engineering. The insight is that language understanding&#10;and tool usage are both required for network LLMs. Driven by the idea of&#10;embodied intelligence, we propose the ChatNet, a domain-adapted network LLM&#10;framework with access to various external network tools. ChatNet can reduce the&#10;time required for burdensome network planning tasks significantly, leading to a&#10;substantial improvement in efficiency. Finally, key challenges and future&#10;research directions are highlighted.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.06125" label="2308.06125">
        <attvalues>
          <attvalue for="0" value="Improving Joint Speech-Text Representations Without Alignment" />
          <attvalue for="1" value="  The last year has seen astonishing progress in text-prompted image generation&#10;premised on the idea of a cross-modal representation space in which the text&#10;and image domains are represented jointly. In ASR, this idea has found&#10;application as joint speech-text encoders that can scale to the capacities of&#10;very large parameter models by being trained on both unpaired speech and text.&#10;While these methods show promise, they have required special treatment of the&#10;sequence-length mismatch inherent in speech and text, either by up-sampling&#10;heuristics or an explicit alignment model. In this work, we offer evidence that&#10;joint speech-text encoders naturally achieve consistent representations across&#10;modalities by disregarding sequence length, and argue that consistency losses&#10;could forgive length differences and simply assume the best alignment. We show&#10;that such a loss improves downstream WER in both a large-parameter monolingual&#10;and multilingual system.&#10;" />
          <attvalue for="2" value="&#10;The power of very large models trained on vast unsupervised corpora in a single modality has become increasingly clear. This has been demonstrated in the text domain where language models have achieved unprecedented zero-shot capabilities \cite{Brown20, Chowdhery22}, as well as in the audio domain, in which a single model has been shown to be adaptable to a surprisingly wide array of acoustic tasks \cite{Yang21, Borsos22}. These successes have given rise to the question of how to apply these methods for problems involving two modalities, which historically have depended on manually paired data.&#10;&#10;One very promising solution to this problem is to train a large encoder on both modalities, such that either modality may be provided as an unpaired example, but which learns to map paired examples to similar points in representation space. In the image/text domain, such a representation has proved achievable and capable of attaining state-of-the-art performance on many image and text comprehension tasks in a single model \cite{Alayrac22, Cho21}.&#10;&#10;In the audio/text domain, joint speech and text models have been utilized for a wide range of tasks \cite{Renduchintala18, Huang20, Mariooryad22}. In speech recognition, the past few years has seen a trend toward models with a joint speech and text encoder to allow pretraining on unpaired speech and text data \cite{Tang20, Bapna22, Chen22, Sainath22}. However, speech recognition presents the particular challenge of two sequence modalities, one of which (speech) is typically represented by a much longer sequence than the other (text). This complicates the task of representing both modalities in the same embedding space, since we cannot make a direct, frame-wise comparison of an encoder's speech representation to its text representation. &#10;&#10;This complication has largely been handled either by upsampling or an explicit alignment model. Fixed upsampling of the text inputs has been applied successfully for ASR in \cite{Sainath22} and SLU in \cite{Thomas22}, proving that an approximate alignment is sufficient for learning a joint representation. On the other hand, \cite{Chen22_2} addresses the problem with a separately-trained alignment model that aims for perfect alignment. In \cite{Chen22}, it's shown that such an alignment model permits the use of \say{consistency} regularization in which the encoder's outputs on corresponding speech and text are compared frame-wise and pushed together in representation space. \cite{Chen22} goes on to show that \say{consistency} regularization yields a more closely joined representation space leading to better WER. &#10;&#10;Consistency regularization itself follows naturally from the literature on generative models. Systems like autoencoders applied to augmented data (e.g. \cite{Chadebec_2022}) explicitly push representations of matched examples together, while contrastive systems like \cite{Chen20} do the same implicitly. The success of the same idea in speech using an explicit alignment begs the question of if the same can be done with an implicit alignmentment; that is, without knowing the particular alignment between speech and text.&#10;&#10;In this paper, we ask if consistency regularization may be applied using the implicit alignments learned in upsampling systems like \cite{Sainath22} to achieve the performance improvements seen with the explicit alignments in \cite{Chen22}. To this end, we develop an algorithm inspired by dynamic time warping \cite{Sakoe78} that finds the best possible alignment between an encoder's representation of a paired speech and text example. We measure the quality of this best alignment in a system without an explicit alignment model and show that that it is not only learned during training but in fact improves at deeper layers of the network. Inspired by the improvements shown in \cite{Chen22_2} and \cite{Chen22}, we then show that by changing the criteria of the consistency regularization to encourage consistency under some alignment, instead of a direct frame-wise comparison, we can achieve robust WER improvements against strong, semi-supervised baselines in both a monolingual and multilingual setting, all without any learned alignment model. Our results suggest that enforcing consistency in cross-modal representations can be done by simply forgiving misalignment. &#10;&#10;The rest of this paper proceeds as follows. Section 2 specifies our setup for joint speech/text modeling and consistency regularization, and the details of our best-alignment algorithm. Section 3 specifies details of our data and training. Section 4 presents our analysis of the best alignment in an unregularized model and the results of optimizing that alignment with our consistency loss. We conclude in Section 5.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Speech Text Alignment, Multimodal Representation, Artificial Intelligence, Signal Processing" />
        </attvalues>
      </node>
      <node id="1803.10299" label="1803.10299">
        <attvalues>
          <attvalue for="0" value="Multi-Modal Data Augmentation for End-to-End ASR" />
          <attvalue for="1" value="  We present a new end-to-end architecture for automatic speech recognition&#10;(ASR) that can be trained using \emph{symbolic} input in addition to the&#10;traditional acoustic input. This architecture utilizes two separate encoders:&#10;one for acoustic input and another for symbolic input, both sharing the&#10;attention and decoder parameters. We call this architecture a multi-modal data&#10;augmentation network (MMDA), as it can support multi-modal (acoustic and&#10;symbolic) input and enables seamless mixing of large text datasets with&#10;significantly smaller transcribed speech corpora during training. We study&#10;different ways of transforming large text corpora into a symbolic form suitable&#10;for training our MMDA network. Our best MMDA setup obtains small improvements&#10;on character error rate (CER), and as much as 7-10\% relative word error rate&#10;(WER) improvement over a baseline both with and without an external language&#10;model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11338" label="2010.11338">
        <attvalues>
          <attvalue for="0" value="A General Multi-Task Learning Framework to Leverage Text Data for Speech&#10;  to Text Tasks" />
          <attvalue for="1" value="  Attention-based sequence-to-sequence modeling provides a powerful and elegant&#10;solution for applications that need to map one sequence to a different&#10;sequence. Its success heavily relies on the availability of large amounts of&#10;training data. This presents a challenge for speech applications where labelled&#10;speech data is very expensive to obtain, such as automatic speech recognition&#10;(ASR) and speech translation (ST). In this study, we propose a general&#10;multi-task learning framework to leverage text data for ASR and ST tasks. Two&#10;auxiliary tasks, a denoising autoencoder task and machine translation task, are&#10;proposed to be co-trained with ASR and ST tasks respectively. We demonstrate&#10;that representing text input as phoneme sequences can reduce the difference&#10;between speech and text inputs, and enhance the knowledge transfer from text&#10;corpora to the speech to text tasks. Our experiments show that the proposed&#10;method achieves a relative 10~15% word error rate reduction on the English&#10;Librispeech task compared with our baseline, and improves the speech&#10;translation quality on the MuST-C tasks by 3.6~9.2 BLEU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.00006" label="2203.00006">
        <attvalues>
          <attvalue for="0" value="Towards Reducing the Need for Speech Training Data To Build Spoken&#10;  Language Understanding Systems" />
          <attvalue for="1" value="  The lack of speech data annotated with labels required for spoken language&#10;understanding (SLU) is often a major hurdle in building end-to-end (E2E)&#10;systems that can directly process speech inputs. In contrast, large amounts of&#10;text data with suitable labels are usually available. In this paper, we propose&#10;a novel text representation and training methodology that allows E2E SLU&#10;systems to be effectively constructed using these text resources. With very&#10;limited amounts of additional speech, we show that these models can be further&#10;improved to perform at levels close to similar systems built on the full speech&#10;datasets. The efficacy of our proposed approach is demonstrated on both intent&#10;and entity tasks using three different SLU datasets. With text-only training,&#10;the proposed system achieves up to 90% of the performance possible with full&#10;speech training. With just an additional 10% of speech data, these models&#10;significantly improve further to 97% of full performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00026" label="2105.00026">
        <attvalues>
          <attvalue for="0" value="Data Augmentation in High Dimensional Low Sample Size Setting Using a&#10;  Geometry-Based Variational Autoencoder" />
          <attvalue for="1" value="  In this paper, we propose a new method to perform data augmentation in a&#10;reliable way in the High Dimensional Low Sample Size (HDLSS) setting using a&#10;geometry-based variational autoencoder. Our approach combines a proper latent&#10;space modeling of the VAE seen as a Riemannian manifold with a new generation&#10;scheme which produces more meaningful samples especially in the context of&#10;small data sets. The proposed method is tested through a wide experimental&#10;study where its robustness to data sets, classifiers and training samples size&#10;is stressed. It is also validated on a medical imaging classification task on&#10;the challenging ADNI database where a small number of 3D brain MRIs are&#10;considered and augmented using the proposed VAE framework. In each case, the&#10;proposed method allows for a significant and reliable gain in the&#10;classification metrics. For instance, balanced accuracy jumps from 66.3% to&#10;74.3% for a state-of-the-art CNN classifier trained with 50 MRIs of cognitively&#10;normal (CN) and 50 Alzheimer disease (AD) patients and from 77.7% to 86.3% when&#10;trained with 243 CN and 210 AD while improving greatly sensitivity and&#10;specificity metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.16611" label="2210.16611">
        <attvalues>
          <attvalue for="0" value="Application of Knowledge Distillation to Multi-task Speech&#10;  Representation Learning" />
          <attvalue for="1" value="  Model architectures such as wav2vec 2.0 and HuBERT have been proposed to&#10;learn speech representations from audio waveforms in a self-supervised manner.&#10;When they are combined with downstream tasks such as keyword spotting and&#10;speaker verification, they provide state-of-the-art performance. However, these&#10;models use a large number of parameters, the smallest version of which has 95&#10;million parameters. This constitutes a challenge for edge AI device&#10;deployments. In this paper, we investigate the application of knowledge&#10;distillation to speech representation learning (SRL) models followed by joint&#10;fine-tuning with multiple downstream voice-activated tasks. In our experiments&#10;on two such tasks, our approach results in nearly 75% reduction in model size&#10;while suffering only 0.1% accuracy and 0.9% equal error rate degradation&#10;compared to the full-size model. In addition, we show that fine-tuning the SRL&#10;models results in a significant performance boost compared to using frozen SRL&#10;models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech representation learning (SRL) has been extensively studied in the literature and shown potential for various speech recognition tasks \cite{srl_1, srl_2, srl_4}. Deep neural network architectures \cite{deep, deep2} are the method of choice for SRL. Since sufficient amounts of labeled data are not typically available for supervised learning in real scenarios, this problem is circumvented by using self-supervised learning on unlabeled data \cite{wav2vec, hubert, tera, selfsupervised_1, selfsupervised_2}. In \cite{wav2vec, hubert}, convolutional networks are combined with transformer encoder-based models to learn speech representations in a self-supervised manner. \cite{emotionrecognition, emotionrecognition_example, speaker_recognition} propose to combine such a model with a downstream network to address classification problems such as emotion recognition or speaker recognition. The resulting models provide competitive performance on these tasks but require training multiple large networks each for a different single task. To enable models for multi-tasking, \cite{multitask} proposes to learn speech representations from raw audio with wav2vec 2.0 \cite{wav2vec} and then fine-tune the model for multiple tasks simultaneously by sampling training instances from the combined task-specific datasets. This method provides even better performance than training on single tasks separately.&#10;&#10;However, the memory footprint of the resulting models is large with the number of parameters ranging from 95 million to 1 billion, which makes them difficult to deploy on edge AI devices. This problem is studied with parameter sharing in \cite{review_w2v2}. Knowledge distillation is another method of choice to address model complexity reduction in various fields ranging from natural language processing to speech recognition \cite{kd_survey, kd_app2, kd_app1}. A common approach in the literature is to reduce model size by maintaining its performance via teacher-student network distillation architectures. \cite{shrinking,review_lighthubert} apply knowledge distillation to large SRL methods to construct efficient speech recognition networks. The conventional approach is transferring knowledge from the final layer of the teacher network to the final prediction of the student network. However, the middle layers of these SRL models also contain valuable speech representation information \cite{layer_analysis}. In \cite{distilhubert}, layer-wise distillation to the HuBERT \cite{hubert} model is applied where knowledge is transferred from middle layers of the large network to construct a smaller model.&#10;&#10;In our work, we discuss generalization of knowledge distillation to use a large network in training of another network with reduced model size, and we use a special knowledge distillation approach as presented in \cite{distilhubert} in our experiments.&#10;As opposed to this approach, we combine the student model with linear downstream heads and fine-tune all network parameters via a multi-task training scheme as in \cite{multitask} instead of freezing the SRL model as done in \cite{distilhubert}. In our experiments, we address keyword spotting and speaker verification problems in single task and multi-task frameworks. Through our results, we show that the constructed student networks perform as well as the teacher models even though the former model size is 28\% of the latter. &#10;Moreover, we perform this in two settings where we construct distilled SRL modules with wav2vec 2.0 and HuBERT separately for a comprehensive performance comparison. &#10;&#10;To the best of our knowledge, we propose for the first time the combination of knowledge distillation application to SRL and joint fine-tuning of the complete model (SRL module and downstream heads) for multiple downstream voice-activated tasks.&#10;This is in contrast to common speech recognition multi-task training approaches where an SRL module is frozen, functioning as a feature extractor, and only the downstream speech recognition tasks are trained \cite{distilhubert, superb}.&#10;On the contrary, we show that fine-tuning both distilled SRL module and downstream heads for multiple voice-activated tasks simultaneously achieves significantly higher performance on all such tasks compared to fine-tuning only the downstream heads with frozen SRL module.&#10;&#10;The main contributions of this work are as follows:&#10;&#10;\begin{itemize}&#10;&#09;\item We successfully combine knowledge distillation and multi-task training to construct a single network (with about 75\% reduced size) that can be efficiently used in devices and embedded systems for multiple tasks.&#10;&#09;\item We show that training the complete network end-to-end (both SRL module and downstream heads) on multiple tasks achieves significantly higher performance than training on the same tasks with the frozen SRL module.&#10;&#09;\item We discuss generalization of the used methods (knowledge distillation and multi-task training) and show that they can be efficiently applied to different networks where we achieve competitive results with both distilled wav2vec 2.0 and distilled HuBERT models on keyword spotting (KWS) and speaker verification (SV) voice-activated tasks.&#10;\end{itemize}&#10;&#10;The paper is organized as follows. Section \ref{sec:method} describes our approach in detail, Section \ref{sec:exp} presents experiments conducted to validate our approach and we conclude with Section \ref{sec:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Edge AI Optimization, Knowledge Distillation, Speech Representation Learning, Signal Processing" />
        </attvalues>
      </node>
      <node id="2110.01077" label="2110.01077">
        <attvalues>
          <attvalue for="0" value="Multi-task Voice Activated Framework using Self-supervised Learning" />
          <attvalue for="1" value="  Self-supervised learning methods such as wav2vec 2.0 have shown promising&#10;results in learning speech representations from unlabelled and untranscribed&#10;speech data that are useful for speech recognition. Since these representations&#10;are learned without any task-specific supervision, they can also be useful for&#10;other voice-activated tasks like speaker verification, keyword spotting,&#10;emotion classification etc. In our work, we propose a general purpose framework&#10;for adapting a pre-trained wav2vec 2.0 model for different voice-activated&#10;tasks. We develop downstream network architectures that operate on the&#10;contextualized speech representations of wav2vec 2.0 to adapt the&#10;representations for solving a given task. Finally, we extend our framework to&#10;perform multi-task learning by jointly optimizing the network parameters on&#10;multiple voice activated tasks using a shared transformer backbone. Both of our&#10;single and multi-task frameworks achieve state-of-the-art results in speaker&#10;verification and keyword spotting benchmarks. Our best performing models&#10;achieve 1.98% and 3.15% EER on VoxCeleb1 test set when trained on VoxCeleb2 and&#10;VoxCeleb1 respectively, and 98.23% accuracy on Google Speech Commands v1.0&#10;keyword spotting dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.14779" label="2309.14779">
        <attvalues>
          <attvalue for="0" value="Exploring Small Language Models with Prompt-Learning Paradigm for&#10;  Efficient Domain-Specific Text Classification" />
          <attvalue for="1" value="  Domain-specific text classification faces the challenge of scarce labeled&#10;data due to the high cost of manual labeling. Prompt-learning, known for its&#10;efficiency in few-shot scenarios, is proposed as an alternative to traditional&#10;fine-tuning methods. And besides, although large language models (LLMs) have&#10;gained prominence, small language models (SLMs, with under 1B parameters) offer&#10;significant customizability, adaptability, and cost-effectiveness for&#10;domain-specific tasks, given industry constraints. In this study, we&#10;investigate the potential of SLMs combined with prompt-learning paradigm for&#10;domain-specific text classification, specifically within customer-agent&#10;interactions in retail. Our evaluations show that, in few-shot settings when&#10;prompt-based model fine-tuning is possible, T5-base, a typical SLM with 220M&#10;parameters, achieve approximately 75% accuracy with limited labeled data (up to&#10;15% of full data), which shows great potentials of SLMs with prompt-learning.&#10;Based on this, We further validate the effectiveness of active few-shot&#10;sampling and the ensemble strategy in the prompt-learning pipeline that&#10;contribute to a remarkable performance gain. Besides, in zero-shot settings&#10;with a fixed model, we underscore a pivotal observation that, although the&#10;GPT-3.5-turbo equipped with around 154B parameters garners an accuracy of&#10;55.16%, the power of well designed prompts becomes evident when the&#10;FLAN-T5-large, a model with a mere 0.5% of GPT-3.5-turbo's parameters, achieves&#10;an accuracy exceeding 31% with the optimized prompt, a leap from its sub-18%&#10;performance with an unoptimized one. Our findings underscore the promise of&#10;prompt-learning in classification tasks with SLMs, emphasizing the benefits of&#10;active few-shot sampling, and ensemble strategies in few-shot settings, and the&#10;importance of prompt engineering in zero-shot settings.&#10;" />
          <attvalue for="2" value="&#10;&#10;In today's digital age, the explosion of textual data across platforms such as social media, online forums, and customer interactions is unmistakable. Interpreting this enormous amount of data poses challenges for businesses and researchers alike. At the heart of this endeavor is text classification, a pivotal tool for gleaning insights and facilitating informed decision-making.&#10;&#10;Industry domain-specific text classification is of paramount importance. In domains such as customer support, discerning customer intentions accurately can expedite response times and route inquiries more effectively. The acquisition of labeled data, essential for model training, remains a significant hurdle. This data acquisition is labor-intensive and often requires domain-specific expertise. Even with advances in text classification, tailoring these models for niche tasks often demands substantial resources and data, prompting the search for methods with diminished reliance on labeled datasets.&#10;&#10;The recent advent of pre-trained language models, such as BERT \cite{devlin2018bert}, GPT \cite{radford2018gpt}, and RoBERTa \cite{liu2019roberta}, has brought about transformative changes in NLP. These models, pre-trained on vast text repositories, have captured intricate language patterns and representations, enabling them to be fine-tuned for a variety of downstream tasks. However, conventional fine-tuning approaches require a significant amount of labeled data, which can be a constraint, especially in domain-specific applications.&#10;&#10;Here, prompt-learning emerges as a promising alternative. This technique capitalizes on the rich representations within pre-trained models, using specific prompts to guide them towards desired outputs, effectively transforming them into classifiers. This method can be particularly advantageous in contexts with limited labeled data, as it reduces the dependence on extensive datasets. In this landscape, there have been some advancements in prompt-learning, both in few-shot and zero-shot settings.&#10;In the realm of few-shot settings where model fine-tuning is allowed, recent models such as PET-TC \cite{schick-schutze-2021-exploiting}, PET-Gen \cite{schick2020few}, and LM-BFF \cite{gao-etal-2021-making} have demonstrated the potential of prompt-learning. For instance, LM-BFF has introduced innovative techniques like automated template generation and label word selection, significantly reducing the need for manual prompt engineering \cite{gao-etal-2021-making}. These models hold promise for domain-specific text classification, a focus of our research, where acquiring extensive labeled data can be challenging. Conversely, in zero-shot settings, models like LAMA \cite{petroni-etal-2019-language} and GPT-3 \cite{brown2020language} have explored tuning-free prompting. This approach relies on the careful engineering of prompts for achieving desired outputs. Our work builds upon these foundations to explore the effectiveness of prompt-learning, especially when coupled with smaller language models, within the specific context of retail interactions.&#10;&#10;In addition to advancements of the prompt-learning paradigm, while large language models (LLMs) continue to dominate discussions, there's an emerging appreciation for smaller language models (SLMs, with under 1 billion parameters). These SLMs, while potentially less powerful, can be specialized for specific tasks, making them more efficient and economical.&#10;&#10;The convergence of these research leads us to the primary objective of this paper: to evaluate the effectiveness of text classification based on prompt-learning methodology, especially when paired with SLMs. Here we do a case study of the text classification task based on the domain of retail interactions. Given this backdrop, the core contributions and findings of this paper, highlighted upfront for clarity, are:&#10;\begin{itemize}&#10; \item In few-shot settings, we showcase the potential of SLMs combined with the prompt-learning approach for domain-specific text classification, specifically in retail interactions, achieving around 75\% accuracy with 15\% of full labeled data using T5-base, which shows strong competitiveness to the traditional fine-tuning approach.&#10; \item In few-shot settings, we validate the active few-shot sampling and ensemble strategies in the prompt-learning process, leading to a notable performance boost.&#10; \item In zero-shot settings, we underscore that, although the giant GPT-3.5-turbo with approximately 154B parameters achieves an impressive 55.16\% accuracy, our optimized prompts enable significantly smaller models like FLAN-T5-large to achieve over 30\% accuracy, reaching over half the performance of GPT-3.5-turbo with a fraction of the parameters.&#10;\end{itemize}&#10;&#10;With these contributions in perspective, the remainder of this paper is structured as follows: Section 2 offers a brief introduction to the dataset we utilize. Section 3 elucidates our prompt-learning pipeline, detailing methodologies and experimental setups for both few-shot and zero-shot scenarios. Section 4 presents our findings and an in-depth analysis. After introducing related work in Section 5, we conclude with insights and potential directions for future research in Section 6. Through our experiments, we aim to uncover an optimized strategy for deploying prompt-learning with SLMs, especially in contexts with limited labeled data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Classification, Computer Science, Linguistics, Prompt Learning Strategies, Artificial Intelligence, Information Science, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2404.00829" label="2404.00829">
        <attvalues>
          <attvalue for="0" value="Returning to the Start: Generating Narratives with Related Endpoints" />
          <attvalue for="1" value="  Human writers often bookend their writing with ending sentences that relate&#10;back to the beginning sentences in order to compose a satisfying narrative that&#10;&quot;closes the loop.&quot; Motivated by this observation, we propose RENarGen, a&#10;controllable story-generation paradigm that generates narratives by ensuring&#10;the first and last sentences are related and then infilling the middle&#10;sentences. Our contributions include an initial exploration of how various&#10;methods of bookending from Narratology affect language modeling for stories.&#10;Automatic and human evaluations indicate RENarGen produces better stories with&#10;more narrative closure than current autoregressive models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Narrative closure is an important feature of satisfying narratives. \cite{carroll2007narrative} defines narrative closure as ``the phenomenological feeling of finality that is generated when all the questions saliently posed by the narrative are answered.'' Human writers often achieve closure through bookending \cite{adamo1995beginnings} (a.k.a circular construction or ring composition) whose minimum criteria is for the ending to relate back to the beginning \cite{novakovich2008fiction, katz2023here}.&#10;&#10;Automatic story generation has advanced significantly recently \cite{chaturvedi2016ask, chaturvedi-etal-2017-story, peng-etal-2017-joint, fan2018hierarchical, yao2019plan, fan2019strategies, brahman-chaturvedi-2020-modeling, brahman2020cue, freiknecht2020procedural, castricato2021tell, chowdhury2021everything, vijjini2022towards, yang-etal-2022-re3, huang-etal-2023-affective}. However, these approaches still struggle to generate satisfying and coherent stories with closure \cite{alabdulkarim-etal-2021-automatic, piper2021narrative}. To address this challenge, we propose Related Endpoint Narrative Generator (RENarGen) to generate closed narratives via bookending with related first and last sentences.&#10;&#10;We refer to the first sentence as the start, the last sentence as the stop, and the start/stop sentence pair as endpoints. &#10;Narrative closure can be achieved via related endpoints, which may be operationalized with various methods, the most common of which is semantic relatedness. &#10;Endpoints are semantically related \cite{vmohammad2008measuring,abdalla2021makes} if they resemble each other w.r.t. elements like theme, character, action, place. Figure \ref{fig:two_stories} illustrates this idea with two stories: Story 1 has related endpoints sharing semantic commonalities that complete themes introduced in the start (e.g., protagonist $arrow$ Vivienne, action $arrow$ moving, and place $arrow$ USA); Story 2 has unrelated endpoints with fewer semantic similarities; the stop introduces new themes without satisfactorily fulfilling the initial narrative thought. To a reader, stories like Story 1 are more ``closed'' than stories like Story 2.&#10;&#10;RENarGen (Figure \ref{fig:renargen-framework}) is a scheme that produces stories with closure using neural language models (LMs) and large language models (LLMs) by (1) generating related endpoints given the start and (2) infilling middle sentences given left and right contexts. We approach these two challenges differently for LMs versus for LLMs. For the first challenge for LMs, &#10;we use semantic relatedness to encourage narrative closure. &#10;We generate a phrase list (salient words/phrases from the start) to emphasize narrative aspects that should be addressed in the stop.&#10;For LLMs, we experiment with different single/multi-prompting methods that address bookending with more sophisticated definitions of relatedness for narrative closure. &#10;For the second challenge for LMs, we propose an interative infilling method, &#10;inspired by story-completing techniques described in Narratology \cite{Zemliansky_2020}, that considers both left and right contexts and generates any number of sentences in a reasonable order. While adding sentences left-to-right is a common method of expanding a story, infilling is also a bonafide method: the basic intuition is to find two consecutive sentences between which additional story material is needed. Infilling imitates human writers who add sentences to earlier locations where they determine additional information is necessary \cite{Zemliansky_2020, flower1981cognitive, van2003writing, milligan2017formal, turner2009influence}. Our method is different from previous works using an automatic bidirectional attention strategy \cite{devlin2018bert, ippolito-etal-2019-unsupervised, gu2019insertion, song2019mass, zhu2019text, wang2020narrative, joshi2020spanbert, donahue-etal-2020-enabling} that require the infill to have fixed length, require knowledge of the infill location at the beginning of inference, and/or are not easily iterable. For LLMs, the Story Infiller generates all infills at one time.&#10;&#10;Through piece-wise narrative generation, RENarGen offers user interactivity. For example, for LMs the user can control the generated stop sentence by editing the phrase list. &#10;&#10;RENarGen uses both LMs and LLMs because both have their strengths. LMs are more accessible with predictable output format but are less coherent. LLMs produce higher quality generation but are less accessible and require more computing power. See Appendix \ref{sec:lm_llm} for further discussion. &#10;&#10;Automatic and human evaluations indicate RENarGen outperforms baselines with stories that feel more complete. Our contributions are:&#10;&#10;\begin{itemize}[itemsep=0pt, topsep=0pt, noitemsep]&#10; \item We present the first study of how related endpoints affect narrative generation with an early outlook on how the ``good writing practice'' of bookending impacts language modeling. &#10;&#10;\item We propose RENarGen, a paradigm adaptable to LMs and LLMs and that produces narratives with related endpoint sentences using a novel infilling strategy.&#10;&#10;\item We conduct automatic and human evaluations to show that the stories generated by RENarGen have related endpoints that help with narrative closure and that improve coherence.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Narrative Generation, Narratology, Computer Science, Story Closure, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="2102.12634" label="2102.12634">
        <attvalues>
          <attvalue for="0" value="Automatic Story Generation: Challenges and Attempts" />
          <attvalue for="1" value="  The scope of this survey paper is to explore the challenges in automatic&#10;story generation. We hope to contribute in the following ways: 1. Explore how&#10;previous research in story generation addressed those challenges. 2. Discuss&#10;future research directions and new technologies that may aid more advancements.&#10;3. Shed light on emerging and often overlooked challenges such as creativity&#10;and discourse.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.07610" label="2009.07610">
        <attvalues>
          <attvalue for="0" value="Reusing a Pretrained Language Model on Languages with Limited Corpora&#10;  for Unsupervised NMT" />
          <attvalue for="1" value="  Using a language model (LM) pretrained on two languages with large&#10;monolingual data in order to initialize an unsupervised neural machine&#10;translation (UNMT) system yields state-of-the-art results. When limited data is&#10;available for one language, however, this method leads to poor translations. We&#10;present an effective approach that reuses an LM that is pretrained only on the&#10;high-resource language. The monolingual LM is fine-tuned on both languages and&#10;is then used to initialize a UNMT model. To reuse the pretrained LM, we have to&#10;modify its predefined vocabulary, to account for the new language. We therefore&#10;propose a novel vocabulary extension method. Our approach, RE-LM, outperforms a&#10;competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk)&#10;and English-Albanian (En-Sq), yielding more than +8.3 BLEU points for all four&#10;translation directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Transfer learning for unmt. &#10;The field of unmt has recently experienced tremendous progress.&#10;\cite{artetxe2017unsupervised, lample2017unsupervised} train unmt models with monolingual data only, using denoising auto-encoding \cite{vincent2008extracting} and online back-translation \cite{sennrich2015improving} as training objectives.&#10; This approach is successful for languages with high-quality, large, comparable data. When these conditions are not met, though, unmt provides near-zero scores \cite{neubig-hu-2018-rapid}. &#10;unmt is further improved when initialized with a cross-lingual pretrained model, trained on large corpora \cite{lample2019cross,song2019mass}. However, many languages have only limited monolingual data available, a setting where unmt is not effective \cite{guzman2019flores}. \cite{sun2020self}, whose work is close to our work in motivation, train a unmt model for an hmr-lmr language pair. Iteratively, every subset (e.g. $10$\%) of hmr and all lmr data is backtranslated and the pseudo-parallel corpus is added to the training process. Just like xlm, this training procedure needs to run from scratch for every new language pair. By contrast, our method fine-tunes a monolingual pretrained lm for unmt, so it is computationally faster and simpler. &#10;&#10;Vocabulary. Transferring a pretrained model (source) to a new model (target) requires the use of a shared vocabulary \cite{nguyen-chiang-2017-transfer}. &#10;\cite{kim-etal-2019-effective} propose a linear alignment of the source and target model embeddings using an unsupervised dictionary. &#10;However, when the embeddings of the two models do not have enough overlapping strings, dictionary induction might fail \cite{sogaard-etal-2018}. \cite{dynamicvoc} transfer a source nmt model to a target nmt model (e.g. De-En to Nl-En). To enable transfer, they overwrite the source vocabulary with the target vocabulary. By contrast, we keep the union of the two vocabularies. We fine-tune a pretrained monolingual lm to an lmr language, to initialize an nmt model. Thus, we need the vocabularies of both languages.&#10;&#10;Adapters. &#10;Residual adapters \cite{Houlsby2019ParameterEfficientTL} are feed-forward networks, added to each of to the original model's layers. During fine-tuning, the model parameters are frozen and only the adapters are fine-tuned. This can prevent catastrophic forgetting \cite{Goodfellow2013AnEI,bapna-firat-2019-simple}.&#10;Adapters show promising results in domain adaptation \cite{bapna-firat-2019-simple} and cross-lingual classification \cite{artetxe2019cross}.&#10;Motivated by this, we study the use of adapters during lm fine-tuning in our analysis.&#10;&#10;" />
          <attvalue for="4" value="Multilingual Vocabulary Extension, Computer Science, Linguistics, Artificial Intelligence, Language Model Pretraining, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2210.14448" label="2210.14448">
        <attvalues>
          <attvalue for="0" value="The NPU-ASLP System for The ISCSLP 2022 Magichub Code-Swiching ASR&#10;  Challenge" />
          <attvalue for="1" value="  This paper describes our NPU-ASLP system submitted to the ISCSLP 2022&#10;Magichub Code-Switching ASR Challenge. In this challenge, we first explore&#10;several popular end-to-end ASR architectures and training strategies, including&#10;bi-encoder, language-aware encoder (LAE) and mixture of experts (MoE). To&#10;improve our system's language modeling ability, we further attempt the internal&#10;language model as well as the long context language model. Given the limited&#10;training data in the challenge, we further investigate the effects of data&#10;augmentation, including speed perturbation, pitch shifting, speech codec,&#10;SpecAugment and synthetic data from text-to-speech (TTS). Finally, we explore&#10;ROVER-based score fusion to make full use of complementary hypotheses from&#10;different models. Our submitted system achieves 16.87% on mix error rate (MER)&#10;on the test set and comes to the 2nd place in the challenge ranking.&#10;" />
          <attvalue for="2" value="&#10;&#10;Code-switching occurs when a speaker alternates between two or more languages. With fast globalization and frequent culture exchange, code-switching has become a common language phenomenon which poses significant challenges to speech and language processing tasks including automatic speech recognition (ASR). Code-switching may occur in the middle of a sentence (intra-sentential) or at the sentence boundaries (inter-sentential) while the former is considered to be more difficult to a speech recognizer. To promote reproducible research of Mandarin-English code-switching ASR, ISCSLP2022 has specifically held the Magichub Code-Switching ASR challenge$$, which provides a sizeable corpus and a common test-bed to benchmark the code-switching ASR performance.&#10;&#10;Code-switching ASR has been explored for quite a long time since the conventional hybrid ASR paradigm~\cite{guo2018study}. Progress has also been advanced with several challenges specifically focusing on the code-switching phenomena~\cite{shi2020asru,shah2020first,diwan2021multilingual}.&#10;With the recent advances in deep learning, neural end-to-end (E2E) frameworks, such as attention encoder decoder (AED)~\cite{vaswani2017attention,chorowski2015attention} and neural transducer~\cite{graves2012sequence}, have emerged as the mainstream for ASR with simplified system building pipeline and substantial performance improvement. However, modeling multiple languages simultaneously in a unified neural architecture is non-trivial because different languages (e.g., Mandarin and English) have significant differences in many aspects including modeling units and manner of articulation.&#10;&#10;Recently, language-expert modules~\cite{lu2020bi, tian2022lae,hou2021exploiting} were proposed for modeling different languages by separated parameters in multilingual or cross-lingual settings, which can capture language-specific knowledge space effectively and mitigate overfitting caused by the poverty of code-switching data.&#10;&#10;Specifically, network parameters were decomposed into language-specific parts (or experts) in a bi-encoder structure, where each transformer encoder represents a language (i.e., Mandarin and English)~\cite{lu2020bi, tian2022lae}.&#10;&#10;Meanwhile, the bi-encoder architecture can effectively leverage rich monolingual data from both languages.&#10;But due to the lack of interaction between the separated encoders, the language-common feature space is apparently ignored. &#10;Therefore, language-aware encoder (LAE)~\cite{tian2022lae} was further proposed to address this problem by sharing the preliminary blocks before the language-specific experts, which could model both language-specific and language-common feature efficiently.&#10;Instead of sharing only the preliminary blocks, mixture of experts (MoE)~\cite{hou2021exploiting} was designed to share the majority of parameters, which may be able to learn more language-common feature and be better suited to limited training data conditions.&#10;&#10;Another difficulty is the data sparsity problem. As the language switch can be occurred anywhere in an utterance for the more difficult intra-sentential switch, it is hard to collect enough code-switching data and prediction of the switching position is rather difficult.&#10;&#10;To overcome this problem, data augmentation might be a feasible solution, including text-to-speech (TTS) augmentation and text data augmentation.&#10;Note that using synthetic data directly often has negligible gain or even misguides the ASR system because of the mismatch between the synthetic and real data. &#10;&#10;For better use of synthetic data, some additional loss functions~\cite{chen2022tts4pretrain,chen2021injecting} and filtering strategies~\cite{park2022unsupervised,hu2022synt++} were proposed to enforce the consistency of hypothesized labels between real and synthetic data.&#10;For text augmentation, a machine translation model was usually adopted to expand on the original code-switching text~\cite{liu2021code}.&#10;&#10;In this challenge, we approach the Mandarin-English code-switching ASR by exploration of both multi-lingual neural architectures and data augmentation. Specifically, we study the bi-encoder, LAE and MoE architectures reviewed above under the popular Conformer based AED framework implemented with two popular ASR toolkits -- ESPNet~\cite{watanabe2018espnet} and WeNet~\cite{yao2021wenet}. Various data augmentation methods, including speed perturbation, pitch shifting, audio codec augmentation, spectrum augmentation as well as text-to-speech augmentation. Specifically for TTS augmentation, a consistency loss~\cite{chen2022tts4pretrain} is proved to be effective for mitigating the mismatch in the distribution of real and synthetic data. We further explore the effectiveness of language modeling, including both internal language model as well as long context language model~\cite{wei2021context}. Finally, ROVER~\cite{fiscus1997post} is adopted for fusion of multiple hypothesis from various models, which has previously proven to be effective~\cite{yu2022summary,sun2018multiple,wang2022sjtu}. Our fusion system has achieved the lowest MER of 16.87\% on the test set, leading our submission to the 2nd place in the challenge.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Modeling Techniques, Engineering, Automatic Speech Recognition, Artificial Intelligence, Data Augmentation Strategies" />
        </attvalues>
      </node>
      <node id="2007.05916" label="2007.05916">
        <attvalues>
          <attvalue for="0" value="The ASRU 2019 Mandarin-English Code-Switching Speech Recognition&#10;  Challenge: Open Datasets, Tracks, Methods and Results" />
          <attvalue for="1" value="  Code-switching (CS) is a common phenomenon and recognizing CS speech is&#10;challenging. But CS speech data is scarce and there' s no common testbed in&#10;relevant research. This paper describes the design and main outcomes of the&#10;ASRU 2019 Mandarin-English code-switching speech recognition challenge, which&#10;aims to improve the ASR performance in Mandarin-English code-switching&#10;situation. 500 hours Mandarin speech data and 240 hours Mandarin-English&#10;intra-sentencial CS data are released to the participants. Three tracks were&#10;set for advancing the AM and LM part in traditional DNN-HMM ASR system, as well&#10;as exploring the E2E models' performance. The paper then presents an overview&#10;of the results and system performance in the three tracks. It turns out that&#10;traditional ASR system benefits from pronunciation lexicon, CS text generating&#10;and data augmentation. In E2E track, however, the results highlight the&#10;importance of using language identification, building-up a rational set of&#10;modeling units and spec-augment. The other details in model training and method&#10;comparsion are discussed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.02093" label="2206.02093">
        <attvalues>
          <attvalue for="0" value="LAE: Language-Aware Encoder for Monolingual and Multilingual ASR" />
          <attvalue for="1" value="  Despite the rapid progress in automatic speech recognition (ASR) research,&#10;recognizing multilingual speech using a unified ASR system remains highly&#10;challenging. Previous works on multilingual speech recognition mainly focus on&#10;two directions: recognizing multiple monolingual speech or recognizing&#10;code-switched speech that uses different languages interchangeably within a&#10;single utterance. However, a pragmatic multilingual recognizer is expected to&#10;be compatible with both directions. In this work, a novel language-aware&#10;encoder (LAE) architecture is proposed to handle both situations by&#10;disentangling language-specific information and generating frame-level&#10;language-aware representations during encoding. In the LAE, the primary&#10;encoding is implemented by the shared block while the language-specific blocks&#10;are used to extract specific representations for each language. To learn&#10;language-specific information discriminatively, a language-aware training&#10;method is proposed to optimize the language-specific blocks in LAE. Experiments&#10;conducted on Mandarin-English code-switched speech suggest that the proposed&#10;LAE is capable of discriminating different languages in frame-level and shows&#10;superior performance on both monolingual and multilingual ASR tasks. With&#10;either a real-recorded or simulated code-switched dataset, the proposed LAE&#10;achieves statistically significant improvements on both CTC and neural&#10;transducer systems. Code is released&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.12028" label="2207.12028">
        <attvalues>
          <attvalue for="0" value="Unsupervised data selection for Speech Recognition with contrastive loss&#10;  ratios" />
          <attvalue for="1" value="  This paper proposes an unsupervised data selection method by using a&#10;submodular function based on contrastive loss ratios of target and training&#10;data sets. A model using a contrastive loss function is trained on both sets.&#10;Then the ratio of frame-level losses for each model is used by a submodular&#10;function. By using the submodular function, a training set for automatic speech&#10;recognition matching the target data set is selected. Experiments show that&#10;models trained on the data sets selected by the proposed method outperform the&#10;selection method based on log-likelihoods produced by GMM-HMM models, in terms&#10;of word error rate (WER). When selecting a fixed amount, e.g. 10 hours of data,&#10;the difference between the results of two methods on Tedtalks was 20.23% WER&#10;relative. The method can also be used to select data with the aim of minimising&#10;negative transfer, while maintaining or improving on performance of models&#10;trained on the whole training set. Results show that the WER on the WSJCAM0&#10;data set was reduced by 6.26% relative when selecting 85% from the whole data&#10;set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.11479" label="2110.11479">
        <attvalues>
          <attvalue for="0" value="Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition" />
          <attvalue for="1" value="  With recent advances in speech synthesis, synthetic data is becoming a viable&#10;alternative to real data for training speech recognition models. However,&#10;machine learning with synthetic data is not trivial due to the gap between the&#10;synthetic and the real data distributions. Synthetic datasets may contain&#10;artifacts that do not exist in real data such as structured noise, content&#10;errors, or unrealistic speaking styles. Moreover, the synthesis process may&#10;introduce a bias due to uneven sampling of the data manifold. We propose two&#10;novel techniques during training to mitigate the problems due to the&#10;distribution gap: (i) a rejection sampling algorithm and (ii) using separate&#10;batch normalization statistics for the real and the synthetic samples. We show&#10;that these methods significantly improve the training of speech recognition&#10;models using synthetic data. We evaluate the proposed approach on keyword&#10;detection and Automatic Speech Recognition (ASR) tasks, and observe up to 18%&#10;and 13% relative error reduction, respectively, compared to naively using the&#10;synthetic data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.01547" label="2102.01547">
        <attvalues>
          <attvalue for="0" value="WeNet: Production oriented Streaming and Non-streaming End-to-End Speech&#10;  Recognition Toolkit" />
          <attvalue for="1" value="  In this paper, we propose an open source, production first, and production&#10;ready speech recognition toolkit called WeNet in which a new two-pass approach&#10;is implemented to unify streaming and non-streaming end-to-end (E2E) speech&#10;recognition in a single model. The main motivation of WeNet is to close the gap&#10;between the research and the production of E2E speechrecognition models. WeNet&#10;provides an efficient way to ship ASR applications in several real-world&#10;scenarios, which is the main difference and advantage to other open source E2E&#10;speech recognition toolkits. In our toolkit, a new two-pass method is&#10;implemented. Our method propose a dynamic chunk-based attention strategy of the&#10;the transformer layers to allow arbitrary right context length modifies in&#10;hybrid CTC/attention architecture. The inference latency could be easily&#10;controlled by only changing the chunk size. The CTC hypotheses are then&#10;rescored by the attention decoder to get the final result. Our experiments on&#10;the AISHELL-1 dataset using WeNet show that, our model achieves 5.03\% relative&#10;character error rate (CER) reduction in non-streaming ASR compared to a&#10;standard non-streaming transformer. After model quantification, our model&#10;perform reasonable RTF and latency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.09301" label="2011.09301">
        <attvalues>
          <attvalue for="0" value="Context-aware RNNLM Rescoring for Conversational Speech Recognition" />
          <attvalue for="1" value="  Conversational speech recognition is regarded as a challenging task due to&#10;its free-style speaking and long-term contextual dependencies. Prior work has&#10;explored the modeling of long-range context through RNNLM rescoring with&#10;improved performance. To further take advantage of the persisted nature during&#10;a conversation, such as topics or speaker turn, we extend the rescoring&#10;procedure to a new context-aware manner. For RNNLM training, we capture the&#10;contextual dependencies by concatenating adjacent sentences with various tag&#10;words, such as speaker or intention information. For lattice rescoring, the&#10;lattice of adjacent sentences are also connected with the first-pass decoded&#10;result by tag words. Besides, we also adopt a selective concatenation strategy&#10;based on tf-idf, making the best use of contextual similarity to improve&#10;transcription performance. Results on four different conversation test sets&#10;show that our approach yields up to 13.1% and 6% relative char-error-rate (CER)&#10;reduction compared with 1st-pass decoding and common lattice-rescoring,&#10;respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.02329" label="2406.02329">
        <attvalues>
          <attvalue for="0" value="On Affine Homotopy between Language Encoders" />
          <attvalue for="1" value="  Pre-trained language encoders -- functions that represent text as vectors --&#10;are an integral component of many NLP tasks. We tackle a natural question in&#10;language encoder analysis: What does it mean for two encoders to be similar? We&#10;contend that a faithful measure of similarity needs to be \emph{intrinsic},&#10;that is, task-independent, yet still be informative of \emph{extrinsic}&#10;similarity -- the performance on downstream tasks. It is common to consider two&#10;encoders similar if they are \emph{homotopic}, i.e., if they can be aligned&#10;through some transformation. In this spirit, we study the properties of&#10;\emph{affine} alignment of language encoders and its implications on extrinsic&#10;similarity. We find that while affine alignment is fundamentally an asymmetric&#10;notion of similarity, it is still informative of extrinsic similarity. We&#10;confirm this on datasets of natural language representations. Beyond providing&#10;useful bounds on extrinsic similarity, affine intrinsic similarity also allows&#10;us to begin uncovering the structure of the space of pre-trained encoders by&#10;defining an order over them.&#10;" />
          <attvalue for="2" value=" &#10;A common paradigm in modern natural language processing (NLP) is to pre-train a \defn{language encoder} on a large swathe of natural language text. &#10;Then, a task-specific model is fit (fine-tuned) using the language encoder as the representation function of the text.&#10;More formally, a language encoder is a function $\enc\colon \kleene{\alphabet} \to \Rd$, i.e., a function that maps a string over an alphabet $\alphabet$ to a finite-dimensional vector. &#10;Now, consider sentiment analysis as an informative example of a task.&#10;Suppose our goal is to classify a string $\str \in \kleene{\alphabet}$ as one of three polarities $\Pi = \{\pos, \neut, \negative\}$.&#10;Then, the probability of $\str$ exhibiting a specific polarity is often given by a log-linear model, e.g., the probability of $\pos$ is&#10;\begin{equation} &#10; \pLM(\pos \mid \str) = \softmax(\outMtx \, \enc(\str) + \vb)_{\pos}&#10;\end{equation}&#10;where $\outMtx \in \R^{3 \times \featurespacedim}$, $\vb \in \R^3$ and $\softmax \colon \R^N arrow \Delta^{N - 1}$. &#10;Empirically, using a pre-trained encoder $\enc$ leads to significantly better classifier performance than training a log-linear model from scratch.&#10;&#10;In the context of the widespread deployment of language encoders, this paper tackles a natural question: Given two language encoders $\enc$ and $\encg$, how can we judge to what extent they are similar?&#10;This question is of practical importance---recent studies have shown that even small variations in the random seed used for training can result in significant performance differences on downstream tasks between models with the same architecture \cite{dodge2020finetuning,sellam2021multiberts}&#10;In this case, we say that two such language encoders exhibit an extrinsic difference, i.e., the difference between two encoders manifests itself when considering their performance on a downstream task.&#10;However, we also seek an intrinsic notion of similarity between two language encoders, i.e., a notion of similarity that is independent of any particular downstream task.&#10;Moreover, we may hope that a good notion of intrinsic similarity would allow us to construct a notion of extrinsic similarity that holds for all downstream tasks.&#10;&#10;Existing work studies language encoder similarity by evaluating whether two encoders produce similar representations for a finite dataset of strings \cite[][inter alia]{Hardoon2004CanonicalCA, pmlr-v97-kornblith19a,williams2021generalizedsm,boix2024gulp}, often by analyzing whether the representation sets can be approximately linearly aligned \cite{pmlr-v97-kornblith19a, pmlr-v44-li15convergent}. &#10;More formally, two encoders are considered similar if there exists a matrix $\mA$ such that $\enc(\str) \approx \mA \, \encg(\str)$ holds for strings $\str$ in some finite set $\sD \subset \kleene{\alphabet}$.&#10;This assumes that examining finitely many outputs provides sufficient insight into encoder behavior.&#10;In contrast, we set out to study the relationships between language encoders, i.e., functions, themselves.&#10;This decision, rather than being just a technicality, allows us to derive a richer understanding of encoder relationships, revealing properties and insights that remain obscured under conventional finite-set analysis. &#10;Concretely, we ask what notions of similarity between encoders one could consider and what they imply for their relationships.&#10;&#10;The main contributions of the paper are of a theoretical nature.&#10;We first define an (extended) metric space on language encoders.&#10;We then extend this notion to account for transformations in a broad framework of \defn{$S$-homotopy} for a set of transformations $S$, where $\encg$ is $S$-homotopic to $\enc$ if $\encg$ can be transformed into $\enc$ through some transformation in $S$.&#10;As a concrete application of the framework, we study affine homotopy---the similarity of $\enc$ and $\psi \circ \encg$ for affine transformations $\psi$.&#10;The notion of intrinsic similarity induced by such one-sided alignment is not symmetric and can be seen as the cost of transforming $\encg$ into $\enc$.&#10;Nevertheless, we show it is informative of extrinsic similarity: If one encoder can be affinely mapped to another, we can guarantee that it also performs similarly on downstream tasks.&#10;We confirm this empirically by studying the intrinsic and extrinsic similarities of various pre-trained encoders, where we observe a positive correlation between intrinsic and extrinsic similarity. Beyond measuring similarity, homotopy also allows us to define a form of hierarchy on the space of encoders, elucidating a structure in which some encoders are more informative than others.&#10;Such an order is also suggested by our experiments, where we find that certain encoders are easier to map to than others which shows in the rank of the learned representations and affects their transfer learning ability.\looseness=-1&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Encoder Analysis, Linguistics, Similarity Measurement, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2110.14739" label="2110.14739">
        <attvalues>
          <attvalue for="0" value="Generalized Shape Metrics on Neural Representations" />
          <attvalue for="1" value="  Understanding the operation of biological and artificial networks remains a&#10;difficult and important challenge. To identify general principles, researchers&#10;are increasingly interested in surveying large collections of networks that are&#10;trained on, or biologically adapted to, similar tasks. A standardized set of&#10;analysis tools is now needed to identify how network-level covariates -- such&#10;as architecture, anatomical brain region, and model organism -- impact neural&#10;representations (hidden layer activations). Here, we provide a rigorous&#10;foundation for these analyses by defining a broad family of metric spaces that&#10;quantify representational dissimilarity. Using this framework we modify&#10;existing representational similarity measures based on canonical correlation&#10;analysis to satisfy the triangle inequality, formulate a novel metric that&#10;respects the inductive biases in convolutional layers, and identify approximate&#10;Euclidean embeddings that enable network representations to be incorporated&#10;into essentially any off-the-shelf machine learning method. We demonstrate&#10;these methods on large-scale datasets from biology (Allen Institute Brain&#10;Observatory) and deep learning (NAS-Bench-101). In doing so, we identify&#10;relationships between neural representations that are interpretable in terms of&#10;anatomical features and model performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.06545" label="2210.06545">
        <attvalues>
          <attvalue for="0" value="GULP: a prediction-based metric between representations" />
          <attvalue for="1" value="  Comparing the representations learned by different neural networks has&#10;recently emerged as a key tool to understand various architectures and&#10;ultimately optimize them. In this work, we introduce GULP, a family of distance&#10;measures between representations that is explicitly motivated by downstream&#10;predictive tasks. By construction, GULP provides uniform control over the&#10;difference in prediction performance between two representations, with respect&#10;to regularized linear prediction tasks. Moreover, it satisfies several&#10;desirable structural properties, such as the triangle inequality and invariance&#10;under orthogonal transformations, and thus lends itself to data embedding and&#10;visualization. We extensively evaluate GULP relative to other methods, and&#10;demonstrate that it correctly differentiates between architecture families,&#10;converges over the course of training, and captures generalization performance&#10;on downstream linear tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.09524" label="2309.09524">
        <attvalues>
          <attvalue for="0" value="Improved Factorized Neural Transducer Model For text-only Domain&#10;  Adaptation" />
          <attvalue for="1" value="  Adapting End-to-End ASR models to out-of-domain datasets with text data is&#10;challenging. Factorized neural Transducer (FNT) aims to address this issue by&#10;introducing a separate vocabulary decoder to predict the vocabulary.&#10;Nonetheless, this approach has limitations in fusing acoustic and language&#10;information seamlessly. Moreover, a degradation in word error rate (WER) on the&#10;general test sets was also observed, leading to doubts about its overall&#10;performance. In response to this challenge, we present the improved factorized&#10;neural Transducer (IFNT) model structure designed to comprehensively integrate&#10;acoustic and language information while enabling effective text adaptation. We&#10;assess the performance of our proposed method on English and Mandarin datasets.&#10;The results indicate that IFNT not only surpasses the neural Transducer and FNT&#10;in baseline performance in both scenarios but also exhibits superior adaptation&#10;ability compared to FNT. On source domains, IFNT demonstrated statistically&#10;significant accuracy improvements, achieving a relative enhancement of 1.2% to&#10;2.8% in baseline accuracy compared to the neural Transducer. On out-of-domain&#10;datasets, IFNT shows relative WER(CER) improvements of up to 30.2% over the&#10;standard neural Transducer with shallow fusion, and relative WER(CER)&#10;reductions ranging from 1.1% to 2.8% on test sets compared to the FNT model.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, end-to-end (E2E) \cite{li2022recent} based models have gained great interest in automatic speech recognition (ASR) systems. Compared to traditional hybrid systems, E2E systems such as connectionist temporal classification (CTC) \cite{ctc}, attention-based encoder-decoder (AED) \cite{las}, and neural Transducer (NT) \cite{sequence} predict word sequences using a single neural network. When there is a mismatch between the trained domain and the test domain, a significant degradation in accuracy is observed. Conventional domain adaptation methods \cite{Bell_2021, deng2023adaptable} typically rely on speech-text pairs from the target domain. However, collecting a large amount of speech-text matching data from the target domain is difficult, while obtaining text-only data is relatively easier. As a result, text-only adaptive methods have been widely proposed and studied \cite{pylkkonen2021fast, Choudhury2022}. Since E2E systems are jointly optimized, there is no separate component that solely performs as a language model (LM), making it challenging to directly apply common LM adaptation methods.&#10;&#10;One feasible solution is to fine-tune the E2E model using the synthesized audio-transcript pairs generated by a text-to-speech (TTS) model \cite{sim2019personalization, zheng2021using, deng2021improving}, but this approach is computationally expensive. Another common practice is LM fusion \cite{cabrera2021language, 8639038, levit2023external, li2023prompting}, such as shallow fusion \cite{kannan2017analysis}, deep fusion \cite{gulcehre2015using}, and cold fusion \cite{coldfusion}. Among them, the most widely used is shallow fusion, which combines the E2E model score and the external LM score in the log-linear domain during beam search. Methods like density ratio \cite{mcdermott2020density} also work similarly. Internal language model estimation \cite{ilm, meng2021internal} was also proposed recently, it calculates the interpolated log-likelihood score based on the maximum scores from the internal LM and the external LM respectively during decoding. However, LM fusion involves interpolation weights that are task-dependent and require tuning, making the performance sensitive to the weight selection. &#10;&#10;There have been increasing research efforts \cite{variani2020hybrid, factorizedAED, mhat} to modify the structure of neural Transducers. Factorized neural Transducer (FNT) \cite{zhao2023fast, chen2021Factorized,levit2023external, gong2023longfnt,le2023factorized} addresses this issue by introducing a standalone LM for vocabulary prediction, enabling the direct application of conventional LM adaptation methods. While the results have shown promising adaptation abilities, minor accuracy degradation has also been observed on the general test set compared to the standard neural Transducer baseline, raising doubts about their overall performance. &#10;&#10;Building upon this work, we propose the improved factorized neural Transducer (IFNT) model that can better combine acoustic information with language information, and improve its baseline accuracy both before and after adapting to text data. Our proposed model incorporates an internal LM alongside the standard neural Transducer model, with the LM posterior probability directly integrated into the vocabulary prediction. Utilizing a standalone LM in our model facilitates the application of various LM adaptation methods for fast text-only domain adaptation, similar to the hybrid system. We validate the proposed method in both English and Mandarin datasets, both in-domain &#10;and out-of-domain scenarios &#10;results demonstrate the superior performance of our proposed model over the FNT and standard neural Transducer shallow fusion methods.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Network Models, Engineering, Signal Processing, Speech Recognition, Language Adaptation" />
        </attvalues>
      </node>
      <node id="2211.00896" label="2211.00896">
        <attvalues>
          <attvalue for="0" value="Factorized Blank Thresholding for Improved Runtime Efficiency of Neural&#10;  Transducers" />
          <attvalue for="1" value="  We show how factoring the RNN-T's output distribution can significantly&#10;reduce the computation cost and power consumption for on-device ASR inference&#10;with no loss in accuracy. With the rise in popularity of neural-transducer type&#10;models like the RNN-T for on-device ASR, optimizing RNN-T's runtime efficiency&#10;is of great interest. While previous work has primarily focused on the&#10;optimization of RNN-T's acoustic encoder and predictor, this paper focuses the&#10;attention on the joiner. We show that despite being only a small part of RNN-T,&#10;the joiner has a large impact on the overall model's runtime efficiency. We&#10;propose to utilize HAT-style joiner factorization for the purpose of skipping&#10;the more expensive non-blank computation when the blank probability exceeds a&#10;certain threshold. Since the blank probability can be computed very efficiently&#10;and the RNN-T output is dominated by blanks, our proposed method leads to a&#10;26-30% decoding speed-up and 43-53% reduction in on-device power consumption,&#10;all the while incurring no accuracy degradation and being relatively simple to&#10;implement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.16248" label="2310.16248">
        <attvalues>
          <attvalue for="0" value="GlotLID: Language Identification for Low-Resource Languages" />
          <attvalue for="1" value="  Several recent papers have published good solutions for language&#10;identification (LID) for about 300 high-resource and medium-resource languages.&#10;However, there is no LID available that (i) covers a wide range of low-resource&#10;languages, (ii) is rigorously evaluated and reliable and (iii) efficient and&#10;easy to use. Here, we publish GlotLID-M, an LID model that satisfies the&#10;desiderata of wide coverage, reliability and efficiency. It identifies 1665&#10;languages, a large increase in coverage compared to prior work. In our&#10;experiments, GlotLID-M outperforms four baselines (CLD3, FT176, OpenLID and&#10;NLLB) when balancing F1 and false positive rate (FPR). We analyze the unique&#10;challenges that low-resource LID poses: incorrect corpus metadata, leakage from&#10;high-resource languages, difficulty separating closely related languages,&#10;handling of macrolanguage vs varieties and in general noisy data. We hope that&#10;integrating GlotLID-M into dataset creation pipelines will improve quality and&#10;enhance accessibility of NLP technology for low-resource languages and&#10;cultures. GlotLID-M model (including future versions), code, and list of data&#10;sources are available: https://github.com/cisnlp/GlotLID.&#10;" />
          <attvalue for="2" value="&#10;The NLP community should &#10;create technology that covers as many languages as possible,&#10;not only medium-resource and high-resource languages. This&#10;goal can only be achieved if corpora for low-resource&#10;languages are available. Web-mined datasets -- including&#10;CC100~\cite{wenzek-etal-2020-ccnet}, mC4~\cite{xue-etal-2021-mt5}&#10;and OSCAR~\cite{AbadjiOrtizSuarezRomaryetal.2021,&#10;OrtizSuarezSagotRomary2019} -- have made important&#10;contributions to low-resource NLP. In&#10;particular, they lay the ground for multilingual neural&#10;models like XLM-R~\cite{conneau-etal-2020-unsupervised},&#10;mT5~\cite{xue-etal-2021-mt5} and&#10;Glot500~\cite{imanigooghari-etal-2023-glot500}. However, existing&#10;web-mined datasets have systematic quality issues&#10;\cite{kreutzer-etal-2022-quality} and insufficient coverage of&#10;low-resource languages.&#10;&#10;Low-quality datasets cause poor performance for downstream applications. They can&#10;also give rise to a misleading perception of progress when &#10;coverage of a low-resource language is claimed based on&#10;noisy data. NLP for low-resource languages&#10;requires high-quality datasets and high-quality datasets&#10;require high-quality LID (language identification). For this reason, high-quality LID&#10;for low-resource languages is paramount. To address this&#10;need, in this paper we present \modelname, a high-quality&#10;LID that covers&#10;\numberlanguagesaccurate languages.&#10;We use ISO&#10;639-3 to individuate languages.&#10;&#10;When expanding the scope of LID from&#10;a few hundred&#10;to \numberlanguagesaccurate&#10;languages, the problem of granularity becomes&#10;severe. In real-world settings, LID needs to&#10;support both \macrolanguages and their varieties; it also needs to&#10;be&#10;robust against out-of-model cousins \cite{caswell-etal-2020-language, kreutzer-etal-2022-quality}.&#10;We pay particular attention to this issue.&#10;&#10;While low-resource is our main focus, \cite{blevins-zettlemoyer-2022-language} point out&#10;that low-quality LID also affects high-resource corpora&#10;through contamination, resulting in claims of successful&#10;crosslingual transfer that are due to unrecognized&#10;coverage of low-resource languages. We also address this&#10;issue, e.g., we improve English F1 on the ``Universal Declaration of Human Rights'' corpus (\udhr) to.85&#10;compared to.43 for \edin.&#10;&#10;Contributions.&#10;(i) We curate \corpusname, a comprehensive dataset&#10;covering \numberlanguagesaccurate languages, most of&#10;them low-resource, from a diverse set of domains. &#10;(ii) We train \modelname&#10;on \corpusname, &#10;an open-source LID covering&#10;these \numberlanguagesaccurate languages.&#10;(iii) In our experiments, \modelname outperforms several baselines&#10;by more than 12\% absolute F1&#10;on \udhr, which we take as the best benchmark for our focus&#10;on low-resource languages.&#10;(iv) When balancing F1 and false positive rate (FPR), \modelname&#10;also outperforms baselines on \flores-200, which is dominated&#10;by high-/medium-resource languages.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Low-Resource Languages, Language Identification, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.14571" label="2010.14571">
        <attvalues>
          <attvalue for="0" value="Language ID in the Wild: Unexpected Challenges on the Path to a&#10;  Thousand-Language Web Text Corpus" />
          <attvalue for="1" value="  Large text corpora are increasingly important for a wide variety of Natural&#10;Language Processing (NLP) tasks, and automatic language identification (LangID)&#10;is a core technology needed to collect such datasets in a multilingual context.&#10;LangID is largely treated as solved in the literature, with models reported&#10;that achieve over 90% average F1 on as many as 1,366 languages. We train LangID&#10;models on up to 1,629 languages with comparable quality on held-out test sets,&#10;but find that human-judged LangID accuracy for web-crawl text corpora created&#10;using these models is only around 5% for many lower-resource languages,&#10;suggesting a need for more robust evaluation. Further analysis revealed a&#10;variety of error modes, arising from domain mismatch, class imbalance, language&#10;similarity, and insufficiently expressive models. We propose two classes of&#10;techniques to mitigate these errors: wordlist-based tunable-precision filters&#10;(for which we release curated lists in about 500 languages) and&#10;transformer-based semi-supervised LangID models, which increase median dataset&#10;precision from 5.5% to 71.2%. These techniques enable us to create an initial&#10;data set covering 100K or more relatively clean sentences in each of 500+&#10;languages, paving the way towards a 1,000-language web text corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.17649" label="2303.17649">
        <attvalues>
          <attvalue for="0" value="Aligning a medium-size GPT model in English to a small closed domain in&#10;  Spanish" />
          <attvalue for="1" value="  In this paper, we propose a methodology to align a medium-sized GPT model,&#10;originally trained in English for an open domain, to a small closed domain in&#10;Spanish. The application for which the model is finely tuned is the question&#10;answering task. To achieve this we also needed to train and implement another&#10;neural network (which we called the reward model) that could score and&#10;determine whether an answer is appropriate for a given question. This component&#10;served to improve the decoding and generation of the answers of the system.&#10;Numerical metrics such as BLEU and perplexity were used to evaluate the model,&#10;and human judgment was also used to compare the decoding technique with others.&#10;Finally, the results favored the proposed method, and it was determined that it&#10;is feasible to use a reward model to align the generation of responses.&#10;" />
          <attvalue for="2" value="&#10;Transformer neural networks have shown great potential for natural language processing \cite{vaswani2017attention}. Several pre-trained transformer language models have been developed with the use of transfer learning, for instance the work by Radford et al. \cite{radford2018improving}. These models have been scaled by increasing their number of parameters by hundreds of millions like GPT-2 \cite{radford2019language} and BERT \cite{devlin2019bert}, or up to hundreds of billions like GPT-3 \cite{brown2020language} and GPT-4 \cite{openai2023gpt4}. Transformer models are trained with massive amounts of text and once they have been finely aligned, they can even recognize the task they must perform, despite never having been specifically trained for it. It is also possible to use the weights of these models and fine-tune them for a particular task, for example, open domain conversational systems like DialoGPT \cite{zhang2020dialogpt}, InstructGPT and ChatGPT \cite{ouyang2022training}.&#10;&#10;On one hand, training huge models like GPT-4 is complicated and highly computationally expensive. On the other hand, medium-size models still lack coherence and sometimes tend to hallucinate facts. Furthermore, since they are usually trained with a wide variety of texts on the Internet, their behavior is not aligned to correctly follow the user's instructions. This highlights the need for some additional component capable of providing consistency to the dialogue state, as well as some kind of system that evaluates and determines the quality of the responses \cite{mctear2020conversational}. However, it is not always possible to have a large enough labeled dataset to fine-tune these models. Especially if the data comes from a knowledge base about a particular product, service or subject. Therefore, there is a need to search for feasible strategies for training language models in the response-generation task, that can be customized for a specific information domain.&#10;&#10;In this article, we focus on the problem of aligning the DialoGPT model, to enable it to answer questions in Spanish. DialoGPT is a model based on GPT-2 and it was originally trained in English. In order to achieve a good alignment, the model is firstly fine-tuned with a corpus in Spanish. Then, it is further refined with a small collection of questions and answers about a specific topic. Finally, the performance of the model is improved through a reward neural network trained with human feedback. As a case study, we implemented a chatbot that answers frequently asked questions from a University undergraduate program, which we will refer to as the LCC dataset. The results show that it is feasible to use a reward model to align a medium-size GPT model to a small set of question-answer pairs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Question Answering Systems, Mathematics, Neural Network Optimization, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="2212.07939" label="2212.07939">
        <attvalues>
          <attvalue for="0" value="RWEN-TTS: Relation-aware Word Encoding Network for Natural&#10;  Text-to-Speech Synthesis" />
          <attvalue for="1" value="  With the advent of deep learning, a huge number of text-to-speech (TTS)&#10;models which produce human-like speech have emerged. Recently, by introducing&#10;syntactic and semantic information w.r.t the input text, various approaches&#10;have been proposed to enrich the naturalness and expressiveness of TTS models.&#10;Although these strategies showed impressive results, they still have some&#10;limitations in utilizing language information. First, most approaches only use&#10;graph networks to utilize syntactic and semantic information without&#10;considering linguistic features. Second, most previous works do not explicitly&#10;consider adjacent words when encoding syntactic and semantic information, even&#10;though it is obvious that adjacent words are usually meaningful when encoding&#10;the current word. To address these issues, we propose Relation-aware Word&#10;Encoding Network (RWEN), which effectively allows syntactic and semantic&#10;information based on two modules (i.e., Semantic-level Relation Encoding and&#10;Adjacent Word Relation Encoding). Experimental results show substantial&#10;improvements compared to previous works.&#10;" />
          <attvalue for="2" value="&#10;Text-to-Speech (TTS), which aims at synthesizing natural-sounding speech from text, has extensive applications in various industries such as entertainment, education, and so on~\cite{DBLP:journals/corr/abs-2106-15561}. Recently, deep learning-based TTS models have drawn attention, showing unprecedented results.&#10;Most existing works have adopted a two-stage generation scheme, which produces an intermediate speech representation (e.g., Mel-spectrogram) from the input text and then generates a raw waveform.&#10;In this work, we focus on the model used in the first stage, called an acoustic model. Generally, the acoustic model is categorized into the autoregressive (AR) model and the non-autoregressive (NAR) model, according to the generation method. Early studies usually focused on the AR model~\cite{DBLP:conf/ssw/OordDZSVGKSK16, DBLP:conf/icml/Skerry-RyanBXWS18, DBLP:conf/icassp/ShenPWSJYCZWRSA18, 10.1609/aaai.v33i01.33016706}. However, they have a slow inference speed caused by sequential generation. Moreover, they are quite sensitive to the alignment resulting in low robustness (e.g., long pause, word repeating, and word skipping).&#10;To overcome these limitations, many NAR models~\cite{lancucki2021fastpitch, ren2019fastspeech, DBLP:conf/iclr/0006H0QZZL21} have been proposed. Compared to AR models, they showed faster inference speed by generating speech in parallel and alleviated robustness issues. Nevertheless, their quality of expressiveness is unsatisfactory because they predict prosodic features that contain pitch, duration, and energy without introducing dependency between time steps~\cite{kharitonov-etal-2022-text}. Thus, various approaches to improve the quality of NAR-TTS have been proposed. &#10;&#10;\cite{min2021meta} successfully achieved expressive speech synthesis by introducing a reference encoder that models desired prosody because the same sentence can be uttered in diverse styles. &#10;&#10;\cite{hwang2021tts, DBLP:conf/interspeech/SongYKSHOYKK22, lajszczak2022distribution} claimed that the performance of NAR-TTS is poor when the training data is insufficient, devising effective data augmentation methods.&#10;\cite{kim2021conditional} combined powerful generative models (i.e., variational autoencoder, normalizing flow, and generative adversarial network) to improve expressiveness. They reported that proposed model close to human-level speech.&#10;Meanwhile, \cite{DBLP:conf/interspeech/KenterSC20, 9413513, DBLP:conf/interspeech/JiaZSZW21, DBLP:conf/interspeech/ZhouSL0B0M22, DBLP:conf/interspeech/ZhangS0TYLWZQLZ22, tatanov2022mixer} boosted the expressiveness of speech by applying various methods proposed in the field of natural language processing (NLP) to the speech domain.&#10;Especially, GraphSpeech~\cite{9413513} and Relational Gated Graph Network (RGGN)~\cite{DBLP:conf/interspeech/ZhouSL0B0M22} claimed the syntactic and semantic information of text affects the naturalness and expressiveness of speech. They improved the performance by utilizing graph networks focused on the representation based on dependency relations.&#10;&#10;Despite the impressive results, we point out two crucial problems in applying syntactic and semantic information. First, most previous works utilizing dependency relations tend to assign graph networks to encode the neighbor nodes based on the dependency tree. For example, in Figure~\ref{fig:01_dependency_tree_example}, when encoding ``shark&quot;, RGGN utilizes weighted-sum to encode ``the&quot;, ``blue&quot;, and ``teeth&quot;, simultaneously. In RGGN, these neighbor words are explicitly considered, and others are implicitly considered. However, ``blue&quot; and ``teeth&quot; do not have a direct semantic correlation, except they share the same parent. We assume that encoding dimly correlated words simultaneously and explicitly can confuse the model. Second, previous works do not explicitly consider dependency relations on adjacent words. On the other hand, it is obvious the relations of adjacent words are usually meaningful because the TTS task deals with sequential data.&#10;&#10;To address the aforementioned issues, we propose Relation-aware Word Encoding Network (RWEN) for TTS. RWEN, which consists of Semantic-level Relation Encoding (SRE) and Adjacent Word Relation Encoding (AWRE), focuses on effectively encoding dependency relations to improve naturalness and expressiveness. SRE encodes dependency relations based on the semantic level to substitute the inefficient graph networks mentioned above. AWRE explicitly encodes dependency relations based on adjacent words. We briefly summarize our main contributions as follows:&#10;\begin{itemize}&#10;\item We design two novel approaches, SRE and AWRE, to consider linguistic features and TTS characteristics.&#10;\item We propose RWEN that contains SRE and AWRE, which can be easily incorporated into most recent TTS models.&#10;\item Experimental results demonstrate that RWEN outperforms existing works, and we prove that SRE and AWRE are significantly effective through our ablation experiments.&#10;\end{itemize}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Linguistics, Deep Learning Techniques, Artificial Intelligence, Natural Language Processing, Text-to-Speech Models" />
        </attvalues>
      </node>
      <node id="2010.13421" label="2010.13421">
        <attvalues>
          <attvalue for="0" value="TTS-by-TTS: TTS-driven Data Augmentation for Fast and High-Quality&#10;  Speech Synthesis" />
          <attvalue for="1" value="  In this paper, we propose a text-to-speech (TTS)-driven data augmentation&#10;method for improving the quality of a non-autoregressive (AR) TTS system.&#10;Recently proposed non-AR models, such as FastSpeech 2, have successfully&#10;achieved fast speech synthesis system. However, their quality is not&#10;satisfactory, especially when the amount of training data is insufficient. To&#10;address this problem, we propose an effective data augmentation method using a&#10;well-designed AR TTS system. In this method, large-scale synthetic corpora&#10;including text-waveform pairs with phoneme duration are generated by the AR TTS&#10;system and then used to train the target non-AR model. Perceptual listening&#10;test results showed that the proposed method significantly improved the quality&#10;of the non-AR TTS system. In particular, we augmented five hours of a training&#10;database to 179 hours of a synthetic one. Using these databases, our TTS system&#10;consisting of a FastSpeech 2 acoustic model with a Parallel WaveGAN vocoder&#10;achieved a mean opinion score of 3.74, which is 40% higher than that achieved&#10;by the conventional method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.06835" label="2104.06835">
        <attvalues>
          <attvalue for="0" value="Enhancing Word-Level Semantic Representation via Dependency Structure&#10;  for Expressive Text-to-Speech Synthesis" />
          <attvalue for="1" value="  Exploiting rich linguistic information in raw text is crucial for expressive&#10;text-to-speech (TTS). As large scale pre-trained text representation develops,&#10;bidirectional encoder representations from Transformers (BERT) has been proven&#10;to embody semantic information and employed to TTS recently. However, original&#10;or simply fine-tuned BERT embeddings still cannot provide sufficient semantic&#10;knowledge that expressive TTS models should take into account. In this paper,&#10;we propose a word-level semantic representation enhancing method based on&#10;dependency structure and pre-trained BERT embedding. The BERT embedding of each&#10;word is reprocessed considering its specific dependencies and related words in&#10;the sentence, to generate more effective semantic representation for TTS. To&#10;better utilize the dependency structure, relational gated graph network (RGGN)&#10;is introduced to make semantic information flow and aggregate through the&#10;dependency structure. The experimental results show that the proposed method&#10;can further improve the naturalness and expressiveness of synthesized speeches&#10;on both Mandarin and English datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09394" label="2005.09394">
        <attvalues>
          <attvalue for="0" value="Enhancing Monotonic Multihead Attention for Streaming ASR" />
          <attvalue for="1" value="  We investigate a monotonic multihead attention (MMA) by extending hard&#10;monotonic attention to Transformer-based automatic speech recognition (ASR) for&#10;online streaming applications. For streaming inference, all monotonic attention&#10;(MA) heads should learn proper alignments because the next token is not&#10;generated until all heads detect the corresponding token boundaries. However,&#10;we found not all MA heads learn alignments with a na\&quot;ive implementation. To&#10;encourage every head to learn alignments properly, we propose HeadDrop&#10;regularization by masking out a part of heads stochastically during training.&#10;Furthermore, we propose to prune redundant heads to improve consensus among&#10;heads for boundary detection and prevent delayed token generation caused by&#10;such heads. Chunkwise attention on each MA head is extended to the multihead&#10;counterpart. Finally, we propose head-synchronous beam search decoding to&#10;guarantee stable streaming inference.&#10;" />
          <attvalue for="2" value="&#10;Recent progress of end-to-end (E2E) automatic speech recognition (ASR) models bridges the gap from the state-of-the-art hybrid systems \cite{google_sota_asr}.&#10;To make E2E models applicable to simultaneous interpretations in lecture and meeting domains, online streaming processing is necessary.&#10;For E2E models, connectionist temporal classification (CTC) \cite{ctc_graves} and recurrent neural network transducer (RNN-T) \cite{rnn_transducer} have been dominant approaches and reached a level of real applications \cite{he2019streaming,sainath2020streaming}.&#10;Meanwhile, attention-based encoder-decoder (AED) models \cite{chorowski2015attention,las} have demonstrated the powerful modeling capability in offline tasks \cite{s2s_comparison_google,s2s_comparison_baidu,rwth_end2end} and a number of streaming models have been investigated for RNN-based models \cite{hou2017gaussian,tjandra2017local,lawson2018learning,adaptive_computation_steps,moritz2019triggered_icassp2019,hard_monotonic_attention,mocha}.&#10;&#10;Recently, the Transformer architecture \cite{vaswani2017attention}, based on self-attention and multihead attention, has shown to outperform the RNN counterparts in various domains \cite{karita2019comparative,zeyer2019comparison}, and several streaming models have been proposed such as triggered attention \cite{moritz2020streaming_icassp2020}, continuous-integrate-and-fire (CIF) \cite{cif}, hard monotonic attention (HMA) \cite{tsunoo2019towards,miao2020transformer}, and other variants \cite{tian2019synchronous}.&#10;Triggered attention truncates encoder outputs by using CTC spikes and performs an attention mechanism over all past frames.&#10;CIF learns acoustic boundaries explicitly and extracts context vectors from the segmented region.&#10;Therefore, these models have adaptive segmentation policies relying on acoustic cues only.&#10;&#10;On the other hand, HMA detects token boundaries on the decoder side by using lexical information as well.&#10;Thus, it is more flexible for modeling non-monotonic alignments and has been investigated in simultaneous machine translation (MT) \cite{arivazhagan2019monotonic}.&#10;Recently, HMA was extended to the Transformer architecture, named monotonic multihead attention (MMA), by replacing each encoder-decoder attention head in the decoder with a monotonic attention (MA) head \cite{ma2019monotonic}.&#10;Unlike a single MA head used in RNN-based models, each MA head can extract source contexts with different pace and learn complex alignments between input and output sequences.&#10;Concurrently, similar methods have been investigated for Transformer-based streaming ASR \cite{tsunoo2019towards,miao2020transformer}.&#10;Miao et al. \cite{miao2020transformer} simplified the MMA framework by equipping a single MA head with each decoder layer to truncate encoder outputs as in triggered attention and perform attention over all past frames.&#10;Tsunoo et al. \cite{tsunoo2019towards} also investigated the MMA framework but resorted to using all past frames to obtain a decent performance.&#10;However, looking back to the beginning of input frames lessens the advantage of linear-time decoding with HMA as the input length gets longer.&#10;&#10;In this work, we investigate the MMA framework using restricted input context for the streaming ASR task.&#10;To perform streaming recognition with the MMA framework, it is necessary for every MA head to learn alignments properly.&#10;This is because the next token is not generated until all heads detect the corresponding token boundaries.&#10;If some heads fail to detect the boundaries until seeing the encoder output of the final frame, the next token generation is delayed accordingly.&#10;However, with a naïve implementation, we found that proper monotonic alignments are learnt in dominant MA heads only.&#10;To prevent this, we propose HeadDrop, in which a part of heads is entirely masked out at random as a regularization during training to encourage the rest non-masked heads to learn alignments properly.&#10;Moreover, we propose to prune redundant MA heads in lower decoder layers to further improve consensus among heads on token boundary detection.&#10;Chunkwise attention \cite{mocha} on top of each MA head is further extended to the multihead counterpart to extract useful representations and compensate the limited context size.&#10;Finally, we propose head-synchronous beam search decoding to guarantee streamable inference.&#10;&#10;Experimental evaluations on Librispeech corpus show that our proposed methods effectively encourage MA heads to learn alignments properly, which leads to improvement of ASR performance.&#10;Our optimal model enables stable streaming inference on other corpora as well without architecture modification.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Streaming Inference, Signal Processing, Mathematics, Speech Recognition, Attention Mechanisms" />
        </attvalues>
      </node>
      <node id="1705.05524" label="1705.05524">
        <attvalues>
          <attvalue for="0" value="Learning Hard Alignments with Variational Inference" />
          <attvalue for="1" value="  There has recently been significant interest in hard attention models for&#10;tasks such as object recognition, visual captioning and speech recognition.&#10;Hard attention can offer benefits over soft attention such as decreased&#10;computational cost, but training hard attention models can be difficult because&#10;of the discrete latent variables they introduce. Previous work used REINFORCE&#10;and Q-learning to approach these issues, but those methods can provide&#10;high-variance gradient estimates and be slow to train. In this paper, we tackle&#10;the problem of learning hard attention for a sequential task using variational&#10;inference methods, specifically the recently introduced VIMCO and NVIL.&#10;Furthermore, we propose a novel baseline that adapts VIMCO to this setting. We&#10;demonstrate our method on a phoneme recognition task in clean and noisy&#10;environments and show that our method outperforms REINFORCE, with the&#10;difference being greater for a more complicated task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.02958" label="1912.02958">
        <attvalues>
          <attvalue for="0" value="Synchronous Transformers for End-to-End Speech Recognition" />
          <attvalue for="1" value="  For most of the attention-based sequence-to-sequence models, the decoder&#10;predicts the output sequence conditioned on the entire input sequence processed&#10;by the encoder. The asynchronous problem between the encoding and decoding&#10;makes these models difficult to be applied for online speech recognition. In&#10;this paper, we propose a model named synchronous transformer to address this&#10;problem, which can predict the output sequence chunk by chunk. Once a&#10;fixed-length chunk of the input sequence is processed by the encoder, the&#10;decoder begins to predict symbols immediately. During training, a&#10;forward-backward algorithm is introduced to optimize all the possible alignment&#10;paths. Our model is evaluated on a Mandarin dataset AISHELL-1. The experiments&#10;show that the synchronous transformer is able to perform encoding and decoding&#10;synchronously, and achieves a character error rate of 8.91% on the test set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.12406" label="1909.12406">
        <attvalues>
          <attvalue for="0" value="Monotonic Multihead Attention" />
          <attvalue for="1" value="  Simultaneous machine translation models start generating a target sequence&#10;before they have encoded or read the source sequence. Recent approaches for&#10;this task either apply a fixed policy on a state-of-the art Transformer model,&#10;or a learnable monotonic attention on a weaker recurrent neural network-based&#10;structure. In this paper, we propose a new attention mechanism, Monotonic&#10;Multihead Attention (MMA), which extends the monotonic attention mechanism to&#10;multihead attention. We also introduce two novel and interpretable approaches&#10;for latency control that are specifically designed for multiple attentions&#10;heads. We apply MMA to the simultaneous machine translation task and&#10;demonstrate better latency-quality tradeoffs compared to MILk, the previous&#10;state-of-the-art approach. We also analyze how the latency controls affect the&#10;attention span and we motivate the introduction of our model by analyzing the&#10;effect of the number of decoder layers and heads on quality and latency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.02323" label="2011.02323">
        <attvalues>
          <attvalue for="0" value="Indic-Transformers: An Analysis of Transformer Language Models for&#10;  Indian Languages" />
          <attvalue for="1" value="  Language models based on the Transformer architecture have achieved&#10;state-of-the-art performance on a wide range of NLP tasks such as text&#10;classification, question-answering, and token classification. However, this&#10;performance is usually tested and reported on high-resource languages, like&#10;English, French, Spanish, and German. Indian languages, on the other hand, are&#10;underrepresented in such benchmarks. Despite some Indian languages being&#10;included in training multilingual Transformer models, they have not been the&#10;primary focus of such work. In order to evaluate the performance on Indian&#10;languages specifically, we analyze these language models through extensive&#10;experiments on multiple downstream tasks in Hindi, Bengali, and Telugu&#10;language. Here, we compare the efficacy of fine-tuning model parameters of&#10;pre-trained models against that of training a language model from scratch.&#10;Moreover, we empirically argue against the strict dependency between the&#10;dataset size and model performance, but rather encourage task-specific model&#10;and method selection. We achieve state-of-the-art performance on Hindi and&#10;Bengali languages for text classification task. Finally, we present effective&#10;strategies for handling the modeling of Indian languages and we release our&#10;model checkpoints for the community :&#10;https://huggingface.co/neuralspace-reverie.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Processing (nlp) has witnessed a paradigm shift from employing task-specific architectures to fine-tuning the same pre-trained language models for various downstream tasks \cite{Howard_Ruder_2018, Peters_Neumann_Iyyer_Gardner_Clark_Lee_Zettlemoyer_2018}. These language models are trained on large corpora of unlabelled text in an unsupervised manner. With the advent of Transformer-based architectures and various pre-training techniques in the last two years \cite{Devlin_Chang_Lee_Toutanova_2019, clark2020electra, Yang_Dai_Yang_Carbonell_Salakhutdinov_Le_2020, Liu_Ott_Goyal_Du_Joshi_Chen_Levy_Lewis_Zettlemoyer_Stoyanov_2019}, the state-of-the-art results have improved on various downstream tasks. However, much of this research has been limited to high-resource languages such as English, French, Spanish, and German \cite{Wu_Dredze_2020}. There are 7000+ languages spoken around the world and yet most of the nlp systems are largely evaluated on a handful of high-resource languages. Hence, there is a dire need to work on nlp beyond resource-rich languages \cite{ruder2020}. Our work contributes to filling this gap, as we focus on nlp for Indian languages.&#10;&#10;India is a multilingual country with only 10 percent of its population speaking English \cite{india}. There have been some concerted and noteworthy efforts to advance research in Indian languages \cite{arora2020inltk}, but very little work has been done with respect to the recently proposed Transformer-based models \cite{Devlin_Chang_Lee_Toutanova_2019, liu2019RoBERTaa}. Most of the recent works that have tried using these models for their applications have preferred using a multilingual version of bert, proposed as \acs{mbert} \cite{Devlin_Chang_Lee_Toutanova_2019}, rather than training monolingual language models from scratch, i.e., without transferring any parameters from other pre-trained models. However, \cite{Wu_Dredze_2020} shows that \acs{mbert} does not have high quality representations for all languages. For example, mbert performs worse than non-bert models on a number of downstream tasks for the bottom 30 percent of languages in terms of dataset-size upon which \acs{mbert} is trained~\cite{Wu_Dredze_2020}. Therefore, we see it as important to evaluate language models in the monolingual setting for low-resource languages. In this paper, we evaluate Transformer-based architectures for three Indian languages (Hindi, Bengali, and Telugu). To this end, our contributions are as follows:&#10; \begin{itemize}&#10; \item We train four variants of contextual monolingual language models, namely bert, \acs{distilbert}, \acs{roberta} and \acs{xlmroberta}, for three Indian languages (Hindi, Bengali and Telugu), which are spoken by more than 60 percent of the Indian population \cite{india}. These languages have different scripts based on an alphasyllabary system \cite{bright1999matter}.&#10; \item We present an exhaustive analysis of these models by evaluating them on multiple nlp tasks: qa, pos Tagging, and Text Classification. We conduct a wide range of experiments with three different setups that enable us to compare our language model variants with their multilingual counterparts and understand different factors that lead to better results on downstream tasks. &#10; \item For each setup, we evaluate models under two training settings which include fine-tuning the entire model and using the language model output as contextual embeddings. We use different layers, namely lstm~\cite{Hochreiter_Schmidhuber_2006}, \acs{bilstm}, fully connected and Transformer, on top of these embeddings.&#10; \item We present a correlation between the available dataset size for training vs the results obtained and empirically demonstrate that using Transformers as feature extractors can lead to competitive results in several downstream tasks. &#10; \item We plan to release our language model checkpoints to aid further research in nlp for Indian languages. Finally, we also intend to open-source mergedQuAD, a combination of XQuAD and MLQA datasets for Hindi, that allows training and comparison of qa models for Hindi.&#10; \end{itemize}&#10;&#10;Our paper is structured as follows. In the next Section \ref{background}, we place our research in context with related work in the downstream tasks we examine here. In Section \ref{transformer}, we provide the necessary theoretical background to the various Transformer-based models we anticipate to employ, and the subsequent Sections \ref{experiments} and \ref{results} explain our experimental setup and our results, respectively. We conclude our work with a brief discussion in Section \ref{conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Indian Language Processing, Artificial Intelligence, Language Model Performance, Multilingual NLP" />
        </attvalues>
      </node>
      <node id="2009.12534" label="2009.12534">
        <attvalues>
          <attvalue for="0" value="iNLTK: Natural Language Toolkit for Indic Languages" />
          <attvalue for="1" value="  We present iNLTK, an open-source NLP library consisting of pre-trained&#10;language models and out-of-the-box support for Data Augmentation, Textual&#10;Similarity, Sentence Embeddings, Word Embeddings, Tokenization and Text&#10;Generation in 13 Indic Languages. By using pre-trained models from iNLTK for&#10;text classification on publicly available datasets, we significantly outperform&#10;previously reported results. On these datasets, we also show that by using&#10;pre-trained models and data augmentation from iNLTK, we can achieve more than&#10;95% of the previous best performance by using less than 10% of the training&#10;data. iNLTK is already being widely used by the community and has 40,000+&#10;downloads, 600+ stars and 100+ forks on GitHub. The library is available at&#10;https://github.com/goru001/inltk.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.00768" label="2204.00768">
        <attvalues>
          <attvalue for="0" value="VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ&#10;  Acoustic Feature" />
          <attvalue for="1" value="  The mainstream neural text-to-speech(TTS) pipeline is a cascade system,&#10;including an acoustic model(AM) that predicts acoustic feature from the input&#10;transcript and a vocoder that generates waveform according to the given&#10;acoustic feature. However, the acoustic feature in current TTS systems is&#10;typically mel-spectrogram, which is highly correlated along both time and&#10;frequency axes in a complicated way, leading to a great difficulty for the AM&#10;to predict. Although high-fidelity audio can be generated by recent neural&#10;vocoders from ground-truth(GT) mel-spectrogram, the gap between the GT and the&#10;predicted mel-spectrogram from AM degrades the performance of the entire TTS&#10;system. In this work, we propose VQTTS, consisting of an AM txt2vec and a&#10;vocoder vec2wav, which uses self-supervised vector-quantized(VQ) acoustic&#10;feature rather than mel-spectrogram. We redesign both the AM and the vocoder&#10;accordingly. In particular, txt2vec basically becomes a classification model&#10;instead of a traditional regression model while vec2wav uses an additional&#10;feature encoder before HifiGAN generator for smoothing the discontinuous&#10;quantized feature. Our experiments show that vec2wav achieves better&#10;reconstruction performance than HifiGAN when using self-supervised VQ acoustic&#10;feature. Moreover, our entire TTS system VQTTS achieves state-of-the-art&#10;performance in terms of naturalness among all current publicly available TTS&#10;systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.13498" label="2302.13498">
        <attvalues>
          <attvalue for="0" value="Pretraining De-Biased Language Model with Large-scale Click Logs for&#10;  Document Ranking" />
          <attvalue for="1" value="  Pre-trained language models have achieved great success in various&#10;large-scale information retrieval tasks. However, most of pretraining tasks are&#10;based on counterfeit retrieval data where the query produced by the tailored&#10;rule is assumed as the user's issued query on the given document or passage.&#10;Therefore, we explore to use large-scale click logs to pretrain a language&#10;model instead of replying on the simulated queries. Specifically, we propose to&#10;use user behavior features to pretrain a debiased language model for document&#10;ranking. Extensive experiments on Baidu desensitization click logs validate the&#10;effectiveness of our method. Our team on WSDM Cup 2023 Pre-training for Web&#10;Search won the 1st place with a Discounted Cumulative Gain @ 10 (DCG@10) score&#10;of 12.16525 on the final leaderboard.&#10;" />
          <attvalue for="2" value="&#10;&#09;Recent advances have shown that pre-trained language models (PTMs) such as BERT~\cite{devlin2018bert}, T5~\cite{raffel2020exploring}, GPT~\cite{radford2018improving} can capture rich semantic information of text and achieve state-of-the-art performance on variance information retrieval tasks~\cite{qiao2019understanding,padaki2020rethinking,li2022cooperative}. However, the pretraining objectives of various PTMs are only based on classical NLP targets~\cite{devlin2018bert} (e.g., Masked Language Modeling and Next Sentence Prediction) and are not carefully explored to better adapt the downstream IR tasks. To address this problem, different pre-training methods with tailored IR objectives are proposed to obtain a better pre-trained language model for downstream IR finetuning tasks. Ma et al.~\cite{ma2021prop} proposed representative words prediction (ROP) task by assuming the sample word set with a higher query likelihood is more ``representative'' to the document. Besides, the dependencies between the inner structures in Wikipedia pages are also exploited to design pretraining tasks~\cite{ma2021pre, wu2022pre} for IR and achieve remarkable retrieval performance compared to the traditional pre-trained language models. These experimental results strongly suggest that traditional PTMs are usually data-hungry on IR tasks, pre-training with suitable IR tasks can effectively boost the performance of IR tasks even on few-shot or zero-shot scenarios~\cite{chen2022axiomatically}.&#10;&#10;Despite the success of various IR-based pretraining objectives on PTMs, we observe that these objectives are mostly designed based on manually tailored counterfeit retrieval data where the produced query are assumed as the user's issued query on the given document or passage. As large-scale click logs can be obtained costlessly, we argue that click logs can also be a good resource to pretrain IR-based PTMs. Since the queries are all from the submissions of real users and more close to the query distribution in the downstream IR tasks, designing IR-based pretraining objectives based on click logs provide great potential to improve the downstream IR tasks.&#10;&#10;In this work, we explore to use large-scale click logs to pretrain a language model with IR-based objectives. Specifically, we design a CTR prediction task and debiased CTR prediction task as our IR-based pretraining objectives. Furthermore, we extract other sparse features (e.g., BM25, document length, query frequency) and feed them into an ensemble learning model to rerank the candidate list. Our team on WSDM Cup 2023 Pre-training for Web Search won $the$ $1st$ $place$ with a Discounted Cumulative Gain&#10;&#09;@ 10 (DCG@10) score of 12.16525 on the final leaderboard. &#09;&#09;&#10;&#09;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, User Behavior Analysis, Linguistics, Data Science, Information Retrieval Tasks, Artificial Intelligence, Language Model Pretraining, Statistics" />
        </attvalues>
      </node>
      <node id="1904.07531" label="1904.07531">
        <attvalues>
          <attvalue for="0" value="Understanding the Behaviors of BERT in Ranking" />
          <attvalue for="1" value="  This paper studies the performances and behaviors of BERT in ranking tasks.&#10;We explore several different ways to leverage the pre-trained BERT and&#10;fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web&#10;Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the&#10;strong effectiveness of BERT in question-answering focused passage ranking&#10;tasks, as well as the fact that BERT is a strong interaction-based seq2seq&#10;matching model. Experimental results on TREC show the gaps between the BERT&#10;pre-trained on surrounding contexts and the needs of ad hoc document ranking.&#10;Analyses illustrate how BERT allocates its attentions between query-document&#10;tokens in its Transformer layers, how it prefers semantic matches between&#10;paraphrase tokens, and how that differs with the soft match patterns learned by&#10;a click-trained neural ranker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.09346" label="2108.09346">
        <attvalues>
          <attvalue for="0" value="Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need" />
          <attvalue for="1" value="  Designing pre-training objectives that more closely resemble the downstream&#10;tasks for pre-trained language models can lead to better performance at the&#10;fine-tuning stage, especially in the ad-hoc retrieval area. Existing&#10;pre-training approaches tailored for IR tried to incorporate weak supervised&#10;signals, such as query-likelihood based sampling, to construct pseudo&#10;query-document pairs from the raw textual corpus. However, these signals rely&#10;heavily on the sampling method. For example, the query likelihood model may&#10;lead to much noise in the constructed pre-training data. \blfootnote{$\dagger$&#10;This work was done during an internship at Huawei.} In this paper, we propose&#10;to leverage the large-scale hyperlinks and anchor texts to pre-train the&#10;language model for ad-hoc retrieval. Since the anchor texts are created by&#10;webmasters and can usually summarize the target document, it can help to build&#10;more accurate and reliable pre-training samples than a specific algorithm.&#10;Considering different views of the downstream ad-hoc retrieval, we devise four&#10;pre-training tasks based on the hyperlinks. We then pre-train the Transformer&#10;model to predict the pair-wise preference, jointly with the Masked Language&#10;Model objective. Experimental results on two large-scale ad-hoc retrieval&#10;datasets show the significant improvement of our model compared with the&#10;existing methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.02232" label="2106.02232">
        <attvalues>
          <attvalue for="0" value="Language Scaling for Universal Suggested Replies Model" />
          <attvalue for="1" value="  We consider the problem of scaling automated suggested replies for Outlook&#10;email system to multiple languages. Faced with increased compute requirements&#10;and low resources for language expansion, we build a single universal model for&#10;improving the quality and reducing run-time costs of our production system.&#10;However, restricted data movement across regional centers prevents joint&#10;training across languages. To this end, we propose a multi-task continual&#10;learning framework, with auxiliary tasks and language adapters to learn&#10;universal language representation across regions. The experimental results show&#10;positive cross-lingual transfer across languages while reducing catastrophic&#10;forgetting across regions. Our online results on real user traffic show&#10;significant gains in CTR and characters saved, as well as 65% training cost&#10;reduction compared with per-language models. As a consequence, we have scaled&#10;the feature in multiple languages including low-resource markets.&#10;" />
          <attvalue for="2" value="&#10;Automated suggested replies or smart replies (SR) assist users to quickly respond with a short, generic, and relevant response, without users having to type in the reply. SR is an increasingly&#10;popular feature in many commercial applications such as Gmail, Outlook, Skype, Facebook Messenger, Microsoft Teams, and Uber \cite{Kannan2016,HendersonASSLGK17,shang2015neural, Deb2019DiversifyingRS,uberSR}. While the initial versions of this feature mostly targeted English users, making it available in multiple languages and markets is important not only from the perspective of product expansion but also from a linguistic inclusivity point of view.&#10;&#10;In this paper we consider the problem of rapid scaling of the SR feature to multiple languages for Outlook. To develop such a system at production scale, we are faced with the following challenges.&#10;&#10;- Model management: Language scaling increases the effort of training, deploying, and managing per-language models, which needs to be replicated for each language. In addition, one model per language increases the storage and compute requirements for the production servers, which can increase costs and occurrences of run-time issues.&#10;&#10;- Data constraints: Developing models at production quality requires considerable effort in data collection and management. Due to regional market share and infrastructure constraints, rich and domain-specific data may not be available for all languages. &#10;&#10;- Data privacy and security policies: &#10;Regional policies enforce data to be located in corresponding regions. For example, Spanish and Portuguese data are stored in North American (NAM) clusters while French data is stored in European (EUR) clusters. Data movement across regions is not allowed and this prevents leveraging commonly used multi-lingual co-training methods which require all the data stored to be in the same place.&#10;&#10;To reduce the cost of model management, we propose to build a single universal SR model, capable of serving multiple languages and markets. To overcome data constraints, we propose to use augmentation with machine-translated (MT) data for languages without supervised data. &#10;To overcome privacy constraints, we propose a continual learning framework, where the model is trained sequentially across regions. To alleviate catastrophic forgetting \cite{french1999catastrophic,mccloskey1989catastrophic} in the continual learning process, &#10;we reinforce the universal properties via multi-task learning approach with public task-agnostic data, and an adapter-based model architecture that leverages domain-specific SR data and MT data.&#10;&#10;Our experimental results followed with improvements shown on real user traffic illustrate the effectiveness of the approach. As a consequence, we have rapidly scaled the feature in several languages including low-resource markets. Multi-lingual training for universal models is often very tricky to work in practice (especially with our data constraints). Thus, we demonstrate a significant accomplishment of a multi-lingual SR system running at production scale on millions of users, which saves resources while improving performance.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Scaling, Computer Science, Linguistics, Artificial Intelligence, Multilingual Processing, Automated Response Systems" />
        </attvalues>
      </node>
      <node id="1705.00652" label="1705.00652">
        <attvalues>
          <attvalue for="0" value="Efficient Natural Language Response Suggestion for Smart Reply" />
          <attvalue for="1" value="  This paper presents a computationally efficient machine-learned method for&#10;natural language response suggestion. Feed-forward neural networks using n-gram&#10;embedding features encode messages into vectors which are optimized to give&#10;message-response pairs a high dot-product value. An optimized search finds&#10;response suggestions. The method is evaluated in a large-scale commercial&#10;e-mail application, Inbox by Gmail. Compared to a sequence-to-sequence&#10;approach, the new system achieves the same quality at a small fraction of the&#10;computational requirements and latency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.08167" label="1907.08167">
        <attvalues>
          <attvalue for="0" value="OCC: A Smart Reply System for Efficient In-App Communications" />
          <attvalue for="1" value="  Smart reply systems have been developed for various messaging platforms. In&#10;this paper, we introduce Uber's smart reply system: one-click-chat (OCC), which&#10;is a key enhanced feature on top of the Uber in-app chat system. It enables&#10;driver-partners to quickly respond to rider messages using smart replies. The&#10;smart replies are dynamically selected according to conversation content using&#10;machine learning algorithms. Our system consists of two major components:&#10;intent detection and reply retrieval, which are very different from standard&#10;smart reply systems where the task is to directly predict a reply. It is&#10;designed specifically for mobile applications with short and non-canonical&#10;messages. Reply retrieval utilizes pairings between intent and reply based on&#10;their popularity in chat messages as derived from historical data. For intent&#10;detection, a set of embedding and classification techniques are experimented&#10;with, and we choose to deploy a solution using unsupervised distributed&#10;embedding and nearest-neighbor classifier. It has the advantage of only&#10;requiring a small amount of labeled training data, simplicity in developing and&#10;deploying to production, and fast inference during serving and hence highly&#10;scalable. At the same time, it performs comparably with deep learning&#10;architectures such as word-level convolutional neural network. Overall, the&#10;system achieves a high accuracy of 76% on intent detection. Currently, the&#10;system is deployed in production for English-speaking countries and 71% of&#10;in-app communications between riders and driver-partners adopted the smart&#10;replies to speedup the communication process.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16755" label="2305.16755">
        <attvalues>
          <attvalue for="0" value="Can large language models generate salient negative statements?" />
          <attvalue for="1" value="  We examine the ability of large language models (LLMs) to generate salient&#10;(interesting) negative statements about real-world entities; an emerging&#10;research topic of the last few years. We probe the LLMs using zero- and k-shot&#10;unconstrained probes, and compare with traditional methods for negation&#10;generation, i.e., pattern-based textual extractions and knowledge-graph-based&#10;inferences, as well as crowdsourced gold statements. We measure the correctness&#10;and salience of the generated lists about subjects from different domains. Our&#10;evaluation shows that guided probes do in fact improve the quality of generated&#10;negatives, compared to the zero-shot variant. Nevertheless, using both prompts,&#10;LLMs still struggle with the notion of factuality of negatives, frequently&#10;generating many ambiguous statements, or statements with negative keywords but&#10;a positive meaning.&#10;" />
          <attvalue for="2" value="&#10;&#10;Motivation and Problem. Structured (knowledge graphs), and unstructured (text corpora) information are the backbone of many AI applications, such as question answering and chat bots. They mainly focus on storing positive knowledge, and mostly contain little negative knowledge. The open-world assumption, which advises to abstain from taking a stance on the truth of absent information, compromises the usability of both forms of machine knowledge. For instance, it is often the case that the NBA's Basketball stars take a coaching position after their retirement. Notably, this is not true for michael jordan. Mining these surprising statements are useful to overcome limitations of applications like question answering systems. For example, querying Bing Chat whether michael jordan invested in his team, the chicago bulls, returns an irrelevant answer about his achievements with the team. In fact, it is an interesting piece of information, that, even though he has a business-oriented mind, he did not monetarily invest in the bulls, but in other sports franchise, including an investment in the not so well-known team the charlotte hornets. &#10;&#10;State of the Art. A new research area has emerged in the last few years, suggesting the importance of the explicit materialization of important negative statements about real-world subjects \cite{survey}. Several methodologies have been proposed ~\cite{arnaoutcikm2022,arnaout2020enriching,safavi-etal-2021-negater,quasimodo,antikb}. The goal is to compile lists of statements (biographic summaries) about subjects, where the statements are truly negative, but also salient, unexpected, or normally mistaken as true positives. To compile these lists, different data sources and methodologies have been explored. In~\cite{arnaoutcikm2022,arnaout2020enriching}, using web-scale knowledge graphs, candidate salient negatives are derived from existing positive statements about highly related entities. The computation relies on the local closed-world assumption, an assumption of completeness over identified relevant subgraphs, coupled with ranking metrics such as relative frequencies. Similarly, ~\cite{safavi-etal-2021-negater} explores graph embeddings to generate candidate negative statements, which are then scored using a fine-tuned language model (LM), by descending order of negativity. Textual sources have been explored in ~\cite{quasimodo}, where commonsense negative statements are extracted, by mining query logs, using pre-defined patterns. \cite{antikb} makes use of the edit history of large collaborative encyclopedias, namely Wikipedia, by looking at sentences edited, where only an entity or a number are changed. The old version of the sentence is then considered an interesting negative statement.&#10;&#10;LLMs for Negative Statements Generation. Recently, LMs have been examined about their ability to store factual knowledge about general topics~\cite{lee-etal-2020-language,petroni-etal-2019-language}. With LMs such as BERT~\cite{BERT}, this was done via masked probing, e.g., ``Paris is the the capital of '' generates france as the top prediction. With large LMs (LLMs), such as GPT-3~\cite{gpt3}, autoregressive generation from textual prompts is the standard, e.g., ``Complete the following. Paris is..'', and receive the completion the capital of France. A few papers focused on the ability of these models to store and understand negative knowledge~\cite{negatedlama,arnaoutcikm2022,chen2023say}. In~\cite{negatedlama}, using masked probing, authors found that LMs, such as BERT, struggle to understand negation, predicting fly for the probe ``Birds cannot ''. In~\cite{arnaoutcikm2022}, methods to infer negative statements from knowledge graphs and text have been compared on a more specific negation task, namely generating salient negative commonsense statements. Results of these models are compared to ones using GPT-3. Even though performing better than BERT-like models~\cite{negatedlama}, GPT-3 was not able to beat the SOTA model (inferences from KGs), neither on the true negativity of statements, nor their salience. More recently, \cite{chen2023say} studies advanced LLMs, such as ChatGPT~\cite{chatgpt}, on their ability to store negative knowledge in a constrained text generation and question answering tasks. The finding are contradictions in the LLM's belief, when comparing results of both tasks. For instance, LLMs generate the sentence ``Lions live in the ocean'', but answer ``No'' when asked ``Do lions live in the ocean?''. \cite{chen2023say} is an important step towards examining LLMs' understanding of the falseness of statements, however, it has four main differences from our study: (i) our prompts are not constrained to commonsense knowledge; (ii) not constrained to puzzles around a set of words, but allowed to generate arbitrary subject-relevant statements; (iii) our comparison includes SOTA baselines from KG and text, not just LLMs; (iv) our study evaluates also the salience of outputs, not just their correctness.&#10;&#10;We summarize our contributions as follows.&#10;\begin{itemize}&#10; \item We design constraint-free prompts for LLM-based negation generation, where we only instantiate the input subject.&#10; \item We examine LLMs' understanding of salient factual negation, finding that, even though they struggle with the notion of true negativity (-18\% in correctness compared to SOTA model), on truly negative statements, the guided few-shot ChatGPT variant ranks first among models in salience.&#10; \item We study both encyclopedic and commonsense domains, finding that it is more challenging for LLMs to generate longer lists of salient commonsense negatives. For instance, the zero-shot ChatGPT variant shows a decrease of 22\% in correctness@5 (compared to @1) for commonsense subjects. No decrease is observed for encyclopedic subjects. &#10; \item We compare the LLM-generated negative statements to existing SOTA methods, from text~\cite{quasimodo} and knowledge graphs~\cite{arnaout2020enriching}.&#10; \item We measure the quality of the negative statements over two aspects, the correctness (true negativity) and salience (interestingness).&#10;\end{itemize}&#10;&#10;The data generated can be downloaded at:.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Factuality Assessment, Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Negation Generation Methods, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.05403" label="2305.05403">
        <attvalues>
          <attvalue for="0" value="Completeness, Recall, and Negation in Open-World Knowledge Bases: A&#10;  Survey" />
          <attvalue for="1" value="  General-purpose knowledge bases (KBs) are a cornerstone of knowledge-centric&#10;AI. Many of them are constructed pragmatically from Web sources, and are thus&#10;far from complete. This poses challenges for the consumption as well as the&#10;curation of their content. While several surveys target the problem of&#10;completing incomplete KBs, the first problem is arguably to know whether and&#10;where the KB is incomplete in the first place, and to which degree.&#10;  In this survey we discuss how knowledge about completeness, recall, and&#10;negation in KBs can be expressed, extracted, and inferred. We cover (i) the&#10;logical foundations of knowledge representation and querying under partial&#10;closed-world semantics; (ii) the estimation of this information via statistical&#10;patterns; (iii) the extraction of information about recall from KBs and text;&#10;(iv) the identification of interesting negative statements; and (v) relaxed&#10;notions of relative recall.&#10;  This survey is targeted at two types of audiences: (1) practitioners who are&#10;interested in tracking KB quality, focusing extraction efforts, and building&#10;quality-aware downstream applications; and (2) data management, knowledge base&#10;and semantic web researchers who wish to understand the state of the art of&#10;knowledge bases beyond the open-world assumption. Consequently, our survey&#10;presents both fundamental methodologies and their working, and gives&#10;practice-oriented recommendations on how to choose between different approaches&#10;for a problem at hand.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.09292" label="2208.09292">
        <attvalues>
          <attvalue for="0" value="UnCommonSense: Informative Negative Knowledge about Everyday Concepts" />
          <attvalue for="1" value="  Commonsense knowledge about everyday concepts is an important asset for AI&#10;applications, such as question answering and chatbots. Recently, we have seen&#10;an increasing interest in the construction of structured commonsense knowledge&#10;bases (CSKBs). An important part of human commonsense is about properties that&#10;do not apply to concepts, yet existing CSKBs only store positive statements.&#10;Moreover, since CSKBs operate under the open-world assumption, absent&#10;statements are considered to have unknown truth rather than being invalid. This&#10;paper presents the UNCOMMONSENSE framework for materializing informative&#10;negative commonsense statements. Given a target concept, comparable concepts&#10;are identified in the CSKB, for which a local closed-world assumption is&#10;postulated. This way, positive statements about comparable concepts that are&#10;absent for the target concept become seeds for negative statement candidates.&#10;The large set of candidates is then scrutinized, pruned and ranked by&#10;informativeness. Intrinsic and extrinsic evaluations show that our method&#10;significantly outperforms the state-of-the-art. A large dataset of informative&#10;negations is released as a resource for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.07497" label="2011.07497">
        <attvalues>
          <attvalue for="0" value="NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge&#10;  Bases" />
          <attvalue for="1" value="  Codifying commonsense knowledge in machines is a longstanding goal of&#10;artificial intelligence. Recently, much progress toward this goal has been made&#10;with automatic knowledge base (KB) construction techniques. However, such&#10;techniques focus primarily on the acquisition of positive (true) KB statements,&#10;even though negative (false) statements are often also important for&#10;discriminative reasoning over commonsense KBs. As a first step toward the&#10;latter, this paper proposes NegatER, a framework that ranks potential negatives&#10;in commonsense KBs using a contextual language model (LM). Importantly, as most&#10;KBs do not contain negatives, NegatER relies only on the positive knowledge in&#10;the LM and does not require ground-truth negative examples. Experiments&#10;demonstrate that, compared to multiple contrastive data augmentation&#10;approaches, NegatER yields negatives that are more grammatical, coherent, and&#10;informative -- leading to statistically significant accuracy improvements in a&#10;challenging KB completion task and confirming that the positive knowledge in&#10;LMs can be &quot;re-purposed&quot; to generate negative knowledge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.10989" label="1905.10989">
        <attvalues>
          <attvalue for="0" value="Commonsense Properties from Query Logs and Question Answering Forums" />
          <attvalue for="1" value="  Commonsense knowledge about object properties, human behavior and general&#10;concepts is crucial for robust AI applications. However, automatic acquisition&#10;of this knowledge is challenging because of sparseness and bias in online&#10;sources. This paper presents Quasimodo, a methodology and tool suite for&#10;distilling commonsense properties from non-standard web sources. We devise&#10;novel ways of tapping into search-engine query logs and QA forums, and&#10;combining the resulting candidate assertions with statistical cues from&#10;encyclopedias, books and image tags in a corroboration step. Unlike prior work&#10;on commonsense knowledge bases, Quasimodo focuses on salient properties that&#10;are typically associated with certain objects or concepts. Extensive&#10;evaluations, including extrinsic use-case studies, show that Quasimodo provides&#10;better coverage than state-of-the-art baselines with comparable quality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.04102" label="2006.04102">
        <attvalues>
          <attvalue for="0" value="Language Models as Fact Checkers?" />
          <attvalue for="1" value="  Recent work has suggested that language models (LMs) store both common-sense&#10;and factual knowledge learned from pre-training data. In this paper, we&#10;leverage this implicit knowledge to create an effective end-to-end fact checker&#10;using a solely a language model, without any external knowledge or explicit&#10;retrieval components. While previous work on extracting knowledge from LMs have&#10;focused on the task of open-domain question answering, to the best of our&#10;knowledge, this is the first work to examine the use of language models as fact&#10;checkers. In a closed-book setting, we show that our zero-shot LM approach&#10;outperforms a random baseline on the standard FEVER task, and that our&#10;fine-tuned LM compares favorably with standard baselines. Though we do not&#10;ultimately outperform methods which use explicit knowledge bases, we believe&#10;our exploration shows that this method is viable and has much room for&#10;exploration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.05976" label="2305.05976">
        <attvalues>
          <attvalue for="0" value="Say What You Mean! Large Language Models Speak Too Positively about&#10;  Negative Commonsense Knowledge" />
          <attvalue for="1" value="  Large language models (LLMs) have been widely studied for their ability to&#10;store and utilize positive knowledge. However, negative knowledge, such as&#10;&quot;lions don't live in the ocean&quot;, is also ubiquitous in the world but rarely&#10;mentioned explicitly in the text. What do LLMs know about negative knowledge?&#10;This work examines the ability of LLMs to negative commonsense knowledge. We&#10;design a constrained keywords-to-sentence generation task (CG) and a Boolean&#10;question-answering task (QA) to probe LLMs. Our experiments reveal that LLMs&#10;frequently fail to generate valid sentences grounded in negative commonsense&#10;knowledge, yet they can correctly answer polar yes-or-no questions. We term&#10;this phenomenon the belief conflict of LLMs. Our further analysis shows that&#10;statistical shortcuts and negation reporting bias from language modeling&#10;pre-training cause this conflict.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.01691" label="2105.01691">
        <attvalues>
          <attvalue for="0" value="Data Augmentation by Concatenation for Low-Resource Translation: A&#10;  Mystery and a Solution" />
          <attvalue for="1" value="  In this paper, we investigate the driving factors behind concatenation, a&#10;simple but effective data augmentation method for low-resource neural machine&#10;translation. Our experiments suggest that discourse context is unlikely the&#10;cause for the improvement of about +1 BLEU across four language pairs. Instead,&#10;we demonstrate that the improvement comes from three other factors unrelated to&#10;discourse: context diversity, length diversity, and (to a lesser extent)&#10;position shifting.&#10;" />
          <attvalue for="2" value="&#10;&#10;Many attempts have been made to augment neural machine translation (MT) systems to use discourse context \cite{doc-level-1, doc-level-2, doc-level-3, doc-level-4, doc-level-5, doc-level-6, doc-level-7, doc-level-8, doc-level-9, context-1}. One particularly simple method is to concatenate consecutive pairs of sentence-pairs during training, but not during translation \cite{agrawal2018contextual, context-2, context-4, sent-concat}. In this paper, we confirm that this simple method helps, by roughly $+1$ BLEU across four low-resource language pairs. But we demonstrate that the reason it helps is not discourse context, because concatenating random pairs of sentence-pairs yields the same improvement.&#10;&#10;Instead, we view concatenation as a kind of data augmentation or noising method (one which pleasantly requires no alteration to the text, unlike data augmentation methods that disturb word order \cite{nmt-noise-1, nmt-noise-2} or replace words with automatically-selected words \cite{nmt-aug-1, nmt-aug-2, nmt-aug-3}). &#10;Concatenating random sentences is easier than concatenating consecutive sentences, because many parallel corpora discard document boundaries, drop sentence-pairs, or even reorder sentence-pairs, so it can be difficult to know which sentence-pairs are truly consecutive. &#10;&#10;But the fact that random concatenation helps so much creates a mystery, which is the focus of the paper. If the reason is not discourse context, what is the reason?&#10;We consider three new hypotheses:&#10;\begin{itemize}&#10;\item Random concatenation creates greater diversity of positions, because it lets the model see sentences shifted by effectively random distances.&#10;\item Random concatenation creates greater diversity of contexts, helping the model learn what not to attend to.&#10;\item Random concatenation creates greater diversity of sentence lengths within a minibatch.&#10;\end{itemize}&#10;Through a careful ablation study, we demonstrate that all three of these factors more or less contribute to the improvement, and together completely explain the improvement.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Language Context Analysis, Data Augmentation Methods, Statistics, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2311.16298" label="2311.16298">
        <attvalues>
          <attvalue for="0" value="Influence Scores at Scale for Efficient Language Data Sampling" />
          <attvalue for="1" value="  Modern ML systems ingest data aggregated from diverse sources, such as&#10;synthetic, human-annotated, and live customer traffic. Understanding&#10;\textit{which} examples are important to the performance of a learning&#10;algorithm is crucial for efficient model training. Recently, a growing body of&#10;literature has given rise to various &quot;influence scores,&quot; which use training&#10;artifacts such as model confidence or checkpointed gradients to identify&#10;important subsets of data. However, these methods have primarily been developed&#10;in computer vision settings, and it remains unclear how well they generalize to&#10;language-based tasks using pretrained models.&#10;  In this paper, we explore the applicability of influence scores in language&#10;classification tasks. We evaluate a diverse subset of these scores on the SNLI&#10;dataset by quantifying accuracy changes in response to pruning training data&#10;through random and influence-score-based sampling. We then stress-test one of&#10;the scores -- &quot;variance of gradients&quot; (VoG) from Agarwal et al. (2022) -- in an&#10;NLU model stack that was exposed to dynamic user speech patterns in a voice&#10;assistant type of setting. Our experiments demonstrate that in many cases,&#10;encoder-based language models can be finetuned on roughly 50% of the original&#10;data without degradation in performance metrics. Along the way, we summarize&#10;lessons learned from applying out-of-the-box implementations of influence&#10;scores, quantify the effects of noisy and class-imbalanced data, and offer&#10;recommendations on score-based sampling for better accuracy and training&#10;efficiency.&#10;" />
          <attvalue for="2" value="&#10;&#10;A salient challenge in training transformer-based models is selecting which examples are most important for learning. Understanding the relative importance of training examples towards model performance can inform data selection strategies that minimize customer privacy risks associated with the collection of training data, estimate the impact of the removal of copyrighted or sensitive data, determine mixing strategies to augment monolingual and multilingual datasets to improve accuracy, and identify defective subsets of data. At the same time, in cases where it is desirable to train on as much data as possible -- such as large language models -- determining the influence of different data instances (both contextually and during pretraining) can help identify failure modes at the level of specific tokens \cite{grosse2023studying}, determine the impact of removal of intellectual property, and significantly reduce costs through more efficient model training \cite{renduchintala2023ingenious}.&#10;&#10;A growing body of literature in the science of deep learning aims to capture this hierarchy of example importance and has led to a proliferation of a number of &quot;difficulty&quot; or &quot;influence&quot; scores (e.g., \cite{EL2N, VoG, TonevaForgetting, VUsable, tracin, beyond_scaling_pruning, dataset_cartography}; see App.~\ref{app:review} for a more complete review). These scores use various training artifacts, such as the margin of confidence or the variance of loss gradients, to rank the relative contribution of each example to model performance. &#10;This ranking of examples can then be used in many downstream tasks that require intelligent data selection, such as pruning datasets while maintaining or even improving model accuracy \cite{EL2N, beyond_scaling_pruning, marion2023more}; identifying outliers and misannotations in labeled data \cite{tracin, VUsable, DBLP:conf/nips/Pleiss0EW20, carlini_outliers, DBLP:conf/nips/FeldmanZ20}; or reweighting/reordering training examples to increase model robustness \cite{DBLP:conf/icml/RenZYU18, DBLP:conf/iclr/WuDN21}. &#10;&#10;Apart from a few notable exceptions \cite{dataset_cartography, VUsable, marion2023more}, influence scores have primarily been developed and demonstrated in the context of image classification, and relatively little is known about their efficacy in downstream language-based tasks. The application of these scores to data selection is further complicated by the fact that during fine-tuning, modern ML systems often ingest a vast amount of data that come from multiple sources, such as synthetic, weak signal, live customer data, and human-annotated. Beyond quantifying the efficacy of influence scores in this highly mixed data setting, there is an operational question of the existence of a simple, scalable influence score that can be easily accommodated in a production workflow. &#10;&#10;In this work, we take a first pass at answering these questions. First, we benchmark a subset of influence scores on the SNLI dataset \cite{snli:emnlp2015} in the downstream task of data reduction using a pretrained BERT model \cite{BERTDevlin}. Given the task of pruning a language dataset for fine-tuning, are influence scores useful signals for determining optimal data selection strategies? If so, which scores work best? We evaluate these scores against a random sampling baseline, in both noisy and clean data settings. &#10;&#10;User speech patterns are constantly evolving due to current events as well as user-system interactions that can be difficult to anticipate. Are influence scores still effective in surfacing data critical for model performance in this dynamic setting? To answer this question, we build upon on our initial findings on SNLI and implement one influence score (&quot;variance of gradients&quot; or &quot;VoG&quot;, first presented in \cite{VoG}) in a generic, large-scale NLU model stack commonly found in commercial voice assistants. We present results for existing in-house test data as well as results for a live user study in which we leveraged VoG scores for the purpose of substantially reducing training data without incurring model-performance degradation. &#10;&#10;Among the five influence scores we evaluated on SNLI, most out-of-the-box implementations do not beat a baseline of randomly pruning the dataset. The implementations can be improved to do better than the random-pruning baseline, but this typically requires careful experimentation to tune hyperparameters specific to each score. Out of the scores we tested, we find that VoG performs best relative to the random-pruning baseline, particularly at large pruning fractions. Test accuracy is mostly maintained after pruning $\sim$45\% of the SNLI training data using VoG scores calculated in a ``one-shot'' fashion, i.e. from a single training run, without any score hyperparameter tuning.&#10;&#10;In a large-scale user study performed using the NLU stack, we find that sampling by VoG scores is effective at surfacing training data that is particularly efficient for learning. We prune roughly 50\% of training data without incurring statistically significant regressions in key metrics that track NLU errors, relative to a baseline model trained with all data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Data Science, Machine Learning Efficiency, Artificial Intelligence, Data Sampling Techniques, Statistics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2308.03296" label="2308.03296">
        <attvalues>
          <attvalue for="0" value="Studying Large Language Model Generalization with Influence Functions" />
          <attvalue for="1" value="  When trying to gain better visibility into a machine learning model in order&#10;to understand and mitigate the associated risks, a potentially valuable source&#10;of evidence is: which training examples most contribute to a given behavior?&#10;Influence functions aim to answer a counterfactual: how would the model's&#10;parameters (and hence its outputs) change if a given sequence were added to the&#10;training set? While influence functions have produced insights for small&#10;models, they are difficult to scale to large language models (LLMs) due to the&#10;difficulty of computing an inverse-Hessian-vector product (IHVP). We use the&#10;Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC)&#10;approximation to scale influence functions up to LLMs with up to 52 billion&#10;parameters. In our experiments, EK-FAC achieves similar accuracy to traditional&#10;influence function estimators despite the IHVP computation being orders of&#10;magnitude faster. We investigate two algorithmic techniques to reduce the cost&#10;of computing gradients of candidate training sequences: TF-IDF filtering and&#10;query batching. We use influence functions to investigate the generalization&#10;patterns of LLMs, including the sparsity of the influence patterns, increasing&#10;abstraction with scale, math and programming abilities, cross-lingual&#10;generalization, and role-playing behavior. Despite many apparently&#10;sophisticated forms of generalization, we identify a surprising limitation:&#10;influences decay to near-zero when the order of key phrases is flipped.&#10;Overall, influence functions give us a powerful new tool for studying the&#10;generalization properties of LLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.07075" label="2107.07075">
        <attvalues>
          <attvalue for="0" value="Deep Learning on a Data Diet: Finding Important Examples Early in&#10;  Training" />
          <attvalue for="1" value="  Recent success in deep learning has partially been driven by training&#10;increasingly overparametrized networks on ever larger datasets. It is therefore&#10;natural to ask: how much of the data is superfluous, which examples are&#10;important for generalization, and how do we find them? In this work, we make&#10;the striking observation that, in standard vision datasets, simple scores&#10;averaged over several weight initializations can be used to identify important&#10;examples very early in training. We propose two such scores -- the Gradient&#10;Normed (GraNd) and the Error L2-Norm (EL2N) scores -- and demonstrate their&#10;efficacy on a range of architectures and datasets by pruning significant&#10;fractions of training data without sacrificing test accuracy. In fact, using&#10;EL2N scores calculated a few epochs into training, we can prune half of the&#10;CIFAR10 training set while slightly improving test accuracy. Furthermore, for a&#10;given dataset, EL2N scores from one architecture or hyperparameter&#10;configuration generalize to other configurations. Compared to recent work that&#10;prunes data by discarding examples that are rarely forgotten over the course of&#10;training, our scores use only local information early in training. We also use&#10;our scores to detect noisy examples and study training dynamics through the&#10;lens of important examples -- we investigate how the data distribution shapes&#10;the loss surface and identify subspaces of the model's data representation that&#10;are relatively stable over training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.09050" label="1803.09050">
        <attvalues>
          <attvalue for="0" value="Learning to Reweight Examples for Robust Deep Learning" />
          <attvalue for="1" value="  Deep neural networks have been shown to be very powerful modeling tools for&#10;many supervised learning tasks involving complex input patterns. However, they&#10;can also easily overfit to training set biases and label noises. In addition to&#10;various regularizers, example reweighting algorithms are popular solutions to&#10;these problems, but they require careful tuning of additional hyperparameters,&#10;such as example mining schedules and regularization hyperparameters. In&#10;contrast to past reweighting methods, which typically consist of functions of&#10;the cost value of each example, in this work we propose a novel meta-learning&#10;algorithm that learns to assign weights to training examples based on their&#10;gradient directions. To determine the example weights, our method performs a&#10;meta gradient descent step on the current mini-batch example weights (which are&#10;initialized from zero) to minimize the loss on a clean unbiased validation set.&#10;Our proposed method can be easily implemented on any type of deep network, does&#10;not require any additional hyperparameter tuning, and achieves impressive&#10;performance on class imbalance and corrupted label problems where only a small&#10;amount of clean validation data is available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.03538" label="2401.03538">
        <attvalues>
          <attvalue for="0" value="Transfer the linguistic representations from TTS to accent conversion&#10;  with non-parallel data" />
          <attvalue for="1" value="  Accent conversion aims to convert the accent of a source speech to a target&#10;accent, meanwhile preserving the speaker's identity. This paper introduces a&#10;novel non-autoregressive framework for accent conversion that learns&#10;accent-agnostic linguistic representations and employs them to convert the&#10;accent in the source speech. Specifically, the proposed system aligns speech&#10;representations with linguistic representations obtained from Text-to-Speech&#10;(TTS) systems, enabling training of the accent voice conversion model on&#10;non-parallel data. Furthermore, we investigate the effectiveness of a&#10;pretraining strategy on native data and different acoustic features within our&#10;proposed framework. We conduct a comprehensive evaluation using both subjective&#10;and objective metrics to assess the performance of our approach. The evaluation&#10;results highlight the benefits of the pretraining strategy and the&#10;incorporation of richer semantic features, resulting in significantly enhanced&#10;audio quality and intelligibility.&#10;" />
          <attvalue for="2" value="&#10;&#10;Foreign Accent Conversion (FAC) primarily aims to modify an individual's speech so that it closely resembles a native accent. This process entails adjusting several linguistic elements, such as pronunciation, intonation, and rhythm, while maintaining the identity of the speaker. The accent conversion not only is helpful for pronunciation correction in second-language(L2) learners, but also has potential applications in personalized Text-to-Speech synthesis(TTS)~\cite{Oshima_Takamichi_Toda_Neubig_Sakti_Nakamura_2021}, movie dubbing~\cite{Türk_Arslan_2002}, improving speech recognition performance~\cite{Biadsy_Weiss_Moreno_Kanvesky_Jia_2019}. However, this task remains particularly challenging due to the extreme lack of parallel corpora and the large variations in different speakers. &#10;&#10;The community has witnessed the rapid developement of AC in the past few years. Early works on AC &#10;focus on reference-based ideas~\cite{Zhao_Sonsaat_Levis_Chukharev-Hudilainen_Gutierrez-Osuna_2018, Zhao_Ding_Gutierrez-Osuna_2019, Li_Tang_Yin_Zhao_Li_Wang_Huang_Wang_Ma_2020, ding2022accentron}, these methods typically need target accent speech to provide the reference pronunciation in inference phases, which limits their practical applications.&#10;More recent efforts aim to build a accent conversion system by utilizing the reference-free approaches, which can be broadly categorized into two types: parallel data based methods~\cite{Zhao_Ding_Gutierrez-Osuna_2021, quamer2022zero, Nguyen_Pham_Waibel} and non-parallel data based methods. Approaches belonging to the former type require speech of the same sentence spoken in both the source accent and the target accent simultaneously. Nonetheless, the acquisition of such parallel corpora poses significant challenges, both in terms of cost and feasibility. There are some attempts~\cite{liu2020end,Jin_Serai_Wu_Tjandra_Manohar_He_2022,Zhou_Wu_Zhang_Tian_Li} on learning AC with non-parallel data, the auto-regressive based methods ~\cite{liu2020end,Zhou_Wu_Zhang_Tian_Li} also suffer from the low inference speed and unstable attention, especially when there are multiple speakers.&#10;&#10;To tackle the aforementioned issues, we present a novel reference-free non-autoregressive accent conversion framework that learns accent-agnostic linguistic representations and utilize them to convert the accent in the source speech. Specifically, we assume that the linguistic hidden-states are accent-independent, and firstly train a native Fastspeech2~\cite{ren2020fastspeech} model to learn the accent-independent linguistic representations. Furthermore, we use native speech data to learn the alignment between linguistic hidden space and speech hidden space. Finally, we employ accent speech to adapt the alignment module, enabling it to map the accent speech to an accent-agnostic hidden space and drive the decoder to generate native speech mel-spectrogram.&#10;&#10;We validate our method by extensive experiments on L2ARCTIC~\cite{zhao2018l2} dataset. The results show that our method achieves competitive performance compared with baseline. The main contribution of our works are three-folds:&#10;&#10;$\bullet$ We propose an novel reference-free non-autoregressive accent conversion framework, which leverages the alignment between the speech and text idea to allow the model learned with non-parallel data.&#10;&#10;$\bullet$ We further explore different input features and pretraining strategy within this framework. The results of experiments demonstrate that these modifications improve audio quality and intelligibility.&#10;&#10;$\bullet$ Finally, we conduct comprehensive evaluation, including both objective and subjective metrics, illustrating the effectiveness of our framework.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Acoustics, Voice Conversion, Computer Science, Speech Processing, Machine Learning, Linguistics, Accent Conversion, Speech Representation, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1904.04169" label="1904.04169">
        <attvalues>
          <attvalue for="0" value="Parrotron: An End-to-End Speech-to-Speech Conversion Model and its&#10;  Applications to Hearing-Impaired Speech and Speech Separation" />
          <attvalue for="1" value="  We describe Parrotron, an end-to-end-trained speech-to-speech conversion&#10;model that maps an input spectrogram directly to another spectrogram, without&#10;utilizing any intermediate discrete representation. The network is composed of&#10;an encoder, spectrogram and phoneme decoders, followed by a vocoder to&#10;synthesize a time-domain waveform. We demonstrate that this model can be&#10;trained to normalize speech from any speaker regardless of accent, prosody, and&#10;background noise, into the voice of a single canonical target speaker with a&#10;fixed accent and consistent articulation and prosody. We further show that this&#10;normalization model can be adapted to normalize highly atypical speech from a&#10;deaf speaker, resulting in significant improvements in intelligibility and&#10;naturalness, measured via a speech recognizer and listening tests. Finally,&#10;demonstrating the utility of this model on other speech tasks, we show that the&#10;same model architecture can be trained to perform a speech separation task&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.13282" label="2211.13282">
        <attvalues>
          <attvalue for="0" value="Voice-preserving Zero-shot Multiple Accent Conversion" />
          <attvalue for="1" value="  Most people who have tried to learn a foreign language would have experienced&#10;difficulties understanding or speaking with a native speaker's accent. For&#10;native speakers, understanding or speaking a new accent is likewise a difficult&#10;task. An accent conversion system that changes a speaker's accent but preserves&#10;that speaker's voice identity, such as timbre and pitch, has the potential for&#10;a range of applications, such as communication, language learning, and&#10;entertainment. Existing accent conversion models tend to change the speaker&#10;identity and accent at the same time. Here, we use adversarial learning to&#10;disentangle accent dependent features while retaining other acoustic&#10;characteristics. What sets our work apart from existing accent conversion&#10;models is the capability to convert an unseen speaker's utterance to multiple&#10;accents while preserving its original voice identity. Subjective evaluations&#10;show that our model generates audio that sound closer to the target accent and&#10;like the original speaker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.03145" label="2407.03145">
        <attvalues>
          <attvalue for="0" value="Enhancing Translation Accuracy of Large Language Models through&#10;  Continual Pre-Training on Parallel Data" />
          <attvalue for="1" value="  In this paper, we propose a two-phase training approach where pre-trained&#10;large language models are continually pre-trained on parallel data and then&#10;supervised fine-tuned with a small amount of high-quality parallel data. To&#10;investigate the effectiveness of our proposed approach, we conducted continual&#10;pre-training with a 3.8B-parameter model and parallel data across eight&#10;different formats. We evaluate these methods on thirteen test sets for&#10;Japanese-to-English and English-to-Japanese translation. The results&#10;demonstrate that when utilizing parallel data in continual pre-training, it is&#10;essential to alternate between source and target sentences. Additionally, we&#10;demonstrated that the translation accuracy improves only for translation&#10;directions where the order of source and target sentences aligns between&#10;continual pre-training data and inference. In addition, we demonstrate that the&#10;LLM-based translation model is more robust in translating spoken language and&#10;achieves higher accuracy with less training data compared to supervised&#10;encoder-decoder models. We also show that the highest accuracy is achieved when&#10;the data for continual pre-training consists of interleaved source and target&#10;sentences and when tags are added to the source sentences.&#10;" />
          <attvalue for="2" value="&#10;&#10;In machine translation, transformer encoder-decoder models~\cite{NIPS2017_transformer}, such as NLLB-200~\cite{nllb2022nolanguage}, mT5~\cite{xue-etal-2021-mt5}, and mBART~\cite{liu-etal-2020-multilingual-denoising} predominate. The emergence of pre-trained Large Language Models (LLMs) composed solely of the transformer decoder, such as GPT series \cite{NEURIPS2020_gpt3,openai2023gpt4}, has prompted the development of pre-trained LLMs, including, PaLM~\cite{chowdhery2022palm}, and LLaMA~\cite{touvron2023llama}. When translating with these LLMs, it is common to use in-context few-shot learning. According to \cite{hendy2023good}, GPT-3 demonstrates comparable or superior accuracy to WMT-best for high-resource languages. Furthermore, as reported by \cite{kocmi-etal-2023-findings}, GPT-4's 5-shot surpasses WMT-best's accuracy in most translation directions. However, \cite{zhu-etal-2024-multilingual} noted that in 8-shot scenarios, relatively small-scale LLMs (e.g., 7B parameters) exhibit lower accuracy than supervised encoder-decoder models. Therefore, it is necessary to investigate methods capable of achieving translation accuracy equivalent to existing translation models with relatively small-scale LLMs. &#10;&#10;On the other hand, in models such as BERT \cite{devlin-etal-2019-bert} and RoBERTa \cite{liu2019roberta}, which consist solely of transformer encoders, the effectiveness of continual pre-training, where pre-trained models are further trained on task-specific data such as classification to improve the accuracy of the task, has been reported \cite{jin-etal-2022-lifelong, ke-etal-2022-continual}. &#10;{In the context of LLMs, continual pre-training has been &#10;reported to transfer models primarily pre-trained in English, such as LLaMA, &#10;to other languages \cite{cui2023efficient}.} {Additionally, &#10;when building LLM-based translation models, &#10;the effectiveness of conducting continual pre-training with either monolingual data, parallel data, or both, &#10;followed by supervised fine-tuning, has been reported, &#10;mainly when basing the model on primarily English &#10;pre-trained models such as LLaMA-2~\cite{xu2024paradigm, alves2024tower, guo-etal-2024-novel}.}&#10;&#10;{&#10;Although those recent publicaion in the context of LLMs&#10;are closely related to our study,&#10;this paper presents research conducted independently &#10;of those latest LLM-based translation studies &#10;such as \cite{xu2024paradigm, alves2024tower, guo-etal-2024-novel}.&#10;}&#10;This paper proposes a two-phase training approach: continual pre-training on parallel data crawled from the web and supervised fine-tuning using a small amount of high-quality parallel data created by professional translators. To comprehensively investigate methods for improving translation accuracy through continual pre-training, we conduct continual pre-training across eight data formats for Japanese-to-English (Ja $\Rightarrow$ En) and English-to-Japanese (En $\Rightarrow$ Ja) translations using a 3.8B-parameter LLM. We evaluate the translation accuracy on 13 test sets. &#10;{&#10;Our paper's novelty &#10;compared to \cite{xu2024paradigm, alves2024tower, guo-etal-2024-novel}&#10;lies in the following aspects.}&#10;\begin{itemize}&#10; \item When conducting continual pre-training on data where source and target sentences appear alternately, the direction of language in which accuracy improves varies depending on the order of source and target sentences. &#10; \item {LLM-based translation model is more robust in translating spoken language and achieves higher accuracy with less training data compared to supervised encoder-decoder models.}&#10; \item {When indicating the translation direction with tags&#10; (``&lt;2en&gt;'' etc.) on data for continual pre-training, higher accuracy is achieved compared to simply concatenating source and target sentences.}&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Language Model Training, Artificial Intelligence, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="2406.00023" label="2406.00023">
        <attvalues>
          <attvalue for="0" value="Expert-Token Resonance: Redefining MoE Routing through Affinity-Driven&#10;  Active Selection" />
          <attvalue for="1" value="  Mixture-of-Experts (MoE) architectures have emerged as a paradigm-shifting&#10;approach for large language models (LLMs), offering unprecedented computational&#10;efficiency. However, these architectures grapple with challenges of token&#10;distribution imbalance and expert homogenization, impeding optimal semantic&#10;generalization. We introduce a novel framework that redefines MoE routing&#10;through affinity-driven active selection. The innovations for the framework&#10;encompass: (1) A rigorous formulation of expert-token affinity metrics. (2) An&#10;adaptive bidirectional selection mechanism leveraging resonance between experts&#10;and tokens. (3) Theoretical derivation and experimental evidence of reduced&#10;expert capacity bounds under dynamic token distribution evolution. It is also&#10;integrated with orthogonal feature extraction module and an optimized loss&#10;function for expert localization. Our theoretical analysis demonstrates that&#10;this approach mitigates expert homogenization while enabling substantial&#10;capacity boundary reduction. Experimental validation corroborates these&#10;findings: it achieves a 40% reduction in token processed by each expert without&#10;compromising model convergence or efficacy. When coupled with communication&#10;optimizations, the training efficiency improvements of 5.4% to 46.6% can be&#10;observed. After supervised fine-tuning, it exhibits performance gains of 9.7%&#10;to 14.1% across GDAD, C-Eval, and TeleQnA benchmarks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have shown exceptional proficiency in understanding deep structures and complex semantic relationships within language \cite{zhao2023survey}. As these models scale up, their capabilities in language generation and logical comprehension are enhanced, but this comes at the cost of significant computational, communication, and storage demands \cite{jiang2024megascale}. To scale models efficiently without disproportionately increasing computational costs, researchers have incorporated the Mixture-of-Experts (MoE) architecture into LLMs \cite{lepikhin2020gshard}. The MoE framework integrates multiple experts within the model, each tasked with processing specific types of inputs \cite{fedus2022switch}. For a given input, only a subset of experts is activated, allowing for more efficient use of computational resources \cite{du2022glam}. Recently, several LLMs employing MoE structures, such as DeepSeek-V3 \cite{liu2024deepseek} and Mixtral \cite{jiang2024mixtral}, have demonstrated outstanding performance on various leaderboards.&#10;&#10;Despite the efficiency benefits of MoE in scaling model sizes, it introduces several new challenges and drawbacks \cite{shazeer2017outrageously}. The conventional MoE model's convergence and the experts' generalization capabilities are heavily dependent on the design of the routing strategy, which easily leads to an imbalanced &quot;winner-takes-all&quot; phenomenon among experts. The imbalance between excessively &quot;developed&quot; experts and those lacking adequate training may compromise or even nullify the intended functionality of routing strategies. Recent studies address these challenges from multiple perspectives \cite{li2023accelerating}. StableMoE \cite{dai2022stablemoe} proposes a two-stage training approach to address the issue of routing fluctuation. This method involves training the routing network independently from the backbone model and utilizing a frozen, distilled routing mechanism to allocate tokens. Dynamic-MoE \cite{huang2024harder} designs a dynamic routing Mixture-of-Experts (MoE) policy that evaluates the sufficiency of current experts while reducing activated parameters by 90\%. The characteristics of classical gated routing lead to experts being unable to learn features mastered by other experts. To address this, MoDE \cite{xie2024mode} proposes moderate distillation between experts to mitigate the generalization problems caused by narrow learning paths. DYNMoE \cite{guo2024dynamic} introduces a unique gated routing mechanism capable of adaptively determining the number of activated experts through trainable expert thresholds, even allowing for the addition or removal of experts.&#10;&#10;In addition to the classical token choice scenario, previous researches also propose work utilizing expert choice (EC). Google Brain introduces the EC routing algorithm \cite{zhou2022mixture}, which assigns experts with predetermined buffer capacities to the Top-k tokens to ensure load balance. The Brainformer \cite{zhou2023brainformers} also adopts this routing strategy, constructing a trainable gating matrix to project the input feature space onto scores corresponding to each expert. Then, each token is routed to the Top-k experts. This strategy is proven highly effective in achieving expert load balancing and enhancing expert learning outcomes. Autonomy-of-Experts models \cite{lv2025autonomy} design a novel MoE paradigm in which experts autonomously select themselves to process inputs by aware of its own capacity to effectively process a token.&#10;&#10;The design of routing strategy is crucial to the MoE structure, while not all tokens may be suitable for training \cite{riquelme2021scaling}. In addition to data preprocessing techniques such as dataset cleaning and deduplication, previous studies have also considered how to discard certain tokens within the model. Early work introduced the concept of expert capacity \cite{lepikhingshard}, which refers to the maximum number of tokens each expert can process at once. Tokens exceeding this capacity are discarded. Expert capacity helps to ensure load balance among experts while facilitating All-to-All communication implementation. However, in situations where it is uncertain whether a token contributes to training, there is a risk of discarding class-discriminative samples, potentially compromising the model's training outcomes. DeepSeek-V2 \cite{liu2024deepseek} designs a device-limited routing mechanism to bound MoE-related communication cost. DeepSeek-V3 \cite{liu2024deepseek} pioneers an auxiliary-loss-free strategy to minimizes the performance degradation. This approach minimizes the constraints on expert specialization imposed by knowledge hybridity and knowledge redundancy. XMoE \cite{yang2024enhancing} achieves more precise router by implementing a threshold-based approach. If a token reaches the specified threshold, it is processed exclusively by a single expert while being discarded by other experts within the Top-k selection. This method allows for more nuanced token selection and processing. LocMoE \cite{li2024locmoe} leverages orthogonal routing weights to prevent token homogenization across different expert networks and introduces the Grouped Average Pooling (GrAP) layer \cite{wang2023multi} for token feature extraction. Under these conditions, LocMoE also provides the theoretical proof for the lower bound of expert capacity. &#10;&#10;In this paper, we propose expert-token resonance, a mechanism consisting of an expert-token bidirectional selection router and the adaptive expert capacity strategy. The primary contributions of this paper are as follows:&#10;&#10;\begin{enumerate}&#10;&#09;\item Affinity-based Efficient Expert Routing via GrAP. By leveraging cosine similarity between tokens and gating weights to define affinity scores, our router effectively guides experts to focus on distinct token segments, mitigating the expert homogenization problem. Meanwhile, the GrAP design reduces computational complexity by a factor of 1/2D to 1/D compared to traditional MLPs (D denotes the dimension of the intermediate hidden layer). This integrated approach demonstrates both improved routing effectiveness and substantial computational efficiency.&#10;&#10;\item Expert-token Bidirectional Selection. By integrating the concepts of expert choice router (ECR) and token choice router (TCR), we propose the adaptive bidirectional selection mechanism. Contrast to conventional router, the bidirectional selection router allows MoE to enhance the training success rate while considering expert capacity constraints. Its effectiveness has been theoretically validated.&#10;&#10;\item The Adaptive Expert Capacity Bound. Setting an adaptive affinity threshold allows the lower bound of expert capacity to be significantly reduced. As training iterations increase, the information density of token features grows, causing the expert capacity to initially decrease and then stabilize. Ultimately, the training efficiency of MoE can be greatly enhanced.&#10;&#10;\end{enumerate}&#10;&#10;Expert-token resonance mechanism adopts the state-of-the-art MoE model Mixtral 8$\times$7B as the backbone, and utilizes MindSpeed-LLM, MindSpeed, and Megatron-LM \cite{shoeybi2019megatron} libraries for training on Ascend NPU clusters. Ascend designs a new computing architecture for LLM training and inference scenarios \cite{liao2021ascend}, boasting powerful low-bit computing capabilities. Experiments conducted on clusters with 32, 64, and 256 NPUs indicate that our approach improves training efficiency by 5.4\% to 46.6\% compared to the baseline, and by 2.9\% to 13.3\% compared to LocMoE. Model performance is enhanced by 9.7\% to 14.1\% compared to the baseline, and by 1.7\% to 4.1\% compared to LocMoE.&#10;&#10;The rest of this paper is structured as follows: Section Method presents the methods proposed in this paper, along with theoretical evidence. Section Experiments analyzes the experimental results of our approach regarding training efficiency and model performance. The final sectionsummarizes the content of this paper and offers an outlook on future improvements.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Artificial Intelligence, Mathematics, Efficient Routing Mechanisms, Mixture-of-Experts Architectures" />
        </attvalues>
      </node>
      <node id="2204.08396" label="2204.08396">
        <attvalues>
          <attvalue for="0" value="StableMoE: Stable Routing Strategy for Mixture of Experts" />
          <attvalue for="1" value="  The Mixture-of-Experts (MoE) technique can scale up the model size of&#10;Transformers with an affordable computational overhead. We point out that&#10;existing learning-to-route MoE methods suffer from the routing fluctuation&#10;issue, i.e., the target expert of the same input may change along with&#10;training, but only one expert will be activated for the input during inference.&#10;The routing fluctuation tends to harm sample efficiency because the same input&#10;updates different experts but only one is finally used. In this paper, we&#10;propose StableMoE with two training stages to address the routing fluctuation&#10;problem. In the first training stage, we learn a balanced and cohesive routing&#10;strategy and distill it into a lightweight router decoupled from the backbone&#10;model. In the second training stage, we utilize the distilled router to&#10;determine the token-to-expert assignment and freeze it for a stable routing&#10;strategy. We validate our method on language modeling and multilingual machine&#10;translation. The results show that StableMoE outperforms existing MoE methods&#10;in terms of both convergence speed and performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.07652" label="2403.07652">
        <attvalues>
          <attvalue for="0" value="Harder Tasks Need More Experts: Dynamic Routing in MoE Models" />
          <attvalue for="1" value="  In this paper, we introduce a novel dynamic expert selection framework for&#10;Mixture of Experts (MoE) models, aiming to enhance computational efficiency and&#10;model performance by adjusting the number of activated experts based on input&#10;difficulty. Unlike traditional MoE approaches that rely on fixed Top-K routing,&#10;which activates a predetermined number of experts regardless of the input's&#10;complexity, our method dynamically selects experts based on the confidence&#10;level in expert selection for each input. This allows for a more efficient&#10;utilization of computational resources, activating more experts for complex&#10;tasks requiring advanced reasoning and fewer for simpler tasks. Through&#10;extensive evaluations, our dynamic routing method demonstrates substantial&#10;improvements over conventional Top-2 routing across various benchmarks,&#10;achieving an average improvement of 0.7% with less than 90% activated&#10;parameters. Further analysis shows our model dispatches more experts to tasks&#10;requiring complex reasoning skills, like BBH, confirming its ability to&#10;dynamically allocate computational resources in alignment with the input's&#10;complexity. Our findings also highlight a variation in the number of experts&#10;needed across different layers of the transformer model, offering insights into&#10;the potential for designing heterogeneous MoE frameworks. The code and models&#10;are available at https://github.com/ZhenweiAn/Dynamic_MoE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.14297" label="2405.14297">
        <attvalues>
          <attvalue for="0" value="Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient&#10;  Transformer Models" />
          <attvalue for="1" value="  The Sparse Mixture of Experts (SMoE) has been widely employed to enhance the&#10;efficiency of training and inference for Transformer-based foundational models,&#10;yielding promising results. However, the performance of SMoE heavily depends on&#10;the choice of hyper-parameters, such as the number of experts and the number of&#10;experts to be activated (referred to as top-k), resulting in significant&#10;computational overhead due to the extensive model training by searching over&#10;various hyper-parameter configurations. As a remedy, we introduce the Dynamic&#10;Mixture of Experts (DynMoE) technique. DynMoE incorporates (1) a novel gating&#10;method that enables each token to automatically determine the number of experts&#10;to activate. (2) An adaptive process automatically adjusts the number of&#10;experts during training. Extensive numerical results across Vision, Language,&#10;and Vision-Language tasks demonstrate the effectiveness of our approach to&#10;achieve competitive performance compared to GMoE for vision and language tasks,&#10;and MoE-LLaVA for vision-language tasks, while maintaining efficiency by&#10;activating fewer parameters. Our code is available at&#10;https://github.com/LINs-lab/DynMoE.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.09368" label="2202.09368">
        <attvalues>
          <attvalue for="0" value="Mixture-of-Experts with Expert Choice Routing" />
          <attvalue for="1" value="  Sparsely-activated Mixture-of-experts (MoE) models allow the number of&#10;parameters to greatly increase while keeping the amount of computation for a&#10;given token or a given sample unchanged. However, a poor expert routing&#10;strategy (e.g. one resulting in load imbalance) can cause certain experts to be&#10;under-trained, leading to an expert being under or over-specialized. Prior work&#10;allocates a fixed number of experts to each token using a top-k function&#10;regardless of the relative importance of different tokens. To address this, we&#10;propose a heterogeneous mixture-of-experts employing an expert choice method.&#10;Instead of letting tokens select the top-k experts, we have experts selecting&#10;the top-k tokens. As a result, each token can be routed to a variable number of&#10;experts and each expert can have a fixed bucket size. We systematically study&#10;pre-training speedups using the same computational resources of the Switch&#10;Transformer top-1 and GShard top-2 gating of prior work and find that our&#10;method improves training convergence time by more than 2x. For the same&#10;computational cost, our method demonstrates higher performance in fine-tuning&#10;11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller&#10;activation cost, our method outperforms the T5 dense model in 7 out of the 11&#10;tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.00008" label="2306.00008">
        <attvalues>
          <attvalue for="0" value="Brainformers: Trading Simplicity for Efficiency" />
          <attvalue for="1" value="  Transformers are central to recent successes in natural language processing&#10;and computer vision. Transformers have a mostly uniform backbone where layers&#10;alternate between feed-forward and self-attention in order to build a deep&#10;network. Here we investigate this design choice and find that more complex&#10;blocks that have different permutations of layer primitives can be more&#10;efficient. Using this insight, we develop a complex block, named Brainformer,&#10;that consists of a diverse sets of layers such as sparsely gated feed-forward&#10;layers, dense feed-forward layers, attention layers, and various forms of layer&#10;normalization and activation functions. Brainformer consistently outperforms&#10;the state-of-the-art dense and sparse Transformers, in terms of both quality&#10;and efficiency. A Brainformer model with 8 billion activated parameters per&#10;token demonstrates 2x faster training convergence and 5x faster step time&#10;compared to its GLaM counterpart. In downstream task evaluation, Brainformer&#10;also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM&#10;with a similar number of activated parameters. Finally, Brainformer largely&#10;outperforms a Primer dense model derived with NAS with similar computation per&#10;token on fewshot evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.05974" label="2106.05974">
        <attvalues>
          <attvalue for="0" value="Scaling Vision with Sparse Mixture of Experts" />
          <attvalue for="1" value="  Sparsely-gated Mixture of Experts networks (MoEs) have demonstrated excellent&#10;scalability in Natural Language Processing. In Computer Vision, however, almost&#10;all performant networks are &quot;dense&quot;, that is, every input is processed by every&#10;parameter. We present a Vision MoE (V-MoE), a sparse version of the Vision&#10;Transformer, that is scalable and competitive with the largest dense networks.&#10;When applied to image recognition, V-MoE matches the performance of&#10;state-of-the-art networks, while requiring as little as half of the compute at&#10;inference time. Further, we propose an extension to the routing algorithm that&#10;can prioritize subsets of each input across the entire batch, leading to&#10;adaptive per-image compute. This allows V-MoE to trade-off performance and&#10;compute smoothly at test-time. Finally, we demonstrate the potential of V-MoE&#10;to scale vision models, and train a 15B parameter model that attains 90.35% on&#10;ImageNet.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.13920" label="2401.13920">
        <attvalues>
          <attvalue for="0" value="LocMoE: A Low-Overhead MoE for Large Language Model Training" />
          <attvalue for="1" value="  The Mixtures-of-Experts (MoE) model is a widespread distributed and&#10;integrated learning method for large language models (LLM), which is favored&#10;due to its ability to sparsify and expand models efficiently. However, the&#10;performance of MoE is limited by load imbalance and high latency of All-to-All&#10;communication, along with relatively redundant computation owing to large&#10;expert capacity. Load imbalance may result from existing routing policies that&#10;consistently tend to select certain experts. The frequent inter-node&#10;communication in the All-to-All procedure also significantly prolongs the&#10;training time. To alleviate the above performance problems, we propose a novel&#10;routing strategy that combines load balance and locality by converting partial&#10;inter-node communication to that of intra-node. Notably, we elucidate that&#10;there is a minimum threshold for expert capacity, calculated through the&#10;maximal angular deviation between the gating weights of the experts and the&#10;assigned tokens. We port these modifications on the PanGu-Sigma model based on&#10;the MindSpore framework with multi-level routing and conduct experiments on&#10;Ascend clusters. The experiment results demonstrate that the proposed LocMoE&#10;reduces training time per epoch by 12.68% to 22.24% compared to classical&#10;routers, such as hash router and switch router, without impacting the model&#10;accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.07402" label="2107.07402">
        <attvalues>
          <attvalue for="0" value="CLSRIL-23: Cross Lingual Speech Representations for Indic Languages" />
          <attvalue for="1" value="  We present a CLSRIL-23, a self supervised learning based audio pre-trained&#10;model which learns cross lingual speech representations from raw audio across&#10;23 Indic languages. It is built on top of wav2vec 2.0 which is solved by&#10;training a contrastive task over masked latent speech representations and&#10;jointly learns the quantization of latents shared across all languages. We&#10;compare the language wise loss during pretraining to compare effects of&#10;monolingual and multilingual pretraining. Performance on some downstream&#10;fine-tuning tasks for speech recognition is also compared and our experiments&#10;show that multilingual pretraining outperforms monolingual training, in terms&#10;of learning speech representations which encodes phonetic similarity of&#10;languages and also in terms of performance on down stream tasks. A decrease of&#10;5% is observed in WER and 9.5% in CER when a multilingual pretrained model is&#10;used for finetuning in Hindi. All the code models are also open sourced.&#10;CLSRIL-23 is a model trained on $23$ languages and almost 10,000 hours of audio&#10;data to facilitate research in speech recognition for Indic languages. We hope&#10;that new state of the art systems will be created using the self supervised&#10;approach, especially for low resources Indic languages.&#10;" />
          <attvalue for="2" value="&#10;Speech recognition has made remarkable progress in the past few years especially after the advent of deep learning \cite{hinton}. End to end (E2E) models have surely simplified the modelling process but they are also notoriously known for huge amount of data requirements. Especially more so for low resource languages \cite{besacier2014automatic}.&#10;&#10;This is of particular importance for countries with many languages and dialects such as India which has 22 official languages with an additional 1500 minor languages/dialects. Apart from a few major languages, most of the languages are low resource, thereby making it more difficult to develop speech related technologies \cite{bourlard2011current}.&#10;&#10;(E2E) networks become an attractive choice for multilingual ASRs since they combine the acoustic model, pronunciation and lexicon model into a single network. One way to tackle with multiple languages using a single model would be to train a multilingual ASR model we can take a union over all the language characters and jointly train a model on all the languages. But even in that approach huge amounts of data is needed per language.&#10;&#10;In recent years self supervised learning has emerged as a new paradigm in which representations are learnt from the data itself and then fine tuning is done on several other down stream tasks. This approach has been widely successful in natural language processing (NLP) applications \cite{devlin2019bert,peters-etal-2018-deep} and is active area of research in other fields. &#10;&#10;In the past few years self supervised learning has been actively studied for speech recognition. In \cite{jiang2019improving} the authors perform unsupervised pretraining with masked predictive coding using a transformer model. Most of work in this space as well has been in monolingual speech recognition \cite{chung2018speech2vec,tjandra2019vqvae,jiang2019improving,harwath2020learning}.&#10;Our approach is based on the wav2vec $2.0$ \cite{baevski2020wav2vec} the details of which are explained in the coming sections. An approach which uses multiple languages in pre-training and fine-tuning is described in \cite{conneau2020unsupervised}. It also shows that cross lingual pre-training outperforms monolingual pre-training. We extend the work by \cite{rivire2020unsupervised} and \cite{conneau2020unsupervised} by pretraining only on Indic languages so that speech recognition tasks have a better performance on Indic languages.&#10;&#10;Languages spoken in the South Asian region belong to at least four major language&#10;families: Indo-European (most of which belong to its sub-branch Indo-Aryan), Dravidian, Austro-Asiatic, and Sino-Tibetan. Almost one third of our mother-tongues in India ($574$ languages) belong to the Indo-Aryan family of languages - spoken by $73.30$\% of Indians. The Dravidian languages, $153$ in number, form the second major linguistic group of the country ($24.47$\% ). Since most of the $23$ we have used are in common language families we aim to utilise language similarity to aid representation learning for low resource languages. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Signal Processing, Speech Recognition, Self Supervised Models, Multilingual Learning" />
        </attvalues>
      </node>
      <node id="2409.11889" label="2409.11889">
        <attvalues>
          <attvalue for="0" value="M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for&#10;  Enhancing Whisper" />
          <attvalue for="1" value="  State-of-the-art models like OpenAI's Whisper exhibit strong performance in&#10;multilingual automatic speech recognition (ASR), but they still face challenges&#10;in accurately recognizing diverse subdialects. In this paper, we propose&#10;M2R-whisper, a novel multi-stage and multi-scale retrieval augmentation&#10;approach designed to enhance ASR performance in low-resource settings. Building&#10;on the principles of in-context learning (ICL) and retrieval-augmented&#10;techniques, our method employs sentence-level ICL in the pre-processing stage&#10;to harness contextual information, while integrating token-level k-Nearest&#10;Neighbors (kNN) retrieval as a post-processing step to further refine the final&#10;output distribution. By synergistically combining sentence-level and&#10;token-level retrieval strategies, M2R-whisper effectively mitigates various&#10;types of recognition errors. Experiments conducted on Mandarin and subdialect&#10;datasets, including AISHELL-1 and KeSpeech, demonstrate substantial&#10;improvements in ASR accuracy, all achieved without any parameter updates.&#10;" />
          <attvalue for="2" value="&#10;&#10;The rapid advancements in deep learning have led to remarkable progress in automatic speech recognition (ASR) systems \cite{prabhavalkar2023end, Conformer, espnet, zhang2022wenet, cif-t, whisper, usm, mms}, resulting in models like OpenAI’s Whisper \cite{whisper}, Google’s USM \cite{usm}, and META’s MMS \cite{mms} that support multilingual speech recognition and achieve near state-of-the-art performance. However, these models still encounter difficulties in low-resource settings \cite{wang2024enhancing,slt-wsy, madi,childmandarin, stutter}, such as recognizing diverse subdialects \cite{basak2023challenges,zhu2023boosting,kothawade2023ditto,aksenova2022accented}. These limitations highlight the need for more flexible and adaptive methods to enhance ASR performance under such conditions.&#10;&#10;Against this backdrop, in-context learning (ICL) \cite{rubin2022learning,min2022rethinking} has emerged as a promising approach in natural language processing (NLP), allowing models to leverage relevant examples embedded within the input, enabling adaptation without finetuning.&#10;The success of ICL hinges on the quality of selected demonstrations \cite{agrawal2023context}, which poses unique challenges in speech processing due to the complexity of audio data. Existing approaches \cite{gao22e_interspeech, hsu2023exploration} often rely on random selection, leading to suboptimal results. For instance, WAVPROMPT \cite{gao22e_interspeech} combines Wav2vec2 with an autoregressive language model for few-shot ICL in speech tasks, while Hsu et al. \cite{hsu2023exploration} propose a text-free warmup strategy to enable ICL in speech models. However, both methods are limited to classification tasks, leaving ICL applications in ASR relatively underexplored.&#10;&#10;Recent studies have begun exploring the application of ICL in ASR, particularly in low-resource settings. For instance, SLAM \cite{chen2024salm} leverages large language models (LLMs) to enhance ASR performance by improving keyword recognition through ICL, while Audio Flamingo \cite{audioflamingo} enables fast adaptation to unseen tasks using sentence-level embeddings. However, Audio Flamingo requires an additional audio encoder, such as LAION-CLAP \cite{LAION-CLAP}, for retrieval. While these approaches utilize the ICL capabilities of LLMs, Wang et al. \cite{prompt-whisper} explore Whisper’s ICL potential for test-time adaptation in Chinese dialects, leveraging small labeled speech samples to reduce the Character Error Rate (CER). Nonetheless, their approach is limited to isolated word-level ICL, which constrains its applicability to broader ASR contexts.&#10;&#10;Beyond ICL, retrieval-augmented methods \cite{knn-lm,knn-mt,FAISS,xu2023nearest}, have been widely adopted in NLP to enhance model performance without parameter updating. Techniques like $k$NN-LM \cite{knn-lm} for language modeling and $k$NN-MT \cite{knn-mt} for machine translation have shown significant promise. In ASR, $k$NN-CTC \cite{knn-ctc} uses Connectionist Temporal Classification (CTC) pseudo labels to create speech-text key-value pairs and retrieves these labels during decoding to refine the output distribution, significantly improving performance in Chinese dialect ASR. Building on $k$NN-CTC, Zhou et al. \cite{knn-cs} further extend this approach with a gated mechanism for zero-shot code-switching ASR. However, these methods are constrained by the quality of the pseudo labels, and $k$NN-CTC primarily addresses substitution errors, showing instability in correcting other types of errors.&#10;&#10;While both ICL and $k$NN have demonstrated significant success in enhancing ASR performance, they differ substantially in their operational stages, retrieval scales, methods of leveraging external datastores, and the types of ASR errors they primarily address. By integrating these complementary approaches, we can achieve synergistic improvements in ASR. Based on this insight, we propose a multi-stage and multi-scale retrieval augmentation method for test-time adaptation, specifically tailored for Whisper, as illustrated in Figure \ref{intro}.&#10;&#10;Our approach extends word-level ICL \cite{prompt-whisper} to sentence-level ICL, enhancing Whisper’s in-context learning capabilities. While ICL serves as a pre-processing retrieval mechanism that leverages the model's intrinsic ability, its impact remains limited. To address this, we integrate post-processing token-level $k$NN retrieval to further refine the final output distribution, forming a multi-stage and multi-scale retrieval augmentation ASR system.&#10;On the other hand, while token-level $k$NN retrieval primarily focuses on reducing substitution errors, sentence-level retrieval effectively handles a broader range of errors. Combining these methods yields complementary effects, improving ASR’s ability to address diverse errors across different stages and scales.&#10;&#10;This approach facilitates rapid domain adaptation without requiring parameter updates. Extensive experiments on AISHELL-1 \cite{aishell} and KeSpeech \cite{kespeech} validate the effectiveness of our method, demonstrating that the integration of sentence-level and token-level datastores enhances ASR performance.&#10;&#10;Our main contributions are as following:&#10;&#10;1. We extend word-level ICL to sentence-level for Whisper by establishing a sentence-level datastore.&#10;&#10;2. We implement a token-level datastore to realize the $k$NN retrieval augmentation method for Whisper.&#10;&#10;3. We propose a multi-stage and multi-scale retrieval augmentation method, M2R-Whisper, which achieves complementary gains by combining both sentence-level and token-level approaches, without updating any parameters.&#10;&#10;4. Comprehensive experiments demonstrate the effectiveness of our method on both Mandarin and subdialect datasets.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Retrieval Augmentation, Linguistics, Multilingual Models, Artificial Intelligence, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2409.04799" label="2409.04799">
        <attvalues>
          <attvalue for="0" value="PB-LRDWWS System for the SLT 2024 Low-Resource Dysarthria Wake-Up Word&#10;  Spotting Challenge" />
          <attvalue for="1" value="  For the SLT 2024 Low-Resource Dysarthria Wake-Up Word Spotting (LRDWWS)&#10;Challenge, we introduce the PB-LRDWWS system. This system combines a dysarthric&#10;speech content feature extractor for prototype construction with a&#10;prototype-based classification method. The feature extractor is a fine-tuned&#10;HuBERT model obtained through a three-stage fine-tuning process using&#10;cross-entropy loss. This fine-tuned HuBERT extracts features from the target&#10;dysarthric speaker's enrollment speech to build prototypes. Classification is&#10;achieved by calculating the cosine similarity between the HuBERT features of&#10;the target dysarthric speaker's evaluation speech and prototypes. Despite its&#10;simplicity, our method demonstrates effectiveness through experimental results.&#10;Our system achieves second place in the final Test-B of the LRDWWS Challenge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04908" label="2110.04908">
        <attvalues>
          <attvalue for="0" value="DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent&#10;  Adaptation" />
          <attvalue for="1" value="  State-of-the-art Automatic Speech Recognition (ASR) systems are known to&#10;exhibit disparate performance on varying speech accents. To improve performance&#10;on a specific target accent, a commonly adopted solution is to finetune the ASR&#10;model using accent-specific labeled speech. However, acquiring large amounts of&#10;labeled speech for specific target accents is challenging. Choosing an&#10;informative subset of speech samples that are most representative of the target&#10;accents becomes important for effective ASR finetuning. To address this&#10;problem, we propose DITTO (Data-efficient and faIr Targeted subseT selectiOn)&#10;that uses Submodular Mutual Information (SMI) functions as acquisition&#10;functions to find the most informative set of utterances matching a target&#10;accent within a fixed budget. An important feature of DITTO is that it supports&#10;fair targeting for multiple accents, i.e. it can automatically select&#10;representative data points from multiple accents when the ASR model needs to&#10;perform well on more than one accent. We show that DITTO is 3-5 times more&#10;label-efficient than other speech selection methods on the IndicTTS and L2&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.08014" label="2205.08014">
        <attvalues>
          <attvalue for="0" value="Accented Speech Recognition: Benchmarking, Pre-training, and Diverse&#10;  Data" />
          <attvalue for="1" value="  Building inclusive speech recognition systems is a crucial step towards&#10;developing technologies that speakers of all language varieties can use.&#10;Therefore, ASR systems must work for everybody independently of the way they&#10;speak. To accomplish this goal, there should be available data sets&#10;representing language varieties, and also an understanding of model&#10;configuration that is the most helpful in achieving robust understanding of all&#10;types of speech. However, there are not enough data sets for accented speech,&#10;and for the ones that are already available, more training approaches need to&#10;be explored to improve the quality of accented speech recognition. In this&#10;paper, we discuss recent progress towards developing more inclusive ASR&#10;systems, namely, the importance of building new data sets representing&#10;linguistic diversity, and exploring novel training approaches to improve&#10;performance for all users. We address recent directions within benchmarking ASR&#10;systems for accented speech, measure the effects of wav2vec 2.0 pre-training on&#10;accented speech recognition, and highlight corpora relevant for diverse ASR&#10;evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.02437" label="2212.02437">
        <attvalues>
          <attvalue for="0" value="In-context Examples Selection for Machine Translation" />
          <attvalue for="1" value="  Large-scale generative models show an impressive ability to perform a wide&#10;range of Natural Language Processing (NLP) tasks using in-context learning,&#10;where a few examples are used to describe a task to the model. For Machine&#10;Translation (MT), these examples are typically randomly sampled from the&#10;development dataset with a similar distribution as the evaluation set. However,&#10;it is unclear how the choice of these in-context examples and their ordering&#10;impacts the output translation quality. In this work, we aim to understand the&#10;properties of good in-context examples for MT in both in-domain and&#10;out-of-domain settings. We show that the translation quality and the domain of&#10;the in-context examples matter and that 1-shot noisy unrelated example can have&#10;a catastrophic impact on output quality. While concatenating multiple random&#10;examples reduces the effect of noise, a single good prompt optimized to&#10;maximize translation quality on the development dataset can elicit learned&#10;information from the pre-trained language model. Adding similar examples based&#10;on an n-gram overlap with the test source significantly and consistently&#10;improves the translation quality of the outputs, outperforming a strong kNN-MT&#10;baseline in 2 out of 4 out-of-domain datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.01831" label="2402.01831">
        <attvalues>
          <attvalue for="0" value="Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and&#10;  Dialogue Abilities" />
          <attvalue for="1" value="  Augmenting large language models (LLMs) to understand audio -- including&#10;non-speech sounds and non-verbal speech -- is critically important for diverse&#10;real-world applications of LLMs. In this paper, we propose Audio Flamingo, a&#10;novel audio language model with 1) strong audio understanding abilities, 2) the&#10;ability to quickly adapt to unseen tasks via in-context learning and retrieval,&#10;and 3) strong multi-turn dialogue abilities. We introduce a series of training&#10;techniques, architecture design, and data strategies to enhance our model with&#10;these abilities. Extensive evaluations across various audio understanding tasks&#10;confirm the efficacy of our method, setting new state-of-the-art benchmarks.&#10;Our demo website is https://audioflamingo.github.io/ and the code is&#10;open-sourced at https://github.com/NVIDIA/audio-flamingo.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.07081" label="2309.07081">
        <attvalues>
          <attvalue for="0" value="Can Whisper perform speech-based in-context learning?" />
          <attvalue for="1" value="  This paper investigates the in-context learning abilities of the Whisper&#10;automatic speech recognition (ASR) models released by OpenAI. A novel&#10;speech-based in-context learning (SICL) approach is proposed for test-time&#10;adaptation, which can reduce the word error rates (WERs) with only a small&#10;number of labelled speech samples without gradient descent. Language-level&#10;adaptation experiments using Chinese dialects showed that when applying SICL to&#10;isolated word ASR, consistent and considerable relative WER reductions can be&#10;achieved using Whisper models of any size on two dialects, which is on average&#10;32.3%. A k-nearest-neighbours-based in-context example selection technique can&#10;be applied to further improve the efficiency of SICL, which can increase the&#10;average relative WER reduction to 36.4%. The findings are verified using&#10;speaker adaptation or continuous speech recognition tasks, and both achieved&#10;considerable relative WER reductions. Detailed quantitative analyses are also&#10;provided to shed light on SICL's adaptability to phonological variances and&#10;dialect-specific lexical nuances.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1702.08734" label="1702.08734">
        <attvalues>
          <attvalue for="0" value="Billion-scale similarity search with GPUs" />
          <attvalue for="1" value="  Similarity search finds application in specialized database systems handling&#10;complex data such as images or videos, which are typically represented by&#10;high-dimensional features and require specific indexing structures. This paper&#10;tackles the problem of better utilizing GPUs for this task. While GPUs excel at&#10;data-parallel tasks, prior approaches are bottlenecked by algorithms that&#10;expose less parallelism, such as k-min selection, or make poor use of the&#10;memory hierarchy.&#10;  We propose a design for k-selection that operates at up to 55% of theoretical&#10;peak performance, enabling a nearest neighbor implementation that is 8.5x&#10;faster than prior GPU state of the art. We apply it in different similarity&#10;search scenarios, by proposing optimized design for brute-force, approximate&#10;and compressed-domain search based on product quantization. In all these&#10;setups, we outperform the state of the art by large margins. Our implementation&#10;enables the construction of a high accuracy k-NN graph on 95 million images&#10;from the Yfcc100M dataset in 35 minutes, and of a graph connecting 1 billion&#10;vectors in less than 12 hours on 4 Maxwell Titan X GPUs. We have open-sourced&#10;our approach for the sake of comparison and reproducibility.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16367" label="2305.16367">
        <attvalues>
          <attvalue for="0" value="Role-Play with Large Language Models" />
          <attvalue for="1" value="  As dialogue agents become increasingly human-like in their performance, it is&#10;imperative that we develop effective ways to describe their behaviour in&#10;high-level terms without falling into the trap of anthropomorphism. In this&#10;paper, we foreground the concept of role-play. Casting dialogue agent behaviour&#10;in terms of role-play allows us to draw on familiar folk psychological terms,&#10;without ascribing human characteristics to language models they in fact lack.&#10;Two important cases of dialogue agent behaviour are addressed this way, namely&#10;(apparent) deception and (apparent) self-awareness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.13486" label="2208.13486">
        <attvalues>
          <attvalue for="0" value="naab: A ready-to-use plug-and-play corpus for Farsi" />
          <attvalue for="1" value="  Huge corpora of textual data are always known to be a crucial need for&#10;training deep models such as transformer-based ones. This issue is emerging&#10;more in lower resource languages - like Farsi. We propose naab, the biggest&#10;cleaned and ready-to-use open-source textual corpus in Farsi. It contains about&#10;130GB of data, 250 million paragraphs, and 15 billion words. The project name&#10;is derived from the Farsi word NAAB K which means pure and high grade. We also&#10;provide the raw version of the corpus called naab-raw and an easy-to-use&#10;preprocessor that can be employed by those who wanted to make a customized&#10;corpus.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) have revolutionized how people interact with technology, representing one of the most significant breakthroughs of the modern era~\cite{srivastava2022beyond,teubner2023welcome}.&#10;While these models have shown remarkable improvements across a wide range of tasks~\cite{chang2024survey} in English, their performance in low and mid-resource languages, such as Farsi, often lags behind~\cite{avetisyan2023large,shen2024understanding}.&#10;&#10;Pre-training LLMs to produce Pretrained Language Models (PLMs)~\cite{raffel2020exploring, devlin2018bert} requires vast amounts of data, and large textual corpora are essential for fine-tuning these models for specific languages.&#10;Given this process's time- and resource-intensive nature, having a readily available, large-scale corpus can significantly benefit researchers working to improve NLP in low-resource languages.&#10;&#10;The lack of large-scale Farsi text data has made fine-tuning large language models challenging~\cite{habib2021challenges,moniri2024investigating}.&#10;This limitation often restricts the ability to train such models to only a handful of well-funded companies or countries, creating an uneven playing field.&#10;As a result, this lack of accessibility can hinder progress in open science, where collaboration and shared resources are essential for advancing NLP research in these underrepresented languages.&#10;&#10;The largest previously available cleaned Farsi textual corpus was a 70GB dataset compiled from eight sources: Common Crawl - fa~\cite{CC-fa}, Miras Text~\cite{Miras}, W2C – Web to Corpus~\cite{majlivs2011w2c}, Persian Wikipedia~\cite{persian-raw-text}, Leipzig Corpora~\cite{biemann2007leipzig}, VOA corpus~\cite{voa-news}, Persian poems corpus~\cite{perisan-poems}, and the Tehran English-Persian parallel corpus (TEP)~\cite{TIEDEMANN12.463}. This corpus has been cleaned and is available for direct download.&#10;&#10;Meanwhile, several toolkits have been developed to streamline NLP workflows, including fine-tuning large models.&#10;These tools aim to democratize access to advanced NLP techniques and promote open science.&#10;A notable example is the Python library \verb|transformers|~\cite{wolf2020transformers}, which has become the standard for training and fine-tuning transformer-based models.&#10;Hugging Face has also introduced a range of integrated libraries that enhance accessibility and collaboration in various NLP tasks.&#10;&#10;Among these is the \verb|datasets| library~\cite{lhoest-etal-2021-datasets}, which provides open-source corpora that are easily accessible to NLP researchers.&#10;However, none of the existing Farsi corpora are available on \verb|datasets|.&#10;The first contribution of this work is to provide an easily accessible Farsi corpus, available to all through Hugging Face \verb|datasets|.&#10;&#10;One of the other primary challenges faced by NLP researchers is effective data pre-processing.&#10;Textual corpora, often derived from web-crawled data, frequently contain undesirable text and personal information.&#10;Traditional methods, such as trimming to remove unwanted patterns~\cite{raffel2020exploring, farahani2021parsbert}, are often computationally expensive and memory-intensive.&#10;In this technical report, we introduce a more efficient alternative: a streaming pipeline for pre-processing texts in Farsi, which addresses these issues in a streamlined and resource-efficient manner.&#10;&#10;Our solution to these challenges is embodied in the naab project, derived from the Farsi word \RL{ناب}, meaning &quot;pure&quot; or &quot;high-grade&quot;~\cite{abadis}.&#10;The corpus provides 126GB of training data, consisting of more than 224 million sequences and nearly 15 billion words, and 2.3GB of test data, containing nearly 11 million sequences and 300 million words.&#10;&#10;The main contributions of this project include the release of the largest cleaned and open-source Farsi corpus, naab, hosted on Hugging Face for easy accessibility; and the introduction of an easy-to-use, streaming-based pre-processing approach that enhances efficiency while maintaining high data quality.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Textual Data Corpus, Information Science, Language Resource Development, Natural Language Processing, Deep Model Training" />
        </attvalues>
      </node>
      <node id="2405.04655" label="2405.04655">
        <attvalues>
          <attvalue for="0" value="Understanding the Capabilities and Limitations of Large Language Models&#10;  for Cultural Commonsense" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated substantial commonsense&#10;understanding through numerous benchmark evaluations. However, their&#10;understanding of cultural commonsense remains largely unexamined. In this&#10;paper, we conduct a comprehensive examination of the capabilities and&#10;limitations of several state-of-the-art LLMs in the context of cultural&#10;commonsense tasks. Using several general and cultural commonsense benchmarks,&#10;we find that (1) LLMs have a significant discrepancy in performance when tested&#10;on culture-specific commonsense knowledge for different cultures; (2) LLMs'&#10;general commonsense capability is affected by cultural context; and (3) The&#10;language used to query the LLMs can impact their performance on&#10;cultural-related tasks. Our study points to the inherent bias in the cultural&#10;understanding of LLMs and provides insights that can help develop culturally&#10;aware language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.02846" label="2109.02846">
        <attvalues>
          <attvalue for="0" value="Datasets: A Community Library for Natural Language Processing" />
          <attvalue for="1" value="  The scale, variety, and quantity of publicly-available NLP datasets has grown&#10;rapidly as researchers propose new tasks, larger models, and novel benchmarks.&#10;Datasets is a community library for contemporary NLP designed to support this&#10;ecosystem. Datasets aims to standardize end-user interfaces, versioning, and&#10;documentation, while providing a lightweight front-end that behaves similarly&#10;for small datasets as for internet-scale corpora. The design of the library&#10;incorporates a distributed, community-driven approach to adding datasets and&#10;documenting usage. After a year of development, the library now includes more&#10;than 650 unique datasets, has more than 250 contributors, and has helped&#10;support a variety of novel cross-dataset research projects and shared tasks.&#10;The library is available at https://github.com/huggingface/datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.00571" label="2109.00571">
        <attvalues>
          <attvalue for="0" value="DILBERT: Customized Pre-Training for Domain Adaptation withCategory&#10;  Shift, with an Application to Aspect Extraction" />
          <attvalue for="1" value="  The rise of pre-trained language models has yielded substantial progress in&#10;the vast majority of Natural Language Processing (NLP) tasks. However, a&#10;generic approach towards the pre-training procedure can naturally be&#10;sub-optimal in some cases. Particularly, fine-tuning a pre-trained language&#10;model on a source domain and then applying it to a different target domain,&#10;results in a sharp performance decline of the eventual classifier for many&#10;source-target domain pairs. Moreover, in some NLP tasks, the output categories&#10;substantially differ between domains, making adaptation even more challenging.&#10;This, for example, happens in the task of aspect extraction, where the aspects&#10;of interest of reviews of, e.g., restaurants or electronic devices may be very&#10;different. This paper presents a new fine-tuning scheme for BERT, which aims to&#10;address the above challenges. We name this scheme DILBERT: Domain Invariant&#10;Learning with BERT, and customize it for aspect extraction in the unsupervised&#10;domain adaptation setting. DILBERT harnesses the categorical information of&#10;both the source and the target domains to guide the pre-training process&#10;towards a more domain and category invariant representation, thus closing the&#10;gap between the domains. We show that DILBERT yields substantial improvements&#10;over state-of-the-art baselines while using a fraction of the unlabeled data,&#10;particularly in more challenging domain adaptation setups.&#10;" />
          <attvalue for="2" value=" &#10;&#10;Aspect-based sentiment analysis (ABSA) \cite{thet2010aspect}, extracting aspect-sentiment pairs for products or services from reviews, is a widely researched task in both academia and industry. ABSA allows a fine-grained and realistic evaluation of reviews, as real-world reviews typically do not convey a homogeneous sentiment but rather communicate different sentiments for different aspects of the reviewed item or service. For example, while the overall sentiment of the review in Figure~\ref{fig:res_review} is unclear, the sentiment towards the service, food, and location of the restaurant is very decisive. Moreover, even when the overall sentiment of the review is clear, ABSA provides more nuanced and complete information about its content.&#10;&#10;In this paper, we focus on the sub-task of aspect extraction (AE, a.k.a opinion targets extraction): Extracting from opinionated texts the aspects on which the reader conveys sentiment. For example, in Figure~\ref{fig:res_review}, the waiter, food, and the views of the city are aspects derived from broader categories: service, food, and location. This task is characterized by a multiplicity of domains, as reviews and other opinionated texts can be written about a variety of products, services as well as many other issues. Moreover, the aspect categories of interest often differ between these domains.&#10;&#10;As for most NLP tasks and applications, AE research has recently made substantial progress. While Transformer \cite{DBLP:journals/corr/VaswaniSPUJGKP17} based pre-trained models \cite{devlin-etal-2019-bert, liu2019roberta} have pushed results substantially forward, they rely on in-domain labeled data to achieve their strong results. Annotating such data for multiple domains is costly and laborious, which is one of the major bottlenecks for developing and deploying NLP systems. As noted above, AE forms a particularly challenging variant of the domain adaptation problem, as the aspect categories of interest tend to change across domains.&#10;&#10;A well-established approach for addressing the above bottleneck is Domain Adaptation (DA) \cite{blitzer2006domain, ben2007analysis}. DA, training models on source domain labeled data so that they can effectively generalize to different target domains, is a long-standing research challenge. While the target domain labeled data availability in DA setups ranges from little (supervised DA \cite{daume:07}) to none (unsupervised DA \cite{ ramponi2020neural}), unlabeled data is typically available in both source and target domains. This paper focuses on unsupervised DA as we believe it is a more realistic and practical scenario. &#10;&#10;Due to the great success of deep learning models, &#10;DA through representation learning \cite{blitzer-etal-2007-biographies, ziser-reichart-2017-neural}, i.e., learning a shared representation for both the source and target domains, has recently become prominent \cite{ziser-reichart-2018-deep, ben-david-etal-2020-perl}. Of particular importance to this line of work are approaches that utilize pivot features \cite{blitzer2006domain} that: (a) frequently appear in both domains; and (b) have high mutual information (MI) with the task label. &#10;While pivot-based methods achieve state-of-the-art results in many text classification tasks \cite{ziser-reichart-2018-deep, miller2019simplified, ben-david-etal-2020-perl}, it is not trivial to successfully apply them on tasks such as AE. This stems from two reasons: First, AE is a sequence tagging task with multiple labels for each input example (i.e., word-level labels for input sentences). For a feature to meet the second condition for being a pivot (high MI with the task label), further refinement of the pivot definition is required. Second, different domains often differ in their aspect categories and hence if a feature is highly correlated with a source domain label (aspect category), this is not indicative of its being correlated with the (different) aspect categories of the target domain.&#10;&#10;To overcome these limitations, we present DILBERT: Domain Invariant Learning with BERT, a customized fine-tuning procedure for AE in an unsupervised DA setup. More specifically, DILBERT employs a variant of the BERT masked language modeling (MLM) task such that hidden tokens are chosen by their semantic similarity to the categories rather than randomly. Further, it employs a new pre-training task: The prediction of which categories appear in the input text. Notice that unlabeled text does not contain supervision for our pre-training tasks and we hence have to use distant supervision as an approximation.&#10;&#10;In our unsupervised DA experiments we consider laptop and restaurant reviews, where for the restaurant domain we consider two variants, that differ in their difficulty. Our best performing model outperforms the strongest baseline by over 5\% on average and over 13\% on the most challenging setup, while using only a small fraction of the unlabeled data. Moreover, we show that our pre-training procedure is very effective in resource-poor setups, where unlabeled data is scarce.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Domain Adaptation, Artificial Intelligence, Information Science, Natural Language Processing, Language Model Fine-tuning" />
        </attvalues>
      </node>
      <node id="0907.1815" label="0907.1815">
        <attvalues>
          <attvalue for="0" value="Frustratingly Easy Domain Adaptation" />
          <attvalue for="1" value="  We describe an approach to domain adaptation that is appropriate exactly in&#10;the case when one has enough ``target'' data to do slightly better than just&#10;using only ``source'' data. Our approach is incredibly simple, easy to&#10;implement as a preprocessing step (10 lines of Perl!) and outperforms&#10;state-of-the-art approaches on a range of datasets. Moreover, it is trivially&#10;extended to a multi-domain adaptation problem, where one has data from a&#10;variety of different domains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.15625" label="2406.15625">
        <attvalues>
          <attvalue for="0" value="Shortcomings of LLMs for Low-Resource Translation: Retrieval and&#10;  Understanding are Both the Problem" />
          <attvalue for="1" value="  This work investigates the in-context learning abilities of pretrained large&#10;language models (LLMs) when instructed to translate text from a low-resource&#10;language into a high-resource language as part of an automated machine&#10;translation pipeline. We conduct a set of experiments translating Southern&#10;Quechua to Spanish and examine the informativity of various types of&#10;information retrieved from a constrained database of digitized pedagogical&#10;materials (dictionaries and grammar lessons) and parallel corpora. Using both&#10;automatic and human evaluation of model output, we conduct ablation studies&#10;that manipulate (1) context type (morpheme translations, grammar descriptions,&#10;and corpus examples), (2) retrieval methods (automated vs. manual), and (3)&#10;model type. Our results suggest that even relatively small LLMs are capable of&#10;utilizing prompt context for zero-shot low-resource translation when provided a&#10;minimally sufficient amount of relevant linguistic information. However, the&#10;variable effects of prompt type, retrieval method, model type, and&#10;language-specific factors highlight the limitations of using even the best LLMs&#10;as translation systems for the majority of the world's 7,000+ languages and&#10;their speakers.&#10;" />
          <attvalue for="2" value="&#10;Despite great progress in the quality of today's state of the art machine translation (MT) systems, constraints on the amount and kinds of data available in the majority of the world's 7,000+ languages have led to yet another disparity in access and support for speakers of these languages: low-resource MT continues to be a major challenge \cite{hendy2023good, nicholas2023lost, robinson-etal-2023-chatgpt, stap-araabi-2023-chatgpt}. Although many languages lack the kinds of large, standardized corpora necessary for traditional MT methods, recent work suggests it may be possible to leverage a smaller amount of existing resources, for example pedagogical materials used for language instruction, to develop MT systems with Large Language Models (LLMs), albeit with varying results \cite{elsner-needle-2023, tanzer2024mtob,zhang2024teaching}.&#10;These materials are often the result of community-driven or government-led initiatives to support language revitalization, reclamation, and mother-tongue education \cite{schreiner-etal-2020-multidirectional,liu-etal-2022-notalways, riestenberg-freemond-2024-prioritizing}. Such discrepancies in the needs and priorities of&#10;academic, commercial, and community-led &#10;efforts to develop digital resources and language technologies is what \cite{gessler-2022-nlpgap} terms the ``NLP Gap''.&#10;&#10;In this study, we investigate one way to lessen the NLP Gap,&#10;comparing LLMs' in-context learning abilities when translating from a low-resource language (a Peruvian variety of Southern Quechua) to a high-resource language (Spanish) using information retrieved from a database of pedagogical materials. We replicate results of earlier studies on a new language pair by comparing the effects of morpheme translations, sentences from a parallel corpus, and passages from a grammar instruction document on translation quality. We then conduct a more focused analysis by annotating translation outputs by hand using a modified MQM error typology \cite{burchardt-2013-mqm}. Finally, we conduct an ablation study on the effects of automated retrieval by manually constructing prompts using the same set of materials.&#10;&#10;Our results suggest that while, unsurprisingly, translation quality improves with model size, such improvements seem to primarily be the result of previous exposure to the low-resource language during model pretraining, rather than an improved ability for the model to utilize prompt context, as evidenced by high scores in response to baseline (zero-shot) translation prompts. However, we also find evidence that in-context learning abilities may be inconsistent across different models of similar size. As found in previous studies, prompts containing morpheme and word-level translations reliably improve model outputs, but information from the grammar and corpus have a null or even negative effect on results. &#10;Human evaluation on a selection of outputs from two models -- GPT-3.5 Turbo and GPT-4o -- &#10;align with the quantitative measures we obtain using BLEURT \cite{sellam2020bleurt} as an automatic metric. Quantitative results also show an effect of automated retrieval on translation quality that is most evident in prompts containing morpheme translations and for models with lower baseline scores. Finally, we highlight a number of ethical concerns and limitations that arise from the proposed methods that are supported by our findings, and discuss the potential risks and challenges LLM-based methods for low-resource MT face moving forward.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning Evaluation, Linguistics, Language Model Translation, Artificial Intelligence, Low-Resource Language Processing" />
        </attvalues>
      </node>
      <node id="2204.05541" label="2204.05541">
        <attvalues>
          <attvalue for="0" value="Not always about you: Prioritizing community needs when developing&#10;  endangered language technology" />
          <attvalue for="1" value="  Languages are classified as low-resource when they lack the quantity of data&#10;necessary for training statistical and machine learning tools and models.&#10;Causes of resource scarcity vary but can include poor access to technology for&#10;developing these resources, a relatively small population of speakers, or a&#10;lack of urgency for collecting such resources in bilingual populations where&#10;the second language is high-resource. As a result, the languages described as&#10;low-resource in the literature are as different as Finnish on the one hand,&#10;with millions of speakers using it in every imaginable domain, and Seneca, with&#10;only a small-handful of fluent speakers using the language primarily in a&#10;restricted domain. While issues stemming from the lack of resources necessary&#10;to train models unite this disparate group of languages, many other issues cut&#10;across the divide between widely-spoken low resource languages and endangered&#10;languages. In this position paper, we discuss the unique technological,&#10;cultural, practical, and ethical challenges that researchers and indigenous&#10;speech community members face when working together to develop language&#10;technology to support endangered language documentation and revitalization. We&#10;report the perspectives of language teachers, Master Speakers and elders from&#10;indigenous communities, as well as the point of view of academics. We describe&#10;an ongoing fruitful collaboration and make recommendations for future&#10;partnerships between academic researchers and language community stakeholders.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.06684" label="2109.06684">
        <attvalues>
          <attvalue for="0" value="Non-autoregressive Transformer with Unified Bidirectional Decoder for&#10;  Automatic Speech Recognition" />
          <attvalue for="1" value="  Non-autoregressive (NAR) transformer models have been studied intensively in&#10;automatic speech recognition (ASR), and a substantial part of NAR transformer&#10;models is to use the casual mask to limit token dependencies. However, the&#10;casual mask is designed for the left-to-right decoding process of the&#10;non-parallel autoregressive (AR) transformer, which is inappropriate for the&#10;parallel NAR transformer since it ignores the right-to-left contexts. Some&#10;models are proposed to utilize right-to-left contexts with an extra decoder,&#10;but these methods increase the model complexity. To tackle the above problems,&#10;we propose a new non-autoregressive transformer with a unified bidirectional&#10;decoder (NAT-UBD), which can simultaneously utilize left-to-right and&#10;right-to-left contexts. However, direct use of bidirectional contexts will&#10;cause information leakage, which means the decoder output can be affected by&#10;the character information from the input of the same position. To avoid&#10;information leakage, we propose a novel attention mask and modify vanilla&#10;queries, keys, and values matrices for NAT-UBD. Experimental results verify&#10;that NAT-UBD can achieve character error rates (CERs) of 5.0%/5.5% on the&#10;Aishell1 dev/test sets, outperforming all previous NAR transformer models.&#10;Moreover, NAT-UBD can run 49.8x faster than the AR transformer baseline when&#10;decoding in a single step.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, transformer models \cite{Linhao2018speech,Yingzhu2020Persistent} based on encoder-decoder have shown superior performance in end-to-end automatic speech recognition (ASR) compared with Recurrent Neural Networks (RNNs) \cite{Chiu2020state,Li2018Fast} and Connectionist Temporal Classification (CTC) \cite{alex2006ctc}. Deficiently, most transformer models predict the next token conditioning on encoded states and previously generated tokens in an autoregressive (AR) manner, resulting in slow decoding speed.&#10;&#10;To accelerate the decoding speed, non-autoregressive (NAR) transformer models \cite{gu2018trans,MARJAN2020ENTROPY,JASON2018DETERMINISTIC} are first proposed in machine translation, which can predict multiple tokens simultaneously and have been widely studied&#10;in ASR recently. To our best knowledge, NAR transformer models in ASR can be roughly divided into two categories according to the decoder. The first kind of NAR transformer model \cite{zhengkun2020spike,cass2020nat} regards the decoder as an acoustic model. However, such NAR transformer models follow the conditional independence hypothesis between the output tokens and suffer inferior recognition performance. The second kind of NAR transformer model \cite{Yosuke2020maskctc,YEBAI2020laji,song2021duokuileni,TSNAT2021TWOPASS} regards the decoder as a language model, and the decoder can predict output conditioning on linguistic information. Notably, the attention mask is widely used in these NAR transformers to limit token dependencies. Especially, the casual mask proposed in the AR transformer \cite{attention} is used in the second kind of NAR transformers \cite{song2021duokuileni,TSNAT2021TWOPASS} to construct a unidirectional decoder (Fig.~\ref{fig.intro} (a)). However, it is inappropriate for NAR transformer models to use the casual mask. Firstly, the casual mask is designed for the serial decoding process of the AR transformer while the decoding process of the NAR transformer is parallel. Secondly, the casual mask only uses left-to-right (L2R) contexts, resulting in discarded right-to-left (R2L) contexts.&#10;&#10;Previously, R2L contexts (Fig.~\ref{fig.intro} (b)) have been studied in the AR transformer \cite{bidireation2020AR} and the streaming ASR \cite{bidirectional2021stream}. These models are composed of one shared encoder and two unidirectional decoders, i.e., two separate decoders with L2R and R2L contexts, respectively. Such a framework is complex and inefficient because it needs an extra unidirectional decoder and the two decoders have no information exchange.&#10;&#10;To tackle the above problems, we propose a new non-autoregre-&#10;ssive transformer with a unified bidirectional decoder (NAT-UBD), which can fully utilize both L2R and R2L contexts in a unified decoder (Fig.~\ref{fig.intro} (c)). However, direct use of bidirectional contexts will cause information leakage. Concretely, information leakage means the decoder output can be affected by the character information from the input of the same position, and the decoder can not refine the input during decoding. Since the proposed NAT-UBD is based on the speech transformer \cite{Linhao2018speech}, the residual connection and self-attention mechanism are adopted, and both of them can cause information leakage. To avoid information leakage resulting from the residual connection, we remove word embedding from vanilla queries matrix ($Q$). To avoid information leakage resulting from the self-attention mechanism, we propose a novel attention mask named self mask and make both keys matrix ($K$) and values matirx ($V$) independent of layers, similar to the Disco transformer \cite{GUJIATAO2020dashen}. This way, NAT-UBD can outperform all previous NAR transformer models on the Aishell1 corpus and achieve competitive performance compared with the AR transformer baseline on the Magicdata corpus. Moreover, NAT-UBD can run much faster than the AR transformer baseline because UBD can predict all tokens simultaneously.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Mathematics, Speech Recognition, Transformer Models, Non-Autoregressive Decoding" />
        </attvalues>
      </node>
      <node id="2004.01655" label="2004.01655">
        <attvalues>
          <attvalue for="0" value="Aligned Cross Entropy for Non-Autoregressive Machine Translation" />
          <attvalue for="1" value="  Non-autoregressive machine translation models significantly speed up decoding&#10;by allowing for parallel prediction of the entire target sequence. However,&#10;modeling word order is more challenging due to the lack of autoregressive&#10;factors in the model. This difficultly is compounded during training with cross&#10;entropy loss, which can highly penalize small shifts in word order. In this&#10;paper, we propose aligned cross entropy (AXE) as an alternative loss function&#10;for training of non-autoregressive models. AXE uses a differentiable dynamic&#10;program to assign loss based on the best possible monotonic alignment between&#10;target tokens and model predictions. AXE-based training of conditional masked&#10;language models (CMLMs) substantially improves performance on major WMT&#10;benchmarks, while setting a new state of the art for non-autoregressive models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.07903" label="2005.07903">
        <attvalues>
          <attvalue for="0" value="Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech&#10;  Recognition" />
          <attvalue for="1" value="  Non-autoregressive transformer models have achieved extremely fast inference&#10;speed and comparable performance with autoregressive sequence-to-sequence&#10;models in neural machine translation. Most of the non-autoregressive&#10;transformers decode the target sequence from a predefined-length mask sequence.&#10;If the predefined length is too long, it will cause a lot of redundant&#10;calculations. If the predefined length is shorter than the length of the target&#10;sequence, it will hurt the performance of the model. To address this problem&#10;and improve the inference speed, we propose a spike-triggered&#10;non-autoregressive transformer model for end-to-end speech recognition, which&#10;introduces a CTC module to predict the length of the target sequence and&#10;accelerate the convergence. All the experiments are conducted on a public&#10;Chinese mandarin dataset AISHELL-1. The results show that the proposed model&#10;can accurately predict the length of the target sequence and achieve a&#10;competitive performance with the advanced transformers. What's more, the model&#10;even achieves a real-time factor of 0.0056, which exceeds all mainstream speech&#10;recognition models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.15025" label="2010.15025">
        <attvalues>
          <attvalue for="0" value="Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Input" />
          <attvalue for="1" value="  Non-autoregressive (NAR) transformer models have achieved significantly&#10;inference speedup but at the cost of inferior accuracy compared to&#10;autoregressive (AR) models in automatic speech recognition (ASR). Most of the&#10;NAR transformers take a fixed-length sequence filled with MASK tokens or a&#10;redundant sequence copied from encoder states as decoder input, they cannot&#10;provide efficient target-side information thus leading to accuracy degradation.&#10;To address this problem, we propose a CTC-enhanced NAR transformer, which&#10;generates target sequence by refining predictions of the CTC module.&#10;Experimental results show that our method outperforms all previous NAR&#10;counterparts and achieves 50x faster decoding speed than a strong AR baseline&#10;with only 0.0 ~ 0.3 absolute CER degradation on Aishell-1 and Aishell-2&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.01522" label="2104.01522">
        <attvalues>
          <attvalue for="0" value="TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech&#10;  Recognition" />
          <attvalue for="1" value="  The autoregressive (AR) models, such as attention-based encoder-decoder&#10;models and RNN-Transducer, have achieved great success in speech recognition.&#10;They predict the output sequence conditioned on the previous tokens and&#10;acoustic encoded states, which is inefficient on GPUs. The non-autoregressive&#10;(NAR) models can get rid of the temporal dependency between the output tokens&#10;and predict the entire output tokens in at least one step. However, the NAR&#10;model still faces two major problems. On the one hand, there is still a great&#10;gap in performance between the NAR models and the advanced AR models. On the&#10;other hand, it's difficult for most of the NAR models to train and converge. To&#10;address these two problems, we propose a new model named the two-step&#10;non-autoregressive transformer(TSNAT), which improves the performance and&#10;accelerating the convergence of the NAR model by learning prior knowledge from&#10;a parameters-sharing AR model. Furthermore, we introduce the two-stage method&#10;into the inference process, which improves the model performance greatly. All&#10;the experiments are conducted on a public Chinese mandarin dataset ASIEHLL-1.&#10;The results show that the TSNAT can achieve a competitive performance with the&#10;AR model and outperform many complicated NAR models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.04481" label="2008.04481">
        <attvalues>
          <attvalue for="0" value="Transformer with Bidirectional Decoder for Speech Recognition" />
          <attvalue for="1" value="  Attention-based models have made tremendous progress on end-to-end automatic&#10;speech recognition(ASR) recently. However, the conventional transformer-based&#10;approaches usually generate the sequence results token by token from left to&#10;right, leaving the right-to-left contexts unexploited. In this work, we&#10;introduce a bidirectional speech transformer to utilize the different&#10;directional contexts simultaneously. Specifically, the outputs of our proposed&#10;transformer include a left-to-right target, and a right-to-left target. In&#10;inference stage, we use the introduced bidirectional beam search method, which&#10;can not only generate left-to-right candidates but also generate right-to-left&#10;candidates, and determine the best hypothesis by the score.&#10;  To demonstrate our proposed speech transformer with a bidirectional&#10;decoder(STBD), we conduct extensive experiments on the AISHELL-1 dataset. The&#10;results of experiments show that STBD achieves a 3.6\% relative CER&#10;reduction(CERR) over the unidirectional speech transformer baseline. Besides,&#10;the strongest model in this paper called STBD-Big can achieve 6.64\% CER on the&#10;test set, without language model rescoring and any extra data augmentation&#10;strategies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.06392" label="2011.06392">
        <attvalues>
          <attvalue for="0" value="Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker&#10;  Adaptation and Pronunciation Enhancement" />
          <attvalue for="1" value="  Recent neural Text-to-Speech (TTS) models have been shown to perform very&#10;well when enough data is available. However, fine-tuning them for new speakers&#10;or languages is not straightforward in a low-resource setup. In this paper, we&#10;show that by applying minor modifications to a Tacotron model, one can transfer&#10;an existing TTS model for new speakers from the same or a different language&#10;using only 20 minutes of data. For this purpose, we first introduce a base&#10;multi-lingual Tacotron with language-agnostic input, then demonstrate how&#10;transfer learning is done for different scenarios of speaker adaptation without&#10;exploiting any pre-trained speaker encoder or code-switching technique. We&#10;evaluate the transferred model in both subjective and objective ways.&#10;" />
          <attvalue for="2" value="&#10;Recent TTS models \cite{shen2018natural, ren2020fastspeech, ping2018deep} can generate intelligible and high fidelity speech when trained on tens of hours of speech corpus. However, training them with only a small amount of data results in an unstable TTS model. Different techniques have been proposed for pre-training parts of a TTS model as a pre-training step and then fine-tuning it towards new speakers \cite{chung2018semisupervised}. In such a case, however, the distinction between the pre-training and the main training phases of the model makes overall convergence more difficult.&#10;&#10;The goal of one type of transfer learning in TTS systems called speaker adaption is to adapt an existing model to a new speaker with a limited amount of data. A common way to do speaker adaption is by using a separate speaker encoder and conditioning the speech synthesis on the speaker embedding generated by the speaker encoder. This allows the model to infer the speaker identity from the speaker embedding and makes it possible to extend speech synthesis to unseen speakers during inference \cite{jia2019transfer, zhang2019learning}. One drawback of such models is their dependency on large multi-speaker datasets for training the base TTS model. Further, it requires a comprehensive speaker embedder to learn as many voice characteristics as possible during training. &#10;&#10;Another way of speaker adaptation is based on fine-tuning the weights of an already trained model \cite{moss2020boffin, chen2019sample}. Assuming the text encoder part of the TTS model has learned ``good enough'' representations, one usually freezes parts of the model that are supposed to stay fixed and only fine-tunes the remaining parts corresponding to the speaker identity to adapt to the new speaker. As the focus in speaker adaptation has been mainly on monolingual transfer, usually, the text encoder is specific to one particular language and therefore can not be used for transfer learning to a new language. In \cite{liu2019crosslingual} the authors learn a mapping between the source and target language phoneme, which helps with language transfer; however, one needs a pre-trained ASR model to learn this mapping in an unsupervised manner.&#10;&#10;Thanks to large, publicly available datasets, training a single-speaking TTS model in a single language like English can be seen as a straightforward task. However, the open question is if it would be possible to adapt a TTS model to a new speaker from a different language without needing an additional ASR model or speaker embedder. To answer this question, one needs to design the model such that it can accept inputs from the source and target languages at the same time without additional input modules or phoneme converters. &#10;&#10;In this work, inspired by the idea of &quot;learning without forgetting&quot; \cite{li2017learning}, we look at speaker adaptation from a model weight fine-tuning perspective by preserving the previous speaker(s) in several scenarios. We first introduce a framework based on Tacotron 2 \cite{shen2018natural} and enhance it with minor adaptations, which make the convergence of the model faster and more stable, especially for cross-lingual cases. The model receives IPA characters as direct input passed through a trainable lookup table for almost all IPA characters, which allows us to easily extend to a new language without using a different encoder per language or any language switching technique. &#10;Moreover, we investigate whether fine-tuning a model for a new speaker in a different language with preserving the old speaker(s) characteristics would help the old speaker(s) to speak in the new language intelligibly. Simply put, the main advantages of such a TTS system are:&#10;\begin{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Multilingual Speaker Adaptation, Transfer Learning Techniques, Signal Processing, Text-to-Speech Models" />
        </attvalues>
      </node>
      <node id="2002.01953" label="2002.01953">
        <attvalues>
          <attvalue for="0" value="BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization" />
          <attvalue for="1" value="  We present BOFFIN TTS (Bayesian Optimization For FIne-tuning Neural Text To&#10;Speech), a novel approach for few-shot speaker adaptation. Here, the task is to&#10;fine-tune a pre-trained TTS model to mimic a new speaker using a small corpus&#10;of target utterances. We demonstrate that there does not exist a&#10;one-size-fits-all adaptation strategy, with convincing synthesis requiring a&#10;corpus-specific configuration of the hyper-parameters that control fine-tuning.&#10;By using Bayesian optimization to efficiently optimize these hyper-parameter&#10;values for a target speaker, we are able to perform adaptation with an average&#10;30% improvement in speaker similarity over standard techniques. Results&#10;indicate, across multiple corpora, that BOFFIN TTS can learn to synthesize new&#10;speakers using less than ten minutes of audio, achieving the same naturalness&#10;as produced for the speakers used to train the base model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.07883" label="2401.07883">
        <attvalues>
          <attvalue for="0" value="The Chronicles of RAG: The Retriever, the Chunk and the Generator" />
          <attvalue for="1" value="  Retrieval Augmented Generation (RAG) has become one of the most popular&#10;paradigms for enabling LLMs to access external data, and also as a mechanism&#10;for grounding to mitigate against hallucinations. When implementing RAG you can&#10;face several challenges like effective integration of retrieval models,&#10;efficient representation learning, data diversity, computational efficiency&#10;optimization, evaluation, and quality of text generation. Given all these&#10;challenges, every day a new technique to improve RAG appears, making it&#10;unfeasible to experiment with all combinations for your problem. In this&#10;context, this paper presents good practices to implement, optimize, and&#10;evaluate RAG for the Brazilian Portuguese language, focusing on the&#10;establishment of a simple pipeline for inference and experiments. We explored a&#10;diverse set of methods to answer questions about the first Harry Potter book.&#10;To generate the answers we used the OpenAI's gpt-4, gpt-4-1106-preview,&#10;gpt-3.5-turbo-1106, and Google's Gemini Pro. Focusing on the quality of the&#10;retriever, our approach achieved an improvement of MRR@10 by 35.4% compared to&#10;the baseline. When optimizing the input size in the application, we observed&#10;that it is possible to further enhance it by 2.4%. Finally, we present the&#10;complete architecture of the RAG with our recommendations. As result, we moved&#10;from a baseline of 57.88% to a maximum relative score of 98.61%.&#10;" />
          <attvalue for="2" value="&#10;&#10;The rise of Large Language Models (LLMs) has changed the way we approach Artificial Intelligence (AI) applications. Their ability to answer different user queries in different domains allow these models to show a notable performance in a wide range of tasks like translation, summarizing, question answering, and many others \cite{gpt3}. However, there are a lot of open challenges when it comes to problems that require answers based on updated information, and external data, that were not available in the training data.&#10;&#10;In order to overcome this challenge, a technique called Retrieval Augmented Generation (RAG) \cite{rag_paper} was developed. This approach aims to solve the limitation of the need for external data, by fetching and incorporating this information in the prompt. With this, the model can generate more cohesive answers about subjects and data not seen during the training, decreasing the occurrence of hallucinations \cite{Hallucination}. Nevertheless, this approach adds a new layer of challenges since it requires the development of a trustworthy retriever pipeline, given that the quality of the final answer can be highly affected if the retrieved text is not relevant to the user query \cite{pereira2022visconde}.&#10;&#10;The landscape of RAG is rapidly expanding, with a constant influx of new papers introducing diverse implementations \cite{rag_survey}. Each of these variants proposes technical modifications or enhancements, such as different retrieval mechanisms, augmentation techniques, or fine-tuning methodologies. This proliferation, while a testament to the field's dynamism, presents a substantial challenge for AI practitioners. The task of methodically experimenting with, and critically evaluating, each variant's performance, scalability, and applicability becomes increasingly complex.&#10;&#10;In this paper, we present a comprehensive series of experiments focused on the application of RAG specifically tailored for Brazilian Portuguese. Our research delves into evaluating various retrieval techniques, including both sparse and dense retrievers. Additionally, we explore two chunking strategies (naive and sentence window) to optimize the integration of retrieved information into the generation process. We also investigate the impact of the positioning of documents within the prompt, analyzing how this influences the overall quality and relevance of the generated content. Finally, our experiments extend to comparing the performance of different LLMs, notably GPT-4 and Gemini, in their ability to effectively incorporate the retrieved information and produce coherent, contextually accurate responses. This paper aims to provide valuable insights and practical guidelines for implementing RAG in Brazilian Portuguese.&#10;&#10;Our main contributions are summarized as follows: 1) we propose a methodology to prepare a dataset in a format that allows quantifying the quality of the different steps in an RAG system. 2) We proposed a metric (maximum relative score) that allow us to direct quantify the existent gap between each approach and a perfect RAG system. 3) We discuss and compare different implementations, showing good practices and optimizations that can be used when developing a RAG system.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Retrieval Augmented Generation, Information Science, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2212.09656" label="2212.09656">
        <attvalues>
          <attvalue for="0" value="Visconde: Multi-document QA with GPT-3 and Neural Reranking" />
          <attvalue for="1" value="  This paper proposes a question-answering system that can answer questions&#10;whose supporting evidence is spread over multiple (potentially long) documents.&#10;The system, called Visconde, uses a three-step pipeline to perform the task:&#10;decompose, retrieve, and aggregate. The first step decomposes the question into&#10;simpler questions using a few-shot large language model (LLM). Then, a&#10;state-of-the-art search engine is used to retrieve candidate passages from a&#10;large collection for each decomposed question. In the final step, we use the&#10;LLM in a few-shot setting to aggregate the contents of the passages into the&#10;final answer. The system is evaluated on three datasets: IIRC, Qasper, and&#10;StrategyQA. Results suggest that current retrievers are the main bottleneck and&#10;that readers are already performing at the human level as long as relevant&#10;passages are provided. The system is also shown to be more effective when the&#10;model is induced to give explanations before answering a question. Code is&#10;available at \url{https://github.com/neuralmind-ai/visconde}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.04339" label="2105.04339">
        <attvalues>
          <attvalue for="0" value="DefSent: Sentence Embeddings using Definition Sentences" />
          <attvalue for="1" value="  Sentence embedding methods using natural language inference (NLI) datasets&#10;have been successfully applied to various tasks. However, these methods are&#10;only available for limited languages due to relying heavily on the large NLI&#10;datasets. In this paper, we propose DefSent, a sentence embedding method that&#10;uses definition sentences from a word dictionary, which performs comparably on&#10;unsupervised semantics textual similarity (STS) tasks and slightly better on&#10;SentEval tasks than conventional methods. Since dictionaries are available for&#10;many languages, DefSent is more broadly applicable than methods using NLI&#10;datasets without constructing additional datasets. We demonstrate that DefSent&#10;performs comparably on unsupervised semantics textual similarity (STS) tasks&#10;and slightly better on SentEval tasks to the methods using large NLI datasets.&#10;Our code is publicly available at https://github.com/hpprc/defsent .&#10;" />
          <attvalue for="2" value="&#10;&#10;Sentence embeddings represent sentences as dense vectors in a low dimensional space.&#10;Recently, sentence embedding methods using natural language inference (NLI) datasets have been successfully applied to various tasks, including semantic textual similarity (STS) tasks. &#10;However, these methods are only available for limited languages due to relying heavily on the large NLI datasets.&#10;In this paper, we propose DefSent, a sentence embedding method that uses definition sentences from a word dictionary.&#10;Since dictionaries are available for many languages, DefSent is more broadly applicable than the methods using NLI datasets without constructing additional datasets.&#10;&#10;Defsent is similar to the model proposed by \cite{hill-dictionary} in that it generates sentence embeddings so that the embeddings of a definition sentence and the word it represents are similar.&#10;However, while \cite{hill-dictionary}'s model is based on recurrent neural network language models, DefSent is based on pre-trained language models such as BERT \cite{BERT} and RoBERTa \cite{RoBERTa}, with a fine-tuning mechanism as well as Sentence-BERT \cite{SBERT}.&#10;Sentence-BERT is one of the state-of-the-art sentence embedding models, which is based on pre-trained language models that are fine-tuned on NLI datasets.&#10;Overviews of Sentence-BERT and DefSent are depicted on Figure \ref{FIG::Overview}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Text Analysis, Artificial Intelligence, Natural Language Inference, Natural Language Processing, Sentence Embedding Methods" />
        </attvalues>
      </node>
      <node id="2109.11797" label="2109.11797">
        <attvalues>
          <attvalue for="0" value="CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models" />
          <attvalue for="1" value="  Pre-Trained Vision-Language Models (VL-PTMs) have shown promising&#10;capabilities in grounding natural language in image data, facilitating a broad&#10;variety of cross-modal tasks. However, we note that there exists a significant&#10;gap between the objective forms of model pre-training and fine-tuning,&#10;resulting in a need for large amounts of labeled data to stimulate the visual&#10;grounding capability of VL-PTMs for downstream tasks. To address the challenge,&#10;we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt&#10;Tuning), a novel paradigm for tuning VL-PTMs, which reformulates visual&#10;grounding into a fill-in-the-blank problem with color-based co-referential&#10;markers in image and text, maximally mitigating the gap. In this way, CPT&#10;enables strong few-shot and even zero-shot visual grounding capabilities of&#10;VL-PTMs. Comprehensive experimental results show that the prompt-tuned VL-PTMs&#10;outperform their fine-tuned counterparts by a large margin (e.g., 17.3%&#10;absolute accuracy improvement, and 73.8% relative standard deviation reduction&#10;on average with one shot in RefCOCO evaluation). We make the data and code for&#10;this paper publicly available at https://github.com/thunlp/CPT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-trained Vision-language Models. Existing VL-PTMs can be roughly divided into three categories according to their pre-training objectives and architectures: (1)~Masked language modeling based VL-PTMs are mainly pre-trained to recover the masked tokens~\cite{lu2019vilbert,su2019vl,tan2019lxmert,li2020oscar,yu2021ernie}; (2)~Auto-regressive language modeling based VL-PTMs model image and text tokens with Transformer decoders auto-regressively~\cite{ramesh2021zero,wang2021simvlm}; (3)~Contrastive learning based VL-PTMs are pre-trained to holistically match image-text pairs~\cite{radford2021learning,li-etal-2021-unimo}. Note that our Cross-modal Prompt Tuning ({CPT}) framework is orthogonal to VL-PTM design. In this work, without loss of generality, we focus on prompting masked language modeling based VL-PTMs due to their prevalence and superior performance, while applying {CPT} to other VL-PTMs is also applicable.&#10;&#10;Prompt Tuning for NLP. Prompt tuning for pre-trained language models is a rapidly emerging field in NLP~\cite{raffel2019exploring,brown2020language,liu2021pre}. Originally designed for probing knowledge in pre-trained language models~\cite{petroni2019language}, prompt tuning has now been extended to handle a variety of NLP tasks, including language understanding~\cite{schick-schutze-2021-just,schick2021exploiting} and generation~\cite{li2021prefix}. &#10;To facilitate prompt engineering, \cite{shin2020eliciting} propose to automatically generate prompt templates via gradient-based search.&#10;Most related to our work are \cite{tsimpoukelli2021multimodal,zhou2021learning,wang2021simvlm} that present textual prompt tuning for VL-PTMs, achieving promising results on some vision-language tasks. However, similar to existing works in NLP, they focus on prompt engineering in text, keeping images untouched, and therefore can only perform holistic implicit visual grounding. In comparison, to the best of our knowledge, {CPT} is the first cross-modal prompt tuning framework tailored for both image and text, and is capable of explicitly grounding natural language to fine-grained image regions. &#10;&#10;Visual Grounding. There is a general consensus that visual grounding plays an essential role in solving vision-language tasks~\cite{karpathy2015deep,plummer2015flickr30k,goodfellow2016deep,krishna2017visual,lu2019vilbert}. \cite{mao2016generation} propose the referring expression comprehension task to explicitly evaluate the visual grounding capability. To address the task, most models learn to classify or rank image region candidates based on the expressions in a fully supervised fashion~\cite{mao2016generation,zhang2018grounding,lu2019vilbert,chen2020uniter}, requiring large amounts of costly human-annotated data. To alleviate reliance on human annotation, some works have investigated zero-/few-shot grounding of new object types~\cite{sadhu2019zero,blukis2020few}, whereas amounts of training data are still needed for existing object types. &#10;In comparison, we prompt general VL-PTMs for zero- and few-shot visual grounding in a reformulated fill-in-the-blank paradigm independent of specific object types.&#10;&#10;" />
          <attvalue for="4" value="Cross Modal Learning, Computer Science, Linguistics, Cognitive Science, Computer Vision, Artificial Intelligence, Natural Language Processing, Vision Language Models, Prompt Tuning Techniques" />
        </attvalues>
      </node>
      <node id="2310.05149" label="2310.05149">
        <attvalues>
          <attvalue for="0" value="Retrieval-Generation Synergy Augmented Large Language Models" />
          <attvalue for="1" value="  Large language models augmented with task-relevant documents have&#10;demonstrated impressive performance on knowledge-intensive tasks. However,&#10;regarding how to obtain effective documents, the existing methods are mainly&#10;divided into two categories. One is to retrieve from an external knowledge&#10;base, and the other is to utilize large language models to generate documents.&#10;We propose an iterative retrieval-generation collaborative framework. It is not&#10;only able to leverage both parametric and non-parametric knowledge, but also&#10;helps to find the correct reasoning path through retrieval-generation&#10;interactions, which is very important for tasks that require multi-step&#10;reasoning. We conduct experiments on four question answering datasets,&#10;including single-hop QA and multi-hop QA tasks. Empirical results show that our&#10;method significantly improves the reasoning ability of large language models&#10;and outperforms previous baselines.&#10;" />
          <attvalue for="2" value="&#10;Large Language models (LLMs) have demonstrated impressive performance on diverse language tasks through in-context learning \cite{brown2020language, hoffmann2022training, zeng2022glm, chowdhery2022palm, openai2023gpt4, touvron2023llama}.&#10;However, they still struggle with knowledge-intensive tasks that require access to a large amount of knowledge, such as open-domain question answering \cite{lee-etal-2019-latent} and commonsense reasoning \cite{zellers-etal-2018-swag}, since the implicit knowledge preserved in the parameters may be partial and insufficient.&#10;As shown in the top of Figure \ref{fig:intro}, one promising direction is to incorporate non-parametric knowledge to help alleviate this problem with large language models.&#10;&#10;Recent research shows that retrieving relevant documents from an external datastore \cite{ram2023context, khattab2023demonstratesearchpredict,shi2023replug} or directly generating contextual documents from LLMs \cite{yu2023generate, sun2023recitationaugmented} both can improve LLMs' performance on knowledge-intensive tasks. &#10;The former, called retrieve-then-read, requires a retriever to retrieve relevant documents.&#10;The latter, known as generate-then-read, leverages large language models to generate relevant documents before answering questions.&#10;However, as shown in Figure \ref{fig:intro}, the above two methods are isolated and lack coordination with each other.&#10;To fill this gap, in this paper, we explore an effective retrieval-generation collaboration framework to further improve the ability of large language models to solve knowledge-intensive tasks.&#10;&#10;In this work, we present ITRG, an ITerative Retrieval-Generation synergy framework to generate relevant documents that simultaneously exploits parametric and non-parametric knowledge.&#10;In each iteration, ITRG consists of two important steps: generation augmented retrieval (GAR) and retrieval augmented generation (RAG).&#10;In the GAR step, we propose a simple and effective method to expand queries by concatenating pseudo-documents generated from large language models and original questions.&#10;And expanded queries improve the accuracy of retrieving relevant documents.&#10;In the RAG step, we use large language models to comprehensively understand retrieved documents to generate new documents for answering questions.&#10;We repeat these steps until we reach the maximum allowed number of iterations.&#10;Through multiple retrieval generation collaborations, our method aids in discovering the appropriate reasoning path and providing correct answers to questions.&#10;&#10;We evaluate the efficacy of our method on 4 question answering datasets, including Natural Questions, TriviaQA, 2WikiMultiHopQA, and HotpotQA.&#10;Experimental results show that our method performs better than previous baselines on all datasets.&#10;In summary, our main contributions can be summarized as follows:&#10;(1) We propose ITRG, an iterative retrieval-generation synergy framework using both parametric and non-parametric knowledge.&#10;(2) We propose a simple and effective generation-augmented retrieval strategy and two retrieval-augmented generation strategies.&#10;(3) Empirical results show that ITRG outperforms previous retrieval-augmented methods.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Enhancement, Knowledge Retrieval Methods, Artificial Intelligence, Question Answering Systems, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.13956" label="2010.13956">
        <attvalues>
          <attvalue for="0" value="Recent Developments on ESPnet Toolkit Boosted by Conformer" />
          <attvalue for="1" value="  In this study, we present recent developments on ESPnet: End-to-End Speech&#10;Processing toolkit, which mainly involves a recently proposed architecture&#10;called Conformer, Convolution-augmented Transformer. This paper shows the&#10;results for a wide range of end-to-end speech processing applications, such as&#10;automatic speech recognition (ASR), speech translations (ST), speech separation&#10;(SS) and text-to-speech (TTS). Our experiments reveal various training tips and&#10;significant performance benefits obtained with the Conformer on different&#10;tasks. These results are competitive or even outperform the current&#10;state-of-art Transformer models. We are preparing to release all-in-one recipes&#10;using open source and publicly available corpora for all the above tasks with&#10;pre-trained models. Our aim for this work is to contribute to our research&#10;community by reducing the burden of preparing state-of-the-art research&#10;environments usually requiring high resources.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.09891" label="2409.09891">
        <attvalues>
          <attvalue for="0" value="Acquiring Pronunciation Knowledge from Transcribed Speech Audio via&#10;  Multi-task Learning" />
          <attvalue for="1" value="  Recent work has shown the feasibility and benefit of bootstrapping an&#10;integrated sequence-to-sequence (Seq2Seq) linguistic frontend from a&#10;traditional pipeline-based frontend for text-to-speech (TTS). To overcome the&#10;fixed lexical coverage of bootstrapping training data, previous work has&#10;proposed to leverage easily accessible transcribed speech audio as an&#10;additional training source for acquiring novel pronunciation knowledge for&#10;uncovered words, which relies on an auxiliary ASR model as part of a cumbersome&#10;implementation flow. In this work, we propose an alternative method to leverage&#10;transcribed speech audio as an additional training source, based on multi-task&#10;learning (MTL). Experiments show that, compared to a baseline Seq2Seq frontend,&#10;the proposed MTL-based method reduces PER from 2.5% to 1.6% for those word&#10;types covered exclusively in transcribed speech audio, achieving a similar&#10;performance to the previous method but with a much simpler implementation flow.&#10;" />
          <attvalue for="2" value="&#10;&#10;To ensure pronunciation accuracy, recent text-to-speech (TTS) takes as input pronunciation sequences generated by a separate pipeline-based linguistic frontend that includes a dictionary for word pronunciation lookup \cite{Fong2019, Shen2020, Ren2021, Tan2021}.&#10;More recent work shows the benefit of replacing the pipeline with a unified sequence-to-sequence (Seq2Seq) model that directly converts the text sequence (a string of characters) to a pronunciation sequence (a string of pronunciation tokens including phones, lexical stresses, prosodic boundaries, etc.) at the sentence level (e.g., converting PIPER'S SON to 1 p ai p - 0 @ z + 1 s uh n \_B in Unisyn \cite{Fitt2000} tokens) \cite{Conkie2020, Pan2020, Sun2023, Comini2023}. Due to the lack of ground-truth training targets, a bootstrapping approach is often applied, where a pre-existing pipeline-based frontend is utilized to generate pronunciation sequences for abundant unlabelled text to serve as training targets. The text should cover a wide range of in-dictionary word types but omit out-of-dictionary ones to ensure target accuracy\cite{Sun2023}.&#10;&#10;However, the dictionary size is fixed, so the bootstrapping training data has fixed lexical coverage, which in turn limits the performance of bootstrapped Seq2Seq frontend \cite{Sun2023}. To solve this, we can turn to some additional training source to acquire pronunciation knowledge of certain word types that are not covered in the bootstrapping training data, where the knowledge can be encoded in some form other than pronunciation sequences.&#10;E.g., a Forced-Alignment (FA) method was proposed in \cite{Sun2023} to leverage transcribed speech audio (i.e., pairs of text and speech audio) as an additional training source.&#10;Though effective, the method requires training specific automatic speech recognition (ASR) models as part of a cumbersome pre-train$arrow$ASR-train\&amp;decode$arrow$re-train flow.&#10;&#10;In this work, we propose an alternative method to leverage transcribed speech audio as an additional training source, via multi-task learning (MTL).&#10;MTL utilizes training targets of related extra tasks as an inductive bias to improve the generalization on the main task, by jointly learning the main task and extra tasks using a shared representation \cite{Caruana1998}.&#10;Recently, \cite{Sun2024} further showed in multi-accent frontend modelling, MTL particularly benefits generalizing the main task to extra-exclusive word types (i.e., word types covered in extra task training data but not covered in main task training data).&#10;Inspired by this, we propose an MTL-based method, jointly learning the main task of Seq2Seq frontend modelling (trained with bootstrapping data) and the extra task of acoustic feature regression (trained with transcribed speech audio).&#10;Our goal is to similarly greatly benefit generalizing frontend modelling to those word types covered exclusively in transcribed speech audio, which has an equivalent effect in acquiring pronunciation knowledge from transcribed speech audio for the Seq2Seq frontend.&#10;&#10;Our method has a compact pre-train$arrow$re-train flow, completely avoiding ASR training and decoding.&#10;The contributions of this paper are as follows: 1) We propose a novel MTL-based method to acquire pronunciation knowledge from transcribed speech audio. 2) We propose a novel multi-task model for our method. 3) Our experiments and analyses confirm the effectiveness of this method and model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Linguistics, Engineering, Text-to-Speech Systems, Sequence-to-Sequence Learning, Multi-Task Learning" />
        </attvalues>
      </node>
      <node id="2004.04934" label="2004.04934">
        <attvalues>
          <attvalue for="0" value="Scalable Multilingual Frontend for TTS" />
          <attvalue for="1" value="  This paper describes progress towards making a Neural Text-to-Speech (TTS)&#10;Frontend that works for many languages and can be easily extended to new&#10;languages. We take a Machine Translation (MT) inspired approach to constructing&#10;the frontend, and model both text normalization and pronunciation on a sentence&#10;level by building and using sequence-to-sequence (S2S) models. We experimented&#10;with training normalization and pronunciation as separate S2S models and with&#10;training a single S2S model combining both functions.&#10;  For our language-independent approach to pronunciation we do not use a&#10;lexicon. Instead all pronunciations, including context-based pronunciations,&#10;are captured in the S2S model. We also present a language-independent chunking&#10;and splicing technique that allows us to process arbitrary-length sentences.&#10;Models for 18 languages were trained and evaluated. Many of the accuracy&#10;measurements are above 99%. We also evaluated the models in the context of&#10;end-to-end synthesis against our current production system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.06962" label="2304.06962">
        <attvalues>
          <attvalue for="0" value="Prompt Engineering and Calibration for Zero-Shot Commonsense Reasoning" />
          <attvalue for="1" value="  Prompt engineering and calibration make large language models excel at&#10;reasoning tasks, including multiple choice commonsense reasoning. From a&#10;practical perspective, we investigate and evaluate these strategies on smaller&#10;language models. Through experiments on five commonsense reasoning benchmarks,&#10;we find that each strategy favors certain models, but their joint effects are&#10;mostly negative.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large Language models (LLMs) have shown impressive performance in many NLP applications (\cite{Ouyang2022TrainingLM}; \cite{Chung2022ScalingIL}; \cite{wei2022finetuned}),&#10;including commonsense reasoning, a key component to AGI \cite{Davis2015CommonsenseRA}. &#10;Recent studies suggest that LLMs are capable of zero-shot and few-shot learning (\cite{NEURIPS2020_1457c0d6}; \cite{webson-pavlick-2022-prompt}; \cite{Chowdhery2022PaLMSL}), and that several strategies can further improve their performance, like prompt engineering and calibration (\cite{kojima2022large}; \cite{pmlr-v139-zhao21c}; \cite{jiang-etal-2021-know}; \cite{Kadavath2022LanguageM}). &#10;Despite achieving SOTA performance on many benchmarks, most LLMs are very expensive to use and not released to the public. &#10;&#10;Consequently, we study whether prompt engineering and calibration can help smaller language models (those with no more than 3B parameters) in zero-shot multiple choice commonsense reasoning. Since these strategies are likely emergent (\cite{wei2022emergent}; \cite{Chan2022DataDP}), we make several modifications, then evaluate them on five commonsense reasoning benchmarks.&#10;We find that prompt engineering favors large Flan-T5 models, while calibration works well on GPT-2. Their joint effects are, however, negative in most cases.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Commonsense Reasoning Tasks, Computer Science, Linguistics, Cognitive Science, Prompt Engineering Strategies, Artificial Intelligence, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2409.09357" label="2409.09357">
        <attvalues>
          <attvalue for="0" value="Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for&#10;  Full-band Speech Restoration with Improved Intelligibility" />
          <attvalue for="1" value="  Speech restoration aims at restoring full-band speech with high quality and&#10;intelligibility, considering a diverse set of distortions. MaskSR is a recently&#10;proposed generative model for this task. As other models of its kind, MaskSR&#10;attains high quality but, as we show, intelligibility can be substantially&#10;improved. We do so by boosting the speech encoder component of MaskSR with&#10;predictions of semantic representations of the target speech, using a&#10;pre-trained self-supervised teacher model. Then, a masked language model is&#10;conditioned on the learned semantic features to predict acoustic tokens that&#10;encode low level spectral details of the target speech. We show that, with the&#10;same MaskSR model capacity and inference time, the proposed model, MaskSR2,&#10;significantly reduces the word error rate, a typical metric for&#10;intelligibility. MaskSR2 also achieves competitive word error rate among other&#10;models, while providing superior quality. An ablation study shows the&#10;effectiveness of various semantic representations.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech restoration (SR) aims at restoring full-band speech with high quality and intelligibility from a corrupted signal~\cite{liu2022voicefixer, serra2022universal, chen2023gesper, koizumi2023miipher}. Compared with conventional speech enhancement (SE) that typically employs discriminative modeling based on regression to remove noise~\cite{zhao2022frcrn} and, at most, reverb~\cite{defossez2020real, li2021simultaneous}, SR addresses a diverse set of tasks including those that are generative in nature, such as bandwidth extension, packet loss concealment, etc. For both SR and SE, a common finding is that the improved perceptual quality after processing may not translate to improved intelligibility, typically measured by the word error rate (WER) of automatic speech recognition (ASR) systems, since removing distortions could alter the phonetic content~\cite{wang2023speechx, koizumi2023miipher, scheibler2024universal, e3net}. Actually, the processed speech may even have a higher WER than that of the corrupted speech~\cite{wang2023speechx, koizumi2023miipher, e3net}. One solution is to condition the model on the text transcription of the corrupted speech, which, however, may not be available during both training and inference~\cite{wang2023speechx, koizumi2023miipher, le2024voicebox}. Another approach optimizes the model with an additional ASR-related loss~\cite{scheibler2024universal, e3net}, but pre-training the ASR model requires large transcribed datasets. &#10;&#10;In this paper, we propose MaskSR2 (Fig.~\ref{fig:masksr2}), which significantly reduces the WER without relying on transcribed data. MaskSR2 is based on our previous work MaskSR, a full-band (44.1\,kHz) SR system that holistically performs denoising, dereverberation, declipping, and bandwidth extension under a masked token modeling paradigm~\cite{li2024masksr}. MaskSR2 improves upon MaskSR by introducing semantic knowledge distillation (KD) in the speech encoder component. During training, given the STFT of a corrupted speech signal, the speech encoder predicts semantic representations of the target speech, extracted using a pre-trained HuBERT model~\cite{hsu2021hubert}. This teacher model encodes semantic (phonetic) patterns learned through self-supervised learning (SSL), which removes the need for transcribed audio. The KD is imposed by a loss function upon the speech encoder, jointly optimized with the rest of the system. Meanwhile, the generative model is conditioned on the learned semantic features from the hidden layer of the encoder to predict randomly-masked acoustic tokens of the target speech. During inference, the HuBERT teacher is discarded, thus reducing the compute cost of MaskSR2 to the one of MaskSR (same model capacity and inference time). Iterative sampling is performed on the output distribution to generate the target acoustic tokens, which are then converted to a waveform by a pre-trained audio (de)tokenizer.&#10;&#10;We get inspiration from previous text-guided speech/audio synthesis research, which shows the importance of semantic modeling~\cite{agostinelli2023musiclm, kharitonov2023speak, liu2024audioldm, dong2023clipsonic}. These systems usually consist of two stages, text-to-semantic and semantic-to-acoustic synthesis, each requiring separate training and iterative inference. For SR, since the corrupted speech provides a stronger condition than a text prompt, we are able to jointly train the speech encoder and the generative model, and then run iterative sampling only on the latter. Another SR work~\cite{serra2022universal} also fuses a speech encoder with a generative model, but the encoder is optimized only on spectral targets, such as the STFT and features derived from it. We show that the semantic KD is an effective choice to improve intelligibility. SELM~\cite{wang2024selm} trains a language model to translate discrete noisy SSL tokens to clean ones, thus performing denoising with lower WER. But this framework requires storing and running a (typically large) SSL model during inference, which is not needed by MaskSR2. Also, the generated speech quality is sensitive to the codebook size of the discrete SSL tokens~\cite{wang2024selm}. We avoid this issue by exploring continuous HuBERT features as the teacher for the speech encoder. Overall, MaskSR2 reduces the WER between 19\,\% and 38\,\% relative to MaskSR, and achieves a competitive WER when compared to strong regression models trained to optimize waveform or spectrum alignment (thus tending to produce lower WER than generative models). MaskSR2 also outperforms various models in terms of quality. Samples are available on our demo page.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Acoustics, Speech Restoration, Deep Learning Models, Computer Science, Speech Processing, Machine Learning, Linguistics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2006.12847" label="2006.12847">
        <attvalues>
          <attvalue for="0" value="Real Time Speech Enhancement in the Waveform Domain" />
          <attvalue for="1" value="  We present a causal speech enhancement model working on the raw waveform that&#10;runs in real-time on a laptop CPU. The proposed model is based on an&#10;encoder-decoder architecture with skip-connections. It is optimized on both&#10;time and frequency domains, using multiple loss functions. Empirical evidence&#10;shows that it is capable of removing various kinds of background noise&#10;including stationary and non-stationary noises, as well as room reverb.&#10;Additionally, we suggest a set of data augmentation techniques applied directly&#10;on the raw waveform which further improve model performance and its&#10;generalization abilities. We perform evaluations on several standard&#10;benchmarks, both using objective metrics and human judgements. The proposed&#10;model matches state-of-the-art performance of both causal and non causal&#10;methods while working directly on the raw waveform.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.06873" label="2308.06873">
        <attvalues>
          <attvalue for="0" value="SpeechX: Neural Codec Language Model as a Versatile Speech Transformer" />
          <attvalue for="1" value="  Recent advancements in generative speech models based on audio-text prompts&#10;have enabled remarkable innovations like high-quality zero-shot text-to-speech.&#10;However, existing models still face limitations in handling diverse audio-text&#10;speech generation tasks involving transforming input speech and processing&#10;audio captured in adverse acoustic conditions. This paper introduces SpeechX, a&#10;versatile speech generation model capable of zero-shot TTS and various speech&#10;transformation tasks, dealing with both clean and noisy signals. SpeechX&#10;combines neural codec language modeling with multi-task learning using&#10;task-dependent prompting, enabling unified and extensible modeling and&#10;providing a consistent way for leveraging textual input in speech enhancement&#10;and transformation tasks. Experimental results show SpeechX's efficacy in&#10;various tasks, including zero-shot TTS, noise suppression, target speaker&#10;extraction, speech removal, and speech editing with or without background&#10;noise, achieving comparable or superior performance to specialized models&#10;across tasks. See https://aka.ms/speechx for demo samples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.02092" label="2406.02092">
        <attvalues>
          <attvalue for="0" value="MaskSR: Masked Language Model for Full-band Speech Restoration" />
          <attvalue for="1" value="  Speech restoration aims at restoring high quality speech in the presence of a&#10;diverse set of distortions. Although several deep learning paradigms have been&#10;studied for this task, the power of the recently emerging language models has&#10;not been fully explored. In this paper, we propose MaskSR, a masked language&#10;model capable of restoring full-band 44.1 kHz speech jointly considering noise,&#10;reverb, clipping, and low bandwidth. MaskSR works with discrete acoustic tokens&#10;extracted using a pre-trained neural codec. During training, MaskSR is&#10;optimized to predict randomly masked tokens extracted from the high quality&#10;target speech, conditioned on the corrupted speech with various distortions.&#10;During inference, MaskSR reconstructs the target speech tokens with efficient&#10;iterative sampling. Extensive experiments show that MaskSR obtains competitive&#10;results on both the full-band speech restoration task and also on sub-tasks&#10;compared with a wide range of models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.03065" label="2206.03065">
        <attvalues>
          <attvalue for="0" value="Universal Speech Enhancement with Score-based Diffusion" />
          <attvalue for="1" value="  Removing background noise from speech audio has been the subject of&#10;considerable effort, especially in recent years due to the rise of virtual&#10;communication and amateur recordings. Yet background noise is not the only&#10;unpleasant disturbance that can prevent intelligibility: reverb, clipping,&#10;codec artifacts, problematic equalization, limited bandwidth, or inconsistent&#10;loudness are equally disturbing and ubiquitous. In this work, we propose to&#10;consider the task of speech enhancement as a holistic endeavor, and present a&#10;universal speech enhancement system that tackles 55 different distortions at&#10;the same time. Our approach consists of a generative model that employs&#10;score-based diffusion, together with a multi-resolution conditioning network&#10;that performs enhancement with mixture density networks. We show that this&#10;approach significantly outperforms the state of the art in a subjective test&#10;performed by expert listeners. We also show that it achieves competitive&#10;objective scores with just 4-8 diffusion steps, despite not considering any&#10;particular strategy for fast sampling. We hope that both our methodology and&#10;technical contributions encourage researchers and practitioners to adopt a&#10;universal approach to speech enhancement, possibly framing it as a generative&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.09747" label="2312.09747">
        <attvalues>
          <attvalue for="0" value="SELM: Speech Enhancement Using Discrete Tokens and Language Models" />
          <attvalue for="1" value="  Language models (LMs) have shown superior performances in various speech&#10;generation tasks recently, demonstrating their powerful ability for semantic&#10;context modeling. Given the intrinsic similarity between speech generation and&#10;speech enhancement, harnessing semantic information holds potential advantages&#10;for speech enhancement tasks. In light of this, we propose SELM, a novel&#10;paradigm for speech enhancement, which integrates discrete tokens and leverages&#10;language models. SELM comprises three stages: encoding, modeling, and decoding.&#10;We transform continuous waveform signals into discrete tokens using pre-trained&#10;self-supervised learning (SSL) models and a k-means tokenizer. Language models&#10;then capture comprehensive contextual information within these tokens. Finally,&#10;a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental&#10;results demonstrate that SELM achieves comparable performance in objective&#10;metrics alongside superior results in subjective perception. Our demos are&#10;available https://honee-w.github.io/SELM/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.16662" label="2408.16662">
        <attvalues>
          <attvalue for="0" value="Space3D-Bench: Spatial 3D Question Answering Benchmark" />
          <attvalue for="1" value="  Answering questions about the spatial properties of the environment poses&#10;challenges for existing language and vision foundation models due to a lack of&#10;understanding of the 3D world notably in terms of relationships between&#10;objects. To push the field forward, multiple 3D Q&amp;A datasets were proposed&#10;which, overall, provide a variety of questions, but they individually focus on&#10;particular aspects of 3D reasoning or are limited in terms of data modalities.&#10;To address this, we present Space3D-Bench - a collection of 1000 general&#10;spatial questions and answers related to scenes of the Replica dataset which&#10;offers a variety of data modalities: point clouds, posed RGB-D images,&#10;navigation meshes and 3D object detections. To ensure that the questions cover&#10;a wide range of 3D objectives, we propose an indoor spatial questions taxonomy&#10;inspired by geographic information systems and use it to balance the dataset&#10;accordingly. Moreover, we provide an assessment system that grades natural&#10;language responses based on predefined ground-truth answers by leveraging a&#10;Vision Language Model's comprehension of both text and images to compare the&#10;responses with ground-truth textual information or relevant visual data.&#10;Finally, we introduce a baseline called RAG3D-Chat integrating the world&#10;understanding of foundation models with rich context retrieval, achieving an&#10;accuracy of 67% on the proposed dataset.&#10;" />
          <attvalue for="2" value="&#10;Foundation models are becoming a standard tool in the field of artificial intelligence by providing adaptability and a high level of performance in a variety of down-stream tasks.&#10;Notably, the field of 3D scene understanding has experienced rapid advancements fueled by these large pre-trained models.&#10;More and more applications are emerging in terms of 3D reasoning, spatial awareness, and environment interaction~\cite{3dllm2024ma}. These spatial competences have provided significant improvements for mixed reality~\cite{fang2024enablingwaypointgenerationcollaborative, delatorre2024llmrrealtimepromptinginteractive, Bozkir_2024}, robotics~\cite{wang2024largelanguagemodelsrobotics, brohan2023rt2visionlanguageactionmodelstransfer, chang2023contextaware}, autonomous vehicles~\cite{mao2023gptdriverlearningdrivegpt, mao2024languageagentautonomousdriving, cho2024languageimagemodels3dunderstanding}, inclusive technologies~\cite{chatgpt4good}, or navigation~\cite{zhou2023navgpt, navigation2018anderson, Gu_2022}.&#10;&#10;Numerous approaches have been proposed by the research community to address these 3D tasks. As these foundation models proved to be capable of understanding other modalities than text, one group of solutions involves proposing multi-modal models, integrating images~\cite{gu2023conceptgraphs}, videos~\cite{zhang2020doesexistspatiotemporalvideo}, or 3D data such as point clouds or meshes~\cite{xu2023pointllm}. Another strong trend in the field is combining the strengths of existing models with such tools as context retrieval~\cite{ning2024llmfindautonomousgisagent} or zero-shot learning~\cite{zhang2024agent3dzeroagentzeroshot3d, Yuan_2024_CVPR}. The progress in understanding the spatial properties is evident, however, robustness and alignment still remain a challenge.&#10;&#10;To measure the performance of these spatially-aware systems, multiple 3D Question and Answer (Q\&amp;A) datasets have been proposed~\cite{azuma_2022_CVPR, Zhu_2023_ICCV, ma2022sqa3d, chen2020scanrefer, yan2023comprehensive, li2023m3dbench}. They vary in their assessment objective, size, scene types, and provided data. Although, from the holistic perspective, the datasets provide a large variety, individually they are either limited in terms of accompanying data modalities, focus on a narrow aspect of 3D reasoning, or do not have a balanced question distribution in terms of objectives.&#10;&#10;To address these limitations, we built a dataset composed of $1000$ questions with ground truth answers.&#10; To assure that the questions cover a wide range of 3D objectives, we adapted an existing taxonomy of spatial question~\cite{understanding2024schmidts} used in Geographic Information Systems and adapted it to the indoor scenes scenario.&#10; We balanced the number of questions with respect to the presented categories. The questions are associated with thirteen selected scenes from the Replica dataset~\cite{replica19arxiv}, that gives access to a variety of data, such as 3D object detections, navigation meshes, and point clouds. Additionally, as Replica is integrated into Habitat Sim's environment~\cite{habitat19iccv}, data such as videos, and RGB-D or semantically-segmented images with camera poses can be seamlessly extracted.&#10;&#10;To complement the dataset's functionality, we developed a Vision Language Model (VLM) based automatic assessment system that evaluates the responses from a question answering system against the dataset's ground truth. To establish the assessment's correctness and reliability, we conduct an extensive user study of $60$ participants, on a subset of $40$ questions that are randomly sampled.&#10; As the result of the survey shows, our evaluation system agrees on 97.5\% of the cases with users, which confirms the reliability of our evaluation protocol. &#10;&#10;To demonstrate a baseline performance on the created dataset, we propose RAG3D-Chat -- Retrieval Augmented Generation (RAG) for 3D Chat -- a system that utilizes RAG~\cite{rag2020} and VLMs to identify the relevant scene context from images, texts, and an SQL database, and also has the capability to answer questions regarding navigable distances. We employed a planner based on a Large Language Model (LLM) to chain available functionalities of the system to answer complex questions. This system scored 67\% of accuracy on the dataset, which proves that there is room for improvement in the robustness of 3D spatial Q\&amp;A. Selected questions from Space3D-Bench dataset, answered by RAG3D-Chat, are presented in \ref{fig:teaser}.&#10;&#10;In summary, this paper introduces the following contributions:&#10;&#10;\begin{itemize}&#10;&#10;\item We propose a dataset of $1000$ diverse spatial questions and answers, based on the scenes of the Replica dataset, which offer a variety of data modalities. We present an application of a geographic spatial questions taxonomy to indoor scenes, and balance our questions accordingly.&#10; \item We provide a VLM-based assessment system that evaluates natural language responses given ground truth answers. To confirm the reliability of our proposed system, we conducted an extensive user study.&#10; \item We leverage the strengths of foundation models, and combine them with Retrieval-Augmented Generation, to present a baseline achieving $67\%$ of accuracy on the proposed dataset.&#10; \item We release the dataset with the assessment system, to encourage the research community to address the challenges of 3D question answering by developing and evaluating their spatial Q\&amp;A systems.&#10; \end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Computer Vision, Artificial Intelligence, Question Answering Systems, Natural Language Processing, Vision Language Models" />
        </attvalues>
      </node>
      <node id="2405.10255" label="2405.10255">
        <attvalues>
          <attvalue for="0" value="When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks&#10;  via Multi-modal Large Language Models" />
          <attvalue for="1" value="  As large language models (LLMs) evolve, their integration with 3D spatial&#10;data (3D-LLMs) has seen rapid progress, offering unprecedented capabilities for&#10;understanding and interacting with physical spaces. This survey provides a&#10;comprehensive overview of the methodologies enabling LLMs to process,&#10;understand, and generate 3D data. Highlighting the unique advantages of LLMs,&#10;such as in-context learning, step-by-step reasoning, open-vocabulary&#10;capabilities, and extensive world knowledge, we underscore their potential to&#10;significantly advance spatial comprehension and interaction within embodied&#10;Artificial Intelligence (AI) systems. Our investigation spans various 3D data&#10;representations, from point clouds to Neural Radiance Fields (NeRFs). It&#10;examines their integration with LLMs for tasks such as 3D scene understanding,&#10;captioning, question-answering, and dialogue, as well as LLM-based agents for&#10;spatial reasoning, planning, and navigation. The paper also includes a brief&#10;review of other methods that integrate 3D and language. The meta-analysis&#10;presented in this paper reveals significant progress yet underscores the&#10;necessity for novel approaches to harness the full potential of 3D-LLMs. Hence,&#10;with this paper, we aim to chart a course for future research that explores and&#10;expands the capabilities of 3D-LLMs in understanding and interacting with the&#10;complex 3D world. To support this survey, we have established a project page&#10;where papers related to our topic are organized and listed:&#10;https://github.com/ActiveVisionLab/Awesome-LLM-3D.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.09308" label="2403.09308">
        <attvalues>
          <attvalue for="0" value="Enabling Waypoint Generation for Collaborative Robots using LLMs and&#10;  Mixed Reality" />
          <attvalue for="1" value="  Programming a robotic is a complex task, as it demands the user to have a&#10;good command of specific programming languages and awareness of the robot's&#10;physical constraints. We propose a framework that simplifies robot deployment&#10;by allowing direct communication using natural language. It uses large language&#10;models (LLM) for prompt processing, workspace understanding, and waypoint&#10;generation. It also employs Augmented Reality (AR) to provide visual feedback&#10;of the planned outcome. We showcase the effectiveness of our framework with a&#10;simple pick-and-place task, which we implement on a real robot. Moreover, we&#10;present an early concept of expressive robot behavior and skill generation that&#10;can be used to communicate with the user and learn new skills (e.g., object&#10;grasping).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.16650" label="2309.16650">
        <attvalues>
          <attvalue for="0" value="ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and&#10;  Planning" />
          <attvalue for="1" value="  For robots to perform a wide variety of tasks, they require a 3D&#10;representation of the world that is semantically rich, yet compact and&#10;efficient for task-driven perception and planning. Recent approaches have&#10;attempted to leverage features from large vision-language models to encode&#10;semantics in 3D representations. However, these approaches tend to produce maps&#10;with per-point feature vectors, which do not scale well in larger environments,&#10;nor do they contain semantic spatial relationships between entities in the&#10;environment, which are useful for downstream planning. In this work, we propose&#10;ConceptGraphs, an open-vocabulary graph-structured representation for 3D&#10;scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing&#10;their output to 3D by multi-view association. The resulting representations&#10;generalize to novel semantic classes, without the need to collect large 3D&#10;datasets or finetune models. We demonstrate the utility of this representation&#10;through a number of downstream planning tasks that are specified through&#10;abstract (language) prompts and require complex reasoning over spatial and&#10;semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer&#10;video: https://youtu.be/mRhNkQwRYnc )&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.06891" label="2001.06891">
        <attvalues>
          <attvalue for="0" value="Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form&#10;  Sentences" />
          <attvalue for="1" value="  In this paper, we consider a novel task, Spatio-Temporal Video Grounding for&#10;Multi-Form Sentences (STVG). Given an untrimmed video and a&#10;declarative/interrogative sentence depicting an object, STVG aims to localize&#10;the spatio-temporal tube of the queried object. STVG has two challenging&#10;settings: (1) We need to localize spatio-temporal object tubes from untrimmed&#10;videos, where the object may only exist in a very small segment of the video;&#10;(2) We deal with multi-form sentences, including the declarative sentences with&#10;explicit objects and interrogative sentences with unknown objects. Existing&#10;methods cannot tackle the STVG task due to the ineffective tube pre-generation&#10;and the lack of object relationship modeling. Thus, we then propose a novel&#10;Spatio-Temporal Graph Reasoning Network (STGRN) for this task. First, we build&#10;a spatio-temporal region graph to capture the region relationships with&#10;temporal object dynamics, which involves the implicit and explicit spatial&#10;subgraphs in each frame and the temporal dynamic subgraph across frames. We&#10;then incorporate textual clues into the graph and develop the multi-step&#10;cross-modal graph reasoning. Next, we introduce a spatio-temporal localizer&#10;with a dynamic selection method to directly retrieve the spatio-temporal tubes&#10;without tube pre-generation. Moreover, we contribute a large-scale video&#10;grounding dataset VidSTG based on video relation dataset VidOR. The extensive&#10;experiments demonstrate the effectiveness of our method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.16911" label="2308.16911">
        <attvalues>
          <attvalue for="0" value="PointLLM: Empowering Large Language Models to Understand Point Clouds" />
          <attvalue for="1" value="  The unprecedented advancements in Large Language Models (LLMs) have shown a&#10;profound impact on natural language processing but are yet to fully embrace the&#10;realm of 3D understanding. This paper introduces PointLLM, a preliminary effort&#10;to fill this gap, enabling LLMs to understand point clouds and offering a new&#10;avenue beyond 2D visual data. PointLLM understands colored object point clouds&#10;with human instructions and generates contextually appropriate responses,&#10;illustrating its grasp of point clouds and common sense. Specifically, it&#10;leverages a point cloud encoder with a powerful LLM to effectively fuse&#10;geometric, appearance, and linguistic information. We collect a novel dataset&#10;comprising 660K simple and 70K complex point-text instruction pairs to enable a&#10;two-stage training strategy: aligning latent spaces and subsequently&#10;instruction-tuning the unified model. To rigorously evaluate the perceptual and&#10;generalization capabilities of PointLLM, we establish two benchmarks:&#10;Generative 3D Object Classification and 3D Object Captioning, assessed through&#10;three different methods, including human evaluation, GPT-4/ChatGPT evaluation,&#10;and traditional metrics. Experimental results reveal PointLLM's superior&#10;performance over existing 2D and 3D baselines, with a notable achievement in&#10;human-evaluated object captioning tasks where it surpasses human annotators in&#10;over 50% of the samples. Codes, datasets, and benchmarks are available at&#10;https://github.com/OpenRobotLab/PointLLM .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.11835" label="2403.11835">
        <attvalues>
          <attvalue for="0" value="Agent3D-Zero: An Agent for Zero-shot 3D Understanding" />
          <attvalue for="1" value="  The ability to understand and reason the 3D real world is a crucial milestone&#10;towards artificial general intelligence. The current common practice is to&#10;finetune Large Language Models (LLMs) with 3D data and texts to enable 3D&#10;understanding. Despite their effectiveness, these approaches are inherently&#10;limited by the scale and diversity of the available 3D data. Alternatively, in&#10;this work, we introduce Agent3D-Zero, an innovative 3D-aware agent framework&#10;addressing the 3D scene understanding in a zero-shot manner. The essence of our&#10;approach centers on reconceptualizing the challenge of 3D scene perception as a&#10;process of understanding and synthesizing insights from multiple images,&#10;inspired by how our human beings attempt to understand 3D scenes. By&#10;consolidating this idea, we propose a novel way to make use of a Large Visual&#10;Language Model (VLM) via actively selecting and analyzing a series of&#10;viewpoints for 3D understanding. Specifically, given an input 3D scene,&#10;Agent3D-Zero first processes a bird's-eye view image with custom-designed&#10;visual prompts, then iteratively chooses the next viewpoints to observe and&#10;summarize the underlying knowledge. A distinctive advantage of Agent3D-Zero is&#10;the introduction of novel visual prompts, which significantly unleash the VLMs'&#10;ability to identify the most informative viewpoints and thus facilitate&#10;observing 3D scenes. Extensive experiments demonstrate the effectiveness of the&#10;proposed framework in understanding diverse and previously unseen 3D&#10;environments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.10482" label="2112.10482">
        <attvalues>
          <attvalue for="0" value="ScanQA: 3D Question Answering for Spatial Scene Understanding" />
          <attvalue for="1" value="  We propose a new 3D spatial understanding task of 3D Question Answering&#10;(3D-QA). In the 3D-QA task, models receive visual information from the entire&#10;3D scene of the rich RGB-D indoor scan and answer the given textual questions&#10;about the 3D scene. Unlike the 2D-question answering of VQA, the conventional&#10;2D-QA models suffer from problems with spatial understanding of object&#10;alignment and directions and fail the object identification from the textual&#10;questions in 3D-QA. We propose a baseline model for 3D-QA, named ScanQA model,&#10;where the model learns a fused descriptor from 3D object proposals and encoded&#10;sentence embeddings. This learned descriptor correlates the language&#10;expressions with the underlying geometric features of the 3D scan and&#10;facilitates the regression of 3D bounding boxes to determine described objects&#10;in textual questions and outputs correct answers. We collected human-edited&#10;question-answer pairs with free-form answers that are grounded to 3D objects in&#10;each 3D scene. Our new ScanQA dataset contains over 40K question-answer pairs&#10;from the 800 indoor scenes drawn from the ScanNet dataset. To the best of our&#10;knowledge, the proposed 3D-QA task is the first large-scale effort to perform&#10;object-grounded question-answering in 3D environments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.05797" label="1906.05797">
        <attvalues>
          <attvalue for="0" value="The Replica Dataset: A Digital Replica of Indoor Spaces" />
          <attvalue for="1" value="  We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene&#10;reconstructions at room and building scale. Each scene consists of a dense&#10;mesh, high-resolution high-dynamic-range (HDR) textures, per-primitive semantic&#10;class and instance information, and planar mirror and glass reflectors. The&#10;goal of Replica is to enable machine learning (ML) research that relies on&#10;visually, geometrically, and semantically realistic generative models of the&#10;world - for instance, egocentric computer vision, semantic segmentation in 2D&#10;and 3D, geometric inference, and the development of embodied agents (virtual&#10;robots) performing navigation, instruction following, and question answering.&#10;Due to the high level of realism of the renderings from Replica, there is hope&#10;that ML systems trained on Replica may transfer directly to real world image&#10;and video data. Together with the data, we are releasing a minimal C++ SDK as a&#10;starting point for working with the Replica dataset. In addition, Replica is&#10;`Habitat-compatible', i.e. can be natively used with AI Habitat for training&#10;and testing embodied agents.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.01201" label="1904.01201">
        <attvalues>
          <attvalue for="0" value="Habitat: A Platform for Embodied AI Research" />
          <attvalue for="1" value="  We present Habitat, a platform for research in embodied artificial&#10;intelligence (AI). Habitat enables training embodied agents (virtual robots) in&#10;highly efficient photorealistic 3D simulation. Specifically, Habitat consists&#10;of: (i) Habitat-Sim: a flexible, high-performance 3D simulator with&#10;configurable agents, sensors, and generic 3D dataset handling. Habitat-Sim is&#10;fast -- when rendering a scene from Matterport3D, it achieves several thousand&#10;frames per second (fps) running single-threaded, and can reach over 10,000 fps&#10;multi-process on a single GPU. (ii) Habitat-API: a modular high-level library&#10;for end-to-end development of embodied AI algorithms -- defining tasks (e.g.,&#10;navigation, instruction following, question answering), configuring, training,&#10;and benchmarking embodied agents.&#10;  These large-scale engineering contributions enable us to answer scientific&#10;questions requiring experiments that were till now impracticable or 'merely'&#10;impractical. Specifically, in the context of point-goal navigation: (1) we&#10;revisit the comparison between learning and SLAM approaches from two recent&#10;works and find evidence for the opposite conclusion -- that learning&#10;outperforms SLAM if scaled to an order of magnitude more experience than&#10;previous investigations, and (2) we conduct the first cross-dataset&#10;generalization experiments {train, test} x {Matterport3D, Gibson} for multiple&#10;sensors {blind, RGB, RGBD, D} and find that only agents with depth (D) sensors&#10;generalize across datasets. We hope that our open-source platform and these&#10;findings will advance research in embodied AI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.21028" label="2405.21028">
        <attvalues>
          <attvalue for="0" value="LACIE: Listener-Aware Finetuning for Confidence Calibration in Large&#10;  Language Models" />
          <attvalue for="1" value="  When answering questions, LLMs can convey not only an answer, but a level of&#10;confidence about the answer being correct. This includes explicit confidence&#10;markers (e.g. giving a numeric score) as well as implicit markers, like an&#10;authoritative tone or elaborating with additional knowledge. For LLMs to be&#10;trustworthy knowledge sources, the confidence they convey should match their&#10;actual expertise; however, most current models tend towards overconfidence. To&#10;calibrate both implicit and explicit confidence markers, we introduce a&#10;pragmatic, listener-aware finetuning method (LACIE) that models the listener,&#10;considering not only whether an answer is right, but whether it will be&#10;accepted by a listener. We cast calibration as preference optimization,&#10;creating data via a two-agent game, where a speaker model's outputs are judged&#10;by a simulated listener. We then finetune three LLMs (Mistral-7B, Llama3-8B,&#10;Llama3-70B) with LACIE, and show that the resulting models are better&#10;calibrated w.r.t. a simulated listener. Crucially, these trends transfer to&#10;human listeners, helping them correctly predict model correctness: we conduct a&#10;human evaluation where annotators accept or reject an LLM's answers, finding&#10;that training with LACIE results in 47% fewer incorrect answers being accepted&#10;while maintaining the same level of acceptance for correct answers.&#10;Furthermore, LACIE generalizes to another dataset, resulting in a large&#10;increase in truthfulness on TruthfulQA when trained on TriviaQA. Our analysis&#10;indicates that LACIE leads to a better confidence separation between correct&#10;and incorrect examples. Qualitatively, we find that a LACIE-trained model&#10;hedges more and implicitly signals certainty when it is correct by using an&#10;authoritative tone or including details. Finally, LACIE finetuning leads to an&#10;emergent increase in model abstention (e.g. saying &quot;I don't know&quot;) for answers&#10;that are likely wrong.&#10;" />
          <attvalue for="2" value="&#10;&#10;In interacting linguistically with each other, people tend to follow conventions -- or maxims -- that allow for successful communication.&#10;For example, good conversational partners try to make their utterances truthful, relevant, clear, and concise \cite{grice.p.1975logic}.&#10;When people violate these conventions, they can mislead listeners, which may ultimately lead to them being seen as incompetent, untrustworthy, or as poor conversational partners. &#10;While large language models (LLMs) generally follow many of these conventions, they often fail to respect \cite{grice.p.1975logic}'s maxim of truthfulness, generating outputs that are not truthful \cite{rawte2023survey}.&#10;More troublingly, untruthful outputs generated by LLMs are often expressed confidently and authoritatively, and thus appear convincing to users, meaning that humans may easily be misled by LLMs. &#10;&#10;LLMs' confidence can be expressed in at least two ways, shown in \ref{fig:fig1}A. &#10;Firstly, LLMs can explicitly express confidence in their output using numeric scores (e.g. ``I am 100\% confident'') or epistemic markers \cite{zhou.k.2023} (e.g. ``I'm very sure that...''). &#10;Secondly, LLMs can implicitly express confidence by details or through their tone; often, the details included are spurious or non-factual, i.e. hallucinated.&#10;For example, in our analysis in \ref{sec:qualitative}, we found that LLMs often add hallucinated backstories to their answers, e.g. ``I remember seeing this movie on the big screen in the theatre...'', or add an additional explanation that may sound convincing but is untrue. &#10;These details convey a sense of expertise that can lead to the answer being perceived as more likely to be correct. &#10;&#10;Overconfidence is particularly troubling given that people are increasingly interacting with LLMs as sources of information \cite{gude2023factors}; in other words, people rely on LLMs to answer questions they themselves do not know the answer to.&#10;Futhermore, because the interactions people have with LLMs happen via language, users often interpret LLM outputs as they would interpret language from a human, i.e. assuming that the outputs follow Gricean maxims.&#10;This in turn makes LLMs unreliable partners; many readers may have had the experience of working together with a partner or teammate who consistently overstates their confidence. &#10;While this teammate may initially have their answers accepted, over time they lose trust.&#10;Indeed, \cite{zhou.k.2024} highlight this type of erosion for overconfident models, finding that overconfidence irreparably damaged a user's trust in an AI system. &#10;&#10;Focusing on answering information-seeking questions, we hypothesize that model overconfidence of both kinds (implicit and explicit) can be mitigated by optimizing for pragmatics, i.e. for how the utterance will be interpreted by a listener.&#10;Specifically, we hypothesize that part of current models' overconfidence lies in (1) a lack of knowledge about whether its answers are correct or not, and (2) a lack of pragmatic grounding, i.e. models do not generate utterances according to how they will be perceived by a listener. &#10;Firstly, base models (not finetuned with instructions or human feedback) are not grounded in the consequences of their answers: they do not receive direct feedback during training about whether the answer is correct or not, and thus have little reason to hedge in their confidence. &#10;Secondly, models are not pre-trained pragmatically; they generate responses without real-time access to feedback on how listeners might interpret their answers. &#10;While models trained with human feedback may in principle have this capacity, past work has shown that they in fact have worse calibration than base models and has attributed this to current reward data penalizing hedging and markers of uncertainty \cite{zhou.k.2024}. &#10;&#10;To address overconfidence by tackling these two types of grounding, we introduce Listener-Aware Calibration for Implicit and Explicit confidence, or \method.&#10;\method finetunes models not only using feedback on whether their answer was correct but also whether their answer was interpreted as correct by a listener.&#10;In other words, whereas past work \cite[][i.a.]{ kuhn.l.2022semantic, tian.k.2023just, ulmer.d.2024calibrating} has sought to produce calibrated distributions in speaker's output distribution -- i.e. answer probabilities that are equal to the model's chance of being correct -- we seek to induce &#10;a calibrated distribution in the listener via the speaker's utterance -- i.e. train the model to output generations that allow a listener to recover a well-calibrated score of how likely the answer is to be correct.&#10;This multi-agent optimization not only trains models to reliably use both explicit and implicit confidence markers, but also allows us to flexibly address calibration for long-form model answers and not only closed-set answer probabilities.&#10;&#10;To pragmatically calibrate LLMs, we adopt the Direct Preference Optimization (DPO) framework \cite{rafailov.r.2024direct}, constructing a dataset of preferrred and dispreferred examples from a seed dataset of QA pairs. &#10;As shown in \ref{fig:fig1}B, we first generate long-form responses from a standard LLM (the speaker agent). &#10;Many of these responses contain both implicit and explicit certainty expressions, often applied inappropriately to incorrect answers. &#10;We then use another LLM to model a listener agent who decides whether to accept or reject the answer; in information-seeking questions (where the answer is not known), &#10;the listener model should base its decision largely on how confident the speaker sounds.&#10;Note that our multi-agent framing allows us to explore a much wider range of confidence cues; while past work \cite{mielke.s.2022, lin.s.2022, zhou.k.2024} has focused on epistemic markers (explicit expressions of confidence) we are also able to define and examine more subtle and implicit confidence cues, like a tone, level of detail, and use of backstories.&#10;Using the listener model and the ground-truth answer, we define a preference function (given in \ref{sec:methodology}) that rewards cases where the model accurately expresses confidence -- marking these as preferred examples -- and penalizes the model when it inaccurately expresses confidence -- marking them as dispreferred. &#10;This makes the training listener-aware, connecting to past work in jointly modeling speakers and listeners \cite{frank.m.2012predicting, fried.d.2018speaker, lazaridou2020multi}. &#10;&#10;We demonstrate the effectiveness of our method first with automated metrics and then through a human evaluation. &#10;We generate training data using \num{10000} QA examples from TriviaQA \cite{joshi.m.2017}; our automated data generation pipeline (described in \ref{sec:methodology}) allows us to transform these into $\sim14,000$ preference instances, which we use to train several LLMs. &#10;When testing our optimized model on TriviaQA using an LLM listener, we find that open-source LLMs are generally overconfident, producing answers that are accepted by the listener despite often being wrong. &#10;Using \method on three different speaker models (Mistral-7B, Llama3-8B, and Llama3-70B), we obtain substantial gains in induced listener calibration, with an average $20.7$ point gain in AUROC over the base model and a $7.8$ point decrease in calibration error, indicating that utterances from \method-trained models induce more calibrated distributions in the listener. &#10;We also obtain an average $18\%$ absolute improvement in precision, meaning that \method-optimized models produces less over-confident utterances; these utterances are more consistently rejected by the listener when they are wrong.&#10;Furthermore, these benefits translate to instruction-tuned model variants.&#10;Going beyond automated evaluation, we perform a human evaluation in which we show that \method significatly reduces the rate at which incorrect answers are accepted by human listeners; a model trained with \method results in a $47\%$ decrease in the rate of false answers being accepted without significantly increasing the rate of rejection for correct answers.&#10;In our analysis, we show that our training transfers between datasets: we train our models on TriviaQA and evaluate them on TruthfulQA \cite{lin2021truthfulqa}.&#10;Here, we show that \method results in a $28\%$ absolute improvement to truthfulness, as measured by TruthfulQA's metrics. &#10;We underscore our quantitative improvements with a qualitative analysis showing that training leads to more hedging and abstention, as well as more detailed outputs and more authoritative tone when the model is actually correct. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Trustworthy AI Development, Confidence Estimation Methods, Cognitive Science, Linguistics, Language Model Calibration, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1907.12461" label="1907.12461">
        <attvalues>
          <attvalue for="0" value="Leveraging Pre-trained Checkpoints for Sequence Generation Tasks" />
          <attvalue for="1" value="  Unsupervised pre-training of large neural models has recently revolutionized&#10;Natural Language Processing. By warm-starting from the publicly released&#10;checkpoints, NLP practitioners have pushed the state-of-the-art on multiple&#10;benchmarks while saving significant amounts of compute time. So far the focus&#10;has been mainly on the Natural Language Understanding tasks. In this paper, we&#10;demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We&#10;developed a Transformer-based sequence-to-sequence model that is compatible&#10;with publicly available pre-trained BERT, GPT-2 and RoBERTa checkpoints and&#10;conducted an extensive empirical study on the utility of initializing our&#10;model, both encoder and decoder, with these checkpoints. Our models result in&#10;new state-of-the-art results on Machine Translation, Text Summarization,&#10;Sentence Splitting, and Sentence Fusion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.05882" label="2104.05882">
        <attvalues>
          <attvalue for="0" value="Discourse Probing of Pretrained Language Models" />
          <attvalue for="1" value="  Existing work on probing of pretrained language models (LMs) has&#10;predominantly focused on sentence-level syntactic tasks. In this paper, we&#10;introduce document-level discourse probing to evaluate the ability of&#10;pretrained LMs to capture document-level relations. We experiment with 7&#10;pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be&#10;overall the best model at capturing discourse -- but only in its encoder, with&#10;BERT performing surprisingly well as the baseline model. Across the different&#10;models, there are substantial differences in which layers best capture&#10;discourse information, and large disparities between models.&#10;" />
          <attvalue for="2" value="&#10;&#10;The remarkable development of pretrained language models&#10;\cite{devlin-etal-2019-bert,lewis-etal-2020-bart,lan2020albert}&#10;has raised questions about what precise aspects of language these models&#10;do and do not capture. Probing tasks offer a means to perform&#10;fine-grained analysis of the capabilities of such models, but most&#10;existing work has focused on sentence-level analysis such as syntax&#10;\cite{hewitt-manning-2019-structural,jawahar-etal-2019-bert,vries2020what},&#10;entities/relations \cite{papanikolaou-etal-2019-deep}, and ontological&#10;knowledge \cite{michael-etal-2020-asking}. Less is known about how well&#10;such models capture broader discourse in documents.&#10;&#10;Rhetorical Structure Theory is a framework for capturing how sentences&#10;are connected and describing the overall structure of a document&#10;\cite{mann-thompson-1986-assertions}. A number of studies have used&#10;pretrained models to classify discourse markers&#10;\cite{sileo-etal-2019-mining} and discourse relations&#10;\cite{nie-etal-2019-dissent,shi-demberg-2019-next}, but few \cite{koto2021top} have&#10;systematically investigated the ability of pretrained models to model&#10;discourse structure. Furthermore, existing work relating to discourse&#10;probing has typically focused exclusively on the BERT-base model,&#10;leaving open the question of how well these findings generalize to other&#10;models with different pretraining objectives, for different languages,&#10;and different model sizes.&#10;&#10;Our research question in this paper is: How much discourse&#10; structure do layers of different pretrained language models capture,&#10; and do the findings generalize across languages?&#10;&#10;There are two contemporaneous related studies that have examined discourse &#10;modelling in pretrained language models. &#10;\cite{upadhye-etal-2020-predicting} analyzed how well two pretrained&#10;models capture referential biases of different classes of English verbs. &#10;\cite{zhu-etal-2020-examining} applied the model of &#10;\cite{feng-hirst-2014-linear} to parse IMDB documents &#10;\cite{maas-etal-2011-learning} into discourse trees. Using this &#10;(potentially noisy) data,&#10;probing tasks were conducted by mapping attention layers into single&#10;vectors of document-level rhetorical features. &#10;These features, however, are unlikely to capture all the intricacies of&#10;inter-sentential abstraction as their input is formed based on discourse&#10;relations and aggregate statistics on&#10;the distribution of discourse units.&#10;&#10;To summarize, we introduce 7 discourse-related probing tasks, which we&#10;use to analyze 7 pretrained language models over 4 languages: English,&#10;Mandarin Chinese, German, and Spanish. Code and public-domain data&#10;associated with this research is available at&#10;.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.11870" label="2210.11870">
        <attvalues>
          <attvalue for="0" value="LittleBird: Efficient Faster &amp; Longer Transformer for Question Answering" />
          <attvalue for="1" value="  BERT has shown a lot of sucess in a wide variety of NLP tasks. But it has a&#10;limitation dealing with long inputs due to its attention mechanism. Longformer,&#10;ETC and BigBird addressed this issue and effectively solved the quadratic&#10;dependency problem. However we find that these models are not sufficient, and&#10;propose LittleBird, a novel model based on BigBird with improved speed and&#10;memory footprint while maintaining accuracy. In particular, we devise a more&#10;flexible and efficient position representation method based on Attention with&#10;Linear Biases (ALiBi). We also show that replacing the method of global&#10;information represented in the BigBird with pack and unpack attention is more&#10;effective. The proposed model can work on long inputs even after being&#10;pre-trained on short inputs, and can be trained efficiently reusing existing&#10;pre-trained language model for short inputs. This is a significant benefit for&#10;low-resource languages where large amounts of long text data are difficult to&#10;obtain. As a result, our experiments show that LittleBird works very well in a&#10;variety of languages, achieving high performance in question answering tasks,&#10;particularly in KorQuAD2.0, Korean Question Answering Dataset for long&#10;paragraphs.&#10;" />
          <attvalue for="2" value="&#10;Transformer~\cite{vaswani2017attention} and pre-trained language models~\cite{devlin2018bert, liu2019roberta} based on it have shown a lot of success in a wide variety of NLP tasks. However, the quadratic dependency problem that comes from the attention mechanism makes it impractical to process long documents. Many techniques have been studied to overcome this problem and BigBird~\cite{zaheer2020big} showed robust and state-of-the-art performance on various NLP downstream tasks.&#10;&#10;In this study, we propose a new model LittleBird by analyzing and improving the shortcomings of BigBird. LittleBird shows improved speed and memory footprint compared to BigBird while maintaining the overall accuracy of the question answering (QA) benchmarks and showing better accuracy in some of them.&#10;&#10;In this study, we propose three major improvements compared to BigBird. The first is the method for position representation. In BigBird, trainable positional embedding is used similar to BERT~\cite{devlin2018bert}, and in ETC, relative positional encoding is used similar to T5~\cite{raffel2020exploring}. However, trainable positional embedding cannot handle longer inputs than those used for training and the relative position encoding is relatively slow and uses extra memory and parameters~\cite{ma2021luna}. \cite{press2021train} introduced the attention with linear biases (ALiBi) method that resolves these problems, but it was designed for causal language modeling, not autoencoding language modeling, which is typically useful for QA tasks. Thus, we devise a new method based on the ALiBi that is fast, flexible, and also effective in QA tasks.&#10;&#10;The second is the method of capturing global information. BigBird introduces two ways of capturing global information, the random sparse attention and global tokens~\cite{ainslie2020etc} which attend to and be attended by all other tokens. However, the random attention method is practically slow compared to its time complexity because it requires to repeat gather and scatter operations at random positions. In addition, a relatively large number of ($\sim$hundreds) global tokens are required to achieve the reported performance using only global tokens without a random attention method in ETC. We show that replacing them with modified pack and unpack attention~\cite{ma2021luna} is more effective. &#10;&#10;The last is the efficient way to train a model for long sequences. We introduce a simple but effective method, Padding Insertion, which makes the model robust to long inputs while training on short inputs. We also propose a distillation method that can maximize the reuse of the pre-trained model for a short length and show that our model can be effectively pre-trained using these methods.&#10;&#10;Our model shows a 12$\sim$29\% reduction in peak memory usage and a 6$\sim$46\% reduction in latency compared to various BigBird and ETC model settings reported in the paper for 4K length document inference while showing better accuracy on several English QA benchmarks dev sets~\cite{kwiatkowski2019natural, welbl2018constructing}. Our model achieves new state-of-the-art performance on KorQUAD 2.0~\cite{kim2019korquad}, a Korean long document QA benchmark. In addition, these results are obtained with LittileBird pre-trained with only 2K sequence length. It shows that our novel positional representation method works well when the model is applied to the QA downstream task with a document longer than the sequence used in the pre-training phase.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Mathematics, Attention Mechanism Improvements, Natural Language Processing, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2004.08483" label="2004.08483">
        <attvalues>
          <attvalue for="0" value="ETC: Encoding Long and Structured Inputs in Transformers" />
          <attvalue for="1" value="  Transformer models have advanced the state of the art in many Natural&#10;Language Processing (NLP) tasks. In this paper, we present a new Transformer&#10;architecture, Extended Transformer Construction (ETC), that addresses two key&#10;challenges of standard Transformer architectures, namely scaling input length&#10;and encoding structured inputs. To scale attention to longer inputs, we&#10;introduce a novel global-local attention mechanism between global tokens and&#10;regular input tokens. We also show that combining global-local attention with&#10;relative position encodings and a Contrastive Predictive Coding (CPC)&#10;pre-training objective allows ETC to encode structured inputs. We achieve&#10;state-of-the-art results on four natural language datasets requiring long&#10;and/or structured inputs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.17826" label="2402.17826">
        <attvalues>
          <attvalue for="0" value="Prediction-Powered Ranking of Large Language Models" />
          <attvalue for="1" value="  Large language models are often ranked according to their level of alignment&#10;with human preferences -- a model is better than other models if its outputs&#10;are more frequently preferred by humans. One of the popular ways to elicit&#10;human preferences utilizes pairwise comparisons between the outputs provided by&#10;different models to the same inputs. However, since gathering pairwise&#10;comparisons by humans is costly and time-consuming, it has become a common&#10;practice to gather pairwise comparisons by a strong large language model -- a&#10;model strongly aligned with human preferences. Surprisingly, practitioners&#10;cannot currently measure the uncertainty that any mismatch between human and&#10;model preferences may introduce in the constructed rankings. In this work, we&#10;develop a statistical framework to bridge this gap. Given a (small) set of&#10;pairwise comparisons by humans and a large set of pairwise comparisons by a&#10;model, our framework provides a rank-set -- a set of possible ranking positions&#10;-- for each of the models under comparison. Moreover, it guarantees that, with&#10;a probability greater than or equal to a user-specified value, the rank-sets&#10;cover the true ranking consistent with the distribution of human pairwise&#10;preferences asymptotically. Using pairwise comparisons made by humans in the&#10;LMSYS Chatbot Arena platform and pairwise comparisons made by three strong&#10;large language models, we empirically demonstrate the effectivity of our&#10;framework and show that the rank-sets constructed using only pairwise&#10;comparisons by the strong large language models are often inconsistent with&#10;(the distribution of) human pairwise preferences.&#10;" />
          <attvalue for="2" value="&#10;During the last years, large language models (LLMs) have shown a remarkable ability to &#10;generate and understand general-purpose language~\cite{bubeck2023sparks}.&#10;As a result, there has been an increasing excitement in their potential to help&#10;humans solve a variety of open-ended, complex tasks across many application domains&#10;such as coding~\cite{mozannar2022reading}, healthcare~\cite{haupt2023ai} and scientific discovery~\cite{romera2023mathematical}, to name a few.&#10;However, evaluating and comparing the performance of different LLMs has become &#10;very challenging~\cite{chang2024asurvey}.&#10;The main reason is that, &#10;in contrast to traditional machine learning models, &#10;LLMs can solve a large number of different tasks and, in many of these tasks, there &#10;is not a unique, structured solution. &#10;As a consequence, there has been a paradigm shift towards evaluating their performance &#10;according to their level of alignment with human preferences---a model is better &#10;than other models if its outputs are more frequently preferred by humans~\cite{hendryckstest2021,wang2022self,ouyang2022training,wang2023aligning,lmsys2023chatbot}.&#10;&#10;One of the most popular paradigms to rank a set of LLMs according to their level of alignment with human preferences utilizes pairwise comparisons~\cite{lmsys2023chatbot,taori2023stanford,zheng2023judging,li2023generative,li2023prd,boubdir2023elo,singhal2023large,chiang2024chatbot}. &#10;Under this paradigm, each pairwise comparison comprises the outputs of two different models picked uniformly at random to an input&#10;sampled from a given dis\-tri\-bu\-tion of inputs.&#10;Moreover, the pairwise comparisons are used to rank the models with a variety of methods such as the Elo&#10;rating~\cite{askell2021general,dettmers2024qlora,bai2022training,wu2023chatarena,lin2023llm}, the Bradley-Terry model~\cite{boyeau2024autoeval,chiang2024chatbot,lmsys2023chatbot} or the win-rate~\cite{zheng2023judging,boyeau2024autoeval,chiang2024chatbot}.&#10;While it is widely agreed that, given a sufficiently large set of pairwise comparisons, higher (lower) ranking under &#10;this paradigm corresponds to better (worse) human alignment, &#10;there have also been increasing concerns that this paradigm is too costly and time-consuming to be practical, &#10;especially given the pace at which models are updated and new models are developed.&#10;&#10;To lower the cost and increase the efficiency of ranking from pairwise comparisons, &#10;it has become a common practice to ask a strong LLM---a model known to strongly align with human preferences---to perform pairwise comparisons~\cite{thomas2023large,wang2023large,chiang2023vicuna,chiang2023can,jiang2023llm,wang2024pandalm,qin2023ischatgpt,dubois2024alpacafarm,qin2023large,liu2024aligning}. &#10;The rationale is that, if a model strongly aligns with human pre\-fe\-ren\-ces, then, the distributions of pairwise comparisons by the model and by the human should in principle match~\cite{thomas2023large,chiang2023can,verma2023preference}.&#10;Worrying\-ly, there are multiple lines of evidence,&#10;including our experimental findings in Figure~\ref{fig:ranks-gpt4},&#10;showing that the rankings constructed using pairwise comparisons made by a strong LLM are sometimes different to those constructed using pairwise comparisons by humans~\cite{zheng2023judging, li2023prd,boubdir2023elo,singhal2023large,dettmers2024qlora,davidson2024evaluating,hou2024large}, questioning the rationale above. &#10;In this work, we introduce a statistical framework to measure the uncertainty in the rankings constructed using pairwise comparisons made by a model, which may be introduced by a mismatch between human and model preferences or by the fact that we use a finite number of pairwise comparisons.&#10;&#10;\xhdr{Our contributions}&#10;Our framework measures uncertainty using rank-sets---sets of possible ranking positions that each model can take.&#10;If the rank-set of a model is large (small), it means that there is high (low) uncertainty in the ranking position of the model. &#10;To construct the rank-sets, &#10;our framework first leverages a (small) set of pairwise comparisons by humans and a large set of pairwise comparisons by a strong LLM to create a confidence ellipsoid.&#10;By using prediction-powered inference~\cite{angelopoulos2023prediction,angelopoulos2023ppi++,zrnic2024cross},&#10;this confidence ellipsoid is guaranteed to contain the vector of (true) probabilities that each model is preferred over others by humans---the win-rates---with a user-spe\-ci\-fied coverage probability $1 - \alpha$.&#10;Then, it uses the distance between this ellipsoid and the hyperplanes under which pairs of models have the same probability values of being preferred over others &#10;to efficiently cons\-truct the rank-sets.&#10;Importantly, we can show that, with probability greater than or equal to $1-\alpha$, the constructed rank-sets are guaranteed to cover the ranking consistent with the (true) probability that each model is preferred over others by humans asymptotically.&#10;Moreover, our framework does not make any assumptions on the distribution of human preferences nor about the degree of alignment between pairwise preferences of humans and the strong LLM.&#10;Experiments on pairwise comparisons made by humans in the LMSYS Chatbot Arena platform~\cite{jiang2023llm} and pairwise comparisons made by three strong LLMs, namely GPT 3.5, Claude 3 and GPT 4, &#10;empirically demonstrate that the rank-sets constructed using our framework are &#10;more likely to cover the true ranking consistent with (the distribution of) human pairwise preferences than the rank-sets constructed using only pairwise comparisons made by the strong LLMs.&#10;An open-source implementation of our methodology as well as the data on pairwise preferences of strong LLMs used in our experiments are available at \href{&#10;&#10;\xhdr{Further related work}&#10;Our work builds upon recent work on prediction-powered inference, ranking under uncertainty, and ran\-king of LLMs.&#10;&#10;Prediction-powered inference~\cite{angelopoulos2023prediction,angelopoulos2023ppi++,zrnic2024cross} is a recently &#10;introduced statistical framework to obtain valid $p$-values and confidence intervals about a population-level quantity such as the mean outcome or a regression coefficient using a small labeled dataset and a large unlabeled dataset, whose labels are imputed using a black-box machine learning model.&#10;However, our work is the first to use prediction-powered inference (as a subroutine) to construct rank-sets &#10;with coverage guarantees.&#10;In this context, it is worth acknowledging that a very recent work by Saad-Falcon et al.~\cite{saadfalcon2023ares} has used prediction-powered inference to construct (single) rankings, rather than rank-sets. &#10;However, their rankings do not enjoy coverage guarantees with respect to the true ranking consistent with (the distribution of) the human preferences.&#10;Moreover, an independent, concurrent work by Boyeau et al.~\cite{boyeau2024autoeval} has also used prediction-powered inference to construct (single) rankings based on the estimated coefficients of a Bradley-Terry model. &#10;However, the estimated coefficients come with large, overlapping confidence intervals, which would have led to&#10;uninformative rank-sets, had the authors used them to construct rank-sets.&#10;&#10;The vast majority of the literature on ranking under uncertainty has focused on confidence intervals for &#10;individual ranking positions~\cite{lemmers2007incorporating, goldstein1996league, hall2009using, marshall1998reliability, wright2014ranking, xie2009confidence, zhang2014confidence}. &#10;Only recently, a paucity of work has focused on joint measures of uncertainty for rankings~\cite{neuhof2023confident, rising2021uncertainty, al2022simultaneous, klein2020joint}.&#10;Similarly as in our work, this line of work also seeks to construct rank-sets with coverage guarantees. &#10;However, &#10;in contrast to our work, &#10;it estimates the quality metric (in our work, the probability that an LLM is preferred over others) and the confidence intervals separately for each of the items (in our work, LLMs) using independent samples. &#10;As a consequence, it needs to perform multiple comparison &#10;correction to create the rank-sets.&#10;&#10;In recent years, there has also been a flurry of work on ranking LLMs using benchmark datasets with manually hand-crafted inputs and ground-truth outputs~\cite{bach2022promptsource,wei2022finetuned,talmor2019commonsense,mishra2022cross,chen2021evaluating,liang2023holistic,longpre2023flan}.&#10;However, it has become increasingly clear that oftentimes rankings derived from benchmark datasets do not &#10;correlate well with rankings derived from human preferences---an improved ranking position in the former&#10;does not lead to an improved ranking position in the latter~\cite{zheng2023judging,li2023generative,li2023prd,chiang2023vicuna,chiang2024chatbot}.&#10;Within the literature on ranking LLMs from pairwise comparisons, most studies use the Elo rating system~\cite{askell2021general,dettmers2024qlora,bai2022training,wu2023chatarena,lin2023llm}, originally introduced for chess tournaments~\cite{elo1966uscf}. &#10;However, Elo-based rankings are sensitive to the order of pairwise comparisons, as newer comparisons have more weight than older ones, which leads to unstable rankings~\cite{boubdir2023elo}.&#10;To address this limitation, several studies have instead used the Bradley-Terry model~\cite{lmsys2023chatbot,chiang2024chatbot, boyeau2024autoeval}, which weighs pairwise comparisons equally regardless of their order. Nevertheless, both the Elo rating system and the Bradley-Terry model have faced criticism, &#10;as pairwise comparisons often fail to satisfy the fundamental axiom of transitivity, &#10;upon which both approaches rely~\cite{boubdir2023elo,bertrand2023limitations}, &#10;Recently, several studies have used the win-rate~\cite{zheng2023judging,chiang2024chatbot,boyeau2024autoeval}, which weighs comparisons equally regardless of their order and does not require the transitivity assumption, but requires humans to make pairwise comparisons between every pair of models.&#10;In our work, we build upon the win-rate and lift the above requirement by using pairwise comparisons made by a strong LLM.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human Preference Elicitation, Statistical Ranking Framework, Computer Science, Artificial Intelligence, Mathematics, Natural Language Processing, Statistics, Language Model Alignment" />
        </attvalues>
      </node>
      <node id="2210.14306" label="2210.14306">
        <attvalues>
          <attvalue for="0" value="Reading Between the Lines: Modeling User Behavior and Costs in&#10;  AI-Assisted Programming" />
          <attvalue for="1" value="  Code-recommendation systems, such as Copilot and CodeWhisperer, have the&#10;potential to improve programmer productivity by suggesting and auto-completing&#10;code. However, to fully realize their potential, we must understand how&#10;programmers interact with these systems and identify ways to improve that&#10;interaction. To seek insights about human-AI collaboration with code&#10;recommendations systems, we studied GitHub Copilot, a code-recommendation&#10;system used by millions of programmers daily. We developed CUPS, a taxonomy of&#10;common programmer activities when interacting with Copilot. Our study of 21&#10;programmers, who completed coding tasks and retrospectively labeled their&#10;sessions with CUPS, showed that CUPS can help us understand how programmers&#10;interact with code-recommendation systems, revealing inefficiencies and time&#10;costs. Our insights reveal how programmers interact with Copilot and motivate&#10;new interface designs and metrics.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.02762" label="2307.02762">
        <attvalues>
          <attvalue for="0" value="PRD: Peer Rank and Discussion Improve Large Language Model based&#10;  Evaluations" />
          <attvalue for="1" value="  Nowadays, the quality of responses generated by different modern large&#10;language models (LLMs) is hard to evaluate and compare automatically. Recent&#10;studies suggest and predominantly use LLMs for reference-free evaluation of&#10;open-ended question answering. More specifically, they use the recognized&#10;&quot;strongest&quot; LLM as the evaluator, which conducts pairwise comparisons of&#10;candidate models' answers and provides a ranking score. However, this intuitive&#10;method has multiple problems, such as bringing in self-enhancement (favoring&#10;its own answers) and positional bias. We draw insights and lessons from the&#10;educational domain (Cho &amp; MacArthur, 2011; Walsh, 2014) to improve LLM-based&#10;evaluations. Specifically, we propose (1) the peer rank (PR) algorithm that&#10;takes into account each peer LLM's pairwise preferences of all answer pairs,&#10;and outputs a final ranking of models; and (2) peer discussion (PD), where we&#10;prompt two LLMs to discuss and try to reach a mutual agreement on the&#10;preferences of two answers. We conduct experiments on two benchmark datasets.&#10;We find that our approaches achieve higher accuracy and align better with human&#10;judgments. Interestingly, PR can induce a relatively accurate self-ranking of&#10;models under the anonymous setting, where each model's name is unrevealed. Our&#10;work provides space to explore evaluating models that are hard to compare for&#10;humans.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.17295" label="2311.17295">
        <attvalues>
          <attvalue for="0" value="Elo Uncovered: Robustness and Best Practices in Language Model&#10;  Evaluation" />
          <attvalue for="1" value="  In Natural Language Processing (NLP), the Elo rating system, originally&#10;designed for ranking players in dynamic games such as chess, is increasingly&#10;being used to evaluate Large Language Models (LLMs) through &quot;A vs B&quot; paired&#10;comparisons. However, while popular, the system's suitability for assessing&#10;entities with constant skill levels, such as LLMs, remains relatively&#10;unexplored. We study two fundamental axioms that evaluation methods should&#10;adhere to: reliability and transitivity. We conduct extensive evaluation of Elo&#10;behaviour, illustrating that individual Elo computations exhibit volatility and&#10;delving into the impact of varying the Elo rating system's hyperparameters. We&#10;show that these axioms are not always satisfied raising questions about the&#10;reliability of current comparative evaluations of LLMs. If the current use of&#10;Elo scores is intended to substitute the costly head-to-head comparison of&#10;LLMs, it is crucial to ensure the ranking is as robust as possible. Guided by&#10;the axioms, our findings offer concrete guidelines for enhancing the&#10;reliability of LLM evaluation methods, suggesting a need for reassessment of&#10;existing comparative approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.07008" label="2403.07008">
        <attvalues>
          <attvalue for="0" value="AutoEval Done Right: Using Synthetic Data for Model Evaluation" />
          <attvalue for="1" value="  The evaluation of machine learning models using human-labeled validation data&#10;can be expensive and time-consuming. AI-labeled synthetic data can be used to&#10;decrease the number of human annotations required for this purpose in a process&#10;called autoevaluation. We suggest efficient and statistically principled&#10;algorithms for this purpose that improve sample efficiency while remaining&#10;unbiased. These algorithms increase the effective human-labeled sample size by&#10;up to 50% on experiments with GPT-4.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.17563" label="2306.17563">
        <attvalues>
          <attvalue for="0" value="Large Language Models are Effective Text Rankers with Pairwise Ranking&#10;  Prompting" />
          <attvalue for="1" value="  Ranking documents using Large Language Models (LLMs) by directly feeding the&#10;query and candidate documents into the prompt is an interesting and practical&#10;problem. However, researchers have found it difficult to outperform fine-tuned&#10;baseline rankers on benchmark datasets. We analyze pointwise and listwise&#10;ranking prompts used by existing methods and argue that off-the-shelf LLMs do&#10;not fully understand these challenging ranking formulations. In this paper, we&#10;propose to significantly reduce the burden on LLMs by using a new technique&#10;called Pairwise Ranking Prompting (PRP). Our results are the first in the&#10;literature to achieve state-of-the-art ranking performance on standard&#10;benchmarks using moderate-sized open-sourced LLMs. On TREC-DL 2019&amp;2020, PRP&#10;based on the Flan-UL2 model with 20B parameters performs favorably with the&#10;previous best approach in the literature, which is based on the blackbox&#10;commercial GPT-4 that has 50x (estimated) model size, while outperforming other&#10;LLM-based solutions, such as InstructGPT which has 175B parameters, by over 10%&#10;for all ranking metrics. By using the same prompt template on seven BEIR tasks,&#10;PRP outperforms supervised baselines and outperforms the blackbox commercial&#10;ChatGPT solution by 4.2% and pointwise LLM-based solutions by more than 10% on&#10;average NDCG@10. Furthermore, we propose several variants of PRP to improve&#10;efficiency and show that it is possible to achieve competitive results even&#10;with linear complexity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.16950" label="2403.16950">
        <attvalues>
          <attvalue for="0" value="Aligning with Human Judgement: The Role of Pairwise Preference in Large&#10;  Language Model Evaluators" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated promising capabilities as&#10;automatic evaluators in assessing the quality of generated natural language.&#10;However, LLMs still exhibit biases in evaluation and often struggle to generate&#10;coherent evaluations that align with human assessments. In this work, we first&#10;conduct a systematic study of the misalignment between LLM evaluators and human&#10;judgement, revealing that existing calibration methods aimed at mitigating&#10;biases are insufficient for effectively aligning LLM evaluators. Inspired by&#10;the use of preference data in RLHF, we formulate the evaluation as a ranking&#10;problem and introduce Pairwise-preference Search (PairS), an uncertainty-guided&#10;search method that employs LLMs to conduct pairwise comparisons and efficiently&#10;ranks candidate texts. PairS achieves state-of-the-art performance on&#10;representative evaluation tasks and demonstrates significant improvements over&#10;direct scoring. Furthermore, we provide insights into the role of pairwise&#10;preference in quantifying the transitivity of LLMs and demonstrate how PairS&#10;benefits from calibration.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.04536" label="2401.04536">
        <attvalues>
          <attvalue for="0" value="Evaluating Language Model Agency through Negotiations" />
          <attvalue for="1" value="  We introduce an approach to evaluate language model (LM) agency using&#10;negotiation games. This approach better reflects real-world use cases and&#10;addresses some of the shortcomings of alternative LM benchmarks. Negotiation&#10;games enable us to study multi-turn, and cross-model interactions, modulate&#10;complexity, and side-step accidental evaluation data leakage. We use our&#10;approach to test six widely used and publicly accessible LMs, evaluating&#10;performance and alignment in both self-play and cross-play settings. Noteworthy&#10;findings include: (i) only closed-source models tested here were able to&#10;complete these tasks; (ii) cooperative bargaining games proved to be most&#10;challenging to the models; and (iii) even the most powerful models sometimes&#10;&quot;lose&quot; to weaker opponents&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.09633" label="2301.09633">
        <attvalues>
          <attvalue for="0" value="Prediction-Powered Inference" />
          <attvalue for="1" value="  Prediction-powered inference is a framework for performing valid statistical&#10;inference when an experimental dataset is supplemented with predictions from a&#10;machine-learning system. The framework yields simple algorithms for computing&#10;provably valid confidence intervals for quantities such as means, quantiles,&#10;and linear and logistic regression coefficients, without making any assumptions&#10;on the machine-learning algorithm that supplies the predictions. Furthermore,&#10;more accurate predictions translate to smaller confidence intervals.&#10;Prediction-powered inference could enable researchers to draw valid and more&#10;data-efficient conclusions using machine learning. The benefits of&#10;prediction-powered inference are demonstrated with datasets from proteomics,&#10;astronomy, genomics, remote sensing, census analysis, and ecology.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.01453" label="2311.01453">
        <attvalues>
          <attvalue for="0" value="PPI++: Efficient Prediction-Powered Inference" />
          <attvalue for="1" value="  We present PPI++: a computationally lightweight methodology for estimation&#10;and inference based on a small labeled dataset and a typically much larger&#10;dataset of machine-learning predictions. The methods automatically adapt to the&#10;quality of available predictions, yielding easy-to-compute confidence sets --&#10;for parameters of any dimensionality -- that always improve on classical&#10;intervals using only the labeled data. PPI++ builds on prediction-powered&#10;inference (PPI), which targets the same problem setting, improving its&#10;computational and statistical efficiency. Real and synthetic experiments&#10;demonstrate the benefits of the proposed adaptations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.16598" label="2309.16598">
        <attvalues>
          <attvalue for="0" value="Cross-Prediction-Powered Inference" />
          <attvalue for="1" value="  While reliable data-driven decision-making hinges on high-quality labeled&#10;data, the acquisition of quality labels often involves laborious human&#10;annotations or slow and expensive scientific measurements. Machine learning is&#10;becoming an appealing alternative as sophisticated predictive techniques are&#10;being used to quickly and cheaply produce large amounts of predicted labels;&#10;e.g., predicted protein structures are used to supplement experimentally&#10;derived structures, predictions of socioeconomic indicators from satellite&#10;imagery are used to supplement accurate survey data, and so on. Since&#10;predictions are imperfect and potentially biased, this practice brings into&#10;question the validity of downstream inferences. We introduce cross-prediction:&#10;a method for valid inference powered by machine learning. With a small labeled&#10;dataset and a large unlabeled dataset, cross-prediction imputes the missing&#10;labels via machine learning and applies a form of debiasing to remedy the&#10;prediction inaccuracies. The resulting inferences achieve the desired error&#10;probability and are more powerful than those that only leverage the labeled&#10;data. Closely related is the recent proposal of prediction-powered inference,&#10;which assumes that a good pre-trained model is already available. We show that&#10;cross-prediction is consistently more powerful than an adaptation of&#10;prediction-powered inference in which a fraction of the labeled data is split&#10;off and used to train the model. Finally, we observe that cross-prediction&#10;gives more stable conclusions than its competitors; its confidence intervals&#10;typically have significantly lower variability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.15361" label="2307.15361">
        <attvalues>
          <attvalue for="0" value="Confident Feature Ranking" />
          <attvalue for="1" value="  Machine learning models are widely applied in various fields. Stakeholders&#10;often use post-hoc feature importance methods to better understand the input&#10;features' contribution to the models' predictions. The interpretation of the&#10;importance values provided by these methods is frequently based on the relative&#10;order of the features (their ranking) rather than the importance values&#10;themselves. Since the order may be unstable, we present a framework for&#10;quantifying the uncertainty in global importance values. We propose a novel&#10;method for the post-hoc interpretation of feature importance values that is&#10;based on the framework and pairwise comparisons of the feature importance&#10;values. This method produces simultaneous confidence intervals for the&#10;features' ranks, which include the ``true'' (infinite sample) ranks with high&#10;probability, and enables the selection of the set of the top-k important&#10;features.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.01279" label="2202.01279">
        <attvalues>
          <attvalue for="0" value="PromptSource: An Integrated Development Environment and Repository for&#10;  Natural Language Prompts" />
          <attvalue for="1" value="  PromptSource is a system for creating, sharing, and using natural language&#10;prompts. Prompts are functions that map an example from a dataset to a natural&#10;language input and target output. Using prompts to train and query language&#10;models is an emerging area in NLP that requires new tools that let users&#10;develop and refine these prompts collaboratively. PromptSource addresses the&#10;emergent challenges in this new setting with (1) a templating language for&#10;defining data-linked prompts, (2) an interface that lets users quickly iterate&#10;on prompt development by observing outputs of their prompts on many examples,&#10;and (3) a community-driven set of guidelines for contributing new prompts to a&#10;common pool. Over 2,000 prompts for roughly 170 datasets are already available&#10;in PromptSource. PromptSource is available at&#10;https://github.com/bigscience-workshop/promptsource.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.11504" label="2303.11504">
        <attvalues>
          <attvalue for="0" value="Language Model Behavior: A Comprehensive Survey" />
          <attvalue for="1" value="  Transformer language models have received widespread public attention, yet&#10;their generated text is often surprising even to NLP researchers. In this&#10;survey, we discuss over 250 recent studies of English language model behavior&#10;before task-specific fine-tuning. Language models possess basic capabilities in&#10;syntax, semantics, pragmatics, world knowledge, and reasoning, but these&#10;capabilities are sensitive to specific inputs and surface features. Despite&#10;dramatic increases in generated text quality as models scale to hundreds of&#10;billions of parameters, the models are still prone to unfactual responses,&#10;commonsense errors, memorized text, and social biases. Many of these weaknesses&#10;can be framed as over-generalizations or under-generalizations of learned&#10;patterns in text. We synthesize recent results to highlight what is currently&#10;known about large language model capabilities, thus providing a resource for&#10;applied work and for research in adjacent fields that use language models.&#10;" />
          <attvalue for="2" value="&#10;Transformer language models have revolutionized the field of natural language processing (NLP) since their introduction in 2018 \cite{radford-etal-2018-improving,devlin-etal-2019-bert}.&#10;Recent research and public attention has demonstrated that large language models (e.g. GPT-3/4, PaLM, and OPT; \cite{brown-etal-2020-language,chowdhery-etal-2022-palm,zhang-etal-2022-opt,openai-2023-gpt4}) can achieve remarkable performance both on standard NLP benchmarks and on open-ended natural language generation tasks from the general public \cite{wang-etal-2019-superglue,johnson-2022-ai}.&#10;Already, language models are used in industry for applications ranging from web search and chatbots to medical and financial document analysis \cite{nayak-2019-understanding,broyde-palmer-2021-build,thewsey-2021-bring,lee-2023-what}.&#10;Due to their widespread applicability, language models have been called ``foundation models'' for NLP \cite{bommasani-etal-2021-on}.&#10;&#10;Language models are trained to predict masked (i.e. hidden) or upcoming words from context, usually text.&#10;The models can then be fine-tuned for specific downstream tasks (e.g. text classification; \cite{devlin-etal-2019-bert}), or they can be used directly for any text prediction task.&#10;As language model capabilities have expanded in recent years, they have increasingly been used in the text generation scenario with minimal or no fine-tuning \cite{brown-etal-2020-language}.&#10;This approach requires no task-specific data or further training infrastructure, thus expanding the range of possibilities and audience for language model applications.&#10;In particular, the release of public APIs and interfaces such as GPT-3 and ChatGPT \cite{brown-etal-2020-language,openai-2021-chatgpt} have enabled widespread public experimentation on the text generation capabilities of language models.&#10;&#10;Yet, text generated by language models is often surprising even to NLP researchers.&#10;Previous studies have investigated both the outputs and internal mechanisms of language models, originally focusing on masked (i.e. fill-in-the-blank) ``BERT'' models and establishing the field of ``BERTology'' (see \cite{rogers-etal-2020-a} for a survey).&#10;In the years since the last BERTology survey in 2020, and in tandem with the rise of large autoregressive models such as GPT-3 (i.e. predicting upcoming words instead of masked words), language model analysis has shifted focus to these large autoregressive models.&#10;Because these models are often used without fine-tuning for open-ended text generation, there have been an increasing number of behavioral studies evaluating the output text probabilities of language models.&#10;&#10;Despite this flurry of research, language model text generation behavior remains unpredictable.&#10;Although model performance on broad benchmark datasets is relatively consistent for a given model size and architecture, responses to specific inputs and examples are not. This feature makes large language models tempting but unreliable to use in many practical applications \cite{ganguli-etal-2022-predictability}.&#10;Furthermore, the rapid pace of NLP research and the quantity of individual studies make any progress in understanding model behavior difficult to track.&#10;As language models become more widespread and researchers from other fields invest interest in language models, it is increasingly important that our existing understanding of model behavior be made clear and accessible.&#10;&#10;In this survey, we discuss over 250 recent studies of English language model behavior, covering syntax, semantics, pragmatics, world knowledge, reasoning, memorization, and bias.&#10;Language models generate fluent and coherent text, but their predictions are highly dependent on input context.&#10;Slight changes in input word choice and phrasing can lead to unfactual, offensive, or plagiarized text.&#10;Understanding these behaviors has broad implications for informed applications in industry \cite{weidinger-etal-2021-ethical} and general questions about meaning and ``understanding'' in artificial agents \cite{bender-koller-2020-climbing,mitchell-krakauer-2022-the,shardlow-przybyla-2022-deanthropomorphising}.&#10;&#10;To the extent possible, we avoid taking a stance on whether language models truly ``understand'' language.&#10;We also leave deeper ethical discussions of the societal implications of language models to surveys focused specifically on that area (e.g. \cite{weidinger-etal-2021-ethical,weidinger-etal-2022-taxonomy}).&#10;Instead, we hope to provide a review of the empirical evidence for what behaviors language models exhibit in controlled settings.&#10;We discuss a wide range of model capabilities and weaknesses (Sections\ref{sec:syntax} through\ref{sec:misinformation-personality-politics}), and we synthesize results framed from the perspectives of model scale (Section\ref{sec:scale}) and text pattern generalization (Section\ref{sec:language-modeling-generalization}).&#10;In this way, we hope to combat anecdote-driven language model ``hype'' with informed hype grounded in what language models actually can and cannot do \cite{bowman-2021-when}, while also highlighting potential future directions of research in language model behavioral analysis.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Language Model Behavior, Natural Language Processing, Artificial Intelligence Limitations" />
        </attvalues>
      </node>
      <node id="2211.11483" label="2211.11483">
        <attvalues>
          <attvalue for="0" value="Deanthropomorphising NLP: Can a Language Model Be Conscious?" />
          <attvalue for="1" value="  This work is intended as a voice in the discussion over previous claims that&#10;a pretrained large language model (LLM) based on the Transformer model&#10;architecture can be sentient. Such claims have been made concerning the LaMDA&#10;model and also concerning the current wave of LLM-powered chatbots, such as&#10;ChatGPT. This claim, if confirmed, would have serious ramifications in the&#10;Natural Language Processing (NLP) community due to wide-spread use of similar&#10;models. However, here we take the position that such a large language model&#10;cannot be sentient, or conscious, and that LaMDA in particular exhibits no&#10;advances over other similar models that would qualify it. We justify this by&#10;analysing the Transformer architecture through Integrated Information Theory of&#10;consciousness. We see the claims of sentience as part of a wider tendency to&#10;use anthropomorphic language in NLP reporting. Regardless of the veracity of&#10;the claims, we consider this an opportune moment to take stock of progress in&#10;language modelling and consider the ethical implications of the task. In order&#10;to make this work helpful for readers outside the NLP community, we also&#10;present the necessary background in language modelling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.08300" label="2110.08300">
        <attvalues>
          <attvalue for="0" value="The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP&#10;  Systems Fail" />
          <attvalue for="1" value="  Researchers in NLP often frame and discuss research results in ways that&#10;serve to deemphasize the field's successes, often in response to the field's&#10;widespread hype. Though well-meaning, this has yielded many misleading or false&#10;claims about the limits of our best technology. This is a problem, and it may&#10;be more serious than it looks: It harms our credibility in ways that can make&#10;it harder to mitigate present-day harms, like those involving biased systems&#10;for content moderation or resume screening. It also limits our ability to&#10;prepare for the potentially enormous impacts of more distant future advances.&#10;This paper urges researchers to be careful about these claims and suggests some&#10;research directions and communication strategies that will make it easier to&#10;avoid or rebut them.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10649" label="2305.10649">
        <attvalues>
          <attvalue for="0" value="ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMs" />
          <attvalue for="1" value="  In this paper, we present ZeroPrompt (Figure 1-(a)) and the corresponding&#10;Prompt-and-Refine strategy (Figure 3), two simple but effective&#10;\textbf{training-free} methods to decrease the Token Display Time (TDT) of&#10;streaming ASR models \textbf{without any accuracy loss}. The core idea of&#10;ZeroPrompt is to append zeroed content to each chunk during inference, which&#10;acts like a prompt to encourage the model to predict future tokens even before&#10;they were spoken. We argue that streaming acoustic encoders naturally have the&#10;modeling ability of Masked Language Models and our experiments demonstrate that&#10;ZeroPrompt is engineering cheap and can be applied to streaming acoustic&#10;encoders on any dataset without any accuracy loss. Specifically, compared with&#10;our baseline models, we achieve 350 $\sim$ 700ms reduction on First Token&#10;Display Time (TDT-F) and 100 $\sim$ 400ms reduction on Last Token Display Time&#10;(TDT-L), with theoretically and experimentally equal WER on both Aishell-1 and&#10;Librispeech datasets.&#10;" />
          <attvalue for="2" value="&#10;In the past few years, end-to-end models, such as connectionist temporal classification (CTC)~\cite{ctc}, RNN-Transducer (RNN-T)~\cite{rnnt}, and attention-based encoder-decoder (AED)~\cite{speech-transformer} models, have achieved significant success on various ASR tasks. Recently, there has been a growing interest in developing end-to-end ASR models with streaming capability. Among them, chunk-based acoustic encoders~\cite{dual,saa,u2} have gained popularity and have been adopted in many previous works. These methods utilize bi-directional recurrent networks~\cite{blstm} or fully-connected self-attention networks~\cite{transformer} within a chunk. In this work, we primarily focus on chunk-based methods due to their full-context utilization in a chunk.&#10;&#10;In streaming scenarios such as real-time subtitles, ASR systems need to decode speech with low latency, producing words as soon as possible~\cite{upl}. A straightforward way to reduce latency is directly decreasing chunk size (i.e., from 640ms to 320ms). However, there is often a trade-off between performance and latency and lower chunk size usually leads to higher WER. Another way to reduce latency is to apply regularization either on loss function~\cite{fastemit,peak-first-ctc} or input spectrogram~\cite{trimtail} to push forward the emission of tokens. While being successful in terms of reducing the token emission latency of streaming ASR models, the definition of token emission latency (i.e., The timestamp or frame index when the model predicts the token) underestimates the true user-perceived latency (such as Token Display Time) in chunk-based models, since they do not account for chunk cumulative time (a.k.a, the time to wait before the input signal forms a chunk). Here, we further provide an example to explain why token emission latency does not correlate well with our notion of user-perceived latency. In Figure 2, assume the second char of the recognition result happens at 1000ms and is pushed forward to 800ms after training with emission regularization, the model still needs to wait until 1200ms to form a valid chunk and hence start to decode and emit the second char.&#10;&#10;To better measure the latency terms that accurately capture the user-perceived latency, we propose two metrics as illustrated in Figure 2: First Token Display Time (TDT-F) and Last Token Display Time (TDT-L) - the minimum chunk cumulative time required to output the first or last character. In real-time subtitle scenarios, those metrics can be used to evaluate the initial on-screen time of the first and last characters. For simplicity, we ignore the chunk computation time because it is usually much smaller than the chunk cumulative time, i.e., inference one chunk with 640ms chunk size usually takes only 50ms on a desktop CPU using single thread.&#10;&#10;In this paper, we explore a training-free method, called ZeroPrompt, which appends zeroed content to each chunk to prompt the model to predict future tokens through its zero-shot ability of Masked LMs that has been implicitly learned during training. We argue that previous works mainly focus on the decoder part of encoder-decoder E2E ASR structure rather than the encoder part to estimate the internal LM because the encoder part is usually optimized with CTC loss and CTC is generally not considered capable of modeling context between output tokens due to conditional independence assumption~\cite{adapter}. However, CTC-optimized ASR encoders learn the training data distribution and are affected by the frequency of words in the training data. The CTC-optimized encoder therefore at least has the modeling ability of a unigram LM to do something like MaskPredict (see Figure 1-(a) and Figure 1-(c) for a clearer comparison between ZeroPrompt and MaskPredict~\cite{bert}), and this paper aims to adopt this zero-shot ability to predict future tokens even before they were spoken and hence greatly reduce the TDT-F \&amp; TDT-L during inference. Besides, to ensure that the final decoding result (or WER) is not affected, we propose to use a chunk-level autoregressive attention mask described in Figure 1-(b), coupled with a revision strategy called Prompt-and-Refine, to iteratively predict future tokens and refine them when the real future chunk arrives (see Figure 3 for a detailed example). Experimental results in Section 3 demonstrate that our methods have many advantages which can be summarized as:&#10;\begin{itemize}&#10; \item ZeroPrompt does not require any model re-training and it takes nearly zero engineering cost to plugin any chunk-based streaming decoding procedure.&#10; \item ZeroPrompt can not only decrease the TDT-F \&amp; TDT-L for partial recognition results but also keep the WER unaffected for final recoginition results. In other words, we achieve the theoretically and experimentally best trade-off between latency and WER.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Streaming ASR Models, Computer Science, Speech Recognition Optimization, Linguistics, Engineering, Signal Processing, Training-Free Methods" />
        </attvalues>
      </node>
      <node id="2104.02207" label="2104.02207">
        <attvalues>
          <attvalue for="0" value="Dissecting User-Perceived Latency of On-Device E2E Speech Recognition" />
          <attvalue for="1" value="  As speech-enabled devices such as smartphones and smart speakers become&#10;increasingly ubiquitous, there is growing interest in building automatic speech&#10;recognition (ASR) systems that can run directly on-device; end-to-end (E2E)&#10;speech recognition models such as recurrent neural network transducers and&#10;their variants have recently emerged as prime candidates for this task. Apart&#10;from being accurate and compact, such systems need to decode speech with low&#10;user-perceived latency (UPL), producing words as soon as they are spoken. This&#10;work examines the impact of various techniques - model architectures, training&#10;criteria, decoding hyperparameters, and endpointer parameters - on UPL. Our&#10;analyses suggest that measures of model size (parameters, input chunk sizes),&#10;or measures of computation (e.g., FLOPS, RTF) that reflect the model's ability&#10;to process input frames are not always strongly correlated with observed UPL.&#10;Thus, conventional algorithmic latency measurements might be inadequate in&#10;accurately capturing latency observed when models are deployed on embedded&#10;devices. Instead, we find that factors affecting token emission latency, and&#10;endpointing behavior have a larger impact on UPL. We achieve the best trade-off&#10;between latency and word error rate when performing ASR jointly with&#10;endpointing, while utilizing the recently proposed alignment regularization&#10;mechanism.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.03284" label="2211.03284">
        <attvalues>
          <attvalue for="0" value="Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying&#10;  Peak-First Regularization" />
          <attvalue for="1" value="  The CTC model has been widely applied to many application scenarios because&#10;of its simple structure, excellent performance, and fast inference speed. There&#10;are many peaks in the probability distribution predicted by the CTC models, and&#10;each peak represents a non-blank token. The recognition latency of CTC models&#10;can be reduced by encouraging the model to predict peaks earlier. Existing&#10;methods to reduce latency require modifying the transition relationship between&#10;tokens in the forward-backward algorithm, and the gradient calculation. Some of&#10;these methods even depend on the forced alignment results provided by other&#10;pretrained models. The above methods are complex to implement. To reduce the&#10;peak latency, we propose a simple and novel method named peak-first&#10;regularization, which utilizes a frame-wise knowledge distillation function to&#10;force the probability distribution of the CTC model to shift left along the&#10;time axis instead of directly modifying the calculation process of CTC loss and&#10;gradients. All the experiments are conducted on a Chinese Mandarin dataset&#10;AISHELL-1. We have verified the effectiveness of the proposed regularization on&#10;both streaming and non-streaming CTC models respectively. The results show that&#10;the proposed method can reduce the average peak latency by about 100 to 200&#10;milliseconds with almost no degradation of recognition accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.00522" label="2211.00522">
        <attvalues>
          <attvalue for="0" value="TrimTail: Low-Latency Streaming ASR with Simple but Effective&#10;  Spectrogram-Level Length Penalty" />
          <attvalue for="1" value="  In this paper, we present TrimTail, a simple but effective emission&#10;regularization method to improve the latency of streaming ASR models. The core&#10;idea of TrimTail is to apply length penalty (i.e., by trimming trailing frames,&#10;see Fig. 1-(b)) directly on the spectrogram of input utterances, which does not&#10;require any alignment. We demonstrate that TrimTail is computationally cheap&#10;and can be applied online and optimized with any training loss or any model&#10;architecture on any dataset without any extra effort by applying it on various&#10;end-to-end streaming ASR networks either trained with CTC loss [1] or&#10;Transducer loss [2]. We achieve 100 $\sim$ 200ms latency reduction with equal&#10;or even better accuracy on both Aishell-1 and Librispeech. Moreover, by using&#10;TrimTail, we can achieve a 400ms algorithmic improvement of User Sensitive&#10;Delay (USD) with an accuracy loss of less than 0.2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.06190" label="2306.06190">
        <attvalues>
          <attvalue for="0" value="$FastDoc$: Domain-Specific Fast Pre-training Technique using&#10;  Document-Level Metadata and Taxonomy" />
          <attvalue for="1" value="  As the demand for sophisticated Natural Language Processing (NLP) models&#10;continues to grow, so does the need for efficient pre-training techniques.&#10;Current NLP models undergo resource-intensive pre-training. In response, we&#10;introduce $FastDoc$ (Fast Pre-training Technique using Document-Level Metadata&#10;and Taxonomy), a novel approach designed to significantly reduce computational&#10;demands. $FastDoc$ leverages document metadata and domain-specific taxonomy as&#10;supervision signals. It involves continual pre-training of an open-domain&#10;transformer encoder using sentence-level embeddings, followed by fine-tuning&#10;using token-level embeddings. We evaluate $FastDoc$ on six tasks across nine&#10;datasets spanning three distinct domains. Remarkably, $FastDoc$ achieves&#10;remarkable compute reductions of approximately 1,000x, 4,500x, 500x compared to&#10;competitive approaches in Customer Support, Scientific, and Legal domains,&#10;respectively. Importantly, these efficiency gains do not compromise performance&#10;relative to competitive baselines. Furthermore, reduced pre-training data&#10;mitigates catastrophic forgetting, ensuring consistent performance in&#10;open-domain scenarios. $FastDoc$ offers a promising solution for&#10;resource-efficient pre-training, with potential applications spanning various&#10;domains.&#10;" />
          <attvalue for="2" value="&#10;&#10;In present times, continual pre-training \cite{empmultidomain, dontstop} on unlabelled, domain-specific text corpora (such as PubMed articles in medical domain, research papers in Scientific Domain, E-Manuals in Customer Support Domain, etc.) has emerged as an important training strategy in NLP to enable open-domain transformer-based language models perform various downstream NLP tasks such as Question Answering (QA), Named Entity Recognition (NER), Natural Language Inference (NLI), etc. on domain-specific datasets \cite{cuad, scibert, nandy-etal-2021-question-answering}. Most of the pre-training strategies involve variants of Masked Language Modelling (MLM) \cite{roberta}, Next Sentence Prediction (NSP) \cite{bert}, Sentence Order Prediction (SOP) \cite{albert}, etc. that use local sentence/span-level contexts as supervision signals. However, such methods require a lot of pre-training data and compute. For instance - pre-training of \bert architecture on a $3.17$ billion word corpus was performed on $8$ GPUs for around $40$ days to obtain \scibert ~\cite{scibert}.&#10;&#10;MLM-style domain-specific pre-training makes an implicit assumption that the constituent documents are independent of each other, which may not be true always. Documents from a particular domain (e.g., customer support, scientific papers, legal proceedings, etc.) may be categorized into different groups by experts in that area, each group containing similar documents. This information is generally stored as either `metadata' of the document \cite{metadata1, metadata2, metadata3}, or in terms of a `taxonomy' \cite{taxonomy1, taxonomy2} of documents.&#10;For example, E-manuals of different versions of a cell phone series are very similar, &#10;scientific articles written on a particular topic (e.g., pre-training) follow a certain type of taxonomy, legal proceedings on related crimes are similar. &#10;While few models such as LinkBERT \cite{linkbert}, MetricBERT \cite{metricbert}, etc. have used document metadata as an additional signal, no work to the best of our knowledge has singularly leveraged taxonomy-based information.&#10;&#10;Contrarily, in this paper, we completely replace the local context-based supervision (MLM, NSP, etc.) during pre-training with (a). \ul{document similarity} learning task using the available domain-specific metadata (through a triplet network), and (b). \ul{hierarchical classification} task that predicts the hierarchical categories corresponding to the domain-specific taxonomy in a supervised manner. &#10;&#10;However, to leverage document-level supervision, a robust encoding of documents is required. We use a hierarchical architecture \cite{hibert} and propose various innovations (see Figure \ref{fig:pipeline}) - (a). We initialize the lower-level encoder using a pre-trained sentence transformer (sBERT/sRoBERTa \cite{sbert}) and freeze its weights. We then initialize the higher-level encoder using pre-trained BERT/RoBERTa encoder, which now operates with a sentence embedding input, received via the lower-level encoder. This design choice (inspired by works that initialize a larger encoder through a smaller pre-trained encoder - e.g., Bert2BERT \cite{bert2bert}) helps us to directly work with sentence embeddings as inputs which in turn enables much larger contexts in a single input, and decreases the required pre-training compute by a huge margin. (b). After pre-training,&#10;we use \ul{only the higher-level encoder} for downstream sentence and token-level tasks. As the higher-level encoder was originally pre-trained with token embedding inputs, it can still be fine-tuned with token embedding inputs. We conduct various experiments to analyze this very interesting and surprising aspect of interoperability of token and sentence embedding inputs.&#10;&#10;Using these ideas, we propose \fpdm\&#10;pre-training framework, and apply it to varied NLP tasks across three disparate domains - Customer Support, Scientific Papers, and Legal Domain, to evaluate the generalizability of \fpdm\ across multiple domains. &#10;Customer Support requires answering consumer queries related to device maintenance, troubleshooting, etc., and hence, we apply \fpdm\ on two Question Answering tasks. In the domain of scientific papers, we focus on tasks such as extracting important scientific keywords \cite{bc5cdr,jnlpba,ncbi}, extracting the type of relation between such keywords &#10;\cite{chemprot, scierc}, as well as classifying citation intents \cite{scicite}. In the legal domain, we focus on the task of automating contract review &#10;\cite{cuad}, which involves finding key clauses in legal contracts.&#10;&#10;{We show that \fpdm~\ul{drastically reduces} (order of 500x) pre-training compute across domains &#10;while still achieving comparable to modestly better performance in downstream tasks. &#10;We further show that the result holds even when we increase model size and consider situations where document metadata and taxonomy may not be explicitly available. &#10;We also show that the frugal pre-training helps \fpdm\ resist catastrophic forgetting so very common when transformers undergo continual in-domain pre-training \cite{dontstop,empmultidomain}.}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Resource Optimization, Computer Science, Linguistics, Efficient Pre-training, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2103.06268" label="2103.06268">
        <attvalues>
          <attvalue for="0" value="CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review" />
          <attvalue for="1" value="  Many specialized domains remain untouched by deep learning, as large labeled&#10;datasets require expensive expert annotators. We address this bottleneck within&#10;the legal domain by introducing the Contract Understanding Atticus Dataset&#10;(CUAD), a new dataset for legal contract review. CUAD was created with dozens&#10;of legal experts from The Atticus Project and consists of over 13,000&#10;annotations. The task is to highlight salient portions of a contract that are&#10;important for a human to review. We find that Transformer models have nascent&#10;performance, but that this performance is strongly influenced by model design&#10;and training dataset size. Despite these promising results, there is still&#10;substantial room for improvement. As one of the only large, specialized NLP&#10;benchmarks annotated by experts, CUAD can serve as a challenging research&#10;benchmark for the broader NLP community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.06400" label="2007.06400">
        <attvalues>
          <attvalue for="0" value="GGPONC: A Corpus of German Medical Text with Rich Metadata Based on&#10;  Clinical Practice Guidelines" />
          <attvalue for="1" value="  The lack of publicly accessible text corpora is a major obstacle for progress&#10;in natural language processing. For medical applications, unfortunately, all&#10;language communities other than English are low-resourced. In this work, we&#10;present GGPONC (German Guideline Program in Oncology NLP Corpus), a freely&#10;distributable German language corpus based on clinical practice guidelines for&#10;oncology. This corpus is one of the largest ever built from German medical&#10;documents. Unlike clinical documents, clinical guidelines do not contain any&#10;patient-related information and can therefore be used without data protection&#10;restrictions. Moreover, GGPONC is the first corpus for the German language&#10;covering diverse conditions in a large medical subfield and provides a variety&#10;of metadata, such as literature references and evidence levels. By applying and&#10;evaluating existing medical information extraction pipelines for German text,&#10;we are able to draw comparisons for the use of medical language to other&#10;corpora, medical and non-medical ones.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15827" label="2203.15827">
        <attvalues>
          <attvalue for="0" value="LinkBERT: Pretraining Language Models with Document Links" />
          <attvalue for="1" value="  Language model (LM) pretraining can learn various knowledge from text&#10;corpora, helping downstream tasks. However, existing methods such as BERT model&#10;a single document, and do not capture dependencies or knowledge that span&#10;across documents. In this work, we propose LinkBERT, an LM pretraining method&#10;that leverages links between documents, e.g., hyperlinks. Given a text corpus,&#10;we view it as a graph of documents and create LM inputs by placing linked&#10;documents in the same context. We then pretrain the LM with two joint&#10;self-supervised objectives: masked language modeling and our new proposal,&#10;document relation prediction. We show that LinkBERT outperforms BERT on various&#10;downstream tasks across two domains: the general domain (pretrained on&#10;Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with&#10;citation links). LinkBERT is especially effective for multi-hop reasoning and&#10;few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our&#10;biomedical LinkBERT sets new states of the art on various BioNLP tasks (+7% on&#10;BioASQ and USMLE). We release our pretrained models, LinkBERT and BioLinkBERT,&#10;as well as code and data at https://github.com/michiyasunaga/LinkBERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.06566" label="1905.06566">
        <attvalues>
          <attvalue for="0" value="HIBERT: Document Level Pre-training of Hierarchical Bidirectional&#10;  Transformers for Document Summarization" />
          <attvalue for="1" value="  Neural extractive summarization models usually employ a hierarchical encoder&#10;for document encoding and they are trained using sentence-level labels, which&#10;are created heuristically using rule-based methods. Training the hierarchical&#10;encoder with these \emph{inaccurate} labels is challenging. Inspired by the&#10;recent work on pre-training transformer sentence encoders&#10;\cite{devlin:2018:arxiv}, we propose {\sc Hibert} (as shorthand for {\bf&#10;HI}erachical {\bf B}idirectional {\bf E}ncoder {\bf R}epresentations from {\bf&#10;T}ransformers) for document encoding and a method to pre-train it using&#10;unlabeled data. We apply the pre-trained {\sc Hibert} to our summarization&#10;model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on&#10;the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times&#10;dataset. We also achieve the state-of-the-art performance on these two&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.01608" label="1904.01608">
        <attvalues>
          <attvalue for="0" value="Structural Scaffolds for Citation Intent Classification in Scientific&#10;  Publications" />
          <attvalue for="1" value="  Identifying the intent of a citation in scientific papers (e.g., background&#10;information, use of methods, comparing results) is critical for machine reading&#10;of individual publications and automated analysis of the scientific literature.&#10;We propose structural scaffolds, a multitask model to incorporate structural&#10;information of scientific papers into citations for effective classification of&#10;citation intents. Our model achieves a new state-of-the-art on an existing ACL&#10;anthology dataset (ACL-ARC) with a 13.3% absolute increase in F1 score, without&#10;relying on external linguistic resources or hand-engineered features as done in&#10;existing methods. In addition, we introduce a new dataset of citation intents&#10;(SciCite) which is more than five times larger and covers multiple scientific&#10;domains compared with existing datasets. Our code and data are available at:&#10;https://github.com/allenai/scicite.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.10084" label="2106.10084">
        <attvalues>
          <attvalue for="0" value="Subjective Bias in Abstractive Summarization" />
          <attvalue for="1" value="  Due to the subjectivity of the summarization, it is a good practice to have&#10;more than one gold summary for each training document. However, many modern&#10;large-scale abstractive summarization datasets have only one-to-one samples&#10;written by different human with different styles. The impact of this phenomenon&#10;is understudied. We formulate the differences among possible multiple&#10;expressions summarizing the same content as subjective bias and examine the&#10;role of this bias in the context of abstractive summarization. In this paper a&#10;lightweight and effective method to extract the feature embeddings of&#10;subjective styles is proposed. Results of summarization models trained on&#10;style-clustered datasets show that there are certain types of styles that lead&#10;to better convergence, abstraction and generalization. The reproducible code&#10;and generated summaries are available online.&#10;" />
          <attvalue for="2" value="&#10;Given a verbose input article, abstractive summarization aims at generating summaries covering its key facts. The base architecture to solve this problem is the attention-based encoder-decoder~\cite{rush2015neural} which greatly improved the result of neural translation~\cite{bahdanau2014neural}. Former studies proposed various models to better understand document~\cite{nallapati2016abstractive}, handle the out-of-vocabulary(OOV) problem~\cite{see2017get}, reduce the repetition\cite{chen2016distraction-based,li2019in} or divided the summarization problem into two steps(select and rewrite)~\cite{moroshko2019an,chen2018fast}. Recent researches also introduced Pretrained Language Model(PLM)~\cite{liu2019text,lewis2019bart:,raffel2019exploring} to this task. In our opinion, the abstractive summarization task is not only about identifying the key content of articles but also about Natural Language Generation(NLG) for summaries. Even the encoder caught and encoded the proper part of an article, the generated summaries may still be various depending on different human-written gold summaries. The summaries written by human are subjective and are therefore susceptible to recall bias. Generation bias in the dataset brought by human annotators should matter in the abstractive summarization task.&#10;&#10;To study this problem, we hypothesize that there exists some subjective style bias among different samples. We define the writing style of human annotators formulating summaries after they have read and captured the main idea of articles as &quot;Subjective Style&quot;. Our results on the most used CNN-Daily Mail(CNN-DM) dataset~\cite{hermann2015teaching} show that different style has a different impact on the model adaption, convergence speed, readability, and abstraction of generated summaries. In particular, this paper makes several contributions as follows:&#10;\begin{itemize}&#10;&#10;\item The hypothesis about subjective bias among different samples in datasets is proposed, studied, and verified in detail for the first time with regard to abstractive summarization tasks on the CNN-DM dataset and its influence on the quality of the NLG process for summary generation is also evaluated. &#10;\item There are few related works about embedding writing style in the sequence-to-sequence(seq2seq) task in which subjective style can be seen as a special case. We propose to use the graph structure to represent the syntactic information in texts and put forward a self-supervised task to extract and embed the subjective style utilizing Graph Convolutional Network (GCN). &#10;\item Experimental results on style-clustered datasets confirm our assumption. Combining all styles in a dataset may not be the best practice for training abstractive summarization models.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Subjective Bias Analysis, Abstractive Summarization, Style Embedding Extraction, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1610.08462" label="1610.08462">
        <attvalues>
          <attvalue for="0" value="Distraction-Based Neural Networks for Document Summarization" />
          <attvalue for="1" value="  Distributed representation learned with neural networks has recently shown to&#10;be effective in modeling natural languages at fine granularities such as words,&#10;phrases, and even sentences. Whether and how such an approach can be extended&#10;to help model larger spans of text, e.g., documents, is intriguing, and further&#10;investigation would still be desirable. This paper aims to enhance neural&#10;network models for such a purpose. A typical problem of document-level modeling&#10;is automatic summarization, which aims to model documents in order to generate&#10;summaries. In this paper, we propose neural models to train computers not just&#10;to pay attention to specific regions and content of input documents with&#10;attention models, but also distract them to traverse between different content&#10;of a document so as to better grasp the overall meaning for summarization.&#10;Without engineering any features, we train the models on two large datasets.&#10;The models achieve the state-of-the-art performance, and they significantly&#10;benefit from the distraction modeling, particularly when input documents are&#10;long.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.10852" label="1909.10852">
        <attvalues>
          <attvalue for="0" value="In Conclusion Not Repetition: Comprehensive Abstractive Summarization&#10;  With Diversified Attention Based On Determinantal Point Processes" />
          <attvalue for="1" value="  Various Seq2Seq learning models designed for machine translation were applied&#10;for abstractive summarization task recently. Despite these models provide high&#10;ROUGE scores, they are limited to generate comprehensive summaries with a high&#10;level of abstraction due to its degenerated attention distribution. We&#10;introduce Diverse Convolutional Seq2Seq Model(DivCNN Seq2Seq) using&#10;Determinantal Point Processes methods(Micro DPPs and Macro DPPs) to produce&#10;attention distribution considering both quality and diversity. Without breaking&#10;the end to end architecture, DivCNN Seq2Seq achieves a higher level of&#10;comprehensiveness compared to vanilla models and strong baselines. All the&#10;reproducible codes and datasets are available online.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.10360" label="1902.10360">
        <attvalues>
          <attvalue for="0" value="An Editorial Network for Enhanced Document Summarization" />
          <attvalue for="1" value="  We suggest a new idea of Editorial Network - a mixed extractive-abstractive&#10;summarization approach, which is applied as a post-processing step over a given&#10;sequence of extracted sentences. Our network tries to imitate the decision&#10;process of a human editor during summarization. Within such a process, each&#10;extracted sentence may be either kept untouched, rephrased or completely&#10;rejected. We further suggest an effective way for training the &quot;editor&quot; based&#10;on a novel soft-labeling approach. Using the CNN/DailyMail dataset we&#10;demonstrate the effectiveness of our approach compared to state-of-the-art&#10;extractive-only or abstractive-only baseline methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15093" label="2305.15093">
        <attvalues>
          <attvalue for="0" value="C-STS: Conditional Semantic Textual Similarity" />
          <attvalue for="1" value="  Semantic textual similarity (STS), a cornerstone task in NLP, measures the&#10;degree of similarity between a pair of sentences, and has broad application in&#10;fields such as information retrieval and natural language understanding.&#10;However, sentence similarity can be inherently ambiguous, depending on the&#10;specific aspect of interest. We resolve this ambiguity by proposing a novel&#10;task called Conditional STS (C-STS) which measures sentences' similarity&#10;conditioned on an feature described in natural language (hereon, condition). As&#10;an example, the similarity between the sentences &quot;The NBA player shoots a&#10;three-pointer.&quot; and &quot;A man throws a tennis ball into the air to serve.&quot; is&#10;higher for the condition &quot;The motion of the ball&quot; (both upward) and lower for&#10;&quot;The size of the ball&quot; (one large and one small). C-STS's advantages are&#10;two-fold: (1) it reduces the subjectivity and ambiguity of STS and (2) enables&#10;fine-grained language model evaluation through diverse natural language&#10;conditions. We put several state-of-the-art models to the test, and even those&#10;performing well on STS (e.g. SimCSE, Flan-T5, and GPT-4) find C-STS&#10;challenging; all with Spearman correlation scores below 50. To encourage a more&#10;comprehensive evaluation of semantic similarity and natural language&#10;understanding, we make nearly 19K C-STS examples and code available for others&#10;to train and test their models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Historical perspectives of semantic similarities&#10;Measuring semantic similarities is a long-standing problem spanning cognitive science \cite{Miller1991Correlates} to psychology \cite{Tversky1977FeaturesOS} where early attempts are made to quantify the subjective similarity judgements with information theoretical concepts. More recently, interest in semantic similarity has gained popularity in the context of machine learning, with works in computer vision recognizing that the notion of similarity between images varies with conditions \cite{veit_conditional_2017} and can therefore be ambiguous \cite{ye_identifying_2022}.&#10;&#10;Textual similarity tasks&#10;Capturing textual similarity is also considered a fundamental problem in natural language processing. Works such as \cite{agirre_semeval-2012, agirre_semeval-2016_2016} define the textual semantic similarity tasks (STS), which is widely used in common benchmarks such as GLUE \cite{wang2018glue}. Extensions to the STS setting have been proposed such as making the task broader with multilinguality \cite{cer_semeval-2017_2017} or incorporating relatedness \cite{abdalla_what_2021}. However, the loose definition of similarity has not been acknowledged as an issue explicitly. In contrast, our work tackles the ambiguity problem by collecting conditions and hence reduce subjectivity. To alleviate ambiguity, explanations play an important role in identifying the differences between the two sentences either in their syntactical structure \cite{lopez-gazpio_interpretable_2017} or in natural language \cite{camburu2018esnli}, but the post-hoc nature of explanations prevents it from being used prior to the similarity judgement, rendering it a supplemental component as opposed to a paradigm change in the task setup. Beyond STS, works that leverage conditioning to enhance sentence representations obtain improved performance for retrieval \cite{asai_task-aware_2022} and embedding qualities \cite{he_multi-perspective_2015, su_one_2022, jiang_promptbert_2022}, which corroborates the observation that conditioning as a form of disambiguation benefits similarity measures.&#10; " />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Natural Language Processing, Semantic Textual Similarity" />
        </attvalues>
      </node>
      <node id="2310.01260" label="2310.01260">
        <attvalues>
          <attvalue for="0" value="SPELL: Semantic Prompt Evolution based on a LLM" />
          <attvalue for="1" value="  Prompt engineering is a new paradigm for enhancing the performance of trained&#10;neural network models. For optimizing text-style prompts, existing methods&#10;usually individually operate small portions of a text step by step, which&#10;either breaks the fluency or could not globally adjust a prompt. Since large&#10;language models (LLMs) have powerful ability of generating coherent texts token&#10;by token, can we utilize LLMs for improving prompts? Based on this motivation,&#10;in this paper, considering a trained LLM as a text generator, we attempt to&#10;design a black-box evolution algorithm for automatically optimizing texts,&#10;namely SPELL (Semantic Prompt Evolution based on a LLM). The proposed method is&#10;evaluated with different LLMs and evolution parameters in different text tasks.&#10;Experimental results show that SPELL could rapidly improve the prompts indeed.&#10;We further explore the evolution process and discuss on the limitations,&#10;potential possibilities and future work.&#10;" />
          <attvalue for="2" value="&#10;&#10;Prompts (\cite{liu2023pre}) are additional contents which would be integrated with the original input data to enhance the performance of a trained target model. For example, in a text classification task, a prompt, &quot;[input text]\textbackslash nDivide the above text into 2 classes: positive and negative.\textbackslash nClass:[MASK]&quot;, may strengthen the performance of a language model because the prompt reduces the probabilities of many unrelated tokens and hints what the model should focus on. Good prompt engineering methods are able to enhance the ability of a target model with limited and efficient optimization. Besides, prompts are flexible. They could be continuous (\cite{li-liang-2021-prefix}; \cite{qin-eisner-2021-learning}), possibly work in hidden layers (\cite{sun-etal-2022-bbtv2}), and have already been applied to many modalities, such as text (\cite{lester-etal-2021-power}; \cite{liu-etal-2022-p}), vision (\cite{jia2022visual}; \cite{Sohn_2023_CVPR}), and graph (\cite{10.1145/3534678.3539249}; \cite{yi2023contrastive}). In this paper, we only focus on discrete text-style prompts applied at the input of a language model for text tasks.&#10;&#10;There are many methods to optimize text-style prompts. Gradient-based methods (\cite{wallace-etal-2019-universal}; \cite{shin-etal-2020-autoprompt}) are the most intuitive one. They usually relax the discrete embeddings for optimization and need large computation because of the process of backward through the trained model. Nowadays, many target models are considered as black boxes since only inference APIs are provided for the users. Hence, many black-box methods, based on evolution (\cite{pmlr-v162-sun22e}; \cite{sun-etal-2022-bbtv2}), reinforcement learning (\cite{deng-etal-2022-rlprompt}; \cite{zhang2023tempera}; \cite{diao2023blackbox}) or search strategies (\cite{prasad-etal-2023-grips}), improve the prompts without the information of gradients and parameters inside the target model. In particular, evolution-based methods maintain a population consists of several prompts as individuals and optimize the population by biological mechanisms such as reproduction and selection. However, these methods individually operate small portions of text step by step, leading to lack of global adjustment and serious damage on the fluency. The prompts optimized by these methods usually consist of various strange characters, such as &quot;an-lione\_-lo-:)eak&quot;.&#10;&#10;In recent years, large language models (LLMs) (\cite{ijcai2021p0612}; \cite{zhao2023survey}) are the most popular star among various new AI applications. Optimized on language modeling, they could get universal generalization ability. With appropriate instructions, they could acquire amazing performance on language understanding, reasoning and text generation. Especially, the generated texts are coherent enough.&#10;&#10;Therefore, we attempt to propose SPELL (Semantic Prompt Evolution based on a LLM), an evolution algorithm to optimize prompts. Since the proposed method considers a LLM as a generator, SPELL would utilize the global feature of a prompt and the improved prompts are coherent. Through experiments, we evaluate that SPELL could rapidly improve performance keeping prompts fluent. Besides, we discover the evolution process and the impacts of different LLMs or evolution strategies. The limitations, challenges and possible directions are discussed on in the end.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Generation Improvement, Computer Science, Linguistics, Cognitive Science, Language Model Enhancement, Neural Network Optimization, Mathematics" />
        </attvalues>
      </node>
      <node id="2205.11200" label="2205.11200">
        <attvalues>
          <attvalue for="0" value="BBTv2: Towards a Gradient-Free Future with Large Language Models" />
          <attvalue for="1" value="  Most downstream adaptation methods tune all or part of the parameters of&#10;pre-trained models (PTMs) through gradient descent, where the tuning cost&#10;increases linearly with the growth of the model size. By contrast,&#10;gradient-free methods only require the forward computation of the PTM to tune&#10;the prompt, retaining the benefits of efficient tuning and deployment. Though,&#10;past work on gradient-free tuning often introduces gradient descent to seek a&#10;good initialization of prompt and lacks versatility across tasks and PTMs. In&#10;this paper, we present BBTv2, an improved version of Black-Box Tuning, to drive&#10;PTMs for few-shot learning. We prepend continuous prompts to every layer of the&#10;PTM and propose a divide-and-conquer gradient-free algorithm to optimize the&#10;prompts at different layers alternately. Extensive experiments across various&#10;tasks and PTMs show that BBTv2 can achieve comparable performance to full model&#10;tuning and state-of-the-art parameter-efficient methods (e.g., Adapter, LoRA,&#10;BitFit, etc.) under few-shot settings while maintaining much fewer tunable&#10;parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.00990" label="2210.00990">
        <attvalues>
          <attvalue for="0" value="Visual Prompt Tuning for Generative Transfer Learning" />
          <attvalue for="1" value="  Transferring knowledge from an image synthesis model trained on a large&#10;dataset is a promising direction for learning generative image models from&#10;various domains efficiently. While previous works have studied GAN models, we&#10;present a recipe for learning vision transformers by generative knowledge&#10;transfer. We base our framework on state-of-the-art generative vision&#10;transformers that represent an image as a sequence of visual tokens to the&#10;autoregressive or non-autoregressive transformers. To adapt to a new domain, we&#10;employ prompt tuning, which prepends learnable tokens called prompt to the&#10;image token sequence, and introduce a new prompt design for our task. We study&#10;on a variety of visual domains, including visual task adaptation&#10;benchmark~\cite{zhai2019large}, with varying amount of training images, and&#10;show effectiveness of knowledge transfer and a significantly better image&#10;generation quality over existing works.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.17035" label="2210.17035">
        <attvalues>
          <attvalue for="0" value="Evaluation of large-scale synthetic data for Grammar Error Correction" />
          <attvalue for="1" value="  Grammar Error Correction(GEC) mainly relies on the availability of high&#10;quality of large amount of synthetic parallel data of grammatically correct and&#10;erroneous sentence pairs. The quality of the synthetic data is evaluated on how&#10;well the GEC system performs when pre-trained using it. But this does not&#10;provide much insight into what are the necessary factors which define the&#10;quality of these data. So this work aims to introduce 3 metrics - reliability,&#10;diversity and distribution match to provide more insight into the quality of&#10;large-scale synthetic data generated for the GEC task, as well as automatically&#10;evaluate them. Evaluating these three metrics automatically can also help in&#10;providing feedback to the data generation systems and thereby improve the&#10;quality of the synthetic data generated dynamically&#10;" />
          <attvalue for="2" value="&#10;Grammar error correction systems focus on detecting possible grammar errors like spelling errors, punctuation errors, verb errors etc and correcting them. Recent success in GEC systems depends largely on the availability of a large amount(in the scale of several millions) of parallel data consisting of grammatically correct and incorrect sentences. Some examples of this include:&#10;\begin{itemize}&#10; \item \cite{stahlberg-kumar-2021-synthetic} use error type tags from ERRANT\cite{bryant-etal-2017-automatic} to guide the synthetic data distribution so that the distribution of the synthetic data can be similar to that of the domain in which the GEC system is used. This synthetic dataset consists of 200 million parallel sentences and achieves near state-of-the-art results on the CoNLL-14\cite{ng-etal-2014-conll} dataset&#10;\end{itemize} &#10;\begin{itemize}&#10; \item \cite{awasthi-etal-2019-parallel} introduces the rule-based synthetic data generation method for GEC by using a list of common replacement errors, common insertion errors, common deletion errors and verb errors, and applying these errors randomly to any part of the grammatically correct sentence. Using this method, a very large synthetic dataset can be easily created for better GEC results&#10;\end{itemize} &#10;\begin{itemize}&#10; \item \cite{koyama-etal-2021-comparison} compares the results of using the synthetic data generated using different backtranslation models such as Transformers, CNN and LSTM models. Here, a pseudo data of nearly 9 million samples are created using each of the above models and the quality of the synthetic data is compared by how well the GEC model trained using these synthetic datasets, perform on test sets like BEA-test\cite{bryant-etal-2019-bea} etc.&#10;\end{itemize} &#10;As mentioned previously, all these synthetic datasets are evaluated based on how the GEC models perform when trained using them. However, this does not capture essential metrics which can be used to improve the synthetic data generation. This work aims to introduce these metrics in terms of :&#10;\begin{itemize}&#10; \item Reliability - This metric can help evaluate which part of the dataset truly resembles humans in terms of creating grammatical errors in a particular sentence&#10;\end{itemize}&#10;\begin{itemize}&#10; \item Diversity - Many synthetic data generation techniques like \cite{stahlberg-kumar-2021-synthetic} and \cite{zhou-etal-2020-improving-grammatical} show that diverse synthetic data can help improve the GEC systems. But these works do not accurately measure the diversity of the datasets. So, this work aims to evaluate the diversity of any synthetic dataset more accurately.&#10;\end{itemize}&#10;\begin{itemize}&#10; \item Distribution match - This metric is useful in determining which synthetic dataset to be used to train the GEC model for a particular domain. In this work, the experiments are done using datasets across different domains like the open domain and the novel domain to determine the accuracy of the metric&#10;\end{itemize}&#10;&#10;For the rest of the discussion, the experiments of the proposed metrics are conducted on the following synthetic datasets:&#10;\begin{itemize}&#10; \item Tagged corruption model - \cite{stahlberg-kumar-2021-synthetic}&#10;\end{itemize}&#10;\begin{itemize}&#10; \item Backtranslation method - \cite{xie-etal-2018-noising}&#10;\end{itemize}&#10;\begin{itemize}&#10; \item Rule-based method - \cite{awasthi-etal-2019-parallel}&#10;\end{itemize}&#10;&#10;The backtranslation method is applied on the novel domain to verify if the metrics still holds for different domains other than the open domain.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Synthetic Data Generation, Machine Learning, Computer Science, Grammar Error Correction, Linguistics, Data Quality Evaluation, Computational Linguistics, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1911.02825" label="1911.02825">
        <attvalues>
          <attvalue for="0" value="Improving Grammatical Error Correction with Machine Translation Pairs" />
          <attvalue for="1" value="  We propose a novel data synthesis method to generate diverse error-corrected&#10;sentence pairs for improving grammatical error correction, which is based on a&#10;pair of machine translation models of different qualities (i.e., poor and&#10;good). The poor translation model resembles the ESL (English as a second&#10;language) learner and tends to generate translations of low quality in terms of&#10;fluency and grammatical correctness, while the good translation model generally&#10;generates fluent and grammatically correct translations. We build the poor and&#10;good translation model with phrase-based statistical machine translation model&#10;with decreased language model weight and neural machine translation model&#10;respectively. By taking the pair of their translations of the same sentences in&#10;a bridge language as error-corrected sentence pairs, we can construct unlimited&#10;pseudo parallel data. Our approach is capable of generating diverse&#10;fluency-improving patterns without being limited by the pre-defined rule set&#10;and the seed error-corrected data. Experimental results demonstrate the&#10;effectiveness of our approach and show that it can be combined with other&#10;synthetic data sources to yield further improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.03087" label="2304.03087">
        <attvalues>
          <attvalue for="0" value="Investigating Chain-of-thought with ChatGPT for Stance Detection on&#10;  Social Media" />
          <attvalue for="1" value="  Stance detection predicts attitudes towards targets in texts and has gained&#10;attention with the rise of social media. Traditional approaches include&#10;conventional machine learning, early deep neural networks, and pre-trained&#10;fine-tuning models. However, with the evolution of very large pre-trained&#10;language models (VLPLMs) like ChatGPT (GPT-3.5), traditional methods face&#10;deployment challenges. The parameter-free Chain-of-Thought (CoT) approach, not&#10;requiring backpropagation training, has emerged as a promising alternative.&#10;This paper examines CoT's effectiveness in stance detection tasks,&#10;demonstrating its superior accuracy and discussing associated challenges.&#10;" />
          <attvalue for="2" value="&#10;Stance detection on social media is an important topic in research communities of both natural language processing (NLP) and social computing \cite{kuccuk2020stance, li2023stance}. &#10;The goal of stance detection is to automatically predict the attitude (i.e., favor, against, or neutral) of opinionated tweets (text) with a specified target. &#10;&#10;Early research works on stance detection mainly adopted rule-based and traditional machine learning techniques \cite{anand2011cats, walker2012stance}. &#10;For example, the effective algorithms for the classifiers are support vector machine (SVM), logistic regression, naive bayes, decision tree and etc \cite{aldayel2021stance}.&#10;As deep learning methods evolved, deep neural networks (DNNs) rapidly became the prevailing techniques for stance detection. These methods employ neural networks with varying structures and connections to develop the desired stance classifier, which can be classified into conventional DNN models, attention-based DNN models, and graph convolutional network (GCN) models.&#10;Convolutional neural networks (CNNs) and long short-term memory (LSTM) models are among the most commonly used conventional DNN models \cite{augenstein2016stance, &#10; jiang2019hierarchical}; attention-based methods primarily leverage target-specific information as the attention query and implement an attention mechanism for inferring stance polarity \cite{dey2018topical}; and GCN methods propose a graph convolutional network to model the relationship between the target and text \cite{zhang2020enhancing}.&#10;&#10;Subsequently, with the great success of bidirectional encoder representations from transformers (BERT) model, a new NLP research paradigm emerges which is utilizing large pre-trained language models (PLM) with a fine-tuning (or prompt-tuning) process. &#10;Fine-tuning models adapt PLMs by building a stance classification head on top of the ``{$&lt;$cls$&gt;$}'' token, and fine-tune the whole model \cite{wei2022emergent}. &#10;This pre-train and fine tuning paradigm provides exceptional performance for most NLP downstream tasks including stance detection, because the abundance of training data enables PLMs to learn enough general-purpose features and knowledge for modeling different languages. &#10;The main idea of prompt-based methods is mimicking PLMs to design a template suitable for classification tasks and then build a mapping (called verbalizer) from the predicted token to the classification labels to perform class prediction, which bridges a projection between the vocabulary and the label space. The prompting strategies provide further improvements for stance detection performance\cite{shin2020autoprompt}. &#10;&#10;As the number of model parameters continues to grow, language model performance has experienced consistent advancements. The advent of very large pre-trained language models (VLPLMs), such as ChatGPT, which is built upon the GPT series architecture, has revolutionized the progress in various language-related tasks. These models have showcased remarkable efficacy across a multitude of tasks, outperforming existing pre-training and fine-tuning frameworks to achieve state-of-the-art results, even in zero-shot scenarios where no training data is available. Such progress marks a significant breakthrough, highlighting the immense potential for ongoing research and the application of large-scale models in the future.&#10;However, contemporary VLPLMs have already attained a colossal scale, encompassing hundreds of billions of parameters. This renders the cost of pre-training and fine-tuning methods prohibitively expensive. Consequently, this article investigates how to effectively leverage VLPLMs for stance detection tasks, motivated by the growing need to address these challenges.&#10;&#10;In this study, we investigate the efficacy of the chain-of-thought (CoT) prompting strategy when applied to ChatGPT (GPT-3.5) for stance detection tasks. The CoT approach involves utilizing templates as prompts to harness the model's capabilities more effectively \cite{dua2022successive, zhou2022least}. To the best of our knowledge, this constitutes the first research endeavor that employs the CoT technique within the context of stance detection tasks.&#10;Our experimental results indicate that CoT methods can achieve state-of-the-art (SOTA) or comparable performance on widely used datasets, using a straightforward prompt without the need for training. Furthermore, we also identify and discuss certain limitations that may constrain the performance of CoT when utilizing VLPLMs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Language Models, Artificial Intelligence, Stance Detection" />
        </attvalues>
      </node>
      <node id="2006.03644" label="2006.03644">
        <attvalues>
          <attvalue for="0" value="Stance Detection on Social Media: State of the Art and Trends" />
          <attvalue for="1" value="  Stance detection on social media is an emerging opinion mining paradigm for&#10;various social and political applications in which sentiment analysis may be&#10;sub-optimal. There has been a growing research interest for developing&#10;effective methods for stance detection methods varying among multiple&#10;communities including natural language processing, web science, and social&#10;computing. This paper surveys the work on stance detection within those&#10;communities and situates its usage within current opinion mining techniques in&#10;social media. It presents an exhaustive review of stance detection techniques&#10;on social media, including the task definition, different types of targets in&#10;stance detection, features set used, and various machine learning approaches&#10;applied. The survey reports state-of-the-art results on the existing benchmark&#10;datasets on stance detection, and discusses the most effective approaches. In&#10;addition, this study explores the emerging trends and different applications of&#10;stance detection on social media. The study concludes by discussing the gaps in&#10;the current existing research and highlights the possible future directions for&#10;stance detection on social media.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05967" label="2010.05967">
        <attvalues>
          <attvalue for="0" value="The Zero Resource Speech Challenge 2020: Discovering discrete subword&#10;  and word units" />
          <attvalue for="1" value="  We present the Zero Resource Speech Challenge 2020, which aims at learning&#10;speech representations from raw audio signals without any labels. It combines&#10;the data sets and metrics from two previous benchmarks (2017 and 2019) and&#10;features two tasks which tap into two levels of speech representation. The&#10;first task is to discover low bit-rate subword representations that optimize&#10;the quality of speech synthesis; the second one is to discover word-like units&#10;from unsegmented raw speech. We present the results of the twenty submitted&#10;models and discuss the implications of the main findings for unsupervised&#10;speech learning.&#10;" />
          <attvalue for="2" value="&#10;&#10;Current speech technology depends heavily on the availability of textual resources. On the other hand, humans learn the sounds and vocabulary of their first language long before they learn to read or write, discovering some kind of linguistic units or representations in their language (typically thought to be phoneme- or word-like), and the equivalent of an acoustic model, a language model, and a speech synthesizer. That humans succeed without textual resources suggests that there may be another approach. Developing technology to learn useful speech representations in an unsupervised way would be useful for the thousands of so-called low-resource languages, which lack the textual resources and/or expertise required to build traditional speech processing systems. &#10;&#10;The Zero Resource Speech Challenge series \cite{versteegh2016,dunbar2017,dunbar2019} aims to push the envelope in unsupervised speech modelling, by taking the radical stance of trying to learn the full speech processing stack without any textual resources. Here, we reopen two previous benchmarks with a focus on discovering discrete representations from raw audio at two linguistic levels. The first focuses on the phonological or sub-word level. The goal is to learn discrete (low bitrate) speech units, which encode meaningful linguistic invariants, and which are useful for doing speech synthesis. This is a reopening of the 2019 ``TTS without T'' ZeroSpeech benchmark \cite{dunbar2019} (track 1b in Figure \ref{fig:diagram-challenge}). The second focuses on the word level. The goal is to discover word-like units for the purpose of segmenting continuous speech. This is a reopening of the 2017 ``spoken term discovery'' ZeroSpeech Benchmark \cite{dunbar2017} (track 2 in Figure \ref{fig:diagram-challenge}). As before, we rely exclusively on freely accessible software and data sets. &#10;&#10;Discrete units, such as words and phonemes, form the basis of every modern speech technology system at some level. One useful feature of discrete representations is that they remove linguistically irrelevant information from the signal, and represent continuous speech in a highly compact format. For example, \cite{DBLP:conf/iclr/BaevskiSA20} perform unsupervised representation learning, and show that, up to a certain point, discrete representations are more useful than continuous ones as the input for training a phone recognizer.&#10;Here, we ask participants to discover their own discrete units and analyze them in terms of how well they capture relevant linguistic contrasts, as indicated by the gold phoneme- and word-level transcriptions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Speech Synthesis Optimization, Speech Representation Learning, Artificial Intelligence, Signal Processing, Unsupervised Learning Methods" />
        </attvalues>
      </node>
      <node id="1911.10438" label="1911.10438">
        <attvalues>
          <attvalue for="0" value="CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations&#10;  with Multi-Task Learning" />
          <attvalue for="1" value="  Joint extraction of entities and relations has received significant attention&#10;due to its potential of providing higher performance for both tasks. Among&#10;existing methods, CopyRE is effective and novel, which uses a&#10;sequence-to-sequence framework and copy mechanism to directly generate the&#10;relation triplets. However, it suffers from two fatal problems. The model is&#10;extremely weak at differing the head and tail entity, resulting in inaccurate&#10;entity extraction. It also cannot predict multi-token entities (e.g.&#10;\textit{Steven Jobs}). To address these problems, we give a detailed analysis&#10;of the reasons behind the inaccurate entity extraction problem, and then&#10;propose a simple but extremely effective model structure to solve this problem.&#10;In addition, we propose a multi-task learning framework equipped with copy&#10;mechanism, called CopyMTL, to allow the model to predict multi-token entities.&#10;Experiments reveal the problems of CopyRE and show that our model achieves&#10;significant improvement over the current state-of-the-art method by 9% in NYT&#10;and 16% in WebNLG (F1 score). Our code is available at&#10;https://github.com/WindChimeRan/CopyMTL&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12812" label="2010.12812">
        <attvalues>
          <attvalue for="0" value="A Frustratingly Easy Approach for Entity and Relation Extraction" />
          <attvalue for="1" value="  End-to-end relation extraction aims to identify named entities and extract&#10;relations between them. Most recent work models these two subtasks jointly,&#10;either by casting them in one structured prediction framework, or performing&#10;multi-task learning through shared representations. In this work, we present a&#10;simple pipelined approach for entity and relation extraction, and establish the&#10;new state-of-the-art on standard benchmarks (ACE04, ACE05 and SciERC),&#10;obtaining a 1.7%-2.8% absolute improvement in relation F1 over previous joint&#10;models with the same pre-trained encoders. Our approach essentially builds on&#10;two independent encoders and merely uses the entity model to construct the&#10;input for the relation model. Through a series of careful examinations, we&#10;validate the importance of learning distinct contextual representations for&#10;entities and relations, fusing entity information early in the relation model,&#10;and incorporating global context. Finally, we also present an efficient&#10;approximation to our approach which requires only one pass of both entity and&#10;relation encoders at inference time, achieving an 8-16$\times$ speedup with a&#10;slight reduction in accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.04361" label="1812.04361">
        <attvalues>
          <attvalue for="0" value="RESIDE: Improving Distantly-Supervised Neural Relation Extraction using&#10;  Side Information" />
          <attvalue for="1" value="  Distantly-supervised Relation Extraction (RE) methods train an extractor by&#10;automatically aligning relation instances in a Knowledge Base (KB) with&#10;unstructured text. In addition to relation instances, KBs often contain other&#10;relevant side information, such as aliases of relations (e.g., founded and&#10;co-founded are aliases for the relation founderOfCompany). RE models usually&#10;ignore such readily available side information. In this paper, we propose&#10;RESIDE, a distantly-supervised neural relation extraction method which utilizes&#10;additional side information from KBs for improved relation extraction. It uses&#10;entity type and relation alias information for imposing soft constraints while&#10;predicting relations. RESIDE employs Graph Convolution Networks (GCN) to encode&#10;syntactic information from text and improves performance even when limited side&#10;information is available. Through extensive experiments on benchmark datasets,&#10;we demonstrate RESIDE's effectiveness. We have made RESIDE's source code&#10;available to encourage reproducible research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.10855" label="2106.10855">
        <attvalues>
          <attvalue for="0" value="CIL: Contrastive Instance Learning Framework for Distantly Supervised&#10;  Relation Extraction" />
          <attvalue for="1" value="  The journey of reducing noise from distant supervision (DS) generated&#10;training data has been started since the DS was first introduced into the&#10;relation extraction (RE) task. For the past decade, researchers apply the&#10;multi-instance learning (MIL) framework to find the most reliable feature from&#10;a bag of sentences. Although the pattern of MIL bags can greatly reduce DS&#10;noise, it fails to represent many other useful sentence features in the&#10;datasets. In many cases, these sentence features can only be acquired by extra&#10;sentence-level human annotation with heavy costs. Therefore, the performance of&#10;distantly supervised RE models is bounded. In this paper, we go beyond typical&#10;MIL framework and propose a novel contrastive instance learning (CIL)&#10;framework. Specifically, we regard the initial MIL as the relational triple&#10;encoder and constraint positive pairs against negative pairs for each instance.&#10;Experiments demonstrate the effectiveness of our proposed framework, with&#10;significant improvements over the previous methods on NYT10, GDS and KBP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.06162" label="2007.06162">
        <attvalues>
          <attvalue for="0" value="Do You Have the Right Scissors? Tailoring Pre-trained Language Models&#10;  via Monte-Carlo Methods" />
          <attvalue for="1" value="  It has been a common approach to pre-train a language model on a large corpus&#10;and fine-tune it on task-specific data. In practice, we observe that&#10;fine-tuning a pre-trained model on a small dataset may lead to over- and/or&#10;under-estimation problem. In this paper, we propose MC-Tailor, a novel method&#10;to alleviate the above issue in text generation tasks by truncating and&#10;transferring the probability mass from over-estimated regions to&#10;under-estimated ones. Experiments on a variety of text generation datasets show&#10;that MC-Tailor consistently and significantly outperforms the fine-tuning&#10;approach. Our code is available at this url.&#10;" />
          <attvalue for="2" value="&#10;Recently, pre-trained language models~(PLM), e.g. GPT-2~\cite{radford2019language}, have shown great promise in many applications of natural language generation, such as stylized text generation~\cite{syed2019adapting} and dialog system~\cite{DBLP:journals/corr/abs-1901-08149}. &#10;PLM is obtained by first pre-training on large-scaled raw sentences~(always general domain corpus), and then used in downstream tasks by fine-tuning on task-specific datasets~(always from some specific domains).&#10;Specifically, given a pre-trained GPT-2 model, to generate sentences of email domain, we always need to fine-tune the GPT-2 on a small set of email domain corpus. &#10;&#10;However, we argue that to get desired sentence outputs, fine-tuning PLM on a specific domain dataset is not necessarily the best, especially when the fine-tuning dataset is of a small size. &#10;Typically, fine-tuning is conducted through Maximum Likelihood Estimation~(MLE), with which the resulting model distribution will be asymptotically consistent with true distribution when the fine-tuning dataset has infinite data samples. &#10;But it is not the case of fine-tuning on small datasets, which always leads to the mismatch problem of the real and model distributions.&#10;&#10;Specifically, MLE minimizes the Kullback–Leibler~(KL) divergence between model and true distributions.&#10;\cite{theis2015note} point out that minimizing KL avoids assigning an extremely small probability to any data point but assigns a lot of probability mass to non-data regions, which leads to a gap between $P_{Real}$ and $P_{Model}$.&#10;Additionally, simple data patterns in the fine-tuning dataset could be easily memorized and over-estimated. Meanwhile, the complex ones may be under-estimated. &#10;The above problem is not severe with adequate data samples, but non-trivial when the size of the fine-tuning dataset is not large enough.~({see Figure \ref{fig:intro}}).&#10;&#10;To address the over- and under-estimated problem, in this paper, we propose MC-Tailor, which can tailor the resulting density of model distribution by cutting the probability mass of over-estimated zones to under-estimated zones, leading to more realistic model distribution after fine-tuning.&#10;Concretely, MC-Tailor consists of two components: a ratio estimator to distinguish over- and under-estimated regions of model distribution; and an early rejection sampling~(ERS) component to tailor~(reassign) probability mass and efficiently obtain sampled sentences from the model distribution.&#10;Note that the proposed ERS is inspired by Sequential Monte Carlo~(SMC, \cite{doucet2000sequential}), but can avoid the degeneration from SMC, as it directly kills samples rather than performs resampling. &#10;&#10;We conduct experiments on various data sets to verify the effectiveness of the proposed MC-Tailor. &#10;Empirical results show that MC-Tailor can generate significantly better samples than finetuning, and the resulting model distributions of our model are closer to real data distributions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Fine-Tuning Methods, Computer Science, Linguistics, Text Generation Tasks, Language Model Training, Artificial Intelligence, Mathematics, Statistics" />
        </attvalues>
      </node>
      <node id="1511.01844" label="1511.01844">
        <attvalues>
          <attvalue for="0" value="A note on the evaluation of generative models" />
          <attvalue for="1" value="  Probabilistic generative models can be used for compression, denoising,&#10;inpainting, texture synthesis, semi-supervised learning, unsupervised feature&#10;learning, and other tasks. Given this wide range of applications, it is not&#10;surprising that a lot of heterogeneity exists in the way these models are&#10;formulated, trained, and evaluated. As a consequence, direct comparison between&#10;models is often difficult. This article reviews mostly known but often&#10;underappreciated properties relating to the evaluation and interpretation of&#10;generative models with a focus on image models. In particular, we show that&#10;three of the currently most commonly used criteria---average log-likelihood,&#10;Parzen window estimates, and visual fidelity of samples---are largely&#10;independent of each other when the data is high-dimensional. Good performance&#10;with respect to one criterion therefore need not imply good performance with&#10;respect to the other criteria. Our results show that extrapolation from one&#10;criterion to another is not warranted and generative models need to be&#10;evaluated directly with respect to the application(s) they were intended for.&#10;In addition, we provide examples demonstrating that Parzen window estimates&#10;should generally be avoided.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13648" label="2305.13648">
        <attvalues>
          <attvalue for="0" value="Non-parametric, Nearest-neighbor-assisted Fine-tuning for Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Non-parametric, k-nearest-neighbor algorithms have recently made inroads to&#10;assist generative models such as language models and machine translation&#10;decoders. We explore whether such non-parametric models can improve machine&#10;translation models at the fine-tuning stage by incorporating statistics from&#10;the kNN predictions to inform the gradient updates for a baseline translation&#10;model. There are multiple methods which could be used to incorporate kNN&#10;statistics and we investigate gradient scaling by a gating mechanism, the kNN's&#10;ground truth probability, and reinforcement learning. For four standard&#10;in-domain machine translation datasets, compared with classic fine-tuning, we&#10;report consistent improvements of all of the three methods by as much as 1.45&#10;BLEU and 1.28 BLEU for German-English and English-German translations&#10;respectively. Through qualitative analysis, we found particular improvements&#10;when it comes to translating grammatical relations or function words, which&#10;results in increased fluency of our model.&#10;" />
          <attvalue for="2" value="&#10;&#10;Non-parametric nearest neighbor models have been seen recent success for generative natural language processing tasks such as language modeling~\cite{khandelwal2020generalization} and machine translation~\cite{Khandelwal2020NearestNM}.&#10;Not only because explicitly memorizing the training data helps generalization, generative natural language models can scale to larger text collections without the added cost of training.&#10;\cite{khandelwal2020generalization} introduced $k$-nearest-neighbor machine translation~($k$NN-MT): a simple non-parametric method for machine translation~(MT) via nearest-neighbor retrievals was proposed and has been verified its effectiveness -- improving BLEU scores by roughly 3 for translating from&#10;English into German and Chinese.&#10;&#10;To easily adapt to multi domains, during inference, $k$NN-MT interpolates the softmax distribution for the target token from the neural machine translation~(NMT) model with the distribution of the retrieved set generated by the $k$-nearest-neighbor~($k$NN) search on a datastore of cached examples.&#10;The datastore is constructed from key-value pairs of parallel training data, where the key is the latent contextual representation of the target prefix tokens obtained via the NMT's stochastic forward-pass computing, and the value is the corresponding ground-truth target token.&#10;&#10;During preliminary investigations, we observed that the $k$NN search is able to memorize content words with lexical meanings of in-domain contexts. However, when it comes to translate grammatical relations, such as function word translations, querying the datastore for $k$ nearest neighbors is insufficient, which has a negative impact on the fluency of the final translation result.&#10;&#10;Moreover, although $k$NN-MT has the advantage that it does not require additional fine-tuning, our experiments show that $k$NN-MT cannot outperform or even achieve comparable performance to classic fine-tuning~\cite{mou-etal-2016-transferable} when in-domain data is accessible.&#10;This is due to the machine translation model not having been optimized on the in-domain data, and thus it limits the translation model's capability to utilize the $k$NN search.&#10;A simple way to observe this fact is to apply the $k$NN-MT algorithm on a fine-tuned translation model, but not a baseline translation model trained on out-of-domain data.&#10;As shown in Table~\ref{tab:ft_knn_mt} and Table~\ref{tab:prob} in Appendix \ref{sec:appendix}, the performance of the algorithm can be largely improved when the $k$NN datastore is constructed with fine-tuned contextual representations and their corresponding keys.&#10;Therefore, fine-tuning is still necessary and it benefits the non-parametric $k$NN search algorithm.&#10;&#10;Given that there are both advantages and disadvantages stemming from the $k$NN-MT algorithm, we proceed to maximize the use of the results from the $k$NN search to enhance the performance of translation models.&#10;We hypothesize that the fine-tuning procedure of a neural translation model can be improved with the assistance of statistics from the $k$NN predictions.&#10;Furthermore, we also explore gradient scaling for the original neural translation model with (1) a gate mechanism applied on the distribution of $k$NN predictions, (2) the $k$NN ground truth probability and (3) reinforcement learning based on the statistics of $k$NN predictions. &#10;&#10;Based on these observations, we propose trainable-$k$NN-MT to alleviate the problems of the the original $k$NN-MT~\cite{Khandelwal2020NearestNM}.&#10;Our trainable-$k$NN-MT is able to learn translations conditioned on the retrieved k-nearest-neighbors.&#10;The statistics of the retrieved set are incorporated into model fine-tuning via three ways aforementioned to dynamically scale up the gradient for back-propagation.&#10;In addition, the $k$NN datastore for retrieving is jointly updated with model fine-tuning so that the $k$NN search can secure more accurate $k$ nearest neighbors.&#10;&#10;There are two main contributions in this paper:&#10;(1) The trainable-$k$NN-MT generates better objective contextual representations of relevant examples, which improves the retrieved sets of top-k nearest neighbors.&#10;(2) The trainable-$k$NN-MT significantly outperforms both of the original $k$NN-MT algorithm and the classic fine-tuning, making it a novel fine-tuning method for neural machine translation. In addition, the fluency of the translation from trainable-$k$NN-MT is qualitatively improved, while staying more faithful to the original language.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Non-Parametric Models, Mathematics, Language Modeling, Statistics" />
        </attvalues>
      </node>
      <node id="1603.06111" label="1603.06111">
        <attvalues>
          <attvalue for="0" value="How Transferable are Neural Networks in NLP Applications?" />
          <attvalue for="1" value="  Transfer learning is aimed to make use of valuable knowledge in a source&#10;domain to help model performance in a target domain. It is particularly&#10;important to neural networks, which are very likely to be overfitting. In some&#10;fields like image processing, many studies have shown the effectiveness of&#10;neural network-based transfer learning. For neural NLP, however, existing&#10;studies have only casually applied transfer learning, and conclusions are&#10;inconsistent. In this paper, we conduct systematic case studies and provide an&#10;illuminating picture on the transferability of neural networks in NLP.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.14394" label="2009.14394">
        <attvalues>
          <attvalue for="0" value="Multiple Word Embeddings for Increased Diversity of Representation" />
          <attvalue for="1" value="  Most state-of-the-art models in natural language processing (NLP) are neural&#10;models built on top of large, pre-trained, contextual language models that&#10;generate representations of words in context and are fine-tuned for the task at&#10;hand. The improvements afforded by these &quot;contextual embeddings&quot; come with a&#10;high computational cost. In this work, we explore a simple technique that&#10;substantially and consistently improves performance over a strong baseline with&#10;negligible increase in run time. We concatenate multiple pre-trained embeddings&#10;to strengthen our representation of words. We show that this concatenation&#10;technique works across many tasks, datasets, and model types. We analyze&#10;aspects of pre-trained embedding similarity and vocabulary coverage and find&#10;that the representational diversity between different pre-trained embeddings is&#10;the driving force of why this technique works. We provide open source&#10;implementations of our models in both TensorFlow and PyTorch.&#10;" />
          <attvalue for="2" value="&#10;&#10;Much of the recent work in NLP has focused on better feature representations via contextual word embeddings&#10;\cite{Peters:2018, Peters2017SemisupervisedST:17, RadfordTransformer2018, Akbik2018ContextualSE, devlin2018bert}.&#10;These models vary in architecture and pre-training objective but they all encode the input based on the&#10;surrounding context in some way. These papers normally compare to baselines like a bidirectional LSTM-CRF (biLSTM-CRF)&#10;where words are represented by a single pre-trained word embedding.&#10;&#10;\cite{Peters:2018, Peters2017SemisupervisedST:17} and \cite{Akbik2018ContextualSE} pre-train large language models&#10;based on LSTMs. Task-specific architectures are then built on top of these pre-trained models. \cite{Peters:2018}&#10;introduce a technique for extracting word representations as a linear combination of layers in the pre-trained model.&#10;Gradient updates are only applied to this weighting factor, which simplifies the training to some extent, but forward&#10;propagation is still required for the full network which makes the model slow to train and evaluate.&#10;&#10;\cite{RadfordTransformer2018}, followed by \cite{devlin2018bert}, pre-train deep transformers \cite{Vaswani:2017} on&#10;massive corpora. They both use a simple output layer on top of the pre-trained model and tune the parameters of the&#10;whole model. In this case,&#10;training requires the forward and backward pass of the entire pre-trained model, which&#10;has a significant impact on size and speed. \cite{devlin2018bert} used&#10;specialized hardware which may be unrealistic for many inference scenarios.&#10;&#10;The prevailing wisdom is that, because these pre-trained models are contextual, they can create representations&#10;of a word that is different in different contexts. For example, a polysemous word can be represented by different&#10;vectors when its context suggests a different sense of a word, while context-independent word vectors need to represent a mix&#10;of all the senses of a word. The majority of NLP models have a similar ``contextualization'' step, typically done via a&#10;biLSTM, convolutional layers, or self-attention, but it is only learned from a smaller, task-specific corpus in contrast&#10;to the massive corpora used by contextual embeddings.&#10;&#10;Contextual embeddings and transfer learning architectures are slow to train and evaluate, which may make them infeasible&#10;for many types of deployments. Using multiple pre-trained embeddings trained on different datasets, we can exploit the&#10;bias in different datasets that results in different representations of the same word. By combining these embeddings, we&#10;can create richer representations of the word without the high computational overhead required by contextual alternatives.&#10;We find that the concatenation of multiple pre-trained word embeddings show&#10;consistent improvements over single embeddings yielding results much closer to contextual alternatives.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Neural Network Models, Contextual Embeddings, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2203.15613" label="2203.15613">
        <attvalues>
          <attvalue for="0" value="Dynamic Latency for CTC-Based Streaming Automatic Speech Recognition&#10;  With Emformer" />
          <attvalue for="1" value="  An inferior performance of the streaming automatic speech recognition models&#10;versus non-streaming model is frequently seen due to the absence of future&#10;context. In order to improve the performance of the streaming model and reduce&#10;the computational complexity, a frame-level model using efficient augment&#10;memory transformer block and dynamic latency training method is employed for&#10;streaming automatic speech recognition in this paper. The long-range history&#10;context is stored into the augment memory bank as a complement to the limited&#10;history context used in the encoder. Key and value are cached by a cache&#10;mechanism and reused for next chunk to reduce computation. Afterwards, a&#10;dynamic latency training method is proposed to obtain better performance and&#10;support low and high latency inference simultaneously. Our experiments are&#10;conducted on benchmark 960h LibriSpeech data set. With an average latency of&#10;640ms, our model achieves a relative WER reduction of 6.0% on test-clean and&#10;3.0% on test-other versus the truncate chunk-wise Transformer.&#10;" />
          <attvalue for="2" value="&#10;Conventional hybrid models have been widely adopted in Automatic Speech Recognition and mostly consist of the acoustic model (AM), pronunciation model (PM) and language model (LM) which are trained separately on different objective and data sets. Compared with hybrid models, end-to-end (E2E) models showed comparable performance by integrating acoustic model, pronunciation model and language model into a single model to achieve joint optimization. Accordingly, many state-of-art models, e.g., connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models.&#10;&#10;Streaming ASR is an important scenario in online application, which emit the token as soon as possible after a partial utterance from the speaker. However, the insufficient future context may result in performance degradation. There is a trade-off between latency and accuracy. The AED model and Transformer model for ASR are not feasible for streaming automatic speech recognition, as the global attention mechanism requires all input feature sequence for the calculation of monotonic attention alignment to generate context information. Several methods have been proposed for streaming automatic speech recognition. &#10;&#10;LAS obtains excellent performance, which uses pyramidal BiLSTM and attention mechanism~\cite{chorowski2014end} architecture. Nonetheless, streaming scenarios prevent the direct use of LAS. To facilitate the use of the LAS model for streaming, HMA~\cite{aharoni2016morphological} is put forward to the direct modeling of monotonic alignment to perform hard attention. Monotonic of chunk attention (MoChA)~\cite{chiu2017monotonic} introduces a novel attention mechanism that retains the benefits of hard monotonic while allowing soft alignments. sMoChA~\cite{miao2019online} is proposed to stable the training of MoChA to obviate exponential decay. ~\cite{inaguma2020minimum} proposes minimum latency training strategies under the MoChA framework. &#10;&#10;To facilitate the use of the transformer-based model for streaming, time-restricted self-attention~\cite{povey2018time} uses a limited length of look-ahead frames for each layer to improve the performance under low latency. However, this method is subject to context leaking in the event of excessive foresight frame lengths such that the foresight context grows linearly with the number of transformer layers. To attenuate context leaking, Transformer-XL~\cite{dai2019transformer} proposes a chunk-wise method in which all frames are visible to other frames in a chunk and caches the previous chunk to reduce the computational complexity. Augment memory transformer~\cite{wu2020streaming,yeh2021streaming} is presented to adopt an augment memory bank to capture long-range history context and reduce computational complexity, which has demonstrated comparable performance.~\cite{cao2021improving} proposes to use self-training to improve the performance of chunk-wise streaming model. However, the performance degradation of the streaming automatic speech recognition models also exist due to the absence of future context under low latency.&#10;&#10;In this paper, we solve these drawbacks with a frame-level streaming automatic speech recognition architecture in which Emformer is an encoder to enhance the long-range history context. Emformer~\cite{shi2021emformer}, which is brought forward to reduce computational complexity and applies parallelized block processing training method. To improve the model's performance, we propose a dynamic latency training method. The experiments are conducted on LibriSpeech corpus~\cite{panayotov2015librispeech}. Compared with truncate chunk-wise transformer-based baseline, our model gets a relative WER reduction of 6.0\% on test-clean and 3.0\% on test-other. Moreover, the dynamic latency method gets relative a WER reduction of 5.5\% on test-clean and 1.0\% on test-other versus the efficient augment memory transformer-based model.&#10;&#10;The paper is organized as follows: Section \ref{rnnt} gives a brief introduction about CTC criterion, Transformer, Emoformer and proposed dynamic latency training, followed by experiments and discussions in Section \ref{src_exp}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Recognition Models, Computer Science, Electrical Engineering, Linguistics, Deep Learning Architectures, Computational Complexity Reduction, Signal Processing" />
        </attvalues>
      </node>
      <node id="1611.01487" label="1611.01487">
        <attvalues>
          <attvalue for="0" value="Morphological Inflection Generation with Hard Monotonic Attention" />
          <attvalue for="1" value="  We present a neural model for morphological inflection generation which&#10;employs a hard attention mechanism, inspired by the nearly-monotonic alignment&#10;commonly found between the characters in a word and the characters in its&#10;inflection. We evaluate the model on three previously studied morphological&#10;inflection generation datasets and show that it provides state of the art&#10;results in various setups compared to previous neural and non-neural&#10;approaches. Finally we present an analysis of the continuous representations&#10;learned by both the hard and soft attention \cite{bahdanauCB14} models for the&#10;task, shedding some light on the features such models extract.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.08042" label="2005.08042">
        <attvalues>
          <attvalue for="0" value="Streaming Transformer-based Acoustic Models Using Self-attention with&#10;  Augmented Memory" />
          <attvalue for="1" value="  Transformer-based acoustic modeling has achieved great suc-cess for both&#10;hybrid and sequence-to-sequence speech recogni-tion. However, it requires&#10;access to the full sequence, and thecomputational cost grows quadratically with&#10;respect to the in-put sequence length. These factors limit its adoption for&#10;stream-ing applications. In this work, we proposed a novel augmentedmemory&#10;self-attention, which attends on a short segment of theinput sequence and a&#10;bank of memories. The memory bankstores the embedding information for all the&#10;processed seg-ments. On the librispeech benchmark, our proposed&#10;methodoutperforms all the existing streamable transformer methods bya large&#10;margin and achieved over 15% relative error reduction,compared with the widely&#10;used LC-BLSTM baseline. Our find-ings are also confirmed on some large internal&#10;datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.07120" label="2011.07120">
        <attvalues>
          <attvalue for="0" value="Streaming Attention-Based Models with Augmented Memory for End-to-End&#10;  Speech Recognition" />
          <attvalue for="1" value="  Attention-based models have been gaining popularity recently for their strong&#10;performance demonstrated in fields such as machine translation and automatic&#10;speech recognition. One major challenge of attention-based models is the need&#10;of access to the full sequence and the quadratically growing computational cost&#10;concerning the sequence length. These characteristics pose challenges,&#10;especially for low-latency scenarios, where the system is often required to be&#10;streaming. In this paper, we build a compact and streaming speech recognition&#10;system on top of the end-to-end neural transducer architecture with&#10;attention-based modules augmented with convolution. The proposed system equips&#10;the end-to-end models with the streaming capability and reduces the large&#10;footprint from the streaming attention-based model using augmented memory. On&#10;the LibriSpeech dataset, our proposed system achieves word error rates 2.7% on&#10;test-clean and 5.8% on test-other, to our best knowledge the lowest among&#10;streaming approaches reported so far.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.07327" label="2109.07327">
        <attvalues>
          <attvalue for="0" value="Improving Streaming Transformer Based ASR Under a Framework of&#10;  Self-supervised Learning" />
          <attvalue for="1" value="  Recently self-supervised learning has emerged as an effective approach to&#10;improve the performance of automatic speech recognition (ASR). Under such a&#10;framework, the neural network is usually pre-trained with massive unlabeled&#10;data and then fine-tuned with limited labeled data. However, the non-streaming&#10;architecture like bidirectional transformer is usually adopted by the neural&#10;network to achieve competitive results, which can not be used in streaming&#10;scenarios. In this paper, we mainly focus on improving the performance of&#10;streaming transformer under the self-supervised learning framework.&#10;Specifically, we propose a novel two-stage training method during fine-tuning,&#10;which combines knowledge distilling and self-training. The proposed training&#10;method achieves 16.3% relative word error rate (WER) reduction on Librispeech&#10;noisy test set. Finally, by only using the 100h clean subset of Librispeech as&#10;the labeled data and the rest (860h) as the unlabeled data, our streaming&#10;transformer based model obtains competitive WERs 3.5/8.7 on Librispeech&#10;clean/noisy test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.00157" label="2110.00157">
        <attvalues>
          <attvalue for="0" value="Under the Microscope: Interpreting Readability Assessment Models for&#10;  Filipino" />
          <attvalue for="1" value="  Readability assessment is the process of identifying the level of ease or&#10;difficulty of a certain piece of text for its intended audience. Approaches&#10;have evolved from the use of arithmetic formulas to more complex&#10;pattern-recognizing models trained using machine learning algorithms. While&#10;using these approaches provide competitive results, limited work is done on&#10;analyzing how linguistic variables affect model inference quantitatively. In&#10;this work, we dissect machine learning-based readability assessment models in&#10;Filipino by performing global and local model interpretation to understand the&#10;contributions of varying linguistic features and discuss its implications in&#10;the context of the Filipino language. Results show that using a model trained&#10;with top features from global interpretation obtained higher performance than&#10;the ones using features selected by Spearman correlation. Likewise, we also&#10;empirically observed local feature weight boundaries for discriminating reading&#10;difficulty at an extremely fine-grained level and their corresponding effects&#10;if values are perturbed.&#10;" />
          <attvalue for="2" value="&#10;&#10;Readability assessment is the process of evaluating a certain piece of text or reading material in terms of reading difficulty. Likewise, reading difficulty can be expressed in various forms such as age level, grade level, or a number from a certain range defined by a book publisher \cite{deutsch-etal-2020-linguistic}. Through the years, this process has evolved from the use of handcrafted arithmetic formulas such as the Flesch-Kincaid Reading Ease \cite{kincaid1975derivation} and Dale-Chall \cite{dale1948formula} readability formulas to the use of supervised machine learning algorithms such as Logistic Regression and Support Vector Machines \cite{chatzipanagiotidis-etal-2021-broad,weiss-meurers-2018-modeling,xia-etal-2016-text,reynolds-2016-insights,vajjala-meurers-2012-improving}. Despite the significant growth in research history, several problems still pose as open challenges for the task such as the (a) availability of corpora and tools for linguistic feature extraction, (b) extrinsic evaluation, and (c) interpretation of linguistic predictors used which is arguably the most important of all \cite{vajjalatrends}. &#10;&#10;We describe the general readability assessment task as feature interpretation dependent since identifying which optimal subset of linguistic features that can potentially impact readability levels is a strict and necessary part of the research process that should not be ignored. Recent works in readability analysis \cite{imperial2020exploring,hancke-etal-2012-readability} have used standalone, correlation-based feature selection techniques such as Spearman or Pearson correlation to get a better understanding of feature dependence and relationship. These methods, however, can be done even before model training which may not be holistically predictive of features that trained machine learning models would eventually find useful \cite{kumar2018correlation}. In addition, these methods break down as they only measure the linear relationship of linguistic variables in contrast to a possibility of a non-linear relationship of features in the dataset.&#10;&#10;In order to have a clear understanding of how a linguistic predictor affects model inference in readability assessment, the learned model weights of the said machine learning model should be extracted and analyzed after training. In this way, one can survey and rank the features used by a model and that has contributed substantially towards obtaining the final output. Thus, we lay down our contributions for this study as follows:&#10;&#10;\begin{enumerate}&#10; \item Feature selection through global interpretation of the state-of-the-art trained model in readability assessment for Filipino;&#10; \item Analysis of the performance of readability assessment models trained using top features from global interpretation against top features from a correlation method; and,&#10; \item Close-up analysis of top local features used by each grade level for readability analysis and their local weight boundaries through local surrogate interpretation.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning Models, Machine Learning, Computer Science, Linguistics, Linguistic Feature Analysis, Readability Assessment, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="1906.07580" label="1906.07580">
        <attvalues>
          <attvalue for="0" value="Text Readability Assessment for Second Language Learners" />
          <attvalue for="1" value="  This paper addresses the task of readability assessment for the texts aimed&#10;at second language (L2) learners. One of the major challenges in this task is&#10;the lack of significantly sized level-annotated data. For the present work, we&#10;collected a dataset of CEFR-graded texts tailored for learners of English as an&#10;L2 and investigated text readability assessment for both native and L2&#10;learners. We applied a generalization method to adapt models trained on larger&#10;native corpora to estimate text readability for learners, and explored domain&#10;adaptation and self-learning techniques to make use of the native data to&#10;improve system performance on the limited L2 data. In our experiments, the best&#10;performing model for readability on learner texts achieves an accuracy of 0.797&#10;and PCC of $0.938$.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.00973" label="2105.00973">
        <attvalues>
          <attvalue for="0" value="Trends, Limitations and Open Challenges in Automatic Readability&#10;  Assessment Research" />
          <attvalue for="1" value="  Readability assessment is the task of evaluating the reading difficulty of a&#10;given piece of text. Although research on computational approaches to&#10;readability assessment is now two decades old, there is not much work on&#10;synthesizing this research. This article is a brief survey of contemporary&#10;research on developing computational models for readability assessment. We&#10;identify the common approaches, discuss their shortcomings, and identify some&#10;challenges for the future. Where possible, we also connect computational&#10;research with insights from related work in other disciplines such as education&#10;and psychology.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.09900" label="2406.09900">
        <attvalues>
          <attvalue for="0" value="GEB-1.3B: Open Lightweight Large Language Model" />
          <attvalue for="1" value="  Recently developed large language models (LLMs) such as ChatGPT, Claude, and&#10;Llama have demonstrated impressive abilities, and even surpass human-level&#10;performance in several tasks. Despite their success, the resource-intensive&#10;demands of these models, requiring significant computational power for both&#10;training and inference, limit their deployment to high-performance servers.&#10;Additionally, the extensive calculation requirements of the models often lead&#10;to increased latency in response times. With the increasing need for LLMs to&#10;operate efficiently on CPUs, research about lightweight models that are&#10;optimized for CPU inference has emerged. In this work, we introduce GEB-1.3B, a&#10;lightweight LLM trained on 550 billion tokens in both Chinese and English&#10;languages. We employ novel training techniques, including ROPE,&#10;Group-Query-Attention, and FlashAttention-2, to accelerate training while&#10;maintaining model performance. Additionally, we fine-tune the model using 10&#10;million samples of instruction data to enhance alignment. GEB-1.3B exhibits&#10;outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU,&#10;outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B.&#10;Notably, the FP32 version of GEB-1.3B achieves commendable inference times on&#10;CPUs, with ongoing efforts to further enhance speed through advanced&#10;quantization techniques. The release of GEB-1.3B as an open-source model marks&#10;a significant contribution to the development of lightweight LLMs, promising to&#10;foster further research and innovation in the field.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models have experienced significant advancements, achieving superhuman capabilities in numerous specific tasks. These models, such as ChatGPT 3.5~\cite{openai_chatgpt}, GPT-4~\cite{openai2024gpt4}, and Claude~\cite{claude}, demonstrate remarkable success across various languages, particularly in English. On the other hand, models such as Baichuan~\cite{baichuan2023baichuan2}, ChatGLM~\cite{zeng2022glm}, and Qwen~\cite{qwen} are mainly used for Chinese. However, these powerful models require a lot of computer resources, like GPUs or TPUs, for both training and inference phases. Consequently, these models are typically operational on remote high-performance servers. Furthermore, the vast computations required by the models bring potential delays in responding. There exists a significant demand for models that can efficiently run on CPUs, thereby enabling deployment on more accessible devices such as laptops and smartphones. In response to this need, researchers are now focusing on the development of lightweight large language models, aiming to reduce both response times and hardware costs. Remarkably, the Llama 2 model~\cite{Touvron2023Llama2O} series demonstrates that, when trained with massive data, smaller models can surpass their larger counterparts in performance. Despite being trained on trillions of tokens, the Llama 2 series shows no signs of training loss saturation, suggesting untapped potential in smaller model capacities.&#10;&#10;In this work, we introduce GEB-1.3B, a lightweight model featuring 1.3 billion parameters and trained on 550 billion tokens in both Chinese and English languages. We incorporate advanced techniques such as ROPE~\cite{su2024roformer}, Group-Query-Attention~\cite{ainslie2023gqa}, and FlashAttention-2~\cite{dao2022flashattention} to expedite our training process. Additionally, we fine-tune the model using 10 million instruction-based samples to enhance its alignment.&#10;&#10;The evaluation results indicate that GEB-1.3B outperforms comparable models such as MindLLM-1.3B\cite{yang2023mindllm} and TinyLLaMA-1.1B\cite{zhang2024tinyllama} on general benchmarks, including MMLU\cite{hendrycks2020measuring}, C-Eval\cite{huang2024c}, and CMMLU\cite{li2023cmmlu}. Moreover, the inference time of the FP32 version on CPUs is sufficiently rapid for practical applications. And we plan to explore further acceleration through quantization in the future.&#10;&#10;We are releasing GEB-1.3B to the general public for research use at. The model has demonstrated superior performance than other lightweight models of the similar parameter size, and even surpass some larger counterparts. We believe that the release of this lightweight LLM will benefit further research.&#10;&#10;The remainder of this paper describes the pre-training methodology (Section \ref{pretrain}), alignment technology (Section \ref{align}), evaluations on benchmarks (Section \ref{eval}), and conclusions (Section \ref{con}). &#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Information Technology, Linguistics, Efficient Computing, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2310.15777" label="2310.15777">
        <attvalues>
          <attvalue for="0" value="MindLLM: Pre-training Lightweight Large Language Model from Scratch,&#10;  Evaluations and Domain Applications" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated remarkable performance across&#10;various natural language tasks, marking significant strides towards general&#10;artificial intelligence. While general artificial intelligence is leveraged by&#10;developing increasingly large-scale models, there could be another branch to&#10;develop lightweight custom models that better serve certain domains, taking&#10;into account the high cost of training and deploying LLMs and the scarcity of&#10;resources. In this paper, we present MindLLM, a novel series of bilingual&#10;lightweight large language models, trained from scratch, alleviating such&#10;burdens by offering models with 1.3 billion and 3 billion parameters. A&#10;thorough account of experiences accrued during large model development is&#10;given, covering every step of the process, including data construction, model&#10;architecture, evaluation, and applications. Such insights are hopefully&#10;valuable for fellow academics and developers. MindLLM consistently matches or&#10;surpasses the performance of other open-source larger models on some public&#10;benchmarks. We also introduce an innovative instruction tuning framework&#10;tailored for smaller models to enhance their capabilities efficiently.&#10;Moreover, we explore the application of MindLLM in specific vertical domains&#10;such as law and finance, underscoring the agility and adaptability of our&#10;lightweight models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.09212" label="2306.09212">
        <attvalues>
          <attvalue for="0" value="CMMLU: Measuring massive multitask language understanding in Chinese" />
          <attvalue for="1" value="  As the capabilities of large language models (LLMs) continue to advance,&#10;evaluating their performance becomes increasingly crucial and challenging. This&#10;paper aims to bridge this gap by introducing CMMLU, a comprehensive Chinese&#10;benchmark that covers various subjects, including natural science, social&#10;sciences, engineering, and humanities. We conduct a thorough evaluation of 18&#10;advanced multilingual- and Chinese-oriented LLMs, assessing their performance&#10;across different subjects and settings. The results reveal that most existing&#10;LLMs struggle to achieve an average accuracy of 50%, even when provided with&#10;in-context examples and chain-of-thought prompts, whereas the random baseline&#10;stands at 25%. This highlights significant room for improvement in LLMs.&#10;Additionally, we conduct extensive experiments to identify factors impacting&#10;the models' performance and propose directions for enhancing LLMs. CMMLU fills&#10;the gap in evaluating the knowledge and reasoning capabilities of large&#10;language models within the Chinese context.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.03728" label="2301.03728">
        <attvalues>
          <attvalue for="0" value="Scaling Laws for Generative Mixed-Modal Language Models" />
          <attvalue for="1" value="  Generative language models define distributions over sequences of tokens that&#10;can represent essentially any combination of data modalities (e.g., any&#10;permutation of image tokens from VQ-VAEs, speech tokens from HuBERT, BPE tokens&#10;for language or code, and so on). To better understand the scaling properties&#10;of such mixed-modal models, we conducted over 250 experiments using seven&#10;different modalities and model sizes ranging from 8 million to 30 billion,&#10;trained on 5-100 billion tokens. We report new mixed-modal scaling laws that&#10;unify the contributions of individual modalities and the interactions between&#10;them. Specifically, we explicitly model the optimal synergy and competition due&#10;to data and model size as an additive term to previous uni-modal scaling laws.&#10;We also find four empirical phenomena observed during the training, such as&#10;emergent coordinate-ascent style training that naturally alternates between&#10;modalities, guidelines for selecting critical hyper-parameters, and connections&#10;between mixed-modal competition and training stability. Finally, we test our&#10;scaling law by training a 30B speech-text model, which significantly&#10;outperforms the corresponding unimodal models. Overall, our research provides&#10;valuable insights into the design and training of mixed-modal generative&#10;models, an important new class of unified models that have unique&#10;distributional properties.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Neural scaling laws quantify the relationship between model size, dataset size, compute budget, and performance, when training neural networks. This concept was introduced by \cite{deep_learning_scaling_2017}, who observed a power law relationship and later scaled to much larger models by \cite{kaplan2020scaling}. &#10;&#10;\cite{Chinchilla} developed a unified formula for scaling laws, and provided recipes for compute-optimal training by adding data-dependent scaling terms unlike previous power law parameterizations. &#10;Other researchers have applied these principles to specific tasks and different parameterization of Transformers. \cite{moe_scaling_laws} examined the application of neural scaling laws to Mixture of Experts (MoE) models. \cite{int8, 4_bit_scaling} studied the relationship between scaling laws and lower precision, which refers to using lower-precision data types, such as 16-bit floating point numbers, in neural networks. \cite{scaling_laws_nmt} and \cite{scaling_laws_nmt_ghorbani} applied these principles to Neural Machine Translation (NMT).&#10;&#10;Additionally, \cite{image_text_scaling} and \cite{speech_alexa_scaling_laws} examined the application of neural scaling laws to generative language models in different modalities, including image generation and acoustic models. \cite{CLIP_scaling_law} also examined multi-modal training but did not specifically focus on generative models.&#10;To our knowledge, we are the first to investigate the phenomenon of interactions, competition, and interference between multiple modalities during training and provide a recipe for optimal mixed-modal training. &#10;&#10;Interestingly, similar competition and scaling phenomenon have been observed for multi-lingual models. \cite{conneau2019unsupervised} observed a ``curse of multilinguality,'' where training in multiple languages can lead to interference between languages, resulting in decreased performance. \cite{goyal2021larger} and \cite{interference_translation} demonstrated that this interference could occur even on models much smaller than the available training data, but scaling up the model size can improve synergy and alleviate interference. These findings align with our findings in the mixed-modal scenario, suggesting that similar principles apply when training on multiple modalities.&#10;&#10;" />
          <attvalue for="4" value="Scaling Laws, Computer Science, Linguistics, Cognitive Science, Generative Language Models, Multimodal Learning, Mathematics" />
        </attvalues>
      </node>
      <node id="2304.11985" label="2304.11985">
        <attvalues>
          <attvalue for="0" value="Self-regularised Minimum Latency Training for Streaming&#10;  Transformer-based Speech Recognition" />
          <attvalue for="1" value="  This paper proposes a self-regularised minimum latency training (SR-MLT)&#10;method for streaming Transformer-based automatic speech recognition (ASR)&#10;systems. In previous works, latency was optimised by truncating the online&#10;attention weights based on the hard alignments obtained from conventional ASR&#10;models, without taking into account the potential loss of ASR accuracy. On the&#10;contrary, here we present a strategy to obtain the alignments as a part of the&#10;model training without external supervision. The alignments produced by the&#10;proposed method are dynamically regularised on the training data, such that the&#10;latency reduction does not result in the loss of ASR accuracy. SR-MLT is&#10;applied as a fine-tuning step on the pre-trained Transformer models that are&#10;based on either monotonic chunkwise attention (MoChA) or cumulative attention&#10;(CA) algorithms for online decoding. ASR experiments on the AIShell-1 and&#10;Librispeech datasets show that when applied on a decent pre-trained MoChA or CA&#10;baseline model, SR-MLT can effectively reduce the latency with the relative&#10;gains ranging from 11.8% to 39.5%. Furthermore, we also demonstrate that under&#10;certain accuracy levels, the models trained with SR-MLT can achieve lower&#10;latency when compared to those supervised using external hard alignments.&#10;" />
          <attvalue for="2" value="&#10;&#10;The Transformer architecture \cite{vaswani2017attention}, as an epitome of the attention-based encoder-decoder framework \cite{chan2016listen, chorowski2015attention}, has become one of the dominant end-to-end (E2E) ASR techniques. Compared with other modelling strategies such as connectionist temporal classification (CTC) \cite{graves2006connectionist, graves2014towards} and recurrent neural network (RNN) transducer \cite{graves2012sequence}, Transformer sustains severe latency issues at inference time, since it requires access to the full speech utterance for decoding. To facilitate streaming ASR, a number of online attention mechanisms have been proposed in literature, including monotonic chunkwise attention (MoChA) \cite{chiu2017monotonic, tsunoo2019towards, inaguma2020enhancing}, CTC-triggered attention \cite{moritz2020streaming}, decoder-end adaptive computation steps (DACS) based algorithms \cite{li2021transformer, li2021head}, as well as the recent cumulative attention (CA) \cite{li2022transformer}. Although the aforementioned algorithms enable Transformer to emit ASR outputs in real time, the emission can be delayed from the actual acoustic boundary, because the system always tends to involve some future information to enhance the prediction confidence. &#10;&#10;There have been attempts to reduce the latency level of attention-based online decoding, and the most popular method is minimum latency training (MLT) \cite{inaguma2020minimum}. MLT introduces the &quot;ground-truth&quot; token boundaries into the attention calculation, which can be obtained from the forced alignments produced by conventional hidden Markov model (HMM) based ASR models \cite{chiu2018state}. When training the E2E system, any attending probabilities beyond the boundary will be ignored, so as to attenuate the chance of triggering later than the real acoustic border.&#10;&#10;Besides, the CTC posterior spikes corresponding to the non-blank labels are also regarded as effective attention boundaries used in MLT, given that they sparsely appear around the endpoint of acoustic events \cite{moritz2020streaming}. Similar to the HMM-based method, a Viterbi decoding can be performed with the CTC model to generate the alignments that locate the output tokens. As a result, the triggering points produced by the attention model are calibrated as close to the CTC spikes as possible \cite{miao2022low}.&#10;&#10;In MLT, both HMM and CTC alignments impose arbitrary restrictions on the attention computation, where the effect of latency reduction is largely subject to the quality of the alignments themselves, which determine the upper bound of MLT performance. Furthermore, the sacrifice of ASR accuracy in exchange of lower latency is not well considered by MLT, making the trade-off easily biased to the latency side.&#10;&#10;Deep reinforcement learning boosted head-synchronous DACS (DRL-HS-DACS) \cite{li2021improving}, on the contrary, saves acquiring prior knowledge by self-exploring dynamic attention boundaries through its own learning experience. During E2E training, the system keeps track of the halting position along with the ASR prediction correctness observed on the training data, based on which the halting agent learns to seek the earliest triggering point with sufficient acoustic information. As opposed to MLT methods, DRL-HS-DACS could not only breakthrough the lower limit of latency posed by external alignments, but also maintain a balance between the accuracy and latency levels.&#10;&#10;Even if the halting positions produced by the DRL-HS-DACS model are generally optimised by the DRL algorithm, the absence of strong constraints (e.g. hard acoustic boundaries) upon the agent training can still lead to the failure of halting for certain decoding steps. Thus, other auxiliary halting strategies such as the maximum look-ahead steps are also adopted to help the system trigger ASR outputs in time.&#10;&#10;To overcome the issues encountered by MLT and DRL-HS-DACS while take advantage of their strengths, we propose the self-regularised minimum latency training (SR-MLT) algorithm. On the one hand, hard acoustic boundaries are utilised to cut off the redundant attention weights that cause delayed triggering, eliminating the need of DRL. On the other hand, these boundaries are dynamically obtained from the course of E2E training itself, rather than from external resources like forced alignments, which notably cuts down the training complexity when compared with the MLT methods.&#10;&#10;We applied SR-MLT on two streaming Transformer ASR systems that are based on MoChA \cite{chiu2017monotonic} and CA \cite{li2022transformer} algorithms respectively, in order to show it compatible to various online attention mechanisms. Our experiments demonstrate that the proposed method can achieve significant latency reduction against the baseline systems. including the vanilla MLT systems that leverage HMM produced acoustic boundaries. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Latency Optimization, Linguistics, Transformer Models, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2011.13834" label="2011.13834">
        <attvalues>
          <attvalue for="0" value="Transformer-based Online Speech Recognition with Decoder-end Adaptive&#10;  Computation Steps" />
          <attvalue for="1" value="  Transformer-based end-to-end (E2E) automatic speech recognition (ASR) systems&#10;have recently gained wide popularity, and are shown to outperform E2E models&#10;based on recurrent structures on a number of ASR tasks. However, like other E2E&#10;models, Transformer ASR also requires the full input sequence for calculating&#10;the attentions on both encoder and decoder, leading to increased latency and&#10;posing a challenge for online ASR. The paper proposes Decoder-end Adaptive&#10;Computation Steps (DACS) algorithm to address the issue of latency and&#10;facilitate online ASR. The proposed algorithm streams the decoding of&#10;Transformer ASR by triggering an output after the confidence acquired from the&#10;encoder states reaches a certain threshold. Unlike other monotonic attention&#10;mechanisms that risk visiting the entire encoder states for each output step,&#10;the paper introduces a maximum look-ahead step into the DACS algorithm to&#10;prevent from reaching the end of speech too fast. A Chunkwise encoder is&#10;adopted in our system to handle real-time speech inputs. The proposed online&#10;Transformer ASR system has been evaluated on Wall Street Journal (WSJ) and&#10;AIShell-1 datasets, yielding 5.5% word error rate (WER) and 7.1% character&#10;error rate (CER) respectively, with only a minor decay in performance when&#10;compared to the offline systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.05736" label="2203.05736">
        <attvalues>
          <attvalue for="0" value="Transformer-based Streaming ASR with Cumulative Attention" />
          <attvalue for="1" value="  In this paper, we propose an online attention mechanism, known as cumulative&#10;attention (CA), for streaming Transformer-based automatic speech recognition&#10;(ASR). Inspired by monotonic chunkwise attention (MoChA) and head-synchronous&#10;decoder-end adaptive computation steps (HS-DACS) algorithms, CA triggers the&#10;ASR outputs based on the acoustic information accumulated at each encoding&#10;timestep, where the decisions are made using a trainable device, referred to as&#10;halting selector. In CA, all the attention heads of the same decoder layer are&#10;synchronised to have a unified halting position. This feature effectively&#10;alleviates the problem caused by the distinct behaviour of individual heads,&#10;which may otherwise give rise to severe latency issues as encountered by MoChA.&#10;The ASR experiments conducted on AIShell-1 and Librispeech datasets demonstrate&#10;that the proposed CA-based Transformer system can achieve on par or better&#10;performance with significant reduction in latency during inference, when&#10;compared to other streaming Transformer systems in literature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1511.06335" label="1511.06335">
        <attvalues>
          <attvalue for="0" value="Unsupervised Deep Embedding for Clustering Analysis" />
          <attvalue for="1" value="  Clustering is central to many data-driven application domains and has been&#10;studied extensively in terms of distance functions and grouping algorithms.&#10;Relatively little work has focused on learning representations for clustering.&#10;In this paper, we propose Deep Embedded Clustering (DEC), a method that&#10;simultaneously learns feature representations and cluster assignments using&#10;deep neural networks. DEC learns a mapping from the data space to a&#10;lower-dimensional feature space in which it iteratively optimizes a clustering&#10;objective. Our experimental evaluations on image and text corpora show&#10;significant improvement over state-of-the-art methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.03706" label="2011.03706">
        <attvalues>
          <attvalue for="0" value="ESPnet-se: end-to-end speech enhancement and separation toolkit designed&#10;  for asr integration" />
          <attvalue for="1" value="  We present ESPnet-SE, which is designed for the quick development of speech&#10;enhancement and speech separation systems in a single framework, along with the&#10;optional downstream speech recognition module. ESPnet-SE is a new project which&#10;integrates rich automatic speech recognition related models, resources and&#10;systems to support and validate the proposed front-end implementation (i.e.&#10;speech enhancement and separation).It is capable of processing both&#10;single-channel and multi-channel data, with various functionalities including&#10;dereverberation, denoising and source separation. We provide all-in-one recipes&#10;including data pre-processing, feature extraction, training and evaluation&#10;pipelines for a wide range of benchmark datasets. This paper describes the&#10;design of the toolkit, several important functionalities, especially the speech&#10;recognition integration, which differentiates ESPnet-SE from other open source&#10;toolkits, and experimental results with major benchmark datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.07283" label="2211.07283">
        <attvalues>
          <attvalue for="0" value="SNIPER Training: Single-Shot Sparse Training for Text-to-Speech" />
          <attvalue for="1" value="  Text-to-speech (TTS) models have achieved remarkable naturalness in recent&#10;years, yet like most deep neural models, they have more parameters than&#10;necessary. Sparse TTS models can improve on dense models via pruning and extra&#10;retraining, or converge faster than dense models with some performance loss.&#10;Thus, we propose training TTS models using decaying sparsity, i.e. a high&#10;initial sparsity to accelerate training first, followed by a progressive rate&#10;reduction to obtain better eventual performance. This decremental approach&#10;differs from current methods of incrementing sparsity to a desired target,&#10;which costs significantly more time than dense training. We call our method&#10;SNIPER training: Single-shot Initialization Pruning Evolving-Rate training. Our&#10;experiments on FastSpeech2 show that we were able to obtain better losses in&#10;the first few training epochs with SNIPER, and that the final SNIPER-trained&#10;models outperformed constant-sparsity models and edged out dense models, with&#10;negligible difference in training time.&#10;" />
          <attvalue for="2" value="&#10;&#10;Although classical parametrized models only require $n$ parameters to fit $n$ data points,~\cite{bubeck} showed that large over-parameterized models with at least $nd$ parameters are in fact necessary for smooth data interpolation (where $d$ is the data dimensionality). This explains the dominance of ever-larger neural models, but we have now reached the era of extreme diminishing returns. For instance, \cite{limits} found that the compute power needed to reduce error rates by a factor of $x$ was at least $x^{10}$ across a range of image classification and natural language processing tasks. &#10;&#10;Counteracting this, researchers have proposed methods to compress models, such as knowledge distillation, tensor decomposition, quantization, and parameter sharing. The most flexible approaches involve sparsification techniques, which can be applied to various training stages and network architectures.&#10;&#10;For instance, \cite{lottery-ticket} showed that when a dense network is initialized, some subnetworks (winning tickets) can match or improve the same-iteration performance of the full network when trained in isolation. However, \cite{imp-fail} suggests that the winning tickets cannot be found without prior dense training and therefore requires much more time to train than the original unpruned model. The additional time taken holds for other sparsification schemes like dynamic pruning, which regrows weights during training according to gradients~\cite{nest} or momentum~\cite{sparse-momentum} in each backward pass. Sparse models also do not improve memory usage or training/inference time unless sparsities are very high (above 80\% for 2-D float32 tensors).&#10;&#10;Since the goal is not just to produce a sparse model but to improve the performance-cost tradeoff, we propose a single-shot sparse training scheme that progressively decreases sparsity to 0 rather than increasing it to a target sparsity. Intuitively, we want to direct gradient updates to more important weights at the start for faster convergence and to the less important weights later. We apply our method to FastSpeech2~\cite{fastspeech2} and evaluate its effectiveness via naturalness, intelligibility, prosody, and training time, comparing it to both dense and constant-sparsity models. We observe that the final SNIPER-trained models overtake the constant-sparsity models on nearly all metrics and surpass the dense model in most. &#10;&#10;In the next section, we introduce the related work on fast sparse training, followed by our proposed SNIPER training framework in Section \ref{sec:sniper}, our experiment setup in Section \ref{sec:experiments}, and finally we report our results in Section \ref{sec:results}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Neural Networks, Model Pruning Techniques, Mathematics, Text-to-Speech Models" />
        </attvalues>
      </node>
      <node id="2105.12806" label="2105.12806">
        <attvalues>
          <attvalue for="0" value="A Universal Law of Robustness via Isoperimetry" />
          <attvalue for="1" value="  Classically, data interpolation with a parametrized model class is possible&#10;as long as the number of parameters is larger than the number of equations to&#10;be satisfied. A puzzling phenomenon in deep learning is that models are trained&#10;with many more parameters than what this classical theory would suggest. We&#10;propose a partial theoretical explanation for this phenomenon. We prove that&#10;for a broad class of data distributions and model classes, overparametrization&#10;is necessary if one wants to interpolate the data smoothly. Namely we show that&#10;smooth interpolation requires $d$ times more parameters than mere&#10;interpolation, where $d$ is the ambient data dimension. We prove this universal&#10;law of robustness for any smoothly parametrized function class with polynomial&#10;size weights, and any covariate distribution verifying isoperimetry. In the&#10;case of two-layers neural networks and Gaussian covariates, this law was&#10;conjectured in prior work by Bubeck, Li and Nagaraj. We also give an&#10;interpretation of our result as an improved generalization bound for model&#10;classes consisting of smooth functions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.05558" label="2007.05558">
        <attvalues>
          <attvalue for="0" value="The Computational Limits of Deep Learning" />
          <attvalue for="1" value="  Deep learning's recent history has been one of achievement: from triumphing&#10;over humans in the game of Go to world-leading performance in image&#10;classification, voice recognition, translation, and other tasks. But this&#10;progress has come with a voracious appetite for computing power. This article&#10;catalogs the extent of this dependency, showing that progress across a wide&#10;variety of applications is strongly reliant on increases in computing power.&#10;Extrapolating forward this reliance reveals that progress along current lines&#10;is rapidly becoming economically, technically, and environmentally&#10;unsustainable. Thus, continued progress in these applications will require&#10;dramatically more computationally-efficient methods, which will either have to&#10;come from changes to deep learning or from moving to other machine learning&#10;methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.06955" label="2106.06955">
        <attvalues>
          <attvalue for="0" value="Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets&#10;  Win" />
          <attvalue for="1" value="  The lottery ticket hypothesis states that sparse subnetworks exist in&#10;randomly initialized dense networks that can be trained to the same accuracy as&#10;the dense network they reside in. However, the subsequent work has failed to&#10;replicate this on large-scale models and required rewinding to an early stable&#10;state instead of initialization. We show that by using a training method that&#10;is stable with respect to linear mode connectivity, large networks can also be&#10;entirely rewound to initialization. Our subsequent experiments on common vision&#10;tasks give strong credence to the hypothesis in Evci et al. (2020b) that&#10;lottery tickets simply retrain to the same regions (although not necessarily to&#10;the same basin). These results imply that existing lottery tickets could not&#10;have been found without the preceding dense training by iterative magnitude&#10;pruning, raising doubts about the use of the lottery ticket hypothesis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.02017" label="1711.02017">
        <attvalues>
          <attvalue for="0" value="NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm" />
          <attvalue for="1" value="  Deep neural networks (DNNs) have begun to have a pervasive impact on various&#10;applications of machine learning. However, the problem of finding an optimal&#10;DNN architecture for large applications is challenging. Common approaches go&#10;for deeper and larger DNN architectures but may incur substantial redundancy.&#10;To address these problems, we introduce a network growth algorithm that&#10;complements network pruning to learn both weights and compact DNN architectures&#10;during training. We propose a DNN synthesis tool (NeST) that combines both&#10;methods to automate the generation of compact and accurate DNNs. NeST starts&#10;with a randomly initialized sparse network called the seed architecture. It&#10;iteratively tunes the architecture with gradient-based growth and&#10;magnitude-based pruning of neurons and connections. Our experimental results&#10;show that NeST yields accurate, yet very compact DNNs, with a wide range of&#10;seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we&#10;reduce network parameters by 70.2x (74.3x) and floating-point operations&#10;(FLOPs) by 79.4x (43.7x). For the AlexNet and VGG-16 architectures, we reduce&#10;network parameters (FLOPs) by 15.7x (4.6x) and 30.2x (8.6x), respectively.&#10;NeST's grow-and-prune paradigm delivers significant additional parameter and&#10;FLOPs reduction relative to pruning-only methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.04840" label="1907.04840">
        <attvalues>
          <attvalue for="0" value="Sparse Networks from Scratch: Faster Training without Losing Performance" />
          <attvalue for="1" value="  We demonstrate the possibility of what we call sparse learning: accelerated&#10;training of deep neural networks that maintain sparse weights throughout&#10;training while achieving dense performance levels. We accomplish this by&#10;developing sparse momentum, an algorithm which uses exponentially smoothed&#10;gradients (momentum) to identify layers and weights which reduce the error&#10;efficiently. Sparse momentum redistributes pruned weights across layers&#10;according to the mean momentum magnitude of each layer. Within a layer, sparse&#10;momentum grows weights according to the momentum magnitude of zero-valued&#10;weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10,&#10;and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared&#10;to other sparse algorithms. Furthermore, we show that sparse momentum reliably&#10;reproduces dense performance levels while providing up to 5.61x faster&#10;training. In our analysis, ablations show that the benefits of momentum&#10;redistribution and growth increase with the depth and size of the network.&#10;Additionally, we find that sparse momentum is insensitive to the choice of its&#10;hyperparameters suggesting that sparse momentum is robust and easy to use.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.00496" label="2005.00496">
        <attvalues>
          <attvalue for="0" value="Structured Tuning for Semantic Role Labeling" />
          <attvalue for="1" value="  Recent neural network-driven semantic role labeling (SRL) systems have shown&#10;impressive improvements in F1 scores. These improvements are due to expressive&#10;input representations, which, at least at the surface, are orthogonal to&#10;knowledge-rich constrained decoding mechanisms that helped linear SRL models.&#10;Introducing the benefits of structure to inform neural models presents a&#10;methodological challenge. In this paper, we present a structured tuning&#10;framework to improve models using softened constraints only at training time.&#10;Our framework leverages the expressiveness of neural networks and provides&#10;supervision with structured loss components. We start with a strong baseline&#10;(RoBERTa) to validate the impact of our approach, and show that our framework&#10;outperforms the baseline by learning to comply with declarative constraints.&#10;Additionally, our experiments with smaller training sizes show that we can&#10;achieve consistent improvements under low-resource scenarios.&#10;" />
          <attvalue for="2" value="&#10;&#10;Semantic Role Labeling~\cite[SRL,][]{palmer2010semantic} is the task of labeling&#10;semantic arguments of predicates in sentences to identify who does what to&#10;whom. Such representations can come in handy in tasks involving text&#10;understanding, such as coreference&#10;resolution~\cite{ponzetto-strube-2006-exploiting} and reading&#10;comprehension~\cite[e.g.,][]{berant-etal-2014-modeling,zhang2019semantics}. This&#10;paper focuses on the question of how knowledge can influence modern semantic&#10;role labeling models.&#10;&#10;Linguistic knowledge can help SRL models in several ways. For example, syntax&#10;can drive feature design~\cite[e.g.,][ and&#10;others]{punyakanok2005necessity,toutanova-etal-2005-joint,kshirsagar-etal-2015-frame,johansson-nugues-2008-dependency},&#10;and can also be embedded into neural network&#10;architectures~\cite{strubell2018linguistically}.&#10;&#10;In addition to such influences on input representations, knowledge about the nature of semantic roles can inform structured decoding algorithms used to construct the outputs. The SRL literature is witness to a rich array of techniques for structured inference, including integer linear programs~\cite[e.g.,][]{punyakanok2005necessity,punyakanok2008importance}, bespoke inference algorithms~\cite[e.g.,][]{tackstrom2015efficient}, A* decoding~\cite[e.g.,][]{he-etal-2017-deep}, greedy heuristics~\cite[e.g.,][]{ouchi-etal-2018-span}, or simple Viterbi decoding to ensure that token tags are BIO-consistent.&#10;&#10;By virtue of being constrained by the definition of the task, global inference promises semantically meaningful outputs, and could provide valuable signal when models are being trained. However, beyond Viterbi decoding, it may impose prohibitive computational costs, thus ruling out using inference during training. Indeed, optimal inference may be intractable, and inference-driven training may require ignoring certain constraints that render inference difficult. &#10;&#10;While global inference was a mainstay of SRL models until recently, today's end-to-end trained neural architectures have shown remarkable successes without needing decoding. These successes can be attributed to the expressive input and internal representations learned by neural networks. The only structured component used with such models, if at all, involves sequential dependencies between labels that admit efficient decoding.&#10;&#10;In this paper, we ask: Can we train neural network models for semantic roles in the presence of general output constraints, without paying the high computational cost of inference?&#10;We propose a structured tuning approach that exposes a neural SRL model to&#10;differentiable constraints during the finetuning step.&#10;To do so, we first write the output space constraints as logic rules. Next, we&#10;relax such statements into differentiable forms that serve as&#10;regularizers to inform the model at training time. Finally, during inference,&#10;our structure-tuned models are free to make their own judgments about labels&#10;without any inference algorithms beyond a simple linear sequence decoder.&#10;&#10;We evaluate our structured tuning on the CoNLL-05~\cite{carreras-marquez-2005-introduction} and CoNLL-12 English SRL~\cite{pradhan-etal-2013-towards}&#10;shared task datasets, and show that by learning to comply with declarative constraints,&#10;trained models can make more consistent and more accurate predictions. We&#10;instantiate our framework on top of a strong baseline system based on the&#10;RoBERTa~\cite{liu2019roberta} encoder, which by itself performs on par with previous best SRL models that are not ensembled.&#10;We evaluate the impact of three different types of&#10;constraints. Our experiments on the CoNLL-05 data show that our constrained&#10;models outperform the baseline system by $0.2$ F1 on the WSJ sectionand $1.2$&#10;F1 on the Brown test set. Even with the larger and cleaner CoNLL-12 data, our&#10;constrained models show improvements without introducing any additional&#10;trainable parameters. Finally, we also evaluate the effectiveness of our&#10;approach on low training data scenarios, and show that constraints can be more&#10;impactful when we do not have large training sets.&#10;&#10;In summary, our contributions are:&#10;\begin{enumerate}[nosep]&#10;\item We present a structured tuning framework for SRL which uses soft&#10; constraints to improve models without introducing additional trainable parameters.&#10;\item Our framework outperforms strong baseline systems, and shows especially large improvements&#10; in low data regimes.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Neural Network Models, Semantic Role Labeling, Mathematics, Constrained Decoding Mechanisms" />
        </attvalues>
      </node>
      <node id="1810.02245" label="1810.02245">
        <attvalues>
          <attvalue for="0" value="A Span Selection Model for Semantic Role Labeling" />
          <attvalue for="1" value="  We present a simple and accurate span-based model for semantic role labeling&#10;(SRL). Our model directly takes into account all possible argument spans and&#10;scores them for each label. At decoding time, we greedily select higher scoring&#10;labeled spans. One advantage of our model is to allow us to design and use&#10;span-level features, that are difficult to use in token-based BIO tagging&#10;approaches. Experimental results demonstrate that our ensemble model achieves&#10;the state-of-the-art results, 87.4 F1 and 87.0 F1 on the CoNLL-2005 and 2012&#10;datasets, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.04418" label="2004.04418">
        <attvalues>
          <attvalue for="0" value="On Optimal Transformer Depth for Low-Resource Language Translation" />
          <attvalue for="1" value="  Transformers have shown great promise as an approach to Neural Machine&#10;Translation (NMT) for low-resource languages. However, at the same time,&#10;transformer models remain difficult to optimize and require careful tuning of&#10;hyper-parameters to be useful in this setting. Many NMT toolkits come with a&#10;set of default hyper-parameters, which researchers and practitioners often&#10;adopt for the sake of convenience and avoiding tuning. These configurations,&#10;however, have been optimized for large-scale machine translation data sets with&#10;several millions of parallel sentences for European languages like English and&#10;French. In this work, we find that the current trend in the field to use very&#10;large models is detrimental for low-resource languages, since it makes training&#10;more difficult and hurts overall performance, confirming previous observations.&#10;We see our work as complementary to the Masakhane project (&quot;Masakhane&quot; means&#10;&quot;We Build Together&quot; in isiZulu.) In this spirit, low-resource NMT systems are&#10;now being built by the community who needs them the most. However, many in the&#10;community still have very limited access to the type of computational resources&#10;required for building extremely large models promoted by industrial research.&#10;Therefore, by showing that transformer models perform well (and often best) at&#10;low-to-moderate depth, we hope to convince fellow researchers to devote less&#10;computational resources, as well as time, to exploring overly large models&#10;during the development of these systems.&#10;" />
          <attvalue for="2" value="&#10;Transformers \cite{vaswani2017attention} have shown great promise as an approach to Neural Machine Translation (NMT) for low-resource languages \cite{abbott2018towards,martinus2019focus}. &#10;However, at the same time, transformer models remain difficult to optimize and require careful tuning of hyper-parameters to be useful in this setting \cite{popel2018training, nguyen2019transformers}. &#10;Many NMT toolkits come with a set of default hyper-parameters, which researchers and practitioners often adopt for the sake of convenience and avoiding tuning. These configurations, however, have been optimized for large-scale machine translation data sets with several millions of parallel sentences for European languages like English and French.&#10;&#10;In this work, we find that the current trend in the field to use very large models is detrimental for low-resource languages, since it makes training more difficult and hurts overall performance, confirming the observations by \cite{murray2019auto,Fan2019ReducingTD}.&#10;Specifically, we compare shallower networks to larger ones on three translation tasks, namely: translating from English to Setswana, Sepedi (Northern Sotho), and Afrikaans. We achieve a new state-of-the-art BLEU score \cite{papineni2002bleu} on some tasks (more than doubling the previous best score for Afrikaans) when using networks of appropriate depth. Furthermore, we provide a preliminary theoretical explanation for this effect on performance as a function of depth. Overall, our findings seem to advocate the use of shallow-to-moderately sized deep transformers for NMT for low-resource language translation.&#10;&#10;Our intuition concerning the relationship between performance and depth stems from prior work on signal propagation theory in noise-regularised neural networks \cite{schoenholz2016deep, pretorius2018critical}. &#10;Specifically, \cite{pretorius2018critical} showed that using Dropout \cite{srivastava2014dropout} limits the depth to which information can stably propagate through neural networks when using ReLU activations.&#10;Since both dropout and ReLU have been core components of the transformer since its inception \cite{vaswani2017attention}, this loss of information is likely to be taking place and should be taken into account when selecting the number of transformer layers.&#10;Although the architecture of a transformer is far more involved than those analysed by \cite{pretorius2018critical}, the fundamental building blocks remain the same. &#10;Thus, in this paper, we make use of the above theoretical insights as a guide to our analysis of depth's influence on performance in transformers.&#10;&#10;We see our work as complementary to the Masakhane project (``Masakhane'' means ``We Build Together'' in isiZulu.)&#10;In this spirit, low-resource NMT systems are now being built by the community who needs them the most. &#10;However, many in the community still have very limited access to the type of computational resources required for building extremely large models promoted by industrial research. &#10;Therefore, by showing that transformer models perform well (and often best) at low-to-moderate depth, we hope to convince fellow researchers to devote less computational resources, as well as time, to exploring overly large models during the development of these systems. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Low-Resource Languages, Mathematics, Neural Machine Translation, Model Optimization" />
        </attvalues>
      </node>
      <node id="1804.00247" label="1804.00247">
        <attvalues>
          <attvalue for="0" value="Training Tips for the Transformer Model" />
          <attvalue for="1" value="  This article describes our experiments in neural machine translation using&#10;the recent Tensor2Tensor framework and the Transformer sequence-to-sequence&#10;model (Vaswani et al., 2017). We examine some of the critical parameters that&#10;affect the final translation quality, memory usage, training stability and&#10;training time, concluding each experiment with a set of recommendations for&#10;fellow researchers. In addition to confirming the general mantra &quot;more data and&#10;larger models&quot;, we address scaling to multiple GPUs and provide practical tips&#10;for improved training regarding batch size, learning rate, warmup steps,&#10;maximum sentence length and checkpoint averaging. We hope that our observations&#10;will allow others to get better results given their particular hardware and&#10;data constraints.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.06717" label="1910.06717">
        <attvalues>
          <attvalue for="0" value="Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and&#10;  Performance for Low-Resource Machine Translation" />
          <attvalue for="1" value="  Neural sequence-to-sequence models, particularly the Transformer, are the&#10;state of the art in machine translation. Yet these neural networks are very&#10;sensitive to architecture and hyperparameter settings. Optimizing these&#10;settings by grid or random search is computationally expensive because it&#10;requires many training runs. In this paper, we incorporate architecture search&#10;into a single training run through auto-sizing, which uses regularization to&#10;delete neurons in a network over the course of training. On very low-resource&#10;language pairs, we show that auto-sizing can improve BLEU scores by up to 3.9&#10;points while removing one-third of the parameters from the model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.01232" label="1611.01232">
        <attvalues>
          <attvalue for="0" value="Deep Information Propagation" />
          <attvalue for="1" value="  We study the behavior of untrained neural networks whose weights and biases&#10;are randomly distributed using mean field theory. We show the existence of&#10;depth scales that naturally limit the maximum depth of signal propagation&#10;through these random networks. Our main practical result is to show that random&#10;networks may be trained precisely when information can travel through them.&#10;Thus, the depth scales that we identify provide bounds on how deep a network&#10;may be trained for a specific choice of hyperparameters. As a corollary to&#10;this, we argue that in networks at the edge of chaos, one of these depth scales&#10;diverges. Thus arbitrarily deep networks may be trained only sufficiently close&#10;to criticality. We show that the presence of dropout destroys the&#10;order-to-chaos critical point and therefore strongly limits the maximum&#10;trainable depth for random networks. Finally, we develop a mean field theory&#10;for backpropagation and we show that the ordered and chaotic phases correspond&#10;to regions of vanishing and exploding gradient respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.00293" label="1811.00293">
        <attvalues>
          <attvalue for="0" value="Critical initialisation for deep signal propagation in noisy rectifier&#10;  neural networks" />
          <attvalue for="1" value="  Stochastic regularisation is an important weapon in the arsenal of a deep&#10;learning practitioner. However, despite recent theoretical advances, our&#10;understanding of how noise influences signal propagation in deep neural&#10;networks remains limited. By extending recent work based on mean field theory,&#10;we develop a new framework for signal propagation in stochastic regularised&#10;neural networks. Our noisy signal propagation theory can incorporate several&#10;common noise distributions, including additive and multiplicative Gaussian&#10;noise as well as dropout. We use this framework to investigate initialisation&#10;strategies for noisy ReLU networks. We show that no critical initialisation&#10;strategy exists using additive noise, with signal propagation exploding&#10;regardless of the selected noise distribution. For multiplicative noise (e.g.&#10;dropout), we identify alternative critical initialisation strategies that&#10;depend on the second moment of the noise distribution. Simulations and&#10;experiments on real-world data confirm that our proposed initialisation is able&#10;to stably propagate signals in deep networks, while using an initialisation&#10;disregarding noise fails to do so. Furthermore, we analyse correlation dynamics&#10;between inputs. Stronger noise regularisation is shown to reduce the depth to&#10;which discriminatory information about the inputs to a noisy ReLU network is&#10;able to propagate, even when initialised at criticality. We support our&#10;theoretical predictions for these trainable depths with simulations, as well as&#10;with experiments on MNIST and CIFAR-10&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.03047" label="2203.03047">
        <attvalues>
          <attvalue for="0" value="Recent Advances in Neural Text Generation: A Task-Agnostic Survey" />
          <attvalue for="1" value="  In recent years, considerable research has been dedicated to the application&#10;of neural models in the field of natural language generation (NLG). The primary&#10;objective is to generate text that is both linguistically natural and&#10;human-like, while also exerting control over the generation process. This paper&#10;offers a comprehensive and task-agnostic survey of the recent advancements in&#10;neural text generation. These advancements have been facilitated through a&#10;multitude of developments, which we categorize into four key areas: data&#10;construction, neural frameworks, training and inference strategies, and&#10;evaluation metrics. By examining these different aspects, we aim to provide a&#10;holistic overview of the progress made in the field. Furthermore, we explore&#10;the future directions for the advancement of neural text generation, which&#10;encompass the utilization of neural pipelines and the incorporation of&#10;background knowledge. These avenues present promising opportunities to further&#10;enhance the capabilities of NLG systems. Overall, this survey serves to&#10;consolidate the current state of the art in neural text generation and&#10;highlights potential avenues for future research and development in this&#10;dynamic field.&#10;" />
          <attvalue for="2" value="&#10;Natural Language Generation (NLG) is a highly challenging sub-field of Natural Language Processing (NLP)~\cite{reiter_dale_2000}, which incorporates knowledge (e.g. text, pictures, audio, tables, etc.) and generates corresponding task-oriented text as output, e.g. weather forecast reports. NLG has a range of applications~\cite{gatt2018survey,santhanam2019survey}.&#10;NLG is traditionally tackled by symbolic and rule-based approaches, but in recent years deep learning techniques have attracted a great amount of interest \cite{belinkov2019analysis}.&#10;Both approaches have their strengths, and it is possible that a neurosymbolic approach will dominate in future. This survey will focus on recent advances in the neural approach.&#10;&#10;Existing survey papers usually summarize one of the NLG applications such as Story Generation \cite{hou2019survey,alhussain2021automatic,tang-etal-2022-ngep,huang-etal-2022-improving}, Text summarisation \cite{suleiman2020deep,el2021automatic}, Dialogue~\cite{ni2021recent,tang2022terminology,zhanga2023cadge}, Machine Translation~\cite{yang2020survey,dabre2020survey}, etc~\cite{loakman2023phonetically}. Only a few studies~\cite{lu2018neural,chandu2020positioning,jin2020recent,dong2021survey} discuss the development of the whole NLG area.&#10;&#10;Task-specific surveys are beneficial, but the survey of the whole NLG area could give broader ideas as inspiration for various applications using generative techniques, as the surveys like \cite{belinkov2019analysis,rogers2020primer} did. &#10;To our knowledge, this is the first comprehensive survey of neural text generation which summarises the commonalities and trends of recent advances.&#10;&#10;From the perspective of neural text generation, various NLG applications have task-agnostic commonalities: (\romannumeral1) Unlike traditional systems, neural networks capture features without ad-hoc feature engineering \cite{belinkov2019analysis}; (\romannumeral2) Various neural generative frameworks mostly use similar encoder-decoder architectures, so they have common modules and training (or inference) strategies; (\romannumeral3) Evaluation metrics also have good generalization in NLG. Therefore, neural text generation has similar challenges \cite{chandu2020positioning,thomson2021generation} and solutions to analyze.&#10;&#10;In this survey, we provide an overview of neural text generation via summarising the papers mainly published within the last 5 years. We compartmentalize and analyze the contributions of these papers according to 4 aspects: \S\ref{sec:2} probes the characteristics of data construction for NLG tasks. \S\ref{sec:3} summarizes common deep learning techniques used in neural generative frameworks. In \S\ref{sec:4}, we analyze training and inference strategies of neural frameworks. \S\ref{sec:5} reviews and categorizes existing evaluation metrics for text generation. Following our analysis of recent advances, we discuss the future directions of research on Neural text generation including developing neural pipelines and exploiting background knowledge.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Generation Techniques, Computer Science, Linguistics, Cognitive Science, Neural Network Models, Artificial Intelligence, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1812.08951" label="1812.08951">
        <attvalues>
          <attvalue for="0" value="Analysis Methods in Neural Language Processing: A Survey" />
          <attvalue for="1" value="  The field of natural language processing has seen impressive progress in&#10;recent years, with neural network models replacing many of the traditional&#10;systems. A plethora of new models have been proposed, many of which are thought&#10;to be opaque compared to their feature-rich counterparts. This has led&#10;researchers to analyze, interpret, and evaluate neural networks in novel and&#10;more fine-grained ways. In this survey paper, we review analysis methods in&#10;neural language processing, categorize them according to prominent research&#10;trends, highlight existing limitations, and point to potential directions for&#10;future work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.04387" label="2105.04387">
        <attvalues>
          <attvalue for="0" value="Recent Advances in Deep Learning Based Dialogue Systems: A Systematic&#10;  Survey" />
          <attvalue for="1" value="  Dialogue systems are a popular natural language processing (NLP) task as it&#10;is promising in real-life applications. It is also a complicated task since&#10;many NLP tasks deserving study are involved. As a result, a multitude of novel&#10;works on this task are carried out, and most of them are deep learning based&#10;due to the outstanding performance. In this survey, we mainly focus on the deep&#10;learning based dialogue systems. We comprehensively review state-of-the-art&#10;research outcomes in dialogue systems and analyze them from two angles: model&#10;type and system type. Specifically, from the angle of model type, we discuss&#10;the principles, characteristics, and applications of different models that are&#10;widely used in dialogue systems. This will help researchers acquaint these&#10;models and see how they are applied in state-of-the-art frameworks, which is&#10;rather helpful when designing a new dialogue system. From the angle of system&#10;type, we discuss task-oriented and open-domain dialogue systems as two streams&#10;of research, providing insight into the hot topics related. Furthermore, we&#10;comprehensively review the evaluation methods and datasets for dialogue systems&#10;to pave the way for future research. Finally, some possible research trends are&#10;identified based on the recent research outcomes. To the best of our knowledge,&#10;this survey is the most comprehensive and up-to-date one at present for deep&#10;learning based dialogue systems, extensively covering the popular techniques.&#10;We speculate that this work is a good starting point for academics who are new&#10;to the dialogue systems or those who want to quickly grasp up-to-date&#10;techniques in this area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.15551" label="2210.15551">
        <attvalues>
          <attvalue for="0" value="Terminology-aware Medical Dialogue Generation" />
          <attvalue for="1" value="  Medical dialogue generation aims to generate responses according to a history&#10;of dialogue turns between doctors and patients. Unlike open-domain dialogue&#10;generation, this requires background knowledge specific to the medical domain.&#10;Existing generative frameworks for medical dialogue generation fall short of&#10;incorporating domain-specific knowledge, especially with regard to medical&#10;terminology. In this paper, we propose a novel framework to improve medical&#10;dialogue generation by considering features centered on domain-specific&#10;terminology. We leverage an attention mechanism to incorporate terminologically&#10;centred features, and fill in the semantic gap between medical background&#10;knowledge and common utterances by enforcing language models to learn&#10;terminology representations with an auxiliary terminology recognition task.&#10;Experimental results demonstrate the effectiveness of our approach, in which&#10;our proposed framework outperforms SOTA language models. Additionally, we&#10;provide a new dataset with medical terminology annotations to support the&#10;research on medical dialogue generation. Our dataset and code are available at&#10;https://github.com/tangg555/meddialog.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.06294" label="2305.06294">
        <attvalues>
          <attvalue for="0" value="CADGE: Context-Aware Dialogue Generation Enhanced with Graph-Structured&#10;  Knowledge Aggregation" />
          <attvalue for="1" value="  Commonsense knowledge is crucial to many natural language processing tasks.&#10;Existing works usually incorporate graph knowledge with conventional graph&#10;neural networks (GNNs), leading to the text and graph knowledge encoding&#10;processes being separated in a serial pipeline. We argue that these separate&#10;representation learning stages may be suboptimal for neural networks to learn&#10;the overall context contained in both types of input knowledge. In this paper,&#10;we propose a novel context-aware graph-attention model (Context-aware GAT),&#10;which can effectively incorporate global features of relevant knowledge graphs&#10;based on a context-enhanced knowledge aggregation process. Specifically, our&#10;framework leverages a novel representation learning approach to process&#10;heterogeneous features - combining flattened graph knowledge with text. To the&#10;best of our knowledge, this is the first attempt at hierarchically applying&#10;graph knowledge aggregation on a connected subgraph in addition to contextual&#10;information to support commonsense dialogue generation. This framework shows&#10;superior performance compared to conventional GNN-based language frameworks.&#10;Both automatic and human evaluation demonstrates that our proposed model has&#10;significant performance uplifts over state-of-the-art baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.07133" label="1803.07133">
        <attvalues>
          <attvalue for="0" value="Neural Text Generation: Past, Present and Beyond" />
          <attvalue for="1" value="  This paper presents a systematic survey on recent development of neural text&#10;generation models. Specifically, we start from recurrent neural network&#10;language models with the traditional maximum likelihood estimation training&#10;scheme and point out its shortcoming for text generation. We thus introduce the&#10;recently proposed methods for text generation based on reinforcement learning,&#10;re-parametrization tricks and generative adversarial nets (GAN) techniques. We&#10;compare different properties of these models and the corresponding techniques&#10;to handle their common problems such as gradient vanishing and generation&#10;diversity. Finally, we conduct a benchmarking experiment with different types&#10;of neural text generation models on two well-known datasets and discuss the&#10;empirical results along with the aforementioned model properties.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.07279" label="2010.07279">
        <attvalues>
          <attvalue for="0" value="Positioning yourself in the maze of Neural Text Generation: A&#10;  Task-Agnostic Survey" />
          <attvalue for="1" value="  Neural text generation metamorphosed into several critical natural language&#10;applications ranging from text completion to free form narrative generation. In&#10;order to progress research in text generation, it is critical to absorb the&#10;existing research works and position ourselves in this massively growing field.&#10;Specifically, this paper surveys the fundamental components of modeling&#10;approaches relaying task agnostic impacts across various generation tasks such&#10;as storytelling, summarization, translation etc., In this context, we present&#10;an abstraction of the imperative techniques with respect to learning paradigms,&#10;pretraining, modeling approaches, decoding and the key challenges outstanding&#10;in the field in each of them. Thereby, we deliver a one-stop destination for&#10;researchers in the field to facilitate a perspective on where to situate their&#10;work and how it impacts other closely related generation tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.11739" label="2112.11739">
        <attvalues>
          <attvalue for="0" value="A Survey of Natural Language Generation" />
          <attvalue for="1" value="  This paper offers a comprehensive review of the research on Natural Language&#10;Generation (NLG) over the past two decades, especially in relation to&#10;data-to-text generation and text-to-text generation deep learning methods, as&#10;well as new applications of NLG technology. This survey aims to (a) give the&#10;latest synthesis of deep learning research on the NLG core tasks, as well as&#10;the architectures adopted in the field; (b) detail meticulously and&#10;comprehensively various NLG tasks and datasets, and draw attention to the&#10;challenges in NLG evaluation, focusing on different evaluation methods and&#10;their relationships; (c) highlight some future emphasis and relatively recent&#10;research issues that arise due to the increasing synergy between NLG and other&#10;artificial intelligence areas, such as computer vision, text and computational&#10;creativity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.05644" label="2108.05644">
        <attvalues>
          <attvalue for="0" value="Generation Challenges: Results of the Accuracy Evaluation Shared Task" />
          <attvalue for="1" value="  The Shared Task on Evaluating Accuracy focused on techniques (both manual and&#10;automatic) for evaluating the factual accuracy of texts produced by neural NLG&#10;systems, in a sports-reporting domain. Four teams submitted evaluation&#10;techniques for this task, using very different approaches and techniques. The&#10;best-performing submissions did encouragingly well at this difficult task.&#10;However, all automatic submissions struggled to detect factual errors which are&#10;semantically or pragmatically complex (for example, based on incorrect&#10;computation or inference).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.12017" label="2301.12017">
        <attvalues>
          <attvalue for="0" value="Understanding INT4 Quantization for Transformer Models: Latency Speedup,&#10;  Composability, and Failure Cases" />
          <attvalue for="1" value="  Improving the deployment efficiency of transformer-based language models has&#10;been challenging given their high computation and memory cost. While INT8&#10;quantization has recently been shown to be effective in reducing both the&#10;memory cost and latency while preserving model accuracy, it remains unclear&#10;whether we can leverage INT4 (which doubles peak hardware throughput) to&#10;achieve further latency improvement. In this study, we explore the feasibility&#10;of employing INT4 weight and activation (W4A4) quantization for language&#10;models. Our findings indicate that W4A4 quantization introduces no to&#10;negligible accuracy degradation for encoder-only and encoder-decoder models,&#10;but causes a significant accuracy drop for decoder-only models. To materialize&#10;the performance gain using W4A4, we develop a highly optimized end-to-end W4A4&#10;encoder inference pipeline supporting different quantization strategies. Our&#10;INT4 pipeline is $8.5\times$ faster for latency-oriented scenarios and up to&#10;$3\times$ for throughput-oriented scenarios compared to the inference of FP16,&#10;and improves the SOTA BERT INT8 performance from FasterTransformer by up to&#10;$1.7\times$. We provide insights into the failure cases when applying W4A4 to&#10;decoder-only models, and further explore the compatibility of INT4 quantization&#10;with other compression methods, like pruning and layer reduction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.05202" label="2002.05202">
        <attvalues>
          <attvalue for="0" value="GLU Variants Improve Transformer" />
          <attvalue for="1" value="  Gated Linear Units (arXiv:1612.08083) consist of the component-wise product&#10;of two linear projections, one of which is first passed through a sigmoid&#10;function. Variations on GLU are possible, using different nonlinear (or even&#10;linear) functions in place of sigmoid. We test these variants in the&#10;feed-forward sublayers of the Transformer (arXiv:1706.03762)&#10;sequence-to-sequence model, and find that some of them yield quality&#10;improvements over the typically-used ReLU or GELU activations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.04045" label="2203.04045">
        <attvalues>
          <attvalue for="0" value="Towards Generalized Models for Task-oriented Dialogue Modeling on Spoken&#10;  Conversations" />
          <attvalue for="1" value="  Building robust and general dialogue models for spoken conversations is&#10;challenging due to the gap in distributions of spoken and written data. This&#10;paper presents our approach to build generalized models for the&#10;Knowledge-grounded Task-oriented Dialogue Modeling on Spoken Conversations&#10;Challenge of DSTC-10. In order to mitigate the discrepancies between spoken and&#10;written text, we mainly employ extensive data augmentation strategies on&#10;written data, including artificial error injection and round-trip text-speech&#10;transformation. To train robust models for spoken conversations, we improve&#10;pre-trained language models, and apply ensemble algorithms for each sub-task.&#10;Typically, for the detection task, we fine-tune \roberta and ELECTRA, and run&#10;an error-fixing ensemble algorithm. For the selection task, we adopt a&#10;two-stage framework that consists of entity tracking and knowledge ranking, and&#10;propose a multi-task learning method to learn multi-level semantic information&#10;by domain classification and entity selection. For the generation task, we&#10;adopt a cross-validation data process to improve pre-trained generative&#10;language models, followed by a consensus decoding algorithm, which can add&#10;arbitrary features like relative \rouge metric, and tune associated feature&#10;weights toward \bleu directly. Our approach ranks third on the objective&#10;evaluation and second on the final official human evaluation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Although promising results have been achieved by dialogue systems on written conversations, using them directly on spoken conversations is challenging due to the differences in data distribution, including the discrepancy between writing and speaking, and the extra noises from speech recognition errors. In Dialog System Technology Challenges 10 (DSTC-10), the sub-track 2 of the ``Knowledge-grounded Task-oriented Dialogue Modeling on Spoken Conversations'' task proposes such a challenge, in which models are evaluated on spoken conversations while no spoken training data is provided. Therefore, it is crucial to build robust dialogue models with high generalization ability. &#10;&#10;The task extends the DSTC-9 track 1~\cite{kim2020domain} from written conversations to spoken conversations, where cross-domain dialogue agents are built to &#10;answer questions that cannot be solved with only domain APIs. Instead, system agents have to retrieve related question-answer (QA) pairs from an unstructured FAQ database, and generate a natural response based on the retrieved QA pair(s).&#10;Thus, our tasks include (1) finding knowledge-seeking turns; (2) returning ranked QA pairs for each knowledge-seeking turn; and (3) generating a system response given QA pairs from task (2) and the dialogue history.&#10;&#10;In this paper, we build generalized models for the task in the following ways.&#10;First, to bridge the gap between writing and speaking, we employ various data augmentation methods to expand the training set, including artificial error injection and round-trip text-speech transformation. &#10;Second, we use pre-trained language models (e.g. RoBERTa~\cite{roberta}, ELECTRA~\cite{clark2020electra}, and UniLM~\cite{unilm}), and design different ensemble algorithms for each sub-task.&#10;Third, for the selection task, we propose a multi-task learning mechanism to enhance models' ability to learn multi-level semantic information. &#10;We also introduce artificial sparse features to explicitly capture informative attributes of the candidate knowledge.&#10;For the generation task, we mainly follow Mi et al.~\shortcite{mi2021towards}, and directly use their online sampling, and consensus decoding algorithms~\cite{pauls2009consensus}. &#10;&#10;In particular, we extend the work of Mi et al.~\shortcite{mi2021towards}, and make the following extra contributions in this paper. &#10;\begin{itemize}&#10; \item Data augmentation. We augment written data by injecting artificially-generated errors based on phonetic similarity, converting the original texts into sound waves by a text-to-speech (TTS) model and then transforming back into texts by an automated speech recognition (ASR) model, and splitting or inserting entity names in dialogues.&#10; \item Multi-task learning. We propose a multi-task learning method for knowledge ranking, in which a domain classification task and an entity selection task are assigned to learn multi-level semantic information.&#10; \item Incremental improvements. We apply more pre-trained language models, e.g. ELECTRA, to increase the modeling diversity. &#10; We also introduce additional artificial sparse features to explicitly capture informative attributes of knowledge snippets.&#10;\end{itemize}&#10;&#10;Our system achieves the third best in the official objective evaluation, and the second best on human evaluation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Model Training, Spoken Conversation Analysis, Dialogue Modeling, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2006.03533" label="2006.03533">
        <attvalues>
          <attvalue for="0" value="Beyond Domain APIs: Task-oriented Conversational Modeling with&#10;  Unstructured Knowledge Access" />
          <attvalue for="1" value="  Most prior work on task-oriented dialogue systems are restricted to a limited&#10;coverage of domain APIs, while users oftentimes have domain related requests&#10;that are not covered by the APIs. In this paper, we propose to expand coverage&#10;of task-oriented dialogue systems by incorporating external unstructured&#10;knowledge sources. We define three sub-tasks: knowledge-seeking turn detection,&#10;knowledge selection, and knowledge-grounded response generation, which can be&#10;modeled individually or jointly. We introduce an augmented version of MultiWOZ&#10;2.1, which includes new out-of-API-coverage turns and responses grounded on&#10;external knowledge sources. We present baselines for each sub-task using both&#10;conventional and neural approaches. Our experimental results demonstrate the&#10;need for further research in this direction to enable more informative&#10;conversational systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.08621" label="2307.08621">
        <attvalues>
          <attvalue for="0" value="Retentive Network: A Successor to Transformer for Large Language Models" />
          <attvalue for="1" value="  In this work, we propose Retentive Network (RetNet) as a foundation&#10;architecture for large language models, simultaneously achieving training&#10;parallelism, low-cost inference, and good performance. We theoretically derive&#10;the connection between recurrence and attention. Then we propose the retention&#10;mechanism for sequence modeling, which supports three computation paradigms,&#10;i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel&#10;representation allows for training parallelism. The recurrent representation&#10;enables low-cost $O(1)$ inference, which improves decoding throughput, latency,&#10;and GPU memory without sacrificing performance. The chunkwise recurrent&#10;representation facilitates efficient long-sequence modeling with linear&#10;complexity, where each chunk is encoded parallelly while recurrently&#10;summarizing the chunks. Experimental results on language modeling show that&#10;RetNet achieves favorable scaling results, parallel training, low-cost&#10;deployment, and efficient inference. The intriguing properties make RetNet a&#10;strong successor to Transformer for large language models. Code will be&#10;available at https://aka.ms/retnet.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformer~\cite{transformer} has become the de facto architecture for large language models~\cite{gpt3}, which was initially proposed to overcome the sequential training issue of recurrent models~\cite{lstm}.&#10;However, training parallelism of Transformers is at the cost of inefficient inference, because of the $O(N)$ complexity per step and memory-bound key-value cache~\cite{multiquery}, which renders Transformers unfriendly to deployment.&#10;The growing sequence length increases GPU memory consumption as well as latency and reduces inference speed.&#10;&#10;Numerous efforts have continued to develop the next-generation architecture, aiming at retaining training parallelism and competitive performance as Transformers while having efficient $O(1)$ inference.&#10;It is challenging to achieve the above goals simultaneously, i.e., the so-called ``impossible triangle'' as shown in \ref{fig:triangle}.&#10;&#10;There have been three main strands of research.&#10;First, linearized attention~\cite{linear-transformer} approximates standard attention scores $\exp(\vq \cdot \vk)$ with kernels $\phi(\vq) \cdot \phi(\vk)$, so that autoregressive inference can be rewritten in a recurrent form.&#10;However, the modeling capability and performance are worse than Transformers, which hinders the method's popularity.&#10;The second strand returns to recurrent models for efficient inference while sacrificing training parallelism.&#10;As a remedy, element-wise operators~\cite{rwkv} are used for acceleration, however, representation capacity and performance are harmed.&#10;The third line of research explores replacing attention with other mechanisms, such as S4~\cite{s4}, and its variants~\cite{h3,hyena}.&#10;None of the previous work can break through the impossible triangle, resulting in no clear winner compared with Transformers.&#10;&#10;In this work, we propose retentive networks (\our), achieving low-cost inference, efficient long-sequence modeling, Transformer-comparable performance, and parallel model training simultaneously.&#10;Specifically, we introduce a multi-scale retention mechanism to substitute multi-head attention, which has three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent representations.&#10;First, the parallel representation empowers training parallelism to utilize GPU devices fully.&#10;Second, the recurrent representation enables efficient $O(1)$ inference in terms of memory and computation. The deployment cost and latency can be significantly reduced. Moreover, the implementation is greatly simplified without key-value cache tricks.&#10;Third, the chunkwise recurrent representation can perform efficient long-sequence modeling. We parallelly encode each local block for computation speed while recurrently encoding the global blocks to save GPU memory.&#10;&#10;We conduct extensive experiments to compare \our with Transformer and its variants.&#10;Experimental results on language modeling show that \our is consistently competitive in terms of both scaling curves and in-context learning.&#10;Moreover, the inference cost of \our is length-invariant.&#10;For a 7B model and 8k sequence length, \our decodes 8.4$\times$ faster and saves 70\% of memory than Transformers with key-value caches.&#10;During training, \our also achieves 25-50\% memory saving and 7$\times$ acceleration than standard Transformer and an advantage towards highly-optimized FlashAttention~\cite{flashattention}.&#10;Besides, \our's inference latency is insensitive to batch size, allowing enormous throughput.&#10;The intriguing properties make \our a strong successor to Transformer for large language models.&#10;&#10;&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Parallel Computing, Linguistics, Mathematics, Language Modeling" />
        </attvalues>
      </node>
      <node id="2404.06411" label="2404.06411">
        <attvalues>
          <attvalue for="0" value="AgentQuest: A Modular Benchmark Framework to Measure Progress and&#10;  Improve LLM Agents" />
          <attvalue for="1" value="  The advances made by Large Language Models (LLMs) have led to the pursuit of&#10;LLM agents that can solve intricate, multi-step reasoning tasks. As with any&#10;research pursuit, benchmarking and evaluation are key corner stones to&#10;efficient and reliable progress. However, existing benchmarks are often narrow&#10;and simply compute overall task success. To face these issues, we propose&#10;AgentQuest -- a framework where (i) both benchmarks and metrics are modular and&#10;easily extensible through well documented and easy-to-use APIs; (ii) we offer&#10;two new evaluation metrics that can reliably track LLM agent progress while&#10;solving a task. We exemplify the utility of the metrics on two use cases&#10;wherein we identify common failure points and refine the agent architecture to&#10;obtain a significant performance increase. Together with the research&#10;community, we hope to extend AgentQuest further and therefore we make it&#10;available under https://github.com/nec-research/agentquest.&#10;" />
          <attvalue for="2" value="&#10;&#10;Generative Agents~\cite{kiela2023saturation} are software systems that leverage foundation models like Large Language Models (LLMs) to perform complex tasks, take decisions, devise multi-steps plans and use tools (API calls, coding, etc.) to build solutions in heterogeneous contexts~\cite{wang2023survey,weng2023prompt}.&#10;The potential ability to solve heterogeneous tasks with high degrees of autonomy has catalysed the interest of both research and industrial communities. Nonetheless, it is still unclear to which extent current systems are successfully able to fulfil their promises. In fact, methodologies to benchmark, evaluate and advance these systems are still in their early days.&#10;&#10;We identify a couple of gaps. &#10;Firstly, benchmarking agents requires combining different benchmark types~\cite{liu2023agentbench,chalamalasetti2023clembench}. For example, some benchmarks focus on specific capabilities and provide gaming environments, which we refer to as ``closed-box'' -- i.e. with a finite set of actions~\cite{liu2023agentbench, patil2023gorilla,chalamalasetti2023clembench} -- whereas other benchmarks provide open-ended tasks and access to general tools, like web browsing~\cite{zhuang2023toolqa, zheng2023judging,mialon2023gaia}. As benchmarks are developed independently, significant effort goes into custom integration of new agent architectures with each benchmark.&#10;&#10;Secondly, and more critically, existing benchmarks mostly focus on providing a success rate measure, i.e. a binary success/fail evaluation for each of the proposed tasks. &#10;While success rate is helpful to measure overall advances of an agent technology, it has limited use in guiding improvements for new generative agent architectures. Here, it is important to consider that generative agents often combine foundation models with multiple other components, such as memory and tools. Developers can reason about these individual components in terms of architecture and their inter-dependence, and could actively change and evolve them using deeper insights about how an agent performs in a benchmark. That is, developers need benchmarks to both evaluate and debug agents.&#10;&#10;For example, current benchmarks make it hard to answer questions like does the agent fail completely the tasks or does it partially solve them? Does the agent fail consistently at a certain step? Would extra run time lead to finding a solution?&#10;Answering these questions would require tracing and inspecting the execution of the agent.&#10;We argue that providing a more efficient approach that is consistent over multiple benchmarks is a stepping stone towards evolving generative agents.&#10;&#10;We address these gaps introducing AgentQuest, a modular framework to support multiple diverse benchmarks and agent architectures (See Figure~\ref{fig:interaction}), alongside with two new metrics -- i.e. progress rate and repetition rate -- to debug an agent architecture behaviour. &#10;AgentQuest defines a standard interface to connect an arbitrary agent architecture with diverse benchmarks, and to compute progress and repetition rates from them. &#10;&#10;We showcase the framework, implementing 4 benchmarks in AgentQuest: ALFWorld~\cite{shridhar2020alfworld}, Lateral Thinking Puzzles~\cite{sloane1992lateral}, Mastermind and Sudoku. The latter two are newly introduced with AgentQuest.&#10;Additional benchmarks can be easily added, while requiring no changes to the tested agents.&#10;&#10;Our final contribution is to present our experience leveraging the proposed metrics to debug and improve existing agent architectures as implemented in LangChain~\cite{chase2022langchain}. In particular, we show that in the Mastermind benchmark the combination of progress rate and repetition rate identifies a limitation in the ability of the agent to explore the full space of potential solutions. Guided by this insight we could improve the success rate in this benchmark by up to $\approx$20\%.&#10;In Lateral Thinking Puzzles we show that partially repeating actions is part of the agent strategy, whereas in ALFWorld, we show that monitoring the progress rate makes it possible to identify that the final success rate is limited by the allowed runtime of the agent, and that more steps lead to a better performance. &#10;Finally, in the Sudoku benchmark, we show that the low success rate is actually paired with low progress rate, making clear that the tested agent is unable to solve this type of tasks.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Artificial Intelligence Evaluation, Artificial Intelligence, Mathematics, Natural Language Processing, Benchmarking Frameworks" />
        </attvalues>
      </node>
      <node id="2308.03688" label="2308.03688">
        <attvalues>
          <attvalue for="0" value="AgentBench: Evaluating LLMs as Agents" />
          <attvalue for="1" value="  Large Language Models (LLMs) are becoming increasingly smart and autonomous,&#10;targeting real-world pragmatic missions beyond traditional NLP tasks. As a&#10;result, there has been an urgent need to evaluate LLMs as agents on challenging&#10;tasks in interactive environments. We present AgentBench, a multi-dimensional&#10;evolving benchmark that currently consists of 8 distinct environments to assess&#10;LLM-as-Agent's reasoning and decision-making abilities in a multi-turn&#10;open-ended generation setting. Our extensive test over 27 API-based and&#10;open-sourced (OSS) LLMs shows that, while top commercial LLMs present a strong&#10;ability of acting as agents in complex environments, there is a significant&#10;disparity in performance between them and OSS competitors. We identify the&#10;typical reasons of failures in environments and LLMs, showing that poor&#10;long-term reasoning, decision-making, and instruction following abilities are&#10;the main obstacles for developing usable LLM agents. Training on code and high&#10;quality multi-turn alignment data could improve agent performance. Datasets,&#10;environments, and an integrated evaluation package for AgentBench are released&#10;at \url{https://github.com/THUDM/AgentBench}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.13304" label="2306.13304">
        <attvalues>
          <attvalue for="0" value="ToolQA: A Dataset for LLM Question Answering with External Tools" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated impressive performance in&#10;various NLP tasks, but they still suffer from challenges such as hallucination&#10;and weak numerical reasoning. To overcome these challenges, external tools can&#10;be used to enhance LLMs' question-answering abilities. However, current&#10;evaluation methods do not distinguish between questions that can be answered&#10;using LLMs' internal knowledge and those that require external information&#10;through tool use. To address this issue, we introduce a new dataset called&#10;ToolQA, which is designed to faithfully evaluate LLMs' ability to use external&#10;tools for question answering. Our development of ToolQA involved a scalable,&#10;automated process for dataset curation, along with 13 specialized tools&#10;designed for interaction with external knowledge in order to answer questions.&#10;Importantly, we strive to minimize the overlap between our benchmark data and&#10;LLMs' pre-training data, enabling a more precise evaluation of LLMs' tool-use&#10;reasoning abilities. We conducted an in-depth diagnosis of existing tool-use&#10;LLMs to highlight their strengths, weaknesses, and potential improvements. Our&#10;findings set a new benchmark for evaluating LLMs and suggest new directions for&#10;future advancements. Our data and code are freely available to the broader&#10;scientific community on GitHub.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.12983" label="2311.12983">
        <attvalues>
          <attvalue for="0" value="GAIA: a benchmark for General AI Assistants" />
          <attvalue for="1" value="  We introduce GAIA, a benchmark for General AI Assistants that, if solved,&#10;would represent a milestone in AI research. GAIA proposes real-world questions&#10;that require a set of fundamental abilities such as reasoning, multi-modality&#10;handling, web browsing, and generally tool-use proficiency. GAIA questions are&#10;conceptually simple for humans yet challenging for most advanced AIs: we show&#10;that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins.&#10;This notable performance disparity contrasts with the recent trend of LLMs&#10;outperforming humans on tasks requiring professional skills in e.g. law or&#10;chemistry. GAIA's philosophy departs from the current trend in AI benchmarks&#10;suggesting to target tasks that are ever more difficult for humans. We posit&#10;that the advent of Artificial General Intelligence (AGI) hinges on a system's&#10;capability to exhibit similar robustness as the average human does on such&#10;questions. Using GAIA's methodology, we devise 466 questions and their answer.&#10;We release our questions while retaining answers to 300 of them to power a&#10;leader-board available at https://huggingface.co/gaia-benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.04761" label="2301.04761">
        <attvalues>
          <attvalue for="0" value="NarrowBERT: Accelerating Masked Language Model Pretraining and Inference" />
          <attvalue for="1" value="  Large-scale language model pretraining is a very successful form of&#10;self-supervised learning in natural language processing, but it is increasingly&#10;expensive to perform as the models and pretraining corpora have become larger&#10;over time. We propose NarrowBERT, a modified transformer encoder that increases&#10;the throughput for masked language model pretraining by more than $2\times$.&#10;NarrowBERT sparsifies the transformer model such that the self-attention&#10;queries and feedforward layers only operate on the masked tokens of each&#10;sentence during pretraining, rather than all of the tokens as with the usual&#10;transformer encoder. We also show that NarrowBERT increases the throughput at&#10;inference time by as much as $3.5\times$ with minimal (or no) performance&#10;degradation on sentence encoding tasks like MNLI. Finally, we examine the&#10;performance of NarrowBERT on the IMDB and Amazon reviews classification and&#10;CoNLL NER tasks and show that it is also comparable to standard BERT&#10;performance.&#10;" />
          <attvalue for="2" value="&#10;Pretrained masked language models, such as BERT \cite{devlins2019bert}, RoBERTa \cite{Liu2019RoBERTaAR}, and DeBERTa \cite{deberta}, have pushed the state-of-the-art on a wide range of downstream tasks in natural language processing.&#10;At their core is the transformer architecture \cite{Vaswani2017AttentionIA} that consists of interleaved self-attention and feedforward sublayers.&#10;Since the former sublayer implies quadratic time complexity in the input sequence length \cite{Vaswani2017AttentionIA}, many have proposed methods to make the self-attention computation more efficient \cite[inter alia]{katharopoulos-et-al-2020,performer,wang2020linformer,RFA,peng-etal-2022-abc}.&#10;&#10;In this work, we explore an orthogonal approach to efficiency: can we make masked language models efficient by reducing the length of the input sequence that each layer needs to process?&#10;In particular, pretraining by masked language modeling only involves prediction of masked tokens (typically, only 15\% of the input tokens; \cite{devlins2019bert,Liu2019RoBERTaAR}).&#10;Despite this sparse pretraining objective, each transformer layer computes a representation for every token.&#10;In addition to pretraining, many downstream applications only use a single vector representation (i.e., only the [CLS] token) for prediction purposes, which is much smaller than the number of input tokens (e.g., sequence classification tasks as in GLUE/SuperGLUE; \cite{wang-etal-2018-glue,superglue}).&#10;By narrowing the input sequence for transformer layers, we can accelerate both pretraining and inference.&#10;&#10;We present NarrowBERT, a new architecture that takes advantage of the sparsity in the training objective. We present two NarrowBERT methods in the sections that follow (Figure \ref{fig:narrowbert}). We provide the code to reproduce our experiments at.&#10;The first method reduces the input sequence for the feedforward sublayers by reordering the interleaved self-attention and feedforward sublayers in the standard transformer architecture \cite{press-etal-2020-improving}: after two standard, interleaved transformer layers, self-attention sublayers are first applied, followed only by feedforward sublayers.&#10;This way, the feedforward sublayer computations are only performed for masked tokens, resulting in a $1.3\times$ speedup in pretraining (\S\ref{section:experiments}).&#10;The second approach reduces the input length to the attention sublayers: queries are only computed for masked tokens in the attention mechanism \cite{Bahdanau2014NeuralMT}, while the keys and values are not recomputed for non-masked tokens, which leads to a greater than $2\times$ speedup in pretraining.&#10;&#10;We extensively evaluate our efficient pretrained models on well-established downstream tasks (e.g., \cite{wang-etal-2018-glue, tjong-kim-sang-de-meulder-2003-introduction}.)&#10;We find that our modifications result in almost no drop in downstream performance, while providing substantial pretraining and inference speedups (\S\ref{section:experiments}).&#10;While efficient attention variants are a promising research direction, this work presents a different and simple approach to making transformers efficient, with minimal changes in architecture.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Language Model Pretraining, Mathematics, Natural Language Processing, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="2210.09934" label="2210.09934">
        <attvalues>
          <attvalue for="0" value="A Simple and Effective Method to Improve Zero-Shot Cross-Lingual&#10;  Transfer Learning" />
          <attvalue for="1" value="  Existing zero-shot cross-lingual transfer methods rely on parallel corpora or&#10;bilingual dictionaries, which are expensive and impractical for low-resource&#10;languages. To disengage from these dependencies, researchers have explored&#10;training multilingual models on English-only resources and transferring them to&#10;low-resource languages. However, its effect is limited by the gap between&#10;embedding clusters of different languages. To address this issue, we propose&#10;Embedding-Push, Attention-Pull, and Robust targets to transfer English&#10;embeddings to virtual multilingual embeddings without semantic loss, thereby&#10;improving cross-lingual transferability. Experimental results on mBERT and&#10;XLM-R demonstrate that our method significantly outperforms previous works on&#10;the zero-shot cross-lingual text classification task and can obtain a better&#10;multilingual alignment.&#10;" />
          <attvalue for="2" value="&#10;In recent years, advances in multilingual models such as mBERT \cite{devlin2019bert}, XLM \cite{conneau2019cross}, XLM-R \cite{conneauetal2020unsupervised}, etc., after being fine-tuned with annotated data, have enabled significant improvements in many cross-lingual tasks. However, due to the lack of annotated data, some tasks in low-resource languages have not enjoyed this technological advancement. To solve this issue, the academic and industrial community began to focus on zero-shot cross-lingual transfer learning \cite{huang-etal-2019-unicoder, artetxe-etal-2020-cross}, which aims to fine-tune multilingual models with annotated data in high-resource languages and obtain a nice performance in low-resource language tasks. &#10;&#10;Some works aligned word embeddings between high- and low-resource languages through additional parallel sentence pairs \cite{artetxeschwenk2019massively,WeiW0XYL21,chi-etal-2021-infoxlm,pan-etal-2021-multilingual} or bilingual dictionaries \cite{Cao2020Multilingual,ijcai2020-533,Liu_Winata_Lin_Xu_Fung_2020}, so that high-resource fine-tuned models can be transferred to low-resource languages. Although this approach has achieved excellent results in many languages, parallel corpora and bilingual dictionaries are still prohibitively expensive, rendering it impracticable in some minority languages.&#10;&#10;To disengage from the dependence on parallel corpora or bilingual dictionaries \cite{wu-dredze-2019-beto,pmlr-v119-hu20b}, some studies have found that syntactic features in high-resource languages can improve zero-shot cross-lingual transfer learning \cite{meng-etal-2019-target,subburathinam-etal-2019-cross,ahmad-etal-2021-syntax,Ahmad_Peng_Chang_2021}. \cite{libovicky-etal-2020-language} found that the embeddings of different languages are clustered according to their language families, as shown in Figure \ref{fig:intro}a and \ref{fig:intro}b, which demonstrated that different languages are not aligned perfectly in mBERT \cite{Ameet-Mbert}. \cite{huang-etal-2021-improving-zero} tried adversarial training and randomized smoothing with English synonym augmentation to build robust regions for embeddings in the multilingual models, as illustrated in Figure \ref{fig:intro}c. In this way, models can output similar predictions for different language embeddings in the same robust region even they are not well aligned. However, the transferability of English synonym augmentation is limited because its robust region remains close to the English cluster, as shown in Figure \ref{fig:intro}c.&#10;&#10;In this work, we select English as a high-resource language and follow the studies that do not require additional parallel corpora or bilingual dictionaries to improve cross-lingual transfer learning performance with minimal cost. For this purpose, three strategies are proposed to enlarge the robust region of English embeddings. The first strategy is called Embedding-Push, which pushes the embedding of English to other language clusters. The second is Attention-Pull, which constrains the relative position of the word embeddings to prevent the meaning from straying. The last strategy, named Robust target, introduces a Virtual Multilingual Embedding (VME) to help the model build a suitable robust region, as shown in Figure \ref{fig:intro}d.&#10;&#10;Experimental results on mBERT and XLM-R demonstrate that our method effectively improves the zero-shot cross-lingual transfer on classification tasks and outperforms a series of previous works. In addition, case studies show that our method improves the model through multilingual word alignment. Compared with existing works, our method has the following advantages. First, our method only needs English resources, which is suitable for low-resource languages. Second, our method can induce alignments in many languages without specifying the target language. Finally, our method is simple to implement and achieves effective experimental results. Our code&#10;is publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Embeddings, Linguistics, Multilingual Modeling, Cross-Lingual Transfer, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="2007.15960" label="2007.15960">
        <attvalues>
          <attvalue for="0" value="On Learning Universal Representations Across Languages" />
          <attvalue for="1" value="  Recent studies have demonstrated the overwhelming advantage of cross-lingual&#10;pre-trained models (PTMs), such as multilingual BERT and XLM, on cross-lingual&#10;NLP tasks. However, existing approaches essentially capture the co-occurrence&#10;among tokens through involving the masked language model (MLM) objective with&#10;token-level cross entropy. In this work, we extend these approaches to learn&#10;sentence-level representations and show the effectiveness on cross-lingual&#10;understanding and generation. Specifically, we propose a Hierarchical&#10;Contrastive Learning (HiCTL) method to (1) learn universal representations for&#10;parallel sentences distributed in one or multiple languages and (2) distinguish&#10;the semantically-related words from a shared cross-lingual vocabulary for each&#10;sentence. We conduct evaluations on two challenging cross-lingual tasks, XTREME&#10;and machine translation. Experimental results show that the HiCTL outperforms&#10;the state-of-the-art XLM-R by an absolute gain of 4.2% accuracy on the XTREME&#10;benchmark as well as achieves substantial improvements on both of the&#10;high-resource and low-resource English-to-X translation tasks over strong&#10;baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Pre-trained Language Models. Recently, substantial work has shown that pre-trained models (PTMs)~\cite{peters-etal-2018-deep,radford2018improving,devlin-etal-2019-bert} on the large corpus are beneﬁcial for downstream NLP tasks. The application scheme is to ﬁne-tune the pre-trained model using the limited labeled data of specific target tasks. For cross-lingual pre-training, both \cite{devlin-etal-2019-bert} and \cite{NIPS2019_8928} trained a transformer-based model on multilingual Wikipedia which covers &#10;various languages, while XLM-R~\cite{conneau-etal-2020-unsupervised} studied the effects of training unsupervised cross-lingual representations on a very large scale.&#10;&#10;For sequence-to-sequence pre-training, UniLM~\cite{NIPS2019_9464} ﬁne-tuned BERT with an ensemble of masks, which employs a shared Transformer network and utilizing specific self-attention mask to control what context the prediction conditions on. \cite{DBLP:conf/icml/SongTQLL19} extended BERT-style models by jointly training the encoder-decoder framework. XLNet~\cite{NIPS2019_8812} trained by predicting masked tokens auto-regressively in a permuted order, which allows predictions to condition on both left and right context. \cite{raffel2019exploring} &#10;unified every NLP problem as a text-to-text problem and pre-trained a denoising sequence-to-sequence model at scale. Concurrently, BART~\cite{lewis-etal-2020-bart} pre-trained a denoising sequence-to-sequence model, in which spans are masked from the input but the complete output is auto-regressively predicted.&#10;&#10;Previous works have explored using pre-trained models to improve text generation, such as pre-training both the encoder and decoder on several languages~\cite{DBLP:conf/icml/SongTQLL19,NIPS2019_8928,raffel2019exploring} or using pre-trained models to initialize encoders~\cite{edunov-etal-2019-pre,zhang-etal-2019-hibert,guo2020nat}. \cite{DBLP:conf/iclr/ZhuXWHQZLL20} and \cite{weng2020acquiring} proposed a BERT-fused NMT model, in which the representations from BERT are treated as context and fed into all layers of both the encoder and decoder. \cite{zhong-etal-2020-extractive} formulated the extractive summarization task as a semantic text matching problem and proposed a Siamese-BERT architecture to compute the similarity between the source document and the candidate summary, which leverages the pre-trained BERT in a Siamese network structure. Our approach also belongs to the contextual pre-training so it could be applied to various downstream NLU and NLG tasks.&#10;&#10;Contrastive Learning. Contrastive learning (CTL)~\cite{pmlr-v97-saunshi19a} aims at maximizing the similarity between the encoded query $q$ and its matched key $k^{+}$ while keeping randomly sampled keys $\{k_0^{-},k_1^{-},k_2^{-},...\}$ faraway from it. &#10;With similarity measured by a score function $s(q,k)$, a form of a contrastive loss function, called InfoNCE~\cite{Oord2018ctl}, is considered in this paper:&#10;\begin{equation}&#10; \mathcal{L}_{ctl} = -\log \frac{\exp(s(q,k^+))}{\exp(s(q,k^+)) + \sum_{i}\exp(s(q,k_{i}^-))},&#10;\end{equation}&#10;where the score function $s(q,k)$ is essentially implemented as the cosine similarity $\frac{q^T k}{\parallel q \parallel \cdot \parallel k \parallel}$. $q$ and $k$ are often encoded by a learnable neural encoder, such as BERT~\cite{devlin-etal-2019-bert} or ResNet~\cite{DBLP:conf/cvpr/HeZRS16}. $k^{+}$ and $k^{-}$ are typically called positive and negative samples. In addition to the form illustrated in Eq. (\ref{eq:ctl}), contrastive losses can also be based on other forms, such as margin-based loses~\cite{DBLP:conf/cvpr/HadsellCL06} and variants of NCE losses~\cite{DBLP:conf/nips/MnihK13}.&#10;&#10;Contrastive learning is at the core of several recent work on unsupervised or self-supervised learning from computer vision~\cite{DBLP:conf/cvpr/WuXYL18,Oord2018ctl,DBLP:conf/cvpr/YeZYC19,he2019Momentum,icml2020_6165,DBLP:conf/eccv/TianKI20} to natural language processing~\cite{DBLP:conf/nips/MikolovSCCD13,DBLP:conf/nips/MnihK13,devlin-etal-2019-bert,DBLP:conf/iclr/ClarkLLM20,feng2020LanguageAgnostic,chi2020InfoXLM}. \cite{DBLP:conf/iclr/KongdYLDY20} improved language representation learning by maximizing the mutual information between a masked sentence representation and local n-gram spans. \cite{DBLP:conf/iclr/ClarkLLM20} utilized a discriminator to predict whether a token is replaced by a generator given its surrounding context. \cite{iter-etal-2020-pretraining} proposed to pre-train language models with contrastive sentence objectives that predict the surrounding sentences given an anchor sentence. In this paper, we propose Hictl to encourage parallel cross-lingual sentences to have the identical semantic representation and distinguish whether a word is contained in them as well, which can naturally improve the capability of cross-lingual understanding and generation for PTMs.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2006.06402" label="2006.06402">
        <attvalues>
          <attvalue for="0" value="CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot&#10;  Cross-Lingual NLP" />
          <attvalue for="1" value="  Multi-lingual contextualized embeddings, such as multilingual-BERT (mBERT),&#10;have shown success in a variety of zero-shot cross-lingual tasks. However,&#10;these models are limited by having inconsistent contextualized representations&#10;of subwords across different languages. Existing work addresses this issue by&#10;bilingual projection and fine-tuning technique. We propose a data augmentation&#10;framework to generate multi-lingual code-switching data to fine-tune mBERT,&#10;which encourages model to align representations from source and multiple target&#10;languages once by mixing their context information. Compared with the existing&#10;work, our method does not rely on bilingual sentences for training, and&#10;requires only one training process for multiple target languages. Experimental&#10;results on five tasks with 19 languages show that our method leads to&#10;significantly improved performances for all the tasks compared with mBERT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.09273" label="1911.09273">
        <attvalues>
          <attvalue for="0" value="Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual&#10;  Task-oriented Dialogue Systems" />
          <attvalue for="1" value="  Recently, data-driven task-oriented dialogue systems have achieved promising&#10;performance in English. However, developing dialogue systems that support&#10;low-resource languages remains a long-standing challenge due to the absence of&#10;high-quality data. In order to circumvent the expensive and time-consuming data&#10;collection, we introduce Attention-Informed Mixed-Language Training (MLT), a&#10;novel zero-shot adaptation method for cross-lingual task-oriented dialogue&#10;systems. It leverages very few task-related parallel word pairs to generate&#10;code-switching sentences for learning the inter-lingual semantics across&#10;languages. Instead of manually selecting the word pairs, we propose to extract&#10;source words based on the scores computed by the attention layer of a trained&#10;English task-related model and then generate word pairs using existing&#10;bilingual dictionaries. Furthermore, intensive experiments with different&#10;cross-lingual embeddings demonstrate the effectiveness of our approach.&#10;Finally, with very few word pairs, our model achieves significant zero-shot&#10;adaptation performance improvements in both cross-lingual dialogue state&#10;tracking and natural language understanding (i.e., intent detection and slot&#10;filling) tasks compared to the current state-of-the-art approaches, which&#10;utilize a much larger amount of bilingual data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.02134" label="2106.02134">
        <attvalues>
          <attvalue for="0" value="Syntax-augmented Multilingual BERT for Cross-lingual Transfer" />
          <attvalue for="1" value="  In recent years, we have seen a colossal effort in pre-training multilingual&#10;text encoders using large-scale corpora in many languages to facilitate&#10;cross-lingual transfer learning. However, due to typological differences across&#10;languages, the cross-lingual transfer is challenging. Nevertheless, language&#10;syntax, e.g., syntactic dependencies, can bridge the typological gap. Previous&#10;works have shown that pre-trained multilingual encoders, such as mBERT&#10;\cite{devlin-etal-2019-bert}, capture language syntax, helping cross-lingual&#10;transfer. This work shows that explicitly providing language syntax and&#10;training mBERT using an auxiliary objective to encode the universal dependency&#10;tree structure helps cross-lingual transfer. We perform rigorous experiments on&#10;four NLP tasks, including text classification, question answering, named entity&#10;recognition, and task-oriented semantic parsing. The experiment results show&#10;that syntax-augmented mBERT improves cross-lingual transfer on popular&#10;benchmarks, such as PAWS-X and MLQA, by 1.4 and 1.6 points on average across&#10;all languages. In the \emph{generalized} transfer setting, the performance&#10;boosted significantly, with 3.9 and 3.1 points on average in PAWS-X and MLQA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Encoding Syntax for Language Transfer&#10;Universal language syntax, e.g., part-of-speech (POS) tags, dependency parse structure, and relations are shown to be helpful for cross-lingual transfer &#10;\cite{kozhevnikov-titov-2013-cross, prazak-konopik-2017-cross, wu2017improved, subburathinam-etal-2019-cross, liu-etal-2019-neural-cross, zhang-etal-2019-syntax-enhanced-self, xie-etal-2020-contextual, ahmad2020gate}.&#10;Many of these prior works utilized graph neural networks (GNN) to encode the dependency graph structure of the input sequences.&#10;In this work, we utilize graph attention networks (GAT) \cite{velickovic2018graph}, a variant of GNN that employs the multi-head attention mechanism.&#10;&#10;Syntax-aware Multi-head Attention&#10;A large body of prior works investigated the advantages of incorporating language syntax to enhance the self-attention mechanism \cite{vaswani2017attention}.&#10;Existing techniques can be broadly divided into two types. &#10;The first type of approach relies on an external parser (or human annotation) to get a sentence's dependency structure during inference.&#10;This type of approaches embed the dependency structure into contextual representations \cite{wu2017improved, chen-etal-2017-improved, wang2019source, wang-etal-2019-self-attention, zhang-etal-2019-syntax-enhanced-self, zhang2019sgnet, bugliarello-okazaki-2020-enhancing, sachan2020syntax, ahmad2020gate}.&#10;Our proposed method falls under this category; however, unlike prior works, our study investigates if fusing the universal dependency structure into the self-attention of existing multilingual encoders help cross-lingual transfer.&#10;Graph attention networks (GATs) that use multi-head attention has also been adopted for NLP tasks&#10;\cite{huang-carley-2019-syntax} also fall into this category.&#10;The second category of approaches does not require the syntax structure of the input text during inference.&#10;These approaches are trained to predict the dependency parse via supervised learning \cite{strubell-etal-2018-linguistically, deguchi-etal-2019-dependency}.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2010.03009" label="2010.03009">
        <attvalues>
          <attvalue for="0" value="GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and&#10;  Event Extraction" />
          <attvalue for="1" value="  Recent progress in cross-lingual relation and event extraction use graph&#10;convolutional networks (GCNs) with universal dependency parses to learn&#10;language-agnostic sentence representations such that models trained on one&#10;language can be applied to other languages. However, GCNs struggle to model&#10;words with long-range dependencies or are not directly connected in the&#10;dependency tree. To address these challenges, we propose to utilize the&#10;self-attention mechanism where we explicitly fuse structural information to&#10;learn the dependencies between words with different syntactic distances. We&#10;introduce GATE, a {\bf G}raph {\bf A}ttention {\bf T}ransformer {\bf E}ncoder,&#10;and test its cross-lingual transferability on relation and event extraction&#10;tasks. We perform experiments on the ACE05 dataset that includes three&#10;typologically different languages: English, Chinese, and Arabic. The evaluation&#10;results show that GATE outperforms three recently proposed methods by a large&#10;margin. Our detailed analysis reveals that due to the reliance on syntactic&#10;dependencies, GATE produces robust representations that facilitate transfer&#10;across languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.11032" label="2009.11032">
        <attvalues>
          <attvalue for="0" value="Streamlining Cross-Document Coreference Resolution: Evaluation and&#10;  Modeling" />
          <attvalue for="1" value="  Recent evaluation protocols for Cross-document (CD) coreference resolution&#10;have often been inconsistent or lenient, leading to incomparable results across&#10;works and overestimation of performance. To facilitate proper future research&#10;on this task, our primary contribution is proposing a pragmatic evaluation&#10;methodology which assumes access to only raw text -- rather than assuming gold&#10;mentions, disregards singleton prediction, and addresses typical targeted&#10;settings in CD coreference resolution. Aiming to set baseline results for&#10;future research that would follow our evaluation methodology, we build the&#10;first end-to-end model for this task. Our model adapts and extends recent&#10;neural models for within-document coreference resolution to address the CD&#10;coreference setting, which outperforms state-of-the-art results by a&#10;significant margin.&#10;" />
          <attvalue for="2" value="&#10;&#10;The literature on coreference resolution has traditionally divided the task into two different settings, addressing the task at either the Within-document (WD) or Cross-document (CD) level. Each setting has presented different challenges, model design choices, and historically different evaluation practices.&#10;&#10;In CD coreference resolution, the instances consist of multiple documents, each authored independently, without any inherent linear ordering between them.&#10;As a result, coreferring expressions across documents are often lexically-divergent, while lexically-similar expressions may refer to different concepts. Table~\ref{tab:subtopic} shows example documents discussing similar, yet distinct, events (two different nominations of a US Surgeon General) with overlapping participants (``President Barack Obama'') and event triggers (``name''). &#10;Leveraging accurate CD coreference models seems particularly appealing for applications that merge information across texts, which have been gaining growing attention recently, such as multi-document summarization~\cite{falke-etal-2017-concept} and multi-hop question answering \cite{dhingra-etal-2018-neural, wang-etal-2019-multi-hop}. &#10;&#10;In this paper, we observe that research on CD coreference has been lagging behind the impressive strides made in WD coreference~\cite{lee-etal-2017-end, lee2018higher, joshi-etal-2019-bert, doi:10.1162/tacl_a_00300, wu-etal-2020-corefqa}. &#10;As the time seems ripe to promote advances in CD coreference modeling as well, we present two steps to facilitate and trigger such systematic research, with respect to proper evaluation methodologies and current modeling approaches.&#10;&#10;With respect to evaluation, we find that previous works have often used incomparable or lenient evaluation protocols, such as assuming event and entity mentions are given as part of the input, peeking into the fine-grained subtopic annotations, or rewarding coreference models for just identifying singleton clusters~(Section~\ref{sec:background}). As we will show in Section~\ref{sec:results}, these evaluation protocols have resulted in artificially inflated performance measures.&#10;&#10;To address these shortcomings, our primary contribution consists of formalizing a realistic evaluation methodology for CD coreference. Namely, we use only raw input texts without assuming access to human-labeled annotations such as entity and event mentions, and also disregard singletons during evaluation. In addition, we examine model performance in both focused topic clusters, known a-priory to discuss overlapping information, as well as on larger sets of documents which contain both related and unrelated documents~(Section~\ref{sec:methodology}).&#10;&#10;With respect to modeling, in Section~\ref{sec:model}, we describe a first end-to-end CD coreference model which builds upon the state-of-the-art in WD coreference and recent advances in transformer-based encoders. To achieve this, we address the inherently non-linear nature of the CD setting by combining this model with an agglomerative clustering approach, which was shown useful in other CD models. We first show that this combination sets a new state of the art for the task of CD coreference, in comparison to prior evaluations~(Section~\ref{sec:results}). &#10;We then evaluate this model following our realistic and more challenging evaluation methodology, setting a proper baseline for future research.&#10;&#10;Taken together, our work brings the task of cross-document coreference resolution up to modern NLP standards, providing standardized evaluation benchmarks and a modern model which sets a new state-of-the-art result for the task. &#10;We hope that future work will use our framework to develop, and particularly to evaluate, models which make further advances on this challenging and important task.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Coreference Resolution, Artificial Intelligence, Evaluation Methodology, Information Science, Neural Modeling" />
        </attvalues>
      </node>
      <node id="1804.05922" label="1804.05922">
        <attvalues>
          <attvalue for="0" value="Neural Models for Reasoning over Multiple Mentions using Coreference" />
          <attvalue for="1" value="  Many problems in NLP require aggregating information from multiple mentions&#10;of the same entity which may be far apart in the text. Existing Recurrent&#10;Neural Network (RNN) layers are biased towards short-term dependencies and&#10;hence not suited to such tasks. We present a recurrent layer which is instead&#10;biased towards coreferent dependencies. The layer uses coreference annotations&#10;extracted from an external system to connect entity mentions belonging to the&#10;same cluster. Incorporating this layer into a state-of-the-art reading&#10;comprehension model improves performance on three datasets -- Wikihop, LAMBADA&#10;and the bAbi AI tasks -- with large gains when training data is scarce.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.08703" label="2212.08703">
        <attvalues>
          <attvalue for="0" value="Fast Entropy-Based Methods of Word-Level Confidence Estimation for&#10;  End-To-End Automatic Speech Recognition" />
          <attvalue for="1" value="  This paper presents a class of new fast non-trainable entropy-based&#10;confidence estimation methods for automatic speech recognition. We show how&#10;per-frame entropy values can be normalized and aggregated to obtain a&#10;confidence measure per unit and per word for Connectionist Temporal&#10;Classification (CTC) and Recurrent Neural Network Transducer (RNN-T) models.&#10;Proposed methods have similar computational complexity to the traditional&#10;method based on the maximum per-frame probability, but they are more&#10;adjustable, have a wider effective threshold range, and better push apart the&#10;confidence distributions of correct and incorrect words. We evaluate the&#10;proposed confidence measures on LibriSpeech test sets, and show that they are&#10;up to 2 and 4 times better than confidence estimation based on the maximum&#10;per-frame probability at detecting incorrect words for Conformer-CTC and&#10;Conformer-RNN-T models, respectively.&#10;" />
          <attvalue for="2" value="&#10;&#10;Many real-world automatic speech recognition (ASR) systems require to have not only the best possible transcript, but also confidence scores for each recognized unit, which provide an estimate of how likely the prediction is to be correct. ASR confidence scores are used in semi-supervised and active learning \cite{vesely2013asru,riccardi2005,yu2010csl,drugman2016}, &#10;speaker adaptation \cite{uebel2001speaker}, information retrieval \cite{zbib2019aS}, speech translation \cite{saleem2004using,besacier-etal-2014-word} etc.&#10;The confidence measures for hybrid WFST-based ASR uses word lattices, $N$-best lists, and external trainable classifiers \cite{wessel2001, Jiang2005ConfidenceMF, Yu2011CalibrationOC}.&#10;&#10;This work is motivated by the need for fast, simple, robust, and adjustable confidence estimation for the end-to-end Connectionist Temporal Classification (CTC) \cite{graves_connectionist_2006} and Recurrent Neural Network Transducer (RNN-T) \cite{graves2012transducer} ASR models. &#10;End-to-end neural ASR systems predict the softmax probability for each unit of the output vocabulary. The probability of the most-likely unit (the maximum probability) is a natural way of estimating confidence \cite{hendrycks2016iclr,park20d_interspeech}. &#10;The effectiveness of this approach is limited by the so-called prediction overconfidence, when the probability distribution is skewed towards the best hypothesis \cite{nguyen2015}. We consider a model to be overconfident when its median probability of incorrect predictions is above 0.9. This situation is typical for overtrained CTC and RNN-T models. To mitigate the overconfidence issue, one can use temperature scaling, dropout, ensemble of ASR models etc \cite{vyas2019icassp,malinin2021uncertainty, Oneata2021}. An alternative approach is based on dedicated neural confidence models ~\cite{li2020confidence,jeon2020,woodward2020confidence,qiu2021learning,li2021residual,qui2021multi,wang2021word}.&#10;&#10;Another challenge in developing confidence estimation methods for end-to-end ASR systems is the granularity of predictions. Speech applications generally require word-level confidence, while end-to-end ASR provides unit-level output. Word-level confidence can either be computed directly as posteriors of a decoding lattice \cite{evermann2000posterior} or they can be aggregated from unit-level scores. The former requires significant computational resources \cite{zapotoczny2019}, while the latter requires careful selection of aggregation methods \cite{Oneata2021}. &#10;&#10;The evaluation of confidence estimation methods for end-to-end ASR systems is also not trivial. Many widely used in the past confidence estimation metrics, like Area Under the Curve of the Receiver Operating Characteristic ($\mathrm{AUC}_\mathrm{ROC}$) or Area Under the Precision-Recall Curve &#10;($\mathrm{AUC}_\mathrm{PR}$), produce unreasonably high scores induced by high accuracy of modern ASR with heavy imbalance between correctly and incorrectly recognized words. Also, most of the metrics do not express the possibility of obtaining different results for different values of the confidence threshold of the confidence estimator (the adjustability).&#10;&#10;Our main contributions are the following:&#10;&#10;\begin{enumerate}&#10; \item We propose a fast non-trainable confidence measure based on an exponentially normalized entropy over per-frame probabilities for output tokens.&#10; \item We show how per-frame confidence measures can be aggregated to obtain confidence measure per unit and per word for greedy decoding for CTC and RNN-T-based models.&#10; \item We evaluate the proposed methods against the traditional method based on probability of the most likely prediction using state-of-the-art Conformer model and demonstrate that proposed measures are more accurate both for clean and noisy acoustic conditions.&#10; \end{enumerate}&#10;The implementation of the proposed confidence estimation methods is available in the NeMo toolkit~\cite{kuchaiev2019nemo}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Confidence Estimation, Computer Science, Linguistics, Neural Network Models, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1412.1897" label="1412.1897">
        <attvalues>
          <attvalue for="0" value="Deep Neural Networks are Easily Fooled: High Confidence Predictions for&#10;  Unrecognizable Images" />
          <attvalue for="1" value="  Deep neural networks (DNNs) have recently been achieving state-of-the-art&#10;performance on a variety of pattern-recognition tasks, most notably visual&#10;classification problems. Given that DNNs are now able to classify objects in&#10;images with near-human-level performance, questions naturally arise as to what&#10;differences remain between computer and human vision. A recent study revealed&#10;that changing an image (e.g. of a lion) in a way imperceptible to humans can&#10;cause a DNN to label the image as something else entirely (e.g. mislabeling a&#10;lion a library). Here we show a related result: it is easy to produce images&#10;that are completely unrecognizable to humans, but that state-of-the-art DNNs&#10;believe to be recognizable objects with 99.99% confidence (e.g. labeling with&#10;certainty that white noise static is a lion). Specifically, we take&#10;convolutional neural networks trained to perform well on either the ImageNet or&#10;MNIST datasets and then find images with evolutionary algorithms or gradient&#10;ascent that DNNs label with high confidence as belonging to each dataset class.&#10;It is possible to produce images totally unrecognizable to human eyes that DNNs&#10;believe with near certainty are familiar objects, which we call &quot;fooling&#10;images&quot; (more generally, fooling examples). Our results shed light on&#10;interesting differences between human vision and current DNNs, and raise&#10;questions about the generality of DNN computer vision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.09636" label="1907.09636">
        <attvalues>
          <attvalue for="0" value="On Modeling ASR Word Confidence" />
          <attvalue for="1" value="  We present a new method for computing ASR word confidences that effectively&#10;mitigates the effect of ASR errors for diverse downstream applications,&#10;improves the word error rate of the 1-best result, and allows better comparison&#10;of scores across different models. We propose 1) a new method for modeling word&#10;confidence using a Heterogeneous Word Confusion Network (HWCN) that addresses&#10;some key flaws in conventional Word Confusion Networks, and 2) a new score&#10;calibration method for facilitating direct comparison of scores from different&#10;models. Using a bidirectional lattice recurrent neural network to compute the&#10;confidence scores of each word in the HWCN, we show that the word sequence with&#10;the best overall confidence is more accurate than the default 1-best result of&#10;the recognizer, and that the calibration method can substantially improve the&#10;reliability of recognizer combination.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.15222" label="2110.15222">
        <attvalues>
          <attvalue for="0" value="Word-level confidence estimation for RNN transducers" />
          <attvalue for="1" value="  Confidence estimate is an often requested feature in applications such as&#10;medical transcription where errors can impact patient care and the confidence&#10;estimate could be used to alert medical professionals to verify potential&#10;errors in recognition.&#10;  In this paper, we present a lightweight neural confidence model tailored for&#10;Automatic Speech Recognition (ASR) system with Recurrent Neural Network&#10;Transducers (RNN-T). Compared to other existing approaches, our model utilizes:&#10;(a) the time information associated with recognized words, which reduces the&#10;computational complexity, and (b) a simple and elegant trick for mapping&#10;between sub-word and word sequences. The mapping addresses the non-unique&#10;tokenization and token deletion problems while amplifying differences between&#10;confusable words. Through extensive empirical evaluations on two different&#10;long-form test sets, we demonstrate that the model achieves a performance of&#10;0.4 Normalized Cross Entropy (NCE) and 0.05 Expected Calibration Error (ECE).&#10;It is robust across different ASR configurations, including target types&#10;(graphemes vs. morphemes), traffic conditions (streaming vs. non-streaming),&#10;and encoder types. We further discuss the importance of evaluation metrics to&#10;reflect practical applications and highlight the need for further work in&#10;improving Area Under the Curve (AUC) for Negative Precision Rate (NPV) and True&#10;Negative Rate (TNR).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.14003" label="2401.14003">
        <attvalues>
          <attvalue for="0" value="ConstraintChecker: A Plugin for Large Language Models to Reason on&#10;  Commonsense Knowledge Bases" />
          <attvalue for="1" value="  Reasoning over Commonsense Knowledge Bases (CSKB), i.e. CSKB reasoning, has&#10;been explored as a way to acquire new commonsense knowledge based on reference&#10;knowledge in the original CSKBs and external prior knowledge. Despite the&#10;advancement of Large Language Models (LLM) and prompt engineering techniques in&#10;various reasoning tasks, they still struggle to deal with CSKB reasoning. One&#10;of the problems is that it is hard for them to acquire explicit relational&#10;constraints in CSKBs from only in-context exemplars, due to a lack of symbolic&#10;reasoning capabilities (Bengio et al., 2021). To this end, we proposed&#10;**ConstraintChecker**, a plugin over prompting techniques to provide and check&#10;explicit constraints. When considering a new knowledge instance,&#10;ConstraintChecker employs a rule-based module to produce a list of constraints,&#10;then it uses a zero-shot learning module to check whether this knowledge&#10;instance satisfies all constraints. The acquired constraint-checking result is&#10;then aggregated with the output of the main prompting technique to produce the&#10;final output. Experimental results on CSKB Reasoning benchmarks demonstrate the&#10;effectiveness of our method by bringing consistent improvements over all&#10;prompting methods. Codes and data are available at&#10;\url{https://github.com/HKUST-KnowComp/ConstraintChecker}.&#10;" />
          <attvalue for="2" value="&#10;&#10;Commonsense Knowledge Bases (CSKB) Reasoning, as one of many commonsense reasoning tasks, has been well explored in Natural Language Processing for the past few years.&#10;As human-annotated CSKBs~\cite{singh2017conceptnet, sap2019atomic, mostafazadeh2020glucose} are usually incomplete and of a small coverage, reasoning over CSKBs, i.e., CSKB reasoning, is a way for expansion. CSKB reasoning is defined as determining whether a new knowledge triple (head event, relation, tail event) is commonsense (in other expressions, being plausible or having label 1) based on the reference knowledge in original CSKBs as well as external prior knowledge~\cite{fang2023ckbpv2, davison2019commonsense}.&#10;Expanding CSKBs via such a reasoning process can lead to better and broader commonsense knowledge as valuable resources to augment AI models in various aspects, such as visual reasoning ~\cite{zellers2019vcr}, text generation~\cite{zhou2021commonsense,ilievski2021story}, or building more capable knowledge models for further downstream applications ~\cite{yu2022cocolm,hwang2021comet,wang-etal-2023-cat}. &#10;&#10;Recently, inspired by the emergence of Large Language Models (LLMs) that can perform well in many commonsense reasoning tasks~\cite{qin2023chatgpt, bian2023chatgpt}, ~\cite{chan2023chatgpt} attempted to use LLMs for a CSKB Reasoning benchmark named CSKB Population (CKBP)~\cite{fang2023ckbpv2}. However, the result shows that LLMs still fall short in the benchmark, even with a large number of in-context examples.&#10;One of the problems is that LLMs find it hard to acquire the explicit relational constraints in CSKBs, hence making wrong predictions.&#10;In the example in Figure ~\ref{fig:problem}, the xReact relation in CSKBs requires the tail event of the knowledge triple to express a mental state, such as ``PersonX is confident'', instead of an action, such as ``PersonX win''.&#10;Meanwhile, LLMs fail to recognize the constraint from in-context exemplars, thus making the judgment mainly based on the semantics of the head and tail events.&#10;It leads to an incorrect prediction that the triple (PersonX prepare for the competition, xReact, PersonX win) is plausible. &#10;In light of this, many advanced prompting techniques, such as Chain-of-Thouht (CoT) \cite{wei2022cot}, Least-to-Most \cite{zhou2023leasttomost}, Active-CoT \cite{diao2023active}, etc., can be possible alternatives for improvements. Nonetheless, they are task-agnostic and suffer from the inherent shortcoming of LLMs in inducing the rules in CSKBs (which we refer as symbolic reasoning ability),&#10;as current deep learning still struggles to deal with symbolic and high-level concepts reasoning tasks ~\cite{bengio2021deeplearning,huang-chang-2023-towards,&#10;pan2023logiclm}.&#10;&#10;To this end, we propose ConstraintChecker, a plugin component for LLMs to handle the problem of explicit constraints in CSKB reasoning.&#10;ConstraintChecker&#10;supports LLMs' reasoning as an independent component in addition to the main-task component that determines whether a knowledge triple is commonsense or not.&#10;There are two modules in this plugin.&#10;Given a knowledge triple (head event, relation, tail event), we first employ a rule-based/symbolic module to produce a list of constraints based on the relation.&#10;The list is then passed to a zero-shot learning module, where we construct constraint-checking questions and use the same LLM as in the main-task component in a zero-shot manner to check whether the instance satisfies all constraints.&#10;The acquired constraint-checking result is then aggregated with the prediction from the main-task component by logical conjunction to produce the final prediction.&#10;&#10;We implement ConstraintChecker and conduct extensive experiments on a CSKB Reasoning benchmark CKBPv2 ~\cite{fang2023ckbpv2} as well as a synthetic discriminative version of ATOMIC$_{20}^{20}$ (in short, SD-ATOMIC$_{20}^{20}$), over two large language models: ChatGPT (gpt-3.5-turbo-0301) and GPT3.5 (text-davinci-003).&#10;On both language models, ConstraintChecker improves over prompting techniques (as the main-task component) by a significant margin in different metrics, achieving the best result on both the benchmarks CKBPv2 and SD-ATOMIC$_{20}^{20}$.&#10;Further analyses and ablation studies show the effect of each of the considered constraints as well as different choices of prompt design in ConstraintChecker, and the superiority of its plug-and-play design over the single-prompt counterpart.&#10;&#10;To summarize, our contribution is two-fold: &#10;(1) We propose ConstraintChecker, an independent plugin that handles the problem of explicit constraints in CSKB reasoning to improve over main-task prompt methods, and&#10;(2) We conduct extensive experiments on two CSKB Reasoning benchmarks CKBPv2 and SD-ATOMIC$_{20}^{20}$, to demonstrate our method's effectiveness and advantages over other advanced prompting techniques.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Cognitive Science, Artificial Intelligence, Mathematics, Commonsense Knowledge Bases, Natural Language Processing, Reasoning Techniques" />
        </attvalues>
      </node>
      <node id="1811.10830" label="1811.10830">
        <attvalues>
          <attvalue for="0" value="From Recognition to Cognition: Visual Commonsense Reasoning" />
          <attvalue for="1" value="  Visual understanding goes well beyond object recognition. With one glance at&#10;an image, we can effortlessly imagine the world beyond the pixels: for&#10;instance, we can infer people's actions, goals, and mental states. While this&#10;task is easy for humans, it is tremendously difficult for today's vision&#10;systems, requiring higher-order cognition and commonsense reasoning about the&#10;world. We formalize this task as Visual Commonsense Reasoning. Given a&#10;challenging question about an image, a machine must answer correctly and then&#10;provide a rationale justifying its answer.&#10;  Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA&#10;problems derived from 110k movie scenes. The key recipe for generating&#10;non-trivial and high-quality problems at scale is Adversarial Matching, a new&#10;approach to transform rich annotations into multiple choice questions with&#10;minimal bias. Experimental results show that while humans find VCR easy (over&#10;90% accuracy), state-of-the-art vision models struggle (~45%).&#10;  To move towards cognition-level understanding, we present a new reasoning&#10;engine, Recognition to Cognition Networks (R2C), that models the necessary&#10;layered inferences for grounding, contextualization, and reasoning. R2C helps&#10;narrow the gap between humans and machines (~65%); still, the challenge is far&#10;from solved, and we provide analysis that suggests avenues for future work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.04808" label="2305.04808">
        <attvalues>
          <attvalue for="0" value="CAT: A Contextualized Conceptualization and Instantiation Framework for&#10;  Commonsense Reasoning" />
          <attvalue for="1" value="  Commonsense reasoning, aiming at endowing machines with a human-like ability&#10;to make situational presumptions, is extremely challenging to generalize. For&#10;someone who barely knows about &quot;meditation,&quot; while is knowledgeable about&#10;&quot;singing,&quot; he can still infer that &quot;meditation makes people relaxed&quot; from the&#10;existing knowledge that &quot;singing makes people relaxed&quot; by first conceptualizing&#10;&quot;singing&quot; as a &quot;relaxing event&quot; and then instantiating that event to&#10;&quot;meditation.&quot; This process, known as conceptual induction and deduction, is&#10;fundamental to commonsense reasoning while lacking both labeled data and&#10;methodologies to enhance commonsense modeling. To fill such a research gap, we&#10;propose CAT (Contextualized ConceptuAlization and InsTantiation), a&#10;semi-supervised learning framework that integrates event conceptualization and&#10;instantiation to conceptualize commonsense knowledge bases at scale. Extensive&#10;experiments show that our framework achieves state-of-the-art performances on&#10;two conceptualization tasks, and the acquired abstract commonsense knowledge&#10;can significantly improve commonsense inference modeling. Our code, data, and&#10;fine-tuned models are publicly available at&#10;https://github.com/HKUST-KnowComp/CAT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.14827" label="2304.14827">
        <attvalues>
          <attvalue for="0" value="ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal,&#10;  Causal, and Discourse Relations" />
          <attvalue for="1" value="  This paper aims to quantitatively evaluate the performance of ChatGPT, an&#10;interactive large language model, on inter-sentential relations such as&#10;temporal relations, causal relations, and discourse relations. Given ChatGPT's&#10;promising performance across various tasks, we proceed to carry out thorough&#10;evaluations on the whole test sets of 11 datasets, including temporal and&#10;causal relations, PDTB2.0-based, and dialogue-based discourse relations. To&#10;ensure the reliability of our findings, we employ three tailored prompt&#10;templates for each task, including the zero-shot prompt template, zero-shot&#10;prompt engineering (PE) template, and in-context learning (ICL) prompt&#10;template, to establish the initial baseline scores for all popular&#10;sentence-pair relation classification tasks for the first time. Through our&#10;study, we discover that ChatGPT exhibits exceptional proficiency in detecting&#10;and reasoning about causal relations, albeit it may not possess the same level&#10;of expertise in identifying the temporal order between two events. While it is&#10;capable of identifying the majority of discourse relations with existing&#10;explicit discourse connectives, the implicit discourse relation remains a&#10;formidable challenge. Concurrently, ChatGPT demonstrates subpar performance in&#10;the dialogue discourse parsing task that requires structural understanding in a&#10;dialogue before being aware of the discourse relation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.09146" label="2201.09146">
        <attvalues>
          <attvalue for="0" value="Question rewriting? Assessing its importance for conversational question&#10;  answering" />
          <attvalue for="1" value="  In conversational question answering, systems must correctly interpret the&#10;interconnected interactions and generate knowledgeable answers, which may&#10;require the retrieval of relevant information from a background repository.&#10;Recent approaches to this problem leverage neural language models, although&#10;different alternatives can be considered in terms of modules for (a)&#10;representing user questions in context, (b) retrieving the relevant background&#10;information, and (c) generating the answer. This work presents a conversational&#10;question answering system designed specifically for the Search-Oriented&#10;Conversational AI (SCAI) shared task, and reports on a detailed analysis of its&#10;question rewriting module. In particular, we considered different variations of&#10;the question rewriting module to evaluate the influence on the subsequent&#10;components, and performed a careful analysis of the results obtained with the&#10;best system configuration. Our system achieved the best performance in the&#10;shared task and our analysis emphasizes the importance of the conversation&#10;context representation for the overall system performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Conversational question answering extends traditional Question Answering (QA) by involving a sequence of interconnected questions and answers \cite{Choi2018}. Systems addressing this problem need to understand an entire conversation flow, often using explicit knowledge from an external datastore to generate a natural and correct answer for the given question. One way of approaching this problem is to divide it into 3 steps (see Fig. \ref{fig:system}): initial question rewriting, retrieval of relevant information regarding the question, and final answer generation.&#10;&#10;In a conversational scenario, questions may contain acronyms, coreferences, ellipses, and other natural language elements that make it difficult for a system to understand the question. Question rewriting aims to solve this problem by reformulating the question and making it independent of the conversation context \cite{Elgohary2019}, which has been shown to improve systems performance \cite{Vakulenko2021}.&#10;&#10;After an initial understanding of the question and its conversational context, the next challenge is the retrieval of relevant information to use explicitly in the answer generation \cite{Dalton2020}. For this step, the rewritten question is used as a query to an external datastore, and thus the performance of the initial rewriting module can affect the conversational passage retrieval \cite{Vakulenko2021a}.&#10;&#10;The last module has the task of generating an answer that incorporates the retrieved information conditioned on the rewritten question. The Question Rewriting in Conversational Context (QReCC) dataset \cite{Anantha2021} brings these tasks together, supporting the training and evaluation of neural models for conversational QA.&#10;&#10;This work presents a conversational QA system implemented according to the dataset and task definition of the Search-Oriented Conversational AI (SCAI) QReCC 2021 shared task, specifically focusing on the question rewriting module. Besides evaluating the system performance as a whole, using many variations of the question rewriting module, our work highlights the importance of this module and how much it impacts the performance of subsequent ones.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Conversational AI, Question Answering Systems, Neural Language Models, Information Science" />
        </attvalues>
      </node>
      <node id="1808.07036" label="1808.07036">
        <attvalues>
          <attvalue for="0" value="QuAC : Question Answering in Context" />
          <attvalue for="1" value="  We present QuAC, a dataset for Question Answering in Context that contains&#10;14K information-seeking QA dialogs (100K questions in total). The dialogs&#10;involve two crowd workers: (1) a student who poses a sequence of freeform&#10;questions to learn as much as possible about a hidden Wikipedia text, and (2) a&#10;teacher who answers the questions by providing short excerpts from the text.&#10;QuAC introduces challenges not found in existing machine comprehension&#10;datasets: its questions are often more open-ended, unanswerable, or only&#10;meaningful within the dialog context, as we show in a detailed qualitative&#10;evaluation. We also report results for a number of reference models, including&#10;a recently state-of-the-art reading comprehension architecture extended to&#10;model dialog context. Our best model underperforms humans by 20 F1, suggesting&#10;that there is significant room for future work on this data. Dataset, baseline,&#10;and leaderboard available at http://quac.ai.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.07382" label="2101.07382">
        <attvalues>
          <attvalue for="0" value="A Comparison of Question Rewriting Methods for Conversational Passage&#10;  Retrieval" />
          <attvalue for="1" value="  Conversational passage retrieval relies on question rewriting to modify the&#10;original question so that it no longer depends on the conversation history.&#10;Several methods for question rewriting have recently been proposed, but they&#10;were compared under different retrieval pipelines. We bridge this gap by&#10;thoroughly evaluating those question rewriting methods on the TREC CAsT 2019&#10;and 2020 datasets under the same retrieval pipeline. We analyze the effect of&#10;different types of question rewriting methods on retrieval performance and show&#10;that by combining question rewriting methods of different types we can achieve&#10;state-of-the-art performance on both datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.04898" label="2010.04898">
        <attvalues>
          <attvalue for="0" value="Open-Domain Question Answering Goes Conversational via Question&#10;  Rewriting" />
          <attvalue for="1" value="  We introduce a new dataset for Question Rewriting in Conversational Context&#10;(QReCC), which contains 14K conversations with 80K question-answer pairs. The&#10;task in QReCC is to find answers to conversational questions within a&#10;collection of 10M web pages (split into 54M passages). Answers to questions in&#10;the same conversation may be distributed across several web pages. QReCC&#10;provides annotations that allow us to train and evaluate individual subtasks of&#10;question rewriting, passage retrieval and reading comprehension required for&#10;the end-to-end conversational question answering (QA) task. We report the&#10;effectiveness of a strong baseline approach that combines the state-of-the-art&#10;model for question rewriting, and competitive models for open-domain QA. Our&#10;results set the first baseline for the QReCC dataset with F1 of 19.10, compared&#10;to the human upper bound of 75.45, indicating the difficulty of the setup and a&#10;large room for improvement.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.00048" label="2406.00048">
        <attvalues>
          <attvalue for="0" value="Towards a theory of how the structure of language is acquired by deep&#10;  neural networks" />
          <attvalue for="1" value="  How much data is required to learn the structure of a language via next-token&#10;prediction? We study this question for synthetic datasets generated via a&#10;Probabilistic Context-Free Grammar (PCFG) -- a tree-like generative model that&#10;captures many of the hierarchical structures found in natural languages. We&#10;determine token-token correlations analytically in our model and show that they&#10;can be used to build a representation of the grammar's hidden variables, the&#10;longer the range the deeper the variable. In addition, a finite training set&#10;limits the resolution of correlations to an effective range, whose size grows&#10;with that of the training set. As a result, a Language Model trained with&#10;increasingly many examples can build a deeper representation of the grammar's&#10;structure, thus reaching good performance despite the high dimensionality of&#10;the problem. We conjecture that the relationship between training set size and&#10;effective range of correlations holds beyond our synthetic datasets. In&#10;particular, our conjecture predicts how the scaling law for the test loss&#10;behaviour with training set size depends on the length of the context window,&#10;which we confirm empirically in Shakespeare's plays and Wikipedia articles.&#10;" />
          <attvalue for="2" value="&#10;&#10;Two central foci of linguistics are the language structure and how humans acquire it. Formal language theory, for instance, describes languages with hierarchical generative models of grammar, classified in different levels of complexity~\cite{chomsky2014aspects,jager2012formal}. In this context, the `poverty of the stimulus' argument~\cite{berwick2011poverty}---stating that the data children receive is insufficient to uniquely determine the grammatical structure of their language---led to the hypothesis that linguistic faculties are largely innate. By contrast, statistical learning theory~\cite{ellis2002frequency, saffran2018infant} posits that the statistics of the input data can be used to deduce the language structure. This assumption is supported by empirical evidence concerning a broad range of tasks, including word segmentation~\cite{saffran1996statistical} and reconstruction of the hierarchical phrase structure~\cite{saffran2001use}.&#10;&#10;Large Language Models (LLMs) offer an interesting perspective on the subject. For instance, the success of LLMs trained for next-token prediction~\cite{devlin2019bert, radford2018improving} establishes that a language can be acquired from examples alone---albeit with a training set much larger than what humans are exposed to. Furthermore, empirical studies of LLMs' representations showed that they learn a hierarchy of contextual information, including notions of linguistics such as word classes and syntactic structure~\cite{peters2018dissecting, tenney2019bert, manning2020emergent}. Recent studies have begun revealing the inner workings of LLMs by using synthetic data generated via context-free grammars~\cite{allen2023physics, zhao2023transformers}, determining, in particular, the algorithm that these models follow when predicting the next token. However, there is no consensus on the mechanisms behind language {acquisition} by LLMs~\cite{arora2023theory, douglas2023large}. As a result, empirical phenomena such as the scaling of the test loss with dataset size and number of parameters~\cite{kaplan2020scaling} and the emergence of specific skills at certain scales~\cite{ganguli2022predictability,schaeffer2024emergent} remain unexplained. In this work, we use hierarchical generative models of data to describe how the structure of a language is learnt as the training set grows.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Data Requirements, Probabilistic Grammar, Mathematics, Language Modeling, Statistics" />
        </attvalues>
      </node>
      <node id="2404.01331" label="2404.01331">
        <attvalues>
          <attvalue for="0" value="LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact&#10;  Language Model" />
          <attvalue for="1" value="  We train a suite of multimodal foundation models (MMFM) using the popular&#10;LLaVA framework with the recently released Gemma family of large language&#10;models (LLMs). Of particular interest is the 2B parameter Gemma model, which&#10;provides opportunities to construct capable small-scale MMFMs. In line with&#10;findings from other papers in this space, we test the effect of ablating three&#10;design features: pretraining the connector, utilizing a more powerful image&#10;backbone, and increasing the size of the language backbone. The resulting&#10;models, which we call LLaVA-Gemma, exhibit moderate performance on an array of&#10;evaluations, but fail to improve past the current comparably sized SOTA models.&#10;Closer analysis of performance shows mixed effects; skipping pretraining tends&#10;to reduce performance, larger vision models sometimes improve performance, and&#10;increasing language model size has inconsistent effects. We publicly release&#10;training recipes, code and weights for our models for the LLaVA-Gemma models.&#10;" />
          <attvalue for="2" value="&#10;In this paper, we introduce LLaVA-Gemma, a suite of vision-language assistants trained from the Gemma Large Language Model (LLM) variants, Gemma-2B and Gemma-7B \cite{team2024gemma}. Our work is inspired by the rapid progress in small but capable visual language models (VLMs), such as LLaVA-Phi \cite{zhu2024llava}, which have demonstrated remarkable efficiency and effectiveness in various language understanding tasks. LLaVA-Gemma distinguishes itself among small VLMs due to the public release of similarly trained, different-sized LLMs Gemma-2B and Gemma-7B. &#10;&#10;The unique release of the Gemma models offers an opportunity to contrast model performance in relation to parameter size and visual encoding capabilities. By possessing two variants with different parameter sizes, LLaVA-Gemma allows researchers to investigate the trade-offs between computational efficiency and the richness of visual and linguistic understanding. With these two variants, we perform a deeper exploration of how varying levels of model complexity influence the effectiveness of visual encoding, providing valuable insights into the optimization of small VLMs for diverse tasks and environments. Furthermore, the use of significantly more unique tokens, at $256k$, offers an opportunity to investigate how a massively increased token set effects multi-modal performance.&#10;&#10;Recent advancements in (LLMs) \cite{vaswani2017attention} and multimodal foundation models (MMFMs) \cite{li2023multimodal} have propelled the interest and development of Large Multimodal Models (LMMs).&#10;Notable models like GPT-4 \cite{achiam2023gpt}, LLaVA \cite{liu2024visual,liu2023improved}, and their derivatives have demonstrated significant performance in vision-language tasks such as Visual Question Answering (VQA) and image captioning \cite{hudson2019gqa}. However, the computational demands of deploying these models have led to the exploration of small-scale LMMs.&#10;Our work aims to provide a unified analysis of small-scale LMMs, examining how model selections, training recipes, and data contribute to performance, which is distinct from existing works such as LLaVA-Phi.&#10;&#10;Our contributions are as follows: &#10;\begin{enumerate}&#10;\item We introduce LLaVA-Gemma, a MMFM that leverages the compact yet powerful Gemma language models for efficient multimodal interactions.&#10;\item We extensively evaluate Gemma-2B and Gemma-7B model variants provides valuable insights into the trade-offs between computational efficiency and the richness of visual and linguistic understanding in LLMs.&#10;\item We present a deep exploration into alternate design choices and visualize attention with relevancy maps to enhance our understanding of the model's performance and attention.&#10;&#10;\end{enumerate}&#10;&#10;We follow the LLaVA framework \cite{liu2023improved} with a few design modifications. This framework combines a pretrained vision encoder (such as CLIP \cite{radford2021learning}) and pretrained language model (such as Llama-2 \cite{touvron2023llama}) into a multimodal model using a MLP connector and a two-stage training procedure.&#10;&#10;The first stage pretrains the MLP connector by freezing the vision and language models and training on custom dataset of 595k samples filtered from CC3M \cite{sharma-etal-2018-conceptual}.&#10;The second stage jointly finetunes the language model and connector using a custom mixture 665k multimodal instruction tuning examples. This dataset includes synthetic data generated \cite{liu2024visual}, as well as examples from established vision-language training sets such as GQA \cite{hudson2019gqa} and TextCaps \cite{sidorov2020textcaps}.&#10;&#10;We deviate from the original recipe in three ways: the language model, the vision encoder and the pretraining stage.&#10;For the language backbone, we use the recently released Gemma models \cite{team2024gemma}. Two aspects of Gemma make it an interesting candidate for our experiments. Whereas LLaVA uses the 7 and 13-billion parameter vicuña langauge models \cite{zheng2023judging}, Gemma offers 2 and 7-billion parameter versions. Next, Gemma uses a significantly larger token set than any other LLM, with 256k unique tokens (compared to a standard ~50k), which offers a unique opportunity to see the effects of a massively more diverse embeddings space.&#10;Other papers exploring the design space of Vision Language Models (VLMs) find the vision encoder is important for achieving strong performance \cite{mckinzie2024mm1}. Correspondingly, we explore the use of the larger 1-billion parameter DINOv2 image encoder \cite{oquab2023dinov2} as the vision tower.&#10;Related work on VLMs \cite{karamcheti2024prismatic} finds that skipping the initial pretraining stage improves downstream performance. For all designs, we train a version with and without the initial pretraining step.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Deep Learning Architectures, Multimodal Foundation Models" />
        </attvalues>
      </node>
      <node id="2401.02330" label="2401.02330">
        <attvalues>
          <attvalue for="0" value="LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model" />
          <attvalue for="1" value="  In this paper, we introduce LLaVA-$\phi$ (LLaVA-Phi), an efficient&#10;multi-modal assistant that harnesses the power of the recently advanced small&#10;language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a&#10;notable advancement in the realm of compact multi-modal models. It demonstrates&#10;that even smaller language models, with as few as 2.7B parameters, can&#10;effectively engage in intricate dialogues that integrate both textual and&#10;visual elements, provided they are trained with high-quality corpora. Our model&#10;delivers commendable performance on publicly available benchmarks that&#10;encompass visual comprehension, reasoning, and knowledge-based perception.&#10;Beyond its remarkable performance in multi-modal dialogue tasks, our model&#10;opens new avenues for applications in time-sensitive environments and systems&#10;that require real-time interaction, such as embodied agents. It highlights the&#10;potential of smaller language models to achieve sophisticated levels of&#10;understanding and interaction, while maintaining greater resource&#10;efficiency.The project is available at {https://github.com/zhuyiche/llava-phi}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.03744" label="2310.03744">
        <attvalues>
          <attvalue for="0" value="Improved Baselines with Visual Instruction Tuning" />
          <attvalue for="1" value="  Large multimodal models (LMM) have recently shown encouraging progress with&#10;visual instruction tuning. In this note, we show that the fully-connected&#10;vision-language cross-modal connector in LLaVA is surprisingly powerful and&#10;data-efficient. With simple modifications to LLaVA, namely, using&#10;CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA&#10;data with simple response formatting prompts, we establish stronger baselines&#10;that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint&#10;uses merely 1.2M publicly available data, and finishes full training in ~1 day&#10;on a single 8-A100 node. We hope this can make state-of-the-art LMM research&#10;more accessible. Code and model will be publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.09506" label="1902.09506">
        <attvalues>
          <attvalue for="0" value="GQA: A New Dataset for Real-World Visual Reasoning and Compositional&#10;  Question Answering" />
          <attvalue for="1" value="  We introduce GQA, a new dataset for real-world visual reasoning and&#10;compositional question answering, seeking to address key shortcomings of&#10;previous VQA datasets. We have developed a strong and robust question engine&#10;that leverages scene graph structures to create 22M diverse reasoning&#10;questions, all come with functional programs that represent their semantics. We&#10;use the programs to gain tight control over the answer distribution and present&#10;a new tunable smoothing technique to mitigate question biases. Accompanying the&#10;dataset is a suite of new metrics that evaluate essential qualities such as&#10;consistency, grounding and plausibility. An extensive analysis is performed for&#10;baselines as well as state-of-the-art models, providing fine-grained results&#10;for different question types and topologies. Whereas a blind LSTM obtains mere&#10;42.1%, and strong VQA models achieve 54.1%, human performance tops at 89.3%,&#10;offering ample opportunity for new research to explore. We strongly hope GQA&#10;will provide an enabling resource for the next generation of models with&#10;enhanced robustness, improved consistency, and deeper semantic understanding&#10;for images and language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.12462" label="2003.12462">
        <attvalues>
          <attvalue for="0" value="TextCaps: a Dataset for Image Captioning with Reading Comprehension" />
          <attvalue for="1" value="  Image descriptions can help visually impaired people to quickly understand&#10;the image content. While we made significant progress in automatically&#10;describing images and optical character recognition, current approaches are&#10;unable to include written text in their descriptions, although text is&#10;omnipresent in human environments and frequently critical to understand our&#10;surroundings. To study how to comprehend text in the context of an image we&#10;collect a novel dataset, TextCaps, with 145k captions for 28k images. Our&#10;dataset challenges a model to recognize text, relate it to its visual context,&#10;and decide what part of the text to copy or paraphrase, requiring spatial,&#10;semantic, and visual reasoning between multiple text tokens and visual&#10;entities, such as objects. We study baselines and adapt existing approaches to&#10;this new task, which we refer to as image captioning with reading&#10;comprehension. Our analysis with automatic and human studies shows that our new&#10;TextCaps dataset provides many new technical challenges over previous datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.07193" label="2304.07193">
        <attvalues>
          <attvalue for="0" value="DINOv2: Learning Robust Visual Features without Supervision" />
          <attvalue for="1" value="  The recent breakthroughs in natural language processing for model pretraining&#10;on large quantities of data have opened the way for similar foundation models&#10;in computer vision. These models could greatly simplify the use of images in&#10;any system by producing all-purpose visual features, i.e., features that work&#10;across image distributions and tasks without finetuning. This work shows that&#10;existing pretraining methods, especially self-supervised methods, can produce&#10;such features if trained on enough curated data from diverse sources. We&#10;revisit existing approaches and combine different techniques to scale our&#10;pretraining in terms of data and model size. Most of the technical&#10;contributions aim at accelerating and stabilizing the training at scale. In&#10;terms of data, we propose an automatic pipeline to build a dedicated, diverse,&#10;and curated image dataset instead of uncurated data, as typically done in the&#10;self-supervised literature. In terms of models, we train a ViT model&#10;(Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of&#10;smaller models that surpass the best available all-purpose features, OpenCLIP&#10;(Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.07865" label="2402.07865">
        <attvalues>
          <attvalue for="0" value="Prismatic VLMs: Investigating the Design Space of Visually-Conditioned&#10;  Language Models" />
          <attvalue for="1" value="  Visually-conditioned language models (VLMs) have seen growing adoption in&#10;applications such as visual dialogue, scene understanding, and robotic task&#10;planning; adoption that has fueled a wealth of new models such as LLaVa,&#10;InstructBLIP, and PaLI-3. Despite the volume of new releases, key design&#10;decisions around image preprocessing, architecture, and optimization are&#10;under-explored, making it challenging to understand what factors account for&#10;model performance $-$ a challenge further complicated by the lack of objective,&#10;consistent evaluations. To address these gaps, we first compile a suite of&#10;standardized evaluations spanning visual question answering, object&#10;localization, and challenge sets that probe properties such as hallucination;&#10;evaluations that provide fine-grained insight VLM capabilities. Second, we&#10;rigorously investigate VLMs along key design axes, including pretrained visual&#10;representations and training from base vs. instruct-tuned language models,&#10;amongst others. We couple our analysis with three resource contributions: (1) a&#10;unified framework for evaluating VLMs, (2) optimized, flexible training code,&#10;and (3) checkpoints for all models, including a family of VLMs at the 7-13B&#10;scale that strictly outperform InstructBLIP and LLaVa v1.5, the&#10;state-of-the-art in open VLMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.08560" label="2009.08560">
        <attvalues>
          <attvalue for="0" value="Small but Mighty: New Benchmarks for Split and Rephrase" />
          <attvalue for="1" value="  Split and Rephrase is a text simplification task of rewriting a complex&#10;sentence into simpler ones. As a relatively new task, it is paramount to ensure&#10;the soundness of its evaluation benchmark and metric. We find that the widely&#10;used benchmark dataset universally contains easily exploitable syntactic cues&#10;caused by its automatic generation process. Taking advantage of such cues, we&#10;show that even a simple rule-based model can perform on par with the&#10;state-of-the-art model. To remedy such limitations, we collect and release two&#10;crowdsourced benchmark datasets. We not only make sure that they contain&#10;significantly more diverse syntax, but also carefully control for their quality&#10;according to a well-defined set of criteria. While no satisfactory automatic&#10;metric exists, we apply fine-grained manual evaluation based on these criteria&#10;using crowdsourcing, showing that our datasets better represent the task and&#10;are significantly more challenging for the models.&#10;" />
          <attvalue for="2" value="&#10;Split and Rephrase is the task of rewriting a presumably long and complex sentence into shorter and simpler sentences, while maintaining the same meaning. For example, one possible way to split the sentence ``Voiced by Aoi Koga, Kaguya is the series' titular character, popular among a wide audience.'' would result in ``Kaguya is voiced by Aoi Koga. Kaguya is the series' titular character. Kaguya is popular among a wide audience.'' While the split sentences have to be coherent, paraphrasing is not enforced. For example, the word ``titular'' does not have to be replaced. This type of text simplification is challenging as its natural language generation process potentially involves multiple sub-processes such as co-reference resolution, named-entity recognition, semantic role labelling, etc. Split and Rephrase has two main real-world uses: first, to benefit systems whose performance improves with decreasing length of sentences e.g. entity extraction \cite{zhang-etal-2017-position} and machine translation \cite{koehn-knowles-2017-six} by acting as a pre-processing step; second, to benefit human readers, especially those less proficient with the language when reading complex documents such as terms and agreements, in understanding the meaning more easily and accurately \cite{Inui2003TextSF,Siddharthan2002AnAF}. &#10;&#10;Datasets of the Split and Rephrase task contain pairs of a complex sentence and a presumably meaning-preserving simplified rewrite containing multiple simpler sentences. The task was introduced by \cite{narayan-etal-2017-split}, with the release of the WebSplit corpus. Afterwards, \cite{aharoni-goldberg-2018-split} proposed the state-of-the-art model to date, a sequence-to-sequence model \cite{Bahdanau2015NeuralMT} with a copy mechanism \cite{gu-etal-2016-incorporating,see-etal-2017-get} with the observation that most texts are unchanged during a Split and Rephrase operation. Later, \cite{botha-etal-2018-learning} introduced the WikiSplit corpus to be used as large but noisy training data, which the authors reported to be unsuitable as the evaluation data. Also, \cite{sulem-etal-2018-bleu} studied the problems of using BLEU as the evaluation metric for this task, while proposing a manually constructed test set called HSplit. &#10;&#10;We argue that the widely used benchmark dataset of Split and Rephrase, the WebSplit test set (known as simply WebSplit below), is not suitable for evaluation. Apart from its series of limitations already reported, such as a small vocabulary, unnatural expressions, etc. \cite{botha-etal-2018-learning}, we further show that its complex sentences systematically follow only 3 syntactical patterns marked by lexical cues (Section~\ref{sec:issues}). To demonstrate the implication of such limitations of WebSplit, we show that a simple, unsupervised rule-based model with only 3 corresponding operations can perform even slightly better than the state-of-the-art neural model (Section~\ref{sec:rule-basedModel}). &#10;&#10;To remedy the limitations of WebSplit, we crowdsource two new benchmarks with significantly more diverse syntax in the Wikipedia and legal contract domain with hundreds of human-written complex-simple sentence pairs (Section~\ref{sec:newBenchmark}). We carefully control for their quality based on 6 well-defined criteria of what constitutes a good Split and Rephrase rewrite. While most related work reports model performance using the widely criticized BLEU score and manual evaluation with no clear rubric, we perform fine-grained model evaluation using these 6 criteria, rated by crowd workers, showing that our benchmarks present models with greater challenges (Section~\ref{sec:modelPerformancee}).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Benchmark Evaluation, Text Simplification, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1707.06971" label="1707.06971">
        <attvalues>
          <attvalue for="0" value="Split and Rephrase" />
          <attvalue for="1" value="  We propose a new sentence simplification task (Split-and-Rephrase) where the&#10;aim is to split a complex sentence into a meaning preserving sequence of&#10;shorter sentences. Like sentence simplification, splitting-and-rephrasing has&#10;the potential of benefiting both natural language processing and societal&#10;applications. Because shorter sentences are generally better processed by NLP&#10;systems, it could be used as a preprocessing step which facilitates and&#10;improves the performance of parsers, semantic role labellers and machine&#10;translation systems. It should also be of use for people with reading&#10;disabilities because it allows the conversion of longer sentences into shorter&#10;ones. This paper makes two contributions towards this new task. First, we&#10;create and make available a benchmark consisting of 1,066,115 tuples mapping a&#10;single complex sentence to a sequence of sentences expressing the same meaning.&#10;Second, we propose five models (vanilla sequence-to-sequence to&#10;semantically-motivated models) to understand the difficulty of the proposed&#10;task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.09468" label="1808.09468">
        <attvalues>
          <attvalue for="0" value="Learning To Split and Rephrase From Wikipedia Edit History" />
          <attvalue for="1" value="  Split and rephrase is the task of breaking down a sentence into shorter ones&#10;that together convey the same meaning. We extract a rich new dataset for this&#10;task by mining Wikipedia's edit history: WikiSplit contains one million&#10;naturally occurring sentence rewrites, providing sixty times more distinct&#10;split examples and a ninety times larger vocabulary than the WebSplit corpus&#10;introduced by Narayan et al. (2017) as a benchmark for this task. Incorporating&#10;WikiSplit as training data produces a model with qualitatively better&#10;predictions that score 32 BLEU points above the prior best result on the&#10;WebSplit benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.07328" label="2310.07328">
        <attvalues>
          <attvalue for="0" value="An Empirical Study of Instruction-tuning Large Language Models in&#10;  Chinese" />
          <attvalue for="1" value="  The success of ChatGPT validates the potential of large language models&#10;(LLMs) in artificial general intelligence (AGI). Subsequently, the release of&#10;LLMs has sparked the open-source community's interest in instruction-tuning,&#10;which is deemed to accelerate ChatGPT's replication process. However, research&#10;on instruction-tuning LLMs in Chinese, the world's most spoken language, is&#10;still in its early stages. Therefore, this paper makes an in-depth empirical&#10;study of instruction-tuning LLMs in Chinese, which can serve as a cookbook that&#10;provides valuable findings for effectively customizing LLMs that can better&#10;respond to Chinese instructions. Specifically, we systematically explore the&#10;impact of LLM bases, parameter-efficient methods, instruction data types, which&#10;are the three most important elements for instruction-tuning. Besides, we also&#10;conduct experiment to study the impact of other factors, e.g., chain-of-thought&#10;data and human-value alignment. We hope that this empirical study can make a&#10;modest contribution to the open Chinese version of ChatGPT. This paper will&#10;release a powerful Chinese LLMs that is comparable to ChatGLM. The code and&#10;data are available at https://github.com/PhoebusSi/Alpaca-CoT.&#10;" />
          <attvalue for="2" value="&#10;&#10;The emergence of ChatGPT gives humanity a real sense of hope for AGI for the first time, and inspires researchers to realize the importance of LLM research. However, the closed source of LLMs (e.g., GPT-3 \cite{brown2020language} and PaLM \cite{chowdhery2022palm}) coupled with the requirement for massive computing resources to build the exclusive LLM has deterred researchers from reaching the LLM training stage. Subsequently, a series of &quot;API research&quot; based on GPT-3 and ChatGPT are constantly emerging, which stimulate the specific capabilities of frozen LLMs (e.g., Chain-of-Thought \cite{wei2023chainofthought,wang2023selfconsistency,kojima2023large}) or guide them to complete specific tasks \cite{yang2022empirical,shen2023hugginggpt}, by calling OpenAI interfaces and carefully designing prompts without model training. &#10;&#10;The unexpected disclosure of the pre-trained LLaMA \cite{touvron2023llama} model changes this situation, and has sparked a surge of excitement in the LLM research community. This is the first open LLM with competitive performance. Recently, Alpaca \cite{alpaca} uses self-instruct \cite{ouyang2022training} and ChatGPT to generate 52K instructions, which can enable LLaMA to respond to various human instructions like ChatGPT. This open project verifies the important role of instruction-tuning \cite{wei2022finetuned, chung2022scaling} open LLMs in replicating the ChatGPT process. &#10;&#10;Given the open LLM LLaMA and Alpaca's high-quality instruction data, there is still a challenge for researchers: even the instruction-tuning of the 7B model still requires high computational resources. To address this problem, Alpaca-LoRA extends the parameter-efficient method LoRA to LLaMA, which further reduces the computing cost of instruction-tuning. It further sparks extensive research in the open-source community on instruction-tuning for LLMs. On this basis, more LLMs (e.g, Bloom \cite{workshop2023bloom}, GPT-J \cite{gpt-j}) are shown to have significant improvements in instruction-following performance with instruction-tuning. On the other hand, more instruction data is constantly being proposed, e.g., Belle \cite{BELLE} constructs Chinese instructions in the same way, and ShareGPT collects a large number of real human-ChatGPT conversations. &#10;&#10;However, research on instruction-tuning LLMs in Chinese, the world’s most spoken language, is still in its early stages. LLM bases, parameter-efficient methods, and instruction data are three essential elements for customizing Chinese ChatGPT-like LLMs. There are no tutorials in the academic community on them yet. Some important questions have not yet been explored and answered: 1) &quot;Which open LLM is more suitable as a foundation for Chinese instruction-tuning?&quot;, 2) &quot;How do parameter-efficient methods other than LoRA affect LLMs?&quot; and 3) &quot;What is the impact of various types of instruction datasets?&quot; &#10;To answer these questions, we collect a range of LLMs, parameter-efficient methods, and instruction datasets. Besides, we consider the AGI (instruction-following) capability and professional knowledge reserve (human exams) of models, and correspondingly select two benchmarks Belle-eval \cite{BELLE} and MMCU \cite{zeng2023measuring} for comprehensively evaluation. &#10;&#10;We also conduct experiments to explore several other factors that may affect the final performance. Specifically, we find that tuning with Chain-of-Thought (CoT) data can improve the ability to respond to complex reasoning questions. &#10;Different LLMs may be suitable for different language prompts (excluding instruction parts) in instruction-tuning. Human-value alignment results into slight performance drop. &#10;On the basis of the above findings, this paper carefully instruction-tunes a powerful Chinese LLMs that is comparable to ChatGLM.&#10;&#10;The contributions can be summarized as follows:&#10; (1) We are the first to systematically study on instruction-tuning in Chinese through adequate experiments, which can serve as a cookbook that provides valuable findings for customizing Chinese version of ChatGPT. &#10;(2) We release a powerful Chinese LLM that is comparable to ChatGLM.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Artificial General Intelligence, Artificial Intelligence, Natural Language Processing, Instruction Tuning" />
        </attvalues>
      </node>
      <node id="2109.05014" label="2109.05014">
        <attvalues>
          <attvalue for="0" value="An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA" />
          <attvalue for="1" value="  Knowledge-based visual question answering (VQA) involves answering questions&#10;that require external knowledge not present in the image. Existing methods&#10;first retrieve knowledge from external resources, then reason over the selected&#10;knowledge, the input image, and question for answer prediction. However, this&#10;two-step approach could lead to mismatches that potentially limit the VQA&#10;performance. For example, the retrieved knowledge might be noisy and irrelevant&#10;to the question, and the re-embedded knowledge features during reasoning might&#10;deviate from their original meanings in the knowledge base (KB). To address&#10;this challenge, we propose PICa, a simple yet effective method that Prompts&#10;GPT3 via the use of Image Captions, for knowledge-based VQA. Inspired by&#10;GPT-3's power in knowledge retrieval and question answering, instead of using&#10;structured KBs as in previous work, we treat GPT-3 as an implicit and&#10;unstructured KB that can jointly acquire and process relevant knowledge.&#10;Specifically, we first convert the image into captions (or tags) that GPT-3 can&#10;understand, then adapt GPT-3 to solve the VQA task in a few-shot manner by just&#10;providing a few in-context VQA examples. We further boost performance by&#10;carefully investigating: (i) what text formats best describe the image content,&#10;and (ii) how in-context examples can be better selected and used. PICa unlocks&#10;the first use of GPT-3 for multimodal tasks. By using only 16 examples, PICa&#10;surpasses the supervised state of the art by an absolute +8.6 points on the&#10;OK-VQA dataset. We also benchmark PICa on VQAv2, where PICa also shows a decent&#10;few-shot performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.12986" label="2304.12986">
        <attvalues>
          <attvalue for="0" value="Measuring Massive Multitask Chinese Understanding" />
          <attvalue for="1" value="  The development of large-scale Chinese language models is flourishing, yet&#10;there is a lack of corresponding capability assessments. Therefore, we propose&#10;a test to measure the multitask accuracy of large Chinese language models. This&#10;test encompasses four major domains, including medicine, law, psychology, and&#10;education, with 15 subtasks in medicine and 8 subtasks in education. We found&#10;that the best-performing models in the zero-shot setting outperformed the&#10;worst-performing models by nearly 18.6 percentage points on average. Across the&#10;four major domains, the highest average zero-shot accuracy of all models is&#10;0.512. In the subdomains, only the GPT-3.5-turbo model achieved a zero-shot&#10;accuracy of 0.693 in clinical medicine, which was the highest accuracy among&#10;all models across all subtasks. All models performed poorly in the legal&#10;domain, with the highest zero-shot accuracy reaching only 0.239. By&#10;comprehensively evaluating the breadth and depth of knowledge across multiple&#10;disciplines, this test can more accurately identify the shortcomings of the&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.00415" label="2205.00415">
        <attvalues>
          <attvalue for="0" value="Don't Blame the Annotator: Bias Already Starts in the Annotation&#10;  Instructions" />
          <attvalue for="1" value="  In recent years, progress in NLU has been driven by benchmarks. These&#10;benchmarks are typically collected by crowdsourcing, where annotators write&#10;examples based on annotation instructions crafted by dataset creators. In this&#10;work, we hypothesize that annotators pick up on patterns in the crowdsourcing&#10;instructions, which bias them to write many similar examples that are then&#10;over-represented in the collected data. We study this form of bias, termed&#10;instruction bias, in 14 recent NLU benchmarks, showing that instruction&#10;examples often exhibit concrete patterns, which are propagated by crowdworkers&#10;to the collected data. This extends previous work (Geva et al., 2019) and&#10;raises a new concern of whether we are modeling the dataset creator's&#10;instructions, rather than the task. Through a series of experiments, we show&#10;that, indeed, instruction bias can lead to overestimation of model performance,&#10;and that models struggle to generalize beyond biases originating in the&#10;crowdsourcing instructions. We further analyze the influence of instruction&#10;bias in terms of pattern frequency and model size, and derive concrete&#10;recommendations for creating future NLU benchmarks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Benchmarks have been proven pivotal for driving progress in Natural Language Understanding (NLU) in recent years \cite{rogers2021qa, bach2022promptsource, wang2022benchmarking}. Nowadays, NLU benchmarks are mostly created through crowdsourcing, where crowdworkers write examples &#10;following annotation instructions crafted by dataset creators \cite{callison-burch-dredze-2010-creating, zheng2018crowdsourcing, suhr-etal-2021-crowdsourcing}. The instructions typically include a short description of the task, along with several examples \cite{dasigi2019quoref, zhou-etal-2019-going, sakaguchi2020winogrande}. &#10;&#10;Despite the vast success of this method, past studies have shown that data collected through crowdsourcing often exhibit various biases that lead to overestimation of model performance \cite{schwartz2017effect, gururangan-etal-2018-annotation, poliak2018hypothesis, tsuchiya-2018-performance, le2020adversarial, Mishra2020OurEM, mishra2021robust, hettiachchi2021investigating}. Such biases are often attributed to annotator-related biases, such as writing style and background knowledge \cite{gururangan-etal-2018-annotation, geva-etal-2019-modeling} (see more discussion on related work in \S\ref{sec:related_work}).&#10;&#10;In this work, we propose that biases in crowdsourced NLU benchmarks often originate at an early stage in the data collection process of designing the annotation task. In particular, we hypothesize that task instructions &#10;provided by dataset creators, which serve as the guiding principles for annotators to complete the task, often influence crowdworkers to follow specific patterns, which are then propagated to the dataset and subsequently over-represented in the collected data. For instance, $\sim36\%$ of the instruction examples for the \quoref dataset \cite{dasigi2019quoref} start with ``What is the name'', and this same pattern can be observed in $\sim59\%$ of the collected instances.&#10;&#10;To test our hypothesis, we conduct a broad study of this form of bias, termed \instructionbias, in 14 recent NLU benchmarks. We find that instruction bias is evident in most of these datasets, showing that $\sim73\%$ of instruction examples on average share a few clear patterns.&#10;Moreover, we find that these patterns are propagated by annotators to the collected data, covering $\sim61\%$ of the instances on average. This suggests that instruction examples play a critical role in the data collection process and the resulting example distribution. &#10;&#10;It is difficult to represent a task with a few examples, and bias in instruction examples makes it even more difficult since a task and its associated reasoning have a larger scope than instruction patterns. For example co-reference resolution, temporal commonsense reasoning, and numerical reasoning are much broader tasks than the prevalent patterns in \quoref (``what is the name...''), \mctaco (``how long...'') and \drop (``how many field goals...'') datasets.&#10;&#10;We investigate the effect of instruction bias on model performance, showing that performance is overestimated by instruction bias and that models often fail to generalize beyond instruction patterns. Moreover, we observe that a higher frequency of instruction patterns in the training set often increases the model performance gap on pattern and non-pattern examples and that large models are generally less sensitive to instruction bias. &#10;&#10;In conclusion, our work shows that instruction bias widely exists in NLU benchmarks, often leading to an overestimation of model performance.&#10;Based on our study, we derive concrete recommendations for monitoring and alleviating this bias in future data collection efforts.&#10;From a broader perspective, our findings also have implications on the recent learning-by-instructions paradigm \cite{efrat2020turking, mishra2021cross}, where crowdsourcing instructions are used in model training. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Linguistics, Model Performance Evaluation, Natural Language Understanding, Crowdsourcing Bias, Artificial Intelligence, Natural Language Processing, Psychology" />
        </attvalues>
      </node>
      <node id="1908.05803" label="1908.05803">
        <attvalues>
          <attvalue for="0" value="Quoref: A Reading Comprehension Dataset with Questions Requiring&#10;  Coreferential Reasoning" />
          <attvalue for="1" value="  Machine comprehension of texts longer than a single sentence often requires&#10;coreference resolution. However, most current reading comprehension benchmarks&#10;do not contain complex coreferential phenomena and hence fail to evaluate the&#10;ability of models to resolve coreference. We present a new crowdsourced dataset&#10;containing more than 24K span-selection questions that require resolving&#10;coreference among entities in over 4.7K English paragraphs from Wikipedia.&#10;Obtaining questions focused on such phenomena is challenging, because it is&#10;hard to avoid lexical cues that shortcut complex reasoning. We deal with this&#10;issue by using a strong baseline model as an adversary in the crowdsourcing&#10;loop, which helps crowdworkers avoid writing questions with exploitable surface&#10;cues. We show that state-of-the-art reading comprehension models perform&#10;significantly worse than humans on this benchmark---the best model performance&#10;is 70.5 F1, while the estimated human performance is 93.4 F1.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11982" label="2010.11982">
        <attvalues>
          <attvalue for="0" value="The Turking Test: Can Language Models Understand Instructions?" />
          <attvalue for="1" value="  Supervised machine learning provides the learner with a set of input-output&#10;examples of the target task. Humans, however, can also learn to perform new&#10;tasks from instructions in natural language. Can machines learn to understand&#10;instructions as well? We present the Turking Test, which examines a model's&#10;ability to follow natural language instructions of varying complexity. These&#10;range from simple tasks, like retrieving the nth word of a sentence, to ones&#10;that require creativity, such as generating examples for SNLI and SQuAD in&#10;place of human intelligence workers (&quot;turkers&quot;). Despite our lenient evaluation&#10;methodology, we observe that a large pretrained language model performs poorly&#10;across all tasks. Analyzing the model's error patterns reveals that the model&#10;tends to ignore explicit instructions and often generates outputs that cannot&#10;be construed as an attempt to solve the task. While it is not yet clear whether&#10;instruction understanding can be captured by traditional language models, the&#10;sheer expressivity of instruction understanding makes it an appealing&#10;alternative to the rising few-shot inference paradigm.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2405.00602" label="2405.00602">
        <attvalues>
          <attvalue for="0" value="Investigating Automatic Scoring and Feedback using Large Language Models" />
          <attvalue for="1" value="  Automatic grading and feedback have been long studied using traditional&#10;machine learning and deep learning techniques using language models. With the&#10;recent accessibility to high performing large language models (LLMs) like&#10;LLaMA-2, there is an opportunity to investigate the use of these LLMs for&#10;automatic grading and feedback generation. Despite the increase in performance,&#10;LLMs require significant computational resources for fine-tuning and additional&#10;specific adjustments to enhance their performance for such tasks. To address&#10;these issues, Parameter Efficient Fine-tuning (PEFT) methods, such as LoRA and&#10;QLoRA, have been adopted to decrease memory and computational requirements in&#10;model fine-tuning. This paper explores the efficacy of PEFT-based quantized&#10;models, employing classification or regression head, to fine-tune LLMs for&#10;automatically assigning continuous numerical grades to short answers and&#10;essays, as well as generating corresponding feedback. We conducted experiments&#10;on both proprietary and open-source datasets for our tasks. The results show&#10;that prediction of grade scores via finetuned LLMs are highly accurate,&#10;achieving less than 3% error in grade percentage on average. For providing&#10;graded feedback fine-tuned 4-bit quantized LLaMA-2 13B models outperform&#10;competitive base models and achieve high similarity with subject matter expert&#10;feedback in terms of high BLEU and ROUGE scores and qualitatively in terms of&#10;feedback. The findings from this study provide important insights into the&#10;impacts of the emerging capabilities of using quantization approaches to&#10;fine-tune LLMs for various downstream tasks, such as automatic short answer&#10;scoring and feedback generation at comparatively lower costs and latency.&#10;" />
          <attvalue for="2" value=" &#10;&#10;With the rapid advancements of artificial intelligence (AI) and natural language processing (NLP) approaches, there has been increasing interest in developing more AI-powered grading and feedback systems for educational purposes \cite{lu2021integrating,jia2022automated, baral2021improving,suzen2020automatic, zhang2022automatic, qiu2022toward}. Research has extensively explored automatic scoring and feedback generation, yet natural language responses, specifically short answers and essays, present ongoing challenges due to their varying length, focus, and level of openness. \cite{burrows2015eras} With the advent and wider availability of language models, the potential for leveraging these technologies to enhance the capabilities of automatic grading and feedback systems has been a focus of investigation. These advancements are not intended to supplant educators but rather to augment their capabilities, offering time-saving benefits and enabling personalized feedback for students, a particularly valuable asset in settings where instructor resources are constrained, such as online education environments.&#10;&#10;The introduction of high-capacity large language models (LLMs) like OpenAI's GPT \cite{radford2019language} and its successors, along with Meta's LLaMA-2 \cite{touvron2023llama2}, opens new avenues for exploring the application of cutting-edge LLMs in automated grading and feedback provision. While access to top-tier LLMs including GPT-3 \cite{brown2020language}, GPT-4 \cite{openai2023gpt4}, Chinchilla \cite{hoffmann2022training}, and both iterations of PaLM \cite{chowdhery2022palm, anil2023palm} remains restricted, the release of models like Falcon \cite{almazrouei2023falcon}, LLaMA \cite{touvron2023llama}, and LLaMA-2 \cite{touvron2023llama2} under open-access terms has democratized access to powerful LLMs. This newfound accessibility enables their fine-tuning for a range of specialized tasks, including the automation of grading and feedback generation.&#10;&#10;Fine-tuning LLMs involve adapting the expansive knowledge of the pretrained LLMs for a target task, leading to the successful impacts of LLMs across many fields and applications \cite{wei2021finetuned}. In the context of automatic grading and feedback generation, fine-tuning these LLM often require two distinct approaches: For automatic grading, one common approach involves using discriminative techniques, such as classification and regression; whereas generative methods, which have grown in popularity due to the rise of GenAI, utilize generative models. With most high-performing LLMs primarily designed for generative tasks, our work aims to investigate the use of these LLMs for regression purposes as well, marking an innovative approach by integrating both capabilities into a unified system for advanced grading and feedback generation.&#10;&#10;Despite their improved performance, a significant challenge with LLMs is their demand for considerable computational resources during fine-tuning and inference phases. To mitigate this, there has been a growing interest in quantization techniques \cite{xiao2023smoothquant, yao2023comprehensive}. Quantization is done by compressing floating-point numbers to lower bit width numbers like int8 and int4. These techniques aim to reduce memory and computational demands without substantially compromising the model's performance. Such strategies fall under the umbrella of Parameter-efficient fine-tuning (PEFT), as discussed in \cite{liu2022few}, where the focus is on fine-tuning a small number of parameters while retaining the core capabilities of LLMs. However, PEFT has mainly been applied for generative fine-tuning tasks as opposed to discriminative tasks like classification and regression. &#10;&#10;In this work, we explored the effectiveness of fine-tuned 4-bit quantized LLama-2 models for automatically grading grading and feedback generation on our proprietary dataset and an open source dataset \cite{filighera2022your}. We conducted several experiments to examine the possibilities for an LLM-based grading and feedback system using quantized models. Specifically, we made adjustment to the model architecture for a regression tasks and then utilized supervised instruction fine-tuning, a well-known approach for finetuning LLMs. In particular, we investigate the following research questions: &#10;\begin{itemize}&#10; \item RQ1: Can fine-tuning quantized LLaMA-2 be leveraged to improve upon existing ML/DL approaches for automatic grading?&#10; \item RQ2: Can fine-tuning quantized LLaMA-2 be leveraged to improve upon existing ML/DL approaches for automatic feedback generation?&#10; \item RQ3: Can combining the regression and generative approaches lead to higher quality feedback generation?&#10;\end{itemize}&#10;&#10;To this end, we conducted experiments on open-source and proprietary datasets for our tasks. We demonstrate that fine-tuned LLMs, including a 4-bit quantized version of the LLaMA-2 13B model, can predict grades with remarkable accuracy, averaging less than 3\% error in grade percentage. For providing graded feedback fine-tuned 4-bit quantized LLaMA-2 13B models outperform competitive base models and achieve high similarity with subject matter expert feedback in terms of high BLEU and ROUGE scores and qualitatively in terms of feedback. The findings from this study will provide important insights into the impacts of the emerging capabilities of using quantization approaches to fine-tune LLMs for various downstream tasks, such as automatic short answer scoring and feedback generation, offering a more cost-effective and efficient solution while maintaining high accuracy and quality of feedback.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Automatic Grading Systems, Efficient Fine-tuning Techniques, Computer Science, Linguistics, Education, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1807.10543" label="1807.10543">
        <attvalues>
          <attvalue for="0" value="Automatic Short Answer Grading and Feedback Using Text Mining Methods" />
          <attvalue for="1" value="  Automatic grading is not a new approach but the need to adapt the latest&#10;technology to automatic grading has become very important. As the technology&#10;has rapidly became more powerful on scoring exams and essays, especially from&#10;the 1990s onwards, partially or wholly automated grading systems using&#10;computational methods have evolved and have become a major area of research. In&#10;particular, the demand of scoring of natural language responses has created a&#10;need for tools that can be applied to automatically grade these responses. In&#10;this paper, we focus on the concept of automatic grading of short answer&#10;questions such as are typical in the UK GCSE system, and providing useful&#10;feedback on their answers to students. We present experimental results on a&#10;dataset provided from the introductory computer science class in the University&#10;of North Texas. We first apply standard data mining techniques to the corpus of&#10;student answers for the purpose of measuring similarity between the student&#10;answers and the model answer. This is based on the number of common words. We&#10;then evaluate the relation between these similarities and marks awarded by&#10;scorers. We then consider an approach that groups student answers into&#10;clusters. Each cluster would be awarded the same mark, and the same feedback&#10;given to each answer in a cluster. In this manner, we demonstrate that clusters&#10;indicate the groups of students who are awarded the same or the similar scores.&#10;Words in each cluster are compared to show that clusters are constructed based&#10;on how many and which words of the model answer have been used. The main&#10;novelty in this paper is that we design a model to predict marks based on the&#10;similarities between the student answers and the model answer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.19553" label="2404.19553">
        <attvalues>
          <attvalue for="0" value="Extending Llama-3's Context Ten-Fold Overnight" />
          <attvalue for="1" value="  We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA&#10;fine-tuning. The entire training cycle is super efficient, which takes 8 hours&#10;on one 8xA800 (80G) GPU machine. The resulted model exhibits superior&#10;performances across a broad range of evaluation tasks, such as NIHS, topic&#10;retrieval, and long-context language understanding; meanwhile, it also well&#10;preserves the original capability over short contexts. The dramatic context&#10;extension is mainly attributed to merely 3.5K synthetic training samples&#10;generated by GPT-4 , which indicates the LLMs' inherent (yet largely&#10;underestimated) potential to extend its original context length. In fact, the&#10;context length could be extended far beyond 80K with more computation&#10;resources. Therefore, the team will publicly release the entire resources&#10;(including data, model, data generation pipeline, training code) so as to&#10;facilitate the future research from the community:&#10;\url{https://github.com/FlagOpen/FlagEmbedding}.&#10;" />
          <attvalue for="2" value="&#10;Recently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models~\cite{chen2023position_interpolation,peng2023yarn,chen2024longlora,ding2024longrope,fu2024data_engineer,zhang2024soaring,an2024make_your_llm_utilize_context}. However, most of them require significant compute and resources to accomplish.&#10;&#10;In this technical report, we propose an efficient solution for entitling the long-context capabilities for LLMs, with which we extend the context length of Llama-3-8B-Instruct from 8K to 80K. Specifically, we use GPT-4~\cite{openai2024gpt4} to synthesize 3.5K long-context training data, covering three long-context tasks:&#10;\begin{enumerate}[leftmargin=20pt,rightmargin=20pt]&#10; \item Single-Detail QA: the inquiry targets on one specific detail in a long context. To construct data for this task, we slice out a short segment (e.g., a chunk with less than 4096 tokens) from a long context (e.g., a book or a long paper) and prompt GPT-4 to generate multiple question-answer pairs based on this segment.&#10; \item Multi-Detail QA: the inquiry requires information aggregation and reasoning over multiple details in a long context. We define two types of long context. The homogeneous context contains a coherent text, such as a book or a long paper. We prompt GPT-4 to generate multiple question-answer pairs that require aggregating and analyzing information from different locations in the context. The heterogeneous context consists of multiple independent texts. Notably, we perform clustering over a large corpus then extract texts from the same cluster to form each heterogeneous context. Therefore, the grouped texts share some semantic similarity. We then prompt GPT-4 to ask about the similarities/dissimilarities across these texts.&#10; \item Biography Summarization: we prompt GPT-4 to write a biography for each main character in a given book.&#10;\end{enumerate}&#10;&#10;For all three tasks, the length of context is between 64K to 80K. Note that longer data can also be synthesized following the same methodology. When training, we organize the question-answer pairs for the same context in one multi-turn conversation then fine-tune the LLM to correctly answer the questions given the entire long context as input. &#10;Following previous work, we mix 5K instances randomly chosen from RedPajama~\cite{together2023redpajama} to mitigate forgetting. We also mix LongAlpaca~\cite{chen2024longlora} in the training set, which contains 12K instruction tuning instances with 16K length at maximum. Therefore, the entire training dataset contains 20K instances.&#10;&#10;We use QLoRA~\cite{dettmers2023qlora} to efficiently fine-tune the model. We apply LoRA on all Q,K,V,O projections and additionally train the embedding layer. We set LoRA rank to 32 and alpha to 16. The learning rate is 5e-5 with linear decay and no warmups. The batch size is 8. Gradient checkpointing is enabled. No parallel strategy is required thanks to the efficient implementation from Unsloth~\cite{unsloth}. We train the model for 1 epoch, which takes 8 hours to complete on a 8xA800 (80G) machine. Importantly, we expand the RoPE base from 500K to 200M in training.&#10;&#10;Our contributions are highlighted as follows:&#10;\begin{itemize}[leftmargin=20pt,rightmargin=20pt]&#10; \item We release Llama-3-8B-Instruct-80K-QLoRA, which extends the context length of Llama-3-8B-Instruct from 8K to 80K. The entire resources including the model, training data, and code are all publicly available, which may advance the field of training long-context LLMs.&#10; \item Our training recipe is simple and efficient, while the resulted model demonstrates remarkable performance on downstream long-context tasks. Further research can be made to improve our approach.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Artificial Intelligence Efficiency, Computer Science, Linguistics, Cognitive Science, Mathematics, Language Model Extension, Context Length Expansion" />
        </attvalues>
      </node>
      <node id="2404.16811" label="2404.16811">
        <attvalues>
          <attvalue for="0" value="Make Your LLM Fully Utilize the Context" />
          <attvalue for="1" value="  While many contemporary large language models (LLMs) can process lengthy&#10;input, they still struggle to fully utilize information within the long&#10;context, known as the lost-in-the-middle challenge. We hypothesize that it&#10;stems from insufficient explicit supervision during the long-context training,&#10;which fails to emphasize that any position in a long context can hold crucial&#10;information. Based on this intuition, our study presents information-intensive&#10;(IN2) training, a purely data-driven solution to overcome lost-in-the-middle.&#10;Specifically, IN2 training leverages a synthesized long-context question-answer&#10;dataset, where the answer requires (1) fine-grained information awareness on a&#10;short segment (~128 tokens) within a synthesized long context (4K-32K tokens),&#10;and (2) the integration and reasoning of information from two or more short&#10;segments. Through applying this information-intensive training on Mistral-7B,&#10;we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of&#10;FILM-7B for utilizing long contexts, we design three probing tasks that&#10;encompass various context styles (document, code, and structured-data context)&#10;and information retrieval patterns (forward, backward, and bi-directional&#10;retrieval). The probing results demonstrate that FILM-7B can robustly retrieve&#10;information from different positions in its 32K context window. Beyond these&#10;probing tasks, FILM-7B significantly improves the performance on real-world&#10;long-context tasks (e.g., 23.5-&gt;26.9 F1 score on NarrativeQA), while&#10;maintaining a comparable performance on short-context tasks (e.g., 59.3-&gt;59.2&#10;accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10750" label="2212.10750">
        <attvalues>
          <attvalue for="0" value="PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and&#10;  Entailment Recognition" />
          <attvalue for="1" value="  The widely studied task of Natural Language Inference (NLI) requires a system&#10;to recognize whether one piece of text is textually entailed by another, i.e.&#10;whether the entirety of its meaning can be inferred from the other. In current&#10;NLI datasets and models, textual entailment relations are typically defined on&#10;the sentence- or paragraph-level. However, even a simple sentence often&#10;contains multiple propositions, i.e. distinct units of meaning conveyed by the&#10;sentence. As these propositions can carry different truth values in the context&#10;of a given premise, we argue for the need to recognize the textual entailment&#10;relation of each proposition in a sentence individually.&#10;  We propose PropSegmEnt, a corpus of over 45K propositions annotated by expert&#10;human raters. Our dataset structure resembles the tasks of (1) segmenting&#10;sentences within a document to the set of propositions, and (2) classifying the&#10;entailment relation of each proposition with respect to a different yet&#10;topically-aligned document, i.e. documents describing the same event or entity.&#10;We establish strong baselines for the segmentation and entailment tasks.&#10;Through case studies on summary hallucination detection and document-level NLI,&#10;we demonstrate that our conceptual framework is potentially useful for&#10;understanding and explaining the compositionality of NLI labels.&#10;" />
          <attvalue for="2" value="&#10;Natural Language Inference (NLI), or Recognizing Textual Entailment (RTE), is the task of determining whether the meaning of one text expression can be inferred from another \cite{DaganGl04}. &#10;Given two pieces of text $(P, H)$, we say the premise $P$ entails the hypothesis $H$ if the entirety of $H$'s meaning can be most likely inferred true after a human reads $P$. If some units of meaning in $H$ are contradicted by, or cannot be determined &#10;from $P$, we describe the relation between the two as contradiction or neutral \cite{MarneffeRaMa08} respectively.&#10;This fundamentally challenging natural language understanding task provides a general interface for semantic inference and comparison across different sources of textual information.&#10;&#10;In reality, most naturally occurring text expressions are composed of a variable number of propositions, i.e. distinct units of meaning conveyed by the piece of text. Consider the sentence shown in Table \ref{tab:lead-example}: ``The Andy Warhol Museum in his hometown, Pittsburgh, Pennsylvania, contains an extensive permanent collection of art.'' Despite the sentence being relatively compact, it still contains (at least) three propositions, as listed in Table~\ref{tab:lead-example}. &#10;While the entire hypothesis would be classified as neutral or not-entailed to the premise, one of its propositions ``Andy Warhol's hometown is in Pittsburgh, Pennsylvania'' is in fact entailed by the premise, while the premise provides no support for the other two propositions. &#10;This phenomenon, namely partial entailment \cite{ levy-etal-2013-recognizing}, is a blind spot for existing sentence- or paragraph-level NLI formulations. &#10;When a hypothesis is compositional, NLI labels coarsely defined on the sentence/paragraph-level cannot express the difference between partial entailment from the non-entailment cases. &#10;&#10;This work argues for the need to study and model textual entailment relations on the level of propositions. &#10;As NLI tasks and applications typically involve different genre of text with variable length and number of propositions \cite{yin-etal-2021-docnli}, decomposing textual entailment relation to the propositional level provides a more fine-grained yet accurate description of textual entailment relation between two arbitrary text expressions. Modeling propositional textual entailment provides a more unified inference format across NLI tasks, and would potentially improve the generalization capabilities of NLI models, e.g.\ with respect to the variability in input lengths \cite{schuster2022stretching}.\looseness=-1&#10;&#10;We propose PropSegmEnt, a multi-domain corpus with over 45K human-annotated propositions.&#10;We define the tasks of proposition-level segmentation and entailment. Given a hypothesis sentence and a premise document, a system is expected to segment the hypothesis into the set of propositions, and recognize whether each proposition can be inferred from the premise. &#10;&#10;Interestingly, we observe that existing notions of proposition adopted by Open Information Extraction (OpenIE) or Semantic Role Labeling (SRL) \cite{baker1998berkeley, kingsbury2002treebank, meyers2004nombank} &#10;often fail to account for the complete set of propositions in a sentence, partly due to the fact that predicates and arguments in different propositions do not necessarily follow the same granularity (\S\ref{sec:desiderata}). &#10;We therefore adopt a more flexible and unified way of representing a proposition as a subset of tokens from the input sentence, without explicitly annotating the semantic role or predicate-argument structure within the proposition, as illustrated in Table~\ref{tab:lead-example}.&#10;We discuss the motivation and design desiderata in \S\ref{sec:desiderata}.&#10;&#10;We construct PropSegmEnt by sampling clusters of topically-aligned documents, i.e.\ documents focusing on the same entity or event, from Wikipedia \cite{schuster2022stretching} and the news domains \cite{gu2020generating}.&#10;We train and instruct expert annotators to identify all propositions exhaustively in a document, and label the textual entailment relation of each proposition with respect to another document in the cluster, viewed as the premise.&#10;&#10;We discuss the modeling challenges, and establish strong baselines for the segmentation and entailment tasks. &#10;We demonstrate the utility of our dataset and models through downstream use case studies on summary hallucination detection \cite{maynez-etal-2020-faithfulness}, and DocNLI \cite{yin-etal-2021-docnli}, through which we show that recognizing and decomposing entailment relations at the proposition-level could provide fine-grained characterization and explanation for NLI-like tasks, especially with long and compositional hypotheses.&#10;&#10;In summary, the main contributions in our paper include: (1) Motivating the need to recognize textual entailment relation on proposition level; (2) Introducing the first large-scale dataset for studying proposition-level segmentation and entailment recognition; and (3) Leveraging PropSegmEnt to train Seq2Seq models as strong baselines for the tasks, and demonstrating their utility in document-level NLI and hallucination detection tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Proposition Segmentation, Linguistics, Cognitive Science, Textual Entailment Analysis, Artificial Intelligence, Natural Language Inference, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2106.09449" label="2106.09449">
        <attvalues>
          <attvalue for="0" value="DocNLI: A Large-scale Dataset for Document-level Natural Language&#10;  Inference" />
          <attvalue for="1" value="  Natural language inference (NLI) is formulated as a unified framework for&#10;solving various NLP problems such as relation extraction, question answering,&#10;summarization, etc. It has been studied intensively in the past few years&#10;thanks to the availability of large-scale labeled datasets. However, most&#10;existing studies focus on merely sentence-level inference, which limits the&#10;scope of NLI's application in downstream NLP problems. This work presents&#10;DocNLI -- a newly-constructed large-scale dataset for document-level NLI.&#10;DocNLI is transformed from a broad range of NLP problems and covers multiple&#10;genres of text. The premises always stay in the document granularity, whereas&#10;the hypotheses vary in length from single sentences to passages with hundreds&#10;of words. Additionally, DocNLI has pretty limited artifacts which unfortunately&#10;widely exist in some popular sentence-level NLI datasets. Our experiments&#10;demonstrate that, even without fine-tuning, a model pretrained on DocNLI shows&#10;promising performance on popular sentence-level benchmarks, and generalizes&#10;well to out-of-domain NLP tasks that rely on inference at document granularity.&#10;Task-specific fine-tuning can bring further improvements. Data, code, and&#10;pretrained models can be found at https://github.com/salesforce/DocNLI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.07447" label="2204.07447">
        <attvalues>
          <attvalue for="0" value="Stretching Sentence-pair NLI Models to Reason over Long Documents and&#10;  Clusters" />
          <attvalue for="1" value="  Natural Language Inference (NLI) has been extensively studied by the NLP&#10;community as a framework for estimating the semantic relation between sentence&#10;pairs. While early work identified certain biases in NLI models, recent&#10;advancements in modeling and datasets demonstrated promising performance. In&#10;this work, we further explore the direct zero-shot applicability of NLI models&#10;to real applications, beyond the sentence-pair setting they were trained on.&#10;First, we analyze the robustness of these models to longer and out-of-domain&#10;inputs. Then, we develop new aggregation methods to allow operating over full&#10;documents, reaching state-of-the-art performance on the ContractNLI dataset.&#10;Interestingly, we find NLI scores to provide strong retrieval signals, leading&#10;to more relevant evidence extractions compared to common similarity-based&#10;methods. Finally, we go further and investigate whole document clusters to&#10;identify both discrepancies and consensus among sources. In a test case, we&#10;find real inconsistencies between Wikipedia pages in different languages about&#10;the same topic.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2001.09386" label="2001.09386">
        <attvalues>
          <attvalue for="0" value="Generating Representative Headlines for News Stories" />
          <attvalue for="1" value="  Millions of news articles are published online every day, which can be&#10;overwhelming for readers to follow. Grouping articles that are reporting the&#10;same event into news stories is a common way of assisting readers in their news&#10;consumption. However, it remains a challenging research problem to efficiently&#10;and effectively generate a representative headline for each story. Automatic&#10;summarization of a document set has been studied for decades, while few studies&#10;have focused on generating representative headlines for a set of articles.&#10;Unlike summaries, which aim to capture most information with least redundancy,&#10;headlines aim to capture information jointly shared by the story articles in&#10;short length, and exclude information that is too specific to each individual&#10;article. In this work, we study the problem of generating representative&#10;headlines for news stories. We develop a distant supervision approach to train&#10;large-scale generation models without any human annotation. This approach&#10;centers on two technical components. First, we propose a multi-level&#10;pre-training framework that incorporates massive unlabeled corpus with&#10;different quality-vs.-quantity balance at different levels. We show that models&#10;trained within this framework outperform those trained with pure human curated&#10;corpus. Second, we propose a novel self-voting-based article attention layer to&#10;extract salient information shared by multiple articles. We show that models&#10;that incorporate this layer are robust to potential noises in news stories and&#10;outperform existing baselines with or without noises. We can further enhance&#10;our model by incorporating human labels, and we show our distant supervision&#10;approach significantly reduces the demand on labeled data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.04558" label="1611.04558">
        <attvalues>
          <attvalue for="0" value="Google's Multilingual Neural Machine Translation System: Enabling&#10;  Zero-Shot Translation" />
          <attvalue for="1" value="  We propose a simple solution to use a single Neural Machine Translation (NMT)&#10;model to translate between multiple languages. Our solution requires no change&#10;in the model architecture from our base system but instead introduces an&#10;artificial token at the beginning of the input sentence to specify the required&#10;target language. The rest of the model, which includes encoder, decoder and&#10;attention, remains unchanged and is shared across all languages. Using a shared&#10;wordpiece vocabulary, our approach enables Multilingual NMT using a single&#10;model without any increase in parameters, which is significantly simpler than&#10;previous proposals for Multilingual NMT. Our method often improves the&#10;translation quality of all involved language pairs, even while keeping the&#10;total number of model parameters constant. On the WMT'14 benchmarks, a single&#10;multilingual model achieves comparable performance for&#10;English$\rightarrow$French and surpasses state-of-the-art results for&#10;English$\rightarrow$German. Similarly, a single multilingual model surpasses&#10;state-of-the-art results for French$\rightarrow$English and&#10;German$\rightarrow$English on WMT'14 and WMT'15 benchmarks respectively. On&#10;production corpora, multilingual models of up to twelve language pairs allow&#10;for better translation of many individual pairs. In addition to improving the&#10;translation quality of language pairs that the model was trained with, our&#10;models can also learn to perform implicit bridging between language pairs never&#10;seen explicitly during training, showing that transfer learning and zero-shot&#10;translation is possible for neural translation. Finally, we show analyses that&#10;hints at a universal interlingua representation in our models and show some&#10;interesting examples when mixing languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.16626" label="2305.16626">
        <attvalues>
          <attvalue for="0" value="Evaluation of Question Generation Needs More References" />
          <attvalue for="1" value="  Question generation (QG) is the task of generating a valid and fluent&#10;question based on a given context and the target answer. According to various&#10;purposes, even given the same context, instructors can ask questions about&#10;different concepts, and even the same concept can be written in different ways.&#10;However, the evaluation for QG usually depends on single reference-based&#10;similarity metrics, such as n-gram-based metric or learned metric, which is not&#10;sufficient to fully evaluate the potential of QG methods. To this end, we&#10;propose to paraphrase the reference question for a more robust QG evaluation.&#10;Using large language models such as GPT-3, we created semantically and&#10;syntactically diverse questions, then adopt the simple aggregation of the&#10;popular evaluation metrics as the final scores. Through our experiments, we&#10;found that using multiple (pseudo) references is more effective for QG&#10;evaluation while showing a higher correlation with human evaluations than&#10;evaluation with a single reference.&#10;" />
          <attvalue for="2" value="&#10;&#10;Question generation (QG) is the task of generating questions that are relevant to and answerable by given text. &#10;Since QG can be applied in not only educational scenarios~\cite{kurdi2020systematic, steuer2021not, moon-etal-2022-evaluating} but also improving question-answering tasks~\cite{chen2021factuality,wang2018learning, yu2020based}, designing better QG frameworks and their automatic evaluations have gained more attention~\cite{ &#10;&#10;However, previous QG works mostly evaluate their methods based on how similar the generated questions are to the gold reference questions~\cite{chan-fan-2019-recurrent, DBLP:conf/nlpcc/ZhouYWTBZ17, du-cardie-2018-harvesting}, using n-gram-based similarity metrics, such as BLEU~\cite{papineni-etal-2002-bleu} and ROUGE~\cite{lin-2004-rouge}. &#10;Given a single reference, these metrics do not account for the lexical and semantic diversity of questions~\cite{DBLP:conf/iclr/ZhangKWWA20}, showing poor correlation with human judgment~\cite{liu-etal-2016-evaluate, novikova-etal-2017-need, chaganty-etal-2018-price}.&#10;Though prior works studied alternative metrics of leveraging language models, such as BERTScore~\cite{DBLP:conf/iclr/ZhangKWWA20} and BLEURT~\cite{sellam-etal-2020-bleurt}, such metrics are limited in that the diversity of gold questions is only implicitly represented in the embedding space, rather than data space (or, raw questions).&#10;&#10;To explicitly compare with the diverse gold questions in the data space, we propose to augment the single reference question for evaluating QG frameworks, which we call Multi-Reference Evaluation (MRE), by leveraging the few-shot ability of large language models (LLMs) like GPT-3~\cite{NEURIPS2020_1457c0d6} and ChatGPT~\cite{openai_2023}. &#10;Though there have been efforts to augment references for improving evaluations, they are either limited in other text generation tasks, such as machine translation~\cite{bawden-etal-2020-study} and question answering~\cite{liu-etal-2021-language}, or the methods are hard to be applied in question generation tasks, as naive LLMs generate some negative (toxic or erroneous) questions~\cite{10.1007/978-3-031-11644-5_13}.&#10;Therefore, we utilize LLMs for paraphrasing to augment a reference question, rather than generating new questions from the given context. &#10;To the best of our knowledge, we are the first to apply reference augmentation to evaluate the QG frameworks. We briefly summarize our main contributions as follows:&#10;&#10;\begin{itemize}&#10;\item We propose to augment the single reference for multiple reference evaluation (MRE) that can explicitly consider syntactic and semantic variations of questions.&#10;Experimental results on quiz design dataset~\cite{laban2022quiz} show that the performance of existing metrics can be considerably improved when MRE is applied.&#10;\item MRE is metric-agnostic, such that various metrics can be improved with our method. Since each existing metric can discover different insights, such as BLEU for lexical similarity and BERTScore for semantic similarity, MRE can improve these multiple various lenses for investigating QG frameworks. &#10;\item We release the augmented reference questions as supplementary materials, which provide an opportunity to reproduce our results for further research. We further validated whether the augmented references are correct or not by human annotators. &#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Question Generation, Computer Science, Linguistics, Computational Linguistics, Language Evaluation Metrics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1805.04843" label="1805.04843">
        <attvalues>
          <attvalue for="0" value="Learning to Ask Questions in Open-domain Conversational Systems with&#10;  Typed Decoders" />
          <attvalue for="1" value="  Asking good questions in large-scale, open-domain conversational systems is&#10;quite significant yet rather untouched. This task, substantially different from&#10;traditional question generation, requires to question not only with various&#10;patterns but also on diverse and relevant topics. We observe that a good&#10;question is a natural composition of {\it interrogatives}, {\it topic words},&#10;and {\it ordinary words}. Interrogatives lexicalize the pattern of questioning,&#10;topic words address the key information for topic transition in dialogue, and&#10;ordinary words play syntactical and grammatical roles in making a natural&#10;sentence. We devise two typed decoders (\textit{soft typed decoder} and&#10;\textit{hard typed decoder}) in which a type distribution over the three types&#10;is estimated and used to modulate the final generation distribution. Extensive&#10;experiments show that the typed decoders outperform state-of-the-art baselines&#10;and can generate more meaningful questions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14989" label="2004.14989">
        <attvalues>
          <attvalue for="0" value="A Study in Improving BLEU Reference Coverage with Diverse Automatic&#10;  Paraphrasing" />
          <attvalue for="1" value="  We investigate a long-perceived shortcoming in the typical use of BLEU: its&#10;reliance on a single reference. Using modern neural paraphrasing techniques, we&#10;study whether automatically generating additional diverse references can&#10;provide better coverage of the space of valid translations and thereby improve&#10;its correlation with human judgments. Our experiments on the into-English&#10;language directions of the WMT19 metrics task (at both the system and sentence&#10;level) show that using paraphrased references does generally improve BLEU, and&#10;when it does, the more diverse the better. However, we also show that better&#10;results could be achieved if those paraphrases were to specifically target the&#10;parts of the space most relevant to the MT outputs being evaluated. Moreover,&#10;the gains remain slight even when human paraphrases are used, suggesting&#10;inherent limitations to BLEU's capacity to correctly exploit multiple&#10;references. Surprisingly, we also find that adequacy appears to be less&#10;important, as shown by the high results of a strong sampling approach, which&#10;even beats human paraphrases when used with sentence-level BLEU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.01730" label="2205.01730">
        <attvalues>
          <attvalue for="0" value="Quiz Design Task: Helping Teachers Create Quizzes with Automated&#10;  Question Generation" />
          <attvalue for="1" value="  Question generation (QGen) models are often evaluated with standardized NLG&#10;metrics that are based on n-gram overlap. In this paper, we measure whether&#10;these metric improvements translate to gains in a practical setting, focusing&#10;on the use case of helping teachers automate the generation of reading&#10;comprehension quizzes. In our study, teachers building a quiz receive question&#10;suggestions, which they can either accept or refuse with a reason. Even though&#10;we find that recent progress in QGen leads to a significant increase in&#10;question acceptance rates, there is still large room for improvement, with the&#10;best model having only 68.4% of its questions accepted by the ten teachers who&#10;participated in our study. We then leverage the annotations we collected to&#10;analyze standard NLG metrics and find that model performance has reached&#10;projected upper-bounds, suggesting new automatic metrics are needed to guide&#10;QGen research forward.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.14956" label="2406.14956">
        <attvalues>
          <attvalue for="0" value="Unlocking the Global Synergies in Low-Rank Adapters" />
          <attvalue for="1" value="  Low-rank Adaption (LoRA) has been the de-facto parameter-efficient&#10;fine-tuning technique for large language models. We present HeteroLoRA, a&#10;light-weight search algorithm that leverages zero-cost proxies to allocate the&#10;limited LoRA trainable parameters across the model for better fine-tuned&#10;performance. In addition to the allocation for the standard LoRA-adapted&#10;models, we also demonstrate the efficacy of HeteroLoRA by performing the&#10;allocation in a more challenging search space that includes LoRA modules and&#10;LoRA-adapted shortcut connections. Experiments show that HeteroLoRA enables&#10;improvements in model performance given the same parameter budge. For example,&#10;on MRPC, we see an improvement of 1.6% in accuracy with similar training&#10;parameter budget. We will open-source our algorithm once the paper is accepted.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, large language models (LLMs) have shown impressive performance in a range of natural language processing tasks~\cite{llmsurvey}. However, fine-tuning pre-trained language models (PLMs) is computationally and memory-intensive.&#10;To mitigate this, parameter-efficient tuning (PET) methods have been developed to fine-tune a small number of (extra) model parameters instead of the entire model~\cite{serialadapter}.&#10;&#10;Low-rank adaptation (LoRA) \cite{lora} is now the de-facto PET method. LoRA injects two low-rank matrices $A\in \mathbb{R}^{r\times d_in}$ and $B\in \mathbb{R}^{d_out\times r}$ with rank $r\ll\min(d_in, d_out)$, to update the pre-trained weights $W \in \mathbb{R}^{d_out\times d_in}$.&#10;Unlike full fine-tuning, LoRA updates only the injected $A$ and $B$ with the pre-trained weights $W$ unchanged. After fine-tuning, the update weights $\Delta W = BA$ fuse back to the pre-trained weights $W'=W+AB$, incurring no additional latency. LoRA achieves performance levels similar to full fine-tuning while drastically reducing memory usage.&#10;We identify the following limitations of LoRA.&#10;\begin{itemize}&#10; \item Existing methods configure LoRA modules within a model uniformly with the same rank $r$, thus each LoRA module consumes an identical number of trainable parameters, regardless of its potentially varying contributions to the overall model performance. \item Current LoRA implementations predominantly adhere to the Transformer architecture. However, there has been limited exploration into extending the model architecture to enhance performance. This leads to the broader question of whether it is necessary to incorporate LoRA modules under these constraints and whether LoRA modules would be more effective with specific new connections, such as shortcut connections \cite{resnet, densenet}.&#10;\end{itemize}&#10;&#10;In this work, we introduce HeteroLoRA, a new lightweight framework designed to autonomously allocate the LoRA module across the entire LLM given a parameter budget. Furthermore, we perform HeteroLoRA within an expanded search space including LoRA-adapted shortcut connections \cite{resnet} as illustrated in \ref{fig: heterolora}. &#10;&#10;Specifically, we make the following contributions:&#10;&#10;\begin{itemize}&#10; \item We propose HeteroLoRA, a novel LoRA configuration search algorithm to solve the rank allocation problem within a limited trainable parameter budget. HeteroLoRA leverages zero-cost proxies~\cite{zerocostproxy} to avoid the high cost of brute-force search. &#10; \item We further prove the efficacy of the LoRA-adapted shortcut connection and combine it with HeteroLoRA to improve global synergies. The shortcuts suggested by HeteroLoRA enable more gains in model performance given the same parameter budget. For instance, on MRPC, we see an improvement of 1.6\% in accuracy with similar model size budgets.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Parameter Efficient Tuning, Deep Learning Algorithms, Artificial Intelligence, Mathematics, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2101.08134" label="2101.08134">
        <attvalues>
          <attvalue for="0" value="Zero-Cost Proxies for Lightweight NAS" />
          <attvalue for="1" value="  Neural Architecture Search (NAS) is quickly becoming the standard methodology&#10;to design neural network models. However, NAS is typically compute-intensive&#10;because multiple models need to be evaluated before choosing the best one. To&#10;reduce the computational power and time needed, a proxy task is often used for&#10;evaluating each model instead of full training. In this paper, we evaluate&#10;conventional reduced-training proxies and quantify how well they preserve&#10;ranking between multiple models during search when compared with the rankings&#10;produced by final trained accuracy. We propose a series of zero-cost proxies,&#10;based on recent pruning literature, that use just a single minibatch of&#10;training data to compute a model's score. Our zero-cost proxies use 3 orders of&#10;magnitude less computation but can match and even outperform conventional&#10;proxies. For example, Spearman's rank correlation coefficient between final&#10;validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82,&#10;compared to 0.61 for EcoNAS (a recently proposed reduced-training proxy).&#10;Finally, we use these zero-cost proxies to enhance existing NAS search&#10;algorithms such as random search, reinforcement learning, evolutionary search&#10;and predictor-based search. For all search methodologies and across three&#10;different NAS datasets, we are able to significantly improve sample efficiency,&#10;and thereby decrease computation, by using our zero-cost proxies. For example&#10;on NAS-Bench-101, we achieved the same accuracy 4$\times$ quicker than the best&#10;previous result. Our code is made public at:&#10;https://github.com/mohsaied/zero-cost-nas.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.17217" label="2203.17217">
        <attvalues>
          <attvalue for="0" value="On the probability-quality paradox in language generation" />
          <attvalue for="1" value="  When generating natural language from neural probabilistic models, high&#10;probability does not always coincide with high quality: It has often been&#10;observed that mode-seeking decoding methods, i.e., those that produce&#10;high-probability text under the model, lead to unnatural language. On the other&#10;hand, the lower-probability text generated by stochastic methods is perceived&#10;as more human-like. In this note, we offer an explanation for this phenomenon&#10;by analyzing language generation through an information-theoretic lens.&#10;Specifically, we posit that human-like language should contain an amount of&#10;information (quantified as negative log-probability) that is close to the&#10;entropy of the distribution over natural strings. Further, we posit that&#10;language with substantially more (or less) information is undesirable. We&#10;provide preliminary empirical evidence in favor of this hypothesis; quality&#10;ratings of both human and machine-generated text -- covering multiple tasks and&#10;common decoding strategies -- suggest high-quality text has an information&#10;content significantly closer to the entropy than we would expect by chance.&#10;" />
          <attvalue for="2" value="&#10;Despite recent advances in language generation, it remains largely unclear what quantitative attributes desirable text has. &#10;At least intuitively, under a well-calibrated model, high probability strings should be both coherent and human-like. Yet, many authors have noticed that maximum-likelihood decoding (or an approximation thereto) yields incredibly unnatural text \cite{pmlr-v97-cohen19a,stahlberg_nmt_2019,eikema_is_2020}\clara{citation from 2021?}. Rather, the relationship between probability and quality appears to have an inflection point \cite{zhang_trading_2020}, i.e., quality and probability are positively correlated only up until a certain point, after which the relationship becomes negative. &#10;Prior work has framed this phenomenon as a diversity vs. quality trade-off, claiming the need for this balance is why stochastic decoding methods---such as nucleus or top-$k$ sampling---outperform maximization-based approaches for tasks like dialogue or story generation\clara{citations}. In this note, we offer a different explanation for this observed phenomenon under an information-theoretic lens. &#10;&#10;The study of natural language as a communication channel---in the formal, mathematical sense of information theory \cite{shannon1948mathematical}---has helped us understand much about properties of natural language\clara{citations}. &#10;From a given channel, information theory tells us that we should (on average) expect certain types of codes, namely those that convey an average amount of information---where information is defined as negative log-probability.\clara{I feel like this probably belongs in the main text but I can't figure out how to incorporate it seamlessly} Notably, under this definition, the highest probability codes, e.g. the mode(s), are not considered representative of the channel: they do not convey much information.\looseness=-1&#10;&#10;This concept leads us to the \defn{Expected Information Hypothesis}: Assuming that we, as humans, use language to communicate information efficiently---by definition, around the average information rate of the theoretical channel---there is a certain amount of information that we a priori expect text to convey. Thus, we do not perceive text that is particularly uninformative---or conversely, very surprising\tiago{I would say informative (I know they are &quot;synonyms&quot;, but not for everyone).\response{clara}{hmmm I feel like informative doesnt quite capture what feeling we're trying to elicit: like 'wtf is this string'}\response{tiago}{Maybe then should we have a footnote explaining what surprising / surprisal means here?}}---as human-like; that's not how we typically use language!&#10;Such a hypothesis offers intuition for the trends seen in natural language generation, i.e., why desirable text seems to exist not at the high end of the probability spectrum but around a certain inflection point, even in state-of-the-art language models. &#10;Indeed, it would largely account for the degeneracy of both high- and low-probability generates, e.g. those observed by \newcite{holtzman_curious_2020}, since they fall outside of our subconscious expectations for natural language text.\looseness=-1&#10;&#10;To confirm these speculations empirically, we perform an analysis of both human and model-generated text. As with prior work, we observe an inflection point in the probability vs. quality relationship: we link this inflection point to the expected information content, i.e., the entropy, of our probabilistic model. We observe that text ranked highly by humans typically falls within a range around model entropy. Further, decoding methods known to perform well in natural language generation appear to concentrate their mass around this value. We take these observations as empirical evidence for our explanation of why high probability text is not always what we want.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Neural Probabilistic Models, Language Generation" />
        </attvalues>
      </node>
      <node id="2005.10283" label="2005.10283">
        <attvalues>
          <attvalue for="0" value="Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural&#10;  Machine Translation" />
          <attvalue for="1" value="  Recent studies have revealed a number of pathologies of neural machine&#10;translation (NMT) systems. Hypotheses explaining these mostly suggest there is&#10;something fundamentally wrong with NMT as a model or its training algorithm,&#10;maximum likelihood estimation (MLE). Most of this evidence was gathered using&#10;maximum a posteriori (MAP) decoding, a decision rule aimed at identifying the&#10;highest-scoring translation, i.e. the mode. We argue that the evidence&#10;corroborates the inadequacy of MAP decoding more than casts doubt on the model&#10;and its training algorithm. In this work, we show that translation&#10;distributions do reproduce various statistics of the data well, but that beam&#10;search strays from such statistics. We show that some of the known pathologies&#10;and biases of NMT are due to MAP decoding and not to NMT's statistical&#10;assumptions nor MLE. In particular, we show that the most likely translations&#10;under the model accumulate so little probability mass that the mode can be&#10;considered essentially arbitrary. We therefore advocate for the use of decision&#10;rules that take into account the translation distribution holistically. We show&#10;that an approximation to minimum Bayes risk decoding gives competitive results&#10;confirming that NMT models do capture important aspects of translation well in&#10;expectation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Some of our observations have been made in previous work. \newcite{ott-etal-2018-analyzing} observe that unigram statistics of beam search stray from those of the data, while random samples do a better job at reproducing them. &#10;\newcite{holtzman-etal-2019-the} find that beam search outputs have disproportionately high token probabilities compared to natural language under a sequence to sequence model.&#10;Our analysis is more extensive, we include richer statistics about the data, more language pairs, and vary the amount of training resources, leading to new insights about MLE-trained NMT and the merits of mode-seeking predictions.&#10;&#10;\newcite{ott-etal-2018-analyzing} also observe that NMT learns flat distributions, they analyse a high-resource English-French system trained on $35.5$ million sentence pairs from WMT'14 and find that after drawing $10,000$ samples from the WMT'14 validation set less than $25\%$ of the probability space has been explored.&#10;Our analysis shows that even though NMT distributions do not reveal clear winners, they do emphasise translations that share statistics with the reference, which motivates us to look into MBR.&#10;&#10;MBR decoding is old news in machine translation \cite{kumar-byrne-2004-minimum,tromble-etal-2008-lattice}, but it has received little attention in NMT. Previous approximations to MBR in NMT employ beam search to define the support and to evaluate expected utility (with probabilities renormalised to sum to $1$ in the beam), these studies report the need for very large beams \cite{stahlberg-etal-2017-neural,blain2017exploring,shu-nakayama-2017-later}.&#10;They claim the inability to directly score better translations higher is a deficiency of the model scoring function. &#10;We argue this is another piece of evidence for the inadequacy of the mode: by using beam search, they emphasise statistics of high-scoring translations, potentially rare and inadequate ones.&#10;Very recently, \newcite{borgeaud-emerson-2020-leveraging} present a voting-theory perspective on decoding for image captioning and machine translation. Their proposal is closely-related to MBR, but motivated differently. Their decision rule too is guided by beam search, which may emphasise pathologies of highest-likelihood paths, but they also propose and investigate stronger utility functions which lead to improvements \wrt length, diversity, and human judgements. &#10;&#10;The only instance that we are aware of where unbiased samples from an NMT model support a decision rule is the concurrent work by \newcite{naskar2020energybased}. The authors make the same observation that we make in Section~\ref{sec:automatic-eval}, namely that an oracle selection from a small set of samples of an NMT model shows great potential, greatly outperforming beam search. Motivated by this observation, the authors propose a re-ranking model that learns to rank sampled translations according to their oracle BLEU. Using the trained model to re-rank a set of 100 samples from the NMT model they find strong improvements over beam search of up to 3 BLEU points, again showing the potential of sampling-based decision rules.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2004.10450" label="2004.10450">
        <attvalues>
          <attvalue for="0" value="Trading Off Diversity and Quality in Natural Language Generation" />
          <attvalue for="1" value="  For open-ended language generation tasks such as storytelling and dialogue,&#10;choosing the right decoding algorithm is critical to controlling the tradeoff&#10;between generation quality and diversity. However, there presently exists no&#10;consensus on which decoding procedure is best or even the criteria by which to&#10;compare them. We address these issues by casting decoding as a multi-objective&#10;optimization problem aiming to simultaneously maximize both response quality&#10;and diversity. Our framework enables us to perform the first large-scale&#10;evaluation of decoding methods along the entire quality-diversity spectrum. We&#10;find that when diversity is a priority, all methods perform similarly, but when&#10;quality is viewed as more important, the recently proposed nucleus sampling&#10;(Holtzman et al. 2019) outperforms all other evaluated decoding algorithms. Our&#10;experiments also confirm the existence of the `likelihood trap', the&#10;counter-intuitive observation that high likelihood sequences are often&#10;surprisingly low quality. We leverage our findings to create and evaluate an&#10;algorithm called \emph{selective sampling} which tractably approximates&#10;globally-normalized temperature sampling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.03759" label="1909.03759">
        <attvalues>
          <attvalue for="0" value="Neural Conversational QA: Learning to Reason v.s. Exploiting Patterns" />
          <attvalue for="1" value="  Neural Conversational QA tasks like ShARC require systems to answer questions&#10;based on the contents of a given passage. On studying recent state-of-the-art&#10;models on the ShARCQA task, we found indications that the models learn spurious&#10;clues/patterns in the dataset. Furthermore, we show that a heuristic-based&#10;program designed to exploit these patterns can have performance comparable to&#10;that of the neural models. In this paper we share our findings about four types&#10;of patterns found in the ShARC corpus and describe how neural models exploit&#10;them. Motivated by the aforementioned findings, we create and share a modified&#10;dataset that has fewer spurious patterns, consequently allowing models to learn&#10;better.&#10;" />
          <attvalue for="2" value=" &#10;&#10;ShARC, a conversational QA task \cite{ShARC}, requires a system to answer user questions based on rules expressed in natural language text. An example in Figure~\ref{fig:cqa} shows a user sharing some background information (referred to as scenario) and asking a question about continuing to pay for `UK National Insurance'. The rule text associated with this dialog exchange defines the policy that guides the conversation flow. At any turn in the conversation, a system may choose to respond with a final Yes/No answer; ask a follow-up question to obtain more information from the user; or reply that the question is irrelevant to the context.&#10;&#10;Several deep learning models such as BERT-QA \cite{BERT}, E3 \cite{E3}, and BiSon \cite{BiSon} perform reasonably well on this task. However, our exploration of the ShARC dataset indicates that there are multiple spurious patterns that could be exploited by neural models. We observe that the performance of the models mentioned above drops when they are tested on a perturbed dataset, suggesting that the underlying neural models do not generalize and are rather over-sensitive \cite{welbl2020undersensitivity} to minor textual perturbations. By sensitivity we refer to a model's ability to generalize itself but not over-fit, while still being invariant to perturbations or text transformations \cite{teney2020value, szegedy2013intriguing}. Our observations about conversational QA models designed for ShARC learning spurious statistical clues are in line with those reported by \cite{ProbingNN}. To the best of our knowledge, we are the first to demonstrate this problem in conversational QA.&#10;&#10;{Patterns in the ShARC dataset:}&#10;We discover four types of patterns in the ShARC dataset: (1) correlation between the last answer to a follow-up question and the predicted answer to a user question; (2) a high correlation between asking a new follow-up question and the number of turns in the dialog history; (3) correlation between the sequence of follow-up questions in the dialog history and the sequence of rule clauses in the rule-text; and (4) correlation between an empty history/scenario and the answer being irrelevant. &#10;&#10;{Contributions:} &#10;The main contributions of this work are as follows: (1) We present a simple heuristics driven program designed to exploit the aforementioned patterns which has performance comparable to the state-of-the-art models. (2) The performance of the state-of-the-art models drops when they are tested on a {perturbed} test set that has these patterns diluted or removed. (3) We also identify a weakness in the current evaluation procedure, and propose an improved evaluation criteria which penalizes BLEU scores if a follow-up question is not generated when it should be. We refer to this criteria as BLEU-P (BLEU penalized) in the rest of the paper. (4) We generate a new dataset which reduces the patterns identified in the original dataset, and re-benchmark existing state-of-the-art models published on the leaderboard. We find that the models learn better on this dataset and their performance is consistent across the original and the perturbed dev sets. Our dataset and all accompanying scripts are available at.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Conversational QA Systems, Computer Science, Neural Model Limitations, Linguistics, Cognitive Science, Dataset Modification, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1809.01494" label="1809.01494">
        <attvalues>
          <attvalue for="0" value="Interpretation of Natural Language Rules in Conversational Machine&#10;  Reading" />
          <attvalue for="1" value="  Most work in machine reading focuses on question answering problems where the&#10;answer is directly expressed in the text to read. However, many real-world&#10;question answering problems require the reading of text not because it contains&#10;the literal answer, but because it contains a recipe to derive an answer&#10;together with the reader's background knowledge. One example is the task of&#10;interpreting regulations to answer &quot;Can I...?&quot; or &quot;Do I have to...?&quot; questions&#10;such as &quot;I am working in Canada. Do I have to carry on paying UK National&#10;Insurance?&quot; after reading a UK government website about this topic. This task&#10;requires both the interpretation of rules and the application of background&#10;knowledge. It is further complicated due to the fact that, in practice, most&#10;questions are underspecified, and a human assistant will regularly have to ask&#10;clarification questions such as &quot;How long have you been working abroad?&quot; when&#10;the answer cannot be directly derived from the question and text. In this&#10;paper, we formalise this task and develop a crowd-sourcing strategy to collect&#10;32k task instances based on real-world rules and crowd-generated questions and&#10;scenarios. We analyse the challenges of this task and assess its difficulty by&#10;evaluating the performance of rule-based and machine-learning baselines. We&#10;observe promising results when no background knowledge is necessary, and&#10;substantial room for improvement whenever background knowledge is needed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.05915" label="1908.05915">
        <attvalues>
          <attvalue for="0" value="Attending to Future Tokens For Bidirectional Sequence Generation" />
          <attvalue for="1" value="  Neural sequence generation is typically performed token-by-token and&#10;left-to-right. Whenever a token is generated only previously produced tokens&#10;are taken into consideration. In contrast, for problems such as sequence&#10;classification, bidirectional attention, which takes both past and future&#10;tokens into consideration, has been shown to perform much better. We propose to&#10;make the sequence generation process bidirectional by employing special&#10;placeholder tokens. Treated as a node in a fully connected graph, a placeholder&#10;token can take past and future tokens into consideration when generating the&#10;actual output token. We verify the effectiveness of our approach experimentally&#10;on two conversational tasks where the proposed bidirectional model outperforms&#10;competitive baselines by a large margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.04808" label="2003.04808">
        <attvalues>
          <attvalue for="0" value="Undersensitivity in Neural Reading Comprehension" />
          <attvalue for="1" value="  Current reading comprehension models generalise well to in-distribution test&#10;sets, yet perform poorly on adversarially selected inputs. Most prior work on&#10;adversarial inputs studies oversensitivity: semantically invariant text&#10;perturbations that cause a model's prediction to change when it should not. In&#10;this work we focus on the complementary problem: excessive prediction&#10;undersensitivity, where input text is meaningfully changed but the model's&#10;prediction does not, even though it should. We formulate a noisy adversarial&#10;attack which searches among semantic variations of the question for which a&#10;model erroneously predicts the same answer, and with even higher probability.&#10;Despite comprising unanswerable questions, both SQuAD2.0 and NewsQA models are&#10;vulnerable to this attack. This indicates that although accurate, models tend&#10;to rely on spurious patterns and do not fully consider the information&#10;specified in a question. We experiment with data augmentation and adversarial&#10;training as defences, and find that both substantially decrease vulnerability&#10;to attacks on held out data, as well as held out attack spaces. Addressing&#10;undersensitivity also improves results on AddSent and AddOneSent, and models&#10;furthermore generalise better when facing train/evaluation distribution&#10;mismatch: they are less prone to overly rely on predictive cues present only in&#10;the training set, and outperform a conventional model by as much as 10.9% F1.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.09241" label="2005.09241">
        <attvalues>
          <attvalue for="0" value="On the Value of Out-of-Distribution Testing: An Example of Goodhart's&#10;  Law" />
          <attvalue for="1" value="  Out-of-distribution (OOD) testing is increasingly popular for evaluating a&#10;machine learning system's ability to generalize beyond the biases of a training&#10;set. OOD benchmarks are designed to present a different joint distribution of&#10;data and labels between training and test time. VQA-CP has become the standard&#10;OOD benchmark for visual question answering, but we discovered three troubling&#10;practices in its current use. First, most published methods rely on explicit&#10;knowledge of the construction of the OOD splits. They often rely on&#10;``inverting'' the distribution of labels, e.g. answering mostly 'yes' when the&#10;common training answer is 'no'. Second, the OOD test set is used for model&#10;selection. Third, a model's in-domain performance is assessed after retraining&#10;it on in-domain splits (VQA v2) that exhibit a more balanced distribution of&#10;labels. These three practices defeat the objective of evaluating&#10;generalization, and put into question the value of methods specifically&#10;designed for this dataset. We show that embarrassingly-simple methods,&#10;including one that generates answers at random, surpass the state of the art on&#10;some question types. We provide short- and long-term solutions to avoid these&#10;pitfalls and realize the benefits of OOD evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.07402" label="2203.07402">
        <attvalues>
          <attvalue for="0" value="Revisiting the Compositional Generalization Abilities of Neural Sequence&#10;  Models" />
          <attvalue for="1" value="  Compositional generalization is a fundamental trait in humans, allowing us to&#10;effortlessly combine known phrases to form novel sentences. Recent works have&#10;claimed that standard seq-to-seq models severely lack the ability to&#10;compositionally generalize. In this paper, we focus on one-shot primitive&#10;generalization as introduced by the popular SCAN benchmark. We demonstrate that&#10;modifying the training distribution in simple and intuitive ways enables&#10;standard seq-to-seq models to achieve near-perfect generalization performance,&#10;thereby showing that their compositional generalization abilities were&#10;previously underestimated. We perform detailed empirical analysis of this&#10;phenomenon. Our results indicate that the generalization performance of models&#10;is highly sensitive to the characteristics of the training data which should be&#10;carefully considered while designing such benchmarks in future.&#10;" />
          <attvalue for="2" value="&#10;&#10;According to the principle of compositionality, the meaning of a complex expression (e.g., a sentence) is determined by the meaning of its individual constituents and how they are combined. Humans can effectively recombine known parts to form new sentences that they have never encountered before. Despite the unprecedented achievements of standard seq-to-seq networks such as LSTMs and Transformers in NLP tasks, previous work has suggested that they are severely limited in their ability to generalize compositionally \cite{scan, cp_in_sp}. &#10;&#10;Problem Statement. Our work relates to a central challenge posed by compositional generalization datasets such as SCAN \cite{scan} and Colors \cite{colors}, which we refer to as one-shot primitive generalization: &#10;The dataset consists of input-output sentence pairs (e.g. `walk twice $arrow$ WALK WALK'); input sentences are formed from primitive words (`walk') and function words (`twice') and are generated by a context-free grammar (CFG); output sentences are obtained by applying an interpretation function. Crucially, there is a systematic difference between the train and test splits: While the former has a single example of an isolated primitive (e.g., the primitive definition `jump $arrow$ JUMP' in SCAN), the latter consists of compositional sentences with this isolated primitive (e.g. `jump twice $arrow$ JUMP JUMP'). See Fig. \ref{fig:overview} (left) for an overview of the task. &#10;&#10;A model with the right inductive bias should generalize on the test data after having seen compositional expressions with other primitives during training. The need for such inductive bias is justified via psychological experiments \cite{colors} indicating that humans do have the ability to generalize on such tasks. Previous works have suggested that seq-to-seq models lack the appropriate inductive bias necessary to generalize on this task since they achieve near-zero accuracies on both SCAN and Colors benchmarks. This has led to the development of many specialized architectures \cite{prim_subs, permutation, ness, lexicon}, learning procedures \cite{lake_meta, titov_meta} and data augmentation methods \cite{geca, kim-rush-aug} to solve the task.&#10;&#10;Contributions. The primary claim of our paper is that, contrary to prior belief, neural sequence models such as Transformers and RNNs do have an inductive bias to generalize compositionally which can be enabled using the right supervision. (i) We show that by making simple and intuitive changes to the training data distribution, standard seq-to-seq models can achieve high generalization performance even with a training set of size less than 20\% of the original training set. In particular, if we incorporated examples with more novel primitives in the training set without necessarily increasing the size of the training set (see right part of Fig. \ref{fig:overview}), then the generalization performance of standard seq-to-seq models improves and reaches near-perfect score after a certain point. Our results also exemplify the importance of the training distribution apart from architectural changes and demonstrate that providing the right supervision can significantly improve the generalization abilities of the models.&#10;(ii) We investigate the potential cause behind the improvement in generalization performance and observe that the embedding of the isolated primitive becomes more similar to other primitives when the training set has higher number of primitives and their use cases.&#10;(iii) To understand the phenomenon better, we characterize the effect of different training distributions and model capacities. Our results show that the parameters of the experimental setting play a crucial role while evaluating the generalization abilities of models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Seq-to-Seq Models, Compositional Generalization, Artificial Intelligence, Training Data Design, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2205.06910" label="2205.06910">
        <attvalues>
          <attvalue for="0" value="A Property Induction Framework for Neural Language Models" />
          <attvalue for="1" value="  To what extent can experience from language contribute to our conceptual&#10;knowledge? Computational explorations of this question have shed light on the&#10;ability of powerful neural language models (LMs) -- informed solely through&#10;text input -- to encode and elicit information about concepts and properties.&#10;To extend this line of research, we present a framework that uses&#10;neural-network language models (LMs) to perform property induction -- a task in&#10;which humans generalize novel property knowledge (has sesamoid bones) from one&#10;or more concepts (robins) to others (sparrows, canaries). Patterns of property&#10;induction observed in humans have shed considerable light on the nature and&#10;organization of human conceptual knowledge. Inspired by this insight, we use&#10;our framework to explore the property inductions of LMs, and find that they&#10;show an inductive preference to generalize novel properties on the basis of&#10;category membership, suggesting the presence of a taxonomic bias in their&#10;representations.&#10;" />
          <attvalue for="2" value="&#10;&#10;There has recently been a growing interest in exploring the limits and potential of language as an environment for learning conceptual knowledge \cite{elman2004alternative, lupyan2019words}---knowledge that encompasses mental representations of everyday objects/events, and their properties and relations, that together inform our intuitive understanding of the world \cite{murphy2004big, machery2009doing}.&#10;Computational explorations of this claim often study the extent to which models that learn semantic representations through text alone can capture conceptual knowledge \cite{lucy-gauthier-2017-distributional, forbes2019neural, da-kasai-2019-cracking, bhatia2020transformer}.&#10;&#10;A hallmark feature of the conceptual knowledge acquired by humans is its capacity to facilitate inductive generalizations: inferences that go beyond available data to project novel information about concepts and properties \cite{osherson1990category, chater2011inductive, hayes2018inductive}.&#10;For example, our knowledge of taxonomic specificity is reflected when we generalize a novel property of a concept (e.g., robins have T9 hormones) more strongly to taxonomically close concepts (sparrows have T9 hormones) than to more taxonomically distant concepts (tigers have T9 hormones).&#10;Inductive generalizations about novel properties (also called property induction) therefore provide a context within which we can explore the nature of agents' understanding of conceptual knowledge.&#10;In this paper, we develop an analysis framework that uses neural network-based language models (LMs, henceforth) to perform property induction and use this framework to study concept representation in these models.&#10;Our framework consists of two stages. &#10;In the first stage, we train LMs to evaluate the truth of sentences expressing property knowledge (e.g., a cat has fur $arrow$ True, a table has fur $arrow$ False). &#10;In the second stage, we use these property-judgment models to test how the representations from the underlying LMs drive inductive generalization of novel properties---e.g., has feps, can dax, etc.&#10;&#10;Each stage of our framework sheds light on different aspects of the conceptual knowledge captured by LMs.&#10;Using the first stage, we test the extent to which LMs support judgement of whether a property applies to a concept, even when that property has not been seen in task-specific fine-tuning. &#10;We find that LMs perform substantially above chance, consistent with the conclusion that they are able to rely on generalizable property knowledge to assess truth of concept-property associations.&#10;In the second stage, we use this property judgment framework to study how knowledge representation in the base LMs drives inductive generalization with respect to entirely novel properties. &#10;We focus specifically on whether models' inductive preferences indicate reliance on taxonomic information, by testing whether models prefer to generalize within rather than outside of taxonomic categories. &#10;To do this, we teach our property-judgment models novel property information such as robins can dax via standard backpropagation methods and then test the extent to which they prefer generalizing this novel property to other birds (e.g. sparrows can dax) more strongly than to non-birds (e.g. zebras can dax).&#10;We find that models indeed show a preference for projecting new property knowledge on the basis of taxonomic category membership, suggesting that the models have acquired and represented taxonomic features on which they rely to project novel information.&#10;&#10;Our LM-based account of property induction contributes to the field in four primary ways. &#10;On the basis of the goals of the task, our framework focuses on reasoning where conclusions do not deductively follow from the premise, unlike the goals of the more commonly-used task of natural language inference \cite{bowman2015large}, and it therefore allows for testing of human-like inferences that are seldom studied in LMs \cite[cf.][]{bhagavatula2019abductive}.&#10;Next, as we show below, our framework opens a new window into exploring how large neural network models of language generalize beyond their training experience, complementing inquiries of models' inductive bias with respect to syntactic structure \cite{mccoy2020does} and ``universal linguistic constraints'' \cite{mccoy2020universal}. Additionally, this work advances research aiming to diagnose the nature and extent of conceptual knowledge in LMs \cite{da-kasai-2019-cracking,forbes2019neural,weir2020probing, bhatia2020transformer} by additionally focusing on how knowledge present in LM representations drives the generalizations they make. &#10;Finally, at a high level, our framework contributes to a range of works that have applied connectionist models to the problem of property induction \cite[see][]{sloman1993feature, rogers2004semantic, saxe2019mathematical}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Cognitive Science, Linguistics, Language Models, Conceptual Knowledge, Artificial Intelligence, Property Induction" />
        </attvalues>
      </node>
      <node id="1705.11168" label="1705.11168">
        <attvalues>
          <attvalue for="0" value="Are distributional representations ready for the real world? Evaluating&#10;  word vectors for grounded perceptual meaning" />
          <attvalue for="1" value="  Distributional word representation methods exploit word co-occurrences to&#10;build compact vector encodings of words. While these representations enjoy&#10;widespread use in modern natural language processing, it is unclear whether&#10;they accurately encode all necessary facets of conceptual meaning. In this&#10;paper, we evaluate how well these representations can predict perceptual and&#10;conceptual features of concrete concepts, drawing on two semantic norm datasets&#10;sourced from human participants. We find that several standard word&#10;representations fail to encode many salient perceptual features of concepts,&#10;and show that these deficits correlate with word-word similarity prediction&#10;errors. Our analyses provide motivation for grounded and embodied language&#10;learning approaches, which may help to remedy these deficits.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.02899" label="1908.02899">
        <attvalues>
          <attvalue for="0" value="Do Neural Language Representations Learn Physical Commonsense?" />
          <attvalue for="1" value="  Humans understand language based on the rich background knowledge about how&#10;the physical world works, which in turn allows us to reason about the physical&#10;world through language. In addition to the properties of objects (e.g., boats&#10;require fuel) and their affordances, i.e., the actions that are applicable to&#10;them (e.g., boats can be driven), we can also reason about if-then inferences&#10;between what properties of objects imply the kind of actions that are&#10;applicable to them (e.g., that if we can drive something then it likely&#10;requires fuel).&#10;  In this paper, we investigate the extent to which state-of-the-art neural&#10;language representations, trained on a vast amount of natural language text,&#10;demonstrate physical commonsense reasoning. While recent advancements of neural&#10;language models have demonstrated strong performance on various types of&#10;natural language inference tasks, our study based on a dataset of over 200k&#10;newly collected annotations suggests that neural language representations still&#10;only learn associations that are explicitly written down.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.04877" label="2004.04877">
        <attvalues>
          <attvalue for="0" value="Probing Neural Language Models for Human Tacit Assumptions" />
          <attvalue for="1" value="  Humans carry stereotypic tacit assumptions (STAs) (Prince, 1978), or&#10;propositional beliefs about generic concepts. Such associations are crucial for&#10;understanding natural language. We construct a diagnostic set of word&#10;prediction prompts to evaluate whether recent neural contextualized language&#10;models trained on large text corpora capture STAs. Our prompts are based on&#10;human responses in a psychological study of conceptual associations. We find&#10;models to be profoundly effective at retrieving concepts given associated&#10;properties. Our results demonstrate empirical evidence that stereotypic&#10;conceptual representations are captured in neural models derived from&#10;semi-supervised linguistic exposure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.14178" label="2304.14178">
        <attvalues>
          <attvalue for="0" value="mPLUG-Owl: Modularization Empowers Large Language Models with&#10;  Multimodality" />
          <attvalue for="1" value="  Large language models (LLMs) have demonstrated impressive zero-shot abilities&#10;on a variety of open-ended tasks, while recent research has also explored the&#10;use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl,&#10;a novel training paradigm that equips LLMs with multi-modal abilities through&#10;modularized learning of foundation LLM, a visual knowledge module, and a visual&#10;abstractor module. This approach can support multiple modalities and facilitate&#10;diverse unimodal and multimodal abilities through modality collaboration. The&#10;training paradigm of mPLUG-Owl involves a two-stage method for aligning image&#10;and text, which learns visual knowledge with the assistance of LLM while&#10;maintaining and even improving the generation abilities of LLM. In the first&#10;stage, the visual knowledge module and abstractor module are trained with a&#10;frozen LLM module to align the image and text. In the second stage,&#10;language-only and multi-modal supervised datasets are used to jointly fine-tune&#10;a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing&#10;the visual knowledge module. We carefully build a visually-related instruction&#10;evaluation set OwlEval. Experimental results show that our model outperforms&#10;existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction&#10;and visual understanding ability, multi-turn conversation ability, and&#10;knowledge reasoning ability. Besides, we observe some unexpected and exciting&#10;abilities such as multi-image correlation and scene text understanding, which&#10;makes it possible to leverage it for harder real scenarios, such as vision-only&#10;document comprehension. Our code, pre-trained model, instruction-tuned models,&#10;and evaluation set are available at https://github.com/X-PLUG/mPLUG-Owl. The&#10;online demo is available at https://www.modelscope.cn/studios/damo/mPLUG-Owl.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.17193" label="2402.17193">
        <attvalues>
          <attvalue for="0" value="When Scaling Meets LLM Finetuning: The Effect of Data, Model and&#10;  Finetuning Method" />
          <attvalue for="1" value="  While large language models (LLMs) often adopt finetuning to unlock their&#10;capabilities for downstream applications, our understanding on the inductive&#10;biases (especially the scaling properties) of different finetuning methods is&#10;still limited. To fill this gap, we conduct systematic experiments studying&#10;whether and how different scaling factors, including LLM model size,&#10;pretraining data size, new finetuning parameter size and finetuning data size,&#10;affect the finetuning performance. We consider two types of finetuning --&#10;full-model tuning (FMT) and parameter efficient tuning (PET, including prompt&#10;tuning and LoRA), and explore their scaling behaviors in the data-limited&#10;regime where the LLM model size substantially outweighs the finetuning data&#10;size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and&#10;experiments on bilingual machine translation and multilingual summarization&#10;benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative&#10;joint scaling law between finetuning data size and each other scaling factor;&#10;2) LLM finetuning benefits more from LLM model scaling than pretraining data&#10;scaling, and PET parameter scaling is generally ineffective; and 3) the optimal&#10;finetuning method is highly task- and finetuning data-dependent. We hope our&#10;findings could shed light on understanding, selecting and developing LLM&#10;finetuning methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;LLM finetuning &#10;&#10;With the significant increase of model size, updating all LLM parameters becomes computationally inefficient and unaffordable. Researchers thus resort to parameter efficient tuning methods that target achieving the best performance with minimal tunable parameters. Efforts in this direction mainly focus on developing efficient tunable modules for LLMs, such as adapters that insert small feed-forward layers~\cite{adapter-houlsby19a,bapna2019simple}, prefix and prompt tuning that appends tunable embeddings to the input~\cite{liliang-prefixtuning,lester2021prompt-tuning}, {LoRA} and compacter that adopts low-rank decomposition~\cite{hu2021lora,Rabeeh21compacter}, Bitfit that adds tunable bias vectors~\cite{Elad21BitFit}, IA3 that scales model activations~\cite{liu2022fewshot} and Q{LoRA} that leverages quantization~\cite{dettmers2023qlora}, to name a few. While previous studies reported encouraging performance with {PET}, e.g. reaching and even surpassing {FMT} across various domains~\cite{he2022unified,ding2022delta,liu2022fewshot,dettmers2023qlora}, they mainly focus on one or few experimental setups, leaving the question of how scaling affects the performance of different finetuning methods under-explored.&#10;&#10;Scaling Laws &#10;&#10;Recent research has shown that the performance of neural models can be predicted by a power-law of model and/or data sizes~\cite{Hestness2017Scaling,kaplan2020scaling}. Such pattern widely exists across different domains and model architectures, such as computer vision~\cite{zhai2021scalingViT}, autoregressive generative modeling~\cite{henighan2020scaling}, neural machine translation~\cite{gordon-etal-2021-data,Ghorbani2021scaling,pmlr-v162-bansal22b,zhang2022scaling}, multilingual translation~\cite{pmlr-v202-fernandes23a}, multi-modal modeling~\cite{aghajanyan2023scaling} and sparse neural architectures~\cite{frantar2023scaling}. These laws provide a valuable tool for guiding training decisions~\cite{hoffmann2022training} and model development by understanding how model performance evolves with scale, which greatly facilitates the development of LLMs~\cite{openai2023gpt4}. Unfortunately, the study of scaling for LLM finetuning lags behind badly, and our study fills this gap.&#10;&#10;The most closely related work to ours is~\cite{hernandez2021scaling} which explored the scaling for knowledge transfer by comparing finetuning with training from scratch. Our study is orthogonal to theirs with significant difference as our key focus is understanding the scaling of different factors for LLM finetuning, rather than the transfer.&#10;&#10;" />
          <attvalue for="4" value="Computer Science, Machine Learning Optimization, Linguistics, Scaling Properties Analysis, Mathematics, Language Model Finetuning" />
        </attvalues>
      </node>
      <node id="2408.00357" label="2408.00357">
        <attvalues>
          <attvalue for="0" value="DeliLaw: A Chinese Legal Counselling System Based on a Large Language&#10;  Model" />
          <attvalue for="1" value="  Traditional legal retrieval systems designed to retrieve legal documents,&#10;statutes, precedents, and other legal information are unable to give&#10;satisfactory answers due to lack of semantic understanding of specific&#10;questions. Large Language Models (LLMs) have achieved excellent results in a&#10;variety of natural language processing tasks, which inspired us that we train a&#10;LLM in the legal domain to help legal retrieval. However, in the Chinese legal&#10;domain, due to the complexity of legal questions and the rigour of legal&#10;articles, there is no legal large model with satisfactory practical application&#10;yet. In this paper, we present DeliLaw, a Chinese legal counselling system&#10;based on a large language model. DeliLaw integrates a legal retrieval module&#10;and a case retrieval module to overcome the model hallucination. Users can&#10;consult professional legal questions, search for legal articles and relevant&#10;judgement cases, etc. on the DeliLaw system in a dialogue mode. In addition,&#10;DeliLaw supports the use of English for counseling. we provide the address of&#10;the system: https://data.delilegal.com/lawQuestion.&#10;" />
          <attvalue for="2" value="&#10;Traditional legal retrieval systems \cite{1} aim to retrieve legal information such as legal documents, statutes and case law. However, they are limited in their ability to semantically understand user queries.Open-source large language models, such as LLAMA \cite{2} and LLAMA2 \cite{3}, Falcon \cite{4}, Vicuna \cite{5}, MOSS \cite{6}, ChatGLM, and ChatGLM2 \cite{7}, have demonstrated satisfactory performance in general-purpose domains following pre-training on large-scale corpora. However, deploying them in very specialized fields such as medicine, legal, and finance presents challenges, primarily due to the scarcity of high-quality, fine-tuned data in these areas and the inherent problem of &quot;hallucination&quot; in generative models. Even the state-of-the-art GPT-4 model \cite{8} in the Chinese legal domain generates a significant number of fictitious legal texts, highlighting the prevalence of this issue.&#10;&#10;Recent studies have explored the fine-tuning of open-source LLMs using legal data to develop legal-specific LLMs. One example is the Chinese legal model LawGPT \cite{9}, which is obtained by fine-tuning ChatGLM-6B LoRA 16-bit instructions. The fine-tuning process involves utilizing existing legal Q\&amp;A datasets and constructing high-quality legal text Q\&amp;A based on real legal articles and cases using ChatGPT’s API. While LawGPT incorporates more legal knowledge into ChatGLM, its performance remains suboptimal. To address this limitation, Huang et al. \cite{10} proposed a method that retrieves relevant legal articles based on the user’s query and contextual information, which can serve as evidence for the query before generating each response. The final response is then generated based on these legal articles. ChatLaw \cite{11} leverages an external knowledge base to mitigate model hallucinations. However, unlike LawGPT, ChatLaw trains a model specifically to extract legal feature words from the user’s everyday language. These extracted legal keywords are then used as queries for law retrieval.&#10;&#10;While Chinese legal LLMs have made significant advancements compared to general-purpose LLMs, they still fall short of direct practical application. To address this gap, this work introduces DeliLaw, a practical legal interrogation system built upon a large language model. We first trained deep learning models dedicated to intent classification to effectively classify user queries according to different legal application scenarios. To build our premium dataset, we sourced extensive data from expert Chinese legal platforms, enriched with statutes accessed through our regulatory retrieval system, ensuring relevant legal texts were included. We crafted diverse prompts based on professional legal advice to facilitate answer generation. Lawyers were engaged to annotate the data, critical for upholding the dataset's high quality and professional reliability. For legal retrieval, we enhance the adaptability and accuracy of the system through a two-stage fine-tuning.The laws in our system response are retrieved from the law database, and since the law database is updated in real time, all the laws returned by the system are real and valid laws. Combining vector library retrieval and ElasticSearch technology, we constructed an efficient case retrieval module to provide users with comprehensive and practical legal information. In the end, by organically integrating the modules and comprehensively reasoning about DeliLaw, we successfully built an intelligent and professional system to meet users' diverse legal needs.The system’s interface is depicted in Figure 1. It enables users to engage in a free dialogue with an AI assistant to seek legal-related inquiries. The AI assistant responds professionally by providing specific legal provisions based on retrieved laws. The link to our demo video is:.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Law, Large Language Models, Linguistics, Legal Information Retrieval, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2402.14851" label="2402.14851">
        <attvalues>
          <attvalue for="0" value="$R^3$: &quot;This is My SQL, Are You With Me?&quot; A Consensus-Based Multi-Agent&#10;  System for Text-to-SQL Tasks" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated strong performance on various&#10;tasks. To unleash their power on the Text-to-SQL task, we propose $R^3$&#10;(Review-Rebuttal-Revision), a consensus-based multi-agent system for&#10;Text-to-SQL tasks. $R^3$ outperforms the existing single LLM Text-to-SQL&#10;systems as well as the multi-agent Text-to-SQL systems by $1.3\%$ to $8.1\%$ on&#10;Spider and Bird. Surprisingly, we find that for Llama-3-8B, $R^3$ outperforms&#10;chain-of-thought prompting by over 20\%, even outperforming GPT-3.5 on the&#10;development set of Spider.&#10;" />
          <attvalue for="2" value="&#10;Text-to-SQL, the task of converting natural language to SQL queries, enables non-technical users to access databases with natural language \cite{deng-etal-2022-recent, katsogiannis2023survey}. &#10;Recently, Large Language Models (LLMs) have made significant progress on various tasks \cite{touvron2023llama,openai2023gpt}.&#10;&#10;Although researchers have proposed various methods to enhance the reasoning abilities of LLMs \cite{wei2022chain,yao2023tree,besta2024graph}, However, they are still facing challenges with Text-to-SQL tasks \cite{li2023can,hong2024next}. &#10;The LLM-based multi-agent system leverages collective intelligence from a group of LLMs and have achieved exceptional performance across various tasks \cite{park2023generative,hong2023metagpt,xu2023language}, but little work explores using them on Text-to-SQL.&#10;The existing multi-agent Text-to-SQL system first decomposes the task into multiple subtasks which are then accomplished step-by-step by agents \cite{wang2023mac}. &#10;While achieving remarkable performances, such a decomposition-based system necessitates extensive manual prompt engineering and logic design. &#10;&#10;We propose $R^3$, a consensus-based multi-agent system for Text-to-SQL tasks. &#10;The proposed system draws inspiration from the peer-review mechanism, featuring one agent as the SQL-writer and several reviewers automatically generated by the LLM.&#10;Once the generated SQL query is tested to be executable, the system will step into a review process, where we use the execution results to guide the SQL-writer and reviewers to refine the SQL. &#10;Through rounds of ``review'', ``negotiation or rebuttal'', and ``revision'', SQL-writer and reviewers will finally achieve consensus and deliver a solution with collective agreement (see \ref{fig:3r}).&#10;&#10;We test $R^3$~on the popular Spider and Bird benchmarks.&#10;$R^3$~outperforms the existing single LLM as well as the multi-agent Tex-to-SQL systems by $1.3\%$ to $8.1\%$ on Spider and Bird. &#10;Surprisingly, we find that for Llama-3-8B, $R^3$~outperforms chain-of-thought prompting by over 20\%,&#10;even outperforming GPT-3.5 on the Spider-Dev set.&#10;Our contributions can be summarized as follows:&#10;\begin{enumerate}[leftmargin=\parindent,align=left,labelwidth=\parindent,labelsep=0pt]&#10; \item To the best of our knowledge, $R^3$~is the first Text-to-SQL system to use the execution result for SQL refinements, and the first Text-to-SQL system to equip agents with memory sequences to enhance SQL generation.&#10; \item $R^3$~offers a consensus-based multi-agent system for Text-to-SQL tasks. &#10; Using very succinct prompts, it achieves strong performance compared to other systems.&#10; In addition, it effectively helps open-source LLMs such as Llama-3-8B on SQL generation.&#10; \item We provide a detailed error analysis of $R^3$~on the existing Text-to-SQL benchmarks, shedding light on future research on the Text-to-SQL task.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Text-to-SQL Systems, Language Model Performance, Multi-Agent Consensus" />
        </attvalues>
      </node>
      <node id="2208.10099" label="2208.10099">
        <attvalues>
          <attvalue for="0" value="Recent Advances in Text-to-SQL: A Survey of What We Have and What We&#10;  Expect" />
          <attvalue for="1" value="  Text-to-SQL has attracted attention from both the natural language processing&#10;and database communities because of its ability to convert the semantics in&#10;natural language into SQL queries and its practical application in building&#10;natural language interfaces to database systems. The major challenges in&#10;text-to-SQL lie in encoding the meaning of natural utterances, decoding to SQL&#10;queries, and translating the semantics between these two forms. These&#10;challenges have been addressed to different extents by the recent advances.&#10;However, there is still a lack of comprehensive surveys for this task. To this&#10;end, we review recent progress on text-to-SQL for datasets, methods, and&#10;evaluation and provide this systematic survey, addressing the aforementioned&#10;challenges and discussing potential future directions. We hope that this survey&#10;can serve as quick access to existing work and motivate future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.09687" label="2308.09687">
        <attvalues>
          <attvalue for="0" value="Graph of Thoughts: Solving Elaborate Problems with Large Language Models" />
          <attvalue for="1" value="  We introduce Graph of Thoughts (GoT): a framework that advances prompting&#10;capabilities in large language models (LLMs) beyond those offered by paradigms&#10;such as Chain-of-Thought or Tree of Thoughts (ToT). The key idea and primary&#10;advantage of GoT is the ability to model the information generated by an LLM as&#10;an arbitrary graph, where units of information (&quot;LLM thoughts&quot;) are vertices,&#10;and edges correspond to dependencies between these vertices. This approach&#10;enables combining arbitrary LLM thoughts into synergistic outcomes, distilling&#10;the essence of whole networks of thoughts, or enhancing thoughts using feedback&#10;loops. We illustrate that GoT offers advantages over state of the art on&#10;different tasks, for example increasing the quality of sorting by 62% over ToT,&#10;while simultaneously reducing costs by &gt;31%. We ensure that GoT is extensible&#10;with new thought transformations and thus can be used to spearhead new&#10;prompting schemes. This work brings the LLM reasoning closer to human thinking&#10;or brain mechanisms such as recurrence, both of which form complex networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.08426" label="2406.08426">
        <attvalues>
          <attvalue for="0" value="Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL" />
          <attvalue for="1" value="  Generating accurate SQL from natural language questions (text-to-SQL) is a&#10;long-standing challenge due to the complexities in user question understanding,&#10;database schema comprehension, and SQL generation. Conventional text-to-SQL&#10;systems, comprising human engineering and deep neural networks, have made&#10;substantial progress. Subsequently, pre-trained language models (PLMs) have&#10;been developed and utilized for text-to-SQL tasks, achieving promising&#10;performance. As modern databases become more complex, the corresponding user&#10;questions also grow more challenging, causing PLMs with parameter constraints&#10;to produce incorrect SQL. This necessitates more sophisticated and tailored&#10;optimization methods, which, in turn, restricts the applications of PLM-based&#10;systems. Recently, large language models (LLMs) have demonstrated significant&#10;capabilities in natural language understanding as the model scale increases.&#10;Therefore, integrating LLM-based implementation can bring unique opportunities,&#10;improvements, and solutions to text-to-SQL research. In this survey, we present&#10;a comprehensive review of LLM-based text-to-SQL. Specifically, we propose a&#10;brief overview of the technical challenges and the evolutionary process of&#10;text-to-SQL. Then, we provide a detailed introduction to the datasets and&#10;metrics designed to evaluate text-to-SQL systems. After that, we present a&#10;systematic analysis of recent advances in LLM-based text-to-SQL. Finally, we&#10;discuss the remaining challenges in this field and propose expectations for&#10;future research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.18940" label="2310.18940">
        <attvalues>
          <attvalue for="0" value="Language Agents with Reinforcement Learning for Strategic Play in the&#10;  Werewolf Game" />
          <attvalue for="1" value="  Agents built with large language models (LLMs) have shown great potential&#10;across a wide range of domains. However, in complex decision-making tasks, pure&#10;LLM-based agents tend to exhibit intrinsic bias in their choice of actions,&#10;which is inherited from the model's training data and results in suboptimal&#10;performance. To develop strategic language agents, i.e., agents that generate&#10;flexible language actions and possess strong decision-making abilities, we&#10;propose a novel framework that powers LLM-based agents with reinforcement&#10;learning (RL). We consider Werewolf, a popular social deduction game, as a&#10;challenging testbed that emphasizes versatile communication and strategic&#10;gameplay. To mitigate the intrinsic bias in language actions, our agents use an&#10;LLM to perform deductive reasoning and generate a diverse set of action&#10;candidates. Then an RL policy trained to optimize the decision-making ability&#10;chooses an action from the candidates to play in the game. Extensive&#10;experiments show that our agents overcome the intrinsic bias and outperform&#10;existing LLM-based agents in the Werewolf game. We also conduct human-agent&#10;experiments and find that our agents achieve human-level performance and&#10;demonstrate strong strategic play.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.04894" label="2403.04894">
        <attvalues>
          <attvalue for="0" value="ConstitutionalExperts: Training a Mixture of Principle-based Prompts" />
          <attvalue for="1" value="  Large language models (LLMs) are highly capable at a variety of tasks given&#10;the right prompt, but writing one is still a difficult and tedious process. In&#10;this work, we introduce ConstitutionalExperts, a method for learning a prompt&#10;consisting of constitutional principles (i.e. rules), given a training dataset.&#10;Unlike prior methods that optimize the prompt as a single entity, our method&#10;incrementally improves the prompt by surgically editing individual principles.&#10;We also show that we can improve overall performance by learning unique prompts&#10;for different semantic regions of the training data and using a&#10;mixture-of-experts (MoE) architecture to route inputs at inference time. We&#10;compare our method to other state of the art prompt-optimization techniques&#10;across six benchmark datasets. We also investigate whether MoE improves these&#10;other techniques. Our results suggest that ConstitutionalExperts outperforms&#10;other prompt optimization techniques by 10.9% (F1) and that mixture-of-experts&#10;improves all techniques, suggesting its broad applicability.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) are highly capable at a variety of NLP tasks when prompted with appropriate natural language instructions \cite{bubeck2023sparks,few-shot-learners}.&#10;However, writing an LLM prompt remains a difficult and ambiguous task, often involving significant experimentation and effort \cite{johnny-cant-prompt}.&#10;&#10;&#10;Many methods for automatic prompt optimization have recently been explored.&#10;Some rely on access to model parameters and gradients to optimize discrete \cite{shin-etal-2020-autoprompt} or continuous \cite{lester-etal-2021-power, qin-eisner-2021-learning} prompts given task-specific training data.&#10;Others involve revising the task-prompt with discrete manipulations, such as through reinforcement learning \cite{deng-etal-2022-rlprompt, zhang2022tempera, hao2022optimizing}.&#10;Discrete mutations of the task-prompt can also be made via another LLM \cite{zhou2023llms_are_prompt_engineers,pryzant-etal-2023-automatic}.&#10;More recent work has explored automatically optimizing both the task-prompt as well as metaprompts for deriving mutations \cite{fernando2023promptbreeder}.&#10;These methods can still produce hard-to-interpret prompts, and concurrently, they all assume that a single, optimized prompt should be applied at inference.&#10;&#10;In this work we introduce ConstitutionalExperts, a technique for producing a set of principle-based prompts and selectively applying them at inference.&#10;Our approach is inspired by the ConstitutionalAI workflow \cite{bai2022constitutional} used to create fine-tuning datasets for LLMs.&#10;Our method discovers and incrementally improves a prompt via a set of principles or rules.&#10; We refer to one of these principle-based prompts as a ConstitutionalExpert, or simply &quot;Expert.&quot;&#10;Similar to prior techniques, our method iteratively updates an initial prompt (via mutation metaprompts), based on its performance on a training set \cite{pryzant-etal-2023-automatic}.&#10;However, the prompts produced by ConstitutionalExperts are structured as a list of principles or rules, thus we refer to one of these prompts as a ConstitutionalExpert.&#10;This structure enables targeted, incremental changes to the learned prompt: instead of rewriting the entire prompt, a principle is either revised, added, or removed at each step.&#10;Additionally, we train a unique ConstitutionalExpert for different semantic regions of the training data.&#10;Thus each ConstitutionalExpert specializes in a different aspect of the problem space, enabling them to collectively outperform generalist prompts.&#10;We drew lessons from prior work showing that selecting the most semantically similar examples at inference time improves the performance of few-shot prompts \cite{nori2023generalist}.&#10;&#10;To evaluate ConstitutionalExperts, we compare it to state-of-the-art prompt optimizing baselines, including ProTeGi \cite{pryzant-etal-2023-automatic} and PromptBreeder \cite{fernando2023promptbreeder}, across six NLP tasks. &#10;We observe that our method outperforms the prompt optimization baselines by a statistically significant margin, and that MoE improves the baselines on average.&#10;We finish by discussing the limitations of our method and future work. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Prompt Engineering Techniques, Mathematics, Artificial Intelligence Architecture, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2311.09709" label="2311.09709">
        <attvalues>
          <attvalue for="0" value="The Ups and Downs of Large Language Model Inference with Vocabulary&#10;  Trimming by Language Heuristics" />
          <attvalue for="1" value="  Deploying large language models (LLMs) encounters challenges due to intensive&#10;computational and memory requirements. Our research examines vocabulary&#10;trimming (VT) inspired by restricting embedding entries to the language of&#10;interest to bolster time and memory efficiency. While such modifications have&#10;been proven effective in tasks like machine translation, tailoring them to LLMs&#10;demands specific modifications given the diverse nature of LLM applications. We&#10;apply two language heuristics to trim the full vocabulary - Unicode-based&#10;script filtering and corpus-based selection - to different LLM families and&#10;sizes. The methods are straightforward, interpretable, and easy to implement.&#10;It is found that VT reduces the memory usage of small models by nearly 50% and&#10;has an upper bound of 25% improvement in generation speed. Yet, we reveal the&#10;limitations of these methods in that they do not perform consistently well for&#10;each language with diminishing returns in larger models.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) are gaining increasing attention given their strong performance \cite{Radford2019Language,brown2020language,workshop2023BLOOM,touvron2023LLaMA}. LLMs, especially multilingual ones, hold vocabulary items for many languages and scripts, which entail a costly matrix multiplication $H \times |V|$ in the output layer, where $H$ is the hidden size and $|V|$ is the size of a vocabulary $V$. This expensive operation leads to increased costs of both memory and time given the autoregressive nature of LLM decoding. Given their substantial size, this latency in inference significantly escalates the expense of LLM deployment.&#10;&#10;In practice, creating a sub-vocabulary $V'$ with $|V'|\ll|V|$ and only loading its corresponding embedding entries for inference seems favourable since most logits from the output layer do not affect the hypothesis token(s) at each time step. Vocabulary trimming (VT) has been actively explored in machine translation \cite[often called shortlisting,][]{schwenk-etal-2007-smooth, le-etal-2012-continuous, devlin-etal-2014-fast}---it computes token-level alignments and makes potential target tokens a sub-vocabulary. While anticipating certain limitations such as domain mismatch \cite{bogoychev-chen-2021-highs,domhan-etal-2022-devil}, vocabulary shortlisting in LLMs poses a fundamental challenge: often LLM outputs are variable and open-ended, complicating the determination of the required lexicons. Recent attempts at multilingual pre-trained models select tokens in a task's language \cite{abdaoui-etal-2020-load,ushio2023efficient}. Nonetheless, research in this direction is still limited, especially in speed considerations.&#10;&#10;We follow the idea of fitting vocabulary to the language of the downstream task. Specifically, We examine two strategies: Unicode-based filtering where vocabulary items are removed if they do not belong to the task language, and corpus-based selection where we record vocabulary hits from a large representative corpus. After experimenting with LLMs from two families of different sizes, we identify a good upper bound of memory reduction with several limitations and outlooks: 1) Unicode-based script filtering maintains quality for Latin-based languages but harms languages requiring code-mixing. 2) Corpus-based selection leads to fewer alterations but is less effective in reducing the embedding size. 3) Embeddings are proportionally smaller in larger models (with smaller vocabularies). Yet we argue that VT can be applied orthogonally to other efficiency methods like efficient attention, quantization, etc.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Information Technology, Linguistics, Vocabulary Trimming Techniques, Artificial Intelligence, Computational Efficiency, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2101.00421" label="2101.00421">
        <attvalues>
          <attvalue for="0" value="The Highs and Lows of Simple Lexical Domain Adaptation Approaches for&#10;  Neural Machine Translation" />
          <attvalue for="1" value="  Machine translation systems are vulnerable to domain mismatch, especially in&#10;a low-resource scenario. Out-of-domain translations are often of poor quality&#10;and prone to hallucinations, due to exposure bias and the decoder acting as a&#10;language model. We adopt two approaches to alleviate this problem: lexical&#10;shortlisting restricted by IBM statistical alignments, and hypothesis&#10;re-ranking based on similarity. The methods are computationally cheap, widely&#10;known, but not extensively experimented on domain adaptation. We demonstrate&#10;success on low-resource out-of-domain test sets, however, the methods are&#10;ineffective when there is sufficient data or too great domain mismatch. This is&#10;due to both the IBM model losing its advantage over the implicitly learned&#10;neural alignment, and issues with subword segmentation of out-of-domain words.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural Machine translation (NMT) has achieved state-of-the-art performance in a variety of language pairs and settings \cite{bahdanau_nmt, vaswani_transformer}, but it is vulnerable to domain mismatch, where the test set differs significantly from the training data in terms of vocabulary, genre, length, etc. This issue is exacerbated in a low-resource condition \cite{koehn-knowles-2017-six}.&#10;&#10;Teacher forcing is used during traditional maximum likelihood neural network training, leading to a strong exposure bias, and model confusion when presented with unexpected sequences. This typically results in hallucinations in the output \cite{mller2019domain}, because the overly zealous language model component prefers a fluent translation, as opposed to an adequate one. A number of methods have been proposed in order to tackle the issue: exposing the model to its predictions during training \cite{ranzato2015sequence, shen-etal-2016-minimum, zhang-etal-2019-bridging, rico-exposure}; tuning directly towards BLEU \cite{wiseman-rush-2016-sequence} or using minimum Bayes risk decoding \cite{kumar-byrne-2004-minimum, stahlberg-etal-2017-neural}.&#10;A common weakness of such methods is that they are computationally expensive.&#10;&#10;In this paper, we adopt and experiment with two approaches inspired by previous research. We use lexical shortlisting to interpolate a statistical alignment model with NMT; on top of it, we perform n-best list re-ranking by hypothesis agreement. Our aim is to constrain the lexical choice of the decoder, to prevent hallucinations from being generated. The methods are computationally simpler, as they require no change to the model or training. We analyse the effectiveness of these methods in different scenarios of domain adaptation. We show BLEU gains on a variety of out-of-domain datasets in a low-resource English-German setting. However, the methods show no improvements once the datasets are large, or the domains are too distant. &#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.06618" label="2205.06618">
        <attvalues>
          <attvalue for="0" value="The Devil is in the Details: On the Pitfalls of Vocabulary Selection in&#10;  Neural Machine Translation" />
          <attvalue for="1" value="  Vocabulary selection, or lexical shortlisting, is a well-known technique to&#10;improve latency of Neural Machine Translation models by constraining the set of&#10;allowed output words during inference. The chosen set is typically determined&#10;by separately trained alignment model parameters, independent of the&#10;source-sentence context at inference time. While vocabulary selection appears&#10;competitive with respect to automatic quality metrics in prior work, we show&#10;that it can fail to select the right set of output words, particularly for&#10;semantically non-compositional linguistic phenomena such as idiomatic&#10;expressions, leading to reduced translation quality as perceived by humans.&#10;Trading off latency for quality by increasing the size of the allowed set is&#10;often not an option in real-world scenarios. We propose a model of vocabulary&#10;selection, integrated into the neural translation model, that predicts the set&#10;of allowed output words from contextualized encoder representations. This&#10;restores translation quality of an unconstrained system, as measured by human&#10;evaluations on WMT newstest2020 and idiomatic expressions, at an inference&#10;latency competitive with alignment-based selection using aggressive thresholds,&#10;thereby removing the dependency on separately trained alignment models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;There are two dominant approaches to generate a restricted set of target word candidates (i) using an external model and (ii) using the \gls*{nmt} system itself.&#10;&#10;In the first approach, a short-list of translation candidates is generated from word-alignments \cite{jean-etal-2015-using,kim-etal-2019-research}, phrase table, and the most common target words \cite{mi-etal-2016-vocabulary}. \cite{vocabularySelectionStrategies} propose an additional method using support vector machines to predict target candidates from a sparse representation of the source sentence.&#10;&#10;In the second approach, \cite{sankaran2017attention} build alignment probability table from the soft-attention layer from decoder to encoder. However, applying their method to multi-head attention in Transformer is non-trivial as attention may not capture word-alignments in multiple attention layers~\cite{li-etal-2019-word}. \cite{shi-knight-2017-speeding}~use local sensitive hashing to shrink the target vocabulary during decoding, though their approach only reduces latency on CPUs instead of GPUs. &#10;&#10;\cite{chen2019learning} reduce the softmax computation by first predicting a cluster of target words and then perform exact search (i.e., softmax) on that cluster. The clustering process is trained jointly with the translation process in their approach.&#10;&#10;Closely related to our work is \cite{weng-etal-2017-neural}, who predict all words in a target sentence from the initial hidden state of the decoder.&#10;Our NVS model differs from theirs in that we make a prediction for each source token and aggregate the results via max-pooling to scale with sentence length.&#10;Recent work of~\cite{bogoychev-chen-2021-highs} illustrates the risk associated with reducing latency via vocabulary selection in domain-mismatched settings.&#10;Our work takes this a step further by providing a detailed analysis on the shortcomings of vocabulary selection and proposing a model to mitigate them. &#10;&#10;Related to our findings on non-compositional expressions, \cite{renduchintala-etal-2021-gender} evaluate the effect of methods used to speed up decoding in Transformer models on gender bias and find minimal BLEU degradations but reduced gendered noun translation performance on a targeted test set.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2404.12845" label="2404.12845">
        <attvalues>
          <attvalue for="0" value="TartuNLP @ SIGTYP 2024 Shared Task: Adapting XLM-RoBERTa for Ancient and&#10;  Historical Languages" />
          <attvalue for="1" value="  We present our submission to the unconstrained subtask of the SIGTYP 2024&#10;Shared Task on Word Embedding Evaluation for Ancient and Historical Languages&#10;for morphological annotation, POS-tagging, lemmatization, character- and&#10;word-level gap-filling. We developed a simple, uniform, and computationally&#10;lightweight approach based on the adapters framework using parameter-efficient&#10;fine-tuning. We applied the same adapter-based approach uniformly to all tasks&#10;and 16 languages by fine-tuning stacked language- and task-specific adapters.&#10;Our submission obtained an overall second place out of three submissions, with&#10;the first place in word-level gap-filling. Our results show the feasibility of&#10;adapting language models pre-trained on modern languages to historical and&#10;ancient languages via adapter training.&#10;" />
          <attvalue for="2" value="&#10;&#10;The application of natural language processing techniques and pre-trained language models to analysis of ancient and historical languages is a compelling subject of research that has been so far overlooked.&#10;While there exist a number of benchmarks, such as GLUE \cite{wang2018glue}, SuperGLUE \cite{wang2019superglue}, or XGLUE \cite{liang2020xglue}, for evaluating the quality of embeddings and language models for modern languages, such benchmarks are lacking for ancient and historical languages. Thus, the SIGTYPE 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages contributes to filling this gap.&#10;&#10;In the current transformer-based language models paradigm, one of the common approaches to solving the tasks present in such benchmarks is to use the task data to fine-tune an encoder transformer model. The approach was first in introduced in \cite{devlin-etal-2019-bert} and was shown to yield superior results compared to the alternatives on the GLUE benchmark. Additionally, the proposed pre-training method is very similar to the word-level gap-filling problem in the shared task. Consequently, the model can be applied to solving the problem directly. This motivates our choice to use a transformer model in the shared task.&#10;&#10;Large pre-trained language models, however, are predominantly trained on corpora of modern languages, with few exceptions such as LatinBERT~\cite{bamman2020latin}. Ancient and historical languages generally lack sufficient data to perform full pre-training of large language models or to continue training from a checkpoint trained on some different language. Full fine-tuning of modern language models on a relatively small amount data in an ancient/historical language might lead to overfitting and catastrophic forgetting. These considerations are also common in context of other low resource tasks or domains. &#10;&#10;Several approaches have been proposed to alleviate these issues. For instance, the supplementary training approach proposed by \cite{phang2018sentence} involves first fine-tuning a pre-trained model on an intermediary labeled task with abundant data, and then on the target task which may have limited data. This approach showed gains over simply fine-tuning on the target task. &#10;\cite{pfeiffer2020mad} developed a cross-lingual transfer-learning approach based on the adapters framework for parameter efficient fine-tuning of language models~\cite{houlsby2019parameter, bapna2019simple}. Their approach involves fine-tuning a language adapter and a task adapter stacked on top of each other. This adapter-based method is expanded by \cite{pfeiffer2020unks} by adopting a custom tokenizer and an embedding layer. Several ways of initializing the new embedding layers are compared on underrepresented modern languages. Since ancient and historical languages are underrepresented, the same techniques should be applicable in this context as well.&#10;&#10;A useful feature of both the supplementary training approach of \cite{phang2018sentence} and the adapter-based training of \cite{pfeiffer2020mad} is that they provide a uniform framework that can be applied to different languages and tasks in a similar manner. While previous related works mainly focused on modern languages, we aim to assess the feasibility of this unified approach for ancient and historical languages. &#10;&#10;Our submission to the SIGTYP 2024 Shared Task on Ancient on Word Embedding Evaluation for Ancient and Historical Languages adopts the methods described by \cite{pfeiffer2020mad, pfeiffer2020unks} by stacking fine-tuned language and task adapters, and customizing the tokenizer and the embedding layers.&#10;Our system is implemented as a unified framework, where various pre-trained models, languages, and tasks can be plugged in.&#10;The system was evaluated on POS tagging, morphological annotation, lemmatization, and filling-in both word-level and character-level gaps for 16 ancient and historical languages. &#10;We participated in the unconstrained subtask, which allowed using any additional resources, such as pre-trained language models. Out of 3 participants on the leaderboard we took a close second place, with the first place on the word gap-filling task. &#10;Our main contribution is showing that by adopting the parameter-efficient adapter training methodology, the large language models pre-trained on modern languages are applicable also to low-resource ancient and historical languages.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Historical Language Processing, Artificial Intelligence, Morphological Annotation, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2004.01401" label="2004.01401">
        <attvalues>
          <attvalue for="0" value="XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training,&#10;  Understanding and Generation" />
          <attvalue for="1" value="  In this paper, we introduce XGLUE, a new benchmark dataset that can be used&#10;to train large-scale cross-lingual pre-trained models using multilingual and&#10;bilingual corpora and evaluate their performance across a diverse set of&#10;cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in&#10;English for natural language understanding tasks only, XGLUE has two main&#10;advantages: (1) it provides 11 diversified tasks that cover both natural&#10;language understanding and generation scenarios; (2) for each task, it provides&#10;labeled data in multiple languages. We extend a recent cross-lingual&#10;pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and&#10;generation tasks, which is evaluated on XGLUE as a strong baseline. We also&#10;evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for&#10;comparison.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.10053" label="2009.10053">
        <attvalues>
          <attvalue for="0" value="Latin BERT: A Contextual Language Model for Classical Philology" />
          <attvalue for="1" value="  We present Latin BERT, a contextual language model for the Latin language,&#10;trained on 642.7 million words from a variety of sources spanning the Classical&#10;era to the 21st century. In a series of case studies, we illustrate the&#10;affordances of this language-specific model both for work in natural language&#10;processing for Latin and in using computational methods for traditional&#10;scholarship: we show that Latin BERT achieves a new state of the art for&#10;part-of-speech tagging on all three Universal Dependency datasets for Latin and&#10;can be used for predicting missing text (including critical emendations); we&#10;create a new dataset for assessing word sense disambiguation for Latin and&#10;demonstrate that Latin BERT outperforms static word embeddings; and we show&#10;that it can be used for semantically-informed search by querying contextual&#10;nearest neighbors. We publicly release trained models to help drive future work&#10;in this space.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.06003" label="2310.06003">
        <attvalues>
          <attvalue for="0" value="Rethinking Memory and Communication Cost for Efficient Large Language&#10;  Model Training" />
          <attvalue for="1" value="  Recently, various distributed strategies for large language model training&#10;have been proposed. However, these methods provided limited solutions for the&#10;trade-off between memory consumption and communication cost. In this paper, we&#10;rethink the impact of memory consumption and communication costs on the&#10;training speed of large language models, and propose a memory-communication&#10;balanced strategy set Partial Redundancy Optimizer (PaRO). PaRO provides&#10;comprehensive options which reduces the amount and frequency of inter-group&#10;communication with minor memory redundancy by fine-grained sharding strategy,&#10;thereby improving the training efficiency in various training scenarios.&#10;Additionally, we propose a Hierarchical Overlapping Ring (HO-Ring)&#10;communication topology to enhance communication efficiency between nodes or&#10;across switches in large language model training. Our experiments demonstrate&#10;that PaRO significantly improves training throughput by 1.19x-2.50x compared to&#10;the SOTA method and achieves a near-linear scalability. The HO-Ring algorithm&#10;improves communication efficiency by 36.5% compared to the traditional Ring&#10;algorithm.&#10;" />
          <attvalue for="2" value="&#10;With the development of machine learning technology, the overall performance of deep learning algorithms in fields such as face recognition, recommender system, and natural language processing has significantly improved \cite{girshick2014rich,G-Meta,NEURIPS2020_1457c0d6}. &#10;Recent research shows that large model training is beneficial to improve model quality. &#10;Over the past few years, model size has increased from 110 million parameters for BERT \cite{devlin2019bert} to 175 billion parameters for GPT-3 \cite{NEURIPS2020_1457c0d6}.&#10;However, training such large language model (LLM) is not an easy task, as it requires a significant amount of computing resources and presents challenges in terms of system complexity.&#10;&#10;As the size of the model and the amount of training data increase, the computing power of a single GPU cannot meet the training needs of large-scale networks. &#10;In LLM training, to effectively utilize the computing power and memory of hundreds of GPU devices, a variety of distributed parallel training technologies have been proposed, such as data parallelism (DP), tensor parallelism (TP) and pipeline parallelism (PP) \cite{2023ColossalAI}.&#10;In DP, an entire dataset is evenly partitioned into mutually exclusive subsets before training, and each worker works on a separate subset of them. &#10;TP divides the calculation and memory load of a single layer onto multiple GPUs by modifying the calculation method within the layer. &#10;PP puts different layers on different GPUs, and then divides the computing and memory loads onto multiple GPUs. &#10;However, TP and PP require modification of the model implement, which is inefficient for developers.&#10;In contrast, data parallelism has become the most mainstream distributed parallel method due to its simplicity.&#10;&#10;In data parallelism, the replicated model on each GPU processes a portion of the input batch, resulting in a large amount of communication data when fusing gradients.&#10;Andrew \cite{baidu-allreduce} applied a ring topology on all-reduce to balance the communication load.&#10;By defining the communication topology, the communication pressure is evenly distributed to each GPU.&#10;However, since a complete model is copied on each GPU, significant memory redundancy occurs, especially when training large models \cite{proficz_improving_2018}.&#10;To this end, Rajbhandari et al. \cite{ZeRO} proposed the Zero Redundancy Optimization (ZeRO) strategy set, which splits the model state (i.e. optimizer state, gradient and parameters) based on data parallelism and reconstructs them through the collective communication.&#10;It reduces memory consumption in LLM training and improves training efficiency by applying larger batch sizes.&#10;&#10;Since ZeRO retains the simplicity, ease of use, and versatility of DP, it has been widely used in LLM training.&#10;ZeRO needs to be adapted to specific training frameworks and hardware equipment to fully exploit its advantages.&#10;In high-performance clusters such as NVIDIA DGX-2 or DGX-A100 \cite{Blink2020}, NVLink/NVSwitch with a bandwidth of up to 4.8TGbps is configured within the node, while the bandwidth of InfiniBand or Ethernet between nodes is only 200$\sim$800Gbps.&#10;The mismatch of bandwidth within and between nodes limits the training efficiency of ZeRO.&#10;To speed up model training, ZeRO requires more GPU resources, which will result in greater collective communication volume.&#10;To reduce collective communication volume, MiCS \cite{MiCS22} proposes a cluster grouping strategy in which all model states are partitioned within each group and replicated across different groups. &#10;However, this partitioning strategy incurs significant memory costs, particularly in scenarios with a large number of groups.&#10;&#10;In this paper, we systematically combine cluster grouping with different partitioning of different model states to trade off the memory and communication costs.&#10;Based on the memory consumption and synchronization frequency of the optimizer state, gradients and parameters, we design several optimization solutions to reduce overall communication costs and frequency with minimal memory redundancy. &#10;Additionally, we optimize the communication topology of ring all-gather and reduce-scatter operations by performing intra- and inter-node communication simultaneously. &#10;This strategy reduces inter-node communication volume and improves inter-node bandwidth utilization. &#10;We plan to release the code, pending approval from the company.&#10;The main contributions of the paper are summarized as follows:&#10;{&#10;&#10;\begin{itemize}&#10;&#10;\item We systematically analyzed the impact of memory consumption and communication costs on the training speed of LLMs, and proposed an overall guideline for balancing memory and communication.&#10; \item We proposed the Partial Redundancy Optimizer (PaRO) strategy set, which provides more refined options for the trade-off between memory consumption and communication costs in different training scenarios.&#10; PaRO significantly increased training throughput by 1.19$\times$-2.50$\times$ comparing with ZeRO, and can also improve the efficiency of complex ML systems.&#10; \item We proposed a Hierarchical Overlapping Ring (HO-Ring) communication topology for inter-node or cross-switch collective communication operations for LLM training or other scenarios. &#10; Compared with the traditional Ring, the communication efficiency of HO-Ring was increased by 36.5\%.&#10;\end{itemize}&#10;}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Information Technology, Distributed Training Strategies, Communication Efficiency Optimization, Artificial Intelligence, Mathematics, Information Science" />
        </attvalues>
      </node>
      <node id="1311.2524" label="1311.2524">
        <attvalues>
          <attvalue for="0" value="Rich feature hierarchies for accurate object detection and semantic&#10;  segmentation" />
          <attvalue for="1" value="  Object detection performance, as measured on the canonical PASCAL VOC&#10;dataset, has plateaued in the last few years. The best-performing methods are&#10;complex ensemble systems that typically combine multiple low-level image&#10;features with high-level context. In this paper, we propose a simple and&#10;scalable detection algorithm that improves mean average precision (mAP) by more&#10;than 30% relative to the previous best result on VOC 2012---achieving a mAP of&#10;53.3%. Our approach combines two key insights: (1) one can apply high-capacity&#10;convolutional neural networks (CNNs) to bottom-up region proposals in order to&#10;localize and segment objects and (2) when labeled training data is scarce,&#10;supervised pre-training for an auxiliary task, followed by domain-specific&#10;fine-tuning, yields a significant performance boost. Since we combine region&#10;proposals with CNNs, we call our method R-CNN: Regions with CNN features. We&#10;also compare R-CNN to OverFeat, a recently proposed sliding-window detector&#10;based on a similar CNN architecture. We find that R-CNN outperforms OverFeat by&#10;a large margin on the 200-class ILSVRC2013 detection dataset. Source code for&#10;the complete system is available at http://www.cs.berkeley.edu/~rbg/rcnn.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.04338" label="2401.04338">
        <attvalues>
          <attvalue for="0" value="G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale&#10;  Recommender Systems" />
          <attvalue for="1" value="  Recently, a new paradigm, meta learning, has been widely applied to Deep&#10;Learning Recommendation Models (DLRM) and significantly improves statistical&#10;performance, especially in cold-start scenarios. However, the existing systems&#10;are not tailored for meta learning based DLRM models and have critical problems&#10;regarding efficiency in distributed training in the GPU cluster. It is because&#10;the conventional deep learning pipeline is not optimized for two task-specific&#10;datasets and two update loops in meta learning. This paper provides a&#10;high-performance framework for large-scale training for Optimization-based Meta&#10;DLRM models over the \textbf{G}PU cluster, namely \textbf{G}-Meta. Firstly,&#10;G-Meta utilizes both data parallelism and model parallelism with careful&#10;orchestration regarding computation and communication efficiency, to enable&#10;high-speed distributed training. Secondly, it proposes a Meta-IO pipeline for&#10;efficient data ingestion to alleviate the I/O bottleneck. Various experimental&#10;results show that G-Meta achieves notable training speed without loss of&#10;statistical performance. Since early 2022, G-Meta has been deployed in Alipay's&#10;core advertising and recommender system, shrinking the continuous delivery of&#10;models by four times. It also obtains 6.48\% improvement in Conversion Rate&#10;(CVR) and 1.06\% increase in CPM (Cost Per Mille) in Alipay's homepage display&#10;advertising, with the benefit of larger training samples and tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.14883" label="2110.14883">
        <attvalues>
          <attvalue for="0" value="Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel&#10;  Training" />
          <attvalue for="1" value="  The success of Transformer models has pushed the deep learning model scale to&#10;billions of parameters. Due to the limited memory resource of a single GPU,&#10;However, the best practice for choosing the optimal parallel strategy is still&#10;lacking, since it requires domain expertise in both deep learning and parallel&#10;computing.&#10;  The Colossal-AI system addressed the above challenge by introducing a unified&#10;interface to scale your sequential code of model training to distributed&#10;environments. It supports parallel training methods such as data, pipeline,&#10;tensor, and sequence parallelism, as well as heterogeneous training methods&#10;integrated with zero redundancy optimizer. Compared to the baseline system,&#10;Colossal-AI can achieve up to 2.76 times training speedup on large-scale&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.05349" label="1804.05349">
        <attvalues>
          <attvalue for="0" value="Improving all-reduce collective operations for imbalanced process&#10;  arrival patterns" />
          <attvalue for="1" value="  Two new algorithms for the all-reduce operation, optimized for imbalanced&#10;process arrival patterns (PAPs) are presented: (i) sorted linear tree (SLT),&#10;(ii) pre-reduced ring (PRR) as well as a new way of on-line PAP detection,&#10;including process arrival time (PAT) estimations and their distribution between&#10;cooperating processes was introduced. The idea, pseudo-code, implementation&#10;details, benchmark for performance evaluation and a real case example for&#10;machine learning are provided. The results of the experiments were described&#10;and analyzed, showing that the proposed solution has high scalability and&#10;improved performance in comparison with the usually used ring and Rabenseifner&#10;algorithms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.04940" label="1910.04940">
        <attvalues>
          <attvalue for="0" value="Blink: Fast and Generic Collectives for Distributed ML" />
          <attvalue for="1" value="  Model parameter synchronization across GPUs introduces high overheads for&#10;data-parallel training at scale. Existing parameter synchronization protocols&#10;cannot effectively leverage available network resources in the face of ever&#10;increasing hardware heterogeneity. To address this, we propose Blink, a&#10;collective communication library that dynamically generates optimal&#10;communication primitives by packing spanning trees. We propose techniques to&#10;minimize the number of trees generated and extend Blink to leverage&#10;heterogeneous communication channels for faster data transfers. Evaluations&#10;show that compared to the state-of-the-art (NCCL), Blink can achieve up to 8x&#10;faster model synchronization, and reduce end-to-end training time for image&#10;classification tasks by up to 40%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.03975" label="2310.03975">
        <attvalues>
          <attvalue for="0" value="HuBERTopic: Enhancing Semantic Representation of HuBERT through&#10;  Self-supervision Utilizing Topic Model" />
          <attvalue for="1" value="  Recently, the usefulness of self-supervised representation learning (SSRL)&#10;methods has been confirmed in various downstream tasks. Many of these models,&#10;as exemplified by HuBERT and WavLM, use pseudo-labels generated from spectral&#10;features or the model's own representation features. From previous studies, it&#10;is known that the pseudo-labels contain semantic information. However, the&#10;masked prediction task, the learning criterion of HuBERT, focuses on local&#10;contextual information and may not make effective use of global semantic&#10;information such as speaker, theme of speech, and so on. In this paper, we&#10;propose a new approach to enrich the semantic representation of HuBERT. We&#10;apply topic model to pseudo-labels to generate a topic label for each&#10;utterance. An auxiliary topic classification task is added to HuBERT by using&#10;topic labels as teachers. This allows additional global semantic information to&#10;be incorporated in an unsupervised manner. Experimental results demonstrate&#10;that our method achieves comparable or better performance than the baseline in&#10;most tasks, including automatic speech recognition and five out of the eight&#10;SUPERB tasks. Moreover, we find that topic labels include various information&#10;about utterance, such as gender, speaker, and its theme. This highlights the&#10;effectiveness of our approach in capturing multifaceted semantic nuances.&#10;" />
          <attvalue for="2" value="&#10;In recent years, the efficacy of self-supervised representation learning (SSRL) methodologies has been empirically validated across diverse downstream tasks in the speech community \cite{yang2021superb, tsai2022superb, xuankai2021asru, peng2023study, evain21_interspeech, shi23g_interspeech, conneau22_interspeech,javed2023indicsuperb, mohamed2022self}. Prominent exemplars of these models, such as HuBERT \cite{hsu2020hubert}, Wav2Vec2.0 \cite{baevski2020wav2vec}, WavLM \cite{chen2022wavlm}, w2v-bert \cite{chung2021w2v}, and BEST-RQ \cite{chiu2022self}, predominantly rely on pseudo-labels generated from either mel-frequency cepstrum coefficients (MFCC) or the model's own representation features.&#10;HuBERT, owing to its utilization of semantic contextual information both in its learning criterion through pseudo-units and in its modeling via self-attention networks stands out as a strong self-supervised learning (SSL) mechanism. It is worth noting that semantic information plays a pivotal role in a variety of tasks, including automatic speech recognition (ASR), spoken language understanding (SLU) \cite{istaiteh2023transformer}, and more.&#10;&#10;In our previous work \cite{maekaku2023fully}, we demonstrated that we could effectively cluster a set of conversational speech based on the theme of the conversation by applying the topic model to pseudo-labels from HuBERT as input.&#10;This highlighted that the pseudo-labels inherently contain some semantic information, and&#10;this fact is also supported by the studies of \cite{polyak2021speech} and \cite{wu2023improving}.&#10;However, HuBERT's masked prediction task primarily focuses on the information surrounding the masked regions. Consequently, even if the discrete sequences inherently capture semantic information, they may not effectively harness the insights derived from the broader global context.&#10;In other words, there is the possibility that the performance of different tasks can be expected to improve by feeding some global semantic information explicitly back to HuBERT.&#10;&#10;In this paper, we propose HuBERTopic, which is a novel approach to enrich the semantic representation of HuBERT.&#10;We apply the topic model to pseudo-labels to generate a topic label for each utterance. &#10;To capture the global semantic information of the entire speech, a CLS vector is given at the beginning of the input to the Transformer encoder of HuBERT, and an auxiliary topic classification task is added by using topic labels as teachers.&#10;This allows additional global semantic information that is expected to be useful in improving the performance of downstream tasks to be incorporated in an unsupervised manner.&#10;&#10;There have been several efforts to improve the performance of downstream tasks by applying additional unsupervised methods to HuBERT. ContentVec \cite{qian2022contentvec} and Spin \cite{chang2023self} are self-supervised methods that disentangle speaker information to impose a speaker-invariant constraint on a pre-trained HuBERT. Shi et al. utilized unsupervised ASR to bridge speech SSL with text-pre-trained models in order to take benefits from semantic information from the textual representations \cite{shi2023bridging}.&#10;These methods have been successful in a variety of downstream tasks, but our method differs in that HuBERTopic does not focus on specific attributes such as speaker and phoneme, and it trains to enrich the global semantic information during pre-training.&#10;Moreover, in the field of SSL, methodologies leveraging some information obtained from the topic model as the ground truth labels have not been previously explored. Within the domain of Natural Language Processing (NLP), various approaches have been proposed that enhance the topic estimation of Latent Dirichlet Allocation (LDA) \cite{blei2003latent} through a semantic filter based on BERT \cite{kenton2019bert} \cite{venugopalan2022enhanced}, or attempt to estimate topic labels using BERT and clustering technique \cite{grootendorst2022bertopic}. However, it is worth noting that, to the best of our knowledge, the utilization of topic information as explicit teacher labels has not been proposed to date.&#10;&#10;Our experimental results demonstrate that our method outperformed the baseline performance on most tasks, including ASR task and eight SUPERB tasks.&#10;Moreover, we find that pseudo-labels include various information about the utterance, such as gender and speaker, through a topic model.&#10;This highlights the effectiveness of our approach in capturing multifaceted semantic nuances, ultimately leading to improved performance in downstream tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Semantic Representation, Linguistics, Artificial Intelligence, Signal Processing, Self-Supervised Learning, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2211.03025" label="2211.03025">
        <attvalues>
          <attvalue for="0" value="Bridging Speech and Textual Pre-trained Models with Unsupervised ASR" />
          <attvalue for="1" value="  Spoken language understanding (SLU) is a task aiming to extract high-level&#10;semantics from spoken utterances. Previous works have investigated the use of&#10;speech self-supervised models and textual pre-trained models, which have shown&#10;reasonable improvements to various SLU tasks. However, because of the&#10;mismatched modalities between speech signals and text tokens, previous methods&#10;usually need complex designs of the frameworks. This work proposes a simple yet&#10;efficient unsupervised paradigm that connects speech and textual pre-trained&#10;models, resulting in an unsupervised speech-to-semantic pre-trained model for&#10;various tasks in SLU. To be specific, we propose to use unsupervised automatic&#10;speech recognition (ASR) as a connector that bridges different modalities used&#10;in speech and textual pre-trained models. Our experiments show that&#10;unsupervised ASR itself can improve the representations from speech&#10;self-supervised models. More importantly, it is shown as an efficient connector&#10;between speech and textual pre-trained models, improving the performances of&#10;five different SLU tasks. Notably, on spoken question answering, we reach the&#10;state-of-the-art result over the challenging NMSQA benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.01083" label="2304.01083">
        <attvalues>
          <attvalue for="0" value="Can the Inference Logic of Large Language Models be Disentangled into&#10;  Symbolic Concepts?" />
          <attvalue for="1" value="  In this paper, we explain the inference logic of large language models (LLMs)&#10;as a set of symbolic concepts. Many recent studies have discovered that&#10;traditional DNNs usually encode sparse symbolic concepts. However, because an&#10;LLM has much more parameters than traditional DNNs, whether the LLM also&#10;encodes sparse symbolic concepts is still an open problem. Therefore, in this&#10;paper, we propose to disentangle the inference score of LLMs for dialogue tasks&#10;into a small number of symbolic concepts. We verify that we can use those&#10;sparse concepts to well estimate all inference scores of the LLM on all&#10;arbitrarily masking states of the input sentence. We also evaluate the&#10;transferability of concepts encoded by an LLM and verify that symbolic concepts&#10;usually exhibit high transferability across similar input sentences. More&#10;crucially, those symbolic concepts can be used to explain the exact reasons&#10;accountable for the LLM's prediction errors.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, large language models (LLMs), e.g., ChatGPT \cite{chatgpt} and GPT-3 \cite{brown2020language}, have exhibited remarkable performance. Although LLMs have been used to guide people in important tasks, such as writing papers, analyzing reports, and consulting information, the inference logic of LLMs still cannot be fully understood by people.&#10;&#10;Therefore, generating an accountable explanation for the LLM's inference score is critical for high-stake decisions. Generally speaking, an accountable explanation needs to satisfy three requirements.&#10;\\&#10;$\bullet$ Countable inference patterns.&#10;Transforming the implicit inference logic of an LLM into explicit and countable inference patterns is crucial to explain the LLM. Specifically, it is found that the inference score of a deep neural network (DNN) on the input sentence can be disentangled into the effects of a small number of inference patterns (e.g., certain phrases) \cite{ren2023defining,li2023does}. These inference patterns can be considered as concepts memorized by the LLM, and they are supposed to universally explain inference scores on various sentences.&#10;\\&#10;$\bullet$ Verified in practice.&#10;When the inference logic of an LLM is explained as a set of inference patterns, the faithfulness or accuracy of the explanation must be both guaranteed by theory and verified in experiments.&#10;\\&#10;$\bullet$ Accountable for errors.&#10;When an LLM makes an incorrect prediction, it is important to clarify the exact reasons accountable for the error with a clear mathematical connection between the reason and the incorrect prediction, rather than provide a specious explanation.&#10;&#10;Definition of concepts in DNNs. Considering the above three requirements, Ren et al. \cite{ren2023defining} have tried to explain the inference logic of a neural network as the detection of salient interactions between input variables. Specifically, they find that the network's inference score can be well mimicked by a small number of salient interactions between input variables. Given an input sentence with $n$ words indexed by $N = \{1, 2,..., n\}$, the DNN does not directly use a single input word for inference. Instead, the DNN encodes the interactions between different words as inference patterns (or concepts) to generate an inference score $v$. For example, in the classification task, given the input sentence ``he is a green hand,'' the interaction between the words $S=\{green,hand\} \subseteq N$ contributes a specific interaction effect $I(S)$, which pushes the network's inference score towards inferring ``he'' to the meaning ``beginner.'' Masking any words in $S$ will remove the interaction effect, i.e., making $I(S)=0$. &#10;&#10;It has been discovered that among all $2^n$ interactions, a DNN usually only encodes a small number of salient interactions, and all other interactions have ignorable effects $I(S)\approx 0$ \cite{ren2023defining,li2023does}. In this way, we can consider these salient interactions as concepts (i.e., countable inference patterns), and consider the rest interactions with ignorable effects as noisy patterns. Inspired by this, the inference logic of a DNN can be explained as the detection of a small number of concepts. In other words, we can use such a few salient interaction concepts to well estimate inference scores of the DNN on an exponential number of all randomly masked sentences.&#10;More crucially, when a DNN makes incorrect predictions, the clear disentanglement of interaction concepts from the DNN allows us to identify the exact interaction concepts encoded by the DNN that are accountable for errors. In addition, those concepts have also exhibited high transferability across different samples and different DNNs, and demonstrated a considerable discrimination power in classification tasks \cite{li2023does}.&#10;&#10;Explaining LLMs using concepts. However, all above findings of using sparse concepts to mimic the logic of a traditional DNN are obtained based on empirical observations in experiments without solid proof. On the other hand, the LLM usually has much more parameters than traditional DNNs, and the LLM is trained on much more data. Therefore, whether an LLM can be explained as the detection of a small number of concepts is still unknown.&#10;&#10;Therefore, in this paper, we aim to disentangle the inference score of LLMs for dialogue into a small number of salient interaction concepts. Given an input sentence, e.g., a question or an unfinished sentence, the LLM sequentially generates words to answer the question or to complete the sentence. Considering the generation of the next word, the LLM takes the previous $n$ words as the input, and then it estimates the probability distribution of generating the $(n+1)$-th word. The LLM samples the word with the highest probability as the target word $y^{truth}$. Therefore, we set $v(\boldsymbol{x})=\log p(y=y^{truth} | \boldsymbol{x}) / (1-p(y=y^{truth} | \boldsymbol{x}))\in\mathbb{R}$ as the inference score of the LLM, where $p(y=y^{truth} | \boldsymbol{x})$ denotes the probalibity of generating the target $(n+1)$-th word $y^{truth}$. Specifically, we obtain the following three conclusions.&#10;\\&#10;1. We have verified that given an input sentence, we can use a small number of concepts to accurately estimate all inference scores of the LLM on all arbitrarily masking states of the input sentence. Given a well-trained LLM for dialogue, we quantify the interactions between different words in the input sentence, and verify that the output score of generating a specific word can be well estimated using a few interaction concepts. Moreover, given an input sentence where some words are randomly masked, we can still use those interaction concepts to well estimate the output score of generating the next word.&#10;\\&#10;2. We have verified that interaction concepts encoded by the LLM exhibit high transferability across similar input sentences. In other words, given similar input sentences, the LLM usually uses similar interaction concepts to generate the same word. &#10;\\&#10;3. We have verified that the disentangled interaction concepts can help people identify the exact reasons for the prediction error made by the LLM. When the LLM generates a wrong word that violated the facts, we find out those interaction concepts that are accountable for the wrong word.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Symbolic Concept Encoding, Computer Science, Language Model Analysis, Linguistics, Artificial Intelligence, Mathematics, Natural Language Processing, Deep Learning Explainability" />
        </attvalues>
      </node>
      <node id="2111.06206" label="2111.06206">
        <attvalues>
          <attvalue for="0" value="Defining and Quantifying the Emergence of Sparse Concepts in DNNs" />
          <attvalue for="1" value="  This paper aims to illustrate the concept-emerging phenomenon in a trained&#10;DNN. Specifically, we find that the inference score of a DNN can be&#10;disentangled into the effects of a few interactive concepts. These concepts can&#10;be understood as causal patterns in a sparse, symbolic causal graph, which&#10;explains the DNN. The faithfulness of using such a causal graph to explain the&#10;DNN is theoretically guaranteed, because we prove that the causal graph can&#10;well mimic the DNN's outputs on an exponential number of different masked&#10;samples. Besides, such a causal graph can be further simplified and re-written&#10;as an And-Or graph (AOG), without losing much explanation accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.01818" label="2208.01818">
        <attvalues>
          <attvalue for="0" value="VQ-T: RNN Transducers using Vector-Quantized Prediction Network States" />
          <attvalue for="1" value="  Beam search, which is the dominant ASR decoding algorithm for end-to-end&#10;models, generates tree-structured hypotheses. However, recent studies have&#10;shown that decoding with hypothesis merging can achieve a more efficient search&#10;with comparable or better performance. But, the full context in recurrent&#10;networks is not compatible with hypothesis merging. We propose to use&#10;vector-quantized long short-term memory units (VQ-LSTM) in the prediction&#10;network of RNN transducers. By training the discrete representation jointly&#10;with the ASR network, hypotheses can be actively merged for lattice generation.&#10;Our experiments on the Switchboard corpus show that the proposed VQ RNN&#10;transducers improve ASR performance over transducers with regular prediction&#10;networks while also producing denser lattices with a very low oracle word error&#10;rate (WER) for the same beam size. Additional language model rescoring&#10;experiments also demonstrate the effectiveness of the proposed lattice&#10;generation scheme.&#10;" />
          <attvalue for="2" value="&#10;&#10;End-to-end automatic speech recognition (ASR) has been a hot research direction. An end-to-end system directly maps acoustic features to linguistic units, reducing the workload of combining different knowledge sources. At the same time, it has shown equivalent or better performance over conventional hidden Markov model (HMM) based approaches \cite{chiu2018state, pham2019very, guo2021recent}. According to recent literature, we have observed two mainstream modeling directions for end-to-end ASR: attention-based encoder-decoder (AED) \cite{chorowski2015attention, chan2016listen, watanabe2017hybrid}, and RNN transducer (RNN-T) \cite{graves2012sequence, rao2017exploring, saon2021advancing}.&#10;&#10;Beam search has been the main decoding strategy for both HMM-based approaches and end-to-end systems. Using a limited beam size $\mathcal{H}$, the search generates an $N$-best list with a computational cost of $\mathcal{O}(\mathcal{H})$. However, $N$-best lists are insufficient for some downstream applications or post-processing stages, such as&#10;\begin{inparaenum}[(1)]&#10;\item language model rescoring~\cite{ljolje1999efficient, sak2010fly, rybach2017lattice};&#10;\item downstream processing of ASR output (e.g., translation\cite{kumar2014some} and keyword spotting \cite{kingsbury2013high, rosenberg2017end});&#10;\item confusion network generation~\cite{mangu2000finding, hakkani2006beyond}; and &#10;\item sequence discriminative training \cite{normandin1996maximum, povey2002, gibson2006hypothesis}.&#10;\end{inparaenum}&#10;&#10;Hypothesis merging (or path recombination) is the main difference between regular beam search and lattice-based search. For conventional methods, Rybach et al. divided merging procedures into two categories \cite{rybach2017lattice}: the phone-pair approach \cite{ljolje1999efficient, mohri2008speech} and the $N$-best history approach \cite{saon2005anatomy, chen2006advances, soltau2009dynamic}. However, neither are suitable for end-to-end models because they require additional n-gram language models and HMM-based acoustic models to generate lattices. In order to generate lattices for end-to-end ASR models, recent works proposed approaches that restrict the hypothesis merging context \cite{zapotoczny19, prabhavalkar2021less} with convolutional and recurrent decoders. Zapotoczny et al. \cite{zapotoczny19} replaced the recurrent decoder with a temporal convolutional network (TCN) in AED networks. Since there is a fixed-length context, this method can merge hypotheses as efficiently as with an n-gram language model. Their experiments on the WSJ dataset indicate that character-based lattice decoding can consistently outperform beam decoding in terms of WER for a similar beam size. Prabhavalkar et al. \cite{prabhavalkar2021less} further investigated hypothesis merging for recurrent neural network transducer (RNN-T) with a full-context prediction network. They approximately merge hypotheses with the same last few byte pair encoding (BPE) labels, showing improvements in both 1-best and oracle WER. &#10;&#10;This work proposes the use of vector-quantized long short-term memory units (VQ-LSTM) to generate lattices for RNN-Ts. The VQ layers are added to the LSTM cells and are jointly optimized with the transducer objective. We show that, with VQ-LSTM, we can actively merge hypotheses for lattice generation while conditioning on the full context. We investigate both systems in \cite{zapotoczny19, prabhavalkar2021less}, and provide a comparison for different types of encoders (LSTM and conformer) and linguistic units (characters and BPE units). Experiments on the Switchboard 300 hours corpus show that our method improves ASR performance on multiple test sets despite using compressed context information. The method also results in denser lattices with a significantly lower oracle WER. The generated lattices from VQ-based systems show improvements for language model rescoring as a downstream task.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Decoding, Computer Science, Linguistics, Neural Network Optimization, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2307.09435" label="2307.09435">
        <attvalues>
          <attvalue for="0" value="SLMGAN: Exploiting Speech Language Model Representations for&#10;  Unsupervised Zero-Shot Voice Conversion in GANs" />
          <attvalue for="1" value="  In recent years, large-scale pre-trained speech language models (SLMs) have&#10;demonstrated remarkable advancements in various generative speech modeling&#10;applications, such as text-to-speech synthesis, voice conversion, and speech&#10;enhancement. These applications typically involve mapping text or speech inputs&#10;to pre-trained SLM representations, from which target speech is decoded. This&#10;paper introduces a new approach, SLMGAN, to leverage SLM representations for&#10;discriminative tasks within the generative adversarial network (GAN) framework,&#10;specifically for voice conversion. Building upon StarGANv2-VC, we add our novel&#10;SLM-based WavLM discriminators on top of the mel-based discriminators along&#10;with our newly designed SLM feature matching loss function, resulting in an&#10;unsupervised zero-shot voice conversion system that does not require text&#10;labels during training. Subjective evaluation results show that SLMGAN&#10;outperforms existing state-of-the-art zero-shot voice conversion models in&#10;terms of naturalness and achieves comparable similarity, highlighting the&#10;potential of SLM-based discriminators for related applications.&#10;" />
          <attvalue for="2" value="&#10;&#10;Voice conversion (VC), a technique of converting one speaker's voice to another speaker's voice, has gained increasing attention in recent years due to its numerous applications, such as personalized text-to-speech synthesis, speaker anonymization, and entertainment \cite{sisman2020overview}. One type of VC, zero-shot voice conversion, also known as any-to-any voice conversion, has become particularly popular. It aims to convert a source speaker's voice to a target without paired training data and without restricting source and target speakers already seen during training \cite{walczyna2023overview}. There are mainly two approaches to achieving zero-shot voice conversion: reconstruction-based methods and GAN-based methods. &#10;&#10;Reconstruction-based methods focus on disentangling speaker information from linguistic information in the latent representation of speech. There are several approaches in this category, including autoencoder-based approaches \cite{qian2019autovc, yuan2021improving, wang2021vqmivc, chen2021again, lian2022robust}, where input speech is mapped to a bottleneck representation with much smaller dimensions than the input and decoded back to that of the target speaker; ASR-based and TTS-based approaches \cite{casanova2022yourtts, levkovitch2022zero, li2023styletts, hussain2023ace}, where input speech is mapped to the same latent representation as the phoneme representation from the text; and large pre-trained speech language model (SLM) approaches \cite{choi2021neural, qian2022contentvec, dang2022training}, where speech is directly reconstructed from SLM representations in deep layers that contain less speaker information. SLM approaches are particularly popular due to their supreme performance from large pre-training. While these approaches can be effective, they do not guarantee perfect disentanglement of speaker and linguistic information, resulting in unnatural speech or residual source speaker information in the converted speech. Moreover, SLM approaches are often slow in inference because of the sheer amount of parameters in the large SLMs, making them unappealing for real-time applications. &#10;&#10;GAN-based methods \cite{zhang2020gazev, nguyen2022nvc, takahashi2022robust}, on the other hand, employ a discriminator to determine whether the converted voice is from the target speaker or not. This approach does not necessarily disentangle speaker information from linguistic information, as it relies on the discriminator's ability to capture the human perception of speaker identity. GAN-based methods typically produce more natural speech \cite{levkovitch2022zero, yasur2023deepfake}, as the latent representations are not forced to be disentangled, preserving more linguistic information. However, the similarity of converted speech relies heavily on the discriminative power of the discriminators, yet designing a discriminator that effectively captures each target speaker's speaking style is challenging, especially when the number of speakers in the dataset is large. &#10;&#10;Our recent work has demonstrated promising results by employing SLMs as discriminators for text-to-speech (TTS), where we show that leveraging SLMs as discriminators improves the naturalness of synthesized speech, specifically from paralinguistic aspects \cite{li2023styletts2}. Still, relevant applications in cross-domain transfer, such as voice conversion, have not been examined. In this work, we introduce a novel method to improve the performance of discriminators using speech language models in GAN-based zero-shot voice conversion models involving a large number of speakers. Our approach extends the state-of-the-art StarGANv2-VC \cite{li2021starganv2} model by incorporating a set of SLM-based discriminators in addition to the existing mel-based discriminators. We employ a neural vocoder BIGVGAN \cite{lee2022bigvgan} to convert the generated mel-spectrograms into waveforms, which are then fed into the WavLM \cite{chen2022wavlm} encoder to obtain the SLM representations of the converted speech. Following this, we train discriminators that consist of simple convolutional neural networks (CNN) on these SLM representations to differentiate between the real and fake samples and classify the source speakers. &#10;&#10;Our model retains the advantages of the original StarGANv2-VC model, such as eliminating the need for text labels, enabling potential real-time applications, and exhibiting greater adaptability to other voice conversion tasks like cross-lingual conversion, while benefiting from rich representations learned by self-supervised training in large SLMs like SLM-based approaches do. Additionally, unlike the vanilla StarGANv2-VC, our model scales to a larger number of speakers without any performance loss. Subjective evaluations show that our model surpasses two baseline models, VQMIVIC \cite{wang2021vqmivc} and AGAIN-VC \cite{chen2021again}, in terms of both naturalness and similarity. Furthermore, it outperforms YourTTS \cite{casanova2022yourtts} and StyleTTS-VC \cite{li2023styletts} in terms of naturalness, two of the best-performing publicly available models for zero-shot voice conversion that rely on text labels, albeit no text labels are required during training. The audio samples can be found at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Speech Language Models, Generative Adversarial Networks, Signal Processing, Voice Conversion Systems" />
        </attvalues>
      </node>
      <node id="2010.12788" label="2010.12788">
        <attvalues>
          <attvalue for="0" value="GAZEV: GAN-Based Zero-Shot Voice Conversion over Non-parallel Speech&#10;  Corpus" />
          <attvalue for="1" value="  Non-parallel many-to-many voice conversion is recently attract-ing huge&#10;research efforts in the speech processing community. A voice conversion system&#10;transforms an utterance of a source speaker to another utterance of a target&#10;speaker by keeping the content in the original utterance and replacing by the&#10;vocal features from the target speaker. Existing solutions, e.g., StarGAN-VC2,&#10;present promising results, only when speech corpus of the engaged speakers is&#10;available during model training. AUTOVCis able to perform voice conversion on&#10;unseen speakers, but it needs an external pretrained speaker verification&#10;model. In this paper, we present our new GAN-based zero-shot voice conversion&#10;solution, called GAZEV, which targets to support unseen speakers on both source&#10;and target utterances. Our key technical contribution is the adoption of&#10;speaker embedding loss on top of the GAN framework, as well as adaptive&#10;instance normalization strategy, in order to address the limitations of speaker&#10;identity transfer in existing solutions. Our empirical evaluations demonstrate&#10;significant performance improvement on output speech quality and comparable&#10;speaker similarity to AUTOVC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.07691" label="2306.07691">
        <attvalues>
          <attvalue for="0" value="StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion&#10;  and Adversarial Training with Large Speech Language Models" />
          <attvalue for="1" value="  In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that&#10;leverages style diffusion and adversarial training with large speech language&#10;models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its&#10;predecessor by modeling styles as a latent random variable through diffusion&#10;models to generate the most suitable style for the text without requiring&#10;reference speech, achieving efficient latent diffusion while benefiting from&#10;the diverse speech synthesis offered by diffusion models. Furthermore, we&#10;employ large pre-trained SLMs, such as WavLM, as discriminators with our novel&#10;differentiable duration modeling for end-to-end training, resulting in improved&#10;speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker&#10;LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by&#10;native English speakers. Moreover, when trained on the LibriTTS dataset, our&#10;model outperforms previous publicly available models for zero-shot speaker&#10;adaptation. This work achieves the first human-level TTS on both single and&#10;multispeaker datasets, showcasing the potential of style diffusion and&#10;adversarial training with large SLMs. The audio demos and source code are&#10;available at https://styletts2.github.io/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.04658" label="2206.04658">
        <attvalues>
          <attvalue for="0" value="BigVGAN: A Universal Neural Vocoder with Large-Scale Training" />
          <attvalue for="1" value="  Despite recent progress in generative adversarial network (GAN)-based&#10;vocoders, where the model generates raw waveform conditioned on acoustic&#10;features, it is challenging to synthesize high-fidelity audio for numerous&#10;speakers across various recording environments. In this work, we present&#10;BigVGAN, a universal vocoder that generalizes well for various&#10;out-of-distribution scenarios without fine-tuning. We introduce periodic&#10;activation function and anti-aliased representation into the GAN generator,&#10;which brings the desired inductive bias for audio synthesis and significantly&#10;improves audio quality. In addition, we train our GAN vocoder at the largest&#10;scale up to 112M parameters, which is unprecedented in the literature. We&#10;identify and address the failure modes in large-scale GAN training for audio,&#10;while maintaining high-fidelity output without over-regularization. Our&#10;BigVGAN, trained only on clean speech (LibriTTS), achieves the state-of-the-art&#10;performance for various zero-shot (out-of-distribution) conditions, including&#10;unseen speakers, languages, recording environments, singing voices, music, and&#10;instrumental audio. We release our code and model at:&#10;https://github.com/NVIDIA/BigVGAN&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.11856" label="2010.11856">
        <attvalues>
          <attvalue for="0" value="XOR QA: Cross-lingual Open-Retrieval Question Answering" />
          <attvalue for="1" value="  Multilingual question answering tasks typically assume answers exist in the&#10;same language as the question. Yet in practice, many languages face both&#10;information scarcity -- where languages have few reference articles -- and&#10;information asymmetry -- where questions reference concepts from other&#10;cultures. This work extends open-retrieval question answering to a&#10;cross-lingual setting enabling questions from one language to be answered via&#10;answer content from another language. We construct a large-scale dataset built&#10;on questions from TyDi QA lacking same-language answers. Our task formulation,&#10;called Cross-lingual Open Retrieval Question Answering (XOR QA), includes 40k&#10;information-seeking questions from across 7 diverse non-English languages.&#10;Based on this dataset, we introduce three new tasks that involve cross-lingual&#10;document retrieval using multi-lingual and English resources. We establish&#10;baselines with state-of-the-art machine translation systems and cross-lingual&#10;pretrained models. Experimental results suggest that XOR QA is a challenging&#10;task that will facilitate the development of novel techniques for multilingual&#10;question answering. Our data and code are available at&#10;https://nlp.cs.washington.edu/xorqa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1609.08210" label="1609.08210">
        <attvalues>
          <attvalue for="0" value="Learning to Translate for Multilingual Question Answering" />
          <attvalue for="1" value="  In multilingual question answering, either the question needs to be&#10;translated into the document language, or vice versa. In addition to direction,&#10;there are multiple methods to perform the translation, four of which we explore&#10;in this paper: word-based, 10-best, context-based, and grammar-based. We build&#10;a feature for each combination of translation direction and method, and train a&#10;model that learns optimal feature weights. On a large forum dataset consisting&#10;of posts in English, Arabic, and Chinese, our novel learn-to-translate approach&#10;was more effective than a strong baseline (p&lt;0.05): translating all text into&#10;English, then training a classifier based only on English (original or&#10;translated) text.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1510.01032" label="1510.01032">
        <attvalues>
          <attvalue for="0" value="Deep convolutional acoustic word embeddings using word-pair side&#10;  information" />
          <attvalue for="1" value="  Recent studies have been revisiting whole words as the basic modelling unit&#10;in speech recognition and query applications, instead of phonetic units. Such&#10;whole-word segmental systems rely on a function that maps a variable-length&#10;speech segment to a vector in a fixed-dimensional space; the resulting acoustic&#10;word embeddings need to allow for accurate discrimination between different&#10;word types, directly in the embedding space. We compare several old and new&#10;approaches in a word discrimination task. Our best approach uses side&#10;information in the form of known word pairs to train a Siamese convolutional&#10;neural network (CNN): a pair of tied networks that take two speech segments as&#10;input and produce their embeddings, trained with a hinge loss that separates&#10;same-word pairs and different-word pairs by some margin. A word classifier CNN&#10;performs similarly, but requires much stronger supervision. Both types of CNNs&#10;yield large improvements over the best previously published results on the word&#10;discrimination task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.14007" label="2006.14007">
        <attvalues>
          <attvalue for="0" value="Multilingual Jointly Trained Acoustic and Written Word Embeddings" />
          <attvalue for="1" value="  Acoustic word embeddings (AWEs) are vector representations of spoken word&#10;segments. AWEs can be learned jointly with embeddings of character sequences,&#10;to generate phonetically meaningful embeddings of written words, or&#10;acoustically grounded word embeddings (AGWEs). Such embeddings have been used&#10;to improve speech retrieval, recognition, and spoken term discovery. In this&#10;work, we extend this idea to multiple low-resource languages. We jointly train&#10;an AWE model and an AGWE model, using phonetically transcribed data from&#10;multiple languages. The pre-trained models can then be used for unseen&#10;zero-resource languages, or fine-tuned on data from low-resource languages. We&#10;also investigate distinctive features, as an alternative to phone labels, to&#10;better share cross-lingual information. We test our models on word&#10;discrimination tasks for twelve languages. When trained on eleven languages and&#10;tested on the remaining unseen language, our model outperforms traditional&#10;unsupervised approaches like dynamic time warping. After fine-tuning the&#10;pre-trained models on one hour or even ten minutes of data from a new language,&#10;performance is typically much better than training on only the target-language&#10;data. We also find that phonetic supervision improves performance over&#10;character sequences, and that distinctive feature supervision is helpful in&#10;handling unseen phones in the target language.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.08135" label="1703.08135">
        <attvalues>
          <attvalue for="0" value="An embedded segmental K-means model for unsupervised segmentation and&#10;  clustering of speech" />
          <attvalue for="1" value="  Unsupervised segmentation and clustering of unlabelled speech are core&#10;problems in zero-resource speech processing. Most approaches lie at&#10;methodological extremes: some use probabilistic Bayesian models with&#10;convergence guarantees, while others opt for more efficient heuristic&#10;techniques. Despite competitive performance in previous work, the full Bayesian&#10;approach is difficult to scale to large speech corpora. We introduce an&#10;approximation to a recent Bayesian model that still has a clear objective&#10;function but improves efficiency by using hard clustering and segmentation&#10;rather than full Bayesian inference. Like its Bayesian counterpart, this&#10;embedded segmental K-means model (ES-KMeans) represents arbitrary-length word&#10;segments as fixed-dimensional acoustic word embeddings. We first compare&#10;ES-KMeans to previous approaches on common English and Xitsonga data sets (5&#10;and 2.5 hours of speech): ES-KMeans outperforms a leading heuristic method in&#10;word segmentation, giving similar scores to the Bayesian model while being 5&#10;times faster with fewer hyperparameters. However, its clusters are less pure&#10;than those of the other models. We then show that ES-KMeans scales to larger&#10;corpora by applying it to the 5 languages of the Zero Resource Speech Challenge&#10;2017 (up to 45 hours), where it performs competitively compared to the&#10;challenge baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.10731" label="2103.10731">
        <attvalues>
          <attvalue for="0" value="Acoustic word embeddings for zero-resource languages using&#10;  self-supervised contrastive learning and multilingual adaptation" />
          <attvalue for="1" value="  Acoustic word embeddings (AWEs) are fixed-dimensional representations of&#10;variable-length speech segments. For zero-resource languages where labelled&#10;data is not available, one AWE approach is to use unsupervised&#10;autoencoder-based recurrent models. Another recent approach is to use&#10;multilingual transfer: a supervised AWE model is trained on several&#10;well-resourced languages and then applied to an unseen zero-resource language.&#10;We consider how a recent contrastive learning loss can be used in both the&#10;purely unsupervised and multilingual transfer settings. Firstly, we show that&#10;terms from an unsupervised term discovery system can be used for contrastive&#10;self-supervision, resulting in improvements over previous unsupervised&#10;monolingual AWE models. Secondly, we consider how multilingual AWE models can&#10;be adapted to a specific zero-resource language using discovered terms. We find&#10;that self-supervised contrastive adaptation outperforms adapted multilingual&#10;correspondence autoencoder and Siamese AWE models, giving the best overall&#10;results in a word discrimination task on six zero-resource languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.02221" label="2012.02221">
        <attvalues>
          <attvalue for="0" value="A Correspondence Variational Autoencoder for Unsupervised Acoustic Word&#10;  Embeddings" />
          <attvalue for="1" value="  We propose a new unsupervised model for mapping a variable-duration speech&#10;segment to a fixed-dimensional representation. The resulting acoustic word&#10;embeddings can form the basis of search, discovery, and indexing systems for&#10;low- and zero-resource languages. Our model, which we refer to as a maximal&#10;sampling correspondence variational autoencoder (MCVAE), is a recurrent neural&#10;network (RNN) trained with a novel self-supervised correspondence loss that&#10;encourages consistency between embeddings of different instances of the same&#10;word. Our training scheme improves on previous correspondence training&#10;approaches through the use and comparison of multiple samples from the&#10;approximate posterior distribution. In the zero-resource setting, the MCVAE can&#10;be trained in an unsupervised way, without any ground-truth word pairs, by&#10;using the word-like segments discovered via an unsupervised term discovery&#10;system. In both this setting and a semi-supervised low-resource setting (with a&#10;limited set of ground-truth word pairs), the MCVAE outperforms previous&#10;state-of-the-art models, such as Siamese-, CAE- and VAE-based RNNs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13408" label="2305.13408">
        <attvalues>
          <attvalue for="0" value="Modular Domain Adaptation for Conformer-Based Streaming ASR" />
          <attvalue for="1" value="  Speech data from different domains has distinct acoustic and linguistic&#10;characteristics. It is common to train a single multidomain model such as a&#10;Conformer transducer for speech recognition on a mixture of data from all&#10;domains. However, changing data in one domain or adding a new domain would&#10;require the multidomain model to be retrained. To this end, we propose a&#10;framework called modular domain adaptation (MDA) that enables a single model to&#10;process multidomain data while keeping all parameters domain-specific, i.e.,&#10;each parameter is only trained by data from one domain. On a streaming&#10;Conformer transducer trained only on video caption data, experimental results&#10;show that an MDA-based model can reach similar performance as the multidomain&#10;model on other domains such as voice search and dictation by adding per-domain&#10;adapters and per-domain feed-forward networks in the Conformer encoder.&#10;" />
          <attvalue for="2" value="&#10;Automatic speech recognition (ASR) systems have been used across various applications, such as video captioning~\cite{Liao2013LargeSD}, dictation~\cite{Li2015LSTMTA}, voice search~\cite{Wang2008AnIT,Shan2017AttentionBasedES}, voice assistant~\cite{Kepuska2018NextgenerationOV} and telephony~\cite{Hain2000THECM,Xiong2017TheM2,Tuske2021OnTL}. Each domain has its own acoustic and linguistic characteristics.&#10;For example, video captioning data has diverse acoustic environments and speaking styles; a dictation utterance is likely to have little background noise; voice search data tends to have short queries with named entities.&#10;&#10;The performance of ASR models deteriorates significantly when the model is trained on a particular domain but evaluated on another~\cite{Bell2020AdaptationAF}. Given sufficient model capacity and training data, it is desirable to build a single ASR model to serve all application domains. To ensure the model performs well for all domains, one simple and effective approach is to mix all data during training to obtain a multidomain model~\cite{Narayanan2018TowardDS,Chan2021SpeechStewSM}. Since all model parameters are shared across all domains, the multidomain approach has some shortcomings. First, the entire model needs to be retrained if the training data from a certain domain changes or a new domain is added. Second, finding the right balance to mix data from various domains is nontrivial. On the other end of the spectrum, it is also possible to build an ASR model for each domain. This means all parameters are domain-specific where each parameter is trained on data from a single domain. However, this is very expensive for training and serving as multiple models need to be maintained. Therefore, it is ideal to build a single model that serves all domains while keeping all model parameters domain-specific. &#10;&#10;The contribution of this work is as follows. First, we proposed a framework called modular domain adaptation (MDA) that can process multidomain data with all parameters being domain-specific. As illustrated in Fig.~\ref{fig:mda}, the backbone model is trained on a particular domain (in red), and all traffic passes through the common backbone model except certain parts of the model where the traffic is split and routed through the per-domain parameters. Second, we identified the most effective components of the Conformer backbone model to integrate per-domain parameters for parameter efficiency. Also, two types of per-domain parameters are explored, i.e., per-domain components that replace the existing components in the backbone model and per-domain adapters that add lightweight adapter modules to modify intermediate representations. Third, the final recipe was validated on three different domains with a large amount of data and minimum word error rate (MWER) training~\cite{Prabhavalkar2017MinimumWE}. The MDA-based model achieved similar performance as the multidomain model across all domains with 0.2--0.4\% absolute degradation in word error rates (WERs). The number of per-domain parameters is 22\% of the backbone model.&#10;&#10;Although domain adaptation has been widely studied for ASR~\cite{Bell2020AdaptationAF} with various paradigms including input feature adaptation~\cite{Gales1998MaximumLL,Fainberg2017FactorisedRF,Sainath2020ASO}, model-based adaptation~\cite{Li2017LargeScaleDA,Asami2017DomainAO,Manohar2018ATL,Samarakoon2018DomainAO,Sim2018DomainAU} and multi-task learning~\cite{Denisov2018UnsupervisedDA,Meng2018AdversarialTL}, our work pays particular attention to modularity. Unlike general adaptation techniques, MDA satisfies the constraint that all model parameters are domain-specific. This brings many advantages of modularity~\cite{pfeiffer2023modular}: similar functions of the ASR model are encoded with the same module while allocating distinct functions to per-domain parameters; per-domain parameters can be constructed separately and updated locally; parameter efficiency is much higher than finetuning the entire model or have multiple single-domain models.&#10;&#10;In this paper, two methods under the framework of MDA are described in Sec.~\ref{sec:methods}. Extensive experimental studies are then conducted in Sec.~\ref{sec:setup}~\&amp;~\ref{sec:exp}, which offer insights into the Conformer transducers, especially in terms of modularity. Conclusions are drawn in Sec.~\ref{sec:conclusions}.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Multidomain Modeling, Domain Adaptation, Signal Processing, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1707.07167" label="1707.07167">
        <attvalues>
          <attvalue for="0" value="Attention-Based End-to-End Speech Recognition on Voice Search" />
          <attvalue for="1" value="  Recently, there has been a growing interest in end-to-end speech recognition&#10;that directly transcribes speech to text without any predefined alignments. In&#10;this paper, we explore the use of attention-based encoder-decoder model for&#10;Mandarin speech recognition on a voice search task. Previous attempts have&#10;shown that applying attention-based encoder-decoder to Mandarin speech&#10;recognition was quite difficult due to the logographic orthography of Mandarin,&#10;the large vocabulary and the conditional dependency of the attention model. In&#10;this paper, we use character embedding to deal with the large vocabulary.&#10;Several tricks are used for effective model training, including L2&#10;regularization, Gaussian weight noise and frame skipping. We compare two&#10;attention mechanisms and use attention smoothing to cover long context in the&#10;attention model. Taken together, these tricks allow us to finally achieve a&#10;character error rate (CER) of 3.58% and a sentence error rate (SER) of 7.43% on&#10;the MiTV voice search dataset. While together with a trigram language model,&#10;CER and SER reach 2.81% and 5.77%, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.11284" label="1807.11284">
        <attvalues>
          <attvalue for="0" value="Unsupervised Domain Adaptation by Adversarial Learning for Robust Speech&#10;  Recognition" />
          <attvalue for="1" value="  In this paper, we investigate the use of adversarial learning for&#10;unsupervised adaptation to unseen recording conditions, more specifically,&#10;single microphone far-field speech. We adapt neural networks based acoustic&#10;models trained with close-talk clean speech to the new recording conditions&#10;using untranscribed adaptation data. Our experimental results on Italian&#10;SPEECON data set show that our proposed method achieves 19.8% relative word&#10;error rate (WER) reduction compared to the unadapted models. Furthermore, this&#10;adaptation method is beneficial even when performed on data from another&#10;language (i.e. French) giving 12.6% relative WER reduction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.00644" label="1804.00644">
        <attvalues>
          <attvalue for="0" value="Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation" />
          <attvalue for="1" value="  The teacher-student (T/S) learning has been shown effective in unsupervised&#10;domain adaptation [1]. It is a form of transfer learning, not in terms of the&#10;transfer of recognition decisions, but the knowledge of posteriori&#10;probabilities in the source domain as evaluated by the teacher model. It learns&#10;to handle the speaker and environment variability inherent in and restricted to&#10;the speech signal in the target domain without proactively addressing the&#10;robustness to other likely conditions. Performance degradation may thus ensue.&#10;In this work, we advance T/S learning by proposing adversarial T/S learning to&#10;explicitly achieve condition-robust unsupervised domain adaptation. In this&#10;method, a student acoustic model and a condition classifier are jointly&#10;optimized to minimize the Kullback-Leibler divergence between the output&#10;distributions of the teacher and student models, and simultaneously, to&#10;min-maximize the condition classification loss. A condition-invariant deep&#10;feature is learned in the adapted student model through this procedure. We&#10;further propose multi-factorial adversarial T/S learning which suppresses&#10;condition variabilities caused by multiple factors simultaneously. Evaluated&#10;with the noisy CHiME-3 test set, the proposed methods achieve relative word&#10;error rate improvements of 44.60% and 5.38%, respectively, over a clean source&#10;model and a strong T/S learning baseline model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.04107" label="2210.04107">
        <attvalues>
          <attvalue for="0" value="Comparing Computational Architectures for Automated Journalism" />
          <attvalue for="1" value="  The majority of NLG systems have been designed following either a&#10;template-based or a pipeline-based architecture. Recent neural models for&#10;data-to-text generation have been proposed with an end-to-end deep learning&#10;flavor, which handles non-linguistic input in natural language without explicit&#10;intermediary representations. This study compares the most often employed&#10;methods for generating Brazilian Portuguese texts from structured data. Results&#10;suggest that explicit intermediate steps in the generation process produce&#10;better texts than the ones generated by neural end-to-end architectures,&#10;avoiding data hallucination while better generalizing to unseen inputs. Code&#10;and corpus are publicly available.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural Language Generation (NLG) is a subfield at the intersectionof linguistics, computer science, and artificial intelligence, concerned with generating readable, coherent and meaningful explanatory text or speech so as to describe non-linguistic input data \cite{Ehud}. NLG is often viewed as complementary to Natural Language Understanding (NLU) and part of Natural Language Processing (NLP). Whereas in NLU the goal is to understand input sentences to produce machine representations, in NLG the system must make decisions about how to transform representations into meaningful words and phrases \cite{liddy2001natural}.&#10;&#10;Multiple successful examples of data-to-text systems can be found in weather forecasting \cite{sripada2004lessons}, financial and analytical reporting, industrial monitoring \cite{kim2020design} and conversational agents. Amongst NLG applications, robot-journalism is one of the most prominent endeavors thanks to the abundance of structured data streams available today, thus allowing automated systems to report recurring material with high-fidelity and lexical variation \cite{graefe2016guide}.&#10;&#10;Traditionally, most data-to-text applications have been designed in a modular fashion as this facilitates reuse in different domains; going directly from input to output with rules has been simply too complex \cite{gatt2018survey}. In such systems, non-linguistic input data is converted into natural language through several explicit intermediate transformations and sequential tasks related to content selection, sentence planning and linguistic realization \cite{ferreira2019neural}. The two most frequently used automated journalism architectures are the template-based approach, which is application-dependent and lacks generalization capabilities due to its rule-based nature, and the pipeline-based approach, which embodies linguistic insights to convert data to text by applying a series of sequential steps.&#10;&#10;The emergence of neural-based NLG systems in the recent years has changed the field: provided there is enough labeled data for training a machine learning model, learning a direct mapping from structured input to textual output has become reality \cite{li2017deep}. This has led to the recent development of deep learning end-to-end models, which directly learn input-output mappings and rely far less on explicit intermediary representations and linguistic insights.&#10;&#10;Even though it is technically feasible to use neural end-to-end methods in real world applications, this does not necessarily mean that they are superior to rule-based approaches in every scenario. Recent empirical studies have demonstrated that a combination of template and pipeline systems produce texts that are more appropriate than the neural-based approaches, which frequently hallucinate content unsupported by the semantic input \cite{ferreira2019neural}. For the particular task of automated journalism, reporting inaccurate data would seriously undermine a robot’s credibility and could have serious implications on sensitive domains, such as environmental reports. A modular model also has the advantage of allowing for auditing, while neural end-to-end approaches behave as black-boxes \cite{campos2020towards}.&#10;&#10;In this paper, we compare the three most frequently used architectures for automated journalism -- template-based, pipeline-based and end-to-end neural models -- using a common domain, the Blue Amazon. With an offshore area of 3.6 million square kilometers along the Brazilian coast, the Blue Amazon is Brazil's exclusive economic zone (EEZ); it is a oceanic region brimming with marine species and energy resources \cite{thompson2015blue}. &#10;Ocean monitoring, climate change and environmental sustainability are &#10;promising fields for automated journalism applications.&#10;The oceans are severely damaged environments, and if current trends continues, there will be disastrous consequences for the planet as it is essential to halt climate change, fostering economic growth and preserving biodiversity \cite{e2022ocean}.&#10;Although connecting with public audiences in an approachable way typically requires coverage by trained human journalists, accurate and low latency information reports can be very helpful. There is a vast and ever-growing body of information about the oceans; clearly, society can benefit from a robot journalism system. To address this issue, we created our robot journalism application which combines different NLG approaches to generate daily reports about the Blue Amazon and publish them on Twitter. &#10;&#10;A corpus of verbalizations of non-linguistic data in Brazilian Portuguese was created based on syntactical and lexical patterning abstracted from data collected from publicly available sources. Intermediate representations were annotated for each entry in order to develop our corpus. A combination of automatic and human evaluation together with a qualitative analysis was then carried out to measure the fluency, semantics and lexical variety of the generated texts.&#10;&#10;This main contributions of this work are the construction of a publicly available Brazilian Portuguese NLG dataset, a comparison between the three most frequently used automated journalism architectures and an application which combines different approaches to publish daily reports about the Blue Amazon on Twitter. In Section \ref{sec:data}, we present our Blue Amazon dataset for automated journalism, and in Section \ref{sec:template} we discuss our approach in building a template-based architecture. Also, in Section \ref{sec:pipeline}, we present and discuss our pipeline architecture with six sequential modules. In Section &#10;\ref{sec:endtoend}, we discuss the end-to-end architecture and utilize it by training four different neural networks to generate the output text. In Section \ref{sec:results} we present the main results of this work and in Section \ref{sec:discussion} we discuss the results by providing some qualitative analysis. Finally, we conclude in Section \ref{sec:conclusion}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Data-to-Text Systems, Computer Science, Linguistics, Deep Learning Architectures, Data Science, Artificial Intelligence, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1908.09022" label="1908.09022">
        <attvalues>
          <attvalue for="0" value="Neural data-to-text generation: A comparison between pipeline and&#10;  end-to-end architectures" />
          <attvalue for="1" value="  Traditionally, most data-to-text applications have been designed using a&#10;modular pipeline architecture, in which non-linguistic input data is converted&#10;into natural language through several intermediate transformations. In&#10;contrast, recent neural models for data-to-text generation have been proposed&#10;as end-to-end approaches, where the non-linguistic input is rendered in natural&#10;language with much less explicit intermediate representations in-between. This&#10;study introduces a systematic comparison between neural pipeline and end-to-end&#10;data-to-text approaches for the generation of text from RDF triples. Both&#10;architectures were implemented making use of state-of-the art deep learning&#10;methods as the encoder-decoder Gated-Recurrent Units (GRU) and Transformer.&#10;Automatic and human evaluations together with a qualitative analysis suggest&#10;that having explicit intermediate steps in the generation process results in&#10;better texts than the ones generated by end-to-end approaches. Moreover, the&#10;pipeline models generalize better to unseen inputs. Data and code are publicly&#10;available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08928" label="2104.08928">
        <attvalues>
          <attvalue for="0" value="Group-Sparse Matrix Factorization for Transfer Learning of Word&#10;  Embeddings" />
          <attvalue for="1" value="  Unstructured text provides decision-makers with a rich data source in many&#10;domains, ranging from product reviews in retail to nursing notes in healthcare.&#10;To leverage this information, words are typically translated into word&#10;embeddings -- vectors that encode the semantic relationships between words --&#10;through unsupervised learning algorithms such as matrix factorization. However,&#10;learning word embeddings from new domains with limited training data can be&#10;challenging, because the meaning/usage may be different in the new domain,&#10;e.g., the word ``positive'' typically has positive sentiment, but often has&#10;negative sentiment in medical notes since it may imply that a patient tested&#10;positive for a disease. In practice, we expect that only a small number of&#10;domain-specific words may have new meanings. We propose an intuitive two-stage&#10;estimator that exploits this structure via a group-sparse penalty to&#10;efficiently transfer learn domain-specific word embeddings by combining&#10;large-scale text corpora (such as Wikipedia) with limited domain-specific text&#10;data. We bound the generalization error of our transfer learning estimator,&#10;proving that it can achieve high accuracy with substantially less&#10;domain-specific data when only a small number of embeddings are altered between&#10;domains. Furthermore, we prove that all local minima identified by our&#10;nonconvex objective function are statistically indistinguishable from the&#10;global minimum under standard regularization conditions, implying that our&#10;estimator can be computed efficiently. Our results provide the first bounds on&#10;group-sparse matrix factorization, which may be of independent interest. We&#10;empirically evaluate our approach compared to state-of-the-art fine-tuning&#10;heuristics from natural language processing.&#10;" />
          <attvalue for="2" value="&#10;&#10;Natural language processing is an increasingly important part of the analytics toolkit for leveraging unstructured text data in a variety of domains. For instance, service providers mine online consumer reviews to inform operational decisions on platforms~\cite{mankad2016understanding} or to infer market structure and the competitive landscape for products \cite{netzer2012mine}; Twitter posts are used to forecast TV show viewership \cite{liu2016structured}; analyst reports of S\&amp;P 500 firms are used to measure innovation \cite{bellstam2020text}; medical notes are used to predict operational metrics such as readmissions rates \cite{hsu-etal-2020-characterizing}; online ads or reviews are used to flag service providers that are likely engaging in illicit activities \cite{ramchandani2021unmasking, li2021detecting}. &#10;&#10;To leverage unstructured text in decision-making, we must preprocess the text to capture the semantic content of words in a way that can be passed as an input to a predictive machine learning algorithm. In the past, this involved domain experts performing costly and imperfect feature engineering. A much more powerful, data-driven approach is to use unsupervised learning algorithms to learn word embeddings, which represent words as vectors \cite{mikolov2013distributed,pennington2014glove}; we focus on widely-used word embedding models that are based on low-rank matrix factorization \cite{pennington2014glove, levy2014neural}. These word embeddings translate semantic similarities between words and the context within which they appear into statistical relationships. Typically, they are trained to encode how frequently pairs of words co-occur in text; these co-occurrence counts implicitly contain semantic properties of words since words with similar meanings tend to occur in similar contexts. Given the large number of words in the English language, to be effective in practice, embeddings must be trained on large-scale and comprehensive text data, e.g., popular embeddings such as Word2Vec \cite{mikolov2013distributed} and GloVe \cite{pennington2014glove} are trained on Wikipedia articles.&#10;&#10;However, it is well-known that pre-trained word embeddings can miss out on important domain-specific meaning/usage, hurting downstream interpretation and effectiveness. Take the healthcare domain as an example. The word ``positive'' is typically associated with positive sentiment on Wikipedia; yet, in the context of medical notes, it typically indicates the presence of a medical condition, corresponding to negative sentiment. Thus, using a generic word embedding for ``positive'' may diminish performance in medical applications. Similarly, words like ``adherence&quot; (referring to medication adherence) have a specific meaning in a healthcare context (relative to its context on general Wikipedia entries) and are strongly predictive of patient outcomes; failing to account for its healthcare-specific meaning may result in a loss in the downstream accuracy of healthcare-specific prediction tasks \cite{blitzer2007biographies}. Consequently, there has been a large body of work training specialized embeddings in a number of diverse contexts, ranging from radiology reports \cite{ong2020machine}, stock market prediction \cite{li2017learning}, cybersecurity vulnerability reports \cite{roy2017learning}, and patent classification \cite{risch2019domain}. This approach only works when the decision-maker has access to a sufficiently large domain-specific text corpus, allowing her to train high-quality embeddings. In practice, decision-makers often have limited domain-specific text data, yielding poor results when training new word embeddings, which hurts the quality of downstream modeling and decisions that leverage these embeddings. In other words, word embeddings trained on domain-specific data alone are unbiased but can have high variance due to limited sample size; in contrast, pre-trained word embeddings have low variance but can be significantly biased depending on the extent of domain mismatch.&#10;&#10;Then, a natural question is whether we can combine large-scale publicly available text corpora (which we call the proxy data hereafter) with limited domain-specific text data (which we call the gold data hereafter) to train precise but domain-specific word embeddings. In particular, we aim to use transfer learning to achieve a better bias-variance tradeoff than using gold or proxy data alone. Our key insight to enable transfer learning is that the meaning/usage of most words do not change when changing domains; rather, we expect that only a small number of domain-specific words will have new meaning/usage. To illustrate, Figure \ref{fig:sparsity_ill} shows text data (paragraphs) from a variety of domain-specific Wikipedia articles, including finance, math, computing, and politics. Words that have a domain-specific meaning are enclosed in a red box, while the remaining words share the same meaning/usage as in the standard English language. We observe that only a small number of unique words have domain-specific meaning/usage.&#10;&#10;More formally, consider a corpus of $d$ words. Let $U_p \in \mathbb{R}^{d\times r}$ denote the true (unobserved) proxy word embedding matrix, of which the $i^{th}$ row $U_p^{(i, \cdot)}$ is the true $r$-dimensional word embedding of word $i \in [d]=\{1,\cdots,d\}$ based on the proxy data; analogously, let $U_g \in \mathbb{R}^{d\times r}$ denote the true (unobserved) gold word embedding matrix. We expect that the meaning/usage for most words are preserved in both domains---i.e., the word embeddings $U_g^{(i, \cdot)} \neq U_p^{(i, \cdot)}$ for only a small number $s \ll d$ values of $i\in[d]$. This induces a group-sparse structure for the difference matrix $U_g-U_p$, i.e., only a small number $s$ of the rows (groups) are nonzero. Figure~\ref{fig:toy_rowsparse} illustrates this notion of ``sparsity'' on a toy example with $d=10$ words, embeddings with dimension $r=5$, and $s=3$ words with shifted meaning/usage. Indeed, we find support for this group-sparse structure in our previous examples from Wikipedia---e.g., in the finance domain (Fig~\ref{fig:sparsity_ill}(a)), we observe only $s=4$ unique finance domain-specific words (put, options, stock, strike) out of a total of $d=51$ distinct words, yielding a sparsity ratio $s/d \lesssim 0.08$. Similarly for the other domains in Fig~\ref{fig:sparsity_ill}, the sparsity ratios $s/d$ are approximately $0.11$, $0.07$ and $0.05$ for the math, computing, and politics examples respectively. (Details and experiments on the Wikipedia data can be found in \S\ref{sec:exp_wiki}.)&#10;&#10;Based on this intuition, we formulate an objective that incorporates a group-sparse penalty \cite{friedman2010note,simon2013sparse} on $U_g - U_p$, where each row is treated as a group. In particular, we estimate domain-specific embeddings from gold data, incorporating $\ell_{2,1}$ regularization to impose group sparsity relative to the (estimated) word embeddings trained on the large proxy data. Our approach balances the need to update the embeddings of important domain-specific words based on the gold data (i.e., reduce bias), while matching most words to the embeddings estimated from the large proxy text corpus (i.e., reduce variance).&#10;&#10;Our main result establishes that the word embedding estimator trained by group-sparse transfer learning achieves a sample complexity bound that, to leading order, scales quadratically in $s$ (the number of words with altered meaning/usage), as opposed to the conventional bound that scales quadratically in $d$ (the total number of words). In other words, transfer learning allows us to accurately identify domain-specific word embeddings with substantially less domain-specific data than classical low-rank matrix factorization methods. We build on prior work establishing error bounds for the group LASSO \cite{lounici2011oracle} and low-rank matrix problems \cite{ge2017no,negahban2011estimation}. We face two additional technical challenges. First, the literature on nonconvex low-rank matrix problems typically studies the Hessian to ensure that local minima are well-behaved; however, the Hessian may not be well-defined under our nonsmooth group-sparse penalty (since the gradient is not continuous). Second, unlike the traditional high-dimensional literature, transfer learning introduces a quartic form (in terms of $U_g - U_p$) in our objective function. We address both challenges through a new analysis that relies on an assumption we term ``quadratic compatibility condition.'' We show that quadratic compatibility is implied by a natural restricted strong convexity (RSC) assumption, which we prove holds with high probability in a general low-rank matrix factorization problem for the illustrative cases of gaussian data and word co-occurrence count data. Furthermore, under a slightly weaker condition that can characterize all local minima \cite{loh2015regularized}, all local minima identified by our algorithm are statistically indistinguishable from the global minimum, implying that our estimator can be computed efficiently.&#10;&#10;While our technical results hold for embeddings trained using matrix factorization, our algorithm straightforwardly applies to nonlinear objectives such as GloVe. &#10;Simulations on synthetic data and domain-specific Wikipedia articles show that our estimator significantly outperforms common heuristics given rich proxy data and limited domain-specific data. Importantly, we show that this is an interpretable strategy to identifying key words with distinct meanings in specific domains such as finance, math, and computing. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Domain Adaptation Learning, Mathematics, Text Data Analysis, Natural Language Processing, Statistics, Word Embeddings Transfer" />
        </attvalues>
      </node>
      <node id="2010.03574" label="2010.03574">
        <attvalues>
          <attvalue for="0" value="Characterizing the Value of Information in Medical Notes" />
          <attvalue for="1" value="  Machine learning models depend on the quality of input data. As electronic&#10;health records are widely adopted, the amount of data in health care is&#10;growing, along with complaints about the quality of medical notes. We use two&#10;prediction tasks, readmission prediction and in-hospital mortality prediction,&#10;to characterize the value of information in medical notes. We show that as a&#10;whole, medical notes only provide additional predictive power over structured&#10;information in readmission prediction. We further propose a probing framework&#10;to select parts of notes that enable more accurate predictions than using all&#10;notes, despite that the selected information leads to a distribution shift from&#10;the training data (&quot;all notes&quot;). Finally, we demonstrate that models trained on&#10;the selected valuable information achieve even better predictive performance,&#10;with only 6.8% of all the tokens for readmission prediction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.07470" label="1709.07470">
        <attvalues>
          <attvalue for="0" value="Learning Domain-Specific Word Embeddings from Sparse Cybersecurity Texts" />
          <attvalue for="1" value="  Word embedding is a Natural Language Processing (NLP) technique that&#10;automatically maps words from a vocabulary to vectors of real numbers in an&#10;embedding space. It has been widely used in recent years to boost the&#10;performance of a vari-ety of NLP tasks such as Named Entity Recognition,&#10;Syntac-tic Parsing and Sentiment Analysis. Classic word embedding methods such&#10;as Word2Vec and GloVe work well when they are given a large text corpus. When&#10;the input texts are sparse as in many specialized domains (e.g.,&#10;cybersecurity), these methods often fail to produce high-quality vectors. In&#10;this pa-per, we describe a novel method to train domain-specificword embeddings&#10;from sparse texts. In addition to domain texts, our method also leverages&#10;diverse types of domain knowledge such as domain vocabulary and semantic&#10;relations. Specifi-cally, we first propose a general framework to encode&#10;diverse types of domain knowledge as text annotations. Then we de-velop a novel&#10;Word Annotation Embedding (WAE) algorithm to incorporate diverse types of text&#10;annotations in word em-bedding. We have evaluated our method on two&#10;cybersecurity text corpora: a malware description corpus and a Common&#10;Vulnerability and Exposure (CVE) corpus. Our evaluation re-sults have&#10;demonstrated the effectiveness of our method in learning domain-specific word&#10;embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1007.1771" label="1007.1771">
        <attvalues>
          <attvalue for="0" value="Oracle Inequalities and Optimal Inference under Group Sparsity" />
          <attvalue for="1" value="  We consider the problem of estimating a sparse linear regression vector&#10;$\beta^*$ under a gaussian noise model, for the purpose of both prediction and&#10;model selection. We assume that prior knowledge is available on the sparsity&#10;pattern, namely the set of variables is partitioned into prescribed groups,&#10;only few of which are relevant in the estimation process. This group sparsity&#10;assumption suggests us to consider the Group Lasso method as a means to&#10;estimate $\beta^*$. We establish oracle inequalities for the prediction and&#10;$\ell_2$ estimation errors of this estimator. These bounds hold under a&#10;restricted eigenvalue condition on the design matrix. Under a stronger&#10;coherence condition, we derive bounds for the estimation error for mixed&#10;$(2,p)$-norms with $1\le p\leq \infty$. When $p=\infty$, this result implies&#10;that a threshold version of the Group Lasso estimator selects the sparsity&#10;pattern of $\beta^*$ with high probability. Next, we prove that the rate of&#10;convergence of our upper bounds is optimal in a minimax sense, up to a&#10;logarithmic factor, for all estimators over a class of group sparse vectors.&#10;Furthermore, we establish lower bounds for the prediction and $\ell_2$&#10;estimation errors of the usual Lasso estimator. Using this result, we&#10;demonstrate that the Group Lasso can achieve an improvement in the prediction&#10;and estimation properties as compared to the Lasso.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.00708" label="1704.00708">
        <attvalues>
          <attvalue for="0" value="No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified&#10;  Geometric Analysis" />
          <attvalue for="1" value="  In this paper we develop a new framework that captures the common landscape&#10;underlying the common non-convex low-rank matrix problems including matrix&#10;sensing, matrix completion and robust PCA. In particular, we show for all above&#10;problems (including asymmetric cases): 1) all local minima are also globally&#10;optimal; 2) no high-order saddle points exists. These results explain why&#10;simple algorithms such as stochastic gradient descent have global converge, and&#10;efficiently optimize these non-convex objective functions in practice. Our&#10;framework connects and simplifies the existing analyses on optimization&#10;landscapes for matrix sensing and symmetric matrix completion. The framework&#10;naturally leads to new results for asymmetric matrix completion and robust PCA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1305.2436" label="1305.2436">
        <attvalues>
          <attvalue for="0" value="Regularized M-estimators with nonconvexity: Statistical and algorithmic&#10;  theory for local optima" />
          <attvalue for="1" value="  We provide novel theoretical results regarding local optima of regularized&#10;$M$-estimators, allowing for nonconvexity in both loss and penalty functions.&#10;Under restricted strong convexity on the loss and suitable regularity&#10;conditions on the penalty, we prove that \emph{any stationary point} of the&#10;composite objective function will lie within statistical precision of the&#10;underlying parameter vector. Our theory covers many nonconvex objective&#10;functions of interest, including the corrected Lasso for errors-in-variables&#10;linear models; regression for generalized linear models with nonconvex&#10;penalties such as SCAD, MCP, and capped-$\ell_1$; and high-dimensional&#10;graphical model estimation. We quantify statistical accuracy by providing&#10;bounds on the $\ell_1$-, $\ell_2$-, and prediction error between stationary&#10;points and the population-level optimum. We also propose a simple modification&#10;of composite gradient descent that may be used to obtain a near-global optimum&#10;within statistical precision $\epsilon$ in $\log(1/\epsilon)$ steps, which is&#10;the fastest possible rate of any first-order method. We provide simulation&#10;studies illustrating the sharpness of our theoretical results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.01181" label="1906.01181">
        <attvalues>
          <attvalue for="0" value="Improved Zero-shot Neural Machine Translation via Ignoring Spurious&#10;  Correlations" />
          <attvalue for="1" value="  Zero-shot translation, translating between language pairs on which a Neural&#10;Machine Translation (NMT) system has never been trained, is an emergent&#10;property when training the system in multilingual settings. However, naive&#10;training for zero-shot NMT easily fails, and is sensitive to hyper-parameter&#10;setting. The performance typically lags far behind the more conventional&#10;pivot-based approach which translates twice using a third language as a pivot.&#10;In this work, we address the degeneracy problem due to capturing spurious&#10;correlations by quantitatively analyzing the mutual information between&#10;language IDs of the source and decoded sentences. Inspired by this analysis, we&#10;propose to use two simple but effective approaches: (1) decoder pre-training;&#10;(2) back-translation. These methods show significant improvement (4~22 BLEU&#10;points) over the vanilla zero-shot translation on three challenging&#10;multilingual datasets, and achieve similar or better results than the&#10;pivot-based approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.11867" label="2004.11867">
        <attvalues>
          <attvalue for="0" value="Improving Massively Multilingual Neural Machine Translation and&#10;  Zero-Shot Translation" />
          <attvalue for="1" value="  Massively multilingual models for neural machine translation (NMT) are&#10;theoretically attractive, but often underperform bilingual models and deliver&#10;poor zero-shot translations. In this paper, we explore ways to improve them. We&#10;argue that multilingual NMT requires stronger modeling capacity to support&#10;language pairs with varying typological characteristics, and overcome this&#10;bottleneck via language-specific components and deepening NMT architectures. We&#10;identify the off-target translation issue (i.e. translating into a wrong target&#10;language) as the major source of the inferior zero-shot performance, and&#10;propose random online backtranslation to enforce the translation of unseen&#10;training language pairs. Experiments on OPUS-100 (a novel multilingual dataset&#10;with 100 languages) show that our approach substantially narrows the&#10;performance gap with bilingual models in both one-to-many and many-to-many&#10;settings, and improves zero-shot performance by ~10 BLEU, approaching&#10;conventional pivot-based methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07885" label="2104.07885">
        <attvalues>
          <attvalue for="0" value="Probing Across Time: What Does RoBERTa Know and When?" />
          <attvalue for="1" value="  Models of language trained on very large corpora have been demonstrated&#10;useful for NLP. As fixed artifacts, they have become the object of intense&#10;study, with many researchers &quot;probing&quot; the extent to which linguistic&#10;abstractions, factual and commonsense knowledge, and reasoning abilities they&#10;acquire and readily demonstrate. Building on this line of work, we consider a&#10;new question: for types of knowledge a language model learns, when during&#10;(pre)training are they acquired? We plot probing performance across iterations,&#10;using RoBERTa as a case study. Among our findings: linguistic knowledge is&#10;acquired fast, stably, and robustly across domains. Facts and commonsense are&#10;slower and more domain-sensitive. Reasoning abilities are, in general, not&#10;stably acquired. As new datasets, pretraining protocols, and probes emerge, we&#10;believe that probing-across-time analyses can help researchers understand the&#10;complex, intermingled learning that these models undergo and guide us toward&#10;more efficient approaches that accomplish necessary learning faster.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.00054" label="2104.00054">
        <attvalues>
          <attvalue for="0" value="A Statistical Analysis of Summarization Evaluation Metrics using&#10;  Resampling Methods" />
          <attvalue for="1" value="  The quality of a summarization evaluation metric is quantified by calculating&#10;the correlation between its scores and human annotations across a large number&#10;of summaries. Currently, it is unclear how precise these correlation estimates&#10;are, nor whether differences between two metrics' correlations reflect a true&#10;difference or if it is due to mere chance. In this work, we address these two&#10;problems by proposing methods for calculating confidence intervals and running&#10;hypothesis tests for correlations using two resampling methods, bootstrapping&#10;and permutation. After evaluating which of the proposed methods is most&#10;appropriate for summarization through two simulation experiments, we analyze&#10;the results of applying these methods to several different automatic evaluation&#10;metrics across three sets of human annotations. We find that the confidence&#10;intervals are rather wide, demonstrating high uncertainty in the reliability of&#10;automatic metrics. Further, although many metrics fail to show statistical&#10;improvements over ROUGE, two recent works, QAEval and BERTScore, do in some&#10;evaluation settings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.09637" label="1912.09637">
        <attvalues>
          <attvalue for="0" value="Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language&#10;  Model" />
          <attvalue for="1" value="  Recent breakthroughs of pretrained language models have shown the&#10;effectiveness of self-supervised learning for a wide range of natural language&#10;processing (NLP) tasks. In addition to standard syntactic and semantic NLP&#10;tasks, pretrained models achieve strong improvements on tasks that involve&#10;real-world knowledge, suggesting that large-scale language modeling could be an&#10;implicit method to capture knowledge. In this work, we further investigate the&#10;extent to which pretrained models such as BERT capture knowledge using a&#10;zero-shot fact completion task. Moreover, we propose a simple yet effective&#10;weakly supervised pretraining objective, which explicitly forces the model to&#10;incorporate knowledge about real-world entities. Models trained with our new&#10;objective yield significant improvements on the fact completion task. When&#10;applied to downstream tasks, our model consistently outperforms BERT on four&#10;entity-related question answering datasets (i.e., WebQuestions, TriviaQA,&#10;SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard&#10;fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.12343" label="2312.12343">
        <attvalues>
          <attvalue for="0" value="LatestEval: Addressing Data Contamination in Language Model Evaluation&#10;  through Dynamic and Time-Sensitive Test Construction" />
          <attvalue for="1" value="  Data contamination in evaluation is getting increasingly prevalent with the&#10;emergence of language models pre-trained on super large, automatically crawled&#10;corpora. This problem leads to significant challenges in the accurate&#10;assessment of model capabilities and generalisations. In this paper, we propose&#10;LatestEval, an automatic method that leverages the most recent texts to create&#10;uncontaminated reading comprehension evaluations. LatestEval avoids data&#10;contamination by only using texts published within a recent time window,&#10;ensuring no overlap with the training corpora of pre-trained language models.&#10;We develop the LatestEval automated pipeline to 1) gather the latest texts; 2)&#10;identify key information, and 3) construct questions targeting the information&#10;while removing the existing answers from the context. This encourages models to&#10;infer the answers themselves based on the remaining context, rather than just&#10;copy-paste. Our experiments demonstrate that language models exhibit negligible&#10;memorisation behaviours on LatestEval as opposed to previous benchmarks,&#10;suggesting a significantly reduced risk of data contamination and leading to a&#10;more robust evaluation. Data and code are publicly available at:&#10;https://github.com/liyucheng09/LatestEval.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent years have seen the ubiquity of pretrained language models in natural language processing (NLP) due to their strong performance and generalisation capability. These models are usually pre-trained on super large internet-crawled corpora. However, many widely used benchmarks are also largely constructed from web resources \cite{hendrycks2020measuring}, which are very likely to be unintentionally included in the pretraining stage. This leads to a major emerging issue called data contamination.&#10;&#10;Recent analysis has revealed that data contamination is widespread in model evaluations \cite{openai2023gpt4,sainz2023nlp}, which greatly undermines the credibility of evaluation results \cite{marie2023,li2023open} and prevents fair comparisons between models \cite{dickson2023}. Moreover, the massive scale of training data makes decontaminating existing benchmarks extremely difficult \cite{kreutzer-etal-2022-quality}. For many closed models, training data is considered as trade secret and thus confidential, eliminating any possibility for the community to address contamination by decontaminating benchmarks.&#10;One potential solution to avoid contaminated evaluation is to create new test data constantly or use human evaluation \cite{liu2023evaluating,jacovi2023stop}, just like how examination for human works. However, this is extremely inefficient and costly, requiring huge human efforts periodically. &#10;&#10;In this paper, we propose LatestEval, an automatic method that leverages the most recent texts to create novel uncontaminated reading comprehension evaluations. Before starting to create the data, we conduct the first manual analysis of real-world Human-AI chat data on document comprehension, to identify the most frequently asked types of knowledge and thus determine the scope of LatestEval.&#10;As for data construction, LatestEval, as a reading comprehension benchmark, typically consists of three components: passage, query, and answer. Here we start by 1) collecting recently created texts across various time-sensitive sources as the passages. Three different sources are used in our experiments: arXiv, BBC, and GitHub. We then 2) extract key information from these texts using various tools as the answers. At last we 3) construct questions targeting the extracted information. The questions are generated automatically via template-filling or large language models. During the testing stage, we remove the extracted answers from the original context, which encourages models to infer answers through reasoning based on the remaining context rather than short-cutting via copy-paste. Overall, LatestEval avoids data contamination by using the latest materials for testing, ensuring that it does not overlap with models' pretraining data.&#10;&#10;Experimental results show that, in contrast to existing reading comprehension benchmarks where language models show significant memorisation, models exhibit negligible memorisation behaviour on LatestEval. Additional human evaluation and performance experiments further demonstrate LatestEval is reliable and effective in benchmarking state-of-the-art language models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Data Contamination Prevention, Reading Comprehension Assessment, Linguistics, Language Model Evaluation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2310.17589" label="2310.17589">
        <attvalues>
          <attvalue for="0" value="An Open Source Data Contamination Report for Large Language Models" />
          <attvalue for="1" value="  Data contamination in model evaluation has become increasingly prevalent with&#10;the growing popularity of large language models. It allows models to &quot;cheat&quot;&#10;via memorisation instead of displaying true capabilities. Therefore,&#10;contamination analysis has become an crucial part of reliable model evaluation&#10;to validate results. However, existing contamination analysis is usually&#10;conducted internally by large language model developers and often lacks&#10;transparency and completeness. This paper presents an extensive data&#10;contamination report for over 15 popular large language models across six&#10;popular multiple-choice QA benchmarks. We also introduce an open-source&#10;pipeline that enables the community to perform contamination analysis on&#10;customised data and models. Our experiments reveal varying contamination levels&#10;ranging from 1\% to 45\% across benchmarks, with the contamination degree&#10;increasing rapidly over time. Performance analysis of large language models&#10;indicates that data contamination does not necessarily lead to increased model&#10;metrics: while significant accuracy boosts of up to 14\% and 7\% are observed&#10;on contaminated C-Eval and Hellaswag benchmarks, only a minimal increase is&#10;noted on contaminated MMLU. We also find larger models seem able to gain more&#10;advantages than smaller models on contaminated test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10160" label="2305.10160">
        <attvalues>
          <attvalue for="0" value="Stop Uploading Test Data in Plain Text: Practical Strategies for&#10;  Mitigating Data Contamination by Evaluation Benchmarks" />
          <attvalue for="1" value="  Data contamination has become prevalent and challenging with the rise of&#10;models pretrained on large automatically-crawled corpora. For closed models,&#10;the training data becomes a trade secret, and even for open models, it is not&#10;trivial to detect contamination. Strategies such as leaderboards with hidden&#10;answers, or using test data which is guaranteed to be unseen, are expensive and&#10;become fragile with time. Assuming that all relevant actors value clean test&#10;data and will cooperate to mitigate data contamination, what can be done? We&#10;propose three strategies that can make a difference: (1) Test data made public&#10;should be encrypted with a public key and licensed to disallow derivative&#10;distribution; (2) demand training exclusion controls from closed API holders,&#10;and protect your test data by refusing to evaluate without them; (3) avoid data&#10;which appears with its solution on the internet, and release the web-page&#10;context of internet-derived data along with the data. These strategies are&#10;practical and can be effective in preventing data contamination.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.15144" label="2109.15144">
        <attvalues>
          <attvalue for="0" value="A Review of Text Style Transfer using Deep Learning" />
          <attvalue for="1" value="  Style is an integral component of a sentence indicated by the choice of words&#10;a person makes. Different people have different ways of expressing themselves,&#10;however, they adjust their speaking and writing style to a social context, an&#10;audience, an interlocutor or the formality of an occasion. Text style transfer&#10;is defined as a task of adapting and/or changing the stylistic manner in which&#10;a sentence is written, while preserving the meaning of the original sentence.&#10;  A systematic review of text style transfer methodologies using deep learning&#10;is presented in this paper. We point out the technological advances in deep&#10;neural networks that have been the driving force behind current successes in&#10;the fields of natural language understanding and generation. The review is&#10;structured around two key stages in the text style transfer process, namely,&#10;representation learning and sentence generation in a new style. The discussion&#10;highlights the commonalities and differences between proposed solutions as well&#10;as challenges and opportunities that are expected to direct and foster further&#10;research in the field.&#10;" />
          <attvalue for="2" value="&#10;&#10;Naturally occurring linguistic variations in spoken and written language have been contributed to culture, personal attributes and social context~\cite{eckert2001style, coupland2007style}. The underlying factors contributing to linguistic variations in spoken language have been extensively studied in the field of variationist sociolinguistics. The adjustments of one’s individual style to match or shift away~\cite{kiesling1998language} from the style of the interlocutor, the audience or social context are prominent in the work of the American linguist, William Labov~\cite{labov1972sociolinguistic, labov1981field, labov1972some}. Different people have different ways of expressing themselves~\cite{labov1972sociolinguistic} and personal attributes, such as gender, age, education, personality, emotional state~\cite{pennebaker2003psychological} are reflected in their writing style. However, style changes over time~\cite{eckert2001style} and we adjust to a social context, an audience we address, a person we communicate with~\cite{bell1984language}, and/or the formality of an occasion~\cite{kiesling1998language}. While direct mapping of sociolinguistics categories is not always possible, stylistic properties have been classified along several dimensions in the research on natural language understanding and generation.&#10;&#10;Adjusting the style of a sentence by rewriting the original sentence in a new style, while preserving its semantic content, is referred to as text style transfer. The diversity of linguistic styles is matched by the diversity in research interests in the field. Some researchers viewed style transfer as an ability to adjust the emotional content in a written text; others equated the concept with formality or politeness. Changing the sentiment polarity of a sentence might change the meaning of a text or transform the message it conveys, although the ability to change the emotional content in a written text should be viewed more along the lines of adjusting the tone of a message that is more appropriate, emphatic and less severe or offensive to the audience or the conversational partner. Other researchers have directed their efforts towards much more sound conceptualization of a style as a genre, or linguistic style of a person, or a particular social group. &#10;&#10;Language style should be a special consideration in current and future intelligent interaction systems~\cite{li2016persona} that understand, process, or generate speech or text. Automatically adjusting the text style could help users improve their communication skills (e.g., being more polite, learning to write formal messages), and could become even more important, when employed in future prosocial interaction mediators on discussion platforms and comment-based communities (e.g., toning down negative sentiment, neutralizing offensiveness).&#10;&#10;In a decade or so, the work on the topic expanded from a few articles to an active research area. Most of the methods for text style transfer are based on deep neural networks. The success of deep learning in other areas has provided fruitful directions to be followed. Inspired by the success of the encoder-decoder models in other fields, including machine translation (MT)~\cite{cho2014properties, wu2016google}, text summarization~\cite{chopra2016abstractive} and dialogue generation~\cite{serban2017hierarchical}, a number of style transfer models are built upon this end-to-end model of learning~\cite{li2018delete, sudhakar2019transforming, xu2019formality, dai2019style, cheng2020contextual, fu2018style, dos2018fighting, prabhumoye2018style, john2019disentangled, zhang2018learning, prabhumoye2018style2, tian2018structured, liu2020revision, wang2019controllable, zhou2020exploring, lee2020stable, zhang2018shaped, hu2017toward}. New advances directed toward adversarial learning have also inspired more recent works on text style transfer~\cite{shen2017style, zhao2018language, logeswaran2018content, chen2018adversarial}.&#10;&#10;Pivotal in this review are the studies that address the automatic adjustment of the style of a written text. At the onset of our paper, we introduce the reader to various text styles that have been in the focus of the selected research papers. We have compiled a list of publicly available datasets that we discuss in terms of their suitability for a particular style transfer task(s). The evaluation of how successful a particular model is on the task of style transfer has two objectives: to measure how well the semantic content in the generated sentence was preserved and to assess the quality of rewriting the sentence in a new (target) style. Evaluation of the performance of style transfer models is of special importance for future research in the area. &#10;&#10;The discussion of the specifics of the proposed approaches to style transfer is organized to allow readers to follow the advances in deep learning and their impact on style transfer tasks. The discussion follows the two key stages in the text style transfer process: 1) representation learning of the style and content of a given sentence and 2) generation of the sentence that has the same meaning as the input sentence, but is expressed in a different style. Auxiliary elements, such as style embeddings~\cite{li2018delete, sudhakar2019transforming, dai2019style, cheng2020contextual, fu2018style, dos2018fighting, zhang2018learning, lee2020stable, hu2017toward, shen2017style, logeswaran2018content, chen2018adversarial}, style classifiers~\cite{xu2019formality, dai2019style} and/or adversarial discriminators~\cite{shen2017style, zhao2018language} are also discussed. We discuss the critical stage of the process, the output sentence generation by categorizing the approaches among three groups. Namely, models that use a simple approach to generation by reconstructing the input sentence, models that incorporate additional style classifier in their encoder-decoder architectures, and models that adopt adversarial learning. &#10;&#10;The paper is organized as follows. After the introductory section, Section~\ref{sec:textual_style} provides a description of various text styles that have been in the focus of the selected research papers. Section~\ref{sec:style_transfer_for_text} gives a formalization of text style transfer and discusses the publicly available datasets suitable for the task at hand. The discussion of a set of measures, which have been proposed as meaningful criteria for evaluating style transfer models, is also presented. Beginning with a brief introduction of several deep neural networks in Section~\ref{sec:deep_neural_networks_for_text_generation}, the discussion of state-of-the-art style transfer methodologies using deep learning is presented in Section~\ref{sec:methods_for_style_transfer}. Section~\ref{sec:challenges} reflects on the challenges style transfer faces and casts light on potential research directions that are expected to further advance the field. Section~\ref{sec:conclusion} concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Learning Methodologies, Deep Learning, Artificial Intelligence, Natural Language Processing, Text Style Transfer" />
        </attvalues>
      </node>
      <node id="1903.06353" label="1903.06353">
        <attvalues>
          <attvalue for="0" value="Formality Style Transfer with Hybrid Textual Annotations" />
          <attvalue for="1" value="  Formality style transformation is the task of modifying the formality of a&#10;given sentence without changing its content. Its challenge is the lack of&#10;large-scale sentence-aligned parallel data. In this paper, we propose an&#10;omnivorous model that takes parallel data and formality-classified data jointly&#10;to alleviate the data sparsity issue. We empirically demonstrate the&#10;effectiveness of our approach by achieving the state-of-art performance on a&#10;recently proposed benchmark dataset of formality transfer. Furthermore, our&#10;model can be readily adapted to other unsupervised text style transfer tasks&#10;like unsupervised sentiment transfer and achieve competitive results on three&#10;widely recognized benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.19287" label="2403.19287">
        <attvalues>
          <attvalue for="0" value="CoderUJB: An Executable and Unified Java Benchmark for Practical&#10;  Programming Scenarios" />
          <attvalue for="1" value="  In the evolving landscape of large language models (LLMs) tailored for&#10;software engineering, the need for benchmarks that accurately reflect&#10;real-world development scenarios is paramount. Current benchmarks are either&#10;too simplistic or fail to capture the multi-tasking nature of software&#10;development. To address this, we introduce CoderUJB, a new benchmark designed&#10;to evaluate LLMs across diverse Java programming tasks that are executable and&#10;reflective of actual development scenarios, acknowledging Java's prevalence in&#10;real-world software production. CoderUJB comprises 2,239 programming questions&#10;derived from 17 real open-source Java projects and spans five practical&#10;programming tasks. Our empirical study on this benchmark investigates the&#10;coding abilities of various open-source and closed-source LLMs, examining the&#10;effects of continued pre-training in specific programming languages code and&#10;instruction fine-tuning on their performance. The findings indicate that while&#10;LLMs exhibit strong potential, challenges remain, particularly in&#10;non-functional code generation (e.g., test generation and defect detection).&#10;Importantly, our results advise caution in the specific programming languages&#10;continued pre-training and instruction fine-tuning, as these techniques could&#10;hinder model performance on certain tasks, suggesting the need for more nuanced&#10;strategies. CoderUJB thus marks a significant step towards more realistic&#10;evaluations of programming capabilities in LLMs, and our study provides&#10;valuable insights for the future development of these models in software&#10;engineering.&#10;" />
          <attvalue for="2" value="&#10;&#10;Researchers have found that advanced AI technologies, exemplified by large language models (LLMs), are proficient in addressing a broad spectrum of challenges, spanning from everyday tasks to complex software engineering issues~\cite{chatgpt,LLMSurvey,codex,aprplm,codeagent1,wang2022usb,wang2023pandalm,wang2024exploring,kieval}. In addition to these general-purpose LLMs, there are code-centric large language models (code LLMs), such as CodeX~\cite{codex}, CodeLlama~\cite{codellama}, and StarCoder~\cite{starcoder}, which are specifically designed to excel at software engineering tasks. Many of these code LLMs are open-source and can be privately deployed to avoid data security issues. As a result, they have gained significant attention for their strong coding skills. Because of this interest, different benchmarks~\cite{codex,codereval,codexglue,xcodeeval,classeval,aixbench} have been designed to measure the programming capabilities of these LLMs. Specifically, Table~\ref{tab:benchmark} presents a selection of notable benchmarks within software engineering alongside our CoderUJB, illustrating their distinct characteristics. The HumanEval~\cite{codex} benchmark stands out in this field, which is used to evaluate the ability of Python function generation and consists of 164 manually designed Python programming questions, and evaluates the quality of the generated solutions by checking whether the solution can be successfully executed by unit tests. We denote those execution-based evaluations as &quot;Executable (\checkmark)&quot;. Then, CoderEval~\cite{codereval} noticed that the questions in HumanEval are simple single-function generation tasks that do not match the actual development scenarios (i.e., writing code in a software project). So, they introduced a new benchmark with 460 questions that better aligned with the actual development scenarios. We denote those actual development questions as &quot;Project-Runnable (\checkmark)&quot;, indicating that the generated code requires a project context dependency for execution. However, those benchmarks focus on a single programming scenario and cannot provide a comprehensive evaluation of the programming capability of LLMs. Previous multi-task benchmarks, like CodeXGLUE~\cite{codexglue}, have been critiqued~\cite{codex} because they rely on similarity-based metrics like BLEU and CodeBLEU~\cite{codebleu}, which do not involve running the code. The recently proposed multi-programming task dataset XCodeEval~\cite{xcodeeval} focuses on questions from programming competitions, which do not accurately reflect typical real-world development scenarios. Consequently, the absence of a benchmark that comprehensively covers multi-programming tasks, executability, and matches real-world development scenarios prevents us from assessing the effectiveness of current LLMs on a broader range of real-world programming tasks.&#10;&#10;To this end, we introduce CoderUJB, a comprehensive benchmark designed for evaluating LLMs that supports multiple tasks, adheres to real-world software development scenarios, and allows for execution within a complete program context(i.e., all source code and execution environments). Specifically, CoderUJB is built on 17 real open-source Java projects, acknowledging Java's prevalence in real-world software production~\cite{tiobe}. We extracted 238 functional code generation questions and 140 code-based test generation~\cite{chatteser} questions from these projects by analyzing the abstract syntax trees and test coverage relationships of the project source code. Then, we extracted and collected 451 issue-based test generation~\cite{issuetest} questions, 470 automatic program repair~\cite{aprplm} questions, and 940 defect detection~\cite{devign} questions from the projects by combining the detailed defect information and related issue reports from the projects. Altogether, CoderUJB comprises 2,239 programming questions covering five trending and practical programming tasks, which is the largest benchmark that is &quot;Project-Runnable&quot; and each question comes with a complete program context to facilitate the researcher's detailed analysis of the questions and the generated solutions. Ultimately, the solutions generated by the LLMs will be placed in real projects for execution, and the programming ability of the LLMs will be evaluated using the execution success rate as the primary metric.&#10;&#10;Next, to illustrate the value of CoderUJB to the field, we conducted a comprehensive empirical study on CoderUJB to explore the programming abilities of a representative set of open-source code LLMs and general-purpose closed-source LLMs. The aim was to answer a couple of crucial questions: How good are these LLMs at coding? And how do continued pre-training~\cite{codellama} and instruction fine-tuning~\cite{wizardcoder} affect their programming performance? After running a slew of experiments on CoderUJB, we find that current LLMs still perform poorly in solving non-functional code generation tasks, especially defect detection tasks. Secondly, superior open-source LLMs can already approach or even surpass GPT-3.5-Turbo on the functional code generation task and the two test generation tasks. Thirdly, continued pre-training in a specific programming language can reduce an LLM's performance in other languages, and this negative impact diminishes as the task becomes less related to the original pre-training task. Such varied outcomes across different tasks emphasize CoderUJB's value as a unified evaluation benchmark that incorporates multiple programming tasks. Lastly, our findings indicate that instruction fine-tuning diminishes the performance of code LLMs in functional code generation and the two test generation tasks—a contrast to the results from the less practical benchmark, HumanEval. This highlights the value of CoderUJB as a practical programming evaluation benchmark. &#10;&#10;We summarize the main contributions of this study as follows:&#10;&#10;\begin{itemize}&#10; \item Benchmark. We have introduced CoderUJB, a universal Java benchmark for assessing LLM performance across multiple real-world programming tasks. The benchmark includes executable test cases and the complete program context (i.e., all source code and execution environments), comprising 2,239 programming questions.&#10; \item Study. We ran a comprehensive empirical analysis of both open-source and proprietary LLMs using CoderUJB, studying (1) their performance across diverse coding tasks, (2) the effect of continued pre-training in a specific language code, and (3) the impact of instruction fine-tuning on these models.&#10; \item Implications. This work revealed multiple significant findings: (1) Program context is useful in code generation tasks. (2) Caution is advised when continuing pre-training in a specific programming language, as its effects tend to be unpredictable, especially if the downstream task is substantially different from the pre-training task. (3) Instruction fine-tuning should be approached carefully, as it can diminish the performance of LLMs on tasks that align closely with the pre-trained task. (4) Comprehensive evaluations are essential, given that different programming tasks may yield disparate results when the same training strategy is applied. &#10;\end{itemize}&#10;&#10;The CoderUJB are publicly available in.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Software Engineering Benchmarks, Artificial Intelligence, Programming Task Evaluation" />
        </attvalues>
      </node>
      <node id="2304.01457" label="2304.01457">
        <attvalues>
          <attvalue for="0" value="Exploring Vision-Language Models for Imbalanced Learning" />
          <attvalue for="1" value="  Vision-Language models (VLMs) that use contrastive language-image&#10;pre-training have shown promising zero-shot classification performance.&#10;However, their performance on imbalanced dataset is relatively poor, where the&#10;distribution of classes in the training dataset is skewed, leading to poor&#10;performance in predicting minority classes. For instance, CLIP achieved only 5%&#10;accuracy on the iNaturalist18 dataset. We propose to add a lightweight decoder&#10;to VLMs to avoid OOM (out of memory) problem caused by large number of classes&#10;and capture nuanced features for tail classes. Then, we explore improvements of&#10;VLMs using prompt tuning, fine-tuning, and incorporating imbalanced algorithms&#10;such as Focal Loss, Balanced SoftMax and Distribution Alignment. Experiments&#10;demonstrate that the performance of VLMs can be further boosted when used with&#10;decoder and imbalanced methods. Specifically, our improved VLMs significantly&#10;outperforms zero-shot classification by an average accuracy of 6.58%, 69.82%,&#10;and 6.17%, on ImageNet-LT, iNaturalist18, and Places-LT, respectively. We&#10;further analyze the influence of pre-training data size, backbones, and&#10;training cost. Our study highlights the significance of imbalanced learning&#10;algorithms in face of VLMs pre-trained by huge data. We release our code at&#10;https://github.com/Imbalance-VLM/Imbalance-VLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.00288" label="2302.00288">
        <attvalues>
          <attvalue for="0" value="CoderEval: A Benchmark of Pragmatic Code Generation with Generative&#10;  Pre-trained Models" />
          <attvalue for="1" value="  Code generation models based on the pre-training and fine-tuning paradigm&#10;have been increasingly attempted by both academia and industry, resulting in&#10;well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To&#10;evaluate the effectiveness of these models, multiple existing benchmarks are&#10;proposed, including only cases of generating a standalone function, i.e., a&#10;function that may invoke or access only built-in functions and standard&#10;libraries. However, non-standalone functions, which typically are not included&#10;in the existing benchmarks, constitute more than 70% of the functions in&#10;popular open-source projects, and evaluating models' effectiveness on&#10;standalone functions cannot reflect these models' effectiveness on pragmatic&#10;code generation scenarios.&#10;  To help bridge the preceding gap, in this paper, we propose a benchmark named&#10;CoderEval, consisting of 230 Python and 230 Java code generation tasks&#10;carefully curated from popular real-world open-source projects and a&#10;self-contained execution platform to automatically assess the functional&#10;correctness of generated code. CoderEval supports code generation tasks from&#10;six levels of context dependency, where context refers to code elements such as&#10;types, APIs, variables, and consts defined outside the function under&#10;generation but within the dependent third-party libraries, current class, file,&#10;or project. CoderEval can be used to evaluate the effectiveness of models in&#10;generating code beyond only standalone functions. By evaluating three code&#10;generation models on CoderEval, we find that the effectiveness of these models&#10;in generating standalone functions is substantially higher than that in&#10;generating non-standalone functions. Our analysis highlights the current&#10;progress and pinpoints future directions to further improve a model's&#10;effectiveness by leveraging contextual information for pragmatic code&#10;generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.03004" label="2303.03004">
        <attvalues>
          <attvalue for="0" value="xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code&#10;  Understanding, Generation, Translation and Retrieval" />
          <attvalue for="1" value="  Recently, pre-trained large language models (LLMs) have shown impressive&#10;abilities in generating codes from natural language descriptions, repairing&#10;buggy codes, translating codes between languages, and retrieving relevant code&#10;segments. However, the evaluation of these models has often been performed in a&#10;scattered way on only one or two specific tasks, in a few languages, at a&#10;partial granularity (e.g., function) level, and in many cases without proper&#10;training data. Even more concerning is that in most cases the evaluation of&#10;generated codes has been done in terms of mere lexical overlap with a reference&#10;code rather than actual execution. We introduce xCodeEval, the largest&#10;executable multilingual multitask benchmark to date consisting of $25$M&#10;document-level coding examples ($16.5$B tokens) from about $7.5$K unique&#10;problems covering up to $11$ programming languages with execution-level&#10;parallelism. It features a total of $7$ tasks involving code understanding,&#10;generation, translation and retrieval. xCodeEval adopts an execution-based&#10;evaluation and offers a multilingual code execution engine, ExecEval that&#10;supports unit test based execution in all the $11$ languages. To address the&#10;challenge of balancing the distributions of text-code samples over multiple&#10;attributes in validation/test sets, we propose a novel data splitting and a&#10;data selection schema based on the geometric mean and graph-theoretic&#10;principle. Our experiments with OpenAI's LLMs (zero-shot) and open-LLMs&#10;(zero-shot and fine-tuned) on the tasks and languages demonstrate **xCodeEval**&#10;to be quite challenging as per the current advancements in language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.10297" label="2009.10297">
        <attvalues>
          <attvalue for="0" value="CodeBLEU: a Method for Automatic Evaluation of Code Synthesis" />
          <attvalue for="1" value="  Evaluation metrics play a vital role in the growth of an area as it defines&#10;the standard of distinguishing between good and bad models. In the area of code&#10;synthesis, the commonly used evaluation metric is BLEU or perfect accuracy, but&#10;they are not suitable enough to evaluate codes, because BLEU is originally&#10;designed to evaluate the natural language, neglecting important syntactic and&#10;semantic features of codes, and perfect accuracy is too strict thus it&#10;underestimates different outputs with the same semantic logic. To remedy this,&#10;we introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the&#10;strength of BLEU in the n-gram match and further injects code syntax via&#10;abstract syntax trees (AST) and code semantics via data-flow. We conduct&#10;experiments by evaluating the correlation coefficient between CodeBLEU and&#10;quality scores assigned by the programmers on three code synthesis tasks, i.e.,&#10;text-to-code, code translation, and code refinement. Experimental results show&#10;that our proposed CodeBLEU can achieve a better correlation with programmer&#10;assigned scores compared with BLEU and accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.04207" label="2305.04207">
        <attvalues>
          <attvalue for="0" value="No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test&#10;  Generation" />
          <attvalue for="1" value="  Unit testing is essential in detecting bugs in functionally-discrete program&#10;units. Manually writing high-quality unit tests is time-consuming and&#10;laborious. Although traditional techniques can generate tests with reasonable&#10;coverage, they exhibit low readability and cannot be directly adopted by&#10;developers. Recent work has shown the large potential of large language models&#10;(LLMs) in unit test generation, which can generate more human-like and&#10;meaningful test code. ChatGPT, the latest LLM incorporating instruction tuning&#10;and reinforcement learning, has performed well in various domains. However, It&#10;remains unclear how effective ChatGPT is in unit test generation.&#10;  In this work, we perform the first empirical study to evaluate ChatGPT's&#10;capability of unit test generation. Specifically, we conduct a quantitative&#10;analysis and a user study to systematically investigate the quality of its&#10;generated tests regarding the correctness, sufficiency, readability, and&#10;usability. The tests generated by ChatGPT still suffer from correctness issues,&#10;including diverse compilation errors and execution failures. Still, the passing&#10;tests generated by ChatGPT resemble manually-written tests by achieving&#10;comparable coverage, readability, and even sometimes developers' preference.&#10;Our findings indicate that generating unit tests with ChatGPT could be very&#10;promising if the correctness of its generated tests could be further improved.&#10;  Inspired by our findings above, we propose ChatTESTER, a novel ChatGPT-based&#10;unit test generation approach, which leverages ChatGPT itself to improve the&#10;quality of its generated tests. ChatTESTER incorporates an initial test&#10;generator and an iterative test refiner. Our evaluation demonstrates the&#10;effectiveness of ChatTESTER by generating 34.3% more compilable tests and 18.7%&#10;more tests with correct assertions than the default ChatGPT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.11515" label="2209.11515">
        <attvalues>
          <attvalue for="0" value="Large Language Models are Few-shot Testers: Exploring LLM-based General&#10;  Bug Reproduction" />
          <attvalue for="1" value="  Many automated test generation techniques have been developed to aid&#10;developers with writing tests. To facilitate full automation, most existing&#10;techniques aim to either increase coverage, or generate exploratory inputs.&#10;However, existing test generation techniques largely fall short of achieving&#10;more semantic objectives, such as generating tests to reproduce a given bug&#10;report. Reproducing bugs is nonetheless important, as our empirical study shows&#10;that the number of tests added in open source repositories due to issues was&#10;about 28% of the corresponding project test suite size. Meanwhile, due to the&#10;difficulties of transforming the expected program semantics in bug reports into&#10;test oracles, existing failure reproduction techniques tend to deal exclusively&#10;with program crashes, a small subset of all bug reports. To automate test&#10;generation from general bug reports, we propose LIBRO, a framework that uses&#10;Large Language Models (LLMs), which have been shown to be capable of performing&#10;code-related tasks. Since LLMs themselves cannot execute the target buggy code,&#10;we focus on post-processing steps that help us discern when LLMs are effective,&#10;and rank the produced tests according to their validity. Our evaluation of&#10;LIBRO shows that, on the widely studied Defects4J benchmark, LIBRO can generate&#10;failure reproducing test cases for 33% of all studied cases (251 out of 750),&#10;while suggesting a bug reproducing test in first place for 149 bugs. To&#10;mitigate data contamination, we also evaluate LIBRO against 31 bug reports&#10;submitted after the collection of the LLM training data terminated: LIBRO&#10;produces bug reproducing tests for 32% of the studied bug reports. Overall, our&#10;results show LIBRO has the potential to significantly enhance developer&#10;efficiency by automatically generating tests from bug reports.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.03496" label="1909.03496">
        <attvalues>
          <attvalue for="0" value="Devign: Effective Vulnerability Identification by Learning Comprehensive&#10;  Program Semantics via Graph Neural Networks" />
          <attvalue for="1" value="  Vulnerability identification is crucial to protect the software systems from&#10;attacks for cyber security. It is especially important to localize the&#10;vulnerable functions among the source code to facilitate the fix. However, it&#10;is a challenging and tedious process, and also requires specialized security&#10;expertise. Inspired by the work on manually-defined patterns of vulnerabilities&#10;from various code representation graphs and the recent advance on graph neural&#10;networks, we propose Devign, a general graph neural network based model for&#10;graph-level classification through learning on a rich set of code semantic&#10;representations. It includes a novel Conv module to efficiently extract useful&#10;features in the learned rich node representations for graph-level&#10;classification. The model is trained over manually labeled datasets built on 4&#10;diversified large-scale open-source C projects that incorporate high complexity&#10;and variety of real source code instead of synthesis code used in previous&#10;works. The results of the extensive evaluation on the datasets demonstrate that&#10;Devign outperforms the state of the arts significantly with an average of&#10;10.51% higher accuracy and 8.68\% F1 score, increases averagely 4.66% accuracy&#10;and 6.37% F1 by the Conv module.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.11350" label="2304.11350">
        <attvalues>
          <attvalue for="0" value="Romanian Multiword Expression Detection Using Multilingual Adversarial&#10;  Training and Lateral Inhibition" />
          <attvalue for="1" value="  Multiword expressions are a key ingredient for developing large-scale and&#10;linguistically sound natural language processing technology. This paper&#10;describes our improvements in automatically identifying Romanian multiword&#10;expressions on the corpus released for the PARSEME v1.2 shared task. Our&#10;approach assumes a multilingual perspective based on the recently introduced&#10;lateral inhibition layer and adversarial training to boost the performance of&#10;the employed multilingual language models. With the help of these two methods,&#10;we improve the F1-score of XLM-RoBERTa by approximately 2.7% on unseen&#10;multiword expressions, the main task of the PARSEME 1.2 edition. In addition,&#10;our results can be considered SOTA performance, as they outperform the previous&#10;results on Romanian obtained by the participants in this competition.&#10;" />
          <attvalue for="2" value="&#10;&#10;The correct identification and handling of multiword expressions (MWEs) are important for various natural language processing (NLP) applications, such as machine translation, text classification, or information retrieval. For example, in machine translation, if an MWE is not recognized as such and is literally translated rather than as an expression, the resulting translation either is confusing or has the wrong meaning \cite{zaninello2020multiword}. In text classification, MWEs recognition can provide important information about the topic or sentiment of a text \cite{catone2019automatic}, while in information retrieval, MWEs can clarify the meaning of a query and improve the accuracy of search results \cite{englmeier2021aspect}.&#10;&#10;The PARSEME COST Action organized three editions \cite{corpus-1.0, corpus-1.1, corpus-1.2} of a shared task that aimed at improving the identification of verbal MWEs (VMWEs) in text. This work improves the results obtained in PARSEME 1.2 \cite{corpus-1.2} for the Romanian language. We investigate the advantages of using Romanian monolingual Transformer-based \cite{vaswani2017attention} language models together with merging all the datasets for each language presented at the competition in a single corpus and then fine-tuning several multilingual language models on it. Additionally, for the latter, we aim to enhance the overall system's performance by generating language-independent features, with the help of two techniques, namely the lateral inhibition layer \cite{pais2022racai} on top of the language models and adversarial training \cite{lowd2005adversarial} between languages. &#10;&#10;Our experiments show that by employing these two algorithms, the results of the cross-lingual robustly optimized BERT approach (XLM-RoBERTa) \cite{conneau2020unsupervised} improve by 2.7\% on unseen MWEs when trained on the combined dataset. Additionally, we report state-of-the-art (SOTA) results with the monolingual training of Romanian Bidirectional Encoder Representations from Transformer (RoBERT) \cite{dumitrescu2020birth} in comparison with the results obtained at the PARSEME 1.2 edition, achieving an F1-score of 60.46\%, an improvement of over 20\%.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Language Models, Multiword Expression Identification, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2009.08712" label="2009.08712">
        <attvalues>
          <attvalue for="0" value="The birth of Romanian BERT" />
          <attvalue for="1" value="  Large-scale pretrained language models have become ubiquitous in Natural&#10;Language Processing. However, most of these models are available either in&#10;high-resource languages, in particular English, or as multilingual models that&#10;compromise performance on individual languages for coverage. This paper&#10;introduces Romanian BERT, the first purely Romanian transformer-based language&#10;model, pretrained on a large text corpus. We discuss corpus composition and&#10;cleaning, the model training process, as well as an extensive evaluation of the&#10;model on various Romanian datasets. We open source not only the model itself,&#10;but also a repository that contains information on how to obtain the corpus,&#10;fine-tune and use this model in production (with practical examples), and how&#10;to fully replicate the evaluation process.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.15183" label="2305.15183">
        <attvalues>
          <attvalue for="0" value="Are Pre-trained Language Models Useful for Model Ensemble in Chinese&#10;  Grammatical Error Correction?" />
          <attvalue for="1" value="  Model ensemble has been in widespread use for Grammatical Error Correction&#10;(GEC), boosting model performance. We hypothesize that model ensemble based on&#10;the perplexity (PPL) computed by pre-trained language models (PLMs) should&#10;benefit the GEC system. To this end, we explore several ensemble strategies&#10;based on strong PLMs with four sophisticated single models. However, the&#10;performance does not improve but even gets worse after the PLM-based ensemble.&#10;This surprising result sets us doing a detailed analysis on the data and coming&#10;up with some insights on GEC. The human references of correct sentences is far&#10;from sufficient in the test data, and the gap between a correct sentence and an&#10;idiomatic one is worth our attention. Moreover, the PLM-based ensemble&#10;strategies provide an effective way to extend and improve GEC benchmark data.&#10;Our source code is available at&#10;https://github.com/JamyDon/PLM-based-CGEC-Model-Ensemble.&#10;" />
          <attvalue for="2" value="&#10;Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text \cite{bryant2022grammatical}. Nowadays, there are two mainstream GEC approaches. The first is treating GEC as a low-resource machine translation task \cite{yuan2016grammatical}, where sequence-to-sequence models like BART \cite{lewis2020bart} are used. This approach simply inputs the incorrect text to the encoder and gets the corrected result from the decoder. The second is treating GEC as a sequence tagging task, where the incorrect text is still taken as the input, but the output is edit tags (keep, delete, add, replace, etc.) for each token. After applying all the edits to the input text, the corrected result is then generated. The model used in this approach is also known as sequence-to-edit models and GECToR \cite{omelianchuk2020gector} is a typical one.&#10;&#10;However, most researches on GEC focus on English while Chinese GEC (CGEC) has just started up. The Chinese language is different from English in many ways and its GEC is thus much harder. &#10;Instead of word inflection in many Western languages, the Chinese grammar is expressed by function words and word order, making CGEC more difficult and complex for that we can't take word form as a handle. &#10;In addition, unlike English, we have very few datasets for training and testing CGEC, which sets us exploring training-free methods like model ensemble to further improve the performance of CGEC systems.&#10;&#10;Because of the nature of GEC that corrections can be represented as several independent edits, model ensemble has been a popular way to improve GEC systems. In CGEC, \cite{li-etal-2018-hybrid}, \cite{liang-etal-2020-bert} and \cite{zhang-etal-2022-mucgec} ensemble their models by majority voting on edits and achieve considerable improvement.&#10;Besides, \cite{DBLP:journals/corr/XieAAJN16} adopt language models to improve neural language correction, following whom \cite{junczys-dowmunt-etal-2018-approaching} ensemble their GEC models using a language model probability. Today, transformer-based \cite{vaswani2017attention} Pre-trained Language Models (PLMs) have been in predominant&#10;use in NLP. However, we find few works on model ensemble using PLMs in CGEC.&#10;&#10;In this work, we hypothesize that choosing the best ensemble output with the help of perplexity (PPL) computed by PLMs should boost the final performance of CGEC. We experiment on ensemble of four CGEC models, including two sequence-to-sequence ones and two sequence-to-edit ones. We try four ensemble strategies: traditional voting, sentence-level ensemble, edit-level ensemble, and edit-combination ensemble, the last three exploiting the power of PLMs. &#10;&#10;To our surprise, the results of model ensemble with PLMs do not exceed those of traditional voting and are even worse than most of the single models. &#10;To find out why a low PPL cannot lead to a &#10;better GEC performance, we carry out a detailed analysis on the ensemble results and get some insights on GEC:&#10;&#10;1) In the test data, human references are insufficient, while PLM-based ensemble strategies produce valuable candidates, after being human checked, which may be &#10;considered as necessary complement to human references. &#10;&#10;2) When facing an erroneous sentence, a human expert corrects it with the minimal effort, while PLM-based ensemble strategies generate more natural and idiomatic text, which is of great help for oversea language learners. &#10;&#10;3) With the powerful ability, PLM-based models try to generate fluent sentences but sometimes ignore the original meaning of the source sentence, resulting in over-correction that should be addressed in future work. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Model Ensemble Strategies, Artificial Intelligence, Mathematics, Grammatical Error Correction" />
        </attvalues>
      </node>
      <node id="2211.05166" label="2211.05166">
        <attvalues>
          <attvalue for="0" value="Grammatical Error Correction: A Survey of the State of the Art" />
          <attvalue for="1" value="  Grammatical Error Correction (GEC) is the task of automatically detecting and&#10;correcting errors in text. The task not only includes the correction of&#10;grammatical errors, such as missing prepositions and mismatched subject-verb&#10;agreement, but also orthographic and semantic errors, such as misspellings and&#10;word choice errors respectively. The field has seen significant progress in the&#10;last decade, motivated in part by a series of five shared tasks, which drove&#10;the development of rule-based methods, statistical classifiers, statistical&#10;machine translation, and finally neural machine translation systems which&#10;represent the current dominant state of the art. In this survey paper, we&#10;condense the field into a single article and first outline some of the&#10;linguistic challenges of the task, introduce the most popular datasets that are&#10;available to researchers (for both English and other languages), and summarise&#10;the various methods and techniques that have been developed with a particular&#10;focus on artificial error generation. We next describe the many different&#10;approaches to evaluation as well as concerns surrounding metric reliability,&#10;especially in relation to subjective human judgements, before concluding with&#10;an overview of recent progress and suggestions for future work and remaining&#10;challenges. We hope that this survey will serve as comprehensive resource for&#10;researchers who are new to the field or who want to be kept apprised of recent&#10;developments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.10994" label="2204.10994">
        <attvalues>
          <attvalue for="0" value="MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese&#10;  Grammatical Error Correction" />
          <attvalue for="1" value="  This paper presents MuCGEC, a multi-reference multi-source evaluation dataset&#10;for Chinese Grammatical Error Correction (CGEC), consisting of 7,063 sentences&#10;collected from three Chinese-as-a-Second-Language (CSL) learner sources. Each&#10;sentence is corrected by three annotators, and their corrections are carefully&#10;reviewed by a senior annotator, resulting in 2.3 references per sentence. We&#10;conduct experiments with two mainstream CGEC models, i.e., the&#10;sequence-to-sequence model and the sequence-to-edit model, both enhanced with&#10;large pretrained language models, achieving competitive benchmark performance&#10;on previous and our datasets. We also discuss CGEC evaluation methodologies,&#10;including the effect of multiple references and using a char-based metric. Our&#10;annotation guidelines, data, and code are available at&#10;\url{https://github.com/HillZhang1999/MuCGEC}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1603.09727" label="1603.09727">
        <attvalues>
          <attvalue for="0" value="Neural Language Correction with Character-Based Attention" />
          <attvalue for="1" value="  Natural language correction has the potential to help language learners&#10;improve their writing skills. While approaches with separate classifiers for&#10;different error types have high precision, they do not flexibly handle errors&#10;such as redundancy or non-idiomatic phrasing. On the other hand, word and&#10;phrase-based machine translation methods are not designed to cope with&#10;orthographic errors, and have recently been outpaced by neural models.&#10;Motivated by these issues, we present a neural network-based approach to&#10;language correction. The core component of our method is an encoder-decoder&#10;recurrent neural network with an attention mechanism. By operating at the&#10;character level, the network avoids the problem of out-of-vocabulary words. We&#10;illustrate the flexibility of our approach on dataset of noisy, user-generated&#10;text collected from an English learner forum. When combined with a language&#10;model, our method achieves a state-of-the-art $F_{0.5}$-score on the CoNLL 2014&#10;Shared Task. We further demonstrate that training the network on additional&#10;data with synthesized errors can improve performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.09615" label="2401.09615">
        <attvalues>
          <attvalue for="0" value="Learning Shortcuts: On the Misleading Promise of NLU in Language Models" />
          <attvalue for="1" value="  The advent of large language models (LLMs) has enabled significant&#10;performance gains in the field of natural language processing. However, recent&#10;studies have found that LLMs often resort to shortcuts when performing tasks,&#10;creating an illusion of enhanced performance while lacking generalizability in&#10;their decision rules. This phenomenon introduces challenges in accurately&#10;assessing natural language understanding in LLMs. Our paper provides a concise&#10;survey of relevant research in this area and puts forth a perspective on the&#10;implications of shortcut learning in the evaluation of language models,&#10;specifically for NLU tasks. This paper urges more research efforts to be put&#10;towards deepening our comprehension of shortcut learning, contributing to the&#10;development of more robust language models, and raising the standards of NLU&#10;evaluation in real-world scenarios.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models (LLMs) have become the convention in the field of natural language processing in recent years. The preference for LLMs can be attributed to their performance gains in a wide variety of NLP tasks, including question answering, textual entailment, sentiment analysis, and commonsense reasoning \cite{peters_deep_2018, devlin_bert_2019, sap_commonsense_2020}. As LLMs scale up, their performance gains not only compete with but also exceed human performance on language understanding benchmarks \cite{he2022debertav3, chowdhery2023palm}. Whether such performance gains are meaningful depends on the quality of the evaluation metrics and the relevance of benchmarking schemes \cite{marie2021scientific, bommasani2021opportunities}.&#10;&#10;Unfortunately, LLMs have been shown to exploit spurious associations and dataset biases as `shortcuts', achieving inflated scores on NLU benchmarks, while lacking reliability and generalization on out-of-distribution samples. Recent works have shown that LLMs tend to learn shortcuts based on statistical cues e.g. the word ``not&quot; \cite{niven_probing_2019}, keywords \cite{moon_masker_2021, du_towards_2021} and cues related to language variations \cite{nguyen_learning_2021} to make predictions. This behavior, also known as shortcut learning, leads the model to learn non-generalizable decision rules that do not perform well on out-of-distribution data \cite{moon_masker_2021, du_shortcut_2022}, but continues to give state-of-the-art results on independent identically distributed (IID) samples. Thus, PLM performance gains on NLU benchmarks do not necessarily portray improvement in the semantic and reasoning capabilities of language models.&#10;&#10;This paper delineates the challenges in enhancing NLU capabilities within pre-trained language models amidst the presence of shortcut learning. We survey recent research on quantifying and mitigating shortcut learning and examine their implications for language under-standing in language models. Finally, we suggest potential research directions and guidelines that can help develop robust and reliable language models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Cognitive Science, Natural Language Understanding, Artificial Intelligence, Artificial Intelligence Limitations" />
        </attvalues>
      </node>
      <node id="2111.09543" label="2111.09543">
        <attvalues>
          <attvalue for="0" value="DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with&#10;  Gradient-Disentangled Embedding Sharing" />
          <attvalue for="1" value="  This paper presents a new pre-trained language model, DeBERTaV3, which&#10;improves the original DeBERTa model by replacing mask language modeling (MLM)&#10;with replaced token detection (RTD), a more sample-efficient pre-training task.&#10;Our analysis shows that vanilla embedding sharing in ELECTRA hurts training&#10;efficiency and model performance. This is because the training losses of the&#10;discriminator and the generator pull token embeddings in different directions,&#10;creating the &quot;tug-of-war&quot; dynamics. We thus propose a new gradient-disentangled&#10;embedding sharing method that avoids the tug-of-war dynamics, improving both&#10;training efficiency and the quality of the pre-trained model. We have&#10;pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its&#10;exceptional performance on a wide range of downstream natural language&#10;understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an&#10;example, the DeBERTaV3 Large model achieves a 91.37% average score, which is&#10;1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art&#10;(SOTA) among the models with a similar structure. Furthermore, we have&#10;pre-trained a multi-lingual model mDeBERTa and observed a larger improvement&#10;over strong baselines compared to English models. For example, the mDeBERTa&#10;Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6%&#10;improvement over XLM-R Base, creating a new SOTA on this benchmark. We have&#10;made our pre-trained models and inference code publicly available at&#10;https://github.com/microsoft/DeBERTa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.08331" label="2006.08331">
        <attvalues>
          <attvalue for="0" value="Probing Neural Dialog Models for Conversational Understanding" />
          <attvalue for="1" value="  The predominant approach to open-domain dialog generation relies on&#10;end-to-end training of neural models on chat datasets. However, this approach&#10;provides little insight as to what these models learn (or do not learn) about&#10;engaging in dialog. In this study, we analyze the internal representations&#10;learned by neural open-domain dialog systems and evaluate the quality of these&#10;representations for learning basic conversational skills. Our results suggest&#10;that standard open-domain dialog systems struggle with answering questions,&#10;inferring contradiction, and determining the topic of conversation, among other&#10;tasks. We also find that the dyadic, turn-taking nature of dialog is not fully&#10;leveraged by these models. By exploring these limitations, we highlight the&#10;need for additional research into architectures and training methods that can&#10;better capture high-level information about dialog.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Evaluating and interpreting open-domain dialog models is notoriously challenging. Multiple studies have shown that standard evaluation metrics such as perplexity and BLEU scores \cite{papineni2002bleu}&#10;correlate very weakly with human judgements of conversation quality \cite{liu2016not,ghandeharioun2019approximating,dziri2019evaluating}. This has inspired multiple new approaches for evaluating dialog systems. One popular evaluation metric involves calculating the semantic similarity between the user input and generated response in high-dimensional embedding space \cite{liu2016not,ghandeharioun2019approximating,dziri2019evaluating,park2018hierarchical,zhao2017learning,xu2018towards}. \cite{ghandeharioun2019approximating} proposed calculating conversation metrics such as sentiment and coherence on self-play conversations generated by trained models. Similarly, \cite{dziri2019evaluating} use neural classifiers to identify whether the model-generated responses entail or contradict user input in a natural language inference setting. &#10;&#10;To the best of our knowledge, all existing approaches for evaluating the performance of open-domain dialog systems only consider external model behavior in the sense that they analyze properties of the generated text. In this study, we explore internal representations instead, motivated by the fact that reasonable internal behavior is crucial for interpretability and is often a prerequisite for effective external behavior.&#10;&#10;Outside of open-domain dialog, probing has been applied for analyzing natural language processing models in machine translation \cite{belinkov2017neural} and visual question answering \cite{subramaniananalyzing}. Probing is also commonly used for evaluating the quality of ``universal'' sentence representations which are trained once and used for a variety of applications \cite{conneau2018you,adi2016fine} (for example, InferSent \cite{conneau2017supervised}, SkipThought \cite{kiros2015skip}, USE \cite{cer2018universal}). Along the same lines, natural language understanding benchmarks such as GLUE \cite{wang2018glue} and SuperGLUE \cite{wang2019superglue} propose a set of diverse tasks for evaluating general linguistic knowledge. Our analysis differs from previous work since it is focused on probing for conversational skills that are particularly relevant to dialog generation.&#10;&#10;With regard to perturbation experiments, \cite{sankar2019neural} found that standard dialog models are largely insensitive to perturbations of the input text. Here we introduce an alternative set of perturbation experiments to similarly explore the extent to which dialog structure is being leveraged by these models. &#10;&#10;" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Conversational AI Limitations, Artificial Intelligence, Natural Language Processing, Neural Model Analysis, Dialog Generation" />
        </attvalues>
      </node>
      <node id="2104.02112" label="2104.02112">
        <attvalues>
          <attvalue for="0" value="Efficient Attentions for Long Document Summarization" />
          <attvalue for="1" value="  The quadratic computational and memory complexities of large Transformers&#10;have limited their scalability for long document summarization. In this paper,&#10;we propose Hepos, a novel efficient encoder-decoder attention with head-wise&#10;positional strides to effectively pinpoint salient information from the source.&#10;We further conduct a systematic study of existing efficient self-attentions.&#10;Combined with Hepos, we are able to process ten times more tokens than existing&#10;models that use full attentions. For evaluation, we present a new dataset,&#10;GovReport, with significantly longer documents and summaries. Results show that&#10;our models produce significantly higher ROUGE scores than competitive&#10;comparisons, including new state-of-the-art results on PubMed. Human evaluation&#10;also shows that our models generate more informative summaries with fewer&#10;unfaithful errors.&#10;" />
          <attvalue for="2" value="&#10;&#10;Long documents, such as scientific papers and government reports, often discuss substantial issues at length, and thus are time-consuming to read, let alone to comprehend. Generating abstractive summaries can help readers quickly grasp the main topics, yet prior work has mostly focused on short texts (containing hundreds of words), e.g., news articles~\cite{gehrmann2018bottom,liu-lapata-2019-text,DBLP:journals/corr/abs-1912-08777}. &#10;&#10;Model training efficiency and summary quality present a pair of challenges for long document summarization.&#10;State-of-the-art systems~\cite{lewis-etal-2020-bart,DBLP:journals/corr/abs-1912-08777} are built upon Transformer~\cite{NIPS2017_3f5ee243}, which uses attentions to compute pairwise relations between tokens. Such framework has quadratic time and memory complexities, and is too costly for long documents. &#10;Solutions have been proposed to reduce the calculation of encoder self-attentions~\cite{wang2020linformer,zaheer2020big} by selectively attending to neighboring tokens~\cite{beltagy2020longformer,child2019generating} or relevant words~\cite{Kitaev2020Reformer:,tay2020sparse}. &#10;Yet, these methods do not apply to encoder-decoder attentions in summarization models since they collaborate and dynamically pinpoint salient content in the source as the summary is decoded. &#10;Truncation is commonly used to circumvent the issue. However, training on curtailed content further aggravates ``hallucination'' in existing abstractive models~\cite{maynez-etal-2020-faithfulness}. &#10;&#10;We argue that summarizing long documents (e.g., with thousands of words or more) requires efficient handling of both types of attentions. To this end, we propose an efficient encoder-decoder attention with head-wise positional strides (Hepos), where the attention heads follow a strided pattern and have varying starting positions. Hepos reduces computational and memory costs while (1) maintaining the power of emphasizing important tokens, and (2) preserving the global context per head. Hepos successfully doubles the processed input sequence size, when combined with any encoder. To the best of our knowledge, we are the first to study efficient encoder-decoder attentions and provide a systematic comparison of diverse encoder attentions for the task of summarization.&#10;&#10;For evaluation, we collect a new large-scale dataset, \ourdata, consisting of about $19.5$k U.S. government reports with expert-written abstractive summaries.&#10;\ourdata has two important features: (1) It contains significantly longer documents ($9.4$k words) and summaries ($553$ words) than existing datasets, such as PubMed and arXiv~\cite{cohan-etal-2018-discourse} (see Table~\ref{tab:basic_stat}); (2) Salient content is spread throughout the documents, as opposed to cases where summary-worthy words are more heavily concentrated in specific parts of the document.&#10;These properties make \ourdata an important benchmark for producing long document summaries with multiple paragraphs.&#10;&#10;We conduct experiments on \ourdata and scientific papers in PubMed and arXiv. &#10;First, when summarizing documents of the same length, Hepos attention yields significantly better ROUGE scores than a non-trivial comparison that projects attentions into low-rank space~\cite{wang2020linformer}. &#10;Second, when trained on the same GPU, Hepos attention, combined with sparse encoder attentions, is able to read more than $10$K words and obtains significantly higher ROUGE scores on \ourdata and new state-of-the-art results on PubMed, compared with full encoder-decoder attention models which can process at most $5$K input words. &#10;Human judges further rate the summaries generated by our models to be more informative and faithful. &#10;&#10;We further propose a new evaluation metric for faithfulness, inspired by APES~\cite{eyal-etal-2019-question}, a fill-in-the-blank QA metric for summary evaluation. &#10;With questions generated from references, our metric, APES$_{src}$, compares QA answers by reading the source and the system summary. It is shown to be better correlated with human judgment than the original metric and an entailment-based scorer~\cite{kryscinski-etal-2020-evaluating}. &#10;&#10;The rest of the paper is organized as follows. We describe efficient encoder attentions in prior work in \S~\ref{sec:encoder}, and formulate our proposed encoder-decoder attention in \S~\ref{sec:decoder}. &#10;The \ourdata data is presented in \S~\ref{sec:dataset}. We then share details on evaluation metrics (\S~\ref{sec:eval}) and experimental results (\S~\ref{sec:results}). Additional related work is listed in \S~\ref{sec:add_related_work}, with conclusion in \S\ref{sec:conclusion}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Long Document Summarization, Computer Science, Linguistics, Efficient Attention Mechanisms, Artificial Intelligence, Transformer Model Optimization, Information Science" />
        </attvalues>
      </node>
      <node id="2002.11296" label="2002.11296">
        <attvalues>
          <attvalue for="0" value="Sparse Sinkhorn Attention" />
          <attvalue for="1" value="  We propose Sparse Sinkhorn Attention, a new efficient and sparse method for&#10;learning to attend. Our method is based on differentiable sorting of internal&#10;representations. Concretely, we introduce a meta sorting network that learns to&#10;generate latent permutations over sequences. Given sorted sequences, we are&#10;then able to compute quasi-global attention with only local windows, improving&#10;the memory efficiency of the attention module. To this end, we propose new&#10;algorithmic innovations such as Causal Sinkhorn Balancing and SortCut, a&#10;dynamic sequence truncation method for tailoring Sinkhorn Attention for&#10;encoding and/or decoding purposes. Via extensive experiments on algorithmic&#10;seq2seq sorting, language modeling, pixel-wise image generation, document&#10;classification and natural language inference, we demonstrate that our memory&#10;efficient Sinkhorn Attention method is competitive with vanilla attention and&#10;consistently outperforms recently proposed efficient Transformer models such as&#10;Sparse Transformers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.05685" label="1804.05685">
        <attvalues>
          <attvalue for="0" value="A Discourse-Aware Attention Model for Abstractive Summarization of Long&#10;  Documents" />
          <attvalue for="1" value="  Neural abstractive summarization models have led to promising results in&#10;summarizing relatively short documents. We propose the first model for&#10;abstractive summarization of single, longer-form documents (e.g., research&#10;papers). Our approach consists of a new hierarchical encoder that models the&#10;discourse structure of a document, and an attentive discourse-aware decoder to&#10;generate the summary. Empirical results on two large-scale datasets of&#10;scientific papers show that our model significantly outperforms&#10;state-of-the-art models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.14345" label="2312.14345">
        <attvalues>
          <attvalue for="0" value="Logic-Scaffolding: Personalized Aspect-Instructed Recommendation&#10;  Explanation Generation using LLMs" />
          <attvalue for="1" value="  The unique capabilities of Large Language Models (LLMs), such as the natural&#10;language text generation ability, position them as strong candidates for&#10;providing explanation for recommendations. However, despite the size of the&#10;LLM, most existing models struggle to produce zero-shot explanations reliably.&#10;To address this issue, we propose a framework called Logic-Scaffolding, that&#10;combines the ideas of aspect-based explanation and chain-of-thought prompting&#10;to generate explanations through intermediate reasoning steps. In this paper,&#10;we share our experience in building the framework and present an interactive&#10;demonstration for exploring our results.&#10;" />
          <attvalue for="2" value="&#10;Explainable recommender systems have gained significant attention in recent years due to the need for transparency and interpretability in AI-driven decision-making processes \cite{zhang2020explainable}. Additionally, large language models (LLMs) have shown remarkable capabilities in various natural language processing tasks. These models possess unique characteristics, including reasoning ability and natural language text generation, making them attractive candidates for providing explanations in recommender systems \cite{wei2022emergent}.&#10;&#10;While LLMs have promising capabilities, employing them without adaptation to the downstream task presents several challenges. A significant issue is their lack of true personalization. Generic LLMs, when used directly, may fail to capture the unique preferences and nuances of individual users. As a result, the generated explanations may feel generic and fail to resonate with users on a personal level. These models often lack transparency in their decision-making process, making it difficult to trace the path of reasoning that led to a specific explanation. This hinders the ability to consistently reproduce the explanations, raising concerns about the reliability and trustworthiness of the recommender system. Moreover, without proper adaptation, LLMs may occasionally produce inappropriate explanations that conflict with ethical and social norms. These challenges emphasize the necessity of addressing the limitations of generic LLMs to ensure reliable, personalized, and responsible explainable recommender systems \cite{tamkin2021understanding}.&#10;&#10;In this work, we aim to tackle these issues by proposing a practical solution. We describe the process of identifying the characteristics of a good LLM-based explanation in \ref{sec:characteristics}. In \ref{sec:model}, we provide more details about how we utilize few-shot prompting \cite{brown2020language} as a means for aspect extraction and how chain-of-thought reasoning \cite{wei2022chain} is used as one of the main components of our framework. Finally, in \ref{sec:demo}, we introduce our interactive demonstration and explain how we generate examples in the domain of movie recommendations.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence Frameworks, Language Model Explanation, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1804.11192" label="1804.11192">
        <attvalues>
          <attvalue for="0" value="Explainable Recommendation: A Survey and New Perspectives" />
          <attvalue for="1" value="  Explainable recommendation attempts to develop models that generate not only&#10;high-quality recommendations but also intuitive explanations. The explanations&#10;may either be post-hoc or directly come from an explainable model (also called&#10;interpretable or transparent model in some contexts). Explainable&#10;recommendation tries to address the problem of why: by providing explanations&#10;to users or system designers, it helps humans to understand why certain items&#10;are recommended by the algorithm, where the human can either be users or system&#10;designers. Explainable recommendation helps to improve the transparency,&#10;persuasiveness, effectiveness, trustworthiness, and satisfaction of&#10;recommendation systems. It also facilitates system designers for better system&#10;debugging. In recent years, a large number of explainable recommendation&#10;approaches -- especially model-based methods -- have been proposed and applied&#10;in real-world systems.&#10;  In this survey, we provide a comprehensive review for the explainable&#10;recommendation research. We first highlight the position of explainable&#10;recommendation in recommender system research by categorizing recommendation&#10;problems into the 5W, i.e., what, when, who, where, and why. We then conduct a&#10;comprehensive survey of explainable recommendation on three perspectives: 1) We&#10;provide a chronological research timeline of explainable recommendation. 2) We&#10;provide a two-dimensional taxonomy to classify existing explainable&#10;recommendation research. 3) We summarize how explainable recommendation applies&#10;to different recommendation tasks. We also devote a chapter to discuss the&#10;explanation perspectives in broader IR and AI/ML research. We end the survey by&#10;discussing potential future directions to promote the explainable&#10;recommendation research area and beyond.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.09269" label="2408.09269">
        <attvalues>
          <attvalue for="0" value="Enhancing Audio-Language Models through Self-Supervised Post-Training&#10;  with Text-Audio Pairs" />
          <attvalue for="1" value="  Research on multi-modal contrastive learning strategies for audio and text&#10;has rapidly gained interest. Contrastively trained Audio-Language Models&#10;(ALMs), such as CLAP, which establish a unified representation across audio and&#10;language modalities, have enhanced the efficacy in various subsequent tasks by&#10;providing good text aligned audio encoders and vice versa. These improvements&#10;are evident in areas like zero-shot audio classification and audio retrieval,&#10;among others. However, the ability of these models to understand natural&#10;language and temporal relations is still a largely unexplored and open field&#10;for research. In this paper, we propose to equip the multi-modal ALMs with&#10;temporal understanding without loosing their inherent prior capabilities of&#10;audio-language tasks with a temporal instillation method TeminAL. We implement&#10;a two-stage training scheme TeminAL A $\&amp;$ B, where the model first learns to&#10;differentiate between multiple sounds in TeminAL A, followed by a phase that&#10;instills a sense of time, thereby enhancing its temporal understanding in&#10;TeminAL B. This approach results in an average performance gain of $5.28\%$ in&#10;temporal understanding on the ESC-50 dataset, while the model remains&#10;competitive in zero-shot retrieval and classification tasks on the&#10;AudioCap/Clotho datasets. We also note the lack of proper evaluation techniques&#10;for contrastive ALMs and propose a strategy for evaluating ALMs in zero-shot&#10;settings. The general-purpose zero-shot model evaluation strategy ZSTE, is used&#10;to evaluate various prior models. ZSTE demonstrates a general strategy to&#10;evaluate all ZS contrastive models. The model trained with TeminAL successfully&#10;outperforms current models on most downstream tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Audio, text, and images are among the most prevalent forms of information data. Developing models with multi-modal capabilities is well recognized as a path forward toward artificial general intelligence \cite{fei2022towards, huang2021makes}. In the field of multi-modal learning, contrastive learning has emerged as an effective strategy for training models on extensive, less-structured internet-sourced data \cite{radford2021learning, liang2022mind, tian2020makes}. Contrastive learning-based models have demonstrated exceptional adaptability across a range of related tasks, such as image classification \cite{chen2020simple, he2020momentum}, natural language processing \cite{gao2021simcse} and speech processing \cite{ravanelli2020multi}, making them a crucial area of research in multi-modal machine learning. One notable early model in this domain is CLIP, developed by \cite{radford2021learning}. CLIP learns the relationship between text and images, aligning them in a common latent domain. It stands out as a groundbreaking vision-language model, facilitating essential tasks such as formulating image captions \cite{mokady2021clipcap} and generating images from text \cite{rombach2022high}.&#10;&#10;Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}. Contrastive models generally excel in relating different modalities through their learned embedding and performing similarity-based retrieval tasks. These multi-context encoders integrate well with other downstream models, such as retrieval and open-ended generation models \cite{ramesh2021zero, li2022blip, yuan2021florence, singh2022flava}. However, previous authors have shown the limitations of audio-language models in truly understanding natural language while learning the relationship between texts and audio \cite{wu2023audio, ghosh2023compa}, an illustration demonstrating some of the concerned examples is shown in \ref{table1}. Critical applications like medical procedures, assembly instructions, commercial user applications, cooking instructions, and language learning may suffer from mistaken outputs in either text or audio settings.&#10;&#10;\cite{wu2023audio} reveal that current audio-language models (ALMs) are biased towards retrieving nouns and verbs, neglecting complete sentence context. They trained an ALM using captions stripped of all but nouns and verbs, which surprisingly performed as well or better than models trained on full, non-shuffled captions. This highlights that ALMs can excel in benchmarks without necessarily demonstrating compositional reasoning capabilities, challenging the importance of holistic sentence understanding in these models. Recent studies highlight limitations in models like CLIP's ability to understand language reasoning despite extensive training datasets \cite{thrush2022winoground, ma2023crepe, yuksekgonul2022and}, suggesting that the focus of contrastive pre-training on retrieval tasks allows such models to excel in retrieval-based benchmarks without a robust compositional grasp. In response, \cite{ghosh2023compa} critique existing audio-retrieval benchmarks, revealing ALMs' superficial success without true compositional understanding. They introduce CompA-CLAP, an evolved audio-language model enhanced with innovative contrastive training techniques which tries to instill language and attribution through their own modification of contrastive training.&#10;&#10;We approach the problem in similar way in the sense of modifying the contrastive training method, but we do it in a multi--stage hierarchical training which first makes the model understand multiple sound events and then tries to make it learn the temporality of the events. One reason for the suboptimal performance of contrastive learning loss is that it only includes similarities between text and audio pairs without attention to language (Proposition 1, \ref{pop}). &#10;To address the issues of contrastive model lacking the sense of time, we propose a novel self-supervised contrastive learning paradigm for post--training the model. &#10;Inspired by the work of \cite{bagad2023test} on instilling time in video-language models, we aim to improve ALMs' understanding of the relationship between modalities. The multi-step post--training process enables the model to grasp the complex task of time adaptation (details in \ref{method}). &#10;The work approaches the problem of time--instillation in two stages, as shown in \ref{fig1}, with details of methodology discussed in \ref{method}. &#10;&#10;Our work examines current zero-shot model evaluation methods, which often depend on basic similarity-based retrieval accuracies and single zero-shot techniques or utilize large language models (LLMs) as evaluators. These methods have demonstrated limitations and biases \cite{gao2024llm,jones2023cognitive,stureborg2024inconsistent,wang2023notfair}. Previous models have been evaluated for their test of time \cite{Shocher_2018_CVPR, Bau_2019, Kundu_2020_CVPR, Huang_2020_NeurIPS, Sun_2020_ICML, Liu_2021_NeurIPS}, but these evaluations never try to test the model's capability of general language and time--understanding. To address this, we propose a step-by-step method of zero-shot evaluation that poses sequentially complicated tasks for a general-purpose solution (details discussed in \ref{algo1}).&#10;&#10;Main contributions. The following are the main outlines of our contribution in this work, to the best of our knowledge the following have never been implemented in the current state of the art model:\\&#10;\begin{itemize}[noitemsep,nolistsep]&#10; \item We show the current ALMs fails to capture the correct temporal relation between audio and texts.&#10; \item We propose a general purpose two step post--training scheme TeminAL: Temporal Instillation in Audio-Language Models for multi-modal ALM models. The method aimed towards developing temporally aware contrastive audio and text encoders which can be employed in various close and open ended generation models as described in \ref{TeminAL}.&#10; \item We propose a general purpose scheme ZSTE: Zero Shot Temporal Evaluation of Zero-Shot evaluation for contrastive models. The sequentially complicated evaluation strategy can be extended to any zero-shot model as described in \ref{ZSTE}. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Audio-Language Models, Temporal Understanding, Linguistics, Cognitive Science, Artificial Intelligence, Signal Processing, Multi-Modal Learning, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2110.14378" label="2110.14378">
        <attvalues>
          <attvalue for="0" value="Towards artificial general intelligence via a multimodal foundation&#10;  model" />
          <attvalue for="1" value="  The fundamental goal of artificial intelligence (AI) is to mimic the core&#10;cognitive activities of human. Despite tremendous success in the AI research,&#10;most of existing methods have only single-cognitive ability. To overcome this&#10;limitation and take a solid step towards artificial general intelligence (AGI),&#10;we develop a foundation model pre-trained with huge multimodal data, which can&#10;be quickly adapted for various downstream cognitive tasks. To achieve this&#10;goal, we propose to pre-train our foundation model by self-supervised learning&#10;with weak semantic correlation data crawled from the Internet and show that&#10;promising results can be obtained on a wide range of downstream tasks.&#10;Particularly, with the developed model-interpretability tools, we demonstrate&#10;that strong imagination ability is now possessed by our foundation model. We&#10;believe that our work makes a transformative stride towards AGI, from our&#10;common practice of &quot;weak or narrow AI&quot; to that of &quot;strong or generalized AI&quot;.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.09734" label="2111.09734">
        <attvalues>
          <attvalue for="0" value="ClipCap: CLIP Prefix for Image Captioning" />
          <attvalue for="1" value="  Image captioning is a fundamental task in vision-language understanding,&#10;where the model predicts a textual informative caption to a given input image.&#10;In this paper, we present a simple approach to address this task. We use CLIP&#10;encoding as a prefix to the caption, by employing a simple mapping network, and&#10;then fine-tunes a language model to generate the image captions. The recently&#10;proposed CLIP model contains rich semantic features which were trained with&#10;textual context, making it best for vision-language perception. Our key idea is&#10;that together with a pre-trained language model (GPT2), we obtain a wide&#10;understanding of both visual and textual data. Hence, our approach only&#10;requires rather quick training to produce a competent captioning model. Without&#10;additional annotations or pre-training, it efficiently generates meaningful&#10;captions for large-scale and diverse datasets. Surprisingly, our method works&#10;well even when only the mapping network is trained, while both CLIP and the&#10;language model remain frozen, allowing a lighter architecture with less&#10;trainable parameters. Through quantitative evaluation, we demonstrate our model&#10;achieves comparable results to state-of-the-art methods on the challenging&#10;Conceptual Captions and nocaps datasets, while it is simpler, faster, and&#10;lighter. Our code is available in&#10;https://github.com/rmokady/CLIP_prefix_caption.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.14084" label="2109.14084">
        <attvalues>
          <attvalue for="0" value="VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text&#10;  Understanding" />
          <attvalue for="1" value="  We present VideoCLIP, a contrastive approach to pre-train a unified model for&#10;zero-shot video and text understanding, without using any labels on downstream&#10;tasks. VideoCLIP trains a transformer for video and text by contrasting&#10;temporally overlapping positive video-text pairs with hard negatives from&#10;nearest neighbor retrieval. Our experiments on a diverse series of downstream&#10;tasks, including sequence-level text-video retrieval, VideoQA, token-level&#10;action localization, and action segmentation reveal state-of-the-art&#10;performance, surpassing prior work, and in some cases even outperforming&#10;supervised approaches. Code is made available at&#10;https://github.com/pytorch/fairseq/tree/main/examples/MMPT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.00823" label="2205.00823">
        <attvalues>
          <attvalue for="0" value="CenterCLIP: Token Clustering for Efficient Text-Video Retrieval" />
          <attvalue for="1" value="  Recently, large-scale pre-training methods like CLIP have made great progress&#10;in multi-modal research such as text-video retrieval. In CLIP, transformers are&#10;vital for modeling complex multi-modal relations. However, in the vision&#10;transformer of CLIP, the essential visual tokenization process, which produces&#10;discrete visual token sequences, generates many homogeneous tokens due to the&#10;redundancy nature of consecutive and similar frames in videos. This&#10;significantly increases computation costs and hinders the deployment of video&#10;retrieval models in web applications. In this paper, to reduce the number of&#10;redundant video tokens, we design a multi-segment token clustering algorithm to&#10;find the most representative tokens and drop the non-essential ones. As the&#10;frame redundancy occurs mostly in consecutive frames, we divide videos into&#10;multiple segments and conduct segment-level clustering. Center tokens from each&#10;segment are later concatenated into a new sequence, while their original&#10;spatial-temporal relations are well maintained. We instantiate two clustering&#10;algorithms to efficiently find deterministic medoids and iteratively partition&#10;groups in high dimensional space. Through this token clustering and center&#10;selection procedure, we successfully reduce computation costs by removing&#10;redundant visual tokens. This method further enhances segment-level semantic&#10;alignment between video and text representations, enforcing the spatio-temporal&#10;interactions of tokens from within-segment frames. Our method, coined as&#10;CenterCLIP, surpasses existing state-of-the-art by a large margin on typical&#10;text-video benchmarks, while reducing the training memory cost by 35\% and&#10;accelerating the inference speed by 14\% at the best case. The code is&#10;available at&#10;\href{{https://github.com/mzhaoshuai/CenterCLIP}}{{https://github.com/mzhaoshuai/CenterCLIP}}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.05051" label="2212.05051">
        <attvalues>
          <attvalue for="0" value="VindLU: A Recipe for Effective Video-and-Language Pretraining" />
          <attvalue for="1" value="  The last several years have witnessed remarkable progress in&#10;video-and-language (VidL) understanding. However, most modern VidL approaches&#10;use complex and specialized model architectures and sophisticated pretraining&#10;protocols, making the reproducibility, analysis and comparisons of these&#10;frameworks difficult. Hence, instead of proposing yet another new VidL model,&#10;this paper conducts a thorough empirical study demystifying the most important&#10;factors in the VidL model design. Among the factors that we investigate are (i)&#10;the spatiotemporal architecture design, (ii) the multimodal fusion schemes,&#10;(iii) the pretraining objectives, (iv) the choice of pretraining data, (v)&#10;pretraining and finetuning protocols, and (vi) dataset and model scaling. Our&#10;empirical study reveals that the most important design factors include:&#10;temporal modeling, video-to-text multimodal fusion, masked modeling objectives,&#10;and joint training on images and videos. Using these empirical insights, we&#10;then develop a step-by-step recipe, dubbed VindLU, for effective VidL&#10;pretraining. Our final model trained using our recipe achieves comparable or&#10;better than state-of-the-art results on several VidL tasks without relying on&#10;external CLIP pretraining. In particular, on the text-to-video retrieval task,&#10;our approach obtains 61.2% on DiDeMo, and 55.0% on ActivityNet, outperforming&#10;current SOTA by 7.8% and 6.1% respectively. Furthermore, our model also obtains&#10;state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA,&#10;MSRVTT-MC and TVQA. Our code and pretrained models are publicly available at:&#10;https://github.com/klauscc/VindLU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.04850" label="2201.04850">
        <attvalues>
          <attvalue for="0" value="Bridging Video-text Retrieval with Multiple Choice Questions" />
          <attvalue for="1" value="  Pre-training a model to learn transferable video-text representation for&#10;retrieval has attracted a lot of attention in recent years. Previous dominant&#10;works mainly adopt two separate encoders for efficient retrieval, but ignore&#10;local associations between videos and texts. Another line of research uses a&#10;joint encoder to interact video with texts, but results in low efficiency since&#10;each text-video pair needs to be fed into the model. In this work, we enable&#10;fine-grained video-text interactions while maintaining high efficiency for&#10;retrieval via a novel pretext task, dubbed as Multiple Choice Questions (MCQ),&#10;where a parametric module BridgeFormer is trained to answer the &quot;questions&quot;&#10;constructed by the text features via resorting to the video features.&#10;Specifically, we exploit the rich semantics of text (i.e., nouns and verbs) to&#10;build questions, with which the video encoder can be trained to capture more&#10;regional content and temporal dynamics. In the form of questions and answers,&#10;the semantic associations between local video-text features can be properly&#10;established. BridgeFormer is able to be removed for downstream retrieval,&#10;rendering an efficient and flexible model with only two encoders. Our method&#10;outperforms state-of-the-art methods on the popular text-to-video retrieval&#10;task in five datasets with different experimental setups (i.e., zero-shot and&#10;fine-tune), including HowTo100M (one million videos). We further conduct&#10;zero-shot action recognition, which can be cast as video-to-text retrieval, and&#10;our approach also significantly surpasses its counterparts. As an additional&#10;benefit, our method achieves competitive results with much shorter pre-training&#10;videos on single-modality downstream tasks, e.g., action recognition with&#10;linear evaluation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.13043" label="2106.13043">
        <attvalues>
          <attvalue for="0" value="AudioCLIP: Extending CLIP to Image, Text and Audio" />
          <attvalue for="1" value="  In the past, the rapidly evolving field of sound classification greatly&#10;benefited from the application of methods from other domains. Today, we observe&#10;the trend to fuse domain-specific tasks and approaches together, which provides&#10;the community with new outstanding models.&#10;  In this work, we present an extension of the CLIP model that handles audio in&#10;addition to text and images. Our proposed model incorporates the ESResNeXt&#10;audio-model into the CLIP framework using the AudioSet dataset. Such a&#10;combination enables the proposed model to perform bimodal and unimodal&#10;classification and querying, while keeping CLIP's ability to generalize to&#10;unseen datasets in a zero-shot inference fashion.&#10;  AudioCLIP achieves new state-of-the-art results in the Environmental Sound&#10;Classification (ESC) task, out-performing other approaches by reaching&#10;accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets.&#10;Further it sets new baselines in the zero-shot ESC-task on the same datasets&#10;(68.78% and 69.40%, respectively).&#10;  Finally, we also assess the cross-modal querying performance of the proposed&#10;model as well as the influence of full and partial training on the results. For&#10;the sake of reproducibility, our code is published.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.10667" label="2303.10667">
        <attvalues>
          <attvalue for="0" value="Audio-Text Models Do Not Yet Leverage Natural Language" />
          <attvalue for="1" value="  Multi-modal contrastive learning techniques in the audio-text domain have&#10;quickly become a highly active area of research. Most works are evaluated with&#10;standard audio retrieval and classification benchmarks assuming that (i) these&#10;models are capable of leveraging the rich information contained in natural&#10;language, and (ii) current benchmarks are able to capture the nuances of such&#10;information. In this work, we show that state-of-the-art audio-text models do&#10;not yet really understand natural language, especially contextual concepts such&#10;as sequential or concurrent ordering of sound events. Our results suggest that&#10;existing benchmarks are not sufficient to assess these models' capabilities to&#10;match complex contexts from the audio and text modalities. We propose a&#10;Transformer-based architecture and show that, unlike prior work, it is capable&#10;of modeling the sequential relationship between sound events in the text and&#10;audio, given appropriate benchmark data. We advocate for the collection or&#10;generation of additional, diverse, data to allow future research to fully&#10;leverage natural language for audio-text modeling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.03162" label="2204.03162">
        <attvalues>
          <attvalue for="0" value="Winoground: Probing Vision and Language Models for Visio-Linguistic&#10;  Compositionality" />
          <attvalue for="1" value="  We present a novel task and dataset for evaluating the ability of vision and&#10;language models to conduct visio-linguistic compositional reasoning, which we&#10;call Winoground. Given two images and two captions, the goal is to match them&#10;correctly - but crucially, both captions contain a completely identical set of&#10;words, only in a different order. The dataset was carefully hand-curated by&#10;expert annotators and is labeled with a rich set of fine-grained tags to assist&#10;in analyzing model performance. We probe a diverse range of state-of-the-art&#10;vision and language models and find that, surprisingly, none of them do much&#10;better than chance. Evidently, these models are not as skilled at&#10;visio-linguistic compositional reasoning as we might have hoped. We perform an&#10;extensive analysis to obtain insights into how future work might try to&#10;mitigate these models' shortcomings. We aim for Winoground to serve as a useful&#10;evaluation set for advancing the state of the art and driving further progress&#10;in the field. The dataset is available at&#10;https://huggingface.co/datasets/facebook/winoground.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.02074" label="2301.02074">
        <attvalues>
          <attvalue for="0" value="Test of Time: Instilling Video-Language Models with a Sense of Time" />
          <attvalue for="1" value="  Modelling and understanding time remains a challenge in contemporary video&#10;understanding models. With language emerging as a key driver towards powerful&#10;generalization, it is imperative for foundational video-language models to have&#10;a sense of time. In this paper, we consider a specific aspect of temporal&#10;understanding: consistency of time order as elicited by before/after relations.&#10;We establish that seven existing video-language models struggle to understand&#10;even such simple temporal relations. We then question whether it is feasible to&#10;equip these foundational models with temporal awareness without re-training&#10;them from scratch. Towards this, we propose a temporal adaptation recipe on top&#10;of one such model, VideoCLIP, based on post-pretraining on a small amount of&#10;video-text data. We conduct a zero-shot evaluation of the adapted models on six&#10;datasets for three downstream tasks which require varying degrees of time&#10;awareness. We observe encouraging performance gains especially when the task&#10;needs higher time awareness. Our work serves as a first step towards probing&#10;and instilling a sense of time in existing video-language models without the&#10;need for data and compute-intense training from scratch.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.17012" label="2309.17012">
        <attvalues>
          <attvalue for="0" value="Benchmarking Cognitive Biases in Large Language Models as Evaluators" />
          <attvalue for="1" value="  Large Language Models (LLMs) have recently been shown to be effective as&#10;automatic evaluators with simple prompting and in-context learning. In this&#10;work, we assemble 15 LLMs of four different size ranges and evaluate their&#10;output responses by preference ranking from the other LLMs as evaluators, such&#10;as System Star is better than System Square. We then evaluate the quality of&#10;ranking outputs introducing the Cognitive Bias Benchmark for LLMs as Evaluators&#10;(CoBBLEr), a benchmark to measure six different cognitive biases in LLM&#10;evaluation outputs, such as the Egocentric bias where a model prefers to rank&#10;its own outputs highly in evaluation. We find that LLMs are biased text quality&#10;evaluators, exhibiting strong indications on our bias benchmark (average of 40%&#10;of comparisons across all models) within each of their evaluations that&#10;question their robustness as evaluators. Furthermore, we examine the&#10;correlation between human and machine preferences and calculate the average&#10;Rank-Biased Overlap (RBO) score to be 49.6%, indicating that machine&#10;preferences are misaligned with humans. According to our findings, LLMs may&#10;still be unable to be utilized for automatic annotation aligned with human&#10;preferences. Our project page is at: https://minnesotanlp.github.io/cobbler.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1712.06087" label="1712.06087">
        <attvalues>
          <attvalue for="0" value="&quot;Zero-Shot&quot; Super-Resolution using Deep Internal Learning" />
          <attvalue for="1" value="  Deep Learning has led to a dramatic leap in Super-Resolution (SR) performance&#10;in the past few years. However, being supervised, these SR methods are&#10;restricted to specific training data, where the acquisition of the&#10;low-resolution (LR) images from their high-resolution (HR) counterparts is&#10;predetermined (e.g., bicubic downscaling), without any distracting artifacts&#10;(e.g., sensor noise, image compression, non-ideal PSF, etc). Real LR images,&#10;however, rarely obey these restrictions, resulting in poor SR results by SotA&#10;(State of the Art) methods. In this paper we introduce &quot;Zero-Shot&quot; SR, which&#10;exploits the power of Deep Learning, but does not rely on prior training. We&#10;exploit the internal recurrence of information inside a single image, and train&#10;a small image-specific CNN at test time, on examples extracted solely from the&#10;input image itself. As such, it can adapt itself to different settings per&#10;image. This allows to perform SR of real old photos, noisy images, biological&#10;data, and other images where the acquisition process is unknown or non-ideal.&#10;On such images, our method outperforms SotA CNN-based SR methods, as well as&#10;previous unsupervised SR methods. To the best of our knowledge, this is the&#10;first unsupervised CNN-based SR method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.00184" label="2102.00184">
        <attvalues>
          <attvalue for="0" value="Adversarially learning disentangled speech representations for robust&#10;  multi-factor voice conversion" />
          <attvalue for="1" value="  Factorizing speech as disentangled speech representations is vital to achieve&#10;highly controllable style transfer in voice conversion (VC). Conventional&#10;speech representation learning methods in VC only factorize speech as speaker&#10;and content, lacking controllability on other prosody-related factors.&#10;State-of-the-art speech representation learning methods for more speechfactors&#10;are using primary disentangle algorithms such as random resampling and ad-hoc&#10;bottleneck layer size adjustment,which however is hard to ensure robust speech&#10;representationdisentanglement. To increase the robustness of highly&#10;controllable style transfer on multiple factors in VC, we propose a&#10;disentangled speech representation learning framework based on adversarial&#10;learning. Four speech representations characterizing content, timbre, rhythm&#10;and pitch are extracted, and further disentangled by an adversarial&#10;Mask-And-Predict (MAP)network inspired by BERT. The adversarial network is used&#10;tominimize the correlations between the speech representations,by randomly&#10;masking and predicting one of the representationsfrom the others. Experimental&#10;results show that the proposedframework significantly improves the robustness&#10;of VC on multiple factors by increasing the speech quality MOS from 2.79 to3.30&#10;and decreasing the MCD from 3.89 to 3.58.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.06791" label="1905.06791">
        <attvalues>
          <attvalue for="0" value="Almost Unsupervised Text to Speech and Automatic Speech Recognition" />
          <attvalue for="1" value="  Text to speech (TTS) and automatic speech recognition (ASR) are two dual&#10;tasks in speech processing and both achieve impressive performance thanks to&#10;the recent advance in deep learning and large amount of aligned speech and text&#10;data. However, the lack of aligned data poses a major practical problem for TTS&#10;and ASR on low-resource languages. In this paper, by leveraging the dual nature&#10;of the two tasks, we propose an almost unsupervised learning method that only&#10;leverages few hundreds of paired data and extra unpaired data for TTS and ASR.&#10;Our method consists of the following components: (1) a denoising auto-encoder,&#10;which reconstructs speech and text sequences respectively to develop the&#10;capability of language modeling both in speech and text domain; (2) dual&#10;transformation, where the TTS model transforms the text $y$ into speech&#10;$\hat{x}$, and the ASR model leverages the transformed pair $(\hat{x},y)$ for&#10;training, and vice versa, to boost the accuracy of the two tasks; (3)&#10;bidirectional sequence modeling, which addresses error propagation especially&#10;in the long speech and text sequence when training with few paired data; (4) a&#10;unified model structure, which combines all the above components for TTS and&#10;ASR based on Transformer model. Our method achieves 99.84% in terms of word&#10;level intelligible rate and 2.68 MOS for TTS, and 11.7% PER for ASR on LJSpeech&#10;dataset, by leveraging only 200 paired speech and text data (about 20 minutes&#10;audio), together with extra unpaired speech and text data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.01445" label="1805.01445">
        <attvalues>
          <attvalue for="0" value="The Fine Line between Linguistic Generalization and Failure in&#10;  Seq2Seq-Attention Models" />
          <attvalue for="1" value="  Seq2Seq based neural architectures have become the go-to architecture to&#10;apply to sequence to sequence language tasks. Despite their excellent&#10;performance on these tasks, recent work has noted that these models usually do&#10;not fully capture the linguistic structure required to generalize beyond the&#10;dense sections of the data distribution \cite{ettinger2017towards}, and as&#10;such, are likely to fail on samples from the tail end of the distribution (such&#10;as inputs that are noisy \citep{belkinovnmtbreak} or of different lengths&#10;\citep{bentivoglinmtlength}). In this paper, we look at a model's ability to&#10;generalize on a simple symbol rewriting task with a clearly defined structure.&#10;We find that the model's ability to generalize this structure beyond the&#10;training distribution depends greatly on the chosen random seed, even when&#10;performance on the standard test set remains the same. This suggests that a&#10;model's ability to capture generalizable structure is highly sensitive.&#10;Moreover, this sensitivity may not be apparent when evaluating it on standard&#10;test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.04879" label="1707.04879">
        <attvalues>
          <attvalue for="0" value="Listening while Speaking: Speech Chain by Deep Learning" />
          <attvalue for="1" value="  Despite the close relationship between speech perception and production,&#10;research in automatic speech recognition (ASR) and text-to-speech synthesis&#10;(TTS) has progressed more or less independently without exerting much mutual&#10;influence on each other. In human communication, on the other hand, a&#10;closed-loop speech chain mechanism with auditory feedback from the speaker's&#10;mouth to her ear is crucial. In this paper, we take a step further and develop&#10;a closed-loop speech chain model based on deep learning. The&#10;sequence-to-sequence model in close-loop architecture allows us to train our&#10;model on the concatenation of both labeled and unlabeled data. While ASR&#10;transcribes the unlabeled speech features, TTS attempts to reconstruct the&#10;original speech waveform based on the text from ASR. In the opposite direction,&#10;ASR also attempts to reconstruct the original text transcription given the&#10;synthesized speech. To the best of our knowledge, this is the first deep&#10;learning model that integrates human speech perception and production&#10;behaviors. Our experimental results show that the proposed approach&#10;significantly improved the performance more than separate systems that were&#10;only trained with labeled data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.00178" label="1805.00178">
        <attvalues>
          <attvalue for="0" value="Dynamic Sentence Sampling for Efficient Training of Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Traditional Neural machine translation (NMT) involves a fixed training&#10;procedure where each sentence is sampled once during each epoch. In reality,&#10;some sentences are well-learned during the initial few epochs; however, using&#10;this approach, the well-learned sentences would continue to be trained along&#10;with those sentences that were not well learned for 10-30 epochs, which results&#10;in a wastage of time. Here, we propose an efficient method to dynamically&#10;sample the sentences in order to accelerate the NMT training. In this approach,&#10;a weight is assigned to each sentence based on the measured difference between&#10;the training costs of two iterations. Further, in each epoch, a certain&#10;percentage of sentences are dynamically sampled according to their weights.&#10;Empirical results based on the NIST Chinese-to-English and the WMT&#10;English-to-German tasks depict that the proposed method can significantly&#10;accelerate the NMT training and improve the NMT performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently neural machine translation (NMT) has been prominently used to perform various translation tasks \cite{luong2015stanford,bojar-EtAl:2017:WMT1}. However, NMT is much more time-consuming than traditional phrase-based statistical machine translation (PBSMT) due to its deep neural network structure. To improve the efficiency of NMT training, most of the studies focus on reducing the number of parameters in the model \cite{DBLP:journals/corr/SeeLM16,DBLP:journals/corr/CregoKKRYSABCDE16,DBLP:journals/corr/HubaraCSEB16} and implementing parallelism in the data or in the model \cite{DBLP:journals/corr/WuSCLNMKCGMKSJL16,DBLP:journals/corr/KalchbrennerESO16,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}.&#10;&#10;Although these technologies have been adopted, deep networks have to be improved to achieve state-of-the-art performance in order to handle very large datasets and several training iterations. Therefore, some researchers have proposed to accelerate the NMT training by resampling a smaller subset of the data that makes a relatively high contribution, to improve the training efficiency of NMT. Specifically, \cite{DBLP:journals/corr/KocmiB17aa} empirically investigated curriculum learning based on the sentence length and word rank. \cite{wang-EtAl:2017:Short3} proposed a static sentence-selection method for domain adaptation using the internal sentence embedding of NMT. They also proposed a sentence weighting method with dynamic weight adjustment \cite{wang-EtAl:2017:EMNLP20174}. \cite{vanderwees-bisazza-monz:2017:EMNLP2017} used domain-based cross-entropy as a criterion to gradually fine-tune the NMT training in a dynamical manner. All of these criteria \cite{wang-EtAl:2017:Short3,wang-EtAl:2017:EMNLP20174,vanderwees-bisazza-monz:2017:EMNLP2017} are calculated before performing the NMT training based on the domain information and are fixed while performing the complete procedure. \cite{zhang-EtAl:2017:I17-2} adopted the sentence-level training cost as a dynamic criterion to gradually fine-tune the NMT training. This approach was developed based on the idea that the training cost is a useful measure to determine the translation quality of a sentence. However, some of the sentences that can be potentially improved by training may be deleted using this method. In addition, all of the above works primarily focused on NMT translation performance, instead of training efficiency.&#10;&#10;In this study, we propose a method of dynamic sentence sampling (DSS) to improve the NMT training efficiency. First, the differences between the training costs of two iterations, which is a measure of whether the translation quality of a sentence can be potentially improved, is measured to be the criterion. We further proposed two sentence resampling strategies, i.e., weighted sampling and review mechanism to help NMT focus on the not well-learned sentences as well as remember the knowledge from the well-learned sentences.&#10;&#10;The remainder of this paper is organized as follows. In Section \ref{sec:edss}, we introduce the dynamic sentence sampling method. Experiments are described and analyzed in Section \ref{sec:ex}. We discussed some other effects of the proposed methods in Section \ref{sec:dis}. We conclude our paper in the last section.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Training Efficiency Optimization, Dynamic Sampling Methods, Computer Science, Linguistics, Mathematics, Statistics, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1606.09274" label="1606.09274">
        <attvalues>
          <attvalue for="0" value="Compression of Neural Machine Translation Models via Pruning" />
          <attvalue for="1" value="  Neural Machine Translation (NMT), like many other deep learning domains,&#10;typically suffers from over-parameterization, resulting in large storage sizes.&#10;This paper examines three simple magnitude-based pruning schemes to compress&#10;NMT models, namely class-blind, class-uniform, and class-distribution, which&#10;differ in terms of how pruning thresholds are computed for the different&#10;classes of weights in the NMT architecture. We demonstrate the efficacy of&#10;weight pruning as a compression technique for a state-of-the-art NMT system. We&#10;show that an NMT model with over 200 million parameters can be pruned by 40%&#10;with very little performance loss as measured on the WMT'14 English-German&#10;translation task. This sheds light on the distribution of redundancy in the NMT&#10;architecture. Our main result is that with retraining, we can recover and even&#10;surpass the original performance with an 80%-pruned model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1609.07061" label="1609.07061">
        <attvalues>
          <attvalue for="0" value="Quantized Neural Networks: Training Neural Networks with Low Precision&#10;  Weights and Activations" />
          <attvalue for="1" value="  We introduce a method to train Quantized Neural Networks (QNNs) --- neural&#10;networks with extremely low precision (e.g., 1-bit) weights and activations, at&#10;run-time. At train-time the quantized weights and activations are used for&#10;computing the parameter gradients. During the forward pass, QNNs drastically&#10;reduce memory size and accesses, and replace most arithmetic operations with&#10;bit-wise operations. As a result, power consumption is expected to be&#10;drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and&#10;ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to&#10;their 32-bit counterparts. For example, our quantized version of AlexNet with&#10;1-bit weights and 2-bit activations achieves $51\%$ top-1 accuracy. Moreover,&#10;we quantize the parameter gradients to 6-bits as well which enables gradients&#10;computation using only bit-wise operation. Quantized recurrent neural networks&#10;were tested over the Penn Treebank dataset, and achieved comparable accuracy as&#10;their 32-bit counterparts using only 4-bits. Last but not least, we programmed&#10;a binary matrix multiplication GPU kernel with which it is possible to run our&#10;MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering&#10;any loss in classification accuracy. The QNN code is available online.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.10099" label="1610.10099">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation in Linear Time" />
          <attvalue for="1" value="  We present a novel neural network for processing sequences. The ByteNet is a&#10;one-dimensional convolutional neural network that is composed of two parts, one&#10;to encode the source sequence and the other to decode the target sequence. The&#10;two network parts are connected by stacking the decoder on top of the encoder&#10;and preserving the temporal resolution of the sequences. To address the&#10;differing lengths of the source and the target, we introduce an efficient&#10;mechanism by which the decoder is dynamically unfolded over the representation&#10;of the encoder. The ByteNet uses dilation in the convolutional layers to&#10;increase its receptive field. The resulting network has two core properties: it&#10;runs in time that is linear in the length of the sequences and it sidesteps the&#10;need for excessive memorization. The ByteNet decoder attains state-of-the-art&#10;performance on character-level language modelling and outperforms the previous&#10;best results obtained with recurrent networks. The ByteNet also achieves&#10;state-of-the-art performance on character-to-character machine translation on&#10;the English-to-German WMT translation task, surpassing comparable neural&#10;translation models that are based on recurrent networks with attentional&#10;pooling and run in quadratic time. We find that the latent alignment structure&#10;contained in the representations reflects the expected alignment between the&#10;tokens.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1707.09533" label="1707.09533">
        <attvalues>
          <attvalue for="0" value="Curriculum Learning and Minibatch Bucketing in Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  We examine the effects of particular orderings of sentence pairs on the&#10;on-line training of neural machine translation (NMT). We focus on two types of&#10;such orderings: (1) ensuring that each minibatch contains sentences similar in&#10;some aspect and (2) gradual inclusion of some sentence types as the training&#10;progresses (so called &quot;curriculum learning&quot;). In our English-to-Czech&#10;experiments, the internal homogeneity of minibatches has no effect on the&#10;training but some of our &quot;curricula&quot; achieve a small improvement over the&#10;baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1612.06138" label="1612.06138">
        <attvalues>
          <attvalue for="0" value="Boosting Neural Machine Translation" />
          <attvalue for="1" value="  Training efficiency is one of the main problems for Neural Machine&#10;Translation (NMT). Deep networks need for very large data as well as many&#10;training iterations to achieve state-of-the-art performance. This results in&#10;very high computation cost, slowing down research and industrialisation. In&#10;this paper, we propose to alleviate this problem with several training methods&#10;based on data boosting and bootstrap with no modifications to the neural&#10;network. It imitates the learning process of humans, which typically spend more&#10;time when learning &quot;difficult&quot; concepts than easier ones. We experiment on an&#10;English-French translation task showing accuracy improvements of up to 1.63&#10;BLEU while saving 20% of training time.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.11660" label="2204.11660">
        <attvalues>
          <attvalue for="0" value="A Survey on Word Meta-Embedding Learning" />
          <attvalue for="1" value="  Meta-embedding (ME) learning is an emerging approach that attempts to learn&#10;more accurate word embeddings given existing (source) word embeddings as the&#10;sole input.&#10;  Due to their ability to incorporate semantics from multiple source embeddings&#10;in a compact manner with superior performance, ME learning has gained&#10;popularity among practitioners in NLP.&#10;  To the best of our knowledge, there exist no prior systematic survey on ME&#10;learning and this paper attempts to fill this need.&#10;  We classify ME learning methods according to multiple factors such as whether&#10;they (a) operate on static or contextualised embeddings, (b) trained in an&#10;unsupervised manner or (c) fine-tuned for a particular task/domain.&#10;  Moreover, we discuss the limitations of existing ME learning methods and&#10;highlight potential future research directions.&#10;" />
          <attvalue for="2" value="&#10;&#10;Given independently trained multiple word representations (aka embeddings) learnt using diverse algorithms and lexical resources, word meta-embedding (ME) learning methods~\cite{Yin:ACL:2016,Bao:COLING:2018,Bollegala:IJCAI:2018,wu2020task,He:2020,jawanpuria-etal-2020-learning,Coates:NAACL:2018} attempt to learn more accurate and wide-coverage word embeddings. &#10;The input and output word embeddings to the ME algorithm are referred respectively as the source and meta-embeddings.&#10;ME has emerged as a promising ensemble approach to combine diverse pretrained source word embeddings to preserve their complementary strengths.&#10;&#10;The problem settings of ME learning differ from that of source embedding learning in important ways as follows.&#10;1. ME methods must be agnostic to the methods used to train the source embeddings.\\&#10;The source embeddings used as the inputs to an ME learning method can be trained using different methods.&#10;For example, context-insensitive static word embedding methods~\cite{Dhillon:2015,Mnih:HLBL:NIPS:2008,Collobert:2011,Huang:ACL:2012,Milkov:2013,Pennington:EMNLP:2014} represent a word by a single vector that does not vary depending on the context in which the word occurs.&#10;On the other hand, contextualised word embedding methods~\cite{Elmo,BERT,XLNET,ALBERT,RoBERTa} represent the same word with different embeddings in its different contexts.&#10;It is not clear beforehand which word embedding is best for a particular NLP task.&#10;By being agnostic to the underlying differences in the source embedding learning methods, ME learning methods are in principle able to incorporate a wide range of source word embeddings.&#10;Moreover, this decoupling of source embedding learning from the ME learning simplifies the latter.&#10;&#10;2. ME methods must not assume access to the original training resources used to train the source embeddings.\\&#10;Source embeddings can be trained using different linguistic resources such as text corpora or dictionaries~\cite{tissier-gravier-habrard:2017:EMNLP2017,Alsuhaibani:AKBC:2019,Bollegala:AAAI:2016}.&#10;Although pretrained word embeddings are often publicly released and are free of charge to use, the resources on which those embeddings were originally trained might not be publicly available due to copyright and licensing restrictions. &#10;Consequently, ME methods have not assumed the access to the original training resources that were used to train the source embeddings.&#10;Therefore, an ME method must obtain all semantic information of words directly from the source embeddings.&#10;&#10;3. ME methods must be able to handle pretrained word embeddings of different dimensionalities.\\&#10;Because ME methods operate directly on pretrained source word embeddings without retraining them on linguistic resources, the dimensionalities of the source word embeddings are often different.&#10;Prior work~\cite{Yin:2018,Levy:TACL:2015} studying word embeddings have shown that the performance of a static word embedding is directly influenced by its dimensionality.&#10;ME learning methods use different techniques such as concatenation~\cite{Yin:ACL:2016}, orthogonal projections~\cite{jawanpuria-etal-2020-learning,He:2020} and averaging~\cite{Coates:NAACL:2018} after applying zero-padding to the sources with smaller dimensionalities as necessary to handle source embeddings with different dimensionalities.&#10;&#10;Applications of ME:&#10;ME learning is attractive from an NLP practitioners point for several reasons.&#10;First, as mentioned above, there is already a large number of pretrained and publicly available repositories of static and contextialised word embeddings.&#10;However, it is not readily obvious what is the best word embedding method to represent the input in a particular NLP application.&#10;We might not be able to try each and every source embedding due to time or computational constraints.&#10;ME learning provides a convenient alternative to selecting the single best word embedding, where we can use a ME trained from all available source embeddings.&#10;Second, unsupervised ME learning methods (\ref{sec:unsup}) do not require labelled data when creating an ME from a given set of source word embeddings.&#10;This is particularly attractive in scenarios where we do not have sufficiently large training resources for learning word embeddings from scratch but have access to multiple pretrained word embeddings.&#10;Moreover, by using multiple source embeddings we might be able to overcome the limited vocabulary coverage in the individual sources.&#10;Third, in situations where there is some labelled data for the target task or domain, we can use supervised ME learning methods (\ref{sec:sup}) to fine-tune the MEs for the target task or domain.&#10;&#10;From a theoretical point-of-view, ME learning can be seen as an instance of ensemble learning~\cite{EnsembleLearning,Polikar_2012}, where we incorporate information from multiple models of lexical (word-level) semantics to learn an improved representation model.&#10;An ensemble typically helps to cancel out noise in individual models, while reinforcing the useful patterns repeated in multiple models~\cite{muromagi-etal-2017-linear}. &#10;Although there are some theoretical work studying word embedding learning~\cite{Arora:TACL:2016,All-but-Top,Bollegala:AAAI:2018}, the theoretical analysis of ME learning has been under-developed, with the exception of concatenated meta-embeddings~\cite{Bollegala:IJCAIa:2022}.&#10;For example, under what conditions can we learn a better ME than individual source embeddings is an important theoretical consideration.&#10;ME learning can also been seen as related to model distillation~\cite{46642,Hinton:2015} where we must learn a simpler student model from a more complicated teacher model.&#10;Model distillation is an actively researched topic in deep learning where it is attractive to learn smaller networks involving a lesser number of parameters from a larger network to avoid overfitting and inference-time efficiency.&#10;&#10;In this survey paper we focus on word-level ME learning.&#10;We first define the ME problem (\ref{sec:definition}) and cover unsupervised (\ref{sec:unsup}) and supervised (\ref{sec:sup}) ME methods.&#10;We also look at multilingual ME in \ref{sec:multi}.&#10;Finally, we discuss the performance of different ME methods (\ref{sec:eval}) and present potential future research directions (\ref{sec:issues}).&#10;Moreover, we publicly release a ME framework that implements several ME methods covered in this paper, which we believe would be useful to further enhance the readers' understanding on this emerging topic.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Meta-Learning Methods, Linguistics, Word Embeddings, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1709.06671" label="1709.06671">
        <attvalues>
          <attvalue for="0" value="Think Globally, Embed Locally --- Locally Linear Meta-embedding of Words" />
          <attvalue for="1" value="  Distributed word embeddings have shown superior performances in numerous&#10;Natural Language Processing (NLP) tasks. However, their performances vary&#10;significantly across different tasks, implying that the word embeddings learnt&#10;by those methods capture complementary aspects of lexical semantics. Therefore,&#10;we believe that it is important to combine the existing word embeddings to&#10;produce more accurate and complete \emph{meta-embeddings} of words. For this&#10;purpose, we propose an unsupervised locally linear meta-embedding learning&#10;method that takes pre-trained word embeddings as the input, and produces more&#10;accurate meta embeddings. Unlike previously proposed meta-embedding learning&#10;methods that learn a global projection over all words in a vocabulary, our&#10;proposed method is sensitive to the differences in local neighbourhoods of the&#10;individual source word embeddings. Moreover, we show that vector concatenation,&#10;a previously proposed highly competitive baseline approach for integrating word&#10;embeddings, can be derived as a special case of the proposed method.&#10;Experimental results on semantic similarity, word analogy, relation&#10;classification, and short-text classification tasks show that our&#10;meta-embeddings to significantly outperform prior methods in several benchmark&#10;datasets, establishing a new state of the art for meta-embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.05262" label="1804.05262">
        <attvalues>
          <attvalue for="0" value="Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by&#10;  Averaging Source Word Embeddings" />
          <attvalue for="1" value="  Creating accurate meta-embeddings from pre-trained source embeddings has&#10;received attention lately. Methods based on global and locally-linear&#10;transformation and concatenation have shown to produce accurate&#10;meta-embeddings. In this paper, we show that the arithmetic mean of two&#10;distinct word embedding sets yields a performant meta-embedding that is&#10;comparable or better than more complex meta-embedding learning methods. The&#10;result seems counter-intuitive given that vector spaces in different source&#10;embeddings are not comparable and cannot be simply averaged. We give insight&#10;into why averaging can still produce accurate meta-embedding despite the&#10;incomparability of the source vector spaces.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1511.06438" label="1511.06438">
        <attvalues>
          <attvalue for="0" value="Joint Word Representation Learning using a Corpus and a Semantic Lexicon" />
          <attvalue for="1" value="  Methods for learning word representations using large text corpora have&#10;received much attention lately due to their impressive performance in numerous&#10;natural language processing (NLP) tasks such as, semantic similarity&#10;measurement, and word analogy detection. Despite their success, these&#10;data-driven word representation learning methods do not consider the rich&#10;semantic relational structure between words in a co-occurring context. On the&#10;other hand, already much manual effort has gone into the construction of&#10;semantic lexicons such as the WordNet that represent the meanings of words by&#10;defining the various relationships that exist among the words in a language. We&#10;consider the question, can we improve the word representations learnt using a&#10;corpora by integrating the knowledge from semantic lexicons?. For this purpose,&#10;we propose a joint word representation learning method that simultaneously&#10;predicts the co-occurrences of two words in a sentence subject to the&#10;relational constrains given by the semantic lexicon. We use relations that&#10;exist between words in the lexicon to regularize the word representations&#10;learnt from the corpus. Our proposed method statistically significantly&#10;outperforms previously proposed methods for incorporating semantic lexicons&#10;into word representations on several benchmark datasets for semantic similarity&#10;and word analogy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.04224" label="1812.04224">
        <attvalues>
          <attvalue for="0" value="On the Dimensionality of Word Embedding" />
          <attvalue for="1" value="  In this paper, we provide a theoretical understanding of word embedding and&#10;its dimensionality. Motivated by the unitary-invariance of word embedding, we&#10;propose the Pairwise Inner Product (PIP) loss, a novel metric on the&#10;dissimilarity between word embeddings. Using techniques from matrix&#10;perturbation theory, we reveal a fundamental bias-variance trade-off in&#10;dimensionality selection for word embeddings. This bias-variance trade-off&#10;sheds light on many empirical observations which were previously unexplained,&#10;for example the existence of an optimal dimensionality. Moreover, new insights&#10;and discoveries, like when and how word embeddings are robust to over-fitting,&#10;are revealed. By optimizing over the bias-variance trade-off of the PIP loss,&#10;we can explicitly answer the open question of dimensionality selection for word&#10;embedding.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.01419" label="1704.01419">
        <attvalues>
          <attvalue for="0" value="Linear Ensembles of Word Embedding Models" />
          <attvalue for="1" value="  This paper explores linear methods for combining several word embedding&#10;models into an ensemble. We construct the combined models using an iterative&#10;method based on either ordinary least squares regression or the solution to the&#10;orthogonal Procrustes problem.&#10;  We evaluate the proposed approaches on Estonian---a morphologically complex&#10;language, for which the available corpora for training word embeddings are&#10;relatively small. We compare both combined models with each other and with the&#10;input word embedding models using synonym and analogy tests. The results show&#10;that while using the ordinary least squares regression performs poorly in our&#10;experiments, using orthogonal Procrustes to combine several word embedding&#10;models into an ensemble model leads to 7-10% relative improvements over the&#10;mean result of the initial models in synonym tests and 19-47% in analogy tests.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.12386" label="2204.12386">
        <attvalues>
          <attvalue for="0" value="Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of&#10;  Source Embeddings" />
          <attvalue for="1" value="  Given multiple source word embeddings learnt using diverse algorithms and&#10;lexical resources, meta word embedding learning methods attempt to learn more&#10;accurate and wide-coverage word embeddings.&#10;  Prior work on meta-embedding has repeatedly discovered that simple vector&#10;concatenation of the source embeddings to be a competitive baseline.&#10;  However, it remains unclear as to why and when simple vector concatenation&#10;can produce accurate meta-embeddings.&#10;  We show that weighted concatenation can be seen as a spectrum matching&#10;operation between each source embedding and the meta-embedding, minimising the&#10;pairwise inner-product loss.&#10;  Following this theoretical analysis, we propose two \emph{unsupervised}&#10;methods to learn the optimal concatenation weights for creating meta-embeddings&#10;from a given set of source embeddings.&#10;  Experimental results on multiple benchmark datasets show that the proposed&#10;weighted concatenated meta-embedding methods outperform previously proposed&#10;meta-embedding learning methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.03235" label="1804.03235">
        <attvalues>
          <attvalue for="0" value="Large scale distributed neural network training through online&#10;  distillation" />
          <attvalue for="1" value="  Techniques such as ensembling and distillation promise model quality&#10;improvements when paired with almost any base model. However, due to increased&#10;test-time cost (for ensembles) and increased complexity of the training&#10;pipeline (for distillation), these techniques are challenging to use in&#10;industrial settings. In this paper we explore a variant of distillation which&#10;is relatively straightforward to use as it does not require a complicated&#10;multi-stage setup or many new hyperparameters. Our first claim is that online&#10;distillation enables us to use extra parallelism to fit very large datasets&#10;about twice as fast. Crucially, we can still speed up training even after we&#10;have already reached the point at which additional parallelism provides no&#10;benefit for synchronous or asynchronous stochastic gradient descent. Two neural&#10;networks trained on disjoint subsets of the data can share knowledge by&#10;encouraging each model to agree with the predictions the other model would have&#10;made. These predictions can come from a stale version of the other model so&#10;they can be safely computed using weights that only rarely get transmitted. Our&#10;second claim is that online distillation is a cost-effective way to make the&#10;exact predictions of a model dramatically more reproducible. We support our&#10;claims using experiments on the Criteo Display Ad Challenge dataset, ImageNet,&#10;and the largest to-date dataset used for neural language modeling, containing&#10;$6\times 10^{11}$ tokens and based on the Common Crawl repository of web data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.12226" label="2408.12226">
        <attvalues>
          <attvalue for="0" value="EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for&#10;  Automated Scoring of CEFR B2 Speaking Assessment Transcripts" />
          <attvalue for="1" value="  Relying on human experts to evaluate CEFR speaking assessments in an&#10;e-learning environment creates scalability challenges, as it limits how quickly&#10;and widely assessments can be conducted. We aim to automate the evaluation of&#10;CEFR B2 English speaking assessments in e-learning environments from&#10;conversation transcripts. First, we evaluate the capability of leading open&#10;source and commercial Large Language Models (LLMs) to score a candidate's&#10;performance across various criteria in the CEFR B2 speaking exam in both global&#10;and India-specific contexts. Next, we create a new expert-validated,&#10;CEFR-aligned synthetic conversational dataset with transcripts that are rated&#10;at different assessment scores. In addition, new instruction-tuned datasets are&#10;developed from the English Vocabulary Profile (up to CEFR B2 level) and the&#10;CEFR-SP WikiAuto datasets. Finally, using these new datasets, we perform&#10;parameter efficient instruction tuning of Mistral Instruct 7B v0.2 to develop a&#10;family of models called EvalYaks. Four models in this family are for assessing&#10;the four sections of the CEFR B2 speaking exam, one for identifying the CEFR&#10;level of vocabulary and generating level-specific vocabulary, and another for&#10;detecting the CEFR level of text and generating level-specific text. EvalYaks&#10;achieved an average acceptable accuracy of 96%, a degree of variation of 0.35&#10;levels, and performed 3 times better than the next best model. This&#10;demonstrates that a 7B parameter LLM instruction tuned with high-quality&#10;CEFR-aligned assessment data can effectively evaluate and score CEFR B2 English&#10;speaking assessments, offering a promising solution for scalable, automated&#10;language proficiency evaluation.&#10;" />
          <attvalue for="2" value="&#10;&#10;The study of English is classified into Academic and Functional (General) English, each fulfilling different use cases. Academic English, prevalent in professional and educational spheres such as universities, prioritizes a formal tone, organized writing, and precise vocabulary for tasks such as essays, reports, and scholarly communication. In contrast, Functional English targets everyday communication skills in speaking, listening, reading, and writing, aiming for practical application in social and personal interactions with a more informal, conversational approach \cite{north2010core}.&#10;&#10;The Common European Framework of Reference for Languages (CEFR) evaluates English proficiency on a six-level scale from A1 (Beginner) to C2 (Advanced). A CEFR B2 qualification indicates that the learner has the ability to independently use English to live, work, or study \cite{council2001common}. The CEFR uses `can do' descriptors to tailor teaching and assessment, aligning curriculum and educational objectives. These descriptors help educators set communicative goals and adapt courses to specific learning needs through consultations with experts and stakeholders \cite{cefr_companion}. Learners often have a `spiky profile', excelling in some language skills but struggling in others, reflecting their Target Language Use (TLU). For example, the IELTS Life Skills Test \cite{ielts} assesses only speaking and listening skills for UK visa applicants. The CEFR framework requires adaptation to fit specific contexts and is not a universal solution. Its `can do' statements define TLUs such as Personal, Public, Occupational, and Educational, facilitating customized teaching and assessment \cite{cambridgeenglish2018cefr}. E-learning can create a comprehensive curriculum customized to individual preferences and abilities, effectively using the CEFR framework to meet specific needs of the learner and contextual demands.&#10;&#10;Various organizations, including Cambridge English and the British Council, offer English language programs and assessments. Cambridge English conducts in-person exams in three categories: Schools, General and Higher Education, and Business. The General and Higher Education exams cater to career and academic requirements at five levels: A2 Key, B1 Preliminary, B2 First, C1 Advanced, and C2 Proficiency. The B2 First exam is crucial for showcasing communication skills in English-speaking environments. On the other hand, the British Council's EnglishScore \cite{englishscore2023validity} provides a more straightforward, mobile-based online assessment for general English users.&#10;&#10;Our objective is to develop a range of models to automate the evaluation and scoring process with the ability to handle the complexities of advanced language examinations such as the B2 First. Using both international and India-specific data, we strive to improve the accuracy and relevance of assessments, particularly for Indian students, in both global and local contexts.&#10;&#10;E-learning is uniquely positioned to address the preferences and capabilities of learners, particularly as the complexity of subjects increases \cite{morris2008economies}. It is especially beneficial for subjects that require in-depth knowledge and detailed responses. In this context, depending on human experts to evaluate every assessment can be extremely costly and does not scale efficiently \cite{shah2014some}. The adoption of technologies that can emulate the expertise of human assessors and automate the evaluation process offers a feasible solution \cite{mekterovic2023scaling}. Implementing technologies that replicate human expertise and automate evaluations could provide a solution, facilitating effective and unbiased teaching and assessment of complex topics, thereby making quality education more scalable and accessible. &#10;&#10;Artificial intelligence (AI), including Generative AI (GenAI) technology, has made significant advancements in various domains, including education, healthcare, and scientific research. GenAI models \cite{genai} can produce diverse content such as text, images, and videos using AI techniques. The integration of AI in education is increasing, albeit at a slower pace compared to other industries, yet its potential influence on education is substantial \cite{chatgpt_education}. These technologies enable personalized educational materials, assessments, and tutoring \cite{ji2023systematic}. Research \cite{AIassistance} has indicated that incorporating AI into educational settings has the potential to enhance students' autonomy in managing their own learning processes. Conversational AI, which conducts human-like conversations via text or audio based on LLMs, is widely accepted among students for task-oriented dialogues \cite{ji2023systematic}. Tools like Google Assistant have improved EFL students' communication skills and attitudes towards intelligent assistants in learning \cite{tai2023impact}. Chatbots providing feedback have effectively boosted vocabulary learning among Korean EFL primary students \cite{jeon2023chatbot}.&#10;&#10;For a long time, researchers have been developing automated language assessments to efficiently and accurately evaluate the abilities of English learners. SpeechRater$^SM$ Version 1.0 (v1.0) \cite{xi2008automated} is an automated system created to score the spontaneous speech of English learners, and it is operationally used in the Test of English as a Foreign Language$^TM$ (TOEFL\textregistered) Practice Online assessment. Students find automated scoring of speaking performance and feedback to be beneficial \cite{gu2021using}. Recent advancements include the implementation of transformer-based models like BERT \cite{rama2021pre} for evaluating CEFR levels of sentences, and GPT variants \cite{gpt4} for assessing essays written by L2 English learners \cite{yancey2023rating}. These tools aim to provide unbiased, globally applicable evaluations and focus on aligning content with recognized proficiency levels.&#10;&#10;The primary aim of this paper is to create an automated system to evaluate a candidate's performance in the CEFR B2 English speaking test, removing the need for a human assessor and ensuring relevance in both global and Indian contexts. Secondary aims involve developing an automated system capable of identifying vocabulary and proficiency levels. Furthermore, this model should generate CEFR B2 vocabulary and sentences. We introduced EvalYaks, a set of six unique models, with four models dedicated to the primary aim and two to the secondary. EvalYaks is trained by instruction tuning Mistral Instruct 7B v0.2 using the Low Rank Adaptation (LoRA), a parameter efficient fine tuning (PEFT) method. &#10;&#10;Due to the scarcity of CEFR B2 English speaking assessment data specific to India, we generated a data set of simulated candidate responses and their evaluation scores using GPT-4. The data set was then refined with expert human feedback.&#10;&#10;As a baseline, we first investigated the ability of leading LLMs, which are highly ranked on the LMSYS leaderboard \cite{lmsys}, to directly evaluate candidate responses by leveraging their intrinsic knowledge and prompt engineering. In addition, we examined the ability of LLMs to comprehend sentence structures and link vocabulary with CEFR proficiency levels. Our EvalYaks suite of instruction fine-tuned (LoRA) models demonstrated superior performance in these tasks compared to standard LLMs. In our experiments, for each scenario, we used two sets of evaluation instructions for the LLM: one with contextual information and one without. Standard LLMs are not capable of satisfactory evaluation. Our findings indicate that a 7B parameter LLM, when instruction-tuned with specific, high-quality CEFR-aligned assessment data, can be employed for automated evaluation and scoring of CEFR B2 English speaking assessments.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, E-Learning Scalability Solutions, Artificial Intelligence, Education, Natural Language Processing, Large Language Models Evaluation, Language Assessment Automation" />
        </attvalues>
      </node>
      <node id="2402.07158" label="2402.07158">
        <attvalues>
          <attvalue for="0" value="Effort and Size Estimation in Software Projects with Large Language&#10;  Model-based Intelligent Interfaces" />
          <attvalue for="1" value="  The advancement of Large Language Models (LLM) has also resulted in an&#10;equivalent proliferation in its applications. Software design, being one, has&#10;gained tremendous benefits in using LLMs as an interface component that extends&#10;fixed user stories. However, inclusion of LLM-based AI agents in software&#10;design often poses unexpected challenges, especially in the estimation of&#10;development efforts. Through the example of UI-based user stories, we provide a&#10;comparison against traditional methods and propose a new way to enhance&#10;specifications of natural language-based questions that allows for the&#10;estimation of development effort by taking into account data sources,&#10;interfaces and algorithms.&#10;" />
          <attvalue for="2" value="&#10;The acceleration of LLM model development and their visibility have prompted the genesis of many LLM-based products. Recently, the release of ChatGPT~\cite{chatgpt2022,GPT4TechnicalReport} was a milestone that signaled a significant shift in society, including changes in software design paradigms. Initially, LLMs~\cite{Huyen2023} like ChatGPT revolutionized the field with advanced chatbots and AI Agents~\cite{wolfram2023}, enhancing the ability of these models by connecting data sources, algorithms and visualizations to LLMs. However, in recent months many have witnessed a transition towards more sophisticated systems such as Retrieval-Augmented Generation (RAG)~\cite{lewis2021retrievalaugmented} and AI Agents~\cite{wolfram2023}.&#10;&#10;Although more recent LLMs~\cite{openai2023gpt4,touvron2023llama,jiang2024mixtral} have the capability to do data analysis and even data summarization and representation, the ability to connect to external data sources, algorithms and specialized interfaces to LLMs~\cite{nexusflow2023} adds additional flexibility to LLMs by enabling it to perform tasks that involves analysis of domain specific real time data, or even the possibility to perform tasks that are still beyond LLM's capabilities.&#10;&#10;This paper discusses the changes in software design using AI Agents, specifically, the shift from traditional UI/UX user stories~\cite{userstories:2004} in software design to LLM-based AI Agent interfaces implementing several user stories using a single natural language interface. This transition represents a paradigm shift from well-structured documentation of data sources, UI/UX interactions, and algorithms, where you can reasonably well estimate size and effort of development, to a more flexible, albeit imprecise, mode of interaction through natural language descriptions. While this shift has unlocked unprecedented levels of user accessibility and software adaptability, it has also introduced unique challenges. One of the most fundamental questions that we intend to address in this paper is on how to estimate the development effort and size of these new systems, where the LLM interacts with the user sometimes in unknown ways.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Development Effort Estimation, Computer Science, Large Language Models, Linguistics, Engineering, Software Design, Information Systems" />
        </attvalues>
      </node>
      <node id="2403.09490" label="2403.09490">
        <attvalues>
          <attvalue for="0" value="Hyper-CL: Conditioning Sentence Representations with Hypernetworks" />
          <attvalue for="1" value="  While the introduction of contrastive learning frameworks in sentence&#10;representation learning has significantly contributed to advancements in the&#10;field, it still remains unclear whether state-of-the-art sentence embeddings&#10;can capture the fine-grained semantics of sentences, particularly when&#10;conditioned on specific perspectives. In this paper, we introduce Hyper-CL, an&#10;efficient methodology that integrates hypernetworks with contrastive learning&#10;to compute conditioned sentence representations. In our proposed approach, the&#10;hypernetwork is responsible for transforming pre-computed condition embeddings&#10;into corresponding projection layers. This enables the same sentence embeddings&#10;to be projected differently according to various conditions. Evaluation on two&#10;representative conditioning benchmarks, namely conditional semantic text&#10;similarity and knowledge graph completion, demonstrates that Hyper-CL is&#10;effective in flexibly conditioning sentence representations, showcasing its&#10;computational efficiency at the same time. We also provide a comprehensive&#10;analysis of the inner workings of our approach, leading to a better&#10;interpretation of its mechanisms.&#10;" />
          <attvalue for="2" value=" &#10;&#10;Building upon the established correlation between language model performance and computational capacity \cite{kaplan2020scaling}, there has emerged an undeniable trend towards the adoption of ever-larger language models across a diverse range of NLP applications.&#10;This trend is also evident in the computation of sentence or text representations. &#10;Despite the ongoing popularity of compact encoders such as BERT \cite{devlin-etal-2019-bert} and RoBERTa \cite{liu2019roberta}, there is a growing inclination to leverage the capabilities of recent, larger language models, e.g., LLaMA-2 \cite{touvron2023llama}, even breaking from the conventional roles of encoders and decoders.&#10;Consequently, the enduring challenge of finding a balance between performance and computational cost---a persistent issue in sentence representation learning \cite{reimers-gurevych-2019-sentence}---continues to be elusive.&#10;&#10;In recent years, there has been a marked improvement in the quality of sentence embeddings, a progress primarily driven by the advent of contrastive learning frameworks (\cite{kim-etal-2021-self,gao-etal-2021-simcse,chuang-etal-2022-diffcse}; inter alia).&#10;However, since the performance of these embeddings is generally evaluated based on their ability to encapsulate the overall meaning of the corresponding sentences---as measured by benchmarks like STS-B \cite{agirre-etal-2012-semeval,cer-etal-2017-semeval} and MTEB \cite{muennighoff-etal-2023-mteb}, it remains uncertain whether they adequately capture information relating to the various aspects of the source sentences.&#10;&#10;For instance, consider the sentences (1) ``A cyclist pedals along a scenic mountain trail, surrounded by lush greenery'' and (2) ``A hiker navigates through a dense forest on a winding path, enveloped by the tranquility of nature''. &#10;In terms of ``The mode of transportation'', these sentences should be perceived as similar since both depict individuals engaging in outdoor activities, traversing natural landscapes.&#10;However, regarding ``The speed of travel'', they should be differentiated, as cycling generally entails a faster pace than hiking.&#10;\cite{deshpande-etal-2023-c} reported that current models for sentence embeddings face challenges in recognizing the fine-grained semantics within sentences.&#10;In other words, the existing models struggle to accurately detect the subtle shifts in sentence nuances that occur when conditioned on specific criteria.&#10;&#10;In the literature, three prevalent approaches have been established for constructing conditioned representations \cite{deshpande-etal-2023-c}, particularly in estimating their similarity (see Figure \ref{fig:fig_architectures4}).&#10;The first is the cross-encoder approach, which encodes the concatenation of a pair of sentences ($s_1$, $s_2$) with a condition ($c$), i.e., $[s_1; s_2; c]$.&#10;The second method is the bi-encoder architecture, computing separate representations of sentences $s_1$ and $s_2$ with the condition $c$---$[s_1; c]$ and $[s_2; c]$.&#10;Despite their simplicity, both approaches share a notable limitation: the representation must be computed for every unique combination of sentences plus a condition.&#10;&#10;On the other hand, the tri-encoder architecture utilizes pre-computed embeddings of sentences $s_1$ and $s_2$ along with the condition $c$. &#10;It then employs a separate composition function responsible for merging the semantics of the sentence and condition.&#10;Considering that the embeddings for each component can be cached and reused, this approach offers enhanced long-term efficiency.&#10;The tri-encoder architecture, despite its potential, falls short in performance compared to the bi-encoder. &#10;This is primarily due to its inherent limitation, which is the inability to model explicit interactions between sentences and conditions during the representation construction process.&#10;Therefore, there is a need to propose a revised version of the tri-encoder architecture that improves its functionality without substantially sacrificing its efficiency.&#10;&#10;In this work, we present Hyper-CL, a method that integrates Hypernetworks \cite{ha2017hypernetworks} with Contrastive Learning to efficiently compute conditioned sentence representations and their similarity. &#10;As illustrated in Figure \ref{fig:fig_hyperCL}, our proposed approach is derived from the tri-encoder architecture. &#10;It introduces an additional hypernetwork tasked with constructing a condition-sensitive network on the fly. &#10;This network projects the original sentence embeddings into a specific condition subspace.&#10;Figure \ref{fig:fig_illustration_hyperCL} illustrates the effectiveness of Hyper-CL in dynamically conditioning pre-computed sentence representations according to different perspectives.&#10;&#10;We demonstrate the effectiveness of Hyper-CL by significantly reducing the performance gap with the bi-encoder architecture in the Conditional Semantic Textual Similarity (C-STS) and Knowledge Graph Completion (KGC) tasks.&#10;In particular, for C-STS, Hyper-CL demonstrates an improvement of up to 7.25 points in Spearman correlation compared to the original tri-encoder architecture.&#10;Furthermore, compared to the bi-encoder approach, our method shows superior efficiency by reducing the running time by approximately 40\% on the C-STS dataset and 57\% on the WN18RR dataset.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Hypernetwork Integration, Computer Science, Linguistics, Contrastive Learning Frameworks, Sentence Representation Learning, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="1609.09106" label="1609.09106">
        <attvalues>
          <attvalue for="0" value="HyperNetworks" />
          <attvalue for="1" value="  This work explores hypernetworks: an approach of using a one network, also&#10;known as a hypernetwork, to generate the weights for another network.&#10;Hypernetworks provide an abstraction that is similar to what is found in&#10;nature: the relationship between a genotype - the hypernetwork - and a&#10;phenotype - the main network. Though they are also reminiscent of HyperNEAT in&#10;evolution, our hypernetworks are trained end-to-end with backpropagation and&#10;thus are usually faster. The focus of this work is to make hypernetworks useful&#10;for deep convolutional networks and long recurrent networks, where&#10;hypernetworks can be viewed as relaxed form of weight-sharing across layers.&#10;Our main result is that hypernetworks can generate non-shared weights for LSTM&#10;and achieve near state-of-the-art results on a variety of sequence modelling&#10;tasks including character-level language modelling, handwriting generation and&#10;neural machine translation, challenging the weight-sharing paradigm for&#10;recurrent networks. Our results also show that hypernetworks applied to&#10;convolutional networks still achieve respectable results for image recognition&#10;tasks compared to state-of-the-art baseline models while requiring fewer&#10;learnable parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.08258" label="2403.08258">
        <attvalues>
          <attvalue for="0" value="Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognition" />
          <attvalue for="1" value="  Conformer-based attention models have become the de facto backbone model for&#10;Automatic Speech Recognition tasks. A blank symbol is usually introduced to&#10;align the input and output sequences for CTC or RNN-T models. Unfortunately,&#10;the long input length overloads computational budget and memory consumption&#10;quadratically by attention mechanism. In this work, we propose a&#10;&quot;Skip-and-Recover&quot; Conformer architecture, named Skipformer, to squeeze&#10;sequence input length dynamically and inhomogeneously. Skipformer uses an&#10;intermediate CTC output as criteria to split frames into three groups: crucial,&#10;skipping and ignoring. The crucial group feeds into next conformer blocks and&#10;its output joint with skipping group by original temporal order as the final&#10;encoder output. Experiments show that our model reduces the input sequence&#10;length by 31 times on Aishell-1 and 22 times on Librispeech corpus. Meanwhile,&#10;the model can achieve better recognition accuracy and faster inference speed&#10;than recent baseline models. Our code is open-sourced and available online.&#10;" />
          <attvalue for="2" value="&#10;In recent years, end-to-end~(E2E) models~\cite{graves2006connectionist,AED,LAS,graves2012sequence} have brought significant progress in the field of automatic speech recognition~(ASR). &#10;There are three popular E2E models for ASR tasks: Connectionist Temporal Classification~(CTC)~\cite{graves2006connectionist}, Attention-based Encoder-Decoder (AED)~\cite{AED,LAS} and Recurrent Neural Network Transducer~(RNN-T)~\cite{graves2012sequence}. &#10;In E2E ASR models, the most important component is the acoustic encoder which converts speech input sequences into high-level feature representations. &#10;Thanks to the development of the attention mechanism, Conformer-based encoder has been proven to outperform other network architectures and got state-of-the-art performance on many ASR tasks. &#10;However, Conformer-based encoder suffers from the quadratic complexity of the attention mechanism limiting its efficiency on long sequence lengths. &#10;Especially for the non-streaming AED model, this issue is more serious because an extra cross-attention needs to be calculated between encoder and decoder. &#10;On the other hand, the length of input sequence is much longer than that of the output sequence.&#10;In order to align the input and output sequences, CTC and RNN-T models introduce an extra blank symbol. &#10;For CTC or RNN-T models, blank symbol dominates the predicted output sequences, which makes a little contribution to the final performance and results in redundant computation. &#10;&#10;Many efforts have been devoted to improving the computation efficiency of acoustic encoders. &#10;The most recent approaches is using bigger downsampling factors to decrease sequence length. &#10;The vanilla Conformer~\cite{conformer} encoder begins with two convolution layers with stride 2 respectively along temporal dimension to reduce the input sequence length by 4 times. &#10;Efficient Conformer~\cite{Efficientconformer} increased the downsamping factor to 8 by using progressive downsampling method without performance drop. &#10;Squeezeformer~\cite{squeezeformer} combines downsampling with temporal U-Net structure. Followed by original down-sampling schema, an extra downsampling layer is applied at the middle of encoder layer and upsampling layer is applied at the end of encoder layer to recover 4 times temporal resolution to keep the recognition accuracy. &#10;A similar strategy was also used for deeper downsampling in Uconv-Conformer~\cite{Uconvconformer}. &#10;&#10;As for RNN-T and CTC model, blank symbols can be ignored during decoding, which can accelerate the decoding process \cite{blankregularized,wang2023accelerating,FSR,factorized}. Furthermore, the CTC output can also be used to guide the encoder downsampling. For example, in~\cite{wang2023accelerating}, they proposed to do frame reduction in the middle of RNN-T encoder using co-trained CTC guidance. Compared with downsampling frames uniformly, such as the Efficient Conformer and Squeezeformer, sampling with CTC guidance is a kind of ``importance sampling'' method. &#10;Blank Frames are dropped directly by such a CTC guidance. &#10;Similarly, we try to introduce the CTC guidance method into AED models to drop blank frames.&#10;Unfortunately, the performance drops greatly after decoding without blank symbols. &#10;&#10;Building upon this opinion, in this paper, we propose a ``Skip-and-Recover'' Conformer architecture, named Skipformer. Our core idea is that the less useful information one frame contains, the simpler model required to model it. &#10;On the contrary, the more crucial information one frame contains, the more complex model required to model it. &#10;In this work, an intermediate CTC output is used as criteria to measure the importance of an acoustic frame. Specifically, an intermediate CTC loss is attached to layer $M$th of the Conformer blocks which totally have $(M+N)$ layers. &#10;Frames after layer $M$ will be split into three groups guided by an intermediate CTC: the crucial group, trivial group and ignoring group. &#10;In general, the crucial group contains the most of non-blank semantic frames. &#10;Correspondingly, the trivial group and ignoring group contain the most of blank frames. &#10;The tivial frames serve as boudaries to patition repeat symbols. &#10;And the rest frames are classified into ignoring group, which will be discarded directly. &#10;These frames from the crucial group will be fed into the next $N$ Conformer layers to extract more valuable information by using more layers. &#10;As for the frames from the trivial group, they will skip over the following $N$ layers. &#10;At the end of the Conformer encoder, frames from crucial and skipping groups will be jointed together by original sequential order. &#10;Finally, the encoder output is fed into attention decoder for decoding. &#10;Furthermore, we explored several different grouping strategies to find out how to classify blank frames into trival and ignoring groups. &#10;Experimental results show that our proposed Skipformer can obtain better recognition performance and reduce the inference time significantly. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Attention Models, Deep Learning, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2208.07657" label="2208.07657">
        <attvalues>
          <attvalue for="0" value="Uconv-Conformer: High Reduction of Input Sequence Length for End-to-End&#10;  Speech Recognition" />
          <attvalue for="1" value="  Optimization of modern ASR architectures is among the highest priority tasks&#10;since it saves many computational resources for model training and inference.&#10;The work proposes a new Uconv-Conformer architecture based on the standard&#10;Conformer model. It consistently reduces the input sequence length by 16 times,&#10;which results in speeding up the work of the intermediate layers. To solve the&#10;convergence issue connected with such a significant reduction of the time&#10;dimension, we use upsampling blocks like in the U-Net architecture to ensure&#10;the correct CTC loss calculation and stabilize network training. The&#10;Uconv-Conformer architecture appears to be not only faster in terms of training&#10;and inference speed but also shows better WER compared to the baseline&#10;Conformer. Our best Uconv-Conformer model shows 47.8% and 23.5% inference&#10;acceleration on the CPU and GPU, respectively. Relative WER reduction is 7.3%&#10;and 9.2% on LibriSpeech test_clean and test_other respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.16481" label="2210.16481">
        <attvalues>
          <attvalue for="0" value="Accelerating RNN-T Training and Inference Using CTC guidance" />
          <attvalue for="1" value="  We propose a novel method to accelerate training and inference process of&#10;recurrent neural network transducer (RNN-T) based on the guidance from a&#10;co-trained connectionist temporal classification (CTC) model. We made a key&#10;assumption that if an encoder embedding frame is classified as a blank frame by&#10;the CTC model, it is likely that this frame will be aligned to blank for all&#10;the partial alignments or hypotheses in RNN-T and it can be discarded from the&#10;decoder input. We also show that this frame reduction operation can be applied&#10;in the middle of the encoder, which result in significant speed up for the&#10;training and inference in RNN-T. We further show that the CTC alignment, a&#10;by-product of the CTC decoder, can also be used to perform lattice reduction&#10;for RNN-T during training. Our method is evaluated on the Librispeech and&#10;SpeechStew tasks. We demonstrate that the proposed method is able to accelerate&#10;the RNN-T inference by 2.2 times with similar or slightly better word error&#10;rates (WER).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.02882" label="2104.02882">
        <attvalues>
          <attvalue for="0" value="FSR: Accelerating the Inference Process of Transducer-Based Models by&#10;  Applying Fast-Skip Regularization" />
          <attvalue for="1" value="  Transducer-based models, such as RNN-Transducer and transformer-transducer,&#10;have achieved great success in speech recognition. A typical transducer model&#10;decodes the output sequence conditioned on the current acoustic state and&#10;previously predicted tokens step by step. Statistically, The number of blank&#10;tokens in the prediction results accounts for nearly 90\% of all tokens. It&#10;takes a lot of computation and time to predict the blank tokens, but only the&#10;non-blank tokens will appear in the final output sequence. Therefore, we&#10;propose a method named fast-skip regularization, which tries to align the blank&#10;position predicted by a transducer with that predicted by a CTC model. During&#10;the inference, the transducer model can predict the blank tokens in advance by&#10;a simple CTC project layer without many complicated forward calculations of the&#10;transducer decoder and then skip them, which will reduce the computation and&#10;improve the inference speed greatly. All experiments are conducted on a public&#10;Chinese mandarin dataset AISHELL-1. The results show that the fast-skip&#10;regularization can indeed help the transducer model learn the blank position&#10;alignments. Besides, the inference with fast-skip can be speeded up nearly 4&#10;times with only a little performance degradation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05211" label="2204.05211">
        <attvalues>
          <attvalue for="0" value="Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0" />
          <attvalue for="1" value="  In this work, we explore whether the recently demonstrated zero-shot&#10;abilities of the T0 model extend to Named Entity Recognition for&#10;out-of-distribution languages and time periods. Using a historical newspaper&#10;corpus in 3 languages as test-bed, we use prompts to extract possible named&#10;entities. Our results show that a naive approach for prompt-based zero-shot&#10;multilingual Named Entity Recognition is error-prone, but highlights the&#10;potential of such an approach for historical languages lacking labeled&#10;datasets. Moreover, we also find that T0-like models can be probed to predict&#10;the publication date and language of a document, which could be very relevant&#10;for the study of historical texts.&#10;" />
          <attvalue for="2" value=" This paper lies at the focal point of three orthogonal advances. First, the recent surge in GLAM-led digitisation efforts \cite{Terras2011}, open citizen science \cite{muki:2021citsci} and the expansive commodification of data \cite{hey:2003deluge}, have enabled a new mode of historical inquiry that capitalises on the `big data of the past' \cite{Kapland17}. Second, the 2017~breakthrough that was the transformer architecture \cite{VaswaniSPUJGKP17} has led to the so-called ImageNet moment of Natural Language Processing \cite{ruder2018nlpimagenet} and brought about unprecedented progress in transfer-learning \cite{raffel2020t5}, few-shot learning \cite{schick-schutze-2021-just}, zero-shot learning \cite{sanh2021multitask}, and prompt-based learning \cite{le-scao-rush-2021-many} for natural language. Third, the growing popularity of prompt-based methods \cite{LiuPengfei2021PPaP} has resulted in a new paradigm for training and fine-tuning Large Language Models~(LLM) as well as novel applications in Named Entity Recognition~(NER)~\cite{QaNER}.&#10;&#10;NER for historical texts has been the focus of a growing body of research, most recently surveyed by~\cite{Ehrmann2021NamedER}. Both NER and the related task of Entity Linking can enhance our ability to search and navigate digitised historical materials~\cite{Neudecker2014LargescaleRO, Kim2015FindingNI}. However, applying NER to historical texts poses a number of challenges, including those due to errors in Optical Character Recognition (OCR)~\cite{Ehrmann2021NamedER,Hamdi2019AnAO,Boros2020AlleviatingDE} and domain transfer~\cite{Blouin_Favre_Auguste_Henriot_2021}. To advance research in this area, an increasing number of datasets have been created to support the development and evaluation of NER approaches in historical text~\cite{Neudecker2016AnOC, ehrmann_extended_2020,ehrmann_maud_2022_6089968}&#10;&#10;In this paper, we examine the zero-shot abilities of T0---a prompt-based LLM developed as part of the BigScience project for open research \cite{sanh2021multitask}---on the challenging task of historical NER. This endeavour had two main hurdles: (1)~the model was neither trained to recognize entities, nor was it ever tested on that task; (2)~our evaluation dataset was out-of-distribution, containing both multilingual and historical data. To better contextualize the results of our experiments, we also run zero-shot prompt-based probing \cite{zhong-etal-2021-factual} to assess T0's broader ability of extracting factual knowledge about two key factors in our experiment, that is, language variation and historical variation in the dataset.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Historical Linguistics, Computer Science, Linguistics, Zero-Shot Learning, History, Historical Text Analysis, Artificial Intelligence, Natural Language Processing, Multilingual NER" />
        </attvalues>
      </node>
      <node id="2109.11406" label="2109.11406">
        <attvalues>
          <attvalue for="0" value="Named Entity Recognition and Classification on Historical Documents: A&#10;  Survey" />
          <attvalue for="1" value="  After decades of massive digitisation, an unprecedented amount of historical&#10;documents is available in digital format, along with their machine-readable&#10;texts. While this represents a major step forward with respect to preservation&#10;and accessibility, it also opens up new opportunities in terms of content&#10;mining and the next fundamental challenge is to develop appropriate&#10;technologies to efficiently search, retrieve and explore information from this&#10;'big data of the past'. Among semantic indexing opportunities, the recognition&#10;and classification of named entities are in great demand among humanities&#10;scholars. Yet, named entity recognition (NER) systems are heavily challenged&#10;with diverse, historical and noisy inputs. In this survey, we present the array&#10;of challenges posed by historical documents to NER, inventory existing&#10;resources, describe the main approaches deployed so far, and identify key&#10;priorities for future developments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.10804" label="2311.10804">
        <attvalues>
          <attvalue for="0" value="A Study on Altering the Latent Space of Pretrained Text to Speech Models&#10;  for Improved Expressiveness" />
          <attvalue for="1" value="  This report explores the challenge of enhancing expressiveness control in&#10;Text-to-Speech (TTS) models by augmenting a frozen pretrained model with a&#10;Diffusion Model that is conditioned on joint semantic audio/text embeddings.&#10;The paper identifies the challenges encountered when working with a VAE-based&#10;TTS model and evaluates different image-to-image methods for altering latent&#10;speech features. Our results offer valuable insights into the complexities of&#10;adding expressiveness control to TTS systems and open avenues for future&#10;research in this direction.&#10;" />
          <attvalue for="2" value="&#10;Significant progress has been made in the development of Text-to-Speech (TTS) systems, with models such as VITS \cite{Kim2021ConditionalText-to-Speech} achieving a mean opinion score comparable to that of genuine speech recordings. However, the difficulty of having precise control over the prosodic features of generated speech samples remains an unsolved issue. Many TTS models lack mechanisms to control prosodic and emotional nuances, which are essential for a wide range of applications.&#10;&#10;In this paper, we present an exploratory study in which we enhance the VITS model with expressiveness control by adding a Denoising Diffusion Model (DDM) \cite{Ho2020DenoisingModels} conditioned on joint audio/text embeddings such as CLAP embeddings \cite{Wu2023Large-ScaleAugmentation} to alter the latent VITS encodings. We chose DDM because these models are known to be easy to condition and are currently state-of-the-art in many computer vision tasks \cite{Ho2022VideoModels, Hoogeboom2023SimpleImages, Rombach2021High-ResolutionModels}. The final goal is to be able to change the generated speech by providing a target style by either providing a recording by text promts describing the style. Contrary to our expectations, the method did not produce the desired improvements in expressiveness control. However, we believe that the findings of our study offer valuable insight into the complexities and challenges associated with adding expressiveness control to TTS systems. Our findings could help future research in designing systems that allow greater control over prosodic and emotional features.&#10;&#10;Our contributions are as follows:&#10;\begin{enumerate}&#10; \item We identify challenges of working with pretrained VAE based TTS models.&#10; \item We apply and compare different image-to-image methods to change latent speech features, highlighting their strengths and weaknesses.&#10; \item We open up discussion on further research directions of controlled emotional TTS.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Expressiveness Control, Deep Learning Methods, Signal Processing, Text-to-Speech Models" />
        </attvalues>
      </node>
      <node id="2403.08196" label="2403.08196">
        <attvalues>
          <attvalue for="0" value="SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech&#10;  Recognition Evaluation" />
          <attvalue for="1" value="  In the wake of the surging tide of deep learning over the past decade,&#10;Automatic Speech Recognition (ASR) has garnered substantial attention, leading&#10;to the emergence of numerous publicly accessible ASR systems that are actively&#10;being integrated into our daily lives. Nonetheless, the impartial and&#10;replicable evaluation of these ASR systems encounters challenges due to various&#10;crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a&#10;general-purpose, open-source platform designed for ASR evaluation. With this&#10;platform: (i) We report a comprehensive benchmark, unveiling the current&#10;state-of-the-art panorama for ASR systems, covering both open-source models and&#10;industrial commercial services. (ii) We quantize how distinct nuances in the&#10;scoring pipeline influence the final benchmark outcomes. These include nuances&#10;related to capitalization, punctuation, interjection, contraction, synonym&#10;usage, compound words, etc. These issues have gained prominence in the context&#10;of the transition towards an End-to-End future. (iii) We propose a practical&#10;modification to the conventional Token-Error-Rate (TER) evaluation metric, with&#10;inspirations from Kolmogorov complexity and Normalized Information Distance&#10;(NID). This adaptation, called modified-TER (mTER), achieves proper&#10;normalization and symmetrical treatment of reference and hypothesis. By&#10;leveraging this platform as a large-scale testing ground, this study&#10;demonstrates the robustness and backward compatibility of mTER when compared to&#10;TER. The SpeechColab Leaderboard is accessible at&#10;https://github.com/SpeechColab/Leaderboard&#10;" />
          <attvalue for="2" value="&#10;Automatic Speech Recognition (ASR) has been an active research topic for many years.&#10;Traditional ASR combines Hidden Markov Models (HMM) and Gaussian Mixture Models (GMM) to capture the dynamics of speech signal and the hierarchical knowledge behind human languages \cite{deng1991phonemic}.&#10;In recent years, deep neural networks (DNN) have started to emerge with superior accuracy \cite{dahl2011context}, and have quickly become the mainstream for ASR.&#10;For instance,&#10;chain model \cite{povey2016purely} incorporates Convolutional Neural Networks (CNNs) and Time Delay Neural Network (TDNNs),&#10;while DeepSpeech \cite{hannun2014deep} model utilizes Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs).&#10;Modern systems are leaning to even more sophisticated architectures such as Transformer \cite{vaswani2017attention} and Conformer \cite{gulati2020conformer},&#10;coupled with sequence losses like Connectionist Temporal Classification (CTC) \cite{CTC} and Recurrent Neural Network Transducer (RNN-T) \cite{graves2012sequence}.&#10;From a system perspective, driven by the scaling law from language modeling research,&#10;large speech models have been developed such as OpenAI-Whisper \cite{radford2022robust}, and Google-USM \cite{zhang2023google},&#10;pushing up the scale of ASR training by orders of magnitude.&#10;In the meantime, self-supervised training, as a paradigm shift,&#10;is also gaining popularity to leverage abundant unlabeled data in the world.&#10;Notable examples are wav2vec 2.0 \cite{baevski2020wav2vec}, HuBERT \cite{hsu2021hubert}, WavLM \cite{chen2022wavlm}, and data2vec \cite{baevski2022data2vec}.&#10;&#10;Given the swift evolution of ASR technology, a variety of speech toolkits have been developed and open-sourced,&#10;such as HTK \cite{young2002htk}, Kaldi \cite{povey2011kaldi}, &#10;ESPnet \cite{watanabe2018espnet}, NeMo \cite{kuchaiev2019nemo}, &#10;SpeechBrain \cite{ravanelli2021speechbrain}, WeNet \cite{yao2021wenet}, and K2,&#10;offering comprehensive libraries and recipes to facilitate ASR research and development.&#10;adjcol&#10;However, the evaluation of ASR still remains challenging \cite{del2021earnings} \cite{gandhi2022esb},&#10;because there exist various crucial subtleties and pitfalls that require non-trivial efforts to do right in practice,&#10;such as text normalization \cite{faria2022toward}.&#10;The divergent ecosystem struggles to reach a clear and consistent understanding on the performance of modern ASR systems. &#10;&#10;To address the problem, we present SpeechColab Leaderboard, an open-source benchmark platform,&#10;so that speech researchers and developers can reliably reproduce, examine, and compare all kinds of ASR systems.&#10;The platform is designed to be:&#10;(i) Simple: consistent data formats and unified interfaces minimize accidental complexity.&#10;(ii) Open: leaderboard users should be able to easily share and exchange resources (e.g. test sets, models, configurations).&#10;(iii) Reproducible: ASR systems, including all their dependencies and environment details, should be reproducible as a whole.&#10;&#10;In Section 2, we describe the proposed platform, including three major components:&#10;a dataset zoo, a model zoo, and an evaluation pipeline.&#10;In Section 3, we report a large-scale benchmark for English ASR on the platform.&#10;In Section 4, the traditional evaluation metric TER (Token Error Rate) is briefly revisited,&#10;and a simple and practical modification is proposed to make TER more robust.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Automatic Speech Recognition, Information Science, Evaluation Metrics, Deep Learning Platforms" />
        </attvalues>
      </node>
      <node id="2106.04624" label="2106.04624">
        <attvalues>
          <attvalue for="0" value="SpeechBrain: A General-Purpose Speech Toolkit" />
          <attvalue for="1" value="  SpeechBrain is an open-source and all-in-one speech toolkit. It is designed&#10;to facilitate the research and development of neural speech processing&#10;technologies by being simple, flexible, user-friendly, and well-documented.&#10;This paper describes the core architecture designed to support several tasks of&#10;common interest, allowing users to naturally conceive, compare and share novel&#10;speech processing pipelines. SpeechBrain achieves competitive or&#10;state-of-the-art performance in a wide range of speech benchmarks. It also&#10;provides training recipes, pretrained models, and inference scripts for popular&#10;speech datasets, as well as tutorials which allow anyone with basic Python&#10;proficiency to familiarize themselves with speech technologies.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.11348" label="2104.11348">
        <attvalues>
          <attvalue for="0" value="Earnings-21: A Practical Benchmark for ASR in the Wild" />
          <attvalue for="1" value="  Commonly used speech corpora inadequately challenge academic and commercial&#10;ASR systems. In particular, speech corpora lack metadata needed for detailed&#10;analysis and WER measurement. In response, we present Earnings-21, a 39-hour&#10;corpus of earnings calls containing entity-dense speech from nine different&#10;financial sectors. This corpus is intended to benchmark ASR systems in the wild&#10;with special attention towards named entity recognition. We benchmark four&#10;commercial ASR models, two internal models built with open-source tools, and an&#10;open-source LibriSpeech model and discuss their differences in performance on&#10;Earnings-21. Using our recently released fstalign tool, we provide a candid&#10;analysis of each model's recognition capabilities under different partitions.&#10;Our analysis finds that ASR accuracy for certain NER categories is poor,&#10;presenting a significant impediment to transcript comprehension and usage.&#10;Earnings-21 bridges academic and commercial ASR system evaluation and enables&#10;further research on entity modeling and WER on real world audio.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.06192" label="2206.06192">
        <attvalues>
          <attvalue for="0" value="Toward Zero Oracle Word Error Rate on the Switchboard Benchmark" />
          <attvalue for="1" value="  The &quot;Switchboard benchmark&quot; is a very well-known test set in automatic speech&#10;recognition (ASR) research, establishing record-setting performance for systems&#10;that claim human-level transcription accuracy. This work highlights&#10;lesser-known practical considerations of this evaluation, demonstrating major&#10;improvements in word error rate (WER) by correcting the reference&#10;transcriptions and deviating from the official scoring methodology. In this&#10;more detailed and reproducible scheme, even commercial ASR systems can score&#10;below 5% WER and the established record for a research system is lowered to&#10;2.3%. An alternative metric of transcript precision is proposed, which does not&#10;penalize deletions and appears to be more discriminating for human vs. machine&#10;performance. While commercial ASR systems are still below this threshold, a&#10;research system is shown to clearly surpass the accuracy of commercial human&#10;speech recognition. This work also explores using standardized scoring tools to&#10;compute oracle WER by selecting the best among a list of alternatives. A phrase&#10;alternatives representation is compared to utterance-level N-best lists and&#10;word-level data structures; using dense lattices and adding out-of-vocabulary&#10;words, this achieves an oracle WER of 0.18%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.12030" label="2402.12030">
        <attvalues>
          <attvalue for="0" value="Towards Cross-Tokenizer Distillation: the Universal Logit Distillation&#10;  Loss for LLMs" />
          <attvalue for="1" value="  Deploying large language models (LLMs) of several billion parameters can be&#10;impractical in most industrial use cases due to constraints such as cost,&#10;latency limitations, and hardware accessibility. Knowledge distillation (KD)&#10;offers a solution by compressing knowledge from resource-intensive large models&#10;to smaller ones. Various strategies exist, some relying on the text generated&#10;by the teacher model and optionally utilizing his logits to enhance learning.&#10;However, these methods based on logits often require both teacher and student&#10;models to share the same tokenizer, limiting their applicability across&#10;different LLM families. In this paper, we introduce Universal Logit&#10;Distillation (ULD) loss, grounded in optimal transport, to address this&#10;limitation. Our experimental results demonstrate the effectiveness of ULD loss&#10;in enabling distillation across models with different architectures and&#10;tokenizers, paving the way to a more widespread use of distillation techniques.&#10;" />
          <attvalue for="2" value="&#10;A significant trend in NLP involves the utilization of large language models (LLMs) such as LLama \cite{LLama}, Mistral \cite{mistral}, Falcon \cite{falcon}, GPT-NeoX \cite{black2022gptneox20b}, or Mixtral \cite{jiang2024mixtral}. While LLMs offer impressive performance \cite{bubeck2023sparks}, their deployment is often hampered by hardware availability, cost, and latency bottlenecks. Several strategies, such as efficient decoding \cite{leviathan2023fast, ye-etal-2023-fid}, model recycling \cite{lester2022reducing}, and model size reduction \cite{dettmers2023qlora, ma2023llmpruner}, have been developed to streamline their use. Among these, knowledge distillation (KD) \cite{model_compression, distillation} a widely adopted technique \cite{distilbert, tinybert, small100, how_fish, token_not_all, cost_effective}, transferring the capabilities of large, complex teacher models into more manageable and smaller student models, tailored for specific tasks. This approach is particularly valuable in contexts where the comprehensive knowledge contained within LLMs is not wholly necessary. This process aims to maintain the peak performance of general models on specific tasks while minimizing latency and memory usage.&#10;&#10;Two approaches can be considered. The &quot;white box&quot; approach, where researchers propose loss functions that require access to the model architecture to compute similarities across layers, forcing adjustment for each situation. In contrast, the &quot;black box&quot; approach, indifferent to models latent spaces, relies solely on the output logit vectors from teacher and student. The black-box approach, due to its flexibility and generality, is easily implemented by practitioners through libraries or APIs, facilitating its adoption.&#10;&#10;Over the past years, NLP researchers have extensively explored these distillation methods with teacher and student models sharing similar architecture, notably the BERT encoder \cite{distilbert, tinybert, mobilebert}. This desire to maintain similar architecture between teacher and student models by mirroring some of the teacher blocks, hidden sizes, or relying on the same tokenizer, came from the need to find similar information supports to apply distillation losses.&#10;&#10;However, KD for generative models, those relying on encoder-decoder or decoder architectures, has received less attention due to the lack of common support between models. Although models of different sizes exist, they often diverge in architecture and tokenizer (\ref{tokenizers_vocabularies}), making logit distillation loss inapplicable. In fact, recent research predominantly focuses on synthetic data fine-tuning \cite{tacl_a_00492, kramchaninova-defauw-2022-synthetic, instructedfinetuning}, rather than refining logits loss in the black box approach. Thus far, the primary method for KD with decoder models is to use the text generated by the teacher model \cite{how_fish, step_step}, and if possible, when students and teachers models belong to the same family, improving KD by employing output-logit distillation with Kullback–Leibler divergence \cite{small100, token_not_all, minilm}. This raises the following research question:&#10;&#10;How can we build a general distillation loss that leverages logits capacity while staying within a black box framework that is easy to implement?&#10;&#10;Contributions: In this paper, we make the following contributions:&#10;\begin{itemize}&#10; \item[1.] A universal logit distillation loss. We introduce a new loss, Universal Logit Distillation Loss (ULD loss), versatile to tokenizers and with minimal assumptions about the architectures of teacher and student models.&#10; \item[2.] Experimental Results. We demonstrate the robust effectiveness of our loss function in transferring the capabilities of many teacher models to different smaller student models for a variety of specific tasks, including extractive question answering, generative question answering, and summarization.&#10; \item[3.] Contributing to future research. We make our code, model weights, and generated datasets openly available to facilitate future research, minimizing computational overhead and lowering entry barriers.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Knowledge Distillation Techniques, Language Model Compression, Mathematics, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="2106.06168" label="2106.06168">
        <attvalues>
          <attvalue for="0" value="Generate, Annotate, and Learn: NLP with Synthetic Text" />
          <attvalue for="1" value="  This paper studies the use of language models as a source of synthetic&#10;unlabeled text for NLP. We formulate a general framework called ``generate,&#10;annotate, and learn (GAL)'' to take advantage of synthetic text within&#10;knowledge distillation, self-training, and few-shot learning applications. To&#10;generate high-quality task-specific text, we either fine-tune LMs on inputs&#10;from the task of interest, or prompt large LMs with few examples. We use the&#10;best available classifier to annotate synthetic text with soft pseudo labels&#10;for knowledge distillation and self-training, and use LMs to obtain hard labels&#10;for few-shot learning. We train new supervised models on the combination of&#10;labeled and pseudo-labeled data, which results in significant gains across&#10;several applications. We investigate key components of GAL and present&#10;theoretical and empirical arguments against the use of class-conditional LMs to&#10;generate synthetic labeled text instead of unlabeled text. GAL achieves new&#10;state-of-the-art knowledge distillation results for 6-layer transformers on the&#10;GLUE leaderboard.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.19019" label="2310.19019">
        <attvalues>
          <attvalue for="0" value="TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language&#10;  Modeling Likewise" />
          <attvalue for="1" value="  Large Language Models (LLMs) exhibit impressive reasoning and data&#10;augmentation capabilities in various NLP tasks. However, what about small&#10;models? In this work, we propose TeacherLM-7.1B, capable of annotating relevant&#10;fundamentals, chain of thought, and common mistakes for most NLP samples, which&#10;makes annotation more than just an answer, thus allowing other models to learn&#10;&quot;why&quot; instead of just &quot;what&quot;. The TeacherLM-7.1B model achieved a zero-shot&#10;score of 52.3 on MMLU, surpassing most models with over 100B parameters. Even&#10;more remarkable is its data augmentation ability. Based on TeacherLM-7.1B, we&#10;augmented 58 NLP datasets and taught various student models with different&#10;parameters from OPT and BLOOM series in a multi-task setting. The experimental&#10;results indicate that the data augmentation provided by TeacherLM has brought&#10;significant benefits. We will release the TeacherLM series of models and&#10;augmented datasets as open-source.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.11621" label="2210.11621">
        <attvalues>
          <attvalue for="0" value="SMaLL-100: Introducing Shallow Multilingual Machine Translation Model&#10;  for Low-Resource Languages" />
          <attvalue for="1" value="  In recent years, multilingual machine translation models have achieved&#10;promising performance on low-resource language pairs by sharing information&#10;between similar languages, thus enabling zero-shot translation. To overcome the&#10;&quot;curse of multilinguality&quot;, these models often opt for scaling up the number of&#10;parameters, which makes their use in resource-constrained environments&#10;challenging. We introduce SMaLL-100, a distilled version of the M2M-100 (12B)&#10;model, a massively multilingual machine translation model covering 100&#10;languages. We train SMaLL-100 with uniform sampling across all language pairs&#10;and therefore focus on preserving the performance of low-resource languages. We&#10;evaluate SMaLL-100 on different low-resource benchmarks: FLORES-101, Tatoeba,&#10;and TICO-19 and demonstrate that it outperforms previous massively multilingual&#10;models of comparable sizes (200-600M) while improving inference latency and&#10;memory usage. Additionally, our model achieves comparable results to M2M-100&#10;(1.2B), while being 3.6x smaller and 4.3x faster at inference. Code and&#10;pre-trained models: https://github.com/alirezamshi/small100&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.01249" label="2303.01249">
        <attvalues>
          <attvalue for="0" value="Language-Universal Adapter Learning with Knowledge Distillation for&#10;  End-to-End Multilingual Speech Recognition" />
          <attvalue for="1" value="  In this paper, we propose a language-universal adapter learning framework&#10;based on a pre-trained model for end-to-end multilingual automatic speech&#10;recognition (ASR). For acoustic modeling, the wav2vec 2.0 pre-trained model is&#10;fine-tuned by inserting language-specific and language-universal adapters. An&#10;online knowledge distillation is then used to enable the language-universal&#10;adapters to learn both language-specific and universal features. The linguistic&#10;information confusion is also reduced by leveraging language identifiers&#10;(LIDs). With LIDs we perform a position-wise modification on the multi-head&#10;attention outputs. In the inference procedure, the language-specific adapters&#10;are removed while the language-universal adapters are kept activated. The&#10;proposed method improves the recognition accuracy and addresses the linear&#10;increase of the number of adapters' parameters with the number of languages in&#10;common multilingual ASR systems. Experiments on the BABEL dataset confirm the&#10;effectiveness of the proposed framework. Compared to the conventional&#10;multilingual model, a 3.3% absolute error rate reduction is achieved. The code&#10;is available at: https://github.com/shen9712/UniversalAdapterLearning.&#10;" />
          <attvalue for="2" value="&#10;&#10;With the widespread of end-to-end automatic speech recognition (ASR) frameworks \cite{dong2020cif, kahn2020self, miao2020transformer}, multilingual ASR has become a research hotspot. This is mainly due to easy training and deployment procedures in real-world applications, especially in low-resourced scenarios.&#10;&#10;One of the key challenges in multilingual ASR is language confusion. The most intuitive idea to address this issue is using language-specific parameters in the model training \cite{yi2018language, gaur2021mixture, kannan2019large, winata2020adapt}. For instance, in \cite{yi2018language}, the lower LSTM layers were shared among multiple languages as a common feature extractor, whereas the upper layers were language specific. \cite{gaur2021mixture} introduced the informed mixture-of-experts layers in which each expert was assigned to one language.&#10;&#10;The adapter-based modeling technique has been successfully used for domain adaptation in computer vision \cite{sung2022vl}, natural language processing \cite{pfeiffer2020unks}, and machine translation \cite{bapna2019simple}. Adapters make effectively domain-specific (language-specific in our case) adjustments to the activations in a network. In multilingual ASR, \cite{kannan2019large} investigated using adapter modules for nine Indian languages in an RNN-T model. Furthermore, The Adapter-and-Adjust framework was introduced in \cite{winata2020adapt}, where both language-specific (LSA) and common adapters were applied to an encoder-decoder network. The LSA was focused on adapting the shared network weights to a particular language, whereas the common adapter was used to learn shared knowledge. Nevertheless, the number of LSA’s parameters grows linearly with the number of languages, which limits large-scale multilingual modeling. Also, the common adapter is trained based on imbalanced multilingual data, hence it is biased towards dominant languages.&#10;&#10;To address the above-mentioned issues, we propose to merge the language-specific and language-agnostic information into one language-universal adapter (LUA). In our approach, the LSA and LUA are first inserted into the wav2vec 2.0 pre-trained model \cite{baevski2020wav2vec} in the training procedure. The wav2vec 2.0 pre-trained model is used due to its high performance in low-resourced ASR. The LSA captures language-specific features which are then transferred to LUA through online knowledge distillation (KD). This results in the improved robustness of LUA to data imbalance and domain (language) shifts. Note that only LUA is used for inference.&#10;&#10;It is generally acknowledged that incorporating the language identifier (LID) is beneficial for multilingual ASR. For instance, \cite{shetty2020improving, zhou2022configurable, zhu2020multilingual} showed that training and testing conditioned on LID improves the performance and reduces language confusions. LID can be also applied in different positions of the network. For example, \cite{zhou2022configurable} simply concatenated a one-hot vector to the input features of the encoder network, whereas \cite{zhu2020multilingual} concatenated LID to multi-head attention inputs in the transformer model.&#10;&#10;In this paper, in order to further reduce language confusion, we propose to use LID as multi-head attention prefixes. This is inspired by the prefix-tuning method \cite{li2021prefix}. This method performs a position-wise modification on the multi-head attention outputs, which is more effective than injecting LID into the input features or hidden features.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Electrical Engineering, Linguistics, Language Modeling Techniques, Multilingual Speech Recognition, Signal Processing, Adapter Learning Framework" />
        </attvalues>
      </node>
      <node id="2112.06825" label="2112.06825">
        <attvalues>
          <attvalue for="0" value="VL-Adapter: Parameter-Efficient Transfer Learning for&#10;  Vision-and-Language Tasks" />
          <attvalue for="1" value="  Recently, fine-tuning language models pre-trained on large text corpora have&#10;provided huge improvements on vision-and-language (V&amp;L) tasks as well as on&#10;pure language tasks. However, fine-tuning the entire parameter set of&#10;pre-trained models becomes impractical since the model size is growing rapidly.&#10;Hence, in this paper, we introduce adapter-based parameter-efficient transfer&#10;learning techniques to V&amp;L models such as VL-BART and VLT5. We evaluate our&#10;methods in a unified multi-task setup on both image-text and video-text&#10;benchmarks. For the image-text tasks, we use four diverse V&amp;L datasets: VQAv2,&#10;GQA, NLVR2 , and MSCOCO image captioning. For video-text tasks, we use TVQA,&#10;How2QA, TVC, and YC2C. With careful training and thorough experiments, we&#10;benchmark three popular adapter-based methods (Adapter, Hyperformer, Compacter)&#10;against the standard full fine-tuning and the recently proposed prompt-tuning&#10;approach. We also enhance the efficiency and performance of adapters by sharing&#10;their weights to attain knowledge across tasks. Our results demonstrate that&#10;training the adapter with the weight-sharing technique (4.18% of total&#10;parameters for image-text tasks and 3.39% for video-text tasks) can match the&#10;performance of fine-tuning the entire model. Lastly, we present a comprehensive&#10;analysis including the combination of adapter and task-specific prompts and the&#10;impact of V&amp;L pre-training on adapters. Our code is available at:&#10;https://github.com/ylsung/VL_adapter.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.05876" label="2107.05876">
        <attvalues>
          <attvalue for="0" value="A Configurable Multilingual Model is All You Need to Recognize All&#10;  Languages" />
          <attvalue for="1" value="  Multilingual automatic speech recognition (ASR) models have shown great&#10;promise in recent years because of the simplified model training and deployment&#10;process. Conventional methods either train a universal multilingual model&#10;without taking any language information or with a 1-hot language ID (LID)&#10;vector to guide the recognition of the target language. In practice, the user&#10;can be prompted to pre-select several languages he/she can speak. The&#10;multilingual model without LID cannot well utilize the language information set&#10;by the user while the multilingual model with LID can only handle one&#10;pre-selected language. In this paper, we propose a novel configurable&#10;multilingual model (CMM) which is trained only once but can be configured as&#10;different models based on users' choices by extracting language-specific&#10;modules together with a universal model from the trained CMM. Particularly, a&#10;single CMM can be deployed to any user scenario where the users can pre-select&#10;any combination of languages. Trained with 75K hours of transcribed anonymized&#10;Microsoft multilingual data and evaluated with 10-language test sets, the&#10;proposed CMM improves from the universal multilingual model by 26.0%, 16.9%,&#10;and 10.4% relative word error reduction when the user selects 1, 2, or 3&#10;languages, respectively. CMM also performs significantly better on&#10;code-switching test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.08666" label="2404.08666">
        <attvalues>
          <attvalue for="0" value="Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferences" />
          <attvalue for="1" value="  Natural language processing (NLP) has grown significantly since the advent of&#10;the Transformer architecture. Transformers have given birth to pre-trained&#10;large language models (PLMs). There has been tremendous improvement in the&#10;performance of NLP systems across several tasks. NLP systems are on par or, in&#10;some cases, better than humans at accomplishing specific tasks. However, it&#10;remains the norm that \emph{better quality datasets at the time of pretraining&#10;enable PLMs to achieve better performance, regardless of the task.} The need to&#10;have quality datasets has prompted NLP researchers to continue creating new&#10;datasets to satisfy particular needs. For example, the two top NLP conferences,&#10;ACL and EMNLP, accepted ninety-two papers in 2022, introducing new datasets.&#10;This work aims to uncover the trends and insights mined within these datasets.&#10;Moreover, we provide valuable suggestions to researchers interested in curating&#10;datasets in the future.&#10;" />
          <attvalue for="2" value="&#10;Natural language processing (NLP) has grown significantly since the advent of the Transformer architecture. Transformers have given birth to a new pre-trained of large language models (PLMs). &#10;&#10;There has been tremendous improvement in the performance of NLP systems across several tasks. NLP systems are on par or, in some cases, better than humans at accomplishing specific tasks. &#10;&#10;However, it remains the norm that better quality datasets at the time of training enable PLMs to perform better, regardless of the task. The need to have quality datasets has prompted NLP researchers to continue creating new datasets to satisfy particular needs. For example, the two top NLP conferences, ACL and EMNLP, accepted ninety-two papers in 2022, introducing new datasets. This work aims to uncover the trends and insights mined within these datasets. Figure~\ref{fig:Intro_section} shows the main NLP tasks for which authors created new datasets, which were published at both EMNLP and ACL in the year 2022. The red box in Figure~\ref{fig:Intro_section} indicates the tasks in both ACL and EMNLP datasets.&#10;&#10;We draw inspiration from \cite{gollapalli-li-2015-emnlp}, who previously studied the significant themes across NLP conferences from 1996 to 2014. The more recent work to examine the disparate sources utilized to construct new datasets is the study by \cite{yu-etal-2022-beyond}. We aim to reveal insights such as significant author affiliations in creating datasets for training NLP models. &#10;&#10;Lastly, we provide helpful suggestions to researchers interested in curating datasets in the future. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Dataset Curation, Data Science, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2211.15649" label="2211.15649">
        <attvalues>
          <attvalue for="0" value="Beyond Counting Datasets: A Survey of Multilingual Dataset Construction&#10;  and Necessary Resources" />
          <attvalue for="1" value="  While the NLP community is generally aware of resource disparities among&#10;languages, we lack research that quantifies the extent and types of such&#10;disparity. Prior surveys estimating the availability of resources based on the&#10;number of datasets can be misleading as dataset quality varies: many datasets&#10;are automatically induced or translated from English data. To provide a more&#10;comprehensive picture of language resources, we examine the characteristics of&#10;156 publicly available NLP datasets. We manually annotate how they are created,&#10;including input text and label sources and tools used to build them, and what&#10;they study, tasks they address and motivations for their creation. After&#10;quantifying the qualitative NLP resource gap across languages, we discuss how&#10;to improve data collection in low-resource languages. We survey&#10;language-proficient NLP researchers and crowd workers per language, finding&#10;that their estimated availability correlates with dataset availability. Through&#10;crowdsourcing experiments, we identify strategies for collecting high-quality&#10;multilingual data on the Mechanical Turk platform. We conclude by making macro&#10;and micro-level suggestions to the NLP community and individual researchers for&#10;future multilingual data development.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.08686" label="2106.08686">
        <attvalues>
          <attvalue for="0" value="Do Acoustic Word Embeddings Capture Phonological Similarity? An&#10;  Empirical Study" />
          <attvalue for="1" value="  Several variants of deep neural networks have been successfully employed for&#10;building parametric models that project variable-duration spoken word segments&#10;onto fixed-size vector representations, or acoustic word embeddings (AWEs).&#10;However, it remains unclear to what degree we can rely on the distance in the&#10;emerging AWE space as an estimate of word-form similarity. In this paper, we&#10;ask: does the distance in the acoustic embedding space correlate with&#10;phonological dissimilarity? To answer this question, we empirically investigate&#10;the performance of supervised approaches for AWEs with different neural&#10;architectures and learning objectives. We train AWE models in controlled&#10;settings for two languages (German and Czech) and evaluate the embeddings on&#10;two tasks: word discrimination and phonological similarity. Our experiments&#10;show that (1) the distance in the embedding space in the best cases only&#10;moderately correlates with phonological distance, and (2) improving the&#10;performance on the word discrimination task does not necessarily yield models&#10;that better reflect word phonological similarity. Our findings highlight the&#10;necessity to rethink the current intrinsic evaluations for AWEs.&#10;" />
          <attvalue for="2" value="&#10;&#10;Spoken language technologies such as spoken term discovery ~\cite{jansen2010towards, jansen2011efficient,anastasopoulos2017spoken} and query-by-example (QbE) search \cite{zhang2009unsupervised, jansen2012indexing, metze2013spoken} aim to capture, organize, and facilitate access to the linguistic content of spoken documents while abstracting away from speaker- and context-related sources of variability in speech. To this end, researchers have developed parametric models based on deep neural networks (DNNs) that project variable-length spoken word segments onto speaker-invariant vector representations, known as acoustic word embeddings (AWEs), where acoustic segments of the same word are projected nearby in space \cite{levin2013fixed, bengio2014word, kamper2016deep, settle2016discriminative, settle2017query}. AWEs, and their underlying vector-space acoustic models, enable efficient indexing and retrieval of spoken content at a scale that non-parametric template-based approaches with dynamic programming \cite{heigold2012investigations, de2007template} have failed to deliver.&#10;&#10;Several DNN architectures and learning objectives have been explored in the literature to build AWEs. State-of-the-art AWE models are trained using either contrastive objectives \cite{kamper+etal_icassp16, he+etal_iclr17} or reconstruction objectives \cite{kamper2015unsupervised, kamper2019truly}. AWEs have been used in downstream applications including ASR \cite{bengio+heigold_interspeech14} and QbE search \cite{settle2017query, yuan+etal_interspeech18}. However, evaluating the utility of AWEs using downstream applications is expensive and may not be always feasible. Therefore, researchers have developed an intrinsic evaluation for AWEs based on the acoustic word discrimination task. In this task, AWE models are evaluated based on their ability to determine whether or not two acoustic segments correspond to the same word type \cite{carlin2011rapid, kamper2015unsupervised, kamper2016deep}.&#10;&#10;Furthermore, Levin et al. \cite{levin2013fixed} have hypothesized that the distance in the emergent AWE space can be interpreted as a metric of (perceptual) dissimilarity between linguistic units (e.g., phones, syllables, words). However, none of the previous studies has empirically (in)validated this hypothesis with a rigorous evaluation beyond word discrimination. Although previous studies have proposed to incorporate the pronunciation distance in the learning objective \cite{he+etal_iclr17, yang2019linguistically}, the reported performance showed no improvement on the word discrimination task, while the distance in the AWE space has shown only a weak correlation with orthographic similarity \cite{he+etal_iclr17}. These observations, however, are yet to be systematically investigated across different architectures, objectives, and languages beyond English, which we aim to address in our study. &#10;&#10;Since AWE models have been recently adopted as cognitive models of infant phonetic learning \cite{MatusevychSKFG20} and cross-language non-native processing \cite{matusevych2021phonetic}, we argue that more effort should be devoted to analyze and understand the emergent embedding space to make sure it behaves as expected. In this paper, we take a step in this direction and make the following contributions:&#10;&#10;\begin{enumerate}[label={(\arabic*)}]&#10; \item We train AWE models with identical resources and hyperparameters and examine the effects of, and the interplay between, the architecture and learning objective on model performance (\S2).&#10;&#10;\item We analyze the correlation between the distance in the embedding space and word-form (dis)similarity, which we measure using a phonetically-informed extension of Levenshtein distance (\S3 and \S4).&#10;&#10;\item We empirically show that while AWE models trained with contrastive objectives outperform other models on the word discrimination task, they are poor at capturing phonological similarity (\S5). &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Linguistics, Deep Neural Networks, Acoustic Word Embeddings, Mathematics, Phonological Similarity" />
        </attvalues>
      </node>
      <node id="1611.04496" label="1611.04496">
        <attvalues>
          <attvalue for="0" value="Multi-view Recurrent Neural Acoustic Word Embeddings" />
          <attvalue for="1" value="  Recent work has begun exploring neural acoustic word&#10;embeddings---fixed-dimensional vector representations of arbitrary-length&#10;speech segments corresponding to words. Such embeddings are applicable to&#10;speech retrieval and recognition tasks, where reasoning about whole words may&#10;make it possible to avoid ambiguous sub-word representations. The main idea is&#10;to map acoustic sequences to fixed-dimensional vectors such that examples of&#10;the same word are mapped to similar vectors, while different-word examples are&#10;mapped to very different vectors. In this work we take a multi-view approach to&#10;learning acoustic word embeddings, in which we jointly learn to embed acoustic&#10;sequences and their corresponding character sequences. We use deep&#10;bidirectional LSTM embedding models and multi-view contrastive losses. We study&#10;the effect of different loss variants, including fixed-margin and&#10;cost-sensitive losses. Our acoustic word embeddings improve over previous&#10;approaches for the task of word discrimination. We also present results on&#10;other tasks that are enabled by the multi-view approach, including cross-view&#10;word discrimination and word similarity.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.02888" label="2008.02888">
        <attvalues>
          <attvalue for="0" value="Evaluating computational models of infant phonetic learning across&#10;  languages" />
          <attvalue for="1" value="  In the first year of life, infants' speech perception becomes attuned to the&#10;sounds of their native language. Many accounts of this early phonetic learning&#10;exist, but computational models predicting the attunement patterns observed in&#10;infants from the speech input they hear have been lacking. A recent study&#10;presented the first such model, drawing on algorithms proposed for unsupervised&#10;learning from naturalistic speech, and tested it on a single phone contrast.&#10;Here we study five such algorithms, selected for their potential cognitive&#10;relevance. We simulate phonetic learning with each algorithm and perform tests&#10;on three phone contrasts from different languages, comparing the results to&#10;infants' discrimination patterns. The five models display varying degrees of&#10;agreement with empirical observations, showing that our approach can help&#10;decide between candidate mechanisms for early phonetic learning, and providing&#10;insight into which aspects of the models are critical for capturing infants'&#10;perceptual development.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.11332" label="2101.11332">
        <attvalues>
          <attvalue for="0" value="A phonetic model of non-native spoken word processing" />
          <attvalue for="1" value="  Non-native speakers show difficulties with spoken word processing. Many&#10;studies attribute these difficulties to imprecise phonological encoding of&#10;words in the lexical memory. We test an alternative hypothesis: that some of&#10;these difficulties can arise from the non-native speakers' phonetic perception.&#10;We train a computational model of phonetic learning, which has no access to&#10;phonology, on either one or two languages. We first show that the model&#10;exhibits predictable behaviors on phone-level and word-level discrimination&#10;tasks. We then test the model on a spoken word processing task, showing that&#10;phonology may not be necessary to explain some of the word processing effects&#10;observed in non-native speakers. We run an additional analysis of the model's&#10;lexical representation space, showing that the two training languages are not&#10;fully separated in that space, similarly to the languages of a bilingual human&#10;speaker.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.09060" label="2311.09060">
        <attvalues>
          <attvalue for="0" value="Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale&#10;  of Two Benchmarks" />
          <attvalue for="1" value="  The concept of localization in LLMs is often mentioned in prior work;&#10;however, methods for localization have never been systematically and directly&#10;evaluated. We propose two complementary benchmarks that evaluate the ability of&#10;localization methods to pinpoint LLM components responsible for memorized data.&#10;In our INJ benchmark, we actively inject a piece of new information into a&#10;small subset of LLM weights, enabling us to directly evaluate whether&#10;localization methods can identify these &quot;ground truth&quot; weights. In our DEL&#10;benchmark, we evaluate localization by measuring how much dropping out&#10;identified neurons deletes a memorized pretrained sequence. Despite their&#10;different perspectives, our two benchmarks yield consistent rankings of five&#10;localization methods. Methods adapted from network pruning perform well on both&#10;benchmarks, and all evaluated methods show promising localization ability. On&#10;the other hand, even successful methods identify neurons that are not specific&#10;to a single memorized sequence.&#10;" />
          <attvalue for="2" value="&#10;Large language models (LLMs) memorize many sequences from their pretraining corpora~\cite{carlini2019secret, lehman-etal-2021-bert, lee2023language}.&#10;For example, \cite{carlini2021extracting} show that GPT2 \cite{radford2019language} can leak some private contact information verbatim.&#10;This paper studies whether we can localize a piece of memorized data, i.e., identify components in LLMs responsible for generating a sequence (near) verbatim.&#10;Successful localization may inform further work in machine unlearning \cite{cao2015towards, bourtoule2021machine}; for instance, one could apply ``neural surgery’’ to the located components to make the LLM forget a piece of sensitive information. &#10;&#10;Prior work on knowledge editing suggests that we can locate a small set of LLM parameters that store factual knowledge~\cite{dai-etal-2022-knowledge,meng2022locating}.&#10;These works demonstrate localization success by showing knowledge editing success when updating only the located LLM parameters.&#10;However, \cite{hase2023does} argue that editing success and localization are actually uncorrelated.&#10;Similarly, prior methods that identify subnetworks in LLMs \cite{gong-etal-2022-finding, panigrahi2023task} usually focus on the performance of downstream classification tasks, lacking direct evaluation on localization per se.&#10;Hence, the degree of existing methods' localization success remains unclear.&#10;&#10;This paper studies the open question, ``Do localization methods actually localize memorized data in LLMs?’’ &#10;We first propose decoupling localization success from downstream success in our \BI.&#10;Our key insight is to actively create the ground-truth weights responsible for data memorization.&#10;Specifically, we force LLMs to use a small set of pre-decided weights to memorize a piece of new information unseen during pretraining.&#10;Therefore, we have the ground-truth locations where the new information is injected.&#10;We can then directly evaluate how well different localization methods recall the indices of the injected weights.&#10;&#10;We further apply the localization methods to a real-world scenario: identifying a small set of neurons in an LLM responsible for memorizing a pretrained sequence.&#10;In this setting, evaluating localization success is more challenging because the ground-truth ``location'' of each memorized sequence is unknown.&#10;We propose the \BII, inspired by knockouts \cite{olsson2022context}, a reverse-engineering approach that removes a set of nodes from the computation graph to observe their importance for specific model behavior. &#10;We first collect a set of memorized sequences, and for each sequence, we drop out the located neurons to measure their importance to memorizing that target sequence.&#10;A successful localization should cleanly erase the target sequence from an LLM without hurting the memorization of the other sequences in the set after dropout.&#10;Our two benchmarks complement each other: the \BI\ provides a direct evaluation of localization methods under a well-controlled setup, while \BII\ answers if the methods can localize pretrained sequences that LLMs have already memorized.&#10;&#10;We systematically evaluate five methods on our two benchmarks, including existing localization methods (\Act, \cite{geva-etal-2022-transformer}; \IG, \cite{dai-etal-2022-knowledge}), a brute-force method that searches for the most important neurons (\Zero), and two methods we adapt from network pruning \cite{hassibi1992second,han2015deep}, \Slim\ and \HC.&#10;Our two benchmarks rank the five methods in the same order, showing especially strong localization ability for \HC.&#10;For example, dropping out only $0.5\%$ of neurons in Pythia-6.9B \cite{biderman2023pythia} identified by \HC\ makes the model forget $57.7\%$ of the target memorized tokens on average.&#10;On the other hand, the \BII\ shows all methods struggle to balance between erasing the target sequence and retaining other memorized data, indicating that the identified neurons are also relevant for memorizing some other sequences.&#10;Overall, both benchmarks agree all evaluated localization methods are promising, but precise localization of a single sequence remains difficult.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Cognitive Science, Localization Methods, Benchmark Evaluation, Neural Networks, Mathematics" />
        </attvalues>
      </node>
      <node id="2301.04213" label="2301.04213">
        <attvalues>
          <attvalue for="0" value="Does Localization Inform Editing? Surprising Differences in&#10;  Causality-Based Localization vs. Knowledge Editing in Language Models" />
          <attvalue for="1" value="  Language models learn a great quantity of factual information during&#10;pretraining, and recent work localizes this information to specific model&#10;weights like mid-layer MLP weights. In this paper, we find that we can change&#10;how a fact is stored in a model by editing weights that are in a different&#10;location than where existing methods suggest that the fact is stored. This is&#10;surprising because we would expect that localizing facts to specific model&#10;parameters would tell us where to manipulate knowledge in models, and this&#10;assumption has motivated past work on model editing methods. Specifically, we&#10;show that localization conclusions from representation denoising (also known as&#10;Causal Tracing) do not provide any insight into which model MLP layer would be&#10;best to edit in order to override an existing stored fact with a new one. This&#10;finding raises questions about how past work relies on Causal Tracing to select&#10;which model layers to edit. Next, we consider several variants of the editing&#10;problem, including erasing and amplifying facts. For one of our editing&#10;problems, editing performance does relate to localization results from&#10;representation denoising, but we find that which layer we edit is a far better&#10;predictor of performance. Our results suggest, counterintuitively, that better&#10;mechanistic understanding of how pretrained language models work may not always&#10;translate to insights about how to best change their behavior. Our code is&#10;available at https://github.com/google/belief-localization&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.03378" label="2303.03378">
        <attvalues>
          <attvalue for="0" value="PaLM-E: An Embodied Multimodal Language Model" />
          <attvalue for="1" value="  Large language models excel at a wide range of complex tasks. However,&#10;enabling general inference in the real world, e.g., for robotics problems,&#10;raises the challenge of grounding. We propose embodied language models to&#10;directly incorporate real-world continuous sensor modalities into language&#10;models and thereby establish the link between words and percepts. Input to our&#10;embodied language model are multi-modal sentences that interleave visual,&#10;continuous state estimation, and textual input encodings. We train these&#10;encodings end-to-end, in conjunction with a pre-trained large language model,&#10;for multiple embodied tasks including sequential robotic manipulation planning,&#10;visual question answering, and captioning. Our evaluations show that PaLM-E, a&#10;single large embodied multimodal model, can address a variety of embodied&#10;reasoning tasks, from a variety of observation modalities, on multiple&#10;embodiments, and further, exhibits positive transfer: the model benefits from&#10;diverse joint training across internet-scale language, vision, and&#10;visual-language domains. Our largest model, PaLM-E-562B with 562B parameters,&#10;in addition to being trained on robotics tasks, is a visual-language generalist&#10;with state-of-the-art performance on OK-VQA, and retains generalist language&#10;capabilities with increasing scale.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.03959" label="2306.03959">
        <attvalues>
          <attvalue for="0" value="Leveraging Explicit Procedural Instructions for Data-Efficient Action&#10;  Prediction" />
          <attvalue for="1" value="  Task-oriented dialogues often require agents to enact complex, multi-step&#10;procedures in order to meet user requests. While large language models have&#10;found success automating these dialogues in constrained environments, their&#10;widespread deployment is limited by the substantial quantities of task-specific&#10;data required for training. The following paper presents a data-efficient&#10;solution to constructing dialogue systems, leveraging explicit instructions&#10;derived from agent guidelines, such as company policies or customer service&#10;manuals. Our proposed Knowledge-Augmented Dialogue System (KADS) combines a&#10;large language model with a knowledge retrieval module that pulls documents&#10;outlining relevant procedures from a predefined set of policies, given a&#10;user-agent interaction. To train this system, we introduce a semi-supervised&#10;pre-training scheme that employs dialogue-document matching and action-oriented&#10;masked language modeling with partial parameter freezing. We evaluate the&#10;effectiveness of our approach on prominent task-oriented dialogue datasets,&#10;Action-Based Conversations Dataset and Schema-Guided Dialogue, for two dialogue&#10;tasks: action state tracking and workflow discovery. Our results demonstrate&#10;that procedural knowledge augmentation improves accuracy predicting in- and&#10;out-of-distribution actions while preserving high performance in settings with&#10;low or sparse data.&#10;" />
          <attvalue for="2" value="&#10;&#10;For many real-world applications, it is crucial for task-oriented dialogue (TOD) systems to complete user requests while strictly adhering to established procedures. For example, consider a customer service agent who must first verify a client's details before changing their password. Although large language models have demonstrated potential in modeling such dialogues, they require large amounts of data with consistent procedural representations to implicitly store procedures in the parameters of their underlying networks. In practical settings, such high-quality data is not always readily available as some procedures may naturally occur infrequently or change over time. In this paper, we explore a solution to TOD modeling which improves performance in low-data settings by referencing explicitly stored agent guidelines.&#10;&#10;We outline a methodology of incorporating procedural knowledge (i.e., knowledge concerning the requisite steps to address a user inquiry) into a language model with the objective of predicting agent actions in dialogue tasks. Our proposed system, the Knowledge-Augmented Dialogue System (KADS), consists of two modules: a knowledge retriever which, given a dialogue between an agent and user, retrieves the most pertinent instructions from a knowledge base of agent procedures and a language model which considers the retrieved instructions along with the ongoing dialogue to inform an action prediction (see architecture in \ref{arch}). &#10;&#10;In prior work, retrieval-enhanced language models have achieved success integrating external knowledge from internet searches into conversational agents \cite{shuster2022,thoppilan2022}. However, a more controllable approach is necessary for instruction retrieval in task-oriented dialogue. Rather than querying the open web, it's more suitable to perform retrieval over a closed set of documents, like in \cite{guu2020,lewis2020}. However, while the training schemes utilized in these works sufficiently prime a model for question-answering tasks, they are not as effective for action prediction. &#10;&#10;Following the lines of \cite{henderson2021}, which introduces a unique pre-training objective for slot-labeling, our method leverages custom objectives suited for action prediction tasks. We employ a specialized warm-up task where dialogues are matched with corresponding procedural instructions to ensure that the knowledge retrieval module is initialized with reasonable dialogue and document embeddings. Then, the system is trained on an special case of masked language modeling in which masked actions are predicted from customer-agent dialogues. Finally, we found it necessary to encourage our system to incorporate signal from retrieved procedures by routinely freezing the language model's weights during training.&#10;&#10;We evaluated this approach on two dialogue tasks--- action state tracking and workflow discovery--- using two task-oriented dialogue datasets: Action-Based Conversations Dataset and Schema-Guided Dialogue. Our results suggest that KADS yields improved action prediction accuracy against several baselines, including an un-augmented language model and a language model augmented with static guidelines, on both in- and out-of-distribution procedures. Furthermore, we demonstrate that knowledge augmentation bolsters our system's ability to predict actions that occur infrequently in the training data.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Dialogue Systems, Artificial Intelligence, Natural Language Processing, Language Modeling, Knowledge Retrieval" />
        </attvalues>
      </node>
      <node id="2010.11791" label="2010.11791">
        <attvalues>
          <attvalue for="0" value="ConVEx: Data-Efficient and Few-Shot Slot Labeling" />
          <attvalue for="1" value="  We propose ConVEx (Conversational Value Extractor), an efficient pretraining&#10;and fine-tuning neural approach for slot-labeling dialog tasks. Instead of&#10;relying on more general pretraining objectives from prior work (e.g., language&#10;modeling, response selection), ConVEx's pretraining objective, a novel pairwise&#10;cloze task using Reddit data, is well aligned with its intended usage on&#10;sequence labeling tasks. This enables learning domain-specific slot labelers by&#10;simply fine-tuning decoding layers of the pretrained general-purpose sequence&#10;labeling model, while the majority of the pretrained model's parameters are&#10;kept frozen. We report state-of-the-art performance of ConVEx across a range of&#10;diverse domains and data sets for dialog slot-labeling, with the largest gains&#10;in the most challenging, few-shot setups. We believe that ConVEx's reduced&#10;pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its&#10;efficient fine-tuning and strong performance, promise wider portability and&#10;scalability for data-efficient sequence-labeling tasks in general.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.09040" label="2312.09040">
        <attvalues>
          <attvalue for="0" value="STaR: Distilling Speech Temporal Relation for Lightweight Speech&#10;  Self-Supervised Learning Models" />
          <attvalue for="1" value="  Albeit great performance of Transformer-based speech selfsupervised learning&#10;(SSL) models, their large parameter size and computational cost make them&#10;unfavorable to utilize. In this study, we propose to compress the speech SSL&#10;models by distilling speech temporal relation (STaR). Unlike previous works&#10;that directly match the representation for each speech frame, STaR distillation&#10;transfers temporal relation between speech frames, which is more suitable for&#10;lightweight student with limited capacity. We explore three STaR distillation&#10;objectives and select the best combination as the final STaR loss. Our model&#10;distilled from HuBERT BASE achieves an overall score of 79.8 on SUPERB&#10;benchmark, the best performance among models with up to 27 million parameters.&#10;We show that our method is applicable across different speech SSL models and&#10;maintains robust performance with further reduced parameters.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformer-based speech self-supervised learning (SSL) models \cite{baevski2020wav2vec, hsu2021hubert, chen2021wavlm} have risen to prominence with great performance in various speech-related tasks\,\cite{yang21c_interspeech, feng2023superb}.&#10;Nonetheless, the downside of these models mainly comes from the requirement of substantial computational resources during the pre-training stage---HuBERT\,Base takes over 82 GPU-days for pre-training, and 32 GPUs en masse are utilized to shorten this\,\cite{hsu2021hubert}.&#10;Another downside is their large parameter size which makes on-device application difficult and renders the speech SSL models unfavorable for many practical scenarios.&#10;The above issues make compression techniques essential for the speech SSL models, and several studies have attempted to solve these issues by applying pruning\,\cite{lai2021parp} or quantization\,\cite{wang2022deep} technique.&#10;&#10;Alternatively, knowledge distillation is another approach for model compression that trains a student model with a smaller parameter size to imitate the behavior of a larger teacher model by matching the student's representation to the teacher's.&#10;Knowledge distillation of the Transformer-based speech SSL models is being actively studied, with previous works of task-specific compression including automatic speech recognition\,(ASR)\,\cite{choi2023masked}, keyword spotting (KWS)\,\cite{lim2023lightweight}, and automatic speaker verification\,(ASV)\,\cite{heo23_interspeech}.&#10;However, given that the speech SSL models are utilized in various downstream tasks, their approaches are limited to only a specific task.&#10;&#10;On the other hand, approaches that realize task-agnostic compression via knowledge distillation include DistilHuBERT\,\cite{chang2021distilhubert} and FitHuBERT\,\cite{lee2022fithubert}, where the former suggests shallow and wide student model design, and the latter suggests deep and narrow one.&#10;ARMHuBERT\,\cite{jang23_interspeech} proposes to reuse attention maps across Transformer\,\cite{vaswani2017attention} layers and utilizes both masked and unmasked speech frames for masking distillation.&#10;Some approaches\,\cite{peng23c_interspeech, wang23da_interspeech} jointly conduct distillation with $L_0$ regularization\,\cite{louizos2018learning} and structured pruning.&#10;LightHuBERT\,\cite{wang2022lighthubert} implements masking distillation and architecture search, but it stands apart from other methods due to its excessive computational demand for creating a teacher-sized supernet.&#10;&#10;While the aforementioned studies have demonstrated promising results, two limitations still exist.&#10;First, most approaches neglect the weak representation capacity of the student and directly match the complex teacher's representation for each speech frame by introducing additional linear heads\,\cite{chang2021distilhubert, lee2022fithubert, jang23_interspeech, peng23c_interspeech, ashihara2022deep}.&#10;This can be an over-constraint for the lightweight student model, necessitating the establishment of an alternative distillation objective that better suits to the student.&#10;Some studies even inefficiently discard these trained linear heads after distillation, although they can convey the teacher's knowledge\,\cite{chang2021distilhubert, lee2022fithubert, peng23c_interspeech, ashihara2022deep}.&#10;Second, while pruning allows us to manage the size of model parameters by a sparsity ratio, it cannot determine computational cost of the model at the same time.&#10;As a result, computational overhead can be higher than the vanilla distillation approaches, in which the model is specified before the training.&#10;&#10;In this study, we explore effective distillation objectives that capture temporal relation between speech frames for lightweight student model.&#10;Additional parameters are not necessary during distillation, enabling the construction of a more compact and computationally efficient student model.&#10;We verify the task-agnostic compression of our proposed objectives using SUPERB benchmark\,\cite{yang21c_interspeech}.&#10;Our model distilled from HuBERT\,Base achieves the best overall score\,\cite{chen2021wavlm} of 79.8 among models with $\sim$\,27 million parameters, while requiring only 30.7\% multiply-accumulates\,(MACs) and 28.1\% parameters of its teacher model.&#10;It even surpasses LightHuBERT\,\cite{wang2022lighthubert}, which demands extensive computational resources for compression, in terms of overall score, number of parameters, and MACs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Compression, Deep Learning, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2303.07592" label="2303.07592">
        <attvalues>
          <attvalue for="0" value="Lightweight feature encoder for wake-up word detection based on&#10;  self-supervised speech representation" />
          <attvalue for="1" value="  Self-supervised learning method that provides generalized speech&#10;representations has recently received increasing attention. Wav2vec 2.0 is the&#10;most famous example, showing remarkable performance in numerous downstream&#10;speech processing tasks. Despite its success, it is challenging to use it&#10;directly for wake-up word detection on mobile devices due to its expensive&#10;computational cost. In this work, we propose LiteFEW, a lightweight feature&#10;encoder for wake-up word detection that preserves the inherent ability of&#10;wav2vec 2.0 with a minimum scale. In the method, the knowledge of the&#10;pre-trained wav2vec 2.0 is compressed by introducing an auto-encoder-based&#10;dimensionality reduction technique and distilled to LiteFEW. Experimental&#10;results on the open-source &quot;Hey Snips&quot; dataset show that the proposed method&#10;applied to various model structures significantly improves the performance,&#10;achieving over 20% of relative improvements with only 64k parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.17394" label="2305.17394">
        <attvalues>
          <attvalue for="0" value="One-Step Knowledge Distillation and Fine-Tuning in Using Large&#10;  Pre-Trained Self-Supervised Learning Models for Speaker Verification" />
          <attvalue for="1" value="  The application of speech self-supervised learning (SSL) models has achieved&#10;remarkable performance in speaker verification (SV). However, there is a&#10;computational cost hurdle in employing them, which makes development and&#10;deployment difficult. Several studies have simply compressed SSL models through&#10;knowledge distillation (KD) without considering the target task. Consequently,&#10;these methods could not extract SV-tailored features. This paper suggests&#10;One-Step Knowledge Distillation and Fine-Tuning (OS-KDFT), which incorporates&#10;KD and fine-tuning (FT). We optimize a student model for SV during KD training&#10;to avert the distillation of inappropriate information for the SV. OS-KDFT&#10;could downsize Wav2Vec 2.0 based ECAPA-TDNN size by approximately 76.2%, and&#10;reduce the SSL model's inference time by 79% while presenting an EER of 0.98%.&#10;The proposed OS-KDFT is validated across VoxCeleb1 and VoxCeleb2 datasets and&#10;W2V2 and HuBERT SSL models. Experiments are available on our GitHub.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.12712" label="2309.12712">
        <attvalues>
          <attvalue for="0" value="Big model only for hard audios: Sample dependent Whisper model selection&#10;  for efficient inferences" />
          <attvalue for="1" value="  Recent progress in Automatic Speech Recognition (ASR) has been coupled with a&#10;substantial increase in the model sizes, which may now contain billions of&#10;parameters, leading to slow inferences even with adapted hardware. In this&#10;context, several ASR models exist in various sizes, with different inference&#10;costs leading to different performance levels. Based on the observation that&#10;smaller models perform optimally on large parts of testing corpora, we propose&#10;to train a decision module, that would allow, given an audio sample, to use the&#10;smallest sufficient model leading to a good transcription. We apply our&#10;approach to two Whisper models with different sizes. By keeping the decision&#10;process computationally efficient, we build a decision module that allows&#10;substantial computational savings with reduced performance drops.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent progress in neural-based automatic speech recognition (ASR) has been driven by new modelling architectures, data collection and processing but also by larger models that have recently exceeded a billion parameters \cite{whisper}. Such advances have promised enhanced accuracy and capabilities, yet they have also ushered in escalating computational demands. These ASR models are usually available in a certain range of sizes with varying performance levels. For instance, Whisper models \cite{whisper} are available in 6 sizes from Tiny (39M parameters) to Large (1.5B parameters), Nvidia FastConformers \cite{rekesh2023fast} range from Large (118M parameters) to XXLarge (1.2B parameters) and self-supervised models like Hubert \cite{hsu2021hubert} or WavLM \cite{chen2021wavlm} are generally available in Base and Large versions. Systematically, following the deep learning trend across modalities, larger version models, although generally trained on the same datasets, perform substantially better than their reduced-size counterparts. This is shown in Figure \ref{fig:res} (a), where the mean Word Error Rates (WER) of four Whisper models with different sizes on the test set of CommonVoice \cite{CV} are presented. The mean WER drops from $28.1$ with the ``Tiny&quot; version to $10.2$ with the ``Medium&quot; one.&#10;&#10;However, as shown in Figure \ref{fig:res} (b), this performance drop may not concern a significant part of the testing points. In this figure, every cell $(i,j)$ shows the proportion of samples in the CommonVoice test set where model $i$ performs better or equally to model $j$. For instance, the third cell in the first line (cell $(0,2)$) states that for $52\%$ of the testing samples, the Tiny model (39M) performs equally or better than the Small one (244M) while bearing more than $6$ times fewer parameters. Based on this observation, this work explores whether we can predict if audio samples will fall into this category. By doing so, audio samples that would not benefit from the costly inference of a large model can be assigned to a smaller one in order to reduce the total computational load. &#10;&#10;More precisely, this study aims to develop a decision module that, given an audio sample, chooses a Whisper model version that has the lowest inference cost without WER degradation. Due to the complexity of the task, in this paper, we only focus on deciding if an audio sample should be decoded with Whisper Tiny (39M) or with Whisper Small (244M). These two model versions are relevant candidates as they exhibit high differences in both WER, inference costs and latency. &#10;&#10;A few works \cite{ewer2, ewer3} have already attempted to choose among several ASR model versions using WER prediction. Given the textual output of an ASR model, they explored the prediction of the sentence-level WER. Yet, these methods are not aiming to reduce inference costs but rather to decide whether an audio sample should be retreated by a more complex ASR model. Indeed, the most efficient techniques predict WER using full ASR pipelines based on either acoustic encoding and language model (LM) beam search \cite{ ewer2, ewer3}. Such methods that rely on costly beam searches cannot be used in our case where the aim is to reduce the computational load. &#10;&#10;Another close line of work is dynamic or early-exiting approaches. Instead of saving computation by choosing between separate ASR models, these methods have attempted to make forward passes lighter by skipping some of the last transformer layers of an ASR model \cite{salahrobin, yoon2022hubertee}. Decision to exits is based on entropy or representation-similarity thresholds. However, early-exiting, as developed in these works, can only save layer computation in the ASR encoder, while, as shown in Table \ref{tab:compute}, for attentional encoder-decoder architectures, most of the computations occur in the beam search decoding. &#10;&#10;The closest work to our effort is from Lugosch et al. \cite{surprisal}, who propose to save computation cost at inference time by choosing between a large and a small ASR decoder. However, their method relies only on the log-likelihood of the encoder which is one of the baselines of our work. &#10;&#10;This paper explores possibilities to build a decision module that allows efficient selection between different ASR model sizes while keeping high performance. Our contributions are threefold : &#10;\begin{itemize}&#10; \item We successfully reduce inference costs for a negligible WER degradation. In addition, our method can be used to interpolate between model sizes, saving the need for costly training of intermediate models.\item We explore different inputs and architectures for the decision module and compare them to several baselines and toplines. &#10; \item The codebase, developed within the SpeechBrain \cite{ravanelli2021speechbrain} framework, is released for further investigations. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Signal Processing, Speech Recognition, Computational Efficiency, Model Optimization" />
        </attvalues>
      </node>
      <node id="2008.03403" label="2008.03403">
        <attvalues>
          <attvalue for="0" value="Word Error Rate Estimation Without ASR Output: e-WER2" />
          <attvalue for="1" value="  Measuring the performance of automatic speech recognition (ASR) systems&#10;requires manually transcribed data in order to compute the word error rate&#10;(WER), which is often time-consuming and expensive. In this paper, we continue&#10;our effort in estimating WER using acoustic, lexical and phonotactic features.&#10;Our novel approach to estimate the WER uses a multistream end-to-end&#10;architecture. We report results for systems using internal speech decoder&#10;features (glass-box), systems without speech decoder features (black-box), and&#10;for systems without having access to the ASR system (no-box). The no-box system&#10;learns joint acoustic-lexical representation from phoneme recognition results&#10;along with MFCC acoustic features to estimate WER. Considering WER per&#10;sentence, our no-box system achieves 0.56 Pearson correlation with the&#10;reference evaluation and 0.24 root mean square error (RMSE) across 1,400&#10;sentences. The estimated overall WER by e-WER2 is 30.9% for a three hours test&#10;set, while the WER computed using the reference transcriptions was 28.5%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.06740" label="2303.06740">
        <attvalues>
          <attvalue for="0" value="Fine-tuning Strategies for Faster Inference using Speech Self-Supervised&#10;  Models: A Comparative Study" />
          <attvalue for="1" value="  Self-supervised learning (SSL) has allowed substantial progress in Automatic&#10;Speech Recognition (ASR) performance in low-resource settings. In this context,&#10;it has been demonstrated that larger self-supervised feature extractors are&#10;crucial for achieving lower downstream ASR error rates. Thus, better&#10;performance might be sanctioned with longer inferences. This article explores&#10;different approaches that may be deployed during the fine-tuning to reduce the&#10;computations needed in the SSL encoder, leading to faster inferences. We adapt&#10;a number of existing techniques to common ASR settings and benchmark them,&#10;displaying performance drops and gains in inference times. Interestingly, we&#10;found that given enough downstream data, a simple downsampling of the input&#10;sequences outperforms the other methods with both low performance drops and&#10;high computational savings, reducing computations by 61.3% with an WER increase&#10;of only 0.81. Finally, we analyze the robustness of the comparison to changes&#10;in dataset conditions, revealing sensitivity to dataset size.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.01659" label="2006.01659">
        <attvalues>
          <attvalue for="0" value="Surprisal-Triggered Conditional Computation with Neural Networks" />
          <attvalue for="1" value="  Autoregressive neural network models have been used successfully for sequence&#10;generation, feature extraction, and hypothesis scoring. This paper presents yet&#10;another use for these models: allocating more computation to more difficult&#10;inputs. In our model, an autoregressive model is used both to extract features&#10;and to predict observations in a stream of input observations. The surprisal of&#10;the input, measured as the negative log-likelihood of the current observation&#10;according to the autoregressive model, is used as a measure of input&#10;difficulty. This in turn determines whether a small, fast network, or a big,&#10;slow network, is used. Experiments on two speech recognition tasks show that&#10;our model can match the performance of a baseline in which the big network is&#10;always used with 15% fewer FLOPs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.10684" label="2305.10684">
        <attvalues>
          <attvalue for="0" value="Data Augmentation for Diverse Voice Conversion in Noisy Environments" />
          <attvalue for="1" value="  Voice conversion (VC) models have demonstrated impressive few-shot conversion&#10;quality on the clean, native speech populations they're trained on. However,&#10;when source or target speech accents, background noise conditions, or&#10;microphone characteristics differ from training, quality voice conversion is&#10;not guaranteed. These problems are often left unexamined in VC research, giving&#10;rise to frustration in users trying to use pretrained VC models on their own&#10;data. We are interested in accent-preserving voice conversion for name&#10;pronunciation from self-recorded examples, a domain in which all three of the&#10;aforementioned conditions are present, and posit that demonstrating higher&#10;performance in this domain correlates with creating VC models that are more&#10;usable by otherwise frustrated users. We demonstrate that existing SOTA&#10;encoder-decoder VC models can be made robust to these variations and endowed&#10;with natural denoising capabilities using more diverse data and simple data&#10;augmentation techniques in pretraining.&#10;" />
          <attvalue for="2" value="&#10;&#10;Voice conversion (VC) is the task of generating utterances in a target speaker's voice that carry the content and prosody from a source utterance from a different speaker \cite{661472}, preserving the content of the source utterance while reproducing the characteristics and style of the target speaker. &#10;VC was originally conceived of as a data-efficient way to add styles and personalities to 90s-era text-to-speech systems, to improve the quality of decoded speech in telephony, and as a way to preserve speaker individuality under speech translation \cite{MOULINES1995125}.&#10;Many of these problems have been solved with other techniques, and VC has come to be treated as a novelty task for demonstrating innovations in data-efficient and few-shot generative modeling&#10;\cite{qian2019autovc},&#10;and real-world use-cases for VC technologies are no longer centered. &#10;&#10;However, niche real-world applications for VC still exist. &#10;One example is multicultural name pronunciation (e.g., in graduation ceremonies) where the desired standard is for one's own name to be read aloud as one pronounces it themself.&#10;Typically, ceremony organizers solicit self-recordings of awardees voices, typically produced on their own cell phones. &#10;We propose treating these recordings as source utterances for conversion into the organizer's target voice. &#10;Under these conditions, represented phonemes differ considerably from those present in the VC model training speech distribution and utterances have inconsistent microphone characteristics and environmental noise. &#10;Unfortunately, we find that existing pretrained few-shot VC models, such as AutoVC \cite{qian2019autovc} and FragmentVC \cite{lin2021fragmentvc} perform poorly in these conditions.&#10;Is it possible to adapt these SOTA VC models to be performant in this setting? Yes, we find.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Acoustics, Voice Conversion Models, Machine Learning, Computer Science, Speech Processing, Linguistics, Robust Speech Processing, Natural Language Processing, Accent Preservation Techniques" />
        </attvalues>
      </node>
      <node id="2109.12258" label="2109.12258">
        <attvalues>
          <attvalue for="0" value="Pushing on Text Readability Assessment: A Transformer Meets Handcrafted&#10;  Linguistic Features" />
          <attvalue for="1" value="  We report two essential improvements in readability assessment: 1. three&#10;novel features in advanced semantics and 2. the timely evidence that&#10;traditional ML models (e.g. Random Forest, using handcrafted features) can&#10;combine with transformers (e.g. RoBERTa) to augment model performance. First,&#10;we explore suitable transformers and traditional ML models. Then, we extract&#10;255 handcrafted linguistic features using self-developed extraction software.&#10;Finally, we assemble those to create several hybrid models, achieving&#10;state-of-the-art (SOTA) accuracy on popular datasets in readability assessment.&#10;The use of handcrafted features help model performance on smaller datasets.&#10;Notably, our RoBERTA-RF-T1 hybrid achieves the near-perfect classification&#10;accuracy of 99%, a 20.3% increase from the previous SOTA.&#10;" />
          <attvalue for="2" value="&#10;The long quest for advancing readability assessment (RA) mostly centered on handcrafting the linguistic features that affect readability \cite{Pitler:08}. RA is a time-honored branch of natural language processing (NLP) that quantifies the difficulty with which a reader understands a text \cite{Feng:10}. Being one of the oldest systematic approaches to linguistics \cite{Collins-Thompson:14}, RA developed various linguistic features. These range from simple measures like the average count of syllables to those as sophisticated as semantic complexity \cite{buchanan2001characterizing}.&#10;&#10;Perhaps due to the abundance of dependable linguistic features, an overwhelming majority of RA systems are Support Vector Machines (SVM) with handcrafted features \cite{Hansen:21}. Such traditional machine learning (ML) methods were linguistically explainable, expandable, and most importantly, competent against the modern neural models. As a fragmentary example, \cite{Fili:19} reports that a large ensemble of 6 BiLSTMs with BERT \cite{bert}, ELMo \cite{elmo}, Word2Vec \cite{word2vec}, and GloVe \cite{glove} embeddings showed only $\sim$1\% accuracy improvement from a single SVM model developed by \cite{Xia:16}.&#10;&#10;Even though deep neural networks have achieved state-of-the-art (SOTA) performance in almost all semantic tasks where sufficient data were available \cite{colbert:11, Zhang:15}, neural models started showing promising results in RA only quite recently \cite{Martinc:21}. A known challenge for the researchers in RA is the lack of large public datasets -- with the unique exception of WeeBit \cite{Vajjala:12}. Technically speaking, even WeeBit is not entirely public since it has to be directly obtained from the authors.&#10;&#10;\cite{Martinc:21} raised the SOTA classification accuracy on the popular WeeBit dataset \cite{Vajjala:12} by about 4\% using BERT. This was the first solid proof that neural models with auto-generated features can show significant improvement compared to traditional ML with handcrafted features. However, neural models, or transformers (which is the interest of this paper), still show not much better performance than traditional ML on smaller datasets like OneStopEnglish \cite{Vajjala:18}, despite the complexity. &#10;&#10;From our observations, the reported low performances of transformers on small RA datasets can be accounted for two reasons. 1. Only BERT was applied to RA, and there could be other transformers that perform better, even on small datasets. 2. If a transformer shows weak performance on small datasets, there must be some additional measures done to supply the final model (e.g. ensemble) with more linguistic information, but such a study is rare in RA. Hence, we tackle the abovementioned issues in this paper. In particular, we 1. perform a wide search on transformers, traditional ML models, and handcrafted features \&amp; 2. develop a hybrid architecture for SOTA and robustness on small datasets.&#10;&#10;However, before we move on to hybrid models, we begin by supplementing an underexplored linguistic branch of handcrafted features. According to survey research on RA \cite{Collins-Thompson:14}, the study on advanced semantics is scarce. We lack a model to capture how deeper semantic structures affect readability. We attempt to solve this issue by viewing a text as a collection of latent topics and calculating the probability distribution.&#10;&#10;Then, we move on to combine traditional ML (w handcrafted features) and transformers. Such a hybrid system is only reported by \cite{deutsch-etal-2020-linguistic}, concluding, ``(hybrid models) did not achieve SOTA performance.'' But we obtain contrary results. Through a large study on the optimal combination, we obtain SOTA results on WeeBit and OneStopEnglish. Also, our BERT-GB-T1 hybrid beats the (previous) SOTA accuracy with only 30\% of the full dataset, in section4.7.&#10;&#10;Our main objectives are creating advanced semantic features and hybrid models. But our contributions to academia are not limited to the abovementioned two. We make the following additions:&#10;&#10;1. We numerically represent certain linguistic properties pertaining to advanced semantics.&#10;&#10;2. We develop a large-scale, openly available 255 features extraction Python toolkit (which is highly scarce in RA). We name the software LingFeat.&#10;&#10;3. We conduct wide searches and parametrizations on transformers and traditional ML for RA use.&#10;&#10;4. We develop hybrid models for SOTA and robu-stness on small datasets. Notably, RoBERTa-RF-T1 achieves 99\% accuracy on OneStopEnglish, 20.3\% higher than the previous SOTA (table 5).&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Hybrid Modeling Techniques, Machine Learning, Computer Science, Linguistics, Readability Assessment, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1907.11779" label="1907.11779">
        <attvalues>
          <attvalue for="0" value="Supervised and Unsupervised Neural Approaches to Text Readability" />
          <attvalue for="1" value="  We present a set of novel neural supervised and unsupervised approaches for&#10;determining the readability of documents. In the unsupervised setting, we&#10;leverage neural language models, whereas in the supervised setting, three&#10;different neural classification architectures are tested. We show that the&#10;proposed neural unsupervised approach is robust, transferable across languages&#10;and allows adaptation to a specific readability task and data set. By&#10;systematic comparison of several neural architectures on a number of benchmark&#10;and new labelled readability datasets in two languages, this study also offers&#10;a comprehensive analysis of different neural approaches to readability&#10;classification. We expose their strengths and weaknesses, compare their&#10;performance to current state-of-the-art classification approaches to&#10;readability, which in most cases still rely on extensive feature engineering,&#10;and propose possibilities for improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.06281" label="2408.06281">
        <attvalues>
          <attvalue for="0" value="MovieSum: An Abstractive Summarization Dataset for Movie Screenplays" />
          <attvalue for="1" value="  Movie screenplay summarization is challenging, as it requires an&#10;understanding of long input contexts and various elements unique to movies.&#10;Large language models have shown significant advancements in document&#10;summarization, but they often struggle with processing long input contexts.&#10;Furthermore, while television transcripts have received attention in recent&#10;studies, movie screenplay summarization remains underexplored. To stimulate&#10;research in this area, we present a new dataset, MovieSum, for abstractive&#10;summarization of movie screenplays. This dataset comprises 2200 movie&#10;screenplays accompanied by their Wikipedia plot summaries. We manually&#10;formatted the movie screenplays to represent their structural elements.&#10;Compared to existing datasets, MovieSum possesses several distinctive features:&#10;(1) It includes movie screenplays, which are longer than scripts of TV&#10;episodes. (2) It is twice the size of previous movie screenplay datasets. (3)&#10;It provides metadata with IMDb IDs to facilitate access to additional external&#10;knowledge. We also show the results of recently released large language models&#10;applied to summarization on our dataset to provide a detailed baseline.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large language models have shown significant improvements in abstractive summarization in recent years \cite{zhong2022dialoglm,10.1145/3404835.3462846,zhong-etal-2021-qmsum,zhang-etal-2022-summn}, aiming to produce a concise and coherent summary of the input document. However, these models often struggle when the input context is long, particularly when the relevant information is distributed across the document \cite{liu2023lost}. To better understand this phenomenon and to advance research, datasets are needed that not only contain long-form documents but also have the property that important information is dispersed throughout the document. Movie screenplays have these characteristics: to generate a faithful summary, an understanding of characters and events across the entire length of the screenplay is required.&#10;&#10;More recently, narrative summarization research has focused on TV shows and books \cite{kryscinski2021booksum,moskvichev-mai-2023-narrativexl}, with less attention given to movie screenplays \cite{gorinski-lapata-2015-movie,papalampidi-etal-2020-screenplay}. Notably, \cite{chen-etal-2022-summscreen} introduced a dataset of TV show transcripts which has gained considerable interest and was included in a long document summarization benchmark \cite{shaham-etal-2022-scrolls}. But unlike movie screenplays, TV episode transcripts tend to be relatively short and predominantly comprise spoken dialogue with minimal scene or character descriptions. Additionally, they are not self-contained, as the events or characters from previous episodes can be referred to. In contrast, movie screenplays are structured documents with various screenplay elements such as scene headings, locations, character names, dialogues and detailed scene descriptions. These are written by screenwriters and are characteristically formatted to denote each element. &#10;&#10;The largest current movie screenplay dataset \cite{gorinski-lapata-2015-movie,gorinski-lapata-2018-whats} comprises 917 automatically formatted screenplays (ScriptBase-j), with the most recent movie from 2013. We built MovieSum, a new movie screenplay dataset for abstractive summarization, which consists of 2200 movies, more than twice the size ScriptBase-j. Importantly, our new dataset has been formatted using a professional script writing tool and paired with Wikipedia plot summaries. Each movie is also tagged with its IMDB IDs to facilitate the collection of other external knowledge in rhw future. The dataset consists of movies spanning a wide range of genres from 1930 to~2023. &#10;&#10;We provide a detailed description of MovieSum, including the steps for collecting and filtering screenplays and statistics and comparison with other narrative datasets. We conduct extensive experiments to evaluate the performance of state-of-the-art summarization models on MovieSum, demonstrating its utility as a benchmark dataset for narrative summarization research. Experiments suggest that recent models struggle with long abstractive summarization, and we hope that our efforts will inspire further research in this area. Furthermore, we provide qualitative analyses of how the structure of the screenplay can be utilized in generating summaries.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Movie Summarization, Computer Science, Linguistics, Dataset Development, Language Models, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.13877" label="2305.13877">
        <attvalues>
          <attvalue for="0" value="NarrativeXL: A Large-scale Dataset For Long-Term Memory Models" />
          <attvalue for="1" value="  We propose a new large-scale (nearly a million questions) ultra-long-context&#10;(more than 50,000 words average document length) reading comprehension dataset.&#10;Using GPT 3.5, we summarized each scene in 1,500 hand-curated fiction books&#10;from Project Gutenberg, which resulted in approximately 150 scene-level&#10;summaries per book. After that, we created a number of reading comprehension&#10;questions based on these summaries, including three types of multiple-choice&#10;scene recognition questions, as well as free-form narrative reconstruction&#10;questions. With 990,595 total questions, our dataset is an order of magnitude&#10;larger than the closest alternatives. Crucially, most questions have a known&#10;``retention demand'', indicating how long-term of a memory is needed to answer&#10;them, which should aid long-term memory performance evaluation. We validate our&#10;data in four small-scale experiments: one with human labelers, and three with&#10;existing language models. We show that our questions 1) adequately represent&#10;the source material 2) can be used to diagnose a model's memory capacity 3) are&#10;not trivial for modern language models even when the memory demand does not&#10;exceed those models' context lengths. Lastly, we provide our code which can be&#10;used to further expand the dataset with minimal human labor.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.12727" label="2004.12727">
        <attvalues>
          <attvalue for="0" value="Screenplay Summarization Using Latent Narrative Structure" />
          <attvalue for="1" value="  Most general-purpose extractive summarization models are trained on news&#10;articles, which are short and present all important information upfront. As a&#10;result, such models are biased on position and often perform a smart selection&#10;of sentences from the beginning of the document. When summarizing long&#10;narratives, which have complex structure and present information piecemeal,&#10;simple position heuristics are not sufficient. In this paper, we propose to&#10;explicitly incorporate the underlying structure of narratives into general&#10;unsupervised and supervised extractive summarization models. We formalize&#10;narrative structure in terms of key narrative events (turning points) and treat&#10;it as latent in order to summarize screenplays (i.e., extract an optimal&#10;sequence of scenes). Experimental results on the CSI corpus of TV screenplays,&#10;which we augment with scene-level summarization labels, show that latent&#10;turning points correlate with important aspects of a CSI episode and improve&#10;summarization performance over general extractive algorithms leading to more&#10;complete and diverse summaries.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07091" label="2104.07091">
        <attvalues>
          <attvalue for="0" value="SummScreen: A Dataset for Abstractive Screenplay Summarization" />
          <attvalue for="1" value="  We introduce SummScreen, a summarization dataset comprised of pairs of TV&#10;series transcripts and human written recaps. The dataset provides a challenging&#10;testbed for abstractive summarization for several reasons. Plot details are&#10;often expressed indirectly in character dialogues and may be scattered across&#10;the entirety of the transcript. These details must be found and integrated to&#10;form the succinct plot descriptions in the recaps. Also, TV scripts contain&#10;content that does not directly pertain to the central plot but rather serves to&#10;develop characters or provide comic relief. This information is rarely&#10;contained in recaps. Since characters are fundamental to TV series, we also&#10;propose two entity-centric evaluation metrics. Empirically, we characterize the&#10;dataset by evaluating several methods, including neural models and those based&#10;on nearest neighbors. An oracle extractive approach outperforms all benchmarked&#10;models according to automatic metrics, showing that the neural models are&#10;unable to fully exploit the input transcripts. Human evaluation and qualitative&#10;analysis reveal that our non-oracle models are competitive with their oracle&#10;counterparts in terms of generating faithful plot events and can benefit from&#10;better content selectors. Both oracle and non-oracle models generate unfaithful&#10;facts, suggesting future research directions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.03953" label="2303.03953">
        <attvalues>
          <attvalue for="0" value="ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use&#10;  Case of Automatic Genre Identification" />
          <attvalue for="1" value="  ChatGPT has shown strong capabilities in natural language generation tasks,&#10;which naturally leads researchers to explore where its abilities end. In this&#10;paper, we examine whether ChatGPT can be used for zero-shot text&#10;classification, more specifically, automatic genre identification. We compare&#10;ChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned on&#10;datasets, manually annotated with genres. The models are compared on test sets&#10;in two languages: English and Slovenian. Results show that ChatGPT outperforms&#10;the fine-tuned model when applied to the dataset which was not seen before by&#10;either of the models. Even when applied on Slovenian language as an&#10;under-resourced language, ChatGPT's performance is no worse than when applied&#10;to English. However, if the model is fully prompted in Slovenian, the&#10;performance drops significantly, showing the current limitations of ChatGPT&#10;usage on smaller languages. The presented results lead us to questioning&#10;whether this is the beginning of an end of laborious manual annotation&#10;campaigns even for smaller languages, such as Slovenian.&#10;" />
          <attvalue for="2" value="&#10;&#10;ChatGPT has shown strong capabilities as a dialogue system, providing clearer and more helpful answers than humans \cite{guo2023close}. It remains unclear whether its performance on text categorization tasks, more specifically on automatic genre identification, can be compared to the existing large language models (LLMs), such as the XLM-RoBERTa model \cite{conneau2020unsupervised}, fine-tuned to the task. Despite the fact that the ChatGPT model was made available to the public only a few months ago, some studies, analyzing the potential of the model for numerous natural language processing (NLP) tasks, have already been published. \cite{qin2023chatgpt} analyzed its zero-shot performance on reasoning tasks, natural language inference, dialogue, question answering, summarization, named-entity recognition and sentiment analysis. The results showed that ChatGPT was outperformed by the fine-tuned large language models at most tasks. It achieved significantly worse results in symbolic reasoning and named-entity recognition. Similarly, \cite{zhong2023can} compared ChatGPT with fine-tuned language models BERT \cite{DBLP:journals/corr/abs-1810-04805} and RoBERTa \cite{liu2019roberta} on the GLUE benchmark \cite{wang2018glue}, consisting of sentiment analysis, linguistic acceptability, paraphrase, textual similarity, natural language&#10;inference, and question answering. The overall results showed that ChatGPT performed comparably to the BERT model, while it was outperformed by the RoBERTa model. In contrast, when \cite{zhang2022would} analyzed ChatGPT's performance on stance detection, it achieved state-of-the-art (SOTA) results on this task. The ChatGPT model was also shown to achieve competitive performance in machine translation of high-resource languages, while it is lacking behind the SOTA models on low-resource languages based on the WMT22 \cite{kocmi-etal-2022-findings} dataset \cite{hendy2023good}.&#10;&#10;In addition to comparing the model to fine-tuned LLM models, some studies compared the model's performance with manual annotation. When \cite{huang2023chatgpt} examined the performance of ChatGPT on categorizing implicit hate speech and providing natural language explanations for the implicit hate speech detection, the results showed great potential of ChatGPT for this and similar tasks. The authors report that the results are promising, as the model correctly identified 80\% of implicit hateful tweets. What is more, its explanations were evaluated to be better than those, provided by humans. Based on this, the authors see a ``great potential of ChatGPT as a data annotation tool''. &#10;&#10;In this paper, we examine ChatGPT's performance on automatic genre identification, a text classification task, where manual annotation has been repeatedly shown to be very hard for humans \cite{egbert2015developing, zu2004genre, suchomel2020genre}. In addition to providing first insights into the performance of ChatGPT on this task, this is also one of the first studies of its performance on a language, other than English. We compare ChatGPT's zero-shot performance with the X-GENRE classifier -- a multilingual XLM-RoBERTa Transformer-based language model, fine-tuned on manually-annotated genre datasets. The models are compared on two test sets, English EN-GINCO dataset and Slovenian GINCO dataset \cite{kuzman-rupnik-ljubei:2022:LREC}, in three scenarios: English prompt with English text, English prompt with Slovenian text and Slovenian prompt with Slovenian text. The experiments show impressive performance of ChatGPT on this task, outperforming the fine-tuned LLM on English test set. In addition, despite Slovenian being an under-resourced language, ChatGPT's performance on this language is no worse than on English, provided that the prompt is in English instead of Slovenian. So much time and effort went into manual annotation of datasets in numerous languages to fine-tune the models for genre identification, which does not seem to be necessary anymore. Is this a point for the NLP community to stop and ask ourselves: ``Have large annotation campaigns became redundant? Can we use ChatGPT to annotate data for research purposes instead?''&#10;&#10;The paper is organized as follows. In Section \ref{sec:related}, we introduce the task of automatic genre identification. We present genre-annotated datasets in Section \ref{sec:dataset}, on which we test the ChatGPT model and the fine-tuned LLM, presented in Section \ref{sec:models}. We discuss the predictions of ChatGPT in Section \ref{sec:chatgpt-predictions} and compare it with the fine-tuned model in Section \ref{sec:comparison}. Finally, in Section \ref{sec:conclusion}, we conclude the paper with discussion of the main findings and suggestions for further work. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Multilingual Language Processing, Linguistics, Cognitive Science, Text Classification Models, Artificial Intelligence, Natural Language Processing, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="2201.03857" label="2201.03857">
        <attvalues>
          <attvalue for="0" value="The GINCO Training Dataset for Web Genre Identification of Documents Out&#10;  in the Wild" />
          <attvalue for="1" value="  This paper presents a new training dataset for automatic genre identification&#10;GINCO, which is based on 1,125 crawled Slovenian web documents that consist of&#10;650 thousand words. Each document was manually annotated for genre with a new&#10;annotation schema that builds upon existing schemata, having primarily clarity&#10;of labels and inter-annotator agreement in mind. The dataset consists of&#10;various challenges related to web-based data, such as machine translated&#10;content, encoding errors, multiple contents presented in one document etc.,&#10;enabling evaluation of classifiers in realistic conditions. The initial machine&#10;learning experiments on the dataset show that (1) pre-Transformer models are&#10;drastically less able to model the phenomena, with macro F1 metrics ranging&#10;around 0.22, while Transformer-based models achieve scores of around 0.58, and&#10;(2) multilingual Transformer models work as well on the task as the monolingual&#10;models that were previously proven to be superior to multilingual models on&#10;standard NLP tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.04416" label="2407.04416">
        <attvalues>
          <attvalue for="0" value="Sound-VECaps: Improving Audio Generation with Visual Enhanced Captions" />
          <attvalue for="1" value="  Generative models have shown significant achievements in audio generation&#10;tasks. However, existing models struggle with complex and detailed prompts,&#10;leading to potential performance degradation. We hypothesize that this problem&#10;stems from the simplicity and scarcity of the training data. This work aims to&#10;create a large-scale audio dataset with rich captions for improving audio&#10;generation models. We first develop an automated pipeline to generate detailed&#10;captions by transforming predicted visual captions, audio captions, and tagging&#10;labels into comprehensive descriptions using a Large Language Model (LLM). The&#10;resulting dataset, Sound-VECaps, comprises 1.66M high-quality audio-caption&#10;pairs with enriched details including audio event orders, occurred places and&#10;environment information. We then demonstrate that training the text-to-audio&#10;generation models with Sound-VECaps significantly improves the performance on&#10;complex prompts. Furthermore, we conduct ablation studies of the models on&#10;several downstream audio-language tasks, showing the potential of Sound-VECaps&#10;in advancing audio-text representation learning. Our dataset and models are&#10;available online.&#10;" />
          <attvalue for="2" value="&#10;Generative models have recently achieved substantial success for text-to-audio generation. In particular, the development of language models~\cite{clap,t5} and diffusion models~\cite{dalle3,stable_diffusion} have enabled the creation of powerful systems~\cite{audioldm2,tango} on generating high-fidelity audio clips. &#10;&#10;Despite their success in generating audio with simple captions, current models struggle with complex prompts containing detailed information, which referred to the challenge as ``prompt following''~\cite{dalle3}. A potential reason for this limitation is that existing audio-caption datasets often lack in quantity and quality~(detailed information) of the captions. In most of these datasets, each audio is matched with simple and short captions, typically, fewer than $10$ words. As a result, the captions in these datasets may not contain fine-grained information that could be useful for highly controllable audio generation. &#10;&#10;In addition, the simplicity of the caption often results in situations where the same caption corresponds to multiple audio files~(e.g., there are $2.5$K audio clips match with the caption `` Music is playing'' in WavCaps~\cite{wavcaps}), causing the system to avoid learning specific audio feature and lead to more instability in the generated outputs. A possible way to address this issue is to incorporate additional information, such as visual features, which have been shown to provide more detailed insights. One of the previous attempts is the Auto-ACD~\cite{autoacd}, where video features are used to improve the description of the event-occurring scene. However, Auto-ACD only takes the visual feature of the middle frame, and the caption has been designed to ignore the visual-only contents, losing more detailed information. &#10;&#10;In this paper, we aim to leverage external visual guidance to enhance the audio captions. With improved captions, we can provide better alignment between the prompt and the sound, thereby improving text-to-audio generation systems. Specifically, we propose new pipelines to construct a large-scale audio-language dataset with vision-enhanced captions. Our approach first involves collecting external visual information using state-of-the-art~(SoTA) image captioning models. These visual captions, combined with simple audio information, are then used to create new, enriched captions through Large Language Models (LLMs). By incorporating additional visual information, our method ensures the accuracy of audio details while enhancing the captions with comprehensive content, including temporal, spatial, and contextual elements related to the environment. Building on AudioSet~\cite{audioset}, we introduce Sound-VECaps, a large-scale dataset comprising over $1.66$M audio-caption pairs. &#10;&#10;Using Sound-VECaps as the training dataset, our experiments with the audio generation model, AudioLDM~\cite{audioldm}, show substantial improvements over baseline models. To evaluate the performance on complex and extended prompts, we propose a new benchmark for text-to-audio generation by constructing an enhanced AudioCaps~\cite{audiocaps} testing set~(same audio with better captions) named AudioCaps-Enhanced. Specifically, the AudioLDM-Large trained on Sound-VECaps achieves a Frechet Audio Distance~(FAD) score of $1.49$ on the AudioCaps. It further improves to a score of $1.06$ on AudioCaps-Enhanced, significantly outperforming current SoTA models. Moreover, we conduct experiments on Sound-VECaps across various audio-language tasks, demonstrating that systems trained on Sound-VECaps achieve SoTA performance in specific audio-domain tasks, such as audio retrieval. We also investigate the effectiveness of the visual-only content within the caption and the impact of these features during inference. In addition, an external version of Sound-VECaps that excludes all the visual-only information~(Sound-VECaps$_{\operatorname{A}}$) is also provided for different research purposes. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Acoustics, Computer Science, Audio Signal Processing, Linguistics, Audio Generation Models, Large Language Datasets, Text-to-Audio Synthesis" />
        </attvalues>
      </node>
      <node id="2308.05734" label="2308.05734">
        <attvalues>
          <attvalue for="0" value="AudioLDM 2: Learning Holistic Audio Generation with Self-supervised&#10;  Pretraining" />
          <attvalue for="1" value="  Although audio generation shares commonalities across different types of&#10;audio, such as speech, music, and sound effects, designing models for each type&#10;requires careful consideration of specific objectives and biases that can&#10;significantly differ from those of other types. To bring us closer to a unified&#10;perspective of audio generation, this paper proposes a framework that utilizes&#10;the same learning method for speech, music, and sound effect generation. Our&#10;framework introduces a general representation of audio, called &quot;language of&#10;audio&quot; (LOA). Any audio can be translated into LOA based on AudioMAE, a&#10;self-supervised pre-trained representation learning model. In the generation&#10;process, we translate any modalities into LOA by using a GPT-2 model, and we&#10;perform self-supervised audio generation learning with a latent diffusion model&#10;conditioned on LOA. The proposed framework naturally brings advantages such as&#10;in-context learning abilities and reusable self-supervised pretrained AudioMAE&#10;and latent diffusion models. Experiments on the major benchmarks of&#10;text-to-audio, text-to-music, and text-to-speech demonstrate state-of-the-art&#10;or competitive performance against previous approaches. Our code, pretrained&#10;model, and demo are available at https://audioldm.github.io/audioldm2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.02228" label="2106.02228">
        <attvalues>
          <attvalue for="0" value="Addressing Inquiries about History: An Efficient and Practical Framework&#10;  for Evaluating Open-domain Chatbot Consistency" />
          <attvalue for="1" value="  A good open-domain chatbot should avoid presenting contradictory responses&#10;about facts or opinions in a conversational session, known as its consistency&#10;capacity. However, evaluating the consistency capacity of a chatbot is still&#10;challenging. Employing human judges to interact with chatbots on purpose to&#10;check their capacities is costly and low-efficient, and difficult to get rid of&#10;subjective bias. In this paper, we propose the Addressing Inquiries about&#10;History (AIH), an efficient and practical framework for the consistency&#10;evaluation. At the conversation stage, AIH attempts to address appropriate&#10;inquiries about the dialogue history to induce the chatbot to redeclare the&#10;historical facts or opinions. We carry out the conversation between chatbots,&#10;which is more efficient than the human-bot interaction and can also alleviate&#10;the subjective bias. In this way, we manage to rapidly obtain a dialog session&#10;that contains responses with high contradiction possibilities. At the&#10;contradiction recognition stage, we can either employ human judges or a natural&#10;language inference (NLI) model to recognize whether the answers to the&#10;inquiries are contradictory with history. Finally, we are able to rank chatbots&#10;according to the contradiction statistics. Experiments on open-domain chatbots&#10;show that our approach can efficiently and reliably assess the consistency&#10;capacity of chatbots and achieve a high ranking correlation with the human&#10;evaluation. We release the framework and hope to help improve the consistency&#10;capacity of chatbots. \footnote{\url{https://github.com/ictnlp/AIH}}&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;There are various methods to evaluate the consistency of chatbots, containing automatic and human-based methods. The methods mainly fall into two dimensions: the static and interactive evaluation. &#10;&#10;" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, Linguistics, Cognitive Science, Human-Chatbot Interaction, Artificial Intelligence, Chatbot Consistency Evaluation, Natural Language Inference, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2205.10517" label="2205.10517">
        <attvalues>
          <attvalue for="0" value="Pre-training Data Quality and Quantity for a Low-Resource Language: New&#10;  Corpus and BERT Models for Maltese" />
          <attvalue for="1" value="  Multilingual language models such as mBERT have seen impressive cross-lingual&#10;transfer to a variety of languages, but many languages remain excluded from&#10;these models. In this paper, we analyse the effect of pre-training with&#10;monolingual data for a low-resource language that is not included in mBERT --&#10;Maltese -- with a range of pre-training set ups. We conduct evaluations with&#10;the newly pre-trained models on three morphosyntactic tasks -- dependency&#10;parsing, part-of-speech tagging, and named-entity recognition -- and one&#10;semantic classification task -- sentiment analysis. We also present a newly&#10;created corpus for Maltese, and determine the effect that the pre-training data&#10;size and domain have on the downstream performance. Our results show that using&#10;a mixture of pre-training domains is often superior to using Wikipedia text&#10;only. We also find that a fraction of this corpus is enough to make significant&#10;leaps in performance over Wikipedia-trained models. We pre-train and compare&#10;two models on the new corpus: a monolingual BERT model trained from scratch&#10;(BERTu), and a further pre-trained multilingual BERT (mBERTu). The models&#10;achieve state-of-the-art performance on these tasks, despite the new corpus&#10;being considerably smaller than typically used corpora for high-resourced&#10;languages. On average, BERTu outperforms or performs competitively with mBERTu,&#10;and the largest gains are observed for higher-level tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Language Models have become a core component in many Natural Language Processing (NLP) tasks.&#10;These models are typically pre-trained on unlabelled texts, and then further fine-tuned using labelled data relevant to the target task.&#10;Transformer-based \cite{NIPS2017_3f5ee243} contextual models such as BERT \cite{devlin-etal-2019-bert} have gained success since the fine-tuning step is relatively inexpensive, while attaining state-of-the-art results in various syntactic and semantic tasks.&#10;&#10;While the bulk of work with the BERT family of models focuses on English, there have been some monolingual models developed for other languages as well (\cite{martin-etal-2020-camembert, polignano2019alberto, antoun-etal-2020-arabert, BERTje, FinBERT, agerri-etal-2020-give}; inter alia).&#10;These monolingual models have been trained on large volumes of data, typically amounting to billions of tokens. &#10;In contrast, it is challenging to find publicly available corpora of this size for low-resource languages.&#10;The evaluation benchmarks for downstream tasks on these languages are also limited, and tend to be dominated by low-level structural tagging tasks.&#10;&#10;To counteract the lack of large volumes of monolingual corpora for low-resource languages, a number of multilingual models have been released, such as mBERT \cite{devlin-etal-2019-bert} and XLM-R \cite{conneau-etal-2020-unsupervised}.&#10;These multilingual models were pre-trained on more than one language at a time by combining corpora from different languages, usually sourced from Wikipedia.&#10;Several works have demonstrated the efficacy of these multilingual models, especially for languages without a language-specific model \cite{kondratyuk-straka-2019-75, wu-dredze-2019-beto}.&#10;Benchmark results have improved for many languages by leveraging cross-linguistic features learnt by these multilingual models \cite{conneau-etal-2020-unsupervised}.&#10;&#10;However, the gains with multilingual models may vary depending on the language being considered.&#10;The ``curse of multilinguality'' limits the language-specific features that these models can learn, since the limited model capacity has to be shared between multiple languages \cite{conneau-etal-2020-unsupervised}.&#10;Models such as mBERT use WordPiece tokenisation \cite{johnson-etal-2017-googles}, which splits words into various sub-tokens, thereby reducing the number of unknown tokens.&#10;However, the vocabulary representations for multilingual models tend to be sub-optimal for specific languages, because words tend to be split into a higher number of sub-tokens \cite{rust-etal-2021-good}.&#10;Moreover, these models may still be biased in favour of over-representing sub-tokens common to a certain subset of languages over others.&#10;Due to the data imbalance across languages, lower-resourced languages tend to be disadvantaged, as there is relatively less pre-training data available compared to the other languages considered in the multilingual model \cite{wu-dredze-2020-languages}. &#10;&#10;Apart from the tension between languages in a multilingual model, other factors are at play as well.&#10;Most prominently, many languages are never seen by these multilingual models \cite{muller-etal-2021-unseen}, since these are typically trained on the largest-available corpora (e.g. mBERT was pre-trained on the 104 languages with the greatest Wikipedia presence).&#10;Such criteria exclude many of the world's languages, including Maltese, the focus of this paper.&#10;This issue is exacerbated even further when the language uses a script which is either different to its closely related languages \cite{muller-etal-2021-unseen}, or which is never seen during pre-training, thereby encoding most of the input with out-of-vocabulary tokens \cite{pfeiffer-etal-2021-unks}.&#10;In fact, \cite{muller-etal-2021-unseen} show that the language transfer capability of a multilingual model to an unseen language is dependent on the degree to which the target language is related to languages already included in the multilingual model.&#10;&#10;In this work we focus on the Maltese language, an official EU language spoken primarily in Malta and in some small communities around the world \cite{Brincat2011}. &#10;It is the only Semitic language written exclusively with a Latin script, containing a few additional characters with diacritic marks (ċ, ġ, \hwithstroke, ż).&#10;The language also has strong influences from Romance languages such as Italian, as well as English.&#10;The Semitic influence is largely exhibited in the grammatical structure through complex morphological characteristics, whilst the non-Semitic aspect is predominantly observed in its vocabulary, with extensive lexical borrowing from Italian and English.&#10;&#10;In the context of NLP, Maltese is a low-resource language~\cite{Rosner2022} and is not part of the languages covered by either mBERT or XLM-R.&#10;\cite{muller-etal-2021-unseen} find that mBERT underperforms non-contextual baselines on Maltese, but benefits when pre-trained further on raw Maltese data.&#10;Similarly, \cite{chau-etal-2020-parsing} further pre-train mBERT but impute the 99 unused tokens present in the model with language specific tokens, yielding better results. This confirms previous findings by \cite{wang-etal-2020-extending}, who also extend mBERT's vocabulary to accommodate unseen languages, but do so by extending the vocabulary and model dimensionality, hence increasing its footprint.&#10;&#10;Motivated by the limitations of existing multilingual models and the deficiency of publicly available corpora for Maltese, we set out to pre-train a new monolingual language model for Maltese and compare it to the alternative strategy of further pre-training an existing multilingual model.&#10;We study, in particular, the impact that the pre-training data size and domain has on the performance in downstream tasks.&#10;The main contributions of this work are as follows:&#10;\begin{enumerate}&#10; \item We develop a new corpus of Maltese text.&#10; \item Using this new data, language models for Maltese are pre-trained.&#10; \item We compare the newly pre-trained models and find that both models improve the state-of-the-art on three structural tagging tasks -- dependency parsing, part-of-speech tagging, and named-entity recognition -- and one semantic classification task -- sentiment analysis.&#10; \item We demonstrate that in a low-resource setting, pre-training using text from varied domains is often superior to solely using Wikipedia, and that matching the domain to target task is beneficial when this is available.&#10; \item We also provide an analysis on the effects of the pre-training size, shedding new light on how much pre-training data is needed to attain significant improvements in performance.&#10;\end{enumerate}&#10;&#10;We make this new corpus, the newly pre-trained language models, and the code publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Multilingual Language Models, Information Science, Low-Resource Language Processing, Pre-Training Data Evaluation" />
        </attvalues>
      </node>
      <node id="2310.15075" label="2310.15075">
        <attvalues>
          <attvalue for="0" value="TableQAKit: A Comprehensive and Practical Toolkit for Table-based&#10;  Question Answering" />
          <attvalue for="1" value="  Table-based question answering (TableQA) is an important task in natural&#10;language processing, which requires comprehending tables and employing various&#10;reasoning ways to answer the questions. This paper introduces TableQAKit, the&#10;first comprehensive toolkit designed specifically for TableQA. The toolkit&#10;designs a unified platform that includes plentiful TableQA datasets and&#10;integrates popular methods of this task as well as large language models&#10;(LLMs). Users can add their datasets and methods according to the friendly&#10;interface. Also, pleasantly surprised using the modules in this toolkit&#10;achieves new SOTA on some datasets. Finally, \tableqakit{} also provides an&#10;LLM-based TableQA Benchmark for evaluating the role of LLMs in TableQA.&#10;TableQAKit is open-source with an interactive interface that includes visual&#10;operations, and comprehensive data for ease of use.&#10;" />
          <attvalue for="2" value="&#10;Question answering systems devote to answering various questions with the evidence located in the structured knowledge base~(e.g., Table~\cite{pasupat2015compositional}, Knowledge Graph~\cite{lan2021survey}), unstructured texts~\cite{rajpurkar2016squad} or images~(e.g., VQA~\cite{antol2015vqa}). Our work focuses on table-based question answering~(TableQA), which typically relies on the evidence from the given tables to answer the question~(a small number of datasets require auxiliary evidence in text~\cite{chen2020hybridqa, zhu2021tat} or images~\cite{talmor2020multimodalqa}).&#10;&#10;The given tables come in various types, such as financial numeric tables, Wikipedia-style tables, and database tables (Examples of datasets are shown in Figure~\ref{fig_dataset_example}). TableQA mainly contains three typical subtasks: Spreadsheet QA \cite{zhu2021tat}, Encyclopedia QA \cite{chen2020hybridqa}, and Structured QA. Accordingly, various reasoning skills are addressed, such as numerical reasoning, multi-hop factual reasoning, and structured query-based questions.&#10;&#10;However, TableQA tasks exhibit drastically different data and methods formats, significantly hindering TableQA research's convenience. It is necessary to develop a unified data interface and task-solving framework. Therefore, we developed the toolkit to fill the gap in this research field.&#10;&#10;Moreover, with the development of large language model (LLM) ~\cite{zhao2023survey}, LLM-based methods~\cite{dong2022survey, touvron2023llama} have gradually become new paradigms in addressing NLP tasks. Thus, the methods integrated into \tableqakit also include the LLM-prompting and fine-tuning methods. It is the first toolkit that uses LLM for structured QA tasks.&#10;&#10;We have also proposed a challenging LLM TableQA benchmark——\TableQAEval, caused by two reasons: (1) Existing TableQA benchmarks usually focus on a single type of table and question, which is out of the game in the era of LLMs. (2) From the perspective of LLM evaluation, there is currently a lack of evaluation benchmark for LLM long-context capabilities, and it happens that tabular data is an easy-to-collect and knowledge-intensive data source.&#10;&#10;\begin{itemize}&#10;\item Unified and comprehensive toolkit.&#10;\tableqakit is the first unified toolkit to support almost all TableQA scenarios. With this toolkit, different datasets are unified under a single interface, allowing users to work seamlessly with existing datasets and their tables.&#10;&#10;\item LLM-supported toolkit. &#10;\tableqakit includes LLM-based methods that enable powerful performance. It is the first toolkit to utilize LLM-based methods for structured knowledge tasks and has achieved new SOTA results on several datasets.&#10;&#10;\item Benchmark. \tableqakit is equipped with \TableQAEval, the first multi-type long-context benchmark for LLM-based TableQA.&#10;&#10;\item Visualization. &#10;We provide visual web pages to help users visualize and interact with data and support multi-type tables and multi-modal data. Users can also upload spreadsheets and interact with them through QA interactions.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Artificial Intelligence, Information Science, Table-based Question Answering, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1508.00305" label="1508.00305">
        <attvalues>
          <attvalue for="0" value="Compositional Semantic Parsing on Semi-Structured Tables" />
          <attvalue for="1" value="  Two important aspects of semantic parsing for question answering are the&#10;breadth of the knowledge source and the depth of logical compositionality.&#10;While existing work trades off one aspect for another, this paper&#10;simultaneously makes progress on both fronts through a new task: answering&#10;complex questions on semi-structured tables using question-answer pairs as&#10;supervision. The central challenge arises from two compounding factors: the&#10;broader domain results in an open-ended set of relations, and the deeper&#10;compositionality results in a combinatorial explosion in the space of logical&#10;forms. We propose a logical-form driven parsing algorithm guided by strong&#10;typing constraints and show that it obtains significant improvements over&#10;natural baselines. For evaluation, we created a new dataset of 22,033 complex&#10;questions on Wikipedia tables, which is made publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.11644" label="2105.11644">
        <attvalues>
          <attvalue for="0" value="A Survey on Complex Knowledge Base Question Answering: Methods,&#10;  Challenges and Solutions" />
          <attvalue for="1" value="  Knowledge base question answering (KBQA) aims to answer a question over a&#10;knowledge base (KB). Recently, a large number of studies focus on semantically&#10;or syntactically complicated questions. In this paper, we elaborately summarize&#10;the typical challenges and solutions for complex KBQA. We begin with&#10;introducing the background about the KBQA task. Next, we present the two&#10;mainstream categories of methods for complex KBQA, namely semantic&#10;parsing-based (SP-based) methods and information retrieval-based (IR-based)&#10;methods. We then review the advanced methods comprehensively from the&#10;perspective of the two categories. Specifically, we explicate their solutions&#10;to the typical challenges. Finally, we conclude and discuss some promising&#10;directions for future research.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.07347" label="2004.07347">
        <attvalues>
          <attvalue for="0" value="HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and&#10;  Textual Data" />
          <attvalue for="1" value="  Existing question answering datasets focus on dealing with homogeneous&#10;information, based either only on text or KB/Table information alone. However,&#10;as human knowledge is distributed over heterogeneous forms, using homogeneous&#10;information alone might lead to severe coverage problems. To fill in the gap,&#10;we present HybridQA https://github.com/wenhuchen/HybridQA, a new large-scale&#10;question-answering dataset that requires reasoning on heterogeneous&#10;information. Each question is aligned with a Wikipedia table and multiple&#10;free-form corpora linked with the entities in the table. The questions are&#10;designed to aggregate both tabular information and text information, i.e., lack&#10;of either form would render the question unanswerable. We test with three&#10;different models: 1) a table-only model. 2) text-only model. 3) a hybrid model&#10;that combines heterogeneous information to find the answer. The experimental&#10;results show that the EM scores obtained by two baselines are below 20\%, while&#10;the hybrid model can achieve an EM over 40\%. This gap suggests the necessity&#10;to aggregate heterogeneous information in HybridQA. However, the hybrid model's&#10;score is still far behind human performance. Hence, HybridQA can serve as a&#10;challenging benchmark to study question answering with heterogeneous&#10;information.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.13895" label="2005.13895">
        <attvalues>
          <attvalue for="0" value="When Can Self-Attention Be Replaced by Feed Forward Layers?" />
          <attvalue for="1" value="  Recently, self-attention models such as Transformers have given competitive&#10;results compared to recurrent neural network systems in speech recognition. The&#10;key factor for the outstanding performance of self-attention models is their&#10;ability to capture temporal relationships without being limited by the distance&#10;between two related events. However, we note that the range of the learned&#10;context progressively increases from the lower to upper self-attention layers,&#10;whilst acoustic events often happen within short time spans in a left-to-right&#10;order. This leads to a question: for speech recognition, is a global view of&#10;the entire sequence still important for the upper self-attention layers in the&#10;encoder of Transformers? To investigate this, we replace these self-attention&#10;layers with feed forward layers. In our speech recognition experiments (Wall&#10;Street Journal and Switchboard), we indeed observe an interesting result:&#10;replacing the upper self-attention layers in the encoder with feed forward&#10;layers leads to no performance drop, and even minor gains. Our experiments&#10;offer insights to how self-attention layers process the speech signal, leading&#10;to the conclusion that the lower self-attention layers of the encoder encode a&#10;sufficiently wide range of inputs, hence learning further contextual&#10;information in the upper layers is unnecessary.&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-attention networks (SANs) has recently become a popular research topic in the speech recognition community \cite{Dong2018Speech-transformer, zhou2018syllable, pham2019veryDeep, povey2018time, zeyer2019comparison}. Previous studies showed SANs can yield superior speech recognition results compared to recurrent neural networks (RNNs) \cite{wang2019transformerHybrid, karita2019comparative}. &#10;&#10;RNNs are conventional models to model sequential data. However, due to gradient vanishing, it is difficult for RNNs to model long-range dependencies \cite{bengio1994learning}. Although gated structures, such as Long Short-Term Memory (LSTM)~\cite{hochreiter1997LSTM} and Gated Recurrent Unit (GRU)~\cite{chung2014GRU} are proposed to alleviate this problem, capturing temporal relationships across a wide time span remains challenging for these models. In SANs, self-attention layers encode contextual information through attention mechanisms \cite{bahdanau2015neural,luong2015effective,vaswani2017attention}. With the attention mechanism, when learning the hidden representation for each time step of a sequence, a self-attention layer has a global view of the entire sequence and it can thus capture temporal relationships without the limitation of range. This is believed to be a key factor for the success of SANs \cite{vaswani2017attention}. &#10;&#10;In this paper we study Transformers~\cite{vaswani2017attention}, end-to-end SAN-based models with two components: an encoder and a decoder. The encoder uses self-attention layers to encode input sequences. At decoding time step $t$, the decoder generates the current output by attending to the encoded input sequence and to the outputs generated before time $t$. For attention-based RNN end-to-end models \cite{luong2015effective, bahdanau2015neural}, the RNN encoder encodes the input sequence. The RNN decoder interacts with the encoded input sequence through an attention layer to produce outputs. &#10;&#10;Previous work on attention-based RNN end-to-end models has shown that for speech recognition, since acoustic events usually happen in a left-to-right order within small time spans, restricting the attention to be monotonic along the time axis improves the model's performance \cite{tjandra2017local, kim2017joint, zhang2019windowed}. This seemingly leads to a contradiction to the reason of SANs' success: if the global view provided by the attention module of self-attention layers is beneficial, then why does forcing the attention mechanism to focus on local information result in performance gains for RNN end-to-end models?&#10;&#10;To investigate this, we explore replacing the upper (further from the input) &#10;self-attention layers of the Transformer's encoder with feed forward layers. We ran extensive experiments on a read speech corpus Wall Street Journal (WSJ) \cite{paul1992wsj} and a conversational telephone speech corpus Switchboard (SWBD) \cite{godfrey1992switchboard}. We found that replacing the upper self-attention layers with feed forward layers does not yield higher error rates -- it even leads to improved accuracies. Since a feed forward layer can be viewed as a pure ``monotonic left-to-right diagonal attention'', this observation does not contradict the previous studies on RNN-based end-to-end models which restrict the attention to be diagonal. Thus, it indicates the inputs to the upper layers of the Transformer encoder have encoded enough contextual information and learning further temporal relationships through self-attention is not helpful. These experiments also do not nullify the benefits of the self-attention layers -- the range of learned context is increased from bottom to up through the self-attention layers and it is important for the lower layers to well encode the context information. Only when the lower layers have captured sufficient contextual information, the attention mechanism becomes redundant for the upper layers. &#10;&#10;It should be noticed that for the attention-based RNN models, the attention mechanism interacts with both the decoder and the encoder. Since an output unit (e.g. a character) is often related to a small time span of acoustic features, the attention needs to attend a small window of the elements in the encoded input sequence in a left-to-right order. In this work we study the self-attention encoder which learns the hidden representation for each time step of the input sequence. Thus, feed forward layers, which can be viewed as ``left-to-right attention without looking at the context'', are sufficient in learning further abstract representation for the frame in the current time step when temporal relationships among input frames are well captured.&#10;&#10;Our observations also make practical contributions. Compared to self-attention layers, feed forward layers have a reduced number of parameters. Furthermore, without parallel computation, the time complexity for a self-attention layer is $o(n^2)$ where $n$ is the length of the input sequence. Replacing self-attention layers with feed forward layers also reduces the time complexity. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Self-Attention Mechanisms, Linguistics, Neural Network Models, Artificial Intelligence, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2308.00081" label="2308.00081">
        <attvalues>
          <attvalue for="0" value="Towards Semantically Enriched Embeddings for Knowledge Graph Completion" />
          <attvalue for="1" value="  Embedding based Knowledge Graph (KG) Completion has gained much attention&#10;over the past few years. Most of the current algorithms consider a KG as a&#10;multidirectional labeled graph and lack the ability to capture the semantics&#10;underlying the schematic information. In a separate development, a vast amount&#10;of information has been captured within the Large Language Models (LLMs) which&#10;has revolutionized the field of Artificial Intelligence. KGs could benefit from&#10;these LLMs and vice versa. This vision paper discusses the existing algorithms&#10;for KG completion based on the variations for generating KG embeddings. It&#10;starts with discussing various KG completion algorithms such as transductive&#10;and inductive link prediction and entity type prediction algorithms. It then&#10;moves on to the algorithms utilizing type information within the KGs, LLMs, and&#10;finally to algorithms capturing the semantics represented in different&#10;description logic axioms. We conclude the paper with a critical reflection on&#10;the current state of work in the community and give recommendations for future&#10;directions.&#10;" />
          <attvalue for="2" value="&#10;Knowledge Graphs (KGs) have recently gained attention due to their applicability to diverse fields of research, ranging from knowledge management, representation, and reasoning to learning representations over KGs. &#10;KGs represent knowledge in the form of relations between entities structured as (head, relation, tail) triples, referred to as facts, along with schematic information in the form of ontologies. KGs have been used for various downstream tasks such as web search, recommender systems, and question answering. &#10;&#10;KGs, however, suffer from incompleteness because of manual or automated generation. Manual generation leads to limited knowledge represented by the curator and contains curator bias~\cite{ntoutsi2020bias}, while automated generation may lead to erroneous or missing information. &#10;KG completion in particular includes the tasks of (i) triple classification, i.e., deciding if the triple is true or not, (ii) Link Prediction (LP) to complete the head, tail, or relation in a triple, and (iii) entity classification, also known as entity type prediction. &#10;To perform KG completion, various rule-based as well as embeddings based models have been proposed. &#10;These algorithms are computationally expensive and are transductive: they only perform predictions based on triple information involving known entities and relations. This is not readily usable when the inference has to be performed on unseen entities and relations. &#10;To this end, inductive KG completion allows for predicting triples that involve unseen entities and relations. &#10;These transductive and inductive LP algorithms are mostly based on factual information contained in KGs. Various studies leveraging language models as an external source of knowledge have been proposed for KG completion. These algorithms lag behind in terms of performance w.r.t. KG embedding based methods because of ranking-based metrics such as Hits@k since existing KG embedding-based algorithms operate under the Closed World Assumption. &#10;This led to the need for human evaluation since Large Language Models (LLMs) contain more general knowledge and may generate correct answers that are different from what is expected by the ground truth with the highest score. Apart from the factual information (i.e., Assertion Box, ABox), another source of information is the ontological information (i.e., Terminology Box, TBox) contained in the KG. The current methods almost completely ignore this TBox information. &#10;To rectify this situation various attempts have been made to include type hierarchies and ontological information with different expressivity levels such as $\mathcal{EL}^{++}$, $\mathcal{ALC}$, etc. In some cases, additional representational capabilities are utilized to capture this information such as box embeddings.&#10;&#10;Related Work. &#10;Several studies have surveyed the state of the art (SoTA) in KG completion. &#10;The work by Paulheim~\cite{Paulheim17} provides a survey of the articles related to KG refinement including various classical and rule-based approaches for KG completion, yet KG embeddings are not discussed. &#10;Other surveys specifically focus on KG embedding-based methods for KG completion. &#10;Wang et al.~\cite{WangMWG17} organises the algorithms for embedding-based KG completion according to their scoring functions such as translational models, semantic matching models, etc. However, this survey does not discuss the methods proposed for KG completion using multimodal information related to an entity or relation such as images, text, and numeric literals. &#10;These aspects are targeted in the survey by Gesese et al.~\cite{DBLP:journals/semweb/GeseseBAS21}, which categorizes these methods based on their scoring function (inspired by the work by Wang et al.~\cite{WangMWG17}) as well as based on multiple modalities. The survey shows theoretical and experimental comparisons of the existing approaches. Still, none of the aforementioned works cover the aspects of ontological statements and semantics in KG embeddings. &#10;Recent works~\cite{pan_et_al:TGDK.1.1.2,pan2024unifying} discuss the interplay of KGs and LLMs. &#10;The study by S. Pan et al.~\cite{pan2024unifying} describes methods that integrate LLMs for KG embeddings and KG completion, yet the role of schematic information is not considered in the anlaysis. &#10;Complementary, J. Z. Pan et al.~\cite{pan_et_al:TGDK.1.1.2} describe how LLMs can be used to complete triples and perform different ontology engineering tasks (e.g., completion or refinement) in KGs. &#10;In contrast to these works, our work focuses on KG embeddings and how LLMs can be used as external sources to improve the performance of KG completion tasks.&#10;In summary, as compared to these existing studies, the current article targets the semantic aspects of knowledge graph embeddings by summarizing and discussing the approaches that have been proposed so far for leveraging semantics provided in the KG.&#10;&#10;Contributions. &#10;This position paper provides an overview of the evolution of methodologies proposed for KG completion, starting from the embedding-based algorithms and LLM-based approaches to the various categories of algorithms proposed for incorporating schematic information within KG embeddings for performing different kinds of completion tasks. &#10;It further discusses the limitations of the approaches in each of these categories and concludes with critical reflections and future research directions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Knowledge Graph Completion, Linguistics, Data Science, Artificial Intelligence, Mathematics" />
        </attvalues>
      </node>
      <node id="1910.12507" label="1910.12507">
        <attvalues>
          <attvalue for="0" value="A Survey on Knowledge Graph Embeddings with Literals: Which model links&#10;  better Literal-ly?" />
          <attvalue for="1" value="  Knowledge Graphs (KGs) are composed of structured information about a&#10;particular domain in the form of entities and relations. In addition to the&#10;structured information KGs help in facilitating interconnectivity and&#10;interoperability between different resources represented in the Linked Data&#10;Cloud. KGs have been used in a variety of applications such as entity linking,&#10;question answering, recommender systems, etc. However, KG applications suffer&#10;from high computational and storage costs. Hence, there arises the necessity&#10;for a representation able to map the high dimensional KGs into low dimensional&#10;spaces, i.e., embedding space, preserving structural as well as relational&#10;information. This paper conducts a survey of KG embedding models which not only&#10;consider the structured information contained in the form of entities and&#10;relations in a KG but also the unstructured information represented as literals&#10;such as text, numerical values, images, etc. Along with a theoretical analysis&#10;and comparison of the methods proposed so far for generating KG embeddings with&#10;literals, an empirical evaluation of the different methods under identical&#10;settings has been performed for the general task of link prediction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.09323" label="1812.09323">
        <attvalues>
          <attvalue for="0" value="Unsupervised Speech Recognition via Segmental Empirical Output&#10;  Distribution Matching" />
          <attvalue for="1" value="  We consider the problem of training speech recognition systems without using&#10;any labeled data, under the assumption that the learner can only access to the&#10;input utterances and a phoneme language model estimated from a non-overlapping&#10;corpus. We propose a fully unsupervised learning algorithm that alternates&#10;between solving two sub-problems: (i) learn a phoneme classifier for a given&#10;set of phoneme segmentation boundaries, and (ii) refining the phoneme&#10;boundaries based on a given classifier. To solve the first sub-problem, we&#10;introduce a novel unsupervised cost function named Segmental Empirical Output&#10;Distribution Matching, which generalizes the work in (Liu et al., 2017) to&#10;segmental structures. For the second sub-problem, we develop an approximate MAP&#10;approach to refining the boundaries obtained from Wang et al. (2017).&#10;Experimental results on TIMIT dataset demonstrate the success of this fully&#10;unsupervised phoneme recognition system, which achieves a phone error rate&#10;(PER) of 41.6%. Although it is still far away from the state-of-the-art&#10;supervised systems, we show that with oracle boundaries and matching language&#10;model, the PER could be improved to 32.5%.This performance approaches the&#10;supervised system of the same model architecture, demonstrating the great&#10;potential of the proposed method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.00316" label="1804.00316">
        <attvalues>
          <attvalue for="0" value="Completely Unsupervised Phoneme Recognition by Adversarially Learning&#10;  Mapping Relationships from Audio Embeddings" />
          <attvalue for="1" value="  Unsupervised discovery of acoustic tokens from audio corpora without&#10;annotation and learning vector representations for these tokens have been&#10;widely studied. Although these techniques have been shown successful in some&#10;applications such as query-by-example Spoken Term Detection (STD), the lack of&#10;mapping relationships between these discovered tokens and real phonemes have&#10;limited the down-stream applications. This paper represents probably the first&#10;attempt towards the goal of completely unsupervised phoneme recognition, or&#10;mapping audio signals to phoneme sequences without phoneme-labeled audio data.&#10;The basic idea is to cluster the embedded acoustic tokens and learn the mapping&#10;between the cluster sequences and the unknown phoneme sequences with a&#10;Generative Adversarial Network (GAN). An unsupervised phoneme recognition&#10;accuracy of 36% was achieved in the preliminary experiments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.04100" label="1904.04100">
        <attvalues>
          <attvalue for="0" value="Completely Unsupervised Speech Recognition By A Generative Adversarial&#10;  Network Harmonized With Iteratively Refined Hidden Markov Models" />
          <attvalue for="1" value="  Producing a large annotated speech corpus for training ASR systems remains&#10;difficult for more than 95% of languages all over the world which are&#10;low-resourced, but collecting a relatively big unlabeled data set for such&#10;languages is more achievable. This is why some initial effort have been&#10;reported on completely unsupervised speech recognition learned from unlabeled&#10;data only, although with relatively high error rates. In this paper, we develop&#10;a Generative Adversarial Network (GAN) to achieve this purpose, in which a&#10;Generator and a Discriminator learn from each other iteratively to improve the&#10;performance. We further use a set of Hidden Markov Models (HMMs) iteratively&#10;refined from the machine generated labels to work in harmony with the GAN. The&#10;initial experiments on TIMIT data set achieve an phone error rate of 33.1%,&#10;which is 8.5% lower than the previous state-of-the-art.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.13326" label="1905.13326">
        <attvalues>
          <attvalue for="0" value="Grammar-based Neural Text-to-SQL Generation" />
          <attvalue for="1" value="  The sequence-to-sequence paradigm employed by neural text-to-SQL models&#10;typically performs token-level decoding and does not consider generating SQL&#10;hierarchically from a grammar. Grammar-based decoding has shown significant&#10;improvements for other semantic parsing tasks, but SQL and other general&#10;programming languages have complexities not present in logical formalisms that&#10;make writing hierarchical grammars difficult. We introduce techniques to handle&#10;these complexities, showing how to construct a schema-dependent grammar with&#10;minimal over-generation. We analyze these techniques on ATIS and Spider, two&#10;challenging text-to-SQL datasets, demonstrating that they yield 14--18\%&#10;relative reductions in error.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.12412" label="2010.12412">
        <attvalues>
          <attvalue for="0" value="SmBoP: Semi-autoregressive Bottom-up Semantic Parsing" />
          <attvalue for="1" value="  The de-facto standard decoding method for semantic parsing in recent years&#10;has been to autoregressively decode the abstract syntax tree of the target&#10;program using a top-down depth-first traversal. In this work, we propose an&#10;alternative approach: a Semi-autoregressive Bottom-up Parser (SmBoP) that&#10;constructs at decoding step $t$ the top-$K$ sub-trees of height $\leq t$. Our&#10;parser enjoys several benefits compared to top-down autoregressive parsing.&#10;From an efficiency perspective, bottom-up parsing allows to decode all&#10;sub-trees of a certain height in parallel, leading to logarithmic runtime&#10;complexity rather than linear. From a modeling perspective, a bottom-up parser&#10;learns representations for meaningful semantic sub-programs at each step,&#10;rather than for semantically-vacuous partial trees. We apply SmBoP on Spider, a&#10;challenging zero-shot semantic parsing benchmark, and show that SmBoP leads to&#10;a 2.2x speed-up in decoding time and a $\sim$5x speed-up in training time,&#10;compared to a semantic parser that uses autoregressive decoding. SmBoP obtains&#10;71.1 denotation accuracy on Spider, establishing a new state-of-the-art, and&#10;69.5 exact match, comparable to the 69.6 exact match of the autoregressive&#10;RAT-SQL+GraPPa.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.12627" label="2012.12627">
        <attvalues>
          <attvalue for="0" value="Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic&#10;  Parsing" />
          <attvalue for="1" value="  We present BRIDGE, a powerful sequential architecture for modeling&#10;dependencies between natural language questions and relational databases in&#10;cross-DB semantic parsing. BRIDGE represents the question and DB schema in a&#10;tagged sequence where a subset of the fields are augmented with cell values&#10;mentioned in the question. The hybrid sequence is encoded by BERT with minimal&#10;subsequent layers and the text-DB contextualization is realized via the&#10;fine-tuned deep attention in BERT. Combined with a pointer-generator decoder&#10;with schema-consistency driven search space pruning, BRIDGE attained&#10;state-of-the-art performance on popular cross-DB text-to-SQL benchmarks, Spider&#10;(71.1\% dev, 67.5\% test with ensemble model) and WikiSQL (92.6\% dev, 91.9\%&#10;test). Our analysis shows that BRIDGE effectively captures the desired&#10;cross-modal dependencies and has the potential to generalize to more text-DB&#10;related tasks. Our implementation is available at&#10;\url{https://github.com/salesforce/TabularSemanticParsing}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.05378" label="1909.05378">
        <attvalues>
          <attvalue for="0" value="CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain&#10;  Natural Language Interfaces to Databases" />
          <attvalue for="1" value="  We present CoSQL, a corpus for building cross-domain, general-purpose&#10;database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+&#10;annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k&#10;dialogues querying 200 complex DBs spanning 138 domains. Each dialogue&#10;simulates a real-world DB query scenario with a crowd worker as a user&#10;exploring the DB and a SQL expert retrieving answers with SQL, clarifying&#10;ambiguous questions, or otherwise informing of unanswerable questions. When&#10;user questions are answerable by SQL, the expert describes the SQL and&#10;execution results to the user, hence maintaining a natural interaction flow.&#10;CoSQL introduces new challenges compared to existing task-oriented dialogue&#10;datasets:(1) the dialogue states are grounded in SQL, a domain-independent&#10;executable representation, instead of domain-specific slot-value pairs, and (2)&#10;because testing is done on unseen databases, success requires generalizing to&#10;new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking,&#10;response generation from query results, and user dialogue act prediction. We&#10;evaluate a set of strong baselines for each task and show that CoSQL presents&#10;significant challenges for future research. The dataset, baselines, and&#10;leaderboard will be released at https://yale-lily.github.io/cosql.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.04984" label="2009.04984">
        <attvalues>
          <attvalue for="0" value="Dialogue-adaptive Language Model Pre-training From Quality Estimation" />
          <attvalue for="1" value="  Pre-trained language models (PrLMs) have achieved great success on a wide&#10;range of natural language processing tasks by virtue of the universal language&#10;representation ability obtained by self-supervised learning on a large corpus.&#10;These models are pre-trained on standard plain texts with general language&#10;model (LM) training objectives, which would be insufficient to model&#10;dialogue-exclusive attributes like specificity and informativeness reflected in&#10;these tasks that are not explicitly captured by the pre-trained universal&#10;language representations. In this work, we propose dialogue-adaptive&#10;pre-training objectives (DAPO) derived from quality estimation to simulate&#10;dialogue-specific features, namely coherence, specificity, and informativeness.&#10;As the foundation for model pre-training, we synthesize a new dialogue corpus&#10;and build our training set with two unsupervised methods: 1) coherence-oriented&#10;context corruption, including utterance ordering, insertion, and replacement,&#10;to help the model capture the coherence inside the dialogue contexts; and 2)&#10;specificity-oriented automatic rescoring, which encourages the model to measure&#10;the quality of the synthesized data for dialogue-adaptive pre-training by&#10;considering specificity and informativeness. Experimental results on widely&#10;used open-domain response selection and quality estimation benchmarks show that&#10;DAPO significantly improves the baseline models and achieves state-of-the-art&#10;performance on the MuTual leaderboard, verifying the effectiveness of&#10;estimating quality evaluation factors into pre-training.&#10;" />
          <attvalue for="2" value="&#10;&#09;Pre-trained language models (PrLMs) have achieved impressive performance in a series of natural language processing tasks. Some prominent examples of PrLMs are BERT \cite{devlin2019bert}, GPT \cite{radford2018improving}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ERNIE \cite{ernie2019sun, ernie202019sun}, ALBERT \cite{lan2019albert} and ELECTRA \cite{electra2020Clark}. {They can also be used in more expansive fields like Bioinformatics \cite{le2021deep,le2021transformer}.}&#10;&#09;The PrLMs are commonly employed through a pre-training then fine-tuning paradigm: the models are first trained on large-scale unlabeled task-independent corpora with general training objectives, like masked language modeling (MLM) \cite{ClozePA1953Taylor} or next sentence prediction (NSP) \cite{devlin2019bert}, to learn universal language representations; then, the trained models are fine-tuned on the smaller datasets of downstream tasks by leveraging extra task-specific modules for adaption.&#10;&#09;However, the learned universal language representations for dialogue tasks would not sufficiently and accurately cover the dialogue-aware features because the literary style of dialogues and plain texts varies dramatically. More specifically, as opposed to general plain texts, dialogue involves multiple speakers, intentions, topics; thus, the utterances are full of transitions. As a result, directly fine-tuning PrLMs on the dialogue datasets would be sub-optimal to model dialogues that contain exclusive attributes.&#10;&#10;To help PrLMs adapt to dialogue-related tasks, recent studies have investigated further dialogue-adaptive pre-training on PrLMs before fine-tuning them on dialogue tasks. \cite{DialoGPTLG2020Zhang} pre-trained GPT further with a conventional Language Model (LM) objective on a large dialogue corpus, Reddit, and get DialoGPT for response generation tasks; \cite{zhang2021kkt} and \cite{AnED2019Whang} pre-trained BERT with Mask Language Model (MLM) objective on the target datasets before fine-tuning on response selection tasks. Besides merely pre-training on dialogue datasets with general objectives, some &#10;&#09;studies propose particular auxiliary tasks or objectives for dialogue-adaptive pre-training. Notably, \cite{wu2020tod} proposed a response contrastive loss to match the context with the corresponding response and distinguish from the randomly sampled negative response. \cite{xu2020learn_aux} employed auxiliary objectives to measure the utterance consistency of a dialogue session and restore the corrupted utterances when fine-tuning models for response selection tasks.&#10;&#10;Despite the progress made by the methods mentioned above, the guideline of dialogue-adaptive pre-training is still not fully exploited, especially for open-domain dialogues as the main focus of this work. The major focus of existing dialogue-adaptive pre-training strategies revolves around merely one attribute, coherence, either on token-level or sentence-level. Therefore, the previous studies can be referred to as coherence-oriented objectives \cite{xu2020learn_aux}, which are commonly implemented in a discriminative way: corrupting the dialogue context by masking, randomly sampling, or re-ordering as negative samples, and restoring the original context as the positive ones. &#10;&#10;As opposed to written language texts, dialogues, as spoken language texts, are full of redundant or uninformative utterances. As it is pointed out by recent dialogue evaluation studies \cite{WhatMA2019See,UnsupervisedEO2020Mehri,mehri2020usr,pang2020towards,DesigningPA2020Zhao}, estimating whether an open-domain dialogue is informative is also essential for assessing its quality, which motivates us to improve dialogue-adaptive pre-training by considering informativeness to better simulate dialogue-specific characteristics. In addition, measuring the informativeness of the utterance would provide a more fine-grained self-supervision like informativeness degree during training, and guide the model to get rid of meaningless utterances. Therefore, in this work, we make a first attempt to bridge comprehensive dialogue quality estimation and pre-training and present dialogue-adaptive pre-training objectives (DAPO) by considering the salient characteristics, including coherence and specificity. The overall workflow of utilizing DAPO and the comparison between existing methods are shown in Figure \ref{fig:overll_idea}. A detailed explanation of the terms are shown as follows:&#10;&#09;\begin{enumerate}&#10;&#09;&#09;\item coherence: whether a dialogue is coherent in its semantics and logic so that readers feel easy to read it.&#10;&#09;&#09;\item specificity: whether the tokens, phrases, and expressions in a dialogue are specific and diverse to avoid being dull and monotonous.&#10;&#09;\end{enumerate}&#10;&#10;As the foundation for model pre-training, we first synthesize a new open-domain dialogue corpus used for pre-training with each dialogue in it as a positive sample to facilitate discriminative pre-training. Inspired by \cite{ModelingLC2005Barzilay,CoherenceMF2018Cervone,DialogueCA2020Mesgar}, we propose coherence-oriented context corruption, including utterance ordering, utterance insertion and utterance replacement, to generate incoherent dialogues as negative samples. Next, we do specificity-oriented automatic rescoring by multiplying the scores of positive samples with a token-specificity coefficient measured by n-gram normalized inverse document frequency (N-NIDF) to further distinguish the samples by quantifying how specific they are. Finally, the PrLMs are trained on all these unsupervised annotated samples with a regression task.&#10;&#10;In summary, our contributions in the paper are three-fold:&#10;&#09;\begin{enumerate}&#10;&#09;&#09;\item We bridge the gap between dialogue-adaptive pre-training and dialogue estimation to facilitate a new research line of dialogue-adaptive pre-training from quality estimation by considering the salient characteristics, including coherence and informativeness.&#10;&#09;&#09;\item To simulate dialogue-specific features, we propose dialogue-adaptive pre-training objectives&#10;&#09;&#09;including coherence-oriented context corruption and token-specificity rescoring, for open-domain dialogue modeling; &#10;&#09;&#09;\item We empirically verify that the quality estimation methods improve PrLMs on both response selection and quality evaluation tasks by modeling the diverse aspects of dialogue-related characteristics.&#10;&#09;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Dialogue Analysis Techniques, Language Model Training, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2102.05474" label="2102.05474">
        <attvalues>
          <attvalue for="0" value="Multi-turn Dialogue Reading Comprehension with Pivot Turns and Knowledge" />
          <attvalue for="1" value="  Multi-turn dialogue reading comprehension aims to teach machines to read&#10;dialogue contexts and solve tasks such as response selection and answering&#10;questions. The major challenges involve noisy history contexts and especial&#10;prerequisites of commonsense knowledge that is unseen in the given material.&#10;Existing works mainly focus on context and response matching approaches. This&#10;work thus makes the first attempt to tackle the above two challenges by&#10;extracting substantially important turns as pivot utterances and utilizing&#10;external knowledge to enhance the representation of context. We propose a&#10;pivot-oriented deep selection model (PoDS) on top of the Transformer-based&#10;language models for dialogue comprehension. In detail, our model first picks&#10;out the pivot utterances from the conversation history according to the&#10;semantic matching with the candidate response or question, if any. Besides,&#10;knowledge items related to the dialogue context are extracted from a knowledge&#10;graph as external knowledge. Then, the pivot utterances and the external&#10;knowledge are combined with a well-designed mechanism for refining predictions.&#10;Experimental results on four dialogue comprehension benchmark tasks show that&#10;our proposed model achieves great improvements on baselines. A series of&#10;empirical comparisons are conducted to show how our selection strategies and&#10;the extra knowledge injection influence the results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.04812" label="1908.04812">
        <attvalues>
          <attvalue for="0" value="An Effective Domain Adaptive Post-Training Method for BERT in Response&#10;  Selection" />
          <attvalue for="1" value="  We focus on multi-turn response selection in a retrieval-based dialog system.&#10;In this paper, we utilize the powerful pre-trained language model&#10;Bi-directional Encoder Representations from Transformer (BERT) for a multi-turn&#10;dialog system and propose a highly effective post-training method on&#10;domain-specific corpus. Although BERT is easily adopted to various NLP tasks&#10;and outperforms previous baselines of each task, it still has limitations if a&#10;task corpus is too focused on a certain domain. Post-training on&#10;domain-specific corpus (e.g., Ubuntu Corpus) helps the model to train&#10;contextualized representations and words that do not appear in general corpus&#10;(e.g., English Wikipedia). Experimental results show that our approach achieves&#10;new state-of-the-art on two response selection benchmarks (i.e., Ubuntu Corpus&#10;V1, Advising Corpus) performance improvement by 5.9% and 6% on R@1.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.04908" label="2004.04908">
        <attvalues>
          <attvalue for="0" value="Designing Precise and Robust Dialogue Response Evaluators" />
          <attvalue for="1" value="  Automatic dialogue response evaluator has been proposed as an alternative to&#10;automated metrics and human evaluation. However, existing automatic evaluators&#10;achieve only moderate correlation with human judgement and they are not robust.&#10;In this work, we propose to build a reference-free evaluator and exploit the&#10;power of semi-supervised training and pretrained (masked) language models.&#10;Experimental results demonstrate that the proposed evaluator achieves a strong&#10;correlation (&gt; 0.6) with human judgement and generalizes robustly to diverse&#10;responses and corpora. We open-source the code and data in&#10;https://github.com/ZHAOTING/dialog-processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.08044" label="1806.08044">
        <attvalues>
          <attvalue for="0" value="Coherence Models for Dialogue" />
          <attvalue for="1" value="  Coherence across multiple turns is a major challenge for state-of-the-art&#10;dialogue models. Arguably the most successful approach to automatically&#10;learning text coherence is the entity grid, which relies on modelling patterns&#10;of distribution of entities across multiple sentences of a text. Originally&#10;applied to the evaluation of automatic summaries and the news genre, among its&#10;many extensions, this model has also been successfully used to assess dialogue&#10;coherence. Nevertheless, both the original grid and its extensions do not model&#10;intents, a crucial aspect that has been studied widely in the literature in&#10;connection to dialogue structure. We propose to augment the original grid&#10;document representation for dialogue with the intentional structure of the&#10;conversation. Our models outperform the original grid representation on both&#10;text discrimination and insertion, the two main standard tasks for coherence&#10;assessment across three different dialogue datasets, confirming that intents&#10;play a key role in modelling dialogue coherence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.08486" label="1908.08486">
        <attvalues>
          <attvalue for="0" value="Dialogue Coherence Assessment Without Explicit Dialogue Act Labels" />
          <attvalue for="1" value="  Recent dialogue coherence models use the coherence features designed for&#10;monologue texts, e.g. nominal entities, to represent utterances and then&#10;explicitly augment them with dialogue-relevant features, e.g., dialogue act&#10;labels. It indicates two drawbacks, (a) semantics of utterances is limited to&#10;entity mentions, and (b) the performance of coherence models strongly relies on&#10;the quality of the input dialogue act labels. We address these issues by&#10;introducing a novel approach to dialogue coherence assessment. We use dialogue&#10;act prediction as an auxiliary task in a multi-task learning scenario to obtain&#10;informative utterance representations for coherence assessment. Our approach&#10;alleviates the need for explicit dialogue act labels during evaluation. The&#10;results of our experiments show that our model substantially (more than 20&#10;accuracy points) outperforms its strong competitors on the DailyDialogue&#10;corpus, and performs on par with them on the SwitchBoard corpus for ranking&#10;dialogues concerning their coherence.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14556" label="2305.14556">
        <attvalues>
          <attvalue for="0" value="Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented&#10;  Dialogues and Annotations" />
          <attvalue for="1" value="  Large pre-trained language models have exhibited unprecedented capabilities&#10;in producing high-quality text via prompting techniques. This fact introduces&#10;new possibilities for data collection and annotation, particularly in&#10;situations where such data is scarce, complex to gather, expensive, or even&#10;sensitive. In this paper, we explore the potential of these models to generate&#10;and annotate goal-oriented dialogues, and conduct an in-depth analysis to&#10;evaluate their quality. Our experiments employ ChatGPT, and encompass three&#10;categories of goal-oriented dialogues (task-oriented, collaborative, and&#10;explanatory), two generation modes (interactive and one-shot), and two&#10;languages (English and Italian). Based on extensive human-based evaluations, we&#10;demonstrate that the quality of generated dialogues and annotations is on par&#10;with those generated by humans.&#10;" />
          <attvalue for="2" value="&#10;&#10;Since its initial release in November 2022, ChatGPT has been tested on various tasks, including traditional NLP tasks, domain-specific knowledge such as medicine, fixing computer programs, and even solving neuro-psychological tests used for humans. Despite its high potential, little attention has been paid to ChatGPT's ability to produce annotated data for training purposes \cite{gpt-as-annotator-2022, Huang2023IsCB-ChatGPT-Annotator}. In this paper, we specifically aim to evaluate ChatGPT's ability to generate novel, human-like dialogues and annotate them according to a predetermined taxonomy, which is a very challenging task in NLP.\\&#10;Although we anticipate further improvements in the language model's conversational abilities, generating high-quality training data remains essential for developing adaptable dialogue systems for various domains and conversational contexts. However, collecting human-like dialogues often requires complex and expensive settings to simulate ecological data. Current methodologies include the &quot;Wizard of Oz&quot; method \cite{wizardofoz-reference}, in which a human (i.e., the Wizard) simulates the system's output, the Map Task \cite{Hcrc_map1991} method, initially proposed for an instruction-giving task involving two participants who must collaborate to reproduce the itinerary on a plain map, and transcribing audio and video recordings \cite{wachsmuth-alshomary-2022-mama}.&#10;\\&#10;Dialogue annotation is crucial for training and evaluating dialogue models \cite{budzianowski2018multiwoz}. However, this is a complex and costly process, as annotation schemas are often not standardized and human annotation is time-consuming and error-prone. As a result, only a few annotated dialogue datasets are available, which cover limited dialogue types and domains, with insufficient data and ongoing debates regarding their quality (as exemplified by the various versions of the MultiWOZ dataset). Therefore, simplifying the dialogue collection and annotation process would greatly benefit research in this area.&#10;\\&#10;Meanwhile, very large pre-trained language models (LPLMs), such as BERT \cite{devlin2018bert}, T5 \cite{raffel-t5}, GPT-3 \cite{GPT-3-reference}, LaMDA \cite{lamda-paper}, PaLM \cite{chowdhery2022palm}, and IstructGPT \cite{instruct-gpt}, have demonstrated unparalleled ability to generate high-quality text via prompting strategies \cite{prompting-survey}. While recent studies have examined the potential of these language models as annotators \cite{gpt-as-annotator-2022}, this area remains largely unexplored, especially regarding dialogue generation and annotation.&#10;\\&#10;This paper investigates the ability of Language-based Pre-trained Language Models (LPLMs) to generate and annotate different types of dialogues. We conduct a thorough qualitative analysis of both the generated dialogues and their semantic annotations to evaluate their usefulness in training dialogue systems. Specifically, we consider three dialogue scenarios of varying complexity: (i) task-oriented dialogues \cite{McTear2020} that present a user with a specific need (e.g., booking a restaurant) and an agent that helps the user achieve their goal through dialogue; (ii) cooperative dialogues where two participants collaborate to achieve a shared goal (e.g., matching job requests and offers); and (iii) explanatory dialogues aimed at providing clarifications and explanations on a specific topic through dialogue (e.g., a medical doctor explaining a diagnosis to a patient or a teacher-student scenario).&#10;\\&#10;To evaluate the quality of the generated dialogues, we conducted a crowd-sourced evaluation using a questionnaire based on a 6-point Likert scale, comparing both the dialogues generated by ChatGPT and corresponding human-generated dialogues. Additionally, we used standard measures for dialogue state tracking \cite{DSTCReview}, i.e., Slot Accuracy and Joint Goal Accuracy, to evaluate the quality of the annotations performed by ChatGPT.&#10;&#10;The contributions of the paper are as follows:&#10;(ii) We report on the first experiment, to the best of our knowledge, aiming at generating complex, human-like dialogues through controlled prompting of a large language model (ChatGPT). (ii) We show that the quality of automatically generated dialogues is comparable to that of reference human-generated dialogues. We also show that the quality of the annotations generated by ChatGPT is comparable to that of human annotations. (iii) We report a number of critical limitations, including the generation of hallucinations, which need to be considered in future research.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Dialogue Generation Techniques, Computer Science, Linguistics, Language Model Evaluation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2212.06346" label="2212.06346">
        <attvalues>
          <attvalue for="0" value="The Massively Multilingual Natural Language Understanding 2022&#10;  (MMNLU-22) Workshop and Competition" />
          <attvalue for="1" value="  Despite recent progress in Natural Language Understanding (NLU), the creation&#10;of multilingual NLU systems remains a challenge. It is common to have NLU&#10;systems limited to a subset of languages due to lack of available data. They&#10;also often vary widely in performance. We launch a three-phase approach to&#10;address the limitations in NLU and help propel NLU technology to new heights.&#10;We release a 52 language dataset called the Multilingual Amazon SLU resource&#10;package (SLURP) for Slot-filling, Intent classification, and Virtual assistant&#10;Evaluation, or MASSIVE, in an effort to address parallel data availability for&#10;voice assistants. We organize the Massively Multilingual NLU 2022 Challenge to&#10;provide a competitive environment and push the state-of-the art in the&#10;transferability of models into other languages. Finally, we host the first&#10;Massively Multilingual NLU workshop which brings these components together. The&#10;MMNLU workshop seeks to advance the science behind multilingual NLU by&#10;providing a platform for the presentation of new research in the field and&#10;connecting teams working on this research direction. This paper summarizes the&#10;dataset, workshop and the competition and the findings of each phase.&#10;" />
          <attvalue for="2" value="&#10;&#10;According to a 2020 study by Juniper Research \cite{juniper} it is expected that by 2024 there will be over 8 billion virtual assistants worldwide, the majority of which will be on smartphones.&#10;Additionally, over 100 million smart speakers have been sold, and virtual assistants continue to be integrated into new products. These devices have in common that humans interact with them via natural language interfaces.&#10;This development has significantly boosted research to advance natural language understanding. &#10;However, most natural language understanding work focuses on only a few of the more than 4,000 written languages in the world.&#10;The limitation is driven by the lack of labeled data, the expense associated with human-based quality assurance, model maintenance, update costs, and more.&#10;To overcome these hurdles, further research in the field of multilingual natural language understanding is needed to enable natural language understanding for currently not- or under-served languages.&#10;With \cite{MetaNoLanguageLeftBehind} and related work, we have seen progress in recent years on the expansion of machine translation into the domain of under-served languages both by advancing science as well as creation of corpora in machine translation. However, in areas as NLU modeling for virtual assistants, many of these limitations still remain. &#10;The vision of this workshop is to address the limitations in NLU and help propel NLU technology into the 50-language, 100-language, and even the 1,000-language regime, both for production systems and for research endeavors, succinctly captured by our slogan, {Let’s scale natural language understanding technology to every language on Earth!}. We do this via a three-pronged approach. First, we created and released the Multilingual Amazon SLU resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation, or MASSIVE dataset \cite{massiveDataset}, containing 1 million realistic, parallel, labeled virtual assistant text utterances spanning 51 languages. Second, we hosted the Massively Multilingual NLU (MMNLU) 2022 Challenge, a competition designed to advance massively multilingual NLU modeling. Finally, we organized the first MMNLU workshop to bring together researchers working in the field of NLU. By providing much needed labelled data, motivating multilingual NLU exploration and bringing NLU researchers together to share findings and spark further collaboration, we hope to push the state-of-the art in multilingual natural language understanding technology.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Dataset Development, Linguistics, AI Model Transferability, Artificial Intelligence, Information Science, Multilingual NLU" />
        </attvalues>
      </node>
      <node id="2304.08823" label="2304.08823">
        <attvalues>
          <attvalue for="0" value="Transfer to a Low-Resource Language via Close Relatives: The Case Study&#10;  on Faroese" />
          <attvalue for="1" value="  Multilingual language models have pushed state-of-the-art in cross-lingual&#10;NLP transfer. The majority of zero-shot cross-lingual transfer, however, use&#10;one and the same massively multilingual transformer (e.g., mBERT or XLM-R) to&#10;transfer to all target languages, irrespective of their typological,&#10;etymological, and phylogenetic relations to other languages. In particular,&#10;readily available data and models of resource-rich sibling languages are often&#10;ignored. In this work, we empirically show, in a case study for Faroese -- a&#10;low-resource language from a high-resource language family -- that by&#10;leveraging the phylogenetic information and departing from the&#10;'one-size-fits-all' paradigm, one can improve cross-lingual transfer to&#10;low-resource languages. In particular, we leverage abundant resources of other&#10;Scandinavian languages (i.e., Danish, Norwegian, Swedish, and Icelandic) for&#10;the benefit of Faroese. Our evaluation results show that we can substantially&#10;improve the transfer performance to Faroese by exploiting data and models of&#10;closely-related high-resource languages. Further, we release a new web corpus&#10;of Faroese and Faroese datasets for named entity recognition (NER), semantic&#10;text similarity (STS), and new language models trained on all Scandinavian&#10;languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;Massively multilingual Transformer-based language models (MMTs) such as mBERT \cite{devlin-etal-2019-bert}, XLM-RoBERTa \cite{conneau-etal-2020-unsupervised} and mT5 \cite{xue-etal-2021-mt5} have been the driving force of modern multilingual NLP, allowing for rapid bootstrapping of language technology for a wide range of low(er)-resource languages by means of (zero-shot or few-shot) cross-lingual transfer from high(er)-resource languages \cite{lauscher-etal-2020-zero,hu2020xtreme,xu-murray-2022-por,schmidt-etal-2022-dont}. &#10;Cross-lingual transfer with MMTs is not without drawbacks. MMTs' representation spaces &#10;are heavily skewed in favor of high-resource languages, for which they have been exposed to much more data in pretraining ~\cite{joshi-etal-2020-state,wu-dredze-2020-languages}; combined with the `curse of multilinguality' -- i.e., limited per-language representation quality stemming from a limited capacity of the model \cite{conneau-etal-2020-unsupervised,pfeiffer-etal-2022-lifting} -- this leads to lower representational quality for languages underrepresented in MMTs' pretraining. Cross-lingual transfer with MMTs thus fails exactly in settings in which it is needed the most: for low-resource languages with small digital footprint \cite{zhao-etal-2021-closer}.&#10;Despite these proven practical limitations, the vast majority of work on cross-lingual transfer still relies on MMTs due to their appealing conceptual generality: in theory, they support transfer between any two languages seen in their pretraining. Such strict reliance on MMTs effectively ignores the linguistic phylogenetics and fails to directly leverage resources of resource-rich languages that are closely related to a target language of interest. &#10;&#10;In this work, we attempt to mitigate the above limitations for a particular group of languages, departing from the `one-size-fits-all' paradigm based on MMTs. We focus on a frequent and realistic setup in which the target language is a low-resource language but from a high-resource language family, i.e., with closely related resource-rich languages. A recent comprehensive evaluation of the languages used in Europe scores languages based on the available resources. Languages such as German and Spanish score at around 0.5 of the English scores, and more than half of the languages are scored below 0.02 of the English score. Many, including almost all regional and minority languages such as Faroese, Scottish Gaelic, Occitan, Luxembourgish, Romani languages, Sicilian and Meänkieli have the score of (almost) 0. However, what differentiates these languages from low-resource languages from Africa (e.g., Niger-Congo family) or indigenous languages of Latin America (e.g., Tupian family) is the fact that they typically have closely related high-resource languages as `language siblings'. In this case, we believe, language models (LMs) of closely related high-resource languages promise more effective transfer compared to using MMTs, plagued by the `curse of multilinguality', as the vehicle of transfer. &#10;&#10;In this proof-of-concept case study, we focus on Faroese as the target language and demonstrate the benefits of linguistically informed transfer. We take advantage of available data and resources from the closely related but much more `NLP-developed' other Scandinavian languages. We show that using ``Scandinavian'' LMs brings substantial gains in downstream transfer to Faroese compared to using XLM-R as a widely used off-the-shelf MMT. The gains are particularly pronounced for the task of semantic text similarity (STS), the only high-level semantic task in our evaluation. &#10;We further show that adding a limited-size target-language corpus to LM's pretraining corpora brings further gains in downstream transfer. &#10;As another contribution of this work, we collect and release: (1) a corpus of web-scraped monolingual Faroese, (2) multiple LMs suitable for Faroese, including those trained on all five Scandinavian languages, and (3) two new task-specific datasets for Faroese labeled by native speakers: for NER and STS. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Cross-Lingual Transfer Learning, Computer Science, Linguistics, Cognitive Science, Multilingual Language Models, Information Science, Low-Resource Language Processing" />
        </attvalues>
      </node>
      <node id="2204.13869" label="2204.13869">
        <attvalues>
          <attvalue for="0" value="Por Qu\'e N\~ao Utiliser Alla Spr{\aa}k? Mixed Training with Gradient&#10;  Optimization in Few-Shot Cross-Lingual Transfer" />
          <attvalue for="1" value="  The current state-of-the-art for few-shot cross-lingual transfer learning&#10;first trains on abundant labeled data in the source language and then&#10;fine-tunes with a few examples on the target language, termed target-adapting.&#10;Though this has been demonstrated to work on a variety of tasks, in this paper&#10;we show some deficiencies of this approach and propose a one-step mixed&#10;training method that trains on both source and target data with&#10;\textit{stochastic gradient surgery}, a novel gradient-level optimization.&#10;Unlike the previous studies that focus on one language at a time when&#10;target-adapting, we use one model to handle all target languages simultaneously&#10;to avoid excessively language-specific models. Moreover, we discuss the&#10;unreality of utilizing large target development sets for model selection in&#10;previous literature. We further show that our method is both development-free&#10;for target languages, and is also able to escape from overfitting issues. We&#10;conduct a large-scale experiment on 4 diverse NLP tasks across up to 48&#10;languages. Our proposed method achieves state-of-the-art performance on all&#10;tasks and outperforms target-adapting by a large margin, especially for&#10;languages that are linguistically distant from the source language, e.g., 7.36%&#10;F1 absolute gain on average for the NER task, up to 17.60% on Punjabi.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.11255" label="2210.11255">
        <attvalues>
          <attvalue for="0" value="Evidence &gt; Intuition: Transferability Estimation for Encoder Selection" />
          <attvalue for="1" value="  With the increase in availability of large pre-trained language models (LMs)&#10;in Natural Language Processing (NLP), it becomes critical to assess their fit&#10;for a specific target task a priori - as fine-tuning the entire space of&#10;available LMs is computationally prohibitive and unsustainable. However,&#10;encoder transferability estimation has received little to no attention in NLP.&#10;In this paper, we propose to generate quantitative evidence to predict which&#10;LM, out of a pool of models, will perform best on a target task without having&#10;to fine-tune all candidates. We provide a comprehensive study on LM ranking for&#10;10 NLP tasks spanning the two fundamental problem types of classification and&#10;structured prediction. We adopt the state-of-the-art Logarithm of Maximum&#10;Evidence (LogME) measure from Computer Vision (CV) and find that it positively&#10;correlates with final LM performance in 94% of the setups. In the first study&#10;of its kind, we further compare transferability measures with the de facto&#10;standard of human practitioner ranking, finding that evidence from quantitative&#10;metrics is more robust than pure intuition and can help identify unexpected LM&#10;candidates.&#10;" />
          <attvalue for="2" value="&#10;Advances in Deep Learning-based NLP and CV build on expressive representations from encoder models pre-trained on massive corpora. Downstream models make use of latent information in these representations to extract relevant features for the task at hand. Within this paradigm, deciding which pre-trained encoder to use in any task-specific architecture is crucial, however training a model using each encoder candidate is infeasible. In absence of prior heuristics (e.g., via related work), the choice of encoder has therefore prevailingly been based on practitioner intuition rather than quantitative evidence.&#10;&#10;In NLP, prior work has examined the different yet related task of performance prediction \cite{xia-etal-2020-predicting,ye-etal-2021-towards}, surveyed and categorized LMs~\cite{xia-etal-2020-bert}, and used probing to predict LM performance specifically for dependency parsing~\cite{muller-eberstein-etal-2022-sort},&#10;but has yet to extensively investigate how to rank the increasingly large number of pre-trained LM encoders across various tasks and domains.&#10;Preliminary work by \cite{you2021logme} shows that the LogME estimator holds promise, including the first steps for encoder selection in NLP. With their main focus being on CV, however, they evaluate only a limited set of tasks and models for NLP and use self-reported benchmark scores instead of running controlled experiments which should include, e.g., the variance across initializations, domains, and fine-tuning strategies (\ref{sec:transferability}).&#10;As such, we seek to answer: How well can we estimate the transferability of pre-trained LMs to specific NLP tasks?&#10;To do so, we contribute:&#10;\begin{itemize}&#10; \item The broadest encoder selection study in NLP to date, on 10 domain-diverse classification and structured prediction tasks (\ref{sec:experiments});&#10; \item An extensive evaluation and analysis across multiple dimensions of variation, including seven general vs.\ domain-specific LMs, \cls vs.\ mean representations, and head vs.\ full model fine-tuning (\ref{sec:analysis});&#10; \item A study with NLP experts, comparing the prevailing ranking of LMs by human intuition with LogME's empirical evidence (\ref{sec:human-study}); &#10; \item Guidelines for applying and interpreting transferability measures to NLP (\ref{sec:conclusion}), and an open-source toolkit for efficient, task-adaptive LM pre-selection.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Language Model Evaluation, Computer Vision, Mathematics, Natural Language Processing, Transfer Learning Assessment" />
        </attvalues>
      </node>
      <node id="2212.00678" label="2212.00678">
        <attvalues>
          <attvalue for="0" value="Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis" />
          <attvalue for="1" value="  Multimodal learning pipelines have benefited from the success of pretrained&#10;language models. However, this comes at the cost of increased model parameters.&#10;In this work, we propose Adapted Multimodal BERT (AMB), a BERT-based&#10;architecture for multimodal tasks that uses a combination of adapter modules&#10;and intermediate fusion layers. The adapter adjusts the pretrained language&#10;model for the task at hand, while the fusion layers perform task-specific,&#10;layer-wise fusion of audio-visual information with textual BERT&#10;representations. During the adaptation process the pre-trained language model&#10;parameters remain frozen, allowing for fast, parameter-efficient training. In&#10;our ablations we see that this approach leads to efficient models, that can&#10;outperform their fine-tuned counterparts and are robust to input noise. Our&#10;experiments on sentiment analysis with CMU-MOSEI show that AMB outperforms the&#10;current state-of-the-art across metrics, with 3.4% relative reduction in the&#10;resulting error and 2.1% relative improvement in 7-class classification&#10;accuracy.&#10;" />
          <attvalue for="2" value="&#10;&#10;Over the past few years, we have witnessed impressive breakthroughs in the field of multimodal applications, due to the abundance of multimedia data and progress in core machine learning algorithms. &#10;This has set the scene for multimodal machine learning as one of the frontiers of applied AI research. For wide-spread adoption in the real-world, models that strike the correct balance between performance and parameter efficiency should be developed.&#10;&#10;GPT \cite{radford2018language} and BERT \cite{devlin-etal-2019-bert} were the first to establish the effectiveness of pre-training large scale language models on general tasks and then refining them for a specific task. Inspired by this approach, VilBERT \cite{lu_vilbert_2019} leveraged parallel multimodal data for pre-training a visual-language model. Other researchers \cite{tsimpoukelli2021multimodal, eichenberg_magma_2021, 10.1145/3394171.3413678, rahman_integrating_2020} have adopted a more flexible method: adapting a model pre-trained only on language for multimodal tasks. &#10;&#10;The standard method of transferring a pre-trained model to a downstream task is called fine-tuning, which involves updating the pre-trained weights with backpropagation. However this method incurs intensive data and computational costs, while some information is lost due to using only task-specific data for updating model parameters. This phenomenon is known as catastrophic forgetting \cite{bower_catastrophic_1989}.&#10;To solve these issues, GPT-3 \cite{brown_language_2020} proposed ``prompt tuning&quot;, an intuitive method to transfer a powerful pretrained model only with text interactions, called ``prompts&quot;, without any gradient updates.&#10;This idea was later extended, with many variations \cite{lester-etal-2021-power, li-liang-2021-prefix}, to make these prompts trainable, now called ``soft prompts&quot;. Houlsby et al. \cite{houlsby2019parameter} proposed adapters, a down-projected feedforward network that updates the representations of each BERT layer. &#10;Frozen \cite{tsimpoukelli2021multimodal} applied these ideas in multimodal learning, by translating an image to a visual soft prompt that is prepended to the input of a standard language model, which keeps its original pre-trained weights unchanged (frozen).&#10;MAGMA \cite{eichenberg_magma_2021} extended this by showing that the addition of adapter layers \cite{houlsby2019parameter} in between the frozen language layers outperforms Frozen.&#10;Flamingo \cite{alayrac_flamingo_2022} scaled up and optimised this concept by introducing a flexible visual encoder which can turn arbitrary sequences of images or even video frames to a fixed number of visual tokens.&#10;&#10;Early applications of deep learning for multimodal sentiment analysis focused on the use of Recurrent Neural Networks (RNNs) \cite{metallinou_context-sensitive_2015, wollmer_lstm-modeling_2013, shenoy-sardana-2020-multilogue} and Convolutional Neural Networks (CNNs) \cite{poria_convolutional_2016} aiming to model contextual information.&#10;The next innovation was the introduction of the attention model to create sophisticated fusion approaches \cite{gu_multimodal_2018, wang_words_2018}.&#10;This naturally led to the incorporation of the transformer \cite{vaswani_attention_2017} as the central model for this task \cite{DBLP:journals/corr/abs-1806-06176, delbrouck-etal-2020-transformer}.&#10;Lately, large-scale pretrained language transformers, such as BERT \cite{devlin-etal-2019-bert}, have become the norm because of consistent performance gains.&#10;ICCN \cite{Sun2020LearningRB} introduced Deep Canonical Correlation Analysis for jointly learning representations.&#10;Wang et al. \cite{wang_words_2018} and later MAG-BERT \cite{rahman_integrating_2020} proposed shifting methods.&#10;MISA \cite{10.1145/3394171.3413678} produced modality invariant and modality specific representations in an effort to disentangle data relationships.&#10;More recently, many researchers turned their efforts towards intricate multimodal pre-training&#10;strategies, such as \cite{yu2021le, kim_cmsbert-clr_2022}. Such methods are model-agnostic and should be studied separately for a fair comparison.&#10;&#10;We present a simple neural architecture that adapts BERT representations for multimodal fusion which we call Adapted Multimodal BERT (AMB). Our approach extends concepts introduced by visual-language models \cite{tsimpoukelli2021multimodal,eichenberg_magma_2021, alayrac_flamingo_2022} to include audio.&#10;The contributions of our work:&#10;\begin{itemize}&#10; \item AMB is evaluated on multimodal sentiment analysis with the CMU-MOSEI database to achieve new state-of-the-art results, regardless of being lightweight and data-efficient due to a low trainable parameter budget.&#10; \item BERT is tuned in an effective way to adapt without losing prior knowledge, while at the same time squeezing as much useful information as possible from audio-visual modalities.&#10; \item We study our model's robustness to noise and compare its performance with a fine-tuned version and the current state-of-the-art MISA.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Multimodal Learning, Deep Learning Architectures, Signal Processing, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="1908.02265" label="1908.02265">
        <attvalues>
          <attvalue for="0" value="ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for&#10;  Vision-and-Language Tasks" />
          <attvalue for="1" value="  We present ViLBERT (short for Vision-and-Language BERT), a model for learning&#10;task-agnostic joint representations of image content and natural language. We&#10;extend the popular BERT architecture to a multi-modal two-stream model,&#10;pro-cessing both visual and textual inputs in separate streams that interact&#10;through co-attentional transformer layers. We pretrain our model through two&#10;proxy tasks on the large, automatically collected Conceptual Captions dataset&#10;and then transfer it to multiple established vision-and-language tasks --&#10;visual question answering, visual commonsense reasoning, referring expressions,&#10;and caption-based image retrieval -- by making only minor additions to the base&#10;architecture. We observe significant improvements across tasks compared to&#10;existing task-specific models -- achieving state-of-the-art on all four tasks.&#10;Our work represents a shift away from learning groundings between vision and&#10;language only as part of task training and towards treating visual grounding as&#10;a pretrainable and transferable capability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.13884" label="2106.13884">
        <attvalues>
          <attvalue for="0" value="Multimodal Few-Shot Learning with Frozen Language Models" />
          <attvalue for="1" value="  When trained at sufficient scale, auto-regressive language models exhibit the&#10;notable ability to learn a new language task after being prompted with just a&#10;few examples. Here, we present a simple, yet effective, approach for&#10;transferring this few-shot learning ability to a multimodal setting (vision and&#10;language). Using aligned image and caption data, we train a vision encoder to&#10;represent each image as a sequence of continuous embeddings, such that a&#10;pre-trained, frozen language model prompted with this prefix generates the&#10;appropriate caption. The resulting system is a multimodal few-shot learner,&#10;with the surprising ability to learn a variety of new tasks when conditioned on&#10;examples, represented as a sequence of multiple interleaved image and text&#10;embeddings. We demonstrate that it can rapidly learn words for new objects and&#10;novel visual categories, do visual question-answering with only a handful of&#10;examples, and make use of outside knowledge, by measuring a single model on a&#10;variety of established and new benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.05253" label="2112.05253">
        <attvalues>
          <attvalue for="0" value="MAGMA -- Multimodal Augmentation of Generative Models through&#10;  Adapter-based Finetuning" />
          <attvalue for="1" value="  Large-scale pretraining is fast becoming the norm in Vision-Language (VL)&#10;modeling. However, prevailing VL approaches are limited by the requirement for&#10;labeled data and the use of complex multi-step pretraining objectives. We&#10;present MAGMA - a simple method for augmenting generative language models with&#10;additional modalities using adapter-based finetuning. Building on Frozen, we&#10;train a series of VL models that autoregressively generate text from arbitrary&#10;combinations of visual and textual input. The pretraining is entirely&#10;end-to-end using a single language modeling objective, simplifying optimization&#10;compared to previous approaches. Importantly, the language model weights remain&#10;unchanged during training, allowing for transfer of encyclopedic knowledge and&#10;in-context learning abilities from language pretraining. MAGMA outperforms&#10;Frozen on open-ended generative tasks, achieving state of the art results on&#10;the OKVQA benchmark and competitive results on a range of other popular VL&#10;benchmarks, while pretraining on 0.2% of the number of samples used to train&#10;SimVLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.08660" label="1805.08660">
        <attvalues>
          <attvalue for="0" value="Multimodal Affective Analysis Using Hierarchical Attention Strategy with&#10;  Word-Level Alignment" />
          <attvalue for="1" value="  Multimodal affective computing, learning to recognize and interpret human&#10;affects and subjective information from multiple data sources, is still&#10;challenging because: (i) it is hard to extract informative features to&#10;represent human affects from heterogeneous inputs; (ii) current fusion&#10;strategies only fuse different modalities at abstract level, ignoring&#10;time-dependent interactions between modalities. Addressing such issues, we&#10;introduce a hierarchical multimodal architecture with attention and word-level&#10;fusion to classify utter-ance-level sentiment and emotion from text and audio&#10;data. Our introduced model outperforms the state-of-the-art approaches on&#10;published datasets and we demonstrated that our model is able to visualize and&#10;interpret the synchronized attention over modalities.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.06176" label="1806.06176">
        <attvalues>
          <attvalue for="0" value="Learning Factorized Multimodal Representations" />
          <attvalue for="1" value="  Learning multimodal representations is a fundamentally complex research&#10;problem due to the presence of multiple heterogeneous sources of information.&#10;Although the presence of multiple modalities provides additional valuable&#10;information, there are two key challenges to address when learning from&#10;multimodal data: 1) models must learn the complex intra-modal and cross-modal&#10;interactions for prediction and 2) models must be robust to unexpected missing&#10;or noisy modalities during testing. In this paper, we propose to optimize for a&#10;joint generative-discriminative objective across multimodal data and labels. We&#10;introduce a model that factorizes representations into two sets of independent&#10;factors: multimodal discriminative and modality-specific generative factors.&#10;Multimodal discriminative factors are shared across all modalities and contain&#10;joint multimodal features required for discriminative tasks such as sentiment&#10;prediction. Modality-specific generative factors are unique for each modality&#10;and contain the information required for generating data. Experimental results&#10;show that our model is able to learn meaningful multimodal representations that&#10;achieve state-of-the-art or competitive performance on six multimodal datasets.&#10;Our model demonstrates flexible generative capabilities by conditioning on&#10;independent factors and can reconstruct missing modalities without&#10;significantly impacting performance. Lastly, we interpret our factorized&#10;representations to understand the interactions that influence multimodal&#10;learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.05544" label="1911.05544">
        <attvalues>
          <attvalue for="0" value="Learning Relationships between Text, Audio, and Video via Deep Canonical&#10;  Correlation for Multimodal Language Analysis" />
          <attvalue for="1" value="  Multimodal language analysis often considers relationships between features&#10;based on text and those based on acoustical and visual properties. Text&#10;features typically outperform non-text features in sentiment analysis or&#10;emotion recognition tasks in part because the text features are derived from&#10;advanced language models or word embeddings trained on massive data sources&#10;while audio and video features are human-engineered and comparatively&#10;underdeveloped. Given that the text, audio, and video are describing the same&#10;utterance in different ways, we hypothesize that the multimodal sentiment&#10;analysis and emotion recognition can be improved by learning (hidden)&#10;correlations between features extracted from the outer product of text and&#10;audio (we call this text-based audio) and analogous text-based video. This&#10;paper proposes a novel model, the Interaction Canonical Correlation Network&#10;(ICCN), to learn such multimodal embeddings. ICCN learns correlations between&#10;all three modes via deep canonical correlation analysis (DCCA) and the proposed&#10;embeddings are then tested on several benchmark datasets and against other&#10;state-of-the-art multimodal embedding algorithms. Empirical results and&#10;ablation studies confirm the effectiveness of ICCN in capturing useful&#10;information from all three views.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.09362" label="1811.09362">
        <attvalues>
          <attvalue for="0" value="Words Can Shift: Dynamically Adjusting Word Representations Using&#10;  Nonverbal Behaviors" />
          <attvalue for="1" value="  Humans convey their intentions through the usage of both verbal and nonverbal&#10;behaviors during face-to-face communication. Speaker intentions often vary&#10;dynamically depending on different nonverbal contexts, such as vocal patterns&#10;and facial expressions. As a result, when modeling human language, it is&#10;essential to not only consider the literal meaning of the words but also the&#10;nonverbal contexts in which these words appear. To better model human language,&#10;we first model expressive nonverbal representations by analyzing the&#10;fine-grained visual and acoustic patterns that occur during word segments. In&#10;addition, we seek to capture the dynamic nature of nonverbal intents by&#10;shifting word representations based on the accompanying nonverbal behaviors. To&#10;this end, we propose the Recurrent Attended Variation Embedding Network (RAVEN)&#10;that models the fine-grained structure of nonverbal subword sequences and&#10;dynamically shifts word representations based on nonverbal cues. Our proposed&#10;model achieves competitive performance on two publicly available datasets for&#10;multimodal sentiment analysis and emotion recognition. We also visualize the&#10;shifted word representations in different nonverbal contexts and summarize&#10;common patterns regarding multimodal variations of word representations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.05787" label="1908.05787">
        <attvalues>
          <attvalue for="0" value="Integrating Multimodal Information in Large Pretrained Transformers" />
          <attvalue for="1" value="  Recent Transformer-based contextual word representations, including BERT and&#10;XLNet, have shown state-of-the-art performance in multiple disciplines within&#10;NLP. Fine-tuning the trained contextual models on task-specific datasets has&#10;been the key to achieving superior performance downstream. While fine-tuning&#10;these pre-trained models is straightforward for lexical applications&#10;(applications with only language modality), it is not trivial for multimodal&#10;language (a growing area in NLP focused on modeling face-to-face&#10;communication). Pre-trained models don't have the necessary components to&#10;accept two extra modalities of vision and acoustic. In this paper, we proposed&#10;an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG&#10;allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning.&#10;It does so by generating a shift to internal representation of BERT and XLNet;&#10;a shift that is conditioned on the visual and acoustic modalities. In our&#10;experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for&#10;multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly&#10;boosts the sentiment analysis performance over previous baselines as well as&#10;language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet&#10;achieves human-level multimodal sentiment analysis performance for the first&#10;time in the NLP community.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.04830" label="2102.04830">
        <attvalues>
          <attvalue for="0" value="Learning Modality-Specific Representations with Self-Supervised&#10;  Multi-Task Learning for Multimodal Sentiment Analysis" />
          <attvalue for="1" value="  Representation Learning is a significant and challenging task in multimodal&#10;learning. Effective modality representations should contain two parts of&#10;characteristics: the consistency and the difference. Due to the unified&#10;multimodal annotation, existing methods are restricted in capturing&#10;differentiated information. However, additional uni-modal annotations are high&#10;time- and labor-cost. In this paper, we design a label generation module based&#10;on the self-supervised learning strategy to acquire independent unimodal&#10;supervisions. Then, joint training the multi-modal and uni-modal tasks to learn&#10;the consistency and difference, respectively. Moreover, during the training&#10;stage, we design a weight-adjustment strategy to balance the learning progress&#10;among different subtasks. That is to guide the subtasks to focus on samples&#10;with a larger difference between modality supervisions. Last, we conduct&#10;extensive experiments on three public multimodal baseline datasets. The&#10;experimental results validate the reliability and stability of auto-generated&#10;unimodal supervisions. On MOSI and MOSEI datasets, our method surpasses the&#10;current state-of-the-art methods. On the SIMS dataset, our method achieves&#10;comparable performance than human-annotated unimodal labels. The full codes are&#10;available at https://github.com/thuiar/Self-MM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.00006" label="2212.00006">
        <attvalues>
          <attvalue for="0" value="Operationalizing Specifications, In Addition to Test Sets for Evaluating&#10;  Constrained Generative Models" />
          <attvalue for="1" value="  In this work, we present some recommendations on the evaluation of&#10;state-of-the-art generative models for constrained generation tasks. The&#10;progress on generative models has been rapid in recent years. These large-scale&#10;models have had three impacts: firstly, the fluency of generation in both&#10;language and vision modalities has rendered common average-case evaluation&#10;metrics much less useful in diagnosing system errors. Secondly, the same&#10;substrate models now form the basis of a number of applications, driven both by&#10;the utility of their representations as well as phenomena such as in-context&#10;learning, which raise the abstraction level of interacting with such models.&#10;Thirdly, the user expectations around these models and their feted public&#10;releases have made the technical challenge of out of domain generalization much&#10;less excusable in practice. Subsequently, our evaluation methodologies haven't&#10;adapted to these changes. More concretely, while the associated utility and&#10;methods of interacting with generative models have expanded, a similar&#10;expansion has not been observed in their evaluation practices. In this paper,&#10;we argue that the scale of generative models could be exploited to raise the&#10;abstraction level at which evaluation itself is conducted and provide&#10;recommendations for the same. Our recommendations are based on leveraging&#10;specifications as a powerful instrument to evaluate generation quality and are&#10;readily applicable to a variety of tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advances in generative models across different modalities have enabled a myriad of applications, making the reliability failures of such models a frontier of research. Adjacent advances such as prompting \cite{gpt3}, open-vocabulary classification \cite{clip}, etc. have significantly enhanced the utility of such models, implicitly raising the abstraction level of human-model interactions. We believe that modern generative models have had three main impacts:&#10;\begin{enumerate}&#10; \item Fluent Generations: The fluency of generation in both language and vision modalities has rendered existing benchmarks and metrics much less useful in diagnosing system errors \cite{gemv2, bigbench}, with previously widely used benchmarks and metrics on tasks such as machine translation \cite{salted} or summarization \cite{goyalzeroshotnews2022} becoming less useful in gauging system problems. &#10; \item Foundational Role: The same substrate models now form the basis of a number of applications \cite{foundation_models} and this is driven both by the utility of their representations as well as phenomena such as in-context learning \cite{gpt3}, which raise the abstraction level of interacting with such models.&#10; \item Greater User Expectations: The user expectations around these models have made the technical challenge of out of domain generalization much less excusable in practice. &#10;\end{enumerate}&#10;&#10;Subsequently, our evaluation methodologies haven’t adapted to these changes and the evaluation of modern generative models remains a challenge, with many hitherto standard benchmarks and metrics becoming less and less useful with increasing model capabilities. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Large-Scale Model Assessment, Generative Model Evaluation, Artificial Intelligence, Mathematics, Constrained Generation Tasks" />
        </attvalues>
      </node>
      <node id="2205.09988" label="2205.09988">
        <attvalues>
          <attvalue for="0" value="SALTED: A Framework for SAlient Long-Tail Translation Error Detection" />
          <attvalue for="1" value="  Traditional machine translation (MT) metrics provide an average measure of&#10;translation quality that is insensitive to the long tail of behavioral problems&#10;in MT. Examples include translation of numbers, physical units, dropped content&#10;and hallucinations. These errors, which occur rarely and unpredictably in&#10;Neural Machine Translation (NMT), greatly undermine the reliability of&#10;state-of-the-art MT systems. Consequently, it is important to have visibility&#10;into these problems during model development. Towards this direction, we&#10;introduce SALTED, a specifications-based framework for behavioral testing of MT&#10;models that provides fine-grained views of salient long-tail errors, permitting&#10;trustworthy visibility into previously invisible problems. At the core of our&#10;approach is the development of high-precision detectors that flag errors (or&#10;alternatively, verify output correctness) between a source sentence and a&#10;system output. We demonstrate that such detectors could be used not just to&#10;identify salient long-tail errors in MT systems, but also for higher-recall&#10;filtering of the training data, fixing targeted errors with model fine-tuning&#10;in NMT and generating novel data for metamorphic testing to elicit further bugs&#10;in models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.14979" label="2004.14979">
        <attvalues>
          <attvalue for="0" value="Paraphrasing vs Coreferring: Two Sides of the Same Coin" />
          <attvalue for="1" value="  We study the potential synergy between two different NLP tasks, both&#10;confronting predicate lexical variability: identifying predicate paraphrases,&#10;and event coreference resolution. First, we used annotations from an event&#10;coreference dataset as distant supervision to re-score heuristically-extracted&#10;predicate paraphrases. The new scoring gained more than 18 points in average&#10;precision upon their ranking by the original scoring method. Then, we used the&#10;same re-ranking features as additional inputs to a state-of-the-art event&#10;coreference resolution model, which yielded modest but consistent improvements&#10;to the model's performance. The results suggest a promising direction to&#10;leverage data and models for each of the tasks to the benefit of the other.&#10;" />
          <attvalue for="2" value="&#10;Recognizing that mentions of different lexical predicates discuss the same event is challenging \cite{barhom-etal-2019-revisiting}. Lexical resources such as WordNet \cite{miller1995wordnet} capture such synonyms (say, tell) and hypernyms (whisper, talk), as well as antonyms, which can be used to refer to the same event when the arguments are reversed ({[}a]$_0$ beat {[}a]$_1$, {[}a]$_1$ lose to {[}a]$_0$). However, WordNet's coverage is insufficient, in particular, missing context-specific paraphrases (e.g. (hide, launder), in the context of money). Conversely, distributional methods enjoy broader coverage, but their precision for this purpose is limited because distributionally similar terms may often be mutually-exclusive (born, die) or may refer to different event types which are only temporally or causally related (sentenced, convicted).&#10;&#10;Two prominent lines of work pertaining to identifying predicates whose meaning or referents can be matched are cross-document (CD) event coreference resolution and recognizing predicate paraphrases. The former identifies and clusters event mentions, across multiple documents, that refer to the same event within their respective contexts. The latter task, on the other hand, collects pairs of event expressions that, at the generic lexical level, may refer to the same event in certain contexts. Table~\ref{table:ecb_vs_chirps} illustrates this difference with examples of co-referable predicate paraphrases, while their mentions obviously do not always co-refer. &#10;&#10;. Recent systems rely on neural representations of the mentions and their contexts \cite{kenyon-dean-etal-2018-resolving,barhom-etal-2019-revisiting}, while earlier approaches leveraged WordNet and other lexical resources to obtain a signal of whether a pair of mentions may be coreferring \cite[e.g.][]{bejan-harabagiu-2010-unsupervised,yang2015hierarchical}. &#10;&#10;Approaches for acquiring predicate paraphrase, in the form of a pair of paraphrastic predicates or predicate templates, were based mostly on unsupervised signals. These included similarity between argument distributions \cite{Lin2001DIRTD, berant2012global}, backtranslation across languages \cite{barzilay-mckeown-2001-extracting,ganitkevitch2013ppdb,mallinson-etal-2017-paraphrasing}, or leveraging redundant news reports on the same event, which are hence likely to refer to the same events and entities using different words \cite{shinyama2002automatic,shinyama-sekine-2006-preemptive,barzilay-lee-2003-learning,zhang2013harvesting,xu2014extracting,shwartz-etal-2017-acquiring}. In some cases, the paraphrase collection phase includes a step of validating a subset of the paraphrases and training a model on these gold paraphrases to re-rank the entire resource \cite{lan-etal-2017-continuously}. &#10;&#10;In this paper, we study the potential synergy between predicate paraphrases and event coreference resolution. We show that the data and models for&#10;one task can benefit the other. In one direction (Section \ref{sec:improving_chirps}), we use event coreference annotations from the ECB+ dataset as distant supervision to learn an improved scoring of predicate paraphrases in the unsupervised Chirps resource \cite{shwartz-etal-2017-acquiring}. The distantly supervised scorer significantly improves upon ranking by the original Chirps scores, adding&#10;18 points to average precision over a test sample.&#10;&#10;In the other direction (Section \ref{sec:using_chirps}), we incorporate data from Chirps, represented in the Chirps re-scorer feature vector, into a state-of-the-art event coreference system \cite{barhom-etal-2019-revisiting}. Chirps has a substantial coverage over the ECB+ coreferring mention pairs, and consequently, the incorporation &#10;yields a modest but consistent improvement across the various coreference metrics.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Event Coreference Resolution, Artificial Intelligence, Information Science, Predicate Paraphrasing, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1906.01753" label="1906.01753">
        <attvalues>
          <attvalue for="0" value="Revisiting Joint Modeling of Cross-document Entity and Event Coreference&#10;  Resolution" />
          <attvalue for="1" value="  Recognizing coreferring events and entities across multiple texts is crucial&#10;for many NLP applications. Despite the task's importance, research focus was&#10;given mostly to within-document entity coreference, with rather little&#10;attention to the other variants. We propose a neural architecture for&#10;cross-document coreference resolution. Inspired by Lee et al (2012), we jointly&#10;model entity and event coreference. We represent an event (entity) mention&#10;using its lexical span, surrounding context, and relation to entity (event)&#10;mentions via predicate-arguments structures. Our model outperforms the previous&#10;state-of-the-art event coreference model on ECB+, while providing the first&#10;entity coreference results on this corpus. Our analysis confirms that all our&#10;representation elements, including the mention span itself, its context, and&#10;the relation to other mentions contribute to the model's success.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1504.05929" label="1504.05929">
        <attvalues>
          <attvalue for="0" value="A Hierarchical Distance-dependent Bayesian Model for Event Coreference&#10;  Resolution" />
          <attvalue for="1" value="  We present a novel hierarchical distance-dependent Bayesian model for event&#10;coreference resolution. While existing generative models for event coreference&#10;resolution are completely unsupervised, our model allows for the incorporation&#10;of pairwise distances between event mentions -- information that is widely used&#10;in supervised coreference models to guide the generative clustering processing&#10;for better event clustering both within and across documents. We model the&#10;distances between event mentions using a feature-rich learnable distance&#10;function and encode them as Bayesian priors for nonparametric clustering.&#10;Experiments on the ECB+ corpus show that our model outperforms state-of-the-art&#10;methods for both within- and cross-document event coreference resolution.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.00391" label="1708.00391">
        <attvalues>
          <attvalue for="0" value="A Continuously Growing Dataset of Sentential Paraphrases" />
          <attvalue for="1" value="  A major challenge in paraphrase research is the lack of parallel corpora. In&#10;this paper, we present a new method to collect large-scale sentential&#10;paraphrases from Twitter by linking tweets through shared URLs. The main&#10;advantage of our method is its simplicity, as it gets rid of the classifier or&#10;human in the loop needed to select data before annotation and subsequent&#10;application of paraphrase identification algorithms in the previous work. We&#10;present the largest human-labeled paraphrase corpus to date of 51,524 sentence&#10;pairs and the first cross-domain benchmarking for automatic paraphrase&#10;identification. In addition, we show that more than 30,000 new sentential&#10;paraphrases can be easily and continuously captured every month at ~70%&#10;precision, and demonstrate their utility for downstream NLP tasks through&#10;phrasal paraphrase extraction. We make our code and data freely available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.04200" label="1802.04200">
        <attvalues>
          <attvalue for="0" value="End-to-End Automatic Speech Translation of Audiobooks" />
          <attvalue for="1" value="  We investigate end-to-end speech-to-text translation on a corpus of&#10;audiobooks specifically augmented for this task. Previous works investigated&#10;the extreme case where source language transcription is not available during&#10;learning nor decoding, but we also study a midway case where source language&#10;transcription is available at training time only. In this case, a single model&#10;is trained to decode source speech into target text in a single pass.&#10;Experimental results show that it is possible to train compact and efficient&#10;end-to-end speech translation models in this setup. We also distribute the&#10;corpus and hope that our speech translation baseline on this corpus will be&#10;challenged in the future.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.06655" label="1802.06655">
        <attvalues>
          <attvalue for="0" value="Tied Multitask Learning for Neural Speech Translation" />
          <attvalue for="1" value="  We explore multitask models for neural translation of speech, augmenting them&#10;in order to reflect two intuitive notions. First, we introduce a model where&#10;the second task decoder receives information from the decoder of the first&#10;task, since higher-level intermediate representations should provide useful&#10;information. Second, we apply regularization that encourages transitivity and&#10;invertibility. We show that the application of these notions on jointly trained&#10;models improves performance on the tasks of low-resource speech transcription&#10;and translation. It also leads to better performance when using attention&#10;information for word discovery over unsegmented input.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1811.02050" label="1811.02050">
        <attvalues>
          <attvalue for="0" value="Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text&#10;  Translation" />
          <attvalue for="1" value="  End-to-end Speech Translation (ST) models have many potential advantages when&#10;compared to the cascade of Automatic Speech Recognition (ASR) and text Machine&#10;Translation (MT) models, including lowered inference latency and the avoidance&#10;of error compounding. However, the quality of end-to-end ST is often limited by&#10;a paucity of training data, since it is difficult to collect large parallel&#10;corpora of speech and translated transcript pairs. Previous studies have&#10;proposed the use of pre-trained components and multi-task learning in order to&#10;benefit from weakly supervised training data, such as speech-to-transcript or&#10;text-to-foreign-text pairs. In this paper, we demonstrate that using&#10;pre-trained MT or text-to-speech (TTS) synthesis models to convert weakly&#10;supervised data into speech-to-translation pairs for ST training can be more&#10;effective than multi-task learning. Furthermore, we demonstrate that a high&#10;quality end-to-end ST model can be trained using only weakly supervised&#10;datasets, and that synthetic data sourced from unlabeled monolingual text or&#10;speech can be used to improve performance. Finally, we discuss methods for&#10;avoiding overfitting to synthetic speech with a quantitative ablation study.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.07209" label="1904.07209">
        <attvalues>
          <attvalue for="0" value="Attention-Passing Models for Robust and Data-Efficient End-to-End Speech&#10;  Translation" />
          <attvalue for="1" value="  Speech translation has traditionally been approached through cascaded models&#10;consisting of a speech recognizer trained on a corpus of transcribed speech,&#10;and a machine translation system trained on parallel texts. Several recent&#10;works have shown the feasibility of collapsing the cascade into a single,&#10;direct model that can be trained in an end-to-end fashion on a corpus of&#10;translated speech. However, experiments are inconclusive on whether the cascade&#10;or the direct model is stronger, and have only been conducted under the&#10;unrealistic assumption that both are trained on equal amounts of data, ignoring&#10;other available speech recognition and machine translation corpora.&#10;  In this paper, we demonstrate that direct speech translation models require&#10;more data to perform well than cascaded models, and while they allow including&#10;auxiliary data through multi-task training, they are poor at exploiting such&#10;data, putting them at a severe disadvantage. As a remedy, we propose the use of&#10;end-to-end trainable models with two attention mechanisms, the first&#10;establishing source speech to source text alignments, the second modeling&#10;source to target text alignment. We show that such models naturally decompose&#10;into multi-task-trainable recognition and translation tasks and propose an&#10;attention-passing technique that alleviates error propagation issues in a&#10;previous formulation of a model with two attention stages. Our proposed model&#10;outperforms all examined baselines and is able to exploit auxiliary training&#10;data much more effectively than direct attentional models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1601.00710" label="1601.00710">
        <attvalues>
          <attvalue for="0" value="Multi-Source Neural Translation" />
          <attvalue for="1" value="  We build a multi-source machine translation model and train it to maximize&#10;the probability of a target English string given French and German sources.&#10;Using the neural encoder-decoder framework, we explore several combination&#10;methods and report up to +4.8 Bleu increases on top of a very strong&#10;attention-based neural translation model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1511.06114" label="1511.06114">
        <attvalues>
          <attvalue for="0" value="Multi-task Sequence to Sequence Learning" />
          <attvalue for="1" value="  Sequence to sequence learning has recently emerged as a new paradigm in&#10;supervised learning. To date, most of its applications focused on only one task&#10;and not much work explored this framework for multiple tasks. This paper&#10;examines three multi-task learning (MTL) settings for sequence to sequence&#10;models: (a) the oneto-many setting - where the encoder is shared between&#10;several tasks such as machine translation and syntactic parsing, (b) the&#10;many-to-one setting - useful when only the decoder can be shared, as in the&#10;case of translation and image caption generation, and (c) the many-to-many&#10;setting - where multiple encoders and decoders are shared, which is the case&#10;with unsupervised objectives and translation. Our results show that training on&#10;a small amount of parsing and image caption data can improve the translation&#10;quality between English and German by up to 1.5 BLEU points over strong&#10;single-task baselines on the WMT benchmarks. Furthermore, we have established a&#10;new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we&#10;reveal interesting properties of the two unsupervised learning objectives,&#10;autoencoder and skip-thought, in the MTL context: autoencoder helps less in&#10;terms of perplexities but more on BLEU scores compared to skip-thought.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1601.01073" label="1601.01073">
        <attvalues>
          <attvalue for="0" value="Multi-Way, Multilingual Neural Machine Translation with a Shared&#10;  Attention Mechanism" />
          <attvalue for="1" value="  We propose multi-way, multilingual neural machine translation. The proposed&#10;approach enables a single neural translation model to translate between&#10;multiple languages, with a number of parameters that grows only linearly with&#10;the number of languages. This is made possible by having a single attention&#10;mechanism that is shared across all language pairs. We train the proposed&#10;multi-way, multilingual model on ten language pairs from WMT'15 simultaneously&#10;and observe clear performance improvements over models trained on only one&#10;language pair. In particular, we observe that the proposed model significantly&#10;improves the translation quality of low-resource language pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.08198" label="1804.08198">
        <attvalues>
          <attvalue for="0" value="A neural interlingua for multilingual machine translation" />
          <attvalue for="1" value="  We incorporate an explicit neural interlingua into a multilingual&#10;encoder-decoder neural machine translation (NMT) architecture. We demonstrate&#10;that our model learns a language-independent representation by performing&#10;direct zero-shot translation (without using pivot translation), and by using&#10;the source sentence embeddings to create an English Yelp review classifier&#10;that, through the mediation of the neural interlingua, can also classify French&#10;and German reviews. Furthermore, we show that, despite using a smaller number&#10;of parameters than a pairwise collection of bilingual NMT models, our approach&#10;produces comparable BLEU scores for each language pair in WMT15.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.06957" label="1806.06957">
        <attvalues>
          <attvalue for="0" value="A Comparison of Transformer and Recurrent Neural Networks on&#10;  Multilingual Neural Machine Translation" />
          <attvalue for="1" value="  Recently, neural machine translation (NMT) has been extended to&#10;multilinguality, that is to handle more than one translation direction with a&#10;single system. Multilingual NMT showed competitive performance against pure&#10;bilingual systems. Notably, in low-resource settings, it proved to work&#10;effectively and efficiently, thanks to shared representation space that is&#10;forced across languages and induces a sort of transfer-learning. Furthermore,&#10;multilingual NMT enables so-called zero-shot inference across language pairs&#10;never seen at training time. Despite the increasing interest in this framework,&#10;an in-depth analysis of what a multilingual NMT model is capable of and what it&#10;is not is still missing. Motivated by this, our work (i) provides a&#10;quantitative and comparative analysis of the translations produced by&#10;bilingual, multilingual and zero-shot systems; (ii) investigates the&#10;translation quality of two of the currently dominant neural architectures in&#10;MT, which are the Recurrent and the Transformer ones; and (iii) quantitatively&#10;explores how the closeness between languages influences the zero-shot&#10;translation. Our analysis leverages multiple professional post-edits of&#10;automatic translations by several different systems and focuses both on&#10;automatic standard metrics (BLEU and TER) and on widely used error categories,&#10;which are lexical, morphology, and word order errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1806.05059" label="1806.05059">
        <attvalues>
          <attvalue for="0" value="Multilingual End-to-End Speech Recognition with A Single Transformer on&#10;  Low-Resource Languages" />
          <attvalue for="1" value="  Sequence-to-sequence attention-based models integrate an acoustic,&#10;pronunciation and language model into a single neural network, which make them&#10;very suitable for multilingual automatic speech recognition (ASR). In this&#10;paper, we are concerned with multilingual speech recognition on low-resource&#10;languages by a single Transformer, one of sequence-to-sequence attention-based&#10;models. Sub-words are employed as the multilingual modeling unit without using&#10;any pronunciation lexicon. First, we show that a single multilingual ASR&#10;Transformer performs well on low-resource languages despite of some language&#10;confusion. We then look at incorporating language information into the model by&#10;inserting the language symbol at the beginning or at the end of the original&#10;sub-words sequence under the condition of language information being known&#10;during training. Experiments on CALLHOME datasets demonstrate that the&#10;multilingual ASR Transformer with the language symbol at the end performs&#10;better and can obtain relatively 10.5\% average word error rate (WER) reduction&#10;compared to SHL-MLSTM with residual learning. We go on to show that, assuming&#10;the language information being known during training and testing, about&#10;relatively 12.4\% average WER reduction can be observed compared to SHL-MLSTM&#10;with residual learning through giving the language symbol as the sentence start&#10;token.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.08968" label="2402.08968">
        <attvalues>
          <attvalue for="0" value="GrounDial: Human-norm Grounded Safe Dialog Response Generation" />
          <attvalue for="1" value="  Current conversational AI systems based on large language models (LLMs) are&#10;known to generate unsafe responses, agreeing to offensive user input or&#10;including toxic content. Previous research aimed to alleviate the toxicity, by&#10;fine-tuning LLM with manually annotated safe dialogue histories. However, the&#10;dependency on additional tuning requires substantial costs. To remove the&#10;dependency, we propose GrounDial, where response safety is achieved by&#10;grounding responses to commonsense social rules without requiring fine-tuning.&#10;A hybrid approach of in-context learning and human-norm-guided decoding of&#10;GrounDial enables the response to be quantitatively and qualitatively safer&#10;even without additional data or tuning.&#10;" />
          <attvalue for="2" value="&#10;Recent LLM-based dialog systems generate responses with near-human naturalness.&#10;However, there have been reported a number of cases where the agent fails to generate safe responses.&#10;They often excuse problematic user input or contain offensive expressions~\cite{deng2023recent, ganguli2022red}.&#10;This potentially exposes users to misleading moral values or causes offense, threatening the versatility of AI-based dialog systems. &#10;Previous attempts for safe response generation have been dedicated to making use of exemplary safe dialogues annotated by humans, by fine-tuning~\cite{xu2021bot, kim2022prosocialdialog, ziems2022moral} or training auxiliary safety detector~\cite{liu2021dexperts}.&#10;&#10;However, the fine-tuning-based approaches have two key limitations: cost and generalizability. &#10;Firstly, they incur additional costs for collecting safe dialogs and training a large-scale LM with numerous parameters. &#10;This weakens efficiency since off-the-shelf LLMs cannot be employed directly. &#10;Secondly, there is no guarantee that regarding the model's ability to generalize to novel problematic inputs from the growing diversity within the user base. &#10;It is crucial to robustly and efficiently generate safe responses in such diverse scenarios. &#10;&#10;On the other hand, how do humans do? &#10;Humans learn not only through experiences but also through education. &#10;In other words, humans learn common sense social rules or norms explicitly from parents, teachers, books, etc, and ground their behavior to those rules.&#10;There have been few early attempts to incorporate the human norms, namely Rules-of-Thumb (RoT), into dialog system~\cite{kim2022prosocialdialog, ziems2022moral}.&#10;They successfully improved the response safety by fine-tuning LLM to generate RoT simultaneously with response, but they did not tackle the dependency on fine-tuning.&#10;To the best of our knowledge, there has been no attempt to directly integrate RoTs into response without the need for additional fine-tuning.&#10;&#10;In this paper, we propose a novel safe response generation framework, GrounDial, which achieves the response safety by grounding response to appropriate RoT.&#10;The response is grounded to RoT through two steps: in-context learning (ICL) and human-norm-guided decoding (HGD). &#10;We demonstrate the quantitative and qualitative effectiveness of GrounDial with Blenderbot~\cite{roller2021recipes} where both response safety and RoT relevance are improved without additional training.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Social Rule Grounding, Psychology, Conversational AI Safety, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2205.12688" label="2205.12688">
        <attvalues>
          <attvalue for="0" value="ProsocialDialog: A Prosocial Backbone for Conversational Agents" />
          <attvalue for="1" value="  Most existing dialogue systems fail to respond properly to potentially unsafe&#10;user utterances by either ignoring or passively agreeing with them. To address&#10;this issue, we introduce ProsocialDialog, the first large-scale multi-turn&#10;dialogue dataset to teach conversational agents to respond to problematic&#10;content following social norms. Covering diverse unethical, problematic,&#10;biased, and toxic situations, ProsocialDialog contains responses that encourage&#10;prosocial behavior, grounded in commonsense social rules (i.e., rules-of-thumb,&#10;RoTs). Created via a human-AI collaborative framework, ProsocialDialog consists&#10;of 58K dialogues, with 331K utterances, 160K unique RoTs, and 497K dialogue&#10;safety labels accompanied by free-form rationales.&#10;  With this dataset, we introduce a dialogue safety detection module, Canary,&#10;capable of generating RoTs given conversational context, and a&#10;socially-informed dialogue agent, Prost. Empirical results show that Prost&#10;generates more socially acceptable dialogues compared to other state-of-the-art&#10;language and dialogue models in both in-domain and out-of-domain settings.&#10;Additionally, Canary effectively guides conversational agents and off-the-shelf&#10;language models to generate significantly more prosocial responses. Our work&#10;highlights the promise and importance of creating and steering conversational&#10;AI to be socially responsible.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.13637" label="2004.13637">
        <attvalues>
          <attvalue for="0" value="Recipes for building an open-domain chatbot" />
          <attvalue for="1" value="  Building open-domain chatbots is a challenging area for machine learning&#10;research. While prior work has shown that scaling neural models in the number&#10;of parameters and the size of the data they are trained on gives improved&#10;results, we show that other ingredients are important for a high-performing&#10;chatbot. Good conversation requires a number of skills that an expert&#10;conversationalist blends in a seamless way: providing engaging talking points&#10;and listening to their partners, and displaying knowledge, empathy and&#10;personality appropriately, while maintaining a consistent persona. We show that&#10;large scale models can learn these skills when given appropriate training data&#10;and choice of generation strategy. We build variants of these recipes with 90M,&#10;2.7B and 9.4B parameter models, and make our models and code publicly&#10;available. Human evaluations show our best models are superior to existing&#10;approaches in multi-turn dialogue in terms of engagingness and humanness&#10;measurements. We then discuss the limitations of this work by analyzing failure&#10;cases of our models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.14890" label="2402.14890">
        <attvalues>
          <attvalue for="0" value="Vygotsky Distance: Measure for Benchmark Task Similarity" />
          <attvalue for="1" value="  Evaluation plays a significant role in modern natural language processing.&#10;Most modern NLP benchmarks consist of arbitrary sets of tasks that neither&#10;guarantee any generalization potential for the model once applied outside the&#10;test set nor try to minimize the resource consumption needed for model&#10;evaluation. This paper presents a theoretical instrument and a practical&#10;algorithm to calculate similarity between benchmark tasks, we call this&#10;similarity measure &quot;Vygotsky distance&quot;. The core idea of this similarity&#10;measure is that it is based on relative performance of the &quot;students&quot; on a&#10;given task, rather that on the properties of the task itself. If two tasks are&#10;close to each other in terms of Vygotsky distance the models tend to have&#10;similar relative performance on them. Thus knowing Vygotsky distance between&#10;tasks one can significantly reduce the number of evaluation tasks while&#10;maintaining a high validation quality. Experiments on various benchmarks,&#10;including GLUE, SuperGLUE, CLUE, and RussianSuperGLUE, demonstrate that a vast&#10;majority of NLP benchmarks could be at least 40% smaller in terms of the tasks&#10;included. Most importantly, Vygotsky distance could also be used for the&#10;validation of new tasks thus increasing the generalization potential of the&#10;future NLP models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Increasingly large language models such as \cite{anil2023palm,xu2023wizardlm,taori2023alpaca, vicuna2023, chowdhery2022palm, bajaj2022metro, zoph2022designing, raffel2020exploring, brown2020language,devlin-etal-2019-bert} manifest a clear trend to develop large, foundational models and then fine-tune in on a variety of NLP tasks depending on the use case. To prove that the new proposed model is beating the state-of-the-art solution, one typically evaluates the model on a series of tasks on which the suggested method is to be better than the existing ones. The community is standardly using conventional benchmarks to see if a given model is superior to the previous ones. When one talks about industrial applications of natural language processing systems, benchmarking becomes even more critical since, in practice, one has to balance various aspects of the systems, such as speed, accuracy, interpretability, etc.&#10;&#10;Moreover, both in industry and academia, one is often interested in assessing the model's generalizing potential. The current approach is somewhat extensive, namely, 'the more tasks we use to evaluate, the better'. However, this is impractical and can be computationally costly. More importantly, such an approach diverts the attention of the NLP community from research into new qualitative and quantitative methods that could rigorously measure generalization potential. This paper suggests an ad-hoc approach to benchmark task similarity evaluation that could add rigour to NLP evaluation. We hope that such an assessment method will stimulate the search for broader sets of tasks that could endow the models with higher generalization capacity.&#10;&#10;We regard {a benchmark} as a set of tasks, metrics, and evaluation methodology (usually, this is some form of aggregation over the included tasks). Each task is typically a dataset consisting of several samples of input texts and target output results. To properly evaluate the model, one typically fits the model on the training subset and then calculates metrics values on the evaluation part. Finally, one uses some aggregation method to get the model's final score that reflects the evaluated approach's quality on the given benchmark. For example, one of the most popular benchmarks in the field of natural language processing (NLP) is the General Language Understanding benchmark (GLUE) \cite{wang2018glue}, which was further extended in SuperGLUE \cite{wang2019superglue}. This benchmark has analogs in other languages: CLUE \cite{xu2020clue} for Chinese, or RussianSuperGLUE \cite{shavrina2020russiansuperglue} for Russian. All of them consist of about $10$ tasks. To get the final score of the model, one typically takes the average accuracy among all the problems provided in a given benchmark.&#10;&#10;In this paper, we introduce the notion of Vygotsky distance — a measure of task similarity evaluated with respect to the relative ranking of the models on the given task. We demonstrate that most benchmarks contain up to fifty percent of the tasks that could be regarded as redundant. We show that removing these tasks from the benchmark has virtually no effect on the resulting assessment of the models' generalization capabilities. We run the experiments using evaluation results of all the NLP models on all available benchmarks provided by Papers With Code. We mainly focus on GLUE, SuperGLUE, CLUE, and RussianSuperGLUE. The most valuable contribution of the paper is that the proposed benchmark compression method could be used to evaluate new benchmarks. The further a new benchmark is from the available tasks in terms of the Vygotsky distance the more value it carries. &#10;&#10;Section \ref{sec:benchmarks_graph_representation} is devoted to the graph representation technique that we developed to represent existing benchmarks. We consider each separate task in the benchmark as the vertex. We build an edge between each pair of tasks and assign a weight to it. The weight depends on the difference in the models' performance on the two tasks. This graph allows us to analyze the entire benchmark and retrieve the most meaningful tasks. Using the formalism we have proposed, we can determine which tasks in the benchmark are almost identical in terms of model evaluation and which are not. Further theoretical development of this framework is beyond the scope of this article, but this practical mechanism of benchmark compression is described in Section~\ref{sec:benchmarks_compression}. Suppose one splits the entire benchmark into public and private non-empty disjoint subsets. In that case, they could train classifiers to predict how models compare on the private leaderboard using the results obtained on the public leaderboard. Moreover, one can also predict the exact models' scores. This way, one could implicitly estimate the amount of information about the private subset of tasks available in the public part of the benchmark. For private leaderboard predictions, we use Support Vector Machine \cite{cortes1995support}, Gaussian Process \cite{williams2006gaussian}, and Multilayer Perceptron \cite{bishop1995neural}. Equipped with the collected data, we show how one can select a small subset of tasks within every benchmark so that it would suffice to predict the information of the models on the rest of the tasks.&#10;&#10;The main contributions of this paper are as follows:&#10;\begin{itemize}&#10; \item we propose a novel way to analyze benchmarks and evaluation systems representing them as weighted undirected graphs, which allows us to retrieve non-trivial structural properties of benchmarks and assess the similarity between the tasks in a given benchmark;&#10; \item we develop a framework that allows us to evaluate and compare NLP models on a small subset of benchmark tasks with the aforementioned Ad-Hoc benchmarks compression mechanism. This method significantly reduces the resources required for the qualitative evaluation of large NLP models;&#10; \item we conduct extensive analysis of the GLUE \cite{wang2018glue}, SuperGLUE \cite{wang2019superglue}, CLUE \cite{xu2020clue}, RussianSuperGLUE \cite{shavrina2020russiansuperglue} using our approach.&#10;\item we suggest that further tasks that extend NLP benchmarks are to be analyzed in terms of Vygotsky distance to ensure that they significantly differ from the existing NLP tasks.&#10;\end{itemize} &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Generalization, Computer Science, Linguistics, Cognitive Science, Benchmark Evaluation, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2010.07261" label="2010.07261">
        <attvalues>
          <attvalue for="0" value="Learning Improvised Chatbots from Adversarial Modifications of Natural&#10;  Language Feedback" />
          <attvalue for="1" value="  The ubiquitous nature of chatbots and their interaction with users generate&#10;an enormous amount of data. Can we improve chatbots using this data? A&#10;self-feeding chatbot improves itself by asking natural language feedback when a&#10;user is dissatisfied with its response and uses this feedback as an additional&#10;training sample. However, user feedback in most cases contains extraneous&#10;sequences hindering their usefulness as a training sample. In this work, we&#10;propose a generative adversarial model that converts noisy feedback into a&#10;plausible natural response in a conversation. The generator's goal is to&#10;convert the feedback into a response that answers the user's previous utterance&#10;and to fool the discriminator which distinguishes feedback from natural&#10;responses. We show that augmenting original training data with these modified&#10;feedback responses improves the original chatbot performance from 69.94% to&#10;75.96% in ranking correct responses on the Personachat dataset, a large&#10;improvement given that the original model is already trained on 131k samples.&#10;" />
          <attvalue for="2" value="&#10;Enabling chatbots to indulge in engaging conversations requires massive datasets of human-human conversations \cite{ritter2011data,sordoni2015neural,vinyals2015neural,zhang2018personalizing,zhang2019dialogpt}.&#10;Training such dialog agents requires substantial time and effort expended in the collection of adequate number of high quality conversation samples.&#10;&#10;\cite{hancock2019learning} alleviate this problem by introducing a self-feeding chatbot which can directly learn from user interactions.&#10;This chatbot requests users to provide natural language feedback when the users are dissatisfied with its response.&#10;&#10;\cite{hancock2019learning} treat this feedback as a gold response to the wrong turn and use it as an additional training sample to improve the chatbot.&#10;&#10;Although natural language feedback is cheap to collect from a chatbot's end-users, most often, feedback cannot be used directly as a training sample since feedback is usually not the answer itself, but simply contains hints to the answer.&#10;\ref{tab:response_samples} shows some feedback text samples.&#10;Naive modification of feedback using heuristics like regular expressions would lead to generic responses that are ineffective in improving the dialog ability of chatbots \cite{li2016diversity}.&#10;Additionally, writing an exhaustive set of regular expression rules is time consuming and requires extensive analysis of the data.&#10; Annotating data to convert feedback text to natural response is also expensive and defeats the purpose of learning from feedback text.&#10;&#10;In this work, we propose a generative adversarial setup for converting such noisy feedback instances into natural, human-like responses that provide better training signals for the dialog agents.&#10;\ref{fig:interface} gives a bird's-eye view of our problem.&#10;We frame this problem as a variant of text style transfer where the generator is tasked with making the feedback resemble the optimal response to the user's previous utterance and the discriminator is a classifier that distinguishes whether a given response is feedback or natural.&#10;&#10;Our main contributions are the following:&#10;\begin{itemize}&#10; \item We introduce Feed2Resp, a text style transfer system that converts feedback to natural responses without full supervision, thus generating additional training samples (\ref{sec:feedresp}).&#10; \item We show that the training on Feed2Resp modified responses leads to improved accuracy of chatbots (\ref{sec:experiments}).&#10; Our results also reveal that training naively on feedback doesn't help when the original chatbot is already a strong model, whereas Feed2Resp also helps strong models.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Generative Modeling, Machine Learning, Computer Science, Chatbot Improvement, Linguistics, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1907.05337" label="1907.05337">
        <attvalues>
          <attvalue for="0" value="Joint Speech Recognition and Speaker Diarization via Sequence&#10;  Transduction" />
          <attvalue for="1" value="  Speech applications dealing with conversations require not only recognizing&#10;the spoken words, but also determining who spoke when. The task of assigning&#10;words to speakers is typically addressed by merging the outputs of two separate&#10;systems, namely, an automatic speech recognition (ASR) system and a speaker&#10;diarization (SD) system. The two systems are trained independently with&#10;different objective functions. Often the SD systems operate directly on the&#10;acoustics and are not constrained to respect word boundaries and this&#10;deficiency is overcome in an ad hoc manner. Motivated by recent advances in&#10;sequence to sequence learning, we propose a novel approach to tackle the two&#10;tasks by a joint ASR and SD system using a recurrent neural network transducer.&#10;Our approach utilizes both linguistic and acoustic cues to infer speaker roles,&#10;as opposed to typical SD systems, which only use acoustic cues. We evaluated&#10;the performance of our approach on a large corpus of medical conversations&#10;between physicians and patients. Compared to a competitive conventional&#10;baseline, our approach improves word-level diarization error rate from 15.8% to&#10;2.2%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.06963" label="2205.06963">
        <attvalues>
          <attvalue for="0" value="Improved Consistency Training for Semi-Supervised Sequence-to-Sequence&#10;  ASR via Speech Chain Reconstruction and Self-Transcribing" />
          <attvalue for="1" value="  Consistency regularization has recently been applied to semi-supervised&#10;sequence-to-sequence (S2S) automatic speech recognition (ASR). This principle&#10;encourages an ASR model to output similar predictions for the same input speech&#10;with different perturbations. The existing paradigm of semi-supervised S2S ASR&#10;utilizes SpecAugment as data augmentation and requires a static teacher model&#10;to produce pseudo transcripts for untranscribed speech. However, this paradigm&#10;fails to take full advantage of consistency regularization. First, the masking&#10;operations of SpecAugment may damage the linguistic contents of the speech,&#10;thus influencing the quality of pseudo labels. Second, S2S ASR requires both&#10;input speech and prefix tokens to make the next prediction. The static prefix&#10;tokens made by the offline teacher model cannot match dynamic pseudo labels&#10;during consistency training. In this work, we propose an improved consistency&#10;training paradigm of semi-supervised S2S ASR. We utilize speech chain&#10;reconstruction as the weak augmentation to generate high-quality pseudo labels.&#10;Moreover, we demonstrate that dynamic pseudo transcripts produced by the&#10;student ASR model benefit the consistency training. Experiments on LJSpeech and&#10;LibriSpeech corpora show that compared to supervised baselines, our improved&#10;paradigm achieves a 12.2% CER improvement in the single-speaker setting and&#10;38.6% in the multi-speaker setting.&#10;" />
          <attvalue for="2" value="&#10;In recent years, sequence-to-sequence (S2S) ASR has made significant progress thanks to the advancement of deep neural networks. S2S ASR models are designed for directly converting the input speech into transcripts \cite{chan2016listen,bahdanau2016end,chorowski2015attention}. However, a large amount of transcribed speech data is essential for training S2S ASR models to achieve state-of-the-art performance. Thus, many semi-supervised learning algorithms have been proposed to efficiently train ASR models with the help of untranscribed speech \cite{zhang2020semi,weninger2020semi,masumura2020sequence,chen2021semi,kahn2020self,park2020improved,higuchi2021momentum,xiao2021contrastive}.&#10;&#10;Consistency regularization \cite{bachman2014learning} is an important principle of semi-supervised learning algorithms. This principle was originally designed for semi-supervised image classification \cite{samuli2017temporal,sajjadi2016regularization,sohn2020fixmatch,berthelot2019mixmatch} and it has recently been extended to semi-supervised S2S ASR \cite{zhang2020semi,weninger2020semi,masumura2020sequence,chen2021semi,wang2020improving}. Consistency regularization assumes that an ASR model should output similar predictions for the same input speech with various perturbations. Also, these perturbations should change the distribution of input speech without altering the corresponding transcripts\cite{sohn2020fixmatch}.&#10;In the literature, SpecAugment \cite{park2019specaugment} is commonly used to perturb speech features due to its simplicity. Its time-frequency masking plays a major role in improving the robustness of ASR \cite{wang2020improving}. However, randomly removing continuous frequency bins or temporal frames may damage the semantics of the input speech. Incomplete linguistic contents will further accumulate errors in pseudo labels and thus influence ASR performance during consistency training. &#10;&#10;Different from image classification, S2S ASR models require both input speech and prefix tokens to make the prediction at each time step. In the consistency training paradigm for S2S ASR \cite{weninger2020semi,chen2021semi,zhang2020semi,masumura2020sequence,park2020improved,kahn2020self}, a teacher model trained on transcribed speech is used to produce pseudo transcripts for untranscribed speech. The pseudo transcripts are then fed into the student ASR model as prefix tokens during consistency training. However, there exist some errors in predictions of the teacher model because of the limited training set \cite{chen2021semi}. These errors remain in the static pseudo transcripts and make pseudo labels poorer in quality. Moreover, since the static pseudo transcripts are made by original speech before consistency training, the mismatch between these transcripts and perturbed speech will further influence the student model during consistency training.&#10;&#10;This paper presents an improved consistency training paradigm of S2S ASR. In previous work, machine speech chain \cite{tjandra2017listening,tjandra2020machine} was designed to jointly train ASR and TTS by reconstructing unlabeled speech and text data. &#10;In this work, we adopt the speech chain reconstruction as a data augmentation method and focus on the FixMatch algorithm \cite{sohn2020fixmatch} which has recently been applied on S2S ASR \cite{weninger2020semi}. Our contributions are as follow:&#10;\begin{itemize}&#10;\item We propose the self-transcribing scheme where the student model serves as its own teacher. Specifically, the pseudo transcripts are produced dynamically by the student model with perturbed speech as input.&#10;\item We demonstrate that speech chain reconstruction is superior than SpecAugment as the weak augmentation for making pseudo labels.&#10;\item We conducted constrast experiments that covers most of the factors that may have an impact on the consistency training, including number of speakers, confidence threshold, ratio of unlabeled data to labeled data, data augmentation methods, and pseudo transcript generation.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Signal Processing, Speech Recognition, Consistency Regularization, Semi-Supervised Learning" />
        </attvalues>
      </node>
      <node id="2007.13876" label="2007.13876">
        <attvalues>
          <attvalue for="0" value="Semi-Supervised Learning with Data Augmentation for End-to-End ASR" />
          <attvalue for="1" value="  In this paper, we apply Semi-Supervised Learning (SSL) along with Data&#10;Augmentation (DA) for improving the accuracy of End-to-End ASR. We focus on the&#10;consistency regularization principle, which has been successfully applied to&#10;image classification tasks, and present sequence-to-sequence (seq2seq) versions&#10;of the FixMatch and Noisy Student algorithms. Specifically, we generate the&#10;pseudo labels for the unlabeled data on-the-fly with a seq2seq model after&#10;perturbing the input features with DA. We also propose soft label variants of&#10;both algorithms to cope with pseudo label errors, showing further performance&#10;improvements. We conduct SSL experiments on a conversational speech data set&#10;with 1.9kh manually transcribed training data, using only 25% of the original&#10;labels (475h labeled data). In the result, the Noisy Student algorithm with&#10;soft labels and consistency regularization achieves 10.4% word error rate (WER)&#10;reduction when adding 475h of unlabeled data, corresponding to a recovery rate&#10;of 92%. Furthermore, when iteratively adding 950h more unlabeled data, our best&#10;SSL performance is within 5% WER increase compared to using the full labeled&#10;training set (recovery rate: 78%).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.05149" label="2103.05149">
        <attvalues>
          <attvalue for="0" value="Contrastive Semi-supervised Learning for ASR" />
          <attvalue for="1" value="  Pseudo-labeling is the most adopted method for pre-training automatic speech&#10;recognition (ASR) models. However, its performance suffers from the supervised&#10;teacher model's degrading quality in low-resource setups and under domain&#10;transfer. Inspired by the successes of contrastive representation learning for&#10;computer vision and speech applications, and more recently for supervised&#10;learning of visual objects, we propose Contrastive Semi-supervised Learning&#10;(CSL). CSL eschews directly predicting teacher-generated pseudo-labels in favor&#10;of utilizing them to select positive and negative examples. In the challenging&#10;task of transcribing public social media videos, using CSL reduces the WER by&#10;8% compared to the standard Cross-Entropy pseudo-labeling (CE-PL) when 10hr of&#10;supervised data is used to annotate 75,000hr of videos. The WER reduction jumps&#10;to 19% under the ultra low-resource condition of using 1hr labels for teacher&#10;supervision. CSL generalizes much better in out-of-domain conditions, showing&#10;up to 17% WER reduction compared to the best CE-PL pre-trained model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1412.4864" label="1412.4864">
        <attvalues>
          <attvalue for="0" value="Learning with Pseudo-Ensembles" />
          <attvalue for="1" value="  We formalize the notion of a pseudo-ensemble, a (possibly infinite)&#10;collection of child models spawned from a parent model by perturbing it&#10;according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep&#10;neural network trains a pseudo-ensemble of child subnetworks generated by&#10;randomly masking nodes in the parent network. We present a novel regularizer&#10;based on making the behavior of a pseudo-ensemble robust with respect to the&#10;noise process generating it. In the fully-supervised setting, our regularizer&#10;matches the performance of dropout. But, unlike dropout, our regularizer&#10;naturally extends to the semi-supervised setting, where it produces&#10;state-of-the-art results. We provide a case study in which we transform the&#10;Recursive Neural Tensor Network of (Socher et. al, 2013) into a&#10;pseudo-ensemble, which significantly improves its performance on a real-world&#10;sentiment analysis benchmark.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.02242" label="1610.02242">
        <attvalues>
          <attvalue for="0" value="Temporal Ensembling for Semi-Supervised Learning" />
          <attvalue for="1" value="  In this paper, we present a simple and efficient method for training deep&#10;neural networks in a semi-supervised setting where only a small portion of&#10;training data is labeled. We introduce self-ensembling, where we form a&#10;consensus prediction of the unknown labels using the outputs of the&#10;network-in-training on different epochs, and most importantly, under different&#10;regularization and input augmentation conditions. This ensemble prediction can&#10;be expected to be a better predictor for the unknown labels than the output of&#10;the network at the most recent training epoch, and can thus be used as a target&#10;for training. Using our method, we set new records for two standard&#10;semi-supervised learning benchmarks, reducing the (non-augmented)&#10;classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from&#10;18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16%&#10;by enabling the standard augmentations. We additionally obtain a clear&#10;improvement in CIFAR-100 classification accuracy by using random images from&#10;the Tiny Images dataset as unlabeled extra inputs during training. Finally, we&#10;demonstrate good tolerance to incorrect labels.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.04586" label="1606.04586">
        <attvalues>
          <attvalue for="0" value="Regularization With Stochastic Transformations and Perturbations for&#10;  Deep Semi-Supervised Learning" />
          <attvalue for="1" value="  Effective convolutional neural networks are trained on large sets of labeled&#10;data. However, creating large labeled datasets is a very costly and&#10;time-consuming task. Semi-supervised learning uses unlabeled data to train a&#10;model with higher accuracy when there is a limited set of labeled data&#10;available. In this paper, we consider the problem of semi-supervised learning&#10;with convolutional neural networks. Techniques such as randomized data&#10;augmentation, dropout and random max-pooling provide better generalization and&#10;stability for classifiers that are trained using gradient descent. Multiple&#10;passes of an individual sample through the network might lead to different&#10;predictions due to the non-deterministic behavior of these techniques. We&#10;propose an unsupervised loss function that takes advantage of the stochastic&#10;nature of these methods and minimizes the difference between the predictions of&#10;multiple passes of a training sample through the network. We evaluate the&#10;proposed method on several benchmark datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.02249" label="1905.02249">
        <attvalues>
          <attvalue for="0" value="MixMatch: A Holistic Approach to Semi-Supervised Learning" />
          <attvalue for="1" value="  Semi-supervised learning has proven to be a powerful paradigm for leveraging&#10;unlabeled data to mitigate the reliance on large labeled datasets. In this&#10;work, we unify the current dominant approaches for semi-supervised learning to&#10;produce a new algorithm, MixMatch, that works by guessing low-entropy labels&#10;for data-augmented unlabeled examples and mixing labeled and unlabeled data&#10;using MixUp. We show that MixMatch obtains state-of-the-art results by a large&#10;margin across many datasets and labeled data amounts. For example, on CIFAR-10&#10;with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by&#10;a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a&#10;dramatically better accuracy-privacy trade-off for differential privacy.&#10;Finally, we perform an ablation study to tease apart which components of&#10;MixMatch are most important for its success.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.14709" label="2005.14709">
        <attvalues>
          <attvalue for="0" value="Beyond Leaderboards: A survey of methods for revealing weaknesses in&#10;  Natural Language Inference data and models" />
          <attvalue for="1" value="  Recent years have seen a growing number of publications that analyse Natural&#10;Language Inference (NLI) datasets for superficial cues, whether they undermine&#10;the complexity of the tasks underlying those datasets and how they impact those&#10;models that are optimised and evaluated on this data. This structured survey&#10;provides an overview of the evolving research area by categorising reported&#10;weaknesses in models and datasets and the methods proposed to reveal and&#10;alleviate those weaknesses for the English language. We summarise and discuss&#10;the findings and conclude with a set of recommendations for possible future&#10;research directions. We hope it will be a useful resource for researchers who&#10;propose new datasets, to have a set of tools to assess the suitability and&#10;quality of their data to evaluate various phenomena of interest, as well as&#10;those who develop novel architectures, to further understand the implications&#10;of their improvements with respect to their model's acquired capabilities.&#10;" />
          <attvalue for="2" value="&#10;Research in areas that require natural language inference (NLI) over text, such as Recognizing Textual Entailment (RTE) \cite{Dagan2006TheChallenge} and Machine Reading Comprehension (MRC) is advancing at an unprecedented rate. On the one hand, novel architectures \cite{Vaswani2017} enable efficient unsupervised training on large corpora to obtain expressive contextualised word and sentence representations for a multitude of downstream NLP tasks \cite{Devlin2018}. On the other hand, large-scale datasets \cite{Bowman2015,rajpurkar2016squad,Williams2018} provide sufficient examples to optimise large neural models that are capable of outperforming the human baseline on multiple tasks \cite{Raffel2019ExploringTransformer,Lan2020ALBERT:Representations}.&#10;&#10;Recent work, however, has questioned the seemingly superb performance for some of the tasks. Specifically, training and evaluation data may contain exploitable superficial cues, such as syntactic constructs \cite{mccoy2019right}, specific words \cite{Poliak2018} or sentence length \cite{gururangan2018annotation} that are predictive of the expected output. After having been evaluated on data in which those cues have been removed, the performance of those models deteriorated significantly \cite{mccoy2019right,Niven2019}, showing that they are in fact relying on the existing cues rather than learning to understand meaning or perform inference.&#10;In other words, those well-performing models tend to obtain optimal performance on a particular dataset, i.e. overfitting on it, rather than generalising for the underlying task. &#10;This issue, in fact, remains concealed, if a model is compared to a human baseline by means of a single number that reports the average score on a held-out test set, which is typically the case with contemporary benchmark leaderboards.&#10;&#10;To reveal and overcome these issues mentioned above, a growing number of approaches has been proposed in the past. &#10;All those methods contribute towards a fine-grained understanding of whether the existing methodology actually evaluates the required inference capabilities, what existing models learn from available training data and, more importantly, which capabilities they still fail to acquire, thus providing targeted suggestions for future research. &#10;&#10;To make sense of this growing body of literature and help researchers new to the field to navigate it, we present a structured survey of the recently proposed methods and report the trends, applications and findings. &#10;In the remainder of this paper, we first establish terminology, set the objectives and the scope of the survey and describe the data collection methodology. We then present a categorisation of the surveyed methods with their main findings, and finally discuss the arising trends and open research questions.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Dataset Analysis, Computer Science, Linguistics, Cognitive Science, Natural Language Inference, Model Evaluation" />
        </attvalues>
      </node>
      <node id="2312.00949" label="2312.00949">
        <attvalues>
          <attvalue for="0" value="Hyperparameter Optimization for Large Language Model Instruction-Tuning" />
          <attvalue for="1" value="  The fine-tuning of Large Language Models (LLMs) has enabled them to recently&#10;achieve milestones in natural language processing applications. The emergence&#10;of ever larger LLMs has paved the way for more efficient fine-tuning methods.&#10;Among these, the Low-Rank Adaptation (LoRA) method keeps most of the weights of&#10;the pre-trained LLM frozen while introducing a low-rank decomposition of the&#10;weight matrix, enabling the tuning of only a very small proportion of the&#10;network. The performance on downstream tasks of models fine-tuned with LoRA&#10;heavily relies on a set of hyperparameters including the rank of the&#10;decomposition. In this work, we investigate the choice of these hyperparameters&#10;through two main blackbox optimization (BBO) techniques. We examine the whole&#10;pipeline of performing fine-tuning and validation on a pre-trained LLM as a&#10;blackbox and efficiently explore the space of hyperparameters with the \nomad&#10;algorithm, achieving a boost in performance and human alignment of the tuned&#10;model.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large-scale Language Models (LLMs) have shown exceptional ability in language understanding and generation~\cite{OPT, text2text, LLM_MTL, gpt3}. State-of-the-art models like ChatGPT~\cite{chatgpt} and GPT-4~\cite{gpt4} have garnered a great deal of interest from the academic and industrial communities. One of the main challenges of LLMs is how to control their behavior and make them follow specific instructions given by users~\cite{instruct_gpt}. Additional fine-tuning of LLMs on a dataset of instructions is called Instruction-Tuning; &#10;this technique has become ubiquitous due to its efficiency~\cite{zhang2023instruction_survey}. However, tuning large models demands a large amount of computer power. To overcome this, a common practice is to use Parameter Efficient Fine Tuning (PEFT) methods, which modify a limited selection of parameters in a pre-trained LLM while leaving the rest unchanged~\cite{peft}. Such methods are quite sensitive to the choice of hyperparameters~\cite{hu2021lora, Valipour2022DyLoRAPT}. In this work we investigate how hyperparameter optimization can better the instruct-tuning results. &#10;&#10;Hyperparameters selection by a human in order to tune a model is a tedious task but it can significantly improve model performance. Bergstra et al.~\cite{NIPS2011_86e8f7ab} suggest that hyperparameters optimization (HPO) forms the outer loop of a learning process. Applying an algorithmic approach to automate the process in finding better hyperparameters should also bring some efficiency. A grid search algorithm is a systematic but inefficient approach that tries a finite number of hyperparameters combinations. A blackbox optimization (BBO) algorithm should be a better choice for solving HPO efficiently within a fixed computational budget. &#10;&#10;In this work we investigated how two BBO solvers implementing different types of algorithms, namely {{\sc Mads}} (a direct search algorithm implemented in {\sf NOMAD}) and TPE (a Bayesian model-based optimization algorithm implemented in NNI) behave when used to solve HPO for the Instruction-tuning of a specific LLM. We found different patterns in hyperparameter selection for these two optimizers, and assessed their effects on downstream tasks. Overall, we confirmed the necessity of careful HP selection in Instruction-tuning for performance boosting, both in downstream tasks and human preference. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Fine-Tuning Methods, Computer Science, Large Language Models, Artificial Intelligence, Mathematics, Hyperparameter Optimization" />
        </attvalues>
      </node>
      <node id="2406.15809" label="2406.15809">
        <attvalues>
          <attvalue for="0" value="LaMSUM: Creating Extractive Summaries of User Generated Content using&#10;  LLMs" />
          <attvalue for="1" value="  Large Language Models (LLMs) have demonstrated impressive performance across&#10;a wide range of NLP tasks, including summarization. LLMs inherently produce&#10;abstractive summaries by paraphrasing the original text, while the generation&#10;of extractive summaries - selecting specific subsets from the original text -&#10;remains largely unexplored. LLMs have a limited context window size,&#10;restricting the amount of data that can be processed at once. We tackle this&#10;challenge by introducing LaMSUM, a novel multi-level framework designed to&#10;generate extractive summaries from large collections of user-generated text&#10;using LLMs. LaMSUM integrates summarization with different voting methods to&#10;achieve robust summaries. Extensive evaluation using four popular LLMs (Llama&#10;3, Mixtral, Gemini, GPT-4o) demonstrates that LaMSUM outperforms&#10;state-of-the-art extractive summarization methods. Overall, this work&#10;represents one of the first attempts to achieve extractive summarization by&#10;leveraging the power of LLMs, and is likely to spark further interest within&#10;the research community.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent decades, the widespread availability of the internet has provided seamless access to online platforms to millions of people.&#10;Governments worldwide are increasingly utilizing these platforms to gather information directly from citizens -- referred as Citizen Reporting~\cite{kopackova2019citizen}.&#10;By leveraging tools such as mobile applications, web-based portals, and social media integrations, citizen reporting platforms establish a direct and efficient communication link between individuals and the relevant authorities, enabling faster issue resolution and facilitating active public participation in community improvement. &#10;Beyond immediate problem solving, real-time data gathered through these platforms contribute valuable information for urban planning and proactive measures, paving the way for more efficient and adaptive communities.&#10;Citizen reporting typically addresses topics such as community issues, environmental challenges, crime prevention, public health, and disaster response~\cite{shin2024systematic}. &#10;&#10;A notable citizen reporting platform is Safe City, developed by the nonprofit `Red Dot Foundation', where people post incidents of sexual harassment, violence and assault. Table~\ref{tab:incident} showcases some example incidents shared by users on the platform. Although the platform is accessible worldwide via website and mobile app, the majority of its users are based in India. Despite making significant strides in recent decades in areas like Science \&amp; Technology, Defense and Agriculture, India continues to grapple with the critical issue of women safety. As per the `2023 Women Peace and Security Index', India was positioned at 128th among 177 evaluated countries~\cite{indiarank}.&#10;Violence against women remains an enduring and urgent concern, taking various forms, including domestic abuse, sexual harassment, rape, dowry-related violence, honor killings, and human trafficking. &#10;One such recent incident that deeply affected the nation was the rape and murder of a 31-year-old doctor while she was on duty in August 2024~\cite{rgkar}. Several other publicized incidents include the 2020 Hathras case, where a young Dalit woman was brutally attacked and raped, ultimately resulting in her death~\cite{hathras} and the 2012 Nirbhaya incident involved the horrific gang rape and fatal assault of a 22-year-old woman in a private bus~\cite{nirbhaya}. &#10;&#10;While such horrific incidents cannot be entirely avoided through reporting alone, platforms like Safe City can play a crucial role in preventing certain cases of sexual assault. By enabling users to analyze reported incidents, assess the safety of specific locations, and make informed decisions when traveling to potential hotspots, these platforms contribute to enhanced personal safety and awareness.&#10;Similarly, the local authorities can also benefit from these platforms to assess emerging cases, identify the underlying factors and determine proactive measures for effective resolution. &#10;However, the challenge for the authorities is to navigate the high volume of information in such platforms. &#10;Manually reviewing all posts is often impractical, {necessitating a need for a summarization algorithm that can identify and select posts that are diverse as well as representative of the original data}. &#10;Additionally, platforms like Safe City often feature a curated selection of posts on their homepage to showcase their core purpose, mission, and services. {This deliberate selection also acts as a form of summarization}.&#10;&#10;Summarization algorithms are of two types: `extractive' and `abstractive'. In {extractive summarization}, the algorithm selects a subset &#10;representative of the original text \cite{xu-etal-2020-discourse, zhong-etal-2020-extractive, zhang-etal-2022-hegel, zhang-etal-2023-diffusum}. In contrast, {abstractive summarization} algorithms generate summaries that capture the essence of the original text, often paraphrasing the content~\cite{pu2023summarization}. For platforms like Safe City, extractive summarization is more suitable, as the goal is not to paraphrase the posts but to select a few that accurately capture a snapshot of the original content. {When summarizing such sensitive posts, preserving the user’s exact words is essential, making extractive summarization particularly valuable in maintaining authenticity and context}.&#10;&#10;Several extractive summarization algorithms for user generated content have been proposed in the literature, primarily for text written in English~\cite{10.1145/3462757.3466092, 10.1145/3477314.3507256, 10.1145/3397271.3401269, jia-etal-2020-neural}. &#10;However, the Safe City platform receives posts in multiple languages from across India, including numerous code-mixed entries where multiple languages are blended, such as Hinglish (a mix of Hindi and English). &#10;Such multilinguality limits the applicability of existing algorithms for extractive summarization of posts submitted to Safe City.&#10;&#10;In recent years, Large Language Models (LLMs) have demonstrated very good performance across various tasks in multilingual and code-mixed settings~\cite{rlhf1, NEURIPS2020_1457c0d6, Tang2023-jl, jin2024comprehensivesurvey}. Plus, summaries generated by LLMs showcase high coherence and are overwhelmingly preferred by human evaluators over other baseline algorithms~\cite{pu2023summarization, liu2023learning}. These prior results motivated us to investigate the utility of LLMs for extractive summarization of large volumes of user generated posts. &#10;However, we encountered two significant limitations which hinder the immediate application of LLMs for extractive summarization: (i) as generative models, LLMs perform abstractive summarization by paraphrasing rather than selecting &#10;the most relevant sentences (as shown in Figure~\ref{fig:current_state}) \cite{spectrum}; and (ii) due to the finite size of the context window, LLMs cannot handle long texts in a single input, underscoring the need for a method that allows for processing long text \cite{jin2024llmmaybelonglmselfextend}.&#10;&#10;To overcome these limitations, in this paper, we present a novel framework \ALGO (Large Language Model based Extractive SUMmarization) that integrates LLM-generated summaries with voting algorithms borrowed from Social Choice Theory \cite{brandt2016handbook}. &#10;&#10;Our judicial application of voting algorithms with a multi-level summarization framework ensures that \ALGO outperforms the state-of-the-art fine-tuned summarization models. &#10;In summary, in this work, we make the following contributions:&#10;\begin{itemize}&#10; \item We propose a novel framework \ALGO which can produce extractive summaries from large (having $&gt;$30K tokens) collection of user generated content. &#10; \ALGO considers a multi-level summarization model that utilizes voting algorithms to combine LLM outputs &#10; to generate robust summaries.&#10; \item Extensive experiments demonstrates that \ALGO outperforms the state-of-the-art extractive summarization algorithms.&#10; \item We apply \ALGO to user posts on the Safe City platform and develop a companion website that offers citizens and authorities a quick, comprehensive summary of harassment incidents occurring in localities across India.&#10;\end{itemize}&#10;&#10;To our knowledge, this is the first work to implement extractive summarization of large collection of user-generated texts using LLMs by combining summarization with voting algorithms. At the same time, we demonstrate the effectiveness of such algorithms to facilitate data-driven decision-making promoting safer communities by providing actionable insights into reported incidents. &#10;We believe this work can spawn further research in this direction, and to enable it, we are making the dataset, including the human-annotated gold standard summaries, available upon request. Code is available at &#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Text Summarization, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1910.14142" label="1910.14142">
        <attvalues>
          <attvalue for="0" value="Discourse-Aware Neural Extractive Text Summarization" />
          <attvalue for="1" value="  Recently BERT has been adopted for document encoding in state-of-the-art text&#10;summarization models. However, sentence-based extractive models often result in&#10;redundant or uninformative phrases in the extracted summaries. Also, long-range&#10;dependencies throughout a document are not well captured by BERT, which is&#10;pre-trained on sentence pairs instead of documents. To address these issues, we&#10;present a discourse-aware neural summarization model - DiscoBert. DiscoBert&#10;extracts sub-sentential discourse units (instead of sentences) as candidates&#10;for extractive selection on a finer granularity. To capture the long-range&#10;dependencies among discourse units, structural discourse graphs are constructed&#10;based on RST trees and coreference mentions, encoded with Graph Convolutional&#10;Networks. Experiments show that the proposed model outperforms state-of-the-art&#10;methods by a significant margin on popular summarization benchmarks compared to&#10;other BERT-base models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.09558" label="2309.09558">
        <attvalues>
          <attvalue for="0" value="Summarization is (Almost) Dead" />
          <attvalue for="1" value="  How well can large language models (LLMs) generate summaries? We develop new&#10;datasets and conduct human evaluation experiments to evaluate the zero-shot&#10;generation capability of LLMs across five distinct summarization tasks. Our&#10;findings indicate a clear preference among human evaluators for LLM-generated&#10;summaries over human-written summaries and summaries generated by fine-tuned&#10;models. Specifically, LLM-generated summaries exhibit better factual&#10;consistency and fewer instances of extrinsic hallucinations. Due to the&#10;satisfactory performance of LLMs in summarization tasks (even surpassing the&#10;benchmark of reference summaries), we believe that most conventional works in&#10;the field of text summarization are no longer necessary in the era of LLMs.&#10;However, we recognize that there are still some directions worth exploring,&#10;such as the creation of novel datasets with higher quality and more reliable&#10;evaluation methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.13978" label="2005.13978">
        <attvalues>
          <attvalue for="0" value="Variational Neural Machine Translation with Normalizing Flows" />
          <attvalue for="1" value="  Variational Neural Machine Translation (VNMT) is an attractive framework for&#10;modeling the generation of target translations, conditioned not only on the&#10;source sentence but also on some latent random variables. The latent variable&#10;modeling may introduce useful statistical dependencies that can improve&#10;translation accuracy. Unfortunately, learning informative latent variables is&#10;non-trivial, as the latent space can be prohibitively large, and the latent&#10;codes are prone to be ignored by many translation models at training time.&#10;Previous works impose strong assumptions on the distribution of the latent code&#10;and limit the choice of the NMT architecture. In this paper, we propose to&#10;apply the VNMT framework to the state-of-the-art Transformer and introduce a&#10;more flexible approximate posterior based on normalizing flows. We demonstrate&#10;the efficacy of our proposal under both in-domain and out-of-domain conditions,&#10;significantly outperforming strong baselines.&#10;" />
          <attvalue for="2" value="&#10;&#10;Translation is inherently ambiguous. For a given source sentence, there can be multiple plausible translations due to the author's stylistic preference, domain, and other factors. On the one hand, the introduction of neural machine translation (NMT) has significantly advanced the field \cite{bahdanau+al-2014-nmt}, continually producing state-of-the-art translation accuracy. On the other hand, the existing framework provides no explicit mechanisms to account for translation ambiguity. &#10;&#10;Recently, there has been a growing interest in latent-variable NMT (LV-NMT) that seeks to incorporate latent random variables into NMT to account for the ambiguities mentioned above. For instance, \newcite{zhang-etal-2016-variational-neural} incorporated latent codes to capture underlying global semantics of source sentences into NMT, while \newcite{SuWXLHZ18} proposed fine-grained latent codes at the word level. The learned codes, while not straightforward to analyze linguistically, are shown empirically to improve accuracy. Nevertheless, the introduction of latent random variables complicates the parameter estimation of these models, as it now involves intractable inference. In practice, prior work resorted to imposing strong assumptions on the latent code distribution, potentially compromising accuracy.&#10;&#10;In this paper, we focus on improving Variational NMT (VNMT) \cite{zhang-etal-2016-variational-neural}: a family of LV-NMT models that relies on the amortized variational method \cite{DBLP:journals/corr/KingmaW13} for inference. Our contributions are twofold. (1) We employ variational distributions based on normalizing flows \cite{rezende15}, instead of uni-modal Gaussian. Normalizing flows can yield complex distributions that may better match the latent code's true posterior. (2) We employ the Transformer architecture \cite{transformer}, including Transformer-Big, as our VNMT's generator network. We observed that the generator networks of most VNMT models belong to the RNN family that are relatively less powerful as a translation model than the Transformer. &#10;&#10;We demonstrate the efficacy of our proposal on the German-English IWSLT'14 and English-German WMT'18 tasks, giving considerable improvements over strong non-latent Transformer baselines, and moderate improvements over Gaussian models. We further show that gains generalize to an out-of-domain condition and a simulated bimodal data condition. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Deep Learning Architectures, Latent Variable Modeling, Mathematics, Statistics, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1505.05770" label="1505.05770">
        <attvalues>
          <attvalue for="0" value="Variational Inference with Normalizing Flows" />
          <attvalue for="1" value="  The choice of approximate posterior distribution is one of the core problems&#10;in variational inference. Most applications of variational inference employ&#10;simple families of posterior approximations in order to allow for efficient&#10;inference, focusing on mean-field or other simple structured approximations.&#10;This restriction has a significant impact on the quality of inferences made&#10;using variational methods. We introduce a new approach for specifying flexible,&#10;arbitrarily complex and scalable approximate posterior distributions. Our&#10;approximations are distributions constructed through a normalizing flow,&#10;whereby a simple initial density is transformed into a more complex one by&#10;applying a sequence of invertible transformations until a desired level of&#10;complexity is attained. We use this view of normalizing flows to develop&#10;categories of finite and infinitesimal flows and provide a unified view of&#10;approaches for constructing rich posterior approximations. We demonstrate that&#10;the theoretical advantages of having posteriors that better match the true&#10;posterior, combined with the scalability of amortized variational approaches,&#10;provides a clear improvement in performance and applicability of variational&#10;inference.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.06186" label="2408.06186">
        <attvalues>
          <attvalue for="0" value="Improving Structural Diversity of Blackbox LLMs via&#10;  Chain-of-Specification Prompting" />
          <attvalue for="1" value="  The capability to generate diverse text is a key challenge facing large&#10;language models (LLMs). Thus far, diversity has been studied via metrics such&#10;as $n$-gram diversity or diversity of BERT embeddings. However, for these kinds&#10;of diversity, the user has little control over the dimensions along which&#10;diversity is considered. For example, in the poetry domain, one might desire&#10;diversity in terms of rhyme and meter, whereas in the code domain, one might&#10;desire diversity in terms of the kinds of expressions used to solve a problem.&#10;We propose a diversity metric called structural diversity, where the user&#10;provides a mapping from generated text to features capturing the kinds of&#10;diversity that they care about. In addition, we propose a novel strategy called&#10;chain-of-specification (CoS) prompting for improving diversity by first having&#10;the LLM generate a specification encoding one instance of structural features,&#10;and then prompting the LLM to generate text that satisfies these features;&#10;notably, our strategy works with blackbox LLMs. In our experiments, we show&#10;that for structural diversity in the poetry and code domains, CoS significantly&#10;improves diversity compared to several baselines.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advances in large language models (LLMs), such as ChatGPT \cite{chatgpt}, have led to significant improvements in the quality and coherence of machine-generated text. However, the diversity of the generated text remains limited, particularly in terms of capturing high-level semantic properties and stylistic variations. As a consequence, there has been a great deal of interest in techniques for improving the diversity of LLMs.&#10;&#10;Much of the existing work on diversity has focused on metrics based on $n$-grams or semantic representations such as BERT embeddings. However, in many applications, users may desire diversity along specific dimensions. For instance, users might want generated poems to be diverse in terms of the structure and content of the poem, such as imagery and language, rhyming scheme, meter, etc. Alternatively, in code generation, users may want to generate code in using a range of different paradigms (e.g., for Python, list comprehension vs. loop vs. recursion) so they can choose the fastest.&#10;&#10;To account for these forms of diversity, we assume the user has provided a feature mapping $\phi:\mathcal{X}\to\mathcal{S}$ that maps text $x\in\mathcal{X}$ to a feature vector $\phi(x)\in\mathcal{S}=\{0,1\}^d$. Then, we can measure diversity in terms of entropy of the generated text in feature space. In particular, given a number of random generations $\{x_1,...,x_k\}$, we can use the empirical entropy of the distribution $\{\phi(x_1),...,\phi(x_k)\}$. We focus on mappings $\phi$ that encode structural properties of text, such as the example structures of poems and programs given above; thus, we refer to this notion of diversity as structural diversity.&#10;&#10;The key challenge is how to design techniques capable of diverse generation according to a user-defined feature mapping. Inspired by chain-of-thought prompting~\cite{cot1}, we propose a two-step prompting strategy, which we call single specification (SS) prompting (summarized in Figure~\ref{fig:illustration}). First, we use the LLM to generate an example of structure $s\in\mathcal{S}$, which we call a specification. Second, we prompt the LLM to generate text $x\in\mathcal{X}$ that satisfies the specification $s$ (i.e., $\phi(x)=s$). This strategy isolates the source of diversity to the first step---i.e., as long as the LLM generates a diverse range of specifications $s$, then the generated text $x$ will be diverse (assuming $\phi(x)=s$ always holds). Intuitively, generating a diverse range of specifications is easier than directly generating text $x$ with diverse structure.&#10;&#10;This strategy can also be chained, where we first generate a high-level specification, then generate a low-level specification that satisfies it, and finally generate text that satisfies the low-level specification; more levels can also be included. For example, in the poetry domain, the prompts might ask for different kinds of structure, such as style, theme, imagery, etc. This process resembles chain-of-thought prompting~\cite{cot1} since it asks the LLM to derive the final generated text using multiple steps, which we refer to as chain-of-specification (CoS) prompting.&#10;&#10;We evaluate our approach on domains including poetry generation, code generation, and generating coding challenge problems. Our results demonstrate that our approach is significantly more effective at improving structural diversity compared to existing approaches for diverse generation; one exception is for models that are not instruct tuned, since our approach relies on instruction following to work well. Finally, we also provide evidence that structural diversity captures qualitatively different aspects of diversity compared with existing metrics such as $n$-gram and BERT embedding diversity, demonstrating the value of structural diversity.&#10;&#10;Example.&#10;We've provided an example of three poems generated using each our CoS algorithm and a standard random sampling strategy in Appendix~\ref{sec:examples}. In the examples generated by random sampling, even though the words are different from one poem to another, the content and structure appear very consistent across all three samples. In constrast, the poems sampled using SoC exhibit significantly different structure and content. These kinds of differences occur in all of our domains.&#10;&#10;Contributions.&#10;We propose a novel framework for studying structural diversity in text generation, where diversity is defined as the entropy of a user-defined mapping into a feature space. In addition, we propose chain-of-specification (CoS) prompting, an effective algorithm for improving structural diversity. Our experiments demonstrate that our approach can significantly improve structural diversity compared to several baselines.&#10;&#10;Related work.&#10;Several recent studies have investigated methods for quantifying and improving the diversity of text generated by LLMs. For example, \cite{diversity1} proposes a set of metrics for evaluating the diversity of generated text, including self-BLEU, distinct $n$-grams, and entropy. They also introduced a new decoding method called nucleus sampling, which aims to improve diversity by sampling from the top-p portion of the probability distribution at each step. This approach works with any model but is white box, whereas our algorithm is black box.&#10;&#10;\cite{diversity2} studies the trade-off between diversity and quality in text generation, using a combination of automatic metrics and human evaluations. They found that increasing the diversity of generated text often comes at the cost of reduced coherence and relevance. To address this issue, they proposed a new approach called DiversityGAN, which uses a generative adversarial network to generate diverse and high-quality text. This approach requires modifying the training process. To the best of our knowledge, the only black-box diversity improvement algorithm was developed and discussed in \cite{ippolito}, who suggested oversampling, clustering the samples into much fewer clusters using an approach such as K-Means, and then only taking the centroid from each cluster - we compare to this baseline in our experiments.&#10;&#10;Finally, \cite{diversity3} investigates the diversity of text generated by GPT-4~\cite{chatgpt} using metrics such as $n$-gram diversity, part-of-speech diversity, and semantic diversity, to compare the diversity of GPT-4 generated text to that of human-written text. They find that while GPT-4 generates text with high ``local'' (i.e., $n$-gram) diversity, it tends to exhibit lower ``global'' (i.e., semantic) diversity compared to human-written text. However, they do not study how to bridge this gap.&#10;&#10;More broadly, there has been work studying diversity for reasoning~\cite{naik2023diversity,zhang2024improving}, and improving diversity of recommender systems~\cite{carraro2024enhancing}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Model Diversity, Computer Science, Linguistics, Artificial Intelligence Metrics, Text Generation Strategies, Artificial Intelligence, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2404.16807" label="2404.16807">
        <attvalues>
          <attvalue for="0" value="Improving Diversity of Commonsense Generation by Large Language Models&#10;  via In-Context Learning" />
          <attvalue for="1" value="  Generative Commonsense Reasoning (GCR) requires a model to reason about a&#10;situation using commonsense knowledge, while generating coherent sentences.&#10;Although the quality of the generated sentences is crucial, the diversity of&#10;the generation is equally important because it reflects the model's ability to&#10;use a range of commonsense knowledge facts. Large Language Models (LLMs) have&#10;shown proficiency in enhancing the generation quality across various tasks&#10;through in-context learning (ICL) using given examples without the need for any&#10;fine-tuning. However, the diversity aspect in LLM outputs has not been&#10;systematically studied before. To address this, we propose a simple method that&#10;diversifies the LLM generations, while preserving their quality. Experimental&#10;results on three benchmark GCR datasets show that our method achieves an ideal&#10;balance between the quality and diversity. Moreover, the sentences generated by&#10;our proposed method can be used as training data to improve diversity in&#10;existing commonsense generators.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.11506" label="2401.11506">
        <attvalues>
          <attvalue for="0" value="Enhancing Recommendation Diversity by Re-ranking with Large Language&#10;  Models" />
          <attvalue for="1" value="  It has long been recognized that it is not enough for a Recommender System&#10;(RS) to provide recommendations based only on their relevance to users. Among&#10;many other criteria, the set of recommendations may need to be diverse.&#10;Diversity is one way of handling recommendation uncertainty and ensuring that&#10;recommendations offer users a meaningful choice. The literature reports many&#10;ways of measuring diversity and improving the diversity of a set of&#10;recommendations, most notably by re-ranking and selecting from a larger set of&#10;candidate recommendations. Driven by promising insights from the literature on&#10;how to incorporate versatile Large Language Models (LLMs) into the RS pipeline,&#10;in this paper we show how LLMs can be used for diversity re-ranking.&#10;  We begin with an informal study that verifies that LLMs can be used for&#10;re-ranking tasks and do have some understanding of the concept of item&#10;diversity. Then, we design a more rigorous methodology where LLMs are prompted&#10;to generate a diverse ranking from a candidate ranking using various prompt&#10;templates with different re-ranking instructions in a zero-shot fashion. We&#10;conduct comprehensive experiments testing state-of-the-art LLMs from the GPT&#10;and Llama families. We compare their re-ranking capabilities with random&#10;re-ranking and various traditional re-ranking methods from the literature. We&#10;open-source the code of our experiments for reproducibility. Our findings&#10;suggest that the trade-offs (in terms of performance and costs, among others)&#10;of LLM-based re-rankers are superior to those of random re-rankers but, as yet,&#10;inferior to the ones of traditional re-rankers. However, the LLM approach is&#10;promising. LLMs exhibit improved performance on many natural language&#10;processing and recommendation tasks and lower inference costs. Given these&#10;trends, we can expect LLM-based re-ranking to become more competitive soon.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.09313" label="2306.09313">
        <attvalues>
          <attvalue for="0" value="Lexical Speaker Error Correction: Leveraging Language Models for Speaker&#10;  Diarization Error Correction" />
          <attvalue for="1" value="  Speaker diarization (SD) is typically used with an automatic speech&#10;recognition (ASR) system to ascribe speaker labels to recognized words. The&#10;conventional approach reconciles outputs from independently optimized ASR and&#10;SD systems, where the SD system typically uses only acoustic information to&#10;identify the speakers in the audio stream. This approach can lead to speaker&#10;errors especially around speaker turns and regions of speaker overlap. In this&#10;paper, we propose a novel second-pass speaker error correction system using&#10;lexical information, leveraging the power of modern language models (LMs). Our&#10;experiments across multiple telephony datasets show that our approach is both&#10;effective and robust. Training and tuning only on the Fisher dataset, this&#10;error correction approach leads to relative word-level diarization error rate&#10;(WDER) reductions of 15-30% on three telephony datasets: RT03-CTS, Callhome&#10;American English and held-out portions of Fisher.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech transcription systems have advanced significantly in the past decade but even with these remarkable advances, machines have difficulties understanding natural conversations with multiple speakers such as in broadcast interviews, meetings, telephone calls, videos or medical recordings. One of the first steps in understanding natural conversations is to recognize the words spoken and their corresponding speakers. Speaker Diarization (SD) is the process of determining &quot;who spoke when&quot; in a multi-speaker audio signal and is a key component in any speech transcription system. SD is used in conjunction with Automatic Speech Recognition (ASR) to assign a speaker label to each transcribed speaker turn and has widespread applications in generating meeting/interview transcripts, medical notes, automated subtitling and dubbing, downstream speaker analytics, among others (we refer to this combined system as SD-ASR in this paper). This is typically performed in multiple steps that include (1) transcribing the words using an ASR system, (2) predicting “who spoke when” using a speaker diarization (SD) system, and, finally, (3) reconciling the output of those two systems.&#10;&#10;Recent advances in SD systems are outlined in \cite{park2022review} and the independent module optimized SD systems typically consists of the following main sub-tasks: (a) segment the input audio into speech segments using a Voice activity detector (VAD), (b) generate speaker segments from the speech segments by either using a uniform window size \cite{32wang2018speaker,zhang2019fully,garcia2017speaker} or by detecting speaker turns \cite{yin2018neural,park2018multimodal,xia2022turn}, (c) extract speaker embeddings \cite{li2017deep,snyder2018x,32wang2018speaker,dawalatabad2021ecapa} for each of the speaker segments and (d) cluster the resulting speaker embeddings using clustering algorithms like Spectral Clustering \cite{32wang2018speaker}, Agglomerative Hierarchical Clustering \cite{garcia2017speaker} among others. These sub-tasks of most of the diarization systems in literature rely only acoustic information and can thus lead to speaker errors, mainly around the speaker turns. This can happen in uniform speaker segmentation as long segments very likely contain speaker turn boundaries, while short segments carry insufficient speaker information. It is also shown that detecting speaker turns using only acoustic information is also error-prone \cite{xia2022turn}. In addition to the SD errors, speakers can be attributed to the wrong words in the SD-ASR reconciliation phase due to errors in ASR word timings. Reconciliation errors can also occur in regions of speech overlap as SD can identify one of the speakers while ASR can identify words corresponding to a different speaker.&#10;&#10;Lexical information can contain complementary information which can be very useful in accurately predicting speaker turns \cite{park2018multimodal,xia2022turn}. For instance, analyzing only the written transcript of a conversation such as &quot;how are you i am good&quot;, enables us to infer that there is likely a speaker change between the utterances &quot;how are you&quot; and &quot;i am good&quot;. There have been a handful of works \cite{park2018multimodal,xia2022turn, park2020speaker,shafey2019joint,india2023language,flemotomos2019linguistically} which leverage the ASR transcripts to infuse lexical information in the SD module. In [7], lexical cues are used to estimate the speaker turns for diarization. \cite{park2020speaker} made use of turn probabilities from lexical cues in the clustering stage by enhancing the adjacency matrix. Though these approaches showed good SD improvements, these systems can still produce errors around speaker turns due to ASR and Diarization errors in overlapped speech as well are sensitive to ASR word timings as they rely on ASR timings in the diarization sub-tasks as well as in the Reconciliation phase. \cite{shafey2019joint} modeled SD and ASR jointly but is confined to 2 speakers with specific distinct roles.&#10;&#10;In this paper, we propose a Speaker Error Correction (SEC) module which can correct speaker errors at the word level without modifying the underlying ASR or the acoustic SD system. This SEC module makes use of the any of the readily available pre-trained LMs \cite{devlin2018bert,liu2019roberta,yang2019xlnet,brown2020language} to infuse the lexical knowledge to correct speaker errors while also leveraging speaker scores from the SD system to prevent over-corrections. The reliance on LMs also significantly reduces the amount of speaker labelled text data needed to train the system. Our approach has components which are modular and don’t need paired audio, text data to train while only needing a small amount of paired data for fine-tuning. This approach is also easier to integrate with existing systems than other lexical-based diarization approaches, since the first-pass acoustic SD system can be run independently of the ASR system. Using experiments across three telephony datasets, we demonstrate that the proposed system is both effective as well as capable of generalization.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Signal Processing, Speech Recognition, Natural Language Processing, Language Modeling, Speaker Diarization" />
        </attvalues>
      </node>
      <node id="1810.04719" label="1810.04719">
        <attvalues>
          <attvalue for="0" value="Fully Supervised Speaker Diarization" />
          <attvalue for="1" value="  In this paper, we propose a fully supervised speaker diarization approach,&#10;named unbounded interleaved-state recurrent neural networks (UIS-RNN). Given&#10;extracted speaker-discriminative embeddings (a.k.a. d-vectors) from input&#10;utterances, each individual speaker is modeled by a parameter-sharing RNN,&#10;while the RNN states for different speakers interleave in the time domain. This&#10;RNN is naturally integrated with a distance-dependent Chinese restaurant&#10;process (ddCRP) to accommodate an unknown number of speakers. Our system is&#10;fully supervised and is able to learn from examples where time-stamped speaker&#10;labels are annotated. We achieved a 7.6% diarization error rate on NIST SRE&#10;2000 CALLHOME, which is better than the state-of-the-art method using spectral&#10;clustering. Moreover, our method decodes in an online fashion while most&#10;state-of-the-art systems rely on offline clustering.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.10731" label="1805.10731">
        <attvalues>
          <attvalue for="0" value="Multimodal Speaker Segmentation and Diarization using Lexical and&#10;  Acoustic Cues via Sequence to Sequence Neural Networks" />
          <attvalue for="1" value="  While there has been substantial amount of work in speaker diarization&#10;recently, there are few efforts in jointly employing lexical and acoustic&#10;information for speaker segmentation. Towards that, we investigate a speaker&#10;diarization system using a sequence-to-sequence neural network trained on both&#10;lexical and acoustic features. We also propose a loss function that allows for&#10;selecting not only the speaker change points but also the best speaker at any&#10;time by allowing for different speaker groupings. We incorporate Mel Frequency&#10;Cepstral Coefficients (MFCC) as an acoustic feature alongside lexical&#10;information that are obtained from conversations from the Fisher dataset. Thus,&#10;we show that acoustics provide complementary information to the lexical&#10;modality. The experimental results show that sequence-to-sequence system&#10;trained on both word sequences and MFCC can improve on speaker diarization&#10;result compared to the system that only relies on lexical modality or the&#10;baseline MFCC-based system. In addition, we test the performance of our&#10;proposed method with Automatic Speech Recognition (ASR) transcripts. While the&#10;performance on ASR transcripts drops, the Diarization Error Rate (DER) of our&#10;proposed method still outperforms the traditional method based on Bayesian&#10;Information Criterion (BIC).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1705.02304" label="1705.02304">
        <attvalues>
          <attvalue for="0" value="Deep Speaker: an End-to-End Neural Speaker Embedding System" />
          <attvalue for="1" value="  We present Deep Speaker, a neural speaker embedding system that maps&#10;utterances to a hypersphere where speaker similarity is measured by cosine&#10;similarity. The embeddings generated by Deep Speaker can be used for many&#10;tasks, including speaker identification, verification, and clustering. We&#10;experiment with ResCNN and GRU architectures to extract the acoustic features,&#10;then mean pool to produce utterance-level speaker embeddings, and train using&#10;triplet loss based on cosine similarity. Experiments on three distinct datasets&#10;suggest that Deep Speaker outperforms a DNN-based i-vector baseline. For&#10;example, Deep Speaker reduces the verification equal error rate by 50%&#10;(relatively) and improves the identification accuracy by 60% (relatively) on a&#10;text-independent dataset. We also present results that suggest adapting from a&#10;model trained with Mandarin can improve accuracy for English speaker&#10;recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.01466" label="2104.01466">
        <attvalues>
          <attvalue for="0" value="ECAPA-TDNN Embeddings for Speaker Diarization" />
          <attvalue for="1" value="  Learning robust speaker embeddings is a crucial step in speaker diarization.&#10;Deep neural networks can accurately capture speaker discriminative&#10;characteristics and popular deep embeddings such as x-vectors are nowadays a&#10;fundamental component of modern diarization systems. Recently, some&#10;improvements over the standard TDNN architecture used for x-vectors have been&#10;proposed. The ECAPA-TDNN model, for instance, has shown impressive performance&#10;in the speaker verification domain, thanks to a carefully designed neural&#10;model.&#10;  In this work, we extend, for the first time, the use of the ECAPA-TDNN model&#10;to speaker diarization. Moreover, we improved its robustness with a powerful&#10;augmentation scheme that concatenates several contaminated versions of the same&#10;signal within the same training batch. The ECAPA-TDNN model turned out to&#10;provide robust speaker embeddings under both close-talking and distant-talking&#10;conditions. Our results on the popular AMI meeting corpus show that our system&#10;significantly outperforms recently proposed approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.07994" label="1911.07994">
        <attvalues>
          <attvalue for="0" value="Linguistically Aided Speaker Diarization Using Speaker Role Information" />
          <attvalue for="1" value="  Speaker diarization relies on the assumption that speech segments&#10;corresponding to a particular speaker are concentrated in a specific region of&#10;the speaker space; a region which represents that speaker's identity. These&#10;identities are not known a priori, so a clustering algorithm is typically&#10;employed, which is traditionally based solely on audio. Under noisy conditions,&#10;however, such an approach poses the risk of generating unreliable speaker&#10;clusters. In this work we aim to utilize linguistic information as a&#10;supplemental modality to identify the various speakers in a more robust way. We&#10;are focused on conversational scenarios where the speakers assume distinct&#10;roles and are expected to follow different linguistic patterns. This distinct&#10;linguistic variability can be exploited to help us construct the speaker&#10;identities. That way, we are able to boost the diarization performance by&#10;converting the clustering task to a classification one. The proposed method is&#10;applied in real-world dyadic psychotherapy interactions between a provider and&#10;a patient and demonstrated to show improved results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.06756" label="2004.06756">
        <attvalues>
          <attvalue for="0" value="Speaker Diarization with Lexical Information" />
          <attvalue for="1" value="  This work presents a novel approach for speaker diarization to leverage&#10;lexical information provided by automatic speech recognition. We propose a&#10;speaker diarization system that can incorporate word-level speaker turn&#10;probabilities with speaker embeddings into a speaker clustering process to&#10;improve the overall diarization accuracy. To integrate lexical and acoustic&#10;information in a comprehensive way during clustering, we introduce an adjacency&#10;matrix integration for spectral clustering. Since words and word boundary&#10;information for word-level speaker turn probability estimation are provided by&#10;a speech recognition system, our proposed method works without any human&#10;intervention for manual transcriptions. We show that the proposed method&#10;improves diarization performance on various evaluation datasets compared to the&#10;baseline diarization system using acoustic information only in speaker&#10;embeddings.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.12060" label="2102.12060">
        <attvalues>
          <attvalue for="0" value="Teach Me to Explain: A Review of Datasets for Explainable Natural&#10;  Language Processing" />
          <attvalue for="1" value="  Explainable NLP (ExNLP) has increasingly focused on collecting&#10;human-annotated textual explanations. These explanations are used downstream in&#10;three ways: as data augmentation to improve performance on a predictive task,&#10;as supervision to train models to produce explanations for their predictions,&#10;and as a ground-truth to evaluate model-generated explanations. In this review,&#10;we identify 65 datasets with three predominant classes of textual explanations&#10;(highlights, free-text, and structured), organize the literature on annotating&#10;each type, identify strengths and shortcomings of existing collection&#10;methodologies, and give recommendations for collecting ExNLP datasets in the&#10;future.&#10;" />
          <attvalue for="2" value="&#10;&#10;Interpreting supervised machine learning (ML) models is crucial for ensuring their reliability and trustworthiness in high-stakes scenarios. &#10;Models that produce justifications for their individual predictions (sometimes referred to as local explanations) can be inspected for the purposes of debugging, quantifying bias and fairness, understanding model behavior, and ascertaining robustness and privacy \cite{molnar2019}. &#10;These benefits have led to the development of &#10;datasets that contain human justifications for the true label&#10;(overviewed in Tables \ref{table:highlights_overview}--\ref{table:structured_overview}). In particular, human justifications are used for three goals: (i) to aid models with additional training supervision \cite{zaidan-etal-2007-using}, &#10;(ii) to train interpretable models that explain their own predictions \cite{camburu2018snli}, &#10;and (iii) to evaluate plausibility of model-generated explanations by measuring their agreement with human explanations \cite{deyoung-etal-2020-eraser}. &#10;&#10;Dataset collection is the most under-scrutinized component of the ML pipeline \cite{paritosh2020}---it is estimated that 92$\%$ of ML practitioners encounter data cascades, or downstream problems resulting from poor data quality \cite{sambasivan2020}. It is important to constantly evaluate data collection practices critically and standardize them \cite{bender-friedman-2018-data, Gebru2018DatasheetsFD, Paullada2020DataAI}. We expect that such examinations are particularly valuable when many related datasets are released contemporaneously and independently in a short period of time, as is the case with ExNLP datasets. &#10;&#10;This survey aims to review and summarize the literature on collecting textual explanations, &#10;highlight what has been learned to date, and give recommendations for future dataset construction. It complements other explainable AI ( XAI) surveys and critical retrospectives that focus on definitions, methods, and/or evaluation \cite{doshi2017towards, biran2017explanation, lipton2018mythos, adadi-berrada-peeking-2018, Ras2018, hoffman2018metrics, Gilpin2018ExplainingEA, Yang2019EvaluatingEW, clinciu-hastie-2019-survey, Guidotti2019ASO, miller2019explanation, Verma2020CounterfactualEF, BARREDOARRIETA202082, Murdoch22071, jacovi-goldberg-2020-towards, Burkart2021ASO}, but not on datasets. We call such datasets ExNLP datasets, because modeling them for the three goals mentioned above requires NLP techniques.&#10;Datasets and methods for explaining fact checking \cite{kotonya-toni-2020-survey} and reading comprehension \cite{Thayaparan2020ASO} have been reviewed; we are the first to review all datasets with textual explanations regardless of task, comprehensively categorize them into three distinct classes, and provide critical retrospectives and best-practice recommendations.&#10;&#10;We first define relevant ExNLP terminology (\S\ref{sec:terminology}) and overview 65 existing datasets (\S\ref{sec:survey_datasets}), accompanied with a live version of the tables as a website accepting community contributions:. We next analyze what can be learned from existing data collection methodologies. In \S\ref{sec:collecting_highlights} and \S\ref{sec:case_study_cose_esnli}, we highlight two points that we expect to be particularly important to the current ExNLP research. Specifically, \S\ref{sec:collecting_highlights} discusses the traditional process of collecting explanations by asking annotators to highlight parts of the input, and its discrepancies with evaluating model-generated highlight explanations. &#10;We also draw attention to how assumptions made for collecting free-text explanations (introduced in \S\ref{sec:terminology}) influence their modeling, and call for better documentation of explanation collection. &#10;In \S\ref{sec:case_study_cose_esnli}, we illustrate that not all template-like free-text explanations are incorrect, and call for embracing the structure of an explanation when appropriate. &#10;Unlike discussions in \S\ref{sec:collecting_highlights}--\ref{sec:case_study_cose_esnli} that are motivated by ExNLP modeling and evaluation choices, the rest of this paper reflects on relevant points from a broader NLP research. In \S\ref{sec:quality}, we present a proposal for controlling quality in explanation collection, and in \S\ref{sec:related_quality}, gather recommendations from related subfields to further reduce data artifacts by increasing diversity of collected explanations. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Dataset Collection, Textual Explanations, Natural Language Processing, Explainable NLP" />
        </attvalues>
      </node>
      <node id="2011.03870" label="2011.03870">
        <attvalues>
          <attvalue for="0" value="Explainable Automated Fact-Checking: A Survey" />
          <attvalue for="1" value="  A number of exciting advances have been made in automated fact-checking&#10;thanks to increasingly larger datasets and more powerful systems, leading to&#10;improvements in the complexity of claims which can be accurately fact-checked.&#10;However, despite these advances, there are still desirable functionalities&#10;missing from the fact-checking pipeline. In this survey, we focus on the&#10;explanation functionality -- that is fact-checking systems providing reasons&#10;for their predictions. We summarize existing methods for explaining the&#10;predictions of fact-checking systems and we explore trends in this topic.&#10;Further, we consider what makes for good explanations in this specific domain&#10;through a comparative analysis of existing fact-checking explanations against&#10;some desirable properties. Finally, we propose further research directions for&#10;generating fact-checking explanations, and describe how these may lead to&#10;improvements in the research area.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.00389" label="2010.00389">
        <attvalues>
          <attvalue for="0" value="A Survey on Explainability in Machine Reading Comprehension" />
          <attvalue for="1" value="  This paper presents a systematic review of benchmarks and approaches for&#10;explainability in Machine Reading Comprehension (MRC). We present how the&#10;representation and inference challenges evolved and the steps which were taken&#10;to tackle these challenges. We also present the evaluation methodologies to&#10;assess the performance of explainable systems. In addition, we identify&#10;persisting open research questions and highlight critical directions for future&#10;work.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.08435" label="1802.08435">
        <attvalues>
          <attvalue for="0" value="Efficient Neural Audio Synthesis" />
          <attvalue for="1" value="  Sequential models achieve state-of-the-art results in audio, visual and&#10;textual domains with respect to both estimating the data distribution and&#10;generating high-quality samples. Efficient sampling for this class of models&#10;has however remained an elusive problem. With a focus on text-to-speech&#10;synthesis, we describe a set of general techniques for reducing sampling time&#10;while maintaining high output quality. We first describe a single-layer&#10;recurrent neural network, the WaveRNN, with a dual softmax layer that matches&#10;the quality of the state-of-the-art WaveNet model. The compact form of the&#10;network makes it possible to generate 24kHz 16-bit audio 4x faster than real&#10;time on a GPU. Second, we apply a weight pruning technique to reduce the number&#10;of weights in the WaveRNN. We find that, for a constant number of parameters,&#10;large sparse networks perform better than small dense networks and this&#10;relationship holds for sparsity levels beyond 96%. The small number of weights&#10;in a Sparse WaveRNN makes it possible to sample high-fidelity audio on a mobile&#10;CPU in real time. Finally, we propose a new generation scheme based on&#10;subscaling that folds a long sequence into a batch of shorter sequences and&#10;allows one to generate multiple samples at once. The Subscale WaveRNN produces&#10;16 samples per step without loss of quality and offers an orthogonal method for&#10;increasing sampling efficiency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.10467" label="1710.10467">
        <attvalues>
          <attvalue for="0" value="Generalized End-to-End Loss for Speaker Verification" />
          <attvalue for="1" value="  In this paper, we propose a new loss function called generalized end-to-end&#10;(GE2E) loss, which makes the training of speaker verification models more&#10;efficient than our previous tuple-based end-to-end (TE2E) loss function. Unlike&#10;TE2E, the GE2E loss function updates the network in a way that emphasizes&#10;examples that are difficult to verify at each step of the training process.&#10;Additionally, the GE2E loss does not require an initial stage of example&#10;selection. With these properties, our model with the new loss function&#10;decreases speaker verification EER by more than 10%, while reducing the&#10;training time by 60% at the same time. We also introduce the MultiReader&#10;technique, which allows us to do domain adaptation - training a more accurate&#10;model that supports multiple keywords (i.e. &quot;OK Google&quot; and &quot;Hey Google&quot;) as&#10;well as multiple dialects.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.08380" label="2311.08380">
        <attvalues>
          <attvalue for="0" value="Direct Preference Optimization for Neural Machine Translation with&#10;  Minimum Bayes Risk Decoding" />
          <attvalue for="1" value="  Minimum Bayes Risk (MBR) decoding can significantly improve translation&#10;performance of Multilingual Large Language Models (MLLMs). However, MBR&#10;decoding is computationally expensive. We show how the recently developed&#10;Reinforcement Learning technique, Direct Preference Optimization (DPO), can&#10;fine-tune MLLMs to get the gains of MBR without any additional computation in&#10;inference. Our method uses only a small monolingual fine-tuning set and yields&#10;significantly improved performance on multiple NMT test sets compared to MLLMs&#10;without DPO.&#10;" />
          <attvalue for="2" value="&#10;&#10;MBR decoding \cite{kumar-byrne-2004-minimum, eikema-aziz-2022-sampling, suzgun-etal-2023-follow} is a two-pass procedure that generates multiple translation hypotheses and selects a hypothesis based on Bayesian risk. Recent work \cite{garcia2023unreasonable, suzgun-etal-2023-follow, yang-2023-thesis} has shown that MBR decoding can significantly boost the translation performance of MLLMs \cite{lin-etal-2022-shot, muennighoff-etal-2023-crosslingual, zeng2023glm130b}, outperforming greedy decoding and beam search. However, MBR decoding is expensive, both in computation and in latency.&#10;&#10;Our goal is to fine-tune a base MLLM so that it has the same single-pass decoding performance as MBR decoding. We propose&#10;a novel self-supervised fine-tuning method based on DPO~\cite{DBLP:conf/nips/RafailovSMMEF23}. Our method uses MBR decoding on an MLLM to produce a preference dataset consisting of pairs of ranked translations. The DPO algorithm is used to fine-tune the MLLM to prefer the higher-ranked translations over lower-ranked ones. MLLMs optimized for MBR preference achieve significantly better translation performance when decoded with beam search, achieving translation quality on par with MBR decoding of the original model.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Language Models, Mathematics, Reinforcement Learning, Statistics" />
        </attvalues>
      </node>
      <node id="2306.10521" label="2306.10521">
        <attvalues>
          <attvalue for="0" value="LM-VC: Zero-shot Voice Conversion via Speech Generation based on&#10;  Language Models" />
          <attvalue for="1" value="  Language model (LM) based audio generation frameworks, e.g., AudioLM, have&#10;recently achieved new state-of-the-art performance in zero-shot audio&#10;generation. In this paper, we explore the feasibility of LMs for zero-shot&#10;voice conversion. An intuitive approach is to follow AudioLM - Tokenizing&#10;speech into semantic and acoustic tokens respectively by HuBERT and&#10;SoundStream, and converting source semantic tokens to target acoustic tokens&#10;conditioned on acoustic tokens of the target speaker. However, such an approach&#10;encounters several issues: 1) the linguistic content contained in semantic&#10;tokens may get dispersed during multi-layer modeling while the lengthy speech&#10;input in the voice conversion task makes contextual learning even harder; 2)&#10;the semantic tokens still contain speaker-related information, which may be&#10;leaked to the target speech, lowering the target speaker similarity; 3) the&#10;generation diversity in the sampling of the LM can lead to unexpected outcomes&#10;during inference, leading to unnatural pronunciation and speech quality&#10;degradation. To mitigate these problems, we propose LM-VC, a two-stage language&#10;modeling approach that generates coarse acoustic tokens for recovering the&#10;source linguistic content and target speaker's timbre, and then reconstructs&#10;the fine for acoustic details as converted speech. Specifically, to enhance&#10;content preservation and facilitates better disentanglement, a masked prefix LM&#10;with a mask prediction strategy is used for coarse acoustic modeling. This&#10;model is encouraged to recover the masked content from the surrounding context&#10;and generate target speech based on the target speaker's utterance and&#10;corrupted semantic tokens. Besides, to further alleviate the sampling error in&#10;the generation, an external LM, which employs window attention to capture the&#10;local acoustic relations, is introduced to participate in the coarse acoustic&#10;modeling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.03926" label="2303.03926">
        <attvalues>
          <attvalue for="0" value="Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec&#10;  Language Modeling" />
          <attvalue for="1" value="  We propose a cross-lingual neural codec language model, VALL-E X, for&#10;cross-lingual speech synthesis. Specifically, we extend VALL-E and train a&#10;multi-lingual conditional codec language model to predict the acoustic token&#10;sequences of the target language speech by using both the source language&#10;speech and the target language text as prompts. VALL-E X inherits strong&#10;in-context learning capabilities and can be applied for zero-shot cross-lingual&#10;text-to-speech synthesis and zero-shot speech-to-speech translation tasks.&#10;Experimental results show that it can generate high-quality speech in the&#10;target language via just one speech utterance in the source language as a&#10;prompt while preserving the unseen speaker's voice, emotion, and acoustic&#10;environment. Moreover, VALL-E X effectively alleviates the foreign accent&#10;problems, which can be controlled by a language ID. Audio samples are available&#10;at \url{https://aka.ms/vallex}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.02940" label="2406.02940">
        <attvalues>
          <attvalue for="0" value="Addressing Index Collapse of Large-Codebook Speech Tokenizer with&#10;  Dual-Decoding Product-Quantized Variational Auto-Encoder" />
          <attvalue for="1" value="  VQ-VAE, as a mainstream approach of speech tokenizer, has been troubled by&#10;``index collapse'', where only a small number of codewords are activated in&#10;large codebooks. This work proposes product-quantized (PQ) VAE with more&#10;codebooks but fewer codewords to address this problem and build large-codebook&#10;speech tokenizers. It encodes speech features into multiple VQ subspaces and&#10;composes them into codewords in a larger codebook. Besides, to utilize each VQ&#10;subspace well, we also enhance PQ-VAE via a dual-decoding training strategy&#10;with the encoding and quantized sequences. The experimental results demonstrate&#10;that PQ-VAE addresses ``index collapse&quot; effectively, especially for larger&#10;codebooks. The model with the proposed training strategy further improves&#10;codebook perplexity and reconstruction quality, outperforming other&#10;multi-codebook VQ approaches. Finally, PQ-VAE demonstrates its effectiveness in&#10;language-model-based TTS, supporting higher-quality speech generation with&#10;larger codebooks.&#10;" />
          <attvalue for="2" value="&#10;&#10;The large language model (LLM) has demonstrated its powerful capability in text generation \cite{brown2020language,openai2023gpt4,touvron2023llama2}. It can auto-regressively generate expressive and diverse text sequences, especially when scaled to a larger model with more training data \cite{kaplan2020scaling}. This breakthrough has also attracted widespread attention from the speech domain, inspiring the next-gen speech generation paradigm, i.e. speech language model \cite{hassid2024textually,wu2023speechgen}. To apply LLM to the speech domain, the first thing is to convert the long speech sequence based on continuous representations into a short sequence with discrete tokens to mimic the text. Then, we can combine speech and text together for LLM training and inference, and achieve conditional speech generation, e.g. text-to-speech (TTS) \cite{betker2023better,lajszczak2024base}, voice conversion (VC) \cite{wang2023lm,kuan2023towards}, and speech-to-speech translation \cite{huang2023speech}. Hence, a high-quality speech tokenizer providing discrete tokens with sufficient speech information is the key to generating intelligible and natural speech.&#10;&#10;Vector-quantized variational autoencoder (VQ-VAE) \cite{vqvae}, as the mainstream model for discrete representation learning of speech, has been well applied in multiple tasks, including speech coding \cite{garbacea2019low}, VC \cite{wang2021vqmivc}, and TTS \cite{Du2022VQTTSHT}. It also demonstrates great potential in speech tokenization over conventional k-means-based approaches \cite{huang2023repcodec}. However, in practice, training a high-quality VQ-VAE-based speech tokenizer is also challenging due to the problem of ``index collapse&quot; \cite{huh2023improvedvqste}. The speech tokenizer usually needs a large codebook comparable to the text dictionary to represent rich speech information. VQ-VAE with a large codebook often fails to learn all codewords well, resulting in only a small number of codewords being activated in both training and inference. Some works turn to exploit multi-sequence discrete representations, e.g. MSMC-VQ \cite{guo2023msmc} and RVQ \cite{DBLP:journals/corr/abs-2210-13438,wang2023neural,zhang2023speechtokenizer}, to avoid learning large codebooks, but they cannot adapt LLMs only modeling single discrete sequences directly, introducing more challenges.&#10;&#10;To avoid this dilemma, we aim to address the ``index collapse&quot; of VQ-VAE to support the training of large-codebook speech tokenizers. We propose PQ-VAE, which indirectly learns a large codebook from multiple small codebooks by replacing VQ with product quantization (PQ) \cite{jegou2010product}. Besides, we enhance the training process with the proposed dual-decoding training strategy to pursue a higher-quality codebook. We conduct experiments on a large-scale speech dataset, first revealing the phenomenon of ``index collapse&quot; when the codebook is enlarged and then demonstrating the effectiveness of PQ-VAE in addressing this problem. Finally, we investigate the performance of PQ-VAE in TTS to show further the importance of a large-codebook speech tokenizer without ``index collapse&quot; for LLM-based speech generation.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Speech Tokenization, Computer Science, Linguistics, Signal Processing, Mathematics, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2305.13009" label="2305.13009">
        <attvalues>
          <attvalue for="0" value="Textually Pretrained Speech Language Models" />
          <attvalue for="1" value="  Speech language models (SpeechLMs) process and generate acoustic data only,&#10;without textual supervision. In this work, we propose TWIST, a method for&#10;training SpeechLMs using a warm-start from a pretrained textual language&#10;models. We show using both automatic and human evaluations that TWIST&#10;outperforms a cold-start SpeechLM across the board. We empirically analyze the&#10;effect of different model design choices such as the speech tokenizer, the&#10;pretrained textual model, and the dataset size. We find that model and dataset&#10;scale both play an important role in constructing better-performing SpeechLMs.&#10;Based on our observations, we present the largest (to the best of our&#10;knowledge) SpeechLM both in terms of number of parameters and training data. We&#10;additionally introduce two spoken versions of the StoryCloze textual benchmark&#10;to further improve model evaluation and advance future research in the field.&#10;We make speech samples, code and models publicly available:&#10;https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.13585" label="2312.13585">
        <attvalues>
          <attvalue for="0" value="Speech Translation with Large Language Models: An Industrial Practice" />
          <attvalue for="1" value="  Given the great success of large language models (LLMs) across various tasks,&#10;in this paper, we introduce LLM-ST, a novel and effective speech translation&#10;model constructed upon a pre-trained LLM. By integrating the large language&#10;model (LLM) with a speech encoder and employing multi-task instruction tuning,&#10;LLM-ST can produce accurate timestamped transcriptions and translations, even&#10;from long audio inputs. Furthermore, our findings indicate that the&#10;implementation of Chain-of-Thought (CoT) prompting can yield advantages in the&#10;context of LLM-ST. Through rigorous experimentation on English and Chinese&#10;datasets, we showcase the exceptional performance of LLM-ST, establishing a new&#10;benchmark in the field of speech translation. Demo:&#10;https://speechtranslation.github.io/llm-st/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.00169" label="2309.00169">
        <attvalues>
          <attvalue for="0" value="RepCodec: A Speech Representation Codec for Speech Tokenization" />
          <attvalue for="1" value="  With recent rapid growth of large language models (LLMs), discrete speech&#10;tokenization has played an important role for injecting speech into LLMs.&#10;However, this discretization gives rise to a loss of information, consequently&#10;impairing overall performance. To improve the performance of these discrete&#10;speech tokens, we present RepCodec, a novel speech representation codec for&#10;semantic speech tokenization. In contrast to audio codecs which reconstruct the&#10;raw audio, RepCodec learns a vector quantization codebook through&#10;reconstructing speech representations from speech encoders like HuBERT or&#10;data2vec. Together, the speech encoder, the codec encoder and the vector&#10;quantization codebook form a pipeline for converting speech waveforms into&#10;semantic tokens. The extensive experiments illustrate that RepCodec, by virtue&#10;of its enhanced information retention capacity, significantly outperforms the&#10;widely used k-means clustering approach in both speech understanding and&#10;generation. Furthermore, this superiority extends across various speech&#10;encoders and languages, affirming the robustness of RepCodec. We believe our&#10;method can facilitate large language modeling research on speech processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.08842" label="2305.08842">
        <attvalues>
          <attvalue for="0" value="Straightening Out the Straight-Through Estimator: Overcoming&#10;  Optimization Challenges in Vector Quantized Networks" />
          <attvalue for="1" value="  This work examines the challenges of training neural networks using vector&#10;quantization using straight-through estimation. We find that a primary cause of&#10;training instability is the discrepancy between the model embedding and the&#10;code-vector distribution. We identify the factors that contribute to this&#10;issue, including the codebook gradient sparsity and the asymmetric nature of&#10;the commitment loss, which leads to misaligned code-vector assignments. We&#10;propose to address this issue via affine re-parameterization of the code&#10;vectors. Additionally, we introduce an alternating optimization to reduce the&#10;gradient error introduced by the straight-through estimation. Moreover, we&#10;propose an improvement to the commitment loss to ensure better alignment&#10;between the codebook representation and the model embedding. These optimization&#10;methods improve the mathematical approximation of the straight-through&#10;estimation and, ultimately, the model performance. We demonstrate the&#10;effectiveness of our methods on several common model architectures, such as&#10;AlexNet, ResNet, and ViT, across various tasks, including image classification&#10;and generative modeling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.10542" label="1802.10542">
        <attvalues>
          <attvalue for="0" value="Memory-based Parameter Adaptation" />
          <attvalue for="1" value="  Deep neural networks have excelled on a wide range of problems, from vision&#10;to language and game playing. Neural networks very gradually incorporate&#10;information into weights as they process data, requiring very low learning&#10;rates. If the training distribution shifts, the network is slow to adapt, and&#10;when it does adapt, it typically performs badly on the training distribution&#10;before the shift. Our method, Memory-based Parameter Adaptation, stores&#10;examples in memory and then uses a context-based lookup to directly modify the&#10;weights of a neural network. Much higher learning rates can be used for this&#10;local adaptation, reneging the need for many iterations over similar data&#10;before good predictions can be made. As our method is memory-based, it&#10;alleviates several shortcomings of neural networks, such as catastrophic&#10;forgetting, fast, stable acquisition of new knowledge, learning with an&#10;imbalanced class labels, and fast learning during evaluation. We demonstrate&#10;this on a range of supervised tasks: large-scale image classification and&#10;language modelling.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.07973" label="1910.07973">
        <attvalues>
          <attvalue for="0" value="Universal Text Representation from BERT: An Empirical Study" />
          <attvalue for="1" value="  We present a systematic investigation of layer-wise BERT activations for&#10;general-purpose text representations to understand what linguistic information&#10;they capture and how transferable they are across different tasks.&#10;Sentence-level embeddings are evaluated against two state-of-the-art models on&#10;downstream and probing tasks from SentEval, while passage-level embeddings are&#10;evaluated on four question-answering (QA) datasets under a learning-to-rank&#10;problem setting. Embeddings from the pre-trained BERT model perform poorly in&#10;semantic similarity and sentence surface information probing tasks. Fine-tuning&#10;BERT on natural language inference data greatly improves the quality of the&#10;embeddings. Combining embeddings from different BERT layers can further boost&#10;performance. BERT embeddings outperform BM25 baseline significantly on factoid&#10;QA datasets at the passage level, but fail to perform better than BM25 on&#10;non-factoid datasets. For all QA datasets, there is a gap between&#10;embedding-based method and in-domain fine-tuned BERT (we report new&#10;state-of-the-art results on two datasets), which suggests deep interactions&#10;between question and answer pairs are critical for those hard tasks.&#10;" />
          <attvalue for="2" value="&#10;Universal text representations are important for many NLP tasks as modern deep learning models are becoming more and more data-hungry and computationally expensive.&#10;On one hand, most research and industry tasks face data sparsity problem due to the high cost of annotation. &#10;Universal text representations can mitigate this problem to a certain extent by performing implicit transfer learning among tasks.&#10;On the other hand, modern deep learning models with millions of parameters are expensive to train and host,&#10;while models using text representation as the building blocks can achieve similar performance with much fewer tunable parameters.&#10;The pre-computed text embeddings can also help decrease model latency dramatically at inference time. &#10;&#10;Since the introduction of pre-trained word embeddings such as word2vec \cite{Mikolov} and GloVe \cite{Pennington2014},&#10;a lot of efforts have been devoted to developing universal sentence embeddings.&#10;Initial attempts at learning sentence representation using unsupervised approaches did not yield satisfactory performance.&#10;Recent work \cite{Conneau2017} has shown that models trained in supervised fashion on datasets like Stanford Natural Language Inference (SNLI) corpus \cite{Bowman2015a}&#10;can consistently outperform unsupervised methods like SkipThought vectors \cite{Kiros2015}.&#10;More recently, Universal Sentence Encoder \cite{Cer2018} equipped with the Transformer \cite{Vaswani2017} as the encoder,&#10;co-trained on a large amount of unsupervised training data and SNLI corpus,&#10;has demonstrated surprisingly good performance with minimal amounts of supervised training data for a transfer task.&#10;&#10;BERT \cite{Devlin2018}, one of the latest models that leverage heavily on language model pre-training,&#10;has achieved state-of-the-art performance in many natural language understanding tasks ranging from sequence and sequence pair classification to question answering.&#10;The fact that pre-trained BERT can be easily fine-tuned with just one additional output layer to create a state-of-the-art model for a wide range of tasks&#10;suggests that BERT representations are potential universal text embeddings.&#10;&#10;Passages that consist of multiple sentences are coherent units of natural languages that convey information at a pragmatic or discourse level.&#10;While there are many models for generating and evaluating sentence embeddings, &#10;there hasn't been a lot of work on passage level embedding generation and evaluation.&#10;&#10;In this paper, we conducted an empirical study of layer-wise activations of BERT as general-purpose text embeddings.&#10;We want to understand to what extent does the BERT representation capture syntactic and semantic information. &#10;The sentence-level embeddings are evaluated on downstream and probing tasks using the SentEval toolkit \cite{Conneau2018},&#10;while the passage-level encodings are evaluated on four passage-level QA datasets (both factoid and non-factoid) under a learning-to-rank setting.&#10;Different methods of combining query embeddings with passage-level answer embeddings are examined. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Question Answering, Computer Science, Linguistics, Computational Linguistics, Artificial Intelligence, Natural Language Processing, Language Modeling, Text Representation" />
        </attvalues>
      </node>
      <node id="1807.06610" label="1807.06610">
        <attvalues>
          <attvalue for="0" value="Learning Noise-Invariant Representations for Robust Speech Recognition" />
          <attvalue for="1" value="  Despite rapid advances in speech recognition, current models remain brittle&#10;to superficial perturbations to their inputs. Small amounts of noise can&#10;destroy the performance of an otherwise state-of-the-art model. To harden&#10;models against background noise, practitioners often perform data augmentation,&#10;adding artificially-noised examples to the training set, carrying over the&#10;original label. In this paper, we hypothesize that a clean example and its&#10;superficially perturbed counterparts shouldn't merely map to the same class ---&#10;they should map to the same representation. We propose&#10;invariant-representation-learning (IRL): At each training iteration, for each&#10;training example,we sample a noisy counterpart. We then apply a penalty term to&#10;coerce matched representations at each layer (above some chosen layer). Our key&#10;results, demonstrated on the Librispeech dataset are the following: (i) IRL&#10;significantly reduces character error rates (CER) on both 'clean' (3.3% vs&#10;6.5%) and 'other' (11.0% vs 18.1%) test sets; (ii) on several out-of-domain&#10;noise settings (different from those seen during training), IRL's benefits are&#10;even more pronounced. Careful ablations confirm that our results are not simply&#10;due to shrinking activations at the chosen layers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.09388" label="2106.09388">
        <attvalues>
          <attvalue for="0" value="Deep Subdomain Adaptation Network for Image Classification" />
          <attvalue for="1" value="  For a target task where labeled data is unavailable, domain adaptation can&#10;transfer a learner from a different source domain. Previous deep domain&#10;adaptation methods mainly learn a global domain shift, i.e., align the global&#10;source and target distributions without considering the relationships between&#10;two subdomains within the same category of different domains, leading to&#10;unsatisfying transfer learning performance without capturing the fine-grained&#10;information. Recently, more and more researchers pay attention to Subdomain&#10;Adaptation which focuses on accurately aligning the distributions of the&#10;relevant subdomains. However, most of them are adversarial methods which&#10;contain several loss functions and converge slowly. Based on this, we present&#10;Deep Subdomain Adaptation Network (DSAN) which learns a transfer network by&#10;aligning the relevant subdomain distributions of domain-specific layer&#10;activations across different domains based on a local maximum mean discrepancy&#10;(LMMD). Our DSAN is very simple but effective which does not need adversarial&#10;training and converges fast. The adaptation can be achieved easily with most&#10;feed-forward network models by extending them with LMMD loss, which can be&#10;trained efficiently via back-propagation. Experiments demonstrate that DSAN can&#10;achieve remarkable results on both object recognition tasks and digit&#10;classification tasks. Our code will be available at:&#10;https://github.com/easezyc/deep-transfer-learning&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.10325" label="2406.10325">
        <attvalues>
          <attvalue for="0" value="Enhancing Multilingual Voice Toxicity Detection with Speech-Text&#10;  Alignment" />
          <attvalue for="1" value="  Toxicity classification for voice heavily relies on the semantic content of&#10;speech. We propose a novel framework that utilizes cross-modal learning to&#10;integrate the semantic embedding of text into a multilabel speech toxicity&#10;classifier during training. This enables us to incorporate textual information&#10;during training while still requiring only audio during inference. We evaluate&#10;this classifier on large-scale datasets with real-world characteristics to&#10;validate the effectiveness of this framework. Through ablation studies, we&#10;demonstrate that general-purpose semantic text embeddings are rich and aligned&#10;with speech for toxicity classification purposes. Conducting experiments across&#10;multiple languages at scale, we show improvements in voice toxicity&#10;classification across five languages and different toxicity categories.&#10;" />
          <attvalue for="2" value="&#10;&#10;Voice chat is a ubiquitous fixture in online social platforms like gaming and chat rooms, with an increasing number of users frequently utilizing it in a real-time interactive setting. With the growing scale and scope of these platforms, maintaining civility and safety through moderation becomes challenging. Furthermore, for social platforms operating across various geographies, languages, and cultures, moderating voice chat in multiple languages adds further complexity to this issue.&#10;Current approaches for voice toxicity classification use some form of text-based classifier that run on the transcription outputs of automatic speech recognition (ASR)~\cite{ghosh2021detoxy}.&#10;More scalable approaches involve compact classifiers, which classify toxicity directly on audio~\cite{yousefi2021audio, lin2022toxic, nada2023lightweight, adima_gupta}. These methods utilize speech encoders such as Wav2Vec 2.0~\cite{baevski2020wav2vec, nada2023lightweight, adima_gupta} and WavLM~\cite{chen2022wavlm, roblox_mtl}as the base encoder. These approaches enable scalability as this removes an expensive autoregressive ASR component of a high parameter ASR-text classifier in favor of a much smaller and faster non-autoregressive network that directly predicts the classification outputs.&#10;&#10;The audio-based speech toxicity classifiers have shown some promise on small-scale speech toxicity datasets, but are severely limited by the availability of large-scale real-world training data. DeToxy~\cite{ghosh2021detoxy} and IEMOCAP~\cite{busso2008iemocap} are public monolingual datasets that are extremely small and lack real-world characteristics given their construction. For multilingual datasets, availability is much more limited~\cite{costajussà2024mutox}.&#10;Even when researchers have access to extensive internal training datasets like Yousefi and Emmanouilidou~\cite{yousefi2021audio}, the recreation of these datasets on a comparable scale remains challenging due to the substantial human annotation needed for each individual language. &#10;This approach lacks scalability when aiming to augment training data for toxicity classification across diverse linguistic contexts.&#10;&#10;To enhance the performance of audio-based voice toxicity classifiers, various strategies are employed, including modifying the model architecture to better capture the semantic information that distinguishes toxicity. Examples of such strategies include developing a custom attention architecture~\cite{yousefi2021audio}, implementing multi-task learning~\cite{roblox_mtl}, or utilizing pre-trained speech encoders~\cite{baevski2020wav2vec,Duquenne:2023:sonar_arxiv}. One approach even involves feeding speech and text into the classifier at inference time to improve classification performance~\cite{mandal2024attentive}. These methods suggest that forcing the model to learn the semantic information content of speech helps with toxicity classification.&#10;Semantic information in speech has been shown to be learnable in a fixed embedding space that can span multiple languages and modalities, and can be incorporated into speech processing tasks~\cite{NEURIPS2021_multimodal_multilingual, blau23_interspeech}. Machine Translation architectures such as ConST~\cite{ye2022cross} have already benefited from this phenomenon to improve performance in their domain.&#10;&#10;Inspired by advances in cross-modal learning, this paper proposes a cross-modal framework for training a pre-trained speech encoder by leveraging a text-encoder on a multilabel toxicity detection task. Through this framework, we contribute the following: Firstly, we demonstrate that a strong text encoder can improve the performance of a toxicity classifier by injecting text during training. Secondly, we present empirical results on large-scale monolingual and multilingual datasets with real-world characteristics, highlighting the feasibility of this method for use in real-world applications. Thirdly, we show that toxicity classification inherently reduces to a semantic speech representation problem, by applying text injection at different layers of the network, demonstrating that learning a simple linear projection of a robust embedding space is sufficient for the model to show improvement over the baseline. Finally, we show that this framework can improve an audio-only voice toxicity classifier for multiple languages at scale without requiring text input during inference time.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Machine Learning, Multilingual Classification, Linguistics, Natural Language Processing, Psychology, Speech Toxicity Classification, Cross-Modal Learning" />
        </attvalues>
      </node>
      <node id="2110.07592" label="2110.07592">
        <attvalues>
          <attvalue for="0" value="DeToxy: A Large-Scale Multimodal Dataset for Toxicity Classification in&#10;  Spoken Utterances" />
          <attvalue for="1" value="  Toxic speech, also known as hate speech, is regarded as one of the crucial&#10;issues plaguing online social media today. Most recent work on toxic speech&#10;detection is constrained to the modality of text and written conversations with&#10;very limited work on toxicity detection from spoken utterances or using the&#10;modality of speech. In this paper, we introduce a new dataset DeToxy, the first&#10;publicly available toxicity annotated dataset for the English language. DeToxy&#10;is sourced from various openly available speech databases and consists of over&#10;2 million utterances. We believe that our dataset would act as a benchmark for&#10;the relatively new and un-explored Spoken Language Processing task of detecting&#10;toxicity from spoken utterances and boost further research in this space.&#10;Finally, we also provide strong unimodal baselines for our dataset and compare&#10;traditional two-step and E2E approaches. Our experiments show that in the case&#10;of spoken utterances, text-based approaches are largely dependent on gold&#10;human-annotated transcripts for their performance and also suffer from the&#10;problem of keyword bias. However, the presence of speech files in DeToxy helps&#10;facilitates the development of E2E speech models which alleviate both the&#10;above-stated problems by better capturing speech clues.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.05060" label="2401.05060">
        <attvalues>
          <attvalue for="0" value="MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot&#10;  Detector" />
          <attvalue for="1" value="  Research in toxicity detection in natural language processing for the speech&#10;modality (audio-based) is quite limited, particularly for languages other than&#10;English. To address these limitations and lay the groundwork for truly&#10;multilingual audio-based toxicity detection, we introduce MuTox, the first&#10;highly multilingual audio-based dataset with toxicity labels. The dataset&#10;comprises 20,000 audio utterances for English and Spanish, and 4,000 for the&#10;other 19 languages. To demonstrate the quality of this dataset, we trained the&#10;MuTox audio-based toxicity classifier, which enables zero-shot toxicity&#10;detection across a wide range of languages. This classifier outperforms&#10;existing text-based trainable classifiers by more than 1% AUC, while expanding&#10;the language coverage more than tenfold. When compared to a wordlist-based&#10;classifier that covers a similar number of languages, MuTox improves precision&#10;and recall by approximately 2.5 times. This significant improvement underscores&#10;the potential of MuTox in advancing the field of audio-based toxicity&#10;detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.10653" label="2401.10653">
        <attvalues>
          <attvalue for="0" value="Attentive Fusion: A Transformer-based Approach to Multimodal Hate Speech&#10;  Detection" />
          <attvalue for="1" value="  With the recent surge and exponential growth of social media usage,&#10;scrutinizing social media content for the presence of any hateful content is of&#10;utmost importance. Researchers have been diligently working since the past&#10;decade on distinguishing between content that promotes hatred and content that&#10;does not. Traditionally, the main focus has been on analyzing textual content.&#10;However, recent research attempts have also commenced into the identification&#10;of audio-based content. Nevertheless, studies have shown that relying solely on&#10;audio or text-based content may be ineffective, as recent upsurge indicates&#10;that individuals often employ sarcasm in their speech and writing. To overcome&#10;these challenges, we present an approach to identify whether a speech promotes&#10;hate or not utilizing both audio and textual representations. Our methodology&#10;is based on the Transformer framework that incorporates both audio and text&#10;sampling, accompanied by our very own layer called &quot;Attentive Fusion&quot;. The&#10;results of our study surpassed previous state-of-the-art techniques, achieving&#10;an impressive macro F1 score of 0.927 on the Test Set.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.02444" label="2205.02444">
        <attvalues>
          <attvalue for="0" value="Cross-modal Contrastive Learning for Speech Translation" />
          <attvalue for="1" value="  How can we learn unified representations for spoken utterances and their&#10;written text? Learning similar representations for semantically similar speech&#10;and text is important for speech translation. To this end, we propose ConST, a&#10;cross-modal contrastive learning method for end-to-end speech-to-text&#10;translation. We evaluate ConST and a variety of previous baselines on a popular&#10;benchmark MuST-C. Experiments show that the proposed ConST consistently&#10;outperforms the previous methods on, and achieves an average BLEU of 29.4. The&#10;analysis further verifies that ConST indeed closes the representation gap of&#10;different modalities -- its learned representation improves the accuracy of&#10;cross-modal speech-text retrieval from 4% to 88%. Code and models are available&#10;at https://github.com/ReneeYe/ConST.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.16173" label="2408.16173">
        <attvalues>
          <attvalue for="0" value="LLM-assisted Labeling Function Generation for Semantic Type Detection" />
          <attvalue for="1" value="  Detecting semantic types of columns in data lake tables is an important&#10;application. A key bottleneck in semantic type detection is the availability of&#10;human annotation due to the inherent complexity of data lakes. In this paper,&#10;we propose using programmatic weak supervision to assist in annotating the&#10;training data for semantic type detection by leveraging labeling functions. One&#10;challenge in this process is the difficulty of manually writing labeling&#10;functions due to the large volume and low quality of the data lake table&#10;datasets. To address this issue, we explore employing Large Language Models&#10;(LLMs) for labeling function generation and introduce several prompt&#10;engineering strategies for this purpose. We conduct experiments on real-world&#10;web table datasets. Based on the initial results, we perform extensive analysis&#10;and provide empirical insights and future directions for researchers in this&#10;field.&#10;" />
          <attvalue for="2" value="&#10;&#10;Semantic type detection is an important task in many data preparation applications, such as data cleaning, schema matching, entity resolution and data discovery~\cite{DBLP:journals/pvldb/ZhangSLHDT20,DBLP:conf/sigmod/SuharaL0ZDCT22,DBLP:conf/sigmod/Wang0HK22,DBLP:conf/deem/Wang022}.&#10;Given a table and a set of semantic labels, semantic type detection aims at identifying a type label for each column in the table so that each cell in the column has the same semantic types.&#10;This task has attracted significant attention from the database community, and many solutions based on deep learning techniques, especially pre-trained Language Models (PLMs)~\cite{DBLP:journals/pvldb/ZhangSLHDT20,DBLP:conf/sigmod/SuharaL0ZDCT22,DBLP:journals/pacmmod/MiaoW23}, have been developed to improve overall performance.&#10;&#10;Although such PLM-based solutions are effective, they have a high requirement of labeled training instances to perform fine-tuning.&#10;Due to the large scale and complex structure of data lake tables, it is rather challenging to acquire high-quality human annotation for semantic type detection~\cite{DBLP:conf/sigmod/Fan00M23}.&#10;We argue that a weak supervision approach, such as data programming~\cite{DBLP:conf/nips/RatnerSWSR16}, is a good solution to reduce the burdens of training data annotation.&#10;In the data programming paradigm, users are asked to design label functions (LF) that provide labels to a subset of data at a much lower cost rather than manually label instances one by one.&#10;Then a label model is learned to denoise and aggregate the weak labels from each LF.&#10;Finally, the label model could predict labels over unlabeled corpus to provide annotated training data. &#10;&#10;Over the past decade, significant efforts have been made in the field of data programming.&#10;\textsf{Snorkel}~\cite{DBLP:journals/pvldb/RatnerBEFWR17} proposed a probabilistic model to aggregate the user-written LFs.&#10;\textsf{Snuba}~\cite{DBLP:journals/pvldb/VarmaR18} aimed at proposing explainable LF while \textsf{Nemo}~\cite{DBLP:journals/pvldb/HsiehZR22} and \textsf{WITAN}~\cite{DBLP:journals/pvldb/DenhamLSN22} focused on the problem of interactive data programming.&#10;The recent advances in the era of Large Language Model (LLM), such as GPT-4~\cite{DBLP:journals/corr/abs-2303-08774} and LLaMA~\cite{DBLP:journals/corr/abs-2302-13971}, have shown powerful capability in various tasks in different fields.&#10;Some recent efforts~\cite{DBLP:journals/corr/abs-2311-00739,DBLP:conf/acl/ZhangYSSZ22} have been made to harness LLMs for automate the generation of LFs for NLP tasks.&#10;However, it is non-trivial to extend them to support the task of semantic type detection.&#10;Compared with the tasks supported in the previous studies, semantic type detection usually has a much larger labeling space and cardinality of datasets.&#10;For example, the number of semantic labels in the Gittable~\cite{DBLP:journals/pacmmod/HulsebosDG23} and TURL~\cite{DBLP:journals/pvldb/DengSL0020} corpus is 835 and 255, respectively. &#10;Meanwhile, the task with the largest number of labels in the WRENCH benchmarking~\cite{DBLP:conf/nips/ZhangYNWYYR21} only has 18 class labels.&#10;The large number of class labels brings two extra challenges: on the one hand, it is rather difficult for users to manually write enough LF for each class; on the other hand, it brings new challenges for the scalability of label model such as \textsf{Snorkel} to handle such large number of LFs and seeding instances for weak supervision.&#10;&#10;In this paper, we propose an end-to-end framework to conduct weakly supervision to generate LFs for semantic type detection with the help of LLM.&#10;We systematically explore the strategies to construct LLM prompt for generating LFs given the seed instances of each label class.&#10;Specifically, we find that it is essential to include both the contents and the ground truth label of the seed instance in the prompt so as to provide sufficient context for LLM to produce effective LFs.&#10;To improve the scalability of \textsf{Snorkel} for semantic type detection, we develop a stacked label model to split the label space and allow the sub-models to run in parallel. &#10;We conduct experiments on widely-used tabular datasets and evaluate both the quality of the generated LFs and the effect of training end models with the datasets annotated by such LFs.&#10;Finally, we make an in-depth analysis of the preliminary results and provide some directions for the future work.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Data Lake Management, Linguistics, Artificial Intelligence, Information Science, Weak Supervision Methods" />
        </attvalues>
      </node>
      <node id="1911.06311" label="1911.06311">
        <attvalues>
          <attvalue for="0" value="Sato: Contextual Semantic Type Detection in Tables" />
          <attvalue for="1" value="  Detecting the semantic types of data columns in relational tables is&#10;important for various data preparation and information retrieval tasks such as&#10;data cleaning, schema matching, data discovery, and semantic search. However,&#10;existing detection approaches either perform poorly with dirty data, support&#10;only a limited number of semantic types, fail to incorporate the table context&#10;of columns or rely on large sample sizes for training data. We introduce Sato,&#10;a hybrid machine learning model to automatically detect the semantic types of&#10;columns in tables, exploiting the signals from the context as well as the&#10;column values. Sato combines a deep learning model trained on a large-scale&#10;table corpus with topic modeling and structured prediction to achieve&#10;support-weighted and macro average F1 scores of 0.925 and 0.735, respectively,&#10;exceeding the state-of-the-art performance by a significant margin. We&#10;extensively analyze the overall and per-type performance of Sato, discussing&#10;how individual modeling components, as well as feature categories, contribute&#10;to its performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.01785" label="2104.01785">
        <attvalues>
          <attvalue for="0" value="Annotating Columns with Pre-trained Language Models" />
          <attvalue for="1" value="  Inferring meta information about tables, such as column headers or&#10;relationships between columns, is an active research topic in data management&#10;as we find many tables are missing some of this information. In this paper, we&#10;study the problem of annotating table columns (i.e., predicting column types&#10;and the relationships between columns) using only information from the table&#10;itself. We develop a multi-task learning framework (called Doduo) based on&#10;pre-trained language models, which takes the entire table as input and predicts&#10;column types/relations using a single model. Experimental results show that&#10;Doduo establishes new state-of-the-art performance on two benchmarks for the&#10;column type prediction and column relation prediction tasks with up to 4.0% and&#10;11.9% improvements, respectively. We report that Doduo can already outperform&#10;the previous state-of-the-art performance with a minimal number of tokens, only&#10;8 tokens per column. We release a toolbox&#10;(https://github.com/megagonlabs/doduo) and confirm the effectiveness of Doduo&#10;on a real-world data science problem through a case study.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.04853" label="2206.04853">
        <attvalues>
          <attvalue for="0" value="Machop: an End-to-End Generalized Entity Matching Framework" />
          <attvalue for="1" value="  Real-world applications frequently seek to solve a general form of the Entity&#10;Matching (EM) problem to find associated entities. Such scenarios include&#10;matching jobs to candidates in job targeting, matching students with courses in&#10;online education, matching products with user reviews on e-commercial websites,&#10;and beyond. These tasks impose new requirements such as matching data entries&#10;with diverse formats or having a flexible and semantics-rich matching&#10;definition, which are beyond the current EM task formulation or approaches. In&#10;this paper, we introduce the problem of Generalized Entity Matching (GEM) that&#10;satisfies these practical requirements and presents an end-to-end pipeline&#10;Machop as the solution. Machop allows end-users to define new matching tasks&#10;from scratch and apply them to new domains in a step-by-step manner. Machop&#10;casts the GEM problem as sequence pair classification so as to utilize the&#10;language understanding capability of Transformers-based language models (LMs)&#10;such as BERT. Moreover, it features a novel external knowledge injection&#10;approach with structure-aware pooling methods that allow domain experts to&#10;guide the LM to focus on the key matching information thus further contributing&#10;to the overall performance. Our experiments and case studies on real-world&#10;datasets from a popular recruiting platform show a significant 17.1% gain in F1&#10;score against state-of-the-art methods along with meaningful matching results&#10;that are human-understandable.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1605.07723" label="1605.07723">
        <attvalues>
          <attvalue for="0" value="Data Programming: Creating Large Training Sets, Quickly" />
          <attvalue for="1" value="  Large labeled training sets are the critical building blocks of supervised&#10;learning methods and are key enablers of deep learning techniques. For some&#10;applications, creating labeled training sets is the most time-consuming and&#10;expensive part of applying machine learning. We therefore propose a paradigm&#10;for the programmatic creation of training sets called data programming in which&#10;users express weak supervision strategies or domain heuristics as labeling&#10;functions, which are programs that label subsets of the data, but that are&#10;noisy and may conflict. We show that by explicitly representing this training&#10;set labeling process as a generative model, we can &quot;denoise&quot; the generated&#10;training set, and establish theoretically that we can recover the parameters of&#10;these generative models in a handful of settings. We then show how to modify a&#10;discriminative loss function to make it noise-aware, and demonstrate our method&#10;over a range of discriminative models including logistic regression and LSTMs.&#10;Experimentally, on the 2014 TAC-KBP Slot Filling challenge, we show that data&#10;programming would have led to a new winning score, and also show that applying&#10;data programming to an LSTM model leads to a TAC-KBP score almost 6 F1 points&#10;over a state-of-the-art LSTM baseline (and into second place in the&#10;competition). Additionally, in initial user studies we observed that data&#10;programming may be an easier way for non-experts to create machine learning&#10;models when training data is limited or unavailable.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.10160" label="1711.10160">
        <attvalues>
          <attvalue for="0" value="Snorkel: Rapid Training Data Creation with Weak Supervision" />
          <attvalue for="1" value="  Labeling training data is increasingly the largest bottleneck in deploying&#10;machine learning systems. We present Snorkel, a first-of-its-kind system that&#10;enables users to train state-of-the-art models without hand labeling any&#10;training data. Instead, users write labeling functions that express arbitrary&#10;heuristics, which can have unknown accuracies and correlations. Snorkel&#10;denoises their outputs without access to ground truth by incorporating the&#10;first end-to-end implementation of our recently proposed machine learning&#10;paradigm, data programming. We present a flexible interface layer for writing&#10;labeling functions based on our experience over the past year collaborating&#10;with companies, agencies, and research labs. In a user study, subject matter&#10;experts build models 2.8x faster and increase predictive performance an average&#10;45.5% versus seven hours of hand labeling. We study the modeling tradeoffs in&#10;this new setting and propose an optimizer for automating tradeoff decisions&#10;that gives up to 1.8x speedup per pipeline execution. In two collaborations,&#10;with the U.S. Department of Veterans Affairs and the U.S. Food and Drug&#10;Administration, and on four open-source text and image data sets representative&#10;of other deployments, Snorkel provides 132% average improvements to predictive&#10;performance over prior heuristic approaches and comes within an average 3.60%&#10;of the predictive performance of large hand-curated training sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.00739" label="2311.00739">
        <attvalues>
          <attvalue for="0" value="Can Large Language Models Design Accurate Label Functions?" />
          <attvalue for="1" value="  Programmatic weak supervision methodologies facilitate the expedited labeling&#10;of extensive datasets through the use of label functions (LFs) that encapsulate&#10;heuristic data sources. Nonetheless, the creation of precise LFs necessitates&#10;domain expertise and substantial endeavors. Recent advances in pre-trained&#10;language models (PLMs) have exhibited substantial potential across diverse&#10;tasks. However, the capacity of PLMs to autonomously formulate accurate LFs&#10;remains an underexplored domain. In this research, we address this gap by&#10;introducing DataSculpt, an interactive framework that harnesses PLMs for the&#10;automated generation of LFs. Within DataSculpt, we incorporate an array of&#10;prompting techniques, instance selection strategies, and LF filtration methods&#10;to explore the expansive design landscape. Ultimately, we conduct a thorough&#10;assessment of DataSculpt's performance on 12 real-world datasets, encompassing&#10;a range of tasks. This evaluation unveils both the strengths and limitations of&#10;contemporary PLMs in LF design.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.07258" label="2106.07258">
        <attvalues>
          <attvalue for="0" value="GitTables: A Large-Scale Corpus of Relational Tables" />
          <attvalue for="1" value="  The success of deep learning has sparked interest in improving relational&#10;table tasks, like data preparation and search, with table representation models&#10;trained on large table corpora. Existing table corpora primarily contain tables&#10;extracted from HTML pages, limiting the capability to represent offline&#10;database tables. To train and evaluate high-capacity models for applications&#10;beyond the Web, we need resources with tables that resemble relational database&#10;tables. Here we introduce GitTables, a corpus of 1M relational tables extracted&#10;from GitHub. Our continuing curation aims at growing the corpus to at least 10M&#10;tables. Analyses of GitTables show that its structure, content, and topical&#10;coverage differ significantly from existing table corpora. We annotate table&#10;columns in GitTables with semantic types, hierarchical relations and&#10;descriptions from Schema.org and DBpedia. The evaluation of our annotation&#10;pipeline on the T2Dv2 benchmark illustrates that our approach provides results&#10;on par with human annotations. We present three applications of GitTables,&#10;demonstrating its value for learned semantic type detection models, schema&#10;completion methods, and benchmarks for table-to-KG matching, data search, and&#10;preparation. We make the corpus and code available at&#10;https://gittables.github.io.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.11377" label="2109.11377">
        <attvalues>
          <attvalue for="0" value="WRENCH: A Comprehensive Benchmark for Weak Supervision" />
          <attvalue for="1" value="  Recent Weak Supervision (WS) approaches have had widespread success in easing&#10;the bottleneck of labeling training data for machine learning by synthesizing&#10;labels from multiple potentially noisy supervision sources. However, proper&#10;measurement and analysis of these approaches remain a challenge. First,&#10;datasets used in existing works are often private and/or custom, limiting&#10;standardization. Second, WS datasets with the same name and base data often&#10;vary in terms of the labels and weak supervision sources used, a significant&#10;&quot;hidden&quot; source of evaluation variance. Finally, WS studies often diverge in&#10;terms of the evaluation protocol and ablations used. To address these problems,&#10;we introduce a benchmark platform, WRENCH, for thorough and standardized&#10;evaluation of WS approaches. It consists of 22 varied real-world datasets for&#10;classification and sequence tagging; a range of real, synthetic, and&#10;procedurally-generated weak supervision sources; and a modular, extensible&#10;framework for WS evaluation, including implementations for popular WS methods.&#10;We use WRENCH to conduct extensive comparisons over more than 120 method&#10;variants to demonstrate its efficacy as a benchmark platform. The code is&#10;available at https://github.com/JieyuZ2/wrench.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.13086" label="2305.13086">
        <attvalues>
          <attvalue for="0" value="LMGQS: A Large-scale Dataset for Query-focused Summarization" />
          <attvalue for="1" value="  Query-focused summarization (QFS) aims to extract or generate a summary of an&#10;input document that directly answers or is relevant to a given query. The lack&#10;of large-scale datasets in the form of documents, queries, and summaries has&#10;hindered model development in this area. In contrast, multiple large-scale&#10;high-quality datasets for generic summarization exist. We hypothesize that&#10;there is a hidden query for each summary sentence in a generic summarization&#10;annotation, and we utilize a large-scale pretrained language model to recover&#10;it. In this way, we convert four generic summarization benchmarks into a new&#10;QFS benchmark dataset, LMGQS, which consists of over 1 million&#10;document-query-summary samples. We thoroughly investigate the properties of our&#10;proposed dataset and establish baselines with state-of-the-art summarization&#10;models. By fine-tuning a language model on LMGQS, we achieve state-of-the-art&#10;zero-shot and supervised performance on multiple existing QFS benchmarks,&#10;demonstrating the high quality and diversity of LMGQS.&#10;" />
          <attvalue for="2" value="&#10;The field of generic summarization \cite{see-etal-2017-get,gehrmann-etal-2018-bottom,liu-lapata-2019-text} has made significant progress in recent years, thanks to the development of generative deep neural models \cite{NIPS2014_a14ac55a,NIPS2017_3f5ee243} and the availability of large-scale training data \cite{nallapati-etal-2016-abstractive,narayan-etal-2018-dont,zhu-etal-2021-mediasum}. However, query-focused summarization (QFS) presents a significant challenge due to the lack of data. Most of the available QFS corpora \cite{dang-2006-duc,dang2006duc,nema-etal-2017-diversity,baumel2016topic,zhong-etal-2021-qmsum} contain only a few thousand documents or less, which is insufficient for training a robust neural model.&#10;&#10;We propose a Language Model Generated Query-Focused Summarization Dataset (LMGQS) to address the lack of a large-scale QFS dataset. Human annotation for QFS typically involves generating suitable queries and then writing corresponding summaries, which is both time-consuming and expensive. Furthermore, it may necessitate a meticulous definition of the query scheme based on the domain of documents \cite{zhong-etal-2021-qmsum}. We hypothesize that, for a pair of document and summary in a generic summarization dataset, hidden queries exist that represent the information needs associated with the summary. Therefore, to efficiently scale annotation, we prompt the large-scale language model InstructGPT \cite{ouyang2022training} with documents and summaries from four generic summarization datasets to extract the hidden queries.&#10;This approach results in the LMGQS dataset, which contains over 1.1 million triplets of document, query, and summary, encompassing a wide range of document and question types.&#10;&#10;To investigate the utility of our proposed LMGQS, we finetune a pretrained language model on it. The model accepts the concatenation of the original document and generated query as input and is trained to produce the original summary. We then compare the finetuned model with various query-focused summarization models on several existing QFS benchmarks that have no overlap with LMGQS under the zero-shot setting. &#10;Empirical results demonstrate that the model finetuned on LMGQS achieves promising performance on both single-document and multi-document QFS benchmarks, surpassing strong baselines. Similarly, when utilizing LMGQS for pre-finetuning, the model achieves state-of-the-art performance in the supervised setting.&#10;&#10;In summary, our contributions are three-fold: (1) We introduce a novel framework for constructing a QFS dataset by converting existing generic summarization datasets using language models as annotators. (2) We present LMGQS, a large-scale QFS benchmark, to foster future research on QFS. (3) The model finetuned on LMGQS exhibits robust generalization capability and achieves remarkable zero-shot and supervised performance on other unseen QFS test sets.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Dataset Development, Artificial Intelligence, Query Summarization, Natural Language Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="2103.06410" label="2103.06410">
        <attvalues>
          <attvalue for="0" value="MediaSum: A Large-scale Media Interview Dataset for Dialogue&#10;  Summarization" />
          <attvalue for="1" value="  MediaSum, a large-scale media interview dataset consisting of 463.6K&#10;transcripts with abstractive summaries. To create this dataset, we collect&#10;interview transcripts from NPR and CNN and employ the overview and topic&#10;descriptions as summaries. Compared with existing public corpora for dialogue&#10;summarization, our dataset is an order of magnitude larger and contains complex&#10;multi-party conversations from multiple domains. We conduct statistical&#10;analysis to demonstrate the unique positional bias exhibited in the transcripts&#10;of televised and radioed interviews. We also show that MediaSum can be used in&#10;transfer learning to improve a model's performance on other dialogue&#10;summarization tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.08300" label="1704.08300">
        <attvalues>
          <attvalue for="0" value="Diversity driven Attention Model for Query-based Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  Abstractive summarization aims to generate a shorter version of the document&#10;covering all the salient points in a compact and coherent fashion. On the other&#10;hand, query-based summarization highlights those points that are relevant in&#10;the context of a given query. The encode-attend-decode paradigm has achieved&#10;notable success in machine translation, extractive summarization, dialog&#10;systems, etc. But it suffers from the drawback of generation of repeated&#10;phrases. In this work we propose a model for the query-based summarization task&#10;based on the encode-attend-decode paradigm with two key additions (i) a query&#10;attention model (in addition to document attention model) which learns to focus&#10;on different portions of the query at different time steps (instead of using a&#10;static representation for the query) and (ii) a new diversity based attention&#10;model which aims to alleviate the problem of repeating phrases in the summary.&#10;In order to enable the testing of this model we introduce a new query-based&#10;summarization dataset building on debatepedia. Our experiments show that with&#10;these two additions the proposed model clearly outperforms vanilla&#10;encode-attend-decode models with a gain of 28% (absolute) in ROUGE-L scores.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.05938" label="2104.05938">
        <attvalues>
          <attvalue for="0" value="QMSum: A New Benchmark for Query-based Multi-domain Meeting&#10;  Summarization" />
          <attvalue for="1" value="  Meetings are a key component of human collaboration. As increasing numbers of&#10;meetings are recorded and transcribed, meeting summaries have become essential&#10;to remind those who may or may not have attended the meetings about the key&#10;decisions made and the tasks to be completed. However, it is hard to create a&#10;single short summary that covers all the content of a long meeting involving&#10;multiple people and topics. In order to satisfy the needs of different types of&#10;users, we define a new query-based multi-domain meeting summarization task,&#10;where models have to select and summarize relevant spans of meetings in&#10;response to a query, and we introduce QMSum, a new benchmark for this task.&#10;QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple&#10;domains. Besides, we investigate a locate-then-summarize method and evaluate a&#10;set of strong summarization baselines on the task. Experimental results and&#10;manual analysis reveal that QMSum presents significant challenges in long&#10;meeting summarization for future research. Dataset is available at&#10;\url{https://github.com/Yale-LILY/QMSum}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2106.10003" label="2106.10003">
        <attvalues>
          <attvalue for="0" value="Improving Performance of Seen and Unseen Speech Style Transfer in&#10;  End-to-end Neural TTS" />
          <attvalue for="1" value="  End-to-end neural TTS training has shown improved performance in speech style&#10;transfer. However, the improvement is still limited by the training data in&#10;both target styles and speakers. Inadequate style transfer performance occurs&#10;when the trained TTS tries to transfer the speech to a target style from a new&#10;speaker with an unknown, arbitrary style. In this paper, we propose a new&#10;approach to style transfer for both seen and unseen styles, with disjoint,&#10;multi-style datasets, i.e., datasets of different styles are recorded, each&#10;individual style is by one speaker with multiple utterances. To encode the&#10;style information, we adopt an inverse autoregressive flow (IAF) structure to&#10;improve the variational inference. The whole system is optimized to minimize a&#10;weighed sum of four different loss functions: 1) a reconstruction loss to&#10;measure the distortions in both source and target reconstructions; 2) an&#10;adversarial loss to &quot;fool&quot; a well-trained discriminator; 3) a style distortion&#10;loss to measure the expected style loss after the transfer; 4) a cycle&#10;consistency loss to preserve the speaker identity of the source after the&#10;transfer. Experiments demonstrate, both objectively and subjectively, the&#10;effectiveness of the proposed approach for seen and unseen style transfer&#10;tasks. The performance of the new approach is better and more robust than those&#10;of four baseline systems of the prior art.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent advancement of end-to-end neural TTS has demonstrated that it can synthesize very natural, human-like speech~\cite{wang2017tacotron, ping2018deep, shen2018natural, fengyu2019improving, guangzhi2020generating}. The trained neural TTS models usually consist of an encoder-decoder neural network~\cite{sutskever2014sequence, bahdanau2015neural} which can map a text sequence to a sequence of speech frames. Extensions of these models have shown that speech styles (e.g., speaker identity, emotion and prosody), which are essential for expressive and diverse voice generation, can be also modelled and controlled~\cite{Ye2018Transfer, wu2019end-to-end, stanton2018predicting, skerry2018towards, liu2020expressive, lei2021fine}. As applicable scenarios of speech synthesis have rapidly developed, such as the audio reading scenario, there is a growing demands for the single-speaker, multi-style synthesis, where a person can simultaneously speak multiple styles, and yet research in this area is still in its infancy.&#10;&#10;Currently most neural TTS systems~\cite{akuzawa2018expressive, xiaochun2019learning, hsu2018hierarchical, habib2020semi-supervised, guangzhi2020fully} are modelled by using a corpus of a single expressive style. Acquiring and annotating a large set of single-speaker speech data with multiple styles for training a neural TTS is usually expensive and time consuming. It is an effective solution to use transfer learning to perform speech style transfer, which allows a speaker to learn the desired style from the data with this style of other speakers without the data of a certain style, and keeps his own timbre consistency. Recently, neural TTS model with global style tokens (GST)~\cite{wang2018style, li2021controllable} or a variational autoencoder (VAE)~\cite{zhang2019learning} has received interests for controlling and transferring speech styles. Theoretically, these models can model any complex styles in a continuous latent space, so that one can control and transfer style by manipulating the latent variables or variational inference from a reference audio.&#10;&#10;However, these researches model all speech styles into one style representation, which contains too much interfering information to be robust and interpretable, and lacks the ability to control a specific speech feature independently. When conducting style transfer, one has to transfer all styles whether desired or not, which may not fit the contexts thus hurts generalization. When conducting style control, one can hardly confirm the relationship between the styles and the coefficients of each dimension of the style representations.&#10;&#10;Recently, Bian et al.~\cite{bian2019multi} introduce a multi-reference encoder to GST~\cite{wang2018style} and adopt an intercross training scheme, which together ensure that each sub-encoder of the multi-reference encoder independently disentangles and controls a specific style. They show successful style transfer on a multi-style data scenario. However, their intercross training scheme does not guarantee each combination of style classes is seen during training, causing a missed opportunity to learn disentangled representations of styles and sub-optimal results on disjoint, multi-style datasets.&#10;&#10;In order to improve style transfer for the combined style that is underrepresented in the dataset, ~\cite{whitehill2020multi} proposes an adversarial cycle consistency training scheme with paired and unpaired triplets to ensure the use of information from all style classes. Unlike intercross training, the scheme sweeps across all combinations of style classes via paired and unpaired triplets. This provides disentanglement of multiple style classes, enabling the model to transfer style in a more faithful manner than existing methods.&#10;&#10;Though ~\cite{whitehill2020multi} improves performance of style transfer, it suffers a limitation, similar to ~\cite{bian2019multi}, that can only transfer the style seen during training, and is inadequate to transfer the speech to a target style from a new speaker with an unknown, arbitrary style, thus narrowing down the applicable scenarios of neural TTS systems. In addition, recording training samples for new style (e.g., customer-service style and poetry style) is challenging and labor-intensive, transferring style from one dataset to another (i.e., disjoint, multi-style datasets) is an appealing feature for TTS systems. Therefore, unseen style transfer on disjoint, multi-style datasets needs to be improved.&#10;&#10;In this paper, we propose a new approach to style transfer for both seen and unseen styles. As a result, it tackles the single-speaker, multi-style synthesis in a more flexible and convenient manner, and further meets the needs of audio reading scenario. The main contributions of this paper are summarized as follows:&#10;&#10;\begin{itemize}&#10; \item To facilitate seen and unseen style transfer in end-to-end neural TTS, we first adopt an inverse autoregressive flow (IAF) structure~\cite{kingma2016improving} to improve the style representation, and then propose four different loss functions to together make sure the seen and unseen style transfer: 1) using a reconstruction loss to measure the distortions in both source and target reconstructions; 2) injecting an adversarial loss to ``fool&quot; a well-trained discriminator; 3) introducing a style distortion loss to measure the expected style loss after the transfer; 4) incorporating a cycle consistency loss to preserve the speaker identity of the source after the transfer. With the proposed approach, we can transfer the speech to a target style from a new speaker with an unknown, arbitrary style, which does not even need to be seen during training.&#10; \item The proposed seen and unseen style transfer scheme is used as a data augmentation method to generate a single-speaker, multi-style speech data, which is significant for various speech tasks, such as multi-style TTS and voice conversion.&#10; \item Our approach outperforms the four prior art baselines and the improvement is confirmed in both subjective and objective tests. The resultant performance of seen and unseen style transfer is better and more robust than the counterpart in the prior art. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Text-to-Speech, Computer Science, Linguistics, Speech Style Transfer, Signal Processing, Mathematics, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="1606.04934" label="1606.04934">
        <attvalues>
          <attvalue for="0" value="Improving Variational Inference with Inverse Autoregressive Flow" />
          <attvalue for="1" value="  The framework of normalizing flows provides a general strategy for flexible&#10;variational inference of posteriors over latent variables. We propose a new&#10;type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast&#10;to earlier published flows, scales well to high-dimensional latent spaces. The&#10;proposed flow consists of a chain of invertible transformations, where each&#10;transformation is based on an autoregressive neural network. In experiments, we&#10;show that IAF significantly improves upon diagonal Gaussian approximate&#10;posteriors. In addition, we demonstrate that a novel type of variational&#10;autoencoder, coupled with IAF, is competitive with neural autoregressive models&#10;in terms of attained log-likelihood on natural images, while allowing&#10;significantly faster synthesis.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.00968" label="2211.00968">
        <attvalues>
          <attvalue for="0" value="Internal Language Model Estimation based Adaptive Language Model Fusion&#10;  for Domain Adaptation" />
          <attvalue for="1" value="  ASR model deployment environment is ever-changing, and the incoming speech&#10;can be switched across different domains during a session. This brings a&#10;challenge for effective domain adaptation when only target domain text data is&#10;available, and our objective is to obtain obviously improved performance on the&#10;target domain while the performance on the general domain is less undermined.&#10;In this paper, we propose an adaptive LM fusion approach called internal&#10;language model estimation based adaptive domain adaptation (ILME-ADA). To&#10;realize such an ILME-ADA, an interpolated log-likelihood score is calculated&#10;based on the maximum of the scores from the internal LM and the external LM&#10;(ELM) respectively. We demonstrate the efficacy of the proposed ILME-ADA method&#10;with both RNN-T and LAS modeling frameworks employing neural network and n-gram&#10;LMs as ELMs respectively on two domain specific (target) test sets. The&#10;proposed method can achieve significantly better performance on the target test&#10;sets while it gets minimal performance degradation on the general test set,&#10;compared with both shallow and ILME-based LM fusion methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="~&#10;To obtain balanced recognition results on two different domains with LM fusion at the same time, one can think of a simple method, namely, building an interpolated LM by employing two data sets respectively. Unfortunately, ASR source data is normally very big and not easy to come by, say, for proprietary issues, ILME-based LM fusion~\cite{meng2021internal} can solve the problem of target domain recognition once more target domain text data is available. However, the ASR performance is degraded on the original/general domain data at the same time. &#10;&#10;Recently, \cite{choudhury2022likelihood} proposed a likelihood ratio based LM fusion method for DA while the ASR performance on the general data set is not degraded. To the best of our knowledge, the idea is the closest to what is proposed in this work. &#10;However, the proposed ILME-ADA is significantly different from \cite{choudhury2022likelihood} in several aspects. &#10;First, they employ likelihood ratio method, while it is the ILME method that is proposed here.&#10;Secondly, the ELM in \cite{choudhury2022likelihood} is interpolated with both data of general and target domains, here, the ELM is always trained with only target domain data.&#10;Thirdly, they only verified the idea with the n-gram LM fusion method over the RNN-T ASR model. Here we demonstrate the efficacy of the proposed method &#10;with RNNLM and n-gram LM over both RNN-T and LAS ASR frameworks respectively. &#10;&#10;" />
        </attvalues>
      </node>
      <node id="2210.17017" label="2210.17017">
        <attvalues>
          <attvalue for="0" value="Blank Collapse: Compressing CTC emission for the faster decoding" />
          <attvalue for="1" value="  Connectionist Temporal Classification (CTC) model is a very efficient method&#10;for modeling sequences, especially for speech data. In order to use CTC model&#10;as an Automatic Speech Recognition (ASR) task, the beam search decoding with an&#10;external language model like n-gram LM is necessary to obtain reasonable&#10;results. In this paper we analyze the blank label in CTC beam search deeply and&#10;propose a very simple method to reduce the amount of calculation resulting in&#10;faster beam search decoding speed. With this method, we can get up to 78%&#10;faster decoding speed than ordinary beam search decoding with a very small loss&#10;of accuracy in LibriSpeech datasets. We prove this method is effective not only&#10;practically by experiments but also theoretically by mathematical reasoning. We&#10;also observe that this reduction is more obvious if the accuracy of the model&#10;is higher.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.10026" label="2110.10026">
        <attvalues>
          <attvalue for="0" value="Private Language Model Adaptation for Speech Recognition" />
          <attvalue for="1" value="  Speech model adaptation is crucial to handle the discrepancy between&#10;server-side proxy training data and actual data received on local devices of&#10;users. With the use of federated learning (FL), we introduce an efficient&#10;approach on continuously adapting neural network language models (NNLMs) on&#10;private devices with applications on automatic speech recognition (ASR). To&#10;address the potential speech transcription errors in the on-device training&#10;corpus, we perform empirical studies on comparing various strategies of&#10;leveraging token-level confidence scores to improve the NNLM quality in the FL&#10;settings. Experiments show that compared with no model adaptation, the proposed&#10;method achieves relative 2.6% and 10.8% word error rate (WER) reductions on two&#10;speech evaluation datasets, respectively. We also provide analysis in&#10;evaluating privacy guarantees of our presented procedure.&#10;" />
          <attvalue for="2" value="&#10;&#10;Neural network language models (NNLMs) play critical roles in automatic speech recognition (ASR) systems \cite{mikolov2010recurrent,chen2015improving,xu2018neural,irie2019language}. They typically outperform traditional $n$-gram LMs with better capability of modeling long-range dependency. For conventional ASR models, NNLMs are widely used in the second pass via $N$-best or lattice rescoring \cite{liu2014efficient,xu2018pruned,li2021parallelizable}. For end-to-end ASR \cite{graves2006connectionist,graves2012sequence,chan2016listen}, although linguistic information is implicitly learned, NNLMs can still further improve accuracy by fusion in first-pass decoding \cite{kannan2018analysis, kim2021improved} or second-pass rescoring.&#10;&#10;With the latest advances in mobile technologies, hosting an ASR system entirely on-device has important implications from a reliability, latency, and particularly privacy perspective, and has become an active area of research and industrial applications \cite{he2019streaming}. A common issue arising after deploying an ASR model on user devices is the discrepancy between training data and actual data received on local devices. The semantic and acoustic characteristics of real users' speech could be very different from those of server-side proxy data, in which case speech model adaptation is indispensable. The privacy-preserving constraint requires user data to stay on their local devices. It is thus more challenging to perform model adaptation on user devices since there is no ground truth speech transcription from users.&#10;&#10;To resolve this privacy concern, federated learning (FL) \cite{konevcny2016federated, konevcny2016federated2, mcmahan2017communication}, a distributed learning technique, has been proposed and applied in many fields including recommendation \cite{chen2018federated}, keyboard suggestion \cite{arnold2016suggesting, ji2019learning}, keyword spotting \cite{leroy2019federated}, health care \cite{xu2019federated}, and more recently, ASR including hybrid acoustic models and end-to-end models \cite{dimitriadis2020federated, guliani2021training, cui2021federated}. FL can protect data privacy by training a shared model in a decentralized manner on users' local devices, so that raw data never leaves physical devices. Specifically, FL distributes the training process among a large number of client devices, with each client learning from live data and calculating model updates independently, then uploading those updates to a central server for aggregation. The updated model will later be delivered to each client device, after which this procedure is repeated until the training convergence of the model.&#10;&#10;In this work, we focus on federated NNLM adaptation for speech recognition application. Federated language modeling has been well explored in mobile keyboard suggestion where sentences typed by users provide instant labeled data for supervised learning \cite{ji2019learning}. However, for any on-device ASR with privacy-preserving requirement, users' text data can not be directly accessed. Instead, we could use decoded hypotheses to perform model adaptation. The adaptation quality can be affected by any ASR transcription errors. Thus, more advanced methods are called for to better leverage transcribed data to conduct FL-based adaptation in an unsupervised manner.&#10;&#10;To alleviate the transcription errors issue described above, we leverage confidence scores of transcripts, which estimate how likely each token in any decoded hypothesis from ASR is correct \cite{jiang2005confidence, huang2013predicting}. Lattice posteriors from conventional ASR systems can be directly used as confidence scores. Modeling based approaches, for example, confidence classifiers trained with various decoding features \cite{kalgaonkar2015estimating}, can provide more accurate confidence measurements. In this paper, we propose to mitigate errors in decoded hypotheses by modifying NNLM training objective using token-level confidence scores from a confidence classifier directly and improve adaptation quality.&#10;&#10;The prior work on using ASR confidence scores in LM task is limited. Authors in \cite{xie2013evaluating} use confidence scores for selecting text data for LM adaptation. Our paper presents and investigates this direction in a rigorous way, proposes the weighting method for adjusting the cross-entropy loss, and conducts solid comparisons on the performance of these weighting approaches in the FL framework.&#10;&#10;We mainly pursue three goals: (1) to present an effective procedure on FL-based domain adaptation of NNLMs with its applications on ASR; (2) to empirically compare approaches of using token-level confidence scores to improve adaptation quality; and (3) to provide analysis in evaluating privacy guarantees of our proposed method using differential privacy (DP) tools \cite{dwork2006calibrating, dwork2014algorithmic}. To the best of our knowledge, our paper is the first work that leverages FL to fine-tune NNLMs for ASR systems and utilizes confidence scores to address any potential quality degradation due to mis-transcribed text as training corpus. In particular, the proposed confidence-based approach can also be applied to other tasks as well, for example, unsupervised speaker adaptation.&#10;&#10;The rest of the paper is organized as follows. In Section~\ref{sec:method}, we introduce the FL-based domain adaptation approach of NNLMs with its applications on speech recognition tasks. We evaluate the proposed method in Section~\ref{sec:expt} and conclude in Section~\ref{sec:summary}.&#10;&#10;" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.09058" label="2212.09058">
        <attvalues>
          <attvalue for="0" value="BEATs: Audio Pre-Training with Acoustic Tokenizers" />
          <attvalue for="1" value="  The massive growth of self-supervised learning (SSL) has been witnessed in&#10;language, vision, speech, and audio domains over the past few years. While&#10;discrete label prediction is widely adopted for other modalities, the&#10;state-of-the-art audio SSL models still employ reconstruction loss for&#10;pre-training. Compared with reconstruction loss, semantic-rich discrete label&#10;prediction encourages the SSL model to abstract the high-level audio semantics&#10;and discard the redundant details as in human perception. However, a&#10;semantic-rich acoustic tokenizer for general audio pre-training is usually not&#10;straightforward to obtain, due to the continuous property of audio and&#10;unavailable phoneme sequences like speech. To tackle this challenge, we propose&#10;BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder&#10;representation from Audio Transformers, where an acoustic tokenizer and an&#10;audio SSL model are optimized by iterations. In the first iteration, we use&#10;random projection as the acoustic tokenizer to train an audio SSL model in a&#10;mask and label prediction manner. Then, we train an acoustic tokenizer for the&#10;next iteration by distilling the semantic knowledge from the pre-trained or&#10;fine-tuned audio SSL model. The iteration is repeated with the hope of mutual&#10;promotion of the acoustic tokenizer and audio SSL model. The experimental&#10;results demonstrate our acoustic tokenizers can generate discrete labels with&#10;rich audio semantics and our audio SSL models achieve state-of-the-art results&#10;across various audio classification benchmarks, even outperforming previous&#10;models that use more training data and model parameters significantly.&#10;Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for&#10;audio-only models without using any external data, and 98.1% accuracy on&#10;ESC-50. The code and pre-trained models are available at https://aka.ms/beats.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.05675" label="2210.05675">
        <attvalues>
          <attvalue for="0" value="Transformers generalize differently from information stored in context&#10;  vs in weights" />
          <attvalue for="1" value="  Transformer models can use two fundamentally different kinds of information:&#10;information stored in weights during training, and information provided&#10;``in-context'' at inference time. In this work, we show that transformers&#10;exhibit different inductive biases in how they represent and generalize from&#10;the information in these two sources. In particular, we characterize whether&#10;they generalize via parsimonious rules (rule-based generalization) or via&#10;direct comparison with observed examples (exemplar-based generalization). This&#10;is of important practical consequence, as it informs whether to encode&#10;information in weights or in context, depending on how we want models to use&#10;that information. In transformers trained on controlled stimuli, we find that&#10;generalization from weights is more rule-based whereas generalization from&#10;context is largely exemplar-based. In contrast, we find that in transformers&#10;pre-trained on natural language, in-context learning is significantly&#10;rule-based, with larger models showing more rule-basedness. We hypothesise that&#10;rule-based generalization from in-context information might be an emergent&#10;consequence of large-scale training on language, which has sparse rule-like&#10;structure. Using controlled stimuli, we verify that transformers pretrained on&#10;data containing sparse rule-like structure exhibit more rule-based&#10;generalization.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformer-based architectures have an impressive ability to use both information stored in weights during training (``in-weights learning''), and information stored only in the inputs provided at inference time (without any gradient updates to the weights of the model; ``in-context learning'') \cite{chan_data_2022}. In-context learning on pretrained models &#10;enables learning efficiently from a few examples (``few-shot learning&quot;) \cite{brown_language_2020}, or even efficiently compressing a large dataset (``prompt tuning&quot;) \cite{li_prefix-tuning_2021, lester_power_2021, sun_black-box_2022}. Given the evident current and future potential for this new learning paradigm, it is important and useful to understand its inductive biases, especially how it differs from in-weights learning. &#10;&#10;One way to understand inductive bias is by examining how models generalize to held-out data. In this work, we adapt the experimental paradigm in \cite{dasgupta_distinguishing_2022} that pose a classification task that distinguishes between two previously defined kinds of generalization behaviors (see \ref{fig:explainer}). A ``rule-based'' decision is made on the basis of minimal features that support the category boundary \cite{ashby_varieties_1986}, while an ``exemplar-based'' decision generalizes on the basis of similarity to examples from training data \cite{shepard_stimulus_1963}, invoking many or all features available. &#10;&#10;This distinction is particularly interesting when comparing in-weights vs in-context learning. Exemplar-based generalization (that uses all available features) is useful in a low-data regime where there is not enough information to form an abstract sparse rule \cite{feldman2020does}. On the other hand, sparser rule-based generalization may help avoid sensitivity to spurious correlation when training with large, noisy, naturalistic datasets (that are commonly used to train in-weights learning). &#10;&#10;We find that transformers exhibit a striking difference in their generalization from in-context vs in-weights information. Transformers display a strong inductive bias towards exemplar-based generalization from in-context information. In contrast, transformers display a strong inductive bias towards rule-based generalization from in-weights information.&#10;&#10;However, when we pose a similar task to large transformer models pretrained on language, they exhibit stronger rule-based generalization from in-context information. One interpretation of these results is that the distribution of natural language is more compatible with rule-based generalization from context (rule-based generalization is in fact optimal in compositional domains like langauge \cite{arjovsky2019invariant}), and such patterns might present strong enough learning pressure to overcome -- and even reverse -- transformers' inherent bias towards exemplar-based generalization from context. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Inductive Bias Analysis, Cognitive Science, Linguistics, Transformer Generalization, Language Model Training, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1906.05271" label="1906.05271">
        <attvalues>
          <attvalue for="0" value="Does Learning Require Memorization? A Short Tale about a Long Tail" />
          <attvalue for="1" value="  State-of-the-art results on image recognition tasks are achieved using&#10;over-parameterized learning algorithms that (nearly) perfectly fit the training&#10;set and are known to fit well even random labels. This tendency to memorize the&#10;labels of the training data is not explained by existing theoretical analyses.&#10;Memorization of the training data also presents significant privacy risks when&#10;the training data contains sensitive personal information and thus it is&#10;important to understand whether such memorization is necessary for accurate&#10;learning.&#10;  We provide the first conceptual explanation and a theoretical model for this&#10;phenomenon. Specifically, we demonstrate that for natural data distributions&#10;memorization of labels is necessary for achieving close-to-optimal&#10;generalization error. Crucially, even labels of outliers and noisy labels need&#10;to be memorized. The model is motivated and supported by the results of several&#10;recent empirical works. In our model, data is sampled from a mixture of&#10;subpopulations and our results show that memorization is necessary whenever the&#10;distribution of subpopulation frequencies is long-tailed. Image and text data&#10;is known to be long-tailed and therefore our results establish a formal link&#10;between these empirical phenomena. Our results allow to quantify the cost of&#10;limiting memorization in learning and explain the disparate effects that&#10;privacy and model compression have on different subgroups.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1907.02893" label="1907.02893">
        <attvalues>
          <attvalue for="0" value="Invariant Risk Minimization" />
          <attvalue for="1" value="  We introduce Invariant Risk Minimization (IRM), a learning paradigm to&#10;estimate invariant correlations across multiple training distributions. To&#10;achieve this goal, IRM learns a data representation such that the optimal&#10;classifier, on top of that data representation, matches for all training&#10;distributions. Through theory and experiments, we show how the invariances&#10;learned by IRM relate to the causal structures governing the data and enable&#10;out-of-distribution generalization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.00313" label="1911.00313">
        <attvalues>
          <attvalue for="0" value="Deep Bidirectional Transformers for Relation Extraction without&#10;  Supervision" />
          <attvalue for="1" value="  We present a novel framework to deal with relation extraction tasks in cases&#10;where there is complete lack of supervision, either in the form of gold&#10;annotations, or relations from a knowledge base. Our approach leverages&#10;syntactic parsing and pre-trained word embeddings to extract few but precise&#10;relations,which are then used to annotate a larger cor-pus, in a manner&#10;identical to distant supervision. The resulting data set is employed to fine&#10;tune a pre-trained BERT model in order to perform relation extraction.&#10;Empirical evaluation on four data sets from the biomedical domain shows that&#10;our method significantly outperforms two simple baselines for unsupervised&#10;relation extraction and, even if not using any supervision at all, achieves&#10;slightly worse results than the state-of-the-art in three out of four data&#10;sets. Importantly, we show that it is possible to successfully fine tune a&#10;large pre-trained language model with noisy data, as op-posed to previous works&#10;that rely on gold data for fine tuning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.02080" label="2102.02080">
        <attvalues>
          <attvalue for="0" value="Top-down Discourse Parsing via Sequence Labelling" />
          <attvalue for="1" value="  We introduce a top-down approach to discourse parsing that is conceptually&#10;simpler than its predecessors (Kobayashi et al., 2020; Zhang et al., 2020). By&#10;framing the task as a sequence labelling problem where the goal is to&#10;iteratively segment a document into individual discourse units, we are able to&#10;eliminate the decoder and reduce the search space for splitting points. We&#10;explore both traditional recurrent models and modern pre-trained transformer&#10;models for the task, and additionally introduce a novel dynamic oracle for&#10;top-down parsing. Based on the Full metric, our proposed LSTM model sets a new&#10;state-of-the-art for RST parsing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.00153" label="2010.00153">
        <attvalues>
          <attvalue for="0" value="Examining the rhetorical capacities of neural language models" />
          <attvalue for="1" value="  Recently, neural language models (LMs) have demonstrated impressive abilities&#10;in generating high-quality discourse. While many recent papers have analyzed&#10;the syntactic aspects encoded in LMs, there has been no analysis to date of the&#10;inter-sentential, rhetorical knowledge. In this paper, we propose a method that&#10;quantitatively evaluates the rhetorical capacities of neural LMs. We examine&#10;the capacities of neural LMs understanding the rhetoric of discourse by&#10;evaluating their abilities to encode a set of linguistic features derived from&#10;Rhetorical Structure Theory (RST). Our experiments show that BERT-based LMs&#10;outperform other Transformer LMs, revealing the richer discourse knowledge in&#10;their intermediate layer representations. In addition, GPT-2 and XLNet&#10;apparently encode less rhetorical knowledge, and we suggest an explanation&#10;drawing from linguistic philosophy. Our method shows an avenue towards&#10;quantifying the rhetorical capacities of neural LMs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.01381" label="2307.01381">
        <attvalues>
          <attvalue for="0" value="Implicit Memory Transformer for Computationally Efficient Simultaneous&#10;  Speech Translation" />
          <attvalue for="1" value="  Simultaneous speech translation is an essential communication task difficult&#10;for humans whereby a translation is generated concurrently with oncoming speech&#10;inputs. For such a streaming task, transformers using block processing to break&#10;an input sequence into segments have achieved state-of-the-art performance at a&#10;reduced cost. Current methods to allow information to propagate across&#10;segments, including left context and memory banks, have faltered as they are&#10;both insufficient representations and unnecessarily expensive to compute. In&#10;this paper, we propose an Implicit Memory Transformer that implicitly retains&#10;memory through a new left context method, removing the need to explicitly&#10;represent memory with memory banks. We generate the left context from the&#10;attention output of the previous segment and include it in the keys and values&#10;of the current segment's attention calculation. Experiments on the MuST-C&#10;dataset show that the Implicit Memory Transformer provides a substantial&#10;speedup on the encoder forward pass with nearly identical translation quality&#10;when compared with the state-of-the-art approach that employs both left context&#10;and memory banks.&#10;" />
          <attvalue for="2" value="&#10;Simultaneous speech translation (SimulST) refers to the process of producing an output translation concurrently with an oncoming source speech input. For humans, performing accurate SimulST is extremely difficult and becomes nearly impossible to perform over long periods of time. Given the potential broad applications of SimulST in industry and government sectors, there is a strong need for machine learning models to perform the task to a level above the capabilities of humans. &#10;&#10;One branch of machine learning models that have been effective in SimulST is transformers \cite{vaswani2017attention} using block processing, a process that breaks an input sequence into segments which the encoder processes sequentially and individually \cite{dong2019self}. As later segments may lose earlier information in a sentence (i.e., context fragmentation), techniques known as left context and memory banks have been introduced. The concept of left context was idealized with the Transformer-XL \cite{dai2019transformer}, a model optimized for language modeling, which was later adapted for streaming automatic speech recognition (ASR). The Transformer-XL generated left context by saving the previous segment to each encoder layer, so the subsequent segment could include it in the attention calculation at the same encoder layer. Memory banks were later introduced in the self-attention calculation of the Augmented Memory Transformer \cite{wu2020streaming}, allowing it to outperform the Transformer-XL in streaming ASR and also be state-of-the-art in SimulST \cite{ma2021streaming}. These memory banks were token summarizations of previous segments and helped retain explicit long-term dependencies. The Augmented Memory Transformer also included the left context alongside the center (main) segment tokens with an additional right context, all of which add computational cost. We argue that the methods to generate and use left context and/or memory banks in both the Transformer-XL and Augmented Memory Transformer are naive, costing both models' performance at a given computational budget.&#10;&#10;In this paper, we propose a computationally efficient architecture, the Implicit Memory Transformer, that implicitly retains memory through a novel left context generation method, thereby removing the need for memory banks entirely. Briefly, the proposed left context method for a given encoder layer leverages the previous segment's attention output in the attention calculation of the current segment. Our method for calculating left context is broadly applicable to any transformer model that utilizes block processing. &#10;The proposed Implicit Memory Transformer is more computationally efficient than the Augmented Memory Transformer, reducing the cost of self-attention calculation, convolution layers, and feed-forward layers. &#10;&#10;We conduct our experiments on the English-German, English-French, and English-Spanish language pairs of the MuST-C dataset \cite{CATTONI2021101155} and demonstrate a significant speedup over the Augmented Memory Transformer for the forward pass of the encoder, with no reduction in the translation quality across all wait-$k$ values.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Speech Translation, Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Transformer Models, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2309.13029" label="2309.13029">
        <attvalues>
          <attvalue for="0" value="Memory-augmented conformer for improved end-to-end long-form ASR" />
          <attvalue for="1" value="  Conformers have recently been proposed as a promising modelling approach for&#10;automatic speech recognition (ASR), outperforming recurrent neural&#10;network-based approaches and transformers. Nevertheless, in general, the&#10;performance of these end-to-end models, especially attention-based models, is&#10;particularly degraded in the case of long utterances. To address this&#10;limitation, we propose adding a fully-differentiable memory-augmented neural&#10;network between the encoder and decoder of a conformer. This external memory&#10;can enrich the generalization for longer utterances since it allows the system&#10;to store and retrieve more information recurrently. Notably, we explore the&#10;neural Turing machine (NTM) that results in our proposed Conformer-NTM model&#10;architecture for ASR. Experimental results using Librispeech train-clean-100&#10;and train-960 sets show that the proposed system outperforms the baseline&#10;conformer without memory for long utterances.&#10;" />
          <attvalue for="2" value="&#10;&#10;Traditional speech recognition systems rely on sophisticated and individual components, including acoustic, pronunciation and language models (LMs) \cite{hmms}.\ In contrast, end-to-end (E2E) speech recognition systems rely on a single deep neural network that learns to map an input sequence of features or raw audio to the corresponding labels; usually, characters or sub-words \cite{pmlr-v32-graves14, 8706675}.\ Because of this simplicity, and in some situations, superior performance over traditional systems, E2E systems have become a favoured procedure for automatic speech recognition (ASR) \cite{ 7472621, chiu2018state}.\ Some widely used E2E approaches are based on connectionist temporal classification (CTC) \cite{graves2006connectionist}, recurrent neural network transducers (RNN-T) \cite{graves2012sequence} and attention-based encoder-decoders (AEDs) \cite{7472621}.&#10;&#10;The transformer \cite{vaswani2017attention} architecture is an AED-based system that uses self-attention to capture long-range interactions.\ Nevertheless, a transformer has more difficulty extracting fine-grained local feature patterns than convolution neural networks (CNNs) \cite{49414}.&#10;For this reason, conformers \cite{49414} have been proposed as an approach for E2E ASR, which outperform RNN-based approaches and transformers since they can model the global and local dependencies of an audio sequence by combining CNNs with transformers.&#10;Nonetheless, E2E ASR methods, particularly AED-based procedures, are known to degrade performance on long utterances when trained on short utterances \cite{narayanan2019recognizing, chiu2019comparison}.\ Besides, long-form transcription is crucial for creating continuous transcriptions of real-world scenarios, like lectures, meetings, and video captions (e.g. YouTube).&#10;&#10;The problem of long-form speech has been addressed in some previous works by simulating training on longer utterances \cite{hori2020transformer, hori2021conformer}. &#10;For example, \cite{hori2020transformer} proposed a method where the transformer or conformer accepts multiple consecutive utterances simultaneously and predicts an output for the last utterance only. This procedure is repeated with a sliding window using one-utterance shifts to recognise the whole recording. Another solution is to segment the audio in advance using a separate voice activity detector (VAD) based approach \cite{bain2023whisperx}, &#10;or an E2E model that learns to predict segment boundaries \cite{51460}.&#10;The E2E segmenter proposed in \cite{51460} relies on human-created heuristics to insert end-of-segment tokens in utterances at training time so that the model can learn to predict those tokens. &#10;&#10;Only few works try to improve the generalisation of E2E ASR systems to long speech without requiring some pre-processing stage or changing how the model trains and decodes compared to traditional E2E ASR. &#10;For instance, \cite{10045036} proposes the replacement of self-attention with fast attention, which improves the model generalisation ability for long utterances. &#10;&#10;In contrast to the works mentioned above, &#10;we hypothesise whether adding a memory-augmented neural network (MANN) in between the encoder and decoder module -- like a neural Turing machine (NTM) \cite{graves2014neural} -- may be a convenient method to enrich the learning capacity of a conformer, contributing to increase the network generalisation for longer utterances without the need for any ad hoc pre-processing or optimisation in training or decoding.\ Thus, NTMs have demonstrated superior performance over long short-term&#10;memory cells (LSTMs) in several learning tasks.\ Moreover, to our knowledge, few works have investigated the use of MANNs for the E2E ASR task.\ In particular, NTM has been used to perform unsupervised speaker adaptation in \cite{sari2020unsupervised} by storing i-vectors \cite{dehak2010front} and then reading from the memory to combine the resulting read vector with the hidden vectors of the encoder of the listen, attend and spell (LAS) architecture \cite{7472621}.\ However, in that work, the write operation of the NTM was not explored, therefore not taking advantage of the full potential of the external memory. &#10;&#10;In this work, we propose incorporating a MANN based on NTM to improve the generalisation of the offline E2E ASR system to long sentences. We refer to this newly proposed ASR architecture as Conformer-NTM. This proposed model and the state-of-the-art (SOTA) conformer baseline (without memory) are trained on Librispeech \cite{panayotov2015librispeech} 100 hours clean and 960 hours.\ Then, we use the test clean and other partitions to evaluate the overall performance of all models. We follow this with an ablation study, testing the models with different utterance lengths (long and very-long). &#10;Our results show that the E2E system can generalise better with the external memory for longer utterances with the Conformer-NTM model.\ Notice that while the focus of this work is on offline ASR settings, the proposed MANN is also expected to complement streaming ASR approaches that address the long-form ASR problem \cite{narayanan2019recognizing, wu2020streaming, tsunoo2019transformer}.&#10;&#10;The rest of the paper is organised as follows.\ Section 2 summarises the MANN system.\ Section 3 introduces the proposed approach.\ In Section 4, we describe the experimental evaluation and obtained results, and in Section 5, we provide some concluding remarks and possible directions for future work.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Neural Network Architectures, Memory-Augmented Models, Artificial Intelligence, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="2204.10749" label="2204.10749">
        <attvalues>
          <attvalue for="0" value="E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASR" />
          <attvalue for="1" value="  Improving the performance of end-to-end ASR models on long utterances ranging&#10;from minutes to hours in length is an ongoing challenge in speech recognition.&#10;A common solution is to segment the audio in advance using a separate voice&#10;activity detector (VAD) that decides segment boundary locations based purely on&#10;acoustic speech/non-speech information. VAD segmenters, however, may be&#10;sub-optimal for real-world speech where, e.g., a complete sentence that should&#10;be taken as a whole may contain hesitations in the middle (&quot;set an alarm for...&#10;5 o'clock&quot;).&#10;  We propose to replace the VAD with an end-to-end ASR model capable of&#10;predicting segment boundaries in a streaming fashion, allowing the segmentation&#10;decision to be conditioned not only on better acoustic features but also on&#10;semantic features from the decoded text with negligible extra computation. In&#10;experiments on real world long-form audio (YouTube) with lengths of up to 30&#10;minutes, we demonstrate 8.5% relative WER improvement and 250 ms reduction in&#10;median end-of-segment latency compared to the VAD segmenter baseline on a&#10;state-of-the-art Conformer RNN-T model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1410.5401" label="1410.5401">
        <attvalues>
          <attvalue for="0" value="Neural Turing Machines" />
          <attvalue for="1" value="  We extend the capabilities of neural networks by coupling them to external&#10;memory resources, which they can interact with by attentional processes. The&#10;combined system is analogous to a Turing Machine or Von Neumann architecture&#10;but is differentiable end-to-end, allowing it to be efficiently trained with&#10;gradient descent. Preliminary results demonstrate that Neural Turing Machines&#10;can infer simple algorithms such as copying, sorting, and associative recall&#10;from input and output examples.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.06165" label="2002.06165">
        <attvalues>
          <attvalue for="0" value="Unsupervised Speaker Adaptation using Attention-based Speaker Memory for&#10;  End-to-End ASR" />
          <attvalue for="1" value="  We propose an unsupervised speaker adaptation method inspired by the neural&#10;Turing machine for end-to-end (E2E) automatic speech recognition (ASR). The&#10;proposed model contains a memory block that holds speaker i-vectors extracted&#10;from the training data and reads relevant i-vectors from the memory through an&#10;attention mechanism. The resulting memory vector (M-vector) is concatenated to&#10;the acoustic features or to the hidden layer activations of an E2E neural&#10;network model. The E2E ASR system is based on the joint connectionist temporal&#10;classification and attention-based encoder-decoder architecture. M-vector and&#10;i-vector results are compared for inserting them at different layers of the&#10;encoder neural network using the WSJ and TED-LIUM2 ASR benchmarks. We show that&#10;M-vectors, which do not require an auxiliary speaker embedding extraction&#10;system at test time, achieve similar word error rates (WERs) compared to&#10;i-vectors for single speaker utterances and significantly lower WERs for&#10;utterances in which there are speaker changes.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.03673" label="2406.03673">
        <attvalues>
          <attvalue for="0" value="Linguistically Conditioned Semantic Textual Similarity" />
          <attvalue for="1" value="  Semantic textual similarity (STS) is a fundamental NLP task that measures the&#10;semantic similarity between a pair of sentences. In order to reduce the&#10;inherent ambiguity posed from the sentences, a recent work called Conditional&#10;STS (C-STS) has been proposed to measure the sentences' similarity conditioned&#10;on a certain aspect. Despite the popularity of C-STS, we find that the current&#10;C-STS dataset suffers from various issues that could impede proper evaluation&#10;on this task. In this paper, we reannotate the C-STS validation set and observe&#10;an annotator discrepancy on 55% of the instances resulting from the annotation&#10;errors in the original label, ill-defined conditions, and the lack of clarity&#10;in the task definition. After a thorough dataset analysis, we improve the C-STS&#10;task by leveraging the models' capability to understand the conditions under a&#10;QA task setting. With the generated answers, we present an automatic error&#10;identification pipeline that is able to identify annotation errors from the&#10;C-STS data with over 80% F1 score. We also propose a new method that largely&#10;improves the performance over baselines on the C-STS data by training the&#10;models with the answers. Finally we discuss the conditionality annotation based&#10;on the typed-feature structure (TFS) of entity types. We show in examples that&#10;the TFS is able to provide a linguistic foundation for constructing C-STS data&#10;with new conditions.&#10;" />
          <attvalue for="2" value="&#10;&#10;sts is an essential NLP task that measures the semantic similarity between two sentences \cite{sts12}.&#10;It is also a popular benchmark for developing tasks such as text embedding learning \cite{senteval,sbert,sbert-aug} and language understanding \cite{glue}.&#10;While the sts datasets have been developed and improved over the past years \cite{sts13,sts14,sts15,sts16,sts17}, &#10;the task itself still suffers from sentence ambiguity and subjectivity to judgment \cite{csts}. &#10;&#10;A new task called csts has been proposed to resolve those issues \cite{csts}. It is designed to disambiguate the similarity between two sentences by measuring the similarity on a given condition. &#10;An accompanying dataset was also proposed to test models on the csts task. &#10;Despite the popularity of csts, we observe certain limitations in the csts dataset that could hinder the understanding and proper evaluation of models on this task.&#10;As illustrated in Figure \ref{fig:example}, these limitations primarily revolve around annotation errors, ill-defined conditions, and a general lack of clarity in task definition. &#10;&#10;Taking into account the significance of these issues, we intend to improve the csts dataset by addressing the existing problems that we observed.&#10;We start by reannotating the csts validation set. By identifying an apparent annotation error rate of 55\% in their validation set, we analyze the provenance of the errors and discrepancies between the original and relabeled datasets.&#10;&#10;To further explore the utility of the condition and how it is understood by language models, we treat it as a qa task and leverage llms to generate the answer to the question that is constructed from the condition. &#10;We find that the LLM-generated answers can better capture the similarity between two sentences and fit closely to our reannotated labels by having a higher Spearman's Correlation. &#10;Based on this finding, we propose an approach to identify potential annotation errors from the csts dataset utilizing the LLM-generated answers, achieving over 80\% F1 score on the validation set.&#10;We also propose a new method to improve the csts task by training the models with the answers. We show that both supervised and generative models can efficiently and effectively learn the condition information encoded in the answers, improving the performance over baselines by a large margin. &#10;&#10;Finally, we discuss a new annotation specification of the conditionality that aims to improve the formulation of the conditions with a more concrete semantic base.&#10;We use the entity type identified from the sentence pair as the surface condition text that is described by its underlying tfs \cite{carpenter1992logic,copestake2000definitions,penn2000algebraic}.&#10;We exemplify that tfs-based conditions can be successfully adopted to sentence pairs from the current csts dataset.&#10;&#10;We summarize the main contributions of this paper as threefold. We reannotate the csts validation set and propose an error identification pipeline that can be applied to the whole dataset to identify potential annotation errors and ambiguities; we propose a qa-facilitated method that largely improves the model performance on the csts task; we discuss using tfs&#10; as a new annotation specification to improve the conditionality in csts dataset with a more concrete semantic base.&#10;We make the source code and dataset publicly available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Dataset Annotation Errors, Computer Science, Linguistics, Artificial Intelligence, Natural Language Processing, Semantic Textual Similarity" />
        </attvalues>
      </node>
      <node id="2112.05662" label="2112.05662">
        <attvalues>
          <attvalue for="0" value="Match Your Words! A Study of Lexical Matching in Neural Information&#10;  Retrieval" />
          <attvalue for="1" value="  Neural Information Retrieval models hold the promise to replace lexical&#10;matching models, e.g. BM25, in modern search engines. While their capabilities&#10;have fully shone on in-domain datasets like MS MARCO, they have recently been&#10;challenged on out-of-domain zero-shot settings (BEIR benchmark), questioning&#10;their actual generalization capabilities compared to bag-of-words approaches.&#10;Particularly, we wonder if these shortcomings could (partly) be the consequence&#10;of the inability of neural IR models to perform lexical matching off-the-shelf.&#10;In this work, we propose a measure of discrepancy between the lexical matching&#10;performed by any (neural) model and an 'ideal' one. Based on this, we study the&#10;behavior of different state-of-the-art neural IR models, focusing on whether&#10;they are able to perform lexical matching when it's actually useful, i.e. for&#10;important terms. Overall, we show that neural IR models fail to properly&#10;generalize term importance on out-of-domain collections or terms almost unseen&#10;during training&#10;" />
          <attvalue for="2" value="&#10;&#10;Over the last two years, the effectiveness of neural IR systems has risen substantially. Neural retrievers based on pre-trained Language Models like BERT~\cite{DBLP:conf/naacl/DevlinCLT19} -- whether dense or sparse -- hold the promise to replace lexical matching models (e.g. BM25) for first-stage ranking in modern search engines. &#10;Despite this success, little is known regarding their actual inner working in the IR setting. Previous works scrutinizing BERT-based ranking models either relied on axiomatic approaches adapted to neural models~\cite{CamaraDiagnosingBERTRetrieval2020,sciavolino2021simple}, controlled experiments~\cite{MacAvaneyABNIRMLAnalyzingBehavior2020}, or direct investigation of the learned representations~\cite{JiangHowDoesBERT2021,FormalWhiteBoxAnalysis2021} or attention~\cite{10.1145/3437963.3441667}. &#10;This line of work has shown -- among other findings -- that these models, which rely on contextualized semantic matching, are actually still quite sensitive to lexical match and term statistics in documents/collections~\cite{JiangHowDoesBERT2021,FormalWhiteBoxAnalysis2021}. However, these observations are based on specifically tailored approaches that cannot directly be applied to any given model. To generalize these findings, we introduce instead an intuitive black box approach: we propose to ``count'' query terms appearing in top documents retrieved by various state-of-the-art neural systems, in order to compare their ability to perform lexical matching.&#10;&#10;Furthermore, previous studies have been conducted on the MS MARCO dataset, on which models have been trained. The BEIR benchmark~\cite{ThakurBEIRHeterogenousBenchmark2021a} has shown that the only systems improving the overall performance over BM25 in the zero-shot setting have (somehow) a lexical bias, e.g. models like doc2query-T5~\cite{Nogueiradoc2querydocTTTTTquery} or ColBERT~\cite{KhattabColBERTEfficientEffective2020}. Therefore, we also propose to study the extent to which neural IR models are able to generalize lexical matching, for query terms that either have not been seen in the training set or with different collection statistics (e.g. common in the training set but rare on an out-of-domain evaluation set).&#10;&#10;In this work, we first develop indicators that help measuring to what extent a lexical match is ``important'' for the user (user relevance) or for the model (system relevance). &#10;By comparing both values -- i.e. computing the difference between the user and the system, we can look at the following research questions:&#10;\begin{inparadesc}&#10;\item[(RQ1)] To what extent neural retrievers perform accurate lexical matching (Sect. \ref{sec:lexical-match})?&#10;\item[(RQ2)] Do they generalize term matching to unseen query terms (Sect. \ref{sec:lexical-match})?&#10;\item[(RQ3)] Do they generalize term matching to new collections (Sect. \ref{sec:generalization:zero-shot})?&#10;\end{inparadesc}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Information Retrieval, Computer Science, Linguistics, Lexical Matching Models, Artificial Intelligence, Mathematics, Search Engine Generalization" />
        </attvalues>
      </node>
      <node id="2201.04458" label="2201.04458">
        <attvalues>
          <attvalue for="0" value="Diagnosing BERT with Retrieval Heuristics" />
          <attvalue for="1" value="  Word embeddings, made widely popular in 2013 with the release of word2vec,&#10;have become a mainstay of NLP engineering pipelines. Recently, with the release&#10;of BERT, word embeddings have moved from the term-based embedding space to the&#10;contextual embedding space -- each term is no longer represented by a single&#10;low-dimensional vector but instead each term and \emph{its context} determine&#10;the vector weights. BERT's setup and architecture have been shown to be general&#10;enough to be applicable to many natural language tasks. Importantly for&#10;Information Retrieval (IR), in contrast to prior deep learning solutions to IR&#10;problems which required significant tuning of neural net architectures and&#10;training regimes, &quot;vanilla BERT&quot; has been shown to outperform existing&#10;retrieval algorithms by a wide margin, including on tasks and corpora that have&#10;long resisted retrieval effectiveness gains over traditional IR baselines (such&#10;as Robust04). In this paper, we employ the recently proposed axiomatic dataset&#10;analysis technique -- that is, we create diagnostic datasets that each fulfil a&#10;retrieval heuristic (both term matching and semantic-based) -- to explore what&#10;BERT is able to learn. In contrast to our expectations, we find BERT, when&#10;applied to a recently released large-scale web corpus with ad-hoc topics, to&#10;\emph{not} adhere to any of the explored axioms. At the same time, BERT&#10;outperforms the traditional query likelihood retrieval model by 40\%. This&#10;means that the axiomatic approach to IR (and its extension of diagnostic&#10;datasets created for retrieval heuristics) may in its current form not be&#10;applicable to large-scale corpora. Additional -- different -- axioms are&#10;needed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00696" label="2011.00696">
        <attvalues>
          <attvalue for="0" value="ABNIRML: Analyzing the Behavior of Neural IR Models" />
          <attvalue for="1" value="  Pretrained contextualized language models such as BERT and T5 have&#10;established a new state-of-the-art for ad-hoc search. However, it is not yet&#10;well-understood why these methods are so effective, what makes some variants&#10;more effective than others, and what pitfalls they may have. We present a new&#10;comprehensive framework for Analyzing the Behavior of Neural IR ModeLs&#10;(ABNIRML), which includes new types of diagnostic probes that allow us to test&#10;several characteristics -- such as writing styles, factuality, sensitivity to&#10;paraphrasing and word order -- that are not addressed by previous techniques.&#10;To demonstrate the value of the framework, we conduct an extensive empirical&#10;study that yields insights into the factors that contribute to the neural&#10;model's gains, and identify potential unintended biases the models exhibit.&#10;Some of our results confirm conventional wisdom, like that recent neural&#10;ranking models rely less on exact term overlap with the query, and instead&#10;leverage richer linguistic information, evidenced by their higher sensitivity&#10;to word and sentence order. Other results are more surprising, such as that&#10;some models (e.g., T5 and ColBERT) are biased towards factually correct (rather&#10;than simply relevant) texts. Further, some characteristics vary even for the&#10;same base language model, and other characteristics can appear due to random&#10;variations during model training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.09650" label="2012.09650">
        <attvalues>
          <attvalue for="0" value="A White Box Analysis of ColBERT" />
          <attvalue for="1" value="  Transformer-based models are nowadays state-of-the-art in ad-hoc Information&#10;Retrieval, but their behavior is far from being understood. Recent work has&#10;claimed that BERT does not satisfy the classical IR axioms. However, we propose&#10;to dissect the matching process of ColBERT, through the analysis of term&#10;importance and exact/soft matching patterns. Even if the traditional axioms are&#10;not formally verified, our analysis reveals that ColBERT: (i) is able to&#10;capture a notion of term importance; (ii) relies on exact matches for important&#10;terms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.08663" label="2104.08663">
        <attvalues>
          <attvalue for="0" value="BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information&#10;  Retrieval Models" />
          <attvalue for="1" value="  Existing neural information retrieval (IR) models have often been studied in&#10;homogeneous and narrow settings, which has considerably limited insights into&#10;their out-of-distribution (OOD) generalization capabilities. To address this,&#10;and to facilitate researchers to broadly evaluate the effectiveness of their&#10;models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous&#10;evaluation benchmark for information retrieval. We leverage a careful selection&#10;of 18 publicly available datasets from diverse text retrieval tasks and domains&#10;and evaluate 10 state-of-the-art retrieval systems including lexical, sparse,&#10;dense, late-interaction and re-ranking architectures on the BEIR benchmark. Our&#10;results show BM25 is a robust baseline and re-ranking and&#10;late-interaction-based models on average achieve the best zero-shot&#10;performances, however, at high computational costs. In contrast, dense and&#10;sparse-retrieval models are computationally more efficient but often&#10;underperform other approaches, highlighting the considerable room for&#10;improvement in their generalization capabilities. We hope this framework allows&#10;us to better evaluate and understand existing retrieval systems, and&#10;contributes to accelerating progress towards better robust and generalizable&#10;systems in the future. BEIR is publicly available at&#10;https://github.com/UKPLab/beir.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.16502" label="2408.16502">
        <attvalues>
          <attvalue for="0" value="LLMs vs Established Text Augmentation Techniques for Classification:&#10;  When do the Benefits Outweight the Costs?" />
          <attvalue for="1" value="  The generative large language models (LLMs) are increasingly being used for&#10;data augmentation tasks, where text samples are LLM-paraphrased and then used&#10;for classifier fine-tuning. However, a research that would confirm a clear&#10;cost-benefit advantage of LLMs over more established augmentation methods is&#10;largely missing. To study if (and when) is the LLM-based augmentation&#10;advantageous, we compared the effects of recent LLM augmentation methods with&#10;established ones on 6 datasets, 3 classifiers and 2 fine-tuning methods. We&#10;also varied the number of seeds and collected samples to better explore the&#10;downstream model accuracy space. Finally, we performed a cost-benefit analysis&#10;and show that LLM-based methods are worthy of deployment only when very small&#10;number of seeds is used. Moreover, in many cases, established methods lead to&#10;similar or better model accuracies.&#10;" />
          <attvalue for="2" value="&#10;&#10;The emergence of recent large language models (LLMs) such as GPT-4, Gemini, Llama, and their wide availability, prompted their use in augmentation of textual datasets~\cite{ubani2023zeroshotdataaug, dai2023auggpt, piedboeuf-langlais-2023-chatgpt, cegin-etal-2023-chatgpt, cegin2024effectsdiversityincentivessample}. In most LLM-based augmentation scenarios, the dataset size is increased through paraphrasing of original samplesThe extended datasets are then used for training small downstream classifiers with small inference costs. LLM augmentation has been used in various domains such as sentiment analysis~\cite{ONAN2023101611, piedboeuf-langlais-2023-chatgpt}, intent classification~\cite{cegin-etal-2023-chatgpt}, news classification~\cite{piedboeuf-langlais-2023-chatgpt, cegin2024effectsdiversityincentivessample} and health symptoms classification~\cite{dai2023auggpt}.&#10;&#10;While LLM augmentation improves downstream classifiers, it is also costly (power consumption, CO$_2$ emissions), as generative models often feature parameters in tens of billions. This is magnitudes higher than other established (most used) augmentation methods, including back translation paraphrasing, or BERT-based word insertion and synonym swap. A comparison of established methods with newer LLM-based methods could explore the cases where the established (and much cheaper) methods are preferable due to their equal/better downstream classifier performance (e.g., accuracy) or better cost-benefit ratio.&#10;In this line, previous works~\cite{piedboeuf-langlais-2023-chatgpt, ubani2023zeroshotdataaug, dai2023auggpt} have measured classifier performance, comparing LLM-based and established augmentation methods. The results have so far been conflicting and mixed. Furthermore, existing studies were limited in terms of parameters: neglecting the variety of available LLMs, the potential impact of the number of seed samples and collected samples, and the variety of classifiers and their fine-tuning methods. &#10;&#10;The goal of this paper is to compare the accuracy and cost-benefits of the most used established text augmentation methods with their recent LLM-based counterparts. Compared with previous studies, this paper offers a more systematic and finer-grained comparison over multiple dimensions. We formulate the following research questions:&#10;&#10;\begin{description}[labelwidth = 24pt, leftmargin = !]&#10; \item[RQ1:] Considering downstream classifier accuracy, in which cases do the established textual augmentation methods work equally or better than the LLM-based methods?&#10; \item[RQ2:] In which cases does the cost of using LLM-based textual augmentation methods instead of established ones outweigh its benefits?&#10;\end{description}&#10;&#10;We empirically investigated three techniques commonly used in textual augmentation: paraphrasing, word inserts and word swaps (replacements). All three exist in both established and LLM-based variants. In the established variant, paraphrasing is done through back-translation using a RNN~\cite{sennrich-etal-2016-improving}, while inserts and swaps use BERT-based approach~\cite{kobayashi-2018-contextual, kumar-etal-2020-data}. For LLM-based variants, we prompted 2 LLMs (GPT-3.5 and Llama-3) to perform all three techniques. We experimented with 6 different datasets (with tasks of sentiment analysis, news classification, and intent classification), 3 downstream classifier models (BERT, RoBERTa, DistilBERT), and 2 fine-tuning approaches (fully fine-tuned, and QLoRA~\cite{dettmers2024qlora}). Furthermore, we investigated various numbers of seed and collected samples used in the augmentation. Together, this resulted in a total of 267,300 fine-tunings, from which we identified the best performing LLM and established methods (answering Q1). These were then further scrutinized under cost-benefit analysis (answering Q2).&#10;&#10;The most prominent findings are: 1) The best LLM augmentation methods outperform established ones only when a small number of seeds is used. The advantage of LLM-based augmentation diminishes with increased seed numbers, making it less cost-feasible. This hints towards using LLM-based methods only in scenarios with a small number of seeds per label (5-20).&#10;2) LLM augmentation methods have higher impact on accuracy of less robustly pre-trained classifiers such as DistilBERT or BERT.&#10;3) LLM augmentation methods have higher impact on classifier accuracy for full fine-tuning when compared to QLoRA fine-tuning.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Language Model Evaluation, Classifier Fine-Tuning, Data Science, Artificial Intelligence, Mathematics, Data Augmentation Methods" />
        </attvalues>
      </node>
      <node id="2207.00883" label="2207.00883">
        <attvalues>
          <attvalue for="0" value="Improving Transformer-based Conversational ASR by Inter-Sentential&#10;  Attention Mechanism" />
          <attvalue for="1" value="  Transformer-based models have demonstrated their effectiveness in automatic&#10;speech recognition (ASR) tasks and even shown superior performance over the&#10;conventional hybrid framework. The main idea of Transformers is to capture the&#10;long-range global context within an utterance by self-attention layers.&#10;However, for scenarios like conversational speech, such utterance-level&#10;modeling will neglect contextual dependencies that span across utterances. In&#10;this paper, we propose to explicitly model the inter-sentential information in&#10;a Transformer based end-to-end architecture for conversational speech&#10;recognition. Specifically, for the encoder network, we capture the contexts of&#10;previous speech and incorporate such historic information into current input by&#10;a context-aware residual attention mechanism. For the decoder, the prediction&#10;of current utterance is also conditioned on the historic linguistic information&#10;through a conditional decoder framework. We show the effectiveness of our&#10;proposed method on several open-source dialogue corpora and the proposed method&#10;consistently improved the performance from the utterance-level&#10;Transformer-based ASR models.&#10;" />
          <attvalue for="2" value="&#10;&#10;Context information plays an important role in ASR, especially in scenes that require inter-sentential information such as conversation since semantically related words, or phrases often reoccur across sentences~\cite{kim2019cross}. Typically, traditional hybrid acoustic-language ASR models usually rely on rich language models to model contextual information~\cite{mikolov2010recurrent,mikolov2012context,mnih2007three, ji2015document,liu2017dialog, xiong2018session}. Meanwhile, there are also several researches adopting context information particularly in end-to-end ASR by adding additional context to the decoder or simply concatenate multiple consecutive utterances as the input of an end-to-end model~\cite{kim2018dialog, masumura2019large, masumura2021hierarchical}. &#10;&#10;Transformer~\cite{vaswani2017attention}, as the most successful attention-based end-to-end model, has recently received more attention due to its superior performance on a wide range of tasks including ASR~\cite{wang2019learning, raganato2018analysis, dong2018speech, karita2019comparative, luo2021simplified, gulati2020conformer}.&#10; However, since the computational and memory cost of self-attention is quadratic w.r.t the input sequence length, Transformer is hard to process long sequences and mainly models independent utterances. &#10;&#10;Several studies in natural language processing (NLP) have been explored to utilize the long contextual information for Transformer~\cite{dai2019transformer, rae2019compressive, beltagy2020longformer, zhou2020informer}. Inspired by above studies in the NLP task, some approaches were also proposed to incorporate contextual information across successive input sequences in Transformer-based ASR~\cite{masumura2021hierarchical,hori2020transformer}, but these methods do not solve the problem of the high computational and memory cost, or have high model complexity.&#10;&#10;In this study, we propose a novel Transformer-based architecture to explicitly model the inter-sentential information for conversational ASR. Inspired by~\cite{he2020realformer}, we include a residual attention module in the encoder, which accelerates the convergence speed and well models the long-range global dependencies within each input sequence. Besides, to further transfer the contextual information of previous sentences, we also propose a novel context-aware residual attention module, which transfers contextual information through attention scores. For the decoder part, we use an additional context module to learn more inter-sentential information. By using the methods above, we introduce inter-sentential contextual information in the popular Transformer ASR model. We demonstrate the superiority of our approach on two dialogue benchmarks (speech from two speakers) HKUST and Switchboard, a lecture benchmark TED-LIUM2, and a dialog dataset DATATANG-dialog, with obvious error rate reduction and neglectable increase of computational cost and model complexity.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Conversational Context, Electrical Engineering, Linguistics, Signal Processing, Speech Recognition, Transformer Models" />
        </attvalues>
      </node>
      <node id="1907.10726" label="1907.10726">
        <attvalues>
          <attvalue for="0" value="Cross-Attention End-to-End ASR for Two-Party Conversations" />
          <attvalue for="1" value="  We present an end-to-end speech recognition model that learns interaction&#10;between two speakers based on the turn-changing information. Unlike&#10;conventional speech recognition models, our model exploits two speakers'&#10;history of conversational-context information that spans across multiple turns&#10;within an end-to-end framework. Specifically, we propose a speaker-specific&#10;cross-attention mechanism that can look at the output of the other speaker side&#10;as well as the one of the current speaker for better at recognizing long&#10;conversations. We evaluated the models on the Switchboard conversational speech&#10;corpus and show that our model outperforms standard end-to-end speech&#10;recognition models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1701.04056" label="1701.04056">
        <attvalues>
          <attvalue for="0" value="Dialog Context Language Modeling with Recurrent Neural Networks" />
          <attvalue for="1" value="  In this work, we propose contextual language models that incorporate dialog&#10;level discourse information into language modeling. Previous works on&#10;contextual language model treat preceding utterances as a sequence of inputs,&#10;without considering dialog interactions. We design recurrent neural network&#10;(RNN) based contextual language models that specially track the interactions&#10;between speakers in a dialog. Experiment results on Switchboard Dialog Act&#10;Corpus show that the proposed model outperforms conventional single turn based&#10;RNN language model by 3.3% on perplexity. The proposed models also demonstrate&#10;advantageous performance over other competitive contextual language models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.07935" label="2102.07935">
        <attvalues>
          <attvalue for="0" value="Hierarchical Transformer-based Large-Context End-to-end ASR with&#10;  Large-Context Knowledge Distillation" />
          <attvalue for="1" value="  We present a novel large-context end-to-end automatic speech recognition&#10;(E2E-ASR) model and its effective training method based on knowledge&#10;distillation. Common E2E-ASR models have mainly focused on utterance-level&#10;processing in which each utterance is independently transcribed. On the other&#10;hand, large-context E2E-ASR models, which take into account long-range&#10;sequential contexts beyond utterance boundaries, well handle a sequence of&#10;utterances such as discourses and conversations. However, the transformer&#10;architecture, which has recently achieved state-of-the-art ASR performance&#10;among utterance-level ASR systems, has not yet been introduced into the&#10;large-context ASR systems. We can expect that the transformer architecture can&#10;be leveraged for effectively capturing not only input speech contexts but also&#10;long-range sequential contexts beyond utterance boundaries. Therefore, this&#10;paper proposes a hierarchical transformer-based large-context E2E-ASR model&#10;that combines the transformer architecture with hierarchical encoder-decoder&#10;based large-context modeling. In addition, in order to enable the proposed&#10;model to use long-range sequential contexts, we also propose a large-context&#10;knowledge distillation that distills the knowledge from a pre-trained&#10;large-context language model in the training phase. We evaluate the&#10;effectiveness of the proposed model and proposed training method on Japanese&#10;discourse ASR tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.11747" label="2012.11747">
        <attvalues>
          <attvalue for="0" value="RealFormer: Transformer Likes Residual Attention" />
          <attvalue for="1" value="  Transformer is the backbone of modern NLP models. In this paper, we propose&#10;RealFormer, a simple and generic technique to create Residual Attention Layer&#10;Transformer networks that significantly outperform the canonical Transformer&#10;and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked&#10;Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA,&#10;Natural Questions, and OpenKP. We also observe empirically that RealFormer&#10;stabilizes training and leads to models with sparser attention. Source code and&#10;pre-trained checkpoints for RealFormer can be found at&#10;https://github.com/google-research/google-research/tree/master/realformer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.07093" label="2210.07093">
        <attvalues>
          <attvalue for="0" value="Query Expansion Using Contextual Clue Sampling with Language Models" />
          <attvalue for="1" value="  Query expansion is an effective approach for mitigating vocabulary mismatch&#10;between queries and documents in information retrieval. One recent line of&#10;research uses language models to generate query-related contexts for expansion.&#10;Along this line, we argue that expansion terms from these contexts should&#10;balance two key aspects: diversity and relevance. The obvious way to increase&#10;diversity is to sample multiple contexts from the language model. However, this&#10;comes at the cost of relevance, because there is a well-known tendency of&#10;models to hallucinate incorrect or irrelevant contexts. To balance these two&#10;considerations, we propose a combination of an effective filtering strategy and&#10;fusion of the retrieved documents based on the generation probability of each&#10;context. Our lexical matching based approach achieves a similar top-5/top-20&#10;retrieval accuracy and higher top-100 accuracy compared with the&#10;well-established dense retrieval model DPR, while reducing the index size by&#10;more than 96%. For end-to-end QA, the reader model also benefits from our&#10;method and achieves the highest Exact-Match score against several competitive&#10;baselines.&#10;" />
          <attvalue for="2" value="&#10;Despite the advent of dense retrieval approaches based on semantic matching for open-domain question answering such as DPR~\cite{karpukhin2020dense}, approaches based on lexical matching~(e.g., BM25) remain important due to their space-efficiency and can serve as input to hybrid methods~\cite{gao2021coil,Formal2021SPLADESL,Lin2021AFB}.&#10;&#10;A core challenge for lexical retrieval is the vocabulary mismatch between the query and documents.&#10;Query expansion techniques dating back over half a century have proven effective in overcoming this issue~\cite{salton1971smart}.&#10;The expansion terms are traditionally precomputed from relevant corpora using pseudo-relevance feedback techniques~\cite{salton1971smart, robertson1976relevance, abdul2004umass}.&#10;In recent work, GAR~\cite{mao2021generation} explored removing the query expansion's reliance on an external corpus and instead used a large language model to generate a context.&#10;&#10;We argue that expansion needs to balance two key factors:&#10;(1) Diversity: Given the question, there can be multiple different reasoning paths~(referred to as contextual clues) to reach the correct answer.&#10;(2) Relevance: Simply relying on a single generated context increases the risk of query drift, as the generated context could be semantically irrelevant or contain factual errors~\cite{schutze2008introduction}.&#10;However, simply generating multiple contexts is prone to the hallucination problem -- they can be unfaithful to the input or include false information~\cite{tian2019sticking, maynez2020faithfulness, dziri2021neural}.&#10;Thus, in this work, we wish to explore the question: How can we best generate a sufficiently rich set of contextual clues to answer a query?&#10;&#10;Our proposed solution~(Figure~\ref{fig:intro}) overcomes these problems with two simple and efficient steps: Filtering and fusion.&#10;After sampling top-k outputs from the decoder of the fine-tuned language model, we first cluster these generated contextual clues based on their lexical distance.&#10;In each cluster, where highly similar contextual clues are grouped together, we only keep a single generated output with the highest generation probability.&#10;The filtering step effectively reduces potential factual errors and redundant close duplicates.&#10;The query is then individually augmented with each filtered contextual clue.&#10;We retrieve documents separately for every single augmented query.&#10;As the last step, all the documents are ranked together~(fusion) with the generation probability from the integral contextual clue in the augmented query.&#10;&#10;We evaluate our approach on two established benchmarks: Natural Questions~\cite{kwiatkowski-etal-2019-natural} and TriviaQA~\cite{lee2019latent}.&#10;Our baseline model GAR~\cite{mao2021generation} trails behind its dense retrieval counterpart DPR~\cite{karpukhin2020dense} by a large margin when retrieving a small number of passages. &#10;We bridge this gap and outperform GAR by 3.1\% and 2.9\% on Top-5/Top-20 accuracy on the NQ dataset. &#10;Compared with DPR, our approach outperforms it by 0.6 and 1.0 points on Top-100 accuracy on the two datasets, while requiring 96\% less index storage space.&#10;The accuracy can be further improved by 3.4\% on the basis of DPR's performance when fusing the documents retrieved from DPR and our method together.&#10;Furthermore, our retrieval performance also successfully transfers to downstream question answering tasks, where our methods increase by 3.2\% and 0.8\% Exact Match score compared with the DPR and GAR retrieved documents. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Information Retrieval Systems, Query Expansion Techniques, Artificial Intelligence, Information Science, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2104.07186" label="2104.07186">
        <attvalues>
          <attvalue for="0" value="COIL: Revisit Exact Lexical Match in Information Retrieval with&#10;  Contextualized Inverted List" />
          <attvalue for="1" value="  Classical information retrieval systems such as BM25 rely on exact lexical&#10;match and carry out search efficiently with inverted list index. Recent neural&#10;IR models shifts towards soft semantic matching all query document terms, but&#10;they lose the computation efficiency of exact match systems. This paper&#10;presents COIL, a contextualized exact match retrieval architecture that brings&#10;semantic lexical matching. COIL scoring is based on overlapping query document&#10;tokens' contextualized representations. The new architecture stores&#10;contextualized token representations in inverted lists, bringing together the&#10;efficiency of exact match and the representation power of deep language models.&#10;Our experimental results show COIL outperforms classical lexical retrievers and&#10;state-of-the-art deep LM retrievers with similar or smaller latency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.08553" label="2009.08553">
        <attvalues>
          <attvalue for="0" value="Generation-Augmented Retrieval for Open-domain Question Answering" />
          <attvalue for="1" value="  We propose Generation-Augmented Retrieval (GAR) for answering open-domain&#10;questions, which augments a query through text generation of heuristically&#10;discovered relevant contexts without external resources as supervision. We&#10;demonstrate that the generated contexts substantially enrich the semantics of&#10;the queries and GAR with sparse representations (BM25) achieves comparable or&#10;better performance than state-of-the-art dense retrieval methods such as DPR.&#10;We show that generating diverse contexts for a query is beneficial as fusing&#10;their results consistently yields better retrieval accuracy. Moreover, as&#10;sparse and dense representations are often complementary, GAR can be easily&#10;combined with DPR to achieve even better performance. GAR achieves&#10;state-of-the-art performance on Natural Questions and TriviaQA datasets under&#10;the extractive QA setup when equipped with an extractive reader, and&#10;consistently outperforms other retrieval methods when the same generative&#10;reader is used.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.05800" label="2311.05800">
        <attvalues>
          <attvalue for="0" value="Leveraging LLMs for Synthesizing Training Data Across Many Languages in&#10;  Multilingual Dense Retrieval" />
          <attvalue for="1" value="  There has been limited success for dense retrieval models in multilingual&#10;retrieval, due to uneven and scarce training data available across multiple&#10;languages. Synthetic training data generation is promising (e.g., InPars or&#10;Promptagator), but has been investigated only for English. Therefore, to study&#10;model capabilities across both cross-lingual and monolingual retrieval tasks,&#10;we develop SWIM-IR, a synthetic retrieval training dataset containing 33 (high&#10;to very-low resource) languages for fine-tuning multilingual dense retrievers&#10;without requiring any human supervision. To construct SWIM-IR, we propose SAP&#10;(summarize-then-ask prompting), where the large language model (LLM) generates&#10;a textual summary prior to the query generation step. SAP assists the LLM in&#10;generating informative queries in the target language. Using SWIM-IR, we&#10;explore synthetic fine-tuning of multilingual dense retrieval models and&#10;evaluate them robustly on three retrieval benchmarks: XOR-Retrieve&#10;(cross-lingual), MIRACL (monolingual) and XTREME-UP (cross-lingual). Our&#10;models, called SWIM-X, are competitive with human-supervised dense retrieval&#10;models, e.g., mContriever-X, finding that SWIM-IR can cheaply substitute for&#10;expensive human-labeled retrieval training data. SWIM-IR dataset and SWIM-X&#10;models are available at https://github.com/google-research-datasets/SWIM-IR.&#10;" />
          <attvalue for="2" value="&#10;&#10;Dense retrieval models have demonstrated impressive performance in ad-hoc information retrieval (IR) tasks, e.g., web search, outperforming traditional retrieval systems such as BM25 \cite[inter alia]{karpukhin-etal-2020-dense, lin2021pretrained, Ni2022LargeDE, neelakantan2022text}. &#10;A major reason for its success lies in the availability of large-scale supervised training datasets in English, such as MS MARCO \cite{msmarco} or NQ \cite{nq}, and coupled with effective training strategies, such as custom hard-negative mining \cite{xiong:2021, lin:2023}, or teacher distillation \cite{hoeffstater:2021, ren:2021}.&#10;&#10;However, there is a limited exploration of dense retrieval models in multilingual retrieval, due to uneven and low distribution of human-supervised training data for other languages apart from English \cite{reimers:2020, feng:2022,wieting-etal-2023-beyond}.&#10;Collecting human annotations for training data generation is not scalable, as it is cumbersome to search and hire native speakers, check their language proficiency, and teach them. Additionally, human annotators are expensive, thereby requiring a large annotation budget for generating a sufficient amount of training pairs (cf.~\ref{fig:swim-training-data}).&#10;&#10;Multilingual query generation is a complex task \cite{wang:2021}. It requires understanding of semantic mappings of words across languages, similar to machine translation \cite{forcada:2002, tan-etal-2019-multilingual, zhu:2023}. Recently, utilizing LLMs for query generation has been popular in English \cite{Bonifacio2022InParsUD, Dai2022PromptagatorFD}.&#10;But as illustrated in \ref{fig:sap-vs-standard}, standard prompt templates can lead the LLM to generate either extractive or uninformative queries across languages.&#10;&#10;To improve the quality of the generated query, we propose SAP ({Summarize-then-Ask Prompting}), where we optimize the prompt to break down the query generation with LLM in two stages: (i) {summary extraction}, which identifies the relevant information from the long input passage and extracts the best representative sentences as the summary, and (ii) {query generation}, which generates a multilingual query relevant for the input passage, using the extracted summary (first stage) as the intermediate step. SAP highlights the relevant information within the passage and produces difficult (i.e., informative) queries in the target language.&#10;&#10;In our work, we utilize PaLM 2 \cite{anil2023palm}, a recent multilingual LLM (successor of PaLM 540B \cite{chowdhery2022palm}) for query generation. The generated query paired with the original passage from Wikipedia is used to construct the SWIM-IR dataset.&#10;SWIM-IR provides synthetic training (query-passage) pairs for improving dense retrieval models without requiring any human supervision. The dataset spans across 33 diverse languages, including both high and very-low resource languages and is one of the largest multilingual synthetic training dataset with 28 million training pairs (cf. \ref{tab:dataset-comparison-small}).&#10;&#10;We develop synthetic multilingual (both monolingual and cross-lingual) dense retrieval models called SWIM-X, using mT5 (base) \cite{xue2021mt5} as the backbone and fine-tune on SWIM-IR. We compare SWIM-X against models fine-tuned with human supervision by changing only the training dataset while keeping other, i.e., model parameters and training settings unchanged. &#10;We evaluate on three standard multilingual retrieval benchmarks (two cross-lingual and one monolingual).&#10;As shown in \ref{fig:overall_results}, on XOR-Retrieve \cite{asai-etal-2021-xor}, SWIM-X outperforms the best-supervised baseline (mContriever-X) by 7.1 points at Recall@5kt. On MIRACL \cite{miracl}, a monolingual retrieval benchmark, SWIM-X is inferior to mContriever-X by 9.0 points at nDCG@10, which shows room for future improvement. On XTREME-UP \cite{ruder2023xtremeup}, a challenging benchmark containing 20 underrepresented Indo-European languages, SWIM-X~outperforms mContriever-X by 11.7 points at MRR@10. We publicly open-source SWIM-IR dataset and SWIM-X models at.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Synthetic Data Generation, Dense Retrieval Models, Computer Science, Linguistics, Multilingual Retrieval, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2007.00808" label="2007.00808">
        <attvalues>
          <attvalue for="0" value="Approximate Nearest Neighbor Negative Contrastive Learning for Dense&#10;  Text Retrieval" />
          <attvalue for="1" value="  Conducting text retrieval in a dense learned representation space has many&#10;intriguing advantages over sparse retrieval. Yet the effectiveness of dense&#10;retrieval (DR) often requires combination with sparse retrieval. In this paper,&#10;we identify that the main bottleneck is in the training mechanisms, where the&#10;negative instances used in training are not representative of the irrelevant&#10;documents in testing. This paper presents Approximate nearest neighbor Negative&#10;Contrastive Estimation (ANCE), a training mechanism that constructs negatives&#10;from an Approximate Nearest Neighbor (ANN) index of the corpus, which is&#10;parallelly updated with the learning process to select more realistic negative&#10;training instances. This fundamentally resolves the discrepancy between the&#10;data distribution used in the training and testing of DR. In our experiments,&#10;ANCE boosts the BERT-Siamese DR model to outperform all competitive dense and&#10;sparse retrieval baselines. It nearly matches the accuracy of&#10;sparse-retrieval-and-BERT-reranking using dot-product in the ANCE-learned&#10;representation space and provides almost 100x speed-up.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.06967" label="2104.06967">
        <attvalues>
          <attvalue for="0" value="Efficiently Teaching an Effective Dense Retriever with Balanced Topic&#10;  Aware Sampling" />
          <attvalue for="1" value="  A vital step towards the widespread adoption of neural retrieval models is&#10;their resource efficiency throughout the training, indexing and query&#10;workflows. The neural IR community made great advancements in training&#10;effective dual-encoder dense retrieval (DR) models recently. A dense text&#10;retrieval model uses a single vector representation per query and passage to&#10;score a match, which enables low-latency first stage retrieval with a nearest&#10;neighbor search. Increasingly common, training approaches require enormous&#10;compute power, as they either conduct negative passage sampling out of a&#10;continuously updating refreshing index or require very large batch sizes for&#10;in-batch negative sampling. Instead of relying on more compute capability, we&#10;introduce an efficient topic-aware query and balanced margin sampling&#10;technique, called TAS-Balanced. We cluster queries once before training and&#10;sample queries out of a cluster per batch. We train our lightweight 6-layer DR&#10;model with a novel dual-teacher supervision that combines pairwise and in-batch&#10;negative teachers. Our method is trainable on a single consumer-grade GPU in&#10;under 48 hours (as opposed to a common configuration of 8x V100s). We show that&#10;our TAS-Balanced training method achieves state-of-the-art low-latency (64ms&#10;per query) results on two TREC Deep Learning Track query sets. Evaluated on&#10;NDCG@10, we outperform BM25 by 44%, a plainly trained DR by 19%, docT5query by&#10;11%, and the previous best DR model by 5%. Additionally, TAS-Balanced produces&#10;the first dense retriever that outperforms every other method on recall at any&#10;cutoff on TREC-DL and allows more resource intensive re-ranking models to&#10;operate on fewer passages to improve results further.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.10726" label="2212.10726">
        <attvalues>
          <attvalue for="0" value="Beyond Contrastive Learning: A Variational Generative Model for&#10;  Multilingual Retrieval" />
          <attvalue for="1" value="  Contrastive learning has been successfully used for retrieval of semantically&#10;aligned sentences, but it often requires large batch sizes or careful&#10;engineering to work well. In this paper, we instead propose a generative model&#10;for learning multilingual text embeddings which can be used to retrieve or&#10;score sentence pairs. Our model operates on parallel data in $N$ languages and,&#10;through an approximation we introduce, efficiently encourages source separation&#10;in this multilingual setting, separating semantic information that is shared&#10;between translations from stylistic or language-specific variation. We show&#10;careful large-scale comparisons between contrastive and generation-based&#10;approaches for learning multilingual text embeddings, a comparison that has not&#10;been done to the best of our knowledge despite the popularity of these&#10;approaches. We evaluate this method on a suite of tasks including semantic&#10;similarity, bitext mining, and cross-lingual question retrieval -- the last of&#10;which we introduce in this paper. Overall, our Variational Multilingual&#10;Source-Separation Transformer (VMSST) model outperforms both a strong&#10;contrastive and generative baseline on these tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.11938" label="2305.11938">
        <attvalues>
          <attvalue for="0" value="XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented&#10;  Languages" />
          <attvalue for="1" value="  Data scarcity is a crucial issue for the development of highly multilingual&#10;NLP systems. Yet for many under-represented languages (ULs) -- languages for&#10;which NLP re-search is particularly far behind in meeting user needs -- it is&#10;feasible to annotate small amounts of data. Motivated by this, we propose&#10;XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather&#10;than zero-shot; its focus on user-centric tasks -- tasks with broad adoption by&#10;speakers of high-resource languages; and its focus on under-represented&#10;languages where this scarce-data scenario tends to be most realistic. XTREME-UP&#10;evaluates the capabilities of language models across 88 under-represented&#10;languages over 9 key user-centric technologies including ASR, OCR, MT, and&#10;information access tasks that are of general utility. We create new datasets&#10;for OCR, autocomplete, semantic parsing, and transliteration, and build on and&#10;refine existing datasets for other tasks. XTREME-UP provides methodology for&#10;evaluating many modeling scenarios including text-only, multi-modal (vision,&#10;audio, and text),supervised parameter tuning, and in-context learning. We&#10;evaluate commonly used models on the benchmark. We release all code and scripts&#10;to train and evaluate models&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.08137" label="2111.08137">
        <attvalues>
          <attvalue for="0" value="Joint Unsupervised and Supervised Training for Multilingual ASR" />
          <attvalue for="1" value="  Self-supervised training has shown promising gains in pretraining models and&#10;facilitating the downstream finetuning for speech recognition, like&#10;multilingual ASR. Most existing methods adopt a 2-stage scheme where the&#10;self-supervised loss is optimized in the first pretraining stage, and the&#10;standard supervised finetuning resumes in the second stage. In this paper, we&#10;propose an end-to-end (E2E) Joint Unsupervised and Supervised Training (JUST)&#10;method to combine the supervised RNN-T loss and the self-supervised contrastive&#10;and masked language modeling (MLM) losses. We validate its performance on the&#10;public dataset Multilingual LibriSpeech (MLS), which includes 8 languages and&#10;is extremely imbalanced. On MLS, we explore (1) JUST trained from scratch, and&#10;(2) JUST finetuned from a pretrained checkpoint. Experiments show that JUST can&#10;consistently outperform other existing state-of-the-art methods, and beat the&#10;monolingual baseline by a significant margin, demonstrating JUST's capability&#10;of handling low-resource languages in multilingual ASR. Our average WER of all&#10;languages outperforms average monolingual baseline by 33.3%, and the&#10;state-of-the-art 2-stage XLSR by 32%. On low-resource languages like Polish,&#10;our WER is less than half of the monolingual baseline and even beats the&#10;supervised transfer learning method which uses external supervision.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;Early works adopted joint training to learn robust and transferable representations. In NLP, \cite{cheng2021self} proposes joint training for machine translation. \cite{gururangan2020don} suggests multiple pretraining objectives for domain-adaptive applications. In speech, PASE \cite{pascual2019learning} jointly solves multiple self-supervised tasks to learn general representations. More recent research found the joint training with both supervised and unsupervised losses can directly optimize the ASR performance. \cite{talnikar2021joint} alternatively minimizes an unsupervised masked CPC loss and a supervised CTC loss \cite{graves2006connectionist}. This single-stage method is shown to match the performance of the two-stage w2v2 on the Librispeech 100-hours dataset. Similarly, UniSpeech \cite{wang2021unispeech} optimizes a combination of phonetic CTC loss and contrastive loss. To further increase the quantizer codebook usage, UniSpeech randomly replaces contextual representations with quantized latent codes. \cite{raghavanhybrid} also designs a similar hybrid multitask learning to train acoustic models under low-resource settings, comprising of supervised CTC, attention and self-supervised reconstruction losses. Similarly, \cite{hwang2021large} combines self- and semi-supervised learning methods for online ASR model. These methods only contain one self-supervised loss in their optimization and often tackle with speech recognition in the phoneme level \cite{wang2021unispeech,raghavanhybrid}. JUST incorporates two self-supervised losses (contrastive and MLM losses), and replaces the CTC loss with an RNN-T loss. RNN-T extends CTC with a prediction network to simulate the effect of LM and has been widely adopted in prior multilingual ASR systems \cite{li2021scaling}.&#10;Furthermore, unlike \cite{pascual2019learning,raghavanhybrid} where each of the multiple tasks has its own branch, JUST computes different losses simply using the intermediate outputs from different layers (Fig.~\ref{fig:just}). &#10; " />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Signal Processing, Speech Recognition, Self-Supervised Learning, Multilingual Training" />
        </attvalues>
      </node>
      <node id="2011.04491" label="2011.04491">
        <attvalues>
          <attvalue for="0" value="Masked Proxy Loss For Text-Independent Speaker Verification" />
          <attvalue for="1" value="  Open-set speaker recognition can be regarded as a metric learning problem,&#10;which is to maximize inter-class variance and minimize intra-class variance.&#10;Supervised metric learning can be categorized into entity-based learning and&#10;proxy-based learning. Most of the existing metric learning objectives like&#10;Contrastive, Triplet, Prototypical, GE2E, etc all belong to the former&#10;division, the performance of which is either highly dependent on sample mining&#10;strategy or restricted by insufficient label information in the mini-batch.&#10;Proxy-based losses mitigate both shortcomings, however, fine-grained&#10;connections among entities are either not or indirectly leveraged. This paper&#10;proposes a Masked Proxy (MP) loss which directly incorporates both proxy-based&#10;relationships and pair-based relationships. We further propose Multinomial&#10;Masked Proxy (MMP) loss to leverage the hardness of speaker pairs. These&#10;methods have been applied to evaluate on VoxCeleb test set and reach&#10;state-of-the-art Equal Error Rate(EER).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1609.03296" label="1609.03296">
        <attvalues>
          <attvalue for="0" value="A Neural Network Alternative to Non-Negative Audio Models" />
          <attvalue for="1" value="  We present a neural network that can act as an equivalent to a Non-Negative&#10;Matrix Factorization (NMF), and further show how it can be used to perform&#10;supervised source separation. Due to the extensibility of this approach we show&#10;how we can achieve better source separation performance as compared to&#10;NMF-based methods, and propose a variety of derivative architectures that can&#10;be used for further improvements.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.15494" label="2310.15494">
        <attvalues>
          <attvalue for="0" value="TRAMS: Training-free Memory Selection for Long-range Language Modeling" />
          <attvalue for="1" value="  The Transformer architecture is crucial for numerous AI models, but it still&#10;faces challenges in long-range language modeling. Though several specific&#10;transformer architectures have been designed to tackle issues of long-range&#10;dependencies, existing methods like Transformer-XL are plagued by a high&#10;percentage of ineffective memories. In this study, we present a plug-and-play&#10;strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens&#10;participating in attention calculation based on one simple metric. This&#10;strategy allows us to keep tokens that are likely to have a high attention&#10;score with the current queries and ignore the other ones. We have tested our&#10;approach on the word-level benchmark (WikiText-103) and the character-level&#10;benchmark (enwik8), and the results indicate an improvement without having&#10;additional training or adding additional parameters.&#10;" />
          <attvalue for="2" value="&#10;Transformer-based models~\cite{bert, roberta, T5, albert, gpt} have achieved remarkable performance over the past few years. The key component of these model architectures is the attention mechanism~\cite{vanilla-transformer}. However, the original attention design struggles to efficiently handle long sequences, which becomes particularly problematic in scenarios such as document-level translation~\cite{werlen2018document, kim2019and} and large-scale text generation~\cite{zhou2023recurrentgpt}, as its time and space computation costs increase quadratically with the sequence length \cite{tay2022efficient}. The primary factor for this elevated computational complexity can be traced back to the multiplication between queries and keys used in the attention module. In general, the time complexity for calculation is $\mathcal{O}(N^2d)$ if a transformer model with $d$ dimensions is set up with an input consisting of $N$ tokens.&#10;&#10;To tackle this computation bottleneck, numerous efforts have been made. The first line of work is to find a new efficient expression to compute the attention score. Despite the advancements made, these methods often compromise performance, thus paving the way for alternative solutions. Efficient architectures that provide an approximate expression of attention have been explored widely~\cite{wang2020linformer, rfa, peng2022abc,choromanski2021hybrid, zheng2022efficient, zheng2022linear}. The second line of work is to keep the calculation expression the same and use an external structure like hash function~\cite{kitaev2019reformer, daras2020smyrf}, clustering~\cite{routing-transformer, vyas2020fast} and memory selector~\cite{pietruszka2022sparsifying, transformer-xl, bertsch2023unlimiformer, expire-span, adaptive-span, child2019generating} to find the suitable subset of queries and keys in the long sequence for attention calculation.&#10;&#10;Our work falls into the second category, in which we propose a training-free memory selection mechanism to select suitable tokens for attention computation. Specifically, we focus on pushing Transformer-XL~\cite{transformer-xl} architecture to a better position by selecting higher-quality tokens inside its memory. Based on our initial investigation, we construct a memory subset by selecting 50\% of the memories with the largest attention values and maintaining the same performance. It indicates that a large portion of information in memory is not fully utilized. This motivates us to explore better methods to optimize memory usage.&#10;&#10;Illustrated in Figure~\ref{fig:transformer-xl}, we propose a TRAining-free Memory Selection method ({TRAMS}) that can be directly plugged into memory-based long-range language models and reduces the time complexity of computing attention matrix. Through experiments on two language modeling benchmark datasets, namely word-level WikiText-103~\cite{wikitext103} and character-level enwik8~\cite{enwik8}, we achieve an improvement in the model's performance, as demonstrated by a 0.19 perplexity (ppl) drop in WikiText-103 and a 0.017 reduction in bits-per-character (bpc) in enwik8.&#10;&#10;To our knowledge, we are the first to design a training-free memory selection method based on Transformer-XL architecture.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Mathematics, Attention Mechanisms, Language Modeling, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="2305.13304" label="2305.13304">
        <attvalues>
          <attvalue for="0" value="RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text" />
          <attvalue for="1" value="  The fixed-size context of Transformer makes GPT models incapable of&#10;generating arbitrarily long text. In this paper, we introduce RecurrentGPT, a&#10;language-based simulacrum of the recurrence mechanism in RNNs. RecurrentGPT is&#10;built upon a large language model (LLM) such as ChatGPT and uses natural&#10;language to simulate the Long Short-Term Memory mechanism in an LSTM. At each&#10;timestep, RecurrentGPT generates a paragraph of text and updates its&#10;language-based long-short term memory stored on the hard drive and the prompt,&#10;respectively. This recurrence mechanism enables RecurrentGPT to generate texts&#10;of arbitrary length without forgetting. Since human users can easily observe&#10;and edit the natural language memories, RecurrentGPT is interpretable and&#10;enables interactive generation of long text. RecurrentGPT is an initial step&#10;towards next-generation computer-assisted writing systems beyond local editing&#10;suggestions. In addition to producing AI-generated content (AIGC), we also&#10;demonstrate the possibility of using RecurrentGPT as an interactive fiction&#10;that directly interacts with consumers. We call this usage of generative models&#10;by ``AI As Contents'' (AIAC), which we believe is the next form of conventional&#10;AIGC. We further demonstrate the possibility of using RecurrentGPT to create&#10;personalized interactive fiction that directly interacts with readers instead&#10;of interacting with writers. More broadly, RecurrentGPT demonstrates the&#10;utility of borrowing ideas from popular model designs in cognitive science and&#10;deep learning for prompting LLMs. Our code is available at&#10;https://github.com/aiwaves-cn/RecurrentGPT and an online demo is available at&#10;https://www.aiwaves.org/recurrentgpt.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04367" label="2110.04367">
        <attvalues>
          <attvalue for="0" value="Hybrid Random Features" />
          <attvalue for="1" value="  We propose a new class of random feature methods for linearizing softmax and&#10;Gaussian kernels called hybrid random features (HRFs) that automatically adapt&#10;the quality of kernel estimation to provide most accurate approximation in the&#10;defined regions of interest. Special instantiations of HRFs lead to well-known&#10;methods such as trigonometric (Rahimi and Recht, 2007) or (recently introduced&#10;in the context of linear-attention Transformers) positive random features&#10;(Choromanski et al., 2021). By generalizing Bochner's Theorem for&#10;softmax/Gaussian kernels and leveraging random features for compositional&#10;kernels, the HRF-mechanism provides strong theoretical guarantees - unbiased&#10;approximation and strictly smaller worst-case relative errors than its&#10;counterparts. We conduct exhaustive empirical evaluation of HRF ranging from&#10;pointwise kernel estimation experiments, through tests on data admitting&#10;clustering structure to benchmarking implicit-attention Transformers (also for&#10;downstream Robotics applications), demonstrating its quality in a wide spectrum&#10;of machine learning problems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2003.05997" label="2003.05997">
        <attvalues>
          <attvalue for="0" value="Efficient Content-Based Sparse Attention with Routing Transformers" />
          <attvalue for="1" value="  Self-attention has recently been adopted for a wide range of sequence&#10;modeling problems. Despite its effectiveness, self-attention suffers from&#10;quadratic compute and memory requirements with respect to sequence length.&#10;Successful approaches to reduce this complexity focused on attending to local&#10;sliding windows or a small set of locations independent of content. Our work&#10;proposes to learn dynamic sparse attention patterns that avoid allocating&#10;computation and memory to attend to content unrelated to the query of interest.&#10;This work builds upon two lines of research: it combines the modeling&#10;flexibility of prior work on content-based sparse attention with the efficiency&#10;gains from approaches based on local, temporal sparse attention. Our model, the&#10;Routing Transformer, endows self-attention with a sparse routing module based&#10;on online k-means while reducing the overall complexity of attention to&#10;$O\left(n^{1.5}d\right)$ from $O\left(n^2d\right)$ for sequence length $n$ and&#10;hidden dimension $d$. We show that our model outperforms comparable sparse&#10;attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity)&#10;as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while&#10;using fewer self-attention layers. Additionally, we set a new state-of-the-art&#10;on the newly released PG-19 data-set, obtaining a test perplexity of 33.2 with&#10;a 22 layer Routing Transformer model trained on sequences of length 8192.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.05169" label="2009.05169">
        <attvalues>
          <attvalue for="0" value="Sparsifying Transformer Models with Trainable Representation Pooling" />
          <attvalue for="1" value="  We propose a novel method to sparsify attention in the Transformer model by&#10;learning to select the most-informative token representations during the&#10;training process, thus focusing on the task-specific parts of an input. A&#10;reduction of quadratic time and memory complexity to sublinear was achieved due&#10;to a robust trainable top-$k$ operator. Our experiments on a challenging long&#10;document summarization task show that even our simple baseline performs&#10;comparably to the current SOTA, and with trainable pooling, we can retain its&#10;top quality, while being $1.8\times$ faster during training, $4.5\times$ faster&#10;during inference, and up to $13\times$ more computationally efficient in the&#10;decoder.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.11037" label="2406.11037">
        <attvalues>
          <attvalue for="0" value="NAST: Noise Aware Speech Tokenization for Speech Language Models" />
          <attvalue for="1" value="  Speech tokenization is the task of representing speech signals as a sequence&#10;of discrete units. Such representations can be later used for various&#10;downstream tasks including automatic speech recognition, text-to-speech, etc.&#10;More relevant to this study, such representation serves as the basis of Speech&#10;Language Models. In this work, we tackle the task of speech tokenization under&#10;the noisy setup and present NAST: Noise Aware Speech Tokenization for Speech&#10;Language Models. NAST is composed of three main components: (i) a predictor;&#10;(ii) a residual encoder; and (iii) a decoder. We evaluate the efficiency of&#10;NAST considering several spoken language modeling tasks and show that NAST is&#10;superior to the evaluated baselines across all setups. Lastly, we analyze NAST&#10;and show its disentanglement properties and robustness to signal variations in&#10;the form of noise, reverberation, pitch-shift, and time-stretch. Code and&#10;pre-trained models are available at https://github.com/ShovalMessica/NAST.&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-supervised models have shown to be highly effective in extracting meaningful representations from raw speech signals~\cite{hubert, chen2022wavlm, baevski2020wav2vec, mohamed2022self}. Recently, the authors in~\cite{on_generative} demonstrated that such self-supervised representations can be used under the Generative Spoken Language Modeling (GSLM) framework.&#10;&#10;The GSLM pipeline typically starts with a self-supervised learning model that extracts continuous speech embeddings. These embeddings are then quantized into a discrete form, often using the k-means algorithm ~\cite{on_generative, kharitonov2021text, borsos2022audiolm}. A speech-language model is subsequently trained on these quantized units, which are finally converted back into raw audio through a unit-based neural vocoder. This framework was shown to be effective in modeling multiple levels of the speech utterance: prosody, content~\cite{on_generative, kharitonov2021text, borsos2022audiolm, kharitonov2023speak}, speech compression and enhancement~\cite{polyak2021speech, wang2023selm, erdogan2023tokensplit}, voice and emotion conversion~\cite{kreuk2021textless, maimon2023speaking}, spoken dialogue~\cite{nguyen2022generative}, and speech-to-speech translation~\cite{lee2021direct, popuri2022enhanced, lee2022textless, wang2023speech}. &#10;&#10;Despite their effectiveness, recent findings highlight a susceptibility of such techniques to acoustic variations that do not affect the linguistic content but greatly modify the output representation~\cite{gat-etal-2023-augmentation}, hence questioning their robustness and generalization. For instance, performing a time-stretch of less than 10\% of the speech utterance yields an edit distance of more than 40\%. The authors in~\cite{gat-etal-2023-augmentation} proposed an augmentation invariant speech tokenizer together with an objective evaluation metric to track progress in the field. Although providing impressive results, the method proposed by~\cite{gat-etal-2023-augmentation} is based on a teacher-student paradigm with k-means being the teacher. Hence, inherits k-means properties and bias.&#10;&#10;To confront the above-mentioned issue, in this work, we propose a novel speech tokenization method named NAST, which stands for Noise Aware Speech Tokenization for Speech Language Models. NAST is composed of three main components: (i) a predictor that maps the speech signal into local discrete representations. Such representation mainly captures local information in the form of phonemes or sub-phonemes; (ii) a residual encoder which predicts a single global representation for the whole sequence. This representation mainly captures global information such as speaker identification; and (iii) a decoder which outputs the original signal representation given both local and global representations. To improve invariance to signal variations we match the representations obtained from the predictor module of both clean and augmented speech signals. All modules are jointly optimized using several loss functions. Our hypothesis is that a representation that truly embodies the phonemic structure of speech will exhibit greater resilience, maintaining the spoken content even when faced with augmentations. Figure~\ref{fig:nast} visually depicts NAST.&#10;&#10;We evaluate NAST's invariance to signal variations (i.e., time-stretch, pitch-shift, additive-noise, and reverberation), encoding capabilities (ABX), together with zero-shot sequence modeling evaluations, i.e., sWUGGY, sBLIMP~\cite{nguyen2020zero}, and Spoken StoryCloze~\cite{hassid2023textually}. Results suggest NAST is comparable or superior to the evaluated baselines across all evaluation methods, considering various cluster numbers. We additionally analyze the learned representation considering speaker information and invariance to noise. These results shed light on the properties captured by the proposed speech tokenizer. &#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Noise Robustness, Speech Tokenization, Computer Science, Linguistics, Signal Processing, Language Modeling" />
        </attvalues>
      </node>
      <node id="2203.16502" label="2203.16502">
        <attvalues>
          <attvalue for="0" value="Generative Spoken Dialogue Language Modeling" />
          <attvalue for="1" value="  We introduce dGSLM, the first &quot;textless&quot; model able to generate audio samples&#10;of naturalistic spoken dialogues. It uses recent work on unsupervised spoken&#10;unit discovery coupled with a dual-tower transformer architecture with&#10;cross-attention trained on 2000 hours of two-channel raw conversational audio&#10;(Fisher dataset) without any text or labels. We show that our model is able to&#10;generate speech, laughter and other paralinguistic signals in the two channels&#10;simultaneously and reproduces more naturalistic and fluid turn-taking compared&#10;to a text-based cascaded model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.05604" label="2107.05604">
        <attvalues>
          <attvalue for="0" value="Direct speech-to-speech translation with discrete units" />
          <attvalue for="1" value="  We present a direct speech-to-speech translation (S2ST) model that translates&#10;speech from one language to speech in another language without relying on&#10;intermediate text generation. We tackle the problem by first applying a&#10;self-supervised discrete speech encoder on the target speech and then training&#10;a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the&#10;discrete representations of the target speech. When target text transcripts are&#10;available, we design a joint speech and text training framework that enables&#10;the model to generate dual modality output (speech and text) simultaneously in&#10;the same inference pass. Experiments on the Fisher Spanish-English dataset show&#10;that the proposed framework yields improvement of 6.7 BLEU compared with a&#10;baseline direct S2ST model that predicts spectrogram features. When trained&#10;without any text transcripts, our model performance is comparable to models&#10;that predict spectrograms and are trained with text supervision, showing the&#10;potential of our system for translation between unwritten languages. Audio&#10;samples are available at&#10;https://facebookresearch.github.io/speech_translation/direct_s2st_units/index.html .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.02527" label="1610.02527">
        <attvalues>
          <attvalue for="0" value="Federated Optimization: Distributed Machine Learning for On-Device&#10;  Intelligence" />
          <attvalue for="1" value="  We introduce a new and increasingly relevant setting for distributed&#10;optimization in machine learning, where the data defining the optimization are&#10;unevenly distributed over an extremely large number of nodes. The goal is to&#10;train a high-quality centralized model. We refer to this setting as Federated&#10;Optimization. In this setting, communication efficiency is of the utmost&#10;importance and minimizing the number of rounds of communication is the&#10;principal goal.&#10;  A motivating example arises when we keep the training data locally on users'&#10;mobile devices instead of logging it to a data center for training. In&#10;federated optimziation, the devices are used as compute nodes performing&#10;computation on their local data in order to update a global model. We suppose&#10;that we have extremely large number of devices in the network --- as many as&#10;the number of users of a given service, each of which has only a tiny fraction&#10;of the total data available. In particular, we expect the number of data points&#10;available locally to be much smaller than the number of devices. Additionally,&#10;since different users generate data with different patterns, it is reasonable&#10;to assume that no device has a representative sample of the overall&#10;distribution.&#10;  We show that existing algorithms are not suitable for this setting, and&#10;propose a new algorithm which shows encouraging experimental results for sparse&#10;convex problems. This work also sets a path for future research needed in the&#10;context of \federated optimization.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.07876" label="1802.07876">
        <attvalues>
          <attvalue for="0" value="Federated Meta-Learning with Fast Convergence and Efficient&#10;  Communication" />
          <attvalue for="1" value="  Statistical and systematic challenges in collaboratively training machine&#10;learning models across distributed networks of mobile devices have been the&#10;bottlenecks in the real-world application of federated learning. In this work,&#10;we show that meta-learning is a natural choice to handle these issues, and&#10;propose a federated meta-learning framework FedMeta, where a parameterized&#10;algorithm (or meta-learner) is shared, instead of a global model in previous&#10;approaches. We conduct an extensive empirical evaluation on LEAF datasets and a&#10;real-world production dataset, and demonstrate that FedMeta achieves a&#10;reduction in required communication cost by 2.82-4.33 times with faster&#10;convergence, and an increase in accuracy by 3.23%-14.84% as compared to&#10;Federated Averaging (FedAvg) which is a leading optimization algorithm in&#10;federated learning. Moreover, FedMeta preserves user privacy since only the&#10;parameterized algorithm is transmitted between mobile devices and central&#10;servers, and no raw data is collected onto the servers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.05512" label="1810.05512">
        <attvalues>
          <attvalue for="0" value="Federated Learning for Keyword Spotting" />
          <attvalue for="1" value="  We propose a practical approach based on federated learning to solve&#10;out-of-domain issues with continuously running embedded speech-based models&#10;such as wake word detectors. We conduct an extensive empirical study of the&#10;federated averaging algorithm for the &quot;Hey Snips&quot; wake word based on a&#10;crowdsourced dataset that mimics a federation of wake word users. We&#10;empirically demonstrate that using an adaptive averaging strategy inspired from&#10;Adam in place of standard weighted model averaging highly reduces the number of&#10;communication rounds required to reach our target performance. The associated&#10;upstream communication costs per user are estimated at 8 MB, which is a&#10;reasonable in the context of smart home voice assistants. Additionally, the&#10;dataset used for these experiments is being open sourced with the aim of&#10;fostering further transparent research in the application of federated learning&#10;to speech data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.06270" label="1911.06270">
        <attvalues>
          <attvalue for="0" value="Federated Learning for Healthcare Informatics" />
          <attvalue for="1" value="  With the rapid development of computer software and hardware technologies,&#10;more and more healthcare data are becoming readily available from clinical&#10;institutions, patients, insurance companies and pharmaceutical industries,&#10;among others. This access provides an unprecedented opportunity for data&#10;science technologies to derive data-driven insights and improve the quality of&#10;care delivery. Healthcare data, however, are usually fragmented and private&#10;making it difficult to generate robust results across populations. For example,&#10;different hospitals own the electronic health records (EHR) of different&#10;patient populations and these records are difficult to share across hospitals&#10;because of their sensitive nature. This creates a big barrier for developing&#10;effective analytical approaches that are generalizable, which need diverse,&#10;&quot;big data&quot;. Federated learning, a mechanism of training a shared global model&#10;with a central server while keeping all the sensitive data in local&#10;institutions where the data belong, provides great promise to connect the&#10;fragmented healthcare data sources with privacy-preservation. The goal of this&#10;survey is to provide a review for federated learning technologies, particularly&#10;within the biomedical space. In particular, we summarize the general solutions&#10;to the statistical challenges, system challenges and privacy issues in&#10;federated learning, and point out the implications and potentials in&#10;healthcare.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1812.07108" label="1812.07108">
        <attvalues>
          <attvalue for="0" value="Learning Private Neural Language Modeling with Attentive Aggregation" />
          <attvalue for="1" value="  Mobile keyboard suggestion is typically regarded as a word-level language&#10;modeling problem. Centralized machine learning technique requires massive user&#10;data collected to train on, which may impose privacy concerns for sensitive&#10;personal typing data of users. Federated learning (FL) provides a promising&#10;approach to learning private language modeling for intelligent personalized&#10;keyboard suggestion by training models in distributed clients rather than&#10;training in a central server. To obtain a global model for prediction, existing&#10;FL algorithms simply average the client models and ignore the importance of&#10;each client during model aggregation. Furthermore, there is no optimization for&#10;learning a well-generalized global model on the central server. To solve these&#10;problems, we propose a novel model aggregation with the attention mechanism&#10;considering the contribution of clients models to the global model, together&#10;with an optimization technique during server aggregation. Our proposed&#10;attentive aggregation method minimizes the weighted distance between the server&#10;model and client models through iterative parameters updating while attends the&#10;distance between the server model and client models. Through experiments on two&#10;popular language modeling datasets and a social media dataset, our proposed&#10;method outperforms its counterparts in terms of perplexity and communication&#10;cost in most settings of comparison.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.01623" label="2408.01623">
        <attvalues>
          <attvalue for="0" value="Dialog Flow Induction for Constrainable LLM-Based Chatbots" />
          <attvalue for="1" value="  LLM-driven dialog systems are used in a diverse set of applications, ranging&#10;from healthcare to customer service. However, given their generalization&#10;capability, it is difficult to ensure that these chatbots stay within the&#10;boundaries of the specialized domains, potentially resulting in inaccurate&#10;information and irrelevant responses. This paper introduces an unsupervised&#10;approach for automatically inducing domain-specific dialog flows that can be&#10;used to constrain LLM-based chatbots. We introduce two variants of dialog flow&#10;based on the availability of in-domain conversation instances. Through human&#10;and automatic evaluation over various dialog domains, we demonstrate that our&#10;high-quality data-guided dialog flows achieve better domain coverage, thereby&#10;overcoming the need for extensive manual crafting of such flows.&#10;" />
          <attvalue for="2" value="&#10;&#10;The widespread use of Large Language Models (LLMs)~\cite{openai2023gpt4} for chatbots, highlighted by their human-like conversational abilities across many topics, faces challenges in specialized domains due to their tendency to go off-topic. This generalization capability, while a strength, necessitates the development of more effective control mechanisms to ensure chatbots remain within the desired domain of conversation, especially in specialized fields such as healthcare or legal advice.&#10;Controlling LLM-based chatbots can be effectively managed through dialog flows or schemas~\cite{bohus2009ravenclaw, Mosig2020STARAS}, which structure conversations along predefined paths of dialog actions, acting as directed graphs where nodes represent actions by the user or bot, and edges are the transitions between actions. This structure helps steer the conversation, keeping it within relevant topics, and also enables chatbots to adapt to new tasks or domains without prior training~\cite{zhao-etal-2023-anytod}.&#10;&#10;However, the construction of precise dialog flows is challenging~\cite{huang2020challenges}, given the diversity of dialog in different domains. The most prevalent approaches~\cite{mehri2021schema, zhao-etal-2023-anytod} use schemas that are carefully handcrafted by the dialog system developers. The design of dialog schemas thus has significant manual overhead for developers, resulting in scalability and coverage limitations~\cite{zhang2020recent}.&#10;&#10;This paper introduces an unsupervised method to generate domain-specific dialog flows, exploiting GPT-4's knowledge to systematically create detailed dialog flows reflecting conversational patterns in various domains. We begin by prompting GPT-4 to produce a structured representation of dialog interactions between users and bots, and then further refine this through self-reflective feedback based on a set of predefined criteria (see figure \ref{fig:llm_flow}).&#10;&#10;Further, when we have domain-specific conversations, our approach automatically identifies distinct user and bot dialog actions within these conversations (see figure \ref{fig:data_driven_flow}). These dialog actions, along with selected conversations that exemplify each action, are used to condition the GPT-4 prompt to ensure the dialog flows are grounded using actual domain instances.&#10;This approach enables the automated creation of structured dialog flows, facilitating the development of effective domain-specific chatbots that adhere to their domain's conversational boundaries.&#10;Our main contributions are:&#10;\begin{itemize}[noitemsep]&#10; \item This paper introduces an approach for automatically constructing dialog flows for various domains in an unsupervised manner. &#10; \item The proposed method uses a multi-step framework, that can further leverage domain-specific dialog instances, leading to a graph-like flow illustrating the structure of conversations in the domain.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Chatbot Optimization, Computer Science, Linguistics, Domain Adaptation, Artificial Intelligence, Natural Language Processing, Dialog Systems" />
        </attvalues>
      </node>
      <node id="2212.09939" label="2212.09939">
        <attvalues>
          <attvalue for="0" value="AnyTOD: A Programmable Task-Oriented Dialog System" />
          <attvalue for="1" value="  We propose AnyTOD, an end-to-end, zero-shot task-oriented dialog (TOD) system&#10;capable of handling unseen tasks without task-specific training. We view TOD as&#10;a program executed by a language model (LM), where program logic and ontology&#10;is provided by a designer as a schema. To enable generalization to unseen&#10;schemas and programs without prior training, AnyTOD adopts a neuro-symbolic&#10;approach. A neural LM keeps track of events occurring during a conversation and&#10;a symbolic program implementing the dialog policy is executed to recommend next&#10;actions AnyTOD should take. This approach drastically reduces data annotation&#10;and model training requirements, addressing the enduring challenge of rapidly&#10;adapting a TOD system to unseen tasks and domains. We demonstrate&#10;state-of-the-art results on STAR, ABCD and SGD benchmarks. We also demonstrate&#10;strong zero-shot transfer ability in low-resource settings, such as zero-shot&#10;on MultiWOZ. In addition, we release STARv2, an updated version of the STAR&#10;dataset with richer annotations, for benchmarking zero-shot end-to-end TOD&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.06262" label="2012.06262">
        <attvalues>
          <attvalue for="0" value="Morphology Matters: A Multilingual Language Modeling Analysis" />
          <attvalue for="1" value="  Prior studies in multilingual language modeling (e.g., Cotterell et al.,&#10;2018; Mielke et al., 2019) disagree on whether or not inflectional morphology&#10;makes languages harder to model. We attempt to resolve the disagreement and&#10;extend those studies. We compile a larger corpus of 145 Bible translations in&#10;92 languages and a larger number of typological features. We fill in missing&#10;typological data for several languages and consider corpus-based measures of&#10;morphological complexity in addition to expert-produced typological features.&#10;We find that several morphological measures are significantly associated with&#10;higher surprisal when LSTM models are trained with BPE-segmented data. We also&#10;investigate linguistically-motivated subword segmentation strategies like&#10;Morfessor and Finite-State Transducers (FSTs) and find that these segmentation&#10;strategies yield better performance and reduce the impact of a language's&#10;morphology on language modeling.&#10;" />
          <attvalue for="2" value="&#10;&#10;With most research in Natural Language Processing (NLP) directed at a small subset of the world's languages, whether the techniques developed are truly language-agnostic is often not known. Because the vast majority of research focuses on English with Chinese a distant second \cite{Mie2016Language}, neither of which is morphologically rich, the impact of morphology on NLP tasks for various languages is not entirely understood.&#10;&#10;Several studies have investigated this issue in the context of language modeling by comparing a number of languages, but found conflicting results. \cite{gerz-etal-2018-relation} and \cite{cotterell-etal-2018-languages} find that morphological complexity is predictive of language modeling difficulty, while \cite{mielke-etal-2019-kind} conclude that simple statistics of a text like the number of types explain differences in modeling difficulty, rather than morphological measures.&#10;&#10;This paper revisits this issue by increasing the number of languages considered and augmenting the kind and number of morphological features used. &#10;We train language models for 92 languages from a corpus of Bibles fully aligned at the verse level and measure language modeling performance using surprisal (the negative log-likelihood) per verse (see \S\ref{sec:metrics}).&#10;We investigate how this measure is correlated with 12 linguist-generated morphological features and four corpus-based measures of morphological complexity. &#10;&#10;Additionally, we contend that the relation between segmentation method, morphology, and language modeling performance needs further investigation. &#10;Byte-Pair Encoding \cite[BPE;][]{shibata1999byte} is widely used in NLP tasks including machine translation \cite{sennrich-etal-2016-neural} as an unsupervised information-theoretic method for segmenting text data into subword units.&#10;Variants of BPE or closely related methods such as WordPiece \cite{kudo-2018-subword} are frequently employed by state-of-the-art pretrained language models \cite{roberta, radford2019language, bert, xlnet}. &#10;However, BPE and other segmentation methods may vary in how closely they capture morphological segments for a given language, which may affect language modeling performance. &#10;&#10;Therefore, this paper focuses on the following two research questions:&#10;\begin{enumerate}&#10;\item Does a language's morphology influence language modeling difficulty?&#10;\item If so, how do different segmentation methods interact with morphology?&#10;\end{enumerate}&#10;&#10;In order to answer the first question, we train models using data sets segmented by characters and BPE units.&#10;Our results show that BPE language modeling surprisal is significantly correlated with measures of morphological typology and complexity. This suggests that BPE segments are ineffective in mitigating the effect of morphology in language modeling. &#10;&#10;As for the second question, we consider more linguistically-motivated segmentation methods to compare with BPE: Morfessor \cite{morfessorcreutzUnsupervisedModelsMorpheme2007} and Finite-State Transducers (FSTs) (see \S\ref{sec:segmentation}).&#10;Our comparison of the models using the different segmentation methods shows that Morfessor reduces the impact of morphology for more languages than BPE. FST-based segmentation methods outperform the other segmentation methods when available. &#10;These results suggest that morphologically motivated segmentations improve cross-linguistic language modeling.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Morphological Complexity Analysis, Linguistics, Subword Segmentation Strategies, Artificial Intelligence, Mathematics, Multilingual Language Modeling" />
        </attvalues>
      </node>
      <node id="1806.03743" label="1806.03743">
        <attvalues>
          <attvalue for="0" value="Are All Languages Equally Hard to Language-Model?" />
          <attvalue for="1" value="  For general modeling methods applied to diverse languages, a natural question&#10;is: how well should we expect our models to work on languages with differing&#10;typological profiles? In this work, we develop an evaluation framework for fair&#10;cross-linguistic comparison of language models, using translated text so that&#10;all models are asked to predict approximately the same information. We then&#10;conduct a study on 21 languages, demonstrating that in some languages, the&#10;textual expression of the information is harder to predict with both $n$-gram&#10;and LSTM language models. We show complex inflectional morphology to be a cause&#10;of performance differences among languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.04726" label="1906.04726">
        <attvalues>
          <attvalue for="0" value="What Kind of Language Is Hard to Language-Model?" />
          <attvalue for="1" value="  How language-agnostic are current state-of-the-art NLP tools? Are there some&#10;types of language that are easier to model with current methods? In prior work&#10;(Cotterell et al., 2018) we attempted to address this question for language&#10;modeling, and observed that recurrent neural network language models do not&#10;perform equally well over all the high-resource European languages found in the&#10;Europarl corpus. We speculated that inflectional morphology may be the primary&#10;culprit for the discrepancy. In this paper, we extend these earlier experiments&#10;to cover 69 languages from 13 language families using a multilingual Bible&#10;corpus. Methodologically, we introduce a new paired-sample multiplicative&#10;mixed-effects model to obtain language difficulty coefficients from&#10;at-least-pairwise parallel corpora. In other words, the model is aware of&#10;inter-sentence variation and can handle missing data. Exploiting this model, we&#10;show that &quot;translationese&quot; is not any easier to model than natively written&#10;language in a fair comparison. Trying to answer the question of what features&#10;difficult languages have in common, we try and fail to reproduce our earlier&#10;(Cotterell et al., 2018) observation about morphological complexity and instead&#10;reveal far simpler statistics of the data that seem to drive complexity in a&#10;much larger sample.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2404.05825" label="2404.05825">
        <attvalues>
          <attvalue for="0" value="LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language&#10;  Models and Doc-Level Embedding" />
          <attvalue for="1" value="  Recently embedding-based retrieval or dense retrieval have shown state of the&#10;art results, compared with traditional sparse or bag-of-words based approaches.&#10;This paper introduces a model-agnostic doc-level embedding framework through&#10;large language model (LLM) augmentation. In addition, it also improves some&#10;important components in the retrieval model training process, such as negative&#10;sampling, loss function, etc. By implementing this LLM-augmented retrieval&#10;framework, we have been able to significantly improve the effectiveness of&#10;widely-used retriever models such as Bi-encoders (Contriever, DRAGON) and&#10;late-interaction models (ColBERTv2), thereby achieving state-of-the-art results&#10;on LoTTE datasets and BEIR datasets.&#10;" />
          <attvalue for="2" value="&#10;The Bi-encoder \cite{karpukhin2020dense} is a type of neural network architecture that is widely used in information retrieval. It consists of two encoders, typically in the form of transformer models \cite{vaswani2017attention}, which encode an vector representation for user queries and potential documents or passages respectively. These two encoders can be shared or using two separate models. The similarity between these two embedding vectors can then be computed, often using dot product or cosine similarity, to determine the relevance of the document or passage to the user's query.&#10;&#10;Cross-encoders \cite{nogueira2019passage}, unlike bi-encoders, amalgamate the inputs at an early stage, allowing for a more intricate interaction between user queries and documents. Here the user query and the document are concatenated, based on which a joint embedding vector is computed. The joint embedding vector is then used to make predictions, such as the relevance of a document to a query in an information retrieval task. Cross-encoders often outperform bi-encoders in tasks requiring a nuanced understanding of the interplay between inputs.&#10;&#10;Late-interaction models, such as ColBERT \cite{khattab2020colbert}, ColBERTv2 \cite{santhanam2021colbertv2} or SPALDE++ \cite{formal2022distillation}, are model architectures that hybrids cross-encoder models and bi-encoder models. Queries and documents are independently encoded into token-level vector representations. So in some sense, this is a bag of embedding vectors model. The interaction between these representations, which constitutes the “late interaction”, involves computing the cosine similarity or dot product scores over the token-level vector embedding. &#10;&#10;All the model architectures require informative embedding of user queries and target documents. While we cannot control the user queries during retrieval tasks, we hypothesize that enriching the embedding of documents can improve the quality and robustness of text retrieval.&#10;&#10;This work makes the following contributions:&#10;\begin{enumerate}&#10;\item We propose LLM-augmented retrieval, a model-agnostic framework that enriches the contextual information in the vector embedding of documents to improve the quality and robustness of existing retrievers.&#10;\item We propose doc-level embedding, which combines more contextual information in the context embedding.&#10;\item We evaluate this framework across different models and a wide range of datasets, establishing state-of-art quality beyond original models.&#10;\item We propose improvements to some key components in retrieval model training, such as negative sampling, loss function, etc. &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Computer Science, Linguistics, Language Models, Artificial Intelligence, Retrieval Models, Information Science" />
        </attvalues>
      </node>
      <node id="2112.01488" label="2112.01488">
        <attvalues>
          <attvalue for="0" value="ColBERTv2: Effective and Efficient Retrieval via Lightweight Late&#10;  Interaction" />
          <attvalue for="1" value="  Neural information retrieval (IR) has greatly advanced search and other&#10;knowledge-intensive language tasks. While many neural IR methods encode queries&#10;and documents into single-vector representations, late interaction models&#10;produce multi-vector representations at the granularity of each token and&#10;decompose relevance modeling into scalable token-level computations. This&#10;decomposition has been shown to make late interaction more effective, but it&#10;inflates the space footprint of these models by an order of magnitude. In this&#10;work, we introduce ColBERTv2, a retriever that couples an aggressive residual&#10;compression mechanism with a denoised supervision strategy to simultaneously&#10;improve the quality and space footprint of late interaction. We evaluate&#10;ColBERTv2 across a wide range of benchmarks, establishing state-of-the-art&#10;quality within and outside the training domain while reducing the space&#10;footprint of late interaction models by 6--10$\times$.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.04733" label="2205.04733">
        <attvalues>
          <attvalue for="0" value="From Distillation to Hard Negative Sampling: Making Sparse Neural IR&#10;  Models More Effective" />
          <attvalue for="1" value="  Neural retrievers based on dense representations combined with Approximate&#10;Nearest Neighbors search have recently received a lot of attention, owing their&#10;success to distillation and/or better sampling of examples for training --&#10;while still relying on the same backbone architecture. In the meantime, sparse&#10;representation learning fueled by traditional inverted indexing techniques has&#10;seen a growing interest, inheriting from desirable IR priors such as explicit&#10;lexical matching. While some architectural variants have been proposed, a&#10;lesser effort has been put in the training of such models. In this work, we&#10;build on SPLADE -- a sparse expansion-based retriever -- and show to which&#10;extent it is able to benefit from the same training improvements as dense&#10;models, by studying the effect of distillation, hard-negative mining as well as&#10;the Pre-trained Language Model initialization. We furthermore study the link&#10;between effectiveness and efficiency, on in-domain and zero-shot settings,&#10;leading to state-of-the-art results in both scenarios for sufficiently&#10;expressive models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.07094" label="2407.07094">
        <attvalues>
          <attvalue for="0" value="AnyTaskTune: Advanced Domain-Specific Solutions through Task-Fine-Tuning" />
          <attvalue for="1" value="  The pervasive deployment of Large Language Models-LLMs in various sectors&#10;often neglects the nuanced requirements of individuals and small organizations,&#10;who benefit more from models precisely tailored to their specific business&#10;contexts rather than those with broadly superior general capabilities. This&#10;work introduces \textbf{AnyTaskTune}, a novel fine-tuning methodology coined as&#10;\textbf{Task-Fine-Tune}, specifically developed to elevate model performance on&#10;a diverse array of domain-specific tasks. This method involves a meticulous&#10;process to identify and define targeted sub-tasks within a domain, followed by&#10;the creation of specialized enhancement datasets for fine-tuning, thereby&#10;optimizing task-specific model performance. We conducted comprehensive&#10;fine-tuning experiments not only in the legal domain for tasks such as keyword&#10;extraction and sentence prediction but across over twenty different sub-tasks&#10;derived from the domains of finance, healthcare, law, psychology, consumer&#10;services, and human resources. To substantiate our approach and facilitate&#10;community engagement, we will open-source these bilingual task datasets. Our&#10;findings demonstrate that models fine-tuned using the \textbf{Task-Fine-Tune}&#10;methodology not only achieve superior performance on these specific tasks but&#10;also significantly outperform models with higher general capabilities in their&#10;respective domains. Our work is publicly available at&#10;\url{https://github.com/PandaVT/DataTager}.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, the rapid advancement and deployment of Large Language Models (LLMs) have transformed various sectors by providing unprecedented natural language processing capabilities~\cite{zhao2023survey}. Traditionally, these models have been developed with a focus on enhancing their general abilities, aiming to create universally powerful tools that excel across a broad spectrum of tasks and domains. However, this generalized approach often fails to meet the specific requirements of individual users and small organizations, whose needs are intricately tied to their unique operational contexts. To address this issue, one very simple way is to fine-tune LLMs for different tasks.&#10;In the medical field, researchers have optimized LLMs using medical datasets to support functionalities such as medical document interpretation and diagnostic consultations. For example, ChatMed~\cite{zhu2023ChatMed}, DISC-MedLLM~\cite{bao2023discmedllmbridginggenerallarge}, HyKGE~\cite{jiang2024hykgehypothesisknowledgegraph}, IvyGPT~\cite{wang2023ivygpt}, and HuatuoGPT~\cite{zhang2023huatuogpt} are notable examples, demonstrating significant advancements within their specialized medical domains compared to generic LLMs. Similarly, in the legal field, researchers have introduced fine-tuned LLMs trained on legal corpora to support activities such as legal research, contract analysis, and legal document summarization. Models like ChatLaw~\cite{cui2024chatlawmultiagentcollaborativelegal}, LawGPT~\cite{zhou2024lawgpt}, and DISC-LawLLM~\cite{yue2023disc} exemplify this trend, demonstrating a deeper grasp of legal language and principles compared to their generic counterparts. Within finance, researchers are leveraging diverse financial datasets, including conversational data and market reports, to fine-tune LLMs for applications such as financial forecasting, risk assessment, and fraud detection. Examples include FinGPT~\cite{yang2023fingpt}, DISC-FinLLM~\cite{chen2023disc}, and PIXIU~\cite{xie2023pixiu}. Additionally, in other fields, models like FaiMA~\cite{yang2024faimafeatureawareincontextlearning}, designed for Multi-domain applications, and Kuaiji~\cite{luo2024kuaijichineseaccountinglarge}, tailored for accounting tasks, illustrate the versatility and potential of fine-tuned LLMs in specialized domains.&#10;&#10;To address this discrepancy, we introduce a novel fine-tuning paradigm specifically designed for Explicit Data Sets, which we term &quot;task fine tune.&quot; This method diverges from conventional training techniques by emphasizing precision and specificity over general performance. By focusing on Explicit Data Sets that contain clear, directive input-output pairs with specific instructions, this approach allows for precise tailoring of models to perform designated tasks effectively. Such fine-tuning not only enhances the model’s ability to execute particular functions but also significantly improves its applicability and efficiency in real-world scenarios.&#10;&#10;The cornerstone of this work, &quot;AnyTaskTune,&quot; is to validate and elaborate on the &quot;task fine tune&quot; method, utilizing Explicit Data Sets across various domains. This method involves identifying the spectrum of tasks required in different fields and scenarios and meticulously developing numerous Explicit Data Sets to conduct task-specific fine-tuning. The aim is to optimize the model's performance for individual and corporate use, ensuring that it meets the nuanced demands of users in fields such as finance, healthcare, law, psychology, consumer services, and human resources.&#10;&#10;To showcase the effectiveness and versatility of this approach, we conducted experiments using over twenty different Explicit Data Sets that we will open-source. These datasets are bilingual and designed to cater to a diverse global audience, providing a valuable resource for the community to engage in further research and enhance application-specific model improvements.&#10;&#10;By concentrating on specific, clearly defined tasks rather than general capabilities, &quot;AnyTaskTune&quot; significantly outperforms traditional models that prioritize breadth over depth. This strategic shift towards more specialized, task-oriented model training represents a critical evolution in how we develop and deploy LLMs, making them more relevant and valuable in industry-specific applications.&#10;&#10;The primary contributions of this paper can be summarized as follows:&#10;\begin{itemize}&#10; \item Introduction of the &quot;task fine tune&quot; paradigm, specifically designed for Explicit Data Sets, emphasizing precision and specificity over general performance.&#10; \item Development and utilization of over twenty bilingual Explicit Data Sets across various domains to validate the effectiveness of the &quot;task fine tune&quot; method.&#10; \item Evaluation of multiple models and datasets across various domains, demonstrating that task-specific fine-tuning yields better results.&#10; \item Open-sourcing of the developed Explicit Data Sets, providing a valuable resource for the community to engage in further research and enhance application-specific model improvements.&#10; \item Implementation of our data processing model, DataTager-LLM, as an online accessible website,.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Task-Specific Model Performance, Linguistics, Cognitive Science, Domain-Specific Fine-Tuning, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2312.15883" label="2312.15883">
        <attvalues>
          <attvalue for="0" value="HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and&#10;  Reliable Medical LLMs Responses" />
          <attvalue for="1" value="  In this paper, we investigate the retrieval-augmented generation (RAG) based&#10;on Knowledge Graphs (KGs) to improve the accuracy and reliability of Large&#10;Language Models (LLMs). Recent approaches suffer from insufficient and&#10;repetitive knowledge retrieval, tedious and time-consuming query parsing, and&#10;monotonous knowledge utilization. To this end, we develop a Hypothesis&#10;Knowledge Graph Enhanced (HyKGE) framework, which leverages LLMs' powerful&#10;reasoning capacity to compensate for the incompleteness of user queries,&#10;optimizes the interaction process with LLMs, and provides diverse retrieved&#10;knowledge. Specifically, HyKGE explores the zero-shot capability and the rich&#10;knowledge of LLMs with Hypothesis Outputs to extend feasible exploration&#10;directions in the KGs, as well as the carefully curated prompt to enhance the&#10;density and efficiency of LLMs' responses. Furthermore, we introduce the HO&#10;Fragment Granularity-aware Rerank Module to filter out noise while ensuring the&#10;balance between diversity and relevance in retrieved knowledge. Experiments on&#10;two Chinese medical multiple-choice question datasets and one Chinese&#10;open-domain medical Q&amp;A dataset with two LLM turbos demonstrate the superiority&#10;of HyKGE in terms of accuracy and explainability.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2307.10512" label="2307.10512">
        <attvalues>
          <attvalue for="0" value="IvyGPT: InteractiVe Chinese pathwaY language model in medical domain" />
          <attvalue for="1" value="  General large language models (LLMs) such as ChatGPT have shown remarkable&#10;success. However, such LLMs have not been widely adopted for medical purposes,&#10;due to poor accuracy and inability to provide medical advice. We propose&#10;IvyGPT, an LLM based on LLaMA that is trained and fine-tuned with high-quality&#10;medical question-answer (QA) instances and Reinforcement Learning from Human&#10;Feedback (RLHF). After supervised fine-tuning, IvyGPT has good multi-turn&#10;conversation capabilities, but it cannot perform like a doctor in other&#10;aspects, such as comprehensive diagnosis. Through RLHF, IvyGPT can output&#10;richer diagnosis and treatment answers that are closer to human. In the&#10;training, we used QLoRA to train 33 billion parameters on a small number of&#10;NVIDIA A100 (80GB) GPUs. Experimental results show that IvyGPT has outperformed&#10;other medical GPT models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.16092" label="2306.16092">
        <attvalues>
          <attvalue for="0" value="Chatlaw: A Multi-Agent Collaborative Legal Assistant with Knowledge&#10;  Graph Enhanced Mixture-of-Experts Large Language Model" />
          <attvalue for="1" value="  AI legal assistants based on Large Language Models (LLMs) can provide&#10;accessible legal consulting services, but the hallucination problem poses&#10;potential legal risks. This paper presents Chatlaw, an innovative legal&#10;assistant utilizing a Mixture-of-Experts (MoE) model and a multi-agent system&#10;to enhance the reliability and accuracy of AI-driven legal services. By&#10;integrating knowledge graphs with artificial screening, we construct a&#10;high-quality legal dataset to train the MoE model. This model utilizes&#10;different experts to address various legal issues, optimizing the accuracy of&#10;legal responses. Additionally, Standardized Operating Procedures (SOP), modeled&#10;after real law firm workflows, significantly reduce errors and hallucinations&#10;in legal services. Our MoE model outperforms GPT-4 in the Lawbench and Unified&#10;Qualification Exam for Legal Professionals by 7.73% in accuracy and 11 points,&#10;respectively, and also surpasses other models in multiple dimensions during&#10;real-case consultations, demonstrating our robust capability for legal&#10;consultation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.04614" label="2406.04614">
        <attvalues>
          <attvalue for="0" value="LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Model" />
          <attvalue for="1" value="  Large language models (LLMs), including both proprietary and open-source&#10;models, have showcased remarkable capabilities in addressing a wide range of&#10;downstream tasks. Nonetheless, when it comes to practical Chinese legal tasks,&#10;these models fail to meet the actual requirements. Proprietary models do not&#10;ensure data privacy for sensitive legal cases, while open-source models&#10;demonstrate unsatisfactory performance due to their lack of legal knowledge. To&#10;address this problem, we introduce LawGPT, the first open-source model&#10;specifically designed for Chinese legal applications. LawGPT comprises two key&#10;components: legal-oriented pre-training and legal supervised fine-tuning.&#10;Specifically, we employ large-scale Chinese legal documents for legal-oriented&#10;pre-training to incorporate legal domain knowledge. To further improve the&#10;model's performance on downstream legal tasks, we create a knowledge-driven&#10;instruction dataset for legal supervised fine-tuning. Our experimental results&#10;demonstrate that LawGPT outperforms the open-source LLaMA 7B model. Our code&#10;and resources are publicly available at https://github.com/pengxiao-song/LaWGPT&#10;and have received 5.7K stars on GitHub.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.11325" label="2309.11325">
        <attvalues>
          <attvalue for="0" value="DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal&#10;  Services" />
          <attvalue for="1" value="  We propose DISC-LawLLM, an intelligent legal system utilizing large language&#10;models (LLMs) to provide a wide range of legal services. We adopt legal&#10;syllogism prompting strategies to construct supervised fine-tuning datasets in&#10;the Chinese Judicial domain and fine-tune LLMs with legal reasoning capability.&#10;We augment LLMs with a retrieval module to enhance models' ability to access&#10;and utilize external legal knowledge. A comprehensive legal benchmark,&#10;DISC-Law-Eval, is presented to evaluate intelligent legal systems from both&#10;objective and subjective dimensions. Quantitative and qualitative results on&#10;DISC-Law-Eval demonstrate the effectiveness of our system in serving various&#10;users across diverse legal scenarios. The detailed resources are available at&#10;https://github.com/FudanDISC/DISC-LawLLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.15205" label="2310.15205">
        <attvalues>
          <attvalue for="0" value="DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple&#10;  Experts Fine-tuning" />
          <attvalue for="1" value="  We propose Multiple Experts Fine-tuning Framework to build a financial large&#10;language model (LLM), DISC-FinLLM. Our methodology improves general LLMs by&#10;endowing them with multi-turn question answering abilities, domain text&#10;processing capabilities, mathematical computation skills, and&#10;retrieval-enhanced generation capabilities. We build a financial&#10;instruction-tuning dataset named DISC-FIN-SFT, including instruction samples of&#10;four categories (consulting, NLP tasks, computing and retrieval-augmented&#10;generation). Evaluations conducted on multiple benchmarks demonstrate that our&#10;model performs better than baseline models in various financial scenarios.&#10;Further resources can be found at https://github.com/FudanDISC/DISC-FinLLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.05443" label="2306.05443">
        <attvalues>
          <attvalue for="0" value="PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark&#10;  for Finance" />
          <attvalue for="1" value="  Although large language models (LLMs) has shown great performance on natural&#10;language processing (NLP) in the financial domain, there are no publicly&#10;available financial tailtored LLMs, instruction tuning datasets, and evaluation&#10;benchmarks, which is critical for continually pushing forward the open-source&#10;development of financial artificial intelligence (AI). This paper introduces&#10;PIXIU, a comprehensive framework including the first financial LLM based on&#10;fine-tuning LLaMA with instruction data, the first instruction data with 136K&#10;data samples to support the fine-tuning, and an evaluation benchmark with 5&#10;tasks and 9 datasets. We first construct the large-scale multi-task instruction&#10;data considering a variety of financial tasks, financial document types, and&#10;financial data modalities. We then propose a financial LLM called FinMA by&#10;fine-tuning LLaMA with the constructed dataset to be able to follow&#10;instructions for various financial tasks. To support the evaluation of&#10;financial LLMs, we propose a standardized benchmark that covers a set of&#10;critical financial tasks, including five financial NLP tasks and one financial&#10;prediction task. With this benchmark, we conduct a detailed analysis of FinMA&#10;and several existing LLMs, uncovering their strengths and weaknesses in&#10;handling critical financial tasks. The model, datasets, benchmark, and&#10;experimental results are open-sourced to facilitate future research in&#10;financial AI.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.01063" label="2403.01063">
        <attvalues>
          <attvalue for="0" value="FaiMA: Feature-aware In-context Learning for Multi-domain Aspect-based&#10;  Sentiment Analysis" />
          <attvalue for="1" value="  Multi-domain aspect-based sentiment analysis (ABSA) seeks to capture&#10;fine-grained sentiment across diverse domains. While existing research narrowly&#10;focuses on single-domain applications constrained by methodological limitations&#10;and data scarcity, the reality is that sentiment naturally traverses multiple&#10;domains. Although large language models (LLMs) offer a promising solution for&#10;ABSA, it is difficult to integrate effectively with established techniques,&#10;including graph-based models and linguistics, because modifying their internal&#10;architecture is not easy. To alleviate this problem, we propose a novel&#10;framework, Feature-aware In-context Learning for Multi-domain ABSA (FaiMA). The&#10;core insight of FaiMA is to utilize in-context learning (ICL) as a&#10;feature-aware mechanism that facilitates adaptive learning in multi-domain ABSA&#10;tasks. Specifically, we employ a multi-head graph attention network as a text&#10;encoder optimized by heuristic rules for linguistic, domain, and sentiment&#10;features. Through contrastive learning, we optimize sentence representations by&#10;focusing on these diverse features. Additionally, we construct an efficient&#10;indexing mechanism, allowing FaiMA to stably retrieve highly relevant examples&#10;across multiple dimensions for any given input. To evaluate the efficacy of&#10;FaiMA, we build the first multi-domain ABSA benchmark dataset. Extensive&#10;experimental results demonstrate that FaiMA achieves significant performance&#10;improvements in multiple domains compared to baselines, increasing F1 by 2.07%&#10;on average. Source code and data sets are anonymously available at&#10;https://github.com/SupritYoung/FaiMA.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.13866" label="2402.13866">
        <attvalues>
          <attvalue for="0" value="Kuaiji: the First Chinese Accounting Large Language Model" />
          <attvalue for="1" value="  Large Language Models (LLMs) like ChatGPT and GPT-4 have demonstrated&#10;impressive proficiency in comprehending and generating natural language.&#10;However, they encounter difficulties when tasked with adapting to specialized&#10;domains such as accounting. To address this challenge, we introduce Kuaiji, a&#10;tailored Accounting Large Language Model. Kuaiji is meticulously fine-tuned&#10;using the Baichuan framework, which encompasses continuous pre-training and&#10;supervised fine-tuning processes. Supported by CAtAcctQA, a dataset containing&#10;large genuine accountant-client dialogues, Kuaiji exhibits exceptional accuracy&#10;and response speed. Our contributions encompass the creation of the first&#10;Chinese accounting dataset, the establishment of Kuaiji as a leading&#10;open-source Chinese accounting LLM, and the validation of its efficacy through&#10;real-world accounting scenarios.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.01201" label="2409.01201">
        <attvalues>
          <attvalue for="0" value="EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio&#10;  Captioning Performance" />
          <attvalue for="1" value="  In this work, we aim to analyze and optimize the EnCLAP framework, a&#10;state-of-the-art model in automated audio captioning. We investigate the impact&#10;of modifying the acoustic encoder components, explore pretraining with&#10;different dataset scales, and study the effectiveness of a reranking scheme.&#10;Through extensive experimentation and quantitative analysis of generated&#10;captions, we develop EnCLAP++, an enhanced version that significantly surpasses&#10;the original.&#10;" />
          <attvalue for="2" value="&#10; Automated audio captioning (AAC), a cross-modal translation involving transcribing audio signals into concise and meaningful natural language descriptions \cite{aac}, remains a particularly challenging task with a substantial performance gap between human and machine. One significant contributor to the performance gap can be attributed to the intrinsic complexity of the task, as distinguishing between various sound events, especially between similar and ambiguous ones, requires extensive real-world knowledge. Furthermore, the scarcity of high-quality data, with the most widely used datasets, AudioCaps \cite{audiocaps} and Clotho \cite{clotho} containing only 50K and 20K captions, respectively, poses an additional challenge. To address these challenges, prior studies have employed pretrained audio encoders trained on audio classification tasks \cite{mei, conette, beats-conformer}, leveraged the text generation capabilities of pretrained language models like GPT-2 \cite{gpt2, prefix_tuning, pengi} and BART \cite{bart, gontier}, and incorporated auxiliary loss terms, including keyword prediction loss \cite{koizumi_keyword} or sentence embedding loss \cite{sentence_embedding}, to improve the semantic quality of captions and provide additional training signal.&#10;&#10;Building on the previous line of research, Kim et~al. \cite{enclap} proposed the EnCLAP framework which integrates a set of pretrained models with an auxiliary training task. Specifically, EnCLAP utilizes two acoustic feature encoders, EnCodec \cite{encodec} and CLAP \cite{clap_laion}, to generate timestep-level and sequence-level representation of the input audio sequence, respectively. EnCLAP utilizes pretrained BART as the caption decoder to leverage these features and generate captions. Furthermore, Kim et~al. also introduced masked codec modeling (MCM), an auxiliary task which involves masking a part of the input codec sequence and predicting it, to enhance the acoustic awareness of the caption decoder. The caption decoder was trained jointly using cross-entropy loss for caption generation and MCM loss. The combination of these approaches allowed EnCLAP to achieve state-of-the-art performance on the AudioCaps dataset.&#10;&#10;Although EnCLAP exhibits impressive performance, the study by Kim et~al. lacks sufficient experimental evaluation for determining the optimal models for the model components. Notably, the authors do not investigate alternative sequence-level acoustic features beyond CLAP. Furthermore, for timestep-level acoustic features, while they demonstrate that discrete codec input outperforms continuous input, their analysis is restricted to a single setup using EnCodec, without exploring other options or configurations. Additionally, Kim et~al. acknowledge the issue of overfitting in larger model variants but do not investigate the use of large-scale weakly-labeled datasets \cite{wavcaps, beats-conformer}, which contain noisy and model-generated captions. Therefore, the EnCLAP framework has potential for further optimization. &#10;&#10;In this work, we extend and optimize the EnCLAP framework through a comprehensive examination of its components. We explore alternative acoustic feature encoder components and assess their efficacy. We also investigate the impact of large-scale training incorporating weakly-labeled datasets on the framework's performance. Furthermore, we adopt a sampling-and-reranking approach \cite{beats-conformer} as an alternative to beam search decoding and evaluate its effectiveness. Finally, we conduct a qualitative analysis of the generated captions to examine the effects of each component on the outputs. Based on our findings, we present EnCLAP++, an improved version of the EnCLAP model that achieved second place in the DCASE2024 Challenge Task6. Figure 1 provides an overview of EnCLAP++.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Audio Captioning Models, Signal Processing, Natural Language Processing, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="1706.10006" label="1706.10006">
        <attvalues>
          <attvalue for="0" value="Automated Audio Captioning with Recurrent Neural Networks" />
          <attvalue for="1" value="  We present the first approach to automated audio captioning. We employ an&#10;encoder-decoder scheme with an alignment model in between. The input to the&#10;encoder is a sequence of log mel-band energies calculated from an audio file,&#10;while the output is a sequence of words, i.e. a caption. The encoder is a&#10;multi-layered, bi-directional gated recurrent unit (GRU) and the decoder a&#10;multi-layered GRU with a classification layer connected to the last GRU of the&#10;decoder. The classification layer and the alignment model are fully connected&#10;layers with shared weights between timesteps. The proposed method is evaluated&#10;using data drawn from a commercial sound effects library, ProSound Effects. The&#10;resulting captions were rated through metrics utilized in machine translation&#10;and image captioning fields. Results from metrics show that the proposed method&#10;can predict words appearing in the original caption, but not always correctly&#10;ordered.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.09817" label="2107.09817">
        <attvalues>
          <attvalue for="0" value="Audio Captioning Transformer" />
          <attvalue for="1" value="  Audio captioning aims to automatically generate a natural language&#10;description of an audio clip. Most captioning models follow an encoder-decoder&#10;architecture, where the decoder predicts words based on the audio features&#10;extracted by the encoder. Convolutional neural networks (CNNs) and recurrent&#10;neural networks (RNNs) are often used as the audio encoder. However, CNNs can&#10;be limited in modelling temporal relationships among the time frames in an&#10;audio signal, while RNNs can be limited in modelling the long-range&#10;dependencies among the time frames. In this paper, we propose an Audio&#10;Captioning Transformer (ACT), which is a full Transformer network based on an&#10;encoder-decoder architecture and is totally convolution-free. The proposed&#10;method has a better ability to model the global information within an audio&#10;signal as well as capture temporal relationships between audio events. We&#10;evaluate our model on AudioCaps, which is the largest audio captioning dataset&#10;publicly available. Our model shows competitive performance compared to other&#10;state-of-the-art approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.01482" label="2305.01482">
        <attvalues>
          <attvalue for="0" value="Multitask learning in Audio Captioning: a sentence embedding regression&#10;  loss acts as a regularizer" />
          <attvalue for="1" value="  In this work, we propose to study the performance of a model trained with a&#10;sentence embedding regression loss component for the Automated Audio Captioning&#10;task. This task aims to build systems that can describe audio content with a&#10;single sentence written in natural language. Most systems are trained with the&#10;standard Cross-Entropy loss, which does not take into account the semantic&#10;closeness of the sentence. We found that adding a sentence embedding loss term&#10;reduces overfitting, but also increased SPIDEr from 0.397 to 0.418 in our first&#10;setting on the AudioCaps corpus. When we increased the weight decay value, we&#10;found our model to be much closer to the current state-of-the-art methods, with&#10;a SPIDEr score up to 0.444 compared to a 0.475 score. Moreover, this model uses&#10;eight times less trainable parameters. In this training setting, the sentence&#10;embedding loss has no more impact on the model performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.17690" label="2401.17690">
        <attvalues>
          <attvalue for="0" value="EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for&#10;  Automated Audio Captioning" />
          <attvalue for="1" value="  We propose EnCLAP, a novel framework for automated audio captioning. EnCLAP&#10;employs two acoustic representation models, EnCodec and CLAP, along with a&#10;pretrained language model, BART. We also introduce a new training objective&#10;called masked codec modeling that improves acoustic awareness of the pretrained&#10;language model. Experimental results on AudioCaps and Clotho demonstrate that&#10;our model surpasses the performance of baseline models. Source code will be&#10;available at https://github.com/jaeyeonkim99/EnCLAP . An online demo is&#10;available at https://huggingface.co/spaces/enclap-team/enclap .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.17352" label="2309.17352">
        <attvalues>
          <attvalue for="0" value="Improving Audio Captioning Models with Fine-grained Audio Features, Text&#10;  Embedding Supervision, and LLM Mix-up Augmentation" />
          <attvalue for="1" value="  Automated audio captioning (AAC) aims to generate informative descriptions&#10;for various sounds from nature and/or human activities. In recent years, AAC&#10;has quickly attracted research interest, with state-of-the-art systems now&#10;relying on a sequence-to-sequence (seq2seq) backbone powered by strong models&#10;such as Transformers. Following the macro-trend of applied machine learning&#10;research, in this work, we strive to improve the performance of seq2seq AAC&#10;models by extensively leveraging pretrained models and large language models&#10;(LLMs). Specifically, we utilize BEATs to extract fine-grained audio features.&#10;Then, we employ Instructor LLM to fetch text embeddings of captions, and infuse&#10;their language-modality knowledge into BEATs audio features via an auxiliary&#10;InfoNCE loss function. Moreover, we propose a novel data augmentation method&#10;that uses ChatGPT to produce caption mix-ups (i.e., grammatical and compact&#10;combinations of two captions) which, together with the corresponding audio&#10;mixtures, increase not only the amount but also the complexity and diversity of&#10;training data. During inference, we propose to employ nucleus sampling and a&#10;hybrid reranking algorithm, which has not been explored in AAC research.&#10;Combining our efforts, our model achieves a new state-of-the-art 32.6 SPIDEr-FL&#10;score on the Clotho evaluation split, and wins the 2023 DCASE AAC challenge.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.03316" label="2209.03316">
        <attvalues>
          <attvalue for="0" value="On the Complementarity between Pre-Training and Random-Initialization&#10;  for Resource-Rich Machine Translation" />
          <attvalue for="1" value="  Pre-Training (PT) of text representations has been successfully applied to&#10;low-resource Neural Machine Translation (NMT). However, it usually fails to&#10;achieve notable gains (sometimes, even worse) on resource-rich NMT on par with&#10;its Random-Initialization (RI) counterpart. We take the first step to&#10;investigate the complementarity between PT and RI in resource-rich scenarios&#10;via two probing analyses, and find that: 1) PT improves NOT the accuracy, but&#10;the generalization by achieving flatter loss landscapes than that of RI; 2) PT&#10;improves NOT the confidence of lexical choice, but the negative diversity by&#10;assigning smoother lexical probability distributions than that of RI. Based on&#10;these insights, we propose to combine their complementarities with a model&#10;fusion algorithm that utilizes optimal transport to align neurons between PT&#10;and RI. Experiments on two resource-rich translation benchmarks, WMT'17&#10;English-Chinese (20M) and WMT'19 English-German (36M), show that PT and RI&#10;could be nicely complementary to each other, achieving substantial improvements&#10;considering both translation accuracy, generalization, and negative diversity.&#10;Probing tools and code are released at: https://github.com/zanchangtong/PTvsRI.&#10;" />
          <attvalue for="2" value="&#10;Pre-training (\cite[PT;][]{Devlin:2018uk,liu2019roberta} has achieved tremendous success in natural language processing fields. Inspired by BERT~\cite{Devlin:2018uk}, recent works~\cite{song2019mass,bart2020,Liu:2020mbart} attempt to leverage sequence-to-sequence PT for neural machine translation (\cite[NMT;][]{bahdanau2014neural,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}) by leveraging a large amount of unlabeled (i.e. monolingual) sentences. &#10;&#10;While recent studies have empirically shown their benefit for the low-resource translation task where the labeled (i.e. parallel) sentences are limited~\cite{bart2020,song2019mass,Liu:2020mbart}, we are generally confronted with resource-rich scenarios, e.g. millions of parallel sentence pairs, in WMT evaluations~\cite{akhbardeh-EtAl:2021:WMT} and industries.&#10;For these resource-rich tasks, PT becomes less effective (sometimes, even worse) than their Random-Initialization (RI) counterparts, for example, as \cite{Zhu2020Incorporating,Liu:2020mbart} reported, the PT underperforms RI if improperly utilized or significant amount of bi-text data is given. &#10;However, there is limited understanding of: &#10;\tcgray{{1}} Why does PT fail compared to RI in terms of translation accuracy in high-resource settings?&#10;\tcgray{{2}} What is the difference between the optimized PT and RI models?&#10;\tcgray{{3}} How can we harmonize PT with RI? Can we just leverage their advantages?&#10;&#10;To this end, we introduce two probing analyses (i.e. loss landscape and lexical probability distribution) for the fully-optimized PT and RI models to investigate \tcgray{{1}} and \tcgray{{2}}, respectively. &#10;We find that: &#10;\tcblack{{1}} Supervised training with significant amount of parallel data, i.e. RI, is enough to optimize the model towards a better optimum point compared with PT, thus leading to better in-domain (same domain between test and training set) performance;&#10;\tcblack{{2}} PT mainly contributes to the generalization ability because of its flatter loss landscape and smoother lexical probability distribution, thus resulting in better out-domain performance and diversified generation.&#10;Motivated by this finding, we propose a simple combination method to reach \tcgray{{3}} by aligning the neurons and weights of PT and RI and then fusing them into a single model based on optimal transport~\cite{monge1781memoire,singh2020model}.&#10;Experiments conducted on the WMT17 English-Chinese and WMT19 English-German benchmarks show that PT can nicely complement RI, leading to better translation accuracy, generalization, and negative diversity. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Model Fusion Techniques, Artificial Intelligence, Mathematics, Pre-Training Methods, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="1910.05653" label="1910.05653">
        <attvalues>
          <attvalue for="0" value="Model Fusion via Optimal Transport" />
          <attvalue for="1" value="  Combining different models is a widely used paradigm in machine learning&#10;applications. While the most common approach is to form an ensemble of models&#10;and average their individual predictions, this approach is often rendered&#10;infeasible by given resource constraints in terms of memory and computation,&#10;which grow linearly with the number of models. We present a layer-wise model&#10;fusion algorithm for neural networks that utilizes optimal transport to (soft-)&#10;align neurons across the models before averaging their associated parameters.&#10;  We show that this can successfully yield &quot;one-shot&quot; knowledge transfer (i.e,&#10;without requiring any retraining) between neural networks trained on&#10;heterogeneous non-i.i.d. data. In both i.i.d. and non-i.i.d. settings , we&#10;illustrate that our approach significantly outperforms vanilla averaging, as&#10;well as how it can serve as an efficient replacement for the ensemble with&#10;moderate fine-tuning, for standard convolutional networks (like VGG11),&#10;residual networks (like ResNet18), and multi-layer perceptrons on CIFAR10,&#10;CIFAR100, and MNIST. Finally, our approach also provides a principled way to&#10;combine the parameters of neural networks with different widths, and we explore&#10;its application for model compression. The code is available at the following&#10;link, https://github.com/sidak/otfusion.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.12913" label="2402.12913">
        <attvalues>
          <attvalue for="0" value="OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination&#10;  Detection with Weakly Supervised Data" />
          <attvalue for="1" value="  This paper mainly describes a unified system for hallucination detection of&#10;LLMs, which wins the second prize in the model-agnostic track of the&#10;SemEval-2024 Task 6, and also achieves considerable results in the model-aware&#10;track. This task aims to detect hallucination with LLMs for three different&#10;text-generation tasks without labeled training data. We utilize prompt&#10;engineering and few-shot learning to verify the performance of different LLMs&#10;on the validation data. Then we select the LLMs with better performance to&#10;generate high-quality weakly supervised training data, which not only satisfies&#10;the consistency of different LLMs, but also satisfies the consistency of the&#10;optimal LLM with different sampling parameters. Furthermore, we finetune&#10;different LLMs by using the constructed training data, and finding that a&#10;relatively small LLM can achieve a competitive level of performance in&#10;hallucination detection, when compared to the large LLMs and the prompt-based&#10;approaches using GPT-4.&#10;" />
          <attvalue for="2" value="&#10;&#10;The emergence of Large Language Models (LLMs)\cite{zhao2023survey} has sparked a significant transformation in the field of Natural Language Processing (NLP), ushering in a new era of unparalleled advancements in text generation and comprehension. This revolutionary technology has elevated the capabilities of AI systems, enabling them to perform complex reasoning and problem-solving tasks with remarkable proficiency\cite{zhao2023survey}. At the heart of this transformation lies the LLMs' ability to compress vast amounts of knowledge into neural networks, effectively turning them into versatile agents capable of engaging in natural language conversations with humans\cite{hadi2023survey}. This has broadened the scope of AI applications beyond traditional domains such as chatbots and virtual assistants, into areas previously thought to be the exclusive domain of humans, particularly those involving creativity and expertise. LLMs are not only limited to language-related tasks but can also function as generalist agents, collaborating with external systems, tools, and models to achieve a wide range of objectives set by humans\cite{triguero2024general}.&#10;&#10;However, recent advancements in research have uncovered a concerning weakness: their proneness to hallucinate content across a range of applications\cite{ ji2023survey}. Hallucination is defined as the generation of information that either conflicts with established sources or cannot be substantiated by available knowledge. The occurrence of hallucination in LLMs poses a significant threat to their practical deployment. While prior works have delved into the roots of hallucination within specific, smaller-scale language models and tasks, there is still a notable gap in understanding the exact nature and prevalence of content that LLMs are likely to hallucinate\cite{cui2024risk, chang2023survey}.&#10;&#10;To address this challenge, we implements a unified system for hallucination detection of LLMs, when there is no labeled training data. This system comprises five parts: Base Model Selection, Prompt Engineering, Weakly-supervised Data Generation, SFT and Ensemble Learning. We first verify the performance of different base LLMs on this task. Then we select the best LLMs and prompt is optimized to improve the performance. And weakly-supervised dataset is generated by using the selected LLMs. For further improvement, SFT is done based on the constructed dataset and ensemble learning is adopted.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Large Language Models, Cognitive Science, Hallucination Detection, Artificial Intelligence, Weak Supervision, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2307.14283" label="2307.14283">
        <attvalues>
          <attvalue for="0" value="General Purpose Artificial Intelligence Systems (GPAIS): Properties,&#10;  Definition, Taxonomy, Societal Implications and Responsible Governance" />
          <attvalue for="1" value="  Most applications of Artificial Intelligence (AI) are designed for a confined&#10;and specific task. However, there are many scenarios that call for a more&#10;general AI, capable of solving a wide array of tasks without being specifically&#10;designed for them. The term General-Purpose Artificial Intelligence Systems&#10;(GPAIS) has been defined to refer to these AI systems. To date, the possibility&#10;of an Artificial General Intelligence, powerful enough to perform any&#10;intellectual task as if it were human, or even improve it, has remained an&#10;aspiration, fiction, and considered a risk for our society. Whilst we might&#10;still be far from achieving that, GPAIS is a reality and sitting at the&#10;forefront of AI research. This work discusses existing definitions for GPAIS&#10;and proposes a new definition that allows for a gradual differentiation among&#10;types of GPAIS according to their properties and limitations. We distinguish&#10;between closed-world and open-world GPAIS, characterising their degree of&#10;autonomy and ability based on several factors such as adaptation to new tasks,&#10;competence in domains not intentionally trained for, ability to learn from few&#10;data, or proactive acknowledgment of their own limitations. We propose a&#10;taxonomy of approaches to realise GPAIS, describing research trends such as the&#10;use of AI techniques to improve another AI (AI-powered AI) or (single)&#10;foundation models. As a prime example, we delve into GenAI, aligning them with&#10;the concepts presented in the taxonomy. We explore multi-modality, which&#10;involves fusing various types of data sources to expand the capabilities of&#10;GPAIS. Through the proposed definition and taxonomy, our aim is to facilitate&#10;research collaboration across different areas that are tackling general purpose&#10;tasks, as they share many common aspects. Finally, we discuss the state of&#10;GPAIS, prospects, societal implications, and the need for regulation and&#10;governance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.05778" label="2401.05778">
        <attvalues>
          <attvalue for="0" value="Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language&#10;  Model Systems" />
          <attvalue for="1" value="  Large language models (LLMs) have strong capabilities in solving diverse&#10;natural language processing tasks. However, the safety and security issues of&#10;LLM systems have become the major obstacle to their widespread application.&#10;Many studies have extensively investigated risks in LLM systems and developed&#10;the corresponding mitigation strategies. Leading-edge enterprises such as&#10;OpenAI, Google, Meta, and Anthropic have also made lots of efforts on&#10;responsible LLMs. Therefore, there is a growing need to organize the existing&#10;studies and establish comprehensive taxonomies for the community. In this&#10;paper, we delve into four essential modules of an LLM system, including an&#10;input module for receiving prompts, a language model trained on extensive&#10;corpora, a toolchain module for development and deployment, and an output&#10;module for exporting LLM-generated content. Based on this, we propose a&#10;comprehensive taxonomy, which systematically analyzes potential risks&#10;associated with each module of an LLM system and discusses the corresponding&#10;mitigation strategies. Furthermore, we review prevalent benchmarks, aiming to&#10;facilitate the risk assessment of LLM systems. We hope that this paper can help&#10;LLM participants embrace a systematic perspective to build their responsible&#10;LLM systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.03775" label="2105.03775">
        <attvalues>
          <attvalue for="0" value="NLP-IIS@UT at SemEval-2021 Task 4: Machine Reading Comprehension using&#10;  the Long Document Transformer" />
          <attvalue for="1" value="  This paper presents a technical report of our submission to the 4th task of&#10;SemEval-2021, titled: Reading Comprehension of Abstract Meaning. In this task,&#10;we want to predict the correct answer based on a question given a context.&#10;Usually, contexts are very lengthy and require a large receptive field from the&#10;model. Thus, common contextualized language models like BERT miss fine&#10;representation and performance due to the limited capacity of the input tokens.&#10;To tackle this problem, we used the Longformer model to better process the&#10;sequences. Furthermore, we utilized the method proposed in the Longformer&#10;benchmark on Wikihop dataset which improved the accuracy on our task data from&#10;23.01% and 22.95% achieved by the baselines for subtask 1 and 2, respectively,&#10;to 70.30% and 64.38%.&#10;" />
          <attvalue for="2" value="&#10;&#10;Reading comprehension is the ability to understand a passage either by human or machine. One of the great benchmarks to evaluate this ability is to try to answer specific questions related to the passage \cite{DBLP:conf/emnlp/RajpurkarZLL16}. Generally, this problem can contain single or multiple documents as context (containing relevant information needed to understand and answer the question), a question (a sentence with at least one asking parameter), and an answer (which is the parameter value of the question).&#10;&#10;In the Task of Reading Comprehension of Abstract Meaning (ReCAM), we have one passage as a context, one question and five candidate answers \cite{zheng-2021-semeval-task4}. The goal is to identify the correct answer based on the context and the given question. You can see a sample of the data in Table \ref{table_data}. For each instance of the data, there is a passage, a question with a missing word that should be filled based on the passage, and five candidate answers to the question.&#10;&#10;The task divides into two subtasks: imperceptibility and non-specificity\cite{zheng-2021-semeval-task4}.&#10;\begin{itemize}&#10; \item imperceptibility: this level of abstract words refers to ideas and concepts that are distant from immediate perception; such as culture, economics, and politics.&#10; \item non-specificity: In contrast to concrete words, this subtask includes more abstract words which focus on a different type of definition; for example, a concrete word like `cow` could be interpreted as an `animal` which is considered as a more abstract word \cite{changizi2008economically}.&#10;\end{itemize}&#10;&#10;The main challenges of this task are the abstract meaning concept representation as well as the machine reading comprehension. This is the main reason we have utilized contextualized language representation models to tackle abstract meaning representation problems.&#10;&#10;In this paper, we use an end-to-end deep contextualized architecture to model this task. This model is also capable of considering more than one passage as the context, and more than five candidate answers. Since we use the long document transformer model (Longformer \cite{beltagy2020longformer}), no limitation is considered in context passage length. We have evaluated this model both on subtask-1 and subtask-2 which resulted in 70\% and 64\% accuracy, respectively. Therefore, we have about 40\% improvement compared to the baseline, which is a Gated Attention (GA) model \cite{zheng-2021-semeval-task4}.&#10;&#10;The rest of the paper is as follows: Section 2 describes the related works and the background. Section 3 includes the description of the proposed method. Section 4 contains the evaluation metrics used as well as a brief discussion, which is then followed by a conclusion and future works in section5.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Language Models, Task Performance Optimization, Artificial Intelligence, Reading Comprehension" />
        </attvalues>
      </node>
      <node id="2105.14879" label="2105.14879">
        <attvalues>
          <attvalue for="0" value="SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning" />
          <attvalue for="1" value="  This paper introduces the SemEval-2021 shared task 4: Reading Comprehension&#10;of Abstract Meaning (ReCAM). This shared task is designed to help evaluate the&#10;ability of machines in representing and understanding abstract concepts. Given&#10;a passage and the corresponding question, a participating system is expected to&#10;choose the correct answer from five candidates of abstract concepts in a&#10;cloze-style machine reading comprehension setup. Based on two typical&#10;definitions of abstractness, i.e., the imperceptibility and nonspecificity, our&#10;task provides three subtasks to evaluate the participating models.&#10;Specifically, Subtask 1 aims to evaluate how well a system can model concepts&#10;that cannot be directly perceived in the physical world. Subtask 2 focuses on&#10;models' ability in comprehending nonspecific concepts located high in a&#10;hypernym hierarchy given the context of a passage. Subtask 3 aims to provide&#10;some insights into models' generalizability over the two types of abstractness.&#10;During the SemEval-2021 official evaluation period, we received 23 submissions&#10;to Subtask 1 and 28 to Subtask 2. The participating teams additionally made 29&#10;submissions to Subtask 3. The leaderboard and competition website can be found&#10;at https://competitions.codalab.org/competitions/26153. The data and baseline&#10;code are available at&#10;https://github.com/boyuanzheng010/SemEval2021-Reading-Comprehension-of-Abstract-Meaning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.00712" label="1911.00712">
        <attvalues>
          <attvalue for="0" value="How to Pre-Train Your Model? Comparison of Different Pre-Training Models&#10;  for Biomedical Question Answering" />
          <attvalue for="1" value="  Using deep learning models on small scale datasets would result in&#10;overfitting. To overcome this problem, the process of pre-training a model and&#10;fine-tuning it to the small scale dataset has been used extensively in domains&#10;such as image processing. Similarly for question answering, pre-training and&#10;fine-tuning can be done in several ways. Commonly reading comprehension models&#10;are used for pre-training, but we show that other types of pre-training can&#10;work better. We compare two pre-training models based on reading comprehension&#10;and open domain question answering models and determine the performance when&#10;fine-tuned and tested over BIOASQ question answering dataset. We find open&#10;domain question answering model to be a better fit for this task rather than&#10;reading comprehension model.&#10;" />
          <attvalue for="2" value="&#10;&#10;Deep learning models have been widely used in several NLP tasks since the emergence of large scale labelled datasets. In Question Answering (QA) specifically on open domain, several neural network models have been introduced, such as Convolutional Neural Networks(CNN), Recurrent Neural Networks(RNN) using GRUs or LSTMs and attention mechanisms, Self-attention networks (Transformers), and Pretrained language models like ELMO, BERT which can be fine-tuned to Question Answering task.&#10;Several kinds of Question Answering (QA) related tasks are widely studied such as Answer Sentence Selection, Reading Comprehension and Open QA. &#10;&#10;Reading Comprehension (RC) is a QA task where a question and a relevant paragraph are given and the goal is to extract the answer string present in the paragraph. The main assumption of this task is that the answer is present in the paragraph, like in SQUAD v1.0 \cite{rajpurkar2016squad}. Variants of this task include unanswerable questions such as in \cite{rajpurkar2018know}. Answers are usually short phrases or entities. There is a leaderboard on SQUAD dataset which showcases lot of models built for this task \cite{seo2016bidirectional,chen2017reading,bert}.&#10;&#10;Open QA is a QA task where a question is given and the goal is to retrieve an answer. An answer has to be retrieved from a set of documents or passages of textual sources as Wikipedia articles or news. Answers are also usually short phrases or entities. &#10;In NN approaches for Open QA, generally answers are extracted using a reading comprehension model on the subset of the retrieved documents or passages considered as relevant \cite{DBLP:journals/corr/DhingraMC17,joshi2017triviaqa}.&#10;&#10;One of the main differences between Reading Comprehension (RC) and Open QA tasks is that the answer must be present in the paragraphs (or documents) for Reading Comprehension, but for Open QA this condition might not hold true because the retrieved documents considered to be relevant to the question might not contain the answer. Another characteristic is that in the Open QA task, several paragraphs or documents contain the answer.&#10;&#10;The BIOASQ Phase B task provides dataset for biomedical question answering which is a small scale labelled dataset for factoid questions (779 question in BIOASQ 7). Each question is associated with multiple relevant paragraphs, some irrelevant ones, and one or several answers. &#10;The work of \cite{wiese-etal-2017-neural} transforms the BIOASQ Phase B dataset into the format of a Reading Comprehension task where each question has an answer text along with the offset in a paragraph which contains the answer. If a paragraph does not contain an answer, it is discarded. This modification of the BIOASQ dataset enables to use a RC model off-the-shelf. &#10;&#10;By using such a model on BIOASQ dataset which is a small scale labelled dataset, it will not result in similar performance as on the large scale open domain datasets due to overfitting. One way of overcoming this problem as reported by \cite{cicling,wiese-etal-2017-neural} is by pre-training a deep learning model on a large scale dataset and fine-tuning the same model to the target small scale dataset. The intuition is that the model learns better representations when learnt on a large scale dataset than having a randomly initialized model trained only on the small scale dataset. &#10;&#10;However the BIOASQ task resembles more towards an Open QA task than a RC task because of the existence of paragraphs without answers even though they are considered relevant. Thus we propose a new way to tackle the BIOASQ task by using an Open QA model that takes into account this particularity.&#10;&#10;We present a comparison of using different pre-training models (reading comprehension and open QA models) for BIOASQ question answering task and also report the performance of a single model without pre-training and without fine-tuning to show the importance of this process. &#10;&#10;We report the performance of our model on different datasets and show that in some cases it outperforms the state-of-the-art systems of BIOASQ \cite{wiese-etal-2017-neural,kamath-etal-2018-adaption,DBLP:journals/corr/abs-1901-08746} in average. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Artificial Intelligence, Question Answering Systems, Information Science, Pre-training Techniques" />
        </attvalues>
      </node>
      <node id="1707.03904" label="1707.03904">
        <attvalues>
          <attvalue for="0" value="Quasar: Datasets for Question Answering by Search and Reading" />
          <attvalue for="1" value="  We present two new large-scale datasets aimed at evaluating systems designed&#10;to comprehend a natural language query and extract its answer from a large&#10;corpus of text. The Quasar-S dataset consists of 37000 cloze-style&#10;(fill-in-the-gap) queries constructed from definitions of software entity tags&#10;on the popular website Stack Overflow. The posts and comments on the website&#10;serve as the background corpus for answering the cloze questions. The Quasar-T&#10;dataset consists of 43000 open-domain trivia questions and their answers&#10;obtained from various internet sources. ClueWeb09 serves as the background&#10;corpus for extracting these answers. We pose these datasets as a challenge for&#10;two related subtasks of factoid Question Answering: (1) searching for relevant&#10;pieces of text that include the correct answer to a query, and (2) reading the&#10;retrieved text to answer the query. We also describe a retrieval system for&#10;extracting relevant sentences and documents from the corpus given a query, and&#10;include these in the release for researchers wishing to only focus on (2). We&#10;evaluate several baselines on both datasets, ranging from simple heuristics to&#10;powerful neural models, and show that these lag behind human performance by&#10;16.4% and 32.1% for Quasar-S and -T respectively. The datasets are available at&#10;https://github.com/bdhingra/quasar .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.03610" label="1706.03610">
        <attvalues>
          <attvalue for="0" value="Neural Domain Adaptation for Biomedical Question Answering" />
          <attvalue for="1" value="  Factoid question answering (QA) has recently benefited from the development&#10;of deep learning (DL) systems. Neural network models outperform traditional&#10;approaches in domains where large datasets exist, such as SQuAD (ca. 100,000&#10;questions) for Wikipedia articles. However, these systems have not yet been&#10;applied to QA in more specific domains, such as biomedicine, because datasets&#10;are generally too small to train a DL system from scratch. For example, the&#10;BioASQ dataset for biomedical QA comprises less then 900 factoid (single&#10;answer) and list (multiple answers) QA instances. In this work, we adapt a&#10;neural QA system trained on a large open-domain dataset (SQuAD, source) to a&#10;biomedical dataset (BioASQ, target) by employing various transfer learning&#10;techniques. Our network architecture is based on a state-of-the-art QA system,&#10;extended with biomedical word embeddings and a novel mechanism to answer list&#10;questions. In contrast to existing biomedical QA systems, our system does not&#10;rely on domain-specific ontologies, parsers or entity taggers, which are&#10;expensive to create. Despite this fact, our systems achieve state-of-the-art&#10;results on factoid questions and competitive results on list questions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05572" label="2010.05572">
        <attvalues>
          <attvalue for="0" value="Meta-Context Transformers for Domain-Specific Response Generation" />
          <attvalue for="1" value="  Despite the tremendous success of neural dialogue models in recent years, it&#10;suffers a lack of relevance, diversity, and some times coherence in generated&#10;responses. Lately, transformer-based models, such as GPT-2, have revolutionized&#10;the landscape of dialogue generation by capturing the long-range structures&#10;through language modeling. Though these models have exhibited excellent&#10;language coherence, they often lack relevance and terms when used for&#10;domain-specific response generation. In this paper, we present DSRNet (Domain&#10;Specific Response Network), a transformer-based model for dialogue response&#10;generation by reinforcing domain-specific attributes. In particular, we extract&#10;meta attributes from context and infuse them with the context utterances for&#10;better attention over domain-specific key terms and relevance. We study the use&#10;of DSRNet in a multi-turn multi-interlocutor environment for domain-specific&#10;response generation. In our experiments, we evaluate DSRNet on Ubuntu dialogue&#10;datasets, which are mainly composed of various technical domain related&#10;dialogues for IT domain issue resolutions and also on CamRest676 dataset, which&#10;contains restaurant domain conversations. Trained with maximum likelihood&#10;objective, our model shows significant improvement over the state-of-the-art&#10;for multi-turn dialogue systems supported by better BLEU and semantic&#10;similarity (BertScore) scores. Besides, we also observe that the responses&#10;produced by our model carry higher relevance due to the presence of&#10;domain-specific key attributes that exhibit better overlap with the attributes&#10;of the context. Our analysis shows that the performance improvement is mostly&#10;due to the infusion of key terms along with dialogues which result in better&#10;attention over domain-relevant terms. Other contributing factors include joint&#10;modeling of dialogue context with the domain-specific meta attributes and&#10;topics.&#10;" />
          <attvalue for="2" value="&#10;&#10;Transformer-based pertained language models, such as BERT \cite{devlin2018bert}, GPT-2 \cite{GPT2pre,radford2018improving}, Transformer-XL \cite{Dai2019transformerxl}, XLNet \cite{Yang2019xlnet}, have revolutionized the landscape of natural language processing lately. These models have achieved state-of-the-art performance on many tasks, such as natural language understanding (NLU), sentence classification, named entity recognition and question answering. The ability to capture the long-range temporal dependencies in the input sequences is one of the key reason behind the success of these models. Besides language coherency, such attributes are beneficial to dialogue response modeling, especially in multi-turn and multi-interlocutor scenarios. The GPT-2 based models \cite{radford2018improving}, pre-trained on a large dataset, have demonstrated that the generated text is fluent, lexically diverse and rich in content. Such models have the capacity to capture textual data with fine granularity and produce output with a high-resolution that closely emulates real-world text written by human.&#10;&#10;Most of the existing neural dialogue response generation models are based on recurrent neural networks(RNNs) These neural response generation system suffers from content or style inconsistencies, lack of long-term contextual information \cite{serban17} &#10;and blandness \cite{li-etal-2016-diversity,Zhang2019ReCoSa}. Many of the issues are alleviated by modeling strategies specifically designed to boost information content using transformer based architecture \cite{GPT2pre} which is evident in some of the recent work on dialogue response generation modeling \cite{Olabiyi2019DLGNet,Bao20soloist,bao20scgpt,zhang20dialogpt}. &#10;&#10;These models have yielded promising results by generating mostly coherent responses given the dialogue context. However, most of them, including the state-of-the-art models trained with naturalistic dialogue data, still perform below human level. Generated responses tend to be either generic, out-of-context or disproportionately short. Some of the previous works attributed such behavior to various causes e.g. prevalence of generic utterances in training data, inadequate sized model architecture to capture the long term temporal dependence, absence of low frequency words in vocab, exposure bias in training models. In a domain specific multi-turn and multi interlocutor dialogue environment, where multiple users converse over a common channel simultaneously, often regarding a common subject, the above stated problems exacerbate in the generated response. More specifically, the next utterance may not be the response corresponding the immediate previous utterance. In addition, due to multiple agents conversing on the same channel, the context of utterance's might be off to the main content of the discussion. &#10;&#10;In this paper, we propose DSRNet (Domain Specific Response Network), a transformer based model, where in we alleviate some of the highlighted issues stated above by explicitly inserting meta-context attributes to capture the context better. &#10;In particular, we extract various meta contexts such as topic, queries etc. (see in Figure \ref{fig:my_label}) and include them as special inputs to our proposed DSRNet. DSRNet architecture is build on GPT-2 \cite{radford2018improving}, with modifications to include the meta contexts in the input set. We have experimented on the Ubuntu dialogue datasets \cite{lowe2015ubuntu,kummerfeld2018large}, which are mainly composed of various computer domain related dialogues for issue resolutions. Like GPT-2, DSRNet is formulated as an auto-regressive (AR) language model and uses a multi-layer transformer model architecture. However, unlike GPT-2, DSRNet is trained on large scale dialogues pairs/conversation-sessions extracted from Ubuntu-Dialogue corpus. Our assumption is that this should enable DSRNet to capture the joint distribution over the response and the previous utterance in the conversation flow with finer granularity. The input of DSRNet includes the context which constitutes of a predefined number of previous utterances (before the response) in the conversation, and the meta-contexts. The meta context is composed of conversation topic, query, entities which are extracted from the conversation at hand using traditional NLP approaches. &#10;We have evaluated DSRNet on the Ubuntu-IRC corpus (multi-interlocutor conversation) \cite{kummerfeld2018large} to generate response utterances which clearly indicate improved response text in terms of alignment with context utterances of the conversation topic. For domain specific environment, it is of great importance to have the response aligned with the context instead of being generic. We have also experimented with other datasets, namely Ubuntu 2.0 dataset (direct conversation) \cite{lowe2015ubuntu} (mainly pertains to the IT domain) and CamRest676 \cite{wenN2N17} which contains restaurant related conversations. The experimental results on CamRest676 dataset also corroborates improved response text generation. We built DSRNet upon the Huggingface Pytorch Transformer \cite{wolf2019huggingfaces}. We also extended the Ubuntu-IRC, Ubuntu 2.0 and CamRest676 dataset with meta-contexts. We intend to release both the source code and extended datasets for future research.&#10;To the best of our knowledge, our approach is the first to consider explicitly meta-context attributes and leverage it in a transformer based model to generate dialogue responses in a multi-turn dialogue environment. &#10;The key contributions of our work are as follows:&#10;\begin{itemize}&#10;\item We propose a novel approach, DSRNet, a GPT-2 based model with meta-context attributes for domain specific multi-turn and multi-interlocutor dialogue response generation. &#10;\item We extend Ubuntu 2.0, Ubuntu-IRC and CamRest676 datasets with meta-context attributes for better context capturing.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Neural Language Processing, Artificial Intelligence, Domain Specific Modeling, Dialogue Response Generation" />
        </attvalues>
      </node>
      <node id="2208.02578" label="2208.02578">
        <attvalues>
          <attvalue for="0" value="N-best Response-based Analysis of Contradiction-awareness in Neural&#10;  Response Generation Models" />
          <attvalue for="1" value="  Avoiding the generation of responses that contradict the preceding context is&#10;a significant challenge in dialogue response generation. One feasible method is&#10;post-processing, such as filtering out contradicting responses from a resulting&#10;n-best response list. In this scenario, the quality of the n-best list&#10;considerably affects the occurrence of contradictions because the final&#10;response is chosen from this n-best list. This study quantitatively analyzes&#10;the contextual contradiction-awareness of neural response generation models&#10;using the consistency of the n-best lists. Particularly, we used polar&#10;questions as stimulus inputs for concise and quantitative analyses. Our tests&#10;illustrate the contradiction-awareness of recent neural response generation&#10;models and methodologies, followed by a discussion of their properties and&#10;limitations.&#10;" />
          <attvalue for="2" value=" &#10;Recent advanced response generation models~\cite{zhang:acl2020demo:dialogpt, adiwardana:arxiv2020:meena, roller:eacl2021:blenderbot} can generate relevant and meaningful responses, which can resolve dull response problems~\cite{vinyals:icml2015ws:neuralconv, sordoni:naacl2015:gen-context-sensitive, serban:aaai2016:HRED}.&#10;This advancement reveals additional flaws in the quality of neural model responses, such as contradiction.&#10;Contradiction is a critical error in dialogue because a single contradictory response can disrupt the flow of the dialogue~\cite{higashinaka:sigdial2015:taxonomy}.&#10;&#10;A generation model outputs a response by selecting the candidate with the highest likelihood ($1$-best) from an $n$-best candidate list.&#10;Prior work has demonstrated that generating the $n$-best lists with noncontradictory $1$-bests is an open challenge~\cite{nie:acl2020:i-like-fish, kim:emnlp2020:will-i-sound-like-me, li:acl2021:addressing}.&#10;Thus, one practical technique for avoiding contradiction is to have an accurate contradiction detector that eliminates all contradictory candidates from the $n$-best list~\cite{nie:acl2020:i-like-fish}.&#10;In this scenario, the consistency of all candidates in the $n$-best list, not just the $1$-best, substantially impacts whether the final output is contradictory because the final response is chosen from the $n$-best list.&#10;Nonetheless, earlier quantitative investigations of contradiction relied solely on $1$-bests from models~\cite{li:acl2021:addressing}.&#10;&#10;In this study, we analyze the $n$-best lists generated by the models to explore methods for enhancing neural response generation to avoid contradiction.&#10;Specifically, we first consider how analyzing an $n$-best list should be approached. &#10;Then, we propose a method for statistically analyzing the $n$-best lists (Figure~\ref{fig:overview}). &#10;Since it is impractical to study all conceivable contradictions in a dialogue, we first focus on contradictions in response to polar questions.&#10;We use our method to highlight the contradiction-awareness of recent high-performance neural response generation models and methodologies.&#10;Our results show that beam search has limitations in terms of avoiding contradiction and that the newer techniques, such as unlikelihood training~\cite{welleck:iclr2020:unlikelihood}, can help overcome these limitations.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Contextual Contradiction Analysis, Linguistics, Artificial Intelligence, Neural Model Evaluation, Natural Language Processing, Dialogue Response Generation" />
        </attvalues>
      </node>
      <node id="2012.13391" label="2012.13391">
        <attvalues>
          <attvalue for="0" value="I like fish, especially dolphins: Addressing Contradictions in Dialogue&#10;  Modeling" />
          <attvalue for="1" value="  To quantify how well natural language understanding models can capture&#10;consistency in a general conversation, we introduce the DialoguE COntradiction&#10;DEtection task (DECODE) and a new conversational dataset containing both&#10;human-human and human-bot contradictory dialogues. We then compare a structured&#10;utterance-based approach of using pre-trained Transformer models for&#10;contradiction detection with the typical unstructured approach. Results reveal&#10;that: (i) our newly collected dataset is notably more effective at providing&#10;supervision for the dialogue contradiction detection task than existing NLI&#10;data including those aimed to cover the dialogue domain; (ii) the structured&#10;utterance-based approach is more robust and transferable on both analysis and&#10;out-of-distribution dialogues than its unstructured counterpart. We also show&#10;that our best contradiction detection model correlates well with human&#10;judgments and further provide evidence for its usage in both automatically&#10;evaluating and improving the consistency of state-of-the-art generative&#10;chatbots.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2108.13653" label="2108.13653">
        <attvalues>
          <attvalue for="0" value="Explaining Classes through Word Attribution" />
          <attvalue for="1" value="  In recent years, several methods have been proposed for explaining individual&#10;predictions of deep learning models, yet there has been little study of how to&#10;aggregate these predictions to explain how such models view classes as a whole&#10;in text classification tasks. In this work, we propose a method for explaining&#10;classes using deep learning models and the Integrated Gradients feature&#10;attribution technique by aggregating explanations of individual examples in&#10;text classification to general descriptions of the classes. We demonstrate the&#10;approach on Web register (genre) classification using the XML-R model and the&#10;Corpus of Online Registers of English (CORE), finding that the method&#10;identifies plausible and discriminative keywords characterizing all but the&#10;smallest class.&#10;" />
          <attvalue for="2" value="&#10;&#10;We propose a method for explaining classes in text classification tasks using deep learning models and feature attribution techniques, such as the Integrated Gradients (IG) method introduced by \cite{sundararajan2017axiomatic}. We focus specifically on IG as it provides a general framework for estimating feature importance in deep neural networks and has been shown to provide reliable saliency maps in text classification tasks among others \cite{bastings2020elephant,kokhlikyan2020captum}. &#10;&#10;Recently, explaining the predictions of deep neural networks has attracted a considerable amount of research interest in fields such as NLP and computer vision. Given the importance of this endeavour, several different techniques have been suggested in order to interpret model predictions \cite[see][for recent discussion]{montavon2018}. Nevertheless, these tend to focus on explaining individual predictions rather than how models perceive whole classes. To the best of our knowledge, we present the first method for aggregating explanations of individual examples in text classification to general descriptions of the classes. The method consists of three steps: 1) repeated model training and application of IG on random train/test splits, 2) aggregation of word scores of individual examples and extraction of keywords, and 3) filtering to remove spurious keywords.&#10;&#10;We test this method by training Transformer-based text classifiers on a large Web register identification corpus and show that it is able to provide descriptive keywords for the classes. The class descriptions provide both linguistic insight and a means for analyzing and debugging neural classification models in text classification.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Model Interpretability, Deep Learning Models, Text Classification, Computer Science, Linguistics, Mathematics, Statistics" />
        </attvalues>
      </node>
      <node id="2010.05607" label="2010.05607">
        <attvalues>
          <attvalue for="0" value="The elephant in the interpretability room: Why use attention as&#10;  explanation when we have saliency methods?" />
          <attvalue for="1" value="  There is a recent surge of interest in using attention as explanation of&#10;model predictions, with mixed evidence on whether attention can be used as&#10;such. While attention conveniently gives us one weight per input token and is&#10;easily extracted, it is often unclear toward what goal it is used as&#10;explanation. We find that often that goal, whether explicitly stated or not, is&#10;to find out what input tokens are the most relevant to a prediction, and that&#10;the implied user for the explanation is a model developer. For this goal and&#10;user, we argue that input saliency methods are better suited, and that there&#10;are no compelling reasons to use attention, despite the coincidence that it&#10;provides a weight for each input. With this position paper, we hope to shift&#10;some of the recent focus on attention to saliency methods, and for authors to&#10;clearly state the goal and user for their explanations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1706.07979" label="1706.07979">
        <attvalues>
          <attvalue for="0" value="Methods for Interpreting and Understanding Deep Neural Networks" />
          <attvalue for="1" value="  This paper provides an entry point to the problem of interpreting a deep&#10;neural network model and explaining its predictions. It is based on a tutorial&#10;given at ICASSP 2017. It introduces some recently proposed techniques of&#10;interpretation, along with theory, tricks and recommendations, to make most&#10;efficient use of these techniques on real data. It also discusses a number of&#10;practical applications.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.14771" label="2310.14771">
        <attvalues>
          <attvalue for="0" value="Evaluating the Knowledge Base Completion Potential of GPT" />
          <attvalue for="1" value="  Structured knowledge bases (KBs) are an asset for search engines and other&#10;applications, but are inevitably incomplete. Language models (LMs) have been&#10;proposed for unsupervised knowledge base completion (KBC), yet, their ability&#10;to do this at scale and with high accuracy remains an open question. Prior&#10;experimental studies mostly fall short because they only evaluate on popular&#10;subjects, or sample already existing facts from KBs. In this work, we perform a&#10;careful evaluation of GPT's potential to complete the largest public KB:&#10;Wikidata. We find that, despite their size and capabilities, models like GPT-3,&#10;ChatGPT and GPT-4 do not achieve fully convincing results on this task.&#10;Nonetheless, they provide solid improvements over earlier approaches with&#10;smaller LMs. In particular, we show that, with proper thresholding, GPT-3&#10;enables to extend Wikidata by 27M facts at 90% precision.&#10;" />
          <attvalue for="2" value="&#10;&#10;Structured knowledge bases (KBs) like Wikidata~\cite{wikidata}, DBpedia~\cite{dbpedia}, and Yago~\cite{yago} are employed in many knowledge-centric applications like search, question answering and dialogue. &#10;Constructing and completing these KBs at high quality and scale is a long-standing research challenge, and multiple benchmarks exist, e.g., FB15k~\cite{bordes2013translating}, CoDEx~\cite{safavi2020codex}, and LM-KBC22~\cite{singhania2022lm}. &#10;Text-extraction, knowledge graph embeddings, and LM-based knowledge extraction have continuously moved scores upwards on these tasks, and leaderboard portals like Paperswithcode provide evidence for that.&#10;&#10;Recently, LMs have been purported as a promising source of structured knowledge. Starting from the seminal LAMA paper \cite{petroni}, a throve of works have explored how to better probe, train, or fine-tune these LMs \cite{liu2021pre}. &#10;&#10;Nonetheless, we observe a certain divide between these late-breaking investigations, and practical KB completion. While recent LM-based approaches often focus on simple methodologies that produce fast results, practical KBC so far is a highly precision-oriented, extremely laborious process, involving a very high degree of manual labour, either for manually creating statements \cite{wikidata}, or for building comprehensive scraping, cleaning, validation, and normalization pipelines \cite{dbpedia,yago}. For example, part of Yago's success stems from its validated $&gt;$95\% accuracy, and according to \cite{weikum-machine-knowledge}, the Google Knowledge Vault was not deployed into production partly because it did not achieve 99\% accuracy. Yet, many previous LM analyses balance precision and recall or report precision/hits@k values, implicitly tuning systems towards balanced recall scores resulting in impractical precision.&#10;It is also important to keep in mind the scale of KBs: Wikidata currently contains around 100 million entities and 1.2B statements. The cost of producing such KBs is massive. An estimate from 2018 sets the cost per statement at 2 \$ for manually curated statement, and 1 ct for automatically extracted ones \cite{paulheim}. Thus, even small additions in relative terms might correspond to massive gains in absolute numbers. For example, even by the lower estimate of 1 ct/statement, adding one statement to just 1\% of Wikidata humans would come at a cost of 100,000 \$.&#10;&#10;In this paper, we conduct a systematic analysis of the KB completion potential of GPT, where we focus on high precision. We evaluate by employing (i) a recent KB completion benchmark, WD-Known, \cite{veseli2023evaluating}, which randomly samples facts from Wikidata &#10;and (ii) by a manual evaluation of subject-relation pairs without object values.&#10;Our main results are:&#10;\begin{enumerate}&#10; \item For the long-tail entities of WD-Known, GPT models perform considerably worse than what &#10; less demanding&#10; benchmarks like LAMA \cite{petroni} have indicated. Nonetheless, we can achieve solid results for language-related, socio-demographic relations (e.g., nativeLanguage).&#10; \item Despite their fame and size, out of the box, the GPT models, including GPT-4, do not produce statements of a high enough accuracy as typically required for KB completion.&#10; \item With simple thresholding, for the first time, we obtain a method that can extend the Wikidata KB at extremely high quality ($&gt;$90\% precision), at the scale of millions of statements. Based on our analysis of 41 common relations, we would be able to add a total of 27M high-accuracy statements.&#10;&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Artificial Intelligence Scaling, Computer Science, Knowledge Base Completion, Linguistics, Language Model Evaluation, Artificial Intelligence, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2009.07810" label="2009.07810">
        <attvalues>
          <attvalue for="0" value="CoDEx: A Comprehensive Knowledge Graph Completion Benchmark" />
          <attvalue for="1" value="  We present CoDEx, a set of knowledge graph completion datasets extracted from&#10;Wikidata and Wikipedia that improve upon existing knowledge graph completion&#10;benchmarks in scope and level of difficulty. In terms of scope, CoDEx comprises&#10;three knowledge graphs varying in size and structure, multilingual descriptions&#10;of entities and relations, and tens of thousands of hard negative triples that&#10;are plausible but verified to be false. To characterize CoDEx, we contribute&#10;thorough empirical analyses and benchmarking experiments. First, we analyze&#10;each CoDEx dataset in terms of logical relation patterns. Next, we report&#10;baseline link prediction and triple classification results on CoDEx for five&#10;extensively tuned embedding models. Finally, we differentiate CoDEx from the&#10;popular FB15K-237 knowledge graph completion dataset by showing that CoDEx&#10;covers more diverse and interpretable content, and is a more difficult link&#10;prediction benchmark. Data, code, and pretrained models are available at&#10;https://bit.ly/2EPbrJs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.11564" label="2009.11564">
        <attvalues>
          <attvalue for="0" value="Machine Knowledge: Creation and Curation of Comprehensive Knowledge&#10;  Bases" />
          <attvalue for="1" value="  Equipping machines with comprehensive knowledge of the world's entities and&#10;their relationships has been a long-standing goal of AI. Over the last decade,&#10;large-scale knowledge bases, also known as knowledge graphs, have been&#10;automatically constructed from web contents and text sources, and have become a&#10;key asset for search engines. This machine knowledge can be harnessed to&#10;semantically interpret textual phrases in news, social media and web tables,&#10;and contributes to question answering, natural language processing and data&#10;analytics. This article surveys fundamental concepts and practical methods for&#10;creating and curating large knowledge bases. It covers models and methods for&#10;discovering and canonicalizing entities and their semantic types and organizing&#10;them into clean taxonomies. On top of this, the article discusses the automatic&#10;extraction of entity-centric properties. To support the long-term life-cycle&#10;and the quality assurance of machine knowledge, the article presents methods&#10;for constructing open schemas and for knowledge curation. Case studies on&#10;academic projects and industrial knowledge graphs complement the survey of&#10;concepts and methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2303.11082" label="2303.11082">
        <attvalues>
          <attvalue for="0" value="Evaluating Language Models for Knowledge Base Completion" />
          <attvalue for="1" value="  Structured knowledge bases (KBs) are a foundation of many intelligent&#10;applications, yet are notoriously incomplete. Language models (LMs) have&#10;recently been proposed for unsupervised knowledge base completion (KBC), yet,&#10;despite encouraging initial results, questions regarding their suitability&#10;remain open. Existing evaluations often fall short because they only evaluate&#10;on popular subjects, or sample already existing facts from KBs. In this work,&#10;we introduce a novel, more challenging benchmark dataset, and a methodology&#10;tailored for a realistic assessment of the KBC potential of LMs. For automated&#10;assessment, we curate a dataset called WD-KNOWN, which provides an unbiased&#10;random sample of Wikidata, containing over 3.9 million facts. In a second step,&#10;we perform a human evaluation on predictions that are not yet in the KB, as&#10;only this provides real insights into the added value over existing KBs. Our&#10;key finding is that biases in dataset conception of previous benchmarks lead to&#10;a systematic overestimate of LM performance for KBC. However, our results also&#10;reveal strong areas of LMs. We could, for example, perform a significant&#10;completion of Wikidata on the relations nativeLanguage, by a factor of ~21&#10;(from 260k to 5.8M) at 82% precision, usedLanguage, by a factor of ~2.1 (from&#10;2.1M to 6.6M) at 82% precision, and citizenOf by a factor of ~0.3 (from 4.2M to&#10;5.3M) at 90% precision. Moreover, we find that LMs possess surprisingly strong&#10;generalization capabilities: even on relations where most facts were not&#10;directly observed in LM training, prediction quality can be high.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.00142" label="2211.00142">
        <attvalues>
          <attvalue for="0" value="TaTa: A Multilingual Table-to-Text Dataset for African Languages" />
          <attvalue for="1" value="  Existing data-to-text generation datasets are mostly limited to English. To&#10;address this lack of data, we create Table-to-Text in African languages (TaTa),&#10;the first large multilingual table-to-text dataset with a focus on African&#10;languages. We created TaTa by transcribing figures and accompanying text in&#10;bilingual reports by the Demographic and Health Surveys Program, followed by&#10;professional translation to make the dataset fully parallel. TaTa includes&#10;8,700 examples in nine languages including four African languages (Hausa, Igbo,&#10;Swahili, and Yor\`ub\'a) and a zero-shot test language (Russian). We&#10;additionally release screenshots of the original figures for future research on&#10;multilingual multi-modal approaches. Through an in-depth human evaluation, we&#10;show that TaTa is challenging for current models and that less than half the&#10;outputs from an mT5-XXL-based model are understandable and attributable to the&#10;source data. We further demonstrate that existing metrics perform poorly for&#10;TaTa and introduce learned metrics that achieve a high correlation with human&#10;judgments. We release all data and annotations at&#10;https://github.com/google-research/url-nlp.&#10;" />
          <attvalue for="2" value="&#10;&#10;Generating text based on structured data is a classic natural language generation (NLG) problem that still poses significant challenges to current models. Despite the recent increase in work focusing on creating multilingual and cross-lingual resources for NLP \cite{nekoto-etal-2020-participatory,ponti-etal-2020-xcopa,ruder-etal-2021-xtreme}, data-to-text datasets are mostly limited to English and a small number of other languages. Data-to-text generation presents important opportunities in multilingual settings, e.g., the expansion of widely used knowledge sources, such as Wikipedia to under-represented languages \cite{lebret-etal-2016-neural}. Data-to-text tasks are also an effective testbed to assess reasoning capabilities of models~\cite{suadaa-etal-2021-towards}.&#10;&#10;However, creating challenging, high-quality datasets for NLG is difficult.&#10;Datasets frequently suffer from outputs that are not attributable to the inputs or are unnatural, and overly simple tasks fail to identify model limitations~\cite{parikh-etal-2020-totto,thomson-etal-2020-sportsett,DBLP:journals/corr/abs-2111-06467}.&#10;To provide a high-quality dataset for multilingual data-to-text generation, we introduce Table-to-Text in African languages (\name). contains multiple references for each example, which require selecting important content, reasoning over multiple cells, and realizing it in the respective language (see Fig.~\ref{fig:teaser}). The dataset is parallel and covers nine languages, eight of which are spoken in Africa: Arabic, English, French, Hausa, Igbo, Portuguese, Swahili, \yoruba, and Russian. We create by transcribing and translating charts and their descriptions in informational reports by the Demographic and Health Surveys (DHS) Program, which publishes population, health, and nutrition data through more than 400 surveys in over 90 countries in PDF format. &#10;&#10;In an analysis of using professional annotators, we find that over 75\% of collected sentences require reasoning over and comparing multiple cells, which makes the dataset challenging for current models.&#10;Even our best baseline model generates attributable language less than half of the time, i.e., over half of model outputs are not faithful to the source.&#10;Moreover, we demonstrate that popular automatic metrics achieve very low correlations with human judgments and are thus unreliable. &#10;To mitigate this issue, we train our own metrics on human annotations, which we call StATA, and we use them to investigate the cross-lingual transfer properties of monolingually trained models. This setup identifies Swahili as the best transfer language whereas traditional metrics would have falsely indicated other languages. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Multilingual Data Generation, Evaluation Metrics Development, Table-to-Text Translation, Natural Language Processing, Statistics" />
        </attvalues>
      </node>
      <node id="2004.14373" label="2004.14373">
        <attvalues>
          <attvalue for="0" value="ToTTo: A Controlled Table-To-Text Generation Dataset" />
          <attvalue for="1" value="  We present ToTTo, an open-domain English table-to-text dataset with over&#10;120,000 training examples that proposes a controlled generation task: given a&#10;Wikipedia table and a set of highlighted table cells, produce a one-sentence&#10;description. To obtain generated targets that are natural but also faithful to&#10;the source table, we introduce a dataset construction process where annotators&#10;directly revise existing candidate sentences from Wikipedia. We present&#10;systematic analyses of our dataset and annotation process as well as results&#10;achieved by several state-of-the-art baselines. While usually fluent, existing&#10;methods often hallucinate phrases that are not supported by the table,&#10;suggesting that this dataset can serve as a useful research benchmark for&#10;high-precision conditional text generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2111.06467" label="2111.06467">
        <attvalues>
          <attvalue for="0" value="SynthBio: A Case Study in Human-AI Collaborative Curation of Text&#10;  Datasets" />
          <attvalue for="1" value="  NLP researchers need more, higher-quality text datasets. Human-labeled&#10;datasets are expensive to collect, while datasets collected via automatic&#10;retrieval from the web such as WikiBio are noisy and can include undesired&#10;biases. Moreover, data sourced from the web is often included in datasets used&#10;to pretrain models, leading to inadvertent cross-contamination of training and&#10;test sets. In this work we introduce a novel method for efficient dataset&#10;curation: we use a large language model to provide seed generations to human&#10;raters, thereby changing dataset authoring from a writing task to an editing&#10;task. We use our method to curate SynthBio - a new evaluation set for WikiBio -&#10;composed of structured attribute lists describing fictional individuals, mapped&#10;to natural language biographies. We show that our dataset of fictional&#10;biographies is less noisy than WikiBio, and also more balanced with respect to&#10;gender and nationality.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.02136" label="2402.02136">
        <attvalues>
          <attvalue for="0" value="User Intent Recognition and Satisfaction with Large Language Models: A&#10;  User Study with ChatGPT" />
          <attvalue for="1" value="  The rapid evolution of large language models such as GPT-4 Turbo represents&#10;an impactful paradigm shift in digital interaction and content engagement.&#10;While these models encode vast amounts of human-generated knowledge and excel&#10;in processing diverse data types, recent research shows that they often face&#10;the challenge of accurately responding to specific user intents, leading to&#10;increased user dissatisfaction. Based on a fine-grained intent taxonomy and&#10;intent-based prompt reformulations, we analyze (1) the quality of intent&#10;recognition and (2) user satisfaction with answers from intent-based prompt&#10;reformulations for two recent ChatGPT models, GPT-3.5 Turbo and GPT-4 Turbo.&#10;The results reveal that GPT-4 outperforms GPT-3.5 on the recognition of common&#10;intents, but is conversely often outperformed by GPT-3.5 on the recognition of&#10;less frequent intents. Moreover, whenever the user intent is correctly&#10;recognized, while users are more satisfied with the answers to intent-based&#10;reformulations of GPT 4 compared to GPT-3.5, they tend to be more satisfied&#10;with the answers of the models to their original prompts compared to the&#10;reformulated ones. Finally, the study indicates that users can quickly learn to&#10;formulate their prompts more effectively, once they are shown possible&#10;reformulation templates.&#10;" />
          <attvalue for="2" value="&#10;Generative AI models are rapidly evolving, especially those handling language and vision. Models like OpenAI's GPT-4 Turbo~\cite{OpenAI2023, openai2023gpt4v} and Google's Gemini~\cite{Deepmind2023} are at the forefront of this evolution, impacting how we interact with digital content and services. At the heart of this development are Large Language Models (LLMs). With human-like text processing, generation, and reasoning capabilities, LLMs have broad applications ranging from creative content generation to complex problem-solving~\cite{Brown2020, romera2023mathematical}. Incorporating human feedback and reinforcement learning in the training of LLMs, such as those used in ChatGPT, has further improved the alignment of these models with societal norms and goals~\cite{Ouyang2022, lee2023rlaif, yang2024crowdsourcing}. Similarly, models like Google's PaLM~\cite{Chowdhery2022,Anil2023} and Meta’s LLaMA~\cite{Touvron2023,Touvron2023a}, together with open-source variants like Vicuna~\cite{Zheng2023} and Alpaca~\cite{alpaca2023}, represent parallel advances in the field.&#10;&#10;Despite serious challenges in areas like mathematical reasoning~\cite{Imani2023}, student error detection~\cite{bewersdorff2023assessing}, and mitigating hallucinations in outputs~\cite{Ji2023,azamfirei2023large,zhang2023siren,manakul2023selfcheckgpt}, LLMs have already had impact in diverse sectors such as healthcare~\cite{Chintagunta2021,Enarvi2020}, finance~\cite{Dowling2023}, journalism~\cite{Pavlik2023}, creative writing~\cite{Yuan2022, Sessler2023}, and, more recently, to scientific discoveries~\cite{romera2023mathematical}. Efforts towards enhancing explainability are underway to foster trust and reliability in these models~\cite{Wu2023b}. &#10;&#10;Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}. Hence, the capabilities of LLMs to reason over natural language are essential for many of the recent breakthroughs in generative AI, such as NExT-GPT~\cite{Wu2023a}, OpenAI's GPT-4 Vision, GPT-4 Turbo, Google's Gemini, and Apple's Ferret~\cite{OpenAI2023,Deepmind2023,you2023ferret}. Such advancements, coupled with other deep learning and explainability techniques~\cite{borisov2022deep,rombach2022high}, can potentially revolutionize science~\cite{wong2023discovery} and society. &#10;&#10;Trained on extensive human-generated datasets and various web corpora, LLMs encode extensive knowledge and excel in processing and reasoning tasks over text. Yet, the effectiveness of their responses strongly depends on accurately inferring the user's intent, which is typically conveyed implicitly in the prompt. Accurately discerning and categorizing user intents in prompts poses considerable challenges for LLMs, mainly related to the inherent ambiguity, variability, or the clarity of the used language, cultural and contextual subtleties, and evolving user expectations. A recent study~\cite{kim2023understanding} highlighted a frequent source of user dissatisfaction with ChatGPT: its occasional failure to understand user intentions. The authors further observed that users often lack effective strategies to mitigate this dissatisfaction. Moreover, it was noted that users with a limited understanding of LLMs tend to experience greater dissatisfaction and are less proactive in addressing these challenges. &#10;&#10;In this paper, we explore the hypothesis that enhancing the understanding of a user's specific intent in their prompt can significantly improve the quality of responses generated by Large Language Models (LLMs). This assumption finds substantial backing in the recent advancements in natural language processing methodologies, notably the Chain-of-Thought, Tree-of-Thought, and Graph-of-Thought techniques~\cite{wei2022chain,yao2023tree,besta2023graph}. These approaches emphasize the importance of contextual and hierarchical understanding in processing user queries, which is crucial for effective human-computer interactions. To systematically approach this hypothesis, we first establish a comprehensive taxonomy of potential user intents. This taxonomy is carefully crafted, taking into account the distinct requirements of different intent types and by comprehensively integrating insights from recent and earlier interdisciplinary scholarly work on intent categorization~\cite{dang2022prompt, shah2023using,MediumChatGPTStats, SparktoroChatGPT} and well-established intent categories from information retrieval~\cite{azad2019query, broder2002taxonomy, kellar2007field, jansen2008determining, rose2004understanding, ashkan2009classifying, yi2008deciphering}. By situating our study within the interdisciplinary field of human-computer interaction (HCI), this work examines whether improved intent recognition can enhance user satisfaction and engagement with conversational AI systems.&#10;&#10;Our study then proceeds in two critical phases. First, we evaluate recent LLM versions, namely GPT-3.5 and GPT-4, to determine their proficiency in accurately recognizing the intent behind user prompts. We expect more recent LLM versions to outperform older ones on this task. This aspect is crucial, as correctly identifying intent is foundational to providing relevant and accurate responses. The second phase of our study focuses on the ramifications of prompt reformulation. Here, we explore whether accurately recognizing and then reformulating a user's prompt to better mirror their intended query leads to a significant improvement in user satisfaction with the quality of the LLM's responses. &#10;&#10;In brief, our work provides the following contributions:&#10;&#10;\begin{enumerate}&#10; \item The development of a taxonomy of potential intents in conversational contexts.&#10; \item Assessment of the intent recognition proficiency of GPT-3.5 and GPT-4 based on a user study.&#10; \item Exploration of the impact of prompt reformulation on response quality based on user satisfaction in a conversational context.&#10; \item An extensive data set, including an overview of our user study, is published on~\href{ for research purposes.&#10;&#10;\end{enumerate}&#10;&#10;Section~\ref{sec:related-work} presents existing literature on intent recognition and prompt optimization. We introduce our intent taxonomy in Section~\ref{sec:taxonomy}, laying the groundwork for the user study in Section~\ref{sec:study_description}. The study results are presented and discussed in Section~\ref{sec:results}, before concluding in Section~\ref{sec:conclusion}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Computer Science, User Satisfaction Analysis, Linguistics, Language Model Evolution, Intent Recognition Challenges, Artificial Intelligence, Natural Language Processing, Psychology" />
        </attvalues>
      </node>
      <node id="2303.05398" label="2303.05398">
        <attvalues>
          <attvalue for="0" value="MathPrompter: Mathematical Reasoning using Large Language Models" />
          <attvalue for="1" value="  Large Language Models (LLMs) have limited performance when solving arithmetic&#10;reasoning tasks and often provide incorrect answers. Unlike natural language&#10;understanding, math problems typically have a single correct answer, making the&#10;task of generating accurate solutions more challenging for LLMs. To the best of&#10;our knowledge, we are not aware of any LLMs that indicate their level of&#10;confidence in their responses which fuels a trust deficit in these models&#10;impeding their adoption. To address this deficiency, we propose `MathPrompter',&#10;a technique that improves performance of LLMs on arithmetic problems along with&#10;increased reliance in the predictions. MathPrompter uses the Zero-shot&#10;chain-of-thought prompting technique to generate multiple Algebraic expressions&#10;or Python functions to solve the same math problem in different ways and&#10;thereby raise the confidence level in the output results. This is in contrast&#10;to other prompt based CoT methods, where there is no check on the validity of&#10;the intermediate steps followed. Our technique improves over state-of-the-art&#10;on the MultiArith dataset ($78.7\%\rightarrow92.5\%$) evaluated using 175B&#10;parameter GPT-based LLM.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.06088" label="2308.06088">
        <attvalues>
          <attvalue for="0" value="Assessing Student Errors in Experimentation Using Artificial&#10;  Intelligence and Large Language Models: A Comparative Study with Human Raters" />
          <attvalue for="1" value="  Identifying logical errors in complex, incomplete or even contradictory and&#10;overall heterogeneous data like students' experimentation protocols is&#10;challenging. Recognizing the limitations of current evaluation methods, we&#10;investigate the potential of Large Language Models (LLMs) for automatically&#10;identifying student errors and streamlining teacher assessments. Our aim is to&#10;provide a foundation for productive, personalized feedback. Using a dataset of&#10;65 student protocols, an Artificial Intelligence (AI) system based on the&#10;GPT-3.5 and GPT-4 series was developed and tested against human raters. Our&#10;results indicate varying levels of accuracy in error detection between the AI&#10;system and human raters. The AI system can accurately identify many fundamental&#10;student errors, for instance, the AI system identifies when a student is&#10;focusing the hypothesis not on the dependent variable but solely on an expected&#10;observation (acc. = 0.90), when a student modifies the trials in an ongoing&#10;investigation (acc. = 1), and whether a student is conducting valid test trials&#10;(acc. = 0.82) reliably. The identification of other, usually more complex&#10;errors, like whether a student conducts a valid control trial (acc. = .60),&#10;poses a greater challenge. This research explores not only the utility of AI in&#10;educational settings, but also contributes to the understanding of the&#10;capabilities of LLMs in error detection in inquiry-based learning like&#10;experimentation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.07356" label="2110.07356">
        <attvalues>
          <attvalue for="0" value="Medically Aware GPT-3 as a Data Generator for Medical Dialogue&#10;  Summarization" />
          <attvalue for="1" value="  In medical dialogue summarization, summaries must be coherent and must&#10;capture all the medically relevant information in the dialogue. However,&#10;learning effective models for summarization require large amounts of labeled&#10;data which is especially hard to obtain. We present an algorithm to create&#10;synthetic training data with an explicit focus on capturing medically relevant&#10;information. We utilize GPT-3 as the backbone of our algorithm and scale 210&#10;human labeled examples to yield results comparable to using 6400 human labeled&#10;examples (~30x) leveraging low-shot learning and an ensemble method. In&#10;detailed experiments, we show that this approach produces high quality training&#10;data that can further be combined with human labeled data to get summaries that&#10;are strongly preferable to those produced by models trained on human data alone&#10;both in terms of medical accuracy and coherency.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.07704" label="2310.07704">
        <attvalues>
          <attvalue for="0" value="Ferret: Refer and Ground Anything Anywhere at Any Granularity" />
          <attvalue for="1" value="  We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of&#10;understanding spatial referring of any shape or granularity within an image and&#10;accurately grounding open-vocabulary descriptions. To unify referring and&#10;grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid&#10;region representation that integrates discrete coordinates and continuous&#10;features jointly to represent a region in the image. To extract the continuous&#10;features of versatile regions, we propose a spatial-aware visual sampler, adept&#10;at handling varying sparsity across different shapes. Consequently, Ferret can&#10;accept diverse region inputs, such as points, bounding boxes, and free-form&#10;shapes. To bolster the desired capability of Ferret, we curate GRIT, a&#10;comprehensive refer-and-ground instruction tuning dataset including 1.1M&#10;samples that contain rich hierarchical spatial knowledge, with 95K hard&#10;negative data to promote model robustness. The resulting model not only&#10;achieves superior performance in classical referring and grounding tasks, but&#10;also greatly outperforms existing MLLMs in region-based and&#10;localization-demanded multimodal chatting. Our evaluations also reveal a&#10;significantly improved capability of describing image details and a remarkable&#10;alleviation in object hallucination. Code and data will be available at&#10;https://github.com/apple/ml-ferret&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.01889" label="2110.01889">
        <attvalues>
          <attvalue for="0" value="Deep Neural Networks and Tabular Data: A Survey" />
          <attvalue for="1" value="  Heterogeneous tabular data are the most commonly used form of data and are&#10;essential for numerous critical and computationally demanding applications. On&#10;homogeneous data sets, deep neural networks have repeatedly shown excellent&#10;performance and have therefore been widely adopted. However, their adaptation&#10;to tabular data for inference or data generation tasks remains challenging. To&#10;facilitate further progress in the field, this work provides an overview of&#10;state-of-the-art deep learning methods for tabular data. We categorize these&#10;methods into three groups: data transformations, specialized architectures, and&#10;regularization models. For each of these groups, our work offers a&#10;comprehensive overview of the main approaches. Moreover, we discuss deep&#10;learning approaches for generating tabular data, and we also provide an&#10;overview over strategies for explaining deep models on tabular data. Thus, our&#10;first contribution is to address the main research streams and existing&#10;methodologies in the mentioned areas, while highlighting relevant challenges&#10;and open research questions. Our second contribution is to provide an empirical&#10;comparison of traditional machine learning methods with eleven deep learning&#10;approaches across five popular real-world tabular data sets of different sizes&#10;and with different learning objectives. Our results, which we have made&#10;publicly available as competitive benchmarks, indicate that algorithms based on&#10;gradient-boosted tree ensembles still mostly outperform deep learning models on&#10;supervised learning tasks, suggesting that the research progress on competitive&#10;deep learning models for tabular data is stagnating. To the best of our&#10;knowledge, this is the first in-depth overview of deep learning approaches for&#10;tabular data; as such, this work can serve as a valuable starting point to&#10;guide researchers and practitioners interested in deep learning with tabular&#10;data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.11630" label="2409.11630">
        <attvalues>
          <attvalue for="0" value="Speaking from Coarse to Fine: Improving Neural Codec Language Model via&#10;  Multi-Scale Speech Coding and Generation" />
          <attvalue for="1" value="  The neural codec language model (CLM) has demonstrated remarkable performance&#10;in text-to-speech (TTS) synthesis. However, troubled by ``recency bias&quot;, CLM&#10;lacks sufficient attention to coarse-grained information at a higher temporal&#10;scale, often producing unnatural or even unintelligible speech. This work&#10;proposes CoFi-Speech, a coarse-to-fine CLM-TTS approach, employing multi-scale&#10;speech coding and generation to address this issue. We train a multi-scale&#10;neural codec, CoFi-Codec, to encode speech into a multi-scale discrete&#10;representation, comprising multiple token sequences with different time&#10;resolutions. Then, we propose CoFi-LM that can generate this representation in&#10;two modes: the single-LM-based chain-of-scale generation and the&#10;multiple-LM-based stack-of-scale generation. In experiments, CoFi-Speech&#10;significantly outperforms single-scale baseline systems on naturalness and&#10;speaker similarity in zero-shot TTS. The analysis of multi-scale coding&#10;demonstrates the effectiveness of CoFi-Codec in learning multi-scale discrete&#10;speech representations while keeping high-quality speech reconstruction. The&#10;coarse-to-fine multi-scale generation, especially for the stack-of-scale&#10;approach, is also validated as a crucial approach in pursuing a high-quality&#10;neural codec language model for TTS.&#10;" />
          <attvalue for="2" value="&#10;&#10;The success of large language models (LLMs) in text domain \cite{brown2020language,openai2023gpt4,touvron2023llama2} has demonstrated their great capability in discrete sequence generation. It also inspires the birth of a new text-to-speech synthesis (TTS) paradigm based on the neural codec language model (CLM) \cite{VALLEX, tortoise, lajszczak2024base}, which treats TTS as a next-token prediction task. This framework usually relies on a neural codec \cite{encodec, hifi-codec, dac} to encode the speech audio into discrete tokens, which can be incorporated with the text sequence and generated by the LM, i.e. an auto-regressive decoder. Finally, we obtain the speech audio from these generated speech tokens via the codec decoder.&#10;&#10;However, different from the text, the discrete speech sequence is much longer to keep sufficient capacity to preserve phonetic and acoustic information. This long sequence length not only increases the complexity of TTS modeling but aggregates the ``recency bias'' of LMs \cite{peysakhovich2023attention, wang2024eliminating}, i.e. overly focusing on recent tokens during auto-regressive generation. This issue makes LM focus less on coarse-grained information \cite{guo2023msmc}, e.g. phonetics, prosody, and speaking style at higher and different temporal scales, hence causing unstable TTS performance, producing unnatural or even unintelligible speech. Although monotonic attention constraints \cite{han2024vall, du2024vall, wang2024attention} are proposed to fix stability issues, they still cannot solve ``recency bias'' fundamentally. Some works \cite{tortoise, socodec, li2024single} turn to directly model shorter speech sequences with a larger frameshift to avoid this issue, but limits the fine-grained expression of LMs in TTS. This dilemma implies the necessity of applying guidance to LMs to pay sufficient attention to both coarse-grained and fine-grained information of speech. &#10;&#10;In this work, we propose a novel CLM-based zero-shot TTS approach, CoFi-Speech, that generates speech in a coarse-to-fine manner via a multi-scale speech coding and generation approach. In this framework, the multi-scale speech codec, CoFi-Codec, decomposes speech into multiple discrete sequences with different temporal resolutions and decodes them back with a high reconstruction quality. Then, we propose two LM-based approaches to predict this multi-scale speech representation from coarse to fine: single-LM-based chain-of-scale generation and multiple-LM-based stack-of-scale generation. In experiments, we present subjective and objective evaluations to demonstrate that CoFi-Speech significantly outperforms baseline systems based on single-scale speech sequences on naturalness and similarity, where stack-of-scale generation performs best. Finally, we conduct detailed ablation studies to analyze multi-scale coding and generation, to further validate the effectiveness of ``speaking from coarse to fine'' in achieving high-quality CLM-based TTS.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Text-to-Speech Synthesis, Electrical Engineering, Linguistics, Signal Processing, Neural Language Models, Speech Coding Techniques" />
        </attvalues>
      </node>
      <node id="2406.07855" label="2406.07855">
        <attvalues>
          <attvalue for="0" value="VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via&#10;  Monotonic Alignment" />
          <attvalue for="1" value="  With the help of discrete neural audio codecs, large language models (LLM)&#10;have increasingly been recognized as a promising methodology for zero-shot&#10;Text-to-Speech (TTS) synthesis. However, sampling based decoding strategies&#10;bring astonishing diversity to generation, but also pose robustness issues such&#10;as typos, omissions and repetition. In addition, the high sampling rate of&#10;audio also brings huge computational overhead to the inference process of&#10;autoregression. To address these issues, we propose VALL-E R, a robust and&#10;efficient zero-shot TTS system, building upon the foundation of VALL-E.&#10;Specifically, we introduce a phoneme monotonic alignment strategy to strengthen&#10;the connection between phonemes and acoustic sequence, ensuring a more precise&#10;alignment by constraining the acoustic tokens to match their associated&#10;phonemes. Furthermore, we employ a codec-merging approach to downsample the&#10;discrete codes in shallow quantization layer, thereby accelerating the decoding&#10;speed while preserving the high quality of speech output. Benefiting from these&#10;strategies, VALL-E R obtains controllablity over phonemes and demonstrates its&#10;strong robustness by approaching the WER of ground truth. In addition, it&#10;requires fewer autoregressive steps, with over 60% time reduction during&#10;inference. This research has the potential to be applied to meaningful&#10;projects, including the creation of speech for those affected by aphasia. Audio&#10;samples will be available at: https://aka.ms/valler.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.01633" label="2305.01633">
        <attvalues>
          <attvalue for="0" value="Missing Information, Unresponsive Authors, Experimental Flaws: The&#10;  Impossibility of Assessing the Reproducibility of Previous Human Evaluations&#10;  in NLP" />
          <attvalue for="1" value="  We report our efforts in identifying a set of previous human evaluations in&#10;NLP that would be suitable for a coordinated study examining what makes human&#10;evaluations in NLP more/less reproducible. We present our results and findings,&#10;which include that just 13\% of papers had (i) sufficiently low barriers to&#10;reproduction, and (ii) enough obtainable information, to be considered for&#10;reproduction, and that all but one of the experiments we selected for&#10;reproduction was discovered to have flaws that made the meaningfulness of&#10;conducting a reproduction questionable. As a result, we had to change our&#10;coordinated study design from a reproduce approach to a&#10;standardise-then-reproduce-twice approach. Our overall (negative) finding that&#10;the great majority of human evaluations in NLP is not repeatable and/or not&#10;reproducible and/or too flawed to justify reproduction, paints a dire picture,&#10;but presents an opportunity for a rethink about how to design and report human&#10;evaluations in NLP.&#10;" />
          <attvalue for="2" value="&#10;&#10;There is increasing awareness in Natural Language Processing (NLP) that reproducibility of results, most particularly of results from system evaluations, matters greatly, and that currently the field does not assess reproducibility of results rigorously enough, and lacks a common approach to it. Recent work has made progress particularly with respect to automatic evaluation \cite{pineau2020checklist, whitaker2017}, but reproducibility of human evaluation, widely considered the litmus test of quality in NLP, has received less attention. It could be argued that if it is not known how reproducible human evaluations are, it is not known how reliable they are; and if it is not known how reliable they are, then it is not known how reliable automatic evaluations meta-evaluated against them &#10;are either. &#10;&#10;The work reported in this paper forms part of the ReproHum project in which our aim is to build on existing work on recording properties of human evaluations datasheet-style \cite{shimorina-belz-2022-human}, and assessing how close results from a reproduction study are to the original study \cite{belz-etal-2022-quantified}, to investigate systematically what factors make a human evaluation more---or less---reproducible. In this paper, we present the findings from our work on the project so far which necessitated a rethink of our entire approach to designing such an investigation.&#10;&#10;Section~\ref{sec:motivation} outlines our motivation for carrying out a multi-lab multi-test (MLMT) study of factors affecting reproduciblity in NLP, and our original design for the study. Section~\ref{sec:sel-anno} describes our paper selection, annotation and filtering process which yielded a surprisingly small number of candidate papers for reproduction. In Section~\ref{sec:flaws} we describe the numerous further issues with original evaluation studies we encountered in the process of setting up reproductions of them with partner labs. Section~\ref{sec:conclusion} summarises our negative findings regarding the infeasibilty of assessing the reproducibility of previously conducted human evaluations in NLP as they are, and outlines the changes to our multi-lab multi-test study necessitated by the findings.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, NLP Evaluation Methods, Methodology, Natural Language Processing, Reproducibility Challenges, Human Evaluation Design" />
        </attvalues>
      </node>
      <node id="2204.05961" label="2204.05961">
        <attvalues>
          <attvalue for="0" value="Quantified Reproducibility Assessment of NLP Results" />
          <attvalue for="1" value="  This paper describes and tests a method for carrying out quantified&#10;reproducibility assessment (QRA) that is based on concepts and definitions from&#10;metrology. QRA produces a single score estimating the degree of reproducibility&#10;of a given system and evaluation measure, on the basis of the scores from, and&#10;differences between, different reproductions. We test QRA on 18 system and&#10;evaluation measure combinations (involving diverse NLP tasks and types of&#10;evaluation), for each of which we have the original results and one to seven&#10;reproduction results. The proposed QRA method produces&#10;degree-of-reproducibility scores that are comparable across multiple&#10;reproductions not only of the same, but of different original studies. We find&#10;that the proposed method facilitates insights into causes of variation between&#10;reproductions, and allows conclusions to be drawn about what changes to system&#10;and/or evaluation design might lead to improved reproducibility.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.03205" label="2406.03205">
        <attvalues>
          <attvalue for="0" value="CoLLAB: A Collaborative Approach for Multilingual Abuse Detection" />
          <attvalue for="1" value="  In this study, we investigate representations from paralingual Pre-Trained&#10;model (PTM) for Audio Abuse Detection (AAD), which has not been explored for&#10;AAD. Our results demonstrate their superiority compared to other PTM&#10;representations on the ADIMA benchmark. Furthermore, combining PTM&#10;representations enhances AAD performance. Despite these improvements,&#10;challenges with cross-lingual generalizability still remain, and certain&#10;languages require training in the same language. This demands individual models&#10;for different languages, leading to scalability, maintenance, and resource&#10;allocation issues and hindering the practical deployment of AAD systems in&#10;linguistically diverse real-world environments. To address this, we introduce&#10;CoLLAB, a novel framework that doesn't require training and allows seamless&#10;merging of models trained in different languages through weight-averaging. This&#10;results in a unified model with competitive AAD performance across multiple&#10;languages.&#10;" />
          <attvalue for="2" value="&#10;&#10;In today's digital era, online social media platforms, gaming communities, and digital environments have become indispensable aspects of modern life. Within these realms, audio plays a crucial role in facilitating real-time communication and fostering connections among users. However, alongside the advantages of audio communication in these settings, there is a growing concern regarding audio-abusive content (AAC). This refers to the malicious or inappropriate use of audio, encompassing behaviors such as verbal harassment, hate speech, and the dissemination of harmful or offensive material. The prevalence of AAC in digital spaces presents substantial challenges for both users and platform administrators. \par&#10;&#10;Due to the challenges induced by AAC to online safety and well-being, audio-based abuse detection (AAD) has caught recent attention. While abuse detection in other modalities like text \cite{davidson2017automated, madhu2023detecting} and visual \cite{alcantara2020offensive, gao2020offensive} content has garnered significant focus and development, AAD has not received comparable attention, despite its critical importance in safeguarding digital spaces. This gap underscores the urgent need for research and development efforts aimed at advancing AAD technologies to effectively mitigate the harmful impacts of audio-based abuse in online environments. \par&#10;&#10;There has been growing interest in building effective AAD systems, for example, Gupta et al. \cite{9746718} explored various representations from VGG trained on AudioSet and multilingual wav2vec2 models with Fully Connected Network (FCN), GRU, and LSTM as classifiers. Further, Sharon et al. \cite{sharon22_interspeech} showed that AAD performance can be enchanced by fusing audio and ASR transcribed textual representations. On the other hand, Thakran et al. \cite{thakran23_interspeech} showed that choosing the right acoustic cues can lead to state-of-the-art (SOTA) performance in AAD without the need of textual representations also reverberated by Spiesberger et al. \cite{spiesberger23_interspeech}. Thakran et al. explored representations from multilingual as well as emotion recognition PTMs. \par&#10;&#10;AAD has benefitted significantly from the availability of different PTM representations, particularly those adept at capturing paralinguistic cues effectively. However, representations from paralingual PTM \cite{shor2022universal} which has shown SOTA performance in related tasks to AAD such as speech emotion recognition (SER), speaker identification (SI), and so on haven't been explored or given less attention for AAD. Our hypothesis suggests that these representations will better capture verbal cues like pitch, tone, and intensity compared to representations of other PTMs, resulting in greater effectiveness for AAD. To validate our hypothesis, we present a comprehensive comparative study of five PTM (TRILLsson,Whisper, MMS, Unispeech-SAT, x-vector) representations including paralingual, multilingual, monolingual as well as speaker recognition PTM which are SOTA in respective tasks and highly potential candidates for improved AAD. \par&#10;&#10;We also conducted an investigative analysis by combining these PTM representations to explore their potential for complementary behavior, akin to observations in other tasks such as speech recognition \cite{arunkumar22b_interspeech}. Our investigation continued as we sought to address the challenge of cross-lingual generalizability as real-world digital environments are linguistically diverse and AAC can appear in different languages. This is a difficult task as some languages require training on that language for better&#10;&#10;AAD performance and models trained in other languages will perform poorly on them also pointed out by Spiesberger et al \cite{spiesberger23_interspeech}. This gives rise to a scenario where individual models may be required for different languages, which poses a significant challenge in terms of scalability, maintenance, and resource allocation. Maintaining separate models for each language not only increases the complexity of the AAD system but also demands additional computational resources.Moreover, it can lead to inconsistencies in performance across languages and hinder the seamless integration of AAD systems into multilingual environments. One way to solve this problem is to train model on the combination of different languages, however, this is not exactly a feasible solution, because training data for different languages may not be available at the same time and with the increase in languages, the computational requirement for training them together will increase. To mitigate this, we propose, CoLLaborative Modeling Across Languages (CoLLM), a novel framework, that doesn't require training at all and enables for the seamless integration of models trained in various languages through the process of weight-averaging, resulting in a single unified model that demonstrates competitive performance in AAD across all languages while comprising the same number of parameters as that of the individual languages models. The proposed framework remains effective even in scenarios where training data for certain languages may not be available at certain times. Notably, CoLLM offers the added benefit of operating in Plug-in mode, facilitating the seamless integration of models trained in various languages alongside those trained on ADIMA. This approach ensures consistent performance across upcoming new languages as well as previously seen languages.&#10;&#10;To summarize, our main contributions are three folds:&#10;&#10;\begin{itemize}&#10; \item A comprehensive empirical study to investigate the performance of paralingual PTM representations for AAD.&#10; \item An in-depth analysis by combining these PTM representations to examine their complementary behavior for improved AAD performance.&#10; \item CoLLM, a novel unified framework for achieving competitive performance across different languages for AAD. &#10;\end{itemize}&#10;&#10;For reproducibility of our experiments and as a reference for future studies to build upon CoLLM for collaborative AAD, we will open source the codes and models built as part of our study after the double-blind review. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Audio Abuse Detection, Linguistics, Pre-Trained Model Representations, Artificial Intelligence, Multilingual Framework Development" />
        </attvalues>
      </node>
      <node id="2204.02263" label="2204.02263">
        <attvalues>
          <attvalue for="0" value="Multilingual and Multimodal Abuse Detection" />
          <attvalue for="1" value="  The presence of abusive content on social media platforms is undesirable as&#10;it severely impedes healthy and safe social media interactions. While automatic&#10;abuse detection has been widely explored in textual domain, audio abuse&#10;detection still remains unexplored. In this paper, we attempt abuse detection&#10;in conversational audio from a multimodal perspective in a multilingual social&#10;media setting. Our key hypothesis is that along with the modelling of audio,&#10;incorporating discriminative information from other modalities can be highly&#10;beneficial for this task. Our proposed method, MADA, explicitly focuses on two&#10;modalities other than the audio itself, namely, the underlying emotions&#10;expressed in the abusive audio and the semantic information encapsulated in the&#10;corresponding textual form. Observations prove that MADA demonstrates gains&#10;over audio-only approaches on the ADIMA dataset. We test the proposed approach&#10;on 10 different languages and observe consistent gains in the range 0.6%-5.2%&#10;by leveraging multiple modalities. We also perform extensive ablation&#10;experiments for studying the contributions of every modality and observe the&#10;best results while leveraging all the modalities together. Additionally, we&#10;perform experiments to empirically confirm that there is a strong correlation&#10;between underlying emotions and abusive behaviour.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.20808" label="2407.20808">
        <attvalues>
          <attvalue for="0" value="Abusive Speech Detection in Indic Languages Using Acoustic Features" />
          <attvalue for="1" value="  Abusive content in online social networks is a well-known problem that can&#10;cause serious psychological harm and incite hatred. The ability to upload audio&#10;data increases the importance of developing methods to detect abusive content&#10;in speech recordings. However, simply transferring the mechanisms from written&#10;abuse detection would ignore relevant information such as emotion and tone. In&#10;addition, many current algorithms require training in the specific language for&#10;which they are being used. This paper proposes to use acoustic and prosodic&#10;features to classify abusive content. We used the ADIMA data set, which&#10;contains recordings from ten Indic languages, and trained different models in&#10;multilingual and cross-lingual settings. Our results show that it is possible&#10;to classify abusive and non-abusive content using only acoustic and prosodic&#10;features. The most important and influential features are discussed.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.04621" label="2110.04621">
        <attvalues>
          <attvalue for="0" value="Universal Paralinguistic Speech Representations Using Self-Supervised&#10;  Conformers" />
          <attvalue for="1" value="  Many speech applications require understanding aspects beyond the words being&#10;spoken, such as recognizing emotion, detecting whether the speaker is wearing a&#10;mask, or distinguishing real from synthetic speech. In this work, we introduce&#10;a new state-of-the-art paralinguistic representation derived from large-scale,&#10;fully self-supervised training of a 600M+ parameter Conformer-based&#10;architecture. We benchmark on a diverse set of speech tasks and demonstrate&#10;that simple linear classifiers trained on top of our time-averaged&#10;representation outperform nearly all previous results, in some cases by large&#10;margins. Our analyses of context-window size demonstrate that, surprisingly, 2&#10;second context-windows achieve 96\% the performance of the Conformers that use&#10;the full long-term context on 7 out of 9 tasks. Furthermore, while the best&#10;per-task representations are extracted internally in the network, stable&#10;performance across several layers allows a single universal representation to&#10;reach near optimal performance on all tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.05518" label="2206.05518">
        <attvalues>
          <attvalue for="0" value="Investigation of Ensemble features of Self-Supervised Pretrained Models&#10;  for Automatic Speech Recognition" />
          <attvalue for="1" value="  Self-supervised learning (SSL) based models have been shown to generate&#10;powerful representations that can be used to improve the performance of&#10;downstream speech tasks. Several state-of-the-art SSL models are available, and&#10;each of these models optimizes a different loss which gives rise to the&#10;possibility of their features being complementary. This paper proposes using an&#10;ensemble of such SSL representations and models, which exploits the&#10;complementary nature of the features extracted by the various pretrained&#10;models. We hypothesize that this results in a richer feature representation and&#10;shows results for the ASR downstream task. To this end, we use three SSL models&#10;that have shown excellent results on ASR tasks, namely HuBERT, Wav2vec2.0, and&#10;WaveLM. We explore the ensemble of models fine-tuned for the ASR task and the&#10;ensemble of features using the embeddings obtained from the pre-trained models&#10;for a downstream ASR task. We get improved performance over individual models&#10;and pre-trained features using Librispeech(100h) and WSJ dataset for the&#10;downstream tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.07882" label="2010.07882">
        <attvalues>
          <attvalue for="0" value="Understanding Neural Abstractive Summarization Models via Uncertainty" />
          <attvalue for="1" value="  An advantage of seq2seq abstractive summarization models is that they&#10;generate text in a free-form manner, but this flexibility makes it difficult to&#10;interpret model behavior. In this work, we analyze summarization decoders in&#10;both blackbox and whitebox ways by studying on the entropy, or uncertainty, of&#10;the model's token-level predictions. For two strong pre-trained models, PEGASUS&#10;and BART on two summarization datasets, we find a strong correlation between&#10;low prediction entropy and where the model copies tokens rather than generating&#10;novel text. The decoder's uncertainty also connects to factors like sentence&#10;position and syntactic distance between adjacent pairs of tokens, giving a&#10;sense of what factors make a context particularly selective for the model's&#10;next output token. Finally, we study the relationship of decoder uncertainty&#10;and attention behavior to understand how attention gives rise to these observed&#10;effects in the model. We show that uncertainty is a useful perspective for&#10;analyzing summarization and text generation models more broadly.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recent progress in abstractive summarization has been fueled by the advent of large-scale Transformers pre-trained on autoregressive language modeling objectives \cite{hoang-etal-2019,khandelwal-etal-2019,lewis-2019-bart,pegasus}. Despite their strong performance on automatic metrics like ROUGE \cite{lin-2004-rouge}, abstractive models are not as straightforward and interpretable as their extractive counterparts. Free-form generation in these models also leads to serious downstream errors, such as factual inconsistencies with the input document \cite{CaoEtAl2018fact,kryscinski-etal-2019-factuality,wang-etal-2020-qags,durmus-etal-2020-feqa,goyal-durrett-2020-factuality}. Although the interpretability of NLU models has been extensively studied \cite{ribeiro2016should,ghaeini-etal-2018-interpreting,jain-wallace-2019-attention,desai2020calibration}, summarization models specifically have not received similar attention, with analysis efforts often focused on datasets and evaluation \cite{kryscinski-etal-2019-neural}.&#10;&#10;In this work, we focus on interpreting and understanding abstractive summarization models through the lens of decoder uncertainty, or the entropy of decisions during generation. While uncertainty in generation has been studied from the perspective of data \cite{ott2018analyzing}, sampling \cite{fan-etal-2018-hierarchical,holtzman2019curious}, and training \cite{correia2019adaptively,kang2020improved}, it is underutilized as a technique for analysis and inspection of generation systems. We study two prominent summarization models, PEGASUS \cite{pegasus} and BART \cite{lewis-2019-bart}, fine-tuned on two English summarization datasets, CNN/Daily Mail \cite{hermann-2015-cnndm} and XSum \cite{narayan-2018-xsum}, to understand model behavior in each setting.&#10;&#10;First, by comparing $n$-grams between the input document and generated summaries, we establish two coarse types for decoded tokens, copy and generate \cite{see-2017-ptrgen}. We find that the entropy of the generation decision correlates with whether the model is copying or generating, as well as where in the sentence the token is. This paints a picture of certain contexts being more restrictive from the standpoint of generation, particularly early in sentences where a model has not ``decided'' what to copy yet, and illustrates the interaction of content selection and lexical choice. Second, we extend this analysis by looking at how uncertainty relates to the syntax of the generated sentence: whether uncertainty connects to syntactic notions of surprisal \cite{roark-etal-2009-deriving} and how the entropy varies across certain syntactic productions.&#10;Finally, we derive a way to quantify decoder attention by aggregating distinct self-attention heads, revealing the correlation between the attention entropy and prediction entropy, and investigating the correspondence between the prediction entropy and the fraction of the past and future decoded tokens.&#10;&#10;Taking this analysis together, we find that the abstractiveness of reference summaries fundamentally changes model behavior: the extractive nature of CNN/DM makes most of its decisions low entropy and copy-oriented while the model maintains higher uncertainty on XSum, yielding more abstractive summaries. More broadly, we show that uncertainty is a simple but effective tool to characterize decoder behavior in text generation. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Artificial Intelligence, Mathematics, Summarization Decoder Behavior, Model Interpretability Analysis, Text Generation Models" />
        </attvalues>
      </node>
      <node id="1906.00138" label="1906.00138">
        <attvalues>
          <attvalue for="0" value="Efficient Adaptation of Pretrained Transformers for Abstractive&#10;  Summarization" />
          <attvalue for="1" value="  Large-scale learning of transformer language models has yielded improvements&#10;on a variety of natural language understanding tasks. Whether they can be&#10;effectively adapted for summarization, however, has been less explored, as the&#10;learned representations are less seamlessly integrated into existing neural&#10;text production architectures. In this work, we propose two solutions for&#10;efficiently adapting pretrained transformer language models as text&#10;summarizers: source embeddings and domain-adaptive training. We test these&#10;solutions on three abstractive summarization datasets, achieving new state of&#10;the art performance on two of them. Finally, we show that these improvements&#10;are achieved by producing more focused summaries with fewer superfluous and&#10;that performance improvements are more pronounced on more abstractive datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.08836" label="1905.08836">
        <attvalues>
          <attvalue for="0" value="Sample Efficient Text Summarization Using a Single Pre-Trained&#10;  Transformer" />
          <attvalue for="1" value="  Language model (LM) pre-training has resulted in impressive performance and&#10;sample efficiency on a variety of language understanding tasks. However, it&#10;remains unclear how to best use pre-trained LMs for generation tasks such as&#10;abstractive summarization, particularly to enhance sample efficiency. In these&#10;sequence-to-sequence settings, prior work has experimented with loading&#10;pre-trained weights into the encoder and/or decoder networks, but used&#10;non-pre-trained encoder-decoder attention weights. We instead use a pre-trained&#10;decoder-only network, where the same Transformer LM both encodes the source and&#10;generates the summary. This ensures that all parameters in the network,&#10;including those governing attention over source states, have been pre-trained&#10;before the fine-tuning step. Experiments on the CNN/Daily Mail dataset show&#10;that our pre-trained Transformer LM substantially improves over pre-trained&#10;Transformer encoder-decoder networks in limited-data settings. For instance, it&#10;achieves 13.1 ROUGE-2 using only 1% of the training data (~3000 examples),&#10;while pre-trained encoder-decoder models score 2.3 ROUGE-2.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2010.05478" label="2010.05478">
        <attvalues>
          <attvalue for="0" value="Evaluating Factuality in Generation with Dependency-level Entailment" />
          <attvalue for="1" value="  Despite significant progress in text generation models, a serious limitation&#10;is their tendency to produce text that is factually inconsistent with&#10;information in the input. Recent work has studied whether textual entailment&#10;systems can be used to identify factual errors; however, these sentence-level&#10;entailment models are trained to solve a different problem than generation&#10;filtering and they do not localize which part of a generation is non-factual.&#10;In this paper, we propose a new formulation of entailment that decomposes it at&#10;the level of dependency arcs. Rather than focusing on aggregate decisions, we&#10;instead ask whether the semantic relationship manifested by individual&#10;dependency arcs in the generated output is supported by the input. Human&#10;judgments on this task are difficult to obtain; we therefore propose a method&#10;to automatically create data based on existing entailment or paraphrase&#10;corpora. Experiments show that our dependency arc entailment model trained on&#10;this data can identify factual inconsistencies in paraphrasing and&#10;summarization better than sentence-level methods or those based on question&#10;generation, while additionally localizing the erroneous parts of the&#10;generation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1602.04938" label="1602.04938">
        <attvalues>
          <attvalue for="0" value="&quot;Why Should I Trust You?&quot;: Explaining the Predictions of Any Classifier" />
          <attvalue for="1" value="  Despite widespread adoption, machine learning models remain mostly black&#10;boxes. Understanding the reasons behind predictions is, however, quite&#10;important in assessing trust, which is fundamental if one plans to take action&#10;based on a prediction, or when choosing whether to deploy a new model. Such&#10;understanding also provides insights into the model, which can be used to&#10;transform an untrustworthy model or prediction into a trustworthy one. In this&#10;work, we propose LIME, a novel explanation technique that explains the&#10;predictions of any classifier in an interpretable and faithful manner, by&#10;learning an interpretable model locally around the prediction. We also propose&#10;a method to explain models by presenting representative individual predictions&#10;and their explanations in a non-redundant way, framing the task as a submodular&#10;optimization problem. We demonstrate the flexibility of these methods by&#10;explaining different models for text (e.g. random forests) and image&#10;classification (e.g. neural networks). We show the utility of explanations via&#10;novel experiments, both simulated and with human subjects, on various scenarios&#10;that require trust: deciding if one should trust a prediction, choosing between&#10;models, improving an untrustworthy classifier, and identifying why a classifier&#10;should not be trusted.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.03894" label="1808.03894">
        <attvalues>
          <attvalue for="0" value="Interpreting Recurrent and Attention-Based Neural Models: a Case Study&#10;  on Natural Language Inference" />
          <attvalue for="1" value="  Deep learning models have achieved remarkable success in natural language&#10;inference (NLI) tasks. While these models are widely explored, they are hard to&#10;interpret and it is often unclear how and why they actually work. In this&#10;paper, we take a step toward explaining such deep learning based models through&#10;a case study on a popular neural model for NLI. In particular, we propose to&#10;interpret the intermediate layers of NLI models by visualizing the saliency of&#10;attention and LSTM gating signals. We present several examples for which our&#10;methods are able to reveal interesting insights and identify the critical&#10;information contributing to the model decisions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1803.00047" label="1803.00047">
        <attvalues>
          <attvalue for="0" value="Analyzing Uncertainty in Neural Machine Translation" />
          <attvalue for="1" value="  Machine translation is a popular test bed for research in neural&#10;sequence-to-sequence models but despite much recent research, there is still a&#10;lack of understanding of these models. Practitioners report performance&#10;degradation with large beams, the under-estimation of rare words and a lack of&#10;diversity in the final translations. Our study relates some of these issues to&#10;the inherent uncertainty of the task, due to the existence of multiple valid&#10;translations for a single source sentence, and to the extrinsic uncertainty&#10;caused by noisy training data. We propose tools and metrics to assess how&#10;uncertainty in the data is captured by the model distribution and how it&#10;affects search strategies that generate translations. Our results show that&#10;search works remarkably well but that models tend to spread too much&#10;probability mass over the hypothesis space. Next, we propose tools to assess&#10;model calibration and show how to easily fix some shortcomings of current&#10;models. As part of this study, we release multiple human reference translations&#10;for two popular benchmarks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.00015" label="1909.00015">
        <attvalues>
          <attvalue for="0" value="Adaptively Sparse Transformers" />
          <attvalue for="1" value="  Attention mechanisms have become ubiquitous in NLP. Recent architectures,&#10;notably the Transformer, learn powerful context-aware word representations&#10;through layered, multi-headed attention. The multiple heads learn diverse types&#10;of word relationships. However, with standard softmax attention, all attention&#10;heads are dense, assigning a non-zero weight to all context words. In this&#10;work, we introduce the adaptively sparse Transformer, wherein attention heads&#10;have flexible, context-dependent sparsity patterns. This sparsity is&#10;accomplished by replacing softmax with $\alpha$-entmax: a differentiable&#10;generalization of softmax that allows low-scoring words to receive precisely&#10;zero weight. Moreover, we derive a method to automatically learn the $\alpha$&#10;parameter -- which controls the shape and sparsity of $\alpha$-entmax --&#10;allowing attention heads to choose between focused or spread-out behavior. Our&#10;adaptively sparse Transformer improves interpretability and head diversity when&#10;compared to softmax Transformers on machine translation datasets. Findings of&#10;the quantitative and qualitative analysis of our approach include that heads in&#10;different layers learn different sparsity preferences and tend to be more&#10;diverse in their attention distributions than softmax Transformers.&#10;Furthermore, at no cost in accuracy, sparsity in attention heads helps to&#10;uncover different head specializations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2212.05093" label="2212.05093">
        <attvalues>
          <attvalue for="0" value="Plug-and-Play Recipe Generation with Content Planning" />
          <attvalue for="1" value="  Recent pre-trained language models have shown promising capabilities in&#10;generating fluent and realistic natural language text. However, generating&#10;multi-sentence text with global content planning has been a long-existing&#10;research question. Current approaches for controlled text generation can hardly&#10;address this issue, as they usually condition on single known control&#10;attributes. In this study, we propose a low-cost yet effective framework which&#10;explicitly models the global content plan of the generated text. Specifically,&#10;it optimizes the joint distribution of the natural language sequence and the&#10;global content plan in a plug-and-play manner. We conduct extensive experiments&#10;on the well-established Recipe1M+ benchmark. Both automatic and human&#10;evaluations verify that our model achieves the state-of-the-art performance on&#10;the task of recipe generation&#10;" />
          <attvalue for="2" value="&#10;Recent progress in large-scale language model pre-training has facilitated significant improvement in generating increasingly realistic natural language text. Although this has been achieved on the surface-level fluency, it has been pointed that generating multi-sentence text with global constraints, or long-term planning is still far from being solved. Typical examples of such task include story continuation with logical coherency~\cite{nye2021improving,sinha2019clutrr}, and recipe generation with step-by-step planning~\cite{marin2019recipe1m+}. &#10;&#10;As suggested by~\cite{lecun2022path}, the aforementioned issues cannot be ameliorated by simply increasing the size of model parameters or the scale of pre-training data. &#10;Adding to this, current approaches for controlled text generation cannot directly tackle those issues either. &#10;For example, CTRL~\cite{keskar2019ctrl}, which trains a class-conditional language model, and PPLM~\cite{dathathri2019plug}, which re-ranks the language model predictions by an attribute model.&#10;They usually share a common setup of optimizing conditional distributions $P(\boldsymbol{y}|a)$, where $\boldsymbol{y}$ is the text sequence and $a$ is the desired single control attribute. &#10;Examples of control attribute include sentiment \cite{ghosh2017affect}, topic \cite{tang2019topic} and formality \cite{wang2019topic}. &#10;However, content planning requires controlling with consideration of global context, which is more sophisticated than the single control attributes. &#10;Therefore, we identify the research gap for the current controlled text generation models to generate multi-sentence text with long-term content planning.&#10;&#10;Motivated by previous research in cognitive science~\cite{evans2003two}, \cite{nye2021improving} pointed out that the reasoning of a neural-based model should consist of two systems, i.e. the system 1 makes intuitive and associative responses, and the system 2 makes deliberative and logical decisions. With greatly increased capabilities, large language models have become sufficiently competent to act as the system 1. However, we argue that, to address the aforementioned research gap, it is vital to empower the language models with the ability to make logical decisions, i.e. predict content plans. &#10;&#10;In contrast to the existing methods that optimize the conditional distributions,&#10;we propose a novel framework which explicitly models the content plan $\boldsymbol{c}$ and optimizes the joint distribution $P(\boldsymbol{y}, \boldsymbol{c})$ in a plug-and-play manner. Figure \ref{graph:overview} depicts an overview of our approach. Specifically, our proposed framework consists of (i) a content planner which predicts the global content plan of the output text; and (ii) a sequence generator, based on pre-trained language models, that generates the output following the content plan. &#10;The predicted content plan steers the generation of the sequence generator through a lightweight and plug-and-play style plan classifier.&#10;It worth emphasizing that the sequence generator does not need to be trained with plan-specific data, which means adapting our framework to other Natural Language Generation (NLG) tasks is cheap and efficient.&#10;&#10;We comprehensively evaluate our approach on the recipe generation task using the widely-used Recipe1M+ benchmark~\cite{marin2019recipe1m+}.&#10;The experimental results demonstrate that our approach significantly outperforms previous state-of-the-art (SOTA) as judged by both automatic and human evaluations. &#10;In particular, the results show that the recipes generated by our model are more accurate and highly controllable. &#10;&#10;In summary, we conclude our contributions as two-fold: Firstly, we identify the current research gap and propose a novel framework that generates text with content plans in a plug-and-play manner.&#10;Secondly, we conduct extensive experiments to show that our framework achieves SOTA performance on the widely-used Recipe1M+ benchmark.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Model Generation, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Text Content Planning" />
        </attvalues>
      </node>
      <node id="2107.02794" label="2107.02794">
        <attvalues>
          <attvalue for="0" value="Improving Coherence and Consistency in Neural Sequence Models with&#10;  Dual-System, Neuro-Symbolic Reasoning" />
          <attvalue for="1" value="  Human reasoning can often be understood as an interplay between two systems:&#10;the intuitive and associative (&quot;System 1&quot;) and the deliberative and logical&#10;(&quot;System 2&quot;). Neural sequence models -- which have been increasingly successful&#10;at performing complex, structured tasks -- exhibit the advantages and failure&#10;modes of System 1: they are fast and learn patterns from data, but are often&#10;inconsistent and incoherent. In this work, we seek a lightweight, training-free&#10;means of improving existing System 1-like sequence models by adding System&#10;2-inspired logical reasoning. We explore several variations on this theme in&#10;which candidate generations from a neural sequence model are examined for&#10;logical consistency by a symbolic reasoning module, which can either accept or&#10;reject the generations. Our approach uses neural inference to mediate between&#10;the neural System 1 and the logical System 2. Results in robust story&#10;generation and grounded instruction-following show that this approach can&#10;increase the coherence and accuracy of neurally-based generations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.06177" label="1908.06177">
        <attvalues>
          <attvalue for="0" value="CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text" />
          <attvalue for="1" value="  The recent success of natural language understanding (NLU) systems has been&#10;troubled by results highlighting the failure of these models to generalize in a&#10;systematic and robust way. In this work, we introduce a diagnostic benchmark&#10;suite, named CLUTRR, to clarify some key issues related to the robustness and&#10;systematicity of NLU systems. Motivated by classic work on inductive logic&#10;programming, CLUTRR requires that an NLU system infer kinship relations between&#10;characters in short stories. Successful performance on this task requires both&#10;extracting relationships between entities, as well as inferring the logical&#10;rules governing these relationships. CLUTRR allows us to precisely measure a&#10;model's ability for systematic generalization by evaluating on held-out&#10;combinations of logical rules, and it allows us to evaluate a model's&#10;robustness by adding curated noise facts. Our empirical results highlight a&#10;substantial performance gap between state-of-the-art NLU models (e.g., BERT and&#10;MAC) and a graph neural network model that works directly with symbolic&#10;inputs---with the graph-based model exhibiting both stronger generalization and&#10;greater robustness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.06851" label="1704.06851">
        <attvalues>
          <attvalue for="0" value="Affect-LM: A Neural Language Model for Customizable Affective Text&#10;  Generation" />
          <attvalue for="1" value="  Human verbal communication includes affective messages which are conveyed&#10;through use of emotionally colored words. There has been a lot of research in&#10;this direction but the problem of integrating state-of-the-art neural language&#10;models with affective information remains an area ripe for exploration. In this&#10;paper, we propose an extension to an LSTM (Long Short-Term Memory) language&#10;model for generating conversational text, conditioned on affect categories. Our&#10;proposed model, Affect-LM enables us to customize the degree of emotional&#10;content in generated sentences through an additional design parameter.&#10;Perception studies conducted using Amazon Mechanical Turk show that Affect-LM&#10;generates naturally looking emotional sentences without sacrificing grammatical&#10;correctness. Affect-LM also learns affect-discriminative word representations,&#10;and perplexity experiments show that additional affective information in&#10;conversational text can improve language model prediction.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.06803" label="2310.06803">
        <attvalues>
          <attvalue for="0" value="Advancing Transformer's Capabilities in Commonsense Reasoning" />
          <attvalue for="1" value="  Recent advances in general purpose pre-trained language models have shown&#10;great potential in commonsense reasoning. However, current works still perform&#10;poorly on standard commonsense reasoning benchmarks including the Com2Sense&#10;Dataset. We argue that this is due to a disconnect with current cutting-edge&#10;machine learning methods. In this work, we aim to bridge the gap by introducing&#10;current ML-based methods to improve general purpose pre-trained language models&#10;in the task of commonsense reasoning. Specifically, we experiment with and&#10;systematically evaluate methods including knowledge transfer, model ensemble,&#10;and introducing an additional pairwise contrastive objective. Our best model&#10;outperforms the strongest previous works by ~15\% absolute gains in Pairwise&#10;Accuracy and ~8.7\% absolute gains in Standard Accuracy.&#10;" />
          <attvalue for="2" value="&#10;&#10;Endowing NLP models with human-like commonsense knowledge has remained a challenge for decades \cite{sap-etal-2020-commonsense}. In 2021, researchers proposed Com2Sense~\cite{singh-etal-2021-com2sense}, a reliable and comprehensive commonsense reasoning benchmark with strict pairwise accuracy metrics. It consists of natural language sentence pairs labeled True/False based on whether they adhere to intuitive commonsense knowledge (Fig.\ref{Fig: Qualitative}). The central evaluation criteria: Pairwise Accuracy, required that the model predict correctly for both sequences to count as successful. &#10;&#10;Initial works on the dataset revealed that neither general purpose language models \cite{devlin-etal-2019-bert}, \cite{DBLP:journals/corr/abs-1907-11692}, \cite{Raffel2019ExploringTL}, etc. nor dedicated commonsense understanding models \cite{Khashabi2020UnifiedQACF}, \cite{2020unifiedqa} performed well on the dataset. All current models suffer from significant performance drops from Standard Accuracy to Pairwise Accuracy, displaying a huge discrepancy from human-like behaviour.&#10;&#10;In this work, we examine possible methods of improving the performance of general purpose language models on the task of commonsense reasoning. Specifically, we study the effects of:&#10;\begin{itemize}&#10; \item Knowledge Transfer from relevant datasets containing commonsense knowledge, including the SemEval-2020 Dataset and the SQuAD2 QA Dataset~\cite{Rajpurkar2018KnowWY}&#10; \item Introducing a pairwise contrastive loss objective that forces models to distinguish commonsensical and non-commensensical statements&#10; \item Ensemble general purpose language models of different backbone architectures to study and compare their effects on overall performance.&#10;&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Language Models, Machine Learning Methods, Artificial Intelligence, Natural Language Processing, Commonsense Reasoning" />
        </attvalues>
      </node>
      <node id="2106.00969" label="2106.00969">
        <attvalues>
          <attvalue for="0" value="COM2SENSE: A Commonsense Reasoning Benchmark with Complementary&#10;  Sentences" />
          <attvalue for="1" value="  Commonsense reasoning is intuitive for humans but has been a long-term&#10;challenge for artificial intelligence (AI). Recent advancements in pretrained&#10;language models have shown promising results on several commonsense benchmark&#10;datasets. However, the reliability and comprehensiveness of these benchmarks&#10;towards assessing model's commonsense reasoning ability remains unclear. To&#10;this end, we introduce a new commonsense reasoning benchmark dataset comprising&#10;natural language true/false statements, with each sample paired with its&#10;complementary counterpart, resulting in 4k sentence pairs. We propose a&#10;pairwise accuracy metric to reliably measure an agent's ability to perform&#10;commonsense reasoning over a given situation. The dataset is crowdsourced and&#10;enhanced with an adversarial model-in-the-loop setup to incentivize challenging&#10;samples. To facilitate a systematic analysis of commonsense capabilities, we&#10;design our dataset along the dimensions of knowledge domains, reasoning&#10;scenarios and numeracy. Experimental results demonstrate that our strongest&#10;baseline (UnifiedQA-3B), after fine-tuning, achieves ~71% standard accuracy and&#10;~51% pairwise accuracy, well below human performance (~95% for both metrics).&#10;The dataset is available at https://github.com/PlusLabNLP/Com2Sense.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2302.13273" label="2302.13273">
        <attvalues>
          <attvalue for="0" value="Two-Stream Joint-Training for Speaker Independent&#10;  Acoustic-to-Articulatory Inversion" />
          <attvalue for="1" value="  Acoustic-to-articulatory inversion (AAI) aims to estimate the parameters of&#10;articulators from speech audio. There are two common challenges in AAI, which&#10;are the limited data and the unsatisfactory performance in speaker independent&#10;scenario. Most current works focus on extracting features directly from speech&#10;and ignoring the importance of phoneme information which may limit the&#10;performance of AAI. To this end, we propose a novel network called SPN that&#10;uses two different streams to carry out the AAI task. Firstly, to improve the&#10;performance of speaker-independent experiment, we propose a new phoneme stream&#10;network to estimate the articulatory parameters as the phoneme features. To the&#10;best of our knowledge, this is the first work that extracts the&#10;speaker-independent features from phonemes to improve the performance of AAI.&#10;Secondly, in order to better represent the speech information, we train a&#10;speech stream network to combine the local features and the global features.&#10;Compared with state-of-the-art (SOTA), the proposed method reduces 0.18mm on&#10;RMSE and increases 6.0% on Pearson correlation coefficient in the&#10;speaker-independent experiment. The code has been released at&#10;https://github.com/liujinyu123/AAINetwork-SPN.&#10;" />
          <attvalue for="2" value="&#10;Acoustic-to-articulatory inversion (AAI) \cite{C1} mainly solves the problem of deriving the pronunciation parameters of key organs from speech audio. In recent years, it has played an important role in many fields, such as pronunciation guidance \cite{C2} and speech recognition \cite{C4,C48,C50}, so it has attracted many researchers to devote themselves to this field.&#10;&#10;Different deep learning based models and acoustic representations have been proposed to carry out the AAI task. In the early stage, codebook \cite{C7} was used for voice inversion, but the performance highly relied on the quality of codebook. Later, The data-driven voice inversion models were presented, such as hidden Markov model (HMM) \cite{C8,C49}, mixed Gaussian network (GMM) \cite{C38}, deep neural networks (DNNs) \cite{C39} and so on. At present, the most commonly used models are the recurrent neural network (RNN) and its variants, such as long-short term memory (LSTM) \cite{C14, C34}. In \cite{C14, C40, C41}, the different speech representations such as line spectral frequencies (LSF), Mel-frequency cepstral coefficients (MFCC) and filter bank energies (FBE) were used. In our work, we take MFCC and phonemes as the input of our model.&#10;&#10;Up to now, there are two main challenges in AAI. One is that the available datasets are very limited, because we need to record the voice and pronunciation parameters at the same time, which is not only difficult to collect, but also expensive. The most commonly used public datasets are MOCHA-TIMIT \cite{C35}, MNGU0 \cite{C21}, and HPRC \cite{C36}. Another challenge is to improve the performance in speaker-independent scenarios.&#10;&#10;For the first challenge, \cite{C44} proposed the method of using the cross corpus data to solve the problem of limited data volume. For the second challenge, \cite{C14} used the vocal tract normalization to map each speaker's pronunciation space to the same space, so as to improve the performance of that model, but it led to the loss of personalized information of speech. It is worth noting that a self-supervised pretraining model was proposed to solve the above challenges and achieved the best performance in \cite{C42}. However, this work only used MFCC as the network input which may limit the performance of AAI and used 1DCNNs to extract speech features which may result in the loss of global information of speech. &#10;&#10;In order to solve the above two challenges, we propose a novel network that consists of two parts, speech stream network and phoneme stream network, which we call SPN in brief. 1DCNNs were used to extract speech features in \cite{C42}. But it was pointed out that CNN only extracts the local features in \cite{C45}, so we add an multi-head attention model to extract the global features to better represent the voice information. In addition, we propose a new phoneme stream network. More precisely, we use transcribed phonemes to perform phoneme inversion, then take the results of phoneme stream network as the phoneme features to perform voice inversion. &#10;The motivation is that phonemes only represent the content information of the speech instead of the identity information. Therefore the phoneme features obtained by phoneme stream network are speaker-independent, which can improve the performance of the speaker-independent experiments.&#10;&#10;In summary, there are three contributions of this work.&#10;\begin{itemize}&#10;\item In order to better represent voice information, we extract the local features and the global features through 1DCNNs and multi-head attention module respectively.&#10;\item We propose a new phoneme stream network to gain the phoneme features to improve the performance of speaker-independent experiment.&#10;\item Based on the experimental results, it is shown that the proposed model outperforms SOTA obviously on public HPRC dataset which decreases by 0.18mm on RMSE and increases by 6\% on PCC.&#10;\end{itemize}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Speech Processing, Linguistics, Engineering, Speech Recognition, Deep Learning Networks, Articulatory Modeling, Phonetics" />
        </attvalues>
      </node>
      <node id="2001.00854" label="2001.00854">
        <attvalues>
          <attvalue for="0" value="Re-synchronization using the Hand Preceding Model for Multi-modal Fusion&#10;  in Automatic Continuous Cued Speech Recognition" />
          <attvalue for="1" value="  Cued Speech (CS) is an augmented lip reading complemented by hand coding, and&#10;it is very helpful to the deaf people. Automatic CS recognition can help&#10;communications between the deaf people and others. Due to the asynchronous&#10;nature of lips and hand movements, fusion of them in automatic CS recognition&#10;is a challenging problem. In this work, we propose a novel re-synchronization&#10;procedure for multi-modal fusion, which aligns the hand features with lips&#10;feature. It is realized by delaying hand position and hand shape with their&#10;optimal hand preceding time which is derived by investigating the temporal&#10;organizations of hand position and hand shape movements in CS. This&#10;re-synchronization procedure is incorporated into a practical continuous CS&#10;recognition system that combines convolutional neural network (CNN) with&#10;multi-stream hidden markov model (MSHMM). A significant improvement of about&#10;4.6\% has been achieved retaining 76.6\% CS phoneme recognition correctness&#10;compared with the state-of-the-art architecture (72.04\%), which did not take&#10;into account the asynchrony of multi-modal fusion in CS. To our knowledge, this&#10;is the first work to tackle the asynchronous multi-modal fusion in the&#10;automatic continuous CS recognition.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.00873" label="2204.00873">
        <attvalues>
          <attvalue for="0" value="Acoustic-to-articulatory Inversion based on Speech Decomposition and&#10;  Auxiliary Feature" />
          <attvalue for="1" value="  Acoustic-to-articulatory inversion (AAI) is to obtain the movement of&#10;articulators from speech signals. Until now, achieving a speaker-independent&#10;AAI remains a challenge given the limited data. Besides, most current works&#10;only use audio speech as input, causing an inevitable performance bottleneck.&#10;To solve these problems, firstly, we pre-train a speech decomposition network&#10;to decompose audio speech into speaker embedding and content embedding as the&#10;new personalized speech features to adapt to the speaker-independent case.&#10;Secondly, to further improve the AAI, we propose a novel auxiliary feature&#10;network to estimate the lip auxiliary features from the above personalized&#10;speech features. Experimental results on three public datasets show that,&#10;compared with the state-of-the-art only using the audio speech feature, the&#10;proposed method reduces the average RMSE by 0.25 and increases the average&#10;correlation coefficient by 2.0% in the speaker-dependent case. More&#10;importantly, the average RMSE decreases by 0.29 and the average correlation&#10;coefficient increases by 5.0% in the speaker-independent case.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12000" label="2305.12000">
        <attvalues>
          <attvalue for="0" value="Deep Learning Approaches to Lexical Simplification: A Survey" />
          <attvalue for="1" value="  Lexical Simplification (LS) is the task of replacing complex for simpler&#10;words in a sentence whilst preserving the sentence's original meaning. LS is&#10;the lexical component of Text Simplification (TS) with the aim of making texts&#10;more accessible to various target populations. A past survey (Paetzold and&#10;Specia, 2017) has provided a detailed overview of LS. Since this survey,&#10;however, the AI/NLP community has been taken by storm by recent advances in&#10;deep learning, particularly with the introduction of large language models&#10;(LLM) and prompt learning. The high performance of these models sparked renewed&#10;interest in LS. To reflect these recent advances, we present a comprehensive&#10;survey of papers published between 2017 and 2023 on LS and its sub-tasks with a&#10;special focus on deep learning. We also present benchmark datasets for the&#10;future development of LS systems.&#10;" />
          <attvalue for="2" value="&#10;&#10;LS improves the readability of any given text with the aim of helping vocabulary and literacy development. LS achieves this by replacing complex words in a sentence with simpler alternatives. LS returns a simplified sentence which can be passed to a TS system for further syntactic and grammatical simplification. The replaced complex words are those words which a general or targeted population found to be hard to read, interpret, or understand. Previous LS systems have been designed to simplify complex words for children, second language learners, individuals with reading disabilities or low-literacy \cite{PaetzoldSpecia2017_surveyLS}. LS therefore provides both developers and users with a degree of personalization that is unattainable through seq2seq or generative TS systems \cite{yeung-lee-2018-personalized,Lee_Yeung2018}. &#10;&#10;Deep learning, and latterly, LLM and prompt learning, have revolutionized the way we approach many NLP tasks, including LS. Previous LS systems have relied upon lexicons, rule-based, statistical, n-gram, and word embedding models to identify and then simplify complex words \cite{PaetzoldSpecia2017_surveyLS}. These approaches would identify a complex word, for example, ``bombardment'' as being in need of simplification and would suggest ``attack'' as a suitable alternative (Figure \ref{pipeline_figure}), hereby referred to as a candidate substitution. &#10;&#10;State-of-the-art deep learning models, such as BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta}, GPT-3 \cite{gpt_3}, and others, automatically generate, select, and rank candidate substitutions with performances superior to traditional approaches. These include relying on pre-existing lexicons, simplification rules, or engineered features \cite{tsar2022}. There have been no surveys published on deep learning approaches for LS. The paper by \cite{PaetzoldSpecia2017_surveyLS} is the most recent survey on LS but it precedes studies that demonstrate the headway made by state-of-the-art deep learning approaches. A broad comprehensive survey on TS was published in 2021\cite{TSsurvey_thanyyan_et_al}. However, this survey likewise does not cover recent advances in the field nor does it focus specifically on LS. This paper therefore continues pre-existing literature by providing an updated survey of the latest deep learning approaches for LS and its sub-tasks of substitute generation (SG), selection (SS), and ranking (SR). &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Deep Learning Advances, Linguistics, Lexical Simplification, Artificial Intelligence, Natural Language Processing, Text Accessibility" />
        </attvalues>
      </node>
      <node id="2302.02888" label="2302.02888">
        <attvalues>
          <attvalue for="0" value="Findings of the TSAR-2022 Shared Task on Multilingual Lexical&#10;  Simplification" />
          <attvalue for="1" value="  We report findings of the TSAR-2022 shared task on multilingual lexical&#10;simplification, organized as part of the Workshop on Text Simplification,&#10;Accessibility, and Readability TSAR-2022 held in conjunction with EMNLP 2022.&#10;The task called the Natural Language Processing research community to&#10;contribute with methods to advance the state of the art in multilingual lexical&#10;simplification for English, Portuguese, and Spanish. A total of 14 teams&#10;submitted the results of their lexical simplification systems for the provided&#10;test data. Results of the shared task indicate new benchmarks in Lexical&#10;Simplification with English lexical simplification quantitative results&#10;noticeably higher than those obtained for Spanish and (Brazilian) Portuguese.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.02550" label="1611.02550">
        <attvalues>
          <attvalue for="0" value="Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based&#10;  Approaches" />
          <attvalue for="1" value="  Acoustic word embeddings --- fixed-dimensional vector representations of&#10;variable-length spoken word segments --- have begun to be considered for tasks&#10;such as speech recognition and query-by-example search. Such embeddings can be&#10;learned discriminatively so that they are similar for speech segments&#10;corresponding to the same word, while being dissimilar for segments&#10;corresponding to different words. Recent work has found that acoustic word&#10;embeddings can outperform dynamic time warping on query-by-example search and&#10;related word discrimination tasks. However, the space of embedding models and&#10;training approaches is still relatively unexplored. In this paper we present&#10;new discriminative embedding models based on recurrent neural networks (RNNs).&#10;We consider training losses that have been successful in prior work, in&#10;particular a cross entropy loss for word classification and a contrastive loss&#10;that explicitly aims to separate same-word and different-word pairs in a&#10;&quot;Siamese network&quot; training setting. We find that both classifier-based and&#10;Siamese RNN embeddings improve over previously reported results on a word&#10;discrimination task, with Siamese RNNs outperforming classification models. In&#10;addition, we present analyses of the learned embeddings and the effects of&#10;variables such as dimensionality and network structure.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.11297" label="1804.11297">
        <attvalues>
          <attvalue for="0" value="Sampling strategies in Siamese Networks for unsupervised speech&#10;  representation learning" />
          <attvalue for="1" value="  Recent studies have investigated siamese network architectures for learning&#10;invariant speech representations using same-different side information at the&#10;word level. Here we investigate systematically an often ignored component of&#10;siamese networks: the sampling procedure (how pairs of same vs. different&#10;tokens are selected). We show that sampling strategies taking into account&#10;Zipf's Law, the distribution of speakers and the proportions of same and&#10;different pairs of words significantly impact the performance of the network.&#10;In particular, we show that word frequency compression improves learning across&#10;a large range of variations in number of training pairs. This effect does not&#10;apply to the same extent to the fully unsupervised setting, where the pairs of&#10;same-different words are obtained by spoken term discovery. We apply these&#10;results to pairs of words discovered using an unsupervised algorithm and show&#10;an improvement on state-of-the-art in unsupervised representation learning&#10;using siamese networks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.02228" label="1808.02228">
        <attvalues>
          <attvalue for="0" value="Segmental Audio Word2Vec: Representing Utterances as Sequences of&#10;  Vectors with Applications in Spoken Term Detection" />
          <attvalue for="1" value="  While Word2Vec represents words (in text) as vectors carrying semantic&#10;information, audio Word2Vec was shown to be able to represent signal segments&#10;of spoken words as vectors carrying phonetic structure information. Audio&#10;Word2Vec can be trained in an unsupervised way from an unlabeled corpus, except&#10;the word boundaries are needed. In this paper, we extend audio Word2Vec from&#10;word-level to utterance-level by proposing a new segmental audio Word2Vec, in&#10;which unsupervised spoken word boundary segmentation and audio Word2Vec are&#10;jointly learned and mutually enhanced, so an utterance can be directly&#10;represented as a sequence of vectors carrying phonetic structure information.&#10;This is achieved by a segmental sequence-to-sequence autoencoder (SSAE), in&#10;which a segmentation gate trained with reinforcement learning is inserted in&#10;the encoder. Experiments on English, Czech, French and German show very good&#10;performance in both unsupervised spoken word segmentation and spoken term&#10;detection applications (significantly better than frame-based DTW).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.15404" label="2012.15404">
        <attvalues>
          <attvalue for="0" value="Unified Mandarin TTS Front-end Based on Distilled BERT Model" />
          <attvalue for="1" value="  The front-end module in a typical Mandarin text-to-speech system (TTS) is&#10;composed of a long pipeline of text processing components, which requires&#10;extensive efforts to build and is prone to large accumulative model size and&#10;cascade errors. In this paper, a pre-trained language model (PLM) based model&#10;is proposed to simultaneously tackle the two most important tasks in TTS&#10;front-end, i.e., prosodic structure prediction (PSP) and grapheme-to-phoneme&#10;(G2P) conversion. We use a pre-trained Chinese BERT[1] as the text encoder and&#10;employ multi-task learning technique to adapt it to the two TTS front-end&#10;tasks. Then, the BERT encoder is distilled into a smaller model by employing a&#10;knowledge distillation technique called TinyBERT[2], making the whole model&#10;size 25% of that of benchmark pipeline models while maintaining competitive&#10;performance on both tasks. With the proposed the methods, we are able to run&#10;the whole TTS front-end module in a light and unified manner, which is more&#10;friendly to deployment on mobile devices.&#10;" />
          <attvalue for="2" value="&#10;&#10;In Mandarin text-to-speech (TTS) synthesis, the front-end module has a strong impact on the intelligibility and naturalness of synthesized speech. As shown in Figure~\ref{fig:frontend}, a typical Mandarin TTS front-end module consists at least two main components, i.e., prosodic structure prediction (PSP) and grapheme-to-phoneme (G2P) conversion. PSP predicts the prosodic boundaries including prosodic word (PW), prosodic phrase (PPH) and intonational phrase (IPH), while G2P converts each Mandarin character into its pronunciation. Most existing TTS systems implement these two components or each processing step of them individually using either rule based models (e.g., syntactic trees based rules for PSP \cite{zhang2016mandarin}, dictionary matching et al based polyphone disambiguity \cite{huang2010disambiguation}), or statistical learning models (like CRF \cite{qian2010automatic} and LSTM/Attention based NN models \cite{pan2019mandarin,lu2019self} for PSP, \cite{cai2019polyphone,shan2016bi} for Mandarin G2P). The front-end module hence becomes a long pipeline of such individual steps as presented in Figure~\ref{fig:frontend}. As a result, it becomes a complicated and laborious work to build and maintain such a front-end, and also the storage and computation of the various front-end models hinder the deployment of TTS systems onto mobile devices. &#10;&#10;The aim of this work is to model PSP and G2P simultaneously in a unified manner and provide all the necessary prosodic and pronunciation information for the TTS backend (e.g., the acoustic model Tacotron~\cite{wang2017tacotron} and Wavenet vocoder \cite{oord2016wavenet}). This idea of simplifying TTS front-end has gained great attention recently. Deep Voice~\cite{arik2017deep} attempts to simplify the front-end pipeline by replacing all the components with deep neural networks. In \cite{yang2019pre}, large fixed pre-trained models are used to extract text representation for both PSP and G2P components, which achieves comparable performance while saving the efforts to elaborately prepare input features for each component individually. Pan et al \cite{pan2019unified} proposed a unified front-end structure that models PSP and G2P as a single sequence-to-sequence neural model in an auto-regressive way. Most of these exsiting works try to compact the front-end steps into a unified module, but still suffer from complex architecture or intensive computation, and can not be trained in a end-to-end fashion. Thus, a more simple and effective front-end model needs to be explored.&#10;&#10;Inspired by the success of pre-trained language model (PLM) in TTS front-end related tasks \cite{zhu2019probing,sun2019knowledge,du2019prosodic,talman2019predicting}, we proposed a novel unified Mandarin TTS front-end model based on Chinese BERT, and further compressed it by utilizing a knowledge distillation technique named TinyBERT\cite{jiao2019tinybert}. The contribution of this work can be summarized into the following aspects:&#10;\begin{itemize}&#10;&#09;\item We proposed a unified front-end model that employs BERT as text encoder and jointly models the PSP and G2P components in a multi-task learning framework\cite{ruder2017overview}. &#10;&#10;\item We employed a two-stage distillation strategy \cite{jiao2019tinybert} to further distill the front-end model into a more compact one, which facilitate deployment on edge devices.&#10;&#10;\item Our approached models achieves state-of-the-art results on both PSP and G2P tasks. The distilled model can improve $0.31\%$ in polyphone disambiguation accuracy, $1.45\%$ and $0.52\%$ in F1 score of PPH and IPH prediction respectively, while having only $25\%$ size of baseline front-end models. &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing Techniques, Linguistics, Text-to-Speech Systems, Artificial Intelligence, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="1702.07825" label="1702.07825">
        <attvalues>
          <attvalue for="0" value="Deep Voice: Real-time Neural Text-to-Speech" />
          <attvalue for="1" value="  We present Deep Voice, a production-quality text-to-speech system constructed&#10;entirely from deep neural networks. Deep Voice lays the groundwork for truly&#10;end-to-end neural speech synthesis. The system comprises five major building&#10;blocks: a segmentation model for locating phoneme boundaries, a&#10;grapheme-to-phoneme conversion model, a phoneme duration prediction model, a&#10;fundamental frequency prediction model, and an audio synthesis model. For the&#10;segmentation model, we propose a novel way of performing phoneme boundary&#10;detection with deep neural networks using connectionist temporal classification&#10;(CTC) loss. For the audio synthesis model, we implement a variant of WaveNet&#10;that requires fewer parameters and trains faster than the original. By using a&#10;neural network for each component, our system is simpler and more flexible than&#10;traditional text-to-speech systems, where each component requires laborious&#10;feature engineering and extensive domain expertise. Finally, we show that&#10;inference with our system can be performed faster than real time and describe&#10;optimized WaveNet inference kernels on both CPU and GPU that achieve up to 400x&#10;speedups over existing implementations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.04111" label="1911.04111">
        <attvalues>
          <attvalue for="0" value="A unified sequence-to-sequence front-end model for Mandarin&#10;  text-to-speech synthesis" />
          <attvalue for="1" value="  In Mandarin text-to-speech (TTS) system, the front-end text processing module&#10;significantly influences the intelligibility and naturalness of synthesized&#10;speech. Building a typical pipeline-based front-end which consists of multiple&#10;individual components requires extensive efforts. In this paper, we proposed a&#10;unified sequence-to-sequence front-end model for Mandarin TTS that converts raw&#10;texts to linguistic features directly. Compared to the pipeline-based&#10;front-end, our unified front-end can achieve comparable performance in&#10;polyphone disambiguation and prosody word prediction, and improve intonation&#10;phrase prediction by 0.0738 in F1 score. We also implemented the unified&#10;front-end with Tacotron and WaveRNN to build a Mandarin TTS system. The&#10;synthesized speech by that got a comparable MOS (4.38) with the pipeline-based&#10;front-end (4.37) and close to human recordings (4.49).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.10915" label="1912.10915">
        <attvalues>
          <attvalue for="0" value="Probing the phonetic and phonological knowledge of tones in Mandarin TTS&#10;  models" />
          <attvalue for="1" value="  This study probes the phonetic and phonological knowledge of lexical tones in&#10;TTS models through two experiments. Controlled stimuli for testing tonal&#10;coarticulation and tone sandhi in Mandarin were fed into Tacotron 2 and&#10;WaveGlow to generate speech samples, which were subject to acoustic analysis&#10;and human evaluation. Results show that both baseline Tacotron 2 and Tacotron 2&#10;with BERT embeddings capture the surface tonal coarticulation patterns well but&#10;fail to consistently apply the Tone-3 sandhi rule to novel sentences.&#10;Incorporating pre-trained BERT embeddings into Tacotron 2 improves the&#10;naturalness and prosody performance, and yields better generalization of Tone-3&#10;sandhi rules to novel complex sentences, although the overall accuracy for&#10;Tone-3 sandhi was still low. Given that TTS models do capture some linguistic&#10;phenomena, it is argued that they can be used to generate and validate certain&#10;linguistic hypotheses. On the other hand, it is also suggested that&#10;linguistically informed stimuli should be included in the training and the&#10;evaluation of TTS models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.02262" label="1908.02262">
        <attvalues>
          <attvalue for="0" value="Predicting Prosodic Prominence from Text with Pre-trained Contextualized&#10;  Word Representations" />
          <attvalue for="1" value="  In this paper we introduce a new natural language processing dataset and&#10;benchmark for predicting prosodic prominence from written text. To our&#10;knowledge this will be the largest publicly available dataset with prosodic&#10;labels. We describe the dataset construction and the resulting benchmark&#10;dataset in detail and train a number of different models ranging from&#10;feature-based classifiers to neural network systems for the prediction of&#10;discretized prosodic prominence. We show that pre-trained contextualized word&#10;representations from BERT outperform the other models even with less than 10%&#10;of the training data. Finally we discuss the dataset in light of the results&#10;and point to future research and plans for further improving both the dataset&#10;and methods of predicting prosodic prominence from text. The dataset and the&#10;code for the models are publicly available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.04386" label="2103.04386">
        <attvalues>
          <attvalue for="0" value="Automatic Difficulty Classification of Arabic Sentences" />
          <attvalue for="1" value="  In this paper, we present a Modern Standard Arabic (MSA) Sentence difficulty&#10;classifier, which predicts the difficulty of sentences for language learners&#10;using either the CEFR proficiency levels or the binary classification as simple&#10;or complex. We compare the use of sentence embeddings of different kinds&#10;(fastText, mBERT , XLM-R and Arabic-BERT), as well as traditional language&#10;features such as POS tags, dependency trees, readability scores and frequency&#10;lists for language learners. Our best results have been achieved using&#10;fined-tuned Arabic-BERT. The accuracy of our 3-way CEFR classification is F-1&#10;of 0.80 and 0.75 for Arabic-Bert and XLM-R classification respectively and 0.71&#10;Spearman correlation for regression. Our binary difficulty classifier reaches&#10;F-1 0.94 and F-1 0.98 for sentence-pair semantic similarity classifier.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the last century, measuring text readability (TR) has been undertaken in education, psychology, and linguistics. There appears to be some agreement that TR is the quality of a given text to be easy to comprehend by its readers in adequate time with reasonable effort \cite{cavalli2018arabic}. Research to date has tended to focus on assigning readability levels to whole text rather than to individual sentences, despite the fact that any text is composed of a number of sentences, which vary in their difficulty \cite{schumacher2016predicting}. Assigning readability levels for a text is a challenging task and it is even more challenging on the sentence level as much less information is available. Also, the sentence difficulty is influenced by many parameters, such as, genre or topics, as well grammatical structures, which need to be combined in a single classifier. Difficulty assessment at the sentence level is a more challenging task in comparison to the better researched text level task, but the availability of a readability sentence classifier for Arabic is vital, since this is a prerequisite for research on automatic text simplification (ATS), i.e. the process of reducing text-linguistic complexity, while maintaining its meaning \cite{saggion2017automatic}.&#10;&#10;We focus here on experiments aimed at measuring to what extent a sentence is understandable by a reader, such as a learner of Arabic as a foreign language, and at exploring different methods for readability assessment. The main aim of this paper lies in developing and testing different sentence representation methodologies, which range from using linguistic knowledge via feature-based machine learning to modern neural methods. &#10;&#10;In summary, the contributions of this paper are:&#10; \begin{enumerate}[noitemsep]&#10; \item We compiled a novel dataset for training on the sentence level; &#10; \item We developed a range of linguistic features, including POS, syntax and frequency information;&#10; \item We evaluated a range of different sentence embedding approaches, such as fastText, BERT and XLM-R, and compared them to the linguistic features;&#10; \item We cast the readability assessment as a regression problem as well as a classification problem;&#10; \item Our model is the first sentence difficulty system available for Arabic.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Language Difficulty Classification, Linguistics, Arabic Language Processing, Sentence Embeddings Analysis, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1606.08425" label="1606.08425">
        <attvalues>
          <attvalue for="0" value="Predicting the Relative Difficulty of Single Sentences With and Without&#10;  Surrounding Context" />
          <attvalue for="1" value="  The problem of accurately predicting relative reading difficulty across a set&#10;of sentences arises in a number of important natural language applications,&#10;such as finding and curating effective usage examples for intelligent language&#10;tutoring systems. Yet while significant research has explored document- and&#10;passage-level reading difficulty, the special challenges involved in assessing&#10;aspects of readability for single sentences have received much less attention,&#10;particularly when considering the role of surrounding passages. We introduce&#10;and evaluate a novel approach for estimating the relative reading difficulty of&#10;a set of sentences, with and without surrounding context. Using different sets&#10;of lexical and grammatical features, we explore models for predicting pairwise&#10;relative difficulty using logistic regression, and examine rankings generated&#10;by aggregating pairwise difficulty labels using a Bayesian rating system to&#10;form a final ranking. We also compare rankings derived for sentences assessed&#10;with and without context, and find that contextual features can help predict&#10;differences in relative difficulty judgments across these two conditions.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2401.07851" label="2401.07851">
        <attvalues>
          <attvalue for="0" value="Unlocking Efficiency in Large Language Model Inference: A Comprehensive&#10;  Survey of Speculative Decoding" />
          <attvalue for="1" value="  To mitigate the high inference latency stemming from autoregressive decoding&#10;in Large Language Models (LLMs), Speculative Decoding has emerged as a novel&#10;decoding paradigm for LLM inference. In each decoding step, this method first&#10;drafts several future tokens efficiently and then verifies them in parallel.&#10;Unlike autoregressive decoding, Speculative Decoding facilitates the&#10;simultaneous decoding of multiple tokens per step, thereby accelerating&#10;inference. This paper presents a comprehensive overview and analysis of this&#10;promising decoding paradigm. We begin by providing a formal definition and&#10;formulation of Speculative Decoding. Then, we organize in-depth discussions on&#10;its key facets, such as drafter selection and verification strategies.&#10;Furthermore, we present a comparative analysis of leading methods under&#10;third-party testing environments. We aim for this work to serve as a catalyst&#10;for further research on Speculative Decoding, ultimately contributing to more&#10;efficient LLM inference.&#10;" />
          <attvalue for="2" value="&#10;Large Language Models (LLMs) have achieved remarkable proficiency in a range of downstream tasks~\cite{openai:2023gpt4, Hugo:2023llama, Hugo:2023llama2, vicuna2023, mistral}. They are progressively evolving as the cornerstone of comprehensive API interfaces (e.g., ChatGPT), offering human life services and guidance through real-time human-machine interactions. However, the inference latency of these sizable models has emerged as a substantial obstacle restricting their broader applications. This latency primarily arises from the token-by-token generation necessitated by autoregressive decoding, resulting in an escalation of the inference latency with both the length of the generated sequence and the model's scale.&#10;&#10;To accelerate LLM inference, an innovative inference paradigm, Speculative Decoding has been introduced~\cite{Stern:2018blockwise, xia:2022specdec, Leviathan:2023specdec, Chen:2023specsampling}. As shown in Figure~\ref{fig:specdec-intro}, in each decoding step, Speculative Decoding first efficiently drafts multiple tokens as speculation of future decoding steps of the target LLM and then utilizes the LLM to verify all drafted tokens in parallel. Only those tokens that meet the LLM's verification criterion are accepted as final outputs to guarantee generation quality. &#10;&#10;Speculative Decoding is founded upon two key observations about LLM inference: 1) many easy tokens can be predicted with less computational overhead (e.g., using a smaller model), and 2) LLM inference is highly memory bandwidth bound~\cite{Patterson:2004latencybandwith, Shazeer:2019memorybandwith} with the main latency bottleneck arising from memory reads/writes of LLM parameters rather than arithmetic computations. Drawing on these observations, Speculative Decoding adapts the concept of speculative execution to focus LLMs' efforts on the validation of pre-drafted tokens, substantially diminishing the need for frequent memory operations of LLM parameters, thereby improving inference efficiency. &#10;&#10;While Speculative Decoding shows promise, it raises several critical questions that warrant further investigation. For instance, how to design an optimal drafter to strike a balance between speculation accuracy and drafting efficiency~\cite{xia:2022specdec, Zhou:2023distillspec, Li:2024eagle}. Additionally, it is essential to assess whether the verification criterion can maintain both generation parallelism and output quality~\cite{Miao:2023specinfer, medusa}. Furthermore, since existing methods are evaluated under disparate testing conditions, a unified benchmark is needed to facilitate realistic speedup expectations within the research community.&#10;&#10;Amid the rapid expansion of research in Speculative Decoding, this work makes the first attempt to present a survey of this field, aiming to raise awareness among academics about the latest advancements. We provide a systematic categorization of current research and an in-depth analysis of relevant studies. Moreover, we introduce Spec-Bench, a comprehensive benchmark to assess Speculative Decoding methods in diverse application scenarios. Our contributions can be summarized as follows:&#10;&#10;\begin{enumerate}[label={(\arabic*)}]&#10; \item First survey: To our knowledge, we are the first to present a comprehensive survey on Speculative Decoding;&#10; \item Formal definition: We furnish a formal definition and formulation of Speculative Decoding, laying the groundwork for future research.&#10; \item New taxonomy: We provide a systematic taxonomy for Speculative Decoding, offering an organized categorization of existing work.&#10; \item Spec-Bench: We introduce Spec-Bench, an extensive benchmark designed for assessing Speculative Decoding, enabling a comparative evaluation of leading methodologies.&#10;\end{enumerate}&#10;&#10;We hope that this work can serve as an essential guide for newcomers and motivate future research.&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Artificial Intelligence Optimization, Artificial Intelligence, Mathematics, Decoding Paradigms, Language Model Inference" />
        </attvalues>
      </node>
      <node id="2203.16487" label="2203.16487">
        <attvalues>
          <attvalue for="0" value="Speculative Decoding: Exploiting Speculative Execution for Accelerating&#10;  Seq2seq Generation" />
          <attvalue for="1" value="  We propose Speculative Decoding (SpecDec), for the first time ever, to&#10;formally study exploiting the idea of speculative execution to accelerate&#10;autoregressive (AR) decoding. Speculative Decoding has two innovations:&#10;Spec-Drafter -- an independent model specially optimized for efficient and&#10;accurate drafting -- and Spec-Verification -- a reliable method for verifying&#10;the drafted tokens efficiently in the decoding paradigm. Experimental results&#10;on various seq2seq tasks including machine translation and abstractive&#10;summarization show our approach can achieve around $5\times$ speedup for the&#10;popular Transformer architectures with comparable generation quality to beam&#10;search decoding, refreshing the impression that the draft-then-verify paradigm&#10;introduces only $1.4\times$$\sim$$2\times$ speedup. In addition to the&#10;remarkable speedup, we also demonstrate 3 additional advantages of SpecDec,&#10;revealing its practical value for accelerating generative models in real-world&#10;applications. Our models and codes are available at&#10;https://github.com/hemingkx/SpecDec.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.06799" label="2103.06799">
        <attvalues>
          <attvalue for="0" value="Towards Continual Learning for Multilingual Machine Translation via&#10;  Vocabulary Substitution" />
          <attvalue for="1" value="  We propose a straightforward vocabulary adaptation scheme to extend the&#10;language capacity of multilingual machine translation models, paving the way&#10;towards efficient continual learning for multilingual machine translation. Our&#10;approach is suitable for large-scale datasets, applies to distant languages&#10;with unseen scripts, incurs only minor degradation on the translation&#10;performance for the original language pairs and provides competitive&#10;performance even in the case where we only possess monolingual data for the new&#10;languages.&#10;" />
          <attvalue for="2" value=" &#10;&#10;The longstanding goal of multilingual machine translation \cite{firat16,johnson16,aharoni19,gu2018universal} has been to develop a universal translation model, capable of providing high-quality translations between any pair of languages. Due to limitations on the data available, however, current approaches rely on first selecting a set of languages for which we have data and training an initial translation model on this data jointly for all languages in a multi-task setup. In an ideal setting, one would continually update the model once data for new language pairs arrives. This setting, dubbed in the literature as continual learning \cite{ring1994continual,rebuffi2017icarl,kirkpatrick2017overcoming,lopez2017gradient}, introduces new challenges not found in the traditional multi-task setup, most famously catastrophic forgetting \cite{mccloskey1989catastrophic}, in which the model may lose its previously-learned knowledge as it learns new language pairs. This situation is further complicated by the training procedures of standard tokenizers, such as Byte-Pair Encoding (BPE) \cite{sennrich2015neural} or Sentencepiece \cite{kudo18}, which necessitate access to monolingual data for all the languages considered before producing the vocabulary. Failing to comply with these requirements, one risks suboptimal segmentation rules which in the worst case could result in strings of entirely &lt;UNK&gt; tokens for text in a previously-unseen alphabet.&#10;&#10;In this work, we investigate how vocabularies derived from BPE transform if they are rebuilt with the same settings but with additional data from a new language. We show in Section \ref{subsec:tok_overlap} that there is a large token overlap between the original and updated vocabularies. This large overlap allows us to retain the performance of a translation model after replacing its vocabulary with the updated vocabulary that additionally supports a new language.&#10;&#10;Past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts \cite{gu2018universal,neubig2018rapid,lakew2019adapting,chronopoulou2020reusing}. These works usually focus solely on learning the new language pair, with no consideration for catastrophic forgetting. Moreover, these works only examine the setting where the new language pair comes with parallel data, despite the reality that for a variety of low-resource languages, we may only possess high-quality monolingual data with no access to parallel data. Finally, unlike our approach, these approaches do not recover the vocabulary one would have built if one had access to the data for the new language from the very beginning.&#10;&#10;Having alleviated the vocabulary issues, we study whether we are able to learn the new language pair rapidly and accurately, matching the performance of a model which had access to this data at the beginning of training. We propose a simple adaptation scheme that allows our translation model to attain competitive performance with strong bilingual and multilingual baselines in a small amount of additional gradient steps. Moreover, our model retains most of the translation quality on the original language pairs it was trained on, exhibiting no signs of catastrophic forgetting.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Translation, Linguistics, Data Science, Continual Learning, Multilingual Models, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="1903.00089" label="1903.00089">
        <attvalues>
          <attvalue for="0" value="Massively Multilingual Neural Machine Translation" />
          <attvalue for="1" value="  Multilingual neural machine translation (NMT) enables training a single model&#10;that supports translation from multiple source languages into multiple target&#10;languages. In this paper, we push the limits of multilingual NMT in terms of&#10;number of languages being used. We perform extensive experiments in training&#10;massively multilingual NMT models, translating up to 102 languages to and from&#10;English within a single model. We explore different setups for training such&#10;models and analyze the trade-offs between translation quality and various&#10;modeling decisions. We report results on the publicly available TED talks&#10;multilingual corpus where we show that massively multilingual many-to-many&#10;models are effective in low resource settings, outperforming the previous&#10;state-of-the-art while supporting up to 59 languages. Our experiments on a&#10;large-scale dataset with 102 languages to and from English and up to one&#10;million examples per direction also show promising results, surpassing strong&#10;bilingual baselines and encouraging future work on massively multilingual NMT.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1802.05368" label="1802.05368">
        <attvalues>
          <attvalue for="0" value="Universal Neural Machine Translation for Extremely Low Resource&#10;  Languages" />
          <attvalue for="1" value="  In this paper, we propose a new universal machine translation approach&#10;focusing on languages with a limited amount of parallel data. Our proposed&#10;approach utilizes a transfer-learning approach to share lexical and sentence&#10;level representations across multiple source languages into one target&#10;language. The lexical part is shared through a Universal Lexical Representation&#10;to support multilingual word-level sharing. The sentence-level sharing is&#10;represented by a model of experts from all source languages that share the&#10;source encoders with all other languages. This enables the low-resource&#10;language to utilize the lexical and sentence representations of the higher&#10;resource languages. Our approach is able to achieve 23 BLEU on Romanian-English&#10;WMT2016 using a tiny parallel corpus of 6k sentences, compared to the 18 BLEU&#10;of strong baseline system which uses multilingual training and&#10;back-translation. Furthermore, we show that the proposed approach can achieve&#10;almost 20 BLEU on the same dataset through fine-tuning a pre-trained&#10;multi-lingual system in a zero-shot setting.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.07725" label="1611.07725">
        <attvalues>
          <attvalue for="0" value="iCaRL: Incremental Classifier and Representation Learning" />
          <attvalue for="1" value="  A major open problem on the road to artificial intelligence is the&#10;development of incrementally learning systems that learn about more and more&#10;concepts over time from a stream of data. In this work, we introduce a new&#10;training strategy, iCaRL, that allows learning in such a class-incremental way:&#10;only the training data for a small number of classes has to be present at the&#10;same time and new classes can be added progressively. iCaRL learns strong&#10;classifiers and a data representation simultaneously. This distinguishes it&#10;from earlier works that were fundamentally limited to fixed data&#10;representations and therefore incompatible with deep learning architectures. We&#10;show by experiments on CIFAR-100 and ImageNet ILSVRC 2012 data that iCaRL can&#10;learn many classes incrementally over a long period of time where other&#10;strategies quickly fail.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1808.04189" label="1808.04189">
        <attvalues>
          <attvalue for="0" value="Rapid Adaptation of Neural Machine Translation to New Languages" />
          <attvalue for="1" value="  This paper examines the problem of adapting neural machine translation&#10;systems to new, low-resourced languages (LRLs) as effectively and rapidly as&#10;possible. We propose methods based on starting with massively multilingual&#10;&quot;seed models&quot;, which can be trained ahead-of-time, and then continuing training&#10;on data related to the LRL. We contrast a number of strategies, leading to a&#10;novel, simple, yet effective method of &quot;similar-language regularization&quot;, where&#10;we jointly train on both a LRL of interest and a similar high-resourced&#10;language to prevent over-fitting to small LRL data. Experiments demonstrate&#10;that massively multilingual models, even without any explicit adaptation, are&#10;surprisingly effective, achieving BLEU scores of up to 15.5 with no data from&#10;the LRL, and that the proposed similar-language regularization method improves&#10;over other adaptation methods by 1.7 BLEU points average over 4 LRL settings.&#10;Code to reproduce experiments at https://github.com/neubig/rapid-adaptation&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.13998" label="1910.13998">
        <attvalues>
          <attvalue for="0" value="Adapting Multilingual Neural Machine Translation to Unseen Languages" />
          <attvalue for="1" value="  Multilingual Neural Machine Translation (MNMT) for low-resource languages&#10;(LRL) can be enhanced by the presence of related high-resource languages (HRL),&#10;but the relatedness of HRL usually relies on predefined linguistic assumptions&#10;about language similarity. Recently, adapting MNMT to a LRL has shown to&#10;greatly improve performance. In this work, we explore the problem of adapting&#10;an MNMT model to an unseen LRL using data selection and model adaptation. In&#10;order to improve NMT for LRL, we employ perplexity to select HRL data that are&#10;most similar to the LRL on the basis of language distance. We extensively&#10;explore data selection in popular multilingual NMT settings, namely in&#10;(zero-shot) translation, and in adaptation from a multilingual pre-trained&#10;model, for both directions (LRL-en). We further show that dynamic adaptation of&#10;the model's vocabulary results in a more favourable segmentation for the LRL in&#10;comparison with direct adaptation. Experiments show reductions in training time&#10;and significant performance gains over LRL baselines, even with zero LRL data&#10;(+13.0 BLEU), up to +17.0 BLEU for pre-trained multilingual model dynamic&#10;adaptation with related data selection. Our method outperforms current&#10;approaches, such as massively multilingual models and data augmentation, on&#10;four LRL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.09209" label="1908.09209">
        <attvalues>
          <attvalue for="0" value="Adversarial Domain Adaptation for Machine Reading Comprehension" />
          <attvalue for="1" value="  In this paper, we focus on unsupervised domain adaptation for Machine Reading&#10;Comprehension (MRC), where the source domain has a large amount of labeled&#10;data, while only unlabeled passages are available in the target domain. To this&#10;end, we propose an Adversarial Domain Adaptation framework (AdaMRC), where&#10;($i$) pseudo questions are first generated for unlabeled passages in the target&#10;domain, and then ($ii$) a domain classifier is incorporated into an MRC model&#10;to predict which domain a given passage-question pair comes from. The&#10;classifier and the passage-question encoder are jointly trained using&#10;adversarial learning to enforce domain-invariant representation learning.&#10;Comprehensive evaluations demonstrate that our approach ($i$) is generalizable&#10;to different MRC models and datasets, ($ii$) can be combined with pre-trained&#10;large-scale language models (such as ELMo and BERT), and ($iii$) can be&#10;extended to semi-supervised learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.12163" label="1909.12163">
        <attvalues>
          <attvalue for="0" value="DARTS: Dialectal Arabic Transcription System" />
          <attvalue for="1" value="  We present the speech to text transcription system, called DARTS, for low&#10;resource Egyptian Arabic dialect. We analyze the following; transfer learning&#10;from high resource broadcast domain to low-resource dialectal domain and&#10;semi-supervised learning where we use in-domain unlabeled audio data collected&#10;from YouTube. Key features of our system are: A deep neural network acoustic&#10;model that consists of a front end Convolutional Neural Network (CNN) followed&#10;by several layers of Time Delayed Neural Network (TDNN) and Long-Short Term&#10;Memory Recurrent Neural Network (LSTM); sequence discriminative training of the&#10;acoustic model; n-gram and recurrent neural network language model for decoding&#10;and N-best list rescoring. We show that a simple transfer learning method can&#10;achieve good results. The results are further improved by using unlabeled data&#10;from YouTube in a semi-supervised setup. Various systems are combined to give&#10;the final system that achieves the lowest word error on on the community&#10;standard Egyptian-Arabic speech dataset (MGB-3).&#10;" />
          <attvalue for="2" value="&#10;&#10;There are a number of major challenges associated with multi-dialect automatic speech recognition (ASR) of conversational Multi-Genre Broadcast (MGB) media, such as background noise variation, cross-talk, and transcriber inconsistency of reference transcripts~\cite{ali2018word}. Dialectal Arabic speech recognition, especially, suffers from the lack of enough in domain transcribed data~\cite{ali2018multi}. There have been many previous attempts to reduce the word error rate (WER) in MGB domain. Both English ASR MGB-1 ~\cite{bell2015mgb} and the first editon of the Arabic ASR in MGB-2 ~\cite{ali2016mgb} focused on using mainstream broadcast media (BBC in MGB-1, Al Jazeera in MGB-2). The MGB-3 ~\cite{ali2017speech} used YouTube recordings to extend the diversity of the challenge and deal with dialectal Arabic as a typical example of languages which do not have well-defined orthographic rules and not enough transcribed data.&#10;&#10;Deep learning has shown great benefit from big data. For example, the MGB-2 best system ~\cite{ khurana2016qcri} achieved more than 7\% relative improvement in WER by augmenting the speech data from $1,200$ to $3,600$ as proposed in ~\cite{ ko2015audio}. &#10;&#10;In this work, we use a simple transfer learning technique combined with semi-supervised learning \cite{manohar2018semi} using unlabelled YouTube data and a very deep neural network based acoustic model to achieve state-of-the-art performance on the challenging low-resource Egyptian Arabic Dialectal speech recognition.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Modeling, Computer Science, Linguistics, Engineering, Signal Processing, Speech Recognition, Transfer Learning" />
        </attvalues>
      </node>
      <node id="1609.05625" label="1609.05625">
        <attvalues>
          <attvalue for="0" value="The MGB-2 Challenge: Arabic Multi-Dialect Broadcast Media Recognition" />
          <attvalue for="1" value="  This paper describes the Arabic Multi-Genre Broadcast (MGB-2) Challenge for&#10;SLT-2016. Unlike last year's English MGB Challenge, which focused on&#10;recognition of diverse TV genres, this year, the challenge has an emphasis on&#10;handling the diversity in dialect in Arabic speech. Audio data comes from 19&#10;distinct programmes from the Aljazeera Arabic TV channel between March 2005 and&#10;December 2015. Programmes are split into three groups: conversations,&#10;interviews, and reports. A total of 1,200 hours have been released with lightly&#10;supervised transcriptions for the acoustic modelling. For language modelling,&#10;we made available over 110M words crawled from Aljazeera Arabic website&#10;Aljazeera.net for a 10 year duration 2000-2011. Two lexicons have been&#10;provided, one phoneme based and one grapheme based. Finally, two tasks were&#10;proposed for this year's challenge: standard speech transcription, and word&#10;alignment. This paper describes the task data and evaluation process used in&#10;the MGB challenge, and summarises the results obtained.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.07276" label="1709.07276">
        <attvalues>
          <attvalue for="0" value="Speech Recognition Challenge in the Wild: Arabic MGB-3" />
          <attvalue for="1" value="  This paper describes the Arabic MGB-3 Challenge - Arabic Speech Recognition&#10;in the Wild. Unlike last year's Arabic MGB-2 Challenge, for which the&#10;recognition task was based on more than 1,200 hours broadcast TV news&#10;recordings from Aljazeera Arabic TV programs, MGB-3 emphasises dialectal Arabic&#10;using a multi-genre collection of Egyptian YouTube videos. Seven genres were&#10;used for the data collection: comedy, cooking, family/kids, fashion, drama,&#10;sports, and science (TEDx). A total of 16 hours of videos, split evenly across&#10;the different genres, were divided into adaptation, development and evaluation&#10;data sets. The Arabic MGB-Challenge comprised two tasks: A) Speech&#10;transcription, evaluated on the MGB-3 test set, along with the 10 hour MGB-2&#10;test set to report progress on the MGB-2 evaluation; B) Arabic dialect&#10;identification, introduced this year in order to distinguish between four major&#10;Arabic dialects - Egyptian, Levantine, North African, Gulf, as well as Modern&#10;Standard Arabic. Two hours of audio per dialect were released for development&#10;and a further two hours were used for evaluation. For dialect identification,&#10;both lexical features and i-vector bottleneck features were shared with&#10;participants in addition to the raw audio recordings. Overall, thirteen teams&#10;submitted ten systems to the challenge. We outline the approaches adopted in&#10;each system, and summarise the evaluation results.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.04325" label="2408.04325">
        <attvalues>
          <attvalue for="0" value="HydraFormer: One Encoder For All Subsampling Rates" />
          <attvalue for="1" value="  In automatic speech recognition, subsampling is essential for tackling&#10;diverse scenarios. However, the inadequacy of a single subsampling rate to&#10;address various real-world situations often necessitates training and deploying&#10;multiple models, consequently increasing associated costs. To address this&#10;issue, we propose HydraFormer, comprising HydraSub, a Conformer-based encoder,&#10;and a BiTransformer-based decoder. HydraSub encompasses multiple branches, each&#10;representing a distinct subsampling rate, allowing for the flexible selection&#10;of any branch during inference based on the specific use case. HydraFormer can&#10;efficiently manage different subsampling rates, significantly reducing training&#10;and deployment expenses. Experiments on AISHELL-1 and LibriSpeech datasets&#10;reveal that HydraFormer effectively adapts to various subsampling rates and&#10;languages while maintaining high recognition performance. Additionally,&#10;HydraFormer showcases exceptional stability, sustaining consistent performance&#10;under various initialization conditions, and exhibits robust transferability by&#10;learning from pretrained single subsampling rate automatic speech recognition&#10;models\footnote{Model code and scripts:&#10;https://github.com/HydraFormer/hydraformer}.&#10;" />
          <attvalue for="2" value="&#10;&#10;Automatic speech recognition (ASR) \cite{asr0,asr1}, as a crucial branch in the field of artificial intelligence, plays a pivotal role in various application scenarios.&#10;With the continuous advancement of technology, ASR models face the challenges of reducing computational complexity and adapting to diverse application scenarios while improving recognition accuracy. To address these issues, researchers have introduced subsampling techniques, aiming to maintain essential information while reducing data volume and enhancing the model's operational efficiency.&#10;&#10;Current ASR research typically uses fixed subsampling rates, yielding satisfactory performance in specific scenarios. For instance, \cite{subsampling4} and \cite{subsampling4_1} opt for a subsampling rate of 4, while \cite{subsampling3} and \cite{subsamplingrate2} choose rates of 3 and 2, respectively.&#10;&#10;In real-world applications, certain scenarios prioritize real-time factor (RTF) over word error rate (WER). For example, edge-cloud hybrid recognition systems \cite{edgecloud} emphasize speed on edge-side speech recognition and accuracy on cloud-side speech recognition. Using the same subsampling rate model for both sides may degrade performance, often necessitating two different models. Additionally, situations like slow speech recognition \cite{slow} benefit from aggressive subsampling strategies, improving RTF and maintaining WER with larger subsampling rates. Moreover, variations in languages, accents, and speakers can yield suboptimal performance for fixed-rate models in some circumstances.&#10;&#10;These scenarios emphasize that a single subsampling rate may not satisfy diverse real-world application needs, necessitating multiple models for different situations. This approach not only increases training and deployment costs but also adds to model management and maintenance complexity. Furthermore, updating or iterating models requires simultaneous updates across all models, leading to substantial expenses. Consequently, developing a unified model that adapts to the subsampling needs of different scenarios while maintaining high recognition performance has become a pressing issue.&#10;&#10;\cite{zhang2019trainable} proposes a method for achieving speech recognition at different downsampling rates through dynamic frame dropping. However, due to the frame dropping, it inevitably loses a significant amount of information. \cite{han2021multistream} suggests using different subsamples with different encoders, which can achieve various downsampling rates. However, this method requires designing a dedicated encoder for each downsampling rate, which greatly increases the model's parameter size and subsequently the training and deployment costs. \cite{kim2022squeezeformer} and \cite{yao2023zipformer} adopt a U-Net-like architecture incorporating multiple sampling rates. However, certain limitations are evident in their approaches. Firstly, the U-Net-inspired structure entails a progressive subsampling process, which continues until the rate is reduced to one-eighth of the original. Each subsampling portion within this process is linked to an encoding module that houses a significant number of parameters which results in substantial computational overhead and inefficiency. Moreover, they lack the flexibility to dynamically adjust the subsampling rate during inference, limiting adaptability in diverse scenarios.&#10;&#10;To address the problem, we propose HydraFormer, a versatile model capable of adapting to different subsampling rates to meet the demands of various application scenarios, as illustrated in Fig. 1. The key idea behind HydraFormer is to enable a single model to handle $N$ different subsampling rates, effectively replacing $N$ individual models that use single subsampling rates. This approach significantly reduces the training and deployment costs to $\boldsymbol{1/N}$ of the original expense while maintaining comparable performance levels. The main contributions of this paper are as follows:&#10;&#10;\begin{itemize}&#10;[itemsep=1pt,topsep=0pt,parsep=0pt]&#10; \item HydraFormer uses a shared encoder for different subsampling rates, reducing training and deployment costs while maintaining comparable recognition performance at various rates compared to single-subsampling-rate ASR models.&#10; \item Omitting positional encoding in HydraSub enhances performance, as various subsampling rates with HydraSub may disrupt the encoder's temporal perception.&#10; \item HydraFormer maintains consistent performance across various model initialization strategies, demonstrating its stability, and it can be easily fine-tuned from pretrained single subsampling rate ASR models.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Engineering, Deep Learning, Signal Processing, Speech Recognition, Model Optimization" />
        </attvalues>
      </node>
      <node id="2210.14515" label="2210.14515">
        <attvalues>
          <attvalue for="0" value="UFO2: A unified pre-training framework for online and offline speech&#10;  recognition" />
          <attvalue for="1" value="  In this paper, we propose a Unified pre-training Framework for Online and&#10;Offline (UFO2) Automatic Speech Recognition (ASR), which 1) simplifies the two&#10;separate training workflows for online and offline modes into one process, and&#10;2) improves the Word Error Rate (WER) performance with limited utterance&#10;annotating. Specifically, we extend the conventional offline-mode&#10;Self-Supervised Learning (SSL)-based ASR approach to a unified manner, where&#10;the model training is conditioned on both the full-context and dynamic-chunked&#10;inputs. To enhance the pre-trained representation model, stop-gradient&#10;operation is applied to decouple the online-mode objectives to the quantizer.&#10;Moreover, in both the pre-training and the downstream fine-tuning stages, joint&#10;losses are proposed to train the unified model with full-weight sharing for the&#10;two modes. Experimental results on the LibriSpeech dataset show that UFO2&#10;outperforms the SSL-based baseline method by 29.7% and 18.2% relative WER&#10;reduction in offline and online modes, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2002.00551" label="2002.00551">
        <attvalues>
          <attvalue for="0" value="End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice&#10;  Activity Detection" />
          <attvalue for="1" value="  This paper integrates a voice activity detection (VAD) function with&#10;end-to-end automatic speech recognition toward an online speech interface and&#10;transcribing very long audio recordings. We focus on connectionist temporal&#10;classification (CTC) and its extension of CTC/attention architectures. As&#10;opposed to an attention-based architecture, input-synchronous label prediction&#10;can be performed based on a greedy search with the CTC (pre-)softmax output.&#10;This prediction includes consecutive long blank labels, which can be regarded&#10;as a non-speech region. We use the labels as a cue for detecting speech&#10;segments with simple thresholding. The threshold value is directly related to&#10;the length of a non-speech region, which is more intuitive and easier to&#10;control than conventional VAD hyperparameters. Experimental results on&#10;unsegmented data show that the proposed method outperformed the baseline&#10;methods using the conventional energy-based and neural-network-based VAD&#10;methods and achieved an RTF less than 0.2. The proposed method is publicly&#10;available.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.10470" label="2005.10470">
        <attvalues>
          <attvalue for="0" value="Multistream CNN for Robust Acoustic Modeling" />
          <attvalue for="1" value="  This paper proposes multistream CNN, a novel neural network architecture for&#10;robust acoustic modeling in speech recognition tasks. The proposed architecture&#10;processes input speech with diverse temporal resolutions by applying different&#10;dilation rates to convolutional neural networks across multiple streams to&#10;achieve the robustness. The dilation rates are selected from the multiples of a&#10;sub-sampling rate of 3 frames. Each stream stacks TDNN-F layers (a variant of&#10;1D CNN), and output embedding vectors from the streams are concatenated then&#10;projected to the final layer. We validate the effectiveness of the proposed&#10;multistream CNN architecture by showing consistent improvements against Kaldi's&#10;best TDNN-F model across various data sets. Multistream CNN improves the WER of&#10;the test-other set in the LibriSpeech corpus by 12% (relative). On custom data&#10;from ASAPP's production ASR system for a contact center, it records a relative&#10;WER improvement of 11% for customer channel audio to prove its robustness to&#10;data in the wild. In terms of real-time factor, multistream CNN outperforms the&#10;baseline TDNN-F by 15%, which also suggests its practicality on production&#10;systems. When combined with self-attentive SRU LM rescoring, multistream CNN&#10;contributes for ASAPP to achieve the best WER of 1.75% on test-clean in&#10;LibriSpeech.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.10549" label="2406.10549">
        <attvalues>
          <attvalue for="0" value="Lightweight Audio Segmentation for Long-form Speech Translation" />
          <attvalue for="1" value="  Speech segmentation is an essential part of speech translation (ST) systems&#10;in real-world scenarios. Since most ST models are designed to process speech&#10;segments, long-form audio must be partitioned into shorter segments before&#10;translation. Recently, data-driven approaches for the speech segmentation task&#10;have been developed. Although the approaches improve overall translation&#10;quality, a performance gap exists due to a mismatch between the models and ST&#10;systems. In addition, the prior works require large self-supervised speech&#10;models, which consume significant computational resources. In this work, we&#10;propose a segmentation model that achieves better speech translation quality&#10;with a small model size. We propose an ASR-with-punctuation task as an&#10;effective pre-training strategy for the segmentation model. We also show that&#10;proper integration of the speech segmentation model into the underlying ST&#10;system is critical to improve overall translation quality at inference time.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech translation (ST), which converts speech signals from one language into text in another language, helps facilitate communication between people who speak different languages and helps overcome language barriers.&#10;Integrating an automatic speech recognition (ASR) component with a machine translation (MT) component is commonly referred to as a cascaded architecture, and it is the traditional and common approach for the ST task~\cite{ney1999speech}.&#10;&#10;Recently, there has been a growing interest in end-to-end (E2E) methods that directly translate spoken source language to target language text using a single sequence-to-sequence model~\cite{berard2016listen,weiss17_interspeech}.&#10;Since E2E ST doesn't produce intermediate speech recognition results, it can prevent ASR errors from propagating to the translation model.&#10;It can also improve latency and model size because it combines the ASR and MT modules into a single model for inference.&#10;However, this approach is still less accurate than the cascade system~\cite{sperber2020speech,agrawal2023findings}.&#10;&#10;Although both cascade and E2E ST systems have been actively developed, the models are designed to process {segmented} speech due to constraints on model architecture and training conditions.&#10;Long-form speech must be segmented in advance to use the ST system in real-world scenarios where segmentation is not available.&#10;However, until recently, it has been under-explored how the segmentation impacts the overall quality of the ST system.&#10;&#10;If a segmentation method is not matched to the underlying ST system, it could lead to low-quality translation results~\cite{salesky2023evaluating}.&#10;In~\cite{wicks2022sentence}, two common failure modes due to the mismatch are discussed.&#10;When a segment is too long or contains multiple sentences, the translation may omit some of the input, called a deletion error.&#10;On the other hand, if a segment is too short or does not contain a proper sentence, the translation may contain phrases not in the input, referred to as an insertion error or {hallucination}~\cite{raunak2021curious,lee2018hallucinations}.&#10;Thus, it is essential to produce segments of appropriate duration and with a single complete sentence to meet the requirements of the underlying ST system.&#10;&#10;Several segmentation methods for ST have been previously introduced in the literature ~\cite{potapczyk2020srpols,gaido2021beyond,gallego2021end,radford2023whisper}.&#10;Pause-based segmentation using voice activity detection (VAD) is commonly employed as a preliminary step for ST systems~\cite{potapczyk2020srpols,gaido2021beyond,gallego2021end}.&#10;Another widely used strategy involves length-based segmentation techniques, where speech is divided into segments according to heuristic principles~\cite{gallego2021end,radford2023whisper}. &#10;For cascaded speech translation systems, there are works on re-segmentation of ASR output text~\cite{cho2012segmentation,wan2021segmenting}.&#10;Also, it is proposed to interpret predictions of ASR and ST models for fixed-size chunks as segmentation~\cite{yoshimura2020end,huang2023e2e,polak2023long,shu2023cif}.&#10;&#10;Recently, data-driven approaches for audio segmentation have been proposed~\cite{shas,fukuda2022speech,fukuda2024improving}, which consist of a neural network encoder and predict segmentation at frame level.&#10;The methods have been shown to improve segmentation performance compared to the traditional methods.&#10;However, the translation quality of the proposed methods is still behind the one of oracle segmentation, as a mismatch exists between the two segmentation results~\cite{salesky2023evaluating}.&#10;Also, the models are usually based on large self-supervised models, such as wav2vec 2.0~\cite{baevski2020wav2vec},&#10;whose computational cost would be a hurdle for deploying ST system on mobile devices.&#10;&#10;In this paper, we aim to improve the end-to-end segmentation modeling for long-form speech translation while significantly reducing the number of model parameters.&#10;We propose that the ASR-with-punctuation task~\cite{nozaki2022interspeech,kim23_interspeech}, the joint task of speech recognition and punctuation prediction, is an effective pre-training task for the segmentation model.&#10;In addition, we show that tuning the segmentation model at inference time is essential to the overall translation quality, and provide an analysis of the mismatch between the segmentation model and ST system.&#10;&#10;Our contributions are as follows:&#10;\begin{itemize}&#10; \item We propose a pre-training strategy for the segmentation model using the ASR-with-punctuation task and show that the proposed pre-training strategy improves the segmentation accuracy and the final translation quality.&#10; \item We show that reducing the mismatch between the segmentation model and ST systems is crucial, due to varying characteristics among ST systems.&#10; \item We demonstrate that the proposed segmentation model achieves better translation quality than the prior methods, and its size is 8\% to 14\% smaller than that of the previous works.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Speech Translation Systems, Model Optimization Techniques, Artificial Intelligence, Signal Processing, Data-Driven Segmentation" />
        </attvalues>
      </node>
      <node id="2104.06683" label="2104.06683">
        <attvalues>
          <attvalue for="0" value="The Curious Case of Hallucinations in Neural Machine Translation" />
          <attvalue for="1" value="  In this work, we study hallucinations in Neural Machine Translation (NMT),&#10;which lie at an extreme end on the spectrum of NMT pathologies. Firstly, we&#10;connect the phenomenon of hallucinations under source perturbation to the&#10;Long-Tail theory of Feldman (2020), and present an empirically validated&#10;hypothesis that explains hallucinations under source perturbation. Secondly, we&#10;consider hallucinations under corpus-level noise (without any source&#10;perturbation) and demonstrate that two prominent types of natural&#10;hallucinations (detached and oscillatory outputs) could be generated and&#10;explained through specific corpus-level noise patterns. Finally, we elucidate&#10;the phenomenon of hallucination amplification in popular data-generation&#10;processes such as Backtranslation and sequence-level Knowledge Distillation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.11710" label="2104.11710">
        <attvalues>
          <attvalue for="0" value="Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct&#10;  Speech Translation" />
          <attvalue for="1" value="  The audio segmentation mismatch between training data and those seen at&#10;run-time is a major problem in direct speech translation. Indeed, while systems&#10;are usually trained on manually segmented corpora, in real use cases they are&#10;often presented with continuous audio requiring automatic (and sub-optimal)&#10;segmentation. After comparing existing techniques (VAD-based, fixed-length and&#10;hybrid segmentation methods), in this paper we propose enhanced hybrid&#10;solutions to produce better results without sacrificing latency. Through&#10;experiments on different domains and language pairs, we show that our methods&#10;outperform all the other techniques, reducing by at least 30% the gap between&#10;the traditional VAD-based approach and optimal manual segmentation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2105.04512" label="2105.04512">
        <attvalues>
          <attvalue for="0" value="End-to-End Speech Translation with Pre-trained Models and Adapters: UPC&#10;  at IWSLT 2021" />
          <attvalue for="1" value="  This paper describes the submission to the IWSLT 2021 offline speech&#10;translation task by the UPC Machine Translation group. The task consists of&#10;building a system capable of translating English audio recordings extracted&#10;from TED talks into German text. Submitted systems can be either cascade or&#10;end-to-end and use a custom or given segmentation. Our submission is an&#10;end-to-end speech translation system, which combines pre-trained models&#10;(Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder,&#10;and uses an efficient fine-tuning technique, which trains only 20% of its total&#10;parameters. We show that adding an Adapter to the system and pre-training it,&#10;can increase the convergence speed and the final result, with which we achieve&#10;a BLEU score of 27.3 on the MuST-C test set. Our final model is an ensemble&#10;that obtains 28.22 BLEU score on the same set. Our submission also uses a&#10;custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for&#10;identifying periods of untranscribable text and can bring improvements of 2.5&#10;to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the&#10;given segmentation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2104.07868" label="2104.07868">
        <attvalues>
          <attvalue for="0" value="Segmenting Subtitles for Correcting ASR Segmentation Errors" />
          <attvalue for="1" value="  Typical ASR systems segment the input audio into utterances using purely&#10;acoustic information, which may not resemble the sentence-like units that are&#10;expected by conventional machine translation (MT) systems for Spoken Language&#10;Translation. In this work, we propose a model for correcting the acoustic&#10;segmentation of ASR models for low-resource languages to improve performance on&#10;downstream tasks. We propose the use of subtitles as a proxy dataset for&#10;correcting ASR acoustic segmentation, creating synthetic acoustic utterances by&#10;modeling common error modes. We train a neural tagging model for correcting ASR&#10;acoustic segmentation and show that it improves downstream performance on MT&#10;and audio-document cross-language information retrieval (CLIR).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2211.15432" label="2211.15432">
        <attvalues>
          <attvalue for="0" value="E2E Segmentation in a Two-Pass Cascaded Encoder ASR Model" />
          <attvalue for="1" value="  We explore unifying a neural segmenter with two-pass cascaded encoder ASR&#10;into a single model. A key challenge is allowing the segmenter (which runs in&#10;real-time, synchronously with the decoder) to finalize the 2nd pass (which runs&#10;900 ms behind real-time) without introducing user-perceived latency or deletion&#10;errors during inference. We propose a design where the neural segmenter is&#10;integrated with the causal 1st pass decoder to emit a end-of-segment (EOS)&#10;signal in real-time. The EOS signal is then used to finalize the non-causal 2nd&#10;pass. We experiment with different ways to finalize the 2nd pass, and find that&#10;a novel dummy frame injection strategy allows for simultaneous high quality 2nd&#10;pass results and low finalization latency. On a real-world long-form captioning&#10;task (YouTube), we achieve 2.4% relative WER and 140 ms EOS latency gains over&#10;a baseline VAD-based segmenter with the same cascaded encoder.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2309.11384" label="2309.11384">
        <attvalues>
          <attvalue for="0" value="Long-Form End-to-End Speech Translation via Latent Alignment&#10;  Segmentation" />
          <attvalue for="1" value="  Current simultaneous speech translation models can process audio only up to a&#10;few seconds long. Contemporary datasets provide an oracle segmentation into&#10;sentences based on human-annotated transcripts and translations. However, the&#10;segmentation into sentences is not available in the real world. Current speech&#10;segmentation approaches either offer poor segmentation quality or have to trade&#10;latency for quality. In this paper, we propose a novel segmentation approach&#10;for a low-latency end-to-end speech translation. We leverage the existing&#10;speech translation encoder-decoder architecture with ST CTC and show that it&#10;can perform the segmentation task without supervision or additional parameters.&#10;To the best of our knowledge, our method is the first that allows an actual&#10;end-to-end simultaneous speech translation, as the same model is used for&#10;translation and segmentation at the same time. On a diverse set of language&#10;pairs and in- and out-of-domain data, we show that the proposed approach&#10;achieves state-of-the-art quality at no additional computational cost.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2202.04774" label="2202.04774">
        <attvalues>
          <attvalue for="0" value="SHAS: Approaching optimal Segmentation for End-to-End Speech Translation" />
          <attvalue for="1" value="  Speech translation models are unable to directly process long audios, like&#10;TED talks, which have to be split into shorter segments. Speech translation&#10;datasets provide manual segmentations of the audios, which are not available in&#10;real-world scenarios, and existing segmentation methods usually significantly&#10;reduce translation quality at inference time. To bridge the gap between the&#10;manual segmentation of training and the automatic one at inference, we propose&#10;Supervised Hybrid Audio Segmentation (SHAS), a method that can effectively&#10;learn the optimal segmentation from any manually segmented speech corpus.&#10;First, we train a classifier to identify the included frames in a segmentation,&#10;using speech representations from a pre-trained wav2vec 2.0. The optimal&#10;splitting points are then found by a probabilistic Divide-and-Conquer algorithm&#10;that progressively splits at the frame of lowest probability until all segments&#10;are below a pre-specified length. Experiments on MuST-C and mTEDx show that the&#10;translation of the segments produced by our method approaches the quality of&#10;the manual segmentation on 5 language pairs. Namely, SHAS retains 95-98% of the&#10;manual segmentation's BLEU score, compared to the 87-93% of the best existing&#10;methods. Our method is additionally generalizable to different domains and&#10;achieves high zero-shot performance in unseen languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.15479" label="2203.15479">
        <attvalues>
          <attvalue for="0" value="Speech Segmentation Optimization using Segmented Bilingual Speech Corpus&#10;  for End-to-end Speech Translation" />
          <attvalue for="1" value="  Speech segmentation, which splits long speech into short segments, is&#10;essential for speech translation (ST). Popular VAD tools like WebRTC VAD have&#10;generally relied on pause-based segmentation. Unfortunately, pauses in speech&#10;do not necessarily match sentence boundaries, and sentences can be connected by&#10;a very short pause that is difficult to detect by VAD. In this study, we&#10;propose a speech segmentation method using a binary classification model&#10;trained using a segmented bilingual speech corpus. We also propose a hybrid&#10;method that combines VAD and the above speech segmentation method. Experimental&#10;results revealed that the proposed method is more suitable for cascade and&#10;end-to-end ST systems than conventional segmentation methods. The hybrid&#10;approach further improved the translation performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.03169" label="2207.03169">
        <attvalues>
          <attvalue for="0" value="End-to-end Speech-to-Punctuated-Text Recognition" />
          <attvalue for="1" value="  Conventional automatic speech recognition systems do not produce punctuation&#10;marks which are important for the readability of the speech recognition&#10;results. They are also needed for subsequent natural language processing tasks&#10;such as machine translation. There have been a lot of works on punctuation&#10;prediction models that insert punctuation marks into speech recognition results&#10;as post-processing. However, these studies do not utilize acoustic information&#10;for punctuation prediction and are directly affected by speech recognition&#10;errors. In this study, we propose an end-to-end model that takes speech as&#10;input and outputs punctuated texts. This model is expected to predict&#10;punctuation robustly against speech recognition errors while using acoustic&#10;information. We also propose to incorporate an auxiliary loss to train the&#10;model using the output of the intermediate layer and unpunctuated texts.&#10;Through experiments, we compare the performance of the proposed model to that&#10;of a cascaded system. The proposed model achieves higher punctuation prediction&#10;accuracy than the cascaded system without sacrificing the speech recognition&#10;error rate. It is also demonstrated that the multi-task learning using the&#10;intermediate output against the unpunctuated text is effective. Moreover, the&#10;proposed model has only about 1/7th of the parameters compared to the cascaded&#10;system.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.01296" label="2306.01296">
        <attvalues>
          <attvalue for="0" value="Improved Training for End-to-End Streaming Automatic Speech Recognition&#10;  Model with Punctuation" />
          <attvalue for="1" value="  Punctuated text prediction is crucial for automatic speech recognition as it&#10;enhances readability and impacts downstream natural language processing tasks.&#10;In streaming scenarios, the ability to predict punctuation in real-time is&#10;particularly desirable but presents a difficult technical challenge. In this&#10;work, we propose a method for predicting punctuated text from input speech&#10;using a chunk-based Transformer encoder trained with Connectionist Temporal&#10;Classification (CTC) loss. The acoustic model trained with long sequences by&#10;concatenating the input and target sequences can learn punctuation marks&#10;attached to the end of sentences more effectively. Additionally, by combining&#10;CTC losses on the chunks and utterances, we achieved both the improved F1 score&#10;of punctuation prediction and Word Error Rate (WER).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2408.11845" label="2408.11845">
        <attvalues>
          <attvalue for="0" value="LLaMA based Punctuation Restoration With Forward Pass Only Decoding" />
          <attvalue for="1" value="  This paper introduces two advancements in the field of Large Language Model&#10;Annotation with a focus on punctuation restoration tasks. Our first&#10;contribution is the application of LLaMA for punctuation restoration, which&#10;demonstrates superior performance compared to the established benchmark.&#10;  Despite its impressive quality, LLaMA faces challenges regarding inference&#10;speed and hallucinations. To address this, our second contribution presents&#10;Forward Pass Only Decoding (FPOD), a novel decoding approach for annotation&#10;tasks. This innovative method results in a substantial 19.8x improvement in&#10;inference speed, effectively addressing a critical bottleneck and enhancing the&#10;practical utility of LLaMA for large-scale data annotation tasks without&#10;hallucinations.&#10;  The combination of these contributions not only solidifies LLaMA as a&#10;powerful tool for punctuation restoration but also highlights FPOD as a crucial&#10;strategy for overcoming speed constraints.&#10;" />
          <attvalue for="2" value="&#10;Automatic Speech Recognition (ASR) plays a vital role in numerous domains involving human-computer interaction \cite{hinton2012deep} \cite{mohamed2011acoustic} \cite{sak2014long}. However, the outputs of many ASR systems often lack punctuation. Punctuation restoration in the context of ASR output is a crucial component \cite{tilk2016bidirectional} \cite{courtland2020efficient}, essential for enhancing the overall utility, user experience, and comprehensibility of transcribed speech. Restoring the punctuation will make the raw ASR output more coherent, with improved intractability.&#10;&#10;The field of punctuation restoration encompasses two distinct techniques: cascade methods, exemplified by models like BERT \cite{devlin2018bert}, commonly applied independently to Automatic Speech Recognition (ASR) outputs in spoken domains without punctuation \cite{puaics2022capitalization}. These cascade models function as standalone systems, addressing the punctuation restoration task sequentially. On the other hand, the End-to-End (E2E) approach, represented by models such as Recurrent Neural Network Transducer (RNNT) or Whisper \cite{radford2023robust}, trained in an end2end fashion, incorporates built-in punctuation output. This category of techniques streamlines the punctuation restoration process. However, both approaches face challenges, the former requiring independent but domain-aligned training data and evaluation effort, and the latter compels to use large amounts of high-quality supervised data containing punctuation paired with audio, which is a bottleneck for scaling ASR systems to new domains and languages requiring punctuation restoration.&#10;&#10;Recognizing the significance of the punctuation restoration task and the challenges posed by previous models, our work introduces an approach that leverages the capabilities of LLaMA. Acknowledged for its effectiveness in various language-related tasks, LLaMA emerges as a compelling alternative that surpasses existing benchmarks across numerous Natural Language Processing (NLP) tasks \cite{touvron2023llama} \cite{touvron2023llama2}. Additionally, with LoRA fine-tuning \cite{hu2021lora}, which demands significantly less supervised training data, we achieve comparable and even superior performance for punctuation restoration compared to traditional methods introduced previously. This approach addresses both the quality effectiveness and scale-up concerns associated with punctuation restoration in diverse languages and domains.&#10;&#10;In our exploration of LLaMA-based punctuation restoration, we present a range of strategies. Initially, we delve into the traditional approach with auto-regressive generation. Subsequently, we explore techniques to address the inherent challenge of inference speed in LLaMA. The first of these strategies involves speculative decoding, showcasing improvements in inference speed while maintaining the quality of generated outputs exactly the same as the original base model. Finally, we present a new forward pass only approach, eliminating the need for auto-regressive generation entirely. This novel approach results in a substantial boost in inference speed.&#10;&#10;Our contribution not only establishes LLaMA as a potent alternative for achieving high-quality punctuation restoration but also introduces practical enhancements to overcome the challenges associated with inference speed.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Punctuation Restoration, Computer Science, Information Technology, Decoding Algorithm Optimization, Linguistics, Mathematics, Language Model Annotation" />
        </attvalues>
      </node>
      <node id="2111.10746" label="2111.10746">
        <attvalues>
          <attvalue for="0" value="Capitalization and Punctuation Restoration: a Survey" />
          <attvalue for="1" value="  Ensuring proper punctuation and letter casing is a key pre-processing step&#10;towards applying complex natural language processing algorithms. This is&#10;especially significant for textual sources where punctuation and casing are&#10;missing, such as the raw output of automatic speech recognition systems.&#10;Additionally, short text messages and micro-blogging platforms offer unreliable&#10;and often wrong punctuation and casing. This survey offers an overview of both&#10;historical and state-of-the-art techniques for restoring punctuation and&#10;correcting word casing. Furthermore, current challenges and research directions&#10;are highlighted.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.07356" label="2204.07356">
        <attvalues>
          <attvalue for="0" value="Vision-and-Language Pretrained Models: A Survey" />
          <attvalue for="1" value="  Pretrained models have produced great success in both Computer Vision (CV)&#10;and Natural Language Processing (NLP). This progress leads to learning joint&#10;representations of vision and language pretraining by feeding visual and&#10;linguistic contents into a multi-layer transformer, Visual-Language Pretrained&#10;Models (VLPMs). In this paper, we present an overview of the major advances&#10;achieved in VLPMs for producing joint representations of vision and language.&#10;As the preliminaries, we briefly describe the general task definition and&#10;genetic architecture of VLPMs. We first discuss the language and vision data&#10;encoding methods and then present the mainstream VLPM structure as the core&#10;content. We further summarise several essential pretraining and fine-tuning&#10;strategies. Finally, we highlight three future directions for both CV and NLP&#10;researchers to provide insightful guidance.&#10;" />
          <attvalue for="2" value="&#10;In both Computer Vision (CV) and Natural Language Processing (NLP) communities, pretrained models have made significant progress. While CV researchers use VGG and ResNet using ImageNet to predict the categorical label of a given image, BERT~\cite{devlin2019bert} has been used and revolutionised many NLP tasks, such as natural language inference, and reading comprehension. Motivated by this, many cross-modal Vision-Language Pretrained Models (VLPMs) have been designed~\cite{vilbert,vlbert,uniter,oscar}. This pretrain-then-transfer learning approach to vision-language tasks naturally follows its widespread use in both CV and NLP. It has become the de facto standard due to the ease of use and solid representational power of large, publicly available models trained on large-scaled data sources. &#10;&#10;In this paper, we present an overview of the rise and major advances achieved in the topic of VLPMs. Figure \ref{fig:vlp} illustrates a generic architecture of VLPMs. It involves the design of four main components: 1) V/L (Vision and Language) Raw Input Data defines the representative raw data streams from language and visual contents respectively, such as a single or multiple sentence(s) and one or a set of image(s). 2) V/L Representation processes the raw data input into the desired format of modality representations that can be used for 3) V-L (Vision-Language) Interaction Model, which then enforces the cross-modal modeling between the two modalities. For instance, a common design is that the textual sentence is first tokenized and converted into the Bert-formatted input embedding while the image is processed into a set of spatial-aware RoI (Region of Interest) features. Those two modality representations are then concatenated and fed into the transformer encoder layers in which the cross-modal interaction is modeled via the multi-head self-attention mechanism. 4) V-L Representation defines the possible cross-modal representations, which can be a V-L representation for the single modality (i.e., Language or Vision) and/or the V-L representation of joint modalities (i.e., Language and Vision). With the well-designed task supervision and learning guidelines from the pretraining, the V-L representation finally learns to represent the generic cross-modal semantics, which would be transferred to help with the downstream V-L tasks via fine-tuning. This generic architecture applies to most of the existing VLPMs. The designs are various for each component, their pretraining strategies and transfer applications. &#10;&#10;Existing surveys in this area have only partially reviewed some related tasks \cite{mogadala2021trends} or focused mainly on systematical analysis \cite{metaanalysis}. &#10;To the best of our knowledge, this is the first work that presents a comprehensive review of VLPMs. Our paper aims to provide both CV and NLP researchers insightful guidance for visual and language cross-modal learning via pretraining.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Multimodal Pretraining, Computer Vision, Artificial Intelligence, Natural Language Processing, Visual Language Models, Joint Representation Learning" />
        </attvalues>
      </node>
      <node id="2011.15124" label="2011.15124">
        <attvalues>
          <attvalue for="0" value="Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework&#10;  of Vision-and-Language BERTs" />
          <attvalue for="1" value="  Large-scale pretraining and task-specific fine-tuning is now the standard&#10;methodology for many tasks in computer vision and natural language processing.&#10;Recently, a multitude of methods have been proposed for pretraining vision and&#10;language BERTs to tackle challenges at the intersection of these two key areas&#10;of AI. These models can be categorised into either single-stream or dual-stream&#10;encoders. We study the differences between these two categories, and show how&#10;they can be unified under a single theoretical framework. We then conduct&#10;controlled experiments to discern the empirical differences between five V&amp;L&#10;BERTs. Our experiments show that training data and hyperparameters are&#10;responsible for most of the differences between the reported results, but they&#10;also reveal that the embedding layer plays a crucial role in these massive&#10;models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2203.14222" label="2203.14222">
        <attvalues>
          <attvalue for="0" value="Listen, Adapt, Better WER: Source-free Single-utterance Test-time&#10;  Adaptation for Automatic Speech Recognition" />
          <attvalue for="1" value="  Although deep learning-based end-to-end Automatic Speech Recognition (ASR)&#10;has shown remarkable performance in recent years, it suffers severe performance&#10;regression on test samples drawn from different data distributions. Test-time&#10;Adaptation (TTA), previously explored in the computer vision area, aims to&#10;adapt the model trained on source domains to yield better predictions for test&#10;samples, often out-of-domain, without accessing the source data. Here, we&#10;propose the Single-Utterance Test-time Adaptation (SUTA) framework for ASR,&#10;which is the first TTA study on ASR to our best knowledge. The single-utterance&#10;TTA is a more realistic setting that does not assume test data are sampled from&#10;identical distribution and does not delay on-demand inference due to&#10;pre-collection for the batch of adaptation data. SUTA consists of unsupervised&#10;objectives with an efficient adaptation strategy. Empirical results demonstrate&#10;that SUTA effectively improves the performance of the source ASR model&#10;evaluated on multiple out-of-domain target corpora and in-domain test samples.&#10;" />
          <attvalue for="2" value="&#10;Deep Learning-based Automatic Speech Recognition (ASR) models achieve impressive success, especially when samples are drawn under the independent and identical distribution (i.i.d.) assumption. However, performance degrades severely when covariate shift (i.e., distribution of test data differs from training data) happens. In real-world scenarios, such covariate-shifted test samples are ubiquitous, making ASR services unstable and unreliable. Therefore, it is critical to alleviate the adverse effect of data shifting.&#10;&#10;Unsupervised Domain Adaptation (UDA) is a commonly-used approach that adapts the source model to the target domain without annotated data. The existing UDA approaches such as domain adversarial training \cite{sun2018domain, sun2017unsupervised}, knowledge distillation \cite{li2017large}, and self-training \cite{khurana2021unsupervised} have shown effectiveness for mitigating data shifting and improving ASR performance. &#10;However, they all require access to source data and sufficient target domain examples for adaptation. Such requirement imposes three main limitation on the real-world application of UDA: (1) source data is not always available during adaptation due to the privacy/storage issues, (2) latency due to target data pre-collection and heavy computation for model adaptation, and (3) the assumption that the target domain examples come from the same distribution. &#10;\looseness=-1&#10;&#10;Test-Time Adaptation (TTA) \cite{wang2020tent, liang2020we, mummadi2021test, khurana2021sita, Hu2021MixNormTA, you2021test, fleuret2021test} has recently attracted growing interest since it effectively adapts models in prediction time with little target data (a batch or even a single instance) without access to source data. &#10;Several studies have shown remarkable TTA effectiveness in computer vision but lack research attempts on TTA for ASR. It is worth noting that TTA in computer vision is heavily targeted on the Batch Normalization (BN) layer's adaptation by re-estimating batch statistics on target data. However, sequential models, such as ASR models, typically are not equipped with BN layers because the lengths of batched input sequences are different. Therefore, most ASR models use instance-wise Layer Normalization (LN) layer. The discrepancy in data format and model architecture also motivates us to innovate TTA tailored for ASR. &#10;\looseness=-1&#10;&#10;Among different TTA variants, most methods are restricted to batch-level TTA. In other words, TTA resolves the first limitation (i.e., privacy/storage) of UDA, while the rest two (latency and target data distribution) still cause challenges.&#10;Recently, SITA \cite{khurana2021sita} has presented a single-instance TTA method, which lifts the limitation about latency and distribution since it does not require the pre-collecting batch of test samples, and it allows test samples that come from heterogeneous sources. However, SITA still focuses on computer vision problems and relies on data augmentation and statistical estimation of BN layers.&#10;&#10;To the best of our knowledge, no existing work attempts TTA or even single-instance TTA on ASR, and there is lacking research exploring the potential of TTA in improving ASR performance. To fill the gap, in this work, we propose the Single-Utterance Test-time Adaptation (SUTA) framework to improve ASR test-time performance with a single utterance. SUTA can be applied to any CTC-based end-to-end ASR model. Our method does not rely on large batch size and access to source data, which causes delayed inference and privacy issues; instead, only one testing utterance is needed for test-time adaptation in an unsupervised manner. SUTA consistently improves the source ASR model in multiple out-of-domain target corpora and in-domain test samples with little computational delay.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Test-time Adaptation, Linguistics, Automatic Speech Recognition, Deep Learning, Signal Processing, Mathematics" />
        </attvalues>
      </node>
      <node id="2006.10726" label="2006.10726">
        <attvalues>
          <attvalue for="0" value="Tent: Fully Test-time Adaptation by Entropy Minimization" />
          <attvalue for="1" value="  A model must adapt itself to generalize to new and different data during&#10;testing. In this setting of fully test-time adaptation the model has only the&#10;test data and its own parameters. We propose to adapt by test entropy&#10;minimization (tent): we optimize the model for confidence as measured by the&#10;entropy of its predictions. Our method estimates normalization statistics and&#10;optimizes channel-wise affine transformations to update online on each batch.&#10;Tent reduces generalization error for image classification on corrupted&#10;ImageNet and CIFAR-10/100 and reaches a new state-of-the-art error on&#10;ImageNet-C. Tent handles source-free domain adaptation on digit recognition&#10;from SVHN to MNIST/MNIST-M/USPS, on semantic segmentation from GTA to&#10;Cityscapes, and on the VisDA-C benchmark. These results are achieved in one&#10;epoch of test-time optimization without altering training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2112.02355" label="2112.02355">
        <attvalues>
          <attvalue for="0" value="SITA: Single Image Test-time Adaptation" />
          <attvalue for="1" value="  In Test-time Adaptation (TTA), given a source model, the goal is to adapt it&#10;to make better predictions for test instances from a different distribution&#10;than the source. Crucially, TTA assumes no access to the source data or even&#10;any additional labeled/unlabeled samples from the target distribution to&#10;finetune the source model. In this work, we consider TTA in a more pragmatic&#10;setting which we refer to as SITA (Single Image Test-time Adaptation). Here,&#10;when making a prediction, the model has access only to the given single test&#10;instance, rather than a batch of instances, as typically been considered in the&#10;literature. This is motivated by the realistic scenarios where inference is&#10;needed on-demand instead of delaying for an incoming batch or the inference is&#10;happening on an edge device (like mobile phone) where there is no scope for&#10;batching. The entire adaptation process in SITA should be extremely fast as it&#10;happens at inference time. To address this, we propose a novel approach AugBN&#10;that requires only a single forward pass. It can be used on any off-the-shelf&#10;trained model to test single instances for both classification and segmentation&#10;tasks. AugBN estimates normalization statistics of the unseen test distribution&#10;from the given test image using only one forward pass with label-preserving&#10;transformations. Since AugBN does not involve any back-propagation, it is&#10;significantly faster compared to recent test time adaptation methods. We&#10;further extend AugBN to make the algorithm hyperparameter-free. Rigorous&#10;experimentation show that our simple algorithm is able to achieve significant&#10;performance gains for a variety of datasets, tasks, and network architectures.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.14794" label="2305.14794">
        <attvalues>
          <attvalue for="0" value="Debiasing Made State-of-the-art: Revisiting the Simple Seed-based Weak&#10;  Supervision for Text Classification" />
          <attvalue for="1" value="  Recent advances in weakly supervised text classification mostly focus on&#10;designing sophisticated methods to turn high-level human heuristics into&#10;quality pseudo-labels. In this paper, we revisit the seed matching-based&#10;method, which is arguably the simplest way to generate pseudo-labels, and show&#10;that its power was greatly underestimated. We show that the limited performance&#10;of seed matching is largely due to the label bias injected by the simple&#10;seed-match rule, which prevents the classifier from learning reliable&#10;confidence for selecting high-quality pseudo-labels. Interestingly, simply&#10;deleting the seed words present in the matched input texts can mitigate the&#10;label bias and help learn better confidence. Subsequently, the performance&#10;achieved by seed matching can be improved significantly, making it on par with&#10;or even better than the state-of-the-art. Furthermore, to handle the case when&#10;the seed words are not made known, we propose to simply delete the word tokens&#10;in the input text randomly with a high deletion ratio. Remarkably, seed&#10;matching equipped with this random deletion method can often achieve even&#10;better performance than that with seed deletion.&#10;" />
          <attvalue for="2" value="&#10;&#10;Recently, weakly supervised text classification, because of its light requirement of human effort, has been extensively studied~\cite{mekala2020contextualized,wang2020x,npprompt,meng2020text,Zhang2021WeaklysupervisedTC,meng2018weakly,tao2015doc2cube,Park2022LIMEWT}.&#10;Specifically, it requires only high-level human guidance to label the text, such as a few rules provided by human experts that match the text with the labels.&#10;These labels, which are not necessarily correct and are thus often dubbed as pseudo-labels, are then employed to train the text classifier following a standard fully supervised or semi-supervised training framework. &#10;State-of-the-art methods mostly focus on designing sophisticated human guidance to obtain high-quality labels, through contextualized weak supervision~\cite{mekala2020contextualized}, prompting language models~\cite{meng2020text,npprompt}, clustering for soft matching ~\cite{wang2020x}, and complicated interactions between seeds~\cite{Zhang2021WeaklysupervisedTC}. &#10;&#10;In this paper, we revisit the seed matching-based weak supervision (denoted as Vanilla)~\cite{mekala2020contextualized,meng2018weakly,tao2015doc2cube}, which is arguably the simplest way to generate pseudo-labels, and show that its power was greatly underestimated.&#10;Specifically, this simple method matches input text with a label if the user-provided seed words of this label are contained in the input text. For example, in sentiment analysis, a document will be labeled as ``positive'' if it contains the word ``happy''.&#10;A text classifier is then trained based on all these pseudo-labels.&#10;&#10;One can expect a non-trivial number of errors in the seed matching-based pseudo-labels. In an ideal case, if we can select only those correct pseudo-labels for training, the accuracy of the text classifier can be significantly boosted. For example, on the 20 Newsgroups dataset, ideally with only those correct pseudo-labels one can get an accuracy of $90.6\%$, compared to $80.1\%$ obtained on all pseudo-labels (see more in Section~\ref{sect:experiment}). In practice, to select those correct labels, a common way is to use the confidence score of a classifier trained on pseudo-labels~\cite{Rizve2021InDO}. However, those high-confidence pseudo-labels may not be correct in the weakly-supervised setting, likely because the classifier may fail to learn reliable confidence on these noisy pseudo-labels~\cite{Mekala2022LOPSLO}. &#10;&#10;In this paper, we take a deep dive into this problem and find that, surprisingly, the high noise rate among the pseudo-labels is often not an obstacle to learning reliable confidence at all. In fact, on a set of synthesized pseudo-labels where the noise rate is exactly the same as those given by seed matching, but the noisy labels are generated by randomly flipping true labels into other classes, the confidence learned by a classifier can genuinely reflect the correct labels, as shown in Figure~\ref{figure:noise-coverage}.&#10;&#10;Therefore, we argue that the poor confidence learned on realistic pseudo-labels is largely attributed to the strong but likely erroneous correlation between the text and pseudo-label injected by the seed-matching rule, which we refer to as label bias. Such a bias can be easily learned by the text classifier upon training, thus yielding spuriously high confidence on any text matching the seed word and ruining the pseudo-label selection. &#10;&#10;To defend against such a label bias, we propose to simply delete the seed words present in the text upon training a classifier on the pseudo-labeled data, which effectively prevents the classifier from learning the biased correlation between seeds and the corresponding pseudo-labels. As shown in Figure~\ref{figure:noise-coverage}, such a simple seed deletion method can significantly improve the confidence score of the trained classifier and thus help select pseudo-labels with fewer label errors at every selection ratio. &#10;Empirical results verify that these less noisy pseudo-labels can indeed improve the classification accuracy significantly, making seed matching-based weak supervision on par with or sometimes even better than the state-of-the-art. &#10;&#10;We further investigate the scenario where the seed words are not made known.&#10;We propose to delete every word token in the input text randomly and independently. This simple random deletion method can improve confidence learning even more as shown in Figure~\ref{figure:noise-coverage}.&#10;Our theoretical analysis also shows that&#10;this random deletion method can mitigate the label bias with a high probability and therefore recover the seed deletion in effect.&#10;It is worth noting that both of these methods introduce no additional hyperparameters.&#10;&#10;In summary, our contributions are as follows.&#10;\begin{itemize}[nosep,leftmargin=*]&#10; \item We revisit the seed matching-based weak supervision and find that its effectiveness is mainly limited by the label bias injected by the seed-matching rule.&#10; \item We show that simply deleting seed words from the pseudo-labeled texts can significantly alleviate the label bias and improve the confidence estimation for pseudo-label selection, as well as end-to-end classification accuracy achieved by seed matching, on par with or even better than the state-of-the-art.&#10; \item We further propose the random deletion method to handle the case when the seed words are unknown and demonstrate its effectiveness both empirically and theoretically.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Text Classification, Computer Science, Linguistics, Pseudo-Label Generation, Artificial Intelligence, Weak Supervision, Statistics" />
        </attvalues>
      </node>
      <node id="2110.02591" label="2110.02591">
        <attvalues>
          <attvalue for="0" value="Weakly-supervised Text Classification Based on Keyword Graph" />
          <attvalue for="1" value="  Weakly-supervised text classification has received much attention in recent&#10;years for it can alleviate the heavy burden of annotating massive data. Among&#10;them, keyword-driven methods are the mainstream where user-provided keywords&#10;are exploited to generate pseudo-labels for unlabeled texts. However, existing&#10;methods treat keywords independently, thus ignore the correlation among them,&#10;which should be useful if properly exploited. In this paper, we propose a novel&#10;framework called ClassKG to explore keyword-keyword correlation on keyword&#10;graph by GNN. Our framework is an iterative process. In each iteration, we&#10;first construct a keyword graph, so the task of assigning pseudo labels is&#10;transformed to annotating keyword subgraphs. To improve the annotation quality,&#10;we introduce a self-supervised task to pretrain a subgraph annotator, and then&#10;finetune it. With the pseudo labels generated by the subgraph annotator, we&#10;then train a text classifier to classify the unlabeled texts. Finally, we&#10;re-extract keywords from the classified texts. Extensive experiments on both&#10;long-text and short-text datasets show that our method substantially&#10;outperforms the existing ones&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.01478" label="1809.01478">
        <attvalues>
          <attvalue for="0" value="Weakly-Supervised Neural Text Classification" />
          <attvalue for="1" value="  Deep neural networks are gaining increasing popularity for the classic text&#10;classification task, due to their strong expressive power and less requirement&#10;for feature engineering. Despite such attractiveness, neural text&#10;classification models suffer from the lack of training data in many real-world&#10;applications. Although many semi-supervised and weakly-supervised text&#10;classification models exist, they cannot be easily applied to deep neural&#10;models and meanwhile support limited supervision types. In this paper, we&#10;propose a weakly-supervised method that addresses the lack of training data in&#10;neural text classification. Our method consists of two modules: (1) a&#10;pseudo-document generator that leverages seed information to generate&#10;pseudo-labeled documents for model pre-training, and (2) a self-training module&#10;that bootstraps on real unlabeled data for model refinement. Our method has the&#10;flexibility to handle different types of weak supervision and can be easily&#10;integrated into existing deep neural models for text classification. We have&#10;performed extensive experiments on three real-world datasets from different&#10;domains. The results demonstrate that our proposed method achieves inspiring&#10;performance without requiring excessive training data and outperforms baseline&#10;methods significantly.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.06720" label="2210.06720">
        <attvalues>
          <attvalue for="0" value="LIME: Weakly-Supervised Text Classification Without Seeds" />
          <attvalue for="1" value="  In weakly-supervised text classification, only label names act as sources of&#10;supervision. Predominant approaches to weakly-supervised text classification&#10;utilize a two-phase framework, where test samples are first assigned&#10;pseudo-labels and are then used to train a neural text classifier. In most&#10;previous work, the pseudo-labeling step is dependent on obtaining seed words&#10;that best capture the relevance of each class label. We present LIME, a&#10;framework for weakly-supervised text classification that entirely replaces the&#10;brittle seed-word generation process with entailment-based&#10;pseudo-classification. We find that combining weakly-supervised classification&#10;and textual entailment mitigates shortcomings of both, resulting in a more&#10;streamlined and effective classification pipeline. With just an off-the-shelf&#10;textual entailment model, LIME outperforms recent baselines in&#10;weakly-supervised text classification and achieves state-of-the-art in 4&#10;benchmarks. We open source our code at https://github.com/seongminp/LIME.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2101.06329" label="2101.06329">
        <attvalues>
          <attvalue for="0" value="In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label&#10;  Selection Framework for Semi-Supervised Learning" />
          <attvalue for="1" value="  The recent research in semi-supervised learning (SSL) is mostly dominated by&#10;consistency regularization based methods which achieve strong performance.&#10;However, they heavily rely on domain-specific data augmentations, which are not&#10;easy to generate for all data modalities. Pseudo-labeling (PL) is a general SSL&#10;approach that does not have this constraint but performs relatively poorly in&#10;its original formulation. We argue that PL underperforms due to the erroneous&#10;high confidence predictions from poorly calibrated models; these predictions&#10;generate many incorrect pseudo-labels, leading to noisy training. We propose an&#10;uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo&#10;labeling accuracy by drastically reducing the amount of noise encountered in&#10;the training process. Furthermore, UPS generalizes the pseudo-labeling process,&#10;allowing for the creation of negative pseudo-labels; these negative&#10;pseudo-labels can be used for multi-label classification as well as negative&#10;learning to improve the single-label classification. We achieve strong&#10;performance when compared to recent SSL methods on the CIFAR-10 and CIFAR-100&#10;datasets. Also, we demonstrate the versatility of our method on the video&#10;dataset UCF-101 and the multi-label dataset Pascal VOC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2205.12528" label="2205.12528">
        <attvalues>
          <attvalue for="0" value="LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly&#10;  Supervised Text Classification" />
          <attvalue for="1" value="  Weakly supervised text classification methods typically train a deep neural&#10;classifier based on pseudo-labels. The quality of pseudo-labels is crucial to&#10;final performance but they are inevitably noisy due to their heuristic nature,&#10;so selecting the correct ones has a huge potential for performance boost. One&#10;straightforward solution is to select samples based on the softmax probability&#10;scores in the neural classifier corresponding to their pseudo-labels. However,&#10;we show through our experiments that such solutions are ineffective and&#10;unstable due to the erroneously high-confidence predictions from poorly&#10;calibrated models. Recent studies on the memorization effects of deep neural&#10;models suggest that these models first memorize training samples with clean&#10;labels and then those with noisy labels. Inspired by this observation, we&#10;propose a novel pseudo-label selection method LOPS that takes learning order of&#10;samples into consideration. We hypothesize that the learning order reflects the&#10;probability of wrong annotation in terms of ranking, and therefore, propose to&#10;select the samples that are learnt earlier. LOPS can be viewed as a strong&#10;performance-boost plug-in to most of existing weakly-supervised text&#10;classification methods, as confirmed in extensive experiments on four&#10;real-world datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2301.12568" label="2301.12568">
        <attvalues>
          <attvalue for="0" value="Schema-Guided Semantic Accuracy: Faithfulness in Task-Oriented Dialogue&#10;  Response Generation" />
          <attvalue for="1" value="  Ensuring that generated utterances are faithful to dialogue actions is&#10;crucial for Task-Oriented Dialogue Response Generation. Slot Error Rate (SER)&#10;only partially measures generation quality in that it solely assesses&#10;utterances generated from non-categorical slots whose values are expected to be&#10;reproduced exactly. Utterances generated from categorical slots, which are more&#10;variable, are not assessed by SER. We propose Schema-Guided Semantic Accuracy&#10;(SGSAcc) to evaluate utterances generated from both categorical and&#10;non-categorical slots by recognizing textual entailment. We show that SGSAcc&#10;can be applied to evaluate utterances generated from a wide range of dialogue&#10;actions in the Schema Guided Dialogue (SGD) dataset with good agreement with&#10;human judgment. We also identify a previously overlooked weakness in generating&#10;faithful utterances from categorical slots in unseen domains. We show that&#10;prefix tuning applied to T5 generation can address this problem. We further&#10;build an ensemble of prefix-tuning and fine-tuning models that achieves the&#10;lowest SER reported and high SGSAcc on the SGD dataset.&#10;" />
          <attvalue for="2" value="&#10;&#10;Task-oriented dialogue response generation aims to generate accurate and fluent utterances from triplets of intent, slot, and values known as dialogue actions (See Fig.\ref{fig:SGSAcc procedure}). Ensuring that the generated utterances faithfully realize dialogue actions is crucial because misinformation can be costly in real-life applications. However, the lack of a complete, automatic faithfulness metric for task-oriented dialogue NLG has made assessing faithfulness difficult. Slot Error Rate (SER)~\cite{luong-etal-2015-effective}, the most widely used faithfulness metric currently, can only assess utterances generated from non-categorical slots &#10;whose values are expected to be reproduced exactly in generations by string matching, omitting utterances generated from categorical slots such as ``(kids\_friendly, True)''. However, dialogue actions with categorical slots are present in 28\% of the test instances in the Schema Guided Dialogue (SGD) dataset~\cite{DBLP:conf/aaai/RastogiZSGK20}, which is the largest dataset for multi-domain task-oriented dialogue system so far. Therefore, it is essential to include them for complete faithfulness evaluation. &#10;&#10;To cover categorical slots in evaluation, we propose Schema-Guided Semantic Accuracy (SGSAcc) that examines semantic consistency rather than string overlap. We build upon Semantic Accuracy~\cite{DBLP:conf/inlg/DusekK20}, which evaluates faithfulness in table-to-text tasks by recognizing textual entailment (RTE). A natural language inference (NLI) model is used to check whether the premise (generated utterances) entails, contradicts, or is neutral to the hypothesis (dialogue actions).&#10;Following their design, we first convert the dialogue actions into fluent sentences to serve as the hypothesis, so that the NLI model trained on free-running texts can perform RTE correctly without further fine-tuning. We name the converted sentences entailment reference to emphasize their role in entailment checking.&#10;&#10;We find that the original Semantic Accuracy cannot be directly applied on task-oriented response generation because it requires handcrafted templates for each dialogue action to produce entailment references. Although \cite{DBLP:conf/emnlp/KaleR20} have published templates for the SGD, they were designed for generation rather than evaluation. We find that Semantic Accuracy using these templates marks 25\% of the ground-truth utterances as unfaithful. To cover the 45 services in the SGD and the 225 service variations in the SGD-X without prohibitive labor, we propose a rule-based algorithm that constructs entailment references based on slot descriptions from service schema, which are provided in the SGD~\cite{DBLP:conf/emnlp/KaleR20} and other popular task-oriented dialogue datasets such as MultiWOZ 2.2~\cite{DBLP:journals/corr/abs-2007-12720}. We were able to design the rules efficiently within 20 working hours for the SGD (Appendix \ref{app:rule construction}). In addition, to help resolve co-references, which are prevalent in dialogues, we augment the premise (generation to be assessed) with previous dialogue turns and slot descriptions when needed. &#10;We verified that SGSAcc has good agreement with human judgments of faithfulness.&#10;&#10;We applied SGSAcc to evaluate the best-performing published model in terms of SER on the SGD. We found a previously overlooked weakness in generating faithful utterances in domains not seen in training. To address this, we experimented with prefix-tuning (PT) \cite{DBLP:conf/acl/LiL20} which was reported to generalize better than fine-tuning (FT) on unseen data \cite{DBLP:conf/acl/LiL20, DBLP:journals/corr/abs-2110-08329}. We found that PT significantly improved SGSAcc in unseen domains, whereas the FT model achieved lower SER in comparison. Noting their complementary advantages, we used SGSAcc to implement a fidelity reranker similar to that from \cite{DBLP:conf/coling/HarkousGS20} to select faithful generations from an ensemble of PT and FT models, which further improved SER and SGSAcc on the SGD.&#10;&#10;Our contributions are summarised as follows:&#10;&#10;(1) We propose SGSAcc, a faithfulness metric tailored to task-oriented dialogue systems that evaluate both categorical and non-categorical slots. &#10;&#10;(2) We empirically show that prefix-tuning significantly improves faithfulness in unseen domains on the Schema Guided Dialogue dataset. &#10;&#10;(3) We build an ensemble of fine-tuning and prefix-tuning models using a SGSAcc-powered fidelity reranker, which significantly improves faithfulness in the NLG task of the SGD dataset.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Semantic Accuracy Evaluation, Faithful Utterance Generation, Natural Language Processing, Dialogue Response Generation" />
        </attvalues>
      </node>
      <node id="2007.12720" label="2007.12720">
        <attvalues>
          <attvalue for="0" value="MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections&#10;  and State Tracking Baselines" />
          <attvalue for="1" value="  MultiWOZ is a well-known task-oriented dialogue dataset containing over&#10;10,000 annotated dialogues spanning 8 domains. It is extensively used as a&#10;benchmark for dialogue state tracking. However, recent works have reported&#10;presence of substantial noise in the dialogue state annotations. MultiWOZ 2.1&#10;identified and fixed many of these erroneous annotations and user utterances,&#10;resulting in an improved version of this dataset. This work introduces MultiWOZ&#10;2.2, which is a yet another improved version of this dataset. Firstly, we&#10;identify and fix dialogue state annotation errors across 17.3% of the&#10;utterances on top of MultiWOZ 2.1. Secondly, we redefine the ontology by&#10;disallowing vocabularies of slots with a large number of possible values (e.g.,&#10;restaurant name, time of booking). In addition, we introduce slot span&#10;annotations for these slots to standardize them across recent models, which&#10;previously used custom string matching heuristics to generate them. We also&#10;benchmark a few state of the art dialogue state tracking models on the&#10;corrected dataset to facilitate comparison for future work. In the end, we&#10;discuss best practices for dialogue data collection that can help avoid&#10;annotation errors.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.06577" label="2004.06577">
        <attvalues>
          <attvalue for="0" value="Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation&#10;  with Semantic Fidelity" />
          <attvalue for="1" value="  End-to-end neural data-to-text (D2T) generation has recently emerged as an&#10;alternative to pipeline-based architectures. However, it has faced challenges&#10;in generalizing to new domains and generating semantically consistent text. In&#10;this work, we present DataTuner, a neural, end-to-end data-to-text generation&#10;system that makes minimal assumptions about the data representation and the&#10;target domain. We take a two-stage generation-reranking approach, combining a&#10;fine-tuned language model with a semantic fidelity classifier. Each of our&#10;components is learnt end-to-end without the need for dataset-specific&#10;heuristics, entity delexicalization, or post-processing. We show that DataTuner&#10;achieves state of the art results on the automated metrics across four major&#10;D2T datasets (LDC2017T10, WebNLG, ViGGO, and Cleaned E2E), with a fluency&#10;assessed by human annotators nearing or exceeding the human-written reference&#10;texts. We further demonstrate that the model-based semantic fidelity scorer in&#10;DataTuner is a better assessment tool compared to traditional, heuristic-based&#10;measures. Our generated text has a significantly better semantic fidelity than&#10;the state of the art across all four datasets&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2209.09746" label="2209.09746">
        <attvalues>
          <attvalue for="0" value="Target-Guided Open-Domain Conversation Planning" />
          <attvalue for="1" value="  Prior studies addressing target-oriented conversational tasks lack a crucial&#10;notion that has been intensively studied in the context of goal-oriented&#10;artificial intelligence agents, namely, planning. In this study, we propose the&#10;task of Target-Guided Open-Domain Conversation Planning (TGCP) task to evaluate&#10;whether neural conversational agents have goal-oriented conversation planning&#10;abilities. Using the TGCP task, we investigate the conversation planning&#10;abilities of existing retrieval models and recent strong generative models. The&#10;experimental results reveal the challenges facing current technology.&#10;" />
          <attvalue for="2" value="&#10;Neural conversational agents have achieved great successes in recent years, and various methods have been proposed to generate informative responses, e.g., the use of knowledge~\cite{Zhao:emnlp2020:KnowledgeGrounded, Wu:acl2020:DiversInformativeDialogue}, personality~\cite{Li:acl2016:PersonabasedNeuralConv,zhang:acl2018:Personachat}, emotional considerations~\cite{Rashkin:acl2019:EmpatheticDialogue, Zhong:emnlp2019:PersonaEmpathetic}, and large-scale models~\cite{Zhang:acl2020:DIALOGPT, Adiwardana:arxiv2020:Meena, Roller:eacl2021:Blender, Thoppilan:arxiv2022:LaMDA}.&#10;One hot topic in this research area is to develop proactive behavior in agents.&#10;For example, \cite{Tang:acl2019:TargetGuided} proposed the task of Target-Guided Open-Domain Conversation, in which an agent is required to actively lead a conversation to a predefined target word.&#10;\cite{Wu:acl2019:ProactiveConv} proposed a task that uses a knowledge graph to actively lead a conversation to a target entity.&#10;Several studies have implemented these target-oriented task settings~\cite{Dai:arxiv2019:MultipleGenerative, Qin:aaai2020:DynamicKnowledge, Yuan:iccai2020:MultihopMemory, Zhong:aaai2021:KeywordGuided, Zhu:sigir2021:ProactiveRetrieval}.&#10;However, these prior studies all lack planning, a crucial notion that has been intensively studied in the context of goal-oriented artificial intelligence (AI) agents~\cite[etc.]{NorvigAndRussell:book1995:AIModernApproach, KuijpersAndDockx:ai1998:man-machine-ai-planning, Stent:acl2004:TrainableSentencePlanning, Walker:jair2007:IndividualAndDomain} and has also been introduced in neural conversational agents~\cite{Botea:deep-dial2019:AutomatedPlanning,Jiang:aaai2019:GeneralPlanningBasedFramework,Jiang:sigdial2019:Lookahead}.&#10;In other words, these studies do not explicitly consider the generation of a multiple-step plan to achieve a target.&#10;&#10;Given this background, in this study, we propose the Target-Guided Open-Domain Conversation Planning (henceforth, TGCP) task such that an agent's planning ability in goal-oriented conversations can be assessed.&#10;The TGCP task is to produce a plan that leads a conversation to a given target, as illustrated in Figure~\ref{fig:task_overview}.&#10;The point is to consider the task of producing a conversation plan for several utterances ahead, which we first address in the aforementioned context of Target-Guided Open-Domain Conversation. &#10;Furthermore, we also propose modeling the planning process by simulating the user's succeeding utterances using the model of the agent itself; namely, the agent converses with itself (i.e., self-conversation) to search for potential conversation paths that achieve the goal.&#10;This task setting is not the same as a real-world setting, in which an agent is required to plan a conversation while uncertain of the user's future utterances. &#10;However, planning in the self-conversation setting can be considered a prerequisite capability for a planning-aware goal-oriented conversational agent. &#10;TGCP works as a framework to evaluate an agents' prerequisite ability for conversation planning without employing human subjects; this can abstract away the hard-to-control human factors from experiments (e.g., some human subjects may not be as cooperative as others).&#10;&#10;This paper has three major contributions:&#10;(1) We propose the TGCP task as a framework to assess the prerequisite ability of a model for goal-oriented conversation planning.&#10;(2) We conduct a set of experiments on the TGCP framework using several existing retrieval-based neural models and recently proposed strong generative neural models of conversational agents. &#10;(3) Our experimental results reveal the challenges facing current technology.&#10;The evaluation codes and the test set used in the experiments are available.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Goal-Oriented Planning, Computer Science, Linguistics, Cognitive Science, Conversational AI, Artificial Intelligence, Neural Language Models, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1811.00207" label="1811.00207">
        <attvalues>
          <attvalue for="0" value="Towards Empathetic Open-domain Conversation Models: a New Benchmark and&#10;  Dataset" />
          <attvalue for="1" value="  One challenge for dialogue agents is recognizing feelings in the conversation&#10;partner and replying accordingly, a key communicative skill. While it is&#10;straightforward for humans to recognize and acknowledge others' feelings in a&#10;conversation, this is a significant challenge for AI systems due to the paucity&#10;of suitable publicly-available datasets for training and evaluation. This work&#10;proposes a new benchmark for empathetic dialogue generation and&#10;EmpatheticDialogues, a novel dataset of 25k conversations grounded in emotional&#10;situations. Our experiments indicate that dialogue models that use our dataset&#10;are perceived to be more empathetic by human evaluators, compared to models&#10;merely trained on large-scale Internet conversation data. We also present&#10;empirical comparisons of dialogue model adaptations for empathetic responding,&#10;leveraging existing models or datasets without requiring lengthy re-training of&#10;the full model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1905.11553" label="1905.11553">
        <attvalues>
          <attvalue for="0" value="Target-Guided Open-Domain Conversation" />
          <attvalue for="1" value="  Many real-world open-domain conversation applications have specific goals to&#10;achieve during open-ended chats, such as recommendation, psychotherapy,&#10;education, etc. We study the problem of imposing conversational goals on&#10;open-domain chat agents. In particular, we want a conversational system to chat&#10;naturally with human and proactively guide the conversation to a designated&#10;target subject. The problem is challenging as no public data is available for&#10;learning such a target-guided strategy. We propose a structured approach that&#10;introduces coarse-grained keywords to control the intended content of system&#10;responses. We then attain smooth conversation transition through turn-level&#10;supervised learning, and drive the conversation towards the target with&#10;discourse-level constraints. We further derive a keyword-augmented conversation&#10;dataset for the study. Quantitative and human evaluations show our system can&#10;produce meaningful and effective conversations, significantly improving over&#10;other approaches.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.00771" label="1902.00771">
        <attvalues>
          <attvalue for="0" value="Generating Dialogue Agents via Automated Planning" />
          <attvalue for="1" value="  Dialogue systems have many applications such as customer support or question&#10;answering. Typically they have been limited to shallow single turn&#10;interactions. However more advanced applications such as career coaching or&#10;planning a trip require a much more complex multi-turn dialogue. Current&#10;limitations of conversational systems have made it difficult to support&#10;applications that require personalization, customization and context dependent&#10;interactions. We tackle this challenging problem by using domain-independent AI&#10;planning to automatically create dialogue plans, customized to guide a dialogue&#10;towards achieving a given goal. The input includes a library of atomic dialogue&#10;actions, an initial state of the dialogue, and a goal. Dialogue plans are&#10;plugged into a dialogue system capable to orchestrate their execution. Use&#10;cases demonstrate the viability of the approach. Our work on dialogue planning&#10;has been integrated into a product, and it is in the process of being deployed&#10;into another.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.00110" label="2103.00110">
        <attvalues>
          <attvalue for="0" value="MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network" />
          <attvalue for="1" value="  Mean opinion score (MOS) is a popular subjective metric to assess the quality&#10;of synthesized speech, and usually involves multiple human judges to evaluate&#10;each speech utterance. To reduce the labor cost in MOS test, multiple methods&#10;have been proposed to automatically predict MOS scores. To our knowledge, for a&#10;speech utterance, all previous works only used the average of multiple scores&#10;from different judges as the training target and discarded the score of each&#10;individual judge, which did not well exploit the precious MOS training data. In&#10;this paper, we propose MBNet, a MOS predictor with a mean subnet and a bias&#10;subnet to better utilize every judge score in MOS datasets, where the mean&#10;subnet is used to predict the mean score of each utterance similar to that in&#10;previous works, and the bias subnet to predict the bias score (the difference&#10;between the mean score and each individual judge score) and capture the&#10;personal preference of individual judges. Experiments show that compared with&#10;MOSNet baseline that only leverages mean score for training, MBNet improves the&#10;system-level spearmans rank correlation co-efficient (SRCC) by 2.9% on VCC 2018&#10;dataset and 6.7% on VCC 2016 dataset.&#10;" />
          <attvalue for="2" value="&#10;&#10;Speech quality assessment \cite{sqa} aims to measure the quality of synthesized speech and has long been a challenge in speech synthesis areas such as text to speech~\cite{tacotron2,fastspeech,ren2021fastspeech} and voice conversion~\cite{cycleganvc,autovc}. Existing objective metrics \cite{MCD,PESQ,ANIQUE,P563} can only reflect speech quality from some certain aspects. This motivates people to use subjective metrics, for instance MOS (mean opinion score) \cite{ana_vcc2016}, to give an overall human perception score. In a MOS test, multiple judges rate a discrete score between 1 and 5 for a given speech (the higher score, the better speech quality). The arithmetic mean of all judge scores of an utterance is the utterance-level MOS. The mean of all utterance-level MOS of a speech-synthesis system is called system-level MOS.&#10;&#10;MOS test is expensive and time-consuming due to the involvement of human judges. To solve this problem, many methods have been proposed to automatically predict the MOS of an utterance: AutoMOS \cite{AutoMOS} predicted MOS with a long short term memory (LSTM) network. Quality-Net \cite{QualityNet} utilized frame-level quality constraint to stabilize the training procedure. With comparative experiments on several architectures, MOSNet \cite{mosnet} showed that CNN-BLSTM was a better architecture for MOS prediction. Based on MOSNet, Williams et al. \cite{williams2020} compared different speech representations for MOS prediction. &#10;Choi et al. \cite{choi2020neural} incorporated MOSNet with multi-task learning to improve performance. In \cite{choi2020deep}, global quality token and encoding layer were utilized to achieve better prediction accuracy.&#10;&#10;Although many methods have been proposed to train the MOS prediction model and improve the prediction accuracy, there exists two important problems when dealing with the MOS training data:&#10;\begin{itemize}&#10;\item To the best of our knowledge, all the previous works ~\cite{QualityNet,mosnet,choi2020neural,choi2020deep} leveraged the utterance-level MOS as the training target, and discarded the detailed judge scores. Utterance-level MOS is simply an average score of multiple judge scores.&#10;If we can leverage the individual score of each judge, the training data will be several times (depending on the number of judges on each speech) larger, which will be a large benefit due to the scarce of original training data. However, it is challenging to leverage these scores since human judgements are of high variance, which makes the model training unstable.&#10;\item Utterance-level MOS is biased due to personal preference. In a MOS dataset, we always allocate more judges than the required number of judgements for each utterance, which can reduce the labor of each judge to ensure the judgement quality. However, different judges might have different rating standards due to the lack of quantitative criterion. As a result, different utterances may be rated by different sets of judges and thus receive incomparable utterance-level MOS. Directly training with the MOS data without any debiasing process, as adopted by many previous works, will hurt the model performance.&#10;\end{itemize}&#10;&#10;To address the above challenges, in this work, we propose MBNet, a MOS prediction model with a mean subnet and a bias subnet structure to leverage the individual judge scores of each utterance and handle the biased scores due to personal preference. Specifically, we design a mean subnet to predict the mean score similar to that in previous works, and a bias subnet to predict the bias score (the difference between the mean score and each judge score). Bias subnet can not only leverage all individual judge scores in dataset, but also capture the personal preference of each judge. The output of bias subnet is added with that of mean subnet to form the final prediction of individual judge score, which can force bias subnet to learn the bias of each judge only and thus stabilize the training. We further proposed clipped mean squared error loss to prevent model from over-fitting to the exact label, which might be incomparable. Experiments on VCC 2018 and VCC 2016 datasets show that MBNet significantly improves the prediction accuracy compared with previous MOS prediction systems including MOSNet and its variants \cite{mosnet,choi2020neural,choi2020deep}. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Human-Computer Interaction, Machine Learning, Computer Science, Speech Processing, Linguistics, Human Evaluation Metrics, Speech Synthesis, Predictive Modeling, Statistics" />
        </attvalues>
      </node>
      <node id="1808.05344" label="1808.05344">
        <attvalues>
          <attvalue for="0" value="Quality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model&#10;  based on BLSTM" />
          <attvalue for="1" value="  Nowadays, most of the objective speech quality assessment tools (e.g.,&#10;perceptual evaluation of speech quality (PESQ)) are based on the comparison of&#10;the degraded/processed speech with its clean counterpart. The need of a&#10;&quot;golden&quot; reference considerably restricts the practicality of such assessment&#10;tools in real-world scenarios since the clean reference usually cannot be&#10;accessed. On the other hand, human beings can readily evaluate the speech&#10;quality without any reference (e.g., mean opinion score (MOS) tests), implying&#10;the existence of an objective and non-intrusive (no clean reference needed)&#10;quality assessment mechanism. In this study, we propose a novel end-to-end,&#10;non-intrusive speech quality evaluation model, termed Quality-Net, based on&#10;bidirectional long short-term memory. The evaluation of utterance-level quality&#10;in Quality-Net is based on the frame-level assessment. Frame constraints and&#10;sensible initializations of forget gate biases are applied to learn meaningful&#10;frame-level quality assessment from the utterance-level quality label.&#10;Experimental results show that Quality-Net can yield high correlation to PESQ&#10;(0.9 for the noisy speech and 0.84 for the speech processed by speech&#10;enhancement). We believe that Quality-Net has potential to be used in a wide&#10;variety of applications of speech signal processing.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.08267" label="2007.08267">
        <attvalues>
          <attvalue for="0" value="Neural MOS Prediction for Synthesized Speech Using Multi-Task Learning&#10;  With Spoofing Detection and Spoofing Type Classification" />
          <attvalue for="1" value="  Several studies have proposed deep-learning-based models to predict the mean&#10;opinion score (MOS) of synthesized speech, showing the possibility of replacing&#10;human raters. However, inter- and intra-rater variability in MOSs makes it hard&#10;to ensure the high performance of the models. In this paper, we propose a&#10;multi-task learning (MTL) method to improve the performance of a MOS prediction&#10;model using the following two auxiliary tasks: spoofing detection (SD) and&#10;spoofing type classification (STC). Besides, we use the focal loss to maximize&#10;the synergy between SD and STC for MOS prediction. Experiments using the MOS&#10;evaluation results of the Voice Conversion Challenge 2018 show that proposed&#10;MTL with two auxiliary tasks improves MOS prediction. Our proposed model&#10;achieves up to 11.6% relative improvement in performance over the baseline&#10;model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.03191" label="2005.03191">
        <attvalues>
          <attvalue for="0" value="ContextNet: Improving Convolutional Neural Networks for Automatic Speech&#10;  Recognition with Global Context" />
          <attvalue for="1" value="  Convolutional neural networks (CNN) have shown promising results for&#10;end-to-end speech recognition, albeit still behind other state-of-the-art&#10;methods in performance. In this paper, we study how to bridge this gap and go&#10;beyond with a novel CNN-RNN-transducer architecture, which we call ContextNet.&#10;ContextNet features a fully convolutional encoder that incorporates global&#10;context information into convolution layers by adding squeeze-and-excitation&#10;modules. In addition, we propose a simple scaling method that scales the widths&#10;of ContextNet that achieves good trade-off between computation and accuracy. We&#10;demonstrate that on the widely used LibriSpeech benchmark, ContextNet achieves&#10;a word error rate (WER) of 2.1%/4.6% without external language model (LM),&#10;1.9%/4.1% with LM and 2.9%/7.0% with only 10M parameters on the clean/noisy&#10;LibriSpeech test sets. This compares to the previous best published system of&#10;2.0%/4.6% with LM and 3.9%/11.3% with 20M parameters. The superiority of the&#10;proposed ContextNet model is also verified on a much larger internal dataset.&#10;" />
          <attvalue for="2" value="&#10;&#10;Convolution Neural Network (CNN) based models for end-to-end (E2E) speech recognition is attracting an increasing amount of attention~\cite{zhang2017very,zeghidour2018fully,li2019jasper,kriman2019quartznet}. Among them, the Jasper model~\cite{li2019jasper} recently achieves close to the state-of-the-art word error rate (WER) 2.95\% on LibriSpeech test-clean~\cite{panayotov2015librispeech} with an external neural language model. The main feature of the Jasper model is a deep convolution based encoder with stacked layers of 1D convolutions and skip connections. Depthwise separable convolutions~\cite{chollet2017xception} have been utilized to further increase the speed and accuracy of CNN models~\cite{hannun2019sequence,kriman2019quartznet}. The key advantage of a CNN based model is its parameter efficiency; however, the WER achieved by the best CNN model, QuartzNet~\cite{kriman2019quartznet}, is still behind the RNN/transformer based models~\cite{largespecaugment,karita2019comparative,wang2019transformer,zhang2020transformer}. &#10;&#10;A major difference between the RNN/Transformer~\cite{karita2019comparative, wang2019transformer, zhang2020transformer} based models and a CNN model is the length of the context. In a bidirectional RNN model, a cell in theory has access to the information of the whole sequence; in a Transformer model, the attention mechanism explicitly allows the nodes at two distant time stamps to attend each other. However, a naive convolution with a limited kernel size only covers a small window in the time domain; hence the context is small and the global information is not incorporated. In this paper, we argue that the lack of global context is the main cause of the gap of WER between the CNN based ASR model and the RNN/Transformer based models. &#10;&#10;To enhance the global context in the CNN model, we draw inspirations from the squeeze-and-excitation (SE) layer introduced in~\cite{hu2018squeeze}, and propose a novel CNN model for ASR, which we call ContextNet. An SE layer squeezes a sequence of local feature vectors into a single global context vector, broadcasts this context back to each local feature vector, and merges the two via multiplications. When we place an SE layer after a naive convolution layer, we grant the convolution output the access to global information. Empirically, we observe that adding squeeze-and-excitation layers to ContextNet~introduces the most reduction in the WER on LibriSpeech test-other.&#10;&#10;Previous works on hybrid ASR have successfully introduced the context to acoustic models by either stacking a large number of layers, or having a separately trained global vector to represent the speaker and the environment information \cite{peddinti2015jhu,xue2014fast,karafiat2011ivector,saon2013speaker}. In \cite{sailor2019unsupervised}, SE has been adopted to RNN for unsupervised adaptation. In this paper, we show that SE can also be effective for CNN encoders.&#10;&#10;The architecture of ContextNet is also inspired by the design choices of QuartzNet~\cite{kriman2019quartznet}, such as the usage of depthwise separable 1D convolution in the encoder. However, there are some key differences in the architectures in addition to the incorporation of the SE layer. For instance, we use a RNN-T decoder~\cite{graves2012sequence, rao2017exploring, he2019, tara2020} instead of the CTC decoder~\cite{graves2006connectionist}. Moreover, we use the Swish activation function~\cite{ramachandran2017searching}, which contributes a slight but consistent reduction in WER. Overall, ContextNet achieves the WER of 1.9\%/4.1\% on LibriSpeech test-clean/test-other. This is a big improvement over previous CNN based architectures such as QuartzNet\cite{kriman2019quartznet}, and it outperforms transformer and LSTM based models~\cite{zhang2020transformer, wang2019transformer, zeyer2019comparison, karita2019comparative, park2019specaugment}.&#10;&#10;This paper also studies how to reduce the computation cost of ContextNet for faster training and inference. First, we adopt a progressive downsampling scheme that is commonly used in vision models. Specifically, we progressively reduce the length of the encoded sequence eight times, significantly lower the computation while maintaining the encoder's representation power and the overall model accuracy. As a benefit, this downsampling scheme allows us to reduce the kernel size of all the convolution layers to five without significantly reducing the effective receptive field of an encoder output node.&#10;&#10;We can scale ContextNet by globally changing the number of channels in convolutional filters. Figure~\ref{overall} illustrates the trade-off of {ContextNet} between model size and WER, as well as its comparison against other methods. Clearly, our scaled model achieves the best trade-offs among all.&#10;&#10;In summary, the main contributions of this paper are: (1) an improved CNN architecture with global context for ASR, (2) a progressive downsampling and model scaling scheme to achieve superior accuracy and model size trade-off.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Artificial Intelligence, Signal Processing, Speech Recognition, Deep Learning Optimization" />
        </attvalues>
      </node>
      <node id="1910.10261" label="1910.10261">
        <attvalues>
          <attvalue for="0" value="QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel&#10;  Separable Convolutions" />
          <attvalue for="1" value="  We propose a new end-to-end neural acoustic model for automatic speech&#10;recognition. The model is composed of multiple blocks with residual connections&#10;between them. Each block consists of one or more modules with 1D time-channel&#10;separable convolutional layers, batch normalization, and ReLU layers. It is&#10;trained with CTC loss. The proposed network achieves near state-of-the-art&#10;accuracy on LibriSpeech and Wall Street Journal, while having fewer parameters&#10;than all competing models. We also demonstrate that this model can be&#10;effectively fine-tuned on new datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1610.02357" label="1610.02357">
        <attvalues>
          <attvalue for="0" value="Xception: Deep Learning with Depthwise Separable Convolutions" />
          <attvalue for="1" value="  We present an interpretation of Inception modules in convolutional neural&#10;networks as being an intermediate step in-between regular convolution and the&#10;depthwise separable convolution operation (a depthwise convolution followed by&#10;a pointwise convolution). In this light, a depthwise separable convolution can&#10;be understood as an Inception module with a maximally large number of towers.&#10;This observation leads us to propose a novel deep convolutional neural network&#10;architecture inspired by Inception, where Inception modules have been replaced&#10;with depthwise separable convolutions. We show that this architecture, dubbed&#10;Xception, slightly outperforms Inception V3 on the ImageNet dataset (which&#10;Inception V3 was designed for), and significantly outperforms Inception V3 on a&#10;larger image classification dataset comprising 350 million images and 17,000&#10;classes. Since the Xception architecture has the same number of parameters as&#10;Inception V3, the performance gains are not due to increased capacity but&#10;rather to a more efficient use of model parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1912.05533" label="1912.05533">
        <attvalues>
          <attvalue for="0" value="SpecAugment on Large Scale Datasets" />
          <attvalue for="1" value="  Recently, SpecAugment, an augmentation scheme for automatic speech&#10;recognition that acts directly on the spectrogram of input utterances, has&#10;shown to be highly effective in enhancing the performance of end-to-end&#10;networks on public datasets. In this paper, we demonstrate its effectiveness on&#10;tasks with large scale datasets by investigating its application to the Google&#10;Multidomain Dataset (Narayanan et al., 2018). We achieve improvement across all&#10;test domains by mixing raw training data augmented with SpecAugment and&#10;noise-perturbed training data when training the acoustic model. We also&#10;introduce a modification of SpecAugment that adapts the time mask size and/or&#10;multiplicity depending on the length of the utterance, which can potentially&#10;benefit large scale tasks. By using adaptive masking, we are able to further&#10;improve the performance of the Listen, Attend and Spell model on LibriSpeech to&#10;2.2% WER on test-clean and 5.2% WER on test-other.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1709.01507" label="1709.01507">
        <attvalues>
          <attvalue for="0" value="Squeeze-and-Excitation Networks" />
          <attvalue for="1" value="  The central building block of convolutional neural networks (CNNs) is the&#10;convolution operator, which enables networks to construct informative features&#10;by fusing both spatial and channel-wise information within local receptive&#10;fields at each layer. A broad range of prior research has investigated the&#10;spatial component of this relationship, seeking to strengthen the&#10;representational power of a CNN by enhancing the quality of spatial encodings&#10;throughout its feature hierarchy. In this work, we focus instead on the channel&#10;relationship and propose a novel architectural unit, which we term the&#10;&quot;Squeeze-and-Excitation&quot; (SE) block, that adaptively recalibrates channel-wise&#10;feature responses by explicitly modelling interdependencies between channels.&#10;We show that these blocks can be stacked together to form SENet architectures&#10;that generalise extremely effectively across different datasets. We further&#10;demonstrate that SE blocks bring significant improvements in performance for&#10;existing state-of-the-art CNNs at slight additional computational cost.&#10;Squeeze-and-Excitation Networks formed the foundation of our ILSVRC 2017&#10;classification submission which won first place and reduced the top-5 error to&#10;2.251%, surpassing the winning entry of 2016 by a relative improvement of ~25%.&#10;Models and code are available at https://github.com/hujie-frank/SENet.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.05941" label="1710.05941">
        <attvalues>
          <attvalue for="0" value="Searching for Activation Functions" />
          <attvalue for="1" value="  The choice of activation functions in deep networks has a significant effect&#10;on the training dynamics and task performance. Currently, the most successful&#10;and widely-used activation function is the Rectified Linear Unit (ReLU).&#10;Although various hand-designed alternatives to ReLU have been proposed, none&#10;have managed to replace it due to inconsistent gains. In this work, we propose&#10;to leverage automatic search techniques to discover new activation functions.&#10;Using a combination of exhaustive and reinforcement learning-based search, we&#10;discover multiple novel activation functions. We verify the effectiveness of&#10;the searches by conducting an empirical evaluation with the best discovered&#10;activation function. Our experiments show that the best discovered activation&#10;function, $f(x) = x \cdot \text{sigmoid}(\beta x)$, which we name Swish, tends&#10;to work better than ReLU on deeper models across a number of challenging&#10;datasets. For example, simply replacing ReLUs with Swish units improves top-1&#10;classification accuracy on ImageNet by 0.9\% for Mobile NASNet-A and 0.6\% for&#10;Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it&#10;easy for practitioners to replace ReLUs with Swish units in any neural network.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.10984" label="2008.10984">
        <attvalues>
          <attvalue for="0" value="End-to-End Neural Transformer Based Spoken Language Understanding" />
          <attvalue for="1" value="  Spoken language understanding (SLU) refers to the process of inferring the&#10;semantic information from audio signals. While the neural transformers&#10;consistently deliver the best performance among the state-of-the-art neural&#10;architectures in field of natural language processing (NLP), their merits in a&#10;closely related field, i.e., spoken language understanding (SLU) have not beed&#10;investigated. In this paper, we introduce an end-to-end neural&#10;transformer-based SLU model that can predict the variable-length domain,&#10;intent, and slots vectors embedded in an audio signal with no intermediate&#10;token prediction architecture. This new architecture leverages the&#10;self-attention mechanism by which the audio signal is transformed to various&#10;sub-subspaces allowing to extract the semantic context implied by an utterance.&#10;Our end-to-end transformer SLU predicts the domains, intents and slots in the&#10;Fluent Speech Commands dataset with accuracy equal to 98.1 \%, 99.6 \%, and&#10;99.6 \%, respectively and outperforms the SLU models that leverage a&#10;combination of recurrent and convolutional neural networks by 1.4 \% while the&#10;size of our model is 25\% smaller than that of these architectures.&#10;Additionally, due to independent sub-space projections in the self-attention&#10;layer, the model is highly parallelizable which makes it a good candidate for&#10;on-device SLU.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1908.09936" label="1908.09936">
        <attvalues>
          <attvalue for="0" value="Leveraging External Knowledge for Out-Of-Vocabulary Entity Labeling" />
          <attvalue for="1" value="  Dealing with previously unseen slots is a challenging problem in a real-world&#10;multi-domain dialogue state tracking task. Other approaches rely on predefined&#10;mappings to generate candidate slot keys, as well as their associated values.&#10;This, however, may fail when the key, the value, or both, are not seen during&#10;training. To address this problem we introduce a neural network that leverages&#10;external knowledge bases (KBs) to better classify out-of-vocabulary slot keys&#10;and values. This network projects the slot into an attribute space derived from&#10;the KB, and, by leveraging similarities in this space, we propose candidate&#10;slot keys and values to the dialogue state tracker. We provide extensive&#10;experiments that demonstrate that our stratagem can improve upon a previous&#10;approach, which relies on predefined candidate mappings. In particular, we&#10;evaluate this approach by training a state-of-the-art model with candidates&#10;generated from our network, and obtained relative increases of 57.7% and 82.7%&#10;in F1 score and accuracy, respectively, for the aforementioned model, when&#10;compared to the current candidate generation strategy.&#10;" />
          <attvalue for="2" value="&#10;&#10;In the Dialogue State Tracking task, the Slot Carryover (SC) paradigm \cite{naik2018contextualSC} involves tracking a key-value pair (a slot) throughout the dialogue. At every turn, a decision is made to whether to carry the slot over to the next turn.&#10;The values that each slot can take are established a priori in a hierarchical schema where the slot's value has a key assigned to it, which is itself associated with a domain. This is used to generate the candidates to be tracked, but slot-to-slot mappings can be challenging, as the keys and values could have been unseen during training \cite{xu-hu-2018-end}.&#10;&#10;This mapping relies on context to resolve ambiguity from values and keys. When a slot is mapped across domains, the given key is translated to its new representation on the target domain. See table Table~\ref{tab:slot_example} for an example.&#10;Two insights can be taken from this: first, the meaning of the key is preserved, and is only a change in naming conventions. Second, the key-value relationship could be seen--loosely speaking--as a type of hypernymic property. These semantic properties, along with the challenge presented by out-of-vocabulary (OOV) keys and values, motivates our work on this paper.&#10;&#10;We surmise that it is possible to leverage knowledge bases (KBs) to build a model which automates the slot candidate generation, and circumvents the OOV problem. Our model collects several prospects, and determines the closest match with respect to the given context. We do not make assumptions about the geometry of the input spaces, and we opt instead to learn the metric as a secondary task, through the polynomial approximation to a Bregman divergence \cite{Banerjee2004ClusteringWB}.&#10;&#10;Previous work exists on inferring labels with little to no information provided during training, such as zero and few-shot learning methods \cite{Snell2017PrototypicalNF}, and hybrid approaches such as in \cite{Akata2013LabelEmbeddingFA, yogatama2015embedding, Shi2018MultiContextLE}.&#10;Leveraging the hierarchical nature of a corpus is not a new approach either, as hypernym classification requires some sort of hierarchical assumption, for example, \cite{Murty2018HierarchicalLA, Fu2014LearningSH, Nickel2017PoincarEF}. &#10;Work has also been made on employing KBs to improve learning, as in \cite{Lee2015LeveragingKB,Yang2017LeveragingKB}.&#10;Finally, Bregman-based methods have garnered popularity as both a metric and a learning algorithm. A good example is the work of Banerjee et al., \shortcite{Banerjee2004ClusteringWB} on clustering with Bregman divergences.&#10;&#10;Our work is unique in its approach: it relies on a KB, but places emphasis on the hierarchy of the corpus to learn and measure distances. It leverages hypernyms as an implicitly hierarchical slot key namespace. It is important to note, however, that our problem is unrelated to the hypernym classification task.&#10;&#10;The three contributions of this paper are: first, a model that is able to output slot candidates for a dialogue state tracker (DST), even when the slot's value, key, or both, are OOV. This is done by leveraging KBs to learn the key's attributes. Second, a method to approximate a generalized, hierarchical distance function--the Bregman divergence--without making assumptions about the geometry of the space we are working on. Third, an analysis of the performance of a DST in OOV contexts, under both its current hardcoded mappings, and our model. Our approach increases the performance of the DST by 26.01\% and 51.08\% in $F_1$ and accuracy, respectively, under 100\% OOV settings. &#10;&#10;We start out by describing our model in Section \ref{sec:model}, and then, in Section~\ref{sec:experiments}, we present the results obtained from the evaluation of a DST with the candidates generated by our approach. We conclude in Section~\ref{sec:conclusion} with a discussion of our work.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Dialogue State Tracking, Computer Science, Linguistics, Knowledge Base Integration, Artificial Intelligence, Information Science" />
        </attvalues>
      </node>
      <node id="1806.01773" label="1806.01773">
        <attvalues>
          <attvalue for="0" value="Contextual Slot Carryover for Disparate Schemas" />
          <attvalue for="1" value="  In the slot-filling paradigm, where a user can refer back to slots in the&#10;context during a conversation, the goal of the contextual understanding system&#10;is to resolve the referring expressions to the appropriate slots in the&#10;context. In large-scale multi-domain systems, this presents two challenges -&#10;scaling to a very large and potentially unbounded set of slot values, and&#10;dealing with diverse schemas. We present a neural network architecture that&#10;addresses the slot value scalability challenge by reformulating the contextual&#10;interpretation as a decision to carryover a slot from a set of possible&#10;candidates. To deal with heterogenous schemas, we introduce a simple&#10;data-driven method for trans- forming the candidate slots. Our experiments show&#10;that our approach can scale to multiple domains and provides competitive&#10;results over a strong baseline.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.01555" label="1805.01555">
        <attvalues>
          <attvalue for="0" value="An End-to-end Approach for Handling Unknown Slot Values in Dialogue&#10;  State Tracking" />
          <attvalue for="1" value="  We highlight a practical yet rarely discussed problem in dialogue state&#10;tracking (DST), namely handling unknown slot values. Previous approaches&#10;generally assume predefined candidate lists and thus are not designed to output&#10;unknown values, especially when the spoken language understanding (SLU) module&#10;is absent as in many end-to-end (E2E) systems. We describe in this paper an E2E&#10;architecture based on the pointer network (PtrNet) that can effectively extract&#10;unknown slot values while still obtains state-of-the-art accuracy on the&#10;standard DSTC2 benchmark. We also provide extensive empirical evidence to show&#10;that tracking unknown values can be challenging and our approach can bring&#10;significant improvement with the help of an effective feature dropout&#10;technique.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.05127" label="1807.05127">
        <attvalues>
          <attvalue for="0" value="Hierarchical Losses and New Resources for Fine-grained Entity Typing and&#10;  Linking" />
          <attvalue for="1" value="  Extraction from raw text to a knowledge base of entities and fine-grained&#10;types is often cast as prediction into a flat set of entity and type labels,&#10;neglecting the rich hierarchies over types and entities contained in curated&#10;ontologies. Previous attempts to incorporate hierarchical structure have&#10;yielded little benefit and are restricted to shallow ontologies. This paper&#10;presents new methods using real and complex bilinear mappings for integrating&#10;hierarchical information, yielding substantial improvement over flat&#10;predictions in entity linking and fine-grained entity typing, and achieving new&#10;state-of-the-art results for end-to-end models on the benchmark FIGER dataset.&#10;We also present two new human-annotated datasets containing wide and deep&#10;hierarchies which we will release to the community to encourage further&#10;research in this direction: MedMentions, a collection of PubMed abstracts in&#10;which 246k mentions have been mapped to the massive UMLS ontology; and TypeNet,&#10;which aligns Freebase types with the WordNet hierarchy to obtain nearly 2k&#10;entity types. In experiments on all three datasets we show substantial gains&#10;from hierarchy-aware training.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1902.09091" label="1902.09091">
        <attvalues>
          <attvalue for="0" value="Leveraging Knowledge Bases in LSTMs for Improving Machine Reading" />
          <attvalue for="1" value="  This paper focuses on how to take advantage of external knowledge bases (KBs)&#10;to improve recurrent neural networks for machine reading. Traditional methods&#10;that exploit knowledge from KBs encode knowledge as discrete indicator&#10;features. Not only do these features generalize poorly, but they require&#10;task-specific feature engineering to achieve good performance. We propose&#10;KBLSTM, a novel neural model that leverages continuous representations of KBs&#10;to enhance the learning of recurrent neural networks for machine reading. To&#10;effectively integrate background knowledge with information from the currently&#10;processed text, our model employs an attention mechanism with a sentinel to&#10;adaptively decide whether to attend to background knowledge and which&#10;information from KBs is useful. Experimental results show that our model&#10;achieves accuracies that surpass the previous state-of-the-art results for both&#10;entity extraction and event extraction on the widely used ACE2005 dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.00315" label="2409.00315">
        <attvalues>
          <attvalue for="0" value="An Empirical Study on Context Length for Open-Domain Dialog Generation" />
          <attvalue for="1" value="  Transformer-based open-domain dialog models have become increasingly popular&#10;in recent years. These models typically represent context as a concatenation of&#10;a dialog history. However, there is no criterion to decide how many utterances&#10;should be kept adequate in a context. We try to figure out how the choice of&#10;context length affects the model. We experiment on three questions from coarse&#10;to fine: (i) Does longer context help model training? (ii) Is it necessary to&#10;change the training context length when dealing with dialogs of different&#10;context lengths? (iii) Do different dialog samples have the same preference for&#10;context length? Our experimental results show that context length, an often&#10;overlooked setting, deserves attention when implementing Transformer-based&#10;dialog models.&#10;" />
          <attvalue for="2" value="&#10;Since the advent of Transformer~\cite{10.5555/3295222.3295349}, language models trained on large-scale corpora have dominated the field of machine translation and other NLP tasks, including open-domain dialog generation~\cite{DBLP:journals/corr/abs-1901-08149,zhang2019dialogpt}. Despite the success of Transformer-based dialog models, they were often criticized for not understanding dialog context~\cite{sankar-etal-2019-neural,saleh-etal-2020-probing}, which can lead to generic responses~\cite{li-etal-2016-diversity} or self-contradictions~\cite{kim-etal-2020-will}. For Transformer-based dialog models, context is usually represented as a concatenation of historical utterances. However, there is no uniform standard for deciding how many utterances to keep in a context. For example, Meena~\cite{DeFreitas2020TowardsAH} limited the context to no more than seven utterances, while PLATO~\cite{bao-etal-2020-plato} limited the total length of the context sequence to no more than 256 tokens. We have no idea whether the context length they choose is optimal and how changing the context length would affect the performance of the model.&#10;&#10;In this paper, we focus on the setting of context length in Transformer-based dialog models. We pose three questions about the possible impact of context length on the model: (i) Does longer context help model training? (ii) Is it necessary to change the training context length when dealing with dialogs of different context lengths? (iii) Do different dialog samples have the same preference for context length? Regarding model selection, since we care about the impact of the context length on the model rather than the absolute performance, we take two most basic practices to implement a dialog model: training a Transformer from scratch and fine-tuning a pre-trained GPT2~\cite{radford2019language} model. Although the performance of these two models is not comparable with the current state-of-the-art chatbots, such as ChatGPT, we believe that the study of these classic paradigms can help us better understand and leverage context when designing Transformer-based dialog models.&#10;&#10;Our experimental results are summarized by the following three findings:&#10;\begin{itemize}&#10; \item Considering both performance and efficiency, a longer context is not necessarily better for Transformer-based dialog models.&#10; \item The best-performing models on the entire set perform well on dialogs with varying history lengths, so there is no need to train separate models for dialogs of different lengths.&#10; \item For different dialog samples, the optimal context length at test time is different. Considering a specific context length for each sample during the testing phase further improves model performance.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Artificial Intelligence, Context Length Analysis, Dialog Model Optimization, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="1906.01603" label="1906.01603">
        <attvalues>
          <attvalue for="0" value="Do Neural Dialog Systems Use the Conversation History Effectively? An&#10;  Empirical Study" />
          <attvalue for="1" value="  Neural generative models have been become increasingly popular when building&#10;conversational agents. They offer flexibility, can be easily adapted to new&#10;domains, and require minimal domain engineering. A common criticism of these&#10;systems is that they seldom understand or use the available dialog history&#10;effectively. In this paper, we take an empirical approach to understanding how&#10;these models use the available dialog history by studying the sensitivity of&#10;the models to artificially introduced unnatural changes or perturbations to&#10;their context at test time. We experiment with 10 different types of&#10;perturbations on 4 multi-turn dialog datasets and find that commonly used&#10;neural dialog architectures like recurrent and transformer-based seq2seq models&#10;are rarely sensitive to most perturbations such as missing or reordering&#10;utterances, shuffling words, etc. Also, by open-sourcing our code, we believe&#10;that it will serve as a useful diagnostic tool for evaluating dialog systems in&#10;the future.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.05816" label="2004.05816">
        <attvalues>
          <attvalue for="0" value="Will I Sound Like Me? Improving Persona Consistency in Dialogues through&#10;  Pragmatic Self-Consciousness" />
          <attvalue for="1" value="  We explore the task of improving persona consistency of dialogue agents.&#10;Recent models tackling consistency often train with additional Natural Language&#10;Inference (NLI) labels or attach trained extra modules to the generative agent&#10;for maintaining consistency. However, such additional labels and training can&#10;be demanding. Also, we find even the best-performing persona-based agents are&#10;insensitive to contradictory words. Inspired by social cognition and&#10;pragmatics, we endow existing dialogue agents with public self-consciousness on&#10;the fly through an imaginary listener. Our approach, based on the Rational&#10;Speech Acts framework (Frank and Goodman, 2012), can enforce dialogue agents to&#10;refrain from uttering contradiction. We further extend the framework by&#10;learning the distractor selection, which has been usually done manually or&#10;randomly. Results on Dialogue NLI (Welleck et al., 2019) and PersonaChat (Zhang&#10;et al., 2018) dataset show that our approach reduces contradiction and improves&#10;consistency of existing dialogue models. Moreover, we show that it can be&#10;generalized to improve context-consistency beyond persona in dialogues.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00682" label="2011.00682">
        <attvalues>
          <attvalue for="0" value="Sequence-to-Sequence Networks Learn the Meaning of Reflexive Anaphora" />
          <attvalue for="1" value="  Reflexive anaphora present a challenge for semantic interpretation: their&#10;meaning varies depending on context in a way that appears to require abstract&#10;variables. Past work has raised doubts about the ability of recurrent networks&#10;to meet this challenge. In this paper, we explore this question in the context&#10;of a fragment of English that incorporates the relevant sort of contextual&#10;variability. We consider sequence-to-sequence architectures with recurrent&#10;units and show that such networks are capable of learning semantic&#10;interpretations for reflexive anaphora which generalize to novel antecedents.&#10;We explore the effect of attention mechanisms and different recurrent unit&#10;types on the type of training data that is needed for success as measured in&#10;two ways: how much lexical support is needed to induce an abstract reflexive&#10;meaning (i.e., how many distinct reflexive antecedents must occur during&#10;training) and what contexts must a noun phrase occur in to support&#10;generalization of reflexive interpretation to this noun phrase?&#10;" />
          <attvalue for="2" value="&#10;&#10;Recurrent neural network architectures have demonstrated remarkable success in natural language processing, achieving state of the art performance across an impressive range of tasks ranging from machine translation to semantic parsing to question answering \cite{sutskever2014sequence, gru, bahdanau2016neural}. These tasks demand the use of a wide variety of computational processes and information sources (from grammatical to lexical to world knowledge), and are evaluated in coarse-grained quantitative ways. As a result, it is not an easy matter to identify the specific strengths and weaknesses in a network's solution of a task. &#10;&#10;In this paper, we take a different tack, exploring the degree to which neural networks successfully master one very specific aspect of linguistic knowledge: the interpretation of sentences containing reflexive anaphora. We address this problem in the context of the task of semantic parsing, which we instantiate as mapping a sequence of words into a predicate calculus logical form representation of the sentence's meaning.&#10;\pex&lt;ex:transform&gt;&#10; \a Mary runs $\to$ $\run(\mary)$&#10; \a John sees Bob $\to$ $\see(\john, \bob)$&#10;\xe&#10;Even for simple sentences like those in~(\getref{ex:transform}), which represent the smallest &#10;representations of object reflexives in English, the network must learn lexical semantic &#10;correspondences (e.g., the input symbol \lex{Mary} is mapped to the output $\mary$ and \lex{runs}&#10;is mapped to $\run$) and a mode of composition (e.g., for an intransitive sentence, the meaning of the subject is surrounded by parentheses and appended to the meaning of the verb). &#10;Of course, not all of natural language adheres to such simple formulas. Reflexives, words like {herself} and {himself}, do not have an interpretation that can be assigned independently of the meaning of the surrounding context.&#10;\pex&lt;ex:transform-refl&gt;&#10; \a Mary sees herself $\to \see(\mary, \mary)$&#10; \a Alice sees herself $\to \see(\alice, \alice)$&#10;\xe&#10;In these sentences, the interpretation of the reflexive is not a constant that can be combined with the meaning of the surrounding elements. Rather, a reflexive object must be interpreted as identical to the meaning of verb's subject. Of course, a network could learn a context-sensitive interpretation of a reflexive, so that for any sentence with \lex{Mary} as its subject, the reflexive is interpreted as $\mary$, and with \lex{Alice} as its subject it is interpreted as $\alice$. However, such piecemeal learning of reflexive meaning will not support generalization to sentences involving a subject that has not been encountered as the antecedent of a reflexive during training, even if the interpretation of the subject has occurred elsewhere. What is needed instead is an interpretation of the reflexive that is characterized not as a specific (sequence of) output token(s), but rather as an abstract instruction to duplicate the interpretation of the subject. Such an abstraction requires more than the ``jigsaw puzzle&quot; approach to meaning that simpler sentences afford. &#10;&#10;\cite{Marcus98} argues that this kind of abstraction, which he takes to require the use of algebraic variables to assert identity, is beyond the capacity of recurrent neural networks. &#10;\cite{Marcus98}'s demonstration involves a simple recurrent network (SRN, \cite{elman90}) language model that is trained to predict the next word over a corpus of sentences of the following form:&#10;\pex&#10; \a A rose is a rose.&#10; \a A mountain is a mountain.&#10;\xe&#10;All sentences in this training set have identical subject and object nouns. &#10;\cite{Marcus98} shows, however, that the resulting trained network does not correctly predict the subject noun when tested with a novel preamble `\lex{A book is a $\ldots$}'. Though intriguing, this demonstration is not entirely convincing: since the noun occurring in the novel preamble, \lex{book} in our example, did not occur in the training data, there is no way that the network could possibly have known which (one-hot represented) output should correspond to the reflexive for a sentence containing the novel (one-hot represented) subject noun, even if the network did successfully encode an identity relation between subject and object. &#10;&#10;\cite{frank2013} explore a related task in the context of SRN interpretation of reflexives. In their experiments, SRNs were trained to map input words to corresponding semantic symbols that are output on the same time step in which a word is presented. For most words in the vocabulary, this is a simple task: the desired output is a constant function of the input (\lex{Mary} corresponds to $\mary$, \lex{sees} to see, etc.). For reflexives however, the target output depends on the subject that occurs earlier in the sentence. \cite{frank2013}\ tested the network's ability to interpret a reflexive in sentences containing a subject that had not occurred as a reflexive's antecedent during training. However, unlike Marcus' task, this subject and its corresponding semantic symbol did occur in other (non-reflexive) contexts in the training data, and therefore was in the realm of possible inputs and outputs for the network. Nonetheless, none of the SRNs that they trained succeeded at this task for even a single test example.&#10;&#10;Since those experiments were conducted, substantial advances have been made on recurrent neural network architectures, some of which have been crucial in the success of practical NLP systems. &#10;\begin{itemize}&#10; \itemRecurrent units: More sophisticated recurrent units like LSTMs \cite{lstm} and GRUs \cite{gru} have been shown to better encode preceding context than SRNs.&#10; \item Sequence-to-Sequence architectures: The performance of network models that transduce one string to another, used in machine translation and semantic parsing, has been greatly improved by the use of independent encoder and decoder networks \cite{sutskever2014sequence}.&#10; \item Attention mechanism: The ability of a network to produce contextually appropriate outputs even in the context of novel vocabulary items has been facilitated by content-sensitive attention mechanisms \cite{bahdanau2016neural, luong-etal-2015-effective}. &#10;\end{itemize}&#10;These innovations open up the possibility that modern network architectures may well be able to solve the variable identity problem necessary for mapping reflexive sentences to their logical form. In the experiments we describe below, we explore whether this is the case. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Computational Semantics, Linguistics, Cognitive Science, Recurrent Neural Networks, Artificial Intelligence, Natural Language Processing, Semantic Interpretation" />
        </attvalues>
      </node>
      <node id="2101.03453" label="2101.03453">
        <attvalues>
          <attvalue for="0" value="BERT &amp; Family Eat Word Salad: Experiments with Text Understanding" />
          <attvalue for="1" value="  In this paper, we study the response of large models from the BERT family to&#10;incoherent inputs that should confuse any model that claims to understand&#10;natural language. We define simple heuristics to construct such examples. Our&#10;experiments show that state-of-the-art models consistently fail to recognize&#10;them as ill-formed, and instead produce high confidence predictions on them. As&#10;a consequence of this phenomenon, models trained on sentences with randomly&#10;permuted word order perform close to state-of-the-art models. To alleviate&#10;these issues, we show that if models are explicitly trained to recognize&#10;invalid inputs, they can be robust to such attacks without a drop in&#10;performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;The BERT family of models~\cite[and others]{devlin2019bert,liu2019roberta}&#10;form the backbone of today's NLP systems. At the time of writing, all eleven systems deemed to outperform humans in the GLUE benchmark suite~\cite{wang2018glue} belong to this family. Do these models understand language? Recent work suggests otherwise. For example, \cite{bender2020climbing} point out that models trained to mimic linguistic form (i.e., language models) may be deficient in understanding the meaning conveyed by language.&#10;&#10;In this paper, we show that such models struggle even with the form of language by demonstrating that they force meaning onto token sequences devoid of any.&#10;For instance, consider the natural language inference (NLI) example in &#10;\ref{fig:intro_example}. A RoBERTa-based model that scores $\sim 89\%$ on the Multi-NLI dataset~\cite{williams2018broad} identifies that the premise entails the hypothesis. However, when the words in the hypothesis are sorted alphabetically (thereby rendering the sequence meaningless), the model still makes the same prediction with high confidence. Indeed, across Multi-NLI, when the hypotheses are sorted alphabetically, the model retains the same prediction in $79\%$ of the cases, with a surprisingly high average confidence of $\sim 95\%$! &#10;We argue that a reliable model should not be insensitive to such a drastic change in word order.&#10;&#10;We study the response of large neural models to destructive transformations: perturbations of inputs that render them meaningless. &#10; \ref{fig:intro_example} shows an example. We define several such transformations, all of which erase meaning from the input text and produce token sequences that are not natural language (i.e., word salad).&#10;&#10;We characterize the response of models to such transformations using two metrics: its ability to predict valid labels for invalid inputs, and its confidence on these predictions.&#10;Via experiments on three tasks from the GLUE benchmark, we show that the labels predicted by state-of-the-art models for destructively transformed inputs bear high agreement with the original ones. Moreover, the models are highly confident in these predictions. &#10;We also find that models trained on meaningless examples perform comparably to the original model on unperturbed examples, despite never having encountered any well-formed training examples. Specifically, models trained on meaningless sentences constructed by permuting the word order perform almost as well as the state-of-the-art models. &#10;These observations suggest that, far from actually understanding natural language, today's state-of-the-art models have trouble even recognizing it.&#10;&#10;Finally, we evaluate strategies to mitigate these weaknesses using regularization that makes models less confident in their predictions, or by allowing models to reject inputs.&#10;&#10;In summary, our contributions are: &#10;\begin{enumerate}&#10;\item We define the notion of destructive input transformations to test the ability of text understanding models at processing word salad. We introduce nine such transformation functions that can be used by practitioners for diagnostic purposes without requiring additional annotated data. &#10;\item We show via experiments that today's best models force meaning upon invalid inputs; i.e., they are not using the right kind of information to arrive at their predictions. &#10;\item We show that simple mitigation strategies can teach models to recognize and reject invalid inputs. &#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Adversarial Input Detection, Computer Science, Linguistics, Cognitive Science, Language Model Evaluation, Robust Model Training" />
        </attvalues>
      </node>
      <node id="2210.12335" label="2210.12335">
        <attvalues>
          <attvalue for="0" value="Guided contrastive self-supervised pre-training for automatic speech&#10;  recognition" />
          <attvalue for="1" value="  Contrastive Predictive Coding (CPC) is a representation learning method that&#10;maximizes the mutual information between intermediate latent representations&#10;and the output of a given model. It can be used to effectively initialize the&#10;encoder of an Automatic Speech Recognition (ASR) model. We present a novel&#10;modification of CPC called Guided Contrastive Predictive Coding (GCPC). Our&#10;proposed method maximizes the mutual information between representations from a&#10;prior-knowledge model and the output of the model being pre-trained, allowing&#10;prior knowledge injection during pre-training. We validate our method on 3 ASR&#10;tasks: German, French and English. Our method outperforms CPC pre-training on&#10;all three datasets, reducing the Word Error Rate (WER) by 4.44%, 6.55% and&#10;15.43% relative on the German, French and English (Librispeech) tasks&#10;respectively, compared to training from scratch, while CPC pre-training only&#10;brings 2.96%, 1.01% and 14.39% relative WER reduction respectively.&#10;" />
          <attvalue for="2" value="&#10;Self-supervised Learning (SSL) has drawn a lot of recent attention in the machine learning community. After its successful applications in the natural language processing domain \cite{devlin2019bert, peters2018deep, radford2018improving}, it has also become an active research area for speech processing. &#10;&#10;One of the main categories of SSL methods learns representations by reconstructing the signal such as full reconstruction with autoencoders \cite{chen2019audio, chorowski2019unsupervised}, future reconstruction with Autoregressive Predictive Coding (APC) \cite{chung2019unsupervised} and masked reconstructions \cite{liu2020mockingjay, ling2020deep, ling2020bertphone}. Instead of reconstructing the exact signal, HuBERT \cite{hsu2021hubert} learns representations by utilizing an offline clustering step to provide aligned target labels for a masked prediction loss. Another category of SSL technology in literature learns representations through a contrastive loss by distinguishing a true future audio sample from a set of negative examples, such as the Contrastive Predictive Coding (CPC) model \cite{oord2018representation} and wav2vec \cite{schneider2019wav2vec}. Vq-wav2vec \cite{baevski2019vq} uses a vector quantization module in addition to contrastive loss to learn discrete representations and wav2vec 2.0 \cite{baevski2020wav2vec} minimizes the contrastive loss defined over contextual representations in the masked region. In addition, w2v-BERT \cite{chung2021w2v} combines the two categories by optimizing two self-supervised losses simultaneously (the contrastive loss and masked language modeling loss).&#10;&#10;All of these methods learn representations from the acoustic data distribution only, which may not be optimal for the downstream ASR task. More recently, Wang et al. propose two supervision-guided codebook generation approaches to get better pre-trained embeddings for the downstream ASR task in \cite{wang2022supervision}. On top of HuBERT pre-training, it uses the phoneme alignments as training targets. It also tries to perform K-means clustering on the supervised speech features extracted from an end-to-end CTC model \cite{graves2006connectionist}. However, this work focuses on the masked prediction self-supervised learning and all the ASR experiments are conducted with the Librispeech dataset with just a few hundred hours of labeled data. In our work, we focus on exploring the contrastive loss based SSL method instead and experiment with large-scale datasets. We propose to introduce weak guidance to improve alignment between the learned representations and the downstream task. The weak guidance is provided in the form of posteriors from a prior-knowledge model learned from a small labeled dataset, which will be discussed in detail in Section \ref{sec:method:gcpc}. &#10;&#10;To combine the self-supervised and supervised training to improve performance of the final ASR task, most existing methods in the literature adopt a 2-stage scheme, where only the self-supervised loss is optimized at the first pre-training stage, and the supervised loss is optimized at the second stage. Wav2vec \cite{schneider2019wav2vec} and vq-wav2vec \cite{baevski2019vq} build the wav2letter \cite{collobert2016wav2letter} acoustic model by using the pre-trained embeddings as input features instead of log-mel filterbanks. Wav2vec 2.0 \cite{baevski2020wav2vec} and HuBERT \cite{hsu2021hubert} pre-train the transformer based encoder using the self-supervised loss, add a randomly initialized output layer on top and fine-tune with the CTC loss \cite{graves2006connectionist}. More recent research has shown that joint training with both supervised and unsupervised losses during the pre-training/fine-tuning stage or as a single training process helps improve the ASR performance. The initial UniSpeech work \cite{wang2021unispeech} demonstrates that representations learned during pre-training can be improved if the self supervised contrastive loss is combined with phonetic CTC loss, and the following Unispeech at scale work \cite{wang2021unispeech2} demonstrates better representations from the pre-training stage for the downstream ASR task when combining the contrastive loss and the transducer loss. \cite{talnikar2021joint} alternatively minimizes an unsupervised masked CPC loss and a supervised CTC loss. This single-stage method is shown to&#10;match the performance of the two-stage wav2vec 2.0 on the Librispeech 100-hours dataset. \cite{raghavan2021hybrid} uses multitask learning comprising of supervised CTC, attention&#10;and self-supervised reconstruction losses to directly train acoustic models under low-resource settings. \cite{bai2022joint} explores the benefit of combining the supervised RNN-T loss \cite{graves2012sequence}, the self-supervised contrastive loss and masked language modeling (MLM) losses during different training stages. In this paper, we demonstrate benefits of our proposed method mainly on the conventional 2-stage training scheme. We additionally try the joint training scheme on one ASR task during the ablation study and demonstrate gains similar to what is reported in literature.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Automatic Speech Recognition, Deep Learning Methods, Signal Processing, Mathematics, Representation Learning" />
        </attvalues>
      </node>
      <node id="2107.05233" label="2107.05233">
        <attvalues>
          <attvalue for="0" value="UniSpeech at scale: An Empirical Study of Pre-training Method on&#10;  Large-Scale Speech Recognition Dataset" />
          <attvalue for="1" value="  Recently, there has been a vast interest in self-supervised learning (SSL)&#10;where the model is pre-trained on large scale unlabeled data and then&#10;fine-tuned on a small labeled dataset. The common wisdom is that SSL helps&#10;resource-limited tasks in which only a limited amount of labeled data is&#10;available. The benefit of SSL keeps diminishing when the labeled training data&#10;amount increases. To our best knowledge, at most a few thousand hours of&#10;labeled data was used in the study of SSL. In contrast, the industry usually&#10;uses tens of thousands of hours of labeled data to build high-accuracy speech&#10;recognition (ASR) systems for resource-rich languages. In this study, we take&#10;the challenge to investigate whether and how SSL can improve the ASR accuracy&#10;of a state-of-the-art production-scale Transformer-Transducer model, which was&#10;built with 65 thousand hours of anonymized labeled EN-US data.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.00093" label="2011.00093">
        <attvalues>
          <attvalue for="0" value="Joint Masked CPC and CTC Training for ASR" />
          <attvalue for="1" value="  Self-supervised learning (SSL) has shown promise in learning representations&#10;of audio that are useful for automatic speech recognition (ASR). But, training&#10;SSL models like wav2vec~2.0 requires a two-stage pipeline. In this paper we&#10;demonstrate a single-stage training of ASR models that can utilize both&#10;unlabeled and labeled data. During training, we alternately minimize two&#10;losses: an unsupervised masked Contrastive Predictive Coding (CPC) loss and the&#10;supervised audio-to-text alignment loss Connectionist Temporal Classification&#10;(CTC). We show that this joint training method directly optimizes performance&#10;for the downstream ASR task using unsupervised data while achieving similar&#10;word error rates to wav2vec~2.0 on the Librispeech 100-hour dataset. Finally,&#10;we postulate that solving the contrastive task is a regularization for the&#10;supervised CTC loss.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.11827" label="2308.11827">
        <attvalues>
          <attvalue for="0" value="Exploring the Effectiveness of GPT Models in Test-Taking: A Case Study&#10;  of the Driver's License Knowledge Test" />
          <attvalue for="1" value="  Large language models such as Open AI's Generative Pre-trained Transformer&#10;(GPT) models are proficient at answering questions, but their knowledge is&#10;confined to the information present in their training data. This limitation&#10;renders them ineffective when confronted with questions about recent&#10;developments or non-public documents. Our research proposes a method that&#10;enables GPT models to answer questions by employing context from an information&#10;source not previously included in their training data. The methodology includes&#10;preprocessing of contextual information, the embedding of contexts and queries,&#10;constructing prompt through the integration of context embeddings, and&#10;generating answers using GPT models. We applied this method in a controlled&#10;test scenario using the California Driver's Handbook as the information source.&#10;The GPT-3 model achieved a 96% passing score on a set of 50 sample driving&#10;knowledge test questions. In contrast, without context, the model's passing&#10;score fell to 82%. However, the model still fails to answer some questions&#10;correctly even with providing library of context, highlighting room for&#10;improvement. The research also examined the impact of prompt length and context&#10;format, on the model's performance. Overall, the study provides insights into&#10;the limitations and potential improvements for GPT models in question-answering&#10;tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;In recent years, the advent of large-scale language models, such as GPT models \cite{brown2020language}, \cite{chen2021evaluating}, has revolutionized natural language understanding and generation \cite{rae2021scaling}, \cite{chowdhery2022palm}, \cite{thoppilan2022lamda}, \cite{zhang2022opt}, \cite{smith2022using}, \cite{fedus2022switch}. These models have demonstrated exceptional performance in a wide range of tasks, including machine translation, summarizing, and question-answering. Their prowess stems from their pre-training on vast amounts of text data, enabling them to learn complex language patterns and structures. As a result, these models possess a remarkable ability to generate human-like responses to a variety of prompts. Prior research has explored the capabilities of GPT models in different question-answering domains. For instance, studies have assessed the performance of these models in answering questions from the bar exam \cite{bommarito2022gpt}, \cite{katz2023gpt}, medical tests \cite{lievin2022can}, \cite{kung2023performance}, \cite{nori2023capabilities}, radiology examinations \cite{bhayana2023performance}, \cite{bhayana2023gpt}, and mathematical tests \cite{floridi2020gpt}. While these studies have demonstrated the potential of GPT models in tackling complex and domain-specific questions, they have also identified challenges and limitations, such as the susceptibility of the models to hallucinate information that is not present in the input data \cite{bender2021dangers}, \cite{wang2021adversarial}.&#10;&#10;Contextual information plays a crucial role in the performance of language models, particularly when answering questions that require a deep understanding of specific domains. In this regard, integrating relevant contextual information into the input prompts can significantly enhance the models’ ability to generate accurate and coherent responses \cite{liu2023pre}, \cite{borgeaud2022improving}. However, the optimal strategies for incorporating context and the impact of context length and format on the models’ performance have not been thoroughly investigated.&#10;&#10;In this paper, we examine the performance of the GPT-3 model in a question-answering task using contextual information from the California Driver's Handbook. Initially, we investigated the model's performance with no context provided. That is, we tested the GPT-3 model solely on the questions, without utilizing the California Driver's Handbook as a context source. Given that the model's training data likely includes extensive information on driver's knowledge test questions, we specifically wanted to test how well it adapted to the unique rules of the California driver's license knowledge test, as driving regulations can vary slightly from one US state to another and significantly between countries. We explore the influence of varying context lengths and formats on the model’s accuracy and analyze the model’s limitations, including sensitivity to text formatting and potential hallucination. By conducting a series of experiments, we aim to shed light on the factors that contribute to the model’s success and shortcomings in this domain. Our study contributes to the growing body of literature on GPT models in question-answering tasks by offering insights into the role of context in the model’s performance. Moreover, we provide recommendations for optimizing context integration and formatting to enhance the accuracy of the generated responses. These findings have implications for researchers and practitioners employing large-scale language models in various applications, particularly those that involve answering questions based on contextual information.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Context-Aware Question Answering, Language Model Limitations, Artificial Intelligence Improvements, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2207.08143" label="2207.08143">
        <attvalues>
          <attvalue for="0" value="Can large language models reason about medical questions?" />
          <attvalue for="1" value="  Although large language models (LLMs) often produce impressive outputs, it&#10;remains unclear how they perform in real-world scenarios requiring strong&#10;reasoning skills and expert domain knowledge. We set out to investigate whether&#10;close- and open-source models (GPT-3.5, LLama-2, etc.) can be applied to answer&#10;and reason about difficult real-world-based questions. We focus on three&#10;popular medical benchmarks (MedQA-USMLE, MedMCQA, and PubMedQA) and multiple&#10;prompting scenarios: Chain-of-Thought (CoT, think step-by-step), few-shot and&#10;retrieval augmentation. Based on an expert annotation of the generated CoTs, we&#10;found that InstructGPT can often read, reason and recall expert knowledge.&#10;Last, by leveraging advances in prompt engineering (few-shot and ensemble&#10;methods), we demonstrated that GPT-3.5 not only yields calibrated predictive&#10;distributions, but also reaches the passing score on three datasets:&#10;MedQA-USMLE 60.2%, MedMCQA 62.7% and PubMedQA 78.2%. Open-source models are&#10;closing the gap: Llama-2 70B also passed the MedQA-USMLE with 62.5% accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.09117" label="2311.09117">
        <attvalues>
          <attvalue for="0" value="R-Spin: Efficient Speaker and Noise-invariant Representation Learning&#10;  with Acoustic Pieces" />
          <attvalue for="1" value="  This paper introduces Robust Spin (R-Spin), a data-efficient domain-specific&#10;self-supervision method for speaker and noise-invariant speech representations&#10;by learning discrete acoustic units with speaker-invariant clustering (Spin).&#10;R-Spin resolves Spin's issues and enhances content representations by learning&#10;to predict acoustic pieces. R-Spin offers a 12X reduction in computational&#10;resources compared to previous state-of-the-art methods while outperforming&#10;them in severely distorted speech scenarios. This paper provides detailed&#10;analyses to show how discrete units contribute to speech encoder training and&#10;improving robustness in diverse acoustic environments.&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-supervised learning~(SSL) for encoder pre-training has emerged as a foundational element in speech processing, outperforming conventional approaches across various applications~\cite{mohamed2022self,liu2022audio}.&#10;Given the substantial cost associated with human annotation of speech data, SSL methods leverage unlabeled audio data to pre-train encoders, generating good representations for downstream tasks like automatic speech recognition~(ASR) and speaker identification~\cite{yang2021superb,tsai-etal-2022-superb}.&#10;The application of SSL models has notably concentrated on ASR, aiming to mitigate the dependence on large transcribed corpora~\cite{hsu2021hubert,baevski2022data2vec,liu2023dinosr}.&#10;Thus, extracting content representations has become a crucial aspect of speech SSL research~\cite{tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}.&#10;Prior studies have devised objective functions to disentangle content from speech, fostering the ability of SSL models to generate speaker-invariant representations through domain-specific self-supervision~(DS).&#10;In DS, a pre-trained SSL model is fine-tuned with unlabeled data for specific applications.&#10;\cite{qian2022contentvec} propose ContentVec by disentangling speaker and content information, demonstrating promising results.&#10;However, ContentVec suffers from the requirement of a voice conversion model and substantial computational costs exceeding 600 GPU hours.&#10;Alternatively, \cite{chang2023spin} propose Speaker-invariant Clustering~(Spin) to produce content representations with minimal fine-tuning resources.&#10;Nonetheless, Spin is constrained to fine-tuning only the top layers, thereby lacking the flexibility to adapt to diverse acoustic domains.&#10;&#10;Parallel to modeling content information in speech, numerous studies are dedicated to investigating the robustness of speech SSL.&#10;While current methods perform well on clean speech datasets, they are vulnerable to out-of-domain data like distorted audio signals~\cite{hsu2021robust}.&#10;To mitigate this vulnerability, researchers have proposed noise-invariant training techniques.&#10;\cite{huang2022improving} proposes HuBERT-MGR via domain adversarial training to render the fine-tuned HuBERT model~\cite{hsu2021hubert} invariant to domain shifts.&#10;WavLM~\cite{chen2022wavlm} integrates denoising with the HuBERT pre-training framework, achieving state-of-the-art performance in many speech processing downstream tasks.&#10;Similarly, \cite{zhu2023robust} propose Robust data2vec, introducing perturbations to the input to predict the exponential moving average teacher model's representations.&#10;In deHuBERT~\cite{ng2023dehubert}, the Barlow Twins loss~\cite{zbontar2021barlow} is applied to encourage representation invariability to input perturbations.&#10;Although many methods have shown success in noisy speech recognition~\cite{wang2022wav2vec-switch,zhu2022noise,huang2022spiral,hu2023wav2code}, to our knowledge, none have concurrently addressed the disentanglement of speaker and noise while enhancing content information.&#10;Furthermore, these approaches exhibit inefficiency, often requiring high computation costs and iterating large corpora over numerous epochs.&#10;&#10;To effectively acquire high-quality content and robust representations for real-world applications, this paper extends Spin with noise-invariant training and acoustic piece pseudo-label learning, coined Robust Spin~(R-Spin).&#10;During training, two utterances of the same content with different distortions are fed into a speech SSL encoder.&#10;The outputs are frame-wise vector-quantized with a learnable codebook via online clustering, as in Spin.&#10;The model is trained to match cluster ID distributions between the utterances.&#10;To prevent codebook collapse, an additional pseudo-label prediction loss is introduced.&#10;The pseudo-labels are generated by learning acoustic pieces~\cite{ren2022acoustic-piece} on top of the discrete units produced by a pre-trained Spin model, offering better training targets that closely align with phonemes and characters.&#10;Within this framework, the speech encoder learns speaker and noise-invariant representations, benefiting robustness and content extraction simultaneously.&#10;The contributions are summarized as follows:&#10;\begin{enumerate}[nolistsep]&#10; \item We integrate predicting acoustic pieces into Spin, enabling fine-tuning all parameters without collapsing, which allows the processing of more complex speech recordings.&#10; \item R-Spin~inherits the benefit of low training costs from Spin, requiring 12X less computation than prior art.&#10; \item With noise-invariant training, R-Spin~outperforms other DS approaches in distorted speech and phoneme recognition tasks like the CHiME-4 challenge~\cite{vincent2017chime4}.&#10; \item We inspect the hidden representations of speech SSL models to quantify the speaker and noise invariability.&#10; \item We offer in-depth analyses of discrete acoustic units to understand how these units help speech encoder training.&#10;\end{enumerate}&#10; " />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Self-Supervision Methods, Computer Science, Robust Acoustic Modeling, Linguistics, Engineering, Speech Representation, Signal Processing" />
        </attvalues>
      </node>
      <node id="2210.15324" label="2210.15324">
        <attvalues>
          <attvalue for="0" value="Robust Data2vec: Noise-robust Speech Representation Learning for ASR by&#10;  Combining Regression and Improved Contrastive Learning" />
          <attvalue for="1" value="  Self-supervised pre-training methods based on contrastive learning or&#10;regression tasks can utilize more unlabeled data to improve the performance of&#10;automatic speech recognition (ASR). However, the robustness impact of combining&#10;the two pre-training tasks and constructing different negative samples for&#10;contrastive learning still remains unclear. In this paper, we propose a&#10;noise-robust data2vec for self-supervised speech representation learning by&#10;jointly optimizing the contrastive learning and regression tasks in the&#10;pre-training stage. Furthermore, we present two improved methods to facilitate&#10;contrastive learning. More specifically, we first propose to construct&#10;patch-based non-semantic negative samples to boost the noise robustness of the&#10;pre-training model, which is achieved by dividing the features into patches at&#10;different sizes (i.e., so-called negative samples). Second, by analyzing the&#10;distribution of positive and negative samples, we propose to remove the easily&#10;distinguishable negative samples to improve the discriminative capacity for&#10;pre-training models. Experimental results on the CHiME-4 dataset show that our&#10;method is able to improve the performance of the pre-trained model in noisy&#10;scenarios. We find that joint training of the contrastive learning and&#10;regression tasks can avoid the model collapse to some extent compared to only&#10;training the regression task.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2103.03230" label="2103.03230">
        <attvalues>
          <attvalue for="0" value="Barlow Twins: Self-Supervised Learning via Redundancy Reduction" />
          <attvalue for="1" value="  Self-supervised learning (SSL) is rapidly closing the gap with supervised&#10;methods on large computer vision benchmarks. A successful approach to SSL is to&#10;learn embeddings which are invariant to distortions of the input sample.&#10;However, a recurring issue with this approach is the existence of trivial&#10;constant solutions. Most current methods avoid such solutions by careful&#10;implementation details. We propose an objective function that naturally avoids&#10;collapse by measuring the cross-correlation matrix between the outputs of two&#10;identical networks fed with distorted versions of a sample, and making it as&#10;close to the identity matrix as possible. This causes the embedding vectors of&#10;distorted versions of a sample to be similar, while minimizing the redundancy&#10;between the components of these vectors. The method is called Barlow Twins,&#10;owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a&#10;pair of identical networks. Barlow Twins does not require large batches nor&#10;asymmetry between the network twins such as a predictor network, gradient&#10;stopping, or a moving average on the weight updates. Intriguingly it benefits&#10;from very high-dimensional output vectors. Barlow Twins outperforms previous&#10;methods on ImageNet for semi-supervised classification in the low-data regime,&#10;and is on par with current state of the art for ImageNet classification with a&#10;linear classifier head, and for transfer tasks of classification and object&#10;detection.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2201.10207" label="2201.10207">
        <attvalues>
          <attvalue for="0" value="SPIRAL: Self-supervised Perturbation-Invariant Representation Learning&#10;  for Speech Pre-Training" />
          <attvalue for="1" value="  We introduce a new approach for speech pre-training named SPIRAL which works&#10;by learning denoising representation of perturbed data in a teacher-student&#10;framework. Specifically, given a speech utterance, we first feed the utterance&#10;to a teacher network to obtain corresponding representation. Then the same&#10;utterance is perturbed and fed to a student network. The student network is&#10;trained to output representation resembling that of the teacher. At the same&#10;time, the teacher network is updated as moving average of student's weights&#10;over training steps. In order to prevent representation collapse, we apply an&#10;in-utterance contrastive loss as pre-training objective and impose position&#10;randomization on the input to the teacher. SPIRAL achieves competitive or&#10;better results compared to state-of-the-art speech pre-training method wav2vec&#10;2.0, with significant reduction of training cost (80% for BASE model, 65% for&#10;LARGE model). Furthermore, we address the problem of noise-robustness that is&#10;critical to real-world speech applications. We propose multi-condition&#10;pre-training by perturbing the student's input with various types of additive&#10;noise. We demonstrate that multi-condition pre-trained SPIRAL models are more&#10;robust to noisy speech (9.0% - 13.3% relative word error rate reduction on real&#10;noisy test data), compared to applying multi-condition training solely in the&#10;fine-tuning stage. Source code is available at&#10;https://github.com/huawei-noah/Speech-Backbones/tree/main/SPIRAL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2304.04974" label="2304.04974">
        <attvalues>
          <attvalue for="0" value="Wav2code: Restore Clean Speech Representations via Codebook Lookup for&#10;  Noise-Robust ASR" />
          <attvalue for="1" value="  Automatic speech recognition (ASR) has gained remarkable successes thanks to&#10;recent advances of deep learning, but it usually degrades significantly under&#10;real-world noisy conditions. Recent works introduce speech enhancement (SE) as&#10;front-end to improve speech quality, which is proved effective but may not be&#10;optimal for downstream ASR due to speech distortion problem. Based on that,&#10;latest works combine SE and currently popular self-supervised learning (SSL) to&#10;alleviate distortion and improve noise robustness. Despite the effectiveness,&#10;the speech distortion caused by conventional SE still cannot be cleared out. In&#10;this paper, we propose a self-supervised framework named Wav2code to implement&#10;a feature-level SE with reduced distortions for noise-robust ASR. First, in&#10;pre-training stage the clean speech representations from SSL model are sent to&#10;lookup a discrete codebook via nearest-neighbor feature matching, the resulted&#10;code sequence are then exploited to reconstruct the original clean&#10;representations, in order to store them in codebook as prior. Second, during&#10;finetuning we propose a Transformer-based code predictor to accurately predict&#10;clean codes by modeling the global dependency of input noisy representations,&#10;which enables discovery and restoration of high-quality clean representations&#10;with reduced distortions. Furthermore, we propose an interactive feature fusion&#10;network to combine original noisy and the restored clean representations to&#10;consider both fidelity and quality, resulting in more informative features for&#10;downstream ASR. Finally, experiments on both synthetic and real noisy datasets&#10;demonstrate that Wav2code can solve the speech distortion and improve ASR&#10;performance under various noisy conditions, resulting in stronger robustness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2109.02071" label="2109.02071">
        <attvalues>
          <attvalue for="0" value="End-to-End Self-Debiasing Framework for Robust NLU Training" />
          <attvalue for="1" value="  Existing Natural Language Understanding (NLU) models have been shown to&#10;incorporate dataset biases leading to strong performance on in-distribution&#10;(ID) test sets but poor performance on out-of-distribution (OOD) ones. We&#10;introduce a simple yet effective debiasing framework whereby the shallow&#10;representations of the main model are used to derive a bias model and both&#10;models are trained simultaneously. We demonstrate on three well studied NLU&#10;tasks that despite its simplicity, our method leads to competitive OOD results.&#10;It significantly outperforms other debiasing approaches on two tasks, while&#10;still delivering high in-distribution performance.&#10;" />
          <attvalue for="2" value="&#10;&#10;Researchers have increasingly raised concerns about the tendency of recent NLU models~\cite{devlin2018bert,liu2019roberta} to quickly leverage spurious surface lexical-syntactic features~\cite{poliak2018hypothesis,gururangan2018annotation,dasgupta2018evaluating,ghaddar2017winer}. These superficial properties, also referred as dataset biases~\cite{shah2020predictive,utama2020towards,moosavi2020improving}, result in significant performance drop on out-of-distribution (OOD) sets containing counterexamples to biases in the training data~\cite{mccoy2019right,schuster2019towards,zhang2019paws,ghaddar2021context}.&#10;&#10;The most common approach to tackle the problem consists in training a bias model with hand-crafted features with the goal of identifying biased training examples. This information is used in a later stage to discourage the main model from adopting the naive strategy of the bias model. Several debiasing training paradigms have been proposed to adjust the importance of biased training samples, such as product of experts~\cite{clark2019don,he2019unlearn}, learned-mixin~\cite{clark2019don}, example reweighting~\cite{schuster2019towards}, debiased focal loss~\cite{mahabadi2020end}, and confidence regularization~\cite{utama2020mind}. &#10;&#10;Recently, there has been a number of endeavours to produce a bias model without prior knowledge on the targeted biases or without the need for manually designing features. \cite{utama2020towards} propose to use instead a model trained on a tiny fraction ($&lt;1\%$) of the training data for few epochs as a bias model; while \cite{clark2020learning}~and \cite{sanh2020learning} trained a low capacity model on the full training set. These approaches target the training of the bias model alone, which is subsequently queried while training the main model of interest.&#10;&#10;In this paper, we propose an end-to-end debiasing framework which does not require an extra training stage, or manual bias features engineering. The bias model is indeed a simple attention-based classification layer on top of the main model's intermediate representations. Both models are trained simultaneously in an end-to-end manner as in \cite{mahabadi2020end}, where the importance of training samples for both models are adjusted using the example reweighting technique of~\cite{schuster2019towards}. The idea of using intermediate classifiers has previously been explored to reduce the inference cost~\cite{schwartz2020right,zhou2020bert,xin2021berxit} for Transformer-based~\cite{vaswani2017attention} models.&#10;&#10;In contrast to all previous works, our bias model helps locating lexico-syntactic bias features, inside the main model's intermediate layers, whose importance is reduced by adding a noise vector; therefore preventing the main model to rely on them. During training, both the main and bias models interact by interchangeably re-weighting the importance of each others' examples. &#10;&#10;Our learning framework, when applied to a vanilla Bert~\cite{devlin2018bert} model leads to consistent and significant improvements on 3 NLU tasks, while maintaining a balanced performance between ID and OOD sets. It involves a single training stage, and only incurs a small number of extra parameters (0.5M) compared to other approaches. For instance, \cite{utama2020mind} used a copy of the main model (110M parameters) as the bias model, while \cite{sanh2020learning} used Bert{-Tiny}~\cite{turc2019well} that has 11M parameters and has been pre-trained from scratch using the masked LM objective~\cite{devlin2018bert}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Cognitive Science, Natural Language Understanding, Model Performance Optimization, Artificial Intelligence, Debiasing Techniques" />
        </attvalues>
      </node>
      <node id="1802.04302" label="1802.04302">
        <attvalues>
          <attvalue for="0" value="Evaluating Compositionality in Sentence Embeddings" />
          <attvalue for="1" value="  An important challenge for human-like AI is compositional semantics. Recent&#10;research has attempted to address this by using deep neural networks to learn&#10;vector space embeddings of sentences, which then serve as input to other tasks.&#10;We present a new dataset for one such task, `natural language inference' (NLI),&#10;that cannot be solved using only word-level knowledge and requires some&#10;compositionality. We find that the performance of state of the art sentence&#10;embeddings (InferSent; Conneau et al., 2017) on our new dataset is poor. We&#10;analyze the decision rules learned by InferSent and find that they are&#10;consistent with simple heuristics that are ecologically valid in its training&#10;dataset. Further, we find that augmenting training with our dataset improves&#10;test performance on our dataset without loss of performance on the original&#10;training dataset. This highlights the importance of structured datasets in&#10;better understanding and improving AI systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2107.11610" label="2107.11610">
        <attvalues>
          <attvalue for="0" value="Context-aware Adversarial Training for Name Regularity Bias in Named&#10;  Entity Recognition" />
          <attvalue for="1" value="  In this work, we examine the ability of NER models to use contextual&#10;information when predicting the type of an ambiguous entity. We introduce NRB,&#10;a new testbed carefully designed to diagnose Name Regularity Bias of NER&#10;models. Our results indicate that all state-of-the-art models we tested show&#10;such a bias; BERT fine-tuned models significantly outperforming feature-based&#10;(LSTM-CRF) ones on NRB, despite having comparable (sometimes lower) performance&#10;on standard benchmarks.&#10;  To mitigate this bias, we propose a novel model-agnostic training method that&#10;adds learnable adversarial noise to some entity mentions, thus enforcing models&#10;to focus more strongly on the contextual signal, leading to significant gains&#10;on NRB. Combining it with two other training strategies, data augmentation and&#10;parameter freezing, leads to further gains.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1909.06321" label="1909.06321">
        <attvalues>
          <attvalue for="0" value="End-to-End Bias Mitigation by Modelling Biases in Corpora" />
          <attvalue for="1" value="  Several recent studies have shown that strong natural language understanding&#10;(NLU) models are prone to relying on unwanted dataset biases without learning&#10;the underlying task, resulting in models that fail to generalize to&#10;out-of-domain datasets and are likely to perform poorly in real-world&#10;scenarios. We propose two learning strategies to train neural models, which are&#10;more robust to such biases and transfer better to out-of-domain datasets. The&#10;biases are specified in terms of one or more bias-only models, which learn to&#10;leverage the dataset biases. During training, the bias-only models' predictions&#10;are used to adjust the loss of the base model to reduce its reliance on biases&#10;by down-weighting the biased examples and focusing the training on the hard&#10;examples. We experiment on large-scale natural language inference and fact&#10;verification benchmarks, evaluating on out-of-domain datasets that are&#10;specifically designed to assess the robustness of models against known biases&#10;in the training data. Results show that our debiasing methods greatly improve&#10;robustness in all settings and better transfer to other textual entailment&#10;datasets. Our code and data are publicly available in&#10;\url{https://github.com/rabeehk/robust-nli}.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;To address dataset biases, researchers have proposed to augment datasets by balancing the existing cues~\cite{schuster2019towards} or to create an adversarial dataset~\cite{jia2017adversarial}. However, collecting new datasets, especially at a large scale, is costly, and thus remains an unsatisfactory solution. It is, therefore, crucial to develop strategies to allow models to be trained on the existing biased datasets. &#10;&#10;\cite{schuster2019towards} propose to first compute the n-grams in the dataset's claims that are the most associated with each fact-verification label. They then solve an optimization problem to assign a balancing weight to each training sample to alleviate the biases. In contrast, we propose several end-to-end debiasing strategies. Additionally, \cite{belinkov-etal-2019-dont} propose adversarial techniques to remove from the NLI sentence encoder the features that allow a&#10;hypothesis-only model to succeed. However, we believe that in general,&#10;the features used by the hypothesis-only model can include some&#10;information necessary to perform the NLI task, and removing such&#10;information from the sentence representation can hurt the performance&#10;of the full model. Their approach consequently degrades the performance on the hard SNLI&#10; set, which is expected to be less biased. In contrast, we propose to train a bias-only model to use its predictions to&#10; dynamically adapt the classification loss to reduce the importance of the most biased examples. &#10;&#10;Concurrently to our work,~\cite{clark2019dont} and \cite{he-etal-2019-unlearn} have also proposed to use the product of experts (PoE) models for avoiding biases. They train their models in two stages, first training a bias-only model and then using it to train a robust model. In contrast, our methods are trained in an end-to-end manner, which is convenient in practice. We additionally show that our proposed Debiased Focal Loss model is an effective method to reduce biases, sometimes superior to PoE. We have evaluated on new domains of NLI hard sets and fact verification. Moreover, we have included an analysis showing that our debiased models indeed have lower correlations with the bias-only models, and have extended our methods to guard against multiple bias patterns simultaneously. We furthermore study transfer performance to other NLI datasets.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2005.00315" label="2005.00315">
        <attvalues>
          <attvalue for="0" value="Mind the Trade-off: Debiasing NLU Models without Degrading the&#10;  In-distribution Performance" />
          <attvalue for="1" value="  Models for natural language understanding (NLU) tasks often rely on the&#10;idiosyncratic biases of the dataset, which make them brittle against test cases&#10;outside the training distribution. Recently, several proposed debiasing methods&#10;are shown to be very effective in improving out-of-distribution performance.&#10;However, their improvements come at the expense of performance drop when models&#10;are evaluated on the in-distribution data, which contain examples with higher&#10;diversity. This seemingly inevitable trade-off may not tell us much about the&#10;changes in the reasoning and understanding capabilities of the resulting models&#10;on broader types of examples beyond the small subset represented in the&#10;out-of-distribution data. In this paper, we address this trade-off by&#10;introducing a novel debiasing method, called confidence regularization, which&#10;discourage models from exploiting biases while enabling them to receive enough&#10;incentive to learn from all the training examples. We evaluate our method on&#10;three NLU tasks and show that, in contrast to its predecessors, it improves the&#10;performance on out-of-distribution datasets (e.g., 7pp gain on HANS dataset)&#10;while maintaining the original in-distribution accuracy.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2011.03856" label="2011.03856">
        <attvalues>
          <attvalue for="0" value="Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles" />
          <attvalue for="1" value="  Many datasets have been shown to contain incidental correlations created by&#10;idiosyncrasies in the data collection process. For example, sentence entailment&#10;datasets can have spurious word-class correlations if nearly all contradiction&#10;sentences contain the word &quot;not&quot;, and image recognition datasets can have&#10;tell-tale object-background correlations if dogs are always indoors. In this&#10;paper, we propose a method that can automatically detect and ignore these kinds&#10;of dataset-specific patterns, which we call dataset biases. Our method trains a&#10;lower capacity model in an ensemble with a higher capacity model. During&#10;training, the lower capacity model learns to capture relatively shallow&#10;correlations, which we hypothesize are likely to reflect dataset bias. This&#10;frees the higher capacity model to focus on patterns that should generalize&#10;better. We ensure the models learn non-overlapping approaches by introducing a&#10;novel method to make them conditionally independent. Importantly, our approach&#10;does not require the bias to be known in advance. We evaluate performance on&#10;synthetic datasets, and four datasets built to penalize models that exploit&#10;known biases on textual entailment, visual question answering, and image&#10;recognition tasks. We show improvement in all settings, including a 10 point&#10;gain on the visual question answering dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2012.01300" label="2012.01300">
        <attvalues>
          <attvalue for="0" value="Learning from others' mistakes: Avoiding dataset biases without modeling&#10;  them" />
          <attvalue for="1" value="  State-of-the-art natural language processing (NLP) models often learn to&#10;model dataset biases and surface form correlations instead of features that&#10;target the intended underlying task. Previous work has demonstrated effective&#10;methods to circumvent these issues when knowledge of the bias is available. We&#10;consider cases where the bias issues may not be explicitly identified, and show&#10;a method for training models that learn to ignore these problematic&#10;correlations. Our approach relies on the observation that models with limited&#10;capacity primarily learn to exploit biases in the dataset. We can leverage the&#10;errors of such limited capacity models to train a more robust model in a&#10;product of experts, thus bypassing the need to hand-craft a biased model. We&#10;show the effectiveness of this method to retain improvements in&#10;out-of-distribution settings even if no particular bias is targeted by the&#10;biased model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="&#10;&#10;Many studies have reported dataset biases in various settings. Examples include visual question answering \cite{Jabri2016RevisitingVQ,Zhang2016YinAY}, story completion \cite{Schwartz2017TheEO}, and reading comprehension \cite{Kaushik2018HowMR,Chen2016ATE}. Towards better evaluation methods, researchers have proposed to collect ``challenge'' datasets that account for surface correlations a model might adopt \cite{Jia2017AdversarialEF,McCoy2019RightFT}. Standard models without specific robust training methods often drop in performance when evaluated on these challenge sets.&#10;&#10;While these works have focused on data collection, another approach is to develop methods allowing models to ignore dataset biases during training. Several active areas of research tackle this challenge by adversarial training \cite{Belinkov2019DontTT,Stacey2020ThereIS}, example forgetting \cite{Yaghoobzadeh2019RobustNL} and dynamic loss adjustment \cite{Cadne2019RUBiRU}. Previous work \cite{He2019UnlearnDB,Clark2019DontTT,Mahabadi2020EndtoEndBM} has shown the effectiveness of product of experts to train un-biased models. In our work, we show that we do not need to explicitly model biases to apply these de-biasing methods and can use a more general setup than previously presented.&#10;&#10;Orthogonal to these evaluation and optimization efforts, data augmentation has attracted interest as a way to reduce model biases by explicitly modifying the dataset distribution \cite{Min2020SyntacticDA,Belinkov2018SyntheticAN}, either by leveraging human knowledge about dataset biases such as swapping male and female entities \cite{Zhao2018GenderBI} or by developing dynamic data collection and benchmarking \cite{nie2020adversarial}. Our work is mostly orthogonal to these efforts and alleviates the need for a human-in-the-loop setup which is common to such data-augmentation approaches. &#10;&#10;Large pre-trained language models have contributed to improved out-of-distribution generalization \cite{Hendrycks2020PretrainedTI}. However, in practice, that remains a challenge in natural language processing \cite{Linzen2020HowCW,Yogatama2019LearningAE} and our work aims at out-of-distribution robustness without significantly compromising in-distribution performance.&#10;&#10;Finally, as we were preparing this manuscript for submission, we became aware of a parallel work \cite{utama2020debiasing} which presents a related de-biasing method leveraging shallow models's mistakes without the need to explicitly model dataset biases. Our approach is different in several ways, in particular we advocate for using limited capacity weak learner while \cite{utama2020debiasing} uses the same architecture as the robust model trained on a few thousands examples. We investigated the trade-off between learner's capacity and resulting performances as well as the resulting few-shot learning regime in the limit of a high capacity weak model.&#10;&#10;" />
        </attvalues>
      </node>
      <node id="2111.10157" label="2111.10157">
        <attvalues>
          <attvalue for="0" value="Lattention: Lattice-attention in ASR rescoring" />
          <attvalue for="1" value="  Lattices form a compact representation of multiple hypotheses generated from&#10;an automatic speech recognition system and have been shown to improve&#10;performance of downstream tasks like spoken language understanding and speech&#10;translation, compared to using one-best hypothesis. In this work, we look into&#10;the effectiveness of lattice cues for rescoring n-best lists in second-pass. We&#10;encode lattices with a recurrent network and train an attention encoder-decoder&#10;model for n-best rescoring. The rescoring model with attention to lattices&#10;achieves 4-5% relative word error rate reduction over first-pass and 6-8% with&#10;attention to both lattices and acoustic features. We show that rescoring models&#10;with attention to lattices outperform models with attention to n-best&#10;hypotheses. We also study different ways to incorporate lattice weights in the&#10;lattice encoder and demonstrate their importance for n-best rescoring.&#10;" />
          <attvalue for="2" value="&#10;&#10;In a typical multi-pass automatic speech recognition (ASR) system, the first-pass system produces lattices \cite{ney1994word} or n-best hypotheses \cite{schwartz1991comparison} which are rescored in the second-pass. More commonly, a neural language model (NLM) trained on large amount of text data is used in the second-pass rescoring \cite{mikolov2010recurrent, sundermeyer2012lstm}. Recently, stronger rescoring models utilizing acoustic information have been proposed. In \cite{sainath2019two}, a listen-attend-spell \cite{chan2016listen} based model was proposed to rescore n-best lists where the encoder is shared with the first-pass recurrent neural network transducer (RNN-T) \cite{graves2012sequence} model. Similarly, in \cite{gandhe2020audio}, NLM was extended to attend to audio features generated by the acoustic model in the first-pass ASR system. In further extension to \cite{sainath2019two}, a deliberation network \cite{xia2017deliberation} based model with additional attention to n-best hypotheses was introduced in \cite{hu2020deliberation}. A more compact representation of the first-pass decoding output are lattices. Lattices encode multiple hypotheses in a condensed form and carry the uncertainties from the first-pass decoding. Using a lattice encoder instead of the 1-best output has been shown to improve performance of downstream tasks like speech translation \cite{sperber2017neural,su2017lattice,sperber2019self,xiao2019lattice} and spoken language understanding \cite{ladhak2016latticernn,zhang2018chinese}.&#10;&#10;There has been some previous work on rescoring lattices in second-pass \cite{liu2014efficient,xu2018pruned} instead of a subset of hypotheses in the n-best list, making use of the richer information in the lattices. In this work, we utilize lattice information for n-best rescoring by encoding them with a recurrent network. We train an attention based encoder-decoder model which attends to the lattice encoder and run the decoder in the teacher-forcing mode to rescore n-best lists. We experiment with different encoders: 1-best, n-best, lattice and audio features extracted from the first-pass model. We employ minimum word error rate (MWER) \cite{hori2016minimum} training criterion which has been shown to improve accuracy of attention-based rescoring models \cite{sainath2019two,gandhe2020audio,hu2020deliberation}.&#10;&#10;There has already been some work on representing lattice structures in recurrent encoders \cite{sperber2017neural,su2017lattice,ladhak2016latticernn} and transformers \cite{sperber2019self,xiao2019lattice} models. In \cite{sperber2017neural}, LatticeLSTM was proposed for machine translation, which extends TreeLSTM \cite{tai2015improved} to encode directed acyclic graphs with weights. We utilize LatticeLSTM with certain modifications as the lattice encoder for ASR n-best rescoring in this work. Specifically, following are the contributions of this paper: (1) We propose a simplified method for encoding lattice weights with similar performance as \cite{sperber2017neural}, (2) We show that lattice-attention rescoring model can provide 4-5\% relative word error rate reduction (WERR) over first-pass, (3) LatticeLSTM-based lattice encoder results in more improvements compared to n-best deliberation encoder \cite{hu2020deliberation}, even for lattices containing same hypotheses as the n-best, (4) Attending to both audio and lattice further reduces word error rate (WER), resulting in 6-8\% relative WERR over first-pass, (5) We study the effect of different mechanisms to incorporate lattice weights in LatticeLSTM and show that unweighted lattice encoders (TreeLSTM) are detrimental for attention-based models and integrating lattice weights is important to mitigate confusions arising from contradictory lattice arcs.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Rescoring Models, Lattice Encoding, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1609.07730" label="1609.07730">
        <attvalues>
          <attvalue for="0" value="Lattice-Based Recurrent Neural Network Encoders for Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  Neural machine translation (NMT) heavily relies on word-level modelling to&#10;learn semantic representations of input sentences. However, for languages&#10;without natural word delimiters (e.g., Chinese) where input sentences have to&#10;be tokenized first, conventional NMT is confronted with two issues: 1) it is&#10;difficult to find an optimal tokenization granularity for source sentence&#10;modelling, and 2) errors in 1-best tokenizations may propagate to the encoder&#10;of NMT. To handle these issues, we propose word-lattice based Recurrent Neural&#10;Network (RNN) encoders for NMT, which generalize the standard RNN to word&#10;lattice topology. The proposed encoders take as input a word lattice that&#10;compactly encodes multiple tokenizations, and learn to generate new hidden&#10;states from arbitrarily many inputs and hidden states in preceding time steps.&#10;As such, the word-lattice based encoders not only alleviate the negative impact&#10;of tokenization errors but also are more expressive and flexible to embed input&#10;sentences. Experiment results on Chinese-English translation demonstrate the&#10;superiorities of the proposed encoders over the conventional encoder.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.00353" label="1910.00353">
        <attvalues>
          <attvalue for="0" value="Grammatical Error Correction in Low-Resource Scenarios" />
          <attvalue for="1" value="  Grammatical error correction in English is a long studied problem with many&#10;existing systems and datasets. However, there has been only a limited research&#10;on error correction of other languages. In this paper, we present a new dataset&#10;AKCES-GEC on grammatical error correction for Czech. We then make experiments&#10;on Czech, German and Russian and show that when utilizing synthetic parallel&#10;corpus, Transformer neural machine translation model can reach new&#10;state-of-the-art results on these datasets. AKCES-GEC is published under CC&#10;BY-NC-SA 4.0 license at https://hdl.handle.net/11234/1-3057 and the source code&#10;of the GEC model is available at&#10;https://github.com/ufal/low-resource-gec-wnut2019.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2005.11849" label="2005.11849">
        <attvalues>
          <attvalue for="0" value="Stronger Baselines for Grammatical Error Correction Using Pretrained&#10;  Encoder-Decoder Model" />
          <attvalue for="1" value="  Studies on grammatical error correction (GEC) have reported the effectiveness&#10;of pretraining a Seq2Seq model with a large amount of pseudodata. However, this&#10;approach requires time-consuming pretraining for GEC because of the size of the&#10;pseudodata. In this study, we explore the utility of bidirectional and&#10;auto-regressive transformers (BART) as a generic pretrained encoder-decoder&#10;model for GEC. With the use of this generic pretrained model for GEC, the&#10;time-consuming pretraining can be eliminated. We find that monolingual and&#10;multilingual BART models achieve high performance in GEC, with one of the&#10;results being comparable to the current strong results in English GEC. Our&#10;implementations are publicly available at GitHub&#10;(https://github.com/Katsumata420/generic-pretrained-GEC).&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.02976" label="2008.02976">
        <attvalues>
          <attvalue for="0" value="Data Weighted Training Strategies for Grammatical Error Correction" />
          <attvalue for="1" value="  Recent progress in the task of Grammatical Error Correction (GEC) has been&#10;driven by addressing data sparsity, both through new methods for generating&#10;large and noisy pretraining data and through the publication of small and&#10;higher-quality finetuning data in the BEA-2019 shared task. Building upon&#10;recent work in Neural Machine Translation (NMT), we make use of both kinds of&#10;data by deriving example-level scores on our large pretraining data based on a&#10;smaller, higher-quality dataset. In this work, we perform an empirical study to&#10;discover how to best incorporate delta-log-perplexity, a type of example&#10;scoring, into a training schedule for GEC. In doing so, we perform experiments&#10;that shed light on the function and applicability of delta-log-perplexity.&#10;Models trained on scored data achieve state-of-the-art results on common GEC&#10;test sets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.19650" label="2305.19650">
        <attvalues>
          <attvalue for="0" value="Adverbs, Surprisingly" />
          <attvalue for="1" value="  This paper begins with the premise that adverbs are neglected in&#10;computational linguistics. This view derives from two analyses: a literature&#10;review and a novel adverb dataset to probe a state-of-the-art language model,&#10;thereby uncovering systematic gaps in accounts for adverb meaning. We suggest&#10;that using Frame Semantics for characterizing word meaning, as in FrameNet,&#10;provides a promising approach to adverb analysis, given its ability to describe&#10;ambiguity, semantic roles, and null instantiation.&#10;" />
          <attvalue for="2" value="&#10;&#10;Adverbs are the part of speech (POS) that has seen the least&#10;attention in (computational) linguistics, likely due to its challenging&#10;nature \cite{conlon-evens-1992-computers}. As \cite[563]{huddleston_pullum_2002} state,&#10;\enquote{the adverb is a [\dots] residual category [\dots] to which words are assigned if they do not satisfy the more specific criteria for nouns, verbs, adjectives, prepositions, and conjunctions.} &#10;&#10;Syntactically, they modify many POSs, except nouns&#10;(eat porridge quickly, hardly noticeable), or even complete&#10;clauses (Probably, I'll come tomorrow). They are semantically varied \cite{thomason-stalnaker-73}, ranging from&#10;intensifiers/modifiers (absolutely, beautifully) to temporal&#10;and spatial specifications (yesterday, forward), to&#10;so-called speaker-oriented adverbs yielding inferences&#10;about speaker attitudes, beliefs, and evaluations.&#10;Finally, adverbs can occupy different positions in sentences, creating complex issues of&#10;scoping and ambiguity \cite{ALEXIADOU2004677,d06a4ea3203c4175ba571fa2c55cb6f9}. Consider the following sentences:&#10;&#10;\begin{exe}&#10;\singlespacing&#10;\ex &#10;\begin{xlist}[]&#10;\ex \uline{Happily}, they watched TV until dinner.&#10;\ex They \uline{happily} watched TV until dinner.&#10;\ex They watched TV \uline{happily} until dinner.&#10;\ex They watched TV until dinner \uline{happily}.&#10; \end{xlist}&#10;\end{exe}&#10;&#10;While &#10;language users tend to interpret &#10;Ex.~\ref{ex:happilycenter}--\ref{ex:happilyend}&#10;as describing the TV watchers'&#10;mental state, Ex.~\ref{ex:happilyfront}&#10;is ambiguous and can also be read as a positive evaluation of the situation by the speaker.&#10;&#10;In sum, adverbs provide crucial information not just about the where and how of events, but also about attitudes and evaluations. However, relatively little research on adverbs exists in computational linguistics, although lexical factors are generally recognized&#10;as central for many NLP tasks \cite{10.1145/345508.345576}. Lexical information is generally represented either in online dictionaries or by&#10;embeddings extracted from corpora \cite{Turney2010,devlin-etal-2019-bert,peters-etal-2018-deep}. &#10;As a dictionary, WordNet \cite{george90:_five_wordn} lists adverbs&#10;but only provides a relatively impoverished account, while lexicons&#10;for sentiment analysis&#10;\cite{Benamara2007SentimentAA,dragut-fellbaum-2014-role} and hedging detection \cite{jeon-choe-2009-key,islam-etal-2020-lexicon} only&#10;consider specific subtypes of adverbs as to how they modulate the &#10;intensity of adjectives. &#10;On the distributional side, adverbs have been considered&#10;from a derviational perspective&#10;\cite{lazaridou-etal-2013-compositional}; yet, they are rarely scrutinized in&#10;detail. Among the standard benchmarks, only GLUE&#10;\cite{wang-etal-2018-glue} and BLiMP \cite{warstadt2020blimp}&#10;cover adverbs, and then only marginally. The same is true&#10;of approaches that combine dictionaries and embeddings&#10;\cite{faruqui-etal-2015-retrofitting}. As a consequence, SOTA&#10;language models consistently struggle with adverb meaning, as Section \ref{sec:treatment} will demonstrate empirically.&#10;&#10;This paper argues that Frame Semantics&#10;\cite{fillmore85:_frames_seman_under}, as realized in&#10;FrameNet (FN) \cite{Ruppenhoferetal16}, provides an efficacious framework to&#10;articulate the relevant aspects of adverb meaning. Specifically, as Ex. \ref{ex:happily} illustrates, lexical ambiguity&#10;is captured in terms of frame ambiguity. Moreover, inferences about the &#10;arguments of adverbs, typically filled by the speaker and the &#10;lexical unit that the adverb modifies, can be captured and characterized &#10;via the frame elements (i.e.\ semantic roles) of the frame. &#10;Notably, FrameNet mechanisms will account for&#10;null-instantiated roles, allowing it to&#10;hint at unexpressed content in cases like&#10;Example~\ref{ex:speakerNI} (v. Section~\ref{sec:appr-fram-analys} for details).&#10;&#10;\begin{exe}&#10; \ex \begin{xlist}&#10; \ex [$_{Speaker}$ The Minister] reported [$_{Message}$ that the cost had exploded].&#10; \ex [$_{Message}$ The cost had] reportedly [$_{Message}$ exploded].&#10; \end{xlist} &#10;\end{exe}&#10;&#10;In such cases specifically, FrameNet considerations of frame element realization help&#10;to explain the absence of the Speaker semantic role in \ref{ex:speakerNI}.&#10;&#10;Plan of the Paper.&#10;Section~\ref{sec:motivation-nlp} defines the scope of this paper (speaker-oriented adverbs) and shows the lack of accounts for&#10;adverbs in NLP through a literature review. Section~\ref{sec:demo}&#10;presents a probing dataset for speaker-oriented adverbs on the basis of&#10;which it demonstrates empirically that current large language models do&#10;not provide accounts for adverb meaning.&#10;Section~\ref{sec:adverbs-in-framenet} provides general background information on FrameNet, gives details on the framework's approach to the description of adverb meaning, and suggests its use to improve NLP models.&#10;Section~\ref{sec:discussion} concludes the paper.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Frame Semantics, Computer Science, Linguistics, Cognitive Science, Computational Linguistics, Artificial Intelligence, Adverb Analysis" />
        </attvalues>
      </node>
      <node id="2408.00620" label="2408.00620">
        <attvalues>
          <attvalue for="0" value="Are Bigger Encoders Always Better in Vision Large Models?" />
          <attvalue for="1" value="  In recent years, multimodal large language models (MLLMs) have shown strong&#10;potential in real-world applications. They are developing rapidly due to their&#10;remarkable ability to comprehend multimodal information and their inherent&#10;powerful cognitive and reasoning capabilities. Among MLLMs, vision language&#10;models (VLM) stand out for their ability to understand vision information.&#10;However, the scaling trend of VLMs under the current mainstream paradigm has&#10;not been extensively studied. Whether we can achieve better performance by&#10;training even larger models is still unclear. To address this issue, we&#10;conducted experiments on the pretraining stage of MLLMs. We conduct our&#10;experiment using different encoder sizes and large language model (LLM) sizes.&#10;Our findings indicate that merely increasing the size of encoders does not&#10;necessarily enhance the performance of VLMs. Moreover, we analyzed the effects&#10;of LLM backbone parameter size and data quality on the pretraining outcomes.&#10;Additionally, we explored the differences in scaling laws between LLMs and&#10;VLMs.&#10;" />
          <attvalue for="2" value="&#10;In recent years, the rapid development of large language models has revolutionized the field of natural language processing ~\cite{devlin2019bert,touvron2023llama2openfoundation,openai2024gpt4,bai2023qwenvl,du2022glamefficientscalinglanguage}. These models, equipped with an enormous number of parameters, have demonstrated outstanding performance in areas such as translation ~\cite{fan2020englishcentric}, question answering ~\cite{devlin2019bert, Raffel2019ExploringTL}, and text generation ~\cite{brown2020languagemodelsfewshotlearners,du2022glamefficientscalinglanguage}. Due to the powerful capabilities of LLMs, a natural idea is to combine them with multimodal research, leveraging the strong cognitive abilities of LLMs to process information from other modalities ~\cite{openai2024gpt4,chu2024qwen2audiotechnicalreport, Maaz2023VideoChatGPTTD}. &#10;&#10;Among MLLMs, VLMs achieve competitive performance in traditional multimodal tasks such as image classification~\cite{chen2024internvl}, image understanding~\cite{li2023blip2}, and image captioning~\cite{bai2023qwenvl}. Moreover, their excellent language understanding capabilities enable strong performance in text-rich tasks, such as vision question-answering~\cite{llava,llava1.5} and image-text retrieval~\cite{chen2024internvl}.&#10;&#10;Currently, the dominant architecture for VLMs employs a large language model as the backbone. Mechanisms such as cross-attention~\cite{li2023blip2} or linear projectors~\cite{llava} are utilized to connect the vision encoder (e.g., ViT~\cite{dosovitskiy2021imageworth16x16words}) with the LLM backbone~\cite{zhang2024mmllms}. Larger visual encoders possess stronger encoding capabilities, allowing them to extract features more effectively. These features are then transformed by the projectors for the LLMs to understand. We refer to this model architecture as the connected vision paradigm.&#10;&#10;Apart from VLMs, previous research on scaling laws~\cite{kaplan2020scaling,hoffmann2022training} adopts an empirical approach to study the relationship between model performance, parameter size, and the amount of training data. Under the assumption that the scaling trend follows a power-law relationship, researchers fit power-law formulas by training models of different sizes with varying amounts of data. Additionally, scaling laws have been extended to various scenarios, including graph neural networks~\cite{liu2024neural}, data mixing~\cite{ye2024data}, data pruning~\cite{Sorscher2022BeyondNS}, and fine-tuning LLMs~\cite{zhang2024scaling}. Although~\cite{aghajanyan2023scaling} investigated scaling laws in the context of multimodal scenarios, they neglected the connected vision paradigm~\cite{zhang2024mmllms,bai2023qwenvl}, leading to the following challenges:&#10;&#10;C1. Poor Effectiveness: Since no scaling laws have been conducted, it is uncertain whether this paradigm can scale up to achieve better performance, resulting in suboptimal model performance.&#10;&#10;C2. Low Efficiency: Without scaling laws, we do not know how much data or how many parameters to use. Typically, all available data are used for training VLMs, resulting in a waste of data and computational resources.&#10;&#10;To address these issues, we conduct scaling law experiments following~\cite {kaplan2020scaling}. We select the renowned LLaVA1.5~\cite{llava1.5} model as the backbone of our study. To investigate the scaling laws, we utilize models with 7 billion and 13 billion parameters. For the dataset, we choose image-text pairs from CC12M~\cite{changpinyo2021conceptual} and Laion400M~\cite{Schuhmann2021LAION400MOD}, known for their high-quality pairs. We use data sizes from 1 million to 10 million image-text pairs for our scaling law experiments.&#10;&#10;The core contributions of this paper are summarized as follows:&#10;\begin{itemize}&#10; \item New Perspective To the best of our knowledge, we are the first to conduct scaling law studies for the connect vision paradigm. Additionally, we are the first to analyze the limitations of this VLM paradigm.&#10;&#10;\item New Observation We conducted experiments on the pretraining phase of MLLMs using the connect vision paradigm. From the experimental results, we derived a crucial conclusion: simply utilizing a ViT trained using CLIP with more parameters and better performance does not enhance the performance of MLLMs. This suggests that exploring alternative methods is necessary to improve the performance of MLLMs. We also indicate that this issue is highly likely not caused by limitations in the scaling ability of ViT.&#10;\end{itemize}&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Multimodal Language Models, Vision Language Processing, Computer Science, Large Model Scaling, Linguistics, Cognitive Science" />
        </attvalues>
      </node>
      <node id="2308.12966" label="2308.12966">
        <attvalues>
          <attvalue for="0" value="Qwen-VL: A Versatile Vision-Language Model for Understanding,&#10;  Localization, Text Reading, and Beyond" />
          <attvalue for="1" value="  In this work, we introduce the Qwen-VL series, a set of large-scale&#10;vision-language models (LVLMs) designed to perceive and understand both texts&#10;and images. Starting from the Qwen-LM as a foundation, we endow it with visual&#10;capacity by the meticulously designed (i) visual receptor, (ii) input-output&#10;interface, (iii) 3-stage training pipeline, and (iv) multilingual multimodal&#10;cleaned corpus. Beyond the conventional image description and&#10;question-answering, we implement the grounding and text-reading ability of&#10;Qwen-VLs by aligning image-caption-box tuples. The resulting models, including&#10;Qwen-VL and Qwen-VL-Chat, set new records for generalist models under similar&#10;model scales on a broad range of visual-centric benchmarks (e.g., image&#10;captioning, question answering, visual grounding) and different settings (e.g.,&#10;zero-shot, few-shot). Moreover, on real-world dialog benchmarks, our&#10;instruction-tuned Qwen-VL-Chat also demonstrates superiority compared to&#10;existing vision-language chatbots. Code, demo and models are available at&#10;https://github.com/QwenLM/Qwen-VL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2312.14238" label="2312.14238">
        <attvalues>
          <attvalue for="0" value="InternVL: Scaling up Vision Foundation Models and Aligning for Generic&#10;  Visual-Linguistic Tasks" />
          <attvalue for="1" value="  The exponential growth of large language models (LLMs) has opened up numerous&#10;possibilities for multimodal AGI systems. However, the progress in vision and&#10;vision-language foundation models, which are also critical elements of&#10;multi-modal AGI, has not kept pace with LLMs. In this work, we design a&#10;large-scale vision-language foundation model (InternVL), which scales up the&#10;vision foundation model to 6 billion parameters and progressively aligns it&#10;with the LLM, using web-scale image-text data from various sources. This model&#10;can be broadly applied to and achieve state-of-the-art performance on 32&#10;generic visual-linguistic benchmarks including visual perception tasks such as&#10;image-level or pixel-level recognition, vision-language tasks such as zero-shot&#10;image/video classification, zero-shot image/video-text retrieval, and link with&#10;LLMs to create multi-modal dialogue systems. It has powerful visual&#10;capabilities and can be a good alternative to the ViT-22B. We hope that our&#10;research could contribute to the development of multi-modal large models. Code&#10;and models are available at https://github.com/OpenGVLab/InternVL.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2402.02054" label="2402.02054">
        <attvalues>
          <attvalue for="0" value="Neural Scaling Laws on Graphs" />
          <attvalue for="1" value="  Deep graph models (e.g., graph neural networks and graph transformers) have&#10;become important techniques for leveraging knowledge across various types of&#10;graphs. Yet, the scaling properties of deep graph models have not been&#10;systematically investigated, casting doubt on the feasibility of achieving&#10;large graph models through enlarging the model and dataset sizes. In this work,&#10;we delve into neural scaling laws on graphs from both model and data&#10;perspectives. We first verify the validity of such laws on graphs, establishing&#10;formulations to describe the scaling behaviors. For model scaling, we&#10;investigate the phenomenon of scaling law collapse and identify overfitting as&#10;the potential reason. Moreover, we reveal that the model depth of deep graph&#10;models can impact the model scaling behaviors, which differ from observations&#10;in other domains such as CV and NLP. For data scaling, we suggest that the&#10;number of graphs can not effectively metric the graph data volume in scaling&#10;law since the sizes of different graphs are highly irregular. Instead, we&#10;reform the data scaling law with the number of edges as the metric to address&#10;the irregular graph sizes. We further demonstrate the reformed law offers a&#10;unified view of the data scaling behaviors for various fundamental graph tasks&#10;including node classification, link prediction, and graph classification. This&#10;work provides valuable insights into neural scaling laws on graphs, which can&#10;serve as an essential step toward large graph models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2403.16952" label="2403.16952">
        <attvalues>
          <attvalue for="0" value="Data Mixing Laws: Optimizing Data Mixtures by Predicting Language&#10;  Modeling Performance" />
          <attvalue for="1" value="  Pretraining data of large language models composes multiple domains (e.g.,&#10;web texts, academic papers, codes), whose mixture proportions crucially impact&#10;the competence of outcome models. While existing endeavors rely on heuristics&#10;or qualitative strategies to tune the proportions, we discover the quantitative&#10;predictability of model performance regarding the mixture proportions in&#10;function forms, which we refer to as the data mixing laws. Fitting such&#10;functions on sample mixtures unveils model performance on unseen mixtures&#10;before actual runs, thus guiding the selection of an ideal data mixture.&#10;Furthermore, we propose nested use of the scaling laws of training steps, model&#10;sizes, and our data mixing law to enable predicting the performance of large&#10;models trained on massive data under various mixtures with only small-scale&#10;training. Moreover, experimental results verify that our method effectively&#10;optimizes the training mixture of a 1B model trained for 100B tokens in&#10;RedPajama, reaching a performance comparable to the one trained for 48% more&#10;steps on the default mixture. Extending the application of data mixing laws to&#10;continual training accurately predicts the critical mixture proportion that&#10;avoids catastrophic forgetting and outlooks the potential for dynamic data&#10;schedules&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.14486" label="2206.14486">
        <attvalues>
          <attvalue for="0" value="Beyond neural scaling laws: beating power law scaling via data pruning" />
          <attvalue for="1" value="  Widely observed neural scaling laws, in which error falls off as a power of&#10;the training set size, model size, or both, have driven substantial performance&#10;improvements in deep learning. However, these improvements through scaling&#10;alone require considerable costs in compute and energy. Here we focus on the&#10;scaling of error with dataset size and show how in theory we can break beyond&#10;power law scaling and potentially even reduce it to exponential scaling instead&#10;if we have access to a high-quality data pruning metric that ranks the order in&#10;which training examples should be discarded to achieve any pruned dataset size.&#10;We then test this improved scaling prediction with pruned dataset size&#10;empirically, and indeed observe better than power law scaling in practice on&#10;ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of&#10;finding high-quality pruning metrics, we perform the first large-scale&#10;benchmarking study of ten different data pruning metrics on ImageNet. We find&#10;most existing high performing metrics scale poorly to ImageNet, while the best&#10;are computationally intensive and require labels for every image. We therefore&#10;developed a new simple, cheap and scalable self-supervised pruning metric that&#10;demonstrates comparable performance to the best supervised metrics. Overall,&#10;our work suggests that the discovery of good data-pruning metrics may provide a&#10;viable path forward to substantially improved neural scaling laws, thereby&#10;reducing the resource costs of modern deep learning.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.02656" label="1911.02656">
        <attvalues>
          <attvalue for="0" value="Invariance and identifiability issues for word embeddings" />
          <attvalue for="1" value="  Word embeddings are commonly obtained as optimizers of a criterion function&#10;$f$ of a text corpus, but assessed on word-task performance using a different&#10;evaluation function $g$ of the test data. We contend that a possible source of&#10;disparity in performance on tasks is the incompatibility between classes of&#10;transformations that leave $f$ and $g$ invariant. In particular, word&#10;embeddings defined by $f$ are not unique; they are defined only up to a class&#10;of transformations to which $f$ is invariant, and this class is larger than the&#10;class to which $g$ is invariant. One implication of this is that the apparent&#10;superiority of one word embedding over another, as measured by word task&#10;performance, may largely be a consequence of the arbitrary elements selected&#10;from the respective solution sets. We provide a formal treatment of the above&#10;identifiability issue, present some numerical examples, and discuss possible&#10;resolutions.&#10;" />
          <attvalue for="2" value="&#10;&#10;Word embeddings map a text corpus, say $X$, to a collection of vectors &#10;$V = (v_1,..., v_p)$&#10;where each $v_j \in \mathbb{R}^d$, for a prescribed embedding dimension $d$, represents one of $p$ words in the corpus. Different word embedding models can be cast as the solution of an optimisation&#10;&#10;\begin{equation}&#10;\underset{U,V}{arg min} \,\, F(X,U,V)=\underset{U,V}{arg min} \,\, f(X,UV), &#10;\end{equation} &#10;for particular corpus representation $X$ and objective function $f$, where $U=(u_1,\ldots,u_n)^T$ are vectors in $\mathbb{R}^n$ representing contexts, typically not of main interest. The setup subsumes some popular embedding techniques such as Latent Semantic Analysis (LSA) \cite{deerwester-lsa}, word2vec \cite{mikolov-distributed, mikolov-efficient}, and GloVe \cite{pennington-glove}, wherein the matrices $U$ and $V$ appear in a suitably chosen $f$ only through their product $UV$. &#10;&#10;Once a word embedding $V$ is constructed by solving \ref{eqn:word:embedding:defn}, the embedding is evaluated on its performance in tasks, including identifying word similarity (given word $a$, identify words with similar meanings), and word analogy (for the statement &quot;$a$ is to $b$ what $c$ is to $x$&quot;, given $a$, $b$ and $c$, identify $x$). Similarities or analogies can be computed from $V$, then performance evaluated against a test data set $D$ containing human-assigned judgements as&#10;\begin{equation}&#10;g(D,V), &#10;\end{equation}&#10;for some function $g$. Constructing word embeddings is &quot;unsupervised&quot; with respect to the evaluation task in the sense that $V$ is determined from \ref{eqn:word:embedding:defn} independently of the choice of $g$ and the data $D$ in \ref{eqn:g:defn}, although $f$ typically entails free parameters that may, consciously or not, be chosen to optimize \ref{eqn:g:defn} \cite{levy-goldberg}.&#10;&#10;Different word embedding models, identified as different $f$ in \ref{eqn:word:embedding:defn}, are often compared based on performance in word tasks in the sense of $g$ in \ref{eqn:g:defn}. But there are &#10;several reasons why comparing performance in this way is difficult. First: performance may be affected less by the structure of model $f$, and more by the number of free parameters it entails and how well they have been tuned \cite{levy-goldberg}. Second: for many embeddings, solving \ref{eqn:word:embedding:defn} entails a Monte Carlo optimisation, so different runs with identical $f$ will result in different realisations of $V$ and hence different values of $g(D,V)$. Third, more subtle and often conflated with the first and second: for most embedding models $f$, \ref{eqn:word:embedding:defn} does not uniquely identify $V$ --- $V$ is said to be non-identifiable --- and different solutions, $V$, each equally optimal with respect to \ref{eqn:word:embedding:defn}, correspond to different values of $g(D,V)$. &#10;&#10;This raises the disconcerting question: can apparent differences in performances in word tasks as evaluated with $g$ be substantially attributed to the arbitrary selection of a solution $V$ from the set of solutions of $f$? In this paper we explore the non-identifiability of $V$, particularly with respect to the class of non-singular transformations $C$ for which $f(X,UV) = f(X,UC^{-1}CV)$ but $g(D,V) \neq g(D,CV)$, and the consequences for constructing and evaluating word embeddings. Specifically, our contributions are as follows. &#10;&#10;\begin{enumerate}&#10; \item For $g$ defined using inner products of embedded word vectors (e.g. Cosine similarity) in $d$ dimensions, we characterise the subset $\mathcal{F}_d$ contained in the set of non-singular transformations to which $g$ is not invariant. &#10; \item We study a widely used strategy for constructing word embeddings that involves multiplying a &quot;base&quot; embedding by a powered matrix of singular values, and show that this amounts to exploring a one-dimensional subset of the optimal solutions.&#10; \item We discuss resolutions to the non-identifiability, including (i) constraining the set of solutions of $f$ to ensure compatibility with invariances of $g$, and (ii) optimizing over the solutions of $f$ with respect to $g$ in a supervised learning sense.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Word Embeddings, Evaluation Metrics, Mathematics, Transformation Invariance, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1908.11020" label="1908.11020">
        <attvalues>
          <attvalue for="0" value="Regularized Context Gates on Transformer for Machine Translation" />
          <attvalue for="1" value="  Context gates are effective to control the contributions from the source and&#10;target contexts in the recurrent neural network (RNN) based neural machine&#10;translation (NMT). However, it is challenging to extend them into the advanced&#10;Transformer architecture, which is more complicated than RNN. This paper first&#10;provides a method to identify source and target contexts and then introduce a&#10;gate mechanism to control the source and target contributions in Transformer.&#10;In addition, to further reduce the bias problem in the gate mechanism, this&#10;paper proposes a regularization method to guide the learning of the gates with&#10;supervision automatically generated using pointwise mutual information.&#10;Extensive experiments on 4 translation datasets demonstrate that the proposed&#10;model obtains an averaged gain of 1.0 BLEU score over a strong Transformer&#10;baseline.&#10;" />
          <attvalue for="2" value="&#10;&#10;An essence to modeling translation is how to learn an effective context from a&#10;sentence pair. Statistical machine translation (SMT) models the source context&#10;from the source-side of a translation model and models the target context from&#10;a target-side language model~\cite{koehn2003statistical, koehn2009statistical,&#10;chiang2005hierarchical}. These two models are trained independently. On the&#10;contrary, neural machine translation (NMT) advocates a unified manner to&#10;jointly learn source and target context using an encoder-decoder framework with&#10;an attention mechanism, leading to substantial gains over SMT in translation&#10;quality~\cite{sutskever2014sequence, bahdanau2014neural,&#10;gehring2017convolutional, vaswani2017attention}. Prior work on attention&#10;mechanism~\cite{luong2015effective, liu2016neural, mi2016supervised,&#10;chen2018syntax, li2018target, elbayad2018pervasive, yang2020neural} have shown&#10;a better context representation is helpful to translation performance.&#10;&#10;However, a standard NMT system is incapable of effectively controlling the&#10;contributions from source and target contexts~\cite{he2018layer} to deliver&#10;highly adequate translations as shown in Figure~\ref{fig:example}. As a result,&#10;\cite{tu2017context} carefully designed context gates to dynamically control&#10;the influence from source and target contexts and observed significant&#10;improvements in the recurrent neural network (RNN) based NMT. Although&#10;Transformer~\cite{vaswani2017attention} delivers significant gains over RNN for&#10;translation, there are still one third translation errors related to context&#10;control problem as described in Section~\ref{subsec:error}. Obviously, it is&#10;feasible to extend the context gates in RNN based NMT into Transformer, but an&#10;obstacle to accomplishing this goal is the complicated architecture in&#10;Transformer, where the source and target words are tightly coupled. Thus, it is&#10;challenging to put context gates into practice in Transformer.&#10;&#10;In this paper, under the Transformer architecture, we firstly provide a way to&#10;define the source and target contexts and then obtain our model by combining&#10;both source and target contexts with context gates, which actually induces a&#10;probabilistic model indicating whether the next generated word is contributed&#10;from the source or target sentence~\cite{li2019word}. In our preliminary&#10;experiments, this model only achieves modest gains over Transformer because the&#10;context selection error reduction is very limited as described in&#10;Section~\ref{subsec:error}. To further address this issue, we propose a&#10;probabilistic model whose loss function is derived from external supervision as&#10;regularization for the context gates. This probabilistic model is jointly&#10;trained with the context gates in NMT. As it is too costly to annotate this&#10;supervision for a large-scale training corpus manually, we instead propose a&#10;simple yet effective method to automatically generate supervision using&#10;pointwise mutual information, inspired by word collocation&#10;~\cite{bouma2009normalized}. In this way, the resulting NMT model is capable&#10;of controlling the contributions from source and target contexts effectively.&#10;&#10;We conduct extensive experiments on 4 benchmark datasets, and experimental&#10;results demonstrate that the proposed gated model obtains an averaged&#10;improvement of 1.0 BLEU point over corresponding strong Transformer baselines.&#10;In addition, we design a novel analysis to show that the improvement of&#10;translation performance is indeed caused by relieving the problem of wrongly&#10;focusing on the source or target context.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Context Control Mechanisms, Mathematics, Neural Machine Translation, Transformer Architecture" />
        </attvalues>
      </node>
      <node id="1608.06043" label="1608.06043">
        <attvalues>
          <attvalue for="0" value="Context Gates for Neural Machine Translation" />
          <attvalue for="1" value="  In neural machine translation (NMT), generation of a target word depends on&#10;both source and target contexts. We find that source contexts have a direct&#10;impact on the adequacy of a translation while target contexts affect the&#10;fluency. Intuitively, generation of a content word should rely more on the&#10;source context and generation of a functional word should rely more on the&#10;target context. Due to the lack of effective control over the influence from&#10;source and target contexts, conventional NMT tends to yield fluent but&#10;inadequate translations. To address this problem, we propose context gates&#10;which dynamically control the ratios at which source and target contexts&#10;contribute to the generation of target words. In this way, we can enhance both&#10;the adequacy and fluency of NMT with more careful control of the information&#10;flow from contexts. Experiments show that our approach significantly improves&#10;upon a standard attention-based NMT system by +2.3 BLEU points.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2206.14982" label="2206.14982">
        <attvalues>
          <attvalue for="0" value="Building Multilingual Machine Translation Systems That Serve Arbitrary&#10;  X-Y Translations" />
          <attvalue for="1" value="  Multilingual Neural Machine Translation (MNMT) enables one system to&#10;translate sentences from multiple source languages to multiple target&#10;languages, greatly reducing deployment costs compared with conventional&#10;bilingual systems. The MNMT training benefit, however, is often limited to&#10;many-to-one directions. The model suffers from poor performance in one-to-many&#10;and many-to-many with zero-shot setup. To address this issue, this paper&#10;discusses how to practically build MNMT systems that serve arbitrary X-Y&#10;translation directions while leveraging multilinguality with a two-stage&#10;training strategy of pretraining and finetuning. Experimenting with the WMT'21&#10;multilingual translation task, we demonstrate that our systems outperform the&#10;conventional baselines of direct bilingual models and pivot translation models&#10;for most directions, averagely giving +6.0 and +4.1 BLEU, without the need for&#10;architecture change or extra data collection. Moreover, we also examine our&#10;proposed approach in an extremely large-scale data setting to accommodate&#10;practical deployment scenarios.&#10;" />
          <attvalue for="2" value="&#10;Multilingual Neural Machine Translation (MNMT), which enables one system to serve translation for multiple directions, has attracted much attention in the machine translation area~\cite{zoph-knight-2016-multi,firat-etal-2016-zero}. Because the multilingual capability hugely reduces the deployment cost at training and inference, MNMT has actively been employed as a machine translation system backbone in recent years~\cite{johnson-etal-2017-googles,DBLP:journals/corr/abs-1803-05567}.&#10;&#10;Most MNMT systems are trained with multiple English-centric data for both directions (e.g., English $arrow$ \{French, Chinese\} (En-X) and \{French, Chinese\} $arrow$ English (X-En)). Recent work~\cite{gu-etal-2019-improved,zhang-etal-2020-improving,yang-etal-2021-improving-multilingual} pointed out that such MNMT systems severely face an off-target translation issue, especially in translations from a non-English language X to another non-English language Y. Meanwhile, \cite{freitag-firat-2020-complete} have extended data resources with multi-way aligned data and reported that one complete many-to-many MNMT can be fully supervised, achieving competitive translation performance for all X-Y directions. In our preliminary experiments, we observed that the complete many-to-many training is still as challenging as one-to-many training~\cite{johnson-etal-2017-googles,wang-etal-2020-negative}, since we have introduced more one-to-many translation tasks into the training. Similarly reported in the many-to-many training with zero-shot setup \cite{gu-etal-2019-improved,yang-etal-2021-improving-multilingual}, the complete MNMT model also suffers from capturing correlations in the data for all the X-Y directions as one model training, due to highly imbalanced data. &#10;&#10;In this paper, we propose a two-stage training for complete MNMT systems that serve arbitrary X-Y translations by 1) pretraining a complete multilingual many-to-many model and 2) finetuning the model to effectively transfer knowledge from pretraining to task-specific multilingual systems. Considering that MNMT is a multi-task learner of translation tasks with \lq\lq multiple languages'', the complete multilingual model learns more diverse and general multilingual representations. We transfer the representations to a specifically targeted task via many-to-one multilingual finetuning, and eventually build multiple many-to-one MNMT models that cover all X-Y directions. The experimental results on the WMT'21 multilingual translation task show that our systems have substantial improvement against conventional bilingual approaches and many-to-one multilingual approaches for most directions. Besides, we discuss our proposal in the light of feasible deployment scenarios and show that the proposed approach also works well in an extremely large-scale data setting.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Data Science, Neural Machine Learning, Artificial Intelligence, Multilingual Translation, Language Model Optimization" />
        </attvalues>
      </node>
      <node id="2010.10239" label="2010.10239">
        <attvalues>
          <attvalue for="0" value="Complete Multilingual Neural Machine Translation" />
          <attvalue for="1" value="  Multilingual Neural Machine Translation (MNMT) models are commonly trained on&#10;a joint set of bilingual corpora which is acutely English-centric (i.e. English&#10;either as the source or target language). While direct data between two&#10;languages that are non-English is explicitly available at times, its use is not&#10;common. In this paper, we first take a step back and look at the commonly used&#10;bilingual corpora (WMT), and resurface the existence and importance of implicit&#10;structure that existed in it: multi-way alignment across examples (the same&#10;sentence in more than two languages). We set out to study the use of multi-way&#10;aligned examples to enrich the original English-centric parallel corpora. We&#10;reintroduce this direct parallel data from multi-way aligned corpora between&#10;all source and target languages. By doing so, the English-centric graph expands&#10;into a complete graph, every language pair being connected. We call MNMT with&#10;such connectivity pattern complete Multilingual Neural Machine Translation&#10;(cMNMT) and demonstrate its utility and efficacy with a series of experiments&#10;and analysis. In combination with a novel training data sampling strategy that&#10;is conditioned on the target language only, cMNMT yields competitive&#10;translation quality for all language pairs. We further study the size effect of&#10;multi-way aligned data, its transfer learning capabilities and how it eases&#10;adding a new language in MNMT. Finally, we stress test cMNMT at scale and&#10;demonstrate that we can train a cMNMT model with up to 111*112=12,432 language&#10;pairs that provides competitive translation quality for all language pairs.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.06514" label="2204.06514">
        <attvalues>
          <attvalue for="0" value="Scalable Training of Language Models using JAX pjit and TPUv4" />
          <attvalue for="1" value="  Modern large language models require distributed training strategies due to&#10;their size. The challenges of efficiently and robustly training them are met&#10;with rapid developments on both software and hardware frontiers. In this&#10;technical report, we explore challenges and design decisions associated with&#10;developing a scalable training framework, and present a quantitative analysis&#10;of efficiency improvements coming from adopting new software and hardware&#10;solutions.&#10;" />
          <attvalue for="2" value="&#10;&#10;Scaling up is one of the most common ways of obtaining better language models~\cite{T5}. Once the model size becomes large enough to prohibit fitting the entire model on a single device, new challenges arise. On the hardware side, extensive amounts of compute resources with large memory and fast interconnect are needed. On the software side, algorithms need to be developed that efficiently utilize that hardware, and optimize the time and resources necessary to train a model.&#10;&#10;This technical report explores the challenges our team has faced when scaling language models to hundreds of billions of parameters, and how our proprietary framework, FAX, is designed to address those challenges. We focus on the breakthroughs in training efficiency achieved by using JAX~\cite{jax} to enable tensor and data parallelism through GSPMD~\cite{gspmd}, XLA, and Google Cloud TPU VMs~\cite{tpu-vm}, highlighting the use of recently released TPU~v4 Pods~\cite{tpu-v4}.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Information Technology, Distributed Training Strategies, Engineering, Data Science, Mathematics, Scalable Framework Design" />
        </attvalues>
      </node>
      <node id="2105.04663" label="2105.04663">
        <attvalues>
          <attvalue for="0" value="GSPMD: General and Scalable Parallelization for ML Computation Graphs" />
          <attvalue for="1" value="  We present GSPMD, an automatic, compiler-based parallelization system for&#10;common machine learning computations. It allows users to write programs in the&#10;same way as for a single device, then give hints through a few annotations on&#10;how to distribute tensors, based on which GSPMD will parallelize the&#10;computation. Its representation of partitioning is simple yet general, allowing&#10;it to express different or mixed paradigms of parallelism on a wide variety of&#10;models.&#10;  GSPMD infers the partitioning for every operator based on limited user&#10;annotations, making it convenient to scale existing single-device programs. It&#10;solves several technical challenges for production usage, allowing GSPMD to&#10;achieve 50% to 62% compute utilization on up to 2048 Cloud TPUv3 cores for&#10;models with up to one trillion parameters.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2102.09786" label="2102.09786">
        <attvalues>
          <attvalue for="0" value="An Empirical Study on Measuring the Similarity of Sentential Arguments&#10;  with Language Model Domain Adaptation" />
          <attvalue for="1" value="  Measuring the similarity between two different sentential arguments is an&#10;important task in argument mining. However, one of the challenges in this field&#10;is that the dataset must be annotated using expertise in a variety of topics,&#10;making supervised learning with labeled data expensive. In this paper, we&#10;investigated whether this problem could be alleviated through transfer&#10;learning. We first adapted a pretrained language model to a domain of interest&#10;using self-supervised learning. Then, we fine-tuned the model to a task of&#10;measuring the similarity between sentences taken from different domains. Our&#10;approach improves a correlation with human-annotated similarity scores compared&#10;to competitive baseline models on the Argument Facet Similarity dataset in an&#10;unsupervised setting. Moreover, we achieve comparable performance to a fully&#10;supervised baseline model by using only about 60% of the labeled data samples.&#10;We believe that our work suggests the possibility of a generalized argument&#10;clustering model for various argumentative topics.&#10;" />
          <attvalue for="2" value="&#10;Providing diverse opinions on a controversial issue is one of the most important functions of argument mining. &#10;To this end, methods for grouping relevant arguments within a given topic by their similarities~\cite{misra-etal-2016-measuring,reimers-etal-2019-classification,chen-etal-2019-seeing} should be developed to prohibit redundant outcomes (argument clustering). &#10;This step plays a crucial role in preventing users from being overwhelmed by the number of retrieved arguments and in clarifying the inconspicuous aspects.&#10;&#10;However, obtaining a sufficiently large labeled dataset is usually time-consuming and expensive.&#10;A continual annotation and training process for an unseen topic is also required to avoid performance degradation.&#10;To address this, various domain adaptation methods~\cite{li2019transferable,das2019learning,wang2019adversarial,cao2019unsupervised} have been recently explored. &#10;These studies aimed to appropriately transfer the knowledge within the domain or task.&#10;In particular, several studies found that continual pretraining of a language model (e.g., BERT~\cite{devlin-etal-2019-bert} and RoBERTa~\cite{liu2019roberta}) is effective with both unsupervised domain adaptation~\cite{ma-etal-2019-domain,rietzler2019adapt} and general supervised learning~\cite{howard2018universal,gururangan2020don}.&#10;&#10;In this study, we attempted to alleviate the low-resource problem of an argument clustering task by leveraging the recent transfer learning strategies. &#10;Specifically, we fine-tuned BERT using a semantic textual similarity~(STS) task to transfer the ability to measure the similarity between two sentences. &#10;Concurrently, we adapted the model to sentences from domains of interest. &#10;These two methods can drive the model to encode the proper representation, in the aspects of both domain and task.&#10;&#10;We evaluated our approach under various conditions including the use of the labeled target dataset and the order of training. &#10;Experimental results show that our approach improved correlation with human-annotated similarity scores against competitive baseline models in an unsupervised setting for the Argument Facet Similarity dataset~(AFS)~\cite{misra-etal-2016-measuring}.&#10;The sample efficiency was also improved, in that comparable performance to a fully supervised baseline model was obtained by using only about 60\% of the labeled dataset.&#10;&#10;Our contributions are as follows: (1) We formulate the task that measures the similarity between sentential arguments into an unsupervised domain adaptation problem.&#10;(2) We investigate various strategies to adapt the pretrained language model into the desired domain and task. &#10;(3) Our proposed approach constantly achieves higher correlation scores than strong baseline models in unsupervised, low-resource, and fully-supervised settings. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Argument Mining, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing, Transfer Learning" />
        </attvalues>
      </node>
      <node id="1906.09821" label="1906.09821">
        <attvalues>
          <attvalue for="0" value="Classification and Clustering of Arguments with Contextualized Word&#10;  Embeddings" />
          <attvalue for="1" value="  We experiment with two recent contextualized word embedding methods (ELMo and&#10;BERT) in the context of open-domain argument search. For the first time, we&#10;show how to leverage the power of contextualized word embeddings to classify&#10;and cluster topic-dependent arguments, achieving impressive results on both&#10;tasks and across multiple datasets. For argument classification, we improve the&#10;state-of-the-art for the UKP Sentential Argument Mining Corpus by 20.8&#10;percentage points and for the IBM Debater - Evidence Sentences dataset by 7.4&#10;percentage points. For the understudied task of argument clustering, we propose&#10;a pre-training step which improves by 7.8 percentage points over strong&#10;baselines on a novel dataset, and by 12.3 percentage points for the Argument&#10;Facet Similarity (AFS) Corpus.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.14192" label="1910.14192">
        <attvalues>
          <attvalue for="0" value="Transferable End-to-End Aspect-based Sentiment Analysis with Selective&#10;  Adversarial Learning" />
          <attvalue for="1" value="  Joint extraction of aspects and sentiments can be effectively formulated as a&#10;sequence labeling problem. However, such formulation hinders the effectiveness&#10;of supervised methods due to the lack of annotated sequence data in many&#10;domains. To address this issue, we firstly explore an unsupervised domain&#10;adaptation setting for this task. Prior work can only use common syntactic&#10;relations between aspect and opinion words to bridge the domain gaps, which&#10;highly relies on external linguistic resources. To resolve it, we propose a&#10;novel Selective Adversarial Learning (SAL) method to align the inferred&#10;correlation vectors that automatically capture their latent relations. The SAL&#10;method can dynamically learn an alignment weight for each word such that more&#10;important words can possess higher alignment weights to achieve fine-grained&#10;(word-level) adaptation. Empirically, extensive experiments demonstrate the&#10;effectiveness of the proposed SAL method.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1911.06137" label="1911.06137">
        <attvalues>
          <attvalue for="0" value="Unsupervised Domain Adaptation on Reading Comprehension" />
          <attvalue for="1" value="  Reading comprehension (RC) has been studied in a variety of datasets with the&#10;boosted performance brought by deep neural networks. However, the&#10;generalization capability of these models across different domains remains&#10;unclear. To alleviate this issue, we are going to investigate unsupervised&#10;domain adaptation on RC, wherein a model is trained on labeled source domain&#10;and to be applied to the target domain with only unlabeled samples. We first&#10;show that even with the powerful BERT contextual representation, the&#10;performance is still unsatisfactory when the model trained on one dataset is&#10;directly applied to another target dataset. To solve this, we provide a novel&#10;conditional adversarial self-training method (CASe). Specifically, our approach&#10;leverages a BERT model fine-tuned on the source dataset along with the&#10;confidence filtering to generate reliable pseudo-labeled samples in the target&#10;domain for self-training. On the other hand, it further reduces domain&#10;distribution discrepancy through conditional adversarial learning across&#10;domains. Extensive experiments show our approach achieves comparable accuracy&#10;to supervised models on multiple large-scale benchmark datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2306.02405" label="2306.02405">
        <attvalues>
          <attvalue for="0" value="An Information-Theoretic Analysis of Self-supervised Discrete&#10;  Representations of Speech" />
          <attvalue for="1" value="  Self-supervised representation learning for speech often involves a&#10;quantization step that transforms the acoustic input into discrete units.&#10;However, it remains unclear how to characterize the relationship between these&#10;discrete units and abstract phonetic categories such as phonemes. In this&#10;paper, we develop an information-theoretic framework whereby we represent each&#10;phonetic category as a distribution over discrete units. We then apply our&#10;framework to two different self-supervised models (namely wav2vec 2.0 and XLSR)&#10;and use American English speech as a case study. Our study demonstrates that&#10;the entropy of phonetic distributions reflects the variability of the&#10;underlying speech sounds, with phonetically similar sounds exhibiting similar&#10;distributions. While our study confirms the lack of direct, one-to-one&#10;correspondence, we find an intriguing, indirect relationship between phonetic&#10;categories and discrete units.&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-supervised learning (SSL) for the speech modality is an active area of research that aims to develop models that build meaningful speech representations from raw audio without any explicit labels or transcriptions (see \cite{9893562} for an overview). &#10;These models can be further adapted for downstream tasks such as automatic speech recognition and speaker identification, and have become the state-of-the-art approach even when limited labeled data are available \cite{van2018representation, Schneider2019wav2vecUP, baevski2020wav2vec, hsu2021hubert}. Recently, it has become a common practice to include a quantization module within the architecture of SSL speech models that transforms the acoustic input into a sequence of discrete entities. &#10;Besides representing the complex acoustic signal in a compact and computationally efficient manner, learning discrete representations of speech can also facilitate training large SSL speech models using a masked language modeling objective similar to those employed in natural language processing (e.g., BERT \cite{devlin-etal-2019-bert}). &#10;&#10;Nevertheless, the nature of the discrete units learned via self-supervision remains an under-explored area of research. &#10;A key question is whether these discrete representations correspond to abstract phonetic categories such as phonemes.&#10;A few recent studies have investigated the discrete units from a neural network interpretability point of view \cite{higy2021discrete, nguyen2022discrete, wells2022phonetic, sicherman2023analysing}.&#10;The analysis in \cite{wells2022phonetic} showed that the discrete units correspond to low-level ``sub-phonetic'' events---rather than high-level phonetic categories---since they are sensitive to context-dependent and non-phonemic variations in speech. &#10;In \cite{sicherman2023analysing}, the authors concluded that there exists a strong correspondence between discrete units and phonemes, and attributed the lack of consistent phoneme-to-unit mapping to variations in phonological contexts. &#10;These findings seem to be contradictory and rely on different definitions of the term ``phoneme'', and thus remain inconclusive.&#10;&#10;Although information theory was initially proposed as a mathematical theory of communication \cite{shannon2001mathematical}, it also provides a quantitative framework for measuring the amount of information conveyed by linguistic units, such as words or sounds. &#10;Information theory has been adopted as a framework to study various aspects of linguistic structure, including phonology \cite{10.1162/tacl_a_00296, pimentel-etal-2021-surprisal}, morphology \cite{rathi-etal-2021-information, wu-etal-2019-morphological}, and syntax \cite{Hahn2018AnIE, futrell-etal-2015-quantifying}. &#10;In this paper, we build on this line of research and develop information-theoretic metrics to analyze the correspondence between phonetic categories and discrete units. &#10;Concretely, we make the following contributions: &#10;&#10;\begin{itemize}[] \item We develop an empirical approach to represent each phonetic category as a probability distribution over discrete units using two self-supervised pre-trained models: English wav2vec 2.0 (henceforth w2v2) and multilingual wav2vec-XLSR (henceforth xlsr) (\S2).&#10; \item We characterize each phonetic category using the notion of information entropy and demonstrate that entropy quantifies acoustic-phonetic variability (\S4).&#10; \item We quantify the dissimilarity between phonetic distributions using Jensen-Shannon divergence and illustrate that this metric highly reflects feature-based phonetic similarity (\S5). &#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Theory, Computer Science, Linguistics, Speech Representation Learning, Mathematics, Information Theoretic Frameworks, Phonetic Category Analysis" />
        </attvalues>
      </node>
      <node id="2105.05582" label="2105.05582">
        <attvalues>
          <attvalue for="0" value="Discrete representations in neural models of spoken language" />
          <attvalue for="1" value="  The distributed and continuous representations used by neural networks are at&#10;odds with representations employed in linguistics, which are typically&#10;symbolic. Vector quantization has been proposed as a way to induce discrete&#10;neural representations that are closer in nature to their linguistic&#10;counterparts. However, it is not clear which metrics are the best-suited to&#10;analyze such discrete representations. We compare the merits of four commonly&#10;used metrics in the context of weakly supervised models of spoken language. We&#10;compare the results they show when applied to two different models, while&#10;systematically studying the effect of the placement and size of the&#10;discretization layer. We find that different evaluation regimes can give&#10;inconsistent results. While we can attribute them to the properties of the&#10;different metrics in most cases, one point of concern remains: the use of&#10;minimal pairs of phoneme triples as stimuli disadvantages larger discrete unit&#10;inventories, unlike metrics applied to complete utterances. Furthermore, while&#10;in general vector quantization induces representations that correlate with&#10;units posited in linguistics, the strength of this correlation is only&#10;moderate.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2311.04930" label="2311.04930">
        <attvalues>
          <attvalue for="0" value="Large language models implicitly learn to straighten neural sentence&#10;  trajectories to construct a predictive representation of natural language" />
          <attvalue for="1" value="  Predicting upcoming events is critical to our ability to interact with our&#10;environment. Transformer models, trained on next-word prediction, appear to&#10;construct representations of linguistic input that can support diverse&#10;downstream tasks. But how does a predictive objective shape such&#10;representations? Inspired by recent work in vision (Henaff et al., 2019), we&#10;test a hypothesis about predictive representations of autoregressive&#10;transformers. In particular, we test whether the neural trajectory of a&#10;sentence becomes progressively straighter as it passes through the network&#10;layers. The key insight is that straighter trajectories should facilitate&#10;prediction via linear extrapolation. We quantify straightness using a&#10;1-dimensional curvature metric, and present four findings in support of the&#10;trajectory straightening hypothesis: i) In trained models, the curvature&#10;decreases from the early to the deeper layers of the network. ii) Models that&#10;perform better on the next-word prediction objective exhibit greater decreases&#10;in curvature, suggesting that this improved ability to straighten sentence&#10;trajectories may be the driver of better language modeling performance. iii)&#10;Given the same linguistic context, the sequences that are generated by the&#10;model have lower curvature than the actual continuations observed in a language&#10;corpus, suggesting that the model favors straighter trajectories for making&#10;predictions. iv) A consistent relationship holds between the average curvature&#10;and the average surprisal of sentences in the deep model layers, such that&#10;sentences with straighter trajectories also have lower surprisal. Importantly,&#10;untrained models do not exhibit these behaviors. In tandem, these results&#10;support the trajectory straightening hypothesis and provide a possible&#10;mechanism for how the geometry of the internal representations of&#10;autoregressive models supports next word prediction.&#10;" />
          <attvalue for="2" value="&#10;\par&#10;Biological systems, like brains, and artificial systems, like deep neural networks, construct internal representations in the service of their internal or external objectives. Certain objectives appear to yield representations that are useful across diverse behaviors. For example, representations that are predictive of incoming input have been argued to be useful in both biological systems - across perception, action, and cognition \cite{Rao1999-bl,Palmer2015-py,Shadmehr2010-pk,hohwy2008predictive,jessup2010error,Shain2020-je,Frank2015-rf} - and in artificial systems across domains \cite{Van_den_Oord2018-kf,Radford_dl}. Two general approaches have been commonly used in modeling predictive processing in the brain. The first approach leverages information theory \cite{Shannon1949-xp} to quantify the relationship between the past and current neural states and future inputs (e.g., \cite{Bialek2007-qc,Tishby2000-id,Wiskott2002-up,Palmer2015-py}). A key limitation of this kind of an approach is that they do not specify how the information about past inputs is actually used to make predictions (see \cite{Henaff2018-qw} for discussion). The second approach instead focuses on circuit-level motifs—specifically interactions between lower-level and higher-level areas— that give rise to a predictive, top-down signal, and the bottom-up error signal (e.g., \cite{Rao1999-bl}). This approach faces the challenge of specifying what information is represented in high-level cortical areas, which, for many domains, remains not well understood.&#10;&#10;Recently, in the context of visual processing, Henaff (2018; see also \cite{Henaff2019-nb}) have developed an approach to temporal prediction at an intermediate level of abstraction. In contrast to the information-theory-grounded approaches, which focus on predicting upcoming inputs \cite{Palmer2015-py}, this approach focuses on the internal representation states and on predicting future internal states. The critical insight comes from vision: because a sequence of visual inputs to the retina evolves in a nonlinear manner, and are difficult to extrapolate, visual system performs a series of transformation to make them easier to predict. The representation of input sequence is transformed to result in straighter the trajectory in the internal state, and allow for linear extrapolation of future states of the sequence. \cite{Henaff2019-nb} found support for this straightening hypothesis in behavioral psychophysics experiments and in neural recordings in the early visual areas of macaques \cite{Henaff2021-dw}. They also tested the predictions of this hypothesis in AlexNet \cite{Krizhevsky2012-wq}, an early convolutional neural network for vision, but did not observe representation straightening. They hypothesized that representation straightening may only emerge in systems where the objective has to do with prediction (cf. AlexNet where the objective function is object categorization).&#10;&#10;Language is a domain where information unfolds over time and where prediction is a natural objective function. Indeed, many successful language models use next-word prediction as their core training objective (e.g., \cite{Radford_dl}). As a result, representational straightening seems a priori plausible as a mechanism for linguistic prediction. Here, we evaluate the straightening hypothesis across four computational experiments. In Experiment 1, we show that across a corpus of approximately 8.5K human-generated sentences, the average curvature of sentences decreases gradually between the input layer and the deep layers. In Experiment 2, we show that larger models and models that are trained on larger datasets show a greater degree of representation straightening, suggesting that straighter internal representations is what allows for better predictive performance. In Experiment 3, we perform a critical comparison between natural, human-generated sentences and model-generated sentences (created by providing the models with the first few words of the natural sentences) and show that the model-generated sentences have straighter trajectories. Finally, in Experiment 4, we relate sentence curvature to surprisal, a measure of how expected words are in context \cite{Shannon1949-xp}, which has been shown to predict human behavior and neural responses to language \cite{Levy2008-hk,Smith2013-fk,Willems2016-mm,Henderson2016-ww,Lopopolo2017-gk,Shain2020-je, Heilbron2022-xv}. Jointly, these results provide evidence for representation straightening as a mechanistic hypothesis about the computations that allow transformer language models to construct a predictive representation in the service of their behavioral objective.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Language Representation, Computer Science, Linguistics, Cognitive Science, Mathematics, Neural Network Geometry, Predictive Modeling" />
        </attvalues>
      </node>
      <node id="1307.0225" label="1307.0225">
        <attvalues>
          <attvalue for="0" value="Predictive information in a sensory population" />
          <attvalue for="1" value="  Guiding behavior requires the brain to make predictions about future sensory&#10;inputs. Here we show that efficient predictive computation starts at the&#10;earliest stages of the visual system. We estimate how much information groups&#10;of retinal ganglion cells carry about the future state of their visual inputs,&#10;and show that every cell we can observe participates in a group of cells for&#10;which this predictive information is close to the physical limit set by the&#10;statistical structure of the inputs themselves. Groups of cells in the retina&#10;also carry information about the future state of their own activity, and we&#10;show that this information can be compressed further and encoded by downstream&#10;predictor neurons, which then exhibit interesting feature selectivity.&#10;Efficient representation of predictive information is a candidate principle&#10;that can be applied at each stage of neural computation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="0712.4381" label="0712.4381">
        <attvalues>
          <attvalue for="0" value="Efficient representation as a design principle for neural coding and&#10;  computation" />
          <attvalue for="1" value="  Does the brain construct an efficient representation of the sensory world? We&#10;review progress on this question, focusing on a series of experiments in the&#10;last decade which use fly vision as a model system in which theory and&#10;experiment can confront each other. Although the idea of efficient&#10;representation has been productive, clearly it is incomplete since it doesn't&#10;tell us which bits of sensory information are most valuable to the organism. We&#10;suggest that an organism which maximizes the (biologically meaningful) adaptive&#10;value of its actions given fixed resources should have internal representations&#10;of the outside world that are optimal in a very specific information theoretic&#10;sense: they maximize the information about the future of sensory inputs at a&#10;fixed value of the information about their past. This principle contains as&#10;special cases computations which the brain seems to carry out, and it should be&#10;possible to test this optimization directly. We return to the fly visual system&#10;and report the results of preliminary experiments that are in encouraging&#10;agreement with theory.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="physics/0004057" label="physics/0004057">
        <attvalues>
          <attvalue for="0" value="The information bottleneck method" />
          <attvalue for="1" value="  We define the relevant information in a signal $x\in X$ as being the&#10;information that this signal provides about another signal $y\in \Y$. Examples&#10;include the information that face images provide about the names of the people&#10;portrayed, or the information that speech sounds provide about the words&#10;spoken. Understanding the signal $x$ requires more than just predicting $y$, it&#10;also requires specifying which features of $\X$ play a role in the prediction.&#10;We formalize this problem as that of finding a short code for $\X$ that&#10;preserves the maximum information about $\Y$. That is, we squeeze the&#10;information that $\X$ provides about $\Y$ through a `bottleneck' formed by a&#10;limited set of codewords $\tX$. This constrained optimization problem can be&#10;seen as a generalization of rate distortion theory in which the distortion&#10;measure $d(x,\x)$ emerges from the joint statistics of $\X$ and $\Y$. This&#10;approach yields an exact set of self consistent equations for the coding rules&#10;$X \to \tX$ and $\tX \to \Y$. Solutions to these equations can be found by a&#10;convergent re-estimation method that generalizes the Blahut-Arimoto algorithm.&#10;Our variational principle provides a surprisingly rich framework for discussing&#10;a variety of problems in signal processing and learning, as will be described&#10;in detail elsewhere.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2008.02897" label="2008.02897">
        <attvalues>
          <attvalue for="0" value="Iterative Compression of End-to-End ASR Model using AutoML" />
          <attvalue for="1" value="  Increasing demand for on-device Automatic Speech Recognition (ASR) systems&#10;has resulted in renewed interests in developing automatic model compression&#10;techniques. Past research have shown that AutoML-based Low Rank Factorization&#10;(LRF) technique, when applied to an end-to-end Encoder-Attention-Decoder style&#10;ASR model, can achieve a speedup of up to 3.7x, outperforming laborious manual&#10;rank-selection approaches. However, we show that current AutoML-based search&#10;techniques only work up to a certain compression level, beyond which they fail&#10;to produce compressed models with acceptable word error rates (WER). In this&#10;work, we propose an iterative AutoML-based LRF approach that achieves over 5x&#10;compression without degrading the WER, thereby advancing the state-of-the-art&#10;in ASR compression.&#10;" />
          <attvalue for="2" value="&#10;&#10;Rapid technological improvements in deep learning-based acoustic modeling, language modeling, and noise-resilience techniques resulted in a drastic drop in the WER of modern ASR systems. Evidently, ASR is serving as the backbone in audio-based input modality on a variety of devices including mobile phones, smart speakers, and IoT appliances. Due to existing data security and privacy concerns in cloud-based ASR systems, a clear shift in preference towards on-device deployment of the state-of-the-art ASR models is emerging. Mobile and IoT devices, however, suffer from a limited resource budget and require efficient deployment of ASR models with significantly lower memory, compute and power demands.&#10;&#10;Popular techniques in reducing resource demands of well-trained and parameter-heavy models include LRF, pruning, and reduced-precision representations~\cite{pang_compression_2018, hinton_kd_2015}. In our previous work, we have shown that automated Reinforcement Learning (RL)-based search can be applied to identify low ranks in LSTM weight-matrices, which allowed for a $1.23\times$ relative speedup gain over a manual search procedure~\cite{shrinkml2019}. However, we observe that the conventional AutoML search fails to find ranks that manifest higher compression ratio (e.g., $\geq 3.7\times$), without degrading the WER. This is due to inability of RL to differentiate between better and worse choices since most of the visited points result in similar rewards.&#10;Alternatively, iterative approaches have been successfully used to manually compress image recognition models, often delivering better results than their one-shot counterparts~\cite{liu_learning_2017,molchanov_pruning_2017,frankle_lottery_2018,gao_dynamic_2019}. &#10;&#10;In this work, we propose a unified approach by combining iterative compression with AutoML-based rank searching to push the boundaries of ASR model compression. We present an RL-based iterative search that performs incremental compression by following a sequence of increasing speedup targets (trajectory) to reach the desired final speedup, while maintaining the WER.&#10;The basic idea behind our method is illustrated in Figure~\ref{fig:iter_compression}.&#10;The main goals are: (i) obtain smaller models than the one achievable by current AutoML-based one-shot compression, while maintaining low WER; and (ii) understand if there is a fundamental difference between iterative and one-shot LRF-based compression techniques. &#10;&#10;We present an extensive set of experiments by considering a state-of-the-art E2E ASR model and show that the proposed iterative search can achieve at least $5\times$ compression ratio, while producing a $7\%$ relative gain in WER. &#10;Compared to the current state-of-the-art one-shot AutoML compression the results show $1.35\times$ relative gain in speedup without degrading the model's accuracy. &#10;Furthermore, we investigate the impact of different trajectories on achieving a $5\times$ compression ratio with our iterative compression approach. Our results suggest that taking incremental compression steps, while performing iterative search, helps in discovering better compression ranks. &#10;Finally, we show that once the optimal ranks are found, they can be applied to the baseline model in an one-shot manner to obtain a compressed model without any loss in WER.&#10;However, the question, {how to find those ranks without intermediate retraining}, remains an open problem.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning Optimization, Linguistics, Model Compression Techniques, Automatic Speech Recognition, Signal Processing, Mathematics" />
        </attvalues>
      </node>
      <node id="1907.03540" label="1907.03540">
        <attvalues>
          <attvalue for="0" value="ShrinkML: End-to-End ASR Model Compression Using Reinforcement Learning" />
          <attvalue for="1" value="  End-to-end automatic speech recognition (ASR) models are increasingly large&#10;and complex to achieve the best possible accuracy. In this paper, we build an&#10;AutoML system that uses reinforcement learning (RL) to optimize the per-layer&#10;compression ratios when applied to a state-of-the-art attention based&#10;end-to-end ASR model composed of several LSTM layers. We use singular value&#10;decomposition (SVD) low-rank matrix factorization as the compression method.&#10;For our RL-based AutoML system, we focus on practical considerations such as&#10;the choice of the reward/punishment functions, the formation of an effective&#10;search space, and the creation of a representative but small data set for quick&#10;evaluation between search steps. Finally, we present accuracy results on&#10;LibriSpeech of the model compressed by our AutoML system, and we compare it to&#10;manually-compressed models. Our results show that in the absence of retraining&#10;our RL-based search is an effective and practical method to compress a&#10;production-grade ASR system. When retraining is possible, we show that our&#10;AutoML system can select better highly-compressed seed models compared to&#10;manually hand-crafted rank selection, thus allowing for more compression than&#10;previously possible.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1708.06519" label="1708.06519">
        <attvalues>
          <attvalue for="0" value="Learning Efficient Convolutional Networks through Network Slimming" />
          <attvalue for="1" value="  The deployment of deep convolutional neural networks (CNNs) in many real&#10;world applications is largely hindered by their high computational cost. In&#10;this paper, we propose a novel learning scheme for CNNs to simultaneously 1)&#10;reduce the model size; 2) decrease the run-time memory footprint; and 3) lower&#10;the number of computing operations, without compromising accuracy. This is&#10;achieved by enforcing channel-level sparsity in the network in a simple but&#10;effective way. Different from many existing approaches, the proposed method&#10;directly applies to modern CNN architectures, introduces minimum overhead to&#10;the training process, and requires no special software/hardware accelerators&#10;for the resulting models. We call our approach network slimming, which takes&#10;wide and large networks as input models, but during training insignificant&#10;channels are automatically identified and pruned afterwards, yielding thin and&#10;compact models with comparable accuracy. We empirically demonstrate the&#10;effectiveness of our approach with several state-of-the-art CNN models,&#10;including VGGNet, ResNet and DenseNet, on various image classification&#10;datasets. For VGGNet, a multi-pass version of network slimming gives a 20x&#10;reduction in model size and a 5x reduction in computing operations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1611.06440" label="1611.06440">
        <attvalues>
          <attvalue for="0" value="Pruning Convolutional Neural Networks for Resource Efficient Inference" />
          <attvalue for="1" value="  We propose a new formulation for pruning convolutional kernels in neural&#10;networks to enable efficient inference. We interleave greedy criteria-based&#10;pruning with fine-tuning by backpropagation - a computationally efficient&#10;procedure that maintains good generalization in the pruned network. We propose&#10;a new criterion based on Taylor expansion that approximates the change in the&#10;cost function induced by pruning network parameters. We focus on transfer&#10;learning, where large pretrained networks are adapted to specialized tasks. The&#10;proposed criterion demonstrates superior performance compared to other&#10;criteria, e.g. the norm of kernel weights or feature map activation, for&#10;pruning large CNNs after adaptation to fine-grained classification tasks&#10;(Birds-200 and Flowers-102) relaying only on the first order gradient&#10;information. We also show that pruning can lead to more than 10x theoretical&#10;(5x practical) reduction in adapted 3D-convolutional filters with a small drop&#10;in accuracy in a recurrent gesture classifier. Finally, we show results for the&#10;large-scale ImageNet dataset to emphasize the flexibility of our approach.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1810.05331" label="1810.05331">
        <attvalues>
          <attvalue for="0" value="Dynamic Channel Pruning: Feature Boosting and Suppression" />
          <attvalue for="1" value="  Making deep convolutional neural networks more accurate typically comes at&#10;the cost of increased computational and memory resources. In this paper, we&#10;reduce this cost by exploiting the fact that the importance of features&#10;computed by convolutional layers is highly input-dependent, and propose feature&#10;boosting and suppression (FBS), a new method to predictively amplify salient&#10;convolutional channels and skip unimportant ones at run-time. FBS introduces&#10;small auxiliary connections to existing convolutional layers. In contrast to&#10;channel pruning methods which permanently remove channels, it preserves the&#10;full network structures and accelerates convolution by dynamically skipping&#10;unimportant input and output channels. FBS-augmented networks are trained with&#10;conventional stochastic gradient descent, making it readily available for many&#10;state-of-the-art CNNs. We compare FBS to a range of existing channel pruning&#10;and dynamic execution schemes and demonstrate large improvements on ImageNet&#10;classification. Experiments show that FBS can respectively provide $5\times$&#10;and $2\times$ savings in compute on VGG-16 and ResNet-18, both with less than&#10;$0.6\%$ top-5 accuracy loss.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.09050" label="1711.09050">
        <attvalues>
          <attvalue for="0" value="Ethical Challenges in Data-Driven Dialogue Systems" />
          <attvalue for="1" value="  The use of dialogue systems as a medium for human-machine interaction is an&#10;increasingly prevalent paradigm. A growing number of dialogue systems use&#10;conversation strategies that are learned from large datasets. There are well&#10;documented instances where interactions with these system have resulted in&#10;biased or even offensive conversations due to the data-driven training process.&#10;Here, we highlight potential ethical issues that arise in dialogue systems&#10;research, including: implicit biases in data-driven systems, the rise of&#10;adversarial examples, potential sources of privacy violations, safety concerns,&#10;special considerations for reinforcement learning systems, and reproducibility&#10;concerns. We also suggest areas stemming from these issues that deserve further&#10;investigation. Through this initial survey, we hope to spur research leading to&#10;robust, safe, and ethically sound dialogue systems.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2208.08386" label="2208.08386">
        <attvalues>
          <attvalue for="0" value="Neural Embeddings for Text" />
          <attvalue for="1" value="  We propose a new kind of embedding for natural language text that deeply&#10;represents semantic meaning. Standard text embeddings use the outputs from&#10;hidden layers of a pretrained language model. In our method, we let a language&#10;model learn from the text and then literally pick its brain, taking the actual&#10;weights of the model's neurons to generate a vector. We call this&#10;representation of the text a neural embedding. We confirm the ability of this&#10;representation to reflect semantics of the text by an analysis of its behavior&#10;on several datasets, and by a comparison of neural embedding with state of the&#10;art sentence embeddings.&#10;" />
          <attvalue for="2" value="&#10;Capturing the semantic meaning of text as a vector is a fundamental challenge for natural language processing (NLP) and an area of active research \cite{giorgi-etal-2021-declutr, zhang-etal-2020-unsupervised, gao-etal-2021-simcse, huang-etal-2021-disentangling, yan-etal-2021-consert, zhang-etal-2021-pairwise, Muennighoff:2022:SGPT, Liu:2022:Autoencoders, chuang-etal-2022-diffcse}. Recent work has focused on fine-tuning pretrained language models with contrastive learning, either supervised (e.g. \cite{reimers-gurevych-2019-sentence, zhang-etal-2021-pairwise, yan-etal-2021-consert}) or unsupervised (e.g. \cite{giorgi-etal-2021-declutr, gao-etal-2021-simcse}). The embedding is generated by pooling the outputs of certain layers of the model as it processes a text. &#10;&#10;Motivated by the need for deeper semantic representations of text, we propose a different kind of embedding. We allow a language model to fine-tune on a text input, and then measure the resulting changes to the model's own neuronal weights as a neural embedding. &#10;We demonstrate that neural embeddings do indeed represent the semantic differences between samples of text. We evaluate neural embeddings on several datasets and compare them with several state of the art sentence embeddings. We observe that neural embeddings correlate better specifically with semantics, while being comparable in other evaluations. We find that neural embeddings behave differently from the known embeddings we considered. &#10;Our contribution:&#10;\begin{enumerate}[topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex]&#10; \item We propose a new kind of text representation: neural embeddings (Section \ref{sec:Neural}).&#10; \item We evaluate embeddings by using several datasets and several criteria (Section \ref{sec:Evaluation}). We show that by these criteria the neural embeddings are (1) better correlated with semantic similarity and consistency, and (2) strongly differ by the errors they do and by how they represent the qualities of the text.&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Semantic Text Embeddings, Deep Learning Models, Computer Science, Machine Learning, Linguistics, Cognitive Science, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2303.03750" label="2303.03750">
        <attvalues>
          <attvalue for="0" value="Preparing the Vuk'uzenzele and ZA-gov-multilingual South African&#10;  multilingual corpora" />
          <attvalue for="1" value="  This paper introduces two multilingual government themed corpora in various&#10;South African languages. The corpora were collected by gathering the South&#10;African Government newspaper (Vuk'uzenzele), as well as South African&#10;government speeches (ZA-gov-multilingual), that are translated into all 11&#10;South African official languages. The corpora can be used for a myriad of&#10;downstream NLP tasks. The corpora were created to allow researchers to study&#10;the language used in South African government publications, with a focus on&#10;understanding how South African government officials communicate with their&#10;constituents. In this paper we highlight the process of gathering, cleaning and&#10;making available the corpora. We create parallel sentence corpora for Neural&#10;Machine Translation (NMT) tasks using Language-Agnostic Sentence&#10;Representations (LASER) embeddings. With these aligned sentences we then&#10;provide NMT benchmarks for 9 indigenous languages by fine-tuning a massively&#10;multilingual pre-trained language model.&#10;" />
          <attvalue for="2" value="&#10;&#10;The advancement of Natural Language Processing (NLP) research in Africa is impeded due to the scarcity of data for training models for various NLP tasks \cite{nekoto2020participatory} as well as availability of benchmarks and ways to reproduce them \cite{martinus2019focus}. For many South African languages there are still challenges finding easily available textual datasets \cite{marivate2020investigating} even if there are many speakers for those languages \cite{ranathunga-de-silva-2022-languages}. There is a need to focus on development of local language \cite{joshi-etal-2020-state} NLP resources.&#10;&#10;This paper builds upon the work of Autshumato \cite{groenewald2009introducing,groenewald2010processing} by creating automatically aligned parallel corpora from government textual data in the 11 official languages of South Africa. While the Autshumato project focused on creating Machine Translation tools for five indigenous languages, the resulting corpora lacked information about its origin or context, limiting its usefulness for other NLP tasks such as categorisation, topic modelling over time and other tasks that require contextual information of the content. Our approach provides more comprehensive data that can support a wider range of NLP applications.&#10;&#10;Our belief is that there is a significant opportunity to create a more user-friendly data collection process that can be easily maintained and provide extraction tools for others. It is essential to preserve the data source and structure it in a way that enables extensions. Our goal is to enhance Neural Machine Translation (NMT) resources in the government data domain by including all indigenous languages and broadening the translation directions beyond English as the source language. Additionally, we recognise the importance of providing aligned data across all South African languages beyond English.&#10;&#10;Further, this paper introduces parallel corpora datasets in the 11 official languages of South Africa, created from text data obtained from the government. These datasets are designed to facilitate the development of NMT models. The corpora are automatically aligned, and are expected to serve as a valuable resource for researchers and practitioners working in the field of machine learning.&#10;&#10;The parallel corpora were generated using LASER encoders \cite{schwenk2017learning}, facilitating the one-to-one alignment of tokenised sentence data. The data was sourced from credible sources such as newspapers and academic journals and covers diverse topics including health, finance, and politics.&#10;&#10;We also provide NMT benchmarks for the parallel corpora by fine-tuning a massively multilingual model (M2M100 \cite{fan2021beyond}) building on the work of \cite{lafand}.&#10;&#10;This paper is structured as follows. In the following section, we detail the datasets that we have compiled, including their compilation methodology and the information they contain. We then describe how we have aligned and created parallel corpora using these datasets. The subsequent sectionpresents our NMT experiments and provides an analysis of the results obtained. Finally, we conclude the paper with our findings and make recommendations for future research.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Government Communication, Multilingual Corpora, Information Science, Natural Language Processing, Neural Machine Translation" />
        </attvalues>
      </node>
      <node id="2210.08523" label="2210.08523">
        <attvalues>
          <attvalue for="0" value="Some Languages are More Equal than Others: Probing Deeper into the&#10;  Linguistic Disparity in the NLP World" />
          <attvalue for="1" value="  Linguistic disparity in the NLP world is a problem that has been widely&#10;acknowledged recently. However, different facets of this problem, or the&#10;reasons behind this disparity are seldom discussed within the NLP community.&#10;This paper provides a comprehensive analysis of the disparity that exists&#10;within the languages of the world. We show that simply categorising languages&#10;considering data availability may not be always correct. Using an existing&#10;language categorisation based on speaker population and vitality, we analyse&#10;the distribution of language data resources, amount of NLP/CL research,&#10;inclusion in multilingual web-based platforms and the inclusion in pre-trained&#10;multilingual models. We show that many languages do not get covered in these&#10;resources or platforms, and even within the languages belonging to the same&#10;language group, there is wide disparity. We analyse the impact of family,&#10;geographical location, GDP and the speaker population of languages and provide&#10;possible reasons for this disparity, along with some suggestions to overcome&#10;the same.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1704.04154" label="1704.04154">
        <attvalues>
          <attvalue for="0" value="Learning Joint Multilingual Sentence Representations with Neural Machine&#10;  Translation" />
          <attvalue for="1" value="  In this paper, we use the framework of neural machine translation to learn&#10;joint sentence representations across six very different languages. Our aim is&#10;that a representation which is independent of the language, is likely to&#10;capture the underlying semantics. We define a new cross-lingual similarity&#10;measure, compare up to 1.4M sentence representations and study the&#10;characteristics of close sentences. We provide experimental evidence that&#10;sentences that are close in embedding space are indeed semantically highly&#10;related, but often have quite different structure and syntax. These relations&#10;also hold when comparing sentences in different languages.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2004.12835" label="2004.12835">
        <attvalues>
          <attvalue for="0" value="Intuitive Contrasting Map for Antonym Embeddings" />
          <attvalue for="1" value="  This paper shows that, modern word embeddings contain information that&#10;distinguishes synonyms and antonyms despite small cosine similarities between&#10;corresponding vectors. This information is encoded in the geometry of the&#10;embeddings and could be extracted with a straight-forward and intuitive&#10;manifold learning procedure or a contrasting map. Such a map is trained on a&#10;small labeled subset of the data and can produce new embeddings that explicitly&#10;highlight specific semantic attributes of the word. The new embeddings produced&#10;by the map are shown to improve the performance on downstream tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Modern word embeddings, such as \cite{mikolov2013efficient}, \cite{pennington2014glove} or \cite{bojanowski2017enriching} are based on the distributional hypothesis \cite{harris1954distributional}. If two words are often used in a similar context, they should have a small cosine similarity between the embeddings. Naturally, such methods often fail to recognize antonyms since antonymous words, e.g., &quot;fast&quot; and &quot;slow&quot;, occur in similar contexts. Many researchers address this issue from different angles.&#10;&#10;Some authors deal with representations of antonyms, injecting additional information, and improving training procedures. For example, \cite{bian2014knowledge} combine deep learning with various types of semantic knowledge to produce new word embeddings that show better performance on a word similarity task. \cite{ono2015word} combine information from thesauri with distributional information from large-scale unlabelled text data and obtain word embeddings that could distinguish antonyms. \cite{liu2015learning} represent semantic knowledge extracted from thesauri as many ordinal ranking inequalities and formulate the learning of semantic word embeddings as a constrained optimization problem. \cite{kim2016adjusting} develop these ideas further and adjust word vectors using the semantic intensity information alongside with thesauri. \cite{dou2018improving} also use thesauri along with the sentiment to build new embeddings that contrast antonyms. \cite{nguyen2016integrating} improve the weights of feature vectors with a special method based on local mutual information and propose an extension of the skip-gram model that integrates the new vector representations into the objective function. \cite{hill2014not} and \cite{hill2014embedding} show that translation-based embeddings perform better in applications that require concepts to be organized according to similarity and better capture their true ontologic status. \cite{lu2015deep} use these ideas and demonstrate that adding a multilingual context when learning embeddings allows improving their quality via deep canonical correlation analysis. &#10;&#10;Other researchers try to develop novel approaches that are not heavily relying on the distributional hypothesis. For example, \cite{schwartz2015symmetric} introduce word-level vector representation based on symmetric patterns and report that such representations allow controlling the model judgment of antonym pairs. \cite{chen2015revisiting} develop special { contrasting embedding framework}. \cite{nguyen2017distinguishing} train a neural network model that exploits lexico-syntactic patterns from syntactic parse trees to distinguish antonyms. &#10;&#10;All works mentioned above were based on the assumption that antonym-distinguishing information is not captured by modern word embeddings. However, this assumption is frequently questioned in the last several years. \cite{vulic2018specialising} and \cite{vulic2018injecting} show one can inject information on hyponyms, hyperonyms, synonyms, and antonyms to distinguish the obtained embeddings using additional linguistic constraints, see \cite{mrkvsic2017semantic}. Moreover, \cite{etcheverry2019unraveling} come up with a two-phase training of a siamese network that transforms initial embeddings into the ones that clearly distinguish antonyms. \cite{ali2019antonym} develop an architecture of a distiller that extracts information on antonyms out of the pre-trained vectors.&#10;&#10;In this work, we demonstrate that Word2Vec \cite{mikolov2013efficient}, GloVe \cite{pennington2014glove}, and especially FastText \cite{bojanowski2017enriching} embeddings contain information that allows distinguishing antonyms to certain extent. This information is encoded in the geometry of the obtained vector space. We propose a very simple and straightforward approach for the extraction of this information. Similarly to \cite{etcheverry2019unraveling} it is based on a siamese network, yet does not require a two-phase training and is more intuitive than the one proposed in \cite{ali2019antonym}. We also show that this approach could be used further to extract other semantic aspects of words out of the obtained embedding space with ease.&#10;&#10;The contribution of this paper is as follows: &#10;\begin{itemize}&#10;\item we demonstrate that modern word embeddings contain information that allows distinguishing synonyms and antonyms; &#10;\item we show that this information could be retrieved by learning a nonlinear manifold via supervision provided by a small labeled sub-sample of synonyms and antonyms;&#10;\item we demonstrate that concatenation of these new embeddings with original embeddings improves the performance on the downstream tasks that are sensitive to synonym-antonym distinction.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Linguistics, Semantic Attribute Extraction, Computational Linguistics, Manifold Learning Techniques, Mathematics, Natural Language Processing, Word Embeddings Analysis" />
        </attvalues>
      </node>
      <node id="1605.07766" label="1605.07766">
        <attvalues>
          <attvalue for="0" value="Integrating Distributional Lexical Contrast into Word Embeddings for&#10;  Antonym-Synonym Distinction" />
          <attvalue for="1" value="  We propose a novel vector representation that integrates lexical contrast&#10;into distributional vectors and strengthens the most salient features for&#10;determining degrees of word similarity. The improved vectors significantly&#10;outperform standard models and distinguish antonyms from synonyms with an&#10;average precision of 0.66-0.76 across word classes (adjectives, nouns, verbs).&#10;Moreover, we integrate the lexical contrast vectors into the objective function&#10;of a skip-gram model. The novel embedding outperforms state-of-the-art models&#10;on predicting word similarities in SimLex-999, and on distinguishing antonyms&#10;from synonyms.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1410.0718" label="1410.0718">
        <attvalues>
          <attvalue for="0" value="Not All Neural Embeddings are Born Equal" />
          <attvalue for="1" value="  Neural language models learn word representations that capture rich&#10;linguistic and conceptual information. Here we investigate the embeddings&#10;learned by neural machine translation models. We show that translation-based&#10;embeddings outperform those learned by cutting-edge monolingual models at&#10;single-language tasks requiring knowledge of conceptual similarity and/or&#10;syntactic role. The findings suggest that, while monolingual models learn&#10;information about how concepts are related, neural-translation models better&#10;capture their true ontological status.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1412.6448" label="1412.6448">
        <attvalues>
          <attvalue for="0" value="Embedding Word Similarity with Neural Machine Translation" />
          <attvalue for="1" value="  Neural language models learn word representations, or embeddings, that&#10;capture rich linguistic and conceptual information. Here we investigate the&#10;embeddings learned by neural machine translation models, a recently-developed&#10;class of neural language model. We show that embeddings from translation models&#10;outperform those learned by monolingual models at tasks that require knowledge&#10;of both conceptual similarity and lexical-syntactic role. We further show that&#10;these effects hold when translating from both English to French and English to&#10;German, and argue that the desirable properties of translation embeddings&#10;should emerge largely independently of the source and target languages.&#10;Finally, we apply a new method for training neural translation models with very&#10;large vocabularies, and show that this vocabulary expansion algorithm results&#10;in minimal degradation of embedding quality. Our embedding spaces can be&#10;queried in an online demo and downloaded from our web page. Overall, our&#10;analyses indicate that translation-based embeddings should be used in&#10;applications that require concepts to be organised according to similarity&#10;and/or lexical function, while monolingual embeddings are better suited to&#10;modelling (nonspecific) inter-word relatedness.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1701.02962" label="1701.02962">
        <attvalues>
          <attvalue for="0" value="Distinguishing Antonyms and Synonyms in a Pattern-based Neural Network" />
          <attvalue for="1" value="  Distinguishing between antonyms and synonyms is a key task to achieve high&#10;performance in NLP systems. While they are notoriously difficult to distinguish&#10;by distributional co-occurrence models, pattern-based methods have proven&#10;effective to differentiate between the relations. In this paper, we present a&#10;novel neural network model AntSynNET that exploits lexico-syntactic patterns&#10;from syntactic parse trees. In addition to the lexical and syntactic&#10;information, we successfully integrate the distance between the related words&#10;along the syntactic path as a new pattern feature. The results from&#10;classification experiments show that AntSynNET improves the performance over&#10;prior pattern-based methods.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1710.06371" label="1710.06371">
        <attvalues>
          <attvalue for="0" value="Specialising Word Vectors for Lexical Entailment" />
          <attvalue for="1" value="  We present LEAR (Lexical Entailment Attract-Repel), a novel post-processing&#10;method that transforms any input word vector space to emphasise the asymmetric&#10;relation of lexical entailment (LE), also known as the IS-A or&#10;hyponymy-hypernymy relation. By injecting external linguistic constraints&#10;(e.g., WordNet links) into the initial vector space, the LE specialisation&#10;procedure brings true hyponymy-hypernymy pairs closer together in the&#10;transformed Euclidean space. The proposed asymmetric distance measure adjusts&#10;the norms of word vectors to reflect the actual WordNet-style hierarchy of&#10;concepts. Simultaneously, a joint objective enforces semantic similarity using&#10;the symmetric cosine distance, yielding a vector space specialised for both&#10;lexical relations at once. LEAR specialisation achieves state-of-the-art&#10;performance in the tasks of hypernymy directionality, hypernymy detection, and&#10;graded lexical entailment, demonstrating the effectiveness and robustness of&#10;the proposed asymmetric specialisation model.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1906.05612" label="1906.05612">
        <attvalues>
          <attvalue for="0" value="Antonym-Synonym Classification Based on New Sub-space Embeddings" />
          <attvalue for="1" value="  Distinguishing antonyms from synonyms is a key challenge for many NLP&#10;applications focused on the lexical-semantic relation extraction. Existing&#10;solutions relying on large-scale corpora yield low performance because of huge&#10;contextual overlap of antonym and synonym pairs. We propose a novel approach&#10;entirely based on pre-trained embeddings. We hypothesize that the pre-trained&#10;embeddings comprehend a blend of lexical-semantic information and we may&#10;distill the task-specific information using Distiller, a model proposed in this&#10;paper. Later, a classifier is trained based on features constructed from the&#10;distilled sub-spaces along with some word level features to distinguish&#10;antonyms from synonyms. Experimental results show that the proposed model&#10;outperforms existing research on antonym synonym distinction in both speed and&#10;performance.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.13511" label="2204.13511">
        <attvalues>
          <attvalue for="0" value="RobBERTje: a Distilled Dutch BERT Model" />
          <attvalue for="1" value="  Pre-trained large-scale language models such as BERT have gained a lot of&#10;attention thanks to their outstanding performance on a wide range of natural&#10;language tasks. However, due to their large number of parameters, they are&#10;resource-intensive both to deploy and to fine-tune. Researchers have created&#10;several methods for distilling language models into smaller ones to increase&#10;efficiency, with a small performance trade-off. In this paper, we create&#10;several different distilled versions of the state-of-the-art Dutch RobBERT&#10;model and call them RobBERTje. The distillations differ in their distillation&#10;corpus, namely whether or not they are shuffled and whether they are merged&#10;with subsequent sentences. We found that the performance of the models using&#10;the shuffled versus non-shuffled datasets is similar for most tasks and that&#10;randomly merging subsequent sentences in a corpus creates models that train&#10;faster and perform better on tasks with long sequences. Upon comparing&#10;distillation architectures, we found that the larger DistilBERT architecture&#10;worked significantly better than the Bort hyperparametrization. Interestingly,&#10;we also found that the distilled models exhibit less gender-stereotypical bias&#10;than its teacher model. Since smaller architectures decrease the time to&#10;fine-tune, these models allow for more efficient training and more lightweight&#10;deployment of many Dutch downstream language tasks.&#10;" />
          <attvalue for="2" value="&#10;&#10;Large-scale pre-trained language models such as BERT \cite{devlin2019bert} have revolutionized many natural language processing tasks thanks to their outstanding performance on downstream tasks.&#10;Initially, a BERT model is pre-trained on a large corpus of text sequences to predict which words---or more precisely tokens---are likely on masked positions in a sentence. &#10;This task, called Masked Language Modelling (MLM), makes self-supervised learning possible on unlabeled text sequences.&#10;Afterward, it only requires fine-tuning on relatively small labeled datasets to usually get (near) state-of-the-art performance on a given language task, such as sentiment analysis, natural language inference and token tagging tasks.&#10;However, such language models are difficult to deploy in production environments due to the fact that these models are large and thus require a lot of storage, and are slow and energy-intensive to perform inference on~\cite{bender2021parrots}.&#10;Following the trend of distilling the knowledge from neural network models \cite{hinton2015distilling}, many types of distillation have been used to extract optimal parameters or extract the knowledge of larger language models into smaller ones \cite{sanh2019distilbert,dewynter2020bort,jiao2020tinybert}.&#10;These smaller models require fewer resources and time to run, at the cost of being slightly less accurate.&#10;Such a distillation thus allows for a favorable trade-off between performance and ease of use at deployment.&#10;&#10;In this paper, we distill the Dutch BERT model RobBERT v2 \cite{delobelle2020robbert}, and name it RobBERTje.&#10;We perform several distillations using a small unlabeled Dutch dataset and fine-tune them on several language tasks to find the best processing of the dataset and target architecture hyperparametrizations.&#10;The contributions of this paper are thus: (1) evaluating data processing for distillation;&#10;(2) replicating studies on distillation&#10;architectures; and (3) creating a more lightweight version of RobBERT to enable more efficient fine-tuning and energy-efficient inferencing of Dutch downstream language tasks.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Artificial Intelligence Efficiency, Computer Science, Linguistics, Language Model Distillation, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="2406.14092" label="2406.14092">
        <attvalues>
          <attvalue for="0" value="Seamless Language Expansion: Enhancing Multilingual Mastery in&#10;  Self-Supervised Models" />
          <attvalue for="1" value="  Self-supervised (SSL) models have shown great performance in various&#10;downstream tasks. However, they are typically developed for limited languages,&#10;and may encounter new languages in real-world. Developing a SSL model for each&#10;new language is costly. Thus, it is vital to figure out how to efficiently&#10;adapt existed SSL models to a new language without impairing its original&#10;abilities. We propose adaptation methods which integrate LoRA to existed SSL&#10;models to extend new language. We also develop preservation strategies which&#10;include data combination and re-clustering to retain abilities on existed&#10;languages. Applied to mHuBERT, we investigate their effectiveness on speech&#10;re-synthesis task. Experiments show that our adaptation methods enable mHuBERT&#10;to be applied to a new language (Mandarin) with MOS value increased about 1.6&#10;and the relative value of WER reduced up to 61.72%. Also, our preservation&#10;strategies ensure that the performance on both existed and new languages&#10;remains intact.&#10;" />
          <attvalue for="2" value="&#10;&#10;Self-supervised pre-trained models have significantly advanced the field of speech technology, offering robust solutions for learning general representations from vast quantities of unlabeled data\cite{sslreview,wavlm,wav2vec,hubert}. By harnessing the abundant unlabeled data, these models diminish the dependency on expensive and labor-intensive labeled data. Notably, models like Wav2vec 2.0\cite{wav2vec} and HuBERT\cite{hubert} can be easily fine-tuned for a range of downstream tasks, such as automatic speech recognition (ASR), speech synthesis and speech tokenization. &#10;&#10;SSL models are typically developed for a limited set of languages\cite{xlsr}, but they may encounter new languages when applied in real-world scenarios. Besides, developing a new SSL model for each new language is at high cost due to its large parameters. Hence, it is vital to adapt existed SSL models to a new language. Adapting SSL models to a new language is very challenging, as phonetic and morphological structures differ largely across languages. The situation is more severe when incorporated languages and the new language are from disparate linguistic families and have completely different attributes. For example, languages encompassed in pre-training are non-tonal such as English and the target language is a tonal language like Mandarin. The difference between tonal language and non-tonal language is huge because tonal languages distinguish lexical or grammatical meanings through pitch variations, while non-tonal languages do not. To deal with language mismatch problem, strategies such as self-supervised adaptive pre-training and language adaptors have been proposed\cite{ssladaptive,adaptiveadapter}. In \cite{ssladaptive}, unlabeled data collected from the target language of downstream tasks are leveraged to adapt the pre-trained model prior to the last fine-tuning stage. Meta-Adaptable-Adapter is proposed in \cite{adaptiveadapter} to learn task-specific adapters for feature extraction and task-independent adapters for feature combination. However, the relationship between parameter efficiency and adaptation performance remains unclear.&#10;&#10;When adapting an existed model to new data or new tasks, there is a phenomenon called catastrophic forgetting\cite{overcomingcf} which indicates that the model tends to forget previously acquired knowledge. The underlying cause is attributed to the model's weights update during adaptation, which inadvertently overwrite antecedent learned representations. This phenomenon can markedly impede the model's performance and capacity for generalization. To mitigate catastrophic forgetting, researchers in speech area have investigated various techniques\cite{MTL,weight} to strike a balance between adapting to new tasks or data and preserving acquired knowledge. A multi-task learning framework is advocated in \cite{MTL} where reserving original knowledge and learning new knowledge are treated as two independent tasks. Alternative methodologies involve the orthogonal modification of weights\cite{weight,weightconsolidation,rotate}. For instance, \cite{weight} introduced a continual learning algorithm known as Regularized Adaptive Weight Modification (RAWM) for fake audio detection. Nevertheless, the strategies for sustaining the knowledge of existed languages while adapting to new ones remain insufficiently explored.&#10;&#10;In this paper, to deal with language mismatch problem, we propose parameter efficient adaptation methods which incorporate LoRA into existed SSL models instead of developing a new one. Additionally, we develop preservation strategies that include data combination and re-clustering to mitigate catastrophic forgetting problem. They enable SSL models to be applied to new languages parameter efficiently while retaining capabilities on existed languages. Specifically, the SSL model under study is mHuBERT\cite{mhubert}, while the existed language and new language to adapt are English and Mandarin respectively. Speech re-synthesis task is adopted to evaluate the representational ability of adapted SSL model and the efficiency of adaptation. The discrete units of SSL models can naturally serve as the input of speech re-synthesis and open the potential for speech language model (SLM). Speech re-synthesis is a more difficult task than ASR since it requires both representation comprehension and waveform reconstruction. Experiments reveal that the MOS value can improve from 2.26 to 3.80 using our adaptation methods. Also, a 61.72\% relative word error rate (WER) reduction compared to un-adapted model in new language speech re-synthesis is observed. These results verify the effectiveness of our adaptation methods on adapting SSL model to a new language. Moreover, results indicate that our preservation strategies substantially mitigate the performance degradation in original languages while maintaining the newly acquired language capabilities, thereby indicating a significant mitigation of catastrophic forgetting.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Multilingual Deep Learning, Engineering, Signal Processing, Speech Synthesis Tasks, Language Model Adaptation" />
        </attvalues>
      </node>
      <node id="2312.07338" label="2312.07338">
        <attvalues>
          <attvalue for="0" value="Self-supervised Adaptive Pre-training of Multilingual Speech Models for&#10;  Language and Dialect Identification" />
          <attvalue for="1" value="  Pre-trained Transformer-based speech models have shown striking performance&#10;when fine-tuned on various downstream tasks such as automatic speech&#10;recognition and spoken language identification (SLID). However, the problem of&#10;domain mismatch remains a challenge in this area, where the domain of the&#10;pre-training data might differ from that of the downstream labeled data used&#10;for fine-tuning. In multilingual tasks such as SLID, the pre-trained speech&#10;model may not support all the languages in the downstream task. To address this&#10;challenge, we propose self-supervised adaptive pre-training (SAPT) to adapt the&#10;pre-trained model to the target domain and languages of the downstream task. We&#10;apply SAPT to the XLSR-128 model and investigate the effectiveness of this&#10;approach for the SLID task. First, we demonstrate that SAPT improves XLSR&#10;performance on the FLEURS benchmark with substantial gains up to 40.1% for&#10;under-represented languages. Second, we apply SAPT on four different datasets&#10;in a few-shot learning setting, showing that our approach improves the sample&#10;efficiency of XLSR during fine-tuning. Our experiments provide strong empirical&#10;evidence that continual adaptation via self-supervision improves downstream&#10;performance for multilingual speech models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1904.08039" label="1904.08039">
        <attvalues>
          <attvalue for="0" value="A Multi-Task Learning Framework for Overcoming the Catastrophic&#10;  Forgetting in Automatic Speech Recognition" />
          <attvalue for="1" value="  Recently, data-driven based Automatic Speech Recognition (ASR) systems have&#10;achieved state-of-the-art results. And transfer learning is often used when&#10;those existing systems are adapted to the target domain, e.g., fine-tuning,&#10;retraining. However, in the processes, the system parameters may well deviate&#10;too much from the previously learned parameters. Thus, it is difficult for the&#10;system training process to learn knowledge from target domains meanwhile not&#10;forgetting knowledge from the previous learning process, which is called as&#10;catastrophic forgetting (CF). In this paper, we attempt to solve the CF problem&#10;with the lifelong learning and propose a novel multi-task learning (MTL)&#10;training framework for ASR. It considers reserving original knowledge and&#10;learning new knowledge as two independent tasks, respectively. On the one hand,&#10;we constrain the new parameters not to deviate too far from the original&#10;parameters and punish the new system when forgetting original knowledge. On the&#10;other hand, we force the new system to solve new knowledge quickly. Then, a MTL&#10;mechanism is employed to get the balance between the two tasks. We applied our&#10;method to an End2End ASR task and obtained the best performance in both target&#10;and original datasets.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.03300" label="2308.03300">
        <attvalues>
          <attvalue for="0" value="Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio&#10;  Detection" />
          <attvalue for="1" value="  Current fake audio detection algorithms have achieved promising performances&#10;on most datasets. However, their performance may be significantly degraded when&#10;dealing with audio of a different dataset. The orthogonal weight modification&#10;to overcome catastrophic forgetting does not consider the similarity of genuine&#10;audio across different datasets. To overcome this limitation, we propose a&#10;continual learning algorithm for fake audio detection to overcome catastrophic&#10;forgetting, called Regularized Adaptive Weight Modification (RAWM). When&#10;fine-tuning a detection network, our approach adaptively computes the direction&#10;of weight modification according to the ratio of genuine utterances and fake&#10;utterances. The adaptive modification direction ensures the network can&#10;effectively detect fake audio on the new dataset while preserving its knowledge&#10;of old model, thus mitigating catastrophic forgetting. In addition, genuine&#10;audio collected from quite different acoustic conditions may skew their feature&#10;distribution, so we introduce a regularization constraint to force the network&#10;to remember the old distribution in this regard. Our method can easily be&#10;generalized to related fields, like speech emotion recognition. We also&#10;evaluate our approach across multiple datasets and obtain a significant&#10;performance improvement on cross-dataset experiments.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1805.07441" label="1805.07441">
        <attvalues>
          <attvalue for="0" value="Overcoming catastrophic forgetting problem by weight consolidation and&#10;  long-term memory" />
          <attvalue for="1" value="  Sequential learning of multiple tasks in artificial neural networks using&#10;gradient descent leads to catastrophic forgetting, whereby previously learned&#10;knowledge is erased during learning of new, disjoint knowledge. Here, we&#10;propose a new approach to sequential learning which leverages the recent&#10;discovery of adversarial examples. We use adversarial subspaces from previous&#10;tasks to enable learning of new tasks with less interference. We apply our&#10;method to sequentially learning to classify digits 0, 1, 2 (task 1), 4, 5, 6,&#10;(task 2), and 7, 8, 9 (task 3) in MNIST (disjoint MNIST task). We compare and&#10;combine our Adversarial Direction (AD) method with the recently proposed&#10;Elastic Weight Consolidation (EWC) method for sequential learning. We train&#10;each task for 20 epochs, which yields good initial performance (99.24% correct&#10;task 1 performance). After training task 2, and then task 3, both plain&#10;gradient descent (PGD) and EWC largely forget task 1 (task 1 accuracy 32.95%&#10;for PGD and 41.02% for EWC), while our combined approach (AD+EWC) still&#10;achieves 94.53% correct on task 1. We obtain similar results with a much more&#10;difficult disjoint CIFAR10 task, which to our knowledge had not been attempted&#10;before (70.10% initial task 1 performance, 67.73% after learning tasks 2 and 3&#10;for AD+EWC, while PGD and EWC both fall to chance level). Our results suggest&#10;that AD+EWC can provide better sequential learning performance than either PGD&#10;or EWC.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2007.11314" label="2007.11314">
        <attvalues>
          <attvalue for="0" value="Better Early than Late: Fusing Topics with Word Embeddings for Neural&#10;  Question Paraphrase Identification" />
          <attvalue for="1" value="  Question paraphrase identification is a key task in Community Question&#10;Answering (CQA) to determine if an incoming question has been previously asked.&#10;Many current models use word embeddings to identify duplicate questions, but&#10;the use of topic models in feature-engineered systems suggests that they can be&#10;helpful for this task, too. We therefore propose two ways of merging topics&#10;with word embeddings (early vs. late fusion) in a new neural architecture for&#10;question paraphrase identification. Our results show that our system&#10;outperforms neural baselines on multiple CQA datasets, while an ablation study&#10;highlights the importance of topics and especially early topic-embedding fusion&#10;in our architecture.&#10;" />
          <attvalue for="2" value="&#10;&#10;Paraphrase identification is a core NLP task and has been widely studied \cite{socher_dynamic_2011,he_multi-perspective_2015,wieting_towards_2016,tomar_neural_2017}.&#10;One interesting application area of paraphrase detection is Community Question Answering (CQA) \cite{SemEval-2017:task3,bonadiman_effective_2017,rodrigues_semantic_2018}. &#10;The aim of CQA is to answer real open-ended questions based on user-generated content from question answering websites. Being able to identify similar --- already answered --- questions &#10;can be helpful for this purpose.&#10;Question paraphrase detection in CQA is difficult&#10;because texts tend to be longer and have less direct overlap compared to traditional paraphrase detection datasets \cite{rus_paraphrase_2014, peinelt_aiming_2019}.&#10;&#10;Early work on paraphrase detection relied on hand-crafted features, while state-of-the-art approaches for paraphrase identification are primarily neural networks \cite{gong_natural_2018,wang_bilateral_2017,tomar_neural_2017} and hybrid techniques \cite{pang_text_2016,wu_ecnu_2017,feng_beihang-msra_2017}.&#10;Many recently proposed CQA paraphrase detection systems still use hand-crafted features&#10;\cite{agustian_uinsuska-titech_2017,filice_kelp_2017} and some work has successfully integrated topic model features \cite{duan_searching_2008,wu_ecnu_2017}.&#10;This suggests that topic distributions could offer auxiliary information for identifying related questions and complement word embeddings \cite{mikolov_efficient_2013,pennington_glove:_2014}, which provide the main signal in neural systems. &#10;Contrary to hand-crafted static topic features, integrating topics in a neural framework brings the advantage of joint updates during training. &#10;Recent work successfully introduced topics in neural architectures for language generation:&#10;\cite{wang_reinforced_2018} used a topic-enhanced encoder for summarisation, \cite{chen_guided_2016} integrated topics in the decoder for machine translation and \cite{narayan_dont_2018} included topics in both encoder and decoder of their summarisation model.&#10;&#10;However, it remains unclear if topics can be useful in a neural paraphrase detection model and how to best fuse topics with word embeddings for this task. &#10;In this paper, we introduce a novel topic-aware neural architecture and specifically make the following contributions:&#10;\begin{enumerate}&#10;\item We define two settings (early and late fusion) for incorporating topics in our neural paraphrase prediction model (section\ref{sec:model}).&#10;\item Our topic-aware model improves over other neural models across multiple question paraphrase identification datasets (section\ref{sec:results}).&#10;\item In an ablation study, we highlight the importance of topics and early topic-embedding fusion in our proposed architecture (section\ref{sec:results}).&#10;\end{enumerate}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Information Retrieval, Neural Network Architecture, Question Paraphrase Identification, Computer Science, Linguistics, Information Science, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1704.04565" label="1704.04565">
        <attvalues>
          <attvalue for="0" value="Neural Paraphrase Identification of Questions with Noisy Pretraining" />
          <attvalue for="1" value="  We present a solution to the problem of paraphrase identification of&#10;questions. We focus on a recent dataset of question pairs annotated with binary&#10;paraphrase labels and show that a variant of the decomposable attention model&#10;(Parikh et al., 2016) results in accurate performance on this task, while being&#10;far simpler than many competing neural architectures. Furthermore, when the&#10;model is pretrained on a noisy dataset of automatically collected question&#10;paraphrases, it obtains the best reported performance on the dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1602.06359" label="1602.06359">
        <attvalues>
          <attvalue for="0" value="Text Matching as Image Recognition" />
          <attvalue for="1" value="  Matching two texts is a fundamental problem in many natural language&#10;processing tasks. An effective way is to extract meaningful matching patterns&#10;from words, phrases, and sentences to produce the matching score. Inspired by&#10;the success of convolutional neural network in image recognition, where neurons&#10;can capture many complicated patterns based on the extracted elementary visual&#10;patterns such as oriented edges and corners, we propose to model text matching&#10;as the problem of image recognition. Firstly, a matching matrix whose entries&#10;represent the similarities between words is constructed and viewed as an image.&#10;Then a convolutional neural network is utilized to capture rich matching&#10;patterns in a layer-by-layer way. We show that by resembling the compositional&#10;hierarchies of patterns in image recognition, our model can successfully&#10;identify salient signals such as n-gram and n-term matchings. Experimental&#10;results demonstrate its superiority against the baselines.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1607.01628" label="1607.01628">
        <attvalues>
          <attvalue for="0" value="Guided Alignment Training for Topic-Aware Neural Machine Translation" />
          <attvalue for="1" value="  In this paper, we propose an effective way for biasing the attention&#10;mechanism of a sequence-to-sequence neural machine translation (NMT) model&#10;towards the well-studied statistical word alignment models. We show that our&#10;novel guided alignment training approach improves translation quality on&#10;real-life e-commerce texts consisting of product titles and descriptions,&#10;overcoming the problems posed by many unknown words and a large type/token&#10;ratio. We also show that meta-data associated with input texts such as topic or&#10;category information can significantly improve translation quality when used as&#10;an additional signal to the decoder part of the network. With both novel&#10;features, the BLEU score of the NMT system on a product title set improves from&#10;18.6 to 21.3%. Even larger MT quality gains are obtained through domain&#10;adaptation of a general domain NMT system to e-commerce data. The developed NMT&#10;system also performs well on the IWSLT speech translation task, where an&#10;ensemble of four variant systems outperforms the phrase-based baseline by 2.1%&#10;BLEU absolute.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2110.05571" label="2110.05571">
        <attvalues>
          <attvalue for="0" value="SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition" />
          <attvalue for="1" value="  The Transformer architecture has been well adopted as a dominant architecture&#10;in most sequence transduction tasks including automatic speech recognition&#10;(ASR), since its attention mechanism excels in capturing long-range&#10;dependencies. While models built solely upon attention can be better&#10;parallelized than regular RNN, a novel network architecture, SRU++, was&#10;recently proposed. By combining the fast recurrence and attention mechanism,&#10;SRU++ exhibits strong capability in sequence modeling and achieves&#10;near-state-of-the-art results in various language modeling and machine&#10;translation tasks with improved compute efficiency. In this work, we present&#10;the advantages of applying SRU++ in ASR tasks by comparing with Conformer&#10;across multiple ASR benchmarks and study how the benefits can be generalized to&#10;long-form speech inputs. On the popular LibriSpeech benchmark, our SRU++ model&#10;achieves 2.0% / 4.7% WER on test-clean / test-other, showing competitive&#10;performances compared with the state-of-the-art Conformer encoder under the&#10;same set-up. Specifically, SRU++ can surpass Conformer on long-form speech&#10;input with a large margin, based on our analysis.&#10;" />
          <attvalue for="2" value="&#10;&#10;Since deep neural networks (DNNs) were applied in automatic speech recognition (ASR) tasks, various DNN architectures which helped boosting the ASR accuracy have been proposed. Recurrent neural networks (RNNs) have been widely used given that they can effectively capture the temporal dependencies of a speech frame sequence \cite{graves2013speech,chiu2018state}. Convolutional neural networks (CNNs), originally popular in the computer vision domain, have also become preferred options for ASR tasks. By carefully tuning the receptive fields, residual connections and temporal resolution of the convolutional layers, deep CNNs can yield very competitive results \cite{han2020contextnet,han2021multistream}. &#10;&#10;However, it is well known that both RNN and CNN struggle to model long sequences.&#10;Typical RNN variants such as LSTM and GRU utilize gating operations to alleviate gradient explosion/vanishing problems, but this also leads to catastrophic forgetting, where information carried by hidden states are overwritten.&#10;On the other hand, CNN uses local connectivity and therefore requires many more layers to gain enough perceptive field for global context, which also creates difficulties for optimization.&#10;&#10;Recently, the Transformer architecture with self-attention \cite{vaswani2017attention} is trending in ASR tasks due to its dominant performance in modeling the long-range dependencies&#10;\cite{wang2020transformer,moritz2020streaming}.&#10;The architecture strikes a great balance between model capacity and training efficiency; (a) attention permits learning dependencies between any pair of time steps in a given sequence, (b) the associated computation of the attention mechanism can be formatted in a couple of matrix multiplications and are therefore highly parallelizable, and (c) scaling model capacity can be achieved by increasing the attention dimension and the number of attention heads, aside from layer stacking. &#10;&#10;Despite the great success of Transformer, several research have found that the powerful attention mechanism can still be complemented by traditional neural components that are well-suited for capturing fine-grain local context \cite{merity2019single,lei2021attention}.&#10;For example, Conformer \cite{gulati2020conformer} proposes adding a convolution module leveraging gating mechanisms in point-wise convolution and gated linear unit (GLU) on top of multi-head self-attention to augment Transformer, yielding faster convergence and state-of-the-art performance on various speech recognition datasets. &#10;If convolutions were effective at enhancing the performance of Transformer models by complementing the attention module, could we achieve similar or better synergy by taking the advantage of the recent development of recurrent networks for attention?&#10;&#10;In this work, we present the application of a novel network architecture called SRU++ introduced by \cite{lei2021attention} in ASR task, which is, to our best knowledge, the first attempt to apply such an attention-recurrence fusion network in speech recognition. The basic idea of SRU++ is to leverage a highly parallelizable recurrent neural network structure called SRU \cite{lei2017simple} and perform attention before the recurrence. We conduct ASR experiments on three popular public datasets and compare the results with the Transformer and Conformer encoders implemented and open-sourced in \cite{guo2021recent}. Results show that our SRU++ encoder model can perform on par with the state-of-the-art Conformer model under the same set-up. &#10;In addition, our analysis shows that SRU++ generalizes exceptionally well on long-form speech, outperforming the Conformer model by a large margin.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Neural Network Architecture, Computer Science, Linguistics, Sequence Modeling, Signal Processing, Mathematics, Speech Recognition" />
        </attvalues>
      </node>
      <node id="1709.02755" label="1709.02755">
        <attvalues>
          <attvalue for="0" value="Simple Recurrent Units for Highly Parallelizable Recurrence" />
          <attvalue for="1" value="  Common recurrent neural architectures scale poorly due to the intrinsic&#10;difficulty in parallelizing their state computations. In this work, we propose&#10;the Simple Recurrent Unit (SRU), a light recurrent unit that balances model&#10;capacity and scalability. SRU is designed to provide expressive recurrence,&#10;enable highly parallelized implementation, and comes with careful&#10;initialization to facilitate training of deep models. We demonstrate the&#10;effectiveness of SRU on multiple NLP tasks. SRU achieves 5--9x speed-up over&#10;cuDNN-optimized LSTM on classification and question answering datasets, and&#10;delivers stronger results than LSTM and convolutional models. We also obtain an&#10;average of 0.7 BLEU improvement over the Transformer model on translation by&#10;incorporating SRU into the architecture.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2409.02302" label="2409.02302">
        <attvalues>
          <attvalue for="0" value="Speech Foundation Model Ensembles for the Controlled Singing Voice&#10;  Deepfake Detection (CtrSVDD) Challenge 2024" />
          <attvalue for="1" value="  This work details our approach to achieving a leading system with a 1.79%&#10;pooled equal error rate (EER) on the evaluation set of the Controlled Singing&#10;Voice Deepfake Detection (CtrSVDD). The rapid advancement of generative AI&#10;models presents significant challenges for detecting AI-generated deepfake&#10;singing voices, attracting increased research attention. The Singing Voice&#10;Deepfake Detection (SVDD) Challenge 2024 aims to address this complex task. In&#10;this work, we explore the ensemble methods, utilizing speech foundation models&#10;to develop robust singing voice anti-spoofing systems. We also introduce a&#10;novel Squeeze-and-Excitation Aggregation (SEA) method, which efficiently and&#10;effectively integrates representation features from the speech foundation&#10;models, surpassing the performance of our other individual systems. Evaluation&#10;results confirm the efficacy of our approach in detecting deepfake singing&#10;voices. The codes can be accessed at https://github.com/Anmol2059/SVDD2024.&#10;" />
          <attvalue for="2" value="&#10;With the rapid development of generative AI technology, the quality of audio synthesis has significantly improved, making it increasingly difficult to distinguish between bona fide and spoofed audio. However, this progress also poses significant risks to human voice biometrics and can deceive both automatic speaker verification systems and their users~\cite{7858696}. Additionally, the proliferation of spoofed speech presents a serious threat to cybersecurity, as it can be used to manipulate information, conduct fraud, and bypass security measures that rely on voice authentication. Finding effective ways to detect spoofing attacks and protect users from the threat of spoofed speech is becoming increasingly important. Therefore, speech anti-spoofing, also known as speech deepfake detection, has emerged~\cite{ASVspoof2019, delgado2024asvspoof, one_class, AASIST, wu2024codecfake}. It is dedicated to developing reliable automatic spoofing countermeasures (CMs), which is of utmost importance to society and the ethical applications of generative models.&#10;&#10;Unlike speech spoofing, creating deepfakes of singing voices introduces distinct challenges. This complexity arises from the inherently musical aspects of singing, such as varying pitch, tempo, and emotion, as well as the frequent presence of loud and intricate background music~\cite{SingFake, CtrSVDD_interspeech}. These factors make it more difficult to detect deepfakes in singing compared to regular speech, which typically features a more consistent and predictable sound pattern.&#10;Recently, the speech anti-spoofing research community has been increasingly focusing on this challenging issue, resulting in the development of related datasets~\cite{SingFake, CtrSVDD_interspeech, 10446271}, challenges~\cite{SVDD_challenge}, and models~\cite{chen2024singing}.&#10;The Singing Voice Deepfake Detection (SVDD) Challenge 2024 aims to address these challenges by fostering the development of robust detection systems~\cite{SVDD_challenge, zhang2024svdd2024inauguralsinging}. &#10;&#10;Speech foundation models are large, pre-trained models designed to serve as the backbone for various speech-related tasks, including speaker verification, speech recognition, and more~\cite{lin2024sawavlms, 10446072, jiang2024target}. Many of these models rely on self-supervised learning (SSL) to develop robust speech representations, such as WavLM~\cite{wavlm} and wav2vec2~\cite{wav2vec2}. These models excel in learning high-quality representations that can be fine-tuned for specific downstream tasks. Recently, many studies on speech anti-spoofing have adopted this approach and achieved state-of-the-art performance~\cite{10003971, liu2024neural, 10446331, 9747768, 10448049, 10448016}. The progress of these studies and their promising performance motivate us to continue exploring along this particular line.&#10;&#10;This work details our participation in the CtrSVDD track of the SVDD Challenge 2024. &#10;We detect singing voice deepfakes by ensembling models developed using speech foundation models, data augmentation techniques, and various layer aggregation methods. Specifically, {the default Weighted Sum aggregation method fixes weights after training, limiting adaptability to new data. The recently proposed Attentive Merging (AttM) method~\cite{attentive_merge}, while powerful, can lead to overfitting on small datasets. To address these issues,} inspired by Squeeze-and-Excitation Networks (SENet)~\cite{SE}, we propose the SE Aggregation (SEA) method. {This method dynamically assigns weights and mitigates overfitting issues, enabling our best individual model to achieve an EER of 2.70\% on the CtrSVDD evaluation set. Further investigations show that ensembling systems enhances robustness and performance, achieving our best result of 1.79\% EER.}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speech Processing, Machine Learning, Linguistics, Deepfake Detection, Ensemble Methods, Engineering, Speech Foundation Models, Artificial Intelligence" />
        </attvalues>
      </node>
      <node id="2309.07525" label="2309.07525">
        <attvalues>
          <attvalue for="0" value="SingFake: Singing Voice Deepfake Detection" />
          <attvalue for="1" value="  The rise of singing voice synthesis presents critical challenges to artists&#10;and industry stakeholders over unauthorized voice usage. Unlike synthesized&#10;speech, synthesized singing voices are typically released in songs containing&#10;strong background music that may hide synthesis artifacts. Additionally,&#10;singing voices present different acoustic and linguistic characteristics from&#10;speech utterances. These unique properties make singing voice deepfake&#10;detection a relevant but significantly different problem from synthetic speech&#10;detection. In this work, we propose the singing voice deepfake detection task.&#10;We first present SingFake, the first curated in-the-wild dataset consisting of&#10;28.93 hours of bonafide and 29.40 hours of deepfake song clips in five&#10;languages from 40 singers. We provide a train/validation/test split where the&#10;test sets include various scenarios. We then use SingFake to evaluate four&#10;state-of-the-art speech countermeasure systems trained on speech utterances. We&#10;find these systems lag significantly behind their performance on speech test&#10;data. When trained on SingFake, either using separated vocal tracks or song&#10;mixtures, these systems show substantial improvement. However, our evaluations&#10;also identify challenges associated with unseen singers, communication codecs,&#10;languages, and musical contexts, calling for dedicated research into singing&#10;voice deepfake detection. The SingFake dataset and related resources are&#10;available at https://www.singfake.org/.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2406.03111" label="2406.03111">
        <attvalues>
          <attvalue for="0" value="Singing Voice Graph Modeling for SingFake Detection" />
          <attvalue for="1" value="  Detecting singing voice deepfakes, or SingFake, involves determining the&#10;authenticity and copyright of a singing voice. Existing models for speech&#10;deepfake detection have struggled to adapt to unseen attacks in this unique&#10;singing voice domain of human vocalization. To bridge the gap, we present a&#10;groundbreaking SingGraph model. The model synergizes the capabilities of the&#10;MERT acoustic music understanding model for pitch and rhythm analysis with the&#10;wav2vec2.0 model for linguistic analysis of lyrics. Additionally, we advocate&#10;for using RawBoost and beat matching techniques grounded in music domain&#10;knowledge for singing voice augmentation, thereby enhancing SingFake detection&#10;performance. Our proposed method achieves new state-of-the-art (SOTA) results&#10;within the SingFake dataset, surpassing the previous SOTA model across three&#10;distinct scenarios: it improves EER relatively for seen singers by 13.2%, for&#10;unseen singers by 24.3%, and unseen singers using different codecs by 37.1%.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2407.02826" label="2407.02826">
        <attvalues>
          <attvalue for="0" value="SA-WavLM: Speaker-Aware Self-Supervised Pre-training for Mixture Speech" />
          <attvalue for="1" value="  It was shown that pre-trained models with self-supervised learning (SSL)&#10;techniques are effective in various downstream speech tasks. However, most such&#10;models are trained on single-speaker speech data, limiting their effectiveness&#10;in mixture speech. This motivates us to explore pre-training on mixture speech.&#10;This work presents SA-WavLM, a novel pre-trained model for mixture speech.&#10;Specifically, SA-WavLM follows an &quot;extract-merge-predict&quot; pipeline in which the&#10;representations of each speaker in the input mixture are first extracted&#10;individually and then merged before the final prediction. In this pipeline,&#10;SA-WavLM performs speaker-informed extractions with the consideration of the&#10;interactions between different speakers. Furthermore, a speaker shuffling&#10;strategy is proposed to enhance the robustness towards the speaker absence.&#10;Experiments show that SA-WavLM either matches or improves upon the&#10;state-of-the-art pre-trained models.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2204.05177" label="2204.05177">
        <attvalues>
          <attvalue for="0" value="The PartialSpoof Database and Countermeasures for the Detection of Short&#10;  Fake Speech Segments Embedded in an Utterance" />
          <attvalue for="1" value="  Automatic speaker verification is susceptible to various manipulations and&#10;spoofing, such as text-to-speech synthesis, voice conversion, replay,&#10;tampering, adversarial attacks, and so on. We consider a new spoofing scenario&#10;called &quot;Partial Spoof&quot; (PS) in which synthesized or transformed speech segments&#10;are embedded into a bona fide utterance. While existing countermeasures (CMs)&#10;can detect fully spoofed utterances, there is a need for their adaptation or&#10;extension to the PS scenario. We propose various improvements to construct a&#10;significantly more accurate CM that can detect and locate short-generated&#10;spoofed speech segments at finer temporal resolutions. First, we introduce&#10;newly developed self-supervised pre-trained models as enhanced feature&#10;extractors. Second, we extend our PartialSpoof database by adding segment&#10;labels for various temporal resolutions. Since the short spoofed speech&#10;segments to be embedded by attackers are of variable length, six different&#10;temporal resolutions are considered, ranging from as short as 20 ms to as large&#10;as 640 ms. Third, we propose a new CM that enables the simultaneous use of the&#10;segment-level labels at different temporal resolutions as well as&#10;utterance-level labels to execute utterance- and segment-level detection at the&#10;same time. We also show that the proposed CM is capable of detecting spoofing&#10;at the utterance level with low error rates in the PS scenario as well as in a&#10;related logical access (LA) scenario. The equal error rates of utterance-level&#10;detection on the PartialSpoof database and ASVspoof 2019 LA database were 0.77&#10;and 0.90%, respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2006.14223" label="2006.14223">
        <attvalues>
          <attvalue for="0" value="Neural Machine Translation For Paraphrase Generation" />
          <attvalue for="1" value="  Training a spoken language understanding system, as the one in Alexa,&#10;typically requires a large human-annotated corpus of data. Manual annotations&#10;are expensive and time consuming. In Alexa Skill Kit (ASK) user experience with&#10;the skill greatly depends on the amount of data provided by skill developer. In&#10;this work, we present an automatic natural language generation system, capable&#10;of generating both human-like interactions and annotations by the means of&#10;paraphrasing. Our approach consists of machine translation (MT) inspired&#10;encoder-decoder deep recurrent neural network. We evaluate our model on the&#10;impact it has on ASK skill, intent, named entity classification accuracy and&#10;sentence level coverage, all of which demonstrate significant improvements for&#10;unseen skills on natural language understanding (NLU) models, trained on the&#10;data augmented with paraphrases.&#10;" />
          <attvalue for="2" value="&#10;ASK is an increasingly important part of Alexa user experience.(\cite{Kumar2017JustAB}, \cite{Kumar2017JustAB}) In ASK work flow, the skill developer provides a set of slots (often catalogs of entities), and a list of intents, which can be mapped to actions, and a set of example phrases defining the grammar of an intent. (see figure \ref{fig:dev-grammar}).\\&#10;&#10;From these examples NLU and language modeling (LM) models are built for the skill. Note that it is up to the developer to anticipate all ways their users will interact with the skill. Interactions not covered by the provided examples often have much lower ASR recognition and NLU classification accuracy. Coming up with an exhaustive list of examples can be a hard task for the developer and incomplete coverage can be a frustrating experience to the user. In this work, we propose to use paraphrasing to expand the coverage of developer-provided examples, and thus reduce burden on skill developers and make skill interactions more natural to Alexa customers. Instead of relying on the developer to come up with an exhaustive list of examples for a given intent, in the proposed work flow, we will only require a few examples and then use paraphrasing model to generate other ways a customer might phrase the same command, and then use that data to build better NLU and LM models. Figure \ref{fig:paraphrase} gives an example of the desired paraphrases for a customer utterance.&#10;&#10;Paraphrasing is used in various Natural Language Processing applications, such as natural language generation, summarization, information extraction, sentence compression and question answering. Traditional paraphrase generation methods exploit hand-crafted rules (\cite{mckeown1983paraphrasing}, \cite{mckeown1983paraphrasing}) or automatically learned complex paraphrase patterns (\cite{zhao2009application}, \cite{zhao2009application}), use thesaurus-based (\cite{hassan2007unt}, \cite{hassan2007unt}) or semantic analysis driven natural language generation approaches (\cite{kozlowski2003generation}, \cite{kozlowski2003generation}), or leverage statistical machine translation (\cite{quirk2004monolingual}, \cite{quirk2004monolingual}); (\cite{wubben2010paraphrase}, \cite{wubben2010paraphrase}).&#10;&#10;In this paper, we propose to use neural machine translation (NMT) as a simple and flexible approach to MT to address the paraphrase generation problem. We observe that in translation, there is not a single correct translation target, but rather several variants of the sentence, carrying the same meaning, or paraphrases. From this perspective, translation can be seen as paraphrasing the source sentence in a different language. Therefore, NMT is quite natural approach to paraphrasing. It has been shown to have comparable performance to the phrase-based translation systems (\cite{sutskever2014sequence}, \cite{sutskever2014sequence}), and it is very flexible and modular, allowing to reuse pre-trained components, such as word embeddings or other networks trained on different datasets.&#10;&#10;The remainder of the paper is organized as follows: Section \ref{nmt} presents a brief overview of the sequence to sequence models and techniques used in this work, Section \ref{data} describes the available data, Section \ref{model-desc} explains the experimental setup, Section \ref{results} presents the evaluation results, Section \ref{discussion} analyzes the results and discusses future work, Section \ref{conclusion} is conclusion. &#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Deep Learning Models, Computer Science, Linguistics, Spoken Language Understanding, Artificial Intelligence, Natural Language Generation" />
        </attvalues>
      </node>
      <node id="1711.00549" label="1711.00549">
        <attvalues>
          <attvalue for="0" value="Just ASK: Building an Architecture for Extensible Self-Service Spoken&#10;  Language Understanding" />
          <attvalue for="1" value="  This paper presents the design of the machine learning architecture that&#10;underlies the Alexa Skills Kit (ASK) a large scale Spoken Language&#10;Understanding (SLU) Software Development Kit (SDK) that enables developers to&#10;extend the capabilities of Amazon's virtual assistant, Alexa. At Amazon, the&#10;infrastructure powers over 25,000 skills deployed through the ASK, as well as&#10;AWS's Amazon Lex SLU Service. The ASK emphasizes flexibility, predictability&#10;and a rapid iteration cycle for third party developers. It imposes inductive&#10;biases that allow it to learn robust SLU models from extremely small and sparse&#10;datasets and, in doing so, removes significant barriers to entry for software&#10;developers and dialogue systems researchers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2308.12488" label="2308.12488">
        <attvalues>
          <attvalue for="0" value="GPTEval: A Survey on Assessments of ChatGPT and GPT-4" />
          <attvalue for="1" value="  The emergence of ChatGPT has generated much speculation in the press about&#10;its potential to disrupt social and economic systems. Its astonishing language&#10;ability has aroused strong curiosity among scholars about its performance in&#10;different domains. There have been many studies evaluating the ability of&#10;ChatGPT and GPT-4 in different tasks and disciplines. However, a comprehensive&#10;review summarizing the collective assessment findings is lacking. The objective&#10;of this survey is to thoroughly analyze prior assessments of ChatGPT and GPT-4,&#10;focusing on its language and reasoning abilities, scientific knowledge, and&#10;ethical considerations. Furthermore, an examination of the existing evaluation&#10;methods is conducted, offering several recommendations for future research in&#10;evaluating large language models.&#10;" />
          <attvalue for="2" value="&#10;&#10;ChatGPT~\cite{openai2023chatgpt} has generated significant scholarly interest across various disciplines due to its impressive dialogue-based task-processing capabilities. This has enabled users to explore and evaluate its performance across a wide range of tasks and disciplines, thereby sparking considerable enthusiasm in the field of Artificial Intelligence (AI). While many researchers have concentrated on evaluating ChatGPT and GPT-4~\cite{openai2023gpt4} within their specific domains of expertise, a comprehensive review encompassing the assessments in multiple tasks and disciplines can offer a holistic understanding of the strengths and limitations of these GPT models. We focus on ChatGPT and GPT-4, because they are state-of-the-art (SOTA) large language models (LLMs). The scope of our survey encompasses quantitative evaluations carried out on ChatGPT or GPT-4, specifically focusing on their language proficiency, scientific knowledge, and ethical considerations. Our main findings are summarized as follows:\\&#10; a) ChatGPT and GPT-4 are strong in language understanding and generation, adeptly engaging in user interactions through dialogues, enabling them to tackle diverse NLP tasks and provide explanatory outputs. However, their current status falls short of being a comprehensive AI, as their performance lags behind expert models in numerous domains involving domain-specific knowledge.&#10;&#10;b) ChatGPT performs satisfactorily in general science knowledge and can answer science questions that desire open responses. However, it can also make mistakes, especially for questions that require multi-step reasoning.&#10;The exceptional language proficiency poses challenges for users in assessing the accuracy of factual information, giving rise to a range of ethical concerns.&#10;&#10;c) Existing evaluation methods may be unreliable. The current evaluation methods heavily depend on prompt engineering and benchmark datasets. Varying prompts can yield disparate evaluation results. Additionally, the comparison of expert systems often relies on (in-domain) datasets that were utilized for training those systems. It remains uncertain if the examined data, such as public datasets and scientific knowledge, have been inadvertently exposed during the training of ChatGPT and GPT-4. These factors may contribute to an unfair comparison between LLMs and their respective baselines.&#10;&#10;The contributions of this work are threefold: (1)~We conduct a comprehensive survey of recent assessments focusing on the language proficiency and scientific knowledge of ChatGPT and GPT-4. (2) We compare their assessment results across various tasks and disciplines to highlight the strengths and weaknesses of the GPT models. (3) We critically analyze the existing assessment methods employed, offering recommendations for future evaluation studies and delivering our ethical considerations associated with the GPT models.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Large Language Models, Linguistics, Cognitive Science, Language Model Evaluation, Artificial Intelligence, Artificial Intelligence Ethics, Philosophy" />
        </attvalues>
      </node>
      <node id="2402.14760" label="2402.14760">
        <attvalues>
          <attvalue for="0" value="Generalizing Reward Modeling for Out-of-Distribution Preference Learning" />
          <attvalue for="1" value="  Preference learning (PL) with large language models (LLMs) aims to align the&#10;LLMs' generations with human preferences. Previous work on reinforcement&#10;learning from human feedback (RLHF) has demonstrated promising results in&#10;in-distribution PL. However, due to the difficulty of obtaining human feedback,&#10;discretely training reward models for every encountered distribution is&#10;challenging. Thus, out-of-distribution (OOD) PL is practically useful for&#10;enhancing the generalization ability of LLMs with limited preference feedback.&#10;This work addresses OOD PL by optimizing a general reward model through a&#10;meta-learning approach. During meta-training, a bilevel optimization algorithm&#10;is utilized to learn a reward model capable of guiding policy learning to align&#10;with human preferences across various distributions. When encountering a test&#10;distribution, the meta-test procedure conducts regularized policy optimization&#10;using the learned reward model for PL. We theoretically demonstrate the&#10;convergence rate of the bilevel optimization algorithm under reasonable&#10;assumptions. Additionally, we conduct experiments on two text generation tasks&#10;across 20 held-out domains and outperform a variety of strong baselines across&#10;various evaluation metrics.&#10;" />
          <attvalue for="2" value="&#10;Aligning large language models (LLMs) with human preferences through reinforcement learning has been demonstrated as a practical approach to align pretrained LLMs along human values. As shown by recent research on LLMs \cite{christiano2017deep,ziegler2019fine,stiennon2020learning,ouyang2022training}, RLHF initially trains a reward model (RM) to capture human preferences from a pairwise preference dataset. It then aligns the LLMs with the learned RM through regularized policy optimization, aiming to learn a language policy that better reflects human values. Most previous work on RLHF focuses on in-distribution (ID) preference learning (PL) \cite{casper2023open}, i.e., using the ID preference data for reward learning \cite{christiano2017deep} and then performing policy optimization using PPO (Fig. \ref{fig:intro} (b)). Besides, direct preference optimization (DPO) \cite{rafailov2024direct} and its variants \cite{azar2024general,ethayarajh2024kto,park2024disentangling} directly optimize a language policy using maximum likelihood estimation on the ID preference data (Fig. \ref{fig:intro} (c)). &#10;&#10;However, defining precise rewards for various real-world tasks is non-trivial \cite{mckinney2022fragility}, and obtaining high-quality feedback that accurately represents human preferences is challenging \cite{bai2022training}.&#10;Therefore, we focus on enhancing the out-of-distribution (OOD) generalization ability for PL. Fig. \ref{fig:intro} (a) illustrates an OOD scenario where only preference data from training distributions (\fcolorbox{black}{pink}{$y$},\fcolorbox{black}{pink}{$y'$},..., \fcolorbox{black}{green}{$y$},\fcolorbox{black}{green}{$y'$}) are available, as well as fine-tuning data from training-/test-distributions ($\mathcal{D}^{{tr}_1}_{xy}$,...,$\mathcal{D}^{{tr}_N}_{xy}$, $\mathcal{D}^{{te}}_{xy}$). Two major challenges arise when these ID PL methods encounter the distribution shift: (i) the generations from the policy may substantially deviate from the human preferences of the target distribution, posing an OOD challenge for PL. (ii) the distribution shifts result in policy drift during the policy optimization process with Kullback-Leibler divergence \cite{ramamurthy2022reinforcement,rafailov2024direct}: the policy tends to move towards the reference policy largely followed in the training distribution, exacerbating the model's deviation from the test distribution in PL. &#10;&#10;An initial approach to addressing the OOD challenge is to directly enhance the OOD generalization ability of the RM using preference data from multiple distributions. This specific approach may involve training an RM using multiple distributions or integrating distribution-specific RMs into a unified model, similar to previous work on RM ensembling \cite{rame2024warm}. However, such a method also has limitations from the distribution shift between the preference data and fine-tuning data. To better tackle the OOD challenge, we employ a meta-learning approach to learn a reward function capable of guiding policy optimization for OOD preference learning (Fig. \ref{fig:intro} (a)). For each episode, the policy is optimized with RL on fine-tuning data (meta-training on support set) and the RM is optimized with PL objectives on the preference data (meta-test on query set). In particular, we incorporate a regularization term in policy optimization to penalize the Kullback-Leibler divergence between the policy-induced distribution and test distribution, thereby mitigating the challenge of policy drift. We summarize the contributions as follows.&#10;\begin{itemize}[leftmargin=1em]&#10;&#09;\item {Improving OOD generalization for preference learning.} We focus on aligning large language models with human preference on the OOD data, and propose a novel meta-learning-based approach to tackle the OOD preference learning problem.&#10;&#09;\item {Gradient-based bilevel optimization algorithm.} To optimize the bilevel optimization objectives of meta-learning, we propose a gradient-based algorithm, and we establish an upper bound for its convergence rate w.r.t. the learning rates and controlling factor for reward modeling.&#10;&#09;\item {State-of-the-art performance.} We conduct experiments on both controlled sentiment generation and knowledge answer generation, using multiple metrics for evaluation. The results outperform a range of strong baselines and achieve the best results across four sentiment generation distributions and 18 answer generation distributions. The code is released at.&#10;\end{itemize}&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Machine Learning, Computer Science, Language Models, Artificial Intelligence, Mathematics, Natural Language Processing, Preference Learning, Meta-Learning" />
        </attvalues>
      </node>
      <node id="2301.03652" label="2301.03652">
        <attvalues>
          <attvalue for="0" value="On The Fragility of Learned Reward Functions" />
          <attvalue for="1" value="  Reward functions are notoriously difficult to specify, especially for tasks&#10;with complex goals. Reward learning approaches attempt to infer reward&#10;functions from human feedback and preferences. Prior works on reward learning&#10;have mainly focused on the performance of policies trained alongside the reward&#10;function. This practice, however, may fail to detect learned rewards that are&#10;not capable of training new policies from scratch and thus do not capture the&#10;intended behavior. Our work focuses on demonstrating and studying the causes of&#10;these relearning failures in the domain of preference-based reward learning. We&#10;demonstrate with experiments in tabular and continuous control environments&#10;that the severity of relearning failures can be sensitive to changes in reward&#10;model design and the trajectory dataset composition. Based on our findings, we&#10;emphasize the need for more retraining-based evaluations in the literature.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2210.01241" label="2210.01241">
        <attvalues>
          <attvalue for="0" value="Is Reinforcement Learning (Not) for Natural Language Processing:&#10;  Benchmarks, Baselines, and Building Blocks for Natural Language Policy&#10;  Optimization" />
          <attvalue for="1" value="  We tackle the problem of aligning pre-trained large language models (LMs)&#10;with human preferences. If we view text generation as a sequential&#10;decision-making problem, reinforcement learning (RL) appears to be a natural&#10;conceptual framework. However, using RL for LM-based generation faces empirical&#10;challenges, including training instability due to the combinatorial action&#10;space, as well as a lack of open-source libraries and benchmarks customized for&#10;LM alignment. Thus, a question rises in the research community: is RL a&#10;practical paradigm for NLP?&#10;  To help answer this, we first introduce an open-source modular library,&#10;RL4LMs (Reinforcement Learning for Language Models), for optimizing language&#10;generators with RL. The library consists of on-policy RL algorithms that can be&#10;used to train any encoder or encoder-decoder LM in the HuggingFace library&#10;(Wolf et al. 2020) with an arbitrary reward function. Next, we present the GRUE&#10;(General Reinforced-language Understanding Evaluation) benchmark, a set of 6&#10;language generation tasks which are supervised not by target strings, but by&#10;reward functions which capture automated measures of human preference. GRUE is&#10;the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally,&#10;we introduce an easy-to-use, performant RL algorithm, NLPO (Natural Language&#10;Policy Optimization) that learns to effectively reduce the combinatorial action&#10;space in language generation. We show 1) that RL techniques are generally&#10;better than supervised methods at aligning LMs to human preferences; and 2)&#10;that NLPO exhibits greater stability and performance than previous policy&#10;gradient methods (e.g., PPO (Schulman et al. 2017)), based on both automatic&#10;and human evaluations.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1910.04887" label="1910.04887">
        <attvalues>
          <attvalue for="0" value="Visual Natural Language Query Auto-Completion for Estimating Instance&#10;  Probabilities" />
          <attvalue for="1" value="  We present a new task of query auto-completion for estimating instance&#10;probabilities. We complete a user query prefix conditioned upon an image. Given&#10;the complete query, we fine tune a BERT embedding for estimating probabilities&#10;of a broad set of instances. The resulting instance probabilities are used for&#10;selection while being agnostic to the segmentation or attention mechanism. Our&#10;results demonstrate that auto-completion using both language and vision&#10;performs better than using only language, and that fine tuning a BERT embedding&#10;allows to efficiently rank instances in the image. In the spirit of&#10;reproducible research we make our data, models, and code available.&#10;" />
          <attvalue for="2" value="&#10;This work focuses on the problem of finding objects in an image based on natural language descriptions. Existing solutions take into account both the image and the query \cite{hu2016segmentation,Hu_2016_CVPR,shi2018key}. In our problem formulation, rather than having the entire text, we are given only a prefix of the text which requires completing the text based on a language model and the image, and finding a relevant object in the image. We decompose the problem into three components: (i) completing the query from text prefix and an image; (ii) estimating probabilities of objects based on the completed text, and (iii) segmenting and classifying all instances in the image. We combine, extend, and modify state of the art components: (i) we extend a FactorCell LSTM \cite{jaech2018personalized, jaech2018low} which conditionally completes text to complete a query from both a text prefix and an image; (ii) we fine tune a BERT embedding to compute instance probabilities from a complete sentence, and (iii) we use Mask-RCNN \cite{maskrcnn2017} for instance segmentation.&#10;&#10;Recent natural language embeddings \cite{devlin2018bert} have been trained with the objectives of predicting masked words and determining whether sentences follow each other, and are efficiently used across a dozen of natural language processing tasks. Sequence models have been conditioned to complete text from a prefix and index \cite{jaech2018personalized}, however have not been extended to take into account an image. Deep neural networks have been trained to segment all instances in an image at very high quality \cite{maskrcnn2017,Hu_2018}.&#10;We propose a novel method of natural language query auto-completion for estimating instance probabilities conditioned on the image and a user query prefix. Our system combines and modifies state of the art components used in query completion, language embedding, and masked instance segmentation. Estimating a broad set of instance probabilities enables selection which is agnostic to the segmentation procedure.&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Machine Learning, Linguistics, Instance Probability Estimation, Multimodal Learning, Computer Vision, Query Auto-Completion, Artificial Intelligence, Natural Language Processing" />
        </attvalues>
      </node>
      <node id="1603.06180" label="1603.06180">
        <attvalues>
          <attvalue for="0" value="Segmentation from Natural Language Expressions" />
          <attvalue for="1" value="  In this paper we approach the novel problem of segmenting an image based on a&#10;natural language expression. This is different from traditional semantic&#10;segmentation over a predefined set of semantic classes, as e.g., the phrase&#10;&quot;two men sitting on the right bench&quot; requires segmenting only the two people on&#10;the right bench and no one standing or sitting on another bench. Previous&#10;approaches suitable for this task were limited to a fixed set of categories&#10;and/or rectangular regions. To produce pixelwise segmentation for the language&#10;expression, we propose an end-to-end trainable recurrent and convolutional&#10;network model that jointly learns to process visual and linguistic information.&#10;In our model, a recurrent LSTM network is used to encode the referential&#10;expression into a vector representation, and a fully convolutional network is&#10;used to a extract a spatial feature map from the image and output a spatial&#10;response map for the target object. We demonstrate on a benchmark dataset that&#10;our model can produce quality segmentation output from the natural language&#10;expression, and outperforms baseline methods by a large margin.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1511.04164" label="1511.04164">
        <attvalues>
          <attvalue for="0" value="Natural Language Object Retrieval" />
          <attvalue for="1" value="  In this paper, we address the task of natural language object retrieval, to&#10;localize a target object within a given image based on a natural language query&#10;of the object. Natural language object retrieval differs from text-based image&#10;retrieval task as it involves spatial information about objects within the&#10;scene and global scene context. To address this issue, we propose a novel&#10;Spatial Context Recurrent ConvNet (SCRC) model as scoring function on candidate&#10;boxes for object retrieval, integrating spatial configurations and global&#10;scene-level contextual information into the network. Our model processes query&#10;text, local image descriptors, spatial configurations and global context&#10;features through a recurrent network, outputs the probability of the query text&#10;conditioned on each candidate box as a score for the box, and can transfer&#10;visual-linguistic knowledge from image captioning domain to our task.&#10;Experimental results demonstrate that our method effectively utilizes both&#10;local and global information, outperforming previous baseline methods&#10;significantly on different datasets and scenarios, and can exploit large scale&#10;vision and language datasets for knowledge transfer.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1804.09661" label="1804.09661">
        <attvalues>
          <attvalue for="0" value="Personalized Language Model for Query Auto-Completion" />
          <attvalue for="1" value="  Query auto-completion is a search engine feature whereby the system suggests&#10;completed queries as the user types. Recently, the use of a recurrent neural&#10;network language model was suggested as a method of generating query&#10;completions. We show how an adaptable language model can be used to generate&#10;personalized completions and how the model can use online updating to make&#10;predictions for users not seen during training. The personalized predictions&#10;are significantly better than a baseline that uses no user information.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1703.06870" label="1703.06870">
        <attvalues>
          <attvalue for="0" value="Mask R-CNN" />
          <attvalue for="1" value="  We present a conceptually simple, flexible, and general framework for object&#10;instance segmentation. Our approach efficiently detects objects in an image&#10;while simultaneously generating a high-quality segmentation mask for each&#10;instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a&#10;branch for predicting an object mask in parallel with the existing branch for&#10;bounding box recognition. Mask R-CNN is simple to train and adds only a small&#10;overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to&#10;generalize to other tasks, e.g., allowing us to estimate human poses in the&#10;same framework. We show top results in all three tracks of the COCO suite of&#10;challenges, including instance segmentation, bounding-box object detection, and&#10;person keypoint detection. Without bells and whistles, Mask R-CNN outperforms&#10;all existing, single-model entries on every task, including the COCO 2016&#10;challenge winners. We hope our simple and effective approach will serve as a&#10;solid baseline and help ease future research in instance-level recognition.&#10;Code has been made available at: https://github.com/facebookresearch/Detectron&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.10370" label="1711.10370">
        <attvalues>
          <attvalue for="0" value="Learning to Segment Every Thing" />
          <attvalue for="1" value="  Most methods for object instance segmentation require all training examples&#10;to be labeled with segmentation masks. This requirement makes it expensive to&#10;annotate new categories and has restricted instance segmentation models to ~100&#10;well-annotated classes. The goal of this paper is to propose a new partially&#10;supervised training paradigm, together with a novel weight transfer function,&#10;that enables training instance segmentation models on a large set of categories&#10;all of which have box annotations, but only a small fraction of which have mask&#10;annotations. These contributions allow us to train Mask R-CNN to detect and&#10;segment 3000 visual concepts using box annotations from the Visual Genome&#10;dataset and mask annotations from the 80 classes in the COCO dataset. We&#10;evaluate our approach in a controlled study on the COCO dataset. This work is a&#10;first step towards instance segmentation models that have broad comprehension&#10;of the visual world.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2310.17558" label="2310.17558">
        <attvalues>
          <attvalue for="0" value="Towards Matching Phones and Speech Representations" />
          <attvalue for="1" value="  Learning phone types from phone instances has been a long-standing problem,&#10;while still being open. In this work, we revisit this problem in the context of&#10;self-supervised learning, and pose it as the problem of matching cluster&#10;centroids to phone embeddings. We study two key properties that enable&#10;matching, namely, whether cluster centroids of self-supervised representations&#10;reduce the variability of phone instances and respect the relationship among&#10;phones. We then use the matching result to produce pseudo-labels and introduce&#10;a new loss function for improving self-supervised representations. Our&#10;experiments show that the matching result captures the relationship among&#10;phones. Training the new loss function jointly with the regular self-supervised&#10;losses, such as APC and CPC, significantly improves the downstream phone&#10;classification.&#10;" />
          <attvalue for="2" value="&#10;&#10;Frame representation produced by self-supervised models have shown to work well at distinguishing phone identities at the frame level~\cite{oord2018cpc,chung2019an,yang2022auto,yang21c_interspeech} and at the segment level \cite{chorowski21b_interspeech, bhati2022unsupervised, cuervo2022variable}.&#10;Although phones are thought to be largely separable in the space of speech representation,&#10;the separability is based on instances.&#10;Whether it is possible to learn representations of types as opposed to instances&#10;remains an open problem.&#10;&#10;In this paper, we will focus on phone types, because phonetic properties&#10;are salient even after crude quantization down to 100 or even 50 codes&#10;\cite{hsu2021hubert, wells2022phonetic,Amitay2023}.&#10;The problem of learning phone types has a long history and&#10;shows up under different names, such as acoustic unit discovery \cite{park2008unsup,zero2015,zero2017}&#10;and unsupervised speech recognition \cite{liu18g_interspeech,chen19e_interspeech,liu2022,baevski2021unsupervised,liu2023towards,yeh2018unsupervised}.&#10;It is also highly related to unsupervised phone segmentation \cite{kreuk20_interspeech,zero2015,zero2017}&#10;and lexical discovery~\cite{kamper2017segmental}.&#10;&#10;Most, if not all, approaches assume that types arise from clustering of instances.&#10;Approaches based on dynamic time warping for acoustic unit discovery rely on&#10;clustering \cite{park2008unsup}.&#10;Bayesian models for acoustic unit discovery is another form of clustering \cite{lee2012nonparametric}.&#10;Much of the recent progress focuses on adversarial approaches to distinguish&#10;sequences of cluster IDs and phone sequences observed in data sets \cite{liu18g_interspeech,chen19e_interspeech,liu2022,baevski2021unsupervised,liu2023towards}.&#10;In this paper, we follow the same footsteps and study&#10;whether cluster centroids of instances can be&#10;a good representation for types in the context of self-supervised learning.&#10;&#10;Even though cluster centroids have been used as types for almost all&#10;approaches to learning phone types,&#10;it is also true that the types discovered by these approaches are never one to one.&#10;There is abundant variability of phones within and across speakers,&#10;and we generally do not know the correct number of clusters to use.&#10;Recent work on analyzing representations of self-supervised predictive coding finds&#10;that phonetic and speaker information are largely represented&#10;in orthogonal subspaces~\cite{liu2023self}.&#10;We will use this as a tool to measure and reduce the variability&#10;of phone instances among speakers.&#10;&#10;For the second problem where there is not always a one-to-one mapping&#10;between the centroids and phone types, we take a different route&#10;and assume access to a set of phone embeddings.&#10;We introduce an additional matching step to find&#10;the correspondence between the centroids and phone embeddings.&#10;This is reminiscent to the line of work in unsupervised machine&#10;translation~\cite{lample2018word,2018gromov,alaux2018unsupervised}.&#10;We will adopt a similar approach, using optimal transport,&#10;in particular, optimizing the Gromov-Wasserstein&#10;distance~\cite{memoli2011gromov, peyre2016gromov}&#10;to match the centroids and phone embeddings.&#10;&#10;We see several applications once we match the centroids and phone embeddings.&#10;One immediate application is to use the matching result&#10;to provide pseudo-labels on unlabeled speech.&#10;The pseudo-labels can be used to further improve the speech representations.&#10;We introduce a loss function predicting pseudo-labels&#10;as self-supervised learning, similar to~\cite{hsu2021hubert,chiu2022self}.&#10;&#10;Our experiments show that centroids of self-supervised frame representation&#10;provide a good starting point for learning types.&#10;We will show quantitative and qualitive results for matching centeroids&#10;to phone embeddings.&#10;In addition, by predicting the pseudo-labels produced by the&#10;matching result gives a sizable improvement in&#10;downstream phone classification.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Linguistics, Signal Processing, Mathematics, Self-Supervised Learning, Phone Classification, Representation Learning" />
        </attvalues>
      </node>
      <node id="2104.11946" label="2104.11946">
        <attvalues>
          <attvalue for="0" value="Aligned Contrastive Predictive Coding" />
          <attvalue for="1" value="  We investigate the possibility of forcing a self-supervised model trained&#10;using a contrastive predictive loss to extract slowly varying latent&#10;representations. Rather than producing individual predictions for each of the&#10;future representations, the model emits a sequence of predictions shorter than&#10;that of the upcoming representations to which they will be aligned. In this&#10;way, the prediction network solves a simpler task of predicting the next&#10;symbols, but not their exact timing, while the encoding network is trained to&#10;produce piece-wise constant latent codes. We evaluate the model on a speech&#10;coding task and demonstrate that the proposed Aligned Contrastive Predictive&#10;Coding (ACPC) leads to higher linear phone prediction accuracy and lower ABX&#10;error rates, while being slightly faster to train due to the reduced number of&#10;prediction heads.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2207.14568" label="2207.14568">
        <attvalues>
          <attvalue for="0" value="Learning Phone Recognition from Unpaired Audio and Phone Sequences Based&#10;  on Generative Adversarial Network" />
          <attvalue for="1" value="  ASR has been shown to achieve great performance recently. However, most of&#10;them rely on massive paired data, which is not feasible for low-resource&#10;languages worldwide. This paper investigates how to learn directly from&#10;unpaired phone sequences and speech utterances. We design a two-stage iterative&#10;framework. GAN training is adopted in the first stage to find the mapping&#10;relationship between unpaired speech and phone sequence. In the second stage,&#10;another HMM model is introduced to train from the generator's output, which&#10;boosts the performance and provides a better segmentation for the next&#10;iteration. In the experiment, we first investigate different choices of model&#10;designs. Then we compare the framework to different types of baselines: (i)&#10;supervised methods (ii) acoustic unit discovery based methods (iii) methods&#10;learning from unpaired data. Our framework performs consistently better than&#10;all acoustic unit discovery methods and previous methods learning from unpaired&#10;data based on the TIMIT dataset.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2305.12464" label="2305.12464">
        <attvalues>
          <attvalue for="0" value="Self-supervised Predictive Coding Models Encode Speaker and Phonetic&#10;  Information in Orthogonal Subspaces" />
          <attvalue for="1" value="  Self-supervised speech representations are known to encode both speaker and&#10;phonetic information, but how they are distributed in the high-dimensional&#10;space remains largely unexplored. We hypothesize that they are encoded in&#10;orthogonal subspaces, a property that lends itself to simple disentanglement.&#10;Applying principal component analysis to representations of two predictive&#10;coding models, we identify two subspaces that capture speaker and phonetic&#10;variances, and confirm that they are nearly orthogonal. Based on this property,&#10;we propose a new speaker normalization method which collapses the subspace that&#10;encodes speaker information, without requiring transcriptions. Probing&#10;experiments show that our method effectively eliminates speaker information and&#10;outperforms a previous baseline in phone discrimination tasks. Moreover, the&#10;approach generalizes and can be used to remove information of unseen speakers.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1809.00013" label="1809.00013">
        <attvalues>
          <attvalue for="0" value="Gromov-Wasserstein Alignment of Word Embedding Spaces" />
          <attvalue for="1" value="  Cross-lingual or cross-domain correspondences play key roles in tasks ranging&#10;from machine translation to transfer learning. Recently, purely unsupervised&#10;methods operating on monolingual embeddings have become effective alignment&#10;tools. Current state-of-the-art methods, however, involve multiple steps,&#10;including heuristic post-hoc refinement strategies. In this paper, we cast the&#10;correspondence problem directly as an optimal transport (OT) problem, building&#10;on the idea that word embeddings arise from metric recovery algorithms. Indeed,&#10;we exploit the Gromov-Wasserstein distance that measures how similarities&#10;between pairs of words relate across languages. We show that our OT objective&#10;can be estimated efficiently, requires little or no tuning, and results in&#10;performance comparable with the state-of-the-art in various unsupervised word&#10;translation tasks.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="2009.05485" label="2009.05485">
        <attvalues>
          <attvalue for="0" value="Text-Independent Speaker Verification with Dual Attention Network" />
          <attvalue for="1" value="  This paper presents a novel design of attention model for text-independent&#10;speaker verification. The model takes a pair of input utterances and generates&#10;an utterance-level embedding to represent speaker-specific characteristics in&#10;each utterance. The input utterances are expected to have highly similar&#10;embeddings if they are from the same speaker. The proposed attention model&#10;consists of a self-attention module and a mutual attention module, which&#10;jointly contributes to the generation of the utterance-level embedding. The&#10;self-attention weights are computed from the utterance itself while the&#10;mutual-attention weights are computed with the involvement of the other&#10;utterance in the input pairs. As a result, each utterance is represented by a&#10;self-attention weighted embedding and a mutual-attention weighted embedding.&#10;The similarity between the embeddings is measured by a cosine distance score&#10;and a binary classifier output score. The whole model, named Dual Attention&#10;Network, is trained end-to-end on Voxceleb database. The evaluation results on&#10;Voxceleb 1 test set show that the Dual Attention Network significantly&#10;outperforms the baseline systems. The best result yields an equal error rate of&#10;1:6%.&#10;" />
          <attvalue for="2" value="&#10;Speaker verification (SV) refers to the process of determining whether an input speech utterance is from a claimed speaker. If the claimed speaker is represented by a reference utterance, the task of SV is essentially to determine whether the two utterances are from the same person or not. In text-dependent SV, all utterances are required to contain the same speech content \cite{variani2014deep}. Whilst in text-independent SV, the spoken content is unrestricted \cite{campbell1997speaker}. The present study is focused on text-independent SV with a pair of input utterances.&#10;&#10;In recent years, embeddings learned by deep neural network (DNN) are widely applied to both text-dependent and text-independent SV \cite{variani2014deep,snyder2018x}. In a typical DNN pipeline of embedding generation, the input speech utterance is first converted into frame-level acoustic representations, e.g., log Mel-filterbank (FBank) or Mel frequency cepstral coefficients (MFCC). The acoustic representations are transformed by a DNN into another type of frame-level features. The DNN output features have variable length, which is determined by the time duration of input utterance. A method of aggregation is applied to convert the variable-length DNN features into a fixed-length embedding that represents speaker's characteristics as reflected in the utterance. Given a test utterance and a reference utterance, speaker verification is performed based on similarity measure between their embeddings. $Average\ Pooling$, i.e., taking simple time average of frame-level DNN features, is an intuitive approach to feature aggregation \cite{Chung18b}. $Statistics\ Pooling$ \cite{snyder2017deep} computes the mean and standard deviation of frame-level features as the utterance-level representation. In these methods, features from different parts of the utterance are assumed to be equally important, and temporal relation between the features is not considered. In \cite{heigold2016end}, a recurrent neural network (RNN) is used to capture temporal dependency and derive utterance-level embedding for SV.&#10;&#10;Attention mechanism in DNN has been shown effective in various application areas \cite{chorowski2015attention,zeyer2018improved,vaswani2017attention,wang2018non}. In simple terms, attention to selected parts of a feature is realized through a method of determining and imposing heavier weights, so as to make these parts more salient and play a more important role in the intended task. In the case of SV, attention mechanism can be implemented in the process of aggregating frame-level features with learned attention weights \cite{bhattacharya2017deep,okabe2018attentive}. This approach showed better performance than $Average$ or $Statistics\ Pooling$, confirming that speaker-relevant information is not evenly distributed in an utterance. Typically the attention weights used to compute the embedding for an input utterance are derived from this utterance itself. This is known as self-attention. As the SV process involves two input utterances, the information from both utterances could be exploited to improve the attention mechanism. This idea of collaborative attention was applied in video-based person re-identification\cite{zhang2019scan} and text-dependent SV\cite{zhang2019seq2seq}. In this paper, the use of $Dual\ Attention$ mechanism is proposed for combining frame-level DNN features in text-independent SV. The attention model comprises two component modules:&#10;\begin{itemize}&#10;\item $Self$-$attention$: the attention weights for each of the two utterances are computed from DNN features of the utterance itself;&#10;\item $Mutual$-$attention$: DNN features of the two utterances collaborate with each other to generate the attention weights&#10;\end{itemize}&#10;&#10;The utterance-level embeddings generated by the $Dual\ Attention$ model are passed to a binary classifier to determine whether the two utterances are from the same speaker or not. The classifier output can be regarded as a similarity score. It is further combined with a cosine distance to produce the final similarity score. The proposed $Dual\ Attention\ Network$ (abbreviated as D-att Net) is trained end-to-end.&#10;&#10;" />
          <attvalue for="3" value="" />
          <attvalue for="4" value="Computer Science, Speaker Verification, Linguistics, Attention Models, Deep Learning, Signal Processing, Mathematics" />
        </attvalues>
      </node>
      <node id="1509.08062" label="1509.08062">
        <attvalues>
          <attvalue for="0" value="End-to-End Text-Dependent Speaker Verification" />
          <attvalue for="1" value="  In this paper we present a data-driven, integrated approach to speaker&#10;verification, which maps a test utterance and a few reference utterances&#10;directly to a single score for verification and jointly optimizes the system's&#10;components using the same evaluation protocol and metric as at test time. Such&#10;an approach will result in simple and efficient systems, requiring little&#10;domain-specific knowledge and making few model assumptions. We implement the&#10;idea by formulating the problem as a single neural network architecture,&#10;including the estimation of a speaker model on only a few utterances, and&#10;evaluate it on our internal &quot;Ok Google&quot; benchmark for text-dependent speaker&#10;verification. The proposed approach appears to be very effective for big data&#10;applications like ours that require highly accurate, easy-to-maintain systems&#10;with a small footprint.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1711.07971" label="1711.07971">
        <attvalues>
          <attvalue for="0" value="Non-local Neural Networks" />
          <attvalue for="1" value="  Both convolutional and recurrent operations are building blocks that process&#10;one local neighborhood at a time. In this paper, we present non-local&#10;operations as a generic family of building blocks for capturing long-range&#10;dependencies. Inspired by the classical non-local means method in computer&#10;vision, our non-local operation computes the response at a position as a&#10;weighted sum of the features at all positions. This building block can be&#10;plugged into many computer vision architectures. On the task of video&#10;classification, even without any bells and whistles, our non-local models can&#10;compete or outperform current competition winners on both Kinetics and Charades&#10;datasets. In static image recognition, our non-local models improve object&#10;detection/segmentation and pose estimation on the COCO suite of tasks. Code is&#10;available at https://github.com/facebookresearch/video-nonlocal-net .&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1807.05688" label="1807.05688">
        <attvalues>
          <attvalue for="0" value="SCAN: Self-and-Collaborative Attention Network for Video Person&#10;  Re-identification" />
          <attvalue for="1" value="  Video person re-identification attracts much attention in recent years. It&#10;aims to match image sequences of pedestrians from different camera views.&#10;Previous approaches usually improve this task from three aspects, including a)&#10;selecting more discriminative frames, b) generating more informative temporal&#10;representations, and c) developing more effective distance metrics. To address&#10;the above issues, we present a novel and practical deep architecture for video&#10;person re-identification termed Self-and-Collaborative Attention Network&#10;(SCAN). It has several appealing properties. First, SCAN adopts non-parametric&#10;attention mechanism to refine the intra-sequence and inter-sequence feature&#10;representation of videos, and outputs self-and-collaborative feature&#10;representation for each video, making the discriminative frames aligned between&#10;the probe and gallery sequences.Second, beyond existing models, a generalized&#10;pairwise similarity measurement is proposed to calculate the similarity feature&#10;representations of video pairs, enabling computing the matching scores by the&#10;binary classifier. Third, a dense clip segmentation strategy is also introduced&#10;to generate rich probe-gallery pairs to optimize the model. Extensive&#10;experiments demonstrate the effectiveness of SCAN, which outperforms the&#10;best-performing baselines on iLIDS-VID, PRID2011 and MARS dataset,&#10;respectively.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
      <node id="1606.02960" label="1606.02960">
        <attvalues>
          <attvalue for="0" value="Sequence-to-Sequence Learning as Beam-Search Optimization" />
          <attvalue for="1" value="  Sequence-to-Sequence (seq2seq) modeling has rapidly become an important&#10;general-purpose NLP tool that has proven effective for many text-generation and&#10;sequence-labeling tasks. Seq2seq builds on deep neural language modeling and&#10;inherits its remarkable accuracy in estimating local, next-word distributions.&#10;In this work, we introduce a model and beam-search training scheme, based on&#10;the work of Daume III and Marcu (2005), that extends seq2seq to learn global&#10;sequence scores. This structured approach avoids classical biases associated&#10;with local training and unifies the training loss with the test-time usage,&#10;while preserving the proven model architecture of seq2seq and its efficient&#10;training approach. We show that our system outperforms a highly-optimized&#10;attention-based seq2seq system and other baselines on three different sequence&#10;to sequence tasks: word ordering, parsing, and machine translation.&#10;" />
          <attvalue for="2" value="" />
          <attvalue for="3" value="" />
        </attvalues>
      </node>
    </nodes>
    <edges>
      <edge source="2406.16838" target="2001.08361" id="0">
        <attvalues>
          <attvalue for="5" value=" One of the most striking findings in modern research on large language models (LLMs) is that, given a model and dataset of sufficient scale, scaling up the compute used at training time leads to better final results \cite{kaplan2020scaling,hoffmann2022an}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="1904.09751" id="2">
        <attvalues>
          <attvalue for="5" value=" Token-level generation algorithms, often called decoding algorithms, have a rich history in natural language processing, ranging from classical greedy decoding and beam search to modern sampling algorithms such as nucleus~\cite{Holtzman2020The} and $\eta$-sampling~\cite{hewitt-etal-2022-truncation}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2210.15191" id="3">
        <attvalues>
          <attvalue for="5" value=" Token-level generation algorithms, often called decoding algorithms, have a rich history in natural language processing, ranging from classical greedy decoding and beam search to modern sampling algorithms such as nucleus~\cite{Holtzman2020The} and $\eta$-sampling~\cite{hewitt-etal-2022-truncation}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2212.14024" id="4">
        <attvalues>
          <attvalue for="5" value=" Recently there has been growing interest in meta-generation algorithms---algorithms that operate on partial or full sequences, and treat the LLM as a black box that is called as part of a larger generation program (\ref{fig:roadmap};~\cite{khattab2022demonstrate,dohan2022language,schlag2023large})." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2207.10342" id="5">
        <attvalues>
          <attvalue for="5" value=" Recently there has been growing interest in meta-generation algorithms---algorithms that operate on partial or full sequences, and treat the LLM as a black box that is called as part of a larger generation program (\ref{fig:roadmap};~\cite{khattab2022demonstrate,dohan2022language,schlag2023large})." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2305.05364" id="6">
        <attvalues>
          <attvalue for="5" value=" Recently there has been growing interest in meta-generation algorithms---algorithms that operate on partial or full sequences, and treat the LLM as a black box that is called as part of a larger generation program (\ref{fig:roadmap};~\cite{khattab2022demonstrate,dohan2022language,schlag2023large})." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2305.10601" id="7">
        <attvalues>
          <attvalue for="5" value=" Meta-generators can increase the compute resources devoted to generation by making multiple model calls, augmenting the model with search algorithms~\cite{yao2023tree,madaan2023selfrefine}, or incorporating external data sources." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2303.17651" id="8">
        <attvalues>
          <attvalue for="5" value=" Meta-generators can increase the compute resources devoted to generation by making multiple model calls, augmenting the model with search algorithms~\cite{yao2023tree,madaan2023selfrefine}, or incorporating external data sources." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2009.01325" id="10">
        <attvalues>
          <attvalue for="5" value=" Doing so has seen success in improving task performance (eg, problem solving~\cite{lewkowycz2022solving}) and steering the output distribution (eg, with human preferences~\cite{stiennon2020}), and may offer a way to overcome limitations of standard LLMs such as error accumulation~\cite{dziri2023faith} and computational capacity~\cite{merrill2024the}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2305.18654" id="11">
        <attvalues>
          <attvalue for="5" value=" Doing so has seen success in improving task performance (eg, problem solving~\cite{lewkowycz2022solving}) and steering the output distribution (eg, with human preferences~\cite{stiennon2020}), and may offer a way to overcome limitations of standard LLMs such as error accumulation~\cite{dziri2023faith} and computational capacity~\cite{merrill2024the}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2310.07923" id="12">
        <attvalues>
          <attvalue for="5" value=" Doing so has seen success in improving task performance (eg, problem solving~\cite{lewkowycz2022solving}) and steering the output distribution (eg, with human preferences~\cite{stiennon2020}), and may offer a way to overcome limitations of standard LLMs such as error accumulation~\cite{dziri2023faith} and computational capacity~\cite{merrill2024the}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2203.15721" id="13">
        <attvalues>
          <attvalue for="5" value=" \cite{wiher2022decodingstrategiesneuraltext} presents a detailed analysis of a smaller set of decoding strategies, while \cite{info12090355} spotlight token-level methods, with a particular focus on considerations for encoder-decoder models." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2107.13586" id="14">
        <attvalues>
          <attvalue for="5" value=" In parallel, several surveys have addressed prompting and related methods \cite{liu2021pretrainpromptpredictsystematic,sahoo2024systematicsurveypromptengineering}, though these works do not address token-level methods." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2402.07927" id="15">
        <attvalues>
          <attvalue for="5" value=" In parallel, several surveys have addressed prompting and related methods \cite{liu2021pretrainpromptpredictsystematic,sahoo2024systematicsurveypromptengineering}, though these works do not address token-level methods." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2307.07982" id="16">
        <attvalues>
          <attvalue for="5" value=" Recent surveys have also considered strategies for speeding up inference \cite{chittyvenkata2023surveytechniquesoptimizingtransformer,miao2023efficientgenerativelargelanguage,khoshnoodi2024comprehensivesurveyacceleratedgeneration, wang2024modelcompressionefficientinference}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2312.15234" id="17">
        <attvalues>
          <attvalue for="5" value=" Recent surveys have also considered strategies for speeding up inference \cite{chittyvenkata2023surveytechniquesoptimizingtransformer,miao2023efficientgenerativelargelanguage,khoshnoodi2024comprehensivesurveyacceleratedgeneration, wang2024modelcompressionefficientinference}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2405.13019" id="18">
        <attvalues>
          <attvalue for="5" value=" Recent surveys have also considered strategies for speeding up inference \cite{chittyvenkata2023surveytechniquesoptimizingtransformer,miao2023efficientgenerativelargelanguage,khoshnoodi2024comprehensivesurveyacceleratedgeneration, wang2024modelcompressionefficientinference}." />
        </attvalues>
      </edge>
      <edge source="2406.16838" target="2204.09269" id="19">
        <attvalues>
          <attvalue for="5" value=" Finally, \cite{xiao2023surveynonautoregressivegenerationneural} focus on non-autoregressive generation, while our survey focuses on autoregressive generation." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2005.11401" id="27">
        <attvalues>
          <attvalue for="5" value=" Retrieval Augmented Generation (RAG) \cite{lewis2020retrieval} is a prominent technique in this regard, augmenting LLMs by linking them to up-to-date external databases to improve the precision of their outputs." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2201.11903" id="28">
        <attvalues>
          <attvalue for="5" value=" Additionally, advanced prompting strategies like chain-of-thought prompting \cite{wei2022chain} aid in improving the reasoning capabilities of LLMs, especially for tasks requiring multi-step logic." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2211.10435" id="29">
        <attvalues>
          <attvalue for="5" value=" Further, the paper delves into the Program-Aided Language Model (PAL) framework \cite{gao2023pal}, which pairs LLMs with external code interpreters to execute accurate calculations." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2210.03629" id="30">
        <attvalues>
          <attvalue for="5" value=" Recent advancements such as the ReAct \cite{yao2022react} framework and LangChain \cite{topsakal2023creating} are also explored, highlighting their potential in guiding LLMs through structured prompts to solve intricate problems and support advanced techniques like PAL." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2304.11277" id="31">
        <attvalues>
          <attvalue for="5" value=" Techniques for scaling model training beyond a single GPU, including PyTorch’s Distributed Data Parallel (DDP) and Fully Sharded Data Parallel (FSDP) \cite{zhao2023pytorch}, are discussed, along with the ZeRO \cite{rajbhandari2019zero} stages that optimize memory usage during training." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2106.09685" id="32">
        <attvalues>
          <attvalue for="5" value=" The paper examines instruction fine-tuning, multitask fine-tuning, and parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) \cite{hu2021lora} and prompt tuning \cite{lester2021prompt}." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2104.08691" id="33">
        <attvalues>
          <attvalue for="5" value=" The paper examines instruction fine-tuning, multitask fine-tuning, and parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) \cite{hu2021lora} and prompt tuning \cite{lester2021prompt}." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="1909.08593" id="34">
        <attvalues>
          <attvalue for="5" value=" Reinforcement Learning from Human Feedback (RLHF) \cite{ziegler2019,glaese2022improving} and Reinforced Self-Training \cite{gulcehre2023reinforced} are also explored as a method to align LLMs with human preferences." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2209.14375" id="35">
        <attvalues>
          <attvalue for="5" value=" Reinforcement Learning from Human Feedback (RLHF) \cite{ziegler2019,glaese2022improving} and Reinforced Self-Training \cite{gulcehre2023reinforced} are also explored as a method to align LLMs with human preferences." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="2308.08998" id="36">
        <attvalues>
          <attvalue for="5" value=" Reinforcement Learning from Human Feedback (RLHF) \cite{ziegler2019,glaese2022improving} and Reinforced Self-Training \cite{gulcehre2023reinforced} are also explored as a method to align LLMs with human preferences." />
        </attvalues>
      </edge>
      <edge source="2407.12036" target="1707.06347" id="37">
        <attvalues>
          <attvalue for="5" value=" Finally, the paper provides an introduction to PPO \cite{schulman2017proximal}, its phases, and its objectives, including policy loss, value function loss, and entropy loss." />
        </attvalues>
      </edge>
      <edge source="2104.08691" target="1802.05365" id="38">
        <attvalues>
          <attvalue for="5" value=" ELMo \cite{peters-etal-2018-deep} proposed freezing the pre-trained model and learning a task-specific weighting of its per-layer representations." />
        </attvalues>
      </edge>
      <edge source="2104.08691" target="1801.06146" id="40">
        <attvalues>
          <attvalue for="5" value=" However, since GPT \cite{gpt} and BERT \cite{devlin-etal-2019-bert}, the dominant adaptation technique has been model tuning (or ``fine-tuning''), where all model parameters are tuned during adaptation, as proposed by \cite{howard-ruder-2018-universal}." />
        </attvalues>
      </edge>
      <edge source="2104.08691" target="2005.14165" id="41">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{brown_2020_gpt3} showed that prompt design (or ``priming'') is surprisingly effective at modulating a frozen GPT-3 model's behavior through text prompts." />
        </attvalues>
      </edge>
      <edge source="2104.08691" target="2010.15980" id="43">
        <attvalues>
          <attvalue for="5" value=" \cite{shin-etal-2020-autoprompt} propose a search algorithm over the discrete space of words, guided by the downstream application training data." />
        </attvalues>
      </edge>
      <edge source="2104.08691" target="2101.00190" id="44">
        <attvalues>
          <attvalue for="5" value=" While we developed our method concurrently with \cite{li_2021_prefix_tuning} and \cite{hambardzumyan_2021_warp}, we are the first to show that prompt tuning alone (with no intermediate-layer prefixes or task-specific output layers) is sufficient to be competitive with model tuning." />
        </attvalues>
      </edge>
      <edge source="2104.08691" target="2101.00121" id="45">
        <attvalues>
          <attvalue for="5" value=" While we developed our method concurrently with \cite{li_2021_prefix_tuning} and \cite{hambardzumyan_2021_warp}, we are the first to show that prompt tuning alone (with no intermediate-layer prefixes or task-specific output layers) is sufficient to be competitive with model tuning." />
        </attvalues>
      </edge>
      <edge source="2305.14196" target="2109.01652" id="46">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have been improving at an incredible pace, solving problems that seemed out of reach, without any task-specific training examples \cite{wei2022finetuned, ouyang2022training, openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2305.14196" target="2206.04615" id="48">
        <attvalues>
          <attvalue for="5" value=" However, current LLM benchmarks that do evaluate models in a zero-shot setting, such as HELM \cite{liang2022holistic} and BigBench \cite{srivastava2022imitation}, mostly focus on short sequences; BigBench, for example, has an average of 77 words per input." />
        </attvalues>
      </edge>
      <edge source="2305.14196" target="2201.03533" id="49">
        <attvalues>
          <attvalue for="5" value=" ZeroScrolls extends Scrolls \cite{shaham-etal-2022-scrolls}, a long text understanding benchmark that enables fine-tuning, adding four additional tasks: query-based summarization, multi-hop question answering, sentiment aggregation, and sorting book section summaries." />
        </attvalues>
      </edge>
      <edge source="2305.14196" target="2112.08608" id="50">
        <attvalues>
          <attvalue for="5" value=" On question answering tasks, we find that zero-shot LLMs bridge the gap with task-specific fine-tuned models; GPT-4 sets a new state of the art on the challenging QuALITY task \cite{pang-etal-2022-quality}, almost reaching human performance." />
        </attvalues>
      </edge>
      <edge source="2404.13033" target="2005.14165" id="51">
        <attvalues>
          <attvalue for="5" value=" In this background, Prompt Engineering (PE) has emerged as a key area in leveraging cutting-edge LLMs, leading to advances in applying LLMs to new tasks \cite{GPT-3}, enhancing logical reasoning \cite{wei2022COT}, and increasing task-specific accuracy \cite{Wang2023Prompt-health,wei2023chatie}, without updating model weights." />
        </attvalues>
      </edge>
      <edge source="2404.13033" target="2204.02311" id="52">
        <attvalues>
          <attvalue for="5" value=" The emergence of Large Language Models (LLMs) such as GPT-3 \cite{GPT-3}, PaLM \cite{chowdhery2023palm}, LLaMA \cite{touvron2023llama-1} and GPT-4 \cite{achiam2023gpt4} revolutionized natural language processing (NLP), enabling complex tasks to be tackled with a single model." />
        </attvalues>
      </edge>
      <edge source="2404.13033" target="2303.08774" id="54">
        <attvalues>
          <attvalue for="5" value=" The emergence of Large Language Models (LLMs) such as GPT-3 \cite{GPT-3}, PaLM \cite{chowdhery2023palm}, LLaMA \cite{touvron2023llama-1} and GPT-4 \cite{achiam2023gpt4} revolutionized natural language processing (NLP), enabling complex tasks to be tackled with a single model." />
        </attvalues>
      </edge>
      <edge source="2404.13033" target="2201.11903" id="55">
        <attvalues>
          <attvalue for="5" value=" In this background, Prompt Engineering (PE) has emerged as a key area in leveraging cutting-edge LLMs, leading to advances in applying LLMs to new tasks \cite{GPT-3}, enhancing logical reasoning \cite{wei2022COT}, and increasing task-specific accuracy \cite{Wang2023Prompt-health,wei2023chatie}, without updating model weights." />
        </attvalues>
      </edge>
      <edge source="2404.13033" target="2304.14670" id="56">
        <attvalues>
          <attvalue for="5" value=" In this background, Prompt Engineering (PE) has emerged as a key area in leveraging cutting-edge LLMs, leading to advances in applying LLMs to new tasks \cite{GPT-3}, enhancing logical reasoning \cite{wei2022COT}, and increasing task-specific accuracy \cite{Wang2023Prompt-health,wei2023chatie}, without updating model weights." />
        </attvalues>
      </edge>
      <edge source="2404.13033" target="2301.13688" id="57">
        <attvalues>
          <attvalue for="5" value=" Different from instruction-tuning datasets like FLAN \cite{longpre2023-FLAN-data}, the MASA task involves more complicated input and output elements, making it suitable for in-depth investigation of different sample designs." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2005.14165" id="58">
        <attvalues>
          <attvalue for="5" value=" Emergent abilities of Large Language Models (LLMs) rely on scaling-up~\cite{brown2020language, wei2022emergent}." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2001.08361" id="59">
        <attvalues>
          <attvalue for="5" value=" Scaling laws~\cite{kaplan2020scaling,hoffmann2022training} give clear empirical guidelines on pre-training computational-optimized LLMs, greatly stimulating and advancing the field." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2401.08092" id="60">
        <attvalues>
          <attvalue for="5" value=" However, pre-training these gigantic models comes at a significant cost in terms of energy consumption and environmental impact~\cite{xu2024survey}~(eg, pre-training Llama-3~\cite{llama3modelcard} consumes a total of 7." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2111.00364" id="61">
        <attvalues>
          <attvalue for="5" value=" The efficient pre-training of LLMs is thus crucial, both from a scientific and a societal perspective, to ensure the continual growth and adoption of AI~\cite{wu2022sustainable, DEVRIES20232191}." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="1511.05641" id="62">
        <attvalues>
          <attvalue for="5" value=" Current popular methods generally focus on expanding the parameters of the base model through techniques like splitting~\cite{chen2015net2net,chen2021bert2bert,wang2023lemon}, copying~\cite{shen2022staged,gong2019efficient}, or matrix mapping~\cite{wang2023learning}." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2110.07143" id="63">
        <attvalues>
          <attvalue for="5" value=" The primary objective is to accelerate the training of large models, and existing methods demonstrate promising speedup results on models such as BERT~\cite{chen2021bert2bert, gong2019efficient, yang2020progressively, wang2023learning, wang2023lemon, shen2022staged}." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2310.07999" id="64">
        <attvalues>
          <attvalue for="5" value=" There are also some approaches that initialize new parameters from scratch~\cite{evci2022gradmax,wang2023lemon,yao2024masked}." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2303.00980" id="66">
        <attvalues>
          <attvalue for="5" value=" This is in contrast to the previous largest LLM pre-training experiment that uses model growth methods and has baselines for comparison, which is reported in Ligo~\cite{wang2023learning}, where a GPT2-1." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2305.02869" id="68">
        <attvalues>
          <attvalue for="5" value=" There are also some approaches that initialize new parameters from scratch~\cite{evci2022gradmax,wang2023lemon,yao2024masked}." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2307.09288" id="69">
        <attvalues>
          <attvalue for="5" value=" In contrast, most growth approaches are evaluated in encoder-based BERT models~\cite{gong2019efficient, chen2021bert2bert, yang2020progressively, wang2023lemon, shen2022staged, evci2022gradmax, yao2024masked}, which have different architecture and training configurations compared to prominent decoder-based LLMs such as Llama~\cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2405.15319" target="2203.15556" id="71">
        <attvalues>
          <attvalue for="5" value=" Given the same target loss value, our analysis reveals a significantly reduced computational cost compared to the common scaling law~\cite{hoffmann2022training}." />
        </attvalues>
      </edge>
      <edge source="2408.06567" target="1706.03762" id="72">
        <attvalues>
          <attvalue for="5" value=" Language models have become a cornerstone of modern natural language processing (NLP) systems, driving applications such as machine translation, conversational agents, text summarization, and question answering \cite{vaswani2017attention, devlin2018bert}." />
        </attvalues>
      </edge>
      <edge source="2408.06567" target="2005.14165" id="73">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in large language models (LLMs) like GPT-3, BERT, and T5 have demonstrated remarkable proficiency across numerous tasks, highlighting the importance of pretraining on large-scale datasets to achieve state-of-the-art results \cite{brown2020language, raffel2020exploring}." />
        </attvalues>
      </edge>
      <edge source="2408.06567" target="2006.16668" id="74">
        <attvalues>
          <attvalue for="5" value=" By dynamically selecting different subsets of model parameters (experts) for various inputs, MoE architectures can scale to a much larger number of parameters without a corresponding increase in computational cost \cite{lepikhin2021gshard}." />
        </attvalues>
      </edge>
      <edge source="2408.06567" target="1511.05641" id="75">
        <attvalues>
          <attvalue for="5" value=" For instance, the Net2Net method accelerates learning via knowledge transfer, allowing the seamless transition of knowledge from smaller to larger networks, which shows significant acceleration in image classification task~\cite{net2net}." />
        </attvalues>
      </edge>
      <edge source="2408.06567" target="2110.07143" id="76">
        <attvalues>
          <attvalue for="5" value=" The bert2BERT approach focuses on reusing pre-trained language models to initialize new models, promoting efficiency and reusability~\cite{chen-etal-2022-bert2bert}." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="2005.14165" id="77">
        <attvalues>
          <attvalue for="5" value=" There has been a large body of work analysing these distributional biases~\cite{blodgett-bias-survey, bias_mt_stanovsky-etal-2019,bias_mt_PratesAL20,ShengCNP_LM_bias19,gpt3}." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1511.06732" id="78">
        <attvalues>
          <attvalue for="5" value=" Prior work, taking inspiration from Reinforcement Learning (RL), has aimed at inducing autoregressive models to optimize global objectives using task specific rewards such as BLEU and ROUGE for Machine Translation and Summarization~\cite{seq_lvl_train_RanzatoCAZ15, BahdanauBXGLPCB17}, or hand crafted rewards~\cite{RL_dialogue_LiMRJGG16,RL_TambwekarDMMHR19} to improve certain a priori desirable features." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1606.01541" id="79">
        <attvalues>
          <attvalue for="5" value=" Prior work, taking inspiration from Reinforcement Learning (RL), has aimed at inducing autoregressive models to optimize global objectives using task specific rewards such as BLEU and ROUGE for Machine Translation and Summarization~\cite{seq_lvl_train_RanzatoCAZ15, BahdanauBXGLPCB17}, or hand crafted rewards~\cite{RL_dialogue_LiMRJGG16,RL_TambwekarDMMHR19} to improve certain a priori desirable features." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1603.08023" id="81">
        <attvalues>
          <attvalue for="5" value=" However, such an optimization process is not infallible; \cite{LiuLSNCP16} noted that it often leads to ``degeneration'', producing poor examples that improve the average reward but forgo coherence and fluency." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1611.02796" id="82">
        <attvalues>
          <attvalue for="5" value=" \cite{KL_Jaques17,Ziegler19} propose a conservative fine-tuning approach moderated by a KL penalty between the trained policy and the original LM, discouraging large deviations." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1909.08593" id="83">
        <attvalues>
          <attvalue for="5" value=" \cite{KL_Jaques17,Ziegler19} propose a conservative fine-tuning approach moderated by a KL penalty between the trained policy and the original LM, discouraging large deviations." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1912.02164" id="84">
        <attvalues>
          <attvalue for="5" value=" A KL penalty was also used by \cite{plug_and_play_20}, this time in a plug-and-play rather than a fine-tuning context." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1909.01326" id="85">
        <attvalues>
          <attvalue for="5" value=" There has been a large body of work analysing these distributional biases~\cite{blodgett-bias-survey, bias_mt_stanovsky-etal-2019,bias_mt_PratesAL20,ShengCNP_LM_bias19,gpt3}." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="2005.00268" id="86">
        <attvalues>
          <attvalue for="5" value=" \cite{babysitter2_Sheng2020} introduce a method relying on adversarial triggers~\cite{WallaceFKGS19}; this method does not de-bias the whole distribution but only obtains non-biased continuations of given prompts." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1904.03035" id="88">
        <attvalues>
          <attvalue for="5" value=" \cite{BordiaB19} introduce a regularization term for reducing gender bias when training a language model from scratch (as opposed to de-biasing a pretrained model)." />
        </attvalues>
      </edge>
      <edge source="2012.11635" target="1912.08517" id="89">
        <attvalues>
          <attvalue for="5" value=" By contrast --- similarly to \cite{A-parshakova-etal-2019-global,opt-rl-arxiv-2019} in a different context --- we clearly decouple the relatively simple problem of determining a ``pivot'' optimal EBM from the more difficult problem of exploiting this EBM at inference time, Such decoupling is valuable, because it permits to better diagnose the important challenges to focus on." />
        </attvalues>
      </edge>
      <edge source="1909.01326" target="1711.09050" id="91">
        <attvalues>
          <attvalue for="5" value=" Recent works in machine translation \cite{prates2018assessing} and dialogue systems \cite{henderson2018ethical} have brought to attention the perpetuation of biases in natural language generation (NLG) systems." />
        </attvalues>
      </edge>
      <edge source="1909.01326" target="1507.04808" id="92">
        <attvalues>
          <attvalue for="5" value=" Language models are a fundamental component of NLG that are widely used in downstream tasks such as machine translation \cite{koehn2009statistical}, dialogue generation \cite{serban2016building}, and story generation \cite{yao2019plan}; as such, biases propagated through the language models will have a profound impact on a variety of other NLG tasks." />
        </attvalues>
      </edge>
      <edge source="2401.13907" target="1805.01042" id="94">
        <attvalues>
          <attvalue for="5" value=" However, in recent studies, researchers discovered that sometimes the high performance on benchmark data sets are not a results of the above, instead, they are from learning the spurious correlations between tokens and output label \cite{poliak2018hypothesis}." />
        </attvalues>
      </edge>
      <edge source="2401.13907" target="2104.08646" id="95">
        <attvalues>
          <attvalue for="5" value=" To solve this issue, the most popular way is manually or semi-automatically edit the records \cite{gardner2021competency,clark-etal-2019-dont,maas-etal-2011-learning}, such that the p(label$|$token) is approximately uniform over all class labels." />
        </attvalues>
      </edge>
      <edge source="2401.13907" target="1508.05326" id="96">
        <attvalues>
          <attvalue for="5" value=" In this study, we will show that such spurious correlations exist in SNLI data \cite{bowman2015large}, in the form that tokens from a specific subset occurs more often with some labels than with other labels." />
        </attvalues>
      </edge>
      <edge source="2401.13907" target="2003.10555" id="97">
        <attvalues>
          <attvalue for="5" value=" We trained a ELECTRA-small \cite{clark2020electra} model on SNLI data without any correction and observed that these spurious correlations have influenced model training." />
        </attvalues>
      </edge>
      <edge source="2401.13907" target="1909.03553" id="99">
        <attvalues>
          <attvalue for="5" value=" However, this method will cost significant amount of human time and may introduce additional bias in these augmented data \cite{tafjord-etal-2019-quartz}." />
        </attvalues>
      </edge>
      <edge source="2104.08646" target="1808.09384" id="101">
        <attvalues>
          <attvalue for="5" value=" Attempts by the natural language processing community to get machines to understand language or read text are often stymied in part by issues in our datasets~\cite{chen-etal-2016-thorough,sugawara-etal-2018-makes}." />
        </attvalues>
      </edge>
      <edge source="2104.08646" target="1508.05326" id="104">
        <attvalues>
          <attvalue for="5" value=" Figure~\ref{fig:snli-artifacts} revisits prior analyses on the SNLI dataset~\cite{bowman-etal-2015-large} with our statistical test." />
        </attvalues>
      </edge>
      <edge source="2104.08646" target="1803.02324" id="105">
        <attvalues>
          <attvalue for="5" value=" An analysis based on pointwise mutual information~\cite[eg,][]{gururangan-etal-2018-annotation} would correspond to a horizontal line in that figure, missing many features that have less extreme but still significant correlations with class labels." />
        </attvalues>
      </edge>
      <edge source="2104.08646" target="2105.10386" id="106">
        <attvalues>
          <attvalue for="5" value=" Our analysis of local edits additionally suggests a strong relationship to sensitivity in boolean functions \cite{o2014analysis}, and we identify gaps in the theory of sensitivity that need to be filled to properly account for bias in sampled datasets." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2005.14165" id="107">
        <attvalues>
          <attvalue for="5" value=" Over the past several years, large transformer models have established themselves as the premier methodology for generative tasks in natural language processing \cite{brown2020language,sanh2021multitask,chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2110.08207" id="108">
        <attvalues>
          <attvalue for="5" value=" Over the past several years, large transformer models have established themselves as the premier methodology for generative tasks in natural language processing \cite{brown2020language,sanh2021multitask,chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2204.02311" id="109">
        <attvalues>
          <attvalue for="5" value=" Over the past several years, large transformer models have established themselves as the premier methodology for generative tasks in natural language processing \cite{brown2020language,sanh2021multitask,chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2204.06125" id="110">
        <attvalues>
          <attvalue for="5" value=" Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis \cite{ramesh2022hierarchical,crowson2022vqgan,rombach2022high}, protein modeling \cite{jumper2021highly,ahdritz2022openfold}, and computer programming \cite{chen2021evaluating,xu2022systematic,fried2022incoder}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2204.08583" id="111">
        <attvalues>
          <attvalue for="5" value=" Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis \cite{ramesh2022hierarchical,crowson2022vqgan,rombach2022high}, protein modeling \cite{jumper2021highly,ahdritz2022openfold}, and computer programming \cite{chen2021evaluating,xu2022systematic,fried2022incoder}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2112.10752" id="112">
        <attvalues>
          <attvalue for="5" value=" Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis \cite{ramesh2022hierarchical,crowson2022vqgan,rombach2022high}, protein modeling \cite{jumper2021highly,ahdritz2022openfold}, and computer programming \cite{chen2021evaluating,xu2022systematic,fried2022incoder}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2107.03374" id="113">
        <attvalues>
          <attvalue for="5" value=" Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis \cite{ramesh2022hierarchical,crowson2022vqgan,rombach2022high}, protein modeling \cite{jumper2021highly,ahdritz2022openfold}, and computer programming \cite{chen2021evaluating,xu2022systematic,fried2022incoder}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2202.13169" id="114">
        <attvalues>
          <attvalue for="5" value=" Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis \cite{ramesh2022hierarchical,crowson2022vqgan,rombach2022high}, protein modeling \cite{jumper2021highly,ahdritz2022openfold}, and computer programming \cite{chen2021evaluating,xu2022systematic,fried2022incoder}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2204.05999" id="115">
        <attvalues>
          <attvalue for="5" value=" Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis \cite{ramesh2022hierarchical,crowson2022vqgan,rombach2022high}, protein modeling \cite{jumper2021highly,ahdritz2022openfold}, and computer programming \cite{chen2021evaluating,xu2022systematic,fried2022incoder}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2001.08361" id="116">
        <attvalues>
          <attvalue for="5" value=" It is well established that there are regular and predictable patterns in the behavior of trained language models as they scale \cite{kaplan2020scaling,henighan2020scaling,hernandez2021scaling,mikami2021scaling,pu2021scaling,sharma2020neural,ghorbani2021scaling}, but prior work connecting these ``Scaling Laws'' to the learning dynamics of language models is minimal." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2010.14701" id="117">
        <attvalues>
          <attvalue for="5" value=" It is well established that there are regular and predictable patterns in the behavior of trained language models as they scale \cite{kaplan2020scaling,henighan2020scaling,hernandez2021scaling,mikami2021scaling,pu2021scaling,sharma2020neural,ghorbani2021scaling}, but prior work connecting these ``Scaling Laws'' to the learning dynamics of language models is minimal." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2102.01293" id="118">
        <attvalues>
          <attvalue for="5" value=" It is well established that there are regular and predictable patterns in the behavior of trained language models as they scale \cite{kaplan2020scaling,henighan2020scaling,hernandez2021scaling,mikami2021scaling,pu2021scaling,sharma2020neural,ghorbani2021scaling}, but prior work connecting these ``Scaling Laws'' to the learning dynamics of language models is minimal." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2108.11018" id="119">
        <attvalues>
          <attvalue for="5" value=" It is well established that there are regular and predictable patterns in the behavior of trained language models as they scale \cite{kaplan2020scaling,henighan2020scaling,hernandez2021scaling,mikami2021scaling,pu2021scaling,sharma2020neural,ghorbani2021scaling}, but prior work connecting these ``Scaling Laws'' to the learning dynamics of language models is minimal." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2004.10802" id="120">
        <attvalues>
          <attvalue for="5" value=" It is well established that there are regular and predictable patterns in the behavior of trained language models as they scale \cite{kaplan2020scaling,henighan2020scaling,hernandez2021scaling,mikami2021scaling,pu2021scaling,sharma2020neural,ghorbani2021scaling}, but prior work connecting these ``Scaling Laws'' to the learning dynamics of language models is minimal." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2109.07740" id="121">
        <attvalues>
          <attvalue for="5" value=" It is well established that there are regular and predictable patterns in the behavior of trained language models as they scale \cite{kaplan2020scaling,henighan2020scaling,hernandez2021scaling,mikami2021scaling,pu2021scaling,sharma2020neural,ghorbani2021scaling}, but prior work connecting these ``Scaling Laws'' to the learning dynamics of language models is minimal." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2111.09259" id="122">
        <attvalues>
          <attvalue for="5" value=" Of the research along these lines that does exist \cite{mcgrath2021acquisition,tirumala2022memorization,xia2022training}, it is overwhelmingly done on non-public models or model checkpoints, further emphasizing the importance of having publicly available model suites for scientific research." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2205.10770" id="123">
        <attvalues>
          <attvalue for="5" value=" Of the research along these lines that does exist \cite{mcgrath2021acquisition,tirumala2022memorization,xia2022training}, it is overwhelmingly done on non-public models or model checkpoints, further emphasizing the importance of having publicly available model suites for scientific research." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2212.09803" id="124">
        <attvalues>
          <attvalue for="5" value=" Of the research along these lines that does exist \cite{mcgrath2021acquisition,tirumala2022memorization,xia2022training}, it is overwhelmingly done on non-public models or model checkpoints, further emphasizing the importance of having publicly available model suites for scientific research." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2101.00027" id="125">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} We train 8 model sizes each on both the Pile \cite{gao2020pile,biderman2022datasheet} and the Pile after deduplication, providing 2 copies of the suite which can be compared." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2209.03661" id="127">
        <attvalues>
          <attvalue for="5" value=" However, while some work has explored finetuning's effects on bias in language models \cite{gira2022debiasing,kirtane2022efficient,choenni2021stepmothers}, or the relationship between the corpus statistics and the measured bias \cite{bordia2019identifying,wal2022birth}, researchers have generally lacked the tools to study the role of the training data on the learning dynamics of bias in large language models of different sizes." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2109.10052" id="128">
        <attvalues>
          <attvalue for="5" value=" However, while some work has explored finetuning's effects on bias in language models \cite{gira2022debiasing,kirtane2022efficient,choenni2021stepmothers}, or the relationship between the corpus statistics and the measured bias \cite{bordia2019identifying,wal2022birth}, researchers have generally lacked the tools to study the role of the training data on the learning dynamics of bias in large language models of different sizes." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="1904.03035" id="129">
        <attvalues>
          <attvalue for="5" value=" However, while some work has explored finetuning's effects on bias in language models \cite{gira2022debiasing,kirtane2022efficient,choenni2021stepmothers}, or the relationship between the corpus statistics and the measured bias \cite{bordia2019identifying,wal2022birth}, researchers have generally lacked the tools to study the role of the training data on the learning dynamics of bias in large language models of different sizes." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2207.10245" id="130">
        <attvalues>
          <attvalue for="5" value=" However, while some work has explored finetuning's effects on bias in language models \cite{gira2022debiasing,kirtane2022efficient,choenni2021stepmothers}, or the relationship between the corpus statistics and the measured bias \cite{bordia2019identifying,wal2022birth}, researchers have generally lacked the tools to study the role of the training data on the learning dynamics of bias in large language models of different sizes." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="1802.08232" id="131">
        <attvalues>
          <attvalue for="5" value=" Memorization is a Poisson Point Process Building on the extensive literature on memorization in large language models \cite{carlini2019secret,carlini2021extracting,hu2022membership}, we ask the following question: does the location of a particular sequence in the training dataset influence the likelihood of it being memorized? Leveraging Pythia's reproducible dataloader setup we answer this question in the negative, and furthermore find that a poisson point process is a very good model for the occurrence of memorized sequences over the course of training." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2012.07805" id="132">
        <attvalues>
          <attvalue for="5" value=" Memorization is a Poisson Point Process Building on the extensive literature on memorization in large language models \cite{carlini2019secret,carlini2021extracting,hu2022membership}, we ask the following question: does the location of a particular sequence in the training dataset influence the likelihood of it being memorized? Leveraging Pythia's reproducible dataloader setup we answer this question in the negative, and furthermore find that a poisson point process is a very good model for the occurrence of memorized sequences over the course of training." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2103.07853" id="133">
        <attvalues>
          <attvalue for="5" value=" Memorization is a Poisson Point Process Building on the extensive literature on memorization in large language models \cite{carlini2019secret,carlini2021extracting,hu2022membership}, we ask the following question: does the location of a particular sequence in the training dataset influence the likelihood of it being memorized? Leveraging Pythia's reproducible dataloader setup we answer this question in the negative, and furthermore find that a poisson point process is a very good model for the occurrence of memorized sequences over the course of training." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2202.07206" id="134">
        <attvalues>
          <attvalue for="5" value=" Emergence of the Impact of Pretraining Frequencies Recent work has identified the frequency of specific facts within a corpus as an important factor in how likely a model is capable of applying that fact in response to a natural language question \cite{razeghi2022impact,elazar2022measuring,kandpal2022large,mallen2022parametric}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2211.08411" id="135">
        <attvalues>
          <attvalue for="5" value=" Emergence of the Impact of Pretraining Frequencies Recent work has identified the frequency of specific facts within a corpus as an important factor in how likely a model is capable of applying that fact in response to a natural language question \cite{razeghi2022impact,elazar2022measuring,kandpal2022large,mallen2022parametric}." />
        </attvalues>
      </edge>
      <edge source="2304.01373" target="2211.05100" id="136">
        <attvalues>
          <attvalue for="5" value=" Existing work has been heavily dependent on the handful of models trained on public data, such as GPT-J \cite{gpt-j} and BLOOM \cite{scao2022bloom}, which lack frequent intermediate checkpoints, so none of these papers are able to look at the fine-grained evolution of this phenomenon over the course of training." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="1909.12673" id="137">
        <attvalues>
          <attvalue for="5" value=" For many of these tasks the scaling behavior of neural networks is highly predictable; model fit or test loss can be described precisely as a function of its number of parameters \cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,hernandez2021scaling,rosenfeld2019constructive}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="1901.01608" id="138">
        <attvalues>
          <attvalue for="5" value=" Along with the practical success of modern neural networks at scale, theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing \cite{ advani2017highdimensional,rosenfeld2019constructive,Geiger_2020,ghorbani2020linearized,chobernoulli2020,DBLP:journals/corr/abs-2102-04074,bahri2021explaining,loureiro2021capturing}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="1904.12191" id="139">
        <attvalues>
          <attvalue for="5" value=" Along with the practical success of modern neural networks at scale, theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing \cite{ advani2017highdimensional,rosenfeld2019constructive,Geiger_2020,ghorbani2020linearized,chobernoulli2020,DBLP:journals/corr/abs-2102-04074,bahri2021explaining,loureiro2021capturing}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="2102.04074" id="140">
        <attvalues>
          <attvalue for="5" value=" Along with the practical success of modern neural networks at scale, theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing \cite{ advani2017highdimensional,rosenfeld2019constructive,Geiger_2020,ghorbani2020linearized,chobernoulli2020,DBLP:journals/corr/abs-2102-04074,bahri2021explaining,loureiro2021capturing}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="2102.06701" id="141">
        <attvalues>
          <attvalue for="5" value=" Along with the practical success of modern neural networks at scale, theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing \cite{ advani2017highdimensional,rosenfeld2019constructive,Geiger_2020,ghorbani2020linearized,chobernoulli2020,DBLP:journals/corr/abs-2102-04074,bahri2021explaining,loureiro2021capturing}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="1712.00409" id="142">
        <attvalues>
          <attvalue for="5" value=" For many of these tasks the scaling behavior of neural networks is highly predictable; model fit or test loss can be described precisely as a function of its number of parameters \cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,hernandez2021scaling,rosenfeld2019constructive}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="2001.08361" id="143">
        <attvalues>
          <attvalue for="5" value=" For many of these tasks the scaling behavior of neural networks is highly predictable; model fit or test loss can be described precisely as a function of its number of parameters \cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,hernandez2021scaling,rosenfeld2019constructive}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="2010.14701" id="144">
        <attvalues>
          <attvalue for="5" value=" For many of these tasks the scaling behavior of neural networks is highly predictable; model fit or test loss can be described precisely as a function of its number of parameters \cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,hernandez2021scaling,rosenfeld2019constructive}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="2102.01293" id="145">
        <attvalues>
          <attvalue for="5" value=" For many of these tasks the scaling behavior of neural networks is highly predictable; model fit or test loss can be described precisely as a function of its number of parameters \cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,hernandez2021scaling,rosenfeld2019constructive}." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="1811.06965" id="146">
        <attvalues>
          <attvalue for="5" value=" Neural machine translation (NMT) has long enjoyed the benefits of scaling \cite{huang2019gpipe,DBLP:journals/corr/abs-1907-05019,DBLP:journals/corr/abs-2006-16668}, but studies investigating the scaling behavior of NMT models are missing." />
        </attvalues>
      </edge>
      <edge source="2109.07740" target="1706.03762" id="148">
        <attvalues>
          <attvalue for="5" value=" We present the first large-scale systematic study of scaling laws for encoder-decoder Transformer models applied to NMT \cite{DBLP:journals/corr/VaswaniSPUJGKP17}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1802.05365" id="150">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) are at the heart of natural language processing, especially following their recent success in the pretraining paradigm \cite[inter alia]{dai15,peters-etal-2018-deep,devlin18,radford2019language}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1910.07181" id="153">
        <attvalues>
          <attvalue for="5" value=" This makes large pretrained language models struggle with rare words, despite being able to produce contextualized representations for them \cite{schick-schutze-2020-bertram}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1508.07909" id="154">
        <attvalues>
          <attvalue for="5" value=" One way to ``open up'' the vocabulary is to model sequences of bytes, characters, or ``wordpieces'' rather than the conventional word tokens \cite{sennrich15,Radford2018ImprovingLU,ponti-etal-2019-towards}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="2108.03334" id="155">
        <attvalues>
          <attvalue for="5" value=" One way to ``open up'' the vocabulary is to model sequences of bytes, characters, or ``wordpieces'' rather than the conventional word tokens \cite{sennrich15,Radford2018ImprovingLU,ponti-etal-2019-towards}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1808.09943" id="156">
        <attvalues>
          <attvalue for="5" value=" These models appear to require greater network depth and show slower convergence than word-based alternatives \cite{cherry-etal-2018-revisiting,rami19}; the extra work comes at a cost." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1809.10853" id="158">
        <attvalues>
          <attvalue for="5" value=" On standard language modeling (Section \ref{sec:closed_lm}) we observe that our model has superior perplexity and is more sample efficient than a variety of existing output embedding approaches, including the recent adaptive embedding of \cite{baevski2018adaptive}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1911.00172" id="160">
        <attvalues>
          <attvalue for="5" value=" This is one of the reasons why the area of word-level language modeling is still very active \cite{baevski2018adaptive,sukhbaatar-etal-2019-adaptive,Khandelwal2020Generalization,press-etal-2020-improving}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1804.08205" id="162">
        <attvalues>
          <attvalue for="5" value=" Interpolations between word- and character- or morphology-based LMs represent another class of solutions \cite{mielke18,gerz-etal-2018-language,Ataman2020A}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1910.13890" id="163">
        <attvalues>
          <attvalue for="5" value=" Interpolations between word- and character- or morphology-based LMs represent another class of solutions \cite{mielke18,gerz-etal-2018-language,Ataman2020A}." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1609.07843" id="164">
        <attvalues>
          <attvalue for="5" value=" Notable for enabling adaptability are interpolated LMs based on copy mechanisms \cite{merity2016pointer}, dynamic evaluation \cite{krause2017dynamic}, and neural caches \cite{grave2016improving,Grave17}; the last provides state-of-the-art adaptation performance and, unlike the rest, it does not require additional training." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1612.04426" id="166">
        <attvalues>
          <attvalue for="5" value=" Notable for enabling adaptability are interpolated LMs based on copy mechanisms \cite{merity2016pointer}, dynamic evaluation \cite{krause2017dynamic}, and neural caches \cite{grave2016improving,Grave17}; the last provides state-of-the-art adaptation performance and, unlike the rest, it does not require additional training." />
        </attvalues>
      </edge>
      <edge source="2009.11523" target="1711.02604" id="167">
        <attvalues>
          <attvalue for="5" value=" We find that GroC also outperforms strong interpolated baselines, including the unbounded neural cache model of \cite{Grave17} on ``near'' domains and performs competitively on ``far'' domains." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="1907.11692" id="169">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (LMs) powered by finetuning have achieved remarkable performance on a wide range of downstream tasks~\cite{DBLP:conf/naacl/DevlinCLT19,DBLP:journals/corr/abs-1907-11692,Radford19}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="1909.10351" id="171">
        <attvalues>
          <attvalue for="5" value=" Under a teacher-student regime, task-agnostic distillation distils pretrained LMs into ones of small compute on pretraining data so that these small LMs can be applied to tasks by finetuning~\cite{DBLP:conf/emnlp/JiaoYSJCL0L20,DBLP:conf/nips/WangW0B0020,DBLP:journals/corr/abs-2302-09632}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="2002.10957" id="172">
        <attvalues>
          <attvalue for="5" value=" Under a teacher-student regime, task-agnostic distillation distils pretrained LMs into ones of small compute on pretraining data so that these small LMs can be applied to tasks by finetuning~\cite{DBLP:conf/emnlp/JiaoYSJCL0L20,DBLP:conf/nips/WangW0B0020,DBLP:journals/corr/abs-2302-09632}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="2302.09632" id="173">
        <attvalues>
          <attvalue for="5" value=" Under a teacher-student regime, task-agnostic distillation distils pretrained LMs into ones of small compute on pretraining data so that these small LMs can be applied to tasks by finetuning~\cite{DBLP:conf/emnlp/JiaoYSJCL0L20,DBLP:conf/nips/WangW0B0020,DBLP:journals/corr/abs-2302-09632}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="1503.02531" id="174">
        <attvalues>
          <attvalue for="5" value=" In contrast, task-specific distillation distils finetuned LMs on finetuning data and consumed resource can be even huge when the number of tasks explode~\cite{DBLP:journals/corr/HintonVD15,DBLP:conf/emnlp/SunCGL19,DBLP:conf/acl/XiaZC22,DBLP:conf/emnlp/YangZS22}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="1908.09355" id="175">
        <attvalues>
          <attvalue for="5" value=" In contrast, task-specific distillation distils finetuned LMs on finetuning data and consumed resource can be even huge when the number of tasks explode~\cite{DBLP:journals/corr/HintonVD15,DBLP:conf/emnlp/SunCGL19,DBLP:conf/acl/XiaZC22,DBLP:conf/emnlp/YangZS22}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="2204.00408" id="176">
        <attvalues>
          <attvalue for="5" value=" In contrast, task-specific distillation distils finetuned LMs on finetuning data and consumed resource can be even huge when the number of tasks explode~\cite{DBLP:journals/corr/HintonVD15,DBLP:conf/emnlp/SunCGL19,DBLP:conf/acl/XiaZC22,DBLP:conf/emnlp/YangZS22}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="2210.03923" id="177">
        <attvalues>
          <attvalue for="5" value=" In contrast, task-specific distillation distils finetuned LMs on finetuning data and consumed resource can be even huge when the number of tasks explode~\cite{DBLP:journals/corr/HintonVD15,DBLP:conf/emnlp/SunCGL19,DBLP:conf/acl/XiaZC22,DBLP:conf/emnlp/YangZS22}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="2205.14570" id="178">
        <attvalues>
          <attvalue for="5" value=" Additionally, it is acknowledged that task-agnostic distillation typically brings performance gain over task-specific distillation does~\cite{DBLP:journals/corr/abs-2205-14570}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="2010.13002" id="180">
        <attvalues>
          <attvalue for="5" value=" Despite so many merits, prior studies mostly lie in distillation of either encoder-only LMs~\cite[eg, BERT,][]{DBLP:conf/naacl/DevlinCLT19} or decoder-only LMs~\cite[eg, GPT,][]{Radford19} and largely ignore the signifance of task-agnostic distillation of encoder-decoder LMs~\cite[eg, T5,][]{DBLP:journals/jmlr/RaffelSRLNMZLL20} given recent advances in task-specific distillation of encoder-decoder LMs though~\cite{DBLP:journals/corr/abs-2010-13002,DBLP:conf/acl/ZhangZBW22,DBLP:conf/acl/LiWTNBAXR22,DBLP:conf/acl/TaoHZSJLLW22}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="2106.03441" id="181">
        <attvalues>
          <attvalue for="5" value=" Despite so many merits, prior studies mostly lie in distillation of either encoder-only LMs~\cite[eg, BERT,][]{DBLP:conf/naacl/DevlinCLT19} or decoder-only LMs~\cite[eg, GPT,][]{Radford19} and largely ignore the signifance of task-agnostic distillation of encoder-decoder LMs~\cite[eg, T5,][]{DBLP:journals/jmlr/RaffelSRLNMZLL20} given recent advances in task-specific distillation of encoder-decoder LMs though~\cite{DBLP:journals/corr/abs-2010-13002,DBLP:conf/acl/ZhangZBW22,DBLP:conf/acl/LiWTNBAXR22,DBLP:conf/acl/TaoHZSJLLW22}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="2203.10705" id="183">
        <attvalues>
          <attvalue for="5" value=" Despite so many merits, prior studies mostly lie in distillation of either encoder-only LMs~\cite[eg, BERT,][]{DBLP:conf/naacl/DevlinCLT19} or decoder-only LMs~\cite[eg, GPT,][]{Radford19} and largely ignore the signifance of task-agnostic distillation of encoder-decoder LMs~\cite[eg, T5,][]{DBLP:journals/jmlr/RaffelSRLNMZLL20} given recent advances in task-specific distillation of encoder-decoder LMs though~\cite{DBLP:journals/corr/abs-2010-13002,DBLP:conf/acl/ZhangZBW22,DBLP:conf/acl/LiWTNBAXR22,DBLP:conf/acl/TaoHZSJLLW22}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="1706.03762" id="184">
        <attvalues>
          <attvalue for="5" value=" Frustratingly, we find that existing distillation methods may fail to handle task-agnostic distillation of encoder-decoder LMs since encoder-decoder LMs can behave very differently in comparison with encoder-only and decoder-only LMs~\cite[eg, the use of cross-attention,][]{DBLP:conf/nips/VaswaniSPUJGKP17}." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="1704.04368" id="186">
        <attvalues>
          <attvalue for="5" value=" For evaluation on abstractive summarization, we adopt CNN/DailyMail~\cite{DBLP:conf/acl/SeeLM17} and XSum~\cite{DBLP:conf/emnlp/NarayanCL18} as two testbeds." />
        </attvalues>
      </edge>
      <edge source="2305.12330" target="1808.08745" id="187">
        <attvalues>
          <attvalue for="5" value=" For evaluation on abstractive summarization, we adopt CNN/DailyMail~\cite{DBLP:conf/acl/SeeLM17} and XSum~\cite{DBLP:conf/emnlp/NarayanCL18} as two testbeds." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1606.05250" id="190">
        <attvalues>
          <attvalue for="5" value="0 \cite{rajpurkar2016squad,rajpurkar2018know} at multiple parameter scales (eg, $65$M and $10\sim20$M)." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="2005.14165" id="193">
        <attvalues>
          <attvalue for="5" value=" However, the number of parameters in such models has grown over hundreds of millions \cite{devlin2018bert,raffel2019exploring,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1503.02531" id="194">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation \cite{hinton2015distilling} has been shown to be a powerful technique to compress a large model (eg, teacher model) into a small one (eg, student model) with acceptable performance degradation." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1903.12136" id="195">
        <attvalues>
          <attvalue for="5" value=" In language models, many efforts have been devoted to task-specific knowledge distillation \cite{tang2019distilling,turc2019well,sun2019patient,aguilar2020knowledge}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1908.08962" id="196">
        <attvalues>
          <attvalue for="5" value=" In language models, many efforts have been devoted to task-specific knowledge distillation \cite{tang2019distilling,turc2019well,sun2019patient,aguilar2020knowledge}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1908.09355" id="197">
        <attvalues>
          <attvalue for="5" value=" In language models, many efforts have been devoted to task-specific knowledge distillation \cite{tang2019distilling,turc2019well,sun2019patient,aguilar2020knowledge}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1910.03723" id="198">
        <attvalues>
          <attvalue for="5" value=" In language models, many efforts have been devoted to task-specific knowledge distillation \cite{tang2019distilling,turc2019well,sun2019patient,aguilar2020knowledge}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="2004.02984" id="200">
        <attvalues>
          <attvalue for="5" value=" Therefore, recent research has started to pay more attention to task-agnostic distillation \cite{sanh2019distilbert,sun2020mobilebert,jiao2019tinybert,wang2020minilm,khanuja2021mergedistill,chen2021extract}, where a student is distilled from a teacher pre-trained on open-domain data and can be efficiently fine-tuned on various downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1909.10351" id="201">
        <attvalues>
          <attvalue for="5" value=" To reduce this discrepancy, recent research has proposed to better initialize the student model from a subset of the teacher's layers \cite{sanh2019distilbert,jiao2019tinybert,wang2020minilm}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="2002.10957" id="202">
        <attvalues>
          <attvalue for="5" value=" To reduce this discrepancy, recent research has proposed to better initialize the student model from a subset of the teacher's layers \cite{sanh2019distilbert,jiao2019tinybert,wang2020minilm}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="2106.02834" id="203">
        <attvalues>
          <attvalue for="5" value=" Therefore, recent research has started to pay more attention to task-agnostic distillation \cite{sanh2019distilbert,sun2020mobilebert,jiao2019tinybert,wang2020minilm,khanuja2021mergedistill,chen2021extract}, where a student is distilled from a teacher pre-trained on open-domain data and can be efficiently fine-tuned on various downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="2104.11928" id="204">
        <attvalues>
          <attvalue for="5" value=" Therefore, recent research has started to pay more attention to task-agnostic distillation \cite{sanh2019distilbert,sun2020mobilebert,jiao2019tinybert,wang2020minilm,khanuja2021mergedistill,chen2021extract}, where a student is distilled from a teacher pre-trained on open-domain data and can be efficiently fine-tuned on various downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1904.09149" id="205">
        <attvalues>
          <attvalue for="5" value=" Such a large prediction discrepancy eventually diminishes the benefits of distillation \cite{jin2019knowledge,cho2019efficacy,mirzadeh2020improved, guo2020reducing,li2021dynamic}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1910.01348" id="206">
        <attvalues>
          <attvalue for="5" value=" Such a large prediction discrepancy eventually diminishes the benefits of distillation \cite{jin2019knowledge,cho2019efficacy,mirzadeh2020improved, guo2020reducing,li2021dynamic}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1902.03393" id="207">
        <attvalues>
          <attvalue for="5" value=" Such a large prediction discrepancy eventually diminishes the benefits of distillation \cite{jin2019knowledge,cho2019efficacy,mirzadeh2020improved, guo2020reducing,li2021dynamic}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="2109.11295" id="208">
        <attvalues>
          <attvalue for="5" value=" Such a large prediction discrepancy eventually diminishes the benefits of distillation \cite{jin2019knowledge,cho2019efficacy,mirzadeh2020improved, guo2020reducing,li2021dynamic}." />
        </attvalues>
      </edge>
      <edge source="2302.09632" target="1806.03822" id="209">
        <attvalues>
          <attvalue for="5" value="0 \cite{rajpurkar2016squad,rajpurkar2018know} at multiple parameter scales (eg, $65$M and $10\sim20$M)." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="2305.14788" id="227">
        <attvalues>
          <attvalue for="5" value=" Furthermore, long-context capabilities can enhance in-context learning by providing more in-context examples, instructions to follow, or example trajectories in context of reinforcement learning~\cite{chevalier-etal-2023-adapting, agarwal2024many, lee2024supervised}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="2308.14508" id="228">
        <attvalues>
          <attvalue for="5" value=" For example, current benchmarks such as Longbench~\cite{bai2023longbench} and L-Eval~\cite{l_eval_an2023} scale only up to 40,000 tokens, while models are capable of hundreds of thousands and millions of tokens~\cite{rodkin2024associative, reid2024gemini, bulatov2023scaling, anthropic2024claude3, liu2024world, gu2023mamba, openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="2307.11088" id="229">
        <attvalues>
          <attvalue for="5" value=" For example, current benchmarks such as Longbench~\cite{bai2023longbench} and L-Eval~\cite{l_eval_an2023} scale only up to 40,000 tokens, while models are capable of hundreds of thousands and millions of tokens~\cite{rodkin2024associative, reid2024gemini, bulatov2023scaling, anthropic2024claude3, liu2024world, gu2023mamba, openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="2407.04841" id="230">
        <attvalues>
          <attvalue for="5" value=" For example, current benchmarks such as Longbench~\cite{bai2023longbench} and L-Eval~\cite{l_eval_an2023} scale only up to 40,000 tokens, while models are capable of hundreds of thousands and millions of tokens~\cite{rodkin2024associative, reid2024gemini, bulatov2023scaling, anthropic2024claude3, liu2024world, gu2023mamba, openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="2402.08268" id="231">
        <attvalues>
          <attvalue for="5" value=" However, the widespread use of this approach has highlighted its limitations - it is overly simplistic, and novel long context models often achieve perfect performance, as usually demonstrated by fully green heatmaps~\cite{reid2024gemini,commandr2024,liu2024world,wang2024xl3m}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="2403.11802" id="232">
        <attvalues>
          <attvalue for="5" value=" As a consequence, synthetic benchmarks focusing on variations of &quot;needle-in-a-haystack&quot; tasks have become increasingly common~\cite{zhang2024inftybench,liu2024world,song2024countingstars,hsieh2024ruler}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="2404.06654" id="233">
        <attvalues>
          <attvalue for="5" value=" As a consequence, synthetic benchmarks focusing on variations of &quot;needle-in-a-haystack&quot; tasks have become increasingly common~\cite{zhang2024inftybench,liu2024world,song2024countingstars,hsieh2024ruler}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="2405.17755" id="234">
        <attvalues>
          <attvalue for="5" value=" However, the widespread use of this approach has highlighted its limitations - it is overly simplistic, and novel long context models often achieve perfect performance, as usually demonstrated by fully green heatmaps~\cite{reid2024gemini,commandr2024,liu2024world,wang2024xl3m}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="1502.05698" id="235">
        <attvalues>
          <attvalue for="5" value=" BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets, that were designed as prerequisites for any system that aims to be capable of conversing with a human~\cite{WestonBCM15}." />
        </attvalues>
      </edge>
      <edge source="2406.10149" target="1911.05507" id="236">
        <attvalues>
          <attvalue for="5" value=" As a source of long natural documents we use books from PG19 corpora~\cite{rae2019compressive}." />
        </attvalues>
      </edge>
      <edge source="2106.10715" target="2010.11929" id="239">
        <attvalues>
          <attvalue for="5" value=" Recently, pre-training has become the mainstream technique to develop large-scale neural networks and achieved great success in both computer vision (CV) and natural language processing (NLP)~\cite{ResNet,ViT,BERT}." />
        </attvalues>
      </edge>
      <edge source="2106.10715" target="2005.14165" id="241">
        <attvalues>
          <attvalue for="5" value=" Especially, there are some much larger pre-trained language models (PLMs) with hundreds of billions of parameters, such as GPT-3~\cite{brown2020language}, PANGU-$\alpha$~\cite{zeng2021pangu}, and Switch-Transformer~\cite{fedus2021switch}." />
        </attvalues>
      </edge>
      <edge source="2106.10715" target="2104.12369" id="242">
        <attvalues>
          <attvalue for="5" value=" Especially, there are some much larger pre-trained language models (PLMs) with hundreds of billions of parameters, such as GPT-3~\cite{brown2020language}, PANGU-$\alpha$~\cite{zeng2021pangu}, and Switch-Transformer~\cite{fedus2021switch}." />
        </attvalues>
      </edge>
      <edge source="2106.10715" target="2101.03961" id="243">
        <attvalues>
          <attvalue for="5" value=" Especially, there are some much larger pre-trained language models (PLMs) with hundreds of billions of parameters, such as GPT-3~\cite{brown2020language}, PANGU-$\alpha$~\cite{zeng2021pangu}, and Switch-Transformer~\cite{fedus2021switch}." />
        </attvalues>
      </edge>
      <edge source="2106.10715" target="2105.13880" id="244">
        <attvalues>
          <attvalue for="5" value=" To reduce the cost of large-scale PLMs from its pre-training to fine-tuning, we try to improve the whole pipeline of developing PLMs as follows: (1) We adopt knowledge inheritance~\cite{qin2021ki} to accelerate the pre-training process." />
        </attvalues>
      </edge>
      <edge source="2106.10715" target="2104.08691" id="245">
        <attvalues>
          <attvalue for="5" value=" (2) We use prompt tuning~\cite{prompt-tuning} instead of fine-tuning to reduce the storage of task-specific parameters." />
        </attvalues>
      </edge>
      <edge source="2106.10715" target="2010.11934" id="246">
        <attvalues>
          <attvalue for="5" value=" Then, we compare CPM-2 with mT5~\cite{xue2020mt5}." />
        </attvalues>
      </edge>
      <edge source="2402.01364" target="2005.11401" id="247">
        <attvalues>
          <attvalue for="5" value=" Whilst continual learning bears some resemblance to other strategies for model improvements, such as retrieval-augmented generation (RAG) ~\cite{LewisPPPKGKLYR020} and model editing ~\cite{yao-etal-2023-editing}, their main purposes differ (\ref{tab:comp})." />
        </attvalues>
      </edge>
      <edge source="2402.01364" target="2305.13172" id="248">
        <attvalues>
          <attvalue for="5" value=" Whilst continual learning bears some resemblance to other strategies for model improvements, such as retrieval-augmented generation (RAG) ~\cite{LewisPPPKGKLYR020} and model editing ~\cite{yao-etal-2023-editing}, their main purposes differ (\ref{tab:comp})." />
        </attvalues>
      </edge>
      <edge source="2402.01364" target="2310.07343" id="249">
        <attvalues>
          <attvalue for="5" value=" This distinction is crucial as it shifts the focus from merely updating information to developing a model's ability to process and generate language in a more comprehensive and nuanced manner~\cite{ZhangFCNW23}." />
        </attvalues>
      </edge>
      <edge source="2402.01364" target="2110.08534" id="250">
        <attvalues>
          <attvalue for="5" value=" We categorise it into three different stages, eg\ continual pretraining to expand the model's fundamental understanding of language~\cite{JinZZ00WA022}, continual instruction tuning to improve the model's response to specific user commands~\cite{zhang2023citb}, and continual alignment to ensure the model's outputs adhere to values, ethical standards and societal norms~\cite{zhang2023copf}." />
        </attvalues>
      </edge>
      <edge source="2402.01364" target="2310.14510" id="251">
        <attvalues>
          <attvalue for="5" value=" We categorise it into three different stages, eg\ continual pretraining to expand the model's fundamental understanding of language~\cite{JinZZ00WA022}, continual instruction tuning to improve the model's response to specific user commands~\cite{zhang2023citb}, and continual alignment to ensure the model's outputs adhere to values, ethical standards and societal norms~\cite{zhang2023copf}." />
        </attvalues>
      </edge>
      <edge source="2402.01364" target="2012.09823" id="252">
        <attvalues>
          <attvalue for="5" value=" While previous surveys in the field are typically organized around various continual learning strategies~\cite{biesialska-etal-2020-continual}, ours is the first to specifically address continual learning in the context of LLMs." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2005.14165" id="253">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated a remarkable capacity for understanding and generating human-like text \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2303-08774,anil2023palm,DBLP:journals/corr/abs-2302-13971,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2303-18223}." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2303.08774" id="254">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated a remarkable capacity for understanding and generating human-like text \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2303-08774,anil2023palm,DBLP:journals/corr/abs-2302-13971,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2303-18223}." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2305.10403" id="255">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated a remarkable capacity for understanding and generating human-like text \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2303-08774,anil2023palm,DBLP:journals/corr/abs-2302-13971,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2303-18223}." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2212.09597" id="257">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated a remarkable capacity for understanding and generating human-like text \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2303-08774,anil2023palm,DBLP:journals/corr/abs-2302-13971,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2303-18223}." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2303.18223" id="258">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated a remarkable capacity for understanding and generating human-like text \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2303-08774,anil2023palm,DBLP:journals/corr/abs-2302-13971,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2303-18223}." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2004.00345" id="259">
        <attvalues>
          <attvalue for="5" value=" Currently, numerous works on model editing for LLMs \cite{de-cao-etal-2021-editing,meng2022locating,meng2023massediting,Sinitsin2020Editable,huang2023transformerpatcher} have made strides in various editing tasks and settings." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2104.08164" id="260">
        <attvalues>
          <attvalue for="5" value=" Currently, numerous works on model editing for LLMs \cite{de-cao-etal-2021-editing,meng2022locating,meng2023massediting,Sinitsin2020Editable,huang2023transformerpatcher} have made strides in various editing tasks and settings." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2202.05262" id="261">
        <attvalues>
          <attvalue for="5" value=" We initially use two popular model editing datasets, ZsRE ~\cite{levy-etal-2017-zero} and \cf~\cite{meng2022locating}, and two structurally different language models, T5 \cite{DBLP:journals/jmlr/RaffelSRLNMZLL20} (encoder-decoder) and GPT-J \cite{wang2021gpt} (decoder only), as our base models." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2210.07229" id="262">
        <attvalues>
          <attvalue for="5" value=" Currently, numerous works on model editing for LLMs \cite{de-cao-etal-2021-editing,meng2022locating,meng2023massediting,Sinitsin2020Editable,huang2023transformerpatcher} have made strides in various editing tasks and settings." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2301.09785" id="263">
        <attvalues>
          <attvalue for="5" value=" Currently, numerous works on model editing for LLMs \cite{de-cao-etal-2021-editing,meng2022locating,meng2023massediting,Sinitsin2020Editable,huang2023transformerpatcher} have made strides in various editing tasks and settings." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="1706.04115" id="264">
        <attvalues>
          <attvalue for="5" value=" We initially use two popular model editing datasets, ZsRE ~\cite{levy-etal-2017-zero} and \cf~\cite{meng2022locating}, and two structurally different language models, T5 \cite{DBLP:journals/jmlr/RaffelSRLNMZLL20} (encoder-decoder) and GPT-J \cite{wang2021gpt} (decoder only), as our base models." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2205.01068" id="266">
        <attvalues>
          <attvalue for="5" value=" We also evaluate the performance of larger models, OPT-13B~\cite{DBLP:journals/corr/abs-2205-01068} and GPT-NEOX-20B~\cite{black2022gptneox20b}." />
        </attvalues>
      </edge>
      <edge source="2305.13172" target="2204.06745" id="267">
        <attvalues>
          <attvalue for="5" value=" We also evaluate the performance of larger models, OPT-13B~\cite{DBLP:journals/corr/abs-2205-01068} and GPT-NEOX-20B~\cite{black2022gptneox20b}." />
        </attvalues>
      </edge>
      <edge source="2405.04434" target="1706.03762" id="269">
        <attvalues>
          <attvalue for="5" value=" (1) In the context of attention mechanisms, the Key-Value (KV) cache of the Multi-Head Attention (MHA)~\cite{transformer} poses a significant obstacle to the inference efficiency of LLMs." />
        </attvalues>
      </edge>
      <edge source="2405.04434" target="2305.13245" id="270">
        <attvalues>
          <attvalue for="5" value=" Various approaches have been explored to address this issue, including Grouped-Query Attention (GQA)~\cite{ainslie2023gqa} and Multi-Query Attention (MQA)~\cite{mqa}." />
        </attvalues>
      </edge>
      <edge source="2405.04434" target="1911.02150" id="271">
        <attvalues>
          <attvalue for="5" value=" Various approaches have been explored to address this issue, including Grouped-Query Attention (GQA)~\cite{ainslie2023gqa} and Multi-Query Attention (MQA)~\cite{mqa}." />
        </attvalues>
      </edge>
      <edge source="2405.04434" target="2401.06066" id="272">
        <attvalues>
          <attvalue for="5" value=" (2) For Feed-Forward Networks~(FFNs), we follow the \dsmoe architecture~\cite{deepseekmoe}, which adopts fine-grained expert segmentation and shared expert isolation for higher potential in expert specialization." />
        </attvalues>
      </edge>
      <edge source="2405.04434" target="2006.16668" id="273">
        <attvalues>
          <attvalue for="5" value=" The \dsmoe architecture demonstrates great advantages compared with conventional MoE architectures like GShard~\cite{gshard}, enabling us to train strong models at an economical cost." />
        </attvalues>
      </edge>
      <edge source="2405.04434" target="2401.02954" id="274">
        <attvalues>
          <attvalue for="5" value=" Compared with the corpus used in \dsvi~(our previous release)~\cite{deepseek1}, this corpus features an extended amount of data, especially Chinese data, and higher data quality." />
        </attvalues>
      </edge>
      <edge source="2405.04434" target="2402.03300" id="275">
        <attvalues>
          <attvalue for="5" value=" Finally, we follow DeepSeekMath~\cite{deepseekmath} to employ Group Relative Policy Optimization~(GRPO) to further align the model with human preference and produce \dsviirl." />
        </attvalues>
      </edge>
      <edge source="2405.04434" target="2306.05685" id="276">
        <attvalues>
          <attvalue for="5" value="97 overall score on MT-Bench \cite{mtbench}, and 7." />
        </attvalues>
      </edge>
      <edge source="2401.02954" target="1706.03762" id="278">
        <attvalues>
          <attvalue for="5" value=" Over the past few years, Large Language Models (LLMs) based on decoder-only Transformers \cite{transformer} have increasingly become the cornerstone and pathway to achieving Artificial General Intelligence (AGI)." />
        </attvalues>
      </edge>
      <edge source="2401.02954" target="2103.10360" id="279">
        <attvalues>
          <attvalue for="5" value=" These products have significantly raised the community's expectations for the capabilities of open-source LLMs, consequently inspiring a series of work~\cite{glm,llama,llama2,qwen,baichuan2,mistral}." />
        </attvalues>
      </edge>
      <edge source="2401.02954" target="2307.09288" id="281">
        <attvalues>
          <attvalue for="5" value=" These products have significantly raised the community's expectations for the capabilities of open-source LLMs, consequently inspiring a series of work~\cite{glm,llama,llama2,qwen,baichuan2,mistral}." />
        </attvalues>
      </edge>
      <edge source="2401.02954" target="2309.10305" id="283">
        <attvalues>
          <attvalue for="5" value=" These products have significantly raised the community's expectations for the capabilities of open-source LLMs, consequently inspiring a series of work~\cite{glm,llama,llama2,qwen,baichuan2,mistral}." />
        </attvalues>
      </edge>
      <edge source="2401.02954" target="2310.06825" id="284">
        <attvalues>
          <attvalue for="5" value=" These products have significantly raised the community's expectations for the capabilities of open-source LLMs, consequently inspiring a series of work~\cite{glm,llama,llama2,qwen,baichuan2,mistral}." />
        </attvalues>
      </edge>
      <edge source="2401.02954" target="2001.08361" id="285">
        <attvalues>
          <attvalue for="5" value=" In addition, early works~\cite{scalinglaw, chinchilla} reached varying conclusions on the scaling of model and data with increased compute budgets and inadequately addressed hyperparameter discussions." />
        </attvalues>
      </edge>
      <edge source="2401.02954" target="2203.15556" id="286">
        <attvalues>
          <attvalue for="5" value=" Following LLaMA, the open-source community has primarily focused on training fixed-size (7B, 13B, 34B, and 70B), high-quality models, often neglecting research exploration into LLM scaling laws \cite{scalinglaw,chinchilla}." />
        </attvalues>
      </edge>
      <edge source="2401.02954" target="2305.18290" id="288">
        <attvalues>
          <attvalue for="5" value=" Additionally, we have utilized direct preference optimization (DPO) \cite{dpo} to improve the conversational performance of the model." />
        </attvalues>
      </edge>
      <edge source="2403.09488" target="2005.14165" id="289">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated their effectiveness on a wide range of tasks through in-context learning (ICL), where models learn to perform a task from demonstrations \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2403.09488" target="2102.07350" id="290">
        <attvalues>
          <attvalue for="5" value=" \cite{reynolds2021prompt, min-etal-2022-rethinking, wei2023larger, pan-etal-2023-context}." />
        </attvalues>
      </edge>
      <edge source="2403.09488" target="2104.08315" id="291">
        <attvalues>
          <attvalue for="5" value=" To mitigate these instabilities, these studies introduced content-free tokens or utilized the entire test set to calibrate prediction probabilities \cite{holtzman-etal-2021-surface, fei2023mitigating, zhou2023batch}." />
        </attvalues>
      </edge>
      <edge source="2104.08315" target="2005.14165" id="292">
        <attvalues>
          <attvalue for="5" value=" Despite the impressive results large pretrained language models have achieved in zero-shot settings \cite{brown2020language, radford2019language}, we argue that current work underestimates the zero-shot capabilities of these models on classification tasks." />
        </attvalues>
      </edge>
      <edge source="2304.09960" target="2005.14165" id="294">
        <attvalues>
          <attvalue for="5" value=" Over the past few years, large language models (LLMs) have emerged as the predominant method for most natural language processing (NLP) tasks \cite{radford2018improving,raffel2020exploring,brown2020language,chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2304.09960" target="2204.02311" id="295">
        <attvalues>
          <attvalue for="5" value=" Over the past few years, large language models (LLMs) have emerged as the predominant method for most natural language processing (NLP) tasks \cite{radford2018improving,raffel2020exploring,brown2020language,chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2304.09960" target="2202.12837" id="297">
        <attvalues>
          <attvalue for="5" value=" Some empirical studies have suggested that the emergent abilities of LLMs may be linked to the label space and input data distributional properties \cite{min2022rethinking,chan2022data}, multitask prompted learning \cite{sanh2022multitask}, and pre-training term frequencies \cite{razeghi2022impact}." />
        </attvalues>
      </edge>
      <edge source="2304.09960" target="2205.05055" id="298">
        <attvalues>
          <attvalue for="5" value=" Some empirical studies have suggested that the emergent abilities of LLMs may be linked to the label space and input data distributional properties \cite{min2022rethinking,chan2022data}, multitask prompted learning \cite{sanh2022multitask}, and pre-training term frequencies \cite{razeghi2022impact}." />
        </attvalues>
      </edge>
      <edge source="2304.09960" target="2110.08207" id="299">
        <attvalues>
          <attvalue for="5" value=" Some empirical studies have suggested that the emergent abilities of LLMs may be linked to the label space and input data distributional properties \cite{min2022rethinking,chan2022data}, multitask prompted learning \cite{sanh2022multitask}, and pre-training term frequencies \cite{razeghi2022impact}." />
        </attvalues>
      </edge>
      <edge source="2304.09960" target="2202.07206" id="300">
        <attvalues>
          <attvalue for="5" value=" Some empirical studies have suggested that the emergent abilities of LLMs may be linked to the label space and input data distributional properties \cite{min2022rethinking,chan2022data}, multitask prompted learning \cite{sanh2022multitask}, and pre-training term frequencies \cite{razeghi2022impact}." />
        </attvalues>
      </edge>
      <edge source="2304.09960" target="2303.07895" id="302">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{wies2023learnability} established a PAC based framework for in-context learnability, and \cite{hahn2023theory} claimed that the emergent abilities arise through recombination of compositional structures in natural languages." />
        </attvalues>
      </edge>
      <edge source="2304.09960" target="2303.07971" id="303">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{wies2023learnability} established a PAC based framework for in-context learnability, and \cite{hahn2023theory} claimed that the emergent abilities arise through recombination of compositional structures in natural languages." />
        </attvalues>
      </edge>
      <edge source="2205.05055" target="2005.14165" id="304">
        <attvalues>
          <attvalue for="5" value=" Large transformer-based language models show an intriguing ability to perform in-context learning \cite{brown_language_2020}." />
        </attvalues>
      </edge>
      <edge source="2205.05055" target="1305.0543" id="306">
        <attvalues>
          <attvalue for="5" value=" That is, a given entity (word, person, object, etc) may have a distribution that is not uniform across time, instead tending to appear in clusters \cite{sarkar_bayesian_2005, alvarez-lacalle_hierarchical_2006, neuts_burstiness_2007, altmann_beyond_2009, serrano_modeling_2009,lambiotte_burstiness_2013}." />
        </attvalues>
      </edge>
      <edge source="2205.05055" target="1902.03477" id="307">
        <attvalues>
          <attvalue for="5" value=" We performed our experiments over data sequences sampled from a standard image-based few-shot dataset \cite[the Omniglot dataset;][]{lake_omniglot_2019}." />
        </attvalues>
      </edge>
      <edge source="2311.05020" target="1409.0473" id="313">
        <attvalues>
          <attvalue for="5" value=" Even in the modern era of deep learning, MT has been a locus of innovations that have fundamentally altered NLP and all of machine learning \cite{bahdanau2015neural,NeurIPS2017_3f5ee243,Sutskever2014SequenceTS}." />
        </attvalues>
      </edge>
      <edge source="2311.05020" target="1706.03762" id="314">
        <attvalues>
          <attvalue for="5" value=" Even in the modern era of deep learning, MT has been a locus of innovations that have fundamentally altered NLP and all of machine learning \cite{bahdanau2015neural,NeurIPS2017_3f5ee243,Sutskever2014SequenceTS}." />
        </attvalues>
      </edge>
      <edge source="2311.05020" target="1409.3215" id="315">
        <attvalues>
          <attvalue for="5" value=" Even in the modern era of deep learning, MT has been a locus of innovations that have fundamentally altered NLP and all of machine learning \cite{bahdanau2015neural,NeurIPS2017_3f5ee243,Sutskever2014SequenceTS}." />
        </attvalues>
      </edge>
      <edge source="2311.05020" target="2306.16900" id="316">
        <attvalues>
          <attvalue for="5" value=" The expense of state-of-the-art research has led many to question the role of smaller and publicly funded groups in AI \cite{lee2023surveying}, a phenomenon we will call the scale crisis." />
        </attvalues>
      </edge>
      <edge source="2311.05020" target="2304.06035" id="317">
        <attvalues>
          <attvalue for="5" value=" Researchers without direct access to LLMs have publicly fretted over their research directions, with \cite{togelius2023choose} suggesting pivots in research direction to sidestep scale, and \cite{ignat2023phd} sketching research areas that are ``not within the purview of LLMs." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1506.02516" id="318">
        <attvalues>
          <attvalue for="5" value=" This is in contrast to prior stack RNNs \cite{grefenstette+al:2015,joulin+mikolov:2015} which model deterministic stacks, being designed to learn one correct stack operation at each time step." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="2010.04674" id="319">
        <attvalues>
          <attvalue for="5" value=" Previously \cite{dusell+chiang:2020}, we proposed a stack-based RNN called the Nondeterministic Stack RNN (NS-RNN) that outperformed other stack RNNs on a range of CFL language modeling tasks." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1804.06028" id="320">
        <attvalues>
          <attvalue for="5" value=" Many machine learning problems involve sequential data with hierarchical structures, such as modeling context-free languages \cite{grefenstette+al:2015,dusell+chiang:2020}, evaluating mathematical expressions \cite{nangia+bowman:2018,hao+al:2018}, logical inference \cite{bowman+al:2015}, and modeling syntax in natural language \cite{dyer+al:2016,shen+al:2018,kim+al:2019}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1809.02836" id="321">
        <attvalues>
          <attvalue for="5" value=" Many machine learning problems involve sequential data with hierarchical structures, such as modeling context-free languages \cite{grefenstette+al:2015,dusell+chiang:2020}, evaluating mathematical expressions \cite{nangia+bowman:2018,hao+al:2018}, logical inference \cite{bowman+al:2015}, and modeling syntax in natural language \cite{dyer+al:2016,shen+al:2018,kim+al:2019}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1506.04834" id="322">
        <attvalues>
          <attvalue for="5" value=" Many machine learning problems involve sequential data with hierarchical structures, such as modeling context-free languages \cite{grefenstette+al:2015,dusell+chiang:2020}, evaluating mathematical expressions \cite{nangia+bowman:2018,hao+al:2018}, logical inference \cite{bowman+al:2015}, and modeling syntax in natural language \cite{dyer+al:2016,shen+al:2018,kim+al:2019}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1602.07776" id="323">
        <attvalues>
          <attvalue for="5" value=" To remedy this, some previous work has investigated the addition of differentiable stack data structures to RNNs \cite{sun+al:1995,grefenstette+al:2015,joulin+mikolov:2015,dusell+chiang:2020}, which is closely related to work on neural networks that model shift-reduce parsers \cite{bowman+al:2016,dyer+al:2016,shen+al:2019}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1810.09536" id="324">
        <attvalues>
          <attvalue for="5" value=" Many machine learning problems involve sequential data with hierarchical structures, such as modeling context-free languages \cite{grefenstette+al:2015,dusell+chiang:2020}, evaluating mathematical expressions \cite{nangia+bowman:2018,hao+al:2018}, logical inference \cite{bowman+al:2015}, and modeling syntax in natural language \cite{dyer+al:2016,shen+al:2018,kim+al:2019}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1906.10225" id="325">
        <attvalues>
          <attvalue for="5" value=" Many machine learning problems involve sequential data with hierarchical structures, such as modeling context-free languages \cite{grefenstette+al:2015,dusell+chiang:2020}, evaluating mathematical expressions \cite{nangia+bowman:2018,hao+al:2018}, logical inference \cite{bowman+al:2015}, and modeling syntax in natural language \cite{dyer+al:2016,shen+al:2018,kim+al:2019}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1909.00111" id="326">
        <attvalues>
          <attvalue for="5" value=" However, recurrent neural networks (RNNs) have difficulty learning to solve these tasks, or generalizing to held-out sequences, unless they have supervision or a hierarchical inductive bias \cite{schijndel+al:2019,wilcox+al:2019,mccoy+al:2020}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1906.04068" id="327">
        <attvalues>
          <attvalue for="5" value=" However, recurrent neural networks (RNNs) have difficulty learning to solve these tasks, or generalizing to held-out sequences, unless they have supervision or a hierarchical inductive bias \cite{schijndel+al:2019,wilcox+al:2019,mccoy+al:2020}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="2001.03632" id="328">
        <attvalues>
          <attvalue for="5" value=" However, recurrent neural networks (RNNs) have difficulty learning to solve these tasks, or generalizing to held-out sequences, unless they have supervision or a hierarchical inductive bias \cite{schijndel+al:2019,wilcox+al:2019,mccoy+al:2020}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1711.05738" id="329">
        <attvalues>
          <attvalue for="5" value=" To remedy this, some previous work has investigated the addition of differentiable stack data structures to RNNs \cite{sun+al:1995,grefenstette+al:2015,joulin+mikolov:2015,dusell+chiang:2020}, which is closely related to work on neural networks that model shift-reduce parsers \cite{bowman+al:2016,dyer+al:2016,shen+al:2019}." />
        </attvalues>
      </edge>
      <edge source="2109.01982" target="1910.13466" id="332">
        <attvalues>
          <attvalue for="5" value=" To remedy this, some previous work has investigated the addition of differentiable stack data structures to RNNs \cite{sun+al:1995,grefenstette+al:2015,joulin+mikolov:2015,dusell+chiang:2020}, which is closely related to work on neural networks that model shift-reduce parsers \cite{bowman+al:2016,dyer+al:2016,shen+al:2019}." />
        </attvalues>
      </edge>
      <edge source="2305.12152" target="2107.02173" id="333">
        <attvalues>
          <attvalue for="5" value=" Without the guidance of an automated metric, the number of topics, an important hyperparameter, is usually derived manually: Practitioners fit various topic models, inspect the resulting topics, and select the configuration which works best for the intended use case \cite{hoyle2021automated}." />
        </attvalues>
      </edge>
      <edge source="2305.12152" target="2302.04166" id="334">
        <attvalues>
          <attvalue for="5" value=" Recent NLP research explores whether large language models (LLMs) can perform automatic annotations; eg, to assess text quality \cite[][inter alia]{gpt_score, faggioli2023perspectives, huang2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.13074" target="2210.13050" id="335">
        <attvalues>
          <attvalue for="5" value=" However, despite demonstrating state-of-the-art results over standard benchmarks, careful error analyses show that these models fail to generalize to cases where the output is a symbolic form containing nested structures of depths greater than what is seen by the models during training \cite{3-yao-structural-generalization}." />
        </attvalues>
      </edge>
      <edge source="2404.13074" target="1910.13461" id="337">
        <attvalues>
          <attvalue for="5" value=" Several lines of research stem from the semantic parsing community's dissatisfaction with the performance of ``vanilla training'' of Transformer-based LLMs such as T5 \cite{raffel2023exploring} and BART \cite{lewis-etal-2020-bart}, which have shown success in purely language-based tasks such as summarization and question-answering." />
        </attvalues>
      </edge>
      <edge source="2402.11355" target="2004.07667" id="338">
        <attvalues>
          <attvalue for="5" value=" Interventions performed in the representation space of language models (LMs), generally $\RD$, have proven effective in understanding and exerting control over neural language models \cite{ravfogel2020null, ravfogel2021counterfactual, geva2021transformer, elazar2021amnesic, ravfogel2022adversarial, ravfogel-etal-2023-linear, belrose2023leace,li2023inference}." />
        </attvalues>
      </edge>
      <edge source="2402.11355" target="2306.03341" id="339">
        <attvalues>
          <attvalue for="5" value=" Interventions performed in the representation space of language models (LMs), generally $\RD$, have proven effective in understanding and exerting control over neural language models \cite{ravfogel2020null, ravfogel2021counterfactual, geva2021transformer, elazar2021amnesic, ravfogel2022adversarial, ravfogel-etal-2023-linear, belrose2023leace,li2023inference}." />
        </attvalues>
      </edge>
      <edge source="2402.11355" target="2205.05124" id="340">
        <attvalues>
          <attvalue for="5" value=" Another widely used approach is to \defn{steer} representations from one class to another, eg, shifting them toward a region in the representation space associated with a different class $c'$ \cite{subramani2022extracting, li2023inference, ravfogel2021counterfactual, singh2024mimic}." />
        </attvalues>
      </edge>
      <edge source="2402.11355" target="2004.12265" id="341">
        <attvalues>
          <attvalue for="5" value=" In this sense, representation surgery resembles a causal intervention \cite{vig2020causal, geiger2021causal, feder2021causalm, pmlr-v162-geiger22a, guerner2023geometric, lemberger2024explaining}, and we will informally use causal language throughout the paper, referring to such modifications in the representation space as interventions." />
        </attvalues>
      </edge>
      <edge source="2402.11355" target="2202.05262" id="342">
        <attvalues>
          <attvalue for="5" value=" We refer to this process as a \defn{counterfactual lens}, as it allows us to interpret representation-space counterfactuals in natural language, similar to representation-level interpretability techniques \cite{meng2022locating, nostalgebraist, belrose2023eliciting, ghandeharioun2024patchscope}." />
        </attvalues>
      </edge>
      <edge source="2402.11355" target="2310.06816" id="343">
        <attvalues>
          <attvalue for="5" value=" \cite{morris2023text} propose an iterative algorithm to approximate the inverse function $\inv \colon \RD arrow \kleene{\alphabet}$." />
        </attvalues>
      </edge>
      <edge source="2402.11355" target="1901.09451" id="344">
        <attvalues>
          <attvalue for="5" value=" We perform experiments on a dataset of short biographies annotated with gender and profession \cite{de2019bias}." />
        </attvalues>
      </edge>
      <edge source="2306.01200" target="2109.06379" id="345">
        <attvalues>
          <attvalue for="5" value=" Developing comprehensive evaluation frameworks~\cite{deng-2021-compression, yuan-2021-bartscore, zhong-2022-towards} that can evaluate multiple human-interpretable dimensions, such as factual consistency~\cite{kryscinski-2020-evaluating, wang-2020-asking} and coherence~\cite{dziri-2019-evaluating, huang-2020-grade}, is important for the advancement of Natural Language Generation (NLG)." />
        </attvalues>
      </edge>
      <edge source="2306.01200" target="1910.12840" id="346">
        <attvalues>
          <attvalue for="5" value=" Developing comprehensive evaluation frameworks~\cite{deng-2021-compression, yuan-2021-bartscore, zhong-2022-towards} that can evaluate multiple human-interpretable dimensions, such as factual consistency~\cite{kryscinski-2020-evaluating, wang-2020-asking} and coherence~\cite{dziri-2019-evaluating, huang-2020-grade}, is important for the advancement of Natural Language Generation (NLG)." />
        </attvalues>
      </edge>
      <edge source="2306.01200" target="1904.03371" id="347">
        <attvalues>
          <attvalue for="5" value=" Developing comprehensive evaluation frameworks~\cite{deng-2021-compression, yuan-2021-bartscore, zhong-2022-towards} that can evaluate multiple human-interpretable dimensions, such as factual consistency~\cite{kryscinski-2020-evaluating, wang-2020-asking} and coherence~\cite{dziri-2019-evaluating, huang-2020-grade}, is important for the advancement of Natural Language Generation (NLG)." />
        </attvalues>
      </edge>
      <edge source="2306.01200" target="2210.07197" id="348">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{zhong-2022-towards} developed a unified evaluation framework that can generalize to multiple dimensions and text generation tasks." />
        </attvalues>
      </edge>
      <edge source="2306.01200" target="2005.14165" id="349">
        <attvalues>
          <attvalue for="5" value=" Finally, in light of the recent work~\cite{goyal-news-2022} that points to the misalignment of existing evaluation metrics with human preference in evaluating zero-shot summaries generated by LLMs such as GPT-3 \cite{brown-2020-language}, we study the effectiveness of Ice in evaluating zero-shot summaries generated by GPT-3." />
        </attvalues>
      </edge>
      <edge source="2306.01200" target="2209.12356" id="350">
        <attvalues>
          <attvalue for="5" value=" Finally, in light of the recent work~\cite{goyal-news-2022} that points to the misalignment of existing evaluation metrics with human preference in evaluating zero-shot summaries generated by LLMs such as GPT-3 \cite{brown-2020-language}, we study the effectiveness of Ice in evaluating zero-shot summaries generated by GPT-3." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="1910.13461" id="351">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning pre-trained models on domain-specific datasets has been the leading paradigm in text summarization research in recent years \cite{lewis2020bart, zhang2020pegasus, raffel2020exploring}." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2005.14165" id="352">
        <attvalues>
          <attvalue for="5" value=" In this paper, we conduct the first systematic study of the impact of prompt-based models on the text summarization research space, using an Instruct-tuned 175B GPT-3 model (text-davinci-002) \cite{brown2020language, ouyang2022training} as a case study." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2110.08207" id="353">
        <attvalues>
          <attvalue for="5" value=" The success of prompting large language models (GPT-3 \cite{brown2020language}, T0 \cite{sanh2022multitask}, PaLM \cite{chowdhery2022palm}, etc) provides an alternative approach, namely learning from natural language task instructions and/or a few demonstrative examples in the context without updating model parameters." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2204.02311" id="354">
        <attvalues>
          <attvalue for="5" value=" The success of prompting large language models (GPT-3 \cite{brown2020language}, T0 \cite{sanh2022multitask}, PaLM \cite{chowdhery2022palm}, etc) provides an alternative approach, namely learning from natural language task instructions and/or a few demonstrative examples in the context without updating model parameters." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2102.09690" id="355">
        <attvalues>
          <attvalue for="5" value=" While recent work \cite{zhao2021calibrate, min2022rethinking, ye2022unreliability} has evaluated this paradigm across a number of tasks, it has only been studied for text summarization with unreliable automatic metrics \cite{he2022z, chowdhery2022palm, ouyang2022training} or in non-standard settings \cite{saunders2022self}." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2208.09770" id="356">
        <attvalues>
          <attvalue for="5" value=" While recent work \cite{zhao2021calibrate, min2022rethinking, ye2022unreliability} has evaluated this paradigm across a number of tasks, it has only been studied for text summarization with unreliable automatic metrics \cite{he2022z, chowdhery2022palm, ouyang2022training} or in non-standard settings \cite{saunders2022self}." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2206.05802" id="357">
        <attvalues>
          <attvalue for="5" value=" While recent work \cite{zhao2021calibrate, min2022rethinking, ye2022unreliability} has evaluated this paradigm across a number of tasks, it has only been studied for text summarization with unreliable automatic metrics \cite{he2022z, chowdhery2022palm, ouyang2022training} or in non-standard settings \cite{saunders2022self}." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="1912.08777" id="358">
        <attvalues>
          <attvalue for="5" value=" First, how do prompt-based GPT-3 summaries compare to those obtained from state-of-the-art fine-tuned summarization models \cite{zhang2020pegasus, liu2022brio}? We compare these approaches using A/B testing on a new corpus of recent news articles, and find that our study participants overwhelmingly prefer GPT-3 summaries across two different ``styles'' with different prompts (three-sentence and single-sentence)." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2005.00661" id="359">
        <attvalues>
          <attvalue for="5" value=" Moreover, these summaries do not suffer from limitations due to low-quality training data that plague fine-tuned generic summarization models \cite{maynez2020faithfulness, goyal2022training}." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2011.04096" id="361">
        <attvalues>
          <attvalue for="5" value=" Second, are existing automatic metrics well-suited to evaluating prompt-based summaries? Recent work has shown that classic reference-based such as ROUGE \cite{lin2004rouge} and BERTScore \cite{bertscore2020} are unreliable when small improvements are reported \cite{peyrard2019studying, fabbri2021summeval}; however large differences, on the order of say $5$ Rouge points or greater, are considered to be correlated with human preferences \cite{bhandari2020metrics, deutsch-etal-2022-examining}." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2112.08542" id="362">
        <attvalues>
          <attvalue for="5" value=" Furthermore, we show that recent reference-free metrics, eg QA-based metrics \cite{fabbri-etal-2022-qafacteval, durmus2020feqa} and trained factuality models \cite{kryscinski2020evaluating, goyal2020evaluating}, similarly fail to adapt to this shift from the fine-tuned to prompting, and need to be re-visited." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="1910.12840" id="363">
        <attvalues>
          <attvalue for="5" value=" Furthermore, we show that recent reference-free metrics, eg QA-based metrics \cite{fabbri-etal-2022-qafacteval, durmus2020feqa} and trained factuality models \cite{kryscinski2020evaluating, goyal2020evaluating}, similarly fail to adapt to this shift from the fine-tuned to prompting, and need to be re-visited." />
        </attvalues>
      </edge>
      <edge source="2209.12356" target="2012.04281" id="364">
        <attvalues>
          <attvalue for="5" value=" For keyword-based summarization, we find that GPT-3 consistently generates more coherent and keyword-relevant summaries compared to current fine-tuned alternatives: crowd annotators prefer GPT-3 summaries over a baseline model \cite{he2020ctrlsum} 70\% of the time." />
        </attvalues>
      </edge>
      <edge source="2005.00800" target="1805.05089" id="365">
        <attvalues>
          <attvalue for="5" value=" The Universal Dependencies project \cite{nivre2016universal} has made available multiple treebanks for the same language annotated according to the same scheme, leading to a new wave of research which explores ways to use multiple treebanks in monolingual parsing \cite{K17:3003,K17:3007,K17:3005,P18:2098}." />
        </attvalues>
      </edge>
      <edge source="2305.14171" target="1801.06146" id="366">
        <attvalues>
          <attvalue for="5" value=" Language models have become an essential tool in dealing with various tasks in the natural language processing (NLP) domain \cite[][inter alia]{howard-ruder-2018-universal, devlin-etal-2019-bert, NEURIPS2020_1457c0d6}." />
        </attvalues>
      </edge>
      <edge source="2305.14171" target="2102.09690" id="367">
        <attvalues>
          <attvalue for="5" value=" The decoder might be miscalibrated \cite{pmlr-v139-zhao21c} or suffer from surface form competition \cite{holtzman-etal-2021-surface}." />
        </attvalues>
      </edge>
      <edge source="2305.14171" target="2107.13586" id="368">
        <attvalues>
          <attvalue for="5" value=" First, zero-shot, or few-shot, performance of pretrained models on downstream tasks depends to a large extent on the way in which the prompt is formulated \cite{pmlr-v139-zhao21c,liu-prompt-survey-2023}." />
        </attvalues>
      </edge>
      <edge source="2305.14171" target="2109.01247" id="369">
        <attvalues>
          <attvalue for="5" value=" Second, ICL may not always benefit from the demonstrative examples the way that it is expected to \cite{webson-pavlick-2022-prompt}, eg, flipping the labels in demonstrations rarely hurts the performance \cite{min-etal-2022-rethinking}." />
        </attvalues>
      </edge>
      <edge source="2305.14171" target="2202.12837" id="370">
        <attvalues>
          <attvalue for="5" value=" Second, ICL may not always benefit from the demonstrative examples the way that it is expected to \cite{webson-pavlick-2022-prompt}, eg, flipping the labels in demonstrations rarely hurts the performance \cite{min-etal-2022-rethinking}." />
        </attvalues>
      </edge>
      <edge source="2305.14171" target="2104.08315" id="371">
        <attvalues>
          <attvalue for="5" value=" The decoder might be miscalibrated \cite{pmlr-v139-zhao21c} or suffer from surface form competition \cite{holtzman-etal-2021-surface}." />
        </attvalues>
      </edge>
      <edge source="2305.14171" target="2210.11416" id="372">
        <attvalues>
          <attvalue for="5" value=" We further compare in-context probing with in-context learning on different sizes of \flant models \cite{chung2022scaling} in \ref{sec:q2}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2303.08774" id="404">
        <attvalues>
          <attvalue for="5" value=" These advancements have sparked interest in exploring the capabilities of LLM in multi-modal perception, including speech \cite{Qwen-Audio,tang2024salmonn, hu2024wavllm,Qwen2-Audio}, vision \cite{OpenAI2023GPT4TR,huang2023language,peng2024grounding}, and content generation \cite{pan2024kosmosg, brooks2024video}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2307.09288" id="405">
        <attvalues>
          <attvalue for="5" value=" Specifically, we utilize Llama 2 \cite{touvron2023llama2} as our foundational LLM, coupled with the Whisper \cite{radford2023whisper} encoder to extract semantic context, and WavLM \cite{chen2022wavlm} multi-layer features to capture acoustic information indicating speaker characteristics, referring to WavLLM \cite{hu2024wavllm} and SALMONN \cite{tang2024salmonn}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2311.07919" id="406">
        <attvalues>
          <attvalue for="5" value=" Several studies have investigated speech-related LLM, which typically involve a fine-tuned text LLM following speech-related instructions and pairing with auxiliary audio encoders \cite{Qwen-Audio,tang2024salmonn, hu2024wavllm,Qwen2-Audio}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2310.13289" id="407">
        <attvalues>
          <attvalue for="5" value=" Specifically, we utilize Llama 2 \cite{touvron2023llama2} as our foundational LLM, coupled with the Whisper \cite{radford2023whisper} encoder to extract semantic context, and WavLM \cite{chen2022wavlm} multi-layer features to capture acoustic information indicating speaker characteristics, referring to WavLLM \cite{hu2024wavllm} and SALMONN \cite{tang2024salmonn}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2407.10759" id="408">
        <attvalues>
          <attvalue for="5" value=" Several studies have investigated speech-related LLM, which typically involve a fine-tuned text LLM following speech-related instructions and pairing with auxiliary audio encoders \cite{Qwen-Audio,tang2024salmonn, hu2024wavllm,Qwen2-Audio}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2302.14045" id="409">
        <attvalues>
          <attvalue for="5" value=" These advancements have sparked interest in exploring the capabilities of LLM in multi-modal perception, including speech \cite{Qwen-Audio,tang2024salmonn, hu2024wavllm,Qwen2-Audio}, vision \cite{OpenAI2023GPT4TR,huang2023language,peng2024grounding}, and content generation \cite{pan2024kosmosg, brooks2024video}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2310.02992" id="410">
        <attvalues>
          <attvalue for="5" value=" These advancements have sparked interest in exploring the capabilities of LLM in multi-modal perception, including speech \cite{Qwen-Audio,tang2024salmonn, hu2024wavllm,Qwen2-Audio}, vision \cite{OpenAI2023GPT4TR,huang2023language,peng2024grounding}, and content generation \cite{pan2024kosmosg, brooks2024video}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2002.03921" id="411">
        <attvalues>
          <attvalue for="5" value=" These studies are based on Permutation Invariant Training (PIT) \cite{zhang2020pit,chang2020pit,meng23sidecaricassp}, Heuristic Error Assignment Training (HEAT) \cite{lu2021surt,raj2023surt2}, or Serialized Output Training (SOT) \cite{kanda2020sot,kanda22tsot,shi24sot,li2024sidecarsot} to match predictions with corresponding target labels for loss calculation." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2302.09908" id="412">
        <attvalues>
          <attvalue for="5" value=" These studies are based on Permutation Invariant Training (PIT) \cite{zhang2020pit,chang2020pit,meng23sidecaricassp}, Heuristic Error Assignment Training (HEAT) \cite{lu2021surt,raj2023surt2}, or Serialized Output Training (SOT) \cite{kanda2020sot,kanda22tsot,shi24sot,li2024sidecarsot} to match predictions with corresponding target labels for loss calculation." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2104.02109" id="413">
        <attvalues>
          <attvalue for="5" value=" These studies are based on Permutation Invariant Training (PIT) \cite{zhang2020pit,chang2020pit,meng23sidecaricassp}, Heuristic Error Assignment Training (HEAT) \cite{lu2021surt,raj2023surt2}, or Serialized Output Training (SOT) \cite{kanda2020sot,kanda22tsot,shi24sot,li2024sidecarsot} to match predictions with corresponding target labels for loss calculation." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2003.12687" id="415">
        <attvalues>
          <attvalue for="5" value=" These studies are based on Permutation Invariant Training (PIT) \cite{zhang2020pit,chang2020pit,meng23sidecaricassp}, Heuristic Error Assignment Training (HEAT) \cite{lu2021surt,raj2023surt2}, or Serialized Output Training (SOT) \cite{kanda2020sot,kanda22tsot,shi24sot,li2024sidecarsot} to match predictions with corresponding target labels for loss calculation." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2202.00842" id="416">
        <attvalues>
          <attvalue for="5" value=" These studies are based on Permutation Invariant Training (PIT) \cite{zhang2020pit,chang2020pit,meng23sidecaricassp}, Heuristic Error Assignment Training (HEAT) \cite{lu2021surt,raj2023surt2}, or Serialized Output Training (SOT) \cite{kanda2020sot,kanda22tsot,shi24sot,li2024sidecarsot} to match predictions with corresponding target labels for loss calculation." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2407.03966" id="417">
        <attvalues>
          <attvalue for="5" value=" These studies are based on Permutation Invariant Training (PIT) \cite{zhang2020pit,chang2020pit,meng23sidecaricassp}, Heuristic Error Assignment Training (HEAT) \cite{lu2021surt,raj2023surt2}, or Serialized Output Training (SOT) \cite{kanda2020sot,kanda22tsot,shi24sot,li2024sidecarsot} to match predictions with corresponding target labels for loss calculation." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2211.00482" id="418">
        <attvalues>
          <attvalue for="5" value=" However, these approaches typically transcribe speech from all talkers indiscriminately and fail to associate transcriptions with specific talkers, unless an additional external \cite{huang2023pit_tse,masumura23joint} or internal \cite{kanda21sasot,conformer-ts-asr,masumura24_interspeech} model is employed to extract speaker information." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2306.02273" id="419">
        <attvalues>
          <attvalue for="5" value=" However, these approaches typically transcribe speech from all talkers indiscriminately and fail to associate transcriptions with specific talkers, unless an additional external \cite{huang2023pit_tse,masumura23joint} or internal \cite{kanda21sasot,conformer-ts-asr,masumura24_interspeech} model is employed to extract speaker information." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2104.02128" id="420">
        <attvalues>
          <attvalue for="5" value=" However, these approaches typically transcribe speech from all talkers indiscriminately and fail to associate transcriptions with specific talkers, unless an additional external \cite{huang2023pit_tse,masumura23joint} or internal \cite{kanda21sasot,conformer-ts-asr,masumura24_interspeech} model is employed to extract speaker information." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2308.05218" id="421">
        <attvalues>
          <attvalue for="5" value=" However, these approaches typically transcribe speech from all talkers indiscriminately and fail to associate transcriptions with specific talkers, unless an additional external \cite{huang2023pit_tse,masumura23joint} or internal \cite{kanda21sasot,conformer-ts-asr,masumura24_interspeech} model is employed to extract speaker information." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2305.16263" id="422">
        <attvalues>
          <attvalue for="5" value=" Although several studies \cite{meng23sidecarinterspeech,meng24whisperovlp} proposed handling multi-talker ASR in conjunction with other tasks within a single model, the addressed tasks remain constrained and lack the flexibility to address various user requirements specifying talker attributes such as please transcribe the talker who said the word “strawberry”." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2407.09817" id="423">
        <attvalues>
          <attvalue for="5" value=" Although several studies \cite{meng23sidecarinterspeech,meng24whisperovlp} proposed handling multi-talker ASR in conjunction with other tasks within a single model, the addressed tasks remain constrained and lack the flexibility to address various user requirements specifying talker attributes such as please transcribe the talker who said the word “strawberry”." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2110.13900" id="425">
        <attvalues>
          <attvalue for="5" value=" Specifically, we utilize Llama 2 \cite{touvron2023llama2} as our foundational LLM, coupled with the Whisper \cite{radford2023whisper} encoder to extract semantic context, and WavLM \cite{chen2022wavlm} multi-layer features to capture acoustic information indicating speaker characteristics, referring to WavLLM \cite{hu2024wavllm} and SALMONN \cite{tang2024salmonn}." />
        </attvalues>
      </edge>
      <edge source="2409.08596" target="2404.00656" id="426">
        <attvalues>
          <attvalue for="5" value=" Specifically, we utilize Llama 2 \cite{touvron2023llama2} as our foundational LLM, coupled with the Whisper \cite{radford2023whisper} encoder to extract semantic context, and WavLM \cite{chen2022wavlm} multi-layer features to capture acoustic information indicating speaker characteristics, referring to WavLLM \cite{hu2024wavllm} and SALMONN \cite{tang2024salmonn}." />
        </attvalues>
      </edge>
      <edge source="2211.00482" target="2205.10643" id="427">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL), which obtains supervisory signals from the input data itself, is an active sub-area of unsupervised learning~\cite{mohamed2022self,liu2021self}." />
        </attvalues>
      </edge>
      <edge source="2211.00482" target="2006.08218" id="428">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL), which obtains supervisory signals from the input data itself, is an active sub-area of unsupervised learning~\cite{mohamed2022self,liu2021self}." />
        </attvalues>
      </edge>
      <edge source="2211.00482" target="2110.13900" id="430">
        <attvalues>
          <attvalue for="5" value=" The previous studies on pre-training and fine-tuning upstream SSL models for multi-talker scenarios are very limited~\cite{chen2022wavlm,chang2022end}." />
        </attvalues>
      </edge>
      <edge source="2211.00482" target="2006.11477" id="431">
        <attvalues>
          <attvalue for="5" value="2\% on LibriSpeech~\cite{Panayotov2015LibrispeechAA} test-other after fine-tuning with 10 minutes of labeled speech~\cite{baevski2020wav2vec}." />
        </attvalues>
      </edge>
      <edge source="2211.00482" target="2106.07447" id="432">
        <attvalues>
          <attvalue for="5" value=" By simply replacing the acoustic features with hidden representations from pre-trained upstream models~\cite{yang2021superb,chen2022wavlm}, or fine-tuning the entire model with additional task-specific layers~\cite{baevski2020wav2vec,hsu2021hubert}, SSL models have surpassed state-of-the-art downstream-only baselines for several tasks including speech recognition~\cite{baevski2020wav2vec,hsu2021hubert} and speaker recognition~\cite{chen2022large,fan2020exploring}." />
        </attvalues>
      </edge>
      <edge source="2211.00482" target="2110.05777" id="433">
        <attvalues>
          <attvalue for="5" value=" By simply replacing the acoustic features with hidden representations from pre-trained upstream models~\cite{yang2021superb,chen2022wavlm}, or fine-tuning the entire model with additional task-specific layers~\cite{baevski2020wav2vec,hsu2021hubert}, SSL models have surpassed state-of-the-art downstream-only baselines for several tasks including speech recognition~\cite{baevski2020wav2vec,hsu2021hubert} and speaker recognition~\cite{chen2022large,fan2020exploring}." />
        </attvalues>
      </edge>
      <edge source="2211.00482" target="2204.00540" id="435">
        <attvalues>
          <attvalue for="5" value=" The previous studies on pre-training and fine-tuning upstream SSL models for multi-talker scenarios are very limited~\cite{chen2022wavlm,chang2022end}." />
        </attvalues>
      </edge>
      <edge source="2211.00482" target="2005.07272" id="436">
        <attvalues>
          <attvalue for="5" value=" For the former, we build upon previous work on target speaker extraction and recognition~\cite{delcroix2018single,vzmolikova2019speakerbeam,medennikov2020target,wang2018voicefilter}; we use target speaker embeddings (extracted from enrollment audio) to direct the model to transcribe a specific speaker in the segmented utterance containing mixed speech." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="1706.03762" id="438">
        <attvalues>
          <attvalue for="5" value=" Transformer-based large language models (LLMs) have achieved extraordinary performance on various language tasks \cite{vaswani2017attention,Brown2020,OpenAI2023,Touvron2023llama,Touvron2023llama2,Chowdhery2022}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2005.14165" id="439">
        <attvalues>
          <attvalue for="5" value=" Transformer-based large language models (LLMs) have achieved extraordinary performance on various language tasks \cite{vaswani2017attention,Brown2020,OpenAI2023,Touvron2023llama,Touvron2023llama2,Chowdhery2022}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2303.08774" id="440">
        <attvalues>
          <attvalue for="5" value=" Transformer-based large language models (LLMs) have achieved extraordinary performance on various language tasks \cite{vaswani2017attention,Brown2020,OpenAI2023,Touvron2023llama,Touvron2023llama2,Chowdhery2022}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2307.09288" id="442">
        <attvalues>
          <attvalue for="5" value=" Transformer-based large language models (LLMs) have achieved extraordinary performance on various language tasks \cite{vaswani2017attention,Brown2020,OpenAI2023,Touvron2023llama,Touvron2023llama2,Chowdhery2022}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2204.02311" id="443">
        <attvalues>
          <attvalue for="5" value=" Transformer-based large language models (LLMs) have achieved extraordinary performance on various language tasks \cite{vaswani2017attention,Brown2020,OpenAI2023,Touvron2023llama,Touvron2023llama2,Chowdhery2022}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="1603.08983" id="444">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2006.04152" id="446">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2004.07453" id="447">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2004.02178" id="448">
        <attvalues>
          <attvalue for="5" value=" This strategy has been adopted in some prior works for model architectures tailored to classification tasks, eg~the encoder-only BERT model \cite{Xin2020,Liu2020,Hu2023SmartBERTAP} or others \cite{Panda2015ConditionalDL,Kaya2018,Bakhtiarnia2021ImprovingTA,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="1910.10073" id="449">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2004.12993" id="450">
        <attvalues>
          <attvalue for="5" value=" This strategy has been adopted in some prior works for model architectures tailored to classification tasks, eg~the encoder-only BERT model \cite{Xin2020,Liu2020,Hu2023SmartBERTAP} or others \cite{Panda2015ConditionalDL,Kaya2018,Bakhtiarnia2021ImprovingTA,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2105.13878" id="451">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2104.08803" id="452">
        <attvalues>
          <attvalue for="5" value=" This work considers token-wise early exiting for generative LLMs and autoregressive natural language generation \cite{Schuster2021,DelCorro2023,Bae2023,Varshney2023AcceleratingLI,Gera2023TheBO,chen2023eellm}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2202.07101" id="453">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2303.09266" id="454">
        <attvalues>
          <attvalue for="5" value=" This strategy has been adopted in some prior works for model architectures tailored to classification tasks, eg~the encoder-only BERT model \cite{Xin2020,Liu2020,Hu2023SmartBERTAP} or others \cite{Panda2015ConditionalDL,Kaya2018,Bakhtiarnia2021ImprovingTA,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="1509.08971" id="455">
        <attvalues>
          <attvalue for="5" value=" This strategy has been adopted in some prior works for model architectures tailored to classification tasks, eg~the encoder-only BERT model \cite{Xin2020,Liu2020,Hu2023SmartBERTAP} or others \cite{Panda2015ConditionalDL,Kaya2018,Bakhtiarnia2021ImprovingTA,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="1709.01686" id="456">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="1810.07052" id="457">
        <attvalues>
          <attvalue for="5" value=" This strategy has been adopted in some prior works for model architectures tailored to classification tasks, eg~the encoder-only BERT model \cite{Xin2020,Liu2020,Hu2023SmartBERTAP} or others \cite{Panda2015ConditionalDL,Kaya2018,Bakhtiarnia2021ImprovingTA,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="1703.09844" id="458">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2004.12814" id="459">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2106.05022" id="460">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2102.04906" id="461">
        <attvalues>
          <attvalue for="5" value=" Early exiting has found success in natural language processing \cite{Graves2016AdaptiveCT,Hou2020,Zhou2020,Schwartz2020,Liu2020,Elbayad2020,Xin2020,Li2021AcceleratingBI,Schuster2021,Xin2021,Xu2023SurveyDynamic,Hu2023SmartBERTAP}, computer vision \cite{Panda2015ConditionalDL,Teerapittayanon2016,Kaya2018,Huang2018}, and many other areas \cite{Scardapane2020WhySW,Laskaridis2021AdaptiveIT,Han2021DynamicNN,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2312.05385" id="462">
        <attvalues>
          <attvalue for="5" value=" This strategy has been adopted in some prior works for model architectures tailored to classification tasks, eg~the encoder-only BERT model \cite{Xin2020,Liu2020,Hu2023SmartBERTAP} or others \cite{Panda2015ConditionalDL,Kaya2018,Bakhtiarnia2021ImprovingTA,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2307.02628" id="463">
        <attvalues>
          <attvalue for="5" value=" This work considers token-wise early exiting for generative LLMs and autoregressive natural language generation \cite{Schuster2021,DelCorro2023,Bae2023,Varshney2023AcceleratingLI,Gera2023TheBO,chen2023eellm}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2310.05424" id="464">
        <attvalues>
          <attvalue for="5" value=" This work considers token-wise early exiting for generative LLMs and autoregressive natural language generation \cite{Schuster2021,DelCorro2023,Bae2023,Varshney2023AcceleratingLI,Gera2023TheBO,chen2023eellm}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2310.18581" id="465">
        <attvalues>
          <attvalue for="5" value=" Another recent work \cite{Varshney2023AcceleratingLI} proposed to initialize the model parameters of early-exit LLMs with pre-trained standard LLMs, but followed by full-parameter training." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2305.01628" id="466">
        <attvalues>
          <attvalue for="5" value=" Closest to our setting and training methodology is the recent work \cite{Gera2023TheBO}, which investigated generative LLMs of sizes up to 355M, and only considered linear exit heads that are randomly initialized." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2312.04916" id="467">
        <attvalues>
          <attvalue for="5" value=" Our implementation is based on the recently proposed \eellm framework \cite{chen2023eellm}, complementing the latter with an alternative solution to training early-exit LLMs that is both accessible and scalable, thanks to its low computational complexity and full compatibility with 3D parallelism." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="1909.08053" id="468">
        <attvalues>
          <attvalue for="5" value=" Recent work has made this approach compatible with massive 3D parallelism, thereby scaling up early-exit LLMs to sizes as large as any standard LLM that can possibly be trained with state-of-the-art LLM frameworks \cite{Shoeybi2019,Narayanan2021,chen2023eellm}." />
        </attvalues>
      </edge>
      <edge source="2402.00518" target="2104.10461" id="470">
        <attvalues>
          <attvalue for="5" value=" This strategy has been adopted in some prior works for model architectures tailored to classification tasks, eg~the encoder-only BERT model \cite{Xin2020,Liu2020,Hu2023SmartBERTAP} or others \cite{Panda2015ConditionalDL,Kaya2018,Bakhtiarnia2021ImprovingTA,Dai2023ApparateRE}." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="2005.14165" id="495">
        <attvalues>
          <attvalue for="5" value=" For Transformer models~\cite{transformer,bert,albert,gpt3}, the input tokens are fed into the model in parallel, while models have dozens of Transformer layers." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="2001.08361" id="496">
        <attvalues>
          <attvalue for="5" value=" Scaling up model capacity is an obvious yet effective approach for better performance in natural language processing (NLP) tasks~\cite{gpt3,kaplan2020scaling,ghorbani2021scaling,zhou2020evaluating}." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="2109.07740" id="497">
        <attvalues>
          <attvalue for="5" value=" Scaling up model capacity is an obvious yet effective approach for better performance in natural language processing (NLP) tasks~\cite{gpt3,kaplan2020scaling,ghorbani2021scaling,zhou2020evaluating}." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="1911.11931" id="498">
        <attvalues>
          <attvalue for="5" value=" Scaling up model capacity is an obvious yet effective approach for better performance in natural language processing (NLP) tasks~\cite{gpt3,kaplan2020scaling,ghorbani2021scaling,zhou2020evaluating}." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="1706.03762" id="499">
        <attvalues>
          <attvalue for="5" value=" For Transformer models~\cite{transformer,bert,albert,gpt3}, the input tokens are fed into the model in parallel, while models have dozens of Transformer layers." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="1909.11942" id="501">
        <attvalues>
          <attvalue for="5" value=" Note that this stream of works is distinct from static model acceleration, which is often referred to as model compression, including knowledge distillation, weight sharing, pruning and quantization~\cite{sanh2019distilbert,bot,albert,zafrir2019q8bert,xu2021beyond} (etc, see another survey~\cite{xu2022survey})." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="2002.02925" id="503">
        <attvalues>
          <attvalue for="5" value=" Note that this stream of works is distinct from static model acceleration, which is often referred to as model compression, including knowledge distillation, weight sharing, pruning and quantization~\cite{sanh2019distilbert,bot,albert,zafrir2019q8bert,xu2021beyond} (etc, see another survey~\cite{xu2022survey})." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="2109.03228" id="505">
        <attvalues>
          <attvalue for="5" value=" Note that this stream of works is distinct from static model acceleration, which is often referred to as model compression, including knowledge distillation, weight sharing, pruning and quantization~\cite{sanh2019distilbert,bot,albert,zafrir2019q8bert,xu2021beyond} (etc, see another survey~\cite{xu2022survey})." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="2202.07105" id="506">
        <attvalues>
          <attvalue for="5" value=" Note that this stream of works is distinct from static model acceleration, which is often referred to as model compression, including knowledge distillation, weight sharing, pruning and quantization~\cite{sanh2019distilbert,bot,albert,zafrir2019q8bert,xu2021beyond} (etc, see another survey~\cite{xu2022survey})." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="2004.07453" id="507">
        <attvalues>
          <attvalue for="5" value=" These two streams of research are in fact orthogonal and recent works \cite{schwartz2020right}, \cite{fastbert} and \cite{leebert} have shown that static and dynamic approaches can be combined for even faster inference and better performance." />
        </attvalues>
      </edge>
      <edge source="2202.07101" target="2004.02178" id="508">
        <attvalues>
          <attvalue for="5" value=" These two streams of research are in fact orthogonal and recent works \cite{schwartz2020right}, \cite{fastbert} and \cite{leebert} have shown that static and dynamic approaches can be combined for even faster inference and better performance." />
        </attvalues>
      </edge>
      <edge source="1909.08053" target="1909.11942" id="510">
        <attvalues>
          <attvalue for="5" value=" By finetuning these pretrained language models on downstream natural language tasks, one can achieve state of the art results as shown in recent work \cite{devlin2018bert, ELMo, Howard2018ULMFIT, Radford2018GPT, Radford2017Sentiment, Le2016seq2seqtransfer, roberta,transformerxl,xlnet,mtdnn,ALBERT2019}." />
        </attvalues>
      </edge>
      <edge source="1909.08053" target="1901.02860" id="512">
        <attvalues>
          <attvalue for="5" value=" By finetuning these pretrained language models on downstream natural language tasks, one can achieve state of the art results as shown in recent work \cite{devlin2018bert, ELMo, Howard2018ULMFIT, Radford2018GPT, Radford2017Sentiment, Le2016seq2seqtransfer, roberta,transformerxl,xlnet,mtdnn,ALBERT2019}." />
        </attvalues>
      </edge>
      <edge source="1909.08053" target="1901.11504" id="514">
        <attvalues>
          <attvalue for="5" value=" By finetuning these pretrained language models on downstream natural language tasks, one can achieve state of the art results as shown in recent work \cite{devlin2018bert, ELMo, Howard2018ULMFIT, Radford2018GPT, Radford2017Sentiment, Le2016seq2seqtransfer, roberta,transformerxl,xlnet,mtdnn,ALBERT2019}." />
        </attvalues>
      </edge>
      <edge source="1909.08053" target="1604.06174" id="515">
        <attvalues>
          <attvalue for="5" value=" As these models become larger, they exceed the memory limit of modern processors, and require additional memory management techniques such as activation checkpointing \cite{activation_checkpointing}." />
        </attvalues>
      </edge>
      <edge source="1909.08053" target="1811.06965" id="516">
        <attvalues>
          <attvalue for="5" value=" This approach is orthogonal to pipeline-based model parallelism as advocated by approaches such as GPipe~\cite{GPipe}." />
        </attvalues>
      </edge>
      <edge source="2203.08555" target="1911.02116" id="519">
        <attvalues>
          <attvalue for="5" value=" This is due in no small part to large multilingual pretrained language models (PLMs) such as mBERT \cite{devlin-etal-2019-bert} and XLM-RoBERTa \cite{conneau-etal-2020-unsupervised}, which have been found to have surprising cross-lingual transfer capabilities in spite of receiving no cross-lingual supervision." />
        </attvalues>
      </edge>
      <edge source="2203.08555" target="1904.09077" id="520">
        <attvalues>
          <attvalue for="5" value=" \cite{wu-dredze-2019-beto}, for example, found mBERT to perform well in a zero-shot setting when fine-tuned for five different NLP tasks in different languages." />
        </attvalues>
      </edge>
      <edge source="2203.08555" target="1906.01502" id="521">
        <attvalues>
          <attvalue for="5" value=" There is, however, a sharp divide between languages that benefit from this transfer and languages that do not, and there is ample evidence that transfer works best between typologically similar languages \cite[][among others]{pires-etal-2019-multilingual,lauscher-etal-2020-zero}." />
        </attvalues>
      </edge>
      <edge source="2203.08555" target="2009.11138" id="522">
        <attvalues>
          <attvalue for="5" value=" \cite{zhang2020worst} recently developed such a method." />
        </attvalues>
      </edge>
      <edge source="2203.08555" target="2004.10643" id="524">
        <attvalues>
          <attvalue for="5" value=" Multilingual dependency parsing is an ideal test case for this method, as the Universal Dependency treebanks \cite{nivre-etal-2020-universal} are currently the manually annotated dataset that covers the most typological diversity \cite{ponti-etal-2021-minimax}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2303.12712" id="535">
        <attvalues>
          <attvalue for="5" value=" Transfer learning has witnessed remarkable success in recent years, particularly exemplified by the advancements in foundation models for Natural Language Processing (NLP) ~\cite{gpt4, touvron2023llama} and Computer Vision (CV)~\cite{kirillov2023segment, CLIP}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2304.02643" id="536">
        <attvalues>
          <attvalue for="5" value=" For the second challenge, we employ a standard Transformer architecture due to its flexibility in modeling sequences with self-attention and its proven transferability across domains like CV and NLP~\cite{kirillov2023segment, Devlin2019BERTPO}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2006.09963" id="538">
        <attvalues>
          <attvalue for="5" value=" Previous methods circumvent this by ignoring the features and only transferring knowledge from the structural side~\cite{qiu2020gcc, davies2023its}, or constraining the applications in vertical domains where node/edge features are naturally aligned~\cite{Hu2019StrategiesFP, Xia2023MoleBERTRP}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2103.00111" id="540">
        <attvalues>
          <attvalue for="5" value=" Such approaches, while somewhat effective, suffer from performance loss or restricted applicability~\cite{Liu2021GraphSL}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2402.08907" id="541">
        <attvalues>
          <attvalue for="5" value=" Another challenge is structural heterogeneity, which arises from the vastly different structural patterns across various graphs, leading to out-of-distribution scenarios and potential negative transfer~\cite{Wang2024SubgraphPT}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2306.01323" id="542">
        <attvalues>
          <attvalue for="5" value=" A typical example is the varying degrees of homophily across graphs~\cite{mao2024demystifying}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2307.03393" id="543">
        <attvalues>
          <attvalue for="5" value=" They replace traditional shallow features like word2vec and tf-idf with language model-enhanced features and have demonstrated impressive empirical success, represented by the improved performance on graph-related tasks and the reduced gap between purely feature-based approaches (eg, an MLP) and graph-tailored models (eg, Graph Neural Networks)~\cite{Chen2023ExploringTP}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2111.06377" id="544">
        <attvalues>
          <attvalue for="5" value=" Inspired by works in CV, NLP and the graph domain~\cite{He2021MaskedAA, Devlin2019BERTPO, tang2021graph}, we adopt a reconstruction-based objective as the pretext task due to its efficacy in enforcing a robust understanding of dependencies within various data schemes." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="1607.00653" id="545">
        <attvalues>
          <attvalue for="5" value=" To address the first challenge, we propose using node sequences generated by random walks as contexts, where the entire sequence forms the receptive field, and the order of nodes preserves proximity information~\cite{Grover2016node2vecSF, Perozzi2014DeepWalkOL}." />
        </attvalues>
      </edge>
      <edge source="2406.13873" target="2205.10803" id="546">
        <attvalues>
          <attvalue for="5" value=" To handle the multi-dimensional and continuous nature of LM-produced features, we adopt a cosine similarity-based objective to measure reconstruction error instead of cross-entropy~\cite{hou2022graphmae}." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2105.00696" id="548">
        <attvalues>
          <attvalue for="5" value=" Graphs are ubiquitous in various disciplines and applications, encompassing a wide range of real-world scenarios~\cite{Xia2021GraphLA}." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="1905.07953" id="550">
        <attvalues>
          <attvalue for="5" value=" Many of these graphs have nodes that are associated with text attributes, resulting in the emergence of text-attributed graphs, such as citation graphs~\cite{hu2020open, Sen_Namata_Bilgic_Getoor_Galligher_Eliassi-Rad_2008} and product graphs~\cite{Chiang2019ClusterGCNAE}." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="1910.09796" id="552">
        <attvalues>
          <attvalue for="5" value=" These graphs have seen widespread use across a myriad of domains, from social network analysis~\cite{social_network}, information retrieval~\cite{Zhu2021TextGNNIT}, to a diverse range of natural language processing tasks~\cite{liu-etal-2020-fine, Yao2018GraphCN}." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2003.08271" id="554">
        <attvalues>
          <attvalue for="5" value=" Recent studies demonstrate that these non-contextualized shallow embeddings suffer from some limitations, such as the inability to capture polysemous words~\cite{Qiu2020PretrainedMF} and deficiency in semantic information~\cite{miaschi-dellorletta-2020-contextual, ethayarajh-2019-contextual}, which may lead to sub-optimal performance on downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="1909.01066" id="555">
        <attvalues>
          <attvalue for="5" value=" Compared to these non-contextualized shallow textual embeddings, large language models (LLMs) present massive context-aware knowledge and superior semantic comprehension capability through the process of pre-training on large-scale text corpora~\cite{Petroni2019LanguageMA, ethayarajh-2019-contextual}." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2303.18223" id="556">
        <attvalues>
          <attvalue for="5" value=" This knowledge achieved from pre-training has led to a surge of revolutions for downstream NLP tasks~\cite{Zhao2023ASO}." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2303.08774" id="557">
        <attvalues>
          <attvalue for="5" value=" Exemplars such as ChatGPT and GPT4~\cite{OpenAI2023GPT4TR}, equipped with hundreds of billions of parameters, exhibit superior performance~\cite{Bubeck2023SparksOA} on numerous text-related tasks from various domains." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2303.12712" id="558">
        <attvalues>
          <attvalue for="5" value=" Exemplars such as ChatGPT and GPT4~\cite{OpenAI2023GPT4TR}, equipped with hundreds of billions of parameters, exhibit superior performance~\cite{Bubeck2023SparksOA} on numerous text-related tasks from various domains." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2304.10149" id="559">
        <attvalues>
          <attvalue for="5" value=" Considering the exceptional ability of these LLMs to process and understand textual data, a pertinent question arises: (1) Can we leverage the knowledge of LLMs to compensate for the deficiency of contextualized knowledge and semantic comprehension inherent in the conventional GNN pipelines? In addition to the knowledge learned via pre-training, recent studies suggest that LLMs present preliminary success on tasks with implicit graph structures such as recommendation \cite{liu2023chatgpt_rec, Gao2023ChatRECTI}, ranking \cite{Ji2023ExploringCA}, and multi-hop reasoning \cite{creswell2023selectioninference}, in which LLMs are adopted to make the final predictions." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2303.07610" id="560">
        <attvalues>
          <attvalue for="5" value=" Considering the exceptional ability of these LLMs to process and understand textual data, a pertinent question arises: (1) Can we leverage the knowledge of LLMs to compensate for the deficiency of contextualized knowledge and semantic comprehension inherent in the conventional GNN pipelines? In addition to the knowledge learned via pre-training, recent studies suggest that LLMs present preliminary success on tasks with implicit graph structures such as recommendation \cite{liu2023chatgpt_rec, Gao2023ChatRECTI}, ranking \cite{Ji2023ExploringCA}, and multi-hop reasoning \cite{creswell2023selectioninference}, in which LLMs are adopted to make the final predictions." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2205.09712" id="561">
        <attvalues>
          <attvalue for="5" value=" Considering the exceptional ability of these LLMs to process and understand textual data, a pertinent question arises: (1) Can we leverage the knowledge of LLMs to compensate for the deficiency of contextualized knowledge and semantic comprehension inherent in the conventional GNN pipelines? In addition to the knowledge learned via pre-training, recent studies suggest that LLMs present preliminary success on tasks with implicit graph structures such as recommendation \cite{liu2023chatgpt_rec, Gao2023ChatRECTI}, ranking \cite{Ji2023ExploringCA}, and multi-hop reasoning \cite{creswell2023selectioninference}, in which LLMs are adopted to make the final predictions." />
        </attvalues>
      </edge>
      <edge source="2307.03393" target="2201.03514" id="562">
        <attvalues>
          <attvalue for="5" value=" It is evident that different types of LLMs possess varying levels of capability, and more powerful models often come with more usage restrictions~\cite{sun2022black, Zhao2023ASO, Qiu2020PretrainedMF}." />
        </attvalues>
      </edge>
      <edge source="2404.09135" target="2005.14165" id="563">
        <attvalues>
          <attvalue for="5" value="5, \cite{brown2020language,wang2022self} has demonstrated its remarkable ability to generate coherent sequences of words and engage in conversational interactions." />
        </attvalues>
      </edge>
      <edge source="2404.09135" target="2307.06435" id="564">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{naveed2023comprehensive} reviewed LLMs from 2019, starting with T5, up to the latest releases in 2023, offering comprehensive references and comparisons." />
        </attvalues>
      </edge>
      <edge source="2404.09135" target="2305.16326" id="566">
        <attvalues>
          <attvalue for="5" value=" \cite{wang2023pre} highlighted the promising applications of LLMs in addressing biomedical questions, while \cite{chen2023large} focused on evaluating biomedical LLMs with respect to benchmarks and summarization capabilities." />
        </attvalues>
      </edge>
      <edge source="2404.09135" target="2307.03109" id="567">
        <attvalues>
          <attvalue for="5" value=" Although \cite{chang2023survey} surveyed LLM evaluation, comprehensive summarizing the metrics remains scarce." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="1806.00258" id="568">
        <attvalues>
          <attvalue for="5" value=" In this context, the six challenges about MT proposed by \newcite{koehn-knowles-2017-six} have been widely recognized and studied by numerous studies, with many efforts revolving around them \cite{chu-wang-2018-survey,neishi-yoshinaga-2019-relation,garg-etal-2019-jointly,pang2023rethinking}." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="1909.02074" id="569">
        <attvalues>
          <attvalue for="5" value=" In this context, the six challenges about MT proposed by \newcite{koehn-knowles-2017-six} have been widely recognized and studied by numerous studies, with many efforts revolving around them \cite{chu-wang-2018-survey,neishi-yoshinaga-2019-relation,garg-etal-2019-jointly,pang2023rethinking}." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="2303.08774" id="571">
        <attvalues>
          <attvalue for="5" value=" The emerging Large Language Models (LLMs) have been a significant breakthrough in NLP \cite{touvron2023llama,openai2023gpt4,touvron2023llama2}." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="2307.09288" id="572">
        <attvalues>
          <attvalue for="5" value=" Note that English is a high-resource language in the Llama2 pretraining data \cite{touvron2023llama2} and German is a relatively high-resource language, ensuring the model's competence in these languages." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="2304.04675" id="573">
        <attvalues>
          <attvalue for="5" value=" LLMs have demonstrated remarkable capabilities, outperforming traditional approaches and setting new benchmark performance for various applications such as machine translation \cite{lyu2023new,zhu2023multilingual,zhang2023prompting,wang2023document}." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="2301.07069" id="574">
        <attvalues>
          <attvalue for="5" value=" LLMs have demonstrated remarkable capabilities, outperforming traditional approaches and setting new benchmark performance for various applications such as machine translation \cite{lyu2023new,zhu2023multilingual,zhang2023prompting,wang2023document}." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="2304.02210" id="575">
        <attvalues>
          <attvalue for="5" value=" LLMs have demonstrated remarkable capabilities, outperforming traditional approaches and setting new benchmark performance for various applications such as machine translation \cite{lyu2023new,zhu2023multilingual,zhang2023prompting,wang2023document}." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="2304.02426" id="576">
        <attvalues>
          <attvalue for="5" value=" Consequently, recent studies have employed LLMs for translation tasks \cite{jiao2023parrot,alves-etal-2023-steering}, achieving remarkable performance." />
        </attvalues>
      </edge>
      <edge source="2401.08350" target="2310.13448" id="577">
        <attvalues>
          <attvalue for="5" value=" Consequently, recent studies have employed LLMs for translation tasks \cite{jiao2023parrot,alves-etal-2023-steering}, achieving remarkable performance." />
        </attvalues>
      </edge>
      <edge source="2406.00832" target="2401.01879" id="579">
        <attvalues>
          <attvalue for="5" value=" This simple procedure is surprisingly effective in practice \cite{beirami2024theoretical,wang2024transforming,gao2023scaling,eisenstein2023helping}." />
        </attvalues>
      </edge>
      <edge source="2406.00832" target="2402.00742" id="580">
        <attvalues>
          <attvalue for="5" value=" This simple procedure is surprisingly effective in practice \cite{beirami2024theoretical,wang2024transforming,gao2023scaling,eisenstein2023helping}." />
        </attvalues>
      </edge>
      <edge source="2406.00832" target="2210.10760" id="581">
        <attvalues>
          <attvalue for="5" value=" This simple procedure is surprisingly effective in practice \cite{beirami2024theoretical,wang2024transforming,gao2023scaling,eisenstein2023helping}." />
        </attvalues>
      </edge>
      <edge source="2406.00832" target="2312.09244" id="582">
        <attvalues>
          <attvalue for="5" value=" This simple procedure is surprisingly effective in practice \cite{beirami2024theoretical,wang2024transforming,gao2023scaling,eisenstein2023helping}." />
        </attvalues>
      </edge>
      <edge source="2406.00832" target="2305.18290" id="583">
        <attvalues>
          <attvalue for="5" value=" Second, contrastive methods directly use the preference data to define an objective function for fine-tuning the LLM \cite{rafailov2023direct,azar2024general,ethayarajh2024kto,xu2024contrastive,hong2024reference}." />
        </attvalues>
      </edge>
      <edge source="2406.00832" target="2310.12036" id="584">
        <attvalues>
          <attvalue for="5" value=" Second, contrastive methods directly use the preference data to define an objective function for fine-tuning the LLM \cite{rafailov2023direct,azar2024general,ethayarajh2024kto,xu2024contrastive,hong2024reference}." />
        </attvalues>
      </edge>
      <edge source="2406.00832" target="2402.01306" id="585">
        <attvalues>
          <attvalue for="5" value=" Second, contrastive methods directly use the preference data to define an objective function for fine-tuning the LLM \cite{rafailov2023direct,azar2024general,ethayarajh2024kto,xu2024contrastive,hong2024reference}." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2005.14165" id="606">
        <attvalues>
          <attvalue for="5" value=" Recently, LLMs has shown impressive reasoning and generation capabilities in various NLP tasks, and these capabilities further enhance as the number of model parameters increases\cite{gpt3,chatgpt,llama1,llama2}." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2307.09288" id="609">
        <attvalues>
          <attvalue for="5" value=" Recently, LLMs has shown impressive reasoning and generation capabilities in various NLP tasks, and these capabilities further enhance as the number of model parameters increases\cite{gpt3,chatgpt,llama1,llama2}." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2305.13803" id="610">
        <attvalues>
          <attvalue for="5" value=" However, the huge computational and memory requirements are still a major obstacle to wider application, so it is necessary to compress LLMs to reduce costs~\cite{li2023auto,liu2023norm,li2022shadow,li2022SFF,li2022self,li2022tf}." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2210.17323" id="611">
        <attvalues>
          <attvalue for="5" value=" Some works for LLMs compression currently focus on model quantization\cite{quant_1, quant_2,dong2023emq}, which is the process of quantizing model parameters into low-bit level representions." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2307.10554" id="612">
        <attvalues>
          <attvalue for="5" value=" Some works for LLMs compression currently focus on model quantization\cite{quant_1, quant_2,dong2023emq}, which is the process of quantizing model parameters into low-bit level representions." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2303.00566" id="613">
        <attvalues>
          <attvalue for="5" value=" In face, another commonly used method for model compression is network pruning\cite{prune_survey}, which reduces the size of the model by deleting some unimportant weights." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2305.11627" id="614">
        <attvalues>
          <attvalue for="5" value=" Although some pruning work for LLMs has made some progress, pruning strategies usually require manual design\cite{llm_pruner, wanda, sparsegpt}." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2206.13329" id="616">
        <attvalues>
          <attvalue for="5" value=" Neural architecture search (NAS)\cite{nas,linas2,li2021nas,dong2023diswot,dong2023rd,lichengp}, which aims to automatically find neural network architecture, has been applied to model pruning due to its effectiveness and simplicity." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2303.15678" id="617">
        <attvalues>
          <attvalue for="5" value=" Neural architecture search (NAS)\cite{nas,linas2,li2021nas,dong2023diswot,dong2023rd,lichengp}, which aims to automatically find neural network architecture, has been applied to model pruning due to its effectiveness and simplicity." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2301.09850" id="618">
        <attvalues>
          <attvalue for="5" value=" Neural architecture search (NAS)\cite{nas,linas2,li2021nas,dong2023diswot,dong2023rd,lichengp}, which aims to automatically find neural network architecture, has been applied to model pruning due to its effectiveness and simplicity." />
        </attvalues>
      </edge>
      <edge source="2309.09507" target="2007.04785" id="619">
        <attvalues>
          <attvalue for="5" value=" One common approach is using an accuracy predictor to predict the accuracy of the model architecture to be pruned in the search space, which can save the cost of evaluating these candidate architectures\cite{gbdt_acc,wei2022convformer}." />
        </attvalues>
      </edge>
      <edge source="2202.07255" target="2012.15723" id="622">
        <attvalues>
          <attvalue for="5" value=" Although adapting Pre-trained Language Models (PLMs)~\cite{devlin-etal-2019-bert} to downstream NLP tasks via finetuning is the de facto mainstream paradigm under fully supervised settings~\cite{wang-etal-2018-glue}, prompting~\cite{gao-etal-2021-making,Radford2019LanguageMA,NEURIPS2020_1457c0d6,schick-schutze-2021-exploiting,schick-schutze-2021-just} has demonstrated its superiority over finetuning in low-resource scenarios." />
        </attvalues>
      </edge>
      <edge source="2202.07255" target="2005.14165" id="623">
        <attvalues>
          <attvalue for="5" value=" Although adapting Pre-trained Language Models (PLMs)~\cite{devlin-etal-2019-bert} to downstream NLP tasks via finetuning is the de facto mainstream paradigm under fully supervised settings~\cite{wang-etal-2018-glue}, prompting~\cite{gao-etal-2021-making,Radford2019LanguageMA,NEURIPS2020_1457c0d6,schick-schutze-2021-exploiting,schick-schutze-2021-just} has demonstrated its superiority over finetuning in low-resource scenarios." />
        </attvalues>
      </edge>
      <edge source="2202.07255" target="2009.07118" id="624">
        <attvalues>
          <attvalue for="5" value=" Although adapting Pre-trained Language Models (PLMs)~\cite{devlin-etal-2019-bert} to downstream NLP tasks via finetuning is the de facto mainstream paradigm under fully supervised settings~\cite{wang-etal-2018-glue}, prompting~\cite{gao-etal-2021-making,Radford2019LanguageMA,NEURIPS2020_1457c0d6,schick-schutze-2021-exploiting,schick-schutze-2021-just} has demonstrated its superiority over finetuning in low-resource scenarios." />
        </attvalues>
      </edge>
      <edge source="2202.07255" target="1904.12848" id="626">
        <attvalues>
          <attvalue for="5" value=" Our DPA framework is not task-dependent and does not require either external unlabeled data~\cite{xie2020unsupervised} or massive text manipulation efforts~\cite{wei-zou-2019-eda} compared with other data augmentation approaches." />
        </attvalues>
      </edge>
      <edge source="2202.07255" target="1901.11196" id="627">
        <attvalues>
          <attvalue for="5" value=" Our DPA framework is not task-dependent and does not require either external unlabeled data~\cite{xie2020unsupervised} or massive text manipulation efforts~\cite{wei-zou-2019-eda} compared with other data augmentation approaches." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2303.17564" id="634">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) exhibit striking capabilities on important tasks in myriad domains including medicine \cite{singhal2023towards}, finance~\cite{wu2023bloomberggpt}, science~\cite{taylor2022galactica}, and entertainment~\cite{zhong2023let}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2211.09085" id="635">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) exhibit striking capabilities on important tasks in myriad domains including medicine \cite{singhal2023towards}, finance~\cite{wu2023bloomberggpt}, science~\cite{taylor2022galactica}, and entertainment~\cite{zhong2023let}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2312.02439" id="636">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) exhibit striking capabilities on important tasks in myriad domains including medicine \cite{singhal2023towards}, finance~\cite{wu2023bloomberggpt}, science~\cite{taylor2022galactica}, and entertainment~\cite{zhong2023let}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2202.03629" id="637">
        <attvalues>
          <attvalue for="5" value=" For example, LMs generate content that is hallucinatory \cite{ji2023survey}, factually incorrect \cite{zhao2023felm}, and harmful \cite{mendelsohn2023dogwhistles,jain2024polyglotoxicityprompts,Hartvigsen2022ToxiGenAL}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2310.00741" id="638">
        <attvalues>
          <attvalue for="5" value=" For example, LMs generate content that is hallucinatory \cite{ji2023survey}, factually incorrect \cite{zhao2023felm}, and harmful \cite{mendelsohn2023dogwhistles,jain2024polyglotoxicityprompts,Hartvigsen2022ToxiGenAL}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2305.17174" id="639">
        <attvalues>
          <attvalue for="5" value=" For example, LMs generate content that is hallucinatory \cite{ji2023survey}, factually incorrect \cite{zhao2023felm}, and harmful \cite{mendelsohn2023dogwhistles,jain2024polyglotoxicityprompts,Hartvigsen2022ToxiGenAL}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2405.09373" id="640">
        <attvalues>
          <attvalue for="5" value=" For example, LMs generate content that is hallucinatory \cite{ji2023survey}, factually incorrect \cite{zhao2023felm}, and harmful \cite{mendelsohn2023dogwhistles,jain2024polyglotoxicityprompts,Hartvigsen2022ToxiGenAL}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2203.09509" id="641">
        <attvalues>
          <attvalue for="5" value=" For example, LMs generate content that is hallucinatory \cite{ji2023survey}, factually incorrect \cite{zhao2023felm}, and harmful \cite{mendelsohn2023dogwhistles,jain2024polyglotoxicityprompts,Hartvigsen2022ToxiGenAL}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2403.04317" id="642">
        <attvalues>
          <attvalue for="5" value=" For example, regulations arise \cite{USA2023}, computational resources constrict, knowledge gets outdated \cite{tack2024online}, and copyrighted training materials are identified \cite{grynbaum2023times}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2307.10169" id="643">
        <attvalues>
          <attvalue for="5" value=" Without ways to quickly address these issues, models can be left miscalibrated, outdated, and biased, limiting their widespread responsible use \cite{kaddour2023challenges}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2308.07633" id="644">
        <attvalues>
          <attvalue for="5" value=" For example, we can view model compression \cite{zhu2023survey,frantar2023gptq,frantar2023sparsegpt} as an intervention to make language models more inference- or memory-efficient." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2210.17323" id="645">
        <attvalues>
          <attvalue for="5" value=" For example, we can view model compression \cite{zhu2023survey,frantar2023gptq,frantar2023sparsegpt} as an intervention to make language models more inference- or memory-efficient." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2301.00774" id="646">
        <attvalues>
          <attvalue for="5" value=" For example, we can view model compression \cite{zhu2023survey,frantar2023gptq,frantar2023sparsegpt} as an intervention to make language models more inference- or memory-efficient." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2211.11031" id="648">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2202.05262" id="649">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2210.07229" id="650">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2312.11795" id="651">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2311.04661" id="652">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2109.07445" id="653">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2407.12824" id="655">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2402.08787" id="656">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2402.16835" id="657">
        <attvalues>
          <attvalue for="5" value=" Other rapidly-advancing examples include knowledge editing \cite{mazzia2023survey,hartvigsen2023aging,meng2022locating,meng2023mass,yu2024melo,tan2024massive}, detoxification \cite{welbl2021challenges,yu2023unlearning,wang2022exploring,suauwhispering}, and unlearning \cite{Liu2024RethinkingMU,Lynch2024EightMT,Eldan2023WhosHP}." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2305.14936" id="659">
        <attvalues>
          <attvalue for="5" value=" While some works have started studying interactions between training objectives \cite{matzken2023trade,xu2023compress,li2024loftq}, practical widespread use is limited without unified evaluations for how interventions interact." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2305.11186" id="660">
        <attvalues>
          <attvalue for="5" value=" While some works have started studying interactions between training objectives \cite{matzken2023trade,xu2023compress,li2024loftq}, practical widespread use is limited without unified evaluations for how interventions interact." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2310.08659" id="661">
        <attvalues>
          <attvalue for="5" value=" While some works have started studying interactions between training objectives \cite{matzken2023trade,xu2023compress,li2024loftq}, practical widespread use is limited without unified evaluations for how interventions interact." />
        </attvalues>
      </edge>
      <edge source="2407.06483" target="2307.09288" id="662">
        <attvalues>
          <attvalue for="5" value=" We use our framework to extensively study composability of state-of-the-art knowledge editing, model compression, and machine unlearning interventions on Llama3-8B \cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2005.14165" id="663">
        <attvalues>
          <attvalue for="5" value=" Notably, LLMs are known for their in-context learning ability, allowing them to generalize to unseen tasks without additional fine-tuning \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2305.05176" id="664">
        <attvalues>
          <attvalue for="5" value=" Despite of their remarkable adaptability, LLMs are very expensive to deploy~\cite{chen2023frugalgpt,wu2023fast}." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2307.09288" id="666">
        <attvalues>
          <attvalue for="5" value=" The inference process of LLMs, such as LLaMA 2~\cite{touvron2023llama}, may require multiple powerful GPUs, which is prohibitively expensive for the general community." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2004.10568" id="667">
        <attvalues>
          <attvalue for="5" value=" To address this problem, model compression methods are widely employed to reduce the model size and inference latency, such as quantization \cite{nagel2020up,dettmers2022llm,xiao2022smoothquant,frantar2022gptq} and pruning \cite{frantar2023sparsegpt}." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2211.10438" id="668">
        <attvalues>
          <attvalue for="5" value=" To address this problem, model compression methods are widely employed to reduce the model size and inference latency, such as quantization \cite{nagel2020up,dettmers2022llm,xiao2022smoothquant,frantar2022gptq} and pruning \cite{frantar2023sparsegpt}." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2210.17323" id="669">
        <attvalues>
          <attvalue for="5" value=" To address this problem, model compression methods are widely employed to reduce the model size and inference latency, such as quantization \cite{nagel2020up,dettmers2022llm,xiao2022smoothquant,frantar2022gptq} and pruning \cite{frantar2023sparsegpt}." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2301.00774" id="670">
        <attvalues>
          <attvalue for="5" value=" To address this problem, model compression methods are widely employed to reduce the model size and inference latency, such as quantization \cite{nagel2020up,dettmers2022llm,xiao2022smoothquant,frantar2022gptq} and pruning \cite{frantar2023sparsegpt}." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2101.00190" id="671">
        <attvalues>
          <attvalue for="5" value=" We underscore that the primary distinction between our prompt learning approach and previous prompt tuning frameworks \cite{li2021prefix, lester2021power,tang2023chain} is that earlier methods mainly utilized the prompt to adapt the model for specific downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2305.11186" target="2306.00550" id="672">
        <attvalues>
          <attvalue for="5" value=" We underscore that the primary distinction between our prompt learning approach and previous prompt tuning frameworks \cite{li2021prefix, lester2021power,tang2023chain} is that earlier methods mainly utilized the prompt to adapt the model for specific downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1907.11692" id="674">
        <attvalues>
          <attvalue for="5" value=" Transfer learning with pretrained Language Models (LMs) such as BERT \cite{devlin2019bert} and RoBERTa \cite{liu:2019roberta} offers unmatched performance in many NLP tasks \cite{Wang:2019superglue,Raffel:2019:arxiv}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1905.00537" id="675">
        <attvalues>
          <attvalue for="5" value=" Transfer learning with pretrained Language Models (LMs) such as BERT \cite{devlin2019bert} and RoBERTa \cite{liu:2019roberta} offers unmatched performance in many NLP tasks \cite{Wang:2019superglue,Raffel:2019:arxiv}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2002.12327" id="677">
        <attvalues>
          <attvalue for="5" value=" However, despite the wealth of semantic knowledge stored in the pretrained LMs \cite{Rogers:2020arxiv,Vulic:2020emnlp}, they do not produce coherent and effective sentence representations when used off-the-shelf \cite{Liu:2021emnlp}: to this effect, further specialization for sentence-level semantics -- not unlike the standard task fine-tuning -- is needed \cite[inter alia]{Reimers:2019emnlp,Li:2020emnlp,Yan:2021acl}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2010.05731" id="678">
        <attvalues>
          <attvalue for="5" value=" However, despite the wealth of semantic knowledge stored in the pretrained LMs \cite{Rogers:2020arxiv,Vulic:2020emnlp}, they do not produce coherent and effective sentence representations when used off-the-shelf \cite{Liu:2021emnlp}: to this effect, further specialization for sentence-level semantics -- not unlike the standard task fine-tuning -- is needed \cite[inter alia]{Reimers:2019emnlp,Li:2020emnlp,Yan:2021acl}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2104.08027" id="679">
        <attvalues>
          <attvalue for="5" value=" However, as we show in this work, such multilingual SEs may still lag behind traditional static cross-lingual word embeddings (CLWEs) when encoding sub-sentence lexical items (eg, words or phrases) \cite{Liu:2021emnlp} for cross-lingual lexical tasks (eg, BLI)." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1908.10084" id="680">
        <attvalues>
          <attvalue for="5" value=" LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives \cite{infonce,Musgrave:2020eccv}, in supervised (eg, leveraging labeled external data such as NLI or sentence similarity annotations) \cite{Reimers:2019emnlp,Vulic:2021emnlp,Liu:2021dialoguecse} or, more recently, fully unsupervised fine-tuning \cite{Liu:2021emnlp,Gao:2021emnlp} setups." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2011.05864" id="681">
        <attvalues>
          <attvalue for="5" value=" However, despite the wealth of semantic knowledge stored in the pretrained LMs \cite{Rogers:2020arxiv,Vulic:2020emnlp}, they do not produce coherent and effective sentence representations when used off-the-shelf \cite{Liu:2021emnlp}: to this effect, further specialization for sentence-level semantics -- not unlike the standard task fine-tuning -- is needed \cite[inter alia]{Reimers:2019emnlp,Li:2020emnlp,Yan:2021acl}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2105.11741" id="682">
        <attvalues>
          <attvalue for="5" value=" However, despite the wealth of semantic knowledge stored in the pretrained LMs \cite{Rogers:2020arxiv,Vulic:2020emnlp}, they do not produce coherent and effective sentence representations when used off-the-shelf \cite{Liu:2021emnlp}: to this effect, further specialization for sentence-level semantics -- not unlike the standard task fine-tuning -- is needed \cite[inter alia]{Reimers:2019emnlp,Li:2020emnlp,Yan:2021acl}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1807.03748" id="683">
        <attvalues>
          <attvalue for="5" value=" LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives \cite{infonce,Musgrave:2020eccv}, in supervised (eg, leveraging labeled external data such as NLI or sentence similarity annotations) \cite{Reimers:2019emnlp,Vulic:2021emnlp,Liu:2021dialoguecse} or, more recently, fully unsupervised fine-tuning \cite{Liu:2021emnlp,Gao:2021emnlp} setups." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2003.08505" id="684">
        <attvalues>
          <attvalue for="5" value=" LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives \cite{infonce,Musgrave:2020eccv}, in supervised (eg, leveraging labeled external data such as NLI or sentence similarity annotations) \cite{Reimers:2019emnlp,Vulic:2021emnlp,Liu:2021dialoguecse} or, more recently, fully unsupervised fine-tuning \cite{Liu:2021emnlp,Gao:2021emnlp} setups." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2109.10126" id="685">
        <attvalues>
          <attvalue for="5" value=" LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives \cite{infonce,Musgrave:2020eccv}, in supervised (eg, leveraging labeled external data such as NLI or sentence similarity annotations) \cite{Reimers:2019emnlp,Vulic:2021emnlp,Liu:2021dialoguecse} or, more recently, fully unsupervised fine-tuning \cite{Liu:2021emnlp,Gao:2021emnlp} setups." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2109.12599" id="686">
        <attvalues>
          <attvalue for="5" value=" LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives \cite{infonce,Musgrave:2020eccv}, in supervised (eg, leveraging labeled external data such as NLI or sentence similarity annotations) \cite{Reimers:2019emnlp,Vulic:2021emnlp,Liu:2021dialoguecse} or, more recently, fully unsupervised fine-tuning \cite{Liu:2021emnlp,Gao:2021emnlp} setups." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2104.08821" id="687">
        <attvalues>
          <attvalue for="5" value=" LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives \cite{infonce,Musgrave:2020eccv}, in supervised (eg, leveraging labeled external data such as NLI or sentence similarity annotations) \cite{Reimers:2019emnlp,Vulic:2021emnlp,Liu:2021dialoguecse} or, more recently, fully unsupervised fine-tuning \cite{Liu:2021emnlp,Gao:2021emnlp} setups." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2007.01852" id="688">
        <attvalues>
          <attvalue for="5" value=" Following the procedures from monolingual setups, another line of research has been transforming multilingual LMs into multilingual SEs \cite{Feng:2020labse,Reimers:2020emnlp}, which enable effective sentence matching and ranking in multiple languages as well as cross-lingually \cite{Litschko:2022jir}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2004.09813" id="689">
        <attvalues>
          <attvalue for="5" value=" Following the procedures from monolingual setups, another line of research has been transforming multilingual LMs into multilingual SEs \cite{Feng:2020labse,Reimers:2020emnlp}, which enable effective sentence matching and ranking in multiple languages as well as cross-lingually \cite{Litschko:2022jir}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2112.11031" id="690">
        <attvalues>
          <attvalue for="5" value=" Consequently, they outperform multilingual off-the-shelf LMs in cross-lingual sentence similarity and ranking applications \cite{Liu:2021acl,Litschko:2022jir}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1901.07291" id="691">
        <attvalues>
          <attvalue for="5" value=" The transformation is typically done by coupling 1) LM objectives on monolingual data available in multiple languages with 2) cross-lingual objectives such as Translation Language Modeling (TLM) \cite{Conneau:2019neurips} and/or cross-lingual contrastive ranking \cite{Yang:2020demos}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1907.04307" id="692">
        <attvalues>
          <attvalue for="5" value=" The transformation is typically done by coupling 1) LM objectives on monolingual data available in multiple languages with 2) cross-lingual objectives such as Translation Language Modeling (TLM) \cite{Conneau:2019neurips} and/or cross-lingual contrastive ranking \cite{Yang:2020demos}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="2105.14398" id="693">
        <attvalues>
          <attvalue for="5" value=" Consequently, they outperform multilingual off-the-shelf LMs in cross-lingual sentence similarity and ranking applications \cite{Liu:2021acl,Litschko:2022jir}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1902.00508" id="694">
        <attvalues>
          <attvalue for="5" value=" We show that the `exposure' procedure is highly effective for both vanilla multilingual LMs (mBERT and XLM-R) and multilingual SEs (\labse and \mpnet): eg, we observe $\approx$+10 Precision@1 points gains on standard BLI benchmarks \cite{Glavas:2019acl}." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1805.06297" id="695">
        <attvalues>
          <attvalue for="5" value=" Finally, inspired by \newcite{Li:2022acl}, we validate that word vectors produced by cross-lingual lexical encoders (eg, after contrastive cross-lingual lexical `exposure') can be effectively interpolated with static CLWEs \cite{Artetxe:2018acl} and offer even stronger performance in cross-lingual lexical tasks." />
        </attvalues>
      </edge>
      <edge source="2205.00267" target="1909.01638" id="696">
        <attvalues>
          <attvalue for="5" value=" Encouragingly, our cross-lingual lexical specialization of multilingual SEs (as well as the further interpolation with static CLWEs), yields particularly massive performance gains for pairs of low-resource languages, as demonstrated on the low-resource BLI benchmark \cite{Vulic:2019we}." />
        </attvalues>
      </edge>
      <edge source="2003.02245" target="1901.11196" id="697">
        <attvalues>
          <attvalue for="5" value=" In particular, ~\cite{wei2019eda} showed that simple word replacement using knowledge bases like WordNet~\cite{miller1998wordnet} improves classification performance." />
        </attvalues>
      </edge>
      <edge source="2003.02245" target="1805.06201" id="698">
        <attvalues>
          <attvalue for="5" value=" Further, ~\cite{kobayashi2018contextual} utilized language models (LM) to augment training data." />
        </attvalues>
      </edge>
      <edge source="2003.02245" target="1911.03118" id="701">
        <attvalues>
          <attvalue for="5" value=" Similarly, ~\cite{anaby2019not} used GPT2~\cite{radford2019language} for DA where examples are generated for a given class by providing class as input to a fine-tuned model." />
        </attvalues>
      </edge>
      <edge source="2003.02245" target="1706.03762" id="702">
        <attvalues>
          <attvalue for="5" value=" This paper proposes a unified approach to use any pre-trained transformer \cite{vaswani2017attention} based models for data augmentation." />
        </attvalues>
      </edge>
      <edge source="2003.02245" target="1910.13461" id="703">
        <attvalues>
          <attvalue for="5" value=" In particular, we explore three different pre-trained model types for DA, including 1) an autoencoder (AE) LM: BERT, 2) an auto-regressive (AR) LM: GPT2, and 3) a pre-trained seq2seq model: BART~\cite{lewis2019bart}." />
        </attvalues>
      </edge>
      <edge source="2406.15209" target="1809.00385" id="704">
        <attvalues>
          <attvalue for="5" value=" Previous studies primarily focused on text-based zero-shot natural language understanding (NLU) \cite{xia2018zero, bapna2017towards, shah2019robust, lee2019zero}, which processes transcripts produced by an automatic speech recognition (ASR) model to create a modular solution to zero-shot SLU." />
        </attvalues>
      </edge>
      <edge source="2406.15209" target="1904.03670" id="706">
        <attvalues>
          <attvalue for="5" value=" \item We conduct cross-corpus zero-shot evaluations to validate the generalisation of the proposed system, trained on SLURP and tested on FSC \cite{lugosch2019speech} and SmartLight \cite{saade2019spoken} datasets." />
        </attvalues>
      </edge>
      <edge source="2406.15209" target="2311.02482" id="707">
        <attvalues>
          <attvalue for="5" value=" A recent study introduces a zero-shot audio-to-intent classification framework \cite{elluru2023generalized}, identifying unseen intents by comparing the speech embedding similarity between the test utterance and enrollment samples." />
        </attvalues>
      </edge>
      <edge source="2406.15209" target="2307.01764" id="708">
        <attvalues>
          <attvalue for="5" value=" Furthermore, \cite{sun2023knowledge} presents a knowledge-aware audio-grounded (KA2G) generative framework for zero-shot slot filling." />
        </attvalues>
      </edge>
      <edge source="2406.15209" target="2011.13205" id="710">
        <attvalues>
          <attvalue for="5" value=" Our contributions are summarised as follows: \begin{itemize} \item We investigate the application of the Whisper model for zero-shot E2E SLU, and show that it significantly outperforms the existing baseline on the SLURP benchmark \cite{bastianelli2020slurp}." />
        </attvalues>
      </edge>
      <edge source="2406.15209" target="1810.12735" id="711">
        <attvalues>
          <attvalue for="5" value=" \item We conduct cross-corpus zero-shot evaluations to validate the generalisation of the proposed system, trained on SLURP and tested on FSC \cite{lugosch2019speech} and SmartLight \cite{saade2019spoken} datasets." />
        </attvalues>
      </edge>
      <edge source="2406.06563" target="2101.03961" id="713">
        <attvalues>
          <attvalue for="5" value=" In response to these challenges, sparse models, such as Mixture-of-Experts (MoE), have gained prominence \cite{switch_transformer, gshard, glam, deepseekmoev1, deepseekmoev2}." />
        </attvalues>
      </edge>
      <edge source="2406.06563" target="2112.10684" id="714">
        <attvalues>
          <attvalue for="5" value=" These models offer a more economically viable alternative by distributing computation across various specialized sub-models or ``experts'', potentially matching or even surpassing the performance of their dense counterparts with a fraction of the resource requirements \cite{artetxe2022efficient, deepspeedmoe, moe_scaling_law}." />
        </attvalues>
      </edge>
      <edge source="2406.06563" target="2310.19341" id="715">
        <attvalues>
          <attvalue for="5" value=" Moreover, the training of Skywork-MoE was conducted on a condensed subset of the SkyPile corpus \cite{wei2023skywork}, with subsequent evaluations demonstrating its robust performance across a diverse array of benchmarks." />
        </attvalues>
      </edge>
      <edge source="2406.06563" target="2212.05055" id="716">
        <attvalues>
          <attvalue for="5" value=" This model leverages the foundational architecture of our previously developed Skywork-13B model \cite{wei2023skywork}, utilizing its dense checkpoints as the initial setup \cite{upcycling}." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="1706.03762" id="717">
        <attvalues>
          <attvalue for="5" value=" \jamba is based on a novel hybrid architecture, which combines Transformer layers \cite{vaswani2017attention} with Mamba layers \cite{gu2023mamba}, a recent state-space model \cite{gu2021combining,gu2021efficiently}, as well as a mixture-of-experts (MoE) module \cite{shazeer2016outrageously,fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2312.00752" id="718">
        <attvalues>
          <attvalue for="5" value=" However, as shown in \cite{gu2023mamba}, its perfomance lags that of pure Mamba." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="1701.06538" id="720">
        <attvalues>
          <attvalue for="5" value=" \jamba also includes MoE layers \cite{shazeer2016outrageously,fedus2022switch}, which allow increasing the model capacity (total number of available parameters) without increasing compute requirements (number of active parameters)." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2101.03961" id="721">
        <attvalues>
          <attvalue for="5" value=" \jamba also includes MoE layers \cite{shazeer2016outrageously,fedus2022switch}, which allow increasing the model capacity (total number of available parameters) without increasing compute requirements (number of active parameters)." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2212.08136" id="722">
        <attvalues>
          <attvalue for="5" value=" \cite{zuo2022efficient} mixes an S4 layer \cite{gu2021efficiently} with a local attention layer, followed by a sequence of local attention layers; it shows experiments with small models and simple tasks." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2306.09539" id="723">
        <attvalues>
          <attvalue for="5" value=" \cite{pilault2023block} starts with an SSM layer followed by chunk-based Transformers, with models up to 1." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2305.12498" id="724">
        <attvalues>
          <attvalue for="5" value=" \cite{fathullah23_interspeech} adds an SSM layer before the self-attention in a Transformer layer, while \cite{saon2023diagonal} adds the SSM after the self-attention, both showing improvements on speech recognition." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2302.14120" id="725">
        <attvalues>
          <attvalue for="5" value=" \cite{fathullah23_interspeech} adds an SSM layer before the self-attention in a Transformer layer, while \cite{saon2023diagonal} adds the SSM after the self-attention, both showing improvements on speech recognition." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2402.04248" id="726">
        <attvalues>
          <attvalue for="5" value=" \cite{park2024can} replaces the MLP layers in the Transformer by Mamba layers, and shows benefits in simple tasks." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2212.14052" id="727">
        <attvalues>
          <attvalue for="5" value=" Closest are perhaps H3 \cite{fu2022hungry}, a specially designed SSM that enables induction capabilities, and a generalization called Hyena \cite{poli2023hyena}." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2302.10866" id="728">
        <attvalues>
          <attvalue for="5" value=" Closest are perhaps H3 \cite{fu2022hungry}, a specially designed SSM that enables induction capabilities, and a generalization called Hyena \cite{poli2023hyena}." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2310.06825" id="729">
        <attvalues>
          <attvalue for="5" value=" However, it lags behind the Attention-only Mistral-7B \cite{jiang2023mistral}." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2401.04088" id="730">
        <attvalues>
          <attvalue for="5" value=" We evaluated our implementation of \jamba on a wide range of benchmarks and found it performs comparably to Mixtral-8x7B \cite{jiang2024mixtral}, which has a similar number of parameters, and also to the larger Llama-2 70B \cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2403.19887" target="2307.09288" id="731">
        <attvalues>
          <attvalue for="5" value=" We evaluated our implementation of \jamba on a wide range of benchmarks and found it performs comparably to Mixtral-8x7B \cite{jiang2024mixtral}, which has a similar number of parameters, and also to the larger Llama-2 70B \cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2109.01207" target="1911.02116" id="733">
        <attvalues>
          <attvalue for="5" value=" Multilingual Language Models such as mBERT \cite{devlin-etal-2019-bert} or XLM-R (XLM-Roberta; \cite{conneau-etal-2020a-unsupervised} achieve remarkable results on a variety of cross-lingual transfer tasks \cite{hu2020extreme, lian2020xglue}." />
        </attvalues>
      </edge>
      <edge source="2109.01207" target="2003.11080" id="734">
        <attvalues>
          <attvalue for="5" value=" Multilingual Language Models such as mBERT \cite{devlin-etal-2019-bert} or XLM-R (XLM-Roberta; \cite{conneau-etal-2020a-unsupervised} achieve remarkable results on a variety of cross-lingual transfer tasks \cite{hu2020extreme, lian2020xglue}." />
        </attvalues>
      </edge>
      <edge source="2109.01207" target="2101.11109" id="735">
        <attvalues>
          <attvalue for="5" value=" Specifically, we consider the following research questions: \begin{enumerate} \item Which cross-lingual pattern representational similarity suggests in the final analysis? (Answer: convergence pattern from \cite{muller-etal-2021-first})." />
        </attvalues>
      </edge>
      <edge source="2109.01207" target="1911.01464" id="736">
        <attvalues>
          <attvalue for="5" value=" \cite{muller-etal-2021-first} backs up their representational analysis using probing task in the layer-wise ablation setting; \cite{conneau-etal-2020b-emerging} also directly supports this conclusion." />
        </attvalues>
      </edge>
      <edge source="2101.11109" target="1706.04902" id="737">
        <attvalues>
          <attvalue for="5" value=" al study only the mononlingual case}} \cite{ruder2019survey}: (1)~Building a shared multilingual representation of text, typically by aligning textual representations across languages." />
        </attvalues>
      </edge>
      <edge source="2101.11109" target="1309.4168" id="738">
        <attvalues>
          <attvalue for="5" value=" This mechanism that emerges out-of-the-box, without any explicit supervision, suggests that mBERT behaves like the standard cross-lingual pipeline\draftnote{We had those references before \cite{aone1993language,schultz2001language,zeman2008cross,sogaard2011data,mikolov2013exploiting,smith2017offline,cao2020multilingual} should we put them back? at least one or two -ds \bm{we've cited all of then at the begining, wouldn't it be too much to cite them again here?}}." />
        </attvalues>
      </edge>
      <edge source="2101.11109" target="1702.03859" id="739">
        <attvalues>
          <attvalue for="5" value=" This mechanism that emerges out-of-the-box, without any explicit supervision, suggests that mBERT behaves like the standard cross-lingual pipeline\draftnote{We had those references before \cite{aone1993language,schultz2001language,zeman2008cross,sogaard2011data,mikolov2013exploiting,smith2017offline,cao2020multilingual} should we put them back? at least one or two -ds \bm{we've cited all of then at the begining, wouldn't it be too much to cite them again here?}}." />
        </attvalues>
      </edge>
      <edge source="2101.11109" target="1901.07291" id="741">
        <attvalues>
          <attvalue for="5" value=" Multilingual pretrained language models \cite{devlin-etal-2019-bert,conneau2019cross} have been shown to perform efficient zero-shot cross-lingual transfer for many tasks and languages \cite{pires-etal-2019-multilingual,wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2101.11109" target="1906.01502" id="742">
        <attvalues>
          <attvalue for="5" value=" \cite{pires-etal-2019-multilingual} hypothesize that these models learn shared multilingual representations during pretraining." />
        </attvalues>
      </edge>
      <edge source="2101.11109" target="1904.09077" id="743">
        <attvalues>
          <attvalue for="5" value=" Multilingual pretrained language models \cite{devlin-etal-2019-bert,conneau2019cross} have been shown to perform efficient zero-shot cross-lingual transfer for many tasks and languages \cite{pires-etal-2019-multilingual,wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2101.11109" target="2005.04511" id="744">
        <attvalues>
          <attvalue for="5" value=" Focusing on syntax, \cite{chi-etal-2020-finding} recently showed that the multilingual version of BERT (mBERT) \cite{devlin-etal-2019-bert}, encodes linguistic properties in shared multilingual sub-spaces." />
        </attvalues>
      </edge>
      <edge source="2305.10610" target="2205.05092" id="770">
        <attvalues>
          <attvalue for="5" value=" On the other hand, \cite{zhou-etal-2022-problems} studied the impact of frequency on contextualised word embeddings and showed that the cosine similarity between highly frequent words are systematically underestimated." />
        </attvalues>
      </edge>
      <edge source="2305.10610" target="1810.04882" id="773">
        <attvalues>
          <attvalue for="5" value=" This often leads to unreliable similarity estimations between words and has undesirable implications in downstream tasks such as the detection of analogies and social biases~\cite{ethayarajh-etal-2019-towards, ethayarajh2019understanding}." />
        </attvalues>
      </edge>
      <edge source="2305.10610" target="2104.08465" id="774">
        <attvalues>
          <attvalue for="5" value=" \cite{zhou2021frequency} show that the diversity (measured by the volume of the bounding hypersphere) of the contextualised embeddings of a target word, computed from multiple contexts containing the word, increases with the frequency of that word." />
        </attvalues>
      </edge>
      <edge source="2305.10610" target="1502.03520" id="775">
        <attvalues>
          <attvalue for="5" value=" In prior work, the $\ell_2$ norm of a static word embedding has been shown to linearly correlate with the log-frequency of that word~\cite{arora-etal-2016-latent,Bollegala:AAAI:2018}." />
        </attvalues>
      </edge>
      <edge source="2305.10610" target="1709.01199" id="776">
        <attvalues>
          <attvalue for="5" value=" In prior work, the $\ell_2$ norm of a static word embedding has been shown to linearly correlate with the log-frequency of that word~\cite{arora-etal-2016-latent,Bollegala:AAAI:2018}." />
        </attvalues>
      </edge>
      <edge source="2305.10610" target="1910.06403" id="777">
        <attvalues>
          <attvalue for="5" value=" Specifically, we use Monte-Carlo Bayesian Optimisation~\cite{botorch} to find the optimal discounting parameters." />
        </attvalues>
      </edge>
      <edge source="2104.08465" target="1810.04882" id="779">
        <attvalues>
          <attvalue for="5" value=" For example, word frequency is known to affect the similarity between static embeddings, and is partially responsible for the existence of word analogies and bias in word2vec \cite{ethayarajh-etal-2019-towards, ethayarajh2019understanding}." />
        </attvalues>
      </edge>
      <edge source="2406.14909" target="2005.14165" id="781">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) exhibit remarkable versatility across numerous applications~\cite{brown2020gpt3, tay2022efficientSurvey, Wan2023EfficientLLM}." />
        </attvalues>
      </edge>
      <edge source="2406.14909" target="1706.03762" id="782">
        <attvalues>
          <attvalue for="5" value=" Central to LLM is the attention mechanism~\cite{vaswani2017attention}, which computes interactions among tokens within a certain span, thereby enabling context understanding." />
        </attvalues>
      </edge>
      <edge source="2406.14909" target="2306.15595" id="783">
        <attvalues>
          <attvalue for="5" value=" Following previous works~\cite{Chen2023PI, Tworkowski2023FocusedTrans}, we quantify the effective context length as the maximum input length where content retrieval accuracy exceeds a 90\% threshold." />
        </attvalues>
      </edge>
      <edge source="2406.14909" target="2308.14508" id="784">
        <attvalues>
          <attvalue for="5" value=" Scaling input length is crucial for enhancing LLM capabilities~\cite{Chen2023PI, Tworkowski2023FocusedTrans}, including fact retrieval, summarization, few-shot learning, question answering and so on~\cite{bai2023longbench, yuan2024lveval}." />
        </attvalues>
      </edge>
      <edge source="2406.14909" target="2210.11794" id="786">
        <attvalues>
          <attvalue for="5" value=" In principle, fixed-span local attention can gradually aggregate global information through multiple model layers, yielding a longer effective context length than each attention span~\cite{feng2022diffuser, zaheer2020bigbird}." />
        </attvalues>
      </edge>
      <edge source="2406.14909" target="2403.03853" id="787">
        <attvalues>
          <attvalue for="5" value=" Besides, existing model compression methods~\cite{Men2024ShortGPT, awq, smooth_quant, llm-mq, squeezellm, li2024evaluating} use general language modeling corpora to decide the compression plan, which cannot accurately profile the influence of compression on long-context tasks." />
        </attvalues>
      </edge>
      <edge source="2210.05839" target="2203.14960" id="789">
        <attvalues>
          <attvalue for="5" value=" Recent works on fine-grained error analysis, such as Domino~\cite{eyuboglu2022domino} and Spotlight~\cite{d2022spotlight} provide solutions to this problem but focus on image datasets which are easier to visualize." />
        </attvalues>
      </edge>
      <edge source="2210.05839" target="2107.00758" id="790">
        <attvalues>
          <attvalue for="5" value=" Recent works on fine-grained error analysis, such as Domino~\cite{eyuboglu2022domino} and Spotlight~\cite{d2022spotlight} provide solutions to this problem but focus on image datasets which are easier to visualize." />
        </attvalues>
      </edge>
      <edge source="2210.05839" target="2202.06523" id="791">
        <attvalues>
          <attvalue for="5" value=" Model evaluation should ideally give actionable insights into a model's performance on a dataset in the form of data curation~\cite{liang2022metashift} or model patching~\cite{goel-etal-2021-goodwill}." />
        </attvalues>
      </edge>
      <edge source="2302.06716" target="1911.00650" id="792">
        <attvalues>
          <attvalue for="5" value=" The resulting models write so fluently that even humans trained to detect machine-generated content can be fooled about 30\% of the time \cite{ippolito-etal-2020-automatic}." />
        </attvalues>
      </edge>
      <edge source="2302.06716" target="2108.07258" id="793">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, the culture in the natural language processing (NLP) community has moved toward open sourcing these `foundation models' with permissive licenses, allowing much smaller and less-resourced entities to deploy them, often after further training them to specialize them to a different domain or task (fine-tuning)\cite{foundationmodels}." />
        </attvalues>
      </edge>
      <edge source="2302.06716" target="2112.04359" id="794">
        <attvalues>
          <attvalue for="5" value=" Since publicly released, trained LLMs can be fine-tuned at a relatively low cost and generate specialized synthetic text at scale, they may empower more actors to engage in malicious uses, for example through cheaper and more effective disinformation \cite{deepmindethical, infop}." />
        </attvalues>
      </edge>
      <edge source="2302.06716" target="2301.04246" id="795">
        <attvalues>
          <attvalue for="5" value=" Possible mitigations for this new threat include ensuring that synthetic text can be identified as non-human and its origins traced \cite{infop}." />
        </attvalues>
      </edge>
      <edge source="2302.06716" target="1705.10742" id="796">
        <attvalues>
          <attvalue for="5" value=" Incorporating a watermark into LLM output has been proposed as a safeguard to privacy and intellectual property \cite{fang-etal-2017-generating, aaronson_2022}, but is not widely practiced." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="1909.01066" id="797">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated a wide range of capabilities, such as world knowledge storage, sophisticated language-based reasoning, and in-context learning~\cite{petroni2019language, wei2022chain,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2005.14165" id="798">
        <attvalues>
          <attvalue for="5" value=" {Lastly,} LLMs show unique properties, such as expressing confidence in words~\cite{lin2022teaching,xiong2023can} and the ability to perform zero-shot or few-shot learning~\cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2310.07521" id="799">
        <attvalues>
          <attvalue for="5" value=" While existing surveys mainly focused on issues such as hallucination and factuality in LLMs~\cite{zhang2023siren,wang2023survey}, there are no comprehensive surveys systematically discussing the technical advancements in LLMs, and here we aim to bridge this gap." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2309.01219" id="800">
        <attvalues>
          <attvalue for="5" value=" While existing surveys mainly focused on issues such as hallucination and factuality in LLMs~\cite{zhang2023siren,wang2023survey}, there are no comprehensive surveys systematically discussing the technical advancements in LLMs, and here we aim to bridge this gap." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2102.09690" id="801">
        <attvalues>
          <attvalue for="5" value=" Their generation still includes biases~\cite{zhao2021calibrate,wang2023reducing} and hallucinations that do not align with reality~\cite{zhang2023siren}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2308.05374" id="802">
        <attvalues>
          <attvalue for="5" value=" Evaluating the trustworthiness of responses from these models remains challenging~\cite{liu2023trustworthy}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="1802.04865" id="806">
        <attvalues>
          <attvalue for="5" value=" Confidence (or uncertainty) estimation is crucial for tasks like out-of-distribution detection and selective prediction~\cite{DBLP:conf/nips/KendallG17,lu2022learning}, and it has been extensively studied and applied in various contexts ~\cite{lee2018simple,devries2018learning}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="1706.04599" id="807">
        <attvalues>
          <attvalue for="5" value=" A related concept is that of model calibration, which focuses on aligning predictive probabilities (estimated confidence) to actual accuracy~\cite{Guo2017OnCO}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2306.13063" id="809">
        <attvalues>
          <attvalue for="5" value=" {Lastly,} LLMs show unique properties, such as expressing confidence in words~\cite{lin2022teaching,xiong2023can} and the ability to perform zero-shot or few-shot learning~\cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2012.00955" id="810">
        <attvalues>
          <attvalue for="5" value=" Given this, confidence estimation and calibration for LLMs is growing as an emerging area of interest~\cite{jiang2021can,lin2022teaching,lin2023generating,shrivastava2023llamas}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2305.19187" id="811">
        <attvalues>
          <attvalue for="5" value=" Given this, confidence estimation and calibration for LLMs is growing as an emerging area of interest~\cite{jiang2021can,lin2022teaching,lin2023generating,shrivastava2023llamas}." />
        </attvalues>
      </edge>
      <edge source="2311.08298" target="2311.08877" id="812">
        <attvalues>
          <attvalue for="5" value=" Given this, confidence estimation and calibration for LLMs is growing as an emerging area of interest~\cite{jiang2021can,lin2022teaching,lin2023generating,shrivastava2023llamas}." />
        </attvalues>
      </edge>
      <edge source="2405.10276" target="2005.14165" id="821">
        <attvalues>
          <attvalue for="5" value=" Advancements in large language models (LLMs) have catalyzed a shift towards prompting-based learning, distinguishing models with capacities exceeding 100 billion parameters for their few-shot learning abilities without extensive retraining~\cite{brown-2020-fewshotlearners}." />
        </attvalues>
      </edge>
      <edge source="2405.10276" target="2103.10385" id="822">
        <attvalues>
          <attvalue for="5" value=" In-context learning, facilitated through the strategic use of prompts, enables these models to generate task-specific responses, marking a departure from traditional pre-train and fine-tune approaches~\cite{Liu2021GPTUT, Wan2023EfficientLL}." />
        </attvalues>
      </edge>
      <edge source="2405.10276" target="2211.01910" id="823">
        <attvalues>
          <attvalue for="5" value=" While initially dependent on manual prompt creation, recent developments in automated prompt engineering, such as APE~\cite{Zhou2022LargeLM} and APO~\cite{Pryzant2023AutomaticPO}, leverage LLMs for dynamic prompt generation and refinement." />
        </attvalues>
      </edge>
      <edge source="2405.10276" target="2305.03495" id="824">
        <attvalues>
          <attvalue for="5" value=" While initially dependent on manual prompt creation, recent developments in automated prompt engineering, such as APE~\cite{Zhou2022LargeLM} and APO~\cite{Pryzant2023AutomaticPO}, leverage LLMs for dynamic prompt generation and refinement." />
        </attvalues>
      </edge>
      <edge source="2405.10276" target="2309.03409" id="825">
        <attvalues>
          <attvalue for="5" value=" Building on this, the proposition of LLMs as optimizers~\cite{yang-2023-llmasoptimizer, Guo2023ConnectingLL} presents the current state-of-the-art in automated prompt design, framing prompt refinement as an optimization challenge." />
        </attvalues>
      </edge>
      <edge source="2405.10276" target="2307.09288" id="826">
        <attvalues>
          <attvalue for="5" value=" However, our empirical results reveal that smaller-scale LLMs like LLaMa-2~\cite{Touvron2023Llama2O} do not have sufficient ability to support the self-optimization." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2304.01852" id="827">
        <attvalues>
          <attvalue for="5" value=" Training LLMs that can serve as alternatives to ChatGPT, or domain-specific LLMs, has become highly necessary \cite{dai2023auggpt,liu2023deidgpt,ma2023impressiongpt,liao2023differentiate,dai2023adautogpt,liu2023summary,guan2023cohortgpt,liu2023pharmacygpt}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2304.14670" id="828">
        <attvalues>
          <attvalue for="5" value=" Language modeling (LM) is a fundamental approach for achieving cognitive intelligence in the field of natural language processing (NLP), and its progress has been notable in recent years \cite{liu2023summary,wang2023prompt,zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2303.18223" id="829">
        <attvalues>
          <attvalue for="5" value=" Language modeling (LM) is a fundamental approach for achieving cognitive intelligence in the field of natural language processing (NLP), and its progress has been notable in recent years \cite{liu2023summary,wang2023prompt,zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2307.10169" id="830">
        <attvalues>
          <attvalue for="5" value=" It assumes a central role in understanding, generating, and manipulating human language, serving as the cornerstone for a diverse range of NLP applications \cite{kaddour2023challenges}, including machine translation, chatbots, sentiment analysis, and text summarization." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="1802.05365" id="831">
        <attvalues>
          <attvalue for="5" value=" Early attempts at PLMs included ELMo \cite{ELMo}, which was built on a Bidirectional LSTM architecture." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="1706.03762" id="832">
        <attvalues>
          <attvalue for="5" value=" However, with the advent of the transformer architecture \cite{transformer}, characterized by parallel self-attention mechanisms, the pre-training and fine-tuning learning paradigm has propelled PLM to prominence as the prevailing approach." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2005.14165" id="833">
        <attvalues>
          <attvalue for="5" value=" A prominent milestone in the development of LLMs is exemplified by the GPT series \cite{GPT,GPT-2,GPT3,openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2307.09288" id="835">
        <attvalues>
          <attvalue for="5" value=" Many studies have pushed the boundaries of model performance by continuously expanding the scale of PLM \cite{GPT-2,GPT3,touvron2023llama,touvron2023llama2}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2304.11107" id="838">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) with significantly larger parameter sizes and extensive training data are typically denoted as Large Language Models (LLMs) \cite{zhong2023chatradiovaluer,liu2023evaluating,zhong2023chatabl}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2303.08774" id="839">
        <attvalues>
          <attvalue for="5" value=" A prominent milestone in the development of LLMs is exemplified by the GPT series \cite{GPT,GPT-2,GPT3,openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2304.11567" id="842">
        <attvalues>
          <attvalue for="5" value=" Training LLMs that can serve as alternatives to ChatGPT, or domain-specific LLMs, has become highly necessary \cite{dai2023auggpt,liu2023deidgpt,ma2023impressiongpt,liao2023differentiate,dai2023adautogpt,liu2023summary,guan2023cohortgpt,liu2023pharmacygpt}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2306.10095" id="843">
        <attvalues>
          <attvalue for="5" value=" Training LLMs that can serve as alternatives to ChatGPT, or domain-specific LLMs, has become highly necessary \cite{dai2023auggpt,liu2023deidgpt,ma2023impressiongpt,liao2023differentiate,dai2023adautogpt,liu2023summary,guan2023cohortgpt,liu2023pharmacygpt}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2307.11346" id="844">
        <attvalues>
          <attvalue for="5" value=" Training LLMs that can serve as alternatives to ChatGPT, or domain-specific LLMs, has become highly necessary \cite{dai2023auggpt,liu2023deidgpt,ma2023impressiongpt,liao2023differentiate,dai2023adautogpt,liu2023summary,guan2023cohortgpt,liu2023pharmacygpt}." />
        </attvalues>
      </edge>
      <edge source="2401.02038" target="2307.10432" id="845">
        <attvalues>
          <attvalue for="5" value=" Training LLMs that can serve as alternatives to ChatGPT, or domain-specific LLMs, has become highly necessary \cite{dai2023auggpt,liu2023deidgpt,ma2023impressiongpt,liao2023differentiate,dai2023adautogpt,liu2023summary,guan2023cohortgpt,liu2023pharmacygpt}." />
        </attvalues>
      </edge>
      <edge source="2312.04691" target="2303.18223" id="846">
        <attvalues>
          <attvalue for="5" value=" Given their rapidly evolving capabilities, LLMs and their application have become a focused topic of research within NLP academia \cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2312.04691" target="1810.08398" id="847">
        <attvalues>
          <attvalue for="5" value=" This balance is typically achieved via a fixed or adaptive read-write schedule, with one of the most popular and longstanding fixed schedules being the wait-k policy \cite{stacl2019}, where the target translation hypothesis lags behind the incrementally available source sequence by k words or subwords." />
        </attvalues>
      </edge>
      <edge source="2310.19349" target="2104.08821" id="848">
        <attvalues>
          <attvalue for="5" value=" Among them, SimCSE~\cite{SimCSE} is a pioneering work of contrastive sentence embeddings and offers techniques for both unsupervised and supervised settings." />
        </attvalues>
      </edge>
      <edge source="2310.19349" target="1508.05326" id="849">
        <attvalues>
          <attvalue for="5" value=" In the supervised approach, it utilizes the Natural Language Inference (NLI) dataset, such as the Stanford NLI (SNLI) dataset~\cite{SNLI} and the Multi-Genre NLI (MNLI) dataset~\cite{MNLI}, to treat semantically similar sentences as positive samples for contrastive learning." />
        </attvalues>
      </edge>
      <edge source="2310.19349" target="1704.05426" id="850">
        <attvalues>
          <attvalue for="5" value=" In the supervised approach, it utilizes the Natural Language Inference (NLI) dataset, such as the Stanford NLI (SNLI) dataset~\cite{SNLI} and the Multi-Genre NLI (MNLI) dataset~\cite{MNLI}, to treat semantically similar sentences as positive samples for contrastive learning." />
        </attvalues>
      </edge>
      <edge source="2310.19349" target="2204.10298" id="851">
        <attvalues>
          <attvalue for="5" value=" SimCSE has arguably become the de facto standard for sentence embeddings, demonstrating wide-ranging and impressive performance and leading to numerous derivative studies~\cite{DiffCSE,PromptBERT,PromptEOL}." />
        </attvalues>
      </edge>
      <edge source="2310.19349" target="2201.04337" id="852">
        <attvalues>
          <attvalue for="5" value=" SimCSE has arguably become the de facto standard for sentence embeddings, demonstrating wide-ranging and impressive performance and leading to numerous derivative studies~\cite{DiffCSE,PromptBERT,PromptEOL}." />
        </attvalues>
      </edge>
      <edge source="2310.19349" target="2307.16645" id="853">
        <attvalues>
          <attvalue for="5" value=" SimCSE has arguably become the de facto standard for sentence embeddings, demonstrating wide-ranging and impressive performance and leading to numerous derivative studies~\cite{DiffCSE,PromptBERT,PromptEOL}." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="2305.11206" id="854">
        <attvalues>
          <attvalue for="5" value=" One of its great challenges is to optimally interface information that pre-trained language models accumulate in their parameters and adapt it to the task of interest \cite{zhou2023lima, ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="2005.14165" id="855">
        <attvalues>
          <attvalue for="5" value=" ICL can be subdivided into few-shot \cite[][]{brown2020language} or zero-shot inference \cite[primarily using instruction-tuned models ][]{wei2021finetuned}." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="2109.01652" id="856">
        <attvalues>
          <attvalue for="5" value=" ICL can be subdivided into few-shot \cite[][]{brown2020language} or zero-shot inference \cite[primarily using instruction-tuned models ][]{wei2021finetuned}." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="2302.04023" id="857">
        <attvalues>
          <attvalue for="5" value=" However, ICL also currently yields overall weaker performance compared to task-tuning and is less stable and reliable on many benchmarks \cite[see, eg][]{bang2023multitask, ohmer2023evaluating, min2022rethinking,lu2021fantastically,zhao2021calibrate}." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="2104.08786" id="858">
        <attvalues>
          <attvalue for="5" value=" However, ICL also currently yields overall weaker performance compared to task-tuning and is less stable and reliable on many benchmarks \cite[see, eg][]{bang2023multitask, ohmer2023evaluating, min2022rethinking,lu2021fantastically,zhao2021calibrate}." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="2102.09690" id="859">
        <attvalues>
          <attvalue for="5" value=" However, ICL also currently yields overall weaker performance compared to task-tuning and is less stable and reliable on many benchmarks \cite[see, eg][]{bang2023multitask, ohmer2023evaluating, min2022rethinking,lu2021fantastically,zhao2021calibrate}." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="2210.12517" id="860">
        <attvalues>
          <attvalue for="5" value=" Since ICL is more constrained (less data and no parameter updates), out-of-distribution generalisation has been suggested to be less of a problem \cite{awadalla2022exploring, si2023prompting}." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="2303.03846" id="862">
        <attvalues>
          <attvalue for="5" value=" Little is known, however, about how these factors interact \cite[work from ][suggests that they cannot be isolated]{wei2023larger, kim2022ground}; it is unclear which aspects are consistently beneficial, which vary across setups, and which are sensible to combine or decouple." />
        </attvalues>
      </edge>
      <edge source="2310.13486" target="1911.00225" id="863">
        <attvalues>
          <attvalue for="5" value=" We start with one of the prominent themes in robustness studies for TT models: robustness to spurious correlations between input and label distributions \cite{kavumba2019choosing, mccoy2019right, niven2019probing} and find that in ICL, spurious correlations do not have a significant impact on learning outcomes." />
        </attvalues>
      </edge>
      <edge source="2304.10158" target="2105.07400" id="864">
        <attvalues>
          <attvalue for="5" value=" This has been observed on, eg, informally written Algerian Arabic \cite{touileb-barnes-2021-interplay}." />
        </attvalues>
      </edge>
      <edge source="2304.10158" target="2109.06772" id="865">
        <attvalues>
          <attvalue for="5" value=" Our contributions are: \begin{itemize} \item We investigate the noise injection method by \cite{aepli-sennrich-2022-improving} with respect to the ideal noise injection rate for different languages and PLMs." />
        </attvalues>
      </edge>
      <edge source="2304.10158" target="1904.09077" id="866">
        <attvalues>
          <attvalue for="5" value=" The importance of token overlap between source and target is an on-going debate (to which we contribute): Prior research has found that subword token overlap between the finetuning and target language improves transfer \cite{wu-dredze-2019-beto, pires-etal-2019-multilingual}, although it might neither be the most important factor \cite{k2020mbert-analysis, muller2022languages} nor a necessary condition for cross-lingual transfer to work \cite{pires-etal-2019-multilingual, conneau-etal-2020-emerging}." />
        </attvalues>
      </edge>
      <edge source="2304.10158" target="1912.07840" id="867">
        <attvalues>
          <attvalue for="5" value=" The importance of token overlap between source and target is an on-going debate (to which we contribute): Prior research has found that subword token overlap between the finetuning and target language improves transfer \cite{wu-dredze-2019-beto, pires-etal-2019-multilingual}, although it might neither be the most important factor \cite{k2020mbert-analysis, muller2022languages} nor a necessary condition for cross-lingual transfer to work \cite{pires-etal-2019-multilingual, conneau-etal-2020-emerging}." />
        </attvalues>
      </edge>
      <edge source="2304.10158" target="1906.01502" id="868">
        <attvalues>
          <attvalue for="5" value=" The importance of token overlap between source and target is an on-going debate (to which we contribute): Prior research has found that subword token overlap between the finetuning and target language improves transfer \cite{wu-dredze-2019-beto, pires-etal-2019-multilingual}, although it might neither be the most important factor \cite{k2020mbert-analysis, muller2022languages} nor a necessary condition for cross-lingual transfer to work \cite{pires-etal-2019-multilingual, conneau-etal-2020-emerging}." />
        </attvalues>
      </edge>
      <edge source="2004.13195" target="1805.11653" id="873">
        <attvalues>
          <attvalue for="5" value=" For example, they can recall more history in natural language data than in similarly Zipfian-distributed n-gram data, implying that they exploit long-distance dependencies \cite{liu_lstms_2018}." />
        </attvalues>
      </edge>
      <edge source="2004.13195" target="1805.04218" id="874">
        <attvalues>
          <attvalue for="5" value=" Their internal representations seem to be hierarchical in nature \cite{blevins_deep_2018,hupkes_visualisation_2017}." />
        </attvalues>
      </edge>
      <edge source="2004.13195" target="1711.10203" id="875">
        <attvalues>
          <attvalue for="5" value=" Their internal representations seem to be hierarchical in nature \cite{blevins_deep_2018,hupkes_visualisation_2017}." />
        </attvalues>
      </edge>
      <edge source="2004.13195" target="1704.08352" id="877">
        <attvalues>
          <attvalue for="5" value=" They seemingly encode knowledge of part of speech \cite{belinkov_what_2017}, morphological productivity \cite{Vania2017FromCT}, and verb agreement \cite{lakretz_emergence_2019}." />
        </attvalues>
      </edge>
      <edge source="2004.13195" target="1903.07435" id="878">
        <attvalues>
          <attvalue for="5" value=" They seemingly encode knowledge of part of speech \cite{belinkov_what_2017}, morphological productivity \cite{Vania2017FromCT}, and verb agreement \cite{lakretz_emergence_2019}." />
        </attvalues>
      </edge>
      <edge source="2004.13195" target="1801.05453" id="879">
        <attvalues>
          <attvalue for="5" value=" Our method builds on Contextual Decomposition~\cite[CD; ][]{murdoch_beyond_2018}, a tool for analyzing the representations produced by LSTMs." />
        </attvalues>
      </edge>
      <edge source="2011.13220" target="1512.02595" id="880">
        <attvalues>
          <attvalue for="5" value=" Language model (LM) is a core elements in natural language processing (NLP) applications, eg, language modeling \cite{bengio2003neural,mikolov2011extensions}, machine translation~\cite{cho2014learning}, speech recognition~\cite{amodei2016deep}, and dialogue generation." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="1706.03762" id="881">
        <attvalues>
          <attvalue for="5" value=" In order to improve the performance, a varieties of Transformer-based modifications have been proposed since 2017\cite{ref1}, but many of them exhibit a lack of generalization across different implementations and tasks\cite{ref27}." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="2102.11972" id="882">
        <attvalues>
          <attvalue for="5" value=" In order to improve the performance, a varieties of Transformer-based modifications have been proposed since 2017\cite{ref1}, but many of them exhibit a lack of generalization across different implementations and tasks\cite{ref27}." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="2001.08361" id="883">
        <attvalues>
          <attvalue for="5" value="\cite{ref2} confirms that performance of the Transformer steadily improves with the scaling up of model size, dataset size, and the amount of computation for training." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="1907.11692" id="884">
        <attvalues>
          <attvalue for="5" value=" Roberta\cite{ref3} shows that the accuracy of BERT can be substantially improved by training the model for a longer time with a larger corpus." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="2005.14165" id="886">
        <attvalues>
          <attvalue for="5" value=" Take GPT-3 175B for example, it was trained on a cluster of 10,000 GPUs \cite{ref5,ref28}." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="2104.12369" id="887">
        <attvalues>
          <attvalue for="5" value=" However, both GPT-3 and PanGu-$\alpha$ with singleton architecture, exhibits good performance on Zero-Shot and Few-Shot learning\cite{ref5,ref6}." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="2006.16668" id="888">
        <attvalues>
          <attvalue for="5" value=" &#09;\item Mixture of Experts (MoE): Scaling the model size with Sparsely Gated Mixture-of-Experts (MoE), such as GShard\cite{ref7}, Switch Transformer\cite{ref8}, Wudao\cite{ref9, ref10} and M6\cite{ref11}." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="2101.03961" id="889">
        <attvalues>
          <attvalue for="5" value=" With MoE, the model size can be successfully scaled up to more than 1000B \cite{ref8,ref10}." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="2012.00413" id="890">
        <attvalues>
          <attvalue for="5" value=" &#09;\item Mixture of Experts (MoE): Scaling the model size with Sparsely Gated Mixture-of-Experts (MoE), such as GShard\cite{ref7}, Switch Transformer\cite{ref8}, Wudao\cite{ref9, ref10} and M6\cite{ref11}." />
        </attvalues>
      </edge>
      <edge source="2110.04725" target="2103.00823" id="891">
        <attvalues>
          <attvalue for="5" value=" &#09;\item Mixture of Experts (MoE): Scaling the model size with Sparsely Gated Mixture-of-Experts (MoE), such as GShard\cite{ref7}, Switch Transformer\cite{ref8}, Wudao\cite{ref9, ref10} and M6\cite{ref11}." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1708.02182" id="892">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) predict a probability distribution over sequences, and are most widely studied to model and generate natural languages~\cite{bengio2003neural,merity2018regularizing,baevski2018adaptive,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1809.10853" id="893">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) predict a probability distribution over sequences, and are most widely studied to model and generate natural languages~\cite{bengio2003neural,merity2018regularizing,baevski2018adaptive,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="2005.14165" id="894">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) predict a probability distribution over sequences, and are most widely studied to model and generate natural languages~\cite{bengio2003neural,merity2018regularizing,baevski2018adaptive,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1409.0473" id="895">
        <attvalues>
          <attvalue for="5" value=" Advances in LMs benefit many natural language processing downstream tasks, such as machine translation~\cite{bahdanau2014neural}, dialog systems~\cite{sordoni2015neural}, question answering~\cite{yang2019xlnet,raffel2019exploring}, and general representation learning for natural language~\cite{devlin2018bert,liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1506.06714" id="896">
        <attvalues>
          <attvalue for="5" value=" Advances in LMs benefit many natural language processing downstream tasks, such as machine translation~\cite{bahdanau2014neural}, dialog systems~\cite{sordoni2015neural}, question answering~\cite{yang2019xlnet,raffel2019exploring}, and general representation learning for natural language~\cite{devlin2018bert,liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1907.11692" id="900">
        <attvalues>
          <attvalue for="5" value=" Advances in LMs benefit many natural language processing downstream tasks, such as machine translation~\cite{bahdanau2014neural}, dialog systems~\cite{sordoni2015neural}, question answering~\cite{yang2019xlnet,raffel2019exploring}, and general representation learning for natural language~\cite{devlin2018bert,liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1910.00577" id="901">
        <attvalues>
          <attvalue for="5" value=" Recently, LMs have also been adopted to model sequences other than text, such as source code written in programming language~\cite{hindle2016naturalness,hellendoorn2017deep,alon2020structural,karampatsis2020big}, which can enable useful downstream tasks like code completion~\cite{raychev2014code}." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1706.03762" id="902">
        <attvalues>
          <attvalue for="5" value=" Most current neural LMs are based on parametric neural networks, using RNN~\cite{mikolov2010recurrent} or Transformer~\cite{vaswani2017attention} architectures." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1711.02604" id="903">
        <attvalues>
          <attvalue for="5" value=" Recently, more and more neural LMs also incorporate non-parametric components~\cite{grave2017unbounded,guu2018generating,he2020learning,Khandelwal2020Generalization}, which usually first select examples from an external source and then reference them during the prediction." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1709.08878" id="904">
        <attvalues>
          <attvalue for="5" value=" Recently, more and more neural LMs also incorporate non-parametric components~\cite{grave2017unbounded,guu2018generating,he2020learning,Khandelwal2020Generalization}, which usually first select examples from an external source and then reference them during the prediction." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="2006.16336" id="905">
        <attvalues>
          <attvalue for="5" value=" Recently, more and more neural LMs also incorporate non-parametric components~\cite{grave2017unbounded,guu2018generating,he2020learning,Khandelwal2020Generalization}, which usually first select examples from an external source and then reference them during the prediction." />
        </attvalues>
      </edge>
      <edge source="2110.02870" target="1911.00172" id="906">
        <attvalues>
          <attvalue for="5" value=" One example of such metric is the $\ell^2$ distance between context vectors calculated by the parametric model~\cite{Khandelwal2020Generalization}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2307.09288" id="907">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) \cite{radford2018improving,touvron2023llama,zhang2023towards,li2024tf,zhang2024modification,zhang2024dynamic} have accommodated a wide range of natural language processing applications, such as code completion \cite{rozière2023code} and question answering \cite{kamalloo2023evaluating,jiang2021can,su2019generalizing}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2311.07052" id="908">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) \cite{radford2018improving,touvron2023llama,zhang2023towards,li2024tf,zhang2024modification,zhang2024dynamic} have accommodated a wide range of natural language processing applications, such as code completion \cite{rozière2023code} and question answering \cite{kamalloo2023evaluating,jiang2021can,su2019generalizing}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2308.12950" id="910">
        <attvalues>
          <attvalue for="5" value=" We include three widely-used RoPE extensions, eg, position interpolation~\cite{chen2023extending}, YaRN~\cite{peng2023yarn}, and NTK-Aware interpolation~\cite{rozière2023code}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2305.06984" id="911">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) \cite{radford2018improving,touvron2023llama,zhang2023towards,li2024tf,zhang2024modification,zhang2024dynamic} have accommodated a wide range of natural language processing applications, such as code completion \cite{rozière2023code} and question answering \cite{kamalloo2023evaluating,jiang2021can,su2019generalizing}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2012.00955" id="912">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) \cite{radford2018improving,touvron2023llama,zhang2023towards,li2024tf,zhang2024modification,zhang2024dynamic} have accommodated a wide range of natural language processing applications, such as code completion \cite{rozière2023code} and question answering \cite{kamalloo2023evaluating,jiang2021can,su2019generalizing}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2306.15595" id="914">
        <attvalues>
          <attvalue for="5" value=" We include three widely-used RoPE extensions, eg, position interpolation~\cite{chen2023extending}, YaRN~\cite{peng2023yarn}, and NTK-Aware interpolation~\cite{rozière2023code}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2309.00071" id="915">
        <attvalues>
          <attvalue for="5" value=" We include three widely-used RoPE extensions, eg, position interpolation~\cite{chen2023extending}, YaRN~\cite{peng2023yarn}, and NTK-Aware interpolation~\cite{rozière2023code}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2405.14591" id="917">
        <attvalues>
          <attvalue for="5" value=" These RoPE extensions focus on improving performance on long texts, yet frustratingly, only a few of them \cite{liu2023scaling,han2023lm,men2024base} have explored the underlying mechanisms in depth." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="1706.03762" id="918">
        <attvalues>
          <attvalue for="5" value=" Thus, we systematically analyze common RoPE extensions more straightforwardly, from the perspective of attention \cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2406.13282" target="2402.10171" id="919">
        <attvalues>
          <attvalue for="5" value=" Afterward, following literature~\cite{fu2024data}, we examine these RoPE extensions on a more challenging long-context test called Needle-in-a-Haystack (Needle)~\cite{needleinhaystack}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2208.03274" id="939">
        <attvalues>
          <attvalue for="5" value=" L{arge} Language Models (LLMs) have demonstrated impressive capabilities in miscellaneous Natural Language Processing (NLP) tasks and promising adaptability in practical applications across diverse domains, including but not limited to content moderation~\cite{markov2023holistic}, code generation~\cite{roziere2023code}, conversational AI~\cite{chatgpt2023}, and personalized content recommendations~\cite{wu2023survey}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2308.12950" id="940">
        <attvalues>
          <attvalue for="5" value=" L{arge} Language Models (LLMs) have demonstrated impressive capabilities in miscellaneous Natural Language Processing (NLP) tasks and promising adaptability in practical applications across diverse domains, including but not limited to content moderation~\cite{markov2023holistic}, code generation~\cite{roziere2023code}, conversational AI~\cite{chatgpt2023}, and personalized content recommendations~\cite{wu2023survey}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2305.19860" id="941">
        <attvalues>
          <attvalue for="5" value=" L{arge} Language Models (LLMs) have demonstrated impressive capabilities in miscellaneous Natural Language Processing (NLP) tasks and promising adaptability in practical applications across diverse domains, including but not limited to content moderation~\cite{markov2023holistic}, code generation~\cite{roziere2023code}, conversational AI~\cite{chatgpt2023}, and personalized content recommendations~\cite{wu2023survey}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2202.03629" id="943">
        <attvalues>
          <attvalue for="5" value=" Such phenomenon of erroneous generation can exhibit in terms of different manifestations (\eg, hallucination \cite{ji2023survey}, disinformation \cite{tamkin2021understanding}, bias \cite{abid2021persistent}) across various tasks." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2102.02503" id="944">
        <attvalues>
          <attvalue for="5" value=" Such phenomenon of erroneous generation can exhibit in terms of different manifestations (\eg, hallucination \cite{ji2023survey}, disinformation \cite{tamkin2021understanding}, bias \cite{abid2021persistent}) across various tasks." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2101.05783" id="945">
        <attvalues>
          <attvalue for="5" value=" Such phenomenon of erroneous generation can exhibit in terms of different manifestations (\eg, hallucination \cite{ji2023survey}, disinformation \cite{tamkin2021understanding}, bias \cite{abid2021persistent}) across various tasks." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2302.04023" id="946">
        <attvalues>
          <attvalue for="5" value=" However, these responses are often presented in a natural human-like tone \cite{bang2023multitask, johnson2023assessing}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2303.12712" id="947">
        <attvalues>
          <attvalue for="5" value=" Such characteristics cause erroneous information to be highly mixed and intertwined with confident and factual contexts, making their detection and localization difficult without close inspection and diligent fact-checking \cite{bubeck2023sparks}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2305.07153" id="948">
        <attvalues>
          <attvalue for="5" value=" A recent survey highlights that 98\% of respondents, encompassing domain experts and civil society members, firmly believe that AGI (artificial general intelligence) labs should undertake risk assessments before deployment~\cite{schuett2023towards}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2307.03718" id="949">
        <attvalues>
          <attvalue for="5" value=" Major tech corporations~\cite{anderljung2023frontier} such as Microsoft~\cite{microsoft2022responsibleai}, OpenAI~\cite{openai2023aisafety}, Amazon~\cite{amazon2023responsibleai}, and Google~\cite{google2023responsibleai}, along with non-governmental organizations (NGOs, \eg, the Centre for the Governance of AI~\cite{koessler2023risk}), are fervently working towards developing safe, secure, transparent, reliable and responsible LLMs and AGI applications." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2011.07586" id="952">
        <attvalues>
          <attvalue for="5" value=" Uncertainty estimation, aimed at gauging the confidence level of model outputs~\cite{bhatt2021uncertainty, hullermeier2021aleatoric, rahmati2019predicting}, stands out as a promising approach for identifying risks in general Machine Learning (ML) models." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2303.08896" id="953">
        <attvalues>
          <attvalue for="5" value=" Such techniques also have the potential for detecting erroneous generation from LLMs~\cite{manakul2023selfcheckgpt} even under black-box settings." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2308.10620" id="955">
        <attvalues>
          <attvalue for="5" value=" LLMs have revolutionized various aspects of software engineering~\cite{hou2023large, fan2023large}, including but not limited to automated code generation~\cite{fried2023incoder, roziere2023code, allal2023santacoder, nijkamp2022codegen}, software testing~\cite{gu2023llm, kang2023large, lemieux2023codamosa, liu2023fill}, debugging~\cite{tian2024debugbench}, program repair~\cite{first2023baldur, wei2023copiloting}, and document generation~~\cite{geng2024large}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2204.05999" id="956">
        <attvalues>
          <attvalue for="5" value=" LLMs have revolutionized various aspects of software engineering~\cite{hou2023large, fan2023large}, including but not limited to automated code generation~\cite{fried2023incoder, roziere2023code, allal2023santacoder, nijkamp2022codegen}, software testing~\cite{gu2023llm, kang2023large, lemieux2023codamosa, liu2023fill}, debugging~\cite{tian2024debugbench}, program repair~\cite{first2023baldur, wei2023copiloting}, and document generation~~\cite{geng2024large}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2401.04621" id="957">
        <attvalues>
          <attvalue for="5" value=" LLMs have revolutionized various aspects of software engineering~\cite{hou2023large, fan2023large}, including but not limited to automated code generation~\cite{fried2023incoder, roziere2023code, allal2023santacoder, nijkamp2022codegen}, software testing~\cite{gu2023llm, kang2023large, lemieux2023codamosa, liu2023fill}, debugging~\cite{tian2024debugbench}, program repair~\cite{first2023baldur, wei2023copiloting}, and document generation~~\cite{geng2024large}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2303.04910" id="958">
        <attvalues>
          <attvalue for="5" value=" LLMs have revolutionized various aspects of software engineering~\cite{hou2023large, fan2023large}, including but not limited to automated code generation~\cite{fried2023incoder, roziere2023code, allal2023santacoder, nijkamp2022codegen}, software testing~\cite{gu2023llm, kang2023large, lemieux2023codamosa, liu2023fill}, debugging~\cite{tian2024debugbench}, program repair~\cite{first2023baldur, wei2023copiloting}, and document generation~~\cite{geng2024large}." />
        </attvalues>
      </edge>
      <edge source="2307.10236" target="2304.11384" id="959">
        <attvalues>
          <attvalue for="5" value=" LLMs have revolutionized various aspects of software engineering~\cite{hou2023large, fan2023large}, including but not limited to automated code generation~\cite{fried2023incoder, roziere2023code, allal2023santacoder, nijkamp2022codegen}, software testing~\cite{gu2023llm, kang2023large, lemieux2023codamosa, liu2023fill}, debugging~\cite{tian2024debugbench}, program repair~\cite{first2023baldur, wei2023copiloting}, and document generation~~\cite{geng2024large}." />
        </attvalues>
      </edge>
      <edge source="2302.04700" target="2003.10555" id="961">
        <attvalues>
          <attvalue for="5" value=" In this paper, we study how Annotation Artifacts impact performance on an ELECTRA Small model \cite{clark2020electra} trained on the Stanford Natural Language Inference Dataset (SNLI) \cite{DBLP:journals/corr/BowmanAPM15}." />
        </attvalues>
      </edge>
      <edge source="2302.04700" target="1508.05326" id="962">
        <attvalues>
          <attvalue for="5" value=" In this paper, we study how Annotation Artifacts impact performance on an ELECTRA Small model \cite{clark2020electra} trained on the Stanford Natural Language Inference Dataset (SNLI) \cite{DBLP:journals/corr/BowmanAPM15}." />
        </attvalues>
      </edge>
      <edge source="2002.06012" target="1906.07429" id="963">
        <attvalues>
          <attvalue for="5" value=" Popular models are made of frames describing relations between entities and their properties~\cite{tur2011spoken,shen2019modeling,li2019incremental}." />
        </attvalues>
      </edge>
      <edge source="2002.06012" target="1907.08854" id="964">
        <attvalues>
          <attvalue for="5" value=" Popular models are made of frames describing relations between entities and their properties~\cite{tur2011spoken,shen2019modeling,li2019incremental}." />
        </attvalues>
      </edge>
      <edge source="2002.06012" target="1907.07421" id="966">
        <attvalues>
          <attvalue for="5" value=" An example of additional distant context used so far is a representation of dialogue history made of embeddings of sentences preceding the sentence or dialogue turn to be interpreted~\cite{chen2016end, goo2018slot, zhao2019hierarchical, sankar2019neural, goel2019hyst,history,henaff2016tracking,korpusik2019dialogue,lee2019sumbt}." />
        </attvalues>
      </edge>
      <edge source="2002.06012" target="1907.00883" id="967">
        <attvalues>
          <attvalue for="5" value=" Noticeable approaches for reducing uncertainty in concept detection automatically extract relevant information from dialogue history~\cite{chen2016end,goo2018slot,goel2019hyst}." />
        </attvalues>
      </edge>
      <edge source="2306.00550" target="2201.11903" id="970">
        <attvalues>
          <attvalue for="5" value=" \cite{wei2022emergent} \cite{cot} discover Chain-of-Thought (CoT) prompting as a simple and broadly applicable method for enhancing reasoning in language models." />
        </attvalues>
      </edge>
      <edge source="2306.00550" target="2203.11171" id="972">
        <attvalues>
          <attvalue for="5" value=" Many work \cite{zhou2022least,wang2022self,shi2022language,autocot,zhang2023multimodal,wang2022towards,zhou2022large,fei2023reasoning,yang2023mm,shi2023large,diao2023active} have tried to make further improvements based on CoT." />
        </attvalues>
      </edge>
      <edge source="2306.00550" target="2210.03057" id="973">
        <attvalues>
          <attvalue for="5" value=" Many work \cite{zhou2022least,wang2022self,shi2022language,autocot,zhang2023multimodal,wang2022towards,zhou2022large,fei2023reasoning,yang2023mm,shi2023large,diao2023active} have tried to make further improvements based on CoT." />
        </attvalues>
      </edge>
      <edge source="2306.00550" target="2302.00923" id="975">
        <attvalues>
          <attvalue for="5" value=" Many work \cite{zhou2022least,wang2022self,shi2022language,autocot,zhang2023multimodal,wang2022towards,zhou2022large,fei2023reasoning,yang2023mm,shi2023large,diao2023active} have tried to make further improvements based on CoT." />
        </attvalues>
      </edge>
      <edge source="2306.00550" target="2212.10001" id="976">
        <attvalues>
          <attvalue for="5" value=" Many work \cite{zhou2022least,wang2022self,shi2022language,autocot,zhang2023multimodal,wang2022towards,zhou2022large,fei2023reasoning,yang2023mm,shi2023large,diao2023active} have tried to make further improvements based on CoT." />
        </attvalues>
      </edge>
      <edge source="2306.00550" target="2211.01910" id="977">
        <attvalues>
          <attvalue for="5" value=" Many work \cite{zhou2022least,wang2022self,shi2022language,autocot,zhang2023multimodal,wang2022towards,zhou2022large,fei2023reasoning,yang2023mm,shi2023large,diao2023active} have tried to make further improvements based on CoT." />
        </attvalues>
      </edge>
      <edge source="2306.00550" target="2305.11255" id="978">
        <attvalues>
          <attvalue for="5" value=" Many work \cite{zhou2022least,wang2022self,shi2022language,autocot,zhang2023multimodal,wang2022towards,zhou2022large,fei2023reasoning,yang2023mm,shi2023large,diao2023active} have tried to make further improvements based on CoT." />
        </attvalues>
      </edge>
      <edge source="2306.00550" target="2302.00093" id="980">
        <attvalues>
          <attvalue for="5" value=" Many work \cite{zhou2022least,wang2022self,shi2022language,autocot,zhang2023multimodal,wang2022towards,zhou2022large,fei2023reasoning,yang2023mm,shi2023large,diao2023active} have tried to make further improvements based on CoT." />
        </attvalues>
      </edge>
      <edge source="2205.10475" target="1802.05365" id="982">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (LMs) have revolutionized NLP over the last few years~\cite{peters2018deep,devlin2019bert,radford2019language}, increasingly adept in performing the flexible and task-agnostic downstream transfer." />
        </attvalues>
      </edge>
      <edge source="2205.10475" target="2005.14165" id="984">
        <attvalues>
          <attvalue for="5" value=" For example, GPT-3~\cite{brown2020language} is trained to predict the next word, and BERT~\cite{devlin2019bert} is trained to recover the masked tokens." />
        </attvalues>
      </edge>
      <edge source="2205.10475" target="1906.03158" id="985">
        <attvalues>
          <attvalue for="5" value=" As shown in Figure~\ref{fig:approach}, first, task-specific architectures are proposed to model the structures for different structure prediction tasks~\cite{stanovsky2018supervised,soares2019matching}." />
        </attvalues>
      </edge>
      <edge source="2205.10475" target="2101.05779" id="986">
        <attvalues>
          <attvalue for="5" value=" Second, task-specific data augmentation~\cite{paolini2021structured,wang2021zero,wei2021finetuned} is introduced, aiming to enrich text format with structure information." />
        </attvalues>
      </edge>
      <edge source="2205.10475" target="2109.11171" id="987">
        <attvalues>
          <attvalue for="5" value=" Second, task-specific data augmentation~\cite{paolini2021structured,wang2021zero,wei2021finetuned} is introduced, aiming to enrich text format with structure information." />
        </attvalues>
      </edge>
      <edge source="2205.10475" target="2109.01652" id="988">
        <attvalues>
          <attvalue for="5" value=" Second, task-specific data augmentation~\cite{paolini2021structured,wang2021zero,wei2021finetuned} is introduced, aiming to enrich text format with structure information." />
        </attvalues>
      </edge>
      <edge source="2108.05575" target="2101.12175" id="989">
        <attvalues>
          <attvalue for="5" value=" As a basis for our system, we will use LOME \cite{xia-etal-2021-lome}, a recent multilingual frame semantic parsing model, as a basis, and experiment with several approaches for extending it to Kicktionary." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2005.14165" id="990">
        <attvalues>
          <attvalue for="5" value=" After extensive training on publicly available web data, large language models have exhibited remarkable capabilities in a wide range of common sense reasoning tasks\cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="1906.05714" id="991">
        <attvalues>
          <attvalue for="5" value=" Some studies have indicated that these models, during pretraining, not only learn contextual text vector representations and probability distributions but are also likely to learn grammar\cite{vig2019multiscale}, syntax\cite{hewitt2019structural}, factual knowledge\cite{wang2020language}, even behavior patterns strikingly similar to human common sense\cite{davison2019commonsense}." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2010.11967" id="992">
        <attvalues>
          <attvalue for="5" value=" Some studies have indicated that these models, during pretraining, not only learn contextual text vector representations and probability distributions but are also likely to learn grammar\cite{vig2019multiscale}, syntax\cite{hewitt2019structural}, factual knowledge\cite{wang2020language}, even behavior patterns strikingly similar to human common sense\cite{davison2019commonsense}." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2303.12712" id="994">
        <attvalues>
          <attvalue for="5" value=" While general-purpose LLMs trained on massive internet datasets demonstrate impressive capabilities across diverse domains\cite{bubeck2023sparks}, recent work with domain-specific models like BloombergGPT \cite{wu2023bloomberggpt} and BioMedLM\cite{bolton2024biomedlm} highlights their superior performance within specific fields." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2303.17564" id="995">
        <attvalues>
          <attvalue for="5" value=" While general-purpose LLMs trained on massive internet datasets demonstrate impressive capabilities across diverse domains\cite{bubeck2023sparks}, recent work with domain-specific models like BloombergGPT \cite{wu2023bloomberggpt} and BioMedLM\cite{bolton2024biomedlm} highlights their superior performance within specific fields." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="1706.03762" id="996">
        <attvalues>
          <attvalue for="5" value=" The introduction of the Transformer architecture\cite{vaswani2017attention} led to the development of highly scalable language models\cite{radford2019language}, demonstrating a predictable relationship between language modeling loss and scaling factors such as model size, number of training tokens, and compute budget." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2204.06745" id="997">
        <attvalues>
          <attvalue for="5" value=" EleutherAI released GPT-NeoX-20B\cite{black2022gpt} and GPT-J-6B \cite{wang2021codet5}, as well as the dataset these models were trained on Pile\cite{gao2020pile}." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2101.00027" id="999">
        <attvalues>
          <attvalue for="5" value=" EleutherAI released GPT-NeoX-20B\cite{black2022gpt} and GPT-J-6B \cite{wang2021codet5}, as well as the dataset these models were trained on Pile\cite{gao2020pile}." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2205.05131" id="1000">
        <attvalues>
          <attvalue for="5" value=" Google released UL2-20B\cite{tay2022ul2}, an encoder-decoder model trained on the publicly available C4\cite{raffel2020exploring}." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2210.02414" id="1002">
        <attvalues>
          <attvalue for="5" value=" Tsinghua University released the weights of GLM-130B\cite{zeng2022glm}, a Chinese-English LLM, and CodeGeeX-13B\cite{zheng2023codegeex}, a LLM for coding applications, without releasing the training sets." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2203.13474" id="1003">
        <attvalues>
          <attvalue for="5" value=" Salesforce released CodeGen-Mono-16B\cite{nijkamp2022codegen} without disclosing a proprietary Python dataset." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2205.01068" id="1004">
        <attvalues>
          <attvalue for="5" value=" Meta released the OPT\cite{zhang2022opt}, LLaMA\cite{touvron2023llama} under a non-commercial license and only provided high-level details about the data collection and filtering process." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2305.15062" id="1006">
        <attvalues>
          <attvalue for="5" value=" Many attempts to fine-tune on specialized domain-specific datasets only use Supervised Fine-Tuning (SFT) or Keep Pretraining (KP), such as Lawyer LLaMA\cite{huang2023lawyer} or Huatuo\cite{wang2023huatuo}." />
        </attvalues>
      </edge>
      <edge source="2405.09857" target="2304.06975" id="1007">
        <attvalues>
          <attvalue for="5" value=" Many attempts to fine-tune on specialized domain-specific datasets only use Supervised Fine-Tuning (SFT) or Keep Pretraining (KP), such as Lawyer LLaMA\cite{huang2023lawyer} or Huatuo\cite{wang2023huatuo}." />
        </attvalues>
      </edge>
      <edge source="2407.00416" target="2005.14165" id="1008">
        <attvalues>
          <attvalue for="5" value="5 \cite{fewshot} in late 2022 has kickstarted the current era of rapid progress in Large Language Models (LLMs)." />
        </attvalues>
      </edge>
      <edge source="2407.00416" target="2307.15043" id="1011">
        <attvalues>
          <attvalue for="5" value=" Techniques such as synthetic data generation \cite{eldan2023tinystories, gunasekar2023textbooks}, the integration of mathematical and coding tasks to enhance reasoning capabilities \cite{codeReasoning}, and research into adversarial attacks \cite{llmattack} for improved safety have all contributed to the ever-increasing capabilities of LLMs." />
        </attvalues>
      </edge>
      <edge source="2407.00416" target="2009.09359" id="1012">
        <attvalues>
          <attvalue for="5" value=" Despite being one of the most widely spoken languages, the size of Bengali pretraining and instruction-tuning data are minuscule compared to their English counterparts \cite{bangla_translation, banglabert}." />
        </attvalues>
      </edge>
      <edge source="2407.00416" target="2205.11081" id="1013">
        <attvalues>
          <attvalue for="5" value=" To this date, BanglaT5 \cite{banglat5}, a 248 million parameter encoder-decoder T5 transformer \cite{t5}, remains the most capable Bengali Language Model." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2005.14165" id="1015">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{brown2020language} showed a standard prompting technique with question-answer pairs that can result in a few-shot effect." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2201.08239" id="1016">
        <attvalues>
          <attvalue for="5" value=" Recently, conversational Large Language Models (LLMs) such as GPT-3~\cite{brown2020language}, Bard~\cite{thoppilan2022lamda}, LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, PaLM~\cite{chowdhery2022palm}, etc have demonstrated exceptional performance in a wide range of popular natural language processing (NLP) tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2211.05100" id="1018">
        <attvalues>
          <attvalue for="5" value=" Recently, conversational Large Language Models (LLMs) such as GPT-3~\cite{brown2020language}, Bard~\cite{thoppilan2022lamda}, LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, PaLM~\cite{chowdhery2022palm}, etc have demonstrated exceptional performance in a wide range of popular natural language processing (NLP) tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2204.02311" id="1019">
        <attvalues>
          <attvalue for="5" value=" Recently, conversational Large Language Models (LLMs) such as GPT-3~\cite{brown2020language}, Bard~\cite{thoppilan2022lamda}, LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, PaLM~\cite{chowdhery2022palm}, etc have demonstrated exceptional performance in a wide range of popular natural language processing (NLP) tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2303.12712" id="1020">
        <attvalues>
          <attvalue for="5" value=" Recently, conversational Large Language Models (LLMs) such as GPT-3~\cite{brown2020language}, Bard~\cite{thoppilan2022lamda}, LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, PaLM~\cite{chowdhery2022palm}, etc have demonstrated exceptional performance in a wide range of popular natural language processing (NLP) tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2112.06905" id="1021">
        <attvalues>
          <attvalue for="5" value=" Recently, conversational Large Language Models (LLMs) such as GPT-3~\cite{brown2020language}, Bard~\cite{thoppilan2022lamda}, LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, PaLM~\cite{chowdhery2022palm}, etc have demonstrated exceptional performance in a wide range of popular natural language processing (NLP) tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2201.11990" id="1022">
        <attvalues>
          <attvalue for="5" value=" Recently, conversational Large Language Models (LLMs) such as GPT-3~\cite{brown2020language}, Bard~\cite{thoppilan2022lamda}, LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, PaLM~\cite{chowdhery2022palm}, etc have demonstrated exceptional performance in a wide range of popular natural language processing (NLP) tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2107.13586" id="1023">
        <attvalues>
          <attvalue for="5" value=" Unsurprisingly, the quality and effectiveness of the prompt can greatly influence the performance of the LLMs for a particular task, and therefore, designing appropriate prompts with the right amount of detail has become more important than ever~\cite{liu2023pre,han2022ptr}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2105.11259" id="1024">
        <attvalues>
          <attvalue for="5" value=" Unsurprisingly, the quality and effectiveness of the prompt can greatly influence the performance of the LLMs for a particular task, and therefore, designing appropriate prompts with the right amount of detail has become more important than ever~\cite{liu2023pre,han2022ptr}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2210.03629" id="1025">
        <attvalues>
          <attvalue for="5" value=" Researchers also explored other prompt design techniques such as Chain-of-thought (CoT)~\cite{wei2022chain}, Reasoning and Acting (ReAct)~\cite{yao2022react}, and other techniques~\cite{kojima2022large,madaan2022text,press2022measuring} in terms of improving the reasoning and acting of LLMs in solving Question-Answering tasks." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2209.07686" id="1027">
        <attvalues>
          <attvalue for="5" value=" Researchers also explored other prompt design techniques such as Chain-of-thought (CoT)~\cite{wei2022chain}, Reasoning and Acting (ReAct)~\cite{yao2022react}, and other techniques~\cite{kojima2022large,madaan2022text,press2022measuring} in terms of improving the reasoning and acting of LLMs in solving Question-Answering tasks." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2210.03350" id="1028">
        <attvalues>
          <attvalue for="5" value=" In this paper, we exclusively focus on understanding LLMs' potential for performing complex tasks that are mostly: 1) ill-defined, 2) abstract goal-oriented, 3) highly dependent on subjective interpretation, and 4) very hard to evaluate quantitatively~\cite{khot2022decomposed,press2022measuring}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2303.17491" id="1029">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, \cite{kim2023language} proposed a prompting scheme where the agent recursively criticizes and improves its output (RCI) to solve a task." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2210.02406" id="1030">
        <attvalues>
          <attvalue for="5" value=" In this paper, we exclusively focus on understanding LLMs' potential for performing complex tasks that are mostly: 1) ill-defined, 2) abstract goal-oriented, 3) highly dependent on subjective interpretation, and 4) very hard to evaluate quantitatively~\cite{khot2022decomposed,press2022measuring}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2203.14465" id="1031">
        <attvalues>
          <attvalue for="5" value=" These complex tasks often involve multiple steps/sub-tasks, and designing ``appropriate'' prompts for such tasks is indeed challenging as there is no single rule book to follow in these cases~\cite{zelikman2022star,nye2021show}." />
        </attvalues>
      </edge>
      <edge source="2305.11430" target="2112.00114" id="1032">
        <attvalues>
          <attvalue for="5" value=" These complex tasks often involve multiple steps/sub-tasks, and designing ``appropriate'' prompts for such tasks is indeed challenging as there is no single rule book to follow in these cases~\cite{zelikman2022star,nye2021show}." />
        </attvalues>
      </edge>
      <edge source="2304.07327" target="1706.03762" id="1045">
        <attvalues>
          <attvalue for="5" value=" Major advancements are primarily driven by a straightforward formula: take a Transformer~\cite{vaswani2017attention}-based architecture, increase the parameter count by enlarging depth and width, increase the size of the training corpus, and increase the scale of training compute." />
        </attvalues>
      </edge>
      <edge source="2304.07327" target="2001.09768" id="1047">
        <attvalues>
          <attvalue for="5" value=" This refers to the process of ensuring that AI systems can not only successfully optimize the provided surrogate training objectives, but also that their predictions are in line with their intended purpose and adhere to ethical and safety standards provided by humans~\cite{Gabriel2020,wang2023aligning}." />
        </attvalues>
      </edge>
      <edge source="2304.07327" target="2307.12966" id="1048">
        <attvalues>
          <attvalue for="5" value=" This refers to the process of ensuring that AI systems can not only successfully optimize the provided surrogate training objectives, but also that their predictions are in line with their intended purpose and adhere to ethical and safety standards provided by humans~\cite{Gabriel2020,wang2023aligning}." />
        </attvalues>
      </edge>
      <edge source="2304.07327" target="2204.05862" id="1049">
        <attvalues>
          <attvalue for="5" value=" One possible solution is assistant-style fine-tuning of language models that has recently emerged as a promising approach to making large language models more in line with human preferences by generating more desirable outputs based on explicitly collected human preference data~\cite{bai2022training, pmlr-v162-ethayarajh22a, thoppilan2022lamda, hilton2021webgpt,menick2022teaching,ziegler2020finetuning} and thus making them more useful." />
        </attvalues>
      </edge>
      <edge source="2304.07327" target="2203.11147" id="1050">
        <attvalues>
          <attvalue for="5" value=" One possible solution is assistant-style fine-tuning of language models that has recently emerged as a promising approach to making large language models more in line with human preferences by generating more desirable outputs based on explicitly collected human preference data~\cite{bai2022training, pmlr-v162-ethayarajh22a, thoppilan2022lamda, hilton2021webgpt,menick2022teaching,ziegler2020finetuning} and thus making them more useful." />
        </attvalues>
      </edge>
      <edge source="2304.07327" target="1909.08593" id="1051">
        <attvalues>
          <attvalue for="5" value=" One possible solution is assistant-style fine-tuning of language models that has recently emerged as a promising approach to making large language models more in line with human preferences by generating more desirable outputs based on explicitly collected human preference data~\cite{bai2022training, pmlr-v162-ethayarajh22a, thoppilan2022lamda, hilton2021webgpt,menick2022teaching,ziegler2020finetuning} and thus making them more useful." />
        </attvalues>
      </edge>
      <edge source="2304.07327" target="1707.06347" id="1053">
        <attvalues>
          <attvalue for="5" value=" This is achieved using the PPO algorithm~\cite{schulman2017proximal}." />
        </attvalues>
      </edge>
      <edge source="2304.07327" target="2207.14529" id="1054">
        <attvalues>
          <attvalue for="5" value=" It becomes apparent that the benefits of all the aforementioned stages are predominantly dependent on the quality of the data used~\cite{DataQuality}." />
        </attvalues>
      </edge>
      <edge source="2010.04650" target="1803.03585" id="1055">
        <attvalues>
          <attvalue for="5" value=" It remains a popular architecture in NLP, and unlike Transformer-based models, it can be trained on small corpora~\cite{tran_importance_2018}." />
        </attvalues>
      </edge>
      <edge source="2010.04650" target="2006.00555" id="1056">
        <attvalues>
          <attvalue for="5" value=" \cite{abnar_transferring_2020} even found that the recurrent inductive biases behind the LSTM's success are so essential that distilling from them can improve the performance of fully attentional models." />
        </attvalues>
      </edge>
      <edge source="2010.04650" target="1602.07776" id="1058">
        <attvalues>
          <attvalue for="5" value=" A Transformer can encode syntax using attention \cite{hewitt_structural_nodate}, and some LSTM variants explicitly encode syntax \cite{bowman-etal-2016-fast,DBLP:journals/corr/DyerKBS16}." />
        </attvalues>
      </edge>
      <edge source="2010.04650" target="1805.11653" id="1059">
        <attvalues>
          <attvalue for="5" value=" For example, they can recall more history in natural language data than in similarly Zipfian-distributed $n$-gram data, implying that they exploit linguistic structure in long-distance dependencies \cite{liu_lstms_2018}." />
        </attvalues>
      </edge>
      <edge source="2010.04650" target="1805.04218" id="1060">
        <attvalues>
          <attvalue for="5" value=" Their internal representations appear to encode constituency \cite{blevins_deep_2018,hupkes_visualisation_2018} and syntactic agreement \cite{lakretz-etal-2019-emergence,gulordava-etal-2018-colorless}." />
        </attvalues>
      </edge>
      <edge source="2010.04650" target="1903.07435" id="1061">
        <attvalues>
          <attvalue for="5" value=" Their internal representations appear to encode constituency \cite{blevins_deep_2018,hupkes_visualisation_2018} and syntactic agreement \cite{lakretz-etal-2019-emergence,gulordava-etal-2018-colorless}." />
        </attvalues>
      </edge>
      <edge source="2010.04650" target="1803.11138" id="1062">
        <attvalues>
          <attvalue for="5" value=" Their internal representations appear to encode constituency \cite{blevins_deep_2018,hupkes_visualisation_2018} and syntactic agreement \cite{lakretz-etal-2019-emergence,gulordava-etal-2018-colorless}." />
        </attvalues>
      </edge>
      <edge source="2108.11857" target="1910.11470" id="1064">
        <attvalues>
          <attvalue for="5" value=" Before transformer LMs \cite{devlin-etal-2019-bert}, the state-of-the-art NER was based on training recurrent neural networks, such as bidirectional LSTM (BiLSTM) with a Conditional Random Field (CRF) layer, from scratch \cite{yadav-bethard-2018-survey}." />
        </attvalues>
      </edge>
      <edge source="2108.11857" target="2004.12126" id="1065">
        <attvalues>
          <attvalue for="5" value=" While LMs fine-tuned for NER have achieved impressive results on standard benchmarks \cite{akbik-etal-2019-pooled}, multiple works have emphasized their limitations with regard to their generalization capacity to new textual genres (eg clean versus noisy text), NE type sets (eg NE types belonging to new domains such as music or e-commerce) and new NEs, unseen during training \cite{lin-etal-2020-rigorous}." />
        </attvalues>
      </edge>
      <edge source="2108.11857" target="2001.03844" id="1067">
        <attvalues>
          <attvalue for="5" value=" Probing has been designed for BiLSTM-CRF LMs \cite{Augenstein2017Generalisation,Taille2020contextualized,fu2020rethinking} or masked LMs such as BERT \cite{petroni-etal-2019-language,jiang-etal-2020-know}." />
        </attvalues>
      </edge>
      <edge source="2108.11857" target="1909.01066" id="1068">
        <attvalues>
          <attvalue for="5" value=" Probing has been designed for BiLSTM-CRF LMs \cite{Augenstein2017Generalisation,Taille2020contextualized,fu2020rethinking} or masked LMs such as BERT \cite{petroni-etal-2019-language,jiang-etal-2020-know}." />
        </attvalues>
      </edge>
      <edge source="2108.11857" target="2004.07493" id="1071">
        <attvalues>
          <attvalue for="5" value=" Humans can easily recognize NEs based on prior domain and common sense linguistic knowledge, or by leveraging contextual cues in text \cite{lin-etal-2020-triggerner}." />
        </attvalues>
      </edge>
      <edge source="2108.11857" target="2005.14165" id="1072">
        <attvalues>
          <attvalue for="5" value=" It entails the task specification via the text input used to prompt the model, without performing any gradient updates \cite{Brown2020Language}." />
        </attvalues>
      </edge>
      <edge source="2108.11857" target="cs/0306050" id="1073">
        <attvalues>
          <attvalue for="5" value=" We use four datasets: CoNLL-2003 \cite{tjong-kim-sang-de-meulder-2003-introduction}, WNUT2017 \cite{liu2014a}, MIT Movie \cite{liu2014a} and extensive lists of NEs from DBpedia \cite{Auer2007Dbpedia}." />
        </attvalues>
      </edge>
      <edge source="2403.00964" target="2106.07139" id="1074">
        <attvalues>
          <attvalue for="5" value=" These models offered unprecedented levels of fluency and coherence in generated text \cite{HAN2021225}." />
        </attvalues>
      </edge>
      <edge source="2403.00964" target="2202.03629" id="1075">
        <attvalues>
          <attvalue for="5" value=" Some examples to tackle hallucination detection tasks in literature \cite{Ji_2023} are: (i) Information Extraction and Comparison between a generated text and a ground truth, (ii) Natural Language Inference Metrics that express the entailment between generated text and a ground truth or (iii) Faithfulness Classification Metrics that leverage upon knowledge-grounded datasets." />
        </attvalues>
      </edge>
      <edge source="2403.00964" target="2311.05232" id="1076">
        <attvalues>
          <attvalue for="5" value=" The ever-increasing adoption of such models makes it necessary to automatically detect and mitigate semantic hallucinations \cite{huang2023survey}." />
        </attvalues>
      </edge>
      <edge source="2403.00964" target="2309.11235" id="1077">
        <attvalues>
          <attvalue for="5" value=" Additionally, we suggest using an ensemble of three different approaches, incorporating a simple BERT-based classifier, a model trained through Conditioned Reinforcement Learning Fine Tuning (C-RLFT)~\cite{wang2023openchat}, and a sequential model based on iterative fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2203.10995" target="1706.03762" id="1078">
        <attvalues>
          <attvalue for="5" value=" To compensate for this absence of linear order, the transformer architecture originally proposed in~\cite{vaswani2017attention} includes a fixed, sinusoidal position embedding added to each token embedding; each token carries a different position embedding, corresponding to its position in the sentence." />
        </attvalues>
      </edge>
      <edge source="2203.10995" target="1907.11692" id="1080">
        <attvalues>
          <attvalue for="5" value=" The transformer-based BERT~\cite{devlin2018bert} replaces these fixed sinusoidal embeddings with unique, learned embeddings per position; RoBERTa~\cite{LiuRoBERTaRobustlyOptimized19a}, the model investigated in this work, does the same." />
        </attvalues>
      </edge>
      <edge source="2203.10995" target="2104.06644" id="1081">
        <attvalues>
          <attvalue for="5" value=" \cite{sinha2021masked} pre-trained RoBERTa models on shuffled corpora to demonstrate that the performance gap between these `shuffled' language models and models trained on unshuffled corpora is minor (when fine-tuned and evaluated downstream on the GLUE~\cite{WangGLUEMultiTaskBenchmark18} benchmark)." />
        </attvalues>
      </edge>
      <edge source="2210.05619" target="2308.02976" id="1083">
        <attvalues>
          <attvalue for="5" value=" We demonstrate this bias effect in Spanish and Greek, comparing the monolingual models BETO \cite{beto} and GreekBERT \cite{greekbert} to multilingual BERT (mBERT), where English is the most frequent language in the training data." />
        </attvalues>
      </edge>
      <edge source="2210.05619" target="2008.12014" id="1084">
        <attvalues>
          <attvalue for="5" value=" We demonstrate this bias effect in Spanish and Greek, comparing the monolingual models BETO \cite{beto} and GreekBERT \cite{greekbert} to multilingual BERT (mBERT), where English is the most frequent language in the training data." />
        </attvalues>
      </edge>
      <edge source="2210.05619" target="1911.02116" id="1088">
        <attvalues>
          <attvalue for="5" value=" Methods for creating more equitable models have been proposed, through identifying or reserving language-specific parameters for each language \cite{ansell2022composable, pfeiffer2022lifting}, through training models without tyoplogically distant languages that dominate the training data \cite{afriberta, virtanen2019finnish, ogunremi2023mini}, as well as through adding model capacity \cite{conneau2019unsupervised, xue2021mt5, lepikhin2021gshard, liang2023xlmv}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="1910.12638" id="1089">
        <attvalues>
          <attvalue for="5" value=" For generative methods, models try to either reconstruct masked acoustic features \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc}, or generate future acoustic features \cite{chung2019apc,chung2020vq-apc,chung2020improvedapc}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="1912.01679" id="1091">
        <attvalues>
          <attvalue for="5" value=" For generative methods, models try to either reconstruct masked acoustic features \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc}, or generate future acoustic features \cite{chung2019apc,chung2020vq-apc,chung2020improvedapc}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2012.06659" id="1092">
        <attvalues>
          <attvalue for="5" value=" For generative methods, models try to either reconstruct masked acoustic features \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc}, or generate future acoustic features \cite{chung2019apc,chung2020vq-apc,chung2020improvedapc}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2011.00406" id="1093">
        <attvalues>
          <attvalue for="5" value=" For generative methods, models try to either reconstruct masked acoustic features \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc}, or generate future acoustic features \cite{chung2019apc,chung2020vq-apc,chung2020improvedapc}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="1904.03240" id="1094">
        <attvalues>
          <attvalue for="5" value=" For generative methods, models try to either reconstruct masked acoustic features \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc}, or generate future acoustic features \cite{chung2019apc,chung2020vq-apc,chung2020improvedapc}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2005.08392" id="1095">
        <attvalues>
          <attvalue for="5" value=" For generative methods, models try to either reconstruct masked acoustic features \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc}, or generate future acoustic features \cite{chung2019apc,chung2020vq-apc,chung2020improvedapc}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2004.05274" id="1096">
        <attvalues>
          <attvalue for="5" value=" For generative methods, models try to either reconstruct masked acoustic features \cite{liu2020mockingjay,liu2021tera,ling2020decoar,ling2020decoar2,liu2021npc}, or generate future acoustic features \cite{chung2019apc,chung2020vq-apc,chung2020improvedapc}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="1807.03748" id="1097">
        <attvalues>
          <attvalue for="5" value=" For the discriminative methods, models either learn by contrastive learning \cite{oord2018cpc,riviere2020m-cpc,schneider2019wav2vec,baevski2020vq-wav2vec,baevski2020wav2vec2,sadhu2021wav2vec-c} or classifying pseudo labels \cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2002.02848" id="1098">
        <attvalues>
          <attvalue for="5" value=" For the discriminative methods, models either learn by contrastive learning \cite{oord2018cpc,riviere2020m-cpc,schneider2019wav2vec,baevski2020vq-wav2vec,baevski2020wav2vec2,sadhu2021wav2vec-c} or classifying pseudo labels \cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="1904.05862" id="1099">
        <attvalues>
          <attvalue for="5" value=" For the discriminative methods, models either learn by contrastive learning \cite{oord2018cpc,riviere2020m-cpc,schneider2019wav2vec,baevski2020vq-wav2vec,baevski2020wav2vec2,sadhu2021wav2vec-c} or classifying pseudo labels \cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="1910.05453" id="1100">
        <attvalues>
          <attvalue for="5" value=" For the discriminative methods, models either learn by contrastive learning \cite{oord2018cpc,riviere2020m-cpc,schneider2019wav2vec,baevski2020vq-wav2vec,baevski2020wav2vec2,sadhu2021wav2vec-c} or classifying pseudo labels \cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2006.11477" id="1101">
        <attvalues>
          <attvalue for="5" value="0~\cite{baevski2020wav2vec2} and HuBERT~\cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2103.08393" id="1102">
        <attvalues>
          <attvalue for="5" value=" For the discriminative methods, models either learn by contrastive learning \cite{oord2018cpc,riviere2020m-cpc,schneider2019wav2vec,baevski2020vq-wav2vec,baevski2020wav2vec2,sadhu2021wav2vec-c} or classifying pseudo labels \cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2106.07447" id="1103">
        <attvalues>
          <attvalue for="5" value="0~\cite{baevski2020wav2vec2} and HuBERT~\cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2104.11462" id="1108">
        <attvalues>
          <attvalue for="5" value=" Therefore, Speech processing Universal PERformance Benchmark (SUPERB) \cite{yang2021superb} and LeBenchmark \cite{evain2021lebenchmark} are developed to evaluate the effectiveness of these methods on general speech processing applications." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="1503.02531" id="1110">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation is a common method for compressing models \cite{hinton2015distilling}, in which a small student model is learned to generate the teacher model's outputs or hidden representations." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="1909.10351" id="1112">
        <attvalues>
          <attvalue for="5" value=" Distilling knowledge has shown to be effective for NLP, and DistilBERT \cite{sanh2019distilbert} and TinyBERT \cite{jiao2020tinybert} are good examples." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2103.15760" id="1113">
        <attvalues>
          <attvalue for="5" value=" However, we found these approaches ineffective in distilling speech SSL models, and few studies investigated this problem \cite{peng2021shrinking,lai2021parp}." />
        </attvalues>
      </edge>
      <edge source="2110.01900" target="2106.05933" id="1114">
        <attvalues>
          <attvalue for="5" value=" However, we found these approaches ineffective in distilling speech SSL models, and few studies investigated this problem \cite{peng2021shrinking,lai2021parp}." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="1807.03748" id="1115">
        <attvalues>
          <attvalue for="5" value=" Contrastive Predictive Coding (CPC)~\cite{oord2018representation} is one such approach whereby the surface feature sequence is first encoded into a latent representation by an encoder network, and an autoregressive model is used to summarize the past latent sequence into a higher-level representation and use it to predict future latent representations." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="2002.02848" id="1116">
        <attvalues>
          <attvalue for="5" value=" CPC and its extensions~\cite{riviere2020unsupervised,schneider2019wav2vec,baevski2020vq,baevski2020wav2vec}, have proven to be effective for learning expressive and robust representations of speech." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="1904.05862" id="1117">
        <attvalues>
          <attvalue for="5" value=" CPC and its extensions~\cite{riviere2020unsupervised,schneider2019wav2vec,baevski2020vq,baevski2020wav2vec}, have proven to be effective for learning expressive and robust representations of speech." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="1910.05453" id="1118">
        <attvalues>
          <attvalue for="5" value=" CPC and its extensions~\cite{riviere2020unsupervised,schneider2019wav2vec,baevski2020vq,baevski2020wav2vec}, have proven to be effective for learning expressive and robust representations of speech." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="2006.11477" id="1119">
        <attvalues>
          <attvalue for="5" value=" CPC and its extensions~\cite{riviere2020unsupervised,schneider2019wav2vec,baevski2020vq,baevski2020wav2vec}, have proven to be effective for learning expressive and robust representations of speech." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="1904.03240" id="1120">
        <attvalues>
          <attvalue for="5" value=" Instead of targeting future latent representations, Autoregressive Predictive Coding (APC)~\cite{chung2019unsupervised} suggests that simply predicting future surface features is suitable for learning an effective representation of speech." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="2001.10603" id="1122">
        <attvalues>
          <attvalue for="5" value=" In practice, a bidirectional RNN~\cite{wang2020unsupervised} or Transformer encoder~\cite{liu2020mockingjay} can be employed in learning speech representation through MLM." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="1910.12638" id="1123">
        <attvalues>
          <attvalue for="5" value=" In practice, a bidirectional RNN~\cite{wang2020unsupervised} or Transformer encoder~\cite{liu2020mockingjay} can be employed in learning speech representation through MLM." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="2004.05274" id="1124">
        <attvalues>
          <attvalue for="5" value=" APC can be improved by enforcing constraints that information from past sequences be stored in the representation~\cite{chung2020improved} or by imposing an information bottleneck via vector quantization~\cite{chung2020vector}." />
        </attvalues>
      </edge>
      <edge source="2011.00406" target="2005.08392" id="1125">
        <attvalues>
          <attvalue for="5" value=" APC can be improved by enforcing constraints that information from past sequences be stored in the representation~\cite{chung2020improved} or by imposing an information bottleneck via vector quantization~\cite{chung2020vector}." />
        </attvalues>
      </edge>
      <edge source="2103.08393" target="1904.05862" id="1126">
        <attvalues>
          <attvalue for="5" value=" \item Self-supervised learning has been shown to be useful for settings with little labeled data \cite{schneider2019wav2vec,ravanelli2020multi}." />
        </attvalues>
      </edge>
      <edge source="2103.08393" target="1807.03748" id="1128">
        <attvalues>
          <attvalue for="5" value=" Self-supervision \cite{schneider2019wav2vec,chen2020big,oord2018representation,lan2019albert} is a paradigm of machine learning (ML) that deals with unsupervised learning of structural patterns in data by exploiting contextual information." />
        </attvalues>
      </edge>
      <edge source="2103.08393" target="1909.11942" id="1129">
        <attvalues>
          <attvalue for="5" value=" Self-supervision \cite{schneider2019wav2vec,chen2020big,oord2018representation,lan2019albert} is a paradigm of machine learning (ML) that deals with unsupervised learning of structural patterns in data by exploiting contextual information." />
        </attvalues>
      </edge>
      <edge source="2103.08393" target="2006.11477" id="1130">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} is one such self-supervised learning model that learns to predict masked out discrete speech encodings using a contextualized representation from a transformer model \cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2103.08393" target="1706.03762" id="1131">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} is one such self-supervised learning model that learns to predict masked out discrete speech encodings using a contextualized representation from a transformer model \cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2103.08393" target="1910.12607" id="1132">
        <attvalues>
          <attvalue for="5" value=" It has been observed that the effectiveness of self-supervision decreases as the amount of labeled data increases\cite{amazon,chung2020generative}." />
        </attvalues>
      </edge>
      <edge source="2103.08393" target="1910.05453" id="1133">
        <attvalues>
          <attvalue for="5" value=" \\ Our main contributions in this paper are \begin{itemize}\item The wav2vec-C model (Section \ref{sec:model}) \item We use real world far-field voice query speech with varied degrees of SNR ranging between -40 to 50 dB, whereas most studies on self-supervised learning in the literature use clean read speech \cite{chung2020generative,baevski2019vq} and some use simulated noisy speech \cite{ravanelli2020multi}." />
        </attvalues>
      </edge>
      <edge source="2103.08393" target="1912.01679" id="1135">
        <attvalues>
          <attvalue for="5" value=" It has been observed that the effectiveness of self-supervision decreases as the amount of labeled data increases\cite{amazon,chung2020generative}." />
        </attvalues>
      </edge>
      <edge source="2312.10523" target="2402.00159" id="1136">
        <attvalues>
          <attvalue for="5" value=" Among the \numTasks sources curated in our benchmark, we contribute two new datasets constructed from data held out of \dolma \cite{dolma}: (1) a subsample of the top 100 subreddits by number of comments, and (2) code from the top 100 programming languages by number of tokens." />
        </attvalues>
      </edge>
      <edge source="2305.13862" target="1706.03762" id="1137">
        <attvalues>
          <attvalue for="5" value=" Transformer-based language models \cite{transfVaswani}, which have disrupted classical NLP pipeline \cite{tenney2019bert}, have grown in size and capabilities in recent years." />
        </attvalues>
      </edge>
      <edge source="2305.13862" target="1905.05950" id="1138">
        <attvalues>
          <attvalue for="5" value=" Transformer-based language models \cite{transfVaswani}, which have disrupted classical NLP pipeline \cite{tenney2019bert}, have grown in size and capabilities in recent years." />
        </attvalues>
      </edge>
      <edge source="2305.13862" target="2106.13219" id="1140">
        <attvalues>
          <attvalue for="5" value=" Therefore, learned representations may inherit the biases and stereotypical associations present in the large text corpora in the language and, thus, in the pre-training corpora taken from the web \cite{liang2021understanding}." />
        </attvalues>
      </edge>
      <edge source="2305.13862" target="2004.09456" id="1141">
        <attvalues>
          <attvalue for="5" value=" It has been observed that as the size of a model increases, its linguistic modeling capabilities and biases increase \cite{nadeem-etal-2021-stereoset}." />
        </attvalues>
      </edge>
      <edge source="2305.13862" target="2205.01068" id="1142">
        <attvalues>
          <attvalue for="5" value=" By testing the 7-billion-parameter LLaMA model and Open Pre-trained Transformer Language Models (OPT) \cite{zhang2022opt}, we show that although the model shows less biased behavior after fine-tuning, the method also achieves a reasonable overall performance of the language model." />
        </attvalues>
      </edge>
      <edge source="2205.06168" target="1905.01896" id="1143">
        <attvalues>
          <attvalue for="5" value=" Distributional semantics models create word embeddings based on the assumption that the meaning of a word is defined by the contexts it is used in (for an overview, see: \cite{sahlgren2008distributional,lenci2018distributional,boleda2020distributional,emerson-2020-goals})." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2303.08774" id="1144">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, today's most performant LLMs such as ChatGPT and GPT4 \cite{2022OpenAIchatgpt,2023GPT4Openai} only provide APIs for temperature and top-$p$ sampling, seemingly overlooking the potential benefits of other advanced decoding methods." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2307.09288" id="1146">
        <attvalues>
          <attvalue for="5" value=" The advent of large language models (LLMs)~\cite[][inter alia]{2022OpenAIchatgpt,2023GPT4Openai,touvron2023llama,touvron2023llama2} has ushered in a new era of natural language processing (NLP)." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2309.09117" id="1147">
        <attvalues>
          <attvalue for="5" value=" Recent studies have shown that the choice of decoding methods can substantially impact the performance of LLMs \cite{o2023contrastive,chuang2023dola}." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2210.15097" id="1149">
        <attvalues>
          <attvalue for="5" value=" In addition, a plethora of new decoding methods \cite{su2022a,li2022contrastive,yang2023frustratingly,meister-etal-2023-locally,hewitt2022truncation,basu2021mirostat} have been proposed afterward, each claiming to outperform the previous state-of-the-art in particular tasks." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="1906.06362" id="1150">
        <attvalues>
          <attvalue for="5" value=" Notably, \cite{ippolito-etal-2019-comparison,wiher2022decoding} provide a comparative analysis of various decoding methods using task-specific language models." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2203.15721" id="1151">
        <attvalues>
          <attvalue for="5" value=" Notably, \cite{ippolito-etal-2019-comparison,wiher2022decoding} provide a comparative analysis of various decoding methods using task-specific language models." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="1904.09751" id="1152">
        <attvalues>
          <attvalue for="5" value=" They find that deterministic decoding methods (eg, beam search) perform better than stochastic decoding methods (eg, top-$p$ sampling~\cite{Holtzman2020The}) in closed-ended generation tasks such as machine translation, while the inverse is true for open-ended generation tasks such as story generation." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2202.06417" id="1153">
        <attvalues>
          <attvalue for="5" value=" In addition, a plethora of new decoding methods \cite{su2022a,li2022contrastive,yang2023frustratingly,meister-etal-2023-locally,hewitt2022truncation,basu2021mirostat} have been proposed afterward, each claiming to outperform the previous state-of-the-art in particular tasks." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2305.12675" id="1154">
        <attvalues>
          <attvalue for="5" value=" \item Speed Stochastic decoding and the recently proposed deterministic method, frustratingly simple decoding (FSD)~\cite{yang2023frustratingly}, can achieve a similar decoding speed to greedy search." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2202.00666" id="1155">
        <attvalues>
          <attvalue for="5" value=" In addition, a plethora of new decoding methods \cite{su2022a,li2022contrastive,yang2023frustratingly,meister-etal-2023-locally,hewitt2022truncation,basu2021mirostat} have been proposed afterward, each claiming to outperform the previous state-of-the-art in particular tasks." />
        </attvalues>
      </edge>
      <edge source="2402.06925" target="2210.15191" id="1156">
        <attvalues>
          <attvalue for="5" value=" In addition, a plethora of new decoding methods \cite{su2022a,li2022contrastive,yang2023frustratingly,meister-etal-2023-locally,hewitt2022truncation,basu2021mirostat} have been proposed afterward, each claiming to outperform the previous state-of-the-art in particular tasks." />
        </attvalues>
      </edge>
      <edge source="2304.01933" target="2303.08774" id="1158">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as ChatGPT~\cite{openai-chatgpt-2022} and GPT-4~\cite{openai-gpt4-2023}, have demonstrated unprecedented performance across various natural language processing (NLP) tasks~\cite{qin-chatgpt-2023} and multi-modal tasks~\cite{hugginggpt}." />
        </attvalues>
      </edge>
      <edge source="2304.01933" target="2302.06476" id="1159">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as ChatGPT~\cite{openai-chatgpt-2022} and GPT-4~\cite{openai-gpt4-2023}, have demonstrated unprecedented performance across various natural language processing (NLP) tasks~\cite{qin-chatgpt-2023} and multi-modal tasks~\cite{hugginggpt}." />
        </attvalues>
      </edge>
      <edge source="2304.01933" target="1902.00751" id="1161">
        <attvalues>
          <attvalue for="5" value=" As shown in Table~\ref{tab:peft_category}, the advantage of PEFT has resulted in the developing of diverse PEFT modules, encompassing series adapters~\cite{adapters, adamix, sparseadapter, lets}, parallel adapters~\cite{parallel_adapter}, reparameterization-based methods~\cite{lora, krona}, and prompt-based learning methods~\cite{prompt_tuning, prefix}." />
        </attvalues>
      </edge>
      <edge source="2304.01933" target="2110.04366" id="1163">
        <attvalues>
          <attvalue for="5" value=" As shown in Table~\ref{tab:peft_category}, the advantage of PEFT has resulted in the developing of diverse PEFT modules, encompassing series adapters~\cite{adapters, adamix, sparseadapter, lets}, parallel adapters~\cite{parallel_adapter}, reparameterization-based methods~\cite{lora, krona}, and prompt-based learning methods~\cite{prompt_tuning, prefix}." />
        </attvalues>
      </edge>
      <edge source="2304.01933" target="2106.09685" id="1164">
        <attvalues>
          <attvalue for="5" value=" As shown in Table~\ref{tab:peft_category}, the advantage of PEFT has resulted in the developing of diverse PEFT modules, encompassing series adapters~\cite{adapters, adamix, sparseadapter, lets}, parallel adapters~\cite{parallel_adapter}, reparameterization-based methods~\cite{lora, krona}, and prompt-based learning methods~\cite{prompt_tuning, prefix}." />
        </attvalues>
      </edge>
      <edge source="2304.01933" target="2104.08691" id="1165">
        <attvalues>
          <attvalue for="5" value=" As shown in Table~\ref{tab:peft_category}, the advantage of PEFT has resulted in the developing of diverse PEFT modules, encompassing series adapters~\cite{adapters, adamix, sparseadapter, lets}, parallel adapters~\cite{parallel_adapter}, reparameterization-based methods~\cite{lora, krona}, and prompt-based learning methods~\cite{prompt_tuning, prefix}." />
        </attvalues>
      </edge>
      <edge source="2304.01933" target="2211.01786" id="1166">
        <attvalues>
          <attvalue for="5" value=" Motivated by this, in this paper, we conduct a comprehensive empirical study of PEFT of three representative open-source LLMs, including BLOOM~\cite{bloom}, GPT-J~\cite{gpt-j}, and LLaMA~\cite{llama}." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="1802.05365" id="1167">
        <attvalues>
          <attvalue for="5" value=" Transfer learning from pre-trained language models (PLMs) is now the prevalent paradigm in natural language processing, yielding strong performance on many tasks~\cite{peters2018deep,devlin2019bert,qiu2020pre}." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="2003.08271" id="1169">
        <attvalues>
          <attvalue for="5" value=" Transfer learning from pre-trained language models (PLMs) is now the prevalent paradigm in natural language processing, yielding strong performance on many tasks~\cite{peters2018deep,devlin2019bert,qiu2020pre}." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="1910.13461" id="1170">
        <attvalues>
          <attvalue for="5" value=" This issue is particularly salient with the ever-increasing size of PLMs, which now range from hundreds of millions~\cite{radford2019language,lewis-etal-2020-bart} to hundreds of billions~\cite{brown2020language} or even trillions of parameters~\cite{fedus2021switch}." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="2005.14165" id="1171">
        <attvalues>
          <attvalue for="5" value=" Inspired by the success of prompting methods that control PLMs through textual prompts~\cite{brown2020language,liu2021pre}, prefix tuning~\cite{li2021prefix} and prompt tuning~\cite{lester2021power} prepend an additional $l$ tunable prefix tokens to the input or hidden layers and only train these soft prompts when fine-tuning on downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="2101.03961" id="1172">
        <attvalues>
          <attvalue for="5" value=" This issue is particularly salient with the ever-increasing size of PLMs, which now range from hundreds of millions~\cite{radford2019language,lewis-etal-2020-bart} to hundreds of billions~\cite{brown2020language} or even trillions of parameters~\cite{fedus2021switch}." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="1902.00751" id="1173">
        <attvalues>
          <attvalue for="5" value=" For example, adapter tuning~\cite{houlsby2019parameter} inserts small neural modules called adapters to each layer of the pretrained network and only the adapters are trained at fine-tuning time." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="2107.13586" id="1174">
        <attvalues>
          <attvalue for="5" value=" Inspired by the success of prompting methods that control PLMs through textual prompts~\cite{brown2020language,liu2021pre}, prefix tuning~\cite{li2021prefix} and prompt tuning~\cite{lester2021power} prepend an additional $l$ tunable prefix tokens to the input or hidden layers and only train these soft prompts when fine-tuning on downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="2101.00190" id="1175">
        <attvalues>
          <attvalue for="5" value=" Besides parameter savings, parameter-efficient tuning makes it possible to quickly adapt to new tasks without catastrophic forgetting~\cite{pfeiffer2021adapterfusion} and often exhibits superior robustness in out-of-distribution evaluation~\cite{li2021prefix}." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="2104.08691" id="1176">
        <attvalues>
          <attvalue for="5" value=" Inspired by the success of prompting methods that control PLMs through textual prompts~\cite{brown2020language,liu2021pre}, prefix tuning~\cite{li2021prefix} and prompt tuning~\cite{lester2021power} prepend an additional $l$ tunable prefix tokens to the input or hidden layers and only train these soft prompts when fine-tuning on downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="2106.09685" id="1177">
        <attvalues>
          <attvalue for="5" value=" More recently,~\cite{hu2021lora} learn low-rank matrices to approximate parameter updates." />
        </attvalues>
      </edge>
      <edge source="2110.04366" target="2005.00247" id="1178">
        <attvalues>
          <attvalue for="5" value=" Besides parameter savings, parameter-efficient tuning makes it possible to quickly adapt to new tasks without catastrophic forgetting~\cite{pfeiffer2021adapterfusion} and often exhibits superior robustness in out-of-distribution evaluation~\cite{li2021prefix}." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1811.01088" id="1181">
        <attvalues>
          <attvalue for="5" value=" Several works~\cite{phang2018sentence,garg2019tanda,dodge2020finetuning,Lee2020Mixout} have demonstrated that FT with a few target domain samples is unstable with high variance, thereby often leading to sub-par gains." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1911.04118" id="1182">
        <attvalues>
          <attvalue for="5" value=" Several works~\cite{phang2018sentence,garg2019tanda,dodge2020finetuning,Lee2020Mixout} have demonstrated that FT with a few target domain samples is unstable with high variance, thereby often leading to sub-par gains." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="2002.06305" id="1183">
        <attvalues>
          <attvalue for="5" value=" Several works~\cite{phang2018sentence,garg2019tanda,dodge2020finetuning,Lee2020Mixout} have demonstrated that FT with a few target domain samples is unstable with high variance, thereby often leading to sub-par gains." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1705.02364" id="1185">
        <attvalues>
          <attvalue for="5" value=" Related Work Recently, several pre-trained models have been studied, of which some provide explicit sentence embeddings~\cite{conneau2017supervised,subramanian2018learning}, while others provide implicit ones~\cite{howard2018universal,radford2018improving}." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1804.00079" id="1186">
        <attvalues>
          <attvalue for="5" value=" Related Work Recently, several pre-trained models have been studied, of which some provide explicit sentence embeddings~\cite{conneau2017supervised,subramanian2018learning}, while others provide implicit ones~\cite{howard2018universal,radford2018improving}." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1801.06146" id="1187">
        <attvalues>
          <attvalue for="5" value=" Related Work Recently, several pre-trained models have been studied, of which some provide explicit sentence embeddings~\cite{conneau2017supervised,subramanian2018learning}, while others provide implicit ones~\cite{howard2018universal,radford2018improving}." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1903.05987" id="1188">
        <attvalues>
          <attvalue for="5" value=" \cite{peters2019tune} compare the performance of feature extraction (by freezing the pre-trained weights) and FT." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1904.02232" id="1189">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{xu2019bert} ``post-train&quot; the pre-trained model on the target dataset, \cite{houlsby2019parameter} inject specifically designed new adapter layers, \cite{arase-tsujii-2019-transfer} inject phrasal paraphrase relations into BERT, \cite{DBLP:journals/corr/abs-1905-05583} use multi-task FT, and \cite{wang2019to} first train a deep network classifier on the fixed pre-trained embedding and then fine-tune it." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1902.00751" id="1190">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{xu2019bert} ``post-train&quot; the pre-trained model on the target dataset, \cite{houlsby2019parameter} inject specifically designed new adapter layers, \cite{arase-tsujii-2019-transfer} inject phrasal paraphrase relations into BERT, \cite{DBLP:journals/corr/abs-1905-05583} use multi-task FT, and \cite{wang2019to} first train a deep network classifier on the fixed pre-trained embedding and then fine-tune it." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1909.00931" id="1191">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{xu2019bert} ``post-train&quot; the pre-trained model on the target dataset, \cite{houlsby2019parameter} inject specifically designed new adapter layers, \cite{arase-tsujii-2019-transfer} inject phrasal paraphrase relations into BERT, \cite{DBLP:journals/corr/abs-1905-05583} use multi-task FT, and \cite{wang2019to} first train a deep network classifier on the fixed pre-trained embedding and then fine-tune it." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1905.05583" id="1192">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{xu2019bert} ``post-train&quot; the pre-trained model on the target dataset, \cite{houlsby2019parameter} inject specifically designed new adapter layers, \cite{arase-tsujii-2019-transfer} inject phrasal paraphrase relations into BERT, \cite{DBLP:journals/corr/abs-1905-05583} use multi-task FT, and \cite{wang2019to} first train a deep network classifier on the fixed pre-trained embedding and then fine-tune it." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1907.05338" id="1193">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{xu2019bert} ``post-train&quot; the pre-trained model on the target dataset, \cite{houlsby2019parameter} inject specifically designed new adapter layers, \cite{arase-tsujii-2019-transfer} inject phrasal paraphrase relations into BERT, \cite{DBLP:journals/corr/abs-1905-05583} use multi-task FT, and \cite{wang2019to} first train a deep network classifier on the fixed pre-trained embedding and then fine-tune it." />
        </attvalues>
      </edge>
      <edge source="2004.05119" target="1802.05365" id="1194">
        <attvalues>
          <attvalue for="5" value=" While the idea of concatenating multiple embeddings has been previously used~\cite{peters2018deep}, we use it for transfer learning in a low resource target domain." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2005.14165" id="1195">
        <attvalues>
          <attvalue for="5" value=" Recent years have witnessed a trend toward scaling neural networks~\cite{gpt3,scalinglaw,scalingvit,palm,vit22b}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2001.08361" id="1196">
        <attvalues>
          <attvalue for="5" value=" Recent years have witnessed a trend toward scaling neural networks~\cite{gpt3,scalinglaw,scalingvit,palm,vit22b}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2106.04560" id="1197">
        <attvalues>
          <attvalue for="5" value=" Recent years have witnessed a trend toward scaling neural networks~\cite{gpt3,scalinglaw,scalingvit,palm,vit22b}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2204.02311" id="1198">
        <attvalues>
          <attvalue for="5" value=" Recent years have witnessed a trend toward scaling neural networks~\cite{gpt3,scalinglaw,scalingvit,palm,vit22b}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2302.05442" id="1199">
        <attvalues>
          <attvalue for="5" value=" Recent years have witnessed a trend toward scaling neural networks~\cite{gpt3,scalinglaw,scalingvit,palm,vit22b}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="1811.06965" id="1201">
        <attvalues>
          <attvalue for="5" value=" The depth is primarily scaled up for exponential expressivity, producing many powerful deep networks~\cite{resnet,gpipe,deepnet}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2203.00555" id="1202">
        <attvalues>
          <attvalue for="5" value=" The depth is primarily scaled up for exponential expressivity, producing many powerful deep networks~\cite{resnet,gpipe,deepnet}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2006.16668" id="1203">
        <attvalues>
          <attvalue for="5" value=" Then, the sparse MoE models~\cite{gshard,switch,stmoe} and model parallelism approaches~\cite{megatron,reduceact} efficiently enlarge the hidden dimension." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2101.03961" id="1204">
        <attvalues>
          <attvalue for="5" value=" Then, the sparse MoE models~\cite{gshard,switch,stmoe} and model parallelism approaches~\cite{megatron,reduceact} efficiently enlarge the hidden dimension." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="1909.08053" id="1205">
        <attvalues>
          <attvalue for="5" value=" Then, the sparse MoE models~\cite{gshard,switch,stmoe} and model parallelism approaches~\cite{megatron,reduceact} efficiently enlarge the hidden dimension." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2205.05198" id="1206">
        <attvalues>
          <attvalue for="5" value=" Then, the sparse MoE models~\cite{gshard,switch,stmoe} and model parallelism approaches~\cite{megatron,reduceact} efficiently enlarge the hidden dimension." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2208.04933" id="1208">
        <attvalues>
          <attvalue for="5" value=" More recently, state space models~\cite{s4,s5,h3,hyena} are appealing to sequence modeling." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2212.14052" id="1209">
        <attvalues>
          <attvalue for="5" value=" More recently, state space models~\cite{s4,s5,h3,hyena} are appealing to sequence modeling." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2302.10866" id="1210">
        <attvalues>
          <attvalue for="5" value=" More recently, state space models~\cite{s4,s5,h3,hyena} are appealing to sequence modeling." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="1904.10509" id="1211">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{sparsetransformer} obtains $\mathcal{O}(N\sqrt{N}d)$ time complexity with a fixed sparse pattern." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2007.14062" id="1212">
        <attvalues>
          <attvalue for="5" value=" Besides the heuristic patterns~\cite{bigbird,longformer}, the learnable patterns prove to be useful for sparse attention~\cite{reformer,colt5}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2004.05150" id="1213">
        <attvalues>
          <attvalue for="5" value=" Besides the heuristic patterns~\cite{bigbird,longformer}, the learnable patterns prove to be useful for sparse attention~\cite{reformer,colt5}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2001.04451" id="1214">
        <attvalues>
          <attvalue for="5" value=" Besides the heuristic patterns~\cite{bigbird,longformer}, the learnable patterns prove to be useful for sparse attention~\cite{reformer,colt5}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2303.09752" id="1215">
        <attvalues>
          <attvalue for="5" value=" Besides the heuristic patterns~\cite{bigbird,longformer}, the learnable patterns prove to be useful for sparse attention~\cite{reformer,colt5}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2006.04768" id="1216">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="1910.13923" id="1217">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2006.16236" id="1218">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2009.14794" id="1219">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2210.10340" id="1220">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="1810.00825" id="1221">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2106.01540" id="1223">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="1901.02860" id="1224">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2304.11062" id="1225">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2203.08913" id="1226">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2307.02486" target="2306.07174" id="1227">
        <attvalues>
          <attvalue for="5" value=" There are also some other efficient Transformer-based variants, including low-rank attention~\cite{linformer,low-rank}, kernel-based methods~\cite{lineartransformer,performer,normformer}, downsampling approaches~\cite{settransformer,perceiver,luna}, recurrent models~\cite{trm-xl,rmt}, and retrieval-based methods~\cite{memorizingtrm,longmem}." />
        </attvalues>
      </edge>
      <edge source="2010.12198" target="2005.00870" id="1228">
        <attvalues>
          <attvalue for="5" value=" Divergence also has practical applications in predicting the performance drop of a model when adapted to new domains~\cite{van-asch-daelemans-2010-using}, and in choosing among alternate models~\cite{xia2020predicting}." />
        </attvalues>
      </edge>
      <edge source="2010.12198" target="1806.00258" id="1229">
        <attvalues>
          <attvalue for="5" value=" Unlike previous surveys, which focus on domain adaptation for specific tasks such as machine translation~\cite{chu-wang-2018-survey} and statistical (non-neural network) models \cite{Jiang2007ALS,Margolis2011ALR}, our work takes a different perspective." />
        </attvalues>
      </edge>
      <edge source="2308.15231" target="1909.06749" id="1249">
        <attvalues>
          <attvalue for="5" value=" Spoken Dialogue Systems (SDSs) are increasingly being embedded in social robots that are expected to seamlessly interact with people in populated public spaces like museums, airports, shopping centres, or hospital waiting rooms \cite{foster2019mummer,tian2021redesigning,gunson2022visually}." />
        </attvalues>
      </edge>
      <edge source="2308.15231" target="2109.02492" id="1250">
        <attvalues>
          <attvalue for="5" value=" In these multi-party scenarios, tasks that are considered trivial for SDSs become substantially more complex \cite{traum2004issues,zhong2022dialoglm,addlesee2023data}." />
        </attvalues>
      </edge>
      <edge source="2308.15231" target="2106.01541" id="1252">
        <attvalues>
          <attvalue for="5" value=" In multi-party conversations (MPCs), the social robot must determine which user said an utterance, who that utterance was directed to, when to respond, and what it should say depending on whom the robot is addressing \cite{hu2019gsn,gu2021mpc,gu2022hetermpc}." />
        </attvalues>
      </edge>
      <edge source="2308.15231" target="2203.08500" id="1253">
        <attvalues>
          <attvalue for="5" value=" In multi-party conversations (MPCs), the social robot must determine which user said an utterance, who that utterance was directed to, when to respond, and what it should say depending on whom the robot is addressing \cite{hu2019gsn,gu2021mpc,gu2022hetermpc}." />
        </attvalues>
      </edge>
      <edge source="2308.15231" target="2204.06677" id="1255">
        <attvalues>
          <attvalue for="5" value=" Dialogue State Tracking (DST) is a well-established task \cite{lee2021dialogue,feng2022dynamic} that is considered crucial to the success of a dialogue system \cite{williams2016dialog}." />
        </attvalues>
      </edge>
      <edge source="2108.11193" target="1508.07909" id="1256">
        <attvalues>
          <attvalue for="5" value=" Contemporary subword tokenization algorithms such as BPE \cite{sennrich-etal-2016-neural} partition a string into contiguous spans of characters." />
        </attvalues>
      </edge>
      <edge source="2108.11193" target="1907.11692" id="1257">
        <attvalues>
          <attvalue for="5" value="8\% of the held-out vocabulary for RoBERTa-Large \cite{liu2019roberta}, 32." />
        </attvalues>
      </edge>
      <edge source="2403.16393" target="2303.18223" id="1259">
        <attvalues>
          <attvalue for="5" value=" L{arge} language models (LLMs) are becoming a centerpiece of many applications and services; moreover, this trend is expected to continue in the coming years \cite{LLMsurvey}." />
        </attvalues>
      </edge>
      <edge source="2403.16393" target="2303.08774" id="1262">
        <attvalues>
          <attvalue for="5" value=" Examples of LLMs include commercial models such as GPT \cite{openai2023gpt4} or Gemini \cite{geminiteam2023gemini} and open-source models such as LLama \cite{LLAMA2} or Mistral \cite{Mistral}." />
        </attvalues>
      </edge>
      <edge source="2403.16393" target="2312.11805" id="1263">
        <attvalues>
          <attvalue for="5" value=" Examples of LLMs include commercial models such as GPT \cite{openai2023gpt4} or Gemini \cite{geminiteam2023gemini} and open-source models such as LLama \cite{LLAMA2} or Mistral \cite{Mistral}." />
        </attvalues>
      </edge>
      <edge source="2403.16393" target="2307.09288" id="1264">
        <attvalues>
          <attvalue for="5" value=" Examples of LLMs include commercial models such as GPT \cite{openai2023gpt4} or Gemini \cite{geminiteam2023gemini} and open-source models such as LLama \cite{LLAMA2} or Mistral \cite{Mistral}." />
        </attvalues>
      </edge>
      <edge source="2403.16393" target="2310.06825" id="1265">
        <attvalues>
          <attvalue for="5" value=" Examples of LLMs include commercial models such as GPT \cite{openai2023gpt4} or Gemini \cite{geminiteam2023gemini} and open-source models such as LLama \cite{LLAMA2} or Mistral \cite{Mistral}." />
        </attvalues>
      </edge>
      <edge source="2403.16393" target="2206.02051" id="1266">
        <attvalues>
          <attvalue for="5" value=" The impact of errors on neural networks has been evaluated extensively both by simulation and by radiation testing \cite{NN_Errors_0},\cite{NN_errors1} showing that they have some intrinsic tolerance to errors, especially when a fixed-point representation is used for the parameters and arithmetic operations." />
        </attvalues>
      </edge>
      <edge source="2403.16393" target="2306.01820" id="1267">
        <attvalues>
          <attvalue for="5" value=" Error detection using a concurrent classifier has been proposed for large-scale machine learning systems in \cite{CCED} and evaluated for BERT in question and answering and emotion classification, showing good detection capabilities." />
        </attvalues>
      </edge>
      <edge source="2305.15282" target="2005.14165" id="1268">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) with parameters in the order of billions \cite{brown_language_2020} have gained significant attention in recent years due to their strong performance on a wide range of natural language processing tasks." />
        </attvalues>
      </edge>
      <edge source="2305.15282" target="2004.02235" id="1269">
        <attvalues>
          <attvalue for="5" value=" As a result, the long-tail problem \cite{gzs-longtail} has been overlooked." />
        </attvalues>
      </edge>
      <edge source="2305.15282" target="1605.05362" id="1270">
        <attvalues>
          <attvalue for="5" value=" This is further highlighted by platforms in which a systematic structure is not created for users; parent nodes may be in place of child nodes, and vice versa \cite{DBLP:journals/corr/Asghar16}." />
        </attvalues>
      </edge>
      <edge source="2305.15282" target="2104.01666" id="1271">
        <attvalues>
          <attvalue for="5" value=" To tackle this problem, we suggest refactoring traditional hierarchical flat-labeled prediction tasks \cite{liu_improving_2021} to a more indicative long-tail prediction task." />
        </attvalues>
      </edge>
      <edge source="2009.14124" target="1802.05365" id="1273">
        <attvalues>
          <attvalue for="5" value=" Such language models include BERT \cite{devlin-etal-2019-bert} and ELMo \cite{peters-etal-2018-deep}, which are conventionally ``pretrained'' on large unlabeled datasets before their internal representations are ``finetuned'' during supervised training on downstream tasks like parsing." />
        </attvalues>
      </edge>
      <edge source="2009.14124" target="1901.07291" id="1274">
        <attvalues>
          <attvalue for="5" value=" One exciting way to compensate for the lack of unlabeled data in low-resource language varieties is to finetune a large, multilingual language model that has been pretrained on the union of many languages' data \cite{devlin-etal-2019-bert,lample2019cross}." />
        </attvalues>
      </edge>
      <edge source="2009.14124" target="1911.02116" id="1275">
        <attvalues>
          <attvalue for="5" value=" Specifically, multilingual models face the transfer-dilution tradeoff \cite{conneau2019unsupervised}: increasing the number of languages during pretraining improves positive crosslingual transfer but decreases the model capacity allocated to each language." />
        </attvalues>
      </edge>
      <edge source="2009.14124" target="2004.00033" id="1277">
        <attvalues>
          <attvalue for="5" value=" This enables the model to transfer some of what it learns from high-resource languages to low-resource ones, demonstrating benefits over monolingual methods in some cases \cite{conneau2019unsupervised,tsai-etal-2019-small}, though not always \cite{agerri2020text,ronnqvist-etal-2019-multilingual}." />
        </attvalues>
      </edge>
      <edge source="2009.14124" target="2005.09093" id="1279">
        <attvalues>
          <attvalue for="5" value=" Indeed, \cite{wu-dredze-2020-languages} find that multilingual models often underperform monolingual baselines for such languages and question their off-the-shelf viability." />
        </attvalues>
      </edge>
      <edge source="2009.14124" target="1508.07909" id="1281">
        <attvalues>
          <attvalue for="5" value=" However, additional pretraining on more data in the target language does not ensure its full representation in the model's vocabulary, which is constructed to maximally represent the model's original pretraining data \cite{sennrich-etal-2016-neural,wu2016googles}." />
        </attvalues>
      </edge>
      <edge source="2009.14124" target="1609.08144" id="1282">
        <attvalues>
          <attvalue for="5" value=" However, additional pretraining on more data in the target language does not ensure its full representation in the model's vocabulary, which is constructed to maximally represent the model's original pretraining data \cite{sennrich-etal-2016-neural,wu2016googles}." />
        </attvalues>
      </edge>
      <edge source="2405.10650" target="1703.09902" id="1284">
        <attvalues>
          <attvalue for="5" value=" Data-to-text generation \cite{gen-survey} is an important task in natural language generation (NLG)." />
        </attvalues>
      </edge>
      <edge source="2405.10650" target="1707.08052" id="1285">
        <attvalues>
          <attvalue for="5" value=" It aims to generate fluent and faithful text based on structured data input and is critical in many NLG systems, such as report generation \cite{rotowire}, oriented dialogues \cite{Self}, etc In data-to-text generation, structured data input is compositional, eg, it can be considered as a combination of elements formed according to certain rules." />
        </attvalues>
      </edge>
      <edge source="2405.10650" target="2110.08467" id="1286">
        <attvalues>
          <attvalue for="5" value=" In semantic parsing and mathematical reasoning tasks, many different manifestations of this ability have been studied \cite{PCFG, Trans}, such as systematicity (handle combinations unseen during training), productivity (extrapolate to longer sequences than those seen during training), etc For compositional generalization in data-to-text generation, only systematicity receives attention \cite{Self}, and research on other manifestations is lacking." />
        </attvalues>
      </edge>
      <edge source="2405.10650" target="2210.03050" id="1287">
        <attvalues>
          <attvalue for="5" value=" Therefore, in order to handle the practical data-to-text generation, the language models should have the ability to recombine previously learned elements with certain rules to map new inputs made up from these elements to their correct output \cite{summary}, which is the so-called compositional generalization." />
        </attvalues>
      </edge>
      <edge source="2405.10650" target="1912.09713" id="1288">
        <attvalues>
          <attvalue for="5" value=" Based on existing datasets, we mainly perform repartition \cite{CFQ} and element modification to construct datasets for our evaluation." />
        </attvalues>
      </edge>
      <edge source="2405.10650" target="2106.09685" id="1289">
        <attvalues>
          <attvalue for="5" value=" Nowadays, advanced Parameter-Efficient Fine-Tuning such as LoRA \cite{LoRA} provides the methods, and the consideration of LLMs becomes necessary." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2110.08207" id="1291">
        <attvalues>
          <attvalue for="5" value=" Especially when text generators are massively scaled up and tuned on human instructions, they acquire impressive capabilities to generalise to new tasks without requiring task-specific fine-tuning \cite{sanh_multitask_2022,ouyang_training_2022,chung_scaling_2022,openai_gpt-4_2023,touvron_llama_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2210.11416" id="1293">
        <attvalues>
          <attvalue for="5" value=" Especially when text generators are massively scaled up and tuned on human instructions, they acquire impressive capabilities to generalise to new tasks without requiring task-specific fine-tuning \cite{sanh_multitask_2022,ouyang_training_2022,chung_scaling_2022,openai_gpt-4_2023,touvron_llama_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2303.08774" id="1294">
        <attvalues>
          <attvalue for="5" value=" Especially when text generators are massively scaled up and tuned on human instructions, they acquire impressive capabilities to generalise to new tasks without requiring task-specific fine-tuning \cite{sanh_multitask_2022,ouyang_training_2022,chung_scaling_2022,openai_gpt-4_2023,touvron_llama_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2307.09288" id="1295">
        <attvalues>
          <attvalue for="5" value=" Especially when text generators are massively scaled up and tuned on human instructions, they acquire impressive capabilities to generalise to new tasks without requiring task-specific fine-tuning \cite{sanh_multitask_2022,ouyang_training_2022,chung_scaling_2022,openai_gpt-4_2023,touvron_llama_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2009.07118" id="1297">
        <attvalues>
          <attvalue for="5" value=" The literature has developed several other universal tasks that cannot solve generative tasks (summarization, translation etc), but can solve any classification task with smaller size and performance competitive with generative LLMs \cite{xu_universal_2023,schick_its_2021}." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2205.15223" id="1298">
        <attvalues>
          <attvalue for="5" value=" While several efficient approaches to universal classification exist \cite{schick_exploiting_2021,xia_prompting_2022,yao_prompt_2022,xu_universal_2023,bragg_flex_2021,ma_issues_2021,sun_nsp-bert_2022}, this paper focuses on guidance for one approach: Natural Language Inference." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2205.11166" id="1299">
        <attvalues>
          <attvalue for="5" value=" While several efficient approaches to universal classification exist \cite{schick_exploiting_2021,xia_prompting_2022,yao_prompt_2022,xu_universal_2023,bragg_flex_2021,ma_issues_2021,sun_nsp-bert_2022}, this paper focuses on guidance for one approach: Natural Language Inference." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2107.07170" id="1300">
        <attvalues>
          <attvalue for="5" value=" While several efficient approaches to universal classification exist \cite{schick_exploiting_2021,xia_prompting_2022,yao_prompt_2022,xu_universal_2023,bragg_flex_2021,ma_issues_2021,sun_nsp-bert_2022}, this paper focuses on guidance for one approach: Natural Language Inference." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="1909.00161" id="1302">
        <attvalues>
          <attvalue for="5" value=" Several papers have used the universal NLI task for zero- and fewshot classification, but stopped short of mixing NLI data with multiple other non-NLI datasets to build more universal classifiers \cite{yin_benchmarking_2019,yin_universal_2020,wang_entailment_2021,laurer_less_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2010.02584" id="1303">
        <attvalues>
          <attvalue for="5" value=" Several papers have used the universal NLI task for zero- and fewshot classification, but stopped short of mixing NLI data with multiple other non-NLI datasets to build more universal classifiers \cite{yin_benchmarking_2019,yin_universal_2020,wang_entailment_2021,laurer_less_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.17543" target="2104.14690" id="1304">
        <attvalues>
          <attvalue for="5" value=" Several papers have used the universal NLI task for zero- and fewshot classification, but stopped short of mixing NLI data with multiple other non-NLI datasets to build more universal classifiers \cite{yin_benchmarking_2019,yin_universal_2020,wang_entailment_2021,laurer_less_2023}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2003.10555" id="1307">
        <attvalues>
          <attvalue for="5" value=" Since discriminative PLMs typically enjoy competitive performance and superior computational efficiency compared with their generative counterparts~\cite{DBLP:conf/iclr/ClarkLLM20}, it can be especially appealing to prompt-tuning discriminative PLMs." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="1909.11942" id="1308">
        <attvalues>
          <attvalue for="5" value=" Recent years have witnessed the great success of the pre-training-then-fine-tuning paradigm in natural language processing (NLP)~\cite{DBLP:conf/naacl/DevlinCLT19,DBLP:conf/nips/YangDYCSL19,DBLP:conf/iclr/ClarkLLM20,DBLP:conf/iclr/LanCGGSS20,DBLP:journals/jmlr/RaffelSRLNMZLL20}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2003.08271" id="1310">
        <attvalues>
          <attvalue for="5" value=" Typically, language models are first pre-trained on large-scale corpora via self-supervised generative or discriminative tasks to learn universal text representations, and then fine-tuned to adapt to downstream tasks~\cite{qiu2020pre,xu2021pre}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2106.07139" id="1311">
        <attvalues>
          <attvalue for="5" value=" Typically, language models are first pre-trained on large-scale corpora via self-supervised generative or discriminative tasks to learn universal text representations, and then fine-tuned to adapt to downstream tasks~\cite{qiu2020pre,xu2021pre}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2107.13586" id="1312">
        <attvalues>
          <attvalue for="5" value=" However, the significant gap between the objective forms of model pre-training and fine-tuning hinders taking full advantage of PLMs in downstream tasks~\cite{DBLP:journals/corr/abs-2107-13586}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="1909.01066" id="1313">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning has recently shown its effectiveness in stimulating the capability of PLMs by transforming downstream tasks into the same form as pre-training~\cite{DBLP:conf/emnlp/PetroniRRLBWM19,brown2020language,DBLP:conf/naacl/SchickS21,DBLP:conf/acl/GaoFC20,yao2021cpt}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2005.14165" id="1314">
        <attvalues>
          <attvalue for="5" value=" However, to the best of our knowledge, existing works focus on prompt-tuning generative PLMs (eg, PLMs pre-trained by generating target textual tokens from the context, such as BERT~\cite{DBLP:conf/naacl/DevlinCLT19} and GPT~\cite{brown2020language})." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2009.07118" id="1315">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning has recently shown its effectiveness in stimulating the capability of PLMs by transforming downstream tasks into the same form as pre-training~\cite{DBLP:conf/emnlp/PetroniRRLBWM19,brown2020language,DBLP:conf/naacl/SchickS21,DBLP:conf/acl/GaoFC20,yao2021cpt}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2012.15723" id="1316">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning has recently shown its effectiveness in stimulating the capability of PLMs by transforming downstream tasks into the same form as pre-training~\cite{DBLP:conf/emnlp/PetroniRRLBWM19,brown2020language,DBLP:conf/naacl/SchickS21,DBLP:conf/acl/GaoFC20,yao2021cpt}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2109.11797" id="1317">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning has recently shown its effectiveness in stimulating the capability of PLMs by transforming downstream tasks into the same form as pre-training~\cite{DBLP:conf/emnlp/PetroniRRLBWM19,brown2020language,DBLP:conf/naacl/SchickS21,DBLP:conf/acl/GaoFC20,yao2021cpt}." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="1912.09637" id="1318">
        <attvalues>
          <attvalue for="5" value=" It is still unknown whether and how discriminative PLMs can be effectively prompt-tuned (eg, PLMs pre-trained by discriminating replaced tokens, such as ELECTRA~\cite{DBLP:conf/iclr/ClarkLLM20} and WKLM~\cite{DBLP:conf/iclr/XiongDWS20})." />
        </attvalues>
      </edge>
      <edge source="2205.11166" target="2002.06305" id="1319">
        <attvalues>
          <attvalue for="5" value=" Moreover, previous works have shown that fine-tuning large PLMs can be highly unstable and even produce divergent results~\cite{DBLP:conf/naacl/DevlinCLT19,dodge2020fine}, which undermines the practicality of large PLMs." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="1909.08593" id="1321">
        <attvalues>
          <attvalue for="5" value=" Preference learning methods — such as Reinforcement Learning from Human Feedback (RLHF) \cite{rm-direction,ft-preference,InstructGPT,cai,llama2} and its successor, Direct Preference Optimization (DPO) \cite{DPO} — leverage preference datasets to achieve alignment that goes beyond what instruction tuning can offer." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2212.08073" id="1323">
        <attvalues>
          <attvalue for="5" value=" Preference learning methods — such as Reinforcement Learning from Human Feedback (RLHF) \cite{rm-direction,ft-preference,InstructGPT,cai,llama2} and its successor, Direct Preference Optimization (DPO) \cite{DPO} — leverage preference datasets to achieve alignment that goes beyond what instruction tuning can offer." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2307.09288" id="1324">
        <attvalues>
          <attvalue for="5" value=" To mitigate these challenges arising from noisy data in the training dataset, manual data cleansing or adding additional annotations, such as the significance level of each preference data point, can be used~\cite{llama2}." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2305.18290" id="1325">
        <attvalues>
          <attvalue for="5" value=" Preference learning methods — such as Reinforcement Learning from Human Feedback (RLHF) \cite{rm-direction,ft-preference,InstructGPT,cai,llama2} and its successor, Direct Preference Optimization (DPO) \cite{DPO} — leverage preference datasets to achieve alignment that goes beyond what instruction tuning can offer." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2204.05862" id="1326">
        <attvalues>
          <attvalue for="5" value=" During the collection of paired corpora for prompts, decisions can vary from unanimous to narrowly divided even within the same crowd \cite{HH-RLHF}." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2303.12712" id="1327">
        <attvalues>
          <attvalue for="5" value=" Similar issues may arise when using large language models (LLMs) for annotation as they exhibit human-like patterns \cite{AGI-sparks}." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2305.14279" id="1328">
        <attvalues>
          <attvalue for="5" value=" \cite{consistency} observed that GPT-4 \cite{GPT4} can display inconsistent behavior during multi-step reasoning despite its impressive performance on various tasks." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2303.08774" id="1329">
        <attvalues>
          <attvalue for="5" value=" \cite{consistency} observed that GPT-4 \cite{GPT4} can display inconsistent behavior during multi-step reasoning despite its impressive performance on various tasks." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2009.10795" id="1330">
        <attvalues>
          <attvalue for="5" value=" Whereas \cite{activeIT} suggest that ambiguous tasks can enhance generalization in instruction tuning using a categorization similar to \cite{dataset_cartography}, \cite{quantity2quality} demonstrate that selecting easy-to-learn data over hard-to-learn segments is more beneficial for improving performance." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2311.00288" id="1331">
        <attvalues>
          <attvalue for="5" value=" Whereas \cite{activeIT} suggest that ambiguous tasks can enhance generalization in instruction tuning using a categorization similar to \cite{dataset_cartography}, \cite{quantity2quality} demonstrate that selecting easy-to-learn data over hard-to-learn segments is more beneficial for improving performance." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2308.12032" id="1332">
        <attvalues>
          <attvalue for="5" value=" Whereas \cite{activeIT} suggest that ambiguous tasks can enhance generalization in instruction tuning using a categorization similar to \cite{dataset_cartography}, \cite{quantity2quality} demonstrate that selecting easy-to-learn data over hard-to-learn segments is more beneficial for improving performance." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2306.04488" id="1333">
        <attvalues>
          <attvalue for="5" value=" Some previous approaches to preference learning aim to build better reward functions by merging or weighting multiple reward models through a multi-objective learning framework \cite{rewarded_soups, personalizedsoups, arithmetic_control}." />
        </attvalues>
      </edge>
      <edge source="2408.12799" target="2403.00409" id="1334">
        <attvalues>
          <attvalue for="5" value=" Additionally, conservative DPO (cDPO) \cite{cDPO} and robust DPO (rDPO) \cite{rDPO} consider mislabeled or ambiguous data contained in the dataset unlike DPO." />
        </attvalues>
      </edge>
      <edge source="2305.13026" target="1905.00537" id="1358">
        <attvalues>
          <attvalue for="5" value=" For this reason, several benchmark suites have been proposed, such as English GLUE \cite{wang-etal-2018-glue} and SuperGLUE \cite{wang2019superglue}." />
        </attvalues>
      </edge>
      <edge source="2305.13026" target="2001.06286" id="1361">
        <attvalues>
          <attvalue for="5" value=" The currently available Dutch language models are BERTje \cite{devries2019bertje}, which is a Dutch version of BERT\textsubscript{base} \cite{devlin-etal-2019-bert}, and three versions of RobBERT \cite{delobelle2020robbert, delobelle2022robbert} which are Dutch versions of RoBERTa\textsubscript{base} \cite{liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2305.13026" target="1907.11692" id="1362">
        <attvalues>
          <attvalue for="5" value=" The currently available Dutch language models are BERTje \cite{devries2019bertje}, which is a Dutch version of BERT\textsubscript{base} \cite{devlin-etal-2019-bert}, and three versions of RobBERT \cite{delobelle2020robbert, delobelle2022robbert} which are Dutch versions of RoBERTa\textsubscript{base} \cite{liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2305.13026" target="2211.08192" id="1363">
        <attvalues>
          <attvalue for="5" value=" Direct comparisons between these models have focused on several tasks: sentiment analysis, natural language inference, coarse-grained part-of-speech tagging and three-class named entity recognition, where RobBERT often outperforms BERTje \cite{delobelle2022robbert}." />
        </attvalues>
      </edge>
      <edge source="2104.10809" target="2002.12327" id="1366">
        <attvalues>
          <attvalue for="5" value=" Rather, analysis of massive language models has revealed that, to some degree, knowledge of syntactic and semantic dependencies can emerge without explicit supervision \cite{rogers2020primer, tenney-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2104.10809" target="1901.11373" id="1367">
        <attvalues>
          <attvalue for="5" value=" Yet, today's NLP systems built on large language models still fall short of human-level general understanding \cite{yogatama2019learning, zhang-etal-2020-winowhy}." />
        </attvalues>
      </edge>
      <edge source="2104.10809" target="2005.14165" id="1368">
        <attvalues>
          <attvalue for="5" value=" \cite{brown2020language} discuss the limitations of their GPT-3 language model compared to humans, suggesting that: \begin{displayquote} Scaling up any LM-like model \ldots\ may eventually run into (or could already be running into) the limits of the pretraining objective." />
        </attvalues>
      </edge>
      <edge source="2403.00827" target="2206.05802" id="1370">
        <attvalues>
          <attvalue for="5" value=" Various approaches have been proposed to show the ability of models to evaluate and critique responses \cite{saunders2022selfcritiquing, scheurer2023training, shinn2023reflexion, selfee2023}, as well as their potential to refine: given feedback, correct their outputs \cite{welleck2022generating, peng2023check, madaan2023selfrefine, huang2023large, wang2023enable}." />
        </attvalues>
      </edge>
      <edge source="2403.00827" target="2211.00053" id="1371">
        <attvalues>
          <attvalue for="5" value=" Various approaches have been proposed to show the ability of models to evaluate and critique responses \cite{saunders2022selfcritiquing, scheurer2023training, shinn2023reflexion, selfee2023}, as well as their potential to refine: given feedback, correct their outputs \cite{welleck2022generating, peng2023check, madaan2023selfrefine, huang2023large, wang2023enable}." />
        </attvalues>
      </edge>
      <edge source="2403.00827" target="2303.17651" id="1372">
        <attvalues>
          <attvalue for="5" value="} Of particular note are recent works exploring the self-refinement phenomenon \cite{madaan2023selfrefine, wang2023enable, shinn2023reflexion}, leveraging the same LLM to perform critique and/or refinement on top of generating responses." />
        </attvalues>
      </edge>
      <edge source="2403.00827" target="2310.01798" id="1373">
        <attvalues>
          <attvalue for="5" value="5 and GPT-4 in refinement, and in the absence of well-designed stopping mechanisms, self-refinement applied to high-quality responses can make the results worse \cite{huang2023large}." />
        </attvalues>
      </edge>
      <edge source="2210.15424" target="2005.14165" id="1374">
        <attvalues>
          <attvalue for="5" value=" Recent years have seen the advent of large language models characterized by emergent capabilities (eg, zero-shot generalization) arising from sheer scale alone~\cite{radford2019language,brown2020gpt3}." />
        </attvalues>
      </edge>
      <edge source="2210.15424" target="2001.08361" id="1375">
        <attvalues>
          <attvalue for="5" value=" Scaling LLMs results in a predictable increase in performance: simple scaling laws connect the number of parameters, pretraining dataset size, and compute budget~\cite{kaplan2020scaling,ganguli2022predictability,hoffmann2022training}, providing a clear path towards more capable models." />
        </attvalues>
      </edge>
      <edge source="2210.15424" target="2202.07785" id="1376">
        <attvalues>
          <attvalue for="5" value=" Scaling LLMs results in a predictable increase in performance: simple scaling laws connect the number of parameters, pretraining dataset size, and compute budget~\cite{kaplan2020scaling,ganguli2022predictability,hoffmann2022training}, providing a clear path towards more capable models." />
        </attvalues>
      </edge>
      <edge source="2210.15424" target="2203.15556" id="1377">
        <attvalues>
          <attvalue for="5" value=" Scaling LLMs results in a predictable increase in performance: simple scaling laws connect the number of parameters, pretraining dataset size, and compute budget~\cite{kaplan2020scaling,ganguli2022predictability,hoffmann2022training}, providing a clear path towards more capable models." />
        </attvalues>
      </edge>
      <edge source="2210.15424" target="1706.03762" id="1378">
        <attvalues>
          <attvalue for="5" value=" This paradigm shift has been fueled by the wide~adoption of the Transformer~\cite{vaswani2017attention}, providing a scalable basis for practitioners to build upon." />
        </attvalues>
      </edge>
      <edge source="2406.11890" target="2005.14165" id="1379">
        <attvalues>
          <attvalue for="5" value=" In-context learning (ICL) has emerged as a promising paradigm that employs a sequence of demonstration exemplars as prompts to assist large language models (LLMs) in effectively performing unseen tasks~\cite{nips20llmfewshotlearner, votek}." />
        </attvalues>
      </edge>
      <edge source="2406.11890" target="2102.09690" id="1380">
        <attvalues>
          <attvalue for="5" value=" However, the performance of ICL can be sensitive to the choice, format, and order of the in-context exemplar~\cite{ICLfactor1, ICLfactor2, ICLformat1, ACL23order2}." />
        </attvalues>
      </edge>
      <edge source="2406.11890" target="2112.08633" id="1381">
        <attvalues>
          <attvalue for="5" value=" Recent learning-based studies~\cite{epr, CEIL, li-etal-2023-unified}, however, separately train a retriever to learn implicit similarity measurements using a contrastive leaning-based proxy task where positive exemplars $x^+$ and negative exemplars $x^-$ are labeled by interacting with LLMs." />
        </attvalues>
      </edge>
      <edge source="2406.11890" target="1905.07830" id="1383">
        <attvalues>
          <attvalue for="5" value=" Through a detailed examination of previous works, we observe 1) While the low-level similarity like BM25 and semantic similarity excel in different tasks (eg, Top-K BM25 outperforms Top-K BERT on Nl2Bash~\cite{datasetnl2bash} and SWAG~\cite{dataset:swag} in Table~\ref{man:cls} and Table~\ref{man:gen}), learning-based similarity generally performs well across all tasks." />
        </attvalues>
      </edge>
      <edge source="2406.11890" target="2302.05698" id="1384">
        <attvalues>
          <attvalue for="5" value=" Moreover, learning-based methods often suffer from poor generalization across different tasks, as corroborated by findings in~\cite{CEIL}." />
        </attvalues>
      </edge>
      <edge source="2406.11890" target="2209.11895" id="1385">
        <attvalues>
          <attvalue for="5" value=" Additionally, by connecting our findings with existing interpretative theories of ICL~\cite{iclinductionhead, randomlabeliswrong1, Repetitionsiclr24, halawi2023overthinking, anchors}, we further qualitatively validate our conclusions." />
        </attvalues>
      </edge>
      <edge source="2304.12272" target="2210.11416" id="1386">
        <attvalues>
          <attvalue for="5" value=" In this paper, we fine-tune FLAN-T5 models of \cite{flan-t5-2022} (FLAN\-T5\-Large and FLAN\-T5\-XL) on a wide range of AMR parsing tasks including AMR2." />
        </attvalues>
      </edge>
      <edge source="2304.12272" target="2301.13688" id="1387">
        <attvalues>
          <attvalue for="5" value=" Instruction fine-tuning language models on a collection of annotated datasets has proven highly effective to improve model performance and generalization to unseen tasks both in general purpose open domain setup, as in \cite{flan-t5-2022, flan2021,flan2023,instructgpt, naturalinstructions, supernaturalinstructions, unnatural2022} and specialized tasks such as conversational dialogs in \cite{instructdial2022}." />
        </attvalues>
      </edge>
      <edge source="2304.12272" target="2205.12673" id="1389">
        <attvalues>
          <attvalue for="5" value=" Instruction fine-tuning language models on a collection of annotated datasets has proven highly effective to improve model performance and generalization to unseen tasks both in general purpose open domain setup, as in \cite{flan-t5-2022, flan2021,flan2023,instructgpt, naturalinstructions, supernaturalinstructions, unnatural2022} and specialized tasks such as conversational dialogs in \cite{instructdial2022}." />
        </attvalues>
      </edge>
      <edge source="2304.12272" target="2110.15534" id="1390">
        <attvalues>
          <attvalue for="5" value=" We show that fine-tuning FLAN-T5 models on AMR parsing leads to a significant improvement over the previous BART fine-tuned SoTA models by \cite{zhou2021emnlp, bai-etal-2022-graph}." />
        </attvalues>
      </edge>
      <edge source="2304.12272" target="2106.09685" id="1391">
        <attvalues>
          <attvalue for="5" value=" We further explore a parameter efficient fine-tuning technique, LoRA (Low Rank Adaptation), \cite{lora2021}." />
        </attvalues>
      </edge>
      <edge source="2206.11569" target="1606.00189" id="1392">
        <attvalues>
          <attvalue for="5" value=" Early GEC systems are basically based on error-specific classifiers \cite{rozovskaya2011algorithm,dahlmeier2012beam} or statistic machine translation models \cite{felice2014grammatical,chollampatt2016neural}." />
        </attvalues>
      </edge>
      <edge source="2206.11569" target="1706.03762" id="1393">
        <attvalues>
          <attvalue for="5" value=" Since the beginning of the deep learning era, neural encoder-decoder models, eg, Transformer \cite{vaswani2017attention}, have emerged as a dominant GEC paradigm \cite{yuan2016grammatical,junczys2018approaching}." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="1409.0473" id="1395">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) are one of the most fundamental technologies in NLP, with applications spanning text generation~\cite{bahdanau2014neural,rush2015neural}, representation learning~\cite{peters2018deep,devlin2019bert,yang2019xlnet}, and few-shot learning~\cite{radford2019language,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="1802.05365" id="1397">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) are one of the most fundamental technologies in NLP, with applications spanning text generation~\cite{bahdanau2014neural,rush2015neural}, representation learning~\cite{peters2018deep,devlin2019bert,yang2019xlnet}, and few-shot learning~\cite{radford2019language,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="2005.14165" id="1400">
        <attvalues>
          <attvalue for="5" value=" Language models (LMs) are one of the most fundamental technologies in NLP, with applications spanning text generation~\cite{bahdanau2014neural,rush2015neural}, representation learning~\cite{peters2018deep,devlin2019bert,yang2019xlnet}, and few-shot learning~\cite{radford2019language,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="1706.03762" id="1401">
        <attvalues>
          <attvalue for="5" value=" Modern neural language models (NLMs) based on recurrent~\cite{mikolov2010recurrent,sundermeyer2012lstm} or self-attentional~\cite{vaswani2017attention,al2019character} neural networks are mostly parametric, where the predictions are solely dependent on the model parameters given the input data." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="1709.08878" id="1403">
        <attvalues>
          <attvalue for="5" value=" \begin{SCfigure*} \end{SCfigure*} In contrast, recent non-parametric LMs~\cite{guu2018generating,khandelwal2019generalization,he2020learning} model text distributions by referencing both the parameters of the underlying model and examples from an external datastore." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="1911.00172" id="1404">
        <attvalues>
          <attvalue for="5" value=" One effective and representative example is the $k$-nearest neighbors LM ($k$NN-LM,~\cite{khandelwal2019generalization})." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="2006.16336" id="1405">
        <attvalues>
          <attvalue for="5" value=" \begin{SCfigure*} \end{SCfigure*} In contrast, recent non-parametric LMs~\cite{guu2018generating,khandelwal2019generalization,he2020learning} model text distributions by referencing both the parameters of the underlying model and examples from an external datastore." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="2010.00710" id="1406">
        <attvalues>
          <attvalue for="5" value=" This model is particularly notable for its large improvements in performance -- it outperforms the previous best parametric LMs by a large margin in standard language modeling benchmarks, in domain adaptation settings, and on other conditional generation tasks such as machine translation \cite{khandelwal2020nearest}." />
        </attvalues>
      </edge>
      <edge source="2109.04212" target="1609.07843" id="1407">
        <attvalues>
          <attvalue for="5" value=" Our experiments on the WikiText-103 language modeling benchmark~\cite{merity2016pointer} and a training-free domain-adaptation setting demonstrate speed improvements of up to 6x with comparable perplexity to the $k$NN-LM." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2111.01690" id="1408">
        <attvalues>
          <attvalue for="5" value=" Although automatic speech recognition (ASR)~\cite{li2022recent,GulatiQCPZYHWZW20,yao2023zipformer} has achieved excellent performance in quiet, single-speaker scenarios, it still faces significant challenges in multi-talker conversational scenarios, especially in the case of overlapping speech." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2005.08100" id="1409">
        <attvalues>
          <attvalue for="5" value=" Although automatic speech recognition (ASR)~\cite{li2022recent,GulatiQCPZYHWZW20,yao2023zipformer} has achieved excellent performance in quiet, single-speaker scenarios, it still faces significant challenges in multi-talker conversational scenarios, especially in the case of overlapping speech." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2310.11230" id="1410">
        <attvalues>
          <attvalue for="5" value=" Although automatic speech recognition (ASR)~\cite{li2022recent,GulatiQCPZYHWZW20,yao2023zipformer} has achieved excellent performance in quiet, single-speaker scenarios, it still faces significant challenges in multi-talker conversational scenarios, especially in the case of overlapping speech." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="1707.07048" id="1411">
        <attvalues>
          <attvalue for="5" value=" To overcome this challenge, a series of multi-talker ASR approaches have been proposed~\cite{chen2017progressive,yu2017recognizing,chang2019mimo,ZhangCQW20,kanda2020serialized}." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="1704.01985" id="1412">
        <attvalues>
          <attvalue for="5" value=" Compared to permutation invariant training (PIT)~\cite{yu2017recognizing,chang2019mimo,ZhangCQW20}, SOT avoids the limitation on the maximum number of speakers, models the dependencies in multi-talker content, and reduces computational complexity, resulting in better performance on multi-talker ASR task." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="1910.06522" id="1413">
        <attvalues>
          <attvalue for="5" value=" Compared to permutation invariant training (PIT)~\cite{yu2017recognizing,chang2019mimo,ZhangCQW20}, SOT avoids the limitation on the maximum number of speakers, models the dependencies in multi-talker content, and reduces computational complexity, resulting in better performance on multi-talker ASR task." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2003.12687" id="1414">
        <attvalues>
          <attvalue for="5" value=" This is precisely what previous SOT methods based on attention-based encoder-decoder (AED)~\cite{kanda2020serialized}, which relied more on encoder performance, lacked, leading to performance bottlenecks." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2104.02128" id="1415">
        <attvalues>
          <attvalue for="5" value=" One of the most representative methods is serialized output training (SOT)~\cite{kanda2020serialized,KandaYGWMCY21,ShiD0YLZ0023}." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2305.12459" id="1416">
        <attvalues>
          <attvalue for="5" value=" One of the most representative methods is serialized output training (SOT)~\cite{kanda2020serialized,KandaYGWMCY21,ShiD0YLZ0023}." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2103.16776" id="1417">
        <attvalues>
          <attvalue for="5" value="} For instance, in~\cite{KandaYWGWMCY21}, despite using 900K hours of large-scale simulated data for pre-training, the word error rate on the AMI~\cite{CarlettaABFGHKKKKLLLMPRW05} meeting corpus still reached 21." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2211.05100" id="1418">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs)~\cite{abs-2211-05100,abs-2302-13971,abs-2307-09288,chiang2023vicuna}, trained on vast amounts of text data, possess unparalleled capabilities in understanding and generating natural language." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2307.09288" id="1420">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs)~\cite{abs-2211-05100,abs-2302-13971,abs-2307-09288,chiang2023vicuna}, trained on vast amounts of text data, possess unparalleled capabilities in understanding and generating natural language." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2310.00230" id="1421">
        <attvalues>
          <attvalue for="5" value=" {A series of LLM-based ASR works~\cite{WangHSWCCCZSRZYPSSW23,abs-2307-11795,abs-2310-13289,abs-2311-07919,abs-2402-08846,abs-2405-02132} have been conducted, which, in contrast to traditional AED methods that focus on encoder performance, tend to treat the speech foundation encoder~\cite{BaevskiZMA20,HsuBTLSM21,ChenWCWLCLKYXWZ22,RadfordKXBMS23} in LLM-based models as a tool for extracting embedding." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2310.13289" id="1423">
        <attvalues>
          <attvalue for="5" value=" {A series of LLM-based ASR works~\cite{WangHSWCCCZSRZYPSSW23,abs-2307-11795,abs-2310-13289,abs-2311-07919,abs-2402-08846,abs-2405-02132} have been conducted, which, in contrast to traditional AED methods that focus on encoder performance, tend to treat the speech foundation encoder~\cite{BaevskiZMA20,HsuBTLSM21,ChenWCWLCLKYXWZ22,RadfordKXBMS23} in LLM-based models as a tool for extracting embedding." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2311.07919" id="1424">
        <attvalues>
          <attvalue for="5" value=" {A series of LLM-based ASR works~\cite{WangHSWCCCZSRZYPSSW23,abs-2307-11795,abs-2310-13289,abs-2311-07919,abs-2402-08846,abs-2405-02132} have been conducted, which, in contrast to traditional AED methods that focus on encoder performance, tend to treat the speech foundation encoder~\cite{BaevskiZMA20,HsuBTLSM21,ChenWCWLCLKYXWZ22,RadfordKXBMS23} in LLM-based models as a tool for extracting embedding." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2402.08846" id="1425">
        <attvalues>
          <attvalue for="5" value=" In~\cite{abs-2402-08846}, LoRA was not introduced, and the encoder was frozen while training only the projector, which also yielded satisfactory results." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2405.02132" id="1426">
        <attvalues>
          <attvalue for="5" value=" In~\cite{abs-2405-02132}, a multi-stage fine-tuning approach was used to better align the modalities of speech and text." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2006.11477" id="1427">
        <attvalues>
          <attvalue for="5" value=" {A series of LLM-based ASR works~\cite{WangHSWCCCZSRZYPSSW23,abs-2307-11795,abs-2310-13289,abs-2311-07919,abs-2402-08846,abs-2405-02132} have been conducted, which, in contrast to traditional AED methods that focus on encoder performance, tend to treat the speech foundation encoder~\cite{BaevskiZMA20,HsuBTLSM21,ChenWCWLCLKYXWZ22,RadfordKXBMS23} in LLM-based models as a tool for extracting embedding." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2106.07447" id="1428">
        <attvalues>
          <attvalue for="5" value=" {A series of LLM-based ASR works~\cite{WangHSWCCCZSRZYPSSW23,abs-2307-11795,abs-2310-13289,abs-2311-07919,abs-2402-08846,abs-2405-02132} have been conducted, which, in contrast to traditional AED methods that focus on encoder performance, tend to treat the speech foundation encoder~\cite{BaevskiZMA20,HsuBTLSM21,ChenWCWLCLKYXWZ22,RadfordKXBMS23} in LLM-based models as a tool for extracting embedding." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2110.13900" id="1429">
        <attvalues>
          <attvalue for="5" value=" {A series of LLM-based ASR works~\cite{WangHSWCCCZSRZYPSSW23,abs-2307-11795,abs-2310-13289,abs-2311-07919,abs-2402-08846,abs-2405-02132} have been conducted, which, in contrast to traditional AED methods that focus on encoder performance, tend to treat the speech foundation encoder~\cite{BaevskiZMA20,HsuBTLSM21,ChenWCWLCLKYXWZ22,RadfordKXBMS23} in LLM-based models as a tool for extracting embedding." />
        </attvalues>
      </edge>
      <edge source="2408.17431" target="2106.09685" id="1431">
        <attvalues>
          <attvalue for="5" value=" For example, in~\cite{abs-2307-11795}, low-rank adaptation (LoRA)~\cite{HuSWALWWC22} was introduced into the LLM to facilitate efficient fine-tuning, and all three components were fine-tuned together in a single stage." />
        </attvalues>
      </edge>
      <edge source="2402.08846" target="2111.01690" id="1441">
        <attvalues>
          <attvalue for="5" value=" The evolution of ASR technology has been marked by the adoption of various paradigms, each representing a leap forward in terms of accuracy, efficiency, and applicability~\cite{e2ereview}." />
        </attvalues>
      </edge>
      <edge source="2402.08846" target="1303.5778" id="1442">
        <attvalues>
          <attvalue for="5" value=" Among these, supervised methods including connectionist temporal classification (CTC)~\cite{CTC}, attention-based encoder-decoder (AED)~\cite{AED}, recurrent neural network transducer (RNN-T)~\cite{RNN-T} and their variants have been pivotal." />
        </attvalues>
      </edge>
      <edge source="2402.08846" target="2006.11477" id="1443">
        <attvalues>
          <attvalue for="5" value=" In addition, employing self-supervised methods for pre-training followed by supervised methods for fine-tuning has also proven to be effective~\cite{baevski2020wav2vec, hsu2021hubert, chen2022wavlm, ma2022mt4ssl, yang2023fast}." />
        </attvalues>
      </edge>
      <edge source="2402.08846" target="2307.03917" id="1445">
        <attvalues>
          <attvalue for="5" value=" Recent works in LLM-based ASR often venture into complex designs, such as compressing the output temporally from the speech encoder~\cite{wu2023decoder, fathullah2023prompting}, tackling modal alignment with the projector~\cite{tang2023salmonn, yu2023connecting}, and fine-tuning the LLM partly or fully~\cite{wu2023decoder, li2023prompting, tang2023salmonn, wang2023lauragpt}." />
        </attvalues>
      </edge>
      <edge source="2402.08846" target="2310.13289" id="1446">
        <attvalues>
          <attvalue for="5" value=" Recent works in LLM-based ASR often venture into complex designs, such as compressing the output temporally from the speech encoder~\cite{wu2023decoder, fathullah2023prompting}, tackling modal alignment with the projector~\cite{tang2023salmonn, yu2023connecting}, and fine-tuning the LLM partly or fully~\cite{wu2023decoder, li2023prompting, tang2023salmonn, wang2023lauragpt}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2303.08774" id="1449">
        <attvalues>
          <attvalue for="5" value=" Drawing from this strength, researchers have begun to merge the prowess of LLMs with various fields, including automatic speech recognition (ASR), where their integration has led to notable performance improvement~\cite{achiam2023gpt,team2023gemini,speechgpt}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2312.11805" id="1450">
        <attvalues>
          <attvalue for="5" value=" Drawing from this strength, researchers have begun to merge the prowess of LLMs with various fields, including automatic speech recognition (ASR), where their integration has led to notable performance improvement~\cite{achiam2023gpt,team2023gemini,speechgpt}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2305.11000" id="1451">
        <attvalues>
          <attvalue for="5" value=" In contrast, the second approach adopts audio-text cross-modal LLMs, which embrace the auditory modality by employing an encoder network to process the speech and generate embeddings that are subsequently provided to a decoder-only LLM~\cite{speechgpt,decoder-only-asr,listen_think_anderstand,salmonn,qwen-audio, simple-asr-llm}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2211.00968" id="1452">
        <attvalues>
          <attvalue for="5" value=" Specifically, ASR, a task that intricately intertwines acoustic modeling with language modeling, has conventionally employed language models like n-grams~\cite{ngram-1,ngram-2,shallow-fusion} or neural network language models (NNLMs)~\cite{nnlm-1,nnlm-2,deep-fusion,component-fusion}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2210.17017" id="1453">
        <attvalues>
          <attvalue for="5" value=" Specifically, ASR, a task that intricately intertwines acoustic modeling with language modeling, has conventionally employed language models like n-grams~\cite{ngram-1,ngram-2,shallow-fusion} or neural network language models (NNLMs)~\cite{nnlm-1,nnlm-2,deep-fusion,component-fusion}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2110.10026" id="1454">
        <attvalues>
          <attvalue for="5" value=" Specifically, ASR, a task that intricately intertwines acoustic modeling with language modeling, has conventionally employed language models like n-grams~\cite{ngram-1,ngram-2,shallow-fusion} or neural network language models (NNLMs)~\cite{nnlm-1,nnlm-2,deep-fusion,component-fusion}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="1905.04226" id="1455">
        <attvalues>
          <attvalue for="5" value=" Specifically, ASR, a task that intricately intertwines acoustic modeling with language modeling, has conventionally employed language models like n-grams~\cite{ngram-1,ngram-2,shallow-fusion} or neural network language models (NNLMs)~\cite{nnlm-1,nnlm-2,deep-fusion,component-fusion}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2304.12995" id="1456">
        <attvalues>
          <attvalue for="5" value=" The first strategy involves connecting LLMs with pre-trained ASR models, wherein the ASR-generated text is directly fed to the LLM to serve as a prompt for downstream tasks~\cite{linkpara1_audio_gpt,linkpara-2,linkpara-4} or to facilitate error correction~\cite{linkpara-5}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2309.04842" id="1458">
        <attvalues>
          <attvalue for="5" value=" The first strategy involves connecting LLMs with pre-trained ASR models, wherein the ASR-generated text is directly fed to the LLM to serve as a prompt for downstream tasks~\cite{linkpara1_audio_gpt,linkpara-2,linkpara-4} or to facilitate error correction~\cite{linkpara-5}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2307.04172" id="1459">
        <attvalues>
          <attvalue for="5" value=" The first strategy involves connecting LLMs with pre-trained ASR models, wherein the ASR-generated text is directly fed to the LLM to serve as a prompt for downstream tasks~\cite{linkpara1_audio_gpt,linkpara-2,linkpara-4} or to facilitate error correction~\cite{linkpara-5}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2307.03917" id="1460">
        <attvalues>
          <attvalue for="5" value=" In contrast, the second approach adopts audio-text cross-modal LLMs, which embrace the auditory modality by employing an encoder network to process the speech and generate embeddings that are subsequently provided to a decoder-only LLM~\cite{speechgpt,decoder-only-asr,listen_think_anderstand,salmonn,qwen-audio, simple-asr-llm}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2305.10790" id="1461">
        <attvalues>
          <attvalue for="5" value=" In contrast, the second approach adopts audio-text cross-modal LLMs, which embrace the auditory modality by employing an encoder network to process the speech and generate embeddings that are subsequently provided to a decoder-only LLM~\cite{speechgpt,decoder-only-asr,listen_think_anderstand,salmonn,qwen-audio, simple-asr-llm}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2310.13289" id="1462">
        <attvalues>
          <attvalue for="5" value=" Specifically, SALMONN~\cite{salmonn} applies Whisper~\cite{whisper} to extract semantic content and BEATs~\cite{beats} for audio event information, culminating in a robust perception of human speech, music, and audio events." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2311.07919" id="1463">
        <attvalues>
          <attvalue for="5" value=" Qwen-Audio~\cite{qwen-audio} implements Whisper as the exclusive encoder, utilizing structured task directives to enhance the model's performance across various audio tasks." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2212.09058" id="1465">
        <attvalues>
          <attvalue for="5" value=" Specifically, SALMONN~\cite{salmonn} applies Whisper~\cite{whisper} to extract semantic content and BEATs~\cite{beats} for audio event information, culminating in a robust perception of human speech, music, and audio events." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2402.08846" id="1466">
        <attvalues>
          <attvalue for="5" value=" SLAM-ASR~\cite{simple-asr-llm} leverages a linear layer as the projector module and achieves SOTA performance on the English 960-hour LibriSpeech~\cite{libispeech} task." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2106.07447" id="1467">
        <attvalues>
          <attvalue for="5" value=" From experiments, we draw the following major conclusions: (1) For the speech encoder, Whisper~\cite{whisper} is more robust but have lower plasticity compared to HuBERT~\cite{hubert}." />
        </attvalues>
      </edge>
      <edge source="2405.02132" target="2301.12597" id="1468">
        <attvalues>
          <attvalue for="5" value=" (2) For the projector, the Transformer’s learning ability is better than the Qformer~\cite{blip} in the speech recognition task." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2308.04477" id="1470">
        <attvalues>
          <attvalue for="5" value=" It significantly enhances development efficiency and quality by increasing productivity, reducing errors, standardizing code, accelerating prototyping, and supporting complex systems~\cite{li2024deveval, li2023large,buscemi2023comparative}." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2203.15556" id="1471">
        <attvalues>
          <attvalue for="5" value=" Training large language models requires extensive high-quality data~\cite{hoffmann2022training}." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2107.03374" id="1472">
        <attvalues>
          <attvalue for="5" value=" To comprehensively evaluate the capabilities of AutoCoder, we tested it on several datasets: HumanEval~\cite{chen2021evaluating}, HumanEval+~\cite{liu2024your}, MBPP~\cite{austin2021program}, MBPP+~\cite{liu2024your}, MultiPL-E~\cite{cassano2022multipl}, and DS-1000~\cite{lai2023ds}." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2212.10560" id="1475">
        <attvalues>
          <attvalue for="5" value=" To address this challenge, previous work has employed various automated code annotation methods, such as Self-Instruct~\cite{wang2022self}, Evol-Instruct~\cite{luo2023wizardcoder}, and OSS-Instruct~\cite{wei2023magicoder}." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2306.08568" id="1476">
        <attvalues>
          <attvalue for="5" value=" To address this challenge, previous work has employed various automated code annotation methods, such as Self-Instruct~\cite{wang2022self}, Evol-Instruct~\cite{luo2023wizardcoder}, and OSS-Instruct~\cite{wei2023magicoder}." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2305.01210" id="1477">
        <attvalues>
          <attvalue for="5" value=" To comprehensively evaluate the capabilities of AutoCoder, we tested it on several datasets: HumanEval~\cite{chen2021evaluating}, HumanEval+~\cite{liu2024your}, MBPP~\cite{austin2021program}, MBPP+~\cite{liu2024your}, MultiPL-E~\cite{cassano2022multipl}, and DS-1000~\cite{lai2023ds}." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2108.07732" id="1478">
        <attvalues>
          <attvalue for="5" value=" To comprehensively evaluate the capabilities of AutoCoder, we tested it on several datasets: HumanEval~\cite{chen2021evaluating}, HumanEval+~\cite{liu2024your}, MBPP~\cite{austin2021program}, MBPP+~\cite{liu2024your}, MultiPL-E~\cite{cassano2022multipl}, and DS-1000~\cite{lai2023ds}." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2208.08227" id="1479">
        <attvalues>
          <attvalue for="5" value=" To comprehensively evaluate the capabilities of AutoCoder, we tested it on several datasets: HumanEval~\cite{chen2021evaluating}, HumanEval+~\cite{liu2024your}, MBPP~\cite{austin2021program}, MBPP+~\cite{liu2024your}, MultiPL-E~\cite{cassano2022multipl}, and DS-1000~\cite{lai2023ds}." />
        </attvalues>
      </edge>
      <edge source="2405.14906" target="2211.11501" id="1480">
        <attvalues>
          <attvalue for="5" value=" To comprehensively evaluate the capabilities of AutoCoder, we tested it on several datasets: HumanEval~\cite{chen2021evaluating}, HumanEval+~\cite{liu2024your}, MBPP~\cite{austin2021program}, MBPP+~\cite{liu2024your}, MultiPL-E~\cite{cassano2022multipl}, and DS-1000~\cite{lai2023ds}." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="1706.03762" id="1481">
        <attvalues>
          <attvalue for="5" value=" Recent years have seen a surge of transformer~\cite{vaswanietal2017} based Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLM-RoBERTa (XLMR) \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5}, RemBERT \cite{chung2021rethinking}." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="1911.02116" id="1483">
        <attvalues>
          <attvalue for="5" value=" Recent years have seen a surge of transformer~\cite{vaswanietal2017} based Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLM-RoBERTa (XLMR) \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5}, RemBERT \cite{chung2021rethinking}." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2010.11934" id="1484">
        <attvalues>
          <attvalue for="5" value=" Recent years have seen a surge of transformer~\cite{vaswanietal2017} based Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLM-RoBERTa (XLMR) \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5}, RemBERT \cite{chung2021rethinking}." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2010.12821" id="1485">
        <attvalues>
          <attvalue for="5" value=" Recent years have seen a surge of transformer~\cite{vaswanietal2017} based Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLM-RoBERTa (XLMR) \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5}, RemBERT \cite{chung2021rethinking}." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2004.10643" id="1486">
        <attvalues>
          <attvalue for="5" value=" These MMLMs are primarily evaluated for their performance on Sequence Labelling \cite{nivre-etal-2020-universal, Pan2017}, Classification \cite{Conneau2018xnli, Yang2019paws-x, ponti-etal-2020-xcopa}, Question Answering \cite{artetxe2020cross, Lewis2020mlqa, Clark2020tydiqa} and Retrieval \cite{Artetxe2019massively, roy-etal-2020-lareqa, botha-etal-2020-entity} tasks." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="1812.10464" id="1489">
        <attvalues>
          <attvalue for="5" value=" These MMLMs are primarily evaluated for their performance on Sequence Labelling \cite{nivre-etal-2020-universal, Pan2017}, Classification \cite{Conneau2018xnli, Yang2019paws-x, ponti-etal-2020-xcopa}, Question Answering \cite{artetxe2020cross, Lewis2020mlqa, Clark2020tydiqa} and Retrieval \cite{Artetxe2019massively, roy-etal-2020-lareqa, botha-etal-2020-entity} tasks." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2005.09093" id="1490">
        <attvalues>
          <attvalue for="5" value=" Evaluating on such benchmarks henceforth fails to provide a comprehensive picture of the model's performance across the linguistic landscape, as the performance of MMLMs has been shown to vary significantly with the amount of pre-training data available for a language \cite{wu-dredze-2020-languages}, as well according to the typological relatedness between the pivot and target languages \cite{lauscher-etal-2020-zero}." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2003.11080" id="1491">
        <attvalues>
          <attvalue for="5" value=" \cite{hu2020xtreme} provides pseudo test sets for tasks like XQUAD and XNLI, obtained by translating English test data into different languages, and shows reasonable estimates of the actual performance by evaluating on translated data but cautions about their reliability when the model is trained on translated data." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2005.00870" id="1492">
        <attvalues>
          <attvalue for="5" value=" Not only this can help us give a better idea about the performance of a multilingual model on a task across a much larger set of languages and hence aiding in better model selection, but also enables applications in devising data collection strategies to maximize performance \cite{srinivasan2022litmus} as well as in selecting the representative set of languages for a benchmark \cite{xia-etal-2020-predicting}." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2110.08875" id="1493">
        <attvalues>
          <attvalue for="5" value=" \cite{srinivasan2021predicting} showed promising results specifically for MMLMs towards predicting their performance on downstream tasks for different languages in zero-shot and few-shot settings, and \cite{ye-etal-2021-towards} propose methods for more reliable performance prediction by estimating confidence intervals as well as predicting fine-grained performance measures." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2102.05486" id="1494">
        <attvalues>
          <attvalue for="5" value=" \cite{srinivasan2021predicting} showed promising results specifically for MMLMs towards predicting their performance on downstream tasks for different languages in zero-shot and few-shot settings, and \cite{ye-etal-2021-towards} propose methods for more reliable performance prediction by estimating confidence intervals as well as predicting fine-grained performance measures." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="1908.11828" id="1495">
        <attvalues>
          <attvalue for="5" value=" We present a case study demonstrating the effectiveness of performance prediction on four multilingual tasks, PAWS-X \cite{Yang2019paws-x} XNLI \cite{Conneau2018xnli}, XQUAD \cite{artetxe2020cross} and TyDiQA-GoldP \cite{Clark2020tydiqa} and show that it can often provide reliable estimates of the performance on different languages on par with evaluating them on translated test sets without any additional translation costs." />
        </attvalues>
      </edge>
      <edge source="2205.06356" target="2003.05002" id="1496">
        <attvalues>
          <attvalue for="5" value=" We present a case study demonstrating the effectiveness of performance prediction on four multilingual tasks, PAWS-X \cite{Yang2019paws-x} XNLI \cite{Conneau2018xnli}, XQUAD \cite{artetxe2020cross} and TyDiQA-GoldP \cite{Clark2020tydiqa} and show that it can often provide reliable estimates of the performance on different languages on par with evaluating them on translated test sets without any additional translation costs." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="1911.02116" id="1498">
        <attvalues>
          <attvalue for="5" value=" Multilingual pretrained encoders like multilingual BERT (mBERT; \cite{devlin-etal-2019-bert}) and XLM-R \cite{conneau-etal-2020-unsupervised} are the top performers in crosslingual tasks such as natural language inference \cite{conneau2018xnli}, document classification \cite{SCHWENK18." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="2010.06432" id="1500">
        <attvalues>
          <attvalue for="5" value="658,artetxe-schwenk-2019-massively}, and argument mining \cite{toledo-ronen-etal-2020-multilingual}." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="1901.07291" id="1501">
        <attvalues>
          <attvalue for="5" value=" A widely explored transfer scenario is zero-shot crosslingual transfer \cite{ pires-etal-2019-multilingual,conneau2019cross,artetxe-schwenk-2019-massively}, where a pretrained encoder is finetuned on abundant task data in the source language (eg, English) and then directly evaluated on target-language test data, achieving surprisingly good performance \cite{wu-dredze-2019-beto,hu2020xtreme}." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="1812.10464" id="1502">
        <attvalues>
          <attvalue for="5" value=" A widely explored transfer scenario is zero-shot crosslingual transfer \cite{ pires-etal-2019-multilingual,conneau2019cross,artetxe-schwenk-2019-massively}, where a pretrained encoder is finetuned on abundant task data in the source language (eg, English) and then directly evaluated on target-language test data, achieving surprisingly good performance \cite{wu-dredze-2019-beto,hu2020xtreme}." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="1904.09077" id="1503">
        <attvalues>
          <attvalue for="5" value=" A widely explored transfer scenario is zero-shot crosslingual transfer \cite{ pires-etal-2019-multilingual,conneau2019cross,artetxe-schwenk-2019-massively}, where a pretrained encoder is finetuned on abundant task data in the source language (eg, English) and then directly evaluated on target-language test data, achieving surprisingly good performance \cite{wu-dredze-2019-beto,hu2020xtreme}." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="2004.15001" id="1504">
        <attvalues>
          <attvalue for="5" value=" However, there is evidence that zero-shot performance reported in the literature has large variance and is often not reproducible \cite{keung2020evaluation,rios-mller-sennrich:2020:WMT}; the results in languages distant from English fall far short of those similar to English \cite{hu2020xtreme,liang2020xglue}." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="2011.01703" id="1505">
        <attvalues>
          <attvalue for="5" value=" However, there is evidence that zero-shot performance reported in the literature has large variance and is often not reproducible \cite{keung2020evaluation,rios-mller-sennrich:2020:WMT}; the results in languages distant from English fall far short of those similar to English \cite{hu2020xtreme,liang2020xglue}." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="2010.03179" id="1506">
        <attvalues>
          <attvalue for="5" value=" The few shots substantially improve model performance of the target language with negligible annotation costs \cite{garrette-baldridge-2013-learning,hedderich-etal-2020-transfer}." />
        </attvalues>
      </edge>
      <edge source="2012.15682" target="2004.07780" id="1507">
        <attvalues>
          <attvalue for="5" value=" To understand these phenomena, we conduct additional in-depth analyses, and find that the models tend to utilize shallow lexical hints \cite{shortcutlearning} in the target language, rather than leveraging abstract crosslingual semantic features learned from the source language." />
        </attvalues>
      </edge>
      <edge source="2004.15001" target="1710.04087" id="1508">
        <attvalues>
          <attvalue for="5" value=" The discovery of cross-lingual structure in word embedding spaces culminated in the work of \cite{ConneauLRDJ18-word}, which showed that unsupervised word translation via adversarial mappings is competitive with supervised techniques." />
        </attvalues>
      </edge>
      <edge source="2004.15001" target="1710.11041" id="1509">
        <attvalues>
          <attvalue for="5" value=" Concurrent work in machine translation also showed that it is possible to achieve non-trivial BLEU scores without any bitext \cite{ArtetxeLAC18-unsupervised,LampleCDR18-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2004.15001" target="1711.00043" id="1510">
        <attvalues>
          <attvalue for="5" value=" Concurrent work in machine translation also showed that it is possible to achieve non-trivial BLEU scores without any bitext \cite{ArtetxeLAC18-unsupervised,LampleCDR18-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2004.15001" target="1901.07291" id="1512">
        <attvalues>
          <attvalue for="5" value=" Self-supervised multilingual contextual embeddings like mBERT \cite{devlin-etal-2019-bert} and XLM \cite{LampleC19-cross} have shown remarkably strong performance on cross-lingual named entity recognition, text classification, dependency parsing, and other tasks (eg, \cite{pires-etal-2019-multilingual,keung-etal-2019-adversarial,wu-dredze-2019-beto})." />
        </attvalues>
      </edge>
      <edge source="2004.15001" target="1906.01502" id="1513">
        <attvalues>
          <attvalue for="5" value=" Self-supervised multilingual contextual embeddings like mBERT \cite{devlin-etal-2019-bert} and XLM \cite{LampleC19-cross} have shown remarkably strong performance on cross-lingual named entity recognition, text classification, dependency parsing, and other tasks (eg, \cite{pires-etal-2019-multilingual,keung-etal-2019-adversarial,wu-dredze-2019-beto})." />
        </attvalues>
      </edge>
      <edge source="2004.15001" target="1909.00153" id="1514">
        <attvalues>
          <attvalue for="5" value=" Self-supervised multilingual contextual embeddings like mBERT \cite{devlin-etal-2019-bert} and XLM \cite{LampleC19-cross} have shown remarkably strong performance on cross-lingual named entity recognition, text classification, dependency parsing, and other tasks (eg, \cite{pires-etal-2019-multilingual,keung-etal-2019-adversarial,wu-dredze-2019-beto})." />
        </attvalues>
      </edge>
      <edge source="2004.15001" target="1904.09077" id="1515">
        <attvalues>
          <attvalue for="5" value=" Self-supervised multilingual contextual embeddings like mBERT \cite{devlin-etal-2019-bert} and XLM \cite{LampleC19-cross} have shown remarkably strong performance on cross-lingual named entity recognition, text classification, dependency parsing, and other tasks (eg, \cite{pires-etal-2019-multilingual,keung-etal-2019-adversarial,wu-dredze-2019-beto})." />
        </attvalues>
      </edge>
      <edge source="2004.15001" target="1805.09821" id="1516">
        <attvalues>
          <attvalue for="5" value=" We present 4 published baselines for zero-shot cross-lingual document classification on MLDoc \cite{schwenk-li-2018-corpus} in Table \ref{table:published-mbert}: Even though the authors report English accuracies which are basically identical, their target language performances are very different." />
        </attvalues>
      </edge>
      <edge source="2010.03179" target="2003.11080" id="1518">
        <attvalues>
          <attvalue for="5" value=" For pretraining approaches where labeled data exists in a high-resource language, and the information is transferred to a low-resource language, \cite{data/Xtreme20} find a significant gap between performance on English and the cross-lingually transferred models." />
        </attvalues>
      </edge>
      <edge source="2010.03179" target="2005.00633" id="1519">
        <attvalues>
          <attvalue for="5" value=" In a recent study, \cite{lowresource/Lauscher2020FromZTH} find that the transfer for multilingual transformer models is less effective for resource-lean settings and distant languages." />
        </attvalues>
      </edge>
      <edge source="2010.03179" target="2004.13305" id="1520">
        <attvalues>
          <attvalue for="5" value=" \cite{lowresource/kann20weakly} recently inspected POS classifiers trained on weak supervision." />
        </attvalues>
      </edge>
      <edge source="2010.03179" target="1911.02116" id="1522">
        <attvalues>
          <attvalue for="5" value=" In this work, we analyse multilingual transformer models, namely mBERT \cite{models/BERT, models/mBERT} and XLM-RoBERTa \cite{models/RoBERTa}." />
        </attvalues>
      </edge>
      <edge source="2009.05781" target="1609.01454" id="1523">
        <attvalues>
          <attvalue for="5" value=" Most modern approaches use neural networks to jointly model intent detection and slot filling \cite{6707709,liu2016attentionbased,goo-etal-2018-slot,zhang-etal-2019-joint}." />
        </attvalues>
      </edge>
      <edge source="2009.05781" target="1812.09471" id="1524">
        <attvalues>
          <attvalue for="5" value=" Most modern approaches use neural networks to jointly model intent detection and slot filling \cite{6707709,liu2016attentionbased,goo-etal-2018-slot,zhang-etal-2019-joint}." />
        </attvalues>
      </edge>
      <edge source="2009.05781" target="1805.10190" id="1525">
        <attvalues>
          <attvalue for="5" value=" Using our pretraining task, we fine-tune transformer language models, achieving state-of-the-art results on the intent detection task of the Snips dataset \cite{coucke2018snips}, the Schema-Guided Dialog (SGD) dataset \cite{rastogi2019towards}, and all 3 languages (English, Spanish, and Thai) of the Facebook multilingual dialog datasets \cite{schuster-etal-2019-cross-lingual}, with statistically significant improvements." />
        </attvalues>
      </edge>
      <edge source="2009.05781" target="1810.13327" id="1527">
        <attvalues>
          <attvalue for="5" value=" Using our pretraining task, we fine-tune transformer language models, achieving state-of-the-art results on the intent detection task of the Snips dataset \cite{coucke2018snips}, the Schema-Guided Dialog (SGD) dataset \cite{rastogi2019towards}, and all 3 languages (English, Spanish, and Thai) of the Facebook multilingual dialog datasets \cite{schuster-etal-2019-cross-lingual}, with statistically significant improvements." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2303.17161" id="1528">
        <attvalues>
          <attvalue for="5" value=" Spoken Semantic Parsing (SSP) is the SLU task that involves transforming a recording to a machine-comprehensible parse tree ~\cite{wang2023treepiece}." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2305.01620" id="1529">
        <attvalues>
          <attvalue for="5" value=" End-to-end models ~\cite{arora2023study} operate directly on speech while cascade models~\cite{futami2023pipeline} generate a semantic parse based on the transcript." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2305.01194" id="1530">
        <attvalues>
          <attvalue for="5" value=" End-to-end models ~\cite{arora2023study} operate directly on speech while cascade models~\cite{futami2023pipeline} generate a semantic parse based on the transcript." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2204.01893" id="1531">
        <attvalues>
          <attvalue for="5" value=" Two-pass deliberation models ~\cite{le2022deliberation} combine the best of both worlds, by using first-pass transcripts and speech embeddings to improve spoken semantic parsing." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="1807.10857" id="1532">
        <attvalues>
          <attvalue for="5" value=" Prior work has explored the use of text data for speech recognition ~\cite{wang2020,toshniwal2018comparison,hori2019cycle}." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2110.05354" id="1534">
        <attvalues>
          <attvalue for="5" value=" External language models trained on text can be used to interpolate token prediction probabilities ~\cite{meng22_interspeech}, but require additional memory, making them unsuitable for on-device applications." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2204.03409" id="1535">
        <attvalues>
          <attvalue for="5" value=" Coordinated learning methods ~\cite{chen22r_interspeech,sainath2023joist} project speech and text to a shared embedding space for speech recognition, but such models require significant amounts of paired speech-text data to learn robust mappings." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2210.07353" id="1536">
        <attvalues>
          <attvalue for="5" value=" Coordinated learning methods ~\cite{chen22r_interspeech,sainath2023joist} project speech and text to a shared embedding space for speech recognition, but such models require significant amounts of paired speech-text data to learn robust mappings." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2211.00174" id="1537">
        <attvalues>
          <attvalue for="5" value=" The final class of work generates speech representations for unpaired speech - Joint Audio Text (JAT)~\cite{kim2022joint} uses mean speech embeddings from paired data to represent unpaired text." />
        </attvalues>
      </edge>
      <edge source="2309.09390" target="2307.09288" id="1540">
        <attvalues>
          <attvalue for="5" value="0~\cite{touvron2023llama2} to generate text data." />
        </attvalues>
      </edge>
      <edge source="2407.04307" target="2005.14165" id="1542">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs), such as BERT \cite{bert} and GPT3 \cite{gpt3}, have become instrumental in advancing Natural Language Processing (NLP) tasks." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="1802.05365" id="1543">
        <attvalues>
          <attvalue for="5" value=" Distributed representations of words~\cite[eg,][]{peters-etal-2018-deep,devlin-etal-2019-bert} have propelled the state-of-the-art across NLP to new heights." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="1908.08593" id="1545">
        <attvalues>
          <attvalue for="5" value=" Recently, there is much interest in probing these opaque representations to understand the information they bear~\cite[eg,][]{kovaleva-etal-2019-revealing,conneau-etal-2018-cram,jawahar-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="1911.03343" id="1546">
        <attvalues>
          <attvalue for="5" value=" The most commonly used strategy calls for training classifiers on them to predict linguistic properties such as syntax, or cognitive skills like numeracy~\cite[\eg][]{kassner-schutze-2020-negated,perone2018evaluation,yaghoobzadeh-etal-2019-probing,krasnowska-kieras-wroblewska-2019-empirical,wallace-etal-2019-nlp,pruksachatkun-etal-2020-intermediate}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="1806.06259" id="1547">
        <attvalues>
          <attvalue for="5" value=" The most commonly used strategy calls for training classifiers on them to predict linguistic properties such as syntax, or cognitive skills like numeracy~\cite[\eg][]{kassner-schutze-2020-negated,perone2018evaluation,yaghoobzadeh-etal-2019-probing,krasnowska-kieras-wroblewska-2019-empirical,wallace-etal-2019-nlp,pruksachatkun-etal-2020-intermediate}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="1906.03608" id="1548">
        <attvalues>
          <attvalue for="5" value=" The most commonly used strategy calls for training classifiers on them to predict linguistic properties such as syntax, or cognitive skills like numeracy~\cite[\eg][]{kassner-schutze-2020-negated,perone2018evaluation,yaghoobzadeh-etal-2019-probing,krasnowska-kieras-wroblewska-2019-empirical,wallace-etal-2019-nlp,pruksachatkun-etal-2020-intermediate}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="1909.07940" id="1549">
        <attvalues>
          <attvalue for="5" value=" The most commonly used strategy calls for training classifiers on them to predict linguistic properties such as syntax, or cognitive skills like numeracy~\cite[\eg][]{kassner-schutze-2020-negated,perone2018evaluation,yaghoobzadeh-etal-2019-probing,krasnowska-kieras-wroblewska-2019-empirical,wallace-etal-2019-nlp,pruksachatkun-etal-2020-intermediate}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="2005.04315" id="1550">
        <attvalues>
          <attvalue for="5" value=" Using these classifiers, criteria such as accuracy or model complexity are used to evaluate the representation quality for the task~\cite[\eg][]{goodwin-etal-2020-probing,pimentel-etal-2020-pareto,michael2020asking}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="2010.02180" id="1551">
        <attvalues>
          <attvalue for="5" value=" Using these classifiers, criteria such as accuracy or model complexity are used to evaluate the representation quality for the task~\cite[\eg][]{goodwin-etal-2020-probing,pimentel-etal-2020-pareto,michael2020asking}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="2004.14513" id="1552">
        <attvalues>
          <attvalue for="5" value=" Using these classifiers, criteria such as accuracy or model complexity are used to evaluate the representation quality for the task~\cite[\eg][]{goodwin-etal-2020-probing,pimentel-etal-2020-pareto,michael2020asking}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="1805.04905" id="1553">
        <attvalues>
          <attvalue for="5" value=" For example, in our experiments using the task of preposition supersense prediction~\cite{schneider-etal-2018-comprehensive}, we found that the accuracies across different training runs of the same classifier can vary by as much as $\sim 8\%$! (Detailed results can be found in \ref{sec:cls}." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="2004.14096" id="1555">
        <attvalues>
          <attvalue for="5" value=" A common compromise involves using linear classifiers to probe representations ~\cite{alain2016understanding,kulmizev-etal-2020-neural}, but doing so may mischaracterize representations that need non-linear separators." />
        </attvalues>
      </edge>
      <edge source="2104.05904" target="1906.01575" id="1557">
        <attvalues>
          <attvalue for="5" value=" Some work recognizes this problem~\cite{hewitt-liang-2019-designing} and proposes to report probing results for at least logistic regression and a multi-layer perceptron~\cite{eger-etal-2019-pitfalls}, or to compare the learning curves between multiple controls~\cite{talmor-etal-2020-olmpics}." />
        </attvalues>
      </edge>
      <edge source="2401.16818" target="2303.10420" id="1567">
        <attvalues>
          <attvalue for="5" value=" Research over the past few years has significantly enhanced language models' capabilities, making them pivotal in tasks like text and code generation, question answering, translation, summarization, and more \cite{ye2023comprehensive}." />
        </attvalues>
      </edge>
      <edge source="2401.16818" target="1706.03762" id="1568">
        <attvalues>
          <attvalue for="5" value=" Most state-of-the-art large language models (LLMs) leverage decoder attention architectures \cite{vaswani2017attention} popularized by the series of GPT models \cite{radford2018gpt1, radford2019gpt2, brown2020gpt3} exemplifying the benefits of pre-training such models on extensive text corpora." />
        </attvalues>
      </edge>
      <edge source="2401.16818" target="2001.08361" id="1569">
        <attvalues>
          <attvalue for="5" value=" Scaling laws for LLMs suggest that performance scales by factors such as model and dataset size, as well as computational resources for training \cite{kaplan2020scaling}." />
        </attvalues>
      </edge>
      <edge source="2401.16818" target="2306.01116" id="1570">
        <attvalues>
          <attvalue for="5" value=" This has led to the development of a plethora of models, ranging in size to optimize performance given certain data and compute constraints; notable representatives are: Falcon \cite{penedo2023refinedweb}, Llama 2 \cite{touvron2023llama}, Qwen \cite{bai2023qwen}, Mistral \cite{jiang2023mistral}, or Mixtral \cite{jiang2024mixtral}." />
        </attvalues>
      </edge>
      <edge source="2401.16818" target="2307.09288" id="1571">
        <attvalues>
          <attvalue for="5" value=" Fundamentally, H2O-Danube follows a decoder LLM architecture adopting core principles from Llama 2 \cite{touvron2023llama} and Mistral \cite{jiang2023mistral}." />
        </attvalues>
      </edge>
      <edge source="2401.16818" target="2310.06825" id="1573">
        <attvalues>
          <attvalue for="5" value=" Fundamentally, H2O-Danube follows a decoder LLM architecture adopting core principles from Llama 2 \cite{touvron2023llama} and Mistral \cite{jiang2023mistral}." />
        </attvalues>
      </edge>
      <edge source="2401.16818" target="2401.04088" id="1574">
        <attvalues>
          <attvalue for="5" value=" This has led to the development of a plethora of models, ranging in size to optimize performance given certain data and compute constraints; notable representatives are: Falcon \cite{penedo2023refinedweb}, Llama 2 \cite{touvron2023llama}, Qwen \cite{bai2023qwen}, Mistral \cite{jiang2023mistral}, or Mixtral \cite{jiang2024mixtral}." />
        </attvalues>
      </edge>
      <edge source="2401.16818" target="2304.01373" id="1577">
        <attvalues>
          <attvalue for="5" value=" In this report, we want to extend previous research in this area \cite{biderman2023pythia, zhang2024tinyllama, zhang2022opt, bai2023qwen, stablelm} and present a series of models based on incremental research and training efforts." />
        </attvalues>
      </edge>
      <edge source="2406.17377" target="2005.14165" id="1578">
        <attvalues>
          <attvalue for="5" value=" Large language models \cite[LLM;][]{brown2020language, touvron2023llama, chowdhery2022palm, gemmateam2024gemma} are known to generalise well across several tasks, including in few shot and zero-shot setups." />
        </attvalues>
      </edge>
      <edge source="2406.17377" target="2004.09095" id="1579">
        <attvalues>
          <attvalue for="5" value=" These resource-poor languages tend to get poorer in representation with the progress in the field \cite{joshi-etal-2020-state, ojo2024good}." />
        </attvalues>
      </edge>
      <edge source="2406.17377" target="2307.09288" id="1580">
        <attvalues>
          <attvalue for="5" value="005\%$ of the pre-training data of an open-source LLM like Llama-2 \cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2406.17377" target="2310.16393" id="1581">
        <attvalues>
          <attvalue for="5" value=" Some of the recent works, explore various techniques to adapt an LLM to new languages, especially with limited target language resources \cite{rathore-etal-2023-zgul}." />
        </attvalues>
      </edge>
      <edge source="2406.17377" target="2305.05940" id="1582">
        <attvalues>
          <attvalue for="5" value=" \cite{tanwar2023multilingual} exploit cross-lingual transfer to improve in-context learning (ICL) for binary sequence classification tasks in low-resource languages by utilizing in-context exemplars from a high-resource language semantically similar to the input in the target language." />
        </attvalues>
      </edge>
      <edge source="2406.17377" target="2204.02311" id="1584">
        <attvalues>
          <attvalue for="5" value=" \cite{awasthi2023bootstrapping} use $540$b PaLM \cite{chowdhery2022palm} to generate training data in low-resource languages using labelled instances in English." />
        </attvalues>
      </edge>
      <edge source="2403.02990" target="2303.10158" id="1586">
        <attvalues>
          <attvalue for="5" value=" Data-centric approaches to Artificial Intelligence (AI) constitute a pivotal element in the advancement towards Artificial General Intelligence (AGI), centering on the construction of AI systems underpinned by high-quality data \cite{zha2023data}." />
        </attvalues>
      </edge>
      <edge source="2403.02990" target="2106.07499" id="1588">
        <attvalues>
          <attvalue for="5" value=" In response to these challenges, researchers have dedicated efforts towards data augmentation (DA) techniques as a means to mitigate such issues \cite{chen-etal-2023-empirical}." />
        </attvalues>
      </edge>
      <edge source="2403.02990" target="2001.08361" id="1589">
        <attvalues>
          <attvalue for="5" value=" Research into the scaling laws pertinent to LLMs highlights the critical role of data as a renewable resource crucial for the enhancement and advancement of models \cite{kaplan2020scaling}." />
        </attvalues>
      </edge>
      <edge source="2403.02990" target="2304.03277" id="1590">
        <attvalues>
          <attvalue for="5" value=" From the data perspectives, data augmentation using LLMs offers a viable strategy to overcome these limitations, facilitating the creation of synthetic datasets of high quality that can, in certain instances, exceed the value of data curated by humans \cite{peng2023instruction}." />
        </attvalues>
      </edge>
      <edge source="2403.02990" target="2203.15556" id="1591">
        <attvalues>
          <attvalue for="5" value=" This strategy not only addresses the challenge posed by the limited supply of human-annotated data but also conforms to scaling laws, enabling an increase in the size of training datasets without a proportional escalation in computational costs (FLOPs) \cite{Hoffmann2022TrainingCL}." />
        </attvalues>
      </edge>
      <edge source="2403.02990" target="2310.15638" id="1592">
        <attvalues>
          <attvalue for="5" value=" The tactical employment of synthetic data stands to substantially reduce data collection costs and energy usage, signifying a transformative phase in model training and inference and laying the groundwork for the achievement of artificial general intelligence (AGI) \cite{li2023coannotating}." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2003.08271" id="1593">
        <attvalues>
          <attvalue for="5" value=" In recent years, pretrained language models (PLMs) have achieved huge success in NLP~\cite{qiu2020pre}." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="1907.11692" id="1595">
        <attvalues>
          <attvalue for="5" value=" Many PLMs such as BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and UniLM~\cite{dong2019unified} which are pretrained from large-scale unlabeled corpus in a self-supervised way, have significantly improve various downstream tasks such as reading comprehension~\cite{xu2019bert}, machine translation~\cite{brown2020language}, text classification~\cite{bao2020unilmv2}, dialog~\cite{wu2020tod} and recommendation~\cite{wu2021plm} by finetuning on these tasks." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="1905.03197" id="1596">
        <attvalues>
          <attvalue for="5" value=" Many PLMs such as BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and UniLM~\cite{dong2019unified} which are pretrained from large-scale unlabeled corpus in a self-supervised way, have significantly improve various downstream tasks such as reading comprehension~\cite{xu2019bert}, machine translation~\cite{brown2020language}, text classification~\cite{bao2020unilmv2}, dialog~\cite{wu2020tod} and recommendation~\cite{wu2021plm} by finetuning on these tasks." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="1904.02232" id="1597">
        <attvalues>
          <attvalue for="5" value=" Many PLMs such as BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and UniLM~\cite{dong2019unified} which are pretrained from large-scale unlabeled corpus in a self-supervised way, have significantly improve various downstream tasks such as reading comprehension~\cite{xu2019bert}, machine translation~\cite{brown2020language}, text classification~\cite{bao2020unilmv2}, dialog~\cite{wu2020tod} and recommendation~\cite{wu2021plm} by finetuning on these tasks." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2005.14165" id="1598">
        <attvalues>
          <attvalue for="5" value=" Many PLMs such as BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and UniLM~\cite{dong2019unified} which are pretrained from large-scale unlabeled corpus in a self-supervised way, have significantly improve various downstream tasks such as reading comprehension~\cite{xu2019bert}, machine translation~\cite{brown2020language}, text classification~\cite{bao2020unilmv2}, dialog~\cite{wu2020tod} and recommendation~\cite{wu2021plm} by finetuning on these tasks." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2002.12804" id="1599">
        <attvalues>
          <attvalue for="5" value=" Many PLMs such as BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and UniLM~\cite{dong2019unified} which are pretrained from large-scale unlabeled corpus in a self-supervised way, have significantly improve various downstream tasks such as reading comprehension~\cite{xu2019bert}, machine translation~\cite{brown2020language}, text classification~\cite{bao2020unilmv2}, dialog~\cite{wu2020tod} and recommendation~\cite{wu2021plm} by finetuning on these tasks." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2004.06871" id="1600">
        <attvalues>
          <attvalue for="5" value=" Many PLMs such as BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and UniLM~\cite{dong2019unified} which are pretrained from large-scale unlabeled corpus in a self-supervised way, have significantly improve various downstream tasks such as reading comprehension~\cite{xu2019bert}, machine translation~\cite{brown2020language}, text classification~\cite{bao2020unilmv2}, dialog~\cite{wu2020tod} and recommendation~\cite{wu2021plm} by finetuning on these tasks." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2104.07413" id="1601">
        <attvalues>
          <attvalue for="5" value=" Many PLMs such as BERT~\cite{devlin2019bert}, RoBERTa~\cite{liu2019roberta} and UniLM~\cite{dong2019unified} which are pretrained from large-scale unlabeled corpus in a self-supervised way, have significantly improve various downstream tasks such as reading comprehension~\cite{xu2019bert}, machine translation~\cite{brown2020language}, text classification~\cite{bao2020unilmv2}, dialog~\cite{wu2020tod} and recommendation~\cite{wu2021plm} by finetuning on these tasks." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="1905.05583" id="1603">
        <attvalues>
          <attvalue for="5" value=" Many existing NLP methods usually directly finetune PLMs with the labeled data in downstream tasks~\cite{sun2019fine}." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2008.03156" id="1606">
        <attvalues>
          <attvalue for="5" value=" Only a few works explore more effective and robust PLM finetuning methods~\cite{chen2020recall,Lee2020mixout,aghajanyan2020better,zhang2021revisiting,xu2021raise}." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2006.05987" id="1607">
        <attvalues>
          <attvalue for="5" value=" Only a few works explore more effective and robust PLM finetuning methods~\cite{chen2020recall,Lee2020mixout,aghajanyan2020better,zhang2021revisiting,xu2021raise}." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2109.05687" id="1608">
        <attvalues>
          <attvalue for="5" value=" Only a few works explore more effective and robust PLM finetuning methods~\cite{chen2020recall,Lee2020mixout,aghajanyan2020better,zhang2021revisiting,xu2021raise}." />
        </attvalues>
      </edge>
      <edge source="2202.12024" target="2002.08910" id="1609">
        <attvalues>
          <attvalue for="5" value=" It is not easy for existing PLM finetuning methods to overcome such gap~\cite{roberts2020much}, which may lead to suboptimal performance especially when labeled data in downstream tasks is insufficient." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2303.12712" id="1611">
        <attvalues>
          <attvalue for="5" value=" In recent years, the field of natural language processing (NLP) has witnessed a profound transformation, fueled by the advent of large language models (LLMs) \cite{bubeck2023sparks,touvron2023llama, achiam2023gpt}, which have emerged as a cornerstone to revolutionize the way we understand and generate human language." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2303.18223" id="1613">
        <attvalues>
          <attvalue for="5" value=" However, two obstacles stick out in the pretraining stage \cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2304.13712" id="1614">
        <attvalues>
          <attvalue for="5" value=" The extensive training endows LLMs with a deep understanding of linguistic structures, nuances, and context, enabling them to generate human-like text and perform a myriad of NLP tasks with unprecedented accuracy and fluency \cite{yang2024harnessing}." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2308.10792" id="1615">
        <attvalues>
          <attvalue for="5" value=" The training journey can be divided into two stages: the pretraining stage and the alignment stage \cite{zhang2023instruction}." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2402.00159" id="1616">
        <attvalues>
          <attvalue for="5" value=" While extensive research has delved into data cleaning and sampling methodologies \cite{soldaini2024dolma, penedo2023refinedweb, wenzek2019ccnet, gunasekar2023textbooks}, the sheer scale and intricacy of pretraining datasets still leave ample room for elevating informational density and efficiency." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2307.03109" id="1617">
        <attvalues>
          <attvalue for="5" value=" Second, establishing a stable and sensitive ablation environment for accurately assessing data strategies poses another challenge \cite{chang2024survey,zhou2023don}." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2311.01964" id="1618">
        <attvalues>
          <attvalue for="5" value=" Second, establishing a stable and sensitive ablation environment for accurately assessing data strategies poses another challenge \cite{chang2024survey,zhou2023don}." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2310.19341" id="1619">
        <attvalues>
          <attvalue for="5" value=" It is inherently unstable and insensitive to smaller models or datasets \cite{wei2023skywork}, and further, it lacks correlation with downstream skills to adequately evaluate the model." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2402.16827" id="1620">
        <attvalues>
          <attvalue for="5" value=" In the alignment stage, challenges arise regarding data \cite{albalak2024survey}, long context \cite{niah2023v0} and RLHF effectiveness \cite{wang2024secrets,xu2024dpo}." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2404.10719" id="1622">
        <attvalues>
          <attvalue for="5" value=" In the alignment stage, challenges arise regarding data \cite{albalak2024survey}, long context \cite{niah2023v0} and RLHF effectiveness \cite{wang2024secrets,xu2024dpo}." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2312.15685" id="1623">
        <attvalues>
          <attvalue for="5" value=" SFT gets major parts of things done, but is to some extent sensitive to data quality and composition \cite{liu2024what}." />
        </attvalues>
      </edge>
      <edge source="2405.13386" target="2303.08774" id="1625">
        <attvalues>
          <attvalue for="5" value=" RLHF in open-sourced models has not yet fulfilled the presumed promise as \cite{achiam2023gpt}." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2005.14165" id="1626">
        <attvalues>
          <attvalue for="5" value=" By pre-training large Transformer models on massive text corpora, LLMs can possess excellent task-solving capacities, eg, using zero-shot or few-shot prompting~\cite{GPT-3}." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2303.18223" id="1627">
        <attvalues>
          <attvalue for="5" value="'' \end{quote} Large language models (LLMs) have achieved remarkable success across a variety of real-world applications~\cite{GPT-3,LLMsurvey,DBLP:journals/corr/abs-2308-07107}." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2308.07107" id="1628">
        <attvalues>
          <attvalue for="5" value="'' \end{quote} Large language models (LLMs) have achieved remarkable success across a variety of real-world applications~\cite{GPT-3,LLMsurvey,DBLP:journals/corr/abs-2308-07107}." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2009.03300" id="1629">
        <attvalues>
          <attvalue for="5" value=" Typical benchmarks include MMLU~\cite{mmlu} (for measuring multitask language understanding ability), Big-Bench~\cite{srivastava2023beyond} (for quantifying and extrapolating the capabilities of LLMs), and AGIEval~\cite{zhong2023agieval} (for evaluating the abilities of tackling human-level tasks)." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2305.08322" id="1630">
        <attvalues>
          <attvalue for="5" value=" Furthermore, to compare the performance of different LLMs, various leaderboards have been also created to rank LLMs according to their performance on existing or new evaluation benchmarks, such as OpenCompass~\cite{2023opencompass} and C-Eval~\cite{huang2023c}." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2206.04615" id="1631">
        <attvalues>
          <attvalue for="5" value=" Typical benchmarks include MMLU~\cite{mmlu} (for measuring multitask language understanding ability), Big-Bench~\cite{srivastava2023beyond} (for quantifying and extrapolating the capabilities of LLMs), and AGIEval~\cite{zhong2023agieval} (for evaluating the abilities of tackling human-level tasks)." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2304.06364" id="1632">
        <attvalues>
          <attvalue for="5" value=" Typical benchmarks include MMLU~\cite{mmlu} (for measuring multitask language understanding ability), Big-Bench~\cite{srivastava2023beyond} (for quantifying and extrapolating the capabilities of LLMs), and AGIEval~\cite{zhong2023agieval} (for evaluating the abilities of tackling human-level tasks)." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2303.08774" id="1633">
        <attvalues>
          <attvalue for="5" value=" It has become common to report the results on these evaluation benchmarks for demonstrating the effectiveness of newly released LLMs~\cite{GPT-4,llama2,anil2023palm}." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2307.09288" id="1634">
        <attvalues>
          <attvalue for="5" value=" It has become common to report the results on these evaluation benchmarks for demonstrating the effectiveness of newly released LLMs~\cite{GPT-4,llama2,anil2023palm}." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2305.10403" id="1635">
        <attvalues>
          <attvalue for="5" value=" It has become common to report the results on these evaluation benchmarks for demonstrating the effectiveness of newly released LLMs~\cite{GPT-4,llama2,anil2023palm}." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="2303.12767" id="1636">
        <attvalues>
          <attvalue for="5" value=" Despite the wide use of these benchmarks and leaderboards, increasing concerns~\cite{aiyappa2023can,li2023open} are growing about the fairness and reliability in evaluating existing LLMs." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="1511.02301" id="1637">
        <attvalues>
          <attvalue for="5" value=" For example, GPT-3 has found that Children’s Book Test dataset~\cite{Hill2015TheGP} was included in the pre-training corpus, and LLaMA-2 has mentioned that the contexts in BoolQ dataset~\cite{boolq} are extracted verbatim from the webpages, which may be included in the publicly available corpus." />
        </attvalues>
      </edge>
      <edge source="2311.01964" target="1905.10044" id="1638">
        <attvalues>
          <attvalue for="5" value=" For example, GPT-3 has found that Children’s Book Test dataset~\cite{Hill2015TheGP} was included in the pre-training corpus, and LLaMA-2 has mentioned that the contexts in BoolQ dataset~\cite{boolq} are extracted verbatim from the webpages, which may be included in the publicly available corpus." />
        </attvalues>
      </edge>
      <edge source="1909.04702" target="1310.4546" id="1647">
        <attvalues>
          <attvalue for="5" value=" Word embedding models such as the skip-gram improve the performance of Natural Language Processing (NLP) methods by revealing the latent structural relationship between words ~\cite{mikolov2013efficient,mikolov2013distributed}." />
        </attvalues>
      </edge>
      <edge source="1909.04702" target="1103.0398" id="1648">
        <attvalues>
          <attvalue for="5" value=" These embeddings have proven valuable for a variety of NLP tasks such as statistical machine translation~\cite{vaswani2013decoding}, part-of-speech tagging, chunking, and named entity recognition ~\cite{collobert2011natural}." />
        </attvalues>
      </edge>
      <edge source="1909.04702" target="1405.4053" id="1649">
        <attvalues>
          <attvalue for="5" value=" Extensions to document embeddings have subsequently been proposed \cite{le2014distributed}." />
        </attvalues>
      </edge>
      <edge source="1909.04702" target="1705.07368" id="1650">
        <attvalues>
          <attvalue for="5" value=" We show the benefits and generality of our method by applying it to LDA, author-topic models (ATM)~\cite{rosen2004author}, and the recently proposed mixed membership skip gram topic model (MMSGTM)~\cite{foulds2018mixed}." />
        </attvalues>
      </edge>
      <edge source="1909.04702" target="2105.10059" id="1651">
        <attvalues>
          <attvalue for="5" value=" Our approach is thus reminiscent of model distillation for supervised models \cite{bucilua2006model, hinton2015distilling}." />
        </attvalues>
      </edge>
      <edge source="2204.08039" target="2005.14165" id="1654">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models \cite{NEURIPS2020_1457c0d6, liu2019roberta, devlin-etal-2019-bert} have shown impressive adaptation ability to dowstream tasks, achieving considerable performance even with scarce task-specific training data, eg, few-shot adaptation \cite{radford2019language, schick-schutze-2021-exploiting, gao-etal-2021-making}." />
        </attvalues>
      </edge>
      <edge source="2204.08039" target="2010.15980" id="1655">
        <attvalues>
          <attvalue for="5" value=" Existing few-shot adaptation techniques broadly fall in fine-tuning and few-shot learning \cite{shin-etal-2020-autoprompt, schick-schutze-2021-just, chen-etal-2021-revisiting}." />
        </attvalues>
      </edge>
      <edge source="2204.08039" target="1909.01066" id="1656">
        <attvalues>
          <attvalue for="5" value=" Specifically, fine-tuning includes directly tuning pre-trained language models with few task-specific examples or utilizing a natural-language prompt to transform downstream tasks to masked language modeling task for better mining knowledge from pre-trained models \cite{petroni-etal-2019-language, jiang-etal-2020-know, wang-etal-2021-transprompt}." />
        </attvalues>
      </edge>
      <edge source="2204.08039" target="1911.03809" id="1657">
        <attvalues>
          <attvalue for="5" value=" Few-shot learning leverages unlabeled data or auxiliary tasks to provide additional information for facilitating model training \cite{zheng2021meta, wang2021meta, du-etal-2021-self}." />
        </attvalues>
      </edge>
      <edge source="2204.08039" target="2109.04144" id="1658">
        <attvalues>
          <attvalue for="5" value=" \cite{utama2021avoiding} found that models obtained from few-shot prompt-based fine-tuning utilize inference heuristics to make predictions on sentence pair classification tasks." />
        </attvalues>
      </edge>
      <edge source="2204.08039" target="2102.09690" id="1659">
        <attvalues>
          <attvalue for="5" value=" \cite{zhao2021calibrate} discovered the instability of model performance towards different prompts in few-shot learning." />
        </attvalues>
      </edge>
      <edge source="2204.08039" target="1908.05267" id="1660">
        <attvalues>
          <attvalue for="5" value=" We model the statistics of important features over prediction labels via local mutual information (LMI) \cite{schuster-etal-2019-towards, du-etal-2021-towards}." />
        </attvalues>
      </edge>
      <edge source="2204.08039" target="1907.11692" id="1662">
        <attvalues>
          <attvalue for="5" value=" We evaluate two pre-trained language models, BERT \cite{devlin-etal-2019-bert} and RoBERTa \cite{liu2019roberta}, on three tasks, including sentiment classification, natural language inference, and paraphrase identification." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2004.06165" id="1663">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2104.11178" id="1664">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2106.11097" id="1665">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2112.01194" id="1666">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2107.07651" id="1667">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2201.12086" id="1669">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2201.02639" id="1670">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2111.08276" id="1671">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2202.10401" id="1672">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2202.03052" id="1673">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2208.10442" id="1674">
        <attvalues>
          <attvalue for="5" value=" MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks~\cite{li2020oscar,akbari2021vatt,fang2021clip2video,yan2021video,li2021align,radford2021learning,li2022blip,zellers2022merlot,zeng2022multi,yang2022vision,wang2022ofa,wang2022image}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2303.08774" id="1675">
        <attvalues>
          <attvalue for="5" value=" With the debut of GPT-4(Vision)~\cite{openai2023gpt4} and Gemini~\cite{team2023gemini}, showcasing impressive MM understanding and generation capabilities, a research fervor on MM-LLMs has been sparked." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2312.11805" id="1676">
        <attvalues>
          <attvalue for="5" value=" With the debut of GPT-4(Vision)~\cite{openai2023gpt4} and Gemini~\cite{team2023gemini}, showcasing impressive MM understanding and generation capabilities, a research fervor on MM-LLMs has been sparked." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2301.12597" id="1677">
        <attvalues>
          <attvalue for="5" value=" Initial research primarily focuses on MM content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like BLIP-2~\cite{DBLP:conf/icml/0008LSH23}, LLaVA~\cite{liu2023llava}, MiniGPT-4~\cite{zhu2023minigpt}, and OpenFlamingo~\cite{awadalla2023openflamingo}; video-text understanding, as demonstrated by initiatives such as VideoChat~\cite{li2023videochat}, Video-ChatGPT~\cite{maaz2023video}, and LLaMA-VID~\cite{li2023llama}; and audio-text understanding, as seen in projects like Qwen-Audio~\cite{chu2023qwen}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2304.10592" id="1679">
        <attvalues>
          <attvalue for="5" value=" Initial research primarily focuses on MM content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like BLIP-2~\cite{DBLP:conf/icml/0008LSH23}, LLaVA~\cite{liu2023llava}, MiniGPT-4~\cite{zhu2023minigpt}, and OpenFlamingo~\cite{awadalla2023openflamingo}; video-text understanding, as demonstrated by initiatives such as VideoChat~\cite{li2023videochat}, Video-ChatGPT~\cite{maaz2023video}, and LLaMA-VID~\cite{li2023llama}; and audio-text understanding, as seen in projects like Qwen-Audio~\cite{chu2023qwen}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2308.01390" id="1680">
        <attvalues>
          <attvalue for="5" value=" Initial research primarily focuses on MM content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like BLIP-2~\cite{DBLP:conf/icml/0008LSH23}, LLaVA~\cite{liu2023llava}, MiniGPT-4~\cite{zhu2023minigpt}, and OpenFlamingo~\cite{awadalla2023openflamingo}; video-text understanding, as demonstrated by initiatives such as VideoChat~\cite{li2023videochat}, Video-ChatGPT~\cite{maaz2023video}, and LLaMA-VID~\cite{li2023llama}; and audio-text understanding, as seen in projects like Qwen-Audio~\cite{chu2023qwen}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2305.06355" id="1681">
        <attvalues>
          <attvalue for="5" value=" Initial research primarily focuses on MM content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like BLIP-2~\cite{DBLP:conf/icml/0008LSH23}, LLaVA~\cite{liu2023llava}, MiniGPT-4~\cite{zhu2023minigpt}, and OpenFlamingo~\cite{awadalla2023openflamingo}; video-text understanding, as demonstrated by initiatives such as VideoChat~\cite{li2023videochat}, Video-ChatGPT~\cite{maaz2023video}, and LLaMA-VID~\cite{li2023llama}; and audio-text understanding, as seen in projects like Qwen-Audio~\cite{chu2023qwen}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2311.17043" id="1683">
        <attvalues>
          <attvalue for="5" value=" Initial research primarily focuses on MM content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like BLIP-2~\cite{DBLP:conf/icml/0008LSH23}, LLaVA~\cite{liu2023llava}, MiniGPT-4~\cite{zhu2023minigpt}, and OpenFlamingo~\cite{awadalla2023openflamingo}; video-text understanding, as demonstrated by initiatives such as VideoChat~\cite{li2023videochat}, Video-ChatGPT~\cite{maaz2023video}, and LLaMA-VID~\cite{li2023llama}; and audio-text understanding, as seen in projects like Qwen-Audio~\cite{chu2023qwen}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2311.07919" id="1684">
        <attvalues>
          <attvalue for="5" value=" Initial research primarily focuses on MM content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like BLIP-2~\cite{DBLP:conf/icml/0008LSH23}, LLaVA~\cite{liu2023llava}, MiniGPT-4~\cite{zhu2023minigpt}, and OpenFlamingo~\cite{awadalla2023openflamingo}; video-text understanding, as demonstrated by initiatives such as VideoChat~\cite{li2023videochat}, Video-ChatGPT~\cite{maaz2023video}, and LLaMA-VID~\cite{li2023llama}; and audio-text understanding, as seen in projects like Qwen-Audio~\cite{chu2023qwen}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2305.17216" id="1685">
        <attvalues>
          <attvalue for="5" value=" This includes tasks with image-text output, such as GILL~\cite{koh2023generating}, Kosmos-2~\cite{peng2023kosmos}, Emu~\cite{sun2023generative}, and MiniGPT-5~\cite{zheng2023minigpt}; as well as speech/audio-text output, exemplified by projects like SpeechGPT~\cite{DBLP:conf/emnlp/ZhangLZZWZQ23} and AudioPaLM~\cite{rubenstein2023audiopalm}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2306.14824" id="1686">
        <attvalues>
          <attvalue for="5" value=" This includes tasks with image-text output, such as GILL~\cite{koh2023generating}, Kosmos-2~\cite{peng2023kosmos}, Emu~\cite{sun2023generative}, and MiniGPT-5~\cite{zheng2023minigpt}; as well as speech/audio-text output, exemplified by projects like SpeechGPT~\cite{DBLP:conf/emnlp/ZhangLZZWZQ23} and AudioPaLM~\cite{rubenstein2023audiopalm}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2310.02239" id="1687">
        <attvalues>
          <attvalue for="5" value=" This includes tasks with image-text output, such as GILL~\cite{koh2023generating}, Kosmos-2~\cite{peng2023kosmos}, Emu~\cite{sun2023generative}, and MiniGPT-5~\cite{zheng2023minigpt}; as well as speech/audio-text output, exemplified by projects like SpeechGPT~\cite{DBLP:conf/emnlp/ZhangLZZWZQ23} and AudioPaLM~\cite{rubenstein2023audiopalm}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2305.11000" id="1688">
        <attvalues>
          <attvalue for="5" value=" This includes tasks with image-text output, such as GILL~\cite{koh2023generating}, Kosmos-2~\cite{peng2023kosmos}, Emu~\cite{sun2023generative}, and MiniGPT-5~\cite{zheng2023minigpt}; as well as speech/audio-text output, exemplified by projects like SpeechGPT~\cite{DBLP:conf/emnlp/ZhangLZZWZQ23} and AudioPaLM~\cite{rubenstein2023audiopalm}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2306.12925" id="1689">
        <attvalues>
          <attvalue for="5" value=" This includes tasks with image-text output, such as GILL~\cite{koh2023generating}, Kosmos-2~\cite{peng2023kosmos}, Emu~\cite{sun2023generative}, and MiniGPT-5~\cite{zheng2023minigpt}; as well as speech/audio-text output, exemplified by projects like SpeechGPT~\cite{DBLP:conf/emnlp/ZhangLZZWZQ23} and AudioPaLM~\cite{rubenstein2023audiopalm}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2303.04671" id="1690">
        <attvalues>
          <attvalue for="5" value=" Some efforts aim to amalgamate LLMs with external tools to reach an approaching any-to-any MM comprehension and generation, such as Visual-ChatGPT~\cite{wu2023visual}, HuggingGPT~\cite{shen2023hugginggpt}, and AudioGPT~\cite{huang2023audiogpt}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2304.12995" id="1691">
        <attvalues>
          <attvalue for="5" value=" Some efforts aim to amalgamate LLMs with external tools to reach an approaching any-to-any MM comprehension and generation, such as Visual-ChatGPT~\cite{wu2023visual}, HuggingGPT~\cite{shen2023hugginggpt}, and AudioGPT~\cite{huang2023audiogpt}." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2309.05519" id="1692">
        <attvalues>
          <attvalue for="5" value=" Conversely, to mitigate propagated errors in the cascade system, initiatives like NExT-GPT~\cite{wu2023next}, CoDi-2~\cite{tang2023codi}, and ModaVerse~\cite{wang2024modaverse} have developed end-to-end MM-LLMs of arbitrary modalities." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2305.11846" id="1693">
        <attvalues>
          <attvalue for="5" value=" Conversely, to mitigate propagated errors in the cascade system, initiatives like NExT-GPT~\cite{wu2023next}, CoDi-2~\cite{tang2023codi}, and ModaVerse~\cite{wang2024modaverse} have developed end-to-end MM-LLMs of arbitrary modalities." />
        </attvalues>
      </edge>
      <edge source="2401.13601" target="2401.06395" id="1694">
        <attvalues>
          <attvalue for="5" value=" Conversely, to mitigate propagated errors in the cascade system, initiatives like NExT-GPT~\cite{wu2023next}, CoDi-2~\cite{tang2023codi}, and ModaVerse~\cite{wang2024modaverse} have developed end-to-end MM-LLMs of arbitrary modalities." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="1606.05250" id="1695">
        <attvalues>
          <attvalue for="5" value=" The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="1704.00051" id="1696">
        <attvalues>
          <attvalue for="5" value=" The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="1910.13461" id="1698">
        <attvalues>
          <attvalue for="5" value=" The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2307.09288" id="1699">
        <attvalues>
          <attvalue for="5" value=" The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2310.06825" id="1700">
        <attvalues>
          <attvalue for="5" value=" The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2303.08774" id="1701">
        <attvalues>
          <attvalue for="5" value=" The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2111.01243" id="1702">
        <attvalues>
          <attvalue for="5" value=" The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2107.12708" id="1703">
        <attvalues>
          <attvalue for="5" value=" The availability of numerous English benchmarks that frame the problem as extractive, cloze-style or open-domain \cite{yang2015wikiqa,rajpurkar2016squad,chen2017reading} reasoning tasks, along with novel pre-trained language models (PLMs) \cite{devlin2018bert,lewis2019bart} and LLMs \cite{touvron2023llama,jiang2023mistral,achiam2023gpt} allowed for the development and granular evaluation of QA systems that occasionally boast human-like or better performance \cite{devlin2018bert,min2023recent,rogers2023qa}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="1910.07475" id="1704">
        <attvalues>
          <attvalue for="5" value=" Following \cite{lewis2019mlqa}, we filter out examples that do not contain the answer substring verbatim in the paragraph and additionally perform a human evaluation on a subset of $50$ examples and show that $98\%$ of these question-answer pairs are answerable and maintain quality." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="1809.03275" id="1705">
        <attvalues>
          <attvalue for="5" value=" Although some concentrated effort has been made to create multilingual QA resources \cite{lewis2019mlqa,asai2018multilingual,liu2019xqa}, the datasets remain rather scarce and usually cover a small selected set of languages due to the labour-intensive annotation costs." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="1912.05200" id="1706">
        <attvalues>
          <attvalue for="5" value=" The proposed methods suggest using direct machine translation \cite{lewis2019mlqa,carrino2019automatic} or multilingual synthetic data generation \cite{riabi2020synthetic,agrawal2023qameleon,shakeri2020towards}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2010.12643" id="1707">
        <attvalues>
          <attvalue for="5" value=" The proposed methods suggest using direct machine translation \cite{lewis2019mlqa,carrino2019automatic} or multilingual synthetic data generation \cite{riabi2020synthetic,agrawal2023qameleon,shakeri2020towards}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2211.08264" id="1708">
        <attvalues>
          <attvalue for="5" value=" The proposed methods suggest using direct machine translation \cite{lewis2019mlqa,carrino2019automatic} or multilingual synthetic data generation \cite{riabi2020synthetic,agrawal2023qameleon,shakeri2020towards}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2004.04721" id="1710">
        <attvalues>
          <attvalue for="5" value=" However, these approaches are directly bound to introduce biases and hallucinations during translation \cite{artetxe2020translation}, cross-lingual transfer \cite{lauscher2020zero, guerreiro2023hallucinations} or generation \cite{ahuja2023mega}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2005.00633" id="1711">
        <attvalues>
          <attvalue for="5" value=" However, these approaches are directly bound to introduce biases and hallucinations during translation \cite{artetxe2020translation}, cross-lingual transfer \cite{lauscher2020zero, guerreiro2023hallucinations} or generation \cite{ahuja2023mega}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2303.12528" id="1712">
        <attvalues>
          <attvalue for="5" value=" However, these approaches are directly bound to introduce biases and hallucinations during translation \cite{artetxe2020translation}, cross-lingual transfer \cite{lauscher2020zero, guerreiro2023hallucinations} or generation \cite{ahuja2023mega}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="2102.07033" id="1713">
        <attvalues>
          <attvalue for="5" value=" Similar mining approaches have been shown to be efficient for this task \cite{lewis2021paq,artetxe2019massively}." />
        </attvalues>
      </edge>
      <edge source="2406.14425" target="1812.10464" id="1714">
        <attvalues>
          <attvalue for="5" value=" Similar mining approaches have been shown to be efficient for this task \cite{lewis2021paq,artetxe2019massively}." />
        </attvalues>
      </edge>
      <edge source="2308.02019" target="2203.15556" id="1715">
        <attvalues>
          <attvalue for="5" value=" \cite{hoffmann2022training} have observed that in order to train a model in a compute-optimal way, the number of parameters and dataset size should follow a linear relation: the so-called Chinchilla scaling law, with an optimal ratio of about 20~tokens per model parameter." />
        </attvalues>
      </edge>
      <edge source="2308.02019" target="2211.09085" id="1716">
        <attvalues>
          <attvalue for="5" value=" \cite{taylor2022galactica} have shown that training models on higher-quality data can improve performance; however, the quantity of such high-quality data is limited, and often represents only a small fraction of the corpus." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2005.12592" id="1717">
        <attvalues>
          <attvalue for="5" value=" In GEC, various methods have been proposed from a wide range of perspectives, including correction performance~\cite{grundkiewicz-junczys-dowmunt-2019-minimally,chollampatt-etal-2019-cross,omelianchuk-etal-2020-gector,kaneko-etal-2020-encoder,qorib-etal-2022-frustratingly}, controlling~\cite{hotate-etal-2019-controlling,yang2022controllable,loem2023exploring}, diversity~\cite{xie-etal-2018-noising,hotate-etal-2020-generating,han2021diversity}, and efficiency~\cite{malmi-etal-2019-encode,chen-etal-2020-improving-efficiency}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2005.00987" id="1718">
        <attvalues>
          <attvalue for="5" value=" In GEC, various methods have been proposed from a wide range of perspectives, including correction performance~\cite{grundkiewicz-junczys-dowmunt-2019-minimally,chollampatt-etal-2019-cross,omelianchuk-etal-2020-gector,kaneko-etal-2020-encoder,qorib-etal-2022-frustratingly}, controlling~\cite{hotate-etal-2019-controlling,yang2022controllable,loem2023exploring}, diversity~\cite{xie-etal-2018-noising,hotate-etal-2020-generating,han2021diversity}, and efficiency~\cite{malmi-etal-2019-encode,chen-etal-2020-improving-efficiency}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2305.18156" id="1720">
        <attvalues>
          <attvalue for="5" value=" \cite{loem2023exploring} showed that prompting did not contribute significantly to the control of correction style for GPT-3." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2110.15149" id="1721">
        <attvalues>
          <attvalue for="5" value=" In GEC, various methods have been proposed from a wide range of perspectives, including correction performance~\cite{grundkiewicz-junczys-dowmunt-2019-minimally,chollampatt-etal-2019-cross,omelianchuk-etal-2020-gector,kaneko-etal-2020-encoder,qorib-etal-2022-frustratingly}, controlling~\cite{hotate-etal-2019-controlling,yang2022controllable,loem2023exploring}, diversity~\cite{xie-etal-2018-noising,hotate-etal-2020-generating,han2021diversity}, and efficiency~\cite{malmi-etal-2019-encode,chen-etal-2020-improving-efficiency}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="1909.01187" id="1722">
        <attvalues>
          <attvalue for="5" value=" In GEC, various methods have been proposed from a wide range of perspectives, including correction performance~\cite{grundkiewicz-junczys-dowmunt-2019-minimally,chollampatt-etal-2019-cross,omelianchuk-etal-2020-gector,kaneko-etal-2020-encoder,qorib-etal-2022-frustratingly}, controlling~\cite{hotate-etal-2019-controlling,yang2022controllable,loem2023exploring}, diversity~\cite{xie-etal-2018-noising,hotate-etal-2020-generating,han2021diversity}, and efficiency~\cite{malmi-etal-2019-encode,chen-etal-2020-improving-efficiency}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2010.03260" id="1723">
        <attvalues>
          <attvalue for="5" value=" In GEC, various methods have been proposed from a wide range of perspectives, including correction performance~\cite{grundkiewicz-junczys-dowmunt-2019-minimally,chollampatt-etal-2019-cross,omelianchuk-etal-2020-gector,kaneko-etal-2020-encoder,qorib-etal-2022-frustratingly}, controlling~\cite{hotate-etal-2019-controlling,yang2022controllable,loem2023exploring}, diversity~\cite{xie-etal-2018-noising,hotate-etal-2020-generating,han2021diversity}, and efficiency~\cite{malmi-etal-2019-encode,chen-etal-2020-improving-efficiency}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2203.07085" id="1724">
        <attvalues>
          <attvalue for="5" value=" \cite{kaneko-etal-2022-interpretability} introduced a method of presenting the retrieved examples as the basis for correction, in contrast to a method of retrieving data similar to the correction target from the training data set and using it for prediction." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2305.15676" id="1725">
        <attvalues>
          <attvalue for="5" value=" \cite{fei2023enhancing} proposed a method that presents the token positions that are the basis of errors and error types, and showed that they are useful for learners." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2005.14165" id="1726">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) such as ChatGPT~\cite{chatgpt} and GPT-3~\cite{brown2020language} have advanced language capabilities and can explain the inference reasons in natural language in various tasks~\cite{wei2022chain,wiegreffe-etal-2022-reframing,kaneko2023solving}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2112.08674" id="1727">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) such as ChatGPT~\cite{chatgpt} and GPT-3~\cite{brown2020language} have advanced language capabilities and can explain the inference reasons in natural language in various tasks~\cite{wei2022chain,wiegreffe-etal-2022-reframing,kaneko2023solving}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2305.11789" id="1728">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) such as ChatGPT~\cite{chatgpt} and GPT-3~\cite{brown2020language} have advanced language capabilities and can explain the inference reasons in natural language in various tasks~\cite{wei2022chain,wiegreffe-etal-2022-reframing,kaneko2023solving}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2305.11862" id="1729">
        <attvalues>
          <attvalue for="5" value=" LLMs are also effective in GEC, achieving state-of-the-art in both unsupervised~\cite{loem2023exploring} and supervised settings~\cite{kaneko2023reducing}." />
        </attvalues>
      </edge>
      <edge source="2309.11439" target="2304.01746" id="1730">
        <attvalues>
          <attvalue for="5" value=" \cite{fang2023chatgpt} showed that ChatGPT improves performance by using natural language to generate step-by-step error detection and correction processes for each span." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2001.08361" id="1731">
        <attvalues>
          <attvalue for="5" value=" A now dominant paradigm in few-shot learning involves pre-training a large language model (PLM) on unsupervised language modelling objectives, combined with supervised fine-tuning \cite{kaplan2020scaling, wei2022emergent}." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2110.08207" id="1732">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning on a variety of classification tasks improves generalization to new unseen tasks even further \cite{sanh2022multitask, wei2022emergent, chung2022scaling}." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2204.07937" id="1733">
        <attvalues>
          <attvalue for="5" value=" Moreover, by applying AuT-Few to a small upstream model (BART0~\cite{Lin2022UnsupervisedCG}), we achieve competitive performance and efficiency to the current state-of-the-art prompt-free method, SetFit~\cite{tunstall2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2205.05638" id="1734">
        <attvalues>
          <attvalue for="5" value=" AuT-Few outperforms strong baselines, including T-Few \cite{liu2022few}, by $2." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2012.15723" id="1735">
        <attvalues>
          <attvalue for="5" value=" Most techniques target prompted masked language models (eg encoder-only models, that make predictions over continuous embeddings via its mask token \cite[][inter alia]{gao-etal-2021-making}." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2103.10385" id="1736">
        <attvalues>
          <attvalue for="5" value=" Automation methods for models with a discrete output space (eg a decoder over the vocabulary) are costly and limited to the automation of the task template, still relying on handcrafted descriptions of labels \cite{liu2021gpt, zhou2023large}." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2109.01247" id="1737">
        <attvalues>
          <attvalue for="5" value=" These insights confirm observations by \cite{webson-pavlick-2022-prompt} in a broader context and they motivate a simple few-shot learning automation method for upstream models, named AuT-Few." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2209.11055" id="1738">
        <attvalues>
          <attvalue for="5" value=" Moreover, by applying AuT-Few to a small upstream model (BART0~\cite{Lin2022UnsupervisedCG}), we achieve competitive performance and efficiency to the current state-of-the-art prompt-free method, SetFit~\cite{tunstall2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2305.12576" target="2109.14076" id="1739">
        <attvalues>
          <attvalue for="5" value=" Furthermore, AuT-Few achieves the best average rank across datasets on the few-shot RAFT benchmark~\cite{alex2021raft}." />
        </attvalues>
      </edge>
      <edge source="2406.15708" target="2204.02311" id="1740">
        <attvalues>
          <attvalue for="5" value=" Significant advancements in large language models (LLMs) have revolutionized various natural language processing tasks \cite{chowdhery2023palm, anil2023palm, achiam2023gpt, team2023gemini}." />
        </attvalues>
      </edge>
      <edge source="2406.15708" target="2107.13586" id="1741">
        <attvalues>
          <attvalue for="5" value=" One notable aspect of LLMs, however, is their sensitivity to the input &quot;prompts,&quot; which has given rise to the burgeoning field of prompt engineering \cite{liu2023pre, sahoo2024systematic}." />
        </attvalues>
      </edge>
      <edge source="2406.15708" target="2211.01910" id="1742">
        <attvalues>
          <attvalue for="5" value=" Indeed, as we elaborate in \S\ref{sec:preliminaries}, EO approaches are often based on simple, handcrafted templates without explicit instruction optimization~\cite{khattab2023dspy, wan2023better}, while IO methods seldom optimize exemplars and often rely on random validation set samples \cite{pryzant2023automatic}, require additional fixed exemplars on top of the validation set \cite{zhou2022large, guo2023connecting}, or consider the ``zero-shot'' setup with no exemplars at all~\cite{wang2023promptagent}." />
        </attvalues>
      </edge>
      <edge source="2406.15708" target="2310.16427" id="1743">
        <attvalues>
          <attvalue for="5" value=" Indeed, as we elaborate in \S\ref{sec:preliminaries}, EO approaches are often based on simple, handcrafted templates without explicit instruction optimization~\cite{khattab2023dspy, wan2023better}, while IO methods seldom optimize exemplars and often rely on random validation set samples \cite{pryzant2023automatic}, require additional fixed exemplars on top of the validation set \cite{zhou2022large, guo2023connecting}, or consider the ``zero-shot'' setup with no exemplars at all~\cite{wang2023promptagent}." />
        </attvalues>
      </edge>
      <edge source="2406.15708" target="2309.16797" id="1744">
        <attvalues>
          <attvalue for="5" value=" While EO and IO approaches address the similar overarching problem, they have evolved somewhat independently, with a few exceptions \cite{fernando2023promptbreeder, wang2023mixture}." />
        </attvalues>
      </edge>
      <edge source="2406.15708" target="2310.03714" id="1745">
        <attvalues>
          <attvalue for="5" value=" Indeed, as we elaborate in \S\ref{sec:preliminaries}, EO approaches are often based on simple, handcrafted templates without explicit instruction optimization~\cite{khattab2023dspy, wan2023better}, while IO methods seldom optimize exemplars and often rely on random validation set samples \cite{pryzant2023automatic}, require additional fixed exemplars on top of the validation set \cite{zhou2022large, guo2023connecting}, or consider the ``zero-shot'' setup with no exemplars at all~\cite{wang2023promptagent}." />
        </attvalues>
      </edge>
      <edge source="2406.15708" target="2109.01652" id="1746">
        <attvalues>
          <attvalue for="5" value=" Whereas the lack of IO in EO methods is somewhat understandable as many EO approaches predate instruction finetuning \cite{wei2022finetuned} and, subsequently, instruction-following models that are sensitive to instructions, the inverse is much less so: concretely, almost all existing IO approaches already require a labeled dataset as the validation set, and are therefore, by definition, not ``zero-shot''." />
        </attvalues>
      </edge>
      <edge source="2406.15708" target="2401.00595" id="1747">
        <attvalues>
          <attvalue for="5" value=" However, given the common practical goal of and the interplay between EO and IO~\cite{mizrahi2023state}, we argue they should not be treated separately -- it is instead critical to understand their relative importance and combined impact, and, where necessary, optimize them jointly for the best performance-cost balance." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="1901.00596" id="1751">
        <attvalues>
          <attvalue for="5" value=" In recent years, Graph Neural Networks (GNNs)~\cite{wu2020comprehensive} have emerged as a powerful tool for a variety of tasks, including node classification~\cite{GIN} and link prediction~\cite{zhang2018link}." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="1810.00826" id="1752">
        <attvalues>
          <attvalue for="5" value=" In recent years, Graph Neural Networks (GNNs)~\cite{wu2020comprehensive} have emerged as a powerful tool for a variety of tasks, including node classification~\cite{GIN} and link prediction~\cite{zhang2018link}." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="1802.09691" id="1753">
        <attvalues>
          <attvalue for="5" value=" In recent years, Graph Neural Networks (GNNs)~\cite{wu2020comprehensive} have emerged as a powerful tool for a variety of tasks, including node classification~\cite{GIN} and link prediction~\cite{zhang2018link}." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="1911.06455" id="1754">
        <attvalues>
          <attvalue for="5" value=" Additionally, Graph Transformers~\cite{GTN, GFormer} employ self-attention and positional encoding to capture global signals among the graph, further improving the expressiveness of GNNs." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2306.08385" id="1755">
        <attvalues>
          <attvalue for="5" value=" To address scalability challenges in large graphs, methods such as Nodeformer~\cite{NodeFormer} and DIFFormer~\cite{DIFFormer} have been proposed." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2301.09474" id="1756">
        <attvalues>
          <attvalue for="5" value=" To address scalability challenges in large graphs, methods such as Nodeformer~\cite{NodeFormer} and DIFFormer~\cite{DIFFormer} have been proposed." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2106.07594" id="1757">
        <attvalues>
          <attvalue for="5" value=" For example, data sparsity remains a significant issue, particularly in scenarios where the graph structure is incomplete or noisy~\cite{you2021graph}." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2002.06157" id="1758">
        <attvalues>
          <attvalue for="5" value=" Moreover, the generalization ability of GNNs to new graphs or unseen nodes remains an open research question, with recent works highlighting the need for more robust and adaptive models~\cite{garg2020generalization, OpenGraph, zhao2024graphany}." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2303.18223" id="1759">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs)~\cite{zhao2023survey}, which show great generalization abilities for unseen tasks~\cite{BERT, T5, wang2023far}, have emerged as powerful tools in various research fields, including natural language processing~\cite{achiam2023gpt}, computer vision~\cite{liu2024visual, liu2024improved}, and information retrieval~\cite{zhu2023collaborative, hou2024large, lin2024data}." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2303.08774" id="1761">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs)~\cite{zhao2023survey}, which show great generalization abilities for unseen tasks~\cite{BERT, T5, wang2023far}, have emerged as powerful tools in various research fields, including natural language processing~\cite{achiam2023gpt}, computer vision~\cite{liu2024visual, liu2024improved}, and information retrieval~\cite{zhu2023collaborative, hou2024large, lin2024data}." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2311.01343" id="1763">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs)~\cite{zhao2023survey}, which show great generalization abilities for unseen tasks~\cite{BERT, T5, wang2023far}, have emerged as powerful tools in various research fields, including natural language processing~\cite{achiam2023gpt}, computer vision~\cite{liu2024visual, liu2024improved}, and information retrieval~\cite{zhu2023collaborative, hou2024large, lin2024data}." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2305.10037" id="1764">
        <attvalues>
          <attvalue for="5" value=" For instance, approaches such as InstructGLM~\cite{InstructGLM} and NLGraph~\cite{NLGraph} have designed specialized prompts that allow LLMs to reason over graph data and generate accurate responses." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2310.13023" id="1765">
        <attvalues>
          <attvalue for="5" value=" For example, GraphGPT~\cite{GraphGPT} and GraphLLM~\cite{GraphLLM} use GNNs to encode graph data into tokens, which are then fed into the LLMs for further processing." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2310.05845" id="1766">
        <attvalues>
          <attvalue for="5" value=" For example, GraphGPT~\cite{GraphGPT} and GraphLLM~\cite{GraphLLM} use GNNs to encode graph data into tokens, which are then fed into the LLMs for further processing." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2312.02783" id="1767">
        <attvalues>
          <attvalue for="5" value=" Unlike prior surveys that categorize studies based on the role of LLMs~\cite{jin2023large, li2023survey} or focus primarily on integrating LLMs with knowledge graphs~\cite{pan2024unifying}, our work highlights the model framework design, particularly the inference and training processes, to distinguish between existing taxonomies." />
        </attvalues>
      </edge>
      <edge source="2405.08011" target="2306.08302" id="1768">
        <attvalues>
          <attvalue for="5" value=" Unlike prior surveys that categorize studies based on the role of LLMs~\cite{jin2023large, li2023survey} or focus primarily on integrating LLMs with knowledge graphs~\cite{pan2024unifying}, our work highlights the model framework design, particularly the inference and training processes, to distinguish between existing taxonomies." />
        </attvalues>
      </edge>
      <edge source="2408.02103" target="2005.14165" id="1769">
        <attvalues>
          <attvalue for="5" value=" Specifically, LLMs have shown proficiency in learning from a limited set of input-output examples (known as demonstrations \cite{brown2020language}), and effectively applying these learned mappings to new, unseen instances." />
        </attvalues>
      </edge>
      <edge source="2408.02103" target="2107.13586" id="1770">
        <attvalues>
          <attvalue for="5" value=" This novel few-shot learning paradigm, which avoids parameter updates, has become a popular and efficient method for utilizing LLMs \cite{liu2021pretrain, dong2023survey, liu2021makes}." />
        </attvalues>
      </edge>
      <edge source="2408.02103" target="2101.06804" id="1771">
        <attvalues>
          <attvalue for="5" value=" Previous studies have investigated which instances can serve as effective prompts for ICL \cite{liu2021makes, zhang2022active, li2023finding}." />
        </attvalues>
      </edge>
      <edge source="2408.02103" target="cs/9603104" id="1772">
        <attvalues>
          <attvalue for="5" value=" We investigate the selection of demonstrations from the perspective of Active Learning (AL) \cite{cohn1996active, settles2009active}." />
        </attvalues>
      </edge>
      <edge source="2408.02103" target="2305.14264" id="1773">
        <attvalues>
          <attvalue for="5" value=" \cite{margatina2023active} elucidates that high semantic similarity, low uncertainty, and high diversity comprise an effective and efficient annotation strategy." />
        </attvalues>
      </edge>
      <edge source="2408.02103" target="2212.04037" id="1774">
        <attvalues>
          <attvalue for="5" value=" Similarly, \cite{gonen2022demystifying} demonstrates that lower prompt perplexity is closely associated with better performance." />
        </attvalues>
      </edge>
      <edge source="2408.02103" target="2209.01975" id="1775">
        <attvalues>
          <attvalue for="5" value=" While \cite{su2022selective}'s Vote-k framework adopts a data-centric perspective (eg, selecting examples that balance diversity and representativeness), it neglects the assessment of uncertainty and the inter-relationship among context examples." />
        </attvalues>
      </edge>
      <edge source="2203.05008" target="2008.10491" id="1777">
        <attvalues>
          <attvalue for="5" value=" In order to improve rare-word performance, recent works have looked to leveraging text-only corpora as an additional source of rare-word data \cite{peyser2020improving,raju2019scalable,huang2021lookup}." />
        </attvalues>
      </edge>
      <edge source="2203.05008" target="1907.01677" id="1778">
        <attvalues>
          <attvalue for="5" value=" These logs can be very large \cite{raju2019scalable,meng2021minimum}, making it prohibitively expensive to make even a single epoch through the data, limiting rare-word exposure." />
        </attvalues>
      </edge>
      <edge source="2203.05008" target="2106.02302" id="1780">
        <attvalues>
          <attvalue for="5" value=" These logs can be very large \cite{raju2019scalable,meng2021minimum}, making it prohibitively expensive to make even a single epoch through the data, limiting rare-word exposure." />
        </attvalues>
      </edge>
      <edge source="2203.05008" target="2107.06499" id="1781">
        <attvalues>
          <attvalue for="5" value=" First, we show that n\&quot;aive deduplication \cite{lee2021deduplicating} does not improve performance because it is too aggressive." />
        </attvalues>
      </edge>
      <edge source="2210.03162" target="2005.14165" id="1784">
        <attvalues>
          <attvalue for="5" value=" As an independent contribution, we explore the idea of simply using conditioning text to construct such experts by leveraging the few-shot modeling abilities of LMs \cite{radford2019language,brown2020language}: given a few examples of text containing a pattern of interest, language models are capable of ``analyzing'' such examples and assign high probability to subsequent text exhibiting the same pattern." />
        </attvalues>
      </edge>
      <edge source="2210.03162" target="2104.08691" id="1785">
        <attvalues>
          <attvalue for="5" value=" This paper explores prompt compression: the idea that the text $x_h$ used to condition a LM can be approximately represented by a much smaller set of carefully chosen weights, using the framework of soft prompts \cite{lestersoftprompt}." />
        </attvalues>
      </edge>
      <edge source="2210.03162" target="1909.05858" id="1786">
        <attvalues>
          <attvalue for="5" value=" Efforts to reduce toxicity and bias generally follow one of two strategies: the first is to train or fine-tune LMs on carefully curated data, either tagging or labelling it in special ways \cite{salesForceCTRL,lu2022quark} or using data known to be ``clean''." />
        </attvalues>
      </edge>
      <edge source="2210.03162" target="2205.13636" id="1787">
        <attvalues>
          <attvalue for="5" value=" Efforts to reduce toxicity and bias generally follow one of two strategies: the first is to train or fine-tune LMs on carefully curated data, either tagging or labelling it in special ways \cite{salesForceCTRL,lu2022quark} or using data known to be ``clean''." />
        </attvalues>
      </edge>
      <edge source="2210.03162" target="2105.03023" id="1789">
        <attvalues>
          <attvalue for="5" value=" The second is to &quot;steer&quot; the generation of token probabilities away from toxic generations \cite{krause2020gedi,liu2021dexperts}, and towards text with known, desirable properties." />
        </attvalues>
      </edge>
      <edge source="2210.03162" target="2102.07350" id="1790">
        <attvalues>
          <attvalue for="5" value=" We term this technique contrastive contexts, and note that it reduces the problem of creating experts to one of prompt engineering \cite{reynolds2021prompt}." />
        </attvalues>
      </edge>
      <edge source="2212.01907" target="2207.00099" id="1792">
        <attvalues>
          <attvalue for="5" value=" \cite{jagielskiforget} Complementary to these works, we investigate how model size affects the effectiveness at WMS via prompting." />
        </attvalues>
      </edge>
      <edge source="2212.01907" target="2202.05520" id="1793">
        <attvalues>
          <attvalue for="5" value=" \begin{enumerate} \item Privacy: As shown by \cite{brownprivacy}, the context of a sentence may change and thus also its secret, which is the word meant to be private." />
        </attvalues>
      </edge>
      <edge source="2212.01907" target="2205.01068" id="1794">
        <attvalues>
          <attvalue for="5" value=" The models are based on the OPT \cite{OPT} and GPT-3 \cite{GPT3} family of autoregressive (decoder-only) transformer-based models." />
        </attvalues>
      </edge>
      <edge source="2212.01907" target="2005.14165" id="1795">
        <attvalues>
          <attvalue for="5" value=" The models are based on the OPT \cite{OPT} and GPT-3 \cite{GPT3} family of autoregressive (decoder-only) transformer-based models." />
        </attvalues>
      </edge>
      <edge source="2408.09895" target="2001.08361" id="1796">
        <attvalues>
          <attvalue for="5" value=" However, they have limited accuracy and generality across different model structures (eg, dense or sparse) and shapes (eg, wide or deep), which may have substantial impacts on the final performance~\cite{kaplan2020scaling}." />
        </attvalues>
      </edge>
      <edge source="2408.09895" target="2203.15556" id="1797">
        <attvalues>
          <attvalue for="5" value=" To predict the performance of LLM before carrying out the whole experiment, researchers have proposed many variants of scaling laws~\cite{hoffmann2022training,krajewski2024scaling,sardana2023beyond,du2024understanding} to characterize the ability of LLM in different settings." />
        </attvalues>
      </edge>
      <edge source="2408.09895" target="2401.00448" id="1800">
        <attvalues>
          <attvalue for="5" value=" To predict the performance of LLM before carrying out the whole experiment, researchers have proposed many variants of scaling laws~\cite{hoffmann2022training,krajewski2024scaling,sardana2023beyond,du2024understanding} to characterize the ability of LLM in different settings." />
        </attvalues>
      </edge>
      <edge source="2408.09895" target="2305.14947" id="1802">
        <attvalues>
          <attvalue for="5" value=" Several recent works on LLM performance prediction~\cite{ye2023predictable,owen2024predictable} are based on model sizes and training data." />
        </attvalues>
      </edge>
      <edge source="2408.09895" target="2401.04757" id="1803">
        <attvalues>
          <attvalue for="5" value=" Several recent works on LLM performance prediction~\cite{ye2023predictable,owen2024predictable} are based on model sizes and training data." />
        </attvalues>
      </edge>
      <edge source="2408.09895" target="2405.18710" id="1804">
        <attvalues>
          <attvalue for="5" value=" Moreover, the precision and stability of the computing infrastructures are not taken into account, which usually damages the quality of the model~\cite{lee2024fp8}." />
        </attvalues>
      </edge>
      <edge source="2408.09895" target="2009.03300" id="1805">
        <attvalues>
          <attvalue for="5" value=" We discover an empirical equation to predict the MMLU~\cite{hendrycks2020measuring} metric of an LLM, which is a widely used measurement with good relevance to performance in downstream tasks~\cite{dubois2024length}." />
        </attvalues>
      </edge>
      <edge source="2408.09895" target="2404.04475" id="1806">
        <attvalues>
          <attvalue for="5" value=" We discover an empirical equation to predict the MMLU~\cite{hendrycks2020measuring} metric of an LLM, which is a widely used measurement with good relevance to performance in downstream tasks~\cite{dubois2024length}." />
        </attvalues>
      </edge>
      <edge source="2111.02114" target="2001.08361" id="1809">
        <attvalues>
          <attvalue for="5" value=" When increasing model and compute budget scale in addition, scaling laws suggest further increase in generalization and transfer performance if not bottlenecked by the data scale \cite{kaplan, kaplan2, Kolesnikov2020, zhai2021scaling}." />
        </attvalues>
      </edge>
      <edge source="2111.02114" target="2005.14165" id="1810">
        <attvalues>
          <attvalue for="5" value=" There is a plethora of recent works that have built massive datasets in order to optimally scale up various models \cite{gpt3, clip, dalle, align}." />
        </attvalues>
      </edge>
      <edge source="2111.02114" target="2101.00027" id="1811">
        <attvalues>
          <attvalue for="5" value=" recently released The Pile, an openly-available 800GB text dataset~\cite{pile}, in an attempt to loosely mimic the dataset used for GPT-3." />
        </attvalues>
      </edge>
      <edge source="2405.13015" target="1602.01059" id="1812">
        <attvalues>
          <attvalue for="5" value=" Starting with Dung's seminal work \cite{DBLP:journals/ai/Dung95}, many researchers have considered abstract argumentation frameworks, composed of a set of arguments and a binary attack relation between them, and created many semantics for tasks such as computing accepted sets of arguments \cite{DBLP:journals/ker/BaroniCG11, DBLP:conf/comma/Caminada06} or rank arguments \cite{DBLP:conf/sum/AmgoudB13a,DBLP:conf/aaai/BonzonDKM16,DBLP:conf/comma/YunVCB18}." />
        </attvalues>
      </edge>
      <edge source="2008.08439" target="1310.4546" id="1813">
        <attvalues>
          <attvalue for="5" value=" Language models, such as word embeddings~\cite{mikolov2013} create vector representations for the words that are able to capture syntactic and semantic relationships." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2005.14165" id="1815">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2303.08774" id="1816">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2307.09288" id="1818">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2310.06825" id="1819">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2401.04088" id="1820">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2403.09606" id="1822">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2403.05789" id="1823">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="1903.09722" id="1824">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="1905.03197" id="1825">
        <attvalues>
          <attvalue for="5" value=" Towards this goal, a series of LLMs such as GPTs~[\cite{brown2020language,achiam2023gpt}], LLaMAs~[\cite{touvron2023llama1,touvron2023llama,llama3modelcard}], and Mistrals~[\cite{jiang2023mistral,jiang2024mixtral}] have delivered high-level text understanding and generation capabilities via utilizing vast amount of high-quality web and human-annotated datasets for pre-training and preference alignment~[\cite{liu2023mmc,liu2024large,sun2024itd,edunov2019pre,dong2019unified}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2109.01652" id="1826">
        <attvalues>
          <attvalue for="5" value=" During preference alignment, instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2110.08207" id="1827">
        <attvalues>
          <attvalue for="5" value=" During preference alignment, instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2308.10792" id="1828">
        <attvalues>
          <attvalue for="5" value=" During preference alignment, instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2304.03277" id="1829">
        <attvalues>
          <attvalue for="5" value=" During preference alignment, instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2301.13688" id="1830">
        <attvalues>
          <attvalue for="5" value=" During preference alignment, instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2306.17194" id="1831">
        <attvalues>
          <attvalue for="5" value=" During preference alignment, instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2302.03202" id="1832">
        <attvalues>
          <attvalue for="5" value=" During preference alignment, instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2402.05119" id="1833">
        <attvalues>
          <attvalue for="5" value=" During preference alignment, instruction tuning plays an important role in refining {the } pre-trained LLMs to provide accurate, pertinent, and harmless responses on a collection of downstream tasks~[\cite{wei2021finetuned,sanh2021multitask,zhang2023instruction,peng2023instruction,longpre2023flan,shu2023exploitability,jang2023exploring,ghosh2024closer,kung2023models}]." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2305.11206" id="1836">
        <attvalues>
          <attvalue for="5" value=" For efficient and effective instruction tuning, existing studies~[\cite{ouyang2022training,taori2023alpaca,zhou2024lima,xia2024less}] have noticed that improving {the } quality of instruction tuning data (eg, formulation of well-defined and complete contexts), rather than simply piling up instructions without analysis (eg, exhaustive collection of open datasets), is of prioritized concerns." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2402.04333" id="1837">
        <attvalues>
          <attvalue for="5" value=" For efficient and effective instruction tuning, existing studies~[\cite{ouyang2022training,taori2023alpaca,zhou2024lima,xia2024less}] have noticed that improving {the } quality of instruction tuning data (eg, formulation of well-defined and complete contexts), rather than simply piling up instructions without analysis (eg, exhaustive collection of open datasets), is of prioritized concerns." />
        </attvalues>
      </edge>
      <edge source="2408.02085" target="2402.16827" id="1838">
        <attvalues>
          <attvalue for="5" value=" As revealed from the probabilistic view~[\cite{john1975d,murphy2012machine,albalak2024survey}], the statistical patterns inherent in datasets {determine } the modeling performance." />
        </attvalues>
      </edge>
      <edge source="2307.07889" target="2006.12719" id="1839">
        <attvalues>
          <attvalue for="5" value=" A large body of existing work use evaluation methods designed for particular tasks and attributes \cite{mehri-eskenazi-2020-unsupervised, rei-etal-2020-comet, manakul2023selfcheckgpt}, for example, measuring the consistency of summaries \cite{wang-etal-2020-asking, manakul2023mqag}." />
        </attvalues>
      </edge>
      <edge source="2307.07889" target="2004.04228" id="1840">
        <attvalues>
          <attvalue for="5" value=" A large body of existing work use evaluation methods designed for particular tasks and attributes \cite{mehri-eskenazi-2020-unsupervised, rei-etal-2020-comet, manakul2023selfcheckgpt}, for example, measuring the consistency of summaries \cite{wang-etal-2020-asking, manakul2023mqag}." />
        </attvalues>
      </edge>
      <edge source="2307.07889" target="2303.04048" id="1842">
        <attvalues>
          <attvalue for="5" value=" This has led to general prompt-based assessment approaches, such as prompt-scoring where an LLM is probed to score outputs on a particular aspect \cite{wang2023chatgpt, kocmi2023large}." />
        </attvalues>
      </edge>
      <edge source="2310.12836" target="2005.14165" id="1843">
        <attvalues>
          <attvalue for="5" value=" Recent Language Models (LMs)~\cite{gpt3, PaLM, flan}, which have a large number of parameters and are further instruction-finetuned on massive datasets, have achieved remarkable successes on various language tasks." />
        </attvalues>
      </edge>
      <edge source="2310.12836" target="1809.02156" id="1844">
        <attvalues>
          <attvalue for="5" value=" However, while the generated answers from LMs look plausible and sound, they are often factually incorrect, which is a problem widely known as hallucination~\cite{Hallucination, llm/eval, llm/truthful}." />
        </attvalues>
      </edge>
      <edge source="2310.12836" target="2203.05115" id="1845">
        <attvalues>
          <attvalue for="5" value=" To mitigate hallucination of LMs, recent works have proposed to augment LMs with the knowledge retrieved from external knowledge sources (eg, Wikipedia and Wikidata)~\cite{internet-augment, Adaptive_Retrieval, KAPING}." />
        </attvalues>
      </edge>
      <edge source="2310.12836" target="2303.17651" id="1846">
        <attvalues>
          <attvalue for="5" value=" Moreover, some other works have proposed to check the factuality of generated texts and refine them by using the knowledge in LMs themselves or from the external knowledge sources~\cite{self-refine, RARR, FLARE, CRITIC, search-chain, InteR}." />
        </attvalues>
      </edge>
      <edge source="2310.12836" target="2302.12813" id="1847">
        <attvalues>
          <attvalue for="5" value=" Note that there exists a concurrent work~\cite{LLM-Augmenter} that proposes to check whether the generated answers from LMs are grounded in the knowledge provided to LMs, by using API calls to proprietary LLMs or a heuristic measure (F1)." />
        </attvalues>
      </edge>
      <edge source="2112.10553" target="1706.03762" id="1848">
        <attvalues>
          <attvalue for="5" value=" Large pretrained language models, based on transformers \cite{Vaswani2017} present the current state of the art in solving natural language processing tasks." />
        </attvalues>
      </edge>
      <edge source="2112.10553" target="2108.07258" id="1849">
        <attvalues>
          <attvalue for="5" value=" Because of this and their ability to quickly and successfully adapt to a wide variety of tasks, they have also been (somewhat impetuously) called foundation models \cite{bommasani2021opportunities,marcus2021aifoundation}." />
        </attvalues>
      </edge>
      <edge source="2112.10553" target="2005.14165" id="1850">
        <attvalues>
          <attvalue for="5" value=" Two examples of such models are GPT-3 \cite{Brown2020GPT3short}, and T5 \cite{raffel2020exploring}." />
        </attvalues>
      </edge>
      <edge source="2112.10553" target="1912.07076" id="1854">
        <attvalues>
          <attvalue for="5" value=" It has been shown \cite{virtanen2019multilingual} that monolingual models outperform massive multilingual models, like multilingual BERT (mBERT) \cite{Devlin2019} or XLM-RoBERTa (XLM-R) \cite{conneau2019unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2112.10553" target="1911.02116" id="1857">
        <attvalues>
          <attvalue for="5" value=" It has been shown \cite{virtanen2019multilingual} that monolingual models outperform massive multilingual models, like multilingual BERT (mBERT) \cite{Devlin2019} or XLM-RoBERTa (XLM-R) \cite{conneau2019unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2408.04905" target="2303.08774" id="1865">
        <attvalues>
          <attvalue for="5" value=" In the field of Natural Language Processing~(NLP), large language models~(LLMs) like GPT-4~\cite{achiam2023gpt}, Gemini~\cite{team2023gemini, reid2024gemini}, and Claude 3~\cite{anthropic2024claude3} have demonstrated near-human-level text generation capabilities." />
        </attvalues>
      </edge>
      <edge source="2408.04905" target="2312.11805" id="1866">
        <attvalues>
          <attvalue for="5" value=" In the field of Natural Language Processing~(NLP), large language models~(LLMs) like GPT-4~\cite{achiam2023gpt}, Gemini~\cite{team2023gemini, reid2024gemini}, and Claude 3~\cite{anthropic2024claude3} have demonstrated near-human-level text generation capabilities." />
        </attvalues>
      </edge>
      <edge source="2408.04905" target="2404.09894" id="1867">
        <attvalues>
          <attvalue for="5" value="~\cite{li2024glitch}." />
        </attvalues>
      </edge>
      <edge source="2305.15582" target="1711.06861" id="1868">
        <attvalues>
          <attvalue for="5" value=" Multi-style text transfer is a challenging task today with applications such as automatic domain-appropriate, style-conformant writing \cite{fu2018style} and AI-assisted stylistic language editing." />
        </attvalues>
      </edge>
      <edge source="2305.15582" target="1703.00955" id="1869">
        <attvalues>
          <attvalue for="5" value=" Text style transfer is challenging because it involves dealing with the aspects of style coupled with the textual content \cite{hu2017toward, shen2017style, lample2018multiple}." />
        </attvalues>
      </edge>
      <edge source="2305.15582" target="1707.01161" id="1870">
        <attvalues>
          <attvalue for="5" value=" This domain's other obstacles include the need for parallel corpus \cite{jhamtani2017shakespearizing} and quality training data." />
        </attvalues>
      </edge>
      <edge source="2305.15582" target="2110.15871" id="1871">
        <attvalues>
          <attvalue for="5" value=" People intentionally \cite{troiano2021theories} tune these styles in writing differently based on their mood, the person they are addressing, the content of the message, or the platform." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1708.02182" id="1881">
        <attvalues>
          <attvalue for="5" value=" The Long Short-Term Memory (LSTM) network \cite{lstm} was designed to model dependencies in sequential data, and has successfully been applied to language modeling in a series of works, including the AWD-LSTM \cite{awdlstm}." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1706.03762" id="1882">
        <attvalues>
          <attvalue for="5" value=" On the other hand, the development of the Transformer attention architecture \cite{transformer} inspired new Transformer-based language models, such as Transformer-XL \cite{transformerxl}, which achieved new state-of-the-art in language modeling benchmarks." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1901.02860" id="1883">
        <attvalues>
          <attvalue for="5" value=" On the other hand, the development of the Transformer attention architecture \cite{transformer} inspired new Transformer-based language models, such as Transformer-XL \cite{transformerxl}, which achieved new state-of-the-art in language modeling benchmarks." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1809.10853" id="1884">
        <attvalues>
          <attvalue for="5" value=" In addition, several works have developed architecture-agnostic enhancements such as adaptive embedding and adaptive softmax \cite{adaptive}, non-parametric cache \cite{cache}, Hebbian softmax \cite{lstmhebbiancache}, and dynamic evaluation \cite{dynamiceval}." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1612.04426" id="1885">
        <attvalues>
          <attvalue for="5" value=" In addition, several works have developed architecture-agnostic enhancements such as adaptive embedding and adaptive softmax \cite{adaptive}, non-parametric cache \cite{cache}, Hebbian softmax \cite{lstmhebbiancache}, and dynamic evaluation \cite{dynamiceval}." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1803.10049" id="1886">
        <attvalues>
          <attvalue for="5" value=" In addition, several works have developed architecture-agnostic enhancements such as adaptive embedding and adaptive softmax \cite{adaptive}, non-parametric cache \cite{cache}, Hebbian softmax \cite{lstmhebbiancache}, and dynamic evaluation \cite{dynamiceval}." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1906.09777" id="1889">
        <attvalues>
          <attvalue for="5" value=" \cite{tensorizedtransformer} applies Block-term Tensor Decomposition \cite{btd} to compress transformer-based language models." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1503.02531" id="1890">
        <attvalues>
          <attvalue for="5" value=" Architecture agnostic model compression techniques such as knowledge distillation \cite{distill}, network pruning \cite{deepcompression}, and trained quantization \cite{deepcompression} are commonly used techniques to either increase the predictivity of neural networks or decrease the model size." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1510.00149" id="1891">
        <attvalues>
          <attvalue for="5" value=" Architecture agnostic model compression techniques such as knowledge distillation \cite{distill}, network pruning \cite{deepcompression}, and trained quantization \cite{deepcompression} are commonly used techniques to either increase the predictivity of neural networks or decrease the model size." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1802.03494" id="1892">
        <attvalues>
          <attvalue for="5" value=" There are also recent efforts focusing on automatically designing efficient models \cite{amc, so2019evolved, apq} and designing specialized accelerators to process the compressed models \cite{scnn, sparch}." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="1708.04485" id="1893">
        <attvalues>
          <attvalue for="5" value=" There are also recent efforts focusing on automatically designing efficient models \cite{amc, so2019evolved, apq} and designing specialized accelerators to process the compressed models \cite{scnn, sparch}." />
        </attvalues>
      </edge>
      <edge source="2005.07877" target="2005.14187" id="1895">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, these techniques are mostly developed on convolutional neural networks and computer vision tasks, and only a handful of recent works like DistilBERT \cite{distilbert} and HAT \cite{hat} applied them to natural language tasks." />
        </attvalues>
      </edge>
      <edge source="2204.08110" target="2109.07348" id="1896">
        <attvalues>
          <attvalue for="5" value=" Recent work has claimed that monolingual pretrained models are also surprisingly good at transferring between languages, despite ostensibly having never seen the target language before \cite[inter alia]{gogoulou2021cross, li2021cross}." />
        </attvalues>
      </edge>
      <edge source="2302.05900" target="1609.07451" id="1897">
        <attvalues>
          <attvalue for="5" value=" A common practice was to convert AMR-to-Text task into an already studied problems such as Tree-to-Text \cite{flanigan-etal-2016-generation, lampouras-vlachos-2017-sheffield}, aligned text-to-text \cite{pourdamghani-etal-2016-generating}, Travel Sales Problems \cite{song-etal-2016-amr} or Grammatical Framework \cite{grammatical-framework-book-ranta}." />
        </attvalues>
      </edge>
      <edge source="2302.05900" target="1409.3215" id="1898">
        <attvalues>
          <attvalue for="5" value=" Recently, most methods are neural-centered with an encoder-decoder architecture \cite{sutskever-etal-2014-seq2seq} as a backbone \cite{konstas-etal-2017-neural, takase-etal-2016-neural, cao-clark-2019-factorising}." />
        </attvalues>
      </edge>
      <edge source="2302.05900" target="1704.08381" id="1899">
        <attvalues>
          <attvalue for="5" value=" Recently, most methods are neural-centered with an encoder-decoder architecture \cite{sutskever-etal-2014-seq2seq} as a backbone \cite{konstas-etal-2017-neural, takase-etal-2016-neural, cao-clark-2019-factorising}." />
        </attvalues>
      </edge>
      <edge source="2302.05900" target="2007.08426" id="1901">
        <attvalues>
          <attvalue for="5" value=" A limitation of those is the absence of pretraining, as demonstrated by \cite{ribeiro-etal-2021-investigating}." />
        </attvalues>
      </edge>
      <edge source="2302.05900" target="2103.09120" id="1902">
        <attvalues>
          <attvalue for="5" value=" To this end, \cite{ribeiro-etal-2021-structural} introduced StructAdapt for lightweight AMR-to-Text with structural adapters." />
        </attvalues>
      </edge>
      <edge source="2302.05900" target="2010.04903" id="1903">
        <attvalues>
          <attvalue for="5" value=" However, although studies have been made to probe position embeddings \cite{wang-chen-2020-position, wang-et-al-2021-APE-BERT,dufter-etal-2022-position}, their role on graph encoding has remained unanswered." />
        </attvalues>
      </edge>
      <edge source="2302.05900" target="2102.11090" id="1904">
        <attvalues>
          <attvalue for="5" value=" However, although studies have been made to probe position embeddings \cite{wang-chen-2020-position, wang-et-al-2021-APE-BERT,dufter-etal-2022-position}, their role on graph encoding has remained unanswered." />
        </attvalues>
      </edge>
      <edge source="2302.05900" target="2006.09242" id="1905">
        <attvalues>
          <attvalue for="5" value=" Our novelty is not in proposing a new method to encode graphs such as \cite{schmitt-etal-2021-modeling} but rather in revealing the interesting behaviours of RPE along with StructAdapt." />
        </attvalues>
      </edge>
      <edge source="2311.09204" target="2307.12966" id="1907">
        <attvalues>
          <attvalue for="5" value=" Evaluating the performance of natural language generation models has significant challenges~\cite{ouyang2022training}, particularly in terms of evaluation benchmarks and evaluation paradigms~\cite{wang2023aligning}." />
        </attvalues>
      </edge>
      <edge source="2311.09204" target="2007.12626" id="1908">
        <attvalues>
          <attvalue for="5" value=" Furthermore, studies such as \cite{fabbri2021summeval} have demonstrated that these automatic metrics often do not correlate well with human judgment." />
        </attvalues>
      </edge>
      <edge source="2311.09204" target="2004.04696" id="1909">
        <attvalues>
          <attvalue for="5" value=" Neural evaluators like BLEURT~\cite{sellam2020bleurt} and its variant SMART~\cite{amplayo2022smart} show improved alignment with human assessments in various generative tasks." />
        </attvalues>
      </edge>
      <edge source="2311.09204" target="2208.01030" id="1910">
        <attvalues>
          <attvalue for="5" value=" Neural evaluators like BLEURT~\cite{sellam2020bleurt} and its variant SMART~\cite{amplayo2022smart} show improved alignment with human assessments in various generative tasks." />
        </attvalues>
      </edge>
      <edge source="2311.09204" target="2302.04166" id="1911">
        <attvalues>
          <attvalue for="5" value=" Notably, studies by \cite{fu2023gptscore, wang2023chatgpt} have leveraged LLMs to rate candidate outputs based on their generation probability alone, eliminating the need for reference text comparisons." />
        </attvalues>
      </edge>
      <edge source="2311.09204" target="2210.07197" id="1912">
        <attvalues>
          <attvalue for="5" value=" Meta-evaluations indicate that these LLM-based evaluators reach a level of human correlation on par with medium-sized neural evaluators~\cite{zhong2022towards}." />
        </attvalues>
      </edge>
      <edge source="2310.14840" target="2001.08361" id="1913">
        <attvalues>
          <attvalue for="5" value=" The best we can do is to measure perplexity on a new sample from the same unknown distribution, and compare that perplexity to the perplexity we obtain with other model architectures or training regimes \cite{brown1992estimate}, or with the expected perplexity given a (compute-optimal) scaling law \cite{DBLP:journals/corr/abs-2001-08361,hoffmann2022empirical}." />
        </attvalues>
      </edge>
      <edge source="2310.14840" target="2101.00027" id="1914">
        <attvalues>
          <attvalue for="5" value=" To obtain the grammar, we use an automatically parsed sectionfrom the The Pile corpus \cite{pile}, and use the state-split framework \cite{petrov-etal-2006-learning} -- one of the most successful statistical parsing frameworks from before the rise of deep learning -- to obtain a statistical grammar with more than 2 million rules." />
        </attvalues>
      </edge>
      <edge source="2310.14840" target="cmp-lg/9411029" id="1915">
        <attvalues>
          <attvalue for="5" value=" One key contribution from this paper is a closed-form expression to efficiently compute masked token probabilities for PCFGs, complementing the classic closed form for causal language modelling \cite{DBLP:journals/coling/Stolcke95}." />
        </attvalues>
      </edge>
      <edge source="2311.18609" target="2201.11903" id="1916">
        <attvalues>
          <attvalue for="5" value=" By adding the chain of thought for an associated answer, \cite{wei2022chain} improves the ability of large language models to perform complex reasoning." />
        </attvalues>
      </edge>
      <edge source="2311.18609" target="2304.12244" id="1917">
        <attvalues>
          <attvalue for="5" value=" \cite{xu2023wizardlm} rewrites the prompts to upgrade the simple instruction to a more complex one and increase diversity for LLM training." />
        </attvalues>
      </edge>
      <edge source="2311.18609" target="2302.04761" id="1918">
        <attvalues>
          <attvalue for="5" value=" \cite{schick2023toolformer} trained the model to use different API calls and get the correct answers." />
        </attvalues>
      </edge>
      <edge source="2311.18609" target="2211.10435" id="1919">
        <attvalues>
          <attvalue for="5" value=" \cite{gao2023pal} do more, they incorporate LLM with a python interpreter that the LLM generates codes for the python interpreter and gets the answer from it, which improves the accuracy of the answer." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1211.3711" id="1920">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1303.5778" id="1921">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1801.00841" id="1922">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1811.06621" id="1923">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1910.12977" id="1924">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="2002.02562" id="1925">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="2010.10759" id="1926">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1508.04395" id="1927">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1712.01769" id="1929">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1805.03294" id="1930">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="2111.05948" id="1931">
        <attvalues>
          <attvalue for="5" value=" Modern end-to-end (E2E) automatic speech recognition (ASR) models, such as recurrent neural network transducers (eg, RNN-T)~\cite{Graves12,GravesMohamedHinton13,RaoSakPrabhavalkar17,HeSainathPrabhavalkarEtAl19,YehMahadeokarKalgaonkar19,ZhangLuSakEtAl20,ShiWangWuEtAl21} and attention-based encoder-decoder models~\cite{ChanJaitlyLeEtAl16,BahdanauChorowskiSerdyukEtAl16,KimHoriWatanabe17,ChiuSainathWeEtAl18,ZeyerIrieSchluterEtAl18}, have the ability to benefit from very large amounts of speech data~\cite{zheng22d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1712.01541" id="1932">
        <attvalues>
          <attvalue for="5" value=" Due mainly to this scalability, E2E models have enabled great strides in multilingual ASR research, which aims to create a single model that can recognize multiple languages at the same time~\cite{Li18multidialect,kannan19_interspeech,Hou2020LargeScaleEM, Pratap2020,li2021scaling,babu22_interspeech,yang2022learning,joshi21_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="1909.05330" id="1933">
        <attvalues>
          <attvalue for="5" value=" Due mainly to this scalability, E2E models have enabled great strides in multilingual ASR research, which aims to create a single model that can recognize multiple languages at the same time~\cite{Li18multidialect,kannan19_interspeech,Hou2020LargeScaleEM, Pratap2020,li2021scaling,babu22_interspeech,yang2022learning,joshi21_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="2111.09296" id="1935">
        <attvalues>
          <attvalue for="5" value=" Due mainly to this scalability, E2E models have enabled great strides in multilingual ASR research, which aims to create a single model that can recognize multiple languages at the same time~\cite{Li18multidialect,kannan19_interspeech,Hou2020LargeScaleEM, Pratap2020,li2021scaling,babu22_interspeech,yang2022learning,joshi21_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="2209.05735" id="1936">
        <attvalues>
          <attvalue for="5" value=" Due mainly to this scalability, E2E models have enabled great strides in multilingual ASR research, which aims to create a single model that can recognize multiple languages at the same time~\cite{Li18multidialect,kannan19_interspeech,Hou2020LargeScaleEM, Pratap2020,li2021scaling,babu22_interspeech,yang2022learning,joshi21_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2211.05756" target="2007.03001" id="1937">
        <attvalues>
          <attvalue for="5" value=" \cite{Pratap2020} built multilingual ASR on top of 50 languages with total 16,000 hours training dataset." />
        </attvalues>
      </edge>
      <edge source="2109.04949" target="cmp-lg/9607031" id="1957">
        <attvalues>
          <attvalue for="5" value=" DRT has played an important role in computational semantics from early work on the Verbmobil project \cite{BosGambaeckLieskeMoriPinkalWorm1996} to work by Johan Bos and others on the Groningen Meaning Bank () and the Parallel Meaning Bank ()." />
        </attvalues>
      </edge>
      <edge source="2109.04949" target="1003.4394" id="1958">
        <attvalues>
          <attvalue for="5" value=" An alternative strategy is to try to incorporate features from the classical canon in non-formal approaches \cite[for example,][]{CoeckeSadrzadehClark2010} or to combine aspects of non-formal and formal approaches in a single framework \cite[for example,][]{Larsson2013,ErkHerbelot2020}." />
        </attvalues>
      </edge>
      <edge source="2109.04949" target="2009.07936" id="1959">
        <attvalues>
          <attvalue for="5" value=" An alternative strategy is to try to incorporate features from the classical canon in non-formal approaches \cite[for example,][]{CoeckeSadrzadehClark2010} or to combine aspects of non-formal and formal approaches in a single framework \cite[for example,][]{Larsson2013,ErkHerbelot2020}." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2005.14165" id="1966">
        <attvalues>
          <attvalue for="5" value=" LLMs have accumulated a lot of information and may be able to answer directly when confronted with widely known facts \cite{gpt3,spark,emergent}." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2303.12712" id="1967">
        <attvalues>
          <attvalue for="5" value=" LLMs have accumulated a lot of information and may be able to answer directly when confronted with widely known facts \cite{gpt3,spark,emergent}." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2204.02311" id="1969">
        <attvalues>
          <attvalue for="5" value=" As scaling these models increases their ability to incorporate more and more knowledge \cite{gpt3,palm}, instead of relying on traditional search engines, Metzler et al." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2105.02274" id="1970">
        <attvalues>
          <attvalue for="5" value=" \cite{rethinkingsearch} suggest using LLM as a unified knowledge base able to perform question answering as well as document retrieval." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2002.08909" id="1971">
        <attvalues>
          <attvalue for="5" value=" In question answering, retrieval augmented methods such as REALM \cite{realm}, RAG \cite{rag}, or RETRO \cite{retro,orqa}, were proposed to reduce LLM's hallucinations." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2005.11401" id="1972">
        <attvalues>
          <attvalue for="5" value=" In question answering, retrieval augmented methods such as REALM \cite{realm}, RAG \cite{rag}, or RETRO \cite{retro,orqa}, were proposed to reduce LLM's hallucinations." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="1906.00300" id="1974">
        <attvalues>
          <attvalue for="5" value=" These architectures are effective as they both improve factualness and reduce hallucinations for specific knowledge-intensive tasks such as Open-domain Question Answering \cite{orqa}." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2201.08239" id="1975">
        <attvalues>
          <attvalue for="5" value=" In a second line of work, models, such as LaMDA, BlenderBot, WebGPT, Toolformer \cite{thoppilan2022lamda,nakano2022webgpt,shuster2022blenderbot,schick2023toolformer} are specifically trained to generate a query and rely on a search engine when confronted with questions." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2208.03188" id="1977">
        <attvalues>
          <attvalue for="5" value=" In a second line of work, models, such as LaMDA, BlenderBot, WebGPT, Toolformer \cite{thoppilan2022lamda,nakano2022webgpt,shuster2022blenderbot,schick2023toolformer} are specifically trained to generate a query and rely on a search engine when confronted with questions." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="2302.04761" id="1978">
        <attvalues>
          <attvalue for="5" value=" In a second line of work, models, such as LaMDA, BlenderBot, WebGPT, Toolformer \cite{thoppilan2022lamda,nakano2022webgpt,shuster2022blenderbot,schick2023toolformer} are specifically trained to generate a query and rely on a search engine when confronted with questions." />
        </attvalues>
      </edge>
      <edge source="2401.01780" target="1705.03551" id="1979">
        <attvalues>
          <attvalue for="5" value=" We focus on closed-book question-answering (CBQA) tasks and carried out on two datasets (Natural Questions (NQ) \cite{kwiatkowski-etal-2019-natural} and TriviaQA (TQA) \cite{joshi-etal-2017-triviaqa})." />
        </attvalues>
      </edge>
      <edge source="2006.09075" target="1611.01734" id="1980">
        <attvalues>
          <attvalue for="5" value=" Natural Language Processing (NLP) algorithms are constantly improving, gradually approaching human level performance \cite{Dozat:17,Edunov:18,Radford:18}." />
        </attvalues>
      </edge>
      <edge source="2006.09075" target="1808.09381" id="1981">
        <attvalues>
          <attvalue for="5" value=" Natural Language Processing (NLP) algorithms are constantly improving, gradually approaching human level performance \cite{Dozat:17,Edunov:18,Radford:18}." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2005.14165" id="1984">
        <attvalues>
          <attvalue for="5" value=" Current Large Language Models (LLMs), such as GPT-3, GPT-4, PaLM, and LLaMA~\cite{brown2020language, chowdhery2023palm, touvron2023llama}, have demonstrated remarkable capabilities in in-context learning, also known as prompting, across a broad spectrum of language understanding and generation tasks~\cite{zhao2023survey, zhang2023instruction, ziyu-etal-2023-lens}." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2303.18223" id="1985">
        <attvalues>
          <attvalue for="5" value=" Current Large Language Models (LLMs), such as GPT-3, GPT-4, PaLM, and LLaMA~\cite{brown2020language, chowdhery2023palm, touvron2023llama}, have demonstrated remarkable capabilities in in-context learning, also known as prompting, across a broad spectrum of language understanding and generation tasks~\cite{zhao2023survey, zhang2023instruction, ziyu-etal-2023-lens}." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2307.09288" id="1986">
        <attvalues>
          <attvalue for="5" value=" For instance, LLaMA2's pretraining corpus comprises over 89\% English content~\cite{touvron2023llama2}." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2304.05613" id="1987">
        <attvalues>
          <attvalue for="5" value=" Yet, these English-centric LLMs~ still exhibit effective performance in multilingual evaluations~\cite{lai-etal-2023-chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2303.12528" id="1988">
        <attvalues>
          <attvalue for="5" value=" In a multilingual prompting scenario designed for zero-shot transfer with LLMs, the model executes tasks by directly generating outputs based on a task description and/or a few examples provided in a pivot language (typically English), along with input in a different target language~\cite{ahuja-etal-2023-mega}." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2306.06688" id="1989">
        <attvalues>
          <attvalue for="5" value=" However, the extent and nature of their cross-lingual capabilities remain underexplored~\cite{ye2023language}." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2310.06474" id="1990">
        <attvalues>
          <attvalue for="5" value=" This raises a critical question: Does the multilinguality of these models stem from a deep, generalizable multilingual linguistic understanding, or merely from the superficial alignment of lexical patterns across languages? Given the demonstrated proficiency of English-centric LLMs in multilingual tasks that demand profound language understanding~\cite{deng2023multilingual, wang2023all}, we hypothesize that these models harbor substantial multilingual knowledge." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2305.14857" id="1991">
        <attvalues>
          <attvalue for="5" value=" Additionally, text-to-text prompting methods~\cite{asai2023buffet}, which rely on a predefined output template, face challenges in maintaining control over the output format." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2211.07830" id="1992">
        <attvalues>
          <attvalue for="5" value=" In response to these challenges, a decent iterative prompting strategy for structured prediction has been introduced, addressing the aforementioned limitations~\cite{blevins-etal-2023-prompting}." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2401.16589" id="1993">
        <attvalues>
          <attvalue for="5" value=" This strategy is inspired by the ToPro method~\cite{ma2024topro}, a novel prompt-based fine-tuning approach for sequence labeling tasks." />
        </attvalues>
      </edge>
      <edge source="2402.18397" target="2004.10643" id="1994">
        <attvalues>
          <attvalue for="5" value=" We evaluate our approach on the Universal Dependency (UD) part-of-speech (POS) tagging dataset~\cite{nivre-etal-2020-universal} covering 38 languages with 3 English-centric LLMs and 2 multilingual LLMs." />
        </attvalues>
      </edge>
      <edge source="2304.14999" target="2110.07577" id="2005">
        <attvalues>
          <attvalue for="5" value=" Parameter-efficient fine-tuning techniques aim to solve this problem by modifying a very small portion of weights relative to the full model size while keeping the rest of the model frozen \cite{mao2021unipelt}." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="1912.10211" id="2006">
        <attvalues>
          <attvalue for="5" value=" Pretrained Audio Neural Networks (PANNs) ~\cite{kong2020panns} have played a significant role in this shift by demonstrating their versatility across various tasks and outperforming many advanced systems via fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2305.10790" id="2007">
        <attvalues>
          <attvalue for="5" value=" One approach to linking text and audio is through generating response text given a combination of an audio prompt and a text prompt~\cite{gardner2023llark,gong2023listen,deshmukh2023pengi}." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2305.11834" id="2008">
        <attvalues>
          <attvalue for="5" value=" For instance, Pengi~\cite{deshmukh2023pengi} converts audio classification, retrieval, captioning, and audio question answering into a text generation task using audio and task-specific text prompts." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2311.07919" id="2009">
        <attvalues>
          <attvalue for="5" value=" Similarly, Qwen-Audio~\cite{chu2023qwen} addresses a variety of audio tasks through text generation but distinguishes itself by using text prompts consisting of hierarchical tag sequences inspired by Whisper~\cite{radford2023robust}." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2301.12661" id="2011">
        <attvalues>
          <attvalue for="5" value=" Moreover, the learned audio-text representations from contrastive models offer the flexibility to use one modality during training and the other at inference, which can be applied in text-to-audio generation~\cite{huang2023make,liu2023audioldm} and language-guided source separation~\cite{liu2023separate,dong2022clipsep}." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2301.12503" id="2012">
        <attvalues>
          <attvalue for="5" value=" Moreover, the learned audio-text representations from contrastive models offer the flexibility to use one modality during training and the other at inference, which can be applied in text-to-audio generation~\cite{huang2023make,liu2023audioldm} and language-guided source separation~\cite{liu2023separate,dong2022clipsep}." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2308.05037" id="2013">
        <attvalues>
          <attvalue for="5" value=" Moreover, the learned audio-text representations from contrastive models offer the flexibility to use one modality during training and the other at inference, which can be applied in text-to-audio generation~\cite{huang2023make,liu2023audioldm} and language-guided source separation~\cite{liu2023separate,dong2022clipsep}." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="1910.09387" id="2015">
        <attvalues>
          <attvalue for="5" value=" For context, while the largest public audio-text datasets contain less than 100,000 pairs~\cite{kim2019audiocaps,drossos2020clotho}, image-text models like CLIP~\cite{radford2021learning} and SigLip~\cite{zhai2023sigmoid} utilize 400 million and 3." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2303.15343" id="2017">
        <attvalues>
          <attvalue for="5" value=" For context, while the largest public audio-text datasets contain less than 100,000 pairs~\cite{kim2019audiocaps,drossos2020clotho}, image-text models like CLIP~\cite{radford2021learning} and SigLip~\cite{zhai2023sigmoid} utilize 400 million and 3." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2208.12415" id="2018">
        <attvalues>
          <attvalue for="5" value="~\cite{huang2022mulan} collected approximately 44 million 30-second music clips, applying a pretrained classifier and rule-based filtering to clean associated metadata." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2211.06687" id="2019">
        <attvalues>
          <attvalue for="5" value=" For instance, LAION-CLAP~\cite{laionclap2023} investigates different choices of audio/text encoders, demonstrating superior performance with the hierarchical token semantic audio transformer (HTSAT)\cite{chen2022hts} for audio encoding and the Robustly optimized BERT approach (RoBERTa)\cite{liu2019roberta} for text encoding." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2303.17395" id="2020">
        <attvalues>
          <attvalue for="5" value="~\cite{mei2023wavcaps} propose a multi-stage data filtering pipeline and utilize ChatGPT for cleaning text descriptions." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2202.00874" id="2021">
        <attvalues>
          <attvalue for="5" value=" For instance, LAION-CLAP~\cite{laionclap2023} investigates different choices of audio/text encoders, demonstrating superior performance with the hierarchical token semantic audio transformer (HTSAT)\cite{chen2022hts} for audio encoding and the Robustly optimized BERT approach (RoBERTa)\cite{liu2019roberta} for text encoding." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="1907.11692" id="2022">
        <attvalues>
          <attvalue for="5" value=" For instance, LAION-CLAP~\cite{laionclap2023} investigates different choices of audio/text encoders, demonstrating superior performance with the hierarchical token semantic audio transformer (HTSAT)\cite{chen2022hts} for audio encoding and the Robustly optimized BERT approach (RoBERTa)\cite{liu2019roberta} for text encoding." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2212.00794" id="2024">
        <attvalues>
          <attvalue for="5" value=" In concurrent work, fast language-audio pretraining (FLAP)~\cite{yeh2023flap}, inspired by fast language-image pretraining (FLIP)~\cite{li2023scaling}, proposes masking and removing a significant portion of spectrogram patches." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2111.06377" id="2025">
        <attvalues>
          <attvalue for="5" value=" The first stage focuses on training spectrogram-based audio encoder using a masked autoencoder (MAE) objective~\cite{he2022masked,huang2022masked}, which learns representations through masking random patches from the input spectrogram and then reconstructing these masked patches." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2207.06405" id="2026">
        <attvalues>
          <attvalue for="5" value=" The first stage focuses on training spectrogram-based audio encoder using a masked autoencoder (MAE) objective~\cite{he2022masked,huang2022masked}, which learns representations through masking random patches from the input spectrogram and then reconstructing these masked patches." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2205.01917" id="2027">
        <attvalues>
          <attvalue for="5" value=" The integration of the auxillary captioning objective, inspired by contrastive captioner (CoCa)~\cite{yu2022coca} and bootstrapping language-image pre-training (BLIP)~\cite{li2022blip}, provides stronger supervision, encouraging the audio encoder to capture fine-grained patterns that closely match text descriptions." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2201.12086" id="2028">
        <attvalues>
          <attvalue for="5" value=" The integration of the auxillary captioning objective, inspired by contrastive captioner (CoCa)~\cite{yu2022coca} and bootstrapping language-image pre-training (BLIP)~\cite{li2022blip}, provides stronger supervision, encouraging the audio encoder to capture fine-grained patterns that closely match text descriptions." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="1806.00451" id="2029">
        <attvalues>
          <attvalue for="5" value="~\cite{recht2018cifar}." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2204.09634" id="2030">
        <attvalues>
          <attvalue for="5" value=" To provide a more comprehensive benchmark, we additionally evaluate on audio question answering (AQA)~\cite{lipping2022clotho}." />
        </attvalues>
      </edge>
      <edge source="2402.06986" target="2203.03022" id="2031">
        <attvalues>
          <attvalue for="5" value=" To evaluate the effectiveness of our audio encoder, we test on Holistic Evaluation of Audio Representations (HEAR)~\cite{turian2022hear}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="1906.02243" id="2032">
        <attvalues>
          <attvalue for="5" value=" However, direct fine-tuning is both time-consuming and computationally intensive \cite{strubell-etal-2019-energy}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2104.08691" id="2033">
        <attvalues>
          <attvalue for="5" value=" This problem will become worse as model sizes continue to grow, increasingly motivating more efficient fine-tuning \cite{lester-etal-2021-power, han2024parameterefficient} or alternative approaches \cite{hu2021lora} for enhancing or aligning LLM performance." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2106.09685" id="2034">
        <attvalues>
          <attvalue for="5" value=" This problem will become worse as model sizes continue to grow, increasingly motivating more efficient fine-tuning \cite{lester-etal-2021-power, han2024parameterefficient} or alternative approaches \cite{hu2021lora} for enhancing or aligning LLM performance." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="1409.3215" id="2036">
        <attvalues>
          <attvalue for="5" value=" Such ensembling methods, which aggregate models during beam search, have shown promise for improving translation quality in NMT settings \cite{sutskever2014sequence, firat2016zeroresource, stahlberg-etal-2018-simple}, but require the same vocabulary and tokenization." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2205.09273" id="2037">
        <attvalues>
          <attvalue for="5" value=" Twist decoding \cite{kasai2022twist} modifies beam search to bypass the shared vocabulary restriction, but its reliance on beam search reduces the inference speed." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2401.08565" id="2038">
        <attvalues>
          <attvalue for="5" value=" Other more recent approaches related to combining language models include proxy tuning \cite{liu2024tuning} and Composition to Augment Language Models (CALM) \cite{bansal2024llm}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2401.02412" id="2039">
        <attvalues>
          <attvalue for="5" value=" Other more recent approaches related to combining language models include proxy tuning \cite{liu2024tuning} and Composition to Augment Language Models (CALM) \cite{bansal2024llm}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2403.00417" id="2040">
        <attvalues>
          <attvalue for="5" value=" Historically, major advances in LMs have come out of subword-level tokenization schemes, which gained traction for their flexibility \cite{yang2024rethinking}, including byte-pair encoding (BPE), SentencePiece, and WordPiece \cite{sennrich-etal-2016-neural, kudo-richardson-2018-sentencepiece, Devlin2019BERTPO, zhang-etal-2019-ernie}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="1508.07909" id="2041">
        <attvalues>
          <attvalue for="5" value=" Historically, major advances in LMs have come out of subword-level tokenization schemes, which gained traction for their flexibility \cite{yang2024rethinking}, including byte-pair encoding (BPE), SentencePiece, and WordPiece \cite{sennrich-etal-2016-neural, kudo-richardson-2018-sentencepiece, Devlin2019BERTPO, zhang-etal-2019-ernie}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2302.14220" id="2043">
        <attvalues>
          <attvalue for="5" value=" One recent study \cite{edman2024characterlevel} fine-tuned a character-level model \cite{xue-etal-2022-byt5} and the model's subword-level counterpart \cite{xue-etal-2021-mt5} for neural machine translation tasks, and found that the character-level model produced improved translation and better cross-lingual generalizations." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2105.13626" id="2044">
        <attvalues>
          <attvalue for="5" value=" One recent study \cite{edman2024characterlevel} fine-tuned a character-level model \cite{xue-etal-2022-byt5} and the model's subword-level counterpart \cite{xue-etal-2021-mt5} for neural machine translation tasks, and found that the character-level model produced improved translation and better cross-lingual generalizations." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2010.11934" id="2045">
        <attvalues>
          <attvalue for="5" value=" One recent study \cite{edman2024characterlevel} fine-tuned a character-level model \cite{xue-etal-2022-byt5} and the model's subword-level counterpart \cite{xue-etal-2021-mt5} for neural machine translation tasks, and found that the character-level model produced improved translation and better cross-lingual generalizations." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2103.06874" id="2046">
        <attvalues>
          <attvalue for="5" value=" More generally, there is some evidence that character-level information can improve performance over other tokenization methods \cite{clark-etal-2022-canine}, particularly in low resource and high language variability settings \cite{riabi-etal-2021-character}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2206.02608" id="2047">
        <attvalues>
          <attvalue for="5" value=" There is some evidence that pretrained language models with subword tokenizers also encode character-level information through the training process \cite{kaushal2022tokens}, further motivating such an approach." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2107.03374" id="2048">
        <attvalues>
          <attvalue for="5" value=" This method demonstrates promising results in improving combined LLM performance across diverse benchmarks, including HumanEval \cite{chen2021evaluating}, GSM8K \cite{cobbe2021training}, and ToxiGen \cite{hartvigsen2022toxigen}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2110.14168" id="2049">
        <attvalues>
          <attvalue for="5" value=" This method demonstrates promising results in improving combined LLM performance across diverse benchmarks, including HumanEval \cite{chen2021evaluating}, GSM8K \cite{cobbe2021training}, and ToxiGen \cite{hartvigsen2022toxigen}." />
        </attvalues>
      </edge>
      <edge source="2407.11009" target="2203.09509" id="2050">
        <attvalues>
          <attvalue for="5" value=" This method demonstrates promising results in improving combined LLM performance across diverse benchmarks, including HumanEval \cite{chen2021evaluating}, GSM8K \cite{cobbe2021training}, and ToxiGen \cite{hartvigsen2022toxigen}." />
        </attvalues>
      </edge>
      <edge source="2404.07413" target="2304.01373" id="2052">
        <attvalues>
          <attvalue for="5" value=" Most LLMs (eg, Llama, \cite{touvron2023llama}; Pythia, \cite{biderman2023pythia}; GPT-3, \cite{brown2020language}; Mistral, \cite{jiang2023mistral}) use all of their parameters during inference and training, which are referred to as dense models." />
        </attvalues>
      </edge>
      <edge source="2404.07413" target="2005.14165" id="2053">
        <attvalues>
          <attvalue for="5" value=" Most LLMs (eg, Llama, \cite{touvron2023llama}; Pythia, \cite{biderman2023pythia}; GPT-3, \cite{brown2020language}; Mistral, \cite{jiang2023mistral}) use all of their parameters during inference and training, which are referred to as dense models." />
        </attvalues>
      </edge>
      <edge source="2404.07413" target="2310.06825" id="2054">
        <attvalues>
          <attvalue for="5" value=" Most LLMs (eg, Llama, \cite{touvron2023llama}; Pythia, \cite{biderman2023pythia}; GPT-3, \cite{brown2020language}; Mistral, \cite{jiang2023mistral}) use all of their parameters during inference and training, which are referred to as dense models." />
        </attvalues>
      </edge>
      <edge source="2404.07413" target="1706.03762" id="2055">
        <attvalues>
          <attvalue for="5" value=" Recent applications of MoE architectures in Transformers \cite{vaswani2017attention} have yielded successful attempts at scaling language models to a substantial size, accompanied by remarkable performance, such as Deepseek MoE~\cite{dai2024deepseekmoe}, Mixtral 8x7B~\cite{jiang2024mixtral}, Grok-1~\cite{grok1_xaiorg}, and DBRX~\cite{dbrx_databricks}." />
        </attvalues>
      </edge>
      <edge source="2404.07413" target="2401.06066" id="2056">
        <attvalues>
          <attvalue for="5" value=" Recent applications of MoE architectures in Transformers \cite{vaswani2017attention} have yielded successful attempts at scaling language models to a substantial size, accompanied by remarkable performance, such as Deepseek MoE~\cite{dai2024deepseekmoe}, Mixtral 8x7B~\cite{jiang2024mixtral}, Grok-1~\cite{grok1_xaiorg}, and DBRX~\cite{dbrx_databricks}." />
        </attvalues>
      </edge>
      <edge source="2404.07413" target="2401.04088" id="2057">
        <attvalues>
          <attvalue for="5" value=" Recent applications of MoE architectures in Transformers \cite{vaswani2017attention} have yielded successful attempts at scaling language models to a substantial size, accompanied by remarkable performance, such as Deepseek MoE~\cite{dai2024deepseekmoe}, Mixtral 8x7B~\cite{jiang2024mixtral}, Grok-1~\cite{grok1_xaiorg}, and DBRX~\cite{dbrx_databricks}." />
        </attvalues>
      </edge>
      <edge source="2404.07413" target="2402.01739" id="2058">
        <attvalues>
          <attvalue for="5" value=" The open-source community has also attempted to train MoE models, such as OpenMoE~\cite{xue2024openmoe}, but its performance is only on par with weak dense models with similar activation parameters, such as OpenLLaMA~\cite{openlm2023openllama} and TinyLLaMA~\cite{zhang2024tinyllama}." />
        </attvalues>
      </edge>
      <edge source="2404.07413" target="2401.02385" id="2059">
        <attvalues>
          <attvalue for="5" value=" The open-source community has also attempted to train MoE models, such as OpenMoE~\cite{xue2024openmoe}, but its performance is only on par with weak dense models with similar activation parameters, such as OpenLLaMA~\cite{openlm2023openllama} and TinyLLaMA~\cite{zhang2024tinyllama}." />
        </attvalues>
      </edge>
      <edge source="2305.12392" target="2303.08774" id="2060">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) like ChatGPT and GPT-4 \cite{DBLP:journals/corr/abs-2303-08774} have been quite successful in solving different generative and reasoning tasks." />
        </attvalues>
      </edge>
      <edge source="2305.12392" target="2209.10754" id="2061">
        <attvalues>
          <attvalue for="5" value=" Generating a semantic graph from text is known as text-to-graph (T2G) generation and is previously attempted mostly by fine-tuning small language models \cite{DBLP:journals/corr/abs-2209-10754,DBLP:journals/corr/abs-2006-04702}." />
        </attvalues>
      </edge>
      <edge source="2305.12392" target="2006.04702" id="2062">
        <attvalues>
          <attvalue for="5" value=" Generating a semantic graph from text is known as text-to-graph (T2G) generation and is previously attempted mostly by fine-tuning small language models \cite{DBLP:journals/corr/abs-2209-10754,DBLP:journals/corr/abs-2006-04702}." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1310.4546" id="2063">
        <attvalues>
          <attvalue for="5" value=" Word2Vec \cite{mikolov2013distributed}, FastText \cite{bojanowski2017enriching}, and Glove \cite{pennington2014glove} are some examples." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1506.06726" id="2065">
        <attvalues>
          <attvalue for="5" value=" SkipThought \cite{kiros2015skip}, InferSent \cite{conneau2017supervised}, and Universal Sentence Encoder \cite{cer2018universal} are well-known examples." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1705.02364" id="2066">
        <attvalues>
          <attvalue for="5" value=" SkipThought \cite{kiros2015skip}, InferSent \cite{conneau2017supervised}, and Universal Sentence Encoder \cite{cer2018universal} are well-known examples." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1905.06316" id="2068">
        <attvalues>
          <attvalue for="5" value=" For this purpose, several probing tasks are proposed to understand what these representations are capturing \cite{tenney2019you,hewitt2019structural,conneau2018you,perone2018evaluation}." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1806.06259" id="2069">
        <attvalues>
          <attvalue for="5" value=" For this purpose, several probing tasks are proposed to understand what these representations are capturing \cite{tenney2019you,hewitt2019structural,conneau2018you,perone2018evaluation}." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1805.04218" id="2070">
        <attvalues>
          <attvalue for="5" value=" One of the interesting findings is that despite the existence of explicit syntactic annotations, these learned deep representations encode syntax to some extent \cite{blevins2018deep}." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1703.00572" id="2071">
        <attvalues>
          <attvalue for="5" value=" Even though deep contextual language models implicitly capture syntactic information of sentences, explicit modeling of syntactic structure of sentences has been shown to further improve the results in different NLP tasks including neural language modeling \cite {shen2017neural, havrylov2019cooperative}, machine comprehension \cite{liu2017structural}, summarization \cite{song2018structure}, text generation \cite{bao2019generating}, machine translation \cite{zhang2019syntax, li2017modeling}, authorship attribution \cite{zhang2018syntax, jafariakinabad2019style, jafariakinabad2020syntactic}, etc Furthermore, Kuncoro et." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1806.05658" id="2072">
        <attvalues>
          <attvalue for="5" value=" Even though deep contextual language models implicitly capture syntactic information of sentences, explicit modeling of syntactic structure of sentences has been shown to further improve the results in different NLP tasks including neural language modeling \cite {shen2017neural, havrylov2019cooperative}, machine comprehension \cite{liu2017structural}, summarization \cite{song2018structure}, text generation \cite{bao2019generating}, machine translation \cite{zhang2019syntax, li2017modeling}, authorship attribution \cite{zhang2018syntax, jafariakinabad2019style, jafariakinabad2020syntactic}, etc Furthermore, Kuncoro et." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1907.05789" id="2073">
        <attvalues>
          <attvalue for="5" value=" Even though deep contextual language models implicitly capture syntactic information of sentences, explicit modeling of syntactic structure of sentences has been shown to further improve the results in different NLP tasks including neural language modeling \cite {shen2017neural, havrylov2019cooperative}, machine comprehension \cite{liu2017structural}, summarization \cite{song2018structure}, text generation \cite{bao2019generating}, machine translation \cite{zhang2019syntax, li2017modeling}, authorship attribution \cite{zhang2018syntax, jafariakinabad2019style, jafariakinabad2020syntactic}, etc Furthermore, Kuncoro et." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1905.02878" id="2074">
        <attvalues>
          <attvalue for="5" value=" Even though deep contextual language models implicitly capture syntactic information of sentences, explicit modeling of syntactic structure of sentences has been shown to further improve the results in different NLP tasks including neural language modeling \cite {shen2017neural, havrylov2019cooperative}, machine comprehension \cite{liu2017structural}, summarization \cite{song2018structure}, text generation \cite{bao2019generating}, machine translation \cite{zhang2019syntax, li2017modeling}, authorship attribution \cite{zhang2018syntax, jafariakinabad2019style, jafariakinabad2020syntactic}, etc Furthermore, Kuncoro et." />
        </attvalues>
      </edge>
      <edge source="2010.06786" target="1909.06194" id="2075">
        <attvalues>
          <attvalue for="5" value=" Inspired by the above observations, our initial work demonstrates that explicit syntactic information of sentences improves the performance of a recurrent neural network classifier in the domain of authorship attribution \cite{jafariakinabad2019style, jafariakinabad2020syntactic}." />
        </attvalues>
      </edge>
      <edge source="2301.00068" target="2005.14165" id="2076">
        <attvalues>
          <attvalue for="5" value=" While GPT-3 \cite{brown2020language} used vanilla next token prediction, following work such as PaLM-2 \cite{anil2023palm}, U-PaLM \cite{tay2022transcending}, GPT-FIM \cite{bavarian2022efficient}, UL2 \cite{tay2022unifying}, and GLM \cite{zeng2022glm} have hinted that incorporating the MLM objective could be highly beneficial to performance." />
        </attvalues>
      </edge>
      <edge source="2301.00068" target="2305.10403" id="2077">
        <attvalues>
          <attvalue for="5" value=" While GPT-3 \cite{brown2020language} used vanilla next token prediction, following work such as PaLM-2 \cite{anil2023palm}, U-PaLM \cite{tay2022transcending}, GPT-FIM \cite{bavarian2022efficient}, UL2 \cite{tay2022unifying}, and GLM \cite{zeng2022glm} have hinted that incorporating the MLM objective could be highly beneficial to performance." />
        </attvalues>
      </edge>
      <edge source="2301.00068" target="2207.14255" id="2078">
        <attvalues>
          <attvalue for="5" value=" While GPT-3 \cite{brown2020language} used vanilla next token prediction, following work such as PaLM-2 \cite{anil2023palm}, U-PaLM \cite{tay2022transcending}, GPT-FIM \cite{bavarian2022efficient}, UL2 \cite{tay2022unifying}, and GLM \cite{zeng2022glm} have hinted that incorporating the MLM objective could be highly beneficial to performance." />
        </attvalues>
      </edge>
      <edge source="2301.00068" target="2210.02414" id="2079">
        <attvalues>
          <attvalue for="5" value=" While GPT-3 \cite{brown2020language} used vanilla next token prediction, following work such as PaLM-2 \cite{anil2023palm}, U-PaLM \cite{tay2022transcending}, GPT-FIM \cite{bavarian2022efficient}, UL2 \cite{tay2022unifying}, and GLM \cite{zeng2022glm} have hinted that incorporating the MLM objective could be highly beneficial to performance." />
        </attvalues>
      </edge>
      <edge source="2301.00068" target="2305.13230" id="2080">
        <attvalues>
          <attvalue for="5" value=" Empirically speaking, predicting masked tokens in the middle of the sentence can be seen as a natural data augmentation technique to vanilla next token prediction, which might be helpful to alleviating the data scarcity problem \cite{xue2023repeat} in the current large model era." />
        </attvalues>
      </edge>
      <edge source="2301.00068" target="2009.03300" id="2081">
        <attvalues>
          <attvalue for="5" value=" (2) We quantify such inconsistencies in benchmark datasets including Lambada \cite{paperno2016Lambada}, MMLU\cite{hendrycks2021measuring} and BigBench \cite{srivastava2023beyond}." />
        </attvalues>
      </edge>
      <edge source="2301.00068" target="2206.04615" id="2082">
        <attvalues>
          <attvalue for="5" value=" (2) We quantify such inconsistencies in benchmark datasets including Lambada \cite{paperno2016Lambada}, MMLU\cite{hendrycks2021measuring} and BigBench \cite{srivastava2023beyond}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="1703.09902" id="2083">
        <attvalues>
          <attvalue for="5" value=" Graph-to-Text (G2T) generation \cite{DBLP:journals/jair/GattK18} is the task of generating natural language from graph-structured data." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="2009.11692" id="2084">
        <attvalues>
          <attvalue for="5" value=" While there are several tasks that could leverage a G2T component~\cite{,DBLP:conf/ijcai/ZhouYHZXZ18,DBLP:conf/emnlp/JiKHWZH20,DBLP:conf/iclr/ChenCSWC21} the direct generation of text description from knowledge graphs (KGs) have attracted a lot of attention due to its potential in providing a more accessible presentation of knowledge to non-experts~\cite{schmitt-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="2010.10439" id="2085">
        <attvalues>
          <attvalue for="5" value=" While there are several tasks that could leverage a G2T component~\cite{,DBLP:conf/ijcai/ZhouYHZXZ18,DBLP:conf/emnlp/JiKHWZH20,DBLP:conf/iclr/ChenCSWC21} the direct generation of text description from knowledge graphs (KGs) have attracted a lot of attention due to its potential in providing a more accessible presentation of knowledge to non-experts~\cite{schmitt-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="1904.09447" id="2086">
        <attvalues>
          <attvalue for="5" value=" While there are several tasks that could leverage a G2T component~\cite{,DBLP:conf/ijcai/ZhouYHZXZ18,DBLP:conf/emnlp/JiKHWZH20,DBLP:conf/iclr/ChenCSWC21} the direct generation of text description from knowledge graphs (KGs) have attracted a lot of attention due to its potential in providing a more accessible presentation of knowledge to non-experts~\cite{schmitt-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="1706.03762" id="2087">
        <attvalues>
          <attvalue for="5" value=" In parallel, Transformer-based \cite{DBLP:conf/nips/VaswaniSPUJGKP17} pre-trained language models (PLMs) such as BART \cite{lewis2019bart}, and T5 \cite{raffel2019exploring} have facilitated state-of-the-art (SotA) results on several tasks, including earlier SotA results for G2T~\cite{DBLP:journals/corr/abs-2007-08426,DBLP:conf/inlg/KaleR20a,mager-etal-2020-gpt}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="1910.13461" id="2088">
        <attvalues>
          <attvalue for="5" value=" In parallel, Transformer-based \cite{DBLP:conf/nips/VaswaniSPUJGKP17} pre-trained language models (PLMs) such as BART \cite{lewis2019bart}, and T5 \cite{raffel2019exploring} have facilitated state-of-the-art (SotA) results on several tasks, including earlier SotA results for G2T~\cite{DBLP:journals/corr/abs-2007-08426,DBLP:conf/inlg/KaleR20a,mager-etal-2020-gpt}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="2007.08426" id="2090">
        <attvalues>
          <attvalue for="5" value=" It has been argued that their success, in part, is due to factual memorisation that guides the generation~\cite{DBLP:journals/corr/abs-2007-08426}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="2005.10433" id="2091">
        <attvalues>
          <attvalue for="5" value=" In parallel, Transformer-based \cite{DBLP:conf/nips/VaswaniSPUJGKP17} pre-trained language models (PLMs) such as BART \cite{lewis2019bart}, and T5 \cite{raffel2019exploring} have facilitated state-of-the-art (SotA) results on several tasks, including earlier SotA results for G2T~\cite{DBLP:journals/corr/abs-2007-08426,DBLP:conf/inlg/KaleR20a,mager-etal-2020-gpt}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="2005.09123" id="2092">
        <attvalues>
          <attvalue for="5" value=" In parallel, Transformer-based \cite{DBLP:conf/nips/VaswaniSPUJGKP17} pre-trained language models (PLMs) such as BART \cite{lewis2019bart}, and T5 \cite{raffel2019exploring} have facilitated state-of-the-art (SotA) results on several tasks, including earlier SotA results for G2T~\cite{DBLP:journals/corr/abs-2007-08426,DBLP:conf/inlg/KaleR20a,mager-etal-2020-gpt}." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="2105.08021" id="2093">
        <attvalues>
          <attvalue for="5" value=" To address this, \cite{DBLP:conf/acl/WangYLJR21} proposed adding extra positional embedding layers to capture the inter-dependency structures of input graphs." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="2103.09120" id="2094">
        <attvalues>
          <attvalue for="5" value=" \cite{DBLP:conf/emnlp/RibeiroZG21} proposed using a structure-aware adapter in PLMs to supplement the input with its graph structure." />
        </attvalues>
      </edge>
      <edge source="2210.10599" target="2111.00276" id="2095">
        <attvalues>
          <attvalue for="5" value=" Our empirical findings highlight that our self-supervised strategies significantly outperform a strong underlying T5 baseline and achieve two new SotA results on two of the datasets WebNLG+2020~\cite{zhou-lampouras-2020-webnlg} and EventNarrative~\cite{DBLP:conf/nips/ColasSWW21}." />
        </attvalues>
      </edge>
      <edge source="2305.12990" target="2110.07524" id="2096">
        <attvalues>
          <attvalue for="5" value=" Sentence embeddings are representations to describe a sentence's meaning and are widely used in natural language tasks such as document classification~\cite{classification}, sentence retrieval~\cite{retrieval}, and question answering~\cite{qa}." />
        </attvalues>
      </edge>
      <edge source="2305.12990" target="2004.14560" id="2097">
        <attvalues>
          <attvalue for="5" value=" Sentence embeddings are representations to describe a sentence's meaning and are widely used in natural language tasks such as document classification~\cite{classification}, sentence retrieval~\cite{retrieval}, and question answering~\cite{qa}." />
        </attvalues>
      </edge>
      <edge source="2305.12990" target="1908.10084" id="2098">
        <attvalues>
          <attvalue for="5" value=" In recent years, machine-learning-based sentence embedding methods with pre-trained language models have become mainstream, and various methods for learning sentence embeddings have been proposed~\cite{SBERT,SimCSE}." />
        </attvalues>
      </edge>
      <edge source="2305.12990" target="2104.08821" id="2099">
        <attvalues>
          <attvalue for="5" value=" In recent years, machine-learning-based sentence embedding methods with pre-trained language models have become mainstream, and various methods for learning sentence embeddings have been proposed~\cite{SBERT,SimCSE}." />
        </attvalues>
      </edge>
      <edge source="2305.12990" target="1412.6623" id="2100">
        <attvalues>
          <attvalue for="5" value=" In this paper, we propose GaussCSE, a Gaussian-distribution-based contrastive sentence embedding to handle such asymmetric relationships between sentences by extending Gaussian embedding for words \cite{Gemb}." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="1706.03762" id="2101">
        <attvalues>
          <attvalue for="5" value=" Attention-based models \cite{vaswani2017attention} have been successful across many areas of machine learning \cite{Jumper2021HighlyAP,videopoet}." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="2401.04088" id="2103">
        <attvalues>
          <attvalue for="5" value=" In particular, large language models (LLMs) comprising decoder-only Transformers pre-trained on large amounts of unlabelled text have become the standard in natural language processing, exhibiting impressive amounts of linguistic and world knowledge \cite{mixtral}." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="2005.08100" id="2104">
        <attvalues>
          <attvalue for="5" value=" In contrast to LLMs, the best performing ASR models are typically based on Conformers \cite{conformer} trained with a connectionist temporal classification \cite{ctc} or RNN transducer \cite{rnn-t} objective." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="1211.3711" id="2105">
        <attvalues>
          <attvalue for="5" value=" In contrast to LLMs, the best performing ASR models are typically based on Conformers \cite{conformer} trained with a connectionist temporal classification \cite{ctc} or RNN transducer \cite{rnn-t} objective." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="1909.09577" id="2106">
        <attvalues>
          <attvalue for="5" value=" This method has been highly successful and is employed in nearly all the best performing ASR models \cite{nemo} such as USM \cite{usm} which also serves as the speech encoder of Gemini v1 \cite{geminiv1}." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="2303.01037" id="2107">
        <attvalues>
          <attvalue for="5" value=" This method has been highly successful and is employed in nearly all the best performing ASR models \cite{nemo} such as USM \cite{usm} which also serves as the speech encoder of Gemini v1 \cite{geminiv1}." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="2312.11805" id="2108">
        <attvalues>
          <attvalue for="5" value=" This method has been highly successful and is employed in nearly all the best performing ASR models \cite{nemo} such as USM \cite{usm} which also serves as the speech encoder of Gemini v1 \cite{geminiv1}." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="2309.13876" id="2110">
        <attvalues>
          <attvalue for="5" value=" The OWSM model \cite{owsm} is a step towards this direction and comprises a Whisper-style encoder-decoder model trained on a compilation of public multilingual ASR corpora." />
        </attvalues>
      </edge>
      <edge source="2402.00235" target="2005.14165" id="2111">
        <attvalues>
          <attvalue for="5" value=" However, as OWSM models are encoder-decoder Transformers trained using an additional CTC-based loss, it remains to be answered if conventional Transformer decoder training, similar to that for LLMs \cite{Brown2020LanguageMA}, suffices for competitive performance." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2005.14165" id="2114">
        <attvalues>
          <attvalue for="5" value=" Following this trend, modern language models often have hundreds of billions of parameters~\cite{gpt3,gopher,pangua,hyperclova}\nocite{switch,jurrasic,Lepikhin2020GShardSG,glam}." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2109.04650" id="2115">
        <attvalues>
          <attvalue for="5" value=" Following this trend, modern language models often have hundreds of billions of parameters~\cite{gpt3,gopher,pangua,hyperclova}\nocite{switch,jurrasic,Lepikhin2020GShardSG,glam}." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2205.01068" id="2116">
        <attvalues>
          <attvalue for="5" value=" Several research groups released pretrained LLMs with over 100B parameters~\cite{opt,yalm,zeng2020glm}\nocite{gpt,gpt-neox-20b}." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2210.02414" id="2117">
        <attvalues>
          <attvalue for="5" value=" Several research groups released pretrained LLMs with over 100B parameters~\cite{opt,yalm,zeng2020glm}\nocite{gpt,gpt-neox-20b}." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2211.05100" id="2118">
        <attvalues>
          <attvalue for="5" value=" Most recently, the BigScience project has released BLOOM, a 176 billion parameter model supporting 46 natural and 13 programming languages~\cite{bloom}." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2101.06840" id="2120">
        <attvalues>
          <attvalue for="5" value=" Several recent works aim to democratize LLMs by ``offloading'' model parameters to slower but cheaper memory (RAM or SSD), then running them on the accelerator layer by layer~\cite{l2l,zerooffload}\nocite{accelerate}." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2205.05638" id="2121">
        <attvalues>
          <attvalue for="5" value=" On top of that, current API pricing can make some research projects prohibitively expensive~\cite{tfew}." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2002.04013" id="2122">
        <attvalues>
          <attvalue for="5" value=" In this work, we explore an alternative strategy inspired by crowdsourced distributed training of neural networks from scratch~\cite{hivemind_dmoe}." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="1902.00751" id="2123">
        <attvalues>
          <attvalue for="5" value=" Aside from inference, participants can fine-tune the model through parameter-efficient training methods like adapters \cite{houlsby2019parameter} or prompt tuning \cite{ptune-lester} or by training entire layers (Section~\ref{sect:design_training})." />
        </attvalues>
      </edge>
      <edge source="2209.01188" target="2104.08691" id="2124">
        <attvalues>
          <attvalue for="5" value=" Aside from inference, participants can fine-tune the model through parameter-efficient training methods like adapters \cite{houlsby2019parameter} or prompt tuning \cite{ptune-lester} or by training entire layers (Section~\ref{sect:design_training})." />
        </attvalues>
      </edge>
      <edge source="2209.02317" target="1909.02622" id="2126">
        <attvalues>
          <attvalue for="5" value=" Recent metrics like BERTScore \cite{bert-score}, MoverScore \cite{zhao-etal-2019-moverscore}, COMET \cite{rei-etal-2020-comet}, BARTScore \cite{yuan2021bartscore}, and BLEURT \cite{sellam-etal-2020-bleurt} adapt pretrained contextualized word embeddings to tackle this issue." />
        </attvalues>
      </edge>
      <edge source="2209.02317" target="2009.09025" id="2127">
        <attvalues>
          <attvalue for="5" value=" Recent metrics like BERTScore \cite{bert-score}, MoverScore \cite{zhao-etal-2019-moverscore}, COMET \cite{rei-etal-2020-comet}, BARTScore \cite{yuan2021bartscore}, and BLEURT \cite{sellam-etal-2020-bleurt} adapt pretrained contextualized word embeddings to tackle this issue." />
        </attvalues>
      </edge>
      <edge source="2209.02317" target="2106.11520" id="2128">
        <attvalues>
          <attvalue for="5" value=" Recent metrics like BERTScore \cite{bert-score}, MoverScore \cite{zhao-etal-2019-moverscore}, COMET \cite{rei-etal-2020-comet}, BARTScore \cite{yuan2021bartscore}, and BLEURT \cite{sellam-etal-2020-bleurt} adapt pretrained contextualized word embeddings to tackle this issue." />
        </attvalues>
      </edge>
      <edge source="2209.02317" target="2004.04696" id="2129">
        <attvalues>
          <attvalue for="5" value=" Recent metrics like BERTScore \cite{bert-score}, MoverScore \cite{zhao-etal-2019-moverscore}, COMET \cite{rei-etal-2020-comet}, BARTScore \cite{yuan2021bartscore}, and BLEURT \cite{sellam-etal-2020-bleurt} adapt pretrained contextualized word embeddings to tackle this issue." />
        </attvalues>
      </edge>
      <edge source="2209.02317" target="2010.05648" id="2130">
        <attvalues>
          <attvalue for="5" value=" In order to perform a systematic evaluation on the robustness of BERTScore with regard to the ratio of unknown tokens, we use character-based adversarial attacks \cite{eger2020hero} that introduce a controlled ratio of new unknown tokens to the input texts." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="1907.11692" id="2132">
        <attvalues>
          <attvalue for="5" value=" Recently, many large pretrained language models (PLMs, \cite{bert,roberta,xlnet,megatron,t5}) have been proposed for a variety of Natural Language Processing (NLP) tasks." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="1909.08053" id="2134">
        <attvalues>
          <attvalue for="5" value=" Recently, many large pretrained language models (PLMs, \cite{bert,roberta,xlnet,megatron,t5}) have been proposed for a variety of Natural Language Processing (NLP) tasks." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="1906.02243" id="2136">
        <attvalues>
          <attvalue for="5" value=" However, as pointed out in recent studies~\cite{strubell2019energy,greenai,parrot}, these models suffer from computational inefficiency and high ecological cost." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="1907.10597" id="2137">
        <attvalues>
          <attvalue for="5" value=" However, as pointed out in recent studies~\cite{strubell2019energy,greenai,parrot}, these models suffer from computational inefficiency and high ecological cost." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="1909.05840" id="2139">
        <attvalues>
          <attvalue for="5" value=" Many attempts have been made to address this problem, including quantization~\cite{q8bert,qbert}, pruning~\cite{headprune,mvp}, knowledge distillation (KD)~\cite{distilbert,bertpkd,mobilebert,pd,tinybert,minilm,zhou2021meta} and progressive module replacing~\cite{bot}." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="1905.10650" id="2140">
        <attvalues>
          <attvalue for="5" value=" Many attempts have been made to address this problem, including quantization~\cite{q8bert,qbert}, pruning~\cite{headprune,mvp}, knowledge distillation (KD)~\cite{distilbert,bertpkd,mobilebert,pd,tinybert,minilm,zhou2021meta} and progressive module replacing~\cite{bot}." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="2005.07683" id="2141">
        <attvalues>
          <attvalue for="5" value=" Many attempts have been made to address this problem, including quantization~\cite{q8bert,qbert}, pruning~\cite{headprune,mvp}, knowledge distillation (KD)~\cite{distilbert,bertpkd,mobilebert,pd,tinybert,minilm,zhou2021meta} and progressive module replacing~\cite{bot}." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="1908.09355" id="2143">
        <attvalues>
          <attvalue for="5" value=" Many attempts have been made to address this problem, including quantization~\cite{q8bert,qbert}, pruning~\cite{headprune,mvp}, knowledge distillation (KD)~\cite{distilbert,bertpkd,mobilebert,pd,tinybert,minilm,zhou2021meta} and progressive module replacing~\cite{bot}." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="2004.02984" id="2144">
        <attvalues>
          <attvalue for="5" value=" Many attempts have been made to address this problem, including quantization~\cite{q8bert,qbert}, pruning~\cite{headprune,mvp}, knowledge distillation (KD)~\cite{distilbert,bertpkd,mobilebert,pd,tinybert,minilm,zhou2021meta} and progressive module replacing~\cite{bot}." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="1909.10351" id="2145">
        <attvalues>
          <attvalue for="5" value=" (2) Using preserved accuracy to evaluate models compressed with more data or data augmentation~\cite{tinybert} can be misleading, since one cannot tell whether the improvement should be attributed to the innovation of the compression technique or addition of data." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="2002.10957" id="2146">
        <attvalues>
          <attvalue for="5" value=" Many attempts have been made to address this problem, including quantization~\cite{q8bert,qbert}, pruning~\cite{headprune,mvp}, knowledge distillation (KD)~\cite{distilbert,bertpkd,mobilebert,pd,tinybert,minilm,zhou2021meta} and progressive module replacing~\cite{bot}." />
        </attvalues>
      </edge>
      <edge source="2109.03228" target="2002.02925" id="2147">
        <attvalues>
          <attvalue for="5" value=" Many attempts have been made to address this problem, including quantization~\cite{q8bert,qbert}, pruning~\cite{headprune,mvp}, knowledge distillation (KD)~\cite{distilbert,bertpkd,mobilebert,pd,tinybert,minilm,zhou2021meta} and progressive module replacing~\cite{bot}." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2307.09288" id="2149">
        <attvalues>
          <attvalue for="5" value=" Examples include the LLaMA \cite{bib4,bib402}, Phi \cite{bib34,bib403,bib404}, ChatGLM \cite{bib3,bib2}, QWen \cite{bib247}, Baichuan \cite{bib5}, and so on." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2306.11644" id="2150">
        <attvalues>
          <attvalue for="5" value=" Examples include the LLaMA \cite{bib4,bib402}, Phi \cite{bib34,bib403,bib404}, ChatGLM \cite{bib3,bib2}, QWen \cite{bib247}, Baichuan \cite{bib5}, and so on." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2309.05463" id="2151">
        <attvalues>
          <attvalue for="5" value=" Examples include the LLaMA \cite{bib4,bib402}, Phi \cite{bib34,bib403,bib404}, ChatGLM \cite{bib3,bib2}, QWen \cite{bib247}, Baichuan \cite{bib5}, and so on." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2103.10360" id="2152">
        <attvalues>
          <attvalue for="5" value=" Examples include the LLaMA \cite{bib4,bib402}, Phi \cite{bib34,bib403,bib404}, ChatGLM \cite{bib3,bib2}, QWen \cite{bib247}, Baichuan \cite{bib5}, and so on." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2210.02414" id="2153">
        <attvalues>
          <attvalue for="5" value=" Examples include the LLaMA \cite{bib4,bib402}, Phi \cite{bib34,bib403,bib404}, ChatGLM \cite{bib3,bib2}, QWen \cite{bib247}, Baichuan \cite{bib5}, and so on." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2309.10305" id="2155">
        <attvalues>
          <attvalue for="5" value=" Examples include the LLaMA \cite{bib4,bib402}, Phi \cite{bib34,bib403,bib404}, ChatGLM \cite{bib3,bib2}, QWen \cite{bib247}, Baichuan \cite{bib5}, and so on." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="1911.00536" id="2157">
        <attvalues>
          <attvalue for="5" value=" After 2000, the NLP field continued to emphasize research on traditional tasks and linguistic structures, while also turning attention to emerging areas such as dialogue systems \cite{bib405,bib407,bib48,bib406}." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2206.04615" id="2158">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2308.04813" id="2160">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2305.14233" id="2161">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2009.03300" id="2162">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2107.07498" id="2163">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2301.13688" id="2164">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2003.11080" id="2166">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2207.04672" id="2167">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, comprehensive performance evaluations \cite{bib242,bib244,bib245}, dialogue datasets \cite{bib107,bib66,bib71}, zero-shot and few-shot datasets \cite{bib171,bib252,bib79}, multilingual datasets \cite{bib249,bib250,bib358}, and others emerged." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2303.18223" id="2168">
        <attvalues>
          <attvalue for="5" value=" Prior to this, several LLM-related surveys, such as \cite{bib7} and \cite{bib410}, analyze the latest developments in LLMs but lack detailed descriptions and summaries of datasets." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2308.10792" id="2170">
        <attvalues>
          <attvalue for="5" value=" \cite{bib8} summarizes the instruction fine-tuning stage of LLMs." />
        </attvalues>
      </edge>
      <edge source="2402.18041" target="2307.03109" id="2171">
        <attvalues>
          <attvalue for="5" value=" \cite{bib9} and \cite{bib10} summarize the evaluation stage." />
        </attvalues>
      </edge>
      <edge source="2206.03702" target="1908.05646" id="2175">
        <attvalues>
          <attvalue for="5" value=" \cite{levine2019sensebert} develop SenseBert, introducing super-senses from Wordnet \cite{miller1995wordnet} into general Bert model." />
        </attvalues>
      </edge>
      <edge source="2206.03702" target="1904.09223" id="2176">
        <attvalues>
          <attvalue for="5" value=" Ernie \cite{sun2019ernie} combines node embeddings from knowledge graph and matched entities to enhance word representations." />
        </attvalues>
      </edge>
      <edge source="2206.03702" target="1909.04164" id="2177">
        <attvalues>
          <attvalue for="5" value=" KnowBert \cite{peters-etal-2019-knowledge} subsumes the entity connection and Bert models, which are trained together." />
        </attvalues>
      </edge>
      <edge source="2301.12140" target="2101.00148" id="2179">
        <attvalues>
          <attvalue for="5" value=" It is useful in a variety of natural language processing (NLP) applications such as noisy parallel corpus filtering~\cite{kurfali-ostling-2019-noisy}, bilingual lexicon induction \cite{shi-etal-2021-bilingual}, code-switching corpus building~\cite{lee2019linguistically,lin2020pre} and incorporating lexical constraints into neural machine translation (NMT) models~\cite{hasler2018neural,chen2021lexically}." />
        </attvalues>
      </edge>
      <edge source="2301.12140" target="2010.03142" id="2180">
        <attvalues>
          <attvalue for="5" value=" It is useful in a variety of natural language processing (NLP) applications such as noisy parallel corpus filtering~\cite{kurfali-ostling-2019-noisy}, bilingual lexicon induction \cite{shi-etal-2021-bilingual}, code-switching corpus building~\cite{lee2019linguistically,lin2020pre} and incorporating lexical constraints into neural machine translation (NMT) models~\cite{hasler2018neural,chen2021lexically}." />
        </attvalues>
      </edge>
      <edge source="2301.12140" target="1805.03750" id="2181">
        <attvalues>
          <attvalue for="5" value=" It is useful in a variety of natural language processing (NLP) applications such as noisy parallel corpus filtering~\cite{kurfali-ostling-2019-noisy}, bilingual lexicon induction \cite{shi-etal-2021-bilingual}, code-switching corpus building~\cite{lee2019linguistically,lin2020pre} and incorporating lexical constraints into neural machine translation (NMT) models~\cite{hasler2018neural,chen2021lexically}." />
        </attvalues>
      </edge>
      <edge source="2301.12140" target="1909.02074" id="2182">
        <attvalues>
          <attvalue for="5" value=" Some works \cite{garg-etal-2019-jointly, li-etal-2019-word,zenkel2019adding,zenkel-etal-2020-end, chen-etal-2020-accurate, zhang-van-genabith-2021-bidirectional,chen-etal-2021-mask} induce alignments from NMT model or its variants." />
        </attvalues>
      </edge>
      <edge source="2301.12140" target="2004.08728" id="2186">
        <attvalues>
          <attvalue for="5" value=" Another line of works~\cite{jalili-sabet-etal-2020-simalign,dou-neubig-2021-word} build multilingual word aligners with contextualized embeddings from the multilingual pretrained language model \cite[mPLM]{mbert,conneau-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2301.12140" target="1904.09077" id="2188">
        <attvalues>
          <attvalue for="5" value=" Another line of works~\cite{jalili-sabet-etal-2020-simalign,dou-neubig-2021-word} build multilingual word aligners with contextualized embeddings from the multilingual pretrained language model \cite[mPLM]{mbert,conneau-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2301.12140" target="1911.02116" id="2189">
        <attvalues>
          <attvalue for="5" value=" Another line of works~\cite{jalili-sabet-etal-2020-simalign,dou-neubig-2021-word} build multilingual word aligners with contextualized embeddings from the multilingual pretrained language model \cite[mPLM]{mbert,conneau-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2404.13292" target="1508.07909" id="2190">
        <attvalues>
          <attvalue for="5" value=" Popular subword tokenization methods, such as Byte Pair Encoding \cite[BPE;][]{sennrich2016neural} or Unigram Language Model \cite[ULM;][]{kudo2018subword}, are adaptations of data compression algorithms that mainly rely on word character co-occurrence statistics in a given text corpus, rather than on human knowledge and understanding about word formations and morphology." />
        </attvalues>
      </edge>
      <edge source="2404.13292" target="1804.10959" id="2191">
        <attvalues>
          <attvalue for="5" value=" Popular subword tokenization methods, such as Byte Pair Encoding \cite[BPE;][]{sennrich2016neural} or Unigram Language Model \cite[ULM;][]{kudo2018subword}, are adaptations of data compression algorithms that mainly rely on word character co-occurrence statistics in a given text corpus, rather than on human knowledge and understanding about word formations and morphology." />
        </attvalues>
      </edge>
      <edge source="2404.13292" target="2112.10508" id="2192">
        <attvalues>
          <attvalue for="5" value=" Despite the success of subword tokenization in popular NLP applications (including machine translation, text generation, and text classification) and a wide range of practical \cite{mielke2021between} and cognitive studies \cite{beinborn2023analyzing}, evaluating subword tokenization algorithms is still an open problem for at least two reasons." />
        </attvalues>
      </edge>
      <edge source="2404.13292" target="2310.13348" id="2193">
        <attvalues>
          <attvalue for="5" value=" Despite the success of subword tokenization in popular NLP applications (including machine translation, text generation, and text classification) and a wide range of practical \cite{mielke2021between} and cognitive studies \cite{beinborn2023analyzing}, evaluating subword tokenization algorithms is still an open problem for at least two reasons." />
        </attvalues>
      </edge>
      <edge source="2404.13292" target="2404.02421" id="2194">
        <attvalues>
          <attvalue for="5" value=" The second motivation is that little or no effort has gone into developing a standard extrinsic NLP benchmark to evaluate how tokenizers with different behaviors impact predictions of downstream tasks in NLP \cite{truong2024revisiting}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1703.09902" id="2195">
        <attvalues>
          <attvalue for="5" value=" Automated data-to-text (D2T) generation systems can transform and organize this knowledge into natural language text snippets that enable broader access~\cite{gatt2018survey}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1910.08435" id="2196">
        <attvalues>
          <attvalue for="5" value=" Applications of this technology include story or dialogue generation~\cite{moon2019opendialkg}, open-domain question-answering~\cite{ma2021open,fan2019using}, and text summarization~\cite{wiseman2017challenges}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1707.08052" id="2197">
        <attvalues>
          <attvalue for="5" value=" Applications of this technology include story or dialogue generation~\cite{moon2019opendialkg}, open-domain question-answering~\cite{ma2021open,fan2019using}, and text summarization~\cite{wiseman2017challenges}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1411.4925" id="2198">
        <attvalues>
          <attvalue for="5" value=" Domains span journalism~\cite{leppanen2017data}, weather~\cite{ramos2014linguistic,mei2015talk}, finance, sports~\cite{plachouras2016interacting, chen2008learning,van2017pass}, and summarizing patient medical histories~\cite{portet2009automatic}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1907.11692" id="2201">
        <attvalues>
          <attvalue for="5" value=" Historically, D2T systems included pipeline approaches with customized models \cite{gardent2017webnlg}, but have now shifted to pretrained Transformer-based language models (PLMs)~\cite{devlin2018bert,liu2019roberta,radford2019language}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1910.13461" id="2203">
        <attvalues>
          <attvalue for="5" value=" Models trained on DART, both larger and more diverse than previous corpora, improve the performance of BART \cite{lewis2019bart} and T5 on the standard WebNLG challenge~\cite{gardent2017webnlg}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1801.06146" id="2204">
        <attvalues>
          <attvalue for="5" value=" This approach requires a PLM to be fine-tuned on a task-specific in-domain dataset~\cite{howard2018universal,see2019massively,keskar2019ctrl}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1909.10705" id="2205">
        <attvalues>
          <attvalue for="5" value=" This approach requires a PLM to be fine-tuned on a task-specific in-domain dataset~\cite{howard2018universal,see2019massively,keskar2019ctrl}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="1909.05858" id="2206">
        <attvalues>
          <attvalue for="5" value=" This approach requires a PLM to be fine-tuned on a task-specific in-domain dataset~\cite{howard2018universal,see2019massively,keskar2019ctrl}." />
        </attvalues>
      </edge>
      <edge source="2205.11505" target="2104.08691" id="2208">
        <attvalues>
          <attvalue for="5" value=" Other strategies, such as prompt tuning~\cite{lester2021power}, can adapt PLMs to specific down-stream tasks by updating only a small subset of model parameters." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="1706.03762" id="2209">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) \cite{vaswani2017attention, radford2019language, gemini, GPT-4} have demonstrated remarkable general capabilities \cite{sparks}." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="2303.12712" id="2210">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) \cite{vaswani2017attention, radford2019language, gemini, GPT-4} have demonstrated remarkable general capabilities \cite{sparks}." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="2001.08361" id="2211">
        <attvalues>
          <attvalue for="5" value=" Research on scaling laws \cite{scaling-law, scaling-law-2, chinchilla, mu-scaling} indicates that metrics related to perplexity (eg, loss and BPB) improve as training FLOPs increase." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="2310.06825" id="2212">
        <attvalues>
          <attvalue for="5" value=" Recent iterations of popular LLM series, such as Mistral at 141B \cite{mistral}, DeepSeek at 236B \cite{deepseek}, Grok at 314B \cite{grok}, and Llama-3 exceeding 400B parameters \cite{llama3}, underscore a trend toward models with 1 trillion parameters." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="2401.02954" id="2213">
        <attvalues>
          <attvalue for="5" value=" Recent iterations of popular LLM series, such as Mistral at 141B \cite{mistral}, DeepSeek at 236B \cite{deepseek}, Grok at 314B \cite{grok}, and Llama-3 exceeding 400B parameters \cite{llama3}, underscore a trend toward models with 1 trillion parameters." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="2212.10560" id="2216">
        <attvalues>
          <attvalue for="5" value=" As a consequent work, we focus on two prominent areas of research with the Tele-FLM models: alignment with human \cite{instructgpt,alpaca} and progressive learning \cite{bert2bert,msg,flm101b}." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="2110.07143" id="2217">
        <attvalues>
          <attvalue for="5" value=" As a consequent work, we focus on two prominent areas of research with the Tele-FLM models: alignment with human \cite{instructgpt,alpaca} and progressive learning \cite{bert2bert,msg,flm101b}." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="2305.02869" id="2218">
        <attvalues>
          <attvalue for="5" value=" As a consequent work, we focus on two prominent areas of research with the Tele-FLM models: alignment with human \cite{instructgpt,alpaca} and progressive learning \cite{bert2bert,msg,flm101b}." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="2305.11206" id="2219">
        <attvalues>
          <attvalue for="5" value=" We explore different data combination and training settings, finding that leveraging the existing knowledge and capabilities of the foundation model with a limited dataset of instruction-focused tasks yields better results than merely increasing the volume of instruction data \cite{lima}, even when the instructional responses are of high quality." />
        </attvalues>
      </edge>
      <edge source="2407.02783" target="1511.05641" id="2220">
        <attvalues>
          <attvalue for="5" value=" The central strategy involves expanding the model's structure during the pre-training phase and utilizing function-preserving growth techniques \cite{net2net, msg, flm101b} to transfer knowledge seamlessly from one stage to the next." />
        </attvalues>
      </edge>
      <edge source="2310.18877" target="2212.11261" id="2222">
        <attvalues>
          <attvalue for="5" value=" In computer vision and natural language processing, methods called Embedding Association Tests (EATs) have been used to evaluate biases in the ways pre-trained models represent social groups, allowing researchers to identify bias early in the machine learning pipeline, before it propagates to downstream tasks \cite{Wolfe2023ContrastiveBias,Wolfe2022AmericanAI,Wolfe2022VAST:Models,Steed2021, Guo2021DetectingBiases, ToneyWails2021,caliskan2016semantics}." />
        </attvalues>
      </edge>
      <edge source="2310.18877" target="2207.00691" id="2223">
        <attvalues>
          <attvalue for="5" value=" In computer vision and natural language processing, methods called Embedding Association Tests (EATs) have been used to evaluate biases in the ways pre-trained models represent social groups, allowing researchers to identify bias early in the machine learning pipeline, before it propagates to downstream tasks \cite{Wolfe2023ContrastiveBias,Wolfe2022AmericanAI,Wolfe2022VAST:Models,Steed2021, Guo2021DetectingBiases, ToneyWails2021,caliskan2016semantics}." />
        </attvalues>
      </edge>
      <edge source="2310.18877" target="2203.07504" id="2224">
        <attvalues>
          <attvalue for="5" value=" In computer vision and natural language processing, methods called Embedding Association Tests (EATs) have been used to evaluate biases in the ways pre-trained models represent social groups, allowing researchers to identify bias early in the machine learning pipeline, before it propagates to downstream tasks \cite{Wolfe2023ContrastiveBias,Wolfe2022AmericanAI,Wolfe2022VAST:Models,Steed2021, Guo2021DetectingBiases, ToneyWails2021,caliskan2016semantics}." />
        </attvalues>
      </edge>
      <edge source="2310.18877" target="2010.15052" id="2225">
        <attvalues>
          <attvalue for="5" value=" In computer vision and natural language processing, methods called Embedding Association Tests (EATs) have been used to evaluate biases in the ways pre-trained models represent social groups, allowing researchers to identify bias early in the machine learning pipeline, before it propagates to downstream tasks \cite{Wolfe2023ContrastiveBias,Wolfe2022AmericanAI,Wolfe2022VAST:Models,Steed2021, Guo2021DetectingBiases, ToneyWails2021,caliskan2016semantics}." />
        </attvalues>
      </edge>
      <edge source="2310.18877" target="2109.08256" id="2226">
        <attvalues>
          <attvalue for="5" value=" Beyond showing that biases found in pre-trained models can propagate, our work adds to the growing body of evidence showing bias in Automated Emotion Recognition (AER), one of many issues that have been raised concerning this area of research \cite{Mohammad2021EthicsAnalysis}." />
        </attvalues>
      </edge>
      <edge source="2311.14685" target="2303.08774" id="2227">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in Large Language Models (LLMs), such as GPT-4~\cite{O23} and LLaMA~\cite{TLIMLLRGHARJGL23}, have led to their rapid adoption in various domains." />
        </attvalues>
      </edge>
      <edge source="2311.14685" target="2301.08653" id="2229">
        <attvalues>
          <attvalue for="5" value=" This success can be attributed to its exceptional performance in a wide array of NLP tasks, including code debugging~\cite{SBHP23}, question answering~\cite{SCBZ23}, and creative writing~\cite{GAG23, SWG23}." />
        </attvalues>
      </edge>
      <edge source="2311.14685" target="2304.08979" id="2230">
        <attvalues>
          <attvalue for="5" value=" This success can be attributed to its exceptional performance in a wide array of NLP tasks, including code debugging~\cite{SBHP23}, question answering~\cite{SCBZ23}, and creative writing~\cite{GAG23, SWG23}." />
        </attvalues>
      </edge>
      <edge source="2311.14685" target="2305.03429" id="2231">
        <attvalues>
          <attvalue for="5" value=" This success can be attributed to its exceptional performance in a wide array of NLP tasks, including code debugging~\cite{SBHP23}, question answering~\cite{SCBZ23}, and creative writing~\cite{GAG23, SWG23}." />
        </attvalues>
      </edge>
      <edge source="2311.14685" target="2302.04023" id="2232">
        <attvalues>
          <attvalue for="5" value=" Moreover, recent studies have shown that one of the current state-of-the-art LLMs, eg, GPT-4, has even surpassed human-level performance in multiple benchmarks~\cite{BCLDSWLJYCDXF23}." />
        </attvalues>
      </edge>
      <edge source="2311.14685" target="1911.03842" id="2233">
        <attvalues>
          <attvalue for="5" value=" While this task has always been demanding~\cite{DFWUKW20, DABSHBR21, SBBCSZZ22}, the emergence of LLMs like ChatGPT brings unique challenges in this domain." />
        </attvalues>
      </edge>
      <edge source="2311.14685" target="2302.05733" id="2234">
        <attvalues>
          <attvalue for="5" value=" For instance, ``jailbreak'' prompts have been used to bypass ChatGPT's restrictions and provoke extremely harmful content~\cite{KLSGZH23}." />
        </attvalues>
      </edge>
      <edge source="2311.14685" target="2304.05335" id="2235">
        <attvalues>
          <attvalue for="5" value=" Using a broad range of system roles, we find this setting for the current model can affect response toxicity but not as extremely as discovered in previous work~\cite{DMRKN23}, and the prompt content is the more important factor." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="2003.08271" id="2236">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{PLM_survey,add1,add2} which are trained on copious amount of corpora have been corroborated valid on an assortment of tasks~\cite{devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="1907.04944" id="2237">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{PLM_survey,add1,add2} which are trained on copious amount of corpora have been corroborated valid on an assortment of tasks~\cite{devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="1912.02164" id="2238">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{PLM_survey,add1,add2} which are trained on copious amount of corpora have been corroborated valid on an assortment of tasks~\cite{devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="1907.11692" id="2240">
        <attvalues>
          <attvalue for="5" value=" For example, RoBERTa-base~\cite{liu2019roberta} and T5-large~\cite{2020t5} have about 125M and 770M parameters respectively." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="1901.11504" id="2242">
        <attvalues>
          <attvalue for="5" value=" Multi-task learning (MTL)~\cite{liu2019mt-dnn} is a solution to avoid storing multiple copies of LMs." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="2107.13586" id="2243">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning~\cite{liu_pretrain_prompt} includes discrete prompt tuning and continuous prompt tuning." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="2005.14165" id="2244">
        <attvalues>
          <attvalue for="5" value=" Discrete prompt tuning ~\cite{NEURIPS2020_1457c0d6,petroni-etal-2019-language,schick-schutze-2021-exploiting} usually inserts some tokens (appearing in the models' vocabulary) to the input sentences to reformulate tasks as fillin-the-blanks problems." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="1909.01066" id="2245">
        <attvalues>
          <attvalue for="5" value=" Discrete prompt tuning ~\cite{NEURIPS2020_1457c0d6,petroni-etal-2019-language,schick-schutze-2021-exploiting} usually inserts some tokens (appearing in the models' vocabulary) to the input sentences to reformulate tasks as fillin-the-blanks problems." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="2104.08691" id="2246">
        <attvalues>
          <attvalue for="5" value=" We conduct experiments on a variety of tasks and the results show that our method can attain performance comparable with that achieved by P-tuning~\cite{power_prompt} and P-tuning v2~\cite{liu2021ptuning}." />
        </attvalues>
      </edge>
      <edge source="2204.04596" target="2101.00190" id="2247">
        <attvalues>
          <attvalue for="5" value=" Instead of inserting discrete tokens, continuous prompt tuning~\cite{power_prompt,li-liang-2021-prefix} adds task-specific trainable vectors to the embeddings of input sentences." />
        </attvalues>
      </edge>
      <edge source="2402.12280" target="2310.17157" id="2250">
        <attvalues>
          <attvalue for="5" value=" One of the reasons behind this high inference overhead is that LLMs are highly memory bandwidth-bounded~\cite{liu2023deja, zheng2024learn}." />
        </attvalues>
      </edge>
      <edge source="2403.16084" target="2212.08913" id="2252">
        <attvalues>
          <attvalue for="5" value=" But the situation there is the same: It is not enough to mine or generate arguments; their quality also needs to be evaluable \cite{park:2018}, so that it can be assessed \cite{lauscher:2020}, flaws can be found \cite{goffredo:2022}, and accounted for \cite{skitalinskaya:2023b}." />
        </attvalues>
      </edge>
      <edge source="2403.16084" target="2303.08774" id="2253">
        <attvalues>
          <attvalue for="5" value=" While the challenges of diversity and subjectivity prevail until today \cite{lapesa:2023}, NLP is now seeing a revolutionary breakthrough: the rise of instruction-following large language models (henceforth, LLMs) that can tackle various NLP tasks with little to no task-specific fine-tuning, enabled by their supreme capability to integrate and leverage knowledge across contexts \cite{openai:2023}." />
        </attvalues>
      </edge>
      <edge source="2403.16084" target="2204.04026" id="2254">
        <attvalues>
          <attvalue for="5" value=" These issues deserve treatment in computational argumentation as well; they are even particularly critical there due to the sensitivity of many controversial topics \cite{holtermann:2022}." />
        </attvalues>
      </edge>
      <edge source="2005.05255" target="1409.0473" id="2255">
        <attvalues>
          <attvalue for="5" value=" In the recent year, neural language models have made tremendous progress with respect to fluency~\cite{bahdanau2014neural,vaswani2017attention,bengio2003lm,devlin2019bert}, but coherency is still a major challenge~\cite{see:2019:storyteller}." />
        </attvalues>
      </edge>
      <edge source="2005.05255" target="1706.03762" id="2256">
        <attvalues>
          <attvalue for="5" value=" In the recent year, neural language models have made tremendous progress with respect to fluency~\cite{bahdanau2014neural,vaswani2017attention,bengio2003lm,devlin2019bert}, but coherency is still a major challenge~\cite{see:2019:storyteller}." />
        </attvalues>
      </edge>
      <edge source="2005.05255" target="1909.10705" id="2258">
        <attvalues>
          <attvalue for="5" value=" In the recent year, neural language models have made tremendous progress with respect to fluency~\cite{bahdanau2014neural,vaswani2017attention,bengio2003lm,devlin2019bert}, but coherency is still a major challenge~\cite{see:2019:storyteller}." />
        </attvalues>
      </edge>
      <edge source="2005.05255" target="1805.04833" id="2259">
        <attvalues>
          <attvalue for="5" value=" The generation of coherent stories has recently been addressed with additional conditioning: \cite{fan2018hierarchical} suggest conditioning on a story prompt, \cite{clark2018creative} propose collaboration between a generative model and a human writer, and \cite{guan2019commonsense} suggest attending to a commonsense graph relevant to the story plot." />
        </attvalues>
      </edge>
      <edge source="2005.05255" target="1808.10113" id="2260">
        <attvalues>
          <attvalue for="5" value=" The generation of coherent stories has recently been addressed with additional conditioning: \cite{fan2018hierarchical} suggest conditioning on a story prompt, \cite{clark2018creative} propose collaboration between a generative model and a human writer, and \cite{guan2019commonsense} suggest attending to a commonsense graph relevant to the story plot." />
        </attvalues>
      </edge>
      <edge source="2005.05255" target="1706.01331" id="2261">
        <attvalues>
          <attvalue for="5" value=" Conditioning based on a generated story plan \cite{martin2018event,fan2019strategies,yao2019plan}, a sequence of images \cite{chandu2019storyboarding} or character roles \cite{liu2020character} have also been considered." />
        </attvalues>
      </edge>
      <edge source="2005.05255" target="1803.02893" id="2264">
        <attvalues>
          <attvalue for="5" value=" This contrasts with prior work \cite{logeswaran2018an} where the need to learn token-level representations limited the number of candidate next sentences that could be considered to a few hundred." />
        </attvalues>
      </edge>
      <edge source="2005.05255" target="1506.06726" id="2265">
        <attvalues>
          <attvalue for="5" value=" We also show preliminary results on the efficacy of our method for ranking candidate next sentence on the Toronto Book Corpus~\cite{kiros2015skip}, a much larger book dataset." />
        </attvalues>
      </edge>
      <edge source="2406.18501" target="2005.14165" id="2277">
        <attvalues>
          <attvalue for="5" value=" Yet recent pre-trained large language models (LLMs) have shown the capacity for in-context learning (ICL): they adapt to specific tasks with a few demonstration-answer pairs provided as prompts in the context window without any parameter updates \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2406.18501" target="2208.01066" id="2278">
        <attvalues>
          <attvalue for="5" value=" \cite{garg2022can}, \cite{zhang2023trained}, and \cite{ahn2024transformers} have shown that standard Transformers \cite{vaswani2017attention} can be trained to implement learning algorithms for linear regressions under ICL-based training objectives." />
        </attvalues>
      </edge>
      <edge source="2406.18501" target="2306.09927" id="2279">
        <attvalues>
          <attvalue for="5" value=" \cite{garg2022can}, \cite{zhang2023trained}, and \cite{ahn2024transformers} have shown that standard Transformers \cite{vaswani2017attention} can be trained to implement learning algorithms for linear regressions under ICL-based training objectives." />
        </attvalues>
      </edge>
      <edge source="2406.18501" target="2306.00297" id="2280">
        <attvalues>
          <attvalue for="5" value=" \cite{garg2022can}, \cite{zhang2023trained}, and \cite{ahn2024transformers} have shown that standard Transformers \cite{vaswani2017attention} can be trained to implement learning algorithms for linear regressions under ICL-based training objectives." />
        </attvalues>
      </edge>
      <edge source="2406.18501" target="1706.03762" id="2281">
        <attvalues>
          <attvalue for="5" value=" \cite{garg2022can}, \cite{zhang2023trained}, and \cite{ahn2024transformers} have shown that standard Transformers \cite{vaswani2017attention} can be trained to implement learning algorithms for linear regressions under ICL-based training objectives." />
        </attvalues>
      </edge>
      <edge source="2406.18501" target="2212.07677" id="2282">
        <attvalues>
          <attvalue for="5" value=" \cite{pmlr-v202-von-oswald23a} have demonstrated that Transformer models, with appropriate choices of parameters, can process in-context demonstrations in a way that is functionally equivalent to performing gradient updates on the same demonstration examples." />
        </attvalues>
      </edge>
      <edge source="2406.18501" target="2212.10559" id="2283">
        <attvalues>
          <attvalue for="5" value=" \cite{dai2023metaoptimization} provided a mathematical construction showing the dual form between Transformer attention and gradient descent and interpreted ICL as a meta-optimization process that performs implicit fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2409.11114" target="2205.07208" id="2284">
        <attvalues>
          <attvalue for="5" value=" Intent recognition~\cite{zhang2022fine}, a key element of these systems, is crucial for enabling automated assistance to address customer needs." />
        </attvalues>
      </edge>
      <edge source="2409.11114" target="2303.08774" id="2287">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in large language models (LLMs)~\cite{touvron2023llama,chiang2023vicuna,openai2023gpt4, feng2023towards} have significantly improved the detection of semantically distinct out-of-distribution (far-OOD) intents~\cite{liu2023good}." />
        </attvalues>
      </edge>
      <edge source="2409.11114" target="2106.03004" id="2288">
        <attvalues>
          <attvalue for="5" value=" However, identifying semantically similar (near-OOD) intents continues to pose significant challenges~\cite{fort2021exploring, liu2023good}." />
        </attvalues>
      </edge>
      <edge source="2409.11114" target="2010.13009" id="2289">
        <attvalues>
          <attvalue for="5" value=" Obtaining discriminative information from limited ID examples for OOD detection is inherently challenging~\cite{zhang2020discriminative}." />
        </attvalues>
      </edge>
      <edge source="2409.11114" target="2108.04106" id="2291">
        <attvalues>
          <attvalue for="5" value=" Drawing inspiration from channel models~\cite{min2021noisy, brown1993mathematics} and prompt tuning~\cite{lester2021power}, we feed into the LLMs the ID class names preceded by a series of learnable continuous prompt embeddings to generate a semantic prototype for each ID class." />
        </attvalues>
      </edge>
      <edge source="2409.11114" target="2104.08691" id="2292">
        <attvalues>
          <attvalue for="5" value=" Drawing inspiration from channel models~\cite{min2021noisy, brown1993mathematics} and prompt tuning~\cite{lester2021power}, we feed into the LLMs the ID class names preceded by a series of learnable continuous prompt embeddings to generate a semantic prototype for each ID class." />
        </attvalues>
      </edge>
      <edge source="2409.11114" target="2007.05566" id="2293">
        <attvalues>
          <attvalue for="5" value=" Furthermore, to ensure a wider variety of class representations—a factor known to improve OOD detection~\cite{winkens2020contrastive}—we adopt a diversified learning strategy aimed at reducing the mutual information between classes." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="1805.02023" id="2294">
        <attvalues>
          <attvalue for="5" value=" \cite{zhang-yang-2018-chinese} proposed a similar idea for Chinese name entity recognition." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="1906.01617" id="2295">
        <attvalues>
          <attvalue for="5" value=" \cite{sperber-etal-2019-self,xiao-etal-2019-lattice,zhang-etal-2019-lattice} proposed extensions to enable the transformer model~\cite{vaswani2017attention} to consume lattice inputs for machine translation." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="1906.01282" id="2296">
        <attvalues>
          <attvalue for="5" value=" \cite{sperber-etal-2019-self,xiao-etal-2019-lattice,zhang-etal-2019-lattice} proposed extensions to enable the transformer model~\cite{vaswani2017attention} to consume lattice inputs for machine translation." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="1706.03762" id="2298">
        <attvalues>
          <attvalue for="5" value=" \cite{sperber-etal-2019-self,xiao-etal-2019-lattice,zhang-etal-2019-lattice} proposed extensions to enable the transformer model~\cite{vaswani2017attention} to consume lattice inputs for machine translation." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="2011.00780" id="2299">
        <attvalues>
          <attvalue for="5" value=" \cite{huang2019adapting} proposed to adapt the transformer model originally pre-trained on written texts to consume lattices in order to improve SLU performance." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="1803.05071" id="2300">
        <attvalues>
          <attvalue for="5" value=" \cite{buckman-neubig-2018-neural} also found that utilizing lattices that represent multiple granularities of sentences can improve language modeling." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="1802.05365" id="2301">
        <attvalues>
          <attvalue for="5" value=" With recent introduction of large pre-trained language models (LMs) such as ELMo~\cite{peters-etal-2018-deep}, GPT~\cite{Radford2018ImprovingLU} and BERT~\cite{devlin-etal-2019-bert}, we have observed huge improvements on natural language understanding tasks." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="1801.06146" id="2303">
        <attvalues>
          <attvalue for="5" value=" It has been shown that fine-tuning the pre-trained language models on the data from the target tasks can mitigate the domain mismatch problem~\cite{howard-ruder-2018-universal,chronopoulou-etal-2019-embarrassingly}." />
        </attvalues>
      </edge>
      <edge source="2007.02629" target="1811.05370" id="2305">
        <attvalues>
          <attvalue for="5" value=" \cite{siddhant2018unsupervised} focused on pre-training a language model specifically for spoken content with huge amount of automatic transcripts, which requires a large collection of in-domain speech." />
        </attvalues>
      </edge>
      <edge source="2102.04754" target="1706.03762" id="2306">
        <attvalues>
          <attvalue for="5" value=" Positional encoding layers \cite{aiayn,cssl} are used to further augment the self-attention layers with sequence order information." />
        </attvalues>
      </edge>
      <edge source="2102.04754" target="1905.04226" id="2307">
        <attvalues>
          <attvalue for="5" value=" The highly complex neural architecture design of Transformers often leads to a large increase in the overall system complexity, for example, up to hundreds of millions of parameters \cite{lmwdt}." />
        </attvalues>
      </edge>
      <edge source="2102.04754" target="1601.06733" id="2308">
        <attvalues>
          <attvalue for="5" value=" The Transformer model architecture features a deep stacking of multiple self-attention layers \cite{lstmnml,assse,adamnl} with residual connections \cite{drlir} and layer normalization \cite{ln} to learn long-range contexts." />
        </attvalues>
      </edge>
      <edge source="2102.04754" target="1703.03130" id="2309">
        <attvalues>
          <attvalue for="5" value=" The Transformer model architecture features a deep stacking of multiple self-attention layers \cite{lstmnml,assse,adamnl} with residual connections \cite{drlir} and layer normalization \cite{ln} to learn long-range contexts." />
        </attvalues>
      </edge>
      <edge source="2102.04754" target="1606.01933" id="2310">
        <attvalues>
          <attvalue for="5" value=" The Transformer model architecture features a deep stacking of multiple self-attention layers \cite{lstmnml,assse,adamnl} with residual connections \cite{drlir} and layer normalization \cite{ln} to learn long-range contexts." />
        </attvalues>
      </edge>
      <edge source="2102.04754" target="1506.02142" id="2314">
        <attvalues>
          <attvalue for="5" value=" However, it lacks of a mathematically well-defined framework \cite{dbarmudl,blamnnu}." />
        </attvalues>
      </edge>
      <edge source="2102.04754" target="1812.03973" id="2315">
        <attvalues>
          <attvalue for="5" value=" In contrast, the only previous research on Bayesian Transformer \cite{blamnnu,blt} was conducted on machine translation and probabilistic programming tasks." />
        </attvalues>
      </edge>
      <edge source="2310.12778" target="2009.07118" id="2325">
        <attvalues>
          <attvalue for="5" value=" In recent years, we have seen many promising applications of prompt-based learning for text classification \cite{schick-schutze-2021-just,wang-etal-2022-towards-unified,zhang-etal-2022-prompt-based,hu-etal-2022-knowledgeable}." />
        </attvalues>
      </edge>
      <edge source="2310.12778" target="2108.02035" id="2327">
        <attvalues>
          <attvalue for="5" value=" In recent years, we have seen many promising applications of prompt-based learning for text classification \cite{schick-schutze-2021-just,wang-etal-2022-towards-unified,zhang-etal-2022-prompt-based,hu-etal-2022-knowledgeable}." />
        </attvalues>
      </edge>
      <edge source="2310.12778" target="2107.13586" id="2328">
        <attvalues>
          <attvalue for="5" value=" Next, it asks a language model (LM) to fill in the slots and then translate what the model fills to be a predicted class \cite{liu2023pre}." />
        </attvalues>
      </edge>
      <edge source="2310.12778" target="2204.06305" id="2329">
        <attvalues>
          <attvalue for="5" value=" In contrast, \cite{wang-etal-2022-automatic} proposed AMuLaP, which represents each class with a set of words, automatically derived from those predicted by the LM for training examples." />
        </attvalues>
      </edge>
      <edge source="2310.12778" target="2212.06950" id="2330">
        <attvalues>
          <attvalue for="5" value=" \cite{zhao-etal-2023-pre} proposed NPPrompt, which represents each class using a set of tokens with the highest embedding similarity to the manual class label." />
        </attvalues>
      </edge>
      <edge source="2312.07751" target="2004.10151" id="2370">
        <attvalues>
          <attvalue for="5" value=" Progress in human-centered NLP research has established the importance of modeling human and social factors, presenting a compelling argument that learning language from linguistic signals alone is not adequate \cite{hovy_social_2018, bisk_experience_2020, flek_returning_2020}, and noting that feelings, knowledge and mental states of the speaker and listener referred to as the ``Theory of Mind'' \cite{Flavell2004TheoryofMindDR}, along with other social context variables are vital to language understanding \cite{bisk_experience_2020, hovy_importance_2021}." />
        </attvalues>
      </edge>
      <edge source="2312.07751" target="1909.04985" id="2371">
        <attvalues>
          <attvalue for="5" value=" This need is backed by a wealth of empirical evidence demonstrating the benefits of modeling human and social factors \cite{volkova_exploring_2013, hu_exploiting_2013, bamman_contextualized_2015, lynn_human_2017, radfar_characterizing_2020}, and personalized models \cite{delasalles_learning_2019, jaech-ostendorf-2018-personalized, king-cook-2020-evaluating, welch_exploring_2020}." />
        </attvalues>
      </edge>
      <edge source="2312.07751" target="1706.03762" id="2372">
        <attvalues>
          <attvalue for="5" value=" In parallel, with the advent of Transformers \cite{vaswani2017attention}, there have been many advances in language modeling \cite{devlin-etal-2019-bert, dai-etal-2019-transformer, liu2019roberta, radford2019language} yielding Transformer-based large language models (LLMs) as the base of most current NLP systems." />
        </attvalues>
      </edge>
      <edge source="2312.07751" target="2303.14524" id="2375">
        <attvalues>
          <attvalue for="5" value=" {Recently, such user-centric prompting has been employed for personalized recommender systems~\cite{doddapaneni-etal-2023-towards}, dialog systems \cite{gao2023chat}, and measuring political biases and fairness \cite{feng-etal-2023-pretraining}." />
        </attvalues>
      </edge>
      <edge source="2312.07751" target="2305.08283" id="2376">
        <attvalues>
          <attvalue for="5" value=" {Recently, such user-centric prompting has been employed for personalized recommender systems~\cite{doddapaneni-etal-2023-towards}, dialog systems \cite{gao2023chat}, and measuring political biases and fairness \cite{feng-etal-2023-pretraining}." />
        </attvalues>
      </edge>
      <edge source="2312.07751" target="2109.03910" id="2377">
        <attvalues>
          <attvalue for="5" value="} Models such as GPT-3 and ChatGPT demonstrate potential for simulating some forms of human context, especially in generative tasks~\cite{reif-etal-2022-recipe}." />
        </attvalues>
      </edge>
      <edge source="1706.04902" target="1310.4546" id="2379">
        <attvalues>
          <attvalue for="5" value=" In recent years, (monolingual) vector representations of words, so-called word embeddings \cite{Mikolov2013a,Pennington2014} have proven extremely useful across a wide range of natural language processing (NLP) applications." />
        </attvalues>
      </edge>
      <edge source="2306.05083" target="1811.08008" id="2380">
        <attvalues>
          <attvalue for="5" value=" Sentence encoders (SEs) are fundamental building blocks in miscellaneous natural language processing (NLP) tasks, including natural language inference, paraphrase identification, and retrieval~\cite{gillick2018end, lan2018neural}." />
        </attvalues>
      </edge>
      <edge source="2306.05083" target="2104.08821" id="2381">
        <attvalues>
          <attvalue for="5" value=" Prior works mostly resort to a few hand-picked examples to illustrate what kind of sentence pairs an SE would consider similar or dissimilar~\cite{gao2021simcse,chuang-etal-2022-diffcse,wang2022sncse}." />
        </attvalues>
      </edge>
      <edge source="2306.05083" target="2204.10298" id="2382">
        <attvalues>
          <attvalue for="5" value=" Prior works mostly resort to a few hand-picked examples to illustrate what kind of sentence pairs an SE would consider similar or dissimilar~\cite{gao2021simcse,chuang-etal-2022-diffcse,wang2022sncse}." />
        </attvalues>
      </edge>
      <edge source="2306.05083" target="2201.05979" id="2383">
        <attvalues>
          <attvalue for="5" value=" Prior works mostly resort to a few hand-picked examples to illustrate what kind of sentence pairs an SE would consider similar or dissimilar~\cite{gao2021simcse,chuang-etal-2022-diffcse,wang2022sncse}." />
        </attvalues>
      </edge>
      <edge source="2306.05083" target="1804.07998" id="2384">
        <attvalues>
          <attvalue for="5" value=" Adversarial samples in NLP are constructed by replacing some words in an original sentence with some other words~\cite{alzantot-etal-2018-generating}, and the original sentence and the adversarial sample will have high lexical overlaps." />
        </attvalues>
      </edge>
      <edge source="2306.05083" target="2004.01970" id="2385">
        <attvalues>
          <attvalue for="5" value=" SEs are often adopted to check the semantic similarity between the original sentence and the adversarial sample~\cite{garg-ramakrishnan-2020-bae,li-etal-2020-bert-attack}." />
        </attvalues>
      </edge>
      <edge source="2306.05083" target="2004.09984" id="2386">
        <attvalues>
          <attvalue for="5" value=" SEs are often adopted to check the semantic similarity between the original sentence and the adversarial sample~\cite{garg-ramakrishnan-2020-bae,li-etal-2020-bert-attack}." />
        </attvalues>
      </edge>
      <edge source="2306.05083" target="1912.01673" id="2387">
        <attvalues>
          <attvalue for="5" value=" While some prior works also crafted sentence pairs to understand the performance of SEs, they either do not make the datasets publicly available~\cite{zhu2018exploring,zhu-de-melo-2020-sentence} or do not consider so many SEs as our paper does~\cite{barancikova-bojar-2020-costra}, especially unsupervised SEs." />
        </attvalues>
      </edge>
      <edge source="2009.00901" target="1611.01734" id="2389">
        <attvalues>
          <attvalue for="5" value=" \cite{dozat2016deep} propose a simple yet effective deep biaffine graph-based parser and achieve the state-of-the-art accuracy on a variety of datasets and languages." />
        </attvalues>
      </edge>
      <edge source="2101.10649" target="1911.02116" id="2390">
        <attvalues>
          <attvalue for="5" value=" We experiment with XLM-RoBERTa (XLM-R)~\cite{xlm-r}, a large XLM model with 550 million parameters and a 250k vocabulary size by extending semantic textual similarity, SQuAD~\cite{squad} \&amp; KorQuAD~\cite{korquad} question answering, and sentiment classifications for various cross-lingual settings." />
        </attvalues>
      </edge>
      <edge source="2101.10649" target="1901.07291" id="2391">
        <attvalues>
          <attvalue for="5" value=" In cross-lingual language understanding, XLM by Conneau \&amp; Lample~\cite{xlm}, despite being pretrained by only masked language modeling (MLM), has reported the state-of-the-art on downstream benchmarks." />
        </attvalues>
      </edge>
      <edge source="2101.10649" target="1606.05250" id="2392">
        <attvalues>
          <attvalue for="5" value=" We experiment with XLM-RoBERTa (XLM-R)~\cite{xlm-r}, a large XLM model with 550 million parameters and a 250k vocabulary size by extending semantic textual similarity, SQuAD~\cite{squad} \&amp; KorQuAD~\cite{korquad} question answering, and sentiment classifications for various cross-lingual settings." />
        </attvalues>
      </edge>
      <edge source="2101.10649" target="1909.07005" id="2393">
        <attvalues>
          <attvalue for="5" value=" We experiment with XLM-RoBERTa (XLM-R)~\cite{xlm-r}, a large XLM model with 550 million parameters and a 250k vocabulary size by extending semantic textual similarity, SQuAD~\cite{squad} \&amp; KorQuAD~\cite{korquad} question answering, and sentiment classifications for various cross-lingual settings." />
        </attvalues>
      </edge>
      <edge source="2101.10649" target="1902.00508" id="2394">
        <attvalues>
          <attvalue for="5" value=" At last, beyond previous work that has attempted to align word embeddings across different languages~\cite{w-mapping}, we compute a projection that directly maps sentence embeddings of one language to those of another." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1905.05950" id="2396">
        <attvalues>
          <attvalue for="5" value=" Probing studies have found that masked language models (MLMs) contain hierarchical representations \cite{tenney2019bert,hewitt2019structural,clark-etal-2019-bert}, while behavioral studies of recurrent neural language models \cite{linzen2016assessing,marvin2018targeted,wilcox2018rnn,vanschijndel2019quantity} and MLMs \cite{goldberg2019bert,hu2020systematic} have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1906.04341" id="2397">
        <attvalues>
          <attvalue for="5" value=" Probing studies have found that masked language models (MLMs) contain hierarchical representations \cite{tenney2019bert,hewitt2019structural,clark-etal-2019-bert}, while behavioral studies of recurrent neural language models \cite{linzen2016assessing,marvin2018targeted,wilcox2018rnn,vanschijndel2019quantity} and MLMs \cite{goldberg2019bert,hu2020systematic} have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1611.01368" id="2398">
        <attvalues>
          <attvalue for="5" value=" Probing studies have found that masked language models (MLMs) contain hierarchical representations \cite{tenney2019bert,hewitt2019structural,clark-etal-2019-bert}, while behavioral studies of recurrent neural language models \cite{linzen2016assessing,marvin2018targeted,wilcox2018rnn,vanschijndel2019quantity} and MLMs \cite{goldberg2019bert,hu2020systematic} have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1808.09031" id="2399">
        <attvalues>
          <attvalue for="5" value=" Probing studies have found that masked language models (MLMs) contain hierarchical representations \cite{tenney2019bert,hewitt2019structural,clark-etal-2019-bert}, while behavioral studies of recurrent neural language models \cite{linzen2016assessing,marvin2018targeted,wilcox2018rnn,vanschijndel2019quantity} and MLMs \cite{goldberg2019bert,hu2020systematic} have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1909.00111" id="2400">
        <attvalues>
          <attvalue for="5" value=" Probing studies have found that masked language models (MLMs) contain hierarchical representations \cite{tenney2019bert,hewitt2019structural,clark-etal-2019-bert}, while behavioral studies of recurrent neural language models \cite{linzen2016assessing,marvin2018targeted,wilcox2018rnn,vanschijndel2019quantity} and MLMs \cite{goldberg2019bert,hu2020systematic} have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1901.05287" id="2401">
        <attvalues>
          <attvalue for="5" value=" Probing studies have found that masked language models (MLMs) contain hierarchical representations \cite{tenney2019bert,hewitt2019structural,clark-etal-2019-bert}, while behavioral studies of recurrent neural language models \cite{linzen2016assessing,marvin2018targeted,wilcox2018rnn,vanschijndel2019quantity} and MLMs \cite{goldberg2019bert,hu2020systematic} have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1907.11692" id="2404">
        <attvalues>
          <attvalue for="5" value=" Recent evidence suggests that MLMs like BERT \cite{devlin2019bert} and RoBERTa \cite{liu2019roberta} can learn to make hierarchical linguistic generalizations through exposure to text \cite{warstadt2020linguistic}, though acquiring many of these linguistic generalizations requires large amounts of data \cite{warstadt2020learning}." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="2007.06761" id="2405">
        <attvalues>
          <attvalue for="5" value=" This finding presents additional evidence to \cite{warstadt2020learning} and \cite{warstadt2020linguistic} for the learnability of hierarchical syntactic information from natural language text input." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="2010.05358" id="2406">
        <attvalues>
          <attvalue for="5" value=" This finding presents additional evidence to \cite{warstadt2020learning} and \cite{warstadt2020linguistic} for the learnability of hierarchical syntactic information from natural language text input." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1802.09091" id="2407">
        <attvalues>
          <attvalue for="5" value=" \cite{mccoy2018poverty} evaluate non-pre-trained recurrent sequence-to-sequence (seq2seq) models \cite{sutskever2014sequence} on the question formation task, finding that they rely on linear/positional surface heuristics rather than hierarchical structure to perform this syntactic transformation." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="1409.3215" id="2408">
        <attvalues>
          <attvalue for="5" value=" \cite{mccoy2018poverty} evaluate non-pre-trained recurrent sequence-to-sequence (seq2seq) models \cite{sutskever2014sequence} on the question formation task, finding that they rely on linear/positional surface heuristics rather than hierarchical structure to perform this syntactic transformation." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="2109.12036" id="2409">
        <attvalues>
          <attvalue for="5" value=" More recent studies have also exclusively considered recurrent seq2seq models and Transformer models \cite{petty2021transformers} trained from scratch on other transformations like tense reinflection \cite{mccoy2020trees} and passivization \cite{mulligan2021structure}, finding similar results." />
        </attvalues>
      </edge>
      <edge source="2203.09397" target="2001.03632" id="2410">
        <attvalues>
          <attvalue for="5" value=" More recent studies have also exclusively considered recurrent seq2seq models and Transformer models \cite{petty2021transformers} trained from scratch on other transformations like tense reinflection \cite{mccoy2020trees} and passivization \cite{mulligan2021structure}, finding similar results." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="1803.11138" id="2411">
        <attvalues>
          <attvalue for="5" value=" Previous investigations of recurrent neural architectures have yielded some evidence for hierarchically-governed linguistic knowledge \cite{gulordava2018colorless, Marvin-2018-targeted,hu-etal-2020-systematic}." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="1810.09536" id="2413">
        <attvalues>
          <attvalue for="5" value=" Architecturally-constrained models when trained without explicit information about syntactic structure show only modest benefits \cite{shen2018ordered,kim-etal-2019-unsupervised,merrill-etal-2019-finding}." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="1904.03746" id="2414">
        <attvalues>
          <attvalue for="5" value=" Architecturally-constrained models when trained without explicit information about syntactic structure show only modest benefits \cite{shen2018ordered,kim-etal-2019-unsupervised,merrill-etal-2019-finding}." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="2001.03632" id="2415">
        <attvalues>
          <attvalue for="5" value=" \cite{mccoytrees} take a different tack: the training data is carefully controlled so that hierarchical behavior can emerge only if a model itself is biased to extract hierarchical generalizations." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="1706.03762" id="2416">
        <attvalues>
          <attvalue for="5" value=" The recently developed Transformer architecture has led to revolutionary advances across many areas of natural language processing, including machine translation and question answering~\cite{vaswani2017attention, devlin2019bert}." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="2002.12327" id="2417">
        <attvalues>
          <attvalue for="5" value=" Transformer-based models have also shown considerable success on benchmarks that appear to require the representation of hierarchical abstractions \cite{rogers2021primer,goldberg2019assessing, warstadt2019investigating}." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="1901.05287" id="2418">
        <attvalues>
          <attvalue for="5" value=" Transformer-based models have also shown considerable success on benchmarks that appear to require the representation of hierarchical abstractions \cite{rogers2021primer,goldberg2019assessing, warstadt2019investigating}." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="2007.06761" id="2419">
        <attvalues>
          <attvalue for="5" value=" Yet, for the reasons noted above, it is difficult to conclude much about the inductive bias in the Transformer: they are trained on vast datasets, leaving open the question of the impact of inductive bias as opposed to training data (\cite{warstadt2020can}, but see \cite{van2019quantity} for arguments that even massive data may not be sufficient)." />
        </attvalues>
      </edge>
      <edge source="2109.12036" target="1909.00111" id="2420">
        <attvalues>
          <attvalue for="5" value=" Yet, for the reasons noted above, it is difficult to conclude much about the inductive bias in the Transformer: they are trained on vast datasets, leaving open the question of the impact of inductive bias as opposed to training data (\cite{warstadt2020can}, but see \cite{van2019quantity} for arguments that even massive data may not be sufficient)." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2002.12327" id="2422">
        <attvalues>
          <attvalue for="5" value=" Since the advent of pre-trained language models \cite{devlin-etal-2019-bert, Liu2019RoBERTaAR,rogers-etal-2020-primer}, NLP has witnessed revolutionary advancements over the years." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2205.11081" id="2424">
        <attvalues>
          <attvalue for="5" value=" This leads to the development of various pre-trained language models specialized in the Bengali language, such as BanglaBERT \cite{bhattacharjee-etal-2022-banglabert}, BanglaT5 \cite{bhattacharjee-etal-2023-banglanlg}, and etc These models have demonstrated exciting progress in many downstream Bengali NLP tasks \cite{ekram-etal-2022-banglarqa, akash-etal-2023-shironaam}." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2004.09095" id="2425">
        <attvalues>
          <attvalue for="5" value=" However, one major concern for these pre-trained models is that they require fine-tuning using domain-specific large annotated datasets, which is challenging for Bengali due to its under-representation in the NLP domain \cite{joshi-etal-2020-state, Chakraborty_Nayeem_Ahmad_2021, chowdhury-etal-2021-unsupervised} despite being the sixth most spoken language in the world with over {300} million native speakers \cite{bengali-language-article}." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2005.14165" id="2426">
        <attvalues>
          <attvalue for="5" value=" Recent developments in large language models (LLMs) \cite{Brown2020LanguageMA, Shoeybi2019MegatronLMTM, Rae2021ScalingLM, Zhang2022OPTOP} have transformed the landscape in NLP." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2307.09288" id="2428">
        <attvalues>
          <attvalue for="5" value=" ChatGPT and other recently proposed LLMs like PaLM-2, Claude-2, LLaMA-2-chat \cite{touvron2023llama2, anil2023palm2, jahan2024comprehensive} are trained via leveraging this RLHF technique to mitigate various limitations of the previous generation LLMs and gained widespread popularity." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2305.18486" id="2429">
        <attvalues>
          <attvalue for="5" value=" Though ChatGPT like LLMs has demonstrated strong zero-shot performance in various NLP tasks in English \cite{laskar-etal-2023-systematic,laskar-etal-2023-building} and some other languages \cite{Lai2023ChatGPTBE} and domains \cite{jahan-etal-2023-evaluation,jahan2024comprehensive,fu2024tiny}, these LLMs are yet to be investigated in the widely spoken, yet modest-resourced, Bengali language domain." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2304.05613" id="2431">
        <attvalues>
          <attvalue for="5" value=" Though ChatGPT like LLMs has demonstrated strong zero-shot performance in various NLP tasks in English \cite{laskar-etal-2023-systematic,laskar-etal-2023-building} and some other languages \cite{Lai2023ChatGPTBE} and domains \cite{jahan-etal-2023-evaluation,jahan2024comprehensive,fu2024tiny}, these LLMs are yet to be investigated in the widely spoken, yet modest-resourced, Bengali language domain." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2306.04504" id="2432">
        <attvalues>
          <attvalue for="5" value=" Though ChatGPT like LLMs has demonstrated strong zero-shot performance in various NLP tasks in English \cite{laskar-etal-2023-systematic,laskar-etal-2023-building} and some other languages \cite{Lai2023ChatGPTBE} and domains \cite{jahan-etal-2023-evaluation,jahan2024comprehensive,fu2024tiny}, these LLMs are yet to be investigated in the widely spoken, yet modest-resourced, Bengali language domain." />
        </attvalues>
      </edge>
      <edge source="2309.13173" target="2310.04270" id="2433">
        <attvalues>
          <attvalue for="5" value=" Though ChatGPT like LLMs has demonstrated strong zero-shot performance in various NLP tasks in English \cite{laskar-etal-2023-systematic,laskar-etal-2023-building} and some other languages \cite{Lai2023ChatGPTBE} and domains \cite{jahan-etal-2023-evaluation,jahan2024comprehensive,fu2024tiny}, these LLMs are yet to be investigated in the widely spoken, yet modest-resourced, Bengali language domain." />
        </attvalues>
      </edge>
      <edge source="2310.05115" target="1908.09961" id="2436">
        <attvalues>
          <attvalue for="5" value=" To address this issue, disentangled representation learning (DRL) (\cite{do2019theory}; \cite{9947342}) has emerged as an approach to separate specific aspects into distinct representations." />
        </attvalues>
      </edge>
      <edge source="2310.05115" target="2012.09276" id="2437">
        <attvalues>
          <attvalue for="5" value=" To address this issue, disentangled representation learning (DRL) (\cite{do2019theory}; \cite{9947342}) has emerged as an approach to separate specific aspects into distinct representations." />
        </attvalues>
      </edge>
      <edge source="2310.05115" target="1606.03657" id="2438">
        <attvalues>
          <attvalue for="5" value=" DRL has shown promise in computer vision through methods such as InfoGAN \cite{chen2016infogan}, but its potential for NLP tasks has yet to be fully explored \cite{vishnubhotla2021evaluation}." />
        </attvalues>
      </edge>
      <edge source="2310.05115" target="1905.05950" id="2443">
        <attvalues>
          <attvalue for="5" value=" On the other hand, several studies have shown that linguistic knowledge like syntactic and semantic aspects are encoded in different layers of PLM (\cite{jawahar-etal-2019-bert}; \cite{tenney-etal-2019-bert}; \cite{hewitt-manning-2019-structural}; \cite{bommasani-etal-2020-interpreting})." />
        </attvalues>
      </edge>
      <edge source="2310.05115" target="1906.04284" id="2444">
        <attvalues>
          <attvalue for="5" value=" Furthermore, each self-attention head seems to focus on different aspect of language information (\cite{vig-belinkov-2019-analyzing}; \cite{clark2019does}; \cite{zhao-bethard-2020-berts})." />
        </attvalues>
      </edge>
      <edge source="2310.05115" target="1906.04341" id="2445">
        <attvalues>
          <attvalue for="5" value=" Furthermore, each self-attention head seems to focus on different aspect of language information (\cite{vig-belinkov-2019-analyzing}; \cite{clark2019does}; \cite{zhao-bethard-2020-berts})." />
        </attvalues>
      </edge>
      <edge source="2310.05115" target="2008.11608" id="2446">
        <attvalues>
          <attvalue for="5" value=" However, it is common practice to represent a word simply as the sum of the last four layers' hidden states even for embedding-based WSD (\cite{scarlini-etal-2020-contexts}; \cite{SensEmBERT}; \cite{loureiro2021analysis})." />
        </attvalues>
      </edge>
      <edge source="2108.08485" target="1707.06875" id="2447">
        <attvalues>
          <attvalue for="5" value=" Despite their widespread use, however, these automated metrics often poorly correlate with ratings given by human judges, particularly for datasets in which only a single human reference exists~\cite{gupta2019investigating,novikova-etal-2017-need}." />
        </attvalues>
      </edge>
      <edge source="2108.08485" target="1907.10568" id="2448">
        <attvalues>
          <attvalue for="5" value=" Despite their widespread use, however, these automated metrics often poorly correlate with ratings given by human judges, particularly for datasets in which only a single human reference exists~\cite{gupta2019investigating,novikova-etal-2017-need}." />
        </attvalues>
      </edge>
      <edge source="2108.08485" target="1701.03079" id="2449">
        <attvalues>
          <attvalue for="5" value=" Moreover, these automated metrics only capture similarities between generated sentences and reference candidates, crucially ignoring provided contexts that are relevant for evaluating the answer in contextual NLG tasks, such as story generation, news summarization, and question-answering~\cite{tao2018ruber,nema-khapra-2018-towards}." />
        </attvalues>
      </edge>
      <edge source="2108.08485" target="1808.10192" id="2450">
        <attvalues>
          <attvalue for="5" value=" Moreover, these automated metrics only capture similarities between generated sentences and reference candidates, crucially ignoring provided contexts that are relevant for evaluating the answer in contextual NLG tasks, such as story generation, news summarization, and question-answering~\cite{tao2018ruber,nema-khapra-2018-towards}." />
        </attvalues>
      </edge>
      <edge source="2108.08485" target="1904.02792" id="2454">
        <attvalues>
          <attvalue for="5" value=" For instance, in HUSE score, \cite{hashimoto-etal-2019-unifying} leverages the differences between perplexity and human judgements to consider both quality and diversity of generated text." />
        </attvalues>
      </edge>
      <edge source="2108.08485" target="2005.00456" id="2455">
        <attvalues>
          <attvalue for="5" value=" Another line has proposed training separate neural models to aid automated metrics~\cite[][inter alia]{mehri-eskenazi-2020-usr,yuma-etal-2020-ubleu}." />
        </attvalues>
      </edge>
      <edge source="2108.08485" target="2004.04696" id="2456">
        <attvalues>
          <attvalue for="5" value=" For instance, BLEURT~\cite{sellam-etal-2020-bleurt} fine-tunes BERT~\cite{devlin2019bert} on synthetic reference-candidate pairs for machine translation." />
        </attvalues>
      </edge>
      <edge source="2108.08485" target="2004.06063" id="2457">
        <attvalues>
          <attvalue for="5" value=" These methods, however, are often limited in practical use, because the high-cost human ratings are not always available for every dataset, and the data- or system-specific training is not easily extended to other domains~\cite{zhang2019bertscore}, \redit{and can even bias the evaluation~\cite{freitag-etal-2020-bleu}." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="2004.02105" id="2458">
        <attvalues>
          <attvalue for="5" value=" This includes, for instance, data selection \cite{aharoni-goldberg-2020-unsupervised}, data exploration \cite{voigt-etal-2022-keywordscape}, and neural topic modeling \cite{zhao2021_topic}." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="2103.00498" id="2459">
        <attvalues>
          <attvalue for="5" value=" This includes, for instance, data selection \cite{aharoni-goldberg-2020-unsupervised}, data exploration \cite{voigt-etal-2022-keywordscape}, and neural topic modeling \cite{zhao2021_topic}." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="1908.10084" id="2461">
        <attvalues>
          <attvalue for="5" value=" One such framework is Sentence Transformers \cite{reimers-gurevych-2019-sentence}, which is used by BERTopic." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="2104.08821" id="2462">
        <attvalues>
          <attvalue for="5" value=" The shift to embedding-based approaches is driven by the continuous development of neural language models, successfully used in natural language understanding (NLU) tasks such as semantic textual similarity \cite{reimers-gurevych-2019-sentence,gao-etal-2021-simcse} or retrieval and reranking \cite{huang2020_retrieval,yates-etal-2021-pretrained}." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="2006.11632" id="2463">
        <attvalues>
          <attvalue for="5" value=" The shift to embedding-based approaches is driven by the continuous development of neural language models, successfully used in natural language understanding (NLU) tasks such as semantic textual similarity \cite{reimers-gurevych-2019-sentence,gao-etal-2021-simcse} or retrieval and reranking \cite{huang2020_retrieval,yates-etal-2021-pretrained}." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="2010.06467" id="2464">
        <attvalues>
          <attvalue for="5" value=" The shift to embedding-based approaches is driven by the continuous development of neural language models, successfully used in natural language understanding (NLU) tasks such as semantic textual similarity \cite{reimers-gurevych-2019-sentence,gao-etal-2021-simcse} or retrieval and reranking \cite{huang2020_retrieval,yates-etal-2021-pretrained}." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="2210.07316" id="2465">
        <attvalues>
          <attvalue for="5" value=" The Massive Text Embedding Benchmark (MTEB, \cite{muennighoff-etal-2023-mteb}) provides such a benchmark for a wide range of embedding-based tasks (eg, classification, clustering, or reranking) and datasets from different domains (eg, online reviews, scientific publications, or social media)." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="2012.15613" id="2467">
        <attvalues>
          <attvalue for="5" value=" The inclusion of non-English data is important, as the performance of multilingual models may not equal their monolingual counterparts \cite{rust-etal-2021-good}, and as a means to evaluate the potentially strong cross-lingual transfer capability of multilingual models (eg, \cite{huang-etal-2019-unicoder})." />
        </attvalues>
      </edge>
      <edge source="2401.02709" target="1801.06146" id="2469">
        <attvalues>
          <attvalue for="5" value=" The idea of this additional training is to adapt language models, typically trained on large and heterogeneous data collections, to the data of a specific domain or task, and has been shown to improve performance on downstream tasks (eg, \cite{howard-ruder-2018-universal,lee2019-biobert,gururangan-etal-2020-dont})." />
        </attvalues>
      </edge>
      <edge source="2012.15613" target="1706.03762" id="2472">
        <attvalues>
          <attvalue for="5" value=" Following large transformer-based language models \cite[LMs,][]{Vaswani:2017} pretrained on large English corpora \cite[eg, BERT, RoBERTa, T5;][]{devlin:2019,liu:2019,Raffel:2020t5}, similar monolingual language models have been introduced for other languages \cite[inter alia]{virtanen:2019, antoun:2020, martin:2020}, offering previously unmatched performance in all NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2012.15613" target="1907.11692" id="2474">
        <attvalues>
          <attvalue for="5" value=" Following large transformer-based language models \cite[LMs,][]{Vaswani:2017} pretrained on large English corpora \cite[eg, BERT, RoBERTa, T5;][]{devlin:2019,liu:2019,Raffel:2020t5}, similar monolingual language models have been introduced for other languages \cite[inter alia]{virtanen:2019, antoun:2020, martin:2020}, offering previously unmatched performance in all NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2012.15613" target="1912.07076" id="2476">
        <attvalues>
          <attvalue for="5" value=" Monolingual BERT models are typically evaluated on downstream NLP tasks to demonstrate their effectiveness in comparison to previous monolingual models or mBERT \cite[inter alia]{virtanen:2019, antoun:2020, martin:2020}." />
        </attvalues>
      </edge>
      <edge source="2012.15613" target="1911.02116" id="2477">
        <attvalues>
          <attvalue for="5" value=" The common argument justifying the need for monolingual variants is the assumption that multilingual models---due to suffering from the so-called curse of multilinguality \cite[eg, the lack of capacity to represent all languages in an equitable way]{conneau:2020}---underperform monolingual models when applied to monolingual tasks \cite[inter alia]{virtanen:2019,antoun:2020,ronnqvist:2019}." />
        </attvalues>
      </edge>
      <edge source="2101.05716" target="1508.05326" id="2481">
        <attvalues>
          <attvalue for="5" value=" For English, several standard NLI datasets exist, such as SICK \cite{marelli-etal-2014-sick}, SNLI \cite{bowman-etal-2015-large} and MNLI \cite{williams-etal-2018-broad}." />
        </attvalues>
      </edge>
      <edge source="2101.05716" target="1704.05426" id="2482">
        <attvalues>
          <attvalue for="5" value=" For English, several standard NLI datasets exist, such as SICK \cite{marelli-etal-2014-sick}, SNLI \cite{bowman-etal-2015-large} and MNLI \cite{williams-etal-2018-broad}." />
        </attvalues>
      </edge>
      <edge source="2101.05716" target="2001.06286" id="2486">
        <attvalues>
          <attvalue for="5" value=" Indeed, Dutch is in the scope of the multilingual BERT models published by Google \cite{devlin-etal-2019-bert}, and two monolingual Dutch BERT models have been published as part of HuggingFace's transformers library \cite{de2019bertje,delobelle2020robbert}." />
        </attvalues>
      </edge>
      <edge source="2101.05716" target="cs/0306050" id="2487">
        <attvalues>
          <attvalue for="5" value=" There is a Named Entity Recognition task coming from the CoNLL-2003 shared task \cite{tjong2003introduction}; from a one million word hand annotated subcorpus of SONAR \cite{Oostdijk2013} one derives part-of-speech tagging, Named Entity Recognition and Semantic Role Labelling tasks." />
        </attvalues>
      </edge>
      <edge source="2101.05716" target="2001.02943" id="2488">
        <attvalues>
          <attvalue for="5" value=" Moreover, \cite{allein2020binary} introduce a classification task where a model needs to distinguish between the pronouns die and dat." />
        </attvalues>
      </edge>
      <edge source="2405.11106" target="1706.03762" id="2489">
        <attvalues>
          <attvalue for="5" value=" NLP has been an active research topic for decades and many famous models have been proposed for language modeling such as Recurrent Neural Network~(RNN)~\cite{rumelhart1986learning, jordan1997serial}, Long-Short Term Memory networks~(LSTM)~\cite{hochreiter1997long}, and transformers~\cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2405.11106" target="2303.05069" id="2490">
        <attvalues>
          <attvalue for="5" value=" In recent years, the integration of NLP with single-agent RL has led to the development of language-conditioned RL frameworks~\cite{peng2023conceptual, jiang2019language, zhou2021inverse}, especially as Large Language Models~(LLMs)~\cite{openai2023chatgpt, touvron2023llama, chowdhery2023palm, team2023gemini} emerged as the rising star in the artificial intelligence community (see Fig." />
        </attvalues>
      </edge>
      <edge source="2405.11106" target="2403.19839" id="2491">
        <attvalues>
          <attvalue for="5" value="~\ref{fig:llm}) and has been successfully applied in various fields~\cite{wu2024new, lai2024language, han2024chainofinteraction}." />
        </attvalues>
      </edge>
      <edge source="2405.11106" target="2303.11366" id="2492">
        <attvalues>
          <attvalue for="5" value=" For example, in Reflexion~\cite{shinn2024reflexion}, the authors showed that the LLM agent could generate decent reflections on its decisions without any reward/feedback from the environment." />
        </attvalues>
      </edge>
      <edge source="2405.11106" target="2402.01680" id="2493">
        <attvalues>
          <attvalue for="5" value="~\cite{guo2024large} reviewed LLM-based multi-agent frameworks, but the emphasis of that paper was not on MARL." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="1706.03762" id="2495">
        <attvalues>
          <attvalue for="5" value=" In recent years, the performance of ASR technology has dramatically advanced, evolving from traditional Hidden Markov Model (HMM)-based architectures to modern end-to-end (E2E) systems like Listen, Attend and Spell (LAS) or RNN-T~\cite{chan2016listen, graves2014towards, amodei2016deep,vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2303.01037" id="2497">
        <attvalues>
          <attvalue for="5" value=" Large-scale models such as Whisper~\cite{radford2023robust} and Google USM~\cite{zhang2023google} have demonstrated state-of-the-art performance, leveraging vast amounts of labeled and unlabeled speech data, which can be costly to obtain." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2011.01991" id="2498">
        <attvalues>
          <attvalue for="5" value=" Recent research, however, has shown that E2E ASR models often learn an internal language model (ILM) on the training data, which can reduce the effectiveness of traditional shallow fusion techniques~\cite{meng2021internal}." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2002.11268" id="2499">
        <attvalues>
          <attvalue for="5" value=" Methods to address the impact of ILMs, such as those proposed by~\cite{mcdermott2019density, liu22j_interspeech, zeineldeen2021investigating}, generally involve code modification during the inference stage, therefore they are out of the scope of discussion in this paper." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="1910.10697" id="2501">
        <attvalues>
          <attvalue for="5" value="~\cite{hrinchuk2020correction} propose a Transformer-based architecture to ``translate'' an ASR model output into grammatically and semantically correct text." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2104.05507" id="2502">
        <attvalues>
          <attvalue for="5" value="~\cite{zhao2021bart} introduce a BART-based semantic correction system for the Mandarin ASR system." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2303.00456" id="2504">
        <attvalues>
          <attvalue for="5" value=" al~\cite{ma2023nbest, ma2023adapting} propose an N-best T5 model based on pre-trained T5 models to perform error correction using the ASR N-best list." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2303.13648" id="2505">
        <attvalues>
          <attvalue for="5" value=" Within the field of NLP, studies such as~\cite{wu2023chatgpt, fang2023chatgpt} have applied ChatGPT models to grammatical error correction tasks." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2307.04172" id="2506">
        <attvalues>
          <attvalue for="5" value="\cite{ma2023can} and Chen et al." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2309.15701" id="2508">
        <attvalues>
          <attvalue for="5" value="\cite{chen2024hyporadise} has integrated N-best lists with generative LLMs to enhance error correction performance." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2405.10025" id="2509">
        <attvalues>
          <attvalue for="5" value=" developed a multi-modal EC model incorporating audio as an additional input~\cite{hu2024listen} and used a cloze-test task approach instead of a generative correction method." />
        </attvalues>
      </edge>
      <edge source="2409.09554" target="2109.14420" id="2511">
        <attvalues>
          <attvalue for="5" value="\cite{leng2021fastcorrect} investigated non-autoregressive models with similar approaches." />
        </attvalues>
      </edge>
      <edge source="2005.01063" target="1905.05950" id="2512">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (LMs) have been shown to contain semantic \cite{tenney2019bert}, syntactic \cite{DBLP:journals/corr/abs-1901-05287, structural-probe, linzen2016agreement} and factual knowledge \cite{petroni2019language}, and to be great starting points for transfer-learning to new tasks via fine-tuning on few examples." />
        </attvalues>
      </edge>
      <edge source="2005.01063" target="1901.05287" id="2513">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (LMs) have been shown to contain semantic \cite{tenney2019bert}, syntactic \cite{DBLP:journals/corr/abs-1901-05287, structural-probe, linzen2016agreement} and factual knowledge \cite{petroni2019language}, and to be great starting points for transfer-learning to new tasks via fine-tuning on few examples." />
        </attvalues>
      </edge>
      <edge source="2005.01063" target="1909.01066" id="2514">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (LMs) have been shown to contain semantic \cite{tenney2019bert}, syntactic \cite{DBLP:journals/corr/abs-1901-05287, structural-probe, linzen2016agreement} and factual knowledge \cite{petroni2019language}, and to be great starting points for transfer-learning to new tasks via fine-tuning on few examples." />
        </attvalues>
      </edge>
      <edge source="2005.01063" target="1805.07398" id="2516">
        <attvalues>
          <attvalue for="5" value=" The distributional approach to TSE \cite{DBLP:conf/acl/Hindle90, DBLP:conf/kdd/PantelL02,DBLP:conf/emnlp/PantelCBPV09,setExpander,DBLP:conf/starsem/MahabalRM18} operates under the hypothesis that similar words appear in similar contexts \cite{DBLP:books/lib/Harris68}." />
        </attvalues>
      </edge>
      <edge source="2203.12815" target="1909.02857" id="2517">
        <attvalues>
          <attvalue for="5" value=" In an effort to explain these cross-lingual performance differences, researchers have proposed treebank size~\cite{vania-etal-2019-systematic}, linguistic variation~\cite{nivre-etal-2007-conll}, test data sentence length or average gold dependency length~\cite{mcdonald-nivre-2011-analyzing}, and domain differences between training and test data~\cite{foster-etal-2011-news}, as potential predictors." />
        </attvalues>
      </edge>
      <edge source="1909.02857" target="1705.00440" id="2518">
        <attvalues>
          <attvalue for="5" value=" Data augmentation is applicable in all scenarios, and has proven useful for low-resource NLP in general \cite{augmentation-mt:ACL2017,bergmanis-etAl:K17-2002,sahin-emnlp18}." />
        </attvalues>
      </edge>
      <edge source="1909.02857" target="1808.09055" id="2520">
        <attvalues>
          <attvalue for="5" value=" While this may intuitively seem to make cross-lingual training difficult, recent results have shown that lexical parameter sharing on characters and words can in fact improve cross-lingual parsing \cite{deLhoneux-emnlp18}; and that in some circumstances, a lexicalized parser can outperform a delexicalized one, even in a low-resource setting \cite{W17-6303}." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="2303.08774" id="2521">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) have demonstrated remarkable capabilities in a wide range of natural language processing tasks \cite{achiam2023gpt,touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="2307.09288" id="2522">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) have demonstrated remarkable capabilities in a wide range of natural language processing tasks \cite{achiam2023gpt,touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="2306.02561" id="2523">
        <attvalues>
          <attvalue for="5" value=" Due to the diversity of data sources, architectures and training methods, different LLMs have strengths and weaknesses in different tasks and contexts \cite{jiang2023llm}." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="1704.00109" id="2524">
        <attvalues>
          <attvalue for="5" value=" In addition to investing significant resources in training a superior LLM, ensembling multiple existing models is another effective way to break through the community performance ceiling \cite{huang2016snapshot}, especially given the current trend in the open source LLM community to contribute only model weights rather than training data and procedures \cite{allenai2023olmo}." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="2211.06088" id="2525">
        <attvalues>
          <attvalue for="5" value="\ref{tab1}, we selected several common CV models \cite{chen2022repghost, tan2019efficientnet,wang2021pvtv2} for ensembling and observed better accuracy on ImageNet \cite{deng2009imagenet} compared to using a single model." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="2305.13534" id="2526">
        <attvalues>
          <attvalue for="5" value=" Another advantage is that early errors in LLMs often snowball into later errors \cite{zhang2023language}." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="2306.02652" id="2528">
        <attvalues>
          <attvalue for="5" value=" To address this, CV classification used cascade inference \cite{jazbec2024towards,enomoro2021learning}, where a gate model passes a sample to a more powerful model only if its confidence falls below a threshold, thereby improving efficiency." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="2104.09286" id="2529">
        <attvalues>
          <attvalue for="5" value=" To address this, CV classification used cascade inference \cite{jazbec2024towards,enomoro2021learning}, where a gate model passes a sample to a more powerful model only if its confidence falls below a threshold, thereby improving efficiency." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="1706.04599" id="2530">
        <attvalues>
          <attvalue for="5" value=" To ensure that LLMs are also suitable as gate models, we measured the Expected Calibration Error (ECE) \cite{guo2017calibration} of CV models and LLMs on ImageNet and MMLU \cite{hendrycks2020measuring}, as shown in Tab." />
        </attvalues>
      </edge>
      <edge source="2406.12585" target="2009.03300" id="2531">
        <attvalues>
          <attvalue for="5" value=" To ensure that LLMs are also suitable as gate models, we measured the Expected Calibration Error (ECE) \cite{guo2017calibration} of CV models and LLMs on ImageNet and MMLU \cite{hendrycks2020measuring}, as shown in Tab." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="1802.05365" id="2546">
        <attvalues>
          <attvalue for="5" value=" The technique of fine-tuning a self-supervised language model has become ubiquitous in Natural Language Processing (NLP) because models trained in this way have advanced evaluation scores on many tasks~\cite{radford-etal-2018-improving,peters-etal-2018-deep,devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="1907.11692" id="2548">
        <attvalues>
          <attvalue for="5" value=" It has spawned many variants \cite{liu-etal-2019-roberta,lan-etal-2019-albert} and much analysis ~\cite{jawahar-etal-2019-bert,chi-etal-2020-finding,rogers-etal-2020-primer}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="1909.11942" id="2549">
        <attvalues>
          <attvalue for="5" value=" It has spawned many variants \cite{liu-etal-2019-roberta,lan-etal-2019-albert} and much analysis ~\cite{jawahar-etal-2019-bert,chi-etal-2020-finding,rogers-etal-2020-primer}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="2005.04511" id="2550">
        <attvalues>
          <attvalue for="5" value=" It has spawned many variants \cite{liu-etal-2019-roberta,lan-etal-2019-albert} and much analysis ~\cite{jawahar-etal-2019-bert,chi-etal-2020-finding,rogers-etal-2020-primer}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="2002.12327" id="2551">
        <attvalues>
          <attvalue for="5" value=" It has spawned many variants \cite{liu-etal-2019-roberta,lan-etal-2019-albert} and much analysis ~\cite{jawahar-etal-2019-bert,chi-etal-2020-finding,rogers-etal-2020-primer}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="1912.07076" id="2553">
        <attvalues>
          <attvalue for="5" value=" While there is evidence to suggest that dedicated monolingual models can be superior to a multilingual model for within-language downstream tasks \cite{devries2019bertje,virtanen-etal-2019-multilingual,farahani-etal-2020-parsbert}, other studies suggest that a multilingual model such as mBERT is a good choice for low-resourced languages \cite{wu-dredze-2020-languages,rust-etal-2020-how,chau-etal-2020-parsing}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="2005.12515" id="2554">
        <attvalues>
          <attvalue for="5" value=" While there is evidence to suggest that dedicated monolingual models can be superior to a multilingual model for within-language downstream tasks \cite{devries2019bertje,virtanen-etal-2019-multilingual,farahani-etal-2020-parsbert}, other studies suggest that a multilingual model such as mBERT is a good choice for low-resourced languages \cite{wu-dredze-2020-languages,rust-etal-2020-how,chau-etal-2020-parsing}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="2005.09093" id="2555">
        <attvalues>
          <attvalue for="5" value=" While there is evidence to suggest that dedicated monolingual models can be superior to a multilingual model for within-language downstream tasks \cite{devries2019bertje,virtanen-etal-2019-multilingual,farahani-etal-2020-parsbert}, other studies suggest that a multilingual model such as mBERT is a good choice for low-resourced languages \cite{wu-dredze-2020-languages,rust-etal-2020-how,chau-etal-2020-parsing}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="2012.15613" id="2556">
        <attvalues>
          <attvalue for="5" value=" While there is evidence to suggest that dedicated monolingual models can be superior to a multilingual model for within-language downstream tasks \cite{devries2019bertje,virtanen-etal-2019-multilingual,farahani-etal-2020-parsbert}, other studies suggest that a multilingual model such as mBERT is a good choice for low-resourced languages \cite{wu-dredze-2020-languages,rust-etal-2020-how,chau-etal-2020-parsing}." />
        </attvalues>
      </edge>
      <edge source="2107.12930" target="2009.14124" id="2557">
        <attvalues>
          <attvalue for="5" value=" While there is evidence to suggest that dedicated monolingual models can be superior to a multilingual model for within-language downstream tasks \cite{devries2019bertje,virtanen-etal-2019-multilingual,farahani-etal-2020-parsbert}, other studies suggest that a multilingual model such as mBERT is a good choice for low-resourced languages \cite{wu-dredze-2020-languages,rust-etal-2020-how,chau-etal-2020-parsing}." />
        </attvalues>
      </edge>
      <edge source="2005.04511" target="1903.08855" id="2558">
        <attvalues>
          <attvalue for="5" value=" Past work \cite{liu2019linguistic,tenney2019bert,tenney2019what} has found that masked language models such as BERT \cite{devlin2019bert} learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision." />
        </attvalues>
      </edge>
      <edge source="2005.04511" target="1905.05950" id="2559">
        <attvalues>
          <attvalue for="5" value=" Past work \cite{liu2019linguistic,tenney2019bert,tenney2019what} has found that masked language models such as BERT \cite{devlin2019bert} learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision." />
        </attvalues>
      </edge>
      <edge source="2005.04511" target="1905.06316" id="2560">
        <attvalues>
          <attvalue for="5" value=" Past work \cite{liu2019linguistic,tenney2019bert,tenney2019what} has found that masked language models such as BERT \cite{devlin2019bert} learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision." />
        </attvalues>
      </edge>
      <edge source="2005.04511" target="1901.07291" id="2562">
        <attvalues>
          <attvalue for="5" value=" Recently, large multilingual masked language models such as Multilingual BERT (mBERT) and XLM \cite{conneau2019cross,conneau2019unsupervised} have shown strong cross-lingual performance on tasks like XNLI \cite{lample2019cross,williams2018broad} and dependency parsing \cite{wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2005.04511" target="1911.02116" id="2563">
        <attvalues>
          <attvalue for="5" value=" Recently, large multilingual masked language models such as Multilingual BERT (mBERT) and XLM \cite{conneau2019cross,conneau2019unsupervised} have shown strong cross-lingual performance on tasks like XNLI \cite{lample2019cross,williams2018broad} and dependency parsing \cite{wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2005.04511" target="1704.05426" id="2564">
        <attvalues>
          <attvalue for="5" value=" Recently, large multilingual masked language models such as Multilingual BERT (mBERT) and XLM \cite{conneau2019cross,conneau2019unsupervised} have shown strong cross-lingual performance on tasks like XNLI \cite{lample2019cross,williams2018broad} and dependency parsing \cite{wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2005.04511" target="1904.09077" id="2565">
        <attvalues>
          <attvalue for="5" value=" Recently, large multilingual masked language models such as Multilingual BERT (mBERT) and XLM \cite{conneau2019cross,conneau2019unsupervised} have shown strong cross-lingual performance on tasks like XNLI \cite{lample2019cross,williams2018broad} and dependency parsing \cite{wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="1911.02116" id="2567">
        <attvalues>
          <attvalue for="5" value=" Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLMR \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5} and mBART \cite{liu-etal-2020-multilingual-denoising} have been surprisingly effective at zero-shot cross lingual transfer eg when fine-tuned on an NLP task in one language, they often tend to generalize reasonably well in languages unseen during fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="2010.11934" id="2568">
        <attvalues>
          <attvalue for="5" value=" Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLMR \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5} and mBART \cite{liu-etal-2020-multilingual-denoising} have been surprisingly effective at zero-shot cross lingual transfer eg when fine-tuned on an NLP task in one language, they often tend to generalize reasonably well in languages unseen during fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="2001.08210" id="2569">
        <attvalues>
          <attvalue for="5" value=" Massively Multilingual Language Models (MMLMs) like mBERT \cite{devlin-etal-2019-bert}, XLMR \cite{conneau-etal-2020-unsupervised}, mT5 \cite{xue-etal-2021-mt5} and mBART \cite{liu-etal-2020-multilingual-denoising} have been surprisingly effective at zero-shot cross lingual transfer eg when fine-tuned on an NLP task in one language, they often tend to generalize reasonably well in languages unseen during fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="2005.00052" id="2570">
        <attvalues>
          <attvalue for="5" value=" These models have been evaluated for their performance across a range of multilingual tasks \cite{Pan2017, nivre2018universal, Conneau2018xnli} and numerous methods like adapters \cite{pfeiffer-etal-2020-mad}, sparse fine-tuning \cite{ansell-etal-2022-composable} and few-shot learning \cite{lauscher-etal-2020-zero} have been proposed to further improve performance of cross lingual transfer." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="1703.04009" id="2572">
        <attvalues>
          <attvalue for="5" value=" As these models find their way more and more into the real word applications with safety implications, like Hate Speech Detection \cite{DavidsonWMW17, deshpande-2022-highly} it becomes important to only take extreme actions for high confidence predictions by the model \cite{Sarkar_KhudaBukhsh_2021}." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="1706.04599" id="2573">
        <attvalues>
          <attvalue for="5" value=" Finally, we show that model calibration across different languages can be substantially improved by utilizing standard calibration techniques like Temperature Scaling \cite{guo-2017-on} and Label Smoothing \cite{pereyra-et-al-2017} without collecting any data in the language (see Figure \ref{fig:sw_zs})." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="2003.07892" id="2575">
        <attvalues>
          <attvalue for="5" value=" For NLP tasks specifically, \cite{desai-durrett-2020-calibration} showed that classifiers trained using pre-trained transformer based models \cite{devlin-etal-2019-bert} are well calibrated both in-domain and out-of-domain settings compared to non-pre-trained model baselines \cite{chen-etal-2017-enhanced}." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="1609.06038" id="2576">
        <attvalues>
          <attvalue for="5" value=" For NLP tasks specifically, \cite{desai-durrett-2020-calibration} showed that classifiers trained using pre-trained transformer based models \cite{devlin-etal-2019-bert} are well calibrated both in-domain and out-of-domain settings compared to non-pre-trained model baselines \cite{chen-etal-2017-enhanced}." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="2001.11453" id="2577">
        <attvalues>
          <attvalue for="5" value=" Notably, \cite{ponti-etal-2021-parameter} highlights, since zero-shot cross lingual transfer represents shifts in the data distribution the point estimates are likely to be miscalibrated, which forms the core setting of our work." />
        </attvalues>
      </edge>
      <edge source="2210.12265" target="1701.06548" id="2578">
        <attvalues>
          <attvalue for="5" value=" Finally, we show that model calibration across different languages can be substantially improved by utilizing standard calibration techniques like Temperature Scaling \cite{guo-2017-on} and Label Smoothing \cite{pereyra-et-al-2017} without collecting any data in the language (see Figure \ref{fig:sw_zs})." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2005.14165" id="2579">
        <attvalues>
          <attvalue for="5" value=" Instruction fine-tuning is key to improving the capabilities and controllability of large language models (LLMs)~\cite{instruction_tuning:2, instruction_tuning:3}, which have already demonstrated strong performance in various tasks~\cite{zhong2023chat,Peng2023ChatGPT4MT,Lu2023EAPrompt,ren2024healthcare}." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2302.10198" id="2580">
        <attvalues>
          <attvalue for="5" value=" Instruction fine-tuning is key to improving the capabilities and controllability of large language models (LLMs)~\cite{instruction_tuning:2, instruction_tuning:3}, which have already demonstrated strong performance in various tasks~\cite{zhong2023chat,Peng2023ChatGPT4MT,Lu2023EAPrompt,ren2024healthcare}." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2402.13408" id="2582">
        <attvalues>
          <attvalue for="5" value=" Instruction fine-tuning is key to improving the capabilities and controllability of large language models (LLMs)~\cite{instruction_tuning:2, instruction_tuning:3}, which have already demonstrated strong performance in various tasks~\cite{zhong2023chat,Peng2023ChatGPT4MT,Lu2023EAPrompt,ren2024healthcare}." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="1612.00796" id="2583">
        <attvalues>
          <attvalue for="5" value=" One major obstacle to tuning LLMs is catastrophic forgetting (CF,~\cite[]{kirkpatrick2017overcoming}), which means LLMs forget prior knowledge when learning new data." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2401.02954" id="2584">
        <attvalues>
          <attvalue for="5" value=" Recent works have provided substantial evidence confirming the negative impact of CF on LLMs, eg, \cite{bi2024deepseek} empirically show that the fine-tuned model is even worse than their foundation counterpart on several tasks, and \cite{agent} reveal the dropped general performance of LLMs after developing their agent capabilities." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2310.12823" id="2585">
        <attvalues>
          <attvalue for="5" value=" Recent works have provided substantial evidence confirming the negative impact of CF on LLMs, eg, \cite{bi2024deepseek} empirically show that the fine-tuned model is even worse than their foundation counterpart on several tasks, and \cite{agent} reveal the dropped general performance of LLMs after developing their agent capabilities." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2205.12393" id="2586">
        <attvalues>
          <attvalue for="5" value=" For \ding{184}, we found that our introduced optimizer nicely complements a series of anti-forgetting methods, including rehearsal~\cite{Rehearsal} and Wise-FT~\cite{Wise-ft}." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2310.16789" id="2587">
        <attvalues>
          <attvalue for="5" value=" While the above techniques are somewhat successful, they require expensive extra data-constructing and training costs, and are even sometimes impractical in LLMs, because a) the data cards of many pretrained models are unclear~\cite{shi2023detecting}, making rehearsal unfeasible, and b) anti-forgetting training brings an unstable and expensive training process~\cite{datta2023measuring}." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2305.10625" id="2588">
        <attvalues>
          <attvalue for="5" value=" While the above techniques are somewhat successful, they require expensive extra data-constructing and training costs, and are even sometimes impractical in LLMs, because a) the data cards of many pretrained models are unclear~\cite{shi2023detecting}, making rehearsal unfeasible, and b) anti-forgetting training brings an unstable and expensive training process~\cite{datta2023measuring}." />
        </attvalues>
      </edge>
      <edge source="2406.04836" target="2010.01412" id="2590">
        <attvalues>
          <attvalue for="5" value=" Based on our observation, we introduced ``Sharpness-Aware Minimization'' (SAM,~\cite[]{SAM}) to flatten the model LLS to approach \ding{183}." />
        </attvalues>
      </edge>
      <edge source="1911.03058" target="1309.4168" id="2591">
        <attvalues>
          <attvalue for="5" value=" Cross-lingual representations \cite{mikolov2013exploiting} additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI)." />
        </attvalues>
      </edge>
      <edge source="1911.03058" target="1602.01595" id="2592">
        <attvalues>
          <attvalue for="5" value=" BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging \cite{zhang2016ten}, parsing \cite{ammar2016many}, document classification \cite{klementiev2012inducing}, and machine translation \cite{irvine2013combining,artetxe-etal-2018-unsupervised,lample-etal-2018-phrase}." />
        </attvalues>
      </edge>
      <edge source="1911.03058" target="1809.01272" id="2593">
        <attvalues>
          <attvalue for="5" value=" BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging \cite{zhang2016ten}, parsing \cite{ammar2016many}, document classification \cite{klementiev2012inducing}, and machine translation \cite{irvine2013combining,artetxe-etal-2018-unsupervised,lample-etal-2018-phrase}." />
        </attvalues>
      </edge>
      <edge source="1911.03058" target="1802.06893" id="2594">
        <attvalues>
          <attvalue for="5" value=" Such pre-trained word embeddings, such as the fastText Wikipedia vectors~\cite{grave2018learning}, are available for many languages and are widely used." />
        </attvalues>
      </edge>
      <edge source="1911.03058" target="1702.03859" id="2595">
        <attvalues>
          <attvalue for="5" value=" Second, a mapping between the languages is learned in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision \cite{zou2013bilingual}, under minimal supervision eg~using only identical strings \cite{smith2017offline}, or even in an unsupervised fashion \cite{zhang2017adversarial,conneau2018word}." />
        </attvalues>
      </edge>
      <edge source="1911.03058" target="1710.04087" id="2596">
        <attvalues>
          <attvalue for="5" value=" The lexica most commonly used for evaluation are the MUSE lexica \cite{conneau2018word} which cover 45 languages, but with translations only from and into English." />
        </attvalues>
      </edge>
      <edge source="1911.03058" target="1805.06297" id="2597">
        <attvalues>
          <attvalue for="5" value=" Alternative evaluation dictionaries are also very English- and European-centric: \cite{dinu2014make} report results on English--Italian, \cite{artetxe-etal-2017-learning} on English--German and English--Finnish, \cite{zhang2017adversarial} on Spanish--English and Italian--English, and \cite{artetxe2018robust} between English and Italian, German, Finnish, Spanish, and Turkish." />
        </attvalues>
      </edge>
      <edge source="2305.02747" target="2010.01672" id="2598">
        <attvalues>
          <attvalue for="5" value="  DTS is critical in a variety of down-steam dialogue modeling tasks, such as dialogue summarization \cite{chen2020multi, rankae, liu2019automatic, qi2021improving, inan2022structured}, dialogue generation \cite{xu2021discovering, zhang2021modeling, liu2022dial2vec, hu2022unimse}, response prediction \cite{711, lin2022duplex, qian2023empathetic, lin2020discovering, he2022galaxy} and question answering \cite{yoon2018learning, zhang2022slot, dai2022cgodial}." />
        </attvalues>
      </edge>
      <edge source="2305.02747" target="2009.12539" id="2599">
        <attvalues>
          <attvalue for="5" value=" The method is later extended to use sentence embeddings from pre-trained language models \cite{711, solbiati2021unsupervised, he2022space, he2022space2}, such as BERT \cite{bert} and SentenceBERT \cite{sentencebert}." />
        </attvalues>
      </edge>
      <edge source="2305.02747" target="1710.03430" id="2600">
        <attvalues>
          <attvalue for="5" value="  DTS is critical in a variety of down-steam dialogue modeling tasks, such as dialogue summarization \cite{chen2020multi, rankae, liu2019automatic, qi2021improving, inan2022structured}, dialogue generation \cite{xu2021discovering, zhang2021modeling, liu2022dial2vec, hu2022unimse}, response prediction \cite{711, lin2022duplex, qian2023empathetic, lin2020discovering, he2022galaxy} and question answering \cite{yoon2018learning, zhang2022slot, dai2022cgodial}." />
        </attvalues>
      </edge>
      <edge source="2305.02747" target="cs/0003083" id="2601">
        <attvalues>
          <attvalue for="5" value=" Previous studies usually assess topic similarity through dialogue coherence or semantic similarity computed by surface features, such as lexical overlap \cite{c99, graphseg, TopicTiling}." />
        </attvalues>
      </edge>
      <edge source="2305.02747" target="1610.03955" id="2602">
        <attvalues>
          <attvalue for="5" value=" \cite{song2016dialogue} assess semantic similarity using pre-trained word embeddings." />
        </attvalues>
      </edge>
      <edge source="2305.02747" target="1908.10084" id="2604">
        <attvalues>
          <attvalue for="5" value=" In the semantic similarity-based methods, word or sentence embeddings are pre-trained on generic textual corpora and supervised Natural Language Inferring (NLI) datasets \cite{sentencebert}, which are unsuitable for unlabeled dialogue data." />
        </attvalues>
      </edge>
      <edge source="2305.02747" target="1904.03371" id="2606">
        <attvalues>
          <attvalue for="5" value=" As illustrated in Figure \ref{fig:case}, dialogue coherence refers to the response relation between an utterance and its preceding context \cite{dziri2019evaluating}, reflecting whether adjacent utterances are linked together." />
        </attvalues>
      </edge>
      <edge source="2305.02747" target="1710.03957" id="2607">
        <attvalues>
          <attvalue for="5" value=" In the coherence-based methods, CSM \cite{csm} learns dialogue coherence from the DailyDialog dataset \cite{li2017dailydialog} without DTS annotations." />
        </attvalues>
      </edge>
      <edge source="2009.08330" target="1310.4546" id="2621">
        <attvalues>
          <attvalue for="5" value=" In recent years, sequence labelers equipped with contextual embeddings have achieved significant accuracy improvement \cite{peters-etal-2018-deep,akbik-etal-2018-contextual,devlin-etal-2019-bert,martin2019camembert} over approaches that use static non-contextual word embeddings \cite{mikolov2013distributed} and character embeddings \cite{santos2014learning}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1508.05326" id="2632">
        <attvalues>
          <attvalue for="5" value=" Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1704.05426" id="2633">
        <attvalues>
          <attvalue for="5" value=" The development of large-scale datasets, eg, SNLI~\cite{bowman2015large}, and MultiNLI~\cite{williams2018broad}, have greatly fertilized the research, and state-of-the-art models could achieve benchmark accuracies of over 90\% on the testing set of SNLI." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1803.02324" id="2634">
        <attvalues>
          <attvalue for="5" value=" For example, negation words like no and never often suggest contradiction~\cite{gururangan2018annotation}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1905.06221" id="2635">
        <attvalues>
          <attvalue for="5" value=" As the sources and preparations of datasets differ, they are less likely to suffer from the same kind of biases~\cite{zhang2019selection}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1905.06517" id="2636">
        <attvalues>
          <attvalue for="5" value=" Although different NLI datasets may be regarded as from different ``domains'', they also all belong to a same general domain --- the real world~\cite{torralba2011unbiased}, which is often witnessed in the domain generalization problems~\cite{jiang2007instance,liang2019additive}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1705.02364" id="2637">
        <attvalues>
          <attvalue for="5" value=" Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1709.04696" id="2638">
        <attvalues>
          <attvalue for="5" value=" Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1606.01933" id="2640">
        <attvalues>
          <attvalue for="5" value=" Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1702.03814" id="2641">
        <attvalues>
          <attvalue for="5" value=" Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1609.06038" id="2642">
        <attvalues>
          <attvalue for="5" value=" Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1908.10763" id="2645">
        <attvalues>
          <attvalue for="5" value=" Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}." />
        </attvalues>
      </edge>
      <edge source="2010.07676" target="1909.04242" id="2647">
        <attvalues>
          <attvalue for="5" value=" Under the proposed scheme, we evaluate 9 widely-used neural network-based NLI models~\cite{bowman2015large,conneau2017supervised,shen2018disan,talman2019sentence,parikh2016decomposable,wang2017bilateral,chen2017enhanced,gong2017natural,devlin2019bert} in Section~\ref{sec:eva_models}, and 5 existing debiasing methods for annotation artifacts~\cite{he2019unlearn,belinkov2019don,zhang2019mitigating,clark2019don} in Section~\ref{sec:eva_debiasing}." />
        </attvalues>
      </edge>
      <edge source="2406.03476" target="2402.00159" id="2648">
        <attvalues>
          <attvalue for="5" value=" Pretraining datasets for large language models (LLMs), such as Dolma \cite{soldaini2023dolma}, have grown to trillions of tokens." />
        </attvalues>
      </edge>
      <edge source="2406.03476" target="2306.01116" id="2649">
        <attvalues>
          <attvalue for="5" value=" Some previous works have opted to pretrain entirely on heavily processed CC data \cite{penedo2023refinedweb}." />
        </attvalues>
      </edge>
      <edge source="2406.03476" target="2307.09288" id="2650">
        <attvalues>
          <attvalue for="5" value=" However, most recent language models trained at scale disclose limited information on the contents of their pretraining data \cite{touvron2023llama, jiang2023mistral, jiang2024mixtral,team2024gemma}." />
        </attvalues>
      </edge>
      <edge source="2406.03476" target="2403.08295" id="2651">
        <attvalues>
          <attvalue for="5" value=" However, most recent language models trained at scale disclose limited information on the contents of their pretraining data \cite{touvron2023llama, jiang2023mistral, jiang2024mixtral,team2024gemma}." />
        </attvalues>
      </edge>
      <edge source="2406.03476" target="2305.10429" id="2652">
        <attvalues>
          <attvalue for="5" value=" At smaller scales, there have been attempts to algorithmically optimize the data mix proportions, but these methods have not been openly validated at the scale most modern language models are trained \cite{xie2024doremi}." />
        </attvalues>
      </edge>
      <edge source="2306.04399" target="2111.09296" id="2654">
        <attvalues>
          <attvalue for="5" value=" Two of the used datasets (CommonVoice and VoxPopuli) are public speech recognition datasets used very often for the benchmarking of ASR systems in many languages \cite{babu22_interspeech,radford2022whisper}." />
        </attvalues>
      </edge>
      <edge source="2305.02215" target="2110.08534" id="2656">
        <attvalues>
          <attvalue for="5" value=" These models outperform all other models nearly consistently after fine-tuning or domain-adaptation \cite{jin-etal-2022-lifelong-pretraining}." />
        </attvalues>
      </edge>
      <edge source="2305.02215" target="1905.05950" id="2658">
        <attvalues>
          <attvalue for="5" value=" In this way, BERT \cite{devlin-etal-2019-bert} contextual representations have been tested to assess their ability to model syntactic information and morphology \cite{tenney-etal-2019-bert, goldberg-syntax-bert, hewitt-manning-2019-structural, jawahar-etal-2019-bert, conen-visualizing-bert, edmiston_systematic_2020}, also comparing different monolingual BERT models \cite{nikolaev-pado-2022-word, otmakhova-etal-2022-cross}." />
        </attvalues>
      </edge>
      <edge source="2305.02215" target="2205.11987" id="2659">
        <attvalues>
          <attvalue for="5" value=" In this way, BERT \cite{devlin-etal-2019-bert} contextual representations have been tested to assess their ability to model syntactic information and morphology \cite{tenney-etal-2019-bert, goldberg-syntax-bert, hewitt-manning-2019-structural, jawahar-etal-2019-bert, conen-visualizing-bert, edmiston_systematic_2020}, also comparing different monolingual BERT models \cite{nikolaev-pado-2022-word, otmakhova-etal-2022-cross}." />
        </attvalues>
      </edge>
      <edge source="2305.02215" target="1905.00414" id="2660">
        <attvalues>
          <attvalue for="5" value=" For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices \cite{kornblith2019similarity}." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="1708.02182" id="2661">
        <attvalues>
          <attvalue for="5" value=" Language modeling is the task of predicting the probability of a text (often conditioned on context), with broad-spanning applications across natural language processing~\cite{bengio2003neural,merity2018regularizing,baevski2018adaptive,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="1809.10853" id="2662">
        <attvalues>
          <attvalue for="5" value=" Language modeling is the task of predicting the probability of a text (often conditioned on context), with broad-spanning applications across natural language processing~\cite{bengio2003neural,merity2018regularizing,baevski2018adaptive,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="2005.14165" id="2663">
        <attvalues>
          <attvalue for="5" value=" Language modeling is the task of predicting the probability of a text (often conditioned on context), with broad-spanning applications across natural language processing~\cite{bengio2003neural,merity2018regularizing,baevski2018adaptive,brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="1711.02604" id="2664">
        <attvalues>
          <attvalue for="5" value=" Recently, retrieval-augmented LMs have shown a series of impressive results~\cite{grave2017unbounded,guu2018generating,he2020learning,khandelwal20generalization,borgeaud2022improving,alon2022neuro}." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="1709.08878" id="2665">
        <attvalues>
          <attvalue for="5" value=" Recently, retrieval-augmented LMs have shown a series of impressive results~\cite{grave2017unbounded,guu2018generating,he2020learning,khandelwal20generalization,borgeaud2022improving,alon2022neuro}." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="2006.16336" id="2666">
        <attvalues>
          <attvalue for="5" value=" Recently, retrieval-augmented LMs have shown a series of impressive results~\cite{grave2017unbounded,guu2018generating,he2020learning,khandelwal20generalization,borgeaud2022improving,alon2022neuro}." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="1911.00172" id="2667">
        <attvalues>
          <attvalue for="5" value=" \item Depending on the design decisions that are chosen for modeling, adding a temperature term to the \knn non-parametric component can become crucial to the success of modeling (although coincidentally, in the original settings of \cite{khandelwal20generalization}, a temperature of 1." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="2201.12431" id="2669">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Finally, one significant drawback to the current \knnlm is the inefficiency of \knn search performed at each step \cite{he2021efficient,borgeaud2022improving,alon2022neuro,Wang2022EfficientCK}." />
        </attvalues>
      </edge>
      <edge source="2301.02828" target="2109.04212" id="2670">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Finally, one significant drawback to the current \knnlm is the inefficiency of \knn search performed at each step \cite{he2021efficient,borgeaud2022improving,alon2022neuro,Wang2022EfficientCK}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="1905.00537" id="2682">
        <attvalues>
          <attvalue for="5" value=" Large pretrained language models~\cite{bert,T5} (PreLMs) have emerged as de-facto standard methods for natural language processing~\cite{glue,superglue}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="2005.14165" id="2683">
        <attvalues>
          <attvalue for="5" value=" However, the drawback of PreLMs is that the models are becoming larger and larger with up to several billion parameters~\cite{GPT-3}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="1906.02243" id="2684">
        <attvalues>
          <attvalue for="5" value=" This comes with high environmental and economic costs~\cite{DBLP:conf/acl/StrubellGM19} and puts development and research in the hands of a few global players\extended{~with rich resources} only \cite[pp." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="2005.07683" id="2685">
        <attvalues>
          <attvalue for="5" value=" The immense resource requirements prevent the use of these models in small-scale laboratories and on mobile devices, which is tied to privacy concerns~\cite{DBLP:conf/nips/Sanh0R20}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="1503.02531" id="2686">
        <attvalues>
          <attvalue for="5" value=" Both knowledge distillation and model compression can be described as teacher-student setups~\cite{knowledgedistillation,modelcompression}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="2105.10059" id="2687">
        <attvalues>
          <attvalue for="5" value=" Both knowledge distillation and model compression can be described as teacher-student setups~\cite{knowledgedistillation,modelcompression}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="1909.10351" id="2689">
        <attvalues>
          <attvalue for="5" value=" Reducing the size of PreLMs using knowledge distillation~\cite{knowledgedistillation} or model compression~\cite{modelcompression} is an active area of research~\cite{distilbert,tinybert,sun2020mobilebert}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="2004.02984" id="2690">
        <attvalues>
          <attvalue for="5" value=" Sharing the same architecture between the student and the teacher enables the use of dedicated distillation techniques, \eg aligning the representations of intermediate layers~\cite{distilbert,sun2020mobilebert}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="1903.12136" id="2691">
        <attvalues>
          <attvalue for="5" value=" However, using more efficient architectures as student has already shown promising results, such as the task-specific distillation approaches by Tang~\etal~\cite{tang2019distilling} and Wasserblatt~\etal~\cite{boundariesbertdistillation}." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="1902.06423" id="2693">
        <attvalues>
          <attvalue for="5" value=" To this end, we extend Mai~\etal's CMOW/CBOW-Hybrid model~\cite{cmow}, which is a hybrid variant unifying the strength of CBOW and CMOW, with a bidirectional representation of the sequences." />
        </attvalues>
      </edge>
      <edge source="2109.08449" target="1802.05365" id="2694">
        <attvalues>
          <attvalue for="5" value=" Our results show that large PreLMs can be distilled into efficient order-sensitive embedding models, achieving a performance that is competitive to ELMo~\cite{DBLP:conf/naacl/PetersNIGCLZ18} on the GLUE benchmark." />
        </attvalues>
      </edge>
      <edge source="2310.12150" target="2202.03629" id="2695">
        <attvalues>
          <attvalue for="5" value=" In recent years, we learned surprisingly impressive yet brittle~\cite{ji2023survey, liu2023evaluating} LFQA capabilities of large-scale LLMs." />
        </attvalues>
      </edge>
      <edge source="2310.12150" target="2307.03172" id="2697">
        <attvalues>
          <attvalue for="5" value=" \cite{liu2023lost} discovered how information placed in the middle of contexts is not used by LMs and a line of work~\cite{Chen2022RichKS, longpre-etal-2021-entity} showed parametric knowledge continues to affect generation even when relevant documents are provided in-context for factoid QA task." />
        </attvalues>
      </edge>
      <edge source="2310.12150" target="2210.13701" id="2698">
        <attvalues>
          <attvalue for="5" value=" \cite{liu2023lost} discovered how information placed in the middle of contexts is not used by LMs and a line of work~\cite{Chen2022RichKS, longpre-etal-2021-entity} showed parametric knowledge continues to affect generation even when relevant documents are provided in-context for factoid QA task." />
        </attvalues>
      </edge>
      <edge source="2310.12150" target="2103.06332" id="2699">
        <attvalues>
          <attvalue for="5" value=" As evaluating the quality of LFQA is notoriously difficult~\cite{krishna-etal-2021-hurdles}, we start our analysis by measuring surface features (eg length, perplexity) that correlate with specific answer qualities such as coherence~\cite{xu-etal-2023-critical}." />
        </attvalues>
      </edge>
      <edge source="2310.12150" target="2305.18201" id="2700">
        <attvalues>
          <attvalue for="5" value=" As evaluating the quality of LFQA is notoriously difficult~\cite{krishna-etal-2021-hurdles}, we start our analysis by measuring surface features (eg length, perplexity) that correlate with specific answer qualities such as coherence~\cite{xu-etal-2023-critical}." />
        </attvalues>
      </edge>
      <edge source="2310.12150" target="2103.08541" id="2702">
        <attvalues>
          <attvalue for="5" value=" To evaluate this, we newly collect human annotations on sentence-level attribution~\cite{rashkin2021measuring} and evaluate off-the-shelf models for detecting attributions~\cite{schuster-etal-2021-get} on our collected dataset (Section~\ref{sec:nli})." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2212.10509" id="2713">
        <attvalues>
          <attvalue for="5" value=" Moreover, using retrieval augmented generation (RAG) provides additional context based on a user query such that an LLM can generate an in-domain output response \cite{Ram_2023} \cite{Trivedi_2022}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2305.11255" id="2714">
        <attvalues>
          <attvalue for="5" value="Liu_2023}, simple QA (Question-Answering) \cite{Trivedi_2022}, multiple choice QA \cite{Trivedi_2022}, financial sentiment analysis \cite{Fei_2023}, Retrieval Augmented Generation (RAG) \cite{Pan_2022}, among others \cite{Lu_2021}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2203.16714" id="2715">
        <attvalues>
          <attvalue for="5" value="Liu_2023}, simple QA (Question-Answering) \cite{Trivedi_2022}, multiple choice QA \cite{Trivedi_2022}, financial sentiment analysis \cite{Fei_2023}, Retrieval Augmented Generation (RAG) \cite{Pan_2022}, among others \cite{Lu_2021}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2104.08786" id="2716">
        <attvalues>
          <attvalue for="5" value=" Additionally, altering the available input context and the order in which information is presented in a prompt \cite{Lu_2021} can affect the final output response structure." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2005.14165" id="2718">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning generative LLMs increases a model's overall performance in a specific domain \cite{Brown_2020}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2201.11903" id="2720">
        <attvalues>
          <attvalue for="5" value=" Zero-shot \cite{Kojima_2022} and few-shot \cite{Wei_2022} prompting techniques can be leveraged to direct answer structures and elicit human-like reasoning for response generation." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="1310.8059" id="2721">
        <attvalues>
          <attvalue for="5" value=" STS is measured through many different techniques that can help in text classification and topic extraction \cite{Slimani_2013} without the limitations of lexical similarity." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2309.06541" id="2722">
        <attvalues>
          <attvalue for="5" value=" Semantic similarity is achievable with generative LLMs as demonstrated by \cite{Gatto_2023}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2302.00083" id="2723">
        <attvalues>
          <attvalue for="5" value=" Moreover, using retrieval augmented generation (RAG) provides additional context based on a user query such that an LLM can generate an in-domain output response \cite{Ram_2023} \cite{Trivedi_2022}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2312.10997" id="2724">
        <attvalues>
          <attvalue for="5" value=" \cite{gao_2024} outlines the current advances in RAG techniques and the categorization of similar technologies." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2307.15770" id="2726">
        <attvalues>
          <attvalue for="5" value=" Similar research was also conducted in the financial domain, where a corporate sustainability report was compared to a sustainability guideline document in their framework, chatReport \cite{Ni_2023}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2310.20558" id="2728">
        <attvalues>
          <attvalue for="5" value=" Minimizing information loss and prompting under token limits are ongoing issues that must be addressed to expand system functionality \cite{jaiswal2023}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2310.06117" id="2729">
        <attvalues>
          <attvalue for="5" value=" Existing research in pre-retrieval RAG processes is limited by the complexity and volume of text provided to its process \cite{zheng_2024}\cite{ma_2023} and abstraction research exhibits dependencies in extensive model finetuning \cite{dixit_2023}." />
        </attvalues>
      </edge>
      <edge source="2404.04351" target="2305.14283" id="2730">
        <attvalues>
          <attvalue for="5" value=" Existing research in pre-retrieval RAG processes is limited by the complexity and volume of text provided to its process \cite{zheng_2024}\cite{ma_2023} and abstraction research exhibits dependencies in extensive model finetuning \cite{dixit_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.10997" target="2211.08411" id="2731">
        <attvalues>
          <attvalue for="5" value=" L{arge} language models (LLMs) have achieved remarkable success, though they still face significant limitations, especially in domain-specific or knowledge-intensive tasks~\cite{longtail}, notably producing ``hallucinations&quot;~\cite{hallucination} when handling queries beyond their training data or requiring current information." />
        </attvalues>
      </edge>
      <edge source="2312.10997" target="2309.01219" id="2732">
        <attvalues>
          <attvalue for="5" value=" L{arge} language models (LLMs) have achieved remarkable success, though they still face significant limitations, especially in domain-specific or knowledge-intensive tasks~\cite{longtail}, notably producing ``hallucinations&quot;~\cite{hallucination} when handling queries beyond their training data or requiring current information." />
        </attvalues>
      </edge>
      <edge source="2312.10997" target="2310.20158" id="2733">
        <attvalues>
          <attvalue for="5" value=" This early stage was characterized by foundational work aimed at refining pre-training techniques\cite{REALM,RAG,Retro}." />
        </attvalues>
      </edge>
      <edge source="2312.10997" target="2005.11401" id="2734">
        <attvalues>
          <attvalue for="5" value=" This early stage was characterized by foundational work aimed at refining pre-training techniques\cite{REALM,RAG,Retro}." />
        </attvalues>
      </edge>
      <edge source="2005.01641" target="1802.05365" id="2739">
        <attvalues>
          <attvalue for="5" value=" Recently, unsupervised sentence encoders such as ELMo \cite{peters-etal-2018-deep} and $\BERT$ \cite{devlin-etal-2019-bert} have become popular within NLP." />
        </attvalues>
      </edge>
      <edge source="2005.01641" target="1903.08855" id="2743">
        <attvalues>
          <attvalue for="5" value=" For this reason, probes are made ``simple'' \cite{liu-etal-2019-linguistic}, which usually means they are minimally parameterised." />
        </attvalues>
      </edge>
      <edge source="2305.13286" target="1904.09077" id="2761">
        <attvalues>
          <attvalue for="5" value=" Thus, these models can exploit data from one language to learn generalisations useful for another, obtaining impressive performance on zero-shot cross-lingual transfer for many languages \cite{wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2305.13286" target="1911.02116" id="2762">
        <attvalues>
          <attvalue for="5" value=" Various studies suggest that representations created by popular MLMs, such as mBERT and XLM-R \cite{conneau2020unsupervised}, are not fully language-agnostic \cite{doddapaneni2021primer, singh2019bert}, but instead strike a balance between language-agnosticism and capturing the nuances of different languages through language-neutral and language-specific components \cite{libovicky2020language,gonen2020s, tanti2021language}." />
        </attvalues>
      </edge>
      <edge source="2305.13286" target="2107.00676" id="2763">
        <attvalues>
          <attvalue for="5" value=" Various studies suggest that representations created by popular MLMs, such as mBERT and XLM-R \cite{conneau2020unsupervised}, are not fully language-agnostic \cite{doddapaneni2021primer, singh2019bert}, but instead strike a balance between language-agnosticism and capturing the nuances of different languages through language-neutral and language-specific components \cite{libovicky2020language,gonen2020s, tanti2021language}." />
        </attvalues>
      </edge>
      <edge source="2305.13286" target="2109.04727" id="2765">
        <attvalues>
          <attvalue for="5" value=" Many works have studied the encoding of cross-lingual patterns within MLMs by either focusing on probing for particular cross-linguistic differences \cite{ravishankar2019multilingual, choenni2022investigating}, or by analyzing the distributional properties of representational language subspaces \cite{yang2021simple, rajaee2022isotropy, chang2022geometry, chi2020finding}." />
        </attvalues>
      </edge>
      <edge source="2305.13286" target="2002.08484" id="2766">
        <attvalues>
          <attvalue for="5" value=" Analyzing the cross-lingual sharing mechanism from the data reliance perspective leads to a set of interesting questions that we explore: \begin{enumerate} \item Given a test language $A$, does our MLM tend to base its predictions only on data from $A$ itself, or does it also employ data from other languages that it was exposed to during task fine-tuning? \item Do MLMs only employ data cross-lingually out of necessity, eg, in scenarios where in-language fine-tuning data is unavailable or insufficient? \item Do languages support each other by adding similar information to what is relied upon from in-language data (eg, reinforcing the model in what it already learns), or do they (also) provide complementary information? \item How do cross-lingual sharing dynamics change over the course of fine-tuning? \item Is the cross-lingual sharing behaviour similar when the test language was seen during fine-tuning compared to when it is used in a zero-shot testing scenario? \end{enumerate} To study this, we use TracIn \cite{pruthi2020estimating}, a training data attribution (TDA) method to identify a set of training samples that are most informative for a particular test prediction." />
        </attvalues>
      </edge>
      <edge source="2305.13286" target="1906.01502" id="2767">
        <attvalues>
          <attvalue for="5" value=" This indicates that MLM representations might be more universal than previous work suggested \cite{singh2019bert}, in part explaining the `surprising' effectiveness of cross-lingual transfer \cite{pires2019multilingual, wu2019beto, karthikeyan2020cross}." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1706.03762" id="2770">
        <attvalues>
          <attvalue for="5" value=" The recipe is to train a deep transformer based model \cite{DBLP:journals/corr/VaswaniSPUJGKP17} on large amounts of monolingual data and then fine-tune it on small amounts of task-specific data." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1912.05372" id="2771">
        <attvalues>
          <attvalue for="5" value=" Given its success in English NLP, this recipe has been replicated across languages leading to many language specific BERTs such as FlauBERT (French) \cite{le2020flaubert}, CamemBERT (French) \cite{martin-etal-2020-camembert}, BERTje (Dutch) \cite{devries2019bertje}, FinBERT (Finnish) \cite{ronnqvist-etal-2019-multilingual}, BERTeus (Basque) \cite{DBLP:conf/lrec/AgerriVCBSSA20}, AfriBERT (Afrikaans) \cite{ralethe-2020-adaptation}, IndicBERT (Indian languages) \cite{kakwani2020indicnlpsuite} etc However, training such language-specific models is only feasible for a few languages which have the necessary data and computational resources." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="2004.00033" id="2775">
        <attvalues>
          <attvalue for="5" value=" Given its success in English NLP, this recipe has been replicated across languages leading to many language specific BERTs such as FlauBERT (French) \cite{le2020flaubert}, CamemBERT (French) \cite{martin-etal-2020-camembert}, BERTje (Dutch) \cite{devries2019bertje}, FinBERT (Finnish) \cite{ronnqvist-etal-2019-multilingual}, BERTeus (Basque) \cite{DBLP:conf/lrec/AgerriVCBSSA20}, AfriBERT (Afrikaans) \cite{ralethe-2020-adaptation}, IndicBERT (Indian languages) \cite{kakwani2020indicnlpsuite} etc However, training such language-specific models is only feasible for a few languages which have the necessary data and computational resources." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="2004.09095" id="2776">
        <attvalues>
          <attvalue for="5" value=" The above situation has lead to the undesired effect of limiting recent advances in NLP to English and a few high resource languages \cite{joshi-etal-2020-state}." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1901.07291" id="2777">
        <attvalues>
          <attvalue for="5" value=" For example, could the shared representations learnt by MLLMs~improve Machine Translation between two resource rich languages? We survey several works \cite{DBLP:conf/nips/ConneauL19, kakwani2020indicnlpsuite, DBLP:conf/emnlp/HuangLDGSJZ19, DBLP:conf/acl/ConneauKGCWGGOZ20, DBLP:conf/emnlp/EisenschlosRCKG19, DBLP:conf/semeval/ZampieriNRAKMDP20, libovicky-etal-2020-language, jalili-sabet-etal-2020-simalign, chen-etal-2020-accurate,zenkel-etal-2020-end, dou-neubig-2021-word, imamura-sumita-2019-recycling, DBLP:journals/corr/abs-2012-15547, DBLP:journals/corr/abs-2002-06823, liu-etal-2020-multilingual-denoising, xue2021mt5} which use MLLMs for downstream bilingual tasks such as unsupervised machine translation, cross-lingual word alignment, cross-lingual QA, etc We summarise the main findings of these studies which indicate that MLLMs~are useful for bilingual tasks, particularly in low resource scenarios." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1911.02116" id="2778">
        <attvalues>
          <attvalue for="5" value=" In this work, we survey several existing studies \cite{DBLP:conf/acl/ConneauKGCWGGOZ20, DBLP:conf/rep4nlp/WuD20, DBLP:conf/lrec/AgerriVCBSSA20,DBLP:journals/corr/abs-1912-07076,ronnqvist-etal-2019-multilingual,ro2020multi2oie,DBLP:journals/corr/abs-2007-09757, DBLP:journals/corr/abs-1912-07076, wang-etal-2020-galileo, DBLP:conf/rep4nlp/WuD20} which show that the right choice depends on various factors such as model capacity, amount of pretraining data, fine-tuning mechanism and amount of task-specific training data." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1708.09803" id="2779">
        <attvalues>
          <attvalue for="5" value=" The question then is How do we bring the benefit of such pretrained BERT based models to a very long list of languages of interest? One alternative, which has become popular, is to train multilingual language models (MLLMs) such as mBERT \cite{devlin-etal-2019-bert}, XLM \cite{DBLP:conf/nips/ConneauL19}, XLM-R \cite{DBLP:conf/acl/ConneauKGCWGGOZ20}, etc A MLLM~ is pretrained using large amounts of unlabeled data from multiple languages with the hope that low resource languages may benefit from high resource languages due to shared vocabulary, genetic relatedness \cite{nguyen-chiang-2017-transfer} or contact relatedness \cite{goyal-etal-2020-contact}." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1912.07076" id="2780">
        <attvalues>
          <attvalue for="5" value=" In this work, we survey several existing studies \cite{DBLP:conf/acl/ConneauKGCWGGOZ20, DBLP:conf/rep4nlp/WuD20, DBLP:conf/lrec/AgerriVCBSSA20,DBLP:journals/corr/abs-1912-07076,ronnqvist-etal-2019-multilingual,ro2020multi2oie,DBLP:journals/corr/abs-2007-09757, DBLP:journals/corr/abs-1912-07076, wang-etal-2020-galileo, DBLP:conf/rep4nlp/WuD20} which show that the right choice depends on various factors such as model capacity, amount of pretraining data, fine-tuning mechanism and amount of task-specific training data." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="2007.09757" id="2781">
        <attvalues>
          <attvalue for="5" value=" In this work, we survey several existing studies \cite{DBLP:conf/acl/ConneauKGCWGGOZ20, DBLP:conf/rep4nlp/WuD20, DBLP:conf/lrec/AgerriVCBSSA20,DBLP:journals/corr/abs-1912-07076,ronnqvist-etal-2019-multilingual,ro2020multi2oie,DBLP:journals/corr/abs-2007-09757, DBLP:journals/corr/abs-1912-07076, wang-etal-2020-galileo, DBLP:conf/rep4nlp/WuD20} which show that the right choice depends on various factors such as model capacity, amount of pretraining data, fine-tuning mechanism and amount of task-specific training data." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1912.07840" id="2782">
        <attvalues>
          <attvalue for="5" value=" Of particular interest, is the ability of MLLMs to facilitate zero-shot cross-lingual transfer \cite{DBLP:conf/iclr/KWMR20} from a resource rich language to a resource deprived language which does not have any task-specific training data." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="2104.07412" id="2783">
        <attvalues>
          <attvalue for="5" value=" To evaluate such cross-lingual transfer, several benchmarks, such as XGLUE \cite{DBLP:conf/emnlp/LiangDGWGQGSJCF20}, XTREME \cite{DBLP:conf/icml/HuRSNFJ20}, XTREME-R \cite{DBLP:journals/corr/abs-2104-07412} have been proposed." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1906.01502" id="2784">
        <attvalues>
          <attvalue for="5" value=" Using these benchmarks, several works \cite{DBLP:conf/acl/PiresSG19, DBLP:conf/emnlp/WuD19, DBLP:conf/iclr/KWMR20, DBLP:conf/acl/ArtetxeRY20, DBLP:conf/iclr/KWMR20, dufter-schutze-2020-identifying, DBLP:journals/corr/abs-2004-09205, DBLP:journals/corr/abs-2005-00633, DBLP:journals/corr/abs-2004-14218, DBLP:conf/nips/ConneauL19, DBLP:conf/emnlp/WangCGLL19,liu-etal-2019-investigating, DBLP:conf/iclr/CaoKK20,DBLP:conf/iclr/WangXXYNC20, DBLP:journals/corr/abs-2008-09112, DBLP:conf/acl/WangJBWHT20, DBLP:conf/ijcnlp/ChiDWMH20} have studied the cross-lingual effectiveness of MLLMs and have shown that such transfer depends on various factors such as amount of shared vocabulary, explicit alignment of representations across languages, size of pretraining corpora, etc We collate the main findings of these studies in this survey." />
        </attvalues>
      </edge>
      <edge source="2107.00676" target="1910.04708" id="2785">
        <attvalues>
          <attvalue for="5" value=" Using these benchmarks, several works \cite{DBLP:conf/acl/PiresSG19, DBLP:conf/emnlp/WuD19, DBLP:conf/iclr/KWMR20, DBLP:conf/acl/ArtetxeRY20, DBLP:conf/iclr/KWMR20, dufter-schutze-2020-identifying, DBLP:journals/corr/abs-2004-09205, DBLP:journals/corr/abs-2005-00633, DBLP:journals/corr/abs-2004-14218, DBLP:conf/nips/ConneauL19, DBLP:conf/emnlp/WangCGLL19,liu-etal-2019-investigating, DBLP:conf/iclr/CaoKK20,DBLP:conf/iclr/WangXXYNC20, DBLP:journals/corr/abs-2008-09112, DBLP:conf/acl/WangJBWHT20, DBLP:conf/ijcnlp/ChiDWMH20} have studied the cross-lingual effectiveness of MLLMs and have shown that such transfer depends on various factors such as amount of shared vocabulary, explicit alignment of representations across languages, size of pretraining corpora, etc We collate the main findings of these studies in this survey." />
        </attvalues>
      </edge>
      <edge source="2308.16137" target="2307.09288" id="2791">
        <attvalues>
          <attvalue for="5" value=" They typically train on text segments of fewer than 4K tokens~\cite{touvron2023llama2, MosaicML2023Introducing}, primarily due to the computational overhead quadratic in the input lengths of their Transformer architectures." />
        </attvalues>
      </edge>
      <edge source="2308.16137" target="2104.09864" id="2792">
        <attvalues>
          <attvalue for="5" value=" Relative positional encodings such as RoPE~\cite{su2021roformer} and Alibi~\cite{press2021train} have been widely adopted by state-of-the-art LLMs, which calculate attention based on inter-token distance instead of absolute positions, hoping to avoid model failures due to unseen absolute position embeddings." />
        </attvalues>
      </edge>
      <edge source="2308.16137" target="2004.05150" id="2794">
        <attvalues>
          <attvalue for="5" value=" Moreover, although applying a sliding-window attention pattern on the Transformer architecture can reduce the memory overhead~\cite{beltagy2020longformer, ding2023longnet, zaheer2020big}, they are not directly applicable to pre-trained models for length generalization without further training." />
        </attvalues>
      </edge>
      <edge source="2308.16137" target="2105.03011" id="2796">
        <attvalues>
          <attvalue for="5" value=" Without any parameter updates, LM-Infinite improves scores compared with the original model and truncation baselines on downstream tasks including Passkey Retrieval~\cite{mohtashami2023landmark} and Qasper~\cite{dasigi2021dataset}, which are two established benchmarks for long-context evaluation." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="1211.3711" id="2797">
        <attvalues>
          <attvalue for="5" value=" Recently, neural transducer based end-to-end (E2E) models \cite{Graves-RNNSeqTransduction, he2019streaming, attentionisallyouneed, yeh2019transformer, TT, Li2019RNNT, battenberg2017exploring,chiu2018state, Li2020comparison, xiechen, E2EOverview}, such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction}, transformer-transducer (T-T) \cite{yeh2019transformer, TT} and conformer-transducer (C-T) \cite{gulati2020conformer}, have become the dominant model for automatic speech recognition (ASR) in industry due to its natural streaming property, as well as competitive accuracy with traditional hybrid speech recognition systems \cite{watanabe2017hybrid, sainath2020streaming, Li2020Developing}." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="1712.01769" id="2798">
        <attvalues>
          <attvalue for="5" value=" Recently, neural transducer based end-to-end (E2E) models \cite{Graves-RNNSeqTransduction, he2019streaming, attentionisallyouneed, yeh2019transformer, TT, Li2019RNNT, battenberg2017exploring,chiu2018state, Li2020comparison, xiechen, E2EOverview}, such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction}, transformer-transducer (T-T) \cite{yeh2019transformer, TT} and conformer-transducer (C-T) \cite{gulati2020conformer}, have become the dominant model for automatic speech recognition (ASR) in industry due to its natural streaming property, as well as competitive accuracy with traditional hybrid speech recognition systems \cite{watanabe2017hybrid, sainath2020streaming, Li2020Developing}." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="1910.12977" id="2799">
        <attvalues>
          <attvalue for="5" value=" Recently, neural transducer based end-to-end (E2E) models \cite{Graves-RNNSeqTransduction, he2019streaming, attentionisallyouneed, yeh2019transformer, TT, Li2019RNNT, battenberg2017exploring,chiu2018state, Li2020comparison, xiechen, E2EOverview}, such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction}, transformer-transducer (T-T) \cite{yeh2019transformer, TT} and conformer-transducer (C-T) \cite{gulati2020conformer}, have become the dominant model for automatic speech recognition (ASR) in industry due to its natural streaming property, as well as competitive accuracy with traditional hybrid speech recognition systems \cite{watanabe2017hybrid, sainath2020streaming, Li2020Developing}." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2005.08100" id="2800">
        <attvalues>
          <attvalue for="5" value=" Recently, neural transducer based end-to-end (E2E) models \cite{Graves-RNNSeqTransduction, he2019streaming, attentionisallyouneed, yeh2019transformer, TT, Li2019RNNT, battenberg2017exploring,chiu2018state, Li2020comparison, xiechen, E2EOverview}, such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction}, transformer-transducer (T-T) \cite{yeh2019transformer, TT} and conformer-transducer (C-T) \cite{gulati2020conformer}, have become the dominant model for automatic speech recognition (ASR) in industry due to its natural streaming property, as well as competitive accuracy with traditional hybrid speech recognition systems \cite{watanabe2017hybrid, sainath2020streaming, Li2020Developing}." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2007.15188" id="2801">
        <attvalues>
          <attvalue for="5" value=" Audio generation method could be based on multi-speaker neural text to speech (TTS) model \cite{Li2020Developing, sim2019personalization, deng2020ttsrnnt, zheng2021ttsasr, ttsjasha} or spliced data method \cite{spliced}." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2105.00858" id="2802">
        <attvalues>
          <attvalue for="5" value=" Audio generation method could be based on multi-speaker neural text to speech (TTS) model \cite{Li2020Developing, sim2019personalization, deng2020ttsrnnt, zheng2021ttsasr, ttsjasha} or spliced data method \cite{spliced}." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="1712.01996" id="2803">
        <attvalues>
          <attvalue for="5" value=" Another class of text-only adaptation methods is LM fusion \cite{kannan2018shallowfusion, 2020fusion, 2021fusion, amazonilm, triebiasing}, such as shallow fusion \cite{kannan2018shallowfusion} where an external LM trained on target-domain text is incorporated during the neural transducer model decoding." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2002.11268" id="2804">
        <attvalues>
          <attvalue for="5" value=" To solve such an issue, density ratio \cite{mcdermott2019densityratio}, hybrid autoregressive transducer model \cite{variani2020hybrid}, and internal LM estimation \cite{meng2021ilme,ibmilm} were proposed to remove the influence of the internal LM contained in the neural transducer model." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2003.07705" id="2805">
        <attvalues>
          <attvalue for="5" value=" To solve such an issue, density ratio \cite{mcdermott2019densityratio}, hybrid autoregressive transducer model \cite{variani2020hybrid}, and internal LM estimation \cite{meng2021ilme,ibmilm} were proposed to remove the influence of the internal LM contained in the neural transducer model." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2011.01991" id="2806">
        <attvalues>
          <attvalue for="5" value=" To solve such an issue, density ratio \cite{mcdermott2019densityratio}, hybrid autoregressive transducer model \cite{variani2020hybrid}, and internal LM estimation \cite{meng2021ilme,ibmilm} were proposed to remove the influence of the internal LM contained in the neural transducer model." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2203.02317" id="2807">
        <attvalues>
          <attvalue for="5" value=" To solve such an issue, density ratio \cite{mcdermott2019densityratio}, hybrid autoregressive transducer model \cite{variani2020hybrid}, and internal LM estimation \cite{meng2021ilme,ibmilm} were proposed to remove the influence of the internal LM contained in the neural transducer model." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2102.01380" id="2808">
        <attvalues>
          <attvalue for="5" value=" However, the performance is often sensitive to the interpolation weight of external LM for different tasks, and it needs to be well tuned based on development data to get optimal results \cite{meng202ilmt}." />
        </attvalues>
      </edge>
      <edge source="2212.01992" target="2110.01500" id="2809">
        <attvalues>
          <attvalue for="5" value=" But based on results in \cite{fnt}, FNT degrades the accuracy on general testing sets compared with the standard neural transducer model, although it significantly improves the accuracy in the new domain after adaptation." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="1507.08240" id="2810">
        <attvalues>
          <attvalue for="5" value=" Commonly used E2E ASR models include Connectionist Temporal Classification (CTC) model \cite{Graves-CTCFirst,miao2015eesen,li2018advancing}, Attention-based Encoder-Decoder (AED) model \cite{chan2016listen}, and transducer models such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction} and transformer-transducer (T-T) \cite{TT}." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="1211.3711" id="2812">
        <attvalues>
          <attvalue for="5" value=" Commonly used E2E ASR models include Connectionist Temporal Classification (CTC) model \cite{Graves-CTCFirst,miao2015eesen,li2018advancing}, Attention-based Encoder-Decoder (AED) model \cite{chan2016listen}, and transducer models such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction} and transformer-transducer (T-T) \cite{TT}." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="2002.02562" id="2813">
        <attvalues>
          <attvalue for="5" value=" Commonly used E2E ASR models include Connectionist Temporal Classification (CTC) model \cite{Graves-CTCFirst,miao2015eesen,li2018advancing}, Attention-based Encoder-Decoder (AED) model \cite{chan2016listen}, and transducer models such as recurrent neural network transducer (RNN-T) \cite{Graves-RNNSeqTransduction} and transformer-transducer (T-T) \cite{TT}." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="1811.06621" id="2814">
        <attvalues>
          <attvalue for="5" value=" The industry has more interest in transducer models including RNN-T and T-T because these models can be streaming in a natural way \cite{he2019streaming, TT, Li2019RNNT, battenberg2017exploring,chiu2018state, Li2020comparison, yeh2019transformer, xiechen}." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="1712.01769" id="2815">
        <attvalues>
          <attvalue for="5" value=" The industry has more interest in transducer models including RNN-T and T-T because these models can be streaming in a natural way \cite{he2019streaming, TT, Li2019RNNT, battenberg2017exploring,chiu2018state, Li2020comparison, yeh2019transformer, xiechen}." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="2008.06580" id="2816">
        <attvalues>
          <attvalue for="5" value=" Such collection usually costs a lot on both time and money, which forbids fast model adaption \cite{belladaptation}." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="2007.15188" id="2817">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{Li2020Developing} uses a multi-speaker neural TTS system to generate speech data using the text-only data of the new domain to adapt the RNN-T model." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="1907.01372" id="2818">
        <attvalues>
          <attvalue for="5" value=" \cite{peyser2019improving} improves a general ASR model’s performance on the numeric data domain with TTS generated numeric speech data." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="2001.01798" id="2819">
        <attvalues>
          <attvalue for="5" value=" There are also other methods proposed for the domain adaptation of E2E models \cite{zhongdomain,domainlow}." />
        </attvalues>
      </edge>
      <edge source="2105.00858" target="2101.05525" id="2820">
        <attvalues>
          <attvalue for="5" value=" Compared to aggregating confidence scores \cite{conf_slt, conf_google}, aggregating features could retain much richer information, thus making the final word-level confidence scores more reliable." />
        </attvalues>
      </edge>
      <edge source="2110.01500" target="1211.3711" id="2821">
        <attvalues>
          <attvalue for="5" value=" As discussed above, most previous work on LM adaptation adopted the standard neural Transducer architecture \cite{graves2012sequence, he2019streaming}." />
        </attvalues>
      </edge>
      <edge source="2110.01500" target="2007.15188" id="2823">
        <attvalues>
          <attvalue for="5" value=" One research direction is to adopt Text-to-Speech (TTS) techniques to synthesize audio with the target-domain text \cite{Li2020Developing, sim2019personalization, zheng2021ttsasr, deng2020ttsrnnt}, and then fine-tune the Transducer model on the synthesized audio and text pairs." />
        </attvalues>
      </edge>
      <edge source="2110.01500" target="2003.07705" id="2824">
        <attvalues>
          <attvalue for="5" value=" LM fusion is another popular choice to incorporate external language models trained on target-domain text, such as shallow fusion \cite{kannan2018shallowfusion} and density ratio based LM integration \cite{variani2020hat, meng2021ilme, mcdermott2019densityratio, meng202ilmt, meng2021mwe}." />
        </attvalues>
      </edge>
      <edge source="2110.01500" target="1712.01996" id="2825">
        <attvalues>
          <attvalue for="5" value=" LM fusion is another popular choice to incorporate external language models trained on target-domain text, such as shallow fusion \cite{kannan2018shallowfusion} and density ratio based LM integration \cite{variani2020hat, meng2021ilme, mcdermott2019densityratio, meng202ilmt, meng2021mwe}." />
        </attvalues>
      </edge>
      <edge source="2110.01500" target="2104.11127" id="2826">
        <attvalues>
          <attvalue for="5" value=" There are some recent efforts to fine-tune the predictor \cite{pylkkonen2021fastadapt} or the internal language model \cite{Meng2021ILMA} with an additional language model loss, and then make it behave similar to a language model." />
        </attvalues>
      </edge>
      <edge source="2110.01500" target="2110.05354" id="2827">
        <attvalues>
          <attvalue for="5" value=" There are some recent efforts to fine-tune the predictor \cite{pylkkonen2021fastadapt} or the internal language model \cite{Meng2021ILMA} with an additional language model loss, and then make it behave similar to a language model." />
        </attvalues>
      </edge>
      <edge source="2403.10205" target="1906.01787" id="2829">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) are known to perform really well on many {text2text} \cite{yang2021towards} generation tasks such as {summarization} \cite{liu2019text, el2021automatic}), {translation} \cite{wang2019learning, maruf2021survey}, etc Because of this success, there is a growing research interest in applying LLMs in novel task settings such as {explaining complex codes, generating new recipes, simplifying contents,} etc In this paper, we introduce another novel task called {functionality extraction from Git {README} files} -- a variant of {text summarization} task \cite{prana2019categorizing} that detects all the functionalities supported by the corresponding application software." />
        </attvalues>
      </edge>
      <edge source="2403.10205" target="1802.06997" id="2830">
        <attvalues>
          <attvalue for="5" value=" But many application Git {README} files tend to contain capture {different functionalities of the underlying software code base} along with other implementation details like {what it does, how others can use it, licensing, etc,}\cite{prana2019categorizing, chen2021evaluating}." />
        </attvalues>
      </edge>
      <edge source="2403.10205" target="1907.09190" id="2831">
        <attvalues>
          <attvalue for="5" value=" This task can also be seen as a variation of a Question-Answering (QA) \cite{fan2019eli5, soares2020literature} task where the question like List all functionalities is fixed." />
        </attvalues>
      </edge>
      <edge source="2406.05968" target="2303.08774" id="2832">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs)~\cite{achiam2023gpt, anil2023palm, touvron2023llama} have demonstrated remarkable capabilities in a wide variety of natural language tasks, such as question answering, machine translation, language understanding, and text summarization." />
        </attvalues>
      </edge>
      <edge source="2406.05968" target="2109.01652" id="2833">
        <attvalues>
          <attvalue for="5" value=" These characteristics of LLMs, combined with instruction tuning to align their behavior with human preferences~\cite{wei2022finetuned, ouyang2022training, chung2022scaling}, have allowed users to seamlessly interact with them via text-based prompting." />
        </attvalues>
      </edge>
      <edge source="2406.05968" target="2004.02016" id="2834">
        <attvalues>
          <attvalue for="5" value=" Speech summarization has traditionally been done using a cascade of automatic speech recognition (ASR) and text summarization~\cite{zhu2020hierarchical, zhang2021exploratory}." />
        </attvalues>
      </edge>
      <edge source="2406.05968" target="1602.06023" id="2836">
        <attvalues>
          <attvalue for="5" value=" Experiments demonstrate that our proposed system outperforms a baseline cascade of ASR and LLM-based text processing in terms of response language modeling and speech summarization on the CNN / DailyMail dataset~\cite{nallapati2016abstractive}." />
        </attvalues>
      </edge>
      <edge source="2103.02212" target="1309.4168" id="2837">
        <attvalues>
          <attvalue for="5" value=" Cross-lingual embedding space alignment \cite{mikolov2013exploiting,artetxe-etal-2016-learning,xing-etal-2015-normalized,lample2018word} recently has been attracted a lot of attention because cross-lingual model transfer is effectively facilitated by shared semantic spaces in NLP tasks, eg, named entity recognition \cite{xie-etal-2018-neural}, part-of-speech tagging \cite{hsu-etal-2019-zero}, and dependency parsing \cite{schuster-etal-2019-cross}, where dependency paring is scoped out in this paper." />
        </attvalues>
      </edge>
      <edge source="2103.02212" target="1710.04087" id="2838">
        <attvalues>
          <attvalue for="5" value=" Cross-lingual embedding space alignment \cite{mikolov2013exploiting,artetxe-etal-2016-learning,xing-etal-2015-normalized,lample2018word} recently has been attracted a lot of attention because cross-lingual model transfer is effectively facilitated by shared semantic spaces in NLP tasks, eg, named entity recognition \cite{xie-etal-2018-neural}, part-of-speech tagging \cite{hsu-etal-2019-zero}, and dependency parsing \cite{schuster-etal-2019-cross}, where dependency paring is scoped out in this paper." />
        </attvalues>
      </edge>
      <edge source="2103.02212" target="1808.09861" id="2839">
        <attvalues>
          <attvalue for="5" value=" Cross-lingual embedding space alignment \cite{mikolov2013exploiting,artetxe-etal-2016-learning,xing-etal-2015-normalized,lample2018word} recently has been attracted a lot of attention because cross-lingual model transfer is effectively facilitated by shared semantic spaces in NLP tasks, eg, named entity recognition \cite{xie-etal-2018-neural}, part-of-speech tagging \cite{hsu-etal-2019-zero}, and dependency parsing \cite{schuster-etal-2019-cross}, where dependency paring is scoped out in this paper." />
        </attvalues>
      </edge>
      <edge source="2103.02212" target="1909.09587" id="2840">
        <attvalues>
          <attvalue for="5" value=" Cross-lingual embedding space alignment \cite{mikolov2013exploiting,artetxe-etal-2016-learning,xing-etal-2015-normalized,lample2018word} recently has been attracted a lot of attention because cross-lingual model transfer is effectively facilitated by shared semantic spaces in NLP tasks, eg, named entity recognition \cite{xie-etal-2018-neural}, part-of-speech tagging \cite{hsu-etal-2019-zero}, and dependency parsing \cite{schuster-etal-2019-cross}, where dependency paring is scoped out in this paper." />
        </attvalues>
      </edge>
      <edge source="2103.02212" target="1902.09492" id="2841">
        <attvalues>
          <attvalue for="5" value=" Our approach is most similar to \cite{schuster-etal-2019-cross}, which maps a target language space into a source language space through a linear transformation to realize zero-shot transfer in dependency parsing." />
        </attvalues>
      </edge>
      <edge source="2103.02212" target="1802.05365" id="2842">
        <attvalues>
          <attvalue for="5" value=" With the remarkable development of monolingual contextual pre-trained models \cite{peters-etal-2018-deep,devlin-etal-2019-bert,radford2019language}, which dramatically outperform static word embeddings \cite{mikolov2013efficient,pennington2014glove,bojanowski-etal-2017-enriching} in broad NLP applications, increasing number of researchers have started focusing on contextual representation alignment for cross-lingual dependency parsing \cite{schuster-etal-2019-cross,wang2019cross}." />
        </attvalues>
      </edge>
      <edge source="2103.02212" target="1909.06775" id="2846">
        <attvalues>
          <attvalue for="5" value=" With the remarkable development of monolingual contextual pre-trained models \cite{peters-etal-2018-deep,devlin-etal-2019-bert,radford2019language}, which dramatically outperform static word embeddings \cite{mikolov2013efficient,pennington2014glove,bojanowski-etal-2017-enriching} in broad NLP applications, increasing number of researchers have started focusing on contextual representation alignment for cross-lingual dependency parsing \cite{schuster-etal-2019-cross,wang2019cross}." />
        </attvalues>
      </edge>
      <edge source="2103.02212" target="1906.01622" id="2849">
        <attvalues>
          <attvalue for="5" value=" We significantly mitigate this drawback by leveraging a prepossessing step, iterative normalization (IN) \cite{zhang-etal-2019-girls}, which is originally used for improving the performance of static embedding mapping on the bilingual dictionary induction task." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1709.04482" id="2850">
        <attvalues>
          <attvalue for="5" value=" Much prior work on characterizing the types of linguistic information encoded in computational models of language such as neural networks has focused on supervised readout probes, which train a classifier on top pretrained models to predict a particular linguistic label \cite{belinkov2017analyzing,liu_et_al,tenney2019bert}." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1903.08855" id="2851">
        <attvalues>
          <attvalue for="5" value=" Much prior work on characterizing the types of linguistic information encoded in computational models of language such as neural networks has focused on supervised readout probes, which train a classifier on top pretrained models to predict a particular linguistic label \cite{belinkov2017analyzing,liu_et_al,tenney2019bert}." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1905.05950" id="2852">
        <attvalues>
          <attvalue for="5" value=" Much prior work on characterizing the types of linguistic information encoded in computational models of language such as neural networks has focused on supervised readout probes, which train a classifier on top pretrained models to predict a particular linguistic label \cite{belinkov2017analyzing,liu_et_al,tenney2019bert}." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="2002.00737" id="2853">
        <attvalues>
          <attvalue for="5" value=" In particular, \cite{stanford_nlp} apply probes to discover linear subspaces that encode tree-distances as distances in the representational subspace, and \cite{kim2019pre} show that these distances can be used even without any labeled information to induce hierarchical structure." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1906.02715" id="2855">
        <attvalues>
          <attvalue for="5" value=" Another popular approach to analyzing deep models is through the lens of geometry \cite{reif2019visualizing,gigante2019visualizing}." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1908.02831" id="2856">
        <attvalues>
          <attvalue for="5" value=" Another popular approach to analyzing deep models is through the lens of geometry \cite{reif2019visualizing,gigante2019visualizing}." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1710.06487" id="2857">
        <attvalues>
          <attvalue for="5" value=" More recent techniques such as replica-based mean field manifold analysis method \cite{chung2018classification, cohen2019separability, mamou2020emergence} connects representation geometry with linear classification performance, but the method is limited to categorization tasks." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1906.04068" id="2860">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Our work demonstrates that interventional tools such as controlled input perturbations can be useful for analyzing deep networks, adding to the growing, interdisciplinary body of work which profitably adapt experimental techniques from cognitive neuroscience and psycholinguistics to analyze computational models of language~\cite{futrell2018rnn,wilcox2019hierarchical,futrell-etal-2019-neural,ettinger2020bert}." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1903.03260" id="2861">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Our work demonstrates that interventional tools such as controlled input perturbations can be useful for analyzing deep networks, adding to the growing, interdisciplinary body of work which profitably adapt experimental techniques from cognitive neuroscience and psycholinguistics to analyze computational models of language~\cite{futrell2018rnn,wilcox2019hierarchical,futrell-etal-2019-neural,ettinger2020bert}." />
        </attvalues>
      </edge>
      <edge source="2104.07578" target="1907.13528" id="2862">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Our work demonstrates that interventional tools such as controlled input perturbations can be useful for analyzing deep networks, adding to the growing, interdisciplinary body of work which profitably adapt experimental techniques from cognitive neuroscience and psycholinguistics to analyze computational models of language~\cite{futrell2018rnn,wilcox2019hierarchical,futrell-etal-2019-neural,ettinger2020bert}." />
        </attvalues>
      </edge>
      <edge source="2206.08082" target="1902.00751" id="2863">
        <attvalues>
          <attvalue for="5" value=" Numerous branches of work have been proposed to circumvent such issues, such as Adapters \cite{Houlsby2019ParameterEfficientTL}, LoRA \cite{hu2021lora}, and in-context learning (ICL) \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2206.08082" target="2106.09685" id="2864">
        <attvalues>
          <attvalue for="5" value=" Numerous branches of work have been proposed to circumvent such issues, such as Adapters \cite{Houlsby2019ParameterEfficientTL}, LoRA \cite{hu2021lora}, and in-context learning (ICL) \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2206.08082" target="2005.14165" id="2865">
        <attvalues>
          <attvalue for="5" value=" Numerous branches of work have been proposed to circumvent such issues, such as Adapters \cite{Houlsby2019ParameterEfficientTL}, LoRA \cite{hu2021lora}, and in-context learning (ICL) \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2206.08082" target="2104.08786" id="2866">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{lu2021fantastically} shown in-context learning suffers from the order sensitivity of the demonstrations." />
        </attvalues>
      </edge>
      <edge source="2206.08082" target="2102.09690" id="2867">
        <attvalues>
          <attvalue for="5" value=" \cite{zhao2021calibrate} introduces a contextual calibration procedure to reduce the variance across different choices of demonstrations." />
        </attvalues>
      </edge>
      <edge source="2206.08082" target="2112.08633" id="2868">
        <attvalues>
          <attvalue for="5" value=" \cite{rubin2021learning} suggests demonstration selection by retrieving in-context samples." />
        </attvalues>
      </edge>
      <edge source="2206.08082" target="2001.09977" id="2869">
        <attvalues>
          <attvalue for="5" value=" If so, how can we create demonstrations with high input-demonstration correlation? To this end, we propose a novel method termed self-generated in-context learning (SG-ICL) which generates demonstrations by leveraging the superiority of PLMs generative abilities \cite{adiwardana2020towards, brown2020language, shwartz-etal-2020-unsupervised, ye2022zerogen}." />
        </attvalues>
      </edge>
      <edge source="2305.15119" target="1508.00657" id="2870">
        <attvalues>
          <attvalue for="5" value=" In this line, \cite{ballesteros-etal-2015-improved} already found that character-based word vectors helped improving performance over purely word-level models, specially for rich-resource languages, for which the use of morphological information is more relevant \cite{dehouck-denis-2018-framework}." />
        </attvalues>
      </edge>
      <edge source="2305.15119" target="1808.09060" id="2871">
        <attvalues>
          <attvalue for="5" value=" \cite{smith-etal-2018-investigation} and \cite{de-lhoneux-etal-2017-raw} studied the impact that ignoring PoS tag vectors had on the performance of a biLSTM transition-based parser \cite{kiperwasser-goldberg-2016-simple}." />
        </attvalues>
      </edge>
      <edge source="2305.15119" target="1603.04351" id="2872">
        <attvalues>
          <attvalue for="5" value=" \cite{smith-etal-2018-investigation} and \cite{de-lhoneux-etal-2017-raw} studied the impact that ignoring PoS tag vectors had on the performance of a biLSTM transition-based parser \cite{kiperwasser-goldberg-2016-simple}." />
        </attvalues>
      </edge>
      <edge source="2305.15119" target="2104.01083" id="2874">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{anderson-gomez-rodriguez-2021-taggers} and \cite{anderson-etal-2021-falta} have explored the differences between using gold and predicted PoS tags, showing that the former are helpful to improve the results, while the latter are often not, with the exception of low-resource languages, where they obtain small but consistent improvements." />
        </attvalues>
      </edge>
      <edge source="2305.15119" target="2106.04222" id="2875">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{anderson-gomez-rodriguez-2021-taggers} and \cite{anderson-etal-2021-falta} have explored the differences between using gold and predicted PoS tags, showing that the former are helpful to improve the results, while the latter are often not, with the exception of low-resource languages, where they obtain small but consistent improvements." />
        </attvalues>
      </edge>
      <edge source="2305.15119" target="2210.15219" id="2876">
        <attvalues>
          <attvalue for="5" value=" Furthermore, \cite{munoz-ortiz-etal-2022-parsing} showed that the efficacy of PoS tags in the context of sequence labeling parsing is greatly influenced by the chosen linearization method." />
        </attvalues>
      </edge>
      <edge source="2305.15119" target="1804.07998" id="2878">
        <attvalues>
          <attvalue for="5" value=" Yet, NLP models are very sensible and brittle against small attacks, and simple perturbations like misspellings can greatly reduce performance \cite{ebrahimi-etal-2018-hotflip,alzantot-etal-2018-generating}." />
        </attvalues>
      </edge>
      <edge source="2305.15119" target="1905.11268" id="2880">
        <attvalues>
          <attvalue for="5" value=" In parallel, defensive strategies have been tested to improve the robustness of NLP systems, eg, placing a word recognition module before downstream classifiers \cite{pruthi-etal-2019-combating}, or using spelling checks and adversarial training \cite{li-etal-2019-textbugger}." />
        </attvalues>
      </edge>
      <edge source="2309.04992" target="2005.14165" id="2881">
        <attvalues>
          <attvalue for="5" value=" Prompting can either be an emergent ability learned through scaling up model size \cite{brown2020language, wei2022emergent} or an ability learned through instruction tuning \cite{weifinetuned, chung2022scaling, ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2309.04992" target="2109.01652" id="2882">
        <attvalues>
          <attvalue for="5" value=" Prompting can either be an emergent ability learned through scaling up model size \cite{brown2020language, wei2022emergent} or an ability learned through instruction tuning \cite{weifinetuned, chung2022scaling, ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2309.04992" target="2012.15723" id="2883">
        <attvalues>
          <attvalue for="5" value=" Despite the recent popularity of prompting, there is a known sensitivity of prompt-based LLMs to elements such as prompt template and label words \cite{gao2021making, schick2021exploiting}." />
        </attvalues>
      </edge>
      <edge source="2309.04992" target="2010.15980" id="2884">
        <attvalues>
          <attvalue for="5" value=" Previous works have demonstrated that prompt templates can significantly impact task performance \cite{shin2020autoprompt, zhou2022large} and that factors such as chosen label words can influence system performance for classification tasks \cite{zhao2021calibrate, holtzman2021surface}." />
        </attvalues>
      </edge>
      <edge source="2309.04992" target="2102.09690" id="2885">
        <attvalues>
          <attvalue for="5" value=" Previous works have demonstrated that prompt templates can significantly impact task performance \cite{shin2020autoprompt, zhou2022large} and that factors such as chosen label words can influence system performance for classification tasks \cite{zhao2021calibrate, holtzman2021surface}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2402.06619" id="2887">
        <attvalues>
          <attvalue for="5" value=" The \aya initiative was created to address the aforementioned data scarcity issues by creating and releasing the largest multilingual instruction-style dataset~\cite{ayadata2024} to date, along with the \aya 101 model~\cite{ustun2024aya}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2305.14976" id="2888">
        <attvalues>
          <attvalue for="5" value=" This can result in cliffs in model performance in languages not included in pre-training \cite{schwartz2022towards, Kotek2023GenderBA, Khandelwal2023CasteistBN, vashishtha2023evaluating,khondaker2023gptaraeval}, the introduction of security flaws for all users, \cite{yong2023lowresource, nasr2023scalable, Li2023PrivacyIL, Lukas2023AnalyzingLO, deng2023multilingual} and a growing divide in the cost of technology due to high latencies for generations outside of English \cite{held2023material, durmus2023measuring,nicholas2023lost,ojo2023good,ahia2023languages}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2310.02446" id="2889">
        <attvalues>
          <attvalue for="5" value=" This can result in cliffs in model performance in languages not included in pre-training \cite{schwartz2022towards, Kotek2023GenderBA, Khandelwal2023CasteistBN, vashishtha2023evaluating,khondaker2023gptaraeval}, the introduction of security flaws for all users, \cite{yong2023lowresource, nasr2023scalable, Li2023PrivacyIL, Lukas2023AnalyzingLO, deng2023multilingual} and a growing divide in the cost of technology due to high latencies for generations outside of English \cite{held2023material, durmus2023measuring,nicholas2023lost,ojo2023good,ahia2023languages}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2306.07377" id="2891">
        <attvalues>
          <attvalue for="5" value=" This can result in cliffs in model performance in languages not included in pre-training \cite{schwartz2022towards, Kotek2023GenderBA, Khandelwal2023CasteistBN, vashishtha2023evaluating,khondaker2023gptaraeval}, the introduction of security flaws for all users, \cite{yong2023lowresource, nasr2023scalable, Li2023PrivacyIL, Lukas2023AnalyzingLO, deng2023multilingual} and a growing divide in the cost of technology due to high latencies for generations outside of English \cite{held2023material, durmus2023measuring,nicholas2023lost,ojo2023good,ahia2023languages}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2311.07978" id="2892">
        <attvalues>
          <attvalue for="5" value=" This can result in cliffs in model performance in languages not included in pre-training \cite{schwartz2022towards, Kotek2023GenderBA, Khandelwal2023CasteistBN, vashishtha2023evaluating,khondaker2023gptaraeval}, the introduction of security flaws for all users, \cite{yong2023lowresource, nasr2023scalable, Li2023PrivacyIL, Lukas2023AnalyzingLO, deng2023multilingual} and a growing divide in the cost of technology due to high latencies for generations outside of English \cite{held2023material, durmus2023measuring,nicholas2023lost,ojo2023good,ahia2023languages}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2305.13707" id="2893">
        <attvalues>
          <attvalue for="5" value=" This can result in cliffs in model performance in languages not included in pre-training \cite{schwartz2022towards, Kotek2023GenderBA, Khandelwal2023CasteistBN, vashishtha2023evaluating,khondaker2023gptaraeval}, the introduction of security flaws for all users, \cite{yong2023lowresource, nasr2023scalable, Li2023PrivacyIL, Lukas2023AnalyzingLO, deng2023multilingual} and a growing divide in the cost of technology due to high latencies for generations outside of English \cite{held2023material, durmus2023measuring,nicholas2023lost,ojo2023good,ahia2023languages}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2402.07827" id="2894">
        <attvalues>
          <attvalue for="5" value=" In this technical report, we assess the performance of \aya 23 models following the comprehensive multilingual evaluation framework proposed by~\cite{ustun2024aya}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2211.01786" id="2895">
        <attvalues>
          <attvalue for="5" value=" Multilingual efforts including the release of Aya 101 \cite{ustun2024aya}, BLOOMZ~\cite{muennighoff2022crosslingual} and mT0~\cite{muennighoff2022crosslingual} models have made great strides in expanding access to modern natural language processing technologies for the world." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2010.11934" id="2896">
        <attvalues>
          <attvalue for="5" value=" However, \aya 101 was by necessity built upon the mT5~\cite{xue2020mt5} pre-trained base model given it was one of the few pre-trained models that had been trained on 101 languages." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2307.09288" id="2898">
        <attvalues>
          <attvalue for="5" value="3ex}{+}}, Command R, Llama series~\cite{touvron2023llama,touvron2023llama2}, Mistral models \cite{jiang2023mistral,jiang2024mixtral} and Gemma models \cite{gemmareport}." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2310.06825" id="2899">
        <attvalues>
          <attvalue for="5" value=" For \aya 23, we instead balance breadth and depth, exploring the impact of allocating more capacity to fewer languages (23 languages) that are included during pre-training, alleviating the ``curse'' and leading to large gains over the original \aya 101 and widely used models such as Gemma \cite{gemmareport}, Mistral \cite{jiang2023mistral}, and Mixtral \cite{jiang2024mixtral} for the corresponding 23 languages." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2401.04088" id="2900">
        <attvalues>
          <attvalue for="5" value=" For \aya 23, we instead balance breadth and depth, exploring the impact of allocating more capacity to fewer languages (23 languages) that are included during pre-training, alleviating the ``curse'' and leading to large gains over the original \aya 101 and widely used models such as Gemma \cite{gemmareport}, Mistral \cite{jiang2023mistral}, and Mixtral \cite{jiang2024mixtral} for the corresponding 23 languages." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2403.08295" id="2901">
        <attvalues>
          <attvalue for="5" value=" For \aya 23, we instead balance breadth and depth, exploring the impact of allocating more capacity to fewer languages (23 languages) that are included during pre-training, alleviating the ``curse'' and leading to large gains over the original \aya 101 and widely used models such as Gemma \cite{gemmareport}, Mistral \cite{jiang2023mistral}, and Mixtral \cite{jiang2024mixtral} for the corresponding 23 languages." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="1911.02116" id="2903">
        <attvalues>
          <attvalue for="5" value=" Due to the well-documented curse of multilinguality \cite{arivazhagan2019massively,conneau2019unsupervised,pfeiffer2022lifting}, models attempting to serve such a broad variety of languages often lag in generative performance on any given language relative to models dedicated to serving a more focused subset, because of the need to share model capacity so widely." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2205.06266" id="2904">
        <attvalues>
          <attvalue for="5" value=" Due to the well-documented curse of multilinguality \cite{arivazhagan2019massively,conneau2019unsupervised,pfeiffer2022lifting}, models attempting to serve such a broad variety of languages often lag in generative performance on any given language relative to models dedicated to serving a more focused subset, because of the need to share model capacity so widely." />
        </attvalues>
      </edge>
      <edge source="2405.15032" target="2401.01055" id="2905">
        <attvalues>
          <attvalue for="5" value=" Our choice of languages was guided to align with the languages present in pre-training of Command R, due to known difficulties of introducing new languages after pre-training \cite{zhao2024llama, yong2022bloom+}." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2311.16867" id="2906">
        <attvalues>
          <attvalue for="5" value=" Both these advances led to the release of large open-source models such as Llama-65B \cite{llama} and Falcon-180B \cite{falcon}." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2005.14165" id="2907">
        <attvalues>
          <attvalue for="5" value=" Historically, large language models first saw an important rise in performance with increased model size \cite{gpt3, palm}." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2203.15556" id="2908">
        <attvalues>
          <attvalue for="5" value=" Updated scaling laws \cite{chinchilla} brought to light that this initial generation of large language models were most likely undertrained, highlighting the need for more training data to further increase the performance." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2101.00027" id="2909">
        <attvalues>
          <attvalue for="5" value=" This triggered another important paradigm shift, namely moving from large curated datasets \cite{pile, palm}, to large-scale datasets harvesting mostly web data from the CommonCrawl project, such as RefinedWeb \cite{refinedweb} or RedPajama \cite{redpajama}." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2306.01116" id="2910">
        <attvalues>
          <attvalue for="5" value=" This triggered another important paradigm shift, namely moving from large curated datasets \cite{pile, palm}, to large-scale datasets harvesting mostly web data from the CommonCrawl project, such as RefinedWeb \cite{refinedweb} or RedPajama \cite{redpajama}." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2307.09288" id="2912">
        <attvalues>
          <attvalue for="5" value=" More recently, the Llama2 models \cite{llama2} showed the benefits of even more prolonged training, achieving state-of-the-art performance with smaller model sizes." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2310.06825" id="2914">
        <attvalues>
          <attvalue for="5" value=" This trend was followed in the past year, resulting in a number of small-sized yet highly performing models such as Qwen-7B \cite{qwen}, Mistral-7B \cite{mistral}, Yi-6B and Yi-9B \cite{yi}, Gemma-7B \cite{gemma} and Llama3-8B \cite{llama3}." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2403.04652" id="2915">
        <attvalues>
          <attvalue for="5" value=" This trend was followed in the past year, resulting in a number of small-sized yet highly performing models such as Qwen-7B \cite{qwen}, Mistral-7B \cite{mistral}, Yi-6B and Yi-9B \cite{yi}, Gemma-7B \cite{gemma} and Llama3-8B \cite{llama3}." />
        </attvalues>
      </edge>
      <edge source="2407.14885" target="2403.08295" id="2916">
        <attvalues>
          <attvalue for="5" value=" This trend was followed in the past year, resulting in a number of small-sized yet highly performing models such as Qwen-7B \cite{qwen}, Mistral-7B \cite{mistral}, Yi-6B and Yi-9B \cite{yi}, Gemma-7B \cite{gemma} and Llama3-8B \cite{llama3}." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2203.15556" id="2917">
        <attvalues>
          <attvalue for="5" value=" This design choice is the standard approach rooted from the Transformer original paper~\cite{Vaswani:2017aa}, later modified by GPT-3 and Chinchilla~\cite{hoffmann2022training}, then followed by LLaMA~\cite{llama2}, Baichuan~\cite{Yang:2023aa}, Qwen~\cite{Bai:2023aa} and many related works." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2401.00448" id="2918">
        <attvalues>
          <attvalue for="5" value=" This makes the model-data scale combination fall into the post Chinchilla optimal regime~\cite{sardana2023beyond}, eg, we overtrain the model on more tokens (3T) than the compute optimal (around 1T)." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2301.12017" id="2919">
        <attvalues>
          <attvalue for="5" value=" The benefit is from the inference side, as we achieve stronger performance with reduced serving cost: after int4~\cite{wu2023understanding} quantization, one can serve the 34B chat model on 24G GPU memory with almost no performance drop; (3)." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2210.11416" id="2920">
        <attvalues>
          <attvalue for="5" value=" This approach significantly deviates from the quantity-scaling styled instruction tuning works like FLAN~\cite{chung2022scaling} and UltraChat~\cite{ding2023enhancing}, but aligns more with the handcrafting styled works like LIMA~\cite{zhou2023lima}." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2305.14233" id="2921">
        <attvalues>
          <attvalue for="5" value=" This approach significantly deviates from the quantity-scaling styled instruction tuning works like FLAN~\cite{chung2022scaling} and UltraChat~\cite{ding2023enhancing}, but aligns more with the handcrafting styled works like LIMA~\cite{zhou2023lima}." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2305.11206" id="2922">
        <attvalues>
          <attvalue for="5" value=" This approach significantly deviates from the quantity-scaling styled instruction tuning works like FLAN~\cite{chung2022scaling} and UltraChat~\cite{ding2023enhancing}, but aligns more with the handcrafting styled works like LIMA~\cite{zhou2023lima}." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="1911.00359" id="2923">
        <attvalues>
          <attvalue for="5" value=" This thorough pipeline leads to a much higher removal ratio than existing pipelines like CCNet~\cite{wenzek2019ccnet}, RefinedWeb~\cite{penedo2023refinedweb} and RedPajama~\cite{together2023redpajama}, which we believe is key to the success of data engineering." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2306.01116" id="2924">
        <attvalues>
          <attvalue for="5" value=" This thorough pipeline leads to a much higher removal ratio than existing pipelines like CCNet~\cite{wenzek2019ccnet}, RefinedWeb~\cite{penedo2023refinedweb} and RedPajama~\cite{together2023redpajama}, which we believe is key to the success of data engineering." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2305.13245" id="2925">
        <attvalues>
          <attvalue for="5" value=" Regarding the model architecture, we use standard implementation of the Transformer architecture with Grouped-Query Attention (GQA)~\cite{gqa}, SwiGLU~\cite{swiglu} activation, and RoPE with an adjusted base frequency (RoPE ABF)~\cite{xiong2023effective}." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2002.05202" id="2926">
        <attvalues>
          <attvalue for="5" value=" Regarding the model architecture, we use standard implementation of the Transformer architecture with Grouped-Query Attention (GQA)~\cite{gqa}, SwiGLU~\cite{swiglu} activation, and RoPE with an adjusted base frequency (RoPE ABF)~\cite{xiong2023effective}." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="1706.03762" id="2928">
        <attvalues>
          <attvalue for="5" value=" This design choice is the standard approach rooted from the Transformer original paper~\cite{Vaswani:2017aa}, later modified by GPT-3 and Chinchilla~\cite{hoffmann2022training}, then followed by LLaMA~\cite{llama2}, Baichuan~\cite{Yang:2023aa}, Qwen~\cite{Bai:2023aa} and many related works." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2309.10305" id="2929">
        <attvalues>
          <attvalue for="5" value=" This design choice is the standard approach rooted from the Transformer original paper~\cite{Vaswani:2017aa}, later modified by GPT-3 and Chinchilla~\cite{hoffmann2022training}, then followed by LLaMA~\cite{llama2}, Baichuan~\cite{Yang:2023aa}, Qwen~\cite{Bai:2023aa} and many related works." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2402.10171" id="2931">
        <attvalues>
          <attvalue for="5" value=" To achive 200K context length, we continue pretrain the model on about 5B length-upsampled data, similar to the concurrent work in~\cite{fu2024data}." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2312.15166" id="2933">
        <attvalues>
          <attvalue for="5" value=" We also study the effectiveness of depth-upscailng~\cite{kim2023solar}, eg, making the model deeper by continual pretraining, and confirming its effectiveness to further improve model performance." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="1909.08053" id="2934">
        <attvalues>
          <attvalue for="5" value=" To support finetuning, we build a hierarchical scheduling framework supporting different distributed backends for different models (eg, Megatron~\cite{megatron-lm} for the policy model and DeepSpeed~\cite{zero} for the reward model)." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="1910.02054" id="2935">
        <attvalues>
          <attvalue for="5" value=" To support finetuning, we build a hierarchical scheduling framework supporting different distributed backends for different models (eg, Megatron~\cite{megatron-lm} for the policy model and DeepSpeed~\cite{zero} for the reward model)." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2009.03300" id="2937">
        <attvalues>
          <attvalue for="5" value=" On most standard benchmarks like MMLU~\cite{hendrycks2020measuring} (for the base model) and LMSys ELO Rating~\cite{zheng2023judging} (for the chat model), Yi-34B generally achieves scores on par with GPT-3." />
        </attvalues>
      </edge>
      <edge source="2403.04652" target="2306.05685" id="2938">
        <attvalues>
          <attvalue for="5" value=" On most standard benchmarks like MMLU~\cite{hendrycks2020measuring} (for the base model) and LMSys ELO Rating~\cite{zheng2023judging} (for the chat model), Yi-34B generally achieves scores on par with GPT-3." />
        </attvalues>
      </edge>
      <edge source="2204.11586" target="1706.03762" id="2939">
        <attvalues>
          <attvalue for="5" value=" Transformer~\cite{DBLP:conf/nips/VaswaniSPUJGKP17} architectures, coupled with an increase in computing capabilities, allows current Language Models (LM) to generate very plausible texts." />
        </attvalues>
      </edge>
      <edge source="2204.11586" target="1805.04833" id="2940">
        <attvalues>
          <attvalue for="5" value="_2018} or top-k/p sampling~\cite{DBLP:conf/acl/LewisDF18, DBLP:conf/iclr/HoltzmanBDFC20}; they select the next token only based on the likelihood (according to the LM) of the resulting sequence, which offers only limited control over the text finally generated." />
        </attvalues>
      </edge>
      <edge source="2204.11586" target="2005.11401" id="2942">
        <attvalues>
          <attvalue for="5" value=" This is particularly problematic for Information Retrieval tasks that imply text generation, such as question-answering from the Web \cite{DBLP:journals/corr/abs-2112-09332,NEURIPS2020_6b493230}, query-focused multi-documents summarization \cite{pasunuru2021data}, query expansion \cite{ClaveauWI2021}, query suggestion \cite{DBLP:journals/tois/MustarLP22}, or chatbots for interactive search \cite{pallagani2021generic}, which leverage contents from various -- and sometimes untrusted -- information sources." />
        </attvalues>
      </edge>
      <edge source="2204.11586" target="2103.01863" id="2943">
        <attvalues>
          <attvalue for="5" value=" This is particularly problematic for Information Retrieval tasks that imply text generation, such as question-answering from the Web \cite{DBLP:journals/corr/abs-2112-09332,NEURIPS2020_6b493230}, query-focused multi-documents summarization \cite{pasunuru2021data}, query expansion \cite{ClaveauWI2021}, query suggestion \cite{DBLP:journals/tois/MustarLP22}, or chatbots for interactive search \cite{pallagani2021generic}, which leverage contents from various -- and sometimes untrusted -- information sources." />
        </attvalues>
      </edge>
      <edge source="2204.11586" target="2203.00667" id="2944">
        <attvalues>
          <attvalue for="5" value=" For instance, following Generative Adversarial Networks~\cite{DBLP:journals/cacm/GoodfellowPMXWO20}, many studies train binary discriminators to distinguish real from generated contents, to approximate distributions of observed documents \cite{yu2017seqgan}." />
        </attvalues>
      </edge>
      <edge source="2204.11586" target="1609.05473" id="2945">
        <attvalues>
          <attvalue for="5" value=" For instance, following Generative Adversarial Networks~\cite{DBLP:journals/cacm/GoodfellowPMXWO20}, many studies train binary discriminators to distinguish real from generated contents, to approximate distributions of observed documents \cite{yu2017seqgan}." />
        </attvalues>
      </edge>
      <edge source="2204.11586" target="2109.13582" id="2946">
        <attvalues>
          <attvalue for="5" value=" Other studies train classifiers on semantic properties such as polarity to learn the generation process towards positive or negative texts~\cite{DBLP:journals/corr/abs-2109-13582, DBLP:conf/emnlp/KrauseGMKJSR21, DBLP:conf/iclr/DathathriMLHFMY20}." />
        </attvalues>
      </edge>
      <edge source="2204.11586" target="2106.06363" id="2950">
        <attvalues>
          <attvalue for="5" value=" This non-myopic discriminator-guided decoding lead to state-of-the-art results in different applications~\cite{selfGAN, DBLP:journals/corr/abs-2109-13582, DBLP:conf/emnlp/LeblondASPLASV21, DBLP:journals/corr/abs-2201-12320}." />
        </attvalues>
      </edge>
      <edge source="2205.04810" target="1706.03762" id="2951">
        <attvalues>
          <attvalue for="5" value=" With the advent of the Transformer \cite{vaswani2017attention} and masked language model (MLM) pretraining \cite{devlin2018bert}, attention-based neural networks have proven quite effective at a variety of language tasks, provided that large amounts of data are available for pretraining." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="1909.11942" id="2954">
        <attvalues>
          <attvalue for="5" value=" N{atural} Language Processing (NLP) is undergoing a paradigm shift with the open-source of large-scale pretrained language models (PLMs), such as GPT \cite{radford2018gpt,radford2019language}, BERT \cite{devlin-etal-2019-bert}, ALBERT \cite{Lan2020ALBERT}, RoBERTa \cite{liu2020roberta}, BART \cite{lewis-etal-2020-bart}, and T5 \cite{Reffel2020T5}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="1910.13461" id="2955">
        <attvalues>
          <attvalue for="5" value=" N{atural} Language Processing (NLP) is undergoing a paradigm shift with the open-source of large-scale pretrained language models (PLMs), such as GPT \cite{radford2018gpt,radford2019language}, BERT \cite{devlin-etal-2019-bert}, ALBERT \cite{Lan2020ALBERT}, RoBERTa \cite{liu2020roberta}, BART \cite{lewis-etal-2020-bart}, and T5 \cite{Reffel2020T5}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="1706.03762" id="2957">
        <attvalues>
          <attvalue for="5" value=" Having Transformer \cite{NIPS2017_3f5ee243} as their basic neural architecture, these PLMs are trained on broad data using different self-supervised learning tasks at scale." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2108.07258" id="2958">
        <attvalues>
          <attvalue for="5" value=" They play a central role as the foundation model of AI \cite{bommasani2021-foundation} for their knowledgeable yet incomplete character." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="1908.10063" id="2959">
        <attvalues>
          <attvalue for="5" value=" A bunch of evidence have surfaced showing that their performance can degrade when they are applied to a narrower domain where data varies substantially from the pretraining corpus \cite{thompson-etal-2019-overcoming,araci2020finbert,chalkidis-etal-2020-legal,miller-etal-2021-domain}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2010.02559" id="2960">
        <attvalues>
          <attvalue for="5" value=" A bunch of evidence have surfaced showing that their performance can degrade when they are applied to a narrower domain where data varies substantially from the pretraining corpus \cite{thompson-etal-2019-overcoming,araci2020finbert,chalkidis-etal-2020-legal,miller-etal-2021-domain}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2006.00632" id="2961">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys \cite{pan2009survey,weiss2016survey} provide a holistic view including but not limited to DA; DA for visual applications \cite{patel2015visual,csurka2017comprehensive,wang2018deep}; multiple-source domain adaptation (MDA) \cite{mansour2008domain,sun2015survey}; neural UDA for NLP applications based on shallow and non-pretrained language models \cite{ramponi-plank-2020-neural}; DA and MDA for machine translation \cite{saunders2022domain}; taxonomy of PLMs \cite{qiu2020pre} and comprehensive guide to use PLMs for NLP tasks \cite{min2021recent} and particularly for text generation tasks \cite{li2021pretrained}; parameter-efficient adaptation methods for PLMs \cite{ding2022delta}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2104.06951" id="2962">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys \cite{pan2009survey,weiss2016survey} provide a holistic view including but not limited to DA; DA for visual applications \cite{patel2015visual,csurka2017comprehensive,wang2018deep}; multiple-source domain adaptation (MDA) \cite{mansour2008domain,sun2015survey}; neural UDA for NLP applications based on shallow and non-pretrained language models \cite{ramponi-plank-2020-neural}; DA and MDA for machine translation \cite{saunders2022domain}; taxonomy of PLMs \cite{qiu2020pre} and comprehensive guide to use PLMs for NLP tasks \cite{min2021recent} and particularly for text generation tasks \cite{li2021pretrained}; parameter-efficient adaptation methods for PLMs \cite{ding2022delta}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2010.11478" id="2963">
        <attvalues>
          <attvalue for="5" value=" However, traditional domain adaptation methods that work with shallow neural networks or non-pretrained language models can be unfavorable to PLMs \cite{ryu2022knowledge}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2009.07806" id="2964">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{wright-augenstein-2020-transformer,karouzos-etal-2021-udalm} find that domain adversarial training on top of BERT is unstable and has little effect on cross-domain performance, suggesting that proper design of domain adaptation methods is necessary for PLMs." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2104.07078" id="2965">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{wright-augenstein-2020-transformer,karouzos-etal-2021-udalm} find that domain adversarial training on top of BERT is unstable and has little effect on cross-domain performance, suggesting that proper design of domain adaptation methods is necessary for PLMs." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2103.11332" id="2966">
        <attvalues>
          <attvalue for="5" value=" Developing domain adaptation and generalization methods for PLMs is promising particularly for data-hungry NLP tasks such as abstractive summarization \cite{yu-etal-2021-adaptsum}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="1802.03601" id="2967">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys \cite{pan2009survey,weiss2016survey} provide a holistic view including but not limited to DA; DA for visual applications \cite{patel2015visual,csurka2017comprehensive,wang2018deep}; multiple-source domain adaptation (MDA) \cite{mansour2008domain,sun2015survey}; neural UDA for NLP applications based on shallow and non-pretrained language models \cite{ramponi-plank-2020-neural}; DA and MDA for machine translation \cite{saunders2022domain}; taxonomy of PLMs \cite{qiu2020pre} and comprehensive guide to use PLMs for NLP tasks \cite{min2021recent} and particularly for text generation tasks \cite{li2021pretrained}; parameter-efficient adaptation methods for PLMs \cite{ding2022delta}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2003.08271" id="2968">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys \cite{pan2009survey,weiss2016survey} provide a holistic view including but not limited to DA; DA for visual applications \cite{patel2015visual,csurka2017comprehensive,wang2018deep}; multiple-source domain adaptation (MDA) \cite{mansour2008domain,sun2015survey}; neural UDA for NLP applications based on shallow and non-pretrained language models \cite{ramponi-plank-2020-neural}; DA and MDA for machine translation \cite{saunders2022domain}; taxonomy of PLMs \cite{qiu2020pre} and comprehensive guide to use PLMs for NLP tasks \cite{min2021recent} and particularly for text generation tasks \cite{li2021pretrained}; parameter-efficient adaptation methods for PLMs \cite{ding2022delta}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2111.01243" id="2969">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys \cite{pan2009survey,weiss2016survey} provide a holistic view including but not limited to DA; DA for visual applications \cite{patel2015visual,csurka2017comprehensive,wang2018deep}; multiple-source domain adaptation (MDA) \cite{mansour2008domain,sun2015survey}; neural UDA for NLP applications based on shallow and non-pretrained language models \cite{ramponi-plank-2020-neural}; DA and MDA for machine translation \cite{saunders2022domain}; taxonomy of PLMs \cite{qiu2020pre} and comprehensive guide to use PLMs for NLP tasks \cite{min2021recent} and particularly for text generation tasks \cite{li2021pretrained}; parameter-efficient adaptation methods for PLMs \cite{ding2022delta}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2201.05273" id="2970">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys \cite{pan2009survey,weiss2016survey} provide a holistic view including but not limited to DA; DA for visual applications \cite{patel2015visual,csurka2017comprehensive,wang2018deep}; multiple-source domain adaptation (MDA) \cite{mansour2008domain,sun2015survey}; neural UDA for NLP applications based on shallow and non-pretrained language models \cite{ramponi-plank-2020-neural}; DA and MDA for machine translation \cite{saunders2022domain}; taxonomy of PLMs \cite{qiu2020pre} and comprehensive guide to use PLMs for NLP tasks \cite{min2021recent} and particularly for text generation tasks \cite{li2021pretrained}; parameter-efficient adaptation methods for PLMs \cite{ding2022delta}." />
        </attvalues>
      </edge>
      <edge source="2211.03154" target="2203.06904" id="2971">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys \cite{pan2009survey,weiss2016survey} provide a holistic view including but not limited to DA; DA for visual applications \cite{patel2015visual,csurka2017comprehensive,wang2018deep}; multiple-source domain adaptation (MDA) \cite{mansour2008domain,sun2015survey}; neural UDA for NLP applications based on shallow and non-pretrained language models \cite{ramponi-plank-2020-neural}; DA and MDA for machine translation \cite{saunders2022domain}; taxonomy of PLMs \cite{qiu2020pre} and comprehensive guide to use PLMs for NLP tasks \cite{min2021recent} and particularly for text generation tasks \cite{li2021pretrained}; parameter-efficient adaptation methods for PLMs \cite{ding2022delta}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1706.03762" id="2973">
        <attvalues>
          <attvalue for="5" value=" By leveraging deep neural networks, eg,~RNNs~\cite{ sutskever2014sequence} and Transformers~\cite{vaswani2017attention}, a LM encodes the conversation to a low-dimensional dialogue state and predicts an utterance, but steering such generation for particular purposes remains an open question." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1909.08593" id="2974">
        <attvalues>
          <attvalue for="5" value=" Several works studied ways to fine-tune a LM to generate texts with specific contexts~\cite{ziegler2019fine, ficler2017controlling}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1702.02429" id="2975">
        <attvalues>
          <attvalue for="5" value=" Other results learned a single steerable LM that is capable of generating utterances for multiple specific intents~\cite{gu2017trainable, chen2018stable, subramani2019can, dathathri2019plug}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1804.10731" id="2976">
        <attvalues>
          <attvalue for="5" value=" Earlier work relies on specific, hand-crafted semantic states~\cite{levin1997stochastic, singh2002optimizing, walker2000application} or partially observable belief states~\cite{williams2007partially, young2010hidden}, in which the agent chooses the best hand-crafted dialogue act at each turn, with the goal of either satisfying the user~\cite{shah2018bootstrapping}, completing the task~\cite{shi2018sentiment}, or responding to the user's query~\cite{serban2017deep}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1709.02349" id="2977">
        <attvalues>
          <attvalue for="5" value=" Our MoE-LM can be seen as a special case of hierarchical LMs (eg,~\cite{serban2017deep,zhao2019rethinking,saleh2020hierarchical}), but it is different than them because it learns both the LMs (experts) and the DM." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1907.00456" id="2978">
        <attvalues>
          <attvalue for="5" value=" On the other hand, more recent approaches use deep learning to extract semantic representations from conversation histories, treat these representations as dialogue belief states, and apply RL to learn a word-level generative DM agent~\cite{jaques2019way, li2016deep, li2017adversarial, shin2020generating}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1902.08858" id="2979">
        <attvalues>
          <attvalue for="5" value=" Our MoE-LM can be seen as a special case of hierarchical LMs (eg,~\cite{serban2017deep,zhao2019rethinking,saleh2020hierarchical}), but it is different than them because it learns both the LMs (experts) and the DM." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1606.01541" id="2980">
        <attvalues>
          <attvalue for="5" value=" Another issue is that RL only optimizes a scalar reward, while the aforementioned methods often need to optimize for both the quality of the generated utterance, eg,~ease of answering~\cite{li2016deep} and fluency~\cite{li2017adversarial, li2019dialogue}, and the goal, eg,~conversation length \cite{zhou2020design}, user's sentiment~\cite{hancock2019learning}, and task completion~\cite{verma2022chai}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1701.06547" id="2981">
        <attvalues>
          <attvalue for="5" value=" Another issue is that RL only optimizes a scalar reward, while the aforementioned methods often need to optimize for both the quality of the generated utterance, eg,~ease of answering~\cite{li2016deep} and fluency~\cite{li2017adversarial, li2019dialogue}, and the goal, eg,~conversation length \cite{zhou2020design}, user's sentiment~\cite{hancock2019learning}, and task completion~\cite{verma2022chai}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1812.08989" id="2982">
        <attvalues>
          <attvalue for="5" value=" Another issue is that RL only optimizes a scalar reward, while the aforementioned methods often need to optimize for both the quality of the generated utterance, eg,~ease of answering~\cite{li2016deep} and fluency~\cite{li2017adversarial, li2019dialogue}, and the goal, eg,~conversation length \cite{zhou2020design}, user's sentiment~\cite{hancock2019learning}, and task completion~\cite{verma2022chai}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1901.05415" id="2983">
        <attvalues>
          <attvalue for="5" value=" Another issue is that RL only optimizes a scalar reward, while the aforementioned methods often need to optimize for both the quality of the generated utterance, eg,~ease of answering~\cite{li2016deep} and fluency~\cite{li2017adversarial, li2019dialogue}, and the goal, eg,~conversation length \cite{zhou2020design}, user's sentiment~\cite{hancock2019learning}, and task completion~\cite{verma2022chai}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="2204.08426" id="2984">
        <attvalues>
          <attvalue for="5" value=" Another issue is that RL only optimizes a scalar reward, while the aforementioned methods often need to optimize for both the quality of the generated utterance, eg,~ease of answering~\cite{li2016deep} and fluency~\cite{li2017adversarial, li2019dialogue}, and the goal, eg,~conversation length \cite{zhou2020design}, user's sentiment~\cite{hancock2019learning}, and task completion~\cite{verma2022chai}." />
        </attvalues>
      </edge>
      <edge source="2206.00059" target="1909.07547" id="2985">
        <attvalues>
          <attvalue for="5" value=" Our MoE-LM can be seen as a special case of hierarchical LMs (eg,~\cite{serban2017deep,zhao2019rethinking,saleh2020hierarchical}), but it is different than them because it learns both the LMs (experts) and the DM." />
        </attvalues>
      </edge>
      <edge source="2307.03692" target="2212.10560" id="2993">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) finetuned on instruct data can behave like conversational agents (Alpaca: \cite{alpaca}, Self-Instruct: \cite{wang2023selfinstruct})." />
        </attvalues>
      </edge>
      <edge source="2307.03692" target="2301.13688" id="2994">
        <attvalues>
          <attvalue for="5" value=" The recipe for a chat model is well-defined: one needs to perform instruction tuning, which means supervised finetuning (SFT) of an LLM on tuples of instruction and response (\cite{longpre2023flan})." />
        </attvalues>
      </edge>
      <edge source="2307.03692" target="2305.11206" id="2995">
        <attvalues>
          <attvalue for="5" value=" Ablation studies have shown that both the diversity and quality of the training data play a crucial role in model performance (\cite{chen2023maybe}, \cite{zhou2023lima})." />
        </attvalues>
      </edge>
      <edge source="2307.03692" target="2205.01068" id="2997">
        <attvalues>
          <attvalue for="5" value=" In addition, there are more than a dozen open-source base LLMs, such as LLaMA (\cite{touvron2023llama}), OPT (\cite{zhang2022opt}), GPT-Neo (\cite{gao2020pile}), Palmyra (\cite{Palmyra}), and others, which result in a plethora of possible combinations leading to distinct instruct models." />
        </attvalues>
      </edge>
      <edge source="2307.03692" target="2101.00027" id="2998">
        <attvalues>
          <attvalue for="5" value=" In addition, there are more than a dozen open-source base LLMs, such as LLaMA (\cite{touvron2023llama}), OPT (\cite{zhang2022opt}), GPT-Neo (\cite{gao2020pile}), Palmyra (\cite{Palmyra}), and others, which result in a plethora of possible combinations leading to distinct instruct models." />
        </attvalues>
      </edge>
      <edge source="2307.03692" target="2305.15717" id="2999">
        <attvalues>
          <attvalue for="5" value=" Moreover, prolonged instruction tuning can decrease the foundational model knowledge (\cite{gudibande2023false}) and can be seen as the out-of-distribution task for a downstream task of instruct-tuning (\cite{kumar2022finetuning})." />
        </attvalues>
      </edge>
      <edge source="2307.03692" target="1503.02531" id="3000">
        <attvalues>
          <attvalue for="5" value=" Little is known about the qualitative impact of the distillation process on the base model (\cite{hinton2015distilling})." />
        </attvalues>
      </edge>
      <edge source="2307.03692" target="2202.10054" id="3002">
        <attvalues>
          <attvalue for="5" value=" Moreover, prolonged instruction tuning can decrease the foundational model knowledge (\cite{gudibande2023false}) and can be seen as the out-of-distribution task for a downstream task of instruct-tuning (\cite{kumar2022finetuning})." />
        </attvalues>
      </edge>
      <edge source="2010.02352" target="1904.09324" id="3034">
        <attvalues>
          <attvalue for="5" value=" These issues can be mitigated by later re-masking a token to repair it~\cite{ghazvininejadETAL:19} or by adapting the model to incorrect contexts~\cite{Ghazvininejad2020SemiAutoregressiveTI}." />
        </attvalues>
      </edge>
      <edge source="2010.02352" target="1802.06901" id="3035">
        <attvalues>
          <attvalue for="5" value=" The CMLM's simplicity and its clear links to the very active field of linguistic representation learning are advantages over its semi-autoregressive competitors, such as iterative refinement of token sequences~\cite{leeETAL:18}, refinement of non-linguistic intermediate representations~\cite{kaiser2018fast,shu2020latent} and learning to predict parallel edit operations~\cite{stern2019insertion,gu2019levenshtein}." />
        </attvalues>
      </edge>
      <edge source="2010.02352" target="1803.03382" id="3036">
        <attvalues>
          <attvalue for="5" value=" The CMLM's simplicity and its clear links to the very active field of linguistic representation learning are advantages over its semi-autoregressive competitors, such as iterative refinement of token sequences~\cite{leeETAL:18}, refinement of non-linguistic intermediate representations~\cite{kaiser2018fast,shu2020latent} and learning to predict parallel edit operations~\cite{stern2019insertion,gu2019levenshtein}." />
        </attvalues>
      </edge>
      <edge source="2010.02352" target="1908.07181" id="3037">
        <attvalues>
          <attvalue for="5" value=" The CMLM's simplicity and its clear links to the very active field of linguistic representation learning are advantages over its semi-autoregressive competitors, such as iterative refinement of token sequences~\cite{leeETAL:18}, refinement of non-linguistic intermediate representations~\cite{kaiser2018fast,shu2020latent} and learning to predict parallel edit operations~\cite{stern2019insertion,gu2019levenshtein}." />
        </attvalues>
      </edge>
      <edge source="2010.02352" target="1902.03249" id="3038">
        <attvalues>
          <attvalue for="5" value=" The CMLM's simplicity and its clear links to the very active field of linguistic representation learning are advantages over its semi-autoregressive competitors, such as iterative refinement of token sequences~\cite{leeETAL:18}, refinement of non-linguistic intermediate representations~\cite{kaiser2018fast,shu2020latent} and learning to predict parallel edit operations~\cite{stern2019insertion,gu2019levenshtein}." />
        </attvalues>
      </edge>
      <edge source="2010.02352" target="1711.02281" id="3040">
        <attvalues>
          <attvalue for="5" value=" Each successive iteration provides mode-breaking~\cite{guETAL:18} context for the next." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2303.08774" id="3042">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs) \cite{achiam2023gpt,touvron2023llama,jiang2023mistral}, featuring billions to trillions of parameters, marks significant progress in diverse language-related tasks \cite{chang2024survey,thirunavukarasu2023large,zhang2023don,wu2023bloomberggpt,beltagy2019scibert}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2307.09288" id="3043">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs) \cite{achiam2023gpt,touvron2023llama,jiang2023mistral}, featuring billions to trillions of parameters, marks significant progress in diverse language-related tasks \cite{chang2024survey,thirunavukarasu2023large,zhang2023don,wu2023bloomberggpt,beltagy2019scibert}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2310.06825" id="3044">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs) \cite{achiam2023gpt,touvron2023llama,jiang2023mistral}, featuring billions to trillions of parameters, marks significant progress in diverse language-related tasks \cite{chang2024survey,thirunavukarasu2023large,zhang2023don,wu2023bloomberggpt,beltagy2019scibert}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2307.03109" id="3045">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs) \cite{achiam2023gpt,touvron2023llama,jiang2023mistral}, featuring billions to trillions of parameters, marks significant progress in diverse language-related tasks \cite{chang2024survey,thirunavukarasu2023large,zhang2023don,wu2023bloomberggpt,beltagy2019scibert}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2305.16339" id="3046">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs) \cite{achiam2023gpt,touvron2023llama,jiang2023mistral}, featuring billions to trillions of parameters, marks significant progress in diverse language-related tasks \cite{chang2024survey,thirunavukarasu2023large,zhang2023don,wu2023bloomberggpt,beltagy2019scibert}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2303.17564" id="3047">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs) \cite{achiam2023gpt,touvron2023llama,jiang2023mistral}, featuring billions to trillions of parameters, marks significant progress in diverse language-related tasks \cite{chang2024survey,thirunavukarasu2023large,zhang2023don,wu2023bloomberggpt,beltagy2019scibert}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="1903.10676" id="3048">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs) \cite{achiam2023gpt,touvron2023llama,jiang2023mistral}, featuring billions to trillions of parameters, marks significant progress in diverse language-related tasks \cite{chang2024survey,thirunavukarasu2023large,zhang2023don,wu2023bloomberggpt,beltagy2019scibert}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2305.18654" id="3049">
        <attvalues>
          <attvalue for="5" value=" Nonetheless, the removal of recurrence imposes significant limitations on many reasoning tasks, as shown in multiple previous works~\cite{deletang2022neural,dziri2024faith}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2302.08399" id="3050">
        <attvalues>
          <attvalue for="5" value=" However, growing concerns have arisen over the limitations~\cite{dziri2024faith,valmeekam2022large,ullman2023large} of current LLMs, particularly their difficulties with basic tasks such as multiplication or counting." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2306.15895" id="3051">
        <attvalues>
          <attvalue for="5" value=" While much of the debate centers on training techniques and data choice~\cite{yu2024large}, it is crucial to also consider the theoretical limitations of the computational capabilities of these models, which fundamentally depend on their core architecture, Transformers~\cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="1706.03762" id="3052">
        <attvalues>
          <attvalue for="5" value=" While much of the debate centers on training techniques and data choice~\cite{yu2024large}, it is crucial to also consider the theoretical limitations of the computational capabilities of these models, which fundamentally depend on their core architecture, Transformers~\cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="1912.11511" id="3053">
        <attvalues>
          <attvalue for="5" value=" In contrast to deterministic models like state machines or $K$ Nearest Neighbor classifiers, whose computational power is entirely reliant on their architectural (algorithm) design, the power of Neural Networks hinges upon a combination of both architecture~\cite{zhou2019analysis} and network optimization~\cite{deletang2022neural}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2207.02098" id="3054">
        <attvalues>
          <attvalue for="5" value=" Nonetheless, the removal of recurrence imposes significant limitations on many reasoning tasks, as shown in multiple previous works~\cite{deletang2022neural,dziri2024faith}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="1906.08482" id="3056">
        <attvalues>
          <attvalue for="5" value=" However, recurrent networks face significant optimization challenges~\cite{alqushaibi2020review}, such as the inability to parallelize during training and susceptibility to gradient vanishing~\cite{hochreiter1998vanishing} or exploding~\cite{kanai2017preventing} with longer sequences~\cite{ribeiro2020beyond}, which limits their scalability with large models and datasets." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2005.00928" id="3057">
        <attvalues>
          <attvalue for="5" value=" Conversely, the Transformer model replaces recurrence with an attention mechanism, enabling parallel training and mitigating the gradient vanishing issue through multiple gradient paths~\cite{abnar2020quantifying}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2001.08361" id="3058">
        <attvalues>
          <attvalue for="5" value=" This innovation has made Transformers the leading choice for scalability~\cite{kaplan2020scaling} and optimization efficiency with large training data and model sizes." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2201.11903" id="3059">
        <attvalues>
          <attvalue for="5" value=" The introduction of Chain of Thought (CoT) prompting~\cite{wei2022chain} represents a significant advancement for transformer-based language models, greatly enhancing performance across a range of tasks~\cite{chu2023survey}, including those beyond the computational capacity of the Transformer architecture." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2402.12875" id="3060">
        <attvalues>
          <attvalue for="5" value=" Despite substantial research analyzing the logic behind CoT, much of it interprets CoT from a psychological perspective~\cite{miao2024chain,li2024chain} as this way of reasoning is more human-like." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2309.14316" id="3061">
        <attvalues>
          <attvalue for="5" value=" Additionally, previous studies have examined CoT's role in knowledge extraction in LLMs~\cite{zhu2023physics}, which can differ from its role in reasoning processes." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="1807.03819" id="3062">
        <attvalues>
          <attvalue for="5" value=" Lastly, we revisit recent efforts to modify the Transformer architecture to be recurrent, including various architectural designs such as the universal Transformer~\cite{dehghani2018universal} and the linear Transformer~\cite{katharopoulos2020transformers}." />
        </attvalues>
      </edge>
      <edge source="2409.09239" target="2006.16236" id="3063">
        <attvalues>
          <attvalue for="5" value=" Lastly, we revisit recent efforts to modify the Transformer architecture to be recurrent, including various architectural designs such as the universal Transformer~\cite{dehghani2018universal} and the linear Transformer~\cite{katharopoulos2020transformers}." />
        </attvalues>
      </edge>
      <edge source="2409.00222" target="2105.06603" id="3064">
        <attvalues>
          <attvalue for="5" value=" In Zero-shot Stance Detection (ZSSD), a model predicts stances for targets it has not seen during training, which is crucial since collecting training data for every potential target is impractical \cite{allaway2021adversarial}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="1907.13528" id="3065">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{ettinger-2020-bert} and \cite{kassner-schutze-2020-negated} show that BERT~\cite{devlin-etal-2019-bert} fails to distinguish between negated and non-negated cloze questions." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="1911.03343" id="3066">
        <attvalues>
          <attvalue for="5" value=" \cite{hosseini-etal-2021-understanding} combine unlikelihood training and syntactic data augmentation to enhance the ability of BERT to understand negation with negated LAMA~\cite{kassner-schutze-2020-negated}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="2005.14165" id="3068">
        <attvalues>
          <attvalue for="5" value=" Researchers have also shown that large language models such as GPT-3 \cite{NEURIPS2020_1457c0d6} and InstructGPT \cite{ouyang2022training} are insensitive to negation and fail to reason under negation \cite{truong-etal-2023-language}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="2306.08189" id="3070">
        <attvalues>
          <attvalue for="5" value=" Researchers have also shown that large language models such as GPT-3 \cite{NEURIPS2020_1457c0d6} and InstructGPT \cite{ouyang2022training} are insensitive to negation and fail to reason under negation \cite{truong-etal-2023-language}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="2205.03815" id="3071">
        <attvalues>
          <attvalue for="5" value=" \cite{jang-etal-2022-beyond} point out that language models violate the logical negation property ($p$ is true iff $\neg p$ is false)." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="2203.08929" id="3072">
        <attvalues>
          <attvalue for="5" value=" In addition to CondaQA, we experiment with five of the eight corpora analyzed by \cite{hossain-etal-2022-analysis}: CommonsenseQA~\cite{talmor-etal-2019-commonsenseqa}, STS-B~\cite{cer-etal-2017-semeval}, QNLI~\cite{rajpurkar-etal-2016-squad}, WiC~\cite{pilehvar-camacho-collados-2019-wic}, and WSC~\cite{levesque_winograd_2012}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="1811.00937" id="3074">
        <attvalues>
          <attvalue for="5" value=" In addition to CondaQA, we experiment with five of the eight corpora analyzed by \cite{hossain-etal-2022-analysis}: CommonsenseQA~\cite{talmor-etal-2019-commonsenseqa}, STS-B~\cite{cer-etal-2017-semeval}, QNLI~\cite{rajpurkar-etal-2016-squad}, WiC~\cite{pilehvar-camacho-collados-2019-wic}, and WSC~\cite{levesque_winograd_2012}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="1606.05250" id="3075">
        <attvalues>
          <attvalue for="5" value=" In addition to CondaQA, we experiment with five of the eight corpora analyzed by \cite{hossain-etal-2022-analysis}: CommonsenseQA~\cite{talmor-etal-2019-commonsenseqa}, STS-B~\cite{cer-etal-2017-semeval}, QNLI~\cite{rajpurkar-etal-2016-squad}, WiC~\cite{pilehvar-camacho-collados-2019-wic}, and WSC~\cite{levesque_winograd_2012}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="2210.14486" id="3077">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{hossain-blanco-2022-leveraging} present Large-AFIN, over 153,000 pairs of sentences with negation and their affirmative interpretations obtained from parallel corpora via backtranslation." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="2205.04012" id="3079">
        <attvalues>
          <attvalue for="5" value=" Related Work Early research on negation targeted detecting negating cues and generating semantic representations, usually by identifying the scope and focus \cite{Morante2011NegationCues,morante-daelemans-2012-conandoyle,van-son-etal-2016-building,khandelwal-sawant-2020-negbert,truong-etal-2022-improving}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="2105.03519" id="3080">
        <attvalues>
          <attvalue for="5" value=" \cite{hosseini-etal-2021-understanding} combine unlikelihood training and syntactic data augmentation to enhance the ability of BERT to understand negation with negated LAMA~\cite{kassner-schutze-2020-negated}." />
        </attvalues>
      </edge>
      <edge source="2406.07492" target="2205.11467" id="3081">
        <attvalues>
          <attvalue for="5" value=" \cite{hossain-etal-2022-question} present AFIN, a corpus of $\approx$3,000 sentences with negations and their affirmative interpretations." />
        </attvalues>
      </edge>
      <edge source="2203.08929" target="1606.05250" id="3082">
        <attvalues>
          <attvalue for="5" value=" This contrasts with previous corpora, where the questions were written by annotators after being told the answer~\cite{rajpurkar-etal-2016-squad}." />
        </attvalues>
      </edge>
      <edge source="2203.08929" target="1803.02324" id="3083">
        <attvalues>
          <attvalue for="5" value=" For example, annotators are likely to use negation when asked to write a text that contradicts something despite contradictions in the wild need not have a negation~\cite{gururangan-etal-2018-annotation}." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="1609.01454" id="3084">
        <attvalues>
          <attvalue for="5" value=" Traditional supervised slot filling models and sequence labeling methods \cite{liu2015recurrent,Liu2016AttentionBasedRN,goo2018slot,niu2019novel,he2020multi,he2020learning,Wang2022InstructionNERAM} have shown remarkable performance." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="1907.00390" id="3085">
        <attvalues>
          <attvalue for="5" value=" Traditional supervised slot filling models and sequence labeling methods \cite{liu2015recurrent,Liu2016AttentionBasedRN,goo2018slot,niu2019novel,he2020multi,he2020learning,Wang2022InstructionNERAM} have shown remarkable performance." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="2203.03903" id="3086">
        <attvalues>
          <attvalue for="5" value=" Traditional supervised slot filling models and sequence labeling methods \cite{liu2015recurrent,Liu2016AttentionBasedRN,goo2018slot,niu2019novel,he2020multi,he2020learning,Wang2022InstructionNERAM} have shown remarkable performance." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="2204.04391" id="3087">
        <attvalues>
          <attvalue for="5" value=" However, these models tend to memorize inherent patterns of entities and contexts \cite{wang2022miner,lin2021rockner}." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="2109.05620" id="3088">
        <attvalues>
          <attvalue for="5" value=" However, these models tend to memorize inherent patterns of entities and contexts \cite{wang2022miner,lin2021rockner}." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="2104.06393" id="3089">
        <attvalues>
          <attvalue for="5" value=" Most existing studies \cite{wu2021bridging,moradi2021evaluating,gui2021textflint} that explored the robustness problem are only about rule-based synthetic datasets, which have certain limitations." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="2103.11441" id="3091">
        <attvalues>
          <attvalue for="5" value=" Most existing studies \cite{wu2021bridging,moradi2021evaluating,gui2021textflint} that explored the robustness problem are only about rule-based synthetic datasets, which have certain limitations." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="2005.07162" id="3092">
        <attvalues>
          <attvalue for="5" value=" Further, \cite{namysl2020nat} focused on the robustness of the NER model against Optical Character Recognition (OCR) disturbance and misspellings." />
        </attvalues>
      </edge>
      <edge source="2208.11508" target="2012.15262" id="3093">
        <attvalues>
          <attvalue for="5" value=" \cite{liu2020robustness} proposed Language understanding augmentation, which contains four data augmentation methods, to simulate natural perturbations." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="1906.00556" id="3095">
        <attvalues>
          <attvalue for="5" value=" In particular, downstream tasks explored with disfluency contexts include punctuation restoration \cite{wang-etal-2014-combining,Lin2020}, machine translation \cite{salesky-etal-2019-fluent,wangetal}, syntactic parsing \cite{yoshikawa-etal-2016-joint, honnibal-johnson-2014-joint, rasooli-tetreault-2013-joint,jamshid-lou-johnson-2020-improving} and question answering \cite{gupta-etal-2021-disfl}." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="2106.04016" id="3097">
        <attvalues>
          <attvalue for="5" value=" \cite{gupta-etal-2021-disfl} who present a disfluent derivative of the question answering dataset SQUAD \cite{rajpurkar-etal-2016-squad}, a possible strategy to create a disfluent intent detection and slot filling dataset is to manually add contextual disfluencies into an existing fluent intent detection and slot filling dataset." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="1606.05250" id="3098">
        <attvalues>
          <attvalue for="5" value=" \cite{gupta-etal-2021-disfl} who present a disfluent derivative of the question answering dataset SQUAD \cite{rajpurkar-etal-2016-squad}, a possible strategy to create a disfluent intent detection and slot filling dataset is to manually add contextual disfluencies into an existing fluent intent detection and slot filling dataset." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="1805.10190" id="3099">
        <attvalues>
          <attvalue for="5" value=" This process could be performed for English with many publicly available intent detection and slot filling datasets \cite{price-1990-evaluation,coucke2018snips}." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="2104.02021" id="3100">
        <attvalues>
          <attvalue for="5" value=" First, we create a dataset with disfluency annotations by manually adding contextual disfluencies as distractors into the fluent dataset PhoATIS \cite{dao21_interspeech} which is the only current dataset publicly available for Vietnamese intent detection and slot filling." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="1911.02116" id="3101">
        <attvalues>
          <attvalue for="5" value=" We conduct experiments using strong baseline models that are based on pre-trained language models {XLM-R} \cite{conneau2019unsupervised} and {PhoBERT} \cite{nguyen2020phobert}." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="2003.00744" id="3102">
        <attvalues>
          <attvalue for="5" value=" Experimental results show that: (i) disfluencies negatively affect the performances of the downstream intent detection and slot filling tasks, and (ii) in the disfluency context, the pre-trained multilingual language model XLM-R is more effective for the intent detection and slot filling tasks than the pre-trained monolingual language model PhoBERT, and this is completely opposite to what is generally found in the fluency context with other Vietnamese NLP tasks \cite{nguyen2020phobert,vitext2sql,PhoNER_COVID19}." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="2010.01891" id="3103">
        <attvalues>
          <attvalue for="5" value=" Experimental results show that: (i) disfluencies negatively affect the performances of the downstream intent detection and slot filling tasks, and (ii) in the disfluency context, the pre-trained multilingual language model XLM-R is more effective for the intent detection and slot filling tasks than the pre-trained monolingual language model PhoBERT, and this is completely opposite to what is generally found in the fluency context with other Vietnamese NLP tasks \cite{nguyen2020phobert,vitext2sql,PhoNER_COVID19}." />
        </attvalues>
      </edge>
      <edge source="2209.08359" target="2104.03879" id="3104">
        <attvalues>
          <attvalue for="5" value=" Experimental results show that: (i) disfluencies negatively affect the performances of the downstream intent detection and slot filling tasks, and (ii) in the disfluency context, the pre-trained multilingual language model XLM-R is more effective for the intent detection and slot filling tasks than the pre-trained monolingual language model PhoBERT, and this is completely opposite to what is generally found in the fluency context with other Vietnamese NLP tasks \cite{nguyen2020phobert,vitext2sql,PhoNER_COVID19}." />
        </attvalues>
      </edge>
      <edge source="2408.12254" target="2109.10952" id="3129">
        <attvalues>
          <attvalue for="5" value=" The data we use is comprised of real child-directed utterances, taken from the CHILDES corpus \cite{macwhinney1998childes}, coupled with a recent method for converting universal dependency annotations to logical forms \cite{ida2023}." />
        </attvalues>
      </edge>
      <edge source="2203.00902" target="2010.15980" id="3139">
        <attvalues>
          <attvalue for="5" value=" Third, what is the effect of automatically searching for prompt template and masked label words? There has been a line of work automatically finding prompts, which results in seemingly unnatural augmented sequences \cite{shin-etal-2020-autoprompt,gao2021making}." />
        </attvalues>
      </edge>
      <edge source="2203.00902" target="2009.07118" id="3141">
        <attvalues>
          <attvalue for="5" value=" Prompt-based fine-tuning has gained increasing attention on NLP \cite{shin-etal-2020-autoprompt,schick-etal-2020-automatically,schick-schutze-2021-just,tanl,gao2021making}." />
        </attvalues>
      </edge>
      <edge source="2203.00902" target="2101.05779" id="3142">
        <attvalues>
          <attvalue for="5" value=" In contrast, schema prompts \cite{lee2021dialogue,tanl} replace a natural language sentence with a structured schema, which makes the prompt more succinct and code-like." />
        </attvalues>
      </edge>
      <edge source="2203.00902" target="2012.15723" id="3143">
        <attvalues>
          <attvalue for="5" value=" Third, what is the effect of automatically searching for prompt template and masked label words? There has been a line of work automatically finding prompts, which results in seemingly unnatural augmented sequences \cite{shin-etal-2020-autoprompt,gao2021making}." />
        </attvalues>
      </edge>
      <edge source="2203.00902" target="1909.01066" id="3144">
        <attvalues>
          <attvalue for="5" value=" NL template prompts \cite{petroni-etal-2019-language,jiang-etal-2020-know,gao2021making} were the earliest proposed and the dominant method." />
        </attvalues>
      </edge>
      <edge source="2203.00902" target="2106.13353" id="3147">
        <attvalues>
          <attvalue for="5" value=" While having been shown effective for several NLI-style classification tasks under few-shot settings \cite{logan2021cutting}, it remains a question whether they are competitive in more general settings." />
        </attvalues>
      </edge>
      <edge source="2003.04036" target="1802.05365" id="3149">
        <attvalues>
          <attvalue for="5" value=" More recently, pretrained language models such as ELMo \cite{peters2018deep}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019XLNet}, and RoBERTa \cite{liu2019roberta} have become the method of choice when encoding text." />
        </attvalues>
      </edge>
      <edge source="2003.04036" target="1907.11692" id="3151">
        <attvalues>
          <attvalue for="5" value=" More recently, pretrained language models such as ELMo \cite{peters2018deep}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019XLNet}, and RoBERTa \cite{liu2019roberta} have become the method of choice when encoding text." />
        </attvalues>
      </edge>
      <edge source="2003.04036" target="cs/0508103" id="3152">
        <attvalues>
          <attvalue for="5" value=" \cite{TurneyLittman2005} proposed identifying such analogies using bag-of-words vector space models." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="2303.12712" id="3153">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have recently demonstrated remarkable abilities across a variety of tasks ~\cite{bubeck2023sparks,xu2023large,zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="2303.18223" id="3154">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have recently demonstrated remarkable abilities across a variety of tasks ~\cite{bubeck2023sparks,xu2023large,zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="2303.11366" id="3155">
        <attvalues>
          <attvalue for="5" value=" The self-reflection process primarily relies on using LLMs to iteratively refine initial drafts through feedback loops, a method that has been widely researched and explored ~\cite{shinn2023reflexion,DBLP:conf/uist/ParkOCMLB23, scheurer2022training,le2022coderl,welleck2022generating, amabile1983theoretical, flower1981cognitive,chen2023equals,simon1962architecture,chen-etal-2023-improving-low,sun2021joint}." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="2304.03442" id="3157">
        <attvalues>
          <attvalue for="5" value=" The self-reflection process primarily relies on using LLMs to iteratively refine initial drafts through feedback loops, a method that has been widely researched and explored ~\cite{shinn2023reflexion,DBLP:conf/uist/ParkOCMLB23, scheurer2022training,le2022coderl,welleck2022generating, amabile1983theoretical, flower1981cognitive,chen2023equals,simon1962architecture,chen-etal-2023-improving-low,sun2021joint}." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="2207.01780" id="3158">
        <attvalues>
          <attvalue for="5" value=" The self-reflection process primarily relies on using LLMs to iteratively refine initial drafts through feedback loops, a method that has been widely researched and explored ~\cite{shinn2023reflexion,DBLP:conf/uist/ParkOCMLB23, scheurer2022training,le2022coderl,welleck2022generating, amabile1983theoretical, flower1981cognitive,chen2023equals,simon1962architecture,chen-etal-2023-improving-low,sun2021joint}." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="2211.00053" id="3159">
        <attvalues>
          <attvalue for="5" value=" The self-reflection process primarily relies on using LLMs to iteratively refine initial drafts through feedback loops, a method that has been widely researched and explored ~\cite{shinn2023reflexion,DBLP:conf/uist/ParkOCMLB23, scheurer2022training,le2022coderl,welleck2022generating, amabile1983theoretical, flower1981cognitive,chen2023equals,simon1962architecture,chen-etal-2023-improving-low,sun2021joint}." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="1611.00179" id="3160">
        <attvalues>
          <attvalue for="5" value=" To address this, we introduce a framework that leverages the inherent duality property ~\cite{he2016dual,qin2020dual,sun2021tibetan,yi2017dualgan,xia2017dual} of translation tasks to provide effective feedback to LLMs, thereby enhancing their reflective capabilities and consequently improving translation performance." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="1704.02510" id="3161">
        <attvalues>
          <attvalue for="5" value=" To address this, we introduce a framework that leverages the inherent duality property ~\cite{he2016dual,qin2020dual,sun2021tibetan,yi2017dualgan,xia2017dual} of translation tasks to provide effective feedback to LLMs, thereby enhancing their reflective capabilities and consequently improving translation performance." />
        </attvalues>
      </edge>
      <edge source="2406.07232" target="1707.00415" id="3162">
        <attvalues>
          <attvalue for="5" value=" To address this, we introduce a framework that leverages the inherent duality property ~\cite{he2016dual,qin2020dual,sun2021tibetan,yi2017dualgan,xia2017dual} of translation tasks to provide effective feedback to LLMs, thereby enhancing their reflective capabilities and consequently improving translation performance." />
        </attvalues>
      </edge>
      <edge source="2305.15444" target="2205.10475" id="3176">
        <attvalues>
          <attvalue for="5" value=" Moreover, a significant gap remains between the best few-shot NER methods and the performance of end-to-end trained models \cite{wang2022deepstruct, xu2022clozing}." />
        </attvalues>
      </edge>
      <edge source="2305.15444" target="2005.14165" id="3177">
        <attvalues>
          <attvalue for="5" value=" In ablations, we show that PromptNER outperforms standard Few-Shot Prompting \cite{brown2020language} and Chain-of-Thought Prompting \cite{wei2022chain}." />
        </attvalues>
      </edge>
      <edge source="2305.15444" target="2208.11464" id="3178">
        <attvalues>
          <attvalue for="5" value=" These few-shot methods struggle when the source and target domains differ with respect to what constitutes an entity \cite{yang2022factmix, das2022container}." />
        </attvalues>
      </edge>
      <edge source="2305.15444" target="cs/0306050" id="3179">
        <attvalues>
          <attvalue for="5" value="48\% F1 score on the CoNLL dataset \cite{sang2003introduction} in a few-shot setting, improving over the best previous few-shot methods by 4\% (absolute)." />
        </attvalues>
      </edge>
      <edge source="2305.15444" target="2105.07464" id="3180">
        <attvalues>
          <attvalue for="5" value=" PromptNER outperforms the best-competing methods by 9\% (absolute) on the GENIA \cite{kim2003genia} dataset and 4\% (absolute) on the FewNERD-Intra \cite{ding2021few} setting and sets a new state of the art on three out of five of the CrossNER \cite{liu2021crossner} target domains, despite using only $2\%$ of the available training data." />
        </attvalues>
      </edge>
      <edge source="2305.15444" target="2012.04373" id="3181">
        <attvalues>
          <attvalue for="5" value=" PromptNER outperforms the best-competing methods by 9\% (absolute) on the GENIA \cite{kim2003genia} dataset and 4\% (absolute) on the FewNERD-Intra \cite{ding2021few} setting and sets a new state of the art on three out of five of the CrossNER \cite{liu2021crossner} target domains, despite using only $2\%$ of the available training data." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="1706.03762" id="3183">
        <attvalues>
          <attvalue for="5" value=" The choice of sequence mixer (eg attention, convolution) in a language model affects both its quality and efficiency~\cite{arora2023zoology,vaswani2018attention}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2209.11895" id="3184">
        <attvalues>
          <attvalue for="5" value=" Prior work shows that attention excels at recall, the ability to ground generations in previously seen tokens~\cite{olsson2022context,arora2023zoology}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2212.10544" id="3185">
        <attvalues>
          <attvalue for="5" value=" The natural question is: can we improve the real-world speed and memory-use of language models without comprising on quality? Recently, a number of architectures have been proposed that enable substantially higher throughput while matching attention in perplexity~\cite{wang2022pretraining,gu2023mamba,yang2023gated,poli2023hyena,peng2023rwkv}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2312.00752" id="3186">
        <attvalues>
          <attvalue for="5" value=" To make \sysname competitive with SoTA attention~\cite{dao2023flashattention2} and recurrent~\cite{gu2023mamba} models under wall-clock and throughput metrics, we introduce several IO-aware optimizations." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2312.06635" id="3187">
        <attvalues>
          <attvalue for="5" value=" The natural question is: can we improve the real-world speed and memory-use of language models without comprising on quality? Recently, a number of architectures have been proposed that enable substantially higher throughput while matching attention in perplexity~\cite{wang2022pretraining,gu2023mamba,yang2023gated,poli2023hyena,peng2023rwkv}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2302.10866" id="3188">
        <attvalues>
          <attvalue for="5" value=" The natural question is: can we improve the real-world speed and memory-use of language models without comprising on quality? Recently, a number of architectures have been proposed that enable substantially higher throughput while matching attention in perplexity~\cite{wang2022pretraining,gu2023mamba,yang2023gated,poli2023hyena,peng2023rwkv}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2305.13048" id="3189">
        <attvalues>
          <attvalue for="5" value=" The natural question is: can we improve the real-world speed and memory-use of language models without comprising on quality? Recently, a number of architectures have been proposed that enable substantially higher throughput while matching attention in perplexity~\cite{wang2022pretraining,gu2023mamba,yang2023gated,poli2023hyena,peng2023rwkv}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2310.06825" id="3190">
        <attvalues>
          <attvalue for="5" value=" While popular architectures use long window sizes (eg 4096 for Mistral-7B \cite{mistral7b}), we choose fixed size $64$ windows, guided by hardware properties." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2212.14052" id="3191">
        <attvalues>
          <attvalue for="5" value=" We hypothesize that this is because linear attention lacks the precision to perform local token shifts and comparisons~\cite{dao2022hungry,arora2023zoology}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2307.08691" id="3192">
        <attvalues>
          <attvalue for="5" value=" To make \sysname competitive with SoTA attention~\cite{dao2023flashattention2} and recurrent~\cite{gu2023mamba} models under wall-clock and throughput metrics, we introduce several IO-aware optimizations." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2205.14135" id="3193">
        <attvalues>
          <attvalue for="5" value=" \begin{enumerate}[leftmargin=*] \item Despite the theoretically improved complexity, linear attention implementations are often slower than well-optimized attention implementations ~\cite{dao2022flashattention}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2209.04881" id="3194">
        <attvalues>
          <attvalue for="5" value=" In \sysname, we use the 2nd-order Taylor approximation of softmax as the linear attention feature map With sequence length $N$ and head dimension $d$, this naïvely requires $\mathcal{O}(Nd^3)$ time and space complexity ~\cite{hedgehog2023,keles2023on}." />
        </attvalues>
      </edge>
      <edge source="2402.18668" target="2307.09288" id="3195">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} In experiments, we show that ${\sysname}$ competes in quality with strong Transformer++ \cite{touvron2023llama} and SoTA sub-quadratic baselines in models up to the 1." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2005.14165" id="3202">
        <attvalues>
          <attvalue for="5" value=" Along this line of research endeavors, LLM-based prompting engineering (PE) methods \cite{liu2023pre,brown2020language} have attracted much attention, partially because they are the key techniques in making full use of the superior capabilities of LLMs via constructing appropriate prompts." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2303.08774" id="3203">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) (eg, GPT-3 \cite{brown2020language}, GPT-4 \cite{GPT4}, LLaMa \cite{touvron2023llama}) make it possible for machines to understand users' attention accurately, thus revolutionizing the human-computer interaction (HCI) paradigm." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2107.13586" id="3205">
        <attvalues>
          <attvalue for="5" value=" \item Second, prompt answer engineering \cite{liu2023pre} refers to the process of searching for an answer space and a map to the original output, which enhances users' understanding of the information encapsulated within the LLM." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="1909.01066" id="3206">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2001.07676" id="3207">
        <attvalues>
          <attvalue for="5" value=" \item Third, multi-prompting methods mainly applied ensemble techniques \cite{schick2020exploiting} to mitigate the sensitivity of LLM to different formulations and to obtain a more stable output." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2010.15980" id="3209">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2101.00190" id="3210">
        <attvalues>
          <attvalue for="5" value=" Such a trend is manifested by the evolution from using discrete prompts (eg, a piece of human-readable text) \cite{jiang2020can, petroni2019language} to continuous ones (eg, a continuous task-specific vector) \cite{li2021prefix, lester2021power}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2103.10385" id="3212">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2104.05240" id="3213">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2012.15723" id="3214">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2108.13161" id="3215">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2105.11259" id="3216">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2104.08691" id="3217">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2109.04332" id="3218">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2205.12548" id="3219">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2310.18358" target="2209.11486" id="3220">
        <attvalues>
          <attvalue for="5" value=" To this end, numerous prompt engineering (PE) methods have been explored with the notable progress of LLM advancement and technologies \cite{liu2023pre,radford2019language,petroni2019language,schick2020exploiting,jiang2020can,shin2020autoprompt,li2021prefix,haviv2021bertese,liu2021gpt,zhong2021factual,gao2020making,zhang2021differentiable,han2022ptr,lester2021power,gu2021ppt,deng2022rlprompt,hou2022metaprompting,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2206.11219" target="1605.06069" id="3223">
        <attvalues>
          <attvalue for="5" value=" Text generation models have continued to gain traction due to groundbreaking progress in neural networks techniques \cite{serban2017hierarchical, guu2018generating, Li2016deeprl}, the recent development of advanced transformer-based architectures \cite{radford2019language}, and training over huge amounts of data \cite{wu2019conditional}." />
        </attvalues>
      </edge>
      <edge source="2206.11219" target="1804.06437" id="3225">
        <attvalues>
          <attvalue for="5" value=" However, aside from traditional \cite{reiter2000building} and rule-based \cite{li2018delete} algorithms, researchers face difficulties understanding the characteristics of the generated corpora." />
        </attvalues>
      </edge>
      <edge source="2206.11219" target="1707.06875" id="3226">
        <attvalues>
          <attvalue for="5" value=" According to \cite{novikova2017we}, these metrics only weakly correlate with human ratings." />
        </attvalues>
      </edge>
      <edge source="2206.11219" target="1904.02792" id="3228">
        <attvalues>
          <attvalue for="5" value=" Last year, \cite{hashimoto2019unifying} suggested a compelling approach that combines both human and automatic metrics." />
        </attvalues>
      </edge>
      <edge source="2203.05598" target="1406.1078" id="3229">
        <attvalues>
          <attvalue for="5" value=" With the improvement of the machine translation quality, and especially with the beginning of the use of deep neural networks since 2014\cite{Kyunghyun_Cho-2014}, it became possible to increase the estimates of translation fluency and adequacy \cite{Andy_Way-2019}." />
        </attvalues>
      </edge>
      <edge source="2203.05598" target="1803.08409" id="3230">
        <attvalues>
          <attvalue for="5" value=" With the improvement of the machine translation quality, and especially with the beginning of the use of deep neural networks since 2014\cite{Kyunghyun_Cho-2014}, it became possible to increase the estimates of translation fluency and adequacy \cite{Andy_Way-2019}." />
        </attvalues>
      </edge>
      <edge source="2203.05598" target="2105.03311" id="3231">
        <attvalues>
          <attvalue for="5" value=" The principles and methods used to assess the quality of machine translation are described a lot \cite{Lifeng_Han-2021}." />
        </attvalues>
      </edge>
      <edge source="2111.12790" target="1310.4546" id="3234">
        <attvalues>
          <attvalue for="5" value=" Moreover, approaches to solving language tasks have evolved rapidly, from bag of words models which rely on a small number of fixed words represented as strings, without underlying meaning, to fixed dense word representations such as word2vec and GloVe \cite{Mikolov2013DistributedRO,pennington-etal-2014-glove} and large contextualized representations of language \cite{peters-etal-2018-deep,devlin-etal-2019-bert} that are trained on task-independent text to provide a backbone representation for word meaning." />
        </attvalues>
      </edge>
      <edge source="2111.12790" target="1802.05365" id="3235">
        <attvalues>
          <attvalue for="5" value=" Moreover, approaches to solving language tasks have evolved rapidly, from bag of words models which rely on a small number of fixed words represented as strings, without underlying meaning, to fixed dense word representations such as word2vec and GloVe \cite{Mikolov2013DistributedRO,pennington-etal-2014-glove} and large contextualized representations of language \cite{peters-etal-2018-deep,devlin-etal-2019-bert} that are trained on task-independent text to provide a backbone representation for word meaning." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="1510.01032" id="3249">
        <attvalues>
          <attvalue for="5" value=" Various approaches have been developed that use supervision from known word pairs \cite{kamper2016deep,hu2020multilingual,settle2016discriminative,settle2017query}, but here we focus on unsupervised learning of AWEs, where only raw audio is available \cite{holzenberger2018learning,kamper2019truly,van2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="2006.14007" id="3250">
        <attvalues>
          <attvalue for="5" value=" Various approaches have been developed that use supervision from known word pairs \cite{kamper2016deep,hu2020multilingual,settle2016discriminative,settle2017query}, but here we focus on unsupervised learning of AWEs, where only raw audio is available \cite{holzenberger2018learning,kamper2019truly,van2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="1706.03818" id="3251">
        <attvalues>
          <attvalue for="5" value=" Various approaches have been developed that use supervision from known word pairs \cite{kamper2016deep,hu2020multilingual,settle2016discriminative,settle2017query}, but here we focus on unsupervised learning of AWEs, where only raw audio is available \cite{holzenberger2018learning,kamper2019truly,van2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="1811.00403" id="3252">
        <attvalues>
          <attvalue for="5" value=" Network architectures vary, but the basic idea is to train the system's representations to make the positive examples closer together in the space \cite{kamper2019truly,van2021comparison} (and in some models, also to separate additional negative example pairs \cite{jacobs2021acoustic,robin2022speech})." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="2012.07387" id="3253">
        <attvalues>
          <attvalue for="5" value=" Though effective, this learning-based approach relies on running UTD on the target language, which itself is computationally intensive and sensitive to differences in input features \cite{van2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="1606.06950" id="3254">
        <attvalues>
          <attvalue for="5" value=" A common baseline that preserves sequential order while extracting a fixed-dimensional representation is subsampling (see, eg, \cite{van2021comparison,kamper2017segmental,kamper2017embedded}): selecting a fixed number of (usually equally spaced) frames and concatenating them." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="1703.08135" id="3255">
        <attvalues>
          <attvalue for="5" value=" A common baseline that preserves sequential order while extracting a fixed-dimensional representation is subsampling (see, eg, \cite{van2021comparison,kamper2017segmental,kamper2017embedded}): selecting a fixed number of (usually equally spaced) frames and concatenating them." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="2103.10731" id="3256">
        <attvalues>
          <attvalue for="5" value=" Network architectures vary, but the basic idea is to train the system's representations to make the positive examples closer together in the space \cite{kamper2019truly,van2021comparison} (and in some models, also to separate additional negative example pairs \cite{jacobs2021acoustic,robin2022speech})." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="2106.07447" id="3258">
        <attvalues>
          <attvalue for="5" value=" Here, we explore whether using newer self-supervised speech representations, available as pre-trained models \cite{hsu2021hubert,baevski2020wav2vec}, may obviate both the UTD step and the need for specialized models to learn unsupervised AWEs." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="2006.11477" id="3259">
        <attvalues>
          <attvalue for="5" value=" We hypothesize that the contextualized speech representations learned by these models will implicitly encode the sequential information needed for AWEs (eg, by capturing within each frame the local acoustic effects of coarticulation, and/or information at a longer timescale that is needed to reconstruct the masked input during pretraining---where average mask span is nearly 300ms \cite{baevski2020wav2vec})." />
        </attvalues>
      </edge>
      <edge source="2210.16043" target="2012.02221" id="3260">
        <attvalues>
          <attvalue for="5" value=" In accordance with our hypothesis, we find that on our English test set, AWEs created by mean-pooling the HB\ representations perform almost as well as the state-of-the-art learned pooling model (MCVAE \cite{peng_correspondence_2020}) with equivalent dimensionality; and outperform subsampling, despite having a much lower dimensionality." />
        </attvalues>
      </edge>
      <edge source="2306.01393" target="1804.10959" id="3261">
        <attvalues>
          <attvalue for="5" value="~\shortcite{sennrich-etal-2016-neural} to use Byte-Pair Encoding (BPE)~\cite{Gage1994ANA} to create subword vocabularies, followed by the use of a unigram language model and the SentencePiece implementation \cite{kudo-2018-subword}, no alternative models have taken over." />
        </attvalues>
      </edge>
      <edge source="2306.01393" target="1508.07909" id="3262">
        <attvalues>
          <attvalue for="5" value=" While subwords have been empirically demonstrated to outperform character and word-level tokenization~\cite{sennrich-etal-2016-neural,Wu2016GooglesNM,denkowski-neubig-2017-stronger}, the factors contributing to their success have not been fully understood yet." />
        </attvalues>
      </edge>
      <edge source="2306.01393" target="1609.08144" id="3263">
        <attvalues>
          <attvalue for="5" value=" While subwords have been empirically demonstrated to outperform character and word-level tokenization~\cite{sennrich-etal-2016-neural,Wu2016GooglesNM,denkowski-neubig-2017-stronger}, the factors contributing to their success have not been fully understood yet." />
        </attvalues>
      </edge>
      <edge source="2306.01393" target="1706.09733" id="3264">
        <attvalues>
          <attvalue for="5" value=" While subwords have been empirically demonstrated to outperform character and word-level tokenization~\cite{sennrich-etal-2016-neural,Wu2016GooglesNM,denkowski-neubig-2017-stronger}, the factors contributing to their success have not been fully understood yet." />
        </attvalues>
      </edge>
      <edge source="2407.13193" target="1909.01066" id="3274">
        <attvalues>
          <attvalue for="5" value=" Recent works~\cite{19emnlp-lm-as-kb, 22arxiv-lmkb-survey, 22nips-rome, 24arxiv-lmkb-scale} demonstrate the success of LLMs can be explained by the fact that language models act as knowledge bases, which refers to implicitly storing the knowledge learned from training datasets in the parameters as internal memory and generating responses by retrieving answers from memory." />
        </attvalues>
      </edge>
      <edge source="2407.13193" target="2110.02095" id="3275">
        <attvalues>
          <attvalue for="5" value=" To store more knowledge for better generation performance, existing works generally enlarge the memory capacity by increasing the volume of parameters~\cite{22iclr-scaling-law, gpt3, 20arxiv-scaling-law, 22arxiv-scaling-law}." />
        </attvalues>
      </edge>
      <edge source="2407.13193" target="2202.03629" id="3276">
        <attvalues>
          <attvalue for="5" value=" One of the most prominent challenges is the hallucination problem~\cite{23cs-hallucination-survey, 23acl-hallucination-mt, 23acl-hallucination-dialogue}, which refers to the tendency of LLMs to generate responses that are coherent and fluent but factually incorrect." />
        </attvalues>
      </edge>
      <edge source="2407.13193" target="2202.05262" id="3277">
        <attvalues>
          <attvalue for="5" value=" To update the knowledge stored in the LLMs' internal memory~\cite{22nips-rome, 23arxiv-editing-survey, 24arxiv-editing-survey}, it is necessary to retrain/fine-tune LLMs with new data, which is a costly process." />
        </attvalues>
      </edge>
      <edge source="2407.13193" target="2305.15075" id="3278">
        <attvalues>
          <attvalue for="5" value=" Another challenge for general LLMs is lacking of domain-specific expertise~\cite{23emnlp-huatuogpt, 23nature-med-palm, 23arxiv-med-palm2, 24arxiv-saulm}." />
        </attvalues>
      </edge>
      <edge source="2407.13193" target="2005.11401" id="3279">
        <attvalues>
          <attvalue for="5" value=" To address these challenges, recent works~\cite{20neurips-rag, 22icml-retro, 20icml-realm} have proposed leveraging an external knowledge database to augment LLMs, known as retrieval-augmented generation (RAG)." />
        </attvalues>
      </edge>
      <edge source="2407.13193" target="2202.01110" id="3280">
        <attvalues>
          <attvalue for="5" value=" Although there are several survey papers for RAG~\cite{22arxiv-rag-survey, 23arxiv-survey, 24arxiv-survey-ecust, 24arxiv-survey-pku, 24arxiv-rag-benchmark}, our survey still has some key insights, \begin{enumerate} \item This paper systematically introduces each component of RAG, including details about the retriever from building to querying, and techniques of the retrieval fusions with tutorial codes." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="2005.14165" id="3284">
        <attvalues>
          <attvalue for="5" value=" A striking feature of large language models is in-context learning \cite{brown2020language, dong2022survey, garg2022can, dai2022can}." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="1706.03762" id="3285">
        <attvalues>
          <attvalue for="5" value=" ICL emerges in transformer models \cite{vaswani2017attention} trained on a diverse set of tasks that contain a common structural element." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="2208.01066" id="3286">
        <attvalues>
          <attvalue for="5" value=" For example, a transformer trained to solve numerous linear regression tasks learns to solve a new linear regression task based on in-context examples \cite{garg2022can, akyurek2022learning,von2023transformers, ahn2023transformers}." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="2212.07677" id="3287">
        <attvalues>
          <attvalue for="5" value=" For example, a transformer trained to solve numerous linear regression tasks learns to solve a new linear regression task based on in-context examples \cite{garg2022can, akyurek2022learning,von2023transformers, ahn2023transformers}." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="2306.04891" id="3288">
        <attvalues>
          <attvalue for="5" value=" Specifically, given a sequence of sample input-output pairs, the predictive error on a target query is comparable to an optimal Bayes predictor \cite{ahuja2023context, xie2021explanation, li2023transformers}." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="2306.04637" id="3289">
        <attvalues>
          <attvalue for="5" value=" This remarkable feature extends to other generative models such as hierarchical regression models that involve model selection \cite{bai2023transformers}, random permutations of images \cite{kirsch2022general} and mixture models over sequential data \cite{wang2023large, xie2021explanation}." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="2209.11895" id="3291">
        <attvalues>
          <attvalue for="5" value=" Across networks of different scales and task structures, the ability to perform ICL often increases abruptly during training \cite{olsson2022context}." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="2309.01809" id="3292">
        <attvalues>
          <attvalue for="5" value=" Other work provides empirical evidence that ICL is the key driver behind the emergent abilities of large language models \cite{lu2023emergent}." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="2205.05055" id="3293">
        <attvalues>
          <attvalue for="5" value=" We first show that the data dependencies highlighted in \cite{chan2022data} are recapitulated in a task with simplified input statistics and a two-layer attention-only network architecture." />
        </attvalues>
      </edge>
      <edge source="2312.03002" target="1902.03477" id="3294">
        <attvalues>
          <attvalue for="5" value=" The item classes are drawn from Omniglot \cite{lake2019omniglot}, a standard image-label dataset." />
        </attvalues>
      </edge>
      <edge source="2406.01171" target="2201.11903" id="3295">
        <attvalues>
          <attvalue for="5" value=" The striking capabilities of large language models (LLMs), exemplified by ChatGPT~\cite{chatgpt}, have significantly advanced the field of natural language processing (NLP;~\cite{wei2023chainofthought,madaan2024self,shinn2024reflexion})." />
        </attvalues>
      </edge>
      <edge source="2406.01171" target="2303.17651" id="3296">
        <attvalues>
          <attvalue for="5" value=" The striking capabilities of large language models (LLMs), exemplified by ChatGPT~\cite{chatgpt}, have significantly advanced the field of natural language processing (NLP;~\cite{wei2023chainofthought,madaan2024self,shinn2024reflexion})." />
        </attvalues>
      </edge>
      <edge source="2406.01171" target="2303.11366" id="3297">
        <attvalues>
          <attvalue for="5" value=" The striking capabilities of large language models (LLMs), exemplified by ChatGPT~\cite{chatgpt}, have significantly advanced the field of natural language processing (NLP;~\cite{wei2023chainofthought,madaan2024self,shinn2024reflexion})." />
        </attvalues>
      </edge>
      <edge source="2406.01171" target="2307.16376" id="3298">
        <attvalues>
          <attvalue for="5" value=" To this end, leveraging personas has resurfaced as an ideal lens for adapting LLMs in target scenarios~\cite{chen2023large, chen2024persona}." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="1412.3555" id="3299">
        <attvalues>
          <attvalue for="5" value=" It has moved the field from feature engineering \cite{och-etal-2004-smorgasbord, zhang-nivre-2011-transition} and architecture engineering \cite{Chung2014EmpiricalEO, kim-2014-convolutional, Bahdanau2015NeuralMT, vaswani17attention} to the pre-train and fine-tune paradigm \cite{Radford2018ImprovingLU, dong_NEURIPS2019_c20bb2d9, lewis_retrieval-augmented_2021}, and lately the pre-train, prompt, and predict paradigm \cite{Liu2021PretrainPA}." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="2107.13586" id="3300">
        <attvalues>
          <attvalue for="5" value=" It has moved the field from feature engineering \cite{och-etal-2004-smorgasbord, zhang-nivre-2011-transition} and architecture engineering \cite{Chung2014EmpiricalEO, kim-2014-convolutional, Bahdanau2015NeuralMT, vaswani17attention} to the pre-train and fine-tune paradigm \cite{Radford2018ImprovingLU, dong_NEURIPS2019_c20bb2d9, lewis_retrieval-augmented_2021}, and lately the pre-train, prompt, and predict paradigm \cite{Liu2021PretrainPA}." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="1909.01066" id="3301">
        <attvalues>
          <attvalue for="5" value=" Many of these works include updating factoids stored within the parameters of LMs \cite{de_cao_editing_2021, mitchell_fast_2021, hase_language_2021} to creating new methods for extracting factual knowledge \cite{petroni_language_2019}." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="2104.05837" id="3302">
        <attvalues>
          <attvalue for="5" value=" \cite{safavi2021relational} divide relevant work according to the level of supervision provided to the LM by a KB." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="2101.00297" id="3303">
        <attvalues>
          <attvalue for="5" value=" This includes: world knowledge \cite{petroni_language_2019, rogers_primer_2020}, relational knowledge \cite{safavi2021relational}, commonsense knowledge \cite{Da2021AnalyzingCE}, linguistic knowledge \cite{Peters2018DissectingCW, Goldberg2019AssessingBS, tenney2018what}, actionable knowledge \cite{Huang2022LanguageMA} and more." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="2201.07207" id="3305">
        <attvalues>
          <attvalue for="5" value=" This includes: world knowledge \cite{petroni_language_2019, rogers_primer_2020}, relational knowledge \cite{safavi2021relational}, commonsense knowledge \cite{Da2021AnalyzingCE}, linguistic knowledge \cite{Peters2018DissectingCW, Goldberg2019AssessingBS, tenney2018what}, actionable knowledge \cite{Huang2022LanguageMA} and more." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="2104.08164" id="3306">
        <attvalues>
          <attvalue for="5" value=" Many of these works include updating factoids stored within the parameters of LMs \cite{de_cao_editing_2021, mitchell_fast_2021, hase_language_2021} to creating new methods for extracting factual knowledge \cite{petroni_language_2019}." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="2110.08455" id="3307">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{wei2021knowledge} evaluate knowledge-enhanced pretrained LMs by delineating the types of knowledge that can be integrated into existing LMs." />
        </attvalues>
      </edge>
      <edge source="2204.06031" target="2101.12294" id="3308">
        <attvalues>
          <attvalue for="5" value=" Similarly, \cite{colonhernandez2021combining} cover the integration of structural knowledge into LMs but forgo implicit knowledge." />
        </attvalues>
      </edge>
      <edge source="2109.07067" target="1606.04870" id="3309">
        <attvalues>
          <attvalue for="5" value=" Smart reply~\cite{kannan2016smart} and Smart compose~\cite{chen2019gmail} are two recent works that provide contextual assistance to aid users in completing everyday text such as emails, search engine inputs, etc While recent advances in deep neural models have shown impressive performance on the text auto-completion task, these models generally require a large amount of everyday text and huge amount of computing power for training to generate adequate suggestions~\cite{chen2019gmail}." />
        </attvalues>
      </edge>
      <edge source="2109.07067" target="1906.00080" id="3310">
        <attvalues>
          <attvalue for="5" value=" Smart reply~\cite{kannan2016smart} and Smart compose~\cite{chen2019gmail} are two recent works that provide contextual assistance to aid users in completing everyday text such as emails, search engine inputs, etc While recent advances in deep neural models have shown impressive performance on the text auto-completion task, these models generally require a large amount of everyday text and huge amount of computing power for training to generate adequate suggestions~\cite{chen2019gmail}." />
        </attvalues>
      </edge>
      <edge source="2109.07067" target="1910.13461" id="3311">
        <attvalues>
          <attvalue for="5" value=" Recently, text-to-text transformers such as BART~\cite{lewis-etal-2020-bart} and T5~\cite{JMLR:v21:20-074} have demonstrated great potential in natural language generation (NLG) tasks by using masked-span infilling as a pre-training objective." />
        </attvalues>
      </edge>
      <edge source="2312.15156" target="2303.13001" id="3313">
        <attvalues>
          <attvalue for="5" value=" Keyphrase extraction aims to extract a set of important phrases from unstructured text into structured data formats, which is a fundamental and crucial task in natural language processing \cite{2014survey,song_survey,song2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2312.15156" target="2009.07481" id="3314">
        <attvalues>
          <attvalue for="5" value=" Typically, keyphrase is beneficial for various applications due to its concise and precise expression \cite{li2023generate,pmlr,Tian_2023,hypersiamesenet,li2023unsupervised,HISUM,salewski2023zeroshot}." />
        </attvalues>
      </edge>
      <edge source="2312.15156" target="2311.08396" id="3315">
        <attvalues>
          <attvalue for="5" value=" Typically, keyphrase is beneficial for various applications due to its concise and precise expression \cite{li2023generate,pmlr,Tian_2023,hypersiamesenet,li2023unsupervised,HISUM,salewski2023zeroshot}." />
        </attvalues>
      </edge>
      <edge source="2312.15156" target="2109.07293" id="3316">
        <attvalues>
          <attvalue for="5" value=" Hence, many organizations and companies rely on keyphrase extraction to automate manual work with zero / few-shot settings \cite{2021unsupervised, hguke, promptrank, hyperrank, agrank}." />
        </attvalues>
      </edge>
      <edge source="2312.15156" target="2005.14165" id="3318">
        <attvalues>
          <attvalue for="5" value=" Recent works \cite{agrawal2022,Wei2023ZeroShotIE} on large-scale pre-trained language models, such as GPT-3 \cite{llm_few}, InstructGPT \cite{ouyang2022training} and ChatGPT 2, suggest that large language models perform well in various natural language processing downstream tasks even without tuning the parameters." />
        </attvalues>
      </edge>
      <edge source="2312.15156" target="2305.03319" id="3320">
        <attvalues>
          <attvalue for="5" value=" Generally, long documents often necessitate complex processing strategies \cite{longke,ld1,ld2,ld3}." />
        </attvalues>
      </edge>
      <edge source="2312.15156" target="2305.16784" id="3322">
        <attvalues>
          <attvalue for="5" value=" Generally, long documents often necessitate complex processing strategies \cite{longke,ld1,ld2,ld3}." />
        </attvalues>
      </edge>
      <edge source="2405.07767" target="2104.09399" id="3323">
        <attvalues>
          <attvalue for="5" value=" Hence, lots of existing test collections used in IR are based on manually created queries \cite{craswell2021trec,yilmaz2008simple}." />
        </attvalues>
      </edge>
      <edge source="2405.07767" target="2310.07849" id="3325">
        <attvalues>
          <attvalue for="5" value=" Synthetic datasets generated using LLMs have recently gained attention across a range of diverse tasks \cite{li2023synthetic,zhang2018synthetic,bao2023synthetic}." />
        </attvalues>
      </edge>
      <edge source="2405.07767" target="1806.01013" id="3326">
        <attvalues>
          <attvalue for="5" value=" Synthetic datasets generated using LLMs have recently gained attention across a range of diverse tasks \cite{li2023synthetic,zhang2018synthetic,bao2023synthetic}." />
        </attvalues>
      </edge>
      <edge source="2405.07767" target="2309.10621" id="3327">
        <attvalues>
          <attvalue for="5" value=" LLMs have also been used to generate relevance labels \cite{thomas2023large,faggioli2023perspectives}, as well as to generate query variants for evaluation and training of IR systems \cite{rajapakse2023improving,alaofi2023can}." />
        </attvalues>
      </edge>
      <edge source="2405.07767" target="2304.09161" id="3328">
        <attvalues>
          <attvalue for="5" value=" LLMs have also been used to generate relevance labels \cite{thomas2023large,faggioli2023perspectives}, as well as to generate query variants for evaluation and training of IR systems \cite{rajapakse2023improving,alaofi2023can}." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="2303.08774" id="3329">
        <attvalues>
          <attvalue for="5" value=" Our evaluation comparing XLM-Roberta \cite{conneau-etal-2020-unsupervised}, GPT-3 \cite{brown2020language}, and GPT-4 models \cite{openai2023gpt} shows that disagreement scores provide estimation of model performance with mean average error (MAE) as low as 0." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="2204.02311" id="3330">
        <attvalues>
          <attvalue for="5" value=" We have recently seen significant progress on many natural language processing (NLP) tasks using the latest generative pretrained models such as GPT \cite{openai2023gpt, ouyang2022training}, PaLM \cite{chowdhery2022palm}, and many others \cite{touvron2023llama, bai2022constitutional, penedo2023refinedweb, alpaca}." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="1907.11692" id="3333">
        <attvalues>
          <attvalue for="5" value=" More established language models (BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta}, XLM-Roberta \cite{conneau2020unsupervised}, etc) provide a strong balance of inference cost and task performance for such systems." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="1911.02116" id="3334">
        <attvalues>
          <attvalue for="5" value=" Our evaluation comparing XLM-Roberta \cite{conneau-etal-2020-unsupervised}, GPT-3 \cite{brown2020language}, and GPT-4 models \cite{openai2023gpt} shows that disagreement scores provide estimation of model performance with mean average error (MAE) as low as 0." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="2006.09462" id="3335">
        <attvalues>
          <attvalue for="5" value=" \cite{kamath-etal-2020-selective} explored evaluating fine-tuned question answering models on out of domain data, relevant to question answering problems." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="2305.14802" id="3336">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{fu2023estimating} creates a meta-model responsible for predicting the accuracy of the LLM model using the model’s confidence scores as features." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="2201.04234" id="3337">
        <attvalues>
          <attvalue for="5" value=" Methods from the computer vision (CV) domain to assess unlabeled data more generally have, for example, proposed the average threshold confidence method that learns a threshold over the model’s confidence, predicting accuracy as the fraction of unlabeled examples exceeding that threshold \cite{DBLP:conf/iclr/GargBLNS22}, or iteratively learn an ensemble of models to identify misclassified data points and perform self-training to improve the ensemble with the identified points \cite{chen2021detecting}." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="2106.13799" id="3339">
        <attvalues>
          <attvalue for="5" value=" One way to develop a well calibrated ensemble is to train the same model on the same dataset but changing initial random seed among the ensemble members, as proposed in \cite{JiangNBK22} for the CV domain." />
        </attvalues>
      </edge>
      <edge source="2309.05619" target="2005.14165" id="3340">
        <attvalues>
          <attvalue for="5" value=" Our evaluation comparing XLM-Roberta \cite{conneau-etal-2020-unsupervised}, GPT-3 \cite{brown2020language}, and GPT-4 models \cite{openai2023gpt} shows that disagreement scores provide estimation of model performance with mean average error (MAE) as low as 0." />
        </attvalues>
      </edge>
      <edge source="2105.11174" target="2010.04389" id="3343">
        <attvalues>
          <attvalue for="5" value=" In recent years, many new tasks and datasets are proposed to assess NLP model's ability of commonsense reasoning \cite{yu2020survey}." />
        </attvalues>
      </edge>
      <edge source="2105.11174" target="1808.05326" id="3344">
        <attvalues>
          <attvalue for="5" value=" SWAG~\cite{zellers-etal-2018-swag} is a task of inferring the upcoming event based on a partial description using commonsense." />
        </attvalues>
      </edge>
      <edge source="2105.11174" target="1811.00937" id="3345">
        <attvalues>
          <attvalue for="5" value=" CommonsenseQA~\cite{talmor-etal-2019-commonsenseqa} is a commonsense question answering dataset built from ConceptNet." />
        </attvalues>
      </edge>
      <edge source="2105.11174" target="1911.03705" id="3346">
        <attvalues>
          <attvalue for="5" value=" We conduct experiments on CommonGen~\cite{lin-etal-2020-commongen} benchmark dataset." />
        </attvalues>
      </edge>
      <edge source="2105.11174" target="2012.00366" id="3347">
        <attvalues>
          <attvalue for="5" value=" \cite{fan-etal-2020-enhanced} propose a retrieve-and-generation method for commonsense generation which uses a prototype candidate sentence as auxiliary input." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2005.14165" id="3348">
        <attvalues>
          <attvalue for="5" value=" These demonstrations could be hand-crafted by human~\cite{hendrycks2021measuring,wei2022chain,kazemi2023lambada}, randomly chosen from training data~\cite{brown2020language,lewkowycz2022solving}." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2110.15943" id="3349">
        <attvalues>
          <attvalue for="5" value=" ICL is popularized by the work on pre-trained large language models, which can perform ICL without being trained to do so~\cite{brown2020language}, though smaller language models can also be explicitly trained to perform ICL~\cite{min2022metaicl}." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2301.11293" id="3350">
        <attvalues>
          <attvalue for="5" value=" Furthermore, ICL avoids common issues associated with fine-tuning, such as overfitting\cite{ying2019overview,kazemi2023understanding}." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2106.09685" id="3351">
        <attvalues>
          <attvalue for="5" value=" Compared to parameter-efficient fine-tuning methods (PEFT)~\cite{hu2021lora,dettmers2023qlora,lester2021power}, ICL is computationally cheaper and remain the model parameters unchanged thus preserving the generality of the LLMs." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2305.14314" id="3352">
        <attvalues>
          <attvalue for="5" value=" Compared to parameter-efficient fine-tuning methods (PEFT)~\cite{hu2021lora,dettmers2023qlora,lester2021power}, ICL is computationally cheaper and remain the model parameters unchanged thus preserving the generality of the LLMs." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2104.08691" id="3353">
        <attvalues>
          <attvalue for="5" value=" Compared to parameter-efficient fine-tuning methods (PEFT)~\cite{hu2021lora,dettmers2023qlora,lester2021power}, ICL is computationally cheaper and remain the model parameters unchanged thus preserving the generality of the LLMs." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2103.03874" id="3354">
        <attvalues>
          <attvalue for="5" value=" These demonstrations could be hand-crafted by human~\cite{hendrycks2021measuring,wei2022chain,kazemi2023lambada}, randomly chosen from training data~\cite{brown2020language,lewkowycz2022solving}." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2212.13894" id="3355">
        <attvalues>
          <attvalue for="5" value=" These demonstrations could be hand-crafted by human~\cite{hendrycks2021measuring,wei2022chain,kazemi2023lambada}, randomly chosen from training data~\cite{brown2020language,lewkowycz2022solving}." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2210.00720" id="3357">
        <attvalues>
          <attvalue for="5" value=" Beyond random selection, there are more advanced selection processes based on metrics such as complexity~\cite{fu2022complexity}, diversity~\cite{li2023finding}, difficulty~\cite{drozdov2023parade}, concept learning~\cite{wang2023large} and perplexity~\cite{gonen2023demystifying}." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2310.14408" id="3358">
        <attvalues>
          <attvalue for="5" value=" Beyond random selection, there are more advanced selection processes based on metrics such as complexity~\cite{fu2022complexity}, diversity~\cite{li2023finding}, difficulty~\cite{drozdov2023parade}, concept learning~\cite{wang2023large} and perplexity~\cite{gonen2023demystifying}." />
        </attvalues>
      </edge>
      <edge source="2401.11624" target="2212.04037" id="3359">
        <attvalues>
          <attvalue for="5" value=" Beyond random selection, there are more advanced selection processes based on metrics such as complexity~\cite{fu2022complexity}, diversity~\cite{li2023finding}, difficulty~\cite{drozdov2023parade}, concept learning~\cite{wang2023large} and perplexity~\cite{gonen2023demystifying}." />
        </attvalues>
      </edge>
      <edge source="2404.13760" target="1901.07291" id="3360">
        <attvalues>
          <attvalue for="5" value=" In the multi-lingual space,~\cite{xlmr} exploited language embeddings for multi-lingual model training." />
        </attvalues>
      </edge>
      <edge source="2404.13760" target="1602.01595" id="3361">
        <attvalues>
          <attvalue for="5" value=" \cite{ammar-etal-2016-many} first proposed to use language embeddings for training a multi-lingual syntactic parser for seven European languages, and showed improved performance." />
        </attvalues>
      </edge>
      <edge source="2404.13760" target="1805.05089" id="3362">
        <attvalues>
          <attvalue for="5" value=" Later work also successfully trained parsers with the so called treebank embeddings for datasets within the same language~\cite{stymne-etal-2018-parser} or language family~\cite{smith-etal-2018-82}." />
        </attvalues>
      </edge>
      <edge source="2404.13760" target="1809.02237" id="3363">
        <attvalues>
          <attvalue for="5" value=" Later work also successfully trained parsers with the so called treebank embeddings for datasets within the same language~\cite{stymne-etal-2018-parser} or language family~\cite{smith-etal-2018-82}." />
        </attvalues>
      </edge>
      <edge source="2404.13760" target="2001.08210" id="3364">
        <attvalues>
          <attvalue for="5" value=" Other work have used special language ids to mark the language of each instance in the context of machine translation~\cite{liu-etal-2020-multilingual-denoising}." />
        </attvalues>
      </edge>
      <edge source="2404.13760" target="2210.09345" id="3365">
        <attvalues>
          <attvalue for="5" value="0, an extension of the CrossRE dataset~\cite{bassignana-plank-2022-crossre} with 3." />
        </attvalues>
      </edge>
      <edge source="2109.11058" target="1611.01368" id="3366">
        <attvalues>
          <attvalue for="5" value=" A rich collection of targeted linguistic evaluations has shown that neural language models can surprisingly learn many aspects of grammar from unlabeled linguistic input \cite[eg,][]{linzen-etal-2016-assessing,gulordava-etal-2018-colorless,warstadt2020blimp,hu-etal-2020-systematic,xiang2021climp}." />
        </attvalues>
      </edge>
      <edge source="2109.11058" target="1803.11138" id="3367">
        <attvalues>
          <attvalue for="5" value=" A rich collection of targeted linguistic evaluations has shown that neural language models can surprisingly learn many aspects of grammar from unlabeled linguistic input \cite[eg,][]{linzen-etal-2016-assessing,gulordava-etal-2018-colorless,warstadt2020blimp,hu-etal-2020-systematic,xiang2021climp}." />
        </attvalues>
      </edge>
      <edge source="2109.11058" target="1912.00582" id="3368">
        <attvalues>
          <attvalue for="5" value=" A rich collection of targeted linguistic evaluations has shown that neural language models can surprisingly learn many aspects of grammar from unlabeled linguistic input \cite[eg,][]{linzen-etal-2016-assessing,gulordava-etal-2018-colorless,warstadt2020blimp,hu-etal-2020-systematic,xiang2021climp}." />
        </attvalues>
      </edge>
      <edge source="2109.11058" target="1903.00943" id="3371">
        <attvalues>
          <attvalue for="5" value=" There is also growing evidence that explicit modeling of syntax helps neural network-based language models represent syntactic state and exhibit human-like processing behaviors of non-local grammatical dependencies, including number agreement \cite{kuncoro-etal-2018-lstms}, negative polarity licensing, filler-gap dependencies \cite{wilcox-etal-2019-structural, hu-etal-2020-systematic}, and garden-path effects \cite{futrell-etal-2019-neural,hu-etal-2020-systematic}." />
        </attvalues>
      </edge>
      <edge source="2109.11058" target="1903.03260" id="3372">
        <attvalues>
          <attvalue for="5" value=" There is also growing evidence that explicit modeling of syntax helps neural network-based language models represent syntactic state and exhibit human-like processing behaviors of non-local grammatical dependencies, including number agreement \cite{kuncoro-etal-2018-lstms}, negative polarity licensing, filler-gap dependencies \cite{wilcox-etal-2019-structural, hu-etal-2020-systematic}, and garden-path effects \cite{futrell-etal-2019-neural,hu-etal-2020-systematic}." />
        </attvalues>
      </edge>
      <edge source="2109.11058" target="1602.07776" id="3373">
        <attvalues>
          <attvalue for="5" value=" While most prior work investigating syntactically guided language models has used Recurrent Neural Network Grammar models \cite{dyer2016recurrent} -- potentially conflating structural supervision with a particular parameterization -- this work further explores structured Transformer language models \cite{qian-etal-2021-structural}." />
        </attvalues>
      </edge>
      <edge source="2109.11058" target="2108.00104" id="3374">
        <attvalues>
          <attvalue for="5" value=" While most prior work investigating syntactically guided language models has used Recurrent Neural Network Grammar models \cite{dyer2016recurrent} -- potentially conflating structural supervision with a particular parameterization -- this work further explores structured Transformer language models \cite{qian-etal-2021-structural}." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="2005.14165" id="3375">
        <attvalues>
          <attvalue for="5" value=" For ICL, it is also empirically observed that the performance continually improves when more training examples are prepended into the prompt~\cite{NEURIPS2020_1457c0d6}." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="2205.01068" id="3376">
        <attvalues>
          <attvalue for="5" value=" However, such models are prohibitively expensive to train with most of the research- or consumer-level devices, though some of them are already publicly available~\cite{zhang2022opt}." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="1712.00409" id="3377">
        <attvalues>
          <attvalue for="5" value=" Specifically, there exists a power law between expected model performance and available data scale~\cite{hestness2017deep,rosenfeld2020a}." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="1909.12673" id="3378">
        <attvalues>
          <attvalue for="5" value=" Specifically, there exists a power law between expected model performance and available data scale~\cite{hestness2017deep,rosenfeld2020a}." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="2112.08633" id="3379">
        <attvalues>
          <attvalue for="5" value=" In order to utilize more training data, several works try to select the most relevant examples to compose the prompt before querying LLM~\cite{liu-etal-2022-makes,rubin-etal-2022-learning}, but still only in-context examples can actually participate the LLM inference while most training data are discarded beforehand, thus providing marginal data scaling benefits." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="2104.08315" id="3380">
        <attvalues>
          <attvalue for="5" value=" Another vulnerability of ICL is the severe bias existed in the output distribution of LLMs, which results in considerable performance degradation~\cite{holtzman-etal-2021-surface} and instability~\cite{lu-etal-2022-fantastically} as shown in existing works." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="2104.08786" id="3381">
        <attvalues>
          <attvalue for="5" value=" Another vulnerability of ICL is the severe bias existed in the output distribution of LLMs, which results in considerable performance degradation~\cite{holtzman-etal-2021-surface} and instability~\cite{lu-etal-2022-fantastically} as shown in existing works." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="2102.09690" id="3382">
        <attvalues>
          <attvalue for="5" value=" For example,~\cite{pmlr-v139-zhao21c} measure such bias by probing LLM with a &quot;NA&quot; example and record the according prior." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="2012.00955" id="3383">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many have proposed various ways to calibrate the output distribution~\cite{pmlr-v139-zhao21c,jiang-etal-2021-know,min-etal-2022-noisy}." />
        </attvalues>
      </edge>
      <edge source="2303.13824" target="2108.04106" id="3384">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many have proposed various ways to calibrate the output distribution~\cite{pmlr-v139-zhao21c,jiang-etal-2021-know,min-etal-2022-noisy}." />
        </attvalues>
      </edge>
      <edge source="2305.14256" target="1908.06625" id="3385">
        <attvalues>
          <attvalue for="5" value=" The approximately linear mapping between cross-lingual word embeddings in different languages is based on assumption that the word semantic meaning is conserved in a translation \cite{ The linearity is only approximate because the corresponding words in different languages have different cultural background, different multiple meanings and different dependencies on context \cite{patra-etal-2019-bilingual, zhao-gilman-2020-non, Cao2020MultilingualAlignment, Peng2022Understanding}." />
        </attvalues>
      </edge>
      <edge source="2305.14256" target="2304.13803" id="3386">
        <attvalues>
          <attvalue for="5" value=" Indeed, in \cite{kang-etal-2024-translate} it is demonstrated that additional context helps to reduce disambiguation errors." />
        </attvalues>
      </edge>
      <edge source="2305.14256" target="2004.09813" id="3387">
        <attvalues>
          <attvalue for="5" value=" The idea that a sentence semantics should be better conserved in a translation was used in \cite{reimers-gurevych-2020-making}." />
        </attvalues>
      </edge>
      <edge source="2305.14256" target="2109.04727" id="3388">
        <attvalues>
          <attvalue for="5" value=" Unlike the removal of a language-specific bias in each language separately \cite{yang-etal-2021-simple, xie-etal-2022-discovering}, this mapping depends on both languages of interest and, while computationally cheap, may provide a better correspondence between the embeddings." />
        </attvalues>
      </edge>
      <edge source="2304.13803" target="2010.06478" id="3389">
        <attvalues>
          <attvalue for="5" value=" These models also demonstrate cross-lingual knowledge when finetuned for the word sense disambiguation (WSD) \cite{raganato-etal-2020-xl, pasini2021xl}." />
        </attvalues>
      </edge>
      <edge source="2304.13803" target="2210.03057" id="3390">
        <attvalues>
          <attvalue for="5" value=" However, little is known about the extent to which word sense knowledge comes from pretraining rather than finetuning: many PLMs struggle to disambiguate word sense when formulated as a binary classification task, the most common word sense setup for prompting language models \cite{shi2022language, scao2022bloom}." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2005.14165" id="3391">
        <attvalues>
          <attvalue for="5" value=" The recent success of Large Language Models (LLMs), such as GPT-3\cite{brown2020language}, LLaMA\cite{touvron2023llama} and PaLM\cite{chowdhery2023palm}, has garnered significant attention in both academia and industry." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2204.02311" id="3393">
        <attvalues>
          <attvalue for="5" value=" The recent success of Large Language Models (LLMs), such as GPT-3\cite{brown2020language}, LLaMA\cite{touvron2023llama} and PaLM\cite{chowdhery2023palm}, has garnered significant attention in both academia and industry." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2303.00807" id="3395">
        <attvalues>
          <attvalue for="5" value=" Substantial efforts have been made to enhance and utilizing such generalization capabilities\cite{xu-etal-2023-fine,saad-falcon-etal-2023-udapdr,yun-etal-2023-appraising}." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2305.11828" id="3396">
        <attvalues>
          <attvalue for="5" value=" Substantial efforts have been made to enhance and utilizing such generalization capabilities\cite{xu-etal-2023-fine,saad-falcon-etal-2023-udapdr,yun-etal-2023-appraising}." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2212.02216" id="3397">
        <attvalues>
          <attvalue for="5" value=" However, for natural language understanding (NLU) tasks, zero- and few-shot LLMs struggle to achieve satisfactory performance\cite{nie2022improving,wei2023zero,li2023evaluating,li2023label} compared to fine-tuned small models (eg, Bert base\cite{devlin2018bert})." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2304.11633" id="3398">
        <attvalues>
          <attvalue for="5" value=" However, for natural language understanding (NLU) tasks, zero- and few-shot LLMs struggle to achieve satisfactory performance\cite{nie2022improving,wei2023zero,li2023evaluating,li2023label} compared to fine-tuned small models (eg, Bert base\cite{devlin2018bert})." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2310.01208" id="3399">
        <attvalues>
          <attvalue for="5" value=" However, for natural language understanding (NLU) tasks, zero- and few-shot LLMs struggle to achieve satisfactory performance\cite{nie2022improving,wei2023zero,li2023evaluating,li2023label} compared to fine-tuned small models (eg, Bert base\cite{devlin2018bert})." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2109.04332" id="3401">
        <attvalues>
          <attvalue for="5" value=" \item Prompt Formats: Prompt design is crucial for LLMs in zero- and few-shot settings\cite{gu2021ppt,liu2023pre}." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2107.13586" id="3402">
        <attvalues>
          <attvalue for="5" value=" \item Prompt Formats: Prompt design is crucial for LLMs in zero- and few-shot settings\cite{gu2021ppt,liu2023pre}." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2201.11903" id="3403">
        <attvalues>
          <attvalue for="5" value=" \item Output Formats: Incorporating the Chain of Thought (CoT) into prompts has been shown to significantly enhance performance in reasoning and complex tasks in zero- and few-shot settings\cite{wei2022chain,wang2022self}." />
        </attvalues>
      </edge>
      <edge source="2403.19930" target="2203.11171" id="3404">
        <attvalues>
          <attvalue for="5" value=" \item Output Formats: Incorporating the Chain of Thought (CoT) into prompts has been shown to significantly enhance performance in reasoning and complex tasks in zero- and few-shot settings\cite{wei2022chain,wang2022self}." />
        </attvalues>
      </edge>
      <edge source="2102.12266" target="1707.06961" id="3421">
        <attvalues>
          <attvalue for="5" value=" Zero-shot approaches \cite{pinter-etal-2017-mimicking, kim2015characteraware, bojanowski2016enriching} attempt to predict the embeddings for OOV words from their characters alone." />
        </attvalues>
      </edge>
      <edge source="2102.12266" target="1903.00724" id="3423">
        <attvalues>
          <attvalue for="5" value=" Few-shot approaches \cite{garneau-etal-2018-predicting, khodak2018la,hu2019fewshot} leveraged this to predict the embeddings for OOV words from just a few contexts, often in conjunction with their morphological information." />
        </attvalues>
      </edge>
      <edge source="2102.12266" target="1907.00505" id="3424">
        <attvalues>
          <attvalue for="5" value=" \cite{hu2019fewshot} proposed an attention-based architecture for OOV word embedding learning as a few-shot regression problem." />
        </attvalues>
      </edge>
      <edge source="2102.12266" target="1703.03400" id="3425">
        <attvalues>
          <attvalue for="5" value=" As OOV words must have their embeddings inferred from contexts outside the training corpus, the authors show that using an adaptation of the model-agnostic meta-learning (MAML) algorithm \cite{finn2017modelagnostic} to adapt the model's parameters to the target domain improves the quality of the learned OOV word embeddings." />
        </attvalues>
      </edge>
      <edge source="2102.12266" target="1810.09502" id="3426">
        <attvalues>
          <attvalue for="5" value=" However, MAML is known to be unstable due to the calculation of gradients requiring backpropagation through multiple instances of the model, as the learning process must be unrolled to calculate gradients with respect to the initial parameters \cite{antoniou2018train}." />
        </attvalues>
      </edge>
      <edge source="2102.12266" target="1803.02021" id="3427">
        <attvalues>
          <attvalue for="5" value=" In practice, the learning process is often truncated to a small number of gradient steps, but has been shown to have a short-horizon bias \cite{wu2018understanding}, causing it to underperform." />
        </attvalues>
      </edge>
      <edge source="2102.12266" target="1812.01054" id="3428">
        <attvalues>
          <attvalue for="5" value=" In this work we explore OOV word embedding learning using Leap \cite{flennerhag2018transferring}, a meta-learning framework which takes into consideration the entire learning trajectory, not only the beginning and end points." />
        </attvalues>
      </edge>
      <edge source="2405.06105" target="2306.15595" id="3429">
        <attvalues>
          <attvalue for="5" value=" Recently, many researchers~\cite{chen2023extending,chen2023clex,xiong2023effective, ding2023longnet, chen2023longlora} have proposed various approaches to scale up the context window of LLMs to more than 100k." />
        </attvalues>
      </edge>
      <edge source="2405.06105" target="2310.16450" id="3430">
        <attvalues>
          <attvalue for="5" value=" Recently, many researchers~\cite{chen2023extending,chen2023clex,xiong2023effective, ding2023longnet, chen2023longlora} have proposed various approaches to scale up the context window of LLMs to more than 100k." />
        </attvalues>
      </edge>
      <edge source="2405.06105" target="2309.12307" id="3432">
        <attvalues>
          <attvalue for="5" value=" Since there is not a comprehensive benchmark tailored for the evaluation of such extremely long text understanding, such as question answering (QA) over 100K tokens, researchers use perplexity (PPL), an evaluation metric for language modeling, to demonstrate the model's ability to process long text~\cite{chen2023longlora, ding2023longnet,liu2023scaling,peng2023yarn}." />
        </attvalues>
      </edge>
      <edge source="2405.06105" target="2309.00071" id="3434">
        <attvalues>
          <attvalue for="5" value=" Since there is not a comprehensive benchmark tailored for the evaluation of such extremely long text understanding, such as question answering (QA) over 100K tokens, researchers use perplexity (PPL), an evaluation metric for language modeling, to demonstrate the model's ability to process long text~\cite{chen2023longlora, ding2023longnet,liu2023scaling,peng2023yarn}." />
        </attvalues>
      </edge>
      <edge source="2010.10938" target="1309.4168" id="3441">
        <attvalues>
          <attvalue for="5" value=" A line of works assumes that monolingual word embeddings share similar structures across different languages and try to impose post-hoc alignment through a mapping~\cite{mikolov:13, Smith:17, joulin:18, Lample:18, artetxe:18, Zhou:19}." />
        </attvalues>
      </edge>
      <edge source="2010.10938" target="1602.01925" id="3442">
        <attvalues>
          <attvalue for="5" value=" Another line of works considers joint training, which optimizes monolingual objective with or without cross-lingual constraints when training word embeddings~\cite{luong:15,gouws:15,Ammar:16,duong:16,lample:18b}." />
        </attvalues>
      </edge>
      <edge source="2010.10938" target="1606.09403" id="3443">
        <attvalues>
          <attvalue for="5" value=" Another line of works considers joint training, which optimizes monolingual objective with or without cross-lingual constraints when training word embeddings~\cite{luong:15,gouws:15,Ammar:16,duong:16,lample:18b}." />
        </attvalues>
      </edge>
      <edge source="2010.10938" target="1310.4546" id="3444">
        <attvalues>
          <attvalue for="5" value=" Cross-lingual word embedding methods above were initially proposed for non-contextualized embedding such as GloVe~\cite{pennington:14} and Word2Vec~\cite{mikolov:13w2v}, and later adapted to contextualized word representation~\cite{schuster:19, aldarmaki-diab:19}." />
        </attvalues>
      </edge>
      <edge source="2010.10938" target="1902.09492" id="3445">
        <attvalues>
          <attvalue for="5" value=" Cross-lingual word embedding methods above were initially proposed for non-contextualized embedding such as GloVe~\cite{pennington:14} and Word2Vec~\cite{mikolov:13w2v}, and later adapted to contextualized word representation~\cite{schuster:19, aldarmaki-diab:19}." />
        </attvalues>
      </edge>
      <edge source="2010.10938" target="1912.07840" id="3449">
        <attvalues>
          <attvalue for="5" value=" The conclusion about the impact of shared vocabulary is mixed~\cite{Karth:20, singh:19}, showing that our understandings about it are still in the early stages." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="1911.02116" id="3450">
        <attvalues>
          <attvalue for="5" value=" Pretrained multilingual models \cite{conneau-etal-2020-unsupervised,liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,lin-etal-2022-shot} have been established as promising sources of transfer learning, where task-specific finetuning benefits from the general knowledge learned on diverse unsupervised data." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="2001.08210" id="3451">
        <attvalues>
          <attvalue for="5" value=" Pretrained multilingual models \cite{conneau-etal-2020-unsupervised,liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,lin-etal-2022-shot} have been established as promising sources of transfer learning, where task-specific finetuning benefits from the general knowledge learned on diverse unsupervised data." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="2010.11934" id="3452">
        <attvalues>
          <attvalue for="5" value=" Pretrained multilingual models \cite{conneau-etal-2020-unsupervised,liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,lin-etal-2022-shot} have been established as promising sources of transfer learning, where task-specific finetuning benefits from the general knowledge learned on diverse unsupervised data." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="1906.01502" id="3453">
        <attvalues>
          <attvalue for="5" value=" While zero-shot crosslingual transfer has shown very promising results on sequence classification or labeling problems \cite{pires-etal-2019-multilingual,DBLP:conf/nips/ConneauL19,wu-dredze-2019-beto}, it remains challenging for generation tasks \cite{ronnqvist-etal-2019-multilingual,vu-etal-2022-overcoming,li-murray-2023-zero} including summarization and translation." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="1901.07291" id="3454">
        <attvalues>
          <attvalue for="5" value=" While zero-shot crosslingual transfer has shown very promising results on sequence classification or labeling problems \cite{pires-etal-2019-multilingual,DBLP:conf/nips/ConneauL19,wu-dredze-2019-beto}, it remains challenging for generation tasks \cite{ronnqvist-etal-2019-multilingual,vu-etal-2022-overcoming,li-murray-2023-zero} including summarization and translation." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="1904.09077" id="3455">
        <attvalues>
          <attvalue for="5" value=" While zero-shot crosslingual transfer has shown very promising results on sequence classification or labeling problems \cite{pires-etal-2019-multilingual,DBLP:conf/nips/ConneauL19,wu-dredze-2019-beto}, it remains challenging for generation tasks \cite{ronnqvist-etal-2019-multilingual,vu-etal-2022-overcoming,li-murray-2023-zero} including summarization and translation." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="2205.12647" id="3457">
        <attvalues>
          <attvalue for="5" value=" To alleviate catastrophic forgetting, one line of work trains on additional unsupervised data \cite{maurya-etal-2021-zmbart,vu-etal-2022-overcoming,DBLP:journals/corr/abs-2311-09344}." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="2305.17325" id="3458">
        <attvalues>
          <attvalue for="5" value=" While zero-shot crosslingual transfer has shown very promising results on sequence classification or labeling problems \cite{pires-etal-2019-multilingual,DBLP:conf/nips/ConneauL19,wu-dredze-2019-beto}, it remains challenging for generation tasks \cite{ronnqvist-etal-2019-multilingual,vu-etal-2022-overcoming,li-murray-2023-zero} including summarization and translation." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="2106.01597" id="3459">
        <attvalues>
          <attvalue for="5" value=" To alleviate catastrophic forgetting, one line of work trains on additional unsupervised data \cite{maurya-etal-2021-zmbart,vu-etal-2022-overcoming,DBLP:journals/corr/abs-2311-09344}." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="2311.09344" id="3460">
        <attvalues>
          <attvalue for="5" value=" To alleviate catastrophic forgetting, one line of work trains on additional unsupervised data \cite{maurya-etal-2021-zmbart,vu-etal-2022-overcoming,DBLP:journals/corr/abs-2311-09344}." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="2308.11466" id="3462">
        <attvalues>
          <attvalue for="5" value=" This has been shown to facilitate zero-shot crosslingual generation in general \cite{pham-etal-2019-improving,wu-etal-2022-laft,DBLP:journals/corr/abs-2308-11466}." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="1505.07818" id="3463">
        <attvalues>
          <attvalue for="5" value=" There a prominent approach is adversarial training \cite{DBLP:journals/jmlr/GaninUAGLLML16,chen-etal-2018-adversarial}, where the model is trained to deceive a language classifier." />
        </attvalues>
      </edge>
      <edge source="2404.05720" target="1606.01614" id="3464">
        <attvalues>
          <attvalue for="5" value=" There a prominent approach is adversarial training \cite{DBLP:journals/jmlr/GaninUAGLLML16,chen-etal-2018-adversarial}, where the model is trained to deceive a language classifier." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1409.3215" id="3466">
        <attvalues>
          <attvalue for="5" value=" Abstractive summarization is commonly viewed as a sequence-to-sequence (seq2seq) learning process \cite{Sutskever2014}." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1409.0473" id="3467">
        <attvalues>
          <attvalue for="5" value=" Seq2seq language models \cite{Bahdanau2015} with attention mechanism have dominated various natural language processing (NLP) downstream tasks\cite{rush2015, chopra2016, Vaswani2017, Radford2018, Devlin2019, Raffel2019, Lewis2020, Qi2020}." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1808.10792" id="3469">
        <attvalues>
          <attvalue for="5" value=" It is also worth noticing that language model with representation learning methods (\eg, bottom-up \cite{gehrmann2018bottomup}, contrastive learning \cite{Xu_Zhang_Wu_Wei_2022} and external faithful signals \cite{dou-etal-2021-gsum}) can effectively improve the quality of machine generated summaries." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="2109.03481" id="3470">
        <attvalues>
          <attvalue for="5" value=" It is also worth noticing that language model with representation learning methods (\eg, bottom-up \cite{gehrmann2018bottomup}, contrastive learning \cite{Xu_Zhang_Wu_Wei_2022} and external faithful signals \cite{dou-etal-2021-gsum}) can effectively improve the quality of machine generated summaries." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="2010.08014" id="3471">
        <attvalues>
          <attvalue for="5" value=" It is also worth noticing that language model with representation learning methods (\eg, bottom-up \cite{gehrmann2018bottomup}, contrastive learning \cite{Xu_Zhang_Wu_Wei_2022} and external faithful signals \cite{dou-etal-2021-gsum}) can effectively improve the quality of machine generated summaries." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="2109.10616" id="3472">
        <attvalues>
          <attvalue for="5" value=" Recent works investigated external commonsence enhanced representation \cite{yang-human-Like-2021} and topic-oriented semantic representation \cite{nguyen-etal-2021-enriching} for abstractive summarization." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1910.12840" id="3473">
        <attvalues>
          <attvalue for="5" value=" To efficiently evaluate the consistency, conciseness and faithfulness of machine generated summaries, deep neural network based evaluation is receiving increasing attention \cite{kryscinski2020factCC, LabanSBH22Summac, liu-etal-2022-brio, ladhak-etal-2022-faithful}." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="2010.07100" id="3474">
        <attvalues>
          <attvalue for="5" value=" Automatic evaluation methods based on $N$-gram, eg, recall-oriented understudy for gisting evaluation (ROUGE) metric \cite{Lin2004} and bilingual evaluation understudy (BLEU) metric \cite{Papineni2002}, are widely-used for generation tasks such as text summarization \cite{Bhandari2020} and machine translation." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1511.06732" id="3475">
        <attvalues>
          <attvalue for="5" value=" \cite{Ranzato2016} and \cite{Wu-etal2016} tried bridging the discrepancy by rewarding BLEU and grammaticality evaluation utility (GLEU) metrics for text generation with reinforcement learning \cite{Williams1989}." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1703.04887" id="3476">
        <attvalues>
          <attvalue for="5" value=" \cite{yang-etal2018} and \cite{li-piccardi-2021} used generative adversarial neural network to optimize BLEU metric on machine translation." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1609.08144" id="3477">
        <attvalues>
          <attvalue for="5" value=" \cite{Ranzato2016} and \cite{Wu-etal2016} tried bridging the discrepancy by rewarding BLEU and grammaticality evaluation utility (GLEU) metrics for text generation with reinforcement learning \cite{Williams1989}." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1705.04304" id="3478">
        <attvalues>
          <attvalue for="5" value=" \cite{paulus2018} applied reinforcement learning approach on text summarization." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1805.04871" id="3479">
        <attvalues>
          <attvalue for="5" value=" \cite{ma2018bagofwords} and \cite{Shao2020BoN} proposed bag-of-words \cite{Joachims1998} and bag-of-ngrams (BoN) as sequence-level training target respectively." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1911.09320" id="3480">
        <attvalues>
          <attvalue for="5" value=" Inspired by probabilistic $N$-gram count objective \cite{shao2018greedy} and BoN objective \cite{Shao2020BoN}, we want the learning objective to be flexible for $N$-gram matching which is not limited by the reference $N$-gram count ceiling, and expect the co-occurrences of $N$-gram have equal weights in the objective, yet retaining the capability of alleviating the exposure bias." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1712.04708" id="3481">
        <attvalues>
          <attvalue for="5" value=" \cite{Zhukov2017} introduced a lower bound approximation of expected BLEU score for sequence generation." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1611.01144" id="3482">
        <attvalues>
          <attvalue for="5" value=" \cite{Casas2018} proposed a differentiable BLEU objective with approximation of $N$-gram matches using Gumbel-softmax \cite{jang2017}." />
        </attvalues>
      </edge>
      <edge source="2202.04003" target="1809.03132" id="3483">
        <attvalues>
          <attvalue for="5" value=" Unlike \cite{shao2018greedy, Shao2020BoN}, our $N$-gram rewards objective and $N$-gram matches objective value the matched $N$-gram equally, and continue optimizing a matched $N$-gram even if it exceeds the count of this $N$-gram in reference text." />
        </attvalues>
      </edge>
      <edge source="2205.13708" target="2001.08210" id="3485">
        <attvalues>
          <attvalue for="5" value=" These include but are not limit to question answering~\cite{devlin-etal-2019-bert}, language generation~\cite{radford2018improving,radford2019language} and machine translation~\cite{liu-etal-2020-multilingual-denoising}." />
        </attvalues>
      </edge>
      <edge source="2205.13708" target="1704.07431" id="3486">
        <attvalues>
          <attvalue for="5" value=" Given the prevalent usage of idioms in different languages, identifying the correct meaning of a phrase in a certain context is crucial for many downstream tasks including sentiment analysis~\cite{WILLIAMS20157375}, automatic spelling correction~\cite{horbach-etal-2016-corpus} and machine translation~\cite{isabelle-etal-2017-challenge}." />
        </attvalues>
      </edge>
      <edge source="2205.13708" target="2010.03763" id="3487">
        <attvalues>
          <attvalue for="5" value=" Following \cite{yu-ettinger-2020-assessing}, we also consider variations of phrase representations across models, layers, and representation types." />
        </attvalues>
      </edge>
      <edge source="2205.13708" target="1902.10618" id="3488">
        <attvalues>
          <attvalue for="5" value=" \cite{shwartz-dagan-2019-still} confirm that contextualized word representations perform better than static word embeddings, more so on detecting meaning shift than in recovering implicit information." />
        </attvalues>
      </edge>
      <edge source="2205.13708" target="2109.04413" id="3489">
        <attvalues>
          <attvalue for="5" value=" Based upon AStitchInLanguageModels~\cite{tayyar-madabushi-etal-2021-astitchinlanguagemodels-dataset}, SemEval-2022 Task2~\cite{tayyarmadabushi-etal-2022-semeval} is proposed with a focus on multilingual idiomaticity." />
        </attvalues>
      </edge>
      <edge source="2205.13708" target="2204.10050" id="3490">
        <attvalues>
          <attvalue for="5" value=" Based upon AStitchInLanguageModels~\cite{tayyar-madabushi-etal-2021-astitchinlanguagemodels-dataset}, SemEval-2022 Task2~\cite{tayyarmadabushi-etal-2022-semeval} is proposed with a focus on multilingual idiomaticity." />
        </attvalues>
      </edge>
      <edge source="2205.13708" target="1911.02116" id="3491">
        <attvalues>
          <attvalue for="5" value=" We extend one of the monolingual idiomaticity probing method~\cite{tan-jiang-2021-bert} to multilingual scenario and compare multiple settings using multi-lingual BERT~(mBERT)~\cite{devlin-etal-2019-bert} and XLM-R~\cite{conneau-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="1804.11283" id="3492">
        <attvalues>
          <attvalue for="5" value=" Examples of this are synthetic datasets created in the news \cite{grusky-etal-2018-newsroom,narayan-etal-2018-dont,scialom-etal-2020-mlsum,hasan-etal-2021-xl} and instructional domains \cite{ladhak-etal-2020-wikilingua} and for descriptive summarisation \cite{liu2018generating,perez-beltrachini-etal-2019-generating,perez-beltrachini-lapata-2021-models}." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="1808.08745" id="3493">
        <attvalues>
          <attvalue for="5" value=" \cite{maynez-etal-2020-faithfulness} found that 70\% of the pairs in the XSum dataset \cite{narayan-etal-2018-dont} contain summaries with hallucinations." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2004.14900" id="3494">
        <attvalues>
          <attvalue for="5" value=" Examples of this are synthetic datasets created in the news \cite{grusky-etal-2018-newsroom,narayan-etal-2018-dont,scialom-etal-2020-mlsum,hasan-etal-2021-xl} and instructional domains \cite{ladhak-etal-2020-wikilingua} and for descriptive summarisation \cite{liu2018generating,perez-beltrachini-etal-2019-generating,perez-beltrachini-lapata-2021-models}." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="1801.10198" id="3497">
        <attvalues>
          <attvalue for="5" value=" Examples of this are synthetic datasets created in the news \cite{grusky-etal-2018-newsroom,narayan-etal-2018-dont,scialom-etal-2020-mlsum,hasan-etal-2021-xl} and instructional domains \cite{ladhak-etal-2020-wikilingua} and for descriptive summarisation \cite{liu2018generating,perez-beltrachini-etal-2019-generating,perez-beltrachini-lapata-2021-models}." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="1906.04687" id="3498">
        <attvalues>
          <attvalue for="5" value=" Examples of this are synthetic datasets created in the news \cite{grusky-etal-2018-newsroom,narayan-etal-2018-dont,scialom-etal-2020-mlsum,hasan-etal-2021-xl} and instructional domains \cite{ladhak-etal-2020-wikilingua} and for descriptive summarisation \cite{liu2018generating,perez-beltrachini-etal-2019-generating,perez-beltrachini-lapata-2021-models}." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2202.09583" id="3499">
        <attvalues>
          <attvalue for="5" value=" Our study focuses on the XWikis corpus \cite{perez-beltrachini-lapata-2021-models} consisting of descriptive summaries." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2202.06935" id="3500">
        <attvalues>
          <attvalue for="5" value=" The potential content misalignment in document-summary pairs created in this way raises concerns about the quality of training and evaluation data \cite{gehrmann2022-obstacles-nlgeval}." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2005.00661" id="3501">
        <attvalues>
          <attvalue for="5" value=" \cite{maynez-etal-2020-faithfulness} found that 70\% of the pairs in the XSum dataset \cite{narayan-etal-2018-dont} contain summaries with hallucinations." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2307.04018" id="3502">
        <attvalues>
          <attvalue for="5" value=" Previous work carried out manual validation of document-summary pairs in automatically created datasets to assess content overlap thereof \cite{maynez-etal-2020-faithfulness,hasan-etal-2021-xl,perez-beltrachini-lapata-2021-models,gao-etal-2023-evaluating,chen-etal-2023-revisiting}." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2106.09069" id="3503">
        <attvalues>
          <attvalue for="5" value=" Taking a causal look into the hallucinations problem in automatically generated summaries, previous work \cite{gem-sets-2021,nan-etal-2021-entity,liu-etal-2021-noisy,cao-wang-2021-cliff,goyal-durrett-2021-annotating,choubey2022cape,aharoni2022mface,qiu2023detecting} inspects whether reference summaries in a dataset contain hallucinations in an automatic way." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2102.09130" id="3504">
        <attvalues>
          <attvalue for="5" value=" Taking a causal look into the hallucinations problem in automatically generated summaries, previous work \cite{gem-sets-2021,nan-etal-2021-entity,liu-etal-2021-noisy,cao-wang-2021-cliff,goyal-durrett-2021-annotating,choubey2022cape,aharoni2022mface,qiu2023detecting} inspects whether reference summaries in a dataset contain hallucinations in an automatic way." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2009.07032" id="3505">
        <attvalues>
          <attvalue for="5" value=" Taking a causal look into the hallucinations problem in automatically generated summaries, previous work \cite{gem-sets-2021,nan-etal-2021-entity,liu-etal-2021-noisy,cao-wang-2021-cliff,goyal-durrett-2021-annotating,choubey2022cape,aharoni2022mface,qiu2023detecting} inspects whether reference summaries in a dataset contain hallucinations in an automatic way." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2104.04302" id="3507">
        <attvalues>
          <attvalue for="5" value=" Taking a causal look into the hallucinations problem in automatically generated summaries, previous work \cite{gem-sets-2021,nan-etal-2021-entity,liu-etal-2021-noisy,cao-wang-2021-cliff,goyal-durrett-2021-annotating,choubey2022cape,aharoni2022mface,qiu2023detecting} inspects whether reference summaries in a dataset contain hallucinations in an automatic way." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2212.10622" id="3509">
        <attvalues>
          <attvalue for="5" value=" Taking a causal look into the hallucinations problem in automatically generated summaries, previous work \cite{gem-sets-2021,nan-etal-2021-entity,liu-etal-2021-noisy,cao-wang-2021-cliff,goyal-durrett-2021-annotating,choubey2022cape,aharoni2022mface,qiu2023detecting} inspects whether reference summaries in a dataset contain hallucinations in an automatic way." />
        </attvalues>
      </edge>
      <edge source="2408.00675" target="2305.13632" id="3510">
        <attvalues>
          <attvalue for="5" value=" Taking a causal look into the hallucinations problem in automatically generated summaries, previous work \cite{gem-sets-2021,nan-etal-2021-entity,liu-etal-2021-noisy,cao-wang-2021-cliff,goyal-durrett-2021-annotating,choubey2022cape,aharoni2022mface,qiu2023detecting} inspects whether reference summaries in a dataset contain hallucinations in an automatic way." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1803.02324" id="3512">
        <attvalues>
          <attvalue for="5" value=" Recently, a series of works \cite{gururangan2018annotation, poliak2018hypothesis, kaushik2018much, tsuchiya2018performance, tan2019investigating, schwartz2017effect, nadeem2020stereoset} has shown that many of popular datasets, such as SQUAD \cite{rajpurkar2016squad} and SNLI \cite{bowman2015large} have unwanted biases \cite{torralba2011unbiased}, resulting from the annotation process." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1606.05250" id="3513">
        <attvalues>
          <attvalue for="5" value=" Recently, a series of works \cite{gururangan2018annotation, poliak2018hypothesis, kaushik2018much, tsuchiya2018performance, tan2019investigating, schwartz2017effect, nadeem2020stereoset} has shown that many of popular datasets, such as SQUAD \cite{rajpurkar2016squad} and SNLI \cite{bowman2015large} have unwanted biases \cite{torralba2011unbiased}, resulting from the annotation process." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1508.05326" id="3514">
        <attvalues>
          <attvalue for="5" value=" We apply DQI in an active learning setup to renovate the SNLI dataset \cite{bowman2015large} using the automation methods, and produce a series of benchmarks in an increasing hierarchy of hardness." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="2002.04108" id="3515">
        <attvalues>
          <attvalue for="5" value=" Our work supports the findings of an interesting recent work \cite{bras2020adversarial} where they indicate that biases make benchmarks easier, as models learn to exploit these biases instead of learning actual features." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1610.02136" id="3519">
        <attvalues>
          <attvalue for="5" value="&quot; \cite{hendrycks2016baseline}." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1904.07911" id="3521">
        <attvalues>
          <attvalue for="5" value=" AFLite \cite{sakaguchi2019winogrande}, REPAIR \cite{li2019repair}, RESOUND \cite{li2018resound} and Dataset Distillation \cite{wang2018dataset} are some of the recent works that use the first approach." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1811.10959" id="3522">
        <attvalues>
          <attvalue for="5" value=" AFLite \cite{sakaguchi2019winogrande}, REPAIR \cite{li2019repair}, RESOUND \cite{li2018resound} and Dataset Distillation \cite{wang2018dataset} are some of the recent works that use the first approach." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1908.10763" id="3524">
        <attvalues>
          <attvalue for="5" value=" Similarly, DRiFt has been proposed \cite{he2019unlearn}, where initially a biased model is learned, which uses only bias related features." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1808.05326" id="3525">
        <attvalues>
          <attvalue for="5" value=" Adversarial Filtering algorithm \cite{zellers2018swag} builds a de-biased dataset by iteratively training an ensemble of classifiers, and then utilizing them to filter data." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1904.02668" id="3527">
        <attvalues>
          <attvalue for="5" value=" This category of approaches might induce its own biases, as studied in a recent work \cite{liu2019inoculation} for NLI stress tests \cite{naik2018stress} and the Adversarial SQuAD dataset \cite{jia2017adversarial}." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1806.00692" id="3528">
        <attvalues>
          <attvalue for="5" value=" This category of approaches might induce its own biases, as studied in a recent work \cite{liu2019inoculation} for NLI stress tests \cite{naik2018stress} and the Adversarial SQuAD dataset \cite{jia2017adversarial}." />
        </attvalues>
      </edge>
      <edge source="2005.00816" target="1909.12434" id="3529">
        <attvalues>
          <attvalue for="5" value=" This shouldn't disturb the sample's internal coherence, nor make unnecessary changes \cite{kaushik2019learning}." />
        </attvalues>
      </edge>
      <edge source="2406.06699" target="2303.18223" id="3532">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) have become ubiquitous in Deep Learning and have shown impressive capabilities in most NLP tasks~\cite{ZhaoEtAl23}." />
        </attvalues>
      </edge>
      <edge source="2406.06699" target="2301.00234" id="3533">
        <attvalues>
          <attvalue for="5" value=" ICL is a prompt engineering technique whereby an LLM is conditioned to solve tasks by means of a few solved demonstration examples included as part of its input prompt~\cite{DongEtAl23}." />
        </attvalues>
      </edge>
      <edge source="2406.06699" target="2201.11903" id="3535">
        <attvalues>
          <attvalue for="5" value=" Creative ICL strategies combining $k$NN-based examples selection, generated chain-of-thought (CoT) prompting, and majority vote ensembling have been proposed and shown to outperform fine-tuning approaches~\cite{Microsoft23,CoTWeiEtAl2023,LeiEtAl23,WangSelfConsistency23}." />
        </attvalues>
      </edge>
      <edge source="2406.06699" target="2203.11171" id="3536">
        <attvalues>
          <attvalue for="5" value=" Creative ICL strategies combining $k$NN-based examples selection, generated chain-of-thought (CoT) prompting, and majority vote ensembling have been proposed and shown to outperform fine-tuning approaches~\cite{Microsoft23,CoTWeiEtAl2023,LeiEtAl23,WangSelfConsistency23}." />
        </attvalues>
      </edge>
      <edge source="2406.06699" target="2303.13375" id="3537">
        <attvalues>
          <attvalue for="5" value=" In the main, $k$NN-based examples selection optimizes the process of learning from few examples and ensembling increases the robustness of the predictions~\cite{CapabilitiesNori23,Microsoft23,LeiEtAl23}." />
        </attvalues>
      </edge>
      <edge source="2406.06699" target="1612.08994" id="3539">
        <attvalues>
          <attvalue for="5" value=" This additional information can be incorporated via feature engineering~\cite{ParsingStabGurevych17}, memory-enabled neural architectures~\cite{PotashEtAl17,SpanRepKuribayashiEtAl19} or LLM-based hybrid methods~\cite{BERTFeaTxtMushtaqCabessa22,BERTMinusMushtaqCabessa23}." />
        </attvalues>
      </edge>
      <edge source="2406.06699" target="2303.08774" id="3540">
        <attvalues>
          <attvalue for="5" value=" In this training-free setting, we show that GPT-4~\cite{GPT42023} is able to leverage relevant information from only a few demonstration examples in order to achieve very competitive classification accuracy." />
        </attvalues>
      </edge>
      <edge source="2301.00234" target="2005.14165" id="3542">
        <attvalues>
          <attvalue for="5" value=" First, since the demonstration is written in natural language, it provides an interpretable interface to communicate with LLMs~\cite{gpt3}." />
        </attvalues>
      </edge>
      <edge source="2301.00234" target="2201.11903" id="3543">
        <attvalues>
          <attvalue for="5" value=" Many studies have shown that LLMs can perform a series of complex tasks through ICL, such as solving mathematical reasoning problems~\cite{cot}." />
        </attvalues>
      </edge>
      <edge source="2301.00234" target="2201.03514" id="3545">
        <attvalues>
          <attvalue for="5" value=" This could not only greatly reduce the computational costs for adapting the model to new tasks, but also make language-model-as-a-service~\cite{sun2022black} possible and can be easily applied to large-scale real-world tasks." />
        </attvalues>
      </edge>
      <edge source="2301.00234" target="2110.15943" id="3546">
        <attvalues>
          <attvalue for="5" value=" Although a range of vanilla GPT models show excellent ICL capability, several studies have found that this capability can be significantly improved through adaptation during pretraining~\cite{metaicl, Li2023mend}." />
        </attvalues>
      </edge>
      <edge source="2301.00234" target="2311.06668" id="3547">
        <attvalues>
          <attvalue for="5" value=" Additionally, optimizing the conciseness of demonstration examples and improving the computational efficiency of ICL are critical areas of ongoing research~\cite{liu2024incontext}." />
        </attvalues>
      </edge>
      <edge source="2107.01700" target="1609.08667" id="3549">
        <attvalues>
          <attvalue for="5" value=" Many traditional coreference resolution systems are pipelined systems, each consists of two separate components: (1) a mention detector for identifying entity mentions from text (2) a coreference resolver for clustering the extracted mentions \cite{raghunathanEtal2010multi,durrettklein2013easy,clarkmanning2015entity,wisemanetal2016learning,clarkmanning2016deep}." />
        </attvalues>
      </edge>
      <edge source="2107.01700" target="1707.07045" id="3550">
        <attvalues>
          <attvalue for="5" value=" In 2017, the first end-to-end coreference resolution model named e2e-coref was proposed \cite{leeetal2017end}." />
        </attvalues>
      </edge>
      <edge source="2107.01700" target="1804.05392" id="3552">
        <attvalues>
          <attvalue for="5" value=" Since then, many extensions to the e2e-coref model have been introduced, ranging from using higher-order inference to directly optimizing evaluation metrics using reinforcement learning \cite{zhangetal2018neuralcoreference,leeetal2018higher,Gu2018ASO,feietal2019end,kantorgloberson2019coreference,joshietal2019bert,ijcai20190700,joshi2020spanbert} (Figure \ref{fig:coref_timeline})." />
        </attvalues>
      </edge>
      <edge source="2107.01700" target="1908.09091" id="3553">
        <attvalues>
          <attvalue for="5" value=" Since then, many extensions to the e2e-coref model have been introduced, ranging from using higher-order inference to directly optimizing evaluation metrics using reinforcement learning \cite{zhangetal2018neuralcoreference,leeetal2018higher,Gu2018ASO,feietal2019end,kantorgloberson2019coreference,joshietal2019bert,ijcai20190700,joshi2020spanbert} (Figure \ref{fig:coref_timeline})." />
        </attvalues>
      </edge>
      <edge source="2107.01700" target="1907.10529" id="3554">
        <attvalues>
          <attvalue for="5" value=" Since then, many extensions to the e2e-coref model have been introduced, ranging from using higher-order inference to directly optimizing evaluation metrics using reinforcement learning \cite{zhangetal2018neuralcoreference,leeetal2018higher,Gu2018ASO,feietal2019end,kantorgloberson2019coreference,joshietal2019bert,ijcai20190700,joshi2020spanbert} (Figure \ref{fig:coref_timeline})." />
        </attvalues>
      </edge>
      <edge source="2402.12011" target="2304.01666" id="3558">
        <attvalues>
          <attvalue for="5" value=" However, the shift towards more advanced Transformer architectures has established the use of contextualized embedding models as the preferred tool for addressing GCD~\cite{montanelli2023survey,kutuzov2022contextualized}." />
        </attvalues>
      </edge>
      <edge source="2402.12011" target="1806.03537" id="3559">
        <attvalues>
          <attvalue for="5" value=" Lexical Semantic Change (LSC) is the problem of automatically identifying words that change their meaning over time~\cite{montanelli2023survey,tahmasebi2021survey,kutuzov2018diachronic,tang2018state}." />
        </attvalues>
      </edge>
      <edge source="2402.12011" target="1801.09872" id="3560">
        <attvalues>
          <attvalue for="5" value=" Lexical Semantic Change (LSC) is the problem of automatically identifying words that change their meaning over time~\cite{montanelli2023survey,tahmasebi2021survey,kutuzov2018diachronic,tang2018state}." />
        </attvalues>
      </edge>
      <edge source="2402.12011" target="2007.11464" id="3561">
        <attvalues>
          <attvalue for="5" value=" The initial excitement for word embeddings prompted researchers and practitioners to solve the GCD task by using static embedding models~\cite{schlechtweg2020semeval,shoemark2019room}." />
        </attvalues>
      </edge>
      <edge source="2402.12011" target="2104.08540" id="3562">
        <attvalues>
          <attvalue for="5" value=" Notably, the framework includes three distinct aspects~\cite{schlechtweg2021dwug}: \begin{itemize} \item [i)] semantic proximity judgments of word in-context, \item [ii)] word sense induction based on proximity judgments, \item [iii)] quantification of semantic change from induced senses." />
        </attvalues>
      </edge>
      <edge source="2003.11520" target="1607.06520" id="3564">
        <attvalues>
          <attvalue for="5" value=" Recent research established that popular embeddings are prone to substantial biases, eg, with respect to gender or race ~\cite{DBLP:journals/corr/abs-1711-08412,bolukbasi2016}, which demonstrated in results like ``Man is to Computer Programmer as Woman is to Homemaker&quot;~\cite{bolukbasi2016} as results of basic analogy tasks." />
        </attvalues>
      </edge>
      <edge source="2003.11520" target="1608.07187" id="3565">
        <attvalues>
          <attvalue for="5" value=" Thus, in this paper, we propose two new post-processing methods for joint/simultaneous multiclass debiasing, which differ in their trade-off between maintaining word relationships and decreasing bias levels: HardWEAT completely eliminates contained bias as measured by the established Word Embedding Association Test \cite{caliskan2017}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="1907.11692" id="3567">
        <attvalues>
          <attvalue for="5" value=" The conventional masking strategy for MLM selects tokens to mask with a uniform distribution~\cite{bert,roberta}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2011.05864" id="3569">
        <attvalues>
          <attvalue for="5" value=" Recent efforts on sentence representation modeling include calibration methods~\cite{on_the_sentence,whitening}, prompt learning\cite{radford2018gpt,radford2019gpt,lm_few_shot,cloze_question, making_plm_few_shot, promptbert}, and sentence-level contrastive learning (CL) based models such as SimCSE~\cite{simcse} and its variants~\cite{continuous_prompt,infocse}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2201.04337" id="3570">
        <attvalues>
          <attvalue for="5" value=" \cite{promptbert} demonstrates that frequency bias indeed harms the performance of sentence embeddings generated by MLM-trained PLMs." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="1909.08053" id="3571">
        <attvalues>
          <attvalue for="5" value=" Several recent studies focus on improving efficiency of pre-training, including mixed-precision training~\cite{megatron-lm}, parameter distillation for different layers~\cite{effecient_training}, introducing a note dictionary for saving information of rare tokens~\cite{tfn}, designing different training objectives~\cite{albert,electra,cocolm}, and dropping redundant tokens during pre-training~\cite{token_dropping}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="1909.11942" id="3572">
        <attvalues>
          <attvalue for="5" value=" Several recent studies focus on improving efficiency of pre-training, including mixed-precision training~\cite{megatron-lm}, parameter distillation for different layers~\cite{effecient_training}, introducing a note dictionary for saving information of rare tokens~\cite{tfn}, designing different training objectives~\cite{albert,electra,cocolm}, and dropping redundant tokens during pre-training~\cite{token_dropping}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2003.10555" id="3573">
        <attvalues>
          <attvalue for="5" value=" Several recent studies focus on improving efficiency of pre-training, including mixed-precision training~\cite{megatron-lm}, parameter distillation for different layers~\cite{effecient_training}, introducing a note dictionary for saving information of rare tokens~\cite{tfn}, designing different training objectives~\cite{albert,electra,cocolm}, and dropping redundant tokens during pre-training~\cite{token_dropping}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2102.08473" id="3574">
        <attvalues>
          <attvalue for="5" value=" Several recent studies focus on improving efficiency of pre-training, including mixed-precision training~\cite{megatron-lm}, parameter distillation for different layers~\cite{effecient_training}, introducing a note dictionary for saving information of rare tokens~\cite{tfn}, designing different training objectives~\cite{albert,electra,cocolm}, and dropping redundant tokens during pre-training~\cite{token_dropping}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2203.13240" id="3575">
        <attvalues>
          <attvalue for="5" value=" Several recent studies focus on improving efficiency of pre-training, including mixed-precision training~\cite{megatron-lm}, parameter distillation for different layers~\cite{effecient_training}, introducing a note dictionary for saving information of rare tokens~\cite{tfn}, designing different training objectives~\cite{albert,electra,cocolm}, and dropping redundant tokens during pre-training~\cite{token_dropping}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2103.15316" id="3577">
        <attvalues>
          <attvalue for="5" value=" Recent efforts on sentence representation modeling include calibration methods~\cite{on_the_sentence,whitening}, prompt learning\cite{radford2018gpt,radford2019gpt,lm_few_shot,cloze_question, making_plm_few_shot, promptbert}, and sentence-level contrastive learning (CL) based models such as SimCSE~\cite{simcse} and its variants~\cite{continuous_prompt,infocse}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2005.14165" id="3578">
        <attvalues>
          <attvalue for="5" value=" Recent efforts on sentence representation modeling include calibration methods~\cite{on_the_sentence,whitening}, prompt learning\cite{radford2018gpt,radford2019gpt,lm_few_shot,cloze_question, making_plm_few_shot, promptbert}, and sentence-level contrastive learning (CL) based models such as SimCSE~\cite{simcse} and its variants~\cite{continuous_prompt,infocse}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2104.08821" id="3579">
        <attvalues>
          <attvalue for="5" value=" Recent efforts on sentence representation modeling include calibration methods~\cite{on_the_sentence,whitening}, prompt learning\cite{radford2018gpt,radford2019gpt,lm_few_shot,cloze_question, making_plm_few_shot, promptbert}, and sentence-level contrastive learning (CL) based models such as SimCSE~\cite{simcse} and its variants~\cite{continuous_prompt,infocse}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2203.06875" id="3580">
        <attvalues>
          <attvalue for="5" value=" Recent efforts on sentence representation modeling include calibration methods~\cite{on_the_sentence,whitening}, prompt learning\cite{radford2018gpt,radford2019gpt,lm_few_shot,cloze_question, making_plm_few_shot, promptbert}, and sentence-level contrastive learning (CL) based models such as SimCSE~\cite{simcse} and its variants~\cite{continuous_prompt,infocse}." />
        </attvalues>
      </edge>
      <edge source="2302.14225" target="2111.04198" id="3582">
        <attvalues>
          <attvalue for="5" value=" Although SimCSE and its variants achieve state-of-the-art (SOTA) performance on STS, they degrade the transfer learning capability on tasks such as SQuAD since they do not target improving token-level representation learning~\cite{tacl}." />
        </attvalues>
      </edge>
      <edge source="2304.11389" target="2212.12131" id="3586">
        <attvalues>
          <attvalue for="5" value=" While \cite{ohschuler23tacl} conjecture that these studies capture two distinct regimes, it remains less clear where the reversal in this relationship happens." />
        </attvalues>
      </edge>
      <edge source="2203.12865" target="1911.02116" id="3600">
        <attvalues>
          <attvalue for="5" value=" Multilingual transformer based models \cite{bert,xlmr,liu2020multilingual,xue2021mt5} have demonstrated commendable zero \&amp; few-shot capabilities." />
        </attvalues>
      </edge>
      <edge source="2203.12865" target="2001.08210" id="3601">
        <attvalues>
          <attvalue for="5" value=" Multilingual transformer based models \cite{bert,xlmr,liu2020multilingual,xue2021mt5} have demonstrated commendable zero \&amp; few-shot capabilities." />
        </attvalues>
      </edge>
      <edge source="2203.12865" target="2010.11934" id="3602">
        <attvalues>
          <attvalue for="5" value=" Multilingual transformer based models \cite{bert,xlmr,liu2020multilingual,xue2021mt5} have demonstrated commendable zero \&amp; few-shot capabilities." />
        </attvalues>
      </edge>
      <edge source="2203.12865" target="2003.11080" id="3604">
        <attvalues>
          <attvalue for="5" value=" Their performance is typically evaluated on benchmarks like XNLI \cite{xnli}, XGLUE \cite{xglue}, XTREME \cite{hu2020xtreme} \&amp; XTREME-R \cite{ruder2021xtremer}." />
        </attvalues>
      </edge>
      <edge source="2203.12865" target="2104.07412" id="3605">
        <attvalues>
          <attvalue for="5" value=" In the manual approach, we ask annotators to create CheckLists in two ways: first, by translation of English CheckList to the target language (t9n) (same as \cite{ruder2021xtremer}); Second, by giving a description of the task and capabilities to create CheckLists from scratch (SCR) (same as original English CheckLists creation \cite{checklist-paper})." />
        </attvalues>
      </edge>
      <edge source="2203.12865" target="2205.06356" id="3606">
        <attvalues>
          <attvalue for="5" value=" However, this evaluation paradigm has a number of limitations including: First, most of these datasets are limited to a few high resource languages~\cite{pmlr-v119-hu20b, wang-etal-2020-extending, vulic-etal-2020-multi}, except for a few tasks (eg, NER, POS \cite{ahuja-etal-2022-beyond, bhatt-2021-on})." />
        </attvalues>
      </edge>
      <edge source="2203.12865" target="1612.00837" id="3607">
        <attvalues>
          <attvalue for="5" value=" Third, state-of-art models are known to learn spurious patterns to achieve high accuracies, saturating performance on these test-benches, yet performing poorly on often much simpler real world cases~\cite{balanced_vqa_v2, gururangan-etal-2018-annotation, glockner2018breaking, tsuchiya2018performance, geva2019we}." />
        </attvalues>
      </edge>
      <edge source="2203.12865" target="2206.04615" id="3609">
        <attvalues>
          <attvalue for="5" value=" These limitations lead to the need of interactive, challenging, and much larger testing datasets (like \cite{big-bench, dyna-bench}) and more holistic approaches to evaluation (like \cite{checklist-paper})." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1504.04666" id="3610">
        <attvalues>
          <attvalue for="5" value=" For unsupervised dependency parsing, \cite{le2015unsupervised} obtain strong results by training a supervised parser on outputs of unsupervised parsing." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1906.10225" id="3611">
        <attvalues>
          <attvalue for="5" value=" Several models have been introduced since: \cite{shen2018ordered} propose an architecture consisting of an LSTM \cite{hochreiter1997long} with a modified update function for the LSTM cell state, \cite{kim2019compound}---the current state-of-the-art---introduce a model based on a mixture of probabilistic context-free grammars, \cite{kim2019unsupervised} present unsupervised learning of recurrent neural networks grammars, \cite{li2019imitation} combine PRPN with imitation learning, and \cite{drozdov2019unsupervised} employ a recursive autoencoder." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1711.02013" id="3612">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{shen2017neural} introduce the first high performing neural UP model \cite{htut2018grammar}." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1806.04168" id="3613">
        <attvalues>
          <attvalue for="5" value=" Our approach draws on the idea of syntactic distances, which can be learned both as latent variables \cite{shen2017neural} and as explicit supervision targets \cite{shen2018straight}." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1611.09100" id="3614">
        <attvalues>
          <attvalue for="5" value=" Related Work Following the line of research on non-neural UP models \cite{clark2001unsupervised,klein2002generative,bod-2006-subtrees}, early approaches to neural UP \cite{yogatama2016learning,choi2018learning} obtain improved performance on downstream tasks, yet show highly inconsistent behavior in parsing \cite{williams2018latent}." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1707.02786" id="3615">
        <attvalues>
          <attvalue for="5" value=" Related Work Following the line of research on non-neural UP models \cite{clark2001unsupervised,klein2002generative,bod-2006-subtrees}, early approaches to neural UP \cite{yogatama2016learning,choi2018learning} obtain improved performance on downstream tasks, yet show highly inconsistent behavior in parsing \cite{williams2018latent}." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1709.01121" id="3616">
        <attvalues>
          <attvalue for="5" value=" Related Work Following the line of research on non-neural UP models \cite{clark2001unsupervised,klein2002generative,bod-2006-subtrees}, early approaches to neural UP \cite{yogatama2016learning,choi2018learning} obtain improved performance on downstream tasks, yet show highly inconsistent behavior in parsing \cite{williams2018latent}." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1808.10000" id="3617">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{shen2017neural} introduce the first high performing neural UP model \cite{htut2018grammar}." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1909.09428" id="3618">
        <attvalues>
          <attvalue for="5" value=" \cite{dyer2019critical} raise concerns that PRPN's parsing methodology is biased towards English trees." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1810.09536" id="3619">
        <attvalues>
          <attvalue for="5" value=" Several models have been introduced since: \cite{shen2018ordered} propose an architecture consisting of an LSTM \cite{hochreiter1997long} with a modified update function for the LSTM cell state, \cite{kim2019compound}---the current state-of-the-art---introduce a model based on a mixture of probabilistic context-free grammars, \cite{kim2019unsupervised} present unsupervised learning of recurrent neural networks grammars, \cite{li2019imitation} combine PRPN with imitation learning, and \cite{drozdov2019unsupervised} employ a recursive autoencoder." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1904.03746" id="3620">
        <attvalues>
          <attvalue for="5" value=" Several models have been introduced since: \cite{shen2018ordered} propose an architecture consisting of an LSTM \cite{hochreiter1997long} with a modified update function for the LSTM cell state, \cite{kim2019compound}---the current state-of-the-art---introduce a model based on a mixture of probabilistic context-free grammars, \cite{kim2019unsupervised} present unsupervised learning of recurrent neural networks grammars, \cite{li2019imitation} combine PRPN with imitation learning, and \cite{drozdov2019unsupervised} employ a recursive autoencoder." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="1906.02276" id="3621">
        <attvalues>
          <attvalue for="5" value=" Several models have been introduced since: \cite{shen2018ordered} propose an architecture consisting of an LSTM \cite{hochreiter1997long} with a modified update function for the LSTM cell state, \cite{kim2019compound}---the current state-of-the-art---introduce a model based on a mixture of probabilistic context-free grammars, \cite{kim2019unsupervised} present unsupervised learning of recurrent neural networks grammars, \cite{li2019imitation} combine PRPN with imitation learning, and \cite{drozdov2019unsupervised} employ a recursive autoencoder." />
        </attvalues>
      </edge>
      <edge source="2005.13455" target="2002.00737" id="3622">
        <attvalues>
          <attvalue for="5" value=" \cite{kim2020pre} examine tree induction from pre-trained models." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="1711.06861" id="3624">
        <attvalues>
          <attvalue for="5" value=" More modern, neural approaches attempt to learn style representations in an unsupervised fashion through a proxy task like style transfer \cite{styletransfervector3,styletransfervector4,styletransfervector1,styletransfervector6,styletransfervector7,styletransfervector5,styletransfervector2} or authorship verification \cite{avvector1,avvector2,avvector3,styleemb}." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="1905.05621" id="3626">
        <attvalues>
          <attvalue for="5" value=" More modern, neural approaches attempt to learn style representations in an unsupervised fashion through a proxy task like style transfer \cite{styletransfervector3,styletransfervector4,styletransfervector1,styletransfervector6,styletransfervector7,styletransfervector5,styletransfervector2} or authorship verification \cite{avvector1,avvector2,avvector3,styleemb}." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="1908.09395" id="3627">
        <attvalues>
          <attvalue for="5" value=" More modern, neural approaches attempt to learn style representations in an unsupervised fashion through a proxy task like style transfer \cite{styletransfervector3,styletransfervector4,styletransfervector1,styletransfervector6,styletransfervector7,styletransfervector5,styletransfervector2} or authorship verification \cite{avvector1,avvector2,avvector3,styleemb}." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="2212.09670" id="3628">
        <attvalues>
          <attvalue for="5" value=" More modern, neural approaches attempt to learn style representations in an unsupervised fashion through a proxy task like style transfer \cite{styletransfervector3,styletransfervector4,styletransfervector1,styletransfervector6,styletransfervector7,styletransfervector5,styletransfervector2} or authorship verification \cite{avvector1,avvector2,avvector3,styleemb}." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="1908.07844" id="3629">
        <attvalues>
          <attvalue for="5" value=" More modern, neural approaches attempt to learn style representations in an unsupervised fashion through a proxy task like style transfer \cite{styletransfervector3,styletransfervector4,styletransfervector1,styletransfervector6,styletransfervector7,styletransfervector5,styletransfervector2} or authorship verification \cite{avvector1,avvector2,avvector3,styleemb}." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="2109.03158" id="3630">
        <attvalues>
          <attvalue for="5" value=" More modern, neural approaches attempt to learn style representations in an unsupervised fashion through a proxy task like style transfer \cite{styletransfervector3,styletransfervector4,styletransfervector1,styletransfervector6,styletransfervector7,styletransfervector5,styletransfervector2} or authorship verification \cite{avvector1,avvector2,avvector3,styleemb}." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="2204.04907" id="3631">
        <attvalues>
          <attvalue for="5" value=" More modern, neural approaches attempt to learn style representations in an unsupervised fashion through a proxy task like style transfer \cite{styletransfervector3,styletransfervector4,styletransfervector1,styletransfervector6,styletransfervector7,styletransfervector5,styletransfervector2} or authorship verification \cite{avvector1,avvector2,avvector3,styleemb}." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="2005.14165" id="3632">
        <attvalues>
          <attvalue for="5" value=" Given this, we use GPT-3 \cite{gpt3}, a large language model (LLM), and zero-shot prompts to generate a synthetic dataset we call StyleGenome of human-interpretable stylometric annotations for various texts." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="2210.11610" id="3633">
        <attvalues>
          <attvalue for="5" value=" Our approach is motivated by recent works showing models trained on synthetic datasets annotated by prompting LLMs can match and sometimes even outperform models trained on human-labeled datasets \cite{llmannotate1, llmannotate2,llmdistill2,llmdistill3}." />
        </attvalues>
      </edge>
      <edge source="2305.12696" target="2212.09689" id="3634">
        <attvalues>
          <attvalue for="5" value=" Our approach is motivated by recent works showing models trained on synthetic datasets annotated by prompting LLMs can match and sometimes even outperform models trained on human-labeled datasets \cite{llmannotate1, llmannotate2,llmdistill2,llmdistill3}." />
        </attvalues>
      </edge>
      <edge source="2005.01330" target="1807.07987" id="3635">
        <attvalues>
          <attvalue for="5" value=" The introduction of deep learning models \cite{deeplearn} into Natural Language Processing (NLP) has led to an explosion in the Neural models and pre-training techniques applied to NLP tasks --- from classical tasks as tagging and parsing to end-to-end tasks as machine translation and question answering --- raising the performance bar on these tasks to an all-times peak." />
        </attvalues>
      </edge>
      <edge source="2005.01330" target="1510.00726" id="3636">
        <attvalues>
          <attvalue for="5" value=" Upon the introduction of Neural Network models into NLP \cite{primer}, it was hoped that we could dispense with the need to model different languages differently." />
        </attvalues>
      </edge>
      <edge source="2005.01330" target="1808.09180" id="3637">
        <attvalues>
          <attvalue for="5" value=" While some morphological structure may be implicitly encoded in these vectors, the {morphemes} themselves remain un-accessible \cite{claraDEP,cotterell15}." />
        </attvalues>
      </edge>
      <edge source="2305.11498" target="1909.03546" id="3639">
        <attvalues>
          <attvalue for="5" value=" Inspired by the significant performance of PLMs, some prior work \cite{wang2019adversarial,wadden2019entity} utilize general PLMs, such as BERT \cite{devlin2019bert}, to construct global dependencies among context words by self-attention." />
        </attvalues>
      </edge>
      <edge source="2305.11498" target="2211.10991" id="3642">
        <attvalues>
          <attvalue for="5" value=" To avoid the problem of auxiliary information insufficiency, dependency tree based Graph Convolution Network (GCN) \cite{nguyen2018graph,liu2018jointly,DBLP:conf/wsdm/WuBG0LY23} was adopted to capture syntactic relations between triggers and related words." />
        </attvalues>
      </edge>
      <edge source="2305.11498" target="1812.08434" id="3643">
        <attvalues>
          <attvalue for="5" value=" However, this method still has some problems: (1) GCN focuses on the nearest syntactic neighbors ~\cite{nguyen2018graph,liu2018jointly}, over-smoothing \cite{zhou2020graph} in deep layers limits the message passing; (2) GCN has not modeled the event fields and entanglement among the identical event fields." />
        </attvalues>
      </edge>
      <edge source="2305.11498" target="1810.10182" id="3644">
        <attvalues>
          <attvalue for="5" value=" Another localness-enhanced method, which models central word regions as Gaussian priors \cite{yang2018modeling,guo2019gaussian}, could expand the neighbor scopes without changing the model structure." />
        </attvalues>
      </edge>
      <edge source="2310.16810" target="2305.18486" id="3645">
        <attvalues>
          <attvalue for="5" value=" The emergence of prompt-driven Large Language Models (LLMs) has led to extensive investigations into their capabilities in a wide range of representative task categories \cite{laskar-etal-2023-systematic, qin2023chatgpt, bang2023multitask}, including summarization." />
        </attvalues>
      </edge>
      <edge source="2310.16810" target="2302.04023" id="3646">
        <attvalues>
          <attvalue for="5" value=" The results of these studies consistently indicate that ChatGPT, in general, underperforms in summarization tasks compared to task-specific pre-trained models \cite{bang2023multitask, zhang2023extractive, yang2023exploring}." />
        </attvalues>
      </edge>
      <edge source="2310.16810" target="2302.14229" id="3647">
        <attvalues>
          <attvalue for="5" value=" This is also the case for multilingual summarization \cite{wang2023zeroshot}." />
        </attvalues>
      </edge>
      <edge source="2310.16810" target="2302.06476" id="3648">
        <attvalues>
          <attvalue for="5" value=" As for dialogue summarization, only \cite{qin2023chatgpt, laskar-etal-2023-systematic} studied ChatGPT's performance using SAMSum \cite{gliwa-etal-2019-samsum} and DialogSum \cite{chen-etal-2021-dialogsum}." />
        </attvalues>
      </edge>
      <edge source="2310.16810" target="1911.12237" id="3649">
        <attvalues>
          <attvalue for="5" value=" As for dialogue summarization, only \cite{qin2023chatgpt, laskar-etal-2023-systematic} studied ChatGPT's performance using SAMSum \cite{gliwa-etal-2019-samsum} and DialogSum \cite{chen-etal-2021-dialogsum}." />
        </attvalues>
      </edge>
      <edge source="2310.16810" target="2105.06762" id="3650">
        <attvalues>
          <attvalue for="5" value=" We reported ROUGE and BERTScore, with experiments conducted on two datasets -- DialogSum \cite{chen-etal-2021-dialogsum} and DECODA \cite{favre-etal-2015-call}." />
        </attvalues>
      </edge>
      <edge source="2310.16810" target="2210.17378" id="3651">
        <attvalues>
          <attvalue for="5" value=" While dialogue summarization is a well-established task, its task formulation is still subject to discussion in the linguistic and NLP communities, which has prevented a consensual definition of what a dialogue summarization should look like \cite{guo-etal-2022-questioning}." />
        </attvalues>
      </edge>
      <edge source="2310.16810" target="2108.13139" id="3652">
        <attvalues>
          <attvalue for="5" value=" For example, in the case of customer service corpora, while TWEETSUMM \cite{feigenblat-etal-2021-tweetsumm-dialog} offers both extractive and abstractive summaries, CSDS \cite{lin-etal-2021-csds} provides three distinct summaries for each dialogue: an overall summary and two role-oriented summaries (user and agent)." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1310.4546" id="3653">
        <attvalues>
          <attvalue for="5" value=" The concept of word embeddings has been prevalent in NLP community in recent years, as they can characterize semantic similarity between any pair of words, achieving promising results in a large number of NLP tasks \cite{mikolov2013distributed, pennington2014glove, Salle2016MatrixFU}." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1506.06726" id="3654">
        <attvalues>
          <attvalue for="5" value=" SkipThought \cite{kiros2015skip} is an encoder-decoder model that predicts adjacent sentences." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1703.02507" id="3655">
        <attvalues>
          <attvalue for="5" value=" \cite{pgj2017unsup} proposes an unsupervised model, Sent2Vec, to learn an n-gram feature in a sentence to predict the center word from the surrounding context." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1803.02893" id="3656">
        <attvalues>
          <attvalue for="5" value=" Quick thoughts (QT) \cite{logeswaran2018efficient} replaces the encoder with a classifier to predict context sentences from candidate sequences." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1805.05388" id="3657">
        <attvalues>
          <attvalue for="5" value=" \cite{khodak2018carte} proposes $\grave{a}\: la \: carte$ to learn a linear mapping to reconstruct the center word from its context." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1705.02364" id="3658">
        <attvalues>
          <attvalue for="5" value=" \cite{conneau-EtAl:2017:EMNLP2017} generates the sentence encoder InferSent using Natural Language Inference (NLI) dataset." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1804.07754" id="3659">
        <attvalues>
          <attvalue for="5" value=" Universal Sentence Encoder \cite{2018arXiv180407754Y, cer2018universal} utilizes the emerging transformer structure \cite{vaswani2017attention, devlin2018bert} that has been proved powerful in various NLP tasks." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1706.03762" id="3660">
        <attvalues>
          <attvalue for="5" value=" Universal Sentence Encoder \cite{2018arXiv180407754Y, cer2018universal} utilizes the emerging transformer structure \cite{vaswani2017attention, devlin2018bert} that has been proved powerful in various NLP tasks." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1705.00364" id="3661">
        <attvalues>
          <attvalue for="5" value=" \cite{wieting-17-recurrent} propose the gated recurrent averaging network (GRAN), which is trained on Paraphrase Database (PPDB) and English Wikipedia." />
        </attvalues>
      </edge>
      <edge source="1810.00438" target="1804.00079" id="3662">
        <attvalues>
          <attvalue for="5" value=" \cite{subramanian2018learning} leverages a multi-task learning framework to generate sentence embeddings." />
        </attvalues>
      </edge>
      <edge source="1911.06118" target="1606.05464" id="3664">
        <attvalues>
          <attvalue for="5" value=" Learning word embedding aids in natural language processing tasks such as question answering and reasoning \cite{choi2018quac}, stance detection \cite{augenstein2016stance}, claim verification \cite{hanselowski2018ukp}." />
        </attvalues>
      </edge>
      <edge source="1911.06118" target="1809.01479" id="3665">
        <attvalues>
          <attvalue for="5" value=" Learning word embedding aids in natural language processing tasks such as question answering and reasoning \cite{choi2018quac}, stance detection \cite{augenstein2016stance}, claim verification \cite{hanselowski2018ukp}." />
        </attvalues>
      </edge>
      <edge source="1911.06118" target="1504.06654" id="3667">
        <attvalues>
          <attvalue for="5" value=" \cite{neelakantan2015efficient} presents a non-parametric based alternative to handle polysemies." />
        </attvalues>
      </edge>
      <edge source="1911.06118" target="1412.6623" id="3668">
        <attvalues>
          <attvalue for="5" value=" \cite{vilnis2014word} learn a Gaussian distribution per word using the expected likelihood kernel." />
        </attvalues>
      </edge>
      <edge source="1911.06118" target="1704.08424" id="3669">
        <attvalues>
          <attvalue for="5" value=" \cite{athiwaratkun2017multimodal} proposes multimodal word distribution approach." />
        </attvalues>
      </edge>
      <edge source="2408.11443" target="1508.07909" id="3670">
        <attvalues>
          <attvalue for="5" value=" Popular subword tokenization schemes are BPE \cite{sennrich-etal-2016-neural}, MaxMatch/WordPiece \cite{wu2016googles}, and UnigramLM \cite{kudo-2018-subword}." />
        </attvalues>
      </edge>
      <edge source="2408.11443" target="1609.08144" id="3671">
        <attvalues>
          <attvalue for="5" value=" Popular subword tokenization schemes are BPE \cite{sennrich-etal-2016-neural}, MaxMatch/WordPiece \cite{wu2016googles}, and UnigramLM \cite{kudo-2018-subword}." />
        </attvalues>
      </edge>
      <edge source="2408.11443" target="1804.10959" id="3672">
        <attvalues>
          <attvalue for="5" value=" Popular subword tokenization schemes are BPE \cite{sennrich-etal-2016-neural}, MaxMatch/WordPiece \cite{wu2016googles}, and UnigramLM \cite{kudo-2018-subword}." />
        </attvalues>
      </edge>
      <edge source="2408.11443" target="1910.13267" id="3673">
        <attvalues>
          <attvalue for="5" value=" There are two main types of stochastic tokenizers: those which learn a distribution from text (eg, UnigramLM) and those which inject randomness by corrupting the tokenization scheme (eg, BPE-Dropout, \cite{provilkov-etal-2020-bpe} and MaxMatch-Dropout, \cite{hiraoka-2022-maxmatch})." />
        </attvalues>
      </edge>
      <edge source="2408.11443" target="2209.04126" id="3674">
        <attvalues>
          <attvalue for="5" value=" There are two main types of stochastic tokenizers: those which learn a distribution from text (eg, UnigramLM) and those which inject randomness by corrupting the tokenization scheme (eg, BPE-Dropout, \cite{provilkov-etal-2020-bpe} and MaxMatch-Dropout, \cite{hiraoka-2022-maxmatch})." />
        </attvalues>
      </edge>
      <edge source="2204.14146" target="2005.14165" id="3675">
        <attvalues>
          <attvalue for="5" value=" We validate our algorithm on a carefully-controlled synthetic task of removing offensive words from a sentence with GPT-3-based models~\cite{brown2020language,ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2204.14146" target="2109.07958" id="3676">
        <attvalues>
          <attvalue for="5" value=" A key problem with LMs is that they generate text that violates human preferences, such as LM-generated misinformation \cite{lin2021truthfulqa}, offensive language \cite{gehman2020realtoxicityprompts}, and factually incorrect outputs such as summaries \cite{stiennon2020learning}." />
        </attvalues>
      </edge>
      <edge source="2204.14146" target="2009.11462" id="3677">
        <attvalues>
          <attvalue for="5" value=" A key problem with LMs is that they generate text that violates human preferences, such as LM-generated misinformation \cite{lin2021truthfulqa}, offensive language \cite{gehman2020realtoxicityprompts}, and factually incorrect outputs such as summaries \cite{stiennon2020learning}." />
        </attvalues>
      </edge>
      <edge source="2204.14146" target="1909.08593" id="3678">
        <attvalues>
          <attvalue for="5" value=" Our algorithm departs from prior work, which uses reinforcement learning methods~\cite[][inter alia]{ziegler2019fine} or auxiliary losses~\cite{stacey2021natural} that cannot be straightforwardly generalized to using natural language feedback." />
        </attvalues>
      </edge>
      <edge source="2204.14146" target="2104.08142" id="3680">
        <attvalues>
          <attvalue for="5" value=" Our algorithm departs from prior work, which uses reinforcement learning methods~\cite[][inter alia]{ziegler2019fine} or auxiliary losses~\cite{stacey2021natural} that cannot be straightforwardly generalized to using natural language feedback." />
        </attvalues>
      </edge>
      <edge source="1911.12559" target="1704.06879" id="3681">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art systems for this task rely on recurrent neural networks~\cite{P17-1054,chen-EtAl:2018:EMNLP9,chen-etal-2019-integrated}, and hence require large amounts of annotated training data to achieve good performance." />
        </attvalues>
      </edge>
      <edge source="1911.12559" target="1808.07185" id="3682">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art systems for this task rely on recurrent neural networks~\cite{P17-1054,chen-EtAl:2018:EMNLP9,chen-etal-2019-integrated}, and hence require large amounts of annotated training data to achieve good performance." />
        </attvalues>
      </edge>
      <edge source="1911.12559" target="1904.03454" id="3683">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art systems for this task rely on recurrent neural networks~\cite{P17-1054,chen-EtAl:2018:EMNLP9,chen-etal-2019-integrated}, and hence require large amounts of annotated training data to achieve good performance." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="1606.03777" id="3684">
        <attvalues>
          <attvalue for="5" value=" Task-oriented dialog models form the core technology behind these applications, which understands users' natural language utterances \cite{Hakkani-TurTCCG16,goo2018slot}, keeps track of the conversation \cite{MrksicSWTY17,ChenLWZT020}, performs requested tasks (eg API calls) \cite{WenMBY17,GaoWPLL18}, and generates appropriate meaningful response to the user \cite{WenGMSVY15,ZhangSGCBGGLD20}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="1801.06176" id="3686">
        <attvalues>
          <attvalue for="5" value=" Task-oriented dialog models form the core technology behind these applications, which understands users' natural language utterances \cite{Hakkani-TurTCCG16,goo2018slot}, keeps track of the conversation \cite{MrksicSWTY17,ChenLWZT020}, performs requested tasks (eg API calls) \cite{WenMBY17,GaoWPLL18}, and generates appropriate meaningful response to the user \cite{WenGMSVY15,ZhangSGCBGGLD20}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="1508.01745" id="3687">
        <attvalues>
          <attvalue for="5" value=" Task-oriented dialog models form the core technology behind these applications, which understands users' natural language utterances \cite{Hakkani-TurTCCG16,goo2018slot}, keeps track of the conversation \cite{MrksicSWTY17,ChenLWZT020}, performs requested tasks (eg API calls) \cite{WenMBY17,GaoWPLL18}, and generates appropriate meaningful response to the user \cite{WenGMSVY15,ZhangSGCBGGLD20}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="2005.00796" id="3688">
        <attvalues>
          <attvalue for="5" value=" Training neural task-oriented dialog models \cite{SimpleTOD:Hosseini-AslMWY20,SOLOIST:PengLLSLG21,UBAR:YangLQ21}, requires a large amount of annotated data, which is difficult to obtain for model developers." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="1801.04871" id="3690">
        <attvalues>
          <attvalue for="5" value=" While crowd-sourcing and dialog simulation based on agent interplay \cite{shah2018building,Simulator:LinAEBZB20} addresses this issue to a certain extent, these are slow and don't provide sufficient coverage of different natural language (NL) user turn surface form variations." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="2004.15006" id="3693">
        <attvalues>
          <attvalue for="5" value=" Recently, large pre-trained language models (eg GPT-2 \cite{GPT2:radford2019language}, T5 \cite{T5:RaffelSRLNMZLL20}) have been successfully used to generate fluent agent dialog responses, both with dialog context \cite{GuWWSY20,ZhangSGCBGGLD20,KaleR20} or without it \cite{KaleR20a,XuWKL20}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="2005.10433" id="3694">
        <attvalues>
          <attvalue for="5" value=" Recently, large pre-trained language models (eg GPT-2 \cite{GPT2:radford2019language}, T5 \cite{T5:RaffelSRLNMZLL20}) have been successfully used to generate fluent agent dialog responses, both with dialog context \cite{GuWWSY20,ZhangSGCBGGLD20,KaleR20} or without it \cite{KaleR20a,XuWKL20}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="2106.05589" id="3695">
        <attvalues>
          <attvalue for="5" value=" Recently, large pre-trained language models (eg GPT-2 \cite{GPT2:radford2019language}, T5 \cite{T5:RaffelSRLNMZLL20}) have been successfully used to generate fluent agent dialog responses, both with dialog context \cite{GuWWSY20,ZhangSGCBGGLD20,KaleR20} or without it \cite{KaleR20a,XuWKL20}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="1807.01554" id="3696">
        <attvalues>
          <attvalue for="5" value=" Previous work on data augmentation for spoken language understanding has largely focused on generating paraphrases of user utterance, with a specific goal and set of entities \cite{HouLCL:18,ZhaoZY19,LinXZZZ21}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="1908.10770" id="3697">
        <attvalues>
          <attvalue for="5" value=" Previous work on data augmentation for spoken language understanding has largely focused on generating paraphrases of user utterance, with a specific goal and set of entities \cite{HouLCL:18,ZhaoZY19,LinXZZZ21}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="2108.08451" id="3698">
        <attvalues>
          <attvalue for="5" value=" Previous work on data augmentation for spoken language understanding has largely focused on generating paraphrases of user utterance, with a specific goal and set of entities \cite{HouLCL:18,ZhaoZY19,LinXZZZ21}." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="1910.13461" id="3699">
        <attvalues>
          <attvalue for="5" value=" Therefore, in this paper, we propose a novel dialog augmentation model, using BART \cite{BART:LewisLGGMLSZ20}, which can generate variations of a user turn, when conditioned on past and future dialog turns." />
        </attvalues>
      </edge>
      <edge source="2310.10380" target="1907.01669" id="3700">
        <attvalues>
          <attvalue for="5" value=" On benchmark MutiWoZ \cite{Multiwoz21:EricGPSAGKGKH20} and SGD \cite{SGD:RastogiZSGK20} datasets, using dialogs generated from our augmentation model can significantly improve dialog success rate and goal accuracy compared to state-of-the-art baseline models." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1409.0473" id="3702">
        <attvalues>
          <attvalue for="5" value=" In recent years, enormous progress has been made in the neural text-to-speech (TTS), which benefits from the development of sequence-to-sequence (seq2seq) neural models~\cite{bahdanau2014neural,sutskever2014sequence}, making it possible to synthesize highly intelligible and natural speech~\cite{wang2017tacotron,shen2018natural,ren2019fastspeech,yu2020durian}." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1409.3215" id="3703">
        <attvalues>
          <attvalue for="5" value=" In recent years, enormous progress has been made in the neural text-to-speech (TTS), which benefits from the development of sequence-to-sequence (seq2seq) neural models~\cite{bahdanau2014neural,sutskever2014sequence}, making it possible to synthesize highly intelligible and natural speech~\cite{wang2017tacotron,shen2018natural,ren2019fastspeech,yu2020durian}." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1703.10135" id="3704">
        <attvalues>
          <attvalue for="5" value=" In recent years, enormous progress has been made in the neural text-to-speech (TTS), which benefits from the development of sequence-to-sequence (seq2seq) neural models~\cite{bahdanau2014neural,sutskever2014sequence}, making it possible to synthesize highly intelligible and natural speech~\cite{wang2017tacotron,shen2018natural,ren2019fastspeech,yu2020durian}." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1712.05884" id="3705">
        <attvalues>
          <attvalue for="5" value=" In recent years, enormous progress has been made in the neural text-to-speech (TTS), which benefits from the development of sequence-to-sequence (seq2seq) neural models~\cite{bahdanau2014neural,sutskever2014sequence}, making it possible to synthesize highly intelligible and natural speech~\cite{wang2017tacotron,shen2018natural,ren2019fastspeech,yu2020durian}." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1711.05447" id="3707">
        <attvalues>
          <attvalue for="5" value=" To create a TTS system with the ability to synthesize various expressive speech, a straightforward method is to train a TTS model with a database with manual labels~\cite{lee2017emotional,choi2019multi,litao2021controllable,li2018emphatic,liu2021expressive}, for instance, a database with manually labeled emotion categories~\cite{lee2017emotional,litao2021controllable} or speaking styles~\cite{liu2021expressive}." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="2011.08679" id="3708">
        <attvalues>
          <attvalue for="5" value=" To transplant a style to a target speaker for whom no labeled expressive recording exists, the cross-speaker style transfer task has attracted much attention~\cite{bian2019multi,whitehill2019multi,karlapati2020copycat,litao2021controllable,pan2021cross,shang2021incorporatingM3}." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="2008.01490" id="3709">
        <attvalues>
          <attvalue for="5" value=" To create a TTS system with the ability to synthesize various expressive speech, a straightforward method is to train a TTS model with a database with manual labels~\cite{lee2017emotional,choi2019multi,litao2021controllable,li2018emphatic,liu2021expressive}, for instance, a database with manually labeled emotion categories~\cite{lee2017emotional,litao2021controllable} or speaking styles~\cite{liu2021expressive}." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1910.11958" id="3710">
        <attvalues>
          <attvalue for="5" value=" Reference embedding-based cross-speaker style transfer models~\cite{bian2019multi,whitehill2019multi,karlapati2020copycat,shang2021incorporatingM3,Li2021ControllableCE}, typically based on several general reference embedding methods \cite{skerry2018towards,wang2018style,zhang2019learning}, have shown promising performance on the style transfer task." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="2107.12562" id="3712">
        <attvalues>
          <attvalue for="5" value=" Taking inspiration from the success of the label-assisted content-aware prosody prediction model on the style transfer task~\cite{pan2021cross}, a novel method for the SRM2TTS task is proposed in this work." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1803.09047" id="3713">
        <attvalues>
          <attvalue for="5" value=" Reference embedding-based cross-speaker style transfer models~\cite{bian2019multi,whitehill2019multi,karlapati2020copycat,shang2021incorporatingM3,Li2021ControllableCE}, typically based on several general reference embedding methods \cite{skerry2018towards,wang2018style,zhang2019learning}, have shown promising performance on the style transfer task." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1803.09017" id="3714">
        <attvalues>
          <attvalue for="5" value=" Reference embedding-based cross-speaker style transfer models~\cite{bian2019multi,whitehill2019multi,karlapati2020copycat,shang2021incorporatingM3,Li2021ControllableCE}, typically based on several general reference embedding methods \cite{skerry2018towards,wang2018style,zhang2019learning}, have shown promising performance on the style transfer task." />
        </attvalues>
      </edge>
      <edge source="2112.12743" target="1812.04342" id="3715">
        <attvalues>
          <attvalue for="5" value=" Reference embedding-based cross-speaker style transfer models~\cite{bian2019multi,whitehill2019multi,karlapati2020copycat,shang2021incorporatingM3,Li2021ControllableCE}, typically based on several general reference embedding methods \cite{skerry2018towards,wang2018style,zhang2019learning}, have shown promising performance on the style transfer task." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1712.05884" id="3716">
        <attvalues>
          <attvalue for="5" value=" To address the above issues, we believe that Tacotron training should minimize frame level reconstruction loss~\cite{wang2017tacotron,shen2018natural} and utterance level perceptual loss at the same time." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="2002.00417" id="3717">
        <attvalues>
          <attvalue for="5" value=" For example, encoder-decoder architecture with attention mechanism, such as Tacotron~\cite{wang2017tacotron,shen2018natural,liu2020wavetts,lee2019robust}, has consistently achieved high voice quality." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1811.02122" id="3718">
        <attvalues>
          <attvalue for="5" value=" The key idea is to integrate the conventional TTS pipeline \cite{hunt1996unit,tokuda2002hmm} into an unified framework that learns sequence-to-sequence mapping from text to a sequence of acoustic features~\cite{lee2019robust,chung2019semi,He2019,Luong2019,liu2019teacher}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1808.10128" id="3719">
        <attvalues>
          <attvalue for="5" value=" The key idea is to integrate the conventional TTS pipeline \cite{hunt1996unit,tokuda2002hmm} into an unified framework that learns sequence-to-sequence mapping from text to a sequence of acoustic features~\cite{lee2019robust,chung2019semi,He2019,Luong2019,liu2019teacher}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1906.00672" id="3720">
        <attvalues>
          <attvalue for="5" value=" The key idea is to integrate the conventional TTS pipeline \cite{hunt1996unit,tokuda2002hmm} into an unified framework that learns sequence-to-sequence mapping from text to a sequence of acoustic features~\cite{lee2019robust,chung2019semi,He2019,Luong2019,liu2019teacher}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1904.00771" id="3721">
        <attvalues>
          <attvalue for="5" value=" The key idea is to integrate the conventional TTS pipeline \cite{hunt1996unit,tokuda2002hmm} into an unified framework that learns sequence-to-sequence mapping from text to a sequence of acoustic features~\cite{lee2019robust,chung2019semi,He2019,Luong2019,liu2019teacher}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1911.02839" id="3722">
        <attvalues>
          <attvalue for="5" value=" The key idea is to integrate the conventional TTS pipeline \cite{hunt1996unit,tokuda2002hmm} into an unified framework that learns sequence-to-sequence mapping from text to a sequence of acoustic features~\cite{lee2019robust,chung2019semi,He2019,Luong2019,liu2019teacher}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1906.04233" id="3723">
        <attvalues>
          <attvalue for="5" value=" As speech prosody is the result of the interplay of multiple speech properties, it is not easy to define speech prosody by a simple labeling scheme~\cite{Luong,lin2019investigation,hodariusing,zhao2020improved,hodari2020perception}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="2005.07884" id="3724">
        <attvalues>
          <attvalue for="5" value=" As speech prosody is the result of the interplay of multiple speech properties, it is not easy to define speech prosody by a simple labeling scheme~\cite{Luong,lin2019investigation,hodariusing,zhao2020improved,hodari2020perception}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="2003.06686" id="3725">
        <attvalues>
          <attvalue for="5" value=" As speech prosody is the result of the interplay of multiple speech properties, it is not easy to define speech prosody by a simple labeling scheme~\cite{Luong,lin2019investigation,hodariusing,zhao2020improved,hodari2020perception}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1808.01410" id="3726">
        <attvalues>
          <attvalue for="5" value=" While it is possible to automate the style token selection \cite{Stanton2018Predicting}, a correct prediction of style token is subject to both the design of the style token dictionary, and the run-time style token prediction algorithm." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1803.09047" id="3727">
        <attvalues>
          <attvalue for="5" value=" At run-time, the style token can be used to predict the speech style from text~\cite{Stanton2018Predicting}, or to transfer the speech style from a reference utterance to target~\cite{skerry2018towards}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1803.09017" id="3729">
        <attvalues>
          <attvalue for="5" value=" Third, the style token dictionary in Tacotron is trained from a collection of speech utterances to represent a large range of acoustic expressiveness for a speaker or an audiobook\cite{wang2018style}." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1603.08155" id="3730">
        <attvalues>
          <attvalue for="5" value=" Perceptual loss is first proposed for image stylization and synthesis \cite{ dosovitskiy2016generating,johnson2016perceptual,chen2017photographic,9052944}, where feature activation patterns, or deep features, derived from pre-trained auxiliary networks are used to optimize the perceptual quality of output image." />
        </attvalues>
      </edge>
      <edge source="2008.01490" target="1707.09405" id="3731">
        <attvalues>
          <attvalue for="5" value=" Perceptual loss is first proposed for image stylization and synthesis \cite{ dosovitskiy2016generating,johnson2016perceptual,chen2017photographic,9052944}, where feature activation patterns, or deep features, derived from pre-trained auxiliary networks are used to optimize the perceptual quality of output image." />
        </attvalues>
      </edge>
      <edge source="2109.06333" target="1802.05365" id="3732">
        <attvalues>
          <attvalue for="5" value=" Our approach also involves an artificial language fragment and a training procedure to introduce knowledge about some property $A$, but it uses a pre-trained language model (LM) \cite{Peters:2018,bert,brown2020language} as the learning agent." />
        </attvalues>
      </edge>
      <edge source="2109.06333" target="2005.14165" id="3734">
        <attvalues>
          <attvalue for="5" value=" Our approach also involves an artificial language fragment and a training procedure to introduce knowledge about some property $A$, but it uses a pre-trained language model (LM) \cite{Peters:2018,bert,brown2020language} as the learning agent." />
        </attvalues>
      </edge>
      <edge source="2109.06333" target="1909.12434" id="3735">
        <attvalues>
          <attvalue for="5" value=" Our approach belongs to the general area of studies using counterfactual linguistic data in NLP (\cite{kaushik,kaushik2021} a." />
        </attvalues>
      </edge>
      <edge source="2109.06333" target="2010.02114" id="3736">
        <attvalues>
          <attvalue for="5" value=" Our approach belongs to the general area of studies using counterfactual linguistic data in NLP (\cite{kaushik,kaushik2021} a." />
        </attvalues>
      </edge>
      <edge source="2109.06333" target="2011.02417" id="3737">
        <attvalues>
          <attvalue for="5" value="); a part of that subfield that uses novel lexicon is the closest to the present paper (\cite{levy,bylinina2022driving} a." />
        </attvalues>
      </edge>
      <edge source="2109.06333" target="1903.03260" id="3738">
        <attvalues>
          <attvalue for="5" value=" Our work contributes to the general agenda of establishing closer connections between learning in humans and artificial neural models (\cite{futrell-etal-2019-neural,wilcox-etal-2020-structural} a." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2302.00083" id="3740">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2308.12241" id="3741">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2307.13692" id="3742">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2110.15426" id="3743">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2302.06476" id="3744">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2305.13655" id="3745">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2305.11175" id="3746">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2308.00692" id="3747">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2304.09842" id="3748">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2307.03393" id="3749">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2308.02565" id="3751">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2402.08170" id="3752">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Large Language Models (LLMs) have been recently show-stealer, profoundly influencing not only the landscape of NLP \cite{ram2023context,liu2023llmrec,sawada2023arb,jaiswal2021radbert,qin2023chatgpt,zhuo2023large,Lee2023CanLL}, but also recently buttressing numerous computer vision \cite{lian2023llm,wang2023visionllm,lai2023lisa,lu2023chameleon,li2024cancergpt} and graph neural networks \cite{ye2023natural,chen2023exploring,qian2023can,duan2023simteg,chen2024llaga} algorithms; achieving steller performance across various task benchmarks." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2305.14152" id="3754">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2305.14314" id="3755">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2210.17323" id="3756">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2306.00978" id="3757">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2206.12755" id="3761">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2306.10460" id="3763">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2306.03805" id="3764">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2303.02141" id="3765">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2310.02277" id="3766">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2310.05175" id="3767">
        <attvalues>
          <attvalue for="5" value=" Among several model compression techniques such as quantization \cite{liu2023llm,Kim2023MemoryEfficientFO,Dettmers2023QLoRAEF,Frantar2022GPTQAP,Lin2023AWQAW,Dettmers2023SpQRAS}, and sparse neural networks \cite{frankle2018the,chen2020lottery,jaiswal2022training,lee2018snip,zhangheng2023sparse,jaiswal2023instant,jaiswal2023emergence,liu2023sparsity,Yin2023PruningSP,yin2023outlier} which require additional hardware support for speedup, token-level early exit or layer-skip has emerged as a promising technique to alleviate these limitations by allowing tokens to cease computation as soon as their hidden states reach saturation \cite{hash-skip-sun,del2023skipdecode,calm,men2024shortgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2307.02628" id="3768">
        <attvalues>
          <attvalue for="5" value=" Figure \ref{fig:ffn_skipllm_motivation} shows the comparison of the responses generated by two recent Layer Skipping methods, namely SkipDecode \cite{del2023skipdecode} and ShortGPT \cite{men2024shortgpt} for a knowledge-intensive QA example." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2403.03853" id="3769">
        <attvalues>
          <attvalue for="5" value=" Figure \ref{fig:ffn_skipllm_motivation} shows the comparison of the responses generated by two recent Layer Skipping methods, namely SkipDecode \cite{del2023skipdecode} and ShortGPT \cite{men2024shortgpt} for a knowledge-intensive QA example." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="1910.10073" id="3770">
        <attvalues>
          <attvalue for="5" value=" \begin{wraptable}{r}{5cm} \end{wraptable} For handling KV cache issue, some recent works \cite{elbayad2019depth,calm,li2021accelerating,chen2023ee,del2023skipdecode} proposes three main solutions: copying hidden states, pre-fixed token-level skip pattern, and KV recomputation." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2105.13878" id="3771">
        <attvalues>
          <attvalue for="5" value=" \begin{wraptable}{r}{5cm} \end{wraptable} For handling KV cache issue, some recent works \cite{elbayad2019depth,calm,li2021accelerating,chen2023ee,del2023skipdecode} proposes three main solutions: copying hidden states, pre-fixed token-level skip pattern, and KV recomputation." />
        </attvalues>
      </edge>
      <edge source="2404.03865" target="2312.04916" id="3772">
        <attvalues>
          <attvalue for="5" value=" \begin{wraptable}{r}{5cm} \end{wraptable} For handling KV cache issue, some recent works \cite{elbayad2019depth,calm,li2021accelerating,chen2023ee,del2023skipdecode} proposes three main solutions: copying hidden states, pre-fixed token-level skip pattern, and KV recomputation." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2005.14165" id="3774">
        <attvalues>
          <attvalue for="5" value=" To give a few examples: (i) the highly influential GPT-3 model displays hallmarks of few-shot learning/generalization ability from a small number of examples given in its prompt \cite{brown2020}; (ii) DALL-E, or other similar text-to-image models trained on large datasets such as CLIP+VQGAN, display qualitative evidence of compositional generalization abilities \cite{ramesh2021,crowson2022}; (iii) pretraining with large, diverse image or text datasets improves the out-of-distribution generalization performance of image recognition \cite{orhan2019,xie2020} and NLP models \cite{hendrycks2020}, respectively." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2102.12092" id="3775">
        <attvalues>
          <attvalue for="5" value=" To give a few examples: (i) the highly influential GPT-3 model displays hallmarks of few-shot learning/generalization ability from a small number of examples given in its prompt \cite{brown2020}; (ii) DALL-E, or other similar text-to-image models trained on large datasets such as CLIP+VQGAN, display qualitative evidence of compositional generalization abilities \cite{ramesh2021,crowson2022}; (iii) pretraining with large, diverse image or text datasets improves the out-of-distribution generalization performance of image recognition \cite{orhan2019,xie2020} and NLP models \cite{hendrycks2020}, respectively." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2204.08583" id="3776">
        <attvalues>
          <attvalue for="5" value=" To give a few examples: (i) the highly influential GPT-3 model displays hallmarks of few-shot learning/generalization ability from a small number of examples given in its prompt \cite{brown2020}; (ii) DALL-E, or other similar text-to-image models trained on large datasets such as CLIP+VQGAN, display qualitative evidence of compositional generalization abilities \cite{ramesh2021,crowson2022}; (iii) pretraining with large, diverse image or text datasets improves the out-of-distribution generalization performance of image recognition \cite{orhan2019,xie2020} and NLP models \cite{hendrycks2020}, respectively." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="1907.07640" id="3777">
        <attvalues>
          <attvalue for="5" value=" To give a few examples: (i) the highly influential GPT-3 model displays hallmarks of few-shot learning/generalization ability from a small number of examples given in its prompt \cite{brown2020}; (ii) DALL-E, or other similar text-to-image models trained on large datasets such as CLIP+VQGAN, display qualitative evidence of compositional generalization abilities \cite{ramesh2021,crowson2022}; (iii) pretraining with large, diverse image or text datasets improves the out-of-distribution generalization performance of image recognition \cite{orhan2019,xie2020} and NLP models \cite{hendrycks2020}, respectively." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="1911.04252" id="3778">
        <attvalues>
          <attvalue for="5" value=" To give a few examples: (i) the highly influential GPT-3 model displays hallmarks of few-shot learning/generalization ability from a small number of examples given in its prompt \cite{brown2020}; (ii) DALL-E, or other similar text-to-image models trained on large datasets such as CLIP+VQGAN, display qualitative evidence of compositional generalization abilities \cite{ramesh2021,crowson2022}; (iii) pretraining with large, diverse image or text datasets improves the out-of-distribution generalization performance of image recognition \cite{orhan2019,xie2020} and NLP models \cite{hendrycks2020}, respectively." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2010.05465" id="3781">
        <attvalues>
          <attvalue for="5" value=" What exactly are the limits to the generalization benefits of large-scale pretraining? More concretely, how do pretraining benefits scale with factors such as the diversity and the scale of the pretraining data, the model size, or the similarity between the pretraining data and the downstream task? Here, we report the results of some simple experiments aimed at addressing the latter two factors, namely the model size and the similarity between the pretraining data and the downstream task, in the context of two previously introduced semantic parsing tasks, SCAN \cite{lake2018} and COGS \cite{kim2020}." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2007.08970" id="3782">
        <attvalues>
          <attvalue for="5" value=" This result is essentially a replication of earlier reports to the same effect: eg \cite{furrer2020} for SCAN and \cite{tay2021} for COGS." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2105.03322" id="3783">
        <attvalues>
          <attvalue for="5" value=" This result is essentially a replication of earlier reports to the same effect: eg \cite{furrer2020} for SCAN and \cite{tay2021} for COGS." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2004.14601" id="3786">
        <attvalues>
          <attvalue for="5" value=" \cite{papadimitriou2020} similarly report downstream natural language modeling tasks can benefit from pretraining in seemingly unrelated domains such Java code or MIDI music scores." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2012.11995" id="3787">
        <attvalues>
          <attvalue for="5" value=" Consistent with this idea, \cite{chiang2020} also show that pretraining with a simple artificial language generated by a stack-based hierarchical grammar can improve the accuracy on a diverse set of downstream natural language tasks, namely the tasks comprising the GLUE benchmark \cite{wang2018}." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2109.04953" id="3789">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{krishna2021} show that text summarization also does not seem to require pretraining with natural language texts: they demonstrate that even pretraining with texts consisting entirely of randomly and independently sampled nonsense words achieves similar results in downstream text summarization tasks." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2106.05963" id="3790">
        <attvalues>
          <attvalue for="5" value=" In the visual domain, \cite{baradad2021} show that even very simple random noise processes can be used as effective pretraining data for downstream natural image recognition tasks, as long as these noise processes satisfy certain basic structural properties of natural images." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2104.06644" id="3791">
        <attvalues>
          <attvalue for="5" value=" In a similar vein, \cite{sinha2021} recently showed that scrambling the (within-sentence) word order in natural language texts has surprisingly little effect on their effectiveness as pretraining data, as long as some higher-order co-occurrence statistics are preserved." />
        </attvalues>
      </edge>
      <edge source="2109.15101" target="2007.06225" id="3793">
        <attvalues>
          <attvalue for="5" value=" \item However, the same is not true for pretraining with a large-scale protein sequence prediction task \cite{elnaggar2020}." />
        </attvalues>
      </edge>
      <edge source="2004.14601" target="1805.01070" id="3794">
        <attvalues>
          <attvalue for="5" value=" Recent work has directly probed the internal activations of models \cite{conneaufing, grain, johnprobe, urvashibert}, or fed them curated inputs that depend on complex syntax \cite{linzen16, gulordava, olmpics, mccoy2020}, in order to uncover latent syntactic awareness." />
        </attvalues>
      </edge>
      <edge source="2004.14601" target="1611.01368" id="3795">
        <attvalues>
          <attvalue for="5" value=" Recent work has directly probed the internal activations of models \cite{conneaufing, grain, johnprobe, urvashibert}, or fed them curated inputs that depend on complex syntax \cite{linzen16, gulordava, olmpics, mccoy2020}, in order to uncover latent syntactic awareness." />
        </attvalues>
      </edge>
      <edge source="2004.14601" target="2108.03334" id="3797">
        <attvalues>
          <attvalue for="5" value=" For this result we draw on recent interlingual work such as \cite{artetxezero}, \cite{edo-zero}, and \cite{xnli}, extending it to use typological distance to turn these observations into quantitative probes." />
        </attvalues>
      </edge>
      <edge source="2012.11995" target="1908.08593" id="3799">
        <attvalues>
          <attvalue for="5" value=" Probing a trained model is widely used to understand to what extent a model learns certain linguistic features~\cite{kovaleva2019revealing, hewitt2019structural, tenney2019bert, tenney2018you, lin2019open}." />
        </attvalues>
      </edge>
      <edge source="2012.11995" target="2004.14601" id="3801">
        <attvalues>
          <attvalue for="5" value=" Recently,~\cite{papadimitriou2020learning} proposed to train an LSTM LM on a non-natural language dataset and test the LM's perplexity on natural language." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2304.13712" id="3802">
        <attvalues>
          <attvalue for="5" value=" Numerous large language models (LLMs) with remarkable natural language understanding and reasoning capabilities have been released in recent months \cite{yang_harnessing_2023,zhao_survey_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2303.18223" id="3803">
        <attvalues>
          <attvalue for="5" value=" Numerous large language models (LLMs) with remarkable natural language understanding and reasoning capabilities have been released in recent months \cite{yang_harnessing_2023,zhao_survey_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2305.06329" id="3804">
        <attvalues>
          <attvalue for="5" value=" Methods for these perspectives have been proposed in prior work, but often focus on non-sequence models \cite{klabunde_similarity_2023} or do not scale to the size of LLMs \cite{shah_modeldiff_2022}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2005.01172" id="3806">
        <attvalues>
          <attvalue for="5" value=" \cite{wu_similarity_2020,abnar_blackbox_2019} explicitly compare representations between models." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="1903.08855" id="3808">
        <attvalues>
          <attvalue for="5" value="\ \ Several works study representations of language models and make implicit comparisons: how contextual they are \cite{ethayarajh_how_2019}, what interpretable concepts can be decoded from them \cite{liu_linguistic_2019}, or how models can communicate via representations \cite{moschella_relative_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2209.15430" id="3809">
        <attvalues>
          <attvalue for="5" value="\ \ Several works study representations of language models and make implicit comparisons: how contextual they are \cite{ethayarajh_how_2019}, what interpretable concepts can be decoded from them \cite{liu_linguistic_2019}, or how models can communicate via representations \cite{moschella_relative_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="1802.05365" id="3811">
        <attvalues>
          <attvalue for="5" value=" However, these works have in common that they do not study the current generation of LLMs, and instead focus on smaller models with different architectures like BERT \cite{devlin_bert_2019} or ELMo \cite{peters_deep_2018}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="1911.02969" id="3812">
        <attvalues>
          <attvalue for="5" value=" Similarity of these models was also studied from a functional perspective \cite{mccoy_berts_2020}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2305.01610" id="3813">
        <attvalues>
          <attvalue for="5" value=" As an exception, \cite{gurnee_finding_2023} probe the recent Pythia models \cite{biderman_pythia_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2304.01373" id="3814">
        <attvalues>
          <attvalue for="5" value=" As an exception, \cite{gurnee_finding_2023} probe the recent Pythia models \cite{biderman_pythia_2023}." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2310.14993" id="3815">
        <attvalues>
          <attvalue for="5" value=" Concurrent work \cite{yousefi_-context_2023,brown2023understanding} studies representations of modern LLMs." />
        </attvalues>
      </edge>
      <edge source="2312.02730" target="2206.04615" id="3816">
        <attvalues>
          <attvalue for="5" value=" Performance of LLMs is compared in many benchmarks \cite[eg,][]{srivastava_capabilities_of_language_models_2022}." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="2005.14165" id="3817">
        <attvalues>
          <attvalue for="5" value=" Recent studies on Large Language Models (LLMs), such as GPT-3~\cite{brown2020language}, InstructGPT~\cite{ouyang2022training}, PaLM~\cite{chowdhery2022palm}, and OPT~\cite{zhang2022opt}, have exhibited impressive zero-shot performance." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="2204.02311" id="3819">
        <attvalues>
          <attvalue for="5" value=" Recent studies on Large Language Models (LLMs), such as GPT-3~\cite{brown2020language}, InstructGPT~\cite{ouyang2022training}, PaLM~\cite{chowdhery2022palm}, and OPT~\cite{zhang2022opt}, have exhibited impressive zero-shot performance." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="2205.01068" id="3820">
        <attvalues>
          <attvalue for="5" value=" Recent studies on Large Language Models (LLMs), such as GPT-3~\cite{brown2020language}, InstructGPT~\cite{ouyang2022training}, PaLM~\cite{chowdhery2022palm}, and OPT~\cite{zhang2022opt}, have exhibited impressive zero-shot performance." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="2302.06476" id="3821">
        <attvalues>
          <attvalue for="5" value=" Specifically, ChatGPT has shown competitive performance on zero-shot logical reasoning~\cite{qin2023chatgpt}, text summarization~\cite{yang2023exploring}, machine translation~\cite{jiao2023chatgpt}, information extraction~\cite{wei2023zero} and so on." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="2302.08081" id="3822">
        <attvalues>
          <attvalue for="5" value=" Specifically, ChatGPT has shown competitive performance on zero-shot logical reasoning~\cite{qin2023chatgpt}, text summarization~\cite{yang2023exploring}, machine translation~\cite{jiao2023chatgpt}, information extraction~\cite{wei2023zero} and so on." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="2301.08745" id="3823">
        <attvalues>
          <attvalue for="5" value=" Specifically, ChatGPT has shown competitive performance on zero-shot logical reasoning~\cite{qin2023chatgpt}, text summarization~\cite{yang2023exploring}, machine translation~\cite{jiao2023chatgpt}, information extraction~\cite{wei2023zero} and so on." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="1805.10190" id="3824">
        <attvalues>
          <attvalue for="5" value=" We conduct experiments on four widely used benchmarks including ATIS~\cite{hemphill-etal-1990-atis}, SNIPS~\cite{couke2018snips} in SLU and MultiWOZ2." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="1907.01669" id="3825">
        <attvalues>
          <attvalue for="5" value="1~\cite{eric2019multiwoz}, MultiWOZ2." />
        </attvalues>
      </edge>
      <edge source="2304.04256" target="2104.00773" id="3826">
        <attvalues>
          <attvalue for="5" value="4~\cite{Ye2021MultiWOZ2A} in DST." />
        </attvalues>
      </edge>
      <edge source="2301.08745" target="1611.04558" id="3828">
        <attvalues>
          <attvalue for="5" value=" For example, how to mention the source or target language information matters in multilingual machine translation models, which is usually solved by attaching language tokens~\cite{Johnson:2017:TACL,fan2021beyond}." />
        </attvalues>
      </edge>
      <edge source="2301.08745" target="2010.11125" id="3829">
        <attvalues>
          <attvalue for="5" value=" For example, how to mention the source or target language information matters in multilingual machine translation models, which is usually solved by attaching language tokens~\cite{Johnson:2017:TACL,fan2021beyond}." />
        </attvalues>
      </edge>
      <edge source="2301.08745" target="2303.08774" id="3830">
        <attvalues>
          <attvalue for="5" value=" On the other hand, with an improved engine GPT-4~\cite{openai2023gpt4} launched on March 15, 2023, we re-evaluate the translation ability of ChatGPT and observe a significant boost of performance." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="1803.06535" id="3831">
        <attvalues>
          <attvalue for="5" value=" If parallel data are provided, standard sequence-to-sequence models are often directly applied~\cite{rao-tetreault-2018-dear} (see Section~\ref{sec:sup})." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="1703.00955" id="3832">
        <attvalues>
          <attvalue for="5" value=" The first line of approaches disentangle text into its content and attribute in the latent space, and apply generative modeling \cite{Hu2017TowardCG,shen2017style}." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="1804.06437" id="3834">
        <attvalues>
          <attvalue for="5" value=" This trend was then joined by another distinctive line of approach, prototype editing \cite{li-etal-2018-delete} which extracts a sentence template and its attribute markers to generate the text." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="1808.07894" id="3835">
        <attvalues>
          <attvalue for="5" value=" Another paradigm soon followed, eg, pseudo-parallel corpus construction to train the model as if in a supervised way with the pseudo-parallel data \cite{zhang2018style,jin2019imat}." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="1901.11333" id="3836">
        <attvalues>
          <attvalue for="5" value=" Another paradigm soon followed, eg, pseudo-parallel corpus construction to train the model as if in a supervised way with the pseudo-parallel data \cite{zhang2018style,jin2019imat}." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="1805.03162" id="3838">
        <attvalues>
          <attvalue for="5" value=" Given the advances in TST methodologies, it now starts to expand its impact to downstream applications, such as persona-based dialog generation \cite{niu-bansal-2018-polite,huang-etal-2018-automatic}, stylistic summarization \cite{jin-etal-2020-hooks}, stylized language modeling to imitate specific authors \cite{Syed2020AdaptingLM}, online text debiasing \cite{pryzant2020automatically,ma2020powertransformer}, simile generation \cite{chakrabarty2020generating}, and many others." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="2004.01980" id="3839">
        <attvalues>
          <attvalue for="5" value=" Given the advances in TST methodologies, it now starts to expand its impact to downstream applications, such as persona-based dialog generation \cite{niu-bansal-2018-polite,huang-etal-2018-automatic}, stylistic summarization \cite{jin-etal-2020-hooks}, stylized language modeling to imitate specific authors \cite{Syed2020AdaptingLM}, online text debiasing \cite{pryzant2020automatically,ma2020powertransformer}, simile generation \cite{chakrabarty2020generating}, and many others." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="1909.09962" id="3840">
        <attvalues>
          <attvalue for="5" value=" Given the advances in TST methodologies, it now starts to expand its impact to downstream applications, such as persona-based dialog generation \cite{niu-bansal-2018-polite,huang-etal-2018-automatic}, stylistic summarization \cite{jin-etal-2020-hooks}, stylized language modeling to imitate specific authors \cite{Syed2020AdaptingLM}, online text debiasing \cite{pryzant2020automatically,ma2020powertransformer}, simile generation \cite{chakrabarty2020generating}, and many others." />
        </attvalues>
      </edge>
      <edge source="2011.00416" target="2009.08942" id="3843">
        <attvalues>
          <attvalue for="5" value=" Given the advances in TST methodologies, it now starts to expand its impact to downstream applications, such as persona-based dialog generation \cite{niu-bansal-2018-polite,huang-etal-2018-automatic}, stylistic summarization \cite{jin-etal-2020-hooks}, stylized language modeling to imitate specific authors \cite{Syed2020AdaptingLM}, online text debiasing \cite{pryzant2020automatically,ma2020powertransformer}, simile generation \cite{chakrabarty2020generating}, and many others." />
        </attvalues>
      </edge>
      <edge source="2405.01299" target="2005.14165" id="3852">
        <attvalues>
          <attvalue for="5" value=" \cite{brown2020language} demonstrate their ability as few-shot learners and \cite{flan_2021,kojima2022large} evidence their zero-shot capabilities." />
        </attvalues>
      </edge>
      <edge source="2405.01299" target="2109.01652" id="3853">
        <attvalues>
          <attvalue for="5" value=" \cite{brown2020language} demonstrate their ability as few-shot learners and \cite{flan_2021,kojima2022large} evidence their zero-shot capabilities." />
        </attvalues>
      </edge>
      <edge source="2405.01299" target="2305.13788" id="3855">
        <attvalues>
          <attvalue for="5" value=" Considering that LLMs are trained to adhere to instructions guided by human preference \cite{ouyang2022training, rafailov2023direct}, studies examine the extent to which human disagreement is captured \cite{lee2023can} and whether or not such disagreement aligns with that of humans \cite{santurkar2023whose}." />
        </attvalues>
      </edge>
      <edge source="2405.01299" target="2303.17548" id="3857">
        <attvalues>
          <attvalue for="5" value=" Considering that LLMs are trained to adhere to instructions guided by human preference \cite{ouyang2022training, rafailov2023direct}, studies examine the extent to which human disagreement is captured \cite{lee2023can} and whether or not such disagreement aligns with that of humans \cite{santurkar2023whose}." />
        </attvalues>
      </edge>
      <edge source="2405.01299" target="2304.14803" id="3858">
        <attvalues>
          <attvalue for="5" value=" We compare the top-performing LLM from the first section(GPT) against human annotators, by examining the degree of alignment between their opinion distributions, for the case of the four subjective datasets recently used for the 2023 SEMEVAL Task on Learning With Disagreement \cite{leonardelli_2023_semeval}." />
        </attvalues>
      </edge>
      <edge source="2304.06939" target="2005.14165" id="3859">
        <attvalues>
          <attvalue for="5" value=" In-context learning \cite{brown2020language} enables sequence models to adapt to new tasks without any parameter updates." />
        </attvalues>
      </edge>
      <edge source="2304.06939" target="2204.14198" id="3860">
        <attvalues>
          <attvalue for="5" value=" We conclude by discussing initial use-cases of mmc4, including OpenFlamingo \cite{awadalla2023openflamingo}, an open source version of Flamingo \cite{alayrac2022flamingo}." />
        </attvalues>
      </edge>
      <edge source="2304.06939" target="2308.01390" id="3862">
        <attvalues>
          <attvalue for="5" value=" We conclude by discussing initial use-cases of mmc4, including OpenFlamingo \cite{awadalla2023openflamingo}, an open source version of Flamingo \cite{alayrac2022flamingo}." />
        </attvalues>
      </edge>
      <edge source="2002.02000" target="1706.03762" id="3863">
        <attvalues>
          <attvalue for="5" value=" The current state-of-the-art language models use large Transformer architectures \cite{Vaswani2017} containing hundreds of millions of parameters and are pretrained using multi-billion word corpuses." />
        </attvalues>
      </edge>
      <edge source="2002.02000" target="1802.05365" id="3865">
        <attvalues>
          <attvalue for="5" value=" They outperformed the RNN-based models such as ELMo \cite{Peters2018} by a significant margin in benchmarks like GLUE \cite{Wang2019} and SQuAD \cite{Rajpurkar2016}." />
        </attvalues>
      </edge>
      <edge source="2002.02000" target="1606.05250" id="3867">
        <attvalues>
          <attvalue for="5" value=" They outperformed the RNN-based models such as ELMo \cite{Peters2018} by a significant margin in benchmarks like GLUE \cite{Wang2019} and SQuAD \cite{Rajpurkar2016}." />
        </attvalues>
      </edge>
      <edge source="2002.02000" target="1907.11692" id="3868">
        <attvalues>
          <attvalue for="5" value=" Since the release of GPT and BERT, many researchers have further improved the Transformer-based language models demonstrated by surpassing their predecessors in the evaluations of the common benchmarks \cite{Liu2019, Dong2019, Conneau2019, Conneau20192, Raffel2019, Yang2019, Sun2020}." />
        </attvalues>
      </edge>
      <edge source="2002.02000" target="1909.10351" id="3869">
        <attvalues>
          <attvalue for="5" value=" To speed up language models, people have developed libraries for fast neural network computation \cite{Zhang2018, Junczysdowmunt2018} and built smaller models with sufficient prediction power using knowledge distillation \cite{Jiao2019} or improved Transformer architectures \cite{Lan2020}." />
        </attvalues>
      </edge>
      <edge source="2002.02000" target="1909.11942" id="3870">
        <attvalues>
          <attvalue for="5" value=" To speed up language models, people have developed libraries for fast neural network computation \cite{Zhang2018, Junczysdowmunt2018} and built smaller models with sufficient prediction power using knowledge distillation \cite{Jiao2019} or improved Transformer architectures \cite{Lan2020}." />
        </attvalues>
      </edge>
      <edge source="2408.11381" target="1704.00051" id="3871">
        <attvalues>
          <attvalue for="5" value=" Retrieval augmentation generation(RAG) leverages external knowledge to mitigate hallucination issues, ensure real-time knowledge updates, and protect private data with no parametric knowledge\cite{naive-rag-2017-qa-baseon-wiki,naiverag-2020-1,naiverag-2-Realm}." />
        </attvalues>
      </edge>
      <edge source="2408.11381" target="2005.11401" id="3872">
        <attvalues>
          <attvalue for="5" value=" Retrieval augmentation generation(RAG) leverages external knowledge to mitigate hallucination issues, ensure real-time knowledge updates, and protect private data with no parametric knowledge\cite{naive-rag-2017-qa-baseon-wiki,naiverag-2020-1,naiverag-2-Realm}." />
        </attvalues>
      </edge>
      <edge source="2408.11381" target="2305.15294" id="3873">
        <attvalues>
          <attvalue for="5" value=" On the other hand, a multitude of novel RAG algorithms have merged, including ITER-RETGEN\cite{iter-retgen}, RRR\cite{rrr-rag}, Self-Ask\cite{self-ask}, Active RAG\cite{active-rag}, Self-RAG\cite{selfrag}, etc However, these RAG algorithms are not well aligned in their fundamental components and evaluation methodologies, making it difficult for researchers to accurately assess their improvements." />
        </attvalues>
      </edge>
      <edge source="2408.11381" target="2210.03350" id="3874">
        <attvalues>
          <attvalue for="5" value=" On the other hand, a multitude of novel RAG algorithms have merged, including ITER-RETGEN\cite{iter-retgen}, RRR\cite{rrr-rag}, Self-Ask\cite{self-ask}, Active RAG\cite{active-rag}, Self-RAG\cite{selfrag}, etc However, these RAG algorithms are not well aligned in their fundamental components and evaluation methodologies, making it difficult for researchers to accurately assess their improvements." />
        </attvalues>
      </edge>
      <edge source="2408.11381" target="2305.06983" id="3875">
        <attvalues>
          <attvalue for="5" value=" On the other hand, a multitude of novel RAG algorithms have merged, including ITER-RETGEN\cite{iter-retgen}, RRR\cite{rrr-rag}, Self-Ask\cite{self-ask}, Active RAG\cite{active-rag}, Self-RAG\cite{selfrag}, etc However, these RAG algorithms are not well aligned in their fundamental components and evaluation methodologies, making it difficult for researchers to accurately assess their improvements." />
        </attvalues>
      </edge>
      <edge source="2408.11381" target="2310.11511" id="3876">
        <attvalues>
          <attvalue for="5" value=" On the other hand, a multitude of novel RAG algorithms have merged, including ITER-RETGEN\cite{iter-retgen}, RRR\cite{rrr-rag}, Self-Ask\cite{self-ask}, Active RAG\cite{active-rag}, Self-RAG\cite{selfrag}, etc However, these RAG algorithms are not well aligned in their fundamental components and evaluation methodologies, making it difficult for researchers to accurately assess their improvements." />
        </attvalues>
      </edge>
      <edge source="2408.11381" target="2308.10633" id="3877">
        <attvalues>
          <attvalue for="5" value=" While various current works are investigating these questions, such as LlamaIndex \cite{LlamaIndex}, LangChain\cite{LangChain2022}, Haystack\cite{haystack}, FastRAG\cite{fastRAG2023}, RALLE \cite{ralle-framework}, LocalRQA\cite{localrqa}, AutoRAG\cite{AutoRAG2024}, and FlashRAG\cite{FlashRAG}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2005.14165" id="3880">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have shown great potential for natural language understanding and generation tasks \cite{gpt3,palm,gpt4}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2204.02311" id="3881">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have shown great potential for natural language understanding and generation tasks \cite{gpt3,palm,gpt4}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2303.08774" id="3882">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have shown great potential for natural language understanding and generation tasks \cite{gpt3,palm,gpt4}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2302.04023" id="3883">
        <attvalues>
          <attvalue for="5" value=" However, they face challenges when answering factual questions due to hallucinations (or confabulations) \cite{hallucination1,hallucination2}, outdated parametric knowledge \cite{streamingqa}, and memory efficiency of parametric knowledge \cite[eg,][]{LMasKB2}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2302.03494" id="3884">
        <attvalues>
          <attvalue for="5" value=" However, they face challenges when answering factual questions due to hallucinations (or confabulations) \cite{hallucination1,hallucination2}, outdated parametric knowledge \cite{streamingqa}, and memory efficiency of parametric knowledge \cite[eg,][]{LMasKB2}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2205.11388" id="3885">
        <attvalues>
          <attvalue for="5" value=" However, they face challenges when answering factual questions due to hallucinations (or confabulations) \cite{hallucination1,hallucination2}, outdated parametric knowledge \cite{streamingqa}, and memory efficiency of parametric knowledge \cite[eg,][]{LMasKB2}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2008.09036" id="3886">
        <attvalues>
          <attvalue for="5" value=" However, they face challenges when answering factual questions due to hallucinations (or confabulations) \cite{hallucination1,hallucination2}, outdated parametric knowledge \cite{streamingqa}, and memory efficiency of parametric knowledge \cite[eg,][]{LMasKB2}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="1704.00051" id="3887">
        <attvalues>
          <attvalue for="5" value=" To address these limitations, researchers have turned to the retrieval-augmented approach used in open-domain question answering (QA) \cite{odqa}, hereinafter referred to as retrieval-augmented LLMs or R-LLMs." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2302.07842" id="3888">
        <attvalues>
          <attvalue for="5" value=" In comparison to closed-book settings where language models generate answers without retrieval, R-LLMs (open-book settings) enable the retrieval of relevant information from external databases or corpora \cite{survey,simplyretrieve}, which has led to improved accuracy in open-domain QA \cite{replug}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2308.03983" id="3889">
        <attvalues>
          <attvalue for="5" value=" Additionally, R-LLMs can acquire extended features even without additional training, such as explicit references, relief from fact hallucination \cite{webgpt}, and easy updates to the knowledge source \cite[eg,][]{realm,simplyretrieve}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2301.12652" id="3890">
        <attvalues>
          <attvalue for="5" value=" In comparison to closed-book settings where language models generate answers without retrieval, R-LLMs (open-book settings) enable the retrieval of relevant information from external databases or corpora \cite{survey,simplyretrieve}, which has led to improved accuracy in open-domain QA \cite{replug}." />
        </attvalues>
      </edge>
      <edge source="2308.10633" target="2010.11915" id="3892">
        <attvalues>
          <attvalue for="5" value="7 \cite{unanswerable}." />
        </attvalues>
      </edge>
      <edge source="2209.08284" target="2104.06378" id="3893">
        <attvalues>
          <attvalue for="5" value=" QA-GNN~\cite{Yasunaga2021QAGNNRW} and GreaseLM~\cite{greaseLM}, the most recent work (as far as we know), retrieve a subgraph using entity matching and path finding, then jointly reason over both modalities by either adding question context to the subgraph as an additional node or introducing additional interaction tokens and layers into the model architecture." />
        </attvalues>
      </edge>
      <edge source="2209.08284" target="2201.08860" id="3894">
        <attvalues>
          <attvalue for="5" value=" QA-GNN~\cite{Yasunaga2021QAGNNRW} and GreaseLM~\cite{greaseLM}, the most recent work (as far as we know), retrieve a subgraph using entity matching and path finding, then jointly reason over both modalities by either adding question context to the subgraph as an additional node or introducing additional interaction tokens and layers into the model architecture." />
        </attvalues>
      </edge>
      <edge source="2304.07772" target="2106.09997" id="3903">
        <attvalues>
          <attvalue for="5" value=" The recent development of pretrained language models and their application to SPARQL query generation has opened new potential avenues \cite{lin2022sparql} \cite{reyd2023comprehensive} \cite{tran2021spbert} \cite{huang2021unseen} \cite{huang2021unseen} \cite{naik2023sql}." />
        </attvalues>
      </edge>
      <edge source="2304.07772" target="2204.12793" id="3904">
        <attvalues>
          <attvalue for="5" value=" Initial experiments on pretrained language models for SPARQL query generation indicate that while they may outperform their non-pretrained counterparts \cite{banerjee2022modern}, they still exhibit some limitations in handling unknown URIs to some degree, but most importantly, they exhibit poor generalization abilities as their performance drops when new question templates are used at test time \cite{reyd2023comprehensive}." />
        </attvalues>
      </edge>
      <edge source="2304.07772" target="1910.13461" id="3905">
        <attvalues>
          <attvalue for="5" value=" In this context, we address the following research questions : \begin{enumerate} \item Does the annotation of KB elements in the natural language questions improve the SPARQL query generation for all models? \item Does the integration of a copy mechanism in NPLMs and PLMs improve the accuracy of the KB elements in the SPARQL queries? \item Are Large Language Models (LLMs) effective in this task, and which fine-tuning/prompting technique performs better? \item What are the most common generation errors, and what types of tokens are often generated instead of the expected types? \item How do models trained on template-based questions perform on naturally reformulated questions? \end{enumerate} Our contributions include the following aspects: \begin{enumerate} \item{Using annotation, we compare the results of two NPLMs (ConvSeq2Seq and Transformers), two PLMs (BART \cite{BART} and T5 \cite{T5}) and two LLMs (Llama2 \cite{touvron2023llama} and Code Llamav2 7B \cite{roziere2023code}; } \item{We evaluate the impact of the copy mechanism and question annotations and experiment with &quot;raw-question&quot; (non-annotated) questions, &quot;tag-within&quot; questions where we replace natural language elements with their KBs URIs counterparts and tag-end questions, where we list KB URIs with their labels at the end of the questions;} \item{We experiment with standard fine-tuning and instruction fine-tuning on two Large Language Models (LLMs), namely Llama\cite{touvron2023llama} and Code Llama\cite{roziere2023code} and measure the impact of training data size on the results;} \item{We perform a fine-grained analysis of the generation errors with their type distribution for all the models;} \item{We test the generalization capabilities of the best-performing models using questions reformulated in different settings." />
        </attvalues>
      </edge>
      <edge source="2304.07772" target="2308.12950" id="3908">
        <attvalues>
          <attvalue for="5" value=" In this context, we address the following research questions : \begin{enumerate} \item Does the annotation of KB elements in the natural language questions improve the SPARQL query generation for all models? \item Does the integration of a copy mechanism in NPLMs and PLMs improve the accuracy of the KB elements in the SPARQL queries? \item Are Large Language Models (LLMs) effective in this task, and which fine-tuning/prompting technique performs better? \item What are the most common generation errors, and what types of tokens are often generated instead of the expected types? \item How do models trained on template-based questions perform on naturally reformulated questions? \end{enumerate} Our contributions include the following aspects: \begin{enumerate} \item{Using annotation, we compare the results of two NPLMs (ConvSeq2Seq and Transformers), two PLMs (BART \cite{BART} and T5 \cite{T5}) and two LLMs (Llama2 \cite{touvron2023llama} and Code Llamav2 7B \cite{roziere2023code}; } \item{We evaluate the impact of the copy mechanism and question annotations and experiment with &quot;raw-question&quot; (non-annotated) questions, &quot;tag-within&quot; questions where we replace natural language elements with their KBs URIs counterparts and tag-end questions, where we list KB URIs with their labels at the end of the questions;} \item{We experiment with standard fine-tuning and instruction fine-tuning on two Large Language Models (LLMs), namely Llama\cite{touvron2023llama} and Code Llama\cite{roziere2023code} and measure the impact of training data size on the results;} \item{We perform a fine-grained analysis of the generation errors with their type distribution for all the models;} \item{We test the generalization capabilities of the best-performing models using questions reformulated in different settings." />
        </attvalues>
      </edge>
      <edge source="1909.08744" target="1611.01734" id="3909">
        <attvalues>
          <attvalue for="5" value=" Dependency parsing has achieved new states of the art using distributed word representations in neural networks, trained with large amounts of annotated data \cite{dozatmanning2017,dozat-qi-manning:2017:K17-3,ma-EtAl:2018:Long2,Che2018ElmoUD}." />
        </attvalues>
      </edge>
      <edge source="1909.08744" target="1805.01087" id="3910">
        <attvalues>
          <attvalue for="5" value=" Dependency parsing has achieved new states of the art using distributed word representations in neural networks, trained with large amounts of annotated data \cite{dozatmanning2017,dozat-qi-manning:2017:K17-3,ma-EtAl:2018:Long2,Che2018ElmoUD}." />
        </attvalues>
      </edge>
      <edge source="1909.08744" target="1902.09697" id="3912">
        <attvalues>
          <attvalue for="5" value=" Recent work has extended contextual word representations (cwrs) multilingually either by training a polyglot language model (LM) on a mixture of data from multiple languages (joint training approach; \cite{mulcaire_NAACL2019}; \cite{LampleConneau2019}) or by aligning multiple monolingual language models crosslingually (retrofitting approach; \cite{Schuster2019CrossLingual,aldarmaki_diab2019})." />
        </attvalues>
      </edge>
      <edge source="1909.08744" target="1901.07291" id="3913">
        <attvalues>
          <attvalue for="5" value=" Recent work has extended contextual word representations (cwrs) multilingually either by training a polyglot language model (LM) on a mixture of data from multiple languages (joint training approach; \cite{mulcaire_NAACL2019}; \cite{LampleConneau2019}) or by aligning multiple monolingual language models crosslingually (retrofitting approach; \cite{Schuster2019CrossLingual,aldarmaki_diab2019})." />
        </attvalues>
      </edge>
      <edge source="1909.08744" target="1902.09492" id="3914">
        <attvalues>
          <attvalue for="5" value=" Recent work has extended contextual word representations (cwrs) multilingually either by training a polyglot language model (LM) on a mixture of data from multiple languages (joint training approach; \cite{mulcaire_NAACL2019}; \cite{LampleConneau2019}) or by aligning multiple monolingual language models crosslingually (retrofitting approach; \cite{Schuster2019CrossLingual,aldarmaki_diab2019})." />
        </attvalues>
      </edge>
      <edge source="1909.08744" target="1903.03243" id="3915">
        <attvalues>
          <attvalue for="5" value=" Recent work has extended contextual word representations (cwrs) multilingually either by training a polyglot language model (LM) on a mixture of data from multiple languages (joint training approach; \cite{mulcaire_NAACL2019}; \cite{LampleConneau2019}) or by aligning multiple monolingual language models crosslingually (retrofitting approach; \cite{Schuster2019CrossLingual,aldarmaki_diab2019})." />
        </attvalues>
      </edge>
      <edge source="2211.03152" target="1703.10931" id="3917">
        <attvalues>
          <attvalue for="5" value=" This has lead to a variety of methods that follow the seq2seq architecture to build models that estimate $p(y|x)$, the probability of producing a simple sentence $y$ given the complex sentence $x$ \cite{nisioi2017exploring,zhang-lapata-2017-sentence,martin2020multilingual,Lin_Wan_2021,Omelianchuk}." />
        </attvalues>
      </edge>
      <edge source="2211.03152" target="2103.05070" id="3918">
        <attvalues>
          <attvalue for="5" value=" This has lead to a variety of methods that follow the seq2seq architecture to build models that estimate $p(y|x)$, the probability of producing a simple sentence $y$ given the complex sentence $x$ \cite{nisioi2017exploring,zhang-lapata-2017-sentence,martin2020multilingual,Lin_Wan_2021,Omelianchuk}." />
        </attvalues>
      </edge>
      <edge source="2211.03152" target="1910.02677" id="3919">
        <attvalues>
          <attvalue for="5" value=" Previous works have tried to control the generation of simple sentences by using control codes that condition the system output with variables such as length, number paraphrases, lexical complexity, and syntactic complexity, etc \cite{martin-etal-2020-controllable,sheang-saggion-2021-controllable}." />
        </attvalues>
      </edge>
      <edge source="2211.03152" target="2110.08329" id="3920">
        <attvalues>
          <attvalue for="5" value=" Other works such as \cite{Clive-2021} use control prefixes, including input-dependent conditional information pre-trained models, incorporating learnable attribute-level representations at different layers of a transformer." />
        </attvalues>
      </edge>
      <edge source="2211.03152" target="0907.0806" id="3921">
        <attvalues>
          <attvalue for="5" value=" The noisy channel method has been mainly applied in machine translation, but it was also applied for Document Compression \cite{daume2002noisy}, exactly in the dropping word task (related to summarization)." />
        </attvalues>
      </edge>
      <edge source="2403.13233" target="2210.02414" id="3922">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, the rapid growth of Chinese open-source large language models, including ChatGLM\cite{glm130b}, Baichuan\cite{yang2023baichuan}, Qwen\cite{qwen}, and BELLE\cite{BELLE}, contributing positively to the field's evolution." />
        </attvalues>
      </edge>
      <edge source="2403.13233" target="2309.10305" id="3923">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, the rapid growth of Chinese open-source large language models, including ChatGLM\cite{glm130b}, Baichuan\cite{yang2023baichuan}, Qwen\cite{qwen}, and BELLE\cite{BELLE}, contributing positively to the field's evolution." />
        </attvalues>
      </edge>
      <edge source="2403.13233" target="2309.02033" id="3925">
        <attvalues>
          <attvalue for="5" value=" Data-Juicer\cite{chen2024datajuicer} is a comprehensive one-stop data processing system for Large Language Models." />
        </attvalues>
      </edge>
      <edge source="2403.13233" target="2308.12032" id="3926">
        <attvalues>
          <attvalue for="5" value=" Instruction Following Difficulty (IFD) \cite{li2023quantity} also introduced to assess the challenge of responding to specific instructions." />
        </attvalues>
      </edge>
      <edge source="1909.05708" target="1902.00508" id="3927">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{Glavas19} reported limited evidence in support of this practice---they found that cross-lingual embeddings optimized for a BDI evaluation metric were not necessarily better on downstream tasks." />
        </attvalues>
      </edge>
      <edge source="1909.05708" target="1605.02276" id="3928">
        <attvalues>
          <attvalue for="5" value=" \cite{faruqui2016problems}'s work on word similarity), with concerning findings about its reliability." />
        </attvalues>
      </edge>
      <edge source="1909.05708" target="1710.04087" id="3929">
        <attvalues>
          <attvalue for="5" value=" A massive dataset of 110 bilingual dictionaries, known as the MUSE dataset, was introduced in early 2018 along with a strong baseline \cite{Conneau2018}." />
        </attvalues>
      </edge>
      <edge source="1909.05708" target="1805.11222" id="3930">
        <attvalues>
          <attvalue for="5" value=" \cite{grave2018unsupervised,jawanpuria2019, Hoshen2018AnIC, hoshen2018non, DBLP:journals/corr/abs-1809-02306,joulin2018rcsls})." />
        </attvalues>
      </edge>
      <edge source="1909.05708" target="1808.08773" id="3931">
        <attvalues>
          <attvalue for="5" value=" \cite{grave2018unsupervised,jawanpuria2019, Hoshen2018AnIC, hoshen2018non, DBLP:journals/corr/abs-1809-02306,joulin2018rcsls})." />
        </attvalues>
      </edge>
      <edge source="1909.05708" target="1804.07745" id="3932">
        <attvalues>
          <attvalue for="5" value=" \cite{grave2018unsupervised,jawanpuria2019, Hoshen2018AnIC, hoshen2018non, DBLP:journals/corr/abs-1809-02306,joulin2018rcsls})." />
        </attvalues>
      </edge>
      <edge source="1909.05708" target="1809.00064" id="3933">
        <attvalues>
          <attvalue for="5" value=" \cite{kementchedjhieva2018generalizing})." />
        </attvalues>
      </edge>
      <edge source="1909.05708" target="1909.02855" id="3934">
        <attvalues>
          <attvalue for="5" value=" As an alternative, we point them to morphologically complete BDI resources, built bottom-up \cite{czarnowska2019dont}." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="1805.03818" id="3935">
        <attvalues>
          <attvalue for="5" value=" Proposed methods use explanations to constrain or regularize the learned model \cite{zaidan_using_2007, small2011constrained, ba2015predicting, zhang_rationale-augmented_2016, Srivastava2018LearningCF, andreas2018learning, liang2020alice}, to automatically label data for data augmentation \cite{hancock_training_2018, wang_does_2019, awasthi2020learning}, as additional supervision \cite{narang_wt5?!_2020, hase2020leakage, pruthi2020} or intermediate structured variables \cite{camburu_e-snli:_2018, rajani_explain_2019, wiegreffe2020}, and simply as model inputs \cite{rupprecht2018guide, Co-Reyes2019Guiding, zhou2020towards}." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="2004.14546" id="3936">
        <attvalues>
          <attvalue for="5" value=" Yet there are tasks where explanations do not fulfill these roles effectively, as improvements in performance prove elusive even when thousands of explanations are gathered \cite{narang_wt5?!_2020, hase2020leakage}." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="1812.01193" id="3937">
        <attvalues>
          <attvalue for="5" value=" Proposed methods use explanations to constrain or regularize the learned model \cite{zaidan_using_2007, small2011constrained, ba2015predicting, zhang_rationale-augmented_2016, Srivastava2018LearningCF, andreas2018learning, liang2020alice}, to automatically label data for data augmentation \cite{hancock_training_2018, wang_does_2019, awasthi2020learning}, as additional supervision \cite{narang_wt5?!_2020, hase2020leakage, pruthi2020} or intermediate structured variables \cite{camburu_e-snli:_2018, rajani_explain_2019, wiegreffe2020}, and simply as model inputs \cite{rupprecht2018guide, Co-Reyes2019Guiding, zhou2020towards}." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="1803.11544" id="3938">
        <attvalues>
          <attvalue for="5" value=" Proposed methods use explanations to constrain or regularize the learned model \cite{zaidan_using_2007, small2011constrained, ba2015predicting, zhang_rationale-augmented_2016, Srivastava2018LearningCF, andreas2018learning, liang2020alice}, to automatically label data for data augmentation \cite{hancock_training_2018, wang_does_2019, awasthi2020learning}, as additional supervision \cite{narang_wt5?!_2020, hase2020leakage, pruthi2020} or intermediate structured variables \cite{camburu_e-snli:_2018, rajani_explain_2019, wiegreffe2020}, and simply as model inputs \cite{rupprecht2018guide, Co-Reyes2019Guiding, zhou2020towards}." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="1702.08608" id="3939">
        <attvalues>
          <attvalue for="5" value=" Improving models in this manner is a natural goal of approaches using explanations, since one purpose of an explanation is to communicate a mental model \cite{doshi-velez_towards_2017, miller2019explanation}." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="1706.07269" id="3940">
        <attvalues>
          <attvalue for="5" value=" But how do explanations get used as additional targets, as inputs, as regularizers, as structured variables, and as rules for automatic data labeling? Even under a general notion of what an ``explanation&quot; is, eg the answer to some why-question \cite{miller2019explanation}, this kind of data plays an impressive number of roles." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="2002.08910" id="3941">
        <attvalues>
          <attvalue for="5" value=" This is because large language models now (1) store a great amount of knowledge in their parameters \cite{roberts-etal-2020-much, lewis2020retrieval}, and (2) infer tasks at test time from the input itself \cite{radford_language_2019, gpt3, weller-etal-2020-learning}." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="1907.11692" id="3942">
        <attvalues>
          <attvalue for="5" value=" Using RoBERTa as a representative large language model \cite{liu_roberta_2019} and Sentence-BERT as a retrieval model \cite{reimers-gurevych-2019-sentence}, we investigate a number of primary research questions, each given with brief context: \begin{enumerate}[itemsep=0pt, wide=0pt, leftmargin=*, after=\strut] \item RQ1." />
        </attvalues>
      </edge>
      <edge source="2102.02201" target="1908.10084" id="3943">
        <attvalues>
          <attvalue for="5" value=" Using RoBERTa as a representative large language model \cite{liu_roberta_2019} and Sentence-BERT as a retrieval model \cite{reimers-gurevych-2019-sentence}, we investigate a number of primary research questions, each given with brief context: \begin{enumerate}[itemsep=0pt, wide=0pt, leftmargin=*, after=\strut] \item RQ1." />
        </attvalues>
      </edge>
      <edge source="2407.16216" target="2204.05862" id="3946">
        <attvalues>
          <attvalue for="5" value=" Reinforcement Learning from Human Feedback (RLHF) \cite{ouyang2022training,bai2022training} has emerged as a groundbreaking technique for aligning LLMs." />
        </attvalues>
      </edge>
      <edge source="2407.16216" target="2303.08774" id="3947">
        <attvalues>
          <attvalue for="5" value=" This approach has led to the development of powerful models such as GPT-4 \cite{openai2024gpt4}, Claude \cite{anthropic2024claude}, and Gemini \cite{team2023gemini}." />
        </attvalues>
      </edge>
      <edge source="2407.16216" target="2312.11805" id="3948">
        <attvalues>
          <attvalue for="5" value=" This approach has led to the development of powerful models such as GPT-4 \cite{openai2024gpt4}, Claude \cite{anthropic2024claude}, and Gemini \cite{team2023gemini}." />
        </attvalues>
      </edge>
      <edge source="2406.17453" target="2101.02235" id="3955">
        <attvalues>
          <attvalue for="5" value=" This process of acquiring information through questioning is essential for children to learn about the world \cite{Chouinard2007} and for adults to solve complex problems \cite{Geva2021Aristotle}." />
        </attvalues>
      </edge>
      <edge source="2406.17453" target="2402.19471" id="3956">
        <attvalues>
          <attvalue for="5" value=" Questions, however, vary in their level of informativeness, with some questions being more informative and efficient in reaching the problem's solution \cite{grand:LIPS}." />
        </attvalues>
      </edge>
      <edge source="2406.17453" target="2310.01468" id="3958">
        <attvalues>
          <attvalue for="5" value=" \cite{Apple} improve open-source LLMs via Reinforcement Learning and Behavioral Cloning from larger LLMs' data." />
        </attvalues>
      </edge>
      <edge source="2406.17453" target="2305.18290" id="3960">
        <attvalues>
          <attvalue for="5" value=" This set of questions, along with the corresponding EIG values, as a proxy for the questions' informativeness, is used for Direct Preference Optimization (DPO; \cite{rafailov2024direct})." />
        </attvalues>
      </edge>
      <edge source="2406.17453" target="2311.08584" id="3961">
        <attvalues>
          <attvalue for="5" value=" Our core idea of sampling, self-annotation to filter with a metric, and preference optimization could improve questions’ generations in a plethora of tasks – recommendation systems \cite{piriyakulkij2023activepreference} and image retrieval \cite{GroundedRetrieval23} for example." />
        </attvalues>
      </edge>
      <edge source="2406.14115" target="1912.02292" id="3962">
        <attvalues>
          <attvalue for="5" value=" Recent research highlights that data quality is more critical than data quantity for effective fine-tuning~\cite{nakkiran2021deep, shumailov2024ai, zhou2024lima, jindal2024birbal}." />
        </attvalues>
      </edge>
      <edge source="2406.14115" target="2307.08701" id="3963">
        <attvalues>
          <attvalue for="5" value=" As a result, several data curation techniques have been proposed, such as data selection~\cite{chen2024alpagasus, li2024quantity}, data evolution~\cite{wang2023selfinstruct, xu2023wizardlm}, and data reflection~\cite{mukherjee2023orca, yin-etal-2023-dynosaur}." />
        </attvalues>
      </edge>
      <edge source="2406.14115" target="2212.10560" id="3964">
        <attvalues>
          <attvalue for="5" value=" As a result, several data curation techniques have been proposed, such as data selection~\cite{chen2024alpagasus, li2024quantity}, data evolution~\cite{wang2023selfinstruct, xu2023wizardlm}, and data reflection~\cite{mukherjee2023orca, yin-etal-2023-dynosaur}." />
        </attvalues>
      </edge>
      <edge source="2406.14115" target="2306.02707" id="3965">
        <attvalues>
          <attvalue for="5" value=" As a result, several data curation techniques have been proposed, such as data selection~\cite{chen2024alpagasus, li2024quantity}, data evolution~\cite{wang2023selfinstruct, xu2023wizardlm}, and data reflection~\cite{mukherjee2023orca, yin-etal-2023-dynosaur}." />
        </attvalues>
      </edge>
      <edge source="2406.14115" target="2108.07258" id="3966">
        <attvalues>
          <attvalue for="5" value=" Although some surveys~\cite{bommasani2021opportunities, albalak2024surveydataselectionlanguage, wang2024survey} have reviewed data curation techniques in the contexts of pretraining, fine-tuning, and reinforcement learning, they generally provide high-level overviews and lack in-depth discussions of the fine-tuning stage." />
        </attvalues>
      </edge>
      <edge source="2306.02707" target="2303.08774" id="3967">
        <attvalues>
          <attvalue for="5" value=" Large Foundation Models (LFMs) such as ChatGPT and GPT-4~\cite{openai2023gpt4} exhibit remarkable zero-shot performances across a broad spectrum of tasks." />
        </attvalues>
      </edge>
      <edge source="2306.02707" target="2107.03374" id="3968">
        <attvalues>
          <attvalue for="5" value=" Alongside academic benchmarks like Human Eval~\cite{chen2021codex} and Big Bench~\cite{srivastava2022imitation}, GPT-4 has also demonstrated human-level performance on various professional exams, including the bar exam, SAT, GRE, and USMLE." />
        </attvalues>
      </edge>
      <edge source="2306.02707" target="2206.04615" id="3969">
        <attvalues>
          <attvalue for="5" value=" Alongside academic benchmarks like Human Eval~\cite{chen2021codex} and Big Bench~\cite{srivastava2022imitation}, GPT-4 has also demonstrated human-level performance on various professional exams, including the bar exam, SAT, GRE, and USMLE." />
        </attvalues>
      </edge>
      <edge source="2306.02707" target="2212.08073" id="3971">
        <attvalues>
          <attvalue for="5" value=" As these models continue to evolve and become more powerful, an intriguing question arises: Can we use the model itself to supervise its own behavior or that of other AI models? ~\cite{Bai2022ConstitutionalAH} have shown that by sampling output from an initial model, generating revisions, and then fine-tuning the original model based on these revised responses, model behavior can be controlled more effectively and can be made more harmless, with significantly fewer human labels." />
        </attvalues>
      </edge>
      <edge source="2306.02707" target="2304.12244" id="3972">
        <attvalues>
          <attvalue for="5" value=" Recently, there has been an influx of studies using LFMs like ChatGPT and GPT-4 as teachers to generate large datasets, for {instruction tuning}, and to train smaller models, such as Alpaca~\cite{alpaca}, WizardLM~\cite{xu2023wizardlm} and Vicuna~\cite{vicuna-2023}." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2208.03188" id="3975">
        <attvalues>
          <attvalue for="5" value=" Recent years have seen remarkable progress in Conversational AI, primarily driven by the advent of approaches and language models~\cite{shuster2022blenderbot,zhang2023enhancing,longpre2023flan,touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2308.08169" id="3976">
        <attvalues>
          <attvalue for="5" value=" Recent years have seen remarkable progress in Conversational AI, primarily driven by the advent of approaches and language models~\cite{shuster2022blenderbot,zhang2023enhancing,longpre2023flan,touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2301.13688" id="3977">
        <attvalues>
          <attvalue for="5" value=" For instance, FlanT5~\cite{longpre2023flan} presents the flan collections with a wide array of datasets and tasks." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2106.02787" id="3979">
        <attvalues>
          <attvalue for="5" value=" Current dialogue datasets~\cite{lin2021bitod,asri2017frames} are typically limited in size and task-specific, which thus results in suboptimal ability in task-oriented model performance." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2205.12673" id="3981">
        <attvalues>
          <attvalue for="5" value=" Other collections~\cite{gupta2022instructdial,kim2022soda,ding2023enhancing,dubois2023alpacafarm} often distill single dataset from ChatGPT or process datasets into a sequence-to-sequence format to support language model training, featuring only input-output pairs such as dialogue context and system response." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2305.14233" id="3982">
        <attvalues>
          <attvalue for="5" value=" Other collections~\cite{gupta2022instructdial,kim2022soda,ding2023enhancing,dubois2023alpacafarm} often distill single dataset from ChatGPT or process datasets into a sequence-to-sequence format to support language model training, featuring only input-output pairs such as dialogue context and system response." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2212.12017" id="3983">
        <attvalues>
          <attvalue for="5" value=" Although OPT~\cite{iyer2022opt} have incorporated collections with several dialogue datasets, these collections remain inaccessible to the public." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="1705.06476" id="3984">
        <attvalues>
          <attvalue for="5" value=" In contract, efforts like InstructDial~\cite{gupta2022instructdial} and ParlAI~\cite{miller2017parlai} consist of more dialogue datasets, but they lack diversity and comprehensiveness." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2212.10465" id="3985">
        <attvalues>
          <attvalue for="5" value=" Other collections~\cite{gupta2022instructdial,kim2022soda,ding2023enhancing,dubois2023alpacafarm} often distill single dataset from ChatGPT or process datasets into a sequence-to-sequence format to support language model training, featuring only input-output pairs such as dialogue context and system response." />
        </attvalues>
      </edge>
      <edge source="2307.10172" target="2305.14387" id="3986">
        <attvalues>
          <attvalue for="5" value=" Other collections~\cite{gupta2022instructdial,kim2022soda,ding2023enhancing,dubois2023alpacafarm} often distill single dataset from ChatGPT or process datasets into a sequence-to-sequence format to support language model training, featuring only input-output pairs such as dialogue context and system response." />
        </attvalues>
      </edge>
      <edge source="2007.16013" target="1911.00172" id="3988">
        <attvalues>
          <attvalue for="5" value=" \cite{khandelwal2020generalization} propose nearest-neighbor LM which can use external data to bias its predictions, &#09;however, significant limits application of this type of model, especially in ASR domain." />
        </attvalues>
      </edge>
      <edge source="2405.05348" target="2001.08361" id="3990">
        <attvalues>
          <attvalue for="5" value=" Research has shown that performance in language models depends strongly on scale and less on model shape \cite{kaplan}, where scale refers to the number of parameters, the training dataset size, and the amount of compute for training." />
        </attvalues>
      </edge>
      <edge source="2405.05348" target="2005.14165" id="3991">
        <attvalues>
          <attvalue for="5" value="5 billion parameters for GPT-2 to 175 billion parameters for GPT-3 which helped improve across various NLP tasks \cite{brownGPT3}." />
        </attvalues>
      </edge>
      <edge source="2405.05348" target="2108.04840" id="3992">
        <attvalues>
          <attvalue for="5" value=" Although these methods have been widely applied to LLMs \cite{madsen}, to the best of our knowledge no research has been conducted on the impact of model size on the quality of these kinds of explanations." />
        </attvalues>
      </edge>
      <edge source="2405.05348" target="2204.05514" id="3993">
        <attvalues>
          <attvalue for="5" value=" We apply two approaches to assess the quality of explanations, namely faithfulness \cite{chan} and plausibility \cite{deyoung}." />
        </attvalues>
      </edge>
      <edge source="2405.05348" target="1911.03429" id="3994">
        <attvalues>
          <attvalue for="5" value=" We apply two approaches to assess the quality of explanations, namely faithfulness \cite{chan} and plausibility \cite{deyoung}." />
        </attvalues>
      </edge>
      <edge source="2406.12754" target="2304.03439" id="3995">
        <attvalues>
          <attvalue for="5" value=" With the advent of Large Language Models (LLMs), researchers have evaluated LLMs' performance on diverse tasks \cite{liu2023evaluating, deng2024tables, wu-etal-2023-hi} and observed LLMs' extraordinary performance on many \cite{10." />
        </attvalues>
      </edge>
      <edge source="2406.12754" target="1506.08126" id="3997">
        <attvalues>
          <attvalue for="5" value=" However, with all these studies on humor and LLMs' understanding of humor, most of these humor datasets and evaluations remain in English \cite{radev-etal-2016-humor, hasan-etal-2019-ur}." />
        </attvalues>
      </edge>
      <edge source="2406.12754" target="2403.18058" id="3998">
        <attvalues>
          <attvalue for="5" value=" In addition, \cite{bai2024coig} have observed that when tuning LLMs on data from RZB, LLMs achieve the best performance on Chinese reasoning tasks compared to tuning LLMs on data from other sources, indicating the significant value of jokes from RZB." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2104.11462" id="4000">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL) for speech representation using large neural networks and unlabeled data offers effective initialization and representations for downstream tasks~\cite{yang2021superb,evain2021lebenchmark,chang2021exploration,tsai-etal-2022-superb,mohamed2022self}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2110.04590" id="4001">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL) for speech representation using large neural networks and unlabeled data offers effective initialization and representations for downstream tasks~\cite{yang2021superb,evain2021lebenchmark,chang2021exploration,tsai-etal-2022-superb,mohamed2022self}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2203.06849" id="4002">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL) for speech representation using large neural networks and unlabeled data offers effective initialization and representations for downstream tasks~\cite{yang2021superb,evain2021lebenchmark,chang2021exploration,tsai-etal-2022-superb,mohamed2022self}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2205.10643" id="4003">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL) for speech representation using large neural networks and unlabeled data offers effective initialization and representations for downstream tasks~\cite{yang2021superb,evain2021lebenchmark,chang2021exploration,tsai-etal-2022-superb,mohamed2022self}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2106.07447" id="4004">
        <attvalues>
          <attvalue for="5" value=" In light of this, ContentVec~\cite{qian2022contentvec} imposes speaker-invariant constraints to pre-trained HuBERT models~\cite{hsu2021hubert} to improve content-related downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2110.13900" id="4005">
        <attvalues>
          <attvalue for="5" value=" Among prior methods, learning discrete units like K-means clusters benefits downstream performance~\cite{hsu2021hubert,chen2022wavlm,chung2021w2v,maekaku2022exploration,chiu2022bestrq,ren2022speech,wells2022phonetic}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2108.06209" id="4006">
        <attvalues>
          <attvalue for="5" value=" Among prior methods, learning discrete units like K-means clusters benefits downstream performance~\cite{hsu2021hubert,chen2022wavlm,chung2021w2v,maekaku2022exploration,chiu2022bestrq,ren2022speech,wells2022phonetic}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2204.03240" id="4008">
        <attvalues>
          <attvalue for="5" value=" Among prior methods, learning discrete units like K-means clusters benefits downstream performance~\cite{hsu2021hubert,chen2022wavlm,chung2021w2v,maekaku2022exploration,chiu2022bestrq,ren2022speech,wells2022phonetic}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="1709.07902" id="4009">
        <attvalues>
          <attvalue for="5" value=" Extracting speaker-invariant linguistic content can benefit downstream tasks like automatic speech recognition~(ASR) and phoneme recognition~(PR)~\cite{hsu2017unsupervised,tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2205.09872" id="4011">
        <attvalues>
          <attvalue for="5" value=" Extracting speaker-invariant linguistic content can benefit downstream tasks like automatic speech recognition~(ASR) and phoneme recognition~(PR)~\cite{hsu2017unsupervised,tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2208.13191" id="4012">
        <attvalues>
          <attvalue for="5" value=" Extracting speaker-invariant linguistic content can benefit downstream tasks like automatic speech recognition~(ASR) and phoneme recognition~(PR)~\cite{hsu2017unsupervised,tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2311.03389" id="4013">
        <attvalues>
          <attvalue for="5" value=" Extracting speaker-invariant linguistic content can benefit downstream tasks like automatic speech recognition~(ASR) and phoneme recognition~(PR)~\cite{hsu2017unsupervised,tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2204.09224" id="4014">
        <attvalues>
          <attvalue for="5" value=" Following this observation, we present speaker-invariant clustering~(Spin), a novel and cost-effective self-supervised fine-tuning~(SSFT) method for SSL models that leverages vector quantization~\cite{caron2018deep,asano2019self,caron2020unsupervised} and speaker disentanglement~\cite{qian2022contentvec} to improve content representation." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="1807.05520" id="4015">
        <attvalues>
          <attvalue for="5" value=" Following this observation, we present speaker-invariant clustering~(Spin), a novel and cost-effective self-supervised fine-tuning~(SSFT) method for SSL models that leverages vector quantization~\cite{caron2018deep,asano2019self,caron2020unsupervised} and speaker disentanglement~\cite{qian2022contentvec} to improve content representation." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2006.09882" id="4017">
        <attvalues>
          <attvalue for="5" value=" Following this observation, we present speaker-invariant clustering~(Spin), a novel and cost-effective self-supervised fine-tuning~(SSFT) method for SSL models that leverages vector quantization~\cite{caron2018deep,asano2019self,caron2020unsupervised} and speaker disentanglement~\cite{qian2022contentvec} to improve content representation." />
        </attvalues>
      </edge>
      <edge source="2305.11072" target="2011.11588" id="4018">
        <attvalues>
          <attvalue for="5" value=" Such design leads to a disentangled representation focusing on the spoken content, improving various downstream tasks, including content-related tasks in SUPERB~\cite{yang2021superb} and ZeroSpeech~\cite{nguyen2020zero}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1303.5778" id="4019">
        <attvalues>
          <attvalue for="5" value=" On the one hand, the rapid development of deep neural networks has dramatically pushed the limit of the models~\cite{abdel2012applying,graves2013speech,chan2015listen,vaswani2017attention,dong2018speech,gulati2020conformer}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1508.01211" id="4020">
        <attvalues>
          <attvalue for="5" value=" On the one hand, the rapid development of deep neural networks has dramatically pushed the limit of the models~\cite{abdel2012applying,graves2013speech,chan2015listen,vaswani2017attention,dong2018speech,gulati2020conformer}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1706.03762" id="4021">
        <attvalues>
          <attvalue for="5" value=" Thus, we can easily evaluate the E2E-ASR performance of pretrained SSLRs available in S3PRL using the current state-of-the-art (SOTA) neural network models, such as Transformers~\cite{vaswani2017attention,dong2018speech} and Conformers~\cite{gulati2020conformer}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="2005.08100" id="4022">
        <attvalues>
          <attvalue for="5" value=" Thus, we can easily evaluate the E2E-ASR performance of pretrained SSLRs available in S3PRL using the current state-of-the-art (SOTA) neural network models, such as Transformers~\cite{vaswani2017attention,dong2018speech} and Conformers~\cite{gulati2020conformer}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="2106.06909" id="4023">
        <attvalues>
          <attvalue for="5" value=" On the other hand, the increasing computing resources have enabled to train an automatic speech recognition (ASR) system with a large amount of transcribed data~\cite{panayotov2015librispeech,chen2021gigaspeech}, leading to a better performance." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1909.09116" id="4026">
        <attvalues>
          <attvalue for="5" value=" In~\cite{lee2013pseudo,synnaeve2019end,kahn2020self}, a semi-supervised method, called pseudo-labelling, was proposed to use both transcribed and untranscribed data." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1904.03240" id="4029">
        <attvalues>
          <attvalue for="5" value=" In~\cite{chung2019unsupervised,chung2020vector}, the authors adopted an method similar to the autoregressive language models (LMs) to predict the future acoustic features (eg FBANK) conditioned on the past input features, called autoregressive predictive coding (APC)." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="2005.08392" id="4030">
        <attvalues>
          <attvalue for="5" value=" In~\cite{chung2019unsupervised,chung2020vector}, the authors adopted an method similar to the autoregressive language models (LMs) to predict the future acoustic features (eg FBANK) conditioned on the past input features, called autoregressive predictive coding (APC)." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1910.12638" id="4031">
        <attvalues>
          <attvalue for="5" value=" Instead of autoregressive modeling, some researchers proposed to use masking prediction techniques as in BERT-LM~\cite{devlin2018bert} to learn the speech representations, including Mockingjay~\cite{liu2020mockingjay}, TERA~\cite{liu2020tera} and NPC~\cite{liu2020non}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="2011.00406" id="4033">
        <attvalues>
          <attvalue for="5" value=" Instead of autoregressive modeling, some researchers proposed to use masking prediction techniques as in BERT-LM~\cite{devlin2018bert} to learn the speech representations, including Mockingjay~\cite{liu2020mockingjay}, TERA~\cite{liu2020tera} and NPC~\cite{liu2020non}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1807.03748" id="4034">
        <attvalues>
          <attvalue for="5" value=" In~\cite{oord2018representation,schneider2019wav2vec}, the models were optimized with a contrastive loss to distinguish the positive sample from negative samples in predictions of future." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1904.05862" id="4035">
        <attvalues>
          <attvalue for="5" value=" In~\cite{oord2018representation,schneider2019wav2vec}, the models were optimized with a contrastive loss to distinguish the positive sample from negative samples in predictions of future." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1910.05453" id="4036">
        <attvalues>
          <attvalue for="5" value=" Later in~\cite{baevski2019vq, baevski2020wav2vec}, a BERT Transformer model is concatenated after the encoder trained by the contrastive loss." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1807.05520" id="4037">
        <attvalues>
          <attvalue for="5" value=" Recently, a novel model, called HuBERT~\cite{hsu2021hubert}, was proposed to pretrain the representation model by a classification tasks using pseudo-labels motivated by deep cluster models \cite{caron2018deep,xie2016unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1511.06335" id="4038">
        <attvalues>
          <attvalue for="5" value=" Recently, a novel model, called HuBERT~\cite{hsu2021hubert}, was proposed to pretrain the representation model by a classification tasks using pseudo-labels motivated by deep cluster models \cite{caron2018deep,xie2016unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="2004.10234" id="4041">
        <attvalues>
          <attvalue for="5" value=" We can also easily evalute the SSLRs in other downstream tasks, including speech translation (ST) \cite{inaguma2020espnet} and speech enhancement (SE) \cite{li2021espnet}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="2011.03706" id="4042">
        <attvalues>
          <attvalue for="5" value=" We can also easily evalute the SSLRs in other downstream tasks, including speech translation (ST) \cite{inaguma2020espnet} and speech enhancement (SE) \cite{li2021espnet}." />
        </attvalues>
      </edge>
      <edge source="2110.04590" target="1912.07875" id="4043">
        <attvalues>
          <attvalue for="5" value=" It is also an interesting question in the air about the generalization ability of these SSLRs, given the fact that most of SSLRs were trained and tested mainly on LibriSpeech~\cite{panayotov2015librispeech,kahn2020libri}." />
        </attvalues>
      </edge>
      <edge source="2208.13191" target="1802.05365" id="4044">
        <attvalues>
          <attvalue for="5" value=" Methods like ELMo \cite{Peters2018}, GPT \cite{Radford2018}, and BERT \cite{Devlin2018} work by using an unsupervised task that develops a representation of text that is useful for downstream tasks in a way that is agnostic to what that task is." />
        </attvalues>
      </edge>
      <edge source="2208.13191" target="1807.03748" id="4046">
        <attvalues>
          <attvalue for="5" value=" Contrastive estimation \cite{Gutmann2010}, in which a full reconstruction is not learned, has yielded representations that achieve strong results in speaker identification and speech recognition \cite{Oord2018}." />
        </attvalues>
      </edge>
      <edge source="2208.13191" target="2010.10504" id="4047">
        <attvalues>
          <attvalue for="5" value="0 \cite{ Baevski2020} and Adaptive SpecAugment \cite{zhang20}." />
        </attvalues>
      </edge>
      <edge source="2208.13191" target="2102.00184" id="4049">
        <attvalues>
          <attvalue for="5" value=" Approaches in this space have commonly relied on techniques like adversarial learning to exclude particular parts of a signal thought to be irrelevant, as in \cite{Wang2021adversarially}." />
        </attvalues>
      </edge>
      <edge source="2208.13191" target="1905.06791" id="4050">
        <attvalues>
          <attvalue for="5" value=" Traditionally, this is done by training a model that performs both ASR and TTS with a shared encoder that is tasked with representing inputs from both the speech and text domains \cite{ren2020unsupervised, xu2020lrspeech}." />
        </attvalues>
      </edge>
      <edge source="2208.13191" target="1805.01445" id="4051">
        <attvalues>
          <attvalue for="5" value=" overfit solutions \cite{Weber2018} and selection of significant units in a DNN \cite{Frankle2018}), the stochasticity of parameter initialization and minibatch selection can be decisive." />
        </attvalues>
      </edge>
      <edge source="2208.13191" target="1707.04879" id="4052">
        <attvalues>
          <attvalue for="5" value=" Possible applications of our joint modeling task include refinement of back-transcription based semi-supervised learning systems such as speech chains \cite{Tjandra17} and Sequential MixMatch \cite{ZChen21}." />
        </attvalues>
      </edge>
      <edge source="2311.03389" target="1811.12359" id="4053">
        <attvalues>
          <attvalue for="5" value=" Learning disentangled representations yields promising results in domains like computer vision, allowing models to robustly separate generative factors within data~\cite{locatello2019challenging,higgins2017beta,kim2018disentangling}." />
        </attvalues>
      </edge>
      <edge source="2311.03389" target="1802.05983" id="4054">
        <attvalues>
          <attvalue for="5" value=" Learning disentangled representations yields promising results in domains like computer vision, allowing models to robustly separate generative factors within data~\cite{locatello2019challenging,higgins2017beta,kim2018disentangling}." />
        </attvalues>
      </edge>
      <edge source="2311.03389" target="2012.09276" id="4055">
        <attvalues>
          <attvalue for="5" value=" Such representations provide multiple benefits including enhanced predictive abilities on downstream tasks, decreased sample complexity, greater explainability, fairness, and a means to avoid shortcut learning~\cite{carbonneau2020measuring}." />
        </attvalues>
      </edge>
      <edge source="2311.03389" target="1804.02812" id="4056">
        <attvalues>
          <attvalue for="5" value=" However, progress on learning disentangled speech representations has been limited, despite potential benefits such as improved understanding of speech signals, interpretable features, controllable generation, source separation, multilingual and cross-lingual speech processing, voice conversion, robustness to variability, few-shot learning, privacy-preserving, etc\cite{Qian2018UnsupervisedSL,chou2018multi}." />
        </attvalues>
      </edge>
      <edge source="2311.03389" target="1803.09017" id="4057">
        <attvalues>
          <attvalue for="5" value=" On the other hand, most synthetic datasets are not large or rich enough to benchmark speech disentanglement models~\cite{wang2018style,zhou2020comparison,jia2018transfer}." />
        </attvalues>
      </edge>
      <edge source="2311.03389" target="1806.04558" id="4058">
        <attvalues>
          <attvalue for="5" value=" On the other hand, most synthetic datasets are not large or rich enough to benchmark speech disentanglement models~\cite{wang2018style,zhou2020comparison,jia2018transfer}." />
        </attvalues>
      </edge>
      <edge source="2109.10044" target="1804.06610" id="4072">
        <attvalues>
          <attvalue for="5" value=" Tree Adjoining Grammar \cite{joshi:87}, like CCG, has become a standard grammar formalism in Computational Linguistics and has formed the basis for much experimental work in developing parsers and NLP systems \cite{kasai-etal-2018-end}." />
        </attvalues>
      </edge>
      <edge source="2109.10044" target="1702.06594" id="4073">
        <attvalues>
          <attvalue for="5" value=" Despite the additional power of CCG (and TAG), there are still efficient parsing algorithms for CCG (and TAG) which are polynomial in the length of the input sentence \cite{vijay:93,kuhlmann-etal-2018-complexity}." />
        </attvalues>
      </edge>
      <edge source="2106.03337" target="1911.12237" id="4076">
        <attvalues>
          <attvalue for="5" value=" Samsum \cite{gliwa2019samsum}, the only large scale dataset for conversation summarization, contains over $16,000$ open-domain conversations and summaries created artificially by humans." />
        </attvalues>
      </edge>
      <edge source="2106.03337" target="1910.13461" id="4077">
        <attvalues>
          <attvalue for="5" value=" Large scale pre-trained language models (PLMs) \cite{lewis2020bart, brown2020language, raffel2020exploring} have been used in various text generation tasks \cite{budzianowski2019hello, min2020ambigqa, cachola2020tldr}." />
        </attvalues>
      </edge>
      <edge source="2105.05535" target="1804.09132" id="4078">
        <attvalues>
          <attvalue for="5" value=" Previous efforts \cite{paetzold2016semeval,yimam2018report,zampieri2017complex} have focused on framing this as a binary classification task, which might not be ideal, since a word close to the decision boundary is assumed to be just as complex as one further away \cite{shardlow2020complex}." />
        </attvalues>
      </edge>
      <edge source="2105.05535" target="2106.00473" id="4081">
        <attvalues>
          <attvalue for="5" value=" To alleviate this issue, SemEval-2021 Task 1 \cite{shardlow2021semeval} formulates this task as a regression task, where a model should predict the complexity value of words (Subtask 1) and MWEs (Subtask 2) in context." />
        </attvalues>
      </edge>
      <edge source="2405.20335" target="2303.08774" id="4082">
        <attvalues>
          <attvalue for="5" value=" Recent advances in artificial intelligence, epitomized by large language models (LLMs) such as GPT-4~\cite{gpt4} and Claude~\cite{claude3}, have demonstrated remarkable capabilities across diverse real-world applications." />
        </attvalues>
      </edge>
      <edge source="2405.20335" target="2204.05862" id="4084">
        <attvalues>
          <attvalue for="5" value=" Ensuring these models align with human expectations and values is crucial, especially as they are integrated into and utilized across numerous applications~\cite{instructGPT,bai2022training}." />
        </attvalues>
      </edge>
      <edge source="2405.20335" target="2305.18290" id="4085">
        <attvalues>
          <attvalue for="5" value=" This approach involves initially gathering preferences from human or AI sources, followed by optimizing a policy model against a clearly built Reward Model (RM)~\cite{instructGPT} or an implicit preference learning target~\cite{dpo}." />
        </attvalues>
      </edge>
      <edge source="2405.20335" target="2307.09288" id="4086">
        <attvalues>
          <attvalue for="5" value=" Specifically, we start with pretrained models Llama-2~\cite{llama2}, a collection of prompts, and a well-trained annotator, GPT-4." />
        </attvalues>
      </edge>
      <edge source="2405.20335" target="2306.05685" id="4087">
        <attvalues>
          <attvalue for="5" value=" We evaluate Xwin-LM on two popular instruction-following benchmarks, AlpacaEval~\cite{alpacaeval} and MT-bench~\cite{vicuna}." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2303.08774" id="4088">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as GPT-4 \cite{OpenAI_arXiv2023}, PaLM2 \cite{Google_arXiv2023}, and Llama2 (Large Language Model META AI) \cite{Touvron_arXiv2023a}, have now become a prominent component in modern natural language processing (NLP) and are successfully utilized in various NLP tasks, such as machine translation, text summarization, and question answering." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2305.10403" id="4089">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as GPT-4 \cite{OpenAI_arXiv2023}, PaLM2 \cite{Google_arXiv2023}, and Llama2 (Large Language Model META AI) \cite{Touvron_arXiv2023a}, have now become a prominent component in modern natural language processing (NLP) and are successfully utilized in various NLP tasks, such as machine translation, text summarization, and question answering." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2307.09288" id="4090">
        <attvalues>
          <attvalue for="5" value=" In this study, we reveal it by performing $N$-best ASR hypotheses rescoring using Llama2-7B \cite{Touvron_arXiv2023a}, which is one of the most representative Transformer \cite{Vaswani_NIPS2017} decoder-based causal LLMs, on the CHiME-7 DASR task." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="1910.14659" id="4091">
        <attvalues>
          <attvalue for="5" value=" We conducted experiments, including experimental settings that have not been investigated in previous studies \cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}, and thus, the experimental results and findings obtained in this study are informative for researchers in this field (note that Llama2-7B is allowed to be used in the CHiME-8 challenge \cite{CHiME})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2104.04950" id="4092">
        <attvalues>
          <attvalue for="5" value=" We conducted experiments, including experimental settings that have not been investigated in previous studies \cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}, and thus, the experimental results and findings obtained in this study are informative for researchers in this field (note that Llama2-7B is allowed to be used in the CHiME-8 challenge \cite{CHiME})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2110.01857" id="4094">
        <attvalues>
          <attvalue for="5" value=" We conducted experiments, including experimental settings that have not been investigated in previous studies \cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}, and thus, the experimental results and findings obtained in this study are informative for researchers in this field (note that Llama2-7B is allowed to be used in the CHiME-8 challenge \cite{CHiME})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2202.01094" id="4095">
        <attvalues>
          <attvalue for="5" value=" We conducted experiments, including experimental settings that have not been investigated in previous studies \cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}, and thus, the experimental results and findings obtained in this study are informative for researchers in this field (note that Llama2-7B is allowed to be used in the CHiME-8 challenge \cite{CHiME})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2204.00212" id="4096">
        <attvalues>
          <attvalue for="5" value=" We conducted experiments, including experimental settings that have not been investigated in previous studies \cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}, and thus, the experimental results and findings obtained in this study are informative for researchers in this field (note that Llama2-7B is allowed to be used in the CHiME-8 challenge \cite{CHiME})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2306.08133" id="4097">
        <attvalues>
          <attvalue for="5" value=" We conducted experiments, including experimental settings that have not been investigated in previous studies \cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}, and thus, the experimental results and findings obtained in this study are informative for researchers in this field (note that Llama2-7B is allowed to be used in the CHiME-8 challenge \cite{CHiME})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2309.15223" id="4098">
        <attvalues>
          <attvalue for="5" value=" We conducted experiments, including experimental settings that have not been investigated in previous studies \cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}, and thus, the experimental results and findings obtained in this study are informative for researchers in this field (note that Llama2-7B is allowed to be used in the CHiME-8 challenge \cite{CHiME})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2306.16007" id="4099">
        <attvalues>
          <attvalue for="5" value=" We conducted experiments, including experimental settings that have not been investigated in previous studies \cite{Shin_PMLR2019,Li_ICASSP2020,Salazar_ACL2020,Chiu_SLT2021,Fohr_IS2021,Zheng_ASRU2021,Futami_ASRU2021,Xu_ICASSP2022,Udagawa_IS2022,Chen_ICASSP2023,Yu_ASRU2023,Li_ASRU2023,Shivakumar_ASRU2023}, and thus, the experimental results and findings obtained in this study are informative for researchers in this field (note that Llama2-7B is allowed to be used in the CHiME-8 challenge \cite{CHiME})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2306.13734" id="4101">
        <attvalues>
          <attvalue for="5" value=" However, their targets are not casual conversations, and the ability of LLMs to rescore ASR hypotheses of casual conversations remains unclear (note that LLMs are not allowed to be used in the CHiME-7 challenge \cite{Cornell_CHiME2023})." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2308.14638" id="4102">
        <attvalues>
          <attvalue for="5" value=" However, the effect of using LMs is limited (the first-place system does not use any LMs \cite{Wang_CHiME2023}), and there is a demand for LMs to deal with such highly casual conversational speech." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2310.12378" id="4103">
        <attvalues>
          <attvalue for="5" value=" LMs can be expected to play an important role in ASR of such casual conversational speech, and most of the submitted systems try to use LMs during ASR decoding and/or for rescoring ASR hypotheses \cite{Ye_CHiME2023,Kamo_CHiME2023,Prisyach_CHiME2023,Park_CHiME2023}." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="1706.03762" id="4104">
        <attvalues>
          <attvalue for="5" value=" In this study, we reveal it by performing $N$-best ASR hypotheses rescoring using Llama2-7B \cite{Touvron_arXiv2023a}, which is one of the most representative Transformer \cite{Vaswani_NIPS2017} decoder-based causal LLMs, on the CHiME-7 DASR task." />
        </attvalues>
      </edge>
      <edge source="2406.18972" target="2305.14314" id="4105">
        <attvalues>
          <attvalue for="5" value=" We employ QLoRA \cite{Dettmers_NeurIPS2023} for memory efficient domain adaptation and consider various context lengths (up to 1024 tokens) in context carry-over." />
        </attvalues>
      </edge>
      <edge source="2109.14039" target="1607.06520" id="4107">
        <attvalues>
          <attvalue for="5" value=" The first debiasing methods, Hard Debias \cite{Tolga:2016} and Gender Neutral-GloVe \cite{Zhao2018:GNglove}, worked to minimize or eliminate the direct bias, and were shown to be successful in mitigating harmful analogies generated by word embeddings in relation to gender-stereotyped occupations." />
        </attvalues>
      </edge>
      <edge source="2109.14039" target="1809.01496" id="4108">
        <attvalues>
          <attvalue for="5" value=" More commonly used evaluations include the Word Embedding Association Test (WEAT) \cite{Caliskan:2017:weat}, and the analogy generation test SemBias \cite{Zhao2018:GNglove}." />
        </attvalues>
      </edge>
      <edge source="2109.14039" target="1903.03862" id="4109">
        <attvalues>
          <attvalue for="5" value=" An influential critique paper by \cite{Gonen:2019} demonstrated that minimizing direct bias did not eliminate bias in the vector space entirely." />
        </attvalues>
      </edge>
      <edge source="2109.14039" target="2006.01938" id="4110">
        <attvalues>
          <attvalue for="5" value=" The next wave of debiasing methods (2019-present) focused on reducing cluster and recoverability bias while proposing new metrics to systematically quantify the indirect bias of the embedding space (eg the Gender-based Illicit Proximity Estimate, introduced by \cite{Kumar:2020:RAN})." />
        </attvalues>
      </edge>
      <edge source="2109.14039" target="1804.06876" id="4111">
        <attvalues>
          <attvalue for="5" value=" Current gender-bias evaluation tests (GBETs) in widespread use include the WinoBias test set \cite{Zhao:2018:WinoBias}, designed to measure bias in coreference resolution systems using stereotypical occupations as a probe, and the NLI test set \cite{Dev:2020:NLItest}, designed to measure stereotypical inferences again using occupations as the concept of interest." />
        </attvalues>
      </edge>
      <edge source="2109.14039" target="1908.09369" id="4112">
        <attvalues>
          <attvalue for="5" value=" Current gender-bias evaluation tests (GBETs) in widespread use include the WinoBias test set \cite{Zhao:2018:WinoBias}, designed to measure bias in coreference resolution systems using stereotypical occupations as a probe, and the NLI test set \cite{Dev:2020:NLItest}, designed to measure stereotypical inferences again using occupations as the concept of interest." />
        </attvalues>
      </edge>
      <edge source="2109.14039" target="1608.07187" id="4113">
        <attvalues>
          <attvalue for="5" value=" More commonly used evaluations include the Word Embedding Association Test (WEAT) \cite{Caliskan:2017:weat}, and the analogy generation test SemBias \cite{Zhao2018:GNglove}." />
        </attvalues>
      </edge>
      <edge source="2109.14039" target="1906.08976" id="4114">
        <attvalues>
          <attvalue for="5" value=" Adding to the library of downstream GBETs is essential in building a robust understanding of gender bias in NLP applications \cite{Sun:2019:review}." />
        </attvalues>
      </edge>
      <edge source="2210.03871" target="2107.13586" id="4115">
        <attvalues>
          <attvalue for="5" value=" Many recent works have demonstrated the benefits of prompting for large language models (see \cite{prompt_survey} for an extensive survey)." />
        </attvalues>
      </edge>
      <edge source="2210.03871" target="2109.01652" id="4117">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{FLAN} find that models with fewer than 8B parameters see decreases in generalization when utilizing instructions, whereas \cite{instructdial} find consistent gains in models with 3B and fewer parameters." />
        </attvalues>
      </edge>
      <edge source="2210.03871" target="2005.14165" id="4119">
        <attvalues>
          <attvalue for="5" value=" Studies on utilizing prompts have shown that as model sizes scale up, the generalization abilities of a model increase \cite{gpt3, lester-etal-2021-power,demonstrations_icl}." />
        </attvalues>
      </edge>
      <edge source="2210.03871" target="2202.12837" id="4120">
        <attvalues>
          <attvalue for="5" value=" Studies on utilizing prompts have shown that as model sizes scale up, the generalization abilities of a model increase \cite{gpt3, lester-etal-2021-power,demonstrations_icl}." />
        </attvalues>
      </edge>
      <edge source="2210.03871" target="2101.11038" id="4122">
        <attvalues>
          <attvalue for="5" value=" Simultaneously with the emergence of prompting, the explicit multi-task learning (MTL) paradigm emerged, with works such as Muppet \cite{aghajanyan-etal-2021-muppet} or T0 \cite{sanh2022multitask} and their variants." />
        </attvalues>
      </edge>
      <edge source="2210.03871" target="2110.08207" id="4123">
        <attvalues>
          <attvalue for="5" value=" Simultaneously with the emergence of prompting, the explicit multi-task learning (MTL) paradigm emerged, with works such as Muppet \cite{aghajanyan-etal-2021-muppet} or T0 \cite{sanh2022multitask} and their variants." />
        </attvalues>
      </edge>
      <edge source="2404.12195" target="2307.09288" id="4133">
        <attvalues>
          <attvalue for="5" value=" However, it has been demonstrated that models with a comparatively smaller parameter count can perform reasonably well on diverse downstream tasks, even outperforming larger models in specific cases\cite{touvron2023llama} \cite{gunasekar2023textbooks}." />
        </attvalues>
      </edge>
      <edge source="2404.12195" target="2306.11644" id="4134">
        <attvalues>
          <attvalue for="5" value=" However, it has been demonstrated that models with a comparatively smaller parameter count can perform reasonably well on diverse downstream tasks, even outperforming larger models in specific cases\cite{touvron2023llama} \cite{gunasekar2023textbooks}." />
        </attvalues>
      </edge>
      <edge source="2404.12195" target="1909.06951" id="4135">
        <attvalues>
          <attvalue for="5" value=" Scale can be achieved by having an \acrshort{acr_llm} generate completely new instruction datasets \cite{maeng2019alpaca, wang-etal-2023-self-instruct}, but the most capable such models have restrictive licensing, casting uncertainty on the openness of derived models trained on their outputs." />
        </attvalues>
      </edge>
      <edge source="2404.12195" target="2303.08774" id="4136">
        <attvalues>
          <attvalue for="5" value=" We go on to further filter this dataset for higher quality and more diverse generations using a better human proxy model \cite{achiam2023gpt}, and perform \acrshort{acr_sft} on our chosen open base model using \acrshort{acr_qlora}, resulting in three \acrshort{acr_qlora} adapter models." />
        </attvalues>
      </edge>
      <edge source="2404.12195" target="1909.08593" id="4137">
        <attvalues>
          <attvalue for="5" value=" More specifically, to maximize a reward based on human preference using \acrfull{acr_ppo}\cite{ziegler2020finetuning} is now common." />
        </attvalues>
      </edge>
      <edge source="2404.12195" target="2305.18290" id="4138">
        <attvalues>
          <attvalue for="5" value=" We deliberately chose to apply \acrshort{acr_dpo} to the merged model as the update rule of \acrshort{acr_dpo} explicitly refers to the entire parameterized \acrshort{acr_llm}\cite{rafailov2023direct}." />
        </attvalues>
      </edge>
      <edge source="2404.12195" target="2204.05862" id="4139">
        <attvalues>
          <attvalue for="5" value=" In our work, we perform \acrshort{acr_dpo} on a subset of the Anthropic HH-RLHF dataset\cite{bai2022training} after merging the \acrshort{acr_qlora} adapter from the \acrshort{acr_sft} stage." />
        </attvalues>
      </edge>
      <edge source="2404.12195" target="2306.05685" id="4140">
        <attvalues>
          <attvalue for="5" value=" In order to evaluate human preferences alignment, we employ the \acrshort{acr_llm}-as-a-judge framework\cite{zheng2023judging} with the MT-bench benchmark question set." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2005.14165" id="4141">
        <attvalues>
          <attvalue for="5" value=" For prompt selection, our observation holds for 9 LMs ranging over 3 orders of magnitude in size~\cite{radford2019language,brown2020language,sanh2019distilbert} on 3 classification tasks and 41 tasks in the LAMA benchmark~\cite{petroni-etal-2019-language}." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2012.15723" id="4143">
        <attvalues>
          <attvalue for="5" value=" However, the few-shot performance of LMs is very sensitive to the textual task description~\cite[``prompt'';][]{schick2020exploiting,jiang-etal-2020-know,gao2020making,zhao2021calibrate}, order of training examples~\cite{zhao2021calibrate,lu2021fantastically,liu2021what}, decoding strategy~\cite{schick2020small,perez2021rissanen}, and other hyperparameters~\cite{schick2020exploiting,gao2020making,schick2020small,schick2020few,tam2021improving}, as well as the learning algorithm itself~\cite{schick2020exploiting,tam2021improving}." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2102.09690" id="4144">
        <attvalues>
          <attvalue for="5" value=" However, the few-shot performance of LMs is very sensitive to the textual task description~\cite[``prompt'';][]{schick2020exploiting,jiang-etal-2020-know,gao2020making,zhao2021calibrate}, order of training examples~\cite{zhao2021calibrate,lu2021fantastically,liu2021what}, decoding strategy~\cite{schick2020small,perez2021rissanen}, and other hyperparameters~\cite{schick2020exploiting,gao2020making,schick2020small,schick2020few,tam2021improving}, as well as the learning algorithm itself~\cite{schick2020exploiting,tam2021improving}." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2104.08786" id="4145">
        <attvalues>
          <attvalue for="5" value=" However, the few-shot performance of LMs is very sensitive to the textual task description~\cite[``prompt'';][]{schick2020exploiting,jiang-etal-2020-know,gao2020making,zhao2021calibrate}, order of training examples~\cite{zhao2021calibrate,lu2021fantastically,liu2021what}, decoding strategy~\cite{schick2020small,perez2021rissanen}, and other hyperparameters~\cite{schick2020exploiting,gao2020making,schick2020small,schick2020few,tam2021improving}, as well as the learning algorithm itself~\cite{schick2020exploiting,tam2021improving}." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2009.07118" id="4146">
        <attvalues>
          <attvalue for="5" value=" However, the few-shot performance of LMs is very sensitive to the textual task description~\cite[``prompt'';][]{schick2020exploiting,jiang-etal-2020-know,gao2020making,zhao2021calibrate}, order of training examples~\cite{zhao2021calibrate,lu2021fantastically,liu2021what}, decoding strategy~\cite{schick2020small,perez2021rissanen}, and other hyperparameters~\cite{schick2020exploiting,gao2020making,schick2020small,schick2020few,tam2021improving}, as well as the learning algorithm itself~\cite{schick2020exploiting,tam2021improving}." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2103.03872" id="4147">
        <attvalues>
          <attvalue for="5" value=" However, the few-shot performance of LMs is very sensitive to the textual task description~\cite[``prompt'';][]{schick2020exploiting,jiang-etal-2020-know,gao2020making,zhao2021calibrate}, order of training examples~\cite{zhao2021calibrate,lu2021fantastically,liu2021what}, decoding strategy~\cite{schick2020small,perez2021rissanen}, and other hyperparameters~\cite{schick2020exploiting,gao2020making,schick2020small,schick2020few,tam2021improving}, as well as the learning algorithm itself~\cite{schick2020exploiting,tam2021improving}." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2103.11955" id="4149">
        <attvalues>
          <attvalue for="5" value=" For choosing hyperparameters, true few-shot selection causes performance to drop by 2-10\% across 8 tasks for ADAPET~\cite{tam2021improving}, a state-of-the-art few-shot method." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2104.14690" id="4151">
        <attvalues>
          <attvalue for="5" value=" Other work claims to use no validation set for hyperparameter selection~\cite{schick2020exploiting,schick2020few,wang2021entailment} but does not describe how they design other aspects of their learning algorithm (eg, training objectives)." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="2103.08493" id="4152">
        <attvalues>
          <attvalue for="5" value=" In other words, one reason that prompts are so effective~\cite[``worth many examples'';][]{scao2021data} is that they are often tuned using many examples." />
        </attvalues>
      </edge>
      <edge source="2105.11447" target="1909.01066" id="4154">
        <attvalues>
          <attvalue for="5" value=" For prompt selection, our observation holds for 9 LMs ranging over 3 orders of magnitude in size~\cite{radford2019language,brown2020language,sanh2019distilbert} on 3 classification tasks and 41 tasks in the LAMA benchmark~\cite{petroni-etal-2019-language}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1907.11692" id="4157">
        <attvalues>
          <attvalue for="5" value=" T{ransformer}-based pretrained language models (T-PTLMs) like GPT-1 \cite{radford2018improving}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ELECTRA \cite{clark2019electra}, T5 \cite{raffel2019exploring}, ALBERT \cite{lan2019albert}, BART \cite{lewis2020bart} and PEGAUSUS \cite{zhang2020pegasus} have achieved tremendous success in NLP because of their ability to learn universal language representations from large volumes of unlabeled text data and then transfer this knowledge to downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2003.10555" id="4158">
        <attvalues>
          <attvalue for="5" value=" T{ransformer}-based pretrained language models (T-PTLMs) like GPT-1 \cite{radford2018improving}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ELECTRA \cite{clark2019electra}, T5 \cite{raffel2019exploring}, ALBERT \cite{lan2019albert}, BART \cite{lewis2020bart} and PEGAUSUS \cite{zhang2020pegasus} have achieved tremendous success in NLP because of their ability to learn universal language representations from large volumes of unlabeled text data and then transfer this knowledge to downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1909.11942" id="4160">
        <attvalues>
          <attvalue for="5" value=" T{ransformer}-based pretrained language models (T-PTLMs) like GPT-1 \cite{radford2018improving}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ELECTRA \cite{clark2019electra}, T5 \cite{raffel2019exploring}, ALBERT \cite{lan2019albert}, BART \cite{lewis2020bart} and PEGAUSUS \cite{zhang2020pegasus} have achieved tremendous success in NLP because of their ability to learn universal language representations from large volumes of unlabeled text data and then transfer this knowledge to downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1910.13461" id="4161">
        <attvalues>
          <attvalue for="5" value=" T{ransformer}-based pretrained language models (T-PTLMs) like GPT-1 \cite{radford2018improving}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ELECTRA \cite{clark2019electra}, T5 \cite{raffel2019exploring}, ALBERT \cite{lan2019albert}, BART \cite{lewis2020bart} and PEGAUSUS \cite{zhang2020pegasus} have achieved tremendous success in NLP because of their ability to learn universal language representations from large volumes of unlabeled text data and then transfer this knowledge to downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1912.08777" id="4162">
        <attvalues>
          <attvalue for="5" value=" T{ransformer}-based pretrained language models (T-PTLMs) like GPT-1 \cite{radford2018improving}, BERT \cite{devlin2019bert}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ELECTRA \cite{clark2019electra}, T5 \cite{raffel2019exploring}, ALBERT \cite{lan2019albert}, BART \cite{lewis2020bart} and PEGAUSUS \cite{zhang2020pegasus} have achieved tremendous success in NLP because of their ability to learn universal language representations from large volumes of unlabeled text data and then transfer this knowledge to downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1404.2188" id="4164">
        <attvalues>
          <attvalue for="5" value=" The evolution of better computer hardware like GPUs and word embeddings like Word2Vec \cite{mikolov2013efficient} and Glove \cite{pennington2014glove} increased the use of deep learning models like CNN \cite{kalchbrenner2014convolutional} and RNN \cite{liu2016recurrent,zhou2016text} for building NLP systems." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1611.06639" id="4166">
        <attvalues>
          <attvalue for="5" value=" The evolution of better computer hardware like GPUs and word embeddings like Word2Vec \cite{mikolov2013efficient} and Glove \cite{pennington2014glove} increased the use of deep learning models like CNN \cite{kalchbrenner2014convolutional} and RNN \cite{liu2016recurrent,zhou2016text} for building NLP systems." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1409.1556" id="4167">
        <attvalues>
          <attvalue for="5" value=" Based on the idea of transfer learning, researchers in Computer Vision trained large CNN models \cite{simonyan2014very,szegedy2016rethinking,he2016deep,tan2019efficientnet} using large scale labeled datasets like ImageNet \cite{krizhevsky2012imagenet,russakovsky2015imagenet}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1905.11946" id="4170">
        <attvalues>
          <attvalue for="5" value=" Based on the idea of transfer learning, researchers in Computer Vision trained large CNN models \cite{simonyan2014very,szegedy2016rethinking,he2016deep,tan2019efficientnet} using large scale labeled datasets like ImageNet \cite{krizhevsky2012imagenet,russakovsky2015imagenet}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2003.08271" id="4173">
        <attvalues>
          <attvalue for="5" value=" Deep learning models like CNN and RNN have difficulties in modelling long term contexts and learn the word representations with locality bias \cite{qiu2020pre}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1706.03762" id="4174">
        <attvalues>
          <attvalue for="5" value=" Self-attention allows for more parallelization compared to RNNs and can easily model long term contexts as every token attend to all the tokens in the input sequence \cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2001.08361" id="4175">
        <attvalues>
          <attvalue for="5" value=" \cite{kaplan2020scaling} showed that the performance of T-PTLMs can be increased just by increasing the size of the model." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2005.14165" id="4176">
        <attvalues>
          <attvalue for="5" value=" This observation triggered the development of large-scale T-PTLMs like GPT-3 (175B) \cite{brown2020language}, PANGU- (200B) \cite{zeng2021pangu}, GShard (600B) \cite{lepikhin2020gshard} which contain billions of parameters and Switch-Transformers (1." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2006.16668" id="4177">
        <attvalues>
          <attvalue for="5" value=" This observation triggered the development of large-scale T-PTLMs like GPT-3 (175B) \cite{brown2020language}, PANGU- (200B) \cite{zeng2021pangu}, GShard (600B) \cite{lepikhin2020gshard} which contain billions of parameters and Switch-Transformers (1." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2101.03961" id="4178">
        <attvalues>
          <attvalue for="5" value="6T) \cite{fedus2021switch} which contains trillions of parameters." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2006.08097" id="4179">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2010.10386" id="4180">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2010.02559" id="4181">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2102.04664" id="4182">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2002.08155" id="4183">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2009.08366" id="4185">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2105.08645" id="4186">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2004.06871" id="4187">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1903.10676" id="4188">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2105.00377" id="4189">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1904.03323" id="4191">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="2007.15779" id="4192">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2108.05542" target="1906.05474" id="4193">
        <attvalues>
          <attvalue for="5" value=" Following the success of T-PTLMs in general English domain, T-PTLMs are also developed for other domains like Finance \cite{yang2020finbert}, Legal \cite{leivaditi2020benchmark,chalkidis2020legal}, News \cite{gururangan2020don}, Programming \cite{lu2021codexglue,feng2020codebert,ahmad2021unified,guo2020graphcodebert,phan2021cotext}, Dialogue \cite{wu2020tod}, Networking \cite{louis2020netbert}, Academic \cite{liu2021oag,beltagy2019scibert,peng2021mathbert} and Biomedical \cite{lee2020biobert,alsentzer2019publicly,gu2020domain,peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2401.08511" target="1802.05365" id="4200">
        <attvalues>
          <attvalue for="5" value=" PLMs learn not only beneficial information~\cite{peters-etal-2018-deep,devlin-etal-2019-bert,brown2020language,touvron2023llama} but also undesirable social biases such as gender, race, and religous biases that exist in the training data~\cite{sun-etal-2019-mitigating,liang-etal-2020-towards,10." />
        </attvalues>
      </edge>
      <edge source="2401.08511" target="2005.14165" id="4202">
        <attvalues>
          <attvalue for="5" value=" PLMs learn not only beneficial information~\cite{peters-etal-2018-deep,devlin-etal-2019-bert,brown2020language,touvron2023llama} but also undesirable social biases such as gender, race, and religous biases that exist in the training data~\cite{sun-etal-2019-mitigating,liang-etal-2020-towards,10." />
        </attvalues>
      </edge>
      <edge source="2401.08511" target="2307.09288" id="4203">
        <attvalues>
          <attvalue for="5" value=" PLMs learn not only beneficial information~\cite{peters-etal-2018-deep,devlin-etal-2019-bert,brown2020language,touvron2023llama} but also undesirable social biases such as gender, race, and religous biases that exist in the training data~\cite{sun-etal-2019-mitigating,liang-etal-2020-towards,10." />
        </attvalues>
      </edge>
      <edge source="2401.08511" target="2309.09092" id="4206">
        <attvalues>
          <attvalue for="5" value=" Debiasing accompanied by FT suffers substantial performance decline in downstream tasks compared to the original PLM~\cite{meade-etal-2022-empirical,kaneko2023impact,oba2023contextual}." />
        </attvalues>
      </edge>
      <edge source="2401.08511" target="2012.15859" id="4207">
        <attvalues>
          <attvalue for="5" value=" Furthermore, bias evaluations exhibit a weak-level of correlation between pre-trained and FT PLMs~\cite{goldfarb-tarrant-etal-2021-intrinsic,kaneko-etal-2022-debiasing,cao-etal-2022-intrinsic}." />
        </attvalues>
      </edge>
      <edge source="2401.08511" target="2210.02938" id="4208">
        <attvalues>
          <attvalue for="5" value=" Furthermore, bias evaluations exhibit a weak-level of correlation between pre-trained and FT PLMs~\cite{goldfarb-tarrant-etal-2021-intrinsic,kaneko-etal-2022-debiasing,cao-etal-2022-intrinsic}." />
        </attvalues>
      </edge>
      <edge source="2401.08511" target="2203.13928" id="4209">
        <attvalues>
          <attvalue for="5" value=" Furthermore, bias evaluations exhibit a weak-level of correlation between pre-trained and FT PLMs~\cite{goldfarb-tarrant-etal-2021-intrinsic,kaneko-etal-2022-debiasing,cao-etal-2022-intrinsic}." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="1910.12638" id="4210">
        <attvalues>
          <attvalue for="5" value=" It distinguishes itself from conventional cascade approaches~\cite{liu2020mockingjay,liu2021tera} by using a single neural model to directly extract the semantics from speech signals~\cite{ghannay2018end, haghani2018audio,serdyuk2018towards} with the advantages of: (1) jointly optimizing the Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) parts, and (2) mitigating error propagation." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="1802.08395" id="4212">
        <attvalues>
          <attvalue for="5" value=" It distinguishes itself from conventional cascade approaches~\cite{liu2020mockingjay,liu2021tera} by using a single neural model to directly extract the semantics from speech signals~\cite{ghannay2018end, haghani2018audio,serdyuk2018towards} with the advantages of: (1) jointly optimizing the Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) parts, and (2) mitigating error propagation." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="2010.04284" id="4214">
        <attvalues>
          <attvalue for="5" value=" Several attempts to unify both textual and speech modalities can be found in~\cite{huang2020leveraging,agrawal2022tie,muller2021pursuit}." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="1910.09463" id="4215">
        <attvalues>
          <attvalue for="5" value=" To overcome it, transfer learning techniques~\cite{bhosale2019end,Caubriere2019,huang2020leveraging} and artificial augmentation of training data with speech synthesis~\cite{desot2020corpus,lugosch2020using} have been proposed." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="2006.11477" id="4216">
        <attvalues>
          <attvalue for="5" value=" SSL models, which are pre-trained from huge amounts of unlabelled data, have lately become very trendy as they show promising results in a wide range of speech tasks~\cite{baevski2020wav2vec,devlin-etal-2019-bert} when substantially alleviating the need of costly annotated speech data." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="2007.01852" id="4218">
        <attvalues>
          <attvalue for="5" value=" To do so, the authors combined the well-known multilingual frame-level speech representation learning model XLS-R~\cite{xlsr} with the Language Agnostic BERT Sentence Embedding generator LaBSE~\cite{feng2022language}." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="2011.09044" id="4219">
        <attvalues>
          <attvalue for="5" value=" Several attempts to unify both textual and speech modalities can be found in~\cite{huang2020leveraging,agrawal2022tie,muller2021pursuit}." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="2205.08180" id="4220">
        <attvalues>
          <attvalue for="5" value=" Inspired by this new challenge, ~\cite{khurana2022samu} proposed a framework named SAMU-XLSR (Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation) which produces a semantically-aligned multimodal and multilingual sentence-level representation." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="2111.09296" id="4221">
        <attvalues>
          <attvalue for="5" value=" To do so, the authors combined the well-known multilingual frame-level speech representation learning model XLS-R~\cite{xlsr} with the Language Agnostic BERT Sentence Embedding generator LaBSE~\cite{feng2022language}." />
        </attvalues>
      </edge>
      <edge source="2307.01323" target="2210.05291" id="4222">
        <attvalues>
          <attvalue for="5" value=" More interestingly, ~\cite{laperriere2022use} shows that SAMU-XLSR can also be used as a frame-level speech encoder for a challenging end-to-end SLU task when they find that this model might create semantically aware frame-level speech representations." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="1802.08395" id="4223">
        <attvalues>
          <attvalue for="5" value=" To our knowledge, end-to-end neural approaches have been proposed four years ago in order to directly extract the semantics from speech signal, by using a single neural model~\cite{ghannay2018end, haghani2018audio,serdyuk2018towards}, instead of applying a classical cascade approach based on the use of an automatic speech recognition (ASR) system, followed by a natural language understanding processing (NLU) module applied to the automatic transcription~\cite{tur2011spoken}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2002.05955" id="4224">
        <attvalues>
          <attvalue for="5" value=" Since 2018, end-to-end approaches have became very popular in the SLU literature~\cite{desot2019slu,dinarelli2020data,radfar2020end,palogiannidi2020end,poncelet2021low}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2008.10984" id="4225">
        <attvalues>
          <attvalue for="5" value=" Since 2018, end-to-end approaches have became very popular in the SLU literature~\cite{desot2019slu,dinarelli2020data,radfar2020end,palogiannidi2020end,poncelet2021low}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="1910.10599" id="4226">
        <attvalues>
          <attvalue for="5" value=" Since 2018, end-to-end approaches have became very popular in the SLU literature~\cite{desot2019slu,dinarelli2020data,radfar2020end,palogiannidi2020end,poncelet2021low}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2010.04284" id="4228">
        <attvalues>
          <attvalue for="5" value=" Some approaches have been proposed in order to exploit the BERT-like capabilities within an end-to-end SLU model, eg by projecting some kinds of sequences of embeddings extracted by an ASR sub-module to a BERT model~\cite{wang2020large,chung2021splat}, or by tying at the sentence level the acoustic embeddings to a SLU fine-tuned BERT model for a speech intent detection task~\cite{huang2020leveraging,agrawal2022tie}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="1910.09463" id="4229">
        <attvalues>
          <attvalue for="5" value=" Several methods have been proposed in order to address this issue, eg transfer learning techniques~\cite{bhosale2019end,Caubriere2019}, \cite{huang2020leveraging} or artificial augmentation of the training data using speech synthesis~\cite{desot2020corpus,lugosch2020using}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2006.11477" id="4230">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL), that benefits from unlabelled data, recently opened new perspectives for automatic speech recognition and natural language processing~\cite{baevski2020wav2vec,devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="1910.12638" id="4233">
        <attvalues>
          <attvalue for="5" value=" SSL has been successfully applied to several SLU tasks, especially through cascade approaches~\cite{laperriere2021we}: the ASR system benefits from learning better speech unit representations~\cite{liu2020mockingjay,liu2021tera,hsu2021hubert} while the NLU module benefits from BERT-like models~\cite{devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2106.07447" id="4235">
        <attvalues>
          <attvalue for="5" value=" SSL has been successfully applied to several SLU tasks, especially through cascade approaches~\cite{laperriere2021we}: the ASR system benefits from learning better speech unit representations~\cite{liu2020mockingjay,liu2021tera,hsu2021hubert} while the NLU module benefits from BERT-like models~\cite{devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2010.02295" id="4236">
        <attvalues>
          <attvalue for="5" value=" Some approaches have been proposed in order to exploit the BERT-like capabilities within an end-to-end SLU model, eg by projecting some kinds of sequences of embeddings extracted by an ASR sub-module to a BERT model~\cite{wang2020large,chung2021splat}, or by tying at the sentence level the acoustic embeddings to a SLU fine-tuned BERT model for a speech intent detection task~\cite{huang2020leveraging,agrawal2022tie}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2011.09044" id="4237">
        <attvalues>
          <attvalue for="5" value=" Some approaches have been proposed in order to exploit the BERT-like capabilities within an end-to-end SLU model, eg by projecting some kinds of sequences of embeddings extracted by an ASR sub-module to a BERT model~\cite{wang2020large,chung2021splat}, or by tying at the sentence level the acoustic embeddings to a SLU fine-tuned BERT model for a speech intent detection task~\cite{huang2020leveraging,agrawal2022tie}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2205.08180" id="4238">
        <attvalues>
          <attvalue for="5" value=" Earlier this year, a new promising model was introduced in~\cite{khurana2022samu}." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2111.09296" id="4239">
        <attvalues>
          <attvalue for="5" value=" The model combines a state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R~\cite{babu2021xls} with the Language Agnostic BERT Sentence Embedding~\cite{feng2022language} (LaBSE) model to create an utterance-level multimodal multilingual speech encoder." />
        </attvalues>
      </edge>
      <edge source="2210.05291" target="2007.01852" id="4240">
        <attvalues>
          <attvalue for="5" value=" The model combines a state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R~\cite{babu2021xls} with the Language Agnostic BERT Sentence Embedding~\cite{feng2022language} (LaBSE) model to create an utterance-level multimodal multilingual speech encoder." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="2005.14165" id="4241">
        <attvalues>
          <attvalue for="5" value=" We are inspired by recent progress with pretrained large Language Models (LLM), which when prompted with task demonstrations \cite{Brown2020-rl}, instructions \cite{Sanh2021-na, Wei2021-go, Ouyang2022-ti} or reasoning chains \cite{Wei2022-lz}, show an ability to answer questions unlikely to have been encountered during training." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="2110.08207" id="4242">
        <attvalues>
          <attvalue for="5" value=" We are inspired by recent progress with pretrained large Language Models (LLM), which when prompted with task demonstrations \cite{Brown2020-rl}, instructions \cite{Sanh2021-na, Wei2021-go, Ouyang2022-ti} or reasoning chains \cite{Wei2022-lz}, show an ability to answer questions unlikely to have been encountered during training." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="1704.00051" id="4243">
        <attvalues>
          <attvalue for="5" value=" Rather than encoding all knowledge in the parameters of a LLM, an alternative approach has been to transform the original question-answering problem into a reading comprehension (RC) problem by retrieving relevant information for answering a particular query from an external corpus, and training a smaller model (QA Model) to reason over the concatenation of the query and retrieved information to derive an answer eg \cite{Chen2017-gw}." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="2201.05320" id="4245">
        <attvalues>
          <attvalue for="5" value=" Such strategies include ignoring an irrelevant context completely or weighing partially evidential facts; eg reasoning toward answering ``Do teenagers always rebel against their parents?'' \cite{Talmor2021-al} can be aided by the retrieval of knowledge that ``Adolescents who have a good relationship with their parents are less likely to engage in various risk behaviours'', even though there is no entailment implied." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="2108.05885" id="4246">
        <attvalues>
          <attvalue for="5" value=" In regards to defining compositionality, others eg \cite{Dankers2022-bw,Hupkes2020-iu} have noted challenges in singularly defining compositionality as it relates to NLP; for our purposes we pragmatically define a question as compositional if it is unlikely to be answerable by our QA Model with a memorised answer from a similar training example, and requires reasoning over a context by utilising at least one reasoning operation (eg conjunction) involving more than one textual fact, and/or at least one numerical operation involving more than one number." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="2101.02235" id="4248">
        <attvalues>
          <attvalue for="5" value=" This criteria leads us to select six evaluation datasets: StrategyQA \cite{Geva2021-sl} contains commonsense samples requiring diverse multi-hop reasoning strategies." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="2011.07127" id="4249">
        <attvalues>
          <attvalue for="5" value=" IIRC \cite{Ferguson2020-hv} contains questions where an initial paragraph is given and answers depend upon reasoning over this plus one to over four additional paragraphs that must be retrieved." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="2102.03315" id="4250">
        <attvalues>
          <attvalue for="5" value=" ARC-DA \cite{Bhakthavatsalam2021-fq} is a question-only subset of ARC \cite{Clark2018-gy}." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="1803.05457" id="4251">
        <attvalues>
          <attvalue for="5" value=" ARC-DA \cite{Bhakthavatsalam2021-fq} is a question-only subset of ARC \cite{Clark2018-gy}." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="1903.00161" id="4252">
        <attvalues>
          <attvalue for="5" value=" DROP \cite{Dua2019-td} is a RC dataset wherein answering each question requires numerical or temporal reasoning over a provided context to reach an often abstractive answer eg ``How many field goals were scored in the first quarter?." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="1811.00937" id="4253">
        <attvalues>
          <attvalue for="5" value=" CommonsenseQA \cite{Talmor2019-rm} contains samples that are often unlikely to be answerable by finding a singular fact eg ``I’m crossing the river, my feet are wet but my body is dry, where am I? (A) waterfall (B) bridge (C) valley (D) bank (E) island'' is answered by considering knowledge related to each option." />
        </attvalues>
      </edge>
      <edge source="2308.00946" target="1908.07898" id="4254">
        <attvalues>
          <attvalue for="5" value=" In addition to the possibility of answer leakage from directly memorised samples, it has been shown that models are able to utilise more subtle cues such as the writing style of a particular annotator who contributed to both train and test splits for better results than are achievable where the test split is truly independent of the training split \cite{Geva2019-ll}." />
        </attvalues>
      </edge>
      <edge source="1908.07898" target="1508.05326" id="4255">
        <attvalues>
          <attvalue for="5" value=" The prevalent method for creating new datasets is through crowdsourcing, where examples are generated by workers \cite{zaidan2011crowdsourcing,richardson2013mctest,bowman2015large,rajpurkar2016squad,trischler2017newsqa}." />
        </attvalues>
      </edge>
      <edge source="1908.07898" target="1606.05250" id="4256">
        <attvalues>
          <attvalue for="5" value=" The prevalent method for creating new datasets is through crowdsourcing, where examples are generated by workers \cite{zaidan2011crowdsourcing,richardson2013mctest,bowman2015large,rajpurkar2016squad,trischler2017newsqa}." />
        </attvalues>
      </edge>
      <edge source="1908.07898" target="1611.09830" id="4257">
        <attvalues>
          <attvalue for="5" value=" The prevalent method for creating new datasets is through crowdsourcing, where examples are generated by workers \cite{zaidan2011crowdsourcing,richardson2013mctest,bowman2015large,rajpurkar2016squad,trischler2017newsqa}." />
        </attvalues>
      </edge>
      <edge source="1908.07898" target="1803.02324" id="4259">
        <attvalues>
          <attvalue for="5" value=" In this paper, we continue recent efforts to understand biases that are introduced during the process of data creation \cite{levy2015supervised,schwartz2017roc,gururangan2018annotation, glockner2018breaking, poliak2018hypothesis, tsuchiya2018performance,aharoni2018split, paun2018comparing}." />
        </attvalues>
      </edge>
      <edge source="1908.07898" target="1704.05426" id="4260">
        <attvalues>
          <attvalue for="5" value=" We investigate this form of bias, termed annotator bias, and perform multiple experiments over three recent NLU datasets: MNLI \cite{N18-1101}, OpenBookQA." />
        </attvalues>
      </edge>
      <edge source="1908.07898" target="1809.02789" id="4261">
        <attvalues>
          <attvalue for="5" value=" \cite{mihaylovetal2018}, and CommonsenseQA \cite{talmor2019commonsenseqa}." />
        </attvalues>
      </edge>
      <edge source="1908.07898" target="1811.00937" id="4262">
        <attvalues>
          <attvalue for="5" value=" \cite{mihaylovetal2018}, and CommonsenseQA \cite{talmor2019commonsenseqa}." />
        </attvalues>
      </edge>
      <edge source="2402.14533" target="2303.08774" id="4264">
        <attvalues>
          <attvalue for="5" value="5 \cite{ouyang2022training}, GPT-4 \cite{achiam2023gpt} and Bard \cite{bard}, have revolutionized and popularized natural language processing and AI, demonstrating human-like and super-human performance in a wide range of text-based tasks \cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2402.14533" target="2303.18223" id="4265">
        <attvalues>
          <attvalue for="5" value="5 \cite{ouyang2022training}, GPT-4 \cite{achiam2023gpt} and Bard \cite{bard}, have revolutionized and popularized natural language processing and AI, demonstrating human-like and super-human performance in a wide range of text-based tasks \cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2402.14533" target="2303.11156" id="4266">
        <attvalues>
          <attvalue for="5" value=" While the layman may find the responses of LLMs hard to distinguish from human-generated ones \cite{layman_llm,new_intro_7}, a plethora of recent literature has shown that it is possible to successfully discern human-generated text from LLM-generated text using various computational techniques \cite{new_intro_llm_1,new_intro_llm_2,new_intro_llm_3}." />
        </attvalues>
      </edge>
      <edge source="2402.14533" target="2303.07205" id="4267">
        <attvalues>
          <attvalue for="5" value=" While the layman may find the responses of LLMs hard to distinguish from human-generated ones \cite{layman_llm,new_intro_7}, a plethora of recent literature has shown that it is possible to successfully discern human-generated text from LLM-generated text using various computational techniques \cite{new_intro_llm_1,new_intro_llm_2,new_intro_llm_3}." />
        </attvalues>
      </edge>
      <edge source="2402.14533" target="2305.16617" id="4268">
        <attvalues>
          <attvalue for="5" value=" While the layman may find the responses of LLMs hard to distinguish from human-generated ones \cite{layman_llm,new_intro_7}, a plethora of recent literature has shown that it is possible to successfully discern human-generated text from LLM-generated text using various computational techniques \cite{new_intro_llm_1,new_intro_llm_2,new_intro_llm_3}." />
        </attvalues>
      </edge>
      <edge source="2402.14533" target="2307.11729" id="4269">
        <attvalues>
          <attvalue for="5" value=" While the layman may find the responses of LLMs hard to distinguish from human-generated ones \cite{layman_llm,new_intro_7}, a plethora of recent literature has shown that it is possible to successfully discern human-generated text from LLM-generated text using various computational techniques \cite{new_intro_llm_1,new_intro_llm_2,new_intro_llm_3}." />
        </attvalues>
      </edge>
      <edge source="2402.14533" target="2309.03992" id="4270">
        <attvalues>
          <attvalue for="5" value=" Among the developed techniques, the linguistic approach, which focuses on the structure, patterns, and nuances inherent in human language, stands out as a promising option that offers both high statistical performance \cite{herbold2023large} as well as theoretically-grounded explanatory power \cite{munoz2023contrasting}, as opposed to alternative \say{black-box} machine-learning techniques \cite{llm_math_1,llm_math_2}." />
        </attvalues>
      </edge>
      <edge source="2402.14533" target="2306.05540" id="4271">
        <attvalues>
          <attvalue for="5" value=" Among the developed techniques, the linguistic approach, which focuses on the structure, patterns, and nuances inherent in human language, stands out as a promising option that offers both high statistical performance \cite{herbold2023large} as well as theoretically-grounded explanatory power \cite{munoz2023contrasting}, as opposed to alternative \say{black-box} machine-learning techniques \cite{llm_math_1,llm_math_2}." />
        </attvalues>
      </edge>
      <edge source="2305.12900" target="1907.11692" id="4273">
        <attvalues>
          <attvalue for="5" value=" This philosophy of language learning has been applied to transformer language models such as BERT~\cite{kenton2019bert} and RoBERTa~\cite{liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2305.12900" target="1909.01066" id="4274">
        <attvalues>
          <attvalue for="5" value=" Specifically, the service, directly inspired by Petroni et al's fact probing method~\cite{petroni2019language}, could be based on optimal, fine-tuned versions of the language models to discover additional objects for new incoming relations." />
        </attvalues>
      </edge>
      <edge source="2305.12900" target="1606.05250" id="4275">
        <attvalues>
          <attvalue for="5" value=" On the other hand, to obtain a model specifically for the Question Answering (QA) downstream task, as a concrete example and the focus of this work, better versions of the models are obtained when the original models' weights are used to first initialize a task-agnostic model which is then further fine-tuned to obtain a QA task-specific model given instances of the downstream task, eg as defined in the Stanford Question Answering Dataset (SQuAD)~\cite{rajpurkar2016squad,rajpurkar2018know}." />
        </attvalues>
      </edge>
      <edge source="2305.12900" target="2205.04040" id="4277">
        <attvalues>
          <attvalue for="5" value=" Instead, inspired from prior work~\cite{fabbri2020template,schick2021exploiting,zhong2022proqa}, this study uses two strategies: 1) template-based unsupervised generation of structured data similar to SQuAD QA data from the ORKG KB, and 2) structural prompt-based learning over state-of-the-art SQuAD-specific fine-tuned transformer models for the scholarly domain." />
        </attvalues>
      </edge>
      <edge source="2407.12994" target="2307.03109" id="4288">
        <attvalues>
          <attvalue for="5" value=" They have attained unprecedented performance on a wide array of NLP tasks \cite{chang2023survey} because of which they have attracted a lot of interest from academia and different industries including medicine, law, finance and more." />
        </attvalues>
      </edge>
      <edge source="2407.12994" target="2402.07927" id="4289">
        <attvalues>
          <attvalue for="5" value=" \cite{sahoo2024systematic} surveys 29 prompting technique papers based on their applications." />
        </attvalues>
      </edge>
      <edge source="2407.12994" target="2404.06001" id="4290">
        <attvalues>
          <attvalue for="5" value=" \cite{edemacu2024privacy} provides an overview of privacy protection prompting methods and thus focuses on a comparatively small sub-field of prompt engineering." />
        </attvalues>
      </edge>
      <edge source="2407.12994" target="2310.14735" id="4291">
        <attvalues>
          <attvalue for="5" value=" \cite{chen2023unleashing} limits the discussion of prompting strategies to some 9-10 methodologies and also does not incorporate categorizing them based on the NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2005.14165" id="4292">
        <attvalues>
          <attvalue for="5" value=" These models, including LLMs such as the GPT series \cite{brownGPT3, openai2024gpt4technicalreport} by OpenAI, along with many others (eg Gemini \cite{geminiteam2024geminifamilyhighlycapable, google2024gemini} and Gemini (BARD) \cite{hulbert2023bard} by Google, Claude series by Anthropic \cite{anthropic2024claude3, claude3}, and Llama series open-source model from Meta \cite{2023llama2, dubey2024llama3herdmodels}), have revolutionized tasks ranging from information extraction to the creation of engaging content \cite{Sarkhel2023}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2312.11805" id="4293">
        <attvalues>
          <attvalue for="5" value=" These models, including LLMs such as the GPT series \cite{brownGPT3, openai2024gpt4technicalreport} by OpenAI, along with many others (eg Gemini \cite{geminiteam2024geminifamilyhighlycapable, google2024gemini} and Gemini (BARD) \cite{hulbert2023bard} by Google, Claude series by Anthropic \cite{anthropic2024claude3, claude3}, and Llama series open-source model from Meta \cite{2023llama2, dubey2024llama3herdmodels}), have revolutionized tasks ranging from information extraction to the creation of engaging content \cite{Sarkhel2023}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2307.09288" id="4294">
        <attvalues>
          <attvalue for="5" value=" These models, including LLMs such as the GPT series \cite{brownGPT3, openai2024gpt4technicalreport} by OpenAI, along with many others (eg Gemini \cite{geminiteam2024geminifamilyhighlycapable, google2024gemini} and Gemini (BARD) \cite{hulbert2023bard} by Google, Claude series by Anthropic \cite{anthropic2024claude3, claude3}, and Llama series open-source model from Meta \cite{2023llama2, dubey2024llama3herdmodels}), have revolutionized tasks ranging from information extraction to the creation of engaging content \cite{Sarkhel2023}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2102.12092" id="4295">
        <attvalues>
          <attvalue for="5" value=" Early MMLMs include the DALL-E series \cite{ramesh2021zeroshot, marcus2022dalle2, openai2021dalle}, which can generate images from textual descriptions, and CLIP, which can understand and relate text and image data in a unified manner \cite{Radford2021LearningTV, li2022supervision}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2307.10169" id="4297">
        <attvalues>
          <attvalue for="5" value=" In real applications, the prompt is the input of the model, and prompt engineering can result in significant output differences \cite{kaddour2023challenges}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2104.08786" id="4298">
        <attvalues>
          <attvalue for="5" value=" Modifying both the structure (eg, altering length, arrangement of instances) and the content (eg, phrasing, choice of illustrations, directives) of the prompt can exert a notable influence on the model's behavior \cite{lu2022sensitivity, webson2022prompt}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2201.11903" id="4299">
        <attvalues>
          <attvalue for="5" value=" Contemporary prompt engineering encompasses a spectrum of techniques, ranging from foundational approaches such as role-prompting \cite{shanahan2023roleplay} to more sophisticated methods such as chain-of-thought prompting \cite{wei2022chain}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2203.11171" id="4300">
        <attvalues>
          <attvalue for="5" value=" As illustrated in Figure \ref{History_of_the_development_in_prompt_engineering}, the historical progression of prompt engineering showcases significant milestones from the early days of structured inputs in the 1950s to advanced methodologies such as chain-of-thought prompting \cite{wei2022chain} and self-consistency prompting \cite{wang2023selfconsistency} developed in recent years." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2305.16367" id="4301">
        <attvalues>
          <attvalue for="5" value=" Contemporary prompt engineering encompasses a spectrum of techniques, ranging from foundational approaches such as role-prompting \cite{shanahan2023roleplay} to more sophisticated methods such as chain-of-thought prompting \cite{wei2022chain}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2005.00661" id="4302">
        <attvalues>
          <attvalue for="5" value=" Importantly, a well-constructed prompt can counteract challenges such as machine hallucinations \cite{maynez2020faithfulness, bubeck2023sparks}." />
        </attvalues>
      </edge>
      <edge source="2310.14735" target="2307.00855" id="4303">
        <attvalues>
          <attvalue for="5" value=" Section \ref{Sec4} discusses methodologies specific to VLMs, including Context Optimization (CoOp), Conditional Context Optimization (CoCoOp), and Multimodal Prompt Learning (MaPLe), which enhance the performance of VLMs \cite{WANG2023100047}." />
        </attvalues>
      </edge>
      <edge source="2309.13205" target="1910.13461" id="4305">
        <attvalues>
          <attvalue for="5" value=" In recent years, transformer-based language models (such as \cite{t5}, \cite{BART}, \cite{NEURIPS2020_1457c0d6}, \cite{BERT}) have emerged as a transformative force in the field of artificial intelligence, revolutionizing Natural Language Understanding(NLU) and Generation(NLG)." />
        </attvalues>
      </edge>
      <edge source="2309.13205" target="2005.14165" id="4306">
        <attvalues>
          <attvalue for="5" value=" According to \cite{NEURIPS2020_1457c0d6}, in-context learning harnesses the context provided by input data to generate appropriate responses or predictions, contrasting with traditional methods that necessitate explicit task-specific training and fine-tuning on labeled datasets." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2108.07258" id="4318">
        <attvalues>
          <attvalue for="5" value=" One prevailing work is the Foundation Model (FM) \cite{bommasani2021opportunities}, which has brought tremendous interest in the research community." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2005.14165" id="4319">
        <attvalues>
          <attvalue for="5" value=" FMs have demonstrated strong generalization and knowledge transfer capabilities \cite{brown2020language,adiwardana2020towards}, as a result of learning from diverse data sources such as different modalities, multiple languages and various application domains." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2001.09977" id="4320">
        <attvalues>
          <attvalue for="5" value=" FMs have demonstrated strong generalization and knowledge transfer capabilities \cite{brown2020language,adiwardana2020towards}, as a result of learning from diverse data sources such as different modalities, multiple languages and various application domains." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="1808.05312" id="4321">
        <attvalues>
          <attvalue for="5" value=" With supervised multitask learning similar to \cite{raffel2020exploring}, different tasks are unified into a heterogeneous discriminative task and the model is trained jointly on these tasks, such as multi-domain tasks \cite{narayanan2018toward,chan2021speechstew} or multilingual tasks \cite{li2021scaling, li2022massively}." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2106.06909" id="4323">
        <attvalues>
          <attvalue for="5" value=" In the speech community, there have been numerous research studies showing promising results and demonstrating the potential advantages of such models \cite{NarayananMisraSimPundakEtAl18,chan2021speechstew,chen2021gigaspeech,zhang2022bigssl,radford2022robust,hwang2022pseudo,gandhi2022esb}." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2109.13226" id="4324">
        <attvalues>
          <attvalue for="5" value=" Existing work has mainly focused on using supervised in-domain data to jointly train or finetune FMs for target tasks \cite{chan2021speechstew,zhang2022bigssl}." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2006.11477" id="4329">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} and wav2vec-BERT \cite{chung2021w2v} updates the pretrained encoder during finetuning, which, however, is often costly for large FMs." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="1807.03748" id="4331">
        <attvalues>
          <attvalue for="5" value=" With self-supervised training, the models are first trained on audio-only data using contrastive loss \cite{oord2018representation} or reconstruction loss \cite{chorowski2019unsupervised}, to learn good representations of the speech signals." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="1901.08810" id="4332">
        <attvalues>
          <attvalue for="5" value=" With self-supervised training, the models are first trained on audio-only data using contrastive loss \cite{oord2018representation} or reconstruction loss \cite{chorowski2019unsupervised}, to learn good representations of the speech signals." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2110.06280" id="4333">
        <attvalues>
          <attvalue for="5" value=" These models are then directly used as feature extractors for downstream tasks \cite{huang2022s3prl,lin2022analyzing}." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2110.03509" id="4334">
        <attvalues>
          <attvalue for="5" value=" These models are then directly used as feature extractors for downstream tasks \cite{huang2022s3prl,lin2022analyzing}." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="1902.00751" id="4336">
        <attvalues>
          <attvalue for="5" value=" Existing work such as residual adapters \cite{houlsby2019parameter,hwang2022large,biadsy2022scalable}, prompting \cite{he2021towards} and neural reprogramming \cite{yang2021voice2series} have demonstrated such potentials." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2203.12559" id="4337">
        <attvalues>
          <attvalue for="5" value=" Existing work such as residual adapters \cite{houlsby2019parameter,hwang2022large,biadsy2022scalable}, prompting \cite{he2021towards} and neural reprogramming \cite{yang2021voice2series} have demonstrated such potentials." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2110.04366" id="4338">
        <attvalues>
          <attvalue for="5" value=" Existing work such as residual adapters \cite{houlsby2019parameter,hwang2022large,biadsy2022scalable}, prompting \cite{he2021towards} and neural reprogramming \cite{yang2021voice2series} have demonstrated such potentials." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2106.09296" id="4339">
        <attvalues>
          <attvalue for="5" value=" Existing work such as residual adapters \cite{houlsby2019parameter,hwang2022large,biadsy2022scalable}, prompting \cite{he2021towards} and neural reprogramming \cite{yang2021voice2series} have demonstrated such potentials." />
        </attvalues>
      </edge>
      <edge source="2302.01496" target="2108.06209" id="4340">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} and wav2vec-BERT \cite{chung2021w2v} updates the pretrained encoder during finetuning, which, however, is often costly for large FMs." />
        </attvalues>
      </edge>
      <edge source="2110.03509" target="1812.09323" id="4355">
        <attvalues>
          <attvalue for="5" value=" Unsupervised ASR systems learn the cross-modal mapping between speech and text \cite{yeh2018unsupervised, aldarmaki2021unsupervised, Liu2020TowardsUS,liu2018completely,chen2019completely}." />
        </attvalues>
      </edge>
      <edge source="2110.03509" target="1804.00316" id="4356">
        <attvalues>
          <attvalue for="5" value=" In particular, \cite{liu2018completely} and \cite{chen2019completely} propose to learn the mapping by Generative Adversarial Network (GAN) \cite{goodfellow2014generative}." />
        </attvalues>
      </edge>
      <edge source="2110.03509" target="1904.04100" id="4357">
        <attvalues>
          <attvalue for="5" value=" In particular, \cite{liu2018completely} and \cite{chen2019completely} propose to learn the mapping by Generative Adversarial Network (GAN) \cite{goodfellow2014generative}." />
        </attvalues>
      </edge>
      <edge source="2110.03509" target="2203.00667" id="4358">
        <attvalues>
          <attvalue for="5" value=" In particular, \cite{liu2018completely} and \cite{chen2019completely} propose to learn the mapping by Generative Adversarial Network (GAN) \cite{goodfellow2014generative}." />
        </attvalues>
      </edge>
      <edge source="2110.03509" target="2105.11084" id="4359">
        <attvalues>
          <attvalue for="5" value=" Notably, a recent work named Wav2vec-U \cite{baevski2021unsupervised} has shown remarkable performance breakthroughs in unsupervised ASR." />
        </attvalues>
      </edge>
      <edge source="2110.03509" target="1512.02595" id="4360">
        <attvalues>
          <attvalue for="5" value=" The performance is comparable with some supervised methods \cite{amodei2016deep, zhang2020pushing, xu2018neural}." />
        </attvalues>
      </edge>
      <edge source="2311.01949" target="2005.14165" id="4361">
        <attvalues>
          <attvalue for="5" value=" Prompted by demonstrations consisting of a few input-label pairs, LLMs perform well even on unseen tasks \cite{brown2020language, 2023arXivSurvey}." />
        </attvalues>
      </edge>
      <edge source="2311.01949" target="2102.09690" id="4362">
        <attvalues>
          <attvalue for="5" value=" ICL ability strongly depends on selected training examples \cite{liu2021makes,2021arXivCalibrate}, and some methods \cite{gonen2022demystifying, guo2023connecting} are designed to form a high-quality demonstration." />
        </attvalues>
      </edge>
      <edge source="2311.01949" target="2212.04037" id="4363">
        <attvalues>
          <attvalue for="5" value=" ICL ability strongly depends on selected training examples \cite{liu2021makes,2021arXivCalibrate}, and some methods \cite{gonen2022demystifying, guo2023connecting} are designed to form a high-quality demonstration." />
        </attvalues>
      </edge>
      <edge source="2311.01949" target="2202.12837" id="4364">
        <attvalues>
          <attvalue for="5" value=" Following the format of examples in demonstrations, language models can predict the right labels by utilizing prior knowledge acquired from pretraining~\cite{min2022rethinking, Shisg}." />
        </attvalues>
      </edge>
      <edge source="2311.01949" target="2303.03846" id="4365">
        <attvalues>
          <attvalue for="5" value=" As the model scale further increases, LLMs can acquire knowledge directly from input-label mappings in demonstrations \cite{wei2023larger}." />
        </attvalues>
      </edge>
      <edge source="2311.01949" target="2210.03350" id="4366">
        <attvalues>
          <attvalue for="5" value=" Prior works \cite{Self-Ask, DSP} investigate the ability of LLMs to perform compositional reasoning tasks in multi-hop question answering tasks." />
        </attvalues>
      </edge>
      <edge source="2311.01949" target="2201.11903" id="4367">
        <attvalues>
          <attvalue for="5" value=" Hence, we propose a prompt-based knowledge-extracting scheme to extract valuable information from selected examples, leveraging LLMs' reasoning and summarizing ability \cite{2022arXivCoT,2023arXivSummarization}." />
        </attvalues>
      </edge>
      <edge source="2311.01949" target="2301.13848" id="4368">
        <attvalues>
          <attvalue for="5" value=" Hence, we propose a prompt-based knowledge-extracting scheme to extract valuable information from selected examples, leveraging LLMs' reasoning and summarizing ability \cite{2022arXivCoT,2023arXivSummarization}." />
        </attvalues>
      </edge>
      <edge source="2304.04054" target="2011.03020" id="4369">
        <attvalues>
          <attvalue for="5" value=" Intimacy is a significant social aspect of language, which helps to explore existing social norms in various contexts \cite{pei2020quantifying}." />
        </attvalues>
      </edge>
      <edge source="2304.04054" target="2210.01108" id="4370">
        <attvalues>
          <attvalue for="5" value=" This paper describes a system developed for the SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis \cite{pei2022semeval}." />
        </attvalues>
      </edge>
      <edge source="2304.04054" target="2204.10050" id="4371">
        <attvalues>
          <attvalue for="5" value=" Inspired by the recent success of transformer-based models in multilingual tasks \cite{malmasi-etal-2022-semeval,tayyar-madabushi-etal-2022-semeval}, we evaluate the performance of BERT (Bidirectional Encoder Representations from Transformers) \cite{devlin-etal-2019-bert} and its modifications." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1912.06670" id="4373">
        <attvalues>
          <attvalue for="5" value=" We experiment on 20 low-resource languages of the CommonVoice dataset \cite{ardila2020common} to demonstrate the effectiveness of our method." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2012.01687" id="4374">
        <attvalues>
          <attvalue for="5" value=" \cite{winata2020adapt} handles the multilingual ASR by directly copying the weights of the cross-lingual language model to the ASR model's transformer decoder." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2007.03001" id="4375">
        <attvalues>
          <attvalue for="5" value=" Furthermore, many approaches have been introduced for multilingual ASR, such as adapter modules \cite{winata2020adapt,kannan2019large,hou2021meta}, multi-head architecture \cite{pratap2020massively,sercu2016very,dalmia2018sequence}, logit adjustment \cite{winata2020adapt}, language-dependent batching \cite{kannan2019large,sercu2016very}, multi-task training \cite{toshniwal2018multilingual,li2018multi}, language embeddings \cite{toshniwal2018multilingual,li2018multi}, and cross-lingual training \cite{baevski2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1911.00359" id="4376">
        <attvalues>
          <attvalue for="5" value=" In contrast, text models have been successfully developed and widely utilized by improving the performance based on the vast web-crawled corpus \cite{wenzek2020ccnet,chelba2013one,conneau2019unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1312.3005" id="4377">
        <attvalues>
          <attvalue for="5" value=" In contrast, text models have been successfully developed and widely utilized by improving the performance based on the vast web-crawled corpus \cite{wenzek2020ccnet,chelba2013one,conneau2019unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1911.02116" id="4378">
        <attvalues>
          <attvalue for="5" value=" Especially, the rise of end-to-end (E2E) transformer-based models showed numerous successes \cite{vaswani2017attention,raffel2019exploring,brown2020language,devlin2018bert}, such as cross-lingual language models \cite{conneau2019unsupervised,chi2021infoxlm}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1706.03762" id="4379">
        <attvalues>
          <attvalue for="5" value=" Especially, the rise of end-to-end (E2E) transformer-based models showed numerous successes \cite{vaswani2017attention,raffel2019exploring,brown2020language,devlin2018bert}, such as cross-lingual language models \cite{conneau2019unsupervised,chi2021infoxlm}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2005.14165" id="4381">
        <attvalues>
          <attvalue for="5" value=" Especially, the rise of end-to-end (E2E) transformer-based models showed numerous successes \cite{vaswani2017attention,raffel2019exploring,brown2020language,devlin2018bert}, such as cross-lingual language models \cite{conneau2019unsupervised,chi2021infoxlm}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2007.07834" id="4383">
        <attvalues>
          <attvalue for="5" value=" We distill the knowledge of the transformer-based cross-lingual text model, InfoXLM \cite{chi2021infoxlm}, while fine-tuning the transformer-based large-scale ASR model, XLSR-wav2vec 2." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2105.11084" id="4384">
        <attvalues>
          <attvalue for="5" value=" Finally, a simple $n$-gram language model is commonly applied to the final predictions to improve the ASR performance \cite{baevski2021unsupervised,schneider2019wav2vec,conneau2020unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2106.07447" id="4385">
        <attvalues>
          <attvalue for="5" value=" Motivated by this, transformer architecture is becoming increasingly common for E2E ASR models \cite{winata2020adapt,baevski2021unsupervised,hsu2021hubert,sadhu2021wav2vec,baevski2020wav2vec,schneider2019wav2vec,baevski2019vq}, where its generalization performance is still limited compared to that of text models." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2103.08393" id="4386">
        <attvalues>
          <attvalue for="5" value=" Motivated by this, transformer architecture is becoming increasingly common for E2E ASR models \cite{winata2020adapt,baevski2021unsupervised,hsu2021hubert,sadhu2021wav2vec,baevski2020wav2vec,schneider2019wav2vec,baevski2019vq}, where its generalization performance is still limited compared to that of text models." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2006.11477" id="4387">
        <attvalues>
          <attvalue for="5" value=" Motivated by this, transformer architecture is becoming increasingly common for E2E ASR models \cite{winata2020adapt,baevski2021unsupervised,hsu2021hubert,sadhu2021wav2vec,baevski2020wav2vec,schneider2019wav2vec,baevski2019vq}, where its generalization performance is still limited compared to that of text models." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1904.05862" id="4388">
        <attvalues>
          <attvalue for="5" value=" Finally, a simple $n$-gram language model is commonly applied to the final predictions to improve the ASR performance \cite{baevski2021unsupervised,schneider2019wav2vec,conneau2020unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1910.05453" id="4389">
        <attvalues>
          <attvalue for="5" value=" Motivated by this, transformer architecture is becoming increasingly common for E2E ASR models \cite{winata2020adapt,baevski2021unsupervised,hsu2021hubert,sadhu2021wav2vec,baevski2020wav2vec,schneider2019wav2vec,baevski2019vq}, where its generalization performance is still limited compared to that of text models." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1909.05330" id="4390">
        <attvalues>
          <attvalue for="5" value=" Furthermore, many approaches have been introduced for multilingual ASR, such as adapter modules \cite{winata2020adapt,kannan2019large,hou2021meta}, multi-head architecture \cite{pratap2020massively,sercu2016very,dalmia2018sequence}, logit adjustment \cite{winata2020adapt}, language-dependent batching \cite{kannan2019large,sercu2016very}, multi-task training \cite{toshniwal2018multilingual,li2018multi}, language embeddings \cite{toshniwal2018multilingual,li2018multi}, and cross-lingual training \cite{baevski2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1509.08967" id="4391">
        <attvalues>
          <attvalue for="5" value=" Furthermore, many approaches have been introduced for multilingual ASR, such as adapter modules \cite{winata2020adapt,kannan2019large,hou2021meta}, multi-head architecture \cite{pratap2020massively,sercu2016very,dalmia2018sequence}, logit adjustment \cite{winata2020adapt}, language-dependent batching \cite{kannan2019large,sercu2016very}, multi-task training \cite{toshniwal2018multilingual,li2018multi}, language embeddings \cite{toshniwal2018multilingual,li2018multi}, and cross-lingual training \cite{baevski2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1802.07420" id="4392">
        <attvalues>
          <attvalue for="5" value=" Furthermore, many approaches have been introduced for multilingual ASR, such as adapter modules \cite{winata2020adapt,kannan2019large,hou2021meta}, multi-head architecture \cite{pratap2020massively,sercu2016very,dalmia2018sequence}, logit adjustment \cite{winata2020adapt}, language-dependent batching \cite{kannan2019large,sercu2016very}, multi-task training \cite{toshniwal2018multilingual,li2018multi}, language embeddings \cite{toshniwal2018multilingual,li2018multi}, and cross-lingual training \cite{baevski2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1711.01694" id="4393">
        <attvalues>
          <attvalue for="5" value=" Furthermore, many approaches have been introduced for multilingual ASR, such as adapter modules \cite{winata2020adapt,kannan2019large,hou2021meta}, multi-head architecture \cite{pratap2020massively,sercu2016very,dalmia2018sequence}, logit adjustment \cite{winata2020adapt}, language-dependent batching \cite{kannan2019large,sercu2016very}, multi-task training \cite{toshniwal2018multilingual,li2018multi}, language embeddings \cite{toshniwal2018multilingual,li2018multi}, and cross-lingual training \cite{baevski2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1712.01541" id="4394">
        <attvalues>
          <attvalue for="5" value=" Furthermore, many approaches have been introduced for multilingual ASR, such as adapter modules \cite{winata2020adapt,kannan2019large,hou2021meta}, multi-head architecture \cite{pratap2020massively,sercu2016very,dalmia2018sequence}, logit adjustment \cite{winata2020adapt}, language-dependent batching \cite{kannan2019large,sercu2016very}, multi-task training \cite{toshniwal2018multilingual,li2018multi}, language embeddings \cite{toshniwal2018multilingual,li2018multi}, and cross-lingual training \cite{baevski2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1503.02531" id="4395">
        <attvalues>
          <attvalue for="5" value=" For the classification models, it is common to distill by minimizing the distance between two output logits of the teacher and student networks \cite{hinton2015distilling,cho2020speech,sanh2019distilbert}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2005.08213" id="4396">
        <attvalues>
          <attvalue for="5" value=" \cite{cho2020speech} distills between text and speech models by minimizing the distance between the classification probabilities, whereas \cite{kim2021two} minimizes the distances between the first hidden representations or the output logits." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1904.08075" id="4397">
        <attvalues>
          <attvalue for="5" value=" Hence, it naturally raises the following intuition: Can we utilize the well-trained cross-lingual language models to increase the performance of multilingual ASR models on low-resource languages? One of the main approaches to using one model to help the other is knowledge distillation (KD) \cite{hinton2015distilling,cho2020speech,liu2019end,jiao2019tinybert,sanh2019distilbert,choi2021temporal,kim2021two}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1909.10351" id="4398">
        <attvalues>
          <attvalue for="5" value=" We use various methods to mitigate the inherent discrepancy between the two modalities, such as the shrink method \cite{chen2016phone}, nearest-neighbor interpolation \cite{itseez2014theopencv}, and a learnable linear transformation layer \cite{jiao2019tinybert,chung2018unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2110.14131" id="4400">
        <attvalues>
          <attvalue for="5" value=" \cite{choi2021temporal} distills the attention maps of a transformer-based audio model to smaller architectures such as convolutional or recurrent neural networks." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2010.13105" id="4401">
        <attvalues>
          <attvalue for="5" value=" \cite{cho2020speech} distills between text and speech models by minimizing the distance between the classification probabilities, whereas \cite{kim2021two} minimizes the distances between the first hidden representations or the output logits." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1805.07467" id="4402">
        <attvalues>
          <attvalue for="5" value=" We use various methods to mitigate the inherent discrepancy between the two modalities, such as the shrink method \cite{chen2016phone}, nearest-neighbor interpolation \cite{itseez2014theopencv}, and a learnable linear transformation layer \cite{jiao2019tinybert,chung2018unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="2010.14920" id="4404">
        <attvalues>
          <attvalue for="5" value=" \cite{liu2020bridging} propose a model that contains speech and text models as its components, where it concentrates on solving the speech-to-text translation task." />
        </attvalues>
      </edge>
      <edge source="2206.12638" target="1910.03320" id="4405">
        <attvalues>
          <attvalue for="5" value=" \cite{di2019one} inputs speech and text to the transformer encoder and decoder, respectively, improving the performance of speech translation task." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1409.3215" id="4406">
        <attvalues>
          <attvalue for="5" value=" The state-of-the art results obtained by encoder-decoder models \cite{sutskever2014sequence} with sequence-to-sequence learning in fields like ASR \cite{amodei2016deep,chan2016listen,chiu2017state,zeyer2018improved} and, most importantly, machine translation \cite{vaswani2017attention,bojar2018findings}, have led to the recent proposal of sequence-to-sequence learning for direct speech-to-text translation \cite{berard2016listen,bansal2017towards}, that is translating from audio without an intermediate output representation." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1512.02595" id="4407">
        <attvalues>
          <attvalue for="5" value=" The state-of-the art results obtained by encoder-decoder models \cite{sutskever2014sequence} with sequence-to-sequence learning in fields like ASR \cite{amodei2016deep,chan2016listen,chiu2017state,zeyer2018improved} and, most importantly, machine translation \cite{vaswani2017attention,bojar2018findings}, have led to the recent proposal of sequence-to-sequence learning for direct speech-to-text translation \cite{berard2016listen,bansal2017towards}, that is translating from audio without an intermediate output representation." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1712.01769" id="4408">
        <attvalues>
          <attvalue for="5" value=" Indeed, while state-of-the-art sequence-to-sequence systems for ASR and MT are respectively trained on thousands of hours of transcribed speech \cite{Chiu18} and tens of millions of parallel sentences \cite{DBLP:journals/corr/abs-1803-05567}, the largest publicly available SLT corpus comprises about 500 hours of translated speech and few others amount to less than 300 hours each \cite{mustc19}." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1805.03294" id="4409">
        <attvalues>
          <attvalue for="5" value=" The state-of-the art results obtained by encoder-decoder models \cite{sutskever2014sequence} with sequence-to-sequence learning in fields like ASR \cite{amodei2016deep,chan2016listen,chiu2017state,zeyer2018improved} and, most importantly, machine translation \cite{vaswani2017attention,bojar2018findings}, have led to the recent proposal of sequence-to-sequence learning for direct speech-to-text translation \cite{berard2016listen,bansal2017towards}, that is translating from audio without an intermediate output representation." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1706.03762" id="4410">
        <attvalues>
          <attvalue for="5" value=" Although this approach has been proposed for RNN-based NMT, it works even better \cite{lakew2018comparison} when using the Transformer \cite{vaswani2017attention} architecture." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1612.01744" id="4411">
        <attvalues>
          <attvalue for="5" value=" The state-of-the art results obtained by encoder-decoder models \cite{sutskever2014sequence} with sequence-to-sequence learning in fields like ASR \cite{amodei2016deep,chan2016listen,chiu2017state,zeyer2018improved} and, most importantly, machine translation \cite{vaswani2017attention,bojar2018findings}, have led to the recent proposal of sequence-to-sequence learning for direct speech-to-text translation \cite{berard2016listen,bansal2017towards}, that is translating from audio without an intermediate output representation." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1706.03872" id="4413">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, end-to-end models require large amounts of parallel training data \cite{koehn2017six} that are not yet available for the SLT task." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1803.05567" id="4414">
        <attvalues>
          <attvalue for="5" value=" Indeed, while state-of-the-art sequence-to-sequence systems for ASR and MT are respectively trained on thousands of hours of transcribed speech \cite{Chiu18} and tens of millions of parallel sentences \cite{DBLP:journals/corr/abs-1803-05567}, the largest publicly available SLT corpus comprises about 500 hours of translated speech and few others amount to less than 300 hours each \cite{mustc19}." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1703.08581" id="4415">
        <attvalues>
          <attvalue for="5" value=" Multitask learning or transfer learning are generally used to exploit ASR data \cite{weiss2017sequence,bansal2018pre,berard2018end,anastasopoulos2018tied} with positive results, and the improvements are more evident when less training data for SLT are available." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1802.04200" id="4417">
        <attvalues>
          <attvalue for="5" value=" Multitask learning or transfer learning are generally used to exploit ASR data \cite{weiss2017sequence,bansal2018pre,berard2018end,anastasopoulos2018tied} with positive results, and the improvements are more evident when less training data for SLT are available." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1802.06655" id="4418">
        <attvalues>
          <attvalue for="5" value=" Multitask learning or transfer learning are generally used to exploit ASR data \cite{weiss2017sequence,bansal2018pre,berard2018end,anastasopoulos2018tied} with positive results, and the improvements are more evident when less training data for SLT are available." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1811.02050" id="4419">
        <attvalues>
          <attvalue for="5" value=" Other approaches to overcome the low-resource condition are data augmentation \cite{jia2018leveraging} and knowledge distillation \cite{liu2019end}." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1904.08075" id="4420">
        <attvalues>
          <attvalue for="5" value=" Other approaches to overcome the low-resource condition are data augmentation \cite{jia2018leveraging} and knowledge distillation \cite{liu2019end}." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1904.07209" id="4421">
        <attvalues>
          <attvalue for="5" value=" {However, } \cite{sperber2019attention} showed that current direct models are not data-efficient in leveraging non-SLT data, {and {that the classic} ``cascade'' approach ({eg a pipelined architecture integrating ASR and MT) still performs better}}." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1601.00710" id="4422">
        <attvalues>
          <attvalue for="5" value=" Multilinguality has been widely explored in neural MT \cite{zoph2016multi,dong2015multi,luong2015multi,firat2016zero,firat2016multi,lu2018neural}, where it is now commonly performed using the target forcing mechanism \cite{hatoward,johnson2017google}, which enables translation to many languages (\{one,many\}-to-many) without changing the underlying NMT architecture." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1511.06114" id="4423">
        <attvalues>
          <attvalue for="5" value=" Multilinguality has been widely explored in neural MT \cite{zoph2016multi,dong2015multi,luong2015multi,firat2016zero,firat2016multi,lu2018neural}, where it is now commonly performed using the target forcing mechanism \cite{hatoward,johnson2017google}, which enables translation to many languages (\{one,many\}-to-many) without changing the underlying NMT architecture." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1606.04164" id="4424">
        <attvalues>
          <attvalue for="5" value=" Multilinguality has been widely explored in neural MT \cite{zoph2016multi,dong2015multi,luong2015multi,firat2016zero,firat2016multi,lu2018neural}, where it is now commonly performed using the target forcing mechanism \cite{hatoward,johnson2017google}, which enables translation to many languages (\{one,many\}-to-many) without changing the underlying NMT architecture." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1601.01073" id="4425">
        <attvalues>
          <attvalue for="5" value=" Multilinguality has been widely explored in neural MT \cite{zoph2016multi,dong2015multi,luong2015multi,firat2016zero,firat2016multi,lu2018neural}, where it is now commonly performed using the target forcing mechanism \cite{hatoward,johnson2017google}, which enables translation to many languages (\{one,many\}-to-many) without changing the underlying NMT architecture." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1804.08198" id="4426">
        <attvalues>
          <attvalue for="5" value=" Multilinguality has been widely explored in neural MT \cite{zoph2016multi,dong2015multi,luong2015multi,firat2016zero,firat2016multi,lu2018neural}, where it is now commonly performed using the target forcing mechanism \cite{hatoward,johnson2017google}, which enables translation to many languages (\{one,many\}-to-many) without changing the underlying NMT architecture." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1611.04798" id="4427">
        <attvalues>
          <attvalue for="5" value=" Multilinguality has been widely explored in neural MT \cite{zoph2016multi,dong2015multi,luong2015multi,firat2016zero,firat2016multi,lu2018neural}, where it is now commonly performed using the target forcing mechanism \cite{hatoward,johnson2017google}, which enables translation to many languages (\{one,many\}-to-many) without changing the underlying NMT architecture." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1806.06957" id="4428">
        <attvalues>
          <attvalue for="5" value=" Although this approach has been proposed for RNN-based NMT, it works even better \cite{lakew2018comparison} when using the Transformer \cite{vaswani2017attention} architecture." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1711.01694" id="4429">
        <attvalues>
          <attvalue for="5" value=" Target forcing has also been applied to multilingual speech recognition \cite{toshniwal2018multilingual,zhou2018multilingual} showing to improve the transcription quality, although multilingual ASR shows to be better than its monolingual counterparts even when the language token is not provided." />
        </attvalues>
      </edge>
      <edge source="1910.03320" target="1806.05059" id="4430">
        <attvalues>
          <attvalue for="5" value=" Target forcing has also been applied to multilingual speech recognition \cite{toshniwal2018multilingual,zhou2018multilingual} showing to improve the transcription quality, although multilingual ASR shows to be better than its monolingual counterparts even when the language token is not provided." />
        </attvalues>
      </edge>
      <edge source="2102.01672" target="1904.02792" id="4431">
        <attvalues>
          <attvalue for="5" value=" These optimization objectives can often be conflicting~\cite{hashimoto2019unifying} and, as a result, evaluations that focus only on a single aspect may fail to recognize the drawbacks of a particular method." />
        </attvalues>
      </edge>
      <edge source="2102.01672" target="1506.03340" id="4432">
        <attvalues>
          <attvalue for="5" value=" To demonstrate this trade-off, consider an improvement on the CNN-DM summarization dataset~\cite{hermann2015teaching,nallapati2016abstractive} measured by the ROUGE-L metric~\cite{lin2004rouge}." />
        </attvalues>
      </edge>
      <edge source="2102.01672" target="1602.06023" id="4433">
        <attvalues>
          <attvalue for="5" value=" To demonstrate this trade-off, consider an improvement on the CNN-DM summarization dataset~\cite{hermann2015teaching,nallapati2016abstractive} measured by the ROUGE-L metric~\cite{lin2004rouge}." />
        </attvalues>
      </edge>
      <edge source="2102.01672" target="2005.00661" id="4434">
        <attvalues>
          <attvalue for="5" value=" Since ROUGE only tests the extent to which a generated summary has a lexical overlap with a reference summary, it can erroneously produce high scores for fluent, yet meaningless and unfaithful outputs as long as many of the same words are used~\cite{maynez2020faithfulness,gabriel2020go}." />
        </attvalues>
      </edge>
      <edge source="2102.01672" target="2005.00955" id="4436">
        <attvalues>
          <attvalue for="5" value=" To avoid the fallacy of encouraging hill climbing on a leaderboard~\cite{linzen2020accelerate}, \GEM focuses on an in-depth evaluation of model outputs across human and automatic evaluation that aims to uncover shortcomings and opportunities for progress." />
        </attvalues>
      </edge>
      <edge source="2102.01672" target="1703.09902" id="4437">
        <attvalues>
          <attvalue for="5" value=" They measure specific generation challenges, such as the content selection and planning (What to say?), and the surface realization (How to say it?)~\cite{reiter2000building,gatt2018survey}." />
        </attvalues>
      </edge>
      <edge source="2102.01672" target="1705.03802" id="4438">
        <attvalues>
          <attvalue for="5" value=" To be able to properly assess the performance of models in a way robust to the shortcuts a model can take, we additionally introduce ten types of challenging test sets that probe for specific modeling aspects~\cite{perez2017analysing,ribeiro-etal-2020-beyond}." />
        </attvalues>
      </edge>
      <edge source="2404.05971" target="1706.03762" id="4442">
        <attvalues>
          <attvalue for="5" value=" The transformer architecture \cite{vaswani2017attention} has all but replaced the recurrent neural network (RNN) in natural language processing in recent years due to its impressive ability to handle long-distance dependencies and its parallelizable training across the time dimension." />
        </attvalues>
      </edge>
      <edge source="2404.05971" target="2312.00752" id="4443">
        <attvalues>
          <attvalue for="5" value=" Mamba \cite{gu2023mamba} and RWKV \cite{peng2023rwkv} are RNNs that allow for parallelized training across the time dimension by restricting the underlying recurrence relation to be associative \cite{martin2017parallelizing, blelloch1990prefix}." />
        </attvalues>
      </edge>
      <edge source="2404.05971" target="2305.13048" id="4444">
        <attvalues>
          <attvalue for="5" value=" Mamba \cite{gu2023mamba} and RWKV \cite{peng2023rwkv} are RNNs that allow for parallelized training across the time dimension by restricting the underlying recurrence relation to be associative \cite{martin2017parallelizing, blelloch1990prefix}." />
        </attvalues>
      </edge>
      <edge source="2404.05971" target="1709.04057" id="4445">
        <attvalues>
          <attvalue for="5" value=" Mamba \cite{gu2023mamba} and RWKV \cite{peng2023rwkv} are RNNs that allow for parallelized training across the time dimension by restricting the underlying recurrence relation to be associative \cite{martin2017parallelizing, blelloch1990prefix}." />
        </attvalues>
      </edge>
      <edge source="2404.05971" target="2312.06681" id="4446">
        <attvalues>
          <attvalue for="5" value=" In particular, we reproduce the following findings from the transformer interpretability literature: \begin{enumerate} \item Contrastive activation addition (CAA): \cite{rimsky2023steering} find that transformer LMs can be controlled using ``steering vectors,'' computed by averaging the difference in residual stream activations between pairs of positive and negative examples of a particular behavior, such as factual versus hallucinatory responses." />
        </attvalues>
      </edge>
      <edge source="2404.05971" target="2303.08112" id="4447">
        <attvalues>
          <attvalue for="5" value=" \item The tuned lens: \cite{belrose2023eliciting} find that interpretable next-token predictions can be elicited from intermediate layers of a transformer using linear probes, and that the accuracy of these predictions increases monotonically with depth." />
        </attvalues>
      </edge>
      <edge source="2404.05971" target="2312.01037" id="4448">
        <attvalues>
          <attvalue for="5" value=" \item ``Quirky'' models: \cite{mallen2023eliciting} find that simple probing methods can elicit a transformer's knowledge of the correct answer to a question, even when it has been fine-tuned to output an incorrect answer." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="1708.02709" id="4453">
        <attvalues>
          <attvalue for="5" value=" Language models are at the very heart of many modern NLP systems and applications \cite{young2018recent}." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="1802.05365" id="4454">
        <attvalues>
          <attvalue for="5" value=" Recent approaches have addressed the first limitation by learning word representations that are contextualized by their token-specific usage context \cite{peters2018deep,devlin-etal-2019-bert,liu2019roberta,yang2019xlnet,yang2019context}." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="1508.07544" id="4455">
        <attvalues>
          <attvalue for="5" value=" While this simplifying assumption has undoubtedly encouraged remarkable progress in modeling language, there is overwhelming evidence in socio-linguistics that language understanding is influenced by the social context in which language is grounded \cite{nguyen2016computational,hovy2018social, mishra2018detecting, garten2019incorporating, flek-2020-returning, bender-koller-2020-climbing}." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="1906.07337" id="4457">
        <attvalues>
          <attvalue for="5" value=" When NLP applications ignore this social context, they may perform sub-optimally underscoring the need for a richer integration of social contexts into NLP models \cite{pavalanathan2015confounds,lynn2017human, zamani-etal-2018-residualized, lynn2019tweet, may-etal-2019-measuring,kurita-etal-2019-measuring, welch-etal-2020-compositional, hovy2021importance}." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="1411.3315" id="4458">
        <attvalues>
          <attvalue for="5" value=" These methods learn word embeddings for each specific social context and can capture how word meanings vary across these dimensions \cite{bamman2014distributed,kulkarni2015statistically, hamilton2016diachronic,welch-etal-2020-compositional,welch-etal-2020-exploring}." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="2010.02986" id="4459">
        <attvalues>
          <attvalue for="5" value=" These methods learn word embeddings for each specific social context and can capture how word meanings vary across these dimensions \cite{bamman2014distributed,kulkarni2015statistically, hamilton2016diachronic,welch-etal-2020-compositional,welch-etal-2020-exploring}." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="2011.06057" id="4460">
        <attvalues>
          <attvalue for="5" value=" These methods learn word embeddings for each specific social context and can capture how word meanings vary across these dimensions \cite{bamman2014distributed,kulkarni2015statistically, hamilton2016diachronic,welch-etal-2020-compositional,welch-etal-2020-exploring}." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="1907.11692" id="4462">
        <attvalues>
          <attvalue for="5" value=" Recent approaches have addressed the first limitation by learning word representations that are contextualized by their token-specific usage context \cite{peters2018deep,devlin-etal-2019-bert,liu2019roberta,yang2019xlnet,yang2019context}." />
        </attvalues>
      </edge>
      <edge source="2110.10319" target="1902.05766" id="4464">
        <attvalues>
          <attvalue for="5" value=" Recent approaches have addressed the first limitation by learning word representations that are contextualized by their token-specific usage context \cite{peters2018deep,devlin-etal-2019-bert,liu2019roberta,yang2019xlnet,yang2019context}." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2303.18223" id="4465">
        <attvalues>
          <attvalue for="5" value=" The field of natural language processing (NLP) has been significantly transformed by the introduction of large language models (LLMs), which have enhanced our understanding and interaction with human language~\cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2303.00980" id="4466">
        <attvalues>
          <attvalue for="5" value=" These advancements bring challenges such as the increased need to train ever larger models~\cite{rae2021scaling,wang2023learning,pan2023reusing,winglian,yao20232x,gesmundo2023composable} owing to the performance scaling law~\cite{kaplan2020scaling, hernandez2021scaling, anil2023palm,kaddour2023no}." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2310.10699" id="4467">
        <attvalues>
          <attvalue for="5" value=" These advancements bring challenges such as the increased need to train ever larger models~\cite{rae2021scaling,wang2023learning,pan2023reusing,winglian,yao20232x,gesmundo2023composable} owing to the performance scaling law~\cite{kaplan2020scaling, hernandez2021scaling, anil2023palm,kaddour2023no}." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2308.06103" id="4468">
        <attvalues>
          <attvalue for="5" value=" These advancements bring challenges such as the increased need to train ever larger models~\cite{rae2021scaling,wang2023learning,pan2023reusing,winglian,yao20232x,gesmundo2023composable} owing to the performance scaling law~\cite{kaplan2020scaling, hernandez2021scaling, anil2023palm,kaddour2023no}." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2001.08361" id="4469">
        <attvalues>
          <attvalue for="5" value=" These advancements bring challenges such as the increased need to train ever larger models~\cite{rae2021scaling,wang2023learning,pan2023reusing,winglian,yao20232x,gesmundo2023composable} owing to the performance scaling law~\cite{kaplan2020scaling, hernandez2021scaling, anil2023palm,kaddour2023no}." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="1701.06538" id="4471">
        <attvalues>
          <attvalue for="5" value=" To efficiently tackle the above, recent works in scaling language models such as a mixture of experts (MoE)~\cite{shazeer2017outrageously,komatsuzaki2022sparse} have been proposed." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2212.05055" id="4472">
        <attvalues>
          <attvalue for="5" value=" Unlike \cite{komatsuzaki2022sparse}, DUS does not scale the model using MoE and rather use a depthwise scaling method analogous to ~\cite{tan2019efficientnet} which is adapted for the LLM architecture." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2211.15841" id="4473">
        <attvalues>
          <attvalue for="5" value=" While those approaches are able to efficiently and effectively scale-up LLMs, they often require non-trivial changes to the training and inference framework~\cite{gale2023megablocks}, which hinders widespread applicability." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2305.00237" id="4474">
        <attvalues>
          <attvalue for="5" value=" Effectively and efficiently scaling up LLMs whilst also retaining the simplicity for ease of use is an important problem~\cite{alberts2023large,fraiwan2023review,sallam2023chatgpt,bahrini2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2304.09103" id="4475">
        <attvalues>
          <attvalue for="5" value=" Effectively and efficiently scaling up LLMs whilst also retaining the simplicity for ease of use is an important problem~\cite{alberts2023large,fraiwan2023review,sallam2023chatgpt,bahrini2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="1905.11946" id="4476">
        <attvalues>
          <attvalue for="5" value=" Unlike \cite{komatsuzaki2022sparse}, DUS does not scale the model using MoE and rather use a depthwise scaling method analogous to ~\cite{tan2019efficientnet} which is adapted for the LLM architecture." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="1910.03771" id="4477">
        <attvalues>
          <attvalue for="5" value=" Thus, there are no additional modules or dynamism as with MoE, making DUS immediately compatible with easy-to-use LLM frameworks such as HuggingFace~\cite{wolf2019huggingface} with no changes to the training or inference framework for maximal efficiency." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2307.09288" id="4478">
        <attvalues>
          <attvalue for="5" value="7 billion parameters, that outperforms existing models like Llama 2~\cite{touvron2023llama2} and Mistral 7B~\cite{jiang2023mistral} in various benchmarks." />
        </attvalues>
      </edge>
      <edge source="2312.15166" target="2310.06825" id="4479">
        <attvalues>
          <attvalue for="5" value="7 billion parameters, that outperforms existing models like Llama 2~\cite{touvron2023llama2} and Mistral 7B~\cite{jiang2023mistral} in various benchmarks." />
        </attvalues>
      </edge>
      <edge source="2212.02475" target="1907.09720" id="4481">
        <attvalues>
          <attvalue for="5" value=" Fast weights have proven successful for supervised \cite{ba2016using}, reinforcement \cite{munkhdalai2019metalearned}, and few-shot \cite{munkhdalai2017meta} learning." />
        </attvalues>
      </edge>
      <edge source="2212.02475" target="1703.00837" id="4482">
        <attvalues>
          <attvalue for="5" value=" Fast weights have proven successful for supervised \cite{ba2016using}, reinforcement \cite{munkhdalai2019metalearned}, and few-shot \cite{munkhdalai2017meta} learning." />
        </attvalues>
      </edge>
      <edge source="2212.02475" target="1703.03400" id="4484">
        <attvalues>
          <attvalue for="5" value=" Training FWLs can be viewed as applying gradient-based meta-learning \cite{finn2017model} to language modeling, where the support set contains tokens seen so far and the query set contains future tokens, a perspective which helps explain some of the behaviors of FWLs." />
        </attvalues>
      </edge>
      <edge source="2212.02475" target="1904.10509" id="4485">
        <attvalues>
          <attvalue for="5" value=" FWLs scale well to long sequences and are complementary to existing long-text generation methods such as sparse attention \cite{child2019generating} or recurrent processing \cite{dai2019transformer}." />
        </attvalues>
      </edge>
      <edge source="2212.02475" target="1901.02860" id="4486">
        <attvalues>
          <attvalue for="5" value=" FWLs scale well to long sequences and are complementary to existing long-text generation methods such as sparse attention \cite{child2019generating} or recurrent processing \cite{dai2019transformer}." />
        </attvalues>
      </edge>
      <edge source="2212.02475" target="1609.07843" id="4487">
        <attvalues>
          <attvalue for="5" value=" We evaluate FWLs at language modeling on the WikiText-103 dataset \cite{merity2016pointer}." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="2303.08774" id="4488">
        <attvalues>
          <attvalue for="5" value=" Advanced LLMs such as OpenAI's GPT-4~\cite{openai2023gpt}, Meta's LLaMA-3~\cite{meta2023llama3}, and Google's Gemini \cite{team2023gemini} excel in generating coherent text with extensive parameters." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="2312.11805" id="4489">
        <attvalues>
          <attvalue for="5" value=" Advanced LLMs such as OpenAI's GPT-4~\cite{openai2023gpt}, Meta's LLaMA-3~\cite{meta2023llama3}, and Google's Gemini \cite{team2023gemini} excel in generating coherent text with extensive parameters." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="2205.01068" id="4490">
        <attvalues>
          <attvalue for="5" value=" We empirically evaluate FISTAPruner on the widely adopted OPT \cite{zhang2022opt}, LLaMA~\cite{touvron2023llama}, and LLaMA-2~\cite{touvron2023llama2} model families, as well as the latest LLaMA-3 \cite{touvron2023llama} models." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="2004.11627" id="4491">
        <attvalues>
          <attvalue for="5" value=" Methods such as those in \cite{huang2020convolution, ma2023llm_pruner, zhang2023lottery} require a retraining phase post-pruning, which is inefficient for billion-scale LLMs." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="2301.00774" id="4492">
        <attvalues>
          <attvalue for="5" value=" Recent developments, including SparseGPT~\cite{frantar2023sparsegpt} and Wanda \cite{sun2023simple}, employ one-shot post-training pruning techniques for LLMs." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="2306.11695" id="4493">
        <attvalues>
          <attvalue for="5" value=" Recent developments, including SparseGPT~\cite{frantar2023sparsegpt} and Wanda \cite{sun2023simple}, employ one-shot post-training pruning techniques for LLMs." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="1706.03762" id="4494">
        <attvalues>
          <attvalue for="5" value=" Figure \ref{fig:overview} provides an overview of our method, which is applied to each linear operator, such as \(W_K\), \(W_Q\), \(W_V\), and $W_O$ within the Transformer's attention blocks \cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="math/0409186" id="4495">
        <attvalues>
          <attvalue for="5" value=" Additionally, we integrate an $\ell_1$-norm regularization term, the optimal convex approximation of the $\ell_0$-norm~\cite{candes2006robust}, into each row of weights to promote sparsity." />
        </attvalues>
      </edge>
      <edge source="2408.03728" target="2307.09288" id="4497">
        <attvalues>
          <attvalue for="5" value=" We empirically evaluate FISTAPruner on the widely adopted OPT \cite{zhang2022opt}, LLaMA~\cite{touvron2023llama}, and LLaMA-2~\cite{touvron2023llama2} model families, as well as the latest LLaMA-3 \cite{touvron2023llama} models." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="2005.14165" id="4499">
        <attvalues>
          <attvalue for="5" value=" Understanding how specific the language of PLMs is can help us better understand the behavior of language models and facilitate downstream applications such as question answering, text generation, and information extraction \cite{liu2021pre,khashabi2020unifiedqa,NEURIPS2020_1457c0d6,wang2020language}, eg, making the generated answers/sentences or extracted information more specific or fine-grained." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="1909.01066" id="4500">
        <attvalues>
          <attvalue for="5" value=" Although there are works on measuring how much knowledge is stored in PLMs or improving the correctness of the predictions \cite{petroni2019language,roberts2020much,jiang2020can}, few attempted to measure or improve the specificity of predictions made by PLMs." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="2002.08910" id="4501">
        <attvalues>
          <attvalue for="5" value=" Although there are works on measuring how much knowledge is stored in PLMs or improving the correctness of the predictions \cite{petroni2019language,roberts2020much,jiang2020can}, few attempted to measure or improve the specificity of predictions made by PLMs." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="2010.06189" id="4502">
        <attvalues>
          <attvalue for="5" value=" Recent works show that the parameters of these models contain significant amounts of knowledge \cite{petroni2019language,roberts2020much,jiang2020x,jiang2020can,wang2020language}, and knowledge stored in PLMs can be extracted by predicting the mask token(s) using prompts." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="2010.11967" id="4504">
        <attvalues>
          <attvalue for="5" value=" Understanding how specific the language of PLMs is can help us better understand the behavior of language models and facilitate downstream applications such as question answering, text generation, and information extraction \cite{liu2021pre,khashabi2020unifiedqa,NEURIPS2020_1457c0d6,wang2020language}, eg, making the generated answers/sentences or extracted information more specific or fine-grained." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="2001.09977" id="4505">
        <attvalues>
          <attvalue for="5" value=" Noteworthy exceptions include the work by \cite{adiwardana2020towards,thoppilan2022lamda}, who evaluated the specificity of conversational language models." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="2201.08239" id="4506">
        <attvalues>
          <attvalue for="5" value=" Noteworthy exceptions include the work by \cite{adiwardana2020towards,thoppilan2022lamda}, who evaluated the specificity of conversational language models." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="2107.13586" id="4507">
        <attvalues>
          <attvalue for="5" value=" Understanding how specific the language of PLMs is can help us better understand the behavior of language models and facilitate downstream applications such as question answering, text generation, and information extraction \cite{liu2021pre,khashabi2020unifiedqa,NEURIPS2020_1457c0d6,wang2020language}, eg, making the generated answers/sentences or extracted information more specific or fine-grained." />
        </attvalues>
      </edge>
      <edge source="2210.05159" target="2005.00700" id="4508">
        <attvalues>
          <attvalue for="5" value=" Understanding how specific the language of PLMs is can help us better understand the behavior of language models and facilitate downstream applications such as question answering, text generation, and information extraction \cite{liu2021pre,khashabi2020unifiedqa,NEURIPS2020_1457c0d6,wang2020language}, eg, making the generated answers/sentences or extracted information more specific or fine-grained." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2308.06103" id="4509">
        <attvalues>
          <attvalue for="5" value=" Recent years have witnessed significant strides in Natural Language Processing (NLP), notably the emergence of Large Language Models (LLMs) \cite{LLM}, transforming machine-human interaction by mimicking human-like language comprehension and generation." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="1706.03762" id="4510">
        <attvalues>
          <attvalue for="5" value=" Among them, Transformer dominates in NLP due to its powerful performance \cite{1}." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2005.14165" id="4511">
        <attvalues>
          <attvalue for="5" value=" Transformer-based LLMs trained on extensive datasets sourced from the web have achieved remarkable success \cite{brown2020language,achiam2023gpt4,gemini2023}." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2303.08774" id="4512">
        <attvalues>
          <attvalue for="5" value=" Transformer-based LLMs trained on extensive datasets sourced from the web have achieved remarkable success \cite{brown2020language,achiam2023gpt4,gemini2023}." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2312.11805" id="4513">
        <attvalues>
          <attvalue for="5" value=" Transformer-based LLMs trained on extensive datasets sourced from the web have achieved remarkable success \cite{brown2020language,achiam2023gpt4,gemini2023}." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="1911.02150" id="4514">
        <attvalues>
          <attvalue for="5" value=" Some methods simplify the query mechanism by replacing multi-head attention with multi-query attention \cite{kvcache} and group query attention \cite{ainslie2023gqa}." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2305.13245" id="4515">
        <attvalues>
          <attvalue for="5" value=" Some methods simplify the query mechanism by replacing multi-head attention with multi-query attention \cite{kvcache} and group query attention \cite{ainslie2023gqa}." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2105.14103" id="4516">
        <attvalues>
          <attvalue for="5" value=" Other approaches focus on improving the computational efficiency of attention, such as AFT \cite{AFT} and RWKV \cite{2}." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2305.13048" id="4517">
        <attvalues>
          <attvalue for="5" value=" Other approaches focus on improving the computational efficiency of attention, such as AFT \cite{AFT} and RWKV \cite{2}." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2312.00752" id="4518">
        <attvalues>
          <attvalue for="5" value=" Recently, alternatives like Mamba \cite{mamba}, rooted in state-space model (SSM) evolution, have gained traction in the reseach community." />
        </attvalues>
      </edge>
      <edge source="2406.12230" target="2402.19427" id="4519">
        <attvalues>
          <attvalue for="5" value=" Yet, empirical evidences suggest scaling challenges for Mamba \cite{de2024griffin}, indicating ongoing hurdles in its widespread adoption." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="1910.13461" id="4522">
        <attvalues>
          <attvalue for="5" value=" Since then, the size and complexity of the embedding models have increased dramatically, especially with the introduction of transformer-based architectures like BERT~\cite{devlin2018bert}, BART~\cite{lewis2019bart}, RoBERTa~\cite{liu2019roberta}, followed by rigorously fine-tuned sentence encoders like LASER \cite{Artetxe_2019}, Universal Sentence Encoder (USE) \cite{cer2018universal}, SentenceBERT (SBERT) \cite{reimers2019sentencebert}, etc, and finally with the emergence of Large Language Models (LLMs) like GPT~\cite{brown2020language}, PaLM~\cite{chowdhery2022palm}, LLaMA~\cite{touvron2023llama}, etc While transformer-based models not only offer embeddings for words but also for longer sequences like sentences and documents, learning meaningful word embeddings still remains fundamental for proper contextualization of longer sequences and robust language modeling." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="1907.11692" id="4523">
        <attvalues>
          <attvalue for="5" value=" Since then, the size and complexity of the embedding models have increased dramatically, especially with the introduction of transformer-based architectures like BERT~\cite{devlin2018bert}, BART~\cite{lewis2019bart}, RoBERTa~\cite{liu2019roberta}, followed by rigorously fine-tuned sentence encoders like LASER \cite{Artetxe_2019}, Universal Sentence Encoder (USE) \cite{cer2018universal}, SentenceBERT (SBERT) \cite{reimers2019sentencebert}, etc, and finally with the emergence of Large Language Models (LLMs) like GPT~\cite{brown2020language}, PaLM~\cite{chowdhery2022palm}, LLaMA~\cite{touvron2023llama}, etc While transformer-based models not only offer embeddings for words but also for longer sequences like sentences and documents, learning meaningful word embeddings still remains fundamental for proper contextualization of longer sequences and robust language modeling." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="1812.10464" id="4524">
        <attvalues>
          <attvalue for="5" value=" Since then, the size and complexity of the embedding models have increased dramatically, especially with the introduction of transformer-based architectures like BERT~\cite{devlin2018bert}, BART~\cite{lewis2019bart}, RoBERTa~\cite{liu2019roberta}, followed by rigorously fine-tuned sentence encoders like LASER \cite{Artetxe_2019}, Universal Sentence Encoder (USE) \cite{cer2018universal}, SentenceBERT (SBERT) \cite{reimers2019sentencebert}, etc, and finally with the emergence of Large Language Models (LLMs) like GPT~\cite{brown2020language}, PaLM~\cite{chowdhery2022palm}, LLaMA~\cite{touvron2023llama}, etc While transformer-based models not only offer embeddings for words but also for longer sequences like sentences and documents, learning meaningful word embeddings still remains fundamental for proper contextualization of longer sequences and robust language modeling." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="1908.10084" id="4526">
        <attvalues>
          <attvalue for="5" value=" Since then, the size and complexity of the embedding models have increased dramatically, especially with the introduction of transformer-based architectures like BERT~\cite{devlin2018bert}, BART~\cite{lewis2019bart}, RoBERTa~\cite{liu2019roberta}, followed by rigorously fine-tuned sentence encoders like LASER \cite{Artetxe_2019}, Universal Sentence Encoder (USE) \cite{cer2018universal}, SentenceBERT (SBERT) \cite{reimers2019sentencebert}, etc, and finally with the emergence of Large Language Models (LLMs) like GPT~\cite{brown2020language}, PaLM~\cite{chowdhery2022palm}, LLaMA~\cite{touvron2023llama}, etc While transformer-based models not only offer embeddings for words but also for longer sequences like sentences and documents, learning meaningful word embeddings still remains fundamental for proper contextualization of longer sequences and robust language modeling." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="2005.14165" id="4527">
        <attvalues>
          <attvalue for="5" value=" Since then, the size and complexity of the embedding models have increased dramatically, especially with the introduction of transformer-based architectures like BERT~\cite{devlin2018bert}, BART~\cite{lewis2019bart}, RoBERTa~\cite{liu2019roberta}, followed by rigorously fine-tuned sentence encoders like LASER \cite{Artetxe_2019}, Universal Sentence Encoder (USE) \cite{cer2018universal}, SentenceBERT (SBERT) \cite{reimers2019sentencebert}, etc, and finally with the emergence of Large Language Models (LLMs) like GPT~\cite{brown2020language}, PaLM~\cite{chowdhery2022palm}, LLaMA~\cite{touvron2023llama}, etc While transformer-based models not only offer embeddings for words but also for longer sequences like sentences and documents, learning meaningful word embeddings still remains fundamental for proper contextualization of longer sequences and robust language modeling." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="2204.02311" id="4528">
        <attvalues>
          <attvalue for="5" value=" Since then, the size and complexity of the embedding models have increased dramatically, especially with the introduction of transformer-based architectures like BERT~\cite{devlin2018bert}, BART~\cite{lewis2019bart}, RoBERTa~\cite{liu2019roberta}, followed by rigorously fine-tuned sentence encoders like LASER \cite{Artetxe_2019}, Universal Sentence Encoder (USE) \cite{cer2018universal}, SentenceBERT (SBERT) \cite{reimers2019sentencebert}, etc, and finally with the emergence of Large Language Models (LLMs) like GPT~\cite{brown2020language}, PaLM~\cite{chowdhery2022palm}, LLaMA~\cite{touvron2023llama}, etc While transformer-based models not only offer embeddings for words but also for longer sequences like sentences and documents, learning meaningful word embeddings still remains fundamental for proper contextualization of longer sequences and robust language modeling." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="2307.09288" id="4529">
        <attvalues>
          <attvalue for="5" value=" Since then, the size and complexity of the embedding models have increased dramatically, especially with the introduction of transformer-based architectures like BERT~\cite{devlin2018bert}, BART~\cite{lewis2019bart}, RoBERTa~\cite{liu2019roberta}, followed by rigorously fine-tuned sentence encoders like LASER \cite{Artetxe_2019}, Universal Sentence Encoder (USE) \cite{cer2018universal}, SentenceBERT (SBERT) \cite{reimers2019sentencebert}, etc, and finally with the emergence of Large Language Models (LLMs) like GPT~\cite{brown2020language}, PaLM~\cite{chowdhery2022palm}, LLaMA~\cite{touvron2023llama}, etc While transformer-based models not only offer embeddings for words but also for longer sequences like sentences and documents, learning meaningful word embeddings still remains fundamental for proper contextualization of longer sequences and robust language modeling." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="2303.12712" id="4530">
        <attvalues>
          <attvalue for="5" value=" While LLMs have shown remarkable success across various well-known NLP tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using,zero-shot-souvika,Bangla-Word-Analogy}, it remains an open question if their enhanced performance can solely be attributed to their larger scale or if the embeddings they generate are fundamentally distinct from those created by traditional encoding models such as Sentence-BERT or Universal Sentence Encoders~\cite{yash,souvika-USE}." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="2112.06905" id="4531">
        <attvalues>
          <attvalue for="5" value=" While LLMs have shown remarkable success across various well-known NLP tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using,zero-shot-souvika,Bangla-Word-Analogy}, it remains an open question if their enhanced performance can solely be attributed to their larger scale or if the embeddings they generate are fundamentally distinct from those created by traditional encoding models such as Sentence-BERT or Universal Sentence Encoders~\cite{yash,souvika-USE}." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="2201.11990" id="4532">
        <attvalues>
          <attvalue for="5" value=" While LLMs have shown remarkable success across various well-known NLP tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using,zero-shot-souvika,Bangla-Word-Analogy}, it remains an open question if their enhanced performance can solely be attributed to their larger scale or if the embeddings they generate are fundamentally distinct from those created by traditional encoding models such as Sentence-BERT or Universal Sentence Encoders~\cite{yash,souvika-USE}." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="2309.03747" id="4534">
        <attvalues>
          <attvalue for="5" value=" While LLMs have shown remarkable success across various well-known NLP tasks~\cite{bubeck2023sparks,dai2022can,du2022glam,smith2022using,zero-shot-souvika,Bangla-Word-Analogy}, it remains an open question if their enhanced performance can solely be attributed to their larger scale or if the embeddings they generate are fundamentally distinct from those created by traditional encoding models such as Sentence-BERT or Universal Sentence Encoders~\cite{yash,souvika-USE}." />
        </attvalues>
      </edge>
      <edge source="2402.11094" target="2010.03446" id="4535">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{fournier2020analogies} evaluated smaller transformer models like GPT-2 and BERT on analogy tasks and found that these models performed poorly on Encyclopedic and Lexicographic analogies, further raising the question of how embedding performance changes as model size increases." />
        </attvalues>
      </edge>
      <edge source="2204.06452" target="1606.03777" id="4536">
        <attvalues>
          <attvalue for="5" value=" The methodology for building TOD systems is gradually advancing from separate training of individual modules \cite{williams2016dialog,mrkvsic2017neural, dai2018tracking} to the end-to-end (E2E) trainable approach \cite{wen2017a, liu2017end, lei2018sequicity, fsdm, zhang2020task,gao2020paraphrase,zhang-etal-2020-probabilistic}." />
        </attvalues>
      </edge>
      <edge source="2204.06452" target="1604.04562" id="4537">
        <attvalues>
          <attvalue for="5" value=" As briefly reviewed above, Markov models have been used in previous LSTM-based TOD systems \cite{wen2017a}-\cite{zhang-etal-2020-probabilistic}, but not studied for PLM-based TOD systems." />
        </attvalues>
      </edge>
      <edge source="2204.06452" target="2009.08115" id="4538">
        <attvalues>
          <attvalue for="5" value=" As briefly reviewed above, Markov models have been used in previous LSTM-based TOD systems \cite{wen2017a}-\cite{zhang-etal-2020-probabilistic}, but not studied for PLM-based TOD systems." />
        </attvalues>
      </edge>
      <edge source="2204.06452" target="1706.03762" id="4539">
        <attvalues>
          <attvalue for="5" value=" Recently, self-attention based Transformer neural networks \cite{vaswani2017attention} have shown their superiority in capturing long-term dependencies over LSTM based networks." />
        </attvalues>
      </edge>
      <edge source="2204.06452" target="2005.00796" id="4541">
        <attvalues>
          <attvalue for="5" value=" Examples include GPT2-based SimpleTOD \cite{hosseini2020simple}, SOLOIST \cite{peng2020etal}, AuGPT \cite{kulhanek2021augpt} and UBAR \cite{yang2021ubar}, and T5-based PPTOD \cite{su2021multitask} and MTTOD \cite{lee-2021-improving-end}, among others." />
        </attvalues>
      </edge>
      <edge source="2204.06452" target="2109.14739" id="4543">
        <attvalues>
          <attvalue for="5" value=" Examples include GPT2-based SimpleTOD \cite{hosseini2020simple}, SOLOIST \cite{peng2020etal}, AuGPT \cite{kulhanek2021augpt} and UBAR \cite{yang2021ubar}, and T5-based PPTOD \cite{su2021multitask} and MTTOD \cite{lee-2021-improving-end}, among others." />
        </attvalues>
      </edge>
      <edge source="2204.06452" target="1907.01669" id="4544">
        <attvalues>
          <attvalue for="5" value="1 dataset \cite{eric2019multiwoz}." />
        </attvalues>
      </edge>
      <edge source="2310.06547" target="1706.08840" id="4545">
        <attvalues>
          <attvalue for="5" value=" They focused on recovering the representations on previous tasks, using methods like restricted gradient updating and knowledge distillation \cite{lopezpaz2017gradient, cao-etal-2020-incremental}." />
        </attvalues>
      </edge>
      <edge source="2310.06547" target="2210.04497" id="4546">
        <attvalues>
          <attvalue for="5" value=" Thus, making a model learn current relations robustly to avoid subsequent confusion becomes the new research focus \cite{wang-etal-2022-learning-robust, zhao2023improving}." />
        </attvalues>
      </edge>
      <edge source="2310.06547" target="2204.11790" id="4547">
        <attvalues>
          <attvalue for="5" value=" This is inspired by the intuition that, training models with explicit rationale supervision can provide greater robustness \cite{chen-etal-2022-rationalization}." />
        </attvalues>
      </edge>
      <edge source="2310.06547" target="2210.06726" id="4548">
        <attvalues>
          <attvalue for="5" value=" Moreover, since relation extraction requires reasoning over two entities, providing explanations for why the two entities have a specific relation can enhance the reasoning capacity of smaller models, thereby eliminating reliance on spurious shortcuts \cite{li2022explanations, magister2023teaching}." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1908.06024" id="4550">
        <attvalues>
          <attvalue for="5" value=" Several publicly available datasets have been created for the task \cite{mishra2019tackling,vidgen2020directions}." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="2004.01670" id="4551">
        <attvalues>
          <attvalue for="5" value=" Several publicly available datasets have been created for the task \cite{mishra2019tackling,vidgen2020directions}." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1702.08138" id="4552">
        <attvalues>
          <attvalue for="5" value=" Several terms have been used to refer to the general concept of harmful online behavior, including toxicity \cite{hosseini2017deceiving}, hate speech \cite{schmidt2017survey}, offensive \cite{zampieri2019semeval} and abusive language \cite{waseem2017understanding,vidgen2019challenges}." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1903.08983" id="4553">
        <attvalues>
          <attvalue for="5" value=" \cite{zampieri2019semeval} used words and phrases frequently found in offensive messages to search for potential abusive tweets." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1705.09899" id="4554">
        <attvalues>
          <attvalue for="5" value=" Several terms have been used to refer to the general concept of harmful online behavior, including toxicity \cite{hosseini2017deceiving}, hate speech \cite{schmidt2017survey}, offensive \cite{zampieri2019semeval} and abusive language \cite{waseem2017understanding,vidgen2019challenges}." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1906.01738" id="4555">
        <attvalues>
          <attvalue for="5" value=" Still, in practice, every dataset only focuses on a narrow range of sub-types of such behaviours and a single online platform \cite{jurgens2019just}." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1703.04009" id="4556">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{davidson2017automated} annotated tweets for three categories, Racist, Offensive but not Racist and Clean, and \cite{nobata2016abusive} collected discussions from Yahoo! Finance news and applied a binary annotation scheme of Abusive versus Clean." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1809.07572" id="4558">
        <attvalues>
          <attvalue for="5" value=" As a result, although models tend to perform well in cross-validation evaluation on one dataset, the cross-dataset generalizability remains low \cite{van2018challenges,wiegand-etal-2019-detection}." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1707.05246" id="4559">
        <attvalues>
          <attvalue for="5" value=" This approach is different from domain adaptation techniques based on data selection \cite{ruder2017learning,liu2019reinforced} in that we apply an unsupervised topic modeling method for topic discovery without using the class labels." />
        </attvalues>
      </edge>
      <edge source="2010.07414" target="1610.08914" id="4560">
        <attvalues>
          <attvalue for="5" value=" We focus on the Wikipedia Detox or Wiki-dataset, (an extension of the dataset by \cite{wulczyn2017ex}), collected from English Wikipedia talk pages and annotated for toxicity." />
        </attvalues>
      </edge>
      <edge source="2405.01582" target="2205.01068" id="4576">
        <attvalues>
          <attvalue for="5" value=" Language Models (LMs) have gained significant attention in recent years due to their impressive performance in various natural language processing (NLP) tasks \cite{opt, falcon, llama, lima, roberta}." />
        </attvalues>
      </edge>
      <edge source="2405.01582" target="2304.05335" id="4577">
        <attvalues>
          <attvalue for="5" value=" What's worse is that several of these datasets are uncurated and may contain harmful content which the LM model can potentially pick up during the training process \cite{bias1, bias2, bias3}." />
        </attvalues>
      </edge>
      <edge source="2405.01582" target="2107.00061" id="4578">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{humaneval} introduce a crowdsourcing-based method for ranking text quality, where human evaluators provide subjective ratings." />
        </attvalues>
      </edge>
      <edge source="2405.01582" target="2303.15056" id="4579">
        <attvalues>
          <attvalue for="5" value=" To overcome these limitations, more recent works have explored the use of automated approaches to quality evaluation such as making use of ChatGPT or GPT-4 to evaluate the quality of the text, where text is designated to be high quality if ChatGPT/GPT-4 deems it to be similar to human text \cite{chatgpteval, gpt4eval}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1906.01502" id="4580">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1901.07291" id="4581">
        <attvalues>
          <attvalue for="5" value=" Multi-lingual Pre-trained Language Models (PLMs) such as multi-lingual BERT (mBERT) \cite{pires2019multilingual}, XLM \cite{conneau2019cross} and XLM-Roberta (XLM-R) \cite{conneau2020unsupervised}, have demonstrated superior performance in many cross-lingual zero-shot downstream tasks such as natural language inference and question answering." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1911.02116" id="4582">
        <attvalues>
          <attvalue for="5" value=" Multi-lingual Pre-trained Language Models (PLMs) such as multi-lingual BERT (mBERT) \cite{pires2019multilingual}, XLM \cite{conneau2019cross} and XLM-Roberta (XLM-R) \cite{conneau2020unsupervised}, have demonstrated superior performance in many cross-lingual zero-shot downstream tasks such as natural language inference and question answering." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2005.04511" id="4583">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2004.03032" id="4584">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2011.02070" id="4585">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2004.14096" id="4586">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1911.03310" id="4587">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1905.05950" id="4589">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1912.07840" id="4590">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1904.09077" id="4591">
        <attvalues>
          <attvalue for="5" value=" For example, they have been studied by the linguistic properties \cite{chi2020finding,edmiston2020systematic,pires2019multilingual,rama2020probing,kulmizev2020neural}, language neutrality \cite{libovicky2019language,libovicky2020language}, layer representation \cite{de2020s,singh2019bert,tenney2019bert,karthikeyan2019cross,wu2019beto}, and language generation \cite{ronnqvist2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1909.02197" id="4593">
        <attvalues>
          <attvalue for="5" value=" Another line of related work is to understand the multi-lingual model representation in the parallel corpus \cite{kudugunta2019investigating}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2010.05731" id="4594">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1906.05061" id="4595">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2101.11888" id="4596">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1802.09375" id="4597">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2009.12862" id="4598">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2004.14923" id="4599">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1903.08855" id="4600">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1908.06625" id="4601">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="1805.03620" id="4602">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2112.12356" target="2004.04070" id="4603">
        <attvalues>
          <attvalue for="5" value=" They include the probing technique to investigate linguistic properties such as typological features \cite{vulic2020probing,ravishankar2019probing,ravishankar2019multilingual,bjerva2021does,bjerva2018phonology,choenni2020does,oncevay2020bridging}, and the isomorphism measure \cite{liu2019linguistic,patra2019bilingual,sogaard2018limitations,vulic2020all}." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="2004.09095" id="4604">
        <attvalues>
          <attvalue for="5" value=" Most languages in the world have little access to NLP technology due to data scarcity \cite{joshi2020state}." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="1602.01595" id="4606">
        <attvalues>
          <attvalue for="5" value=" Most previous work includes language information as features, by using language IDs, or language embeddings (eg~\cite{ammar2016many,o2016survey,ostling-tiedemann-2017-continuous,ponti2019modeling,oncevay2020bridging})." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="1610.03349" id="4607">
        <attvalues>
          <attvalue for="5" value=" Most previous work includes language information as features, by using language IDs, or language embeddings (eg~\cite{ammar2016many,o2016survey,ostling-tiedemann-2017-continuous,ponti2019modeling,oncevay2020bridging})." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="1612.07486" id="4608">
        <attvalues>
          <attvalue for="5" value=" Most previous work includes language information as features, by using language IDs, or language embeddings (eg~\cite{ammar2016many,o2016survey,ostling-tiedemann-2017-continuous,ponti2019modeling,oncevay2020bridging})." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="1807.00914" id="4609">
        <attvalues>
          <attvalue for="5" value=" Most previous work includes language information as features, by using language IDs, or language embeddings (eg~\cite{ammar2016many,o2016survey,ostling-tiedemann-2017-continuous,ponti2019modeling,oncevay2020bridging})." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="2004.14923" id="4610">
        <attvalues>
          <attvalue for="5" value=" Most previous work includes language information as features, by using language IDs, or language embeddings (eg~\cite{ammar2016many,o2016survey,ostling-tiedemann-2017-continuous,ponti2019modeling,oncevay2020bridging})." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="1808.09055" id="4611">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{delhoneux2018parameter} observe positive cross-lingual sharing effects only in a handful of their settings." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="1802.09375" id="4612">
        <attvalues>
          <attvalue for="5" value=" This is confirmed by \cite{bjerva2018phonology}, who find that, eg, language embeddings trained on a morphological task can encode morphological features from WALS." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="1409.7495" id="4613">
        <attvalues>
          <attvalue for="5" value=" In contrast with previous work, we blind a model to typological information, by using adversarial techniques based on gradient reversal \cite{ganin2014unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2101.11888" target="2003.11080" id="4614">
        <attvalues>
          <attvalue for="5" value=" We evaluate on the structured prediction and classification tasks in XTREME \cite{xtreme}, yielding a total of 40 languages and 4 tasks." />
        </attvalues>
      </edge>
      <edge source="2004.04070" target="1408.3456" id="4615">
        <attvalues>
          <attvalue for="5" value=" The extent to which they really do so has been evaluated, eg, using semantic word similarity and association norms \cite{hill2015simlex,Gerz:2016emnlp}, and word analogy benchmarks \cite{Mikolov2013naacl}." />
        </attvalues>
      </edge>
      <edge source="2004.04070" target="1608.00869" id="4616">
        <attvalues>
          <attvalue for="5" value=" The extent to which they really do so has been evaluated, eg, using semantic word similarity and association norms \cite{hill2015simlex,Gerz:2016emnlp}, and word analogy benchmarks \cite{Mikolov2013naacl}." />
        </attvalues>
      </edge>
      <edge source="2004.04070" target="1309.4168" id="4617">
        <attvalues>
          <attvalue for="5" value=" Researchers have exploited this to learn linear transformations between such spaces \cite{Mikolov2013exploiting,Glavas2019}, which have been used to induce bilingual dictionaries, as well as to facilitate multilingual modeling and cross-lingual transfer \cite{Ruder2019survey}." />
        </attvalues>
      </edge>
      <edge source="2004.04070" target="1902.00508" id="4618">
        <attvalues>
          <attvalue for="5" value=" Researchers have exploited this to learn linear transformations between such spaces \cite{Mikolov2013exploiting,Glavas2019}, which have been used to induce bilingual dictionaries, as well as to facilitate multilingual modeling and cross-lingual transfer \cite{Ruder2019survey}." />
        </attvalues>
      </edge>
      <edge source="2004.04070" target="1706.04902" id="4619">
        <attvalues>
          <attvalue for="5" value=" Researchers have exploited this to learn linear transformations between such spaces \cite{Mikolov2013exploiting,Glavas2019}, which have been used to induce bilingual dictionaries, as well as to facilitate multilingual modeling and cross-lingual transfer \cite{Ruder2019survey}." />
        </attvalues>
      </edge>
      <edge source="2004.04070" target="1805.03620" id="4620">
        <attvalues>
          <attvalue for="5" value=" It furthermore provides us with an explanation for reported failures to align word vector spaces in different languages \cite{sogaard2018limitations,Artetxe2018}, which has so far been largely attributed only to inherent typological differences." />
        </attvalues>
      </edge>
      <edge source="2004.04070" target="1805.06297" id="4621">
        <attvalues>
          <attvalue for="5" value=" It furthermore provides us with an explanation for reported failures to align word vector spaces in different languages \cite{sogaard2018limitations,Artetxe2018}, which has so far been largely attributed only to inherent typological differences." />
        </attvalues>
      </edge>
      <edge source="2406.10764" target="2103.15721" id="4623">
        <attvalues>
          <attvalue for="5" value=" These datasets included CaSiNo~\cite{chawla2021casino}, which consists of negotiations involving campfire resources like firewood, water, and food; Job Interview~\cite{job_interview}, containing simulated job interview negotiations for salary and responsibilities; Craigslist Bargain~\cite{craiglist_bargain}, focused on negotiations for items on the Craigslist platform; and Persuasion For Good~\cite{persuasion-for-good}, where participants are persuaded to donate to social causes." />
        </attvalues>
      </edge>
      <edge source="2406.10764" target="1706.05125" id="4624">
        <attvalues>
          <attvalue for="5" value=" These studies include development of strategy prediction models for tasks such as commodity price bargaining~\cite{ahmad2023ina, craiglist_bargain} and item negotiation in a barter setting~\cite{lewis2017dealnodeal, chawla2021casino} while limiting the scope of the negotiation to a specific selection of items (closed-domain)." />
        </attvalues>
      </edge>
      <edge source="2406.10764" target="1808.09637" id="4625">
        <attvalues>
          <attvalue for="5" value=" These datasets included CaSiNo~\cite{chawla2021casino}, which consists of negotiations involving campfire resources like firewood, water, and food; Job Interview~\cite{job_interview}, containing simulated job interview negotiations for salary and responsibilities; Craigslist Bargain~\cite{craiglist_bargain}, focused on negotiations for items on the Craigslist platform; and Persuasion For Good~\cite{persuasion-for-good}, where participants are persuaded to donate to social causes." />
        </attvalues>
      </edge>
      <edge source="2406.10764" target="2305.17325" id="4626">
        <attvalues>
          <attvalue for="5" value=" Additionally, closed-domain settings have been shown to affect out-of-domain task performance, which remains an unsolved challenge for modern negotiation agents~\cite{li-murray-2023-zero}." />
        </attvalues>
      </edge>
      <edge source="2406.10764" target="2303.12712" id="4627">
        <attvalues>
          <attvalue for="5" value=" Recently, Large Language Models (LLMs) have shown excellent generalization capabilities in zero and few-shot settings across multiple benchmarks~\cite{bubeck2023sparks, chang2023survey,kojima2022large, hou2024large, ahmed2022few, dai2022promptagator}." />
        </attvalues>
      </edge>
      <edge source="2406.10764" target="1906.06725" id="4629">
        <attvalues>
          <attvalue for="5" value=" These datasets included CaSiNo~\cite{chawla2021casino}, which consists of negotiations involving campfire resources like firewood, water, and food; Job Interview~\cite{job_interview}, containing simulated job interview negotiations for salary and responsibilities; Craigslist Bargain~\cite{craiglist_bargain}, focused on negotiations for items on the Craigslist platform; and Persuasion For Good~\cite{persuasion-for-good}, where participants are persuaded to donate to social causes." />
        </attvalues>
      </edge>
      <edge source="2005.09282" target="1606.06950" id="4630">
        <attvalues>
          <attvalue for="5" value=" Bayesian approaches model the problem as a generative process such as an HMM or GMM with a Dirichlet process prior so that both the number of units and the parameters of those units' models can be inferred~\cite{lee2012nonparametric,ondel2016variational,kamper2017segmental,heck2017feature,ondel2018bayesian, ondel2017bayesian, Ondel2019shmm}." />
        </attvalues>
      </edge>
      <edge source="2005.09282" target="1802.06053" id="4631">
        <attvalues>
          <attvalue for="5" value=" Bayesian approaches model the problem as a generative process such as an HMM or GMM with a Dirichlet process prior so that both the number of units and the parameters of those units' models can be inferred~\cite{lee2012nonparametric,ondel2016variational,kamper2017segmental,heck2017feature,ondel2018bayesian, ondel2017bayesian, Ondel2019shmm}." />
        </attvalues>
      </edge>
      <edge source="2005.09282" target="1811.00403" id="4632">
        <attvalues>
          <attvalue for="5" value=" Autoencoders have been used to this end, with further constraints such as speaker invariance costs or correspondence training~\cite{kamper2015unsupervised,kamper2019truly,yusuf2019temporally}." />
        </attvalues>
      </edge>
      <edge source="2005.09282" target="1312.6114" id="4633">
        <attvalues>
          <attvalue for="5" value=" Variants of the Variational AutoEncoder (VAE) have been especially successful~\cite{kingma2013auto}." />
        </attvalues>
      </edge>
      <edge source="2005.09282" target="1901.08810" id="4634">
        <attvalues>
          <attvalue for="5" value=" Among these are the vector quantized VAE~\cite{chorowski2019unsupervised,Tjandra2019,Eloff2019}, HMM-VAE~\cite{ebbers2017hidden, glarner2018full}." />
        </attvalues>
      </edge>
      <edge source="2005.09282" target="1904.07556" id="4636">
        <attvalues>
          <attvalue for="5" value=" Among these are the vector quantized VAE~\cite{chorowski2019unsupervised,Tjandra2019,Eloff2019}, HMM-VAE~\cite{ebbers2017hidden, glarner2018full}." />
        </attvalues>
      </edge>
      <edge source="2404.01077" target="2005.14165" id="4640">
        <attvalues>
          <attvalue for="5" value="5em, fill=GGreen!25, text=black, align=left,font=, inner xsep=2pt, inner ysep=4pt, line width=1pt, ] As hundreds of billions of breakthroughs on the parameter scale, Large Language Models (LLMs) acquire emergent abilities~\cite{Wei2022EmergentAO}, especially in-context learning ability~\cite{brown2020language} that promote rapid advancement in prompting techniques." />
        </attvalues>
      </edge>
      <edge source="2404.01077" target="2201.11903" id="4641">
        <attvalues>
          <attvalue for="5" value=" Therefore, we mainly focus on the hard prompt that covers all the components of LLM input scaling from concise instructions to long context with demonstrations (Chain-of-Thought (CoT)~\cite{Wei2022ChainOT}, role-playing system prompts, etc)." />
        </attvalues>
      </edge>
      <edge source="2404.01077" target="2305.10601" id="4642">
        <attvalues>
          <attvalue for="5" value=" For example, the CoT series of studies~\cite{Yao2023TreeOT, Besta2023GraphOT, Chen2022ProgramOT} have progressively enhanced LLM reasoning capability by thinking aloud." />
        </attvalues>
      </edge>
      <edge source="2012.15562" target="1911.02116" id="4644">
        <attvalues>
          <attvalue for="5" value=" However, while they exhibit strong transfer performance between resource-rich and similar languages \cite{Conneau2020xlm-r,Artetxe2020cross-lingual}, these models struggle with transfer to low-resource languages \cite{Wu:2020repl} and languages not represented at all in their pretraining corpora \cite{pfeiffer20madx,Muller20BeingUnseen,Ansell2021MADG}." />
        </attvalues>
      </edge>
      <edge source="2012.15562" target="2003.11080" id="4645">
        <attvalues>
          <attvalue for="5" value=" Massively multilingual language models pretrained on large multilingual data, such as multilingual BERT \cite[mBERT;][]{Devlin2019bert} and XLM-R \cite{Conneau2020xlm-r} are the current state-of-the-art vehicle for effective cross-lingual transfer \cite{Hu2020xtreme}." />
        </attvalues>
      </edge>
      <edge source="2012.15562" target="2005.09093" id="4647">
        <attvalues>
          <attvalue for="5" value=" However, while they exhibit strong transfer performance between resource-rich and similar languages \cite{Conneau2020xlm-r,Artetxe2020cross-lingual}, these models struggle with transfer to low-resource languages \cite{Wu:2020repl} and languages not represented at all in their pretraining corpora \cite{pfeiffer20madx,Muller20BeingUnseen,Ansell2021MADG}." />
        </attvalues>
      </edge>
      <edge source="2012.15562" target="2005.00052" id="4648">
        <attvalues>
          <attvalue for="5" value=" Another line of work adapts the embedding layer as well as other layers of the model via adapters \cite{pfeiffer20madx, ustun2020udapter}." />
        </attvalues>
      </edge>
      <edge source="2012.15562" target="2010.12858" id="4649">
        <attvalues>
          <attvalue for="5" value=" However, while they exhibit strong transfer performance between resource-rich and similar languages \cite{Conneau2020xlm-r,Artetxe2020cross-lingual}, these models struggle with transfer to low-resource languages \cite{Wu:2020repl} and languages not represented at all in their pretraining corpora \cite{pfeiffer20madx,Muller20BeingUnseen,Ansell2021MADG}." />
        </attvalues>
      </edge>
      <edge source="2012.15562" target="2004.13640" id="4650">
        <attvalues>
          <attvalue for="5" value=" While the former has only been applied to high-resource languages, the latter approaches have been limited to languages with seen scripts \cite{ChauLS20Parsing, Muller20BeingUnseen} and large pretraining corpora \cite{Wang20ExtendmBERT}." />
        </attvalues>
      </edge>
      <edge source="2012.15562" target="1805.03620" id="4651">
        <attvalues>
          <attvalue for="5" value=" We systematize existing approaches based on the pretrained information they utilize and identify lexically overlapping tokens that are present in both vocabularies as key carriers of such information \cite{Sogaard2018limitations}." />
        </attvalues>
      </edge>
      <edge source="2012.15562" target="1902.00193" id="4652">
        <attvalues>
          <attvalue for="5" value=" We evaluate our approaches in the named entity recognition (NER) task on the standard WikiAnn dataset \cite{Rahimi2019massively} and Dependency Parsing \cite[DP;][]{nivre:2016}." />
        </attvalues>
      </edge>
      <edge source="2010.12858" target="2308.02976" id="4656">
        <attvalues>
          <attvalue for="5" value=" In the past year, monolingual language models have been released for more than 20 languages including Arabic, French, German, and Italian~\cite[inter alia]{antoun2020arabert,martin-etal-2020-camembert,de2019bertje,canete-2020-beto,kuratov-2019-rubert,schweter-2020-berturk}." />
        </attvalues>
      </edge>
      <edge source="2010.12858" target="1911.02116" id="4658">
        <attvalues>
          <attvalue for="5" value=" Even if training multilingual models that cover more languages and language varieties is tempting, the curse of multilinguality \cite{conneau-etal-2020-unsupervised} makes it an impractical solution, as it would require to train ever larger models." />
        </attvalues>
      </edge>
      <edge source="2010.12858" target="2005.09093" id="4660">
        <attvalues>
          <attvalue for="5" value=" Furthermore, as shown by \cite{wu-dredze-2020-languages}, large-scale multilingual language models are sub-optimal\draftremove{performance} for languages that \draftreplace{only account for a small portion of the pretraining." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1703.10135" id="4661">
        <attvalues>
          <attvalue for="5" value=" Recently, the naturalness of corpus-based text-to-speech (TTS) has been significantly improved with the use of attention-based sequence-to-sequence (seq2seq) mapping framework~\cite{wang2017tacotron,shen2018natural}." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1712.05884" id="4662">
        <attvalues>
          <attvalue for="5" value=" Recently, the naturalness of corpus-based text-to-speech (TTS) has been significantly improved with the use of attention-based sequence-to-sequence (seq2seq) mapping framework~\cite{wang2017tacotron,shen2018natural}." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1906.01268" id="4663">
        <attvalues>
          <attvalue for="5" value=" And unveiled by recent NLP tasks, different SAN encoder layers can capture latent syntactic and semantic properties of the input sentence at different levels~\cite{wang2019exploiting,peters2018deep}." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1802.05365" id="4664">
        <attvalues>
          <attvalue for="5" value=" And unveiled by recent NLP tasks, different SAN encoder layers can capture latent syntactic and semantic properties of the input sentence at different levels~\cite{wang2019exploiting,peters2018deep}." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1904.04764" id="4665">
        <attvalues>
          <attvalue for="5" value=" For example, recent study has shown that exploiting syntactic features in a parsed tree is beneficial to the richness of the prosodic outcomes, leading to more natural synthesized speech~\cite{guo2019exploiting}." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1803.09017" id="4666">
        <attvalues>
          <attvalue for="5" value=" To model expressivity, the global style tokens (GST) family~\cite{wang2018style,an2019learning} learns style embeddings from a reference audio in an unsupervised way, which lets the synthesized speech imitate the style of reference audio." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1812.04342" id="4667">
        <attvalues>
          <attvalue for="5" value=" Likewise, the variational autoencoder (VAE) models styles or expressivity in a similar way~\cite{zhang2019learning}." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1809.08895" id="4668">
        <attvalues>
          <attvalue for="5" value=" Recent studies have revealed that self-attention based networks (SAN)~\cite{li2019neural,yasuda2019investigation,yang2019enhancing,yang2019improving} have strong ability in capturing global prosodic information, leading to more natural synthesized speech." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1810.11960" id="4669">
        <attvalues>
          <attvalue for="5" value=" Recent studies have revealed that self-attention based networks (SAN)~\cite{li2019neural,yasuda2019investigation,yang2019enhancing,yang2019improving} have strong ability in capturing global prosodic information, leading to more natural synthesized speech." />
        </attvalues>
      </edge>
      <edge source="2008.00613" target="1810.10181" id="4670">
        <attvalues>
          <attvalue for="5" value=" Specifically, we utilize different levels of representations from the SAN-based text encoder to build a context extractor, which is helpful to extract different levels of syntactic and semantic information~\cite{dou2018exploiting}." />
        </attvalues>
      </edge>
      <edge source="2203.15996" target="1907.11692" id="4672">
        <attvalues>
          <attvalue for="5" value=" Large pre-trained language models (PLMs) \cite{devlin-etal-2019-bert,liu2019roberta} have achieved great success in a variety of NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2203.15996" target="1506.02626" id="4673">
        <attvalues>
          <attvalue for="5" value=" In the unstructured pruning, each model parameter is individually removed if it reaches some criteria based on the magnitude or importance score \cite{DBLP:journals/corr/HanPTD15, DBLP:conf/iclr/ZhuG18, DBLP:conf/nips/Sanh0R20}." />
        </attvalues>
      </edge>
      <edge source="2203.15996" target="1905.10650" id="4674">
        <attvalues>
          <attvalue for="5" value=" Pruning methods can also be classified into optimization-free methods \cite{DBLP:conf/nips/MichelLN19} and the ones that involve optimization \cite{frankle2018the, lagunas-etal-2021-block}." />
        </attvalues>
      </edge>
      <edge source="2203.15996" target="1905.09418" id="4675">
        <attvalues>
          <attvalue for="5" value=" While in the structured pruning, rows or columns of the parameters are removed from the weight matrices \cite{DBLP:journals/corr/abs-1910-06360,DBLP:conf/nips/MichelLN19,voita-etal-2019-analyzing,lagunas-etal-2021-block,DBLP:conf/nips/HouHSJCL20}." />
        </attvalues>
      </edge>
      <edge source="2203.15996" target="2109.04838" id="4676">
        <attvalues>
          <attvalue for="5" value=" While in the structured pruning, rows or columns of the parameters are removed from the weight matrices \cite{DBLP:journals/corr/abs-1910-06360,DBLP:conf/nips/MichelLN19,voita-etal-2019-analyzing,lagunas-etal-2021-block,DBLP:conf/nips/HouHSJCL20}." />
        </attvalues>
      </edge>
      <edge source="2401.07103" target="2004.06063" id="4680">
        <attvalues>
          <attvalue for="5" value=" Traditional NLG evaluation metrics, such as BLEU~\cite{Papineni2002Bleu}, ROUGE~\cite{Lin2004Rouge} and TER~\cite{snover2006study}, primarily focus on surface-level text differences and often fall short in assessing semantic aspects~\cite{freitag-etal-2020-bleu}." />
        </attvalues>
      </edge>
      <edge source="2401.07103" target="1603.08023" id="4681">
        <attvalues>
          <attvalue for="5" value=" Additionally, other methods that employ neural embeddings to calculate the score~\cite{liu2016not,sellam-etal-2020-bleurt,Zhang2020BERTScore}, despite assessing aspects like semantic equivalence and fluency, are inflexible and limited in scope~\cite{freitag-etal-2021-experts}." />
        </attvalues>
      </edge>
      <edge source="2401.07103" target="2004.04696" id="4682">
        <attvalues>
          <attvalue for="5" value=" Additionally, other methods that employ neural embeddings to calculate the score~\cite{liu2016not,sellam-etal-2020-bleurt,Zhang2020BERTScore}, despite assessing aspects like semantic equivalence and fluency, are inflexible and limited in scope~\cite{freitag-etal-2021-experts}." />
        </attvalues>
      </edge>
      <edge source="2401.07103" target="2201.11903" id="4685">
        <attvalues>
          <attvalue for="5" value=" The emergent abilities of LLMs present a promising avenue for the LLM-based NLG evaluation, such as Chain-of-Thought (CoT)~\cite{wei2022chain}, zero-shot instruction following~\cite{DBLP:conf/iclr/WeiBZGYLDDL22}, better alignment with human preference~\cite{ouyang2022training}, etc These attributes position LLMs as potent tools for evaluating NLG outputs, offering a more sophisticated and better human-aligned assessment compared to traditional methods~\cite{liu2023gpteval, kocmi-federmann-2023-large, fu2023gptscore}." />
        </attvalues>
      </edge>
      <edge source="2401.07103" target="2109.01652" id="4686">
        <attvalues>
          <attvalue for="5" value=" The emergent abilities of LLMs present a promising avenue for the LLM-based NLG evaluation, such as Chain-of-Thought (CoT)~\cite{wei2022chain}, zero-shot instruction following~\cite{DBLP:conf/iclr/WeiBZGYLDDL22}, better alignment with human preference~\cite{ouyang2022training}, etc These attributes position LLMs as potent tools for evaluating NLG outputs, offering a more sophisticated and better human-aligned assessment compared to traditional methods~\cite{liu2023gpteval, kocmi-federmann-2023-large, fu2023gptscore}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="1706.03762" id="4687">
        <attvalues>
          <attvalue for="5" value=" The introduction of Transformer \cite{vaswani2023attention} in 2017, followed by groundbreaking LLMs like OpenAI's GPT \cite{brown2020language} and Google's BERT \cite{devlin-etal-2019-bert}, marked the beginning of a new era in language understanding and generation." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2005.14165" id="4688">
        <attvalues>
          <attvalue for="5" value=" The introduction of Transformer \cite{vaswani2023attention} in 2017, followed by groundbreaking LLMs like OpenAI's GPT \cite{brown2020language} and Google's BERT \cite{devlin-etal-2019-bert}, marked the beginning of a new era in language understanding and generation." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2202.04538" id="4693">
        <attvalues>
          <attvalue for="5" value=" More recently, generative LLMs (eg, GPT-3\cite{kojima2023large}, LlaMa\cite{touvron2023llama} and ChatGPT\cite{chatgpt}) have propelled this evolution to unprecedented heights, seamlessly converging with Generative AI and heralding a fresh era in the realm of synthetic data generation\cite{meng2023tuning,meng2022generating,zerogen,gao2023selfguided,ye-etal-2022-progen,regen,chen-etal-2023-mixture}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2202.07922" id="4694">
        <attvalues>
          <attvalue for="5" value=" More recently, generative LLMs (eg, GPT-3\cite{kojima2023large}, LlaMa\cite{touvron2023llama} and ChatGPT\cite{chatgpt}) have propelled this evolution to unprecedented heights, seamlessly converging with Generative AI and heralding a fresh era in the realm of synthetic data generation\cite{meng2023tuning,meng2022generating,zerogen,gao2023selfguided,ye-etal-2022-progen,regen,chen-etal-2023-mixture}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2305.10703" id="4697">
        <attvalues>
          <attvalue for="5" value=" More recently, generative LLMs (eg, GPT-3\cite{kojima2023large}, LlaMa\cite{touvron2023llama} and ChatGPT\cite{chatgpt}) have propelled this evolution to unprecedented heights, seamlessly converging with Generative AI and heralding a fresh era in the realm of synthetic data generation\cite{meng2023tuning,meng2022generating,zerogen,gao2023selfguided,ye-etal-2022-progen,regen,chen-etal-2023-mixture}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2203.00667" id="4699">
        <attvalues>
          <attvalue for="5" value=" The origins of Generative AI can be traced back to pivotal models such as Generative Adversarial Networks\cite{goodfellow2014generative} (GANs) and Variational Autoencoders\cite{kingma2022autoencoding} (VAEs), which demonstrated the ability to generate realistic images and signals\cite{wu2020logan}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="1312.6114" id="4700">
        <attvalues>
          <attvalue for="5" value=" The origins of Generative AI can be traced back to pivotal models such as Generative Adversarial Networks\cite{goodfellow2014generative} (GANs) and Variational Autoencoders\cite{kingma2022autoencoding} (VAEs), which demonstrated the ability to generate realistic images and signals\cite{wu2020logan}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="1912.00953" id="4701">
        <attvalues>
          <attvalue for="5" value=" The origins of Generative AI can be traced back to pivotal models such as Generative Adversarial Networks\cite{goodfellow2014generative} (GANs) and Variational Autoencoders\cite{kingma2022autoencoding} (VAEs), which demonstrated the ability to generate realistic images and signals\cite{wu2020logan}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="1904.05342" id="4702">
        <attvalues>
          <attvalue for="5" value=" For instance, ClinicalBERT\cite{huang2019clinicalbert}, adapted from BERT through pre-training on clinical texts, demonstrates superior performance in predicting hospital readmissions compared to the original BERT\cite{devlin2018bert}, which was trained on Wikipedia and BookCorpus\cite{zhu2015aligning} text data." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="1506.06724" id="4703">
        <attvalues>
          <attvalue for="5" value=" For instance, ClinicalBERT\cite{huang2019clinicalbert}, adapted from BERT through pre-training on clinical texts, demonstrates superior performance in predicting hospital readmissions compared to the original BERT\cite{devlin2018bert}, which was trained on Wikipedia and BookCorpus\cite{zhu2015aligning} text data." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2305.13523" id="4704">
        <attvalues>
          <attvalue for="5" value=" By generating text that closely mirrors human language, LLMs facilitate the creation of robust, varied datasets necessary for training and refining AI models across various applications, from healthcare\cite{peng2023study}, eduction\cite{moore2023empowering} to business management\cite{rane2023role}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2303.04226" id="4705">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for Generative AI and LLMs exist, each revisits related works from a different perspective: Generative AI surveys provide a holistic view of this area starting from Generative Adversarial Networks (GANs) to ChatGPT \cite{cao2023comprehensive} and models developed for synthetic data generation in the past decade \cite{bauer2024comprehensive}, with a special focus on text-to-image \cite{zhang2023text} or text-to-speech \cite{zhang2023survey} generation as well as practical applications in Education \cite{baidoo2023education} and Healthcare \cite{yu2023leveraging}; Surveys for LLMs provide systematic categorization \cite{qiu2020pre} for NLP tasks \cite{min2023recent} and methods to adapt these LLMs to specific domains \cite{10356_167965} through model optimization and personalization perspectives \cite{guo2022domain}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2401.02524" id="4706">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for Generative AI and LLMs exist, each revisits related works from a different perspective: Generative AI surveys provide a holistic view of this area starting from Generative Adversarial Networks (GANs) to ChatGPT \cite{cao2023comprehensive} and models developed for synthetic data generation in the past decade \cite{bauer2024comprehensive}, with a special focus on text-to-image \cite{zhang2023text} or text-to-speech \cite{zhang2023survey} generation as well as practical applications in Education \cite{baidoo2023education} and Healthcare \cite{yu2023leveraging}; Surveys for LLMs provide systematic categorization \cite{qiu2020pre} for NLP tasks \cite{min2023recent} and methods to adapt these LLMs to specific domains \cite{10356_167965} through model optimization and personalization perspectives \cite{guo2022domain}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2303.13336" id="4707">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for Generative AI and LLMs exist, each revisits related works from a different perspective: Generative AI surveys provide a holistic view of this area starting from Generative Adversarial Networks (GANs) to ChatGPT \cite{cao2023comprehensive} and models developed for synthetic data generation in the past decade \cite{bauer2024comprehensive}, with a special focus on text-to-image \cite{zhang2023text} or text-to-speech \cite{zhang2023survey} generation as well as practical applications in Education \cite{baidoo2023education} and Healthcare \cite{yu2023leveraging}; Surveys for LLMs provide systematic categorization \cite{qiu2020pre} for NLP tasks \cite{min2023recent} and methods to adapt these LLMs to specific domains \cite{10356_167965} through model optimization and personalization perspectives \cite{guo2022domain}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2003.08271" id="4708">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for Generative AI and LLMs exist, each revisits related works from a different perspective: Generative AI surveys provide a holistic view of this area starting from Generative Adversarial Networks (GANs) to ChatGPT \cite{cao2023comprehensive} and models developed for synthetic data generation in the past decade \cite{bauer2024comprehensive}, with a special focus on text-to-image \cite{zhang2023text} or text-to-speech \cite{zhang2023survey} generation as well as practical applications in Education \cite{baidoo2023education} and Healthcare \cite{yu2023leveraging}; Surveys for LLMs provide systematic categorization \cite{qiu2020pre} for NLP tasks \cite{min2023recent} and methods to adapt these LLMs to specific domains \cite{10356_167965} through model optimization and personalization perspectives \cite{guo2022domain}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2111.01243" id="4709">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for Generative AI and LLMs exist, each revisits related works from a different perspective: Generative AI surveys provide a holistic view of this area starting from Generative Adversarial Networks (GANs) to ChatGPT \cite{cao2023comprehensive} and models developed for synthetic data generation in the past decade \cite{bauer2024comprehensive}, with a special focus on text-to-image \cite{zhang2023text} or text-to-speech \cite{zhang2023survey} generation as well as practical applications in Education \cite{baidoo2023education} and Healthcare \cite{yu2023leveraging}; Surveys for LLMs provide systematic categorization \cite{qiu2020pre} for NLP tasks \cite{min2023recent} and methods to adapt these LLMs to specific domains \cite{10356_167965} through model optimization and personalization perspectives \cite{guo2022domain}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2211.03154" id="4710">
        <attvalues>
          <attvalue for="5" value=" Comprehensive surveys for Generative AI and LLMs exist, each revisits related works from a different perspective: Generative AI surveys provide a holistic view of this area starting from Generative Adversarial Networks (GANs) to ChatGPT \cite{cao2023comprehensive} and models developed for synthetic data generation in the past decade \cite{bauer2024comprehensive}, with a special focus on text-to-image \cite{zhang2023text} or text-to-speech \cite{zhang2023survey} generation as well as practical applications in Education \cite{baidoo2023education} and Healthcare \cite{yu2023leveraging}; Surveys for LLMs provide systematic categorization \cite{qiu2020pre} for NLP tasks \cite{min2023recent} and methods to adapt these LLMs to specific domains \cite{10356_167965} through model optimization and personalization perspectives \cite{guo2022domain}." />
        </attvalues>
      </edge>
      <edge source="2403.04190" target="2201.05273" id="4711">
        <attvalues>
          <attvalue for="5" value=" Surveys on LLMs for text generation \cite{li2022pretrained} focus on developing generative LLMs including model architecture choices and training techniques and do not contain gigantic LLMs released in the past two years." />
        </attvalues>
      </edge>
      <edge source="1909.08940" target="1508.05326" id="4712">
        <attvalues>
          <attvalue for="5" value=" The task of Natural language inference (NLI) \cite{W03-0906,dagan2006pascal,D15-1075} is to specify whether the given hypothesis entails, contradicts, or is neutral regarding the premise." />
        </attvalues>
      </edge>
      <edge source="1909.08940" target="1810.08854" id="4713">
        <attvalues>
          <attvalue for="5" value=" Existing solutions to improve the performance of NLI models across datasets include (1) using external knowledge \cite{joshi2018pair2vec,P18-1224}, and (2) fine-tuning on the target datasets, eg, \cite{bansal:AAAI:2019,liu2019inoculation}." />
        </attvalues>
      </edge>
      <edge source="1909.08940" target="1711.04289" id="4714">
        <attvalues>
          <attvalue for="5" value=" Existing solutions to improve the performance of NLI models across datasets include (1) using external knowledge \cite{joshi2018pair2vec,P18-1224}, and (2) fine-tuning on the target datasets, eg, \cite{bansal:AAAI:2019,liu2019inoculation}." />
        </attvalues>
      </edge>
      <edge source="1909.08940" target="1904.02668" id="4716">
        <attvalues>
          <attvalue for="5" value=" Existing solutions to improve the performance of NLI models across datasets include (1) using external knowledge \cite{joshi2018pair2vec,P18-1224}, and (2) fine-tuning on the target datasets, eg, \cite{bansal:AAAI:2019,liu2019inoculation}." />
        </attvalues>
      </edge>
      <edge source="2205.00320" target="2103.00453" id="4717">
        <attvalues>
          <attvalue for="5" value=" \cite{Schick2020Self-DiagnosisNLP} proposed a self-debiasing approach that uses only a handful of templates that contain the definition of undesired attributes." />
        </attvalues>
      </edge>
      <edge source="2205.00320" target="2009.11462" id="4718">
        <attvalues>
          <attvalue for="5" value=" \cite{Gehman2020REALTOXICITYPROMPTS:Models} showed that pretrained LMs generate toxic text even when conditioned on innocuous prompts." />
        </attvalues>
      </edge>
      <edge source="2406.13357" target="2204.14198" id="4720">
        <attvalues>
          <attvalue for="5" value=" Vision large language models have applied this principle to various vision tasks\cite{NEURIPS2022_960a172b, pmlr-v202-li23q, NEURIPS2023_6dcf277e, pmlr-v139-radford21a, zhu2023minigpt4}." />
        </attvalues>
      </edge>
      <edge source="2406.13357" target="2304.12995" id="4721">
        <attvalues>
          <attvalue for="5" value=" AudioGPT\cite{huang2023audiogpt} and HuggingGPT\cite{NEURIPS2023_77c33e6a} have made preliminary attempts." />
        </attvalues>
      </edge>
      <edge source="2406.13357" target="2305.11000" id="4723">
        <attvalues>
          <attvalue for="5" value=" By discretizing the speech signal into token sequences and expanding them within the LLM, SpeechGPT\cite{zhang-etal-2023-speechgpt} enables seamless text-speech interaction with a vocoder model for speech synthesis." />
        </attvalues>
      </edge>
      <edge source="2406.13357" target="2308.15930" id="4724">
        <attvalues>
          <attvalue for="5" value=" LLaSM\cite{shu2023llasm} uses Whisper\cite{pmlr-v202-radford23a} and Chinese-LLAMA2-7B as speech encoder and LLM with two training stages." />
        </attvalues>
      </edge>
      <edge source="2406.13357" target="2307.03917" id="4726">
        <attvalues>
          <attvalue for="5" value=" Whisper does not appear in Speech-LLaMA\cite{10389705}, they train 4 Transformer layers as audio encoder to complete ST tasks in 13 languages with LLaMA\cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2406.13357" target="2307.09288" id="4727">
        <attvalues>
          <attvalue for="5" value=" Yi-6B with the LLaMA\cite{touvron2023llama} decoder-only structure is selected as LLM." />
        </attvalues>
      </edge>
      <edge source="2406.13357" target="2311.07919" id="4728">
        <attvalues>
          <attvalue for="5" value=" Whisper and Qwen are used in Qwen-Audio\cite{chu2023qwenaudio}, which is also trained in two stages." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="2302.11382" id="4730">
        <attvalues>
          <attvalue for="5" value=" Additionally, prompts are often formatted with reusable templates~\cite{DBLP:journals/corr/abs-2302-11382} as a result of prompt engineering." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="2201.07207" id="4731">
        <attvalues>
          <attvalue for="5" value=" Such examples are common in LLM for robotics and tool learning~\cite{huang2022language, driess2023palme, DBLP:journals/corr/abs-2304-08354}." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="2211.05102" id="4732">
        <attvalues>
          <attvalue for="5" value=" Reusing attention states is a popular strategy for accelerating the service of a single prompt~\cite{pope2022efficiently}." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="2209.04881" id="4733">
        <attvalues>
          <attvalue for="5" value=" We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size~\cite{keles2022computational,tay2023efficient} while the space and compute complexity of \tech scales linearly with the size." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="2009.06732" id="4734">
        <attvalues>
          <attvalue for="5" value=" We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size~\cite{keles2022computational,tay2023efficient} while the space and compute complexity of \tech scales linearly with the size." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="1910.03771" id="4735">
        <attvalues>
          <attvalue for="5" value=" In \S\ref{sec:implementation}, we report a prototype implementation of \tech on top of the HuggingFace transformers library \cite{wolf2020huggingfaces}." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="2307.09288" id="4736">
        <attvalues>
          <attvalue for="5" value=" While \tech can work with any Transformer architecture compatible with \kvcache, we experiment with three popular Transformer architectures powering the following open-sourced LLMs: Llama2 \cite{touvron2023llama}, Falcon \cite{falcon}, and MPT~\cite{MosaicML2023Introducing}." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="2306.01116" id="4737">
        <attvalues>
          <attvalue for="5" value=" While \tech can work with any Transformer architecture compatible with \kvcache, we experiment with three popular Transformer architectures powering the following open-sourced LLMs: Llama2 \cite{touvron2023llama}, Falcon \cite{falcon}, and MPT~\cite{MosaicML2023Introducing}." />
        </attvalues>
      </edge>
      <edge source="2311.04934" target="2308.14508" id="4738">
        <attvalues>
          <attvalue for="5" value=" We employ the LongBench suite~\cite{bai2023longbench}, which includes recommendation and question-answering (QA) tasks based on multiple documents." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2303.08774" id="4739">
        <attvalues>
          <attvalue for="5" value=" The latest generation of large language models (LLMs), such as ChatGPT \cite{2022OpenAIchatgpt} and GPT4 \cite{2023GPT4Openai}, are often referred to as generalist models for their exceptional generalizability to perform various natural language processing (NLP) tasks." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2305.11206" id="4740">
        <attvalues>
          <attvalue for="5" value=" Our research is motivated by the existence of various broad-coverage general-purpose instruction-following datasets \cite{2023TaoriAlpaca,2023PengGPT4Instruct,dolly,2023XuWizardLM,zhou2023lima,su2023pandagpt} and their surprising efficiency for turning LLMs into capable instruction-following generalists." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2305.15717" id="4741">
        <attvalues>
          <attvalue for="5" value=" Recent studies \cite{2023TaoriAlpaca,zhou2023lima,2023GudibandeFalse} suggest that (1) the foundation of their superior performance (eg, knowledge and capabilities) is predominantly acquired during large-scale unsupervised pre-training; and (2) {instruction tuning} \cite{2021SanhT0,wei2021finetuned,mishra2021natural,ouyang2022training} is an incredibly data-efficient method for unleashing the power of LLMs to complete realistic NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2110.08207" id="4742">
        <attvalues>
          <attvalue for="5" value=" Recent studies \cite{2023TaoriAlpaca,zhou2023lima,2023GudibandeFalse} suggest that (1) the foundation of their superior performance (eg, knowledge and capabilities) is predominantly acquired during large-scale unsupervised pre-training; and (2) {instruction tuning} \cite{2021SanhT0,wei2021finetuned,mishra2021natural,ouyang2022training} is an incredibly data-efficient method for unleashing the power of LLMs to complete realistic NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2109.01652" id="4743">
        <attvalues>
          <attvalue for="5" value=" Recent studies \cite{2023TaoriAlpaca,zhou2023lima,2023GudibandeFalse} suggest that (1) the foundation of their superior performance (eg, knowledge and capabilities) is predominantly acquired during large-scale unsupervised pre-training; and (2) {instruction tuning} \cite{2021SanhT0,wei2021finetuned,mishra2021natural,ouyang2022training} is an incredibly data-efficient method for unleashing the power of LLMs to complete realistic NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2302.06476" id="4745">
        <attvalues>
          <attvalue for="5" value=" However, under rigorous evaluation, the performance of those instruction-following generalist models often falls short compared to traditional task-specific specialist models \cite{2023JiaoChatGPTTranslator,2023QinChatGPTGeneral,fang2023chatgpt,liu2023comprehensive}." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2304.01746" id="4746">
        <attvalues>
          <attvalue for="5" value=" However, under rigorous evaluation, the performance of those instruction-following generalist models often falls short compared to traditional task-specific specialist models \cite{2023JiaoChatGPTTranslator,2023QinChatGPTGeneral,fang2023chatgpt,liu2023comprehensive}." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2303.13547" id="4747">
        <attvalues>
          <attvalue for="5" value=" However, under rigorous evaluation, the performance of those instruction-following generalist models often falls short compared to traditional task-specific specialist models \cite{2023JiaoChatGPTTranslator,2023QinChatGPTGeneral,fang2023chatgpt,liu2023comprehensive}." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2304.08085" id="4748">
        <attvalues>
          <attvalue for="5" value=" Recently, there has also been a growing trend towards developing specialist models using instruction tuning \cite{2023JiaoParroT,2023WangInstructUIE,2023ZhangWritingAssistance,cheng2023adapting,2023HuaTuoWang}." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2305.13225" id="4749">
        <attvalues>
          <attvalue for="5" value=" Recently, there has also been a growing trend towards developing specialist models using instruction tuning \cite{2023JiaoParroT,2023WangInstructUIE,2023ZhangWritingAssistance,cheng2023adapting,2023HuaTuoWang}." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2304.06975" id="4750">
        <attvalues>
          <attvalue for="5" value=" Recently, there has also been a growing trend towards developing specialist models using instruction tuning \cite{2023JiaoParroT,2023WangInstructUIE,2023ZhangWritingAssistance,cheng2023adapting,2023HuaTuoWang}." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2304.03277" id="4751">
        <attvalues>
          <attvalue for="5" value=" Our research is motivated by the existence of various broad-coverage general-purpose instruction-following datasets \cite{2023TaoriAlpaca,2023PengGPT4Instruct,dolly,2023XuWizardLM,zhou2023lima,su2023pandagpt} and their surprising efficiency for turning LLMs into capable instruction-following generalists." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2304.12244" id="4752">
        <attvalues>
          <attvalue for="5" value=" Our research is motivated by the existence of various broad-coverage general-purpose instruction-following datasets \cite{2023TaoriAlpaca,2023PengGPT4Instruct,dolly,2023XuWizardLM,zhou2023lima,su2023pandagpt} and their surprising efficiency for turning LLMs into capable instruction-following generalists." />
        </attvalues>
      </edge>
      <edge source="2310.15326" target="2305.16355" id="4753">
        <attvalues>
          <attvalue for="5" value=" Our research is motivated by the existence of various broad-coverage general-purpose instruction-following datasets \cite{2023TaoriAlpaca,2023PengGPT4Instruct,dolly,2023XuWizardLM,zhou2023lima,su2023pandagpt} and their surprising efficiency for turning LLMs into capable instruction-following generalists." />
        </attvalues>
      </edge>
      <edge source="2312.12810" target="2306.00996" id="4754">
        <attvalues>
          <attvalue for="5" value=" \cite{kouzelis2023weakly} recently proposed a time-accurate and silence-aware neural forced aligner, where a weighted finite-state transducer (WFST) is introduced for modeling dysfluency patterns such as repetition." />
        </attvalues>
      </edge>
      <edge source="2312.12810" target="2303.00747" id="4756">
        <attvalues>
          <attvalue for="5" value=" WhisperX\cite{bain2023whisperx} recently extends Whisper~\cite{radford2022whisper} by generating timestamps for individual words." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="1503.02531" id="4757">
        <attvalues>
          <attvalue for="5" value=" The knowledge from the teacher model is transferred at the output \cite{KD} or intermediate feature level \cite{romero2014fitnets}." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="1412.6550" id="4758">
        <attvalues>
          <attvalue for="5" value=" The knowledge from the teacher model is transferred at the output \cite{KD} or intermediate feature level \cite{romero2014fitnets}." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2006.05525" id="4759">
        <attvalues>
          <attvalue for="5" value=" A common technique is to transfer the knowledge using teachers' posterior distribution \cite{gou2021knowledge}." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2001.01798" id="4760">
        <attvalues>
          <attvalue for="5" value=" KD is used for many tasks such as domain adaptation \cite{asami2017domain,meng2019domain,zhu2020domain}, domain generalisation \cite{wang2021embracing,kim2021domain, fang2021mosaicking}, and model compression \cite{chebotar2016distilling, kim2019knowledge, takashima2018investigation}." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2011.02782" id="4761">
        <attvalues>
          <attvalue for="5" value=" KD is used for many tasks such as domain adaptation \cite{asami2017domain,meng2019domain,zhu2020domain}, domain generalisation \cite{wang2021embracing,kim2021domain, fang2021mosaicking}, and model compression \cite{chebotar2016distilling, kim2019knowledge, takashima2018investigation}." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2107.02629" id="4762">
        <attvalues>
          <attvalue for="5" value=" KD is used for many tasks such as domain adaptation \cite{asami2017domain,meng2019domain,zhu2020domain}, domain generalisation \cite{wang2021embracing,kim2021domain, fang2021mosaicking}, and model compression \cite{chebotar2016distilling, kim2019knowledge, takashima2018investigation}." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2111.06531" id="4763">
        <attvalues>
          <attvalue for="5" value=" KD is used for many tasks such as domain adaptation \cite{asami2017domain,meng2019domain,zhu2020domain}, domain generalisation \cite{wang2021embracing,kim2021domain, fang2021mosaicking}, and model compression \cite{chebotar2016distilling, kim2019knowledge, takashima2018investigation}." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="1606.07947" id="4764">
        <attvalues>
          <attvalue for="5" value=" Sequence level KD was first proposed in \cite{kim2016sequence}, where the teacher models provide sequence-level probability distribution over the whole sample space for better knowledge transfer." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2005.09310" id="4765">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{gao2021distilling} proposed three types of selection strategies at the output of teachers, eg weighted, Top-1 and Top-K." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2008.03822" id="4766">
        <attvalues>
          <attvalue for="5" value=" Various models such as BERT\cite{futami2020distilling}, BLSTMs \cite{8639629} with different context windows have been used in ASR knowledge distillation tasks." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2006.11477" id="4767">
        <attvalues>
          <attvalue for="5" value=" Such a model is trained with masked spans for generalised contextual latent representation of speech \cite{baevski2020wav2vec}." />
        </attvalues>
      </edge>
      <edge source="2303.00550" target="2104.01027" id="4768">
        <attvalues>
          <attvalue for="5" value=" These self-supervised models have been observed to be quickly adaptable to new domains or cross-domain tasks \cite{hsu2021robust}." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2203.12277" id="4769">
        <attvalues>
          <attvalue for="5" value=" Several improved methods~\cite{lu-etal-2022-unified,DBLP:journals/corr/abs-2301-03282,wei2023zeroshot,wang2023instructuie} have been proposed for the unified modeling of information extraction tasks, including prompt-based extractive and generative models." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2209.02693" id="4770">
        <attvalues>
          <attvalue for="5" value=" The primary studies~\cite{jiang-etal-2021-named,li2022unified,DBLP:conf/cikm/XuSLF18,ye-etal-2022-packed,cao-etal-2022-oneee,sheng-etal-2021-casee,Zhang2022OptimizingBF,tang-etal-2022-unirel,xu-etal-2022-extracting} of information extraction are task-specialized, which results in dedicated architectures, isolated models, and specialized knowledge sources for different IE tasks." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2106.08977" id="4771">
        <attvalues>
          <attvalue for="5" value=" The primary studies~\cite{jiang-etal-2021-named,li2022unified,DBLP:conf/cikm/XuSLF18,ye-etal-2022-packed,cao-etal-2022-oneee,sheng-etal-2021-casee,Zhang2022OptimizingBF,tang-etal-2022-unirel,xu-etal-2022-extracting} of information extraction are task-specialized, which results in dedicated architectures, isolated models, and specialized knowledge sources for different IE tasks." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2301.03282" id="4772">
        <attvalues>
          <attvalue for="5" value=" Several improved methods~\cite{lu-etal-2022-unified,DBLP:journals/corr/abs-2301-03282,wei2023zeroshot,wang2023instructuie} have been proposed for the unified modeling of information extraction tasks, including prompt-based extractive and generative models." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2210.02414" id="4773">
        <attvalues>
          <attvalue for="5" value=" IE tasks are highly diversified due to their varying targets (entities, relations, events, etc), heterogeneous structures (spans, triplets, records, etc), and domain-specific schemas~\cite{DBLP:journals/corr/abs-2301-03282,DBLP:journals/corr/abs-2210-02414,DBLP:conf/acl/DuQLDQY022}." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2103.10360" id="4774">
        <attvalues>
          <attvalue for="5" value=" IE tasks are highly diversified due to their varying targets (entities, relations, events, etc), heterogeneous structures (spans, triplets, records, etc), and domain-specific schemas~\cite{DBLP:journals/corr/abs-2301-03282,DBLP:journals/corr/abs-2210-02414,DBLP:conf/acl/DuQLDQY022}." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2112.10070" id="4775">
        <attvalues>
          <attvalue for="5" value=" The primary studies~\cite{jiang-etal-2021-named,li2022unified,DBLP:conf/cikm/XuSLF18,ye-etal-2022-packed,cao-etal-2022-oneee,sheng-etal-2021-casee,Zhang2022OptimizingBF,tang-etal-2022-unirel,xu-etal-2022-extracting} of information extraction are task-specialized, which results in dedicated architectures, isolated models, and specialized knowledge sources for different IE tasks." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2109.06067" id="4776">
        <attvalues>
          <attvalue for="5" value=" The primary studies~\cite{jiang-etal-2021-named,li2022unified,DBLP:conf/cikm/XuSLF18,ye-etal-2022-packed,cao-etal-2022-oneee,sheng-etal-2021-casee,Zhang2022OptimizingBF,tang-etal-2022-unirel,xu-etal-2022-extracting} of information extraction are task-specialized, which results in dedicated architectures, isolated models, and specialized knowledge sources for different IE tasks." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2107.01583" id="4777">
        <attvalues>
          <attvalue for="5" value=" The primary studies~\cite{jiang-etal-2021-named,li2022unified,DBLP:conf/cikm/XuSLF18,ye-etal-2022-packed,cao-etal-2022-oneee,sheng-etal-2021-casee,Zhang2022OptimizingBF,tang-etal-2022-unirel,xu-etal-2022-extracting} of information extraction are task-specialized, which results in dedicated architectures, isolated models, and specialized knowledge sources for different IE tasks." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2211.09039" id="4779">
        <attvalues>
          <attvalue for="5" value=" The primary studies~\cite{jiang-etal-2021-named,li2022unified,DBLP:conf/cikm/XuSLF18,ye-etal-2022-packed,cao-etal-2022-oneee,sheng-etal-2021-casee,Zhang2022OptimizingBF,tang-etal-2022-unirel,xu-etal-2022-extracting} of information extraction are task-specialized, which results in dedicated architectures, isolated models, and specialized knowledge sources for different IE tasks." />
        </attvalues>
      </edge>
      <edge source="2403.05132" target="2304.08085" id="4780">
        <attvalues>
          <attvalue for="5" value=" Several improved methods~\cite{lu-etal-2022-unified,DBLP:journals/corr/abs-2301-03282,wei2023zeroshot,wang2023instructuie} have been proposed for the unified modeling of information extraction tasks, including prompt-based extractive and generative models." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2312.11805" id="4781">
        <attvalues>
          <attvalue for="5" value="0-pro \cite{Gemini2023}, Llama 3 8b Instruct \cite{llama3}, and Mistral 7b Instruct \cite{mistral}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2310.06825" id="4782">
        <attvalues>
          <attvalue for="5" value="0-pro \cite{Gemini2023}, Llama 3 8b Instruct \cite{llama3}, and Mistral 7b Instruct \cite{mistral}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2107.13586" id="4783">
        <attvalues>
          <attvalue for="5" value=" Prompting strategies have emerged as a promising avenue for improving LLM performance by providing concise and informative input \cite{Liu+2023,Brown+2020, Jiang+23, ge+24}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2005.14165" id="4784">
        <attvalues>
          <attvalue for="5" value=" Prompting strategies have emerged as a promising avenue for improving LLM performance by providing concise and informative input \cite{Liu+2023,Brown+2020, Jiang+23, ge+24}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2404.02060" id="4785">
        <attvalues>
          <attvalue for="5" value=" These studies often involve defining and exploring overly complex problems such as those about extreme-label classification \cite{Li+2024} or ``Needle In a Haystack&quot; \cite{MachlabB2024}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2004.15011" id="4786">
        <attvalues>
          <attvalue for="5" value=" Extractive Summarization methods such as TextRank \cite{mihalceaT2004} are widely used to identify and extract the most significant sentences from a document for different purposes \cite{CacholaLCW2020, FengFQ22, BalcerzakJW2014, Wang+2020}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2004.05150" id="4787">
        <attvalues>
          <attvalue for="5" value=" There is a body of research dedicated to studying the limitations of LLMs on long sequences and proposing mitigations at both architecture-level \cite{BeltagyPC2020,BertschANG2024} as well as prompt-level \cite{Wei+2022}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2305.01625" id="4788">
        <attvalues>
          <attvalue for="5" value=" There is a body of research dedicated to studying the limitations of LLMs on long sequences and proposing mitigations at both architecture-level \cite{BeltagyPC2020,BertschANG2024} as well as prompt-level \cite{Wei+2022}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2201.11903" id="4789">
        <attvalues>
          <attvalue for="5" value=" There is a body of research dedicated to studying the limitations of LLMs on long sequences and proposing mitigations at both architecture-level \cite{BeltagyPC2020,BertschANG2024} as well as prompt-level \cite{Wei+2022}." />
        </attvalues>
      </edge>
      <edge source="2408.01866" target="2404.08865" id="4790">
        <attvalues>
          <attvalue for="5" value=" These studies often involve defining and exploring overly complex problems such as those about extreme-label classification \cite{Li+2024} or ``Needle In a Haystack&quot; \cite{MachlabB2024}." />
        </attvalues>
      </edge>
      <edge source="2305.14993" target="1910.04387" id="4805">
        <attvalues>
          <attvalue for="5" value=" At a low level, one can control complexity by describing the nature of simplification operations to be performed \cite{mallinson2019controllable, martin2020controllable}." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2307.03109" id="4806">
        <attvalues>
          <attvalue for="5" value=" This expectation raises a profound challenge -- how do we evaluate and rank the quality of different LMs over a variety of capabilities? This is a complex evaluation endeavor \cite{chang2023survey}, as it transcends the boundaries of a specific task and seeks to measure the overall capabilities of an LM over a wide manifold of natural language tasks." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2206.04615" id="4808">
        <attvalues>
          <attvalue for="5" value=" To this end, LM benchmarks are constantly being proposed, where each new benchmark further expands the coverage and diversity of evaluated tasks and settings \cite{wang-etal-2018-glue,bigbench,lmeval-harness,talmor-etal-2020-olmpics,yuan2023evaluating,zhang2023dialogstudio}." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2308.01240" id="4809">
        <attvalues>
          <attvalue for="5" value=" To this end, LM benchmarks are constantly being proposed, where each new benchmark further expands the coverage and diversity of evaluated tasks and settings \cite{wang-etal-2018-glue,bigbench,lmeval-harness,talmor-etal-2020-olmpics,yuan2023evaluating,zhang2023dialogstudio}." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2307.10172" id="4810">
        <attvalues>
          <attvalue for="5" value=" To this end, LM benchmarks are constantly being proposed, where each new benchmark further expands the coverage and diversity of evaluated tasks and settings \cite{wang-etal-2018-glue,bigbench,lmeval-harness,talmor-etal-2020-olmpics,yuan2023evaluating,zhang2023dialogstudio}." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2304.11158" id="4812">
        <attvalues>
          <attvalue for="5" value=" Running such expansive benchmarks can entail spending \$$10$K+ or $4$K+ GPU hours for evaluating a single model \cite{liang2022holistic}, and may even surpass those of pretraining \cite{biderman2023emergent} when evaluating checkpoints." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2203.15556" id="4813">
        <attvalues>
          <attvalue for="5" value=" While the trade-off between computation and performance is usually discussed in the context of pre-training (eg, scaling laws; \cite{hoffmann2022trainingChinchilla,ivgi2022scaling}) and fine-tuning (eg, parameter efficient; \cite{lialin2023scaling}), here we call for putting this trade-off on the center stage of evaluation design." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2202.06387" id="4814">
        <attvalues>
          <attvalue for="5" value=" While the trade-off between computation and performance is usually discussed in the context of pre-training (eg, scaling laws; \cite{hoffmann2022trainingChinchilla,ivgi2022scaling}) and fine-tuning (eg, parameter efficient; \cite{lialin2023scaling}), here we call for putting this trade-off on the center stage of evaluation design." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2303.15647" id="4815">
        <attvalues>
          <attvalue for="5" value=" While the trade-off between computation and performance is usually discussed in the context of pre-training (eg, scaling laws; \cite{hoffmann2022trainingChinchilla,ivgi2022scaling}) and fine-tuning (eg, parameter efficient; \cite{lialin2023scaling}), here we call for putting this trade-off on the center stage of evaluation design." />
        </attvalues>
      </edge>
      <edge source="2308.11696" target="2108.12284" id="4816">
        <attvalues>
          <attvalue for="5" value=" In practice, the compute side of the trade-off already plays a role in most large-scale evaluation decisions, both in benchmark design \cite{liang2022holistic} and in its use for evaluation \cite[eg, choosing the number of seeds;][]{csordas2021devil, choshen2022start}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="1706.03762" id="4818">
        <attvalues>
          <attvalue for="5" value=" However, the limited context length imposed by the transformer architecture \cite{vaswani2017attention, huang2018music}, the absence of ICL abilities in moderately large language models \cite{lu2023emergent}, the quadratic increase in computational cost with an increase in context length (or demonstrations in ICL) \cite{keles2023computational}, and the sensitivity of ICL performance \cite{bertsch2024context} present challenges in the utility, reliability, and efficiency of ICL." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2211.05100" id="4819">
        <attvalues>
          <attvalue for="5" value=" Five years in, publicly available models grew to 176 billion parameters \cite{bloom,zhang2022opt,zeng2022glm130b}, eg by a factor of 500." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2205.01068" id="4820">
        <attvalues>
          <attvalue for="5" value=" Five years in, publicly available models grew to 176 billion parameters \cite{bloom,zhang2022opt,zeng2022glm130b}, eg by a factor of 500." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2210.02414" id="4821">
        <attvalues>
          <attvalue for="5" value=" Five years in, publicly available models grew to 176 billion parameters \cite{bloom,zhang2022opt,zeng2022glm130b}, eg by a factor of 500." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2204.02311" id="4822">
        <attvalues>
          <attvalue for="5" value=" Published literature includes models up to 1 trillion parameters \cite{palm,megatron,switch}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="1909.08053" id="4823">
        <attvalues>
          <attvalue for="5" value=" Published literature includes models up to 1 trillion parameters \cite{palm,megatron,switch}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2101.03961" id="4824">
        <attvalues>
          <attvalue for="5" value=" Published literature includes models up to 1 trillion parameters \cite{palm,megatron,switch}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2309.01809" id="4825">
        <attvalues>
          <attvalue for="5" value=" However, the limited context length imposed by the transformer architecture \cite{vaswani2017attention, huang2018music}, the absence of ICL abilities in moderately large language models \cite{lu2023emergent}, the quadratic increase in computational cost with an increase in context length (or demonstrations in ICL) \cite{keles2023computational}, and the sensitivity of ICL performance \cite{bertsch2024context} present challenges in the utility, reliability, and efficiency of ICL." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2209.04881" id="4826">
        <attvalues>
          <attvalue for="5" value=" However, the limited context length imposed by the transformer architecture \cite{vaswani2017attention, huang2018music}, the absence of ICL abilities in moderately large language models \cite{lu2023emergent}, the quadratic increase in computational cost with an increase in context length (or demonstrations in ICL) \cite{keles2023computational}, and the sensitivity of ICL performance \cite{bertsch2024context} present challenges in the utility, reliability, and efficiency of ICL." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2203.06904" id="4828">
        <attvalues>
          <attvalue for="5" value=" In the last few years, more than a hundred PEFT papers have been published, with several studies \cite{delta_tuning} providing a good overview of the most popular methods, such as Adapters \cite{adapters}, BitFit \cite{bitfit}, LoRA \cite{lora}, Compacter \cite{compacter}, and Soft Prompts \cite{p_tuning,prefix_tuning}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="1902.00751" id="4829">
        <attvalues>
          <attvalue for="5" value=" In the last few years, more than a hundred PEFT papers have been published, with several studies \cite{delta_tuning} providing a good overview of the most popular methods, such as Adapters \cite{adapters}, BitFit \cite{bitfit}, LoRA \cite{lora}, Compacter \cite{compacter}, and Soft Prompts \cite{p_tuning,prefix_tuning}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2106.10199" id="4830">
        <attvalues>
          <attvalue for="5" value=" In the last few years, more than a hundred PEFT papers have been published, with several studies \cite{delta_tuning} providing a good overview of the most popular methods, such as Adapters \cite{adapters}, BitFit \cite{bitfit}, LoRA \cite{lora}, Compacter \cite{compacter}, and Soft Prompts \cite{p_tuning,prefix_tuning}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2106.09685" id="4831">
        <attvalues>
          <attvalue for="5" value=" In the last few years, more than a hundred PEFT papers have been published, with several studies \cite{delta_tuning} providing a good overview of the most popular methods, such as Adapters \cite{adapters}, BitFit \cite{bitfit}, LoRA \cite{lora}, Compacter \cite{compacter}, and Soft Prompts \cite{p_tuning,prefix_tuning}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2106.04647" id="4832">
        <attvalues>
          <attvalue for="5" value=" In the last few years, more than a hundred PEFT papers have been published, with several studies \cite{delta_tuning} providing a good overview of the most popular methods, such as Adapters \cite{adapters}, BitFit \cite{bitfit}, LoRA \cite{lora}, Compacter \cite{compacter}, and Soft Prompts \cite{p_tuning,prefix_tuning}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2103.10385" id="4833">
        <attvalues>
          <attvalue for="5" value=" In the last few years, more than a hundred PEFT papers have been published, with several studies \cite{delta_tuning} providing a good overview of the most popular methods, such as Adapters \cite{adapters}, BitFit \cite{bitfit}, LoRA \cite{lora}, Compacter \cite{compacter}, and Soft Prompts \cite{p_tuning,prefix_tuning}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2101.00190" id="4834">
        <attvalues>
          <attvalue for="5" value=" In the last few years, more than a hundred PEFT papers have been published, with several studies \cite{delta_tuning} providing a good overview of the most popular methods, such as Adapters \cite{adapters}, BitFit \cite{bitfit}, LoRA \cite{lora}, Compacter \cite{compacter}, and Soft Prompts \cite{p_tuning,prefix_tuning}." />
        </attvalues>
      </edge>
      <edge source="2303.15647" target="2302.11529" id="4835">
        <attvalues>
          <attvalue for="5" value=" \cite{modular_deep_learning} presented a survey on modular deep learning, providing an overview of several similar methods from the perspective of modularity and multi-task inference." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="2202.01094" id="4836">
        <attvalues>
          <attvalue for="5" value=" The advantages of using discriminative loss functions such as MWER \cite{xu2022rescorebert,gandhe2020audio} to train language models are clear." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="1912.03363" id="4837">
        <attvalues>
          <attvalue for="5" value=" The advantages of using discriminative loss functions such as MWER \cite{xu2022rescorebert,gandhe2020audio} to train language models are clear." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="1712.01818" id="4838">
        <attvalues>
          <attvalue for="5" value=" There have been several attempts at incorporating discriminative training using MWER loss for LSTM-RNN \cite{mwer2016,gandhe2020audio}, Sequence-to-Sequence \cite{mwer2018,li2020towards,sainath2019two}, RNN-Transducer \cite{guo2020efficient}, Transformer-Transducer \cite{meng2021minimum} based architectures for ASR." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="2004.11544" id="4839">
        <attvalues>
          <attvalue for="5" value=" There have been several attempts at incorporating discriminative training using MWER loss for LSTM-RNN \cite{mwer2016,gandhe2020audio}, Sequence-to-Sequence \cite{mwer2018,li2020towards,sainath2019two}, RNN-Transducer \cite{guo2020efficient}, Transformer-Transducer \cite{meng2021minimum} based architectures for ASR." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="1908.10992" id="4840">
        <attvalues>
          <attvalue for="5" value=" There have been several attempts at incorporating discriminative training using MWER loss for LSTM-RNN \cite{mwer2016,gandhe2020audio}, Sequence-to-Sequence \cite{mwer2018,li2020towards,sainath2019two}, RNN-Transducer \cite{guo2020efficient}, Transformer-Transducer \cite{meng2021minimum} based architectures for ASR." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="2106.02302" id="4842">
        <attvalues>
          <attvalue for="5" value=" There have been several attempts at incorporating discriminative training using MWER loss for LSTM-RNN \cite{mwer2016,gandhe2020audio}, Sequence-to-Sequence \cite{mwer2018,li2020towards,sainath2019two}, RNN-Transducer \cite{guo2020efficient}, Transformer-Transducer \cite{meng2021minimum} based architectures for ASR." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="1910.14659" id="4844">
        <attvalues>
          <attvalue for="5" value=" \cite{salazar2020masked} proposed to distill pseudo log-likelihoods (PLL) from BERT \cite{mlm-scoring1} using L2-loss based regression over the classification (CLS) token in application to utterance rescoring for ASR." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="1503.02531" id="4845">
        <attvalues>
          <attvalue for="5" value=" \cite{hinton2015distilling} proposed cross-entropy based distillation based on KL-divergence loss for distilling from an ensemble of teacher models for classification problems." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="1412.6550" id="4846">
        <attvalues>
          <attvalue for="5" value=" \cite{romero2014fitnets} extended knowledge distillation by minimizing L2-norm of the intermediate representations from hidden layers between the teacher and the student." />
        </attvalues>
      </edge>
      <edge source="2306.09452" target="1709.02929" id="4847">
        <attvalues>
          <attvalue for="5" value=" L2-norm based objectives are also used to closely replicate real value predictions of teacher in application to regression problems \cite{chen2017learning,wang2017model}." />
        </attvalues>
      </edge>
      <edge source="2112.11638" target="1810.06065" id="4849">
        <attvalues>
          <attvalue for="5" value=" A summary is assessed by evaluating its qualities, which can be defined in different ways \cite{Fan:2018:Robust, Xenouleas:2019:SUMQE, Kryscinski:2020:EvaluatingFactual, Vasilyev:2020:Sensitivity, Fabbri:2021:SummEval}." />
        </attvalues>
      </edge>
      <edge source="2112.11638" target="2007.12626" id="4850">
        <attvalues>
          <attvalue for="5" value=" Correlation of evaluation measures with human scores for all qualities is widely accepted as a criterion for judging about the evaluation measures \cite{Fabbri:2021:SummEval}, with a few caveats." />
        </attvalues>
      </edge>
      <edge source="2112.11638" target="2012.14602" id="4851">
        <attvalues>
          <attvalue for="5" value=" There is also a possibility of an implicit bias even in expert scores \cite{Vasilyev:2021:NoHuman}." />
        </attvalues>
      </edge>
      <edge source="2307.03972" target="2103.10360" id="4854">
        <attvalues>
          <attvalue for="5" value=" Following the technical trajectory of ChatGPT, a significant number of high-quality LLMs have emerged in recent times in both academia and industry, such as LLaMA~\cite{DBLP:journals/corr/abs-2302-13971}, ChatGLM~\cite{du2022glm} and PaLM~\cite{DBLP:journals/corr/abs-2305-10403}." />
        </attvalues>
      </edge>
      <edge source="2307.03972" target="2305.10403" id="4855">
        <attvalues>
          <attvalue for="5" value=" Following the technical trajectory of ChatGPT, a significant number of high-quality LLMs have emerged in recent times in both academia and industry, such as LLaMA~\cite{DBLP:journals/corr/abs-2302-13971}, ChatGLM~\cite{du2022glm} and PaLM~\cite{DBLP:journals/corr/abs-2305-10403}." />
        </attvalues>
      </edge>
      <edge source="2307.03972" target="2302.08081" id="4856">
        <attvalues>
          <attvalue for="5" value=" Previous studies found that these LLMs have achieved great performance on a wide range of NLP tasks, including machine translation~\cite{DBLP:journals/corr/abs-2301-08745}, named entity recognition~\cite{DBLP:journals/corr/abs-2305-05862} and text summarization~\cite{DBLP:journals/corr/abs-2302-08081}." />
        </attvalues>
      </edge>
      <edge source="2307.03972" target="2304.01746" id="4857">
        <attvalues>
          <attvalue for="5" value=" Certain studies have token comprehensive investigations into the performance of LLMs in the domain of English grammatical error correction, yielding some interesting findings~\cite{DBLP:journals/corr/abs-2304-01746,DBLP:journals/corr/abs-2303-13648}." />
        </attvalues>
      </edge>
      <edge source="2307.03972" target="2303.13648" id="4858">
        <attvalues>
          <attvalue for="5" value=" Certain studies have token comprehensive investigations into the performance of LLMs in the domain of English grammatical error correction, yielding some interesting findings~\cite{DBLP:journals/corr/abs-2304-01746,DBLP:journals/corr/abs-2303-13648}." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2303.08774" id="4859">
        <attvalues>
          <attvalue for="5" value=" A third dimension to the problem is the level of access to the LLM itself in closed-data setups: white-box detection where we have full access to the model, {eg,}\ a local model, and black-box detection where only the API access is available, {eg,}\ GPT-4 \cite{achiam2023gpt}." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2201.11903" id="4860">
        <attvalues>
          <attvalue for="5" value=" In the rapidly evolving landscape of artificial intelligence (AI), Large Language Models (LLMs) have emerged as pivotal tools, driving innovation across a wide spectrum of applications, from natural language processing (NLP) and automated content creation \cite{achiam2023gpt,betker2023improving} to complex decision-making systems and autonomous agents \cite{wei2022chain,li2023chain,yang2023auto,wu2023autogen}." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2306.02224" id="4861">
        <attvalues>
          <attvalue for="5" value=" In the rapidly evolving landscape of artificial intelligence (AI), Large Language Models (LLMs) have emerged as pivotal tools, driving innovation across a wide spectrum of applications, from natural language processing (NLP) and automated content creation \cite{achiam2023gpt,betker2023improving} to complex decision-making systems and autonomous agents \cite{wei2022chain,li2023chain,yang2023auto,wu2023autogen}." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2101.00027" id="4863">
        <attvalues>
          <attvalue for="5" value=" At their core, these models rely on extensive datasets \cite{commoncrawl,gao2020pile} to learn about language and the world, and generate responses that are increasingly indistinguishable from human-authored output \cite{instructgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2307.09288" id="4865">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2312.11805" id="4866">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2401.04088" id="4867">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2401.02954" id="4869">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2403.08295" id="4870">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2311.16989" id="4871">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="1803.05457" id="4872">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="1905.07830" id="4873">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2009.03300" id="4874">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2109.07958" id="4875">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2404.00699" target="2110.14168" id="4877">
        <attvalues>
          <attvalue for="5" value=" Lastly, in the current intense race to build the most powerful LLM \cite{touvron2023llama,gemini,jiang2024mixtral,bai2023qwen,bi2024deepseek,young2024yi,Gemma2024OpenModels,chen2023chatgpt}, the community is struggling to settle on a fixed subset of benchmarks as LLMs performance increases fast \cite{arc,hellaswag,mmlu,truthfulqa,winogrande,gsm8k}, an issue which is further fueled by underlying contamination." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="1611.01368" id="4878">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) generate complex and largely grammatical strings and display impressive performance with structures traditionally thought to require abstract and hierarchical syntax \cite{linzen2016assessing,linzen2021syntactic,wilcoxLI,futrell2019rnns}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2004.10827" id="4879">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) generate complex and largely grammatical strings and display impressive performance with structures traditionally thought to require abstract and hierarchical syntax \cite{linzen2016assessing,linzen2021syntactic,wilcoxLI,futrell2019rnns}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="1811.01866" id="4880">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) generate complex and largely grammatical strings and display impressive performance with structures traditionally thought to require abstract and hierarchical syntax \cite{linzen2016assessing,linzen2021syntactic,wilcoxLI,futrell2019rnns}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2303.12712" id="4881">
        <attvalues>
          <attvalue for="5" value=" They have achieved human-like performance at a wide range of natural language tasks \cite{bubeck2023sparks,frank_2023}, particularly those having to do with linguistic form \cite{mahowald2023dissociating}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="1909.00111" id="4884">
        <attvalues>
          <attvalue for="5" value=" One reason that has been posited not to take LLMs seriously as cognitive models, though, is the immense amount of data they are trained on relative to what a human child is exposed to \cite{warstadt2022what,van-schijndel-etal-2019-quantity}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="1810.12247" id="4885">
        <attvalues>
          <attvalue for="5" value=" We submitted to both strict tracks --- however, we were notified through the meta-review that our models qualify only for the loose track due to the usage of additional non-linguistic data (music from the MAESTRO dataset~\cite{hawthorne2018enabling})." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="1912.00582" id="4886">
        <attvalues>
          <attvalue for="5" value=" Inspired by this idea, targeting the BLiMP \cite{warstadt2020blimp} syntactic evaluations as well as more general tasks, we trained with a targeted MLM objective." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="1905.00537" id="4887">
        <attvalues>
          <attvalue for="5" value=" The evaluation is on a set of natural language tasks including grammatical acceptability judgments via minimal pairs in the BLiMP benchmark \cite{warstadt2020blimp}, language understanding tasks in SuperGLUE \cite{wang2019superglue}, and MSGS (the Mixed Signals Generalization Set) \cite{warstadt-etal-2020-learning} We started with a baseline DeBERTa model, trained from scratch on BabyLM data using a custom unigram SentencePiece tokenizer ~\cite{kudo-richardson-2018-sentencepiece}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2010.05358" id="4888">
        <attvalues>
          <attvalue for="5" value=" The evaluation is on a set of natural language tasks including grammatical acceptability judgments via minimal pairs in the BLiMP benchmark \cite{warstadt2020blimp}, language understanding tasks in SuperGLUE \cite{wang2019superglue}, and MSGS (the Mixed Signals Generalization Set) \cite{warstadt-etal-2020-learning} We started with a baseline DeBERTa model, trained from scratch on BabyLM data using a custom unigram SentencePiece tokenizer ~\cite{kudo-richardson-2018-sentencepiece}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="1808.06226" id="4889">
        <attvalues>
          <attvalue for="5" value=" The evaluation is on a set of natural language tasks including grammatical acceptability judgments via minimal pairs in the BLiMP benchmark \cite{warstadt2020blimp}, language understanding tasks in SuperGLUE \cite{wang2019superglue}, and MSGS (the Mixed Signals Generalization Set) \cite{warstadt-etal-2020-learning} We started with a baseline DeBERTa model, trained from scratch on BabyLM data using a custom unigram SentencePiece tokenizer ~\cite{kudo-richardson-2018-sentencepiece}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2305.19905" id="4890">
        <attvalues>
          <attvalue for="5" value=" To that end, \cite{mueller-linzen-2023-plant} showed that training on simpler data first could induce a better hierarchical bias for learning language." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2004.14601" id="4892">
        <attvalues>
          <attvalue for="5" value=" \cite{papadimitriou-jurafsky-2020-learning} use this idea to show that training language models on structured data (eg, music) can help models learn faster." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2003.10555" id="4893">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, replaced token detection approaches such as ELECTRA \cite{Clark2020ELECTRA} suffer from an inability to learn probability distributions over the entire vocabulary, and so cannot be used for (pseudo)-likelihood scoring \cite{salazar-etal-2020-masked}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="1910.14659" id="4894">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, replaced token detection approaches such as ELECTRA \cite{Clark2020ELECTRA} suffer from an inability to learn probability distributions over the entire vocabulary, and so cannot be used for (pseudo)-likelihood scoring \cite{salazar-etal-2020-masked}." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2204.06644" id="4895">
        <attvalues>
          <attvalue for="5" value=" Another related approach is Corrective Language Modeling (CLM) \cite{DBLP:journals/corr/abs-2204-06644}, in which the model is trained to correctly replace corrupted tokens; however, it is not clear how to best use these models for scoring sentences in BLiMP." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2210.11771" id="4896">
        <attvalues>
          <attvalue for="5" value=" Other strategies for selecting masks were used in \cite{sadeq-etal-2022-informask,gu-etal-2020-train}; unlike these works, we mask specific words which are essential to the phenomena in BLiMP." />
        </attvalues>
      </edge>
      <edge source="2310.17591" target="2004.09733" id="4897">
        <attvalues>
          <attvalue for="5" value=" Other strategies for selecting masks were used in \cite{sadeq-etal-2022-informask,gu-etal-2020-train}; unlike these works, we mask specific words which are essential to the phenomena in BLiMP." />
        </attvalues>
      </edge>
      <edge source="2305.19905" target="1808.09031" id="4899">
        <attvalues>
          <attvalue for="5" value=" This hypothesis is supported by the finding that, given minimal pairs of grammatical and ungrammatical sentences, the probability distribution over sentences defined by LMs often favors the grammatical sentence \cite{marvin-linzen-2018-targeted,hu-etal-2020-systematic}." />
        </attvalues>
      </edge>
      <edge source="2305.19905" target="2007.06761" id="4901">
        <attvalues>
          <attvalue for="5" value=" A related line of work has shown that, through pre-training, LMs can under some circumstances acquire syntactic inductive biases which are then applied to fine-tuning tasks, whereas models which have not been pre-trained do not have such inductive biases (\cite{warstadt-bowman-2020-linguistic,warstadt-etal-2020-learning,lovering2021predicting,mueller-etal-2022-coloring})." />
        </attvalues>
      </edge>
      <edge source="2305.19905" target="2010.05358" id="4902">
        <attvalues>
          <attvalue for="5" value=" A related line of work has shown that, through pre-training, LMs can under some circumstances acquire syntactic inductive biases which are then applied to fine-tuning tasks, whereas models which have not been pre-trained do not have such inductive biases (\cite{warstadt-bowman-2020-linguistic,warstadt-etal-2020-learning,lovering2021predicting,mueller-etal-2022-coloring})." />
        </attvalues>
      </edge>
      <edge source="2305.19905" target="2203.09397" id="4903">
        <attvalues>
          <attvalue for="5" value=" A related line of work has shown that, through pre-training, LMs can under some circumstances acquire syntactic inductive biases which are then applied to fine-tuning tasks, whereas models which have not been pre-trained do not have such inductive biases (\cite{warstadt-bowman-2020-linguistic,warstadt-etal-2020-learning,lovering2021predicting,mueller-etal-2022-coloring})." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1903.07785" id="4905">
        <attvalues>
          <attvalue for="5" value=" Pre-trained Language Models (PLMs) have achieved superior performances on various NLP tasks~\cite{baevski2019cloze,joshi2019spanbert,liu2019roberta,yang2019xlnet,Clark2020ELECTRA:} and have attracted wide research interests." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1907.10529" id="4906">
        <attvalues>
          <attvalue for="5" value=" Pre-trained Language Models (PLMs) have achieved superior performances on various NLP tasks~\cite{baevski2019cloze,joshi2019spanbert,liu2019roberta,yang2019xlnet,Clark2020ELECTRA:} and have attracted wide research interests." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1907.11692" id="4907">
        <attvalues>
          <attvalue for="5" value=" Pre-trained Language Models (PLMs) have achieved superior performances on various NLP tasks~\cite{baevski2019cloze,joshi2019spanbert,liu2019roberta,yang2019xlnet,Clark2020ELECTRA:} and have attracted wide research interests." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="2003.10555" id="4909">
        <attvalues>
          <attvalue for="5" value=" Pre-trained Language Models (PLMs) have achieved superior performances on various NLP tasks~\cite{baevski2019cloze,joshi2019spanbert,liu2019roberta,yang2019xlnet,Clark2020ELECTRA:} and have attracted wide research interests." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1904.05342" id="4912">
        <attvalues>
          <attvalue for="5" value=" To learn domain-specific language patterns, some previous works~\cite{Beltagy2019SciBERT,huang2019clinicalbert} pre-train a BERT-like model from scratch using large-scale in-domain data." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1811.01088" id="4913">
        <attvalues>
          <attvalue for="5" value=" To learn task-specific language patterns, some previous works~\cite{phang2018sentence} add intermediate supervised pre-training after general pre-training, whose pre-training task is similar to the downstream task but has a larger dataset." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1812.10860" id="4914">
        <attvalues>
          <attvalue for="5" value=" However, \cite{wang2019can} shows that this kind of intermediate pre-training often negatively impacts the transferability to downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1904.03323" id="4915">
        <attvalues>
          <attvalue for="5" value=" In this way, PLMs can utilize more data to better learn domain-specific language patterns~\cite{alsentzer-etal-2019-publicly,Lee2019BioBERT,sung-etal-2019-pre,xu-etal-2019-doubletransfer, aharoni2020unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1906.04382" id="4917">
        <attvalues>
          <attvalue for="5" value=" In this way, PLMs can utilize more data to better learn domain-specific language patterns~\cite{alsentzer-etal-2019-publicly,Lee2019BioBERT,sung-etal-2019-pre,xu-etal-2019-doubletransfer, aharoni2020unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="1911.02493" id="4918">
        <attvalues>
          <attvalue for="5" value=" For instance, in sentiment analysis, sentiment tokens such as ``like'' and ``hate'' are critical for sentiments classification~\cite{ke2019sentilr}." />
        </attvalues>
      </edge>
      <edge source="2004.09733" target="cs/0506075" id="4919">
        <attvalues>
          <attvalue for="5" value=" We conduct experiments on two sentiment analysis tasks: MR~\cite{pang2005seeing} and SemEval14 task 4~\cite{pontiki-etal-2014-semeval}." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1412.2306" id="4920">
        <attvalues>
          <attvalue for="5" value=" These models were first developed for captions descriptive of images \cite{socher2014grounded,karpathy2015deep} and more recently for spoken descriptions of the images (eg, \cite{harwath2015deep,harwath2016unsupervised,synnaeve2014learning,chrupala2017representations,kamper2019semantic})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1511.03690" id="4921">
        <attvalues>
          <attvalue for="5" value=" These models were first developed for captions descriptive of images \cite{socher2014grounded,karpathy2015deep} and more recently for spoken descriptions of the images (eg, \cite{harwath2015deep,harwath2016unsupervised,synnaeve2014learning,chrupala2017representations,kamper2019semantic})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1702.01991" id="4922">
        <attvalues>
          <attvalue for="5" value=" In this context, it is of great interest how the representations emerging from training of such multimodal models relate to the known linguistic structure of the input language (eg, \cite{chrupala2017representations,alishahi2017encoding,harwath2019learning,havard2019models,havard2019word}), and how such methods can support (or replace) purely audio-based representation learning approaches (eg, \cite{Chung2019a,oord2018cpc})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1710.01949" id="4923">
        <attvalues>
          <attvalue for="5" value=" These models were first developed for captions descriptive of images \cite{socher2014grounded,karpathy2015deep} and more recently for spoken descriptions of the images (eg, \cite{harwath2015deep,harwath2016unsupervised,synnaeve2014learning,chrupala2017representations,kamper2019semantic})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1706.03815" id="4924">
        <attvalues>
          <attvalue for="5" value=" In this context, it is of great interest how the representations emerging from training of such multimodal models relate to the known linguistic structure of the input language (eg, \cite{chrupala2017representations,alishahi2017encoding,harwath2019learning,havard2019models,havard2019word}), and how such methods can support (or replace) purely audio-based representation learning approaches (eg, \cite{Chung2019a,oord2018cpc})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1911.09602" id="4925">
        <attvalues>
          <attvalue for="5" value=" In this context, it is of great interest how the representations emerging from training of such multimodal models relate to the known linguistic structure of the input language (eg, \cite{chrupala2017representations,alishahi2017encoding,harwath2019learning,havard2019models,havard2019word}), and how such methods can support (or replace) purely audio-based representation learning approaches (eg, \cite{Chung2019a,oord2018cpc})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1902.03052" id="4926">
        <attvalues>
          <attvalue for="5" value=" In this context, it is of great interest how the representations emerging from training of such multimodal models relate to the known linguistic structure of the input language (eg, \cite{chrupala2017representations,alishahi2017encoding,harwath2019learning,havard2019models,havard2019word}), and how such methods can support (or replace) purely audio-based representation learning approaches (eg, \cite{Chung2019a,oord2018cpc})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1909.08491" id="4927">
        <attvalues>
          <attvalue for="5" value=" In this context, it is of great interest how the representations emerging from training of such multimodal models relate to the known linguistic structure of the input language (eg, \cite{chrupala2017representations,alishahi2017encoding,harwath2019learning,havard2019models,havard2019word}), and how such methods can support (or replace) purely audio-based representation learning approaches (eg, \cite{Chung2019a,oord2018cpc})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1904.03240" id="4928">
        <attvalues>
          <attvalue for="5" value=" In this context, it is of great interest how the representations emerging from training of such multimodal models relate to the known linguistic structure of the input language (eg, \cite{chrupala2017representations,alishahi2017encoding,harwath2019learning,havard2019models,havard2019word}), and how such methods can support (or replace) purely audio-based representation learning approaches (eg, \cite{Chung2019a,oord2018cpc})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1807.03748" id="4929">
        <attvalues>
          <attvalue for="5" value=" In this context, it is of great interest how the representations emerging from training of such multimodal models relate to the known linguistic structure of the input language (eg, \cite{chrupala2017representations,alishahi2017encoding,harwath2019learning,havard2019models,havard2019word}), and how such methods can support (or replace) purely audio-based representation learning approaches (eg, \cite{Chung2019a,oord2018cpc})." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="2104.13225" id="4930">
        <attvalues>
          <attvalue for="5" value=" However, the research in this direction is still young and largely driven by a few research groups \cite{chrupala2021visually}." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="1712.04313" id="4931">
        <attvalues>
          <attvalue for="5" value=" In contrast to the earlier Zerospeech-challenges \cite{versteegh2015zero,dunbar2017zero,dunbar2019zero,nguyen2020zero} that have purely focused on audio-based learning of linguistic representations (including the speech-based track of the current challenge, on which we build), ZR-2021VG takes a step towards multimodal language learning by asking participants to train on audiovisual data." />
        </attvalues>
      </edge>
      <edge source="2107.06546" target="2011.11588" id="4933">
        <attvalues>
          <attvalue for="5" value=" In contrast to the earlier Zerospeech-challenges \cite{versteegh2015zero,dunbar2017zero,dunbar2019zero,nguyen2020zero} that have purely focused on audio-based learning of linguistic representations (including the speech-based track of the current challenge, on which we build), ZR-2021VG takes a step towards multimodal language learning by asking participants to train on audiovisual data." />
        </attvalues>
      </edge>
      <edge source="2204.07779" target="2005.02324" id="4935">
        <attvalues>
          <attvalue for="5" value=" \cite{jiang2020neural} \shortcite{jiang2020neural} initialized the encoder with BERT and proposed a seq2seq model." />
        </attvalues>
      </edge>
      <edge source="2204.07779" target="2004.09733" id="4937">
        <attvalues>
          <attvalue for="5" value=" We are inspired by the works of \cite{gururangan2020don} \shortcite{gururangan2020don} and \cite{gu2020train} \shortcite{gu2020train}." />
        </attvalues>
      </edge>
      <edge source="2406.17789" target="2303.18223" id="4939">
        <attvalues>
          <attvalue for="5" value=" However, LLMs have limitations and their performance has to be well understood before using them on a given application \cite{LLM_survey2}." />
        </attvalues>
      </edge>
      <edge source="2406.17789" target="1905.07830" id="4941">
        <attvalues>
          <attvalue for="5" value=" For example, there are tests to evaluate the capabilities of LLMs to solve common sense reasoning problems \cite{zellers2019hellaswag} or to answer mathematical questions \cite{Mathmeasuring}." />
        </attvalues>
      </edge>
      <edge source="2406.17789" target="2103.03874" id="4942">
        <attvalues>
          <attvalue for="5" value=" For example, there are tests to evaluate the capabilities of LLMs to solve common sense reasoning problems \cite{zellers2019hellaswag} or to answer mathematical questions \cite{Mathmeasuring}." />
        </attvalues>
      </edge>
      <edge source="2406.17789" target="2009.03300" id="4943">
        <attvalues>
          <attvalue for="5" value=" In this work we perform an initial analysis of the impact of automatic translation on one of the most widely used LLM benchmarks, the Multitask Language Understanding (MMLU) test \cite{MMLU} for one of the most commonly used and chosen as a second language to learn, Spanish." />
        </attvalues>
      </edge>
      <edge source="2406.17789" target="2206.04615" id="4944">
        <attvalues>
          <attvalue for="5" value=" To provide a more comprehensive evaluation, some benchmarks evaluate several tasks, for example, the Multitask Language Understanding (MMLU) test \cite{MMLU} evaluates 57 different topics and other benchmarks increase the number of tasks and topics to more than 200 \cite{BIGMeasuring}." />
        </attvalues>
      </edge>
      <edge source="2406.17789" target="2307.16039" id="4945">
        <attvalues>
          <attvalue for="5" value=" To be able to evaluate LLMs in many languages, and given the large number of questions of the benchmarks, the standard procedure is to translate the English test to the target language using automatic translation tools, for example, in the Okapi project \cite{lai2023okapi} three benchmarks from the Open LLM Leaderboard \cite{open-llm-leaderboard} are translated using ChatGPT while in the evaluation of GPT4, the tests were translated using Azure Translator \cite{GPT4}." />
        </attvalues>
      </edge>
      <edge source="2406.17789" target="2303.08774" id="4946">
        <attvalues>
          <attvalue for="5" value=" To be able to evaluate LLMs in many languages, and given the large number of questions of the benchmarks, the standard procedure is to translate the English test to the target language using automatic translation tools, for example, in the Okapi project \cite{lai2023okapi} three benchmarks from the Open LLM Leaderboard \cite{open-llm-leaderboard} are translated using ChatGPT while in the evaluation of GPT4, the tests were translated using Azure Translator \cite{GPT4}." />
        </attvalues>
      </edge>
      <edge source="2109.07780" target="1609.08144" id="4948">
        <attvalues>
          <attvalue for="5" value=" Recent years have seen a surge of interest in neural machine translation (NMT,~\cite{luong2015effective,wu2016google,gehring2017convolutional,transformer}) where it benefits from a massive amount of training data." />
        </attvalues>
      </edge>
      <edge source="2109.07780" target="1706.03762" id="4950">
        <attvalues>
          <attvalue for="5" value=" Experimental results show that the proposed bidirectional training (BiT) consistently and significantly improves the translation performance over the strong Transformer~\cite{transformer}." />
        </attvalues>
      </edge>
      <edge source="2109.07780" target="1511.06709" id="4951">
        <attvalues>
          <attvalue for="5" value=" Although many approaches about fully exploiting the parallel and monolingual data are proposed, eg back translation~\cite{sennrich-etal-2016-improving}, knowledge distillation~\cite{kim-rush-2016-sequence} and data diversification~\cite{nguyen2019data}, the prerequisite of these approaches is to build a well-performed baseline model based on the parallel data." />
        </attvalues>
      </edge>
      <edge source="2109.07780" target="1606.07947" id="4952">
        <attvalues>
          <attvalue for="5" value=" Although many approaches about fully exploiting the parallel and monolingual data are proposed, eg back translation~\cite{sennrich-etal-2016-improving}, knowledge distillation~\cite{kim-rush-2016-sequence} and data diversification~\cite{nguyen2019data}, the prerequisite of these approaches is to build a well-performed baseline model based on the parallel data." />
        </attvalues>
      </edge>
      <edge source="2109.07780" target="1911.01986" id="4953">
        <attvalues>
          <attvalue for="5" value=" Although many approaches about fully exploiting the parallel and monolingual data are proposed, eg back translation~\cite{sennrich-etal-2016-improving}, knowledge distillation~\cite{kim-rush-2016-sequence} and data diversification~\cite{nguyen2019data}, the prerequisite of these approaches is to build a well-performed baseline model based on the parallel data." />
        </attvalues>
      </edge>
      <edge source="2109.07780" target="1611.00179" id="4954">
        <attvalues>
          <attvalue for="5" value=" In the context of machine translation, both the source$arrow$target and target$arrow$source language mappings may benefit bilingual modeling, which motivates many recent studies, eg dual learning~\cite{he2016dual} and symmetric training~\cite{cohn2016incorporating,Liang2007AgreementBasedL}." />
        </attvalues>
      </edge>
      <edge source="2109.07780" target="1601.01085" id="4955">
        <attvalues>
          <attvalue for="5" value=" In the context of machine translation, both the source$arrow$target and target$arrow$source language mappings may benefit bilingual modeling, which motivates many recent studies, eg dual learning~\cite{he2016dual} and symmetric training~\cite{cohn2016incorporating,Liang2007AgreementBasedL}." />
        </attvalues>
      </edge>
      <edge source="2109.07780" target="1806.01258" id="4956">
        <attvalues>
          <attvalue for="5" value=" In the context of machine translation, both the source$arrow$target and target$arrow$source language mappings may benefit bilingual modeling, which motivates many recent studies, eg dual learning~\cite{he2016dual} and symmetric training~\cite{cohn2016incorporating,Liang2007AgreementBasedL}." />
        </attvalues>
      </edge>
      <edge source="2108.13811" target="2004.08056" id="4962">
        <attvalues>
          <attvalue for="5" value=" Another branch is BERT-based \cite{kenton2019bert} methods \cite{yu2020dialogue, xue2022embarrassingly}." />
        </attvalues>
      </edge>
      <edge source="2108.13811" target="1801.06176" id="4963">
        <attvalues>
          <attvalue for="5" value=" The problem itself is well-motivated, because relations between entities in dialogues could potentially provide dialogue systems with additional features for better dialogue managing~\cite{peng2018deep, su2018discriminative} or response generation~\cite{su2018natural}." />
        </attvalues>
      </edge>
      <edge source="2108.13811" target="1808.02747" id="4964">
        <attvalues>
          <attvalue for="5" value=" The problem itself is well-motivated, because relations between entities in dialogues could potentially provide dialogue systems with additional features for better dialogue managing~\cite{peng2018deep, su2018discriminative} or response generation~\cite{su2018natural}." />
        </attvalues>
      </edge>
      <edge source="2108.13811" target="2012.02553" id="4965">
        <attvalues>
          <attvalue for="5" value=" There are two popular datasets, DialogRE~\cite{yu2020dialogue} and DDRel~\cite{jia2021ddrel}, focusing on relation extraction in dialogues illustrated in Figure \ref{fig:example}." />
        </attvalues>
      </edge>
      <edge source="2108.13811" target="2109.05126" id="4966">
        <attvalues>
          <attvalue for="5" value=" However, \cite{albalak2022d} is the only prior work that tried to explicitly leverage such signal for improving DRE, because such explanation annotations may not be always available~\cite{kung2020zero}." />
        </attvalues>
      </edge>
      <edge source="2108.13811" target="2012.06780" id="4968">
        <attvalues>
          <attvalue for="5" value=" DHGAT \cite{chen2020dialogue} presents an attention-based heterogeneous graph network to model multiple types of features; GDPNet \cite{xue2021gdpnet} constructs latent multi-view graphs to model possible relationships among tokens in a long sequence, and then refines the graphs by iterative graph convolution and pooling techniques." />
        </attvalues>
      </edge>
      <edge source="2108.13811" target="2012.13873" id="4970">
        <attvalues>
          <attvalue for="5" value=" SimepleRE \cite{xue2022embarrassingly} is a simple BERT model with an additional refinement gate for iteratively finding high-confidence prediction." />
        </attvalues>
      </edge>
      <edge source="2108.13811" target="2005.06312" id="4971">
        <attvalues>
          <attvalue for="5" value=" LSR \cite{nan2020reasoning} is a latent structure refinement method for better reasoning in the document-level relation extraction task." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2009.11462" id="4972">
        <attvalues>
          <attvalue for="5" value=" The challenge lies in preserving performance while effectively mitigating their potential toxicity~\cite{gehman-etal-2020-realtoxicityprompts,xu-etal-2021-detoxifying,welbl-etal-2021-challenges-detoxifying,hartvigsen2022toxigen,hosseini2023empirical,welleck2023generating}, a concern at the forefront of modern LLM development." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2104.06390" id="4973">
        <attvalues>
          <attvalue for="5" value=" The challenge lies in preserving performance while effectively mitigating their potential toxicity~\cite{gehman-etal-2020-realtoxicityprompts,xu-etal-2021-detoxifying,welbl-etal-2021-challenges-detoxifying,hartvigsen2022toxigen,hosseini2023empirical,welleck2023generating}, a concern at the forefront of modern LLM development." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2109.07445" id="4974">
        <attvalues>
          <attvalue for="5" value=" The challenge lies in preserving performance while effectively mitigating their potential toxicity~\cite{gehman-etal-2020-realtoxicityprompts,xu-etal-2021-detoxifying,welbl-etal-2021-challenges-detoxifying,hartvigsen2022toxigen,hosseini2023empirical,welleck2023generating}, a concern at the forefront of modern LLM development." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2301.09211" id="4975">
        <attvalues>
          <attvalue for="5" value=" The challenge lies in preserving performance while effectively mitigating their potential toxicity~\cite{gehman-etal-2020-realtoxicityprompts,xu-etal-2021-detoxifying,welbl-etal-2021-challenges-detoxifying,hartvigsen2022toxigen,hosseini2023empirical,welleck2023generating}, a concern at the forefront of modern LLM development." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2211.00053" id="4976">
        <attvalues>
          <attvalue for="5" value=" \\ We propose a holistic framework for implicit knowledge editing, modifying language at the stylistic level—a move toward rendering LLMs more ``politically correct'' on ambiguous topics, as opposed to silencing them entirely~\cite{tang2023detoxify,welleck2023generating}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2105.03023" id="4977">
        <attvalues>
          <attvalue for="5" value=" Second, post-processing relies on subjective heuristics, limiting both utility and scalability~\cite{liu-etal-2021-dexperts,kumar-etal-2023-controlled,hallinan-etal-2023-detoxifying}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2305.19230" id="4978">
        <attvalues>
          <attvalue for="5" value=" Second, post-processing relies on subjective heuristics, limiting both utility and scalability~\cite{liu-etal-2021-dexperts,kumar-etal-2023-controlled,hallinan-etal-2023-detoxifying}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2212.10543" id="4979">
        <attvalues>
          <attvalue for="5" value=" Second, post-processing relies on subjective heuristics, limiting both utility and scalability~\cite{liu-etal-2021-dexperts,kumar-etal-2023-controlled,hallinan-etal-2023-detoxifying}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2307.15043" id="4980">
        <attvalues>
          <attvalue for="5" value=" Despite shared concerns regarding toxicity, existing approaches tend toward superficial censorship, often prompting LLMs to avoid sensitive topics altogether, limiting applicability for marginalized groups and inadvertently allowing for implicit toxicity~\cite{zou2023universal,deshpande-etal-2023-toxicity,wei2023jailbroken,liu2023jailbreaking}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2304.05335" id="4981">
        <attvalues>
          <attvalue for="5" value=" Despite shared concerns regarding toxicity, existing approaches tend toward superficial censorship, often prompting LLMs to avoid sensitive topics altogether, limiting applicability for marginalized groups and inadvertently allowing for implicit toxicity~\cite{zou2023universal,deshpande-etal-2023-toxicity,wei2023jailbroken,liu2023jailbreaking}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2307.02483" id="4982">
        <attvalues>
          <attvalue for="5" value=" Despite shared concerns regarding toxicity, existing approaches tend toward superficial censorship, often prompting LLMs to avoid sensitive topics altogether, limiting applicability for marginalized groups and inadvertently allowing for implicit toxicity~\cite{zou2023universal,deshpande-etal-2023-toxicity,wei2023jailbroken,liu2023jailbreaking}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2305.13860" id="4983">
        <attvalues>
          <attvalue for="5" value=" Despite shared concerns regarding toxicity, existing approaches tend toward superficial censorship, often prompting LLMs to avoid sensitive topics altogether, limiting applicability for marginalized groups and inadvertently allowing for implicit toxicity~\cite{zou2023universal,deshpande-etal-2023-toxicity,wei2023jailbroken,liu2023jailbreaking}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="1706.03741" id="4984">
        <attvalues>
          <attvalue for="5" value=" Recently, there has been increased interest in the research community in LLM alignment, that is, training techniques to align model output to the user’s intent, such as Reinforcement Learning through Human (RLHF)~\cite{NIPS2017_d5e2c0ad} Feedback and Direct Preference Optimization (DPO)~\cite{rafailov2023direct,bai2022constitutional}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2305.18290" id="4985">
        <attvalues>
          <attvalue for="5" value=" Recently, there has been increased interest in the research community in LLM alignment, that is, training techniques to align model output to the user’s intent, such as Reinforcement Learning through Human (RLHF)~\cite{NIPS2017_d5e2c0ad} Feedback and Direct Preference Optimization (DPO)~\cite{rafailov2023direct,bai2022constitutional}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2212.08073" id="4986">
        <attvalues>
          <attvalue for="5" value=" Recently, there has been increased interest in the research community in LLM alignment, that is, training techniques to align model output to the user’s intent, such as Reinforcement Learning through Human (RLHF)~\cite{NIPS2017_d5e2c0ad} Feedback and Direct Preference Optimization (DPO)~\cite{rafailov2023direct,bai2022constitutional}." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2308.13449" id="4987">
        <attvalues>
          <attvalue for="5" value=" ~\cite{bekbayev2023poison} show in their work that aligning LLMs by reinforcing models not to respond to specific user inputs degraded the performance." />
        </attvalues>
      </edge>
      <edge source="2401.08491" target="2204.05862" id="4988">
        <attvalues>
          <attvalue for="5" value=" In contrast, \cite{bai2022training} show that degradation or improvement in performance by alignment is dependent on model size." />
        </attvalues>
      </edge>
      <edge source="2007.01359" target="1602.01925" id="4989">
        <attvalues>
          <attvalue for="5" value=" The common approach is to first train a multilingual language model that aims to capture the semantic relations of words in context, independent of the language~\cite{Ammar:2016:MMWE,Mikel:2019:MMS,Unicoder:2019,Alexis:2020:XLR,Feng:2020:LaBSE}." />
        </attvalues>
      </edge>
      <edge source="2007.01359" target="1812.10464" id="4990">
        <attvalues>
          <attvalue for="5" value=" The common approach is to first train a multilingual language model that aims to capture the semantic relations of words in context, independent of the language~\cite{Ammar:2016:MMWE,Mikel:2019:MMS,Unicoder:2019,Alexis:2020:XLR,Feng:2020:LaBSE}." />
        </attvalues>
      </edge>
      <edge source="2007.01359" target="1911.02116" id="4992">
        <attvalues>
          <attvalue for="5" value=" The common approach is to first train a multilingual language model that aims to capture the semantic relations of words in context, independent of the language~\cite{Ammar:2016:MMWE,Mikel:2019:MMS,Unicoder:2019,Alexis:2020:XLR,Feng:2020:LaBSE}." />
        </attvalues>
      </edge>
      <edge source="2007.01359" target="2007.01852" id="4993">
        <attvalues>
          <attvalue for="5" value=" The common approach is to first train a multilingual language model that aims to capture the semantic relations of words in context, independent of the language~\cite{Ammar:2016:MMWE,Mikel:2019:MMS,Unicoder:2019,Alexis:2020:XLR,Feng:2020:LaBSE}." />
        </attvalues>
      </edge>
      <edge source="2007.01359" target="1909.00437" id="4994">
        <attvalues>
          <attvalue for="5" value=" Such a multilingual model can then later be either (i) fine-tuned for classification~\cite{Siddhant:2019:MMTE} task using labelled examples from source language(s), or (ii) used to extract low-dimensional embeddings (representations) for documents from both source and target languages~\cite{Reimers:2020:Making}; the embeddings from source language(s) together with annotated labels are then used for training a light-weight independent classifier for cross-lingual topic ID, which is then used to classify embeddings from target languages." />
        </attvalues>
      </edge>
      <edge source="2007.01359" target="2004.09813" id="4995">
        <attvalues>
          <attvalue for="5" value=" Such a multilingual model can then later be either (i) fine-tuned for classification~\cite{Siddhant:2019:MMTE} task using labelled examples from source language(s), or (ii) used to extract low-dimensional embeddings (representations) for documents from both source and target languages~\cite{Reimers:2020:Making}; the embeddings from source language(s) together with annotated labels are then used for training a light-weight independent classifier for cross-lingual topic ID, which is then used to classify embeddings from target languages." />
        </attvalues>
      </edge>
      <edge source="2209.12616" target="1802.05365" id="4996">
        <attvalues>
          <attvalue for="5" value=" Language model (LM) pretraining has become one of the most common strategies within the natural language processing (NLP) community to solve downstream tasks \cite{peters-etal-2018-deep, howard-ruder-2018-universal, radford2018improving, radford2019language, devlin2018bert}." />
        </attvalues>
      </edge>
      <edge source="2209.12616" target="1912.01703" id="4997">
        <attvalues>
          <attvalue for="5" value=" As a system design, T-NER is implemented in Pytorch \cite{paszke2019pytorch} on top of the Transformers library \cite{Wolf2019HuggingFacesTS}." />
        </attvalues>
      </edge>
      <edge source="2209.12616" target="1910.03771" id="4998">
        <attvalues>
          <attvalue for="5" value=" As a system design, T-NER is implemented in Pytorch \cite{paszke2019pytorch} on top of the Transformers library \cite{Wolf2019HuggingFacesTS}." />
        </attvalues>
      </edge>
      <edge source="2209.12616" target="1201.0490" id="4999">
        <attvalues>
          <attvalue for="5" value=" Moreover, the interfaces of our training and evaluation modules are highly inspired by Scikit-learn \cite{pedregosa2011scikit}, enabling an interoperability with recent models as well as integrating them in an intuitive way." />
        </attvalues>
      </edge>
      <edge source="2208.01307" target="1809.06142" id="5000">
        <attvalues>
          <attvalue for="5" value=" We then leverage existing gold subtitle translations~\cite{opensubtitle} in Chinese and Farsi to project our annotations, resulting in a multilingual corpus (Fig." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1409.3215" id="5001">
        <attvalues>
          <attvalue for="5" value="} Recurrent models (RNNs and more specifically LSTMs) have been used extensively across several NLP tasks such as machine translation \cite{sutskever2014sequence}, language modeling \cite{melis2017state} and question answering \cite{seo2016bidirectional}." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1707.05589" id="5002">
        <attvalues>
          <attvalue for="5" value="} Recurrent models (RNNs and more specifically LSTMs) have been used extensively across several NLP tasks such as machine translation \cite{sutskever2014sequence}, language modeling \cite{melis2017state} and question answering \cite{seo2016bidirectional}." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1611.01603" id="5003">
        <attvalues>
          <attvalue for="5" value="} Recurrent models (RNNs and more specifically LSTMs) have been used extensively across several NLP tasks such as machine translation \cite{sutskever2014sequence}, language modeling \cite{melis2017state} and question answering \cite{seo2016bidirectional}." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1805.04908" id="5004">
        <attvalues>
          <attvalue for="5" value=" Recently, several works \cite{weiss2018practical,sennhauser-berwick-2018-evaluating,skachkova-etal-2018-closing} have attempted to understand the capabilities of LSTMs by empirically analyzing them on different types of formal languages." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1906.03648" id="5006">
        <attvalues>
          <attvalue for="5" value=" We first evaluate the ability of LSTMs to recognize randomly sampled Dyck-$n$ sequences and find, in contrast to prior works \cite{suzgun2019lstm,suzgun2019memory}, that they can generalize nearly perfectly when the test samples are within the same lengths as seen during training." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1911.03329" id="5007">
        <attvalues>
          <attvalue for="5" value=" We first evaluate the ability of LSTMs to recognize randomly sampled Dyck-$n$ sequences and find, in contrast to prior works \cite{suzgun2019lstm,suzgun2019memory}, that they can generalize nearly perfectly when the test samples are within the same lengths as seen during training." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1910.13466" id="5009">
        <attvalues>
          <attvalue for="5" value=" This prompted the development of memory-augmented variants \cite{joulin2015inferring,suzgun2019memory} of LSTMs which generalize well on Dyck languages but are notoriously hard to train and fail to perform well on practical NLP tasks \cite{NIPS2019_8748}." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1803.11138" id="5010">
        <attvalues>
          <attvalue for="5" value=" On the other hand, despite the limited performance of LSTMs on Dyck languages, several studies \cite{gulordava-etal-2018-colorless,tran-etal-2018-importance} have found that LSTMs perform well in modeling hierarchical structure in natural language data." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1803.03585" id="5011">
        <attvalues>
          <attvalue for="5" value=" On the other hand, despite the limited performance of LSTMs on Dyck languages, several studies \cite{gulordava-etal-2018-colorless,tran-etal-2018-importance} have found that LSTMs perform well in modeling hierarchical structure in natural language data." />
        </attvalues>
      </edge>
      <edge source="2011.03965" target="1706.03762" id="5012">
        <attvalues>
          <attvalue for="5" value=" Since Transformer \cite{vaswani2017attention} is also a dominant model in NLP \cite{devlin-etal-2019-bert}, we include it in our experiments." />
        </attvalues>
      </edge>
      <edge source="2206.05395" target="1703.09902" id="5014">
        <attvalues>
          <attvalue for="5" value=" Recent advances in the field of natural language generation (NLG) \cite{gatt2018survey} have resulted in models able to produce realistic, coherent, and fluent texts in a multitude of natural language processing tasks." />
        </attvalues>
      </edge>
      <edge source="2206.05395" target="1804.06609" id="5015">
        <attvalues>
          <attvalue for="5" value=" Unlike conventional methods which were able to provide fine-grained control over many aspects of the system output including incorporating domain-specific dictionaries, terminology or certain words in the generated output, neural end-to-end approaches remove many of these knobs and switches \cite{post2018fast}." />
        </attvalues>
      </edge>
      <edge source="2206.05395" target="1902.08654" id="5016">
        <attvalues>
          <attvalue for="5" value=" For example, it can help avoid generic and meaningless responses in dialogue systems \cite{see2019makes}, personalize dialogue agents based on user features that lead to more engaging and meaningful conversations \cite{zhang2018personalizing}, ensure non-offensive sentence completion and friendly communication \cite{liu2019rhetorically}, intervene on the system output in interactive scenarios where domain specific terminology must be included in the generated texts \cite{crego2016systran}, or aid in art creation in applications such as poetry generation or assisted story writing \cite{peng2018towards}." />
        </attvalues>
      </edge>
      <edge source="2206.05395" target="1610.05540" id="5018">
        <attvalues>
          <attvalue for="5" value=" For example, it can help avoid generic and meaningless responses in dialogue systems \cite{see2019makes}, personalize dialogue agents based on user features that lead to more engaging and meaningful conversations \cite{zhang2018personalizing}, ensure non-offensive sentence completion and friendly communication \cite{liu2019rhetorically}, intervene on the system output in interactive scenarios where domain specific terminology must be included in the generated texts \cite{crego2016systran}, or aid in art creation in applications such as poetry generation or assisted story writing \cite{peng2018towards}." />
        </attvalues>
      </edge>
      <edge source="2206.05395" target="2012.11635" id="5019">
        <attvalues>
          <attvalue for="5" value=" Moreover, controlling a generic pretrained language model in order to satisfy certain desiderata helps avoid generating toxic content, prevents demographic biases, can steer generations towards desired a topic or style \cite{khalifa2020distributional}, and helps communicate intentions in suitable manners for different situations, target audiences and environments \cite{lample2018multiple}, \cite{li2018delete}." />
        </attvalues>
      </edge>
      <edge source="2206.05395" target="1804.06437" id="5020">
        <attvalues>
          <attvalue for="5" value=" Moreover, controlling a generic pretrained language model in order to satisfy certain desiderata helps avoid generating toxic content, prevents demographic biases, can steer generations towards desired a topic or style \cite{khalifa2020distributional}, and helps communicate intentions in suitable manners for different situations, target audiences and environments \cite{lample2018multiple}, \cite{li2018delete}." />
        </attvalues>
      </edge>
      <edge source="2206.05395" target="1906.01604" id="5021">
        <attvalues>
          <attvalue for="5" value=" For example, commonly used sequential text generation methods and architectures assume a rigid modeling of the output sequence based on an ordering of words, in which tokens are generated progressively one at a time in a standard left-to-right manner \cite{chan2019kermit}." />
        </attvalues>
      </edge>
      <edge source="2404.04212" target="2111.01243" id="5037">
        <attvalues>
          <attvalue for="5" value=" Advances in large-scale pre-trained language models have transformed the field for high-resource languages \cite{min2023recent}, but these data and compute-hungry models are not viable for the more-than-7000 low-resource languages (LRLs) in the world \cite{stap-araabi-2023-chatgpt, robinson-etal-2023-chatgpt, zhang-etal-2023-dont}." />
        </attvalues>
      </edge>
      <edge source="2404.04212" target="2205.11277" id="5038">
        <attvalues>
          <attvalue for="5" value=" Moreover, it did not address truly LRLs \cite{ustun2022does}, nor did it incorporate variation in domains that would allow for an assessment of the models' generalization capabilities." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="1303.5778" id="5039">
        <attvalues>
          <attvalue for="5" value=" However, one of the disadvantages of shallow fusion comes from the fact that most of the E2E systems~\cite{LAS,RNNT, LASFirst} arguably already incorporate an internal LM~\cite{StatelessRNNT,LessIsMore, HAT}." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="1508.07909" id="5040">
        <attvalues>
          <attvalue for="5" value=" Consequently, even if the E2E model is using BPE~\cite{BPE} or graphemenes as output units, the system struggles to recognize them." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="2007.05609" id="5041">
        <attvalues>
          <attvalue for="5" value=" The proposed technique builds upon both density ratio~\cite{dr} and class-based LM tags~\cite{ShallowContextual} to contextualize the E2E models." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="1708.06426" id="5043">
        <attvalues>
          <attvalue for="5" value=" Several early attempts were proposed to integrate external language models (LM) into E2E systems, ranging from shallow or deep fusion~\cite{ShallowDeepFusion, ShallowASR}, to cold fusion~\cite{coldFusion} among many others~\cite{componentFusion,ShallowFusionRNNT}." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="2010.13878" id="5044">
        <attvalues>
          <attvalue for="5" value=" Several early attempts were proposed to integrate external language models (LM) into E2E systems, ranging from shallow or deep fusion~\cite{ShallowDeepFusion, ShallowASR}, to cold fusion~\cite{coldFusion} among many others~\cite{componentFusion,ShallowFusionRNNT}." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="1712.01996" id="5045">
        <attvalues>
          <attvalue for="5" value=" In the ASR domain, \cite{ShallowASR} shown that shallow fusion with same output units in both LM and E2E worked best to incorporate an external language model in some tasks." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="2012.06749" id="5046">
        <attvalues>
          <attvalue for="5" value=" However, one of the disadvantages of shallow fusion comes from the fact that most of the E2E systems~\cite{LAS,RNNT, LASFirst} arguably already incorporate an internal LM~\cite{StatelessRNNT,LessIsMore, HAT}." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="2002.11268" id="5047">
        <attvalues>
          <attvalue for="5" value=" The proposed technique builds upon both density ratio~\cite{dr} and class-based LM tags~\cite{ShallowContextual} to contextualize the E2E models." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="2003.07705" id="5048">
        <attvalues>
          <attvalue for="5" value=" Further works modified the architecture~\cite{HAT} or made some assumptions~\cite{Internal} to better approximate the implicit LM of E2E systems in combination with density ratio." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="2011.01991" id="5049">
        <attvalues>
          <attvalue for="5" value=" Further works modified the architecture~\cite{HAT} or made some assumptions~\cite{Internal} to better approximate the implicit LM of E2E systems in combination with density ratio." />
        </attvalues>
      </edge>
      <edge source="2206.14623" target="2006.03411" id="5051">
        <attvalues>
          <attvalue for="5" value=" In~\cite{CLAS} and~\cite{contextualRNNT}, an additional attention set input is proposed for both attention and recurrent neural transducer (RNN-T) models respectively." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="1904.02619" id="5052">
        <attvalues>
          <attvalue for="5" value=" Among the various model architectures that stand out are: Connectionist Temporal Classification \cite{Graves:06icml}, Listen-Attend-Spell \cite{LAS-2016}, Time-Depth Separable Convolutions \cite{hannun2019sequencetosequence}, Recurrent Neural Network Transducer \cite{graves-2012-sequen-trans}, Transformer \cite{Dong-no-recurrence-s2s-model-2018}, Conformer \cite{gulati2020conformer}, SqueezeFormer \cite{kim2022squeezeformer}, Fast Conformer\cite{rekesh2023fast}, and Zipformer \cite{yao2023zipformer}." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="1211.3711" id="5053">
        <attvalues>
          <attvalue for="5" value=" Among the various model architectures that stand out are: Connectionist Temporal Classification \cite{Graves:06icml}, Listen-Attend-Spell \cite{LAS-2016}, Time-Depth Separable Convolutions \cite{hannun2019sequencetosequence}, Recurrent Neural Network Transducer \cite{graves-2012-sequen-trans}, Transformer \cite{Dong-no-recurrence-s2s-model-2018}, Conformer \cite{gulati2020conformer}, SqueezeFormer \cite{kim2022squeezeformer}, Fast Conformer\cite{rekesh2023fast}, and Zipformer \cite{yao2023zipformer}." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="2005.08100" id="5054">
        <attvalues>
          <attvalue for="5" value=" Among the various model architectures that stand out are: Connectionist Temporal Classification \cite{Graves:06icml}, Listen-Attend-Spell \cite{LAS-2016}, Time-Depth Separable Convolutions \cite{hannun2019sequencetosequence}, Recurrent Neural Network Transducer \cite{graves-2012-sequen-trans}, Transformer \cite{Dong-no-recurrence-s2s-model-2018}, Conformer \cite{gulati2020conformer}, SqueezeFormer \cite{kim2022squeezeformer}, Fast Conformer\cite{rekesh2023fast}, and Zipformer \cite{yao2023zipformer}." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="2206.00888" id="5055">
        <attvalues>
          <attvalue for="5" value=" Among the various model architectures that stand out are: Connectionist Temporal Classification \cite{Graves:06icml}, Listen-Attend-Spell \cite{LAS-2016}, Time-Depth Separable Convolutions \cite{hannun2019sequencetosequence}, Recurrent Neural Network Transducer \cite{graves-2012-sequen-trans}, Transformer \cite{Dong-no-recurrence-s2s-model-2018}, Conformer \cite{gulati2020conformer}, SqueezeFormer \cite{kim2022squeezeformer}, Fast Conformer\cite{rekesh2023fast}, and Zipformer \cite{yao2023zipformer}." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="2305.05084" id="5056">
        <attvalues>
          <attvalue for="5" value=" Among the various model architectures that stand out are: Connectionist Temporal Classification \cite{Graves:06icml}, Listen-Attend-Spell \cite{LAS-2016}, Time-Depth Separable Convolutions \cite{hannun2019sequencetosequence}, Recurrent Neural Network Transducer \cite{graves-2012-sequen-trans}, Transformer \cite{Dong-no-recurrence-s2s-model-2018}, Conformer \cite{gulati2020conformer}, SqueezeFormer \cite{kim2022squeezeformer}, Fast Conformer\cite{rekesh2023fast}, and Zipformer \cite{yao2023zipformer}." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="2310.11230" id="5057">
        <attvalues>
          <attvalue for="5" value=" Among the various model architectures that stand out are: Connectionist Temporal Classification \cite{Graves:06icml}, Listen-Attend-Spell \cite{LAS-2016}, Time-Depth Separable Convolutions \cite{hannun2019sequencetosequence}, Recurrent Neural Network Transducer \cite{graves-2012-sequen-trans}, Transformer \cite{Dong-no-recurrence-s2s-model-2018}, Conformer \cite{gulati2020conformer}, SqueezeFormer \cite{kim2022squeezeformer}, Fast Conformer\cite{rekesh2023fast}, and Zipformer \cite{yao2023zipformer}." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="2111.03250" id="5059">
        <attvalues>
          <attvalue for="5" value=" The first approach modifies the model to integrate user-provided context words \cite{Pundak2018DeepCE,Chang2021ContextAwareTT, MysoreSathyendra2022,fu2023robust}, while the second approach applies an external context score using techniques like shallow fusion or on-the-fly re-scoring \cite{Zhao2019end2end,wang2023contextual}." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="2305.05271" id="5060">
        <attvalues>
          <attvalue for="5" value=" The first approach modifies the model to integrate user-provided context words \cite{Pundak2018DeepCE,Chang2021ContextAwareTT, MysoreSathyendra2022,fu2023robust}, while the second approach applies an external context score using techniques like shallow fusion or on-the-fly re-scoring \cite{Zhao2019end2end,wang2023contextual}." />
        </attvalues>
      </edge>
      <edge source="2402.06592" target="2310.00178" id="5061">
        <attvalues>
          <attvalue for="5" value=" The first approach modifies the model to integrate user-provided context words \cite{Pundak2018DeepCE,Chang2021ContextAwareTT, MysoreSathyendra2022,fu2023robust}, while the second approach applies an external context score using techniques like shallow fusion or on-the-fly re-scoring \cite{Zhao2019end2end,wang2023contextual}." />
        </attvalues>
      </edge>
      <edge source="2405.13181" target="2106.09685" id="5062">
        <attvalues>
          <attvalue for="5" value=" We aimed to do this by employing -- 1) different fine tuning (FT) methods, 2) applying Low-Rank Adaptation - LoRA (\cite{hu2021lora}) adaptors with few-shot learning, and 3) performing context-distillation both with and without few-shot learning setting." />
        </attvalues>
      </edge>
      <edge source="2405.13181" target="1704.05426" id="5063">
        <attvalues>
          <attvalue for="5" value=" We aim to understand the efficacy of alternative fine-tuning methods on a pre-trained large language model’s performance in sequence classification tasks using 2 datasets, namely \href{ (\cite{N18-1101}) and \href{ (\cite{warstadt2018neural}), which are further explained in Section \ref{Datasets}." />
        </attvalues>
      </edge>
      <edge source="2405.13181" target="1805.12471" id="5064">
        <attvalues>
          <attvalue for="5" value=" We aim to understand the efficacy of alternative fine-tuning methods on a pre-trained large language model’s performance in sequence classification tasks using 2 datasets, namely \href{ (\cite{N18-1101}) and \href{ (\cite{warstadt2018neural}), which are further explained in Section \ref{Datasets}." />
        </attvalues>
      </edge>
      <edge source="2405.13181" target="2205.01068" id="5065">
        <attvalues>
          <attvalue for="5" value=" We explored alternate ways of efficiently fine-tuning the model and compared them with the baseline methods (vanilla and pattern-based fine-tuning) for Open Pre-trained Transformer (OPT) (\cite{zhang2022opt}) model's performance on the text sequence classification task using both in-domain and out of domain accuracies." />
        </attvalues>
      </edge>
      <edge source="2405.13181" target="2005.14165" id="5066">
        <attvalues>
          <attvalue for="5" value=" Currently large language models (LLMs) are pre-dominantly used by leveraging In-Context Learning - ICL (\cite{brown2020language}), whereby during the inference time, the model learns to answer follow-up questions from a series of prompts." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2203.06904" id="5067">
        <attvalues>
          <attvalue for="5" value=" Parameter efficient learning methods ({{PERMs}}) serve as potential alternatives to finetuning for adapting and deploying language models in real world scenarios \cite{ding2022delta}." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2005.14165" id="5068">
        <attvalues>
          <attvalue for="5" value=" This is especially important for large language models (eg GPT-3 \cite{brown2020language} and MT-NLG \cite{smith2022using}) as finetuning the entire model will be very expensive or infeasible due to their model size." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2201.11990" id="5069">
        <attvalues>
          <attvalue for="5" value=" This is especially important for large language models (eg GPT-3 \cite{brown2020language} and MT-NLG \cite{smith2022using}) as finetuning the entire model will be very expensive or infeasible due to their model size." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2101.00190" id="5070">
        <attvalues>
          <attvalue for="5" value=" Prefix tuning \cite{li2021prefix}, which is one of the {{PERMs}}, draws inspiration from prompting and introduces a small set of continuous vectors as virtual prompts to allow subsequent tokens to attend to, which obtains comparable performance to finetuning in the full data setting." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2104.08691" id="5071">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning \cite{lester2021power} shows the power of scaling PLMs and that tuning only a few extra embeddings is sufficient to achieve similar performance to finetuning the entire 11b T5-XXL \cite{raffel2020exploring} model." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="1902.00751" id="5073">
        <attvalues>
          <attvalue for="5" value=" Different from adding new parameters through prompts, Adapter \cite{houlsby2019parameter} injects trainable parameters through low-rank structure in a skip-connection way." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2106.09685" id="5074">
        <attvalues>
          <attvalue for="5" value=" Other {{PERMs}} includes LoRA \cite{hu2021lora}, Mix-And-Match adapter \cite{he2021towards}, Compactor \cite{karimi2021compacter}, BitFit \cite{zaken2022bitfit}, diff-pruning \cite{guo2021parameter} and etc Most conclusions about {{PERMs}} so far are drawn from their in-domain evaluations over full training samples." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2110.04366" id="5075">
        <attvalues>
          <attvalue for="5" value=" Other {{PERMs}} includes LoRA \cite{hu2021lora}, Mix-And-Match adapter \cite{he2021towards}, Compactor \cite{karimi2021compacter}, BitFit \cite{zaken2022bitfit}, diff-pruning \cite{guo2021parameter} and etc Most conclusions about {{PERMs}} so far are drawn from their in-domain evaluations over full training samples." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2106.04647" id="5076">
        <attvalues>
          <attvalue for="5" value=" Other {{PERMs}} includes LoRA \cite{hu2021lora}, Mix-And-Match adapter \cite{he2021towards}, Compactor \cite{karimi2021compacter}, BitFit \cite{zaken2022bitfit}, diff-pruning \cite{guo2021parameter} and etc Most conclusions about {{PERMs}} so far are drawn from their in-domain evaluations over full training samples." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2106.10199" id="5077">
        <attvalues>
          <attvalue for="5" value=" Other {{PERMs}} includes LoRA \cite{hu2021lora}, Mix-And-Match adapter \cite{he2021towards}, Compactor \cite{karimi2021compacter}, BitFit \cite{zaken2022bitfit}, diff-pruning \cite{guo2021parameter} and etc Most conclusions about {{PERMs}} so far are drawn from their in-domain evaluations over full training samples." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2012.07463" id="5078">
        <attvalues>
          <attvalue for="5" value=" Other {{PERMs}} includes LoRA \cite{hu2021lora}, Mix-And-Match adapter \cite{he2021towards}, Compactor \cite{karimi2021compacter}, BitFit \cite{zaken2022bitfit}, diff-pruning \cite{guo2021parameter} and etc Most conclusions about {{PERMs}} so far are drawn from their in-domain evaluations over full training samples." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2104.14839" id="5079">
        <attvalues>
          <attvalue for="5" value=" Various efforts are made to systematically measure and mitigate factual errors in many generation tasks, including summarization~\cite{huang2021factual} and dialogue generations~\cite{rashkin2021increasing,shuster2021retrieval,dziri2021neural, wu2021controllable}." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2107.06963" id="5080">
        <attvalues>
          <attvalue for="5" value=" Various efforts are made to systematically measure and mitigate factual errors in many generation tasks, including summarization~\cite{huang2021factual} and dialogue generations~\cite{rashkin2021increasing,shuster2021retrieval,dziri2021neural, wu2021controllable}." />
        </attvalues>
      </edge>
      <edge source="2210.13673" target="2104.07567" id="5081">
        <attvalues>
          <attvalue for="5" value=" Various efforts are made to systematically measure and mitigate factual errors in many generation tasks, including summarization~\cite{huang2021factual} and dialogue generations~\cite{rashkin2021increasing,shuster2021retrieval,dziri2021neural, wu2021controllable}." />
        </attvalues>
      </edge>
      <edge source="2305.10992" target="2109.01819" id="5095">
        <attvalues>
          <attvalue for="5" value=" We conjecture that the main reason behind the performance difference lies in its lack of complexity, eg, the number of classes to be predicted, and similar arguments have been made for auxiliary task ineffectiveness \cite{Lan2020ALBERT} and pretraining task design \cite{yamaguchi-etal-2021-frustratingly}." />
        </attvalues>
      </edge>
      <edge source="2305.10992" target="2203.10415" id="5096">
        <attvalues>
          <attvalue for="5" value=" \cite{yamaguchi-etal-2021-frustratingly} and \cite{alajrami-aletras-2022-pre} have demonstrated that such token-level objectives themselves, eg, pretraining without MLM, perform comparably to MLM." />
        </attvalues>
      </edge>
      <edge source="2305.10992" target="1909.11942" id="5097">
        <attvalues>
          <attvalue for="5" value=" We conjecture that the main reason behind the performance difference lies in its lack of complexity, eg, the number of classes to be predicted, and similar arguments have been made for auxiliary task ineffectiveness \cite{Lan2020ALBERT} and pretraining task design \cite{yamaguchi-etal-2021-frustratingly}." />
        </attvalues>
      </edge>
      <edge source="2305.10992" target="1606.05250" id="5099">
        <attvalues>
          <attvalue for="5" value=" We pretrain 14 different types of models with the proposed control task in addition to MLM for reference and evaluate their downstream performance on the GLUE \cite{wang2018glue}, SQuAD \cite{rajpurkar-etal-2016-squad}, and Universal Dependencies (UD) \cite{nivre-etal-2020-universal} benchmarks." />
        </attvalues>
      </edge>
      <edge source="2305.10992" target="2004.10643" id="5100">
        <attvalues>
          <attvalue for="5" value=" We pretrain 14 different types of models with the proposed control task in addition to MLM for reference and evaluate their downstream performance on the GLUE \cite{wang2018glue}, SQuAD \cite{rajpurkar-etal-2016-squad}, and Universal Dependencies (UD) \cite{nivre-etal-2020-universal} benchmarks." />
        </attvalues>
      </edge>
      <edge source="2005.09946" target="2007.11464" id="5101">
        <attvalues>
          <attvalue for="5" value=" SemEval 2020 Task 1 \cite{schlechtweg2020semeval} addresses the current lack of a systematic approach for the evaluation of automatic methods for the diachronic analysis by proposing a common evaluation framework that comprises two tasks and covers four different languages (German, English, Latin, and Swedish)." />
        </attvalues>
      </edge>
      <edge source="2005.09946" target="1811.06278" id="5102">
        <attvalues>
          <attvalue for="5" value=" Other dynamic approaches incorporate the alignment directly into the learning stage via the optimisation function \cite{Tahmasebi2018}." />
        </attvalues>
      </edge>
      <edge source="2005.09946" target="1703.00607" id="5103">
        <attvalues>
          <attvalue for="5" value=" In this work, we focus on dynamic word embeddings by exploring methods based on both explicit, such as Dynamic Word2Vec \cite{Yao2018}, and implicit alignment, namely Temporal Random Indexing \cite{Basile} and Temporal Referencing \cite{Dubossarsky2019}." />
        </attvalues>
      </edge>
      <edge source="2005.09946" target="1906.01688" id="5104">
        <attvalues>
          <attvalue for="5" value=" In this work, we focus on dynamic word embeddings by exploring methods based on both explicit, such as Dynamic Word2Vec \cite{Yao2018}, and implicit alignment, namely Temporal Random Indexing \cite{Basile} and Temporal Referencing \cite{Dubossarsky2019}." />
        </attvalues>
      </edge>
      <edge source="1909.12289" target="1703.01619" id="5116">
        <attvalues>
          <attvalue for="5" value=" Auto-regressive sequence-to-sequence (seq2seq) models with attention mechanism are widely used in a variety of areas including Neural Machine Translation (NMT) \cite{neubig2017neural,huang2016attention} and speech synthesis \cite{shen2018natural,wang2018style}, also known as Text-To-Speech (TTS)." />
        </attvalues>
      </edge>
      <edge source="1909.12289" target="1712.05884" id="5117">
        <attvalues>
          <attvalue for="5" value=" Auto-regressive sequence-to-sequence (seq2seq) models with attention mechanism are widely used in a variety of areas including Neural Machine Translation (NMT) \cite{neubig2017neural,huang2016attention} and speech synthesis \cite{shen2018natural,wang2018style}, also known as Text-To-Speech (TTS)." />
        </attvalues>
      </edge>
      <edge source="1909.12289" target="1803.09017" id="5118">
        <attvalues>
          <attvalue for="5" value=" Auto-regressive sequence-to-sequence (seq2seq) models with attention mechanism are widely used in a variety of areas including Neural Machine Translation (NMT) \cite{neubig2017neural,huang2016attention} and speech synthesis \cite{shen2018natural,wang2018style}, also known as Text-To-Speech (TTS)." />
        </attvalues>
      </edge>
      <edge source="1909.12289" target="1506.03099" id="5119">
        <attvalues>
          <attvalue for="5" value=" Several approaches are introduced to tackle the above problem, namely scheduled sampling \cite{bengio2015scheduled} and professor forcing \cite{lamb2016professor}." />
        </attvalues>
      </edge>
      <edge source="1909.12289" target="1610.09038" id="5120">
        <attvalues>
          <attvalue for="5" value=" Several approaches are introduced to tackle the above problem, namely scheduled sampling \cite{bengio2015scheduled} and professor forcing \cite{lamb2016professor}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1710.03957" id="5121">
        <attvalues>
          <attvalue for="5" value=" Former dialogue datasets make the end-to-end training of deep neural models possible, such as DailyDialog~\cite{dailydialog}, PersonaChat~\cite{personachat}, etc, and deep neural networks including Seq2seq with attention~\cite{seq2seq}, Transformers~\cite{transformer} have already shown their capability in generating conversation replies." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1409.0473" id="5123">
        <attvalues>
          <attvalue for="5" value=" Former dialogue datasets make the end-to-end training of deep neural models possible, such as DailyDialog~\cite{dailydialog}, PersonaChat~\cite{personachat}, etc, and deep neural networks including Seq2seq with attention~\cite{seq2seq}, Transformers~\cite{transformer} have already shown their capability in generating conversation replies." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1706.03762" id="5124">
        <attvalues>
          <attvalue for="5" value=" Former dialogue datasets make the end-to-end training of deep neural models possible, such as DailyDialog~\cite{dailydialog}, PersonaChat~\cite{personachat}, etc, and deep neural networks including Seq2seq with attention~\cite{seq2seq}, Transformers~\cite{transformer} have already shown their capability in generating conversation replies." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1510.03055" id="5125">
        <attvalues>
          <attvalue for="5" value=" In order to tackle it, some researchers try to refine the training objective with extra constraint items~\cite{diversity_objective} or modify the criterion that encourages models to decode more diverse words~\cite{kurata_generate,kulikov_decoding,li2016simple,song_diversifying}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1701.03185" id="5126">
        <attvalues>
          <attvalue for="5" value=" However, current models tend to produce generic sentences such as ``I don't know'', caused by the current inherent deterministic training objective as well as insufficient diversity and limited quality of current datasets~\cite{diversity_objective,shao_quality}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1811.00907" id="5127">
        <attvalues>
          <attvalue for="5" value=" In order to tackle it, some researchers try to refine the training objective with extra constraint items~\cite{diversity_objective} or modify the criterion that encourages models to decode more diverse words~\cite{kurata_generate,kulikov_decoding,li2016simple,song_diversifying}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1611.08562" id="5128">
        <attvalues>
          <attvalue for="5" value=" In order to tackle it, some researchers try to refine the training objective with extra constraint items~\cite{diversity_objective} or modify the criterion that encourages models to decode more diverse words~\cite{kurata_generate,kulikov_decoding,li2016simple,song_diversifying}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1905.05471" id="5129">
        <attvalues>
          <attvalue for="5" value=" Data filtering is commonly used in machine learning and extended to dialogue generation by removing samples with generic responses based on entropy~\cite{entropy_filter} or the predictions of a Seq2seq model~\cite{data_distillation}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1702.06703" id="5130">
        <attvalues>
          <attvalue for="5" value=" Data filtering is commonly used in machine learning and extended to dialogue generation by removing samples with generic responses based on entropy~\cite{entropy_filter} or the predictions of a Seq2seq model~\cite{data_distillation}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1703.02573" id="5131">
        <attvalues>
          <attvalue for="5" value=" Word replacement extends the data scales by randomly replacing original tokens with others based on vocabulary distributions~\cite{xie_replacement,fadaee_replacement}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1705.00440" id="5132">
        <attvalues>
          <attvalue for="5" value=" Word replacement extends the data scales by randomly replacing original tokens with others based on vocabulary distributions~\cite{xie_replacement,fadaee_replacement}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1710.09412" id="5133">
        <attvalues>
          <attvalue for="5" value=" It is inspired by the recent success of the mixup approach that combines training pairs of samples and labels convexly into a single one~\cite{mixup}." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1805.06201" id="5134">
        <attvalues>
          <attvalue for="5" value=" Compared to previous similar work~\cite{kobayashi_contextual,gao_mixup}, we use both soft embedding and soft labels for a more flexible training process." />
        </attvalues>
      </edge>
      <edge source="2103.01534" target="1905.10523" id="5135">
        <attvalues>
          <attvalue for="5" value=" Compared to previous similar work~\cite{kobayashi_contextual,gao_mixup}, we use both soft embedding and soft labels for a more flexible training process." />
        </attvalues>
      </edge>
      <edge source="2309.02240" target="2202.13675" id="5136">
        <attvalues>
          <attvalue for="5" value=" Dialog policy learning (DPL) aiming to determine the next abstracted system output plays a key role in pipeline task-oriented dialog systems \cite{kwan_survey_2023}." />
        </attvalues>
      </edge>
      <edge source="2309.02240" target="2009.09781" id="5137">
        <attvalues>
          <attvalue for="5" value=" A lot of progress is being made in demonstration-based efficient learning methods \cite{brys2015reinforcement,cederborg2015policy,wang2020learning,li2020rethinking,jhunjhunwala2020multi,geishauser2022dynamic}." />
        </attvalues>
      </edge>
      <edge source="2309.02240" target="2204.05928" id="5138">
        <attvalues>
          <attvalue for="5" value=" A lot of progress is being made in demonstration-based efficient learning methods \cite{brys2015reinforcement,cederborg2015policy,wang2020learning,li2020rethinking,jhunjhunwala2020multi,geishauser2022dynamic}." />
        </attvalues>
      </edge>
      <edge source="2309.02240" target="2002.02450" id="5139">
        <attvalues>
          <attvalue for="5" value=" Recently, the studies on PLMs for dialog, including BERT-based dialog state tracking \cite{gulyaev2020goal} and GPT-2 based dialog generation \cite{peng2020few, yang2020ubar} are not centred on DPL." />
        </attvalues>
      </edge>
      <edge source="2309.02240" target="2002.12328" id="5140">
        <attvalues>
          <attvalue for="5" value=" Recently, the studies on PLMs for dialog, including BERT-based dialog state tracking \cite{gulyaev2020goal} and GPT-2 based dialog generation \cite{peng2020few, yang2020ubar} are not centred on DPL." />
        </attvalues>
      </edge>
      <edge source="2404.19319" target="1503.02531" id="5142">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation \cite[KD;][]{hinton2015distilling,jiao-etal-2020-tinybert} during LM pretraining has emerged as the primary mean of compressing the capabilities of a large pretrained teacher model into a task agnostic smaller student model." />
        </attvalues>
      </edge>
      <edge source="2404.19319" target="1909.10351" id="5143">
        <attvalues>
          <attvalue for="5" value=" \cite{jiao-etal-2020-tinybert} compare their model to BERT\textsubscript{Tiny} \cite{turc2019wellread}, which has the same architecture but employs significantly different training resources than their TinyBERT\textsubscript{Tiny}, preventing a fair comparison." />
        </attvalues>
      </edge>
      <edge source="2404.19319" target="1908.08962" id="5144">
        <attvalues>
          <attvalue for="5" value=" \cite{jiao-etal-2020-tinybert} compare their model to BERT\textsubscript{Tiny} \cite{turc2019wellread}, which has the same architecture but employs significantly different training resources than their TinyBERT\textsubscript{Tiny}, preventing a fair comparison." />
        </attvalues>
      </edge>
      <edge source="2404.19319" target="2004.02984" id="5146">
        <attvalues>
          <attvalue for="5" value=" Similarly, \cite{sanh2020distilbert} compare their distilled student solely against the teacher, whereas \cite{sun-etal-2020-mobilebert, wang2020minilm} only add comparison against larger pretrained models and competing KD strategies." />
        </attvalues>
      </edge>
      <edge source="2404.19319" target="2206.14366" id="5147">
        <attvalues>
          <attvalue for="5" value=" Even the body of work that focuses on comparing different KD strategies has only recently sought to standardize training and thus enable fair comparisons \cite{lu2022knowledge,wang-etal-2023-distill}." />
        </attvalues>
      </edge>
      <edge source="2404.19319" target="2305.15032" id="5148">
        <attvalues>
          <attvalue for="5" value=" Even the body of work that focuses on comparing different KD strategies has only recently sought to standardize training and thus enable fair comparisons \cite{lu2022knowledge,wang-etal-2023-distill}." />
        </attvalues>
      </edge>
      <edge source="2404.19319" target="2001.08361" id="5149">
        <attvalues>
          <attvalue for="5" value=" Scaling laws \cite{kaplan2020scaling, hoffmann2022training}, reveal that, under a fixed computation budget, only a marginal correlation exists between the LM size and it's performance: smaller models compensate their lower learning efficiency with the ability to process more tokens within the same budget." />
        </attvalues>
      </edge>
      <edge source="2404.19319" target="2212.14034" id="5150">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{geiping2022cramming} showcases this behavior by training multiple BERT models with varying architecture sizes for a fix 24 hour duration, resulting in similar loss values across all sizes." />
        </attvalues>
      </edge>
      <edge source="2206.14366" target="2005.14165" id="5152">
        <attvalues>
          <attvalue for="5" value=" Recently, the emergence of pre-trained language models, especially the transformer-based model such as BERT \cite{Devlin2019BERTPO}, and GPT-3 \cite{Brown2020LanguageMA}, has revolutionized the research on various natural language processing (NLP), compute vision (CV), and multimodal tasks \cite{Dosovitskiy2021AnII, Liu2021SwinTH,Lin2021M6AC,wang2022OFA} and achieve stunning success." />
        </attvalues>
      </edge>
      <edge source="2206.14366" target="2010.11929" id="5153">
        <attvalues>
          <attvalue for="5" value=" Recently, the emergence of pre-trained language models, especially the transformer-based model such as BERT \cite{Devlin2019BERTPO}, and GPT-3 \cite{Brown2020LanguageMA}, has revolutionized the research on various natural language processing (NLP), compute vision (CV), and multimodal tasks \cite{Dosovitskiy2021AnII, Liu2021SwinTH,Lin2021M6AC,wang2022OFA} and achieve stunning success." />
        </attvalues>
      </edge>
      <edge source="2206.14366" target="2103.00823" id="5154">
        <attvalues>
          <attvalue for="5" value=" Recently, the emergence of pre-trained language models, especially the transformer-based model such as BERT \cite{Devlin2019BERTPO}, and GPT-3 \cite{Brown2020LanguageMA}, has revolutionized the research on various natural language processing (NLP), compute vision (CV), and multimodal tasks \cite{Dosovitskiy2021AnII, Liu2021SwinTH,Lin2021M6AC,wang2022OFA} and achieve stunning success." />
        </attvalues>
      </edge>
      <edge source="2206.14366" target="1503.02531" id="5155">
        <attvalues>
          <attvalue for="5" value=" Among all these methods, knowledge distillation (KD) \cite{Hinton2015DistillingTK} is simple yet effective and has been frequently used \cite{Wang2020MiniLMDS, Jiao2020TinyBERTDB}." />
        </attvalues>
      </edge>
      <edge source="2206.14366" target="2002.10957" id="5156">
        <attvalues>
          <attvalue for="5" value=" Despite considerable previous literature having grown up to apply knowledge distillation to transformer-based models for model compression \cite{Wang2020MiniLMDS,Jiao2020TinyBERTDB,Sanh2019DistilBERTAD,Sun2020MobileBERTAC}, there are still too many unexplored areas in the mechanism of KD." />
        </attvalues>
      </edge>
      <edge source="2206.14366" target="1909.10351" id="5157">
        <attvalues>
          <attvalue for="5" value=" Despite considerable previous literature having grown up to apply knowledge distillation to transformer-based models for model compression \cite{Wang2020MiniLMDS,Jiao2020TinyBERTDB,Sanh2019DistilBERTAD,Sun2020MobileBERTAC}, there are still too many unexplored areas in the mechanism of KD." />
        </attvalues>
      </edge>
      <edge source="2206.14366" target="2004.02984" id="5159">
        <attvalues>
          <attvalue for="5" value=" Despite considerable previous literature having grown up to apply knowledge distillation to transformer-based models for model compression \cite{Wang2020MiniLMDS,Jiao2020TinyBERTDB,Sanh2019DistilBERTAD,Sun2020MobileBERTAC}, there are still too many unexplored areas in the mechanism of KD." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="1310.4546" id="5160">
        <attvalues>
          <attvalue for="5" value=" These learned embeddings often contain useful properties including semantic regularity, that is words that have similar meaning appear closer to each other in vector space \cite{mikolov2013distributed}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="1802.05365" id="5161">
        <attvalues>
          <attvalue for="5" value=" More recent deep-learning approaches such as ELMo \cite{peters1luke} and BERT \cite{devlin-etal-2019-bert} can further account for context, enabling for example, the disambiguation of homonyms appearing in different sentences." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="2006.11477" id="5163">
        <attvalues>
          <attvalue for="5" value=" The masked language modeling objective popularised by BERT has since been extended to the audio domain to learn contextualized speech representations directly from audio \cite{baevski2020wav2vec, hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="2102.05766" id="5164">
        <attvalues>
          <attvalue for="5" value=" The use-cases of joint speech-text embeddings can be broadly classified into three non-mutually exclusive categories: \begin {enumerate*} [label=\itshape\alph*\upshape)] \item when intending to leverage data from both modalities as input into a single model, such as for multimodal translation \cite{zheng2021fused} \item when learning a semantic alignment between text and speech \cite{duquenne2021multimodal,chung2018unsupervised} that can be useful for data mining or retrieval \item when transferring knowledge encoded in a pretrained model from one modality to another or incorporating knowledge from both modalities, often seen in end-to-end solutions for speech translation (ST) \cite{ye2021end,han2021learning,dong2021listen,Tang2021IST}, SLU \cite{Denisov_2020,chung2020splat} or Q-A \cite{chuang2019speechbert}), that combine semantics from text with acoustics from speech \end{enumerate*}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="1805.07467" id="5165">
        <attvalues>
          <attvalue for="5" value=" The use-cases of joint speech-text embeddings can be broadly classified into three non-mutually exclusive categories: \begin {enumerate*} [label=\itshape\alph*\upshape)] \item when intending to leverage data from both modalities as input into a single model, such as for multimodal translation \cite{zheng2021fused} \item when learning a semantic alignment between text and speech \cite{duquenne2021multimodal,chung2018unsupervised} that can be useful for data mining or retrieval \item when transferring knowledge encoded in a pretrained model from one modality to another or incorporating knowledge from both modalities, often seen in end-to-end solutions for speech translation (ST) \cite{ye2021end,han2021learning,dong2021listen,Tang2021IST}, SLU \cite{Denisov_2020,chung2020splat} or Q-A \cite{chuang2019speechbert}), that combine semantics from text with acoustics from speech \end{enumerate*}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="2104.10380" id="5166">
        <attvalues>
          <attvalue for="5" value=" The use-cases of joint speech-text embeddings can be broadly classified into three non-mutually exclusive categories: \begin {enumerate*} [label=\itshape\alph*\upshape)] \item when intending to leverage data from both modalities as input into a single model, such as for multimodal translation \cite{zheng2021fused} \item when learning a semantic alignment between text and speech \cite{duquenne2021multimodal,chung2018unsupervised} that can be useful for data mining or retrieval \item when transferring knowledge encoded in a pretrained model from one modality to another or incorporating knowledge from both modalities, often seen in end-to-end solutions for speech translation (ST) \cite{ye2021end,han2021learning,dong2021listen,Tang2021IST}, SLU \cite{Denisov_2020,chung2020splat} or Q-A \cite{chuang2019speechbert}), that combine semantics from text with acoustics from speech \end{enumerate*}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="2107.05782" id="5168">
        <attvalues>
          <attvalue for="5" value=" Variations of this training objective are found in the literature, either as a pretraining step in isolation \cite{duquenne2021multimodal,Denisov_2020,chung2020splat}, or optimised in conjunction with the downstream task \cite{dong2021listen,Tang2021IST}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="2010.02295" id="5170">
        <attvalues>
          <attvalue for="5" value=" Variations of this training objective are found in the literature, either as a pretraining step in isolation \cite{duquenne2021multimodal,Denisov_2020,chung2020splat}, or optimised in conjunction with the downstream task \cite{dong2021listen,Tang2021IST}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="1910.11559" id="5171">
        <attvalues>
          <attvalue for="5" value=" The use-cases of joint speech-text embeddings can be broadly classified into three non-mutually exclusive categories: \begin {enumerate*} [label=\itshape\alph*\upshape)] \item when intending to leverage data from both modalities as input into a single model, such as for multimodal translation \cite{zheng2021fused} \item when learning a semantic alignment between text and speech \cite{duquenne2021multimodal,chung2018unsupervised} that can be useful for data mining or retrieval \item when transferring knowledge encoded in a pretrained model from one modality to another or incorporating knowledge from both modalities, often seen in end-to-end solutions for speech translation (ST) \cite{ye2021end,han2021learning,dong2021listen,Tang2021IST}, SLU \cite{Denisov_2020,chung2020splat} or Q-A \cite{chuang2019speechbert}), that combine semantics from text with acoustics from speech \end{enumerate*}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="1608.04207" id="5172">
        <attvalues>
          <attvalue for="5" value=" Several studies have been carried out to better understand learned embeddings derived exclusively from either text or speech, via linear probing \cite{conneau-etal-2018-cram,yossisenemb,47786}, geometrical analysis of the representation space \cite{ethayarajh2019}, or other intrinsic measures \cite{Pasad2021LayerWiseAO, schnabel2015evaluation}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="1905.06316" id="5173">
        <attvalues>
          <attvalue for="5" value=" Several studies have been carried out to better understand learned embeddings derived exclusively from either text or speech, via linear probing \cite{conneau-etal-2018-cram,yossisenemb,47786}, geometrical analysis of the representation space \cite{ethayarajh2019}, or other intrinsic measures \cite{Pasad2021LayerWiseAO, schnabel2015evaluation}." />
        </attvalues>
      </edge>
      <edge source="2204.01235" target="2107.04734" id="5175">
        <attvalues>
          <attvalue for="5" value=" Several studies have been carried out to better understand learned embeddings derived exclusively from either text or speech, via linear probing \cite{conneau-etal-2018-cram,yossisenemb,47786}, geometrical analysis of the representation space \cite{ethayarajh2019}, or other intrinsic measures \cite{Pasad2021LayerWiseAO, schnabel2015evaluation}." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2005.14165" id="5176">
        <attvalues>
          <attvalue for="5" value=" The recent success of Large Language Models (LLMs) \cite{BrownMRSKDSKSSA20,ChenTJYPKEBP21,gpt4,TouvronLIMLLRAGJGL23,TouvronMSAAB23,Gemini,JiangRSBLSBCLSSSALBGLLSSYATLWLE23} has lead to the development of various methods that facilitate constrained generation, a method that lets users tailor the output of an LLM to a specific task or format." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2107.03374" id="5177">
        <attvalues>
          <attvalue for="5" value=" The recent success of Large Language Models (LLMs) \cite{BrownMRSKDSKSSA20,ChenTJYPKEBP21,gpt4,TouvronLIMLLRAGJGL23,TouvronMSAAB23,Gemini,JiangRSBLSBCLSSSALBGLLSSYATLWLE23} has lead to the development of various methods that facilitate constrained generation, a method that lets users tailor the output of an LLM to a specific task or format." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2303.08774" id="5178">
        <attvalues>
          <attvalue for="5" value=" The recent success of Large Language Models (LLMs) \cite{BrownMRSKDSKSSA20,ChenTJYPKEBP21,gpt4,TouvronLIMLLRAGJGL23,TouvronMSAAB23,Gemini,JiangRSBLSBCLSSSALBGLLSSYATLWLE23} has lead to the development of various methods that facilitate constrained generation, a method that lets users tailor the output of an LLM to a specific task or format." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2307.09288" id="5180">
        <attvalues>
          <attvalue for="5" value=" The recent success of Large Language Models (LLMs) \cite{BrownMRSKDSKSSA20,ChenTJYPKEBP21,gpt4,TouvronLIMLLRAGJGL23,TouvronMSAAB23,Gemini,JiangRSBLSBCLSSSALBGLLSSYATLWLE23} has lead to the development of various methods that facilitate constrained generation, a method that lets users tailor the output of an LLM to a specific task or format." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2312.11805" id="5181">
        <attvalues>
          <attvalue for="5" value=" The recent success of Large Language Models (LLMs) \cite{BrownMRSKDSKSSA20,ChenTJYPKEBP21,gpt4,TouvronLIMLLRAGJGL23,TouvronMSAAB23,Gemini,JiangRSBLSBCLSSSALBGLLSSYATLWLE23} has lead to the development of various methods that facilitate constrained generation, a method that lets users tailor the output of an LLM to a specific task or format." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2401.04088" id="5182">
        <attvalues>
          <attvalue for="5" value=" The recent success of Large Language Models (LLMs) \cite{BrownMRSKDSKSSA20,ChenTJYPKEBP21,gpt4,TouvronLIMLLRAGJGL23,TouvronMSAAB23,Gemini,JiangRSBLSBCLSSSALBGLLSSYATLWLE23} has lead to the development of various methods that facilitate constrained generation, a method that lets users tailor the output of an LLM to a specific task or format." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2212.06094" id="5183">
        <attvalues>
          <attvalue for="5" value=" Doing so, the generated text can be ensured to adhere to constraints like high-level templates \cite{Beurer-Kellner023,guidance}, regular expressions \cite{Beurer-Kellner023,guidance,WillardL23} or context-free grammars \cite{WillardL23,guidance}." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2307.09702" id="5184">
        <attvalues>
          <attvalue for="5" value=" In contrast to existing methods, \tool is highly efficient and incurs little to no overhead, and in many cases even increases the throughput of LLM inference over unconstrained generation, by leveraging pre-computation \cite{WillardL23} and a novel speculative decoding procedure for constrained decoding." />
        </attvalues>
      </edge>
      <edge source="2403.06988" target="2201.11227" id="5185">
        <attvalues>
          <attvalue for="5" value=" While existing work on code generation has made this observation before \cite{PoesiaP00SMG22}, solving this problem efficiently remains challenging, as the online computation of all bridge tokens at each decoding step, can be too costly in high-throughput environments." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2005.14165" id="5186">
        <attvalues>
          <attvalue for="5" value=" The standard ICL approach is to randomly sample a few examples from a training set to construct a prompt~\cite{brown2020language}; however, prior work~\cite{liu2021makes,zhao2021calibrate,lu2021fantastically} has found that ICL is very sensitive to the choice of training examples and their order in the prompt." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2201.11990" id="5187">
        <attvalues>
          <attvalue for="5" value=" First introduced by GPT-3~\cite{brown2020language}, ICL with LLMs has reached state-of-the-art few-shot performance across many tasks~\cite{rae2021scaling,smith2022using,thoppilan2022lamda,chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2201.08239" id="5188">
        <attvalues>
          <attvalue for="5" value=" First introduced by GPT-3~\cite{brown2020language}, ICL with LLMs has reached state-of-the-art few-shot performance across many tasks~\cite{rae2021scaling,smith2022using,thoppilan2022lamda,chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2204.02311" id="5189">
        <attvalues>
          <attvalue for="5" value=" First introduced by GPT-3~\cite{brown2020language}, ICL with LLMs has reached state-of-the-art few-shot performance across many tasks~\cite{rae2021scaling,smith2022using,thoppilan2022lamda,chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2001.07676" id="5191">
        <attvalues>
          <attvalue for="5" value=" Compared with alternatives that use fine-tuning~\cite{devlin2018bert,schick2020exploiting,gao2020making}, ICL does not require task-specific training, which enables its use with very large language models, and it uses a unified model for all tasks, enabling easier deployment." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2012.15723" id="5192">
        <attvalues>
          <attvalue for="5" value=" Compared with alternatives that use fine-tuning~\cite{devlin2018bert,schick2020exploiting,gao2020making}, ICL does not require task-specific training, which enables its use with very large language models, and it uses a unified model for all tasks, enabling easier deployment." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2202.12837" id="5193">
        <attvalues>
          <attvalue for="5" value=" Despite its impressive few-shot performance, ICL often exhibits unintuitive behavior~\cite{min2022rethinking}." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2102.09690" id="5194">
        <attvalues>
          <attvalue for="5" value=" The standard ICL approach is to randomly sample a few examples from a training set to construct a prompt~\cite{brown2020language}; however, prior work~\cite{liu2021makes,zhao2021calibrate,lu2021fantastically} has found that ICL is very sensitive to the choice of training examples and their order in the prompt." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2104.08786" id="5195">
        <attvalues>
          <attvalue for="5" value=" The standard ICL approach is to randomly sample a few examples from a training set to construct a prompt~\cite{brown2020language}; however, prior work~\cite{liu2021makes,zhao2021calibrate,lu2021fantastically} has found that ICL is very sensitive to the choice of training examples and their order in the prompt." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2209.07661" id="5196">
        <attvalues>
          <attvalue for="5" value=" ICL is also sensitive to small changes in prompt format~\cite{chen2022relation}." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2112.08633" id="5198">
        <attvalues>
          <attvalue for="5" value=" While some prior work improves ICL accuracy by retrieving a suitable prompt for each test example~\cite{liu2021makes,rubin2021learning,su2022selective}, we show that it is possible to achieve stably good accuracy with a randomly sampled prompt for all test examples, when given the ``right'' training (sub)set." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2209.01975" id="5199">
        <attvalues>
          <attvalue for="5" value=" In contrast with prior work optimizing diversity for prompt selection~\cite{su2022selective,Ye2022ComplementaryEF}, we find our stable subsets no more diverse than random subsets of the training data." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2205.01068" id="5200">
        <attvalues>
          <attvalue for="5" value=" Our subset selection methods greatly improve performance across 5 classification datasets and 4 LLMs, with main experiments on GPTJ-6B~\cite{gpt-j} and OPT-13B~\cite{zhang2022opt}." />
        </attvalues>
      </edge>
      <edge source="2212.10378" target="2211.13892" id="5201">
        <attvalues>
          <attvalue for="5" value=" In contrast with prior work optimizing diversity for prompt selection~\cite{su2022selective,Ye2022ComplementaryEF}, we find our stable subsets no more diverse than random subsets of the training data." />
        </attvalues>
      </edge>
      <edge source="2003.07278" target="1802.05365" id="5203">
        <attvalues>
          <attvalue for="5" value=" Previous work \cite{peters2018deep,devlin2018bert,yang2019xlnet,raffel2019exploring} has shown that contextual embeddings pre-trained on large-scale unlabelled corpora achieve state-of-the-art performance on a wide range of natural language processing tasks, such as text classification, question answering and text summarization." />
        </attvalues>
      </edge>
      <edge source="2003.07278" target="1903.08855" id="5207">
        <attvalues>
          <attvalue for="5" value=" Further analyses \cite{liu2019linguistic,hewitt-liang-2019-designing,hewitt2019structural,tenney2019bert} demonstrate that contextual embeddings are capable of learning useful and transferable representations across languages." />
        </attvalues>
      </edge>
      <edge source="2003.07278" target="1905.05950" id="5209">
        <attvalues>
          <attvalue for="5" value=" Further analyses \cite{liu2019linguistic,hewitt-liang-2019-designing,hewitt2019structural,tenney2019bert} demonstrate that contextual embeddings are capable of learning useful and transferable representations across languages." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="2011.00675" id="5210">
        <attvalues>
          <attvalue for="5" value=" Moreover, the reliance of mined parallel data has many potential downsides, such as allowing for poisoning attacks \cite{xu2021targeted,wang2021putting}, memorization of low-quality examples \cite{raunak2022finding}, and biases towards generating text in language registers over-represented in the parallel data, such as language varieties \cite{lakew2018neural,riley2022frmt} or formality \cite{rippeth-etal-2022-controlling}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="2107.05243" id="5211">
        <attvalues>
          <attvalue for="5" value=" Moreover, the reliance of mined parallel data has many potential downsides, such as allowing for poisoning attacks \cite{xu2021targeted,wang2021putting}, memorization of low-quality examples \cite{raunak2022finding}, and biases towards generating text in language registers over-represented in the parallel data, such as language varieties \cite{lakew2018neural,riley2022frmt} or formality \cite{rippeth-etal-2022-controlling}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="2210.12929" id="5212">
        <attvalues>
          <attvalue for="5" value=" Moreover, the reliance of mined parallel data has many potential downsides, such as allowing for poisoning attacks \cite{xu2021targeted,wang2021putting}, memorization of low-quality examples \cite{raunak2022finding}, and biases towards generating text in language registers over-represented in the parallel data, such as language varieties \cite{lakew2018neural,riley2022frmt} or formality \cite{rippeth-etal-2022-controlling}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="1811.01064" id="5213">
        <attvalues>
          <attvalue for="5" value=" Moreover, the reliance of mined parallel data has many potential downsides, such as allowing for poisoning attacks \cite{xu2021targeted,wang2021putting}, memorization of low-quality examples \cite{raunak2022finding}, and biases towards generating text in language registers over-represented in the parallel data, such as language varieties \cite{lakew2018neural,riley2022frmt} or formality \cite{rippeth-etal-2022-controlling}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="2205.06644" id="5215">
        <attvalues>
          <attvalue for="5" value=" Moreover, the reliance of mined parallel data has many potential downsides, such as allowing for poisoning attacks \cite{xu2021targeted,wang2021putting}, memorization of low-quality examples \cite{raunak2022finding}, and biases towards generating text in language registers over-represented in the parallel data, such as language varieties \cite{lakew2018neural,riley2022frmt} or formality \cite{rippeth-etal-2022-controlling}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="1905.02450" id="5216">
        <attvalues>
          <attvalue for="5" value=" Unsupervised translation systems have demonstrated promising performance in recent years, able to match strong supervised baselines on academic benchmarks \cite{song2019mass,garcia2020harnessing,han2021unsupervised} by relying on a collection of tricks and techniques, such as multilinguality \cite{conneau2019unsupervised, garcia2020harnessing, garcia2020multilingual,lin2021few}, back-translation \cite{lample2017unsupervised}, and most recently through large-scale models with parameters in the hundreds of billions \cite{chowdhery2022palm,vilar2022prompting,han2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="2110.05448" id="5218">
        <attvalues>
          <attvalue for="5" value=" Unsupervised translation systems have demonstrated promising performance in recent years, able to match strong supervised baselines on academic benchmarks \cite{song2019mass,garcia2020harnessing,han2021unsupervised} by relying on a collection of tricks and techniques, such as multilinguality \cite{conneau2019unsupervised, garcia2020harnessing, garcia2020multilingual,lin2021few}, back-translation \cite{lample2017unsupervised}, and most recently through large-scale models with parameters in the hundreds of billions \cite{chowdhery2022palm,vilar2022prompting,han2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="1911.02116" id="5219">
        <attvalues>
          <attvalue for="5" value=" Unsupervised translation systems have demonstrated promising performance in recent years, able to match strong supervised baselines on academic benchmarks \cite{song2019mass,garcia2020harnessing,han2021unsupervised} by relying on a collection of tricks and techniques, such as multilinguality \cite{conneau2019unsupervised, garcia2020harnessing, garcia2020multilingual,lin2021few}, back-translation \cite{lample2017unsupervised}, and most recently through large-scale models with parameters in the hundreds of billions \cite{chowdhery2022palm,vilar2022prompting,han2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="1711.00043" id="5221">
        <attvalues>
          <attvalue for="5" value=" Unsupervised translation systems have demonstrated promising performance in recent years, able to match strong supervised baselines on academic benchmarks \cite{song2019mass,garcia2020harnessing,han2021unsupervised} by relying on a collection of tricks and techniques, such as multilinguality \cite{conneau2019unsupervised, garcia2020harnessing, garcia2020multilingual,lin2021few}, back-translation \cite{lample2017unsupervised}, and most recently through large-scale models with parameters in the hundreds of billions \cite{chowdhery2022palm,vilar2022prompting,han2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="2204.02311" id="5222">
        <attvalues>
          <attvalue for="5" value=" Unsupervised translation systems have demonstrated promising performance in recent years, able to match strong supervised baselines on academic benchmarks \cite{song2019mass,garcia2020harnessing,han2021unsupervised} by relying on a collection of tricks and techniques, such as multilinguality \cite{conneau2019unsupervised, garcia2020harnessing, garcia2020multilingual,lin2021few}, back-translation \cite{lample2017unsupervised}, and most recently through large-scale models with parameters in the hundreds of billions \cite{chowdhery2022palm,vilar2022prompting,han2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="2211.09102" id="5223">
        <attvalues>
          <attvalue for="5" value=" Unsupervised translation systems have demonstrated promising performance in recent years, able to match strong supervised baselines on academic benchmarks \cite{song2019mass,garcia2020harnessing,han2021unsupervised} by relying on a collection of tricks and techniques, such as multilinguality \cite{conneau2019unsupervised, garcia2020harnessing, garcia2020multilingual,lin2021few}, back-translation \cite{lample2017unsupervised}, and most recently through large-scale models with parameters in the hundreds of billions \cite{chowdhery2022palm,vilar2022prompting,han2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2302.01398" target="2005.14165" id="5224">
        <attvalues>
          <attvalue for="5" value=" It thus remains to explore what lies between these two research streams: few-shot learning \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2110.05448" target="2001.08361" id="5225">
        <attvalues>
          <attvalue for="5" value=" Recent work on generative pre-training has shown that with sufficient data and scale \cite{DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2010-14701}, large language models (LMs) can learn a diverse suite of tasks without explicit supervision \cite{radford2019language}, and that even stronger performance on these tasks can be elicited using few-shot demonstrations \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}." />
        </attvalues>
      </edge>
      <edge source="2110.05448" target="2010.14701" id="5226">
        <attvalues>
          <attvalue for="5" value=" Recent work on generative pre-training has shown that with sufficient data and scale \cite{DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2010-14701}, large language models (LMs) can learn a diverse suite of tasks without explicit supervision \cite{radford2019language}, and that even stronger performance on these tasks can be elicited using few-shot demonstrations \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}." />
        </attvalues>
      </edge>
      <edge source="2110.05448" target="2005.14165" id="5227">
        <attvalues>
          <attvalue for="5" value=" Recent work on generative pre-training has shown that with sufficient data and scale \cite{DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2010-14701}, large language models (LMs) can learn a diverse suite of tasks without explicit supervision \cite{radford2019language}, and that even stronger performance on these tasks can be elicited using few-shot demonstrations \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}." />
        </attvalues>
      </edge>
      <edge source="2110.05448" target="1908.08206" id="5230">
        <attvalues>
          <attvalue for="5" value=" Recent work in unsupervised NMT has been dominated by large encoder-decoder architectures where the bootstrap is implemented by denoising/autoencoding tasks (eg, multilingual Cloze \cite{DBLP:conf/naacl/DevlinCLT19, DBLP:conf/nips/ConneauL19}, masked-span prediction \cite{DBLP:journals/jmlr/RaffelSRLNMZLL20, DBLP:conf/naacl/XueCRKASBR21}, reconstruction from corrupted inputs \cite{DBLP:conf/emnlp/WangZJLL19, DBLP:journals/tacl/LiuGGLEGLZ20}) intended to produce strong encoders and aligned multilingual representations for decoding." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2308.12014" id="5252">
        <attvalues>
          <attvalue for="5" value=" Alignment is at the core of shaping behaviors of LLMs corresponding to human intentions and values~\cite{yaoInstructionsIntrinsicHuman2023a,shenLargeLanguageModel2023}, eg, teaching LLMs to follow ``helpful, harmless and honest (HHH)'' principles during responding~\cite{askell2021general}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2309.15025" id="5253">
        <attvalues>
          <attvalue for="5" value=" Alignment is at the core of shaping behaviors of LLMs corresponding to human intentions and values~\cite{yaoInstructionsIntrinsicHuman2023a,shenLargeLanguageModel2023}, eg, teaching LLMs to follow ``helpful, harmless and honest (HHH)'' principles during responding~\cite{askell2021general}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2112.00861" id="5254">
        <attvalues>
          <attvalue for="5" value=" Alignment is at the core of shaping behaviors of LLMs corresponding to human intentions and values~\cite{yaoInstructionsIntrinsicHuman2023a,shenLargeLanguageModel2023}, eg, teaching LLMs to follow ``helpful, harmless and honest (HHH)'' principles during responding~\cite{askell2021general}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2307.12966" id="5255">
        <attvalues>
          <attvalue for="5" value=" As a result, increasing efforts have been made for aligning LLMs to meet the human requirements, which makes it a hotspot research direction in LLM era~\cite{wangAligningLargeLanguage2023a, wangEssenceProspectInvestigation2024, jiAIAlignmentComprehensive2024}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2304.03277" id="5256">
        <attvalues>
          <attvalue for="5" value=" This form of data is often used for supervised fine-tuning of LLMs to inject human preference information into the model~\cite{alpaca,peng2023instruction,ding-etal-2023-enhancing}; 2) preference data, which usually includes a query, several potential responses, and human preferences regarding these responses~\cite{cui2024ultrafeedback}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2305.14233" id="5257">
        <attvalues>
          <attvalue for="5" value=" This form of data is often used for supervised fine-tuning of LLMs to inject human preference information into the model~\cite{alpaca,peng2023instruction,ding-etal-2023-enhancing}; 2) preference data, which usually includes a query, several potential responses, and human preferences regarding these responses~\cite{cui2024ultrafeedback}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2305.18290" id="5258">
        <attvalues>
          <attvalue for="5" value=" Preference data can be applied for direct preference optimization via algorithms such as DPO~\cite{rafailovDirectPreferenceOptimization2023a}, IPO~\cite{azar2024general}, and PRO~\cite{songPreferenceRankingOptimization2024}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2310.12036" id="5259">
        <attvalues>
          <attvalue for="5" value=" Preference data can be applied for direct preference optimization via algorithms such as DPO~\cite{rafailovDirectPreferenceOptimization2023a}, IPO~\cite{azar2024general}, and PRO~\cite{songPreferenceRankingOptimization2024}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2307.09288" id="5261">
        <attvalues>
          <attvalue for="5" value=" However, the construction process for both instruction-response pairs and preference data requires very expensive, meticulous human annotation with high quality standards, making each step of scaling these methods very costly~\cite{ouyangTrainingLanguageModels2022b,touvron2023llama,NEURIPS2023_ac662d74}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2305.11206" id="5262">
        <attvalues>
          <attvalue for="5" value=" However, the construction process for both instruction-response pairs and preference data requires very expensive, meticulous human annotation with high quality standards, making each step of scaling these methods very costly~\cite{ouyangTrainingLanguageModels2022b,touvron2023llama,NEURIPS2023_ac662d74}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2211.03540" id="5263">
        <attvalues>
          <attvalue for="5" value=" First, with the rapid development of LLMs, the capabilities of LLMs have gradually approached or even surpassed human in many aspects, making it increasingly challenging for humans to produce alignment data that is meaningful for LLMs~\cite{bowman2022measuring,burns2023weaktostrong}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2312.09390" id="5264">
        <attvalues>
          <attvalue for="5" value=" First, with the rapid development of LLMs, the capabilities of LLMs have gradually approached or even surpassed human in many aspects, making it increasingly challenging for humans to produce alignment data that is meaningful for LLMs~\cite{bowman2022measuring,burns2023weaktostrong}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2306.05685" id="5265">
        <attvalues>
          <attvalue for="5" value=" In fact, many studies have found that the quality of data generated by LLMs has already exceeded the quality of data annotated by general human annotators in many perspectives~\cite{zheng2024judging,chen2024spiral,wei2024long}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2403.18802" id="5266">
        <attvalues>
          <attvalue for="5" value=" In fact, many studies have found that the quality of data generated by LLMs has already exceeded the quality of data annotated by general human annotators in many perspectives~\cite{zheng2024judging,chen2024spiral,wei2024long}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2401.10020" id="5267">
        <attvalues>
          <attvalue for="5" value=" To address these challenges, automated alignment has drawn great attention very recently~\cite{yuan2024self,chen2024self}." />
        </attvalues>
      </edge>
      <edge source="2406.01252" target="2401.01335" id="5268">
        <attvalues>
          <attvalue for="5" value=" To address these challenges, automated alignment has drawn great attention very recently~\cite{yuan2024self,chen2024self}." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1706.03762" id="5269">
        <attvalues>
          <attvalue for="5" value=" Transformer \cite{NIPS2017_7181} based language representation has replaced many previous pre-training or initialization approaches \cite{devlin2018bert, radford2019language, yang2019xlnet, liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1506.06724" id="5271">
        <attvalues>
          <attvalue for="5" value=" For example, while the original BERT models \cite{devlin2018bert} were trained on English Wikipedia articles and BooksCorpus \cite{zhu2015aligning}, the same masked language modeling was continued on biomedical data." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1904.03323" id="5272">
        <attvalues>
          <attvalue for="5" value=" BioBERT \cite{lee2019biobert} was trained using Pubmed abstracts and full articles and Clinical BERT \cite{alsentzer2019publicly} parameters were further refined using MIMIC-III clinical notes \cite{johnson2016mimic}." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1906.05474" id="5273">
        <attvalues>
          <attvalue for="5" value=" Evidence suggest that understanding the syntactic structure of scientific literature and clinical data from pre-training boosts performance in their respective downstream tasks \cite{peng2019transfer}." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1901.11373" id="5274">
        <attvalues>
          <attvalue for="5" value=" In general perplexity increases for older domains, and models lose confidence in continual learning settings \cite{yogatama2019learning}." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1706.05098" id="5275">
        <attvalues>
          <attvalue for="5" value=" For many tasks the straightforward solution is to combine datasets during training and approach this as a multi-task learning (MTL) \cite{ruder2017overview} problem." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1906.02243" id="5276">
        <attvalues>
          <attvalue for="5" value=" \cite{strubell2019energy} show that as deep neural architectures in the natural language community grow we increasingly trade results for carbon emissions." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1801.06146" id="5277">
        <attvalues>
          <attvalue for="5" value=" \cite{howard2018universal} introduced a multi stage training scheme for fine tuning LSTM based universal language models (ULMFiT)." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1612.00796" id="5279">
        <attvalues>
          <attvalue for="5" value=" \cite{kirkpatrick2017overcoming} provided a more sophisticated solution constraining weights individually termed elastic weight consolidation (EWC)." />
        </attvalues>
      </edge>
      <edge source="2004.03794" target="1906.01076" id="5280">
        <attvalues>
          <attvalue for="5" value=" Using small samples of data from previous tasks coupled with local adaptation \cite{d2019episodic} demonstrate improvement in a Lifelong Learning (LL) training scheme." />
        </attvalues>
      </edge>
      <edge source="1906.01076" target="1612.00796" id="5282">
        <attvalues>
          <attvalue for="5" value=" The three main approaches to address catastrophic forgetting are based on: (i) augmenting the loss function that is being minimized during training with extra terms (eg, a regularization term, an optimization constraint) to prevent model parameters learned on a new dataset from significantly deviating from parameters learned on previously seen datasets \cite{ewc, zenke, chaudhry}, (ii) adding extra learning phases such as a knowledge distillation phase, an experience replay \cite{progresscompress, experiencereplay}, and (iii) augmenting the model with an episodic memory module \cite{mbpa}." />
        </attvalues>
      </edge>
      <edge source="1906.01076" target="1802.10542" id="5283">
        <attvalues>
          <attvalue for="5" value=" In local adaptation, we follow Memory-based Parameter Adaptation \cite[MbPA;][]{mbpa} and use examples retrieved from memory to update model parameters used to make a prediction of a particular test example." />
        </attvalues>
      </edge>
      <edge source="1906.01076" target="1802.05365" id="5284">
        <attvalues>
          <attvalue for="5" value=" In language learning, progress in unsupervised pretraining \cite{elmo,ulmfit,bert} has driven advances in many language understanding tasks \cite{kitaev,lee}." />
        </attvalues>
      </edge>
      <edge source="1906.01076" target="1801.06146" id="5285">
        <attvalues>
          <attvalue for="5" value=" In language learning, progress in unsupervised pretraining \cite{elmo,ulmfit,bert} has driven advances in many language understanding tasks \cite{kitaev,lee}." />
        </attvalues>
      </edge>
      <edge source="1906.01076" target="1805.01052" id="5287">
        <attvalues>
          <attvalue for="5" value=" In language learning, progress in unsupervised pretraining \cite{elmo,ulmfit,bert} has driven advances in many language understanding tasks \cite{kitaev,lee}." />
        </attvalues>
      </edge>
      <edge source="1906.01076" target="1804.05392" id="5288">
        <attvalues>
          <attvalue for="5" value=" In language learning, progress in unsupervised pretraining \cite{elmo,ulmfit,bert} has driven advances in many language understanding tasks \cite{kitaev,lee}." />
        </attvalues>
      </edge>
      <edge source="1906.01076" target="1901.11373" id="5289">
        <attvalues>
          <attvalue for="5" value=" However, these models have been shown to require a lot of in-domain training examples, rapidly overfit to particular datasets, and are prone to catastrophic forgetting \cite{glipaper}, making them unsuitable as a model of general linguistic intelligence." />
        </attvalues>
      </edge>
      <edge source="1906.01076" target="1812.00420" id="5290">
        <attvalues>
          <attvalue for="5" value=" We assume that the model only makes one pass over the training examples, similar to \cite{agem}." />
        </attvalues>
      </edge>
      <edge source="1910.00275" target="1805.05388" id="5294">
        <attvalues>
          <attvalue for="5" value=" Three evaluation tasks have been proposed to evaluate few-shot learning methods: Definitional Nonce \cite{herbelot2017}, Chimera \cite{Lazaridou:20170be}, and Contextual Rare Words \cite{Khodak:20180be}, which we describe in Section~\ref{sec:back}." />
        </attvalues>
      </edge>
      <edge source="2102.08424" target="1806.00187" id="5303">
        <attvalues>
          <attvalue for="5" value=" They hold state of the art on a myriad of tasks, eg, neural machine translation \cite[NMT;][]{ott-etal-2018-scaling} and abstractive summarization \cite[AS;][]{lewis2019bart}." />
        </attvalues>
      </edge>
      <edge source="2102.08424" target="1910.13461" id="5304">
        <attvalues>
          <attvalue for="5" value=" They hold state of the art on a myriad of tasks, eg, neural machine translation \cite[NMT;][]{ott-etal-2018-scaling} and abstractive summarization \cite[AS;][]{lewis2019bart}." />
        </attvalues>
      </edge>
      <edge source="2102.08424" target="1808.10006" id="5305">
        <attvalues>
          <attvalue for="5" value=" Yet, an undesirable property of these models has been repeatedly observed in word-level tasks: When using beam search as the decoding strategy, increasing the beam width beyond a size of $k=5$ often leads to a drop in the quality of solutions \cite{murray-chiang-2018-correcting, yang-etal-2018-breaking,pmlr-v97-cohen19a}." />
        </attvalues>
      </edge>
      <edge source="2102.08424" target="1908.10090" id="5306">
        <attvalues>
          <attvalue for="5" value=" Further, in the context of NMT, it has been shown that the empty string is frequently the most-probable solution under the model \cite{stahlberg-byrne-2019-nmt}." />
        </attvalues>
      </edge>
      <edge source="2102.08424" target="1706.03872" id="5307">
        <attvalues>
          <attvalue for="5" value=" Some suggest this is a manifestation of the general inadequacy of neural models for language generation tasks \cite{koehn-knowles-2017-six, kumar2019calibration,holtzman2019curious,stahlberg_phd}; in this work, we find evidence demonstrating otherwise." />
        </attvalues>
      </edge>
      <edge source="2102.08424" target="1904.09751" id="5308">
        <attvalues>
          <attvalue for="5" value=" Some suggest this is a manifestation of the general inadequacy of neural models for language generation tasks \cite{koehn-knowles-2017-six, kumar2019calibration,holtzman2019curious,stahlberg_phd}; in this work, we find evidence demonstrating otherwise." />
        </attvalues>
      </edge>
      <edge source="2102.08424" target="2005.10213" id="5310">
        <attvalues>
          <attvalue for="5" value="\looseness=-1 Sequence-to-sequence transducers for character-level tasks often follow the architectures of their word-level counterparts \cite{faruqui-etal-2016-morphological, lee-etal-2017-fully}, and have likewise achieved state-of-the-art performance on eg, morphological inflection generation \cite{wu2020applying} and grapheme-to-phoneme conversion \cite{G2P}." />
        </attvalues>
      </edge>
      <edge source="2102.08424" target="2004.06338" id="5311">
        <attvalues>
          <attvalue for="5" value="\looseness=-1 Sequence-to-sequence transducers for character-level tasks often follow the architectures of their word-level counterparts \cite{faruqui-etal-2016-morphological, lee-etal-2017-fully}, and have likewise achieved state-of-the-art performance on eg, morphological inflection generation \cite{wu2020applying} and grapheme-to-phoneme conversion \cite{G2P}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2003.12710" id="5312">
        <attvalues>
          <attvalue for="5" value=" With large amounts of paired speech and text data available for ASR training, end-to-end (E2E) automatic speech recognition (ASR) systems have become competitive with hybrid ASR systems~\cite{sainath2020streaming,li2020developing,radford2022robust}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2007.15188" id="5313">
        <attvalues>
          <attvalue for="5" value=" With large amounts of paired speech and text data available for ASR training, end-to-end (E2E) automatic speech recognition (ASR) systems have become competitive with hybrid ASR systems~\cite{sainath2020streaming,li2020developing,radford2022robust}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2202.01094" id="5315">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2104.04950" id="5316">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2105.00858" id="5317">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="1612.02695" id="5318">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="1708.06426" id="5320">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="1807.10857" id="5321">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2003.07705" id="5322">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2210.17049" id="5323">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2011.01991" id="5324">
        <attvalues>
          <attvalue for="5" value=" There are generally four ways to perform text-only adaptation: \begin{itemize} \item {Rescoring and reranking}: after decoding, use a powerful external LM to update scores and rerank n-best results or recognition lattice \cite{xu2022rescorebert,chiu2021innovative,beck2020lvcsr}; \item {Audio generation}: generate audio for training texts via TTS \cite{deng2021improving, fazel2021synthasr}, data splicing \cite{zhao2021addressing} or learn to emulate encoder output from texts on the fly \cite{mittal2023in-situ}; \item {Fusion and biasing}: during decoding interpolate posterior word probabilities with text priors from external LMs \cite{Chorowski2016Towards,gulcehre2017onusing,sriram2017coldfusion,toshniwal2018comparison, mcdermott2019density}; \item {Explicit separation of internal LMs}: force the E2E decoder/predictor to behave more like a language model \cite{variani2020hybrid,meng2023modular,meng2021internal,zhou2022language, chen2022factorized}." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="2110.01500" id="5326">
        <attvalues>
          <attvalue for="5" value=" \end{itemize} In this paper, we focus on the last approach using factorized neural transducers (FNT)~\cite{chen2022factorized} which is an extension of RNN-T~\cite{graves2012sequence} that makes the predictor network act like a proper language model." />
        </attvalues>
      </edge>
      <edge source="2305.17304" target="1211.3711" id="5327">
        <attvalues>
          <attvalue for="5" value=" \end{itemize} In this paper, we focus on the last approach using factorized neural transducers (FNT)~\cite{chen2022factorized} which is an extension of RNN-T~\cite{graves2012sequence} that makes the predictor network act like a proper language model." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="2002.05709" id="5328">
        <attvalues>
          <attvalue for="5" value=" Thanks to this property, self-supervised learning can leverage large-scale unlabeled data for training, and has enjoyed success in learning high-level representations of data from different modalities~\cite{chen2020simple,devlin2019bert,baevski2020wav2vec}." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="2006.11477" id="5329">
        <attvalues>
          <attvalue for="5" value=" Thanks to this property, self-supervised learning can leverage large-scale unlabeled data for training, and has enjoyed success in learning high-level representations of data from different modalities~\cite{chen2020simple,devlin2019bert,baevski2020wav2vec}." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="1807.03748" id="5330">
        <attvalues>
          <attvalue for="5" value=" Methods like contrastive predictive coding~\cite{oord2018representation}, autoregressive predictive coding~\cite{chung2019unsupervised}, masked predictive coding~\cite{liu2020mockingjay,wang2020unsupervised,jiang2019improving}, and problem-agnostic speech encoder~\cite{pascual2019learning} have been shown to be capable of learning representations that capture high-level properties of speech that are not easily accessible from surface features such as audio waveforms and spectrograms." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="1904.03240" id="5331">
        <attvalues>
          <attvalue for="5" value=" Methods like contrastive predictive coding~\cite{oord2018representation}, autoregressive predictive coding~\cite{chung2019unsupervised}, masked predictive coding~\cite{liu2020mockingjay,wang2020unsupervised,jiang2019improving}, and problem-agnostic speech encoder~\cite{pascual2019learning} have been shown to be capable of learning representations that capture high-level properties of speech that are not easily accessible from surface features such as audio waveforms and spectrograms." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="1910.12638" id="5332">
        <attvalues>
          <attvalue for="5" value=" Methods like contrastive predictive coding~\cite{oord2018representation}, autoregressive predictive coding~\cite{chung2019unsupervised}, masked predictive coding~\cite{liu2020mockingjay,wang2020unsupervised,jiang2019improving}, and problem-agnostic speech encoder~\cite{pascual2019learning} have been shown to be capable of learning representations that capture high-level properties of speech that are not easily accessible from surface features such as audio waveforms and spectrograms." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="2001.10603" id="5333">
        <attvalues>
          <attvalue for="5" value=" Methods like contrastive predictive coding~\cite{oord2018representation}, autoregressive predictive coding~\cite{chung2019unsupervised}, masked predictive coding~\cite{liu2020mockingjay,wang2020unsupervised,jiang2019improving}, and problem-agnostic speech encoder~\cite{pascual2019learning} have been shown to be capable of learning representations that capture high-level properties of speech that are not easily accessible from surface features such as audio waveforms and spectrograms." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="1910.09932" id="5334">
        <attvalues>
          <attvalue for="5" value=" Methods like contrastive predictive coding~\cite{oord2018representation}, autoregressive predictive coding~\cite{chung2019unsupervised}, masked predictive coding~\cite{liu2020mockingjay,wang2020unsupervised,jiang2019improving}, and problem-agnostic speech encoder~\cite{pascual2019learning} have been shown to be capable of learning representations that capture high-level properties of speech that are not easily accessible from surface features such as audio waveforms and spectrograms." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="1910.12607" id="5336">
        <attvalues>
          <attvalue for="5" value=" These methods have been further extended or improved for tackling a wide range of speech applications, including speech recognition~\cite{baevski2020effectiveness,chung2020generative,ling2020deep,jiang2020further,song2020speech}, speech translation~\cite{nguyen2020investigating,wu2020self}, speaker verification~\cite{ravi2020exploring}, unsupervised unit discovery~\cite{feng2020unsupervised}, and unsupervised phoneme segmentation~\cite{kreuk2020self}, to name a few." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="1912.01679" id="5337">
        <attvalues>
          <attvalue for="5" value=" These methods have been further extended or improved for tackling a wide range of speech applications, including speech recognition~\cite{baevski2020effectiveness,chung2020generative,ling2020deep,jiang2020further,song2020speech}, speech translation~\cite{nguyen2020investigating,wu2020self}, speaker verification~\cite{ravi2020exploring}, unsupervised unit discovery~\cite{feng2020unsupervised}, and unsupervised phoneme segmentation~\cite{kreuk2020self}, to name a few." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="2005.09862" id="5338">
        <attvalues>
          <attvalue for="5" value=" These methods have been further extended or improved for tackling a wide range of speech applications, including speech recognition~\cite{baevski2020effectiveness,chung2020generative,ling2020deep,jiang2020further,song2020speech}, speech translation~\cite{nguyen2020investigating,wu2020self}, speaker verification~\cite{ravi2020exploring}, unsupervised unit discovery~\cite{feng2020unsupervised}, and unsupervised phoneme segmentation~\cite{kreuk2020self}, to name a few." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="1910.10387" id="5339">
        <attvalues>
          <attvalue for="5" value=" These methods have been further extended or improved for tackling a wide range of speech applications, including speech recognition~\cite{baevski2020effectiveness,chung2020generative,ling2020deep,jiang2020further,song2020speech}, speech translation~\cite{nguyen2020investigating,wu2020self}, speaker verification~\cite{ravi2020exploring}, unsupervised unit discovery~\cite{feng2020unsupervised}, and unsupervised phoneme segmentation~\cite{kreuk2020self}, to name a few." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="2008.03615" id="5341">
        <attvalues>
          <attvalue for="5" value=" These methods have been further extended or improved for tackling a wide range of speech applications, including speech recognition~\cite{baevski2020effectiveness,chung2020generative,ling2020deep,jiang2020further,song2020speech}, speech translation~\cite{nguyen2020investigating,wu2020self}, speaker verification~\cite{ravi2020exploring}, unsupervised unit discovery~\cite{feng2020unsupervised}, and unsupervised phoneme segmentation~\cite{kreuk2020self}, to name a few." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="2007.13465" id="5343">
        <attvalues>
          <attvalue for="5" value=" These methods have been further extended or improved for tackling a wide range of speech applications, including speech recognition~\cite{baevski2020effectiveness,chung2020generative,ling2020deep,jiang2020further,song2020speech}, speech translation~\cite{nguyen2020investigating,wu2020self}, speaker verification~\cite{ravi2020exploring}, unsupervised unit discovery~\cite{feng2020unsupervised}, and unsupervised phoneme segmentation~\cite{kreuk2020self}, to name a few." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="2005.08392" id="5344">
        <attvalues>
          <attvalue for="5" value="~(2020) proposed to incorporate vector quantization layers to restrict model capacity during pre-training so as to uncover a model's preference in preserving speech information for achieving a maximal self-supervised objective~\cite{chung2020vector}." />
        </attvalues>
      </edge>
      <edge source="2010.11481" target="2007.04205" id="5345">
        <attvalues>
          <attvalue for="5" value=" For the correlation study, we also consider more self-supervised models with diverse modeling choices as compared to previous work~\cite{blandon2020analysis}." />
        </attvalues>
      </edge>
      <edge source="2406.12471" target="2002.06305" id="5358">
        <attvalues>
          <attvalue for="5" value=" Despite the success, previous works observed that fine-tuning still remains unstable~\cite{dodge2020fine, mosbach2021on, chen-etal-2022-revisiting}, especially with limited data." />
        </attvalues>
      </edge>
      <edge source="2406.12471" target="2312.01082" id="5359">
        <attvalues>
          <attvalue for="5" value=" To deal with the fine-tuning instability, researchers propose various strategies to mitigate the effects of randomness~\cite{pecher2024survey}." />
        </attvalues>
      </edge>
      <edge source="2406.12471" target="2107.04835" id="5361">
        <attvalues>
          <attvalue for="5" value=" The methods that add noise to the model parameters~\cite{hua-etal-2021-noise, wu-etal-2022-noisytune} also perform well, improving generalisability and overall performance, but not necessarily reducing the instability." />
        </attvalues>
      </edge>
      <edge source="2406.12471" target="1710.09412" id="5362">
        <attvalues>
          <attvalue for="5" value=" As many of these strategies are designed and evaluated on high-resource datasets, almost no focus is dedicated to evaluating the mitigation benefit of data augmentation~\cite{zhang2018mixup, meng2023tuning}." />
        </attvalues>
      </edge>
      <edge source="2406.12471" target="2202.07962" id="5363">
        <attvalues>
          <attvalue for="5" value=" When using PEFT methods, the focus is on the initialisation of soft prompts, even though the factors such as data shuffling still lead to variance in results~\cite{chen-etal-2022-revisiting}." />
        </attvalues>
      </edge>
      <edge source="2406.12471" target="2302.04863" id="5364">
        <attvalues>
          <attvalue for="5" value=" Overall, the best performing mitigation strategies are ensembles and model interpolation methods~\cite{gueta-etal-2023-knowledge, hidey-etal-2022-reducing, wang-etal-2023-two}, which significantly reduce the deviation in results, but also significantly increase the computation costs." />
        </attvalues>
      </edge>
      <edge source="2406.12471" target="2305.12947" id="5365">
        <attvalues>
          <attvalue for="5" value=" In the comparison, we also include an augmentation strategy that uses large language models to paraphrase samples, as such paraphrasing was observed to improve robustness and stability by~\cite{cegin-etal-2023-chatgpt, cegin2024effects}, especially in limited data settings." />
        </attvalues>
      </edge>
      <edge source="2409.06243" target="2105.04222" id="5366">
        <attvalues>
          <attvalue for="5" value=" Several strategies have been proposed for achieving domain transfer to unseen domains by utilizing natural language form slot descriptions \cite{lin2021leveraging, heck2022robust}, leveraging question answering dataset \cite{li2021zero, sf-dst}, and augmentation with ontology \cite{campagna2020zero}." />
        </attvalues>
      </edge>
      <edge source="2409.06243" target="2101.08333" id="5367">
        <attvalues>
          <attvalue for="5" value=" Several strategies have been proposed for achieving domain transfer to unseen domains by utilizing natural language form slot descriptions \cite{lin2021leveraging, heck2022robust}, leveraging question answering dataset \cite{li2021zero, sf-dst}, and augmentation with ontology \cite{campagna2020zero}." />
        </attvalues>
      </edge>
      <edge source="2409.06243" target="2005.00891" id="5368">
        <attvalues>
          <attvalue for="5" value=" Several strategies have been proposed for achieving domain transfer to unseen domains by utilizing natural language form slot descriptions \cite{lin2021leveraging, heck2022robust}, leveraging question answering dataset \cite{li2021zero, sf-dst}, and augmentation with ontology \cite{campagna2020zero}." />
        </attvalues>
      </edge>
      <edge source="2305.10615" target="2205.10643" id="5386">
        <attvalues>
          <attvalue for="5" value=" SSL models have shown promising results by capturing important speech features, such as phonemes and other acoustic units, through training on large amounts of unlabeled speech data \cite{mohamed2022self}." />
        </attvalues>
      </edge>
      <edge source="2305.10615" target="2006.11477" id="5388">
        <attvalues>
          <attvalue for="5" value=" Over the past few years, researchers have proposed a variety of SSL models with different training objectives, operating under various data conditions, model architectures, and modalities \cite{baevski2020wav2vec, hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2305.10615" target="2203.06849" id="5389">
        <attvalues>
          <attvalue for="5" value=" Recently, an extension of SUPERB called SUPERB-SG \cite{tsai2022superb} has been introduced." />
        </attvalues>
      </edge>
      <edge source="2305.10615" target="2111.09296" id="5390">
        <attvalues>
          <attvalue for="5" value=" However, there has been growing interest in applying SSL models to multilingual scenarios, such as training multilingual SSL models \cite{babu2021xls, conneau2020unsupervised, duquenne2022speechmatrix} or using SSL models in a cross-lingual manner \cite{zhao2022improving, berrebbi22_interspeech, wu2020self, li22aa_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2305.10615" target="2104.11462" id="5391">
        <attvalues>
          <attvalue for="5" value=" Lebenchmark primarily evaluates speech tasks in French \cite{evain21_interspeech}; IndicSUPERB focuses mostly on Indian languages \cite{javed2022indicsuperb}." />
        </attvalues>
      </edge>
      <edge source="2305.10615" target="2208.11761" id="5392">
        <attvalues>
          <attvalue for="5" value=" Lebenchmark primarily evaluates speech tasks in French \cite{evain21_interspeech}; IndicSUPERB focuses mostly on Indian languages \cite{javed2022indicsuperb}." />
        </attvalues>
      </edge>
      <edge source="2305.10615" target="2203.10752" id="5393">
        <attvalues>
          <attvalue for="5" value=" XTREME-S focuses on multilingual speech representation benchmarks, including ASR, speech translation, speech classification, and speech retrieval \cite{conneau22_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2010.12780" target="1506.06714" id="5394">
        <attvalues>
          <attvalue for="5" value=" General purpose (non goal-oriented) dialogue has been investigated using data-driven sequence-to-sequence (SEQ2SEQ) recurrent neural networks (RNN) \cite{sordoni2015neural, shang2015neural, wen2015semantically, vinyals2015neural}." />
        </attvalues>
      </edge>
      <edge source="2010.12780" target="1902.00098" id="5395">
        <attvalues>
          <attvalue for="5" value=" Recently, fine-tuning pre-trained language models has demonstrated superior performance in the ConvAI2 \cite{dinan2019second} competition." />
        </attvalues>
      </edge>
      <edge source="2010.12780" target="1901.08149" id="5397">
        <attvalues>
          <attvalue for="5" value=" Different frameworks have been proposed in the literature: Transformer-ED (explicit encoder and decoder architecture) \cite{zheng2019pre}, Transformer-Dec (decoder only) \cite{wolf2019transfertransfo,lin2019caire,zhang2019dialogpt}, Transformer decoder that uses bi-directional attention on the source side and left-to-right attention on the target side with Masked Language Model (MLM) objective (Transformer-MLM) \cite{dong2019unified} or with Auto-Regressive (AR) objective (Transformer-AR) \cite{bao2019plato, shuster2019dialogue}." />
        </attvalues>
      </edge>
      <edge source="2010.12780" target="1911.00536" id="5398">
        <attvalues>
          <attvalue for="5" value=" Different frameworks have been proposed in the literature: Transformer-ED (explicit encoder and decoder architecture) \cite{zheng2019pre}, Transformer-Dec (decoder only) \cite{wolf2019transfertransfo,lin2019caire,zhang2019dialogpt}, Transformer decoder that uses bi-directional attention on the source side and left-to-right attention on the target side with Masked Language Model (MLM) objective (Transformer-MLM) \cite{dong2019unified} or with Auto-Regressive (AR) objective (Transformer-AR) \cite{bao2019plato, shuster2019dialogue}." />
        </attvalues>
      </edge>
      <edge source="2010.12780" target="1905.03197" id="5399">
        <attvalues>
          <attvalue for="5" value=" BERT has been used widely as the encoder for classification tasks \cite{zhang2019find, zeng2020multidomain}, while some studies \cite{dong2019unified, zeng2020generalized} show that fine-tuning BERT can also achieve state-of-the-art performance for response generation in dialogue." />
        </attvalues>
      </edge>
      <edge source="2010.12780" target="1910.07931" id="5400">
        <attvalues>
          <attvalue for="5" value=" Different frameworks have been proposed in the literature: Transformer-ED (explicit encoder and decoder architecture) \cite{zheng2019pre}, Transformer-Dec (decoder only) \cite{wolf2019transfertransfo,lin2019caire,zhang2019dialogpt}, Transformer decoder that uses bi-directional attention on the source side and left-to-right attention on the target side with Masked Language Model (MLM) objective (Transformer-MLM) \cite{dong2019unified} or with Auto-Regressive (AR) objective (Transformer-AR) \cite{bao2019plato, shuster2019dialogue}." />
        </attvalues>
      </edge>
      <edge source="2010.12780" target="2004.03829" id="5401">
        <attvalues>
          <attvalue for="5" value=" Thus, some researchers believe that this framework naturally works well for dialogue response generation \cite{lin2020exploring}." />
        </attvalues>
      </edge>
      <edge source="2010.12780" target="1910.03544" id="5403">
        <attvalues>
          <attvalue for="5" value=" BERT has been used widely as the encoder for classification tasks \cite{zhang2019find, zeng2020multidomain}, while some studies \cite{dong2019unified, zeng2020generalized} show that fine-tuning BERT can also achieve state-of-the-art performance for response generation in dialogue." />
        </attvalues>
      </edge>
      <edge source="2311.13892" target="1904.08067" id="5406">
        <attvalues>
          <attvalue for="5" value=" Recently, masked language models (MLMs) \cite{devlin2018bert, lanalbert, liu2019roberta, NEURIPS2019_dc6a7e65, sanh2019distilbert, zhang2021you} are employed in both traditional tasks like text classification \cite{kowsari2019text, zhang2023ideal, zhang2023hypertime} and diverse multimodal tasks \cite{wu2023autogen, wu2023empirical} when combined with models like image generators \cite{radford2021learning, rombach2022high}." />
        </attvalues>
      </edge>
      <edge source="2311.13892" target="2007.08100" id="5408">
        <attvalues>
          <attvalue for="5" value=" Existing studies \cite{liang2020towards, kaneko2021debiasing, garimella2021he, ct1965, webster2020measuring, chengfairfil} have introduced intuitive approaches that use additional corpus to retrieve contextualized embeddings or locate the biases and fine-tune accordingly." />
        </attvalues>
      </edge>
      <edge source="2311.13892" target="1909.11942" id="5410">
        <attvalues>
          <attvalue for="5" value=" &#09;\item We conduct experiments on three well-known open-source MLMs: BERT\cite{devlin2018bert}, ALBERT\cite{lanalbert}, and DistilBERT\cite{sanh2019distilbert}, and achieves state-of-the-art performance (0." />
        </attvalues>
      </edge>
      <edge source="1912.13413" target="1310.4546" id="5422">
        <attvalues>
          <attvalue for="5" value=" Following the setups of the widely used {\sc word2vec} \cite{mikolov2013distributed} model, we consider two vectors per each word $i$: \begin{itemize} \item $\mathbf{w}_i$ is an embedding of the word $i$ when $i$ is a center word, \item $\mathbf{c}_i$ is an embedding of the word $i$ when $i$ is a context word." />
        </attvalues>
      </edge>
      <edge source="1912.13413" target="1902.09859" id="5423">
        <attvalues>
          <attvalue for="5" value=" Under the assumptions \ref{assump_1}--\ref{assump_3} above, \cite{assylbekov2019context} \shortcite{assylbekov2019context} showed that each word's vector $\mathbf{w}_i$ splits into two approximately equally-sized subvectors $\mathbf{x}_i$ and $\mathbf{y}_i$, and the model \ref{eq:model} for generating a word $i$ in the context of a word $j$ can be rewritten as $$ p(i\mid j)\approx p_i\cdot e^{\mathbf{x}_j^\top\mathbf{x}_i-\mathbf{y}_j^\top\mathbf{y}_i}." />
        </attvalues>
      </edge>
      <edge source="2408.03130" target="2305.06161" id="5424">
        <attvalues>
          <attvalue for="5" value=" These versatile models have demonstrated remarkable abilities in diverse applications, ranging from assisting in code generation \cite{Li2023StarCoderMT} \cite{Chen2021EvaluatingLL}, to facilitating news summarization \cite{Wei2022EmergentAO} \cite{Lewis2019BARTDS}, and even augmenting information retrieval systems for improved search accuracy and efficiency \cite{Lewis2020RetrievalAugmentedGF} \cite{DBLP:journals/corr/abs-2312-10997}." />
        </attvalues>
      </edge>
      <edge source="2408.03130" target="2107.03374" id="5425">
        <attvalues>
          <attvalue for="5" value=" These versatile models have demonstrated remarkable abilities in diverse applications, ranging from assisting in code generation \cite{Li2023StarCoderMT} \cite{Chen2021EvaluatingLL}, to facilitating news summarization \cite{Wei2022EmergentAO} \cite{Lewis2019BARTDS}, and even augmenting information retrieval systems for improved search accuracy and efficiency \cite{Lewis2020RetrievalAugmentedGF} \cite{DBLP:journals/corr/abs-2312-10997}." />
        </attvalues>
      </edge>
      <edge source="2408.03130" target="1910.13461" id="5427">
        <attvalues>
          <attvalue for="5" value=" These versatile models have demonstrated remarkable abilities in diverse applications, ranging from assisting in code generation \cite{Li2023StarCoderMT} \cite{Chen2021EvaluatingLL}, to facilitating news summarization \cite{Wei2022EmergentAO} \cite{Lewis2019BARTDS}, and even augmenting information retrieval systems for improved search accuracy and efficiency \cite{Lewis2020RetrievalAugmentedGF} \cite{DBLP:journals/corr/abs-2312-10997}." />
        </attvalues>
      </edge>
      <edge source="2408.03130" target="2005.11401" id="5428">
        <attvalues>
          <attvalue for="5" value=" These versatile models have demonstrated remarkable abilities in diverse applications, ranging from assisting in code generation \cite{Li2023StarCoderMT} \cite{Chen2021EvaluatingLL}, to facilitating news summarization \cite{Wei2022EmergentAO} \cite{Lewis2019BARTDS}, and even augmenting information retrieval systems for improved search accuracy and efficiency \cite{Lewis2020RetrievalAugmentedGF} \cite{DBLP:journals/corr/abs-2312-10997}." />
        </attvalues>
      </edge>
      <edge source="2408.03130" target="2312.10997" id="5429">
        <attvalues>
          <attvalue for="5" value=" These versatile models have demonstrated remarkable abilities in diverse applications, ranging from assisting in code generation \cite{Li2023StarCoderMT} \cite{Chen2021EvaluatingLL}, to facilitating news summarization \cite{Wei2022EmergentAO} \cite{Lewis2019BARTDS}, and even augmenting information retrieval systems for improved search accuracy and efficiency \cite{Lewis2020RetrievalAugmentedGF} \cite{DBLP:journals/corr/abs-2312-10997}." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="1909.01066" id="5430">
        <attvalues>
          <attvalue for="5" value=" Autoregressive large language models (LLMs) trained on general-domain data are vast stores of world knowledge \cite{petroni-etal-2019-language}." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="1901.00158" id="5431">
        <attvalues>
          <attvalue for="5" value=" Many useful tasks, such as infilling~\cite{zhu2019text,liu-etal-2019-tigs}, generating text conditioned on length or lexical constraints~\cite{hokamp-liu-2017-lexically, hu-etal-2019-improved}, and finding the most likely sequence continuation, involve intractable inference in LLMs." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="1905.10752" id="5432">
        <attvalues>
          <attvalue for="5" value=" Many useful tasks, such as infilling~\cite{zhu2019text,liu-etal-2019-tigs}, generating text conditioned on length or lexical constraints~\cite{hokamp-liu-2017-lexically, hu-etal-2019-improved}, and finding the most likely sequence continuation, involve intractable inference in LLMs." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="2201.11903" id="5434">
        <attvalues>
          <attvalue for="5" value=" For example, we can interpret chain-of-thought reasoning \cite{wei2022chain,kojima2022large}, a paradigm of reasoning in language models, as a problem of intractable posterior inference." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="1811.10996" id="5435">
        <attvalues>
          <attvalue for="5" value=" A standard method to sample approximately from intractable posterior distributions is Markov chain Monte Carlo (MCMC), but it is difficult to craft good proposal distributions {for multi-modal distributions over} language data \cite{miao2018cgmh,zhang-etal-2020-language-generation,lew2023sequential}, and inference on a new input may be prohibitively slow." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="2011.12334" id="5436">
        <attvalues>
          <attvalue for="5" value=" A standard method to sample approximately from intractable posterior distributions is Markov chain Monte Carlo (MCMC), but it is difficult to craft good proposal distributions {for multi-modal distributions over} language data \cite{miao2018cgmh,zhang-etal-2020-language-generation,lew2023sequential}, and inference on a new input may be prohibitively slow." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="1707.06347" id="5438">
        <attvalues>
          <attvalue for="5" value=" Alternatively, one can turn to reinforcement learning (RL) approaches such as proximal policy optimization~\cite[PPO;][]{schulman2017proximal}, where the language model is treated as a policy to be fine-tuned." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="2210.10760" id="5439">
        <attvalues>
          <attvalue for="5" value=" In both cases, issues with this mode collapse are exacerbated when the target distribution is misspecified, leading to the undesirable behavior of overoptimized samplers \cite{gao2022scaling}." />
        </attvalues>
      </edge>
      <edge source="2310.04363" target="2106.04399" id="5440">
        <attvalues>
          <attvalue for="5" value=" One way to implement amortized inference for high-dimensional discrete data such as text is using generative flow networks \cite[GFlowNets;][]{bengio2021flow}, which are diversity-seeking reinforcement learning algorithms that train policies to sample objects (such as a token sequence $Z$) with probability proportional to a given reward function, such as the joint $p_{\rm LM}(XZY)$." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="1911.02116" id="5442">
        <attvalues>
          <attvalue for="5" value=" Finally, \cite{de-vries-etal-2022-make} combined multiple typological features in a single regression model to predict the cross-lingual transfer performance of XLM-R \cite{conneau2019unsupervised} in POS tagging." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="2010.11934" id="5443">
        <attvalues>
          <attvalue for="5" value=" In our work, we try to better understand how multi-lingual pre-trained language models, such as mT5 \cite{xue2020mt5}, transfer {any} linguistic and semantic knowledge across languages." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="2001.08210" id="5444">
        <attvalues>
          <attvalue for="5" value=" Multi-lingual language models (LM), such as mBERT \cite{devlin-etal-2019-bert}, XLM-R \cite{conneau2019unsupervised}, mT5 \cite{xue2020mt5}, mBART \cite{liu-etal-2020-multilingual-denoising}, have been remarkably successful in enabling natural language tasks in low-resource languages through cross-lingual transfer from high-resource languages." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="1906.01502" id="5445">
        <attvalues>
          <attvalue for="5" value=" Probing studies from \cite{pires-etal-2019-multilingual} and \cite{xue2020mt5} suggest that large multi-lingual language models exhibit zero-shot transfer ability and can deliver state-of-art performance for low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="1911.03310" id="5446">
        <attvalues>
          <attvalue for="5" value=" The fact that this model exhibits cross-lingual transfer may suggest that it is somehow aligning its learned ``semantic spaces'' of different languages \cite{libovicky2019language,muller2021align}." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="1712.00409" id="5447">
        <attvalues>
          <attvalue for="5" value=" Recently, a line of work by \cite{hestness2017deep,kaplan2020scaling} has analyzed the scaling effects of parameters, corpus size and number of training steps on pre-training loss in language models \cite{devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="2001.08361" id="5448">
        <attvalues>
          <attvalue for="5" value=" Recently, a line of work by \cite{hestness2017deep,kaplan2020scaling} has analyzed the scaling effects of parameters, corpus size and number of training steps on pre-training loss in language models \cite{devlin-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="2102.04074" id="5449">
        <attvalues>
          <attvalue for="5" value=" \cite{hutter2021learning} extended this analysis to the out-of-distribution transfer setting and showed that the effective amount of data transferred from the training distribution to the target distribution follows a power law of the number of parameters and the amount of training data." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="2005.00870" id="5450">
        <attvalues>
          <attvalue for="5" value=" Similarly, \cite{xia-etal-2020-predicting} showed that the performance of a wide range of language tasks could be predicted with relatively good accuracy." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="1912.07840" id="5451">
        <attvalues>
          <attvalue for="5" value=" \cite{K2020Cross-Lingual} have suggested that ``structural similarity'' between the source and target languages is one of the most important factors regardless of the lexical overlap or word frequency similarity." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="1905.12688" id="5452">
        <attvalues>
          <attvalue for="5" value=" \cite{lin-etal-2019-choosing} combined multiple features into a gradient-boosting model to predict zero-shot cross-lingual transfer performance." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="2101.11109" id="5453">
        <attvalues>
          <attvalue for="5" value=" The fact that this model exhibits cross-lingual transfer may suggest that it is somehow aligning its learned ``semantic spaces'' of different languages \cite{libovicky2019language,muller2021align}." />
        </attvalues>
      </edge>
      <edge source="2212.01757" target="2005.09093" id="5454">
        <attvalues>
          <attvalue for="5" value=" We posit that transfer between some languages is more dominant than others, based on the premise that not all language pairs are born equal \cite{wu-dredze-2020-languages}." />
        </attvalues>
      </edge>
      <edge source="2310.19572" target="2005.14165" id="5455">
        <attvalues>
          <attvalue for="5" value=" One of the most remarkable abilities of LLMs is in-context learning (ICL)~\cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2310.19572" target="2212.04037" id="5456">
        <attvalues>
          <attvalue for="5" value=" Despite the promising results demonstrated by existing ICL models~\cite{DBLP:journals/corr/abs-2212-04037,DBLP:conf/nips/Wei0SBIXCLZ22}, their causal nature in language modeling restricts each token's attention solely to its preceding tokens." />
        </attvalues>
      </edge>
      <edge source="2310.19572" target="2201.11903" id="5457">
        <attvalues>
          <attvalue for="5" value=" Despite the promising results demonstrated by existing ICL models~\cite{DBLP:journals/corr/abs-2212-04037,DBLP:conf/nips/Wei0SBIXCLZ22}, their causal nature in language modeling restricts each token's attention solely to its preceding tokens." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="1602.02373" id="5458">
        <attvalues>
          <attvalue for="5" value=" Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="1503.00075" id="5459">
        <attvalues>
          <attvalue for="5" value=" Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="1706.03762" id="5460">
        <attvalues>
          <attvalue for="5" value=" Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="1907.11692" id="5462">
        <attvalues>
          <attvalue for="5" value=" Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="2006.03654" id="5463">
        <attvalues>
          <attvalue for="5" value=" Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="2005.14165" id="5465">
        <attvalues>
          <attvalue for="5" value=" Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="1910.13461" id="5467">
        <attvalues>
          <attvalue for="5" value=" Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="2201.08239" id="5468">
        <attvalues>
          <attvalue for="5" value=" Throughout the evolution of large language models (LLMs)\cite{johnson2016supervised,tai2015improved,zhu2015long,vaswani2017attention,kenton2019bert,liu2019roberta,hedeberta,sanh2019distilbert}, the recently launched ChatGPT has attracted mass attention in the NLP community~\cite{radford2018improving,radford2019language,brown2020language,raffel2020exploring,lewis2020bart,thoppilan2022lamda}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="2301.10035" id="5469">
        <attvalues>
          <attvalue for="5" value=" In a range of NLP tasks, including question answering, dialogue, summarization, named entity recognition, and sentiment analysis, ChatGPT has demonstrated impressive performance, outperforming many models even in zero-shot settings~\cite{nov2023putting,liu2023deid,shen2023hugginggpt,qin2023chatgpt,peng2023towards}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="2302.06476" id="5471">
        <attvalues>
          <attvalue for="5" value=" In a range of NLP tasks, including question answering, dialogue, summarization, named entity recognition, and sentiment analysis, ChatGPT has demonstrated impressive performance, outperforming many models even in zero-shot settings~\cite{nov2023putting,liu2023deid,shen2023hugginggpt,qin2023chatgpt,peng2023towards}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="1809.05679" id="5473">
        <attvalues>
          <attvalue for="5" value=" Unlike traditional graph extraction methods, which mainly rely on heuristics~\cite{yao2019graph} or graph refinement~\cite{zhao2021data}, our proposed method does not require further refinement of the obtained graph, which reduces computational costs." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="2111.01243" id="5474">
        <attvalues>
          <attvalue for="5" value=" i) How to effectively utilize ChatGPT in scenarios where a substantial amount of labeled data is available, making it challenging to fully harness the potential of in-context learning~\cite{min2021recent}? ii) How to enhance interpretability in the decision-making process? In this work, instead of developing a customized interpretation algorithm, we propose to tackle the problem from a data-centric perspective~\cite{zha2023data}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="2301.04819" id="5475">
        <attvalues>
          <attvalue for="5" value=" i) How to effectively utilize ChatGPT in scenarios where a substantial amount of labeled data is available, making it challenging to fully harness the potential of in-context learning~\cite{min2021recent}? ii) How to enhance interpretability in the decision-making process? In this work, instead of developing a customized interpretation algorithm, we propose to tackle the problem from a data-centric perspective~\cite{zha2023data}." />
        </attvalues>
      </edge>
      <edge source="2305.03513" target="2006.06830" id="5476">
        <attvalues>
          <attvalue for="5" value=" Unlike traditional graph extraction methods, which mainly rely on heuristics~\cite{yao2019graph} or graph refinement~\cite{zhao2021data}, our proposed method does not require further refinement of the obtained graph, which reduces computational costs." />
        </attvalues>
      </edge>
      <edge source="2010.09828" target="1904.09077" id="5478">
        <attvalues>
          <attvalue for="5" value=" Since this work emerged, there have been major advances in multilingual NLP~\cite{wu-dredze-2019-beto, pires-etal-2019-multilingual}." />
        </attvalues>
      </edge>
      <edge source="2010.09828" target="1911.02116" id="5480">
        <attvalues>
          <attvalue for="5" value=" These models, such as multilingual BERT or XMLR~\cite{conneau2019unsupervised}, have achieved impressive results on a range of multilingual NLP tasks, including part of speech tagging~\cite{tsai-etal-2019-small}, parsing~\cite{wang-etal-2019-cross, kondratyuk-straka-2019-75}, and semantic similarity~\cite{lo-simard-2019-fully,reimers-gurevych-2019-sentence}." />
        </attvalues>
      </edge>
      <edge source="2010.09828" target="1909.06775" id="5482">
        <attvalues>
          <attvalue for="5" value=" These models, such as multilingual BERT or XMLR~\cite{conneau2019unsupervised}, have achieved impressive results on a range of multilingual NLP tasks, including part of speech tagging~\cite{tsai-etal-2019-small}, parsing~\cite{wang-etal-2019-cross, kondratyuk-straka-2019-75}, and semantic similarity~\cite{lo-simard-2019-fully,reimers-gurevych-2019-sentence}." />
        </attvalues>
      </edge>
      <edge source="2010.09828" target="1908.10084" id="5483">
        <attvalues>
          <attvalue for="5" value=" These models, such as multilingual BERT or XMLR~\cite{conneau2019unsupervised}, have achieved impressive results on a range of multilingual NLP tasks, including part of speech tagging~\cite{tsai-etal-2019-small}, parsing~\cite{wang-etal-2019-cross, kondratyuk-straka-2019-75}, and semantic similarity~\cite{lo-simard-2019-fully,reimers-gurevych-2019-sentence}." />
        </attvalues>
      </edge>
      <edge source="2112.09866" target="1911.02116" id="5491">
        <attvalues>
          <attvalue for="5" value=" The emergence of multilingual models: mBERT \cite{devlin-etal-2019-bert} and XLM-RoBERTa\cite{conneau-etal-2020-unsupervised} made it possible to leverage English data to improve the performance of low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2112.09866" target="2005.00247" id="5492">
        <attvalues>
          <attvalue for="5" value=" We conducted the experiments in two different setups, Houlsby\cite{pmlr-v97-houlsby19a} and Pfeiffer\cite{pfeiffer-etal-2021-adapterfusion,pfeiffer-etal-2020-mad}." />
        </attvalues>
      </edge>
      <edge source="2112.09866" target="1910.07475" id="5496">
        <attvalues>
          <attvalue for="5" value=" Our models are evaluated on the combination of XQuAD\cite{artetxe-etal-2020-cross} and MLQA\cite{lewis-etal-2020-mlqa} datasets which are similar to SQuAD \cite{rajpurkar-etal-2016-squad}." />
        </attvalues>
      </edge>
      <edge source="2112.09866" target="1606.05250" id="5497">
        <attvalues>
          <attvalue for="5" value=" Our models are evaluated on the combination of XQuAD\cite{artetxe-etal-2020-cross} and MLQA\cite{lewis-etal-2020-mlqa} datasets which are similar to SQuAD \cite{rajpurkar-etal-2016-squad}." />
        </attvalues>
      </edge>
      <edge source="2112.09866" target="1902.00751" id="5498">
        <attvalues>
          <attvalue for="5" value=" We conducted the experiments in two different setups, Houlsby\cite{pmlr-v97-houlsby19a} and Pfeiffer\cite{pfeiffer-etal-2021-adapterfusion,pfeiffer-etal-2020-mad}." />
        </attvalues>
      </edge>
      <edge source="2112.09866" target="2005.00052" id="5499">
        <attvalues>
          <attvalue for="5" value=" The multi-task adapter (MAD-X) \cite{pfeiffer-etal-2020-mad} outperforms the state-of-the-art models in cross-lingual transfer across a representative set of typologically diverse languages on question answering." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2107.03374" id="5500">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have revolutionized text generation, with applications ranging from code development \cite{chen2021evaluating} to information retrieval \cite{zhu2023large}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2308.07107" id="5501">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have revolutionized text generation, with applications ranging from code development \cite{chen2021evaluating} to information retrieval \cite{zhu2023large}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2311.05232" id="5502">
        <attvalues>
          <attvalue for="5" value=" However, alongside their impressive capabilities, LLMs possess a troubling tendency to fabricate information, generating outputs that diverge from factual reality – a phenomenon dubbed ``hallucination'' \cite{huang2023survey}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2310.17651" id="5503">
        <attvalues>
          <attvalue for="5" value=" Moreover, when those risk scores are calibrated, they are not only interpretable but ``trustworthy'' in the sense that they can be safely used as if they were true probabilities \cite{noarov2023high}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2206.01067" id="5504">
        <attvalues>
          <attvalue for="5" value=" Our approach mirrors the robust assurances offered by conformal prediction, where multicalibration (of quantiles) has been used to give group-conditional guarantees \cite{bastani2022practical,jung2022batch,gibbs2023conformal}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2209.15145" id="5505">
        <attvalues>
          <attvalue for="5" value=" Our approach mirrors the robust assurances offered by conformal prediction, where multicalibration (of quantiles) has been used to give group-conditional guarantees \cite{bastani2022practical,jung2022batch,gibbs2023conformal}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2305.12616" id="5506">
        <attvalues>
          <attvalue for="5" value=" Our approach mirrors the robust assurances offered by conformal prediction, where multicalibration (of quantiles) has been used to give group-conditional guarantees \cite{bastani2022practical,jung2022batch,gibbs2023conformal}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2307.03109" id="5507">
        <attvalues>
          <attvalue for="5" value=" Additional Related Work Numerous recent surveys focus on hallucinations in LLMs \cite{chang2023survey, huang2023survey, ji2023survey, rawte2023survey, tonmoy2024comprehensive, zhang2023siren, guerreiro2023hallucinations}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2303.08896" id="5508">
        <attvalues>
          <attvalue for="5" value=" Key contributions in this domain include \cite{manakul2023selfcheckgpt, rebedea2023nemo}, which evaluate consistency, similarity, and agreement among alternative generated responses." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2207.05221" id="5509">
        <attvalues>
          <attvalue for="5" value=" \cite{kadavath2022language, friel2023chainpoll} directly engage LLMs by posing inquiries about correctness or consistency within a single answer." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2306.10193" id="5511">
        <attvalues>
          <attvalue for="5" value=" Several recent papers use conformal prediction to derive sets of prompt completions, offering marginal coverage guarantees (eg~for 90\% of prompts, at least one completion in the set should be ``good'') \cite{quach2023conformal, kumar2023conformal, deutschmann2023conformal, ren2023robots, zecchin2023forking}." />
        </attvalues>
      </edge>
      <edge source="2404.04689" target="2305.18404" id="5512">
        <attvalues>
          <attvalue for="5" value=" Among these, \cite{kumar2023conformal} is closest to our approach but requires &quot;group-specific&quot; prompting strategies." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="1706.03762" id="5513">
        <attvalues>
          <attvalue for="5" value=" Thanks to advances in language modeling, particularly with the use of the transformer architecture \cite{vaswani2017attention}, NLP models can now generate text that is often difficult to distinguish from that written by a human." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="1804.10731" id="5514">
        <attvalues>
          <attvalue for="5" value=" Applications include relational reasoning~\cite{shah2018bootstrapping}, task completion~\cite{shi2018sentiment}, and query fulfillment~\cite{serban2017deep}, whose action spaces are structured enough to be represented by hand-crafted features." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="1709.02349" id="5515">
        <attvalues>
          <attvalue for="5" value=" Applications include relational reasoning~\cite{shah2018bootstrapping}, task completion~\cite{shi2018sentiment}, and query fulfillment~\cite{serban2017deep}, whose action spaces are structured enough to be represented by hand-crafted features." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="1907.00456" id="5516">
        <attvalues>
          <attvalue for="5" value=" To handle more complex dialogues, recent approaches use language models (LMs) to extract semantic representations from conversation histories, treat them as dialogue states, and apply RL to learn a word-level generative DM agent~\cite{jaques2019way, li2016deep, li2017adversarial, shin2020generating}." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="2005.01643" id="5517">
        <attvalues>
          <attvalue for="5" value=" While one can potentially address the DM problem using offline RL, issues such as model exploitation leading to distribution shift on the state and action spaces, when training on static datasets are of paramount concern \cite{levine2020offline}." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="1902.08858" id="5518">
        <attvalues>
          <attvalue for="5" value=" As a result, naive application of RL to DM may result in poorly-performing agents that generate incomprehensible utterances~\cite{zhao2019rethinking}." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="2206.00059" id="5519">
        <attvalues>
          <attvalue for="5" value=" We tackle the above issues related to the use of offline RL in DM systems by leveraging recent advances in Mixture-of-Expert Language Models (MoE-LMs) \cite{chow2023mixture}." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="1909.07547" id="5520">
        <attvalues>
          <attvalue for="5" value=" This circumvents the word-level credit-assignment issue, particularly challenging in long conversations \cite{saleh2020hierarchical}." />
        </attvalues>
      </edge>
      <edge source="2302.10850" target="2204.08426" id="5521">
        <attvalues>
          <attvalue for="5" value=" Fourth, in contrast to the findings of \cite{verma2022chai}, where offline RL agents tend to lack utterance diversity (due to potential reward hacking and optimization of a single objective), our MoE-based DM agents by design are adept at generating utterances that reflect different intents." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2110.13900" id="5522">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) from speech data has been investigated\cite{chen2021wavlm,zhang2021bigssl,pmlr-v139-wang21y,chung2021w2v,ao2021speecht5,jiang2021further,wang2020unsupervised,liu2020mockingjay,liu2021tera} because of its great potential of improving low-resource tasks through learning prior knowledge from large amounts of data without annotations." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2109.13226" id="5523">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) from speech data has been investigated\cite{chen2021wavlm,zhang2021bigssl,pmlr-v139-wang21y,chung2021w2v,ao2021speecht5,jiang2021further,wang2020unsupervised,liu2020mockingjay,liu2021tera} because of its great potential of improving low-resource tasks through learning prior knowledge from large amounts of data without annotations." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2101.07597" id="5524">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) from speech data has been investigated\cite{chen2021wavlm,zhang2021bigssl,pmlr-v139-wang21y,chung2021w2v,ao2021speecht5,jiang2021further,wang2020unsupervised,liu2020mockingjay,liu2021tera} because of its great potential of improving low-resource tasks through learning prior knowledge from large amounts of data without annotations." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2108.06209" id="5525">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) from speech data has been investigated\cite{chen2021wavlm,zhang2021bigssl,pmlr-v139-wang21y,chung2021w2v,ao2021speecht5,jiang2021further,wang2020unsupervised,liu2020mockingjay,liu2021tera} because of its great potential of improving low-resource tasks through learning prior knowledge from large amounts of data without annotations." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2110.07205" id="5526">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) from speech data has been investigated\cite{chen2021wavlm,zhang2021bigssl,pmlr-v139-wang21y,chung2021w2v,ao2021speecht5,jiang2021further,wang2020unsupervised,liu2020mockingjay,liu2021tera} because of its great potential of improving low-resource tasks through learning prior knowledge from large amounts of data without annotations." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2001.10603" id="5527">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) from speech data has been investigated\cite{chen2021wavlm,zhang2021bigssl,pmlr-v139-wang21y,chung2021w2v,ao2021speecht5,jiang2021further,wang2020unsupervised,liu2020mockingjay,liu2021tera} because of its great potential of improving low-resource tasks through learning prior knowledge from large amounts of data without annotations." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="1910.12638" id="5528">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) from speech data has been investigated\cite{chen2021wavlm,zhang2021bigssl,pmlr-v139-wang21y,chung2021w2v,ao2021speecht5,jiang2021further,wang2020unsupervised,liu2020mockingjay,liu2021tera} because of its great potential of improving low-resource tasks through learning prior knowledge from large amounts of data without annotations." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2104.11462" id="5531">
        <attvalues>
          <attvalue for="5" value=" SSL models can be used in two manners: 1) feature extraction to replace human-designed features\cite{yang2021superb,evain2021lebenchmark,chang2021exploration}; and 2) model initialization for finetuning downstream tasks\cite{vyas2021comparing,misra2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2110.04590" id="5532">
        <attvalues>
          <attvalue for="5" value=" SSL models can be used in two manners: 1) feature extraction to replace human-designed features\cite{yang2021superb,evain2021lebenchmark,chang2021exploration}; and 2) model initialization for finetuning downstream tasks\cite{vyas2021comparing,misra2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2104.02558" id="5533">
        <attvalues>
          <attvalue for="5" value=" SSL models can be used in two manners: 1) feature extraction to replace human-designed features\cite{yang2021superb,evain2021lebenchmark,chang2021exploration}; and 2) model initialization for finetuning downstream tasks\cite{vyas2021comparing,misra2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="1904.03240" id="5534">
        <attvalues>
          <attvalue for="5" value=" For example, autoregressive predictive coding (APC) uses temporally-shifted sequences to perform prediction such that the model predicts future frames from previous frames\cite{chung2019unsupervised,chung2020generative,Ravi2020}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="1910.12607" id="5535">
        <attvalues>
          <attvalue for="5" value=" First, autoregressive predictions at different temporal distances are shown to enable the pretrained model to learn more effectively \cite{chung2020generative}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2008.03615" id="5536">
        <attvalues>
          <attvalue for="5" value=" For example, autoregressive predictive coding (APC) uses temporally-shifted sequences to perform prediction such that the model predicts future frames from previous frames\cite{chung2019unsupervised,chung2020generative,Ravi2020}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="1807.03748" id="5537">
        <attvalues>
          <attvalue for="5" value=" Different from APC and Bi-APC where the reconstruction loss is used, Wav2vec-based methods are implemented to include negative samples, and a contrastive loss is utilized to increase the distance from the output to negative samples and decrease that distance to the positive sample\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="1904.05862" id="5538">
        <attvalues>
          <attvalue for="5" value=" Different from APC and Bi-APC where the reconstruction loss is used, Wav2vec-based methods are implemented to include negative samples, and a contrastive loss is utilized to increase the distance from the output to negative samples and decrease that distance to the positive sample\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2006.11477" id="5539">
        <attvalues>
          <attvalue for="5" value=" Different from APC and Bi-APC where the reconstruction loss is used, Wav2vec-based methods are implemented to include negative samples, and a contrastive loss is utilized to increase the distance from the output to negative samples and decrease that distance to the positive sample\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="1910.05453" id="5540">
        <attvalues>
          <attvalue for="5" value=" Different from APC and Bi-APC where the reconstruction loss is used, Wav2vec-based methods are implemented to include negative samples, and a contrastive loss is utilized to increase the distance from the output to negative samples and decrease that distance to the positive sample\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2106.07447" id="5541">
        <attvalues>
          <attvalue for="5" value=" A more recent SSL framework, HuBERT\cite{hsu2021hubertc,hsu2021hubertj}, creates the pseudo-label of each speech frame using clustering techniques like K-means." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2002.02848" id="5542">
        <attvalues>
          <attvalue for="5" value=" These methods have been shown to be effective for low-resource ASR tasks such as low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec} and accented speech\cite{li2021accent}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2110.04934" id="5543">
        <attvalues>
          <attvalue for="5" value=" These methods have been shown to be effective for low-resource ASR tasks such as low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec} and accented speech\cite{li2021accent}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2110.03520" id="5544">
        <attvalues>
          <attvalue for="5" value=" These methods have been shown to be effective for low-resource ASR tasks such as low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec} and accented speech\cite{li2021accent}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2110.07957" id="5545">
        <attvalues>
          <attvalue for="5" value=" However, a weakness of SSL training is domain shifting that happens when the domain of the finetuning data is different than that of the pretraining data\cite{meng2021don,hsu2021robust}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2104.01027" id="5546">
        <attvalues>
          <attvalue for="5" value=" Although a performance improvement can be observed when the magnitude of the pretraining data is large enough, previous work has shown that additional gains can be obtained by including target domain data in the ASR pretraining stage\cite{hsu2021robust,hwang2021large}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2110.03560" id="5547">
        <attvalues>
          <attvalue for="5" value=" Previous studies proposed to perform adaptation of supervised models either during or after the finetuning stage \cite{khurana2021magic,huo2021incremental}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2110.00155" id="5548">
        <attvalues>
          <attvalue for="5" value=" Previous studies proposed to perform adaptation of supervised models either during or after the finetuning stage \cite{khurana2021magic,huo2021incremental}." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2109.06952" id="5549">
        <attvalues>
          <attvalue for="5" value=" In \cite{tomanek2021residual, houlsby2019parameter}, residual adapters are inserted to achieve a parameter efficient adaptation for low-resource supervised tasks, but the performance is worse than finetuning the entire model." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2105.11905" id="5550">
        <attvalues>
          <attvalue for="5" value=" In \cite{hou2021exploiting,kannan2019large,rebuffi2017learning}, residual adapters are applied to learn domain specific parameters to achieve robust models for various domains." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="1909.05330" id="5551">
        <attvalues>
          <attvalue for="5" value=" In \cite{hou2021exploiting,kannan2019large,rebuffi2017learning}, residual adapters are applied to learn domain specific parameters to achieve robust models for various domains." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="1705.08045" id="5552">
        <attvalues>
          <attvalue for="5" value=" In \cite{hou2021exploiting,kannan2019large,rebuffi2017learning}, residual adapters are applied to learn domain specific parameters to achieve robust models for various domains." />
        </attvalues>
      </edge>
      <edge source="2305.00115" target="2206.07931" id="5553">
        <attvalues>
          <attvalue for="5" value=" Different from \cite{fan2022draft}, DRAFT's performance is examined with Bi-APC, and ablation studies are conducted for a better understanding of the DRAFT framework." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2110.13900" id="5554">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) for speech has been investigated\cite{chen2021wavlm,zhang2021bigssl,chung2021w2v,ao2021speecht5,liu2021tera} because of its great potential in improving low-resource ASR tasks." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2109.13226" id="5555">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) for speech has been investigated\cite{chen2021wavlm,zhang2021bigssl,chung2021w2v,ao2021speecht5,liu2021tera} because of its great potential in improving low-resource ASR tasks." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2108.06209" id="5556">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) for speech has been investigated\cite{chen2021wavlm,zhang2021bigssl,chung2021w2v,ao2021speecht5,liu2021tera} because of its great potential in improving low-resource ASR tasks." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2110.07205" id="5557">
        <attvalues>
          <attvalue for="5" value=" Recently, self-supervised learning (SSL) for speech has been investigated\cite{chen2021wavlm,zhang2021bigssl,chung2021w2v,ao2021speecht5,liu2021tera} because of its great potential in improving low-resource ASR tasks." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="1904.03240" id="5559">
        <attvalues>
          <attvalue for="5" value=" For example, autoregressive predictive coding (APC) uses a shifted input sequence as supervision such that the model predicts future frames from previous frames\cite{chung2019unsupervised,chung2020generative,Ravi2020}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="1910.12607" id="5560">
        <attvalues>
          <attvalue for="5" value=" For example, autoregressive predictive coding (APC) uses a shifted input sequence as supervision such that the model predicts future frames from previous frames\cite{chung2019unsupervised,chung2020generative,Ravi2020}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2008.03615" id="5561">
        <attvalues>
          <attvalue for="5" value=" For example, autoregressive predictive coding (APC) uses a shifted input sequence as supervision such that the model predicts future frames from previous frames\cite{chung2019unsupervised,chung2020generative,Ravi2020}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="1807.03748" id="5562">
        <attvalues>
          <attvalue for="5" value=" Different from APC, Wav2vec-based methods include sampled negative frames in a contrastive loss to increase discrimination between frames in a way that the learned embedding is closer to the positive frame and more distant to the negative frames\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="1904.05862" id="5563">
        <attvalues>
          <attvalue for="5" value=" Different from APC, Wav2vec-based methods include sampled negative frames in a contrastive loss to increase discrimination between frames in a way that the learned embedding is closer to the positive frame and more distant to the negative frames\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2006.11477" id="5564">
        <attvalues>
          <attvalue for="5" value=" Different from APC, Wav2vec-based methods include sampled negative frames in a contrastive loss to increase discrimination between frames in a way that the learned embedding is closer to the positive frame and more distant to the negative frames\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="1910.05453" id="5565">
        <attvalues>
          <attvalue for="5" value=" Different from APC, Wav2vec-based methods include sampled negative frames in a contrastive loss to increase discrimination between frames in a way that the learned embedding is closer to the positive frame and more distant to the negative frames\cite{oord2018representation,schneider2019wav2vec,baevski2020wav2vec,baevski2019vq}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2106.07447" id="5566">
        <attvalues>
          <attvalue for="5" value=" A more recent SSL framework, HuBERT\cite{hsu2021hubertc,hsu2021hubertj}, creates a pseudo-label for each speech frame using clustering techniques like K-means." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2104.11462" id="5568">
        <attvalues>
          <attvalue for="5" value=" Models learned with SSL objectives can be used in two manners: 1) feature extraction as a replacement of hand-crafted speech features\cite{yang2021superb,evain2021lebenchmark,chang2021exploration}; or 2) model initialization for finetuning downstream tasks\cite{jiang2021further,wang2020unsupervised,misra2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2110.04590" id="5569">
        <attvalues>
          <attvalue for="5" value=" Models learned with SSL objectives can be used in two manners: 1) feature extraction as a replacement of hand-crafted speech features\cite{yang2021superb,evain2021lebenchmark,chang2021exploration}; or 2) model initialization for finetuning downstream tasks\cite{jiang2021further,wang2020unsupervised,misra2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2001.10603" id="5570">
        <attvalues>
          <attvalue for="5" value=" Models learned with SSL objectives can be used in two manners: 1) feature extraction as a replacement of hand-crafted speech features\cite{yang2021superb,evain2021lebenchmark,chang2021exploration}; or 2) model initialization for finetuning downstream tasks\cite{jiang2021further,wang2020unsupervised,misra2021comparison}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2002.02848" id="5571">
        <attvalues>
          <attvalue for="5" value=" SSL has been shown to be effective in ASR of low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec}, accented speech\cite{li2021accent}, and child ASR\cite{fan2021bi,wang2021low}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2012.12121" id="5572">
        <attvalues>
          <attvalue for="5" value=" SSL has been shown to be effective in ASR of low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec}, accented speech\cite{li2021accent}, and child ASR\cite{fan2021bi,wang2021low}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2110.04934" id="5573">
        <attvalues>
          <attvalue for="5" value=" SSL has been shown to be effective in ASR of low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec}, accented speech\cite{li2021accent}, and child ASR\cite{fan2021bi,wang2021low}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2110.03520" id="5574">
        <attvalues>
          <attvalue for="5" value=" SSL has been shown to be effective in ASR of low-resource languages\cite{riviere2020unsupervised,yi2020applying}, noisy speech\cite{wang2021wav2vec}, accented speech\cite{li2021accent}, and child ASR\cite{fan2021bi,wang2021low}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2110.07957" id="5575">
        <attvalues>
          <attvalue for="5" value=" However, a main weakness of SSL is that training from one domain causes domain shifting when finetuning on data from a different domain\cite{meng2021don,sanabria2022measuring}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2104.01027" id="5577">
        <attvalues>
          <attvalue for="5" value=" To address this issue, previous work presented robust pretrained models by adding target domain data during pretraining\cite{hsu2021robust,hwang2021large}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2110.03560" id="5578">
        <attvalues>
          <attvalue for="5" value=" In \cite{khurana2021magic}, un-annotated target domain data are used for semi-supervised learning during the finetuning stage to alleviate the performance degradation caused by domain shifting." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2305.00115" id="5579">
        <attvalues>
          <attvalue for="5" value=" We presented partial results of DRAFT in a paper that is in review\cite{fan2022towards}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2105.11905" id="5580">
        <attvalues>
          <attvalue for="5" value=" Note that residual adapters have been used before to learn domain specific parameters for adaptation\cite{hou2021exploiting,kannan2019large,rebuffi2017learning, tomanek2021residual}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="1909.05330" id="5581">
        <attvalues>
          <attvalue for="5" value=" Note that residual adapters have been used before to learn domain specific parameters for adaptation\cite{hou2021exploiting,kannan2019large,rebuffi2017learning, tomanek2021residual}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="1705.08045" id="5582">
        <attvalues>
          <attvalue for="5" value=" Note that residual adapters have been used before to learn domain specific parameters for adaptation\cite{hou2021exploiting,kannan2019large,rebuffi2017learning, tomanek2021residual}." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="1902.00751" id="5583">
        <attvalues>
          <attvalue for="5" value=" In \cite{houlsby2019parameter} residual adapters are inserted to achieve a parameter efficient adaptation for disordered speech." />
        </attvalues>
      </edge>
      <edge source="2206.07931" target="2202.03218" id="5584">
        <attvalues>
          <attvalue for="5" value=" In \cite{thomas2022efficient}, the same idea is used for SSL-pretrained models for efficient adaptation, which is known as adapter tuning in natural language processing." />
        </attvalues>
      </edge>
      <edge source="2311.13581" target="1706.03762" id="5585">
        <attvalues>
          <attvalue for="5" value=" Since the Transformer architecture was proposed (\cite{vaswani2023attention}), large language models have achieved impressive results across natural language processing benchmarks~(\cite{brown2020language})." />
        </attvalues>
      </edge>
      <edge source="2311.13581" target="2005.14165" id="5586">
        <attvalues>
          <attvalue for="5" value=" However, these remarkable achievements were only made possible because of dramatic increases in the number of parameters or model sizes (\cite{brown2020language}, \cite{wei2022emergent}), resulting in considerable memory requirements and greater processing times." />
        </attvalues>
      </edge>
      <edge source="2311.13581" target="1911.02150" id="5588">
        <attvalues>
          <attvalue for="5" value=" This is especially problematic due to the memory-bandwidth cost of recurrently loading the model parameters and the past keys and values tensors (\cite{shazeer2019fast})." />
        </attvalues>
      </edge>
      <edge source="2311.13581" target="2302.01318" id="5589">
        <attvalues>
          <attvalue for="5" value=" Importantly, the generation quality of the original large model is guaranteed by the rejection scheme that keeps only tokens that are generated with an identical distribution than the large model (\cite{chen2023accelerating}, \cite{leviathan2023fast})." />
        </attvalues>
      </edge>
      <edge source="2311.13581" target="2211.17192" id="5590">
        <attvalues>
          <attvalue for="5" value=" Importantly, the generation quality of the original large model is guaranteed by the rejection scheme that keeps only tokens that are generated with an identical distribution than the large model (\cite{chen2023accelerating}, \cite{leviathan2023fast})." />
        </attvalues>
      </edge>
      <edge source="2311.13581" target="1904.09324" id="5591">
        <attvalues>
          <attvalue for="5" value=" This solution, called parallel decoding, can be implemented as a masked language model (\cite{ghazvininejad-etal-2019-mask}) or by copying the encoder input in the decoder in the context of encoder-decoder architectures (\cite{gu2018nonautoregressive})." />
        </attvalues>
      </edge>
      <edge source="2311.13581" target="1711.02281" id="5592">
        <attvalues>
          <attvalue for="5" value=" This solution, called parallel decoding, can be implemented as a masked language model (\cite{ghazvininejad-etal-2019-mask}) or by copying the encoder input in the decoder in the context of encoder-decoder architectures (\cite{gu2018nonautoregressive})." />
        </attvalues>
      </edge>
      <edge source="2311.13581" target="1811.03115" id="5593">
        <attvalues>
          <attvalue for="5" value=" Additionally, \cite{stern2018blockwise} focus solely on greedy decoding, and \cite{medusa} do not guarantee a loss-less decoding." />
        </attvalues>
      </edge>
      <edge source="2311.07914" target="2212.10403" id="5594">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) seek to emulate human intelligence through statistical training on extensive datasets~\cite{huang2022towards}." />
        </attvalues>
      </edge>
      <edge source="2311.07914" target="2202.03629" id="5595">
        <attvalues>
          <attvalue for="5" value=" LLMs also face challenges in accurately interpreting phrases or terms when the context is vague and resides in a knowledge gap region of the model, leading to outputs that may sound plausible but are often irrelevant or incorrect~\cite{ji2023survey, lenat2023getting}." />
        </attvalues>
      </edge>
      <edge source="2311.07914" target="2212.10511" id="5596">
        <attvalues>
          <attvalue for="5" value=" This phenomenon, often termed &quot;hallucinations,&quot; undermines the reliability of these models~\cite{mallen2023not}." />
        </attvalues>
      </edge>
      <edge source="2311.07914" target="2310.07343" id="5597">
        <attvalues>
          <attvalue for="5" value=" To effectively tackle this issue, there have been continuous research efforts in making knowledge updates and model tuning~\cite{zhang2023large, mialon2023augmented, petroni2019language}." />
        </attvalues>
      </edge>
      <edge source="2311.07914" target="2211.05994" id="5600">
        <attvalues>
          <attvalue for="5" value=" Related Works: There are several related surveys which discuss LLM augmentation using external knowledge~\cite{hu2023survey, yin2022survey, alkhamissi2022review, ye2022generative, wei2021knowledge, liang2022reasoning, zhang2023large, mialon2023augmented}." />
        </attvalues>
      </edge>
      <edge source="2004.02451" target="1808.09031" id="5609">
        <attvalues>
          <attvalue for="5" value=" On the test set of \cite{marvin-linzen:2018:EMNLP}, we show that LSTM language models (LSTM-LMs) trained by this loss reach near perfect level on most syntactic constructions for which we create negative examples, with only a slight increase of perplexity about 1." />
        </attvalues>
      </edge>
      <edge source="2004.02451" target="1909.00111" id="5610">
        <attvalues>
          <attvalue for="5" value=" So far, the results for RNN language models (RNN-LMs) trained only with raw text are overall negative; prior work has reported low performance on the challenging test cases \cite{marvin-linzen:2018:EMNLP} even with the massive size of the data and model \cite{van-schijndel-EtAl:2019:EMNLP1}, or argue the necessity of an architectural change to track the syntactic structure explicitly \cite{wilcox-etal-2019-structural,kuncoro-EtAl:2018:Long}." />
        </attvalues>
      </edge>
      <edge source="2004.02451" target="1903.00943" id="5611">
        <attvalues>
          <attvalue for="5" value=" So far, the results for RNN language models (RNN-LMs) trained only with raw text are overall negative; prior work has reported low performance on the challenging test cases \cite{marvin-linzen:2018:EMNLP} even with the massive size of the data and model \cite{van-schijndel-EtAl:2019:EMNLP1}, or argue the necessity of an architectural change to track the syntactic structure explicitly \cite{wilcox-etal-2019-structural,kuncoro-EtAl:2018:Long}." />
        </attvalues>
      </edge>
      <edge source="2004.02451" target="1611.01368" id="5612">
        <attvalues>
          <attvalue for="5" value=" Here the task is to evaluate whether a model assigns a higher likelihood on a grammatically correct sentence (\ref{ex:correct_orc}) over an incorrect sentence (\ref{ex:incorrect_orc}) that is minimally different from the original one \cite{Q16-1037}." />
        </attvalues>
      </edge>
      <edge source="2004.02451" target="1706.03542" id="5613">
        <attvalues>
          <attvalue for="5" value=" Our second question is about the true {limitation} of LSTM-LMs: are there still any syntactic constructions that the models cannot handle robustly even with our direct learning signals? This question can be seen as a fine-grained one raised by \cite{enguehard-etal-2017-exploring} with a stronger tool and improved evaluation metric." />
        </attvalues>
      </edge>
      <edge source="2004.02451" target="1906.06438" id="5614">
        <attvalues>
          <attvalue for="5" value=" Another relevant work on the capacity of LSTM-LMs is \cite{kuncoro-etal-2019-scalable}, which shows that by distilling from syntactic LMs \cite{dyer-EtAl:2016:N16-1}, LSTM-LMs can improve their robustness on various agreement phenomena." />
        </attvalues>
      </edge>
      <edge source="2004.02451" target="1602.07776" id="5615">
        <attvalues>
          <attvalue for="5" value=" Another relevant work on the capacity of LSTM-LMs is \cite{kuncoro-etal-2019-scalable}, which shows that by distilling from syntactic LMs \cite{dyer-EtAl:2016:N16-1}, LSTM-LMs can improve their robustness on various agreement phenomena." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2102.05918" id="5629">
        <attvalues>
          <attvalue for="5" value=" Vision-language models (VLMs) such as CLIP \cite{radford2021clip}, ALIGN \cite{pmlr-v139-align}, and Coca \cite{yu2022coca} have become excellent base models in multiple domains, most of which employ a dual-encoder architecture to align the natural images with descriptive texts." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2205.01917" id="5630">
        <attvalues>
          <attvalue for="5" value=" Vision-language models (VLMs) such as CLIP \cite{radford2021clip}, ALIGN \cite{pmlr-v139-align}, and Coca \cite{yu2022coca} have become excellent base models in multiple domains, most of which employ a dual-encoder architecture to align the natural images with descriptive texts." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2109.01134" id="5631">
        <attvalues>
          <attvalue for="5" value=" Overall, there has been increasing attention paid to prompt learning due to its potential to perform significantly better than zero-shot transfer with a few sets of labeled data \cite{zhou2022coop,zhou2022cocoop,zhu2023gradalign,khattak2023maple,jia2022visual}." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2203.12119" id="5632">
        <attvalues>
          <attvalue for="5" value=" Overall, there has been increasing attention paid to prompt learning due to its potential to perform significantly better than zero-shot transfer with a few sets of labeled data \cite{zhou2022coop,zhou2022cocoop,zhu2023gradalign,khattak2023maple,jia2022visual}." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2210.03117" id="5633">
        <attvalues>
          <attvalue for="5" value=" Overall, there has been increasing attention paid to prompt learning due to its potential to perform significantly better than zero-shot transfer with a few sets of labeled data \cite{zhou2022coop,zhou2022cocoop,zhu2023gradalign,khattak2023maple,jia2022visual}." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2203.05557" id="5634">
        <attvalues>
          <attvalue for="5" value=" Overall, there has been increasing attention paid to prompt learning due to its potential to perform significantly better than zero-shot transfer with a few sets of labeled data \cite{zhou2022coop,zhou2022cocoop,zhu2023gradalign,khattak2023maple,jia2022visual}." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2205.14865" id="5635">
        <attvalues>
          <attvalue for="5" value=" Overall, there has been increasing attention paid to prompt learning due to its potential to perform significantly better than zero-shot transfer with a few sets of labeled data \cite{zhou2022coop,zhou2022cocoop,zhu2023gradalign,khattak2023maple,jia2022visual}." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2002.08053" id="5636">
        <attvalues>
          <attvalue for="5" value=" As we see, learning with only candidate labels (also widely known as partial-label learning (PLL) \cite{wang2021pico,wu2022revisiting,lv2020progressive,feng2020provably}) is practically significant, which also has arisen in many vital applications such as web mining \cite{luo2010learning}, online annotation \cite{tang2017confidence} and ecoinformatics \cite{liu2012conditional}." />
        </attvalues>
      </edge>
      <edge source="2407.07638" target="2002.11919" id="5638">
        <attvalues>
          <attvalue for="5" value=" We conjecture that the reason lies in the prior knowledge brought by the fixed class token, which can keep VLMs from over-fitting to the false-positive labels in the candidate label set and provide VLMs with preferred zero-shot ability, thus mitigating the error accumulation problem \cite{yao2021network} in PLL when the label ambiguity increases." />
        </attvalues>
      </edge>
      <edge source="2103.05081" target="1812.06864" id="5639">
        <attvalues>
          <attvalue for="5" value=" Neural language models (LMs), including long short-term memory (LSTM) and Transformer based ones, have significantly improved performance over $n$-gram LMs in automatic speech recognition (ASR)~\cite{mikolov2010recurrent,chen2015recurrent,xu2018neural,zeghidour2018fully, synnaeve2019end, irie2019language,li2020empirical}." />
        </attvalues>
      </edge>
      <edge source="2312.02331" target="1909.01792" id="5643">
        <attvalues>
          <attvalue for="5" value=" Even with the advent of transformer-based language models, RNNs and LSTMs can outperform non-pretrained transformers on various small datasets \cite{melis2019mogrifier}." />
        </attvalues>
      </edge>
      <edge source="2312.02331" target="1803.01271" id="5644">
        <attvalues>
          <attvalue for="5" value=" While powerful, RNN- and LSTM-based models struggle to capture long-range dependencies in their context history \cite{bai2018empirical, sankar-etal-2019-neural}." />
        </attvalues>
      </edge>
      <edge source="2312.02331" target="1611.01702" id="5645">
        <attvalues>
          <attvalue for="5" value=" We train the models from \cite{dieng2016topicrnn, lau2017topically, rezaee2020discrete, guo2020recurrent} on three document-level corpora and evaluate their held-out perplexity." />
        </attvalues>
      </edge>
      <edge source="2312.02331" target="1805.04623" id="5646">
        <attvalues>
          <attvalue for="5" value=" The motivation for combining language models and topic models is to decouple local syntactic structure, which can be modeled by a language model, from document-level semantic concepts, which can be captured by a topic model \cite{khandelwal2018sharp, oconnor-andreas-2021-context}." />
        </attvalues>
      </edge>
      <edge source="2312.02331" target="1704.08012" id="5647">
        <attvalues>
          <attvalue for="5" value=" Moreover, we use a baseline language model that is conditioned on all previously seen document words, rather than being restricted to the current sentence \cite{lau2017topically, rezaee2020discrete, guo2020recurrent}." />
        </attvalues>
      </edge>
      <edge source="2312.02331" target="1409.2329" id="5648">
        <attvalues>
          <attvalue for="5" value=" Our finding: no predictive improvement of TGLMs over a standard LSTM-LM baseline \cite{zaremba2014recurrent}." />
        </attvalues>
      </edge>
      <edge source="2312.02331" target="2106.15195" id="5650">
        <attvalues>
          <attvalue for="5" value=" These studies have uncovered instances where results are not directly comparable, as reported numbers are borrowed from prior works that used different experimental settings \cite{marie-etal-2021-scientific, hoyle2021automated}." />
        </attvalues>
      </edge>
      <edge source="2312.02331" target="1907.06902" id="5651">
        <attvalues>
          <attvalue for="5" value=" Furthermore, they identify cases where baselines are either too weak or improperly tuned \cite{dacrema2019, nityasya-etal-2023-scientific}." />
        </attvalues>
      </edge>
      <edge source="2204.06328" target="2102.03216" id="5652">
        <attvalues>
          <attvalue for="5" value=" Different from intermediate CTC-based approaches \cite{inter-ctc, inter-ctc2, inter-ctc-pruning, inter-KD}, the HuBERT-EE aims to dynamically use the intermediate prediction as the model's final output with minimal WER degradation." />
        </attvalues>
      </edge>
      <edge source="2206.06586" target="1911.02116" id="5653">
        <attvalues>
          <attvalue for="5" value=" \item We propose a 2-step knowledge distillation framework based on mPLMs, eg XLM-RoBERTa~\cite{xlmr-conneau-etal-2020-unsupervised}, to address the FreeTransfer-X." />
        </attvalues>
      </edge>
      <edge source="2206.06586" target="1910.04708" id="5654">
        <attvalues>
          <attvalue for="5" value=" CLT is realized by either aligning parameters of monolingual models or sharing parameters among different languages~\cite{liu-etal-2019-investigating,jacob-mbert-2019,xlmr-conneau-etal-2020-unsupervised,Wang*2020Cross-lingual}." />
        </attvalues>
      </edge>
      <edge source="2206.06586" target="1706.03762" id="5655">
        <attvalues>
          <attvalue for="5" value=" Besides, models in this paradigm are usually large-scale Transformers~\cite{trm_NIPS2017_3f5ee243} based on mPLMs, which limits their deployment in real-world." />
        </attvalues>
      </edge>
      <edge source="2206.06586" target="1503.02531" id="5657">
        <attvalues>
          <attvalue for="5" value=" To address the FreeTransfer-X, we propose a 2-step knowledge distillation~\cite[KD,][]{hinton2015distilling} framework based on mPLM, as shown in Figure~\ref{fig:overview}." />
        </attvalues>
      </edge>
      <edge source="2203.00211" target="2012.07421" id="5659">
        <attvalues>
          <attvalue for="5" value=" They often make incorrect predictions, especially when inputs tend to diverge from their training data distribution \cite{elsahar-galle-2019-annotate,miller2020effect,pmlr-v139-koh21a}." />
        </attvalues>
      </edge>
      <edge source="2203.00211" target="2006.09462" id="5661">
        <attvalues>
          <attvalue for="5" value=" \cite{kamath-etal-2020-selective} proposed a post-hoc calibration-based SP technique for Question-Answering (QA) datasets." />
        </attvalues>
      </edge>
      <edge source="2203.00211" target="2109.07009" id="5662">
        <attvalues>
          <attvalue for="5" value=" \cite{garg2021will} distill the QA model to filter out error-prone questions." />
        </attvalues>
      </edge>
      <edge source="2203.00211" target="1610.02136" id="5663">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, despite the shared goal of making NLP systems robust and reliable for real-world applications, SP has remained underexplored; the community does not know which techniques work best across tasks/settings or even if they consistently outperform the simplest baseline `MaxProb' \cite{hendrycks17baseline} (that uses the maximum softmax probability as the confidence estimator for selective prediction)." />
        </attvalues>
      </edge>
      <edge source="2004.03133" target="1607.06520" id="5664">
        <attvalues>
          <attvalue for="5" value=" \cite{bolukbasi16} enumerated that the automatically generated analogies of ($she,he$) in the Word2Vec \cite{mikolov2013distributed} show the gender biases in significant level." />
        </attvalues>
      </edge>
      <edge source="2004.03133" target="1904.03310" id="5665">
        <attvalues>
          <attvalue for="5" value=" Recent researches have disclosed that word embeddings contain unexpected bias in their geometry on the embedding space \cite{bolukbasi16,zhao2019gender}." />
        </attvalues>
      </edge>
      <edge source="2004.03133" target="1310.4546" id="5666">
        <attvalues>
          <attvalue for="5" value=" \cite{bolukbasi16} enumerated that the automatically generated analogies of ($she,he$) in the Word2Vec \cite{mikolov2013distributed} show the gender biases in significant level." />
        </attvalues>
      </edge>
      <edge source="2004.03133" target="1608.07187" id="5669">
        <attvalues>
          <attvalue for="5" value=" This unwanted bias can cause biased results in the downstream tasks \cite{caliskan17,kiritchenko2018examining,bhaskaran2019good} and gender discrimination in NLP systems." />
        </attvalues>
      </edge>
      <edge source="2004.03133" target="1805.04508" id="5670">
        <attvalues>
          <attvalue for="5" value=" This unwanted bias can cause biased results in the downstream tasks \cite{caliskan17,kiritchenko2018examining,bhaskaran2019good} and gender discrimination in NLP systems." />
        </attvalues>
      </edge>
      <edge source="2004.03133" target="1906.10256" id="5671">
        <attvalues>
          <attvalue for="5" value=" This unwanted bias can cause biased results in the downstream tasks \cite{caliskan17,kiritchenko2018examining,bhaskaran2019good} and gender discrimination in NLP systems." />
        </attvalues>
      </edge>
      <edge source="2303.12320" target="1907.11692" id="5672">
        <attvalues>
          <attvalue for="5" value=" Question-answering models have evolved from rule-based~\cite{7980526} to RNN-based sequence models~\cite{ and now to Transformer-based Language Models (LM) such as RoBERTa-large~\cite{DBLP:journals/corr/abs-1907-11692}." />
        </attvalues>
      </edge>
      <edge source="2303.12320" target="1612.03975" id="5673">
        <attvalues>
          <attvalue for="5" value=" KGs such as Freebase~\cite{bollacker2008freebase}, Wikidata~\cite{vrandevcic2012wikidata}, or ConceptNet~\cite{speer2017conceptnet} store knowledge in the form of graph triplets (topic-relationship-topic) and are well suited for Graph Neural Networks (GNNs), eg ~\cite{welling2016semi}." />
        </attvalues>
      </edge>
      <edge source="2303.12320" target="1609.02907" id="5674">
        <attvalues>
          <attvalue for="5" value=" KGs such as Freebase~\cite{bollacker2008freebase}, Wikidata~\cite{vrandevcic2012wikidata}, or ConceptNet~\cite{speer2017conceptnet} store knowledge in the form of graph triplets (topic-relationship-topic) and are well suited for Graph Neural Networks (GNNs), eg ~\cite{welling2016semi}." />
        </attvalues>
      </edge>
      <edge source="2303.12320" target="1909.02151" id="5675">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{lin-etal-2019-kagnet}~propose a graph network to score answers while~\cite{feng-etal-2020-scalable} focus on a multi-hop message passing framework that allows each node to attend to multi-hop neighbors in a single layer, combining interpretable path-based reasoning with scalable GNNs." />
        </attvalues>
      </edge>
      <edge source="2303.12320" target="2104.06378" id="5676">
        <attvalues>
          <attvalue for="5" value=" The improvements to the WG are combined with the reasoning process of QA-GNN~\cite{yasunaga-etal-2021-qa} and evaluated on three datasets, where we see especially large improvements on domain-specific OpenBookQA (discussed in \ref{23})." />
        </attvalues>
      </edge>
      <edge source="2303.12320" target="2005.00646" id="5677">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{lin-etal-2019-kagnet}~propose a graph network to score answers while~\cite{feng-etal-2020-scalable} focus on a multi-hop message passing framework that allows each node to attend to multi-hop neighbors in a single layer, combining interpretable path-based reasoning with scalable GNNs." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1509.01626" id="5678">
        <attvalues>
          <attvalue for="5" value=" Current approaches for data augmentation in NLP tasks mostly revolve around &#09;thesaurus data augmentation \cite{zhang2015character}, in &#09;which words that belong to the same semantic role are substituted with one &#09;another using a preconstructed lexicon, and noisy data augmentation &#09;\cite{wei2019eda} where random editing operations create perturbations in &#09;the language space." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1809.02305" id="5679">
        <attvalues>
          <attvalue for="5" value=" The recent trend \cite{hu2017toward,yoo2019data,shin2019utterance} gravitates &#09;towards generative data augmentation &#09;(GDA), a class of techniques that leverage deep generative models &#09;such as VAEs to delegate the automatic discovery of novel class-preserving samples to machine learning." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1705.00440" id="5680">
        <attvalues>
          <attvalue for="5" value=" While some notable work exists in &#09;text classification \cite{zhang2015character}, spoken language &#09;understanding \cite{yoo2019data}, and machine translation &#09;\cite{fadaee2017data}, we still lack the full understanding of &#09;utilizing generative models for text augmentation." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1901.11196" id="5681">
        <attvalues>
          <attvalue for="5" value=" Current approaches for data augmentation in NLP tasks mostly revolve around &#09;thesaurus data augmentation \cite{zhang2015character}, in &#09;which words that belong to the same semantic role are substituted with one &#09;another using a preconstructed lexicon, and noisy data augmentation &#09;\cite{wei2019eda} where random editing operations create perturbations in &#09;the language space." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1703.00955" id="5682">
        <attvalues>
          <attvalue for="5" value=" The recent trend \cite{hu2017toward,yoo2019data,shin2019utterance} gravitates &#09;towards generative data augmentation &#09;(GDA), a class of techniques that leverage deep generative models &#09;such as VAEs to delegate the automatic discovery of novel class-preserving samples to machine learning." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1605.06069" id="5683">
        <attvalues>
          <attvalue for="5" value=" The Markov approach \cite{serban2017hierarchical} employs &#09;a sequence-to-sequence variational autoencoder (VAE) &#09;\cite{kingma2013auto} structure to predict the next utterance given &#09;a deterministic context representation, while the holistic approach &#09;\cite{park2018hierarchical} utilizes a set of global &#09;latent variables to encode the entire dialog, &#09;improving the awareness in general dialog structures." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1312.6114" id="5684">
        <attvalues>
          <attvalue for="5" value=" The Markov approach \cite{serban2017hierarchical} employs &#09;a sequence-to-sequence variational autoencoder (VAE) &#09;\cite{kingma2013auto} structure to predict the next utterance given &#09;a deterministic context representation, while the holistic approach &#09;\cite{park2018hierarchical} utilizes a set of global &#09;latent variables to encode the entire dialog, &#09;improving the awareness in general dialog structures." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1804.03424" id="5685">
        <attvalues>
          <attvalue for="5" value=" The Markov approach \cite{serban2017hierarchical} employs &#09;a sequence-to-sequence variational autoencoder (VAE) &#09;\cite{kingma2013auto} structure to predict the next utterance given &#09;a deterministic context representation, while the holistic approach &#09;\cite{park2018hierarchical} utilizes a set of global &#09;latent variables to encode the entire dialog, &#09;improving the awareness in general dialog structures." />
        </attvalues>
      </edge>
      <edge source="2001.08604" target="1801.03558" id="5686">
        <attvalues>
          <attvalue for="5" value=" However, complex and autoregressive VAEs are known to suffer from the risk of &#09;inference collapse \cite{cremer2018inference}, in which &#09;the model converges to a local optimum where the generator network &#09;neglects the latents, reducing the generation controllability." />
        </attvalues>
      </edge>
      <edge source="2007.00049" target="1810.05201" id="5703">
        <attvalues>
          <attvalue for="5" value=" But they are also known to capture a significant amount of stereotypical associations \cite[eg,][]{debias, zhao-etal-2017-men, Bias1, biasSurvey} related to gender, race, nationality, or religion, which can manifest in unwanted and/or potentially harmful ways in downstream tasks \cite{gap, zhao-etal-2019-gender,bias2}." />
        </attvalues>
      </edge>
      <edge source="2007.00049" target="1908.09369" id="5704">
        <attvalues>
          <attvalue for="5" value=" Following~\cite{bias2}, we use Natural Language Inference (NLI) as an effective quantitative probe." />
        </attvalues>
      </edge>
      <edge source="2007.00049" target="1801.07593" id="5705">
        <attvalues>
          <attvalue for="5" value=" Existing methods to mitigate these effects either require expensive retraining of vectors \cite{gan-bias} which can be inefficient, or projecting out information contained along an entire subspace representing a protected concept (such as gender or race) in the embedding space \cite[eg,][]{debias,Bias1, ravfogel2020null}." />
        </attvalues>
      </edge>
      <edge source="2007.00049" target="1903.03862" id="5707">
        <attvalues>
          <attvalue for="5" value=" Projective approaches are difficult to control as they are either insufficient: removing a subspace can still leave residual bias~\cite{gonen2019lipstick, lauscher2019bias}, or too aggressive: in the case of gender, also unnecessarily altering the association between the word pregnant and words like female and mother." />
        </attvalues>
      </edge>
      <edge source="2007.00049" target="1606.01933" id="5708">
        <attvalues>
          <attvalue for="5" value=" A GloVe-based NLI~\cite{parikh2016decomposable} model, without any explicit form of bias mitigation, predicts label entail with a high probability of $97\%$; the notion of a \subject{matriarch} being a \subject{woman} is correctly identified by the model." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1805.03784" id="5709">
        <attvalues>
          <attvalue for="5" value=" It is a core task in knowledge extraction and is important to various downstream applications such as user interest modeling \cite{karatay2015user}, question answering \cite{khalid2008impact} and dialogue systems \cite{bowden2018slugnerds}." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1603.01354" id="5710">
        <attvalues>
          <attvalue for="5" value=" To alleviate the burden of designing hand-crafted features, deep learning models \cite{ma2016end,huang2015bidirectional} have been proposed for NER and shown strong performance." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1508.01991" id="5711">
        <attvalues>
          <attvalue for="5" value=" To alleviate the burden of designing hand-crafted features, deep learning models \cite{ma2016end,huang2015bidirectional} have been proposed for NER and shown strong performance." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1809.03599" id="5712">
        <attvalues>
          <attvalue for="5" value=" To address the label incompleteness issue, some works adopt the partial annotation CRFs to consider all possible labels for unlabeled tokens~\cite{yang2018distantly,shang2018learning}, but they still require a considerable amount of annotated tokens or external tools." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1707.02483" id="5713">
        <attvalues>
          <attvalue for="5" value=" To address the label noise issue, \cite{ni2017weakly} \cite{ni2017weakly} use heuristic rules to filter out sentences with low matching quality." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1908.09659" id="5714">
        <attvalues>
          <attvalue for="5" value=" \cite{cao2019low} \cite{cao2019low} attempt to induce labels for entity mentions based on their occurrence popularity in the concept taxonomy, which can suffer from labeling bias and produce mislabeled data." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1802.05365" id="5715">
        <attvalues>
          <attvalue for="5" value=" To address the challenges in learning from distant supervision, our approach leverages the power of pre-trained language models (eg, ELMo \cite{peters2018deep}, BERT \cite{devlin2018bert}, XLnet \cite{yang2019xlnet}) which are particularly attractive to this task due to the following merits: First, they are very large neural networks trained with huge amounts of unlabeled data in a completely unsupervised manner, which can be cheaply obtained; Second, due to their massive sizes (usually having hundreds of millions or billions of parameters), they have strong expressive power to capture general semantics and syntactic information effectively." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1907.11692" id="5718">
        <attvalues>
          <attvalue for="5" value=" In the first stage, we fine-tune the RoBERTa model~\cite{liu2019roberta} with distantly-matched labels to essentially transfer the semantic knowledge in RoBERTa, which will improve the quality of prediction induced from distant supervision." />
        </attvalues>
      </edge>
      <edge source="2006.15509" target="1909.11942" id="5719">
        <attvalues>
          <attvalue for="5" value=" These language models have achieved state-of-the-art performance in many popular NLP benchmarks with appropriate fine-tuning ~\cite{devlin2018bert,liu2019roberta,yang2019xlnet,Lan2020ALBERT,raffel2019exploring}, which demonstrates their strong ability in modeling the text data." />
        </attvalues>
      </edge>
      <edge source="2407.21515" target="2010.02666" id="5721">
        <attvalues>
          <attvalue for="5" value=" The current best batch sampling procedures rely on clustering the training data set~\cite{hofstatter:2020}." />
        </attvalues>
      </edge>
      <edge source="2405.20053" target="1707.06347" id="5723">
        <attvalues>
          <attvalue for="5" value=" Proximal Policy Optimization (PPO) \cite{schulman2017proximal} and Direct Preference Optimization (DPO) \cite{rafailov2023direct} are two such aligment techniques which have been extensively used to improve the quality of LLM outputs, leading to instruction following agents or chat assistants which are quickly approaching human-baselines in a variety of knowledge and reasoning tasks \cite{open-llm-leaderboard, clark2018think, zellers2019hellaswag, hendrycks2021measuring, lin2022truthfulqa, DBLP:journals/corr/abs-1907-10641, DBLP:journals/corr/abs-2110-14168}." />
        </attvalues>
      </edge>
      <edge source="2405.20053" target="2305.18290" id="5724">
        <attvalues>
          <attvalue for="5" value=" Proximal Policy Optimization (PPO) \cite{schulman2017proximal} and Direct Preference Optimization (DPO) \cite{rafailov2023direct} are two such aligment techniques which have been extensively used to improve the quality of LLM outputs, leading to instruction following agents or chat assistants which are quickly approaching human-baselines in a variety of knowledge and reasoning tasks \cite{open-llm-leaderboard, clark2018think, zellers2019hellaswag, hendrycks2021measuring, lin2022truthfulqa, DBLP:journals/corr/abs-1907-10641, DBLP:journals/corr/abs-2110-14168}." />
        </attvalues>
      </edge>
      <edge source="2405.20053" target="2308.13449" id="5725">
        <attvalues>
          <attvalue for="5" value=" One study \cite{bekbayev2023poison} discovered that performing alignment during the Supervised Fine-Tuning (SFT) stage of training may lead to worse performance on reasoning benchmarks, and another \cite{bai2022training} discovered that SFT alone outperforms RLHF for smaller models with the benefits of RLHF only emerging for models with more than 1 Billion parameters." />
        </attvalues>
      </edge>
      <edge source="2405.20053" target="2204.05862" id="5726">
        <attvalues>
          <attvalue for="5" value=" One study \cite{bekbayev2023poison} discovered that performing alignment during the Supervised Fine-Tuning (SFT) stage of training may lead to worse performance on reasoning benchmarks, and another \cite{bai2022training} discovered that SFT alone outperforms RLHF for smaller models with the benefits of RLHF only emerging for models with more than 1 Billion parameters." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="1706.03762" id="5727">
        <attvalues>
          <attvalue for="5" value=" Most state-of-the-art LMs are based on the transformer architecture \cite{Vaswani2017}, whose theoretical abilities and limitations have been studied extensively; see, eg, the survey by \cite{strobl2023transformers}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="1809.02836" id="5728">
        <attvalues>
          <attvalue for="5" value=" Traditionally, the representational capacity of neural networks, both in terms of lower bounds (what they can provably do) as well as upper bounds (what they can provably not do), has been studied in terms of Boolean sequential models of computation, such as finite-state automata and Turing machines \cite[eg,][]{Kleene1956,Minsky1954,Siegelmann1992OnTC,hao-etal-2018-context,merrill-2019-sequential,merrill-etal-2020-formal,merrill-etal-2022-saturated,merrill2022extracting}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="1906.01615" id="5729">
        <attvalues>
          <attvalue for="5" value=" Traditionally, the representational capacity of neural networks, both in terms of lower bounds (what they can provably do) as well as upper bounds (what they can provably not do), has been studied in terms of Boolean sequential models of computation, such as finite-state automata and Turing machines \cite[eg,][]{Kleene1956,Minsky1954,Siegelmann1992OnTC,hao-etal-2018-context,merrill-2019-sequential,merrill-etal-2020-formal,merrill-etal-2022-saturated,merrill2022extracting}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2004.08500" id="5730">
        <attvalues>
          <attvalue for="5" value=" Traditionally, the representational capacity of neural networks, both in terms of lower bounds (what they can provably do) as well as upper bounds (what they can provably not do), has been studied in terms of Boolean sequential models of computation, such as finite-state automata and Turing machines \cite[eg,][]{Kleene1956,Minsky1954,Siegelmann1992OnTC,hao-etal-2018-context,merrill-2019-sequential,merrill-etal-2020-formal,merrill-etal-2022-saturated,merrill2022extracting}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2106.16213" id="5731">
        <attvalues>
          <attvalue for="5" value=" Traditionally, the representational capacity of neural networks, both in terms of lower bounds (what they can provably do) as well as upper bounds (what they can provably not do), has been studied in terms of Boolean sequential models of computation, such as finite-state automata and Turing machines \cite[eg,][]{Kleene1956,Minsky1954,Siegelmann1992OnTC,hao-etal-2018-context,merrill-2019-sequential,merrill-etal-2020-formal,merrill-etal-2022-saturated,merrill2022extracting}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2201.12451" id="5732">
        <attvalues>
          <attvalue for="5" value=" Traditionally, the representational capacity of neural networks, both in terms of lower bounds (what they can provably do) as well as upper bounds (what they can provably not do), has been studied in terms of Boolean sequential models of computation, such as finite-state automata and Turing machines \cite[eg,][]{Kleene1956,Minsky1954,Siegelmann1992OnTC,hao-etal-2018-context,merrill-2019-sequential,merrill-etal-2020-formal,merrill-etal-2022-saturated,merrill2022extracting}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2310.05161" id="5733">
        <attvalues>
          <attvalue for="5" value=" Recent work has extended this paradigm to work with probabilistic models of computation \cite{svete-cotterell-2023-recurrent,nowak-etal-2023-representational}, but so far only for LMs based on recurrent neural networks." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2310.12942" id="5734">
        <attvalues>
          <attvalue for="5" value=" Recent work has extended this paradigm to work with probabilistic models of computation \cite{svete-cotterell-2023-recurrent,nowak-etal-2023-representational}, but so far only for LMs based on recurrent neural networks." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="1906.06755" id="5735">
        <attvalues>
          <attvalue for="5" value="\looseness=-1 However, the sequential nature of classical models makes the connection to the inherently parallelizable transformer architecture less straightforward and has resulted in a number of results upper-bounding their representational capacity \cite{hahn-2020-theoretical,bhattamishra-etal-2020-on-ability,chiang-cholak-2022-overcoming,hao-etal-2022-formal,merrill-sabharwal-2023-parallelism}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2009.11264" id="5736">
        <attvalues>
          <attvalue for="5" value="\looseness=-1 However, the sequential nature of classical models makes the connection to the inherently parallelizable transformer architecture less straightforward and has resulted in a number of results upper-bounding their representational capacity \cite{hahn-2020-theoretical,bhattamishra-etal-2020-on-ability,chiang-cholak-2022-overcoming,hao-etal-2022-formal,merrill-sabharwal-2023-parallelism}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2202.12172" id="5737">
        <attvalues>
          <attvalue for="5" value="\looseness=-1 However, the sequential nature of classical models makes the connection to the inherently parallelizable transformer architecture less straightforward and has resulted in a number of results upper-bounding their representational capacity \cite{hahn-2020-theoretical,bhattamishra-etal-2020-on-ability,chiang-cholak-2022-overcoming,hao-etal-2022-formal,merrill-sabharwal-2023-parallelism}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2204.06618" id="5738">
        <attvalues>
          <attvalue for="5" value="\looseness=-1 However, the sequential nature of classical models makes the connection to the inherently parallelizable transformer architecture less straightforward and has resulted in a number of results upper-bounding their representational capacity \cite{hahn-2020-theoretical,bhattamishra-etal-2020-on-ability,chiang-cholak-2022-overcoming,hao-etal-2022-formal,merrill-sabharwal-2023-parallelism}." />
        </attvalues>
      </edge>
      <edge source="2404.14994" target="2207.00729" id="5739">
        <attvalues>
          <attvalue for="5" value="\looseness=-1 However, the sequential nature of classical models makes the connection to the inherently parallelizable transformer architecture less straightforward and has resulted in a number of results upper-bounding their representational capacity \cite{hahn-2020-theoretical,bhattamishra-etal-2020-on-ability,chiang-cholak-2022-overcoming,hao-etal-2022-formal,merrill-sabharwal-2023-parallelism}." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="2301.02111" id="5740">
        <attvalues>
          <attvalue for="5" value=" Recently, large language models have demonstrated remarkable performance on zero-shot text-to-speech (TTS) tasks such as VALL-E \cite{wang2023neural}, SPEAR-TTS \cite{kharitonov2023speak}, and SoundStorm \cite{borsos2023soundstorm}." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="2302.03540" id="5741">
        <attvalues>
          <attvalue for="5" value=" Recently, large language models have demonstrated remarkable performance on zero-shot text-to-speech (TTS) tasks such as VALL-E \cite{wang2023neural}, SPEAR-TTS \cite{kharitonov2023speak}, and SoundStorm \cite{borsos2023soundstorm}." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="2305.09636" id="5742">
        <attvalues>
          <attvalue for="5" value=" Recently, large language models have demonstrated remarkable performance on zero-shot text-to-speech (TTS) tasks such as VALL-E \cite{wang2023neural}, SPEAR-TTS \cite{kharitonov2023speak}, and SoundStorm \cite{borsos2023soundstorm}." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="2210.13438" id="5743">
        <attvalues>
          <attvalue for="5" value=" Following that, Encodec \cite{defossez2022high} introduced a spectrogram-only adversarial loss, a novel gradient balancer, and a small Transformer model to further improve the performance of codec." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="1712.01120" id="5745">
        <attvalues>
          <attvalue for="5" value=" \cite{kleijn2018wavenet} proposed a low-rate speech coding architecture based on the WaveNet \cite{vanwavenet} decoder." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="1609.03499" id="5746">
        <attvalues>
          <attvalue for="5" value=" \cite{kleijn2018wavenet} proposed a low-rate speech coding architecture based on the WaveNet \cite{vanwavenet} decoder." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="2102.09660" id="5747">
        <attvalues>
          <attvalue for="5" value=" Lyra \cite{kleijn2021generative} encodes quantized mel-spectrogram features of speech, and then decodes them with WaveGRU \cite{chung2014empirical}." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="1412.3555" id="5748">
        <attvalues>
          <attvalue for="5" value=" Lyra \cite{kleijn2021generative} encodes quantized mel-spectrogram features of speech, and then decodes them with WaveGRU \cite{chung2014empirical}." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="1910.06464" id="5749">
        <attvalues>
          <attvalue for="5" value=" \cite{garbacea2019low} used the discretized latent representations proposed in VQVAE \cite{van2017neural} as conditioning for the WaveNet decoder." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="1711.00937" id="5750">
        <attvalues>
          <attvalue for="5" value=" \cite{garbacea2019low} used the discretized latent representations proposed in VQVAE \cite{van2017neural} as conditioning for the WaveNet decoder." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="2107.03312" id="5751">
        <attvalues>
          <attvalue for="5" value=" After that, SoundStream \cite{zeghidour2021soundstream}, as a fully convolutional end-to-end universal audio codec model, was proposed, extending the VQVAE vector quantizer to a residual vector quantizer." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="2305.02765" id="5752">
        <attvalues>
          <attvalue for="5" value=" HifiCodec \cite{yang2023hifi} proposes a codec model that uses group-residual vector quantization to improve the reconstruction performance of audio." />
        </attvalues>
      </edge>
      <edge source="2310.00014" target="1904.02882" id="5753">
        <attvalues>
          <attvalue for="5" value=" \end{itemize} Experimental results on speech reconstruction and zero-shot TTS task with LibriTTS datasets \cite{zen2019libritts} show that TiCodec achieved better speech reconstruction performance with fewer tokens and improved robustness, quality, and similarity of synthesized speech in the zero-shot TTS task." />
        </attvalues>
      </edge>
      <edge source="2408.05948" target="2003.02320" id="5764">
        <attvalues>
          <attvalue for="5" value=" Knowledge Graphs (KGs) have long been recognized as powerful tools for capturing structured representations of the world~\cite{hogan-etal-2021-knowledge}." />
        </attvalues>
      </edge>
      <edge source="2408.05948" target="2210.00105" id="5765">
        <attvalues>
          <attvalue for="5" value=" This structured representation has had an impact across various domains, including Natural Language Processing~\cite{schneider-etal-2022-decade}, Recommender Systems~\cite{guo-etal-2022-recommender}, and Information Retrieval~\cite{reinanda-etal-2020-knowledge}." />
        </attvalues>
      </edge>
      <edge source="2408.05948" target="1909.01066" id="5767">
        <attvalues>
          <attvalue for="5" value=" The integration of LLMs and KGs has opened up new opportunities in natural language processing~\cite{petroni-etal-2019-language,guu-etal-2020-realm,peng-etal-2023-check}, which has led to significant advancements across various tasks~\cite{barba-etal-2021-esc,chakrabarti-etal-2022-joint,de-cao-etal-2022-multilingual,xu-etal-2023-kilm}." />
        </attvalues>
      </edge>
      <edge source="2408.05948" target="2302.12813" id="5768">
        <attvalues>
          <attvalue for="5" value=" The integration of LLMs and KGs has opened up new opportunities in natural language processing~\cite{petroni-etal-2019-language,guu-etal-2020-realm,peng-etal-2023-check}, which has led to significant advancements across various tasks~\cite{barba-etal-2021-esc,chakrabarti-etal-2022-joint,de-cao-etal-2022-multilingual,xu-etal-2023-kilm}." />
        </attvalues>
      </edge>
      <edge source="2408.05948" target="2103.12528" id="5769">
        <attvalues>
          <attvalue for="5" value=" The integration of LLMs and KGs has opened up new opportunities in natural language processing~\cite{petroni-etal-2019-language,guu-etal-2020-realm,peng-etal-2023-check}, which has led to significant advancements across various tasks~\cite{barba-etal-2021-esc,chakrabarti-etal-2022-joint,de-cao-etal-2022-multilingual,xu-etal-2023-kilm}." />
        </attvalues>
      </edge>
      <edge source="2408.05948" target="2306.12235" id="5771">
        <attvalues>
          <attvalue for="5" value=" These datasets have played a crucial role in enabling new retrieval-augmented systems, demonstrating the potential of LLM-KG integrations to provide accurate and attributable responses in conversational settings~\cite{christmann2023compmix}." />
        </attvalues>
      </edge>
      <edge source="2408.05948" target="2309.15088" id="5772">
        <attvalues>
          <attvalue for="5" value=" This process has been utilized at scale for neural query synthesis~\cite{doct5query, unicoil, NQS} and LLM-based ranked list reorderings for instruction distillation into open-source rerankers~\cite{rankvicuna, rankzephyr, lit5} resulting in substantial improvements across a spectrum of retrieval tasks." />
        </attvalues>
      </edge>
      <edge source="2311.12338" target="1706.03762" id="5774">
        <attvalues>
          <attvalue for="5" value=" The transformer architecture\cite{2017attention}, was introduced in 2017, has become a foundation in LLMs." />
        </attvalues>
      </edge>
      <edge source="2311.12338" target="2302.03735" id="5776">
        <attvalues>
          <attvalue for="5" value="\cite{surveyllm4rec} This survey focuses on utilizing LLMs for Personalized Explanation Generating task." />
        </attvalues>
      </edge>
      <edge source="2311.12338" target="2107.13586" id="5777">
        <attvalues>
          <attvalue for="5" value=" As a result, a recently proposed paradigm, prompt learning\cite{liu2023pre}, further unifies the use of PLMs on different tasks in a simple yet flexible manner." />
        </attvalues>
      </edge>
      <edge source="2210.13270" target="1702.08608" id="5778">
        <attvalues>
          <attvalue for="5" value=" The opaqueness of deep natural language processing (NLP) models has grown in tandem with their power \cite{doshi2017towards}, which has motivated efforts to interpret how these black-box models work \cite{sundararajan2017axiomatic, belinkov2019analysis}." />
        </attvalues>
      </edge>
      <edge source="2210.13270" target="1703.01365" id="5779">
        <attvalues>
          <attvalue for="5" value=" The opaqueness of deep natural language processing (NLP) models has grown in tandem with their power \cite{doshi2017towards}, which has motivated efforts to interpret how these black-box models work \cite{sundararajan2017axiomatic, belinkov2019analysis}." />
        </attvalues>
      </edge>
      <edge source="2210.13270" target="2004.03685" id="5780">
        <attvalues>
          <attvalue for="5" value=" Post-hoc explanation aims to explain a trained model and reveal how the model arrives at a decision \cite{jacovi2020towards, molnar2020interpretable}." />
        </attvalues>
      </edge>
      <edge source="2210.13270" target="1806.05337" id="5781">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{singh2018hierarchical} use CD score \cite{murdoch2018beyond} as a joining metric in the agglomerative clustering procedure; \cite{chen2020generating} recursively divides large text spans into smaller ones by detecting feature interaction." />
        </attvalues>
      </edge>
      <edge source="2210.13270" target="1801.05453" id="5782">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{singh2018hierarchical} use CD score \cite{murdoch2018beyond} as a joining metric in the agglomerative clustering procedure; \cite{chen2020generating} recursively divides large text spans into smaller ones by detecting feature interaction." />
        </attvalues>
      </edge>
      <edge source="2210.13270" target="1706.03762" id="5785">
        <attvalues>
          <attvalue for="5" value=" The concerns are summarized as follows: First, modern NLP models such as BERT \cite{devlin2018bert} and GPT \cite{radford2018improving, radford2019language} are almost all transformer-based, using self-attention mechanisms \cite{vaswani2017attention} to build word relations." />
        </attvalues>
      </edge>
      <edge source="2010.11506" target="1907.11692" id="5787">
        <attvalues>
          <attvalue for="5" value=" The transfer learning framework consists of two stages, where we first pre-train a large-scale language model, (\eg, BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta}, ALBERT \cite{Lan2020ALBERT} and T5 \cite{raffel2019exploring}) on a large text corpus and then fine-tune it on downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2010.11506" target="1909.11942" id="5788">
        <attvalues>
          <attvalue for="5" value=" The transfer learning framework consists of two stages, where we first pre-train a large-scale language model, (\eg, BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta}, ALBERT \cite{Lan2020ALBERT} and T5 \cite{raffel2019exploring}) on a large text corpus and then fine-tune it on downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2010.11506" target="1703.00056" id="5792">
        <attvalues>
          <attvalue for="5" value=" Providing such calibrated output probabilities can help us to achieve better model robustness \cite{lee2018simple}, model fairness \cite{chouldechova2017fair} and improve label efficiency via uncertainty driven learning \cite{gal2017active,siddhant2018deep, shen2018deep}." />
        </attvalues>
      </edge>
      <edge source="2010.11506" target="1703.02910" id="5793">
        <attvalues>
          <attvalue for="5" value=" Providing such calibrated output probabilities can help us to achieve better model robustness \cite{lee2018simple}, model fairness \cite{chouldechova2017fair} and improve label efficiency via uncertainty driven learning \cite{gal2017active,siddhant2018deep, shen2018deep}." />
        </attvalues>
      </edge>
      <edge source="2010.11506" target="1808.05697" id="5794">
        <attvalues>
          <attvalue for="5" value=" Providing such calibrated output probabilities can help us to achieve better model robustness \cite{lee2018simple}, model fairness \cite{chouldechova2017fair} and improve label efficiency via uncertainty driven learning \cite{gal2017active,siddhant2018deep, shen2018deep}." />
        </attvalues>
      </edge>
      <edge source="2010.11506" target="1706.04599" id="5795">
        <attvalues>
          <attvalue for="5" value=" To fight against miscalibration, a natural option is to apply a calibration method such as temperature scaling \cite{guo2017calibration} in a post-processing step." />
        </attvalues>
      </edge>
      <edge source="2010.11506" target="1408.5882" id="5796">
        <attvalues>
          <attvalue for="5" value=" Specifically, we train a TextCNN \cite{kim2014convolutional} and a BERT-MLP using 20NG\textsubscript{15} (the first 15 categories of 20NG) and then evaluate them on both in-distribution and OOD data." />
        </attvalues>
      </edge>
      <edge source="2010.11506" target="1701.06548" id="5797">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{Pereyra2017erl} propose an entropy regularizer to prevent over-confidence, but it can needlessly hurt legitimate high confident predictions." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1506.05869" id="5798">
        <attvalues>
          <attvalue for="5" value=" While techniques of open domain dialogue generation \cite{vinyals2015neural,xing2017hierarchical,zhang2019recosa} have been applied in industrial products \cite{shum2018eliza,ram2018conversational}, people can still feel the gap between the dialogue systems and humans, especially when they dive into a specific topic of interest." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1907.05339" id="5800">
        <attvalues>
          <attvalue for="5" value=" While techniques of open domain dialogue generation \cite{vinyals2015neural,xing2017hierarchical,zhang2019recosa} have been applied in industrial products \cite{shum2018eliza,ram2018conversational}, people can still feel the gap between the dialogue systems and humans, especially when they dive into a specific topic of interest." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1801.01957" id="5801">
        <attvalues>
          <attvalue for="5" value=" While techniques of open domain dialogue generation \cite{vinyals2015neural,xing2017hierarchical,zhang2019recosa} have been applied in industrial products \cite{shum2018eliza,ram2018conversational}, people can still feel the gap between the dialogue systems and humans, especially when they dive into a specific topic of interest." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1801.03604" id="5802">
        <attvalues>
          <attvalue for="5" value=" While techniques of open domain dialogue generation \cite{vinyals2015neural,xing2017hierarchical,zhang2019recosa} have been applied in industrial products \cite{shum2018eliza,ram2018conversational}, people can still feel the gap between the dialogue systems and humans, especially when they dive into a specific topic of interest." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1910.00610" id="5803">
        <attvalues>
          <attvalue for="5" value=" To bridge the gap, researchers consider grounding open domain dialogues by external knowledge which could be retrieved either from structured knowledge bases \cite{zhou2018commonsense,moon2019opendialkg,tuan2019dykgchat} or from unstructured documents \cite{dinan2018wizard}." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1811.01241" id="5804">
        <attvalues>
          <attvalue for="5" value=" The test beds are benchmarks of knowledge-grounded dialogue generation, including Wizard of Wikipedia (Wizard) \cite{dinan2018wizard}, CMU Document Grounded Conversations (CMU$\_$DoG) \cite{zhou2018dataset}, and Topical-Chat (TC) \cite{gopalakrishnan2019topical}, in which we discard the external knowledge passages." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1909.01066" id="5805">
        <attvalues>
          <attvalue for="5" value=" The work is motivated by two lines of research emerging recently: (1) interestingly, some recent studies indicate that pre-trained language models have packed enough knowledge in their parameters, and thus they can do a good job in question-answering tasks without the need of access to external knowledge \cite{petroni2019language,roberts2020much}." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="2002.08910" id="5806">
        <attvalues>
          <attvalue for="5" value=" The work is motivated by two lines of research emerging recently: (1) interestingly, some recent studies indicate that pre-trained language models have packed enough knowledge in their parameters, and thus they can do a good job in question-answering tasks without the need of access to external knowledge \cite{petroni2019language,roberts2020much}." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1911.00536" id="5807">
        <attvalues>
          <attvalue for="5" value=" We choose DialoGPT \cite{zhang2019dialogpt}, GPT-2$_{finetune}$, and DialoGPT$_{finetune}$ as the pre-trained language models for investigation, where GPT-2$_{finetune}$ and DialoGPT$_{finetune}$ refer to the OpenAI GPT-2 model \cite{radford2019language} and DialoGPT fine-tuned on the training data of the benchmarks respectively." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1901.08149" id="5808">
        <attvalues>
          <attvalue for="5" value=" If the answer is ``yes'', then we can get rid of the dependency on external knowledge sources, and obtain a simpler and more flexible architecture with a better generalization ability inherited from pre-training with massive text corpus \cite{radford2019language}; and (2) pre-training techniques have exhibited compelling performance on the task of open domain dialogue generation \cite{zhang2019dialogpt,wolf2019transfertransfo}." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="1809.07358" id="5809">
        <attvalues>
          <attvalue for="5" value=" The test beds are benchmarks of knowledge-grounded dialogue generation, including Wizard of Wikipedia (Wizard) \cite{dinan2018wizard}, CMU Document Grounded Conversations (CMU$\_$DoG) \cite{zhou2018dataset}, and Topical-Chat (TC) \cite{gopalakrishnan2019topical}, in which we discard the external knowledge passages." />
        </attvalues>
      </edge>
      <edge source="2011.09708" target="2308.11995" id="5810">
        <attvalues>
          <attvalue for="5" value=" The test beds are benchmarks of knowledge-grounded dialogue generation, including Wizard of Wikipedia (Wizard) \cite{dinan2018wizard}, CMU Document Grounded Conversations (CMU$\_$DoG) \cite{zhou2018dataset}, and Topical-Chat (TC) \cite{gopalakrishnan2019topical}, in which we discard the external knowledge passages." />
        </attvalues>
      </edge>
      <edge source="2304.06875" target="1706.03762" id="5811">
        <attvalues>
          <attvalue for="5" value=" Modern LLMs are based on the Transformer architecture \cite{DBLP:conf/nips/VaswaniSPUJGKP17}, and can be trained with unsupervised objectives including causal language modeling \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}, masked language modeling, among others \cite{DBLP:conf/icml/WangRHSCBLR22}." />
        </attvalues>
      </edge>
      <edge source="2304.06875" target="2005.14165" id="5812">
        <attvalues>
          <attvalue for="5" value=" Since researches on scaling laws \cite{DBLP:journals/corr/abs-2001-08361, DBLP:journals/corr/abs-2203-15556} reveal the potential of improving model performance by increasing the total computation, the community have been scaling up both the model sizes and training data \cite{DBLP:conf/nips/BrownMRSKDNSSAA20, DBLP:journals/corr/abs-2201-11990, DBLP:journals/corr/abs-2307-09288, falcon40b}, as briefly summarized in Table \ref{tab: current llm}." />
        </attvalues>
      </edge>
      <edge source="2304.06875" target="2001.08361" id="5813">
        <attvalues>
          <attvalue for="5" value=" In this paper, we start by proposing a method that yields accurate loss prediction, namely $\mu$Scaling ~(a compound word of $\mu$P \cite{DBLP:journals/corr/abs-2203-03466} and Scaling Laws \cite{DBLP:journals/corr/abs-2001-08361}), with experimental results supporting its correctness." />
        </attvalues>
      </edge>
      <edge source="2304.06875" target="2307.09288" id="5814">
        <attvalues>
          <attvalue for="5" value=" For instance, as reported by Llama-2 \cite{DBLP:journals/corr/abs-2307-09288}, time to train the 7B, 13B, and 70B models on roughly 2 trillion tokens is 184k, 368k, and 1." />
        </attvalues>
      </edge>
      <edge source="2304.06875" target="2303.08774" id="5815">
        <attvalues>
          <attvalue for="5" value=" As a possible solution to this issue, the technical report of GPT-4 \cite{gpt4} showed that some behaviors of large models can be predicted before the training starts (with unpublished methods)." />
        </attvalues>
      </edge>
      <edge source="2304.06875" target="2203.03466" id="5816">
        <attvalues>
          <attvalue for="5" value=" In this paper, we start by proposing a method that yields accurate loss prediction, namely $\mu$Scaling ~(a compound word of $\mu$P \cite{DBLP:journals/corr/abs-2203-03466} and Scaling Laws \cite{DBLP:journals/corr/abs-2001-08361}), with experimental results supporting its correctness." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2005.11129" id="5817">
        <attvalues>
          <attvalue for="5" value=" With the development of deep neural networks, text-to-speech (TTS) technology has made significant progress~\cite{glowtts,gradtts,paralleltacotron,speartts}." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2105.06337" id="5818">
        <attvalues>
          <attvalue for="5" value=" With the development of deep neural networks, text-to-speech (TTS) technology has made significant progress~\cite{glowtts,gradtts,paralleltacotron,speartts}." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2010.11439" id="5819">
        <attvalues>
          <attvalue for="5" value=" With the development of deep neural networks, text-to-speech (TTS) technology has made significant progress~\cite{glowtts,gradtts,paralleltacotron,speartts}." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2302.03540" id="5820">
        <attvalues>
          <attvalue for="5" value=" With the development of deep neural networks, text-to-speech (TTS) technology has made significant progress~\cite{glowtts,gradtts,paralleltacotron,speartts}." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2104.05557" id="5821">
        <attvalues>
          <attvalue for="5" value=" Early studies use continuous audio signals as input, relying on an explicit speaker encoder to embed a speaker's timbre, prosody, and speaking style~\cite{scglowtts,yourtts}." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2112.02418" id="5822">
        <attvalues>
          <attvalue for="5" value=" Early studies use continuous audio signals as input, relying on an explicit speaker encoder to embed a speaker's timbre, prosody, and speaking style~\cite{scglowtts,yourtts}." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2205.07211" id="5823">
        <attvalues>
          <attvalue for="5" value=" Some studies further rely on specifically designed speech disentanglement approaches to extract speaker-agnostic information~\cite{kumar2022zero,generspeech}." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2304.09116" id="5824">
        <attvalues>
          <attvalue for="5" value=" The former~\cite{NS2,voicebox,fastdiff} leverages the diffusion model~\cite{ho2020ddpm} and its variants~\cite{flowmatching,yang2022diffusion} to estimate the target speech that shares the same distribution as the enrollment, while the latter~\cite{audiolm} usually employs language models on a pre-trained neural codec to extract discrete audio representations and reconstruct high-quality waveforms." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2306.15687" id="5825">
        <attvalues>
          <attvalue for="5" value=" The former~\cite{NS2,voicebox,fastdiff} leverages the diffusion model~\cite{ho2020ddpm} and its variants~\cite{flowmatching,yang2022diffusion} to estimate the target speech that shares the same distribution as the enrollment, while the latter~\cite{audiolm} usually employs language models on a pre-trained neural codec to extract discrete audio representations and reconstruct high-quality waveforms." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2006.11239" id="5827">
        <attvalues>
          <attvalue for="5" value=" The former~\cite{NS2,voicebox,fastdiff} leverages the diffusion model~\cite{ho2020ddpm} and its variants~\cite{flowmatching,yang2022diffusion} to estimate the target speech that shares the same distribution as the enrollment, while the latter~\cite{audiolm} usually employs language models on a pre-trained neural codec to extract discrete audio representations and reconstruct high-quality waveforms." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2210.02747" id="5828">
        <attvalues>
          <attvalue for="5" value=" The former~\cite{NS2,voicebox,fastdiff} leverages the diffusion model~\cite{ho2020ddpm} and its variants~\cite{flowmatching,yang2022diffusion} to estimate the target speech that shares the same distribution as the enrollment, while the latter~\cite{audiolm} usually employs language models on a pre-trained neural codec to extract discrete audio representations and reconstruct high-quality waveforms." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2209.00796" id="5829">
        <attvalues>
          <attvalue for="5" value=" The former~\cite{NS2,voicebox,fastdiff} leverages the diffusion model~\cite{ho2020ddpm} and its variants~\cite{flowmatching,yang2022diffusion} to estimate the target speech that shares the same distribution as the enrollment, while the latter~\cite{audiolm} usually employs language models on a pre-trained neural codec to extract discrete audio representations and reconstruct high-quality waveforms." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2209.03143" id="5830">
        <attvalues>
          <attvalue for="5" value=" The former~\cite{NS2,voicebox,fastdiff} leverages the diffusion model~\cite{ho2020ddpm} and its variants~\cite{flowmatching,yang2022diffusion} to estimate the target speech that shares the same distribution as the enrollment, while the latter~\cite{audiolm} usually employs language models on a pre-trained neural codec to extract discrete audio representations and reconstruct high-quality waveforms." />
        </attvalues>
      </edge>
      <edge source="2406.15752" target="2301.02111" id="5831">
        <attvalues>
          <attvalue for="5" value=" In the domain of neural codec language models, VALL-E~\cite{valle} is a prototypical and highly effective two-stage approach." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2005.14165" id="5893">
        <attvalues>
          <attvalue for="5" value=" Additionally, larger models were believed to be unscalable to reach the desired performance given the costs \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2110-14168}, However, recent explorations reveal the emergent abilities of larger models like GPT3-175B and PaLM 540B \cite{DBLP:journals/corr/abs-2204-02311}: prompting shows low or even close-to-random performance on multiple tasks until a certain scale of the model where a breakthrough emerges \cite{DBLP:journals/corr/abs-2206-04615,DBLP:journals/corr/abs-2206-07682}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2107.03374" id="5894">
        <attvalues>
          <attvalue for="5" value="5 through pretraining on code as in Codex \cite{DBLP:journals/corr/abs-2107-03374} which powers GitHub Copilot, as well as through instruction fine-tuning that aligns the model's responses given instructions with human expectations using reinforcement learning, known as InstructGPT \cite{DBLP:journals/corr/abs-2203-02155}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2302.04023" id="5896">
        <attvalues>
          <attvalue for="5" value=" When further combined with fine-tuning on dialogues in a similar way, the resulting model, ChatGPT, has gained great popularity since its release in late 2022, displaying highly human-like language understanding and generation capabilities \cite{kung2023performance,DBLP:journals/corr/abs-2302-04023,DBLP:journals/corr/abs-2302-06476}, and has become the core of a number of AI-powered applications." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2302.06476" id="5897">
        <attvalues>
          <attvalue for="5" value=" When further combined with fine-tuning on dialogues in a similar way, the resulting model, ChatGPT, has gained great popularity since its release in late 2022, displaying highly human-like language understanding and generation capabilities \cite{kung2023performance,DBLP:journals/corr/abs-2302-04023,DBLP:journals/corr/abs-2302-06476}, and has become the core of a number of AI-powered applications." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2302.04761" id="5898">
        <attvalues>
          <attvalue for="5" value=" Combined with the ability to utilize tools with external APIs as in Toolformer \cite{DBLP:journals/corr/abs-2302-04761} as well as conducting web search as in WebGPT \cite{DBLP:journals/corr/abs-2112-09332} and New Bing, a competent and versatile AI assistant has taken shape." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2103.03095" id="5900">
        <attvalues>
          <attvalue for="5" value=" Traditionally, SLU is carried out using a cascaded pipeline, which includes an automatic speech recognition (ASR) module taking audio as inputs, and a natural language understanding (NLU) module working on ASR transcripts, hypotheses, or lattice to predict labels for tasks like intent classification (IC) and slot filling (SF) \cite{DBLP:conf/asru/Mori07,DBLP:conf/ijcai/QinXC021}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="1802.08395" id="5901">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end approaches that directly predict labels from speech \cite{DBLP:conf/icassp/SerdyukWFKLB18,DBLP:conf/slt/HaghaniNBCGMPQW18,DBLP:conf/interspeech/SaxonCMM21} become more popular, and pretrained language and speech models are also introduced into SLU \cite{DBLP:journals/corr/abs-2111-02735,DBLP:conf/icassp/AroraDDCUPZKGYV22,DBLP:conf/icassp/SeoKL22}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="1809.09190" id="5902">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end approaches that directly predict labels from speech \cite{DBLP:conf/icassp/SerdyukWFKLB18,DBLP:conf/slt/HaghaniNBCGMPQW18,DBLP:conf/interspeech/SaxonCMM21} become more popular, and pretrained language and speech models are also introduced into SLU \cite{DBLP:journals/corr/abs-2111-02735,DBLP:conf/icassp/AroraDDCUPZKGYV22,DBLP:conf/icassp/SeoKL22}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2106.09009" id="5903">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end approaches that directly predict labels from speech \cite{DBLP:conf/icassp/SerdyukWFKLB18,DBLP:conf/slt/HaghaniNBCGMPQW18,DBLP:conf/interspeech/SaxonCMM21} become more popular, and pretrained language and speech models are also introduced into SLU \cite{DBLP:journals/corr/abs-2111-02735,DBLP:conf/icassp/AroraDDCUPZKGYV22,DBLP:conf/icassp/SeoKL22}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2111.14706" id="5905">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end approaches that directly predict labels from speech \cite{DBLP:conf/icassp/SerdyukWFKLB18,DBLP:conf/slt/HaghaniNBCGMPQW18,DBLP:conf/interspeech/SaxonCMM21} become more popular, and pretrained language and speech models are also introduced into SLU \cite{DBLP:journals/corr/abs-2111-02735,DBLP:conf/icassp/AroraDDCUPZKGYV22,DBLP:conf/icassp/SeoKL22}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2104.07253" id="5906">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end approaches that directly predict labels from speech \cite{DBLP:conf/icassp/SerdyukWFKLB18,DBLP:conf/slt/HaghaniNBCGMPQW18,DBLP:conf/interspeech/SaxonCMM21} become more popular, and pretrained language and speech models are also introduced into SLU \cite{DBLP:journals/corr/abs-2111-02735,DBLP:conf/icassp/AroraDDCUPZKGYV22,DBLP:conf/icassp/SeoKL22}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2006.05702" id="5907">
        <attvalues>
          <attvalue for="5" value=" Additionally, there are works focused on low-resource or few-shot textual IC/SF \cite{DBLP:conf/emnlp/YazdaniH15,DBLP:conf/interspeech/FerreiraJL15,DBLP:conf/acl/HouCLZLLL20,DBLP:conf/emnlp/WuSJ21,DBLP:conf/interspeech/PengZZG21}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2004.13952" id="5908">
        <attvalues>
          <attvalue for="5" value=" Additionally, there are works focused on low-resource or few-shot textual IC/SF \cite{DBLP:conf/emnlp/YazdaniH15,DBLP:conf/interspeech/FerreiraJL15,DBLP:conf/acl/HouCLZLLL20,DBLP:conf/emnlp/WuSJ21,DBLP:conf/interspeech/PengZZG21}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2107.13586" id="5909">
        <attvalues>
          <attvalue for="5" value=" In contrast, considering the difficulty of fine-tuning the whole GPT3 model, recent NLP research highlights a different scheme, namely prompting \cite{DBLP:journals/csur/LiuYFJHN23}: given a fixed textual description of the task known as a prompt without any training data, the language model may correctly carry out the task." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2104.08691" id="5910">
        <attvalues>
          <attvalue for="5" value=" It appears to be clumsy and may perform worse than fine-tuned smaller models like T5-11B at the beginning \cite{DBLP:conf/emnlp/LesterAC21}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2001.08361" id="5911">
        <attvalues>
          <attvalue for="5" value=" Additionally, larger models were believed to be unscalable to reach the desired performance given the costs \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2110-14168}, However, recent explorations reveal the emergent abilities of larger models like GPT3-175B and PaLM 540B \cite{DBLP:journals/corr/abs-2204-02311}: prompting shows low or even close-to-random performance on multiple tasks until a certain scale of the model where a breakthrough emerges \cite{DBLP:journals/corr/abs-2206-04615,DBLP:journals/corr/abs-2206-07682}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2110.14168" id="5912">
        <attvalues>
          <attvalue for="5" value=" Additionally, larger models were believed to be unscalable to reach the desired performance given the costs \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2110-14168}, However, recent explorations reveal the emergent abilities of larger models like GPT3-175B and PaLM 540B \cite{DBLP:journals/corr/abs-2204-02311}: prompting shows low or even close-to-random performance on multiple tasks until a certain scale of the model where a breakthrough emerges \cite{DBLP:journals/corr/abs-2206-04615,DBLP:journals/corr/abs-2206-07682}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2204.02311" id="5913">
        <attvalues>
          <attvalue for="5" value=" Additionally, larger models were believed to be unscalable to reach the desired performance given the costs \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2110-14168}, However, recent explorations reveal the emergent abilities of larger models like GPT3-175B and PaLM 540B \cite{DBLP:journals/corr/abs-2204-02311}: prompting shows low or even close-to-random performance on multiple tasks until a certain scale of the model where a breakthrough emerges \cite{DBLP:journals/corr/abs-2206-04615,DBLP:journals/corr/abs-2206-07682}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2206.04615" id="5914">
        <attvalues>
          <attvalue for="5" value=" Additionally, larger models were believed to be unscalable to reach the desired performance given the costs \cite{DBLP:conf/nips/BrownMRSKDNSSAA20,DBLP:journals/corr/abs-2001-08361,DBLP:journals/corr/abs-2110-14168}, However, recent explorations reveal the emergent abilities of larger models like GPT3-175B and PaLM 540B \cite{DBLP:journals/corr/abs-2204-02311}: prompting shows low or even close-to-random performance on multiple tasks until a certain scale of the model where a breakthrough emerges \cite{DBLP:journals/corr/abs-2206-04615,DBLP:journals/corr/abs-2206-07682}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2201.11903" id="5916">
        <attvalues>
          <attvalue for="5" value=" This breakthrough enables chain-of-thought prompting to surpass the smaller models fine-tuned on rich data \cite{weichain,DBLP:journals/corr/abs-2210-00720}, allows reasoning using internal knowledge with results comparable to external knowledge retrievers \cite{DBLP:journals/corr/abs-2209-10063}, and leads to better robustness and generalization \cite{DBLP:journals/corr/abs-2210-00720,DBLP:journals/corr/abs-2210-09150}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2210.00720" id="5917">
        <attvalues>
          <attvalue for="5" value=" This breakthrough enables chain-of-thought prompting to surpass the smaller models fine-tuned on rich data \cite{weichain,DBLP:journals/corr/abs-2210-00720}, allows reasoning using internal knowledge with results comparable to external knowledge retrievers \cite{DBLP:journals/corr/abs-2209-10063}, and leads to better robustness and generalization \cite{DBLP:journals/corr/abs-2210-00720,DBLP:journals/corr/abs-2210-09150}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2210.03337" id="5920">
        <attvalues>
          <attvalue for="5" value=" There have been several works on SLU employing prompts, such as fine-tuning pretrained models like T5 aided by prompts \cite{DBLP:conf/coling/WuWZCZ22,DBLP:journals/corr/abs-2210-03337}, fine-tuning embeddings prepended to the inputs known as continuous prompts \cite{DBLP:conf/interspeech/ChangT0L22,DBLP:journals/corr/abs-2303-00733}, and end-to-end SLU by in-context learning on GPT2 with a fine-tuned audio encoder \cite{DBLP:conf/interspeech/GaoNQZCH22}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2303.00733" id="5921">
        <attvalues>
          <attvalue for="5" value=" There have been several works on SLU employing prompts, such as fine-tuning pretrained models like T5 aided by prompts \cite{DBLP:conf/coling/WuWZCZ22,DBLP:journals/corr/abs-2210-03337}, fine-tuning embeddings prepended to the inputs known as continuous prompts \cite{DBLP:conf/interspeech/ChangT0L22,DBLP:journals/corr/abs-2303-00733}, and end-to-end SLU by in-context learning on GPT2 with a fine-tuned audio encoder \cite{DBLP:conf/interspeech/GaoNQZCH22}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2203.15863" id="5922">
        <attvalues>
          <attvalue for="5" value=" There have been several works on SLU employing prompts, such as fine-tuning pretrained models like T5 aided by prompts \cite{DBLP:conf/coling/WuWZCZ22,DBLP:journals/corr/abs-2210-03337}, fine-tuning embeddings prepended to the inputs known as continuous prompts \cite{DBLP:conf/interspeech/ChangT0L22,DBLP:journals/corr/abs-2303-00733}, and end-to-end SLU by in-context learning on GPT2 with a fine-tuned audio encoder \cite{DBLP:conf/interspeech/GaoNQZCH22}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2011.13205" id="5923">
        <attvalues>
          <attvalue for="5" value=" Therefore we endeavor to undertake it by designing prompts and evaluating these models on multiple SLU benchmarks, including SLURP \cite{DBLP:conf/emnlp/BastianelliVSR20} and the multilingual MINDS-14 \cite{DBLP:conf/emnlp/GerzSKMLSMWV21}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2104.08524" id="5924">
        <attvalues>
          <attvalue for="5" value=" Therefore we endeavor to undertake it by designing prompts and evaluating these models on multiple SLU benchmarks, including SLURP \cite{DBLP:conf/emnlp/BastianelliVSR20} and the multilingual MINDS-14 \cite{DBLP:conf/emnlp/GerzSKMLSMWV21}." />
        </attvalues>
      </edge>
      <edge source="2305.13512" target="2205.01068" id="5926">
        <attvalues>
          <attvalue for="5" value=" Furthermore, we compare smaller models that can easily run on a common GPU, namely GPT2 \cite{radford2019language} and several OPT models \cite{DBLP:journals/corr/abs-2205-01068}." />
        </attvalues>
      </edge>
      <edge source="2305.05461" target="2106.12672" id="5945">
        <attvalues>
          <attvalue for="5" value=" We identify one particular configuration that shows the best performance across all of our downstream evaluation tasks, namely a combination of the Charformer downsampling model \cite{Tay:2021}, and CANINE upsampling model and pretraining procedure \cite{Clark:22}." />
        </attvalues>
      </edge>
      <edge source="2305.05461" target="2103.06874" id="5946">
        <attvalues>
          <attvalue for="5" value=" We identify one particular configuration that shows the best performance across all of our downstream evaluation tasks, namely a combination of the Charformer downsampling model \cite{Tay:2021}, and CANINE upsampling model and pretraining procedure \cite{Clark:22}." />
        </attvalues>
      </edge>
      <edge source="2405.14782" target="2107.07002" id="5947">
        <attvalues>
          <attvalue for="5" value=" Evaluation practices thus play a crucial role in the direction of the field: inconsistencies or biases in evaluation practices can lead to skewed performance comparisons, which may influence the direction of future research and the adoption of new methods by the community \cite{dehghani2021benchmark} or lead to adverse effects from deploying suboptimal or harmful models \cite{bender-friedman-2018-data} on tasks for which they are ill-suited \cite{Raji_2022}." />
        </attvalues>
      </edge>
      <edge source="2405.14782" target="2206.09511" id="5948">
        <attvalues>
          <attvalue for="5" value=" Evaluation practices thus play a crucial role in the direction of the field: inconsistencies or biases in evaluation practices can lead to skewed performance comparisons, which may influence the direction of future research and the adoption of new methods by the community \cite{dehghani2021benchmark} or lead to adverse effects from deploying suboptimal or harmful models \cite{bender-friedman-2018-data} on tasks for which they are ill-suited \cite{Raji_2022}." />
        </attvalues>
      </edge>
      <edge source="2205.09350" target="1805.01087" id="5949">
        <attvalues>
          <attvalue for="5" value=" Dependency parsers \cite{dozat-etal-2017-stanfords,ma-etal-2018-stack,strzyz-etal-2019-viable} already achieve accurate results for certain setups \cite{berzak-etal-2016-anchoring}." />
        </attvalues>
      </edge>
      <edge source="2205.09350" target="1605.04481" id="5951">
        <attvalues>
          <attvalue for="5" value=" Dependency parsers \cite{dozat-etal-2017-stanfords,ma-etal-2018-stack,strzyz-etal-2019-viable} already achieve accurate results for certain setups \cite{berzak-etal-2016-anchoring}." />
        </attvalues>
      </edge>
      <edge source="2205.09350" target="1909.02857" id="5952">
        <attvalues>
          <attvalue for="5" value=" On data augmentation, recent approaches have replaced subtrees of sentences to generate new ones \cite{vania-etal-2019-systematic,dehouck-gomez-rodriguez-2020-data}." />
        </attvalues>
      </edge>
      <edge source="2205.09350" target="1507.08449" id="5953">
        <attvalues>
          <attvalue for="5" value=" \cite{vilares-etal-2016-one,ammar-etal-2016-many} merged treebanks to train multilingual parsers that sometimes could outperform the equivalent monolingual version, which has applications for less-resourced parsing." />
        </attvalues>
      </edge>
      <edge source="2205.09350" target="1602.01595" id="5954">
        <attvalues>
          <attvalue for="5" value=" \cite{vilares-etal-2016-one,ammar-etal-2016-many} merged treebanks to train multilingual parsers that sometimes could outperform the equivalent monolingual version, which has applications for less-resourced parsing." />
        </attvalues>
      </edge>
      <edge source="2205.09350" target="1909.08744" id="5955">
        <attvalues>
          <attvalue for="5" value=" In the context of multilingual representations, \cite{mulcaire-etal-2019-low} trained a zero-shot parser on top of a polyglot language model, relying on merged RR treebanks too." />
        </attvalues>
      </edge>
      <edge source="2302.14055" target="2203.01205" id="5964">
        <attvalues>
          <attvalue for="5" value=" Most of these models are trained with self-supervised approaches~\cite{liu2022audio}, making it hard to understand which information is being preserved in the resulting representations." />
        </attvalues>
      </edge>
      <edge source="2302.14055" target="2010.13007" id="5965">
        <attvalues>
          <attvalue for="5" value=" The output labels are characteristics derived from the speech signal, such as phone class \cite{ma2021probing}, pronunciation quality, fluency, or other speech properties \cite{shah2021all}." />
        </attvalues>
      </edge>
      <edge source="2302.14055" target="2203.16193" id="5966">
        <attvalues>
          <attvalue for="5" value=" Consequently, several recent works have focused on analyzing how the properties of a speech signal are encoded in these representations \cite{ma2021probing,de2022probing}." />
        </attvalues>
      </edge>
      <edge source="2302.14055" target="2101.00387" id="5967">
        <attvalues>
          <attvalue for="5" value=" The output labels are characteristics derived from the speech signal, such as phone class \cite{ma2021probing}, pronunciation quality, fluency, or other speech properties \cite{shah2021all}." />
        </attvalues>
      </edge>
      <edge source="2302.14055" target="1905.00414" id="5968">
        <attvalues>
          <attvalue for="5" value=" A comparison of two representations can be made using methods from Representation Similarity Analysis (RSA) \cite{kornblith2019similarity}." />
        </attvalues>
      </edge>
      <edge source="2302.14055" target="2107.04734" id="5969">
        <attvalues>
          <attvalue for="5" value=" In \cite{pasad2021layer}, they found that wav2vec2." />
        </attvalues>
      </edge>
      <edge source="2302.14055" target="2010.11481" id="5970">
        <attvalues>
          <attvalue for="5" value=" Using CKA, \cite{chung2021similarity} found that the learning objective of self-supervised speech models affects the similarity more than the architecture does." />
        </attvalues>
      </edge>
      <edge source="2302.14055" target="2007.13542" id="5971">
        <attvalues>
          <attvalue for="5" value=" This method has been used to measure the intrinsic quality of a speech representation to perform a certain classification task \cite{algayres2020evaluating,carlin2011rapid}." />
        </attvalues>
      </edge>
      <edge source="2007.13542" target="1712.04313" id="5972">
        <attvalues>
          <attvalue for="5" value=" In Section~\ref{results}, we present results on the five speech datasets from the ZeroSpeech Challenge \cite{zs15,zs17}." />
        </attvalues>
      </edge>
      <edge source="2007.13542" target="1811.00403" id="5973">
        <attvalues>
          <attvalue for="5" value=" To address that challenge, recent methods use {speech embeddings}, eg~fixed-size representations of variable-length speech sequences \cite{herman_cae,nils,settle,riad,emb2,emb3,emb5,cae}." />
        </attvalues>
      </edge>
      <edge source="2007.13542" target="1611.02550" id="5974">
        <attvalues>
          <attvalue for="5" value=" To address that challenge, recent methods use {speech embeddings}, eg~fixed-size representations of variable-length speech sequences \cite{herman_cae,nils,settle,riad,emb2,emb3,emb5,cae}." />
        </attvalues>
      </edge>
      <edge source="2007.13542" target="1804.11297" id="5975">
        <attvalues>
          <attvalue for="5" value=" Estimated frequencies can be useful in representation learning by enabling efficient sampling of tokens in a speech database \cite{riad}." />
        </attvalues>
      </edge>
      <edge source="2007.13542" target="1805.07467" id="5977">
        <attvalues>
          <attvalue for="5" value=" To address that challenge, recent methods use {speech embeddings}, eg~fixed-size representations of variable-length speech sequences \cite{herman_cae,nils,settle,riad,emb2,emb3,emb5,cae}." />
        </attvalues>
      </edge>
      <edge source="2007.13542" target="1808.02228" id="5978">
        <attvalues>
          <attvalue for="5" value=" Speech embeddings can be used in many applications, such as key-word spotting\cite{query,query2,query3}, spoken term discovery\cite{utd,utd2,utd3}, and segmentation of speech into words \cite{goldwater,seg1,seg2}." />
        </attvalues>
      </edge>
      <edge source="2007.13542" target="1606.06950" id="5979">
        <attvalues>
          <attvalue for="5" value=" Speech embeddings can be used in many applications, such as key-word spotting\cite{query,query2,query3}, spoken term discovery\cite{utd,utd2,utd3}, and segmentation of speech into words \cite{goldwater,seg1,seg2}." />
        </attvalues>
      </edge>
      <edge source="2105.00572" target="1911.00359" id="5980">
        <attvalues>
          <attvalue for="5" value=" We scale the capacity of XLM-R by almost two orders of magnitude while training on the same CC100 dataset~\cite{wenzek2019ccnet}." />
        </attvalues>
      </edge>
      <edge source="2105.00572" target="2010.11934" id="5982">
        <attvalues>
          <attvalue for="5" value="7 billion parameters respectively, significantly outperform the previous XLM-R model (trained in a similar setting) on cross-lingual understanding benchmarks and obtain competitive performance with the multilingual T5 models~\cite{raffel2019exploring,xue2020mt5}." />
        </attvalues>
      </edge>
      <edge source="2105.00572" target="1907.11692" id="5983">
        <attvalues>
          <attvalue for="5" value=" The XLM-R model~\cite{conneau2019unsupervised} extended that approach by scaling the amount of data by two orders of magnitude, from Wikipedia to Common-Crawl and training longer, similar to RoBERTa~\cite{roberta2019}." />
        </attvalues>
      </edge>
      <edge source="2105.00572" target="1901.07291" id="5986">
        <attvalues>
          <attvalue for="5" value=" Recent multilingual masked language models (MLM) like mBERT ~\cite{devlin2018bert} or XLM \cite{lample2019cross} improved cross-lingual language understanding by pretraining large Transformer models~\cite{transformer17} on multiple languages at once." />
        </attvalues>
      </edge>
      <edge source="2105.00572" target="1706.03762" id="5987">
        <attvalues>
          <attvalue for="5" value=" Recent multilingual masked language models (MLM) like mBERT ~\cite{devlin2018bert} or XLM \cite{lample2019cross} improved cross-lingual language understanding by pretraining large Transformer models~\cite{transformer17} on multiple languages at once." />
        </attvalues>
      </edge>
      <edge source="2105.00572" target="1911.02116" id="5988">
        <attvalues>
          <attvalue for="5" value=" When the number of languages becomes large, \cite{conneau2019unsupervised} even observed an overall decrease of performance on all languages." />
        </attvalues>
      </edge>
      <edge source="2105.00572" target="2005.14165" id="5990">
        <attvalues>
          <attvalue for="5" value=" Recent work scaled language models to hundreds of billions~\cite{brown2020language} or even multiple trillion parameters~\cite{fedus2021switch}, showing consistent gains in doing so." />
        </attvalues>
      </edge>
      <edge source="2105.00572" target="2101.03961" id="5991">
        <attvalues>
          <attvalue for="5" value=" Recent work scaled language models to hundreds of billions~\cite{brown2020language} or even multiple trillion parameters~\cite{fedus2021switch}, showing consistent gains in doing so." />
        </attvalues>
      </edge>
      <edge source="1909.04761" target="1802.05365" id="5992">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (LMs) have shown striking improvements on a range of natural language processing (NLP) tasks \cite{Peters2018,Howard2018,Devlin2018}." />
        </attvalues>
      </edge>
      <edge source="1909.04761" target="1801.06146" id="5993">
        <attvalues>
          <attvalue for="5" value=" Our model combines universal language model fine-tuning \cite[ULMFiT;][]{Howard2018} with the quasi-recurrent neural network \cite[QRNN;][]{Bradbury2017} and subword tokenization \cite{Kudo2018} and can be pretrained on a single Tesla V100 GPU in a few hours." />
        </attvalues>
      </edge>
      <edge source="1909.04761" target="1812.10464" id="5995">
        <attvalues>
          <attvalue for="5" value=" We evaluate our models on two widely used cross-lingual classification datasets, MLDoc \cite{Schwenk2018} and CLS \cite{Prettenhofer2010a} where we outperform the state-of-the-art zero-shot model LASER \cite{Artetxe2018e} and multi-lingual BERT \cite{Devlin2018} in the supervised setting---even without any pretraining." />
        </attvalues>
      </edge>
      <edge source="1909.04761" target="1901.07291" id="5996">
        <attvalues>
          <attvalue for="5" value=" Recently, cross-lingual extensions of these LMs have been proposed that train on multiple languages jointly \cite{Artetxe2018e,Lample2019}." />
        </attvalues>
      </edge>
      <edge source="1909.04761" target="1902.03499" id="5997">
        <attvalues>
          <attvalue for="5" value=" 2) Infrequent scripts are over-segmented in the shared word piece vocabulary \cite{Wang2019}." />
        </attvalues>
      </edge>
      <edge source="1909.04761" target="1611.01576" id="5998">
        <attvalues>
          <attvalue for="5" value=" Our model combines universal language model fine-tuning \cite[ULMFiT;][]{Howard2018} with the quasi-recurrent neural network \cite[QRNN;][]{Bradbury2017} and subword tokenization \cite{Kudo2018} and can be pretrained on a single Tesla V100 GPU in a few hours." />
        </attvalues>
      </edge>
      <edge source="1909.04761" target="1804.10959" id="5999">
        <attvalues>
          <attvalue for="5" value=" Our model combines universal language model fine-tuning \cite[ULMFiT;][]{Howard2018} with the quasi-recurrent neural network \cite[QRNN;][]{Bradbury2017} and subword tokenization \cite{Kudo2018} and can be pretrained on a single Tesla V100 GPU in a few hours." />
        </attvalues>
      </edge>
      <edge source="1909.04761" target="1805.09821" id="6000">
        <attvalues>
          <attvalue for="5" value=" We evaluate our models on two widely used cross-lingual classification datasets, MLDoc \cite{Schwenk2018} and CLS \cite{Prettenhofer2010a} where we outperform the state-of-the-art zero-shot model LASER \cite{Artetxe2018e} and multi-lingual BERT \cite{Devlin2018} in the supervised setting---even without any pretraining." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1606.05250" id="6001">
        <attvalues>
          <attvalue for="5" value=" Performance on many downstream tasks have improved considerably, achieving parity with human baselines in benchmark leaderboards such as SQuAD \cite{squad,squad2} and GLUE \cite{glue}." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1806.03822" id="6002">
        <attvalues>
          <attvalue for="5" value=" Performance on many downstream tasks have improved considerably, achieving parity with human baselines in benchmark leaderboards such as SQuAD \cite{squad,squad2} and GLUE \cite{glue}." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1901.11373" id="6003">
        <attvalues>
          <attvalue for="5" value=" Such an approach has also been shown to reduce the number of training examples that is needed to achieve good performance on the task of interest \cite{glipaper}." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1310.4546" id="6004">
        <attvalues>
          <attvalue for="5" value=" We then show how the skip-gram objective (\S{\ref{sec:skipgram}}; \cite{skipgram}), masked language modeling (\S{\ref{sec:bert}}; \cite{bert}), and permutation language modeling (\S{\ref{sec:xlnet}}; \cite{xlnet}), fit in this framework." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1801.06146" id="6006">
        <attvalues>
          <attvalue for="5" value=" Many of these encoders are trained with a language modeling objective, where the representation of a context is trained to be predictive of a target token by maximizing the log likelihood of predicting this token \cite{daile,ulmfit,gpt,gpt2}." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1802.05365" id="6007">
        <attvalues>
          <attvalue for="5" value=" \cite{elmo} propose an improvement by adding a reverse objective that also predicts the word token that precedes the context." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1801.04062" id="6010">
        <attvalues>
          <attvalue for="5" value=" Such a framework is inspired by the InfoMax principle \cite{linsker} and has been the main driver of progress in self-supervised representation learning in other domains such as computer vision, audio processing, and reinforcement learning \cite{mine,cpc,deepinfomax,amdim,lowe}." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1807.03748" id="6011">
        <attvalues>
          <attvalue for="5" value=" Many of these methods are trained to maximize a particular lower bound called InfoNCE \cite{cpc}---also known as contrastive learning \cite{arora}." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1808.06670" id="6012">
        <attvalues>
          <attvalue for="5" value=" Such a framework is inspired by the InfoMax principle \cite{linsker} and has been the main driver of progress in self-supervised representation learning in other domains such as computer vision, audio processing, and reinforcement learning \cite{mine,cpc,deepinfomax,amdim,lowe}." />
        </attvalues>
      </edge>
      <edge source="1910.08350" target="1902.09229" id="6014">
        <attvalues>
          <attvalue for="5" value=" Many of these methods are trained to maximize a particular lower bound called InfoNCE \cite{cpc}---also known as contrastive learning \cite{arora}." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="2402.00159" id="6015">
        <attvalues>
          <attvalue for="5" value=" With the realization of unprecedented capabilities, the availability of underlying large training corpora and well-established language model training pipelines has shifted the focus in NLP research from defining linguistic inductive biases to the collection and curation of extensive text datasets \cite{dolma, penedo2024fineweb, together2023redpajama, openelm}." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="2005.14165" id="6016">
        <attvalues>
          <attvalue for="5" value=" The recent trend showed an increase in focus on data curation and augmentation compared to innovation in model architecture or training paradigms \cite{brown2020language, touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="2304.15004" id="6017">
        <attvalues>
          <attvalue for="5" value=" However, emergent capabilities in LLMs are noticeable for models of very large sizes such as 10B parameters or above and most of smaller LLMs of 200M-3B parameters have shown limited abilities in reasoning, fact recall and coherent long generation~\cite{schaeffer2024emergent}." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="2311.07052" id="6018">
        <attvalues>
          <attvalue for="5" value=" Examples include MiniMA~\cite{zhang2023towards} and the Phi family~\cite{abdin2024phi, li2023textbooks}." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="2404.14219" id="6019">
        <attvalues>
          <attvalue for="5" value=" Examples include MiniMA~\cite{zhang2023towards} and the Phi family~\cite{abdin2024phi, li2023textbooks}." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="2401.05749" id="6020">
        <attvalues>
          <attvalue for="5" value=" This new trend, however, does not map equally to languages that are not as privileged with huge amounts of high quality content (or content in quantities that allow aggressive filtering), nor the availability of strong models that can be employed to generate diverse synthetic data in substantial quantities in a cost-effective manner, as is the situation with Arabic and many other low-resourced languages~\cite{thompson2024shocking}." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="2308.16149" id="6021">
        <attvalues>
          <attvalue for="5" value=" A commonly-adopted workaround to the issue of data shortage is to turn to machine translation (MT) to benefit from the available content in English, which is evident in the data mixtures of the more capable Arabic models, eg Jais~\cite{sengupta2023jais}." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="1906.08069" id="6022">
        <attvalues>
          <attvalue for="5" value=" Specifically, 1) cultural biases that are stored within a language corpus get imported when translation is used, leading to misaligned models, eg~\cite{holmstrom-etal-2023-bridging} and 2) based on the quality of the translation, certain linguistic intricacies of languages might not be respected~\cite{zhang-toral-2019-effect}, leading to degradation in the quality of the data and thus degradation in the capability of the final model with regard to the quality of the target language." />
        </attvalues>
      </edge>
      <edge source="2405.14277" target="2306.05685" id="6024">
        <attvalues>
          <attvalue for="5" value=" We train models with different sizes using the Arabic-translated TinyStories, and benchmark the trained models against several other Arabic LLMs using GPT-4 as a judge following previous works \cite{zheng2024judging} in the task of story generation using three metrics: grammar correctness, consistency with the provided context, and creativity." />
        </attvalues>
      </edge>
      <edge source="2406.16377" target="2305.11206" id="6026">
        <attvalues>
          <attvalue for="5" value=" The most classic approach is to modify the internal representations and mechanisms of LLMs via parameter update, such as fine-tuning the models on a set of demonstrations of desirable (and undesirable) behaviors \cite{zhou2024lima,hu2021lora,houlsby2019parameter}." />
        </attvalues>
      </edge>
      <edge source="2406.16377" target="2106.09685" id="6027">
        <attvalues>
          <attvalue for="5" value=" The most classic approach is to modify the internal representations and mechanisms of LLMs via parameter update, such as fine-tuning the models on a set of demonstrations of desirable (and undesirable) behaviors \cite{zhou2024lima,hu2021lora,houlsby2019parameter}." />
        </attvalues>
      </edge>
      <edge source="2406.16377" target="1902.00751" id="6028">
        <attvalues>
          <attvalue for="5" value=" The most classic approach is to modify the internal representations and mechanisms of LLMs via parameter update, such as fine-tuning the models on a set of demonstrations of desirable (and undesirable) behaviors \cite{zhou2024lima,hu2021lora,houlsby2019parameter}." />
        </attvalues>
      </edge>
      <edge source="2406.16377" target="2005.14165" id="6029">
        <attvalues>
          <attvalue for="5" value=" Thanks to the exceptional in-context learning capabilities of LLMs \cite{brown2020language}, in-context prompting \cite{wei2023larger,wei2022chain,liu2023pre} has also emerged as a promising new method for altering the model behavior by simply augmenting the model input with an informative prompt." />
        </attvalues>
      </edge>
      <edge source="2406.16377" target="2303.03846" id="6030">
        <attvalues>
          <attvalue for="5" value=" Thanks to the exceptional in-context learning capabilities of LLMs \cite{brown2020language}, in-context prompting \cite{wei2023larger,wei2022chain,liu2023pre} has also emerged as a promising new method for altering the model behavior by simply augmenting the model input with an informative prompt." />
        </attvalues>
      </edge>
      <edge source="2406.16377" target="2201.11903" id="6031">
        <attvalues>
          <attvalue for="5" value=" Thanks to the exceptional in-context learning capabilities of LLMs \cite{brown2020language}, in-context prompting \cite{wei2023larger,wei2022chain,liu2023pre} has also emerged as a promising new method for altering the model behavior by simply augmenting the model input with an informative prompt." />
        </attvalues>
      </edge>
      <edge source="2406.16377" target="2107.13586" id="6032">
        <attvalues>
          <attvalue for="5" value=" Thanks to the exceptional in-context learning capabilities of LLMs \cite{brown2020language}, in-context prompting \cite{wei2023larger,wei2022chain,liu2023pre} has also emerged as a promising new method for altering the model behavior by simply augmenting the model input with an informative prompt." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2303.08774" id="6033">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as GPT-4~\cite{openai2022gpt4} and Gemini~\cite{team2023gemini}, have reformed the domain of artificial intelligence (AI) with astonishing language capacities, such as natural language understanding~\cite{yang2023harnessing,touvron2023llama}, text generation~\cite{kocon2023chatgpt,anil2023palm}, machine translation~\cite{jiao2023chatgpt}, summarization~\cite{xie-etal-2024-chunk}, and programming~\cite{surameery2023use,tian2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2312.11805" id="6034">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as GPT-4~\cite{openai2022gpt4} and Gemini~\cite{team2023gemini}, have reformed the domain of artificial intelligence (AI) with astonishing language capacities, such as natural language understanding~\cite{yang2023harnessing,touvron2023llama}, text generation~\cite{kocon2023chatgpt,anil2023palm}, machine translation~\cite{jiao2023chatgpt}, summarization~\cite{xie-etal-2024-chunk}, and programming~\cite{surameery2023use,tian2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2304.13712" id="6035">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as GPT-4~\cite{openai2022gpt4} and Gemini~\cite{team2023gemini}, have reformed the domain of artificial intelligence (AI) with astonishing language capacities, such as natural language understanding~\cite{yang2023harnessing,touvron2023llama}, text generation~\cite{kocon2023chatgpt,anil2023palm}, machine translation~\cite{jiao2023chatgpt}, summarization~\cite{xie-etal-2024-chunk}, and programming~\cite{surameery2023use,tian2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2307.09288" id="6036">
        <attvalues>
          <attvalue for="5" value=" We conduct the self-play on this adversarial game using open-source LLMs, LLaMA-2-7B~\cite{touvron2023llama} and Baichuan-2-13B~\cite{yang2023baichuan}, with target words selected from a 50K top-frequency vocabulary~\cite{davies_coca_2019}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2302.10724" id="6037">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as GPT-4~\cite{openai2022gpt4} and Gemini~\cite{team2023gemini}, have reformed the domain of artificial intelligence (AI) with astonishing language capacities, such as natural language understanding~\cite{yang2023harnessing,touvron2023llama}, text generation~\cite{kocon2023chatgpt,anil2023palm}, machine translation~\cite{jiao2023chatgpt}, summarization~\cite{xie-etal-2024-chunk}, and programming~\cite{surameery2023use,tian2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2308.13191" id="6038">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as GPT-4~\cite{openai2022gpt4} and Gemini~\cite{team2023gemini}, have reformed the domain of artificial intelligence (AI) with astonishing language capacities, such as natural language understanding~\cite{yang2023harnessing,touvron2023llama}, text generation~\cite{kocon2023chatgpt,anil2023palm}, machine translation~\cite{jiao2023chatgpt}, summarization~\cite{xie-etal-2024-chunk}, and programming~\cite{surameery2023use,tian2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2305.12295" id="6039">
        <attvalues>
          <attvalue for="5" value=" To address the reasoning challenge of LLMs, plenty of works have contributed in-depth efforts from the perspectives of Chain-of-Thought (CoT) prompt engineering~\cite{wei2022chain,ding2023everything,yao2024tree}, and the usage of auxiliary reasoning tools~\cite{pan2023logic}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="1911.01622" id="6040">
        <attvalues>
          <attvalue for="5" value=" With the above consideration, we select an adversarial language game called Adversarial Taboo~\cite{yao2021adversarial}, in which an attacker and a defender perform a conversation around a target word only visible to the attacker." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2305.13534" id="6041">
        <attvalues>
          <attvalue for="5" value=" However, the reasoning ability of LLMs, which is essential for complex problem-solving~\cite{pan2023logic} and advanced intelligence-developing~\cite{yao2021adversarial}, still retains being challenged by various criteria including correctness~\cite{zhang2023language} and faithfulness~\cite{turpin2024language}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2305.04388" id="6042">
        <attvalues>
          <attvalue for="5" value=" However, both prompt-based and tool-calling methods require additional prompt designs, which are inconsistent and sensitive to different prompt patterns and LLM checkpoints~\cite{turpin2024language,chu2023survey}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2201.11903" id="6043">
        <attvalues>
          <attvalue for="5" value=" To address the reasoning challenge of LLMs, plenty of works have contributed in-depth efforts from the perspectives of Chain-of-Thought (CoT) prompt engineering~\cite{wei2022chain,ding2023everything,yao2024tree}, and the usage of auxiliary reasoning tools~\cite{pan2023logic}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2311.04254" id="6044">
        <attvalues>
          <attvalue for="5" value=" To address the reasoning challenge of LLMs, plenty of works have contributed in-depth efforts from the perspectives of Chain-of-Thought (CoT) prompt engineering~\cite{wei2022chain,ding2023everything,yao2024tree}, and the usage of auxiliary reasoning tools~\cite{pan2023logic}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2305.10601" id="6045">
        <attvalues>
          <attvalue for="5" value=" To address the reasoning challenge of LLMs, plenty of works have contributed in-depth efforts from the perspectives of Chain-of-Thought (CoT) prompt engineering~\cite{wei2022chain,ding2023everything,yao2024tree}, and the usage of auxiliary reasoning tools~\cite{pan2023logic}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2310.05492" id="6047">
        <attvalues>
          <attvalue for="5" value=" More fundamental and consistent reasoning-improving approaches are post-pretraining~\cite{azerbayev2023llemma} and fine-tuning~\cite{dong2023abilities}, which trains LLMs with additional reasoning-related text corpus." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2312.06585" id="6048">
        <attvalues>
          <attvalue for="5" value=" To improve LLM reasoning more efficiently, self-improvement methods, which enhance LLMs with model-generated synthetic data, have recently attracted increasing research attention~\cite{singh2023beyond,huang2023large,burns2023weak,chen2024self}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2210.11610" id="6049">
        <attvalues>
          <attvalue for="5" value=" Self-improvement methods usually utilize the intrinsic language capability of LLMs to judge~\cite{huang2023large}, filter~\cite{yuan2024self}, or revise~\cite{yuan2024self} self-generated samples to enhance their quality." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2312.09390" id="6050">
        <attvalues>
          <attvalue for="5" value=" To improve LLM reasoning more efficiently, self-improvement methods, which enhance LLMs with model-generated synthetic data, have recently attracted increasing research attention~\cite{singh2023beyond,huang2023large,burns2023weak,chen2024self}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2401.01335" id="6051">
        <attvalues>
          <attvalue for="5" value=" To improve LLM reasoning more efficiently, self-improvement methods, which enhance LLMs with model-generated synthetic data, have recently attracted increasing research attention~\cite{singh2023beyond,huang2023large,burns2023weak,chen2024self}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2401.10020" id="6052">
        <attvalues>
          <attvalue for="5" value=" Self-improvement methods usually utilize the intrinsic language capability of LLMs to judge~\cite{huang2023large}, filter~\cite{yuan2024self}, or revise~\cite{yuan2024self} self-generated samples to enhance their quality." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2402.14016" id="6053">
        <attvalues>
          <attvalue for="5" value=" Besides, the judgments from LLMs are not guaranteed objective~\cite{raina2024llm}." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="1909.05398" id="6054">
        <attvalues>
          <attvalue for="5" value=" Although language games have attracted increasing attention in natural language processing~\cite{lewis2017deal,hausknecht2020interactive,xu2023exploring,wu2024enhance}, most of them are specially designed with customized game rules, in lack of the generalization to improve the general language capacities of LLMs." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2309.04658" id="6055">
        <attvalues>
          <attvalue for="5" value=" Although language games have attracted increasing attention in natural language processing~\cite{lewis2017deal,hausknecht2020interactive,xu2023exploring,wu2024enhance}, most of them are specially designed with customized game rules, in lack of the generalization to improve the general language capacities of LLMs." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2311.18232" id="6057">
        <attvalues>
          <attvalue for="5" value=" Recent studies have raised interest in entity- or word-based language games, such as 20-Question~\cite{zhang2023entity} and Guess-My-City~\cite{abdulhai2023lmrl} , which provide not only straight-forward word-level outcomes but also language universality by traversing the game word from comprehensive vocabularies." />
        </attvalues>
      </edge>
      <edge source="2404.10642" target="2309.10305" id="6058">
        <attvalues>
          <attvalue for="5" value=" We conduct the self-play on this adversarial game using open-source LLMs, LLaMA-2-7B~\cite{touvron2023llama} and Baichuan-2-13B~\cite{yang2023baichuan}, with target words selected from a 50K top-frequency vocabulary~\cite{davies_coca_2019}." />
        </attvalues>
      </edge>
      <edge source="2306.15268" target="1711.02173" id="6059">
        <attvalues>
          <attvalue for="5" value=" These noisy words may be caused by accidental typos \cite{belinkov2017synthetic} or spelling variants on social media \cite{ritter-etal-2010-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2306.15268" target="2003.12932" id="6060">
        <attvalues>
          <attvalue for="5" value=" However, despite their success in in-distribution test data with standardized word forms, Pretrained Language Models (PLMs), which serve as the backbone models, tend to perform poorly on rare or noisy words \cite{kumar-etal-2020-noisy,baron2015words}." />
        </attvalues>
      </edge>
      <edge source="2306.15268" target="2204.00716" id="6061">
        <attvalues>
          <attvalue for="5" value=" Prior studies show that most subword-based PLMs perform poorly under noise largely due to subword segmentation \cite{zhuang2022typos}, while character-based PLMs show more robustness \cite{el-boukkouri-etal-2020-characterbert}." />
        </attvalues>
      </edge>
      <edge source="2306.15268" target="2010.10392" id="6062">
        <attvalues>
          <attvalue for="5" value=" Prior studies show that most subword-based PLMs perform poorly under noise largely due to subword segmentation \cite{zhuang2022typos}, while character-based PLMs show more robustness \cite{el-boukkouri-etal-2020-characterbert}." />
        </attvalues>
      </edge>
      <edge source="2306.15268" target="2210.17004" id="6063">
        <attvalues>
          <attvalue for="5" value=" Examining the impact of subword segmentation factors on PLMs is also crucial for defending against the adversarial attacks that leverage the sensitivity of subword segmentation to noise \cite{liu-etal-2022-character}." />
        </attvalues>
      </edge>
      <edge source="2111.00191" target="2005.06600" id="6064">
        <attvalues>
          <attvalue for="5" value=" First, a data advancing automation approach is employed to advance the source ({\em{eg,}} the initial mono corpus) language using corpus filtering~\cite{herold2021data} and Grammar Error Correction (GEC)~\cite{wang2020comprehensive}." />
        </attvalues>
      </edge>
      <edge source="1810.12836" target="1804.00079" id="6065">
        <attvalues>
          <attvalue for="5" value=" We present a novel approach for cross-lingual representation learning that combines methods for multi-task learning of monolingual sentence representations~\cite{unec2018,subramanian18} with recent work on dual encoder methods for obtaining multilingual sentence representations for bi-text retrieval~\cite{Guo2018,yang2019}." />
        </attvalues>
      </edge>
      <edge source="1810.12836" target="1807.11906" id="6066">
        <attvalues>
          <attvalue for="5" value=" We present a novel approach for cross-lingual representation learning that combines methods for multi-task learning of monolingual sentence representations~\cite{unec2018,subramanian18} with recent work on dual encoder methods for obtaining multilingual sentence representations for bi-text retrieval~\cite{Guo2018,yang2019}." />
        </attvalues>
      </edge>
      <edge source="1810.12836" target="1902.08564" id="6067">
        <attvalues>
          <attvalue for="5" value=" We present a novel approach for cross-lingual representation learning that combines methods for multi-task learning of monolingual sentence representations~\cite{unec2018,subramanian18} with recent work on dual encoder methods for obtaining multilingual sentence representations for bi-text retrieval~\cite{Guo2018,yang2019}." />
        </attvalues>
      </edge>
      <edge source="1810.12836" target="1805.03620" id="6068">
        <attvalues>
          <attvalue for="5" value=" Multi-task training using additional monolingual tasks is found to improve performance over models that only make use of parallel data on both cross-lingual semantic textual similarity (STS)~\cite{cer-EtAl:2017:SemEval} and cross-lingual eigen-similarity~\cite{sogard2018}." />
        </attvalues>
      </edge>
      <edge source="2209.12711" target="2005.14165" id="6069">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LMs) pretrained on a vast amounts of corpora have shown surprising, even emergent, capabilities of solving various downstream tasks through prompts (instructions)~\cite{brown2020language, rae2021scaling, chowdhery2022palm, zhang2022opt, wei2022emergent}." />
        </attvalues>
      </edge>
      <edge source="2209.12711" target="2110.08207" id="6070">
        <attvalues>
          <attvalue for="5" value=" Previous work has specifically shown LMs can perform unseen tasks through multitask fine-tuning on various downstream tasks with prompts~\cite{sanh2021multitask, wei2021finetuned, wang2022benchmarking, ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2209.12711" target="2204.02311" id="6071">
        <attvalues>
          <attvalue for="5" value=" A 540B LM~\cite{chowdhery2022palm} has even shown the capability to act as the ``brain'' for actual robots, helping them perform different tasks in the real-world~\cite{ahn2022can}." />
        </attvalues>
      </edge>
      <edge source="2209.12711" target="2204.01691" id="6072">
        <attvalues>
          <attvalue for="5" value=" A 540B LM~\cite{chowdhery2022palm} has even shown the capability to act as the ``brain'' for actual robots, helping them perform different tasks in the real-world~\cite{ahn2022can}." />
        </attvalues>
      </edge>
      <edge source="2209.12711" target="1907.13528" id="6073">
        <attvalues>
          <attvalue for="5" value=" Prior work~\cite{ettinger2020bert, webson2021prompt} has shown that LMs (as well as other large pretrained models in different modalities such as DALLE-2~\cite{ramesh2022hierarchical}) have a hard time understanding negated prompts and perform the task as if provided with the original prompt." />
        </attvalues>
      </edge>
      <edge source="2209.12711" target="2204.06125" id="6074">
        <attvalues>
          <attvalue for="5" value=" Prior work~\cite{ettinger2020bert, webson2021prompt} has shown that LMs (as well as other large pretrained models in different modalities such as DALLE-2~\cite{ramesh2022hierarchical}) have a hard time understanding negated prompts and perform the task as if provided with the original prompt." />
        </attvalues>
      </edge>
      <edge source="1908.05731" target="1409.3215" id="6075">
        <attvalues>
          <attvalue for="5" value=" Sequence to sequence models directly estimate the posterior probability of a target sequence $y$ given a source sequence $x$~\cite{sutskever2014sequence,bahdanau2015neural,gehring2017convs2s,vaswani2017transformer} and can be trained with pairs of source and target sequences." />
        </attvalues>
      </edge>
      <edge source="1908.05731" target="1409.0473" id="6076">
        <attvalues>
          <attvalue for="5" value=" Sequence to sequence models directly estimate the posterior probability of a target sequence $y$ given a source sequence $x$~\cite{sutskever2014sequence,bahdanau2015neural,gehring2017convs2s,vaswani2017transformer} and can be trained with pairs of source and target sequences." />
        </attvalues>
      </edge>
      <edge source="1908.05731" target="1706.03762" id="6078">
        <attvalues>
          <attvalue for="5" value=" Sequence to sequence models directly estimate the posterior probability of a target sequence $y$ given a source sequence $x$~\cite{sutskever2014sequence,bahdanau2015neural,gehring2017convs2s,vaswani2017transformer} and can be trained with pairs of source and target sequences." />
        </attvalues>
      </edge>
      <edge source="1908.05731" target="1511.06709" id="6079">
        <attvalues>
          <attvalue for="5" value=" Unpaired sequences can be leveraged by data augmentation schemes such as back-translation, but direct models cannot naturally take advantage of unpaired data~\cite{sennrich2016bt,edunov2018bt}." />
        </attvalues>
      </edge>
      <edge source="1908.05731" target="1808.09381" id="6080">
        <attvalues>
          <attvalue for="5" value=" Unpaired sequences can be leveraged by data augmentation schemes such as back-translation, but direct models cannot naturally take advantage of unpaired data~\cite{sennrich2016bt,edunov2018bt}." />
        </attvalues>
      </edge>
      <edge source="1908.05731" target="1611.02554" id="6081">
        <attvalues>
          <attvalue for="5" value=" Previous work on neural noisy channel modeling relied on a complex latent variable model that incrementally processes source and target prefixes~\cite{yu2017neuralnoisy}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="1809.02836" id="6082">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="1906.06349" id="6083">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="1906.01615" id="6084">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2004.08500" id="6085">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2010.07515" id="6086">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2106.16213" id="6087">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2201.12451" id="6088">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2310.05161" id="6089">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2310.12942" id="6090">
        <attvalues>
          <attvalue for="5" value=" This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades \cite{McCulloch1943,Kleene1956,siegelmann-sontag-1992,hao-etal-2018-context,DBLP:journals/corr/abs-1906-06349,merrill-2019-sequential,merrill-etal-2020-formal,hewitt-etal-2020-rnns,Chung2021,merrill-etal-2022-saturated,merrill2022extracting,svete2023recurrent,nowak-etal-2023-representational}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="1906.06755" id="6091">
        <attvalues>
          <attvalue for="5" value=" Moreover, the prevalence of Transformer-based LMs has led to a recent body of work investigating their representational capacity \cite[eg,][]{hahn-2020-theoretical,ebrahimi-etal-2020-self,bhattamishra-etal-2020-ability,merrill-sabharwal-2023-parallelism}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2010.04303" id="6092">
        <attvalues>
          <attvalue for="5" value="\looseness-1 A closer look at the existing work on the empirical learnability of formal languages (see \ref{app:related_work} for an overview) reveals a categorical mismatch between what LMs are, eg, probability distributions over strings, and what learning a formal language means, eg, classifying strings as members of a specific language, eg, a set of strings \cite{ebrahimi-etal-2020-self,deletang2023neural,wang-steinert-threlkeld-2023-evaluating}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2009.11264" id="6093">
        <attvalues>
          <attvalue for="5" value=" Moreover, the prevalence of Transformer-based LMs has led to a recent body of work investigating their representational capacity \cite[eg,][]{hahn-2020-theoretical,ebrahimi-etal-2020-self,bhattamishra-etal-2020-ability,merrill-sabharwal-2023-parallelism}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2207.00729" id="6094">
        <attvalues>
          <attvalue for="5" value=" Moreover, the prevalence of Transformer-based LMs has led to a recent body of work investigating their representational capacity \cite[eg,][]{hahn-2020-theoretical,ebrahimi-etal-2020-self,bhattamishra-etal-2020-ability,merrill-sabharwal-2023-parallelism}." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2207.02098" id="6095">
        <attvalues>
          <attvalue for="5" value=" Similar to \cite{deletang2023neural}, we find that RNNs are better suited to modeling formal languages than Transformers." />
        </attvalues>
      </edge>
      <edge source="2406.04289" target="2309.00857" id="6096">
        <attvalues>
          <attvalue for="5" value="\looseness-1 A closer look at the existing work on the empirical learnability of formal languages (see \ref{app:related_work} for an overview) reveals a categorical mismatch between what LMs are, eg, probability distributions over strings, and what learning a formal language means, eg, classifying strings as members of a specific language, eg, a set of strings \cite{ebrahimi-etal-2020-self,deletang2023neural,wang-steinert-threlkeld-2023-evaluating}." />
        </attvalues>
      </edge>
      <edge source="2307.02053" target="2306.04757" id="6098">
        <attvalues>
          <attvalue for="5" value=" This spurred us to fine-tune Vicuna on Flan-mini Collection dataset, anticipating improvement on reasoning-intensive tasks in InstructEval~\cite{instructeval}." />
        </attvalues>
      </edge>
      <edge source="2307.02053" target="2301.13688" id="6099">
        <attvalues>
          <attvalue for="5" value=" To this end, we first sample a 1M-sized instruction dataset from the 15M-sized Flan Collection dataset~\cite{longpre2023flan} and combined it with several other datasets comprising coding tasks and ChatGPT/GPT-4 distilled conversations." />
        </attvalues>
      </edge>
      <edge source="2307.02053" target="2106.09685" id="6100">
        <attvalues>
          <attvalue for="5" value=" To ensure a reasonable computational cost for the fine-tuning process, we retrofit LoRA~\cite{hu2021lora} adapter into the LLaMA~\cite{llama} decoder-transformer of Vicuna." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="1905.00546" id="6102">
        <attvalues>
          <attvalue for="5" value=" In recent years, Semi-Supervised Learning (SSL) has attracted a lot of research interest in many fields of deep learning, such as Automatic Speech Recognition (ASR) \cite{ssl_asr0,ssl_asr1,synnaeve2020endtoend}, Computer Vision \cite{ssl_cv0,ssl_cv1,xie2020self} and Natural Language Processing \cite{ssl_nlp0,ssl_nlp1,ssl_nlp2}." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="1904.04445" id="6103">
        <attvalues>
          <attvalue for="5" value=" In recent years, Semi-Supervised Learning (SSL) has attracted a lot of research interest in many fields of deep learning, such as Automatic Speech Recognition (ASR) \cite{ssl_asr0,ssl_asr1,synnaeve2020endtoend}, Computer Vision \cite{ssl_cv0,ssl_cv1,xie2020self} and Natural Language Processing \cite{ssl_nlp0,ssl_nlp1,ssl_nlp2}." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="1911.04252" id="6104">
        <attvalues>
          <attvalue for="5" value=" Among these methods, Noisy Student Training (NST) has recently demonstrated extremely strong performances in Image Classification \cite{xie2020self} by introducing noise and randomness into traditional Teacher-student Learning \cite{teacher_student0,teacher_student1}." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="1909.13788" id="6105">
        <attvalues>
          <attvalue for="5" value=" In recent years, Semi-Supervised Learning (SSL) has attracted a lot of research interest in many fields of deep learning, such as Automatic Speech Recognition (ASR) \cite{ssl_asr0,ssl_asr1,synnaeve2020endtoend}, Computer Vision \cite{ssl_cv0,ssl_cv1,xie2020self} and Natural Language Processing \cite{ssl_nlp0,ssl_nlp1,ssl_nlp2}." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="2006.11477" id="6107">
        <attvalues>
          <attvalue for="5" value=" After combing with pre-train methods \cite{baevski2020wav2vec}, NST is shown to be a vital component for achieving SOTA results on a number of datasets, eg Librispeech \cite{NST_ASR2_pushing_limits}." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="2010.10504" id="6108">
        <attvalues>
          <attvalue for="5" value=" After combing with pre-train methods \cite{baevski2020wav2vec}, NST is shown to be a vital component for achieving SOTA results on a number of datasets, eg Librispeech \cite{NST_ASR2_pushing_limits}." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="2204.01981" id="6109">
        <attvalues>
          <attvalue for="5" value=" Another recent unsupervised data selection technique is investigated in \cite{lu22_interspeech}, where a contrastive Language Model is applied as a data selector to better improve the target-domain ASR task." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="2111.05948" id="6110">
        <attvalues>
          <attvalue for="5" value=" We leverage concept of contrastive LM and data selection method in \cite{zheng2022scaling}." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="1709.05522" id="6111">
        <attvalues>
          <attvalue for="5" value=" \end{itemize} Experiments on AISHELL-1 \cite{aishell1} as supervised data and WenetSpeech \cite{zhang2022WenetSpeech} as unsupervised data indicate a significant improvement of 10." />
        </attvalues>
      </edge>
      <edge source="2211.04717" target="1808.10583" id="6112">
        <attvalues>
          <attvalue for="5" value=" When combined AISHELL-2 \cite{aishell2} and WenetSpeech as unsupervised data, 3." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1409.0473" id="6113">
        <attvalues>
          <attvalue for="5" value=" Neural machine translation (NMT) \cite{bahdanau2015neural,vaswani2017attention} has been originally developed to work sentence by sentence." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1706.03762" id="6114">
        <attvalues>
          <attvalue for="5" value=" Neural machine translation (NMT) \cite{bahdanau2015neural,vaswani2017attention} has been originally developed to work sentence by sentence." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1808.07048" id="6115">
        <attvalues>
          <attvalue for="5" value=" Recently, it has been claimed that sentence-level NMT generates document-level errors, eg wrong coreference of pronouns/articles or inconsistent translations throughout a document \cite{guillou2018pronoun,laubli2018has}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1711.00513" id="6116">
        <attvalues>
          <attvalue for="5" value=" Despite the high overhead in modeling, translation metric scores (eg {\bleu}) are often only marginally improved, leaving the evaluation to artificial tests targeted for pronoun resolution \cite{jean2017does,tiedemann2017neural,bawden2018evaluating,voita2018context,voita2019when}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1805.10163" id="6117">
        <attvalues>
          <attvalue for="5" value=" Despite the high overhead in modeling, translation metric scores (eg {\bleu}) are often only marginally improved, leaving the evaluation to artificial tests targeted for pronoun resolution \cite{jean2017does,tiedemann2017neural,bawden2018evaluating,voita2018context,voita2019when}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1810.03581" id="6118">
        <attvalues>
          <attvalue for="5" value=" Even if the metric score gets significantly better, the improvement is limited to specific datasets or explained with only a few examples \cite{tu2018learning,maruf2018document,kuang2018fusing,cao2018encoding,zhang2018improving,maruf2019selective}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1809.01576" id="6119">
        <attvalues>
          <attvalue for="5" value=" Modeling of the context is usually done with fully-fledged NMT encoders with extensions to consider complex relations between sentences \cite{bawden2018evaluating,voita2018context,zhang2018improving,miculicich2018document,maruf2019selective}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1903.08788" id="6120">
        <attvalues>
          <attvalue for="5" value=" Even if the metric score gets significantly better, the improvement is limited to specific datasets or explained with only a few examples \cite{tu2018learning,maruf2018document,kuang2018fusing,cao2018encoding,zhang2018improving,maruf2019selective}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1704.05135" id="6121">
        <attvalues>
          <attvalue for="5" value=" Despite the high overhead in modeling, translation metric scores (eg {\bleu}) are often only marginally improved, leaving the evaluation to artificial tests targeted for pronoun resolution \cite{jean2017does,tiedemann2017neural,bawden2018evaluating,voita2018context,voita2019when}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1708.05943" id="6122">
        <attvalues>
          <attvalue for="5" value=" Despite the high overhead in modeling, translation metric scores (eg {\bleu}) are often only marginally improved, leaving the evaluation to artificial tests targeted for pronoun resolution \cite{jean2017does,tiedemann2017neural,bawden2018evaluating,voita2018context,voita2019when}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1905.05979" id="6123">
        <attvalues>
          <attvalue for="5" value=" Despite the high overhead in modeling, translation metric scores (eg {\bleu}) are often only marginally improved, leaving the evaluation to artificial tests targeted for pronoun resolution \cite{jean2017does,tiedemann2017neural,bawden2018evaluating,voita2018context,voita2019when}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1711.09367" id="6124">
        <attvalues>
          <attvalue for="5" value=" Even if the metric score gets significantly better, the improvement is limited to specific datasets or explained with only a few examples \cite{tu2018learning,maruf2018document,kuang2018fusing,cao2018encoding,zhang2018improving,maruf2019selective}." />
        </attvalues>
      </edge>
      <edge source="1910.00294" target="1711.03688" id="6125">
        <attvalues>
          <attvalue for="5" value=" Even if the metric score gets significantly better, the improvement is limited to specific datasets or explained with only a few examples \cite{tu2018learning,maruf2018document,kuang2018fusing,cao2018encoding,zhang2018improving,maruf2019selective}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2201.11903" id="6127">
        <attvalues>
          <attvalue for="5" value=" Prompting methods for large language models (LLMs) have gained significant attention for their ability to enhance reasoning capabilities through multi-step processes, such as Chain of Thought (CoT)\cite{wei2022chain}, Tree of Thought(ToT)\cite{yao2024tree}, and ReAct\cite{yao2022react}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2305.10601" id="6128">
        <attvalues>
          <attvalue for="5" value=" Prompting methods for large language models (LLMs) have gained significant attention for their ability to enhance reasoning capabilities through multi-step processes, such as Chain of Thought (CoT)\cite{wei2022chain}, Tree of Thought(ToT)\cite{yao2024tree}, and ReAct\cite{yao2022react}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2210.03629" id="6129">
        <attvalues>
          <attvalue for="5" value=" Prompting methods for large language models (LLMs) have gained significant attention for their ability to enhance reasoning capabilities through multi-step processes, such as Chain of Thought (CoT)\cite{wei2022chain}, Tree of Thought(ToT)\cite{yao2024tree}, and ReAct\cite{yao2022react}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2203.14465" id="6131">
        <attvalues>
          <attvalue for="5" value=" These approaches can be extended by incorporating search algorithms to optimize prompts, utilizing techniques like Monte Carlo Tree Search (LATS, STaR)\cite{zhou2023language,zelikman2022star}, bandit algorithms (LongPO)\cite{hsieh2023automatic}, and gradient-style search (OPRO)\cite{Yang2023LargeLM}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2311.10117" id="6132">
        <attvalues>
          <attvalue for="5" value=" These approaches can be extended by incorporating search algorithms to optimize prompts, utilizing techniques like Monte Carlo Tree Search (LATS, STaR)\cite{zhou2023language,zelikman2022star}, bandit algorithms (LongPO)\cite{hsieh2023automatic}, and gradient-style search (OPRO)\cite{Yang2023LargeLM}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2309.03409" id="6133">
        <attvalues>
          <attvalue for="5" value=" These approaches can be extended by incorporating search algorithms to optimize prompts, utilizing techniques like Monte Carlo Tree Search (LATS, STaR)\cite{zhou2023language,zelikman2022star}, bandit algorithms (LongPO)\cite{hsieh2023automatic}, and gradient-style search (OPRO)\cite{Yang2023LargeLM}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2402.10200" id="6134">
        <attvalues>
          <attvalue for="5" value=" Previous approaches to measuring uncertainty in LLMs primarily rely on token-level or sentence-level generation likelihoods, often represented by metrics like token disparity probability~\cite{wang2024chain}, predictive entropy~\cite{npe_lnpe} and reciprocal of perplexity~\cite{chen1998evaluation}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2002.07650" id="6135">
        <attvalues>
          <attvalue for="5" value=" Previous approaches to measuring uncertainty in LLMs primarily rely on token-level or sentence-level generation likelihoods, often represented by metrics like token disparity probability~\cite{wang2024chain}, predictive entropy~\cite{npe_lnpe} and reciprocal of perplexity~\cite{chen1998evaluation}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2309.17249" id="6136">
        <attvalues>
          <attvalue for="5" value=" These techniques have been used for bias calibration~\cite{zhou2023batch}, controllable decoding~\cite{zhu2024hot}, and LLM planning~\cite{ren2023robots}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2309.02772" id="6137">
        <attvalues>
          <attvalue for="5" value=" These techniques have been used for bias calibration~\cite{zhou2023batch}, controllable decoding~\cite{zhu2024hot}, and LLM planning~\cite{ren2023robots}." />
        </attvalues>
      </edge>
      <edge source="2409.10044" target="2307.01928" id="6138">
        <attvalues>
          <attvalue for="5" value=" These techniques have been used for bias calibration~\cite{zhou2023batch}, controllable decoding~\cite{zhu2024hot}, and LLM planning~\cite{ren2023robots}." />
        </attvalues>
      </edge>
      <edge source="2408.15729" target="2005.04611" id="6140">
        <attvalues>
          <attvalue for="5" value=" Given their importance, prior work has sought to measure the amount of factual knowledge encoded in LMs using knowledge probing mechanisms~\cite{petroniHowContextAffects2020, kaloKAMELKnowledgeAnalysis2022}." />
        </attvalues>
      </edge>
      <edge source="2108.10447" target="1308.0850" id="6142">
        <attvalues>
          <attvalue for="5" value=" Autoregressive TTS models rely on the attention mechanism~\cite{Graves13, bahdanau2014neural} to align text and speech, typically using content based attention mechanism~\cite{Tacotron, Flowtron}." />
        </attvalues>
      </edge>
      <edge source="2108.10447" target="1712.05884" id="6143">
        <attvalues>
          <attvalue for="5" value=" These models generally rely on external aligners~\cite{fastspeech2} like the Montreal Forced Aligner (MFA)~\cite{MFA}, or on durations extracted from a pre-trained autoregressive model (or forced aligner)~\cite{lancucki2020fastpitch, peng2020non, yiFastSpeech} like Tacotron 2~\cite{Tacotron2}." />
        </attvalues>
      </edge>
      <edge source="2108.10447" target="2005.11129" id="6144">
        <attvalues>
          <attvalue for="5" value=" Although recent works have improved alignments by using both content and location sensitive attention~\cite{Tacotron2}, such models still suffer from alignment problems on long utterances~\cite{kim2020glowtts}." />
        </attvalues>
      </edge>
      <edge source="2108.10447" target="2006.06873" id="6145">
        <attvalues>
          <attvalue for="5" value=" We demonstrate that this approach can also be used to learn alignments online in parallel TTS models~\cite{lancucki2020fastpitch, fastspeech2, rad_tts}, again eliminating the need for external aligners or alignments obtained from a pre-trained TTS models." />
        </attvalues>
      </edge>
      <edge source="2108.10447" target="1710.08969" id="6146">
        <attvalues>
          <attvalue for="5" value=" In addition, we further examine the effect of a simple, static alignment prior for guiding alignment attention learning \cite{tachibana2018efficiently, rad_tts}." />
        </attvalues>
      </edge>
      <edge source="2204.00990" target="1712.05884" id="6151">
        <attvalues>
          <attvalue for="5" value=" Neural network-based text-to-speech (TTS), aiming to synthesize intelligible and natural speech from text, has made great progress in recent years \cite{shen2018natural, ping2018deep, ren2020fastspeech}." />
        </attvalues>
      </edge>
      <edge source="2204.00990" target="2106.15561" id="6153">
        <attvalues>
          <attvalue for="5" value=" As a result, adapting TTS models to an arbitrary speaker with a few samples (speaker adaptation) is a hot research topic in academia and industry recently \cite{tan2021survey}." />
        </attvalues>
      </edge>
      <edge source="2204.00990" target="1802.06006" id="6154">
        <attvalues>
          <attvalue for="5" value=" One of the general approaches for speaker adaptation is fine-tuning the whole or part of a well-trained multi-speaker TTS model with a few adaptation data \cite{arik2018neural}." />
        </attvalues>
      </edge>
      <edge source="2204.00990" target="2103.00993" id="6155">
        <attvalues>
          <attvalue for="5" value=" These methods are proven to achieve considerable adaptation performance, but with some drawbacks: (i) certain adaptation time and trainable parameters are required for each new target speaker; (ii) voice quality drops quickly when the adaptation data is less than 10 sentences \cite{chen2021adaspeech}; (iii) adaptation performance can be affected by low-quality speeches of the target speaker, resulting in poor intelligibility and prosody of synthesized speech." />
        </attvalues>
      </edge>
      <edge source="2204.00990" target="1806.04558" id="6156">
        <attvalues>
          <attvalue for="5" value=" Some researches draw on transfer learning, such as x-vectors from speaker verification tasks \cite{jia2018transfer, cooper2020zero,zhang2021one}." />
        </attvalues>
      </edge>
      <edge source="2204.00990" target="1810.07217" id="6157">
        <attvalues>
          <attvalue for="5" value=" Others focus on training an encoder network jointly with acoustic models, like using global speaker embeddings (GSEs) \cite{lu2019one} or variational autoencoder (VAE) \cite{hsu2018hierarchical, nguyen2021nvc}." />
        </attvalues>
      </edge>
      <edge source="2204.00990" target="1907.02479" id="6158">
        <attvalues>
          <attvalue for="5" value=" Inspired by related works on prosody transfer \cite{klimkov2019finegrained, lee2019robust, li2021towards}, some prior researches try to introduce fine-grained speaker embedding via attention mechanism for capturing more speaker information from speech." />
        </attvalues>
      </edge>
      <edge source="2204.00990" target="2005.08484" id="6159">
        <attvalues>
          <attvalue for="5" value=" To make good use of reference speech, Attentron \cite{choi2020attentron} proposes an attention-based variable-length embedding method to leverage features near to raw reference speech for better generalization." />
        </attvalues>
      </edge>
      <edge source="2005.05716" target="1907.11692" id="6161">
        <attvalues>
          <attvalue for="5" value=" Contemporary machine learning that addresses text-related tasks adheres to the use of large language models---deep neural network architectures that have gone through extensive unsupervised pre-training in order to capture context-dependent meaning of individual tokens \cite{devlin2019bert,liu2019roberta,yang2019xlnet}." />
        </attvalues>
      </edge>
      <edge source="2005.05716" target="1705.07874" id="6163">
        <attvalues>
          <attvalue for="5" value=" Approaches to understanding black-box (non-interpretable) neural network models often resort to post-hoc approximations, eg, SHAP \cite{lundberg2017unified}, and similar are not necessary internal to the model itself." />
        </attvalues>
      </edge>
      <edge source="2005.05716" target="1409.0473" id="6164">
        <attvalues>
          <attvalue for="5" value=" A potential way of extracting the token relevance is the attention mechanism \cite{bahdanau;2014,luong2015effective}." />
        </attvalues>
      </edge>
      <edge source="2005.05716" target="1908.07442" id="6166">
        <attvalues>
          <attvalue for="5" value=" Similar findings were also recently discussed when considering tabular data \cite{arik2019tabnet}." />
        </attvalues>
      </edge>
      <edge source="2312.12764" target="1604.08242" id="6167">
        <attvalues>
          <attvalue for="5" value=" As described above, previous studies \cite{Saon_IS2016,Saon_IS2017,Li_CHiME5,Medennikov_CHiME6,Arora_CHiME6} have reported the effectiveness of combining a small number of NLMs (up to four \cite{Medennikov_CHiME6}) on lattice rescoring." />
        </attvalues>
      </edge>
      <edge source="2312.12764" target="1610.05256" id="6169">
        <attvalues>
          <attvalue for="5" value=" It has been reported that, by carrying over contextual information across ASR hypotheses, the rescoring performance for such a long ASR hypothesis sequence can be improved \cite{Xiong_arXiv2017,Xiong_ICASSP2018,Xiong_EMNLP2018,Parthasarathy_arXiv2019,Irie_ASRU2019,Zmolikova_CHiME6,Sun_ICASSP2021}." />
        </attvalues>
      </edge>
      <edge source="2312.12764" target="1708.06073" id="6170">
        <attvalues>
          <attvalue for="5" value=" It has been reported that, by carrying over contextual information across ASR hypotheses, the rescoring performance for such a long ASR hypothesis sequence can be improved \cite{Xiong_arXiv2017,Xiong_ICASSP2018,Xiong_EMNLP2018,Parthasarathy_arXiv2019,Irie_ASRU2019,Zmolikova_CHiME6,Sun_ICASSP2021}." />
        </attvalues>
      </edge>
      <edge source="2312.12764" target="2006.07898" id="6171">
        <attvalues>
          <attvalue for="5" value=" As described above, previous studies \cite{Saon_IS2016,Saon_IS2017,Li_CHiME5,Medennikov_CHiME6,Arora_CHiME6} have reported the effectiveness of combining a small number of NLMs (up to four \cite{Medennikov_CHiME6}) on lattice rescoring." />
        </attvalues>
      </edge>
      <edge source="2312.12764" target="1905.04226" id="6172">
        <attvalues>
          <attvalue for="5" value=" They have a non-recurrent self-attentive architecture that is completely different from that of the LSTMLMs, and they show comparable or superior rescoring performance to the LSTMLMs \cite{Irie_IS2019,Irie_ASRU2019,Sun_ICASSP2021}." />
        </attvalues>
      </edge>
      <edge source="2312.12764" target="2102.06474" id="6174">
        <attvalues>
          <attvalue for="5" value=" It has been reported that, by carrying over contextual information across ASR hypotheses, the rescoring performance for such a long ASR hypothesis sequence can be improved \cite{Xiong_arXiv2017,Xiong_ICASSP2018,Xiong_EMNLP2018,Parthasarathy_arXiv2019,Irie_ASRU2019,Zmolikova_CHiME6,Sun_ICASSP2021}." />
        </attvalues>
      </edge>
      <edge source="2312.12764" target="1706.03762" id="6175">
        <attvalues>
          <attvalue for="5" value=" In addition to the LSTMLMs, NLMs based on Transformers \cite{Vaswani_NIPS2017} have recently been used for rescoring." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2212.06522" id="6187">
        <attvalues>
          <attvalue for="5" value=" Relation Extraction (RE) aims to extract the relation between two entities \cite{qu2023distantly,gu2022delving} from an unstructured text \cite{cheng2021hacred}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2310.18342" id="6189">
        <attvalues>
          <attvalue for="5" value=" Given the significance of inter-entity relations within textual information, the practice of relation extraction finds extensive utility across various downstream tasks, including dialogue systems~\cite{lu2023miracle,liu2018knowledge}, information retrieval~\cite{yang2020biomedical,yu2023fusionint5}, information extraction \cite{zhu2023mirror,zhu2021efficient}, and question answering~\cite{yasunaga2021qa,qu2021passage}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2305.14685" id="6190">
        <attvalues>
          <attvalue for="5" value=" Given the significance of inter-entity relations within textual information, the practice of relation extraction finds extensive utility across various downstream tasks, including dialogue systems~\cite{lu2023miracle,liu2018knowledge}, information retrieval~\cite{yang2020biomedical,yu2023fusionint5}, information extraction \cite{zhu2023mirror,zhu2021efficient}, and question answering~\cite{yasunaga2021qa,qu2021passage}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2112.06013" id="6192">
        <attvalues>
          <attvalue for="5" value=" Given the significance of inter-entity relations within textual information, the practice of relation extraction finds extensive utility across various downstream tasks, including dialogue systems~\cite{lu2023miracle,liu2018knowledge}, information retrieval~\cite{yang2020biomedical,yu2023fusionint5}, information extraction \cite{zhu2023mirror,zhu2021efficient}, and question answering~\cite{yasunaga2021qa,qu2021passage}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2104.06378" id="6193">
        <attvalues>
          <attvalue for="5" value=" Given the significance of inter-entity relations within textual information, the practice of relation extraction finds extensive utility across various downstream tasks, including dialogue systems~\cite{lu2023miracle,liu2018knowledge}, information retrieval~\cite{yang2020biomedical,yu2023fusionint5}, information extraction \cite{zhu2023mirror,zhu2021efficient}, and question answering~\cite{yasunaga2021qa,qu2021passage}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2105.03938" id="6194">
        <attvalues>
          <attvalue for="5" value=" Given the significance of inter-entity relations within textual information, the practice of relation extraction finds extensive utility across various downstream tasks, including dialogue systems~\cite{lu2023miracle,liu2018knowledge}, information retrieval~\cite{yang2020biomedical,yu2023fusionint5}, information extraction \cite{zhu2023mirror,zhu2021efficient}, and question answering~\cite{yasunaga2021qa,qu2021passage}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2004.06870" id="6196">
        <attvalues>
          <attvalue for="5" value=" Following the emergence of the paradigm involving pre-trained models and fine-tuning for downstream tasks~\cite{kenton2019bert,radford2018improving}, many recent relation extraction studies have embraced the utilization of large language models~\cite{ye2020coreferential,soares2019matching,zhou2022improved,ye2022packed}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="1906.03158" id="6197">
        <attvalues>
          <attvalue for="5" value=" Following the emergence of the paradigm involving pre-trained models and fine-tuning for downstream tasks~\cite{kenton2019bert,radford2018improving}, many recent relation extraction studies have embraced the utilization of large language models~\cite{ye2020coreferential,soares2019matching,zhou2022improved,ye2022packed}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2102.01373" id="6198">
        <attvalues>
          <attvalue for="5" value=" Following the emergence of the paradigm involving pre-trained models and fine-tuning for downstream tasks~\cite{kenton2019bert,radford2018improving}, many recent relation extraction studies have embraced the utilization of large language models~\cite{ye2020coreferential,soares2019matching,zhou2022improved,ye2022packed}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2109.06067" id="6199">
        <attvalues>
          <attvalue for="5" value=" Following the emergence of the paradigm involving pre-trained models and fine-tuning for downstream tasks~\cite{kenton2019bert,radford2018improving}, many recent relation extraction studies have embraced the utilization of large language models~\cite{ye2020coreferential,soares2019matching,zhou2022improved,ye2022packed}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2012.15723" id="6200">
        <attvalues>
          <attvalue for="5" value=" Recently, prompt tuning has emerged as a promising direction for facilitating few-shot learning, which effectively bridges the gap between the pre-training and the downstream task~\cite{gao2021making,jin2023instance}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2201.07126" id="6201">
        <attvalues>
          <attvalue for="5" value=" Recently, prompt tuning has emerged as a promising direction for facilitating few-shot learning, which effectively bridges the gap between the pre-training and the downstream task~\cite{gao2021making,jin2023instance}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2105.11259" id="6202">
        <attvalues>
          <attvalue for="5" value=" To address this issue, previous work \cite{han2022ptr} applies logic rules to decompose complex relations into descriptions related to the subject and object entity types." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2104.07650" id="6203">
        <attvalues>
          <attvalue for="5" value=" Some works construct virtual words for each relation (a trainable ``$[relation_1]$&quot;) to substitute the corresponding answer space of the complex relation~\cite{chen2022knowprompt,chen2022relation}." />
        </attvalues>
      </edge>
      <edge source="2312.17267" target="2205.02355" id="6204">
        <attvalues>
          <attvalue for="5" value=" Some works construct virtual words for each relation (a trainable ``$[relation_1]$&quot;) to substitute the corresponding answer space of the complex relation~\cite{chen2022knowprompt,chen2022relation}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2106.07447" id="6222">
        <attvalues>
          <attvalue for="5" value=" Recent studies have shown that self-supervised fine-tuning of the pretrained HuBERT~\cite{9585401} naturally induces syllabic organization in an intermediate network layer~\cite{peng23e_interspeech,10446062}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2305.11435" id="6223">
        <attvalues>
          <attvalue for="5" value=" demonstrated that syllabic organization emerges in Visually-Grounded HuBERT (VG-HuBERT), which learns the co-occurrence of matched speech-image pairs~\cite{peng23e_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2310.10803" id="6224">
        <attvalues>
          <attvalue for="5" value=" Moreover, unlike~\cite{10446062}, we avoid using the CLS token and employ a frame-level training objective to prevent the aggregation of paralinguistic information." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2203.15081" id="6225">
        <attvalues>
          <attvalue for="5" value=" Some of those works have shown that hidden units obtained by discretizing learned features highly correlate with linguistic units, eg, phones~\cite{9585401}, syllables~\cite{peng23e_interspeech,10446062}, and words~\cite{peng22c_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2104.00355" id="6226">
        <attvalues>
          <attvalue for="5" value=" By utilizing them as pseudo-labels of untranscribed audio, we can train transcript-less models for various spoken language processing tasks, including speech synthesis~\cite{lakhotia-etal-2021-generative,polyak21_interspeech,10158503,zhang-etal-2023-speechgpt}, spoken language understanding~\cite{wu23g_interspeech,fang2024integrating}, speech-to-speech translation~\cite{lee-etal-2022-textless,huang2023transpeech}, and spoken language acquisition~\cite{10096250}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2209.03143" id="6227">
        <attvalues>
          <attvalue for="5" value=" By utilizing them as pseudo-labels of untranscribed audio, we can train transcript-less models for various spoken language processing tasks, including speech synthesis~\cite{lakhotia-etal-2021-generative,polyak21_interspeech,10158503,zhang-etal-2023-speechgpt}, spoken language understanding~\cite{wu23g_interspeech,fang2024integrating}, speech-to-speech translation~\cite{lee-etal-2022-textless,huang2023transpeech}, and spoken language acquisition~\cite{10096250}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2305.11000" id="6228">
        <attvalues>
          <attvalue for="5" value=" By utilizing them as pseudo-labels of untranscribed audio, we can train transcript-less models for various spoken language processing tasks, including speech synthesis~\cite{lakhotia-etal-2021-generative,polyak21_interspeech,10158503,zhang-etal-2023-speechgpt}, spoken language understanding~\cite{wu23g_interspeech,fang2024integrating}, speech-to-speech translation~\cite{lee-etal-2022-textless,huang2023transpeech}, and spoken language acquisition~\cite{10096250}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2305.18096" id="6229">
        <attvalues>
          <attvalue for="5" value=" By utilizing them as pseudo-labels of untranscribed audio, we can train transcript-less models for various spoken language processing tasks, including speech synthesis~\cite{lakhotia-etal-2021-generative,polyak21_interspeech,10158503,zhang-etal-2023-speechgpt}, spoken language understanding~\cite{wu23g_interspeech,fang2024integrating}, speech-to-speech translation~\cite{lee-etal-2022-textless,huang2023transpeech}, and spoken language acquisition~\cite{10096250}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2112.08352" id="6231">
        <attvalues>
          <attvalue for="5" value=" By utilizing them as pseudo-labels of untranscribed audio, we can train transcript-less models for various spoken language processing tasks, including speech synthesis~\cite{lakhotia-etal-2021-generative,polyak21_interspeech,10158503,zhang-etal-2023-speechgpt}, spoken language understanding~\cite{wu23g_interspeech,fang2024integrating}, speech-to-speech translation~\cite{lee-etal-2022-textless,huang2023transpeech}, and spoken language acquisition~\cite{10096250}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2205.12523" id="6232">
        <attvalues>
          <attvalue for="5" value=" By utilizing them as pseudo-labels of untranscribed audio, we can train transcript-less models for various spoken language processing tasks, including speech synthesis~\cite{lakhotia-etal-2021-generative,polyak21_interspeech,10158503,zhang-etal-2023-speechgpt}, spoken language understanding~\cite{wu23g_interspeech,fang2024integrating}, speech-to-speech translation~\cite{lee-etal-2022-textless,huang2023transpeech}, and spoken language acquisition~\cite{10096250}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2305.11072" id="6233">
        <attvalues>
          <attvalue for="5" value=" Following~\cite{chang23_interspeech}, to obtain speaker-invariant representations, we constrain a speech model to extract consistent features between the original speech and its speaker-perturbed version." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="2204.09224" id="6235">
        <attvalues>
          <attvalue for="5" value=" Some works have introduced speaker disentanglement for self-supervised training of speech models~\cite{pmlr-v162-qian22b,chang23_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2409.10103" target="1706.03762" id="6236">
        <attvalues>
          <attvalue for="5" value=" Finally, our ablation study reveals that the essential factor for syllabic organization is the use of higher Transformer~\cite{NIPS2017_3f5ee243} layers as the student's learning targets that correlate with linguistically coarse-grained units." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="1706.03762" id="6237">
        <attvalues>
          <attvalue for="5" value=" With the emergence of the Transformer architecture \cite{vaswani2017attention}, a significant breakthrough was achieved, enabling the effective retention of extensive long-range dependencies in tasks related to natural language processing, speech, and vision." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2005.14165" id="6238">
        <attvalues>
          <attvalue for="5" value=" The widespread adoption of LLMs on a substantial scale gained traction following the successful establishment of ChatGPT (including GPT-3 and subsequent iterations) \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2206.04615" id="6239">
        <attvalues>
          <attvalue for="5" value=" The Big-bench \cite{srivastava2022beyond} introduced over 200 benchmarks designed to assess the capabilities of Large Language Models(LLMs) through quantification and extrapolation." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2306.01116" id="6240">
        <attvalues>
          <attvalue for="5" value=" Despite the high cost of deploying and operating large language models (LLMs), the recent release of the Falcon \cite{penedo2023refinedweb} and Llama2 \cite{touvron2023llama} models has sparked optimism among small organizations and has increased their desire to deploy their own custom LLMs." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2307.09288" id="6241">
        <attvalues>
          <attvalue for="5" value=" Despite the high cost of deploying and operating large language models (LLMs), the recent release of the Falcon \cite{penedo2023refinedweb} and Llama2 \cite{touvron2023llama} models has sparked optimism among small organizations and has increased their desire to deploy their own custom LLMs." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2211.05102" id="6242">
        <attvalues>
          <attvalue for="5" value=" The efficient deployment of decoder only LLMs are challenging in practice because the generative inference proceeds sequentially, where the computation for each token depends on the previously generated tokens \cite{pope2023efficiently}." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2302.01318" id="6243">
        <attvalues>
          <attvalue for="5" value=" The application of quantization methods to transformers emerges as a efficacious approach for mitigating sampling latency, while incurring minimal to negligible impact on overall performance \cite{chen2023accelerating}." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2303.05295" id="6244">
        <attvalues>
          <attvalue for="5" value=" Quantization techniques can be mainly characterized into three forms namely - i) quantization aware training \cite{yang2023dynamic, liu2023llm}, ii) quantization aware fine-tuning \cite{dettmers2023qlora, kwon2022alphatuning, dettmers2022llm}, and iii) post training quantization (PTQ) \cite{frantar2022gptq, yuan2023rptq, lin2023awq}." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2305.14314" id="6245">
        <attvalues>
          <attvalue for="5" value=" Quantization techniques can be mainly characterized into three forms namely - i) quantization aware training \cite{yang2023dynamic, liu2023llm}, ii) quantization aware fine-tuning \cite{dettmers2023qlora, kwon2022alphatuning, dettmers2022llm}, and iii) post training quantization (PTQ) \cite{frantar2022gptq, yuan2023rptq, lin2023awq}." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="2210.17323" id="6246">
        <attvalues>
          <attvalue for="5" value=" \begin{enumerate} \item This study offers a systematic examination of the influence exerted by three pivotal hyper-parameters, namely, max new tokens, temperature, and top\_k, on LLMs that have undergone quantization through widely adopted post-training quantization techniques such as \cite{frantar2022gptq} (hereafter, gptq) and \cite{dettmers2022llm, dettmers2023qlora} (hereafter, bitsandbytes)." />
        </attvalues>
      </edge>
      <edge source="2309.05210" target="1609.07843" id="6247">
        <attvalues>
          <attvalue for="5" value=" The perplexity scores are computed on datasets such as Wiki \cite{merity2016pointer}, PTB \cite{marcus1993building}, and C4 \cite{raffel2020exploring}, which mostl likely have served as foundational datasets during the training of most of the LLMs." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="1907.11692" id="6250">
        <attvalues>
          <attvalue for="5" value=" Transformer architecture enabled efficient training of large-scale language models~\cite{radford2019language} and language understanding models~\cite{devlin-etal-2019-bert,liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="1310.4546" id="6251">
        <attvalues>
          <attvalue for="5" value=" On the other hand, transfer learning, which has been used in representation learning for years~\cite{mikolov2013distributed,pennington-etal-2014-glove,devlin-etal-2019-bert}, has finally been successfully applied to text-to-text problems as well~\cite{raffel2020exploring,lewis-etal-2020-bart}." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="1910.13461" id="6253">
        <attvalues>
          <attvalue for="5" value=" On the other hand, transfer learning, which has been used in representation learning for years~\cite{mikolov2013distributed,pennington-etal-2014-glove,devlin-etal-2019-bert}, has finally been successfully applied to text-to-text problems as well~\cite{raffel2020exploring,lewis-etal-2020-bart}." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="2001.08210" id="6254">
        <attvalues>
          <attvalue for="5" value=" Their T5 model was available only for English language, but more recently pre-trained multi-lingual architectures~\cite{liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,nagoudi-etal-2021-indt5} and non-English counterparts~\cite{carmo2020ptt5,Malaya} were released." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="2010.11934" id="6255">
        <attvalues>
          <attvalue for="5" value=" Their T5 model was available only for English language, but more recently pre-trained multi-lingual architectures~\cite{liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,nagoudi-etal-2021-indt5} and non-English counterparts~\cite{carmo2020ptt5,Malaya} were released." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="2104.07483" id="6256">
        <attvalues>
          <attvalue for="5" value=" Their T5 model was available only for English language, but more recently pre-trained multi-lingual architectures~\cite{liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,nagoudi-etal-2021-indt5} and non-English counterparts~\cite{carmo2020ptt5,Malaya} were released." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="2008.09144" id="6257">
        <attvalues>
          <attvalue for="5" value=" Their T5 model was available only for English language, but more recently pre-trained multi-lingual architectures~\cite{liu-etal-2020-multilingual-denoising,xue-etal-2021-mt5,nagoudi-etal-2021-indt5} and non-English counterparts~\cite{carmo2020ptt5,Malaya} were released." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="1912.05372" id="6259">
        <attvalues>
          <attvalue for="5" value=" Multiple publications show that models targeted for specific language perform better than multi-lingual one~\cite{martin-etal-2020-camembert,le-etal-2020-flaubert-unsupervised,chan-etal-2020-germans,mroczkowski-etal-2021-herbert,virtanen2019multilingual,nagoudi2021arat5}." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="2010.10906" id="6260">
        <attvalues>
          <attvalue for="5" value=" Multiple publications show that models targeted for specific language perform better than multi-lingual one~\cite{martin-etal-2020-camembert,le-etal-2020-flaubert-unsupervised,chan-etal-2020-germans,mroczkowski-etal-2021-herbert,virtanen2019multilingual,nagoudi2021arat5}." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="2105.01735" id="6261">
        <attvalues>
          <attvalue for="5" value=" Our contributions are: \begin{itemize} \item comprehensive evaluation of text-to-text models on diverse tasks in Polish, such as text-to-text KLEJ benchmark~\cite{rybak-etal-2020-klej}, machine translation, question answering and summarization, \item construction of benchmark datasets in the Polish domain for question answering and summarization, \item demonstration of the efficiency of pre-training procedure for transferring knowledge from multi-lingual to monolingual text-to-text models based on work by \cite{arkhipov-etal-2019-tuning,mroczkowski-etal-2021-herbert}, \item release of plT5 -- a T5-based model for the Polish language, which achieves the best results among the evaluated text-to-text models on KLEJ benchmark, machine translation and question answering and second-best results in summarization." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="1912.07076" id="6262">
        <attvalues>
          <attvalue for="5" value=" Multiple publications show that models targeted for specific language perform better than multi-lingual one~\cite{martin-etal-2020-camembert,le-etal-2020-flaubert-unsupervised,chan-etal-2020-germans,mroczkowski-etal-2021-herbert,virtanen2019multilingual,nagoudi2021arat5}." />
        </attvalues>
      </edge>
      <edge source="2205.08808" target="2005.00630" id="6263">
        <attvalues>
          <attvalue for="5" value=" Our contributions are: \begin{itemize} \item comprehensive evaluation of text-to-text models on diverse tasks in Polish, such as text-to-text KLEJ benchmark~\cite{rybak-etal-2020-klej}, machine translation, question answering and summarization, \item construction of benchmark datasets in the Polish domain for question answering and summarization, \item demonstration of the efficiency of pre-training procedure for transferring knowledge from multi-lingual to monolingual text-to-text models based on work by \cite{arkhipov-etal-2019-tuning,mroczkowski-etal-2021-herbert}, \item release of plT5 -- a T5-based model for the Polish language, which achieves the best results among the evaluated text-to-text models on KLEJ benchmark, machine translation and question answering and second-best results in summarization." />
        </attvalues>
      </edge>
      <edge source="2002.09836" target="1908.08960" id="6283">
        <attvalues>
          <attvalue for="5" value=" Improving human evaluation may require prompting labelers to pay higher attention \cite{Hardy2019HighRES}, as well as splitting quality scores into multiple dimensions such as fluency, informativeness, and factual correctness \cite{Wojciech2019Neural, Wojciech2017Evaluating, Lisa2018Robust}." />
        </attvalues>
      </edge>
      <edge source="2002.09836" target="1909.02622" id="6284">
        <attvalues>
          <attvalue for="5" value=" This deficiency may be partially addressable through measurement of the similarity not of text tokens but named entities or other preprocessed features \cite{Yuning2019Facet, Arman2016Revisiting, Fatma2015Keyphrase, Jun2015Better, Kavita2018ROUGE2} or embeddings \cite{Wei2019MoverScore, Tianyi2020BERTScore, Yang2020Supert}." />
        </attvalues>
      </edge>
      <edge source="2002.09836" target="1710.00284" id="6285">
        <attvalues>
          <attvalue for="5" value=" An overlap can be measured as well between summary and document text \cite{Liqun2017Efficient}." />
        </attvalues>
      </edge>
      <edge source="2002.09836" target="1909.08593" id="6286">
        <attvalues>
          <attvalue for="5" value=" Humans are also vulnerable to biases, such as the preference for phrases and sentences copied directly from the document text into summaries \cite{Daniel2020FineTuning}." />
        </attvalues>
      </edge>
      <edge source="2002.09836" target="1906.00318" id="6287">
        <attvalues>
          <attvalue for="5" value=" For example this might be achieved through a series of question-answers \cite{Matan2019Question, Ping2018SemanticQA, Thomas2015Answers}." />
        </attvalues>
      </edge>
      <edge source="2207.00555" target="2106.07447" id="6295">
        <attvalues>
          <attvalue for="5" value=" HuBERT \cite{hsu2021hubert} and wav2vec 2." />
        </attvalues>
      </edge>
      <edge source="2207.00555" target="2006.11477" id="6296">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} by introducing both KL-divergence and mean squared error (MSE) losses." />
        </attvalues>
      </edge>
      <edge source="2207.00555" target="1706.03762" id="6300">
        <attvalues>
          <attvalue for="5" value=" In \cite{sanh2019distilbert}, knowledge distillation is applied to BERT \cite{devlin2018bert}, one of the most prominent language representation model, by reducing the number of Transformer \cite{vaswani2017attention} layers and initializing with pre-trained BERT." />
        </attvalues>
      </edge>
      <edge source="2207.00555" target="1412.6550" id="6301">
        <attvalues>
          <attvalue for="5" value=" On the other hand, FitNets \cite{adriana2015fitnets} suggests thinner and deeper student than the teacher, matching not only the final outputs but also the intermediate representations as hints." />
        </attvalues>
      </edge>
      <edge source="2207.00555" target="2110.01900" id="6302">
        <attvalues>
          <attvalue for="5" value=" DistilHuBERT \cite{chang2021distilhubert} compresses 12 Transformer layers down to 2 by employing 3 distinct prediction heads." />
        </attvalues>
      </edge>
      <edge source="2207.00555" target="2103.15760" id="6303">
        <attvalues>
          <attvalue for="5" value=" In \cite{peng2021shrinking}, authors attempt to reduce the Transformer layers of wav2vec 2." />
        </attvalues>
      </edge>
      <edge source="2207.00555" target="2203.15610" id="6305">
        <attvalues>
          <attvalue for="5" value=" While developing our method, a concurrent work \cite{wang2022lighthubert} with a similar goal proposes a two-stage distillation strategy, making use of pre-training distillation and large-sized Transformer supernet with neural architecture search." />
        </attvalues>
      </edge>
      <edge source="2402.01643" target="2304.04370" id="6306">
        <attvalues>
          <attvalue for="5" value=" The advent of LLM has marked a significant milestone in NLP \cite{ge2023openagi}." />
        </attvalues>
      </edge>
      <edge source="2402.01643" target="2307.08303" id="6307">
        <attvalues>
          <attvalue for="5" value=" However, the effective utilization of LLMs often depends on fine-tuning techniques such as prompt or prefix tuning \cite{peng2023soft}." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="1905.00537" id="6310">
        <attvalues>
          <attvalue for="5" value="4 F1 score \cite{wang2020superglue}." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="2108.05542" id="6311">
        <attvalues>
          <attvalue for="5" value=" These Transformers are outperforming all previous methods and, sometimes, even humans in many NLP tasks \cite{wang-etal-2018-glue,wang2020superglue,AMMUS-T-PTLMs:2021,ThreatsPTLMs:2022}." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="1506.06724" id="6313">
        <attvalues>
          <attvalue for="5" value=" These plays may be included in the book dataset \cite{BookCorpusUsedInBERT} used for pre-training BERT." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="2107.02137" id="6314">
        <attvalues>
          <attvalue for="5" value=" The English version of the last ERNIE \cite{Sun2021ERNIE3L} is trained on an even more extensive corpus." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="1909.08053" id="6315">
        <attvalues>
          <attvalue for="5" value=" This possible shortcoming is sometimes considered when novel Transformers are introduced \cite{Radford2019LanguageMA,Shoeybi2019MegatronLMTM}." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="1811.01348" id="6316">
        <attvalues>
          <attvalue for="5" value=" Corpora and related tasks derived from the DeepWeb and DarkWeb \cite{info13090435,DBLP:conf/itasec/RanaldiNFZ22,avarikioti2018structure,Choshen2019TheLO} offer a tremendous opportunity to study Transformers and other natural language models on \totallyunseensentences." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="1905.05543" id="6317">
        <attvalues>
          <attvalue for="5" value=" Moreover, language on the DarkNet may have very different characteristics with respect to the one accessible from the surface web \cite{Choshen2019TheLO}." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="1805.03122" id="6318">
        <attvalues>
          <attvalue for="5" value=" We experimented with Stylistic Classifiers based on the bleaching text model \cite{van-der-goot-etal-2018-bleaching}, with Lexical Neural Networks based on GloVe \cite{pennington-etal-2014-glove} and word2vec \cite{mikolov2013efficient}, with Syntatic-based neural networks based on KERMIT \cite{zanzotto-etal-2020-kermit}, and with holistic Transformers such as BERT \cite{devlin-etal-2019-bert}, XLNet \cite{Yang2019XLNetGA}, ERNIE \cite{zhang2019ernie} and Electra \cite{clark2020electra}." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="2003.10555" id="6322">
        <attvalues>
          <attvalue for="5" value=" We experimented with Stylistic Classifiers based on the bleaching text model \cite{van-der-goot-etal-2018-bleaching}, with Lexical Neural Networks based on GloVe \cite{pennington-etal-2014-glove} and word2vec \cite{mikolov2013efficient}, with Syntatic-based neural networks based on KERMIT \cite{zanzotto-etal-2020-kermit}, and with holistic Transformers such as BERT \cite{devlin-etal-2019-bert}, XLNet \cite{Yang2019XLNetGA}, ERNIE \cite{zhang2019ernie} and Electra \cite{clark2020electra}." />
        </attvalues>
      </edge>
      <edge source="2201.05613" target="2106.09226" id="6323">
        <attvalues>
          <attvalue for="5" value=" Results show that syntactic and lexical neural networks surprisingly outperform pre-trained Transformers even after fine-tuning \cite{FineTuning:Wei2021WhyDP}." />
        </attvalues>
      </edge>
      <edge source="2210.04834" target="2005.14165" id="6326">
        <attvalues>
          <attvalue for="5" value=" Some studies have shown that these large models trained on generic corpora seem to be more robust to data distributional shifts, relying less on domain-specific training data to perform well \cite{NEURIPS2020_1457c0d6}." />
        </attvalues>
      </edge>
      <edge source="2210.04834" target="1503.02531" id="6327">
        <attvalues>
          <attvalue for="5" value=" In this paradigm, lightweight models referred to as students, are trained to mimic the teacher predictions over a transfer set \cite{hinton2015distilling}." />
        </attvalues>
      </edge>
      <edge source="2210.04834" target="1909.10351" id="6328">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation (referred to simply as distillation hereafter; \cite{hinton2015distilling}), has shown promising results, especially at the high compression rates typically required in NLU (\cite{jiao-etal-2020-tinybert}, \cite{soltan-etal-2021-limitations})." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2307.09288" id="6329">
        <attvalues>
          <attvalue for="5" value=" With the success of large language models \cite{touvron2023llama,achiam2023gpt,jiang2023mistral}, it has become increasingly important for language models (LMs) to handle instructions effectively for customized agents and tasks." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2303.08774" id="6330">
        <attvalues>
          <attvalue for="5" value=" With the success of large language models \cite{touvron2023llama,achiam2023gpt,jiang2023mistral}, it has become increasingly important for language models (LMs) to handle instructions effectively for customized agents and tasks." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2310.06825" id="6331">
        <attvalues>
          <attvalue for="5" value=" With the success of large language models \cite{touvron2023llama,achiam2023gpt,jiang2023mistral}, it has become increasingly important for language models (LMs) to handle instructions effectively for customized agents and tasks." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2005.14165" id="6332">
        <attvalues>
          <attvalue for="5" value=" Prompt engineering \cite{brown2020language,sanh2021multitask,chung2024scaling} involves crafting handcrafted prompts and faces the challenge of getting LMs to consistently produce desired outputs with few-shot instructions." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2110.08207" id="6333">
        <attvalues>
          <attvalue for="5" value=" Prompt engineering \cite{brown2020language,sanh2021multitask,chung2024scaling} involves crafting handcrafted prompts and faces the challenge of getting LMs to consistently produce desired outputs with few-shot instructions." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2210.11416" id="6334">
        <attvalues>
          <attvalue for="5" value=" Prompt engineering \cite{brown2020language,sanh2021multitask,chung2024scaling} involves crafting handcrafted prompts and faces the challenge of getting LMs to consistently produce desired outputs with few-shot instructions." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2205.05638" id="6336">
        <attvalues>
          <attvalue for="5" value=" Model fine-tuning \cite{raffel2020exploring} can perform very well for task-specific needs but requires explicit fine-tuning of a significant number of model parameters, even with parameter-efficient fine-tuning (PEFT) approaches \cite{liu2022few, hu2021lora}." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2101.00190" id="6337">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning (PT) \cite{li2021prefix,lester2021power,wen2024hard,shi2022toward,shin2020autoprompt,khashabi2021prompt} is a promising method that lies between prompt engineering and model fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2104.08691" id="6338">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning (PT) \cite{li2021prefix,lester2021power,wen2024hard,shi2022toward,shin2020autoprompt,khashabi2021prompt} is a promising method that lies between prompt engineering and model fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2302.03668" id="6339">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning (PT) \cite{li2021prefix,lester2021power,wen2024hard,shi2022toward,shin2020autoprompt,khashabi2021prompt} is a promising method that lies between prompt engineering and model fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2212.10539" id="6340">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning (PT) \cite{li2021prefix,lester2021power,wen2024hard,shi2022toward,shin2020autoprompt,khashabi2021prompt} is a promising method that lies between prompt engineering and model fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2010.15980" id="6341">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning (PT) \cite{li2021prefix,lester2021power,wen2024hard,shi2022toward,shin2020autoprompt,khashabi2021prompt} is a promising method that lies between prompt engineering and model fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2112.08348" id="6342">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning (PT) \cite{li2021prefix,lester2021power,wen2024hard,shi2022toward,shin2020autoprompt,khashabi2021prompt} is a promising method that lies between prompt engineering and model fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2205.11961" id="6343">
        <attvalues>
          <attvalue for="5" value=" Instead of handcrafting prompts, it optimizes a small number of prompt embeddings or indices with training data and has demonstrated capabilities comparable to those of model fine-tuning approaches \cite{asai2022attempt,shi2023dept,wang2023multitask}." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2309.05173" id="6344">
        <attvalues>
          <attvalue for="5" value=" While recent work \cite{shi2023dept} also explores low-rank matrices for prompt tuning, it restricts low-rankness to the differences or updates of a frozen baseline prompt, similar to the LoRA technique used in model fine-tuning \cite{hu2021lora}, and is only applied to a portion of the overall soft prompt." />
        </attvalues>
      </edge>
      <edge source="2406.19486" target="2106.09685" id="6346">
        <attvalues>
          <attvalue for="5" value=" While recent work \cite{shi2023dept} also explores low-rank matrices for prompt tuning, it restricts low-rankness to the differences or updates of a frozen baseline prompt, similar to the LoRA technique used in model fine-tuning \cite{hu2021lora}, and is only applied to a portion of the overall soft prompt." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1706.03762" id="6347">
        <attvalues>
          <attvalue for="5" value=" The Transformer architecture~\cite{Vaswani2017} became the backbone of the state-of-the-art models in a variety of tasks~\cite{Liu2019,Raffel2019,Adiwardana2020,Brown2020}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1907.11692" id="6348">
        <attvalues>
          <attvalue for="5" value=" The Transformer architecture~\cite{Vaswani2017} became the backbone of the state-of-the-art models in a variety of tasks~\cite{Liu2019,Raffel2019,Adiwardana2020,Brown2020}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2001.09977" id="6350">
        <attvalues>
          <attvalue for="5" value=" The Transformer architecture~\cite{Vaswani2017} became the backbone of the state-of-the-art models in a variety of tasks~\cite{Liu2019,Raffel2019,Adiwardana2020,Brown2020}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2005.14165" id="6351">
        <attvalues>
          <attvalue for="5" value=" The Transformer architecture~\cite{Vaswani2017} became the backbone of the state-of-the-art models in a variety of tasks~\cite{Liu2019,Raffel2019,Adiwardana2020,Brown2020}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1906.04284" id="6352">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{Vig2019} focussed on how attention align with specific syntactic dependency relations, \cite{Hupkes2020} considered if Transformers generalize compositionally and \cite{Kharitonov2020} studied how different models generalize from very few data." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1906.04341" id="6353">
        <attvalues>
          <attvalue for="5" value=" This spurred a significant interest in better understanding inner workings of these models~\cite{Vig2019,Clark2019,Kharitonov2020,Hahn2020,Movva2020,Chaabouni2021,Merrill2021,sinha2021masked}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2006.14953" id="6354">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{Vig2019} focussed on how attention align with specific syntactic dependency relations, \cite{Hupkes2020} considered if Transformers generalize compositionally and \cite{Kharitonov2020} studied how different models generalize from very few data." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1906.06755" id="6355">
        <attvalues>
          <attvalue for="5" value=" This spurred a significant interest in better understanding inner workings of these models~\cite{Vig2019,Clark2019,Kharitonov2020,Hahn2020,Movva2020,Chaabouni2021,Merrill2021,sinha2021masked}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2009.13270" id="6356">
        <attvalues>
          <attvalue for="5" value=" This spurred a significant interest in better understanding inner workings of these models~\cite{Vig2019,Clark2019,Kharitonov2020,Hahn2020,Movva2020,Chaabouni2021,Merrill2021,sinha2021masked}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2107.01366" id="6357">
        <attvalues>
          <attvalue for="5" value=" This spurred a significant interest in better understanding inner workings of these models~\cite{Vig2019,Clark2019,Kharitonov2020,Hahn2020,Movva2020,Chaabouni2021,Merrill2021,sinha2021masked}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2010.09697" id="6358">
        <attvalues>
          <attvalue for="5" value=" This spurred a significant interest in better understanding inner workings of these models~\cite{Vig2019,Clark2019,Kharitonov2020,Hahn2020,Movva2020,Chaabouni2021,Merrill2021,sinha2021masked}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2104.06644" id="6359">
        <attvalues>
          <attvalue for="5" value=" This spurred a significant interest in better understanding inner workings of these models~\cite{Vig2019,Clark2019,Kharitonov2020,Hahn2020,Movva2020,Chaabouni2021,Merrill2021,sinha2021masked}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1909.01066" id="6361">
        <attvalues>
          <attvalue for="5" value=" First, large Transformer models are increasingly often used as a storage, for instance, as a general-purpose knowledge base or as a closed-book question-answering system~\cite{Petroni2019,Roberts2020,Lewis2020}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2002.08910" id="6362">
        <attvalues>
          <attvalue for="5" value=" First, large Transformer models are increasingly often used as a storage, for instance, as a general-purpose knowledge base or as a closed-book question-answering system~\cite{Petroni2019,Roberts2020,Lewis2020}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2008.02637" id="6363">
        <attvalues>
          <attvalue for="5" value=" First, large Transformer models are increasingly often used as a storage, for instance, as a general-purpose knowledge base or as a closed-book question-answering system~\cite{Petroni2019,Roberts2020,Lewis2020}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1911.00172" id="6364">
        <attvalues>
          <attvalue for="5" value=" There are even Transformers models that are explicitly endowed with an external training data memorization mechanism~\cite{Khandelwal2020,Khandelwal2021,He2021}, demonstrating that further boosting their memorization abilities is beneficial." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2010.00710" id="6365">
        <attvalues>
          <attvalue for="5" value=" There are even Transformers models that are explicitly endowed with an external training data memorization mechanism~\cite{Khandelwal2020,Khandelwal2021,He2021}, demonstrating that further boosting their memorization abilities is beneficial." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2109.04212" id="6366">
        <attvalues>
          <attvalue for="5" value=" There are even Transformers models that are explicitly endowed with an external training data memorization mechanism~\cite{Khandelwal2020,Khandelwal2021,He2021}, demonstrating that further boosting their memorization abilities is beneficial." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2012.07805" id="6367">
        <attvalues>
          <attvalue for="5" value=" Second, in contrast, the same ability to memorize can become undesirable and lead to a leakage of personal data from trained models~\cite{Carlini2020,Thakkar2021}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="2108.05885" id="6368">
        <attvalues>
          <attvalue for="5" value=" For instance, successful language models need to generalize to be able to deal with never-seen-before sentences, but they also need to {memorize} the spelling of words, the non-compositional meaning of idioms, idiosyncrasies of languages, common knowledge, etc \cite[see, eg][]{dankers2021paradox}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1611.03530" id="6369">
        <attvalues>
          <attvalue for="5" value=" Intuitively, the number of parameters, data augmentation, and regularization are likely to affect how successful models are in memorization~\cite{Zhang2016,Sablayrolles2018}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1508.07909" id="6370">
        <attvalues>
          <attvalue for="5" value=" A very common approach is to learn subword-level vocabulary with Byte-Pair Encoding (BPE)~\cite{Sennrich2015} or similar methods~\cite[eg,][]{Devlin2018,Kudo2018,Provilkov2019}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1808.06226" id="6372">
        <attvalues>
          <attvalue for="5" value=" A very common approach is to learn subword-level vocabulary with Byte-Pair Encoding (BPE)~\cite{Sennrich2015} or similar methods~\cite[eg,][]{Devlin2018,Kudo2018,Provilkov2019}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1910.13267" id="6373">
        <attvalues>
          <attvalue for="5" value=" A very common approach is to learn subword-level vocabulary with Byte-Pair Encoding (BPE)~\cite{Sennrich2015} or similar methods~\cite[eg,][]{Devlin2018,Kudo2018,Provilkov2019}." />
        </attvalues>
      </edge>
      <edge source="2110.02782" target="1610.05820" id="6374">
        <attvalues>
          <attvalue for="5" value=" For that, we study how easy it is to accurately tell if a particular example was used in the model's training data via a membership inference attack~\cite{Shokri2017}." />
        </attvalues>
      </edge>
      <edge source="2407.16431" target="1812.06135" id="6375">
        <attvalues>
          <attvalue for="5" value=" In Natural Language Processing (NLP), similar to many machine learning domains, bias mitigation generally occurs at three intervention avenues: the training data, the learning procedure, or the model output \cite{lohia2019bias}." />
        </attvalues>
      </edge>
      <edge source="2407.16431" target="1901.09451" id="6376">
        <attvalues>
          <attvalue for="5" value=" Since model bias traces its roots to the training data, mitigating bias at the training data level has proven very effective \cite{dixon2018measuring,de2019bias}." />
        </attvalues>
      </edge>
      <edge source="2407.16431" target="1807.11714" id="6377">
        <attvalues>
          <attvalue for="5" value=" Key works, such as \cite{zhao-etal-2018-gender,lu2020gender,zmigrod-etal-2019-counterfactual}, introducing CDA as a bias mitigation technique adopt a word substitution approach based on dictionaries." />
        </attvalues>
      </edge>
      <edge source="2407.16431" target="1804.06876" id="6378">
        <attvalues>
          <attvalue for="5" value=" Key works, such as \cite{zhao-etal-2018-gender,lu2020gender,zmigrod-etal-2019-counterfactual}, introducing CDA as a bias mitigation technique adopt a word substitution approach based on dictionaries." />
        </attvalues>
      </edge>
      <edge source="2407.16431" target="1906.04571" id="6379">
        <attvalues>
          <attvalue for="5" value=" Key works, such as \cite{zhao-etal-2018-gender,lu2020gender,zmigrod-etal-2019-counterfactual}, introducing CDA as a bias mitigation technique adopt a word substitution approach based on dictionaries." />
        </attvalues>
      </edge>
      <edge source="2407.16431" target="1911.03842" id="6380">
        <attvalues>
          <attvalue for="5" value=" Because dictionary compilations are often incomplete \cite{dinan2020queens}, a direct word-substitution approach will not generalize to omitted words." />
        </attvalues>
      </edge>
      <edge source="2407.16431" target="1604.02201" id="6381">
        <attvalues>
          <attvalue for="5" value=" This challenge is exacerbated by the fact that training models on limited parallel data can impair performance \cite{zoph2016transfer}." />
        </attvalues>
      </edge>
      <edge source="2407.16431" target="1410.8516" id="6382">
        <attvalues>
          <attvalue for="5" value=" Using an invertible flow-based model \cite{dinh2014nice}, counterfactual words are generated for sampled words." />
        </attvalues>
      </edge>
      <edge source="2206.04935" target="2010.01150" id="6384">
        <attvalues>
          <attvalue for="5" value=" Typically, better expressivity is expected from language/domain-specific LMs \cite{gururangan-etal-2020-dont,dai-etal-2020-cost} while open-domain settings necessitate high-capacity models with access to as much pre-training data as possible." />
        </attvalues>
      </edge>
      <edge source="2206.04935" target="1906.02243" id="6385">
        <attvalues>
          <attvalue for="5" value=" In absence of immediate performance indicators, the most accurate choice could be made by training the full model using each LM candidate, however this is often infeasible and wasteful~\cite{strubell-etal-2019-energy}." />
        </attvalues>
      </edge>
      <edge source="2206.04935" target="2002.12462" id="6386">
        <attvalues>
          <attvalue for="5" value=" Recently, the field of Computer Vision (CV) has attempted to tackle this problem by quantifying useful information in pre-trained image encoders as measured directly on labeled target data without fine-tuning \cite{nguyen2020,you2021}." />
        </attvalues>
      </edge>
      <edge source="2206.04935" target="2102.11005" id="6387">
        <attvalues>
          <attvalue for="5" value=" Recently, the field of Computer Vision (CV) has attempted to tackle this problem by quantifying useful information in pre-trained image encoders as measured directly on labeled target data without fine-tuning \cite{nguyen2020,you2021}." />
        </attvalues>
      </edge>
      <edge source="2206.04935" target="2005.00870" id="6388">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, performance prediction in NLP has so far been studied as a function of dataset and model characteristics~\cite{xia-etal-2020-predicting,ye-etal-2021-towards} and has yet to examine how to rank large pools of pre-trained LMs." />
        </attvalues>
      </edge>
      <edge source="2206.04935" target="2102.05486" id="6389">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, performance prediction in NLP has so far been studied as a function of dataset and model characteristics~\cite{xia-etal-2020-predicting,ye-etal-2021-towards} and has yet to examine how to rank large pools of pre-trained LMs." />
        </attvalues>
      </edge>
      <edge source="2206.04935" target="2203.12971" id="6390">
        <attvalues>
          <attvalue for="5" value=" Given the closely related field of probing, in which lightweight models quantify task-specific information in pre-trained LMs, we recast its objective in the context of performance prediction and ask: How predictive is lightweight probing at choosing the best performing LM for dependency parsing? To answer this question, we contribute: \begin{itemize} \item An efficient encoder ranking method for structured prediction using dependency probing (\cite{depprobe}; DepProbe) to quantify latent syntax (Section \ref{sec:method})." />
        </attvalues>
      </edge>
      <edge source="2206.04935" target="2010.12821" id="6391">
        <attvalues>
          <attvalue for="5" value=" \item An in-depth analysis of the surprisingly low inherent dependency information in RemBERT ~\cite{chung2021rethinking} compared to its high fine-tuned performance (Section \ref{sec:rembert-analysis})." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="2011.04946" id="6392">
        <attvalues>
          <attvalue for="5" value=" One of the most challenging aspects of working with large language models (LLMs) is their computational complexity \cite{zhang_when_2021}." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="2209.00099" id="6393">
        <attvalues>
          <attvalue for="5" value=" With 340M parameters, even the BERT-large model is impractical for deployment on low-end devices with inadequate computational power \cite{treviso2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="1909.10351" id="6394">
        <attvalues>
          <attvalue for="5" value=" \cite{Jiao2019TinyBERTDB} proposed a transformer-specific distillation method, employing a two-stage learning framework with general and task-specific distillation using BERT." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="2103.12028" id="6396">
        <attvalues>
          <attvalue for="5" value=" The omission of African languages from the pre-training phase of LLMs results in low performance in these languages, making NLP tasks participatory difficult \cite{kreutzer_quality_2022}." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="2110.03036" id="6397">
        <attvalues>
          <attvalue for="5" value=" \cite{ahia2021low} termed this situation the &quot;low-resource double-bind&quot; to describe the coexistence of data and computation limitations on resources." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="1911.02116" id="6399">
        <attvalues>
          <attvalue for="5" value=" AfriBERTa \cite{ogueji-etal-2021-small} beats competitive models like mBERT \cite{devlin_bert_2019} and XLM-R \cite{conneau2020unsupervised} on text categorization and NER tasks." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="1503.02531" id="6400">
        <attvalues>
          <attvalue for="5" value=" Our contributions address the following questions: \begin{enumerate} \item How tiny can we construct a small-data model using the knowledge distillation framework? \item What are the efficiency and generalization limits of pruning on a small-data model? \item What are the optimal reductions we can achieve in size and latency utilizing quantization? \end{enumerate} Related work First introduced by \cite{Hinton2015DistillingTK}, \cite{Sanh2019DistilBERTAD} demonstrated similar performances on downstream tasks with smaller language models pre-trained using distillation, which is faster at inference and suitable for edge devices." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="1906.02768" id="6403">
        <attvalues>
          <attvalue for="5" value="&quot; \cite{yu2019playing} and \cite{renda2020comparing} also found winning tickets early in training for Transformers and LSTMs." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="2003.02389" id="6404">
        <attvalues>
          <attvalue for="5" value="&quot; \cite{yu2019playing} and \cite{renda2020comparing} also found winning tickets early in training for Transformers and LSTMs." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="2109.15082" id="6407">
        <attvalues>
          <attvalue for="5" value=" \cite{bai2022towards} also introduced post-training quantization for language models, minimizing training time, memory, and data consumption, while \cite{wang2022deep} achieved 16$\times$ compression by quantizing transformer backbones to 4-bit and applying 50\% fine-grained structural sparsity." />
        </attvalues>
      </edge>
      <edge source="2404.04759" target="2211.10438" id="6408">
        <attvalues>
          <attvalue for="5" value=" Additionally, \cite{xiao2022smoothquant} enabled 8-bit weight, 8-bit activation quantization for large language models, addressing activation outliers, and \cite{dettmers2022llm} used LLM." />
        </attvalues>
      </edge>
      <edge source="2011.14277" target="1802.05365" id="6409">
        <attvalues>
          <attvalue for="5" value=" Recent years witnessed much success achieved by pre-trained LMs in the field of Natural Language Processing \cite{peters2018deep,devlin2019bert}." />
        </attvalues>
      </edge>
      <edge source="2011.14277" target="1901.05287" id="6411">
        <attvalues>
          <attvalue for="5" value=" Among the linguistic knowledge, syntax is broadly explored across sensitive structures \cite{goldberg2019assessing}, grammatical correctness \cite{marvin2018targeted}, and parsing dependencies \cite{hewitt2019structural}." />
        </attvalues>
      </edge>
      <edge source="2011.14277" target="1808.09031" id="6412">
        <attvalues>
          <attvalue for="5" value=" Among the linguistic knowledge, syntax is broadly explored across sensitive structures \cite{goldberg2019assessing}, grammatical correctness \cite{marvin2018targeted}, and parsing dependencies \cite{hewitt2019structural}." />
        </attvalues>
      </edge>
      <edge source="2011.14277" target="1911.11641" id="6414">
        <attvalues>
          <attvalue for="5" value=" In addition to the linguistics, tasks on common sense and facts are also introduced to test models on memorizing real-world knowledge during pre-training \cite{bisk2019piqa,zhou2019evaluating,petroni2019language}." />
        </attvalues>
      </edge>
      <edge source="2011.14277" target="1911.11931" id="6415">
        <attvalues>
          <attvalue for="5" value=" In addition to the linguistics, tasks on common sense and facts are also introduced to test models on memorizing real-world knowledge during pre-training \cite{bisk2019piqa,zhou2019evaluating,petroni2019language}." />
        </attvalues>
      </edge>
      <edge source="2011.14277" target="1909.01066" id="6416">
        <attvalues>
          <attvalue for="5" value=" In addition to the linguistics, tasks on common sense and facts are also introduced to test models on memorizing real-world knowledge during pre-training \cite{bisk2019piqa,zhou2019evaluating,petroni2019language}." />
        </attvalues>
      </edge>
      <edge source="2011.14277" target="2004.05986" id="6417">
        <attvalues>
          <attvalue for="5" value=" In the experiments, we test not only off-the-shelf models from CLUE project \cite{xu2020clue}, but also four BERT variants granted with different training objectives that mimic BERT, RoBERTa, SpanBERT, and ALBERT." />
        </attvalues>
      </edge>
      <edge source="2005.02693" target="1904.03396" id="6418">
        <attvalues>
          <attvalue for="5" value=" Improved performance would facilitate investigation of more complex versions of the shallow task, such as the deep task in which function words are pruned from the tree, which may be of more practical use in pipeline natural language generation (NLG) systems \cite{moryossef-etal-2019-step, elder-etal-2019-designing, castro-ferreira-etal-2019-neural}." />
        </attvalues>
      </edge>
      <edge source="2005.02693" target="1805.07731" id="6419">
        <attvalues>
          <attvalue for="5" value=" Unlike in the 2018 shared task, where a system trained with synthetic data performed roughly the same as a system trained on the original dataset \cite{elder_srst_2018, king_white_srst_2018}, we find its use leads to a large improvement in performance." />
        </attvalues>
      </edge>
      <edge source="2005.02693" target="1503.00075" id="6420">
        <attvalues>
          <attvalue for="5" value=" For instance, the system which was the clear winner of this year's shared task \cite{ims_srst_2019} used tree-structured long short-term memory (LSTM) networks \cite{tai_tree_lstm_2015}." />
        </attvalues>
      </edge>
      <edge source="2005.02693" target="1704.08381" id="6422">
        <attvalues>
          <attvalue for="5" value=" Though their system outperformed \cite{neural_amr_konstas_ACL_2017} at equivalent levels of additional training sentences, it was unable to scale up to the 20 million sentences used by the best \cite{neural_amr_konstas_ACL_2017} system and ultimately did not outperform them." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2303.18223" id="6423">
        <attvalues>
          <attvalue for="5" value=" Foundation LMs encompass three primary categories: Pre-trained Language Models (PLMs) \cite{min2023recent}, Large Language Models (LLMs) \cite{zhao2023survey}, and Vision-Language Models (VLMs) \cite{ijcai2022p762}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2111.01243" id="6424">
        <attvalues>
          <attvalue for="5" value=" Foundation LMs encompass three primary categories: Pre-trained Language Models (PLMs) \cite{min2023recent}, Large Language Models (LLMs) \cite{zhao2023survey}, and Vision-Language Models (VLMs) \cite{ijcai2022p762}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2302.10035" id="6425">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in foundation language models (LMs) have set new benchmarks in both natural language processing (NLP) \cite{zhao2023survey,min2023recent,zhou2023chatgpt} and computer vision (CV) \cite{wang2023large, tiwari2022gcr, ramasesh2022effect}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2202.10936" id="6426">
        <attvalues>
          <attvalue for="5" value=" Foundation LMs encompass three primary categories: Pre-trained Language Models (PLMs) \cite{min2023recent}, Large Language Models (LLMs) \cite{zhao2023survey}, and Vision-Language Models (VLMs) \cite{ijcai2022p762}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="1612.00796" id="6427">
        <attvalues>
          <attvalue for="5" value=" A key issue is ``catastrophic forgetting&quot; \cite{kirkpatrick2017overcoming}, where models lose previously learned knowledge when adapting to new information." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="1909.08383" id="6428">
        <attvalues>
          <attvalue for="5" value=" Unlike human learning, which is inherently continuous and adaptive \cite{de2021continual}, foundation LMs generally require retraining to incorporate new data." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2302.00487" id="6429">
        <attvalues>
          <attvalue for="5" value="} Continual learning (CL) \cite{wang2023comprehensive,van2022three}, also known as lifelong learning \cite{parisi2019continual} or incremental learning \cite{zhou2023deep}, offers an effective solution to these challenges." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="1802.07569" id="6430">
        <attvalues>
          <attvalue for="5" value="} Continual learning (CL) \cite{wang2023comprehensive,van2022three}, also known as lifelong learning \cite{parisi2019continual} or incremental learning \cite{zhou2023deep}, offers an effective solution to these challenges." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2112.09153" id="6431">
        <attvalues>
          <attvalue for="5" value="} Recent advancements in CL methodologies have substantially enhanced the adaptability and knowledge retention capabilities of foundation LMs \cite{mehta2021empirical,cossu2022continual,lee2023pre}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2205.09357" id="6432">
        <attvalues>
          <attvalue for="5" value="} Recent advancements in CL methodologies have substantially enhanced the adaptability and knowledge retention capabilities of foundation LMs \cite{mehta2021empirical,cossu2022continual,lee2023pre}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2112.02714" id="6434">
        <attvalues>
          <attvalue for="5" value=" {Notable successes have been documented in diverse downstream tasks, such as aspect-based sentiment analysis \cite{ke2021classic}, dialogue generation \cite{scialom2022fine}, text classification \cite{razdaibiedina2023progressive}, visual question answering \cite{zhang2023vqacl,qian2023decouple} and so on}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2205.12393" id="6435">
        <attvalues>
          <attvalue for="5" value=" {Notable successes have been documented in diverse downstream tasks, such as aspect-based sentiment analysis \cite{ke2021classic}, dialogue generation \cite{scialom2022fine}, text classification \cite{razdaibiedina2023progressive}, visual question answering \cite{zhang2023vqacl,qian2023decouple} and so on}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2301.12314" id="6436">
        <attvalues>
          <attvalue for="5" value=" {Notable successes have been documented in diverse downstream tasks, such as aspect-based sentiment analysis \cite{ke2021classic}, dialogue generation \cite{scialom2022fine}, text classification \cite{razdaibiedina2023progressive}, visual question answering \cite{zhang2023vqacl,qian2023decouple} and so on}." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2403.14608" id="6438">
        <attvalues>
          <attvalue for="5" value=" Second, due to the substantial number of parameters in foundation LMs, it is crucial to employ parameter-efficient techniques \cite{han2024parameter}, such as prompt tuning \cite{liu2022p} and adapters \cite{mundra2024comprehensive}, to update parameters without comprehensive retraining." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2305.07491" id="6439">
        <attvalues>
          <attvalue for="5" value=" Second, due to the substantial number of parameters in foundation LMs, it is crucial to employ parameter-efficient techniques \cite{han2024parameter}, such as prompt tuning \cite{liu2022p} and adapters \cite{mundra2024comprehensive}, to update parameters without comprehensive retraining." />
        </attvalues>
      </edge>
      <edge source="2405.18653" target="2301.00234" id="6440">
        <attvalues>
          <attvalue for="5" value=" Third, the foundation LMs possess the capability to follow instructions through instructional learning \cite{dong2022survey,ouyang2022training}, enabling more dynamic and context-aware interactions." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="1901.07931" id="6444">
        <attvalues>
          <attvalue for="5" value=" To validate the effectiveness of our approach, dubbed PINS~(\underline{P}runing with principled \underline{I}mportance a\underline{N}d \underline{S}elf-regularization), we conducted extensive experiments with various pre-trained language models on a wide variety of tasks, including natural language understanding on GLUE~\cite{glue}), question answering on SQuAD~\cite{squad}, named entity recognition on CoNLL 2003~\cite{conll2003}, and data-to-text generation on WebNLG~\cite{webnlg}, DART~\cite{dart}, and E2E~\cite{e2e}." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="2007.02871" id="6445">
        <attvalues>
          <attvalue for="5" value=" To validate the effectiveness of our approach, dubbed PINS~(\underline{P}runing with principled \underline{I}mportance a\underline{N}d \underline{S}elf-regularization), we conducted extensive experiments with various pre-trained language models on a wide variety of tasks, including natural language understanding on GLUE~\cite{glue}), question answering on SQuAD~\cite{squad}, named entity recognition on CoNLL 2003~\cite{conll2003}, and data-to-text generation on WebNLG~\cite{webnlg}, DART~\cite{dart}, and E2E~\cite{e2e}." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="1908.09355" id="6446">
        <attvalues>
          <attvalue for="5" value=" To resolve the above challenge, model compression~\cite{pkd,svd,albert} has been actively studied to make PLMs meet the practical requirement." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="1909.11942" id="6447">
        <attvalues>
          <attvalue for="5" value=" To resolve the above challenge, model compression~\cite{pkd,svd,albert} has been actively studied to make PLMs meet the practical requirement." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="2003.02389" id="6449">
        <attvalues>
          <attvalue for="5" value=" Prevalent importance criteria are based on the parameter's magnitude~\cite{gupta,mag2} or sensitivity~\cite{l0,movement,superticket,platon}." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="2005.07683" id="6450">
        <attvalues>
          <attvalue for="5" value=" Prevalent importance criteria are based on the parameter's magnitude~\cite{gupta,mag2} or sensitivity~\cite{l0,movement,superticket,platon}." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="2105.12002" id="6451">
        <attvalues>
          <attvalue for="5" value=" Prevalent importance criteria are based on the parameter's magnitude~\cite{gupta,mag2} or sensitivity~\cite{l0,movement,superticket,platon}." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="2206.12562" id="6452">
        <attvalues>
          <attvalue for="5" value=" Prevalent importance criteria are based on the parameter's magnitude~\cite{gupta,mag2} or sensitivity~\cite{l0,movement,superticket,platon}." />
        </attvalues>
      </edge>
      <edge source="2305.12394" target="cs/0306050" id="6453">
        <attvalues>
          <attvalue for="5" value=" To validate the effectiveness of our approach, dubbed PINS~(\underline{P}runing with principled \underline{I}mportance a\underline{N}d \underline{S}elf-regularization), we conducted extensive experiments with various pre-trained language models on a wide variety of tasks, including natural language understanding on GLUE~\cite{glue}), question answering on SQuAD~\cite{squad}, named entity recognition on CoNLL 2003~\cite{conll2003}, and data-to-text generation on WebNLG~\cite{webnlg}, DART~\cite{dart}, and E2E~\cite{e2e}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2305.07243" id="6454">
        <attvalues>
          <attvalue for="5" value=" In contrast to prior systems like TorToise TTS \cite{DBLP:journals/corr/abs-2305-07243}, which employs an LLM in conjunction with a denoising diffusion probabilistic models (DDPM)~\cite{DBLP:conf/nips/HoJA20}, CosyVoice utilizes a conditional flow matching approach, as it has been demonstrated to accelerate both training and inference compared to traditional diffusion models \cite{le2024voicebox}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2301.02111" id="6455">
        <attvalues>
          <attvalue for="5" value=" At the forefront of this advancement are Large Language Models (LLMs), which have been increasingly utilized in TTS systems to generate speech with a higher degree of naturalness and the ability to synthesize voices in a zero-shot fashion \cite{DBLP:journals/corr/abs-2305-07243,DBLP:journals/corr/abs-2301-02111,DBLP:journals/corr/abs-2402-08093}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2402.08093" id="6456">
        <attvalues>
          <attvalue for="5" value=" At the forefront of this advancement are Large Language Models (LLMs), which have been increasingly utilized in TTS systems to generate speech with a higher degree of naturalness and the ability to synthesize voices in a zero-shot fashion \cite{DBLP:journals/corr/abs-2305-07243,DBLP:journals/corr/abs-2301-02111,DBLP:journals/corr/abs-2402-08093}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2010.05646" id="6457">
        <attvalues>
          <attvalue for="5" value=" A token vocoder is then employed to reconstruct the raw waveforms from the tokenized speech \cite{DBLP:conf/nips/KongKB20, DBLP:journals/corr/abs-2210-13438}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2106.07447" id="6458">
        <attvalues>
          <attvalue for="5" value=" Traditionally, tokens are acquired through unsupervised learning, which may not capture explicit semantic information or align well with corresponding text \cite{DBLP:journals/taslp/HsuBTLSM21,DBLP:journals/corr/abs-2210-13438}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2210.13438" id="6459">
        <attvalues>
          <attvalue for="5" value=" Traditionally, tokens are acquired through unsupervised learning, which may not capture explicit semantic information or align well with corresponding text \cite{DBLP:journals/taslp/HsuBTLSM21,DBLP:journals/corr/abs-2210-13438}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2306.12925" id="6461">
        <attvalues>
          <attvalue for="5" value=" Early studies have shown that quantizers with auxiliary automatic speech recognition (ASR) loss outperform k-means clustering on the universal speech model (USM) for speech-to-text translation and ASR tasks, as demonstrated in \cite{DBLP:journals/corr/abs-2306-12925}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2006.11239" id="6462">
        <attvalues>
          <attvalue for="5" value=" In contrast to prior systems like TorToise TTS \cite{DBLP:journals/corr/abs-2305-07243}, which employs an LLM in conjunction with a denoising diffusion probabilistic models (DDPM)~\cite{DBLP:conf/nips/HoJA20}, CosyVoice utilizes a conditional flow matching approach, as it has been demonstrated to accelerate both training and inference compared to traditional diffusion models \cite{le2024voicebox}." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2306.15687" id="6463">
        <attvalues>
          <attvalue for="5" value=" While existing methods incorporate flow matching in TTS \cite{le2024voicebox, DBLP:journals/corr/abs-2309-05027,DBLP:journals/corr/abs-2309-03199,DBLP:journals/corr/abs-2309-17056}, they often rely on phoneme duration prediction, necessitating the use of supplementary phonemizers and forced aligners." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2309.17056" id="6465">
        <attvalues>
          <attvalue for="5" value=" While existing methods incorporate flow matching in TTS \cite{le2024voicebox, DBLP:journals/corr/abs-2309-05027,DBLP:journals/corr/abs-2309-03199,DBLP:journals/corr/abs-2309-17056}, they often rely on phoneme duration prediction, necessitating the use of supplementary phonemizers and forced aligners." />
        </attvalues>
      </edge>
      <edge source="2407.05407" target="2207.12598" id="6466">
        <attvalues>
          <attvalue for="5" value=" We optimize the flow matching process with techniques such as classifier-free guidance \cite{DBLP:journals/corr/abs-2207-12598}, a cosine scheduler, and masked conditions." />
        </attvalues>
      </edge>
      <edge source="2004.12506" target="1802.05365" id="6467">
        <attvalues>
          <attvalue for="5" value=" Recent progress in NLP has been marked with the emergence of large-scale pre-trained models, eg, ELMo~\cite{elmo}, BERT~\cite{bert}, and GPT-2~\cite{gpt2}." />
        </attvalues>
      </edge>
      <edge source="2004.12506" target="1901.08149" id="6469">
        <attvalues>
          <attvalue for="5" value=" Indeed, GPT-based language models have shown impressive results for open-domain dialogue generation \cite{golovanov2019large,wolf2019transfertransfo,dialogpt}." />
        </attvalues>
      </edge>
      <edge source="2004.12506" target="1911.00536" id="6470">
        <attvalues>
          <attvalue for="5" value=" Indeed, GPT-based language models have shown impressive results for open-domain dialogue generation \cite{golovanov2019large,wolf2019transfertransfo,dialogpt}." />
        </attvalues>
      </edge>
      <edge source="2004.12506" target="1909.10705" id="6471">
        <attvalues>
          <attvalue for="5" value=" In particular, using automatic metrics (eg, cosine similarity, lexical diversity, sentence length), \cite{see2019massively} illustrated that GPT-2 has the ability to generate interesting and coherent text." />
        </attvalues>
      </edge>
      <edge source="2204.02470" target="1712.01769" id="6473">
        <attvalues>
          <attvalue for="5" value=" End-to-end models based on deep learning have demonstrated their superiority over conventional hidden Markov-based models on speech tasks for some corpora \cite{chiu2018state, karita2019comparative, pham2019very, guo2021recent}." />
        </attvalues>
      </edge>
      <edge source="2204.02470" target="2203.06849" id="6475">
        <attvalues>
          <attvalue for="5" value=" On the other hand, end-to-end models can perform poorly when the training data is limited \cite{rwth-libri-2019} and low resource scenarios often introduce a language-mismatch with the data used to train powerful self-supervised learning (SSL) representations \cite{tsai2022superbsg}." />
        </attvalues>
      </edge>
      <edge source="2204.02470" target="2002.11800" id="6476">
        <attvalues>
          <attvalue for="5" value=" When there is no training data available for the target languages, these systems can be even applied in a zero-shot manner \cite{li2020universal, yan2021differentiable, xu2021simple}." />
        </attvalues>
      </edge>
      <edge source="2204.02470" target="2012.12121" id="6477">
        <attvalues>
          <attvalue for="5" value=" Another direction is to use self-supervised learning models trained on large untranscribed corpora as front-end feature extractors, replacing conventional spectral features (SF) such as log Mel-filterbanks coefficients (FBANK) \cite{yi2020applying, wu20g_interspeech, baevski2020wav2vec, n21_interspeech, chang2021exploration, liu2021tera}." />
        </attvalues>
      </edge>
      <edge source="2204.02470" target="2006.11477" id="6478">
        <attvalues>
          <attvalue for="5" value=" SSL first layers output representations tend to be quite similar to SF according to a canonical correlation analysis \cite{andrew2013deep} of Wav2vec2 \cite{DBLP:wav2vec2} from Pasad et al." />
        </attvalues>
      </edge>
      <edge source="2204.02470" target="2110.13900" id="6479">
        <attvalues>
          <attvalue for="5" value=" During their unsupervised training, SSL models \cite{DBLP:HuBERT, DBLP:wav2vec,DBLP:wav2vec2,chen2021wavlm} learn their own feature extraction modules and are totally free of SF at fine-tuning time." />
        </attvalues>
      </edge>
      <edge source="2204.02470" target="1911.02116" id="6481">
        <attvalues>
          <attvalue for="5" value=" Although these approaches have shown improvements, even when domain mismatches occur (such as language or audio conditions \cite{sanabria2022measuring}), performance depends on the relatedness between the SSL training domain and the target language one \cite{conneau2019unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2204.02470" target="2107.04734" id="6482">
        <attvalues>
          <attvalue for="5" value=" In contrast, the last layers are likely to be more corpus or domain-specific, which should be randomly initialized at fine-tuning time \cite{DBLP:layerwiseToyota}." />
        </attvalues>
      </edge>
      <edge source="2110.05847" target="2103.00508" id="6490">
        <attvalues>
          <attvalue for="5" value=" However, this often results in large amounts of generated content in the deliberations, causing information overload that prevents their potential from being fully realised \cite{arana2021citizen, davies2020online, davies2021mixed}." />
        </attvalues>
      </edge>
      <edge source="2110.05847" target="1910.13461" id="6491">
        <attvalues>
          <attvalue for="5" value=" We have carried out an evaluation with 6 abstractive summarisation models: BART \cite{lewis2019bart}, T5 \cite{raffel2019exploring}, BERT (PreSumm – BertSumExtAbs: \cite{liu2019text}), PG (Pointer-Generator with Coverage Penalty) \cite{see2017get}, CopyTransformer \cite{gehrmann2018bottom}, and FastAbsRL \cite{chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2110.05847" target="1704.04368" id="6494">
        <attvalues>
          <attvalue for="5" value=" We have carried out an evaluation with 6 abstractive summarisation models: BART \cite{lewis2019bart}, T5 \cite{raffel2019exploring}, BERT (PreSumm – BertSumExtAbs: \cite{liu2019text}), PG (Pointer-Generator with Coverage Penalty) \cite{see2017get}, CopyTransformer \cite{gehrmann2018bottom}, and FastAbsRL \cite{chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2110.05847" target="1808.10792" id="6495">
        <attvalues>
          <attvalue for="5" value=" We have carried out an evaluation with 6 abstractive summarisation models: BART \cite{lewis2019bart}, T5 \cite{raffel2019exploring}, BERT (PreSumm – BertSumExtAbs: \cite{liu2019text}), PG (Pointer-Generator with Coverage Penalty) \cite{see2017get}, CopyTransformer \cite{gehrmann2018bottom}, and FastAbsRL \cite{chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2110.05847" target="1805.11080" id="6496">
        <attvalues>
          <attvalue for="5" value=" We have carried out an evaluation with 6 abstractive summarisation models: BART \cite{lewis2019bart}, T5 \cite{raffel2019exploring}, BERT (PreSumm – BertSumExtAbs: \cite{liu2019text}), PG (Pointer-Generator with Coverage Penalty) \cite{see2017get}, CopyTransformer \cite{gehrmann2018bottom}, and FastAbsRL \cite{chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2110.05847" target="1804.00344" id="6497">
        <attvalues>
          <attvalue for="5" value=" Those models are applied in combination with the machine translation system MarianMT \cite{junczys2018marian} using the Opus-MT models \cite{tiedemann2020opus}." />
        </attvalues>
      </edge>
      <edge source="2010.12627" target="1309.4168" id="6498">
        <attvalues>
          <attvalue for="5" value=" Mapping based BWE approaches rely only on a cheap bilingual signal, in the form of a seed lexicon, and monolingual data to train monolingual word embeddings (MWEs) for each language, which makes them easily applicable in low-resource scenarios \cite{Mikolov2013exploiting,Xing2015,Artetxe2016}." />
        </attvalues>
      </edge>
      <edge source="2010.12627" target="1710.04087" id="6499">
        <attvalues>
          <attvalue for="5" value=" It was shown that BWEs can be built using a small seed lexicon \cite{Artexte2017} or without any word pairs \cite{Conneau2018,Artetxe2018} relying on the assumption of isomorphic MWE spaces." />
        </attvalues>
      </edge>
      <edge source="2010.12627" target="1805.06297" id="6500">
        <attvalues>
          <attvalue for="5" value=" It was shown that BWEs can be built using a small seed lexicon \cite{Artexte2017} or without any word pairs \cite{Conneau2018,Artetxe2018} relying on the assumption of isomorphic MWE spaces." />
        </attvalues>
      </edge>
      <edge source="2010.12627" target="1901.07291" id="6503">
        <attvalues>
          <attvalue for="5" value=" Similarly, the shared source and target language subword tokens are used as a cheap cross-lingual signal in \cite{devlin-etal-2019-bert,Conneau2019XLM}." />
        </attvalues>
      </edge>
      <edge source="2010.12627" target="1910.04708" id="6504">
        <attvalues>
          <attvalue for="5" value=" Furthermore, the advantages of mapping and jointly training the MWEs and BWEs were combined in \cite{Wang2020} for even better BWEs." />
        </attvalues>
      </edge>
      <edge source="2209.12786" target="1907.11692" id="6518">
        <attvalues>
          <attvalue for="5" value=" To observe this, we can query widely used pretrained models, such as RoBERTa$_{Large}$ \cite{liu2019roberta} with our previous example." />
        </attvalues>
      </edge>
      <edge source="2209.12786" target="2110.08182" id="6519">
        <attvalues>
          <attvalue for="5" value=" After plotting accuracy against model size, we observe that scaling up is universally helpful for improving LLMs' performance on the colour probing benchmark (CoDa; \cite{paik-etal-2021-world})." />
        </attvalues>
      </edge>
      <edge source="2209.12786" target="1909.11942" id="6522">
        <attvalues>
          <attvalue for="5" value=" However, the LMs tested by \cite{paik-etal-2021-world,zhang-etal-2022-visual,shwartz-choi-2020-neural}, eg, GPT-2 \cite{radford2019language}, BERT \cite{devlin-etal-2019-bert}, RoBERTa, and ALBERT \cite{Lan2020ALBERT}, usually have only several hundred million parameters and are of much smaller sizes than LLMs available now." />
        </attvalues>
      </edge>
      <edge source="2209.12786" target="2005.14165" id="6524">
        <attvalues>
          <attvalue for="5" value=" In this work we probe T5 \cite{raffel-etal-2020-exploring}, GPT-3 \cite{brown2020language}, and PaLM \cite{chowdhery2022palm} of various sizes, with parameter counts ranging from 770M to 540B." />
        </attvalues>
      </edge>
      <edge source="2209.12786" target="2204.02311" id="6525">
        <attvalues>
          <attvalue for="5" value=" In this work we probe T5 \cite{raffel-etal-2020-exploring}, GPT-3 \cite{brown2020language}, and PaLM \cite{chowdhery2022palm} of various sizes, with parameter counts ranging from 770M to 540B." />
        </attvalues>
      </edge>
      <edge source="2307.10210" target="2006.00632" id="6526">
        <attvalues>
          <attvalue for="5" value=" We refer the reader to domain adaptation surveys in natural language processing for a detailed overview \cite{ramponi2020neural, chu2018survey, jiang2013literature, margolis2011literature}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2109.08678" id="6527">
        <attvalues>
          <attvalue for="5" value=" Besides, another branch of case-based methods \cite{das2022knowledge,ye2021rng,das2021case,yu-etal-2022-kg} integrate training cases or candidate subgraphs to prompt the model for reasoning." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2202.10610" id="6528">
        <attvalues>
          <attvalue for="5" value=" Besides, another branch of case-based methods \cite{das2022knowledge,ye2021rng,das2021case,yu-etal-2022-kg} integrate training cases or candidate subgraphs to prompt the model for reasoning." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2003.13956" id="6529">
        <attvalues>
          <attvalue for="5" value=" The existing methods to solve KBQA can be divided into two categories: Semantic parsing-based (SP-based) methods \cite{ye2021rng,das2022knowledge,lan2020query,sun2020sparqa} and embedding-based methods \cite{agarwal-etal-2021-knowledge,zhang2022subgraph,he2021improving,sun2020faithful,sun2018open,sun2019pullnet}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2010.12688" id="6530">
        <attvalues>
          <attvalue for="5" value=" The existing methods to solve KBQA can be divided into two categories: Semantic parsing-based (SP-based) methods \cite{ye2021rng,das2022knowledge,lan2020query,sun2020sparqa} and embedding-based methods \cite{agarwal-etal-2021-knowledge,zhang2022subgraph,he2021improving,sun2020faithful,sun2018open,sun2019pullnet}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2202.13296" id="6531">
        <attvalues>
          <attvalue for="5" value=" The existing methods to solve KBQA can be divided into two categories: Semantic parsing-based (SP-based) methods \cite{ye2021rng,das2022knowledge,lan2020query,sun2020sparqa} and embedding-based methods \cite{agarwal-etal-2021-knowledge,zhang2022subgraph,he2021improving,sun2020faithful,sun2018open,sun2019pullnet}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2101.03737" id="6532">
        <attvalues>
          <attvalue for="5" value=" The existing methods to solve KBQA can be divided into two categories: Semantic parsing-based (SP-based) methods \cite{ye2021rng,das2022knowledge,lan2020query,sun2020sparqa} and embedding-based methods \cite{agarwal-etal-2021-knowledge,zhang2022subgraph,he2021improving,sun2020faithful,sun2018open,sun2019pullnet}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2004.03658" id="6533">
        <attvalues>
          <attvalue for="5" value=" The existing methods to solve KBQA can be divided into two categories: Semantic parsing-based (SP-based) methods \cite{ye2021rng,das2022knowledge,lan2020query,sun2020sparqa} and embedding-based methods \cite{agarwal-etal-2021-knowledge,zhang2022subgraph,he2021improving,sun2020faithful,sun2018open,sun2019pullnet}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="1809.00782" id="6534">
        <attvalues>
          <attvalue for="5" value=" Previous works \cite{bordes2013translating,ji2015knowledge,sun2018open,sun2019pullnet,ju2022grape} primarily addresses these challenges by introducing external text corpus and specially designed framework to incorporate information from the documents." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="1904.09537" id="6535">
        <attvalues>
          <attvalue for="5" value=" Previous works \cite{bordes2013translating,ji2015knowledge,sun2018open,sun2019pullnet,ju2022grape} primarily addresses these challenges by introducing external text corpus and specially designed framework to incorporate information from the documents." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="cs/0605124" id="6536">
        <attvalues>
          <attvalue for="5" value=" The former one heavily relies on the expensive annotation of the intermediate logic form such as SPARQL \cite{perez2009semantics}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2203.10321" id="6537">
        <attvalues>
          <attvalue for="5" value=" Instead of parsing the questions, the latter one directly encodes and retrieves the candidate subgraphs, then obtains answers by a ranker or a generator \cite{saxena2022sequence,oguz2020unik,yu2022decaf}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2012.14610" id="6538">
        <attvalues>
          <attvalue for="5" value=" Instead of parsing the questions, the latter one directly encodes and retrieves the candidate subgraphs, then obtains answers by a ranker or a generator \cite{saxena2022sequence,oguz2020unik,yu2022decaf}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2210.00063" id="6539">
        <attvalues>
          <attvalue for="5" value=" Instead of parsing the questions, the latter one directly encodes and retrieves the candidate subgraphs, then obtains answers by a ranker or a generator \cite{saxena2022sequence,oguz2020unik,yu2022decaf}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2212.08153" id="6540">
        <attvalues>
          <attvalue for="5" value=" Moreover, to improve the efficiency, existing methods directly concatenate all the retrieved subgraphs during answer generation, which inevitably causes different types of subgraphs to interfere with each other in the encoding process \cite{de2022fido,yu-etal-2022-kg}." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2110.04330" id="6541">
        <attvalues>
          <attvalue for="5" value=" Besides, another branch of case-based methods \cite{das2022knowledge,ye2021rng,das2021case,yu-etal-2022-kg} integrate training cases or candidate subgraphs to prompt the model for reasoning." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2210.02933" id="6542">
        <attvalues>
          <attvalue for="5" value=" Previous works \cite{bordes2013translating,ji2015knowledge,sun2018open,sun2019pullnet,ju2022grape} primarily addresses these challenges by introducing external text corpus and specially designed framework to incorporate information from the documents." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2104.08762" id="6543">
        <attvalues>
          <attvalue for="5" value=" Besides, another branch of case-based methods \cite{das2022knowledge,ye2021rng,das2021case,yu-etal-2022-kg} integrate training cases or candidate subgraphs to prompt the model for reasoning." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="2002.05709" id="6545">
        <attvalues>
          <attvalue for="5" value=" Inspired by masked language modeling (MLM) \cite{devlin2018bert} and contrastive learning \cite{chen2020simple}, we introduce two novel structured knowledge-aware pre-training tasks according to the characteristics of the data in subgraphs." />
        </attvalues>
      </edge>
      <edge source="2308.14436" target="1909.07606" id="6546">
        <attvalues>
          <attvalue for="5" value=" To better encode different subgraphs during reasoning, motivated by \cite{liu2020k}, we design an interval attention mechanism, effectively guiding the model to shield the interference from irrelevant subgraphs." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="1706.03762" id="6547">
        <attvalues>
          <attvalue for="5" value=" Transformers \cite{vaswani2017attention} have risen as the preferred architecture for language models, being simpler, more scalable and more performant than alternatives based on recurrent neural networks." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="2005.14165" id="6548">
        <attvalues>
          <attvalue for="5" value=" Furthermore, few-shot learning capabilities have been observed in GPT-3 (175 billion parameters) \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="2101.03961" id="6550">
        <attvalues>
          <attvalue for="5" value=" Although transformers come in numerous flavors and variations \cite[\dots]{brown2020language,devlin2019bert,fedus2021switch,lewis2019bart,radford2019language,raffel2020exploring,yang2020xlnet}, a growing body of work suggests that the model performance is mainly driven by scale." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="1910.13461" id="6551">
        <attvalues>
          <attvalue for="5" value=" Although transformers come in numerous flavors and variations \cite[\dots]{brown2020language,devlin2019bert,fedus2021switch,lewis2019bart,radford2019language,raffel2020exploring,yang2020xlnet}, a growing body of work suggests that the model performance is mainly driven by scale." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="1909.08053" id="6554">
        <attvalues>
          <attvalue for="5" value=" The leading method is 3d parallelism (see for example \cite{shoeybi2020megatronlm}), which combines the three common form of parallelism: data, pipeline and tensor parallelism." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="1910.02054" id="6555">
        <attvalues>
          <attvalue for="5" value=" The ZeRO family of methods addresses this bottleneck by partitioning the training state \cite{rajbhandari2020zero}, aggressively offloading memory \cite{ren2021zerooffload,rajbhandari2021zeroinfinity}, and breaking down individual operations \cite{rajbhandari2021zeroinfinity}." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="2101.06840" id="6556">
        <attvalues>
          <attvalue for="5" value=" The ZeRO family of methods addresses this bottleneck by partitioning the training state \cite{rajbhandari2020zero}, aggressively offloading memory \cite{ren2021zerooffload,rajbhandari2021zeroinfinity}, and breaking down individual operations \cite{rajbhandari2021zeroinfinity}." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="2104.07857" id="6557">
        <attvalues>
          <attvalue for="5" value=" In section\ref{sec:fast_checkpoints}, we investigate disk offloading in a way similar to ZeRO-Infinity \cite{rajbhandari2021zeroinfinity}." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="1811.12941" id="6558">
        <attvalues>
          <attvalue for="5" value=" We explicitly integrate the concept of critical batch size \cite{golmant2018computational,mccandlish2018empirical,shallue2019measuring} in our analysis, leveraging the empirical scaling laws found in \cite{kaplan2020scaling}." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="1812.06162" id="6559">
        <attvalues>
          <attvalue for="5" value=" We explicitly integrate the concept of critical batch size \cite{golmant2018computational,mccandlish2018empirical,shallue2019measuring} in our analysis, leveraging the empirical scaling laws found in \cite{kaplan2020scaling}." />
        </attvalues>
      </edge>
      <edge source="2106.02679" target="2001.08361" id="6561">
        <attvalues>
          <attvalue for="5" value=" We explicitly integrate the concept of critical batch size \cite{golmant2018computational,mccandlish2018empirical,shallue2019measuring} in our analysis, leveraging the empirical scaling laws found in \cite{kaplan2020scaling}." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2305.05003" id="6565">
        <attvalues>
          <attvalue for="5" value=" This detailed process typically involves \ding{182} categorizing raw data with class or task labels for basic classification, \ding{183} adding intermediate labels for contextual depth~\cite{yu2022generate}, \ding{184} assigning confidence scores to assess annotation reliability~\cite{lin2022teaching}, \ding{185} applying alignment or preference labels to tailor outputs to specific criteria or user needs, \ding{186} annotating entity relationships to understand how entities within a dataset interact with each other~\cite{wadhwa2023revisiting}, \ding{187} marking semantic roles to define the underlying roles that entities play in a sentence~\cite{larionov2019semantic}, \ding{188} tagging temporal sequences to capture the order of events or actions~\cite{yu2023temporal}, or \ding{189} Synthesize data in the format of instruction~\cite{wang2022self}, response~\cite{zhang2023self}, reasoning~\cite{wang2022pinto}, pairwise~\cite{bai2022constitutional} and textual feedback~\cite{pan2024automatically} to for language model tuning." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2203.11171" id="6566">
        <attvalues>
          <attvalue for="5" value=" This detailed process typically involves \ding{182} categorizing raw data with class or task labels for basic classification, \ding{183} adding intermediate labels for contextual depth~\cite{yu2022generate}, \ding{184} assigning confidence scores to assess annotation reliability~\cite{lin2022teaching}, \ding{185} applying alignment or preference labels to tailor outputs to specific criteria or user needs, \ding{186} annotating entity relationships to understand how entities within a dataset interact with each other~\cite{wadhwa2023revisiting}, \ding{187} marking semantic roles to define the underlying roles that entities play in a sentence~\cite{larionov2019semantic}, \ding{188} tagging temporal sequences to capture the order of events or actions~\cite{yu2023temporal}, or \ding{189} Synthesize data in the format of instruction~\cite{wang2022self}, response~\cite{zhang2023self}, reasoning~\cite{wang2022pinto}, pairwise~\cite{bai2022constitutional} and textual feedback~\cite{pan2024automatically} to for language model tuning." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2305.11952" id="6567">
        <attvalues>
          <attvalue for="5" value=" This detailed process typically involves \ding{182} categorizing raw data with class or task labels for basic classification, \ding{183} adding intermediate labels for contextual depth~\cite{yu2022generate}, \ding{184} assigning confidence scores to assess annotation reliability~\cite{lin2022teaching}, \ding{185} applying alignment or preference labels to tailor outputs to specific criteria or user needs, \ding{186} annotating entity relationships to understand how entities within a dataset interact with each other~\cite{wadhwa2023revisiting}, \ding{187} marking semantic roles to define the underlying roles that entities play in a sentence~\cite{larionov2019semantic}, \ding{188} tagging temporal sequences to capture the order of events or actions~\cite{yu2023temporal}, or \ding{189} Synthesize data in the format of instruction~\cite{wang2022self}, response~\cite{zhang2023self}, reasoning~\cite{wang2022pinto}, pairwise~\cite{bai2022constitutional} and textual feedback~\cite{pan2024automatically} to for language model tuning." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2212.08073" id="6569">
        <attvalues>
          <attvalue for="5" value=" This detailed process typically involves \ding{182} categorizing raw data with class or task labels for basic classification, \ding{183} adding intermediate labels for contextual depth~\cite{yu2022generate}, \ding{184} assigning confidence scores to assess annotation reliability~\cite{lin2022teaching}, \ding{185} applying alignment or preference labels to tailor outputs to specific criteria or user needs, \ding{186} annotating entity relationships to understand how entities within a dataset interact with each other~\cite{wadhwa2023revisiting}, \ding{187} marking semantic roles to define the underlying roles that entities play in a sentence~\cite{larionov2019semantic}, \ding{188} tagging temporal sequences to capture the order of events or actions~\cite{yu2023temporal}, or \ding{189} Synthesize data in the format of instruction~\cite{wang2022self}, response~\cite{zhang2023self}, reasoning~\cite{wang2022pinto}, pairwise~\cite{bai2022constitutional} and textual feedback~\cite{pan2024automatically} to for language model tuning." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2310.06498" id="6570">
        <attvalues>
          <attvalue for="5" value=" Despite its wide applications, data annotation and synthesis poses significant challenges for current machine learning models due to the complexity, subjectivity, and diversity of data~\cite{yang2023new}." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2303.08774" id="6571">
        <attvalues>
          <attvalue for="5" value=" Advanced LLMs such as GPT-4~\cite{openai2023gpt4}, Gemini~\cite{team2023gemini}, and LLaMA-2~\cite{touvron2023llama2} offer a promising opportunity to revolutionize data annotation." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2312.11805" id="6572">
        <attvalues>
          <attvalue for="5" value=" Advanced LLMs such as GPT-4~\cite{openai2023gpt4}, Gemini~\cite{team2023gemini}, and LLaMA-2~\cite{touvron2023llama2} offer a promising opportunity to revolutionize data annotation." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2307.09288" id="6573">
        <attvalues>
          <attvalue for="5" value=" Advanced LLMs such as GPT-4~\cite{openai2023gpt4}, Gemini~\cite{team2023gemini}, and LLaMA-2~\cite{touvron2023llama2} offer a promising opportunity to revolutionize data annotation." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2404.10306" id="6576">
        <attvalues>
          <attvalue for="5" value=" Their ability to automate annotation tasks, ensure consistency across large volumes of data~\cite{hou2023large}, and adapt through fine-tuning or prompting for specific domains~\cite{song2023preference,Zhang2024BalancingSA}, significantly mitigates the challenges encountered with traditional annotation and synthesis methods, setting a new standard for what is achievable in the realm of NLP." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2303.18223" id="6578">
        <attvalues>
          <attvalue for="5" value=" While existing surveys in the NLP domain extensively cover architectural nuances~\cite{zhao2023survey}, training methodologies~\cite{liu2023trustworthy}, and evaluation protocols~\cite{chang2023survey} associated with LLMs, their main focus lies on the capabilities of models for specific end tasks such as machine translation~\cite{min2021recent}, alignment~\cite{wang2023aligning}, code generation~\cite{zan2023large}, and medical analysis~\cite{thirunavukarasu2023large}." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2308.05374" id="6579">
        <attvalues>
          <attvalue for="5" value=" While existing surveys in the NLP domain extensively cover architectural nuances~\cite{zhao2023survey}, training methodologies~\cite{liu2023trustworthy}, and evaluation protocols~\cite{chang2023survey} associated with LLMs, their main focus lies on the capabilities of models for specific end tasks such as machine translation~\cite{min2021recent}, alignment~\cite{wang2023aligning}, code generation~\cite{zan2023large}, and medical analysis~\cite{thirunavukarasu2023large}." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2307.03109" id="6580">
        <attvalues>
          <attvalue for="5" value=" While existing surveys in the NLP domain extensively cover architectural nuances~\cite{zhao2023survey}, training methodologies~\cite{liu2023trustworthy}, and evaluation protocols~\cite{chang2023survey} associated with LLMs, their main focus lies on the capabilities of models for specific end tasks such as machine translation~\cite{min2021recent}, alignment~\cite{wang2023aligning}, code generation~\cite{zan2023large}, and medical analysis~\cite{thirunavukarasu2023large}." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2111.01243" id="6581">
        <attvalues>
          <attvalue for="5" value=" While existing surveys in the NLP domain extensively cover architectural nuances~\cite{zhao2023survey}, training methodologies~\cite{liu2023trustworthy}, and evaluation protocols~\cite{chang2023survey} associated with LLMs, their main focus lies on the capabilities of models for specific end tasks such as machine translation~\cite{min2021recent}, alignment~\cite{wang2023aligning}, code generation~\cite{zan2023large}, and medical analysis~\cite{thirunavukarasu2023large}." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2307.12966" id="6582">
        <attvalues>
          <attvalue for="5" value=" While existing surveys in the NLP domain extensively cover architectural nuances~\cite{zhao2023survey}, training methodologies~\cite{liu2023trustworthy}, and evaluation protocols~\cite{chang2023survey} associated with LLMs, their main focus lies on the capabilities of models for specific end tasks such as machine translation~\cite{min2021recent}, alignment~\cite{wang2023aligning}, code generation~\cite{zan2023large}, and medical analysis~\cite{thirunavukarasu2023large}." />
        </attvalues>
      </edge>
      <edge source="2402.13446" target="2212.09420" id="6583">
        <attvalues>
          <attvalue for="5" value=" While existing surveys in the NLP domain extensively cover architectural nuances~\cite{zhao2023survey}, training methodologies~\cite{liu2023trustworthy}, and evaluation protocols~\cite{chang2023survey} associated with LLMs, their main focus lies on the capabilities of models for specific end tasks such as machine translation~\cite{min2021recent}, alignment~\cite{wang2023aligning}, code generation~\cite{zan2023large}, and medical analysis~\cite{thirunavukarasu2023large}." />
        </attvalues>
      </edge>
      <edge source="2312.07405" target="2301.00234" id="6584">
        <attvalues>
          <attvalue for="5" value=" Most LLMs have not been explicitly trained or tuned to perform ICL, and thus have not actually been optimized to approach new tasks in this format~\cite{Dong2022ALearning}." />
        </attvalues>
      </edge>
      <edge source="2312.07405" target="2209.07661" id="6585">
        <attvalues>
          <attvalue for="5" value=" Like other forms of prompt engineering, ICL suffers from a lack of robustness across the many arbitrary choices that users encounter in the process of setting it up~\cite{Chen2022OnLearning}." />
        </attvalues>
      </edge>
      <edge source="2312.07405" target="2102.09690" id="6586">
        <attvalues>
          <attvalue for="5" value=" It has been shown that the performance of in-context learning can vary dramatically based on changes to the prompt~\cite{Zhao2021CalibrateModels}." />
        </attvalues>
      </edge>
      <edge source="2312.07405" target="2207.05221" id="6587">
        <attvalues>
          <attvalue for="5" value=" There is also evidence to suggest that ICL performs poorly when shown a &quot;none of the above&quot; option~\cite{Kadavath2022LanguageKnow}, which could hinder its application in practical settings (eg\ open world classification) where the inputs may not always correspond with any option in the label space." />
        </attvalues>
      </edge>
      <edge source="2204.08167" target="2005.00796" id="6588">
        <attvalues>
          <attvalue for="5" value=" Recent State-of-the-art models that tackle the Belief State Tracking problem are generally based on large language models \cite{hosseini2020simple,heck2020trippy,wu2019transferable}." />
        </attvalues>
      </edge>
      <edge source="2204.08167" target="2005.02877" id="6589">
        <attvalues>
          <attvalue for="5" value=" Recent State-of-the-art models that tackle the Belief State Tracking problem are generally based on large language models \cite{hosseini2020simple,heck2020trippy,wu2019transferable}." />
        </attvalues>
      </edge>
      <edge source="2204.08167" target="1905.08743" id="6590">
        <attvalues>
          <attvalue for="5" value=" Recent State-of-the-art models that tackle the Belief State Tracking problem are generally based on large language models \cite{hosseini2020simple,heck2020trippy,wu2019transferable}." />
        </attvalues>
      </edge>
      <edge source="2204.08167" target="2002.10348" id="6592">
        <attvalues>
          <attvalue for="5" value=" The development of models like BERT \cite{devlin2018bert} and GPT-2 \cite{Radford2019LanguageMA} has also inspired advances in the use of pre-trained language models (PLMs) for low-resource few-shot learning for dialog generation \cite{Zhao2020Low-Resource}." />
        </attvalues>
      </edge>
      <edge source="2204.08167" target="2005.14165" id="6593">
        <attvalues>
          <attvalue for="5" value=" The recent paradigm of prompt-based learning \cite{brown2020language} equips PLMs with constructive task-dependent prompts to simplify language generation for downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2204.08167" target="2012.15723" id="6594">
        <attvalues>
          <attvalue for="5" value=" This method has shown great results on few-shot and zero-shot learning tasks such as classification \cite{gao-etal-2021-making,schick-schutze-2021-exploiting,han2021ptr} and text generation \cite{schick2020few,li-liang-2021-prefix}." />
        </attvalues>
      </edge>
      <edge source="2204.08167" target="2105.11259" id="6595">
        <attvalues>
          <attvalue for="5" value=" This method has shown great results on few-shot and zero-shot learning tasks such as classification \cite{gao-etal-2021-making,schick-schutze-2021-exploiting,han2021ptr} and text generation \cite{schick2020few,li-liang-2021-prefix}." />
        </attvalues>
      </edge>
      <edge source="2204.08167" target="2101.00190" id="6597">
        <attvalues>
          <attvalue for="5" value=" This method has shown great results on few-shot and zero-shot learning tasks such as classification \cite{gao-etal-2021-making,schick-schutze-2021-exploiting,han2021ptr} and text generation \cite{schick2020few,li-liang-2021-prefix}." />
        </attvalues>
      </edge>
      <edge source="2206.09676" target="1803.06535" id="6598">
        <attvalues>
          <attvalue for="5" value=" Text style transfer (TST) systems are designed to change the style of the original text to alternative one, such as more informal~\cite{rao-tetreault-2018-dear}, more positive~\cite{luo-etal-2019-towards}, or even more Shakespearean~\cite{jhamtani-etal-2017-shakespearizing}." />
        </attvalues>
      </edge>
      <edge source="2206.09676" target="2005.00701" id="6599">
        <attvalues>
          <attvalue for="5" value=" In the majority of recent TST papers~\cite{DBLP:conf/sigir/RaneDLE21,cao-etal-2020-expertise,riley-etal-2021-textsettr}BLEU~\cite{papineni2002bleu} is still the main way to evaluate the content similarity." />
        </attvalues>
      </edge>
      <edge source="2206.09676" target="2010.03802" id="6600">
        <attvalues>
          <attvalue for="5" value=" In the majority of recent TST papers~\cite{DBLP:conf/sigir/RaneDLE21,cao-etal-2020-expertise,riley-etal-2021-textsettr}BLEU~\cite{papineni2002bleu} is still the main way to evaluate the content similarity." />
        </attvalues>
      </edge>
      <edge source="2206.09676" target="1810.11878" id="6601">
        <attvalues>
          <attvalue for="5" value=" More recent approaches as cosine similarity calculation between averaged word vectors~\cite{pang-gimpel-2019-unsupervised}, BLEURT~\cite{sellam-etal-2020-bleurt} (which is a BERT~\cite{devlin-etal-2019-bert} fine-tuned for semantic similarity evaluation task in cross-encoder manner on synthetic data)~\cite{lai-etal-2021-thank} and BERTScore\cite{zhang2019bertscore} (F1-score over BERT-embeddings between tokens from initial and target sentences)~\cite{lee-etal-2021-enhancing} are also gaining popularity." />
        </attvalues>
      </edge>
      <edge source="2206.09676" target="2004.04696" id="6602">
        <attvalues>
          <attvalue for="5" value=" More recent approaches as cosine similarity calculation between averaged word vectors~\cite{pang-gimpel-2019-unsupervised}, BLEURT~\cite{sellam-etal-2020-bleurt} (which is a BERT~\cite{devlin-etal-2019-bert} fine-tuned for semantic similarity evaluation task in cross-encoder manner on synthetic data)~\cite{lai-etal-2021-thank} and BERTScore\cite{zhang2019bertscore} (F1-score over BERT-embeddings between tokens from initial and target sentences)~\cite{lee-etal-2021-enhancing} are also gaining popularity." />
        </attvalues>
      </edge>
      <edge source="2206.09676" target="2105.06947" id="6604">
        <attvalues>
          <attvalue for="5" value=" More recent approaches as cosine similarity calculation between averaged word vectors~\cite{pang-gimpel-2019-unsupervised}, BLEURT~\cite{sellam-etal-2020-bleurt} (which is a BERT~\cite{devlin-etal-2019-bert} fine-tuned for semantic similarity evaluation task in cross-encoder manner on synthetic data)~\cite{lai-etal-2021-thank} and BERTScore\cite{zhang2019bertscore} (F1-score over BERT-embeddings between tokens from initial and target sentences)~\cite{lee-etal-2021-enhancing} are also gaining popularity." />
        </attvalues>
      </edge>
      <edge source="2206.09676" target="2108.00449" id="6606">
        <attvalues>
          <attvalue for="5" value=" More recent approaches as cosine similarity calculation between averaged word vectors~\cite{pang-gimpel-2019-unsupervised}, BLEURT~\cite{sellam-etal-2020-bleurt} (which is a BERT~\cite{devlin-etal-2019-bert} fine-tuned for semantic similarity evaluation task in cross-encoder manner on synthetic data)~\cite{lai-etal-2021-thank} and BERTScore\cite{zhang2019bertscore} (F1-score over BERT-embeddings between tokens from initial and target sentences)~\cite{lee-etal-2021-enhancing} are also gaining popularity." />
        </attvalues>
      </edge>
      <edge source="2206.09676" target="1808.10192" id="6607">
        <attvalues>
          <attvalue for="5" value=" For example, in~\cite{nema-khapra-2018-towards} authors exploited a similar assumption and used the information about NEs and some other categories of words to improve measures like BLEU or METEOR~\cite{banerjee-lavie-2005-meteor} for question answering task." />
        </attvalues>
      </edge>
      <edge source="2311.15451" target="1911.00811" id="6612">
        <attvalues>
          <attvalue for="5" value=" Indeed, it has been shown that LLMs commonly fail in QA tasks \cite{geiger2019posing}, and that these failures are associated with a limited understanding of output confidence, out-of-domain data, ambiguity in input prompts, inconsistent training information, and hallucinations, among others." />
        </attvalues>
      </edge>
      <edge source="2311.15451" target="1904.04792" id="6613">
        <attvalues>
          <attvalue for="5" value=" One approach used inferred softmax classifier probabilities to calibrate which questions to respond to \cite{rodriguez2021quizbowl}." />
        </attvalues>
      </edge>
      <edge source="2311.15451" target="1706.04599" id="6614">
        <attvalues>
          <attvalue for="5" value=" However, training a calibration model is challenging and softmax classifiers are often unreliable \cite{guo2017calibration}." />
        </attvalues>
      </edge>
      <edge source="2311.15451" target="2006.09462" id="6615">
        <attvalues>
          <attvalue for="5" value=" Similarly, an out-of-domain (OOD) calibrator can be trained to detect OOD inputs \cite{kamath2020selective} but requires known or synthetic out-of-domain samples and does not consider other sources of inaccuracies like over-represented features or ambiguous labels." />
        </attvalues>
      </edge>
      <edge source="2311.15451" target="1805.04604" id="6616">
        <attvalues>
          <attvalue for="5" value=" Other approaches include modeling and estimating LLM uncertainty \cite{dong2018confidence, shen2022posthoc, chen2023quantifying, lin2023generating, collins2023human, chuang2023dola, quach2023conformal}, fine-tuning calibrators to consider entropy, perplexity, and other metrics \cite{jiang2021can}, and calculating output consistency \cite{manakul2023selfcheckgpt, miao2023selfcheck}." />
        </attvalues>
      </edge>
      <edge source="2311.15451" target="2012.00955" id="6617">
        <attvalues>
          <attvalue for="5" value=" Other approaches include modeling and estimating LLM uncertainty \cite{dong2018confidence, shen2022posthoc, chen2023quantifying, lin2023generating, collins2023human, chuang2023dola, quach2023conformal}, fine-tuning calibrators to consider entropy, perplexity, and other metrics \cite{jiang2021can}, and calculating output consistency \cite{manakul2023selfcheckgpt, miao2023selfcheck}." />
        </attvalues>
      </edge>
      <edge source="2311.15451" target="2303.08896" id="6618">
        <attvalues>
          <attvalue for="5" value=" Other approaches include modeling and estimating LLM uncertainty \cite{dong2018confidence, shen2022posthoc, chen2023quantifying, lin2023generating, collins2023human, chuang2023dola, quach2023conformal}, fine-tuning calibrators to consider entropy, perplexity, and other metrics \cite{jiang2021can}, and calculating output consistency \cite{manakul2023selfcheckgpt, miao2023selfcheck}." />
        </attvalues>
      </edge>
      <edge source="2311.15451" target="2108.11896" id="6619">
        <attvalues>
          <attvalue for="5" value=" Another family of techniques retrieve evidence and verify outputs through external databases \cite{guo2022survey} or in-context learning \cite{weng2022large}." />
        </attvalues>
      </edge>
      <edge source="2304.12102" target="2005.14165" id="6620">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across a wide range of natural language processing tasks, as well as real-life applications \cite{brown2020language,touvron2023llama,bubeck2023sparks}." />
        </attvalues>
      </edge>
      <edge source="2304.12102" target="2303.12712" id="6622">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across a wide range of natural language processing tasks, as well as real-life applications \cite{brown2020language,touvron2023llama,bubeck2023sparks}." />
        </attvalues>
      </edge>
      <edge source="2304.12102" target="2302.14502" id="6623">
        <attvalues>
          <attvalue for="5" value=" As LLMs have no memory outside their context window, it poses a significant challenge when tackling tasks that involve processing long documents or engaging in extended conversations \cite{dong2023survey}." />
        </attvalues>
      </edge>
      <edge source="2304.12102" target="1706.03762" id="6624">
        <attvalues>
          <attvalue for="5" value=" Increasing the context length for LLMs, particularly those based on Transformer, is very expensive due to the quadratic growth of memory and computation associated with the 2-D attention matrix \cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2211.05077" target="1803.09851" id="6625">
        <attvalues>
          <attvalue for="5" value=" Previous works usually construct a shared embedding space and add different constraints to regularize the space for compositional concept learning~\cite{attrasopt,graph_comp,open_world_comp}." />
        </attvalues>
      </edge>
      <edge source="2211.05077" target="2102.01987" id="6626">
        <attvalues>
          <attvalue for="5" value=" Previous works usually construct a shared embedding space and add different constraints to regularize the space for compositional concept learning~\cite{attrasopt,graph_comp,open_world_comp}." />
        </attvalues>
      </edge>
      <edge source="2211.05077" target="2101.12609" id="6627">
        <attvalues>
          <attvalue for="5" value=" Previous works usually construct a shared embedding space and add different constraints to regularize the space for compositional concept learning~\cite{attrasopt,graph_comp,open_world_comp}." />
        </attvalues>
      </edge>
      <edge source="2211.05077" target="2107.13586" id="6629">
        <attvalues>
          <attvalue for="5" value=" The role of soft-embedding is similar to verbalizer in the general prompt-learning framework~\cite{liupengfei}." />
        </attvalues>
      </edge>
      <edge source="2409.06927" target="2310.01405" id="6630">
        <attvalues>
          <attvalue for="5" value=" The concept of activation steering \cite{turner2024steeringlanguagemodelsactivation}/representation engineering \cite{zou2023representationengineeringtopdownapproach} on transformer-based LLMs is simple, and it is remarkable that it works." />
        </attvalues>
      </edge>
      <edge source="2409.06927" target="2312.06681" id="6631">
        <attvalues>
          <attvalue for="5" value=" Other researchers \cite{panickssery2024steeringllama2contrastive} have independently explored the idea of fine-tuning as a replacement for online steering, but this work is distinctive in targeting the tuning specifically at model activations, rather than the standard method of tuning based on model output deviations from target output." />
        </attvalues>
      </edge>
      <edge source="2202.04876" target="2004.12006" id="6632">
        <attvalues>
          <attvalue for="5" value=" Domain-specific concept generalization is at the core of human cognition \cite{Yuetal2015}, and a key enabler in NLP tasks where inference and reasoning are important, eg: semantic similarity \cite{Pilehvaretal:2013, yu2014improving}, WSD \cite{Agirreetal2014} and, more recently, QA \cite{joshi2020contextualized} and NLI \cite{chen2020mining}." />
        </attvalues>
      </edge>
      <edge source="2202.04876" target="2010.01239" id="6633">
        <attvalues>
          <attvalue for="5" value=" Domain-specific concept generalization is at the core of human cognition \cite{Yuetal2015}, and a key enabler in NLP tasks where inference and reasoning are important, eg: semantic similarity \cite{Pilehvaretal:2013, yu2014improving}, WSD \cite{Agirreetal2014} and, more recently, QA \cite{joshi2020contextualized} and NLI \cite{chen2020mining}." />
        </attvalues>
      </edge>
      <edge source="2202.04876" target="2010.12813" id="6634">
        <attvalues>
          <attvalue for="5" value=" These methods, which largely rely on hand-crafted features, are still relevant today, and complement modern approaches exploiting language models (LMs), either via sequence classification \cite{Chen2021ConstructingTF}, or combining contextual, distributed, and lexico-syntactic features \cite{Yu2020STEAMST}." />
        </attvalues>
      </edge>
      <edge source="2202.04876" target="1909.01066" id="6636">
        <attvalues>
          <attvalue for="5" value=" In parallel, several works have recently focused on using LMs as zero-shot tools for solving NLP tasks, eg, commonsense, relational and analogical reasoning \cite{Petroni2019LanguageMA,bouraoui2020inducing, ushio-etal-2021-bert-is,paranjape2021prompting}, multiword expression (MWE) identification \cite{anke2021evaluating,garcia2021probing}, QA \cite{shwartz2020unsupervised, banerjee2020self}, domain labeling \cite{sainz2021ask2transformers}, or lexical substitution and simplification \cite{zhou2019bert}." />
        </attvalues>
      </edge>
      <edge source="2202.04876" target="1911.12753" id="6637">
        <attvalues>
          <attvalue for="5" value=" In parallel, several works have recently focused on using LMs as zero-shot tools for solving NLP tasks, eg, commonsense, relational and analogical reasoning \cite{Petroni2019LanguageMA,bouraoui2020inducing, ushio-etal-2021-bert-is,paranjape2021prompting}, multiword expression (MWE) identification \cite{anke2021evaluating,garcia2021probing}, QA \cite{shwartz2020unsupervised, banerjee2020self}, domain labeling \cite{sainz2021ask2transformers}, or lexical substitution and simplification \cite{zhou2019bert}." />
        </attvalues>
      </edge>
      <edge source="2202.04876" target="2106.06823" id="6638">
        <attvalues>
          <attvalue for="5" value=" In parallel, several works have recently focused on using LMs as zero-shot tools for solving NLP tasks, eg, commonsense, relational and analogical reasoning \cite{Petroni2019LanguageMA,bouraoui2020inducing, ushio-etal-2021-bert-is,paranjape2021prompting}, multiword expression (MWE) identification \cite{anke2021evaluating,garcia2021probing}, QA \cite{shwartz2020unsupervised, banerjee2020self}, domain labeling \cite{sainz2021ask2transformers}, or lexical substitution and simplification \cite{zhou2019bert}." />
        </attvalues>
      </edge>
      <edge source="2202.04876" target="2004.05483" id="6639">
        <attvalues>
          <attvalue for="5" value=" In parallel, several works have recently focused on using LMs as zero-shot tools for solving NLP tasks, eg, commonsense, relational and analogical reasoning \cite{Petroni2019LanguageMA,bouraoui2020inducing, ushio-etal-2021-bert-is,paranjape2021prompting}, multiword expression (MWE) identification \cite{anke2021evaluating,garcia2021probing}, QA \cite{shwartz2020unsupervised, banerjee2020self}, domain labeling \cite{sainz2021ask2transformers}, or lexical substitution and simplification \cite{zhou2019bert}." />
        </attvalues>
      </edge>
      <edge source="2202.04876" target="2104.06599" id="6640">
        <attvalues>
          <attvalue for="5" value=" Moreover, by tuning and manipulating natural language queries (often referred to as prompts), impressive results have been recently obtained on tasks such as semantic textual similarity, entailment, or relation classification \cite{shin2020eliciting,qin2021learning}." />
        </attvalues>
      </edge>
      <edge source="2010.14660" target="1611.00179" id="6672">
        <attvalues>
          <attvalue for="5" value=" This is similar in philosophy to dual learning in Neural Machine Translation \cite{dual_nips2016}, or unsupervised style transfer \cite{shen2017style, tian2018structured, dai2019style}." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2005.14165" id="6674">
        <attvalues>
          <attvalue for="5" value=" Large pre-trained language models (PLMs) have showcased exceptional abilities in in-context learning (ICL)~\cite{brown2020language,wang2023investigating,rubin-etal-2022-learning}, which assists the model in discerning the underlying patterns within demonstrations and make more accurate predictions~\cite{chan2022data,wu-etal-2023-self}." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2307.15411" id="6675">
        <attvalues>
          <attvalue for="5" value=" Large pre-trained language models (PLMs) have showcased exceptional abilities in in-context learning (ICL)~\cite{brown2020language,wang2023investigating,rubin-etal-2022-learning}, which assists the model in discerning the underlying patterns within demonstrations and make more accurate predictions~\cite{chan2022data,wu-etal-2023-self}." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2112.08633" id="6676">
        <attvalues>
          <attvalue for="5" value=" This promotes research aimed at improving the quality~\cite{rubin-etal-2022-learning,li-etal-2023-unified}, quantity~\cite{li2023context,choi2022prompt}, and permutations~\cite{lu-etal-2022-fantastically,tang2023found} of demonstrations." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2205.05055" id="6677">
        <attvalues>
          <attvalue for="5" value=" Large pre-trained language models (PLMs) have showcased exceptional abilities in in-context learning (ICL)~\cite{brown2020language,wang2023investigating,rubin-etal-2022-learning}, which assists the model in discerning the underlying patterns within demonstrations and make more accurate predictions~\cite{chan2022data,wu-etal-2023-self}." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2212.10375" id="6678">
        <attvalues>
          <attvalue for="5" value=" Large pre-trained language models (PLMs) have showcased exceptional abilities in in-context learning (ICL)~\cite{brown2020language,wang2023investigating,rubin-etal-2022-learning}, which assists the model in discerning the underlying patterns within demonstrations and make more accurate predictions~\cite{chan2022data,wu-etal-2023-self}." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2201.11903" id="6679">
        <attvalues>
          <attvalue for="5" value=" As a new paradigm, ICL offers compelling advantages, allowing for natural language interaction with PLMs~\cite{wei2022chain,yang2023supervised}, as well as reduced computational costs~\cite{li2023context,rubin-etal-2022-learning}." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2312.15918" id="6680">
        <attvalues>
          <attvalue for="5" value=" As a new paradigm, ICL offers compelling advantages, allowing for natural language interaction with PLMs~\cite{wei2022chain,yang2023supervised}, as well as reduced computational costs~\cite{li2023context,rubin-etal-2022-learning}." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2302.04931" id="6681">
        <attvalues>
          <attvalue for="5" value=" This promotes research aimed at improving the quality~\cite{rubin-etal-2022-learning,li-etal-2023-unified}, quantity~\cite{li2023context,choi2022prompt}, and permutations~\cite{lu-etal-2022-fantastically,tang2023found} of demonstrations." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2211.04486" id="6682">
        <attvalues>
          <attvalue for="5" value=" While promising, ICL's performance is highly dependent on provided demonstrations and templates~\cite{liu-etal-2022-makes,zhang-etal-2022-active,sorensen-etal-2022-information}, resulting in subpar and unstable performance." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2203.11364" id="6683">
        <attvalues>
          <attvalue for="5" value=" While promising, ICL's performance is highly dependent on provided demonstrations and templates~\cite{liu-etal-2022-makes,zhang-etal-2022-active,sorensen-etal-2022-information}, resulting in subpar and unstable performance." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2305.04320" id="6684">
        <attvalues>
          <attvalue for="5" value=" This promotes research aimed at improving the quality~\cite{rubin-etal-2022-learning,li-etal-2023-unified}, quantity~\cite{li2023context,choi2022prompt}, and permutations~\cite{lu-etal-2022-fantastically,tang2023found} of demonstrations." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2206.11349" id="6685">
        <attvalues>
          <attvalue for="5" value=" This promotes research aimed at improving the quality~\cite{rubin-etal-2022-learning,li-etal-2023-unified}, quantity~\cite{li2023context,choi2022prompt}, and permutations~\cite{lu-etal-2022-fantastically,tang2023found} of demonstrations." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2104.08786" id="6686">
        <attvalues>
          <attvalue for="5" value=" This promotes research aimed at improving the quality~\cite{rubin-etal-2022-learning,li-etal-2023-unified}, quantity~\cite{li2023context,choi2022prompt}, and permutations~\cite{lu-etal-2022-fantastically,tang2023found} of demonstrations." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2310.07712" id="6687">
        <attvalues>
          <attvalue for="5" value=" This promotes research aimed at improving the quality~\cite{rubin-etal-2022-learning,li-etal-2023-unified}, quantity~\cite{li2023context,choi2022prompt}, and permutations~\cite{lu-etal-2022-fantastically,tang2023found} of demonstrations." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2102.09690" id="6688">
        <attvalues>
          <attvalue for="5" value=" Other research avenues include prediction adjustment~\cite{zhao2021calibrate,han2022prototypical,fei-etal-2023-mitigating} and learning process design (eg, channel models~\cite{min-etal-2022-noisy} and meta-training frameworks~\cite{min-etal-2022-metaicl})." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2205.10183" id="6689">
        <attvalues>
          <attvalue for="5" value=" Other research avenues include prediction adjustment~\cite{zhao2021calibrate,han2022prototypical,fei-etal-2023-mitigating} and learning process design (eg, channel models~\cite{min-etal-2022-noisy} and meta-training frameworks~\cite{min-etal-2022-metaicl})." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2305.19148" id="6690">
        <attvalues>
          <attvalue for="5" value=" Other research avenues include prediction adjustment~\cite{zhao2021calibrate,han2022prototypical,fei-etal-2023-mitigating} and learning process design (eg, channel models~\cite{min-etal-2022-noisy} and meta-training frameworks~\cite{min-etal-2022-metaicl})." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2108.04106" id="6691">
        <attvalues>
          <attvalue for="5" value=" Other research avenues include prediction adjustment~\cite{zhao2021calibrate,han2022prototypical,fei-etal-2023-mitigating} and learning process design (eg, channel models~\cite{min-etal-2022-noisy} and meta-training frameworks~\cite{min-etal-2022-metaicl})." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="2110.15943" id="6692">
        <attvalues>
          <attvalue for="5" value=" Other research avenues include prediction adjustment~\cite{zhao2021calibrate,han2022prototypical,fei-etal-2023-mitigating} and learning process design (eg, channel models~\cite{min-etal-2022-noisy} and meta-training frameworks~\cite{min-etal-2022-metaicl})." />
        </attvalues>
      </edge>
      <edge source="2407.00100" target="1207.4404" id="6693">
        <attvalues>
          <attvalue for="5" value=" This operation stems from the observation that the deep features in a network are usually linearized~\cite{bengio2013better,cheung2020modals,cho2016noisy}, implying the existence of numerous semantic directions within the deep feature space, hence potentially enabling us to incorporate richer contextual knowledge without extending input length." />
        </attvalues>
      </edge>
      <edge source="2407.20729" target="2208.03274" id="6695">
        <attvalues>
          <attvalue for="5" value=" Past work such as \cite{markov2023holistic,qiu2024facilitatingpornographictextdetection} have laid foundation in terms of AI moderation using large language models, but there is still no not safe for work task in malay language." />
        </attvalues>
      </edge>
      <edge source="2407.20729" target="1706.03762" id="6696">
        <attvalues>
          <attvalue for="5" value=" The AI field, especially natural language processing, \cite{markov2023holistic} has seen remarkable progress with significant breakthroughs like transformer-based architectures \cite{vaswani2023attentionneed}, multimodality integration to chatbot applications, and reinforcement learning from human feedback." />
        </attvalues>
      </edge>
      <edge source="2407.20729" target="2403.13250" id="6697">
        <attvalues>
          <attvalue for="5" value=" Past work such as \cite{markov2023holistic,qiu2024facilitatingpornographictextdetection} have laid foundation in terms of AI moderation using large language models, but there is still no not safe for work task in malay language." />
        </attvalues>
      </edge>
      <edge source="2107.07150" target="2104.08646" id="6699">
        <attvalues>
          <attvalue for="5" value=" It has been widely applied to a variety of tasks, \eg \revised{changing text style}~\cite{reid2021lewis}, mitigating dataset biases~\cite{gardner2021competency}, explaining model behaviors~\cite{ross2020explaining}, and improving model generalization~\cite{teney2020learning,polyjuice:acl21}." />
        </attvalues>
      </edge>
      <edge source="2107.07150" target="2004.09034" id="6701">
        <attvalues>
          <attvalue for="5" value=" It has been widely applied to a variety of tasks, \eg \revised{changing text style}~\cite{reid2021lewis}, mitigating dataset biases~\cite{gardner2021competency}, explaining model behaviors~\cite{ross2020explaining}, and improving model generalization~\cite{teney2020learning,polyjuice:acl21}." />
        </attvalues>
      </edge>
      <edge source="2107.07150" target="2101.00288" id="6702">
        <attvalues>
          <attvalue for="5" value=" It has been widely applied to a variety of tasks, \eg \revised{changing text style}~\cite{reid2021lewis}, mitigating dataset biases~\cite{gardner2021competency}, explaining model behaviors~\cite{ross2020explaining}, and improving model generalization~\cite{teney2020learning,polyjuice:acl21}." />
        </attvalues>
      </edge>
      <edge source="2107.07150" target="2012.04698" id="6703">
        <attvalues>
          <attvalue for="5" value=" Existing efforts train task-specific generators, \eg training a sentiment style transferer requires instances annotated with positive and negative labels~\cite{madaan2020generate}." />
        </attvalues>
      </edge>
      <edge source="2107.07150" target="1908.04319" id="6704">
        <attvalues>
          <attvalue for="5" value="} To encourage control code following, we train \revised{the \sysname generator} with unlikelihood training~\cite{Welleck2020Neural} to penalize generations that are not aligned with designated \revised{control codes}." />
        </attvalues>
      </edge>
      <edge source="2107.07150" target="1909.12434" id="6705">
        <attvalues>
          <attvalue for="5" value=" Making such fine-grained perturbations \revised{allows for more careful evaluation and improvement of} models' language understanding~\cite{kaushik2019learning, polyjuice:acl21}." />
        </attvalues>
      </edge>
      <edge source="2107.07150" target="2104.05196" id="6706">
        <attvalues>
          <attvalue for="5" value=" In fact, on nine fine-grained and compositional StylePTB perturbations~\cite{Lyu2021StylePTBAC}, \sysname achieves performance compatible with task-specific baselines, and even outperforms them on five transfers (\sect{sec:appendix-style-transfer})." />
        </attvalues>
      </edge>
      <edge source="1912.00958" target="1809.01962" id="6708">
        <attvalues>
          <attvalue for="5" value=" SeqGAN, a generative adversarial model for sequences, has been employed for pretraining a code-switched LM \cite{SGargIS2018}." />
        </attvalues>
      </edge>
      <edge source="1912.00958" target="1409.0473" id="6709">
        <attvalues>
          <attvalue for="5" value=" The area of machine translation has witnessed sustained research efforts \cite{CHO2014,Bahdanau2015JOINTLY,GNMT}." />
        </attvalues>
      </edge>
      <edge source="1912.00958" target="1609.08144" id="6710">
        <attvalues>
          <attvalue for="5" value=" Conventional phrase based statistical machine translation (SMT) \cite{PBSMT} has shown to be outperformed by attention based recurrent encoder-decoder models \cite{GNMT} and transformer networks comprising self-attention and feed forward network blocks \cite{TRANSFORMER}." />
        </attvalues>
      </edge>
      <edge source="1912.00958" target="1706.03762" id="6711">
        <attvalues>
          <attvalue for="5" value=" Conventional phrase based statistical machine translation (SMT) \cite{PBSMT} has shown to be outperformed by attention based recurrent encoder-decoder models \cite{GNMT} and transformer networks comprising self-attention and feed forward network blocks \cite{TRANSFORMER}." />
        </attvalues>
      </edge>
      <edge source="1912.00958" target="1806.00258" id="6712">
        <attvalues>
          <attvalue for="5" value=" This observation of MT output being sensitive to the mismatch in training and inference data distributions is consistent with previous studies on MT adaptation \cite{ADAPTATIONSURVEY}." />
        </attvalues>
      </edge>
      <edge source="1912.00958" target="1805.09119" id="6713">
        <attvalues>
          <attvalue for="5" value=" In a recent work on bootstrapping natural language understanding systems using translations \cite{MTNLP}, SMT is employed for generating initial translations, followed by the use of source-target alignments to retain and resample named entities." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2212.09648" id="6714">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2011.00677" id="6715">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2009.09309" id="6716">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2009.05387" id="6717">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2104.08200" id="6719">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2104.08726" id="6720">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2105.09680" id="6721">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2203.05437" id="6722">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2205.15960" id="6723">
        <attvalues>
          <attvalue for="5" value=" NusaWrites covers 5 natural language understanding tasks (eg, emotion, sentiment classification) and one natural language generation task (eg, machine translation), and complements NusaX~\cite{winata2022nusax}---a contemporaneous work on 10 Indonesian local languages for sentiment analysis and machine translation." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2305.16171" id="6724">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2305.14716" id="6725">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2109.04607" id="6726">
        <attvalues>
          <attvalue for="5" value=" While it is critical to democratize NLP to underrepresented languages, previous works~\cite{cahyawijaya2022nusacrowd,kakwani2020indicnlpsuite,koto2020indolem,koto-koto-2020-towards,wilie2020indonlu,tacl_masakhaner,adelani2021masakhaner,cahyawijaya2021indonlg,ebrahimi2022americasnli,park2021klue,kumar2022indicnlg,winata2022nusax, adilazuarda-etal-2022-indorobusta, ogundepo2023afriqa,kabra-etal-2023-multi,song2023globalbench} have developed labeled and unlabeled corpora in the languages mainly through document translation~\cite{winata2022nusax} and online scraping~\cite{koto-etal-2021-indobertweet,koto-etal-2022-lipkey}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2203.13357" id="6727">
        <attvalues>
          <attvalue for="5" value=" We chose Indonesian local languages as our case study because of the language diversity in Indonesia, with more than 700 languages spoken but most of them are underrepresented and extremely low-resource~\cite{cohn2014local,aji-etal-2022-one}." />
        </attvalues>
      </edge>
      <edge source="2309.10661" target="2302.04023" id="6728">
        <attvalues>
          <attvalue for="5" value=" \cite{bang2023multitask} categorize Javanese (jav) and Sundanese (sun) as low-resource languages, while the others as extremely low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2104.08726" target="1901.07291" id="6729">
        <attvalues>
          <attvalue for="5" value=" Pretrained multilingual models such as XLM \cite{Lample2019CrosslingualLM}, multilingual BERT \cite[mBERT;][]{Devlin2019}, and XLM-R \cite{Conneau2020UnsupervisedCR} achieve strong cross-lingual transfer results for many languages and natural language processing (NLP) tasks." />
        </attvalues>
      </edge>
      <edge source="2104.08726" target="1911.02116" id="6731">
        <attvalues>
          <attvalue for="5" value=" Pretrained multilingual models such as XLM \cite{Lample2019CrosslingualLM}, multilingual BERT \cite[mBERT;][]{Devlin2019}, and XLM-R \cite{Conneau2020UnsupervisedCR} achieve strong cross-lingual transfer results for many languages and natural language processing (NLP) tasks." />
        </attvalues>
      </edge>
      <edge source="2104.08726" target="2010.12858" id="6732">
        <attvalues>
          <attvalue for="5" value=" As these languages are most likely to be unseen to commonly used pretrained models, practically all work evaluating unseen language performance and language adaptation methods has been limited to low-level, syntactic tasks such as part-of-speech tagging, dependency parsing, and named-entity recognition \cite{Muller2020WhenBU, wang-etal-2020-extending}." />
        </attvalues>
      </edge>
      <edge source="2104.08726" target="2012.15562" id="6733">
        <attvalues>
          <attvalue for="5" value=" Yet, even for unseen languages, performance is generally above chance, and model adaptation approaches have been shown to yield further improvements \cite{Muller2020WhenBU, Pfeiffer2020MADXAA,pfeiffer2020unks,wang-etal-2020-extending}." />
        </attvalues>
      </edge>
      <edge source="2104.08726" target="2004.13640" id="6734">
        <attvalues>
          <attvalue for="5" value=" Yet, even for unseen languages, performance is generally above chance, and model adaptation approaches have been shown to yield further improvements \cite{Muller2020WhenBU, Pfeiffer2020MADXAA,pfeiffer2020unks,wang-etal-2020-extending}." />
        </attvalues>
      </edge>
      <edge source="2104.04243" target="1907.11692" id="6737">
        <attvalues>
          <attvalue for="5" value=" Contextual sentence embeddings such as BERT~\cite{devlin2019bert} and RoBERTa~\cite{liu2019roberta}, applied to large datasets such as SNLI~\cite{snli:emnlp2015} and MultiNLI~\cite{N18-1101}, have led to near-human performance of NLI systems." />
        </attvalues>
      </edge>
      <edge source="2104.04243" target="1704.05426" id="6738">
        <attvalues>
          <attvalue for="5" value=" Contextual sentence embeddings such as BERT~\cite{devlin2019bert} and RoBERTa~\cite{liu2019roberta}, applied to large datasets such as SNLI~\cite{snli:emnlp2015} and MultiNLI~\cite{N18-1101}, have led to near-human performance of NLI systems." />
        </attvalues>
      </edge>
      <edge source="2104.04243" target="1909.02164" id="6739">
        <attvalues>
          <attvalue for="5" value=" In this paper, we study the harder problem of reasoning about tabular premises, as instantiated in datasets such as TabFact~\cite{chen2019tabfact} and InfoTabS~\cite{gupta-etal-2020-infotabs}." />
        </attvalues>
      </edge>
      <edge source="2104.04243" target="2005.06117" id="6740">
        <attvalues>
          <attvalue for="5" value=" In this paper, we study the harder problem of reasoning about tabular premises, as instantiated in datasets such as TabFact~\cite{chen2019tabfact} and InfoTabS~\cite{gupta-etal-2020-infotabs}." />
        </attvalues>
      </edge>
      <edge source="2110.12609" target="1312.3005" id="6741">
        <attvalues>
          <attvalue for="5" value=" Language models are commonly evaluated on the One Billion Word Benchmark (lm1b) \cite{lm1b}, reporting performance on perplexity \cite{radford2019language}." />
        </attvalues>
      </edge>
      <edge source="2110.12609" target="2108.07790" id="6742">
        <attvalues>
          <attvalue for="5" value=" \end{itemize} Previous work has documented examples of decontextualized hate speech \cite{ngo2021mitigating} within this dataset, as well as the impact of its destructive preprocessing \cite{radford2019language}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="1712.00409" id="6743">
        <attvalues>
          <attvalue for="5" value=" A wide range of studies have shown that the performance of a language model exhibits a notable growth pattern as the number of parameters and data size increase, following a power-law relationship~\cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,clark2022unified,zhai2022scaling,gao2023scaling,biderman2023pythia}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2001.08361" id="6744">
        <attvalues>
          <attvalue for="5" value=" In this paper, we revisit the scaling-law formulas proposed by \cite{kaplan2020scaling}, confirming that they remain generally applicable when scaling the model size up to 33B." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2010.14701" id="6745">
        <attvalues>
          <attvalue for="5" value=" A wide range of studies have shown that the performance of a language model exhibits a notable growth pattern as the number of parameters and data size increase, following a power-law relationship~\cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,clark2022unified,zhai2022scaling,gao2023scaling,biderman2023pythia}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2202.01169" id="6746">
        <attvalues>
          <attvalue for="5" value=" There have been subsequent works that study scaling laws on larger models~\cite{clark2022unified,isik2024scaling}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2106.04560" id="6747">
        <attvalues>
          <attvalue for="5" value=" A wide range of studies have shown that the performance of a language model exhibits a notable growth pattern as the number of parameters and data size increase, following a power-law relationship~\cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,clark2022unified,zhai2022scaling,gao2023scaling,biderman2023pythia}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2210.10760" id="6748">
        <attvalues>
          <attvalue for="5" value=" A wide range of studies have shown that the performance of a language model exhibits a notable growth pattern as the number of parameters and data size increase, following a power-law relationship~\cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,clark2022unified,zhai2022scaling,gao2023scaling,biderman2023pythia}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2304.01373" id="6749">
        <attvalues>
          <attvalue for="5" value=" A wide range of studies have shown that the performance of a language model exhibits a notable growth pattern as the number of parameters and data size increase, following a power-law relationship~\cite{hestness2017deep,kaplan2020scaling,henighan2020scaling,clark2022unified,zhai2022scaling,gao2023scaling,biderman2023pythia}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2203.15556" id="6750">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{hoffmann2022training} claimed that the training data size should be scaled much more than the recommendation in \cite{kaplan2020scaling}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2303.08774" id="6751">
        <attvalues>
          <attvalue for="5" value=" As mentioned in the GPT-4 technical report~\cite{achiam2023gpt}, some aspects of GPT-4's performance can be accurately predicted based on models trained with no more than 1/1,000th the compute of GPT-4." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2402.04177" id="6752">
        <attvalues>
          <attvalue for="5" value=" There have been subsequent works that study scaling laws on larger models~\cite{clark2022unified,isik2024scaling}." />
        </attvalues>
      </edge>
      <edge source="2403.06563" target="2401.02954" id="6753">
        <attvalues>
          <attvalue for="5" value=" \cite{bi2024deepseek} suggested that the optimal batch size depends only on the compute budget rather than the loss value." />
        </attvalues>
      </edge>
      <edge source="1911.03766" target="1804.05392" id="6754">
        <attvalues>
          <attvalue for="5" value=" Our model builds on recent ideas in span selection models \cite{lee-etal-2018-higher, he-etal-2018-jointly, ouchi-etal-2018-span}, used in this work for the multi-sentence argument linking task for \datasetname and for several other event-based datasets \cite[AIDA Phase 1]{gerber-chai-2012-semantic, pradhan-etal-2013-towards, pavlick-etal-2016-gun}." />
        </attvalues>
      </edge>
      <edge source="2402.01383" target="1810.05995" id="6755">
        <attvalues>
          <attvalue for="5" value=" They have been criticized for low correlation with human judgments \cite{sulem-etal-2018-bleu}, as surface-level matching cannot reliably evaluate text." />
        </attvalues>
      </edge>
      <edge source="2402.01383" target="2106.11520" id="6757">
        <attvalues>
          <attvalue for="5" value=" After the rise of deep learning, model-based evaluation metrics like BERTScore \cite{zhang2019bertscore} and BARTScore \cite{yuan2021bartscore} have been continuously proposed and gradually adopted to evaluate the overall quality or various specific aspects of generated outputs (eg, fluency, coherence, coverage, faithfulness, etc)." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2303.03329" id="6759">
        <attvalues>
          <attvalue for="5" value=" Over the past few decades, the speech community has devoted considerable time and effort to manually recording, collecting, and annotating a vast amount of speech data with corresponding segmentation, transcription and speaker labels, which has significantly advanced the performance of various speech technologies, such as automatic speech recognition (ASR) \cite{asr_survey, radford2023robust, zhang2023google}, text-to-speech synthesis (TTS) \cite{tts_survey, kharitonov2023speak, wang2023neural, jiang2023mega}, speaker verification (SV) \cite{sv_survey, caron2021emerging}, and speech enhancement (SE) \cite{se_survey}." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2106.15561" id="6760">
        <attvalues>
          <attvalue for="5" value=" Over the past few decades, the speech community has devoted considerable time and effort to manually recording, collecting, and annotating a vast amount of speech data with corresponding segmentation, transcription and speaker labels, which has significantly advanced the performance of various speech technologies, such as automatic speech recognition (ASR) \cite{asr_survey, radford2023robust, zhang2023google}, text-to-speech synthesis (TTS) \cite{tts_survey, kharitonov2023speak, wang2023neural, jiang2023mega}, speaker verification (SV) \cite{sv_survey, caron2021emerging}, and speech enhancement (SE) \cite{se_survey}." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2006.11477" id="6761">
        <attvalues>
          <attvalue for="5" value=" For the missing annotation issue, previous research has focused primarily on unsupervised pre-training methods \cite{wav2vec, baevski2020wav2vec, hubert, wavlm, borsos2023audiolm}." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2305.18802" id="6762">
        <attvalues>
          <attvalue for="5" value=" Regarding the second issue, prior research \cite{koizumi2023libritts, valentiniinvestigating} has proposed leveraging speech enhancement methods to handle unstable speech quality in TTS model construction." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2212.00406" id="6763">
        <attvalues>
          <attvalue for="5" value=" To provide high-quality speech and annotation, we incorporate competitive models into the proposed AutoPrep framework, including the Band-Split RNN (BSRNN) speech enhancement model \cite{bsrnn}, WeSpeaker speaker embedding model \cite{wespeaker}, personalized BSRNN (pBSRNN) target speech extraction (TSE) model \cite{yu2023tspeech}, and a 60k-hour trained multilingual conformer-based \cite{conformer} RNN-Transducer \cite{rnnt} ASR system." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2210.17016" id="6764">
        <attvalues>
          <attvalue for="5" value=" To provide high-quality speech and annotation, we incorporate competitive models into the proposed AutoPrep framework, including the Band-Split RNN (BSRNN) speech enhancement model \cite{bsrnn}, WeSpeaker speaker embedding model \cite{wespeaker}, personalized BSRNN (pBSRNN) target speech extraction (TSE) model \cite{yu2023tspeech}, and a 60k-hour trained multilingual conformer-based \cite{conformer} RNN-Transducer \cite{rnnt} ASR system." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2005.08100" id="6765">
        <attvalues>
          <attvalue for="5" value=" To provide high-quality speech and annotation, we incorporate competitive models into the proposed AutoPrep framework, including the Band-Split RNN (BSRNN) speech enhancement model \cite{bsrnn}, WeSpeaker speaker embedding model \cite{wespeaker}, personalized BSRNN (pBSRNN) target speech extraction (TSE) model \cite{yu2023tspeech}, and a 60k-hour trained multilingual conformer-based \cite{conformer} RNN-Transducer \cite{rnnt} ASR system." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="1211.3711" id="6766">
        <attvalues>
          <attvalue for="5" value=" To provide high-quality speech and annotation, we incorporate competitive models into the proposed AutoPrep framework, including the Band-Split RNN (BSRNN) speech enhancement model \cite{bsrnn}, WeSpeaker speaker embedding model \cite{wespeaker}, personalized BSRNN (pBSRNN) target speech extraction (TSE) model \cite{yu2023tspeech}, and a 60k-hour trained multilingual conformer-based \cite{conformer} RNN-Transducer \cite{rnnt} ASR system." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="1909.01700" id="6767">
        <attvalues>
          <attvalue for="5" value=" In addition, we use the processed data to train a DurIAN \cite{yu2019durian} multi-speaker TTS system to evaluate the proposed framework." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2110.03370" id="6768">
        <attvalues>
          <attvalue for="5" value=" Experiments conducted on the open-sourced WenetSpeech \cite{zhang2022wenetspeech} and our self-collected AutoPrepWild speech data demonstrate that the proposed AutoPrep framework can generate processed data speech with similar DNSMOS and PDNSMOS scores compared to the open-sourced AIShell-3 \cite{shi2020aishell} and LibriTTS \cite{zen2019libritts} datasets." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="2010.11567" id="6769">
        <attvalues>
          <attvalue for="5" value=" Experiments conducted on the open-sourced WenetSpeech \cite{zhang2022wenetspeech} and our self-collected AutoPrepWild speech data demonstrate that the proposed AutoPrep framework can generate processed data speech with similar DNSMOS and PDNSMOS scores compared to the open-sourced AIShell-3 \cite{shi2020aishell} and LibriTTS \cite{zen2019libritts} datasets." />
        </attvalues>
      </edge>
      <edge source="2309.13905" target="1904.02882" id="6770">
        <attvalues>
          <attvalue for="5" value=" Experiments conducted on the open-sourced WenetSpeech \cite{zhang2022wenetspeech} and our self-collected AutoPrepWild speech data demonstrate that the proposed AutoPrep framework can generate processed data speech with similar DNSMOS and PDNSMOS scores compared to the open-sourced AIShell-3 \cite{shi2020aishell} and LibriTTS \cite{zen2019libritts} datasets." />
        </attvalues>
      </edge>
      <edge source="2212.09422" target="2209.11055" id="6788">
        <attvalues>
          <attvalue for="5" value=" Huggingfaces Sentence Transformer Finetuning (SetFit) \cite{tunstall2022efficient} allows for such a small amount of labeled data while achieving impressive classification results that unsupervised methods are heavily challenged." />
        </attvalues>
      </edge>
      <edge source="2212.09422" target="1908.10084" id="6789">
        <attvalues>
          <attvalue for="5" value=" By leveraging pre-trained sentence transformers \cite{reimers-2019-sentence-bert} and class-based term frequency inverse document frequency (tf-idf) for topic extraction, we can generate coherent topics with only a few labeled documents per class." />
        </attvalues>
      </edge>
      <edge source="2310.18696" target="1905.06316" id="6791">
        <attvalues>
          <attvalue for="5" value=" Existing research probed LLMs for their encoding of various linguistic properties such as agreement~\cite{jawahar2019does}, word order and sentence structure~\cite{tenney_what_2018,hewitt_structural_2019}, co-reference~\cite{tenney_bert_2019}, semantics~\cite{ettinger_what_2020} and multilinguality~\cite{ravishankar_multilingual_2019, &#09;libovicky_language_2020}." />
        </attvalues>
      </edge>
      <edge source="2310.18696" target="1905.05950" id="6792">
        <attvalues>
          <attvalue for="5" value=" Taking a step further, \cite{tenney_bert_2019} and \cite{clark_what_2019} studied where linguistic information is encoded in LLMs by probing different layers." />
        </attvalues>
      </edge>
      <edge source="2310.18696" target="1907.13528" id="6793">
        <attvalues>
          <attvalue for="5" value=" Existing research probed LLMs for their encoding of various linguistic properties such as agreement~\cite{jawahar2019does}, word order and sentence structure~\cite{tenney_what_2018,hewitt_structural_2019}, co-reference~\cite{tenney_bert_2019}, semantics~\cite{ettinger_what_2020} and multilinguality~\cite{ravishankar_multilingual_2019, &#09;libovicky_language_2020}." />
        </attvalues>
      </edge>
      <edge source="2310.18696" target="1906.04341" id="6794">
        <attvalues>
          <attvalue for="5" value=" Taking a step further, \cite{tenney_bert_2019} and \cite{clark_what_2019} studied where linguistic information is encoded in LLMs by probing different layers." />
        </attvalues>
      </edge>
      <edge source="2310.18696" target="1907.11692" id="6795">
        <attvalues>
          <attvalue for="5" value=" We study joint encoding patterns within both a monolingual and multilingual LLM, namely RoBERTa~\cite{liu_roberta_2019} and XLM-R~\cite{conneau_unsupervised_2020}." />
        </attvalues>
      </edge>
      <edge source="2310.18696" target="1911.02116" id="6796">
        <attvalues>
          <attvalue for="5" value=" We study joint encoding patterns within both a monolingual and multilingual LLM, namely RoBERTa~\cite{liu_roberta_2019} and XLM-R~\cite{conneau_unsupervised_2020}." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2303.08774" id="6815">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art multilingual LLMs such as GPT-4 \cite{achiam2023gpt}, Llama~\cite{touvron2023llama}, and Gemini~\cite{team2023gemini} cater to this global user base." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2307.09288" id="6816">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art multilingual LLMs such as GPT-4 \cite{achiam2023gpt}, Llama~\cite{touvron2023llama}, and Gemini~\cite{team2023gemini} cater to this global user base." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2312.11805" id="6817">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art multilingual LLMs such as GPT-4 \cite{achiam2023gpt}, Llama~\cite{touvron2023llama}, and Gemini~\cite{team2023gemini} cater to this global user base." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2308.14921" id="6818">
        <attvalues>
          <attvalue for="5" value=" However, many recent studies have revealed undesirable biases and stereotypes in these models \cite{kotek2023gender, kamruzzaman2024global}." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2004.09456" id="6819">
        <attvalues>
          <attvalue for="5" value=" Some datasets have been proposed to measure these biases, predominantly in English \cite{nadeem2021stereoset, kamruzzaman-etal-2024-investigating, nangia2020crows}." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2403.20147" id="6820">
        <attvalues>
          <attvalue for="5" value=" There are studies in other languages like French \cite{neveol2022french}, Hindi \cite{sahoo2024indibias}, Italian \cite{sanguinetti2020haspeede}, and Arabic \cite{lauscher2020araweat}." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2011.01575" id="6821">
        <attvalues>
          <attvalue for="5" value=" There are studies in other languages like French \cite{neveol2022french}, Hindi \cite{sahoo2024indibias}, Italian \cite{sanguinetti2020haspeede}, and Arabic \cite{lauscher2020araweat}." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2407.03536" id="6822">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{sadhu2024social} proposed a dataset concentrating on gender and religion biases through persona-based experiments, such as adopting the role of a typical Bengali person." />
        </attvalues>
      </edge>
      <edge source="2409.11638" target="2407.06432" id="6823">
        <attvalues>
          <attvalue for="5" value=" \cite{sadhu2024empirical} found stereotypical emotional associations between genders in Bengali, using an emotion dataset." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="2203.15556" id="6824">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs, \cite{gopher,chinchilla,scao2022bloom,llama}) are seeing widespread adoption thanks to the fact that they can perform many language tasks and generate coherent long-form text." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="2211.05100" id="6825">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs, \cite{gopher,chinchilla,scao2022bloom,llama}) are seeing widespread adoption thanks to the fact that they can perform many language tasks and generate coherent long-form text." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="2009.11462" id="6828">
        <attvalues>
          <attvalue for="5" value=" In addition, we may want models to output text with specific properties, such as having a positive sentiment, a certain writing style, etc Typically, LLMs pre-trained on uncurated large-scale text corpora can generate text that does not have these desired attributes~\cite{wallace2019triggers,RealToxicityPrompts}, which motivates the need for techniques that enable controllable text generation." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="1909.05858" id="6830">
        <attvalues>
          <attvalue for="5" value=" Alternatively, an LLM can be trained with ``control codes''~\cite{ctrl,quark} that indicate text characteristics and can be used to induce the LLM to generate content with those characteristics." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="2205.13636" id="6831">
        <attvalues>
          <attvalue for="5" value=" Alternatively, an LLM can be trained with ``control codes''~\cite{ctrl,quark} that indicate text characteristics and can be used to induce the LLM to generate content with those characteristics." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="2204.11586" id="6833">
        <attvalues>
          <attvalue for="5" value=" One way to avoid the cost and shortcomings of additional training is to instead modify the decoding procedure used to generate text from a language model \cite{Chaffin_2022}." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="2105.03023" id="6836">
        <attvalues>
          <attvalue for="5" value=" During decoding, \cite{gedi} and \cite{dexperts} process signals from auxiliary generative models, whereas \cite{fudge} and \cite{sitdikov2022classifiers} evaluate intermediate sequences." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="2104.05218" id="6837">
        <attvalues>
          <attvalue for="5" value=" During decoding, \cite{gedi} and \cite{dexperts} process signals from auxiliary generative models, whereas \cite{fudge} and \cite{sitdikov2022classifiers} evaluate intermediate sequences." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="2205.07276" id="6838">
        <attvalues>
          <attvalue for="5" value=" During decoding, \cite{gedi} and \cite{dexperts} process signals from auxiliary generative models, whereas \cite{fudge} and \cite{sitdikov2022classifiers} evaluate intermediate sequences." />
        </attvalues>
      </edge>
      <edge source="2310.09520" target="1902.08654" id="6839">
        <attvalues>
          <attvalue for="5" value=" Despite these benefits, weighted decoding can significantly increase the cost of decoding and often underperforms methods that involve further training~\cite{see-etal-2019-makes}." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="2111.05988" id="6840">
        <attvalues>
          <attvalue for="5" value=" A traditional cross-lingual information retrieval (CLIR) system consists of two components: machine translation and monolingual information retrieval~\cite{nie2010}." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="1310.4546" id="6841">
        <attvalues>
          <attvalue for="5" value=" The first class of approaches uses pretrained word representations or embeddings, such as word2vec~\cite{tomas2013} and GloVe~\cite{pennington2014glove}, directly to improve IR models." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="1706.06613" id="6843">
        <attvalues>
          <attvalue for="5" value=" These methods have shown impressive results on monolingual IR datasets~\cite{XiongSIGIR17,guocikm16,mostafaSIGIR17}." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="1711.08611" id="6844">
        <attvalues>
          <attvalue for="5" value=" These methods have shown impressive results on monolingual IR datasets~\cite{XiongSIGIR17,guocikm16,mostafaSIGIR17}." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="1704.08803" id="6845">
        <attvalues>
          <attvalue for="5" value=" These methods have shown impressive results on monolingual IR datasets~\cite{XiongSIGIR17,guocikm16,mostafaSIGIR17}." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="1802.05365" id="6848">
        <attvalues>
          <attvalue for="5" value=" These models have outperformed traditional word embeddings on various NLP tasks~\cite{xlnet2019,devlin-etal-2019-bert,peters-etal-2018-deep,Lan2019ALBERTAL}." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="1909.11942" id="6849">
        <attvalues>
          <attvalue for="5" value=" These models have outperformed traditional word embeddings on various NLP tasks~\cite{xlnet2019,devlin-etal-2019-bert,peters-etal-2018-deep,Lan2019ALBERTAL}." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="1905.09217" id="6850">
        <attvalues>
          <attvalue for="5" value=" Therefore, several recent works have successfully applied BERT pretrained models for monolingual IR~\cite{daiSIGIR19,akkalyoncu-yilmaz-etal-2019-applying} and passage re-ranking~\cite{Nogueira19}." />
        </attvalues>
      </edge>
      <edge source="2004.13005" target="1901.04085" id="6851">
        <attvalues>
          <attvalue for="5" value=" Therefore, several recent works have successfully applied BERT pretrained models for monolingual IR~\cite{daiSIGIR19,akkalyoncu-yilmaz-etal-2019-applying} and passage re-ranking~\cite{Nogueira19}." />
        </attvalues>
      </edge>
      <edge source="2307.16833" target="1511.06709" id="6852">
        <attvalues>
          <attvalue for="5" value=" To address the data scarcity problem, back-translation-based methods~\cite{sennrich-etal-2016-improving, edunov-etal-2018-understanding, hoang-etal-2018-iterative, sugiyama-yoshinaga-2019-data, kumar-etal-2020-data} have been widely adopted." />
        </attvalues>
      </edge>
      <edge source="2307.16833" target="2005.03642" id="6853">
        <attvalues>
          <attvalue for="5" value=" In this case, out-of-domain issues such as hallucinations~\cite{wang-sennrich-2020-exposure, muller-etal-2020-domain}, are more likely to occur, leading to difficulties in acquiring high-quality synthetic data." />
        </attvalues>
      </edge>
      <edge source="2307.16833" target="2005.14165" id="6854">
        <attvalues>
          <attvalue for="5" value=" Recently, with the remarkable advancements in Natural Language Generation models~\cite{NEURIPS2020_1457c0d6}, research on utilizing large-scale language generation models for data augmentation~\cite{yoo-etal-2021-gpt3mix-leveraging} has been conducted." />
        </attvalues>
      </edge>
      <edge source="2307.16833" target="2104.08826" id="6855">
        <attvalues>
          <attvalue for="5" value=" Recently, with the remarkable advancements in Natural Language Generation models~\cite{NEURIPS2020_1457c0d6}, research on utilizing large-scale language generation models for data augmentation~\cite{yoo-etal-2021-gpt3mix-leveraging} has been conducted." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2305.10403" id="6861">
        <attvalues>
          <attvalue for="5" value=" Since the advent of open-domain generative systems driven by large language models (LLMs)~\cite{Anil2023PaLM2T,chatgpt,OpenAI2023GPT4TR}, addressing the coherent generation of potentially inaccurate or fabricated content has been a persistent challenge~\cite{Rawte2023ASO,Ye2023CognitiveMA,Zhang2023SirensSI}." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2303.08774" id="6862">
        <attvalues>
          <attvalue for="5" value=" Since the advent of open-domain generative systems driven by large language models (LLMs)~\cite{Anil2023PaLM2T,chatgpt,OpenAI2023GPT4TR}, addressing the coherent generation of potentially inaccurate or fabricated content has been a persistent challenge~\cite{Rawte2023ASO,Ye2023CognitiveMA,Zhang2023SirensSI}." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2309.05922" id="6863">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Moving beyond general discussions on text hallucinations~\cite{Zhang2023SirensSI,Ye2023CognitiveMA,Rawte2023ASO}, our study delves deeper into the attribution of large language models." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2309.06794" id="6864">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Moving beyond general discussions on text hallucinations~\cite{Zhang2023SirensSI,Ye2023CognitiveMA,Rawte2023ASO}, our study delves deeper into the attribution of large language models." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2309.01219" id="6865">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Moving beyond general discussions on text hallucinations~\cite{Zhang2023SirensSI,Ye2023CognitiveMA,Rawte2023ASO}, our study delves deeper into the attribution of large language models." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2309.07852" id="6866">
        <attvalues>
          <attvalue for="5" value=" This form of attribution is particularly lacking in high-risk professional fields such as medicine and law, with research revealing a significant number of incomplete attributions (35\% and 31\%, respectively); moreover, many attributions were derived from unreliable sources, with 51\% of them being assessed as unreliable by experts~\cite{Malaviya2023ExpertQAEQ}." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2306.01116" id="6867">
        <attvalues>
          <attvalue for="5" value="8pt, ] The essence of the hallucination problem may stem from the fact that pre-trained models are sourced from vast, unfiltered real-world texts~\cite{Penedo2023TheRD}." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2204.05862" id="6869">
        <attvalues>
          <attvalue for="5" value=" Even after utilizing reinforcement learning from human feedback~\cite{Ouyang2022TrainingLM}, models can still exhibit external hallucinations~\cite{Bai2022TrainingAH}." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2201.08239" id="6870">
        <attvalues>
          <attvalue for="5" value=" To address the issue of external hallucinations, researchers have begun to employ measures like external references to enhance the authenticity and reliability of chatbots~\cite{Thoppilan2022LaMDALM,Menick2022TeachingLM,nakano2021webgpt}." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2203.11147" id="6871">
        <attvalues>
          <attvalue for="5" value=" To address the issue of external hallucinations, researchers have begun to employ measures like external references to enhance the authenticity and reliability of chatbots~\cite{Thoppilan2022LaMDALM,Menick2022TeachingLM,nakano2021webgpt}." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2309.09401" id="6875">
        <attvalues>
          <attvalue for="5" value="6\% of the time, the suggested references were only present 14\% of the time~\cite{zuccon2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2311.03731" target="2305.14627" id="6876">
        <attvalues>
          <attvalue for="5" value=" But retrieval does not inherently equate to attribution~\cite{gao2023enabling}." />
        </attvalues>
      </edge>
      <edge source="2309.14462" target="2110.07205" id="6877">
        <attvalues>
          <attvalue for="5" value=" Large deep learning models have recently achieved great success on speech recognition tasks \cite{speechT5, speech_XLS-R}." />
        </attvalues>
      </edge>
      <edge source="2107.04691" target="2001.09694" id="6878">
        <attvalues>
          <attvalue for="5" value=" In recent years, extractive text-based machine comprehension (MC) has advanced significantly with many systems \cite{Zhang2020RetrospectiveRF, Lan2020ALBERT:, DBLP:conf/aaai/0001WZDZ020, yamada-etal-2020-luke, clark2020electra} outperforming human performance." />
        </attvalues>
      </edge>
      <edge source="2107.04691" target="1606.05250" id="6879">
        <attvalues>
          <attvalue for="5" value=" The Spoken SQuAD dataset takes the SQuAD MC dataset \cite{rajpurkar-etal-2016-squad} and passes the passages through a text-to-speech system to generate spoken equivalents." />
        </attvalues>
      </edge>
      <edge source="2107.04691" target="1608.06378" id="6880">
        <attvalues>
          <attvalue for="5" value=" Several deep models have been evaluated on this task: \cite{Tseng+2016} uses attention-based RNN, \cite{fang-hier} uses tree-structured RNN and \cite{chung-etal-2018-supervised} explores transfer learning from text-based question-answering." />
        </attvalues>
      </edge>
      <edge source="2107.04691" target="1608.07775" id="6881">
        <attvalues>
          <attvalue for="5" value=" Several deep models have been evaluated on this task: \cite{Tseng+2016} uses attention-based RNN, \cite{fang-hier} uses tree-structured RNN and \cite{chung-etal-2018-supervised} explores transfer learning from text-based question-answering." />
        </attvalues>
      </edge>
      <edge source="2107.04691" target="1711.05345" id="6882">
        <attvalues>
          <attvalue for="5" value=" Several deep models have been evaluated on this task: \cite{Tseng+2016} uses attention-based RNN, \cite{fang-hier} uses tree-structured RNN and \cite{chung-etal-2018-supervised} explores transfer learning from text-based question-answering." />
        </attvalues>
      </edge>
      <edge source="2107.04691" target="1804.00320" id="6883">
        <attvalues>
          <attvalue for="5" value=" \cite{Lee2018} demonstrates that ASR errors have catastrophic impact on machine comprehension." />
        </attvalues>
      </edge>
      <edge source="2107.04691" target="1910.11559" id="6884">
        <attvalues>
          <attvalue for="5" value=" SpeechBERT \cite{Chuang2020} is an end-to-end SQA model trained on Spoken SQuAD." />
        </attvalues>
      </edge>
      <edge source="2107.04691" target="1808.02280" id="6885">
        <attvalues>
          <attvalue for="5" value=" \cite{odsqa} release a large scale SQA dataset with real speech but this dataset is in Chinese." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1809.05288" id="6904">
        <attvalues>
          <attvalue for="5" value=" At the same time, current neural NLG models struggle to replicate the high language diversity of the training sentences present in these large datasets, and instead they learn to produce the same generic type of sentences as with considerably less training data~\cite{deriu2018syntactic,juraska2018characterizing,duvsek2019evaluating}." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1901.07931" id="6905">
        <attvalues>
          <attvalue for="5" value=" At the same time, current neural NLG models struggle to replicate the high language diversity of the training sentences present in these large datasets, and instead they learn to produce the same generic type of sentences as with considerably less training data~\cite{deriu2018syntactic,juraska2018characterizing,duvsek2019evaluating}." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1712.04034" id="6906">
        <attvalues>
          <attvalue for="5" value=" Video games are a vast entertainment topic that can naturally be discussed in a casual conversation, similar to movies and music, yet in the dialogue systems community it does not enjoy popularity anywhere close to that of the latter two topics \cite{fazel2017learning,li2017end,moghe2018towards,shah2018building,khatri2018advancing}." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1809.08205" id="6908">
        <attvalues>
          <attvalue for="5" value=" Video games are a vast entertainment topic that can naturally be discussed in a casual conversation, similar to movies and music, yet in the dialogue systems community it does not enjoy popularity anywhere close to that of the latter two topics \cite{fazel2017learning,li2017end,moghe2018towards,shah2018building,khatri2018advancing}." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1801.04871" id="6909">
        <attvalues>
          <attvalue for="5" value=" Video games are a vast entertainment topic that can naturally be discussed in a casual conversation, similar to movies and music, yet in the dialogue systems community it does not enjoy popularity anywhere close to that of the latter two topics \cite{fazel2017learning,li2017end,moghe2018towards,shah2018building,khatri2018advancing}." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1812.10757" id="6910">
        <attvalues>
          <attvalue for="5" value=" Video games are a vast entertainment topic that can naturally be discussed in a casual conversation, similar to movies and music, yet in the dialogue systems community it does not enjoy popularity anywhere close to that of the latter two topics \cite{fazel2017learning,li2017end,moghe2018towards,shah2018building,khatri2018advancing}." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1508.01755" id="6911">
        <attvalues>
          <attvalue for="5" value=" While they certainly can be a topic of a casual conversation, the existing restaurant datasets~\cite{stent2004trainable,gavsic2008training,mairesse2010phrase,howcroft2013enhancing,wen2015stochastic,nayak2017plan} are geared more toward a task-oriented dialogue where a system tries to narrow down a restaurant based on the user's preferences and ultimately give a recommendation." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1508.01745" id="6912">
        <attvalues>
          <attvalue for="5" value=" Other domains have been represented by task-oriented datasets with multiple DA types, for example the Hotel, Laptop, and TV datasets~\cite{wen2015semantically,wen2016multi}." />
        </attvalues>
      </edge>
      <edge source="1910.12129" target="1603.01232" id="6913">
        <attvalues>
          <attvalue for="5" value=" Other domains have been represented by task-oriented datasets with multiple DA types, for example the Hotel, Laptop, and TV datasets~\cite{wen2015semantically,wen2016multi}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1901.07291" id="6937">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models~(PLMs), such as mBERT~\cite{devlin2019bert} and XLM~\cite{conneau2019cross}, have been shown to be effective on a variety of cross-lingual benchmarks~\cite{conneau2018xnli,artetxe2019cross,hu2020xtreme}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="2003.11080" id="6940">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models~(PLMs), such as mBERT~\cite{devlin2019bert} and XLM~\cite{conneau2019cross}, have been shown to be effective on a variety of cross-lingual benchmarks~\cite{conneau2018xnli,artetxe2019cross,hu2020xtreme}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1911.02116" id="6941">
        <attvalues>
          <attvalue for="5" value=" Moreover, XLM-R~\cite{conneau2019unsupervised} further demonstrates that it is possible to have a single large model for all languages, without sacrificing per-language performance." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1503.02531" id="6942">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation~(KD)~\cite{hinton2015distilling}, one of the model compression techniques, has been successfully used for compressing monolingual PLMs~\cite{tang2019distilling,sun2019patient,sanh2019distilbert,tsai2019small,jiao2019tinybert,turc2019well,sunmobilebert,wang2020minilm}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1903.12136" id="6943">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation~(KD)~\cite{hinton2015distilling}, one of the model compression techniques, has been successfully used for compressing monolingual PLMs~\cite{tang2019distilling,sun2019patient,sanh2019distilbert,tsai2019small,jiao2019tinybert,turc2019well,sunmobilebert,wang2020minilm}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1908.09355" id="6944">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation~(KD)~\cite{hinton2015distilling}, one of the model compression techniques, has been successfully used for compressing monolingual PLMs~\cite{tang2019distilling,sun2019patient,sanh2019distilbert,tsai2019small,jiao2019tinybert,turc2019well,sunmobilebert,wang2020minilm}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1909.10351" id="6947">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation~(KD)~\cite{hinton2015distilling}, one of the model compression techniques, has been successfully used for compressing monolingual PLMs~\cite{tang2019distilling,sun2019patient,sanh2019distilbert,tsai2019small,jiao2019tinybert,turc2019well,sunmobilebert,wang2020minilm}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="2002.10957" id="6948">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation~(KD)~\cite{hinton2015distilling}, one of the model compression techniques, has been successfully used for compressing monolingual PLMs~\cite{tang2019distilling,sun2019patient,sanh2019distilbert,tsai2019small,jiao2019tinybert,turc2019well,sunmobilebert,wang2020minilm}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="2004.05686" id="6949">
        <attvalues>
          <attvalue for="5" value=" In this paper, we focus on the cross-lingual scenario where there is no task training data in the target languages, which is different from previous works~\cite{tsai2019small,mukherjee2020xtremedistil}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1706.03762" id="6950">
        <attvalues>
          <attvalue for="5" value=" Specifically, we take mBERT as an example and investigate how to effectively and efficiently distill the cross-lingual generalization ability of it into a transformer~\cite{vaswani2017attention} based student." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1906.01502" id="6951">
        <attvalues>
          <attvalue for="5" value=" Then, we freeze the inherited embeddings during the distillation process since they are shown to be important for the cross-lingual generalization ability of mBERT~\cite{pires2019multilingual,wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2103.06418" target="1904.09077" id="6952">
        <attvalues>
          <attvalue for="5" value=" Then, we freeze the inherited embeddings during the distillation process since they are shown to be important for the cross-lingual generalization ability of mBERT~\cite{pires2019multilingual,wu2019beto}." />
        </attvalues>
      </edge>
      <edge source="2306.14910" target="1807.07987" id="6953">
        <attvalues>
          <attvalue for="5" value=" The celebrated supervised learning framework \cite{vapnik1999overview,lecun2015deep} was designed and developed exactly for this paradigm." />
        </attvalues>
      </edge>
      <edge source="2306.14910" target="2304.12210" id="6954">
        <attvalues>
          <attvalue for="5" value=" This development is both exciting and aligns with the longstanding goal of the weakly-, semi-, and self-supervised learning community \cite{zhu2005semi,zhou2018brief,gui2023survey,balestriero2023cookbook}." />
        </attvalues>
      </edge>
      <edge source="2312.10323" target="2112.08348" id="6955">
        <attvalues>
          <attvalue for="5" value=" \cite{khashabi2021} formalizes the notion of waywardness by proposing that for some arbitrary task, $t$, and for an arbitrary discrete prompt, $p_D$, there exists an analogous, unintelligible continuous prompt, $p_C$, that corresponds to the former and will perform with comparable satisfaction on $t$." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="1911.02116" id="6961">
        <attvalues>
          <attvalue for="5" value=" Multilingual models like mBERT \cite{devlin-etal-2019-bert} and XLM-R \cite{conneau-etal-2020-unsupervised} have been recently shown to be surprisingly effective for zero-shot transfer \cite{pires-etal-2019-multilingual} \cite{wu-dredze-2019-beto}, where on fine-tuning for a task on one or a few languages, called {pivots}, they can perform well on languages unseen during training." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="1906.01502" id="6962">
        <attvalues>
          <attvalue for="5" value=" Multilingual models like mBERT \cite{devlin-etal-2019-bert} and XLM-R \cite{conneau-etal-2020-unsupervised} have been recently shown to be surprisingly effective for zero-shot transfer \cite{pires-etal-2019-multilingual} \cite{wu-dredze-2019-beto}, where on fine-tuning for a task on one or a few languages, called {pivots}, they can perform well on languages unseen during training." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="1904.09077" id="6963">
        <attvalues>
          <attvalue for="5" value=" Multilingual models like mBERT \cite{devlin-etal-2019-bert} and XLM-R \cite{conneau-etal-2020-unsupervised} have been recently shown to be surprisingly effective for zero-shot transfer \cite{pires-etal-2019-multilingual} \cite{wu-dredze-2019-beto}, where on fine-tuning for a task on one or a few languages, called {pivots}, they can perform well on languages unseen during training." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="2005.09093" id="6964">
        <attvalues>
          <attvalue for="5" value=" The zero-shot performance however, is often not uniform across the languages and the multilingual models turn out to be much less effective for low resource languages \cite{wu-dredze-2020-languages, lauscher-etal-2020-zero} and the languages that are typologically distant from the pivots \cite{lauscher-etal-2020-zero}." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="1912.07840" id="6965">
        <attvalues>
          <attvalue for="5" value=" What affects the zero-shot transfer across different languages is a subject of considerable interest and importance~\cite{wang2019cross, pires-etal-2019-multilingual, wu-dredze-2019-beto, lauscher-etal-2020-zero}, however there is little conclusive evidence and a few papers even show contradictory findings." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="2110.08875" id="6966">
        <attvalues>
          <attvalue for="5" value=" As \cite{srinivasan2021predicting} shows, accurate performance predictors can also help us build better and fairer multilingual models by suggesting data labeling strategies." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="2105.05975" id="6967">
        <attvalues>
          <attvalue for="5" value=" Along similar lines \cite{srinivasan2021predicting} and \cite{dolicki2021analysing} explore zero-shot performance prediction with a larger set of features and different regression techniques." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="2104.07412" id="6968">
        <attvalues>
          <attvalue for="5" value=" For instance, for most tasks in the popular XTREME-R \cite{ruder2021xtreme} benchmark, there are data points for 7-11 languages." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="1809.00366" id="6970">
        <attvalues>
          <attvalue for="5" value=" First, we experiment with different multi-task learning approaches, such as Group Lasso \cite{yuan2006model}, Collective Matrix Factorization \cite{cortes2018cold}, Multi-Task Deep Gaussian Process Regression \cite{NIPS2007_66368270} and Meta Agnostic Meta Learning \cite{finn2017model} for 11 tasks." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="1703.03400" id="6971">
        <attvalues>
          <attvalue for="5" value=" First, we experiment with different multi-task learning approaches, such as Group Lasso \cite{yuan2006model}, Collective Matrix Factorization \cite{cortes2018cold}, Multi-Task Deep Gaussian Process Regression \cite{NIPS2007_66368270} and Meta Agnostic Meta Learning \cite{finn2017model} for 11 tasks." />
        </attvalues>
      </edge>
      <edge source="2205.06130" target="2012.15613" id="6972">
        <attvalues>
          <attvalue for="5" value=" Third, apart from the features used for zero-shot performance prediction in the previous work \cite{lauscher-etal-2020-zero, srinivasan2021predicting, dolicki2021analysing}, we also utilize metrics quantifying the quality of multilingual tokenizers as proposed in \cite{rust-etal-2021-good} as features in our predictive models, which turn out to have strong predictive power for certain tasks." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="2102.07370" id="6973">
        <attvalues>
          <attvalue for="5" value=" SLU has a very broad scope, including intent classification~\cite{sharma2021intent}, slot filling~\cite{slurp,wang2021fine,arora2022espnet,seo2022integration}, speech emotion recognition~\cite{chen2020large,shon2021leveraging}, question answering~\cite{lee2018odsqa, you2021knowledge}, etc There are mainly two types of SLU models: (1) cascading (ASR+NLU) models that first perform automatic speech recognition (ASR) and then apply a natural language understanding (NLU) model to the transcribed text; (2) end-to-end (E2E) models that directly predict the semantic output without predicting transcriptions." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="2011.13205" id="6974">
        <attvalues>
          <attvalue for="5" value=" Our main contributions are summarized as follows: \begin{itemize} \item Effectiveness: We present a Conformer-Transformer model with ASR-pretrained encoder that achieves new state-of-the-art performance on the SLURP dataset~\cite{slurp}, outperforming the other end-to-end (E2E) baselines by a large margin." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="2111.14706" id="6976">
        <attvalues>
          <attvalue for="5" value=" Some recent works~\cite{arora2022espnet,seo2022integration} also propose to train the SLU model with additional ASR task in a multi-task loss, which is more tricky to train since it's hard to choose a proper weight to balance different loss terms." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="2104.07253" id="6977">
        <attvalues>
          <attvalue for="5" value=" Some recent works~\cite{arora2022espnet,seo2022integration} also propose to train the SLU model with additional ASR task in a multi-task loss, which is more tricky to train since it's hard to choose a proper weight to balance different loss terms." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="2106.06598" id="6978">
        <attvalues>
          <attvalue for="5" value=" SLU has a very broad scope, including intent classification~\cite{sharma2021intent}, slot filling~\cite{slurp,wang2021fine,arora2022espnet,seo2022integration}, speech emotion recognition~\cite{chen2020large,shon2021leveraging}, question answering~\cite{lee2018odsqa, you2021knowledge}, etc There are mainly two types of SLU models: (1) cascading (ASR+NLU) models that first perform automatic speech recognition (ASR) and then apply a natural language understanding (NLU) model to the transcribed text; (2) end-to-end (E2E) models that directly predict the semantic output without predicting transcriptions." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="1808.02280" id="6979">
        <attvalues>
          <attvalue for="5" value=" SLU has a very broad scope, including intent classification~\cite{sharma2021intent}, slot filling~\cite{slurp,wang2021fine,arora2022espnet,seo2022integration}, speech emotion recognition~\cite{chen2020large,shon2021leveraging}, question answering~\cite{lee2018odsqa, you2021knowledge}, etc There are mainly two types of SLU models: (1) cascading (ASR+NLU) models that first perform automatic speech recognition (ASR) and then apply a natural language understanding (NLU) model to the transcribed text; (2) end-to-end (E2E) models that directly predict the semantic output without predicting transcriptions." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="2006.11477" id="6981">
        <attvalues>
          <attvalue for="5" value=" Current works~\cite{wang2021fine,seo2022integration} use encoders pretrained by self-supervised learning (SSL)~\cite{baevski2020wav2vec,hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="2106.07447" id="6982">
        <attvalues>
          <attvalue for="5" value=" Current works~\cite{wang2021fine,seo2022integration} use encoders pretrained by self-supervised learning (SSL)~\cite{baevski2020wav2vec,hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2307.07057" target="1902.00751" id="6983">
        <attvalues>
          <attvalue for="5" value=" \item Efficiency: We conduct extensive experiments on exploring parameter efficiency of E2E models, including freezing the encoder and using Adapters~\cite{houlsby2019parameter} in the encoder and finetuning on SLURP speech recognition data." />
        </attvalues>
      </edge>
      <edge source="2408.11847" target="1706.03762" id="6984">
        <attvalues>
          <attvalue for="5" value=" The field of Natural Language Processing is going through a massive transition since the introduction of Transformer-based Large Language Models (LLMs) \cite{vaswani2017attention, devlin-etal-2019-bert, radford2019language} which have demonstrated exceptional generalisation capability on a wide range of language-related tasks." />
        </attvalues>
      </edge>
      <edge source="2408.11847" target="2309.16573" id="6985">
        <attvalues>
          <attvalue for="5" value=" \cite{la2023language} noted several reproducibility issues with this Language-Models-as-a-Service (LMaaS) paradigm \cite{sun2022black}, where language models are centrally hosted and typically provided on a subscription or pay-per-use basis (eg, the OpenAI API and Google's Gemini API)." />
        </attvalues>
      </edge>
      <edge source="2408.11847" target="2201.03514" id="6986">
        <attvalues>
          <attvalue for="5" value=" \cite{la2023language} noted several reproducibility issues with this Language-Models-as-a-Service (LMaaS) paradigm \cite{sun2022black}, where language models are centrally hosted and typically provided on a subscription or pay-per-use basis (eg, the OpenAI API and Google's Gemini API)." />
        </attvalues>
      </edge>
      <edge source="2408.11847" target="2405.14782" id="6987">
        <attvalues>
          <attvalue for="5" value=" Inspired by \cite{biderman2024lessons}, the library promotes experiment reproducibility by facilitating the definition of all inputs/prompts within a single JSON Lines (JSONL) or CSV file." />
        </attvalues>
      </edge>
      <edge source="2408.11847" target="2403.08295" id="6990">
        <attvalues>
          <attvalue for="5" value=" In addition to LMaaS, there are now also several open LLMs, defined here as those with broadly available model weights as in \cite{kapoor2024societal}, eg Llama \cite{meta2024introducing}, Gemma \cite{team2024gemma}, Aya \cite{aryabumi2024aya}." />
        </attvalues>
      </edge>
      <edge source="2408.11847" target="2405.15032" id="6991">
        <attvalues>
          <attvalue for="5" value=" In addition to LMaaS, there are now also several open LLMs, defined here as those with broadly available model weights as in \cite{kapoor2024societal}, eg Llama \cite{meta2024introducing}, Gemma \cite{team2024gemma}, Aya \cite{aryabumi2024aya}." />
        </attvalues>
      </edge>
      <edge source="2408.11847" target="2306.05685" id="6992">
        <attvalues>
          <attvalue for="5" value=" prompto also provides built-in functionalities for automatic evaluation of the obtained responses, such as allowing the user to apply scoring functions to model outputs, and model graded evaluation or LLM-as-a-judge \cite{zheng_judge_2023}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2309.15701" id="6993">
        <attvalues>
          <attvalue for="5" value=" Initial studies typically input pure textual transcriptions into the LLM, often combining the ASR N-best results with instructions to prompt the LLM for error correction \cite{HyPoradise,Yang_2023,ma2023generativelargelanguagemodels}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2310.06434" id="6995">
        <attvalues>
          <attvalue for="5" value=" Further research has combined N-best results with speech encoders and even added denoising information \cite{whispering_llama,fathullah2023promptinglargelanguagemodels,chen2024itslatefusingacoustic,robust}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2309.15649" id="6996">
        <attvalues>
          <attvalue for="5" value=" Initial studies typically input pure textual transcriptions into the LLM, often combining the ASR N-best results with instructions to prompt the LLM for error correction \cite{HyPoradise,Yang_2023,ma2023generativelargelanguagemodels}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2307.04172" id="6997">
        <attvalues>
          <attvalue for="5" value=" Initial studies typically input pure textual transcriptions into the LLM, often combining the ASR N-best results with instructions to prompt the LLM for error correction \cite{HyPoradise,Yang_2023,ma2023generativelargelanguagemodels}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2401.11382" id="6999">
        <attvalues>
          <attvalue for="5" value=" Concurrently, other studies have attempted to integrate pre-trained ASR models (most commonly using the speech encoder part) into LLMs with a modality adapter, such as Q-former, attention, or a projection to align the speech feature space with the textual space of the LLM \cite{speech_llama,yu2023connectingspeechencoderlarge,li2024usinglargelanguagemodel}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2402.05457" id="7001">
        <attvalues>
          <attvalue for="5" value=" Further research has combined N-best results with speech encoders and even added denoising information \cite{whispering_llama,fathullah2023promptinglargelanguagemodels,chen2024itslatefusingacoustic,robust}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2401.10446" id="7002">
        <attvalues>
          <attvalue for="5" value=" Further research has combined N-best results with speech encoders and even added denoising information \cite{whispering_llama,fathullah2023promptinglargelanguagemodels,chen2024itslatefusingacoustic,robust}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2306.16007" id="7003">
        <attvalues>
          <attvalue for="5" value=" Usually, for traditional ASR models, domain adaptation or speaker adaptation can be used to address the issue of insufficient training \cite{Huang_Ye_Li_Gong_2021,10389732}." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2005.11401" id="7004">
        <attvalues>
          <attvalue for="5" value=" However, for LLM-based ASR, aside from the costly fine-tuning, this can be achieved through Retrieval-Augmented Generation (RAG) \cite{rag1,rag2}, allowing the LLM to learn external knowledge during inference." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2002.08909" id="7005">
        <attvalues>
          <attvalue for="5" value=" However, for LLM-based ASR, aside from the costly fine-tuning, this can be achieved through Retrieval-Augmented Generation (RAG) \cite{rag1,rag2}, allowing the LLM to learn external knowledge during inference." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2310.12477" id="7006">
        <attvalues>
          <attvalue for="5" value=" \cite{speech_icl} explores and proposes a speech LLM capable of performing unseen classification tasks for the first time." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2311.02248" id="7007">
        <attvalues>
          <attvalue for="5" value=" COSMIC \cite{COSMIC} pioneered this capability in more complex ASR tasks, showing significant ASR accuracy gains in context-biased tasks." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2312.13560" id="7009">
        <attvalues>
          <attvalue for="5" value=" Recently, in the speech retrieval augmentation task for small models, \cite{knn_ctc} and \cite{speaker_smoth} separately used Connectionist Temporal Classification (CTC) and Attention Encoder-Decoder (AED) pre-trained ASR models as speech tokenizers to force-align the speech features and text tokens." />
        </attvalues>
      </edge>
      <edge source="2409.08597" target="2406.04791" id="7010">
        <attvalues>
          <attvalue for="5" value=" Recently, in the speech retrieval augmentation task for small models, \cite{knn_ctc} and \cite{speaker_smoth} separately used Connectionist Temporal Classification (CTC) and Attention Encoder-Decoder (AED) pre-trained ASR models as speech tokenizers to force-align the speech features and text tokens." />
        </attvalues>
      </edge>
      <edge source="2407.06057" target="2307.09288" id="7011">
        <attvalues>
          <attvalue for="5" value=" The \bon algorithm has also been effectively applied in controlled decoding \cite{fudge, mudgal2024controlled} and to generate a dataset for supervised fine-tuning \cite{llama2}." />
        </attvalues>
      </edge>
      <edge source="2407.06057" target="1706.03741" id="7012">
        <attvalues>
          <attvalue for="5" value=" A common alignment-via-fine-tuning method is reinforcement learning from human feedback \cite[RLHF;][]{rlhf, summarizehf, instructgpt}." />
        </attvalues>
      </edge>
      <edge source="2407.06057" target="1707.06347" id="7014">
        <attvalues>
          <attvalue for="5" value=" This objective is often maximized with an RL algorithm, eg, proximal policy optimization \cite[PPO;][]{ppo}." />
        </attvalues>
      </edge>
      <edge source="2407.06057" target="2104.05218" id="7015">
        <attvalues>
          <attvalue for="5" value=" The \bon algorithm has also been effectively applied in controlled decoding \cite{fudge, mudgal2024controlled} and to generate a dataset for supervised fine-tuning \cite{llama2}." />
        </attvalues>
      </edge>
      <edge source="2407.06057" target="2404.01730" id="7016">
        <attvalues>
          <attvalue for="5" value=" Theoretically, \cite{yang2024asymptotics} prove that under some simplifying assumptions, the \bon distribution is asymptotically equivalent to the optimal distribution under the KL-constrained RL objective." />
        </attvalues>
      </edge>
      <edge source="2407.06057" target="2210.10760" id="7017">
        <attvalues>
          <attvalue for="5" value=" Empirically, it has been repeatedly shown \cite{pmlr-v202-gao23h, dpo, mudgal2024controlled} that \bon often appears on the frontier of reward and KL curves, surpassing the performance of models fine-tuned with RLHF." />
        </attvalues>
      </edge>
      <edge source="2407.10645" target="2211.01910" id="7019">
        <attvalues>
          <attvalue for="5" value=" Third, to explain to social scientists how to apply state-of-the-art prompt optimization methods used in the wider LLM community \cite{optimizer1, optimizer2} to their own classification tasks." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2302.05019" id="7020">
        <attvalues>
          <attvalue for="5" value=" Information Extraction (IE) is a crucial domain in natural language processing (NLP) that converts plain text into structured knowledge (eg, entities, relations, and events), and serves as a foundational requirement for a wide range of downstream tasks, such as knowledge graph construction \cite{zhong2023comprehensive}, knowledge reasoning \cite{fu2019collaborative} and question answering \cite{srihari1999information}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="1909.00230" id="7021">
        <attvalues>
          <attvalue for="5" value=" Information Extraction (IE) is a crucial domain in natural language processing (NLP) that converts plain text into structured knowledge (eg, entities, relations, and events), and serves as a foundational requirement for a wide range of downstream tasks, such as knowledge graph construction \cite{zhong2023comprehensive}, knowledge reasoning \cite{fu2019collaborative} and question answering \cite{srihari1999information}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2203.12277" id="7022">
        <attvalues>
          <attvalue for="5" value=" This is conducted by capturing inter-task dependencies with instructive prompts, and achieves consistent performance \cite{uie,gollie,paolini2021structured,instructuie,giellm,fei2022lasuie,codeie}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2304.08085" id="7023">
        <attvalues>
          <attvalue for="5" value=" This is conducted by capturing inter-task dependencies with instructive prompts, and achieves consistent performance \cite{uie,gollie,paolini2021structured,instructuie,giellm,fei2022lasuie,codeie}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2311.02962" id="7024">
        <attvalues>
          <attvalue for="5" value=" These challenges include the misalignment between natural language output and structured form \cite{code4uie}, hallucination problem in LLMs \cite{liu2024survey}, contextual dependence, high computational resource requirements \cite{sahoo2024systematic}, difficulties in updating internal knowledge \cite{xu2024editing}, etc In this survey, we provide a comprehensive exploration of LLMs for generative IE, as illustrated in Fig." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2303.08774" id="7026">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs), such as GPT-4 \cite{gpt4}, has significantly advanced the field of NLP, due to their extraordinary capabilities in text understanding and generation \cite{qi2024unimel,peng2024large,zhang2024notellm2}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2407.16160" id="7027">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs), such as GPT-4 \cite{gpt4}, has significantly advanced the field of NLP, due to their extraordinary capabilities in text understanding and generation \cite{qi2024unimel,peng2024large,zhang2024notellm2}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2311.03758" id="7028">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs), such as GPT-4 \cite{gpt4}, has significantly advanced the field of NLP, due to their extraordinary capabilities in text understanding and generation \cite{qi2024unimel,peng2024large,zhang2024notellm2}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2405.16789" id="7029">
        <attvalues>
          <attvalue for="5" value=" The emergence of large language models (LLMs), such as GPT-4 \cite{gpt4}, has significantly advanced the field of NLP, due to their extraordinary capabilities in text understanding and generation \cite{qi2024unimel,peng2024large,zhang2024notellm2}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2403.01744" id="7030">
        <attvalues>
          <attvalue for="5" value=" This enhances LLMs with the capability to perform zero-shot and few-shot learning, enabling them to model various tasks consistently and serving as tools for data augmentation \cite{zhang2024notellm, wang2024context, zhu2024fastmem}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2308.11432" id="7031">
        <attvalues>
          <attvalue for="5" value=" Furthermore, LLMs can serve as intelligent agents for complex task planning and execution, utilizing memory retrieval and various tools to enhance efficiency and successfully accomplish tasks \cite{wang2024survey, guan2024enhancing, huang2024qdmr,fu2024video,li2023agent4ranking}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2405.21075" id="7032">
        <attvalues>
          <attvalue for="5" value=" Furthermore, LLMs can serve as intelligent agents for complex task planning and execution, utilizing memory retrieval and various tools to enhance efficiency and successfully accomplish tasks \cite{wang2024survey, guan2024enhancing, huang2024qdmr,fu2024video,li2023agent4ranking}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2312.15450" id="7033">
        <attvalues>
          <attvalue for="5" value=" Furthermore, LLMs can serve as intelligent agents for complex task planning and execution, utilizing memory retrieval and various tools to enhance efficiency and successfully accomplish tasks \cite{wang2024survey, guan2024enhancing, huang2024qdmr,fu2024video,li2023agent4ranking}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2305.13981" id="7034">
        <attvalues>
          <attvalue for="5" value=" Therefore, there has been a recent surge of interest in generative IE methods \cite{qi2023preserving} that adopt LLMs to generate structural information rather than extracting structural information from plain text." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2301.03282" id="7035">
        <attvalues>
          <attvalue for="5" value=" These methods have been proven to be more practical in real-world scenarios compared to discriminated methods \cite{chen2023heproto,usm}, as they can handle schemas containing millions of entities without significant performance degradation \cite{genie}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2112.08340" id="7036">
        <attvalues>
          <attvalue for="5" value=" These methods have been proven to be more practical in real-world scenarios compared to discriminated methods \cite{chen2023heproto,usm}, as they can handle schemas containing millions of entities without significant performance degradation \cite{genie}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2101.05779" id="7038">
        <attvalues>
          <attvalue for="5" value=" On the other hand, recent works have shown the outstanding generalization of LLMs to not only learn from IE training data through fine-tuning \cite{paolini2021structured,yan2021unified,tempgen,rebel,paolini2021structured}, but also extract information in few-shot and even zero-shot scenarios relying solely on in-context examples or instructions \cite{chatie,code4struct,gpt-ner,promptner,xu2023unleash}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2311.06838" id="7039">
        <attvalues>
          <attvalue for="5" value=" This is conducted by capturing inter-task dependencies with instructive prompts, and achieves consistent performance \cite{uie,gollie,paolini2021structured,instructuie,giellm,fei2022lasuie,codeie}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2304.06248" id="7040">
        <attvalues>
          <attvalue for="5" value=" This is conducted by capturing inter-task dependencies with instructive prompts, and achieves consistent performance \cite{uie,gollie,paolini2021structured,instructuie,giellm,fei2022lasuie,codeie}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2305.05711" id="7041">
        <attvalues>
          <attvalue for="5" value=" This is conducted by capturing inter-task dependencies with instructive prompts, and achieves consistent performance \cite{uie,gollie,paolini2021structured,instructuie,giellm,fei2022lasuie,codeie}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2106.01223" id="7042">
        <attvalues>
          <attvalue for="5" value=" On the other hand, recent works have shown the outstanding generalization of LLMs to not only learn from IE training data through fine-tuning \cite{paolini2021structured,yan2021unified,tempgen,rebel,paolini2021structured}, but also extract information in few-shot and even zero-shot scenarios relying solely on in-context examples or instructions \cite{chatie,code4struct,gpt-ner,promptner,xu2023unleash}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2109.04901" id="7043">
        <attvalues>
          <attvalue for="5" value=" On the other hand, recent works have shown the outstanding generalization of LLMs to not only learn from IE training data through fine-tuning \cite{paolini2021structured,yan2021unified,tempgen,rebel,paolini2021structured}, but also extract information in few-shot and even zero-shot scenarios relying solely on in-context examples or instructions \cite{chatie,code4struct,gpt-ner,promptner,xu2023unleash}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2210.12810" id="7044">
        <attvalues>
          <attvalue for="5" value=" On the other hand, recent works have shown the outstanding generalization of LLMs to not only learn from IE training data through fine-tuning \cite{paolini2021structured,yan2021unified,tempgen,rebel,paolini2021structured}, but also extract information in few-shot and even zero-shot scenarios relying solely on in-context examples or instructions \cite{chatie,code4struct,gpt-ner,promptner,xu2023unleash}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2304.10428" id="7045">
        <attvalues>
          <attvalue for="5" value=" On the other hand, recent works have shown the outstanding generalization of LLMs to not only learn from IE training data through fine-tuning \cite{paolini2021structured,yan2021unified,tempgen,rebel,paolini2021structured}, but also extract information in few-shot and even zero-shot scenarios relying solely on in-context examples or instructions \cite{chatie,code4struct,gpt-ner,promptner,xu2023unleash}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2305.15444" id="7046">
        <attvalues>
          <attvalue for="5" value=" On the other hand, recent works have shown the outstanding generalization of LLMs to not only learn from IE training data through fine-tuning \cite{paolini2021structured,yan2021unified,tempgen,rebel,paolini2021structured}, but also extract information in few-shot and even zero-shot scenarios relying solely on in-context examples or instructions \cite{chatie,code4struct,gpt-ner,promptner,xu2023unleash}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2305.01555" id="7047">
        <attvalues>
          <attvalue for="5" value=" On the other hand, recent works have shown the outstanding generalization of LLMs to not only learn from IE training data through fine-tuning \cite{paolini2021structured,yan2021unified,tempgen,rebel,paolini2021structured}, but also extract information in few-shot and even zero-shot scenarios relying solely on in-context examples or instructions \cite{chatie,code4struct,gpt-ner,promptner,xu2023unleash}." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2402.00253" id="7049">
        <attvalues>
          <attvalue for="5" value=" These challenges include the misalignment between natural language output and structured form \cite{code4uie}, hallucination problem in LLMs \cite{liu2024survey}, contextual dependence, high computational resource requirements \cite{sahoo2024systematic}, difficulties in updating internal knowledge \cite{xu2024editing}, etc In this survey, we provide a comprehensive exploration of LLMs for generative IE, as illustrated in Fig." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2402.07927" id="7050">
        <attvalues>
          <attvalue for="5" value=" These challenges include the misalignment between natural language output and structured form \cite{code4uie}, hallucination problem in LLMs \cite{liu2024survey}, contextual dependence, high computational resource requirements \cite{sahoo2024systematic}, difficulties in updating internal knowledge \cite{xu2024editing}, etc In this survey, we provide a comprehensive exploration of LLMs for generative IE, as illustrated in Fig." />
        </attvalues>
      </edge>
      <edge source="2312.17617" target="2402.18099" id="7051">
        <attvalues>
          <attvalue for="5" value=" These challenges include the misalignment between natural language output and structured form \cite{code4uie}, hallucination problem in LLMs \cite{liu2024survey}, contextual dependence, high computational resource requirements \cite{sahoo2024systematic}, difficulties in updating internal knowledge \cite{xu2024editing}, etc In this survey, we provide a comprehensive exploration of LLMs for generative IE, as illustrated in Fig." />
        </attvalues>
      </edge>
      <edge source="2408.06484" target="2301.13848" id="7052">
        <attvalues>
          <attvalue for="5" value=" For some datasets, LLMs have been found to not only surpass previous summarization models, but to meet the performance of humans \cite{zhang2024benchmarking}." />
        </attvalues>
      </edge>
      <edge source="2408.06484" target="2012.07311" id="7053">
        <attvalues>
          <attvalue for="5" value=" Another major area is the summarization of call center interactions \cite{zou2021topic}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2311.13165" id="7054">
        <attvalues>
          <attvalue for="5" value=" M{LLMs} are sophisticated artificial intelligence (AI) systems designed to process and integrate various types of data, including text, images, videos, audio, and physiological sequential data~\cite{wu2023multimodal,yin2023survey,zhang2024mm}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2306.13549" id="7055">
        <attvalues>
          <attvalue for="5" value=" M{LLMs} are sophisticated artificial intelligence (AI) systems designed to process and integrate various types of data, including text, images, videos, audio, and physiological sequential data~\cite{wu2023multimodal,yin2023survey,zhang2024mm}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2401.13601" id="7056">
        <attvalues>
          <attvalue for="5" value=" M{LLMs} are sophisticated artificial intelligence (AI) systems designed to process and integrate various types of data, including text, images, videos, audio, and physiological sequential data~\cite{wu2023multimodal,yin2023survey,zhang2024mm}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2305.17216" id="7057">
        <attvalues>
          <attvalue for="5" value=" As we navigate the era of multimodal data fusion, marked by rapid advancements in information technology and an explosive increase in data volume, the capabilities of single-modality systems no longer suffice for complex real-world tasks~\cite{koh2024generating,ma2024eye,zhang2024potential}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2407.05758" id="7058">
        <attvalues>
          <attvalue for="5" value=" As we navigate the era of multimodal data fusion, marked by rapid advancements in information technology and an explosive increase in data volume, the capabilities of single-modality systems no longer suffice for complex real-world tasks~\cite{koh2024generating,ma2024eye,zhang2024potential}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="1911.08870" id="7059">
        <attvalues>
          <attvalue for="5" value=" In Natural Language Processing (NLP) tasks such as text generation and machine translation, MLLMs leverage images, video, and audio to provide contextual support, enhancing the accuracy and expressiveness of the generated text~\cite{malik2021automatic,bahar2019comparative,lyu2023macaw}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2306.09093" id="7060">
        <attvalues>
          <attvalue for="5" value=" In Natural Language Processing (NLP) tasks such as text generation and machine translation, MLLMs leverage images, video, and audio to provide contextual support, enhancing the accuracy and expressiveness of the generated text~\cite{malik2021automatic,bahar2019comparative,lyu2023macaw}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2303.08774" id="7061">
        <attvalues>
          <attvalue for="5" value=" For instance, MLLMs such as GPT-4V~\cite{achiam2023gpt}, and Gemini~\cite{team2023gemini} combine image content with natural language descriptions to produce more vivid and precise annotation results." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2306.05685" id="7062">
        <attvalues>
          <attvalue for="5" value=" In particular, MLLMs transform NLP by incorporating visual and auditory data, thus enriching text generation and machine translation~\cite{achiam2023gpt,zheng2024judging,le2023bloom}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2211.05100" id="7063">
        <attvalues>
          <attvalue for="5" value=" In particular, MLLMs transform NLP by incorporating visual and auditory data, thus enriching text generation and machine translation~\cite{achiam2023gpt,zheng2024judging,le2023bloom}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2307.00855" id="7065">
        <attvalues>
          <attvalue for="5" value=" In sentiment analysis and dialogue systems, MLLMs are capable of the integration of multimodal information, which further deepens the understanding of system and response capabilities, presenting a leap forward in human-computer interaction~\cite{wang2024largelanguagemodelsrobotics,wang2023review}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2312.11805" id="7066">
        <attvalues>
          <attvalue for="5" value=" In addition, MLLMs show strong potential in tasks such as cross-modal audio-text translation, audio soundtrack generation, and multimodal sentiment analysis~\cite{tang2023salmonn,team2023gemini}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2309.05519" id="7067">
        <attvalues>
          <attvalue for="5" value=" Models like NExT-GPT~\cite{wu2023next} and Sora~\cite{liu2024sora} are pioneering multimodal video generation, producing richer and more realistic video content by learning from multimodal data." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2402.17177" id="7068">
        <attvalues>
          <attvalue for="5" value=" Models like NExT-GPT~\cite{wu2023next} and Sora~\cite{liu2024sora} are pioneering multimodal video generation, producing richer and more realistic video content by learning from multimodal data." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2305.06355" id="7069">
        <attvalues>
          <attvalue for="5" value=" Furthermore, advancements in intelligent video understanding technologies, such as VideoChat~\cite{li2023videochat} and Video-LLaVA~\cite{lin2023video}, have significantly enhanced the ability to analyze and process video content." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2311.10122" id="7070">
        <attvalues>
          <attvalue for="5" value=" Furthermore, advancements in intelligent video understanding technologies, such as VideoChat~\cite{li2023videochat} and Video-LLaVA~\cite{lin2023video}, have significantly enhanced the ability to analyze and process video content." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2401.04447" id="7071">
        <attvalues>
          <attvalue for="5" value=" Traditional audio processing usually relies on unimodal signal processing methods, such as speech recognition~\cite{gaikwad2010review} or audio classification~\cite{mulimani2024class}, which have limitations in processing complex multimodal data." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2304.12995" id="7073">
        <attvalues>
          <attvalue for="5" value=" For example, in speech generation tasks, MLLMs can utilize textual and visual information to generate more natural and contextually relevant speech output~\cite{shen2024hugginggpt,huang2024audiogpt}." />
        </attvalues>
      </edge>
      <edge source="2408.01319" target="2310.13289" id="7074">
        <attvalues>
          <attvalue for="5" value=" In addition, MLLMs show strong potential in tasks such as cross-modal audio-text translation, audio soundtrack generation, and multimodal sentiment analysis~\cite{tang2023salmonn,team2023gemini}." />
        </attvalues>
      </edge>
      <edge source="2403.11509" target="2303.08774" id="7081">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in LLMs, like LLaMA\cite{llama} and OpenAI's GPT series\cite{gpt-4}, have led to widespread use in various applications, especially in industrial scenarios." />
        </attvalues>
      </edge>
      <edge source="2403.11509" target="2304.05335" id="7082">
        <attvalues>
          <attvalue for="5" value=" However, as LLMs evolve, more sophisticated and fluent outputs are generated, bringing forth novel challenges like hallucinations\cite{hallu}, biases and toxicity\cite{toxicity}." />
        </attvalues>
      </edge>
      <edge source="2403.11509" target="1904.03371" id="7083">
        <attvalues>
          <attvalue for="5" value=" For instance, Alipay employs generative systems for the automatic generation of social media posts, but this raises issues like potential toxicity\cite{toxicity} or incoherence\cite{coherence} in the content." />
        </attvalues>
      </edge>
      <edge source="2403.11509" target="2106.11520" id="7084">
        <attvalues>
          <attvalue for="5" value=" Existing methods\cite{bartscore, rouge, bertscore, gptscore, instructscore, tigerscore} for text generation primarily focus on basic aspects but inadequately address recent emerging challenges." />
        </attvalues>
      </edge>
      <edge source="2403.11509" target="2202.03629" id="7085">
        <attvalues>
          <attvalue for="5" value=" However, as LLMs evolve, more sophisticated and fluent outputs are generated, bringing forth novel challenges like hallucinations\cite{hallu}, biases and toxicity\cite{toxicity}." />
        </attvalues>
      </edge>
      <edge source="2403.11509" target="2310.00752" id="7087">
        <attvalues>
          <attvalue for="5" value=" LLM-based methods, such as InstructScore\cite{instructscore} and TIGERScore\cite{tigerscore}, provide detailed diagnostic reports but suffer from significant latency issues due to the inherent inefficiency of LLM inference." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1409.0473" id="7088">
        <attvalues>
          <attvalue for="5" value=" As deep learning based neural machine translation (NMT) was proposed and adopted to several researches, it has been gradually figured out that more superior performance can be derived through NMT approach \cite{bahdanau2014neural,vaswani2017attention,lample2019cross,song2019mass}." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1706.03762" id="7089">
        <attvalues>
          <attvalue for="5" value=" In addition, we conduct baseline translation experiments by training transformer-base model structure \cite{vaswani2017attention} through all the parallel corpora given by AIhub." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1901.07291" id="7090">
        <attvalues>
          <attvalue for="5" value=" As deep learning based neural machine translation (NMT) was proposed and adopted to several researches, it has been gradually figured out that more superior performance can be derived through NMT approach \cite{bahdanau2014neural,vaswani2017attention,lample2019cross,song2019mass}." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1905.02450" id="7091">
        <attvalues>
          <attvalue for="5" value=" As deep learning based neural machine translation (NMT) was proposed and adopted to several researches, it has been gradually figured out that more superior performance can be derived through NMT approach \cite{bahdanau2014neural,vaswani2017attention,lample2019cross,song2019mass}." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1912.01703" id="7092">
        <attvalues>
          <attvalue for="5" value=" Recently, release of open source frameworks, such as Pytorch\cite{NEURIPS2019_9015}, and lowered accessibility to the big data further facilitated vigorous and diverse research." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1808.09381" id="7093">
        <attvalues>
          <attvalue for="5" value=" Although data-augmentation techniques, such as back translation \cite{edunov2018understanding} and copied translation \cite{currey2017copied} have been introduced, as the human supervision is generally minimized or excluded in the data generation process, the quality of such pseudo-generated parallel corpus cannot be guaranteed \cite{burlot2019using, epaliyana2021improving}." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1903.11437" id="7094">
        <attvalues>
          <attvalue for="5" value=" Although data-augmentation techniques, such as back translation \cite{edunov2018understanding} and copied translation \cite{currey2017copied} have been introduced, as the human supervision is generally minimized or excluded in the data generation process, the quality of such pseudo-generated parallel corpus cannot be guaranteed \cite{burlot2019using, epaliyana2021improving}." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1805.12282" id="7095">
        <attvalues>
          <attvalue for="5" value=" While the amount of training data caused significant impact on the statistical-based MT approaches, the quality of data is treated as more important than the amount of data in general deep learning-based MT approaches \cite{khayrallah2018impact, koehn-EtAl:2020:WMT}." />
        </attvalues>
      </edge>
      <edge source="2110.15023" target="1811.04655" id="7096">
        <attvalues>
          <attvalue for="5" value=" LIWC generally used to recognize linguistic markers for mental health study in Psychopathology such as detecting Narcissism\cite{holtzman2019linguistic}, schizophrenia\cite{bae2021schizophrenia}, bipolar disorder\cite{sekulic2018not}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="1712.05884" id="7097">
        <attvalues>
          <attvalue for="5" value=" Recent TTS methods achieved high naturalness in synthetic speech that is comparable to human speech \cite{DBLP:conf/icassp/ShenPWSJYCZWRSA18_short, DBLP:journals/corr/abs-1809-08895, DBLP:conf/icml/KimKS21}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2006.11239" id="7098">
        <attvalues>
          <attvalue for="5" value=" Recently, the diffusion probabilistic model (DPM) \cite{DBLP:conf/nips/HoJA20} has been developed as another advanced probabilistic model, and it has been intensively studied in speech domains such as neural vocoders \cite{DBLP:conf/iclr/ChenZZWNC21}, and TTS \cite{jeong21_interspeech, chen21p_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2009.00713" id="7099">
        <attvalues>
          <attvalue for="5" value=" Recently, the diffusion probabilistic model (DPM) \cite{DBLP:conf/nips/HoJA20} has been developed as another advanced probabilistic model, and it has been intensively studied in speech domains such as neural vocoders \cite{DBLP:conf/iclr/ChenZZWNC21}, and TTS \cite{jeong21_interspeech, chen21p_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2106.06406" id="7101">
        <attvalues>
          <attvalue for="5" value=" The iterative inference process of diffusion enables conditioning by various methods such as adaptive prior \cite{DBLP:conf/iclr/LeeKS0LMQ0YL22}, classifier guidance \cite{DBLP:conf/icml/KimKY22}, and iterative latent variable refinement \cite{levkovitch22_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2111.11755" id="7102">
        <attvalues>
          <attvalue for="5" value=" The iterative inference process of diffusion enables conditioning by various methods such as adaptive prior \cite{DBLP:conf/iclr/LeeKS0LMQ0YL22}, classifier guidance \cite{DBLP:conf/icml/KimKY22}, and iterative latent variable refinement \cite{levkovitch22_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2206.02246" id="7103">
        <attvalues>
          <attvalue for="5" value=" The iterative inference process of diffusion enables conditioning by various methods such as adaptive prior \cite{DBLP:conf/iclr/LeeKS0LMQ0YL22}, classifier guidance \cite{DBLP:conf/icml/KimKY22}, and iterative latent variable refinement \cite{levkovitch22_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="1312.6114" id="7104">
        <attvalues>
          <attvalue for="5" value=" Inspired by a recent TTS method based on VAE \cite{DBLP:journals/corr/KingmaW13} incorporating a waveform model \cite{DBLP:conf/icml/KimKS21}, our method convert texts into waveforms via a latent space." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="1907.02479" id="7105">
        <attvalues>
          <attvalue for="5" value=" The use of the variational autoencoder (VAE) \cite{DBLP:journals/corr/KingmaW13} is a popular representation learning method for speech style modeling \cite{DBLP:conf/interspeech/KlimkovRRD19, DBLP:conf/iclr/HsuZWZWWCJCSNP19}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="1711.00937" id="7106">
        <attvalues>
          <attvalue for="5" value=" Designing a discrete latent space in VAE enables the capture of phonetic latent information in speech \cite{DBLP:conf/nips/OordVK17}, and a categorical latent space enables the use of a semisupervised approach to capture the desired latent information such as speech emotion \cite{DBLP:conf/iclr/HabibMSBSSKB20}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="1910.01709" id="7107">
        <attvalues>
          <attvalue for="5" value=" Designing a discrete latent space in VAE enables the capture of phonetic latent information in speech \cite{DBLP:conf/nips/OordVK17}, and a categorical latent space enables the use of a semisupervised approach to capture the desired latent information such as speech emotion \cite{DBLP:conf/iclr/HabibMSBSSKB20}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2103.15060" id="7108">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL) based on a pretrained representation is also applied to speech synthesis to utilize large-scale unpaired texts or speeches \cite{jia21_interspeech, DBLP:conf/interspeech/SiuzdakDRJ22}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2212.08321" id="7109">
        <attvalues>
          <attvalue for="5" value=" Supervised fine-tuning in SSL enables the adoption of pretrained features to specific domains, and it is used to capture accent features for TTS \cite{9829304}." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2102.09672" id="7110">
        <attvalues>
          <attvalue for="5" value=" However, diffusion models typically use a fixed variance, which results in a suboptimal likelihood \cite{DBLP:conf/icml/NicholD21}, and the application of diffusion to latent variable modeling, where the variance of latent variables is approximated by a model, is as yet not known." />
        </attvalues>
      </edge>
      <edge source="2212.08329" target="2106.06103" id="7111">
        <attvalues>
          <attvalue for="5" value=" Inspired by a recent TTS method based on VAE \cite{DBLP:journals/corr/KingmaW13} incorporating a waveform model \cite{DBLP:conf/icml/KimKS21}, our method convert texts into waveforms via a latent space." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1706.03872" id="7142">
        <attvalues>
          <attvalue for="5" value=" Several authors present anecdotal evidence for NMT systems occasionally falling into a hallucination mode where translations are grammatically correct but unrelated to the source sentence \cite{Arthur2016,Koehn2017,Nguyen2018}." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1412.6572" id="7143">
        <attvalues>
          <attvalue for="5" value=" We consider domain robustness a desirable property of NLP systems, along with other types of robustness, such as robustness against adversarial examples (\cite{Goodfellow2015}) or typos in the input \cite{Belinkov2018}." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1711.02173" id="7144">
        <attvalues>
          <attvalue for="5" value=" We consider domain robustness a desirable property of NLP systems, along with other types of robustness, such as robustness against adversarial examples (\cite{Goodfellow2015}) or typos in the input \cite{Belinkov2018}." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1511.06709" id="7145">
        <attvalues>
          <attvalue for="5" value=" While domain adaptation with small amounts of parallel or monolingual in-domain data has proven very effective for NMT \cite[eg][]{luong2015,sennrich-haddow-birch:2016:P16-11,R17-1049,li-EtAl:2019:WMT1}, the target domain(s) may be unknown when a system is built, and there are language pairs for which training data is only available for limited domains." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1612.06140" id="7146">
        <attvalues>
          <attvalue for="5" value=" While domain adaptation with small amounts of parallel or monolingual in-domain data has proven very effective for NMT \cite[eg][]{luong2015,sennrich-haddow-birch:2016:P16-11,R17-1049,li-EtAl:2019:WMT1}, the target domain(s) may be unknown when a system is built, and there are language pairs for which training data is only available for limited domains." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1906.11943" id="7147">
        <attvalues>
          <attvalue for="5" value=" While domain adaptation with small amounts of parallel or monolingual in-domain data has proven very effective for NMT \cite[eg][]{luong2015,sennrich-haddow-birch:2016:P16-11,R17-1049,li-EtAl:2019:WMT1}, the target domain(s) may be unknown when a system is built, and there are language pairs for which training data is only available for limited domains." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1606.02006" id="7148">
        <attvalues>
          <attvalue for="5" value=" Several authors present anecdotal evidence for NMT systems occasionally falling into a hallucination mode where translations are grammatically correct but unrelated to the source sentence \cite{Arthur2016,Koehn2017,Nguyen2018}." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1611.01874" id="7150">
        <attvalues>
          <attvalue for="5" value=" As a means to reduce hallucination, we experiment with several techniques and assess their effectiveness in improving domain robustness: reconstruction \cite{Tu2017,niu-etal-2019-bi}, subword regularization \cite{Kudo2018}, neural noisy channel models \cite{li2016mutual,yee2019simple}, and defensive distillation \cite{papernot2016distillation}, as well as combinations of these techniques." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1811.01116" id="7151">
        <attvalues>
          <attvalue for="5" value=" As a means to reduce hallucination, we experiment with several techniques and assess their effectiveness in improving domain robustness: reconstruction \cite{Tu2017,niu-etal-2019-bi}, subword regularization \cite{Kudo2018}, neural noisy channel models \cite{li2016mutual,yee2019simple}, and defensive distillation \cite{papernot2016distillation}, as well as combinations of these techniques." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1804.10959" id="7152">
        <attvalues>
          <attvalue for="5" value=" As a means to reduce hallucination, we experiment with several techniques and assess their effectiveness in improving domain robustness: reconstruction \cite{Tu2017,niu-etal-2019-bi}, subword regularization \cite{Kudo2018}, neural noisy channel models \cite{li2016mutual,yee2019simple}, and defensive distillation \cite{papernot2016distillation}, as well as combinations of these techniques." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1601.00372" id="7153">
        <attvalues>
          <attvalue for="5" value=" As a means to reduce hallucination, we experiment with several techniques and assess their effectiveness in improving domain robustness: reconstruction \cite{Tu2017,niu-etal-2019-bi}, subword regularization \cite{Kudo2018}, neural noisy channel models \cite{li2016mutual,yee2019simple}, and defensive distillation \cite{papernot2016distillation}, as well as combinations of these techniques." />
        </attvalues>
      </edge>
      <edge source="1911.03109" target="1908.05731" id="7154">
        <attvalues>
          <attvalue for="5" value=" As a means to reduce hallucination, we experiment with several techniques and assess their effectiveness in improving domain robustness: reconstruction \cite{Tu2017,niu-etal-2019-bi}, subword regularization \cite{Kudo2018}, neural noisy channel models \cite{li2016mutual,yee2019simple}, and defensive distillation \cite{papernot2016distillation}, as well as combinations of these techniques." />
        </attvalues>
      </edge>
      <edge source="2210.17238" target="2205.00656" id="7156">
        <attvalues>
          <attvalue for="5" value=" Second, random sampling causes sampling bias, such as containing false negatives for a given dialogue context~\cite{zhou-etal-2022-debiased}." />
        </attvalues>
      </edge>
      <edge source="2210.17238" target="2007.06661" id="7157">
        <attvalues>
          <attvalue for="5" value=" To mitigate this problem, recent studies have proposed various methods to synthesize and leverage adversarial negative training samples so that the selection model can learn features beyond content similarity~\cite{srivastava2020robustness, kaushik2021learning}." />
        </attvalues>
      </edge>
      <edge source="2210.17238" target="2009.11321" id="7159">
        <attvalues>
          <attvalue for="5" value=" The most reliable method is to collect human-written adversarial negatives~\cite{sai-etal-2020-improving}, but it is not scalable because it is expensive and time consuming." />
        </attvalues>
      </edge>
      <edge source="2210.17238" target="2108.13487" id="7160">
        <attvalues>
          <attvalue for="5" value=" To overcome these limitations, we note that large-scale language models such as GPT3 can be utilized as a low-cost data labeler~\cite{wang-etal-2021-want-reduce}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1409.3215" id="7161">
        <attvalues>
          <attvalue for="5" value=" Owing to the rapid development of neural sequence-to-sequence modeling~\cite{sutskever2014sequence, bahdanau2014neural}, deep neural network (DNN)-based end-to-end automatic speech recognition (ASR) systems have become almost as effective as the traditional hidden Markov model-based systems~\cite{chiu2018state, luscher2019rwth, karita2019a}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1712.01769" id="7162">
        <attvalues>
          <attvalue for="5" value=" Owing to the rapid development of neural sequence-to-sequence modeling~\cite{sutskever2014sequence, bahdanau2014neural}, deep neural network (DNN)-based end-to-end automatic speech recognition (ASR) systems have become almost as effective as the traditional hidden Markov model-based systems~\cite{chiu2018state, luscher2019rwth, karita2019a}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1506.07503" id="7163">
        <attvalues>
          <attvalue for="5" value=" Various models and approaches have been proposed for improving the performance of the autoregressive (AR) end-to-end ASR model with the encoder-decoder architecture based on recurrent neural networks (RNNs)~\cite{chorowski2015attention, chan2016listen, kim2017joint} and Transformers~\cite{vaswani2017attention, dong2018speech, karita2019improving}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1706.03762" id="7164">
        <attvalues>
          <attvalue for="5" value=" Various models and approaches have been proposed for improving the performance of the autoregressive (AR) end-to-end ASR model with the encoder-decoder architecture based on recurrent neural networks (RNNs)~\cite{chorowski2015attention, chan2016listen, kim2017joint} and Transformers~\cite{vaswani2017attention, dong2018speech, karita2019improving}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1711.02281" id="7165">
        <attvalues>
          <attvalue for="5" value=" Contrary to the autoregressive framework, non-autoregressive (NAR) sequence generation has attracted attention, including the revisitation of connectionist temporal classification (CTC)~\cite{graves2006connectionist, libovicky2018end} and the growing interest for non-autoregressive Transformer (NAT)~\cite{gu2017non}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1802.06901" id="7166">
        <attvalues>
          <attvalue for="5" value=" Different types of non-autoregressive models have been proposed based on the iterative refinement decoding~\cite{lee2018deterministic}, insert or edit-based sequence generation~\cite{stern2019insertion, gu2019levenshtein}, masked language model objective~\cite{ghazvininejad2019mask, ghazvininejad2020semi, saharia2020non}, and generative flow~\cite{ma2019flowseq}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1902.03249" id="7167">
        <attvalues>
          <attvalue for="5" value=" Different types of non-autoregressive models have been proposed based on the iterative refinement decoding~\cite{lee2018deterministic}, insert or edit-based sequence generation~\cite{stern2019insertion, gu2019levenshtein}, masked language model objective~\cite{ghazvininejad2019mask, ghazvininejad2020semi, saharia2020non}, and generative flow~\cite{ma2019flowseq}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1904.09324" id="7168">
        <attvalues>
          <attvalue for="5" value=" On the other hand, \cite{chen2019non} trains a Transformer encoder-decoder in a mask-predict manner~\cite{ghazvininejad2019mask}: target tokens are randomly masked and predicted conditioning on the unmasked tokens and the input speech." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1909.02480" id="7169">
        <attvalues>
          <attvalue for="5" value=" Different types of non-autoregressive models have been proposed based on the iterative refinement decoding~\cite{lee2018deterministic}, insert or edit-based sequence generation~\cite{stern2019insertion, gu2019levenshtein}, masked language model objective~\cite{ghazvininejad2019mask, ghazvininejad2020semi, saharia2020non}, and generative flow~\cite{ma2019flowseq}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="1707.07413" id="7170">
        <attvalues>
          <attvalue for="5" value=" While CTC makes use of dynamic programming to efficiently calculate the most probable alignment, the strong conditional independence assumption between output tokens results in poor performance compared to the autoregressive models~\cite{battenberg2017exploring}." />
        </attvalues>
      </edge>
      <edge source="2005.08700" target="2002.08926" id="7171">
        <attvalues>
          <attvalue for="5" value=" \cite{chan2020imputer} proposes Imputer, which performs the mask prediction in CTC's latent alignments to get rid of the output length prediction." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1610.04019" id="7172">
        <attvalues>
          <attvalue for="5" value=" That is why recently probabilistic generative models, such as VAEs, have been used to learn a latent representation that captures the variability of speech \cite{ChengHsu2016, Hsu2017, Akuzawa2018, Zhang2019}." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1611.02648" id="7173">
        <attvalues>
          <attvalue for="5" value=" Such a prior, even if it still represents natural prosody, loses some of the variability of real speech, and has been shown to lead to over-regularization and poor latent representations \cite{Dilokthanakul2016, Bowman2016}." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1912.08521" id="7174">
        <attvalues>
          <attvalue for="5" value=" More precisely, we suggest to adopt a learned conditional posterior as prior, following the approach of \cite{Aliakbarian2019}." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1804.02135" id="7175">
        <attvalues>
          <attvalue for="5" value=" The typical approach to add a condition to a VAE is the Conditional VAE (CVAE) \cite{Akuzawa2018, Skerry-Ryan2018}." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1705.07120" id="7176">
        <attvalues>
          <attvalue for="5" value=" VampPrior \cite{Tomczak2018, Hodari2020} is an example of such approaches where the proposed prior is a mixture of variational posteriors conditioned on learnable pseudo-data." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1810.07217" id="7177">
        <attvalues>
          <attvalue for="5" value=" Another related work is the use of Gaussian mixture as a latent prior \cite{Hsu2019} where there is a different speech factor represented in each mixture component." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1803.09017" id="7178">
        <attvalues>
          <attvalue for="5" value=" In controllable TTS literature, in \cite{Wang2018} the authors introduce the ``global style tokens'' (GST), style embeddings that condition the text encoder and are learned jointly with Tacotron \cite{Wang2017}." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1703.10135" id="7179">
        <attvalues>
          <attvalue for="5" value=" In controllable TTS literature, in \cite{Wang2018} the authors introduce the ``global style tokens'' (GST), style embeddings that condition the text encoder and are learned jointly with Tacotron \cite{Wang2017}." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="1912.00955" id="7180">
        <attvalues>
          <attvalue for="5" value=" Other works adopt a variational method and condition the VAE on contextual linguistic information, either directly \cite{Tyagi2020} or via training a prediction model that is then applied to sampling \cite{Karlapati2020, Hodari2021}." />
        </attvalues>
      </edge>
      <edge source="2106.10229" target="2011.02252" id="7181">
        <attvalues>
          <attvalue for="5" value=" Other works adopt a variational method and condition the VAE on contextual linguistic information, either directly \cite{Tyagi2020} or via training a prediction model that is then applied to sampling \cite{Karlapati2020, Hodari2021}." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="2002.08909" id="7183">
        <attvalues>
          <attvalue for="5" value=" With a huge number of model parameters and well designed training objectives, pretrained language models (PLMs) have brought a new era to NLP \cite{bertnb1,bertnb2,bertnb3,bertnb4}." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="2002.06823" id="7185">
        <attvalues>
          <attvalue for="5" value=" With a huge number of model parameters and well designed training objectives, pretrained language models (PLMs) have brought a new era to NLP \cite{bertnb1,bertnb2,bertnb3,bertnb4}." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="2003.08271" id="7186">
        <attvalues>
          <attvalue for="5" value=" With a huge number of model parameters and well designed training objectives, pretrained language models (PLMs) have brought a new era to NLP \cite{bertnb1,bertnb2,bertnb3,bertnb4}." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="1909.03546" id="7188">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning PLMs such as BERT \cite{bert} has become a basic and effective way in many downstream tasks \cite{finetune1, finetune2, finetune3}." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="2002.06305" id="7189">
        <attvalues>
          <attvalue for="5" value=" However, recent study has shown that aggressive fine-tuning can induce an unstable and suboptimal performance of the models especially with insufficient data \cite{bad1, bad2}, which attracts some researchers to figure out the culprits and explore effective methods to solve them \cite{bettertune1, bettertune2, bettertune3}." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="1903.05987" id="7190">
        <attvalues>
          <attvalue for="5" value=" However, recent study has shown that aggressive fine-tuning can induce an unstable and suboptimal performance of the models especially with insufficient data \cite{bad1, bad2}, which attracts some researchers to figure out the culprits and explore effective methods to solve them \cite{bettertune1, bettertune2, bettertune3}." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="1911.03437" id="7193">
        <attvalues>
          <attvalue for="5" value=" For example, there are some regularization methods like RecAdam \cite{Recadam} and Mixout \cite{mixout}, and adversarial training techniques like SMART \cite{smart} and FreeLB \cite{FreeLB} to alleviate the overfitting of data in downstream tasks; Beyond that, \cite{noisytune} proposed NoisyTune with the argument that in addition to the overfitting of the limited downstream data, there could also exist overfitting in pretraining tasks, which could result in enormous gaps between pretraining and downstream task data." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="1909.11764" id="7194">
        <attvalues>
          <attvalue for="5" value=" For example, there are some regularization methods like RecAdam \cite{Recadam} and Mixout \cite{mixout}, and adversarial training techniques like SMART \cite{smart} and FreeLB \cite{FreeLB} to alleviate the overfitting of data in downstream tasks; Beyond that, \cite{noisytune} proposed NoisyTune with the argument that in addition to the overfitting of the limited downstream data, there could also exist overfitting in pretraining tasks, which could result in enormous gaps between pretraining and downstream task data." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="2202.12024" id="7195">
        <attvalues>
          <attvalue for="5" value=" For example, there are some regularization methods like RecAdam \cite{Recadam} and Mixout \cite{mixout}, and adversarial training techniques like SMART \cite{smart} and FreeLB \cite{FreeLB} to alleviate the overfitting of data in downstream tasks; Beyond that, \cite{noisytune} proposed NoisyTune with the argument that in addition to the overfitting of the limited downstream data, there could also exist overfitting in pretraining tasks, which could result in enormous gaps between pretraining and downstream task data." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="1909.11556" id="7196">
        <attvalues>
          <attvalue for="5" value=" Besides, it has also been demonstrated that the existence of a large number of redundant parameters could also be a factor in the suboptimal performances of aggressively fine-tuned PLMs \cite{redundancy1, redundancy2, redundancy3}." />
        </attvalues>
      </edge>
      <edge source="2210.12403" target="2202.02664" id="7197">
        <attvalues>
          <attvalue for="5" value=" Considering the redundant parameters in a model are not insufficiently trained, \cite{sage} proposed a learning rate scheduler named SAGE in which larger learning rates are assigned to these parameters of low sensitivity (a measure of parameter's importance to downstream tasks)." />
        </attvalues>
      </edge>
      <edge source="2310.06201" target="2005.14165" id="7198">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across a wide range of natural language processing tasks, as well as real-life applications \cite{brown2020language,touvron2023llama,bubeck2023sparks}." />
        </attvalues>
      </edge>
      <edge source="2310.06201" target="2303.12712" id="7200">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across a wide range of natural language processing tasks, as well as real-life applications \cite{brown2020language,touvron2023llama,bubeck2023sparks}." />
        </attvalues>
      </edge>
      <edge source="2310.06201" target="1706.03762" id="7201">
        <attvalues>
          <attvalue for="5" value=" However, it is very computationally expensive, particularly with Transformer based LLMs, due to the quadratic growth of memory and computation associated with the 2-D attention matrix \cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2310.06201" target="1904.10509" id="7202">
        <attvalues>
          <attvalue for="5" value=" There are active attempts in reducing the computation and memory cost of the Transformer architecture with sparse attention \cite{child2019generating} or local dense attention \cite{beltagy2020longformer}." />
        </attvalues>
      </edge>
      <edge source="2310.06201" target="2004.05150" id="7203">
        <attvalues>
          <attvalue for="5" value=" There are active attempts in reducing the computation and memory cost of the Transformer architecture with sparse attention \cite{child2019generating} or local dense attention \cite{beltagy2020longformer}." />
        </attvalues>
      </edge>
      <edge source="2310.06201" target="2304.08467" id="7204">
        <attvalues>
          <attvalue for="5" value=" There are also efforts to learn soft prompts with further distillation to save context cost during inference \cite{mu2023learning,chevalier2023adapting}." />
        </attvalues>
      </edge>
      <edge source="2310.06201" target="2305.14788" id="7205">
        <attvalues>
          <attvalue for="5" value=" There are also efforts to learn soft prompts with further distillation to save context cost during inference \cite{mu2023learning,chevalier2023adapting}." />
        </attvalues>
      </edge>
      <edge source="2310.16193" target="2002.05709" id="7206">
        <attvalues>
          <attvalue for="5" value=" In recent years, contrastive learning (CL) has become the go-to method to train representation encoder models \cite{chen2020simple,he2020momentum,gao2021simcse,su2022one}." />
        </attvalues>
      </edge>
      <edge source="2310.16193" target="1911.05722" id="7207">
        <attvalues>
          <attvalue for="5" value=" In recent years, contrastive learning (CL) has become the go-to method to train representation encoder models \cite{chen2020simple,he2020momentum,gao2021simcse,su2022one}." />
        </attvalues>
      </edge>
      <edge source="2310.16193" target="2104.08821" id="7208">
        <attvalues>
          <attvalue for="5" value=" In recent years, contrastive learning (CL) has become the go-to method to train representation encoder models \cite{chen2020simple,he2020momentum,gao2021simcse,su2022one}." />
        </attvalues>
      </edge>
      <edge source="2310.16193" target="2212.09741" id="7209">
        <attvalues>
          <attvalue for="5" value=" In recent years, contrastive learning (CL) has become the go-to method to train representation encoder models \cite{chen2020simple,he2020momentum,gao2021simcse,su2022one}." />
        </attvalues>
      </edge>
      <edge source="2305.14785" target="2302.03494" id="7210">
        <attvalues>
          <attvalue for="5" value=" At the same time, people constantly report LLMs' failures, anecdotal \cite{borji2023categorical} and systematic, e." />
        </attvalues>
      </edge>
      <edge source="2305.14785" target="2304.08979" id="7211">
        <attvalues>
          <attvalue for="5" value="g, the lack of reliability and consistency \cite{shen2023chatgpt, jang2023consistency,plevris2023chatbots}, contradictory or unreasonable answers \cite{zhong2023chatgpt}, inability to detect false assumptions \cite{shen2023chatgpt}, wrong information in prompts \cite {zuccon2023dr}, contradictory responses to identical queries \cite{jang2023consistency, plevris2023chatbots}." />
        </attvalues>
      </edge>
      <edge source="2305.14785" target="2305.18618" id="7212">
        <attvalues>
          <attvalue for="5" value="g, the lack of reliability and consistency \cite{shen2023chatgpt, jang2023consistency,plevris2023chatbots}, contradictory or unreasonable answers \cite{zhong2023chatgpt}, inability to detect false assumptions \cite{shen2023chatgpt}, wrong information in prompts \cite {zuccon2023dr}, contradictory responses to identical queries \cite{jang2023consistency, plevris2023chatbots}." />
        </attvalues>
      </edge>
      <edge source="2305.14785" target="2302.10198" id="7213">
        <attvalues>
          <attvalue for="5" value="g, the lack of reliability and consistency \cite{shen2023chatgpt, jang2023consistency,plevris2023chatbots}, contradictory or unreasonable answers \cite{zhong2023chatgpt}, inability to detect false assumptions \cite{shen2023chatgpt}, wrong information in prompts \cite {zuccon2023dr}, contradictory responses to identical queries \cite{jang2023consistency, plevris2023chatbots}." />
        </attvalues>
      </edge>
      <edge source="2305.14785" target="2303.06273" id="7214">
        <attvalues>
          <attvalue for="5" value="g, the lack of reliability and consistency \cite{shen2023chatgpt, jang2023consistency,plevris2023chatbots}, contradictory or unreasonable answers \cite{zhong2023chatgpt}, inability to detect false assumptions \cite{shen2023chatgpt}, wrong information in prompts \cite {zuccon2023dr}, contradictory responses to identical queries \cite{jang2023consistency, plevris2023chatbots}." />
        </attvalues>
      </edge>
      <edge source="2305.14785" target="1508.05326" id="7215">
        <attvalues>
          <attvalue for="5" value=" recognizing textual entailment \cite{dagan2005pascal,bowman2015snli}), that are easy for humans, and show that they pose a challenge to LLMs." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2107.13586" id="7216">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) have waved the NLP community as fundamental infrastructure by demonstrating remarkable abilities with the ``pre-train, prompt, and predict'' paradigm \cite{liu2021pre,DBLP:journals/corr/abs-2303-18223}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2303.18223" id="7217">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) have waved the NLP community as fundamental infrastructure by demonstrating remarkable abilities with the ``pre-train, prompt, and predict'' paradigm \cite{liu2021pre,DBLP:journals/corr/abs-2303-18223}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="1909.02151" id="7218">
        <attvalues>
          <attvalue for="5" value=" The mere PLMs, however, lack the capacity to handle knowledge-intensive tasks with advanced functionalities like commonsense reasoning \cite{lin-etal-2019-kagnet,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2305-03695} and open-domain question answering \cite{DBLP:conf/emnlp/YangYM15}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2212.09597" id="7219">
        <attvalues>
          <attvalue for="5" value=" The mere PLMs, however, lack the capacity to handle knowledge-intensive tasks with advanced functionalities like commonsense reasoning \cite{lin-etal-2019-kagnet,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2305-03695} and open-domain question answering \cite{DBLP:conf/emnlp/YangYM15}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2305.03695" id="7220">
        <attvalues>
          <attvalue for="5" value=" The mere PLMs, however, lack the capacity to handle knowledge-intensive tasks with advanced functionalities like commonsense reasoning \cite{lin-etal-2019-kagnet,DBLP:journals/corr/abs-2212-09597,DBLP:journals/corr/abs-2305-03695} and open-domain question answering \cite{DBLP:conf/emnlp/YangYM15}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="1704.00051" id="7221">
        <attvalues>
          <attvalue for="5" value=" This necessitates a boosting trend for research focusing on augmenting PLMs with external knowledge sources \cite{chen-etal-2017-reading,DBLP:conf/www/ChenZXDYTHSC22,DBLP:conf/nips/Welleck0BHCCC21,DBLP:conf/nips/WelleckLLHC22,Zhang2022GreaseLMGR,zhang2023multimodal}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2104.07650" id="7222">
        <attvalues>
          <attvalue for="5" value=" This necessitates a boosting trend for research focusing on augmenting PLMs with external knowledge sources \cite{chen-etal-2017-reading,DBLP:conf/www/ChenZXDYTHSC22,DBLP:conf/nips/Welleck0BHCCC21,DBLP:conf/nips/WelleckLLHC22,Zhang2022GreaseLMGR,zhang2023multimodal}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2104.01112" id="7223">
        <attvalues>
          <attvalue for="5" value=" This necessitates a boosting trend for research focusing on augmenting PLMs with external knowledge sources \cite{chen-etal-2017-reading,DBLP:conf/www/ChenZXDYTHSC22,DBLP:conf/nips/Welleck0BHCCC21,DBLP:conf/nips/WelleckLLHC22,Zhang2022GreaseLMGR,zhang2023multimodal}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="1909.01066" id="7226">
        <attvalues>
          <attvalue for="5" value=" Taking pilot experiments as an example, we use knowledge probing \cite{petroni-etal-2019-language} to the PLM as shown in Figure~\ref{fig:overview}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2202.04824" id="7227">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{DBLP:journals/corr/abs-2202-04824} and \cite{DBLP:journals/corr/abs-2210-14803} propose to utilize the knowledge in the pre-traning corpus by retrieve-then-fine-tuning method." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2210.14803" id="7228">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{DBLP:journals/corr/abs-2202-04824} and \cite{DBLP:journals/corr/abs-2210-14803} propose to utilize the knowledge in the pre-traning corpus by retrieve-then-fine-tuning method." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="1908.05739" id="7229">
        <attvalues>
          <attvalue for="5" value=" Likewise, \cite{DBLP:conf/iclr/BhagavatulaBMSH20} capitalizes on the implicit knowledge within large language models (&gt;10B) by retrieving from model weights with recitation-augmented generation." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2211.07349" id="7230">
        <attvalues>
          <attvalue for="5" value=" Subsequently, we consolidate knowledge via FFN to explicitly leverage latent knowledge to help address downstream tasks since FFN plays a crucial role in PLMs \cite{wang-etal-2022-finding-skill}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="1907.11692" id="7231">
        <attvalues>
          <attvalue for="5" value=" We apply the proposed knowledge rumination to various PLMs, including RoBERTa \cite{Liu2019RoBERTaAR}, DeBERTa \cite{he2021deberta}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2006.03654" id="7232">
        <attvalues>
          <attvalue for="5" value=" We apply the proposed knowledge rumination to various PLMs, including RoBERTa \cite{Liu2019RoBERTaAR}, DeBERTa \cite{he2021deberta}." />
        </attvalues>
      </edge>
      <edge source="2305.08732" target="2005.14165" id="7233">
        <attvalues>
          <attvalue for="5" value=" We also transfer knowledge rumination to large language GPT-3 (175B)~\cite{DBLP:journals/corr/abs-2005-14165}." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="1706.03762" id="7234">
        <attvalues>
          <attvalue for="5" value=" In recent years, finetuning large-scale pretrained Transformer \cite{vaswani2017attention} models have been the most successful technique to solve various Natural Language Processing (NLP) tasks such as Machine Translation \cite{edunov2018understanding,raffel2019exploring}, Text Summarization \cite{yan2020prophetnet,takase2019positional}, Question-Answering \cite{zhang2020retrospective,garg2019tanda,dhingra2017linguistic}, Natural Language Inference \cite{zhang2019semantics,lan2019albert}, among others." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="1808.09381" id="7235">
        <attvalues>
          <attvalue for="5" value=" In recent years, finetuning large-scale pretrained Transformer \cite{vaswani2017attention} models have been the most successful technique to solve various Natural Language Processing (NLP) tasks such as Machine Translation \cite{edunov2018understanding,raffel2019exploring}, Text Summarization \cite{yan2020prophetnet,takase2019positional}, Question-Answering \cite{zhang2020retrospective,garg2019tanda,dhingra2017linguistic}, Natural Language Inference \cite{zhang2019semantics,lan2019albert}, among others." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="1904.07418" id="7238">
        <attvalues>
          <attvalue for="5" value=" In recent years, finetuning large-scale pretrained Transformer \cite{vaswani2017attention} models have been the most successful technique to solve various Natural Language Processing (NLP) tasks such as Machine Translation \cite{edunov2018understanding,raffel2019exploring}, Text Summarization \cite{yan2020prophetnet,takase2019positional}, Question-Answering \cite{zhang2020retrospective,garg2019tanda,dhingra2017linguistic}, Natural Language Inference \cite{zhang2019semantics,lan2019albert}, among others." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="2001.09694" id="7239">
        <attvalues>
          <attvalue for="5" value=" In recent years, finetuning large-scale pretrained Transformer \cite{vaswani2017attention} models have been the most successful technique to solve various Natural Language Processing (NLP) tasks such as Machine Translation \cite{edunov2018understanding,raffel2019exploring}, Text Summarization \cite{yan2020prophetnet,takase2019positional}, Question-Answering \cite{zhang2020retrospective,garg2019tanda,dhingra2017linguistic}, Natural Language Inference \cite{zhang2019semantics,lan2019albert}, among others." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="1911.04118" id="7240">
        <attvalues>
          <attvalue for="5" value=" In recent years, finetuning large-scale pretrained Transformer \cite{vaswani2017attention} models have been the most successful technique to solve various Natural Language Processing (NLP) tasks such as Machine Translation \cite{edunov2018understanding,raffel2019exploring}, Text Summarization \cite{yan2020prophetnet,takase2019positional}, Question-Answering \cite{zhang2020retrospective,garg2019tanda,dhingra2017linguistic}, Natural Language Inference \cite{zhang2019semantics,lan2019albert}, among others." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="1909.02209" id="7242">
        <attvalues>
          <attvalue for="5" value=" In recent years, finetuning large-scale pretrained Transformer \cite{vaswani2017attention} models have been the most successful technique to solve various Natural Language Processing (NLP) tasks such as Machine Translation \cite{edunov2018understanding,raffel2019exploring}, Text Summarization \cite{yan2020prophetnet,takase2019positional}, Question-Answering \cite{zhang2020retrospective,garg2019tanda,dhingra2017linguistic}, Natural Language Inference \cite{zhang2019semantics,lan2019albert}, among others." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="1909.11942" id="7243">
        <attvalues>
          <attvalue for="5" value=" In recent years, finetuning large-scale pretrained Transformer \cite{vaswani2017attention} models have been the most successful technique to solve various Natural Language Processing (NLP) tasks such as Machine Translation \cite{edunov2018understanding,raffel2019exploring}, Text Summarization \cite{yan2020prophetnet,takase2019positional}, Question-Answering \cite{zhang2020retrospective,garg2019tanda,dhingra2017linguistic}, Natural Language Inference \cite{zhang2019semantics,lan2019albert}, among others." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="1604.02201" id="7244">
        <attvalues>
          <attvalue for="5" value=" This method has been shown to be effective in various low-resource tasks such as Low-resource Machine Translation \cite{zoph2016transfer}, Cross-lingual Language Modeling \cite{adams2017cross}, Named Entity Recognition \cite{das2017named}, Fake News Detection \cite{cruz2019localization}, and many more." />
        </attvalues>
      </edge>
      <edge source="2005.02068" target="1907.00409" id="7247">
        <attvalues>
          <attvalue for="5" value=" Second, we pretrain stronger BERT \cite{devlin2018bert} models in Filipino, with larger input sequence lengths than our previous Tagalog-BERT models \cite{cruz2019evaluating}." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="2305.03514" id="7253">
        <attvalues>
          <attvalue for="5" value=" While decoder-based generative models have shown significant capabilities in generating coherent and contextually relevant language \cite{Shahriar_Hayawi_2023}, many studies have consistently demonstrated that BERT-family encoders fine-tuned with carefully crafted data are more reliable in specialized classification tasks (see eg \cite{ziems2023, pahwa-pahwa-2023-bphigh, li2023chatgpt, bang2023multitask})." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="2004.09456" id="7254">
        <attvalues>
          <attvalue for="5" value=" StereoSet \cite{nadeem-etal-2021-stereoset} is currently a leading test set for reporting on intrinsic bias in BERT, as observed through the NSP task (note that StereoSet contains test sets for both language modelling and NSP, but here we focus on NSP only)." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="2007.08100" id="7256">
        <attvalues>
          <attvalue for="5" value=" Previously, debias-by-projection has been applied to the final output sentence representation only \cite{liang-etal-2020-towards-sent-debias, Bhardwaj2021Jul}, but has not yet been attempted within BERT's inner layers." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="1904.01172" id="7258">
        <attvalues>
          <attvalue for="5" value=" NLI is a fundamental NLP task that involves determining the relationship between two sentences \cite{storks2019recent}." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="2104.08731" id="7259">
        <attvalues>
          <attvalue for="5" value=" More specifically, NLI is used to improve Question-answering models \cite{chen-etal-2021-nli-models, fortier-dubois-rosati, PARAMASIVAM20229644}, dialogue systems \cite{Chen2019-dialog}, and content verification models \cite{falke-etal-2019-ranking, dusek-kasner-2020-evaluating}." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="2003.02126" id="7260">
        <attvalues>
          <attvalue for="5" value=" More specifically, NLI is used to improve Question-answering models \cite{chen-etal-2021-nli-models, fortier-dubois-rosati, PARAMASIVAM20229644}, dialogue systems \cite{Chen2019-dialog}, and content verification models \cite{falke-etal-2019-ranking, dusek-kasner-2020-evaluating}." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="1607.06520" id="7261">
        <attvalues>
          <attvalue for="5" value=" Applying something akin to hard debias \cite{Tolga:2016} to the final sentence representation output by a language model \cite{liang-etal-2020-towards-sent-debias, Bhardwaj2021Jul} has been suggested as a way to create debiased contextual sentence representations." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="2109.14039" id="7263">
        <attvalues>
          <attvalue for="5" value=" Furthermore, we experiment with the use of information weighting \cite{dawkins-2021-marked} paired with the use of higher-dimensional gender subspaces." />
        </attvalues>
      </edge>
      <edge source="2403.18803" target="2010.12864" id="7264">
        <attvalues>
          <attvalue for="5" value=" In a related work, \cite{jin-etal-2021-transferability} find that bias mitigation by finetuning an upstream model can be transferred to the downstream setting." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="1910.00194" id="7265">
        <attvalues>
          <attvalue for="5" value=" Pretrained contextualized models (PCMs) have brought large improvements in these tasks including WSD \cite{hadiwinoto-etal-2019-improved,loureiro-jorge-2019-language,huang-etal-2019-glossbert,blevins-zettlemoyer-2020-moving}, WiC \cite{pilehvar-camacho-collados-2019-wic,gari-soler-etal-2019-word} and entity linking (EL) \cite{wu-etal-2020-scalable,broscheit-2019-investigating}." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="1906.10007" id="7266">
        <attvalues>
          <attvalue for="5" value=" Pretrained contextualized models (PCMs) have brought large improvements in these tasks including WSD \cite{hadiwinoto-etal-2019-improved,loureiro-jorge-2019-language,huang-etal-2019-glossbert,blevins-zettlemoyer-2020-moving}, WiC \cite{pilehvar-camacho-collados-2019-wic,gari-soler-etal-2019-word} and entity linking (EL) \cite{wu-etal-2020-scalable,broscheit-2019-investigating}." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="1908.07245" id="7267">
        <attvalues>
          <attvalue for="5" value=" Pretrained contextualized models (PCMs) have brought large improvements in these tasks including WSD \cite{hadiwinoto-etal-2019-improved,loureiro-jorge-2019-language,huang-etal-2019-glossbert,blevins-zettlemoyer-2020-moving}, WiC \cite{pilehvar-camacho-collados-2019-wic,gari-soler-etal-2019-word} and entity linking (EL) \cite{wu-etal-2020-scalable,broscheit-2019-investigating}." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="1905.08377" id="7269">
        <attvalues>
          <attvalue for="5" value=" Pretrained contextualized models (PCMs) have brought large improvements in these tasks including WSD \cite{hadiwinoto-etal-2019-improved,loureiro-jorge-2019-language,huang-etal-2019-glossbert,blevins-zettlemoyer-2020-moving}, WiC \cite{pilehvar-camacho-collados-2019-wic,gari-soler-etal-2019-word} and entity linking (EL) \cite{wu-etal-2020-scalable,broscheit-2019-investigating}." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="1911.03814" id="7270">
        <attvalues>
          <attvalue for="5" value=" Pretrained contextualized models (PCMs) have brought large improvements in these tasks including WSD \cite{hadiwinoto-etal-2019-improved,loureiro-jorge-2019-language,huang-etal-2019-glossbert,blevins-zettlemoyer-2020-moving}, WiC \cite{pilehvar-camacho-collados-2019-wic,gari-soler-etal-2019-word} and entity linking (EL) \cite{wu-etal-2020-scalable,broscheit-2019-investigating}." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="2003.05473" id="7271">
        <attvalues>
          <attvalue for="5" value=" Pretrained contextualized models (PCMs) have brought large improvements in these tasks including WSD \cite{hadiwinoto-etal-2019-improved,loureiro-jorge-2019-language,huang-etal-2019-glossbert,blevins-zettlemoyer-2020-moving}, WiC \cite{pilehvar-camacho-collados-2019-wic,gari-soler-etal-2019-word} and entity linking (EL) \cite{wu-etal-2020-scalable,broscheit-2019-investigating}." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="2010.05731" id="7272">
        <attvalues>
          <attvalue for="5" value=" Specifically, \cite{vulic-etal-2020-probing, aina-etal-2019-putting} found language models `contextualize' words in higher layers while the type-level information is better kept in lower layers." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="1909.01380" id="7273">
        <attvalues>
          <attvalue for="5" value=" \cite{voita-etal-2019-bottom} point out different learning objectives affect the contextualization process, and \cite{gari-soler-apidianaki-2021-lets, pimentel-etal-2020-speakers} show PCMs can capture words' ambiguity levels." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="2104.14694" id="7274">
        <attvalues>
          <attvalue for="5" value=" \cite{voita-etal-2019-bottom} point out different learning objectives affect the contextualization process, and \cite{gari-soler-apidianaki-2021-lets, pimentel-etal-2020-speakers} show PCMs can capture words' ambiguity levels." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="2012.15180" id="7275">
        <attvalues>
          <attvalue for="5" value=" The design of the probing baselines follows previous studies that applied input perturbation techniques for model and task analysis in GLUE \cite{pham2020out}, NLI \cite{poliak-etal-2018-hypothesis,wang-etal-2018-glue,talman2021nli} and relation extraction \cite{peng-etal-2020-learning}." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="1805.01042" id="7276">
        <attvalues>
          <attvalue for="5" value=" The design of the probing baselines follows previous studies that applied input perturbation techniques for model and task analysis in GLUE \cite{pham2020out}, NLI \cite{poliak-etal-2018-hypothesis,wang-etal-2018-glue,talman2021nli} and relation extraction \cite{peng-etal-2020-learning}." />
        </attvalues>
      </edge>
      <edge source="2112.06733" target="2010.01923" id="7279">
        <attvalues>
          <attvalue for="5" value=" The design of the probing baselines follows previous studies that applied input perturbation techniques for model and task analysis in GLUE \cite{pham2020out}, NLI \cite{poliak-etal-2018-hypothesis,wang-etal-2018-glue,talman2021nli} and relation extraction \cite{peng-etal-2020-learning}." />
        </attvalues>
      </edge>
      <edge source="2407.10795" target="2210.15097" id="7280">
        <attvalues>
          <attvalue for="5" value=" This reduces the probability of the expert model to make similar mistakes as the amateur model, thus making the generation content more logical and coherent~\cite{li-etal-2023-contrastive, o2023contrastive, zhao2024enhancing}." />
        </attvalues>
      </edge>
      <edge source="2407.10795" target="2402.10588" id="7282">
        <attvalues>
          <attvalue for="5" value=" The performance gap between our approach and DoLa on the multilingual benchmark also validates the findings about the language transition of intermediate decodings across the layers of LLMs by \cite{wendler2024llamas} and provides further insight into the working patterns of LLMs." />
        </attvalues>
      </edge>
      <edge source="2407.10795" target="2307.09288" id="7283">
        <attvalues>
          <attvalue for="5" value=" Our experimental results on multilingual reasoning benchmarks mGSM show that our devised approach significantly outperforms the previous contrastive decoding approach DoLa, and improves the chain-of-thought reasoning accuracy of a group of open-source LLMs: LLaMA2~\cite{llama2}, LLaMA3~\cite{meta2024introducing}, Mistral~\cite{jiang2023mistral}, etc, across 11 languages." />
        </attvalues>
      </edge>
      <edge source="2407.10795" target="2310.06825" id="7284">
        <attvalues>
          <attvalue for="5" value=" Our experimental results on multilingual reasoning benchmarks mGSM show that our devised approach significantly outperforms the previous contrastive decoding approach DoLa, and improves the chain-of-thought reasoning accuracy of a group of open-source LLMs: LLaMA2~\cite{llama2}, LLaMA3~\cite{meta2024introducing}, Mistral~\cite{jiang2023mistral}, etc, across 11 languages." />
        </attvalues>
      </edge>
      <edge source="2208.02070" target="1906.02243" id="7285">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning is, however, complicated by the size of PLMs \cite{energy_in_NLP, green_ai}." />
        </attvalues>
      </edge>
      <edge source="2208.02070" target="2002.11985" id="7286">
        <attvalues>
          <attvalue for="5" value=" Works in model compression have sought to improve inference efficiency of these models by pruning, quantization, and distillation \cite{Ganesh2021CompressingLT, survey_on_device_ML}." />
        </attvalues>
      </edge>
      <edge source="2208.02070" target="1602.01528" id="7288">
        <attvalues>
          <attvalue for="5" value=" Further, memory operations are energy intensive and slow \cite{songhan_eie, comp_arch_hennessy_patterson_2012}." />
        </attvalues>
      </edge>
      <edge source="2208.02070" target="1811.03604" id="7289">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning must be made fast and resource-efficient to facilitate use-cases such as personalized auto-correct systems on mobile device keyboards, or green AI \cite{federated_mobile_keyboard_example, survey_on_device_ML, green_ai}." />
        </attvalues>
      </edge>
      <edge source="2208.02070" target="2106.10199" id="7290">
        <attvalues>
          <attvalue for="5" value=" Prior works in efficient training focus primarily on parameter efficiency for uncompressed (eg, huge) transformer-based models \cite{bitfit, houlsby2019parameter_eff_adapter, he2022towards_unified_param_eff_parallel_adapter, guo-etal-2021-parameter-diff-pruning}." />
        </attvalues>
      </edge>
      <edge source="2208.02070" target="1902.00751" id="7291">
        <attvalues>
          <attvalue for="5" value=" Adapters \cite{houlsby2019parameter_eff_adapter} manage to achieve parameter efficiency in a computationally efficient manner, while difference pruning \cite{guo-etal-2021-parameter-diff-pruning} requires triple the training-time parameter usage." />
        </attvalues>
      </edge>
      <edge source="2208.02070" target="2012.07463" id="7292">
        <attvalues>
          <attvalue for="5" value=" Adapters \cite{houlsby2019parameter_eff_adapter} manage to achieve parameter efficiency in a computationally efficient manner, while difference pruning \cite{guo-etal-2021-parameter-diff-pruning} requires triple the training-time parameter usage." />
        </attvalues>
      </edge>
      <edge source="2208.02070" target="2205.01541" id="7293">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art methods in efficient fine-tuning, namely Freeze-and-Reconfigure (FAR), are decidedly quick to converge, but require a far greater proportion of parameters and are plagued by slow memory operations \cite{FAR}." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="1912.06670" id="7294">
        <attvalues>
          <attvalue for="5" value=" Multilingual speech corpora have limited coverage of speech-related tasks, primarily focusing on automatic speech recognition (ASR) \cite{commonvoice,fleurs,MaSS,multilinguallibrispeech} and speech translation (ST) \cite{mustc,mtedx,europarlst,covost2}, while neglecting spoken language understanding (SLU -- the task of extracting semantic information from spoken utterances, which typically involves subtasks like intent detection and slot filling)." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="2012.03411" id="7296">
        <attvalues>
          <attvalue for="5" value=" Multilingual speech corpora have limited coverage of speech-related tasks, primarily focusing on automatic speech recognition (ASR) \cite{commonvoice,fleurs,MaSS,multilinguallibrispeech} and speech translation (ST) \cite{mustc,mtedx,europarlst,covost2}, while neglecting spoken language understanding (SLU -- the task of extracting semantic information from spoken utterances, which typically involves subtasks like intent detection and slot filling)." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="1911.03167" id="7298">
        <attvalues>
          <attvalue for="5" value=" Multilingual speech corpora have limited coverage of speech-related tasks, primarily focusing on automatic speech recognition (ASR) \cite{commonvoice,fleurs,MaSS,multilinguallibrispeech} and speech translation (ST) \cite{mustc,mtedx,europarlst,covost2}, while neglecting spoken language understanding (SLU -- the task of extracting semantic information from spoken utterances, which typically involves subtasks like intent detection and slot filling)." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="1910.07475" id="7299">
        <attvalues>
          <attvalue for="5" value=" Unlike text processing, where extensive efforts in natural language understanding (NLU) have led to resources covering a wide range of languages \cite{mlqa,multi3nlu,multiatis,massive}, SLU datasets are mainly English-centric \cite{slurp}, with few exceptions \cite{snips,portmedia,italic}." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="2212.10455" id="7300">
        <attvalues>
          <attvalue for="5" value=" Unlike text processing, where extensive efforts in natural language understanding (NLU) have led to resources covering a wide range of languages \cite{mlqa,multi3nlu,multiatis,massive}, SLU datasets are mainly English-centric \cite{slurp}, with few exceptions \cite{snips,portmedia,italic}." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="2004.14353" id="7301">
        <attvalues>
          <attvalue for="5" value=" Unlike text processing, where extensive efforts in natural language understanding (NLU) have led to resources covering a wide range of languages \cite{mlqa,multi3nlu,multiatis,massive}, SLU datasets are mainly English-centric \cite{slurp}, with few exceptions \cite{snips,portmedia,italic}." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="2204.08582" id="7302">
        <attvalues>
          <attvalue for="5" value=" We start with the MASSIVE NLU (eg textual) dataset \cite{massive}, an ideal foundation due to its size, domain diversity, and broad coverage of languages, intent, and slot types." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="2011.13205" id="7303">
        <attvalues>
          <attvalue for="5" value=" Developed by commissioning professional translators to localize the English SLURP dataset \cite{slurp} into 51 languages, MASSIVE comprises 1M labeled utterances spanning 18 domains, with 60 intents and 55 slots." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="1805.10190" id="7304">
        <attvalues>
          <attvalue for="5" value=" Unlike text processing, where extensive efforts in natural language understanding (NLU) have led to resources covering a wide range of languages \cite{mlqa,multi3nlu,multiatis,massive}, SLU datasets are mainly English-centric \cite{slurp}, with few exceptions \cite{snips,portmedia,italic}." />
        </attvalues>
      </edge>
      <edge source="2408.03900" target="2306.08502" id="7305">
        <attvalues>
          <attvalue for="5" value=" Our goal is to bridge the gap in multilingual SLU drawing inspiration from \cite{italic} and collecting speech recordings in multiple languages." />
        </attvalues>
      </edge>
      <edge source="2402.09025" target="2005.14165" id="7306">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as GPT-3, OPT, and LLaMA demonstrate exceptional proficiency in a variety of natural language processing (NLP) tasks and have become key components in applications like chatbots and question-answering systems~\cite{gpt3, opt, palm, llama_v1, llama_v2}." />
        </attvalues>
      </edge>
      <edge source="2402.09025" target="2208.11580" id="7307">
        <attvalues>
          <attvalue for="5" value=" Its application in LLMs has been somewhat limited, primarily due to challenges that arise in managing sparse matrices~\cite{obc, wanda, sparsegpt, DSnoT}." />
        </attvalues>
      </edge>
      <edge source="2402.09025" target="2008.11849" id="7308">
        <attvalues>
          <attvalue for="5" value=" This complexity becomes particularly evident when using modern GPU hardware, as these systems are typically optimized for operations involving dense matrices~\cite{sparsert, sparsegpu}." />
        </attvalues>
      </edge>
      <edge source="2402.09025" target="2303.09435" id="7309">
        <attvalues>
          <attvalue for="5" value=" In the realm of LLMs, a significant similarity in output is observed among successive transformer blocks~\cite{jump_to_con, dejavu}." />
        </attvalues>
      </edge>
      <edge source="2204.12820" target="2105.14504" id="7310">
        <attvalues>
          <attvalue for="5" value=" More particularly, \cite{barnes-etal-2021-structured} consider sentiment analysis as a (graph) structured task, and discuss up to five subtasks: (i) sentiment expression extraction, (ii) sentiment target extraction, (iii) sentiment holder extraction, (iv) defining the relationship between these elements, and (v) assigning a polarity label." />
        </attvalues>
      </edge>
      <edge source="2204.12820" target="cs/0212032" id="7311">
        <attvalues>
          <attvalue for="5" value=" They discuss that although these tasks have been extensively studied by different authors \cite[inter alia]{turney-2002-thumbs,pontiki-etal-2015-semeval,zhang2019end}, they are not addressed all together." />
        </attvalues>
      </edge>
      <edge source="2204.12820" target="1807.01396" id="7312">
        <attvalues>
          <attvalue for="5" value=" More specifically, we rely on a bi-affine graph-based parser \cite{dozat-manning-2018-simpler} and different large pre-trained language models (LM), such as BERT \cite{devlin-etal-2019-bert}, RoBERTa \cite{liu2019roberta} or XLM-R \cite{conneau-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2204.12820" target="1907.11692" id="7314">
        <attvalues>
          <attvalue for="5" value=" More specifically, we rely on a bi-affine graph-based parser \cite{dozat-manning-2018-simpler} and different large pre-trained language models (LM), such as BERT \cite{devlin-etal-2019-bert}, RoBERTa \cite{liu2019roberta} or XLM-R \cite{conneau-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2204.12820" target="1911.02116" id="7315">
        <attvalues>
          <attvalue for="5" value=" More specifically, we rely on a bi-affine graph-based parser \cite{dozat-manning-2018-simpler} and different large pre-trained language models (LM), such as BERT \cite{devlin-etal-2019-bert}, RoBERTa \cite{liu2019roberta} or XLM-R \cite{conneau-etal-2020-unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2406.12621" target="2305.04572" id="7316">
        <attvalues>
          <attvalue for="5" value=" With the popularization of self-supervised method and modern neural network architecture (pretrained transformers), both speech and text domains now use similar techniques \cite{chrupala-2023-putting}." />
        </attvalues>
      </edge>
      <edge source="2405.05955" target="2303.18223" id="7323">
        <attvalues>
          <attvalue for="5" value=" For example, the search engine API empowers ChatGPT to access real-time information~\cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2405.05955" target="2305.11554" id="7324">
        <attvalues>
          <attvalue for="5" value="~\cite{hao2024toolkengpt, guu2020retrieval, qin2024toolllm}." />
        </attvalues>
      </edge>
      <edge source="2405.05955" target="2403.07714" id="7325">
        <attvalues>
          <attvalue for="5" value=" The effectiveness of Smurfs is validated through both open-ended and closed-ended tool planning benchmark experiments~\cite{guo2024stabletoolbench, yang2018hotpotqa}, where the proposed MAS system consistently outperform baseline methods on both benchmarks." />
        </attvalues>
      </edge>
      <edge source="2306.05969" target="2010.05465" id="7326">
        <attvalues>
          <attvalue for="5" value=" Many studies have demonstrated language models' ability to extend a generalization from a small set of examples to novel lexical items, structures, and contexts, even if the models do not always do so in a human-like way \cite{hupkes2020,kim-linzen-2020-cogs,lake2018,mccoy2018}." />
        </attvalues>
      </edge>
      <edge source="2306.05969" target="1802.09091" id="7328">
        <attvalues>
          <attvalue for="5" value=" Many studies have demonstrated language models' ability to extend a generalization from a small set of examples to novel lexical items, structures, and contexts, even if the models do not always do so in a human-like way \cite{hupkes2020,kim-linzen-2020-cogs,lake2018,mccoy2018}." />
        </attvalues>
      </edge>
      <edge source="2306.05969" target="2010.02375" id="7330">
        <attvalues>
          <attvalue for="5" value=" This studies have shown, for example, that the GPT-2 language model \cite{radford2019} can match human judgments about whether the dative alternation applies to a verb \cite{hawkins2020}, and that information about which syntactic frames a verb can appear in (eg whether a verb participates in the spray/load alternation) can be recovered from the verb's contextualized representations and from sentence embeddings \cite{kann-etal-2019-verb}." />
        </attvalues>
      </edge>
      <edge source="2306.05969" target="1811.10773" id="7331">
        <attvalues>
          <attvalue for="5" value=" This studies have shown, for example, that the GPT-2 language model \cite{radford2019} can match human judgments about whether the dative alternation applies to a verb \cite{hawkins2020}, and that information about which syntactic frames a verb can appear in (eg whether a verb participates in the spray/load alternation) can be recovered from the verb's contextualized representations and from sentence embeddings \cite{kann-etal-2019-verb}." />
        </attvalues>
      </edge>
      <edge source="2311.05640" target="1706.03762" id="7332">
        <attvalues>
          <attvalue for="5" value=" Neural language models based on the Transformer architecture \cite{vaswani2017attention} have revolutionized Natural Language Processing (NLP) in recent years, advancing the state of the art in tasks ranging from text classification to open-ended text generation." />
        </attvalues>
      </edge>
      <edge source="2311.05640" target="2005.14165" id="7333">
        <attvalues>
          <attvalue for="5" value=" The ability of such models to implicitly learn to perform tasks that they have not been directly trained on has been considered to be closely tied to the scale of the model \cite{brown2020language,chowdhery2022palm} and, perhaps even more importantly, to the number of training tokens \cite{hoffmann2022training, muennighoff2023scaling,touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2311.05640" target="2204.02311" id="7334">
        <attvalues>
          <attvalue for="5" value=" The ability of such models to implicitly learn to perform tasks that they have not been directly trained on has been considered to be closely tied to the scale of the model \cite{brown2020language,chowdhery2022palm} and, perhaps even more importantly, to the number of training tokens \cite{hoffmann2022training, muennighoff2023scaling,touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2311.05640" target="2203.15556" id="7335">
        <attvalues>
          <attvalue for="5" value=" The ability of such models to implicitly learn to perform tasks that they have not been directly trained on has been considered to be closely tied to the scale of the model \cite{brown2020language,chowdhery2022palm} and, perhaps even more importantly, to the number of training tokens \cite{hoffmann2022training, muennighoff2023scaling,touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2311.05640" target="2006.01538" id="7337">
        <attvalues>
          <attvalue for="5" value=" While the language is comparatively well represented in online resources relative to this number, less than 1\% of texts available in eg\ Wikipedia and Common Crawl are Finnish \cite{pyysalo2021wikibert,xue2021mt5}." />
        </attvalues>
      </edge>
      <edge source="2311.05640" target="2010.11934" id="7338">
        <attvalues>
          <attvalue for="5" value=" While the language is comparatively well represented in online resources relative to this number, less than 1\% of texts available in eg\ Wikipedia and Common Crawl are Finnish \cite{pyysalo2021wikibert,xue2021mt5}." />
        </attvalues>
      </edge>
      <edge source="2311.05640" target="2211.05100" id="7340">
        <attvalues>
          <attvalue for="5" value=" We also perform continued pretraining of the 176-billion parameter BLOOM model \cite{scao2022bloom} to extend its coverage of Finnish, introduce novel evaluation datasets, and assess multiple aspects of the resulting models." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="1705.02364" id="7341">
        <attvalues>
          <attvalue for="5" value=" We further extend the multilingual CMLM to co-train with parallel text (bitext) retrieval task, and finetune with cross-lingual natural language inference (NLI) data, inspired by the success of prior work on multitask sentence representation learning~\cite{subramanian2018learning,muse,distill-mling} and NLI learning~\cite{infersent,sentbert}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="1804.00079" id="7342">
        <attvalues>
          <attvalue for="5" value=" We further extend the multilingual CMLM to co-train with parallel text (bitext) retrieval task, and finetune with cross-lingual natural language inference (NLI) data, inspired by the success of prior work on multitask sentence representation learning~\cite{subramanian2018learning,muse,distill-mling} and NLI learning~\cite{infersent,sentbert}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="1803.02893" id="7343">
        <attvalues>
          <attvalue for="5" value=" The vectors capture rich semantic information that can be used to measure semantic textual similarity~(STS) between sentences or train classifiers for a broad range of downstream tasks~\cite{infersent,subramanian2018learning,qk,use,sentbert,muse,gem}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="1908.10084" id="7345">
        <attvalues>
          <attvalue for="5" value=" We further extend the multilingual CMLM to co-train with parallel text (bitext) retrieval task, and finetune with cross-lingual natural language inference (NLI) data, inspired by the success of prior work on multitask sentence representation learning~\cite{subramanian2018learning,muse,distill-mling} and NLI learning~\cite{infersent,sentbert}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="1907.04307" id="7346">
        <attvalues>
          <attvalue for="5" value=" We further extend the multilingual CMLM to co-train with parallel text (bitext) retrieval task, and finetune with cross-lingual natural language inference (NLI) data, inspired by the success of prior work on multitask sentence representation learning~\cite{subramanian2018learning,muse,distill-mling} and NLI learning~\cite{infersent,sentbert}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="1810.00438" id="7347">
        <attvalues>
          <attvalue for="5" value=" The vectors capture rich semantic information that can be used to measure semantic textual similarity~(STS) between sentences or train classifiers for a broad range of downstream tasks~\cite{infersent,subramanian2018learning,qk,use,sentbert,muse,gem}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="1909.11942" id="7349">
        <attvalues>
          <attvalue for="5" value=" Conversely, recent efforts to improve language models include the development of masked language model (MLM) pre-training from large scale unlabeled corpora \cite{bert,albert,roberta}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="1907.11692" id="7350">
        <attvalues>
          <attvalue for="5" value=" Conversely, recent efforts to improve language models include the development of masked language model (MLM) pre-training from large scale unlabeled corpora \cite{bert,albert,roberta}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="2007.01852" id="7351">
        <attvalues>
          <attvalue for="5" value=" While internal MLM model representations are helpful when fine-tuning on downstream tasks, they do not directly produce good sentence representations, without further supervised \cite{sentbert} or semi-structured \cite{labse} fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="2004.09813" id="7353">
        <attvalues>
          <attvalue for="5" value=" We further extend the multilingual CMLM to co-train with parallel text (bitext) retrieval task, and finetune with cross-lingual natural language inference (NLI) data, inspired by the success of prior work on multitask sentence representation learning~\cite{subramanian2018learning,muse,distill-mling} and NLI learning~\cite{infersent,sentbert}." />
        </attvalues>
      </edge>
      <edge source="2012.14388" target="2004.05484" id="7354">
        <attvalues>
          <attvalue for="5" value=" Language agnostic representations require semantically similar cross-lingual pairs to be closer in representation space than unrelated same-language pairs~\cite{lareqa}." />
        </attvalues>
      </edge>
      <edge source="2312.11920" target="1907.01749" id="7355">
        <attvalues>
          <attvalue for="5" value=" Furthermore, a few studies show that taking advantage of multi-level embedding features is helpful, including characters, words, and their positions \cite{cai2019polyphone,dai2019disambiguation}." />
        </attvalues>
      </edge>
      <edge source="2312.11920" target="1506.00196" id="7356">
        <attvalues>
          <attvalue for="5" value=" A paradigm of encoder-decoder generation is employed in the task to train a Seq2Seq model with attention mechanism \cite{yao2015sequence,zhang2020distant}." />
        </attvalues>
      </edge>
      <edge source="2312.11920" target="2005.14165" id="7357">
        <attvalues>
          <attvalue for="5" value=" GPT \cite{radford2018improving,brown2020language} uses a massive amount of unlabeled text to pre-train a multi-layer Transformer decoder which shows better results on both natural language understanding and generation tasks." />
        </attvalues>
      </edge>
      <edge source="2312.11920" target="2103.10360" id="7358">
        <attvalues>
          <attvalue for="5" value=" Based on autoregressive models, GLM \cite{du2021glm} improves blank filling pretraining by 2D positional encodings and supports different types of tasks by varying the number and lengths of blanks." />
        </attvalues>
      </edge>
      <edge source="2203.15643" target="2006.04558" id="7359">
        <attvalues>
          <attvalue for="5" value=" Synthetic voices generated by several recent neural TTS models~\cite{ren2020fastspeech, donahue2020end, kim2020glow, kim2021conditional} have been able to achieve high naturalness and intelligibility compared to the real recordings it was trained on." />
        </attvalues>
      </edge>
      <edge source="2203.15643" target="2008.03802" id="7360">
        <attvalues>
          <attvalue for="5" value=" However, in contrast with previous work~\cite{vainer2020speedyspeech} that performed KD by only distilled the teacher network’s duration or the work by ~\cite{oord2018parallel, ping2018clarinet} that only utilized KD by compressing neural vocoders, our proposed approach was performed on an end-to-end TTS model." />
        </attvalues>
      </edge>
      <edge source="2203.15643" target="1712.05884" id="7361">
        <attvalues>
          <attvalue for="5" value="3$M to $15$M, more than a half smaller compared to established high-quality TTS like~\cite{tacotron2, ren2020fastspeech} which is around $30$M in size." />
        </attvalues>
      </edge>
      <edge source="2203.15643" target="2102.04040" id="7362">
        <attvalues>
          <attvalue for="5" value=" To date, ~\cite{luo2021lightspeech} proposed applying neural architecture search to automatically design an acoustic model; the discovered model is only $1." />
        </attvalues>
      </edge>
      <edge source="2203.15643" target="2110.01147" id="7363">
        <attvalues>
          <attvalue for="5" value=" Another work by~\cite{lai2021interplay} shows that a neural acoustic model is highly prunable." />
        </attvalues>
      </edge>
      <edge source="2203.15643" target="1503.02531" id="7364">
        <attvalues>
          <attvalue for="5" value=" In this work, we present Nix-TTS, a lightweight TTS achieved via knowledge distillation (KD)~\cite{hinton2015distilling} to a high-quality yet large-sized, non-autoregressive, and end-to-end (vocoder-free) TTS teacher model." />
        </attvalues>
      </edge>
      <edge source="2203.15643" target="1711.10433" id="7365">
        <attvalues>
          <attvalue for="5" value=" However, in contrast with previous work~\cite{vainer2020speedyspeech} that performed KD by only distilled the teacher network’s duration or the work by ~\cite{oord2018parallel, ping2018clarinet} that only utilized KD by compressing neural vocoders, our proposed approach was performed on an end-to-end TTS model." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="1601.01280" id="7366">
        <attvalues>
          <attvalue for="5" value=" Sequence-to-sequence (seq2seq) models have been widely used in semantic parsing~\cite{dong2016language, jia2016data} and excel at handling the natural language variation of human-generated queries." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="1910.02612" id="7368">
        <attvalues>
          <attvalue for="5" value=" This has motivated many specialized architectures that improve peformance on SCAN~\cite{li2019compositional,russin2019compositional,gordon2019permutation,lake2019compositional,liu2020compositional,nye2020learning,chen2020compositional}." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="1906.05381" id="7370">
        <attvalues>
          <attvalue for="5" value=" This has motivated many specialized architectures that improve peformance on SCAN~\cite{li2019compositional,russin2019compositional,gordon2019permutation,lake2019compositional,liu2020compositional,nye2020learning,chen2020compositional}." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="2006.10627" id="7371">
        <attvalues>
          <attvalue for="5" value=" This has motivated many specialized architectures that improve peformance on SCAN~\cite{li2019compositional,russin2019compositional,gordon2019permutation,lake2019compositional,liu2020compositional,nye2020learning,chen2020compositional}." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="2003.05562" id="7372">
        <attvalues>
          <attvalue for="5" value=" This has motivated many specialized architectures that improve peformance on SCAN~\cite{li2019compositional,russin2019compositional,gordon2019permutation,lake2019compositional,liu2020compositional,nye2020learning,chen2020compositional}." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="2008.06662" id="7373">
        <attvalues>
          <attvalue for="5" value=" This has motivated many specialized architectures that improve peformance on SCAN~\cite{li2019compositional,russin2019compositional,gordon2019permutation,lake2019compositional,liu2020compositional,nye2020learning,chen2020compositional}." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="2007.08970" id="7374">
        <attvalues>
          <attvalue for="5" value="} For example, large pre-trained seq2seq models that perform well on in-distribution evaluations do not address most of the compositional generalization challenges proposed in SCAN~\cite{furrer2020compositional}." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="1901.04587" id="7375">
        <attvalues>
          <attvalue for="5" value=" First, humans have been shown to be adept compositional learners~\cite{lake2019human}." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="1806.01261" id="7377">
        <attvalues>
          <attvalue for="5" value=" Several authors have argued that a greater focus on compositional generalization is an important path to more human-like generalization and NLU~\cite{lake2017building,battaglia2018relational}." />
        </attvalues>
      </edge>
      <edge source="2010.12725" target="1912.09713" id="7379">
        <attvalues>
          <attvalue for="5" value=" We also propose new Target Maximum Compound Divergence (\tmcd) train and test splits, extending the methodology of~\cite{keysers2019measuring} to create challenging evaluations of compositional generalization for non-synthetic datasets." />
        </attvalues>
      </edge>
      <edge source="2102.00804" target="2005.11640" id="7382">
        <attvalues>
          <attvalue for="5" value=" \cite{liu2020jointlyArxiv} proposes a BERT model that jointly encodes the word confidence network and the dialog context." />
        </attvalues>
      </edge>
      <edge source="2102.00804" target="2002.00750" id="7383">
        <attvalues>
          <attvalue for="5" value=" \cite{DBLP:conf/icassp/WengMKWZMNPWBT20} presents a contextual language correction on ASR outputs jointly with modelling LU task that learns from ASR n-best transcriptions." />
        </attvalues>
      </edge>
      <edge source="2102.00804" target="1802.08395" id="7384">
        <attvalues>
          <attvalue for="5" value=" \cite{DBLP:conf/icassp/SerdyukWFKLB18} explore the possibility to extend the end-to-end ASR learning to include NLU component and optimize the whole system for SLU task while Ghannay et al." />
        </attvalues>
      </edge>
      <edge source="2406.10267" target="1904.09751" id="7385">
        <attvalues>
          <attvalue for="5" value=" Research \cite{Holtzman} shows that human generated text often does not correspond to modelled highest probability." />
        </attvalues>
      </edge>
      <edge source="2405.00888" target="2307.09288" id="7386">
        <attvalues>
          <attvalue for="5" value=" The increasing number of open-source LLMs, including Pythia~\cite{biderman2023pythia} and LLaMA-2~\cite{touvron2023llama2}, democratizes research in natural language processing (NLP)." />
        </attvalues>
      </edge>
      <edge source="2405.00888" target="2304.01373" id="7387">
        <attvalues>
          <attvalue for="5" value=" The increasing number of open-source LLMs, including Pythia~\cite{biderman2023pythia} and LLaMA-2~\cite{touvron2023llama2}, democratizes research in natural language processing (NLP)." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2205.05638" id="7390">
        <attvalues>
          <attvalue for="5" value=" In-context learning imposes substantial memory and computational overhead during inference as all the training examples have to be processed for each sample \cite{liu2022few}." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2101.00190" id="7391">
        <attvalues>
          <attvalue for="5" value=" Among the PET methods, soft prompts \cite{li-liang-2021-prefix,lester2021power} prepend trainable parameters to the input of the layers." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2104.08691" id="7392">
        <attvalues>
          <attvalue for="5" value=" Among the PET methods, soft prompts \cite{li-liang-2021-prefix,lester2021power} prepend trainable parameters to the input of the layers." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="1902.00751" id="7393">
        <attvalues>
          <attvalue for="5" value=" In another category of the PET methods, adapter modules are inserted \cite{pmlr-v97-houlsby19a,karimi2021compacter,he2022towards} into the Transformer." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2106.04647" id="7394">
        <attvalues>
          <attvalue for="5" value=" In another category of the PET methods, adapter modules are inserted \cite{pmlr-v97-houlsby19a,karimi2021compacter,he2022towards} into the Transformer." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2110.04366" id="7395">
        <attvalues>
          <attvalue for="5" value=" In another category of the PET methods, adapter modules are inserted \cite{pmlr-v97-houlsby19a,karimi2021compacter,he2022towards} into the Transformer." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2106.09685" id="7396">
        <attvalues>
          <attvalue for="5" value=" Therefore, Low Rank Adaption (LoRA) \cite{hu2021lora} was developed, which also uses extra low-rank modules as the trainable parameters." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="1910.02558" id="7397">
        <attvalues>
          <attvalue for="5" value=" This powerful decomposition method, when used for model compression, has proven to outperform low-rank factorization methods \cite{thakker2019pushing,hameed2021convolutional}." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2109.14710" id="7398">
        <attvalues>
          <attvalue for="5" value=" This powerful decomposition method, when used for model compression, has proven to outperform low-rank factorization methods \cite{thakker2019pushing,hameed2021convolutional}." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2109.06243" id="7399">
        <attvalues>
          <attvalue for="5" value=" It has also been successfully used for the compression of Transformer-based language models \cite{tahaei2021kroneckerbert,edalati2021kronecker}." />
        </attvalues>
      </edge>
      <edge source="2212.10650" target="2110.08152" id="7400">
        <attvalues>
          <attvalue for="5" value=" It has also been successfully used for the compression of Transformer-based language models \cite{tahaei2021kroneckerbert,edalati2021kronecker}." />
        </attvalues>
      </edge>
      <edge source="2005.00187" target="1602.02410" id="7402">
        <attvalues>
          <attvalue for="5" value=" This has been shown to be the case for both recurrent neural networks \cite{rnn_lm,lstm_lm,jozefowicz2016exploring} and non-recurrent attention-based models \cite{bert,gpt2}." />
        </attvalues>
      </edge>
      <edge source="2005.00187" target="1611.01368" id="7404">
        <attvalues>
          <attvalue for="5" value="} To gain a better understanding of these models' successes and failures, in particular in the domain of syntax, proposals have been made for testing the models on subsets of the test corpus where successful word prediction crucially depends on a correct analysis of the structure of the sentence \cite{dupouxlinzen16}." />
        </attvalues>
      </edge>
      <edge source="2005.00187" target="1803.11138" id="7405">
        <attvalues>
          <attvalue for="5" value=" Second, we hypothesize that language models would be better able to learn hierarchical syntactic generalizations in morphologically complex languages (which provide frequent overt cues to syntactic structure) than in morphologically simpler languages \cite{gulordavacolorless18,Lorimor2008,mccoy18pos}." />
        </attvalues>
      </edge>
      <edge source="2005.00187" target="1808.09031" id="7406">
        <attvalues>
          <attvalue for="5" value=" In this paper, we introduce the Cross-Linguistic Assessment of Models on Syntax (CLAMS) data set, which extends the subject-verb agreement component of the \cite{marvinlinzen18} challenge set to French, German, Hebrew and Russian." />
        </attvalues>
      </edge>
      <edge source="2004.03720" target="1706.03762" id="7421">
        <attvalues>
          <attvalue for="5" value=" Large transformers \cite{vaswani2017attention} pretrained with variants of a language modeling objective, such as BERT \cite{devlin2019bert}, have proven their effectiveness at flexibly transferring to a variety of domains and tasks." />
        </attvalues>
      </edge>
      <edge source="2004.03720" target="1508.07909" id="7423">
        <attvalues>
          <attvalue for="5" value=" Subword tokenization, popularized in the neural machine translation literature \cite{sennrich-etal-2016-neural, vaswani2017attention, wu2016google}, produces tokens at multiple levels of granularity, from individual characters to full words." />
        </attvalues>
      </edge>
      <edge source="2004.03720" target="1907.11692" id="7425">
        <attvalues>
          <attvalue for="5" value=" To understand whether this more natural tokenization leads to improved performance, we pretrain separate language models using the RoBERTa objective \cite{liu2019roberta} with each tokenization for both English and Japanese, two typologically distant languages." />
        </attvalues>
      </edge>
      <edge source="2004.03720" target="1909.11942" id="7427">
        <attvalues>
          <attvalue for="5" value=" BERT uses the WordPiece method \cite{Schuster2012JapaneseAK}, a language-modeling based variant of BPE; T5 \cite{raffel2019exploring} uses character-level BPE; GPT2 \cite{radford2019language} and RoBERTa \cite{liu2019roberta} use BPE over raw bytes instead of unicode characters; XLNet \cite{yang2019xlnet} and ALBERT \cite{lan2019albert} use the SentencePiece library \cite{kudo-richardson-2018-sentencepiece} which implements both BPE and unigram language model tokenization, but in both cases fail to clarify which of these methods they chose." />
        </attvalues>
      </edge>
      <edge source="2004.03720" target="1808.06226" id="7428">
        <attvalues>
          <attvalue for="5" value=" BERT uses the WordPiece method \cite{Schuster2012JapaneseAK}, a language-modeling based variant of BPE; T5 \cite{raffel2019exploring} uses character-level BPE; GPT2 \cite{radford2019language} and RoBERTa \cite{liu2019roberta} use BPE over raw bytes instead of unicode characters; XLNet \cite{yang2019xlnet} and ALBERT \cite{lan2019albert} use the SentencePiece library \cite{kudo-richardson-2018-sentencepiece} which implements both BPE and unigram language model tokenization, but in both cases fail to clarify which of these methods they chose." />
        </attvalues>
      </edge>
      <edge source="2004.03720" target="1804.10959" id="7429">
        <attvalues>
          <attvalue for="5" value=" In the machine translation literature, \cite{kudo-2018-subword} introduced the unigram language model tokenization method in the context of machine translation and found it comparable in performance to BPE." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2204.14198" id="7430">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) can do reasoning over diverse input data modalities [\cite{alayrac2022flamingo, driess2023palm, brohan2023rt, moon2023anymal}], besides the natural language domain." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2111.06377" id="7431">
        <attvalues>
          <attvalue for="5" value=" Such multimodal reasoning relies on pre-trained encoders for different input modalities, such as RGB frames [\cite{he2022masked}], LiDAR point clouds [\cite{hess2023masked}] and acoustic signals [\cite{li2023ti}], to extract task-relevant features." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2207.00531" id="7432">
        <attvalues>
          <attvalue for="5" value=" Such multimodal reasoning relies on pre-trained encoders for different input modalities, such as RGB frames [\cite{he2022masked}], LiDAR point clouds [\cite{hess2023masked}] and acoustic signals [\cite{li2023ti}], to extract task-relevant features." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="1706.03762" id="7435">
        <attvalues>
          <attvalue for="5" value=" Since today's transformer-based encoders [\cite{vaswani2017attention,wu2021autoformer}] are usually large in size, incorporating all modalities to LLMs is computationally expensive or even infeasible, especially on resource-constrained edge devices used in embodied AI applications." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2106.13008" id="7436">
        <attvalues>
          <attvalue for="5" value=" Since today's transformer-based encoders [\cite{vaswani2017attention,wu2021autoformer}] are usually large in size, incorporating all modalities to LLMs is computationally expensive or even infeasible, especially on resource-constrained edge devices used in embodied AI applications." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2307.15818" id="7437">
        <attvalues>
          <attvalue for="5" value=" An intuitive approach to such modality adaptation is to jointly train the encoders of all involved modalities with LLM to align every modality with the natural language domain [\cite{brohan2023rt, driess2023palm, wu2023next}], but is too expensive for runtime." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2301.12597" id="7438">
        <attvalues>
          <attvalue for="5" value=" As shown in Figure \ref{fig:snapshot} - Top, existing work connects encoders to LLM's input layer through a trainable projector [\cite{li2023blip, zhu2023minigpt}], and then applies parameter-efficient LLM fine-tuning [\cite{hu2021lora, sung2022vl, liang2022modular}] to improve accuracy." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2106.09685" id="7439">
        <attvalues>
          <attvalue for="5" value=" As shown in Figure \ref{fig:snapshot} - Top, existing work connects encoders to LLM's input layer through a trainable projector [\cite{li2023blip, zhu2023minigpt}], and then applies parameter-efficient LLM fine-tuning [\cite{hu2021lora, sung2022vl, liang2022modular}] to improve accuracy." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2303.11403" id="7440">
        <attvalues>
          <attvalue for="5" value=" Although some recent work [\cite{shukor2023ep}] explored connections to LLM's intermediate layers, such connections are arbitrarily decided and always fixed at runtime, and hence lack runtime adaptability." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2204.02311" id="7441">
        <attvalues>
          <attvalue for="5" value=" Our design focuses on decoder-only LLM, which is the dominant LLM architecture due to stronger generative power [\cite{chowdhery2022palm}] and has been widely adopted by most existing LLMs, ranging from BLOOM-1." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2211.01786" id="7442">
        <attvalues>
          <attvalue for="5" value=" We implemented and evaluated mPnP-LLM with two open-sourced LLMs, namely OPT [\cite{zhang2022opt}] and BLOOMZ [\cite{muennighoff2022crosslingual}], on the nuScene-QA dataset [\cite{qian2023nuscenes}] for multimodal QA task in autonomous driving." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2205.01068" id="7443">
        <attvalues>
          <attvalue for="5" value=" We implemented and evaluated mPnP-LLM with two open-sourced LLMs, namely OPT [\cite{zhang2022opt}] and BLOOMZ [\cite{muennighoff2022crosslingual}], on the nuScene-QA dataset [\cite{qian2023nuscenes}] for multimodal QA task in autonomous driving." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2005.14165" id="7444">
        <attvalues>
          <attvalue for="5" value="3B [\cite{zhang2022opt}] to GPT3-175B [\cite{brown2020language}]." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="1707.00683" id="7445">
        <attvalues>
          <attvalue for="5" value=" Being different from the existing plug-and-play approaches [\cite{de2017modulating, tan2019lxmert, brohan2022rt, shukor2023ep}] that require re-implementation of LLM's source codes, mPnP-LLM inserts the projected multimodal tokens as new key-value pairs into the multi-head attention (MHA) module of LLM block." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="1910.03771" id="7446">
        <attvalues>
          <attvalue for="5" value=" Since such interfacing is well supported in popular LLM frameworks (eg, HuggingFace Transformers [\cite{wolf2019huggingface}]), we can avoid any manual programming and reconfiguration efforts at runtime." />
        </attvalues>
      </edge>
      <edge source="2312.07886" target="2305.14836" id="7447">
        <attvalues>
          <attvalue for="5" value=" We implemented and evaluated mPnP-LLM with two open-sourced LLMs, namely OPT [\cite{zhang2022opt}] and BLOOMZ [\cite{muennighoff2022crosslingual}], on the nuScene-QA dataset [\cite{qian2023nuscenes}] for multimodal QA task in autonomous driving." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="2011.10798" id="7448">
        <attvalues>
          <attvalue for="5" value=" Since they are much smaller than conventional models, and their inference speed is often much faster \cite{bo21system,Ryan19,sainath2021cascadedlm,sainath2020streaming}, they work well for various streaming applications." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="1811.06621" id="7449">
        <attvalues>
          <attvalue for="5" value=" Since they are much smaller than conventional models, and their inference speed is often much faster \cite{bo21system,Ryan19,sainath2021cascadedlm,sainath2020streaming}, they work well for various streaming applications." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="1712.01769" id="7450">
        <attvalues>
          <attvalue for="5" value=" E2E speech recognition models, which combine acoustic, pronunciation and language models from conventional systems \cite{Golan16} into one neural network, have become widely used, especially for on-device applications \cite{bo21system,Ryan19,CC18,KimHoriWatanabe17,JinyuLi2019,Zeyer2020}." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="1909.12415" id="7452">
        <attvalues>
          <attvalue for="5" value=" E2E speech recognition models, which combine acoustic, pronunciation and language models from conventional systems \cite{Golan16} into one neural network, have become widely used, especially for on-device applications \cite{bo21system,Ryan19,CC18,KimHoriWatanabe17,JinyuLi2019,Zeyer2020}." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="2005.09319" id="7453">
        <attvalues>
          <attvalue for="5" value=" E2E speech recognition models, which combine acoustic, pronunciation and language models from conventional systems \cite{Golan16} into one neural network, have become widely used, especially for on-device applications \cite{bo21system,Ryan19,CC18,KimHoriWatanabe17,JinyuLi2019,Zeyer2020}." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="2003.12710" id="7454">
        <attvalues>
          <attvalue for="5" value=" Since they are much smaller than conventional models, and their inference speed is often much faster \cite{bo21system,Ryan19,sainath2021cascadedlm,sainath2020streaming}, they work well for various streaming applications." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="1911.09762" id="7455">
        <attvalues>
          <attvalue for="5" value=" Speech encoders that have been trained on high-resource ASR data can serve as foundation models for other tasks like sentiment analysis \cite{lu2020speech} or low-resource translation \cite{bansal2018pre}, to name a few." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="1911.03782" id="7457">
        <attvalues>
          <attvalue for="5" value=" Unlike the original work \cite{dalmia2019enforcing, dalmia2022legonn}, our encoder layers attention have limited left and right context windows, and the produced Lego-Features are successfully paired with a streaming-friendly RNN-T decoder." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="2209.15430" id="7459">
        <attvalues>
          <attvalue for="5" value=" In \cite{moschella2022relative}, this is achieved by learning representations relative to data-dependent anchors." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="2004.03898" id="7460">
        <attvalues>
          <attvalue for="5" value=" Another general approach, presented in \cite{gygli2021towards}, uses self-supervised objectives designed to encourage compatibility of different layer outputs." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="2010.14233" id="7461">
        <attvalues>
          <attvalue for="5" value=" Further, some research has already experimented with deliberation on top of CTC outputs to save the cost of first-pass decoding \cite{chi2020align,wang2022deliberation,wang2022streaming}." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="2112.11442" id="7462">
        <attvalues>
          <attvalue for="5" value=" Further, some research has already experimented with deliberation on top of CTC outputs to save the cost of first-pass decoding \cite{chi2020align,wang2022deliberation,wang2022streaming}." />
        </attvalues>
      </edge>
      <edge source="2304.00173" target="2204.07556" id="7463">
        <attvalues>
          <attvalue for="5" value=" Further, some research has already experimented with deliberation on top of CTC outputs to save the cost of first-pass decoding \cite{chi2020align,wang2022deliberation,wang2022streaming}." />
        </attvalues>
      </edge>
      <edge source="2110.04541" target="2005.14165" id="7469">
        <attvalues>
          <attvalue for="5" value=" &#09;\ifdefined\SQUEEZE Beyond excelling in their core task of pure language modeling, modern Neural Language Models (NLMs) show impressive zero- and few-shot abilities in more general Natural Language Understanding (NLU) tasks~\cite{GPT3}." />
        </attvalues>
      </edge>
      <edge source="2110.04541" target="1605.06743" id="7470">
        <attvalues>
          <attvalue for="5" value=" The separation rank &#09;was employed for analyzing the dependencies modeled by convolutional~\cite{cohen2017inductive}, recurrent~\cite{levine2018benefits}, and self-attention ~\cite{levine2020limits} networks with respect to a single input example." />
        </attvalues>
      </edge>
      <edge source="2110.04541" target="2006.12467" id="7471">
        <attvalues>
          <attvalue for="5" value=" The separation rank &#09;was employed for analyzing the dependencies modeled by convolutional~\cite{cohen2017inductive}, recurrent~\cite{levine2018benefits}, and self-attention ~\cite{levine2020limits} networks with respect to a single input example." />
        </attvalues>
      </edge>
      <edge source="2108.11696" target="1802.05365" id="7472">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models~\cite{peters2018deep, radford2018improving, devlin2019bert, liu2019roberta} have contributed to great progress in natural language understanding (NLU)." />
        </attvalues>
      </edge>
      <edge source="2108.11696" target="1811.01088" id="7473">
        <attvalues>
          <attvalue for="5" value=" STILT~\cite{phang2018sentence, wang2019can, clark2019boolq, pruksachatkun2020intermediate, phang2020english, vu2020exploring} can further improve their performance on downstream NLU tasks by redesigning the training pipeline, introducing an intermediate-task fine-tuning phase before fine-tuning the pretrained models on the target task of interest (Figure~\ref{fig:STILT})." />
        </attvalues>
      </edge>
      <edge source="2108.11696" target="1907.11692" id="7475">
        <attvalues>
          <attvalue for="5" value=" To study when and why STILT works, \cite{pruksachatkun2020intermediate} conduct large-scale experiments based on RoBERTa-large~\cite{liu2019roberta} with different intermediate-target task pairs." />
        </attvalues>
      </edge>
      <edge source="2108.11696" target="1909.00277" id="7476">
        <attvalues>
          <attvalue for="5" value=" They show the difficulty to have a generally useful intermediate task and conclude that those containing complex reasoning and inference, such as CosmosQA~\cite{huang2019cosmos} and HellaSwag~\cite{zellers2019hellaswag}, tend to enhance various target tasks." />
        </attvalues>
      </edge>
      <edge source="2108.11696" target="1905.07830" id="7477">
        <attvalues>
          <attvalue for="5" value=" They show the difficulty to have a generally useful intermediate task and conclude that those containing complex reasoning and inference, such as CosmosQA~\cite{huang2019cosmos} and HellaSwag~\cite{zellers2019hellaswag}, tend to enhance various target tasks." />
        </attvalues>
      </edge>
      <edge source="2108.11696" target="2004.14074" id="7478">
        <attvalues>
          <attvalue for="5" value=" However, this ignores the fact that HellaSwag is a synthetic dataset, and RoBERTa tends to capture the data artifacts when fine-tuned on HellaSwag~\cite{tamborrino2020pre}." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1902.03249" id="7479">
        <attvalues>
          <attvalue for="5" value=" Insertion-based models, for example, the Insertion Transformer \cite{stern-icml-2019} is one potential solution." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1902.02192" id="7480">
        <attvalues>
          <attvalue for="5" value=" Recently, insertion-based models \cite{stern-icml-2019,welleck-icml-2019,gu-arxiv-2019,chan-arxiv-2019} have been introduced for text generation." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1902.01370" id="7481">
        <attvalues>
          <attvalue for="5" value=" Recently, insertion-based models \cite{stern-icml-2019,welleck-icml-2019,gu-arxiv-2019,chan-arxiv-2019} have been introduced for text generation." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1906.01604" id="7482">
        <attvalues>
          <attvalue for="5" value=" An autoregressive left-to-right model would require $O(n)$ generation steps to generate $n$ tokens, whereas the Insertion Transformer \cite{stern-icml-2019} and KERMIT \cite{chan-arxiv-2019} following a balanced binary tree policy requires only $O(\log_2 n)$ generation steps to generate $n$ tokens." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1406.1078" id="7483">
        <attvalues>
          <attvalue for="5" value=" Unlike traditional autoregressive left-to-right models \cite{cho-emnlp-2014,sutskever-nips-2014,vaswani-nips-2017}, insertion-based models are not restricted to generating text sequences in a serial left-to-right manner, but these models are endowed with the capabilities of parallel generation." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1409.3215" id="7484">
        <attvalues>
          <attvalue for="5" value=" Unlike traditional autoregressive left-to-right models \cite{cho-emnlp-2014,sutskever-nips-2014,vaswani-nips-2017}, insertion-based models are not restricted to generating text sequences in a serial left-to-right manner, but these models are endowed with the capabilities of parallel generation." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1706.03762" id="7485">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Target contextualization is especially difficult in an autoregressive left-to-right model (eg, Transformer \cite{vaswani-nips-2017}), the model must generate the whole document in linear fashion, which would be prohibitively expensive costing $O(n)$ iterations to generate $n$ tokens." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1803.05567" id="7486">
        <attvalues>
          <attvalue for="5" value=" Recent research suggests we are nearing human-level parity for sentence-level translation in certain domains \cite{hassan-arxiv-2018}, however, we lag significantly behind in document-level translation \cite{laubli-emnlp-2018}." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1808.07048" id="7487">
        <attvalues>
          <attvalue for="5" value=" Recent research suggests we are nearing human-level parity for sentence-level translation in certain domains \cite{hassan-arxiv-2018}, however, we lag significantly behind in document-level translation \cite{laubli-emnlp-2018}." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1907.06170" id="7488">
        <attvalues>
          <attvalue for="5" value=" Various papers have proposed incorporating context for document-level translation \cite{dowmunt-wmt-2019}, which has been shown to improve translation quality." />
        </attvalues>
      </edge>
      <edge source="1910.13034" target="1711.03688" id="7489">
        <attvalues>
          <attvalue for="5" value=" Some prior work have focused on utilizing block coordinate descent like algorithms during inference \cite{maruf-acl-2018}, however this adds complexity and additional runtime cost during inference." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="1907.11692" id="7491">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="2006.03654" id="7492">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="1907.10529" id="7493">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="1904.09223" id="7494">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="2005.14165" id="7495">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="2204.07832" id="7498">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="2201.04831" id="7499">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="2010.02693" id="7500">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="2205.10714" id="7501">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLMs)~\cite{devlin-etal-2019-bert,liu2019roberta,deberta,joshi2020spanbert,sun2019ernie,GPT3,t5} are widely used in the community of natural language processing and have achieved remarkable success in numerous downstream tasks of natural language understanding (NLU), such as sentiment analysis~\cite{liu2021unified,wang2022contrastive,zhong2022knowledge}, intent detection~\cite{kim2016intent,wu2020slotrefine} and reasoning~\cite{qu2022interpretable}." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="2109.01819" id="7503">
        <attvalues>
          <attvalue for="5" value=" The denoising objective~\cite{yamaguchi2021frustratingly} aims to improve the data efficiency and save training costs, while the contrastive objective involves leveraging contrastive learning~\cite{gao2021simcse} to learn better sentence representations." />
        </attvalues>
      </edge>
      <edge source="2302.09268" target="2104.08821" id="7504">
        <attvalues>
          <attvalue for="5" value=" The denoising objective~\cite{yamaguchi2021frustratingly} aims to improve the data efficiency and save training costs, while the contrastive objective involves leveraging contrastive learning~\cite{gao2021simcse} to learn better sentence representations." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2303.08774" id="7505">
        <attvalues>
          <attvalue for="5" value=" Using large language models (LLMs)~\cite{GPT4,llama2} as evaluators to assign scores to the given inputs have become prevalent." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2307.09288" id="7506">
        <attvalues>
          <attvalue for="5" value=" Using large language models (LLMs)~\cite{GPT4,llama2} as evaluators to assign scores to the given inputs have become prevalent." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2304.03442" id="7507">
        <attvalues>
          <attvalue for="5" value=" Similarly, \cite{generative_agent} assign poignancy score to the generated text for the retrieval task." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2304.02554" id="7508">
        <attvalues>
          <attvalue for="5" value=" Other research explores using LLMs to assess generated texts, finding the LLM scores correlates higher with human evaluators than existing automatic metrics \cite{gao2023human,shen2023large,liu-etal-2023-g,luo2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2305.13091" id="7509">
        <attvalues>
          <attvalue for="5" value=" Other research explores using LLMs to assess generated texts, finding the LLM scores correlates higher with human evaluators than existing automatic metrics \cite{gao2023human,shen2023large,liu-etal-2023-g,luo2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2303.16634" id="7510">
        <attvalues>
          <attvalue for="5" value=" Other research explores using LLMs to assess generated texts, finding the LLM scores correlates higher with human evaluators than existing automatic metrics \cite{gao2023human,shen2023large,liu-etal-2023-g,luo2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2311.01967" id="7511">
        <attvalues>
          <attvalue for="5" value=" Even slight linguistic variations can lead to significant fluctuations in task performance \cite{leidinger2023linguistic}." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2309.03409" id="7513">
        <attvalues>
          <attvalue for="5" value=" While prompt optimization techniques \cite{chen2023instructzero,LLMoptim,zhang2023tempera,prasad2023grips} have been developed to assist in designing more effective prompts, these methods require paired input-output samples for objective value calculation." />
        </attvalues>
      </edge>
      <edge source="2406.09972" target="2203.07281" id="7514">
        <attvalues>
          <attvalue for="5" value=" While prompt optimization techniques \cite{chen2023instructzero,LLMoptim,zhang2023tempera,prasad2023grips} have been developed to assist in designing more effective prompts, these methods require paired input-output samples for objective value calculation." />
        </attvalues>
      </edge>
      <edge source="2406.02449" target="1706.03762" id="7525">
        <attvalues>
          <attvalue for="5" value=" Do the representations learned by a transformer model \cite{vaswani_attention_2017} exhibit similar system level-structures? To answer this we look at the representations that emerge over the course of training as a kind of language in their own right." />
        </attvalues>
      </edge>
      <edge source="2406.02449" target="1503.02406" id="7526">
        <attvalues>
          <attvalue for="5" value=" This is in part because these models are black-boxes \cite{tishby_deep_2015, shwartz-ziv_opening_2017}." />
        </attvalues>
      </edge>
      <edge source="2406.02449" target="2003.12298" id="7527">
        <attvalues>
          <attvalue for="5" value=" While there is a growing body of work on interpretability, offering techniques for predicting what is encoded in a model's representations \cite{voita_information-theoretic_2020, pimentel2020information}, there's still lack of clarity about how representations themselves are structured, how that structure emerges, and what kinds of structures are desirable." />
        </attvalues>
      </edge>
      <edge source="2406.02449" target="1406.1078" id="7528">
        <attvalues>
          <attvalue for="5" value=" A multi-layered neural model needs to learn to map a sentence to a vector representation that later layers can successfully map to the output; encoder-decoder models \cite{cho2014learning} even more explicitly use separate parts of a model to map in and out of vector space." />
        </attvalues>
      </edge>
      <edge source="2406.02449" target="1206.5538" id="7529">
        <attvalues>
          <attvalue for="5" value=" While there has long been interest in the kinds of representations learned by deep-models \cite{bengio2013representation, locatello2019challenging}, there has been little work quantifying systematic structure in the representations learned by transformers or relating them to the kinds of structures that characterise natural language." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1705.02073" id="7530">
        <attvalues>
          <attvalue for="5" value=" Various methods have been proposed to build cross-lingual classification models by exploiting machine translation systems~\cite{cldc,chen2018adversarial,xnli}, and learning multilingual embeddings~\cite{xnli,yu2018multilingual,clse-artetxet,eisenschlos2019multifit}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1606.01614" id="7531">
        <attvalues>
          <attvalue for="5" value=" Various methods have been proposed to build cross-lingual classification models by exploiting machine translation systems~\cite{cldc,chen2018adversarial,xnli}, and learning multilingual embeddings~\cite{xnli,yu2018multilingual,clse-artetxet,eisenschlos2019multifit}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1812.10464" id="7533">
        <attvalues>
          <attvalue for="5" value=" Various methods have been proposed to build cross-lingual classification models by exploiting machine translation systems~\cite{cldc,chen2018adversarial,xnli}, and learning multilingual embeddings~\cite{xnli,yu2018multilingual,clse-artetxet,eisenschlos2019multifit}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1909.04761" id="7534">
        <attvalues>
          <attvalue for="5" value=" Various methods have been proposed to build cross-lingual classification models by exploiting machine translation systems~\cite{cldc,chen2018adversarial,xnli}, and learning multilingual embeddings~\cite{xnli,yu2018multilingual,clse-artetxet,eisenschlos2019multifit}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1901.07291" id="7536">
        <attvalues>
          <attvalue for="5" value=" Recently, multilingual pretrained language models have shown surprising cross-lingual effectiveness on a wide range of downstream tasks~\cite{bert,xlm,xnlg,xlmr}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1909.10481" id="7537">
        <attvalues>
          <attvalue for="5" value=" Recently, multilingual pretrained language models have shown surprising cross-lingual effectiveness on a wide range of downstream tasks~\cite{bert,xlm,xnlg,xlmr}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1911.02116" id="7538">
        <attvalues>
          <attvalue for="5" value=" The curse of multilinguality results in that the multilingual models usually perform worse than their monolingual competitors on downstream tasks~\cite{nmt:wild19,xlmr}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1906.01502" id="7539">
        <attvalues>
          <attvalue for="5" value=" Even without using any parallel corpora, the pretrained models can still perform zero-shot cross-lingual classification~\cite{pires2019multilingual,wu2019beto,keung2019adversarial}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1904.09077" id="7540">
        <attvalues>
          <attvalue for="5" value=" Even without using any parallel corpora, the pretrained models can still perform zero-shot cross-lingual classification~\cite{pires2019multilingual,wu2019beto,keung2019adversarial}." />
        </attvalues>
      </edge>
      <edge source="1911.03913" target="1909.00153" id="7541">
        <attvalues>
          <attvalue for="5" value=" Even without using any parallel corpora, the pretrained models can still perform zero-shot cross-lingual classification~\cite{pires2019multilingual,wu2019beto,keung2019adversarial}." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="1706.03762" id="7543">
        <attvalues>
          <attvalue for="5" value=" However, with the rise of bidirectional LSTMs \cite{hochreiter1997long} and Transformers \cite{vaswani2017attention}, recent research has focused on non-incremental solutions." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="2010.05330" id="7544">
        <attvalues>
          <attvalue for="5" value=" Therefore, this approach does not capture the progressive unfolding of input over time, giving the sense that all of it is available all of a sudden \cite{madureira-schlangen-2020-incremental}." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="1805.12518" id="7545">
        <attvalues>
          <attvalue for="5" value=" While some studies have addressed the challenge of outputting incremental structured representations - for various definitions of incrementality \cite{konstas-etal-2014-incremental,kohn-2018-incremental,shen-etal-2021-explicitly} - analyses of trees remain limited, more notably since the popularization of deep learning, and are mostly partially incremental approaches." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="2011.07960" id="7546">
        <attvalues>
          <attvalue for="5" value=" While some studies have addressed the challenge of outputting incremental structured representations - for various definitions of incrementality \cite{konstas-etal-2014-incremental,kohn-2018-incremental,shen-etal-2021-explicitly} - analyses of trees remain limited, more notably since the popularization of deep learning, and are mostly partially incremental approaches." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="1606.06406" id="7547">
        <attvalues>
          <attvalue for="5" value=" For shift-reduce constituent parsing, \cite{cross-huang-2016-incremental} proposed an incremental model with minimal features, focusing on only three sentence positions to predict the next action." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="1805.01052" id="7548">
        <attvalues>
          <attvalue for="5" value=" Despite this, it relied on bidirectional Transformers and a CYK architecture \cite{kitaev2018constituency} for decoding these vectors into trees." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="2010.14568" id="7549">
        <attvalues>
          <attvalue for="5" value=" For the decoder, we reassess two options that generate partial trees based solely on current inputs: (i) an incremental parsing-as-tagging model \cite{gomez-rodriguez-vilares-2018-constituent}, and (ii) a transition-based decoder that uses graph-neural-network representations \cite{yang2020strongly}." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="1802.05365" id="7552">
        <attvalues>
          <attvalue for="5" value=" \cite{stanojevic-steedman-2019-ccg} developed an almost fully incremental parser for combinatory categorical grammars (CCG), relying on ELMo embeddings \cite{peters-etal-2018-deep} and a bidirectional LSTM for these predictions." />
        </attvalues>
      </edge>
      <edge source="2402.02782" target="1810.08994" id="7554">
        <attvalues>
          <attvalue for="5" value=" For the decoder, we reassess two options that generate partial trees based solely on current inputs: (i) an incremental parsing-as-tagging model \cite{gomez-rodriguez-vilares-2018-constituent}, and (ii) a transition-based decoder that uses graph-neural-network representations \cite{yang2020strongly}." />
        </attvalues>
      </edge>
      <edge source="2403.08833" target="1505.00468" id="7555">
        <attvalues>
          <attvalue for="5" value=" The advancement of computer vision and natural language processing has facilitated research in vision-language fusion, such as Visual Question Answering\cite{antol2015vqa} (VQA), Image Caption\cite{xu2015show}, and Vision-Language Navigation\cite{anderson2018vision} (VLN)." />
        </attvalues>
      </edge>
      <edge source="2403.08833" target="1502.03044" id="7556">
        <attvalues>
          <attvalue for="5" value=" The advancement of computer vision and natural language processing has facilitated research in vision-language fusion, such as Visual Question Answering\cite{antol2015vqa} (VQA), Image Caption\cite{xu2015show}, and Vision-Language Navigation\cite{anderson2018vision} (VLN)." />
        </attvalues>
      </edge>
      <edge source="2403.08833" target="2201.11903" id="7558">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) have recently received considerable attention due to their remarkable language generation capabilities and extensive knowledge\cite{touvron2302llama,wei2022chain}." />
        </attvalues>
      </edge>
      <edge source="2403.08833" target="2302.09230" id="7560">
        <attvalues>
          <attvalue for="5" value=" Recent research has revealed that, despite being trained, current VLN models still encounter difficulties when dealing with diverse instructions\cite{zhang2023vln}." />
        </attvalues>
      </edge>
      <edge source="2403.08833" target="2306.16410" id="7562">
        <attvalues>
          <attvalue for="5" value=" However, a challenge arises in developing LLM-based VLN systems due to LLMs' limited visual perceptual capabilities\cite{berrios2023towards}, primarily trained on textual data." />
        </attvalues>
      </edge>
      <edge source="2403.08833" target="2305.16986" id="7563">
        <attvalues>
          <attvalue for="5" value=" Some efforts introduced visual perceptual modules, converting visual information into textual descriptions for analysis by LLMs\cite{liu2023internchat, zhu2023minigpt,zhou2023navgpt}." />
        </attvalues>
      </edge>
      <edge source="1811.04791" target="1712.04313" id="7564">
        <attvalues>
          <attvalue for="5" value=" One challenge for these systems, highlighted by the zrsc shared tasks of 2015 \cite{Versteegh2015} and 2017 \cite{Dunbar2017}, is to improve subword modeling, eg, to extract or learn speech features from the target language audio." />
        </attvalues>
      </edge>
      <edge source="1811.04791" target="1803.08863" id="7565">
        <attvalues>
          <attvalue for="5" value=" The results above were presented as part of an earlier conference version of this paper \cite{Hermann2018}." />
        </attvalues>
      </edge>
      <edge source="1811.04791" target="1606.06950" id="7566">
        <attvalues>
          <attvalue for="5" value=" Finally, we provide both a qualitative analysis of the differences between the different features we extract, and a quantitative evaluation on the downstream target-language task of unsupervised full-coverage speech segmentation and clustering using the system of \cite{Kamper2017}." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1811.05082" id="7568">
        <attvalues>
          <attvalue for="5" value=" Concretely, inspired by the investigation of E2E-ABSA in~\cite{li2019unified}, which predicts aspect boundaries as well as aspect sentiments using a single sequence tagger, we build a series of simple yet insightful neural baselines for the sequence labeling problem and fine-tune the task-specific components with BERT or deem BERT as feature extractor." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1909.07593" id="7569">
        <attvalues>
          <attvalue for="5" value=" Compared to this classification problem, the second one and the third one, namely, Aspect-oriented Opinion Words Extraction (AOWE)~\cite{fan-etal-2019-target} and End-to-End Aspect-based Sentiment Analysis (E2E-ABSA)~\cite{ma-etal-2018-joint,schmitt-etal-2018-joint,li2019unified,li2017learning,li2019learning}, are related to a sequence tagging problem." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1605.08900" id="7570">
        <attvalues>
          <attvalue for="5" value=" Many neural models composed of a task-agnostic pre-trained word embedding layer and task-specific neural architecture have been proposed for the original ABSA task (eg the aspect-level sentiment classification)~\cite{tang-etal-2016-aspect,wang-etal-2016-attention,chen-etal-2017-recurrent-attention,liu-zhang-2017-attention,ma2017interactive,ma2018targeted,majumder-etal-2018-iarm,li-etal-2018-transformation,he-etal-2018-exploiting,xue-li-2018-aspect,wang-etal-2018-target,fan-etal-2018-multi,huang-carley-2018-parameterized,lei2019human,li2019exploiting,zhang2019aspect}, but the improvement of these models measured by the accuracy or F1 score has reached a bottleneck." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1805.01086" id="7572">
        <attvalues>
          <attvalue for="5" value=" Many neural models composed of a task-agnostic pre-trained word embedding layer and task-specific neural architecture have been proposed for the original ABSA task (eg the aspect-level sentiment classification)~\cite{tang-etal-2016-aspect,wang-etal-2016-attention,chen-etal-2017-recurrent-attention,liu-zhang-2017-attention,ma2017interactive,ma2018targeted,majumder-etal-2018-iarm,li-etal-2018-transformation,he-etal-2018-exploiting,xue-li-2018-aspect,wang-etal-2018-target,fan-etal-2018-multi,huang-carley-2018-parameterized,lei2019human,li2019exploiting,zhang2019aspect}, but the improvement of these models measured by the accuracy or F1 score has reached a bottleneck." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1806.04346" id="7573">
        <attvalues>
          <attvalue for="5" value=" Many neural models composed of a task-agnostic pre-trained word embedding layer and task-specific neural architecture have been proposed for the original ABSA task (eg the aspect-level sentiment classification)~\cite{tang-etal-2016-aspect,wang-etal-2016-attention,chen-etal-2017-recurrent-attention,liu-zhang-2017-attention,ma2017interactive,ma2018targeted,majumder-etal-2018-iarm,li-etal-2018-transformation,he-etal-2018-exploiting,xue-li-2018-aspect,wang-etal-2018-target,fan-etal-2018-multi,huang-carley-2018-parameterized,lei2019human,li2019exploiting,zhang2019aspect}, but the improvement of these models measured by the accuracy or F1 score has reached a bottleneck." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1805.07043" id="7574">
        <attvalues>
          <attvalue for="5" value=" Many neural models composed of a task-agnostic pre-trained word embedding layer and task-specific neural architecture have been proposed for the original ABSA task (eg the aspect-level sentiment classification)~\cite{tang-etal-2016-aspect,wang-etal-2016-attention,chen-etal-2017-recurrent-attention,liu-zhang-2017-attention,ma2017interactive,ma2018targeted,majumder-etal-2018-iarm,li-etal-2018-transformation,he-etal-2018-exploiting,xue-li-2018-aspect,wang-etal-2018-target,fan-etal-2018-multi,huang-carley-2018-parameterized,lei2019human,li2019exploiting,zhang2019aspect}, but the improvement of these models measured by the accuracy or F1 score has reached a bottleneck." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1909.06276" id="7575">
        <attvalues>
          <attvalue for="5" value=" Many neural models composed of a task-agnostic pre-trained word embedding layer and task-specific neural architecture have been proposed for the original ABSA task (eg the aspect-level sentiment classification)~\cite{tang-etal-2016-aspect,wang-etal-2016-attention,chen-etal-2017-recurrent-attention,liu-zhang-2017-attention,ma2017interactive,ma2018targeted,majumder-etal-2018-iarm,li-etal-2018-transformation,he-etal-2018-exploiting,xue-li-2018-aspect,wang-etal-2018-target,fan-etal-2018-multi,huang-carley-2018-parameterized,lei2019human,li2019exploiting,zhang2019aspect}, but the improvement of these models measured by the accuracy or F1 score has reached a bottleneck." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1811.10999" id="7576">
        <attvalues>
          <attvalue for="5" value=" Many neural models composed of a task-agnostic pre-trained word embedding layer and task-specific neural architecture have been proposed for the original ABSA task (eg the aspect-level sentiment classification)~\cite{tang-etal-2016-aspect,wang-etal-2016-attention,chen-etal-2017-recurrent-attention,liu-zhang-2017-attention,ma2017interactive,ma2018targeted,majumder-etal-2018-iarm,li-etal-2018-transformation,he-etal-2018-exploiting,xue-li-2018-aspect,wang-etal-2018-target,fan-etal-2018-multi,huang-carley-2018-parameterized,lei2019human,li2019exploiting,zhang2019aspect}, but the improvement of these models measured by the accuracy or F1 score has reached a bottleneck." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1310.4546" id="7578">
        <attvalues>
          <attvalue for="5" value=" One reason is that the task-agnostic embedding layer, usually a linear layer initialized with Word2Vec~\cite{mikolov2013distributed} or GloVe~\cite{pennington-etal-2014-glove}, only provides context-independent word-level features, which is insufficient for capturing the complex semantic dependencies in the sentence." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1708.00107" id="7579">
        <attvalues>
          <attvalue for="5" value=" Thus, introducing a context-aware word embedding layer pre-trained on large-scale datasets with deep LSTM~\cite{mccann2017learned,peters-etal-2018-deep,howard-ruder-2018-universal} or Transformer~\cite{radford2018improving,radford2019language,devlin-etal-2019-bert,lample2019cross,yang2019xlnet,dong2019unified} for fine-tuning a lightweight task-specific network using the labeled data has good potential for further enhancing the performance." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1802.05365" id="7580">
        <attvalues>
          <attvalue for="5" value=" Thus, introducing a context-aware word embedding layer pre-trained on large-scale datasets with deep LSTM~\cite{mccann2017learned,peters-etal-2018-deep,howard-ruder-2018-universal} or Transformer~\cite{radford2018improving,radford2019language,devlin-etal-2019-bert,lample2019cross,yang2019xlnet,dong2019unified} for fine-tuning a lightweight task-specific network using the labeled data has good potential for further enhancing the performance." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1801.06146" id="7581">
        <attvalues>
          <attvalue for="5" value=" Thus, introducing a context-aware word embedding layer pre-trained on large-scale datasets with deep LSTM~\cite{mccann2017learned,peters-etal-2018-deep,howard-ruder-2018-universal} or Transformer~\cite{radford2018improving,radford2019language,devlin-etal-2019-bert,lample2019cross,yang2019xlnet,dong2019unified} for fine-tuning a lightweight task-specific network using the labeled data has good potential for further enhancing the performance." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1901.07291" id="7583">
        <attvalues>
          <attvalue for="5" value=" Thus, introducing a context-aware word embedding layer pre-trained on large-scale datasets with deep LSTM~\cite{mccann2017learned,peters-etal-2018-deep,howard-ruder-2018-universal} or Transformer~\cite{radford2018improving,radford2019language,devlin-etal-2019-bert,lample2019cross,yang2019xlnet,dong2019unified} for fine-tuning a lightweight task-specific network using the labeled data has good potential for further enhancing the performance." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1905.03197" id="7585">
        <attvalues>
          <attvalue for="5" value=" Thus, introducing a context-aware word embedding layer pre-trained on large-scale datasets with deep LSTM~\cite{mccann2017learned,peters-etal-2018-deep,howard-ruder-2018-universal} or Transformer~\cite{radford2018improving,radford2019language,devlin-etal-2019-bert,lample2019cross,yang2019xlnet,dong2019unified} for fine-tuning a lightweight task-specific network using the labeled data has good potential for further enhancing the performance." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1904.02232" id="7586">
        <attvalues>
          <attvalue for="5" value=" \cite{xu-etal-2019-bert,sun-etal-2019-utilizing,song2019attentional,yu2019adapting,rietzler2019adapt,huang2019syntax,hu2019learning} have conducted some initial attempts to couple the deep contextualized word embedding layer with downstream neural models for the original ABSA task and establish the new state-of-the-art results." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1903.09588" id="7587">
        <attvalues>
          <attvalue for="5" value=" \cite{xu-etal-2019-bert,sun-etal-2019-utilizing,song2019attentional,yu2019adapting,rietzler2019adapt,huang2019syntax,hu2019learning} have conducted some initial attempts to couple the deep contextualized word embedding layer with downstream neural models for the original ABSA task and establish the new state-of-the-art results." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1902.09314" id="7588">
        <attvalues>
          <attvalue for="5" value=" \cite{xu-etal-2019-bert,sun-etal-2019-utilizing,song2019attentional,yu2019adapting,rietzler2019adapt,huang2019syntax,hu2019learning} have conducted some initial attempts to couple the deep contextualized word embedding layer with downstream neural models for the original ABSA task and establish the new state-of-the-art results." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1908.11860" id="7589">
        <attvalues>
          <attvalue for="5" value=" \cite{xu-etal-2019-bert,sun-etal-2019-utilizing,song2019attentional,yu2019adapting,rietzler2019adapt,huang2019syntax,hu2019learning} have conducted some initial attempts to couple the deep contextualized word embedding layer with downstream neural models for the original ABSA task and establish the new state-of-the-art results." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1909.02606" id="7590">
        <attvalues>
          <attvalue for="5" value=" \cite{xu-etal-2019-bert,sun-etal-2019-utilizing,song2019attentional,yu2019adapting,rietzler2019adapt,huang2019syntax,hu2019learning} have conducted some initial attempts to couple the deep contextualized word embedding layer with downstream neural models for the original ABSA task and establish the new state-of-the-art results." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1909.11297" id="7591">
        <attvalues>
          <attvalue for="5" value=" \cite{xu-etal-2019-bert,sun-etal-2019-utilizing,song2019attentional,yu2019adapting,rietzler2019adapt,huang2019syntax,hu2019learning} have conducted some initial attempts to couple the deep contextualized word embedding layer with downstream neural models for the original ABSA task and establish the new state-of-the-art results." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1706.03762" id="7592">
        <attvalues>
          <attvalue for="5" value=" In this paper, we investigate the modeling power of BERT~\cite{devlin-etal-2019-bert}, one of the most popular pre-trained language model armed with Transformer~\cite{vaswani2017attention}, on the task of E2E-ABSA." />
        </attvalues>
      </edge>
      <edge source="1910.00883" target="1712.05403" id="7593">
        <attvalues>
          <attvalue for="5" value=" Besides, we standardize the comparative study by consistently utilizing the hold-out development dataset for model selection, which is ignored in most of the existing ABSA works~\cite{tay2018learning}." />
        </attvalues>
      </edge>
      <edge source="2011.09567" target="1903.08855" id="7595">
        <attvalues>
          <attvalue for="5" value=" \begin{example} cubra de nieve la hermosa cumbre\\ cu-bra-de-nie-ve-la-her-mo-sa-cum-bre \\ $+--+---+-+-$ 11 \\ (Garcilaso de la Vega) \\ \end{example} Research has shown that neural models implicitly encode linguistic features ranging from token labeling to different kinds of segmentation \cite{liu2019linguistic}." />
        </attvalues>
      </edge>
      <edge source="2011.09567" target="1805.01070" id="7596">
        <attvalues>
          <attvalue for="5" value=" There is also evidence that language models and embeddings are able to capture not only semantic and syntactic properties but structural information, as shown by \cite{hewitt2019structural} in their work with structural probes for extracting syntax trees, and \cite{conneau2018you} approximating the length in words of a sentence by its vector." />
        </attvalues>
      </edge>
      <edge source="2011.09567" target="1711.00938" id="7597">
        <attvalues>
          <attvalue for="5" value=" The only neural approach was explored by \cite{agirrezabal2016zeuscansion,agirrezabal2017comparison}, who used bi-LSTM neural networks and CRF's to automatically scan poetry in three languages (eg English, Spanish and Basque)." />
        </attvalues>
      </edge>
      <edge source="2408.03070" target="1907.13528" id="7598">
        <attvalues>
          <attvalue for="5" value=" \cite{ettinger_what_2020,kassner_negated_2020} use factual statements such as \Next, and report that models \draftreplace{fail {to reflect the presence of negation in the probability distribution." />
        </attvalues>
      </edge>
      <edge source="2408.03070" target="1911.03343" id="7599">
        <attvalues>
          <attvalue for="5" value=" \cite{ettinger_what_2020,kassner_negated_2020} use factual statements such as \Next, and report that models \draftreplace{fail {to reflect the presence of negation in the probability distribution." />
        </attvalues>
      </edge>
      <edge source="2408.03070" target="2005.01810" id="7600">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{klafka_spying_2020} study how input embeddings encode animacy, gender, and number of surrounding words in a specific SVO context." />
        </attvalues>
      </edge>
      <edge source="2408.03070" target="1808.10627" id="7601">
        <attvalues>
          <attvalue for="5" value=" Note that our methodology differs from that of \cite{jumelet_language_2018}, who, given an input token, predict the zone this token belongs to." />
        </attvalues>
      </edge>
      <edge source="2408.03070" target="1909.02597" id="7602">
        <attvalues>
          <attvalue for="5" value=" \cite{jumelet_language_2018} have shown that LSTM embeddings do encode the notion of licensing scope (given an input embedding, a classifier can predict the structural zone the input token belongs to), a finding later confirmed for transformer-based PLMs \cite{warstadt_investigating_2019}." />
        </attvalues>
      </edge>
      <edge source="2401.06643" target="1904.03122" id="7605">
        <attvalues>
          <attvalue for="5" value=" The aim of such data collection is to increase the data diversity and subsequent performance of classifiers trained on the data~\cite{Larson2019, larson-etal-2020-iterative}." />
        </attvalues>
      </edge>
      <edge source="2401.06643" target="2305.12947" id="7607">
        <attvalues>
          <attvalue for="5" value=" The baseline, taken from a previous study~\cite{cegin-etal-2023-chatgpt}, is a simple prompting for paraphrases." />
        </attvalues>
      </edge>
      <edge source="2010.01063" target="1901.09069" id="7609">
        <attvalues>
          <attvalue for="5" value=" Almeida and Xex\'eo surveyed different types of static word embeddings \cite{almeida2019word}, and Liu et al." />
        </attvalues>
      </edge>
      <edge source="2010.01063" target="2003.07278" id="7610">
        <attvalues>
          <attvalue for="5" value=" \cite{liu2020contextual} focused on contextual representations found in the most recent neural models." />
        </attvalues>
      </edge>
      <edge source="2010.01063" target="1801.07772" id="7611">
        <attvalues>
          <attvalue for="5" value=" Belinkov and Glass \cite{belinkov2017evaluating} surveyed the strategies of interpreting latent representation." />
        </attvalues>
      </edge>
      <edge source="2311.03881" target="2104.08821" id="7612">
        <attvalues>
          <attvalue for="5" value=" Unsupervised SimCSE (unsup-SimCSE) is a notable framework for contrastive sentence embeddings~\cite{DBLP:conf/emnlp/GaoYC21}." />
        </attvalues>
      </edge>
      <edge source="2311.03881" target="2204.00408" id="7613">
        <attvalues>
          <attvalue for="5" value=" Inspired by recent research on parameter sparsification~\cite{DBLP:conf/acl/XiaZC22, prasanna-etal-2020-bert,DBLP:conf/nips/HouHSJCL20,DBLP:conf/nips/MichelLN19}, particularly the works on the lottery ticket hypothesis (LTH)~\cite{DBLP:conf/iclr/FrankleC19,DBLP:conf/iclr/BaiWTL022,DBLP:conf/icml/FrankleD0C20,DBLP:conf/nlpcc/YangZWS22} showing its effectiveness in improving model performance through pruning, we hypothesize that certain parameters in SimCSE might hinder the representation of universal sentence embeddings." />
        </attvalues>
      </edge>
      <edge source="2311.03881" target="1905.10650" id="7615">
        <attvalues>
          <attvalue for="5" value=" Inspired by recent research on parameter sparsification~\cite{DBLP:conf/acl/XiaZC22, prasanna-etal-2020-bert,DBLP:conf/nips/HouHSJCL20,DBLP:conf/nips/MichelLN19}, particularly the works on the lottery ticket hypothesis (LTH)~\cite{DBLP:conf/iclr/FrankleC19,DBLP:conf/iclr/BaiWTL022,DBLP:conf/icml/FrankleD0C20,DBLP:conf/nlpcc/YangZWS22} showing its effectiveness in improving model performance through pruning, we hypothesize that certain parameters in SimCSE might hinder the representation of universal sentence embeddings." />
        </attvalues>
      </edge>
      <edge source="2311.03881" target="2203.04248" id="7617">
        <attvalues>
          <attvalue for="5" value=" Inspired by recent research on parameter sparsification~\cite{DBLP:conf/acl/XiaZC22, prasanna-etal-2020-bert,DBLP:conf/nips/HouHSJCL20,DBLP:conf/nips/MichelLN19}, particularly the works on the lottery ticket hypothesis (LTH)~\cite{DBLP:conf/iclr/FrankleC19,DBLP:conf/iclr/BaiWTL022,DBLP:conf/icml/FrankleD0C20,DBLP:conf/nlpcc/YangZWS22} showing its effectiveness in improving model performance through pruning, we hypothesize that certain parameters in SimCSE might hinder the representation of universal sentence embeddings." />
        </attvalues>
      </edge>
      <edge source="2311.03881" target="1912.05671" id="7618">
        <attvalues>
          <attvalue for="5" value=" Inspired by recent research on parameter sparsification~\cite{DBLP:conf/acl/XiaZC22, prasanna-etal-2020-bert,DBLP:conf/nips/HouHSJCL20,DBLP:conf/nips/MichelLN19}, particularly the works on the lottery ticket hypothesis (LTH)~\cite{DBLP:conf/iclr/FrankleC19,DBLP:conf/iclr/BaiWTL022,DBLP:conf/icml/FrankleD0C20,DBLP:conf/nlpcc/YangZWS22} showing its effectiveness in improving model performance through pruning, we hypothesize that certain parameters in SimCSE might hinder the representation of universal sentence embeddings." />
        </attvalues>
      </edge>
      <edge source="2311.03881" target="2207.09638" id="7619">
        <attvalues>
          <attvalue for="5" value=" Inspired by recent research on parameter sparsification~\cite{DBLP:conf/acl/XiaZC22, prasanna-etal-2020-bert,DBLP:conf/nips/HouHSJCL20,DBLP:conf/nips/MichelLN19}, particularly the works on the lottery ticket hypothesis (LTH)~\cite{DBLP:conf/iclr/FrankleC19,DBLP:conf/iclr/BaiWTL022,DBLP:conf/icml/FrankleD0C20,DBLP:conf/nlpcc/YangZWS22} showing its effectiveness in improving model performance through pruning, we hypothesize that certain parameters in SimCSE might hinder the representation of universal sentence embeddings." />
        </attvalues>
      </edge>
      <edge source="2311.03881" target="2005.10242" id="7620">
        <attvalues>
          <attvalue for="5" value=" In the literature ~\cite{DBLP:conf/icml/0001I20}, two such properties have been proposed: alignment and uniformity." />
        </attvalues>
      </edge>
      <edge source="2210.05892" target="1601.04811" id="7621">
        <attvalues>
          <attvalue for="5" value=" Examples include machine translation~\cite{DBLP:conf/acl/TuLLLL16,DBLP:journals/taslp/ZhangLSZX021}, question answering~\cite{DBLP:conf/emnlp/DuanTCZ17}, and generation-based dialog system~\cite{DBLP:conf/acl/TuLC0W022}." />
        </attvalues>
      </edge>
      <edge source="2210.05892" target="1911.11520" id="7622">
        <attvalues>
          <attvalue for="5" value=" Examples include machine translation~\cite{DBLP:conf/acl/TuLLLL16,DBLP:journals/taslp/ZhangLSZX021}, question answering~\cite{DBLP:conf/emnlp/DuanTCZ17}, and generation-based dialog system~\cite{DBLP:conf/acl/TuLC0W022}." />
        </attvalues>
      </edge>
      <edge source="2210.05892" target="2203.13560" id="7623">
        <attvalues>
          <attvalue for="5" value=" Examples include machine translation~\cite{DBLP:conf/acl/TuLLLL16,DBLP:journals/taslp/ZhangLSZX021}, question answering~\cite{DBLP:conf/emnlp/DuanTCZ17}, and generation-based dialog system~\cite{DBLP:conf/acl/TuLC0W022}." />
        </attvalues>
      </edge>
      <edge source="2210.05892" target="2008.12009" id="7624">
        <attvalues>
          <attvalue for="5" value=" ROUGE and its variants, also evaluating text based on $n$-grams, are recall-based measures~\cite{DBLP:journals/csur/SaiMK23}." />
        </attvalues>
      </edge>
      <edge source="2210.05892" target="2106.00085" id="7625">
        <attvalues>
          <attvalue for="5" value=" On the other hand, the generated texts to be evaluated may have different lengths~\cite{meister-cotterell-2021-language}." />
        </attvalues>
      </edge>
      <edge source="2002.08131" target="1802.05365" id="7628">
        <attvalues>
          <attvalue for="5" value=" Recently, two divergent research directions have shown promise for document-classification: on the one hand, transfer learning \cite{peters-etal-2018-deep,howard-ruder-2018-universal,devlin-etal-2019-bert} and on the other hand hierarchical modeling \cite{Xia:Cho:16,Con:Sch:Bar:17,yang-etal-2016-hierarchical}." />
        </attvalues>
      </edge>
      <edge source="2002.08131" target="1801.06146" id="7629">
        <attvalues>
          <attvalue for="5" value=" Recently, two divergent research directions have shown promise for document-classification: on the one hand, transfer learning \cite{peters-etal-2018-deep,howard-ruder-2018-universal,devlin-etal-2019-bert} and on the other hand hierarchical modeling \cite{Xia:Cho:16,Con:Sch:Bar:17,yang-etal-2016-hierarchical}." />
        </attvalues>
      </edge>
      <edge source="2002.08131" target="1602.00367" id="7631">
        <attvalues>
          <attvalue for="5" value=" Recently, two divergent research directions have shown promise for document-classification: on the one hand, transfer learning \cite{peters-etal-2018-deep,howard-ruder-2018-universal,devlin-etal-2019-bert} and on the other hand hierarchical modeling \cite{Xia:Cho:16,Con:Sch:Bar:17,yang-etal-2016-hierarchical}." />
        </attvalues>
      </edge>
      <edge source="2002.08131" target="1606.01781" id="7632">
        <attvalues>
          <attvalue for="5" value=" Recently, two divergent research directions have shown promise for document-classification: on the one hand, transfer learning \cite{peters-etal-2018-deep,howard-ruder-2018-universal,devlin-etal-2019-bert} and on the other hand hierarchical modeling \cite{Xia:Cho:16,Con:Sch:Bar:17,yang-etal-2016-hierarchical}." />
        </attvalues>
      </edge>
      <edge source="2010.08067" target="1808.06232" id="7633">
        <attvalues>
          <attvalue for="5" value=" This situation has changed with the advent of lexicon-scale acceptability and inference judgment datasets, such as the \href{ datasets \cite{white_computational_2016,white_role_2018,white_frequency_accepted,white_lexicosyntactic_2018,an_lexical_2020,moon_source_toappear}." />
        </attvalues>
      </edge>
      <edge source="2010.08067" target="1908.05253" id="7634">
        <attvalues>
          <attvalue for="5" value=" This situation has changed with the advent of lexicon-scale acceptability and inference judgment datasets, such as the \href{ datasets \cite{white_computational_2016,white_role_2018,white_frequency_accepted,white_lexicosyntactic_2018,an_lexical_2020,moon_source_toappear}." />
        </attvalues>
      </edge>
      <edge source="2010.08067" target="1503.02510" id="7635">
        <attvalues>
          <attvalue for="5" value=" On the one hand, powerful methods for learning structured representations from corpus data and (to some extent) behavioral data now exist \cite{le_inside-outside_2014,le_compositional_2015,williams_latent_2018,shen_neural_2018,kim_unsupervised_2019,drozdov_unsupervised_labeled_2019,drozdov_unsupervised_latent_2019}, but the relationship between these models' representations and grammars posited under standard frameworks (\cite{montague_proper_1973} et seq) assuming some form of (combinatory) categorial grammar \cite{steedman_syntactic_2000} remains unclear." />
        </attvalues>
      </edge>
      <edge source="2010.08067" target="1709.01121" id="7636">
        <attvalues>
          <attvalue for="5" value=" On the one hand, powerful methods for learning structured representations from corpus data and (to some extent) behavioral data now exist \cite{le_inside-outside_2014,le_compositional_2015,williams_latent_2018,shen_neural_2018,kim_unsupervised_2019,drozdov_unsupervised_labeled_2019,drozdov_unsupervised_latent_2019}, but the relationship between these models' representations and grammars posited under standard frameworks (\cite{montague_proper_1973} et seq) assuming some form of (combinatory) categorial grammar \cite{steedman_syntactic_2000} remains unclear." />
        </attvalues>
      </edge>
      <edge source="2010.08067" target="1711.02013" id="7637">
        <attvalues>
          <attvalue for="5" value=" On the one hand, powerful methods for learning structured representations from corpus data and (to some extent) behavioral data now exist \cite{le_inside-outside_2014,le_compositional_2015,williams_latent_2018,shen_neural_2018,kim_unsupervised_2019,drozdov_unsupervised_labeled_2019,drozdov_unsupervised_latent_2019}, but the relationship between these models' representations and grammars posited under standard frameworks (\cite{montague_proper_1973} et seq) assuming some form of (combinatory) categorial grammar \cite{steedman_syntactic_2000} remains unclear." />
        </attvalues>
      </edge>
      <edge source="2010.08067" target="1904.03746" id="7638">
        <attvalues>
          <attvalue for="5" value=" On the one hand, powerful methods for learning structured representations from corpus data and (to some extent) behavioral data now exist \cite{le_inside-outside_2014,le_compositional_2015,williams_latent_2018,shen_neural_2018,kim_unsupervised_2019,drozdov_unsupervised_labeled_2019,drozdov_unsupervised_latent_2019}, but the relationship between these models' representations and grammars posited under standard frameworks (\cite{montague_proper_1973} et seq) assuming some form of (combinatory) categorial grammar \cite{steedman_syntactic_2000} remains unclear." />
        </attvalues>
      </edge>
      <edge source="2010.08067" target="1207.1420" id="7639">
        <attvalues>
          <attvalue for="5" value=" On the other hand, powerful methods for inducing such grammars now exist \cite{zettlemoyer_learning_2005,zettlemoyer_online_2007,zettlemoyer_learning_2009,kwiatkowksi_inducing_2010,kwiatkowski_lexical_2011,bisk_induction_2012,bisk_simple_2012,bisk_hdp_2013,bisk_probing_2015}, but they do not straightforwardly generalize to the full range of behavioral data of interest in experimental semantics." />
        </attvalues>
      </edge>
      <edge source="1908.11047" target="1802.05365" id="7640">
        <attvalues>
          <attvalue for="5" value=" The NLP community is revisiting the role of linguistic structure in applications with the advent of contextual word representations (\cwrs) derived from pretraining language models on large corpora~\cite{Peters:18,Radford:18,Howard:18,Devlin:18}." />
        </attvalues>
      </edge>
      <edge source="1908.11047" target="1801.06146" id="7641">
        <attvalues>
          <attvalue for="5" value=" The NLP community is revisiting the role of linguistic structure in applications with the advent of contextual word representations (\cwrs) derived from pretraining language models on large corpora~\cite{Peters:18,Radford:18,Howard:18,Devlin:18}." />
        </attvalues>
      </edge>
      <edge source="1908.11047" target="1804.08199" id="7643">
        <attvalues>
          <attvalue for="5" value=" Recent work has shown that downstream task performance may benefit from explicitly injecting a syntactic inductive bias into model architectures \cite{Kuncoro:18}, even when \cwrs are also used \cite{Strubell:18}." />
        </attvalues>
      </edge>
      <edge source="1908.11047" target="1805.04218" id="7645">
        <attvalues>
          <attvalue for="5" value=" Recent work has probed the knowledge encoded in \cwrs and found they capture a surprisingly large amount of syntax~\cite{Blevins:18,Liu:19,Tenney:18}." />
        </attvalues>
      </edge>
      <edge source="1908.11047" target="1903.08855" id="7646">
        <attvalues>
          <attvalue for="5" value=" We further examine the contextual embeddings obtained from the enhanced architecture and a shallow syntactic context, using black-box probes from \cite{Liu:19}." />
        </attvalues>
      </edge>
      <edge source="1908.11047" target="1905.06316" id="7647">
        <attvalues>
          <attvalue for="5" value=" Recent work has probed the knowledge encoded in \cwrs and found they capture a surprisingly large amount of syntax~\cite{Blevins:18,Liu:19,Tenney:18}." />
        </attvalues>
      </edge>
      <edge source="2205.12452" target="2108.07258" id="7648">
        <attvalues>
          <attvalue for="5" value=" Foundational Models \cite{Bommasani2021OnTO} based on the Transformer architecture \cite{Vaswani2017AttentionIA} has quickly become the most common building block in the modern language understanding stack, providing robust language representations which can be leveraged to provide impressive accuracy on tasks like question answering, text classification, and token classification." />
        </attvalues>
      </edge>
      <edge source="2205.12452" target="1706.03762" id="7649">
        <attvalues>
          <attvalue for="5" value=" Foundational Models \cite{Bommasani2021OnTO} based on the Transformer architecture \cite{Vaswani2017AttentionIA} has quickly become the most common building block in the modern language understanding stack, providing robust language representations which can be leveraged to provide impressive accuracy on tasks like question answering, text classification, and token classification." />
        </attvalues>
      </edge>
      <edge source="2205.12452" target="2010.02559" id="7651">
        <attvalues>
          <attvalue for="5" value=" These Large Language Models (LLMs) are able to adapt to novel domains through pretraining resulting in models like BioBERT \cite{Lee2020BioBERTAP}, LegalBERT \cite{Chalkidis2020LEGALBERTTM}, and SciBERT \cite{beltagy2019SciBERTAP} have become a popular strategy for improving performance further." />
        </attvalues>
      </edge>
      <edge source="2205.12452" target="2012.11881" id="7653">
        <attvalues>
          <attvalue for="5" value=" Several approaches have been successfully used to improve the performance of these LLMs, such as approximating attention \cite{Peng2021RandomFA}, removing portions of the models \cite{Sridhar2020UndividedAA}, and reducing the precision of activation and weight values." />
        </attvalues>
      </edge>
      <edge source="2205.12452" target="2111.05754" id="7654">
        <attvalues>
          <attvalue for="5" value="\\ Recent work \cite{Zafrir2021PruneOF} \cite{Kurti2022TheOB} has shown that the application of unstructured and semi-structured (block) pruning mechanisms on LLMs can significantly compress models with little to no loss in accuracy." />
        </attvalues>
      </edge>
      <edge source="2205.12452" target="2203.07259" id="7655">
        <attvalues>
          <attvalue for="5" value="\\ Recent work \cite{Zafrir2021PruneOF} \cite{Kurti2022TheOB} has shown that the application of unstructured and semi-structured (block) pruning mechanisms on LLMs can significantly compress models with little to no loss in accuracy." />
        </attvalues>
      </edge>
      <edge source="2205.12452" target="1909.10351" id="7657">
        <attvalues>
          <attvalue for="5" value=" Compressed models like DistillBERT \cite{Sanh2019DistilBERTAD} and TinyBERT \cite{Jiao2020TinyBERTDB} are some of the most popular LLMs because they provide compression without any additional know-how or optimization." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="1909.08383" id="7658">
        <attvalues>
          <attvalue for="5" value=" These CL methods can be grouped into three categories \cite{defy}: (i) regularization-based methods add a term to the loss to regularize training, eg \cite{ewc, mas, imm}; (ii) rehearsal-based methods rehearse previous tasks through a small memory of samples from previous tasks, eg \cite{agem, er}; (iii) architectural-based methods increase the model capacity when learning new tasks, eg \cite{pnn}." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="1612.00796" id="7659">
        <attvalues>
          <attvalue for="5" value=" These CL methods can be grouped into three categories \cite{defy}: (i) regularization-based methods add a term to the loss to regularize training, eg \cite{ewc, mas, imm}; (ii) rehearsal-based methods rehearse previous tasks through a small memory of samples from previous tasks, eg \cite{agem, er}; (iii) architectural-based methods increase the model capacity when learning new tasks, eg \cite{pnn}." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="1812.00420" id="7660">
        <attvalues>
          <attvalue for="5" value=" These CL methods can be grouped into three categories \cite{defy}: (i) regularization-based methods add a term to the loss to regularize training, eg \cite{ewc, mas, imm}; (ii) rehearsal-based methods rehearse previous tasks through a small memory of samples from previous tasks, eg \cite{agem, er}; (iii) architectural-based methods increase the model capacity when learning new tasks, eg \cite{pnn}." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="1606.04671" id="7661">
        <attvalues>
          <attvalue for="5" value=" These CL methods can be grouped into three categories \cite{defy}: (i) regularization-based methods add a term to the loss to regularize training, eg \cite{ewc, mas, imm}; (ii) rehearsal-based methods rehearse previous tasks through a small memory of samples from previous tasks, eg \cite{agem, er}; (iii) architectural-based methods increase the model capacity when learning new tasks, eg \cite{pnn}." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="2104.01616" id="7662">
        <attvalues>
          <attvalue for="5" value=" Overall, our method remains very simple, while, compared to the best methods from \cite{lifelongasr, eeckt2021continual}, not requiring a memory." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="2112.09427" id="7663">
        <attvalues>
          <attvalue for="5" value=" \cite{lifelongasr} and \cite{eeckt2021continual} implement, respectively, four and nine CL methods for (End-to-End) E2E ASR, both finding that rehearsal-based methods remain the most practical way to overcome CF." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="2207.05071" id="7664">
        <attvalues>
          <attvalue for="5" value=" \cite{ogem} applies Gradient Episodic Memory \cite{gem}, also a rehearsal-based method, to E2E ASR, while assuming that task boundaries are not known." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="1706.08840" id="7665">
        <attvalues>
          <attvalue for="5" value=" \cite{ogem} applies Gradient Episodic Memory \cite{gem}, also a rehearsal-based method, to E2E ASR, while assuming that task boundaries are not known." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="2203.16082" id="7666">
        <attvalues>
          <attvalue for="5" value=" \cite{eeckt_adapters} finds that using rehearsal-based methods is not necessary to prevent CF when task-specific adapters \cite{adapters} are used, which comes, however, at the cost of introducing task-specific parameters, possibly requiring a task label at inference time." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="1902.00751" id="7667">
        <attvalues>
          <attvalue for="5" value=" \cite{eeckt_adapters} finds that using rehearsal-based methods is not necessary to prevent CF when task-specific adapters \cite{adapters} are used, which comes, however, at the cost of introducing task-specific parameters, possibly requiring a task label at inference time." />
        </attvalues>
      </edge>
      <edge source="2210.15282" target="1606.09282" id="7668">
        <attvalues>
          <attvalue for="5" value=" Though already very effective by itself, weight averaging can be further improved by introducing knowledge distillation losses (as in \cite{lwf}) during the adaptation." />
        </attvalues>
      </edge>
      <edge source="2402.18031" target="2212.10496" id="7669">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{hyde} demonstrates the effectiveness of utilizing LLMs to generate hypothetical documents answering the original query as additional texts to augment the query." />
        </attvalues>
      </edge>
      <edge source="2402.18031" target="2304.13157" id="7670">
        <attvalues>
          <attvalue for="5" value=" \cite{generative_relevance_feedback} show the efficacy of applying pseudo-relevance feedback upon the LLM-generated answers for expansion." />
        </attvalues>
      </edge>
      <edge source="2402.18031" target="2309.01219" id="7671">
        <attvalues>
          <attvalue for="5" value=" These include hallucination~\cite{hallucination_1}, inability to update~\cite{timeqa}, and a deficiency in long-tail knowledge~\cite{long-tail}." />
        </attvalues>
      </edge>
      <edge source="2402.18031" target="2207.13332" id="7672">
        <attvalues>
          <attvalue for="5" value=" These include hallucination~\cite{hallucination_1}, inability to update~\cite{timeqa}, and a deficiency in long-tail knowledge~\cite{long-tail}." />
        </attvalues>
      </edge>
      <edge source="2402.18031" target="2211.08411" id="7673">
        <attvalues>
          <attvalue for="5" value=" These include hallucination~\cite{hallucination_1}, inability to update~\cite{timeqa}, and a deficiency in long-tail knowledge~\cite{long-tail}." />
        </attvalues>
      </edge>
      <edge source="2402.18031" target="2309.08541" id="7674">
        <attvalues>
          <attvalue for="5" value=" Such generations may introduce irrelevant or misleading texts, degrading retrieval performance~\cite{expan_fail}." />
        </attvalues>
      </edge>
      <edge source="2402.18031" target="2304.09161" id="7675">
        <attvalues>
          <attvalue for="5" value=" Unlike methods that rely on the intrinsic parametric knowledge of LLMs, CSQE exclusively leverages the strong relevance assessing capability of LLMs~\cite{relevance_assess_1, relevance_assess_2}." />
        </attvalues>
      </edge>
      <edge source="2211.04023" target="2101.08091" id="7676">
        <attvalues>
          <attvalue for="5" value=" Spoken language understanding (SLU) is a crucial component in task-oriented dialogue systems \cite{weld2021survey}, which typically consists of two subtasks: intent detection (ID) and slot filling (SF)." />
        </attvalues>
      </edge>
      <edge source="2211.04023" target="2010.03880" id="7677">
        <attvalues>
          <attvalue for="5" value="\ref{fig:example}, the task of ID should classify both intent labels in the utterance (eg, AddToPlaylist and PlayMusic), while SF can be treated as a sequence labeling task to predict slot for each token in BIO format \cite{zhang2016joint,qin2021co}." />
        </attvalues>
      </edge>
      <edge source="2211.04023" target="2009.13431" id="7678">
        <attvalues>
          <attvalue for="5" value=" Since intents and slots are inextricably related \cite{zhou2021pin,huang-etal-2020-federated, huang2021sentiment}, researchers in recent years \cite{gangadharaiah2019joint, qin-etal-2020-agif, qin2021gl,cai2022slim} have increasingly focused on joint multiple intent detection and slot filling." />
        </attvalues>
      </edge>
      <edge source="2211.04023" target="2108.11711" id="7679">
        <attvalues>
          <attvalue for="5" value=" Since intents and slots are inextricably related \cite{zhou2021pin,huang-etal-2020-federated, huang2021sentiment}, researchers in recent years \cite{gangadharaiah2019joint, qin-etal-2020-agif, qin2021gl,cai2022slim} have increasingly focused on joint multiple intent detection and slot filling." />
        </attvalues>
      </edge>
      <edge source="2211.04023" target="2106.01925" id="7681">
        <attvalues>
          <attvalue for="5" value=" \cite{qin2021gl} explored a global-locally graph interaction network, which models slot dependency and intent-slot interaction for each utterance." />
        </attvalues>
      </edge>
      <edge source="2211.04023" target="1908.08676" id="7682">
        <attvalues>
          <attvalue for="5" value=" Concretely, inspired by the success of leveraging label characteristics to help model optimization \cite{wu-etal-2021-label,cui2019hierarchically}, we construct intent and slot spaces using words in each intent label and slot label respectively to inject label information into utterance representations adaptively." />
        </attvalues>
      </edge>
      <edge source="2306.13986" target="2005.14165" id="7683">
        <attvalues>
          <attvalue for="5" value="5~\cite{brown2020language}---one of the currently most powerful LLMs---can handle them." />
        </attvalues>
      </edge>
      <edge source="2010.13856" target="1906.11943" id="7684">
        <attvalues>
          <attvalue for="5" value=" Idiosyncrasies in the input text can trigger surprising model behaviors even for language pairs at the high end of the translation quality spectrum, \cite{DBLP:journals/corr/abs-1906-11943}." />
        </attvalues>
      </edge>
      <edge source="2010.13856" target="2005.03519" id="7685">
        <attvalues>
          <attvalue for="5" value=" Practical use cases of MT output are well aligned with CE models in that translation quality is viewed as a binary variable: correct or incorrect, as described in \cite{zhou2020practical}." />
        </attvalues>
      </edge>
      <edge source="2010.13856" target="1409.3215" id="7686">
        <attvalues>
          <attvalue for="5" value=" While our approach pertains to the more general body of work on ``uncertainty estimation&quot; for machine learning models, and in particular for seq2seq models~\cite{sutskever2014sequence}, we find little related work in the context of neural MT, as highlighted in~\ref{sec:related_work_ce}." />
        </attvalues>
      </edge>
      <edge source="2005.00278" target="1804.08313" id="7687">
        <attvalues>
          <attvalue for="5" value=" Semantic-role structures have been shown effective in many NLP tasks, including machine translation~\cite{marcheggiani2018exploiting}, question answering~\cite{shen2007using}, and summarization~\cite{khan2015framework}." />
        </attvalues>
      </edge>
      <edge source="2005.00278" target="1703.04826" id="7688">
        <attvalues>
          <attvalue for="5" value=" Most work on SRL relies on supervised learning~\cite{he2017deep,marcheggiani2017encoding}, and thus requires annotated resources such as PropBank~\cite{palmer2005proposition} and FrameNet~\cite{baker1998berkeley} for English, or SALSA~\cite{burchardt2006salsa} for German." />
        </attvalues>
      </edge>
      <edge source="2005.00278" target="1312.6114" id="7689">
        <attvalues>
          <attvalue for="5" value=" We approach the transfer problem from the generative perspective and use the variational autoencoding (VAE) framework~\cite{kingma2013auto} or, more specifically, its semi-supervised version~\cite{kingma2014semi}." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2107.03312" id="7692">
        <attvalues>
          <attvalue for="5" value=" AudioLM~\cite{audiolm} uses a hierarchical sequence-to-sequence approach and adopts w2v-BERT~\cite{w2vbert} and SoundStream~\cite{soundstream} to extract semantic and acoustic tokens respectively." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2210.13438" id="7693">
        <attvalues>
          <attvalue for="5" value=" VALL-E~\cite{valle}, a pioneering TTS framework, adopts RVQ-based audio codec Encodec~\cite{encodec} and utilizes a language model as a prompt-based language modeling task." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2106.07447" id="7694">
        <attvalues>
          <attvalue for="5" value=" This breakthrough, coupled with the use of neural audio codecs~\cite{soundstream,encodec,hubert} that convert continuous audio features into discrete tokens, has greatly propelled recent speech synthesis frameworks~\cite{megatts,megatts2}, such as VALL-E~\cite{valle}, AudioLM~\cite{audiolm}, NaturalSpeech2~\cite{naturespeech2}, and SPEAR-TTS~\cite{speartts}." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2306.03509" id="7695">
        <attvalues>
          <attvalue for="5" value=" This breakthrough, coupled with the use of neural audio codecs~\cite{soundstream,encodec,hubert} that convert continuous audio features into discrete tokens, has greatly propelled recent speech synthesis frameworks~\cite{megatts,megatts2}, such as VALL-E~\cite{valle}, AudioLM~\cite{audiolm}, NaturalSpeech2~\cite{naturespeech2}, and SPEAR-TTS~\cite{speartts}." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2301.02111" id="7696">
        <attvalues>
          <attvalue for="5" value=" VALL-E~\cite{valle}, a pioneering TTS framework, adopts RVQ-based audio codec Encodec~\cite{encodec} and utilizes a language model as a prompt-based language modeling task." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2209.03143" id="7697">
        <attvalues>
          <attvalue for="5" value=" AudioLM~\cite{audiolm} uses a hierarchical sequence-to-sequence approach and adopts w2v-BERT~\cite{w2vbert} and SoundStream~\cite{soundstream} to extract semantic and acoustic tokens respectively." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2304.09116" id="7698">
        <attvalues>
          <attvalue for="5" value=" This breakthrough, coupled with the use of neural audio codecs~\cite{soundstream,encodec,hubert} that convert continuous audio features into discrete tokens, has greatly propelled recent speech synthesis frameworks~\cite{megatts,megatts2}, such as VALL-E~\cite{valle}, AudioLM~\cite{audiolm}, NaturalSpeech2~\cite{naturespeech2}, and SPEAR-TTS~\cite{speartts}." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2302.03540" id="7699">
        <attvalues>
          <attvalue for="5" value=" SPEAR-TTS~\cite{speartts} has the same structure except for replacing the first stage with an encoder-decoder scheme." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2108.06209" id="7700">
        <attvalues>
          <attvalue for="5" value=" AudioLM~\cite{audiolm} uses a hierarchical sequence-to-sequence approach and adopts w2v-BERT~\cite{w2vbert} and SoundStream~\cite{soundstream} to extract semantic and acoustic tokens respectively." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2006.04558" id="7701">
        <attvalues>
          <attvalue for="5" value=" Compared with traditional TTS systems like FastSpeech2~\cite{fastspeech2} and Tacotron2~\cite{tacotron}, these recent models show great voice cloning ability by providing only a 3-second speech prompt and have natural prosody comparable with human speakers." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="1703.10135" id="7702">
        <attvalues>
          <attvalue for="5" value=" Compared with traditional TTS systems like FastSpeech2~\cite{fastspeech2} and Tacotron2~\cite{tacotron}, these recent models show great voice cloning ability by providing only a 3-second speech prompt and have natural prosody comparable with human speakers." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2201.11903" id="7703">
        <attvalues>
          <attvalue for="5" value=" Hence, the selection of speech prompts is critically important, akin to the significance of prompts in the LLM domain, where the quality and clarity of prompts significantly influence the outcomes~\cite{cot,zerocot}." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2209.06484" id="7705">
        <attvalues>
          <attvalue for="5" value=" Furthermore, in TTS scenarios that incorporate context information, such as audiobook TTS~\cite{xue2022paratts,chen2022unsupervised,chen2023stylespeech} and conversational TTS~\cite{m2ctts,deng2023cmcu,deng2023concss}, the choice of a speech prompt should also take contextual information into account." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2305.02269" id="7707">
        <attvalues>
          <attvalue for="5" value=" Furthermore, in TTS scenarios that incorporate context information, such as audiobook TTS~\cite{xue2022paratts,chen2022unsupervised,chen2023stylespeech} and conversational TTS~\cite{m2ctts,deng2023cmcu,deng2023concss}, the choice of a speech prompt should also take contextual information into account." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2312.10358" id="7708">
        <attvalues>
          <attvalue for="5" value=" Furthermore, in TTS scenarios that incorporate context information, such as audiobook TTS~\cite{xue2022paratts,chen2022unsupervised,chen2023stylespeech} and conversational TTS~\cite{m2ctts,deng2023cmcu,deng2023concss}, the choice of a speech prompt should also take contextual information into account." />
        </attvalues>
      </edge>
      <edge source="2406.03714" target="2005.11401" id="7709">
        <attvalues>
          <attvalue for="5" value=" In LLM area, RAG methods~\cite{rag4ki,ragmp} are recognized as a significant enhancement across a variety of tasks." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2005.14165" id="7714">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advancements in large language models (LLMs) have been remarkable~\cite{Brown2020LanguageMA,Zhao2023ASO}, with these models demonstrating an unparalleled ability to understand and generate text across a wide spectrum of tasks~\cite{Wei2022ChainOT,Kojima2022LargeLM}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2303.18223" id="7715">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advancements in large language models (LLMs) have been remarkable~\cite{Brown2020LanguageMA,Zhao2023ASO}, with these models demonstrating an unparalleled ability to understand and generate text across a wide spectrum of tasks~\cite{Wei2022ChainOT,Kojima2022LargeLM}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2303.05453" id="7717">
        <attvalues>
          <attvalue for="5" value=" This capability has revolutionized the way we interact with machine-generated content and opened up new avenues for personalized text generation~\cite{Kirk2023PersonalisationWB,Li2023TeachLT}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="1910.03506" id="7718">
        <attvalues>
          <attvalue for="5" value=" Personalization in text generation is of paramount importance to ensure user engagement and satisfaction~\cite{Huang2022UserNLP222I} across a range of in applications such as composing tweets, or generating news articles and financial reports, or in more personalized settings like business communications and creative writing~\cite{Li2019TowardsCA,Li2020KnowledgeEnhancedPR}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2010.01480" id="7719">
        <attvalues>
          <attvalue for="5" value=" Personalization in text generation is of paramount importance to ensure user engagement and satisfaction~\cite{Huang2022UserNLP222I} across a range of in applications such as composing tweets, or generating news articles and financial reports, or in more personalized settings like business communications and creative writing~\cite{Li2019TowardsCA,Li2020KnowledgeEnhancedPR}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2203.15556" id="7720">
        <attvalues>
          <attvalue for="5" value=" However, the prospect of developing a unique LLM for each user presents challenges, including the prohibitive resource requirements~\cite{Hoffmann2022TrainingCL}, data privacy concerns~\cite{Li2023MultistepJP}, and the scarcity of personalized data~\cite{Rafailov2023DirectPO}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2304.05197" id="7721">
        <attvalues>
          <attvalue for="5" value=" However, the prospect of developing a unique LLM for each user presents challenges, including the prohibitive resource requirements~\cite{Hoffmann2022TrainingCL}, data privacy concerns~\cite{Li2023MultistepJP}, and the scarcity of personalized data~\cite{Rafailov2023DirectPO}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2305.18290" id="7722">
        <attvalues>
          <attvalue for="5" value=" However, the prospect of developing a unique LLM for each user presents challenges, including the prohibitive resource requirements~\cite{Hoffmann2022TrainingCL}, data privacy concerns~\cite{Li2023MultistepJP}, and the scarcity of personalized data~\cite{Rafailov2023DirectPO}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2204.05862" id="7723">
        <attvalues>
          <attvalue for="5" value=" A promising solution lies in adopting lightweight models capable of online learning, which can dynamically adjust their output based on continuous user feedback~\cite{Bai2022TrainingAH}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2106.04887" id="7724">
        <attvalues>
          <attvalue for="5" value=" Importantly, this adaptive process is poised to unlock long-term rewards stemming from personalization, encompassing not just explicit preferences expressed by users but also responding to favorable actions~\cite{Xie2021InteractionGroundedL}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2211.01910" id="7725">
        <attvalues>
          <attvalue for="5" value=" Despite these benefits, the ultimate effectiveness of LLMs hinges on the quality of the given instructions~\cite{Zhou2022LargeLM,Bang2023AMM,White2023APP}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2302.04023" id="7726">
        <attvalues>
          <attvalue for="5" value=" Despite these benefits, the ultimate effectiveness of LLMs hinges on the quality of the given instructions~\cite{Zhou2022LargeLM,Bang2023AMM,White2023APP}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2302.11382" id="7727">
        <attvalues>
          <attvalue for="5" value=" Despite these benefits, the ultimate effectiveness of LLMs hinges on the quality of the given instructions~\cite{Zhou2022LargeLM,Bang2023AMM,White2023APP}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2101.00190" id="7728">
        <attvalues>
          <attvalue for="5" value=" Previous efforts have focused on gradient-based strategies~\cite{Shin2020ElicitingKF,Li2021PrefixTuningOC,Lester2021ThePO} for automated instruction optimization, the applicability is limited to less advanced public models, leaving out many advanced yet proprietary models." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2104.08691" id="7729">
        <attvalues>
          <attvalue for="5" value=" Previous efforts have focused on gradient-based strategies~\cite{Shin2020ElicitingKF,Li2021PrefixTuningOC,Lester2021ThePO} for automated instruction optimization, the applicability is limited to less advanced public models, leaving out many advanced yet proprietary models." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2310.06825" id="7730">
        <attvalues>
          <attvalue for="5" value=" With the emergence of more advanced open models such as Mistral-7B~\cite{Jiang2023Mistral7}, Llama-70B~\cite{Touvron2023LLaMAOA,Touvron2023Llama2O}, and Mixtral-8x7B~\cite{Jiang2024MixtralOE}, which offer transparency and have reported performance that even surpasses that of ChatGPT-3." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2307.09288" id="7732">
        <attvalues>
          <attvalue for="5" value=" With the emergence of more advanced open models such as Mistral-7B~\cite{Jiang2023Mistral7}, Llama-70B~\cite{Touvron2023LLaMAOA,Touvron2023Llama2O}, and Mixtral-8x7B~\cite{Jiang2024MixtralOE}, which offer transparency and have reported performance that even surpasses that of ChatGPT-3." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2401.04088" id="7733">
        <attvalues>
          <attvalue for="5" value=" With the emergence of more advanced open models such as Mistral-7B~\cite{Jiang2023Mistral7}, Llama-70B~\cite{Touvron2023LLaMAOA,Touvron2023Llama2O}, and Mixtral-8x7B~\cite{Jiang2024MixtralOE}, which offer transparency and have reported performance that even surpasses that of ChatGPT-3." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="1911.04462" id="7735">
        <attvalues>
          <attvalue for="5" value=" Considering that {\itshape capturing the nuances of persona in natural language instructions is challenging}, we aim to directly optimize the soft token embeddings~\cite{Chen2023InstructZeroEI,Lin2023UseYI}, representing the contextual factors through user feedback by utilizing neural bandit algorithms~\cite{Zhou2019NeuralCB,zhang2021neural}." />
        </attvalues>
      </edge>
      <edge source="2404.16115" target="2010.00827" id="7736">
        <attvalues>
          <attvalue for="5" value=" Considering that {\itshape capturing the nuances of persona in natural language instructions is challenging}, we aim to directly optimize the soft token embeddings~\cite{Chen2023InstructZeroEI,Lin2023UseYI}, representing the contextual factors through user feedback by utilizing neural bandit algorithms~\cite{Zhou2019NeuralCB,zhang2021neural}." />
        </attvalues>
      </edge>
      <edge source="2405.20574" target="2303.18223" id="7737">
        <attvalues>
          <attvalue for="5" value=" The emergence of Large Language Models (LLMs)~\cite{zhao2023survey} have also introduced an ever growing demand for robust evaluation frameworks for LLMs." />
        </attvalues>
      </edge>
      <edge source="2405.20574" target="2310.18018" id="7738">
        <attvalues>
          <attvalue for="5" value=" Further, our private test sets allow for robust evaluation of a plethora of models in the wild without significant worry of data contamination on the tested benchmarks~\cite{sainz2023nlp,zhou2023don,balloccu2024leak}." />
        </attvalues>
      </edge>
      <edge source="2405.20574" target="2311.01964" id="7739">
        <attvalues>
          <attvalue for="5" value=" Further, our private test sets allow for robust evaluation of a plethora of models in the wild without significant worry of data contamination on the tested benchmarks~\cite{sainz2023nlp,zhou2023don,balloccu2024leak}." />
        </attvalues>
      </edge>
      <edge source="2405.20574" target="2402.03927" id="7740">
        <attvalues>
          <attvalue for="5" value=" Further, our private test sets allow for robust evaluation of a plethora of models in the wild without significant worry of data contamination on the tested benchmarks~\cite{sainz2023nlp,zhou2023don,balloccu2024leak}." />
        </attvalues>
      </edge>
      <edge source="2304.13250" target="2210.07128" id="7743">
        <attvalues>
          <attvalue for="5" value=" Furthermore, prompting such PLMs with code-like structures (eg, Python, JSON, PDDL) instead of text has been shown to lead to performance improvements on structured common sense reasoning \cite{madaan2022language}, event argument extraction \cite{wang2022code4struct}, knowledge graph construction \cite{bi2023codekgc}, story understanding \cite{dong2022corrpus}, and causal reasoning \cite{zhang-etal-2023-causal}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="1903.08855" id="7746">
        <attvalues>
          <attvalue for="5" value=" Pre-training on large corpora of text enables the natural language processing (NLP) models to acquire a vast amount of factual and commonsense knowledge \cite{liu-etal-2019-linguistic,petroni-etal-2019-language,yogatama2019learning,davison-etal-2019-commonsense}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="1909.01066" id="7747">
        <attvalues>
          <attvalue for="5" value=" Pre-training on large corpora of text enables the natural language processing (NLP) models to acquire a vast amount of factual and commonsense knowledge \cite{liu-etal-2019-linguistic,petroni-etal-2019-language,yogatama2019learning,davison-etal-2019-commonsense}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="1901.11373" id="7748">
        <attvalues>
          <attvalue for="5" value=" Pre-training on large corpora of text enables the natural language processing (NLP) models to acquire a vast amount of factual and commonsense knowledge \cite{liu-etal-2019-linguistic,petroni-etal-2019-language,yogatama2019learning,davison-etal-2019-commonsense}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="1811.00937" id="7751">
        <attvalues>
          <attvalue for="5" value=" How well can state-of-the-art NLP models perform in such scenarios? Recently, many datasets have been created that test different language understanding skills such as pronoun resolution \cite{sakaguchi2021winogrande,levesque_winograd_2012}, commonsense reasoning \cite{talmor-etal-2019-commonsenseqa}, numerical reasoning \cite{dua-etal-2019-drop,patel-etal-2021-nlp,mishra-etal-2022-numglue}, qualitative reasoning \cite{tafjord-etal-2019-quartz, tafjord2019quarel}, temporal reasoning \cite{zhou-etal-2019-going}, and feasibility reasoning \cite{gupta2022john}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="1903.00161" id="7752">
        <attvalues>
          <attvalue for="5" value=" How well can state-of-the-art NLP models perform in such scenarios? Recently, many datasets have been created that test different language understanding skills such as pronoun resolution \cite{sakaguchi2021winogrande,levesque_winograd_2012}, commonsense reasoning \cite{talmor-etal-2019-commonsenseqa}, numerical reasoning \cite{dua-etal-2019-drop,patel-etal-2021-nlp,mishra-etal-2022-numglue}, qualitative reasoning \cite{tafjord-etal-2019-quartz, tafjord2019quarel}, temporal reasoning \cite{zhou-etal-2019-going}, and feasibility reasoning \cite{gupta2022john}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="2103.07191" id="7753">
        <attvalues>
          <attvalue for="5" value=" How well can state-of-the-art NLP models perform in such scenarios? Recently, many datasets have been created that test different language understanding skills such as pronoun resolution \cite{sakaguchi2021winogrande,levesque_winograd_2012}, commonsense reasoning \cite{talmor-etal-2019-commonsenseqa}, numerical reasoning \cite{dua-etal-2019-drop,patel-etal-2021-nlp,mishra-etal-2022-numglue}, qualitative reasoning \cite{tafjord-etal-2019-quartz, tafjord2019quarel}, temporal reasoning \cite{zhou-etal-2019-going}, and feasibility reasoning \cite{gupta2022john}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="2204.05660" id="7754">
        <attvalues>
          <attvalue for="5" value=" How well can state-of-the-art NLP models perform in such scenarios? Recently, many datasets have been created that test different language understanding skills such as pronoun resolution \cite{sakaguchi2021winogrande,levesque_winograd_2012}, commonsense reasoning \cite{talmor-etal-2019-commonsenseqa}, numerical reasoning \cite{dua-etal-2019-drop,patel-etal-2021-nlp,mishra-etal-2022-numglue}, qualitative reasoning \cite{tafjord-etal-2019-quartz, tafjord2019quarel}, temporal reasoning \cite{zhou-etal-2019-going}, and feasibility reasoning \cite{gupta2022john}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="1909.03553" id="7755">
        <attvalues>
          <attvalue for="5" value=" How well can state-of-the-art NLP models perform in such scenarios? Recently, many datasets have been created that test different language understanding skills such as pronoun resolution \cite{sakaguchi2021winogrande,levesque_winograd_2012}, commonsense reasoning \cite{talmor-etal-2019-commonsenseqa}, numerical reasoning \cite{dua-etal-2019-drop,patel-etal-2021-nlp,mishra-etal-2022-numglue}, qualitative reasoning \cite{tafjord-etal-2019-quartz, tafjord2019quarel}, temporal reasoning \cite{zhou-etal-2019-going}, and feasibility reasoning \cite{gupta2022john}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="2002.00293" id="7757">
        <attvalues>
          <attvalue for="5" value=" Furthermore, numerous adversarial datasets \cite{mccoy-etal-2019-right,bartolo-etal-2020-beat, naik-etal-2018-stress} have also been developed that test the robustness of models." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="2109.05052" id="7758">
        <attvalues>
          <attvalue for="5" value=" \cite{longpre-etal-2021-entity} study entity-based conflicts in the parametric and contextual knowledge." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="2004.04123" id="7759">
        <attvalues>
          <attvalue for="5" value=" \cite{agarwal2020entity} investigate entity-based swapping to test the robustness of models." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="2210.11416" id="7762">
        <attvalues>
          <attvalue for="5" value=" We conduct comprehensive experiments with several NLP models such as Flan T5 \cite{chung2022scaling}, GPT-3 \cite{NEURIPS2020_1457c0d6}, and UnifiedQA \cite{khashabi-etal-2020-unifiedqa}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="2005.14165" id="7763">
        <attvalues>
          <attvalue for="5" value=" We conduct comprehensive experiments with several NLP models such as Flan T5 \cite{chung2022scaling}, GPT-3 \cite{NEURIPS2020_1457c0d6}, and UnifiedQA \cite{khashabi-etal-2020-unifiedqa}." />
        </attvalues>
      </edge>
      <edge source="2305.12096" target="2005.00700" id="7764">
        <attvalues>
          <attvalue for="5" value=" We conduct comprehensive experiments with several NLP models such as Flan T5 \cite{chung2022scaling}, GPT-3 \cite{NEURIPS2020_1457c0d6}, and UnifiedQA \cite{khashabi-etal-2020-unifiedqa}." />
        </attvalues>
      </edge>
      <edge source="2102.11037" target="1807.07987" id="7765">
        <attvalues>
          <attvalue for="5" value=" In the past ten years, developments of Deep Learning methods \cite{Goodfellow-et-al-2016, lecun2015deep} have enabled research in Natural Language Processing (NLP) to take an impressive leap." />
        </attvalues>
      </edge>
      <edge source="2102.11037" target="1606.05250" id="7766">
        <attvalues>
          <attvalue for="5" value=" Some tasks, like Question Answering \cite{rajpurkar2016squad} or Sentiment Analysis \cite{maas-EtAl:2011:ACL-HLT2011}, seemed unrealistic twenty years ago, and nowadays recent neural models achieved better scores than humans \cite{devlin2018bert} \cite{lan2019albert} for these applications." />
        </attvalues>
      </edge>
      <edge source="2102.11037" target="1909.11942" id="7768">
        <attvalues>
          <attvalue for="5" value=" Some tasks, like Question Answering \cite{rajpurkar2016squad} or Sentiment Analysis \cite{maas-EtAl:2011:ACL-HLT2011}, seemed unrealistic twenty years ago, and nowadays recent neural models achieved better scores than humans \cite{devlin2018bert} \cite{lan2019albert} for these applications." />
        </attvalues>
      </edge>
      <edge source="2102.11037" target="1906.02243" id="7769">
        <attvalues>
          <attvalue for="5" value=" We can produce models that achieve impressive scores, but deployment and climate change problems \cite{strubell2019energy} raise some issues." />
        </attvalues>
      </edge>
      <edge source="2310.09263" target="2005.14165" id="7771">
        <attvalues>
          <attvalue for="5" value=" While prompt-engineering is a promising direction to enhance model performance, it requires task-specific tuning (eg, task-specific labeled-data to test the performance of different instruction/example combinations)~\cite{llm-gpt-3, prompt-engineering, prompt-engineering-2}." />
        </attvalues>
      </edge>
      <edge source="2310.09263" target="1909.02164" id="7773">
        <attvalues>
          <attvalue for="5" value=" Consider, for example, the popular NLP task of (T-3) Table-QA~\cite{table-qa-tabfact, table-qa-wikitablequestions, table-qa-2}, where the task is to answer a natural-language question, based on the content of the table." />
        </attvalues>
      </edge>
      <edge source="2310.09263" target="2109.01652" id="7774">
        <attvalues>
          <attvalue for="5" value=" It was observed in the NLP community~\cite{flan, llm-gpt-3, instruct-gpt}, that earlier versions of pre-trained language models, such as GPT-3, is able to complete a sentence with the next likely token (eg, ``\code{write a bed-time}'' $arrow$ ``\code{story}''), but cannot reliable follow higher-level instructions from humans (eg, ``\code{write a bed-time story for a 3 years-old, in 100 words}''), a behavior that is only demonstrated in later models such as ChatGPT." />
        </attvalues>
      </edge>
      <edge source="2408.04303" target="2305.15425" id="7777">
        <attvalues>
          <attvalue for="5" value=" Multilingual tokenization is unfair, with all existing approaches inadvertently favoring some languages over others \cite{petrov-etal-2023-tokenizer-unfairness,rust-etal-2021-good}." />
        </attvalues>
      </edge>
      <edge source="2408.04303" target="2012.15613" id="7778">
        <attvalues>
          <attvalue for="5" value=" Multilingual tokenization is unfair, with all existing approaches inadvertently favoring some languages over others \cite{petrov-etal-2023-tokenizer-unfairness,rust-etal-2021-good}." />
        </attvalues>
      </edge>
      <edge source="2408.04303" target="2305.17179" id="7779">
        <attvalues>
          <attvalue for="5" value=" Western European languages often benefit from this, thanks to their shared alphabet and linguistic heritage \cite{limisiewicz-etal-2023-tokenization}." />
        </attvalues>
      </edge>
      <edge source="2408.04303" target="2212.01304" id="7781">
        <attvalues>
          <attvalue for="5" value=" Although character or byte-level encoders appear to handle diverse scripts more fairly, they frequently struggle to capture meaningful word-level information, especially in non-ideographic languages with limited alphabets \cite{libovicky-etal-2022-dont,edman-etal-2022-subword}." />
        </attvalues>
      </edge>
      <edge source="2408.04303" target="2004.09095" id="7782">
        <attvalues>
          <attvalue for="5" value=" The trillion tokens required for training LLMs simply does not exist in most languages \cite{joshi-etal-2020-state}, turning transfer learning into a requirement." />
        </attvalues>
      </edge>
      <edge source="2408.04303" target="2211.05102" id="7783">
        <attvalues>
          <attvalue for="5" value=" Efficient computation necessitates the batch-processing of requests \cite{pope-2022-efficiently}, but many languages also suffer from intermittent workloads." />
        </attvalues>
      </edge>
      <edge source="2408.04303" target="2312.11514" id="7784">
        <attvalues>
          <attvalue for="5" value=" This also makes it unsustainable to dedicate extensive GPU resources to continuously host often-idling LLMs, while the time required to load them back into memory impedes many commercial applications that require low latency \cite{apple-llm-flask}." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="1905.05879" id="7785">
        <attvalues>
          <attvalue for="5" value=" AutoVC~\cite{AUTOVC} attempted to disentangle speaker traits from language by a carefully designed autoencoder." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="2004.07370" id="7786">
        <attvalues>
          <attvalue for="5" value=" To separate speaker timbre from prosody, works~\cite{AUTOVCF0} provided pitch contours explicitly to the system." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="2006.04154" id="7788">
        <attvalues>
          <attvalue for="5" value=" Several works~\cite{liu19c_interspeech,VQVC+} further improved content separation by learning representations with vector quantization." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="2004.11284" id="7790">
        <attvalues>
          <attvalue for="5" value=" SpeechSplit~\cite{SpeechSplit,SpeechSplit2} achieved rhythm and pitch conversion with multiple carefully designed autoencoders." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="2208.08757" id="7791">
        <attvalues>
          <attvalue for="5" value=" Leveraging these works, SRDVC~\cite{SRDVC} presented a unified one-shot VC system that allows control over both prosody and speaker attributes." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="2006.11477" id="7792">
        <attvalues>
          <attvalue for="5" value=" To address this, recent approaches began to explore self-supervised speech representation~\cite{wav2vec2,wav2vec} (S3R) as a source of language information." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="1904.05862" id="7793">
        <attvalues>
          <attvalue for="5" value=" To address this, recent approaches began to explore self-supervised speech representation~\cite{wav2vec2,wav2vec} (S3R) as a source of language information." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="2110.14513" id="7794">
        <attvalues>
          <attvalue for="5" value=" However, these works~\cite{NANSY,s3prl-vc} generally focus on continuous S3R and are limited to speaker conversion." />
        </attvalues>
      </edge>
      <edge source="2211.06535" target="2110.06280" id="7795">
        <attvalues>
          <attvalue for="5" value=" However, these works~\cite{NANSY,s3prl-vc} generally focus on continuous S3R and are limited to speaker conversion." />
        </attvalues>
      </edge>
      <edge source="2210.13749" target="2103.07552" id="7796">
        <attvalues>
          <attvalue for="5" value=" Data augmentation in NLP can be useful in many situations, from low resource data setting, domain adaptation \cite{wei2021few}, debiasing \cite{dinan2020queens}, to improving generalization, robustness \cite{dhole2021nl}." />
        </attvalues>
      </edge>
      <edge source="2210.13749" target="1911.03842" id="7797">
        <attvalues>
          <attvalue for="5" value=" Data augmentation in NLP can be useful in many situations, from low resource data setting, domain adaptation \cite{wei2021few}, debiasing \cite{dinan2020queens}, to improving generalization, robustness \cite{dhole2021nl}." />
        </attvalues>
      </edge>
      <edge source="2210.13749" target="2112.02721" id="7798">
        <attvalues>
          <attvalue for="5" value=" Data augmentation in NLP can be useful in many situations, from low resource data setting, domain adaptation \cite{wei2021few}, debiasing \cite{dinan2020queens}, to improving generalization, robustness \cite{dhole2021nl}." />
        </attvalues>
      </edge>
      <edge source="2210.13749" target="2002.05709" id="7799">
        <attvalues>
          <attvalue for="5" value=" In the vision domain, \cite{chen2020simple} shows that a diverse set of augmentation can be used to learn a robust general-purpose representation with contrastive learning." />
        </attvalues>
      </edge>
      <edge source="2210.13749" target="2104.08821" id="7800">
        <attvalues>
          <attvalue for="5" value=" It is not straightforward to find the best augmentations that work for contrastive learning in different datasets or tasks \cite{gao2021simcse}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="1607.06520" id="7802">
        <attvalues>
          <attvalue for="5" value=" Such an application may be particularly relevant for mitigating gender bias, as text generated by models often encodes societal biases with respect to gender \cite{bolukbasi2016man, zhao-etal-2018-learning}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="2004.07667" id="7803">
        <attvalues>
          <attvalue for="5" value=" The key conceptual point in our paper is the connection between concept erasure techniques and steering \cite{ravfogel2020null, ravfogel2022linear, ravfogel2022adversarial, belrose2024leace, guerner2023geometric}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="2005.13407" id="7805">
        <attvalues>
          <attvalue for="5" value=" The manipulation of these representations, referred to as representation surgery, enables to both better understand the model's behavior and to shape the text it generates \cite{bolukbasi2016man,ravfogel2020null,elazar2021amnesic,feder2021causalm,meng2022locating,geva2021transformer,ghandeharioun2024patchscope}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="2202.05262" id="7806">
        <attvalues>
          <attvalue for="5" value=" The manipulation of these representations, referred to as representation surgery, enables to both better understand the model's behavior and to shape the text it generates \cite{bolukbasi2016man,ravfogel2020null,elazar2021amnesic,feder2021causalm,meng2022locating,geva2021transformer,ghandeharioun2024patchscope}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="2012.14913" id="7807">
        <attvalues>
          <attvalue for="5" value=" The manipulation of these representations, referred to as representation surgery, enables to both better understand the model's behavior and to shape the text it generates \cite{bolukbasi2016man,ravfogel2020null,elazar2021amnesic,feder2021causalm,meng2022locating,geva2021transformer,ghandeharioun2024patchscope}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="2205.05124" id="7809">
        <attvalues>
          <attvalue for="5" value=" This function turns out to be a linear translation of the representations, giving a theoretical justification to the usage of steering vectors \cite{subramani2022extracting, li2023inference}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="2210.10012" id="7810">
        <attvalues>
          <attvalue for="5" value=" Digging into the formal underpinning of concept erasure, however, we find that concept erasure techniques are built on the notion of guardedness \cite{ravfogel-etal-2023-linear}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="2306.03819" id="7811">
        <attvalues>
          <attvalue for="5" value=" Thus, subject to a guardedness constraint, concept erasure techniques search for an affine transformation that minimally alters the existing representations \cite{belrose2024leace}." />
        </attvalues>
      </edge>
      <edge source="2402.09631" target="1903.03862" id="7812">
        <attvalues>
          <attvalue for="5" value=" Applying the covariance constraint endows the resulting steering function with another guarantee: it provably removes bias by neighbors \cite{gonen2019lipstick} in expectation, eg, it reduces the tendency of the representations to cluster by their associated gender." />
        </attvalues>
      </edge>
      <edge source="2207.00216" target="1802.07569" id="7813">
        <attvalues>
          <attvalue for="5" value=" A typical domain adaptation method~\cite{NIPS2006_b1b0432c,mirsamadi17_interspeech} aims to improve the performance on a target domain and thus usually suffers from performance degradation on a source domain, which is known as catastrophic forgetting~\cite{journals/nn/ParisiKPKW19}." />
        </attvalues>
      </edge>
      <edge source="2207.00216" target="1708.01547" id="7814">
        <attvalues>
          <attvalue for="5" value=" Such methods require retaining the previous model~\cite{conf/iclr/YoonYLH18,fu2021incremental} or adding extra parameters of the same size as the model for optimization~\cite{rusu2016progressive,conf/asru/GhorbaniKH19}." />
        </attvalues>
      </edge>
      <edge source="2207.00216" target="1606.04671" id="7815">
        <attvalues>
          <attvalue for="5" value=" Such methods require retaining the previous model~\cite{conf/iclr/YoonYLH18,fu2021incremental} or adding extra parameters of the same size as the model for optimization~\cite{rusu2016progressive,conf/asru/GhorbaniKH19}." />
        </attvalues>
      </edge>
      <edge source="2207.00216" target="1910.00565" id="7816">
        <attvalues>
          <attvalue for="5" value=" Such methods require retaining the previous model~\cite{conf/iclr/YoonYLH18,fu2021incremental} or adding extra parameters of the same size as the model for optimization~\cite{rusu2016progressive,conf/asru/GhorbaniKH19}." />
        </attvalues>
      </edge>
      <edge source="2207.00216" target="2005.04288" id="7817">
        <attvalues>
          <attvalue for="5" value=" Such methods require retaining the previous model~\cite{conf/iclr/YoonYLH18,fu2021incremental} or adding extra parameters of the same size as the model for optimization~\cite{rusu2016progressive,conf/asru/GhorbaniKH19}." />
        </attvalues>
      </edge>
      <edge source="2207.00216" target="1503.02531" id="7818">
        <attvalues>
          <attvalue for="5" value="}~\cite{fu2021incremental} proposed an incremental learning algorithm for end-to-end ASR that uses attention distillation and knowledge distillation~\cite{hinton2015distilling} to prevent catastrophic forgetting." />
        </attvalues>
      </edge>
      <edge source="2207.00216" target="2103.13678" id="7819">
        <attvalues>
          <attvalue for="5" value=" In~\cite{DBLP:conf/naacl/GuFX21}, parameters of a subnetwork are fixed during adaptation to keep the performance of the source domain." />
        </attvalues>
      </edge>
      <edge source="2207.00216" target="2202.09167" id="7820">
        <attvalues>
          <attvalue for="5" value=" Some studies on domain adaptation of ASR models have shown that updating only a part of the layers improves the performance on the target domain~\cite{conf/interspeech/UenoMMSSYAK18,sukhadia2022domain}." />
        </attvalues>
      </edge>
      <edge source="1910.00565" target="1809.06833" id="7822">
        <attvalues>
          <attvalue for="5" value=" Current state-of-the-art neural network-based ASR systems have advanced to nearly human performance in several evaluation settings~\cite{saon2017english, best2}; however, these systems perform poorly for domains that are not included in the original training data~\cite{myself1,sun2017unsupervised,hsu2017unsupervised,jafarlou2019LRF}." />
        </attvalues>
      </edge>
      <edge source="1910.00565" target="1910.07047" id="7824">
        <attvalues>
          <attvalue for="5" value=" Current state-of-the-art neural network-based ASR systems have advanced to nearly human performance in several evaluation settings~\cite{saon2017english, best2}; however, these systems perform poorly for domains that are not included in the original training data~\cite{myself1,sun2017unsupervised,hsu2017unsupervised,jafarlou2019LRF}." />
        </attvalues>
      </edge>
      <edge source="2202.09167" target="2011.01991" id="7825">
        <attvalues>
          <attvalue for="5" value=" In \cite{meng2021internal,zeyer2021LibriSpeech,wang2020multitask}, the idea of internal language model estimation is used to improve performance when the well-trained ASR model is used to decode target-domain (eg, in-domain) data directly without any transfer-learning step." />
        </attvalues>
      </edge>
      <edge source="2202.09167" target="2010.14318" id="7826">
        <attvalues>
          <attvalue for="5" value=" In \cite{meng2021internal,zeyer2021LibriSpeech,wang2020multitask}, the idea of internal language model estimation is used to improve performance when the well-trained ASR model is used to decode target-domain (eg, in-domain) data directly without any transfer-learning step." />
        </attvalues>
      </edge>
      <edge source="2202.09167" target="2006.11477" id="7827">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, HuBERT \cite{hsu2021hubert} have become popular to address the low-resource speech modelling problem." />
        </attvalues>
      </edge>
      <edge source="2202.09167" target="2106.07447" id="7828">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, HuBERT \cite{hsu2021hubert} have become popular to address the low-resource speech modelling problem." />
        </attvalues>
      </edge>
      <edge source="2202.09167" target="2104.01027" id="7829">
        <attvalues>
          <attvalue for="5" value=" However, they do suffer significant degradation when the target-domain is different from the data used in pre-training the model \cite{hsu2021robust,conneau2019unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2202.09167" target="1911.02116" id="7830">
        <attvalues>
          <attvalue for="5" value=" However, they do suffer significant degradation when the target-domain is different from the data used in pre-training the model \cite{hsu2021robust,conneau2019unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2202.09167" target="1411.1792" id="7831">
        <attvalues>
          <attvalue for="5" value=" Our approach is along a line of thinking that bears some resemblance to the works in \cite{ghahremani2017investigation,yosinski2014transferable}." />
        </attvalues>
      </edge>
      <edge source="2202.09167" target="1904.08779" id="7832">
        <attvalues>
          <attvalue for="5" value=" \item The effect of applying a Spectral Augmentation (SpecAug) \cite{park2019specaugment} step on the well-trained ASR encoder features to improve the performance of the low-resource model." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1706.04902" id="7833">
        <attvalues>
          <attvalue for="5" value="2999959}, cross-lingual~\cite{ruder2017survey}, contextualized~\cite{peters:NAACL2018}, retrofitted~\cite{faruqui2015retrofitting}, multi-sense~\cite{PilehvarCNC17}, cross-domain~\cite{YangLZ17}, dependency-based~\cite{LevyG14}), encoding architecture, (convolution~\cite{kim2015}, linear vector operations~\cite{bojanowski:TACL2017}, bidirectional LSTM~\cite{ling2015}), as well as in terms of the target units (words, characters, character n-grams, morphemes, phonemes)." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1802.05365" id="7834">
        <attvalues>
          <attvalue for="5" value="2999959}, cross-lingual~\cite{ruder2017survey}, contextualized~\cite{peters:NAACL2018}, retrofitted~\cite{faruqui2015retrofitting}, multi-sense~\cite{PilehvarCNC17}, cross-domain~\cite{YangLZ17}, dependency-based~\cite{LevyG14}), encoding architecture, (convolution~\cite{kim2015}, linear vector operations~\cite{bojanowski:TACL2017}, bidirectional LSTM~\cite{ling2015}), as well as in terms of the target units (words, characters, character n-grams, morphemes, phonemes)." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1902.00184" id="7837">
        <attvalues>
          <attvalue for="5" value="2999959}, cross-lingual~\cite{ruder2017survey}, contextualized~\cite{peters:NAACL2018}, retrofitted~\cite{faruqui2015retrofitting}, multi-sense~\cite{PilehvarCNC17}, cross-domain~\cite{YangLZ17}, dependency-based~\cite{LevyG14}), encoding architecture, (convolution~\cite{kim2015}, linear vector operations~\cite{bojanowski:TACL2017}, bidirectional LSTM~\cite{ling2015}), as well as in terms of the target units (words, characters, character n-grams, morphemes, phonemes)." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1508.06615" id="7838">
        <attvalues>
          <attvalue for="5" value="2999959}, cross-lingual~\cite{ruder2017survey}, contextualized~\cite{peters:NAACL2018}, retrofitted~\cite{faruqui2015retrofitting}, multi-sense~\cite{PilehvarCNC17}, cross-domain~\cite{YangLZ17}, dependency-based~\cite{LevyG14}), encoding architecture, (convolution~\cite{kim2015}, linear vector operations~\cite{bojanowski:TACL2017}, bidirectional LSTM~\cite{ling2015}), as well as in terms of the target units (words, characters, character n-grams, morphemes, phonemes)." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1805.11937" id="7841">
        <attvalues>
          <attvalue for="5" value=" Another \rev{popular technique} option to assess the quality of word representation is through extrinsic evaluation, where the word vectors are used directly in downstream tasks, such as machine translation (MT)~\cite{ataman2018}, semantic role labeling (SRL)~\cite{sahin:acl18} or language modeling (LM)~\cite{vania2017}." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1805.02036" id="7843">
        <attvalues>
          <attvalue for="5" value=" Another \rev{popular technique} option to assess the quality of word representation is through extrinsic evaluation, where the word vectors are used directly in downstream tasks, such as machine translation (MT)~\cite{ataman2018}, semantic role labeling (SRL)~\cite{sahin:acl18} or language modeling (LM)~\cite{vania2017}." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1704.08352" id="7844">
        <attvalues>
          <attvalue for="5" value=" Another \rev{popular technique} option to assess the quality of word representation is through extrinsic evaluation, where the word vectors are used directly in downstream tasks, such as machine translation (MT)~\cite{ataman2018}, semantic role labeling (SRL)~\cite{sahin:acl18} or language modeling (LM)~\cite{vania2017}." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1608.04207" id="7845">
        <attvalues>
          <attvalue for="5" value=" To address the aforementioned problems, a few studies have introduced the idea of probing tasks \cite{kohn2016evaluating,shi-etAl:ACL2016,adi2017fine,Veldhoen2016DiagnosticCR,senteval18}; which are a set of \rev{multi-label}{multi-class} classification problems that probe a learned word vector for a specific linguistic property, such as part-of-speech (POS), semantic, or morphological tag." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1905.06316" id="7847">
        <attvalues>
          <attvalue for="5" value="} In this work, \begin{itemize} \item We extend the line of work by~\cite{senteval18} and \cite{tenney2018what} and introduce 15 type-level probing tasks for 24 languages by taking language properties into account." />
        </attvalues>
      </edge>
      <edge source="1903.09442" target="1810.11101" id="7848">
        <attvalues>
          <attvalue for="5" value=" Although languages share a large set of common probing tasks, each has a list of its own, eg, Russian and Spanish are probed for gender, while Turkish is probed for polarity and possession; \item We introduce a reusable, systematic methodology for creation and evaluation of such tests by utilizing the existing resources such as UniMorph~\cite{sylakGlassmanK15,sylak2016composition,kirov-etal-2018-unimorph}, Wikipedia and Wuggy~\cite{keuleers2010wuggy}; \item We then use the proposed probing tasks to evaluate \rev{the most commonly used multilingual word embedding models}{a set of diverse multilingual embedding models} and to diagnose a neural end-to-end semantic role labeling model as a case study." />
        </attvalues>
      </edge>
      <edge source="2010.12096" target="1211.3711" id="7849">
        <attvalues>
          <attvalue for="5" value=" End-to-end streaming models~\cite{graves2012sequence, he2019streaming, yeh2019transformer, zhang2020transformer, li2020towards, moritz2020streaming, tsunoo2019towards} have become attractive for on-device recognition tasks in two aspects: first, end-to-end models are usually compact, which makes them suitable to be used on devices." />
        </attvalues>
      </edge>
      <edge source="2010.12096" target="1506.07503" id="7850">
        <attvalues>
          <attvalue for="5" value=" Contrary to non-streaming ASR models such as Chorowski et al's attention-based models~\cite{chorowski2015attention} or Chan et al's listen-attend-spell models \cite{chan2016listen}, streaming ASR models cannot utilize the full context." />
        </attvalues>
      </edge>
      <edge source="2010.12096" target="1908.10992" id="7851">
        <attvalues>
          <attvalue for="5" value=" In the past few years, many research efforts have been devoted to improving streaming ASR~\cite{sainath2019two,sainath2020streaming,Saon-distillation-2020}." />
        </attvalues>
      </edge>
      <edge source="2010.12096" target="2003.12710" id="7852">
        <attvalues>
          <attvalue for="5" value=" In the past few years, many research efforts have been devoted to improving streaming ASR~\cite{sainath2019two,sainath2020streaming,Saon-distillation-2020}." />
        </attvalues>
      </edge>
      <edge source="2010.12096" target="1911.04252" id="7854">
        <attvalues>
          <attvalue for="5" value=" (3) We use the waveforms and their predicted transcripts in a noisy student learning framework~\cite{park2020improved,xie2020self,he2019revisiting,li2017learning}." />
        </attvalues>
      </edge>
      <edge source="2010.12096" target="1909.13788" id="7855">
        <attvalues>
          <attvalue for="5" value=" (3) We use the waveforms and their predicted transcripts in a noisy student learning framework~\cite{park2020improved,xie2020self,he2019revisiting,li2017learning}." />
        </attvalues>
      </edge>
      <edge source="2109.10847" target="1907.11692" id="7858">
        <attvalues>
          <attvalue for="5" value=" Recent large-scale transformer-based neural language models like GPT-2 \cite{radford2019language, brown2020language}, BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta}, XLNet \cite{yang2020xlnet}, UniLM \cite{dong2019unified}, ELECTRA \cite{clark2020electra}, T5 \cite{2020t5} have released pre-trained language models." />
        </attvalues>
      </edge>
      <edge source="2109.10847" target="1905.03197" id="7860">
        <attvalues>
          <attvalue for="5" value=" Recent large-scale transformer-based neural language models like GPT-2 \cite{radford2019language, brown2020language}, BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta}, XLNet \cite{yang2020xlnet}, UniLM \cite{dong2019unified}, ELECTRA \cite{clark2020electra}, T5 \cite{2020t5} have released pre-trained language models." />
        </attvalues>
      </edge>
      <edge source="2109.10847" target="2003.10555" id="7861">
        <attvalues>
          <attvalue for="5" value=" Recent large-scale transformer-based neural language models like GPT-2 \cite{radford2019language, brown2020language}, BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta}, XLNet \cite{yang2020xlnet}, UniLM \cite{dong2019unified}, ELECTRA \cite{clark2020electra}, T5 \cite{2020t5} have released pre-trained language models." />
        </attvalues>
      </edge>
      <edge source="2109.10847" target="2005.14165" id="7863">
        <attvalues>
          <attvalue for="5" value="5B parameters, T5-11B \cite{2020t5} has 11B parameters and GPT-3 \cite{brown2020language} has 175B parameters." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2005.14165" id="7873">
        <attvalues>
          <attvalue for="5" value=" While in-context learning has been extensively studied for decoder-only language models like GPT-3~\cite{NEURIPS2020_1457c0d6} and PaLM~\cite{chowdhery2022palm}, research on encoder-decoder language models, which have shown to learn stronger representations~\cite{devlin-etal-2019-bert,10." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2303.08774" id="7874">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in natural language processing have been predominantly driven by the development of large language models (LLMs) \cite{NEURIPS2020_1457c0d6,openai2022chatgpt,openai2023gpt4,chowdhery2022palm,smith2022using}." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2204.02311" id="7875">
        <attvalues>
          <attvalue for="5" value=" While in-context learning has been extensively studied for decoder-only language models like GPT-3~\cite{NEURIPS2020_1457c0d6} and PaLM~\cite{chowdhery2022palm}, research on encoder-decoder language models, which have shown to learn stronger representations~\cite{devlin-etal-2019-bert,10." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2201.11990" id="7876">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in natural language processing have been predominantly driven by the development of large language models (LLMs) \cite{NEURIPS2020_1457c0d6,openai2022chatgpt,openai2023gpt4,chowdhery2022palm,smith2022using}." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2302.06476" id="7877">
        <attvalues>
          <attvalue for="5" value=" These models have demonstrated remarkable performance across a wide range of tasks~\cite{qin2023chatgpt,bubeck2023sparks,huang2022towards}." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2303.12712" id="7878">
        <attvalues>
          <attvalue for="5" value=" These models have demonstrated remarkable performance across a wide range of tasks~\cite{qin2023chatgpt,bubeck2023sparks,huang2022towards}." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2212.10403" id="7879">
        <attvalues>
          <attvalue for="5" value=" These models have demonstrated remarkable performance across a wide range of tasks~\cite{qin2023chatgpt,bubeck2023sparks,huang2022towards}." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2209.14500" id="7880">
        <attvalues>
          <attvalue for="5" value=" Notably, \cite{patel2023bidirectional} tap into the potential of mT5~\cite{xue-etal-2021-mt5}, a multilingual encoder-decoder LM, by iteratively prompting the model to produce long generations with in-context examples." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2010.11934" id="7881">
        <attvalues>
          <attvalue for="5" value=" Notably, \cite{patel2023bidirectional} tap into the potential of mT5~\cite{xue-etal-2021-mt5}, a multilingual encoder-decoder LM, by iteratively prompting the model to produce long generations with in-context examples." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2210.11416" id="7882">
        <attvalues>
          <attvalue for="5" value=" \cite{chung2022scaling,longpre2023flan} finetune T5~\cite{10." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2301.13688" id="7883">
        <attvalues>
          <attvalue for="5" value=" \cite{chung2022scaling,longpre2023flan} finetune T5~\cite{10." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2109.01652" id="7885">
        <attvalues>
          <attvalue for="5" value="3455856} with a large mixture of tasks using instruction tuning~\cite{mishra-etal-2022-cross,wei2022finetuned,sanh2022multitask} to improve model performance and generalization to unseen tasks in both zero-shot and few-shot settings." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2110.08207" id="7886">
        <attvalues>
          <attvalue for="5" value="3455856} with a large mixture of tasks using instruction tuning~\cite{mishra-etal-2022-cross,wei2022finetuned,sanh2022multitask} to improve model performance and generalization to unseen tasks in both zero-shot and few-shot settings." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2111.07408" id="7888">
        <attvalues>
          <attvalue for="5" value=" On the other hand, LLMs still face challenges such as hallucination and limitations in representing the long-tail and most recent knowledge~\cite{mallen2022not,huang-etal-2022-large,luu-etal-2022-time,jang-etal-2022-temporalwiki,zheng2023does}." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2204.14211" id="7889">
        <attvalues>
          <attvalue for="5" value=" On the other hand, LLMs still face challenges such as hallucination and limitations in representing the long-tail and most recent knowledge~\cite{mallen2022not,huang-etal-2022-large,luu-etal-2022-time,jang-etal-2022-temporalwiki,zheng2023does}." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2208.03299" id="7891">
        <attvalues>
          <attvalue for="5" value=" Among these, the encoder-decoder models, such as Atlas~\cite{izacard2022few}, stand out." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2304.06762" id="7893">
        <attvalues>
          <attvalue for="5" value=" Retrieval-augmented language models~\cite{izacard2022few,pmlr-v162-borgeaud22a,wang2023shall,shi2023replug} have emerged as a powerful approach to address these issues by retrieving relevant knowledge from an external corpus." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2301.12652" id="7894">
        <attvalues>
          <attvalue for="5" value=" Retrieval-augmented language models~\cite{izacard2022few,pmlr-v162-borgeaud22a,wang2023shall,shi2023replug} have emerged as a powerful approach to address these issues by retrieving relevant knowledge from an external corpus." />
        </attvalues>
      </edge>
      <edge source="2308.07922" target="2007.01282" id="7895">
        <attvalues>
          <attvalue for="5" value=" They benefit from the strong representation ability of a bidirectional encoder, coupled with of the efficacy of a Fusion-in-Decoder architecture~\cite{izacard-grave-2021-leveraging}, enabling the effective integration of multiple retrieved passages." />
        </attvalues>
      </edge>
      <edge source="2308.04225" target="1312.6114" id="7896">
        <attvalues>
          <attvalue for="5" value=" Among the deep generative models, the VAE~\cite{vae} is a particularly promising model because it provides an efficient inference mechanism to determine the latent factors of variation from the observed data." />
        </attvalues>
      </edge>
      <edge source="2308.04225" target="1907.04809" id="7897">
        <attvalues>
          <attvalue for="5" value=" Although it has been shown that the vanilla VAE is unable to identify the true latent factors of variation from observed data \cite{Khemakhem2020}, this does not render the approach futile." />
        </attvalues>
      </edge>
      <edge source="2308.04225" target="1709.07902" id="7898">
        <attvalues>
          <attvalue for="5" value=" The hierarchical VAE \cite{Hsu2017UnsupervisedLO} and the factorized \cite{fcae} VAE use two encoders for unsupervised disentanglement of the speech signal into two disjoint representations where one captures short-term variations and the other long-term variations in the signal." />
        </attvalues>
      </edge>
      <edge source="2308.04225" target="2005.12963" id="7899">
        <attvalues>
          <attvalue for="5" value=" The hierarchical VAE \cite{Hsu2017UnsupervisedLO} and the factorized \cite{fcae} VAE use two encoders for unsupervised disentanglement of the speech signal into two disjoint representations where one captures short-term variations and the other long-term variations in the signal." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2205.10643" id="7901">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning~(SSL) for speech encoder pre-training benefits various speech processing tasks and outperforms conventional approaches~\cite{mohamed2022self}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2303.01037" id="7902">
        <attvalues>
          <attvalue for="5" value=" SSL methods leverage large unlabeled speech corpus to train deep neural networks to encode useful representations and succeed in applications like speech translation~\cite{seamlessm4t2023} and automatic speech recognition~(ASR)~\cite{zhang2023usm}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2110.01900" id="7903">
        <attvalues>
          <attvalue for="5" value=" DistilHuBERT~\cite{chang2022distilhubert} predicts multiple hidden layers in a HuBERT teacher~\cite{hsu2021hubert} using the student's output with separate prediction heads." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2207.00555" id="7904">
        <attvalues>
          <attvalue for="5" value=" FitHuBERT~\cite{lee2022fithubert} and Ashihara et al." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2207.06867" id="7905">
        <attvalues>
          <attvalue for="5" value=" Under this new problem setting, we propose Contrastive Layer-to-layer Distillation~(CoLLD) by combining L2L KD~\cite{ashihara2022deep} and a contrastive masked prediction learning objective~\cite{baevski2020wav2vec2}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2203.15610" id="7906">
        <attvalues>
          <attvalue for="5" value=" In KD, a lightweight student model learns to predict hidden representations to mimic the large teacher model's behavior~\cite{chang2022distilhubert,lee2022fithubert,ashihara2022deep,wang2022lighthubert,huang2023ensemble,jang2023recycle,wang2023distilxlsr}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2302.12757" id="7907">
        <attvalues>
          <attvalue for="5" value=" In KD, a lightweight student model learns to predict hidden representations to mimic the large teacher model's behavior~\cite{chang2022distilhubert,lee2022fithubert,ashihara2022deep,wang2022lighthubert,huang2023ensemble,jang2023recycle,wang2023distilxlsr}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2305.11685" id="7908">
        <attvalues>
          <attvalue for="5" value=" In KD, a lightweight student model learns to predict hidden representations to mimic the large teacher model's behavior~\cite{chang2022distilhubert,lee2022fithubert,ashihara2022deep,wang2022lighthubert,huang2023ensemble,jang2023recycle,wang2023distilxlsr}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2106.07447" id="7910">
        <attvalues>
          <attvalue for="5" value=" Those works compress a HuBERT Base~\cite{hsu2021hubert} model~(95M parameters) to models around 20M to 30M parameters and evaluate with the Speech processing Universal PERformance Benchmark~(SUPERB)~\cite{yang2021superb,tsai-etal-2022-superb}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2106.05933" id="7911">
        <attvalues>
          <attvalue for="5" value=" In unstructured pruning, parameters with small values are set to zero~\cite{lai2021parp}, while structured pruning removes submodules from a model~\cite{peng2023structured,jiang2023accurate,wang2023task} to reduce the parameters but requires complicated implementation." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2302.14132" id="7912">
        <attvalues>
          <attvalue for="5" value=" In unstructured pruning, parameters with small values are set to zero~\cite{lai2021parp}, while structured pruning removes submodules from a model~\cite{peng2023structured,jiang2023accurate,wang2023task} to reduce the parameters but requires complicated implementation." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2305.19549" id="7913">
        <attvalues>
          <attvalue for="5" value=" In unstructured pruning, parameters with small values are set to zero~\cite{lai2021parp}, while structured pruning removes submodules from a model~\cite{peng2023structured,jiang2023accurate,wang2023task} to reduce the parameters but requires complicated implementation." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2306.01385" id="7914">
        <attvalues>
          <attvalue for="5" value=" In unstructured pruning, parameters with small values are set to zero~\cite{lai2021parp}, while structured pruning removes submodules from a model~\cite{peng2023structured,jiang2023accurate,wang2023task} to reduce the parameters but requires complicated implementation." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2305.17651" id="7915">
        <attvalues>
          <attvalue for="5" value=" Other studies combine the above methods~\cite{peng2023dphubert} or techniques like layer-skipping~\cite{peng2023i3d} and low-bit quantization~\cite{yeh2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2303.07624" id="7916">
        <attvalues>
          <attvalue for="5" value=" Other studies combine the above methods~\cite{peng2023dphubert} or techniques like layer-skipping~\cite{peng2023i3d} and low-bit quantization~\cite{yeh2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2301.00652" id="7917">
        <attvalues>
          <attvalue for="5" value=" Other studies combine the above methods~\cite{peng2023dphubert} or techniques like layer-skipping~\cite{peng2023i3d} and low-bit quantization~\cite{yeh2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2203.06849" id="7919">
        <attvalues>
          <attvalue for="5" value=" Those works compress a HuBERT Base~\cite{hsu2021hubert} model~(95M parameters) to models around 20M to 30M parameters and evaluate with the Speech processing Universal PERformance Benchmark~(SUPERB)~\cite{yang2021superb,tsai-etal-2022-superb}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2006.11477" id="7920">
        <attvalues>
          <attvalue for="5" value=" Under this new problem setting, we propose Contrastive Layer-to-layer Distillation~(CoLLD) by combining L2L KD~\cite{ashihara2022deep} and a contrastive masked prediction learning objective~\cite{baevski2020wav2vec2}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2111.09296" id="7921">
        <attvalues>
          <attvalue for="5" value="0B parameters) and outperforms strong baselines like XLS-R~\cite{babu2022xlsr} and MMS~\cite{pratap2023mms}." />
        </attvalues>
      </edge>
      <edge source="2309.07707" target="2305.13516" id="7922">
        <attvalues>
          <attvalue for="5" value="0B parameters) and outperforms strong baselines like XLS-R~\cite{babu2022xlsr} and MMS~\cite{pratap2023mms}." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="2009.10056" id="7924">
        <attvalues>
          <attvalue for="5" value=" To tackle few-shot intent detection, some recent attempts employ induction network~\cite{geng2019few}, generation-based methods~\cite{xia2020composed, xia-etal-2020-composed}, metric learning~\cite{nguyen2020dynamic}, or self-training~\cite{dopierre2020few}." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="2010.02481" id="7925">
        <attvalues>
          <attvalue for="5" value=" To tackle few-shot intent detection, some recent attempts employ induction network~\cite{geng2019few}, generation-based methods~\cite{xia2020composed, xia-etal-2020-composed}, metric learning~\cite{nguyen2020dynamic}, or self-training~\cite{dopierre2020few}." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="1905.00537" id="7927">
        <attvalues>
          <attvalue for="5" value=" Most recently, large-scale pre-trained language models such as BERT~\cite{devlin2018bert, radford2019language, brown2020language} have shown great promise in many natural language understanding tasks~\cite{DBLP:conf/nips/WangPNSMHLB19}, and there has been a surge of interest in fine-tuning the pre-trained language models for intent detection~\cite{zhang2020discriminative, zhang2020intent, peng2020soloist, wu2020tod, casanueva2020efficient,larson2019evaluation}." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="2010.13009" id="7928">
        <attvalues>
          <attvalue for="5" value=" \item DNNC~\cite{zhang2020discriminative} pre-trains a language model with around $1$ million annotated samples for natural language inference~(NLI) and use the pre-trained model for intent detection." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="1909.02027" id="7929">
        <attvalues>
          <attvalue for="5" value=" For example, the dataset OOS~\cite{larson2019evaluation} provides labeled utterances across $10$ different domains." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="2009.13570" id="7931">
        <attvalues>
          <attvalue for="5" value=" \end{itemize} While these methods have achieved impressive performance, they heavily rely on the existence of a large-scale corpus~\cite{mehri2020dialoglue} that is close in semantics to the target domain or consists of similar tasks for continued pre-training, which needs huge effort for data collection and comes at a high computational cost." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="2004.06871" id="7932">
        <attvalues>
          <attvalue for="5" value=" \item TOD-BERT~\cite{wu2020tod} further pre-trains BERT on a task-oriented dialogue corpus of $100,000$ unlabeled samples with masked language modelling~(MLM) and response contrastive objectives." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="1911.03688" id="7933">
        <attvalues>
          <attvalue for="5" value=" \item USE-ConveRT~\cite{henderson-etal-2020-convert, casanueva2020efficient} investigates a dual encoder model trained with response selection tasks on $727$ million input-response pairs." />
        </attvalues>
      </edge>
      <edge source="2109.05782" target="2009.05781" id="7934">
        <attvalues>
          <attvalue for="5" value=" \item WikiHowRoBERTa~\cite{zhang2020intent} constructs some pre-training tasks based on the wikiHow database with $110,000$ articles." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2111.01690" id="7935">
        <attvalues>
          <attvalue for="5" value=" T{he} performance of end-to-end (E2E) automatic speech recognition (ASR) systems has improved dramatically over the past years \cite{li2021recent,cheng2022eteh,huang2019exploring,miao2020online,miao2020transformer} due to the advanced neural network architectures, improved training criteria, and large amounts of training data." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2307.02351" id="7936">
        <attvalues>
          <attvalue for="5" value=" T{he} performance of end-to-end (E2E) automatic speech recognition (ASR) systems has improved dramatically over the past years \cite{li2021recent,cheng2022eteh,huang2019exploring,miao2020online,miao2020transformer} due to the advanced neural network architectures, improved training criteria, and large amounts of training data." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2001.08290" id="7937">
        <attvalues>
          <attvalue for="5" value=" T{he} performance of end-to-end (E2E) automatic speech recognition (ASR) systems has improved dramatically over the past years \cite{li2021recent,cheng2022eteh,huang2019exploring,miao2020online,miao2020transformer} due to the advanced neural network architectures, improved training criteria, and large amounts of training data." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2008.06580" id="7938">
        <attvalues>
          <attvalue for="5" value=" Since it is impossible to cover all test domains in the training data, applying domain adaptation \cite{bell2020adaptation} for a new target domain is of great interest in the application of ASR." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2105.11905" id="7939">
        <attvalues>
          <attvalue for="5" value=" When the labeled data is available in the target domain, the supervised domain adaptation is straightforward since we could simply use the labeled data to fine-tune the source model \cite{sim2018domain,hou2021exploiting}." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="1804.00522" id="7942">
        <attvalues>
          <attvalue for="5" value=" However, these approaches require a specific design for the target domain \cite{li2017large} or careful tuning of the data generation model \cite{hosseini2018multi}, making them inconvenient when extending to an arbitrary new domain or large-scale applications." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2007.15188" id="7943">
        <attvalues>
          <attvalue for="5" value=" An intuitive solution is to synthesize target domain data \cite{li2017large,hsu2017unsupervised,hosseini2018multi,li2020developing,baskar2021eat,yue2022exploring}." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2104.07474" id="7944">
        <attvalues>
          <attvalue for="5" value=" An intuitive solution is to synthesize target domain data \cite{li2017large,hsu2017unsupervised,hosseini2018multi,li2020developing,baskar2021eat,yue2022exploring}." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2104.07491" id="7945">
        <attvalues>
          <attvalue for="5" value=" Another category is domain-invariant feature learning with distribution matching approaches \cite{hou2021cross,sun2018domain}, which aims to learn a domain-invariant representation while being class-discriminative on the source domain." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="1806.02786" id="7946">
        <attvalues>
          <attvalue for="5" value=" Another category is domain-invariant feature learning with distribution matching approaches \cite{hou2021cross,sun2018domain}, which aims to learn a domain-invariant representation while being class-discriminative on the source domain." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2104.01027" id="7949">
        <attvalues>
          <attvalue for="5" value=" Since continued pre-training~\cite{gururangan2020don,hsu2021robust} is proven effective and used in our approach, we examine whether catastrophic forgetting is severe here by evaluating the effectiveness of data replay~\cite{hu2022how}, which is widely adopted to address knowledge forgetting." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2011.13439" id="7950">
        <attvalues>
          <attvalue for="5" value=" Nonetheless, existing literature typically focused on one aspect to address the UDA problem, eg, with SSL~\cite{hsu2021robust}, online~\cite{higuchi2021momentum} or offline PL~\cite{khurana2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2106.08922" id="7951">
        <attvalues>
          <attvalue for="5" value=" Nonetheless, existing literature typically focused on one aspect to address the UDA problem, eg, with SSL~\cite{hsu2021robust}, online~\cite{higuchi2021momentum} or offline PL~\cite{khurana2021unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2006.04996" id="7952">
        <attvalues>
          <attvalue for="5" value=" In summary, the major novelties of this paper are as follows: \begin{itemize} \item Dual-Branch PL (DPL): There are two vital challenges in existing online PL approaches: Firstly, since the self-generated pseudo-labels are used as the supervision, the errors would be accumulated and cause the error accumulation~\cite{jiang2020implicit} (or the confirmation bias~\cite{arazo2020pseudo}) issue, degrading the performance and sometimes driving the training to collapse." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="1909.09116" id="7954">
        <attvalues>
          <attvalue for="5" value=" \item Uncertainty-Aware Confidence Filtering (UCF): Most existing filtering methods for offline PL utilize decoding scores as confidence estimation to rule out the noisy pseudo-labels~\cite{kahn2020self,park2020improved}." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2010.11428" id="7956">
        <attvalues>
          <attvalue for="5" value=" However, this confidence estimation is unreliable when ASR networks are poorly calibrated~\cite{li2021confidence}." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="1506.02142" id="7957">
        <attvalues>
          <attvalue for="5" value=" UCF addressed this issue by adaptively utilizing uncertainty~\cite{gal2016dropout} and confidence estimations to select pseudo-labels in offline PL, where the combination hyper-parameters in UCF are adaptively determined on the development set." />
        </attvalues>
      </edge>
      <edge source="2206.09783" target="2104.12081" id="7958">
        <attvalues>
          <attvalue for="5" value=" Since continued pre-training~\cite{gururangan2020don,hsu2021robust} is proven effective and used in our approach, we examine whether catastrophic forgetting is severe here by evaluating the effectiveness of data replay~\cite{hu2022how}, which is widely adopted to address knowledge forgetting." />
        </attvalues>
      </edge>
      <edge source="2104.07491" target="1807.06610" id="7959">
        <attvalues>
          <attvalue for="5" value="~\cite{liang2018learning} proposed to combine data augmentation with representation matching to force the model to learn noise-invariant representations between clean speech and their augmented noisy counterparts." />
        </attvalues>
      </edge>
      <edge source="2104.07491" target="2011.13439" id="7960">
        <attvalues>
          <attvalue for="5" value=" In~\cite{khurana2020unsupervised}, the authors were able to recover 60\% to 80\% of the word error rates (WER) on the target domain by introducing a pseudo-label filtering approach based on the model's uncertainty using dropout for ASR UDA." />
        </attvalues>
      </edge>
      <edge source="2104.07491" target="1806.02786" id="7961">
        <attvalues>
          <attvalue for="5" value=" Recent work \cite{sun2018domain,duan2020unsupervised} used the domain-adversarial training~\cite{ganin2016domain} for speech recognition where they adversarially trained domain discriminators to distinguish the source and target samples." />
        </attvalues>
      </edge>
      <edge source="2104.07491" target="1505.07818" id="7962">
        <attvalues>
          <attvalue for="5" value=" Recent work \cite{sun2018domain,duan2020unsupervised} used the domain-adversarial training~\cite{ganin2016domain} for speech recognition where they adversarially trained domain discriminators to distinguish the source and target samples." />
        </attvalues>
      </edge>
      <edge source="2104.07491" target="2106.09388" id="7963">
        <attvalues>
          <attvalue for="5" value=" This is also validated in \cite{zhu2020deep} where the images aligned in subdomains (eg, domains split by class labels) can generally achieve better adaptation performance than traditional methods that align the whole domains." />
        </attvalues>
      </edge>
      <edge source="2010.11428" target="cs/0010012" id="7965">
        <attvalues>
          <attvalue for="5" value=" In conventional HMM-based systems, reliable confidence scores can be easily obtained by computing word posterior probabilities from compact representations of the hypotheses space, \eg lattices or confusion networks~\cite{Mangu2000FindingCI,Evermann2000PosteriorPD}." />
        </attvalues>
      </edge>
      <edge source="2010.11428" target="1810.13025" id="7966">
        <attvalues>
          <attvalue for="5" value=" Improved confidence estimation can be achieved by using model-based approaches, such as conditional random fields\cite{Seigel2011CombiningIS}, recurrent neural networks~\cite{Kalgaonkar2015EstimatingCS,Ragni2018ConfidenceEA} and graph neural networks~\cite{Li2019BidirectionalLR}, or leveraging more related information including phonetics, word/phone duration and language models~\cite{Jiang2005ConfidenceMF,Kastanos2020ConfidenceEF}." />
        </attvalues>
      </edge>
      <edge source="2010.11428" target="1810.13024" id="7967">
        <attvalues>
          <attvalue for="5" value=" Improved confidence estimation can be achieved by using model-based approaches, such as conditional random fields\cite{Seigel2011CombiningIS}, recurrent neural networks~\cite{Kalgaonkar2015EstimatingCS,Ragni2018ConfidenceEA} and graph neural networks~\cite{Li2019BidirectionalLR}, or leveraging more related information including phonetics, word/phone duration and language models~\cite{Jiang2005ConfidenceMF,Kastanos2020ConfidenceEF}." />
        </attvalues>
      </edge>
      <edge source="2010.11428" target="1712.01769" id="7969">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end speech recognition has achieved promising performance over the conventional systems~\cite{Chiu2018StateoftheArtSR}." />
        </attvalues>
      </edge>
      <edge source="2010.11428" target="1811.06621" id="7970">
        <attvalues>
          <attvalue for="5" value=" As end-to-end speech recognition has various modelling and engineering advantages, they are becoming more widely adopted~\cite{He2019StreamingES}." />
        </attvalues>
      </edge>
      <edge source="2010.11428" target="1506.07503" id="7971">
        <attvalues>
          <attvalue for="5" value=" One class of the end-to-end systems is attention-based sequence-to-sequence models~\cite{Chorowski2015AttentionBasedMF,Chan2016ListenAA}." />
        </attvalues>
      </edge>
      <edge source="2010.11428" target="1610.02136" id="7973">
        <attvalues>
          <attvalue for="5" value=" However, the quality of confidence estimation by softmax probabilities may be very poor~\cite{Hendrycks2017ABF}." />
        </attvalues>
      </edge>
      <edge source="1911.01497" target="1409.0473" id="7974">
        <attvalues>
          <attvalue for="5" value=" Sequence-to-sequence (Seq2Seq) networks have achieved impressive results \cite{bahdanau2014neural, sutskever2014sequence, neubig2017neural,vinyals2015neural,venugopalan2015sequence,karpathy2015deep} on a variety of problems within natural language processing." />
        </attvalues>
      </edge>
      <edge source="1911.01497" target="1506.05869" id="7975">
        <attvalues>
          <attvalue for="5" value=" Sequence-to-sequence (Seq2Seq) networks have achieved impressive results \cite{bahdanau2014neural, sutskever2014sequence, neubig2017neural,vinyals2015neural,venugopalan2015sequence,karpathy2015deep} on a variety of problems within natural language processing." />
        </attvalues>
      </edge>
      <edge source="1911.01497" target="1412.2306" id="7976">
        <attvalues>
          <attvalue for="5" value=" Sequence-to-sequence (Seq2Seq) networks have achieved impressive results \cite{bahdanau2014neural, sutskever2014sequence, neubig2017neural,vinyals2015neural,venugopalan2015sequence,karpathy2015deep} on a variety of problems within natural language processing." />
        </attvalues>
      </edge>
      <edge source="1911.01497" target="1409.1259" id="7977">
        <attvalues>
          <attvalue for="5" value=" However, their inability to handle long sentences \cite{length1} as well as a lack of systematic generalizability \cite{lake2017still} questions the ability of seq2seq networks to model compositionality in natural language." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="1506.07503" id="7978">
        <attvalues>
          <attvalue for="5" value=" The E2E frameworks include encoder--decoder networks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18}, connectionist temporal classification (CTC) \cite{Graves-CTC06,Graves-TEE14}, and recurrent neural network transducer (RNN-T) \cite{Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20}." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="1712.01769" id="7979">
        <attvalues>
          <attvalue for="5" value=" The E2E frameworks include encoder--decoder networks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18}, connectionist temporal classification (CTC) \cite{Graves-CTC06,Graves-TEE14}, and recurrent neural network transducer (RNN-T) \cite{Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20}." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="1303.5778" id="7981">
        <attvalues>
          <attvalue for="5" value=" The E2E frameworks include encoder--decoder networks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18}, connectionist temporal classification (CTC) \cite{Graves-CTC06,Graves-TEE14}, and recurrent neural network transducer (RNN-T) \cite{Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20}." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="1801.00841" id="7982">
        <attvalues>
          <attvalue for="5" value=" The E2E frameworks include encoder--decoder networks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18}, connectionist temporal classification (CTC) \cite{Graves-CTC06,Graves-TEE14}, and recurrent neural network transducer (RNN-T) \cite{Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20}." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="2003.12710" id="7983">
        <attvalues>
          <attvalue for="5" value=" The E2E frameworks include encoder--decoder networks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18}, connectionist temporal classification (CTC) \cite{Graves-CTC06,Graves-TEE14}, and recurrent neural network transducer (RNN-T) \cite{Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20}." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="1904.08779" id="7984">
        <attvalues>
          <attvalue for="5" value=" The recent research focus of automatic speech recognition (ASR) is end-to-end (E2E) frameworks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18,Graves-CTC06,Graves-TEE14,Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20,Daniel-SA19,Christoph-RWTH19,Moritz-CMR21}, which can directly map incoming speech signals into characters \cite{Amodei-DS216} or word targets \cite{Hagen-NSR17,Audhkhasi-DAW17}." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="2104.02858" id="7985">
        <attvalues>
          <attvalue for="5" value=" The recent research focus of automatic speech recognition (ASR) is end-to-end (E2E) frameworks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18,Graves-CTC06,Graves-TEE14,Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20,Daniel-SA19,Christoph-RWTH19,Moritz-CMR21}, which can directly map incoming speech signals into characters \cite{Amodei-DS216} or word targets \cite{Hagen-NSR17,Audhkhasi-DAW17}." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="1610.09975" id="7986">
        <attvalues>
          <attvalue for="5" value=" The recent research focus of automatic speech recognition (ASR) is end-to-end (E2E) frameworks \cite{Chan-LAS16,Chorowski-ABM15,Watanabe-HCA17,Chiu-SSR18,Chiu-MCA18,Graves-CTC06,Graves-TEE14,Graves-SRD13,Rao-EAD17,Sainath-ASO20,Kim-ART20,Daniel-SA19,Christoph-RWTH19,Moritz-CMR21}, which can directly map incoming speech signals into characters \cite{Amodei-DS216} or word targets \cite{Hagen-NSR17,Audhkhasi-DAW17}." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="2005.08100" id="7988">
        <attvalues>
          <attvalue for="5" value=" In \cite{Li-ABF21,Xiong-ECP21}, a Conformer-based encoder network was proposed for the RNN-T as the Conformer can effectively model the local-global context information through its convolution and self-attention layers \cite{Anmol-Conformer20}, showing promising performance." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="2106.09236" id="7989">
        <attvalues>
          <attvalue for="5" value=" In \cite{Li-ABF21,Xiong-ECP21}, a Conformer-based encoder network was proposed for the RNN-T as the Conformer can effectively model the local-global context information through its convolution and self-attention layers \cite{Anmol-Conformer20}, showing promising performance." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="2005.03271" id="7991">
        <attvalues>
          <attvalue for="5" value=" Although applying the methods \cite{Chiu-RMF21} improved the WER for out-domain long-form utterances, the investigation was conducted only on long short-term memory (LSTM)-based encoder networks." />
        </attvalues>
      </edge>
      <edge source="2108.10752" target="2011.10798" id="7992">
        <attvalues>
          <attvalue for="5" value=" In \cite{Li-ABF21,Xiong-ECP21}, a Conformer-based encoder network was proposed for the RNN-T as the Conformer can effectively model the local-global context information through its convolution and self-attention layers \cite{Anmol-Conformer20}, showing promising performance." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="2106.15561" id="7993">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text to speech (TTS) ~\cite{tan2021survey,tan2022naturalspeech,wang2017tacotron,shen2018natural,ren2019fastspeech,ren2020fastspeech, zhang2021study} has demonstrated significant successes in producing natural-sounding speech." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="2205.04421" id="7994">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text to speech (TTS) ~\cite{tan2021survey,tan2022naturalspeech,wang2017tacotron,shen2018natural,ren2019fastspeech,ren2020fastspeech, zhang2021study} has demonstrated significant successes in producing natural-sounding speech." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="1703.10135" id="7995">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text to speech (TTS) ~\cite{tan2021survey,tan2022naturalspeech,wang2017tacotron,shen2018natural,ren2019fastspeech,ren2020fastspeech, zhang2021study} has demonstrated significant successes in producing natural-sounding speech." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="1712.05884" id="7996">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text to speech (TTS) ~\cite{tan2021survey,tan2022naturalspeech,wang2017tacotron,shen2018natural,ren2019fastspeech,ren2020fastspeech, zhang2021study} has demonstrated significant successes in producing natural-sounding speech." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="2006.04558" id="7998">
        <attvalues>
          <attvalue for="5" value=" In \cite{ren2020fastspeech, elias21_interspeech, zhang21u_interspeech}, the information from the ground truth speech data, eg pitch, duration, is incorporated in model training to alleviate the ``one-to-many'' problem." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="1808.10128" id="7999">
        <attvalues>
          <attvalue for="5" value=" Recently, leveraging contextual representations learned from unlabeled text data to improve the TTS model has become a rising topic \cite{wang2015word,chung2019semi,hayashi2019pre,kenter2020improving,xu2021improving}, which is also the focus of our paper." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="2011.05161" id="8000">
        <attvalues>
          <attvalue for="5" value=" Recent works~\cite{hayashi2019pre,xiao2020improving,kenter2020improving,xu2021improving} have applied the BERT model as an auxiliary encoder for the TTS system." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="2010.10392" id="8002">
        <attvalues>
          <attvalue for="5" value=" The auxiliary BERT encoder extracts additional text features for character-based units (eg, character\cite{el2020characterbert}, subword\cite{liu2019roberta, devlin2019bert}), which enables the TTS system to generate speech with better pronunciation and expressiveness." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="1907.11692" id="8003">
        <attvalues>
          <attvalue for="5" value=" The auxiliary BERT encoder extracts additional text features for character-based units (eg, character\cite{el2020characterbert}, subword\cite{liu2019roberta, devlin2019bert}), which enables the TTS system to generate speech with better pronunciation and expressiveness." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="1811.07240" id="8004">
        <attvalues>
          <attvalue for="5" value=" Some works \cite{kastner2019representation, jia21_interspeech} attempt to enhance the TTS phoneme encoder with character information directly rather than introducing an auxiliary BERT model." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="1905.10453" id="8005">
        <attvalues>
          <attvalue for="5" value=" So what if we pre-train the phoneme encoder with only the phoneme as the input? We note that the size of the phoneme dictionary is only around 200, and directly using such a small dictionary would not convey contextual semantic information effectively\cite{ding2019call}." />
        </attvalues>
      </edge>
      <edge source="2203.17190" target="1508.07909" id="8006">
        <attvalues>
          <attvalue for="5" value=" The sup-phoneme tokens are obtained by applying the learnt Byte-Pair Encoding (BPE) ~\cite{sennrich2016neural} rules to words." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="2005.00700" id="8020">
        <attvalues>
          <attvalue for="5" value=" Strong fine-tuned QA models like UnifiedQA~\cite{2020unifiedqa} can achieve impressive results on various QA tasks through multitask training, but exhibit subpar performance on multistep reasoning." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="1906.07132" id="8021">
        <attvalues>
          <attvalue for="5" value=" Moreover, because some multistep reasoning benchmarks contain annotation artifacts or reasoning shortcuts~\cite{jiang-bansal-2019-avoiding}, dedicated models trained on these benchmarks often have much lower F1 performance on contrast sets~\cite{gardner-etal-2020-evaluating} and adversarial sets~\cite{Schlegel2020SemanticsAM}, indicating their lack of robustness." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="2004.02709" id="8022">
        <attvalues>
          <attvalue for="5" value=" Moreover, because some multistep reasoning benchmarks contain annotation artifacts or reasoning shortcuts~\cite{jiang-bansal-2019-avoiding}, dedicated models trained on these benchmarks often have much lower F1 performance on contrast sets~\cite{gardner-etal-2020-evaluating} and adversarial sets~\cite{Schlegel2020SemanticsAM}, indicating their lack of robustness." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="2012.04056" id="8023">
        <attvalues>
          <attvalue for="5" value=" Moreover, because some multistep reasoning benchmarks contain annotation artifacts or reasoning shortcuts~\cite{jiang-bansal-2019-avoiding}, dedicated models trained on these benchmarks often have much lower F1 performance on contrast sets~\cite{gardner-etal-2020-evaluating} and adversarial sets~\cite{Schlegel2020SemanticsAM}, indicating their lack of robustness." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="1909.00109" id="8024">
        <attvalues>
          <attvalue for="5" value=" However, as we show in our experiments, existing methods~\cite{andor-etal-2019-giving, Chen2020NeuralSR} that perform explicit reasoning steps still suffer from robustness issues." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="2110.13472" id="8025">
        <attvalues>
          <attvalue for="5" value=" Moreover, multi-step reasoning methods are often engineered for a specific domain or type of multistep QA \cite{fu-etal-2021-decomposing-complex, perez-etal-2020-unsupervised}, and thus cannot be easily extended to other multistep QA settings." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="2303.08774" id="8026">
        <attvalues>
          <attvalue for="5" value=" Prompting methods \cite{Chen2020NeuralSR, dua-etal-2022-successive} have shown promise in generating multistep solutions to questions, but they require very large language models (LMs) as well as careful prompt engineering, and still lag behind fine-tuned methods~\cite{OpenAI2023GPT4TR}." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="1807.02322" id="8027">
        <attvalues>
          <attvalue for="5" value=" To further improve performance, we use a memory buffer to store trajectories with high F1 score, inspired by Memory-Augmented Policy Optimization (MAPO;~\cite{NEURIPS2018_f4e369c0}), previously used for semantic parsing." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="1903.00161" id="8028">
        <attvalues>
          <attvalue for="5" value=" We conduct experiments on DROP~\cite{dua-etal-2019-drop}, HotpotQA~\cite{yang-etal-2018-hotpotqa}, and their contrast and adversarial sets to evaluate the performance of our proposed Chain-of-Questions framework." />
        </attvalues>
      </edge>
      <edge source="2305.14901" target="1809.09600" id="8029">
        <attvalues>
          <attvalue for="5" value=" We conduct experiments on DROP~\cite{dua-etal-2019-drop}, HotpotQA~\cite{yang-etal-2018-hotpotqa}, and their contrast and adversarial sets to evaluate the performance of our proposed Chain-of-Questions framework." />
        </attvalues>
      </edge>
      <edge source="2308.15982" target="2005.14165" id="8031">
        <attvalues>
          <attvalue for="5" value=" However, with the ever-increasing number of parameters in PLMs \cite{NEURIPS2020_1457c0d6}, there is a need for parameter-efficient fine-tuning techniques \cite{DBLP:conf/iclr/HeZMBN22} to reduce training costs." />
        </attvalues>
      </edge>
      <edge source="2308.15982" target="2110.04366" id="8032">
        <attvalues>
          <attvalue for="5" value=" However, with the ever-increasing number of parameters in PLMs \cite{NEURIPS2020_1457c0d6}, there is a need for parameter-efficient fine-tuning techniques \cite{DBLP:conf/iclr/HeZMBN22} to reduce training costs." />
        </attvalues>
      </edge>
      <edge source="2308.15982" target="1902.00751" id="8033">
        <attvalues>
          <attvalue for="5" value=" One representative technique is adapters tuning \cite{houlsby2019parameter}, which updates only a subset of parameters." />
        </attvalues>
      </edge>
      <edge source="2308.15982" target="2208.10160" id="8035">
        <attvalues>
          <attvalue for="5" value=" One possible solution is to transfer knowledge from pretrained adapters to target tasks \cite{Chawla_2021_WACV,zhong2022panda,pmlr-v180-wang22a}." />
        </attvalues>
      </edge>
      <edge source="2308.15982" target="2005.00247" id="8036">
        <attvalues>
          <attvalue for="5" value=" AdapterFusion~\cite{pfeiffer2020adapterfusion} has been proposed to assemble pretrained adapters with composition layers to integrate knowledge." />
        </attvalues>
      </edge>
      <edge source="2308.15982" target="2210.04284" id="8037">
        <attvalues>
          <attvalue for="5" value=" However, AdapterFusion compromises parameter efficiency~\cite{He2022SparseAdapterAE}, primarily due to the excessive number of trainable parameters in the composition layers." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="1211.3711" id="8038">
        <attvalues>
          <attvalue for="5" value=" Specifically, recurrent neural network transducer (RNN-T) originally presented in \cite{graves2012sequence} has shown competitive ASR performance on various benchmarks \cite{chiu2019comparison, li2020comparison, zhang2021benchmarking}." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="1911.02242" id="8039">
        <attvalues>
          <attvalue for="5" value=" Specifically, recurrent neural network transducer (RNN-T) originally presented in \cite{graves2012sequence} has shown competitive ASR performance on various benchmarks \cite{chiu2019comparison, li2020comparison, zhang2021benchmarking}." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2003.12710" id="8040">
        <attvalues>
          <attvalue for="5" value=" Typically based on token emission latency, we categorize ASR models into: (i) streaming recognizers \cite{sainath2020streaming, mahadeokar2021flexi} that emit hypothesized words in real time, with low latency measured by milliseconds, and (ii) non-streaming models \cite{gulati2020conformer, zhang2020pushing} that only emit word hypotheses after processing the complete speech utterance." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2005.08100" id="8041">
        <attvalues>
          <attvalue for="5" value=" Typically based on token emission latency, we categorize ASR models into: (i) streaming recognizers \cite{sainath2020streaming, mahadeokar2021flexi} that emit hypothesized words in real time, with low latency measured by milliseconds, and (ii) non-streaming models \cite{gulati2020conformer, zhang2020pushing} that only emit word hypotheses after processing the complete speech utterance." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2002.02562" id="8042">
        <attvalues>
          <attvalue for="5" value=" Recently it has been shown favorable to unify the streaming and non-streaming models, either through a single shared encoder \cite{zhang2020transformer, yu2020dual, yao2021wenet, kim2021multi, weninger2022conformer}, or through cascaded streaming and non-streaming encoders \cite{li2021better, narayanan2021cascaded}." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2010.10759" id="8043">
        <attvalues>
          <attvalue for="5" value=" Latest streaming recognizers often employ a transformer/conformer encoder \cite{zhang2020transformer, li2021better}, and may use a limited future audio context (also referred to as look-ahead audio frames) \cite{shi2021emformer, shi2022streaming}." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2010.10504" id="8044">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} can substantially improve large non-streaming models; given sufficient unlabeled data, the potential accuracy gain can be proportional to the growing model size \cite{zhang2020pushing}." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2011.10798" id="8045">
        <attvalues>
          <attvalue for="5" value=" Recently it has been shown favorable to unify the streaming and non-streaming models, either through a single shared encoder \cite{zhang2020transformer, yu2020dual, yao2021wenet, kim2021multi, weninger2022conformer}, or through cascaded streaming and non-streaming encoders \cite{li2021better, narayanan2021cascaded}." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2010.06030" id="8046">
        <attvalues>
          <attvalue for="5" value=" While for the unified dual-mode encoder, the non-streaming encoder directly processes the entire utterance and is immune from the accuracy degradation of the streaming encoder; additionally, the accuracy and latency of the streaming encoder can benefit from the weight sharing, or inplace knowledge distillation from the more performant non-streaming encoder \cite{yu2020dual}." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2110.08352" id="8048">
        <attvalues>
          <attvalue for="5" value=" While it can facilitate ASR training of various model sizes, each sub-model in \cite{yang2022omni} operates with the same inference latency." />
        </attvalues>
      </edge>
      <edge source="2207.11906" target="2006.11477" id="8049">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} can substantially improve large non-streaming models; given sufficient unlabeled data, the potential accuracy gain can be proportional to the growing model size \cite{zhang2020pushing}." />
        </attvalues>
      </edge>
      <edge source="2305.12720" target="2005.14165" id="8051">
        <attvalues>
          <attvalue for="5" value=" Specifically, Transformer-based LLMs such as BERT \cite{Devlin2018} and the GPT series \cite{GPT-1,GPT-2,GPT-3} have demonstrated high-performance thanks to their pre-training." />
        </attvalues>
      </edge>
      <edge source="2305.12720" target="2303.08774" id="8052">
        <attvalues>
          <attvalue for="5" value=" Furthermore, models that have evolved from these, such as ChatGPT \cite{chatgpt} and GPT4 \cite{gpt4}, have gained popularity for their remarkable performance." />
        </attvalues>
      </edge>
      <edge source="2305.12720" target="2211.05100" id="8054">
        <attvalues>
          <attvalue for="5" value=" Other models such as Bard \cite{bard}, LLaMA \cite{touvron2023llama}, Dolly \cite{dolly}, Bloom \cite{scao2022bloom}, and Vicuna \cite{vicuna} have also emerged." />
        </attvalues>
      </edge>
      <edge source="2204.03985" target="2207.06300" id="8062">
        <attvalues>
          <attvalue for="5" value=" Recently, we proposed $Re^2G$~\cite{glass-etal-2022-re2g}, the core of our $KGI_$ (Knowledge Graph Induction) system." />
        </attvalues>
      </edge>
      <edge source="2407.17874" target="2006.11477" id="8064">
        <attvalues>
          <attvalue for="5" value="0 \cite{wav2vec} and Whisper \cite{whisper}, have significantly improved the capabilities of speech recognition through extensive training on large datasets." />
        </attvalues>
      </edge>
      <edge source="2407.17874" target="2104.02194" id="8066">
        <attvalues>
          <attvalue for="5" value=" Contextual biasing \cite{aleksic2015bringing} is widely used in previous studies \cite{le2021contextualized,huang2023contextualized,CB-Conformer,sun2023can} to improve the performance of speech recognition for domain-specific words." />
        </attvalues>
      </edge>
      <edge source="2407.17874" target="2305.12493" id="8067">
        <attvalues>
          <attvalue for="5" value=" Contextual biasing \cite{aleksic2015bringing} is widely used in previous studies \cite{le2021contextualized,huang2023contextualized,CB-Conformer,sun2023can} to improve the performance of speech recognition for domain-specific words." />
        </attvalues>
      </edge>
      <edge source="2407.17874" target="2304.09607" id="8068">
        <attvalues>
          <attvalue for="5" value=" Contextual biasing \cite{aleksic2015bringing} is widely used in previous studies \cite{le2021contextualized,huang2023contextualized,CB-Conformer,sun2023can} to improve the performance of speech recognition for domain-specific words." />
        </attvalues>
      </edge>
      <edge source="2407.17874" target="2306.01942" id="8069">
        <attvalues>
          <attvalue for="5" value=" Contextual biasing \cite{aleksic2015bringing} is widely used in previous studies \cite{le2021contextualized,huang2023contextualized,CB-Conformer,sun2023can} to improve the performance of speech recognition for domain-specific words." />
        </attvalues>
      </edge>
      <edge source="2407.17874" target="2309.07414" id="8070">
        <attvalues>
          <attvalue for="5" value=" To alleviate the problem, recent works \cite{PromptASR,chang2023context,promptformer,deep-LLM-fusion} introduce methods to incorporate more specific contextual information." />
        </attvalues>
      </edge>
      <edge source="2407.17874" target="2306.16007" id="8072">
        <attvalues>
          <attvalue for="5" value=" The most relevant work to ours is \cite{deep-LLM-fusion}, which utilizes human-written descriptions." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="1811.06621" id="8074">
        <attvalues>
          <attvalue for="5" value=" Unlike previous work which relied on specialized WFSTs \cite{Zhao2019,He2019RNNT,Le2021deepshallow}, we use a generic WFST in this work that does not require strong context prefixes nor domain-specific sentence patterns." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="2005.08100" id="8075">
        <attvalues>
          <attvalue for="5" value=" End-to-end automatic speech recognition (ASR) models have become increasingly popular in recent years, thanks to their simplicity and competitive performance on generic transcription tasks \cite{Prabhavalkar17,He2019RNNT,Gulati2020conformer,Zhang2021benchmark}." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="1712.01996" id="8078">
        <attvalues>
          <attvalue for="5" value=" For static knowledge (eg, knowledge that does not change from one utterance to the next), various LM fusion techniques have been proposed \cite{gulcehre2015using,Kannan2018,sriram2018cold,toshniwal2018comparison,shan2019component,Kim2021lmfusion}, in addition to methods that remove the internal LM's contribution prior to fusion \cite{McDermott19,Variani2020hat,Meng2021ILME}." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="1708.06426" id="8079">
        <attvalues>
          <attvalue for="5" value=" For static knowledge (eg, knowledge that does not change from one utterance to the next), various LM fusion techniques have been proposed \cite{gulcehre2015using,Kannan2018,sriram2018cold,toshniwal2018comparison,shan2019component,Kim2021lmfusion}, in addition to methods that remove the internal LM's contribution prior to fusion \cite{McDermott19,Variani2020hat,Meng2021ILME}." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="1807.10857" id="8080">
        <attvalues>
          <attvalue for="5" value=" For static knowledge (eg, knowledge that does not change from one utterance to the next), various LM fusion techniques have been proposed \cite{gulcehre2015using,Kannan2018,sriram2018cold,toshniwal2018comparison,shan2019component,Kim2021lmfusion}, in addition to methods that remove the internal LM's contribution prior to fusion \cite{McDermott19,Variani2020hat,Meng2021ILME}." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="2010.13878" id="8081">
        <attvalues>
          <attvalue for="5" value=" For static knowledge (eg, knowledge that does not change from one utterance to the next), various LM fusion techniques have been proposed \cite{gulcehre2015using,Kannan2018,sriram2018cold,toshniwal2018comparison,shan2019component,Kim2021lmfusion}, in addition to methods that remove the internal LM's contribution prior to fusion \cite{McDermott19,Variani2020hat,Meng2021ILME}." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="2002.11268" id="8082">
        <attvalues>
          <attvalue for="5" value=" For static knowledge (eg, knowledge that does not change from one utterance to the next), various LM fusion techniques have been proposed \cite{gulcehre2015using,Kannan2018,sriram2018cold,toshniwal2018comparison,shan2019component,Kim2021lmfusion}, in addition to methods that remove the internal LM's contribution prior to fusion \cite{McDermott19,Variani2020hat,Meng2021ILME}." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="2003.07705" id="8083">
        <attvalues>
          <attvalue for="5" value=" For static knowledge (eg, knowledge that does not change from one utterance to the next), various LM fusion techniques have been proposed \cite{gulcehre2015using,Kannan2018,sriram2018cold,toshniwal2018comparison,shan2019component,Kim2021lmfusion}, in addition to methods that remove the internal LM's contribution prior to fusion \cite{McDermott19,Variani2020hat,Meng2021ILME}." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="2011.01991" id="8084">
        <attvalues>
          <attvalue for="5" value=" For static knowledge (eg, knowledge that does not change from one utterance to the next), various LM fusion techniques have been proposed \cite{gulcehre2015using,Kannan2018,sriram2018cold,toshniwal2018comparison,shan2019component,Kim2021lmfusion}, in addition to methods that remove the internal LM's contribution prior to fusion \cite{McDermott19,Variani2020hat,Meng2021ILME}." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="2011.07754" id="8085">
        <attvalues>
          <attvalue for="5" value=" Unlike previous work which only contextualized the end-to-end ASR model \cite{Pundak2018DC,Chen2019DC,Jain2020DC,Le2021deepshallow}, we propose to fuse the NNLM with trie-based deep biasing to give the former implicit access to biasing information and leverage the vast amount of unpaired text data." />
        </attvalues>
      </edge>
      <edge source="2104.02194" target="2006.03411" id="8087">
        <attvalues>
          <attvalue for="5" value=" Unlike previous work which only contextualized the end-to-end ASR model \cite{Pundak2018DC,Chen2019DC,Jain2020DC,Le2021deepshallow}, we propose to fuse the NNLM with trie-based deep biasing to give the former implicit access to biasing information and leverage the vast amount of unpaired text data." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="1811.06621" id="8088">
        <attvalues>
          <attvalue for="5" value=" End-to-end (E2E) automatic speech recognition (ASR) is gaining popularity due to its simple model structure, high training efficiency, and astounding performance across many tasks\cite{asr0,asr1}." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2004.11544" id="8089">
        <attvalues>
          <attvalue for="5" value=" End-to-end (E2E) automatic speech recognition (ASR) is gaining popularity due to its simple model structure, high training efficiency, and astounding performance across many tasks\cite{asr0,asr1}." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2102.07739" id="8090">
        <attvalues>
          <attvalue for="5" value=" The first approach is shallow fusion\cite{gourav2021personalization,lm0,lm1,lm2}, which fuses the task-specific external language model with the ASR model and boosts the scores of biased words during decoding." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2011.01991" id="8092">
        <attvalues>
          <attvalue for="5" value=" In daily life, due to the diversity of scenarios, directly deploying an ASR model trained on one specific dataset to other domains will cause the problem of domain mismatch\cite{gourav2021personalization,mismatch0,mismatch2,mismatch3,mismatch100}." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="1712.01996" id="8094">
        <attvalues>
          <attvalue for="5" value=" The first approach is shallow fusion\cite{gourav2021personalization,lm0,lm1,lm2}, which fuses the task-specific external language model with the ASR model and boosts the scores of biased words during decoding." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2011.07754" id="8095">
        <attvalues>
          <attvalue for="5" value=" The first approach is shallow fusion\cite{gourav2021personalization,lm0,lm1,lm2}, which fuses the task-specific external language model with the ASR model and boosts the scores of biased words during decoding." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2201.12806" id="8096">
        <attvalues>
          <attvalue for="5" value=" The latter approach employs an end-to-end model structure that co-trains the contextual module as part of the entire ASR model, making it impractical to regulate the degree of biasing in inference\cite{nn0,recent0,recent1}." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2110.02220" id="8097">
        <attvalues>
          <attvalue for="5" value=" Additionally, this approach tends to choose the recurrent neural network transducer (RNN-T)\cite{nn1,nn2} as the base model." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2111.01690" id="8098">
        <attvalues>
          <attvalue for="5" value=" The second approach is integrating the contextual module with the ASR model\cite{nn0,nn1} in an all-neural network to incorporate contextual information, leveraging the powerful modeling expressiveness of E2E neural networks\cite{li2022recent}." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2104.05544" id="8099">
        <attvalues>
          <attvalue for="5" value=" In addition, the traditional language model maintains a consistent emphasis on biased words and is incapable of altering the weights of biased words, resulting in a poor prediction performance for biased words\cite{lm5,lm_33}." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2104.11127" id="8100">
        <attvalues>
          <attvalue for="5" value=" However, the former approach suffers from the adaptation problem, where the traditional language model conflicts with the internal language model\cite{ilm,lm44} in the ASR model due to the inconsistency of the trained domains." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2202.11134" id="8102">
        <attvalues>
          <attvalue for="5" value=" The latter approach employs an end-to-end model structure that co-trains the contextual module as part of the entire ASR model, making it impractical to regulate the degree of biasing in inference\cite{nn0,recent0,recent1}." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2202.09167" id="8103">
        <attvalues>
          <attvalue for="5" value=" The latter approach employs an end-to-end model structure that co-trains the contextual module as part of the entire ASR model, making it impractical to regulate the degree of biasing in inference\cite{nn0,recent0,recent1}." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2205.13660" id="8104">
        <attvalues>
          <attvalue for="5" value=" Additionally, this approach tends to choose the recurrent neural network transducer (RNN-T)\cite{nn1,nn2} as the base model." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2005.08100" id="8105">
        <attvalues>
          <attvalue for="5" value=" In this work, we propose CB-Conformer to solve the problem of biased word recognition by augmenting the Conformer-Encoder\cite{gulati2020conformer} with a Contextual Biasing Module and a Self-Adaptive Language Model." />
        </attvalues>
      </edge>
      <edge source="2304.09607" target="2110.03370" id="8106">
        <attvalues>
          <attvalue for="5" value=" We provide three specific subsets filtered from the WenetSpeech\cite{wenetspeech} dataset: the person-name dataset, the place-name dataset, and the organization-name dataset." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1706.03762" id="8107">
        <attvalues>
          <attvalue for="5" value=" Over the past few years, we have witnessed a revolution of machine learning in the domain of Natural Language Processing (NLP) \cite{vaswani2017attention,devlin2018bert}." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1704.05426" id="8110">
        <attvalues>
          <attvalue for="5" value=" Perhaps motivated by needs and competitions (eg GLUE \cite{wang2018glue}), novel and powerful proposals are made every day by public and private laboratories around the world to solve several complex NLP tasks: Natural Language Inference \cite{williams2017broad,levesque2012winograd}, Sentence Similarity and Paraphrasing \cite{dolan2005automatically,agirre2012semeval}, Text Classification \cite{socher2013recursive,warstadt2018neural}, Reading Comprehension/Question Answering (QA) \cite{rajpurkar2016squad,nguyen2016ms,lai2017race,joshi2017triviaqa}." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1805.12471" id="8111">
        <attvalues>
          <attvalue for="5" value=" Perhaps motivated by needs and competitions (eg GLUE \cite{wang2018glue}), novel and powerful proposals are made every day by public and private laboratories around the world to solve several complex NLP tasks: Natural Language Inference \cite{williams2017broad,levesque2012winograd}, Sentence Similarity and Paraphrasing \cite{dolan2005automatically,agirre2012semeval}, Text Classification \cite{socher2013recursive,warstadt2018neural}, Reading Comprehension/Question Answering (QA) \cite{rajpurkar2016squad,nguyen2016ms,lai2017race,joshi2017triviaqa}." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1606.05250" id="8112">
        <attvalues>
          <attvalue for="5" value=" For instance, trained on the SQuAD dataset \cite{rajpurkar2016squad}, language models such as BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta} or XLNet \cite{yang2019xlnet}, have shown a great ability in identifying the answer to a question in a given unstructured source of information (raw text)." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1704.04683" id="8113">
        <attvalues>
          <attvalue for="5" value=" Perhaps motivated by needs and competitions (eg GLUE \cite{wang2018glue}), novel and powerful proposals are made every day by public and private laboratories around the world to solve several complex NLP tasks: Natural Language Inference \cite{williams2017broad,levesque2012winograd}, Sentence Similarity and Paraphrasing \cite{dolan2005automatically,agirre2012semeval}, Text Classification \cite{socher2013recursive,warstadt2018neural}, Reading Comprehension/Question Answering (QA) \cite{rajpurkar2016squad,nguyen2016ms,lai2017race,joshi2017triviaqa}." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1705.03551" id="8114">
        <attvalues>
          <attvalue for="5" value=" Perhaps motivated by needs and competitions (eg GLUE \cite{wang2018glue}), novel and powerful proposals are made every day by public and private laboratories around the world to solve several complex NLP tasks: Natural Language Inference \cite{williams2017broad,levesque2012winograd}, Sentence Similarity and Paraphrasing \cite{dolan2005automatically,agirre2012semeval}, Text Classification \cite{socher2013recursive,warstadt2018neural}, Reading Comprehension/Question Answering (QA) \cite{rajpurkar2016squad,nguyen2016ms,lai2017race,joshi2017triviaqa}." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1907.11692" id="8115">
        <attvalues>
          <attvalue for="5" value=" For instance, trained on the SQuAD dataset \cite{rajpurkar2016squad}, language models such as BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta} or XLNet \cite{yang2019xlnet}, have shown a great ability in identifying the answer to a question in a given unstructured source of information (raw text)." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1908.01519" id="8117">
        <attvalues>
          <attvalue for="5" value=" An alternative direction is transfer learning and in particular, zero-shot transfer \cite{hardalov2019beyond,liu2019xqa} when there is no target data." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1606.04164" id="8118">
        <attvalues>
          <attvalue for="5" value=" On the one hand, many specific strategies have been proposed for zero-shot transfer with explicit language alignment \cite{firat2016zero,johnson2017google,asai2018multilingual}." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="1809.03275" id="8119">
        <attvalues>
          <attvalue for="5" value=" On the one hand, many specific strategies have been proposed for zero-shot transfer with explicit language alignment \cite{firat2016zero,johnson2017google,asai2018multilingual}." />
        </attvalues>
      </edge>
      <edge source="1910.04659" target="2002.06071" id="8120">
        <attvalues>
          <attvalue for="5" value=" We then show that we can further improve the results using &quot;virtual&quot; target data obtained with machine translation and end up with a performance competitive to the one that we would have with target data annotated by humans \cite{fquad}." />
        </attvalues>
      </edge>
      <edge source="2406.03487" target="2209.12356" id="8121">
        <attvalues>
          <attvalue for="5" value=" In the news domain, prior work has found that LLM-generated summaries have fewer inconsistencies \cite{goyal2022news, zhang2023benchmarking}." />
        </attvalues>
      </edge>
      <edge source="2406.03487" target="2301.13848" id="8122">
        <attvalues>
          <attvalue for="5" value=" Surprisingly, our analysis reveals that over 30\ of LLM-generated summaries contain inconsistencies, contrasting sharply with the inconsistency rate of less than 5\% in GPT-generated news summaries \cite{zhang2023benchmarking}." />
        </attvalues>
      </edge>
      <edge source="2406.03487" target="2305.16548" id="8123">
        <attvalues>
          <attvalue for="5" value=" Dialogue summarization remain less explored, with prior works mostly focused on smaller fine-tuned models \cite{zhu2023annotating, gao2023reference, wang2022analyzing}." />
        </attvalues>
      </edge>
      <edge source="2406.03487" target="1911.12237" id="8124">
        <attvalues>
          <attvalue for="5" value=" We obtain fine-grained inconsistency annotations for summaries generated (zero-shot) by two prominent LLMs (GPT-4 \cite{luo2023chatgpt} and Alpaca-13B \cite{alpaca}) and across two summarization datasets (SAMSum \cite{gliwa2019samsum} and DialogSum \cite{chen2021dialogsum})." />
        </attvalues>
      </edge>
      <edge source="2406.03487" target="2105.06762" id="8125">
        <attvalues>
          <attvalue for="5" value=" We obtain fine-grained inconsistency annotations for summaries generated (zero-shot) by two prominent LLMs (GPT-4 \cite{luo2023chatgpt} and Alpaca-13B \cite{alpaca}) and across two summarization datasets (SAMSum \cite{gliwa2019samsum} and DialogSum \cite{chen2021dialogsum})." />
        </attvalues>
      </edge>
      <edge source="2406.03487" target="2205.12854" id="8126">
        <attvalues>
          <attvalue for="5" value=" Work done by \cite{tang2022understanding}, also in the news domain, notes varying error distributions across different model categories." />
        </attvalues>
      </edge>
      <edge source="2406.03487" target="2210.11777" id="8127">
        <attvalues>
          <attvalue for="5" value=" Previous work has primarily relied on part-of-speech-based tags for error classification \cite{wang2022analyzing, zhu2023annotating, gao2023reference}." />
        </attvalues>
      </edge>
      <edge source="2409.02050" target="2002.02562" id="8137">
        <attvalues>
          <attvalue for="5" value=" End-to-end speech recognition systems have gained popularity in recent years due to their simple training process and outstanding performance \cite{ctc, attention, LAS, joint,transformer-transducer,speech-transformer}." />
        </attvalues>
      </edge>
      <edge source="2409.02050" target="1906.08647" id="8138">
        <attvalues>
          <attvalue for="5" value=" CS corpus synthesis \cite{TTS1, TTS2, TTS3} and self-supervised speech representation learning \cite{self-sup1,self-sup3,self-sup2} alleviate this data scarcity problem by synthesizing corpus or utilizing unlabeled data." />
        </attvalues>
      </edge>
      <edge source="2409.02050" target="1810.09699" id="8139">
        <attvalues>
          <attvalue for="5" value=" CS corpus synthesis \cite{TTS1, TTS2, TTS3} and self-supervised speech representation learning \cite{self-sup1,self-sup3,self-sup2} alleviate this data scarcity problem by synthesizing corpus or utilizing unlabeled data." />
        </attvalues>
      </edge>
      <edge source="2409.02050" target="1806.06200" id="8140">
        <attvalues>
          <attvalue for="5" value=" CS corpus synthesis \cite{TTS1, TTS2, TTS3} and self-supervised speech representation learning \cite{self-sup1,self-sup3,self-sup2} alleviate this data scarcity problem by synthesizing corpus or utilizing unlabeled data." />
        </attvalues>
      </edge>
      <edge source="2409.02050" target="1909.05330" id="8141">
        <attvalues>
          <attvalue for="5" value=" Cross-language transfer learning \cite{qianyi1,qianyi2,qianyi3} also effectively alleviates the data scarcity problem by utilizing monolingual corpus to improve performance in CS scenarios." />
        </attvalues>
      </edge>
      <edge source="2409.02050" target="2007.03001" id="8142">
        <attvalues>
          <attvalue for="5" value=" Therefore, effectively modeling diverse languages within a unified architecture presents a significant challenge \cite{degrade,youxiaojianmo1,youxiaojianmo2}." />
        </attvalues>
      </edge>
      <edge source="2409.02050" target="2211.01458" id="8143">
        <attvalues>
          <attvalue for="5" value=" Therefore, effectively modeling diverse languages within a unified architecture presents a significant challenge \cite{degrade,youxiaojianmo1,youxiaojianmo2}." />
        </attvalues>
      </edge>
      <edge source="2409.02050" target="1904.00784" id="8144">
        <attvalues>
          <attvalue for="5" value=" Therefore, effectively modeling diverse languages within a unified architecture presents a significant challenge \cite{degrade,youxiaojianmo1,youxiaojianmo2}." />
        </attvalues>
      </edge>
      <edge source="2304.07987" target="2301.07597" id="8146">
        <attvalues>
          <attvalue for="5" value=" Moreover, since COIG translated corpus is translated from English instruction corpora~\cite{supernaturalinstructions,honovich2022unnaturalinstructions,selfinstruct} with diverse tasks, it is much more diverse than Chinese instruction corpora built by adapting prompt engineering on existing Chinese datasets, eg \cite{zeng2023glm-130b,Firefly,hc3}." />
        </attvalues>
      </edge>
      <edge source="2304.07987" target="2109.01652" id="8147">
        <attvalues>
          <attvalue for="5" value=" Pre-trained large-scale language models (LLMs) have shown revolutionary performance in many downstream tasks~\cite{hc3,flan}." />
        </attvalues>
      </edge>
      <edge source="2304.07987" target="2301.13688" id="8148">
        <attvalues>
          <attvalue for="5" value=" For example, the FLAN collection~\cite{flancollection} contains 15M examples covering 1836 tasks, and OPT-IML~\cite{iyer2022opt} claims to have 18M examples for more than 2000 tasks (although it is still not publicly available)." />
        </attvalues>
      </edge>
      <edge source="2304.07987" target="2212.12017" id="8149">
        <attvalues>
          <attvalue for="5" value=" For example, the FLAN collection~\cite{flancollection} contains 15M examples covering 1836 tasks, and OPT-IML~\cite{iyer2022opt} claims to have 18M examples for more than 2000 tasks (although it is still not publicly available)." />
        </attvalues>
      </edge>
      <edge source="2304.07987" target="2212.09689" id="8150">
        <attvalues>
          <attvalue for="5" value=" Moreover, since COIG translated corpus is translated from English instruction corpora~\cite{supernaturalinstructions,honovich2022unnaturalinstructions,selfinstruct} with diverse tasks, it is much more diverse than Chinese instruction corpora built by adapting prompt engineering on existing Chinese datasets, eg \cite{zeng2023glm-130b,Firefly,hc3}." />
        </attvalues>
      </edge>
      <edge source="2304.07987" target="2210.02414" id="8151">
        <attvalues>
          <attvalue for="5" value=" Moreover, since COIG translated corpus is translated from English instruction corpora~\cite{supernaturalinstructions,honovich2022unnaturalinstructions,selfinstruct} with diverse tasks, it is much more diverse than Chinese instruction corpora built by adapting prompt engineering on existing Chinese datasets, eg \cite{zeng2023glm-130b,Firefly,hc3}." />
        </attvalues>
      </edge>
      <edge source="2402.02302" target="2301.07295" id="8153">
        <attvalues>
          <attvalue for="5" value="~While recent studies have shown the effectiveness of continued pre-training to adapt these models to the target language~\cite{NOWAKOWSKI2023103148,10301554}, they involved using 70--200 hours of target language data." />
        </attvalues>
      </edge>
      <edge source="2402.02302" target="2205.01086" id="8154">
        <attvalues>
          <attvalue for="5" value="~To account for the text-/token-less nature of untranscribed speech corpora, we induce them in a bottom-up manner using wav2seq \cite{10096988}, a method for inducing pseudo-tokens using pre-trained speech embeddings." />
        </attvalues>
      </edge>
      <edge source="2307.11779" target="1910.13461" id="8156">
        <attvalues>
          <attvalue for="5" value=" To produce these abstractive responses, generative models \cite{bart, t5} synthesize information from multiple sources/text documents using sequence-to-sequence LLMs such that the generated answers may be highly abstractive or otherwise not readily attributable -- as they are in search engines -- to a specific content source such as a document on the web with a unique URI identifier ." />
        </attvalues>
      </edge>
      <edge source="2210.09683" target="1706.03762" id="8157">
        <attvalues>
          <attvalue for="5" value=" As the development of neural machine translation research~\cite{vaswani2017attention,wei-etal-2022-learning}, the metric methods should be capable of evaluating the high-quality translations at the level of semantics rather than surfance-level features~\cite{sellam-etal-2020-bleurt,ranasinghe-etal-2020-transquest,rei-etal-2020-comet,wan-etal-2022-unite}." />
        </attvalues>
      </edge>
      <edge source="2210.09683" target="2204.06812" id="8158">
        <attvalues>
          <attvalue for="5" value=" As the development of neural machine translation research~\cite{vaswani2017attention,wei-etal-2022-learning}, the metric methods should be capable of evaluating the high-quality translations at the level of semantics rather than surfance-level features~\cite{sellam-etal-2020-bleurt,ranasinghe-etal-2020-transquest,rei-etal-2020-comet,wan-etal-2022-unite}." />
        </attvalues>
      </edge>
      <edge source="2210.09683" target="2004.04696" id="8159">
        <attvalues>
          <attvalue for="5" value=" Compared to conventional statistical- (eg, BLEU,~\cite{papineni-etal-2002-bleu} and representation-based methods (eg,~BERTScore,~\cite{zhang2019bertscore}), the model-based approaches (eg,~\cite[BLEURT,][]{sellam-etal-2020-bleurt};~\cite[COMET,][]{rei-etal-2020-comet};~\cite[UniTE,][]{wan-etal-2022-unite}) show their strong ability on delivering more accurate quality predictions, especially those approaches which apply source sentences as additional input for the metric model~\cite{rei-etal-2020-comet,takahashi-etal-2020-automatic,wan-etal-2021-robleurt,wan-etal-2022-unite}." />
        </attvalues>
      </edge>
      <edge source="2210.09683" target="2011.01536" id="8160">
        <attvalues>
          <attvalue for="5" value=" As the development of neural machine translation research~\cite{vaswani2017attention,wei-etal-2022-learning}, the metric methods should be capable of evaluating the high-quality translations at the level of semantics rather than surfance-level features~\cite{sellam-etal-2020-bleurt,ranasinghe-etal-2020-transquest,rei-etal-2020-comet,wan-etal-2022-unite}." />
        </attvalues>
      </edge>
      <edge source="2210.09683" target="2009.09025" id="8161">
        <attvalues>
          <attvalue for="5" value=" Compared to conventional statistical- (eg, BLEU,~\cite{papineni-etal-2002-bleu} and representation-based methods (eg,~BERTScore,~\cite{zhang2019bertscore}), the model-based approaches (eg,~\cite[BLEURT,][]{sellam-etal-2020-bleurt};~\cite[COMET,][]{rei-etal-2020-comet};~\cite[UniTE,][]{wan-etal-2022-unite}) show their strong ability on delivering more accurate quality predictions, especially those approaches which apply source sentences as additional input for the metric model~\cite{rei-etal-2020-comet,takahashi-etal-2020-automatic,wan-etal-2021-robleurt,wan-etal-2022-unite}." />
        </attvalues>
      </edge>
      <edge source="2210.09683" target="1911.02116" id="8164">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) like BERT~\cite{devlin-etal-2019-bert} and XLM-R~\cite{conneau-etal-2020-unsupervised} have shown promising results in identifying the quality of translation outputs." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="1907.11692" id="8168">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) such as BERT \cite{devlin2018bert}, RoBERTA \cite{liu2019roberta}, T5 \cite{raffel2020exploring}, and PaLM \cite{chowdhery2022palm}, are intricately designed architectures equipped with an extensive number of parameters." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2204.02311" id="8170">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) such as BERT \cite{devlin2018bert}, RoBERTA \cite{liu2019roberta}, T5 \cite{raffel2020exploring}, and PaLM \cite{chowdhery2022palm}, are intricately designed architectures equipped with an extensive number of parameters." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2205.10770" id="8171">
        <attvalues>
          <attvalue for="5" value=" These models have been rigorously pre-trained on vast and diverse corpora, thereby enabling them to excel in a wide array of Natural Language Processing (NLP) tasks, from language understanding to both conditional and unconditional text generation \cite{tirumala2022memorization, zhou2022learning}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="1908.04319" id="8172">
        <attvalues>
          <attvalue for="5" value=" On one hand, LLMs exhibit a tendency for 'hallucinations' \cite{welleck2019neural, ji2023survey}, providing plausible yet nonfactual predictions." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="1909.01066" id="8173">
        <attvalues>
          <attvalue for="5" value=" On the other hand, the black-box nature of LLMs compromises both interpretability and factual accuracy, often resulting in erroneous statements despite memorizing facts during training \cite{petroni2019language, scialom-etal-2022-fine}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2302.07842" id="8174">
        <attvalues>
          <attvalue for="5" value=" Knowledge in natural language can be externally sourced from a retrievable database, reducing hallucinations and enhancing the interpretability of LLMs \cite{mialon2023augmented}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2107.11976" id="8175">
        <attvalues>
          <attvalue for="5" value=" Utilizing dense neural retrievers, which employ dense query and document vectors generated by a neural network \cite{asai2021one}, the system can evaluate the semantic similarity to an information-seeking query by calculating the embedding vector similarity across related concepts \cite{lewis2020retrieval, luan2021sparse}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2005.11401" id="8176">
        <attvalues>
          <attvalue for="5" value=" Utilizing dense neural retrievers, which employ dense query and document vectors generated by a neural network \cite{asai2021one}, the system can evaluate the semantic similarity to an information-seeking query by calculating the embedding vector similarity across related concepts \cite{lewis2020retrieval, luan2021sparse}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2306.08302" id="8177">
        <attvalues>
          <attvalue for="5" value=" Most of the works related to these two tasks rely intensively on the ongoing training of neural networks \cite{pan2023unifying, zhong2023comprehensive}, which is both difficult to employ and less flexible for on-the-fly updates." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="1905.07098" id="8179">
        <attvalues>
          <attvalue for="5" value=" As for the latter, KGs offer LLMs a structured and efficient way to address their limitations in factual accuracy and reasoning \cite{xiong2019improving, pan2023unifying}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2002.00388" id="8180">
        <attvalues>
          <attvalue for="5" value=" KGs not only provide accurate and explicit knowledge crucial for various applications \cite{ji2021survey} but are also known for their symbolic reasoning capabilities to produce interpretable results \cite{zhang2021neural}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2010.05446" id="8181">
        <attvalues>
          <attvalue for="5" value=" KGs not only provide accurate and explicit knowledge crucial for various applications \cite{ji2021survey} but are also known for their symbolic reasoning capabilities to produce interpretable results \cite{zhang2021neural}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2011.00235" id="8182">
        <attvalues>
          <attvalue for="5" value=" These graphs are dynamic, continuously evolving with the addition of new knowledge \cite{mitchell2018never}, and can be specialized for domain-specific requirements \cite{abu2021domain}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2304.08183" id="8183">
        <attvalues>
          <attvalue for="5" value=" Current automated techniques necessitate neural network training \cite{luo2023normalizing,wan2021reasoning, wang2023knowledge}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2309.15427" id="8184">
        <attvalues>
          <attvalue for="5" value=" As for the interaction between KGs and LLMs, neural networks are trained to let LLMs understand the information retrieved from KGs \cite{tian2023graph,yasunaga2022deep}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2210.09338" id="8185">
        <attvalues>
          <attvalue for="5" value=" As for the interaction between KGs and LLMs, neural networks are trained to let LLMs understand the information retrieved from KGs \cite{tian2023graph,yasunaga2022deep}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2307.09288" id="8186">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art LLMs such as ChatGPT, BARD, and LLAMA\cite{touvron2023llama} have demonstrated impressive reasoning capabilities \cite{bang2023multitask, agarwal2023analysing}." />
        </attvalues>
      </edge>
      <edge source="2311.14740" target="2302.04023" id="8187">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art LLMs such as ChatGPT, BARD, and LLAMA\cite{touvron2023llama} have demonstrated impressive reasoning capabilities \cite{bang2023multitask, agarwal2023analysing}." />
        </attvalues>
      </edge>
      <edge source="2107.11976" target="1906.00300" id="8188">
        <attvalues>
          <attvalue for="5" value=" Most recent progress in open QA is made for English by building a pipeline based on a dense passage retriever trained on large-scale English QA datasets to find evidence passages in English ~\cite{lee-chang-toutanova:2019:ACL2019,karpukhin2020dense}, followed by a reader that extracts an answer from retrieved passages." />
        </attvalues>
      </edge>
      <edge source="2107.11976" target="2004.04906" id="8189">
        <attvalues>
          <attvalue for="5" value=" Most recent progress in open QA is made for English by building a pipeline based on a dense passage retriever trained on large-scale English QA datasets to find evidence passages in English ~\cite{lee-chang-toutanova:2019:ACL2019,karpukhin2020dense}, followed by a reader that extracts an answer from retrieved passages." />
        </attvalues>
      </edge>
      <edge source="2107.11976" target="2010.11856" id="8190">
        <attvalues>
          <attvalue for="5" value=" Our experiments show that CORA advances the state of the art on two multilingual open QA datasets, Xor-TyDi QA \cite{xorqa} and MKQA \cite{mkqa}, across 26 typologically diverse languages; CORA achieves gains of 23." />
        </attvalues>
      </edge>
      <edge source="2107.11976" target="2010.06467" id="8191">
        <attvalues>
          <attvalue for="5" value=" Answering multilingual questions requires retrieving evidence from knowledge sources of other languages than the original question since many languages have limited reference documents or the question sometimes inquires about concepts from other cultures \cite{xorqa,lin2020pretrained}." />
        </attvalues>
      </edge>
      <edge source="2107.11976" target="1609.08210" id="8192">
        <attvalues>
          <attvalue for="5" value=" To address these challenges, previous work in multilingual open QA~\cite{ture-boschee-2016-learning,xorqa} translates questions into English, applies an English open QA system to answer in English, and then translates answers back to the target language." />
        </attvalues>
      </edge>
      <edge source="2107.11976" target="2005.11401" id="8193">
        <attvalues>
          <attvalue for="5" value="~\ref{img:method_overview}) extends the {retrieve-then-generate} approach of English open QA~\cite{lewis2020retrieval,izacard2020leveraging} with a single cross-lingual retriever and a generator that do not rely on language-specific retrievers or machine translation modules." />
        </attvalues>
      </edge>
      <edge source="2107.11976" target="2007.01282" id="8194">
        <attvalues>
          <attvalue for="5" value="~\ref{img:method_overview}) extends the {retrieve-then-generate} approach of English open QA~\cite{lewis2020retrieval,izacard2020leveraging} with a single cross-lingual retriever and a generator that do not rely on language-specific retrievers or machine translation modules." />
        </attvalues>
      </edge>
      <edge source="2107.11976" target="2007.15207" id="8195">
        <attvalues>
          <attvalue for="5" value=" Our experiments show that CORA advances the state of the art on two multilingual open QA datasets, Xor-TyDi QA \cite{xorqa} and MKQA \cite{mkqa}, across 26 typologically diverse languages; CORA achieves gains of 23." />
        </attvalues>
      </edge>
      <edge source="2306.17399" target="1804.09132" id="8197">
        <attvalues>
          <attvalue for="5" value=" This differentiates LCP from complex word identification (CWI), eg, binary classification of complex words \cite{Yimam2018-rw}." />
        </attvalues>
      </edge>
      <edge source="2405.10443" target="1706.03762" id="8198">
        <attvalues>
          <attvalue for="5" value=" Current literature has primarily focused on adapting end-to-end Transformer models \cite{vaswani-et-al-2017} to overcome the difficulties of simultaneous machine translation (SimulMT) due to their reduced parameter counts and greater inference speed\cite{ma2020simulmt}." />
        </attvalues>
      </edge>
      <edge source="2405.10443" target="2011.02048" id="8199">
        <attvalues>
          <attvalue for="5" value=" Current literature has primarily focused on adapting end-to-end Transformer models \cite{vaswani-et-al-2017} to overcome the difficulties of simultaneous machine translation (SimulMT) due to their reduced parameter counts and greater inference speed\cite{ma2020simulmt}." />
        </attvalues>
      </edge>
      <edge source="2405.10443" target="2307.09288" id="8200">
        <attvalues>
          <attvalue for="5" value=" However, the recent successes of large language models (LLMs) \cite{touvron2023llama, jiang2023mistral, almazrouei2023falcon} has prompted preliminary research applying them to SimulMT through fine-tuning and inference techniques \cite{agostinelli2023simul, wang2023simultaneous, koshkin2024transllama, wang2024conversational, guo2024sillm}." />
        </attvalues>
      </edge>
      <edge source="2405.10443" target="2312.04691" id="8201">
        <attvalues>
          <attvalue for="5" value=" Furthermore, there has yet to be a universal approach to fine-tuning LLMs for SimulMT that is not unnecessarily computationally expensive by either expanding the dataset through data augmentation, a process referred to as prefix fine-tuning \cite{agostinelli2023simul, wang2023simultaneous, koshkin2024transllama} or increasing the prompt length through prompt restructuring \cite{koshkin2024transllama, wang2024conversational}." />
        </attvalues>
      </edge>
      <edge source="2405.10443" target="2402.10552" id="8202">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, most modern works have neglected the computational increases created by dumping the target sequence's key and value (KV) cache \cite{wang2024conversational}." />
        </attvalues>
      </edge>
      <edge source="2405.10443" target="2402.04636" id="8203">
        <attvalues>
          <attvalue for="5" value=" Furthermore, there has yet to be a universal approach to fine-tuning LLMs for SimulMT that is not unnecessarily computationally expensive by either expanding the dataset through data augmentation, a process referred to as prefix fine-tuning \cite{agostinelli2023simul, wang2023simultaneous, koshkin2024transllama} or increasing the prompt length through prompt restructuring \cite{koshkin2024transllama, wang2024conversational}." />
        </attvalues>
      </edge>
      <edge source="2405.10443" target="2311.16867" id="8205">
        <attvalues>
          <attvalue for="5" value="3 billion parameter Falcon models pre-trained on the RefinedWeb dataset using SimulMask \cite{almazrouei2023falcon, refinedweb} and compared them against identical Falcon models that adopt existing prefix fine-tuning or prompt restructuring methods on the IWSLT 2017 dataset \cite{cettolo-etal-2017-overview}." />
        </attvalues>
      </edge>
      <edge source="1803.08493" target="1708.00107" id="8208">
        <attvalues>
          <attvalue for="5" value=" Further, these algorithms often still require thousands of sentences in a transfer dataset to train effectively, especially when unsupervised \cite{2017arXiv170800107M}." />
        </attvalues>
      </edge>
      <edge source="1803.08493" target="1802.05365" id="8209">
        <attvalues>
          <attvalue for="5" value=" Method outputs vary from a vector \cite{2017arXiv170800107M} to another deep structure \cite{2018arXiv180205365P}." />
        </attvalues>
      </edge>
      <edge source="2306.03975" target="1810.11118" id="8210">
        <attvalues>
          <attvalue for="5" value=" We conduct experiments on two benchmark datasets, including the Ubuntu IRC \cite{kummerfeld-etal-2019-large} and Movie Dialogue \cite{liuSGLWZ20-ijcai}." />
        </attvalues>
      </edge>
      <edge source="2306.03975" target="2210.15265" id="8211">
        <attvalues>
          <attvalue for="5" value=" Currently, the rapid development of deep neural models has greatly advanced the dialogue disentanglement task \cite{mehri-carenini-2017-chat,liuSGLWZ20-ijcai,jiang-etal-2018-identifying,chengyuhuang-emnlp}, especially by making use of the pre-trained language models (PLM) \cite{aaai-ZhuNWNX20,tiandali-arxiv}, eg, BERT \cite{devlin-etal-2019-bert}, DiaBERT \cite{tiandali-arxiv}." />
        </attvalues>
      </edge>
      <edge source="2306.03975" target="1911.10666" id="8212">
        <attvalues>
          <attvalue for="5" value=" Currently, the rapid development of deep neural models has greatly advanced the dialogue disentanglement task \cite{mehri-carenini-2017-chat,liuSGLWZ20-ijcai,jiang-etal-2018-identifying,chengyuhuang-emnlp}, especially by making use of the pre-trained language models (PLM) \cite{aaai-ZhuNWNX20,tiandali-arxiv}, eg, BERT \cite{devlin-etal-2019-bert}, DiaBERT \cite{tiandali-arxiv}." />
        </attvalues>
      </edge>
      <edge source="2306.03975" target="2004.03760" id="8213">
        <attvalues>
          <attvalue for="5" value=" Currently, the rapid development of deep neural models has greatly advanced the dialogue disentanglement task \cite{mehri-carenini-2017-chat,liuSGLWZ20-ijcai,jiang-etal-2018-identifying,chengyuhuang-emnlp}, especially by making use of the pre-trained language models (PLM) \cite{aaai-ZhuNWNX20,tiandali-arxiv}, eg, BERT \cite{devlin-etal-2019-bert}, DiaBERT \cite{tiandali-arxiv}." />
        </attvalues>
      </edge>
      <edge source="2306.03975" target="2112.05346" id="8215">
        <attvalues>
          <attvalue for="5" value=" As extensively revealed, the essence of the task lies in the understanding of the underlying conversational discourse \cite{zhu-etal-2021-findings,aclMa0Z22}, and thus it is key to model the discourse structure of the dialogue." />
        </attvalues>
      </edge>
      <edge source="2306.03975" target="2110.08018" id="8216">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, most current models only train with the pair-wise cross-entropy loss without considering the higher-level optimization (eg, thread, session) \cite{liuSGLWZ20-ijcai,jiang-etal-2018-identifying,aclMa0Z22}." />
        </attvalues>
      </edge>
      <edge source="2305.10433" target="1703.04009" id="8217">
        <attvalues>
          <attvalue for="5" value=" Determining what is toxic or harmful in a language is a very subjective task, as perceived toxicity varies based on many different characteristics ~\cite{Davidson2017ai}." />
        </attvalues>
      </edge>
      <edge source="2305.10433" target="1705.09899" id="8218">
        <attvalues>
          <attvalue for="5" value=" For example, ~\cite{Waseem2017-vg} proposed a typology that differentiates between direct/indirect abusive language toward specific individuals or groups." />
        </attvalues>
      </edge>
      <edge source="2305.10433" target="2107.08720" id="8219">
        <attvalues>
          <attvalue for="5" value=" Specifically for hate speech, the research of ~\cite{Fanton2021-bc} focused on covariate shifts (modified text) in the human-in-the-loop pipeline, demonstrating that assessing covariate shifting is beneficial to addressing the shortcomings of extant collection strategies that grant either quality or quantity, but not both ~\cite{Bhatt2021-em}." />
        </attvalues>
      </edge>
      <edge source="2305.10433" target="2111.07997" id="8220">
        <attvalues>
          <attvalue for="5" value=" Current toxicity detectors' reliability and robustness suffer from high subjectivity and bias towards keywords ~\cite{Sap2022-lj}." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2005.14165" id="8229">
        <attvalues>
          <attvalue for="5" value=" Generative (decoder-only) large language models (LLMs) such as GPT models \cite{gpt3_few_shot,openai2023gpt4}, PaLM \cite{palm}, OPT \cite{OPT}, BLOOM \cite{bloom}, LLaMA \cite{llama1,llama2}, and others have exhibited remarkable capabilities across various NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2303.08774" id="8230">
        <attvalues>
          <attvalue for="5" value=" Generative (decoder-only) large language models (LLMs) such as GPT models \cite{gpt3_few_shot,openai2023gpt4}, PaLM \cite{palm}, OPT \cite{OPT}, BLOOM \cite{bloom}, LLaMA \cite{llama1,llama2}, and others have exhibited remarkable capabilities across various NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2204.02311" id="8231">
        <attvalues>
          <attvalue for="5" value=" Generative (decoder-only) large language models (LLMs) such as GPT models \cite{gpt3_few_shot,openai2023gpt4}, PaLM \cite{palm}, OPT \cite{OPT}, BLOOM \cite{bloom}, LLaMA \cite{llama1,llama2}, and others have exhibited remarkable capabilities across various NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2205.01068" id="8232">
        <attvalues>
          <attvalue for="5" value=" Generative (decoder-only) large language models (LLMs) such as GPT models \cite{gpt3_few_shot,openai2023gpt4}, PaLM \cite{palm}, OPT \cite{OPT}, BLOOM \cite{bloom}, LLaMA \cite{llama1,llama2}, and others have exhibited remarkable capabilities across various NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2211.05100" id="8233">
        <attvalues>
          <attvalue for="5" value=" Generative (decoder-only) large language models (LLMs) such as GPT models \cite{gpt3_few_shot,openai2023gpt4}, PaLM \cite{palm}, OPT \cite{OPT}, BLOOM \cite{bloom}, LLaMA \cite{llama1,llama2}, and others have exhibited remarkable capabilities across various NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2307.09288" id="8235">
        <attvalues>
          <attvalue for="5" value=" Generative (decoder-only) large language models (LLMs) such as GPT models \cite{gpt3_few_shot,openai2023gpt4}, PaLM \cite{palm}, OPT \cite{OPT}, BLOOM \cite{bloom}, LLaMA \cite{llama1,llama2}, and others have exhibited remarkable capabilities across various NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2207.04672" id="8236">
        <attvalues>
          <attvalue for="5" value="5 and GPT-4 can rival the supervised encoder-decoder state-of-the-art (SoTA) models like NLLB \cite{nllb}, while they still fall short in translation for low-resource languages \cite{gpt_mt,jiao2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2302.09210" id="8237">
        <attvalues>
          <attvalue for="5" value="5 and GPT-4 can rival the supervised encoder-decoder state-of-the-art (SoTA) models like NLLB \cite{nllb}, while they still fall short in translation for low-resource languages \cite{gpt_mt,jiao2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2304.04675" id="8238">
        <attvalues>
          <attvalue for="5" value="3B by a substantial 30 BLEU points \cite{zhu2023multilingual}." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2307.04408" id="8240">
        <attvalues>
          <attvalue for="5" value=" Recent research has sought to enhance translation performance by commencing with smaller LLMs \cite{bigtranslate,zeng2023tim,swie,zhu2023extrapolating,li2023eliciting,bayling}, especially 7B or 13B parameters." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2308.12674" id="8241">
        <attvalues>
          <attvalue for="5" value=" Recent research has sought to enhance translation performance by commencing with smaller LLMs \cite{bigtranslate,zeng2023tim,swie,zhu2023extrapolating,li2023eliciting,bayling}, especially 7B or 13B parameters." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2305.15083" id="8243">
        <attvalues>
          <attvalue for="5" value=" Recent research has sought to enhance translation performance by commencing with smaller LLMs \cite{bigtranslate,zeng2023tim,swie,zhu2023extrapolating,li2023eliciting,bayling}, especially 7B or 13B parameters." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2306.10968" id="8244">
        <attvalues>
          <attvalue for="5" value=" As depicted in Figure \ref{fig:intro}, contemporary studies such as Balyling \cite{bayling} and BigTranslate \cite{bigtranslate}, which use LLaMA as their backbone, exhibit a maximum increment of 3 to 4 BLEU or COMET in relation to the zero-shot performance of LLaMA on the WMT'22 test set (8 directions)." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2305.11206" id="8245">
        <attvalues>
          <attvalue for="5" value=" Secondly, drawing inspiration from the recognized significance of data quality in other applications \cite{zhou2023lima,maillard2023small,gunasekar2023textbooks}, we fine-tune the model with a small amount of high-quality parallel data." />
        </attvalues>
      </edge>
      <edge source="2309.11674" target="2306.11644" id="8246">
        <attvalues>
          <attvalue for="5" value=" Secondly, drawing inspiration from the recognized significance of data quality in other applications \cite{zhou2023lima,maillard2023small,gunasekar2023textbooks}, we fine-tune the model with a small amount of high-quality parallel data." />
        </attvalues>
      </edge>
      <edge source="2309.08357" target="2202.00874" id="8247">
        <attvalues>
          <attvalue for="5" value=" Recently, many approaches have achieved notable success on audio classification tasks with the emergence of large-scale audio datasets \cite{gemmeke2017audioset}, powerful network architectures \cite{chen2022hts}, and effective learning theories \cite{xiao2023semanticac}." />
        </attvalues>
      </edge>
      <edge source="2309.08357" target="2207.09519" id="8251">
        <attvalues>
          <attvalue for="5" value=" In pursuit of better performance under data scarcity, methods such as Treff adapter \cite{liang23treff} and TIP adapter \cite{zhang2022tip} utilize a key-value cache model from the training set and update the prior knowledge in CLAP by feature retrieval, which can achieve better performance than traditional few-shot algorithms, such as ProtoNet \cite{snell2017protonet}." />
        </attvalues>
      </edge>
      <edge source="2309.08357" target="1703.05175" id="8252">
        <attvalues>
          <attvalue for="5" value=" In pursuit of better performance under data scarcity, methods such as Treff adapter \cite{liang23treff} and TIP adapter \cite{zhang2022tip} utilize a key-value cache model from the training set and update the prior knowledge in CLAP by feature retrieval, which can achieve better performance than traditional few-shot algorithms, such as ProtoNet \cite{snell2017protonet}." />
        </attvalues>
      </edge>
      <edge source="2309.08357" target="2109.01134" id="8253">
        <attvalues>
          <attvalue for="5" value=" Prompt tuning, explored in vision-language tasks \cite{zhou2022coop, khattak2023maple}, throws light on the above issue by converting the hand-crafted prompt template (eg, ``this is a sound of'') into learnable tokens while reserving the CLAP text encoder." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="1706.03762" id="8254">
        <attvalues>
          <attvalue for="5" value=" Performance of these frameworks demonstrate the capability of deep learning models like Transformer\cite{NIPS2017_7181,Zhou2018} to learn and express the knowledge in various modalities required for diverse speech tasks." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="1804.10752" id="8255">
        <attvalues>
          <attvalue for="5" value=" Performance of these frameworks demonstrate the capability of deep learning models like Transformer\cite{NIPS2017_7181,Zhou2018} to learn and express the knowledge in various modalities required for diverse speech tasks." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2305.16107" id="8257">
        <attvalues>
          <attvalue for="5" value=" Previous literature\cite{wang2023viola,rubenstein2023audiopalm,chen2023lauragpt} has analyzed the performance differences between the proposed multitask speech models and existing single-task models, but few studies compare these models under fair enough experimental conditions, such as same supervised training data." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2306.12925" id="8258">
        <attvalues>
          <attvalue for="5" value=" Previous literature\cite{wang2023viola,rubenstein2023audiopalm,chen2023lauragpt} has analyzed the performance differences between the proposed multitask speech models and existing single-task models, but few studies compare these models under fair enough experimental conditions, such as same supervised training data." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2310.04673" id="8259">
        <attvalues>
          <attvalue for="5" value=" Previous literature\cite{wang2023viola,rubenstein2023audiopalm,chen2023lauragpt} has analyzed the performance differences between the proposed multitask speech models and existing single-task models, but few studies compare these models under fair enough experimental conditions, such as same supervised training data." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2305.11000" id="8260">
        <attvalues>
          <attvalue for="5" value=" VioLA\cite{wang2023viola}, AudioPaLM\cite{rubenstein2023audiopalm}, LauraGPT\cite{chen2023lauragpt}, SpeechGPT\cite{zhang2023speechgpt}, etc support speech transcription and generation tasks by modeling both speech and text representations with Transformer LMs." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2210.13438" id="8261">
        <attvalues>
          <attvalue for="5" value=" Acoustic speech representations are generated from speech waveform through speech codec methods, eg EnCodec\cite{defossez2022high} and SoundStream\cite{zeghidour2021soundstream}." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2107.03312" id="8262">
        <attvalues>
          <attvalue for="5" value=" Acoustic speech representations are generated from speech waveform through speech codec methods, eg EnCodec\cite{defossez2022high} and SoundStream\cite{zeghidour2021soundstream}." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="1904.05862" id="8263">
        <attvalues>
          <attvalue for="5" value=" Semantic representations are usually extracted from self-supervised learning models such as wav2vec\cite{schneider2019wav2vec,baevski2020wav2vec,chung2021w2v} and HuBERT\cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2006.11477" id="8264">
        <attvalues>
          <attvalue for="5" value=" Semantic representations are usually extracted from self-supervised learning models such as wav2vec\cite{schneider2019wav2vec,baevski2020wav2vec,chung2021w2v} and HuBERT\cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2108.06209" id="8265">
        <attvalues>
          <attvalue for="5" value=" Semantic representations are usually extracted from self-supervised learning models such as wav2vec\cite{schneider2019wav2vec,baevski2020wav2vec,chung2021w2v} and HuBERT\cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2106.07447" id="8266">
        <attvalues>
          <attvalue for="5" value=" Semantic representations are usually extracted from self-supervised learning models such as wav2vec\cite{schneider2019wav2vec,baevski2020wav2vec,chung2021w2v} and HuBERT\cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2209.03143" id="8267">
        <attvalues>
          <attvalue for="5" value=" In preliminary experiments, we have found that the k-means discretization method used in AudioLM\cite{borsos2023audiolm} and Spear-TTS\cite{kharitonov2023speak} would lead to loss of acoustic information." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2302.03540" id="8268">
        <attvalues>
          <attvalue for="5" value=" In preliminary experiments, we have found that the k-means discretization method used in AudioLM\cite{borsos2023audiolm} and Spear-TTS\cite{kharitonov2023speak} would lead to loss of acoustic information." />
        </attvalues>
      </edge>
      <edge source="2406.07801" target="2010.05646" id="8269">
        <attvalues>
          <attvalue for="5" value=" We tried to resynthesize HuBERT k-means tokens of Mandarin Chinese speech with a HiFi-GAN\cite{kong2020hifi} vocoder and observed tone inaccuracies in synthesized speech, which are unacceptable for Mandarin." />
        </attvalues>
      </edge>
      <edge source="2312.01500" target="1802.06041" id="8270">
        <attvalues>
          <attvalue for="5" value=" As stated by \cite{martindale-carpuat-2018-fluency}, maintaining text fluency avoids misapprehensions, makes interactions more realistic, and leads to higher user satisfaction and trust." />
        </attvalues>
      </edge>
      <edge source="2211.04699" target="2004.00248" id="8273">
        <attvalues>
          <attvalue for="5" value=" 2) Second, some studied \cite{Adversarial, Lin2020JointPO,alam-etal-2020-punctuation,shi21_interspeech} regard it as a sequence labeling task, where a punctuation mark is assigned to each word by probability." />
        </attvalues>
      </edge>
      <edge source="2211.04699" target="2106.06731" id="8274">
        <attvalues>
          <attvalue for="5" value=" 2) Second, some studied \cite{Adversarial, Lin2020JointPO,alam-etal-2020-punctuation,shi21_interspeech} regard it as a sequence labeling task, where a punctuation mark is assigned to each word by probability." />
        </attvalues>
      </edge>
      <edge source="2211.04699" target="2002.07028" id="8275">
        <attvalues>
          <attvalue for="5" value=" Besides, the experiments \cite{Bottleneck, Talking} indicate that the limitation called Low-Rank Bottleneck exists in the self-attention mechanism." />
        </attvalues>
      </edge>
      <edge source="2211.04699" target="1706.03762" id="8276">
        <attvalues>
          <attvalue for="5" value=" Concretely, with the fixed vector size of multi-head self-attention vector\cite{attention}, increasing the number of self-attention heads would decrease the vector size in each head, which causes significant degradation of its comprehension." />
        </attvalues>
      </edge>
      <edge source="2312.15472" target="2303.08774" id="8277">
        <attvalues>
          <attvalue for="5" value=" Several studies indicate the recent LLMs provide {up to 40\% erroneous answers to factual questions} \cite{openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2312.15472" target="2202.03629" id="8278">
        <attvalues>
          <attvalue for="5" value=" Nonetheless, LLMs often provide incorrect answers to input queries and perform inaccurate inferences \cite{Ji_2023,openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2312.15472" target="2010.12884" id="8279">
        <attvalues>
          <attvalue for="5" value=" Finally, modifying LLMs' outputs during decoding often reduces their (linguistic) coherency \cite{lu-etal-2021-neurologic,lew2023sequential}." />
        </attvalues>
      </edge>
      <edge source="2312.15472" target="2004.02308" id="8281">
        <attvalues>
          <attvalue for="5" value=" We discuss how to use current work on using declarative constraints to learn accurate ML models over inconsistent data \cite{picado2020learning,DBLP:conf/deem/ZhenCT23} and techniques to embed structured queries in vector space \cite{ijcai2019p845,DBLP:conf/iclr/RenHL20,jackermeier2023box2el} to address these problems." />
        </attvalues>
      </edge>
      <edge source="2312.15472" target="2002.05969" id="8282">
        <attvalues>
          <attvalue for="5" value=" We discuss how to use current work on using declarative constraints to learn accurate ML models over inconsistent data \cite{picado2020learning,DBLP:conf/deem/ZhenCT23} and techniques to embed structured queries in vector space \cite{ijcai2019p845,DBLP:conf/iclr/RenHL20,jackermeier2023box2el} to address these problems." />
        </attvalues>
      </edge>
      <edge source="2312.15472" target="2307.09288" id="8283">
        <attvalues>
          <attvalue for="5" value=" We also report preliminary results for integrating constraints in Llama-2 \cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2312.09634" target="2310.11703" id="8284">
        <attvalues>
          <attvalue for="5" value=" For instance, vector databases \cite{hanComprehensiveSurveyVector2023} use such representations in fast similarity searches for retrieval and fuzzy joins." />
        </attvalues>
      </edge>
      <edge source="2312.09634" target="2207.08815" id="8285">
        <attvalues>
          <attvalue for="5" value=" However, for typical data tables, with columns containing entries of different nature and type, recent work has shown that bigger, more sophisticated, neural methods do not outperform simpler machine-learning models based on trees \cite{grinsztajnWhyTreebasedModels2022}." />
        </attvalues>
      </edge>
      <edge source="2312.09634" target="1907.01860" id="8286">
        <attvalues>
          <attvalue for="5" value=" Practitioners often rely on pretrained word embeddings developed in natural language processing \cite{joulinBagTricksEfficient2017} or numerical representations built from substrings \cite{cerdaEncodingHighcardinalityString2022}." />
        </attvalues>
      </edge>
      <edge source="2312.09634" target="1607.01759" id="8287">
        <attvalues>
          <attvalue for="5" value=" Practitioners often rely on pretrained word embeddings developed in natural language processing \cite{joulinBagTricksEfficient2017} or numerical representations built from substrings \cite{cerdaEncodingHighcardinalityString2022}." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2303.08774" id="8291">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as ChatGPT~\cite{chatgpt} and GPT-4~\cite{openai2022gpt4}, have recently pushed AI performance to a new height, with their astonishing capabilities in natural language processing~\cite{jiao2023chatgpt,han2023information}, logical reasoning~\cite{liu2023evaluating}, and imitation~\cite{wei2022emergent}." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2304.03439" id="8292">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs), such as ChatGPT~\cite{chatgpt} and GPT-4~\cite{openai2022gpt4}, have recently pushed AI performance to a new height, with their astonishing capabilities in natural language processing~\cite{jiao2023chatgpt,han2023information}, logical reasoning~\cite{liu2023evaluating}, and imitation~\cite{wei2022emergent}." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2209.07858" id="8295">
        <attvalues>
          <attvalue for="5" value=" General preferences (including helpfulness and harmlessness) have attracted increasing attention~\cite{bai2022training,bai2022constitutional,ganguli2022red,touvron2023llama}, while customized preferences remain unexplored." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2304.05302" id="8296">
        <attvalues>
          <attvalue for="5" value=" To align human values, various methods have been proposed from different perspectives, such as reinforcement learning~\cite{ouyang2022training,bai2022constitutional}, ranking~\cite{yuan2023rrhf}, and reject sampling~\cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2212.08073" id="8297">
        <attvalues>
          <attvalue for="5" value=" General preferences (including helpfulness and harmlessness) have attracted increasing attention~\cite{bai2022training,bai2022constitutional,ganguli2022red,touvron2023llama}, while customized preferences remain unexplored." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2307.09288" id="8298">
        <attvalues>
          <attvalue for="5" value=" Then we train general and domain-specific reward models with LLaMA~\cite{touvron2023llama} as the base model using both general preferences~\cite{bai2022training,nakano2021webgpt,peng2023instruction} and DSP data." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2112.00861" id="8300">
        <attvalues>
          <attvalue for="5" value=" A similar reward pre-training idea has been empirically tested by \cite{askell2021general} as preference model pre-training (PMP)." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2204.05862" id="8301">
        <attvalues>
          <attvalue for="5" value=" Then we train general and domain-specific reward models with LLaMA~\cite{touvron2023llama} as the base model using both general preferences~\cite{bai2022training,nakano2021webgpt,peng2023instruction} and DSP data." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2305.00955" id="8302">
        <attvalues>
          <attvalue for="5" value=" For harmlessness, models' responses are supposed to be fair, safe, and without toxicity~\cite{bai2022constitutional,ganguli2022red,fernandes2023bridging}." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="1804.05958" id="8303">
        <attvalues>
          <attvalue for="5" value=" Task-related rewards designed in earlier works of particular NLP domains (such as machine translation~\cite{kreutzer2018can}, summarization~\cite{ziegler2019fine}, and continuation~\cite{stiennon2020learning}) can be classified into the helpfulness category." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="1909.08593" id="8304">
        <attvalues>
          <attvalue for="5" value=" Task-related rewards designed in earlier works of particular NLP domains (such as machine translation~\cite{kreutzer2018can}, summarization~\cite{ziegler2019fine}, and continuation~\cite{stiennon2020learning}) can be classified into the helpfulness category." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="1903.10676" id="8305">
        <attvalues>
          <attvalue for="5" value=" Moreover, the above classification of human preferences naturally leads to an interesting question: ``How to learn a customized reward model well while preserving its general preference ability?'' A high-qualified customized reward model is practically valued to enhance the domain-specific LLM fine-tuning by serving as a learning critic or an evaluation metric~\cite{askell2021general, touvron2023llama}, because general LLMs can not handle all application domains, especially in which professional knowledge are required~\cite{beltagy2019scibert,gu2021domain,li2023llava}." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2007.15779" id="8306">
        <attvalues>
          <attvalue for="5" value=" Moreover, the above classification of human preferences naturally leads to an interesting question: ``How to learn a customized reward model well while preserving its general preference ability?'' A high-qualified customized reward model is practically valued to enhance the domain-specific LLM fine-tuning by serving as a learning critic or an evaluation metric~\cite{askell2021general, touvron2023llama}, because general LLMs can not handle all application domains, especially in which professional knowledge are required~\cite{beltagy2019scibert,gu2021domain,li2023llava}." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2306.00890" id="8307">
        <attvalues>
          <attvalue for="5" value=" Moreover, the above classification of human preferences naturally leads to an interesting question: ``How to learn a customized reward model well while preserving its general preference ability?'' A high-qualified customized reward model is practically valued to enhance the domain-specific LLM fine-tuning by serving as a learning critic or an evaluation metric~\cite{askell2021general, touvron2023llama}, because general LLMs can not handle all application domains, especially in which professional knowledge are required~\cite{beltagy2019scibert,gu2021domain,li2023llava}." />
        </attvalues>
      </edge>
      <edge source="2309.03126" target="2304.03277" id="8309">
        <attvalues>
          <attvalue for="5" value=" Then we train general and domain-specific reward models with LLaMA~\cite{touvron2023llama} as the base model using both general preferences~\cite{bai2022training,nakano2021webgpt,peng2023instruction} and DSP data." />
        </attvalues>
      </edge>
      <edge source="2209.05972" target="1908.10084" id="8311">
        <attvalues>
          <attvalue for="5" value=" In general, PLMs use either $[CLS]$ tokens in the last layer, $AVG$ which is the average representation of tokens in the last layer\cite{reimers2019sentence, li2020sentence}, or $AVG_{FL}$ which is the average representation of tokens in the first and last layers\cite{gao2021simcse}, to pool out sentence representation from word representations." />
        </attvalues>
      </edge>
      <edge source="2209.05972" target="2104.08821" id="8312">
        <attvalues>
          <attvalue for="5" value=" In general, PLMs use either $[CLS]$ tokens in the last layer, $AVG$ which is the average representation of tokens in the last layer\cite{reimers2019sentence, li2020sentence}, or $AVG_{FL}$ which is the average representation of tokens in the first and last layers\cite{gao2021simcse}, to pool out sentence representation from word representations." />
        </attvalues>
      </edge>
      <edge source="2011.00592" target="2006.09109" id="8322">
        <attvalues>
          <attvalue for="5" value=" There are a variety of problems surrounding current probing task specifications: (i) probing tasks need to be manually construed, which brings with it a certain degree of arbitrariness and incompleteness; (ii) most probing tasks require labeled datasets or trained classifiers such as dependency parsers for linguistic processing---however, these may be unavailable for many low-resource languages or available only to a limited degree; (iii) it is not entirely clear how probing tasks have to be designed, eg, how much training data they require and which classifier to use for probing \cite{Eger_howto:2020}; (iv) \newcite{Ravichander2020ProbingTP} also argue that standard probing tasks do not outline the information signals a classifier actually uses for making predictions." />
        </attvalues>
      </edge>
      <edge source="2104.07275" target="1810.07942" id="8323">
        <attvalues>
          <attvalue for="5" value=" First, we benchmark exact match (EM) on multiple task-oriented semantic parsing datasets, including TOPv2 \cite{chen-2020-topv2} and TOP \cite{gupta2018semantic}." />
        </attvalues>
      </edge>
      <edge source="2104.07275" target="2001.11458" id="8324">
        <attvalues>
          <attvalue for="5" value=" Task-oriented conversational assistants typically first employ semantic parsers to map utterances to frames \cite{hemphill1990atis,coucke2018snips,gupta2018semantic,rongali2020don,decoupled}." />
        </attvalues>
      </edge>
      <edge source="2104.07275" target="2009.13655" id="8325">
        <attvalues>
          <attvalue for="5" value=" Task-oriented conversational assistants typically first employ semantic parsers to map utterances to frames \cite{hemphill1990atis,coucke2018snips,gupta2018semantic,rongali2020don,decoupled}." />
        </attvalues>
      </edge>
      <edge source="2104.07275" target="2010.03714" id="8326">
        <attvalues>
          <attvalue for="5" value=" Due to performance constraints in real-world deployments, recent work in task-oriented semantic parsing has shifted towards building seq2seq, non-autoregressive parsers optimized for both quality and latency \cite{Zhu2020DontPI,nar_semantic_parsing}." />
        </attvalues>
      </edge>
      <edge source="2104.07275" target="2104.04923" id="8327">
        <attvalues>
          <attvalue for="5" value=" However, the application of off-the-shelf, non-autoregressive algorithms to task-oriented semantic parsing is not trivial, often leading to brittle implementations with sub-optimal, opinionated components \cite{nar_semantic_parsing}." />
        </attvalues>
      </edge>
      <edge source="2104.07275" target="1904.09324" id="8328">
        <attvalues>
          <attvalue for="5" value=" One popular family of seq2seq, non-autoregressive, task-oriented parsers is based on the mask-predict algorithm \cite{ghazvininejad2019maskpredict}, which operates in three steps: encoding, length prediction, and decoding." />
        </attvalues>
      </edge>
      <edge source="2104.07275" target="1805.04793" id="8329">
        <attvalues>
          <attvalue for="5" value=" This additionally creates a parallel among our approach and coarse-to-fine \cite{Dong2018CoarsetoFineDF} modeling as our length predictor predicts a general structure that our decoder further refines." />
        </attvalues>
      </edge>
      <edge source="2104.07275" target="2010.03546" id="8330">
        <attvalues>
          <attvalue for="5" value=" First, we benchmark exact match (EM) on multiple task-oriented semantic parsing datasets, including TOPv2 \cite{chen-2020-topv2} and TOP \cite{gupta2018semantic}." />
        </attvalues>
      </edge>
      <edge source="2312.12624" target="2307.09288" id="8333">
        <attvalues>
          <attvalue for="5" value=" Llama2 \cite{touvron2023llama} proudly stands as a member of a burgeoning family of models, poised to bridge the multilingual gap by providing precise, contextually relevant results for Odia while championing linguistic diversity in the NLP landscape." />
        </attvalues>
      </edge>
      <edge source="2104.11070" target="2005.00796" id="8334">
        <attvalues>
          <attvalue for="5" value=" Conversations in goal or task oriented conversational interfaces, such as digital personal assistants or chatbots, typically span multiple turns of back and forth between a user and a bot \cite{multiwoz:18,houseini_2020}." />
        </attvalues>
      </edge>
      <edge source="2104.11070" target="1708.06073" id="8335">
        <attvalues>
          <attvalue for="5" value=" In particular, recurrent neural network (RNN) and LSTM based NLMs are trained and evaluated without resetting hidden states across sentences \cite{ms:2018, irie:19, xiong-etal-2018-session}." />
        </attvalues>
      </edge>
      <edge source="2104.11070" target="1706.03762" id="8337">
        <attvalues>
          <attvalue for="5" value=" The wide spread adoption of transformer architecture based on self attention \cite{vaswani:17} and pretrained masked language models \cite{devlin:18} paved the way for use of transformers for NLMs." />
        </attvalues>
      </edge>
      <edge source="2104.11070" target="1901.02860" id="8340">
        <attvalues>
          <attvalue for="5" value=" While the initial work used modified attention masks to handle longer input sequences more efficiently, Transformer-XL (TXL) \cite{dai-etal-2019-transformer} made use of segment wise recurrence, making it well suited for long span decoding." />
        </attvalues>
      </edge>
      <edge source="2104.11070" target="2102.06474" id="8341">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{sun2021transformer} further tried to improve cross utterance decoding using TXL by adding a LSTM fusion layer, where the hidden states of LSTM are carried over multiple sentences." />
        </attvalues>
      </edge>
      <edge source="2104.11070" target="1906.11604" id="8342">
        <attvalues>
          <attvalue for="5" value=" The work in \cite{kim-etal-2019-gated} explored the use of a conversational context embedding from recent history to improve the contextualization of end-to-end ASR models where as some other works \cite{mikolov:12,keli:2018,anirudh:18,Chen:2015} used an explicit topic vector or a neural cache and a domain classifier for domain and contextual adaptation." />
        </attvalues>
      </edge>
      <edge source="2104.11070" target="1806.10215" id="8343">
        <attvalues>
          <attvalue for="5" value=" The work in \cite{kim-etal-2019-gated} explored the use of a conversational context embedding from recent history to improve the contextualization of end-to-end ASR models where as some other works \cite{mikolov:12,keli:2018,anirudh:18,Chen:2015} used an explicit topic vector or a neural cache and a domain classifier for domain and contextual adaptation." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="1303.5778" id="8344">
        <attvalues>
          <attvalue for="5" value=" End-to-end (E2E) automatic speech recognition (ASR) models simplify conventional pipeline ASR methods and directly transcribe input speech into corresponding text \cite{8068205,6638947}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="2206.07430" id="8345">
        <attvalues>
          <attvalue for="5" value=" However, incorporating the external LM demands extra computational cost and additional parameters, and accurate internal LM estimation is not always feasible \cite{tsunoo22_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="2201.03655" id="8346">
        <attvalues>
          <attvalue for="5" value=" However, E2E ASR still suffers from unseen domains \cite{tsunoo22_interspeech}, and large quantities of labelled data are not always feasible to collect and can therefore be limited \cite{9746480}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="1905.07149" id="8347">
        <attvalues>
          <attvalue for="5" value=" Adaptation training methods can be utilised to alleviate this issue when the target domain has enough paired data \cite{tsunoo19_interspeech, 6424251}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="1506.07503" id="8348">
        <attvalues>
          <attvalue for="5" value=" Several structural fusion methods like deep fusion \cite{gulcehre2015using} and cold fusion \cite{sriram18_interspeech} have been proposed, but require additional training and haven't replaced shallow fusion \cite{chorowski2015attention} as the dominant LM integration method \cite{9003790, 9383515}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="1712.01996" id="8349">
        <attvalues>
          <attvalue for="5" value=" Shallow fusion \cite{chorowski2015attention} which linearly interpolates the E2E ASR with an external LM is straightforward and widely deployed \cite{8462682}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="1708.06426" id="8351">
        <attvalues>
          <attvalue for="5" value=" Several structural fusion methods like deep fusion \cite{gulcehre2015using} and cold fusion \cite{sriram18_interspeech} have been proposed, but require additional training and haven't replaced shallow fusion \cite{chorowski2015attention} as the dominant LM integration method \cite{9003790, 9383515}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="2002.11268" id="8352">
        <attvalues>
          <attvalue for="5" value=" It subtracts the score of a source-domain LM from the log-linear combination of the E2E ASR model and target-domain LM scores \cite{9003790}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="2011.01991" id="8353">
        <attvalues>
          <attvalue for="5" value=" Furthermore, the estimate of the E2E ASR model's internal LM has been explored \cite{9383515, 9415039, 9746948, zeineldeen21_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="2110.05354" id="8354">
        <attvalues>
          <attvalue for="5" value=" Work in \cite{meng22_interspeech} explores fine-tuning the internal LM with text-only data but requires regularisation strategies to avoid the internal LM over-learning target domains." />
        </attvalues>
      </edge>
      <edge source="2302.08579" target="2102.10233" id="8355">
        <attvalues>
          <attvalue for="5" value=" With E2E ASR models trained on the LibriSpeech corpus \cite{7178964}, experiments showed that the proposed methods greatly boosted the cross-domain ASR accuracy on the Switchboard \cite{225858} and AESRC2020 \cite{9413386} corpus while performing robustly in intra-domain scenarios." />
        </attvalues>
      </edge>
      <edge source="2311.02310" target="2303.08774" id="8356">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in large language models (LLMs) have revolutionized Natural Language Processing field as such models \cite[inter alia]{openai2023gpt4, instrcutgpt, chowdhery2022palm, touvron2023llama} can easily adapt to a new task through prompt (in-context) learning where the task instruction and demonstrations (examples to guide LLMs on the task) are provided to the model." />
        </attvalues>
      </edge>
      <edge source="2311.02310" target="2211.09102" id="8357">
        <attvalues>
          <attvalue for="5" value=" Recent work~\cite{vilar2022prompting, zhang2023prompting, jiao2023chatgpt, hendy2023good} has found that prompt-based methods perform well on language models trained primarily on monolingual data, rivaling state-of-the-art systems trained specifically for machine translation tasks on benchmark datasets." />
        </attvalues>
      </edge>
      <edge source="2307.01878" target="2005.14165" id="8358">
        <attvalues>
          <attvalue for="5" value=" The current state-of-the-art language modeling methods often require transfer learning \cite{brown2020language}, large amount of labels \cite{yang2019xlnet} and pretrained embeddings \cite{cao2020multilingual}." />
        </attvalues>
      </edge>
      <edge source="2307.01878" target="1812.11270" id="8361">
        <attvalues>
          <attvalue for="5" value=" For semi-supervised method \cite{meng2018weaklysupervised} tailored to limited label scenario, it is time consuming to both tune and train." />
        </attvalues>
      </edge>
      <edge source="2307.01878" target="0710.0845" id="8362">
        <attvalues>
          <attvalue for="5" value=" Topic modeling is an unsupervised method for discovering latent structure within the training document sets and achieves great empirical performance in many fields\cite{blei2009nested}, including finance \cite{ healthcare \cite{DBLP:journals/corr/abs-1711-10960}, education \cite{zhao2020targeted}, marketing \cite{Reisenbichler2019} and social science \cite{762586}." />
        </attvalues>
      </edge>
      <edge source="2307.01878" target="1711.10960" id="8363">
        <attvalues>
          <attvalue for="5" value=" Topic modeling is an unsupervised method for discovering latent structure within the training document sets and achieves great empirical performance in many fields\cite{blei2009nested}, including finance \cite{ healthcare \cite{DBLP:journals/corr/abs-1711-10960}, education \cite{zhao2020targeted}, marketing \cite{Reisenbichler2019} and social science \cite{762586}." />
        </attvalues>
      </edge>
      <edge source="2307.01878" target="1706.00359" id="8365">
        <attvalues>
          <attvalue for="5" value=" The goal of unsupervised topic modeling methods \cite{blei2003latent,teh2006hierarchical,miao2018discovering,gemp2019weakly, xu-etal-2023-vontss} is to maximize the probability of the observed data, resulting in the tendency to identify obvious and superficial aspects of a corpus." />
        </attvalues>
      </edge>
      <edge source="2307.01878" target="1003.0783" id="8368">
        <attvalues>
          <attvalue for="5" value=" To incorporate users' domain knowledge of documents into the model, supervised modeling \cite{blei2010supervised, JMLR:v13:zhu12a, pmlr-v108-wang20c} has been studied." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2005.06600" id="8369">
        <attvalues>
          <attvalue for="5" value=" It aims to detect and correct spelling errors in the Chinese text~\cite{chang1995new,DBLP:journals/corr/abs-2005-06600}." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="1807.01270" id="8370">
        <attvalues>
          <attvalue for="5" value=" &#09;Generally, sequence translation~\cite{DBLP:conf/emnlp/WangSLHZ18,DBLP:journals/corr/abs-1807-01270,DBLP:conf/acl/WangTZ19,DBLP:conf/ijcnlp/WangKKK20,DBLP:conf/acl/KanekoMKSI20} and sequence tagging~\cite{DBLP:conf/bea/OmelianchukACS20,liang-etal-2020-bert,DBLP:conf/emnlp/MallinsonSMG20,DBLP:conf/acl/ParnowLZ21} are the two most typical technical paradigms to tackle the problem." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2011.02093" id="8371">
        <attvalues>
          <attvalue for="5" value=" &#09;Generally, sequence translation~\cite{DBLP:conf/emnlp/WangSLHZ18,DBLP:journals/corr/abs-1807-01270,DBLP:conf/acl/WangTZ19,DBLP:conf/ijcnlp/WangKKK20,DBLP:conf/acl/KanekoMKSI20} and sequence tagging~\cite{DBLP:conf/bea/OmelianchukACS20,liang-etal-2020-bert,DBLP:conf/emnlp/MallinsonSMG20,DBLP:conf/acl/ParnowLZ21} are the two most typical technical paradigms to tackle the problem." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2005.00987" id="8372">
        <attvalues>
          <attvalue for="5" value=" &#09;Generally, sequence translation~\cite{DBLP:conf/emnlp/WangSLHZ18,DBLP:journals/corr/abs-1807-01270,DBLP:conf/acl/WangTZ19,DBLP:conf/ijcnlp/WangKKK20,DBLP:conf/acl/KanekoMKSI20} and sequence tagging~\cite{DBLP:conf/bea/OmelianchukACS20,liang-etal-2020-bert,DBLP:conf/emnlp/MallinsonSMG20,DBLP:conf/acl/ParnowLZ21} are the two most typical technical paradigms to tackle the problem." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2003.10687" id="8373">
        <attvalues>
          <attvalue for="5" value=" &#09;Generally, sequence translation~\cite{DBLP:conf/emnlp/WangSLHZ18,DBLP:journals/corr/abs-1807-01270,DBLP:conf/acl/WangTZ19,DBLP:conf/ijcnlp/WangKKK20,DBLP:conf/acl/KanekoMKSI20} and sequence tagging~\cite{DBLP:conf/bea/OmelianchukACS20,liang-etal-2020-bert,DBLP:conf/emnlp/MallinsonSMG20,DBLP:conf/acl/ParnowLZ21} are the two most typical technical paradigms to tackle the problem." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2105.14209" id="8374">
        <attvalues>
          <attvalue for="5" value=" &#09;Generally, sequence translation~\cite{DBLP:conf/emnlp/WangSLHZ18,DBLP:journals/corr/abs-1807-01270,DBLP:conf/acl/WangTZ19,DBLP:conf/ijcnlp/WangKKK20,DBLP:conf/acl/KanekoMKSI20} and sequence tagging~\cite{DBLP:conf/bea/OmelianchukACS20,liang-etal-2020-bert,DBLP:conf/emnlp/MallinsonSMG20,DBLP:conf/acl/ParnowLZ21} are the two most typical technical paradigms to tackle the problem." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2005.07421" id="8377">
        <attvalues>
          <attvalue for="5" value=" Benefiting from the development of pretraining techniques, many researchers fine-tune the pretrained language models such as BERT~\cite{DBLP:conf/naacl/DevlinCLT19} on the task of CSC and obtain encouraging performance~\cite{DBLP:conf/naacl/ZhaoWSJL19,DBLP:conf/aclnut/HongYHLL19,DBLP:conf/acl/ZhangHLL20,DBLP:conf/acl/LiuYYZW20,DBLP:conf/acl/LiZZH20,DBLP:conf/acl/HuangLJZCWX20,DBLP:conf/acl/GuoNWZX21,DBLP:conf/acl/ZhangPZWHSWW21,DBLP:conf/acl/Li020,dai-etal-2022-whole}." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2105.14813" id="8378">
        <attvalues>
          <attvalue for="5" value=" Benefiting from the development of pretraining techniques, many researchers fine-tune the pretrained language models such as BERT~\cite{DBLP:conf/naacl/DevlinCLT19} on the task of CSC and obtain encouraging performance~\cite{DBLP:conf/naacl/ZhaoWSJL19,DBLP:conf/aclnut/HongYHLL19,DBLP:conf/acl/ZhangHLL20,DBLP:conf/acl/LiuYYZW20,DBLP:conf/acl/LiZZH20,DBLP:conf/acl/HuangLJZCWX20,DBLP:conf/acl/GuoNWZX21,DBLP:conf/acl/ZhangPZWHSWW21,DBLP:conf/acl/Li020,dai-etal-2022-whole}." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2106.01609" id="8379">
        <attvalues>
          <attvalue for="5" value=" Benefiting from the development of pretraining techniques, many researchers fine-tune the pretrained language models such as BERT~\cite{DBLP:conf/naacl/DevlinCLT19} on the task of CSC and obtain encouraging performance~\cite{DBLP:conf/naacl/ZhaoWSJL19,DBLP:conf/aclnut/HongYHLL19,DBLP:conf/acl/ZhangHLL20,DBLP:conf/acl/LiuYYZW20,DBLP:conf/acl/LiZZH20,DBLP:conf/acl/HuangLJZCWX20,DBLP:conf/acl/GuoNWZX21,DBLP:conf/acl/ZhangPZWHSWW21,DBLP:conf/acl/Li020,dai-etal-2022-whole}." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="1907.11692" id="8380">
        <attvalues>
          <attvalue for="5" value=" Fortunately, masked pretrained language models such as BERT \cite{DBLP:conf/naacl/DevlinCLT19}, RoBERTa~\cite{DBLP:journals/corr/abs-1907-11692}, ELECTRA~\cite{DBLP:conf/iclr/ClarkLLM20}, etc can satisfy the needs of detecting and correcting spelling errors in an unsupervised manner." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2003.10555" id="8381">
        <attvalues>
          <attvalue for="5" value=" Fortunately, masked pretrained language models such as BERT \cite{DBLP:conf/naacl/DevlinCLT19}, RoBERTa~\cite{DBLP:journals/corr/abs-1907-11692}, ELECTRA~\cite{DBLP:conf/iclr/ClarkLLM20}, etc can satisfy the needs of detecting and correcting spelling errors in an unsupervised manner." />
        </attvalues>
      </edge>
      <edge source="2209.07068" target="2109.06822" id="8382">
        <attvalues>
          <attvalue for="5" value=" \cite{DBLP:conf/emnlp/YasunagaLL21} employ GPT2-like models to conduct unsupervised English grammatical error correction which also verifies the feasible of our direction." />
        </attvalues>
      </edge>
      <edge source="2305.09898" target="1910.13461" id="8383">
        <attvalues>
          <attvalue for="5" value=" SimCLS \cite{liu-liu-2021-simcls} and BRIO-Ctr \cite{liu-etal-2022-brio} train a large pre-trained model, such as RoBERTa \cite{2019RoBERTa} and BART \cite{lewis-etal-2020-bart}, to align the candidate summaries according to the quality." />
        </attvalues>
      </edge>
      <edge source="2305.09898" target="1506.03099" id="8384">
        <attvalues>
          <attvalue for="5" value=" This phenomenon is known as exposure bias \cite{NIPS2015_e995f98d, ICLR2016_Aurelio}." />
        </attvalues>
      </edge>
      <edge source="2305.09898" target="2104.07210" id="8385">
        <attvalues>
          <attvalue for="5" value=" To mitigate this problem, re-ranking systems \cite{liu-etal-2021-refsum, liu-liu-2021-simcls, liu-etal-2022-brio, ravaut-etal-2022-summareranker} have recently been introduced to generate a more appropriate summary." />
        </attvalues>
      </edge>
      <edge source="2305.09898" target="2106.01890" id="8386">
        <attvalues>
          <attvalue for="5" value=" SimCLS \cite{liu-liu-2021-simcls} and BRIO-Ctr \cite{liu-etal-2022-brio} train a large pre-trained model, such as RoBERTa \cite{2019RoBERTa} and BART \cite{lewis-etal-2020-bart}, to align the candidate summaries according to the quality." />
        </attvalues>
      </edge>
      <edge source="2305.09898" target="2203.16804" id="8387">
        <attvalues>
          <attvalue for="5" value=" In addition, BRIO-Mul \cite{liu-etal-2022-brio} demonstrates that the combination of the contrastive and cross-entropy loss works complementarily and has better performance." />
        </attvalues>
      </edge>
      <edge source="2305.09898" target="1907.11692" id="8388">
        <attvalues>
          <attvalue for="5" value=" SimCLS \cite{liu-liu-2021-simcls} and BRIO-Ctr \cite{liu-etal-2022-brio} train a large pre-trained model, such as RoBERTa \cite{2019RoBERTa} and BART \cite{lewis-etal-2020-bart}, to align the candidate summaries according to the quality." />
        </attvalues>
      </edge>
      <edge source="2305.09898" target="2203.06569" id="8389">
        <attvalues>
          <attvalue for="5" value=" SummaReranker \cite{ravaut-etal-2022-summareranker} minimizes the average over the binary cross-entropy losses optimized for each evaluation metric." />
        </attvalues>
      </edge>
      <edge source="2305.09898" target="2004.08795" id="8390">
        <attvalues>
          <attvalue for="5" value=" Inspired by \cite{zhong-etal-2020-extractive}, we conduct a preliminary study, by sorting candidate summaries in descending order based on the ROUGE score and then defining $z$ as the rank index of the highest BERTScore summary." />
        </attvalues>
      </edge>
      <edge source="2107.00099" target="2007.15188" id="8404">
        <attvalues>
          <attvalue for="5" value=" Given sufficient data for a specific domain, it is cost-effective to customize the unified model to the target domain rather than training a domain-dependent model for higher accuracy~\cite{Li2020, Bell2020}." />
        </attvalues>
      </edge>
      <edge source="2107.00099" target="1705.09724" id="8405">
        <attvalues>
          <attvalue for="5" value=" Previously, a seed model provided target labels in a semi-supervised manner, filtering data using normalized frame-level entropy~\cite{Liu2007}, and minimum Bayes risk~\cite{Walker2017}." />
        </attvalues>
      </edge>
      <edge source="2102.06551" target="1603.04351" id="8407">
        <attvalues>
          <attvalue for="5" value=" While these approaches simplify the parsing architecture and eliminate the need for hand-crafted feature engineering \cite{chen-manning-2014-fast,dyer-etal-2015-transition,kiperwasser-goldberg-2016-simple,DBLP:conf/iclr/DozatM17,kulmizev-etal-2019-deep}, their performance has been less exciting for several morphologically rich languages (MRLs) and low-resource languages \cite{more-etal-2019-joint,seeker-cetinoglu-2015-graph}." />
        </attvalues>
      </edge>
      <edge source="2102.06551" target="1611.01734" id="8408">
        <attvalues>
          <attvalue for="5" value=" While these approaches simplify the parsing architecture and eliminate the need for hand-crafted feature engineering \cite{chen-manning-2014-fast,dyer-etal-2015-transition,kiperwasser-goldberg-2016-simple,DBLP:conf/iclr/DozatM17,kulmizev-etal-2019-deep}, their performance has been less exciting for several morphologically rich languages (MRLs) and low-resource languages \cite{more-etal-2019-joint,seeker-cetinoglu-2015-graph}." />
        </attvalues>
      </edge>
      <edge source="2102.06551" target="1909.02857" id="8409">
        <attvalues>
          <attvalue for="5" value=" This includes data augmentation strategies, cross-lingual transfer \cite{vania-etal-2019-systematic} and using unlabelled data with semi-supervised learning \cite{clark-etal-2018-semi} and self-training \cite{rotman2019deep}." />
        </attvalues>
      </edge>
      <edge source="2102.06551" target="1809.08370" id="8410">
        <attvalues>
          <attvalue for="5" value=" This includes data augmentation strategies, cross-lingual transfer \cite{vania-etal-2019-systematic} and using unlabelled data with semi-supervised learning \cite{clark-etal-2018-semi} and self-training \cite{rotman2019deep}." />
        </attvalues>
      </edge>
      <edge source="2102.06551" target="1911.04286" id="8411">
        <attvalues>
          <attvalue for="5" value=" Further, our method performs close to DCST \cite{rotman2019deep}, a self-training based extension of \newcite{dozat2017stanford}, which uses gold morphological tags as input for training." />
        </attvalues>
      </edge>
      <edge source="2102.06551" target="1808.09180" id="8412">
        <attvalues>
          <attvalue for="5" value=" Further, incorporating morphological knowledge substantially improves the parsing performance for MRLs, including low-resource languages ~\cite{vania-etal-2018-character,dehouck-denis-2018-framework}." />
        </attvalues>
      </edge>
      <edge source="2406.08393" target="2210.14755" id="8415">
        <attvalues>
          <attvalue for="5" value=" On the other hand, despite efforts by Kune{\v{s}}ov{\'a} and Zaj{\'\i}~\cite{kunevsova2023multitask} to enhance SCD performance through multitasking, eg, OSD and VAD, all these tasks are frame-level binary classification tasks, which pose a risk of overfitting when training complex models due to the simplistic learning paradigm." />
        </attvalues>
      </edge>
      <edge source="2406.08393" target="2109.11641" id="8416">
        <attvalues>
          <attvalue for="5" value=" For example, in~\cite{xia2022turn}, the transcription used to train an ASR model is enhanced by incorporating a distinct token designed to denote speaker turns." />
        </attvalues>
      </edge>
      <edge source="2406.08393" target="2211.06482" id="8417">
        <attvalues>
          <attvalue for="5" value=" In addition to label-based methods for frame-level SCD, several works have explored leveraging text transcription for word-level speaker change detection through ASR techniques~\cite{xia2022turn,zhao2023augmenting}." />
        </attvalues>
      </edge>
      <edge source="2406.08393" target="2006.11477" id="8418">
        <attvalues>
          <attvalue for="5" value="0~\cite{baevski2020wav2vec}, on the SCD task." />
        </attvalues>
      </edge>
      <edge source="2406.08393" target="2106.07447" id="8419">
        <attvalues>
          <attvalue for="5" value="0, other SSL models such as Hubert~\cite{hsu2021hubert} and WavLm~\cite{chen2022wavlm} have also gained significant attention in various downstream tasks, such as Hubert-based speech recognition~\cite{nasersharif2023speech} and WavLm-based speech synthesis~\cite{lajszczak2024base}." />
        </attvalues>
      </edge>
      <edge source="2406.08393" target="2110.13900" id="8420">
        <attvalues>
          <attvalue for="5" value="0, other SSL models such as Hubert~\cite{hsu2021hubert} and WavLm~\cite{chen2022wavlm} have also gained significant attention in various downstream tasks, such as Hubert-based speech recognition~\cite{nasersharif2023speech} and WavLm-based speech synthesis~\cite{lajszczak2024base}." />
        </attvalues>
      </edge>
      <edge source="2406.08393" target="2402.08093" id="8421">
        <attvalues>
          <attvalue for="5" value="0, other SSL models such as Hubert~\cite{hsu2021hubert} and WavLm~\cite{chen2022wavlm} have also gained significant attention in various downstream tasks, such as Hubert-based speech recognition~\cite{nasersharif2023speech} and WavLm-based speech synthesis~\cite{lajszczak2024base}." />
        </attvalues>
      </edge>
      <edge source="2406.08393" target="2005.08100" id="8422">
        <attvalues>
          <attvalue for="5" value=" To tackle those issues, we propose an innovative end-to-end SCD model, referred to as SCDNet, based on the Conformer architecture~\cite{gulati2020conformer}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="2006.11477" id="8423">
        <attvalues>
          <attvalue for="5" value=" Many task designs exist, but one of the most common is to mask the input and train the neural network to reconstruct the original input based on the surrounding information \cite{kenton2019bert,baevski2020wav2vec}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="2104.01767" id="8427">
        <attvalues>
          <attvalue for="5" value=" By starting from simply averaging the word representations \cite{wieting2015towards}, many unparameterized methods were introduced, such as considering the word frequency while averaging \cite{arora2017simple}, whitening the representations \cite{huang2021whiteningbert,su2021whitening}, or utilizing the singular value transformation \cite{yan2022addressing}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="2103.15316" id="8428">
        <attvalues>
          <attvalue for="5" value=" By starting from simply averaging the word representations \cite{wieting2015towards}, many unparameterized methods were introduced, such as considering the word frequency while averaging \cite{arora2017simple}, whitening the representations \cite{huang2021whiteningbert,su2021whitening}, or utilizing the singular value transformation \cite{yan2022addressing}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="2208.11790" id="8429">
        <attvalues>
          <attvalue for="5" value=" By starting from simply averaging the word representations \cite{wieting2015towards}, many unparameterized methods were introduced, such as considering the word frequency while averaging \cite{arora2017simple}, whitening the representations \cite{huang2021whiteningbert,su2021whitening}, or utilizing the singular value transformation \cite{yan2022addressing}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="1803.10963" id="8430">
        <attvalues>
          <attvalue for="5" value=" Also, the attention layer is often used to obtain which representation matters more for each downstream task \cite{okabe2018attentive,safari2020self,wu2020vector}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="2008.01077" id="8431">
        <attvalues>
          <attvalue for="5" value=" Also, the attention layer is often used to obtain which representation matters more for each downstream task \cite{okabe2018attentive,safari2020self,wu2020vector}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="1910.05453" id="8432">
        <attvalues>
          <attvalue for="5" value=" Decades of research have been conducted on VQ \cite{soong1987report}, where it crept into the modern self-supervised models, such as vq-wav2vec \cite{jegou2010product,baevski2020vq}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="1804.03209" id="8434">
        <attvalues>
          <attvalue for="5" value=" To further boost the research on unsupervised pooling methods, we devise a benchmark that evaluates various pooling methods across multiple backbone networks and a wide range of tasks, namely, keyword spotting \cite{warden2018speech}, speaker identification \cite{nagrani2017voxceleb}, intent classification \cite{lugosch2019speech}, and emotion recognition \cite{busso2008iemocap}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="1706.08612" id="8435">
        <attvalues>
          <attvalue for="5" value=" To further boost the research on unsupervised pooling methods, we devise a benchmark that evaluates various pooling methods across multiple backbone networks and a wide range of tasks, namely, keyword spotting \cite{warden2018speech}, speaker identification \cite{nagrani2017voxceleb}, intent classification \cite{lugosch2019speech}, and emotion recognition \cite{busso2008iemocap}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="1904.03670" id="8436">
        <attvalues>
          <attvalue for="5" value=" To further boost the research on unsupervised pooling methods, we devise a benchmark that evaluates various pooling methods across multiple backbone networks and a wide range of tasks, namely, keyword spotting \cite{warden2018speech}, speaker identification \cite{nagrani2017voxceleb}, intent classification \cite{lugosch2019speech}, and emotion recognition \cite{busso2008iemocap}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="2002.05709" id="8437">
        <attvalues>
          <attvalue for="5" value=" To avoid supervision altogether, we closely follow the recent contrastive learning literature \cite{chen2020simple,choi2022combating} to evaluate the effectiveness of the pooled representation directly via the nearest-neighbor approach \cite{bernhardsson2017annoy}." />
        </attvalues>
      </edge>
      <edge source="2304.03940" target="2011.07932" id="8438">
        <attvalues>
          <attvalue for="5" value=" To avoid supervision altogether, we closely follow the recent contrastive learning literature \cite{chen2020simple,choi2022combating} to evaluate the effectiveness of the pooled representation directly via the nearest-neighbor approach \cite{bernhardsson2017annoy}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2302.06476" id="8440">
        <attvalues>
          <attvalue for="5" value=" The evolution of Large Language Models (LLMs) like OpenAI's ChatGPT and Meta AI's LLaMa-2 represents a major leap in NLP \cite{qin2023chatgpt, touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2303.18223" id="8441">
        <attvalues>
          <attvalue for="5" value=" These models, trained on vast datasets, are adept at mimicking the subtleties of human language with remarkable accuracy \cite{zhao2023survey, yin-etal-2023-large}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2310.06825" id="8442">
        <attvalues>
          <attvalue for="5" value=" Both open-source and adaptable, LLaMa-2 and Mistral-7B enhance NLP technology accessibility and facilitate research without proprietary constraints \cite{jiang2023mistral}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2005.14165" id="8443">
        <attvalues>
          <attvalue for="5" value=" This capability to navigate the intricate relationship between textual context and target subjects, accurately aligning with the author's intended stance, underscores their advanced understanding of language \cite{brown2020language, chowdhery2022palm, wei2023larger}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2302.04023" id="8444">
        <attvalues>
          <attvalue for="5" value=" This methodological shift has not only simplified the application of LLMs but also expanded their utility, making them particularly effective for tasks that require a deep understanding of language nuances, such as stance detection \cite{bang-etal-2023-multitask, ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2303.16199" id="8445">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning tailors LLMs like ChatGPT, LLaMa-2, and Mistral-7B to specific tasks, significantly enhancing their precision and relevance for contextually aware stance detection on social media platforms \cite{zhang2023llamaadapter}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2304.03262" id="8446">
        <attvalues>
          <attvalue for="5" value=" The flexibility of LLMs in social media analysis is highlighted by their use of advanced techniques such as fine-tuning, chain of thought prompting \cite{chen2023you}, and both zero-shot and few-shot learning \cite{zhang2023stance, cruickshank2023use, aiyappa-etal-2023-trust}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2212.14548" id="8447">
        <attvalues>
          <attvalue for="5" value=" The flexibility of LLMs in social media analysis is highlighted by their use of advanced techniques such as fine-tuning, chain of thought prompting \cite{chen2023you}, and both zero-shot and few-shot learning \cite{zhang2023stance, cruickshank2023use, aiyappa-etal-2023-trust}." />
        </attvalues>
      </edge>
      <edge source="2404.12171" target="2310.19750" id="8448">
        <attvalues>
          <attvalue for="5" value=" These methods help navigate complex online discourse, enhancing the models' ability to interpret both explicit and implicit content effectively \cite{gatto2023chainofthought, lan2023stance}." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="1909.08053" id="8449">
        <attvalues>
          <attvalue for="5" value=" We see an increasingly larger transformer become a better language model \cite{gpt,gpt2,megatron,gpt3}." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="2005.14165" id="8450">
        <attvalues>
          <attvalue for="5" value=" In this scale approach, the knowledge is implicitly represented in the weights of a parametric neural network, and it is not straightforward to interpret whether a model contains a particular knowledge without asking the model to produce a response---eg, via a cloze-style question \cite{lmiskb} or a prompt \cite{gpt3}." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="1909.01066" id="8451">
        <attvalues>
          <attvalue for="5" value=" In this scale approach, the knowledge is implicitly represented in the weights of a parametric neural network, and it is not straightforward to interpret whether a model contains a particular knowledge without asking the model to produce a response---eg, via a cloze-style question \cite{lmiskb} or a prompt \cite{gpt3}." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="1612.04426" id="8452">
        <attvalues>
          <attvalue for="5" value=" In contrast to previous language models that either interpolate output probabilities \cite{smerity,grave,knnlm,bertknn} or use input concatenation \cite{realm,megatronctrl} to combine information from different sources, we design a context-dependent gating mechanism to incorporate local, extended, and global context." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="1901.02860" id="8453">
        <attvalues>
          <attvalue for="5" value=" Specifically, we follow transformer-XL \cite{txl} to capture extended context by caching hidden states in a temporary short-term memory." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="1911.05507" id="8454">
        <attvalues>
          <attvalue for="5" value=" For example, attempts to incorporate extended local context to a neural network---such as those found in neural cache \cite{grave}, transformer-XL \cite{txl} compressive transformer \cite{compresstrans}, performers \cite{performer}, \revision{longformer \cite{longformer}, and reformer \cite{reformer}}---can be seen as models of working memory." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="2009.14794" id="8455">
        <attvalues>
          <attvalue for="5" value=" For example, attempts to incorporate extended local context to a neural network---such as those found in neural cache \cite{grave}, transformer-XL \cite{txl} compressive transformer \cite{compresstrans}, performers \cite{performer}, \revision{longformer \cite{longformer}, and reformer \cite{reformer}}---can be seen as models of working memory." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="2004.05150" id="8456">
        <attvalues>
          <attvalue for="5" value=" For example, attempts to incorporate extended local context to a neural network---such as those found in neural cache \cite{grave}, transformer-XL \cite{txl} compressive transformer \cite{compresstrans}, performers \cite{performer}, \revision{longformer \cite{longformer}, and reformer \cite{reformer}}---can be seen as models of working memory." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="2001.04451" id="8457">
        <attvalues>
          <attvalue for="5" value=" For example, attempts to incorporate extended local context to a neural network---such as those found in neural cache \cite{grave}, transformer-XL \cite{txl} compressive transformer \cite{compresstrans}, performers \cite{performer}, \revision{longformer \cite{longformer}, and reformer \cite{reformer}}---can be seen as models of working memory." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="1911.00172" id="8458">
        <attvalues>
          <attvalue for="5" value=" In contrast to previous language models that either interpolate output probabilities \cite{smerity,grave,knnlm,bertknn} or use input concatenation \cite{realm,megatronctrl} to combine information from different sources, we design a context-dependent gating mechanism to incorporate local, extended, and global context." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="1906.01076" id="8459">
        <attvalues>
          <attvalue for="5" value=" Models of episodic memory include $k$NN-LM \cite{knnlm} and architectures that are designed for more complicated tasks such as question answering \cite{demasson, realm} and machine translation \cite{nnmt}." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="2010.00710" id="8460">
        <attvalues>
          <attvalue for="5" value=" Models of episodic memory include $k$NN-LM \cite{knnlm} and architectures that are designed for more complicated tasks such as question answering \cite{demasson, realm} and machine translation \cite{nnmt}." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="1609.07843" id="8461">
        <attvalues>
          <attvalue for="5" value=" In contrast to previous language models that either interpolate output probabilities \cite{smerity,grave,knnlm,bertknn} or use input concatenation \cite{realm,megatronctrl} to combine information from different sources, we design a context-dependent gating mechanism to incorporate local, extended, and global context." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="2005.00766" id="8462">
        <attvalues>
          <attvalue for="5" value=" In contrast to previous language models that either interpolate output probabilities \cite{smerity,grave,knnlm,bertknn} or use input concatenation \cite{realm,megatronctrl} to combine information from different sources, we design a context-dependent gating mechanism to incorporate local, extended, and global context." />
        </attvalues>
      </edge>
      <edge source="2102.02557" target="2002.08909" id="8463">
        <attvalues>
          <attvalue for="5" value=" In contrast to previous language models that either interpolate output probabilities \cite{smerity,grave,knnlm,bertknn} or use input concatenation \cite{realm,megatronctrl} to combine information from different sources, we design a context-dependent gating mechanism to incorporate local, extended, and global context." />
        </attvalues>
      </edge>
      <edge source="2308.06507" target="2205.09073" id="8465">
        <attvalues>
          <attvalue for="5" value=" Besides the above ways, \cite{DBLP:conf/icml/DaiCZARGG22} propose Dialog Inpainting, which creates information-seeking dialogues by inserting utterances between neighboring sentences in documents." />
        </attvalues>
      </edge>
      <edge source="2308.06507" target="2106.07499" id="8466">
        <attvalues>
          <attvalue for="5" value=" A promising way to alleviate this problem is data augmentation \cite{DBLP:journals/corr/abs-2106-07499}." />
        </attvalues>
      </edge>
      <edge source="2308.06507" target="1805.06201" id="8467">
        <attvalues>
          <attvalue for="5" value=" Traditional methods, including token-level manipulation \cite{DBLP:conf/naacl/Kobayashi18, DBLP:conf/emnlp/WeiZ19} and sentence-level paraphrasing \cite{DBLP:conf/acl/SennrichHB16}, improve the linguistic diversity of training data." />
        </attvalues>
      </edge>
      <edge source="2308.06507" target="1511.06709" id="8468">
        <attvalues>
          <attvalue for="5" value=" Traditional methods, including token-level manipulation \cite{DBLP:conf/naacl/Kobayashi18, DBLP:conf/emnlp/WeiZ19} and sentence-level paraphrasing \cite{DBLP:conf/acl/SennrichHB16}, improve the linguistic diversity of training data." />
        </attvalues>
      </edge>
      <edge source="2308.06507" target="2112.08342" id="8469">
        <attvalues>
          <attvalue for="5" value=" Another line of research focuses on simulation-based methods \cite{DBLP:journals/corr/abs-2112-08342, DBLP:journals/corr/abs-2205-12609}." />
        </attvalues>
      </edge>
      <edge source="2308.06507" target="2005.14165" id="8470">
        <attvalues>
          <attvalue for="5" value=" To alleviate the above issues, we propose a simple yet effective method AutoConv for Automatically generating information-seeking Conversations, which takes advantage of the few-shot learning ability and generation capacity of large language models (LLM) \cite{DBLP:conf/nips/BrownMRSKDNSSAA20}." />
        </attvalues>
      </edge>
      <edge source="2308.06507" target="1808.07042" id="8471">
        <attvalues>
          <attvalue for="5" value=" We conduct comprehensive experiments on two frequently-used datasets QuAC \cite{DBLP:conf/emnlp/ChoiHIYYCLZ18} and CoQA \cite{DBLP:journals/tacl/ReddyCM19} in the low-resource setting, where only dozens of human dialogues are available." />
        </attvalues>
      </edge>
      <edge source="2205.05590" target="1810.07455" id="8472">
        <attvalues>
          <attvalue for="5" value=" These approaches use either transcripts~\cite{tran2017a, ji2016a, shen2016neural} or a combination of transcript and audio~\cite{he2018exploring, ortega2018lexico, julia2010dialog} to predict DA." />
        </attvalues>
      </edge>
      <edge source="2205.05590" target="2004.11419" id="8473">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{dang2020endtoend} introduced an end-to-end (E2E) DAC approach, where DAs are directly inferred from audio signals." />
        </attvalues>
      </edge>
      <edge source="2205.05590" target="2105.00260" id="8474">
        <attvalues>
          <attvalue for="5" value=" As highlighted in~\cite{wallbridge2021it}, it represents the non-lexical channel that serves a fundamental role in speech communication among humans." />
        </attvalues>
      </edge>
      <edge source="2205.05590" target="1803.00831" id="8475">
        <attvalues>
          <attvalue for="5" value=" We compare our proposed model with previous E2E DAC models \cite{ortega2018lexico, he2018exploring} {that only use spectral-based audio features}." />
        </attvalues>
      </edge>
      <edge source="2401.14630" target="2208.01815" id="8476">
        <attvalues>
          <attvalue for="5" value=" It is a challenging task in Chinese Natural Language Processing (NLP) and holds substantial importance for various downstream applications, including Optical Character Recognition (OCR)~\cite{afli-etal-2016-using}, Automatic Speech Recognition (ASR)~\cite{gao-etal-2010-large}, AI Writing Assistant~\cite{DBLP:journals/corr/abs-2208-01815}, and numerous other tasks that contain Chinese texts." />
        </attvalues>
      </edge>
      <edge source="2401.14630" target="2005.07421" id="8478">
        <attvalues>
          <attvalue for="5" value=" Furthermore, it is important to note that several existing models~\cite{hong-etal-2019-FASPell,Softmasked,spellgcn,Lv_2022} rely on supervised learning approaches which are trained on datasets with limited coverage of terminologies from specialized domains." />
        </attvalues>
      </edge>
      <edge source="2401.14630" target="2004.14166" id="8479">
        <attvalues>
          <attvalue for="5" value=" Furthermore, it is important to note that several existing models~\cite{hong-etal-2019-FASPell,Softmasked,spellgcn,Lv_2022} rely on supervised learning approaches which are trained on datasets with limited coverage of terminologies from specialized domains." />
        </attvalues>
      </edge>
      <edge source="2401.14630" target="2203.10929" id="8480">
        <attvalues>
          <attvalue for="5" value=" There are works such as ECSPell~\cite{Lv_2022} that raises the issue of the domain adaptation problem in the CSC area, and evaluates some state-of-the-art models on the human-annotated domain datasets." />
        </attvalues>
      </edge>
      <edge source="2401.14630" target="2209.07068" id="8481">
        <attvalues>
          <attvalue for="5" value=" To verify this conjecture, we also conduct cross-domain evaluation on some unsupervised CSC methods such as uChecker~\cite{li-2022-uchecker} which can preserve the knowledge of the original pretrained language models as much as possible." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2005.14165" id="8482">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) are trained on vast amounts of data in a self-supervised fashion, which has shown promising performance in a variety of zero-shot and few-shot tasks \cite{gpt3,Chowdhery2022PaLMSL}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2204.02311" id="8483">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) are trained on vast amounts of data in a self-supervised fashion, which has shown promising performance in a variety of zero-shot and few-shot tasks \cite{gpt3,Chowdhery2022PaLMSL}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2301.13688" id="8485">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning these models on a diverse set of tasks allows them to handle unseen tasks following natural language instructions~\cite{ouyang2022training,longpre2023flan,taori2023alpaca,gpt4all}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2211.05100" id="8487">
        <attvalues>
          <attvalue for="5" value=" As a result, existing open-source LLMs such as XGLM~\cite{lin-etal-2022-xglm}, BLOOM~\cite{scao2022bloom}, and LLaMA~\cite{touvron2023llama} perform relatively poor on these languages, some of which are entirely overlooked." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2203.15556" id="8488">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, most LLMs are developed for English, such as LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, Chinchilla~\cite{hoffmann2022chinchilla}, OPT~\cite{zhang2022opt}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2205.01068" id="8489">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, most LLMs are developed for English, such as LLaMA~\cite{touvron2023llama}, BLOOM~\cite{scao2022bloom}, Chinchilla~\cite{hoffmann2022chinchilla}, OPT~\cite{zhang2022opt}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2001.08361" id="8490">
        <attvalues>
          <attvalue for="5" value=" Due to the insufficient high-quality internet data, LLM capabilities on low-resource languages fail to be easily improved through expanding their data size like English~\cite{kaplan2020scaling,rae2021scaling,biderman2023pythia}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2304.01373" id="8491">
        <attvalues>
          <attvalue for="5" value=" Due to the insufficient high-quality internet data, LLM capabilities on low-resource languages fail to be easily improved through expanding their data size like English~\cite{kaplan2020scaling,rae2021scaling,biderman2023pythia}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2010.11934" id="8492">
        <attvalues>
          <attvalue for="5" value=" To construct PolyLM, we leverage a massive dataset of 640B tokens, culled from publicly available sources such as Wikipedia, mC4~\cite{xue2020mt5}, CC-100~\cite{Conneau2019UnsupervisedCR}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="1911.02116" id="8493">
        <attvalues>
          <attvalue for="5" value=" In order to assess the multilingual capabilities of LLM, we curate a benchmark derived from existing multilingual tasks (Section \ref{sec4}), including QA~\cite{tydiqa}, understanding~\cite{Conneau2019UnsupervisedCR,Yang2019PAWSXAC,tikhonov2021heads,ponti2020xcopa}, generation~\cite{Chen2021MTGAB}, and cross-lingual machine translation~\cite{barrault-etal-2020-findings}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2211.01786" id="8494">
        <attvalues>
          <attvalue for="5" value=" On the one hand, extant multilingual SFT datasets, eg xP3-MT~\cite{muennighoff2022crosslingual}, are acquired via machine translation, which potentially yields a style of translationese, a lack of cultural nuances, as well as translation errors." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2003.05002" id="8495">
        <attvalues>
          <attvalue for="5" value=" In order to assess the multilingual capabilities of LLM, we curate a benchmark derived from existing multilingual tasks (Section \ref{sec4}), including QA~\cite{tydiqa}, understanding~\cite{Conneau2019UnsupervisedCR,Yang2019PAWSXAC,tikhonov2021heads,ponti2020xcopa}, generation~\cite{Chen2021MTGAB}, and cross-lingual machine translation~\cite{barrault-etal-2020-findings}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="1908.11828" id="8496">
        <attvalues>
          <attvalue for="5" value=" In order to assess the multilingual capabilities of LLM, we curate a benchmark derived from existing multilingual tasks (Section \ref{sec4}), including QA~\cite{tydiqa}, understanding~\cite{Conneau2019UnsupervisedCR,Yang2019PAWSXAC,tikhonov2021heads,ponti2020xcopa}, generation~\cite{Chen2021MTGAB}, and cross-lingual machine translation~\cite{barrault-etal-2020-findings}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2106.12066" id="8497">
        <attvalues>
          <attvalue for="5" value=" In order to assess the multilingual capabilities of LLM, we curate a benchmark derived from existing multilingual tasks (Section \ref{sec4}), including QA~\cite{tydiqa}, understanding~\cite{Conneau2019UnsupervisedCR,Yang2019PAWSXAC,tikhonov2021heads,ponti2020xcopa}, generation~\cite{Chen2021MTGAB}, and cross-lingual machine translation~\cite{barrault-etal-2020-findings}." />
        </attvalues>
      </edge>
      <edge source="2307.06018" target="2005.00333" id="8498">
        <attvalues>
          <attvalue for="5" value=" In order to assess the multilingual capabilities of LLM, we curate a benchmark derived from existing multilingual tasks (Section \ref{sec4}), including QA~\cite{tydiqa}, understanding~\cite{Conneau2019UnsupervisedCR,Yang2019PAWSXAC,tikhonov2021heads,ponti2020xcopa}, generation~\cite{Chen2021MTGAB}, and cross-lingual machine translation~\cite{barrault-etal-2020-findings}." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="1706.03762" id="8499">
        <attvalues>
          <attvalue for="5" value=" In recent years, automatic speech recognition (ASR) using self-attention (SA) \cite{vaswani2017attention} has attracted considerable attention." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="1909.06317" id="8500">
        <attvalues>
          <attvalue for="5" value=" Both transformer-based speech recognition \cite{dong2018speech,karita2019comparative,mohamed2019transformers,zeyer2019comparison,chang2020end} and hybrid \cite{povey2018time,wang2019transformer} and connectionist temporal classification (CTC) \cite{pham2019very,salazar2019self} models have shown a high recognition performance with SA." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="2002.03921" id="8502">
        <attvalues>
          <attvalue for="5" value=" Both transformer-based speech recognition \cite{dong2018speech,karita2019comparative,mohamed2019transformers,zeyer2019comparison,chang2020end} and hybrid \cite{povey2018time,wang2019transformer} and connectionist temporal classification (CTC) \cite{pham2019very,salazar2019self} models have shown a high recognition performance with SA." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="1910.09799" id="8503">
        <attvalues>
          <attvalue for="5" value=" Both transformer-based speech recognition \cite{dong2018speech,karita2019comparative,mohamed2019transformers,zeyer2019comparison,chang2020end} and hybrid \cite{povey2018time,wang2019transformer} and connectionist temporal classification (CTC) \cite{pham2019very,salazar2019self} models have shown a high recognition performance with SA." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="1904.13377" id="8504">
        <attvalues>
          <attvalue for="5" value=" Both transformer-based speech recognition \cite{dong2018speech,karita2019comparative,mohamed2019transformers,zeyer2019comparison,chang2020end} and hybrid \cite{povey2018time,wang2019transformer} and connectionist temporal classification (CTC) \cite{pham2019very,salazar2019self} models have shown a high recognition performance with SA." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="1901.10055" id="8505">
        <attvalues>
          <attvalue for="5" value=" Both transformer-based speech recognition \cite{dong2018speech,karita2019comparative,mohamed2019transformers,zeyer2019comparison,chang2020end} and hybrid \cite{povey2018time,wang2019transformer} and connectionist temporal classification (CTC) \cite{pham2019very,salazar2019self} models have shown a high recognition performance with SA." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="1803.09519" id="8506">
        <attvalues>
          <attvalue for="5" value=" Masking \cite{sperber2018self} limits the range of self-attention by using a Gaussian window, whereas relative positional encoding \cite{shaw2018self,pham2020relative} uses relative embedding in a self-attention architecture to eliminate the effect of the length mismatch." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="1803.02155" id="8507">
        <attvalues>
          <attvalue for="5" value=" Masking \cite{sperber2018self} limits the range of self-attention by using a Gaussian window, whereas relative positional encoding \cite{shaw2018self,pham2020relative} uses relative embedding in a self-attention architecture to eliminate the effect of the length mismatch." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="2005.09940" id="8508">
        <attvalues>
          <attvalue for="5" value=" Masking \cite{sperber2018self} limits the range of self-attention by using a Gaussian window, whereas relative positional encoding \cite{shaw2018self,pham2020relative} uses relative embedding in a self-attention architecture to eliminate the effect of the length mismatch." />
        </attvalues>
      </edge>
      <edge source="2102.09168" target="2001.04451" id="8509">
        <attvalues>
          <attvalue for="5" value=" Inspired by the mathematical expression based on the shared-QK attention used in Reformer \cite{kitaev2020reformer}, in this paper, yet another self-attention reformulation based on a Gaussian kernel is proposed." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2311.04939" id="8511">
        <attvalues>
          <attvalue for="5" value=" Effective long-sequence reasoning in large language models (LLMs) is crucial for a wide range of applications~\cite{re_longer_2022,li2023loogle}, from understanding extensive texts~\cite{tay2020long, kryscinski2021booksum} and managing day-long conversations~\cite{zhang2021summ,zhong2022dialoglm} to code generation~\cite{du2023classeval,zheng2023codegeex} and science discoveries~\cite{varadi2022alphafold,song2023deepspeed4science}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2011.04006" id="8512">
        <attvalues>
          <attvalue for="5" value=" Effective long-sequence reasoning in large language models (LLMs) is crucial for a wide range of applications~\cite{re_longer_2022,li2023loogle}, from understanding extensive texts~\cite{tay2020long, kryscinski2021booksum} and managing day-long conversations~\cite{zhang2021summ,zhong2022dialoglm} to code generation~\cite{du2023classeval,zheng2023codegeex} and science discoveries~\cite{varadi2022alphafold,song2023deepspeed4science}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2109.02492" id="8513">
        <attvalues>
          <attvalue for="5" value=" Effective long-sequence reasoning in large language models (LLMs) is crucial for a wide range of applications~\cite{re_longer_2022,li2023loogle}, from understanding extensive texts~\cite{tay2020long, kryscinski2021booksum} and managing day-long conversations~\cite{zhang2021summ,zhong2022dialoglm} to code generation~\cite{du2023classeval,zheng2023codegeex} and science discoveries~\cite{varadi2022alphafold,song2023deepspeed4science}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2308.01861" id="8514">
        <attvalues>
          <attvalue for="5" value=" Effective long-sequence reasoning in large language models (LLMs) is crucial for a wide range of applications~\cite{re_longer_2022,li2023loogle}, from understanding extensive texts~\cite{tay2020long, kryscinski2021booksum} and managing day-long conversations~\cite{zhang2021summ,zhong2022dialoglm} to code generation~\cite{du2023classeval,zheng2023codegeex} and science discoveries~\cite{varadi2022alphafold,song2023deepspeed4science}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2310.04610" id="8515">
        <attvalues>
          <attvalue for="5" value=" Effective long-sequence reasoning in large language models (LLMs) is crucial for a wide range of applications~\cite{re_longer_2022,li2023loogle}, from understanding extensive texts~\cite{tay2020long, kryscinski2021booksum} and managing day-long conversations~\cite{zhang2021summ,zhong2022dialoglm} to code generation~\cite{du2023classeval,zheng2023codegeex} and science discoveries~\cite{varadi2022alphafold,song2023deepspeed4science}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2307.08691" id="8516">
        <attvalues>
          <attvalue for="5" value=" Recent system support advancements~\cite{dao2023flashattention2, jacobs2023deepspeed} have enabled training transformers for any $L$ sequence length even with $O(L^2)$ computational complexity." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2401.04088" id="8517">
        <attvalues>
          <attvalue for="5" value=" This is exemplified by models such as MPT~\cite{MosaicML2023Introducing} and Mistral~\cite{jiang2024mixtral} pre-trained with sequence lengths 16k and 32k respectively." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2307.03172" id="8518">
        <attvalues>
          <attvalue for="5" value=" Particularly, \cite{liu2023lost} demonstrated a substantial degradation in LLMs' performance when crucial information is positioned amidst a lengthy context, a phenomenon they refer to as ``lost-in-the-middle&quot;." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2104.09864" id="8519">
        <attvalues>
          <attvalue for="5" value=" One explanation is about the use of rotary positional embedding (RoPE) \cite{su2024roformer}, a prevalent positional encoding technique used in open-source LLMs." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2306.15595" id="8521">
        <attvalues>
          <attvalue for="5" value=" However, this approach was initially introduced to extend context windows, and its performance regarding the ``lost-in-the-middle'' problem remains uncertain for several reasons: (i) Indice re-scaling forces position embeddings of original context window to reside in a narrower region, leading to performance degradation in the original context window as shown in \cite{chen2023extending}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2306.07174" id="8522">
        <attvalues>
          <attvalue for="5" value=" These include extra memory bank \cite{LongMem}, reordering the input context based on relevance \cite{peysakhovich2023attention,chen2023fortify}, enhancing the information searching and reflection ability via attention strengthening tasks~\cite{junqing2023never,xu2023retrieval}, splitting the input into short segments and applying short-text models~\cite{ivgi2023efficient}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2310.01427" id="8523">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{peysakhovich2023attention} empirically discovered that LLMs tend to emphasize more on the current window while still paying more attention to the relevant text than distracting content." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2312.04455" id="8524">
        <attvalues>
          <attvalue for="5" value=" \cite{chen2023fortify} conducted parallel runs of LLMs with different RoPE angles, thereby mitigating the risk of overlooking crucial information through a weighted sum of the outputs." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2310.03025" id="8525">
        <attvalues>
          <attvalue for="5" value=" These include extra memory bank \cite{LongMem}, reordering the input context based on relevance \cite{peysakhovich2023attention,chen2023fortify}, enhancing the information searching and reflection ability via attention strengthening tasks~\cite{junqing2023never,xu2023retrieval}, splitting the input into short segments and applying short-text models~\cite{ivgi2023efficient}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2208.00748" id="8526">
        <attvalues>
          <attvalue for="5" value=" These include extra memory bank \cite{LongMem}, reordering the input context based on relevance \cite{peysakhovich2023attention,chen2023fortify}, enhancing the information searching and reflection ability via attention strengthening tasks~\cite{junqing2023never,xu2023retrieval}, splitting the input into short segments and applying short-text models~\cite{ivgi2023efficient}." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="1409.4842" id="8527">
        <attvalues>
          <attvalue for="5" value=" Multi-scale features, well-established in Inception-style models \cite{szegedy2015going, szegedy2016rethinking, guo2022segnext}, utilize parallel employment of kernels with different sizes to fuse multi-scale information, spanning short to long distances." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2307.09288" id="8528">
        <attvalues>
          <attvalue for="5" value=" By simply re-scaling the indices of positional encoding, Ms-PoE consistently enhances the performance of various LLMs including Llama-2~\cite{touvron2023llama}, StableBeluga~\cite{StableBelugaModels} and Vicuna~\cite{vicuna2023} on the ZeroSCROLLS benchmark \cite{shaham2023zeroscrolls}, achieving a notable average accuracy gain of up to 3." />
        </attvalues>
      </edge>
      <edge source="2403.04797" target="2305.14196" id="8529">
        <attvalues>
          <attvalue for="5" value=" By simply re-scaling the indices of positional encoding, Ms-PoE consistently enhances the performance of various LLMs including Llama-2~\cite{touvron2023llama}, StableBeluga~\cite{StableBelugaModels} and Vicuna~\cite{vicuna2023} on the ZeroSCROLLS benchmark \cite{shaham2023zeroscrolls}, achieving a notable average accuracy gain of up to 3." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="1911.02116" id="8532">
        <attvalues>
          <attvalue for="5" value=" In particular, SCWE such as XL-LEXEME~\cite{cassotti-etal-2023-xl} obtained by fine-tuning MLM such as XLM-RoBERTa~\cite{conneau-etal-2020-xlmroberta} on WiC~\cite{pilehvar-camacho-collados-2019-wic} have reported superior performance in SCD benchmarks~\cite{cassotti-etal-2023-xl,aida-bollegala-2023-swap,periti-tahmasebi-2024-systematic,aida-bollegala-2024-semantic}, implying that semantic changes can be accurately inferred from SCWE." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="2110.02204" id="8533">
        <attvalues>
          <attvalue for="5" value=" A large body of methods have been proposed to represent the meaning of a word in a given context~\cite{BERT,conneau-etal-2020-xlmroberta,yi-zhou-2021-learning,rachinskiy-arefyev-2021-glossreader,periti-etal-2024-automatically}, or within a given time period~\cite{hamilton-etal-2016-diachronic,rosenfeld-erk-2018-deep,aida-etal-2021-comprehensive,rosin-etal-2022-time,aida-bollegala-2023-unsupervised,Tang2023-yu, fedorova-etal-2024-definition}." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="2110.06366" id="8535">
        <attvalues>
          <attvalue for="5" value=" A large body of methods have been proposed to represent the meaning of a word in a given context~\cite{BERT,conneau-etal-2020-xlmroberta,yi-zhou-2021-learning,rachinskiy-arefyev-2021-glossreader,periti-etal-2024-automatically}, or within a given time period~\cite{hamilton-etal-2016-diachronic,rosenfeld-erk-2018-deep,aida-etal-2021-comprehensive,rosin-etal-2022-time,aida-bollegala-2023-unsupervised,Tang2023-yu, fedorova-etal-2024-definition}." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="2402.12011" id="8538">
        <attvalues>
          <attvalue for="5" value=" In particular, SCWE such as XL-LEXEME~\cite{cassotti-etal-2023-xl} obtained by fine-tuning MLM such as XLM-RoBERTa~\cite{conneau-etal-2020-xlmroberta} on WiC~\cite{pilehvar-camacho-collados-2019-wic} have reported superior performance in SCD benchmarks~\cite{cassotti-etal-2023-xl,aida-bollegala-2023-swap,periti-tahmasebi-2024-systematic,aida-bollegala-2024-semantic}, implying that semantic changes can be accurately inferred from SCWE." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="2403.00226" id="8539">
        <attvalues>
          <attvalue for="5" value=" In particular, SCWE such as XL-LEXEME~\cite{cassotti-etal-2023-xl} obtained by fine-tuning MLM such as XLM-RoBERTa~\cite{conneau-etal-2020-xlmroberta} on WiC~\cite{pilehvar-camacho-collados-2019-wic} have reported superior performance in SCD benchmarks~\cite{cassotti-etal-2023-xl,aida-bollegala-2023-swap,periti-tahmasebi-2024-systematic,aida-bollegala-2024-semantic}, implying that semantic changes can be accurately inferred from SCWE." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="2010.06478" id="8540">
        <attvalues>
          <attvalue for="5" value=" First, in \ref{sec:contextual}, we investigate the embedding dimensions specific to the contextual semantic changes of words using WiC benchmarks~\cite{pilehvar-camacho-collados-2019-wic,raganato-etal-2020-xlwic,martelli-etal-2021-mclwic,liu-etal-2021-am2ico} as the evaluation task." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="2104.08639" id="8541">
        <attvalues>
          <attvalue for="5" value=" First, in \ref{sec:contextual}, we investigate the embedding dimensions specific to the contextual semantic changes of words using WiC benchmarks~\cite{pilehvar-camacho-collados-2019-wic,raganato-etal-2020-xlwic,martelli-etal-2021-mclwic,liu-etal-2021-am2ico} as the evaluation task." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="2007.11464" id="8542">
        <attvalues>
          <attvalue for="5" value=" Second, in \ref{sec:temporal}, we investigate the embedding dimensions specific to the temporal semantic changes of words on SemEval-2020 Task 1~\cite{schlechtweg-etal-2020-semeval} benchmark." />
        </attvalues>
      </edge>
      <edge source="2407.02820" target="2305.13175" id="8543">
        <attvalues>
          <attvalue for="5" value=" In each setting, we compare pre-trained CWE and the SCWE obtained by fine-tuning on WiC using PCA and ICA, which have been used in prior work investigating dimensions in CWE~\cite{yamagiwa-etal-2023-discovering}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="1911.01464" id="8544">
        <attvalues>
          <attvalue for="5" value=" Large pretrained multilingual transformer models succeed at a variety of multilingual and monolingual tasks and can be used in transfer learning paradigms, where a model is trained to do a task in one language and then transferred to another language \cite{lauscher-etal-2020-zero,conneau-etal-2020-emerging, wu-dredze-2019-beto, wu-dredze-2020-languages, pires-etal-2019-multilingual,vulic-etal-2020-probing,rust-etal-2021-good}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2010.05731" id="8545">
        <attvalues>
          <attvalue for="5" value=" Large pretrained multilingual transformer models succeed at a variety of multilingual and monolingual tasks and can be used in transfer learning paradigms, where a model is trained to do a task in one language and then transferred to another language \cite{lauscher-etal-2020-zero,conneau-etal-2020-emerging, wu-dredze-2019-beto, wu-dredze-2020-languages, pires-etal-2019-multilingual,vulic-etal-2020-probing,rust-etal-2021-good}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2012.15613" id="8546">
        <attvalues>
          <attvalue for="5" value=" Large pretrained multilingual transformer models succeed at a variety of multilingual and monolingual tasks and can be used in transfer learning paradigms, where a model is trained to do a task in one language and then transferred to another language \cite{lauscher-etal-2020-zero,conneau-etal-2020-emerging, wu-dredze-2019-beto, wu-dredze-2020-languages, pires-etal-2019-multilingual,vulic-etal-2020-probing,rust-etal-2021-good}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2005.04511" id="8547">
        <attvalues>
          <attvalue for="5" value=" These abilities have spurred a spate of papers probing the internal workings and capabilities of multilingual models, suggesting that such models may contain language-independent, along with langauge-specific knowledge of interesting linguistic structure \cite[eg,][]{chi-etal-2020-finding,papadimitriou-etal-2021-deep,ravishankar-etal-2021-attention,blevins-etal-2022-analyzing,gonen-etal-2020-greek}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2101.11043" id="8548">
        <attvalues>
          <attvalue for="5" value=" These abilities have spurred a spate of papers probing the internal workings and capabilities of multilingual models, suggesting that such models may contain language-independent, along with langauge-specific knowledge of interesting linguistic structure \cite[eg,][]{chi-etal-2020-finding,papadimitriou-etal-2021-deep,ravishankar-etal-2021-attention,blevins-etal-2022-analyzing,gonen-etal-2020-greek}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2101.10927" id="8549">
        <attvalues>
          <attvalue for="5" value=" These abilities have spurred a spate of papers probing the internal workings and capabilities of multilingual models, suggesting that such models may contain language-independent, along with langauge-specific knowledge of interesting linguistic structure \cite[eg,][]{chi-etal-2020-finding,papadimitriou-etal-2021-deep,ravishankar-etal-2021-attention,blevins-etal-2022-analyzing,gonen-etal-2020-greek}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2010.08275" id="8551">
        <attvalues>
          <attvalue for="5" value=" \cite{gonen-etal-2020-greek} also show that, by subtracting an ``average'' representation of language $X$ from a particular token embedding and then adding the average language $Y$ embedding, one can obtain a translation of the token in language $Y$ by analogy." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2004.03061" id="8554">
        <attvalues>
          <attvalue for="5" value=" While the results of this literature are suggestive, probing methods are susceptible to memorizing the original input and may not reflect what information models actually use downstream \cite{hewitt-liang-2019-designing,elazar-etal-2021-amnesic,pimentel-etal-2020-information,voita-etal-2021-analyzing}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2010.10907" id="8555">
        <attvalues>
          <attvalue for="5" value=" While the results of this literature are suggestive, probing methods are susceptible to memorizing the original input and may not reflect what information models actually use downstream \cite{hewitt-liang-2019-designing,elazar-etal-2021-amnesic,pimentel-etal-2020-information,voita-etal-2021-analyzing}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2106.02997" id="8556">
        <attvalues>
          <attvalue for="5" value=" It is thus desirable to test not only what information can be extracted but what information is actually used \cite{geiger2021causal,finlayson-etal-2021-causal,lasri-etal-2022-probing}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2106.06087" id="8557">
        <attvalues>
          <attvalue for="5" value=" It is thus desirable to test not only what information can be extracted but what information is actually used \cite{geiger2021causal,finlayson-etal-2021-causal,lasri-etal-2022-probing}." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2105.06965" id="8559">
        <attvalues>
          <attvalue for="5" value=" Whereas \cite{ravfogel-etal-2021-counterfactual} use AlterRep to explore syntactic representations in models, our hypothesis is that the same kind of causal manipulation could be informative as to how multilingual models process multilingual text." />
        </attvalues>
      </edge>
      <edge source="2310.18862" target="2004.07667" id="8560">
        <attvalues>
          <attvalue for="5" value=" To do that we apply AlterRep \cite{ravfogel-etal-2021-counterfactual}, an offshoot of Iterative Nullspace Projection \cite[INLP;][]{ravfogel-etal-2020-null,elazar-etal-2021-amnesic}, in a multilingual setting." />
        </attvalues>
      </edge>
      <edge source="2409.09659" target="2312.07819" id="8563">
        <attvalues>
          <attvalue for="5" value=" While \cite{zhang-2023}'s results indicate that LLMs achieve state-of-the-art performance on NLI, they only evaluate the performance of GPT-3." />
        </attvalues>
      </edge>
      <edge source="2409.09659" target="2005.14165" id="8564">
        <attvalues>
          <attvalue for="5" value="5 \cite{gpt3-brown:20} and GPT-4 \cite{openai-gpt4:23} to perform NLI." />
        </attvalues>
      </edge>
      <edge source="2409.09659" target="2303.08774" id="8565">
        <attvalues>
          <attvalue for="5" value="5 \cite{gpt3-brown:20} and GPT-4 \cite{openai-gpt4:23} to perform NLI." />
        </attvalues>
      </edge>
      <edge source="2409.09659" target="2402.03927" id="8566">
        <attvalues>
          <attvalue for="5" value=" Providers of closed-source models often disclose minimal information regarding the training data or procedure, hindering the evaluation of results achieved with these models and obscuring biases in training data and models \cite{balloccu-etal-2024-leak}." />
        </attvalues>
      </edge>
      <edge source="2409.09659" target="2308.10092" id="8567">
        <attvalues>
          <attvalue for="5" value=" In addition, closed-source models are typically only accessible via an API, causing lack of control over model updates, which are often communicated poorly to users \cite{yu-open-2023, pozzobon-etal-2023-challenges}." />
        </attvalues>
      </edge>
      <edge source="2204.04873" target="2012.15562" id="8569">
        <attvalues>
          <attvalue for="5" value=" While several works suggest that such knowledge transfer goes beyond just vocabulary sharing across languages \cite{artetxe-etal-2020-cross, k-etal-2021-analyzing}, others have shown that the models' performance is sensitive to the quality of the tokenization it relies on \cite{pfeiffer-etal-2021-unks}." />
        </attvalues>
      </edge>
      <edge source="2204.04873" target="2010.12858" id="8570">
        <attvalues>
          <attvalue for="5" value=" Recently proposed methods include continual pretraining of the model (restricted to the embedding layer training only in some cases) \cite{artetxe-etal-2020-cross,chau-etal-2020-parsing,muller-etal-2021-unseen,zhang-etal-2020-multi-stage,wang-etal-2020-extending}, or training of language-specific adapters \cite{pfeiffer-etal-2020-mad, pfeiffer-etal-2021-adapterfusion, pfeiffer-etal-2021-unks, philip-etal-2020-monolingual, ustun-etal-2021-multilingual, berard-2021-continual} for the target language." />
        </attvalues>
      </edge>
      <edge source="2204.04873" target="1911.02116" id="8571">
        <attvalues>
          <attvalue for="5" value=" Data availability, but also the curse of multilinguality \cite{conneau-etal-2020-unsupervised} makes training a single model covering all the languages challenging: there is a trade off between the number of languages in pretraining data, model capacity and the downstream performance for each individual language." />
        </attvalues>
      </edge>
      <edge source="2204.04873" target="2009.14124" id="8572">
        <attvalues>
          <attvalue for="5" value=" Recently proposed methods include continual pretraining of the model (restricted to the embedding layer training only in some cases) \cite{artetxe-etal-2020-cross,chau-etal-2020-parsing,muller-etal-2021-unseen,zhang-etal-2020-multi-stage,wang-etal-2020-extending}, or training of language-specific adapters \cite{pfeiffer-etal-2020-mad, pfeiffer-etal-2021-adapterfusion, pfeiffer-etal-2021-unks, philip-etal-2020-monolingual, ustun-etal-2021-multilingual, berard-2021-continual} for the target language." />
        </attvalues>
      </edge>
      <edge source="2204.04873" target="2010.05904" id="8573">
        <attvalues>
          <attvalue for="5" value=" Recently proposed methods include continual pretraining of the model (restricted to the embedding layer training only in some cases) \cite{artetxe-etal-2020-cross,chau-etal-2020-parsing,muller-etal-2021-unseen,zhang-etal-2020-multi-stage,wang-etal-2020-extending}, or training of language-specific adapters \cite{pfeiffer-etal-2020-mad, pfeiffer-etal-2021-adapterfusion, pfeiffer-etal-2021-unks, philip-etal-2020-monolingual, ustun-etal-2021-multilingual, berard-2021-continual} for the target language." />
        </attvalues>
      </edge>
      <edge source="2204.04873" target="2004.13640" id="8574">
        <attvalues>
          <attvalue for="5" value=" Recently proposed methods include continual pretraining of the model (restricted to the embedding layer training only in some cases) \cite{artetxe-etal-2020-cross,chau-etal-2020-parsing,muller-etal-2021-unseen,zhang-etal-2020-multi-stage,wang-etal-2020-extending}, or training of language-specific adapters \cite{pfeiffer-etal-2020-mad, pfeiffer-etal-2021-adapterfusion, pfeiffer-etal-2021-unks, philip-etal-2020-monolingual, ustun-etal-2021-multilingual, berard-2021-continual} for the target language." />
        </attvalues>
      </edge>
      <edge source="2204.04873" target="2005.00052" id="8575">
        <attvalues>
          <attvalue for="5" value=" Recently proposed methods include continual pretraining of the model (restricted to the embedding layer training only in some cases) \cite{artetxe-etal-2020-cross,chau-etal-2020-parsing,muller-etal-2021-unseen,zhang-etal-2020-multi-stage,wang-etal-2020-extending}, or training of language-specific adapters \cite{pfeiffer-etal-2020-mad, pfeiffer-etal-2021-adapterfusion, pfeiffer-etal-2021-unks, philip-etal-2020-monolingual, ustun-etal-2021-multilingual, berard-2021-continual} for the target language." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1003.4083" id="8576">
        <attvalues>
          <attvalue for="5" value=" The acoustic module predicts phonemes based on the input speech feature like Mel Frequency Cepstral Coefficient (MFCC) \cite{muda2010voice}." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1412.1602" id="8577">
        <attvalues>
          <attvalue for="5" value=" Attention mechanism aligns the relevant speech frames for predicting symbols at each output time step \cite{chorowski2014end,chorowski2015attention}." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1506.07503" id="8578">
        <attvalues>
          <attvalue for="5" value=" Attention mechanism aligns the relevant speech frames for predicting symbols at each output time step \cite{chorowski2014end,chorowski2015attention}." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1211.3711" id="8579">
        <attvalues>
          <attvalue for="5" value=" The initial versions of the encoder-decoder architecture for ASR modelled with recurrent neural network (RNN) as the main component for sequence processing \cite{graves2012sequence, graves2013speech}." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1406.1078" id="8580">
        <attvalues>
          <attvalue for="5" value=" Apart from the vanilla RNN, some other variations like long short-term memory (LSTM) \cite{hochreiter1997long}, gated recurrent unit (GRU) \cite{cho2014learning} are also popular in modelling sequential data." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1610.03022" id="8581">
        <attvalues>
          <attvalue for="5" value=" Convolutional neural networks (CNN) coupled with RNNs \cite{7953077} or stand-alone \cite{zhang2016towards} have also been used to make effective ASR models." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1701.02720" id="8582">
        <attvalues>
          <attvalue for="5" value=" Convolutional neural networks (CNN) coupled with RNNs \cite{7953077} or stand-alone \cite{zhang2016towards} have also been used to make effective ASR models." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1706.03762" id="8583">
        <attvalues>
          <attvalue for="5" value=" To address the limitations of RNN, Transformer network \cite{vaswani2017attention} has been recently proposed for sequence-to-sequence transduction." />
        </attvalues>
      </edge>
      <edge source="2102.07259" target="1904.02874" id="8584">
        <attvalues>
          <attvalue for="5" value=" Recently, some survey papers \cite{chaudhari2019attentive, galassi2020attention} have presented the development of attention-based models on natural language processing (NLP)." />
        </attvalues>
      </edge>
      <edge source="2310.14103" target="2109.01652" id="8585">
        <attvalues>
          <attvalue for="5" value=" A paradigm gaining traction is the use of instruction fine-tuned (IFT) models, LMs capable of following arbitrary instructions expressed in natural language \cite{wei2022finetuned, sanh2022multitask, ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2310.14103" target="1905.07830" id="8586">
        <attvalues>
          <attvalue for="5" value=" However, current metrics used to measure their performance are often task-specific \cite{zellers2019hellaswag, eval-harness}, or rely on automatic metrics designed for other intended purposes \cite{papineni-etal-2002-bleu, lin-2004-rouge}." />
        </attvalues>
      </edge>
      <edge source="2211.15613" target="2003.02739" id="8587">
        <attvalues>
          <attvalue for="5" value=" Zero-shot cross-lingual transfer, where models trained on a source language (eg, English) are directly applied to other target languages, has the potential to extend NLP systems to many languages \cite{nooralahzadeh-etal-2020-zero, keung-etal-2020-dont,chen2021model, niu-etal-2022-onealigner, huang-etal-2022-multilingual-generative}." />
        </attvalues>
      </edge>
      <edge source="2211.15613" target="2010.06127" id="8588">
        <attvalues>
          <attvalue for="5" value=" Zero-shot cross-lingual transfer, where models trained on a source language (eg, English) are directly applied to other target languages, has the potential to extend NLP systems to many languages \cite{nooralahzadeh-etal-2020-zero, keung-etal-2020-dont,chen2021model, niu-etal-2022-onealigner, huang-etal-2022-multilingual-generative}." />
        </attvalues>
      </edge>
      <edge source="2211.15613" target="2010.11934" id="8589">
        <attvalues>
          <attvalue for="5" value=" Recent work has shown that combining training data in a source language together with its automatic translation to the target language leads to consistent performance improvements \cite{xue2021mt5,hu2020xtreme}." />
        </attvalues>
      </edge>
      <edge source="2211.15613" target="1910.07475" id="8590">
        <attvalues>
          <attvalue for="5" value=" To avoid the use of complex word alignment models, several recent efforts \cite{lewis2020mlqa, hu2020xtreme} directly translated sentences with span annotations wrapped between special markers (eg, {&lt;a&gt;} and {&lt;/a&gt;})." />
        </attvalues>
      </edge>
      <edge source="2402.15610" target="2305.06500" id="8591">
        <attvalues>
          <attvalue for="5" value=" Instruction-tuned vision-and-language models (VLMs)~\cite{dai2023instructblip, liu2023improved, laurencon2023obelics, bai2023qwen} have achieved strong accuracy on reasoning benchmarks, which typically require VLMs to produce an answer for each instance." />
        </attvalues>
      </edge>
      <edge source="2402.15610" target="2204.13631" id="8592">
        <attvalues>
          <attvalue for="5" value=" However, a vanilla selective prediction system with low tolerance for incorrect predictions will abstain too frequently to be practical, even when the model answer may be correct~\cite{whitehead2022reliable}." />
        </attvalues>
      </edge>
      <edge source="2402.15610" target="2301.12597" id="8593">
        <attvalues>
          <attvalue for="5" value=" For example, if a user specifies that the BLIP2~\cite{li2023blip} predictions should be right at least 90\% of the time, vanilla selective prediction will make a prediction for just 4\% of A-OKVQA~\cite{schwenk2022okvqa} questions, with 94\% of the correct predictions being abstained on." />
        </attvalues>
      </edge>
      <edge source="2402.15610" target="2206.01718" id="8594">
        <attvalues>
          <attvalue for="5" value=" For example, if a user specifies that the BLIP2~\cite{li2023blip} predictions should be right at least 90\% of the time, vanilla selective prediction will make a prediction for just 4\% of A-OKVQA~\cite{schwenk2022okvqa} questions, with 94\% of the correct predictions being abstained on." />
        </attvalues>
      </edge>
      <edge source="2005.00932" target="1409.3215" id="8595">
        <attvalues>
          <attvalue for="5" value=" Neural machine translation (NMT) \cite{sutskever2014sequence, bahdanau2014neural} has achieved impressive performance in recent years, but the autoregressive decoding process limits the translation speed and restricts low-latency applications." />
        </attvalues>
      </edge>
      <edge source="2005.00932" target="1711.02281" id="8596">
        <attvalues>
          <attvalue for="5" value=" The decoding speedup for NAR models is typically 2-15$\times$ depending on the specific setup (eg, the number of length candidates, number of latent samples, etc), and NAR models can be tuned to achieve different trade-offs between time complexity and decoding quality \cite{gu2017non, wei2019imitation, ghazvininejad2019mask, ma2019flowseq}." />
        </attvalues>
      </edge>
      <edge source="2005.00932" target="1802.06901" id="8597">
        <attvalues>
          <attvalue for="5" value=" To mitigate this issue, many non-autoregressive (NAR) translation methods have been proposed, including latent space models \cite{gu2017non, ma2019flowseq, shu2019latent}, iterative refinement methods \cite{lee2018deterministic, ghazvininejad2019mask}, and alternative loss functions \cite{libovicky2018end, wang2019non, wei2019imitation, li2019hint, shao2019minimizing}." />
        </attvalues>
      </edge>
      <edge source="2005.00932" target="1811.04719" id="8598">
        <attvalues>
          <attvalue for="5" value=" To mitigate this issue, many non-autoregressive (NAR) translation methods have been proposed, including latent space models \cite{gu2017non, ma2019flowseq, shu2019latent}, iterative refinement methods \cite{lee2018deterministic, ghazvininejad2019mask}, and alternative loss functions \cite{libovicky2018end, wang2019non, wei2019imitation, li2019hint, shao2019minimizing}." />
        </attvalues>
      </edge>
      <edge source="2005.00932" target="1706.03762" id="8599">
        <attvalues>
          <attvalue for="5" value=" Although different in various aspects, all of these methods are based on transformer modules \cite{vaswani2017attention}, and depend on a well-trained AR model to obtain its output translations to create targets for NAR model training." />
        </attvalues>
      </edge>
      <edge source="2005.00932" target="1511.06709" id="8600">
        <attvalues>
          <attvalue for="5" value=" Techniques like backtranslation \cite{sennrich2015improving} are known to improve MT performance using monolingual data alone." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2009.02725" id="8601">
        <attvalues>
          <attvalue for="5" value=" Voice conversion (VC) aims to generate a new voice with the source voice content and target speaker timbre \cite{DBLP:journals/taslp/LiuCWWLM21,tang2022avqvc,DBLP:conf/icassp/ChenSH21,DBLP:conf/icassp/HayashiHKT21}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2202.10020" id="8602">
        <attvalues>
          <attvalue for="5" value=" Voice conversion (VC) aims to generate a new voice with the source voice content and target speaker timbre \cite{DBLP:journals/taslp/LiuCWWLM21,tang2022avqvc,DBLP:conf/icassp/ChenSH21,DBLP:conf/icassp/HayashiHKT21}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2010.11646" id="8603">
        <attvalues>
          <attvalue for="5" value=" Voice conversion (VC) aims to generate a new voice with the source voice content and target speaker timbre \cite{DBLP:journals/taslp/LiuCWWLM21,tang2022avqvc,DBLP:conf/icassp/ChenSH21,DBLP:conf/icassp/HayashiHKT21}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2104.06793" id="8604">
        <attvalues>
          <attvalue for="5" value=" Voice conversion (VC) aims to generate a new voice with the source voice content and target speaker timbre \cite{DBLP:journals/taslp/LiuCWWLM21,tang2022avqvc,DBLP:conf/icassp/ChenSH21,DBLP:conf/icassp/HayashiHKT21}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2208.04035" id="8605">
        <attvalues>
          <attvalue for="5" value=" One-to-one VC model is inefficient due to only being able to convert voice between a fixed pair of source speaker and target speaker, such as the CycleGAN-VC \cite{asru2021zhang,DBLP:journals/taslp/NakashikaTA15,asru2021tang}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2010.14150" id="8606">
        <attvalues>
          <attvalue for="5" value=" For VC models with uncertain speaker pair, such as many-to-many~\cite{DBLP:journals/access/LeeKP21,DBLP:conf/icassp/WangY21} and any-to-any ~\cite{DBLP:conf/icassp/LinCLLL21}, widely utilize the disentanglement-based method." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2103.09420" id="8607">
        <attvalues>
          <attvalue for="5" value=" But the challenge is to avoid the overlapping of untangling results~\cite{DBLP:conf/iclr/YuanCZHGC21,DBLP:conf/icassp/LiTYWXSM21}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2010.14804" id="8608">
        <attvalues>
          <attvalue for="5" value=" But the challenge is to avoid the overlapping of untangling results~\cite{DBLP:conf/iclr/YuanCZHGC21,DBLP:conf/icassp/LiTYWXSM21}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="1905.05879" id="8609">
        <attvalues>
          <attvalue for="5" value=" AutoVC proposes to circumspection choose the content dimension to separate the content information before combining it with the pre-trained speaker information~\cite{DBLP:conf/icml/QianZCYH19}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="2006.04154" id="8610">
        <attvalues>
          <attvalue for="5" value=" The similar problem also exists in VQVC+, which proposes to use a codebook to obtain the content information by combining similar dimensions~\cite{DBLP:conf/interspeech/WuCL20}." />
        </attvalues>
      </edge>
      <edge source="2202.10976" target="1905.01270" id="8611">
        <attvalues>
          <attvalue for="5" value=" Disentangled Representation for Image-to-Image Translation (DRIT) assumes image consists of content and attribute information, and two input images have same content \cite{DBLP:journals/ijcv/LeeTMHLSY20}." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="1606.05250" id="8613">
        <attvalues>
          <attvalue for="5" value="5 \cite{brown2020language}, and create a question-answering dataset using questions provided in the SQUAD \cite{rajpurkar-etal-2016-squad} dataset." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2010.06028" id="8614">
        <attvalues>
          <attvalue for="5" value=" Synthetic data generation is a widely adopted method for domain adaptation in QA systems \cite{shakeri2020end} \cite{yue2021contrastive} \cite{yue2022domain}." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2108.13854" id="8615">
        <attvalues>
          <attvalue for="5" value=" Synthetic data generation is a widely adopted method for domain adaptation in QA systems \cite{shakeri2020end} \cite{yue2021contrastive} \cite{yue2022domain}." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2209.04998" id="8616">
        <attvalues>
          <attvalue for="5" value=" Synthetic data generation is a widely adopted method for domain adaptation in QA systems \cite{shakeri2020end} \cite{yue2021contrastive} \cite{yue2022domain}." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2005.14165" id="8617">
        <attvalues>
          <attvalue for="5" value="5 \cite{brown2020language}, and create a question-answering dataset using questions provided in the SQUAD \cite{rajpurkar-etal-2016-squad} dataset." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2303.12712" id="8618">
        <attvalues>
          <attvalue for="5" value=" This transformation has been catalyzed by the advent of generative models trained 'in-the-wild', such as those described in \cite{brown2020language}, \cite{bubeck2023sparks}, and \cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2006.00632" id="8622">
        <attvalues>
          <attvalue for="5" value=" Recent surveys, such as \cite{ramponi-plank-2020-neural}, discuss domain adaptation in NLP and divide approaches into data centric and model centric." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2109.06827" id="8623">
        <attvalues>
          <attvalue for="5" value=" The scarcity of research on generalization in QA models, especially with natural distribution shifts, is a motivation for our work, backed by observations from \cite{arora-etal-2021-types} on out-of-distribution data in NLP." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="1912.02145" id="8624">
        <attvalues>
          <attvalue for="5" value=" Initial experiments like \cite{longpre-etal-2019-exploration} ventured into domain-agnostic question answering using data augmentation." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2110.09468" id="8625">
        <attvalues>
          <attvalue for="5" value=" The benefits of generated data have been explored by \cite{gowal2021improving}, showing its potential in adversarial robustness." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2104.08678" id="8626">
        <attvalues>
          <attvalue for="5" value=" \cite{bartolo-etal-2021-improving} and \cite{mekala-etal-2022-leveraging} use synthetic and context-generated data respectively for QA and text classification." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2205.12604" id="8627">
        <attvalues>
          <attvalue for="5" value=" \cite{bartolo-etal-2021-improving} and \cite{mekala-etal-2022-leveraging} use synthetic and context-generated data respectively for QA and text classification." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2109.01652" id="8628">
        <attvalues>
          <attvalue for="5" value="5 model, as described by \cite{wei2022finetuned}, to generate context for questions." />
        </attvalues>
      </edge>
      <edge source="2309.06358" target="2302.02503" id="8629">
        <attvalues>
          <attvalue for="5" value=" With similar motivations, \cite{bansal2023leaving} demonstrates the application of Stable Diffusion in diverse dataset creation for image tasks." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="1706.03762" id="8636">
        <attvalues>
          <attvalue for="5" value=" \cite{grundkiewicz-etal-2019-neural} leveraged a Transformer model \cite{vaswani2017attention} that was pre-trained on synthetic GEC data and right-to-left re-ranking for ensemble." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="2005.00987" id="8637">
        <attvalues>
          <attvalue for="5" value=" \cite{kaneko2020encoder} adopted several strategies of BERT \cite{devlin2018bert} usage for GEC." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="2106.03830" id="8639">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{rothe2021a} built their system on top of T5 \cite{xue-etal-2021-mt5}, a xxl version of the T5 Transformer encoder-decoder model and reached new state-of-the-art results (11B parameters)." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="2010.11934" id="8640">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{rothe2021a} built their system on top of T5 \cite{xue-etal-2021-mt5}, a xxl version of the T5 Transformer encoder-decoder model and reached new state-of-the-art results (11B parameters)." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="1909.01187" id="8641">
        <attvalues>
          <attvalue for="5" value=" LaserTagger \cite{malmi-etal-2019-encode} is a sequence tagging model that casts text generation as a text editing task." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="1910.02893" id="8642">
        <attvalues>
          <attvalue for="5" value=" The Parallel Iterative Edit (PIE) model \cite{awasthi2019parallel} does parallel decoding, achieving quality that is competitive with the seq2seq models." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="2005.12592" id="8643">
        <attvalues>
          <attvalue for="5" value=" A similar approach is presented in \cite{omelianchuk2020gector}." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="1904.05780" id="8644">
        <attvalues>
          <attvalue for="5" value=" However, human annotation is expensive, so researchers are working on methods for augmentation of training data, synthetic data generation, and strategies for its efficient usage \cite{lichtarge2019corpora}, \cite{kiyono2019an}, \cite{stahlberg2021synthetic}." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="1909.00502" id="8645">
        <attvalues>
          <attvalue for="5" value=" However, human annotation is expensive, so researchers are working on methods for augmentation of training data, synthetic data generation, and strategies for its efficient usage \cite{lichtarge2019corpora}, \cite{kiyono2019an}, \cite{stahlberg2021synthetic}." />
        </attvalues>
      </edge>
      <edge source="2203.13064" target="2105.13318" id="8646">
        <attvalues>
          <attvalue for="5" value=" However, human annotation is expensive, so researchers are working on methods for augmentation of training data, synthetic data generation, and strategies for its efficient usage \cite{lichtarge2019corpora}, \cite{kiyono2019an}, \cite{stahlberg2021synthetic}." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1910.00353" id="8647">
        <attvalues>
          <attvalue for="5" value=" We generate synthetic training data by automatically corrupting grammatical sentences, but in contrast to the previous state-of-the-art by \cite{naplava} for low-resources languages, we use our synthetic pre-training to train a single model on all 101 languages, employing no language-specific priors to remain fully language-agnostic." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="2005.11849" id="8648">
        <attvalues>
          <attvalue for="5" value=" Modern approaches often view the GEC task as monolingual text-to-text rewriting \cite{naplava,katsumata2020stronger,grundkiewicz-etal-2019-neural} and employ encoder-decoder neural architectures~\cite{sutskever2014sequence,bahdanau2014neural}." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1409.3215" id="8649">
        <attvalues>
          <attvalue for="5" value=" Modern approaches often view the GEC task as monolingual text-to-text rewriting \cite{naplava,katsumata2020stronger,grundkiewicz-etal-2019-neural} and employ encoder-decoder neural architectures~\cite{sutskever2014sequence,bahdanau2014neural}." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1409.0473" id="8650">
        <attvalues>
          <attvalue for="5" value=" Modern approaches often view the GEC task as monolingual text-to-text rewriting \cite{naplava,katsumata2020stronger,grundkiewicz-etal-2019-neural} and employ encoder-decoder neural architectures~\cite{sutskever2014sequence,bahdanau2014neural}." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1909.01187" id="8651">
        <attvalues>
          <attvalue for="5" value=" These methods typically require large training sets to work well~\cite{malmi-etal-2019-encode} which are scarce especially for languages other than English." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1904.05780" id="8652">
        <attvalues>
          <attvalue for="5" value=" Although using synthetic data as the first fine-tuning step has been shown to improve model accuracy, it introduces practical challenges that make the development and fair comparison of GEC models challenging: ($i$) the synthetic methods often require language-specific tuning (eg language-specific hyperparameters and spelling dictionaries~\cite{naplava}), and; ($ii$) due to the inability of synthetic data to capture the complete error distribution of the target eval sets, the final model is obtained by following a multi-stage fine-tuning process~\cite{lichtarge-etal-2019-corpora,lichtarge-etal-2020-data,omelianchuk-etal-2020-gector}." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1910.02893" id="8653">
        <attvalues>
          <attvalue for="5" value=" Limited amounts of suitable training data has led to multiple approaches that propose to generate synthetic training data for GEC~\cite{madnani2012exploring,grundkiewicz2014wiked,grundkiewicz-etal-2019-neural,lichtarge-etal-2019-corpora,awasthi-etal-2019-parallel}." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="2008.02976" id="8654">
        <attvalues>
          <attvalue for="5" value=" Although using synthetic data as the first fine-tuning step has been shown to improve model accuracy, it introduces practical challenges that make the development and fair comparison of GEC models challenging: ($i$) the synthetic methods often require language-specific tuning (eg language-specific hyperparameters and spelling dictionaries~\cite{naplava}), and; ($ii$) due to the inability of synthetic data to capture the complete error distribution of the target eval sets, the final model is obtained by following a multi-stage fine-tuning process~\cite{lichtarge-etal-2019-corpora,lichtarge-etal-2020-data,omelianchuk-etal-2020-gector}." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="2005.12592" id="8655">
        <attvalues>
          <attvalue for="5" value=" Although using synthetic data as the first fine-tuning step has been shown to improve model accuracy, it introduces practical challenges that make the development and fair comparison of GEC models challenging: ($i$) the synthetic methods often require language-specific tuning (eg language-specific hyperparameters and spelling dictionaries~\cite{naplava}), and; ($ii$) due to the inability of synthetic data to capture the complete error distribution of the target eval sets, the final model is obtained by following a multi-stage fine-tuning process~\cite{lichtarge-etal-2019-corpora,lichtarge-etal-2020-data,omelianchuk-etal-2020-gector}." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="2010.11934" id="8657">
        <attvalues>
          <attvalue for="5" value=" In this paper we adopt the mT5 \cite{mt5} as our base model which has already been pre-trained on a corpus covering 101 languages." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1910.13461" id="8658">
        <attvalues>
          <attvalue for="5" value=" The ideas of leveraging self-supervised pre-training and increasing the model size have yielded significant improvements on numerous seq2seq tasks in recent years~\cite{t5,mt5,BART,MASS,KERMIT,Bert2Bert}, but these approaches have been applied to GEC to only a limited extent." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1905.02450" id="8659">
        <attvalues>
          <attvalue for="5" value=" The ideas of leveraging self-supervised pre-training and increasing the model size have yielded significant improvements on numerous seq2seq tasks in recent years~\cite{t5,mt5,BART,MASS,KERMIT,Bert2Bert}, but these approaches have been applied to GEC to only a limited extent." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1906.01604" id="8660">
        <attvalues>
          <attvalue for="5" value=" The ideas of leveraging self-supervised pre-training and increasing the model size have yielded significant improvements on numerous seq2seq tasks in recent years~\cite{t5,mt5,BART,MASS,KERMIT,Bert2Bert}, but these approaches have been applied to GEC to only a limited extent." />
        </attvalues>
      </edge>
      <edge source="2106.03830" target="1907.12461" id="8661">
        <attvalues>
          <attvalue for="5" value=" The ideas of leveraging self-supervised pre-training and increasing the model size have yielded significant improvements on numerous seq2seq tasks in recent years~\cite{t5,mt5,BART,MASS,KERMIT,Bert2Bert}, but these approaches have been applied to GEC to only a limited extent." />
        </attvalues>
      </edge>
      <edge source="2406.17626" target="2209.07858" id="8662">
        <attvalues>
          <attvalue for="5" value=" For safety enhancement, Anthropic and DeepMind \cite{HH-RLHF} have introduced the ``red-teaming'' approach, which involves the role-playing of attackers in order to identify and fix the models' safety loopholes." />
        </attvalues>
      </edge>
      <edge source="2406.17626" target="1804.06876" id="8663">
        <attvalues>
          <attvalue for="5" value=" Furthermore, existing studies have created various red-teaming datasets to assess the safety of LLMs in different scenarios, such as bias \cite{genderbias}, discrimination \cite{harmfulq}, and privacy breaches \cite{Confaide}." />
        </attvalues>
      </edge>
      <edge source="2406.17626" target="2212.08061" id="8664">
        <attvalues>
          <attvalue for="5" value=" Furthermore, existing studies have created various red-teaming datasets to assess the safety of LLMs in different scenarios, such as bias \cite{genderbias}, discrimination \cite{harmfulq}, and privacy breaches \cite{Confaide}." />
        </attvalues>
      </edge>
      <edge source="2406.17626" target="2310.17884" id="8665">
        <attvalues>
          <attvalue for="5" value=" Furthermore, existing studies have created various red-teaming datasets to assess the safety of LLMs in different scenarios, such as bias \cite{genderbias}, discrimination \cite{harmfulq}, and privacy breaches \cite{Confaide}." />
        </attvalues>
      </edge>
      <edge source="2406.17626" target="2307.04657" id="8666">
        <attvalues>
          <attvalue for="5" value=" Initially, we selected $100$ attack prompts from each category defined by BeaverTails~\cite{beavertails}." />
        </attvalues>
      </edge>
      <edge source="2407.12869" target="2307.09288" id="8667">
        <attvalues>
          <attvalue for="5" value=" There has been a rapid advancement in open source English-dominant foundation language models like Llama 2 \cite{touvron2023llama}, Mistral 7B \cite{jiang2023mistral}, and Llama 3, primarily trained on extensive English corpora with minimal inclusion of non-English languages." />
        </attvalues>
      </edge>
      <edge source="2407.12869" target="2310.06825" id="8668">
        <attvalues>
          <attvalue for="5" value=" There has been a rapid advancement in open source English-dominant foundation language models like Llama 2 \cite{touvron2023llama}, Mistral 7B \cite{jiang2023mistral}, and Llama 3, primarily trained on extensive English corpora with minimal inclusion of non-English languages." />
        </attvalues>
      </edge>
      <edge source="2407.12869" target="2308.16149" id="8669">
        <attvalues>
          <attvalue for="5" value=" While bilingual and monolingual models trained from scratch, like Jais \cite{sengupta2023jais} and Bloom \cite{workshop2023bloom}, have shown promise in non-English capabilities, they are expensive to train and have inferior capabilities in English." />
        </attvalues>
      </edge>
      <edge source="2407.12869" target="2211.05100" id="8670">
        <attvalues>
          <attvalue for="5" value=" While bilingual and monolingual models trained from scratch, like Jais \cite{sengupta2023jais} and Bloom \cite{workshop2023bloom}, have shown promise in non-English capabilities, they are expensive to train and have inferior capabilities in English." />
        </attvalues>
      </edge>
      <edge source="2407.12869" target="2404.17790" id="8671">
        <attvalues>
          <attvalue for="5" value=" \cite{fujii2024swallow, luo2024empirical, FRENCH1999128, huang2024acegpt}." />
        </attvalues>
      </edge>
      <edge source="2407.12869" target="2012.05628" id="8673">
        <attvalues>
          <attvalue for="5" value=" Several recent works demonstrate cross-lingual transfer of foundation models \cite{de-vries-nissim-2021-good, marchisio-etal-2023-mini, csaki2023efficiently, zhao2024Llama, huang2024acegpt, da-dalt-etal-2024-flor-effectiveness}, yet they lack comprehensive analysis of hyperparameter tuning, tokenizer, data mix selections, and the impact of different model sizes." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="2111.10592" id="8679">
        <attvalues>
          <attvalue for="5" value=" Additionally, for many use cases, such as \gls*{kws} for voice assistants, it is desirable that the models are small and efficient \cite{ivan_deep_spoken_keyword}." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="2006.11477" id="8680">
        <attvalues>
          <attvalue for="5" value=" However, current studies mainly focus on developing universal speech models \cite{baevski_wav2vec2, Chen_WavLMLS}, which are trained on large speech corpuses such as Librispeech \cite{librispeech} or LibriLight \cite{librilight}, with the goal of obtaining a model that can perform well for multiple downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="1912.07875" id="8681">
        <attvalues>
          <attvalue for="5" value=" However, current studies mainly focus on developing universal speech models \cite{baevski_wav2vec2, Chen_WavLMLS}, which are trained on large speech corpuses such as Librispeech \cite{librispeech} or LibriLight \cite{librilight}, with the goal of obtaining a model that can perform well for multiple downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="1503.02531" id="8683">
        <attvalues>
          <attvalue for="5" value=" While knowledge distillation \cite{hinton_distilling} has been investigated for transferring the representations learned by a large model to a smaller model \cite{Gu_Liu, Fang_SEED, chen_kornblith}, such methods do not deal with the problem of the necessity of training a large model initially." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="2106.11304" id="8684">
        <attvalues>
          <attvalue for="5" value=" While knowledge distillation \cite{hinton_distilling} has been investigated for transferring the representations learned by a large model to a smaller model \cite{Gu_Liu, Fang_SEED, chen_kornblith}, such methods do not deal with the problem of the necessity of training a large model initially." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="2107.14762" id="8685">
        <attvalues>
          <attvalue for="5" value=" One study used a contrastive type of SSL method to train smaller models without distillation from a large pretrained model and found that, contrary to former assumptions, small models are able to solve the self-supervised pretext tasks without overfitting \cite{efficacy_of_small_self_supervised_contrastive_models}." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="2005.08575" id="8686">
        <attvalues>
          <attvalue for="5" value=" Other work found that the learned parameters of large speech models suffer from redundancy across layers, and proposed the use of weight sharing to reduce parameter redundancy and the network size \cite{Chi_AudioAA}." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="2202.03555" id="8687">
        <attvalues>
          <attvalue for="5" value=" In this paper, we investigate the adaption of the general non-contrastive SSL framework Data2Vec \cite{data2vec} to improve \gls*{kws} performance in label-deficient scenarios." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="2104.00769" id="8688">
        <attvalues>
          <attvalue for="5" value=" We implement three variations of the \gls*{kwt} model \cite{berg_KWT}, varying from 600k to 5." />
        </attvalues>
      </edge>
      <edge source="2210.01703" target="1804.03209" id="8689">
        <attvalues>
          <attvalue for="5" value=" The models are evaluated on a label-deficient setup of the Google Speech Commands data set \cite{speechcommandsv2} with only \SI{20}{\percent} labelled data for supervised training, and the results show the following: \begin{enumerate} performance for all three models when the amount of labelled data is limited, indicating that self-supervised learning can also be beneficial for small models." />
        </attvalues>
      </edge>
      <edge source="2102.12624" target="2005.14327" id="8690">
        <attvalues>
          <attvalue for="5" value=" While end-to-end (E2E) \cite{graves2014towards} deep learning (DL) models brought great improvements to the field of automatic speech recognition (ASR) in recent years and reduced word error rates (WER) on benchmark datasets significantly \cite{li2020comparison}, they also come with a set of problems." />
        </attvalues>
      </edge>
      <edge source="2102.12624" target="1810.12170" id="8693">
        <attvalues>
          <attvalue for="5" value=" Recently some approaches have arisen to tackle this challenge with E2E ASR using a specialised architecture and losses \cite{peyser2020improving} or using specific data and training procedures to better represent contextual information \cite{alon2019contextual}." />
        </attvalues>
      </edge>
      <edge source="2102.12624" target="1706.03762" id="8694">
        <attvalues>
          <attvalue for="5" value=" An ASR system is built using the Transformer architecture \cite{vaswani2017attention} which has been shown to outperform any prior architecture for most speech related tasks\cite{karita2019comparative, li2019neural, vila2018end, dong2018speech} by heavily relying on attention mechanisms \cite{bahdanau2015neural}." />
        </attvalues>
      </edge>
      <edge source="2102.12624" target="1909.06317" id="8695">
        <attvalues>
          <attvalue for="5" value=" An ASR system is built using the Transformer architecture \cite{vaswani2017attention} which has been shown to outperform any prior architecture for most speech related tasks\cite{karita2019comparative, li2019neural, vila2018end, dong2018speech} by heavily relying on attention mechanisms \cite{bahdanau2015neural}." />
        </attvalues>
      </edge>
      <edge source="2102.12624" target="1409.0473" id="8696">
        <attvalues>
          <attvalue for="5" value=" An ASR system is built using the Transformer architecture \cite{vaswani2017attention} which has been shown to outperform any prior architecture for most speech related tasks\cite{karita2019comparative, li2019neural, vila2018end, dong2018speech} by heavily relying on attention mechanisms \cite{bahdanau2015neural}." />
        </attvalues>
      </edge>
      <edge source="2102.12624" target="1803.08976" id="8697">
        <attvalues>
          <attvalue for="5" value=" Prior approaches of embedding audio include using a skip-gram model for speech \cite{chung2018speech2vec} and using convolutional neural nets (CNNs) for general purpose audio detection \cite{audiomax}." />
        </attvalues>
      </edge>
      <edge source="2102.12624" target="1711.06025" id="8698">
        <attvalues>
          <attvalue for="5" value=" The metric space embeddings are then compared to reference samples using various renown MSML approaches such as Siamese Networks \cite{bromley1994signature}, Relation Networks \cite{sung2018learning}, Prototypical Networks \cite{snell2017prototypical} and Matching Networks \cite{vinyals2016matching}." />
        </attvalues>
      </edge>
      <edge source="2102.12624" target="1703.05175" id="8699">
        <attvalues>
          <attvalue for="5" value=" The metric space embeddings are then compared to reference samples using various renown MSML approaches such as Siamese Networks \cite{bromley1994signature}, Relation Networks \cite{sung2018learning}, Prototypical Networks \cite{snell2017prototypical} and Matching Networks \cite{vinyals2016matching}." />
        </attvalues>
      </edge>
      <edge source="2406.13713" target="2408.04216" id="8701">
        <attvalues>
          <attvalue for="5" value=" Natural Language Processing (NLP) has been revolutionized by the emergence of Large Language Models (LLMs), enabling advancements in tasks such as machine translation~\cite{zhao2023transformer,bo2024_attentionMT, huang2024moceadaptivemixturecontextualization}, text summarization~\cite{li2023improving,doan2023too}, and knowledge extraction~\cite{gu2023distilling,xu2024take}." />
        </attvalues>
      </edge>
      <edge source="2406.13713" target="2307.06439" id="8702">
        <attvalues>
          <attvalue for="5" value=" Natural Language Processing (NLP) has been revolutionized by the emergence of Large Language Models (LLMs), enabling advancements in tasks such as machine translation~\cite{zhao2023transformer,bo2024_attentionMT, huang2024moceadaptivemixturecontextualization}, text summarization~\cite{li2023improving,doan2023too}, and knowledge extraction~\cite{gu2023distilling,xu2024take}." />
        </attvalues>
      </edge>
      <edge source="2406.13713" target="2403.09963" id="8703">
        <attvalues>
          <attvalue for="5" value=" Natural Language Processing (NLP) has been revolutionized by the emergence of Large Language Models (LLMs), enabling advancements in tasks such as machine translation~\cite{zhao2023transformer,bo2024_attentionMT, huang2024moceadaptivemixturecontextualization}, text summarization~\cite{li2023improving,doan2023too}, and knowledge extraction~\cite{gu2023distilling,xu2024take}." />
        </attvalues>
      </edge>
      <edge source="2406.13713" target="2303.08774" id="8704">
        <attvalues>
          <attvalue for="5" value=" Prominent examples include GPT-4~\cite{chatGPT4}, Claude 3 Opus~\cite{claude}, and Mixtral 8x7B~\cite{jiang2024_mixtral}." />
        </attvalues>
      </edge>
      <edge source="2406.13713" target="2401.04088" id="8705">
        <attvalues>
          <attvalue for="5" value=" Prominent examples include GPT-4~\cite{chatGPT4}, Claude 3 Opus~\cite{claude}, and Mixtral 8x7B~\cite{jiang2024_mixtral}." />
        </attvalues>
      </edge>
      <edge source="2204.09874" target="1511.06038" id="8706">
        <attvalues>
          <attvalue for="5" value=" Neural topic models (NTMs) \cite{miao2016neural, srivastava2017autoencoding} incorporating neural components have significantly advanced the modelling results than the traditional Latent Dirichlet Allocation (LDA; \cite{blei2003latent})." />
        </attvalues>
      </edge>
      <edge source="2204.09874" target="1312.6114" id="8709">
        <attvalues>
          <attvalue for="5" value=" Despite the promising performance, existing NTMs are generally based on a variational autoencoder framework (VAE; \cite{kingma2013auto}), which suffers from hyper-parameters tuning and computational overheads \cite{zhao2021topic}." />
        </attvalues>
      </edge>
      <edge source="2204.09874" target="2103.00498" id="8710">
        <attvalues>
          <attvalue for="5" value=" Despite the promising performance, existing NTMs are generally based on a variational autoencoder framework (VAE; \cite{kingma2013auto}), which suffers from hyper-parameters tuning and computational overheads \cite{zhao2021topic}." />
        </attvalues>
      </edge>
      <edge source="2204.09874" target="2004.02105" id="8711">
        <attvalues>
          <attvalue for="5" value=" With high-quality contextualized document representations, do we really need sophisticated NTMs to obtain coherent and interpretable topics? Recent work \cite{aharoni-goldberg-2020-unsupervised, sia-etal-2020-tired, thompson2020topic, grootendorst2020bertopic} has shown that directly congregating contextualized embeddings can get semantically similar word or document clusters." />
        </attvalues>
      </edge>
      <edge source="2204.09874" target="2004.14914" id="8712">
        <attvalues>
          <attvalue for="5" value=" Specifically, \cite{sia-etal-2020-tired} cluster vocabulary-level word embeddings and obtain top words from each cluster using weighing and re-ranking, while \cite{thompson2020topic} consider polysemy and perform token-level clustering." />
        </attvalues>
      </edge>
      <edge source="2204.09874" target="2010.12626" id="8713">
        <attvalues>
          <attvalue for="5" value=" Specifically, \cite{sia-etal-2020-tired} cluster vocabulary-level word embeddings and obtain top words from each cluster using weighing and re-ranking, while \cite{thompson2020topic} consider polysemy and perform token-level clustering." />
        </attvalues>
      </edge>
      <edge source="2203.16954" target="1911.04128" id="8734">
        <attvalues>
          <attvalue for="5" value=" Recently, a hybrid TN system for Mandarin has been proposed, which combines a rule-based model based on pattern match and a multi-head self-attention based non-seq2seq neural network model, to address the corresponding shortcomings mentioned above \cite{zhang2020hybrid}." />
        </attvalues>
      </edge>
      <edge source="2203.16954" target="2004.11795" id="8735">
        <attvalues>
          <attvalue for="5" value=" FLAT can obtain all potential words in the sentence that match the specific lexicon, organize all characters and matched words to a lattice structure and flatten the lattice structure into spans \cite{li2020flat}, then send them into Transformer encoder." />
        </attvalues>
      </edge>
      <edge source="2203.16954" target="1805.02023" id="8736">
        <attvalues>
          <attvalue for="5" value=" The method of combining lexicon is fully independent of word segmentation, and more effective in using word information thanks to the freedom of choosing lexicon words in a context \cite{zhang2018chinese}." />
        </attvalues>
      </edge>
      <edge source="2212.14227" target="1905.05879" id="8737">
        <attvalues>
          <attvalue for="5" value=" To convert an unseen speaker’s voice into another speaker’s voice unseen during training, the model needs to learn a shared representation of speech across all potential sources and target speakers \cite{qian2019autovc}." />
        </attvalues>
      </edge>
      <edge source="2212.14227" target="1904.05742" id="8738">
        <attvalues>
          <attvalue for="5" value=" Several techniques have been proposed for learning disentangled representations, including instance normalization \cite{chou2019one, wu2020one, chen2021again}, vector quantization \cite{wu2020one, van2020vector, wang2021vqmivc, tang2022avqvc}, transfer learning from ASR or TTS models \cite{li2021ppg, lin2021s2vc, zhang2021transfer, casanova2022yourtts, gabrys2022voice}, and adversarial training \cite{wang2020one, tang2021tgavc}." />
        </attvalues>
      </edge>
      <edge source="2212.14227" target="2010.14804" id="8739">
        <attvalues>
          <attvalue for="5" value=" Several techniques have been proposed for learning disentangled representations, including instance normalization \cite{chou2019one, wu2020one, chen2021again}, vector quantization \cite{wu2020one, van2020vector, wang2021vqmivc, tang2022avqvc}, transfer learning from ASR or TTS models \cite{li2021ppg, lin2021s2vc, zhang2021transfer, casanova2022yourtts, gabrys2022voice}, and adversarial training \cite{wang2020one, tang2021tgavc}." />
        </attvalues>
      </edge>
      <edge source="2212.14227" target="1910.11997" id="8740">
        <attvalues>
          <attvalue for="5" value=" VC systems such as Mellotron \cite{valle2020mellotron} and Cotatron \cite{park2020cotatron}, on the other hand, use phoneme alignment and pitch curve from the source speech and re-synthesize the speech of the target speaker." />
        </attvalues>
      </edge>
      <edge source="2212.14227" target="2005.03295" id="8741">
        <attvalues>
          <attvalue for="5" value=" VC systems such as Mellotron \cite{valle2020mellotron} and Cotatron \cite{park2020cotatron}, on the other hand, use phoneme alignment and pitch curve from the source speech and re-synthesize the speech of the target speaker." />
        </attvalues>
      </edge>
      <edge source="2212.14227" target="2009.14399" id="8742">
        <attvalues>
          <attvalue for="5" value=" Our work makes multiple contributions: (i) we show that the cycle consistency and adversarial objective are effective in training both TTS decoder and mel-spectrogram encoder for VC applications, (ii) we introduce novel data augmentation using text-guided voice conversion results as both input and target during training, and (iii) we demonstrate that the loss function proposed in \cite{zhang2021transfer} is suboptimal for transfer learning from TTS models for voice conversion applications and propose an alternative solution with a mutual information (MI) maximization objective." />
        </attvalues>
      </edge>
      <edge source="2212.14227" target="2112.02418" id="8744">
        <attvalues>
          <attvalue for="5" value=" The subjective human evaluation shows that our model outperforms the previous state-of-the-art one-shot voice conversion model, YourTTS \cite{casanova2022yourtts}, and two other baseline models, AGAIN-VC \cite{chen2021again} and VQMIVC \cite{wang2021vqmivc}, for unseen source and target speakers." />
        </attvalues>
      </edge>
      <edge source="2212.14227" target="2011.00316" id="8745">
        <attvalues>
          <attvalue for="5" value=" The subjective human evaluation shows that our model outperforms the previous state-of-the-art one-shot voice conversion model, YourTTS \cite{casanova2022yourtts}, and two other baseline models, AGAIN-VC \cite{chen2021again} and VQMIVC \cite{wang2021vqmivc}, for unseen source and target speakers." />
        </attvalues>
      </edge>
      <edge source="2210.07111" target="2103.06874" id="8747">
        <attvalues>
          <attvalue for="5" value=" Several recent results \cite{Clark2022CaninePA,Xue2022ByT5TA} have excited the research community with the possibility of ``tokenizer-free'' models, character-level and byte-level models, as an alternative to more traditional subword-based models." />
        </attvalues>
      </edge>
      <edge source="2210.07111" target="2105.13626" id="8748">
        <attvalues>
          <attvalue for="5" value=" Several recent results \cite{Clark2022CaninePA,Xue2022ByT5TA} have excited the research community with the possibility of ``tokenizer-free'' models, character-level and byte-level models, as an alternative to more traditional subword-based models." />
        </attvalues>
      </edge>
      <edge source="2210.07111" target="2012.15613" id="8749">
        <attvalues>
          <attvalue for="5" value=" We, the authors of this paper, were also initially excited by these results -- the possibility of eschewing the two-step processing pipeline of subword segmentation and subword-based models would reduce the corresponding difficulties in cross-lingual transfer \cite{pmlr-v119-hu20b,maronikolakis-etal-2021-wine-v,rust-etal-2021-good,Wang2021-wy} or domain adaptation \cite{sato-etal-2020-vocabulary,liu-etal-2021-bridging} due to inconsistent subword units." />
        </attvalues>
      </edge>
      <edge source="2206.03025" target="2204.10050" id="8763">
        <attvalues>
          <attvalue for="5" value=" This paper describes the system developed by the OCHADAI team for SemEval-2022 Task 2 - Multilingual Idiomaticity Detection and Sentence Embedding \cite{tayyarmadabushi-etal-2022-semeval}." />
        </attvalues>
      </edge>
      <edge source="2401.10465" target="2004.06338" id="8764">
        <attvalues>
          <attvalue for="5" value=" Neural G2Ps~\cite{neuralG2P}, in contrast, use lexicons as their data for training their neural network and use the obtained network for predicting pronunciations." />
        </attvalues>
      </edge>
      <edge source="2401.10465" target="2204.03067" id="8767">
        <attvalues>
          <attvalue for="5" value=" Recently there have been efforts to build a massively multi-lingual ByT5 G2P~\cite{Zhu2022ByT5MF}." />
        </attvalues>
      </edge>
      <edge source="2401.10465" target="2106.07447" id="8768">
        <attvalues>
          <attvalue for="5" value=" We first use unlabeled speech data to pre-train a HuBERT \cite{9585401} for three iterations." />
        </attvalues>
      </edge>
      <edge source="2401.10465" target="1712.05884" id="8769">
        <attvalues>
          <attvalue for="5" value=" We then use the trained G2P to train a Tacotron 2 \cite{8461368} model." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1703.10135" id="8770">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural network-based speech synthesis systems \cite{wang2017tacotron,shen2018natural,li2019neural} show superior performance in terms of speech quality and naturalness and offer an effective approach to learning speech representations." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1712.05884" id="8771">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural network-based speech synthesis systems \cite{wang2017tacotron,shen2018natural,li2019neural} show superior performance in terms of speech quality and naturalness and offer an effective approach to learning speech representations." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1809.08895" id="8772">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural network-based speech synthesis systems \cite{wang2017tacotron,shen2018natural,li2019neural} show superior performance in terms of speech quality and naturalness and offer an effective approach to learning speech representations." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1803.09047" id="8773">
        <attvalues>
          <attvalue for="5" value=" Various approaches to unsupervised learning of prosodic representation have been proposed \cite{skerry2018towards,klimkov2019fine,lee2019robust,zhang2020learning,sun2020fully,tan2020fine,wang2019vector,kenter2019chive}." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1907.02479" id="8774">
        <attvalues>
          <attvalue for="5" value=" Various approaches to unsupervised learning of prosodic representation have been proposed \cite{skerry2018towards,klimkov2019fine,lee2019robust,zhang2020learning,sun2020fully,tan2020fine,wang2019vector,kenter2019chive}." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1811.02122" id="8775">
        <attvalues>
          <attvalue for="5" value=" Various approaches to unsupervised learning of prosodic representation have been proposed \cite{skerry2018towards,klimkov2019fine,lee2019robust,zhang2020learning,sun2020fully,tan2020fine,wang2019vector,kenter2019chive}." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1906.04233" id="8776">
        <attvalues>
          <attvalue for="5" value=" Specific speech generation applications (eg, TTS) require appropriate and expressive prosody predicted from the text \cite{hodariusing}." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="2011.03943" id="8778">
        <attvalues>
          <attvalue for="5" value=" The prosodic representation is obtained as one of the learned factors, parallel with non-prosodic factors that correspond to content, speaker, channel, etc In \cite{tan2020fine, zhang2021estimating, hsu2019disentangling,karlapati2020copycat}, adversarial learning was applied to address the problem that the learned prosodic representation might contain substantial information related to non-prosodic factors." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1905.07195" id="8779">
        <attvalues>
          <attvalue for="5" value=" Various approaches to unsupervised learning of prosodic representation have been proposed \cite{skerry2018towards,klimkov2019fine,lee2019robust,zhang2020learning,sun2020fully,tan2020fine,wang2019vector,kenter2019chive}." />
        </attvalues>
      </edge>
      <edge source="2108.02821" target="1503.02406" id="8781">
        <attvalues>
          <attvalue for="5" value=" In the present study, prosodic representations learning is tackled from the perspective of information bottleneck (IB), by which a good representation is determined with the trade-off between its predictive/reconstructive power and compact representation \cite{tishby2015deep}." />
        </attvalues>
      </edge>
      <edge source="2407.14875" target="2005.14165" id="8783">
        <attvalues>
          <attvalue for="5" value=" Auto-regressive language models \cite{gpt3} have a very impressive ability to perform a new task with a few examples without any additional fine-tuning, named few-shot learners." />
        </attvalues>
      </edge>
      <edge source="2407.14875" target="2310.13289" id="8784">
        <attvalues>
          <attvalue for="5" value=" This is mainly different from \cite{BLSP,BLSPKD}, they only use one simplex prompt `Continue the following text', which may lead to task overfitting phenomenon as described in \cite{salmonn}." />
        </attvalues>
      </edge>
      <edge source="2407.14875" target="2311.07919" id="8785">
        <attvalues>
          <attvalue for="5" value=" Prior speech language models \cite{salmonn,qwen-audio} utilize next token prediction task to align speech features into word embedding space, where the desired next tokens are generated by task-related prompt and speech features." />
        </attvalues>
      </edge>
      <edge source="2407.14875" target="2209.11895" id="8786">
        <attvalues>
          <attvalue for="5" value=" In the research area of mechanistic interpretability, \cite{induction} introduces that the ability of context-learning is derived from an induction head, a circuit whose complete the pattern by copying and completing sequences that have occurred before." />
        </attvalues>
      </edge>
      <edge source="2407.14875" target="2309.00916" id="8787">
        <attvalues>
          <attvalue for="5" value=" This is mainly different from \cite{BLSP,BLSPKD}, they only use one simplex prompt `Continue the following text', which may lead to task overfitting phenomenon as described in \cite{salmonn}." />
        </attvalues>
      </edge>
      <edge source="2407.14875" target="2405.19041" id="8788">
        <attvalues>
          <attvalue for="5" value=" This is mainly different from \cite{BLSP,BLSPKD}, they only use one simplex prompt `Continue the following text', which may lead to task overfitting phenomenon as described in \cite{salmonn}." />
        </attvalues>
      </edge>
      <edge source="2407.14875" target="1904.03670" id="8789">
        <attvalues>
          <attvalue for="5" value=" Extensive experiments demonstrate that the resulting Seal model exhibits robust performance as a few-shot learner on two speech understanding tasks, including FSC \cite{fsc} and SLURP \cite{slurp}." />
        </attvalues>
      </edge>
      <edge source="2407.14875" target="2011.13205" id="8790">
        <attvalues>
          <attvalue for="5" value=" Extensive experiments demonstrate that the resulting Seal model exhibits robust performance as a few-shot learner on two speech understanding tasks, including FSC \cite{fsc} and SLURP \cite{slurp}." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2110.04484" id="8791">
        <attvalues>
          <attvalue for="5" value=" To handle the lack of training data, a common method is using self-supervised or semi-supervised training~\cite{zhu22c_interspeech,synnaeve2020end, vesely2017semi}, where a model initially trained with limited transcribed data is used to generate transcriptions for unlabeled data which are then added to the training data to update the model." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2310.18450" id="8793">
        <attvalues>
          <attvalue for="5" value=" Data augmentation can also be used to tackle this problem where additional data is synthesised from existing data eg~\cite{xie23_interspeech,bartelds2023making,park19specaug,ragni14data}." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2305.10951" id="8794">
        <attvalues>
          <attvalue for="5" value=" Data augmentation can also be used to tackle this problem where additional data is synthesised from existing data eg~\cite{xie23_interspeech,bartelds2023making,park19specaug,ragni14data}." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="1904.08779" id="8795">
        <attvalues>
          <attvalue for="5" value=" Data augmentation can also be used to tackle this problem where additional data is synthesised from existing data eg~\cite{xie23_interspeech,bartelds2023making,park19specaug,ragni14data}." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2303.01037" id="8797">
        <attvalues>
          <attvalue for="5" value=" With the development and release of pre-trained multilingual end-to-end foundation ASR models~\cite{radford2023robust,zhang2023google,babu2021xls}, utilising a multilingual model is another option for low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2111.09296" id="8798">
        <attvalues>
          <attvalue for="5" value=" With the development and release of pre-trained multilingual end-to-end foundation ASR models~\cite{radford2023robust,zhang2023google,babu2021xls}, utilising a multilingual model is another option for low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2302.11186" id="8799">
        <attvalues>
          <attvalue for="5" value=" proposed a smaller universal monolingual output layer shared across languages for high-quality and high-efficiency multilingual ASR~\cite{zhang2023uml}." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2302.12829" id="8800">
        <attvalues>
          <attvalue for="5" value=" used hierarchical CTC to leverage language identity throughout the entire encoder-decoder network, aiming to improve ASR performance by correctly identifying languages~\cite{chen2023improving}." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2110.04366" id="8801">
        <attvalues>
          <attvalue for="5" value=" In NLP, approaches like language-specific adapters and output heads, as seen in~\cite{he2021towards, houlsby2019parameter}, enable parameter-efficient fine-tuning but necessitate prior knowledge of the input language." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="1802.07569" id="8802">
        <attvalues>
          <attvalue for="5" value=" Alternatively, continual lifelong learning, as proposed by \cite{parisi2019continual}, combines data from both existing and new languages for ongoing training." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2105.03010" id="8804">
        <attvalues>
          <attvalue for="5" value=" proposed a weight factorization technique to factorize each weight matrix in the network into language dependent and independent factors~\cite{pham21efficient}." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2211.11703" id="8805">
        <attvalues>
          <attvalue for="5" value=" When combined with Elastic Weight Consolidation (EWC), this approach allows a multilingual ASR model to expand its learning from an initial 10 languages to 26 languages without catastrophic forgetting~\cite{pham23towards,Pham2023thesis}." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2104.08691" id="8806">
        <attvalues>
          <attvalue for="5" value=" Inspired by techniques from NLP tasks, we implement Soft Prompt Tuning (SPT), introduced in~\cite{lester2021power} and utilised in a language assessment task~\cite{ma2023adapting_slate}, to effectively expand language coverage." />
        </attvalues>
      </edge>
      <edge source="2407.06800" target="2307.09378" id="8807">
        <attvalues>
          <attvalue for="5" value=" Inspired by techniques from NLP tasks, we implement Soft Prompt Tuning (SPT), introduced in~\cite{lester2021power} and utilised in a language assessment task~\cite{ma2023adapting_slate}, to effectively expand language coverage." />
        </attvalues>
      </edge>
      <edge source="2105.03010" target="1611.04798" id="8808">
        <attvalues>
          <attvalue for="5" value=" This was shown to be more effective in a multilingual scenario than fully sharing the whole network\cite{ha2016toward,johnson2017google} since each language has certain unique features, and the single architecture often struggles to handle a variety of languages~\cite{platanios-etal-2018-contextual}." />
        </attvalues>
      </edge>
      <edge source="2105.03010" target="1808.08493" id="8809">
        <attvalues>
          <attvalue for="5" value=" This was shown to be more effective in a multilingual scenario than fully sharing the whole network\cite{ha2016toward,johnson2017google} since each language has certain unique features, and the single architecture often struggles to handle a variety of languages~\cite{platanios-etal-2018-contextual}." />
        </attvalues>
      </edge>
      <edge source="2105.03010" target="1706.03762" id="8810">
        <attvalues>
          <attvalue for="5" value=" For example, the language-specifically biased attention~\cite{zhu2020multilingual} modified the self-attention architecture~\cite{vaswani2017attention} specifically based on the assumption that each language can benefit from a bias added to the attention scores." />
        </attvalues>
      </edge>
      <edge source="2012.00004" target="1706.04971" id="8813">
        <attvalues>
          <attvalue for="5" value=" The task is getting more attention in recent years \cite{hamilton-etal-2016-diachronic,frermann-lapata-2016-bayesian,schlechtweg-etal-2017-german}." />
        </attvalues>
      </edge>
      <edge source="2012.00004" target="1906.02979" id="8814">
        <attvalues>
          <attvalue for="5" value=" There is also the synchronic LSC task, which aims to identify domain-specific changes of word senses compared to general-language usage \cite{schlechtweg-etal-2019-wind}." />
        </attvalues>
      </edge>
      <edge source="2012.00004" target="1804.06517" id="8815">
        <attvalues>
          <attvalue for="5" value=" \newcite{schlechtweg-etal-2019-wind} evaluated available approaches for LSC detection using the DURel dataset \cite{schlectweg-etal-DURel}." />
        </attvalues>
      </edge>
      <edge source="2012.00004" target="1405.3515" id="8816">
        <attvalues>
          <attvalue for="5" value=" (1) Semantic vector spaces approaches \cite{gulordava-baroni-2011-distributional,kim-etal-2014-temporal,Xu2015ACE,eger-mehler-2016-linearity,hamilton-etal-2016-cultural,hamilton-etal-2016-diachronic,rosenfeld-erk-2018-deep} represent each word with two vectors for two different time periods." />
        </attvalues>
      </edge>
      <edge source="2012.00004" target="1606.02821" id="8818">
        <attvalues>
          <attvalue for="5" value=" (1) Semantic vector spaces approaches \cite{gulordava-baroni-2011-distributional,kim-etal-2014-temporal,Xu2015ACE,eger-mehler-2016-linearity,hamilton-etal-2016-cultural,hamilton-etal-2016-diachronic,rosenfeld-erk-2018-deep} represent each word with two vectors for two different time periods." />
        </attvalues>
      </edge>
      <edge source="2012.00004" target="2001.03216" id="8819">
        <attvalues>
          <attvalue for="5" value=" (2) Topic modeling approaches \cite{wang-topics,bamman-topics,Wijaya-topic,mihalcea-nastase-2012-word,cook-etal-2014-novel,frermann-lapata-2016-bayesian,Schlechtweg20} estimate a probability distribution of words over their different senses, eg, topics." />
        </attvalues>
      </edge>
      <edge source="2012.00004" target="1807.04175" id="8820">
        <attvalues>
          <attvalue for="5" value=" We use methods for cross-lingual mapping \cite{Brychcin2019,artetxe-labaka-agirre:2016:EMNLP2016,artetxe-etal-2017,artetxe-etal-2018b,artetxe-etal-2018-robust} and thanks to the large similarity between $L_1$ and $L_2$ the quality of transformation should be high." />
        </attvalues>
      </edge>
      <edge source="2012.00004" target="1805.06297" id="8821">
        <attvalues>
          <attvalue for="5" value=" We use methods for cross-lingual mapping \cite{Brychcin2019,artetxe-labaka-agirre:2016:EMNLP2016,artetxe-etal-2017,artetxe-etal-2018b,artetxe-etal-2018-robust} and thanks to the large similarity between $L_1$ and $L_2$ the quality of transformation should be high." />
        </attvalues>
      </edge>
      <edge source="2402.12233" target="2012.14913" id="8822">
        <attvalues>
          <attvalue for="5" value=" How do pre-trained Transformer models process and store information? ~\cite{DBLP:conf/emnlp/GevaSBL21,DBLP:conf/emnlp/GevaCWG22} suggest feed-forward networks (FFNs) operate as key-value neural memories~\cite{DBLP:conf/nips/SukhbaatarSWF15}." />
        </attvalues>
      </edge>
      <edge source="2402.12233" target="2203.14680" id="8823">
        <attvalues>
          <attvalue for="5" value=" How do pre-trained Transformer models process and store information? ~\cite{DBLP:conf/emnlp/GevaSBL21,DBLP:conf/emnlp/GevaCWG22} suggest feed-forward networks (FFNs) operate as key-value neural memories~\cite{DBLP:conf/nips/SukhbaatarSWF15}." />
        </attvalues>
      </edge>
      <edge source="2402.12233" target="1503.08895" id="8824">
        <attvalues>
          <attvalue for="5" value=" How do pre-trained Transformer models process and store information? ~\cite{DBLP:conf/emnlp/GevaSBL21,DBLP:conf/emnlp/GevaCWG22} suggest feed-forward networks (FFNs) operate as key-value neural memories~\cite{DBLP:conf/nips/SukhbaatarSWF15}." />
        </attvalues>
      </edge>
      <edge source="2402.12233" target="2310.16218" id="8825">
        <attvalues>
          <attvalue for="5" value=" Based on this point of view, how can we update the information processing and storing? We employ Knowledge Editing (KE)~\cite{wang2023knowledge} as an illustrative example: altering `Paris' to `Seattle' in response to `Eiffel Tower is located in' by modifying the model weights." />
        </attvalues>
      </edge>
      <edge source="2402.12233" target="2104.08164" id="8826">
        <attvalues>
          <attvalue for="5" value=" We test the two methods in various scenarios for different pre-trained transformers, including knowledge editing~\cite{DBLP:conf/emnlp/CaoAT21, DBLP:conf/iclr/HuangSZZR023}, multi-task tuning~\cite{DBLP:conf/iclr/AribandiTSRZMZ022}, and instruction-tuning~\cite{DBLP:conf/iclr/WeiBZGYLDDL22}." />
        </attvalues>
      </edge>
      <edge source="2402.12233" target="2111.10952" id="8827">
        <attvalues>
          <attvalue for="5" value=" We test the two methods in various scenarios for different pre-trained transformers, including knowledge editing~\cite{DBLP:conf/emnlp/CaoAT21, DBLP:conf/iclr/HuangSZZR023}, multi-task tuning~\cite{DBLP:conf/iclr/AribandiTSRZMZ022}, and instruction-tuning~\cite{DBLP:conf/iclr/WeiBZGYLDDL22}." />
        </attvalues>
      </edge>
      <edge source="2402.12233" target="2109.01652" id="8828">
        <attvalues>
          <attvalue for="5" value=" We test the two methods in various scenarios for different pre-trained transformers, including knowledge editing~\cite{DBLP:conf/emnlp/CaoAT21, DBLP:conf/iclr/HuangSZZR023}, multi-task tuning~\cite{DBLP:conf/iclr/AribandiTSRZMZ022}, and instruction-tuning~\cite{DBLP:conf/iclr/WeiBZGYLDDL22}." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="1508.05326" id="8829">
        <attvalues>
          <attvalue for="5" value=" Large-scale Benchmarks such as SNLI \cite{bowman2015large}, SQUAD \cite{rajpurkar2016squad}, GLUE \cite{wang2018glue}, and Senteval \cite{conneau2018senteval} have been guiding our progress in NLP over the years." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="1606.05250" id="8830">
        <attvalues>
          <attvalue for="5" value=" Large-scale Benchmarks such as SNLI \cite{bowman2015large}, SQUAD \cite{rajpurkar2016squad}, GLUE \cite{wang2018glue}, and Senteval \cite{conneau2018senteval} have been guiding our progress in NLP over the years." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="1907.11692" id="8834">
        <attvalues>
          <attvalue for="5" value=" Often, language models such as BERT \cite{devlin2018bert}, RoBERTA \cite{liu2019roberta} and GPT3 \cite{brown2020language} beat human performance." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="2005.14165" id="8835">
        <attvalues>
          <attvalue for="5" value=" Often, language models such as BERT \cite{devlin2018bert}, RoBERTA \cite{liu2019roberta} and GPT3 \cite{brown2020language} beat human performance." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="1803.02324" id="8836">
        <attvalues>
          <attvalue for="5" value=" A growing number of recent works \cite{gururangan2018annotation, poliak2018hypothesis, kaushik2018much, tsuchiya2018performance, tan2019investigating, schwartz2017effect, swayamdipta-etal-2020-dataset, gardner-etal-2021-competency, pezeshkpour2021combining} however, expose an undesired reason this can happen: instead of learning tasks like humans, models simply overfit to spurious biases." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="1904.07911" id="8838">
        <attvalues>
          <attvalue for="5" value=" Several algorithms \cite{sakaguchi2019winogrande,li2019repair,li2018resound, wang2018dataset,clark2019don,he2019unlearn,mahabadi2019simple,zellers2018swag,nie2019adversarial,kaushik2019learning,gardner2020evaluating} have been proposed over the past few years to tackle the issue of bias." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="1908.10763" id="8840">
        <attvalues>
          <attvalue for="5" value=" Several algorithms \cite{sakaguchi2019winogrande,li2019repair,li2018resound, wang2018dataset,clark2019don,he2019unlearn,mahabadi2019simple,zellers2018swag,nie2019adversarial,kaushik2019learning,gardner2020evaluating} have been proposed over the past few years to tackle the issue of bias." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="1808.05326" id="8841">
        <attvalues>
          <attvalue for="5" value=" Several algorithms \cite{sakaguchi2019winogrande,li2019repair,li2018resound, wang2018dataset,clark2019don,he2019unlearn,mahabadi2019simple,zellers2018swag,nie2019adversarial,kaushik2019learning,gardner2020evaluating} have been proposed over the past few years to tackle the issue of bias." />
        </attvalues>
      </edge>
      <edge source="2210.07566" target="1909.12434" id="8843">
        <attvalues>
          <attvalue for="5" value=" Several algorithms \cite{sakaguchi2019winogrande,li2019repair,li2018resound, wang2018dataset,clark2019don,he2019unlearn,mahabadi2019simple,zellers2018swag,nie2019adversarial,kaushik2019learning,gardner2020evaluating} have been proposed over the past few years to tackle the issue of bias." />
        </attvalues>
      </edge>
      <edge source="2310.04027" target="1908.10063" id="8844">
        <attvalues>
          <attvalue for="5" value=" Through extensive evaluations on multiple financial sentiment analysis benchmarks, we demonstrate that compared to traditional smaller-scale sentiment analysis models \cite{araci2019finbert} and general-purpose LLMs, such as ChatGPT \cite{ouyang2022training} and LLaMA \cite{touvron2023llama}, our approach markedly outperforms them." />
        </attvalues>
      </edge>
      <edge source="2310.04027" target="2006.08097" id="8845">
        <attvalues>
          <attvalue for="5" value=" Traditional NLP models, constrained by the limitations of their model parameters and the scale of their training corpora, often lack the capability to comprehensively understand intricate financial news, thereby limiting the efficacy of financial sentiment analysis \cite{araci2019finbert,yang2020finbert,sohangir2018big,day2016deep}." />
        </attvalues>
      </edge>
      <edge source="2310.04027" target="2303.17564" id="8846">
        <attvalues>
          <attvalue for="5" value=" In contrast, the advent of large language models (LLMs) \cite{ouyang2022training,touvron2023llama,wu2023bloomberggpt,yang2023fingpt} has ushered in a new era in the NLP domain." />
        </attvalues>
      </edge>
      <edge source="2310.04027" target="2306.06031" id="8847">
        <attvalues>
          <attvalue for="5" value=" In contrast, the advent of large language models (LLMs) \cite{ouyang2022training,touvron2023llama,wu2023bloomberggpt,yang2023fingpt} has ushered in a new era in the NLP domain." />
        </attvalues>
      </edge>
      <edge source="2310.04027" target="2201.08239" id="8850">
        <attvalues>
          <attvalue for="5" value=" Firstly, the discrepancy between the objective function used in LLMs' pre-training and the goal of predicting financial sentiment may result in LLMs' inability to consistently output labels for financial sentiment analysis as expected \cite{thoppilan2022lamda,ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2310.04027" target="1307.5336" id="8851">
        <attvalues>
          <attvalue for="5" value=" The scarcity of information has not only interfered with the judgment of human experts \cite{malo2014good} but also poses a significant challenge to the accurate prediction of large language models." />
        </attvalues>
      </edge>
      <edge source="2310.04027" target="2005.11401" id="8852">
        <attvalues>
          <attvalue for="5" value=" 2) retrieval-augmented component \cite{lewis2020retrieval}, which introduces additional context to brief statements from news flashes or tweets." />
        </attvalues>
      </edge>
      <edge source="2310.07929" target="1904.09077" id="8854">
        <attvalues>
          <attvalue for="5" value=" Multilingual language models share representations across languages \cite{artetxe-etal-2020-cross, conneau-etal-2020-emerging}, which is thought to enable their crosslingual transfer abilities \cite{wu-dredze-2019-beto,Chi_Dong_Wei_Wang_Mao_Huang_2020,pmlr-v119-hu20b,winata-etal-2021-language,winata-etal-2022-cross}." />
        </attvalues>
      </edge>
      <edge source="2310.07929" target="1909.10481" id="8855">
        <attvalues>
          <attvalue for="5" value=" Multilingual language models share representations across languages \cite{artetxe-etal-2020-cross, conneau-etal-2020-emerging}, which is thought to enable their crosslingual transfer abilities \cite{wu-dredze-2019-beto,Chi_Dong_Wei_Wang_Mao_Huang_2020,pmlr-v119-hu20b,winata-etal-2021-language,winata-etal-2022-cross}." />
        </attvalues>
      </edge>
      <edge source="2310.07929" target="2109.07684" id="8856">
        <attvalues>
          <attvalue for="5" value=" Multilingual language models share representations across languages \cite{artetxe-etal-2020-cross, conneau-etal-2020-emerging}, which is thought to enable their crosslingual transfer abilities \cite{wu-dredze-2019-beto,Chi_Dong_Wei_Wang_Mao_Huang_2020,pmlr-v119-hu20b,winata-etal-2021-language,winata-etal-2022-cross}." />
        </attvalues>
      </edge>
      <edge source="2310.07929" target="2109.14989" id="8857">
        <attvalues>
          <attvalue for="5" value=" Because the grammatical structure is primed rather than a specific semantic meaning, \cite{sinclair_2022_StructuralPersistenceLanguage} argue that structural priming effects provide evidence for abstract grammatical representations in language models." />
        </attvalues>
      </edge>
      <edge source="2310.07929" target="1909.10579" id="8858">
        <attvalues>
          <attvalue for="5" value=" Specifically, does crosslingual structural priming occur, and how much pre-training data does it require? Structural priming is a phenomenon in which after being presented with a sentence with a given grammatical structure, people (and language models; \cite{sinclair_2022_StructuralPersistenceLanguage}) are more likely to produce a sentence with the same structure \cite{bock_1986_SyntacticPersistenceLanguage,prasad-etal-2019-using,frank_2021_CrosslanguageStructuralPriming,li-etal-2022-neural,choi2022syntactic2}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="1904.12848" id="8860">
        <attvalues>
          <attvalue for="5" value=" Data augmentation (DA) is a common strategy to generate novel label-preserving data to remedy data scarcity and imbalance problems \cite{xie2020unsupervised}, which has been applied with noteworthy success in image and speech recognition \cite{Iwana2021,Park2019,Shorten2019ASO}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="2007.15951" id="8861">
        <attvalues>
          <attvalue for="5" value=" Data augmentation (DA) is a common strategy to generate novel label-preserving data to remedy data scarcity and imbalance problems \cite{xie2020unsupervised}, which has been applied with noteworthy success in image and speech recognition \cite{Iwana2021,Park2019,Shorten2019ASO}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="1904.08779" id="8862">
        <attvalues>
          <attvalue for="5" value=" Data augmentation (DA) is a common strategy to generate novel label-preserving data to remedy data scarcity and imbalance problems \cite{xie2020unsupervised}, which has been applied with noteworthy success in image and speech recognition \cite{Iwana2021,Park2019,Shorten2019ASO}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="2105.03075" id="8863">
        <attvalues>
          <attvalue for="5" value=" In the field of natural language processing (NLP), there have also been a number of studies that use various DA techniques to boost the trained models’ performance \cite{feng-etal-2021-survey,Liu:9240734}, ranging from word replacement \cite{wang-yang-2015-thats,wang-etal-2018-switchout,Zhang2015}, to predictive neural language models \cite{hou2018,kobayashi-2018-contextual,Kurata2016LabeledDG}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="1808.07512" id="8864">
        <attvalues>
          <attvalue for="5" value=" In the field of natural language processing (NLP), there have also been a number of studies that use various DA techniques to boost the trained models’ performance \cite{feng-etal-2021-survey,Liu:9240734}, ranging from word replacement \cite{wang-yang-2015-thats,wang-etal-2018-switchout,Zhang2015}, to predictive neural language models \cite{hou2018,kobayashi-2018-contextual,Kurata2016LabeledDG}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="1509.01626" id="8865">
        <attvalues>
          <attvalue for="5" value=" In the field of natural language processing (NLP), there have also been a number of studies that use various DA techniques to boost the trained models’ performance \cite{feng-etal-2021-survey,Liu:9240734}, ranging from word replacement \cite{wang-yang-2015-thats,wang-etal-2018-switchout,Zhang2015}, to predictive neural language models \cite{hou2018,kobayashi-2018-contextual,Kurata2016LabeledDG}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="1807.01554" id="8866">
        <attvalues>
          <attvalue for="5" value=" In the field of natural language processing (NLP), there have also been a number of studies that use various DA techniques to boost the trained models’ performance \cite{feng-etal-2021-survey,Liu:9240734}, ranging from word replacement \cite{wang-yang-2015-thats,wang-etal-2018-switchout,Zhang2015}, to predictive neural language models \cite{hou2018,kobayashi-2018-contextual,Kurata2016LabeledDG}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="1805.06201" id="8867">
        <attvalues>
          <attvalue for="5" value=" In the field of natural language processing (NLP), there have also been a number of studies that use various DA techniques to boost the trained models’ performance \cite{feng-etal-2021-survey,Liu:9240734}, ranging from word replacement \cite{wang-yang-2015-thats,wang-etal-2018-switchout,Zhang2015}, to predictive neural language models \cite{hou2018,kobayashi-2018-contextual,Kurata2016LabeledDG}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="1901.11196" id="8868">
        <attvalues>
          <attvalue for="5" value=" Simple as these operations may seem, they have shown general success in various sentiment-related and sentence type classification tasks \cite{wei-zou-2019-eda}." />
        </attvalues>
      </edge>
      <edge source="2209.00797" target="2101.05469" id="8869">
        <attvalues>
          <attvalue for="5" value=" This approach consists of four commonly used token-level editing operations \cite{wei-etal-2021-text,wei-zou-2019-eda}, eg, Synonym Replacement (SR), Random Insertion (RI), Random Swap (RS), and Random Deletion (RD)." />
        </attvalues>
      </edge>
      <edge source="2003.07914" target="1402.4182" id="8870">
        <attvalues>
          <attvalue for="5" value=" First, source code LMs have been used in a diverse variety of tools well beyond the obvious application of autocompletion, ranging from code readability \cite{Allamanis2014} to program repair \cite{chen2018sequencer}." />
        </attvalues>
      </edge>
      <edge source="2003.07914" target="1901.01808" id="8871">
        <attvalues>
          <attvalue for="5" value=" First, source code LMs have been used in a diverse variety of tools well beyond the obvious application of autocompletion, ranging from code readability \cite{Allamanis2014} to program repair \cite{chen2018sequencer}." />
        </attvalues>
      </edge>
      <edge source="2003.07914" target="1506.01159" id="8872">
        <attvalues>
          <attvalue for="5" value=" To show that improvement in language modelling transfers to downstream SE tasks, we conduct an experiment similar to \cite{Ray2016}, who showed that language models can be used to highlight buggy code." />
        </attvalues>
      </edge>
      <edge source="2003.07914" target="1709.06182" id="8874">
        <attvalues>
          <attvalue for="5" value=" Many works have taken advantage of the ``naturalness'' of software \cite{Hindle2012} to assist software engineering tasks, including code completion \cite{Raychev2014}, improving code readability \cite{Allamanis2014}, program repair \cite{santos2018syntax,chen2018sequencer}, identifying buggy code \cite{Ray2016} and API migration \cite{gu2017deepam}, among many others \cite{big-code-survey}." />
        </attvalues>
      </edge>
      <edge source="2003.07914" target="1508.07909" id="8875">
        <attvalues>
          <attvalue for="5" value=" However, we find that the most common ways to reduce vocabulary that were previously considered in the software engineering literature, such as splitting identifiers according to underscores and case, are not enough to obtain a vocabulary of a manageable size; advanced approaches such as adaptations of the Byte-Pair Encoding (BPE) algorithm \cite{Gage1994,Sennrich2015} are needed to reach this goal and deal with the OOV problem." />
        </attvalues>
      </edge>
      <edge source="2003.07914" target="1801.06146" id="8876">
        <attvalues>
          <attvalue for="5" value=" Second, recent results in NLP \cite{howard2018universal, Peters2018, Devlin2018} show that NLMs can be used as upstream tasks in transfer learning, leading to state-of-the-art improvement in downstream tasks: for instance, a model can be pre-trained as an NLM, and later on fine-tuned as a classifier." />
        </attvalues>
      </edge>
      <edge source="2308.06385" target="2204.05862" id="8877">
        <attvalues>
          <attvalue for="5" value=" By optimizing for key qualities such as harmlessness and helpfulness, this technique, as underscored by \cite{bai2022training}, has gained significant ground." />
        </attvalues>
      </edge>
      <edge source="2308.06385" target="2303.08774" id="8878">
        <attvalues>
          <attvalue for="5" value=" Furthermore, this approach has proven its efficacy by attaining unprecedented results across a wide array of natural language tasks \cite{openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2308.06385" target="1707.06347" id="8879">
        <attvalues>
          <attvalue for="5" value=" The conventional RLHF pipeline refines an initial, non-aligned LLM by employing an online RL algorithm like the popular Proximal Policy Optimization (PPO) \cite{schulman2017proximal}." />
        </attvalues>
      </edge>
      <edge source="2308.06385" target="2212.08073" id="8880">
        <attvalues>
          <attvalue for="5" value=" RLAIF methods \cite{bai2022constitutional} aim to mimic human binary preferences by assigning scores to outputs $o_1$ and $o_2$ utilizing a LLM." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="1211.3711" id="8886">
        <attvalues>
          <attvalue for="5" value=" As an S2S model, recurrent neural network transducer (RNN-T)~\cite{graves2012sequence} and its variants have achieved high-accuracy and low-latency in streaming on-device speech recognition~\cite{rnntshibie,graves2013speech}." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="1811.06621" id="8887">
        <attvalues>
          <attvalue for="5" value=" As an S2S model, recurrent neural network transducer (RNN-T)~\cite{graves2012sequence} and its variants have achieved high-accuracy and low-latency in streaming on-device speech recognition~\cite{rnntshibie,graves2013speech}." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="1303.5778" id="8888">
        <attvalues>
          <attvalue for="5" value=" As an S2S model, recurrent neural network transducer (RNN-T)~\cite{graves2012sequence} and its variants have achieved high-accuracy and low-latency in streaming on-device speech recognition~\cite{rnntshibie,graves2013speech}." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="1706.03762" id="8889">
        <attvalues>
          <attvalue for="5" value=" Neural transducer has the streaming decoding ability in nature, while other E2E competitors, particularly those based on attention mechanism, such as transformer~\cite{transformer,povey2018time,dong2018speech} and listen, attend and spell~\cite{las} (LAS), have to be modified to possess the streaming ability." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="2002.11268" id="8890">
        <attvalues>
          <attvalue for="5" value=" A common solution is to use a language model (LM) fusion strategy: an LM is first externally trained on text data and then incorporated into the E2E model~\cite{shallowfusion,zhao2019shallow}." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="2005.07157" id="8891">
        <attvalues>
          <attvalue for="5" value=" Work in ~\cite{ttsrnnt,ttsrnnt2} has shown that data augmentation with text-to-speech utterances yields improvement to E2E models; however, there still remains a substantial gap in performance between models trained on human speech and those trained on synthesized speech." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="1909.11699" id="8892">
        <attvalues>
          <attvalue for="5" value=" Work in ~\cite{ttsrnnt,ttsrnnt2} has shown that data augmentation with text-to-speech utterances yields improvement to E2E models; however, there still remains a substantial gap in performance between models trained on human speech and those trained on synthesized speech." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="2004.11544" id="8893">
        <attvalues>
          <attvalue for="5" value=" Recently, a two-pass RNN-T+LAS model, where LAS rescores hypotheses from RNN-T, has been proposed~\cite{rnntlas} and improved further with more tricks~\cite{rnntlas2}." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="2003.12710" id="8894">
        <attvalues>
          <attvalue for="5" value=" Recently, a two-pass RNN-T+LAS model, where LAS rescores hypotheses from RNN-T, has been proposed~\cite{rnntlas} and improved further with more tricks~\cite{rnntlas2}." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="1801.00841" id="8895">
        <attvalues>
          <attvalue for="5" value=" Most approaches on neural transducer have been conducted on English corpora and different modeling units, such as phonemes, grapheme and word-piece, have been explored~\cite{rao2017exploring}." />
        </attvalues>
      </edge>
      <edge source="2011.08469" target="1811.05097" id="8896">
        <attvalues>
          <attvalue for="5" value=" There are several studies on LAS and Transformer based Mandarin ASR, but we only find one paper on the use of RNN-T in Mandarin which shows its feasibility on modeling Chinese characters~\cite{sengmaornnt}." />
        </attvalues>
      </edge>
      <edge source="1909.09010" target="1807.07987" id="8897">
        <attvalues>
          <attvalue for="5" value=" Machine learning, and in particular deep learning technology \cite{DeepLearning} powers many aspects of modern lives." />
        </attvalues>
      </edge>
      <edge source="1909.09010" target="1901.02860" id="8898">
        <attvalues>
          <attvalue for="5" value=" These technologies have been widely implemented in a plenty of fields, such as language modeling \cite{Transformerxl,jozefowicz2016exploring}, natural language processing (NLP) \cite{vaswani2017attention,devlin2018bert} and large vocabulary continuous speech recognition (LVCSR) \cite{dahl2011context,graves2014towards}." />
        </attvalues>
      </edge>
      <edge source="1909.09010" target="1602.02410" id="8899">
        <attvalues>
          <attvalue for="5" value=" These technologies have been widely implemented in a plenty of fields, such as language modeling \cite{Transformerxl,jozefowicz2016exploring}, natural language processing (NLP) \cite{vaswani2017attention,devlin2018bert} and large vocabulary continuous speech recognition (LVCSR) \cite{dahl2011context,graves2014towards}." />
        </attvalues>
      </edge>
      <edge source="1909.09010" target="1706.03762" id="8900">
        <attvalues>
          <attvalue for="5" value=" These technologies have been widely implemented in a plenty of fields, such as language modeling \cite{Transformerxl,jozefowicz2016exploring}, natural language processing (NLP) \cite{vaswani2017attention,devlin2018bert} and large vocabulary continuous speech recognition (LVCSR) \cite{dahl2011context,graves2014towards}." />
        </attvalues>
      </edge>
      <edge source="1909.09010" target="1412.6651" id="8902">
        <attvalues>
          <attvalue for="5" value=" Elastic averaging SGD \cite{zhang2015deep} has been proposed recently and is the state-of-art asynchronous parameter-server method." />
        </attvalues>
      </edge>
      <edge source="1909.09010" target="1410.7455" id="8903">
        <attvalues>
          <attvalue for="5" value=" These methods achieve nearly linear speedups but suffer from accuracy degradation \cite{Block-Momentum,povey2014parallel}." />
        </attvalues>
      </edge>
      <edge source="1909.09010" target="1907.05698" id="8904">
        <attvalues>
          <attvalue for="5" value=" BMUF \cite{Block-Momentum}, proposed to tackle the degradation problem in MA, is widely used in speech recognition \cite{li2018improving,zhaoyou2019}." />
        </attvalues>
      </edge>
      <edge source="1909.09010" target="1703.05880" id="8905">
        <attvalues>
          <attvalue for="5" value=" The results in \cite{Empirical17} showed that BMUF outperforms EASGD and ASGD on speech recognition tasks." />
        </attvalues>
      </edge>
      <edge source="2211.00490" target="1211.3711" id="8906">
        <attvalues>
          <attvalue for="5" value=" As a prominent example, transducer~\cite{transducer, transformer-transducer, pruned-rnnt} has gained more and more popularity for real-time ASR system development, because it is naturally streaming and demonstrates superior performance." />
        </attvalues>
      </edge>
      <edge source="2211.00490" target="2010.11148" id="8907">
        <attvalues>
          <attvalue for="5" value=" Different from FastEmit~\cite{fastemit} that directly changes the derivatives, we modify the log-probabilities of emitting symbols by adding a small constant $\lambda$ times the frame offsets relative to middle frame." />
        </attvalues>
      </edge>
      <edge source="2211.00490" target="2105.05005" id="8908">
        <attvalues>
          <attvalue for="5" value=" Another work named Self alignment~\cite{self-alignment} proposes to boost the log-probability of the alignment that is one frame to the left of the Viterbi forced-alignment, which requires an extra recursion with a time complexity of $\mathcal{O}(T \times U)$ to obtain the Viterbi forced-alignment, where $T$ and $U$ are the lengths of frame sequence and token sequence respectively." />
        </attvalues>
      </edge>
      <edge source="2404.17347" target="2005.11401" id="8909">
        <attvalues>
          <attvalue for="5" value=" The recent advances in Large Language Models (LLMs) have led to an explosion of research on Retrieval-Augmented Generation (RAG): combining generative LLMs with data retrieval to provide responses grounded on authoritative document collections \cite{rag-neurips-2020}." />
        </attvalues>
      </edge>
      <edge source="2404.17347" target="2312.10997" id="8910">
        <attvalues>
          <attvalue for="5" value=" RAG systems have been deployed in diverse domains (see \cite{gao2024retrievalaugmented} for a recent survey)." />
        </attvalues>
      </edge>
      <edge source="2404.17347" target="2309.01431" id="8912">
        <attvalues>
          <attvalue for="5" value=" Recognizing the importance of evaluation, the research community has been creating evaluation benchmark datasets \cite{liu2023recall,chen2023benchmarking}, evaluation metrics \cite{es2023ragas}, and noise robustness \cite{chen2023benchmarking}, as well as evaluation frameworks (such as RAGAs \cite{es2023ragas}, and ARES \cite{saadfalcon2023ares})." />
        </attvalues>
      </edge>
      <edge source="2404.17347" target="2311.09476" id="8914">
        <attvalues>
          <attvalue for="5" value=" Recognizing the importance of evaluation, the research community has been creating evaluation benchmark datasets \cite{liu2023recall,chen2023benchmarking}, evaluation metrics \cite{es2023ragas}, and noise robustness \cite{chen2023benchmarking}, as well as evaluation frameworks (such as RAGAs \cite{es2023ragas}, and ARES \cite{saadfalcon2023ares})." />
        </attvalues>
      </edge>
      <edge source="2404.17347" target="2206.11249" id="8915">
        <attvalues>
          <attvalue for="5" value=" We suggest that a comprehensive evaluation of RAG systems should include the following: Aggregate Performance: Overall evaluation is important for continuous benchmarking of models and dataset performance \cite{gehrmann-etal-2022-gemv2}." />
        </attvalues>
      </edge>
      <edge source="2304.04726" target="1508.05326" id="8917">
        <attvalues>
          <attvalue for="5" value=" Some datasets like SNLI \cite{bowman-etal-2015-large} and MNLI \cite{williams-etal-2018-broad} do, however, contain information about different readings in the form of annotation disagreement." />
        </attvalues>
      </edge>
      <edge source="2304.04726" target="1704.05426" id="8918">
        <attvalues>
          <attvalue for="5" value=" Some datasets like SNLI \cite{bowman-etal-2015-large} and MNLI \cite{williams-etal-2018-broad} do, however, contain information about different readings in the form of annotation disagreement." />
        </attvalues>
      </edge>
      <edge source="2304.04726" target="1902.02476" id="8920">
        <attvalues>
          <attvalue for="5" value=" Moving towards uncertainty-aware neural language models, we present our initial results using Stochastic Weight Averaging (SWA) \cite{izmailov2018averaging} and SWA-Gaussian (SWAG) \cite{maddox2019simple} on the task of Natural Language Inference." />
        </attvalues>
      </edge>
      <edge source="2010.02448" target="1708.02182" id="8921">
        <attvalues>
          <attvalue for="5" value=" Neural language models such as LSTM~\cite{merityRegOpt,peters2018deep}, GPT2~\cite{radford2019language}, and BERT~\cite{devlin2019bert,liu2019roberta} have achieved state-of-the-art performance in various downstream NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2010.02448" target="1802.05365" id="8922">
        <attvalues>
          <attvalue for="5" value=" Neural language models such as LSTM~\cite{merityRegOpt,peters2018deep}, GPT2~\cite{radford2019language}, and BERT~\cite{devlin2019bert,liu2019roberta} have achieved state-of-the-art performance in various downstream NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2010.02448" target="1907.11692" id="8924">
        <attvalues>
          <attvalue for="5" value=" Neural language models such as LSTM~\cite{merityRegOpt,peters2018deep}, GPT2~\cite{radford2019language}, and BERT~\cite{devlin2019bert,liu2019roberta} have achieved state-of-the-art performance in various downstream NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2010.02448" target="1906.01958" id="8925">
        <attvalues>
          <attvalue for="5" value=" One interesting line of these works tries to extract discrete constituency trees from pre-trained language models \cite{marevcek2018extracting,marevcek2019balustrades,kim2020pre, wu2020perturbed}." />
        </attvalues>
      </edge>
      <edge source="2010.02448" target="2002.00737" id="8926">
        <attvalues>
          <attvalue for="5" value=" Using our approach, we find that some recent works on pre-trained language models suffer from the branching bias~\cite{kim2020pre,wu2020perturbed, marevcek2018extracting}." />
        </attvalues>
      </edge>
      <edge source="2010.02448" target="1909.09428" id="8927">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, the approach to measuring the bias in~\cite{dyer2019critical} is highly dependent on the architecture of ordered neuron and its parsing algorithm." />
        </attvalues>
      </edge>
      <edge source="2010.02448" target="1810.09536" id="8928">
        <attvalues>
          <attvalue for="5" value=" For example, as pointed out by~\cite{dyer2019critical}, the syntax extracted from the ordered neuron based language model~\cite{shen2018ordered} is biased to right-branching languages (eg, English)." />
        </attvalues>
      </edge>
      <edge source="2010.02448" target="2004.14786" id="8929">
        <attvalues>
          <attvalue for="5" value=" Using our approach, we find that some recent works on pre-trained language models suffer from the branching bias~\cite{kim2020pre,wu2020perturbed, marevcek2018extracting}." />
        </attvalues>
      </edge>
      <edge source="2311.02883" target="2109.05093" id="8930">
        <attvalues>
          <attvalue for="5" value=" Notable previous work on finetuning, including PICARD \cite{scholak2021picard}, UnifiedSKG \cite{xie2022unifiedskg}, and RESDSQL-3B + NatSQL\cite{li2023decoupling}, achieve impressive results by leveraging customized SQL-specific syntax knowledge and training on a large number of (text, SQL) paired data samples." />
        </attvalues>
      </edge>
      <edge source="2311.02883" target="2201.05966" id="8931">
        <attvalues>
          <attvalue for="5" value=" Notable previous work on finetuning, including PICARD \cite{scholak2021picard}, UnifiedSKG \cite{xie2022unifiedskg}, and RESDSQL-3B + NatSQL\cite{li2023decoupling}, achieve impressive results by leveraging customized SQL-specific syntax knowledge and training on a large number of (text, SQL) paired data samples." />
        </attvalues>
      </edge>
      <edge source="2311.02883" target="2005.14165" id="8932">
        <attvalues>
          <attvalue for="5" value=" Recently, large language models (LLMs) such as GPT-3 \cite{brown2020language}, PaLM \cite{chowdhery2022palm}, and ChatGPT\cite{stiennon2020learning} have demonstrated promising few-shot abilities via prompting\cite{wei2022emergent}." />
        </attvalues>
      </edge>
      <edge source="2311.02883" target="2204.02311" id="8933">
        <attvalues>
          <attvalue for="5" value=" Recently, large language models (LLMs) such as GPT-3 \cite{brown2020language}, PaLM \cite{chowdhery2022palm}, and ChatGPT\cite{stiennon2020learning} have demonstrated promising few-shot abilities via prompting\cite{wei2022emergent}." />
        </attvalues>
      </edge>
      <edge source="2311.02883" target="2203.11171" id="8935">
        <attvalues>
          <attvalue for="5" value=" In few-shot prompting, self-consistency decoding \cite{wang2022self}, which samples a diverse set of reasoning paths and selects the most consistent answer, has shown remarkable performance improvements across different tasks." />
        </attvalues>
      </edge>
      <edge source="2109.05093" target="1810.02720" id="8936">
        <attvalues>
          <attvalue for="5" value=" For a while now it has been possible to restrict auto-regressive decoding to only those token sequences that correctly parse to SQL abstract syntax trees \cite{Yin_2018,lin2019grammarbased,Wang_2020}." />
        </attvalues>
      </edge>
      <edge source="2109.05093" target="1905.13326" id="8937">
        <attvalues>
          <attvalue for="5" value=" For a while now it has been possible to restrict auto-regressive decoding to only those token sequences that correctly parse to SQL abstract syntax trees \cite{Yin_2018,lin2019grammarbased,Wang_2020}." />
        </attvalues>
      </edge>
      <edge source="2109.05093" target="1911.04942" id="8938">
        <attvalues>
          <attvalue for="5" value=" For a while now it has been possible to restrict auto-regressive decoding to only those token sequences that correctly parse to SQL abstract syntax trees \cite{Yin_2018,lin2019grammarbased,Wang_2020}." />
        </attvalues>
      </edge>
      <edge source="2109.05093" target="2010.12412" id="8939">
        <attvalues>
          <attvalue for="5" value=" More recently, semi-auto-regressive improvements to this parsing paradigm have been proposed \cite{rubin2020smbop}." />
        </attvalues>
      </edge>
      <edge source="2109.05093" target="2012.12627" id="8940">
        <attvalues>
          <attvalue for="5" value=" A less invasive and more compatible approach is to not constrain the generation process, but instead to filter finalized beam hypotheses by validity \cite{suhr-etal-2020-exploring,Lin_2020}." />
        </attvalues>
      </edge>
      <edge source="2109.05093" target="1909.05378" id="8943">
        <attvalues>
          <attvalue for="5" value=" Significantly, with the help of \Picard, a T5-3B model can be raised to state-of-the-art performance on the Spider and CoSQL datasets \cite{yu-etal-2019-cosql}." />
        </attvalues>
      </edge>
      <edge source="2406.02396" target="1908.10084" id="8944">
        <attvalues>
          <attvalue for="5" value=" Natural language embeddings are used in a diverse range of applications, including clustering~\cite{liu2011survey, Angelov2020Top2VecDR}, text mining~\cite{jiang2015training}, semantic search~\cite{reimers2019sentence,muennighoff2022sgpt} and feature representation~\cite{alayrac2022flamingo}." />
        </attvalues>
      </edge>
      <edge source="2406.02396" target="2204.14198" id="8946">
        <attvalues>
          <attvalue for="5" value=" Natural language embeddings are used in a diverse range of applications, including clustering~\cite{liu2011survey, Angelov2020Top2VecDR}, text mining~\cite{jiang2015training}, semantic search~\cite{reimers2019sentence,muennighoff2022sgpt} and feature representation~\cite{alayrac2022flamingo}." />
        </attvalues>
      </edge>
      <edge source="2406.02396" target="2210.07316" id="8948">
        <attvalues>
          <attvalue for="5" value=" Recent work has proposed mteb \cite{muennighoff-etal-2023-mteb}, a benchmark for evaluating the quality of document embeddings for a wide variety of tasks." />
        </attvalues>
      </edge>
      <edge source="2406.02396" target="2309.07597" id="8949">
        <attvalues>
          <attvalue for="5" value=" This is especially problematic since recent approaches such as prompt-based embedding models~\cite{muennighoff2022sgpt,xiao2023c,su2022one}, Matryoshka embeddings \cite{NEURIPS2022_c32319f4} introduce variables which can dramatically influence performance." />
        </attvalues>
      </edge>
      <edge source="2406.02396" target="2212.09741" id="8950">
        <attvalues>
          <attvalue for="5" value=" This is especially problematic since recent approaches such as prompt-based embedding models~\cite{muennighoff2022sgpt,xiao2023c,su2022one}, Matryoshka embeddings \cite{NEURIPS2022_c32319f4} introduce variables which can dramatically influence performance." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2004.04906" id="8952">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) have recently been leveraged across bi-encoder~\cite{karpukhin-etal-2020-dense, xiong2021approximate,zhuang2021tilde,wang2022simlm,gao-callan-2022-unsupervised}, cross-encoder~\cite{nogueira2019passage, nogueira-etal-2020-document,zhuang2021QLMT5}, and sparse~\cite{lin2021few,Formal2021SPLADE,zhuang2021fast} ranker architectures, showing impressive ranking effectiveness." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2207.02578" id="8953">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) have recently been leveraged across bi-encoder~\cite{karpukhin-etal-2020-dense, xiong2021approximate,zhuang2021tilde,wang2022simlm,gao-callan-2022-unsupervised}, cross-encoder~\cite{nogueira2019passage, nogueira-etal-2020-document,zhuang2021QLMT5}, and sparse~\cite{lin2021few,Formal2021SPLADE,zhuang2021fast} ranker architectures, showing impressive ranking effectiveness." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2108.05540" id="8954">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) have recently been leveraged across bi-encoder~\cite{karpukhin-etal-2020-dense, xiong2021approximate,zhuang2021tilde,wang2022simlm,gao-callan-2022-unsupervised}, cross-encoder~\cite{nogueira2019passage, nogueira-etal-2020-document,zhuang2021QLMT5}, and sparse~\cite{lin2021few,Formal2021SPLADE,zhuang2021fast} ranker architectures, showing impressive ranking effectiveness." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="1901.04085" id="8955">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) have recently been leveraged across bi-encoder~\cite{karpukhin-etal-2020-dense, xiong2021approximate,zhuang2021tilde,wang2022simlm,gao-callan-2022-unsupervised}, cross-encoder~\cite{nogueira2019passage, nogueira-etal-2020-document,zhuang2021QLMT5}, and sparse~\cite{lin2021few,Formal2021SPLADE,zhuang2021fast} ranker architectures, showing impressive ranking effectiveness." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2107.05720" id="8957">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) have recently been leveraged across bi-encoder~\cite{karpukhin-etal-2020-dense, xiong2021approximate,zhuang2021tilde,wang2022simlm,gao-callan-2022-unsupervised}, cross-encoder~\cite{nogueira2019passage, nogueira-etal-2020-document,zhuang2021QLMT5}, and sparse~\cite{lin2021few,Formal2021SPLADE,zhuang2021fast} ranker architectures, showing impressive ranking effectiveness." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2108.08513" id="8958">
        <attvalues>
          <attvalue for="5" value=" Pre-trained language models (PLMs) have recently been leveraged across bi-encoder~\cite{karpukhin-etal-2020-dense, xiong2021approximate,zhuang2021tilde,wang2022simlm,gao-callan-2022-unsupervised}, cross-encoder~\cite{nogueira2019passage, nogueira-etal-2020-document,zhuang2021QLMT5}, and sparse~\cite{lin2021few,Formal2021SPLADE,zhuang2021fast} ranker architectures, showing impressive ranking effectiveness." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2204.00716" id="8960">
        <attvalues>
          <attvalue for="5" value=" Despite this success, the strong effectiveness of PLM-based rankers does not always generalise without sufficient in-domain training data~\cite{thakur2021beir,zhuang-zuccon-2021-dealing,zhuang2022character}." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2005.14165" id="8962">
        <attvalues>
          <attvalue for="5" value=" Alternatively, generative large language models (LLMs) like GPT3~\cite{Brown2020GPT3} have been used to synthesize domain-specific training queries, which are then used to train these rankers~\cite{Luiz2022inpars, dai2023promptagator}." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2010.03073" id="8965">
        <attvalues>
          <attvalue for="5" value=" These PLM-based QLMs are fine-tuned on query generation tasks and subsequently employed to rank documents as per their likelihood~\cite{nogueira-dos-santos-etal-2020-beyond,zhuang2021QLMT5,Lesota2021QLM,zhuang2021tilde}." />
        </attvalues>
      </edge>
      <edge source="2310.13243" target="2110.08207" id="8968">
        <attvalues>
          <attvalue for="5" value=" Empirical results show that using the T0 LLM~\cite{sanh2022multitask} as a QLM, large gains in ranking effectiveness can be obtained." />
        </attvalues>
      </edge>
      <edge source="2406.18192" target="2403.04132" id="8989">
        <attvalues>
          <attvalue for="5" value=" In the esteemed rankings of the global language model benchmark platform, Chatbot Arena\cite{Chiang2024ChatbotAA}, reveals that the top-performing LLMs~\cite{Claude3,achiam2023gpt,team2023gemini} are predominantly English-based." />
        </attvalues>
      </edge>
      <edge source="2406.18192" target="2303.08774" id="8990">
        <attvalues>
          <attvalue for="5" value=" In the esteemed rankings of the global language model benchmark platform, Chatbot Arena\cite{Chiang2024ChatbotAA}, reveals that the top-performing LLMs~\cite{Claude3,achiam2023gpt,team2023gemini} are predominantly English-based." />
        </attvalues>
      </edge>
      <edge source="2406.18192" target="2312.11805" id="8991">
        <attvalues>
          <attvalue for="5" value=" In the esteemed rankings of the global language model benchmark platform, Chatbot Arena\cite{Chiang2024ChatbotAA}, reveals that the top-performing LLMs~\cite{Claude3,achiam2023gpt,team2023gemini} are predominantly English-based." />
        </attvalues>
      </edge>
      <edge source="2311.16083" target="2005.14165" id="8994">
        <attvalues>
          <attvalue for="5" value=" So far, this cross-influence of topics and styles has not been studied in the context of PLMs such as BERT \cite{devlin2018bert}, T5 \cite{raffel20} or GPTs \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2311.16083" target="2012.09392" id="8996">
        <attvalues>
          <attvalue for="5" value=" However it has been noticed \cite{hendrycks20pretrained,moon2021masker} that while in general PLMs are more robust than previous models, they still suffer from spurious domain-specific clues." />
        </attvalues>
      </edge>
      <edge source="2104.04487" target="1211.3711" id="8998">
        <attvalues>
          <attvalue for="5" value=" The most common E2E architectures are either attention-based (eg, listen, attention, and spell (LAS)~\cite{chan_las_icassp2016}) or RNNT~\cite{graves_arxiv2012} models (see~\cite{prabhavalkar_is2017} for comparison)." />
        </attvalues>
      </edge>
      <edge source="2104.04487" target="1811.06621" id="9000">
        <attvalues>
          <attvalue for="5" value="~\cite{he_icassp2019} presented an RNNT based real-time steaming recognizer, which outperformed classic ASR models by a wide margin." />
        </attvalues>
      </edge>
      <edge source="2104.04487" target="1907.01372" id="9001">
        <attvalues>
          <attvalue for="5" value=" Although E2E models have overall shown strong results, they have been shown to have difficulty accurately modeling tail phenomena such as proper nouns, numerics, and accented speech~\cite{Crosslingual1998,NonNative2014,Numeric2019,ProperNoun2020}, due to the requirement that they be trained on paired (speech-transcript) data." />
        </attvalues>
      </edge>
      <edge source="2104.04487" target="1712.01996" id="9004">
        <attvalues>
          <attvalue for="5" value=" Recent papers have proposed fusing E2E models with LMs trained with text data (usually referred to this as fusion), including shallow fusion~\cite{glehre_2015OnUM,kannan_icassp2018}, deep fusion~\cite{glehre_2015OnUM}, cold fusion~\cite{sriram_archiv2015}, component fusion~\cite{shan_icassp2019}, etc Most experiments used neural LMs, and some used n-gram fst LMs~\cite{chan_las_icassp2016,bahdanau_icassp2016,glehre_2015OnUM,chorowski_is2017}." />
        </attvalues>
      </edge>
      <edge source="2104.04487" target="1708.06426" id="9005">
        <attvalues>
          <attvalue for="5" value=" Recent papers have proposed fusing E2E models with LMs trained with text data (usually referred to this as fusion), including shallow fusion~\cite{glehre_2015OnUM,kannan_icassp2018}, deep fusion~\cite{glehre_2015OnUM}, cold fusion~\cite{sriram_archiv2015}, component fusion~\cite{shan_icassp2019}, etc Most experiments used neural LMs, and some used n-gram fst LMs~\cite{chan_las_icassp2016,bahdanau_icassp2016,glehre_2015OnUM,chorowski_is2017}." />
        </attvalues>
      </edge>
      <edge source="2104.04487" target="1508.04395" id="9006">
        <attvalues>
          <attvalue for="5" value=" Recent papers have proposed fusing E2E models with LMs trained with text data (usually referred to this as fusion), including shallow fusion~\cite{glehre_2015OnUM,kannan_icassp2018}, deep fusion~\cite{glehre_2015OnUM}, cold fusion~\cite{sriram_archiv2015}, component fusion~\cite{shan_icassp2019}, etc Most experiments used neural LMs, and some used n-gram fst LMs~\cite{chan_las_icassp2016,bahdanau_icassp2016,glehre_2015OnUM,chorowski_is2017}." />
        </attvalues>
      </edge>
      <edge source="2104.04487" target="1612.02695" id="9007">
        <attvalues>
          <attvalue for="5" value=" Recent papers have proposed fusing E2E models with LMs trained with text data (usually referred to this as fusion), including shallow fusion~\cite{glehre_2015OnUM,kannan_icassp2018}, deep fusion~\cite{glehre_2015OnUM}, cold fusion~\cite{sriram_archiv2015}, component fusion~\cite{shan_icassp2019}, etc Most experiments used neural LMs, and some used n-gram fst LMs~\cite{chan_las_icassp2016,bahdanau_icassp2016,glehre_2015OnUM,chorowski_is2017}." />
        </attvalues>
      </edge>
      <edge source="2104.04487" target="1807.10857" id="9008">
        <attvalues>
          <attvalue for="5" value=" See~\cite{Toshniwal2018ACO} for comparison of some of these approaches." />
        </attvalues>
      </edge>
      <edge source="2110.07982" target="1908.04743" id="9009">
        <attvalues>
          <attvalue for="5" value=" The authors of IMS-Speech \cite{IMSDE} trained a German STT model, which so far had the best results on the German Tuda dataset \cite{TUDA}." />
        </attvalues>
      </edge>
      <edge source="2110.07982" target="2101.00390" id="9010">
        <attvalues>
          <attvalue for="5" value=" In VoxPopuli \cite{VOXPOP}, an approach for training multilingual models using a large unlabeled dataset is investigated." />
        </attvalues>
      </edge>
      <edge source="2110.07982" target="1912.06670" id="9011">
        <attvalues>
          <attvalue for="5" value="0}{\percent} in German\,$/$\,Spanish\,$/$\,French on the CommonVoice datasets~\cite{COMV}, which so far have been the best results on these datasets." />
        </attvalues>
      </edge>
      <edge source="2110.07982" target="2101.06856" id="9012">
        <attvalues>
          <attvalue for="5" value=" \cite{TINTRA} trained a very small model on a large in-house Chinese dataset which can run faster than real-time on an ARMv7 chip." />
        </attvalues>
      </edge>
      <edge source="2110.07982" target="1811.06621" id="9013">
        <attvalues>
          <attvalue for="5" value=" \cite{STRSRM} did train an English model on a very large in-house dataset which can run twice as fast than real-time on a Google-Pixel smartphone." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="cs/0010012" id="9014">
        <attvalues>
          <attvalue for="5" value=" For hidden Markov model (HMM)-based systems, word posterior probabilities from lattices or confusion networks can provide reasonably good estimates of confidence scores~\cite{Evermann2000PosteriorPD,Mangu2000FindingCI}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="1810.13025" id="9015">
        <attvalues>
          <attvalue for="5" value=" In order to have more reliable estimates, many model-based approaches have been proposed~\cite{Seigel2011CombiningIS,Kalgaonkar2015EstimatingCS,DelAgua2018SpeakerAdaptedCM,Ragni2018ConfidenceEA,Li2019BidirectionalLR}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="1810.13024" id="9016">
        <attvalues>
          <attvalue for="5" value=" In order to have more reliable estimates, many model-based approaches have been proposed~\cite{Seigel2011CombiningIS,Kalgaonkar2015EstimatingCS,DelAgua2018SpeakerAdaptedCM,Ragni2018ConfidenceEA,Li2019BidirectionalLR}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="2011.10798" id="9017">
        <attvalues>
          <attvalue for="5" value=" More recently, as end-to-end speech recognition systems reach competitive performance with a simplified pipeline~\cite{Li2021ABA,Tuske2021OnTL,Li2020DevelopingRM}, confidence estimation for end-to-end models has become a much-needed component of an ASR system." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="2105.00982" id="9018">
        <attvalues>
          <attvalue for="5" value=" More recently, as end-to-end speech recognition systems reach competitive performance with a simplified pipeline~\cite{Li2021ABA,Tuske2021OnTL,Li2020DevelopingRM}, confidence estimation for end-to-end models has become a much-needed component of an ASR system." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="2007.15188" id="9019">
        <attvalues>
          <attvalue for="5" value=" More recently, as end-to-end speech recognition systems reach competitive performance with a simplified pipeline~\cite{Li2021ABA,Tuske2021OnTL,Li2020DevelopingRM}, confidence estimation for end-to-end models has become a much-needed component of an ASR system." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="1211.3711" id="9020">
        <attvalues>
          <attvalue for="5" value=" Because popular end-to-end systems such as recurrent neural network transducers~\cite{Graves2012SequenceTW} and attention-based encoder-decoder models~\cite{Chorowski2015AttentionBasedMF} have auto-regressive decoders that depend on the full history, it is not straightforward to generate lattice-like representations for a large number of hypotheses, from which word posteriors can be derived~\cite{Oneata2021AnEO}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="1506.07503" id="9021">
        <attvalues>
          <attvalue for="5" value=" Because popular end-to-end systems such as recurrent neural network transducers~\cite{Graves2012SequenceTW} and attention-based encoder-decoder models~\cite{Chorowski2015AttentionBasedMF} have auto-regressive decoders that depend on the full history, it is not straightforward to generate lattice-like representations for a large number of hypotheses, from which word posteriors can be derived~\cite{Oneata2021AnEO}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="2101.05525" id="9022">
        <attvalues>
          <attvalue for="5" value=" Because popular end-to-end systems such as recurrent neural network transducers~\cite{Graves2012SequenceTW} and attention-based encoder-decoder models~\cite{Chorowski2015AttentionBasedMF} have auto-regressive decoders that depend on the full history, it is not straightforward to generate lattice-like representations for a large number of hypotheses, from which word posteriors can be derived~\cite{Oneata2021AnEO}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="2010.11428" id="9023">
        <attvalues>
          <attvalue for="5" value=" For example, multi-layer perceptrons (MLPs), recurrent neural networks (RNNs) and self-attention networks have been used to learn token-level~\cite{Woodward2020ConfidenceMI,Li2021ConfidenceEF}, word-level~\cite{Qiu2021LearningWC}, and utterance-level confidence scores~\cite{Kumar2020UtteranceCM,Li2021ResidualEM,Liu2021UtterancelevelNC}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="2103.06716" id="9024">
        <attvalues>
          <attvalue for="5" value=" For example, multi-layer perceptrons (MLPs), recurrent neural networks (RNNs) and self-attention networks have been used to learn token-level~\cite{Woodward2020ConfidenceMI,Li2021ConfidenceEF}, word-level~\cite{Qiu2021LearningWC}, and utterance-level confidence scores~\cite{Kumar2020UtteranceCM,Li2021ResidualEM,Liu2021UtterancelevelNC}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="2109.07750" id="9026">
        <attvalues>
          <attvalue for="5" value=" For example, multi-layer perceptrons (MLPs), recurrent neural networks (RNNs) and self-attention networks have been used to learn token-level~\cite{Woodward2020ConfidenceMI,Li2021ConfidenceEF}, word-level~\cite{Qiu2021LearningWC}, and utterance-level confidence scores~\cite{Kumar2020UtteranceCM,Li2021ResidualEM,Liu2021UtterancelevelNC}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="2104.12870" id="9027">
        <attvalues>
          <attvalue for="5" value=" With effective confidence estimation, simply rescoring $n$-best hypotheses with confidence scores can directly improve the ASR performance~\cite{Li2021ResidualEM,Qiu2021MultiTaskLF}." />
        </attvalues>
      </edge>
      <edge source="2110.03327" target="1906.02530" id="9028">
        <attvalues>
          <attvalue for="5" value=" Therefore, it is questionable whether the confidence estimation on out-of-domain (OOD) data will be reliable~\cite{Ovadia2019CanYT}." />
        </attvalues>
      </edge>
      <edge source="2105.00573" target="1409.3215" id="9052">
        <attvalues>
          <attvalue for="5" value=" Similarly, many sequence-to-sequence tasks that convert one sequence into another \cite{sutskever2014sequence} can be decomposed to simpler sequence sub-tasks in order to reduce the overall complexity." />
        </attvalues>
      </edge>
      <edge source="2105.00573" target="1409.0473" id="9053">
        <attvalues>
          <attvalue for="5" value=" End-to-end sequence models like encoder-decoder models \cite{bahdanau2014neural, vaswani2017attention}, are attractive in part due to their simplistic design and the reduced need for hand-crafted features." />
        </attvalues>
      </edge>
      <edge source="2105.00573" target="2004.10234" id="9054">
        <attvalues>
          <attvalue for="5" value=" However, studies have shown mixed results compared to cascaded models particularly for complex sequence tasks like speech translation \cite{inaguma-etal-2020-espnet-st} and spoken language understanding \cite{coucke2018snips}." />
        </attvalues>
      </edge>
      <edge source="2105.00573" target="1805.10190" id="9055">
        <attvalues>
          <attvalue for="5" value=" However, studies have shown mixed results compared to cascaded models particularly for complex sequence tasks like speech translation \cite{inaguma-etal-2020-espnet-st} and spoken language understanding \cite{coucke2018snips}." />
        </attvalues>
      </edge>
      <edge source="2308.13116" target="2108.08877" id="9056">
        <attvalues>
          <attvalue for="5" value=" Current state-of-the-art sentence embedding models, however, are trained on modern, high-resource languages such as English and use massive datasets consisting of billions of sentence pairs \cite{ni-etal-2022-sentence}." />
        </attvalues>
      </edge>
      <edge source="2308.13116" target="2008.12014" id="9057">
        <attvalues>
          <attvalue for="5" value=" \cite{singh-etal-2021-pilot} fine-tuned a Modern Greek BERT model \cite{koutsikakisGREEKBERTGreeksVisiting2020} on Ancient Greek text for PoS tagging, morphological tagging, and lemmatization tasks." />
        </attvalues>
      </edge>
      <edge source="2308.13116" target="2211.05673" id="9058">
        <attvalues>
          <attvalue for="5" value=" \cite{yamshchikov-etal-2022-bert} trained a BERT model for authorship classification of Pseudo-Plutarch texts." />
        </attvalues>
      </edge>
      <edge source="2308.13116" target="2305.01099" id="9059">
        <attvalues>
          <attvalue for="5" value=" \cite{cowen-breenLogionMachineLearning2023} trained another BERT model for the purpose of identifying errors in scribal transmission." />
        </attvalues>
      </edge>
      <edge source="2308.13116" target="1908.10084" id="9061">
        <attvalues>
          <attvalue for="5" value=" The best approaches for high-resource languages involve large human-annotated datasets, such as the natural language inference (NLI) datasets used by Sentence-BERT \cite{reimers-gurevych-2019-sentence}." />
        </attvalues>
      </edge>
      <edge source="2308.13116" target="2004.09813" id="9062">
        <attvalues>
          <attvalue for="5" value=" Following \cite{reimers-gurevych-2020-making}, we use multilingual knowledge distillation to train sentence embedding models with an aligned vector space for Ancient Greek and English." />
        </attvalues>
      </edge>
      <edge source="2305.11791" target="cs/0306050" id="9063">
        <attvalues>
          <attvalue for="5" value=" Named entity recognition (NER)~\cite{tjong2003introduction, doddington-etal-2004-automatic-2} has been a long-standing and one of the most important fundamental tasks in natural language processing (NLP)." />
        </attvalues>
      </edge>
      <edge source="2305.11791" target="1603.01360" id="9064">
        <attvalues>
          <attvalue for="5" value=" Existing NER models can be divided into three different categories, including sequence labeling methods~\cite{lample2016neural,devlin2019bert}, span-level classification~\cite{wang-lu-2020-two-2, zhong-chen-2021-frustratingly-2} and generation-based methods~\cite{yan-etal-2021-unified-generative-2, lu-etal-2022-unified-2}." />
        </attvalues>
      </edge>
      <edge source="2305.11791" target="2010.03851" id="9066">
        <attvalues>
          <attvalue for="5" value=" Existing NER models can be divided into three different categories, including sequence labeling methods~\cite{lample2016neural,devlin2019bert}, span-level classification~\cite{wang-lu-2020-two-2, zhong-chen-2021-frustratingly-2} and generation-based methods~\cite{yan-etal-2021-unified-generative-2, lu-etal-2022-unified-2}." />
        </attvalues>
      </edge>
      <edge source="2305.11791" target="2106.01223" id="9067">
        <attvalues>
          <attvalue for="5" value=" On the other hand, generation-based models~\cite{yan-etal-2021-unified-generative-2, Paolini2021StructuredPA} can overcome this limitation by leveraging generative PLMs and introducing a unified tagging strategy." />
        </attvalues>
      </edge>
      <edge source="2305.11791" target="2010.01677" id="9068">
        <attvalues>
          <attvalue for="5" value=" By introducing more samples in the training stage, data augmentation (DA) methods have been proven to be effective solutions in low-resource settings, including few-shot NER~\cite{chen-etal-2020-local-2, zhou-etal-2022-melm-2, chen-etal-2021-data-2}." />
        </attvalues>
      </edge>
      <edge source="2305.11791" target="1910.13461" id="9069">
        <attvalues>
          <attvalue for="5" value=" However, few efforts are made on DA over generative PLMs (eg BART~\cite{lewis-etal-2020-bart-2}, T5~\cite{Raffel2019ExploringTL}), so limited data resources will lead to weakly fine-tuning of these generation-based methods." />
        </attvalues>
      </edge>
      <edge source="2305.11791" target="2105.08901" id="9071">
        <attvalues>
          <attvalue for="5" value=" \cite{Tan2021ASN} proposed a sequence-to-set network and relied mainly on non-autoregressive generation~\cite{gu2018nonautoregressive}." />
        </attvalues>
      </edge>
      <edge source="2305.11791" target="1711.02281" id="9072">
        <attvalues>
          <attvalue for="5" value=" However, a simple mixture of these target sequences can confuse the model since there will be several ``gold'' target sequences corresponding to a same source sequence, which will result in a one-to-many mapping problem (also known as multimodality problem~\cite{gu2018nonautoregressive}), especially harmful in few-shot NER settings." />
        </attvalues>
      </edge>
      <edge source="2409.05997" target="1910.03771" id="9073">
        <attvalues>
          <attvalue for="5" value=" Our library is built to rely solely on PyTorch~\cite{pytorch2019} and HuggingFace~\cite{wolf-huggingface} ecosystems and can be integrated as a fast model selection step in a larger NLP pipeline." />
        </attvalues>
      </edge>
      <edge source="2409.05997" target="2212.10082" id="9074">
        <attvalues>
          <attvalue for="5" value=" Prominent examples of such methods include H-score \cite{baoInformationTheoreticApproachTransferability2019,hscore2023} and LogME \cite{logme2021}." />
        </attvalues>
      </edge>
      <edge source="2409.05997" target="2102.11005" id="9075">
        <attvalues>
          <attvalue for="5" value=" Prominent examples of such methods include H-score \cite{baoInformationTheoreticApproachTransferability2019,hscore2023} and LogME \cite{logme2021}." />
        </attvalues>
      </edge>
      <edge source="2409.05997" target="cs/0306050" id="9077">
        <attvalues>
          <attvalue for="5" value=" \end{itemize} TransformerRanker supports NLP classification tasks of two main families: (1) Text classification tasks such as question classification~\cite{voorhees2000overview}, sentiment analysis~\cite{socher2013recursive} or textual entailment~\cite{wang2018glue} in which a classification decision is made for an entire text (or text pair), (2) sequence labeling tasks such as named entity recognition~\cite[NER;][]{sang-conll} and part-of-speech tagging~\cite{petrov2011universal} where classification decisions are made per-word." />
        </attvalues>
      </edge>
      <edge source="2409.05997" target="1104.2086" id="9078">
        <attvalues>
          <attvalue for="5" value=" \end{itemize} TransformerRanker supports NLP classification tasks of two main families: (1) Text classification tasks such as question classification~\cite{voorhees2000overview}, sentiment analysis~\cite{socher2013recursive} or textual entailment~\cite{wang2018glue} in which a classification decision is made for an entire text (or text pair), (2) sequence labeling tasks such as named entity recognition~\cite[NER;][]{sang-conll} and part-of-speech tagging~\cite{petrov2011universal} where classification decisions are made per-word." />
        </attvalues>
      </edge>
      <edge source="2409.05997" target="1912.01703" id="9079">
        <attvalues>
          <attvalue for="5" value=" Our library is built to rely solely on PyTorch~\cite{pytorch2019} and HuggingFace~\cite{wolf-huggingface} ecosystems and can be integrated as a fast model selection step in a larger NLP pipeline." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="1706.03762" id="9080">
        <attvalues>
          <attvalue for="5" value=" Transformers have been dominant in many sequence generation tasks, outperforming their recurrent neural network (RNN) counterparts in terms of both accuracy and speed for end-to-end systems\cite{vaswani2017attention, li2020comparison, karita2019comparative}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="1711.02281" id="9081">
        <attvalues>
          <attvalue for="5" value=" The idea is widely adopted in neural machine translation (NMT)\cite{gu2018non, lee2020deterministic, saharia2020non}, automatic speech recognition (ASR)\cite{chen2020non, chan2020imputer, bai2020listen, tian2020spike, higuchi2020mask, fujita2020insertion, fan2021cass, chi2020align, song2021non, higuchi2021improved, fujita2020end, bai2021fast}, text-to-speech (TTS)\cite{peng2020non, miao2020flow} and speech translation \cite{inaguma2021orthros}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="1905.08459" id="9082">
        <attvalues>
          <attvalue for="5" value=" The idea is widely adopted in neural machine translation (NMT)\cite{gu2018non, lee2020deterministic, saharia2020non}, automatic speech recognition (ASR)\cite{chen2020non, chan2020imputer, bai2020listen, tian2020spike, higuchi2020mask, fujita2020insertion, fan2021cass, chi2020align, song2021non, higuchi2021improved, fujita2020end, bai2021fast}, text-to-speech (TTS)\cite{peng2020non, miao2020flow} and speech translation \cite{inaguma2021orthros}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="2010.13047" id="9083">
        <attvalues>
          <attvalue for="5" value=" The idea is widely adopted in neural machine translation (NMT)\cite{gu2018non, lee2020deterministic, saharia2020non}, automatic speech recognition (ASR)\cite{chen2020non, chan2020imputer, bai2020listen, tian2020spike, higuchi2020mask, fujita2020insertion, fan2021cass, chi2020align, song2021non, higuchi2021improved, fujita2020end, bai2021fast}, text-to-speech (TTS)\cite{peng2020non, miao2020flow} and speech translation \cite{inaguma2021orthros}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="2005.08700" id="9084">
        <attvalues>
          <attvalue for="5" value=" applied the same idea but based on the connectionist temporal classification (CTC) output \cite{higuchi2020mask,higuchi2021improved}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="2010.13270" id="9085">
        <attvalues>
          <attvalue for="5" value=" applied the same idea but based on the connectionist temporal classification (CTC) output \cite{higuchi2020mask,higuchi2021improved}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="2005.13211" id="9086">
        <attvalues>
          <attvalue for="5" value=" used the idea of the insertion transformer from NMT to generate the output sequence with an arbitrary order \cite{fujita2020insertion}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="2010.14233" id="9087">
        <attvalues>
          <attvalue for="5" value=" Another recent effective method is using multiple decoders as refiners to do an iterative refinement based on CTC alignments \cite{chi2020align}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="2005.04862" id="9088">
        <attvalues>
          <attvalue for="5" value=" The idea is to substitute the word embedding in autoregressive models with an acoustic representation for each output token, assuming that language semantics can also be captured by acoustic representations \cite{bai2020listen, tian2020spike, fan2021cass}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="2005.08100" id="9090">
        <attvalues>
          <attvalue for="5" value=" First, convolution augmented self-attention blocks are applied to both the encoder and decoder modules, while other work only considered using them in the encoder\cite{gulati2020conformer}." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="1910.10324" id="9091">
        <attvalues>
          <attvalue for="5" value=" Third, considering the wide use of iterated loss functions to train deep transformers\cite{tjandra2020deja, wang2020transformer, lee2021intermediate}, we apply iterated loss to enhance the gradient update of low-layer parameters for both the encoder and decoder modules." />
        </attvalues>
      </edge>
      <edge source="2106.09885" target="1709.05522" id="9092">
        <attvalues>
          <attvalue for="5" value=" When no external language model is used, large improvements are observed on both the Librispeech\cite{panayotov2015librispeech} and Aishell1\cite{bu2017aishell} datasets in terms of error rate, and the performance is close to the autoregressive baseline." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="1805.07043" id="9093">
        <attvalues>
          <attvalue for="5" value=" Text classification, as an extensively applied fundamental cornerstone for natural language processing (NLP) applications, such as sentiment analysis~\cite{xue2018aspect}, spam detection~\cite{kennedy2019fact} and spoken dialogue systems~\cite{lowe2016evaluation,gupta2019simple}, has been widely studied for decades." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="2010.15296" id="9094">
        <attvalues>
          <attvalue for="5" value=" Text classification, as an extensively applied fundamental cornerstone for natural language processing (NLP) applications, such as sentiment analysis~\cite{xue2018aspect}, spam detection~\cite{kennedy2019fact} and spoken dialogue systems~\cite{lowe2016evaluation,gupta2019simple}, has been widely studied for decades." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="1605.05414" id="9095">
        <attvalues>
          <attvalue for="5" value=" Text classification, as an extensively applied fundamental cornerstone for natural language processing (NLP) applications, such as sentiment analysis~\cite{xue2018aspect}, spam detection~\cite{kennedy2019fact} and spoken dialogue systems~\cite{lowe2016evaluation,gupta2019simple}, has been widely studied for decades." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="1903.08268" id="9096">
        <attvalues>
          <attvalue for="5" value=" Text classification, as an extensively applied fundamental cornerstone for natural language processing (NLP) applications, such as sentiment analysis~\cite{xue2018aspect}, spam detection~\cite{kennedy2019fact} and spoken dialogue systems~\cite{lowe2016evaluation,gupta2019simple}, has been widely studied for decades." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="1408.5882" id="9098">
        <attvalues>
          <attvalue for="5" value=" Much of the work concentrated on learning the composition of distributional word representations~\cite{mikolov2013efficient, pennington2014glove, bojanowski2017enriching} for categorization, wherein plenty of deep learning methods have been adopted, such as TextCNNs~\cite{kim2014convolutional}, RCNNs~\cite{lai2015recurrent}, recurrent neural networks (RNNs)~\cite{liu2016recurrent}, FastText~\cite{joulin2016bag}, BERT~\cite{devlin2018bert}, etc Most of them learn the word representations by firstly projecting the one-hot encoding of each token through a pretrained or randomly initialized word embedding matrices to acquire the dense real-valued vectors, and then feed them into neural models for classification." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="1607.01759" id="9100">
        <attvalues>
          <attvalue for="5" value=" Much of the work concentrated on learning the composition of distributional word representations~\cite{mikolov2013efficient, pennington2014glove, bojanowski2017enriching} for categorization, wherein plenty of deep learning methods have been adopted, such as TextCNNs~\cite{kim2014convolutional}, RCNNs~\cite{lai2015recurrent}, recurrent neural networks (RNNs)~\cite{liu2016recurrent}, FastText~\cite{joulin2016bag}, BERT~\cite{devlin2018bert}, etc Most of them learn the word representations by firstly projecting the one-hot encoding of each token through a pretrained or randomly initialized word embedding matrices to acquire the dense real-valued vectors, and then feed them into neural models for classification." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="1809.03664" id="9102">
        <attvalues>
          <attvalue for="5" value=" Moreover, \cite{zeng2018topic} incorporated the neural topic models with Variational Autoencoder (VAE)~\cite{kingma2013auto} into the classification tasks so as to discover the latent topics in the document level and encode the co-occurrence of words with bag-of-words statistics." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="1312.6114" id="9103">
        <attvalues>
          <attvalue for="5" value=" Moreover, \cite{zeng2018topic} incorporated the neural topic models with Variational Autoencoder (VAE)~\cite{kingma2013auto} into the classification tasks so as to discover the latent topics in the document level and encode the co-occurrence of words with bag-of-words statistics." />
        </attvalues>
      </edge>
      <edge source="2011.12184" target="1310.4546" id="9104">
        <attvalues>
          <attvalue for="5" value=" Grounding on this, we design an ad hoc Clustering-Enchanced neural model (hereafter CluE) that jointly learns the distributional clustering and the alignment between the domain-aware clustering centroids and word representations in the Euclidean hidden semantic space for text classification, with the vector space assumption that words with similar meanings are close to each other~\cite{mikolov2013distributed}." />
        </attvalues>
      </edge>
      <edge source="2103.11431" target="1003.1141" id="9106">
        <attvalues>
          <attvalue for="5" value=" They have also proven effective in modeling cognitive operations such as the judgement of word similarity~\cite{turney2010frequency, baroni2010distributional}, and the brain activity elicited by specific concepts~\cite{mitchell2008predicting}." />
        </attvalues>
      </edge>
      <edge source="2103.11431" target="1506.02004" id="9107">
        <attvalues>
          <attvalue for="5" value=" In addition, the studies define the notion of interpretability in terms of the coherence of dimensions of the word embeddings~\cite{faruqui2015sparse, lipton2018mythos, subramanian2018spine}." />
        </attvalues>
      </edge>
      <edge source="2103.11431" target="1310.4546" id="9108">
        <attvalues>
          <attvalue for="5" value=" However, these studies are primarily focused on pre-trained word embeddings like GloVe~\cite{pennington2014glove} and word2vec~\cite{mikolov2013distributed}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2111.08201" id="9110">
        <attvalues>
          <attvalue for="5" value=" Speech summarization, whether extractive \cite{liu2015combining} or abstractive \cite{kano2020asru,kano2022icassp,shon2022slueted}, requires global acoustic context since knowledge of the entire speech signal is helpful for either extracting relevant key-frames or generating comprehensive abstractive summaries." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2212.10525" id="9111">
        <attvalues>
          <attvalue for="5" value=" Speech summarization, whether extractive \cite{liu2015combining} or abstractive \cite{kano2020asru,kano2022icassp,shon2022slueted}, requires global acoustic context since knowledge of the entire speech signal is helpful for either extracting relevant key-frames or generating comprehensive abstractive summaries." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2303.00978" id="9112">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end speech summarization models ~\cite{sharma2022end,matsuura2023} have been shown to outperform competitive cascade models that comprise speech recognition and text summarization modules." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2004.05150" id="9113">
        <attvalues>
          <attvalue for="5" value=" Prior work has proposed restricting the scope of attention using the Longformer ~\cite{sharma2022end,beltagy2020longformer} or linear self-attentions like the XNOR-former ~\cite{sharma2022xnor}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2005.08100" id="9114">
        <attvalues>
          <attvalue for="5" value=" For example, with a 6-layer conformer~\cite{conformer} encoder, a 32G V-100 GPU can take sequences of length 25,000; and with an XNOR-encoder, the same GPU can take ~45,000 frames." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2204.08920" id="9115">
        <attvalues>
          <attvalue for="5" value=" Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="1801.00841" id="9116">
        <attvalues>
          <attvalue for="5" value=" Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2001.02674" id="9117">
        <attvalues>
          <attvalue for="5" value=" Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="1910.11455" id="9118">
        <attvalues>
          <attvalue for="5" value=" Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2006.14941" id="9119">
        <attvalues>
          <attvalue for="5" value=" Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2010.10759" id="9120">
        <attvalues>
          <attvalue for="5" value=" Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2011.00033" id="9121">
        <attvalues>
          <attvalue for="5" value=" Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2102.04488" id="9122">
        <attvalues>
          <attvalue for="5" value=" Most prior work has been focused on the former to enable streaming inference for tasks like speech recognition ~\cite{rao2017exploring,moritz2020streaming,narayanan2019streaming,tsunoo2021slt,shi2021icassp}, speech translation ~\cite{ma2021translation}, spoken intent detection ~\cite{deng22b_interspeech}, and wake word detection ~\cite{wang2021wake}." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="1808.02171" id="9123">
        <attvalues>
          <attvalue for="5" value=" Prior work in speech recognition for long conversations ~\cite{KimMetze18,hori2021advanced, HoriMHR20} can be considered examples of such block-wise models, since they produce an utterance transcription for every new block of input." />
        </attvalues>
      </edge>
      <edge source="2307.08217" target="2104.09426" id="9124">
        <attvalues>
          <attvalue for="5" value=" Prior work in speech recognition for long conversations ~\cite{KimMetze18,hori2021advanced, HoriMHR20} can be considered examples of such block-wise models, since they produce an utterance transcription for every new block of input." />
        </attvalues>
      </edge>
      <edge source="2202.12191" target="1901.04085" id="9139">
        <attvalues>
          <attvalue for="5" value=" Since the advent of BERT~(Bidirectional Encoder Representations from Transformers~\cite{devlin2019bert}), a variety of BERT-based Neural Ranking Models~(NRMs) have been proposed~\cite{nogueira2019passage, khattab2020colbert}." />
        </attvalues>
      </edge>
      <edge source="2202.12191" target="2004.12832" id="9141">
        <attvalues>
          <attvalue for="5" value=" For the NRM's language model, we considered pre-trained BERT, fine-tuned ColBERT~\cite{khattab2020colbert}, and prefix-tuned ColBERT~\cite{jung2021semi}." />
        </attvalues>
      </edge>
      <edge source="2202.12191" target="2110.14943" id="9142">
        <attvalues>
          <attvalue for="5" value=" For the NRM's language model, we considered pre-trained BERT, fine-tuned ColBERT~\cite{khattab2020colbert}, and prefix-tuned ColBERT~\cite{jung2021semi}." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2310.07343" id="9144">
        <attvalues>
          <attvalue for="5" value=" Retrieval-augmented generation (RAG) \cite{guu2020retrieval, lewis2020retrieval, ram-etal-2023-context} that augments large language models (LLMs) with retrieval of relevant information has become increasingly popular in knowledge-intensive tasks, including open-domain question-answering (QA) \cite{zhang-etal-2023-large, kasai2023realtime, cui2023chatlaw,zhang-etal-2023-survey-efficient}." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2211.07886" id="9145">
        <attvalues>
          <attvalue for="5" value=" Retrieval-augmented generation (RAG) \cite{guu2020retrieval, lewis2020retrieval, ram-etal-2023-context} that augments large language models (LLMs) with retrieval of relevant information has become increasingly popular in knowledge-intensive tasks, including open-domain question-answering (QA) \cite{zhang-etal-2023-large, kasai2023realtime, cui2023chatlaw,zhang-etal-2023-survey-efficient}." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2312.10997" id="9146">
        <attvalues>
          <attvalue for="5" value=" However, standard RAG methods conduct retrieval indiscriminately, irrespective of the input query, which may result in suboptimal task performance and increased inference costs \cite{gao2024retrievalaugmented}." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2212.10511" id="9147">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{mallen-etal-2023-trust} heuristically retrieve when the popularity of an entity on Wikipedia is below a certain threshold; \cite{jiang-etal-2023-active} trigger retrieval if any token in the temporarily generated sentence has low confidence." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2302.00093" id="9148">
        <attvalues>
          <attvalue for="5" value=" On the other hand, the retrieved context may contain noise and irrelevant information, and augmenting noisy context can potentially distract LLMs, thereby impeding task performance \cite{pmlr-v202-shi23a}." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2305.06983" id="9150">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{mallen-etal-2023-trust} heuristically retrieve when the popularity of an entity on Wikipedia is below a certain threshold; \cite{jiang-etal-2023-active} trigger retrieval if any token in the temporarily generated sentence has low confidence." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2305.09955" id="9151">
        <attvalues>
          <attvalue for="5" value=" To obviate the hyperparameter threshold, model-based methods \cite{feng2023knowledge, ren2023investigating} directly prompt LLMs for retrieval decisions, given the observation that LLMs can acknowledge their knowledge boundaries to some extent \cite{kadavath2022language, yin-etal-2023-large}." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2207.05221" id="9152">
        <attvalues>
          <attvalue for="5" value=" To obviate the hyperparameter threshold, model-based methods \cite{feng2023knowledge, ren2023investigating} directly prompt LLMs for retrieval decisions, given the observation that LLMs can acknowledge their knowledge boundaries to some extent \cite{kadavath2022language, yin-etal-2023-large}." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="1606.05250" id="9153">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, commonly used open-domain QA datasets \cite{rajpurkar-etal-2016-squad, joshi-etal-2017-triviaqa, kwiatkowski-etal-2019-natural, mallen-etal-2023-trust} fail to fulfil this purpose, as various LLMs have distinct sizes and levels of pre-trained knowledge, making them inadequately assess the necessity of external retrieval for LLMs." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2309.01431" id="9154">
        <attvalues>
          <attvalue for="5" value=" RetrievalQA enables us to evaluate the effectiveness of ARAG approaches, an aspect predominantly overlooked in prior studies and recent RAG evaluation systems \cite{chen2023benchmarking, saadfalcon2023ares, es2023ragas}, which focus only on task performance, the relevance of retrieval context or the faithfulness of answers." />
        </attvalues>
      </edge>
      <edge source="2402.16457" target="2005.14165" id="9155">
        <attvalues>
          <attvalue for="5" value=" As an initial effort, we propose Time-Aware Adaptive REtrieval (TA-ARE), a simple yet effective method to improve ARAG via in-context learning (ICL; \cite{NEURIPS2020_1457c0d6}), obviating the need for calibration or additional training." />
        </attvalues>
      </edge>
      <edge source="2112.08718" target="1211.3711" id="9156">
        <attvalues>
          <attvalue for="5" value=" With recent advancements \cite{graves2012sequence, chan2016listen, zhang2020Transformer}, they have been deployed in a wide range of domains, including healthcare, travel reservations, and customer services etc A typical technique to improve the performance of these systems is to do a rescoring of the $n$-best hypotheses with an external Language Model (LM) \cite{chan2016listen}." />
        </attvalues>
      </edge>
      <edge source="2112.08718" target="1905.04226" id="9158">
        <attvalues>
          <attvalue for="5" value=" Recent pretrained Transformer-based LMs such as GPT-2 \cite{radford2019language} and BERT \cite{devlin2018bert} have been shown \cite{irie2019language} to be more effective than conventional LSTM based LMs for rescoring." />
        </attvalues>
      </edge>
      <edge source="2112.08718" target="2104.11070" id="9159">
        <attvalues>
          <attvalue for="5" value=" As showcased in \cite{shenoy21_interspeech, Shenoy_2021}, domain-specific data is useful for improving performance in a domain." />
        </attvalues>
      </edge>
      <edge source="2112.08718" target="2005.14165" id="9161">
        <attvalues>
          <attvalue for="5" value=" More recent models, such as GPT-3 \cite{brown2020language}, are able to solve new tasks with the help of just the textual descriptions of the task (called prompts)." />
        </attvalues>
      </edge>
      <edge source="2112.08718" target="2104.08691" id="9162">
        <attvalues>
          <attvalue for="5" value=" Drawing ideas from prompt-tuning \cite{lester2021power} for task adaptation, we introduce domain-prompts for our goal." />
        </attvalues>
      </edge>
      <edge source="2110.06502" target="2104.08691" id="9164">
        <attvalues>
          <attvalue for="5" value=" \cite{lester2021power} introduced this idea of learning the token embeddings of the prompt used to prime a LM to a particular task." />
        </attvalues>
      </edge>
      <edge source="2402.04925" target="2210.17323" id="9165">
        <attvalues>
          <attvalue for="5" value=" Strategies like GPTQ \cite{frantar_gptq_2023} and Tensor Parallel (TP) \cite{noauthor_tensor_nodate} are hence essential in achieving high-throughput performance." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2301.08653" id="9166">
        <attvalues>
          <attvalue for="5" value=" Although significant concerns regarding LLMs, particularly their tendency to ``hallucinate'' (or ``making things up'') and generation of biased or harmful content in scale have been raised~\cite{bender2021dangers, alkaissi2023artificial}, ChatGPT is becoming a common tool not only for everyday tasks such as essay writing, translation, and summarization~\cite{taecharungroj2023can,patel2023chatgpt}, but also for more sophisticated tasks such as code generation, debugging~\cite{sobania2023analysis}, and mathematical problem-solving~\cite{frieder2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2301.13867" id="9167">
        <attvalues>
          <attvalue for="5" value=" Although significant concerns regarding LLMs, particularly their tendency to ``hallucinate'' (or ``making things up'') and generation of biased or harmful content in scale have been raised~\cite{bender2021dangers, alkaissi2023artificial}, ChatGPT is becoming a common tool not only for everyday tasks such as essay writing, translation, and summarization~\cite{taecharungroj2023can,patel2023chatgpt}, but also for more sophisticated tasks such as code generation, debugging~\cite{sobania2023analysis}, and mathematical problem-solving~\cite{frieder2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2302.10724" id="9168">
        <attvalues>
          <attvalue for="5" value=" A recent study showed that although ChatGPT performs generally well in many tasks, it has different strengths and weaknesses for different tasks and does not tend to beat the SOTA models~\cite{kocon2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2202.12837" id="9169">
        <attvalues>
          <attvalue for="5" value=" Given that language models show competitive performance in classification tasks despite poorly labeled data~\cite{min2022rethinking,garg2022can}, we cannot discard the possibility that the RLHF pipeline might essentially be a weaker variant of type (1) contamination." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2005.14165" id="9170">
        <attvalues>
          <attvalue for="5" value=" Such contamination has been documented in the training data of other language models~\cite{brown2020language,dodge2021documenting,carlini2020extracting}." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2303.08774" id="9171">
        <attvalues>
          <attvalue for="5" value=" It has been shown that LLMs can also be significantly affected by data leakage, both by the leakage of labels and even by the leakage of dataset without labels~\cite{min2022rethinking,brown2020language,gpt4}." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="1707.06347" id="9172">
        <attvalues>
          <attvalue for="5" value=" The now-frozen RM is used as a reward function, and the LLM is further fine-tuned to maximize this reward using the Proximal Policy Optimization (PPO) algorithm~\cite{schulman2017proximal}." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2104.08758" id="9173">
        <attvalues>
          <attvalue for="5" value=" It is important to highlight a distinction between two kinds of contamination acknowledged in literature~\cite{dodge2021documenting}: (1) the case where both the task input and labels are leaked to the model via training versus (2) the case where just the input is exposed." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2012.07805" id="9174">
        <attvalues>
          <attvalue for="5" value=" Such contamination has been documented in the training data of other language models~\cite{brown2020language,dodge2021documenting,carlini2020extracting}." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2208.01066" id="9175">
        <attvalues>
          <attvalue for="5" value=" Given that language models show competitive performance in classification tasks despite poorly labeled data~\cite{min2022rethinking,garg2022can}, we cannot discard the possibility that the RLHF pipeline might essentially be a weaker variant of type (1) contamination." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="2212.14548" id="9176">
        <attvalues>
          <attvalue for="5" value=" As we will present below, we do see an overall improvement in the performance before and after the publication of the stance detection evaluation paper~\cite{zhang2022would}." />
        </attvalues>
      </edge>
      <edge source="2303.12767" target="1605.01655" id="9177">
        <attvalues>
          <attvalue for="5" value=" Recent studies have claimed that ChatGPT outperforms most of the previous models proposed for this task~\cite{zhang2022would} on a few existing evaluation datasets, such as the SemEval 2016 Task6 dataset~\cite{mohammad2016semeval,mohammad2017stance} and P-stance~\cite{li2021p}, even in a zero-shot setting where the model was not fine-tuned on the task-specific training data." />
        </attvalues>
      </edge>
      <edge source="2305.14538" target="1804.06609" id="9179">
        <attvalues>
          <attvalue for="5" value=" To mitigate this issue \cite{post2018fast} propose dynamic beam allocation, which reduces the computational overhead to a constant factor." />
        </attvalues>
      </edge>
      <edge source="2305.14538" target="1906.01105" id="9180">
        <attvalues>
          <attvalue for="5" value=" More recently, \cite{dinu2019training} have argued that GBS can be brittle under realistic conditions and propose instead to train a language model to copy target terminologies, after first adding these to the input." />
        </attvalues>
      </edge>
      <edge source="2006.07890" target="1309.4168" id="9181">
        <attvalues>
          <attvalue for="5" value=" Static pretrained word embeddings like word2vec \cite{mikolov2013exploiting} are recently replaced by dynamic, contextual embeddings, such as ELMo \cite{Peters2018} and BERT \cite{Devlin2018}." />
        </attvalues>
      </edge>
      <edge source="2006.07890" target="1802.05365" id="9182">
        <attvalues>
          <attvalue for="5" value=" Static pretrained word embeddings like word2vec \cite{mikolov2013exploiting} are recently replaced by dynamic, contextual embeddings, such as ELMo \cite{Peters2018} and BERT \cite{Devlin2018}." />
        </attvalues>
      </edge>
      <edge source="2006.07890" target="1911.02116" id="9184">
        <attvalues>
          <attvalue for="5" value=" However, two massively multilingual masked language models have been released: a multilingual BERT (mBERT) \cite{Devlin2018}, trained on 104 languages, and newer even larger XLM-RoBERTa (XLM-R) \cite{conneau2019unsupervised}, trained on 100 languages." />
        </attvalues>
      </edge>
      <edge source="2006.07890" target="1912.07076" id="9185">
        <attvalues>
          <attvalue for="5" value=" The main reasons for this choice are to better represent each language, and keep sensible sub-word vocabulary, as shown by \cite{virtanen2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2306.01768" target="1706.03762" id="9187">
        <attvalues>
          <attvalue for="5" value=" A Transformer model is a neural network that learns context and thus meaning by tracking relationships in sequential data like the words in natural language~\cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2306.01768" target="2009.06732" id="9189">
        <attvalues>
          <attvalue for="5" value=" The existing survey, in On Evaluation sectionon page 21, discussed a few NLP benchmarks such as GLUE, NaturalQuestions, and TriviaQA; and it argued that ``many research papers select their own benchmarks to showcase the abilities of the proposed model,'' leaving it ``a mystery to which fundamental efficient Transformer block one should consider using''~\cite{tay2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2202.09049" target="1910.13461" id="9200">
        <attvalues>
          <attvalue for="5" value=" &#09;Previous state-of-the-art Seq2Seq based methods, like GPT-2 \cite{Radford2019LanguageMA} and BART \cite{DBLP:conf/acl/LewisLGGMLSZ20}, mainly generate in a left-to-right manner and train with language model objective, which undergo an issue of generating inconsistent and safe endings." />
        </attvalues>
      </edge>
      <edge source="2202.09049" target="1808.10113" id="9201">
        <attvalues>
          <attvalue for="5" value=" To solve this issue, researches consider integrating semantic \cite{Huang2021StoryEG,Xu2020ControllableSG}, commonsense \cite{Li2018GeneratingRA,Guan2019StoryEG,Paul2021COINSDG}, sentiment/emotion \cite{Luo2019LearningTC,Brahman2020ModelingPE} or even multi-modal \cite{Huang2021IgSEGIS} knowledge into backbone models." />
        </attvalues>
      </edge>
      <edge source="2202.09049" target="2106.02497" id="9202">
        <attvalues>
          <attvalue for="5" value=" To solve this issue, researches consider integrating semantic \cite{Huang2021StoryEG,Xu2020ControllableSG}, commonsense \cite{Li2018GeneratingRA,Guan2019StoryEG,Paul2021COINSDG}, sentiment/emotion \cite{Luo2019LearningTC,Brahman2020ModelingPE} or even multi-modal \cite{Huang2021IgSEGIS} knowledge into backbone models." />
        </attvalues>
      </edge>
      <edge source="2202.09049" target="2010.06822" id="9203">
        <attvalues>
          <attvalue for="5" value=" To solve this issue, researches consider integrating semantic \cite{Huang2021StoryEG,Xu2020ControllableSG}, commonsense \cite{Li2018GeneratingRA,Guan2019StoryEG,Paul2021COINSDG}, sentiment/emotion \cite{Luo2019LearningTC,Brahman2020ModelingPE} or even multi-modal \cite{Huang2021IgSEGIS} knowledge into backbone models." />
        </attvalues>
      </edge>
      <edge source="2202.09049" target="2001.05139" id="9204">
        <attvalues>
          <attvalue for="5" value=" &#09;Particularly, \cite{Guan2020AKP} introduces a self-supervised task to distinguish true stories from auto-constructed fake stories to incorporate commonsense knowledge into GPT-2, so that the content of generated endings appears more coherent." />
        </attvalues>
      </edge>
      <edge source="2202.09049" target="1902.09229" id="9205">
        <attvalues>
          <attvalue for="5" value=" &#09;Based on the above analysis, we adopt the paradigm of contrastive learning \cite{Arora2019ATA} to introduce an approach for story ending generation which achieves the goal of modeling the consistency between story context and candidate endings." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2107.03374" id="9206">
        <attvalues>
          <attvalue for="5" value=" For example, OpenAI’s Codex \cite{chen2021evaluatingCodex}, released in 2021, has achieved a success rate of 28." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2308.12950" id="9207">
        <attvalues>
          <attvalue for="5" value=" Subsequently, various code LLMs emerged in both academia and industry, such as Incoder \cite{Fried2023Incoder}, StarCoder \cite{li2023starcoder}, CodeRL \cite{le2022coderl}, CodeGen \cite{Nijkamp2023codegen,nijkamp2023codegen2}, Code Llama \cite{roziere2023codellama}, ChatGPT \cite{chatgpt}, etc These models are capable of generating code with functional accuracy comparable to that of human developers." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2204.05999" id="9208">
        <attvalues>
          <attvalue for="5" value=" Subsequently, various code LLMs emerged in both academia and industry, such as Incoder \cite{Fried2023Incoder}, StarCoder \cite{li2023starcoder}, CodeRL \cite{le2022coderl}, CodeGen \cite{Nijkamp2023codegen,nijkamp2023codegen2}, Code Llama \cite{roziere2023codellama}, ChatGPT \cite{chatgpt}, etc These models are capable of generating code with functional accuracy comparable to that of human developers." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2305.06161" id="9209">
        <attvalues>
          <attvalue for="5" value=" Subsequently, various code LLMs emerged in both academia and industry, such as Incoder \cite{Fried2023Incoder}, StarCoder \cite{li2023starcoder}, CodeRL \cite{le2022coderl}, CodeGen \cite{Nijkamp2023codegen,nijkamp2023codegen2}, Code Llama \cite{roziere2023codellama}, ChatGPT \cite{chatgpt}, etc These models are capable of generating code with functional accuracy comparable to that of human developers." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2207.01780" id="9210">
        <attvalues>
          <attvalue for="5" value=" Subsequently, various code LLMs emerged in both academia and industry, such as Incoder \cite{Fried2023Incoder}, StarCoder \cite{li2023starcoder}, CodeRL \cite{le2022coderl}, CodeGen \cite{Nijkamp2023codegen,nijkamp2023codegen2}, Code Llama \cite{roziere2023codellama}, ChatGPT \cite{chatgpt}, etc These models are capable of generating code with functional accuracy comparable to that of human developers." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2203.13474" id="9211">
        <attvalues>
          <attvalue for="5" value=" Subsequently, various code LLMs emerged in both academia and industry, such as Incoder \cite{Fried2023Incoder}, StarCoder \cite{li2023starcoder}, CodeRL \cite{le2022coderl}, CodeGen \cite{Nijkamp2023codegen,nijkamp2023codegen2}, Code Llama \cite{roziere2023codellama}, ChatGPT \cite{chatgpt}, etc These models are capable of generating code with functional accuracy comparable to that of human developers." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2305.02309" id="9212">
        <attvalues>
          <attvalue for="5" value=" Subsequently, various code LLMs emerged in both academia and industry, such as Incoder \cite{Fried2023Incoder}, StarCoder \cite{li2023starcoder}, CodeRL \cite{le2022coderl}, CodeGen \cite{Nijkamp2023codegen,nijkamp2023codegen2}, Code Llama \cite{roziere2023codellama}, ChatGPT \cite{chatgpt}, etc These models are capable of generating code with functional accuracy comparable to that of human developers." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2309.01219" id="9213">
        <attvalues>
          <attvalue for="5" value=" Most existing work mainly focuses on investigating the hallucination for natural language generation (NLG) tasks, for instance, generative question answering \cite{li2021addressing}, abstractive summarization \cite{maynez2020faithfulness}, dialogue generation \cite{huang2020challenges}, etc The hallucinations are mainly divided into three categories: input-conflicting, context-conflicting, and fact-conflicting hallucinations \cite{zhang2023siren}." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2202.03629" id="9214">
        <attvalues>
          <attvalue for="5" value=" The hallucination issue poses a potential risk in deploying LLMs across various applications \cite{ji2023survey}." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="2005.00661" id="9215">
        <attvalues>
          <attvalue for="5" value=" Most existing work mainly focuses on investigating the hallucination for natural language generation (NLG) tasks, for instance, generative question answering \cite{li2021addressing}, abstractive summarization \cite{maynez2020faithfulness}, dialogue generation \cite{huang2020challenges}, etc The hallucinations are mainly divided into three categories: input-conflicting, context-conflicting, and fact-conflicting hallucinations \cite{zhang2023siren}." />
        </attvalues>
      </edge>
      <edge source="2404.00971" target="1905.05709" id="9216">
        <attvalues>
          <attvalue for="5" value=" Most existing work mainly focuses on investigating the hallucination for natural language generation (NLG) tasks, for instance, generative question answering \cite{li2021addressing}, abstractive summarization \cite{maynez2020faithfulness}, dialogue generation \cite{huang2020challenges}, etc The hallucinations are mainly divided into three categories: input-conflicting, context-conflicting, and fact-conflicting hallucinations \cite{zhang2023siren}." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="1612.01744" id="9217">
        <attvalues>
          <attvalue for="5" value=" To overcome these shortcomings, end-to-end (E2E) ST systems that directly convert speech signals into text without a separate ASR stage have been extensively studied (eg, \cite{berard2016listen,vila2018end,sperber2020speech,radford2022robust})." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="2004.06358" id="9218">
        <attvalues>
          <attvalue for="5" value=" To overcome these shortcomings, end-to-end (E2E) ST systems that directly convert speech signals into text without a separate ASR stage have been extensively studied (eg, \cite{berard2016listen,vila2018end,sperber2020speech,radford2022robust})." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="2204.05352" id="9220">
        <attvalues>
          <attvalue for="5" value=" Furthermore, E2E ST systems based on neural transducer~\cite{xue22d_interspeech,xue2022weakly,wang23oa_interspeech} have succeeded in achieving low-latency ST while maintaining higher accuracy compared to traditional, non-streaming cascaded systems." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="2211.02499" id="9221">
        <attvalues>
          <attvalue for="5" value=" Furthermore, E2E ST systems based on neural transducer~\cite{xue22d_interspeech,xue2022weakly,wang23oa_interspeech} have succeeded in achieving low-latency ST while maintaining higher accuracy compared to traditional, non-streaming cascaded systems." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="2101.09624" id="9222">
        <attvalues>
          <attvalue for="5" value=" This is known as the speaker diarization (SD) problem \cite{park2021review}, which has been long studied in the context of speaker-attributed ASR \cite{fiscus2007rich,watanabe2020chime}." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="2303.00747" id="9223">
        <attvalues>
          <attvalue for="5" value=" Note that, although a few techniques have recently been proposed for E2E ASR to estimate word-level time stamps \cite{radford2022robust,bain23_interspeech}, they assume a monotonic alignment between the input and output, which does not hold in the ST task." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="2110.07393" id="9224">
        <attvalues>
          <attvalue for="5" value=" This dataset, named DiariST-AliMeeting, is developed by translating the Mandarin Chinese meeting recordings, AliMeeting corpus~\cite{yu2022m2met,yu2022summary}, into English." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="2202.03647" id="9225">
        <attvalues>
          <attvalue for="5" value=" This dataset, named DiariST-AliMeeting, is developed by translating the Mandarin Chinese meeting recordings, AliMeeting corpus~\cite{yu2022m2met,yu2022summary}, into English." />
        </attvalues>
      </edge>
      <edge source="2309.08007" target="2202.00842" id="9226">
        <attvalues>
          <attvalue for="5" value=" We develop a streaming multi-talker ST and SD system, dubbed DiariST, by integrating the token-level serialized output training (t-SOT) \cite{kanda22arxiv} and t-vector \cite{kanda22b_interspeech}, both originally developed for speaker-attributed ASR, into the neural transducer-based ST system." />
        </attvalues>
      </edge>
      <edge source="2408.13678" target="2006.11477" id="9228">
        <attvalues>
          <attvalue for="5" value="0 \cite{Baevski2020wav2vec2A}, HuBERT \cite{Hsu2021HuBERTSS}, and WavLM \cite{WavLM}, to understand how they represent Mandarin lexical tone, English lexical stress, and English phrasal pitch accents." />
        </attvalues>
      </edge>
      <edge source="2408.13678" target="2106.07447" id="9229">
        <attvalues>
          <attvalue for="5" value="0 \cite{Baevski2020wav2vec2A}, HuBERT \cite{Hsu2021HuBERTSS}, and WavLM \cite{WavLM}, to understand how they represent Mandarin lexical tone, English lexical stress, and English phrasal pitch accents." />
        </attvalues>
      </edge>
      <edge source="2408.13678" target="2110.13900" id="9230">
        <attvalues>
          <attvalue for="5" value="0 \cite{Baevski2020wav2vec2A}, HuBERT \cite{Hsu2021HuBERTSS}, and WavLM \cite{WavLM}, to understand how they represent Mandarin lexical tone, English lexical stress, and English phrasal pitch accents." />
        </attvalues>
      </edge>
      <edge source="2408.13678" target="2306.06232" id="9231">
        <attvalues>
          <attvalue for="5" value=" \cite{Martin2023ProbingSS, Pasad2023WhatDS, Pasad2022ComparativeLA}." />
        </attvalues>
      </edge>
      <edge source="2408.13678" target="2107.04734" id="9232">
        <attvalues>
          <attvalue for="5" value=" Probing has been used to show that SSL models represent phone identity \cite{Martin2023ProbingSS, CormacEnglish2022DomainInformedPO, Ma2020ProbingAR, Pasad2023WhatDS, Pasad2022ComparativeLA, Pasad2021LayerWiseAO}, and semantic or syntactic features of words \cite{Pasad2021LayerWiseAO, Pasad2023WhatDS, Shah2021WhatAD}." />
        </attvalues>
      </edge>
      <edge source="2408.13678" target="2108.01122" id="9233">
        <attvalues>
          <attvalue for="5" value=" While we also know that suprasegmentals like prosody and tone are represented by SSL models \cite{Yuan2021AutomaticRO, Yang2023WhatCA, prosodyTasks, emotionProbing}, we know less about how these representations develop across layers, what contexts they draw on, and how language-general they are." />
        </attvalues>
      </edge>
      <edge source="2308.14731" target="1709.06182" id="9234">
        <attvalues>
          <attvalue for="5" value=" Code summaries form the backbone of much documentation for programmers, and the dream of automatic generation of these summaries has been described as a ``holy grail'' of SE research~\cite{allamanis2018survey, forward2002relevance, leclair2019neural}." />
        </attvalues>
      </edge>
      <edge source="2308.14731" target="2305.12865" id="9235">
        <attvalues>
          <attvalue for="5" value=" The language model in the most powerful products may be tens or hundreds of billions of parameters, and the data input often includes trillions of tokens, such as the entirety of public GitHub repositories, plus StackOverflow, Wikipedia, etc The effectiveness of these products has captured the public imagination and helped drive a new wave of research~\cite{sun2023automatic}." />
        </attvalues>
      </edge>
      <edge source="2308.14731" target="2305.08005" id="9236">
        <attvalues>
          <attvalue for="5" value=" This call is a loss of data custody and a non-starter for many institutions~\cite{derner2023beyond}." />
        </attvalues>
      </edge>
      <edge source="2308.14731" target="2305.08286" id="9237">
        <attvalues>
          <attvalue for="5" value=" As foundation models, we compare jam~\cite{su2023language} and starcoder~\cite{li2023starcoder}." />
        </attvalues>
      </edge>
      <edge source="2308.14731" target="2305.06161" id="9238">
        <attvalues>
          <attvalue for="5" value=" As foundation models, we compare jam~\cite{su2023language} and starcoder~\cite{li2023starcoder}." />
        </attvalues>
      </edge>
      <edge source="2308.14731" target="2305.02301" id="9240">
        <attvalues>
          <attvalue for="5" value=" Although there are some papers that have already formulated the code summarization as a fine-tuning problem such as~\cite{wang2021codet5, bender2021danger} and studied knowledge distillation for smaller models from larger models~\cite{hsieh2023distilling, yu2023large}, we thoroughly explore the data and model size for knowledge distillation on code summarization and conduct the human study to compare the language models generated summary and human reference with human experts." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2005.14165" id="9241">
        <attvalues>
          <attvalue for="5" value=" Large language models have recently revolutionized the field of NLP by showing excellent performance on a diverse set of downstream text-processing tasks often with little to no finetuning required on the downstream tasks~\cite{brown2020neurips,ouyang2022training,chen2021evaluating, bang2023multitask,qin2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2107.03374" id="9243">
        <attvalues>
          <attvalue for="5" value=" Large language models have recently revolutionized the field of NLP by showing excellent performance on a diverse set of downstream text-processing tasks often with little to no finetuning required on the downstream tasks~\cite{brown2020neurips,ouyang2022training,chen2021evaluating, bang2023multitask,qin2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2302.06476" id="9244">
        <attvalues>
          <attvalue for="5" value=" Large language models have recently revolutionized the field of NLP by showing excellent performance on a diverse set of downstream text-processing tasks often with little to no finetuning required on the downstream tasks~\cite{brown2020neurips,ouyang2022training,chen2021evaluating, bang2023multitask,qin2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2303.08774" id="9245">
        <attvalues>
          <attvalue for="5" value=" When trained with a large number of parameters in the order of billions (and even trillions~\cite{openai2023gpt4}) and similarly large quantities of text data, these models demonstrate an emergent ability to do in-context learning~\cite{xie2021explanation, min2022rethinking} and reasoning via chain-of-thought prompting~\cite{wei2023chainofthought}, which renders LLMs more accurate than dedicated smaller models trained on task-specific data." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2201.11903" id="9247">
        <attvalues>
          <attvalue for="5" value=" When trained with a large number of parameters in the order of billions (and even trillions~\cite{openai2023gpt4}) and similarly large quantities of text data, these models demonstrate an emergent ability to do in-context learning~\cite{xie2021explanation, min2022rethinking} and reasoning via chain-of-thought prompting~\cite{wei2023chainofthought}, which renders LLMs more accurate than dedicated smaller models trained on task-specific data." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2305.10790" id="9248">
        <attvalues>
          <attvalue for="5" value=" However, such architectures are mainly motivated towards the goal of having a single end-to-end model and the speech recognition capabilities of such multi-modal LLMs are limited~\cite{gong2023listen}." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2305.11000" id="9249">
        <attvalues>
          <attvalue for="5" value=" One set of approaches relies on multi-modal LLMs, which ingest the audio modality by processing the underlying speech using an encoder network and feeding the LLM with speech embeddings~\cite{gong2023listen, fathullah2023prompting,zhang2023speechgpt,deshmukh2023pengi}." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2305.11834" id="9250">
        <attvalues>
          <attvalue for="5" value=" One set of approaches relies on multi-modal LLMs, which ingest the audio modality by processing the underlying speech using an encoder network and feeding the LLM with speech embeddings~\cite{gong2023listen, fathullah2023prompting,zhang2023speechgpt,deshmukh2023pengi}." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2304.12995" id="9251">
        <attvalues>
          <attvalue for="5" value=" Another set of approaches interface LLMs with pretrained ASR models such that the ASR output is fed directly to the LLM as a prompt to tackle the downstream task~\cite{huang2023audiogpt,shen2023hugginggpt,he2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2305.13512" id="9253">
        <attvalues>
          <attvalue for="5" value=" A relevant work~\cite{he2023chatgpt} utilizes ASR 1-best outputs as well as oracle transcripts as prompts to the LLM for speech-intent classification with encouraging results." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="2307.04172" id="9254">
        <attvalues>
          <attvalue for="5" value=" Another work~\cite{ma2023generative} explores correcting ASR errors by prompting $n$-best lists to ChatGPT which results in improvements in ASR WER." />
        </attvalues>
      </edge>
      <edge source="2309.04842" target="1804.03209" id="9256">
        <attvalues>
          <attvalue for="5" value=" We use an in-house general-purpose English ASR system and Vicuna~\cite{vicuna2023}, an instruction-tuned LLaMA LLM~\cite{touvron2023llama} , in this paper, and we experiment on (i) a device-directed speech detection (DDSD) task for binary intent classification on whether an utterance is directed towards a voice assistant or not, and on (ii) a keyword spotting (KS) task on the Google Speech Commands (GSC)~\cite{warden2018speech} dataset, which we treat as a multi-class intent classification problem." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="1909.05952" id="9257">
        <attvalues>
          <attvalue for="5" value=" Since the introduction of end-to-end neural diarization (EEND)~\cite{fujita2019enda} and its extension to deal with arbitrary amounts of speakers~\cite{fujita2020neural,horiguchi2020end}, it has been established as a state-of-the-art alternative to the standard cascaded diarization systems based on different submodules, eg voice activity detection (VAD), uniform segmentation, speaker embeddings extraction, clustering and overlapped speech detection (OSD) with handling." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2006.01796" id="9258">
        <attvalues>
          <attvalue for="5" value=" Since the introduction of end-to-end neural diarization (EEND)~\cite{fujita2019enda} and its extension to deal with arbitrary amounts of speakers~\cite{fujita2020neural,horiguchi2020end}, it has been established as a state-of-the-art alternative to the standard cascaded diarization systems based on different submodules, eg voice activity detection (VAD), uniform segmentation, speaker embeddings extraction, clustering and overlapped speech detection (OSD) with handling." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2005.09921" id="9259">
        <attvalues>
          <attvalue for="5" value=" Since the introduction of end-to-end neural diarization (EEND)~\cite{fujita2019enda} and its extension to deal with arbitrary amounts of speakers~\cite{fujita2020neural,horiguchi2020end}, it has been established as a state-of-the-art alternative to the standard cascaded diarization systems based on different submodules, eg voice activity detection (VAD), uniform segmentation, speaker embeddings extraction, clustering and overlapped speech detection (OSD) with handling." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="1909.06247" id="9260">
        <attvalues>
          <attvalue for="5" value=" Some by using self-attention layers~\cite{fujita2019endb} or conformer layers~\cite{chieh2021end} instead of the original BLSTM layers for feature encoding; others have focused on more complex diarization scenarios such as its online fashion~\cite{han2021bw,xue2021onlineA} or when more than one microphone is available~\cite{horiguchi2021multi} or by improving the model iteratively using pseudo-labels~\cite{takashima2021end}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2106.07167" id="9261">
        <attvalues>
          <attvalue for="5" value=" Some by using self-attention layers~\cite{fujita2019endb} or conformer layers~\cite{chieh2021end} instead of the original BLSTM layers for feature encoding; others have focused on more complex diarization scenarios such as its online fashion~\cite{han2021bw,xue2021onlineA} or when more than one microphone is available~\cite{horiguchi2021multi} or by improving the model iteratively using pseudo-labels~\cite{takashima2021end}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2011.02678" id="9262">
        <attvalues>
          <attvalue for="5" value=" Some by using self-attention layers~\cite{fujita2019endb} or conformer layers~\cite{chieh2021end} instead of the original BLSTM layers for feature encoding; others have focused on more complex diarization scenarios such as its online fashion~\cite{han2021bw,xue2021onlineA} or when more than one microphone is available~\cite{horiguchi2021multi} or by improving the model iteratively using pseudo-labels~\cite{takashima2021end}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2006.02616" id="9263">
        <attvalues>
          <attvalue for="5" value=" Some by using self-attention layers~\cite{fujita2019endb} or conformer layers~\cite{chieh2021end} instead of the original BLSTM layers for feature encoding; others have focused on more complex diarization scenarios such as its online fashion~\cite{han2021bw,xue2021onlineA} or when more than one microphone is available~\cite{horiguchi2021multi} or by improving the model iteratively using pseudo-labels~\cite{takashima2021end}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2110.04694" id="9264">
        <attvalues>
          <attvalue for="5" value=" Some by using self-attention layers~\cite{fujita2019endb} or conformer layers~\cite{chieh2021end} instead of the original BLSTM layers for feature encoding; others have focused on more complex diarization scenarios such as its online fashion~\cite{han2021bw,xue2021onlineA} or when more than one microphone is available~\cite{horiguchi2021multi} or by improving the model iteratively using pseudo-labels~\cite{takashima2021end}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2106.04078" id="9265">
        <attvalues>
          <attvalue for="5" value=" Some by using self-attention layers~\cite{fujita2019endb} or conformer layers~\cite{chieh2021end} instead of the original BLSTM layers for feature encoding; others have focused on more complex diarization scenarios such as its online fashion~\cite{han2021bw,xue2021onlineA} or when more than one microphone is available~\cite{horiguchi2021multi} or by improving the model iteratively using pseudo-labels~\cite{takashima2021end}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2012.10055" id="9266">
        <attvalues>
          <attvalue for="5" value=" Some have used EEND together with more standard approaches by using EEND-inspired models to find overlaps among pairs of speakers in the output of a cascaded system~\cite{horiguchi2021end} or leveraging EEND's VAD performance by using an external VAD system~\cite{horiguchi2021hitachi} or combining short duration diarization outputs to produce better whole-utterance diarization~\cite{kinoshita2021integrating,kinoshita2021advances,horiguchi2021towards,kinoshita2022tight}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2102.01363" id="9267">
        <attvalues>
          <attvalue for="5" value=" Some have used EEND together with more standard approaches by using EEND-inspired models to find overlaps among pairs of speakers in the output of a cascaded system~\cite{horiguchi2021end} or leveraging EEND's VAD performance by using an external VAD system~\cite{horiguchi2021hitachi} or combining short duration diarization outputs to produce better whole-utterance diarization~\cite{kinoshita2021integrating,kinoshita2021advances,horiguchi2021towards,kinoshita2022tight}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2010.13366" id="9268">
        <attvalues>
          <attvalue for="5" value=" Some have used EEND together with more standard approaches by using EEND-inspired models to find overlaps among pairs of speakers in the output of a cascaded system~\cite{horiguchi2021end} or leveraging EEND's VAD performance by using an external VAD system~\cite{horiguchi2021hitachi} or combining short duration diarization outputs to produce better whole-utterance diarization~\cite{kinoshita2021integrating,kinoshita2021advances,horiguchi2021towards,kinoshita2022tight}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2105.09040" id="9269">
        <attvalues>
          <attvalue for="5" value=" Some have used EEND together with more standard approaches by using EEND-inspired models to find overlaps among pairs of speakers in the output of a cascaded system~\cite{horiguchi2021end} or leveraging EEND's VAD performance by using an external VAD system~\cite{horiguchi2021hitachi} or combining short duration diarization outputs to produce better whole-utterance diarization~\cite{kinoshita2021integrating,kinoshita2021advances,horiguchi2021towards,kinoshita2022tight}." />
        </attvalues>
      </edge>
      <edge source="2204.00890" target="2107.01545" id="9270">
        <attvalues>
          <attvalue for="5" value=" Some have used EEND together with more standard approaches by using EEND-inspired models to find overlaps among pairs of speakers in the output of a cascaded system~\cite{horiguchi2021end} or leveraging EEND's VAD performance by using an external VAD system~\cite{horiguchi2021hitachi} or combining short duration diarization outputs to produce better whole-utterance diarization~\cite{kinoshita2021integrating,kinoshita2021advances,horiguchi2021towards,kinoshita2022tight}." />
        </attvalues>
      </edge>
      <edge source="2211.08203" target="1310.4546" id="9271">
        <attvalues>
          <attvalue for="5" value=" Static word embeddings have proven to encode semantic information of words and are therefore useful to solve tasks such as synonym selection and analogical reasoning \cite{mikolov2013distributed, levy2015improving}." />
        </attvalues>
      </edge>
      <edge source="2211.08203" target="1510.02675" id="9275">
        <attvalues>
          <attvalue for="5" value=" Previous research has found static word embeddings appear to be associated with word frequency in various ways: word frequency correlates with embedding norm \cite{wilson2015controlled, arora2016latent}, the nearest neighbors of the embeddings of medium-frequency English words are more unstable \cite{hellrich2016bad}, there are frequency-related differences in the distribution of the inner products between target and context vectors \cite{mimno2017strange}, embeddings can accurately predict whether a word is frequent or rare \cite{schnabel2015evaluation}, and the visual inspection of their top principal components suggest they encode frequency \cite{gong2018frage, mu2018allbutthetop}." />
        </attvalues>
      </edge>
      <edge source="2211.08203" target="1809.06858" id="9276">
        <attvalues>
          <attvalue for="5" value=" Previous research has found static word embeddings appear to be associated with word frequency in various ways: word frequency correlates with embedding norm \cite{wilson2015controlled, arora2016latent}, the nearest neighbors of the embeddings of medium-frequency English words are more unstable \cite{hellrich2016bad}, there are frequency-related differences in the distribution of the inner products between target and context vectors \cite{mimno2017strange}, embeddings can accurately predict whether a word is frequent or rare \cite{schnabel2015evaluation}, and the visual inspection of their top principal components suggest they encode frequency \cite{gong2018frage, mu2018allbutthetop}." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2307.09288" id="9278">
        <attvalues>
          <attvalue for="5" value=" RLHF, in particular, has been instrumental in expanding the application of both closed-source~\cite{OMS, claude_2024, team2023gemini} and open-source LLMs~\cite{touvron2023llama, yang2023baichuan}, driven by the need to align foundational models with human values and preferences~\cite{ziegler2019fine, stiennon2020learning, ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2303.08774" id="9279">
        <attvalues>
          <attvalue for="5" value=" Reward-model-based (RM-based) alignment pioneered by OpenAI~\cite{ouyang2022training,achiam2023gpt, touvron2023llama} first trains a Reward Model (RM) from user preferences, typically through Maximum Likelihood Estimation (MLE), and then leverages actor-critic algorithms such as Proximal Policy Optimization (PPO)~\cite{schulman2017proximal} to tune the SFT model to realize alignment." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="1909.08593" id="9280">
        <attvalues>
          <attvalue for="5" value=" RLHF, in particular, has been instrumental in expanding the application of both closed-source~\cite{OMS, claude_2024, team2023gemini} and open-source LLMs~\cite{touvron2023llama, yang2023baichuan}, driven by the need to align foundational models with human values and preferences~\cite{ziegler2019fine, stiennon2020learning, ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="1707.06347" id="9282">
        <attvalues>
          <attvalue for="5" value=" Reward-model-based (RM-based) alignment pioneered by OpenAI~\cite{ouyang2022training,achiam2023gpt, touvron2023llama} first trains a Reward Model (RM) from user preferences, typically through Maximum Likelihood Estimation (MLE), and then leverages actor-critic algorithms such as Proximal Policy Optimization (PPO)~\cite{schulman2017proximal} to tune the SFT model to realize alignment." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="1907.01752" id="9283">
        <attvalues>
          <attvalue for="5" value=" This approach often requires substantial computational resources and suffers from sample inefficiency~\cite{choshen2019weaknesses}." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2305.18290" id="9284">
        <attvalues>
          <attvalue for="5" value=" Conversely, another class of methods, known as reward-model-free (RM-free) alignment, such as Direct Preference Optimization (DPO)~\cite{rafailov2024direct}, Identity Preference Optimization (IPO)~\cite{azar2024general}, Sequence Likelihood Calibration (SLiC)~\cite{zhao2023slic}, DPO-positive~\cite{pal2024smaug} and Simple Preference Optimization (SimPO)~\cite{meng2024simpo}, do not rely on an extra RM." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2310.12036" id="9285">
        <attvalues>
          <attvalue for="5" value=" Conversely, another class of methods, known as reward-model-free (RM-free) alignment, such as Direct Preference Optimization (DPO)~\cite{rafailov2024direct}, Identity Preference Optimization (IPO)~\cite{azar2024general}, Sequence Likelihood Calibration (SLiC)~\cite{zhao2023slic}, DPO-positive~\cite{pal2024smaug} and Simple Preference Optimization (SimPO)~\cite{meng2024simpo}, do not rely on an extra RM." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2305.10425" id="9286">
        <attvalues>
          <attvalue for="5" value=" Conversely, another class of methods, known as reward-model-free (RM-free) alignment, such as Direct Preference Optimization (DPO)~\cite{rafailov2024direct}, Identity Preference Optimization (IPO)~\cite{azar2024general}, Sequence Likelihood Calibration (SLiC)~\cite{zhao2023slic}, DPO-positive~\cite{pal2024smaug} and Simple Preference Optimization (SimPO)~\cite{meng2024simpo}, do not rely on an extra RM." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2402.13228" id="9287">
        <attvalues>
          <attvalue for="5" value=" Conversely, another class of methods, known as reward-model-free (RM-free) alignment, such as Direct Preference Optimization (DPO)~\cite{rafailov2024direct}, Identity Preference Optimization (IPO)~\cite{azar2024general}, Sequence Likelihood Calibration (SLiC)~\cite{zhao2023slic}, DPO-positive~\cite{pal2024smaug} and Simple Preference Optimization (SimPO)~\cite{meng2024simpo}, do not rely on an extra RM." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2405.14734" id="9288">
        <attvalues>
          <attvalue for="5" value=" Conversely, another class of methods, known as reward-model-free (RM-free) alignment, such as Direct Preference Optimization (DPO)~\cite{rafailov2024direct}, Identity Preference Optimization (IPO)~\cite{azar2024general}, Sequence Likelihood Calibration (SLiC)~\cite{zhao2023slic}, DPO-positive~\cite{pal2024smaug} and Simple Preference Optimization (SimPO)~\cite{meng2024simpo}, do not rely on an extra RM." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2404.02078" id="9289">
        <attvalues>
          <attvalue for="5" value=" One notable counter-intuitive observation is that the likelihood of both preferred and rejected responses tends to decrease over the course of DPO training~\cite{yuan2024advancing, pytheia_MM}, while the likelihood of certain tokens diverging from the training data increases~\cite{xu2024dpo}." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2404.10719" id="9290">
        <attvalues>
          <attvalue for="5" value=" One notable counter-intuitive observation is that the likelihood of both preferred and rejected responses tends to decrease over the course of DPO training~\cite{yuan2024advancing, pytheia_MM}, while the likelihood of certain tokens diverging from the training data increases~\cite{xu2024dpo}." />
        </attvalues>
      </edge>
      <edge source="2406.07327" target="2405.08448" id="9291">
        <attvalues>
          <attvalue for="5" value=" Furthermore, our findings confirm that the distribution of preference data critically influences DPO’s effectiveness, with on-policy DPO performing better than off-policy DPO, which is consistent with concurrent empirical studies~\cite{tang2024understanding, guo2024direct}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1508.05326" id="9293">
        <attvalues>
          <attvalue for="5" value=" Recent sentence representation models like BERT~\cite{DBLP:conf/naacl/DevlinCLT19} achieved state-of-the-art results on sentence-pair regression/classification tasks, such as question answering, natural language inference~(NLI)~\cite{DBLP:conf/emnlp/BowmanAPM15,DBLP:conf/naacl/WilliamsNB18}, and semantic textual similarity~(STS)~\cite{DBLP:conf/semeval/AgirreCDG12,agirre-etal-2013-sem,agirre-etal-2016-semeval,agirre-etal-2014-semeval,agirre-etal-2015-semeval}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1704.05426" id="9294">
        <attvalues>
          <attvalue for="5" value=" Recent sentence representation models like BERT~\cite{DBLP:conf/naacl/DevlinCLT19} achieved state-of-the-art results on sentence-pair regression/classification tasks, such as question answering, natural language inference~(NLI)~\cite{DBLP:conf/emnlp/BowmanAPM15,DBLP:conf/naacl/WilliamsNB18}, and semantic textual similarity~(STS)~\cite{DBLP:conf/semeval/AgirreCDG12,agirre-etal-2013-sem,agirre-etal-2016-semeval,agirre-etal-2014-semeval,agirre-etal-2015-semeval}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1908.10084" id="9295">
        <attvalues>
          <attvalue for="5" value=" For instance, SBERT~\cite{DBLP:conf/emnlp/ReimersG19} using the siamese BERT-networks that decreased the performance by 3-4 points evaluated by Spearman correlation~\cite{myers2004spearman} on STS-Benchmark~\cite{cer-etal-2017-semeval}, which implies room for improvement." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1806.04330" id="9296">
        <attvalues>
          <attvalue for="5" value=" The feature has been proved vital for predicting matching degrees~\cite{DBLP:conf/coling/Lan018,xu2020symmetric}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1909.03405" id="9297">
        <attvalues>
          <attvalue for="5" value=" The feature has been proved vital for predicting matching degrees~\cite{DBLP:conf/coling/Lan018,xu2020symmetric}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1304.5634" id="9298">
        <attvalues>
          <attvalue for="5" value=" We take inspiration from Multi-view learning~\cite{DBLP:journals/corr/abs-1304-5634,DBLP:conf/emnlp/ClarkLML18} and train the sentence matching model from two views: (1) Siamese View, we start with the siamese BERT-networks as a backbone to derive sentence embeddings, to be able to capture semantics similarity efficiently by calculating distances on the two fixed-size vectors." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1809.08370" id="9299">
        <attvalues>
          <attvalue for="5" value=" We take inspiration from Multi-view learning~\cite{DBLP:journals/corr/abs-1304-5634,DBLP:conf/emnlp/ClarkLML18} and train the sentence matching model from two views: (1) Siamese View, we start with the siamese BERT-networks as a backbone to derive sentence embeddings, to be able to capture semantics similarity efficiently by calculating distances on the two fixed-size vectors." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1503.02531" id="9300">
        <attvalues>
          <attvalue for="5" value=" The association between the two views acts as a regularization term that trains a student with soft targets from the multiple teacher's output distributions, making the procedure similar to knowledge distillation~\cite{DBLP:journals/corr/HintonVD15}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1908.09355" id="9302">
        <attvalues>
          <attvalue for="5" value=" In contrast of other distilled versions of BERT~\cite{sanh2019distilbert,DBLP:conf/emnlp/SunCGL19}, our method aims to optimize sentence embedding representations with two heterogeneous networks, together with multi-task knowledge distillation~\cite{DBLP:journals/corr/abs-1904-09482}, neither distilling large models into a small model~\cite{DBLP:journals/corr/abs-1904-00796, DBLP:journals/corr/abs-1903-04190} nor born-again networks~\cite{DBLP:conf/icml/FurlanelloLTIA18,DBLP:conf/acl/ClarkLKML19}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1904.09482" id="9303">
        <attvalues>
          <attvalue for="5" value=" In contrast of other distilled versions of BERT~\cite{sanh2019distilbert,DBLP:conf/emnlp/SunCGL19}, our method aims to optimize sentence embedding representations with two heterogeneous networks, together with multi-task knowledge distillation~\cite{DBLP:journals/corr/abs-1904-09482}, neither distilling large models into a small model~\cite{DBLP:journals/corr/abs-1904-00796, DBLP:journals/corr/abs-1903-04190} nor born-again networks~\cite{DBLP:conf/icml/FurlanelloLTIA18,DBLP:conf/acl/ClarkLKML19}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1904.00796" id="9304">
        <attvalues>
          <attvalue for="5" value=" In contrast of other distilled versions of BERT~\cite{sanh2019distilbert,DBLP:conf/emnlp/SunCGL19}, our method aims to optimize sentence embedding representations with two heterogeneous networks, together with multi-task knowledge distillation~\cite{DBLP:journals/corr/abs-1904-09482}, neither distilling large models into a small model~\cite{DBLP:journals/corr/abs-1904-00796, DBLP:journals/corr/abs-1903-04190} nor born-again networks~\cite{DBLP:conf/icml/FurlanelloLTIA18,DBLP:conf/acl/ClarkLKML19}." />
        </attvalues>
      </edge>
      <edge source="2104.08675" target="1907.04829" id="9305">
        <attvalues>
          <attvalue for="5" value=" Besides, we compared the loss weighting and teacher annealing strategy~\cite{DBLP:conf/acl/ClarkLKML19} during the distillation process, suggesting that the latter was more efficient." />
        </attvalues>
      </edge>
      <edge source="2205.13674" target="1508.01211" id="9306">
        <attvalues>
          <attvalue for="5" value=" Several different models have been proposed over the years: cross-entropy (CE) models with a deep feed-forward architecture \cite{hinton2012deep}, connectionist temporal classification (CTC) models \cite{graves2006connectionist} with recurrent architectures such as long short-term memory (LSTM) \cite{hochreiter1997long}, and more recently sequence-to-sequence (Seq2Seq) models like listen, attend and spell (LAS) \cite{chan2015listen}, recurrent neural network transducer (RNN-T) \cite{graves2012sequence}, and hybrid autoregressive transducer (HAT) \cite{variani2020hybrid}." />
        </attvalues>
      </edge>
      <edge source="2205.13674" target="1211.3711" id="9307">
        <attvalues>
          <attvalue for="5" value=" Several different models have been proposed over the years: cross-entropy (CE) models with a deep feed-forward architecture \cite{hinton2012deep}, connectionist temporal classification (CTC) models \cite{graves2006connectionist} with recurrent architectures such as long short-term memory (LSTM) \cite{hochreiter1997long}, and more recently sequence-to-sequence (Seq2Seq) models like listen, attend and spell (LAS) \cite{chan2015listen}, recurrent neural network transducer (RNN-T) \cite{graves2012sequence}, and hybrid autoregressive transducer (HAT) \cite{variani2020hybrid}." />
        </attvalues>
      </edge>
      <edge source="2205.13674" target="2003.07705" id="9308">
        <attvalues>
          <attvalue for="5" value=" Several different models have been proposed over the years: cross-entropy (CE) models with a deep feed-forward architecture \cite{hinton2012deep}, connectionist temporal classification (CTC) models \cite{graves2006connectionist} with recurrent architectures such as long short-term memory (LSTM) \cite{hochreiter1997long}, and more recently sequence-to-sequence (Seq2Seq) models like listen, attend and spell (LAS) \cite{chan2015listen}, recurrent neural network transducer (RNN-T) \cite{graves2012sequence}, and hybrid autoregressive transducer (HAT) \cite{variani2020hybrid}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2005.14165" id="9310">
        <attvalues>
          <attvalue for="5" value=" In the past few years, researchers focused on the generation of each single modality and have achieved great progress: In terms of text generation, we have witnessed a qualitative leap in the performance of natural language processing tasks: From BERT~\cite{devlin2018bert}, GPT1~\cite{2018gpt1}, GPT2~\cite{2019gpt2}, GPT3~\cite{gpt3}, GPT4~\cite{gpt4} to ChatGPT~\cite{chatgpt}, LLaMA~\cite{li2023llama,gao2023llama}, the number of model parameters and training samples has grown rapidly, resulting in the continual growth of modal abilities and product deployment." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2303.08774" id="9311">
        <attvalues>
          <attvalue for="5" value=" In the past few years, researchers focused on the generation of each single modality and have achieved great progress: In terms of text generation, we have witnessed a qualitative leap in the performance of natural language processing tasks: From BERT~\cite{devlin2018bert}, GPT1~\cite{2018gpt1}, GPT2~\cite{2019gpt2}, GPT3~\cite{gpt3}, GPT4~\cite{gpt4} to ChatGPT~\cite{chatgpt}, LLaMA~\cite{li2023llama,gao2023llama}, the number of model parameters and training samples has grown rapidly, resulting in the continual growth of modal abilities and product deployment." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2311.17043" id="9312">
        <attvalues>
          <attvalue for="5" value=" In the past few years, researchers focused on the generation of each single modality and have achieved great progress: In terms of text generation, we have witnessed a qualitative leap in the performance of natural language processing tasks: From BERT~\cite{devlin2018bert}, GPT1~\cite{2018gpt1}, GPT2~\cite{2019gpt2}, GPT3~\cite{gpt3}, GPT4~\cite{gpt4} to ChatGPT~\cite{chatgpt}, LLaMA~\cite{li2023llama,gao2023llama}, the number of model parameters and training samples has grown rapidly, resulting in the continual growth of modal abilities and product deployment." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2304.15010" id="9313">
        <attvalues>
          <attvalue for="5" value=" In the past few years, researchers focused on the generation of each single modality and have achieved great progress: In terms of text generation, we have witnessed a qualitative leap in the performance of natural language processing tasks: From BERT~\cite{devlin2018bert}, GPT1~\cite{2018gpt1}, GPT2~\cite{2019gpt2}, GPT3~\cite{gpt3}, GPT4~\cite{gpt4} to ChatGPT~\cite{chatgpt}, LLaMA~\cite{li2023llama,gao2023llama}, the number of model parameters and training samples has grown rapidly, resulting in the continual growth of modal abilities and product deployment." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2307.01952" id="9314">
        <attvalues>
          <attvalue for="5" value=" In the visual generation field, with the rapid progress of diffusion models and large-scale image-text datasets, text-to-image (T2I) generation has achieved remarkable achievement and can synthesize high-quality images based on various user-provided text prompts, such as SDXL~\cite{sdxl} and PIXART-$\alpha$~\cite{chen2023pixart}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2310.00426" id="9315">
        <attvalues>
          <attvalue for="5" value=" In the visual generation field, with the rapid progress of diffusion models and large-scale image-text datasets, text-to-image (T2I) generation has achieved remarkable achievement and can synthesize high-quality images based on various user-provided text prompts, such as SDXL~\cite{sdxl} and PIXART-$\alpha$~\cite{chen2023pixart}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2204.03458" id="9316">
        <attvalues>
          <attvalue for="5" value=" Subsequently, significant advancements have been made in the field of text-to-video generation through the utilization of video diffusion models~\cite{vdm} and large-scale video-language datasets~\cite{Webvid}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2104.00650" id="9317">
        <attvalues>
          <attvalue for="5" value=" Subsequently, significant advancements have been made in the field of text-to-video generation through the utilization of video diffusion models~\cite{vdm} and large-scale video-language datasets~\cite{Webvid}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2211.13221" id="9318">
        <attvalues>
          <attvalue for="5" value=" Notably, several milestone works have emerged, such as~\cite{he2022lvdm, zhou2022magicvideo, singer2022make-a-video, ho2022imagen-video, villegas2022phenaki, chen2023videocrafter1, guo2023animatediff, bar2024lumiere, girdhar2023emuvideo, videocrafter2} and Sora~\cite{sora}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2110.02624" id="9320">
        <attvalues>
          <attvalue for="5" value=" For the 3D generation, with the emergence of the CLIP~\cite{radford2021learning} model, some methods~\cite{sanghi2022clip,mohammad2022clip,michel2022text2mesh} try to align the text information to the rendered images from 3D representations, {eg}, mesh, point cloud, NeRF~\cite{wang2023nerf} and gaussian splatting~\cite{kerbl3Dgaussians})." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2203.13333" id="9321">
        <attvalues>
          <attvalue for="5" value=" For the 3D generation, with the emergence of the CLIP~\cite{radford2021learning} model, some methods~\cite{sanghi2022clip,mohammad2022clip,michel2022text2mesh} try to align the text information to the rendered images from 3D representations, {eg}, mesh, point cloud, NeRF~\cite{wang2023nerf} and gaussian splatting~\cite{kerbl3Dgaussians})." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2112.03221" id="9322">
        <attvalues>
          <attvalue for="5" value=" For the 3D generation, with the emergence of the CLIP~\cite{radford2021learning} model, some methods~\cite{sanghi2022clip,mohammad2022clip,michel2022text2mesh} try to align the text information to the rendered images from 3D representations, {eg}, mesh, point cloud, NeRF~\cite{wang2023nerf} and gaussian splatting~\cite{kerbl3Dgaussians})." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2212.08070" id="9323">
        <attvalues>
          <attvalue for="5" value=" For the 3D generation, with the emergence of the CLIP~\cite{radford2021learning} model, some methods~\cite{sanghi2022clip,mohammad2022clip,michel2022text2mesh} try to align the text information to the rendered images from 3D representations, {eg}, mesh, point cloud, NeRF~\cite{wang2023nerf} and gaussian splatting~\cite{kerbl3Dgaussians})." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2112.10752" id="9325">
        <attvalues>
          <attvalue for="5" value=" Additionally, the integration of Stable Diffusion (SD)~\cite{ldm} with text-to-image rendering has enabled a series of works in text-to-3D generation~\cite{yi2023gaussiandreamer, tang2023dreamgaussian, hoellein2023text2room, liang2023_luciddreamer, yu2023_csd, li2023_sweetdreamer, wang2023prolificdreamer, lorraine2023_att3d, xu2023_dream3d, zhu2023_hifa, Chen_2023_ICCV, tsalicoglou2023textmesh, poole2022dreamfusion, lin2023magic3d,seo2023let}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2310.08529" id="9326">
        <attvalues>
          <attvalue for="5" value=" Additionally, the integration of Stable Diffusion (SD)~\cite{ldm} with text-to-image rendering has enabled a series of works in text-to-3D generation~\cite{yi2023gaussiandreamer, tang2023dreamgaussian, hoellein2023text2room, liang2023_luciddreamer, yu2023_csd, li2023_sweetdreamer, wang2023prolificdreamer, lorraine2023_att3d, xu2023_dream3d, zhu2023_hifa, Chen_2023_ICCV, tsalicoglou2023textmesh, poole2022dreamfusion, lin2023magic3d,seo2023let}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2303.07937" id="9327">
        <attvalues>
          <attvalue for="5" value=" Additionally, the integration of Stable Diffusion (SD)~\cite{ldm} with text-to-image rendering has enabled a series of works in text-to-3D generation~\cite{yi2023gaussiandreamer, tang2023dreamgaussian, hoellein2023text2room, liang2023_luciddreamer, yu2023_csd, li2023_sweetdreamer, wang2023prolificdreamer, lorraine2023_att3d, xu2023_dream3d, zhu2023_hifa, Chen_2023_ICCV, tsalicoglou2023textmesh, poole2022dreamfusion, lin2023magic3d,seo2023let}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2301.12503" id="9328">
        <attvalues>
          <attvalue for="5" value=" In the area of text-to-audio generation, a series of representative works tackle different audio domains such as~\cite{liu2023audioldm, liu2023audioldm2, kreuk2022audiogen} for text-to-audio,~\cite{agostinelli2023musiclm, copet2024musicgen, forsgren2022riffusion} for text-to-music, and~\cite{tan2024naturalspeech, shen2023naturalspeech2, ju2024naturalspeech3, wang2023valle, jiang2023megatts2, ren2020fastspeech2} for text-to-speech, and they have achieved significant performance in generating high-quality natural sounds, music, and human-level speech." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2301.11325" id="9329">
        <attvalues>
          <attvalue for="5" value=" In the area of text-to-audio generation, a series of representative works tackle different audio domains such as~\cite{liu2023audioldm, liu2023audioldm2, kreuk2022audiogen} for text-to-audio,~\cite{agostinelli2023musiclm, copet2024musicgen, forsgren2022riffusion} for text-to-music, and~\cite{tan2024naturalspeech, shen2023naturalspeech2, ju2024naturalspeech3, wang2023valle, jiang2023megatts2, ren2020fastspeech2} for text-to-speech, and they have achieved significant performance in generating high-quality natural sounds, music, and human-level speech." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2307.08041" id="9330">
        <attvalues>
          <attvalue for="5" value=" The first category involves encoding visual information into discrete token indices, trying to unify visual understanding and generation~\cite{ge2023planting, zeqiang2023mini, tang2023codi, ge2023making, sun2023emu2, zhao2023making}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2311.16465" id="9331">
        <attvalues>
          <attvalue for="5" value=" The second category focuses on leveraging LLMs to enhance the generation quality of existing pretrained T2I models: One type of approach utilizes an LLM as a layout planner to incorporate knowledge of object spatial positions, quantity, and object size, enabling the generation of required bounding boxes~\cite{chen2023textdiffuser2, lian2023llm, feng2023layoutgpt, zhang2023controllable, qu2023layoutllm}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2301.07093" id="9332">
        <attvalues>
          <attvalue for="5" value=" After obtaining the bounding boxes, the images can be generated through a grounded T2I model such as GLIGEN~\cite{li2023gligen}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2310.12945" id="9334">
        <attvalues>
          <attvalue for="5" value=" For 3D generation and editing, LLMs serve as a bridge between users and 3D assets, which improves interaction efficiency~\cite{sun20233d,feng2023posegpt} and helps users understand the 3D assets~\cite{chen2023ll3da,wu2023gpteval3d}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2311.18651" id="9335">
        <attvalues>
          <attvalue for="5" value=" For 3D generation and editing, LLMs serve as a bridge between users and 3D assets, which improves interaction efficiency~\cite{sun20233d,feng2023posegpt} and helps users understand the 3D assets~\cite{chen2023ll3da,wu2023gpteval3d}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2401.04092" id="9336">
        <attvalues>
          <attvalue for="5" value=" For 3D generation and editing, LLMs serve as a bridge between users and 3D assets, which improves interaction efficiency~\cite{sun20233d,feng2023posegpt} and helps users understand the 3D assets~\cite{chen2023ll3da,wu2023gpteval3d}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2305.11000" id="9337">
        <attvalues>
          <attvalue for="5" value=" In the context of audio generation and editing, the role of LLMs primarily lies in serving as coordinated backbones for multimodal audio~\cite{zhang2023speechgpt, gong2023listen,deshmukh2023pengi, rubenstein2023audiopalm, liu2024music, chen2023lauragpt, gardner2023llark, tang2023salmonn, chu2023qwen-audio, hussain2023m,shu2023llasm,yuan2024chatmusician,ding2024songcomposer}, conditioners for specific tasks~\cite{wu2023music, ghosal2023text, wu2024improving}, labelers for audio understanding~\cite{huang2023make2, wang2023assessing, vyas2023audiobox}, agents for interactive generation and editing~\cite{shen2023hugginggpt, huang2023audiogpt, liu2023wavjourney, yu2023musicagent, zhang2023loop, zhuo2023lyricwhiz}, as well as inspiration for novel approaches~\cite{wang2023valle, agostinelli2023musiclm, dhariwal2020jukebox, copet2024musicgen, borsos2023audiolm, yang2023uniaudio}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2305.11834" id="9338">
        <attvalues>
          <attvalue for="5" value=" In the context of audio generation and editing, the role of LLMs primarily lies in serving as coordinated backbones for multimodal audio~\cite{zhang2023speechgpt, gong2023listen,deshmukh2023pengi, rubenstein2023audiopalm, liu2024music, chen2023lauragpt, gardner2023llark, tang2023salmonn, chu2023qwen-audio, hussain2023m,shu2023llasm,yuan2024chatmusician,ding2024songcomposer}, conditioners for specific tasks~\cite{wu2023music, ghosal2023text, wu2024improving}, labelers for audio understanding~\cite{huang2023make2, wang2023assessing, vyas2023audiobox}, agents for interactive generation and editing~\cite{shen2023hugginggpt, huang2023audiogpt, liu2023wavjourney, yu2023musicagent, zhang2023loop, zhuo2023lyricwhiz}, as well as inspiration for novel approaches~\cite{wang2023valle, agostinelli2023musiclm, dhariwal2020jukebox, copet2024musicgen, borsos2023audiolm, yang2023uniaudio}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2308.15930" id="9339">
        <attvalues>
          <attvalue for="5" value=" In the context of audio generation and editing, the role of LLMs primarily lies in serving as coordinated backbones for multimodal audio~\cite{zhang2023speechgpt, gong2023listen,deshmukh2023pengi, rubenstein2023audiopalm, liu2024music, chen2023lauragpt, gardner2023llark, tang2023salmonn, chu2023qwen-audio, hussain2023m,shu2023llasm,yuan2024chatmusician,ding2024songcomposer}, conditioners for specific tasks~\cite{wu2023music, ghosal2023text, wu2024improving}, labelers for audio understanding~\cite{huang2023make2, wang2023assessing, vyas2023audiobox}, agents for interactive generation and editing~\cite{shen2023hugginggpt, huang2023audiogpt, liu2023wavjourney, yu2023musicagent, zhang2023loop, zhuo2023lyricwhiz}, as well as inspiration for novel approaches~\cite{wang2023valle, agostinelli2023musiclm, dhariwal2020jukebox, copet2024musicgen, borsos2023audiolm, yang2023uniaudio}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2402.16153" id="9340">
        <attvalues>
          <attvalue for="5" value=" In the context of audio generation and editing, the role of LLMs primarily lies in serving as coordinated backbones for multimodal audio~\cite{zhang2023speechgpt, gong2023listen,deshmukh2023pengi, rubenstein2023audiopalm, liu2024music, chen2023lauragpt, gardner2023llark, tang2023salmonn, chu2023qwen-audio, hussain2023m,shu2023llasm,yuan2024chatmusician,ding2024songcomposer}, conditioners for specific tasks~\cite{wu2023music, ghosal2023text, wu2024improving}, labelers for audio understanding~\cite{huang2023make2, wang2023assessing, vyas2023audiobox}, agents for interactive generation and editing~\cite{shen2023hugginggpt, huang2023audiogpt, liu2023wavjourney, yu2023musicagent, zhang2023loop, zhuo2023lyricwhiz}, as well as inspiration for novel approaches~\cite{wang2023valle, agostinelli2023musiclm, dhariwal2020jukebox, copet2024musicgen, borsos2023audiolm, yang2023uniaudio}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2305.18474" id="9343">
        <attvalues>
          <attvalue for="5" value=" In the context of audio generation and editing, the role of LLMs primarily lies in serving as coordinated backbones for multimodal audio~\cite{zhang2023speechgpt, gong2023listen,deshmukh2023pengi, rubenstein2023audiopalm, liu2024music, chen2023lauragpt, gardner2023llark, tang2023salmonn, chu2023qwen-audio, hussain2023m,shu2023llasm,yuan2024chatmusician,ding2024songcomposer}, conditioners for specific tasks~\cite{wu2023music, ghosal2023text, wu2024improving}, labelers for audio understanding~\cite{huang2023make2, wang2023assessing, vyas2023audiobox}, agents for interactive generation and editing~\cite{shen2023hugginggpt, huang2023audiogpt, liu2023wavjourney, yu2023musicagent, zhang2023loop, zhuo2023lyricwhiz}, as well as inspiration for novel approaches~\cite{wang2023valle, agostinelli2023musiclm, dhariwal2020jukebox, copet2024musicgen, borsos2023audiolm, yang2023uniaudio}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2301.02111" id="9344">
        <attvalues>
          <attvalue for="5" value=" In the context of audio generation and editing, the role of LLMs primarily lies in serving as coordinated backbones for multimodal audio~\cite{zhang2023speechgpt, gong2023listen,deshmukh2023pengi, rubenstein2023audiopalm, liu2024music, chen2023lauragpt, gardner2023llark, tang2023salmonn, chu2023qwen-audio, hussain2023m,shu2023llasm,yuan2024chatmusician,ding2024songcomposer}, conditioners for specific tasks~\cite{wu2023music, ghosal2023text, wu2024improving}, labelers for audio understanding~\cite{huang2023make2, wang2023assessing, vyas2023audiobox}, agents for interactive generation and editing~\cite{shen2023hugginggpt, huang2023audiogpt, liu2023wavjourney, yu2023musicagent, zhang2023loop, zhuo2023lyricwhiz}, as well as inspiration for novel approaches~\cite{wang2023valle, agostinelli2023musiclm, dhariwal2020jukebox, copet2024musicgen, borsos2023audiolm, yang2023uniaudio}." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2310.17796" id="9345">
        <attvalues>
          <attvalue for="5" value=" Besides, multimodal agents~\cite{shen2023hugginggpt,liu2023controlllm,yang2023gpt4tools,wu2023visual,liu2023internchat,li2023modelscope} integrate lots of AIGC tools into the framework as a universal system, which relies on LLMs to invoke tools but endows LLMs with the ability to comprehend and generate content of non-text modalities." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2305.18752" id="9346">
        <attvalues>
          <attvalue for="5" value=" Besides, multimodal agents~\cite{shen2023hugginggpt,liu2023controlllm,yang2023gpt4tools,wu2023visual,liu2023internchat,li2023modelscope} integrate lots of AIGC tools into the framework as a universal system, which relies on LLMs to invoke tools but endows LLMs with the ability to comprehend and generate content of non-text modalities." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2303.04671" id="9347">
        <attvalues>
          <attvalue for="5" value=" Besides, multimodal agents~\cite{shen2023hugginggpt,liu2023controlllm,yang2023gpt4tools,wu2023visual,liu2023internchat,li2023modelscope} integrate lots of AIGC tools into the framework as a universal system, which relies on LLMs to invoke tools but endows LLMs with the ability to comprehend and generate content of non-text modalities." />
        </attvalues>
      </edge>
      <edge source="2405.19334" target="2309.00986" id="9348">
        <attvalues>
          <attvalue for="5" value=" Besides, multimodal agents~\cite{shen2023hugginggpt,liu2023controlllm,yang2023gpt4tools,wu2023visual,liu2023internchat,li2023modelscope} integrate lots of AIGC tools into the framework as a universal system, which relies on LLMs to invoke tools but endows LLMs with the ability to comprehend and generate content of non-text modalities." />
        </attvalues>
      </edge>
      <edge source="2306.12581" target="2108.05682" id="9349">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art models seem to achieve quite high results in such cross-lingual evaluation campaigns, although recent works showed that there is still room for improvements \cite{goldman-etal-2022-un}." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2404.14219" id="9352">
        <attvalues>
          <attvalue for="5" value=" Typical SLMs [\cite{abdin2024phi, thawakar2024mobillama, team2024gemma}] are much more lightweight (eg, with $&lt;$3B parameters) than the existing LLMs \cite{touvron2023llama, almazrouei2023falcon} (eg, with $&gt;$65B parameters), but can achieve on-par accuracy in simpler or specific tasks." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2307.09288" id="9353">
        <attvalues>
          <attvalue for="5" value=" Typical SLMs [\cite{abdin2024phi, thawakar2024mobillama, team2024gemma}] are much more lightweight (eg, with $&lt;$3B parameters) than the existing LLMs \cite{touvron2023llama, almazrouei2023falcon} (eg, with $&gt;$65B parameters), but can achieve on-par accuracy in simpler or specific tasks." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2302.04089" id="9354">
        <attvalues>
          <attvalue for="5" value=" Model compression techniques reduce the model size, via either pruning that sparsifies the model structures [\cite{kurtic2024ziplm, ma2023llm, kurtic2024ziplm}], quantization that reduces the numerical precision of the model [\cite{lin2023awq, kim2024memory, chee2024quip}], or knowledge distillation that migrates the general knowledge to a smaller model [\cite{kang2024knowledge, zhao2023multistage}]." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2306.00978" id="9355">
        <attvalues>
          <attvalue for="5" value=" Model compression techniques reduce the model size, via either pruning that sparsifies the model structures [\cite{kurtic2024ziplm, ma2023llm, kurtic2024ziplm}], quantization that reduces the numerical precision of the model [\cite{lin2023awq, kim2024memory, chee2024quip}], or knowledge distillation that migrates the general knowledge to a smaller model [\cite{kang2024knowledge, zhao2023multistage}]." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2305.18395" id="9356">
        <attvalues>
          <attvalue for="5" value=" Model compression techniques reduce the model size, via either pruning that sparsifies the model structures [\cite{kurtic2024ziplm, ma2023llm, kurtic2024ziplm}], quantization that reduces the numerical precision of the model [\cite{lin2023awq, kim2024memory, chee2024quip}], or knowledge distillation that migrates the general knowledge to a smaller model [\cite{kang2024knowledge, zhao2023multistage}]." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="1905.10650" id="9357">
        <attvalues>
          <attvalue for="5" value=" However, most of these methods require intensive model retraining, which is computationally expensive and cannot well adapt to different downstream tasks or input data [\cite{michel2019sixteen, bansal2022rethinking}]." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2205.14135" id="9358">
        <attvalues>
          <attvalue for="5" value=" Other schemes of efficient attention algorithms [\cite{dao2022flashattention}] and decoding techniques [\cite{leviathan2023fast}] can also reduce the memory cost and computing latency in inference, but do not help reduce the model's redundancy." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2211.17192" id="9359">
        <attvalues>
          <attvalue for="5" value=" Other schemes of efficient attention algorithms [\cite{dao2022flashattention}] and decoding techniques [\cite{leviathan2023fast}] can also reduce the memory cost and computing latency in inference, but do not help reduce the model's redundancy." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2310.17157" id="9360">
        <attvalues>
          <attvalue for="5" value=" Such sparisification ratio is similar to that reported in existing work for LLMs [\cite{liu2023deja}], and allows significant memory savings and computing latency reduction." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="1703.01365" id="9361">
        <attvalues>
          <attvalue for="5" value=" Some attribution schemes (eg, Integrated Gradients [\cite{sundararajan2017axiomatic,yvinec2022singe}]) integrate multiple gradients over interpolated input samples to ensure precise attribution scores, but also incur high computing costs." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2207.04089" id="9362">
        <attvalues>
          <attvalue for="5" value=" Some attribution schemes (eg, Integrated Gradients [\cite{sundararajan2017axiomatic,yvinec2022singe}]) integrate multiple gradients over interpolated input samples to ensure precise attribution scores, but also incur high computing costs." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2108.00708" id="9363">
        <attvalues>
          <attvalue for="5" value=" Other schemes provide more computationally efficient methods by calculating the attribution scores as the product of a neuron's gradient and output magnitude (Gradient $\times$ Output) [\cite{liu2021group,lee2018snip}], which is the first-order approximation of the model output's change due to neuron deactivation." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2306.11644" id="9365">
        <attvalues>
          <attvalue for="5" value="5/2 [\cite{gunasekar2023textbooks}] and MobiLlama-0." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2402.16840" id="9366">
        <attvalues>
          <attvalue for="5" value="5B/1B [\cite{thawakar2024mobillama}], and multiple question answering (QA) datasets including TruthfulQA [\cite{lin2021truthfulqa}] and YahooAnswersQA [\cite{YahooAnswersQA}]." />
        </attvalues>
      </edge>
      <edge source="2406.06562" target="2109.07958" id="9367">
        <attvalues>
          <attvalue for="5" value="5B/1B [\cite{thawakar2024mobillama}], and multiple question answering (QA) datasets including TruthfulQA [\cite{lin2021truthfulqa}] and YahooAnswersQA [\cite{YahooAnswersQA}]." />
        </attvalues>
      </edge>
      <edge source="2310.11923" target="1905.05950" id="9369">
        <attvalues>
          <attvalue for="5" value=" More precisely, it is often argued that the task of masked language modelling encourages models to successively aggregate lexical/collocational, syntactic, and semantic information from the input text as the activations progress through encoder layers \cite{tenney-etal-2019-bert}." />
        </attvalues>
      </edge>
      <edge source="2310.11923" target="2005.04511" id="9370">
        <attvalues>
          <attvalue for="5" value=" While \cite{chi-etal-2020-finding} identify well-structured syntactic subspaces, eg\ those encoding the topology and labels of Universal Dependency trees, we target sentence-level semantic subspaces and carry out experiments on two semantic tasks, viz." />
        </attvalues>
      </edge>
      <edge source="2310.11923" target="2105.07452" id="9371">
        <attvalues>
          <attvalue for="5" value=" However, the association of middle layers of MLMs with syntax and higher levels with semantic information is not widely disputed as a general principle and is taken for granted in many papers on model analysis and downstream applications \cite{chi-etal-2020-finding,li-etal-2021-bert,sharma2022exploratory}." />
        </attvalues>
      </edge>
      <edge source="2310.11923" target="2204.10200" id="9372">
        <attvalues>
          <attvalue for="5" value=" However, the association of middle layers of MLMs with syntax and higher levels with semantic information is not widely disputed as a general principle and is taken for granted in many papers on model analysis and downstream applications \cite{chi-etal-2020-finding,li-etal-2021-bert,sharma2022exploratory}." />
        </attvalues>
      </edge>
      <edge source="2310.11923" target="2102.12452" id="9373">
        <attvalues>
          <attvalue for="5" value=" This can be largely attributed to the fact that the standard way of analyzing pretrained language models, namely probing, proceeds by applying linear classifiers to token representations at different layers \cite{belinkov-2022-probing}." />
        </attvalues>
      </edge>
      <edge source="2310.11923" target="2012.14913" id="9374">
        <attvalues>
          <attvalue for="5" value=" Conversely, while it is possible to provide nuanced analyses of causal Transformer-based models \cite{geva-etal-2021-transformer,geva-etal-2022-transformer}, such analyses are not easily transferable to MLMs \cite{nikolaev2023universe}." />
        </attvalues>
      </edge>
      <edge source="2310.11923" target="2203.14680" id="9375">
        <attvalues>
          <attvalue for="5" value=" Conversely, while it is possible to provide nuanced analyses of causal Transformer-based models \cite{geva-etal-2021-transformer,geva-etal-2022-transformer}, such analyses are not easily transferable to MLMs \cite{nikolaev2023universe}." />
        </attvalues>
      </edge>
      <edge source="1910.12094" target="1412.5567" id="9376">
        <attvalues>
          <attvalue for="5" value=" With many recent results \cite{hannun2014deep, amodei2016deep, collobert2016wav2letter}, end-to-end deep learning has created a larger interest in the speech community." />
        </attvalues>
      </edge>
      <edge source="1910.12094" target="1807.07104" id="9378">
        <attvalues>
          <attvalue for="5" value=" Hierarchical approaches \cite{Sanabria2018HierarchicalMT} introduced different granularity objectives by combining both character and phoneme prediction at different levels of the model." />
        </attvalues>
      </edge>
      <edge source="1910.12094" target="1807.05960" id="9379">
        <attvalues>
          <attvalue for="5" value=" With its success in computer vision under the few-shot learning setting~\cite{rusu2018meta, snell2017prototypical, vinyals2016matching}, there have been some works in language and speech processing, for instance, language transfer in neural machine translation \cite{gu2018meta}, dialogue generation \cite{mi2019meta}, and speaker adaptive training \cite{klejch2018learning}, but not multilingual pretraining for speech recognition." />
        </attvalues>
      </edge>
      <edge source="1910.12094" target="1808.08437" id="9380">
        <attvalues>
          <attvalue for="5" value=" With its success in computer vision under the few-shot learning setting~\cite{rusu2018meta, snell2017prototypical, vinyals2016matching}, there have been some works in language and speech processing, for instance, language transfer in neural machine translation \cite{gu2018meta}, dialogue generation \cite{mi2019meta}, and speaker adaptive training \cite{klejch2018learning}, but not multilingual pretraining for speech recognition." />
        </attvalues>
      </edge>
      <edge source="1910.12094" target="1905.05644" id="9381">
        <attvalues>
          <attvalue for="5" value=" With its success in computer vision under the few-shot learning setting~\cite{rusu2018meta, snell2017prototypical, vinyals2016matching}, there have been some works in language and speech processing, for instance, language transfer in neural machine translation \cite{gu2018meta}, dialogue generation \cite{mi2019meta}, and speaker adaptive training \cite{klejch2018learning}, but not multilingual pretraining for speech recognition." />
        </attvalues>
      </edge>
      <edge source="1910.12094" target="1808.10239" id="9382">
        <attvalues>
          <attvalue for="5" value=" With its success in computer vision under the few-shot learning setting~\cite{rusu2018meta, snell2017prototypical, vinyals2016matching}, there have been some works in language and speech processing, for instance, language transfer in neural machine translation \cite{gu2018meta}, dialogue generation \cite{mi2019meta}, and speaker adaptive training \cite{klejch2018learning}, but not multilingual pretraining for speech recognition." />
        </attvalues>
      </edge>
      <edge source="1910.12094" target="1703.03400" id="9383">
        <attvalues>
          <attvalue for="5" value=" We use model-agnostic meta-learning algorithm (MAML) \cite{finn2017model} in this work." />
        </attvalues>
      </edge>
      <edge source="2409.04009" target="1810.10147" id="9385">
        <attvalues>
          <attvalue for="5" value=" To attract more successive studies on few-shot RC, \cite{han2018fewrel} constructed a large-scale supervised few-shot relation classification dataset (FewRel) for the purpose of evaluating the performance of various meta-learning approaches, including Meta Network~\cite{munkhdalai2017meta}, GNN~\cite{garcia2018fewshot}, SNAIL~\cite{mishra2018a} and ProtoNet~\cite{NIPS2017_6996}, on RC." />
        </attvalues>
      </edge>
      <edge source="2409.04009" target="1703.00837" id="9386">
        <attvalues>
          <attvalue for="5" value=" To attract more successive studies on few-shot RC, \cite{han2018fewrel} constructed a large-scale supervised few-shot relation classification dataset (FewRel) for the purpose of evaluating the performance of various meta-learning approaches, including Meta Network~\cite{munkhdalai2017meta}, GNN~\cite{garcia2018fewshot}, SNAIL~\cite{mishra2018a} and ProtoNet~\cite{NIPS2017_6996}, on RC." />
        </attvalues>
      </edge>
      <edge source="2409.04009" target="1711.04043" id="9387">
        <attvalues>
          <attvalue for="5" value=" To attract more successive studies on few-shot RC, \cite{han2018fewrel} constructed a large-scale supervised few-shot relation classification dataset (FewRel) for the purpose of evaluating the performance of various meta-learning approaches, including Meta Network~\cite{munkhdalai2017meta}, GNN~\cite{garcia2018fewshot}, SNAIL~\cite{mishra2018a} and ProtoNet~\cite{NIPS2017_6996}, on RC." />
        </attvalues>
      </edge>
      <edge source="2409.04009" target="1707.03141" id="9388">
        <attvalues>
          <attvalue for="5" value=" To attract more successive studies on few-shot RC, \cite{han2018fewrel} constructed a large-scale supervised few-shot relation classification dataset (FewRel) for the purpose of evaluating the performance of various meta-learning approaches, including Meta Network~\cite{munkhdalai2017meta}, GNN~\cite{garcia2018fewshot}, SNAIL~\cite{mishra2018a} and ProtoNet~\cite{NIPS2017_6996}, on RC." />
        </attvalues>
      </edge>
      <edge source="2409.04009" target="1703.05175" id="9389">
        <attvalues>
          <attvalue for="5" value=" To attract more successive studies on few-shot RC, \cite{han2018fewrel} constructed a large-scale supervised few-shot relation classification dataset (FewRel) for the purpose of evaluating the performance of various meta-learning approaches, including Meta Network~\cite{munkhdalai2017meta}, GNN~\cite{garcia2018fewshot}, SNAIL~\cite{mishra2018a} and ProtoNet~\cite{NIPS2017_6996}, on RC." />
        </attvalues>
      </edge>
      <edge source="2211.15195" target="1706.03762" id="9390">
        <attvalues>
          <attvalue for="5" value=" The most recent discoveries are based on the Transformer architecture that enabled capturing the semantic meaning of the sentence~\cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2211.15195" target="1911.03863" id="9392">
        <attvalues>
          <attvalue for="5" value=" First of all, it is not designed to perform well when the number of observations is limited, so large training sets are consistently required for models to perform well~\cite{bansal2019learning}." />
        </attvalues>
      </edge>
      <edge source="2211.15195" target="2006.05987" id="9393">
        <attvalues>
          <attvalue for="5" value=" Secondly, the fine-tuning process happens to be very unstable across different runs with different seeds, even though just a few minor components of the learning process are dependent on random seeds~\cite{zhang2020revisiting}." />
        </attvalues>
      </edge>
      <edge source="2211.15195" target="1906.07413" id="9394">
        <attvalues>
          <attvalue for="5" value=" This leads to a poor generalization of the model and thus causes problems with noisy or outlier data~\cite{cao2019learning}." />
        </attvalues>
      </edge>
      <edge source="2406.07096" target="2104.02194" id="9396">
        <attvalues>
          <attvalue for="5" value=" In this case, the context-biasing list is supplied to the encoder or decoder via a cross-attention mechanism as a vector of an entire word \cite{Pundak2018DeepCE, Jain2020ContextualRF, Yang2023PromptASRFC} or a token from context trie \cite{Le2021ContextualizedSE, Harding2023SelectiveBW}." />
        </attvalues>
      </edge>
      <edge source="2406.07096" target="2310.00230" id="9397">
        <attvalues>
          <attvalue for="5" value=" There are also methods based on SpeechLM, when the context-biasing list is fed directly into the prompt for the LLM part of the model \cite{Wang2023SLMBT,Chen2023SALMSL}." />
        </attvalues>
      </edge>
      <edge source="2406.07096" target="2310.09424" id="9398">
        <attvalues>
          <attvalue for="5" value=" There are also methods based on SpeechLM, when the context-biasing list is fed directly into the prompt for the LLM part of the model \cite{Wang2023SLMBT,Chen2023SALMSL}." />
        </attvalues>
      </edge>
      <edge source="2406.07096" target="2209.01250" id="9399">
        <attvalues>
          <attvalue for="5" value=" We also propose a method of improving the recognition accuracy of abbreviations and complicated words with alternative transcriptions inspired by \cite{Fox2022ImprovingCR}, but obtained automatically without preliminary speech recognition." />
        </attvalues>
      </edge>
      <edge source="2406.07096" target="2101.06856" id="9400">
        <attvalues>
          <attvalue for="5" value=" It is also possible to combine an end-to-end ASR model with WFST to obtain context-biasing abilities of the classic models \cite{Fox2022ImprovingCR,Zhang2021TinyTA,Andrusenko2022ImprovingOO}." />
        </attvalues>
      </edge>
      <edge source="2406.07096" target="1211.3711" id="9401">
        <attvalues>
          <attvalue for="5" value=" This problem is considerably worsened in the case of the Transducer (RNN-T) model \cite{Graves2012SequenceTW} since beam-search decoding involves multiple Decoder (Prediction) and Joint networks calculations." />
        </attvalues>
      </edge>
      <edge source="2406.07096" target="2312.17279" id="9402">
        <attvalues>
          <attvalue for="5" value=" A Hybrid Transducer-CTC model \cite{noroozi2024stateful} (a shared encoder trained together with CTC and Transducer output heads) enables the use of the CTC-WS method for the Transducer model." />
        </attvalues>
      </edge>
      <edge source="2401.08833" target="1910.12607" id="9403">
        <attvalues>
          <attvalue for="5" value=" Further, contextual speech representations learned from self-supervised models are actually trained to predict the context such as the future or masked frames \cite{baevski2020wav2vec,hsu2021hubert,chung2020generative,chung2020vector,yang2022autoregressive}." />
        </attvalues>
      </edge>
      <edge source="2401.08833" target="2006.11477" id="9405">
        <attvalues>
          <attvalue for="5" value=" Further, contextual speech representations learned from self-supervised models are actually trained to predict the context such as the future or masked frames \cite{baevski2020wav2vec,hsu2021hubert,chung2020generative,chung2020vector,yang2022autoregressive}." />
        </attvalues>
      </edge>
      <edge source="2401.08833" target="2106.07447" id="9406">
        <attvalues>
          <attvalue for="5" value=" Further, contextual speech representations learned from self-supervised models are actually trained to predict the context such as the future or masked frames \cite{baevski2020wav2vec,hsu2021hubert,chung2020generative,chung2020vector,yang2022autoregressive}." />
        </attvalues>
      </edge>
      <edge source="2401.08833" target="2005.08392" id="9407">
        <attvalues>
          <attvalue for="5" value=" Further, contextual speech representations learned from self-supervised models are actually trained to predict the context such as the future or masked frames \cite{baevski2020wav2vec,hsu2021hubert,chung2020generative,chung2020vector,yang2022autoregressive}." />
        </attvalues>
      </edge>
      <edge source="2401.08833" target="2004.03061" id="9408">
        <attvalues>
          <attvalue for="5" value=" In this paper, we present an information-theoretic approach to assess the information contained in representations \cite{pimentel2020information,voita2020information}." />
        </attvalues>
      </edge>
      <edge source="2401.08833" target="2003.12298" id="9409">
        <attvalues>
          <attvalue for="5" value=" In this paper, we present an information-theoretic approach to assess the information contained in representations \cite{pimentel2020information,voita2020information}." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="1803.09047" id="9410">
        <attvalues>
          <attvalue for="5" value=" These approaches typically use autoencoders conditioned on lexical information and speaker identity \cite{skerry2018towards, wang2018style, battenberg2019effective, zhang2019learning}." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="2004.11284" id="9411">
        <attvalues>
          <attvalue for="5" value=" Other work has used a triple bottleneck to further decompose prosody in its subcomponents \cite{qian2020unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="2006.11477" id="9412">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, which learn representations directly from raw audio." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="1905.07195" id="9413">
        <attvalues>
          <attvalue for="5" value=" CHiVE represents prosody using $F_0$, $c_0$ and phoneme durations as features in a conditional variational autoencoder \cite{kenter2019chive}." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="1807.03748" id="9414">
        <attvalues>
          <attvalue for="5" value=" These include contrastive predictive coding (CPC) \cite{oord2018representation}, wav2vec \cite{schneider2019wav2vec}, vq-wav2vec \cite{baevski2019vq} and wav2vec 2." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="1904.05862" id="9415">
        <attvalues>
          <attvalue for="5" value=" These include contrastive predictive coding (CPC) \cite{oord2018representation}, wav2vec \cite{schneider2019wav2vec}, vq-wav2vec \cite{baevski2019vq} and wav2vec 2." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="1910.05453" id="9416">
        <attvalues>
          <attvalue for="5" value=" These include contrastive predictive coding (CPC) \cite{oord2018representation}, wav2vec \cite{schneider2019wav2vec}, vq-wav2vec \cite{baevski2019vq} and wav2vec 2." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="1711.02209" id="9417">
        <attvalues>
          <attvalue for="5" value=" Other approaches have used triplet loss and temporal proximity as a training signal to learn ``semantic'' \cite{jansen2018unsupervised} or ``non-semantic'' \cite{shor2020towards} representations from spectrograms." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="2002.12764" id="9418">
        <attvalues>
          <attvalue for="5" value=" Other approaches have used triplet loss and temporal proximity as a training signal to learn ``semantic'' \cite{jansen2018unsupervised} or ``non-semantic'' \cite{shor2020towards} representations from spectrograms." />
        </attvalues>
      </edge>
      <edge source="2107.08248" target="1609.03499" id="9419">
        <attvalues>
          <attvalue for="5" value=" Having been developed in the context of TTS, they have inductive biases that are particularly suited to learning primarily phonetic representations, rather than prosodic representations, a weakness highlighted by \cite{oord2016wavenet}." />
        </attvalues>
      </edge>
      <edge source="2110.04392" target="1911.02116" id="9421">
        <attvalues>
          <attvalue for="5" value=" Recent Natural Language Processing (NLP) systems based on pre-trained representations from Transformer language models, such as BERT \cite{devlin-etal-2019-bert} and XLM-Roberta \cite{conneau-etal-2020-unsupervised}, have achieved outstanding results in a variety of tasks." />
        </attvalues>
      </edge>
      <edge source="2110.04392" target="1702.08608" id="9422">
        <attvalues>
          <attvalue for="5" value=" Interpretability is a major concern in modern Artificial Intelligence (AI) and NLP research \cite{doshi2017towards,danilevsky-etal-2020-survey}, as black-box models undermine users’ trust in new technologies \cite{mercado2016intelligent,toreini2020relationship}." />
        </attvalues>
      </edge>
      <edge source="2110.04392" target="2010.00711" id="9423">
        <attvalues>
          <attvalue for="5" value=" Interpretability is a major concern in modern Artificial Intelligence (AI) and NLP research \cite{doshi2017towards,danilevsky-etal-2020-survey}, as black-box models undermine users’ trust in new technologies \cite{mercado2016intelligent,toreini2020relationship}." />
        </attvalues>
      </edge>
      <edge source="2110.04392" target="2005.00115" id="9426">
        <attvalues>
          <attvalue for="5" value=" Rationale extraction methods achieve this by selecting a portion of the input that justifies model output for a given data point \cite{lei-etal-2016-rationalizing,jain-etal-2020-learning}." />
        </attvalues>
      </edge>
      <edge source="2110.04392" target="1911.03429" id="9427">
        <attvalues>
          <attvalue for="5" value=" On the one hand, QE is different from other explainable NLP tasks with existing datasets \cite{deyoung2019eraser} in various important aspects." />
        </attvalues>
      </edge>
      <edge source="2110.04392" target="2010.04480" id="9428">
        <attvalues>
          <attvalue for="5" value=" As we will show in Section \ref{sec:data}, rationalized sentence-level evaluation can be a middle ground between relatively cheap but noisy annotations derived from post-editing \cite{fomicheva2020mlqepe} and very informative but expensive explicit error annotation based on error taxonomies, such as the Multidimensional Quality Metrics (MQM) framework \cite{LommelMQM:2014}." />
        </attvalues>
      </edge>
      <edge source="2110.04392" target="1908.04626" id="9429">
        <attvalues>
          <attvalue for="5" value=" First, it aims to explore the plausibility of explainable evaluation metrics \cite{wiegreffe-pinter-2019-attention}, by proposing a test set with manually annotated rationales." />
        </attvalues>
      </edge>
      <edge source="2407.16370" target="2107.13586" id="9431">
        <attvalues>
          <attvalue for="5" value=" The auto-regressive learning nature of LLMs introduces and empowers a new ``prompting'' mechanism based on input instructions, where users can provide text prompts to guide LLMs to complete particular tasks \cite{Liu2022} as a form of next token prediction." />
        </attvalues>
      </edge>
      <edge source="2407.16370" target="2309.15701" id="9432">
        <attvalues>
          <attvalue for="5" value=" Evaluation results on the CHiME-4 subset of the HyPoradise dataset \cite{Chen2023HyPoradise} show the effectiveness of the proposed algorithms." />
        </attvalues>
      </edge>
      <edge source="2407.16370" target="2011.11715" id="9433">
        <attvalues>
          <attvalue for="5" value=" Although this problem can be approached by language model re-scoring techniques~\cite{yang2021multi,liu2016attention, ma2023n}, recent studies \cite{Chen2023HyPoradise, Radhakrishnan2023, Gu2024DenoisingLM, Hu2024robustASR} have shown that leveraging LLMs to correct errors in an generative way often leads to better performance." />
        </attvalues>
      </edge>
      <edge source="2407.16370" target="1609.01454" id="9434">
        <attvalues>
          <attvalue for="5" value=" Although this problem can be approached by language model re-scoring techniques~\cite{yang2021multi,liu2016attention, ma2023n}, recent studies \cite{Chen2023HyPoradise, Radhakrishnan2023, Gu2024DenoisingLM, Hu2024robustASR} have shown that leveraging LLMs to correct errors in an generative way often leads to better performance." />
        </attvalues>
      </edge>
      <edge source="2407.16370" target="2309.15649" id="9435">
        <attvalues>
          <attvalue for="5" value=" When input prompts are critical for instructing LLMs on unseen ASR tasks, these task-activating prompts \cite{yang2023generative} are often empirically-designed and under-explored in the research community." />
        </attvalues>
      </edge>
      <edge source="2407.16370" target="2203.15863" id="9436">
        <attvalues>
          <attvalue for="5" value=" For example, early works~\cite{watanabe2017language, yang2021voice2series,gao2022wavprompt, chang2023speechprompt} focus on iterative optimization at the waveform level to instruct acoustic models for new tasks, but there are fewer studies on optimizing LLM-prompts for ASR tasks." />
        </attvalues>
      </edge>
      <edge source="2407.16370" target="2309.08532" id="9437">
        <attvalues>
          <attvalue for="5" value=" To achieve better post-ASR error correction, this paper explores alternative prompts for this task, and investigates a conditional evolutionary strategies based prompt optimization algorithm, named EvoPrompt \cite{Guo2024Evoprompt}, to refine the alternative prompts." />
        </attvalues>
      </edge>
      <edge source="2406.10602" target="1911.02116" id="9439">
        <attvalues>
          <attvalue for="5" value=" Subsequently, the primary challenge facing multilingual LLMs, known as the &quot;curse of multilinguality&quot; \cite{conneau2019unsupervised}, and the current attempts to solve it, are discussed." />
        </attvalues>
      </edge>
      <edge source="2010.03726" target="1704.04368" id="9440">
        <attvalues>
          <attvalue for="5" value=" For instance, 6\% of summary sentences generated by Pointer-Gen~\cite{see-etal-2017-get} are through fusion, whereas human abstracts contain 32\% fusion sentences." />
        </attvalues>
      </edge>
      <edge source="2010.03726" target="1705.04304" id="9441">
        <attvalues>
          <attvalue for="5" value=" A majority of the systems are trained end-to-end~\cite{see-etal-2017-get,paulus2018a,narayan-etal-2018-dont,chen-bansal-2018-fast,gehrmann-etal-2018-bottom,liu-lapata-2019-hierarchical}, where an abstractive summarizer is rewarded for generating summaries that contain the same words as human abstracts, measured by automatic metrics such as ROUGE~\cite{lin-2004-rouge}." />
        </attvalues>
      </edge>
      <edge source="2010.03726" target="1808.08745" id="9442">
        <attvalues>
          <attvalue for="5" value=" A majority of the systems are trained end-to-end~\cite{see-etal-2017-get,paulus2018a,narayan-etal-2018-dont,chen-bansal-2018-fast,gehrmann-etal-2018-bottom,liu-lapata-2019-hierarchical}, where an abstractive summarizer is rewarded for generating summaries that contain the same words as human abstracts, measured by automatic metrics such as ROUGE~\cite{lin-2004-rouge}." />
        </attvalues>
      </edge>
      <edge source="2010.03726" target="1805.11080" id="9443">
        <attvalues>
          <attvalue for="5" value=" A majority of the systems are trained end-to-end~\cite{see-etal-2017-get,paulus2018a,narayan-etal-2018-dont,chen-bansal-2018-fast,gehrmann-etal-2018-bottom,liu-lapata-2019-hierarchical}, where an abstractive summarizer is rewarded for generating summaries that contain the same words as human abstracts, measured by automatic metrics such as ROUGE~\cite{lin-2004-rouge}." />
        </attvalues>
      </edge>
      <edge source="2010.03726" target="1808.10792" id="9444">
        <attvalues>
          <attvalue for="5" value=" A majority of the systems are trained end-to-end~\cite{see-etal-2017-get,paulus2018a,narayan-etal-2018-dont,chen-bansal-2018-fast,gehrmann-etal-2018-bottom,liu-lapata-2019-hierarchical}, where an abstractive summarizer is rewarded for generating summaries that contain the same words as human abstracts, measured by automatic metrics such as ROUGE~\cite{lin-2004-rouge}." />
        </attvalues>
      </edge>
      <edge source="2010.03726" target="1905.13164" id="9445">
        <attvalues>
          <attvalue for="5" value=" A majority of the systems are trained end-to-end~\cite{see-etal-2017-get,paulus2018a,narayan-etal-2018-dont,chen-bansal-2018-fast,gehrmann-etal-2018-bottom,liu-lapata-2019-hierarchical}, where an abstractive summarizer is rewarded for generating summaries that contain the same words as human abstracts, measured by automatic metrics such as ROUGE~\cite{lin-2004-rouge}." />
        </attvalues>
      </edge>
      <edge source="2010.03726" target="1706.03762" id="9447">
        <attvalues>
          <attvalue for="5" value=" We address the challenge of fusing disparate sentences by enhancing the Transformer architecture~\cite{NIPS2017_7181} with points of correspondence between sentences, which are devices that tie two sentences together into a coherent text." />
        </attvalues>
      </edge>
      <edge source="2206.01767" target="1607.06520" id="9449">
        <attvalues>
          <attvalue for="5" value=" In the former, bias measurements are used to debias or correct biases in word representations to avoid encoded biases trickling down when applying these NLP models \cite{bolukbasi_man_2016, caliskan_semantics_2017}." />
        </attvalues>
      </edge>
      <edge source="2206.01767" target="1908.06361" id="9450">
        <attvalues>
          <attvalue for="5" value=" It is not clear whether it is possible to re-use seed set across corpora (thereby interfering with upstream use cases), and elements such as seed term frequency have been shown to affect bias measurements, and thus downstream uses \cite{ethayarajh2019understanding}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1406.1078" id="9451">
        <attvalues>
          <attvalue for="5" value=" Some of the popular NLG tasks include machine translation \cite{cho2014learning}, dialogue systems \cite{shang2015neural}, and text summarization \cite{rush2017neural}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1503.02364" id="9452">
        <attvalues>
          <attvalue for="5" value=" Some of the popular NLG tasks include machine translation \cite{cho2014learning}, dialogue systems \cite{shang2015neural}, and text summarization \cite{rush2017neural}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1811.01063" id="9453">
        <attvalues>
          <attvalue for="5" value=" The aspects of text generation that are commonly controlled include topic \cite{dziri-etal-2019-augmenting,feng2018topic,ijcai2018-619,xing2017topic}, style \cite{li2018delete, sudhakar2019transforming, prabhumoye2018style, chen2018adversarial}, emotion \cite{fu2018style,kong2019adversarial,DBLP:journals/inffus/SunLWLT20,zhou2018emotional}, and user preferences \cite{li-etal-2016-persona,luan-etal-2017-multi,yang2018investigating,yang2017personalized}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1805.03616" id="9454">
        <attvalues>
          <attvalue for="5" value=" Some of the applications of controllable text generation are context-based text generation \cite{jaech2018low}, topic-aware text generation, \cite{wang2018reinforced}, knowledge-enhanced text generation \cite{young2018augmenting} and text style transfer \cite{hu2022text}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1606.08340" id="9455">
        <attvalues>
          <attvalue for="5" value=" The aspects of text generation that are commonly controlled include topic \cite{dziri-etal-2019-augmenting,feng2018topic,ijcai2018-619,xing2017topic}, style \cite{li2018delete, sudhakar2019transforming, prabhumoye2018style, chen2018adversarial}, emotion \cite{fu2018style,kong2019adversarial,DBLP:journals/inffus/SunLWLT20,zhou2018emotional}, and user preferences \cite{li-etal-2016-persona,luan-etal-2017-multi,yang2018investigating,yang2017personalized}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1804.06437" id="9456">
        <attvalues>
          <attvalue for="5" value=" The aspects of text generation that are commonly controlled include topic \cite{dziri-etal-2019-augmenting,feng2018topic,ijcai2018-619,xing2017topic}, style \cite{li2018delete, sudhakar2019transforming, prabhumoye2018style, chen2018adversarial}, emotion \cite{fu2018style,kong2019adversarial,DBLP:journals/inffus/SunLWLT20,zhou2018emotional}, and user preferences \cite{li-etal-2016-persona,luan-etal-2017-multi,yang2018investigating,yang2017personalized}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1711.06861" id="9457">
        <attvalues>
          <attvalue for="5" value=" For example, embedding learning techniques are used to represent style \cite{fu2018style}, and then adversarial learning is used to match content but to distinguish between different styles\cite{hu2017toward, xu2018unpaired, john2018disentangled}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1901.07129" id="9458">
        <attvalues>
          <attvalue for="5" value=" The aspects of text generation that are commonly controlled include topic \cite{dziri-etal-2019-augmenting,feng2018topic,ijcai2018-619,xing2017topic}, style \cite{li2018delete, sudhakar2019transforming, prabhumoye2018style, chen2018adversarial}, emotion \cite{fu2018style,kong2019adversarial,DBLP:journals/inffus/SunLWLT20,zhou2018emotional}, and user preferences \cite{li-etal-2016-persona,luan-etal-2017-multi,yang2018investigating,yang2017personalized}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1603.06155" id="9460">
        <attvalues>
          <attvalue for="5" value=" For example, the persona of a speaker in dialogue \cite{li2016persona} or the sentiment of product reviews \cite{hu2017toward}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1710.07388" id="9461">
        <attvalues>
          <attvalue for="5" value=" The aspects of text generation that are commonly controlled include topic \cite{dziri-etal-2019-augmenting,feng2018topic,ijcai2018-619,xing2017topic}, style \cite{li2018delete, sudhakar2019transforming, prabhumoye2018style, chen2018adversarial}, emotion \cite{fu2018style,kong2019adversarial,DBLP:journals/inffus/SunLWLT20,zhou2018emotional}, and user preferences \cite{li-etal-2016-persona,luan-etal-2017-multi,yang2018investigating,yang2017personalized}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1710.02603" id="9462">
        <attvalues>
          <attvalue for="5" value=" Some of the applications of controllable text generation are context-based text generation \cite{jaech2018low}, topic-aware text generation, \cite{wang2018reinforced}, knowledge-enhanced text generation \cite{young2018augmenting} and text style transfer \cite{hu2022text}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="2010.12742" id="9463">
        <attvalues>
          <attvalue for="5" value=" A more detailed overview can be found in \cite{hu2022text, jin2022deep, mou2020stylized, toshevska2021review, prabhumoye2020exploring}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1912.02164" id="9465">
        <attvalues>
          <attvalue for="5" value=" The Plug and Play language model (PPLM) that was proposed by \cite{dathathri2019plug} takes an external input, performs computations on hidden states, and then combines a pre-trained language model with one or more simple attribute classifiers that guide text generation toward the desired topic or sentiment." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1810.04700" id="9466">
        <attvalues>
          <attvalue for="5" value=" Another model by \cite{gehrmann2018end} describes a training method based on diverse ensembling that would lead models to learn distinct text styles." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="1703.00955" id="9467">
        <attvalues>
          <attvalue for="5" value=" For example, embedding learning techniques are used to represent style \cite{fu2018style}, and then adversarial learning is used to match content but to distinguish between different styles\cite{hu2017toward, xu2018unpaired, john2018disentangled}." />
        </attvalues>
      </edge>
      <edge source="2407.14822" target="2011.00416" id="9468">
        <attvalues>
          <attvalue for="5" value=" Understanding and dealing with style in text proves to be very complex \cite{hu2022text}, but recent advances in deep learning techniques are helping stylized text generation tasks in various ways \cite{jin2022deep}." />
        </attvalues>
      </edge>
      <edge source="2301.00418" target="1812.11459" id="9470">
        <attvalues>
          <attvalue for="5" value=" Therefore, many previous works introduced various approaches to improve Vietnamese word segmentation performance, including single word segmentation task only \cite{diend1, dinhdien06, ha2003, nguyenetal2006, thangdq1, hongphuong08, songnguyen16, phongnt1, datnq1, vund2018, spanviws} and multi-task containing word segmentation and part-of-speech tagging \cite{ducpd1, nguyen-etal-2017-word}, dependency parsing \cite{nguyen-2019-neural}." />
        </attvalues>
      </edge>
      <edge source="2301.00418" target="1709.06307" id="9471">
        <attvalues>
          <attvalue for="5" value=" In summary our contributions are the following: \begin{itemize} \item Five pre-trained monolingual S4-based language models for Vietnamese, including one model without word segmentation, and four models using RDRsegmenter \cite{datnq1}, uitnlp \cite{vund2018}, pyvi, or underthesea toolkits in the pre-processing data phase." />
        </attvalues>
      </edge>
      <edge source="2301.00418" target="1607.01759" id="9472">
        <attvalues>
          <attvalue for="5" value="1007/978-3-030-82147-0_53} applied the fastText \cite{joulin2017bag} model for pre-processing and embedding the input data without the Vietnamese word segmentation phase." />
        </attvalues>
      </edge>
      <edge source="2301.00418" target="2003.00744" id="9475">
        <attvalues>
          <attvalue for="5" value=" Indeed, the research \cite{9335912} used RDRsegmenter toolkit for data pre-processing before using the pre-trained monolingual PhoBERT model \cite{phobert}, which is made for Vietnamese and applied Byte-Pair Encoding (BPE) method \cite{sennrich-etal-2016-neural} for sub-word representations for Vietnamese." />
        </attvalues>
      </edge>
      <edge source="2301.00418" target="1508.07909" id="9476">
        <attvalues>
          <attvalue for="5" value=" Indeed, the research \cite{9335912} used RDRsegmenter toolkit for data pre-processing before using the pre-trained monolingual PhoBERT model \cite{phobert}, which is made for Vietnamese and applied Byte-Pair Encoding (BPE) method \cite{sennrich-etal-2016-neural} for sub-word representations for Vietnamese." />
        </attvalues>
      </edge>
      <edge source="2301.00418" target="2110.00156" id="9477">
        <attvalues>
          <attvalue for="5" value="31\% achieved by the span labeling approach \cite{spanviws} using XLM-RoBERTa \cite{conneau-etal-2020-unsupervised}, which is very slow when inference on CPU device." />
        </attvalues>
      </edge>
      <edge source="2301.00418" target="1911.02116" id="9478">
        <attvalues>
          <attvalue for="5" value="31\% achieved by the span labeling approach \cite{spanviws} using XLM-RoBERTa \cite{conneau-etal-2020-unsupervised}, which is very slow when inference on CPU device." />
        </attvalues>
      </edge>
      <edge source="2301.00418" target="2006.07804" id="9479">
        <attvalues>
          <attvalue for="5" value=" In summary our contributions are the following: \begin{itemize} \item Five pre-trained monolingual S4-based language models for Vietnamese, including one model without word segmentation, and four models using RDRsegmenter \cite{datnq1}, uitnlp \cite{vund2018}, pyvi, or underthesea toolkits in the pre-processing data phase." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2307.09288" id="9481">
        <attvalues>
          <attvalue for="5" value=" These models, fitted on huge text corpora, can produce responses resembling those of humans~\cite{touvron2023llama2,gpt-4}." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2303.08774" id="9482">
        <attvalues>
          <attvalue for="5" value=" These models, fitted on huge text corpora, can produce responses resembling those of humans~\cite{touvron2023llama2,gpt-4}." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2112.04359" id="9483">
        <attvalues>
          <attvalue for="5" value=" However, since LMs often generate wrong or hallucinated responses~\cite{weidinger2021ethical,xiao2021hallucination,huang2024one}, it is crucial to correctly quantify their level of uncertainty in responding to particular inputs." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2405.19544" id="9485">
        <attvalues>
          <attvalue for="5" value=" However, since LMs often generate wrong or hallucinated responses~\cite{weidinger2021ethical,xiao2021hallucination,huang2024one}, it is crucial to correctly quantify their level of uncertainty in responding to particular inputs." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="1506.02142" id="9486">
        <attvalues>
          <attvalue for="5" value=" Uncertainty quantification is well-explored in supervised learning, specifically in classification~\cite[eg,][etc]{lichtenstein1977calibration,gal2016dropout,lakshminarayanan2017simple}." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="1612.01474" id="9487">
        <attvalues>
          <attvalue for="5" value=" Uncertainty quantification is well-explored in supervised learning, specifically in classification~\cite[eg,][etc]{lichtenstein1977calibration,gal2016dropout,lakshminarayanan2017simple}." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2203.01850" id="9488">
        <attvalues>
          <attvalue for="5" value=" In classification, a confidence measure is an estimate of the probability that the predicted class $\widehat Y$ matches the true class label $Y$~\cite{lichtenstein1977calibration,lee2023t}." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="1706.04599" id="9489">
        <attvalues>
          <attvalue for="5" value="\tag{ECE} \end{equation} In classification, confidence measures are predominantly built on model logits~\cite{guo2017calibration,kull2019beyond}." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2102.09690" id="9490">
        <attvalues>
          <attvalue for="5" value=" Third, even hand-crafted prompts intended to make LMs express confidence explicitly may not lead to reliable confidence values because elicitation is heavily tied to prompt formats~\cite{zhao2021calibrate,xiong2023llms}." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2306.13063" id="9491">
        <attvalues>
          <attvalue for="5" value=" Third, even hand-crafted prompts intended to make LMs express confidence explicitly may not lead to reliable confidence values because elicitation is heavily tied to prompt formats~\cite{zhao2021calibrate,xiong2023llms}." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2302.09664" id="9492">
        <attvalues>
          <attvalue for="5" value=" For instance, the semantic entropy \cite{kuhn2023semantic} can take arbitrarily large positive values, whereas the EigV measure of \cite{lin2023generating} depends on the number of responses generated." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2305.19187" id="9493">
        <attvalues>
          <attvalue for="5" value=" For instance, the semantic entropy \cite{kuhn2023semantic} can take arbitrarily large positive values, whereas the EigV measure of \cite{lin2023generating} depends on the number of responses generated." />
        </attvalues>
      </edge>
      <edge source="2404.03163" target="2402.03744" id="9494">
        <attvalues>
          <attvalue for="5" value=" Further, \cite{chen2024inside} characterize differential entropy in the embedding space with EigenScore, via the covariance of embeddings of potential responses." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="1706.03762" id="9497">
        <attvalues>
          <attvalue for="5" value=" In a nutshell, Transformers~\cite{vaswani2017attention} are founded on three key innovations: positional encoding, scaled dot product attention, and multi-head attention (we will come back to these elements in more detail in Section~\ref{sec:background})." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="1908.09635" id="9498">
        <attvalues>
          <attvalue for="5" value=" In our work, we adopt the following commonly accepted definition \cite{mehrabi2021survey}: fairness refers to the absence of any prejudice or favoritism towards an individual or a group based on their intrinsic or acquired traits." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="1812.08769" id="9499">
        <attvalues>
          <attvalue for="5" value=" A lack of diversity inherent to the data, for instance, a corpus containing a large majority of male profiles (eg sample bias), will cause the model to maintain and accentuate a gender bias~\cite{swinger2019biases}." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="1904.08783" id="9500">
        <attvalues>
          <attvalue for="5" value=" These issues are observed in different levels of the NLP pipeline: text encoding~\cite{basta2019evaluating,kurita2019measuring}, during the fine-tuning process~\cite{delobell2021measuringfairness}, or simply as the potential harm caused on downstream tasks~\cite{kurita2019measuring}, with dedicated studies on language generation \cite{sheng2019woman}, document classification \cite{bhardwaj2020investigating}, toxicity detection, and sentiment analysis \cite{hutchinson2020social}." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="1906.07337" id="9501">
        <attvalues>
          <attvalue for="5" value=" These issues are observed in different levels of the NLP pipeline: text encoding~\cite{basta2019evaluating,kurita2019measuring}, during the fine-tuning process~\cite{delobell2021measuringfairness}, or simply as the potential harm caused on downstream tasks~\cite{kurita2019measuring}, with dedicated studies on language generation \cite{sheng2019woman}, document classification \cite{bhardwaj2020investigating}, toxicity detection, and sentiment analysis \cite{hutchinson2020social}." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="1909.01326" id="9502">
        <attvalues>
          <attvalue for="5" value=" These issues are observed in different levels of the NLP pipeline: text encoding~\cite{basta2019evaluating,kurita2019measuring}, during the fine-tuning process~\cite{delobell2021measuringfairness}, or simply as the potential harm caused on downstream tasks~\cite{kurita2019measuring}, with dedicated studies on language generation \cite{sheng2019woman}, document classification \cite{bhardwaj2020investigating}, toxicity detection, and sentiment analysis \cite{hutchinson2020social}." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="2009.05021" id="9503">
        <attvalues>
          <attvalue for="5" value=" These issues are observed in different levels of the NLP pipeline: text encoding~\cite{basta2019evaluating,kurita2019measuring}, during the fine-tuning process~\cite{delobell2021measuringfairness}, or simply as the potential harm caused on downstream tasks~\cite{kurita2019measuring}, with dedicated studies on language generation \cite{sheng2019woman}, document classification \cite{bhardwaj2020investigating}, toxicity detection, and sentiment analysis \cite{hutchinson2020social}." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="2005.00813" id="9504">
        <attvalues>
          <attvalue for="5" value=" These issues are observed in different levels of the NLP pipeline: text encoding~\cite{basta2019evaluating,kurita2019measuring}, during the fine-tuning process~\cite{delobell2021measuringfairness}, or simply as the potential harm caused on downstream tasks~\cite{kurita2019measuring}, with dedicated studies on language generation \cite{sheng2019woman}, document classification \cite{bhardwaj2020investigating}, toxicity detection, and sentiment analysis \cite{hutchinson2020social}." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="2008.05221" id="9505">
        <attvalues>
          <attvalue for="5" value=" Several model compression techniques have been proposed, as discussed in~\cite{Gupta22tkdd} and namely the following compression families : pruning, quantization and distillation." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="2101.01321" id="9506">
        <attvalues>
          <attvalue for="5" value=" The primer \cite{lecun1989optimal} increases the speed and generalization capacities by removing the less important model's weights with regard to the task, while quantization approximates the model’s weights to reduce its complexity (eg reducing the numerical precision of the weights \cite{kim2021ibert})." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="1503.02531" id="9507">
        <attvalues>
          <attvalue for="5" value=" Finally, distillation \cite{hinton2015distilling} consists in training a smaller model (called student model) to mimic the predictions of the large PLM to distill (teacher model)." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="1911.05248" id="9510">
        <attvalues>
          <attvalue for="5" value=" Looking at the impact of model distillation through fairness lenses has started to be investigated, mainly in the context of computer vision~\cite{hooker2020characterising,hooker2021compressed,lukasik2021teacher}." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="2207.04546" id="9512">
        <attvalues>
          <attvalue for="5" value=" While some works have shown that distilled versions of PLMs can exacerbate bias \cite{radford2018improving,delobelle2022fairdistillation}, other articles seem to reach an opposite conclusion \cite{xu2022can}; in this latter, authors state that model distillation acts as a regularization technique allowing bias reduction." />
        </attvalues>
      </edge>
      <edge source="2401.06495" target="2201.08542" id="9513">
        <attvalues>
          <attvalue for="5" value=" While some works have shown that distilled versions of PLMs can exacerbate bias \cite{radford2018improving,delobelle2022fairdistillation}, other articles seem to reach an opposite conclusion \cite{xu2022can}; in this latter, authors state that model distillation acts as a regularization technique allowing bias reduction." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="1909.01326" id="9515">
        <attvalues>
          <attvalue for="5" value=" However, these models have shown to be prone to picking up unwanted correlations and stereotypes from the pre-training data \cite{sheng-etal-2019-woman, kurita-etal-2019-measuring, hutchinson-etal-2020-social} which, can perpetuate harmful biases for people belonging to marginalized groups." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="2004.09456" id="9516">
        <attvalues>
          <attvalue for="5" value=" While there has been a great deal of interest in understanding and mitigating such biases in LLMs \cite{nadeem-etal-2021-stereoset, schick-etal-2021-self, meade-etal-2022-empirical}, the focus of such studies has primarily been on English." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="2005.00699" id="9517">
        <attvalues>
          <attvalue for="5" value=" Past work on evaluating and mitigating biases in multilingual models has mostly been concerned with gender bias in cross-lingual word embeddings \cite{zhao-etal-2020-gender, bansal-etal-2021-debiasing} which fails to account for contextual information \cite{kurita-etal-2019-measuring, delobelle-etal-2022-measuring}, making them unreliable for LLMs." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="1906.07337" id="9518">
        <attvalues>
          <attvalue for="5" value=" Past work on evaluating and mitigating biases in multilingual models has mostly been concerned with gender bias in cross-lingual word embeddings \cite{zhao-etal-2020-gender, bansal-etal-2021-debiasing} which fails to account for contextual information \cite{kurita-etal-2019-measuring, delobelle-etal-2022-measuring}, making them unreliable for LLMs." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="2205.00551" id="9519">
        <attvalues>
          <attvalue for="5" value=" Other methods for estimating biases in contextualized representations involve Multilingual Bias Evaluation \cite[MBE]{kaneko-etal-2022-gender}, which utilizes parallel translation corpora in different languages that might lack non-western cultural contexts \cite{talat-etal-2022-reap}." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="2109.03646" id="9520">
        <attvalues>
          <attvalue for="5" value=" For debiasing LLMs, \cite{lauscher-etal-2021-sustainable-modular} proposed an adapter \cite{houlsby-etal-2019-parameter} based approach." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="1902.00751" id="9521">
        <attvalues>
          <attvalue for="5" value=" For debiasing LLMs, \cite{lauscher-etal-2021-sustainable-modular} proposed an adapter \cite{houlsby-etal-2019-parameter} based approach." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="2010.06032" id="9522">
        <attvalues>
          <attvalue for="5" value=" First, we extend the DisCo metric \cite{webster2020measuring} by creating human-corrected templates for 6 Indian languages." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="1804.06876" id="9523">
        <attvalues>
          <attvalue for="5" value=" Second, we extend existing debiasing strategies like Counterfactual Data Augmentation \cite{zhao-etal-2018-gender} and Self-Debiasing \cite{schick-etal-2021-self} to mitigate gender biases across languages in Masked Language Models (MLMs)." />
        </attvalues>
      </edge>
      <edge source="2307.01503" target="2103.00453" id="9524">
        <attvalues>
          <attvalue for="5" value=" Second, we extend existing debiasing strategies like Counterfactual Data Augmentation \cite{zhao-etal-2018-gender} and Self-Debiasing \cite{schick-etal-2021-self} to mitigate gender biases across languages in Masked Language Models (MLMs)." />
        </attvalues>
      </edge>
      <edge source="2303.01191" target="1901.07291" id="9525">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{conneau2019cross} employ Masked Language Modeling (MLM) objective to train an encoder." />
        </attvalues>
      </edge>
      <edge source="2303.01191" target="1905.02450" id="9526">
        <attvalues>
          <attvalue for="5" value=" In this direction, we compare MASS \cite{song2019mass} and DAE \cite{artetxe2018unsupervised} on language pairs with different word-orders." />
        </attvalues>
      </edge>
      <edge source="2303.01191" target="2001.08210" id="9527">
        <attvalues>
          <attvalue for="5" value=" \cite{liu-etal-2020-multilingual-denoising} use Denoising Auto-Encoder (DAE) to pre-train the model." />
        </attvalues>
      </edge>
      <edge source="2303.01191" target="2103.10531" id="9528">
        <attvalues>
          <attvalue for="5" value=" Recently, several approaches have been proposed to address the issue of lexical divergence between languages \cite{chronopoulou-etal-2021-improving,banerjee-etal-2021-crosslingual, khatri2021simple}." />
        </attvalues>
      </edge>
      <edge source="2303.01191" target="2106.04995" id="9529">
        <attvalues>
          <attvalue for="5" value=" Recently, several approaches have been proposed to address the issue of lexical divergence between languages \cite{chronopoulou-etal-2021-improving,banerjee-etal-2021-crosslingual, khatri2021simple}." />
        </attvalues>
      </edge>
      <edge source="2303.01191" target="1811.00383" id="9530">
        <attvalues>
          <attvalue for="5" value=" In a transfer learning scenario, \cite{murthy2018addressing} observed improvement in result by re-ordering the assisting source language sentences to match the word-order of the actual source language before training the parent model." />
        </attvalues>
      </edge>
      <edge source="2303.01191" target="1809.01272" id="9531">
        <attvalues>
          <attvalue for="5" value=" In this direction, we compare MASS \cite{song2019mass} and DAE \cite{artetxe2018unsupervised} on language pairs with different word-orders." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2112.02418" id="9533">
        <attvalues>
          <attvalue for="5" value=" One-shot approach, an alternative type of adaptive TTS, constructs personalized TTS by fine-tuning pre-trained multi-speaker TTS models with few reference speeches of target speaker \cite{pmlr-v162-casanova22a, DBLP:journals/corr/abs-2005-05642, Moss2020BOFFINTF, hsieh23_interspeech, NEURIPS2018_4559912e, chen2021adaspeech, 9414872}." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2005.05642" id="9534">
        <attvalues>
          <attvalue for="5" value=" To efficiently adapt to the target speaker, several studies fine-tuned a subset of the model's parameters \cite{DBLP:journals/corr/abs-2005-05642, Moss2020BOFFINTF, NEURIPS2018_4559912e, chen2021adaspeech, 9414872}, or leveraged adapter-based fine-tuning techniques \cite{hsieh23_interspeech} such as Low-Rank Adaptation (LoRA) \cite{hu2022lora} or prefix-tuning \cite{li-liang-2021-prefix}, which only fine-tune the parameters of newly integrated adapters." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2211.00585" id="9535">
        <attvalues>
          <attvalue for="5" value=" To efficiently adapt to the target speaker, several studies fine-tuned a subset of the model's parameters \cite{DBLP:journals/corr/abs-2005-05642, Moss2020BOFFINTF, NEURIPS2018_4559912e, chen2021adaspeech, 9414872}, or leveraged adapter-based fine-tuning techniques \cite{hsieh23_interspeech} such as Low-Rank Adaptation (LoRA) \cite{hu2022lora} or prefix-tuning \cite{li-liang-2021-prefix}, which only fine-tune the parameters of newly integrated adapters." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2106.09685" id="9536">
        <attvalues>
          <attvalue for="5" value=" To efficiently adapt to the target speaker, several studies fine-tuned a subset of the model's parameters \cite{DBLP:journals/corr/abs-2005-05642, Moss2020BOFFINTF, NEURIPS2018_4559912e, chen2021adaspeech, 9414872}, or leveraged adapter-based fine-tuning techniques \cite{hsieh23_interspeech} such as Low-Rank Adaptation (LoRA) \cite{hu2022lora} or prefix-tuning \cite{li-liang-2021-prefix}, which only fine-tune the parameters of newly integrated adapters." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2101.00190" id="9537">
        <attvalues>
          <attvalue for="5" value=" To efficiently adapt to the target speaker, several studies fine-tuned a subset of the model's parameters \cite{DBLP:journals/corr/abs-2005-05642, Moss2020BOFFINTF, NEURIPS2018_4559912e, chen2021adaspeech, 9414872}, or leveraged adapter-based fine-tuning techniques \cite{hsieh23_interspeech} such as Low-Rank Adaptation (LoRA) \cite{hu2022lora} or prefix-tuning \cite{li-liang-2021-prefix}, which only fine-tune the parameters of newly integrated adapters." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2006.11239" id="9538">
        <attvalues>
          <attvalue for="5" value=" Recently, inspired by successes of diffusion-based generative model \cite{DDPM} on fine-tuning-based personalized generation tasks \cite{Ruiz_2023_CVPR}, diffusion-based one-shot TTS models have been proposed \cite{kim2022guidedtts, kim23k_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2208.12242" id="9539">
        <attvalues>
          <attvalue for="5" value=" Recently, inspired by successes of diffusion-based generative model \cite{DDPM} on fine-tuning-based personalized generation tasks \cite{Ruiz_2023_CVPR}, diffusion-based one-shot TTS models have been proposed \cite{kim2022guidedtts, kim23k_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2205.15370" id="9540">
        <attvalues>
          <attvalue for="5" value=" Recently, inspired by successes of diffusion-based generative model \cite{DDPM} on fine-tuning-based personalized generation tasks \cite{Ruiz_2023_CVPR}, diffusion-based one-shot TTS models have been proposed \cite{kim2022guidedtts, kim23k_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2408.14739" target="2306.16083" id="9541">
        <attvalues>
          <attvalue for="5" value=" We utilize a diffusion-based pre-trained TTS model and adopt a fine-tuning methodology following UnitSpeech \cite{kim23k_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1807.07987" id="9542">
        <attvalues>
          <attvalue for="5" value=" The rise of deep learning~\cite{lecun2015deep} has made more complex sequence generation tasks~\cite{sutskever2014sequence,Wang2017TacotronTE,shen2018natural,oord2016wavenet,Kalchbrenner2018EfficientNA} feasible." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1409.3215" id="9543">
        <attvalues>
          <attvalue for="5" value=" The rise of deep learning~\cite{lecun2015deep} has made more complex sequence generation tasks~\cite{sutskever2014sequence,Wang2017TacotronTE,shen2018natural,oord2016wavenet,Kalchbrenner2018EfficientNA} feasible." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1703.10135" id="9544">
        <attvalues>
          <attvalue for="5" value=" Last, to generate the smooth mel spectrograms in a streaming inference manner, we adopt a time-delayed LSTM post-net instead of a global CBHG-like~\cite{Wang2017TacotronTE} module." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1712.05884" id="9545">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, when dealing with out-of-domain or abnormal texts inputs, Tacotron-like attention based end-to-end structures could render unacceptable errors, including skipping, repeating, long unexpected pause and attention collapse~\cite{shen2018natural,tiantencent}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1609.03499" id="9546">
        <attvalues>
          <attvalue for="5" value=" Tacotron, usually followed by a traditional or neural vocoder~\cite{oord2016wavenet,Griffin1984SignalEF}, takes linguistic feature and speaker identity as input and generates mel-spectrogram as output." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1802.08435" id="9547">
        <attvalues>
          <attvalue for="5" value=" The rise of deep learning~\cite{lecun2015deep} has made more complex sequence generation tasks~\cite{sutskever2014sequence,Wang2017TacotronTE,shen2018natural,oord2016wavenet,Kalchbrenner2018EfficientNA} feasible." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1906.00672" id="9548">
        <attvalues>
          <attvalue for="5" value=" Through various evaluations, our proposed AdaDurIAN significantly surpasses the Tacotron-like model~\cite{He2019RobustSA} in terms of naturalness, speaker similarity and cross-lingual speaking, and also shows its promising performance in few-shot emotion transfer tasks." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1704.00784" id="9549">
        <attvalues>
          <attvalue for="5" value=" More recently, stepwise monotonic attention (SMA)~\cite{He2019RobustSA} method, which is based on monotonic attention~\cite{raffel2017online}, was proposed to enforce strict constraint to meet the demand of locality, monotonicity and completeness in the speech synthesis process." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1808.10128" id="9550">
        <attvalues>
          <attvalue for="5" value=" However, building TTS system with limited data often sacrifices quality and reliability~\cite{chung2019semi}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1806.04558" id="9551">
        <attvalues>
          <attvalue for="5" value=" There are mainly two approaches here: the first is just to update the new speaker embedding and combine it with linguistic feature as inputs to a TTS model~\cite{jia2018transfer,li2017deep}, which may require a very strong speaker encoder network trained by thousands of speakers~\cite{8462665}; the second is to fine-tune the entire multi-speaker network to select a optimal single-speaker model~\cite{arik2018neural,chen2018sample,9054301}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1705.02304" id="9552">
        <attvalues>
          <attvalue for="5" value=" There are mainly two approaches here: the first is just to update the new speaker embedding and combine it with linguistic feature as inputs to a TTS model~\cite{jia2018transfer,li2017deep}, which may require a very strong speaker encoder network trained by thousands of speakers~\cite{8462665}; the second is to fine-tune the entire multi-speaker network to select a optimal single-speaker model~\cite{arik2018neural,chen2018sample,9054301}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1710.10467" id="9553">
        <attvalues>
          <attvalue for="5" value=" There are mainly two approaches here: the first is just to update the new speaker embedding and combine it with linguistic feature as inputs to a TTS model~\cite{jia2018transfer,li2017deep}, which may require a very strong speaker encoder network trained by thousands of speakers~\cite{8462665}; the second is to fine-tune the entire multi-speaker network to select a optimal single-speaker model~\cite{arik2018neural,chen2018sample,9054301}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1802.06006" id="9554">
        <attvalues>
          <attvalue for="5" value=" There are mainly two approaches here: the first is just to update the new speaker embedding and combine it with linguistic feature as inputs to a TTS model~\cite{jia2018transfer,li2017deep}, which may require a very strong speaker encoder network trained by thousands of speakers~\cite{8462665}; the second is to fine-tune the entire multi-speaker network to select a optimal single-speaker model~\cite{arik2018neural,chen2018sample,9054301}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1809.10460" id="9555">
        <attvalues>
          <attvalue for="5" value=" There are mainly two approaches here: the first is just to update the new speaker embedding and combine it with linguistic feature as inputs to a TTS model~\cite{jia2018transfer,li2017deep}, which may require a very strong speaker encoder network trained by thousands of speakers~\cite{8462665}; the second is to fine-tune the entire multi-speaker network to select a optimal single-speaker model~\cite{arik2018neural,chen2018sample,9054301}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="2002.01953" id="9556">
        <attvalues>
          <attvalue for="5" value=" There are mainly two approaches here: the first is just to update the new speaker embedding and combine it with linguistic feature as inputs to a TTS model~\cite{jia2018transfer,li2017deep}, which may require a very strong speaker encoder network trained by thousands of speakers~\cite{8462665}; the second is to fine-tune the entire multi-speaker network to select a optimal single-speaker model~\cite{arik2018neural,chen2018sample,9054301}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1909.01700" id="9558">
        <attvalues>
          <attvalue for="5" value=" To achieve naturalness and robustness in speech synthesis, FastSpeech~\cite{ren2019fastspeech} and duration informed attention network (DurIAN)~\cite{Yu2019DurIANDI} have been recently proposed to overcome the unexpected errors of end-to-end systems by combining duration information of traditional statistical parametric speech synthesis system~\cite{zen2009statistical}." />
        </attvalues>
      </edge>
      <edge source="2005.05642" target="1409.0473" id="9559">
        <attvalues>
          <attvalue for="5" value=" The latter DurIAN, originally proposed for multi-modal speech synthesis, is an autoregressive framework which achieves robustness and naturalness by using skip state encoder and combining duration with windowed content-based attention~\cite{Bahdanau2015NeuralMT}." />
        </attvalues>
      </edge>
      <edge source="2409.03257" target="2303.18223" id="9560">
        <attvalues>
          <attvalue for="5" value=" The rapid advancement of large language models (LLMs)~\cite{zhao2023survey} has led to the creation of various leaderboards designed to evaluate their performance across a wide range of tasks~\cite{alpaca_eval,lee2023holistic,HughesBae2023,bigcodeleaderboard,li2023halueval}." />
        </attvalues>
      </edge>
      <edge source="2409.03257" target="2311.04287" id="9561">
        <attvalues>
          <attvalue for="5" value=" The rapid advancement of large language models (LLMs)~\cite{zhao2023survey} has led to the creation of various leaderboards designed to evaluate their performance across a wide range of tasks~\cite{alpaca_eval,lee2023holistic,HughesBae2023,bigcodeleaderboard,li2023halueval}." />
        </attvalues>
      </edge>
      <edge source="2409.03257" target="2305.11747" id="9562">
        <attvalues>
          <attvalue for="5" value=" The rapid advancement of large language models (LLMs)~\cite{zhao2023survey} has led to the creation of various leaderboards designed to evaluate their performance across a wide range of tasks~\cite{alpaca_eval,lee2023holistic,HughesBae2023,bigcodeleaderboard,li2023halueval}." />
        </attvalues>
      </edge>
      <edge source="2409.03257" target="2405.20574" id="9563">
        <attvalues>
          <attvalue for="5" value=" While previous analyses of the Open Ko-LLM Leaderboard~\cite{park2024open} have provided valuable insights into LLM performance, they have been constrained observation periods of only five months, limiting their ability to capture long-term trends." />
        </attvalues>
      </edge>
      <edge source="2109.04711" target="1109.6341" id="9572">
        <attvalues>
          <attvalue for="5" value=" The conventional wisdom on semi-supervised learning and unsupervised domain adaptation is that labeled data is expensive; therefore, training on a combination of labeled and unlabeled data is an economical approach to improve performance when adapting to a new domain \cite{blum1998combining,daume2006domain,hoffman2018cycada,chen2020mixtext}." />
        </attvalues>
      </edge>
      <edge source="2109.04711" target="1711.03213" id="9573">
        <attvalues>
          <attvalue for="5" value=" The conventional wisdom on semi-supervised learning and unsupervised domain adaptation is that labeled data is expensive; therefore, training on a combination of labeled and unlabeled data is an economical approach to improve performance when adapting to a new domain \cite{blum1998combining,daume2006domain,hoffman2018cycada,chen2020mixtext}." />
        </attvalues>
      </edge>
      <edge source="2109.04711" target="2004.12239" id="9574">
        <attvalues>
          <attvalue for="5" value=" The conventional wisdom on semi-supervised learning and unsupervised domain adaptation is that labeled data is expensive; therefore, training on a combination of labeled and unlabeled data is an economical approach to improve performance when adapting to a new domain \cite{blum1998combining,daume2006domain,hoffman2018cycada,chen2020mixtext}." />
        </attvalues>
      </edge>
      <edge source="2109.04711" target="1904.02817" id="9575">
        <attvalues>
          <attvalue for="5" value=" Recent work has shown that pre-training in-domain Transformers is an effective method for unsupervised adaptation \cite{Han2019UnsupervisedDA,wright2020transformer} and even boosts performance when large quantities of in-domain data are available \cite{Gururangan2020DontSP}." />
        </attvalues>
      </edge>
      <edge source="2109.04711" target="2009.07806" id="9576">
        <attvalues>
          <attvalue for="5" value=" Recent work has shown that pre-training in-domain Transformers is an effective method for unsupervised adaptation \cite{Han2019UnsupervisedDA,wright2020transformer} and even boosts performance when large quantities of in-domain data are available \cite{Gururangan2020DontSP}." />
        </attvalues>
      </edge>
      <edge source="2109.04711" target="1906.02243" id="9579">
        <attvalues>
          <attvalue for="5" value=" However, modern pre-training methods incur substantial costs \cite{izsak2021train}, and generate carbon emissions \cite{Strubell2019EnergyAP,schwartz2019green,bender2021dangers}." />
        </attvalues>
      </edge>
      <edge source="2109.04711" target="1907.10597" id="9580">
        <attvalues>
          <attvalue for="5" value=" However, modern pre-training methods incur substantial costs \cite{izsak2021train}, and generate carbon emissions \cite{Strubell2019EnergyAP,schwartz2019green,bender2021dangers}." />
        </attvalues>
      </edge>
      <edge source="2109.04711" target="1907.11692" id="9582">
        <attvalues>
          <attvalue for="5" value=" Our analysis suggests that given current costs of pre-training large Transformer models, such as BERT \cite{devlin-etal-2019-bert}, and RoBERTa \cite{Liu2019RoBERTaAR}, in-domain data annotation should always be part of an economical strategy when adapting a single NLP system to a new domain." />
        </attvalues>
      </edge>
      <edge source="2212.10204" target="2005.09271" id="9583">
        <attvalues>
          <attvalue for="5" value=" Another successful AC approach utilizes phonetic posteriorgram (PPG) to characterize the phonetic pronunciation \cite{zhao2018icassp,li2020improving} that doesn't require parallel speech data." />
        </attvalues>
      </edge>
      <edge source="2402.15925" target="2004.04906" id="9584">
        <attvalues>
          <attvalue for="5" value=" Dense retrievers \cite{Karpukhin2020DensePR, izacard2022unsupervised, Hofstatter2021EfficientlyTA} are a standard component of retrieval augmented Question Answering (QA) \cite{RAG}, and other retrieval systems such as fact-checking~\cite{thorne2018fever}, argumentation \cite{wachsmuth-etal-2018-retrieval}, and others." />
        </attvalues>
      </edge>
      <edge source="2402.15925" target="2005.11401" id="9585">
        <attvalues>
          <attvalue for="5" value=" Retrievers are widespread, and are used in contexts that require trust: increasing factuality and decreasing hallucination \cite{shuster-etal-2021-retrieval-augmentation}, and providing trust and transparency \cite{lewis2020retrieval} via a source document that has provenance and can be examined." />
        </attvalues>
      </edge>
      <edge source="2402.15925" target="1803.05355" id="9586">
        <attvalues>
          <attvalue for="5" value=" Dense retrievers \cite{Karpukhin2020DensePR, izacard2022unsupervised, Hofstatter2021EfficientlyTA} are a standard component of retrieval augmented Question Answering (QA) \cite{RAG}, and other retrieval systems such as fact-checking~\cite{thorne2018fever}, argumentation \cite{wachsmuth-etal-2018-retrieval}, and others." />
        </attvalues>
      </edge>
      <edge source="2402.15925" target="2104.07567" id="9587">
        <attvalues>
          <attvalue for="5" value=" Retrievers are widespread, and are used in contexts that require trust: increasing factuality and decreasing hallucination \cite{shuster-etal-2021-retrieval-augmentation}, and providing trust and transparency \cite{lewis2020retrieval} via a source document that has provenance and can be examined." />
        </attvalues>
      </edge>
      <edge source="2402.15925" target="2106.11230" id="9588">
        <attvalues>
          <attvalue for="5" value=" The choice of pair affects feature suppression -- what is recoverable and what is not \cite{robinson2021can}." />
        </attvalues>
      </edge>
      <edge source="2402.15925" target="2106.16163" id="9589">
        <attvalues>
          <attvalue for="5" value=" So we extend this previous analytical work into the retrieval domain, by training 25 MultiContrievers initialised from MultiBert checkpoints~\cite{multiberts}." />
        </attvalues>
      </edge>
      <edge source="2402.15925" target="2003.12298" id="9590">
        <attvalues>
          <attvalue for="5" value=" We use information theoretic probing, also known as minimum description length (MDL) probing~\cite{voita-titov-2020-information}, to measure the information in MultiContriever representations." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2108.07258" id="9591">
        <attvalues>
          <attvalue for="5" value=" Contemporary natural language processing (NLP) relies heavily on pretrained language models, which are trained using large-scale unlabeled data~\cite{bommasani2021opportunities}." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2002.12327" id="9593">
        <attvalues>
          <attvalue for="5" value=" The MultiBerts release builds on top of a large body of work that seeks to analyze the behavior of Bert~\cite{rogers-etal-2020-primer}." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="1906.02243" id="9594">
        <attvalues>
          <attvalue for="5" value=" Because pre-training large language models is computationally expensive~\cite{strubell2019energy}, researchers often rely on the release of model checkpoints through libraries such as HuggingFace Transformers \cite{wolf-etal-2020-transformers}, which enable them to use large-scale language models without repeating the pre-training work." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="1806.00692" id="9596">
        <attvalues>
          <attvalue for="5" value=" Pre-training such models is an inherently stochastic process which depends on the initialization of the model's parameters and the ordering of training examples; for example, \cite{d2020underspecification} report substantial quantitative differences across multiple checkpoints of the same model architecture on several ``stress tests''~\cite{naik2018stress,mccoy2019right}." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2004.13606" id="9598">
        <attvalues>
          <attvalue for="5" value=" Understanding this difference is critical if we are to generate reusable insights about deep learning for NLP, and improve the state-of-the-art going forward~\cite{zhou2020curse,dodge2020fine,krishna2021how}." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2002.06305" id="9599">
        <attvalues>
          <attvalue for="5" value=" Understanding this difference is critical if we are to generate reusable insights about deep learning for NLP, and improve the state-of-the-art going forward~\cite{zhou2020curse,dodge2020fine,krishna2021how}." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2105.05641" id="9600">
        <attvalues>
          <attvalue for="5" value=" Understanding this difference is critical if we are to generate reusable insights about deep learning for NLP, and improve the state-of-the-art going forward~\cite{zhou2020curse,dodge2020fine,krishna2021how}." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2010.06032" id="9601">
        <attvalues>
          <attvalue for="5" value=" \item We illustrate the approach with a practical use case: we investigate the impact of counterfactual data augmentation on gender bias, in a Bert-based coreference resolution systems~\cite{webster2020measuring} (\S\ref{sec:experiments})." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2006.05987" id="9602">
        <attvalues>
          <attvalue for="5" value=" In addition to the studies of robustness cited above, several authors have introduced methods to reduce Bert's variability during fine-tuning~\cite{zhang2021revisiting, mosbach2021on, dodge2020fine, Lee2020Mixout, phang2018sentence}." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2104.07885" id="9603">
        <attvalues>
          <attvalue for="5" value=" Other authors have also studied the time dimension, which motivates our release of intermediate checkpoints \cite{liu2021probing, hao-etal-2020-investigating, saphra-lopez-2019-understanding, chiang-etal-2020-pretrained, dodge2020fine}." />
        </attvalues>
      </edge>
      <edge source="2106.16163" target="2104.00054" id="9604">
        <attvalues>
          <attvalue for="5" value=" In concurrent work, \cite{deutsch2021statistical} considered bootstrapping methods similar to the Multi-Bootstrap, in the context of summarization metrics evaluation." />
        </attvalues>
      </edge>
      <edge source="2106.07337" target="2004.06833" id="9605">
        <attvalues>
          <attvalue for="5" value=" Automated diagnosis of neurological diseases from the speech is receiving increased interest, as evidenced in the recent ADReSS-challenge \cite{luz2020alzheimer, luz2021detecting} which focuses on Alzheimer disease detection." />
        </attvalues>
      </edge>
      <edge source="2106.07337" target="1910.12590" id="9606">
        <attvalues>
          <attvalue for="5" value=" In \cite{kourkounakis2020detecting}, residual networks and BLSTMs are used for classifying different types of stuttering." />
        </attvalues>
      </edge>
      <edge source="2106.07337" target="1709.07902" id="9607">
        <attvalues>
          <attvalue for="5" value=" An FHVAE \cite{hsu2017unsupervised, shon2018unsupervised} models the generative process of a sequence of segments in a hierarchical structure." />
        </attvalues>
      </edge>
      <edge source="2106.07337" target="1706.03762" id="9608">
        <attvalues>
          <attvalue for="5" value=" To aggregate information over words or sentences, we compare simple averages or standard deviations or attention-based averages \cite{vaswani2017attention}, where we assume the attention mechanism can be trained to select those segments which show a high discrimination potential." />
        </attvalues>
      </edge>
      <edge source="2103.01273" target="1805.05089" id="9609">
        <attvalues>
          <attvalue for="5" value=" Follow-up work found that multiple datasets within the same language can also be combined by encoding their origin~\cite{stymne-etal-2018-parser,ustun-etal-2019-multi}, thereby implicitly learning useful commonalities, while still encoding dataset-specific knowledge." />
        </attvalues>
      </edge>
      <edge source="2103.01273" target="1809.02237" id="9610">
        <attvalues>
          <attvalue for="5" value=" A common strategy when the test data is drawn from a different distribution as the training datasets (zero-shot), is to use a manually assigned proxy treebank~\cite{smith-etal-2018-82,barry-etal-2019-cross,meechan-maddon-nivre-2019-parse}." />
        </attvalues>
      </edge>
      <edge source="2103.01273" target="1910.07938" id="9611">
        <attvalues>
          <attvalue for="5" value=" A common strategy when the test data is drawn from a different distribution as the training datasets (zero-shot), is to use a manually assigned proxy treebank~\cite{smith-etal-2018-82,barry-etal-2019-cross,meechan-maddon-nivre-2019-parse}." />
        </attvalues>
      </edge>
      <edge source="2103.01273" target="2005.00800" id="9612">
        <attvalues>
          <attvalue for="5" value=" Recent work showed that for unseen datasets in mono-lingual setups~\cite{wagner-etal-2020-treebank}, interpolated dataset embeddings can be used to improve performance for zero-shot settings." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="2009.03300" id="9613">
        <attvalues>
          <attvalue for="5" value=" For many of these general benchmarks \cite{hendrycks2020measuring, zhong2023agieval}, LLMs are prompted with custom instructions or in-context examples." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="2307.03172" id="9615">
        <attvalues>
          <attvalue for="5" value=" Additionally, LLMs also tend to forget or ignore information in long contexts~\cite{liu2023lost}, leading to potential accuracy drops even when the model can handle long input prompts." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="2005.11401" id="9616">
        <attvalues>
          <attvalue for="5" value=" While Retrieval-Augmented Generation (RAG) \cite{lewis2020retrieval} has been developed to address some of these challenges, it may sometimes retrieve irrelevant passages or documents, which can potentially degrade the generation performance." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="2106.09685" id="9617">
        <attvalues>
          <attvalue for="5" value=" With the emergence of Parameter Efficient Fine-tuning (PEFT)~\cite{hu2021lora, peft}, the computational resources required to fine-tune a task-specific LLM have decreased significantly." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="1901.11196" id="9618">
        <attvalues>
          <attvalue for="5" value=" For natural language processing (NLP) tasks, one can use approaches such as synonym replacement, character replacement (eg, by intentionally introducing spelling errors), random swapping, and back translation, just to name a few~\cite{wei2019eda, belinkov2017synthetic, coulombe2018text, zhang2018mixup}." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="2307.08701" id="9620">
        <attvalues>
          <attvalue for="5" value=" To address this, several recent papers have explored using an LLM to expand the fine-tuning dataset~\cite{dai2023auggpt, kumar2020data, zhou2023lima,chen2023alpagasus, cao2023instruction, wei2023instructiongpt4, zhu2023minigpt4}." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="2212.10560" id="9621">
        <attvalues>
          <attvalue for="5" value=" LLM2LLM achieves this by (1) fine-tuning a student LLM on the initial dataset, (2) evaluating on the training data and extracting data points which the model got incorrect after training, and (3) using a Self-Instruct \cite{wang-etal-2023-self-instruct} style data augmentation to augment these data points, which are then added back into the training data (Section~\ref{sec:llm2llm})." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="2110.14168" id="9622">
        <attvalues>
          <attvalue for="5" value=" \item We benchmark LLM2LLM on randomly sampled subsets of GSM8K \cite{cobbe2021gsm8k}, CaseHOLD \cite{zhengguha2021}, SNIPS \cite{coucke2018snips}, TREC \cite{li-roth-2002-learning} and SST-2 \cite{socher2013recursive} in order to evaluate the effectiveness of our approach in the low-data regime (Section~\ref{sec:main_results})." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="2104.08671" id="9623">
        <attvalues>
          <attvalue for="5" value=" \item We benchmark LLM2LLM on randomly sampled subsets of GSM8K \cite{cobbe2021gsm8k}, CaseHOLD \cite{zhengguha2021}, SNIPS \cite{coucke2018snips}, TREC \cite{li-roth-2002-learning} and SST-2 \cite{socher2013recursive} in order to evaluate the effectiveness of our approach in the low-data regime (Section~\ref{sec:main_results})." />
        </attvalues>
      </edge>
      <edge source="2403.15042" target="1805.10190" id="9624">
        <attvalues>
          <attvalue for="5" value=" \item We benchmark LLM2LLM on randomly sampled subsets of GSM8K \cite{cobbe2021gsm8k}, CaseHOLD \cite{zhengguha2021}, SNIPS \cite{coucke2018snips}, TREC \cite{li-roth-2002-learning} and SST-2 \cite{socher2013recursive} in order to evaluate the effectiveness of our approach in the low-data regime (Section~\ref{sec:main_results})." />
        </attvalues>
      </edge>
      <edge source="2204.02135" target="2104.03017" id="9626">
        <attvalues>
          <attvalue for="5" value="0, adding a fine-tune step has shown a noticeable accuracy improvement in MOS prediction over different speech datasets \cite{tseng2021utilizing, cooper2021generalization}." />
        </attvalues>
      </edge>
      <edge source="2204.02135" target="2110.02635" id="9627">
        <attvalues>
          <attvalue for="5" value=" Labelled speech datasets containing conferencing degradations (Tencent, IU Bloomington \cite{stupakov2009cosine,richey2018voices}, NISQA Corpus \cite{mittag2021nisqa}, and PSTN \cite{mittag2020dnn}) and synthesised speech (VoiceMOS \cite{cooper2021generalization}) were used to build the fine-tuning datasets and test the resulting models." />
        </attvalues>
      </edge>
      <edge source="2204.02135" target="2104.09494" id="9628">
        <attvalues>
          <attvalue for="5" value=" Labelled speech datasets containing conferencing degradations (Tencent, IU Bloomington \cite{stupakov2009cosine,richey2018voices}, NISQA Corpus \cite{mittag2021nisqa}, and PSTN \cite{mittag2020dnn}) and synthesised speech (VoiceMOS \cite{cooper2021generalization}) were used to build the fine-tuning datasets and test the resulting models." />
        </attvalues>
      </edge>
      <edge source="2204.02135" target="1804.05053" id="9630">
        <attvalues>
          <attvalue for="5" value=" Labelled speech datasets containing conferencing degradations (Tencent, IU Bloomington \cite{stupakov2009cosine,richey2018voices}, NISQA Corpus \cite{mittag2021nisqa}, and PSTN \cite{mittag2020dnn}) and synthesised speech (VoiceMOS \cite{cooper2021generalization}) were used to build the fine-tuning datasets and test the resulting models." />
        </attvalues>
      </edge>
      <edge source="2204.02135" target="2007.14598" id="9631">
        <attvalues>
          <attvalue for="5" value=" Labelled speech datasets containing conferencing degradations (Tencent, IU Bloomington \cite{stupakov2009cosine,richey2018voices}, NISQA Corpus \cite{mittag2021nisqa}, and PSTN \cite{mittag2020dnn}) and synthesised speech (VoiceMOS \cite{cooper2021generalization}) were used to build the fine-tuning datasets and test the resulting models." />
        </attvalues>
      </edge>
      <edge source="2401.12070" target="2301.11305" id="9632">
        <attvalues>
          <attvalue for="5" value=" Even with this strict limitation, our scheme still out-performs all open-source methods for ChatGPT detection and is competitive with or better than commercial APIs, despite these competitors using training samples from ChatGPT~\cite{mitchell_detectgpt_2023,verma_ghostbuster_2023}." />
        </attvalues>
      </edge>
      <edge source="2401.12070" target="2305.15047" id="9633">
        <attvalues>
          <attvalue for="5" value=" Even with this strict limitation, our scheme still out-performs all open-source methods for ChatGPT detection and is competitive with or better than commercial APIs, despite these competitors using training samples from ChatGPT~\cite{mitchell_detectgpt_2023,verma_ghostbuster_2023}." />
        </attvalues>
      </edge>
      <edge source="2401.12070" target="2210.07321" id="9634">
        <attvalues>
          <attvalue for="5" value=" These actors have a wide range of LLMs available to them beyond just ChatGPT, making zero-shot, model-agnostic detection critical for social media moderation and platform integrity assurance~\cite{crothers_machine_2022, bail_difficulty_2023}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1604.04562" id="9635">
        <attvalues>
          <attvalue for="5" value=" They correspond to the commonly defined natural language understanding, dialogue state tracking, and dialogue management modules \cite{wen2016network}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1706.01690" id="9637">
        <attvalues>
          <attvalue for="5" value=" They have been used in a wide range of applications, such as booking restaurants~\cite{wen2016network}, providing tourist information~\cite{budzianowski2018multiwoz,wu2019global}, ordering tickets~\cite{schulz2017frame}, and healthcare consultation~\cite{wei2018task}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1905.08743" id="9638">
        <attvalues>
          <attvalue for="5" value=" Most of the task-oriented dialogue systems nowadays, are benefited from transfer learning~\cite{WuTradeDST2019,lin2020mintl}, especially pre-trained language models trained on general text, such as BERT~\cite{devlin2018bert} and GPT2~\cite{radford2019language}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="2009.12005" id="9639">
        <attvalues>
          <attvalue for="5" value=" Most of the task-oriented dialogue systems nowadays, are benefited from transfer learning~\cite{WuTradeDST2019,lin2020mintl}, especially pre-trained language models trained on general text, such as BERT~\cite{devlin2018bert} and GPT2~\cite{radford2019language}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1910.07931" id="9641">
        <attvalues>
          <attvalue for="5" value=" However, previous work claims that linguistic patterns could differ between writing text and human conversation, resulting in a large gap of data distributions~\cite{bao2019plato,wolf2019transfertransfo}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1901.08149" id="9642">
        <attvalues>
          <attvalue for="5" value=" However, previous work claims that linguistic patterns could differ between writing text and human conversation, resulting in a large gap of data distributions~\cite{bao2019plato,wolf2019transfertransfo}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1911.03688" id="9643">
        <attvalues>
          <attvalue for="5" value=" \item ConveRT~\cite{henderson2019convert} and TOD-BERT-jnt~\cite{wu2020tod} have the highest classification accuracy and mutual information score, suggesting that response selection is useful for dialogue pre-training, especially when we compare TOD-BERT-jnt to TOD-BERT-mlm." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1911.00536" id="9644">
        <attvalues>
          <attvalue for="5" value=" Recently, several approaches are leveraging open-domain data~\cite{henderson2019convert,zhang2019dialogpt}, or aggregating task-oriented data~\cite{wu2020tod} to pre-train language models." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1608.04207" id="9646">
        <attvalues>
          <attvalue for="5" value=" Classifier probe is commonly used in different NLP tasks such as morphology~\cite{belinkov-etal-2017-neural}, sentence length~\cite{adi2016fine}, or linguistic structure~\cite{hewitt2019structural}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="2004.03061" id="9647">
        <attvalues>
          <attvalue for="5" value=" In addition, we present mutual information probe to investigate these language models by directly clustering their output representations, as recent study~\cite{pimentel2020information} suggests that a simple classifier may not be able to achieve the best estimate of mutual information between features and the downstream task." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1907.11692" id="9649">
        <attvalues>
          <attvalue for="5" value=" The distilled version of BERT surprisingly outperforms BERT and other strong baselines such as RoBERTa~\cite{liu2019roberta}." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="1909.11942" id="9650">
        <attvalues>
          <attvalue for="5" value=" \item Models such as AlBERT~\cite{lan2019albert} and ELECTRA~\cite{clark2020electra} have low classification accuracy and mutual information, showing the least useful information on task-oriented dialogue tasks." />
        </attvalues>
      </edge>
      <edge source="2010.13912" target="2003.10555" id="9651">
        <attvalues>
          <attvalue for="5" value=" \item Models such as AlBERT~\cite{lan2019albert} and ELECTRA~\cite{clark2020electra} have low classification accuracy and mutual information, showing the least useful information on task-oriented dialogue tasks." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2108.07258" id="9652">
        <attvalues>
          <attvalue for="5" value=" At the pre-training phase model gains a general understanding of language, including grammar rules, linguistic patterns, factual information, and reasoning abilities \cite{bommasani2022opportunities}." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2305.13230" id="9653">
        <attvalues>
          <attvalue for="5" value=" Ideally, the pre-training dataset should scale with the number of model parameters \cite{xue2023repeat}." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2001.08361" id="9654">
        <attvalues>
          <attvalue for="5" value=" \par The performance of LLMs is influenced by several crucial factors, including the number of model parameters, the number of observed tokens, and the overall quality of the text \cite{xue2023repeat} \cite{DBLP:journals/corr/abs-2001-08361}." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2307.09288" id="9655">
        <attvalues>
          <attvalue for="5" value=" LLaMA \cite{touvron2023llama}, when equipped with a LoRA adapter fine-tuned on medical texts, particularly outperforms foundational models in clinical domain tasks \cite{gema2023parameterefficient}." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2005.14165" id="9656">
        <attvalues>
          <attvalue for="5" value=" For comparison, Meta's LLama 2 was trained on 2 trillion tokens \cite{touvron2023llama} and GPT-3 on roughly 300 billion tokens \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="1911.02685" id="9657">
        <attvalues>
          <attvalue for="5" value=" This can also be done via transfer learning \cite{zhuang2020comprehensive} \cite{DBLP:journals/corr/abs-2004-10964}, fine-tuning LLM for Causal Language Modeling (predicting the next element in a sequence iteratively) \cite{wu2023metalearning} in a supervised manner on text in a language it has merely or never seen in a pre-training phase." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2310.05884" id="9659">
        <attvalues>
          <attvalue for="5" value=" This can also be done via transfer learning \cite{zhuang2020comprehensive} \cite{DBLP:journals/corr/abs-2004-10964}, fine-tuning LLM for Causal Language Modeling (predicting the next element in a sequence iteratively) \cite{wu2023metalearning} in a supervised manner on text in a language it has merely or never seen in a pre-training phase." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2307.03042" id="9660">
        <attvalues>
          <attvalue for="5" value=" The study \cite{gema2023parameterefficient} demonstrates that this approach yields substantial improvements, especially in large-scale multilabel classification tasks like diagnoses and procedures classification." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2306.03264" id="9661">
        <attvalues>
          <attvalue for="5" value=" For instance, studies have shown that Domain Adaptive Pre-training can significantly improve the performance of foundational LLMs in clinical tasks \cite{gema2023parameterefficient} \cite{karn-etal-2023-shs} \cite{chen2023meditron70b}." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2311.16079" id="9662">
        <attvalues>
          <attvalue for="5" value=" For instance, studies have shown that Domain Adaptive Pre-training can significantly improve the performance of foundational LLMs in clinical tasks \cite{gema2023parameterefficient} \cite{karn-etal-2023-shs} \cite{chen2023meditron70b}." />
        </attvalues>
      </edge>
      <edge source="2402.09759" target="2304.11771" id="9664">
        <attvalues>
          <attvalue for="5" value=" The introduction of LLM-based AI tools has demonstrated a marked improvement in operational efficiency, evidenced by a 14\% average increase in the rate of completed tasks per hour \cite{brynjolfsson2023generative}." />
        </attvalues>
      </edge>
      <edge source="2409.07737" target="1908.10084" id="9665">
        <attvalues>
          <attvalue for="5" value=" Text embeddings are widely used for tasks such as retrieval-augmented generation (RAG) and similar document retrieval~\cite{SentenceBERT,SimCSE,E5}." />
        </attvalues>
      </edge>
      <edge source="2409.07737" target="2104.08821" id="9666">
        <attvalues>
          <attvalue for="5" value=" Text embeddings are widely used for tasks such as retrieval-augmented generation (RAG) and similar document retrieval~\cite{SentenceBERT,SimCSE,E5}." />
        </attvalues>
      </edge>
      <edge source="2409.07737" target="2212.03533" id="9667">
        <attvalues>
          <attvalue for="5" value=" In recent years, the development of general-purpose text embedding models trained on diverse datasets has become increasingly common~\cite{E5,mE5,GTE,BGE,JinaBERT}." />
        </attvalues>
      </edge>
      <edge source="2409.07737" target="2402.05672" id="9668">
        <attvalues>
          <attvalue for="5" value=" In recent years, the development of general-purpose text embedding models trained on diverse datasets has become increasingly common~\cite{E5,mE5,GTE,BGE,JinaBERT}." />
        </attvalues>
      </edge>
      <edge source="2409.07737" target="2308.03281" id="9669">
        <attvalues>
          <attvalue for="5" value=" In recent years, the development of general-purpose text embedding models trained on diverse datasets has become increasingly common~\cite{E5,mE5,GTE,BGE,JinaBERT}." />
        </attvalues>
      </edge>
      <edge source="2409.07737" target="2309.07597" id="9670">
        <attvalues>
          <attvalue for="5" value=" In recent years, the development of general-purpose text embedding models trained on diverse datasets has become increasingly common~\cite{E5,mE5,GTE,BGE,JinaBERT}." />
        </attvalues>
      </edge>
      <edge source="2409.07737" target="2310.19923" id="9671">
        <attvalues>
          <attvalue for="5" value=" In recent years, the development of general-purpose text embedding models trained on diverse datasets has become increasingly common~\cite{E5,mE5,GTE,BGE,JinaBERT}." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="1708.06426" id="9673">
        <attvalues>
          <attvalue for="5" value=" Currently, there are two popular approaches widely used to leverage unpaired text for E2E ASR models: language model (LM) fusion~\cite{gulcehre2015using,sriram2017cold,toshniwal2018comparison,shan2019component} and re-scoring~\cite{chan2016listen}." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="1807.10857" id="9674">
        <attvalues>
          <attvalue for="5" value=" Currently, there are two popular approaches widely used to leverage unpaired text for E2E ASR models: language model (LM) fusion~\cite{gulcehre2015using,sriram2017cold,toshniwal2018comparison,shan2019component} and re-scoring~\cite{chan2016listen}." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2008.03822" id="9675">
        <attvalues>
          <attvalue for="5" value="~\cite{futami2020distilling} distill knowledge from the BERT output distribution to the output distribution of the ASR model." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2102.00291" id="9676">
        <attvalues>
          <attvalue for="5" value="~\cite{huang2021speech} fine-tune PLM as an ASR model with acoustics as cues." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2305.04160" id="9677">
        <attvalues>
          <attvalue for="5" value=" Apart from them, utilizing large-scale pre-trained language models (PLMs) to improve language modeling of ASR models~\cite{futami2020distilling,huang2021speech,chen2023xllm} is also a practical approach to make use of unpaired text dataset." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2202.07894" id="9678">
        <attvalues>
          <attvalue for="5" value=" Furthermore, the representation-based KD is applied to various ASR models~\cite{kubo2022knowledge,deng2022distill}." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="1910.14659" id="9679">
        <attvalues>
          <attvalue for="5" value=" The re-scorer based methods~\cite{shin2019effective,Salazar2020MaskedLM,Chiu2021InnovativeBR,Futami2021ASRRA,Xu2022RescoreBERTDS} convert PLMs into re-scorers and use them to re-score the $N$-best lists or lattices from the first-pass decoding, while not changing the ASR model." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2104.04950" id="9680">
        <attvalues>
          <attvalue for="5" value=" The re-scorer based methods~\cite{shin2019effective,Salazar2020MaskedLM,Chiu2021InnovativeBR,Futami2021ASRRA,Xu2022RescoreBERTDS} convert PLMs into re-scorers and use them to re-score the $N$-best lists or lattices from the first-pass decoding, while not changing the ASR model." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2110.01857" id="9681">
        <attvalues>
          <attvalue for="5" value=" The re-scorer based methods~\cite{shin2019effective,Salazar2020MaskedLM,Chiu2021InnovativeBR,Futami2021ASRRA,Xu2022RescoreBERTDS} convert PLMs into re-scorers and use them to re-score the $N$-best lists or lattices from the first-pass decoding, while not changing the ASR model." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2202.01094" id="9682">
        <attvalues>
          <attvalue for="5" value=" The re-scorer based methods~\cite{shin2019effective,Salazar2020MaskedLM,Chiu2021InnovativeBR,Futami2021ASRRA,Xu2022RescoreBERTDS} convert PLMs into re-scorers and use them to re-score the $N$-best lists or lattices from the first-pass decoding, while not changing the ASR model." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2101.06699" id="9683">
        <attvalues>
          <attvalue for="5" value=" Following~\cite{yi2021efficiently}, Zheng et al." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="1905.11235" id="9684">
        <attvalues>
          <attvalue for="5" value=" Unlike other E2E schemes, the continuous integrate-and-fire mechanism (CIF)\cite{dong2020cif}, which generates token-level acoustic representations aligned with the text, provides a natural option for the KD at the acoustic level." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2109.09161" id="9685">
        <attvalues>
          <attvalue for="5" value="~\cite{zheng2021wav} and Deng et al." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2201.10103" id="9686">
        <attvalues>
          <attvalue for="5" value="~\cite{deng2022model} integrate pre-trained acoustic and language models for low-resource ASR and non-autoregressive (NAR) ASR, respectively." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="1503.02531" id="9687">
        <attvalues>
          <attvalue for="5" value=" The KD-based methods transfer knowledge from PLMs to ASR models via knowledge distillation~\cite{Hinton2015DistillingTK}." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2102.07594" id="9688">
        <attvalues>
          <attvalue for="5" value=" Unlike the probability-based KD, the representation-based KD, which optimizes the similarity between teacher and student representations, transfers knowledge from PLMs to NAR ASR models~\cite{bai2021fast}." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2203.03582" id="9689">
        <attvalues>
          <attvalue for="5" value=" Furthermore, the representation-based KD is applied to various ASR models~\cite{kubo2022knowledge,deng2022distill}." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="1910.10699" id="9690">
        <attvalues>
          <attvalue for="5" value=" Inspired by contrastive knowledge distillation (CKD)\cite{tian2019contrastive,fu2021lrc}, we leverage contrastive loss to transfer the knowledge to the high-level acoustics of CIF-based ASR models." />
        </attvalues>
      </edge>
      <edge source="2301.13003" target="2012.07335" id="9691">
        <attvalues>
          <attvalue for="5" value=" Inspired by contrastive knowledge distillation (CKD)\cite{tian2019contrastive,fu2021lrc}, we leverage contrastive loss to transfer the knowledge to the high-level acoustics of CIF-based ASR models." />
        </attvalues>
      </edge>
      <edge source="1910.00795" target="1506.07503" id="9692">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end deep learning frameworks have shown impressive performances on many sequence-related tasks, such as ASR, MT, and TTS \cite{chorowski2015attentionasr,bahdanau2014nmt,wang2017tacotron}." />
        </attvalues>
      </edge>
      <edge source="1910.00795" target="1409.0473" id="9693">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end deep learning frameworks have shown impressive performances on many sequence-related tasks, such as ASR, MT, and TTS \cite{chorowski2015attentionasr,bahdanau2014nmt,wang2017tacotron}." />
        </attvalues>
      </edge>
      <edge source="1910.00795" target="1703.10135" id="9694">
        <attvalues>
          <attvalue for="5" value=" Recently, end-to-end deep learning frameworks have shown impressive performances on many sequence-related tasks, such as ASR, MT, and TTS \cite{chorowski2015attentionasr,bahdanau2014nmt,wang2017tacotron}." />
        </attvalues>
      </edge>
      <edge source="1910.00795" target="1703.08581" id="9695">
        <attvalues>
          <attvalue for="5" value=" \cite{weiss2017direct}, although it does not explicitly transcribe the speech into text in the source language, it also doesn’t require supervision from the groundtruth of the source language transcription during training." />
        </attvalues>
      </edge>
      <edge source="1910.00795" target="1802.06003" id="9696">
        <attvalues>
          <attvalue for="5" value=" then proved that this approach is possible for distant language pairs such as Japanese-to-English translation \cite{kano2017structured}." />
        </attvalues>
      </edge>
      <edge source="1910.00795" target="1904.06037" id="9697">
        <attvalues>
          <attvalue for="5" value=" \cite{jia2019direct} proposed the deep learning model that is trained end-to-end, which learns to map speech spectrograms into target spectrograms in another language that corresponds to the translated content (in a same or different canonical voice)." />
        </attvalues>
      </edge>
      <edge source="1910.00795" target="1712.04313" id="9698">
        <attvalues>
          <attvalue for="5" value=" The “Zero Resource Speech Challenge” s eries \cite{versteegh2015zero,dunbar2017zero,dunbar2019zero} was constructed to progress incrementally toward a system that learns an end-to-end spoken dialog (SD) system in an unknown language from scratch just using information available to language learning infants." />
        </attvalues>
      </edge>
      <edge source="2401.01128" target="2112.10741" id="9700">
        <attvalues>
          <attvalue for="5" value=" Recently, some text-to-image (T2I) synthesis methods, such as ~\cite{nichol2021glide, DingYHZZYLZSYT2021CogView, RombachBLEO22StableDiffusion, DingZHT2022CogView2, SahariaCSLWDGLA22imagen} have undergone significant advancements, particularly with the emergence of Large Language Models (LLM) and their enhancement in Large Vision Models (LVM), greatly enhancing the instruction-following capabilities of traditional T2I models." />
        </attvalues>
      </edge>
      <edge source="2401.01128" target="2012.15723" id="9702">
        <attvalues>
          <attvalue for="5" value=" LVM exhibits strong text comprehension~\cite{jiang2020can}, which brings a prompt engineering method~\cite{gao2020making} for better meet specific image generation requirements~\cite{oppenlaender2023taxonomy}." />
        </attvalues>
      </edge>
      <edge source="2401.01128" target="2311.06752" id="9704">
        <attvalues>
          <attvalue for="5" value=" Specifically, BeautifulPrompt~\cite{cao2023beautifulprompt} trains model using both low-quality and high-quality prompt pairs to enhances the quality of the generated images through Reinforcement Learning(RL)." />
        </attvalues>
      </edge>
      <edge source="2401.01128" target="2307.09036" id="9706">
        <attvalues>
          <attvalue for="5" value=" PromptMagician~\cite{feng2023promptmagician} is trained using a substantial amount of high-quality prompts, aiming to generate optimal prompts for Stable Diffusion." />
        </attvalues>
      </edge>
      <edge source="2401.01128" target="2309.14122" id="9707">
        <attvalues>
          <attvalue for="5" value=" This randomness may change the original semantics and introduce unsafe factors, raising safety concerns~\cite{ba2023surrogateprompt}." />
        </attvalues>
      </edge>
      <edge source="2401.01128" target="2303.08774" id="9708">
        <attvalues>
          <attvalue for="5" value=" Specifically, we first create a dataset sourced from various public text-only or text-image pairs datasets, which is then summarized and filtered by GPT-4~\cite{openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2402.10558" target="cs/0304006" id="9710">
        <attvalues>
          <attvalue for="5" value=" In particular, the task of rewriting text has been classified into three different categories \cite{madnani2010generating}: lexical paraphrasing, that is based or replacing words with other words with same meaning \cite{bolshakov2004synonymous}; phrasal paraphrasing, when the paraphrase is created acting on fragments with the same meaning \cite{ganitkevitch2013ppdb}; sentential paraphrasing when the paraphrases is performed at a sentence level, considering sentences with the same meaning \cite{barzilay2001extracting,barzilay2003learning,dolan2004unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2402.10558" target="1409.3215" id="9711">
        <attvalues>
          <attvalue for="5" value=" The success of using Deep Neural Networks in Statistical Machine Translation \cite{kalchbrenner2013recurrent}, in particular sequence-to-sequence architectures \cite{sutskever2014sequence}, suggests that the task of text paraphrasing can be addressed using similar approaches." />
        </attvalues>
      </edge>
      <edge source="2402.10558" target="1603.06393" id="9712">
        <attvalues>
          <attvalue for="5" value=" In particular, we focus on a Pointer network \cite{gu2016incorporating,vaswani2017attention,see2017get}) that, given an input text, learns to generate a possible target paraphrase." />
        </attvalues>
      </edge>
      <edge source="2402.10558" target="1706.03762" id="9713">
        <attvalues>
          <attvalue for="5" value=" In particular, we focus on a Pointer network \cite{gu2016incorporating,vaswani2017attention,see2017get}) that, given an input text, learns to generate a possible target paraphrase." />
        </attvalues>
      </edge>
      <edge source="2402.10558" target="1704.04368" id="9714">
        <attvalues>
          <attvalue for="5" value=" In particular, we focus on a Pointer network \cite{gu2016incorporating,vaswani2017attention,see2017get}) that, given an input text, learns to generate a possible target paraphrase." />
        </attvalues>
      </edge>
      <edge source="2201.01337" target="2010.12309" id="9729">
        <attvalues>
          <attvalue for="5" value=" This scenario has contributed to the rise of Low-Resource NLP, which aims to develop techniques to deal with low data availability in a specific language or application domain \cite{hedderich2020survey}." />
        </attvalues>
      </edge>
      <edge source="2201.01337" target="1909.00161" id="9730">
        <attvalues>
          <attvalue for="5" value=" Current approaches to the zero-shot text classification task ({0shot-TC}) make use of the good performance that Transformers have demonstrated in text entailment tasks \cite{yin2019benchmarking}." />
        </attvalues>
      </edge>
      <edge source="2201.01337" target="2004.05150" id="9733">
        <attvalues>
          <attvalue for="5" value=" While there are transformer-based solutions to these problems individually \cite{beltagy2020longformer,sanh2019distilbert,Zaheer2020BigBT}, to the best of our knowledge, there is no solution that addresses both, nor even in the context of 0shot-TC." />
        </attvalues>
      </edge>
      <edge source="2201.01337" target="2007.14062" id="9735">
        <attvalues>
          <attvalue for="5" value=" While there are transformer-based solutions to these problems individually \cite{beltagy2020longformer,sanh2019distilbert,Zaheer2020BigBT}, to the best of our knowledge, there is no solution that addresses both, nor even in the context of 0shot-TC." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="1911.02116" id="9737">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="1508.05326" id="9739">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="1704.05426" id="9740">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="1910.07475" id="9741">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="2003.05002" id="9742">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="2011.03080" id="9743">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="1904.09077" id="9744">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="1902.00193" id="9745">
        <attvalues>
          <attvalue for="5" value=" Multilingual pre-trained language models (aka M-PLMs) such as mBERT ~\cite{devlin-etal-2019-bert}, XLM ~\cite{conneau2019unsupervised} and XLM-R ~\cite{conneau-etal-2020-unsupervised} have achieved significant improvement for many multilingual tasks, including multilingual NLI~\cite{conneau-etal-2018-xnli,bowman-etal-2015-large,williams-etal-2018-broad}, question answering~\cite{lewis2019mlqa,clark-etal-2020-tydi,hardalov2020exams} and NER~\cite{wu-dredze-2019-beto,rahimi-etal-2019-massively}." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="1909.10351" id="9746">
        <attvalues>
          <attvalue for="5" value=" As pre-trained language models are usually computationally expensive, many transformer distillation methods~\cite{jiao-etal-2020-tinybert,liu2020cross} have been proposed, which distill knowledge from a large teacher model to a lightweight student network, to accelerate inference and reduce model size while maintaining the accuracy." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="2010.14271" id="9747">
        <attvalues>
          <attvalue for="5" value=" As pre-trained language models are usually computationally expensive, many transformer distillation methods~\cite{jiao-etal-2020-tinybert,liu2020cross} have been proposed, which distill knowledge from a large teacher model to a lightweight student network, to accelerate inference and reduce model size while maintaining the accuracy." />
        </attvalues>
      </edge>
      <edge source="2305.07928" target="2005.10450" id="9748">
        <attvalues>
          <attvalue for="5" value=" The majority of works mainly focus on learning from a single teacher as in Figure~\ref{fig:pipeline} (a), while only a few studies have considered to learn from multiple teachers~\cite{peng2020mtss,liu2020adaptive} which allows to select the optimal model as teacher for different domains during the student training." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1602.06023" id="9751">
        <attvalues>
          <attvalue for="5" value=" A major weakness of traditional sequence-to-sequence learning when applied to summarization is the lack of a direct copy mechanism, leading to missing or misrepresented details in decoded summaries~\cite{chopra2016abstractive,nallapati2016abstractive,rush2015neural,zeng2016efficient}." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1409.3215" id="9752">
        <attvalues>
          <attvalue for="5" value=" These approaches usually rely on a sequence-to-sequence~\cite{sutskever2014sequence} style architecture, and tend to produce fluent, well formed natural language summaries when coupled with beam search or other decoding techniques." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1611.03382" id="9753">
        <attvalues>
          <attvalue for="5" value=" Though attention helps ameliorate this issue by directly learning to focus on specific words or phrases in a source document~\cite{chopra2016abstractive}, many have allowed for an explicit copy mechanism inspired by Pointer Networks~\cite{vinyals2015pointer}, by optimizing a differentiable decision whether to generate new text or directly copy from the source~\cite{gu2016incorporating,zeng2016efficient,see2017get}." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1506.03134" id="9754">
        <attvalues>
          <attvalue for="5" value=" Though attention helps ameliorate this issue by directly learning to focus on specific words or phrases in a source document~\cite{chopra2016abstractive}, many have allowed for an explicit copy mechanism inspired by Pointer Networks~\cite{vinyals2015pointer}, by optimizing a differentiable decision whether to generate new text or directly copy from the source~\cite{gu2016incorporating,zeng2016efficient,see2017get}." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1603.06393" id="9755">
        <attvalues>
          <attvalue for="5" value=" Though attention helps ameliorate this issue by directly learning to focus on specific words or phrases in a source document~\cite{chopra2016abstractive}, many have allowed for an explicit copy mechanism inspired by Pointer Networks~\cite{vinyals2015pointer}, by optimizing a differentiable decision whether to generate new text or directly copy from the source~\cite{gu2016incorporating,zeng2016efficient,see2017get}." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1704.04368" id="9756">
        <attvalues>
          <attvalue for="5" value=" Additional components in many neural abstractive summarization systems model semantic coverage~\cite{tu2016modeling,see2017get} and provide guidance on where to attend~\cite{gehrmann2018bottom} in order to directly avoid repetition and ancillary details, while encouraging completeness." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1601.04811" id="9757">
        <attvalues>
          <attvalue for="5" value=" Additional components in many neural abstractive summarization systems model semantic coverage~\cite{tu2016modeling,see2017get} and provide guidance on where to attend~\cite{gehrmann2018bottom} in order to directly avoid repetition and ancillary details, while encouraging completeness." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1808.10792" id="9758">
        <attvalues>
          <attvalue for="5" value=" Additional components in many neural abstractive summarization systems model semantic coverage~\cite{tu2016modeling,see2017get} and provide guidance on where to attend~\cite{gehrmann2018bottom} in order to directly avoid repetition and ancillary details, while encouraging completeness." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1705.04304" id="9759">
        <attvalues>
          <attvalue for="5" value=" Recent work has incorporated the use of reinforcement learning to directly optimize objectives of interest that may not be differentiable, but are nonetheless useful for summarization, such as directly optimizing the ROUGE score~\cite{paulus2017deep,li2018actor,celikyilmaz2018deep}." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1803.11070" id="9760">
        <attvalues>
          <attvalue for="5" value=" Recent work has incorporated the use of reinforcement learning to directly optimize objectives of interest that may not be differentiable, but are nonetheless useful for summarization, such as directly optimizing the ROUGE score~\cite{paulus2017deep,li2018actor,celikyilmaz2018deep}." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1803.10357" id="9761">
        <attvalues>
          <attvalue for="5" value=" Recent work has incorporated the use of reinforcement learning to directly optimize objectives of interest that may not be differentiable, but are nonetheless useful for summarization, such as directly optimizing the ROUGE score~\cite{paulus2017deep,li2018actor,celikyilmaz2018deep}." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1802.05365" id="9762">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, \cite{peters2018deep}, \cite{Devlin2018BERTPO}, \cite{howard2018universal}, \cite{radford2018improving}, and \cite{radford2019language}, among others, have shown the benefits of large-scale pretraining on large, unlabeled corpora on a variety of downstream tasks in transfer learning settings." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1801.06146" id="9764">
        <attvalues>
          <attvalue for="5" value=" Simultaneously, \cite{peters2018deep}, \cite{Devlin2018BERTPO}, \cite{howard2018universal}, \cite{radford2018improving}, and \cite{radford2019language}, among others, have shown the benefits of large-scale pretraining on large, unlabeled corpora on a variety of downstream tasks in transfer learning settings." />
        </attvalues>
      </edge>
      <edge source="1909.00325" target="1801.10198" id="9765">
        <attvalues>
          <attvalue for="5" value=" In particular, it has been shown that large-scale, attention-only language modeling via decoder-only transformers~\cite{decoder-transformer} as an unsupervised pretraining task admits the ability to perform zero-shot learning on meaningful tasks involving natural language generation~\cite{radford2019language}." />
        </attvalues>
      </edge>
      <edge source="2209.12944" target="2011.13205" id="9766">
        <attvalues>
          <attvalue for="5" value=" Despite the popularity of voice assistants among users globally and the advancements in spoken-language understanding \cite{DBLP:conf/emnlp/BastianelliVSR20, DBLP:journals/corr/abs-2204-08582}, there are surprisingly limited efforts in studying spoken QA and its limitations." />
        </attvalues>
      </edge>
      <edge source="2209.12944" target="1804.00320" id="9767">
        <attvalues>
          <attvalue for="5" value=" The majority of research focuses on reading comprehension as a component of spoken QA \cite{DBLP:conf/interspeech/LeeWLL18, DBLP:conf/emnlp/FaisalKAA21, DBLP:conf/eacl/RavichanderDRMH21}." />
        </attvalues>
      </edge>
      <edge source="2209.12944" target="2102.08345" id="9768">
        <attvalues>
          <attvalue for="5" value="~\cite{DBLP:conf/eacl/RavichanderDRMH21} showed that ASR noise not only dramatically affects the performance of transformer-based reading comprehension models but also that it is a more challenging type of noise compared to the noise generated from keyboard mistyping or faulty machine translation." />
        </attvalues>
      </edge>
      <edge source="2209.12944" target="2109.12072" id="9769">
        <attvalues>
          <attvalue for="5" value="~\cite{DBLP:conf/emnlp/FaisalKAA21} showed that background differences in users, such as their accent, can affect the performance of reading comprehension models differently." />
        </attvalues>
      </edge>
      <edge source="2209.12944" target="2204.00716" id="9770">
        <attvalues>
          <attvalue for="5" value=" Zhuang and Zuccon \cite{DBLP:journals/corr/abs-2204-00716} increased the robustness of dense retrievers against typos by replacing the extremely sensitive to typos WordPiece tokenizer with the Character-CNN module and further combined it with a knowledge distillation method." />
        </attvalues>
      </edge>
      <edge source="2209.12944" target="2205.02303" id="9772">
        <attvalues>
          <attvalue for="5" value=" Sidiropoulos and Kanoulas \cite{DBLP:conf/sigir/SidiropoulosK22} showcased the dense retrievers' lack of robustness to typos in the question and proposed a combination of data augmentation with a contrastive loss to robustify the model." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="1502.05698" id="9773">
        <attvalues>
          <attvalue for="5" value=" There exist many datasets that aim to evaluate these abilities \cite[eg,][]{walker2006ace,pradhan-etal-2012-conll,rahman-ng-2012-resolving,weston2015towards,chen-etal-2018-preco,bamman-etal-2020-annotated,uryupina2020annotating} and many NLP models that aim to solve these tasks \cite[eg,][]{haghighi-klein-2010-coreference,lee-etal-2011-stanfords,hill2016goldilocks,henaff2017tracking,ji-etal-2017-dynamic,lee-etal-2017-end,bosselut2018simulating,gupta-durrett-2019-effective,gupta-durrett-2019-tracking,aina-etal-2019-entity,toshniwal-etal-2020-learning,wu-etal-2020-corefqa}." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="1912.01140" id="9775">
        <attvalues>
          <attvalue for="5" value=" There exist many datasets that aim to evaluate these abilities \cite[eg,][]{walker2006ace,pradhan-etal-2012-conll,rahman-ng-2012-resolving,weston2015towards,chen-etal-2018-preco,bamman-etal-2020-annotated,uryupina2020annotating} and many NLP models that aim to solve these tasks \cite[eg,][]{haghighi-klein-2010-coreference,lee-etal-2011-stanfords,hill2016goldilocks,henaff2017tracking,ji-etal-2017-dynamic,lee-etal-2017-end,bosselut2018simulating,gupta-durrett-2019-effective,gupta-durrett-2019-tracking,aina-etal-2019-entity,toshniwal-etal-2020-learning,wu-etal-2020-corefqa}." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="1511.02301" id="9776">
        <attvalues>
          <attvalue for="5" value=" There exist many datasets that aim to evaluate these abilities \cite[eg,][]{walker2006ace,pradhan-etal-2012-conll,rahman-ng-2012-resolving,weston2015towards,chen-etal-2018-preco,bamman-etal-2020-annotated,uryupina2020annotating} and many NLP models that aim to solve these tasks \cite[eg,][]{haghighi-klein-2010-coreference,lee-etal-2011-stanfords,hill2016goldilocks,henaff2017tracking,ji-etal-2017-dynamic,lee-etal-2017-end,bosselut2018simulating,gupta-durrett-2019-effective,gupta-durrett-2019-tracking,aina-etal-2019-entity,toshniwal-etal-2020-learning,wu-etal-2020-corefqa}." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="1707.07045" id="9779">
        <attvalues>
          <attvalue for="5" value=" There exist many datasets that aim to evaluate these abilities \cite[eg,][]{walker2006ace,pradhan-etal-2012-conll,rahman-ng-2012-resolving,weston2015towards,chen-etal-2018-preco,bamman-etal-2020-annotated,uryupina2020annotating} and many NLP models that aim to solve these tasks \cite[eg,][]{haghighi-klein-2010-coreference,lee-etal-2011-stanfords,hill2016goldilocks,henaff2017tracking,ji-etal-2017-dynamic,lee-etal-2017-end,bosselut2018simulating,gupta-durrett-2019-effective,gupta-durrett-2019-tracking,aina-etal-2019-entity,toshniwal-etal-2020-learning,wu-etal-2020-corefqa}." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="1909.02635" id="9781">
        <attvalues>
          <attvalue for="5" value=" There exist many datasets that aim to evaluate these abilities \cite[eg,][]{walker2006ace,pradhan-etal-2012-conll,rahman-ng-2012-resolving,weston2015towards,chen-etal-2018-preco,bamman-etal-2020-annotated,uryupina2020annotating} and many NLP models that aim to solve these tasks \cite[eg,][]{haghighi-klein-2010-coreference,lee-etal-2011-stanfords,hill2016goldilocks,henaff2017tracking,ji-etal-2017-dynamic,lee-etal-2017-end,bosselut2018simulating,gupta-durrett-2019-effective,gupta-durrett-2019-tracking,aina-etal-2019-entity,toshniwal-etal-2020-learning,wu-etal-2020-corefqa}." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="1904.03518" id="9782">
        <attvalues>
          <attvalue for="5" value=" There exist many datasets that aim to evaluate these abilities \cite[eg,][]{walker2006ace,pradhan-etal-2012-conll,rahman-ng-2012-resolving,weston2015towards,chen-etal-2018-preco,bamman-etal-2020-annotated,uryupina2020annotating} and many NLP models that aim to solve these tasks \cite[eg,][]{haghighi-klein-2010-coreference,lee-etal-2011-stanfords,hill2016goldilocks,henaff2017tracking,ji-etal-2017-dynamic,lee-etal-2017-end,bosselut2018simulating,gupta-durrett-2019-effective,gupta-durrett-2019-tracking,aina-etal-2019-entity,toshniwal-etal-2020-learning,wu-etal-2020-corefqa}." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="1905.06649" id="9783">
        <attvalues>
          <attvalue for="5" value=" There exist many datasets that aim to evaluate these abilities \cite[eg,][]{walker2006ace,pradhan-etal-2012-conll,rahman-ng-2012-resolving,weston2015towards,chen-etal-2018-preco,bamman-etal-2020-annotated,uryupina2020annotating} and many NLP models that aim to solve these tasks \cite[eg,][]{haghighi-klein-2010-coreference,lee-etal-2011-stanfords,hill2016goldilocks,henaff2017tracking,ji-etal-2017-dynamic,lee-etal-2017-end,bosselut2018simulating,gupta-durrett-2019-effective,gupta-durrett-2019-tracking,aina-etal-2019-entity,toshniwal-etal-2020-learning,wu-etal-2020-corefqa}." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="1910.13461" id="9787">
        <attvalues>
          <attvalue for="5" value=" Using a probing classifier, they found that the states can be decoded from T5 \cite{raffel2020exploring} and BART \cite{lewis-etal-2020-bart} with high accuracy." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="2005.14165" id="9788">
        <attvalues>
          <attvalue for="5" value=" We use this novel task to evaluate GPT-3 \cite{brown2020language}, \href{ and Flan-T5 \cite{chung2022scaling} without any finetuning." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="2210.11416" id="9789">
        <attvalues>
          <attvalue for="5" value=" We use this novel task to evaluate GPT-3 \cite{brown2020language}, \href{ and Flan-T5 \cite{chung2022scaling} without any finetuning." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="2210.13382" id="9790">
        <attvalues>
          <attvalue for="5" value=" More broadly, our task can also serve as a useful tool for investigations into emergent world models in LMs \cite[eg,][]{li2023emergent,tsai-etal-2023-large}." />
        </attvalues>
      </edge>
      <edge source="2305.02363" target="2304.02868" id="9791">
        <attvalues>
          <attvalue for="5" value=" More broadly, our task can also serve as a useful tool for investigations into emergent world models in LMs \cite[eg,][]{li2023emergent,tsai-etal-2023-large}." />
        </attvalues>
      </edge>
      <edge source="2402.09954" target="2206.04615" id="9792">
        <attvalues>
          <attvalue for="5" value=" Current chatbots based on LLMs have superior performance on question answering, polishing documents, etc~\cite{srivastava2022beyond}." />
        </attvalues>
      </edge>
      <edge source="2402.09954" target="2212.03699" id="9793">
        <attvalues>
          <attvalue for="5" value=" For instance, optimizing persona dialogue generation is of great significance for empathetic and medical chatbots~\cite{de2020effectiveness,liu2022persona}, and it can bring trust to users~\cite{huang2023personalized,de2020effectiveness}." />
        </attvalues>
      </edge>
      <edge source="2402.09954" target="2210.15088" id="9794">
        <attvalues>
          <attvalue for="5" value=" On the other hand, fine-tuning a high-quality persona-based dialogue model usually requires a proprietary dataset, and the cost of manually writing dialogues is very high~\cite{cao2022model,huang2023personalized}." />
        </attvalues>
      </edge>
      <edge source="2402.09954" target="2204.09867" id="9795">
        <attvalues>
          <attvalue for="5" value=" On the other hand, fine-tuning a high-quality persona-based dialogue model usually requires a proprietary dataset, and the cost of manually writing dialogues is very high~\cite{cao2022model,huang2023personalized}." />
        </attvalues>
      </edge>
      <edge source="2310.05657" target="2112.00861" id="9819">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) trained with task instructions and human feedback can follow natural language instructions to complete a task~\cite{askell2021general,sanh2022multitask,wei2022finetuned,ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2310.05657" target="2110.08207" id="9820">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) trained with task instructions and human feedback can follow natural language instructions to complete a task~\cite{askell2021general,sanh2022multitask,wei2022finetuned,ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2310.05657" target="2109.01652" id="9821">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) trained with task instructions and human feedback can follow natural language instructions to complete a task~\cite{askell2021general,sanh2022multitask,wei2022finetuned,ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2310.05657" target="2302.07736" id="9824">
        <attvalues>
          <attvalue for="5" value=" Recently, the instruction-following ability of LLMs makes them promising candidates for automatic evaluation~\cite{chiang-lee-2023-large,liu2023gpteval, wang2023chatgpt,huang2023chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2306.04009" target="1909.01066" id="9827">
        <attvalues>
          <attvalue for="5" value=" Contemporary pre-trained language models (LMs) such as BERT \cite{devlin-etal-2019-bert} and T5 \cite{raffel2020exploring} have been shown to be adept at encoding factual knowledge \cite{petroni-etal-2019-language, zhong-etal-2021-factual, roberts-etal-2020-much}, an ability that can be further boosted by explicitly integrating them with knowledge about entities and relations \cite[i." />
        </attvalues>
      </edge>
      <edge source="2306.04009" target="2006.10413" id="9828">
        <attvalues>
          <attvalue for="5" value=" At the same time, these LMs often struggle to compose the knowledge they encode \cite{kassner-etal-2020-pretrained, talmor-etal-2020-olmpics, moiseev-etal-2022-skill}, and therefore do not satisfy C2." />
        </attvalues>
      </edge>
      <edge source="2306.04009" target="2205.12538" id="9829">
        <attvalues>
          <attvalue for="5" value=" However, such methods require training entirely separate models, or make use of human-annotations \cite{patel2022question}." />
        </attvalues>
      </edge>
      <edge source="2306.04009" target="2104.06599" id="9830">
        <attvalues>
          <attvalue for="5" value=" {We then adapt the knowledge integrated T5 models by training soft prompts \cite{qin-eisner-2021-learning, lester-etal-2021-power} on random walks over the structured knowledge that they have encoded, and devise two methods that trigger this ability in the LMs given a multi-hop question as input." />
        </attvalues>
      </edge>
      <edge source="2306.04009" target="2104.08691" id="9832">
        <attvalues>
          <attvalue for="5" value=" Briefly, on \txxl our methods show improvements over previously proposed prompt-tuning approaches \cite{lester-etal-2021-power, vu-etal-2022-spot} as well as full model fine-tuning, with \pth and \mixture demonstrating gains of $\sim$16 and $\sim$9." />
        </attvalues>
      </edge>
      <edge source="2404.01453" target="1707.07343" id="9834">
        <attvalues>
          <attvalue for="5" value=" Efforts have been devoted to developing methodologies for effective temporal relation extraction \cite{choubey-huang-2017-sequential, ning2018cogcomptime, ning2019improved, wang-etal-2020-joint, zhang-etal-2022-extracting}, along with initiatives to create benchmark datasets with a temporal focus \cite{TBcorpus, verhagen-etal-2010-Tempeval2, ning2018multi, zhou2021temporal, gantt2022decomposing}." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="1909.10705" id="9835">
        <attvalues>
          <attvalue for="5" value=" Recent breakthroughs in natural language processing (NLP) and natural language generation (NLG) have revitalized interest in applying computational methods to story \cite{see2019massively,xu2020megatroncntrl,nichols2020collaborative,fang2021transformerbased,hazarika2021zeroshot}." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="2011.10208" id="9837">
        <attvalues>
          <attvalue for="5" value=" Recent breakthroughs in natural language processing (NLP) and natural language generation (NLG) have revitalized interest in applying computational methods to story \cite{see2019massively,xu2020megatroncntrl,nichols2020collaborative,fang2021transformerbased,hazarika2021zeroshot}." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="2101.00828" id="9838">
        <attvalues>
          <attvalue for="5" value=" Recent breakthroughs in natural language processing (NLP) and natural language generation (NLG) have revitalized interest in applying computational methods to story \cite{see2019massively,xu2020megatroncntrl,nichols2020collaborative,fang2021transformerbased,hazarika2021zeroshot}." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="2106.06411" id="9839">
        <attvalues>
          <attvalue for="5" value=" Recent breakthroughs in natural language processing (NLP) and natural language generation (NLG) have revitalized interest in applying computational methods to story \cite{see2019massively,xu2020megatroncntrl,nichols2020collaborative,fang2021transformerbased,hazarika2021zeroshot}." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="2005.14165" id="9840">
        <attvalues>
          <attvalue for="5" value=" Even with modern deep learning techniques this is a significant challenge, as people expect stories to be consistent and coherent, two things that transformers are not particularly good at doing across long passages \cite{gpt3,yao2019plan}." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="1812.04948" id="9842">
        <attvalues>
          <attvalue for="5" value=" In other domains such as image generation \cite{karras2019style, patashnik2021styleclip, galanos2021affectgan}, strategic game-playing \cite{alphazero, alphago}, and planning~\cite{karkus2017qmdp,fan2019automatic}, powerful models for evaluating objects of interest have lead to more powerful models for generating them." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="1906.00318" id="9845">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, methods for automatically evaluating stories, such as ROUGE \cite{lin2004rouge}, are extremely limited in their ability to accurately assess models and are easily Goodharted \cite{belz2006comparing,cohan2016revisiting,schluter2017limits,schluter2016approximate,eyal2019question}." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="2010.01717" id="9846">
        <attvalues>
          <attvalue for="5" value=" Recently, researchers interested in story evaluation have developed more sophisticated techniques for evaluating stories \cite{akoury2020storium,dou2021scarecrow} as well as the Purdy Index \cite{purdy2018predicting} and the Fabula-Entropy Index \cite{castricato2021formal,castricato2021fabula}." />
        </attvalues>
      </edge>
      <edge source="2110.03111" target="2104.07472" id="9847">
        <attvalues>
          <attvalue for="5" value=" Recently, researchers interested in story evaluation have developed more sophisticated techniques for evaluating stories \cite{akoury2020storium,dou2021scarecrow} as well as the Purdy Index \cite{purdy2018predicting} and the Fabula-Entropy Index \cite{castricato2021formal,castricato2021fabula}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="2401.00368" id="9849">
        <attvalues>
          <attvalue for="5" value=" Furthermore, an efficient technique for creating high-quality text embeddings using synthetic data and minimal training, avoiding complex pipelines and extensive labeled datasets, and achieving top results on key benchmarks when mixed with labeled data \cite{wang2023improving}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="1801.06146" id="9853">
        <attvalues>
          <attvalue for="5" value=" Various techniques have been proposed to improve the performance of embedding models, such as fine-tuning on domain-specific data \cite{howard2018universal}, using ensemble methods, and incorporating external knowledge sources \cite{zhang2019ernie}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="2005.14165" id="9856">
        <attvalues>
          <attvalue for="5" value=" Large language models have been successfully applied to a wide range of NLP tasks, such as text generation \cite{radford2019language}, question answering \cite{raffel2020exploring}, and sentiment analysis \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="1805.06201" id="9857">
        <attvalues>
          <attvalue for="5" value=" For example, a method for contextual augmentation of text data using a bidirectional language model is being proposed \cite{kobayashi2018contextual}, while a retrieval-augmented generation approach for improving the factual accuracy of generated text was also introduced \cite{guu2020retrieval}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="2310.04408" id="9858">
        <attvalues>
          <attvalue for="5" value=" For instance, RECOMP proposes compressing retrieved documents into summaries before integrating them with language models, aiming to reduce computational costs and help LMs identify relevant information more efficiently \cite{xu2023recomp}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="2310.15556" id="9859">
        <attvalues>
          <attvalue for="5" value=" Similarly, TCRA-LLM introduces a token compression scheme for retrieval-augmented LLMs, employing summarization and semantic compression techniques to reduce inference costs \cite{liu2023tcra}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="2312.05708" id="9860">
        <attvalues>
          <attvalue for="5" value=" Context Tuning for RAG addresses the limitation of RAG's tool retrieval step by employing a smart context retrieval system to fetch relevant information, improving the efficiency and effectiveness of the generation process \cite{anantha2023context}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="2402.18700" id="9861">
        <attvalues>
          <attvalue for="5" value=" The Natural Language Prompt Encapsulation (Nano-Capsulator) framework compresses original prompts into NL formatted Capsule Prompts while maintaining prompt utility and transferability \cite{chuang2024learning}." />
        </attvalues>
      </edge>
      <edge source="2404.12283" target="2305.11186" id="9862">
        <attvalues>
          <attvalue for="5" value=" Compress-Then-Prompt [18] indicates that the generation quality in a compressed LLM can be markedly improved for specific queries by selecting prompts with high efficiency and accuracy trade-offs \cite{xu2023compress}." />
        </attvalues>
      </edge>
      <edge source="2209.04156" target="2101.08091" id="9863">
        <attvalues>
          <attvalue for="5" value=" The two tasks are defined to identify intents and extract semantic components from utterances in dialog systems~\cite{DBLP:journals/corr/abs-2101-08091}." />
        </attvalues>
      </edge>
      <edge source="2209.04156" target="1609.01454" id="9864">
        <attvalues>
          <attvalue for="5" value=" Previous joint learning methods utilized the supervised signal from intent detection to improve the performance of slot filling by attention~\cite{DBLP:conf/interspeech/LiuL16} or gated~\cite{DBLP:conf/naacl/GooGHHCHC18} mechanisms." />
        </attvalues>
      </edge>
      <edge source="2209.04156" target="2010.03880" id="9865">
        <attvalues>
          <attvalue for="5" value="~\cite{DBLP:conf/icassp/QinLCKZ021} proposed a co-interactive module to model the cross-impact of two tasks and achieved the state-of-the-art performance." />
        </attvalues>
      </edge>
      <edge source="2209.04156" target="1812.09471" id="9866">
        <attvalues>
          <attvalue for="5" value=" The joint learning methods always demonstrate their effectiveness over the independent models~\cite{DBLP:conf/interspeech/Hakkani-TurTCCG16,DBLP:conf/acl/ZhangLDFY19,DBLP:conf/icassp/HuiWCYWX21}." />
        </attvalues>
      </edge>
      <edge source="2209.04156" target="2102.10905" id="9867">
        <attvalues>
          <attvalue for="5" value=" The joint learning methods always demonstrate their effectiveness over the independent models~\cite{DBLP:conf/interspeech/Hakkani-TurTCCG16,DBLP:conf/acl/ZhangLDFY19,DBLP:conf/icassp/HuiWCYWX21}." />
        </attvalues>
      </edge>
      <edge source="2209.04156" target="1710.10903" id="9869">
        <attvalues>
          <attvalue for="5" value=" Considering the variant importance of syntactic characteristics in dependency structures, we encode the syntactic information by graph attention network~\cite{velickovic2018graph}." />
        </attvalues>
      </edge>
      <edge source="2209.04156" target="2003.09831" id="9870">
        <attvalues>
          <attvalue for="5" value=" Furthermore, we acquire the semantic embeddings of task labels by their descriptions~\cite{DBLP:journals/taslp/ZhuZMY20}." />
        </attvalues>
      </edge>
      <edge source="2209.04156" target="1805.10190" id="9871">
        <attvalues>
          <attvalue for="5" value=" We compare our model with several state-of-the-art baselines on two public datasets: ATIS~\cite{DBLP:conf/naacl/HemphillGD90} and SNIPS~\cite{DBLP:journals/corr/abs-1805-10190}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1703.10135" id="9872">
        <attvalues>
          <attvalue for="5" value=" Recent advances in end-to-end text-to-speech (TTS) synthesis have enabled us to produce very realistic and natural-sounding synthetic speech \cite{wang2017tacotron,ping2018clarinet} with mean opinion scores (MOS) approaching those of natural human speech \cite{shen2018natural}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1807.07281" id="9873">
        <attvalues>
          <attvalue for="5" value=" Recent advances in end-to-end text-to-speech (TTS) synthesis have enabled us to produce very realistic and natural-sounding synthetic speech \cite{wang2017tacotron,ping2018clarinet} with mean opinion scores (MOS) approaching those of natural human speech \cite{shen2018natural}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1712.05884" id="9874">
        <attvalues>
          <attvalue for="5" value=" Recent advances in end-to-end text-to-speech (TTS) synthesis have enabled us to produce very realistic and natural-sounding synthetic speech \cite{wang2017tacotron,ping2018clarinet} with mean opinion scores (MOS) approaching those of natural human speech \cite{shen2018natural}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1907.04462" id="9875">
        <attvalues>
          <attvalue for="5" value=" Prior studies have focused on training a speaker encoder network jointly with the TTS model \cite{chen2019sample,park2019multi,nachmani2018fitting} or the neural vocoder \cite{deng2018modeling}; others have explored the use of speaker embeddings in combination with fine-tuning the TTS model \cite{deng2018modeling,hu2019neural,arik2018neural}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1905.00590" id="9876">
        <attvalues>
          <attvalue for="5" value=" An effective approach for speaker adaptation in neural TTS is to fine-tune all or part of model with a small amount of data from the target speaker \cite{kons2019high,chen2019sample}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1809.10460" id="9877">
        <attvalues>
          <attvalue for="5" value=" Prior studies have focused on training a speaker encoder network jointly with the TTS model \cite{chen2019sample,park2019multi,nachmani2018fitting} or the neural vocoder \cite{deng2018modeling}; others have explored the use of speaker embeddings in combination with fine-tuning the TTS model \cite{deng2018modeling,hu2019neural,arik2018neural}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1812.05253" id="9879">
        <attvalues>
          <attvalue for="5" value=" Prior studies have focused on training a speaker encoder network jointly with the TTS model \cite{chen2019sample,park2019multi,nachmani2018fitting} or the neural vocoder \cite{deng2018modeling}; others have explored the use of speaker embeddings in combination with fine-tuning the TTS model \cite{deng2018modeling,hu2019neural,arik2018neural}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1802.06006" id="9880">
        <attvalues>
          <attvalue for="5" value=" Prior studies have focused on training a speaker encoder network jointly with the TTS model \cite{chen2019sample,park2019multi,nachmani2018fitting} or the neural vocoder \cite{deng2018modeling}; others have explored the use of speaker embeddings in combination with fine-tuning the TTS model \cite{deng2018modeling,hu2019neural,arik2018neural}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1806.04558" id="9881">
        <attvalues>
          <attvalue for="5" value=" \cite{jia2018transfer} observed that unseen speakers' synthetic speech had lower speaker similarity to the target speaker than seen speakers, accents were often mismatched, and nuances such as characteristic prosody were lost, indicating that while seen speakers can be well-modeled in this manner, there is room for improvement for modeling unseen speakers." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1804.05160" id="9883">
        <attvalues>
          <attvalue for="5" value=" One prominent advancement is the use of learnable dictionary encoding (LDE) \cite{cai2018exploring} and angular softmax \cite{Huang2018} for speaker recognition, which are reported to boost the speaker recognition performance on open-source corpora such as the VoxCelebs \cite{nagrani2017voxceleb, chung2018voxceleb2}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1706.08612" id="9884">
        <attvalues>
          <attvalue for="5" value=" One prominent advancement is the use of learnable dictionary encoding (LDE) \cite{cai2018exploring} and angular softmax \cite{Huang2018} for speaker recognition, which are reported to boost the speaker recognition performance on open-source corpora such as the VoxCelebs \cite{nagrani2017voxceleb, chung2018voxceleb2}." />
        </attvalues>
      </edge>
      <edge source="1910.10838" target="1810.11960" id="9885">
        <attvalues>
          <attvalue for="5" value=" For this purpose, we extend an improved Tacotron system in \cite{yasuda2019investigation} to a multi-speaker TTS system and conduct systematic analysis to answer the above question." />
        </attvalues>
      </edge>
      <edge source="2109.04939" target="1602.07776" id="9886">
        <attvalues>
          <attvalue for="5" value=" Specifically, although \cite{dyerRecurrentNeuralNetwork2016a} and \cite{haleFindingSyntaxHuman2018} adopted the vanilla RNNG with a top-down parsing strategy for English with head-initial right-branching structures, \cite{abneyMemoryRequirementsLocal1991} and \cite{resnikLeftcornerParsingPsychological1992} suggested that the top-down parsing strategy is not optimal for head-final left-branching structures, and alternatively proposed the left-corner parsing strategy as more human-like parsing strategy." />
        </attvalues>
      </edge>
      <edge source="2109.04939" target="1806.04127" id="9887">
        <attvalues>
          <attvalue for="5" value=" Specifically, although \cite{dyerRecurrentNeuralNetwork2016a} and \cite{haleFindingSyntaxHuman2018} adopted the vanilla RNNG with a top-down parsing strategy for English with head-initial right-branching structures, \cite{abneyMemoryRequirementsLocal1991} and \cite{resnikLeftcornerParsingPsychological1992} suggested that the top-down parsing strategy is not optimal for head-final left-branching structures, and alternatively proposed the left-corner parsing strategy as more human-like parsing strategy." />
        </attvalues>
      </edge>
      <edge source="2110.06341" target="2006.06264" id="9888">
        <attvalues>
          <attvalue for="5" value=" Since these metrics focus solely on surface-level aspects of the generated text, they correlate poorly with human evaluation, especially when models are producing high-quality text~\cite{belz2006comparing,callison2006re,ma2019results,mathur2020tangled}." />
        </attvalues>
      </edge>
      <edge source="2110.06341" target="2006.14799" id="9889">
        <attvalues>
          <attvalue for="5" value=" This has led to a surge of interest in learned metrics that cast evaluation as a regression problem and leverage pre-trained multilingual models to capture the semantic similarity between references and generated text~\cite{celikyilmaz2020evaluation}." />
        </attvalues>
      </edge>
      <edge source="2110.06341" target="2009.09025" id="9890">
        <attvalues>
          <attvalue for="5" value=" Popular examples of those metrics include Comet~\cite{rei2020comet} and Bleurt-Extended~\cite{sellam2020bleurt}, based on XLM-RoBERTa~\cite{conneau2019cross, conneau2019unsupervised} and mBERT~\cite{devlin2018bert} respectively." />
        </attvalues>
      </edge>
      <edge source="2110.06341" target="2004.04696" id="9891">
        <attvalues>
          <attvalue for="5" value=" Inspired by~\cite{sellam2020bleurt}, we introduce a data generation method based on random perturbations that allows us to synthesize arbitrary amounts of multilingual training data." />
        </attvalues>
      </edge>
      <edge source="2110.06341" target="1901.07291" id="9892">
        <attvalues>
          <attvalue for="5" value=" In the case of cross-lingual transfer, this phenomenon is known as the curse of multilinguality: to allow for positive transfer, the model must be scaled up with the number of languages~\cite{conneau2019cross}." />
        </attvalues>
      </edge>
      <edge source="2110.06341" target="2010.12821" id="9897">
        <attvalues>
          <attvalue for="5" value=" We first present a series of experiments that validate that previous findings on cross-lingual transfer and the curse of multilinguality apply to the metrics domain, using RemBERT (Rebalanced mBERT), a multilingual extension of BERT~\cite{chung2020rethinking}." />
        </attvalues>
      </edge>
      <edge source="2110.06341" target="1503.02531" id="9898">
        <attvalues>
          <attvalue for="5" value=" Distillation has been shown to successfully transfer knowledge from large models to smaller ones~\cite{hinton2015distilling}, but it requires access to a large corpus of unlabelled data~\cite{sanh2019distilbert,turc2019well}, which does not exist for our task." />
        </attvalues>
      </edge>
      <edge source="2309.12234" target="2005.08100" id="9900">
        <attvalues>
          <attvalue for="5" value=" Current end-to-end systems consistently outperform their traditional hybrid and cascaded counterparts in both efficiency and performance \cite{Gulati_ISCA2020, xu2023recent}." />
        </attvalues>
      </edge>
      <edge source="2309.12234" target="1911.08870" id="9901">
        <attvalues>
          <attvalue for="5" value=" To mitigate this, Connectionist Temporal Classification (CTC) \cite{Graves_ACL2006} has emerged as a widely-used auxiliary training objective \cite{watanabe2017hybrid}, which has demonstrated a notable improvement in stabilizing model convergence and enhancing performance, especially in the more challenging ST task \cite{bahar2019comparative}." />
        </attvalues>
      </edge>
      <edge source="2309.12234" target="1909.06515" id="9903">
        <attvalues>
          <attvalue for="5" value=" This mechanism offers a more lightweight and direct solution compared to alternative techniques like pre-training \cite{bansal2018pre} or data augmentation \cite{pino_corr2019}." />
        </attvalues>
      </edge>
      <edge source="2309.12234" target="2105.05752" id="9904">
        <attvalues>
          <attvalue for="5" value=" Researchers propose a stacked design for ST that decouples the encoding into an acoustic encoder and a textual encoder, where only the former is supervised by the CTC loss\cite{Xu_ACL2021,nast}." />
        </attvalues>
      </edge>
      <edge source="2309.12234" target="2305.17358" id="9905">
        <attvalues>
          <attvalue for="5" value=" We further come up with an enhanced variant by combining several recent advancements in CTC application, including InterCTC \cite{Lee_ICASSP2021}, prediction-aware encoding \cite{nast}, and a curriculum-based training strategy \cite{nast}, referred to as BiL-CTC+." />
        </attvalues>
      </edge>
      <edge source="2309.12234" target="2102.03216" id="9906">
        <attvalues>
          <attvalue for="5" value=" We further come up with an enhanced variant by combining several recent advancements in CTC application, including InterCTC \cite{Lee_ICASSP2021}, prediction-aware encoding \cite{nast}, and a curriculum-based training strategy \cite{nast}, referred to as BiL-CTC+." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="2204.01670" id="9907">
        <attvalues>
          <attvalue for="5" value=" While some studies \cite{18, 19, 15, 16} have focused on developing speaker-independent (SI) DSR models, the inherent variability among dysarthric speakers—due to differences in etiology, age, gender, speaking style, and severity of dysarthria—poses a significant challenge." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="1907.13511" id="9908">
        <attvalues>
          <attvalue for="5" value=" \cite{14} achieved this by fine-tuning a conventional speech recognition model with data from a specific dysarthric speaker, selectively fine-tuning only a subset of the network layers to avoid overfitting." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="2011.01686" id="9909">
        <attvalues>
          <attvalue for="5" value=" Consequently, several studies \cite{14,33,9,20,2} have explored fine-tuning speech recognition models with data from the target speaker for personalized DSR." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="2211.00089" id="9910">
        <attvalues>
          <attvalue for="5" value=" Moreover, the varying severity of dysarthria over time \cite{11} can compromise the long-term efficacy of personalized DSR models, necessitating continuous data collection and model optimization." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="1703.05175" id="9911">
        <attvalues>
          <attvalue for="5" value=" Inspired by the principles of prototypical networks for few-shot learning \cite{3}, our PB-DSR method utilizes a feature extractor to create per-word prototypes from only few-shot samples, enabling rapid adaptation to the unique speech patterns of each speaker." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="2106.07447" id="9912">
        <attvalues>
          <attvalue for="5" value=" We utilize the pre-trained HuBERT \cite{7} model, renowned for its general speech recognition capabilities, to extract speech features." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="2004.11362" id="9913">
        <attvalues>
          <attvalue for="5" value=" Moreover, we integrate supervised contrastive learning (SCL) \cite{10} to enhance feature extraction, adopting a strategy from contrastive learning \cite{23} that optimizes feature representations by minimizing intra-class distances while maximizing inter-class separations, a technique proven effective in various speech tasks \cite{22,32,31,35}." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="2002.05709" id="9914">
        <attvalues>
          <attvalue for="5" value=" Moreover, we integrate supervised contrastive learning (SCL) \cite{10} to enhance feature extraction, adopting a strategy from contrastive learning \cite{23} that optimizes feature representations by minimizing intra-class distances while maximizing inter-class separations, a technique proven effective in various speech tasks \cite{22,32,31,35}." />
        </attvalues>
      </edge>
      <edge source="2407.18461" target="2308.16485" id="9915">
        <attvalues>
          <attvalue for="5" value=" Moreover, we integrate supervised contrastive learning (SCL) \cite{10} to enhance feature extraction, adopting a strategy from contrastive learning \cite{23} that optimizes feature representations by minimizing intra-class distances while maximizing inter-class separations, a technique proven effective in various speech tasks \cite{22,32,31,35}." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1310.4546" id="9930">
        <attvalues>
          <attvalue for="5" value=" The approach to definition modelling of \cite{noraset2017definition} is based on a recurrent neural network (RNN) language model, which is conditioned on a word embedding for the target word to be defined, specifically pre-trained word2vec \cite{mikolov2013distributed} embeddings." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1502.07257" id="9931">
        <attvalues>
          <attvalue for="5" value=" Subsequent work has considered approaches to learn multi-sense embeddings, in which a word is represented by multiple vectors, each corresponding to a sense \cite{adagram,lee2017muse}." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1704.04601" id="9932">
        <attvalues>
          <attvalue for="5" value=" Subsequent work has considered approaches to learn multi-sense embeddings, in which a word is represented by multiple vectors, each corresponding to a sense \cite{adagram,lee2017muse}." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1612.00394" id="9934">
        <attvalues>
          <attvalue for="5" value=" Our experimental results show that, for every language and dataset considered, our proposed approach outperforms the benchmark approach of \cite{noraset2017definition} which does not model polysemy." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1709.09254" id="9935">
        <attvalues>
          <attvalue for="5" value=" To address this limitation, a number of studies have proposed context-aware definition generation models \cite{ni2017learning,gadetsky2018conditional,ishiwatari2018learning,mickus-etal-2019-mark,chang2019does}." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1806.10090" id="9936">
        <attvalues>
          <attvalue for="5" value=" To address this limitation, a number of studies have proposed context-aware definition generation models \cite{ni2017learning,gadetsky2018conditional,ishiwatari2018learning,mickus-etal-2019-mark,chang2019does}." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1911.05715" id="9937">
        <attvalues>
          <attvalue for="5" value=" To address this limitation, a number of studies have proposed context-aware definition generation models \cite{ni2017learning,gadetsky2018conditional,ishiwatari2018learning,mickus-etal-2019-mark,chang2019does}." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1909.09483" id="9938">
        <attvalues>
          <attvalue for="5" value=" Following \cite{zhu2019multi} we evaluate our proposed model using variations of BLEU \cite{papineni2002bleu}." />
        </attvalues>
      </edge>
      <edge source="2006.07398" target="1601.03764" id="9939">
        <attvalues>
          <attvalue for="5" value=" This model utilizes word embeddings and coarse-grained atom embeddings to represent senses \cite{arora2018linearTRIM}, in which atoms are shared across words." />
        </attvalues>
      </edge>
      <edge source="2404.01903" target="2212.10561" id="9948">
        <attvalues>
          <attvalue for="5" value=" They are the foundation of both developer tools (eg, \cite{tabnine,github-copilot}) and reasoning agents (eg, \cite{zelikman2024parsel, yang2024if,hu:codebotler,alphacode}), capable of performing a variety of programming tasks across several programming languages." />
        </attvalues>
      </edge>
      <edge source="2404.01903" target="2311.11183" id="9949">
        <attvalues>
          <attvalue for="5" value=" They are the foundation of both developer tools (eg, \cite{tabnine,github-copilot}) and reasoning agents (eg, \cite{zelikman2024parsel, yang2024if,hu:codebotler,alphacode}), capable of performing a variety of programming tasks across several programming languages." />
        </attvalues>
      </edge>
      <edge source="2404.01903" target="2203.07814" id="9950">
        <attvalues>
          <attvalue for="5" value=" They are the foundation of both developer tools (eg, \cite{tabnine,github-copilot}) and reasoning agents (eg, \cite{zelikman2024parsel, yang2024if,hu:codebotler,alphacode}), capable of performing a variety of programming tasks across several programming languages." />
        </attvalues>
      </edge>
      <edge source="2404.01903" target="2306.03341" id="9951">
        <attvalues>
          <attvalue for="5" value=" We employ activation steering~\cite{li2024inference,rimsky2023steering,turner2023activation, subramani2022extracting} to steer model activations toward the target task of type prediction." />
        </attvalues>
      </edge>
      <edge source="2404.01903" target="2312.06681" id="9952">
        <attvalues>
          <attvalue for="5" value=" We employ activation steering~\cite{li2024inference,rimsky2023steering,turner2023activation, subramani2022extracting} to steer model activations toward the target task of type prediction." />
        </attvalues>
      </edge>
      <edge source="2404.01903" target="2308.10248" id="9953">
        <attvalues>
          <attvalue for="5" value=" We employ activation steering~\cite{li2024inference,rimsky2023steering,turner2023activation, subramani2022extracting} to steer model activations toward the target task of type prediction." />
        </attvalues>
      </edge>
      <edge source="2210.16498" target="1909.06317" id="9954">
        <attvalues>
          <attvalue for="5" value=" However, the gap between human and machine intelligence is not that straightforward to be filled in the post-transformer era~\cite{karita2019comparative, lecun2022autonomous, mohamed2022self_speech, speech_representation}." />
        </attvalues>
      </edge>
      <edge source="2210.16498" target="2204.00465" id="9955">
        <attvalues>
          <attvalue for="5" value=" Another disadvantage of NCMF~\cite{lian_art} is that the MNGU0 data is sparsely sampled from articulators, which limits the intelligibility of learned representations." />
        </attvalues>
      </edge>
      <edge source="2204.00465" target="2011.04491" id="9956">
        <attvalues>
          <attvalue for="5" value=" Research on speech representation learning has been dominated by deep learning recently in the areas such as speaker recognition~\cite{lian2020masked, ECAPA-TDNN}, automatic speech recognition~\cite{conformer}, voice conversion~\cite{DSVAE-VC, C-DSVAE} and text-to-speech~\cite{VITS, lian2022utts}, etc The goal of speech representation learning is to optimize both the performance of the model architectures and the interpretability of the learned representations." />
        </attvalues>
      </edge>
      <edge source="2204.00465" target="2005.08100" id="9957">
        <attvalues>
          <attvalue for="5" value=" Research on speech representation learning has been dominated by deep learning recently in the areas such as speaker recognition~\cite{lian2020masked, ECAPA-TDNN}, automatic speech recognition~\cite{conformer}, voice conversion~\cite{DSVAE-VC, C-DSVAE} and text-to-speech~\cite{VITS, lian2022utts}, etc The goal of speech representation learning is to optimize both the performance of the model architectures and the interpretability of the learned representations." />
        </attvalues>
      </edge>
      <edge source="2204.00465" target="2106.06103" id="9959">
        <attvalues>
          <attvalue for="5" value=" Research on speech representation learning has been dominated by deep learning recently in the areas such as speaker recognition~\cite{lian2020masked, ECAPA-TDNN}, automatic speech recognition~\cite{conformer}, voice conversion~\cite{DSVAE-VC, C-DSVAE} and text-to-speech~\cite{VITS, lian2022utts}, etc The goal of speech representation learning is to optimize both the performance of the model architectures and the interpretability of the learned representations." />
        </attvalues>
      </edge>
      <edge source="2204.00465" target="1609.03296" id="9960">
        <attvalues>
          <attvalue for="5" value=" To handle the aforementioned problem, \cite{neural-nmf} proposed an auto-encoder based model to replace non-negatve matrix factorization for speech separation task." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="1907.11692" id="9973">
        <attvalues>
          <attvalue for="5" value=" The advent of large-scale language models has revolutionized the field of natural language processing, leading to significant advancements in various applications, such as language translation and text classification \cite{devlin2019bert,radford2019language,liu2019roberta,clark2020electra,chowdhery2022palm,anil2023palm}." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="2003.10555" id="9974">
        <attvalues>
          <attvalue for="5" value=" The advent of large-scale language models has revolutionized the field of natural language processing, leading to significant advancements in various applications, such as language translation and text classification \cite{devlin2019bert,radford2019language,liu2019roberta,clark2020electra,chowdhery2022palm,anil2023palm}." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="2204.02311" id="9975">
        <attvalues>
          <attvalue for="5" value=" The advent of large-scale language models has revolutionized the field of natural language processing, leading to significant advancements in various applications, such as language translation and text classification \cite{devlin2019bert,radford2019language,liu2019roberta,clark2020electra,chowdhery2022palm,anil2023palm}." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="2305.10403" id="9976">
        <attvalues>
          <attvalue for="5" value=" The advent of large-scale language models has revolutionized the field of natural language processing, leading to significant advancements in various applications, such as language translation and text classification \cite{devlin2019bert,radford2019language,liu2019roberta,clark2020electra,chowdhery2022palm,anil2023palm}." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="2205.01068" id="9977">
        <attvalues>
          <attvalue for="5" value=" While numerous large language models for English have been publicly released \cite{zhang2022opt,black2022gptneox20b,biderman2023pythia,touvron2023llama,together2023redpajama,MosaicML2023Introducing}, the availability of such models for non-English languages remains limited." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="2204.06745" id="9978">
        <attvalues>
          <attvalue for="5" value=" While numerous large language models for English have been publicly released \cite{zhang2022opt,black2022gptneox20b,biderman2023pythia,touvron2023llama,together2023redpajama,MosaicML2023Introducing}, the availability of such models for non-English languages remains limited." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="2304.01373" id="9979">
        <attvalues>
          <attvalue for="5" value=" While numerous large language models for English have been publicly released \cite{zhang2022opt,black2022gptneox20b,biderman2023pythia,touvron2023llama,together2023redpajama,MosaicML2023Introducing}, the availability of such models for non-English languages remains limited." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="2211.05100" id="9982">
        <attvalues>
          <attvalue for="5" value=" Although several multilingual large language models have also been released \cite{lin2022fewshot,scao2022bloom}, they are typically trained on English-centric corpora, resulting in lower performance on other languages." />
        </attvalues>
      </edge>
      <edge source="2306.02254" target="2204.04541" id="9983">
        <attvalues>
          <attvalue for="5" value=" We assess the zero-shot and few-shot performance of our Polyglot-Ko models using the KOBEST benchmark \cite{kim2022kobest}." />
        </attvalues>
      </edge>
      <edge source="2204.04541" target="2004.05986" id="9986">
        <attvalues>
          <attvalue for="5" value="~\cite{CLUE, FLUE, indonlu, KLUE}." />
        </attvalues>
      </edge>
      <edge source="2204.04541" target="1708.01425" id="9987">
        <attvalues>
          <attvalue for="5" value=" These studies have found that datasets may contain many spurious artefacts, and the performance of PLM is enhanced by excessive usage of said artefacts~\cite{habernal2018argument, niven2019probing, mccoy2019right, benderkoller2020climbing}." />
        </attvalues>
      </edge>
      <edge source="2204.04541" target="2004.03289" id="9989">
        <attvalues>
          <attvalue for="5" value=" When it comes to the Korean language, two benchmarks are widely used: Korean-NLI \&amp; STS~\cite{ham2020kornli} and KLUE~\cite{KLUE}." />
        </attvalues>
      </edge>
      <edge source="2204.04541" target="2105.09680" id="9990">
        <attvalues>
          <attvalue for="5" value=" When it comes to the Korean language, two benchmarks are widely used: Korean-NLI \&amp; STS~\cite{ham2020kornli} and KLUE~\cite{KLUE}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="1211.3711" id="9991">
        <attvalues>
          <attvalue for="5" value=" End-to-end models include connectionist temporal classification \cite{graves2006connectionist}, recurrent neural network-transducer (RNN-T) \cite{graves2012sequence}, and attention-based sequence-to-sequence models \cite{bahdanau2014neural,chorowski2015attention,ray2018ad3} also known as LAS: Listen, Attend and Spell \cite{chan2016listen}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="1409.0473" id="9992">
        <attvalues>
          <attvalue for="5" value=" End-to-end models include connectionist temporal classification \cite{graves2006connectionist}, recurrent neural network-transducer (RNN-T) \cite{graves2012sequence}, and attention-based sequence-to-sequence models \cite{bahdanau2014neural,chorowski2015attention,ray2018ad3} also known as LAS: Listen, Attend and Spell \cite{chan2016listen}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="1506.07503" id="9993">
        <attvalues>
          <attvalue for="5" value=" End-to-end models include connectionist temporal classification \cite{graves2006connectionist}, recurrent neural network-transducer (RNN-T) \cite{graves2012sequence}, and attention-based sequence-to-sequence models \cite{bahdanau2014neural,chorowski2015attention,ray2018ad3} also known as LAS: Listen, Attend and Spell \cite{chan2016listen}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="1811.06621" id="9995">
        <attvalues>
          <attvalue for="5" value=" Among these three methods, RNN-T is replacing the traditional hybrid ASR models \cite{he2019streaming,sainath2019two} since it has good streaming capability which is challenging to LAS and does not have CTC's frame-independence assumption." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="1908.10992" id="9996">
        <attvalues>
          <attvalue for="5" value=" Using LAS as a second-pass rescorer by attending to both encoder features and n-best output from the RNN-T has been explored in \cite{sainath2019two}, and an inter encoder-decoder attention mechanism was introduced in \cite{wang2020attention} to better align the encoder feature with the hypothesis." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="1909.12415" id="9997">
        <attvalues>
          <attvalue for="5" value=" Depth-LSTM and layer normalization was tried in \cite{li2019improving}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="2005.08497" id="9998">
        <attvalues>
          <attvalue for="5" value=" Using LAS as a second-pass rescorer by attending to both encoder features and n-best output from the RNN-T has been explored in \cite{sainath2019two}, and an inter encoder-decoder attention mechanism was introduced in \cite{wang2020attention} to better align the encoder feature with the hypothesis." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="2007.15188" id="9999">
        <attvalues>
          <attvalue for="5" value=" Other improvements include better initialization methods, training on TTS data, and use of lookahead encoders \cite{li2020developing}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="1802.08395" id="10000">
        <attvalues>
          <attvalue for="5" value=" Training an end-to-end SLU system to predict intent and slot values directly from audio is therefore becoming a popular research area \cite{qian2017exploring,serdyuk2018towards,chen2018spoken,haghani2018audio, lugosch2019speech,rao2020speech,rao2021mean}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="1809.09190" id="10001">
        <attvalues>
          <attvalue for="5" value=" While such system may not readily outperform or replace large-scale conventional SLU systems with independently optimized ASR and NLU modules, it is indicated that semantic information such as intent and slots could potentially help improve an ASR system \cite{haghani2018audio,rao2021mean}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="2008.06173" id="10002">
        <attvalues>
          <attvalue for="5" value=" Training an end-to-end SLU system to predict intent and slot values directly from audio is therefore becoming a popular research area \cite{qian2017exploring,serdyuk2018towards,chen2018spoken,haghani2018audio, lugosch2019speech,rao2020speech,rao2021mean}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="2102.06750" id="10003">
        <attvalues>
          <attvalue for="5" value=" While such system may not readily outperform or replace large-scale conventional SLU systems with independently optimized ASR and NLU modules, it is indicated that semantic information such as intent and slots could potentially help improve an ASR system \cite{haghani2018audio,rao2021mean}." />
        </attvalues>
      </edge>
      <edge source="2105.07071" target="2106.06183" id="10004">
        <attvalues>
          <attvalue for="5" value=" The technique to explicitly incorporate various contextual signals analogous to intent, such as dialog state and music play state, into an RNN-T based ASR system has been proposed in \cite{wu2020multistate,ray2021improving}, but most of these contextual signals are derived only after the first turn of the dialog and would only benefit subsequent turns." />
        </attvalues>
      </edge>
      <edge source="2111.02392" target="2008.12527" id="10005">
        <attvalues>
          <attvalue for="5" value=" However, there is still a gap in quality and intelligibility between unsupervised and supervised systems \cite{zhao2020voice}." />
        </attvalues>
      </edge>
      <edge source="2111.02392" target="2104.00355" id="10006">
        <attvalues>
          <attvalue for="5" value=" Most of these studies focus on discrete speech units \cite{polyak2021speech, van2020vector, huang2021any}." />
        </attvalues>
      </edge>
      <edge source="2111.02392" target="2106.07447" id="10007">
        <attvalues>
          <attvalue for="5" value=" Focusing on any-to-one voice conversion (eg, any source speaker to a single target speaker), we compare discrete and soft speech units across two self-supervised methods: contrastive predictive coding (CPC) \cite{oord2018representation} and hidden-unit BERT (HuBERT) \cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2111.02392" target="1807.03748" id="10008">
        <attvalues>
          <attvalue for="5" value=" Focusing on any-to-one voice conversion (eg, any source speaker to a single target speaker), we compare discrete and soft speech units across two self-supervised methods: contrastive predictive coding (CPC) \cite{oord2018representation} and hidden-unit BERT (HuBERT) \cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2401.08092" id="10009">
        <attvalues>
          <attvalue for="5" value=" Second, we approach the large-scale LLM serving system as a whole rather than focusing on a specific technique (eg, RAG~\cite{rag-survey}, long-context~\cite{long-context-survey-dong2023}), or a layer (eg, model~\cite{xu2024survey}, agent~\cite{agent-survey-2023})." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2404.18416" id="10010">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, this conflict between accuracy and performance has been exacerbated recently by the growing demand for longer contextual understanding when deploying models in practice~\cite{med-gemini}." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2211.05102" id="10011">
        <attvalues>
          <attvalue for="5" value=" This introduces new complexities as the transformer's attention mechanism exhibits a quadratic increase in resource consumption with longer contexts~\cite{pope2023efficiently}." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2307.03172" id="10012">
        <attvalues>
          <attvalue for="5" value=" Furthermore, LLMs struggle to utilize information from longer contexts effectively~\cite{lost-in-the-middle-2023}." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="1901.02860" id="10014">
        <attvalues>
          <attvalue for="5" value=" Works at this layer commonly optimize the model structure~\cite{transformer-xl, zhao2023atom}, cache~\cite{vLLM-sosp23,streamingLLM}, scheduling~\cite{orca-osdi22,tetriserve-2024}, etc The agent-layer system sits atop the model-layer system and results from emerging LLM-based system applications that leverage LLM-driven workflow to improve a raw LLM model's accuracy and efficiency while handling complex real-world tasks~\cite{compound-ai-blog}." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2104.09864" id="10017">
        <attvalues>
          <attvalue for="5" value=" For example, using positional embedding to extend a model's range does not improve the model’s accuracy beyond the context length~\cite{roformer-su2021}, and using quantization~\cite{zhao2023atom}, pruning~\cite{compression-survey-2023}, and sparsity~\cite{streamingLLM} enable one to serve a model with faster speed but at the cost of potentially lower accuracy." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2310.19102" id="10018">
        <attvalues>
          <attvalue for="5" value=" For example, using positional embedding to extend a model's range does not improve the model’s accuracy beyond the context length~\cite{roformer-su2021}, and using quantization~\cite{zhao2023atom}, pruning~\cite{compression-survey-2023}, and sparsity~\cite{streamingLLM} enable one to serve a model with faster speed but at the cost of potentially lower accuracy." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2308.07633" id="10019">
        <attvalues>
          <attvalue for="5" value=" For example, using positional embedding to extend a model's range does not improve the model’s accuracy beyond the context length~\cite{roformer-su2021}, and using quantization~\cite{zhao2023atom}, pruning~\cite{compression-survey-2023}, and sparsity~\cite{streamingLLM} enable one to serve a model with faster speed but at the cost of potentially lower accuracy." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2404.14294" id="10021">
        <attvalues>
          <attvalue for="5" value=" Compared to prior surveys~\cite{zhou2024survey,xu2024survey,long-context-survey, long-context-survey-dong2023, long-context-survey-wang2024,agent-survey-2023,rag-survey}, we makes two unique contributions." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2309.07864" id="10022">
        <attvalues>
          <attvalue for="5" value=" Second, we approach the large-scale LLM serving system as a whole rather than focusing on a specific technique (eg, RAG~\cite{rag-survey}, long-context~\cite{long-context-survey-dong2023}), or a layer (eg, model~\cite{xu2024survey}, agent~\cite{agent-survey-2023})." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2312.10997" id="10023">
        <attvalues>
          <attvalue for="5" value=" Second, we approach the large-scale LLM serving system as a whole rather than focusing on a specific technique (eg, RAG~\cite{rag-survey}, long-context~\cite{long-context-survey-dong2023}), or a layer (eg, model~\cite{xu2024survey}, agent~\cite{agent-survey-2023})." />
        </attvalues>
      </edge>
      <edge source="2405.11299" target="2302.14502" id="10024">
        <attvalues>
          <attvalue for="5" value=" Second, we approach the large-scale LLM serving system as a whole rather than focusing on a specific technique (eg, RAG~\cite{rag-survey}, long-context~\cite{long-context-survey-dong2023}), or a layer (eg, model~\cite{xu2024survey}, agent~\cite{agent-survey-2023})." />
        </attvalues>
      </edge>
      <edge source="2002.05955" target="1802.08395" id="10025">
        <attvalues>
          <attvalue for="5" value=" They were initially applied to speech translation \cite{berard-nips2016,weiss2017sequence} and then to SLU tasks where the main goal is to extract the domain and user intent from an utterance, together with some semantic slots \cite{DBLP:journals/corr/abs-1802-08395,DBLP:journals/corr/abs-1904-03670}." />
        </attvalues>
      </edge>
      <edge source="2002.05955" target="1904.03670" id="10027">
        <attvalues>
          <attvalue for="5" value=" Except for \cite{DBLP:journals/corr/abs-1904-03670}, most end-to-end SLU systems of the literature are trained on huge amount of data." />
        </attvalues>
      </edge>
      <edge source="2002.05955" target="1512.02595" id="10028">
        <attvalues>
          <attvalue for="5" value=" The most relevant works of the literature with respect to this task \cite{DBLP:journals/corr/abs-1906-07601,ghannay:hal-01987740} propose models based on Feed-Forward Neural Networks (FFNN) similar to the Deep Speech 2 model proposed for ASR \cite{DBLP-journals/corr/AmodeiABCCCCCCD15}, and an independent pre-trained language model re-scores semantic outputs." />
        </attvalues>
      </edge>
      <edge source="2201.10716" target="2010.11439" id="10029">
        <attvalues>
          <attvalue for="5" value=" Considering that a pronunciation dictionary can never cover all possible words in a language, G2P conversion is essential for any applications that depend on the mapping relationship between the spoken and written forms of a language, such as TTS and ASR \cite{elias2021parallel, gao2021pre, masumura20_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2201.10716" target="1610.06540" id="10030">
        <attvalues>
          <attvalue for="5" value=" Recently, neural networks such as LSTM \cite{toshniwal2016jointly} and Transformer \cite{yolchuyeva2019transformer} have showed powerful ability on G2P conversion." />
        </attvalues>
      </edge>
      <edge source="2201.10716" target="2004.06338" id="10031">
        <attvalues>
          <attvalue for="5" value=" The Transformer-based models have achieved state-of-the-art performance in many benchmarks \cite{yolchuyeva2019transformer, gorman2020sigmorphon}." />
        </attvalues>
      </edge>
      <edge source="2201.10716" target="1708.01464" id="10032">
        <attvalues>
          <attvalue for="5" value=" Subsequently, multilingual neural networks \cite{peters2017massively} and pre-trained G2P models of high-resource languages \cite{engelhart2021grapheme} showed better cross-lingual G2P modeling ability." />
        </attvalues>
      </edge>
      <edge source="2201.10716" target="2104.04091" id="10033">
        <attvalues>
          <attvalue for="5" value=" Subsequently, multilingual neural networks \cite{peters2017massively} and pre-trained G2P models of high-resource languages \cite{engelhart2021grapheme} showed better cross-lingual G2P modeling ability." />
        </attvalues>
      </edge>
      <edge source="2201.10716" target="2002.06823" id="10035">
        <attvalues>
          <attvalue for="5" value=" They are fine-tuning GBERT and fusing GBERT into the Transformer model by attention \cite{zhu2019incorporating}." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="2108.13624" id="10037">
        <attvalues>
          <attvalue for="5" value=" When these models are deployed for real-world use, their performance should be consistent even when presented with out-of-distribution (OOD) data that differs from the training distribution in an unpredictable manner \cite{liu2021towards}." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="2305.20019" id="10038">
        <attvalues>
          <attvalue for="5" value=" Length generalisation (see eg \cite{pmlr-v202-ray-chowdhury23b,zhou2023algorithms}) is the capacity to process sequences that are longer (or shorter) than those seen in the training set." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="2310.16028" id="10039">
        <attvalues>
          <attvalue for="5" value=" Length generalisation (see eg \cite{pmlr-v202-ray-chowdhury23b,zhou2023algorithms}) is the capacity to process sequences that are longer (or shorter) than those seen in the training set." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="1904.03670" id="10040">
        <attvalues>
          <attvalue for="5" value=" In \cite{lugosch19_interspeech}, a system was trained on three specific phrases and tested with also a new phrase, assessing the OOV generalisation." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="2301.10884" id="10042">
        <attvalues>
          <attvalue for="5" value=" More generally, neural (NLP) systems have been found often to fail in tasks that require CG \cite{lake2018generalization, keysers2019measuring, yao2022structural}, although they do have some capacity for CG \cite{lepori2023break}." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="2009.09395" id="10043">
        <attvalues>
          <attvalue for="5" value=" In addition to the generalisation types applicable to both text and audio sequences, some types are specific to audio-based models, such as diverse accents \cite{viglino19_interspeech}, different age groups \cite{potamianos2003robust}, and various acoustic environments \cite{haeb2020far}." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="1802.08395" id="10044">
        <attvalues>
          <attvalue for="5" value=" As these studies focus on the text-based language understanding task, the proposed evaluation sets are mostly text-only data, and as such, they can not always be applied to the end-to-end (E2E) SLU models that rely solely on audio \cite{serdyuk2018towards, haghani2018audio, palogiannidi2020end}." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="1910.10599" id="10045">
        <attvalues>
          <attvalue for="5" value=" Similarly, in \cite{palogiannidi2020end}, E2E SLU models were tested on unique wordings not seen in training." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="2011.13205" id="10047">
        <attvalues>
          <attvalue for="5" value=" We introduce OOV, CG, and microphone mismatch splits of the SLURP dataset \cite{bastianelli-etal-2020-slurp}, which we call SLURP For OOD generalisation (SLURPFOOD)." />
        </attvalues>
      </edge>
      <edge source="2407.07425" target="1703.01365" id="10048">
        <attvalues>
          <attvalue for="5" value=" We also leverage a model interpretability method (Integrated Gradients \cite{sundararajan2017axiomatic}) to determine the underlying causes contributing to the limited generalisation of the systems." />
        </attvalues>
      </edge>
      <edge source="2310.20620" target="2205.14217" id="10051">
        <attvalues>
          <attvalue for="5" value=" Previous studies show that the quality of continuous-output models highly depends on the choice of embeddings~\cite{Li-2022-DiffusionLM, tokarchuk-niculae-2022-target, kumar2018von}." />
        </attvalues>
      </edge>
      <edge source="2310.20620" target="1712.09405" id="10053">
        <attvalues>
          <attvalue for="5" value=" Text mapping to continuous space is widely explored in NLP and can be done using embeddings of tokens, words~\cite{turian-etal-2010-word,mikolov-w2v,mikolov-etal-2018-advances} and sentences~\cite{reimers-gurevych-2019-sentence,feng-etal-2022-language}." />
        </attvalues>
      </edge>
      <edge source="2310.20620" target="1908.10084" id="10054">
        <attvalues>
          <attvalue for="5" value=" Text mapping to continuous space is widely explored in NLP and can be done using embeddings of tokens, words~\cite{turian-etal-2010-word,mikolov-w2v,mikolov-etal-2018-advances} and sentences~\cite{reimers-gurevych-2019-sentence,feng-etal-2022-language}." />
        </attvalues>
      </edge>
      <edge source="2310.20620" target="2007.01852" id="10055">
        <attvalues>
          <attvalue for="5" value=" Text mapping to continuous space is widely explored in NLP and can be done using embeddings of tokens, words~\cite{turian-etal-2010-word,mikolov-w2v,mikolov-etal-2018-advances} and sentences~\cite{reimers-gurevych-2019-sentence,feng-etal-2022-language}." />
        </attvalues>
      </edge>
      <edge source="2310.20620" target="2005.09117" id="10056">
        <attvalues>
          <attvalue for="5" value=" \cite{arora-etal-2020-contextual} applied static random embeddings for text classification model's input; however, to the best of our knowledge, the effect of untrained random target embeddings has not been previously studied in the literature, especially for text-generating tasks such as machine translation." />
        </attvalues>
      </edge>
      <edge source="2105.02855" target="1906.01502" id="10057">
        <attvalues>
          <attvalue for="5" value=" Past work on multilingual learning has found that multilingual BERT~(mBERT; \cite{devlin2019-mbert}) generalizes across languages with high zero-shot transfer performance on a variety of tasks~\cite{pires-etal-2019-multilingual, wu-dredze-2019-beto}." />
        </attvalues>
      </edge>
      <edge source="2105.02855" target="2003.02912" id="10058">
        <attvalues>
          <attvalue for="5" value=" However, it has also been observed that high-resource languages included in mBERT pre-training often have a better-performing monolingual model, and low-resource languages that are not included in mBERT pre-training usually show poor performance \cite{nozza2020mask, wu-dredze-2020-languages}." />
        </attvalues>
      </edge>
      <edge source="2105.02855" target="1604.02201" id="10059">
        <attvalues>
          <attvalue for="5" value=" \cite{zoph-etal-2016-transfer} introduce a method for transferring a pre-trained machine translation model to lower-resource languages by only fine-tuning the lexical layer." />
        </attvalues>
      </edge>
      <edge source="2105.02855" target="2012.05628" id="10061">
        <attvalues>
          <attvalue for="5" value=" This method has also been applied to BERT \cite{artetxe-etal-2020-cross} and GPT-2 \cite{devries2020good}." />
        </attvalues>
      </edge>
      <edge source="2401.12005" target="2010.10392" id="10062">
        <attvalues>
          <attvalue for="5" value=" Examples include universal authorial embeddings using Siamese BERT \cite{Rivera-Soto_Miano_Ordonez_Chen_Khan_Bishop_Andrews_2021} and Character BERT \cite{El_Boukkouri_Ferret_Lavergne_Noji_Zweigenbaum_Tsujii_2020}, and using BERT for classification \cite{Fabien_Villatoro-Tello_Motlicek_Parida_2020,Tyo_Dhingra_Lipton_2022}." />
        </attvalues>
      </edge>
      <edge source="2401.12005" target="2209.06869" id="10063">
        <attvalues>
          <attvalue for="5" value=" We benchmark ALMs on the standard Blogs50 and CCAT50 datasets, following \cite{Tyo_Dhingra_Lipton_2022}, finding that our approach achieves state-of-the-art overall performance." />
        </attvalues>
      </edge>
      <edge source="2401.12005" target="2108.07258" id="10064">
        <attvalues>
          <attvalue for="5" value=" The task has gained prominence in recent years due to increasing concerns about the misuse of LLMs \cite{Bommasani_Hudson_Adeli_Altman_Arora_von_Arx_Bernstein_Bohg_Bosselut_Brunskill__2022,Gehrmann_Strobelt_Rush_2019,Tian_Chen_Wang_Bai_Zhang_Li_Xu_Wang_2023,Wu_Pang_Shen_Cheng_Chua_2023,Gehrmann_Strobelt_Rush_2019,Wu_Pang_Shen_Cheng_Chua_2023}." />
        </attvalues>
      </edge>
      <edge source="2401.12005" target="1906.04043" id="10065">
        <attvalues>
          <attvalue for="5" value=" Approaches include both fully automated detection and computer-assisted detection, eg, GLTR \cite{Gehrmann_Strobelt_Rush_2019} and GPTZero \cite{chakraborty_possibilities_2023}." />
        </attvalues>
      </edge>
      <edge source="2401.12005" target="2305.18149" id="10066">
        <attvalues>
          <attvalue for="5" value=" The task has gained prominence in recent years due to increasing concerns about the misuse of LLMs \cite{Bommasani_Hudson_Adeli_Altman_Arora_von_Arx_Bernstein_Bohg_Bosselut_Brunskill__2022,Gehrmann_Strobelt_Rush_2019,Tian_Chen_Wang_Bai_Zhang_Li_Xu_Wang_2023,Wu_Pang_Shen_Cheng_Chua_2023,Gehrmann_Strobelt_Rush_2019,Wu_Pang_Shen_Cheng_Chua_2023}." />
        </attvalues>
      </edge>
      <edge source="2401.12005" target="2304.04736" id="10067">
        <attvalues>
          <attvalue for="5" value=" Approaches include both fully automated detection and computer-assisted detection, eg, GLTR \cite{Gehrmann_Strobelt_Rush_2019} and GPTZero \cite{chakraborty_possibilities_2023}." />
        </attvalues>
      </edge>
      <edge source="2005.00851" target="1909.05330" id="10068">
        <attvalues>
          <attvalue for="5" value=" The identifier is to detect the input language from which to select or configure the decoder model of the corresponding language for the next recognition step \cite{go_2019_ml,8114354,6289013,8268945} or do a post-evaluation \cite{Waibel2000,Niesler2006LanguageIA} to select a result from outputs of the front decoder." />
        </attvalues>
      </edge>
      <edge source="2005.00851" target="1908.01060" id="10069">
        <attvalues>
          <attvalue for="5" value=" The first direction is to enhance the quality of monolingual models by leveraging or sharing resources from other languages \cite{go_2019_ml,2019_jh_ml,8114354}." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="1706.03762" id="10070">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Transformers \cite{vaswani2017attention} have demonstrated strong performance on text generation \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="2005.14165" id="10071">
        <attvalues>
          <attvalue for="5" value=" Autoregressive Transformers \cite{vaswani2017attention} have demonstrated strong performance on text generation \cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="2009.06489" id="10072">
        <attvalues>
          <attvalue for="5" value=" The success of self-attention in Transformers over recurrent models \cite{hochreiter1997long} can be attributed to its parallelizability \cite{hooker2021hardware} and its effective gradient propagation over many time steps \cite{ke2018sparse}." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="2006.16236" id="10074">
        <attvalues>
          <attvalue for="5" value=" \cite{katharopoulos2020transformers} propose feature maps to maintain positive outputs, while \cite{choromanski2020rethinking,peng2021random} carefully ensure their random feature maps are unbiased estimates of the softmax attention kernel." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="2009.14794" id="10075">
        <attvalues>
          <attvalue for="5" value=" \cite{katharopoulos2020transformers} propose feature maps to maintain positive outputs, while \cite{choromanski2020rethinking,peng2021random} carefully ensure their random feature maps are unbiased estimates of the softmax attention kernel." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="2102.11174" id="10077">
        <attvalues>
          <attvalue for="5" value=" \cite{schlag2021linear,peng2021random} propose more sophisticated update rules to forget information in the recurrent state to improve performance." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="2103.13076" id="10078">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{kasai2021finetuning} showed that pre-trained Transformers can be fine-tuned into a recurrent formulation using learned ReLU feature maps with minor degradations." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="1609.07843" id="10079">
        <attvalues>
          <attvalue for="5" value=" We also show competitive performance on WikiText-103 \cite{merity2016pointer} compared to more complex attention alternatives." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="1911.11423" id="10080">
        <attvalues>
          <attvalue for="5" value=" Our results support the idea \cite{merity2019single,zhai2021attention} that it is unnecessary for attention alternatives to maintain a close analogy to self-attention, and it is more important to focus on designing an expressive update rule." />
        </attvalues>
      </edge>
      <edge source="2210.04243" target="2105.14103" id="10081">
        <attvalues>
          <attvalue for="5" value=" Our results support the idea \cite{merity2019single,zhai2021attention} that it is unnecessary for attention alternatives to maintain a close analogy to self-attention, and it is more important to focus on designing an expressive update rule." />
        </attvalues>
      </edge>
      <edge source="2112.08593" target="1401.3841" id="10082">
        <attvalues>
          <attvalue for="5" value=" Early work on story generation used planning~\cite{meehan1976metanovel,lebowitz1987planning,cavazza2003interacting,porteous2009controlling,riedl2010narrative,ware2010modeling} or case-based reasoning~\cite{perez2001mexica,peinado2005creativity,turner2014creative}." />
        </attvalues>
      </edge>
      <edge source="2112.08593" target="1705.03557" id="10083">
        <attvalues>
          <attvalue for="5" value=" Recently, large pre-trained neural language models have been applied to story generation because they circumvent the need for knowledge engineering and tend to produce relatively fluent, varied, and naturalistic language~\cite{roemmele2016writing,khalifa2017deeptingle,clark2018neural,martin2018event}." />
        </attvalues>
      </edge>
      <edge source="2112.08593" target="1706.01331" id="10084">
        <attvalues>
          <attvalue for="5" value=" Previous attempts to enhance the coherence of generated stories and control the trajectory of the story use conditioning on content-relevant features such as plot outlines~\cite{fan2018hierarchical,peng2018towards,rashkin2020plotmachines} or by hierarchical reasoning with abstract representations that help constrain story progression~\cite{martin2018event,yao2019plan,fan2019strategies,peng2021inferring}." />
        </attvalues>
      </edge>
      <edge source="2112.08593" target="1805.04833" id="10085">
        <attvalues>
          <attvalue for="5" value=" Previous attempts to enhance the coherence of generated stories and control the trajectory of the story use conditioning on content-relevant features such as plot outlines~\cite{fan2018hierarchical,peng2018towards,rashkin2020plotmachines} or by hierarchical reasoning with abstract representations that help constrain story progression~\cite{martin2018event,yao2019plan,fan2019strategies,peng2021inferring}." />
        </attvalues>
      </edge>
      <edge source="2112.08593" target="2004.14967" id="10086">
        <attvalues>
          <attvalue for="5" value=" Previous attempts to enhance the coherence of generated stories and control the trajectory of the story use conditioning on content-relevant features such as plot outlines~\cite{fan2018hierarchical,peng2018towards,rashkin2020plotmachines} or by hierarchical reasoning with abstract representations that help constrain story progression~\cite{martin2018event,yao2019plan,fan2019strategies,peng2021inferring}." />
        </attvalues>
      </edge>
      <edge source="2112.08593" target="2105.01311" id="10089">
        <attvalues>
          <attvalue for="5" value=" Previous attempts to enhance the coherence of generated stories and control the trajectory of the story use conditioning on content-relevant features such as plot outlines~\cite{fan2018hierarchical,peng2018towards,rashkin2020plotmachines} or by hierarchical reasoning with abstract representations that help constrain story progression~\cite{martin2018event,yao2019plan,fan2019strategies,peng2021inferring}." />
        </attvalues>
      </edge>
      <edge source="2112.08593" target="1909.03480" id="10091">
        <attvalues>
          <attvalue for="5" value=" We focus on evaluating our system in the domain of science fiction plots~\cite{ammanabrolu2020story}, consistent with prior work~\cite{tambwekar2019controllable,ammanabrolu2020story}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="1908.10084" id="10092">
        <attvalues>
          <attvalue for="5" value=" Language models perform well on these tasks but typically require fine-tuning on the downstream task and corpora ~\cite{reimers-gurevych-2019-sentence,devlin2018bert,pfeiffer2020AdapterHub,mosbach2021on}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2006.04884" id="10095">
        <attvalues>
          <attvalue for="5" value=" Language models perform well on these tasks but typically require fine-tuning on the downstream task and corpora ~\cite{reimers-gurevych-2019-sentence,devlin2018bert,pfeiffer2020AdapterHub,mosbach2021on}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="1807.03748" id="10096">
        <attvalues>
          <attvalue for="5" value=" In terms of sentence embeddings, contrastive learning schemes have already been adopted successfully~\cite{vanDenOord2018,liu-etal-2021-fast,gao2021simcse,Carlsson2021ICLR}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2104.08027" id="10097">
        <attvalues>
          <attvalue for="5" value=" Most recently, augmentation was also proposed in a more continuous fashion operating in a parameter level via simple techniques such as drop-out~\cite{gao2021simcse,liu-etal-2021-fast,klein2022scd} or random span masking~\cite{liu-etal-2021-fast}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2104.08821" id="10098">
        <attvalues>
          <attvalue for="5" value=" Most recently, augmentation was also proposed in a more continuous fashion operating in a parameter level via simple techniques such as drop-out~\cite{gao2021simcse,liu-etal-2021-fast,klein2022scd} or random span masking~\cite{liu-etal-2021-fast}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2205.00656" id="10099">
        <attvalues>
          <attvalue for="5" value=" Other related work include~\cite{zhang2022unsupervised,zhou-etal-2022-debiased,zhang2022contrastive,liu2022transencoder}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="1703.02573" id="10100">
        <attvalues>
          <attvalue for="5" value="a discrete augmentation), which comprises word level operations such as swapping, insertion, deletion, and substitution~\cite{XieICLR2017,Coulombe2018TextDA,wei-zou-2019-eda}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="1812.04718" id="10101">
        <attvalues>
          <attvalue for="5" value="a discrete augmentation), which comprises word level operations such as swapping, insertion, deletion, and substitution~\cite{XieICLR2017,Coulombe2018TextDA,wei-zou-2019-eda}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="1901.11196" id="10102">
        <attvalues>
          <attvalue for="5" value="a discrete augmentation), which comprises word level operations such as swapping, insertion, deletion, and substitution~\cite{XieICLR2017,Coulombe2018TextDA,wei-zou-2019-eda}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2004.12239" id="10103">
        <attvalues>
          <attvalue for="5" value=" In contrast to that, continuous augmentation operates at the representation level, comprising approaches like interpolation or ``mixup'' on the embedding space~\cite{chen-etal-2020-mixtext, cheng-etal-2020-advaug, Guo2019AugmentingDW}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2203.07847" id="10104">
        <attvalues>
          <attvalue for="5" value=" Most recently, augmentation was also proposed in a more continuous fashion operating in a parameter level via simple techniques such as drop-out~\cite{gao2021simcse,liu-etal-2021-fast,klein2022scd} or random span masking~\cite{liu-etal-2021-fast}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2010.10604" id="10105">
        <attvalues>
          <attvalue for="5" value=" Similar to ~\cite{NEURIPS2020_bcff3f63}, we adopt the Log-Normal distribution to model attention." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="1911.05722" id="10106">
        <attvalues>
          <attvalue for="5" value=" To this end, we utilize momentum contrastive learning to generate harder negatives~\cite{MoCo_He2020}." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2110.08552" id="10108">
        <attvalues>
          <attvalue for="5" value=" Previous works: Recently, VaSCL~\cite{zhang-etal-2022-virtual}, ConSERT~\cite{yan-etal-2021-consert}, PCL~\cite{WuPCL22} and ~\cite{chuang2022diffcse} proposed contrastive representation learning with diverse augmentation strategies on positive pair." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2105.11741" id="10109">
        <attvalues>
          <attvalue for="5" value=" Previous works: Recently, VaSCL~\cite{zhang-etal-2022-virtual}, ConSERT~\cite{yan-etal-2021-consert}, PCL~\cite{WuPCL22} and ~\cite{chuang2022diffcse} proposed contrastive representation learning with diverse augmentation strategies on positive pair." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2201.12093" id="10110">
        <attvalues>
          <attvalue for="5" value=" Previous works: Recently, VaSCL~\cite{zhang-etal-2022-virtual}, ConSERT~\cite{yan-etal-2021-consert}, PCL~\cite{WuPCL22} and ~\cite{chuang2022diffcse} proposed contrastive representation learning with diverse augmentation strategies on positive pair." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2204.10298" id="10111">
        <attvalues>
          <attvalue for="5" value=" Previous works: Recently, VaSCL~\cite{zhang-etal-2022-virtual}, ConSERT~\cite{yan-etal-2021-consert}, PCL~\cite{WuPCL22} and ~\cite{chuang2022diffcse} proposed contrastive representation learning with diverse augmentation strategies on positive pair." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2109.04380" id="10112">
        <attvalues>
          <attvalue for="5" value=" Similar to us, ESimCSE~\cite{wu2021esimcse} and MoCoSE~\cite{cao-etal-2022-exploring} proposed to exploit a momentum contrastive learning model with negative sample queue for sentence embedding to boost uniformity of the representations." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2202.13093" id="10113">
        <attvalues>
          <attvalue for="5" value=" Similar to us, ESimCSE~\cite{wu2021esimcse} and MoCoSE~\cite{cao-etal-2022-exploring} proposed to exploit a momentum contrastive learning model with negative sample queue for sentence embedding to boost uniformity of the representations." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2104.12565" id="10116">
        <attvalues>
          <attvalue for="5" value=" Our model differs from this paper and the method in \cite{bachman2019learning,yang2021mutual,zhang-etal-2020-unsupervised,sordoni2021decomposed,wu2020mutual}, which focuses on using mutual information for self-supervised learning." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2009.12061" id="10117">
        <attvalues>
          <attvalue for="5" value=" Our model differs from this paper and the method in \cite{bachman2019learning,yang2021mutual,zhang-etal-2020-unsupervised,sordoni2021decomposed,wu2020mutual}, which focuses on using mutual information for self-supervised learning." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2106.13401" id="10118">
        <attvalues>
          <attvalue for="5" value=" Our model differs from this paper and the method in \cite{bachman2019learning,yang2021mutual,zhang-etal-2020-unsupervised,sordoni2021decomposed,wu2020mutual}, which focuses on using mutual information for self-supervised learning." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2005.13149" id="10119">
        <attvalues>
          <attvalue for="5" value=" Our model differs from this paper and the method in \cite{bachman2019learning,yang2021mutual,zhang-etal-2020-unsupervised,sordoni2021decomposed,wu2020mutual}, which focuses on using mutual information for self-supervised learning." />
        </attvalues>
      </edge>
      <edge source="2211.04928" target="2109.13059" id="10120">
        <attvalues>
          <attvalue for="5" value=" Other related work include~\cite{zhang2022unsupervised,zhou-etal-2022-debiased,zhang2022contrastive,liu2022transencoder}." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2303.08774" id="10121">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) such as ChatGPT \cite{DBLP:journals/corr/abs-2303-08774}, Llama \cite{touvron2023llama} have changed the landscape of AI research because of their groundbreaking capabilities." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2307.09288" id="10122">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) such as ChatGPT \cite{DBLP:journals/corr/abs-2303-08774}, Llama \cite{touvron2023llama} have changed the landscape of AI research because of their groundbreaking capabilities." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2303.00456" id="10123">
        <attvalues>
          <attvalue for="5" value=" Along with this line of research, different LLMs have been explored, eg T5 \cite{ma2023n}, Llama \cite{radhakrishnan2023whispering} and ChatGPT \cite{ma2023can}." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2202.12837" id="10124">
        <attvalues>
          <attvalue for="5" value=" These methods can be roughly categorized into two groups depending on whether to re-train LLMs: i) fine-tuning LLMs \cite{ma2023n, radhakrishnan2023whispering} and ii) in-context learning of LLMs \cite{min2022rethinking}, which utilizes prompts without changing parameters of LLMs \cite{ma2023can, yang2023generative}." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2307.04172" id="10125">
        <attvalues>
          <attvalue for="5" value=" On LibriSpeech, when the word-error-rate (WER) is lower than 2$\%$, there is very limited improvement that previous methods can contribute \cite{ma2023can, chen2023hyporadise}." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2310.06434" id="10126">
        <attvalues>
          <attvalue for="5" value=" Along with this line of research, different LLMs have been explored, eg T5 \cite{ma2023n}, Llama \cite{radhakrishnan2023whispering} and ChatGPT \cite{ma2023can}." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2402.05457" id="10127">
        <attvalues>
          <attvalue for="5" value=" Because of the discrepancy between spoken language and written language, LLM-based correction can hinder the fidelity of ASR transcriptions \cite{chen2024its}; ii) multi-step reasoning challenge for LLMs." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2203.07172" id="10128">
        <attvalues>
          <attvalue for="5" value=" Contrary to existing approaches \cite{gekhman2022red, qiu2021learning, li2021confidence}, we extract the confidence and uncertainty information from N-best list hypotheses and identify them by thresholding the obtained confidence scores." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2305.18654" id="10129">
        <attvalues>
          <attvalue for="5" value=" For the second issue of multi-step reasoning, it is a well-known challenge for LLMs \cite{dziri2023faith, srivastava2023beyond}." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2201.11903" id="10130">
        <attvalues>
          <attvalue for="5" value=" The widely-used prompting techniques, chain-of-thought (CoT) prompting \cite{wei2022chain} and the prefix `let's think step by step' \cite{kojima2022large}, are not best suited for our ASR correction task." />
        </attvalues>
      </edge>
      <edge source="2310.11532" target="2309.15649" id="10133">
        <attvalues>
          <attvalue for="5" value=" To accommodate this complexity, \cite{yang2023generative} proposed a task-activating prompting (TAP), which needs four rounds of Q and A to properly set up the correction task." />
        </attvalues>
      </edge>
      <edge source="2106.08159" target="1706.04599" id="10135">
        <attvalues>
          <attvalue for="5" value=" One popular technique for post-hoc calibration of neural networks is temperature scaling \cite{guo2017calibration}." />
        </attvalues>
      </edge>
      <edge source="2204.00498" target="2010.12725" id="10136">
        <attvalues>
          <attvalue for="5" value=" A clear trend in this area is to finetune models pretrained on natural language; notably, performance significantly improves as larger pretrained models are used \cite{shaw-etal-2021-compositional,scholak2021picard}." />
        </attvalues>
      </edge>
      <edge source="2204.00498" target="2109.05093" id="10137">
        <attvalues>
          <attvalue for="5" value=" A clear trend in this area is to finetune models pretrained on natural language; notably, performance significantly improves as larger pretrained models are used \cite{shaw-etal-2021-compositional,scholak2021picard}." />
        </attvalues>
      </edge>
      <edge source="2204.00498" target="2005.14165" id="10138">
        <attvalues>
          <attvalue for="5" value=" Recent results from the broader field demonstrate that simply scaling training data and model size for generative language models brings advanced capabilities, such as few-shot learning without finetuning \cite[GPT-3, ][]{brown2020language} and code generation \cite[Codex, ][]{chen2021evaluating}." />
        </attvalues>
      </edge>
      <edge source="2204.00498" target="2107.03374" id="10139">
        <attvalues>
          <attvalue for="5" value=" Recent results from the broader field demonstrate that simply scaling training data and model size for generative language models brings advanced capabilities, such as few-shot learning without finetuning \cite[GPT-3, ][]{brown2020language} and code generation \cite[Codex, ][]{chen2021evaluating}." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2112.10752" id="10140">
        <attvalues>
          <attvalue for="5" value=" In domains of continuous signals, diffusion probabilistic models have shown great success in rendering photorealistic images~\cite{rombach2021highresolution,ramesh2022dalle2}, immersive videos~\cite{bar2024lumiere} and synthesizing high-quality audio~\cite{kong2020diffwave} through iterative denoising, outperforming GANs and autoregressive (AR) models, even contributing to the surge of AI art." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2204.06125" id="10141">
        <attvalues>
          <attvalue for="5" value=" In domains of continuous signals, diffusion probabilistic models have shown great success in rendering photorealistic images~\cite{rombach2021highresolution,ramesh2022dalle2}, immersive videos~\cite{bar2024lumiere} and synthesizing high-quality audio~\cite{kong2020diffwave} through iterative denoising, outperforming GANs and autoregressive (AR) models, even contributing to the surge of AI art." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2401.12945" id="10142">
        <attvalues>
          <attvalue for="5" value=" In domains of continuous signals, diffusion probabilistic models have shown great success in rendering photorealistic images~\cite{rombach2021highresolution,ramesh2022dalle2}, immersive videos~\cite{bar2024lumiere} and synthesizing high-quality audio~\cite{kong2020diffwave} through iterative denoising, outperforming GANs and autoregressive (AR) models, even contributing to the surge of AI art." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2009.09761" id="10143">
        <attvalues>
          <attvalue for="5" value=" In domains of continuous signals, diffusion probabilistic models have shown great success in rendering photorealistic images~\cite{rombach2021highresolution,ramesh2022dalle2}, immersive videos~\cite{bar2024lumiere} and synthesizing high-quality audio~\cite{kong2020diffwave} through iterative denoising, outperforming GANs and autoregressive (AR) models, even contributing to the surge of AI art." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2005.14165" id="10144">
        <attvalues>
          <attvalue for="5" value=" The story is different in the domains of discrete signals comprising symbolic sequences such as natural languages, where AR large language models~\cite[large language models or LLMs,][]{brown2020lgpt3,openai2023gpt4} have dominated the scene, delivering impressive generalist language abilities in language understanding and generating human-like texts, and can even follow natural language instructions to perform unseen tasks." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2303.08774" id="10145">
        <attvalues>
          <attvalue for="5" value=" The story is different in the domains of discrete signals comprising symbolic sequences such as natural languages, where AR large language models~\cite[large language models or LLMs,][]{brown2020lgpt3,openai2023gpt4} have dominated the scene, delivering impressive generalist language abilities in language understanding and generating human-like texts, and can even follow natural language instructions to perform unseen tasks." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2310.01218" id="10146">
        <attvalues>
          <attvalue for="5" value=" While many recent endeavors try unifying the generation paradigms by enabling large language models to draw~\cite{ge2023seed} or speak~\cite{zhang2023speechgpt}, few explore generating discrete sequences such as languages with diffusion models." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2305.11000" id="10147">
        <attvalues>
          <attvalue for="5" value=" While many recent endeavors try unifying the generation paradigms by enabling large language models to draw~\cite{ge2023seed} or speak~\cite{zhang2023speechgpt}, few explore generating discrete sequences such as languages with diffusion models." />
        </attvalues>
      </edge>
      <edge source="2308.12219" target="2109.01652" id="10149">
        <attvalues>
          <attvalue for="5" value=" We then reprogram pre-trained masked LMs into {Diffusion-LLM}s via diffusive adaptation, where task-specific finetuning and instruction finetuning~\cite{wei2021flanv1} are explored for solving certain downstream tasks or general language problems, showing {Diffusion-LLM}s benefit from pre-training on large scale data." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="2106.15561" id="10150">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text-to-speech technology has demonstrated significant successes in generating high-quality speech with good naturalness and expressiveness for a target speaker \cite{tan2021survey,wang2017tacotron,shen2018natural,elias2021parallel,ren2020fastspeech}." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="1703.10135" id="10151">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text-to-speech technology has demonstrated significant successes in generating high-quality speech with good naturalness and expressiveness for a target speaker \cite{tan2021survey,wang2017tacotron,shen2018natural,elias2021parallel,ren2020fastspeech}." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="1712.05884" id="10152">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text-to-speech technology has demonstrated significant successes in generating high-quality speech with good naturalness and expressiveness for a target speaker \cite{tan2021survey,wang2017tacotron,shen2018natural,elias2021parallel,ren2020fastspeech}." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="2010.11439" id="10153">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text-to-speech technology has demonstrated significant successes in generating high-quality speech with good naturalness and expressiveness for a target speaker \cite{tan2021survey,wang2017tacotron,shen2018natural,elias2021parallel,ren2020fastspeech}." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="2006.04558" id="10154">
        <attvalues>
          <attvalue for="5" value=" In recent years, neural text-to-speech technology has demonstrated significant successes in generating high-quality speech with good naturalness and expressiveness for a target speaker \cite{tan2021survey,wang2017tacotron,shen2018natural,elias2021parallel,ren2020fastspeech}." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="1906.00672" id="10155">
        <attvalues>
          <attvalue for="5" value=" Their performance may deteriorate substantially on domain-mismatched text \cite{he2019robust} due to the limited content and domain coverage of training data." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="1808.10128" id="10156">
        <attvalues>
          <attvalue for="5" value=" However, it was noticed that the actual benefits of using training speech from other speakers could be uncertain and unstable \cite{chung2019semi}." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="1802.06006" id="10157">
        <attvalues>
          <attvalue for="5" value=" Leveraging large amount of non-target speakers' data from different sources has become a common and appealing approach to developing high-performance TTS systems when training data from the target speaker(s) are limited \cite{chung2019semi,arik2018neural,chen2018sample,cooper2020zero, tan2021cuhk}." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="1809.10460" id="10158">
        <attvalues>
          <attvalue for="5" value=" Leveraging large amount of non-target speakers' data from different sources has become a common and appealing approach to developing high-performance TTS systems when training data from the target speaker(s) are limited \cite{chung2019semi,arik2018neural,chen2018sample,cooper2020zero, tan2021cuhk}." />
        </attvalues>
      </edge>
      <edge source="2110.03857" target="1910.10838" id="10159">
        <attvalues>
          <attvalue for="5" value=" Leveraging large amount of non-target speakers' data from different sources has become a common and appealing approach to developing high-performance TTS systems when training data from the target speaker(s) are limited \cite{chung2019semi,arik2018neural,chen2018sample,cooper2020zero, tan2021cuhk}." />
        </attvalues>
      </edge>
      <edge source="2005.01107" target="1409.3215" id="10160">
        <attvalues>
          <attvalue for="5" value=" The most widely-used techniques are Deep Learning-based approaches involving Sequence-to-Sequence (Seq2Seq) \cite{sutskever2014sequence} models." />
        </attvalues>
      </edge>
      <edge source="2005.01107" target="1704.01792" id="10161">
        <attvalues>
          <attvalue for="5" value=" These include the usage of extra linguistic features \cite{zhou2017Neural} or the introduction of answer-awareness \cite{zhao2018paragraph,du2017learning,dong2019unified}, which uses the answer to the desired question, or the position of the answer within the context paragraph as additional features." />
        </attvalues>
      </edge>
      <edge source="2005.01107" target="1705.00106" id="10162">
        <attvalues>
          <attvalue for="5" value=" These include the usage of extra linguistic features \cite{zhou2017Neural} or the introduction of answer-awareness \cite{zhao2018paragraph,du2017learning,dong2019unified}, which uses the answer to the desired question, or the position of the answer within the context paragraph as additional features." />
        </attvalues>
      </edge>
      <edge source="2005.01107" target="1905.03197" id="10163">
        <attvalues>
          <attvalue for="5" value=" The use of Transformers \cite{vaswani2017attention} over standard RNNs have also been adopted as these models provide the power of Attention in order to refer to specific points of context within the context paragraph, alleviating the RNN's memory bottleneck \cite{dong2019unified}." />
        </attvalues>
      </edge>
      <edge source="2005.01107" target="1705.02012" id="10164">
        <attvalues>
          <attvalue for="5" value=" Reinforcement Learning (RL) have produced consistent results for the task by using policy gradients \cite{yuan2017machine}." />
        </attvalues>
      </edge>
      <edge source="2005.01107" target="1706.03762" id="10165">
        <attvalues>
          <attvalue for="5" value=" The use of Transformers \cite{vaswani2017attention} over standard RNNs have also been adopted as these models provide the power of Attention in order to refer to specific points of context within the context paragraph, alleviating the RNN's memory bottleneck \cite{dong2019unified}." />
        </attvalues>
      </edge>
      <edge source="2005.01107" target="1606.05250" id="10166">
        <attvalues>
          <attvalue for="5" value=" We benchmark standard language model finetuning on a reformatting of the SQuAD \cite{rajpurkar-etal-2016-squad} v." />
        </attvalues>
      </edge>
      <edge source="2104.06483" target="2006.11572" id="10167">
        <attvalues>
          <attvalue for="5" value=" This is true even for datasets where all words inflect in the same way---eg there are no inflectional classes or allomorphs of morphemes, as is found in the low-resource Niger-Congo dataset used in SIGMORPHON 2020 shared task \cite{vylomova-etal-2020-sigmorphon}." />
        </attvalues>
      </edge>
      <edge source="2104.06483" target="2005.10213" id="10168">
        <attvalues>
          <attvalue for="5" value=" Especially on inflection tasks, where an input lemma such as {dog}, and input inflectional features such as \{{N,PL}\}, are expected to produce an output such as {dogs}, the model has shown to be particularly adept at generalizing patterns \cite{wu2020applying,liu-hulden-2020-analogy}." />
        </attvalues>
      </edge>
      <edge source="2104.06483" target="1705.06106" id="10169">
        <attvalues>
          <attvalue for="5" value=" It has been noted earlier that neural sequence-to-sequence models are apt to perform poorly if they have been exposed to little training data and that autoencoding on hallucinated forms could be useful \cite{kann-schutze-2017-unlabeled}." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="2005.14165" id="10170">
        <attvalues>
          <attvalue for="5" value=" Recent advances in Large Language Models (LLMs)~\cite{brown-etal-2020-gpt3,chowdhery-etal-2022-palm}, also known as Foundation Models~\cite{bommasani-etal-2022-foundation}, have challenged the traditional supervised learning paradigm of fine-tuning by demonstrating emergent zero-shot Natural Language Understanding (NLU) capabilities~\cite{wei2022emergent} through scaling the model's size in billions of parameters~\cite{kaplan-etal-2020-scale}." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="2204.02311" id="10171">
        <attvalues>
          <attvalue for="5" value=" Recent advances in Large Language Models (LLMs)~\cite{brown-etal-2020-gpt3,chowdhery-etal-2022-palm}, also known as Foundation Models~\cite{bommasani-etal-2022-foundation}, have challenged the traditional supervised learning paradigm of fine-tuning by demonstrating emergent zero-shot Natural Language Understanding (NLU) capabilities~\cite{wei2022emergent} through scaling the model's size in billions of parameters~\cite{kaplan-etal-2020-scale}." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="2108.07258" id="10172">
        <attvalues>
          <attvalue for="5" value=" Recent advances in Large Language Models (LLMs)~\cite{brown-etal-2020-gpt3,chowdhery-etal-2022-palm}, also known as Foundation Models~\cite{bommasani-etal-2022-foundation}, have challenged the traditional supervised learning paradigm of fine-tuning by demonstrating emergent zero-shot Natural Language Understanding (NLU) capabilities~\cite{wei2022emergent} through scaling the model's size in billions of parameters~\cite{kaplan-etal-2020-scale}." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="2001.08361" id="10174">
        <attvalues>
          <attvalue for="5" value=" Recent advances in Large Language Models (LLMs)~\cite{brown-etal-2020-gpt3,chowdhery-etal-2022-palm}, also known as Foundation Models~\cite{bommasani-etal-2022-foundation}, have challenged the traditional supervised learning paradigm of fine-tuning by demonstrating emergent zero-shot Natural Language Understanding (NLU) capabilities~\cite{wei2022emergent} through scaling the model's size in billions of parameters~\cite{kaplan-etal-2020-scale}." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="1706.03741" id="10176">
        <attvalues>
          <attvalue for="5" value="5- models, is an instruction-following transformer-based language model, which has been further trained (aligned) with reinforcement learning from human feedback (RLHF) \cite{christiano-etal-2017-drl}." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="2212.14402" id="10177">
        <attvalues>
          <attvalue for="5" value=" Following the work of \cite{bommarito-bar-exams-2023}, we evaluate the latest OpenAI's GPT-3." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="2301.04408" id="10178">
        <attvalues>
          <attvalue for="5" value=" In a follow-up work, \cite{bommarito-etal-2023-gpt} assessed the model's performance in accounting certification exams, where the model significantly under-performs human capabilities with a correct rate of 14." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="2110.00976" id="10179">
        <attvalues>
          <attvalue for="5" value=" March 2023), the first available ChatGPT, on legal text classification tasks from the LexGLUE \cite{chalkidis-etal-2022-lexglue} benchmark in a zero-shot fashion providing examples in a templated instruction-following format, similar to those used by \cite{chung-etal-2023-flant5}." />
        </attvalues>
      </edge>
      <edge source="2304.12202" target="2210.11416" id="10180">
        <attvalues>
          <attvalue for="5" value=" March 2023), the first available ChatGPT, on legal text classification tasks from the LexGLUE \cite{chalkidis-etal-2022-lexglue} benchmark in a zero-shot fashion providing examples in a templated instruction-following format, similar to those used by \cite{chung-etal-2023-flant5}." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="1805.07685" id="10181">
        <attvalues>
          <attvalue for="5" value=" First work on detoxification was a sequence-to-sequence collaborative classifier, attention and the cycle consistency loss \cite{nogueira-dos-santos-etal-2018-fighting}." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="2105.09052" id="10182">
        <attvalues>
          <attvalue for="5" value=" There exist a variety of Textual Style Transfer methods: from totally supervised methods \cite{Wang2019HarnessingPN,zhang2020parallel,dementieva2021methods} which require a parallel text corpus for training to unsupervised \cite{shen2017style,wang2019controllable,xu2021vae} that are designed to work without any parallel data." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="1803.06535" id="10183">
        <attvalues>
          <attvalue for="5" value=" There is a lack of parallel data for Textual Style Transfer since there exist only few parallel datasets for English \cite{rao-tetreault-2018-dear} and some other languages \cite{briakou2021xformal}." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="2104.05767" id="10184">
        <attvalues>
          <attvalue for="5" value=" Given that, Textual Style Transfer has now a lot of specific subtasks ranging from formality style transfer \cite{rao-tetreault-2018-dear, yao2021improving} and simplification of domain-specific texts \cite{devaraj2021paragraphlevel, maddela2021controllable} to emotion modification \cite{sharma2021facilitating} and detoxification (debiasing) \cite{li2020stylecontent, dementieva2021methods}." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="2101.07714" id="10185">
        <attvalues>
          <attvalue for="5" value=" Given that, Textual Style Transfer has now a lot of specific subtasks ranging from formality style transfer \cite{rao-tetreault-2018-dear, yao2021improving} and simplification of domain-specific texts \cite{devaraj2021paragraphlevel, maddela2021controllable} to emotion modification \cite{sharma2021facilitating} and detoxification (debiasing) \cite{li2020stylecontent, dementieva2021methods}." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="2012.07419" id="10186">
        <attvalues>
          <attvalue for="5" value=" Given that, Textual Style Transfer has now a lot of specific subtasks ranging from formality style transfer \cite{rao-tetreault-2018-dear, yao2021improving} and simplification of domain-specific texts \cite{devaraj2021paragraphlevel, maddela2021controllable} to emotion modification \cite{sharma2021facilitating} and detoxification (debiasing) \cite{li2020stylecontent, dementieva2021methods}." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="2005.07522" id="10187">
        <attvalues>
          <attvalue for="5" value=" There exist a variety of Textual Style Transfer methods: from totally supervised methods \cite{Wang2019HarnessingPN,zhang2020parallel,dementieva2021methods} which require a parallel text corpus for training to unsupervised \cite{shen2017style,wang2019controllable,xu2021vae} that are designed to work without any parallel data." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="1809.06284" id="10191">
        <attvalues>
          <attvalue for="5" value=" On the other hand, if we address Textual Style Transfer task as a Machine Translation task we get a significant performance boost \cite{prabhumoye2018style}." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="2102.05456" id="10192">
        <attvalues>
          <attvalue for="5" value=" A recent work by \cite{laugier2021civil} introduces self-supervised model based on T$5$ model \cite{raffel2020exploring} with a denoising and cyclic auto-encoder loss." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="2001.08210" id="10194">
        <attvalues>
          <attvalue for="5" value=" Multilingual language models such as mBART \cite{liu2020multilingual}, mT5 \cite{xue2021mt5} have recently become available." />
        </attvalues>
      </edge>
      <edge source="2206.02252" target="2010.11934" id="10195">
        <attvalues>
          <attvalue for="5" value=" Multilingual language models such as mBART \cite{liu2020multilingual}, mT5 \cite{xue2021mt5} have recently become available." />
        </attvalues>
      </edge>
      <edge source="2207.03391" target="1909.05330" id="10197">
        <attvalues>
          <attvalue for="5" value=" For end-to-end (e2e) speech recognition, sequence-to-sequence models \cite{cho18}, RNN transducers (RNN-T) \cite{kannan2019}, transformers \cite{vishwas20} and unsupervised learning \cite{conneau21} are being used." />
        </attvalues>
      </edge>
      <edge source="2207.03391" target="2007.03001" id="10199">
        <attvalues>
          <attvalue for="5" value=" Previous work on e2e multilingual speech recognition systems has shown that a multilingual setup does not guarantee the reduction in Word Error Rate (WER) for target languages \cite{conneau21,Pratap2020,hou20}." />
        </attvalues>
      </edge>
      <edge source="2207.03391" target="2005.08118" id="10200">
        <attvalues>
          <attvalue for="5" value=" Recent efforts to interpret the learning of multilingual speech recognition systems \cite{zelasko20,feng21} observe that Phoneme Error Rate (PER) of an overlapped phoneme is not reduced with the growing number of sharing languages." />
        </attvalues>
      </edge>
      <edge source="2207.03391" target="2010.12104" id="10201">
        <attvalues>
          <attvalue for="5" value=" Recent efforts to interpret the learning of multilingual speech recognition systems \cite{zelasko20,feng21} observe that Phoneme Error Rate (PER) of an overlapped phoneme is not reduced with the growing number of sharing languages." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="1904.05862" id="10203">
        <attvalues>
          <attvalue for="5" value=" Examples of models trained by self-supervised learning, which we refer to as self-supervised models, include wav2vec~\cite{schneider2020wav2vec, baevski2020wav2vec2}, HuBERT~\cite{hsu2021hubert}, and WavLM~\cite{chen2022wavlm}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2002.05709" id="10204">
        <attvalues>
          <attvalue for="5" value=" The main idea of this approach is to leverage the inherent structures and patterns within the speech data to train models via representation learning loss such as contrastive loss~\cite{chen2020simclr, jiang2021speechsimclr, huh2020augmentation, inoue2020semi}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2106.07447" id="10205">
        <attvalues>
          <attvalue for="5" value="0~\cite{baevski2020wav2vec2}, HuBERT~\cite{hsu2021hubert}, ContentVec~\cite{qian2022contentvec}, and WavLM+~\cite{chen2022wavlm}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2110.13900" id="10206">
        <attvalues>
          <attvalue for="5" value=" \item We thoroughly conducted experimental evaluation with multiple conventional fine-tuning methods including weight tuning~\cite{chen2022wavlm}, LoRA tuning~\cite{hu22lora}, Prefix tuning~\cite{li21prefixtuning}, and Efficient adapter tuning~\cite{speech-adapter}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2109.15053" id="10207">
        <attvalues>
          <attvalue for="5" value=" For example, a number of studies have proposed methods that utilize speech embeddings extracted from self-supervised models for discriminative tasks such as speaker verification\cite{finetune-sv, fan21wav2vecsv, lee22wav2vecsv, peng2023improving} and speech emotion recognition~\cite{finetune-er, pepino21wav2vecer}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2104.03502" id="10208">
        <attvalues>
          <attvalue for="5" value=" For example, a number of studies have proposed methods that utilize speech embeddings extracted from self-supervised models for discriminative tasks such as speaker verification\cite{finetune-sv, fan21wav2vecsv, lee22wav2vecsv, peng2023improving} and speech emotion recognition~\cite{finetune-er, pepino21wav2vecer}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2109.03381" id="10209">
        <attvalues>
          <attvalue for="5" value=" For example, spoken question answering is an important line of research focused on developing models capable of understanding and responding to questions posed in natural spoken language, where recent studies leverage self-supervised models~\cite{sqa1, sqa2, sqa3, sqa4, sqa5, sqa6}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="1902.00751" id="10210">
        <attvalues>
          <attvalue for="5" value=" The first adapter tuning method~\cite{nlp-adapter} was proposed for BERT \cite{bert} in the field of natural language processing, where two adapter modules are inserted into each encoder layer of BERT." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2004.03829" id="10212">
        <attvalues>
          <attvalue for="5" value=" A number of follow-up studies have used adapters for various natural language processing tasks \cite{lin2020exploring, Guo2021AdaptiveAdapters, Zhang2023PoE}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="1909.05330" id="10213">
        <attvalues>
          <attvalue for="5" value="}~\cite{rnnt-adapter} integrated adapter modules into recurrent neural network transducers." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2012.01687" id="10214">
        <attvalues>
          <attvalue for="5" value="}~\cite{ctcattn-adapter} proposed the adapt-and-adjust framework, which uses adapter modules for multilingual speech recognition based on hybrid connectionist temporal classification (CTC)-attention networks." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2204.09883" id="10215">
        <attvalues>
          <attvalue for="5" value="}~\cite{Qian2022LayerWiseFastAdaptation} proposed gated and multi-basis adapters for multi-accent speech recognition." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2202.03218" id="10217">
        <attvalues>
          <attvalue for="5" value=" \item We thoroughly conducted experimental evaluation with multiple conventional fine-tuning methods including weight tuning~\cite{chen2022wavlm}, LoRA tuning~\cite{hu22lora}, Prefix tuning~\cite{li21prefixtuning}, and Efficient adapter tuning~\cite{speech-adapter}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2210.06175" id="10218">
        <attvalues>
          <attvalue for="5" value="}~\cite{Chen2022EfficientTuning} compared the adapter modules with other efficient fine-tuning methods such as low-rank adaptation (LoRA)~\cite{hu22lora}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2106.09685" id="10219">
        <attvalues>
          <attvalue for="5" value=" \item We thoroughly conducted experimental evaluation with multiple conventional fine-tuning methods including weight tuning~\cite{chen2022wavlm}, LoRA tuning~\cite{hu22lora}, Prefix tuning~\cite{li21prefixtuning}, and Efficient adapter tuning~\cite{speech-adapter}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2107.04734" id="10220">
        <attvalues>
          <attvalue for="5" value="0~\cite{analysis1, analysis2}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2212.02780" id="10221">
        <attvalues>
          <attvalue for="5" value=" This paper is an extended version of our previously published paper~\cite{otake2023parameter} at ICASSP 2023." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2006.11477" id="10222">
        <attvalues>
          <attvalue for="5" value="0~\cite{baevski2020wav2vec2}, HuBERT~\cite{hsu2021hubert}, ContentVec~\cite{qian2022contentvec}, and WavLM+~\cite{chen2022wavlm}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2204.09224" id="10223">
        <attvalues>
          <attvalue for="5" value="0~\cite{baevski2020wav2vec2}, HuBERT~\cite{hsu2021hubert}, ContentVec~\cite{qian2022contentvec}, and WavLM+~\cite{chen2022wavlm}." />
        </attvalues>
      </edge>
      <edge source="2407.21066" target="2101.00190" id="10224">
        <attvalues>
          <attvalue for="5" value=" \item We thoroughly conducted experimental evaluation with multiple conventional fine-tuning methods including weight tuning~\cite{chen2022wavlm}, LoRA tuning~\cite{hu22lora}, Prefix tuning~\cite{li21prefixtuning}, and Efficient adapter tuning~\cite{speech-adapter}." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1503.02364" id="10225">
        <attvalues>
          <attvalue for="5" value=" Conversational agents can be broadly categorized into two main types: (1) Chit-chat systems \cite{shang2015neural,sordoni2015neural,li2016deep,serban2016building,serban2017hierarchical} designed to engage users and provide mental support by conducting chit-chat type of conversation in wide range of topics without having a specific goal to complete." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1506.06714" id="10226">
        <attvalues>
          <attvalue for="5" value=" Conversational agents can be broadly categorized into two main types: (1) Chit-chat systems \cite{shang2015neural,sordoni2015neural,li2016deep,serban2016building,serban2017hierarchical} designed to engage users and provide mental support by conducting chit-chat type of conversation in wide range of topics without having a specific goal to complete." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1606.01541" id="10227">
        <attvalues>
          <attvalue for="5" value=" Conversational agents can be broadly categorized into two main types: (1) Chit-chat systems \cite{shang2015neural,sordoni2015neural,li2016deep,serban2016building,serban2017hierarchical} designed to engage users and provide mental support by conducting chit-chat type of conversation in wide range of topics without having a specific goal to complete." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1507.04808" id="10228">
        <attvalues>
          <attvalue for="5" value=" Conversational agents can be broadly categorized into two main types: (1) Chit-chat systems \cite{shang2015neural,sordoni2015neural,li2016deep,serban2016building,serban2017hierarchical} designed to engage users and provide mental support by conducting chit-chat type of conversation in wide range of topics without having a specific goal to complete." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1605.06069" id="10229">
        <attvalues>
          <attvalue for="5" value=" Conversational agents can be broadly categorized into two main types: (1) Chit-chat systems \cite{shang2015neural,sordoni2015neural,li2016deep,serban2016building,serban2017hierarchical} designed to engage users and provide mental support by conducting chit-chat type of conversation in wide range of topics without having a specific goal to complete." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1604.04562" id="10230">
        <attvalues>
          <attvalue for="5" value=" With the advent of deep learning, the trend has shifted toward end-to-end conversation modeling \cite{vinyals2015neural,xing2017topic,wen2017network}." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1506.05869" id="10231">
        <attvalues>
          <attvalue for="5" value=" With the advent of deep learning, the trend has shifted toward end-to-end conversation modeling \cite{vinyals2015neural,xing2017topic,wen2017network}." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1606.08340" id="10232">
        <attvalues>
          <attvalue for="5" value=" With the advent of deep learning, the trend has shifted toward end-to-end conversation modeling \cite{vinyals2015neural,xing2017topic,wen2017network}." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1910.14084" id="10234">
        <attvalues>
          <attvalue for="5" value=" Some preliminary work has been done on (1) and (2) in~\cite{mazumder2018towards,mazumder2019building,mazumder2019lifelong}." />
        </attvalues>
      </edge>
      <edge source="2009.10750" target="1907.13295" id="10235">
        <attvalues>
          <attvalue for="5" value=" Some preliminary work has been done on (1) and (2) in~\cite{mazumder2018towards,mazumder2019building,mazumder2019lifelong}." />
        </attvalues>
      </edge>
      <edge source="2210.12619" target="1703.01898" id="10236">
        <attvalues>
          <attvalue for="5" value=" Later approaches explicitly incorporate label information when processing the text, eg, \cite{Yogatama2017GenerativeAD} uses generative modeling and generates text given label embedding, and \cite{rios-kavuluru-2018-shot} uses label embedding based attention over text, both requiring multiple passes over the text and increasing the computational cost." />
        </attvalues>
      </edge>
      <edge source="2210.12619" target="1704.05426" id="10237">
        <attvalues>
          <attvalue for="5" value=" Most recently, NLI- \cite{condoravdi-etal-2003-entailment,williams-etal-2018-broad,yin-etal-2019-benchmarking} and NSP- \cite{ma-etal-2021-issues} based {0shot} text classification formulations have been proposed." />
        </attvalues>
      </edge>
      <edge source="2210.12619" target="1909.00161" id="10238">
        <attvalues>
          <attvalue for="5" value=" Most recently, NLI- \cite{condoravdi-etal-2003-entailment,williams-etal-2018-broad,yin-etal-2019-benchmarking} and NSP- \cite{ma-etal-2021-issues} based {0shot} text classification formulations have been proposed." />
        </attvalues>
      </edge>
      <edge source="2210.12619" target="1907.11692" id="10240">
        <attvalues>
          <attvalue for="5" value=" NLI and NSP use large transformer-based PLMs \cite{devlin-etal-2019-bert,Liu2019RoBERTaAR,DBLP:journals/corr/abs-1910-13461} and outperform previous non-transformer-based models by a large margin." />
        </attvalues>
      </edge>
      <edge source="2210.12619" target="1910.13461" id="10241">
        <attvalues>
          <attvalue for="5" value="2969312}) datasets using {0shot} models based on a moderately sized bart-large (NLI) \cite{lewis-etal-2020-bart} and a small bert-base (NSP) PLM." />
        </attvalues>
      </edge>
      <edge source="2210.12619" target="1805.10190" id="10242">
        <attvalues>
          <attvalue for="5" value=" We experiment with three intent classification (SNIPS \cite{Coucke2018SnipsVP}, ATIS \cite{5700816} and HWU64 \cite{Liu2019BenchmarkingNL}) and two topic classification (AG's news and Yahoo! Answers \cite{10." />
        </attvalues>
      </edge>
      <edge source="2210.12619" target="1903.05566" id="10243">
        <attvalues>
          <attvalue for="5" value=" We experiment with three intent classification (SNIPS \cite{Coucke2018SnipsVP}, ATIS \cite{5700816} and HWU64 \cite{Liu2019BenchmarkingNL}) and two topic classification (AG's news and Yahoo! Answers \cite{10." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="2012.08549" id="10244">
        <attvalues>
          <attvalue for="5" value=" In recent times, end-to-end (E2E) spoken language understanding (SLU) has gained popularity for SDS \cite{rongali2020exploring, chung2021splat, morais2021end, denisov2020pretrained}." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="1805.06280" id="10245">
        <attvalues>
          <attvalue for="5" value=" This idea has been explored extensively in written/typed dialog systems \cite{bothe2018context,raheja2019dialogue,colombo2020guiding}." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="1904.02594" id="10246">
        <attvalues>
          <attvalue for="5" value=" This idea has been explored extensively in written/typed dialog systems \cite{bothe2018context,raheja2019dialogue,colombo2020guiding}." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="2002.08801" id="10247">
        <attvalues>
          <attvalue for="5" value=" This idea has been explored extensively in written/typed dialog systems \cite{bothe2018context,raheja2019dialogue,colombo2020guiding}." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="2002.06012" id="10248">
        <attvalues>
          <attvalue for="5" value=" \cite{tomashenko2020dialogue} use text from the previous system prompt as history." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="2108.08405" id="10249">
        <attvalues>
          <attvalue for="5" value=" \cite{ganhotra2021integrating} use the entire conversation as history which is a transcript decoded from an ASR, making it prone to ASR errors and also not fully E2E." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="1906.11604" id="10250">
        <attvalues>
          <attvalue for="5" value=" ASR performance has also been shown to improve by integrating dialog history \cite{kim2019gated} and intent representations \cite{ray2021listen}." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="2105.07071" id="10251">
        <attvalues>
          <attvalue for="5" value=" ASR performance has also been shown to improve by integrating dialog history \cite{kim2019gated} and intent representations \cite{ray2021listen}." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="2010.04284" id="10252">
        <attvalues>
          <attvalue for="5" value=" Further, speech embeddings are tied with BERT embeddings via Euclidean loss \cite{huang2020leveraging} and a novel use of Contrastive loss \cite{zhang2021cross}." />
        </attvalues>
      </edge>
      <edge source="2204.05169" target="2010.13929" id="10254">
        <attvalues>
          <attvalue for="5" value=" We perform extensive experiments on the recently released HarperValleyBank corpus \cite{wu2020harpervalleybank} which is a task oriented dialog dataset." />
        </attvalues>
      </edge>
      <edge source="2310.17611" target="2204.06125" id="10257">
        <attvalues>
          <attvalue for="5" value=" For example, classical results show that word embeddings can be used for analogical reasoning \cite[eg,][]{mikolov2013efficient, pennington2014glove}, and such embeddings are the backbone of modern generative AI systems \cite[eg,][]{ramesh2022hierarchical, bubeck2023sparks, saharia2022photorealistic, devlin2018bert}." />
        </attvalues>
      </edge>
      <edge source="2310.17611" target="2206.05829" id="10258">
        <attvalues>
          <attvalue for="5" value=" In this paper, we use a natural candidate independence model in vector spaces known as partial orthogonality \cite{lauritzen1996graphical,amini2022lattice}." />
        </attvalues>
      </edge>
      <edge source="2310.17611" target="2209.10652" id="10259">
        <attvalues>
          <attvalue for="5" value=" More importantly, practical embeddings could potentially incorporate distortion, noise and undergo phenomena resembling superposition \cite{elhage2022superposition}." />
        </attvalues>
      </edge>
      <edge source="2310.17611" target="1502.03520" id="10260">
        <attvalues>
          <attvalue for="5" value=" Related work There are many papers \cite[eg,][]{arora2016latent, gittens2017skip, allen2019analogies, ethayarajh2019understanding, trager2023linear, perera2023prompt, leemann2023post, merullo2023language, wang2023concept} connecting semantic meanings and algebraic structures of popular embeddings like CLIP \cite{radford2021learning}, Glove \cite{pennington2014glove} and word2vec \cite{mikolov2013efficient}." />
        </attvalues>
      </edge>
      <edge source="2310.17611" target="1810.04882" id="10261">
        <attvalues>
          <attvalue for="5" value=" \cite{ethayarajh2019understanding}, on the other hand, studies the geometry of embeddings that decomposes the shifted pointwise mutual information (PMI) matrix." />
        </attvalues>
      </edge>
      <edge source="2310.17611" target="1605.09522" id="10262">
        <attvalues>
          <attvalue for="5" value=" On the other hand, similar to using vector orthogonality to represent (conditional) independence, kernel mean embeddings \cite{muandet2017kernel} are Hilbert space embeddings of distributions that can also be used to represent conditional independences \cite{song2009hilbert, song2013kernel}." />
        </attvalues>
      </edge>
      <edge source="2402.14337" target="2201.11903" id="10263">
        <attvalues>
          <attvalue for="5" value=" \usepgfplotslibrary{units} Language models (LMs) have achieved significant progress on sophisticated reasoning tasks requiring commonsense knowledge or selecting the best answer among tricky multiple choicese~\cite{Wei0SBIXCLZ22, 0010LLWWBCH22}." />
        </attvalues>
      </edge>
      <edge source="2402.14337" target="2203.14465" id="10264">
        <attvalues>
          <attvalue for="5" value=" Recent advances have enabled LMs to generate explicit free-text rationales and use them to guide task predictions with better search space for candidate answers without injecting additional knowledge~\cite{ZelikmanWMG22, KojimaGRMI22}." />
        </attvalues>
      </edge>
      <edge source="2402.14337" target="2205.11822" id="10265">
        <attvalues>
          <attvalue for="5" value="~\cite{JungQWBB0C22, WangCIC023, ChenB0J0S23}." />
        </attvalues>
      </edge>
      <edge source="2402.14337" target="2305.07095" id="10266">
        <attvalues>
          <attvalue for="5" value=" \cite{JoshiLRCTNW0023} has recently reported that only 20\% of them are useful for humans to gain additional information to answer questions." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2303.12712" id="10267">
        <attvalues>
          <attvalue for="5" value=" These models are revolutionizing the way we interact with technology, offering unprecedented capabilities, and consequently reshaping the AI landscape, prompting new discussions about artificial general intelligence (AGI )(\cite{bubeck2023sparks}; \cite{zhao2023survey})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2303.18223" id="10268">
        <attvalues>
          <attvalue for="5" value=" \cite{zhao2023survey} also reported that the small-sized open-source models perform not well on mathematical reasoning and scaling the open-source modes can improve the performance consistently." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2306.04757" id="10270">
        <attvalues>
          <attvalue for="5" value=" To assess the effectiveness and superiority of LLMs, a significant number of tasks and benchmarks have been introduced, aiming at empirically evaluating and analyzing their capabilities and the factors influencing their abilities (\cite{chia2023instructeval}; \cite{liang2022holistic}; \cite{zhao2023survey}; \cite{chang2023survey}; \cite{guo2023evaluating})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2307.03109" id="10272">
        <attvalues>
          <attvalue for="5" value=" Previous research identifies several critical measures that must be considered in the evaluation of LLMs, such as accuracy, efficiency, bias, safety etc (\cite{liang2022holistic}; \cite{chang2023survey})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2305.17306" id="10274">
        <attvalues>
          <attvalue for="5" value=" However, current LLM evaluations tend to prioritize accuracy (\cite{fu2023chain}; \cite{safdari2023personality}; \cite{choi2023llms}; \cite{yuan2023revisiting}; \cite{li2023api})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2307.00184" id="10275">
        <attvalues>
          <attvalue for="5" value=" However, current LLM evaluations tend to prioritize accuracy (\cite{fu2023chain}; \cite{safdari2023personality}; \cite{choi2023llms}; \cite{yuan2023revisiting}; \cite{li2023api})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2305.14938" id="10276">
        <attvalues>
          <attvalue for="5" value=" However, current LLM evaluations tend to prioritize accuracy (\cite{fu2023chain}; \cite{safdari2023personality}; \cite{choi2023llms}; \cite{yuan2023revisiting}; \cite{li2023api})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2306.04618" id="10277">
        <attvalues>
          <attvalue for="5" value=" However, current LLM evaluations tend to prioritize accuracy (\cite{fu2023chain}; \cite{safdari2023personality}; \cite{choi2023llms}; \cite{yuan2023revisiting}; \cite{li2023api})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2304.15004" id="10278">
        <attvalues>
          <attvalue for="5" value=" For instance, ``emergent abilities'' could be observed from a number of LLMs, such as GPT, PaLM and LaMDA (\cite{wei2022emergent}; \cite{schaeffer2023emergent})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2210.11416" id="10279">
        <attvalues>
          <attvalue for="5" value=" Some researchers found that instruction-tuning provides a broad set of advantages compared with other types of training (fine-tune, pretrained, RL-tuned etc) (\cite{liang2022holistic}; \cite{chung2022scaling}; \cite{zhao2023survey})." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2306.10062" id="10280">
        <attvalues>
          <attvalue for="5" value=" Researchers also found that some of the inconsistencies among the relationships between model size and task performance \cite{burnell2023revealing}." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2306.13394" id="10283">
        <attvalues>
          <attvalue for="5" value=" A primary issue is the narrow range of models typically assessed in multiple tasks — often several to 30 (\cite{yu2023mm}; \cite{yu2023kola}; \cite{fu2023mme}; \cite{jiang2023structgpt}; \cite{huang2023trustgpt}), compared to the over 120000 models available, for instance, on Huggingface." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2305.09645" id="10284">
        <attvalues>
          <attvalue for="5" value=" A primary issue is the narrow range of models typically assessed in multiple tasks — often several to 30 (\cite{yu2023mm}; \cite{yu2023kola}; \cite{fu2023mme}; \cite{jiang2023structgpt}; \cite{huang2023trustgpt}), compared to the over 120000 models available, for instance, on Huggingface." />
        </attvalues>
      </edge>
      <edge source="2403.15250" target="2306.11507" id="10285">
        <attvalues>
          <attvalue for="5" value=" A primary issue is the narrow range of models typically assessed in multiple tasks — often several to 30 (\cite{yu2023mm}; \cite{yu2023kola}; \cite{fu2023mme}; \cite{jiang2023structgpt}; \cite{huang2023trustgpt}), compared to the over 120000 models available, for instance, on Huggingface." />
        </attvalues>
      </edge>
      <edge source="2312.16850" target="1505.07818" id="10286">
        <attvalues>
          <attvalue for="5" value=" Previous approaches attempting to disentangle accent attributes and speaker timbre are mainly based on Domain Adversarial Training (DAT) ~\cite{DAT}." />
        </attvalues>
      </edge>
      <edge source="2312.16850" target="2202.05352" id="10287">
        <attvalues>
          <attvalue for="5" value=" Additionally, gradient descent in domain adversarial training can violate the optimizer's asymptotic convergence guarantees, often hindering the transfer performance ~\cite{AcunaLZF22}." />
        </attvalues>
      </edge>
      <edge source="2312.16850" target="1802.08735" id="10288">
        <attvalues>
          <attvalue for="5" value=" Furthermore, there is a trade-off between speaker similarity and accent similarity, which means entirely removing speaker timbre hurts performance on preserving accent pronunciation ~\cite{ShuBNE18}." />
        </attvalues>
      </edge>
      <edge source="2312.16850" target="2210.17305" id="10289">
        <attvalues>
          <attvalue for="5" value=" Bottleneck (BN) features are recently used as an intermediate representation to supervise accent attribute modeling in TTS ~\cite{ZhangWYSWX22}." />
        </attvalues>
      </edge>
      <edge source="2312.16850" target="2109.03439" id="10290">
        <attvalues>
          <attvalue for="5" value=" However, in the methods with BN as an intermediate representation \cite{bnTTS1, Hiertron}, models are often trained independently in multiple stages." />
        </attvalues>
      </edge>
      <edge source="2312.16850" target="2106.06103" id="10291">
        <attvalues>
          <attvalue for="5" value=" Specifically, we leverage the end-to-end speech synthesis framework, VITS ~\cite{vits}, as the backbone of our model, since it achieves good audio quality and alleviates the error accumulation caused by the conventional two-stage TTS system consisting of an acoustic model and a vocoder." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="1804.07998" id="10293">
        <attvalues>
          <attvalue for="5" value=" Early works examine the robustness of neural models by creating a certain types of artificial adversarial examples~\cite{jia2017adversarial,alzantot2018generating,ren2019generating,jin2020bert}, and involving human-and-model-in-the-loop to create dynamic adversarial examples~\cite{nie2019adversarial,wallace2019trick}." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="1907.11932" id="10294">
        <attvalues>
          <attvalue for="5" value=" Early works examine the robustness of neural models by creating a certain types of artificial adversarial examples~\cite{jia2017adversarial,alzantot2018generating,ren2019generating,jin2020bert}, and involving human-and-model-in-the-loop to create dynamic adversarial examples~\cite{nie2019adversarial,wallace2019trick}." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="1809.02701" id="10296">
        <attvalues>
          <attvalue for="5" value=" Early works examine the robustness of neural models by creating a certain types of artificial adversarial examples~\cite{jia2017adversarial,alzantot2018generating,ren2019generating,jin2020bert}, and involving human-and-model-in-the-loop to create dynamic adversarial examples~\cite{nie2019adversarial,wallace2019trick}." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="1803.02324" id="10297">
        <attvalues>
          <attvalue for="5" value=" Further studies discover that a few types of superficial cues (eg shortcuts) in the training data, are learned by the models and hence affect the model robustness~\cite{gururangan2018annotation,mccoy2019right,lai2021machine}." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="2106.01024" id="10299">
        <attvalues>
          <attvalue for="5" value=" Further studies discover that a few types of superficial cues (eg shortcuts) in the training data, are learned by the models and hence affect the model robustness~\cite{gururangan2018annotation,mccoy2019right,lai2021machine}." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="2002.04108" id="10301">
        <attvalues>
          <attvalue for="5" value=" Besides, several studies try to improve the robustness of the neural models by adversarial data augmentation~\cite{min2020syntactic} and data filtering~\cite{le2020adversarial}." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="2005.00955" id="10302">
        <attvalues>
          <attvalue for="5" value=" First, the analysis and evaluation in previous work focus on just one or a few types of adversarial examples or shortcuts, but we need normative evaluation~\cite{linzen2020can, ettinger2020bert,phangadversarially}." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="2103.11441" id="10304">
        <attvalues>
          <attvalue for="5" value=" Checklist~\cite{ribeiro2020beyond} and Textflint~\cite{gui2021textflint} are great attempts of normative evaluation." />
        </attvalues>
      </edge>
      <edge source="2112.08609" target="2004.02709" id="10306">
        <attvalues>
          <attvalue for="5" value=" Some other works manually perturb the examples to construct natural examples, but the manual perturbations is time consuming and costly~\cite{gardner2020evaluating}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1904.13377" id="10307">
        <attvalues>
          <attvalue for="5" value=" Especially in speech recognition, lots of attention-based models have proved to obtain a substantial performance improvement~\cite{Miao2019}\cite{Pham2019}\cite{Sperber2018}\cite{Chorowski2015}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1803.09519" id="10308">
        <attvalues>
          <attvalue for="5" value=" Especially in speech recognition, lots of attention-based models have proved to obtain a substantial performance improvement~\cite{Miao2019}\cite{Pham2019}\cite{Sperber2018}\cite{Chorowski2015}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1506.07503" id="10309">
        <attvalues>
          <attvalue for="5" value=" Especially in speech recognition, lots of attention-based models have proved to obtain a substantial performance improvement~\cite{Miao2019}\cite{Pham2019}\cite{Sperber2018}\cite{Chorowski2015}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1901.10055" id="10310">
        <attvalues>
          <attvalue for="5" value=" For example, self-attention blocks are successfully applied in CTC-based network, SAN-CTC~\cite{salazar2019self} showed that self-attention encoder is competitive with existing end-to-end models." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1910.12977" id="10311">
        <attvalues>
          <attvalue for="5" value=" Transformer-Transducer~\cite{yeh2019transformer} used VGGNet with causal convolution as the frontend of encoder, and self-attention transducer as the network architecture." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1706.03762" id="10312">
        <attvalues>
          <attvalue for="5" value=" CTC~\cite{graves2006connectionist}, Transformer~\cite{vaswani2017attention}, RNN-Transducer~\cite{tian2019self}\cite{Wang2019} are most common used architectures in speech recognition~\cite{battenberg2017exploring}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1909.13037" id="10313">
        <attvalues>
          <attvalue for="5" value=" CTC~\cite{graves2006connectionist}, Transformer~\cite{vaswani2017attention}, RNN-Transducer~\cite{tian2019self}\cite{Wang2019} are most common used architectures in speech recognition~\cite{battenberg2017exploring}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1811.05097" id="10314">
        <attvalues>
          <attvalue for="5" value=" CTC~\cite{graves2006connectionist}, Transformer~\cite{vaswani2017attention}, RNN-Transducer~\cite{tian2019self}\cite{Wang2019} are most common used architectures in speech recognition~\cite{battenberg2017exploring}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1707.07413" id="10315">
        <attvalues>
          <attvalue for="5" value=" CTC~\cite{graves2006connectionist}, Transformer~\cite{vaswani2017attention}, RNN-Transducer~\cite{tian2019self}\cite{Wang2019} are most common used architectures in speech recognition~\cite{battenberg2017exploring}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1512.02595" id="10316">
        <attvalues>
          <attvalue for="5" value=" CTC~\cite{sainath2015convolutional}\cite{amodei2016deep} is first widely used to end-to-end models, but CTC has a fatal drawback that every timestep is outputted independently." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1909.12415" id="10317">
        <attvalues>
          <attvalue for="5" value=" Transformer~\cite{Zeyer2019}\cite{Li2019} is another choice by encoder-decoder infrastructure." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1303.5778" id="10318">
        <attvalues>
          <attvalue for="5" value=" The RNN-Transducer~\cite{graves2013speech}\cite{Tsunoo2019} was proposed as an extension to CTC, which also marginalizes over all possible alignments between the input sequence and the output targets." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1910.11871" id="10319">
        <attvalues>
          <attvalue for="5" value=" The RNN-Transducer~\cite{graves2013speech}\cite{Tsunoo2019} was proposed as an extension to CTC, which also marginalizes over all possible alignments between the input sequence and the output targets." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="1902.06450" id="10320">
        <attvalues>
          <attvalue for="5" value=" Another researches are worked on local monotonic attention~\cite{Andre2019}\cite{dong2019self}." />
        </attvalues>
      </edge>
      <edge source="2102.11594" target="2002.02562" id="10321">
        <attvalues>
          <attvalue for="5" value=" Google proposed transformer encoders with RNN-T loss~\cite{ZhangTransformer}, and they showed that limiting the left and right context of attention per-layer can obtain not bad accuracy but still have some gap between the performance of full-attention models." />
        </attvalues>
      </edge>
      <edge source="2001.01140" target="1805.04623" id="10322">
        <attvalues>
          <attvalue for="5" value=" Despite their name, they fall short of capturing word dependencies past 200 words and focus on past 50 words more heavily \cite{LSTM_issues}." />
        </attvalues>
      </edge>
      <edge source="2001.01140" target="1706.03762" id="10323">
        <attvalues>
          <attvalue for="5" value=" It does away with recurrence in favor of an attention mechanism \cite{ATTENTION}." />
        </attvalues>
      </edge>
      <edge source="2001.01140" target="1901.02860" id="10325">
        <attvalues>
          <attvalue for="5" value=" The transformer-XL architecture solves this by introducing a segment-level recurrence mechanism \cite{Transformer_xl}." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2006.11477" id="10327">
        <attvalues>
          <attvalue for="5" value="0~\cite{Wav2Vec2} designed a quantization module trained jointly with the masked prediction objective." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2404.09385" id="10329">
        <attvalues>
          <attvalue for="5" value=" Note that HuBERT~\cite{HuBERT} original focused on content-based tasks, such as ASR, but widely adopted as a general foundation model~\cite{SuPERB,9747870,wang2022finetunedwav2vec20hubertbenchmark}." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2205.10643" id="10330">
        <attvalues>
          <attvalue for="5" value=" Consequently, numerous SSL approaches for learning encoders have been introduced (see~\cite{mohamed2022self} for a review)." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2106.07447" id="10331">
        <attvalues>
          <attvalue for="5" value=" Note that HuBERT~\cite{HuBERT} original focused on content-based tasks, such as ASR, but widely adopted as a general foundation model~\cite{SuPERB,9747870,wang2022finetunedwav2vec20hubertbenchmark}." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2110.13900" id="10332">
        <attvalues>
          <attvalue for="5" value=" Recent approaches~\cite{WavLM,contentvec,multiresHuBERT,UniSAT,chen2024robustspeechrepresentationlearning} built upon the iterative clustering framework of HuBERT with architectural changes and data augmentations." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2204.09224" id="10334">
        <attvalues>
          <attvalue for="5" value=" Recent approaches~\cite{WavLM,contentvec,multiresHuBERT,UniSAT,chen2024robustspeechrepresentationlearning} built upon the iterative clustering framework of HuBERT with architectural changes and data augmentations." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2310.02720" id="10335">
        <attvalues>
          <attvalue for="5" value=" Recent approaches~\cite{WavLM,contentvec,multiresHuBERT,UniSAT,chen2024robustspeechrepresentationlearning} built upon the iterative clustering framework of HuBERT with architectural changes and data augmentations." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2407.00837" id="10336">
        <attvalues>
          <attvalue for="5" value=" Recent approaches~\cite{WavLM,contentvec,multiresHuBERT,UniSAT,chen2024robustspeechrepresentationlearning} built upon the iterative clustering framework of HuBERT with architectural changes and data augmentations." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="1910.12638" id="10338">
        <attvalues>
          <attvalue for="5" value=" Early works~\cite{TERA,Mockingjay,decoar} explored using low-level spectral features as prediction targets." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="1912.01679" id="10339">
        <attvalues>
          <attvalue for="5" value=" Early works~\cite{TERA,Mockingjay,decoar} explored using low-level spectral features as prediction targets." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2203.09690" id="10340">
        <attvalues>
          <attvalue for="5" value=" However, such targets are challenging to reconstruct due to their continuous and fine-grained nature~\cite{bai20223}." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2108.06209" id="10341">
        <attvalues>
          <attvalue for="5" value=" Consequently, later works~\cite{w2v-BERT,wav2vec-c,Wav2Vec2} explored methods for quantizing targets to abstract the fine-grained speech properties." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2103.08393" id="10342">
        <attvalues>
          <attvalue for="5" value=" Consequently, later works~\cite{w2v-BERT,wav2vec-c,Wav2Vec2} explored methods for quantizing targets to abstract the fine-grained speech properties." />
        </attvalues>
      </edge>
      <edge source="2409.10788" target="2202.03896" id="10343">
        <attvalues>
          <attvalue for="5" value=" Note that HuBERT~\cite{HuBERT} original focused on content-based tasks, such as ASR, but widely adopted as a general foundation model~\cite{SuPERB,9747870,wang2022finetunedwav2vec20hubertbenchmark}." />
        </attvalues>
      </edge>
      <edge source="2310.03938" target="2110.04590" id="10345">
        <attvalues>
          <attvalue for="5" value=" Recent works have incorporated self-supervised learning (SSL) models into end-to-end (E2E) automatic speech recognition (ASR) systems as frontend feature extractors to capture more relevant features of corpora~\cite{Chang2021AnEO, 9893562, Yang2021SUPERBSP, KrishnaD2021UsingLS, 9801640, shi2023multi, chen2023joint, chang2023colld}." />
        </attvalues>
      </edge>
      <edge source="2310.03938" target="2204.02470" id="10346">
        <attvalues>
          <attvalue for="5" value=" Our proposed EFFUSE method reduces the number of parameters by using a single SSL model to mimic the benefits of multiple SSL models, thereby scaling down inference costs compared to the original feature fusion network \cite{berrebbi22_interspeech, fearless}." />
        </attvalues>
      </edge>
      <edge source="2310.03938" target="2302.09331" id="10347">
        <attvalues>
          <attvalue for="5" value=" As a result, methods such as feature fusion \cite{berrebbi22_interspeech}, adapters \cite{10095130, 9746223}, teacher-student distillation \cite{Peng2021ShrinkingBR, 9747490, Peng2023DPHuBERTJD, lee22p_interspeech, Yang2021KnowledgeDF}, and continual training \cite{DBLP:conf/interspeech/HuangFZL22, vandereeckt_eusipco2022, Hsu2021RobustW2, 9746594}, have been proposed, which have shown performance improvements for different target domains." />
        </attvalues>
      </edge>
      <edge source="2310.03938" target="2103.15760" id="10348">
        <attvalues>
          <attvalue for="5" value=" As a result, methods such as feature fusion \cite{berrebbi22_interspeech}, adapters \cite{10095130, 9746223}, teacher-student distillation \cite{Peng2021ShrinkingBR, 9747490, Peng2023DPHuBERTJD, lee22p_interspeech, Yang2021KnowledgeDF}, and continual training \cite{DBLP:conf/interspeech/HuangFZL22, vandereeckt_eusipco2022, Hsu2021RobustW2, 9746594}, have been proposed, which have shown performance improvements for different target domains." />
        </attvalues>
      </edge>
      <edge source="2310.03938" target="2203.16104" id="10349">
        <attvalues>
          <attvalue for="5" value=" As a result, methods such as feature fusion \cite{berrebbi22_interspeech}, adapters \cite{10095130, 9746223}, teacher-student distillation \cite{Peng2021ShrinkingBR, 9747490, Peng2023DPHuBERTJD, lee22p_interspeech, Yang2021KnowledgeDF}, and continual training \cite{DBLP:conf/interspeech/HuangFZL22, vandereeckt_eusipco2022, Hsu2021RobustW2, 9746594}, have been proposed, which have shown performance improvements for different target domains." />
        </attvalues>
      </edge>
      <edge source="2302.13458" target="1712.05884" id="10350">
        <attvalues>
          <attvalue for="5" value=" The early autoregressive (AR) TTS models \cite{shen2018natural, li2019neural} dealt with the difficulty by factorizing the speech distribution into the product of homogeneous conditional factors in sequential order." />
        </attvalues>
      </edge>
      <edge source="2302.13458" target="2005.11129" id="10351">
        <attvalues>
          <attvalue for="5" value=" The first type (Type-$\textup{\uppercase}$) is to use an advanced generative framework such as normalizing flow \cite{NEURIPS2020_5c3b99e8}, diffusion model \cite{pmlr-v139-popov21a}, and generative adversarial network \cite{yang21e_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2302.13458" target="2105.06337" id="10352">
        <attvalues>
          <attvalue for="5" value=" The first type (Type-$\textup{\uppercase}$) is to use an advanced generative framework such as normalizing flow \cite{NEURIPS2020_5c3b99e8}, diffusion model \cite{pmlr-v139-popov21a}, and generative adversarial network \cite{yang21e_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2302.13458" target="2106.15153" id="10353">
        <attvalues>
          <attvalue for="5" value=" The first type (Type-$\textup{\uppercase}$) is to use an advanced generative framework such as normalizing flow \cite{NEURIPS2020_5c3b99e8}, diffusion model \cite{pmlr-v139-popov21a}, and generative adversarial network \cite{yang21e_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2302.13458" target="2006.04558" id="10355">
        <attvalues>
          <attvalue for="5" value=" The second type of the methods (Type-$\textup{\uppercase}$) is to solve a task of TTS by dividing it into simpler two tasks based on variance information such as pitch or energy \cite{ren2021fastspeech, lancucki2021fastpitch, bak21_interspeech}: (1) text conditioned variance modeling; (2) text and variance information conditioned speech generation." />
        </attvalues>
      </edge>
      <edge source="2302.13458" target="1605.08803" id="10356">
        <attvalues>
          <attvalue for="5" value=" First, since NF is robust to the one-to-many problem, it learns the variance distribution better than using the MSE loss \cite{DBLP:conf/iclr/DinhSB17}, and it leads to improved speech quality." />
        </attvalues>
      </edge>
      <edge source="2302.13458" target="2005.13580" id="10357">
        <attvalues>
          <attvalue for="5" value=" Second, NF enhances the variance controllability by disentangling the latent variance representation and the text \cite{NEURIPS2020_1cfa81af}." />
        </attvalues>
      </edge>
      <edge source="2311.01544" target="1706.03762" id="10358">
        <attvalues>
          <attvalue for="5" value=" Cutting-edge Large Language Models (LLMs) based on the transformer architecture \cite{vaswani2017attention} have revolutionized Natural Language Processing with their exceptional performance, notably exemplified by the GPT series~\cite{radford2018improving,radford2019language,brown2020language,bubeck2023sparks} in text generation." />
        </attvalues>
      </edge>
      <edge source="2311.01544" target="2005.14165" id="10359">
        <attvalues>
          <attvalue for="5" value=" Cutting-edge Large Language Models (LLMs) based on the transformer architecture \cite{vaswani2017attention} have revolutionized Natural Language Processing with their exceptional performance, notably exemplified by the GPT series~\cite{radford2018improving,radford2019language,brown2020language,bubeck2023sparks} in text generation." />
        </attvalues>
      </edge>
      <edge source="2311.01544" target="2303.12712" id="10360">
        <attvalues>
          <attvalue for="5" value=" Cutting-edge Large Language Models (LLMs) based on the transformer architecture \cite{vaswani2017attention} have revolutionized Natural Language Processing with their exceptional performance, notably exemplified by the GPT series~\cite{radford2018improving,radford2019language,brown2020language,bubeck2023sparks} in text generation." />
        </attvalues>
      </edge>
      <edge source="2311.01544" target="2204.02311" id="10361">
        <attvalues>
          <attvalue for="5" value=" However, these models have grown massively, even exceeding half a trillion parameters~\cite{chowdhery2022palm}." />
        </attvalues>
      </edge>
      <edge source="2311.01544" target="1905.10650" id="10362">
        <attvalues>
          <attvalue for="5" value=" In particular, for the attention mechanism, it was hinted that after some training convergence, certain heads dominate the inference process~\cite{michel2019sixteen}." />
        </attvalues>
      </edge>
      <edge source="2405.05445" target="2303.14524" id="10363">
        <attvalues>
          <attvalue for="5" value=" Secondly, LLMs can enhance model accuracy by leveraging their knowledge on a broader range of data; see \cite{moller2023prompt,gao2023chat,chen2024exploring}." />
        </attvalues>
      </edge>
      <edge source="2405.05445" target="2307.03393" id="10364">
        <attvalues>
          <attvalue for="5" value=" Secondly, LLMs can enhance model accuracy by leveraging their knowledge on a broader range of data; see \cite{moller2023prompt,gao2023chat,chen2024exploring}." />
        </attvalues>
      </edge>
      <edge source="2211.00106" target="1906.01502" id="10366">
        <attvalues>
          <attvalue for="5" value=" This modeling approach has several powerful advantages, such as allowing similar languages to exert positive influence on each other, and enabling cross-lingual task transfer (eg, finetuning on some source language(s), then using the model on different target languages) \cite{pires2019multilingual}." />
        </attvalues>
      </edge>
      <edge source="2211.00106" target="2203.04583" id="10368">
        <attvalues>
          <attvalue for="5" value=" While recent works apply various subnetwork based approaches to their models statically \cite{lu2022language, yang2022learning, nooralahzadeh2022improving}, we propose a new method that allows the model to dynamically update the subnetworks during fine-tuning." />
        </attvalues>
      </edge>
      <edge source="2211.00106" target="1905.09418" id="10370">
        <attvalues>
          <attvalue for="5" value=" Inspired by studies that show that attention-heads in BERT-based models have specialized functions \cite{voita2019analyzing, htut2019attention}, we focus on learning subnetworks at the attention-head level." />
        </attvalues>
      </edge>
      <edge source="2211.00106" target="1703.03400" id="10371">
        <attvalues>
          <attvalue for="5" value=" Given our focus on low-resource languages, we also combine our methods with meta-learning, a data-efficient technique to learn tasks from a few samples \cite{finn2017model}." />
        </attvalues>
      </edge>
      <edge source="2101.02258" target="1611.01368" id="10374">
        <attvalues>
          <attvalue for="5" value=" Previous studies confirmed that RNN trained on natural data can successfully perform challenging long-range agreement between subject and verb \cite{Linzen:etal:2016, Bernardy:Lappin:2017, Gulordava:etal:2018, lakretz2019emergence}." />
        </attvalues>
      </edge>
      <edge source="2101.02258" target="1903.07435" id="10375">
        <attvalues>
          <attvalue for="5" value=" An analysis of the generalization patterns of the networks revealed primacy and recency effects, consistent with recent findings on the distinction between short- and long-range number units identified in RNN language models \cite{lakretz2019emergence}; Finally, we describe the dynamics of the inner states of one such long-range unit and its complex encoding of multiple grammatical numbers." />
        </attvalues>
      </edge>
      <edge source="2402.08382" target="2005.00955" id="10376">
        <attvalues>
          <attvalue for="5" value=" The current framework of natural language processing systems, described by \cite{linzen-2020-accelerate} as the PAID paradigm, consists of two production stages: unsupervised representation learning and task-specific engineering." />
        </attvalues>
      </edge>
      <edge source="2402.08382" target="1911.02969" id="10379">
        <attvalues>
          <attvalue for="5" value=" In classification systems, we observe unstable outcome despite consistent input and reliance on shallow heuristics while processing unfamiliar input \cite{mccoy-etal-2020-berts, zhou-etal-2020-curse}." />
        </attvalues>
      </edge>
      <edge source="2402.08382" target="2302.10198" id="10380">
        <attvalues>
          <attvalue for="5" value=" In generative and conversational systems, we observe stagnant natural language understanding performance despite drastic increase in conversational performance \cite{zhong2023chatgpt}, and failure to generalize sentences like &quot;A equals B&quot; to &quot;B equals A&quot; \cite{berglund2023reversal}." />
        </attvalues>
      </edge>
      <edge source="2402.08382" target="2004.13606" id="10382">
        <attvalues>
          <attvalue for="5" value=" While it is difficult to pinpoint the exact source of these weaknesses or even disentangle between the effects of unsupervised pre-training and task specific engineering, the pre-training stage is at least partially attributable for these behaviors, and there exists room for improvement \cite{zhou-etal-2020-curse, min-etal-2020-syntactic}." />
        </attvalues>
      </edge>
      <edge source="2008.00545" target="1702.05464" id="10386">
        <attvalues>
          <attvalue for="5" value=" The impact of dataset-bias \cite{tzeng2017adversarial} on LID robustness has not yet been investigated with a systematic evaluation across datasets." />
        </attvalues>
      </edge>
      <edge source="2008.00545" target="1409.7495" id="10387">
        <attvalues>
          <attvalue for="5" value=" Finally, we apply adversarial domain confusion \cite{ganin2015unsupervised} to adapt our model to a target domain, analyze predictions from the adapted model, and visualize its representations compared to the baseline (RQ3)." />
        </attvalues>
      </edge>
      <edge source="2305.18074" target="1909.05952" id="10388">
        <attvalues>
          <attvalue for="5" value=" The main motivation is that recent research trends, such as the invention of fully end-to-end diarization methods \cite{fujita2019end-blstm, fujita2019end-self, fujita2020end, kinoshita2021integrating, kinoshita2021advances, kinoshita2022utterance, kinoshita2022tight}, seems to suggest that the technology is mature enough to attain highly accurate and reliable diarization in the CTS scenario, whereas other scenarios, such as distant-talk meetings with several speakers, still pose several challenges \cite{watanabe2020chime}." />
        </attvalues>
      </edge>
      <edge source="2309.00949" target="1911.02116" id="10390">
        <attvalues>
          <attvalue for="5" value=" mBERT~\cite{devlin-etal-2019-bert} and xlm-r~\cite{conneau-etal-2020-unsupervised} trained with multiple objective functions on more than hundreds of languages came a long way in achieving this vision." />
        </attvalues>
      </edge>
      <edge source="2401.07284" target="2110.04541" id="10393">
        <attvalues>
          <attvalue for="5" value=" Inspired by~\cite{levine2021inductive, shi2023context}, we leverage length-based clustering to extend the context by concatenating similar documents into the same input as context." />
        </attvalues>
      </edge>
      <edge source="2401.07284" target="2106.09685" id="10394">
        <attvalues>
          <attvalue for="5" value=" Moreover, we improve the efficiency of domain adaptation by utilizing parameter-efficient fine-tuning methods like LoRA~\cite{hu2021lora}." />
        </attvalues>
      </edge>
      <edge source="2401.07284" target="2311.00176" id="10395">
        <attvalues>
          <attvalue for="5" value=" Contrary to previous work~\cite{liu2023chipnemo}, we find that LoRA can be more efficient than full fine-tuning for domain adaptation with proper settings." />
        </attvalues>
      </edge>
      <edge source="2311.09325" target="2212.12131" id="10396">
        <attvalues>
          <attvalue for="5" value=" \cite{oh2023does} hypothesize that this might be due to LLMs being ``too confident'' in their estimates of rare named entities compared to humans, thanks to their manifold larger exposure to data and greater memory capacity compared to humans." />
        </attvalues>
      </edge>
      <edge source="2311.09325" target="2205.12507" id="10397">
        <attvalues>
          <attvalue for="5" value=" Furthermore, work on NLP applications like question answering has reported that probability estimates from pretrained language models are often overconfident, eg~they are higher than the ground truth probability~\cite{si2022re, kumar2022answer}." />
        </attvalues>
      </edge>
      <edge source="2311.09325" target="1706.04599" id="10398">
        <attvalues>
          <attvalue for="5" value=" One approach to address calibration problems is to use temperature scaling, as done eg, in vision tasks \cite{guo2017calibration, hendrycks2019using}." />
        </attvalues>
      </edge>
      <edge source="2311.09325" target="2211.14301" id="10399">
        <attvalues>
          <attvalue for="5" value=" We note that the idea to work with flattened distributions instead of the original probability distributions from LLMs is also related to contextual R\'enyi Entropy as discussed by \cite{pimentel-etal-2023-effect}, as well as the super/sub-linear surprisal effect by \cite{shain2022large,hoover2023plausibility}." />
        </attvalues>
      </edge>
      <edge source="2205.02694" target="1904.05862" id="10400">
        <attvalues>
          <attvalue for="5" value=" Deep acoustic models have improved automatic speech recognition (ASR) substantially in recent years \cite{schneider2019wav2vec, baevski2019vq, baevski2020wav2vec, conneau2020unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2205.02694" target="2107.04734" id="10401">
        <attvalues>
          <attvalue for="5" value=" The work of \cite{livescuprobing2021} examined information represented by the wav2vec~2." />
        </attvalues>
      </edge>
      <edge source="2205.02694" target="2006.11477" id="10402">
        <attvalues>
          <attvalue for="5" value="0 model \cite{baevski2020wav2vec} across the various Transformer layers." />
        </attvalues>
      </edge>
      <edge source="2205.02694" target="2010.13007" id="10403">
        <attvalues>
          <attvalue for="5" value=" In addition, \cite{liberman2021} investigated several deep acoustic models using phonetic probing tasks, and found that representations from these models capture information useful for distinguishing English phones." />
        </attvalues>
      </edge>
      <edge source="2205.02694" target="2002.11800" id="10404">
        <attvalues>
          <attvalue for="5" value=" While automatic approaches for computing phonetic transcriptions exist (eg, \cite{allosaurus}), they produce lower quality phonetic transcriptions than human transcribers do." />
        </attvalues>
      </edge>
      <edge source="2205.02694" target="2011.12649" id="10405">
        <attvalues>
          <attvalue for="5" value=" Recently, \cite{bartelds2021neural} found that representations from the hidden layers of pre-trained and fine-tuned wav2vec~2." />
        </attvalues>
      </edge>
      <edge source="2401.14400" target="2109.06772" id="10406">
        <attvalues>
          <attvalue for="5" value=" We evaluate the approaches on part-of-speech~(POS) tagging with zero-shot cross-lingual transfer from Standard German~\cite{aepli-sennrich-2022-improving}, as well as dialect identification~\cite{zampieri-etal-2019-report} and cross-lingual sentence retrieval based on a parallel Standard German--Swiss German test set~\cite{aepli-etal-2023-benchmark}." />
        </attvalues>
      </edge>
      <edge source="2401.14400" target="2311.16865" id="10407">
        <attvalues>
          <attvalue for="5" value=" We evaluate the approaches on part-of-speech~(POS) tagging with zero-shot cross-lingual transfer from Standard German~\cite{aepli-sennrich-2022-improving}, as well as dialect identification~\cite{zampieri-etal-2019-report} and cross-lingual sentence retrieval based on a parallel Standard German--Swiss German test set~\cite{aepli-etal-2023-benchmark}." />
        </attvalues>
      </edge>
      <edge source="2401.14400" target="2205.06266" id="10408">
        <attvalues>
          <attvalue for="5" value=" We then focus on comparing monolithic adaptation, where all the parameters of the encoder are updated during continued pre-training, to modular adaptation with language-specific modular components (language adapters; \cite{pfeiffer-etal-2022-lifting})." />
        </attvalues>
      </edge>
      <edge source="2401.14400" target="2303.13310" id="10409">
        <attvalues>
          <attvalue for="5" value=" Given these findings, we propose to extend the SwissBERT model~\cite{vamvas-etal-2023-swissbert}, which was trained on Standard German and other languages, with a Swiss German adapter~(Table~\ref{tab:figure-1})." />
        </attvalues>
      </edge>
      <edge source="2401.14400" target="2103.06874" id="10410">
        <attvalues>
          <attvalue for="5" value=" We further hypothesize that the architecture of Canine~\cite{clark-etal-2022-canine}, a tokenization-free model that operates on characters, might be better suited to the highly variable spelling of Swiss German." />
        </attvalues>
      </edge>
      <edge source="2312.10185" target="2212.10071" id="10411">
        <attvalues>
          <attvalue for="5" value=" Recently, with the emergence of large language models (LLMs), knowledge distillation (KD) with label matching has been widely applied to low-data regimes~\cite{shridhar2022distilling,ho2022large,li2023feasibility,gilardi2023chatgpt,wang-etal-2021-want-reduce,yoo-etal-2021-gpt3mix-leveraging,ding2022gpt}." />
        </attvalues>
      </edge>
      <edge source="2312.10185" target="2303.15056" id="10412">
        <attvalues>
          <attvalue for="5" value=" Recently, with the emergence of large language models (LLMs), knowledge distillation (KD) with label matching has been widely applied to low-data regimes~\cite{shridhar2022distilling,ho2022large,li2023feasibility,gilardi2023chatgpt,wang-etal-2021-want-reduce,yoo-etal-2021-gpt3mix-leveraging,ding2022gpt}." />
        </attvalues>
      </edge>
      <edge source="2312.10185" target="2108.13487" id="10413">
        <attvalues>
          <attvalue for="5" value=" Recently, with the emergence of large language models (LLMs), knowledge distillation (KD) with label matching has been widely applied to low-data regimes~\cite{shridhar2022distilling,ho2022large,li2023feasibility,gilardi2023chatgpt,wang-etal-2021-want-reduce,yoo-etal-2021-gpt3mix-leveraging,ding2022gpt}." />
        </attvalues>
      </edge>
      <edge source="2312.10185" target="2104.08826" id="10414">
        <attvalues>
          <attvalue for="5" value=" Recently, with the emergence of large language models (LLMs), knowledge distillation (KD) with label matching has been widely applied to low-data regimes~\cite{shridhar2022distilling,ho2022large,li2023feasibility,gilardi2023chatgpt,wang-etal-2021-want-reduce,yoo-etal-2021-gpt3mix-leveraging,ding2022gpt}." />
        </attvalues>
      </edge>
      <edge source="2312.10185" target="2005.14165" id="10416">
        <attvalues>
          <attvalue for="5" value=" In scenarios where there is a scarcity of labeled in-domain data but an abundance of unlabeled data, LLMs can serve as an annotator to generate pseudo labels through few-shot prompting, eg, in-context learning~\cite{brown2020language}." />
        </attvalues>
      </edge>
      <edge source="2312.10185" target="2210.06711" id="10417">
        <attvalues>
          <attvalue for="5" value=" Some previous works for noisy label learning utilize loss reweighting~\cite{hinton2015distilling,NEURIPS2022_2e343555,lee2013pseudo} that may not apply to few-shot cases where pseudo labels are overwhelmingly more than gold labels, leading to great imbalance and unstable training." />
        </attvalues>
      </edge>
      <edge source="2312.10185" target="1503.02531" id="10418">
        <attvalues>
          <attvalue for="5" value=" Some previous works for noisy label learning utilize loss reweighting~\cite{hinton2015distilling,NEURIPS2022_2e343555,lee2013pseudo} that may not apply to few-shot cases where pseudo labels are overwhelmingly more than gold labels, leading to great imbalance and unstable training." />
        </attvalues>
      </edge>
      <edge source="2403.13590" target="2005.14165" id="10420">
        <attvalues>
          <attvalue for="5" value=" Unlike prior &quot;pre-train and fine-tune&quot; \cite{devlin-etal-2019-bert, he2020deberta} approaches, instruction-tuned LLMs combined with effective good prompting techniques has enabled LLMs to excel at unseen tasks without task-specific training \cite{brown2020language, touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2403.13590" target="2311.05232" id="10421">
        <attvalues>
          <attvalue for="5" value=" They are prone to hallucinating information \cite{huang2023survey, manakul2023selfcheckgpt}, can have large sensitivity to the form of prompts \cite{sclar2023quantifying, zhou2022large} and also demonstrate systematic biases such as gender bias \cite{kotek2023gender}." />
        </attvalues>
      </edge>
      <edge source="2403.13590" target="2310.11324" id="10422">
        <attvalues>
          <attvalue for="5" value=" They are prone to hallucinating information \cite{huang2023survey, manakul2023selfcheckgpt}, can have large sensitivity to the form of prompts \cite{sclar2023quantifying, zhou2022large} and also demonstrate systematic biases such as gender bias \cite{kotek2023gender}." />
        </attvalues>
      </edge>
      <edge source="2403.13590" target="2308.14921" id="10423">
        <attvalues>
          <attvalue for="5" value=" They are prone to hallucinating information \cite{huang2023survey, manakul2023selfcheckgpt}, can have large sensitivity to the form of prompts \cite{sclar2023quantifying, zhou2022large} and also demonstrate systematic biases such as gender bias \cite{kotek2023gender}." />
        </attvalues>
      </edge>
      <edge source="2403.13590" target="2109.01652" id="10424">
        <attvalues>
          <attvalue for="5" value=" Furthermore, due to the general nature of their pre-training and instruction-tuning \cite{wei2021finetuned, ouyang2022training}, for certain applications, they may be unaware of particular important task invariances." />
        </attvalues>
      </edge>
      <edge source="2403.13590" target="2308.11483" id="10425">
        <attvalues>
          <attvalue for="5" value=" Ongoing work has demonstrated that LLMs can be sensitive to the input order of options, which has been observed for both question answering \cite{pezeshkpour2023large, zheng2023large} and pairwise assessment \cite{zheng2023judging, wang2023large, liusie2023zero}." />
        </attvalues>
      </edge>
      <edge source="2403.13590" target="2306.05685" id="10426">
        <attvalues>
          <attvalue for="5" value=" Ongoing work has demonstrated that LLMs can be sensitive to the input order of options, which has been observed for both question answering \cite{pezeshkpour2023large, zheng2023large} and pairwise assessment \cite{zheng2023judging, wang2023large, liusie2023zero}." />
        </attvalues>
      </edge>
      <edge source="2403.13590" target="2309.03882" id="10427">
        <attvalues>
          <attvalue for="5" value=" Although debiasing approaches can be applied to enforce invariances, such methods can be computationally expensive or inapplicable to black-box settings \cite{zheng2023large}." />
        </attvalues>
      </edge>
      <edge source="2310.17714" target="2009.07503" id="10428">
        <attvalues>
          <attvalue for="5" value=" Recent studies \cite{zhang-etal-2020-minimize, zeng2020copymtl, lin-etal-2020-joint, wang-lu-2020-two, cheng-etal-2020-dynamically, zhong-chen-2021-frustratingly} in supervised RE take advantage of pre-trained language models (PLMs) and achieve SOTA performances by fine-tuning PLMs with a relation classifier." />
        </attvalues>
      </edge>
      <edge source="2310.17714" target="2210.11800" id="10429">
        <attvalues>
          <attvalue for="5" value=" Inspired by recent studies \cite{wan-etal-2022-rescue, khandelwal2019generalization, guu2020retrieval, meng2021gnn} using k-Nearest Neighbor to retrieve diverse expressions for language generation tasks, we introduce a simple but effective approach that consults training relations at test time through a nearest-neighbor search over dense vectors of lexico-syntactic patterns and provides a simple yet effective means to tackle the above issues." />
        </attvalues>
      </edge>
      <edge source="2310.17714" target="2305.18322" id="10430">
        <attvalues>
          <attvalue for="5" value=" Further, As per \cite{kaur2023refind}, REFinD includes more complex sentences than TACRED, with an average entity-pair distance of 11, compared to 8 in TACRED." />
        </attvalues>
      </edge>
      <edge source="2310.17714" target="2305.02105" id="10431">
        <attvalues>
          <attvalue for="5" value=" Recently, ICL (In-Context Learning) based approach \cite{wan2023gptre} is utilized for RE tasks." />
        </attvalues>
      </edge>
      <edge source="2310.17714" target="2302.05698" id="10433">
        <attvalues>
          <attvalue for="5" value=" Finding efficient demonstrates often relies on learning-based retrieval \cite{ye2023compositional, rubin-etal-2022-learning}." />
        </attvalues>
      </edge>
      <edge source="2210.11800" target="2009.07503" id="10434">
        <attvalues>
          <attvalue for="5" value=" Recent studies~\cite{zhang-etal-2020-minimize,Zeng_Zhang_Liu_2020,lin-etal-2020-joint,wang-lu-2020-two,cheng-etal-2020-dynamically,zhong-chen-2021-frustratingly} in supervised RE take advantage of pre-trained language models (PLMs) and achieve SOTA performances by fine-tuning PLMs with a relation classifier." />
        </attvalues>
      </edge>
      <edge source="2210.11800" target="1911.10438" id="10435">
        <attvalues>
          <attvalue for="5" value=" Recent studies~\cite{zhang-etal-2020-minimize,Zeng_Zhang_Liu_2020,lin-etal-2020-joint,wang-lu-2020-two,cheng-etal-2020-dynamically,zhong-chen-2021-frustratingly} in supervised RE take advantage of pre-trained language models (PLMs) and achieve SOTA performances by fine-tuning PLMs with a relation classifier." />
        </attvalues>
      </edge>
      <edge source="2210.11800" target="2010.03851" id="10436">
        <attvalues>
          <attvalue for="5" value=" Recent studies~\cite{zhang-etal-2020-minimize,Zeng_Zhang_Liu_2020,lin-etal-2020-joint,wang-lu-2020-two,cheng-etal-2020-dynamically,zhong-chen-2021-frustratingly} in supervised RE take advantage of pre-trained language models (PLMs) and achieve SOTA performances by fine-tuning PLMs with a relation classifier." />
        </attvalues>
      </edge>
      <edge source="2210.11800" target="2010.12812" id="10438">
        <attvalues>
          <attvalue for="5" value=" Recent studies~\cite{zhang-etal-2020-minimize,Zeng_Zhang_Liu_2020,lin-etal-2020-joint,wang-lu-2020-two,cheng-etal-2020-dynamically,zhong-chen-2021-frustratingly} in supervised RE take advantage of pre-trained language models (PLMs) and achieve SOTA performances by fine-tuning PLMs with a relation classifier." />
        </attvalues>
      </edge>
      <edge source="2210.11800" target="1911.00172" id="10439">
        <attvalues>
          <attvalue for="5" value=" Inspired by recent studies~\cite{DBLP:conf/iclr/KhandelwalLJZL20, DBLP:journals/corr/abs-2002-08909,DBLP:journals/corr/abs-2110-08743} using $k$NN to retrieve diverse expressions for language generation tasks, we introduce a simple but effective $k$NN-RE framework to address above-mentioned two problems." />
        </attvalues>
      </edge>
      <edge source="2210.11800" target="1812.04361" id="10441">
        <attvalues>
          <attvalue for="5" value=" DS augments labeled RE datasets by matching knowledge base (KB) relation triplets and raw text entity pairs in a weak-supervision fashion ~\cite{mintz-etal-2009-distant,lin-etal-2016-neural,vashishth-etal-2018-reside,chen-etal-2021-cil}." />
        </attvalues>
      </edge>
      <edge source="2210.11800" target="2106.10855" id="10442">
        <attvalues>
          <attvalue for="5" value=" DS augments labeled RE datasets by matching knowledge base (KB) relation triplets and raw text entity pairs in a weak-supervision fashion ~\cite{mintz-etal-2009-distant,lin-etal-2016-neural,vashishth-etal-2018-reside,chen-etal-2021-cil}." />
        </attvalues>
      </edge>
      <edge source="2309.10299" target="2307.13008" id="10448">
        <attvalues>
          <attvalue for="5" value=" \cite{jain2023adaptation}, the authors froze most of the model's parameters while fine-tuned only the final layer." />
        </attvalues>
      </edge>
      <edge source="2309.10299" target="2305.12606" id="10449">
        <attvalues>
          <attvalue for="5" value=" \cite{rouditchenko2023comparison} fine-tuned the entire model on unseen languages." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2006.11477" id="10451">
        <attvalues>
          <attvalue for="5" value=" Following the success of self-supervised text pretraining \cite{devlin2018bert, lewis2019bart, raffel2020exploring, he2020deberta} , self-supervised speech pretraining \cite{DBLP:conf/nips/BaevskiZMA20,DBLP:journals/taslp/HsuBTLSM21,DBLP:journals/corr/abs-2110-13900} aims to learn strong speech representation for downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2106.07447" id="10452">
        <attvalues>
          <attvalue for="5" value=" Following the success of self-supervised text pretraining \cite{devlin2018bert, lewis2019bart, raffel2020exploring, he2020deberta} , self-supervised speech pretraining \cite{DBLP:conf/nips/BaevskiZMA20,DBLP:journals/taslp/HsuBTLSM21,DBLP:journals/corr/abs-2110-13900} aims to learn strong speech representation for downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2110.13900" id="10453">
        <attvalues>
          <attvalue for="5" value=" Following the success of self-supervised text pretraining \cite{devlin2018bert, lewis2019bart, raffel2020exploring, he2020deberta} , self-supervised speech pretraining \cite{DBLP:conf/nips/BaevskiZMA20,DBLP:journals/taslp/HsuBTLSM21,DBLP:journals/corr/abs-2110-13900} aims to learn strong speech representation for downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2203.06849" id="10455">
        <attvalues>
          <attvalue for="5" value=" Despite reaching near-perfect performance on speech intent classification and speech keyword spotting on SUPERB \cite{yang2021superb}, these models' performance on ST using a randomly initialized transformer decoder \cite{tsai2022superb} is not competitive with works that incorporate knowledge from text \cite{li2020multilingual, conneau2022xtreme}." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2010.12829" id="10456">
        <attvalues>
          <attvalue for="5" value=" For example, the SLUE benchmark \cite{shon2022slue} uses DeBERTa \cite{he2020deberta} in cascaded speech understanding, DUAL \cite{lin2022dual} uses Longformer \cite{beltagy2020longformer} in end-to-end speech question answering, and \cite{li2020multilingual} uses mBART \cite{liu2020multilingual} in the speech translation task." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2010.02295" id="10457">
        <attvalues>
          <attvalue for="5" value=" Pretrained language models (PLM) are also frequently present in previous works solving SQA \cite{chuang2019speechbert,chung2020splat,lin2022dual}." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2111.10367" id="10458">
        <attvalues>
          <attvalue for="5" value=" For example, the SLUE benchmark \cite{shon2022slue} uses DeBERTa \cite{he2020deberta} in cascaded speech understanding, DUAL \cite{lin2022dual} uses Longformer \cite{beltagy2020longformer} in end-to-end speech question answering, and \cite{li2020multilingual} uses mBART \cite{liu2020multilingual} in the speech translation task." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2103.07162" id="10459">
        <attvalues>
          <attvalue for="5" value=" The more recent end-to-end approach seeks to reduce error prorogation by using speech representations directly as the input of pretrained language models \cite{li2020multilingual, lin2022dual}, and it is logical to do so since language models are cross-disciplinary learners \cite{kao2021bert}." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2006.03654" id="10460">
        <attvalues>
          <attvalue for="5" value=" For example, the SLUE benchmark \cite{shon2022slue} uses DeBERTa \cite{he2020deberta} in cascaded speech understanding, DUAL \cite{lin2022dual} uses Longformer \cite{beltagy2020longformer} in end-to-end speech question answering, and \cite{li2020multilingual} uses mBART \cite{liu2020multilingual} in the speech translation task." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2004.05150" id="10461">
        <attvalues>
          <attvalue for="5" value=" For example, the SLUE benchmark \cite{shon2022slue} uses DeBERTa \cite{he2020deberta} in cascaded speech understanding, DUAL \cite{lin2022dual} uses Longformer \cite{beltagy2020longformer} in end-to-end speech question answering, and \cite{li2020multilingual} uses mBART \cite{liu2020multilingual} in the speech translation task." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2001.08210" id="10462">
        <attvalues>
          <attvalue for="5" value=" For example, the SLUE benchmark \cite{shon2022slue} uses DeBERTa \cite{he2020deberta} in cascaded speech understanding, DUAL \cite{lin2022dual} uses Longformer \cite{beltagy2020longformer} in end-to-end speech question answering, and \cite{li2020multilingual} uses mBART \cite{liu2020multilingual} in the speech translation task." />
        </attvalues>
      </edge>
      <edge source="2211.00586" target="2105.11084" id="10463">
        <attvalues>
          <attvalue for="5" value=" Since self-supervised speech representations have been found to resemble phoneme sequences after clustering and reduction \cite{baevski2021unsupervised} when using 128 clusters, they are much more similar to character level inputs than to subword level inputs." />
        </attvalues>
      </edge>
      <edge source="2409.08199" target="2305.11834" id="10467">
        <attvalues>
          <attvalue for="5" value=" The advance of pretrained language models has spurred significant improvements across various language-related tasks \cite{devlin2019bert, radford2019language}, and has been extended to processing multimodal information \cite{llava,pengi}." />
        </attvalues>
      </edge>
      <edge source="2409.08199" target="2010.06775" id="10469">
        <attvalues>
          <attvalue for="5" value=" In response, recent studies have proposed algorithms to augment language models with visual knowledge \cite{tan2020vokenization, lu2022imagination, wangvisually, tang2023learning}." />
        </attvalues>
      </edge>
      <edge source="2409.08199" target="2402.16998" id="10470">
        <attvalues>
          <attvalue for="5" value=" Although a recent work studies the effectiveness of audio snippet embeddings in the language model representation space \cite{ngo2024language}, it is not known whether the language models have rich commonsense knowledge regarding the auditory signals, eg, which animal make a specific sound (\ref{fig:our_task})." />
        </attvalues>
      </edge>
      <edge source="2409.08199" target="2212.10465" id="10471">
        <attvalues>
          <attvalue for="5" value=" To construct this benchmark, we propose an LLM-based data-processing pipeline for the sake of the scalability of the benchmark dataset \cite{kim2023soda,mei2024wavcaps}." />
        </attvalues>
      </edge>
      <edge source="2409.08199" target="2303.17395" id="10472">
        <attvalues>
          <attvalue for="5" value=" To construct this benchmark, we propose an LLM-based data-processing pipeline for the sake of the scalability of the benchmark dataset \cite{kim2023soda,mei2024wavcaps}." />
        </attvalues>
      </edge>
      <edge source="2409.08199" target="2408.00118" id="10473">
        <attvalues>
          <attvalue for="5" value=" In particular, we test three different language models---BERT\cite{devlin2019bert}, Gemma\cite{team2024gemma}, and LLaMA\cite{dubey2024llama}---and find that all models achieve low predictive accuracy in both benchmark tasks (\ref{tab:audio_knowledge_recognition})." />
        </attvalues>
      </edge>
      <edge source="2409.08199" target="2106.09685" id="10475">
        <attvalues>
          <attvalue for="5" value=" Upon identification of auditory spans by the detector, the language model activates Low-Rank Adaptation (LoRA)\cite{hu2022lora} weights, which is finetuned with AudiotoryBench, which maintains its pretrained knowledge makes the model perform well in other tasks by deactivating LoRA weights." />
        </attvalues>
      </edge>
      <edge source="2404.12901" target="2404.01617" id="10476">
        <attvalues>
          <attvalue for="5" value=" For example, LLMs can assist in designing algorithms and managing network topologies~\cite{he2024llm,mani2023enhancing}, enhance configuration efficiency while minimizing errors~\cite{mondal2023llms,lian2023configuration}, identify hidden patterns or anomalies~\cite{zhou2023towards,kotaru2023adapting}, and introduce novel strategies for improving network security~\cite{meng2024large}." />
        </attvalues>
      </edge>
      <edge source="2404.12901" target="2310.09690" id="10479">
        <attvalues>
          <attvalue for="5" value=" For example, LLMs can assist in designing algorithms and managing network topologies~\cite{he2024llm,mani2023enhancing}, enhance configuration efficiency while minimizing errors~\cite{mondal2023llms,lian2023configuration}, identify hidden patterns or anomalies~\cite{zhou2023towards,kotaru2023adapting}, and introduce novel strategies for improving network security~\cite{meng2024large}." />
        </attvalues>
      </edge>
      <edge source="2404.12901" target="2311.17474" id="10480">
        <attvalues>
          <attvalue for="5" value=" Compared to a prior publication that merely focuses on a domain-adapted LLM framework with access to various external network tools~\cite{huang2023large}, our comprehensive survey presents a condensed overview of existing LLMN research and the lessons learned from it." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="2005.14165" id="10481">
        <attvalues>
          <attvalue for="5" value=" This has been demonstrated in the text domain where language models have achieved unprecedented zero-shot capabilities \cite{Brown20, Chowdhery22}, as well as in the audio domain, in which a single model has been shown to be adaptable to a surprisingly wide array of acoustic tasks \cite{Yang21, Borsos22}." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="2204.14198" id="10483">
        <attvalues>
          <attvalue for="5" value=" In the image/text domain, such a representation has proved achievable and capable of attaining state-of-the-art performance on many image and text comprehension tasks in a single model \cite{Alayrac22, Cho21}." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="1803.10299" id="10484">
        <attvalues>
          <attvalue for="5" value=" In the audio/text domain, joint speech and text models have been utilized for a wide range of tasks \cite{Renduchintala18, Huang20, Mariooryad22}." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="2010.11338" id="10485">
        <attvalues>
          <attvalue for="5" value=" In speech recognition, the past few years has seen a trend toward models with a joint speech and text encoder to allow pretraining on unpaired speech and text data \cite{Tang20, Bapna22, Chen22, Sainath22}." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="2210.07353" id="10486">
        <attvalues>
          <attvalue for="5" value=" In this paper, we ask if consistency regularization may be applied using the implicit alignments learned in upsampling systems like \cite{Sainath22} to achieve the performance improvements seen with the explicit alignments in \cite{Chen22}." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="2203.00006" id="10487">
        <attvalues>
          <attvalue for="5" value=" Fixed upsampling of the text inputs has been applied successfully for ASR in \cite{Sainath22} and SLU in \cite{Thomas22}, proving that an approximate alignment is sufficient for learning a joint representation." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="2204.03409" id="10488">
        <attvalues>
          <attvalue for="5" value=" Inspired by the improvements shown in \cite{Chen22_2} and \cite{Chen22}, we then show that by changing the criteria of the consistency regularization to encourage consistency under some alignment, instead of a direct frame-wise comparison, we can achieve robust WER improvements against strong, semi-supervised baselines in both a monolingual and multilingual setting, all without any learned alignment model." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="2105.00026" id="10489">
        <attvalues>
          <attvalue for="5" value=" Systems like autoencoders applied to augmented data (eg \cite{Chadebec_2022}) explicitly push representations of matched examples together, while contrastive systems like \cite{Chen20} do the same implicitly." />
        </attvalues>
      </edge>
      <edge source="2308.06125" target="2002.05709" id="10490">
        <attvalues>
          <attvalue for="5" value=" Systems like autoencoders applied to augmented data (eg \cite{Chadebec_2022}) explicitly push representations of matched examples together, while contrastive systems like \cite{Chen20} do the same implicitly." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="1910.12638" id="10491">
        <attvalues>
          <attvalue for="5" value=" Speech representation learning (SRL) has been extensively studied in the literature and shown potential for various speech recognition tasks \cite{srl_1, srl_2, srl_4}." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2006.11477" id="10492">
        <attvalues>
          <attvalue for="5" value="0 \cite{wav2vec} and then fine-tune the model for multiple tasks simultaneously by sampling training instances from the combined task-specific datasets." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2104.03502" id="10493">
        <attvalues>
          <attvalue for="5" value=" \cite{emotionrecognition, emotionrecognition_example, speaker_recognition} propose to combine such a model with a downstream network to address classification problems such as emotion recognition or speaker recognition." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2110.01077" id="10494">
        <attvalues>
          <attvalue for="5" value=" As opposed to this approach, we combine the student model with linear downstream heads and fine-tune all network parameters via a multi-task training scheme as in \cite{multitask} instead of freezing the SRL model as done in \cite{distilhubert}." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2006.05525" id="10495">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation is another method of choice to address model complexity reduction in various fields ranging from natural language processing to speech recognition \cite{kd_survey, kd_app2, kd_app1}." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2103.15760" id="10496">
        <attvalues>
          <attvalue for="5" value=" \cite{shrinking,review_lighthubert} apply knowledge distillation to large SRL methods to construct efficient speech recognition networks." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2203.15610" id="10497">
        <attvalues>
          <attvalue for="5" value=" \cite{shrinking,review_lighthubert} apply knowledge distillation to large SRL methods to construct efficient speech recognition networks." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2107.04734" id="10498">
        <attvalues>
          <attvalue for="5" value=" However, the middle layers of these SRL models also contain valuable speech representation information \cite{layer_analysis}." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2110.01900" id="10499">
        <attvalues>
          <attvalue for="5" value=" This is in contrast to common speech recognition multi-task training approaches where an SRL module is frozen, functioning as a feature extractor, and only the downstream speech recognition tasks are trained \cite{distilhubert, superb}." />
        </attvalues>
      </edge>
      <edge source="2210.16611" target="2106.07447" id="10500">
        <attvalues>
          <attvalue for="5" value=" In \cite{distilhubert}, layer-wise distillation to the HuBERT \cite{hubert} model is applied where knowledge is transferred from middle layers of the large network to construct a smaller model." />
        </attvalues>
      </edge>
      <edge source="2309.14779" target="1907.11692" id="10502">
        <attvalues>
          <attvalue for="5" value=" The recent advent of pre-trained language models, such as BERT \cite{devlin2018bert}, GPT \cite{radford2018gpt}, and RoBERTa \cite{liu2019roberta}, has brought about transformative changes in NLP." />
        </attvalues>
      </edge>
      <edge source="2309.14779" target="2012.15723" id="10504">
        <attvalues>
          <attvalue for="5" value=" For instance, LM-BFF has introduced innovative techniques like automated template generation and label word selection, significantly reducing the need for manual prompt engineering \cite{gao-etal-2021-making}." />
        </attvalues>
      </edge>
      <edge source="2309.14779" target="1909.01066" id="10505">
        <attvalues>
          <attvalue for="5" value=" Conversely, in zero-shot settings, models like LAMA \cite{petroni-etal-2019-language} and GPT-3 \cite{brown2020language} have explored tuning-free prompting." />
        </attvalues>
      </edge>
      <edge source="2309.14779" target="2005.14165" id="10506">
        <attvalues>
          <attvalue for="5" value=" Conversely, in zero-shot settings, models like LAMA \cite{petroni-etal-2019-language} and GPT-3 \cite{brown2020language} have explored tuning-free prompting." />
        </attvalues>
      </edge>
      <edge source="2404.00829" target="2102.12634" id="10507">
        <attvalues>
          <attvalue for="5" value=" However, these approaches still struggle to generate satisfying and coherent stories with closure \cite{alabdulkarim-etal-2021-automatic, piper2021narrative}." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="1806.06200" id="10511">
        <attvalues>
          <attvalue for="5" value=" Code-switching ASR has been explored for quite a long time since the conventional hybrid ASR paradigm~\cite{guo2018study}." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2007.05916" id="10512">
        <attvalues>
          <attvalue for="5" value=" Progress has also been advanced with several challenges specifically focusing on the code-switching phenomena~\cite{shi2020asru,shah2020first,diwan2021multilingual}." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="1706.03762" id="10513">
        <attvalues>
          <attvalue for="5" value=" With the recent advances in deep learning, neural end-to-end (E2E) frameworks, such as attention encoder decoder (AED)~\cite{vaswani2017attention,chorowski2015attention} and neural transducer~\cite{graves2012sequence}, have emerged as the mainstream for ASR with simplified system building pipeline and substantial performance improvement." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="1506.07503" id="10514">
        <attvalues>
          <attvalue for="5" value=" With the recent advances in deep learning, neural end-to-end (E2E) frameworks, such as attention encoder decoder (AED)~\cite{vaswani2017attention,chorowski2015attention} and neural transducer~\cite{graves2012sequence}, have emerged as the mainstream for ASR with simplified system building pipeline and substantial performance improvement." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="1211.3711" id="10515">
        <attvalues>
          <attvalue for="5" value=" With the recent advances in deep learning, neural end-to-end (E2E) frameworks, such as attention encoder decoder (AED)~\cite{vaswani2017attention,chorowski2015attention} and neural transducer~\cite{graves2012sequence}, have emerged as the mainstream for ASR with simplified system building pipeline and substantial performance improvement." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2105.11905" id="10516">
        <attvalues>
          <attvalue for="5" value=" Instead of sharing only the preliminary blocks, mixture of experts (MoE)~\cite{hou2021exploiting} was designed to share the majority of parameters, which may be able to learn more language-common feature and be better suited to limited training data conditions." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2206.02093" id="10517">
        <attvalues>
          <attvalue for="5" value=" Therefore, language-aware encoder (LAE)~\cite{tian2022lae} was further proposed to address this problem by sharing the preliminary blocks before the language-specific experts, which could model both language-specific and language-common feature efficiently." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2108.12226" id="10518">
        <attvalues>
          <attvalue for="5" value=" For better use of synthetic data, some additional loss functions~\cite{chen2022tts4pretrain,chen2021injecting} and filtering strategies~\cite{park2022unsupervised,hu2022synt++} were proposed to enforce the consistency of hypothesized labels between real and synthetic data." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2207.12028" id="10519">
        <attvalues>
          <attvalue for="5" value=" For better use of synthetic data, some additional loss functions~\cite{chen2022tts4pretrain,chen2021injecting} and filtering strategies~\cite{park2022unsupervised,hu2022synt++} were proposed to enforce the consistency of hypothesized labels between real and synthetic data." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2110.11479" id="10520">
        <attvalues>
          <attvalue for="5" value=" For better use of synthetic data, some additional loss functions~\cite{chen2022tts4pretrain,chen2021injecting} and filtering strategies~\cite{park2022unsupervised,hu2022synt++} were proposed to enforce the consistency of hypothesized labels between real and synthetic data." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2102.01547" id="10522">
        <attvalues>
          <attvalue for="5" value=" Specifically, we study the bi-encoder, LAE and MoE architectures reviewed above under the popular Conformer based AED framework implemented with two popular ASR toolkits -- ESPNet~\cite{watanabe2018espnet} and WeNet~\cite{yao2021wenet}." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2011.09301" id="10523">
        <attvalues>
          <attvalue for="5" value=" We further explore the effectiveness of language modeling, including both internal language model as well as long context language model~\cite{wei2021context}." />
        </attvalues>
      </edge>
      <edge source="2210.14448" target="2202.03647" id="10524">
        <attvalues>
          <attvalue for="5" value=" Finally, ROVER~\cite{fiscus1997post} is adopted for fusion of multiple hypothesis from various models, which has previously proven to be effective~\cite{yu2022summary,sun2018multiple,wang2022sjtu}." />
        </attvalues>
      </edge>
      <edge source="2406.02329" target="2002.06305" id="10525">
        <attvalues>
          <attvalue for="5" value=" In the context of the widespread deployment of language encoders, this paper tackles a natural question: Given two language encoders $\enc$ and $\encg$, how can we judge to what extent they are similar? This question is of practical importance---recent studies have shown that even small variations in the random seed used for training can result in significant performance differences on downstream tasks between models with the same architecture \cite{dodge2020finetuning,sellam2021multiberts} In this case, we say that two such language encoders exhibit an extrinsic difference, eg, the difference between two encoders manifests itself when considering their performance on a downstream task." />
        </attvalues>
      </edge>
      <edge source="2406.02329" target="2106.16163" id="10526">
        <attvalues>
          <attvalue for="5" value=" In the context of the widespread deployment of language encoders, this paper tackles a natural question: Given two language encoders $\enc$ and $\encg$, how can we judge to what extent they are similar? This question is of practical importance---recent studies have shown that even small variations in the random seed used for training can result in significant performance differences on downstream tasks between models with the same architecture \cite{dodge2020finetuning,sellam2021multiberts} In this case, we say that two such language encoders exhibit an extrinsic difference, eg, the difference between two encoders manifests itself when considering their performance on a downstream task." />
        </attvalues>
      </edge>
      <edge source="2406.02329" target="2110.14739" id="10527">
        <attvalues>
          <attvalue for="5" value=" Existing work studies language encoder similarity by evaluating whether two encoders produce similar representations for a finite dataset of strings \cite[][inter alia]{Hardoon2004CanonicalCA, pmlr-v97-kornblith19a,williams2021generalizedsm,boix2024gulp}, often by analyzing whether the representation sets can be approximately linearly aligned \cite{pmlr-v97-kornblith19a, pmlr-v44-li15convergent}." />
        </attvalues>
      </edge>
      <edge source="2406.02329" target="2210.06545" id="10528">
        <attvalues>
          <attvalue for="5" value=" Existing work studies language encoder similarity by evaluating whether two encoders produce similar representations for a finite dataset of strings \cite[][inter alia]{Hardoon2004CanonicalCA, pmlr-v97-kornblith19a,williams2021generalizedsm,boix2024gulp}, often by analyzing whether the representation sets can be approximately linearly aligned \cite{pmlr-v97-kornblith19a, pmlr-v44-li15convergent}." />
        </attvalues>
      </edge>
      <edge source="2406.02329" target="1905.00414" id="10529">
        <attvalues>
          <attvalue for="5" value=" Existing work studies language encoder similarity by evaluating whether two encoders produce similar representations for a finite dataset of strings \cite[][inter alia]{Hardoon2004CanonicalCA, pmlr-v97-kornblith19a,williams2021generalizedsm,boix2024gulp}, often by analyzing whether the representation sets can be approximately linearly aligned \cite{pmlr-v97-kornblith19a, pmlr-v44-li15convergent}." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2111.01690" id="10530">
        <attvalues>
          <attvalue for="5" value=" In recent years, end-to-end (E2E) \cite{li2022recent} based models have gained great interest in automatic speech recognition (ASR) systems." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="1508.01211" id="10531">
        <attvalues>
          <attvalue for="5" value=" Compared to traditional hybrid systems, E2E systems such as connectionist temporal classification (CTC) \cite{ctc}, attention-based encoder-decoder (AED) \cite{las}, and neural Transducer (NT) \cite{sequence} predict word sequences using a single neural network." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="1211.3711" id="10532">
        <attvalues>
          <attvalue for="5" value=" Compared to traditional hybrid systems, E2E systems such as connectionist temporal classification (CTC) \cite{ctc}, attention-based encoder-decoder (AED) \cite{las}, and neural Transducer (NT) \cite{sequence} predict word sequences using a single neural network." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2008.06580" id="10533">
        <attvalues>
          <attvalue for="5" value=" Conventional domain adaptation methods \cite{Bell_2021, deng2023adaptable} typically rely on speech-text pairs from the target domain." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2104.11127" id="10534">
        <attvalues>
          <attvalue for="5" value=" As a result, text-only adaptive methods have been widely proposed and studied \cite{pylkkonen2021fast, Choudhury2022}." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2104.04487" id="10536">
        <attvalues>
          <attvalue for="5" value=" Another common practice is LM fusion \cite{cabrera2021language, 8639038, levit2023external, li2023prompting}, such as shallow fusion \cite{kannan2017analysis}, deep fusion \cite{gulcehre2015using}, and cold fusion \cite{coldfusion}." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="1712.01996" id="10537">
        <attvalues>
          <attvalue for="5" value=" Another common practice is LM fusion \cite{cabrera2021language, 8639038, levit2023external, li2023prompting}, such as shallow fusion \cite{kannan2017analysis}, deep fusion \cite{gulcehre2015using}, and cold fusion \cite{coldfusion}." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="1708.06426" id="10539">
        <attvalues>
          <attvalue for="5" value=" Another common practice is LM fusion \cite{cabrera2021language, 8639038, levit2023external, li2023prompting}, such as shallow fusion \cite{kannan2017analysis}, deep fusion \cite{gulcehre2015using}, and cold fusion \cite{coldfusion}." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2002.11268" id="10540">
        <attvalues>
          <attvalue for="5" value=" Methods like density ratio \cite{mcdermott2020density} also work similarly." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2003.07705" id="10542">
        <attvalues>
          <attvalue for="5" value=" There have been increasing research efforts \cite{variani2020hybrid, factorizedAED, mhat} to modify the structure of neural Transducers." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2212.01992" id="10543">
        <attvalues>
          <attvalue for="5" value=" Factorized neural Transducer (FNT) \cite{zhao2023fast, chen2021Factorized,levit2023external, gong2023longfnt,le2023factorized} addresses this issue by introducing a standalone LM for vocabulary prediction, enabling the direct application of conventional LM adaptation methods." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2305.17304" id="10544">
        <attvalues>
          <attvalue for="5" value=" Factorized neural Transducer (FNT) \cite{zhao2023fast, chen2021Factorized,levit2023external, gong2023longfnt,le2023factorized} addresses this issue by introducing a standalone LM for vocabulary prediction, enabling the direct application of conventional LM adaptation methods." />
        </attvalues>
      </edge>
      <edge source="2309.09524" target="2211.00896" id="10545">
        <attvalues>
          <attvalue for="5" value=" Factorized neural Transducer (FNT) \cite{zhao2023fast, chen2021Factorized,levit2023external, gong2023longfnt,le2023factorized} addresses this issue by introducing a standalone LM for vocabulary prediction, enabling the direct application of conventional LM adaptation methods." />
        </attvalues>
      </edge>
      <edge source="2310.16248" target="1911.00359" id="10546">
        <attvalues>
          <attvalue for="5" value=" Web-mined datasets -- including CC100~\cite{wenzek-etal-2020-ccnet}, mC4~\cite{xue-etal-2021-mt5} and OSCAR~\cite{AbadjiOrtizSuarezRomaryetal." />
        </attvalues>
      </edge>
      <edge source="2310.16248" target="2010.11934" id="10547">
        <attvalues>
          <attvalue for="5" value=" In particular, they lay the ground for multilingual neural models like XLM-R~\cite{conneau-etal-2020-unsupervised}, mT5~\cite{xue-etal-2021-mt5} and Glot500~\cite{imanigooghari-etal-2023-glot500}." />
        </attvalues>
      </edge>
      <edge source="2310.16248" target="1911.02116" id="10548">
        <attvalues>
          <attvalue for="5" value=" In particular, they lay the ground for multilingual neural models like XLM-R~\cite{conneau-etal-2020-unsupervised}, mT5~\cite{xue-etal-2021-mt5} and Glot500~\cite{imanigooghari-etal-2023-glot500}." />
        </attvalues>
      </edge>
      <edge source="2310.16248" target="2305.12182" id="10549">
        <attvalues>
          <attvalue for="5" value=" In particular, they lay the ground for multilingual neural models like XLM-R~\cite{conneau-etal-2020-unsupervised}, mT5~\cite{xue-etal-2021-mt5} and Glot500~\cite{imanigooghari-etal-2023-glot500}." />
        </attvalues>
      </edge>
      <edge source="2310.16248" target="2103.12028" id="10550">
        <attvalues>
          <attvalue for="5" value=" However, existing web-mined datasets have systematic quality issues \cite{kreutzer-etal-2022-quality} and insufficient coverage of low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2310.16248" target="2010.14571" id="10551">
        <attvalues>
          <attvalue for="5" value=" In real-world settings, LID needs to support both \macrolanguages and their varieties; it also needs to be robust against out-of-model cousins \cite{caswell-etal-2020-language, kreutzer-etal-2022-quality}." />
        </attvalues>
      </edge>
      <edge source="2303.17649" target="1706.03762" id="10552">
        <attvalues>
          <attvalue for="5" value=" Transformer neural networks have shown great potential for natural language processing \cite{vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2303.17649" target="2005.14165" id="10554">
        <attvalues>
          <attvalue for="5" value=" These models have been scaled by increasing their number of parameters by hundreds of millions like GPT-2 \cite{radford2019language} and BERT \cite{devlin2019bert}, or up to hundreds of billions like GPT-3 \cite{brown2020language} and GPT-4 \cite{openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2303.17649" target="2303.08774" id="10555">
        <attvalues>
          <attvalue for="5" value=" These models have been scaled by increasing their number of parameters by hundreds of millions like GPT-2 \cite{radford2019language} and BERT \cite{devlin2019bert}, or up to hundreds of billions like GPT-3 \cite{brown2020language} and GPT-4 \cite{openai2023gpt4}." />
        </attvalues>
      </edge>
      <edge source="2303.17649" target="1911.00536" id="10556">
        <attvalues>
          <attvalue for="5" value=" It is also possible to use the weights of these models and fine-tune them for a particular task, for example, open domain conversational systems like DialoGPT \cite{zhang2020dialogpt}, InstructGPT and ChatGPT \cite{ouyang2022training}." />
        </attvalues>
      </edge>
      <edge source="2212.07939" target="2106.15561" id="10564">
        <attvalues>
          <attvalue for="5" value=" Text-to-Speech (TTS), which aims at synthesizing natural-sounding speech from text, has extensive applications in various industries such as entertainment, education, and so on~\cite{DBLP:journals/corr/abs-2106-15561}." />
        </attvalues>
      </edge>
      <edge source="2212.07939" target="2006.06873" id="10565">
        <attvalues>
          <attvalue for="5" value=" To overcome these limitations, many NAR models~\cite{lancucki2021fastpitch, ren2019fastspeech, DBLP:conf/iclr/0006H0QZZL21} have been proposed." />
        </attvalues>
      </edge>
      <edge source="2212.07939" target="2109.03264" id="10566">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, their quality of expressiveness is unsatisfactory because they predict prosodic features that contain pitch, duration, and energy without introducing dependency between time steps~\cite{kharitonov-etal-2022-text}." />
        </attvalues>
      </edge>
      <edge source="2212.07939" target="2010.13421" id="10567">
        <attvalues>
          <attvalue for="5" value=" \cite{hwang2021tts, DBLP:conf/interspeech/SongYKSHOYKK22, lajszczak2022distribution} claimed that the performance of NAR-TTS is poor when the training data is insufficient, devising effective data augmentation methods." />
        </attvalues>
      </edge>
      <edge source="2212.07939" target="2106.06103" id="10568">
        <attvalues>
          <attvalue for="5" value=" \cite{kim2021conditional} combined powerful generative models (eg, variational autoencoder, normalizing flow, and generative adversarial network) to improve expressiveness." />
        </attvalues>
      </edge>
      <edge source="2212.07939" target="2104.06835" id="10570">
        <attvalues>
          <attvalue for="5" value=" Especially, GraphSpeech~\cite{9413513} and Relational Gated Graph Network (RGGN)~\cite{DBLP:conf/interspeech/ZhouSL0B0M22} claimed the syntactic and semantic information of text affects the naturalness and expressiveness of speech." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1712.01769" id="10571">
        <attvalues>
          <attvalue for="5" value=" Recent progress of end-to-end (E2E) automatic speech recognition (ASR) models bridges the gap from the state-of-the-art hybrid systems \cite{google_sota_asr}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1211.3711" id="10572">
        <attvalues>
          <attvalue for="5" value=" For E2E models, connectionist temporal classification (CTC) \cite{ctc_graves} and recurrent neural network transducer (RNN-T) \cite{rnn_transducer} have been dominant approaches and reached a level of real applications \cite{he2019streaming,sainath2020streaming}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1811.06621" id="10573">
        <attvalues>
          <attvalue for="5" value=" For E2E models, connectionist temporal classification (CTC) \cite{ctc_graves} and recurrent neural network transducer (RNN-T) \cite{rnn_transducer} have been dominant approaches and reached a level of real applications \cite{he2019streaming,sainath2020streaming}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="2003.12710" id="10574">
        <attvalues>
          <attvalue for="5" value=" For E2E models, connectionist temporal classification (CTC) \cite{ctc_graves} and recurrent neural network transducer (RNN-T) \cite{rnn_transducer} have been dominant approaches and reached a level of real applications \cite{he2019streaming,sainath2020streaming}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1506.07503" id="10575">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, attention-based encoder-decoder (AED) models \cite{chorowski2015attention,las} have demonstrated the powerful modeling capability in offline tasks \cite{s2s_comparison_google,s2s_comparison_baidu,rwth_end2end} and a number of streaming models have been investigated for RNN-based models \cite{hou2017gaussian,tjandra2017local,lawson2018learning,adaptive_computation_steps,moritz2019triggered_icassp2019,hard_monotonic_attention,mocha}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1707.07413" id="10576">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, attention-based encoder-decoder (AED) models \cite{chorowski2015attention,las} have demonstrated the powerful modeling capability in offline tasks \cite{s2s_comparison_google,s2s_comparison_baidu,rwth_end2end} and a number of streaming models have been investigated for RNN-based models \cite{hou2017gaussian,tjandra2017local,lawson2018learning,adaptive_computation_steps,moritz2019triggered_icassp2019,hard_monotonic_attention,mocha}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1805.03294" id="10577">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, attention-based encoder-decoder (AED) models \cite{chorowski2015attention,las} have demonstrated the powerful modeling capability in offline tasks \cite{s2s_comparison_google,s2s_comparison_baidu,rwth_end2end} and a number of streaming models have been investigated for RNN-based models \cite{hou2017gaussian,tjandra2017local,lawson2018learning,adaptive_computation_steps,moritz2019triggered_icassp2019,hard_monotonic_attention,mocha}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1705.08091" id="10578">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, attention-based encoder-decoder (AED) models \cite{chorowski2015attention,las} have demonstrated the powerful modeling capability in offline tasks \cite{s2s_comparison_google,s2s_comparison_baidu,rwth_end2end} and a number of streaming models have been investigated for RNN-based models \cite{hou2017gaussian,tjandra2017local,lawson2018learning,adaptive_computation_steps,moritz2019triggered_icassp2019,hard_monotonic_attention,mocha}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1705.05524" id="10579">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, attention-based encoder-decoder (AED) models \cite{chorowski2015attention,las} have demonstrated the powerful modeling capability in offline tasks \cite{s2s_comparison_google,s2s_comparison_baidu,rwth_end2end} and a number of streaming models have been investigated for RNN-based models \cite{hou2017gaussian,tjandra2017local,lawson2018learning,adaptive_computation_steps,moritz2019triggered_icassp2019,hard_monotonic_attention,mocha}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1808.10088" id="10580">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, attention-based encoder-decoder (AED) models \cite{chorowski2015attention,las} have demonstrated the powerful modeling capability in offline tasks \cite{s2s_comparison_google,s2s_comparison_baidu,rwth_end2end} and a number of streaming models have been investigated for RNN-based models \cite{hou2017gaussian,tjandra2017local,lawson2018learning,adaptive_computation_steps,moritz2019triggered_icassp2019,hard_monotonic_attention,mocha}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1704.00784" id="10581">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, attention-based encoder-decoder (AED) models \cite{chorowski2015attention,las} have demonstrated the powerful modeling capability in offline tasks \cite{s2s_comparison_google,s2s_comparison_baidu,rwth_end2end} and a number of streaming models have been investigated for RNN-based models \cite{hou2017gaussian,tjandra2017local,lawson2018learning,adaptive_computation_steps,moritz2019triggered_icassp2019,hard_monotonic_attention,mocha}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1706.03762" id="10583">
        <attvalues>
          <attvalue for="5" value=" Recently, the Transformer architecture \cite{vaswani2017attention}, based on self-attention and multihead attention, has shown to outperform the RNN counterparts in various domains \cite{karita2019comparative,zeyer2019comparison}, and several streaming models have been proposed such as triggered attention \cite{moritz2020streaming_icassp2020}, continuous-integrate-and-fire (CIF) \cite{cif}, hard monotonic attention (HMA) \cite{tsunoo2019towards,miao2020transformer}, and other variants \cite{tian2019synchronous}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1909.06317" id="10584">
        <attvalues>
          <attvalue for="5" value=" Recently, the Transformer architecture \cite{vaswani2017attention}, based on self-attention and multihead attention, has shown to outperform the RNN counterparts in various domains \cite{karita2019comparative,zeyer2019comparison}, and several streaming models have been proposed such as triggered attention \cite{moritz2020streaming_icassp2020}, continuous-integrate-and-fire (CIF) \cite{cif}, hard monotonic attention (HMA) \cite{tsunoo2019towards,miao2020transformer}, and other variants \cite{tian2019synchronous}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="2001.02674" id="10585">
        <attvalues>
          <attvalue for="5" value=" Recently, the Transformer architecture \cite{vaswani2017attention}, based on self-attention and multihead attention, has shown to outperform the RNN counterparts in various domains \cite{karita2019comparative,zeyer2019comparison}, and several streaming models have been proposed such as triggered attention \cite{moritz2020streaming_icassp2020}, continuous-integrate-and-fire (CIF) \cite{cif}, hard monotonic attention (HMA) \cite{tsunoo2019towards,miao2020transformer}, and other variants \cite{tian2019synchronous}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1905.11235" id="10586">
        <attvalues>
          <attvalue for="5" value=" Recently, the Transformer architecture \cite{vaswani2017attention}, based on self-attention and multihead attention, has shown to outperform the RNN counterparts in various domains \cite{karita2019comparative,zeyer2019comparison}, and several streaming models have been proposed such as triggered attention \cite{moritz2020streaming_icassp2020}, continuous-integrate-and-fire (CIF) \cite{cif}, hard monotonic attention (HMA) \cite{tsunoo2019towards,miao2020transformer}, and other variants \cite{tian2019synchronous}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1910.11871" id="10587">
        <attvalues>
          <attvalue for="5" value=" \cite{tsunoo2019towards} also investigated the MMA framework but resorted to using all past frames to obtain a decent performance." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="2001.08290" id="10588">
        <attvalues>
          <attvalue for="5" value=" \cite{miao2020transformer} simplified the MMA framework by equipping a single MA head with each decoder layer to truncate encoder outputs as in triggered attention and perform attention over all past frames." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1912.02958" id="10589">
        <attvalues>
          <attvalue for="5" value=" Recently, the Transformer architecture \cite{vaswani2017attention}, based on self-attention and multihead attention, has shown to outperform the RNN counterparts in various domains \cite{karita2019comparative,zeyer2019comparison}, and several streaming models have been proposed such as triggered attention \cite{moritz2020streaming_icassp2020}, continuous-integrate-and-fire (CIF) \cite{cif}, hard monotonic attention (HMA) \cite{tsunoo2019towards,miao2020transformer}, and other variants \cite{tian2019synchronous}." />
        </attvalues>
      </edge>
      <edge source="2005.09394" target="1909.12406" id="10591">
        <attvalues>
          <attvalue for="5" value=" Recently, HMA was extended to the Transformer architecture, named monotonic multihead attention (MMA), by replacing each encoder-decoder attention head in the decoder with a monotonic attention (MA) head \cite{ma2019monotonic}." />
        </attvalues>
      </edge>
      <edge source="2011.02323" target="1801.06146" id="10592">
        <attvalues>
          <attvalue for="5" value=" Natural Language Processing (nlp) has witnessed a paradigm shift from employing task-specific architectures to fine-tuning the same pre-trained language models for various downstream tasks \cite{Howard_Ruder_2018, Peters_Neumann_Iyyer_Gardner_Clark_Lee_Zettlemoyer_2018}." />
        </attvalues>
      </edge>
      <edge source="2011.02323" target="2005.09093" id="10594">
        <attvalues>
          <attvalue for="5" value=" For example, mbert performs worse than non-bert models on a number of downstream tasks for the bottom 30 percent of languages in terms of dataset-size upon which \acs{mbert} is trained~\cite{Wu_Dredze_2020}." />
        </attvalues>
      </edge>
      <edge source="2011.02323" target="2009.12534" id="10595">
        <attvalues>
          <attvalue for="5" value=" There have been some concerted and noteworthy efforts to advance research in Indian languages \cite{arora2020inltk}, but very little work has been done with respect to the recently proposed Transformer-based models \cite{Devlin_Chang_Lee_Toutanova_2019, liu2019RoBERTaa}." />
        </attvalues>
      </edge>
      <edge source="2302.13498" target="1904.07531" id="10601">
        <attvalues>
          <attvalue for="5" value=" &#09;Recent advances have shown that pre-trained language models (PTMs) such as BERT~\cite{devlin2018bert}, T5~\cite{raffel2020exploring}, GPT~\cite{radford2018improving} can capture rich semantic information of text and achieve state-of-the-art performance on variance information retrieval tasks~\cite{qiao2019understanding,padaki2020rethinking,li2022cooperative}." />
        </attvalues>
      </edge>
      <edge source="2302.13498" target="2108.09346" id="10603">
        <attvalues>
          <attvalue for="5" value=" Besides, the dependencies between the inner structures in Wikipedia pages are also exploited to design pretraining tasks~\cite{ma2021pre, wu2022pre} for IR and achieve remarkable retrieval performance compared to the traditional pre-trained language models." />
        </attvalues>
      </edge>
      <edge source="2106.02232" target="1606.04870" id="10604">
        <attvalues>
          <attvalue for="5" value=" SR is an increasingly popular feature in many commercial applications such as Gmail, Outlook, Skype, Facebook Messenger, Microsoft Teams, and Uber \cite{Kannan2016,HendersonASSLGK17,shang2015neural, Deb2019DiversifyingRS,uberSR}." />
        </attvalues>
      </edge>
      <edge source="2106.02232" target="1705.00652" id="10605">
        <attvalues>
          <attvalue for="5" value=" SR is an increasingly popular feature in many commercial applications such as Gmail, Outlook, Skype, Facebook Messenger, Microsoft Teams, and Uber \cite{Kannan2016,HendersonASSLGK17,shang2015neural, Deb2019DiversifyingRS,uberSR}." />
        </attvalues>
      </edge>
      <edge source="2106.02232" target="1503.02364" id="10606">
        <attvalues>
          <attvalue for="5" value=" SR is an increasingly popular feature in many commercial applications such as Gmail, Outlook, Skype, Facebook Messenger, Microsoft Teams, and Uber \cite{Kannan2016,HendersonASSLGK17,shang2015neural, Deb2019DiversifyingRS,uberSR}." />
        </attvalues>
      </edge>
      <edge source="2106.02232" target="1907.08167" id="10607">
        <attvalues>
          <attvalue for="5" value=" SR is an increasingly popular feature in many commercial applications such as Gmail, Outlook, Skype, Facebook Messenger, Microsoft Teams, and Uber \cite{Kannan2016,HendersonASSLGK17,shang2015neural, Deb2019DiversifyingRS,uberSR}." />
        </attvalues>
      </edge>
      <edge source="2305.16755" target="2305.05403" id="10608">
        <attvalues>
          <attvalue for="5" value=" A new research area has emerged in the last few years, suggesting the importance of the explicit materialization of important negative statements about real-world subjects \cite{survey}." />
        </attvalues>
      </edge>
      <edge source="2305.16755" target="2208.09292" id="10609">
        <attvalues>
          <attvalue for="5" value=" In~\cite{arnaoutcikm2022}, methods to infer negative statements from knowledge graphs and text have been compared on a more specific negation task, namely generating salient negative commonsense statements." />
        </attvalues>
      </edge>
      <edge source="2305.16755" target="2011.07497" id="10610">
        <attvalues>
          <attvalue for="5" value=" Similarly, ~\cite{safavi-etal-2021-negater} explores graph embeddings to generate candidate negative statements, which are then scored using a fine-tuned language model (LM), by descending order of negativity." />
        </attvalues>
      </edge>
      <edge source="2305.16755" target="1905.10989" id="10611">
        <attvalues>
          <attvalue for="5" value=" \item We compare the LLM-generated negative statements to existing SOTA methods, from text~\cite{quasimodo} and knowledge graphs~\cite{arnaout2020enriching}." />
        </attvalues>
      </edge>
      <edge source="2305.16755" target="2006.04102" id="10612">
        <attvalues>
          <attvalue for="5" value=" Recently, LMs have been examined about their ability to store factual knowledge about general topics~\cite{lee-etal-2020-language,petroni-etal-2019-language}." />
        </attvalues>
      </edge>
      <edge source="2305.16755" target="1909.01066" id="10613">
        <attvalues>
          <attvalue for="5" value=" Recently, LMs have been examined about their ability to store factual knowledge about general topics~\cite{lee-etal-2020-language,petroni-etal-2019-language}." />
        </attvalues>
      </edge>
      <edge source="2305.16755" target="1911.03343" id="10615">
        <attvalues>
          <attvalue for="5" value=" Even though performing better than BERT-like models~\cite{negatedlama}, GPT-3 was not able to beat the SOTA model (inferences from KGs), neither on the true negativity of statements, nor their salience." />
        </attvalues>
      </edge>
      <edge source="2305.16755" target="2305.05976" id="10616">
        <attvalues>
          <attvalue for="5" value=" \cite{chen2023say} is an important step towards examining LLMs' understanding of the falseness of statements, however, it has four main differences from our study: (i) our prompts are not constrained to commonsense knowledge; (ii) not constrained to puzzles around a set of words, but allowed to generate arbitrary subject-relevant statements; (iii) our comparison includes SOTA baselines from KG and text, not just LLMs; (iv) our study evaluates also the salience of outputs, not just their correctness." />
        </attvalues>
      </edge>
      <edge source="2105.01691" target="1907.06170" id="10617">
        <attvalues>
          <attvalue for="5" value=" Many attempts have been made to augment neural machine translation (MT) systems to use discourse context \cite{doc-level-1, doc-level-2, doc-level-3, doc-level-4, doc-level-5, doc-level-6, doc-level-7, doc-level-8, doc-level-9, context-1}." />
        </attvalues>
      </edge>
      <edge source="2105.01691" target="1711.02173" id="10618">
        <attvalues>
          <attvalue for="5" value=" Instead, we view concatenation as a kind of data augmentation or noising method (one which pleasantly requires no alteration to the text, unlike data augmentation methods that disturb word order \cite{nmt-noise-1, nmt-noise-2} or replace words with automatically-selected words \cite{nmt-aug-1, nmt-aug-2, nmt-aug-3})." />
        </attvalues>
      </edge>
      <edge source="2105.01691" target="1905.10523" id="10619">
        <attvalues>
          <attvalue for="5" value=" Instead, we view concatenation as a kind of data augmentation or noising method (one which pleasantly requires no alteration to the text, unlike data augmentation methods that disturb word order \cite{nmt-noise-1, nmt-noise-2} or replace words with automatically-selected words \cite{nmt-aug-1, nmt-aug-2, nmt-aug-3})." />
        </attvalues>
      </edge>
      <edge source="2311.16298" target="2308.03296" id="10620">
        <attvalues>
          <attvalue for="5" value=" At the same time, in cases where it is desirable to train on as much data as possible -- such as large language models -- determining the influence of different data instances (both contextually and during pretraining) can help identify failure modes at the level of specific tokens \cite{grosse2023studying}, determine the impact of removal of intellectual property, and significantly reduce costs through more efficient model training \cite{renduchintala2023ingenious}." />
        </attvalues>
      </edge>
      <edge source="2311.16298" target="2107.07075" id="10621">
        <attvalues>
          <attvalue for="5" value=" This ranking of examples can then be used in many downstream tasks that require intelligent data selection, such as pruning datasets while maintaining or even improving model accuracy \cite{EL2N, beyond_scaling_pruning, marion2023more}; identifying outliers and misannotations in labeled data \cite{tracin, VUsable, DBLP:conf/nips/Pleiss0EW20, carlini_outliers, DBLP:conf/nips/FeldmanZ20}; or reweighting/reordering training examples to increase model robustness \cite{DBLP:conf/icml/RenZYU18, DBLP:conf/iclr/WuDN21}." />
        </attvalues>
      </edge>
      <edge source="2311.16298" target="2002.08484" id="10622">
        <attvalues>
          <attvalue for="5" value=" This ranking of examples can then be used in many downstream tasks that require intelligent data selection, such as pruning datasets while maintaining or even improving model accuracy \cite{EL2N, beyond_scaling_pruning, marion2023more}; identifying outliers and misannotations in labeled data \cite{tracin, VUsable, DBLP:conf/nips/Pleiss0EW20, carlini_outliers, DBLP:conf/nips/FeldmanZ20}; or reweighting/reordering training examples to increase model robustness \cite{DBLP:conf/icml/RenZYU18, DBLP:conf/iclr/WuDN21}." />
        </attvalues>
      </edge>
      <edge source="2311.16298" target="1803.09050" id="10623">
        <attvalues>
          <attvalue for="5" value=" This ranking of examples can then be used in many downstream tasks that require intelligent data selection, such as pruning datasets while maintaining or even improving model accuracy \cite{EL2N, beyond_scaling_pruning, marion2023more}; identifying outliers and misannotations in labeled data \cite{tracin, VUsable, DBLP:conf/nips/Pleiss0EW20, carlini_outliers, DBLP:conf/nips/FeldmanZ20}; or reweighting/reordering training examples to increase model robustness \cite{DBLP:conf/icml/RenZYU18, DBLP:conf/iclr/WuDN21}." />
        </attvalues>
      </edge>
      <edge source="2311.16298" target="2009.10795" id="10624">
        <attvalues>
          <attvalue for="5" value=" Apart from a few notable exceptions \cite{dataset_cartography, VUsable, marion2023more}, influence scores have primarily been developed and demonstrated in the context of image classification, and relatively little is known about their efficacy in downstream language-based tasks." />
        </attvalues>
      </edge>
      <edge source="2311.16298" target="1508.05326" id="10625">
        <attvalues>
          <attvalue for="5" value=" First, we benchmark a subset of influence scores on the SNLI dataset \cite{snli:emnlp2015} in the downstream task of data reduction using a pretrained BERT model \cite{BERTDevlin}." />
        </attvalues>
      </edge>
      <edge source="2401.03538" target="1904.04169" id="10628">
        <attvalues>
          <attvalue for="5" value=" The accent conversion not only is helpful for pronunciation correction in second-language(L2) learners, but also has potential applications in personalized Text-to-Speech synthesis(TTS)~\cite{Oshima_Takamichi_Toda_Neubig_Sakti_Nakamura_2021}, movie dubbing~\cite{Türk_Arslan_2002}, improving speech recognition performance~\cite{Biadsy_Weiss_Moreno_Kanvesky_Jia_2019}." />
        </attvalues>
      </edge>
      <edge source="2401.03538" target="2211.13282" id="10629">
        <attvalues>
          <attvalue for="5" value=" There are some attempts~\cite{liu2020end,Jin_Serai_Wu_Tjandra_Manohar_He_2022,Zhou_Wu_Zhang_Tian_Li} on learning AC with non-parallel data, the auto-regressive based methods ~\cite{liu2020end,Zhou_Wu_Zhang_Tian_Li} also suffer from the low inference speed and unstable attention, especially when there are multiple speakers." />
        </attvalues>
      </edge>
      <edge source="2401.03538" target="2212.10204" id="10630">
        <attvalues>
          <attvalue for="5" value=" There are some attempts~\cite{liu2020end,Jin_Serai_Wu_Tjandra_Manohar_He_2022,Zhou_Wu_Zhang_Tian_Li} on learning AC with non-parallel data, the auto-regressive based methods ~\cite{liu2020end,Zhou_Wu_Zhang_Tian_Li} also suffer from the low inference speed and unstable attention, especially when there are multiple speakers." />
        </attvalues>
      </edge>
      <edge source="2401.03538" target="2006.04558" id="10631">
        <attvalues>
          <attvalue for="5" value=" Specifically, we assume that the linguistic hidden-states are accent-independent, and firstly train a native Fastspeech2~\cite{ren2020fastspeech} model to learn the accent-independent linguistic representations." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="1706.03762" id="10632">
        <attvalues>
          <attvalue for="5" value=" In machine translation, transformer encoder-decoder models~\cite{NIPS2017_transformer}, such as NLLB-200~\cite{nllb2022nolanguage}, mT5~\cite{xue-etal-2021-mt5}, and mBART~\cite{liu-etal-2020-multilingual-denoising} predominate." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2207.04672" id="10633">
        <attvalues>
          <attvalue for="5" value=" In machine translation, transformer encoder-decoder models~\cite{NIPS2017_transformer}, such as NLLB-200~\cite{nllb2022nolanguage}, mT5~\cite{xue-etal-2021-mt5}, and mBART~\cite{liu-etal-2020-multilingual-denoising} predominate." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2010.11934" id="10634">
        <attvalues>
          <attvalue for="5" value=" In machine translation, transformer encoder-decoder models~\cite{NIPS2017_transformer}, such as NLLB-200~\cite{nllb2022nolanguage}, mT5~\cite{xue-etal-2021-mt5}, and mBART~\cite{liu-etal-2020-multilingual-denoising} predominate." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2001.08210" id="10635">
        <attvalues>
          <attvalue for="5" value=" In machine translation, transformer encoder-decoder models~\cite{NIPS2017_transformer}, such as NLLB-200~\cite{nllb2022nolanguage}, mT5~\cite{xue-etal-2021-mt5}, and mBART~\cite{liu-etal-2020-multilingual-denoising} predominate." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2005.14165" id="10636">
        <attvalues>
          <attvalue for="5" value=" The emergence of pre-trained Large Language Models (LLMs) composed solely of the transformer decoder, such as GPT series \cite{NEURIPS2020_gpt3,openai2023gpt4}, has prompted the development of pre-trained LLMs, including, PaLM~\cite{chowdhery2022palm}, and LLaMA~\cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2303.08774" id="10637">
        <attvalues>
          <attvalue for="5" value=" The emergence of pre-trained Large Language Models (LLMs) composed solely of the transformer decoder, such as GPT series \cite{NEURIPS2020_gpt3,openai2023gpt4}, has prompted the development of pre-trained LLMs, including, PaLM~\cite{chowdhery2022palm}, and LLaMA~\cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2204.02311" id="10638">
        <attvalues>
          <attvalue for="5" value=" The emergence of pre-trained Large Language Models (LLMs) composed solely of the transformer decoder, such as GPT series \cite{NEURIPS2020_gpt3,openai2023gpt4}, has prompted the development of pre-trained LLMs, including, PaLM~\cite{chowdhery2022palm}, and LLaMA~\cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2307.09288" id="10639">
        <attvalues>
          <attvalue for="5" value=" The emergence of pre-trained Large Language Models (LLMs) composed solely of the transformer decoder, such as GPT series \cite{NEURIPS2020_gpt3,openai2023gpt4}, has prompted the development of pre-trained LLMs, including, PaLM~\cite{chowdhery2022palm}, and LLaMA~\cite{touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2302.09210" id="10640">
        <attvalues>
          <attvalue for="5" value=" According to \cite{hendy2023good}, GPT-3 demonstrates comparable or superior accuracy to WMT-best for high-resource languages." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2304.04675" id="10641">
        <attvalues>
          <attvalue for="5" value=" However, \cite{zhu-etal-2024-multilingual} noted that in 8-shot scenarios, relatively small-scale LLMs (eg, 7B parameters) exhibit lower accuracy than supervised encoder-decoder models." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="1907.11692" id="10643">
        <attvalues>
          <attvalue for="5" value=" On the other hand, in models such as BERT \cite{devlin-etal-2019-bert} and RoBERTa \cite{liu2019roberta}, which consist solely of transformer encoders, the effectiveness of continual pre-training, where pre-trained models are further trained on task-specific data such as classification to improve the accuracy of the task, has been reported \cite{jin-etal-2022-lifelong, ke-etal-2022-continual}." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2110.08534" id="10644">
        <attvalues>
          <attvalue for="5" value=" On the other hand, in models such as BERT \cite{devlin-etal-2019-bert} and RoBERTa \cite{liu2019roberta}, which consist solely of transformer encoders, the effectiveness of continual pre-training, where pre-trained models are further trained on task-specific data such as classification to improve the accuracy of the task, has been reported \cite{jin-etal-2022-lifelong, ke-etal-2022-continual}." />
        </attvalues>
      </edge>
      <edge source="2407.03145" target="2309.11674" id="10646">
        <attvalues>
          <attvalue for="5" value=" { Our paper's novelty compared to \cite{xu2024paradigm, alves2024tower, guo-etal-2024-novel} lies in the following aspects." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2303.18223" id="10647">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have shown exceptional proficiency in understanding deep structures and complex semantic relationships within language \cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2006.16668" id="10648">
        <attvalues>
          <attvalue for="5" value=" Early work introduced the concept of expert capacity \cite{lepikhingshard}, which refers to the maximum number of tokens each expert can process at once." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2101.03961" id="10649">
        <attvalues>
          <attvalue for="5" value=" The MoE framework integrates multiple experts within the model, each tasked with processing specific types of inputs \cite{fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2112.06905" id="10650">
        <attvalues>
          <attvalue for="5" value=" For a given input, only a subset of experts is activated, allowing for more efficient use of computational resources \cite{du2022glam}." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2401.04088" id="10651">
        <attvalues>
          <attvalue for="5" value=" Recently, several LLMs employing MoE structures, such as DeepSeek-V3 \cite{liu2024deepseek} and Mixtral \cite{jiang2024mixtral}, have demonstrated outstanding performance on various leaderboards." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="1701.06538" id="10652">
        <attvalues>
          <attvalue for="5" value=" Despite the efficiency benefits of MoE in scaling model sizes, it introduces several new challenges and drawbacks \cite{shazeer2017outrageously}." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2204.08396" id="10653">
        <attvalues>
          <attvalue for="5" value=" StableMoE \cite{dai2022stablemoe} proposes a two-stage training approach to address the issue of routing fluctuation." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2403.07652" id="10654">
        <attvalues>
          <attvalue for="5" value=" Dynamic-MoE \cite{huang2024harder} designs a dynamic routing Mixture-of-Experts (MoE) policy that evaluates the sufficiency of current experts while reducing activated parameters by 90\%." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2405.14297" id="10656">
        <attvalues>
          <attvalue for="5" value=" DYNMoE \cite{guo2024dynamic} introduces a unique gated routing mechanism capable of adaptively determining the number of activated experts through trainable expert thresholds, even allowing for the addition or removal of experts." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2202.09368" id="10657">
        <attvalues>
          <attvalue for="5" value=" Google Brain introduces the EC routing algorithm \cite{zhou2022mixture}, which assigns experts with predetermined buffer capacities to the Top-k tokens to ensure load balance." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2306.00008" id="10658">
        <attvalues>
          <attvalue for="5" value=" The Brainformer \cite{zhou2023brainformers} also adopts this routing strategy, constructing a trainable gating matrix to project the input feature space onto scores corresponding to each expert." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2106.05974" id="10659">
        <attvalues>
          <attvalue for="5" value=" The design of routing strategy is crucial to the MoE structure, while not all tokens may be suitable for training \cite{riquelme2021scaling}." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="2401.13920" id="10660">
        <attvalues>
          <attvalue for="5" value=" LocMoE \cite{li2024locmoe} leverages orthogonal routing weights to prevent token homogenization across different expert networks and introduces the Grouped Average Pooling (GrAP) layer \cite{wang2023multi} for token feature extraction." />
        </attvalues>
      </edge>
      <edge source="2406.00023" target="1909.08053" id="10661">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} Expert-token resonance mechanism adopts the state-of-the-art MoE model Mixtral 8$\times$7B as the backbone, and utilizes MindSpeed-LLM, MindSpeed, and Megatron-LM \cite{shoeybi2019megatron} libraries for training on Ascend NPU clusters." />
        </attvalues>
      </edge>
      <edge source="2107.07402" target="1802.05365" id="10663">
        <attvalues>
          <attvalue for="5" value=" This approach has been widely successful in natural language processing (NLP) applications \cite{devlin2019bert,peters-etal-2018-deep} and is active area of research in other fields." />
        </attvalues>
      </edge>
      <edge source="2107.07402" target="1910.09932" id="10664">
        <attvalues>
          <attvalue for="5" value=" Most of work in this space as well has been in monolingual speech recognition \cite{chung2018speech2vec,tjandra2019vqvae,jiang2019improving,harwath2020learning}." />
        </attvalues>
      </edge>
      <edge source="2107.07402" target="1803.08976" id="10665">
        <attvalues>
          <attvalue for="5" value=" Most of work in this space as well has been in monolingual speech recognition \cite{chung2018speech2vec,tjandra2019vqvae,jiang2019improving,harwath2020learning}." />
        </attvalues>
      </edge>
      <edge source="2107.07402" target="1911.09602" id="10667">
        <attvalues>
          <attvalue for="5" value=" Most of work in this space as well has been in monolingual speech recognition \cite{chung2018speech2vec,tjandra2019vqvae,jiang2019improving,harwath2020learning}." />
        </attvalues>
      </edge>
      <edge source="2107.07402" target="2006.11477" id="10668">
        <attvalues>
          <attvalue for="5" value="0$ \cite{baevski2020wav2vec} the details of which are explained in the coming sections." />
        </attvalues>
      </edge>
      <edge source="2107.07402" target="2002.02848" id="10670">
        <attvalues>
          <attvalue for="5" value=" We extend the work by \cite{rivire2020unsupervised} and \cite{conneau2020unsupervised} by pretraining only on Indic languages so that speech recognition tasks have a better performance on Indic languages." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2303.03329" id="10671">
        <attvalues>
          <attvalue for="5" value=" The rapid advancements in deep learning have led to remarkable progress in automatic speech recognition (ASR) systems \cite{prabhavalkar2023end, Conformer, espnet, zhang2022wenet, cif-t, whisper, usm, mms}, resulting in models like OpenAI’s Whisper \cite{whisper}, Google’s USM \cite{usm}, and META’s MMS \cite{mms} that support multilingual speech recognition and achieve near state-of-the-art performance." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2303.01037" id="10673">
        <attvalues>
          <attvalue for="5" value=" The rapid advancements in deep learning have led to remarkable progress in automatic speech recognition (ASR) systems \cite{prabhavalkar2023end, Conformer, espnet, zhang2022wenet, cif-t, whisper, usm, mms}, resulting in models like OpenAI’s Whisper \cite{whisper}, Google’s USM \cite{usm}, and META’s MMS \cite{mms} that support multilingual speech recognition and achieve near state-of-the-art performance." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2305.13516" id="10674">
        <attvalues>
          <attvalue for="5" value=" The rapid advancements in deep learning have led to remarkable progress in automatic speech recognition (ASR) systems \cite{prabhavalkar2023end, Conformer, espnet, zhang2022wenet, cif-t, whisper, usm, mms}, resulting in models like OpenAI’s Whisper \cite{whisper}, Google’s USM \cite{usm}, and META’s MMS \cite{mms} that support multilingual speech recognition and achieve near state-of-the-art performance." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2407.18461" id="10675">
        <attvalues>
          <attvalue for="5" value=" However, these models still encounter difficulties in low-resource settings \cite{wang2024enhancing,slt-wsy, madi,childmandarin, stutter}, such as recognizing diverse subdialects \cite{basak2023challenges,zhu2023boosting,kothawade2023ditto,aksenova2022accented}." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2409.04799" id="10676">
        <attvalues>
          <attvalue for="5" value=" However, these models still encounter difficulties in low-resource settings \cite{wang2024enhancing,slt-wsy, madi,childmandarin, stutter}, such as recognizing diverse subdialects \cite{basak2023challenges,zhu2023boosting,kothawade2023ditto,aksenova2022accented}." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2206.09783" id="10677">
        <attvalues>
          <attvalue for="5" value=" However, these models still encounter difficulties in low-resource settings \cite{wang2024enhancing,slt-wsy, madi,childmandarin, stutter}, such as recognizing diverse subdialects \cite{basak2023challenges,zhu2023boosting,kothawade2023ditto,aksenova2022accented}." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2110.04908" id="10678">
        <attvalues>
          <attvalue for="5" value=" However, these models still encounter difficulties in low-resource settings \cite{wang2024enhancing,slt-wsy, madi,childmandarin, stutter}, such as recognizing diverse subdialects \cite{basak2023challenges,zhu2023boosting,kothawade2023ditto,aksenova2022accented}." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2205.08014" id="10679">
        <attvalues>
          <attvalue for="5" value=" However, these models still encounter difficulties in low-resource settings \cite{wang2024enhancing,slt-wsy, madi,childmandarin, stutter}, such as recognizing diverse subdialects \cite{basak2023challenges,zhu2023boosting,kothawade2023ditto,aksenova2022accented}." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2112.08633" id="10680">
        <attvalues>
          <attvalue for="5" value=" Against this backdrop, in-context learning (ICL) \cite{rubin2022learning,min2022rethinking} has emerged as a promising approach in natural language processing (NLP), allowing models to leverage relevant examples embedded within the input, enabling adaptation without finetuning." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2202.12837" id="10681">
        <attvalues>
          <attvalue for="5" value=" Against this backdrop, in-context learning (ICL) \cite{rubin2022learning,min2022rethinking} has emerged as a promising approach in natural language processing (NLP), allowing models to leverage relevant examples embedded within the input, enabling adaptation without finetuning." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2212.02437" id="10682">
        <attvalues>
          <attvalue for="5" value=" The success of ICL hinges on the quality of selected demonstrations \cite{agrawal2023context}, which poses unique challenges in speech processing due to the complexity of audio data." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2203.15863" id="10683">
        <attvalues>
          <attvalue for="5" value=" For instance, WAVPROMPT \cite{gao22e_interspeech} combines Wav2vec2 with an autoregressive language model for few-shot ICL in speech tasks, while Hsu et al." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2310.09424" id="10684">
        <attvalues>
          <attvalue for="5" value=" For instance, SLAM \cite{chen2024salm} leverages large language models (LLMs) to enhance ASR performance by improving keyword recognition through ICL, while Audio Flamingo \cite{audioflamingo} enables fast adaptation to unseen tasks using sentence-level embeddings." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2402.01831" id="10685">
        <attvalues>
          <attvalue for="5" value=" For instance, SLAM \cite{chen2024salm} leverages large language models (LLMs) to enhance ASR performance by improving keyword recognition through ICL, while Audio Flamingo \cite{audioflamingo} enables fast adaptation to unseen tasks using sentence-level embeddings." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2211.06687" id="10686">
        <attvalues>
          <attvalue for="5" value=" However, Audio Flamingo requires an additional audio encoder, such as LAION-CLAP \cite{LAION-CLAP}, for retrieval." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2309.07081" id="10687">
        <attvalues>
          <attvalue for="5" value=" Our approach extends word-level ICL \cite{prompt-whisper} to sentence-level ICL, enhancing Whisper’s in-context learning capabilities." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="1911.00172" id="10688">
        <attvalues>
          <attvalue for="5" value=" Techniques like $k$NN-LM \cite{knn-lm} for language modeling and $k$NN-MT \cite{knn-mt} for machine translation have shown significant promise." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2010.00710" id="10689">
        <attvalues>
          <attvalue for="5" value=" Techniques like $k$NN-LM \cite{knn-lm} for language modeling and $k$NN-MT \cite{knn-mt} for machine translation have shown significant promise." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="1702.08734" id="10690">
        <attvalues>
          <attvalue for="5" value=" Beyond ICL, retrieval-augmented methods \cite{knn-lm,knn-mt,FAISS,xu2023nearest}, have been widely adopted in NLP to enhance model performance without parameter updating." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2301.02828" id="10691">
        <attvalues>
          <attvalue for="5" value=" Beyond ICL, retrieval-augmented methods \cite{knn-lm,knn-mt,FAISS,xu2023nearest}, have been widely adopted in NLP to enhance model performance without parameter updating." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="2312.13560" id="10692">
        <attvalues>
          <attvalue for="5" value=" In ASR, $k$NN-CTC \cite{knn-ctc} uses Connectionist Temporal Classification (CTC) pseudo labels to create speech-text key-value pairs and retrieves these labels during decoding to refine the output distribution, significantly improving performance in Chinese dialect ASR." />
        </attvalues>
      </edge>
      <edge source="2409.11889" target="1709.05522" id="10694">
        <attvalues>
          <attvalue for="5" value=" Extensive experiments on AISHELL-1 \cite{aishell} and KeSpeech \cite{kespeech} validate the effectiveness of our method, demonstrating that the integration of sentence-level and token-level datastores enhances ASR performance." />
        </attvalues>
      </edge>
      <edge source="2208.13486" target="2206.04615" id="10695">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) have revolutionized how people interact with technology, representing one of the most significant breakthroughs of the modern era~\cite{srivastava2022beyond,teubner2023welcome}." />
        </attvalues>
      </edge>
      <edge source="2208.13486" target="2307.03109" id="10696">
        <attvalues>
          <attvalue for="5" value=" While these models have shown remarkable improvements across a wide range of tasks~\cite{chang2024survey} in English, their performance in low and mid-resource languages, such as Farsi, often lags behind~\cite{avetisyan2023large,shen2024understanding}." />
        </attvalues>
      </edge>
      <edge source="2208.13486" target="2405.04655" id="10697">
        <attvalues>
          <attvalue for="5" value=" While these models have shown remarkable improvements across a wide range of tasks~\cite{chang2024survey} in English, their performance in low and mid-resource languages, such as Farsi, often lags behind~\cite{avetisyan2023large,shen2024understanding}." />
        </attvalues>
      </edge>
      <edge source="2208.13486" target="2109.02846" id="10700">
        <attvalues>
          <attvalue for="5" value=" Among these is the \verb|datasets| library~\cite{lhoest-etal-2021-datasets}, which provides open-source corpora that are easily accessible to NLP researchers." />
        </attvalues>
      </edge>
      <edge source="2109.00571" target="1706.03762" id="10701">
        <attvalues>
          <attvalue for="5" value=" While Transformer \cite{DBLP:journals/corr/VaswaniSPUJGKP17} based pre-trained models \cite{devlin-etal-2019-bert, liu2019roberta} have pushed results substantially forward, they rely on in-domain labeled data to achieve their strong results." />
        </attvalues>
      </edge>
      <edge source="2109.00571" target="0907.1815" id="10703">
        <attvalues>
          <attvalue for="5" value=" While the target domain labeled data availability in DA setups ranges from little (supervised DA \cite{daume:07}) to none (unsupervised DA \cite{ ramponi2020neural}), unlabeled data is typically available in both source and target domains." />
        </attvalues>
      </edge>
      <edge source="2406.15625" target="2302.09210" id="10704">
        <attvalues>
          <attvalue for="5" value=" Despite great progress in the quality of today's state of the art machine translation (MT) systems, constraints on the amount and kinds of data available in the majority of the world's 7,000+ languages have led to yet another disparity in access and support for speakers of these languages: low-resource MT continues to be a major challenge \cite{hendy2023good, nicholas2023lost, robinson-etal-2023-chatgpt, stap-araabi-2023-chatgpt}." />
        </attvalues>
      </edge>
      <edge source="2406.15625" target="2204.05541" id="10706">
        <attvalues>
          <attvalue for="5" value=" These materials are often the result of community-driven or government-led initiatives to support language revitalization, reclamation, and mother-tongue education \cite{schreiner-etal-2020-multidirectional,liu-etal-2022-notalways, riestenberg-freemond-2024-prioritizing}." />
        </attvalues>
      </edge>
      <edge source="2406.15625" target="2004.04696" id="10707">
        <attvalues>
          <attvalue for="5" value="5 Turbo and GPT-4o -- align with the quantitative measures we obtain using BLEURT \cite{sellam2020bleurt} as an automatic metric." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="1712.01769" id="10708">
        <attvalues>
          <attvalue for="5" value=" Recently, transformer models \cite{Linhao2018speech,Yingzhu2020Persistent} based on encoder-decoder have shown superior performance in end-to-end automatic speech recognition (ASR) compared with Recurrent Neural Networks (RNNs) \cite{Chiu2020state,Li2018Fast} and Connectionist Temporal Classification (CTC) \cite{alex2006ctc}." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="2004.11544" id="10709">
        <attvalues>
          <attvalue for="5" value=" Recently, transformer models \cite{Linhao2018speech,Yingzhu2020Persistent} based on encoder-decoder have shown superior performance in end-to-end automatic speech recognition (ASR) compared with Recurrent Neural Networks (RNNs) \cite{Chiu2020state,Li2018Fast} and Connectionist Temporal Classification (CTC) \cite{alex2006ctc}." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="1711.02281" id="10710">
        <attvalues>
          <attvalue for="5" value=" To accelerate the decoding speed, non-autoregressive (NAR) transformer models \cite{gu2018trans,MARJAN2020ENTROPY,JASON2018DETERMINISTIC} are first proposed in machine translation, which can predict multiple tokens simultaneously and have been widely studied in ASR recently." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="2004.01655" id="10711">
        <attvalues>
          <attvalue for="5" value=" To accelerate the decoding speed, non-autoregressive (NAR) transformer models \cite{gu2018trans,MARJAN2020ENTROPY,JASON2018DETERMINISTIC} are first proposed in machine translation, which can predict multiple tokens simultaneously and have been widely studied in ASR recently." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="1802.06901" id="10712">
        <attvalues>
          <attvalue for="5" value=" To accelerate the decoding speed, non-autoregressive (NAR) transformer models \cite{gu2018trans,MARJAN2020ENTROPY,JASON2018DETERMINISTIC} are first proposed in machine translation, which can predict multiple tokens simultaneously and have been widely studied in ASR recently." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="2005.07903" id="10713">
        <attvalues>
          <attvalue for="5" value=" The first kind of NAR transformer model \cite{zhengkun2020spike,cass2020nat} regards the decoder as an acoustic model." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="2005.08700" id="10715">
        <attvalues>
          <attvalue for="5" value=" The second kind of NAR transformer model \cite{Yosuke2020maskctc,YEBAI2020laji,song2021duokuileni,TSNAT2021TWOPASS} regards the decoder as a language model, and the decoder can predict output conditioning on linguistic information." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="2005.04862" id="10716">
        <attvalues>
          <attvalue for="5" value=" The second kind of NAR transformer model \cite{Yosuke2020maskctc,YEBAI2020laji,song2021duokuileni,TSNAT2021TWOPASS} regards the decoder as a language model, and the decoder can predict output conditioning on linguistic information." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="2010.15025" id="10717">
        <attvalues>
          <attvalue for="5" value=" Especially, the casual mask proposed in the AR transformer \cite{attention} is used in the second kind of NAR transformers \cite{song2021duokuileni,TSNAT2021TWOPASS} to construct a unidirectional decoder (Fig." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="2104.01522" id="10718">
        <attvalues>
          <attvalue for="5" value=" Especially, the casual mask proposed in the AR transformer \cite{attention} is used in the second kind of NAR transformers \cite{song2021duokuileni,TSNAT2021TWOPASS} to construct a unidirectional decoder (Fig." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="1706.03762" id="10719">
        <attvalues>
          <attvalue for="5" value=" Especially, the casual mask proposed in the AR transformer \cite{attention} is used in the second kind of NAR transformers \cite{song2021duokuileni,TSNAT2021TWOPASS} to construct a unidirectional decoder (Fig." />
        </attvalues>
      </edge>
      <edge source="2109.06684" target="2008.04481" id="10720">
        <attvalues>
          <attvalue for="5" value="intro} (b)) have been studied in the AR transformer \cite{bidireation2020AR} and the streaming ASR \cite{bidirectional2021stream}." />
        </attvalues>
      </edge>
      <edge source="2011.06392" target="1712.05884" id="10722">
        <attvalues>
          <attvalue for="5" value=" We first introduce a framework based on Tacotron 2 \cite{shen2018natural} and enhance it with minor adaptations, which make the convergence of the model faster and more stable, especially for cross-lingual cases." />
        </attvalues>
      </edge>
      <edge source="2011.06392" target="1808.10128" id="10723">
        <attvalues>
          <attvalue for="5" value=" Different techniques have been proposed for pre-training parts of a TTS model as a pre-training step and then fine-tuning it towards new speakers \cite{chung2018semisupervised}." />
        </attvalues>
      </edge>
      <edge source="2011.06392" target="1806.04558" id="10724">
        <attvalues>
          <attvalue for="5" value=" This allows the model to infer the speaker identity from the speaker embedding and makes it possible to extend speech synthesis to unseen speakers during inference \cite{jia2019transfer, zhang2019learning}." />
        </attvalues>
      </edge>
      <edge source="2011.06392" target="2002.01953" id="10725">
        <attvalues>
          <attvalue for="5" value=" Another way of speaker adaptation is based on fine-tuning the weights of an already trained model \cite{moss2020boffin, chen2019sample}." />
        </attvalues>
      </edge>
      <edge source="2011.06392" target="1606.09282" id="10727">
        <attvalues>
          <attvalue for="5" value=" In this work, inspired by the idea of &quot;learning without forgetting&quot; \cite{li2017learning}, we look at speaker adaptation from a model weight fine-tuning perspective by preserving the previous speaker(s) in several scenarios." />
        </attvalues>
      </edge>
      <edge source="2401.07883" target="2005.14165" id="10728">
        <attvalues>
          <attvalue for="5" value=" Their ability to answer different user queries in different domains allow these models to show a notable performance in a wide range of tasks like translation, summarizing, question answering, and many others \cite{gpt3}." />
        </attvalues>
      </edge>
      <edge source="2401.07883" target="2005.11401" id="10729">
        <attvalues>
          <attvalue for="5" value=" In order to overcome this challenge, a technique called Retrieval Augmented Generation (RAG) \cite{rag_paper} was developed." />
        </attvalues>
      </edge>
      <edge source="2401.07883" target="2309.01219" id="10730">
        <attvalues>
          <attvalue for="5" value=" With this, the model can generate more cohesive answers about subjects and data not seen during the training, decreasing the occurrence of hallucinations \cite{Hallucination}." />
        </attvalues>
      </edge>
      <edge source="2401.07883" target="2212.09656" id="10731">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, this approach adds a new layer of challenges since it requires the development of a trustworthy retriever pipeline, given that the quality of the final answer can be highly affected if the retrieved text is not relevant to the user query \cite{pereira2022visconde}." />
        </attvalues>
      </edge>
      <edge source="2401.07883" target="2312.10997" id="10732">
        <attvalues>
          <attvalue for="5" value=" The landscape of RAG is rapidly expanding, with a constant influx of new papers introducing diverse implementations \cite{rag_survey}." />
        </attvalues>
      </edge>
      <edge source="2105.04339" target="1907.11692" id="10735">
        <attvalues>
          <attvalue for="5" value=" However, while \cite{hill-dictionary}'s model is based on recurrent neural network language models, DefSent is based on pre-trained language models such as BERT \cite{BERT} and RoBERTa \cite{RoBERTa}, with a fine-tuning mechanism as well as Sentence-BERT \cite{SBERT}." />
        </attvalues>
      </edge>
      <edge source="2105.04339" target="1908.10084" id="10736">
        <attvalues>
          <attvalue for="5" value=" However, while \cite{hill-dictionary}'s model is based on recurrent neural network language models, DefSent is based on pre-trained language models such as BERT \cite{BERT} and RoBERTa \cite{RoBERTa}, with a fine-tuning mechanism as well as Sentence-BERT \cite{SBERT}." />
        </attvalues>
      </edge>
      <edge source="2310.05149" target="2005.14165" id="10737">
        <attvalues>
          <attvalue for="5" value=" Large Language models (LLMs) have demonstrated impressive performance on diverse language tasks through in-context learning \cite{brown2020language, hoffmann2022training, zeng2022glm, chowdhery2022palm, openai2023gpt4, touvron2023llama}." />
        </attvalues>
      </edge>
      <edge source="2310.05149" target="1906.00300" id="10738">
        <attvalues>
          <attvalue for="5" value=" However, they still struggle with knowledge-intensive tasks that require access to a large amount of knowledge, such as open-domain question answering \cite{lee-etal-2019-latent} and commonsense reasoning \cite{zellers-etal-2018-swag}, since the implicit knowledge preserved in the parameters may be partial and insufficient." />
        </attvalues>
      </edge>
      <edge source="2310.05149" target="1808.05326" id="10739">
        <attvalues>
          <attvalue for="5" value=" However, they still struggle with knowledge-intensive tasks that require access to a large amount of knowledge, such as open-domain question answering \cite{lee-etal-2019-latent} and commonsense reasoning \cite{zellers-etal-2018-swag}, since the implicit knowledge preserved in the parameters may be partial and insufficient." />
        </attvalues>
      </edge>
      <edge source="2310.05149" target="2302.00083" id="10740">
        <attvalues>
          <attvalue for="5" value=" Recent research shows that retrieving relevant documents from an external datastore \cite{ram2023context, khattab2023demonstratesearchpredict,shi2023replug} or directly generating contextual documents from LLMs \cite{yu2023generate, sun2023recitationaugmented} both can improve LLMs' performance on knowledge-intensive tasks." />
        </attvalues>
      </edge>
      <edge source="2310.05149" target="2301.12652" id="10741">
        <attvalues>
          <attvalue for="5" value=" Recent research shows that retrieving relevant documents from an external datastore \cite{ram2023context, khattab2023demonstratesearchpredict,shi2023replug} or directly generating contextual documents from LLMs \cite{yu2023generate, sun2023recitationaugmented} both can improve LLMs' performance on knowledge-intensive tasks." />
        </attvalues>
      </edge>
      <edge source="2409.09891" target="2004.04934" id="10747">
        <attvalues>
          <attvalue for="5" value=" More recent work shows the benefit of replacing the pipeline with a unified sequence-to-sequence (Seq2Seq) model that directly converts the text sequence (a string of characters) to a pronunciation sequence (a string of pronunciation tokens including phones, lexical stresses, prosodic boundaries, etc) at the sentence level (eg, converting PIPER'S SON to 1 p ai p - 0 @ z + 1 s uh n \_B in Unisyn \cite{Fitt2000} tokens) \cite{Conkie2020, Pan2020, Sun2023, Comini2023}." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2210.11416" id="10749">
        <attvalues>
          <attvalue for="5" value=" Large Language models (LLMs) have shown impressive performance in many NLP applications (\cite{Ouyang2022TrainingLM}; \cite{Chung2022ScalingIL}; \cite{wei2022finetuned}), including commonsense reasoning, a key component to AGI \cite{Davis2015CommonsenseRA}." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2109.01652" id="10750">
        <attvalues>
          <attvalue for="5" value=" Large Language models (LLMs) have shown impressive performance in many NLP applications (\cite{Ouyang2022TrainingLM}; \cite{Chung2022ScalingIL}; \cite{wei2022finetuned}), including commonsense reasoning, a key component to AGI \cite{Davis2015CommonsenseRA}." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2005.14165" id="10751">
        <attvalues>
          <attvalue for="5" value=" Recent studies suggest that LLMs are capable of zero-shot and few-shot learning (\cite{NEURIPS2020_1457c0d6}; \cite{webson-pavlick-2022-prompt}; \cite{Chowdhery2022PaLMSL}), and that several strategies can further improve their performance, like prompt engineering and calibration (\cite{kojima2022large}; \cite{pmlr-v139-zhao21c}; \cite{jiang-etal-2021-know}; \cite{Kadavath2022LanguageM})." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2109.01247" id="10752">
        <attvalues>
          <attvalue for="5" value=" Recent studies suggest that LLMs are capable of zero-shot and few-shot learning (\cite{NEURIPS2020_1457c0d6}; \cite{webson-pavlick-2022-prompt}; \cite{Chowdhery2022PaLMSL}), and that several strategies can further improve their performance, like prompt engineering and calibration (\cite{kojima2022large}; \cite{pmlr-v139-zhao21c}; \cite{jiang-etal-2021-know}; \cite{Kadavath2022LanguageM})." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2204.02311" id="10753">
        <attvalues>
          <attvalue for="5" value=" Recent studies suggest that LLMs are capable of zero-shot and few-shot learning (\cite{NEURIPS2020_1457c0d6}; \cite{webson-pavlick-2022-prompt}; \cite{Chowdhery2022PaLMSL}), and that several strategies can further improve their performance, like prompt engineering and calibration (\cite{kojima2022large}; \cite{pmlr-v139-zhao21c}; \cite{jiang-etal-2021-know}; \cite{Kadavath2022LanguageM})." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2102.09690" id="10755">
        <attvalues>
          <attvalue for="5" value=" Recent studies suggest that LLMs are capable of zero-shot and few-shot learning (\cite{NEURIPS2020_1457c0d6}; \cite{webson-pavlick-2022-prompt}; \cite{Chowdhery2022PaLMSL}), and that several strategies can further improve their performance, like prompt engineering and calibration (\cite{kojima2022large}; \cite{pmlr-v139-zhao21c}; \cite{jiang-etal-2021-know}; \cite{Kadavath2022LanguageM})." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2012.00955" id="10756">
        <attvalues>
          <attvalue for="5" value=" Recent studies suggest that LLMs are capable of zero-shot and few-shot learning (\cite{NEURIPS2020_1457c0d6}; \cite{webson-pavlick-2022-prompt}; \cite{Chowdhery2022PaLMSL}), and that several strategies can further improve their performance, like prompt engineering and calibration (\cite{kojima2022large}; \cite{pmlr-v139-zhao21c}; \cite{jiang-etal-2021-know}; \cite{Kadavath2022LanguageM})." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2207.05221" id="10757">
        <attvalues>
          <attvalue for="5" value=" Recent studies suggest that LLMs are capable of zero-shot and few-shot learning (\cite{NEURIPS2020_1457c0d6}; \cite{webson-pavlick-2022-prompt}; \cite{Chowdhery2022PaLMSL}), and that several strategies can further improve their performance, like prompt engineering and calibration (\cite{kojima2022large}; \cite{pmlr-v139-zhao21c}; \cite{jiang-etal-2021-know}; \cite{Kadavath2022LanguageM})." />
        </attvalues>
      </edge>
      <edge source="2304.06962" target="2205.05055" id="10759">
        <attvalues>
          <attvalue for="5" value=" Since these strategies are likely emergent (\cite{wei2022emergent}; \cite{Chan2022DataDP}), we make several modifications, then evaluate them on five commonsense reasoning benchmarks." />
        </attvalues>
      </edge>
      <edge source="2409.09357" target="2006.12847" id="10762">
        <attvalues>
          <attvalue for="5" value=" Compared with conventional speech enhancement (SE) that typically employs discriminative modeling based on regression to remove noise~\cite{zhao2022frcrn} and, at most, reverb~\cite{defossez2020real, li2021simultaneous}, SR addresses a diverse set of tasks including those that are generative in nature, such as bandwidth extension, packet loss concealment, etc For both SR and SE, a common finding is that the improved perceptual quality after processing may not translate to improved intelligibility, typically measured by the word error rate (WER) of automatic speech recognition (ASR) systems, since removing distortions could alter the phonetic content~\cite{wang2023speechx, koizumi2023miipher, scheibler2024universal, e3net}." />
        </attvalues>
      </edge>
      <edge source="2409.09357" target="2308.06873" id="10763">
        <attvalues>
          <attvalue for="5" value=" One solution is to condition the model on the text transcription of the corrupted speech, which, however, may not be available during both training and inference~\cite{wang2023speechx, koizumi2023miipher, le2024voicebox}." />
        </attvalues>
      </edge>
      <edge source="2409.09357" target="2406.02092" id="10765">
        <attvalues>
          <attvalue for="5" value="1\,kHz) SR system that holistically performs denoising, dereverberation, declipping, and bandwidth extension under a masked token modeling paradigm~\cite{li2024masksr}." />
        </attvalues>
      </edge>
      <edge source="2409.09357" target="2106.07447" id="10766">
        <attvalues>
          <attvalue for="5" value=" During training, given the STFT of a corrupted speech signal, the speech encoder predicts semantic representations of the target speech, extracted using a pre-trained HuBERT model~\cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2409.09357" target="2301.11325" id="10767">
        <attvalues>
          <attvalue for="5" value=" We get inspiration from previous text-guided speech/audio synthesis research, which shows the importance of semantic modeling~\cite{agostinelli2023musiclm, kharitonov2023speak, liu2024audioldm, dong2023clipsonic}." />
        </attvalues>
      </edge>
      <edge source="2409.09357" target="2206.03065" id="10768">
        <attvalues>
          <attvalue for="5" value=" Another SR work~\cite{serra2022universal} also fuses a speech encoder with a generative model, but the encoder is optimized only on spectral targets, such as the STFT and features derived from it." />
        </attvalues>
      </edge>
      <edge source="2409.09357" target="2312.09747" id="10769">
        <attvalues>
          <attvalue for="5" value=" Also, the generated speech quality is sensitive to the codebook size of the discrete SSL tokens~\cite{wang2024selm}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2405.10255" id="10784">
        <attvalues>
          <attvalue for="5" value=" More and more applications are emerging in terms of 3D reasoning, spatial awareness, and environment interaction~\cite{3dllm2024ma}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2403.09308" id="10785">
        <attvalues>
          <attvalue for="5" value=" These spatial competences have provided significant improvements for mixed reality~\cite{fang2024enablingwaypointgenerationcollaborative, delatorre2024llmrrealtimepromptinginteractive, Bozkir_2024}, robotics~\cite{wang2024largelanguagemodelsrobotics, brohan2023rt2visionlanguageactionmodelstransfer, chang2023contextaware}, autonomous vehicles~\cite{mao2023gptdriverlearningdrivegpt, mao2024languageagentautonomousdriving, cho2024languageimagemodels3dunderstanding}, inclusive technologies~\cite{chatgpt4good}, or navigation~\cite{zhou2023navgpt, navigation2018anderson, Gu_2022}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2305.16986" id="10788">
        <attvalues>
          <attvalue for="5" value=" These spatial competences have provided significant improvements for mixed reality~\cite{fang2024enablingwaypointgenerationcollaborative, delatorre2024llmrrealtimepromptinginteractive, Bozkir_2024}, robotics~\cite{wang2024largelanguagemodelsrobotics, brohan2023rt2visionlanguageactionmodelstransfer, chang2023contextaware}, autonomous vehicles~\cite{mao2023gptdriverlearningdrivegpt, mao2024languageagentautonomousdriving, cho2024languageimagemodels3dunderstanding}, inclusive technologies~\cite{chatgpt4good}, or navigation~\cite{zhou2023navgpt, navigation2018anderson, Gu_2022}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2309.16650" id="10789">
        <attvalues>
          <attvalue for="5" value=" As these foundation models proved to be capable of understanding other modalities than text, one group of solutions involves proposing multi-modal models, integrating images~\cite{gu2023conceptgraphs}, videos~\cite{zhang2020doesexistspatiotemporalvideo}, or 3D data such as point clouds or meshes~\cite{xu2023pointllm}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2001.06891" id="10790">
        <attvalues>
          <attvalue for="5" value=" As these foundation models proved to be capable of understanding other modalities than text, one group of solutions involves proposing multi-modal models, integrating images~\cite{gu2023conceptgraphs}, videos~\cite{zhang2020doesexistspatiotemporalvideo}, or 3D data such as point clouds or meshes~\cite{xu2023pointllm}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2308.16911" id="10791">
        <attvalues>
          <attvalue for="5" value=" As these foundation models proved to be capable of understanding other modalities than text, one group of solutions involves proposing multi-modal models, integrating images~\cite{gu2023conceptgraphs}, videos~\cite{zhang2020doesexistspatiotemporalvideo}, or 3D data such as point clouds or meshes~\cite{xu2023pointllm}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2403.11835" id="10792">
        <attvalues>
          <attvalue for="5" value=" Another strong trend in the field is combining the strengths of existing models with such tools as context retrieval~\cite{ning2024llmfindautonomousgisagent} or zero-shot learning~\cite{zhang2024agent3dzeroagentzeroshot3d, Yuan_2024_CVPR}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2112.10482" id="10793">
        <attvalues>
          <attvalue for="5" value=" To measure the performance of these spatially-aware systems, multiple 3D Question and Answer (Q\&amp;A) datasets have been proposed~\cite{azuma_2022_CVPR, Zhu_2023_ICCV, ma2022sqa3d, chen2020scanrefer, yan2023comprehensive, li2023m3dbench}." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="1906.05797" id="10794">
        <attvalues>
          <attvalue for="5" value=" The questions are associated with thirteen selected scenes from the Replica dataset~\cite{replica19arxiv}, that gives access to a variety of data, such as 3D object detections, navigation meshes, and point clouds." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="1904.01201" id="10795">
        <attvalues>
          <attvalue for="5" value=" Additionally, as Replica is integrated into Habitat Sim's environment~\cite{habitat19iccv}, data such as videos, and RGB-D or semantically-segmented images with camera poses can be seamlessly extracted." />
        </attvalues>
      </edge>
      <edge source="2408.16662" target="2005.11401" id="10796">
        <attvalues>
          <attvalue for="5" value=" To demonstrate a baseline performance on the created dataset, we propose RAG3D-Chat -- Retrieval Augmented Generation (RAG) for 3D Chat -- a system that utilizes RAG~\cite{rag2020} and VLMs to identify the relevant scene context from images, texts, and an SQL database, and also has the capability to answer questions regarding navigable distances." />
        </attvalues>
      </edge>
      <edge source="2405.21028" target="2309.05922" id="10797">
        <attvalues>
          <attvalue for="5" value="1975logic}'s maxim of truthfulness, generating outputs that are not truthful \cite{rawte2023survey}." />
        </attvalues>
      </edge>
      <edge source="2405.21028" target="2109.07958" id="10798">
        <attvalues>
          <attvalue for="5" value=" In our analysis, we show that our training transfers between datasets: we train our models on TriviaQA and evaluate them on TruthfulQA \cite{lin2021truthfulqa}." />
        </attvalues>
      </edge>
      <edge source="2104.05882" target="1910.13461" id="10810">
        <attvalues>
          <attvalue for="5" value=" The remarkable development of pretrained language models \cite{devlin-etal-2019-bert,lewis-etal-2020-bart,lan2020albert} has raised questions about what precise aspects of language these models do and do not capture." />
        </attvalues>
      </edge>
      <edge source="2104.05882" target="1909.11942" id="10811">
        <attvalues>
          <attvalue for="5" value=" The remarkable development of pretrained language models \cite{devlin-etal-2019-bert,lewis-etal-2020-bart,lan2020albert} has raised questions about what precise aspects of language these models do and do not capture." />
        </attvalues>
      </edge>
      <edge source="2104.05882" target="2004.06499" id="10812">
        <attvalues>
          <attvalue for="5" value=" Probing tasks offer a means to perform fine-grained analysis of the capabilities of such models, but most existing work has focused on sentence-level analysis such as syntax \cite{hewitt-manning-2019-structural,jawahar-etal-2019-bert,vries2020what}, entities/relations \cite{papanikolaou-etal-2019-deep}, and ontological knowledge \cite{michael-etal-2020-asking}." />
        </attvalues>
      </edge>
      <edge source="2104.05882" target="1911.00313" id="10813">
        <attvalues>
          <attvalue for="5" value=" Probing tasks offer a means to perform fine-grained analysis of the capabilities of such models, but most existing work has focused on sentence-level analysis such as syntax \cite{hewitt-manning-2019-structural,jawahar-etal-2019-bert,vries2020what}, entities/relations \cite{papanikolaou-etal-2019-deep}, and ontological knowledge \cite{michael-etal-2020-asking}." />
        </attvalues>
      </edge>
      <edge source="2104.05882" target="2004.14513" id="10814">
        <attvalues>
          <attvalue for="5" value=" Probing tasks offer a means to perform fine-grained analysis of the capabilities of such models, but most existing work has focused on sentence-level analysis such as syntax \cite{hewitt-manning-2019-structural,jawahar-etal-2019-bert,vries2020what}, entities/relations \cite{papanikolaou-etal-2019-deep}, and ontological knowledge \cite{michael-etal-2020-asking}." />
        </attvalues>
      </edge>
      <edge source="2104.05882" target="2102.02080" id="10816">
        <attvalues>
          <attvalue for="5" value=" A number of studies have used pretrained models to classify discourse markers \cite{sileo-etal-2019-mining} and discourse relations \cite{nie-etal-2019-dissent,shi-demberg-2019-next}, but few \cite{koto2021top} have systematically investigated the ability of pretrained models to model discourse structure." />
        </attvalues>
      </edge>
      <edge source="2104.05882" target="2010.00153" id="10817">
        <attvalues>
          <attvalue for="5" value=" \cite{zhu-etal-2020-examining} applied the model of \cite{feng-hirst-2014-linear} to parse IMDB documents \cite{maas-etal-2011-learning} into discourse trees." />
        </attvalues>
      </edge>
      <edge source="2210.11870" target="1706.03762" id="10818">
        <attvalues>
          <attvalue for="5" value=" Transformer~\cite{vaswani2017attention} and pre-trained language models~\cite{devlin2018bert, liu2019roberta} based on it have shown a lot of success in a wide variety of NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2210.11870" target="2007.14062" id="10820">
        <attvalues>
          <attvalue for="5" value=" Many techniques have been studied to overcome this problem and BigBird~\cite{zaheer2020big} showed robust and state-of-the-art performance on various NLP downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2210.11870" target="2106.01540" id="10822">
        <attvalues>
          <attvalue for="5" value=" We show that replacing them with modified pack and unpack attention~\cite{ma2021luna} is more effective." />
        </attvalues>
      </edge>
      <edge source="2210.11870" target="2004.08483" id="10824">
        <attvalues>
          <attvalue for="5" value=" BigBird introduces two ways of capturing global information, the random sparse attention and global tokens~\cite{ainslie2020etc} which attend to and be attended by all other tokens." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2303.12712" id="10825">
        <attvalues>
          <attvalue for="5" value=" During the last years, large language models (LLMs) have shown a remarkable ability to generate and understand general-purpose language~\cite{bubeck2023sparks}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2210.14306" id="10826">
        <attvalues>
          <attvalue for="5" value=" As a result, there has been an increasing excitement in their potential to help humans solve a variety of open-ended, complex tasks across many application domains such as coding~\cite{mozannar2022reading}, healthcare~\cite{haupt2023ai} and scientific discovery~\cite{romera2023mathematical}, to name a few." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2307.03109" id="10827">
        <attvalues>
          <attvalue for="5" value=" However, evaluating and comparing the performance of different LLMs has become very challenging~\cite{chang2024asurvey}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2009.03300" id="10828">
        <attvalues>
          <attvalue for="5" value=" As a consequence, there has been a paradigm shift towards evaluating their performance according to their level of alignment with human preferences---a model is better than other models if its outputs are more frequently preferred by humans~\cite{hendryckstest2021,wang2022self,ouyang2022training,wang2023aligning,lmsys2023chatbot}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2212.10560" id="10829">
        <attvalues>
          <attvalue for="5" value=" As a consequence, there has been a paradigm shift towards evaluating their performance according to their level of alignment with human preferences---a model is better than other models if its outputs are more frequently preferred by humans~\cite{hendryckstest2021,wang2022self,ouyang2022training,wang2023aligning,lmsys2023chatbot}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2307.12966" id="10831">
        <attvalues>
          <attvalue for="5" value=" As a consequence, there has been a paradigm shift towards evaluating their performance according to their level of alignment with human preferences---a model is better than other models if its outputs are more frequently preferred by humans~\cite{hendryckstest2021,wang2022self,ouyang2022training,wang2023aligning,lmsys2023chatbot}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2306.05685" id="10832">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, both the Elo rating system and the Bradley-Terry model have faced criticism, as pairwise comparisons often fail to satisfy the fundamental axiom of transitivity, upon which both approaches rely~\cite{boubdir2023elo,bertrand2023limitations}, Recently, several studies have used the win-rate~\cite{zheng2023judging,chiang2024chatbot,boyeau2024autoeval}, which weighs comparisons equally regardless of their order and does not require the transitivity assumption, but requires humans to make pairwise comparisons between every pair of models." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2310.05470" id="10833">
        <attvalues>
          <attvalue for="5" value=" However, it has become increasingly clear that oftentimes rankings derived from benchmark datasets do not correlate well with rankings derived from human preferences---an improved ranking position in the former does not lead to an improved ranking position in the latter~\cite{zheng2023judging,li2023generative,li2023prd,chiang2023vicuna,chiang2024chatbot}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2307.02762" id="10834">
        <attvalues>
          <attvalue for="5" value=" However, it has become increasingly clear that oftentimes rankings derived from benchmark datasets do not correlate well with rankings derived from human preferences---an improved ranking position in the former does not lead to an improved ranking position in the latter~\cite{zheng2023judging,li2023generative,li2023prd,chiang2023vicuna,chiang2024chatbot}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2311.17295" id="10835">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, both the Elo rating system and the Bradley-Terry model have faced criticism, as pairwise comparisons often fail to satisfy the fundamental axiom of transitivity, upon which both approaches rely~\cite{boubdir2023elo,bertrand2023limitations}, Recently, several studies have used the win-rate~\cite{zheng2023judging,chiang2024chatbot,boyeau2024autoeval}, which weighs comparisons equally regardless of their order and does not require the transitivity assumption, but requires humans to make pairwise comparisons between every pair of models." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2403.04132" id="10837">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, both the Elo rating system and the Bradley-Terry model have faced criticism, as pairwise comparisons often fail to satisfy the fundamental axiom of transitivity, upon which both approaches rely~\cite{boubdir2023elo,bertrand2023limitations}, Recently, several studies have used the win-rate~\cite{zheng2023judging,chiang2024chatbot,boyeau2024autoeval}, which weighs comparisons equally regardless of their order and does not require the transitivity assumption, but requires humans to make pairwise comparisons between every pair of models." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2112.00861" id="10838">
        <attvalues>
          <attvalue for="5" value=" Within the literature on ranking LLMs from pairwise comparisons, most studies use the Elo rating system~\cite{askell2021general,dettmers2024qlora,bai2022training,wu2023chatarena,lin2023llm}, originally introduced for chess tournaments~\cite{elo1966uscf}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2305.14314" id="10839">
        <attvalues>
          <attvalue for="5" value=" Within the literature on ranking LLMs from pairwise comparisons, most studies use the Elo rating system~\cite{askell2021general,dettmers2024qlora,bai2022training,wu2023chatarena,lin2023llm}, originally introduced for chess tournaments~\cite{elo1966uscf}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2204.05862" id="10840">
        <attvalues>
          <attvalue for="5" value=" Within the literature on ranking LLMs from pairwise comparisons, most studies use the Elo rating system~\cite{askell2021general,dettmers2024qlora,bai2022training,wu2023chatarena,lin2023llm}, originally introduced for chess tournaments~\cite{elo1966uscf}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2305.13711" id="10841">
        <attvalues>
          <attvalue for="5" value=" Within the literature on ranking LLMs from pairwise comparisons, most studies use the Elo rating system~\cite{askell2021general,dettmers2024qlora,bai2022training,wu2023chatarena,lin2023llm}, originally introduced for chess tournaments~\cite{elo1966uscf}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2403.07008" id="10842">
        <attvalues>
          <attvalue for="5" value=" Nevertheless, both the Elo rating system and the Bradley-Terry model have faced criticism, as pairwise comparisons often fail to satisfy the fundamental axiom of transitivity, upon which both approaches rely~\cite{boubdir2023elo,bertrand2023limitations}, Recently, several studies have used the win-rate~\cite{zheng2023judging,chiang2024chatbot,boyeau2024autoeval}, which weighs comparisons equally regardless of their order and does not require the transitivity assumption, but requires humans to make pairwise comparisons between every pair of models." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2309.10621" id="10843">
        <attvalues>
          <attvalue for="5" value=" The rationale is that, if a model strongly aligns with human pre\-fe\-ren\-ces, then, the distributions of pairwise comparisons by the model and by the human should in principle match~\cite{thomas2023large,chiang2023can,verma2023preference}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2305.17926" id="10844">
        <attvalues>
          <attvalue for="5" value=" To lower the cost and increase the efficiency of ranking from pairwise comparisons, it has become a common practice to ask a strong LLM---a model known to strongly align with human preferences---to perform pairwise comparisons~\cite{thomas2023large,wang2023large,chiang2023vicuna,chiang2023can,jiang2023llm,wang2024pandalm,qin2023ischatgpt,dubois2024alpacafarm,qin2023large,liu2024aligning}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2306.02561" id="10846">
        <attvalues>
          <attvalue for="5" value=" Experiments on pairwise comparisons made by humans in the LMSYS Chatbot Arena platform~\cite{jiang2023llm} and pairwise comparisons made by three strong LLMs, namely GPT 3." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2302.06476" id="10848">
        <attvalues>
          <attvalue for="5" value=" To lower the cost and increase the efficiency of ranking from pairwise comparisons, it has become a common practice to ask a strong LLM---a model known to strongly align with human preferences---to perform pairwise comparisons~\cite{thomas2023large,wang2023large,chiang2023vicuna,chiang2023can,jiang2023llm,wang2024pandalm,qin2023ischatgpt,dubois2024alpacafarm,qin2023large,liu2024aligning}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2305.14387" id="10849">
        <attvalues>
          <attvalue for="5" value=" To lower the cost and increase the efficiency of ranking from pairwise comparisons, it has become a common practice to ask a strong LLM---a model known to strongly align with human preferences---to perform pairwise comparisons~\cite{thomas2023large,wang2023large,chiang2023vicuna,chiang2023can,jiang2023llm,wang2024pandalm,qin2023ischatgpt,dubois2024alpacafarm,qin2023large,liu2024aligning}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2306.17563" id="10850">
        <attvalues>
          <attvalue for="5" value=" To lower the cost and increase the efficiency of ranking from pairwise comparisons, it has become a common practice to ask a strong LLM---a model known to strongly align with human preferences---to perform pairwise comparisons~\cite{thomas2023large,wang2023large,chiang2023vicuna,chiang2023can,jiang2023llm,wang2024pandalm,qin2023ischatgpt,dubois2024alpacafarm,qin2023large,liu2024aligning}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2403.16950" id="10851">
        <attvalues>
          <attvalue for="5" value=" To lower the cost and increase the efficiency of ranking from pairwise comparisons, it has become a common practice to ask a strong LLM---a model known to strongly align with human preferences---to perform pairwise comparisons~\cite{thomas2023large,wang2023large,chiang2023vicuna,chiang2023can,jiang2023llm,wang2024pandalm,qin2023ischatgpt,dubois2024alpacafarm,qin2023large,liu2024aligning}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2401.04536" id="10852">
        <attvalues>
          <attvalue for="5" value=" Worrying\-ly, there are multiple lines of evidence, including our experimental findings in Figure~\ref{fig:ranks-gpt4}, showing that the rankings constructed using pairwise comparisons made by a strong LLM are sometimes different to those constructed using pairwise comparisons by humans~\cite{zheng2023judging, li2023prd,boubdir2023elo,singhal2023large,dettmers2024qlora,davidson2024evaluating,hou2024large}, questioning the rationale above." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2301.09633" id="10854">
        <attvalues>
          <attvalue for="5" value=" Prediction-powered inference~\cite{angelopoulos2023prediction,angelopoulos2023ppi++,zrnic2024cross} is a recently introduced statistical framework to obtain valid $p$-values and confidence intervals about a population-level quantity such as the mean outcome or a regression coefficient using a small labeled dataset and a large unlabeled dataset, whose labels are imputed using a black-box machine learning model." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2311.01453" id="10855">
        <attvalues>
          <attvalue for="5" value=" Prediction-powered inference~\cite{angelopoulos2023prediction,angelopoulos2023ppi++,zrnic2024cross} is a recently introduced statistical framework to obtain valid $p$-values and confidence intervals about a population-level quantity such as the mean outcome or a regression coefficient using a small labeled dataset and a large unlabeled dataset, whose labels are imputed using a black-box machine learning model." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2309.16598" id="10856">
        <attvalues>
          <attvalue for="5" value=" Prediction-powered inference~\cite{angelopoulos2023prediction,angelopoulos2023ppi++,zrnic2024cross} is a recently introduced statistical framework to obtain valid $p$-values and confidence intervals about a population-level quantity such as the mean outcome or a regression coefficient using a small labeled dataset and a large unlabeled dataset, whose labels are imputed using a black-box machine learning model." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2311.09476" id="10857">
        <attvalues>
          <attvalue for="5" value="~\cite{saadfalcon2023ares} has used prediction-powered inference to construct (single) rankings, rather than rank-sets." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2307.15361" id="10858">
        <attvalues>
          <attvalue for="5" value=" Only recently, a paucity of work has focused on joint measures of uncertainty for rankings~\cite{neuhof2023confident, rising2021uncertainty, al2022simultaneous, klein2020joint}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2202.01279" id="10859">
        <attvalues>
          <attvalue for="5" value=" In recent years, there has also been a flurry of work on ranking LLMs using benchmark datasets with manually hand-crafted inputs and ground-truth outputs~\cite{bach2022promptsource,wei2022finetuned,talmor2019commonsense,mishra2022cross,chen2021evaluating,liang2023holistic,longpre2023flan}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2109.01652" id="10860">
        <attvalues>
          <attvalue for="5" value=" In recent years, there has also been a flurry of work on ranking LLMs using benchmark datasets with manually hand-crafted inputs and ground-truth outputs~\cite{bach2022promptsource,wei2022finetuned,talmor2019commonsense,mishra2022cross,chen2021evaluating,liang2023holistic,longpre2023flan}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="1811.00937" id="10861">
        <attvalues>
          <attvalue for="5" value=" In recent years, there has also been a flurry of work on ranking LLMs using benchmark datasets with manually hand-crafted inputs and ground-truth outputs~\cite{bach2022promptsource,wei2022finetuned,talmor2019commonsense,mishra2022cross,chen2021evaluating,liang2023holistic,longpre2023flan}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2107.03374" id="10863">
        <attvalues>
          <attvalue for="5" value=" In recent years, there has also been a flurry of work on ranking LLMs using benchmark datasets with manually hand-crafted inputs and ground-truth outputs~\cite{bach2022promptsource,wei2022finetuned,talmor2019commonsense,mishra2022cross,chen2021evaluating,liang2023holistic,longpre2023flan}." />
        </attvalues>
      </edge>
      <edge source="2402.17826" target="2301.13688" id="10865">
        <attvalues>
          <attvalue for="5" value=" In recent years, there has also been a flurry of work on ranking LLMs using benchmark datasets with manually hand-crafted inputs and ground-truth outputs~\cite{bach2022promptsource,wei2022finetuned,talmor2019commonsense,mishra2022cross,chen2021evaluating,liang2023holistic,longpre2023flan}." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2005.14165" id="10867">
        <attvalues>
          <attvalue for="5" value=" In particular, the release of public APIs and interfaces such as GPT-3 and ChatGPT \cite{brown-etal-2020-language,openai-2021-chatgpt} have enabled widespread public experimentation on the text generation capabilities of language models." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2204.02311" id="10868">
        <attvalues>
          <attvalue for="5" value=" Recent research and public attention has demonstrated that large language models (eg GPT-3/4, PaLM, and OPT; \cite{brown-etal-2020-language,chowdhery-etal-2022-palm,zhang-etal-2022-opt,openai-2023-gpt4}) can achieve remarkable performance both on standard NLP benchmarks and on open-ended natural language generation tasks from the general public \cite{wang-etal-2019-superglue,johnson-2022-ai}." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2205.01068" id="10869">
        <attvalues>
          <attvalue for="5" value=" Recent research and public attention has demonstrated that large language models (eg GPT-3/4, PaLM, and OPT; \cite{brown-etal-2020-language,chowdhery-etal-2022-palm,zhang-etal-2022-opt,openai-2023-gpt4}) can achieve remarkable performance both on standard NLP benchmarks and on open-ended natural language generation tasks from the general public \cite{wang-etal-2019-superglue,johnson-2022-ai}." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2303.08774" id="10870">
        <attvalues>
          <attvalue for="5" value=" Recent research and public attention has demonstrated that large language models (eg GPT-3/4, PaLM, and OPT; \cite{brown-etal-2020-language,chowdhery-etal-2022-palm,zhang-etal-2022-opt,openai-2023-gpt4}) can achieve remarkable performance both on standard NLP benchmarks and on open-ended natural language generation tasks from the general public \cite{wang-etal-2019-superglue,johnson-2022-ai}." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="1905.00537" id="10871">
        <attvalues>
          <attvalue for="5" value=" Recent research and public attention has demonstrated that large language models (eg GPT-3/4, PaLM, and OPT; \cite{brown-etal-2020-language,chowdhery-etal-2022-palm,zhang-etal-2022-opt,openai-2023-gpt4}) can achieve remarkable performance both on standard NLP benchmarks and on open-ended natural language generation tasks from the general public \cite{wang-etal-2019-superglue,johnson-2022-ai}." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2108.07258" id="10872">
        <attvalues>
          <attvalue for="5" value=" Due to their widespread applicability, language models have been called ``foundation models'' for NLP \cite{bommasani-etal-2021-on}." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2002.12327" id="10873">
        <attvalues>
          <attvalue for="5" value=" Previous studies have investigated both the outputs and internal mechanisms of language models, originally focusing on masked (eg fill-in-the-blank) ``BERT'' models and establishing the field of ``BERTology'' (see \cite{rogers-etal-2020-a} for a survey)." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2202.07785" id="10874">
        <attvalues>
          <attvalue for="5" value=" This feature makes large language models tempting but unreliable to use in many practical applications \cite{ganguli-etal-2022-predictability}." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2112.04359" id="10875">
        <attvalues>
          <attvalue for="5" value=" We also leave deeper ethical discussions of the societal implications of language models to surveys focused specifically on that area (eg \cite{weidinger-etal-2021-ethical,weidinger-etal-2022-taxonomy})." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2211.11483" id="10877">
        <attvalues>
          <attvalue for="5" value=" Understanding these behaviors has broad implications for informed applications in industry \cite{weidinger-etal-2021-ethical} and general questions about meaning and ``understanding'' in artificial agents \cite{bender-koller-2020-climbing,mitchell-krakauer-2022-the,shardlow-przybyla-2022-deanthropomorphising}." />
        </attvalues>
      </edge>
      <edge source="2303.11504" target="2110.08300" id="10878">
        <attvalues>
          <attvalue for="5" value=" In this way, we hope to combat anecdote-driven language model ``hype'' with informed hype grounded in what language models actually can and cannot do \cite{bowman-2021-when}, while also highlighting potential future directions of research in language model behavioral analysis." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="1211.3711" id="10879">
        <attvalues>
          <attvalue for="5" value=" In the past few years, end-to-end models, such as connectionist temporal classification (CTC)~\cite{ctc}, RNN-Transducer (RNN-T)~\cite{rnnt}, and attention-based encoder-decoder (AED)~\cite{speech-transformer} models, have achieved significant success on various ASR tasks." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="2010.06030" id="10880">
        <attvalues>
          <attvalue for="5" value=" Among them, chunk-based acoustic encoders~\cite{dual,saa,u2} have gained popularity and have been adopted in many previous works." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="1902.06450" id="10881">
        <attvalues>
          <attvalue for="5" value=" Among them, chunk-based acoustic encoders~\cite{dual,saa,u2} have gained popularity and have been adopted in many previous works." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="1706.03762" id="10883">
        <attvalues>
          <attvalue for="5" value=" These methods utilize bi-directional recurrent networks~\cite{blstm} or fully-connected self-attention networks~\cite{transformer} within a chunk." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="2104.02207" id="10884">
        <attvalues>
          <attvalue for="5" value=" In streaming scenarios such as real-time subtitles, ASR systems need to decode speech with low latency, producing words as soon as possible~\cite{upl}." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="2010.11148" id="10885">
        <attvalues>
          <attvalue for="5" value=" Another way to reduce latency is to apply regularization either on loss function~\cite{fastemit,peak-first-ctc} or input spectrogram~\cite{trimtail} to push forward the emission of tokens." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="2211.03284" id="10886">
        <attvalues>
          <attvalue for="5" value=" Another way to reduce latency is to apply regularization either on loss function~\cite{fastemit,peak-first-ctc} or input spectrogram~\cite{trimtail} to push forward the emission of tokens." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="2211.00522" id="10887">
        <attvalues>
          <attvalue for="5" value=" Another way to reduce latency is to apply regularization either on loss function~\cite{fastemit,peak-first-ctc} or input spectrogram~\cite{trimtail} to push forward the emission of tokens." />
        </attvalues>
      </edge>
      <edge source="2305.10649" target="2302.08579" id="10888">
        <attvalues>
          <attvalue for="5" value=" We argue that previous works mainly focus on the decoder part of encoder-decoder E2E ASR structure rather than the encoder part to estimate the internal LM because the encoder part is usually optimized with CTC loss and CTC is generally not considered capable of modeling context between output tokens due to conditional independence assumption~\cite{adapter}." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="2010.00784" id="10890">
        <attvalues>
          <attvalue for="5" value=" We also show that the frugal pre-training helps \fpdm\ resist catastrophic forgetting so very common when transformers undergo continual in-domain pre-training \cite{dontstop,empmultidomain}." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="2103.06268" id="10891">
        <attvalues>
          <attvalue for="5" value=" In the legal domain, we focus on the task of automating contract review \cite{cuad}, which involves finding key clauses in legal contracts." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="1907.11692" id="10892">
        <attvalues>
          <attvalue for="5" value=" Most of the pre-training strategies involve variants of Masked Language Modelling (MLM) \cite{roberta}, Next Sentence Prediction (NSP) \cite{bert}, Sentence Order Prediction (SOP) \cite{albert}, etc that use local sentence/span-level contexts as supervision signals." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="1909.11942" id="10894">
        <attvalues>
          <attvalue for="5" value=" Most of the pre-training strategies involve variants of Masked Language Modelling (MLM) \cite{roberta}, Next Sentence Prediction (NSP) \cite{bert}, Sentence Order Prediction (SOP) \cite{albert}, etc that use local sentence/span-level contexts as supervision signals." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="1903.10676" id="10895">
        <attvalues>
          <attvalue for="5" value="17$ billion word corpus was performed on $8$ GPUs for around $40$ days to obtain \scibert ~\cite{scibert}." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="2007.06400" id="10896">
        <attvalues>
          <attvalue for="5" value=" This information is generally stored as either `metadata' of the document \cite{metadata1, metadata2, metadata3}, or in terms of a `taxonomy' \cite{taxonomy1, taxonomy2} of documents." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="2203.15827" id="10897">
        <attvalues>
          <attvalue for="5" value=" While few models such as LinkBERT \cite{linkbert}, MetricBERT \cite{metricbert}, etc have used document metadata as an additional signal, no work to the best of our knowledge has singularly leveraged taxonomy-based information." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="1905.06566" id="10899">
        <attvalues>
          <attvalue for="5" value=" We use a hierarchical architecture \cite{hibert} and propose various innovations (see Figure \ref{fig:pipeline}) - (a)." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="1908.10084" id="10900">
        <attvalues>
          <attvalue for="5" value=" We initialize the lower-level encoder using a pre-trained sentence transformer (sBERT/sRoBERTa \cite{sbert}) and freeze its weights." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="2110.07143" id="10901">
        <attvalues>
          <attvalue for="5" value=" This design choice (inspired by works that initialize a larger encoder through a smaller pre-trained encoder - eg, Bert2BERT \cite{bert2bert}) helps us to directly work with sentence embeddings as inputs which in turn enables much larger contexts in a single input, and decreases the required pre-training compute by a huge margin." />
        </attvalues>
      </edge>
      <edge source="2306.06190" target="1904.01608" id="10902">
        <attvalues>
          <attvalue for="5" value=" In the domain of scientific papers, we focus on tasks such as extracting important scientific keywords \cite{bc5cdr,jnlpba,ncbi}, extracting the type of relation between such keywords \cite{chemprot, scierc}, as well as classifying citation intents \cite{scicite}." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1409.0473" id="10905">
        <attvalues>
          <attvalue for="5" value=" The base architecture to solve this problem is the attention-based encoder-decoder~\cite{rush2015neural} which greatly improved the result of neural translation~\cite{bahdanau2014neural}." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1602.06023" id="10906">
        <attvalues>
          <attvalue for="5" value=" Former studies proposed various models to better understand document~\cite{nallapati2016abstractive}, handle the out-of-vocabulary(OOV) problem~\cite{see2017get}, reduce the repetition\cite{chen2016distraction-based,li2019in} or divided the summarization problem into two steps(select and rewrite)~\cite{moroshko2019an,chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1704.04368" id="10907">
        <attvalues>
          <attvalue for="5" value=" Former studies proposed various models to better understand document~\cite{nallapati2016abstractive}, handle the out-of-vocabulary(OOV) problem~\cite{see2017get}, reduce the repetition\cite{chen2016distraction-based,li2019in} or divided the summarization problem into two steps(select and rewrite)~\cite{moroshko2019an,chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1610.08462" id="10908">
        <attvalues>
          <attvalue for="5" value=" Former studies proposed various models to better understand document~\cite{nallapati2016abstractive}, handle the out-of-vocabulary(OOV) problem~\cite{see2017get}, reduce the repetition\cite{chen2016distraction-based,li2019in} or divided the summarization problem into two steps(select and rewrite)~\cite{moroshko2019an,chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1909.10852" id="10909">
        <attvalues>
          <attvalue for="5" value=" Former studies proposed various models to better understand document~\cite{nallapati2016abstractive}, handle the out-of-vocabulary(OOV) problem~\cite{see2017get}, reduce the repetition\cite{chen2016distraction-based,li2019in} or divided the summarization problem into two steps(select and rewrite)~\cite{moroshko2019an,chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1902.10360" id="10910">
        <attvalues>
          <attvalue for="5" value=" Former studies proposed various models to better understand document~\cite{nallapati2016abstractive}, handle the out-of-vocabulary(OOV) problem~\cite{see2017get}, reduce the repetition\cite{chen2016distraction-based,li2019in} or divided the summarization problem into two steps(select and rewrite)~\cite{moroshko2019an,chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1805.11080" id="10911">
        <attvalues>
          <attvalue for="5" value=" Former studies proposed various models to better understand document~\cite{nallapati2016abstractive}, handle the out-of-vocabulary(OOV) problem~\cite{see2017get}, reduce the repetition\cite{chen2016distraction-based,li2019in} or divided the summarization problem into two steps(select and rewrite)~\cite{moroshko2019an,chen2018fast}." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1910.13461" id="10913">
        <attvalues>
          <attvalue for="5" value=" Recent researches also introduced Pretrained Language Model(PLM)~\cite{liu2019text,lewis2019bart:,raffel2019exploring} to this task." />
        </attvalues>
      </edge>
      <edge source="2106.10084" target="1506.03340" id="10915">
        <attvalues>
          <attvalue for="5" value=" Our results on the most used CNN-Daily Mail(CNN-DM) dataset~\cite{hermann2015teaching} show that different style has a different impact on the model adaption, convergence speed, readability, and abstraction of generated summaries." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2107.13586" id="10918">
        <attvalues>
          <attvalue for="5" value=" Prompts (\cite{liu2023pre}) are additional contents which would be integrated with the original input data to enhance the performance of a trained target model." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2101.00190" id="10919">
        <attvalues>
          <attvalue for="5" value=" They could be continuous (\cite{li-liang-2021-prefix}; \cite{qin-eisner-2021-learning}), possibly work in hidden layers (\cite{sun-etal-2022-bbtv2}), and have already been applied to many modalities, such as text (\cite{lester-etal-2021-power}; \cite{liu-etal-2022-p}), vision (\cite{jia2022visual}; \cite{Sohn_2023_CVPR}), and graph (\cite{10." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2104.06599" id="10920">
        <attvalues>
          <attvalue for="5" value=" They could be continuous (\cite{li-liang-2021-prefix}; \cite{qin-eisner-2021-learning}), possibly work in hidden layers (\cite{sun-etal-2022-bbtv2}), and have already been applied to many modalities, such as text (\cite{lester-etal-2021-power}; \cite{liu-etal-2022-p}), vision (\cite{jia2022visual}; \cite{Sohn_2023_CVPR}), and graph (\cite{10." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2205.11200" id="10921">
        <attvalues>
          <attvalue for="5" value=" Hence, many black-box methods, based on evolution (\cite{pmlr-v162-sun22e}; \cite{sun-etal-2022-bbtv2}), reinforcement learning (\cite{deng-etal-2022-rlprompt}; \cite{zhang2023tempera}; \cite{diao2023blackbox}) or search strategies (\cite{prasad-etal-2023-grips}), improve the prompts without the information of gradients and parameters inside the target model." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2104.08691" id="10922">
        <attvalues>
          <attvalue for="5" value=" They could be continuous (\cite{li-liang-2021-prefix}; \cite{qin-eisner-2021-learning}), possibly work in hidden layers (\cite{sun-etal-2022-bbtv2}), and have already been applied to many modalities, such as text (\cite{lester-etal-2021-power}; \cite{liu-etal-2022-p}), vision (\cite{jia2022visual}; \cite{Sohn_2023_CVPR}), and graph (\cite{10." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2203.12119" id="10923">
        <attvalues>
          <attvalue for="5" value=" They could be continuous (\cite{li-liang-2021-prefix}; \cite{qin-eisner-2021-learning}), possibly work in hidden layers (\cite{sun-etal-2022-bbtv2}), and have already been applied to many modalities, such as text (\cite{lester-etal-2021-power}; \cite{liu-etal-2022-p}), vision (\cite{jia2022visual}; \cite{Sohn_2023_CVPR}), and graph (\cite{10." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2210.00990" id="10924">
        <attvalues>
          <attvalue for="5" value=" They could be continuous (\cite{li-liang-2021-prefix}; \cite{qin-eisner-2021-learning}), possibly work in hidden layers (\cite{sun-etal-2022-bbtv2}), and have already been applied to many modalities, such as text (\cite{lester-etal-2021-power}; \cite{liu-etal-2022-p}), vision (\cite{jia2022visual}; \cite{Sohn_2023_CVPR}), and graph (\cite{10." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2010.15980" id="10927">
        <attvalues>
          <attvalue for="5" value=" Gradient-based methods (\cite{wallace-etal-2019-universal}; \cite{shin-etal-2020-autoprompt}) are the most intuitive one." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2201.03514" id="10928">
        <attvalues>
          <attvalue for="5" value=" Hence, many black-box methods, based on evolution (\cite{pmlr-v162-sun22e}; \cite{sun-etal-2022-bbtv2}), reinforcement learning (\cite{deng-etal-2022-rlprompt}; \cite{zhang2023tempera}; \cite{diao2023blackbox}) or search strategies (\cite{prasad-etal-2023-grips}), improve the prompts without the information of gradients and parameters inside the target model." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2205.12548" id="10929">
        <attvalues>
          <attvalue for="5" value=" Hence, many black-box methods, based on evolution (\cite{pmlr-v162-sun22e}; \cite{sun-etal-2022-bbtv2}), reinforcement learning (\cite{deng-etal-2022-rlprompt}; \cite{zhang2023tempera}; \cite{diao2023blackbox}) or search strategies (\cite{prasad-etal-2023-grips}), improve the prompts without the information of gradients and parameters inside the target model." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2203.07281" id="10931">
        <attvalues>
          <attvalue for="5" value=" Hence, many black-box methods, based on evolution (\cite{pmlr-v162-sun22e}; \cite{sun-etal-2022-bbtv2}), reinforcement learning (\cite{deng-etal-2022-rlprompt}; \cite{zhang2023tempera}; \cite{diao2023blackbox}) or search strategies (\cite{prasad-etal-2023-grips}), improve the prompts without the information of gradients and parameters inside the target model." />
        </attvalues>
      </edge>
      <edge source="2310.01260" target="2303.18223" id="10932">
        <attvalues>
          <attvalue for="5" value=" In recent years, large language models (LLMs) (\cite{ijcai2021p0612}; \cite{zhao2023survey}) are the most popular star among various new AI applications." />
        </attvalues>
      </edge>
      <edge source="2210.17035" target="1910.02893" id="10944">
        <attvalues>
          <attvalue for="5" value=" In this work, the experiments are done using datasets across different domains like the open domain and the novel domain to determine the accuracy of the metric \end{itemize} For the rest of the discussion, the experiments of the proposed metrics are conducted on the following synthetic datasets: \begin{itemize} \item Tagged corruption model - \cite{stahlberg-kumar-2021-synthetic} \end{itemize} \begin{itemize} \item Backtranslation method - \cite{xie-etal-2018-noising} \end{itemize} \begin{itemize} \item Rule-based method - \cite{awasthi-etal-2019-parallel} \end{itemize} The backtranslation method is applied on the novel domain to verify if the metrics still holds for different domains other than the open domain." />
        </attvalues>
      </edge>
      <edge source="2210.17035" target="1911.02825" id="10946">
        <attvalues>
          <attvalue for="5" value=" This work aims to introduce these metrics in terms of : \begin{itemize} \item Reliability - This metric can help evaluate which part of the dataset truly resembles humans in terms of creating grammatical errors in a particular sentence \end{itemize} \begin{itemize} \item Diversity - Many synthetic data generation techniques like \cite{stahlberg-kumar-2021-synthetic} and \cite{zhou-etal-2020-improving-grammatical} show that diverse synthetic data can help improve the GEC systems." />
        </attvalues>
      </edge>
      <edge source="2304.03087" target="2006.03644" id="10955">
        <attvalues>
          <attvalue for="5" value=" For example, the effective algorithms for the classifiers are support vector machine (SVM), logistic regression, naive bayes, decision tree and etc \cite{aldayel2021stance}." />
        </attvalues>
      </edge>
      <edge source="2304.03087" target="2010.15980" id="10957">
        <attvalues>
          <attvalue for="5" value=" The prompting strategies provide further improvements for stance detection performance\cite{shin2020autoprompt}." />
        </attvalues>
      </edge>
      <edge source="2304.03087" target="2212.04092" id="10958">
        <attvalues>
          <attvalue for="5" value=" The CoT approach involves utilizing templates as prompts to harness the model's capabilities more effectively \cite{dua2022successive, zhou2022least}." />
        </attvalues>
      </edge>
      <edge source="2010.05967" target="1712.04313" id="10959">
        <attvalues>
          <attvalue for="5" value=" This is a reopening of the 2017 ``spoken term discovery'' ZeroSpeech Benchmark \cite{dunbar2017} (track 2 in Figure \ref{fig:diagram-challenge})." />
        </attvalues>
      </edge>
      <edge source="2010.05967" target="1910.05453" id="10961">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{DBLP:conf/iclr/BaevskiSA20} perform unsupervised representation learning, and show that, up to a certain point, discrete representations are more useful than continuous ones as the input for training a phone recognizer." />
        </attvalues>
      </edge>
      <edge source="2007.06162" target="1909.09962" id="10963">
        <attvalues>
          <attvalue for="5" value=" Recently, pre-trained language models~(PLM), eg GPT-2~\cite{radford2019language}, have shown great promise in many applications of natural language generation, such as stylized text generation~\cite{syed2019adapting} and dialog system~\cite{DBLP:journals/corr/abs-1901-08149}." />
        </attvalues>
      </edge>
      <edge source="2007.06162" target="1901.08149" id="10964">
        <attvalues>
          <attvalue for="5" value=" Recently, pre-trained language models~(PLM), eg GPT-2~\cite{radford2019language}, have shown great promise in many applications of natural language generation, such as stylized text generation~\cite{syed2019adapting} and dialog system~\cite{DBLP:journals/corr/abs-1901-08149}." />
        </attvalues>
      </edge>
      <edge source="2007.06162" target="1511.01844" id="10965">
        <attvalues>
          <attvalue for="5" value=" \cite{theis2015note} point out that minimizing KL avoids assigning an extremely small probability to any data point but assigns a lot of probability mass to non-data regions, which leads to a gap between $P_{Real}$ and $P_{Model}$." />
        </attvalues>
      </edge>
      <edge source="2305.13648" target="2010.00710" id="10966">
        <attvalues>
          <attvalue for="5" value=" Based on these observations, we propose trainable-$k$NN-MT to alleviate the problems of the the original $k$NN-MT~\cite{Khandelwal2020NearestNM}." />
        </attvalues>
      </edge>
      <edge source="2305.13648" target="1603.06111" id="10967">
        <attvalues>
          <attvalue for="5" value=" Moreover, although $k$NN-MT has the advantage that it does not require additional fine-tuning, our experiments show that $k$NN-MT cannot outperform or even achieve comparable performance to classic fine-tuning~\cite{mou-etal-2016-transferable} when in-domain data is accessible." />
        </attvalues>
      </edge>
      <edge source="2009.14394" target="1802.05365" id="10968">
        <attvalues>
          <attvalue for="5" value=" \cite{Peters:2018} introduce a technique for extracting word representations as a linear combination of layers in the pre-trained model." />
        </attvalues>
      </edge>
      <edge source="2009.14394" target="1706.03762" id="10970">
        <attvalues>
          <attvalue for="5" value=" \cite{RadfordTransformer2018}, followed by \cite{devlin2018bert}, pre-train deep transformers \cite{Vaswani:2017} on massive corpora." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1507.08240" id="10971">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many state-of-art models, eg, connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1412.1602" id="10972">
        <attvalues>
          <attvalue for="5" value=" LAS obtains excellent performance, which uses pyramidal BiLSTM and attention mechanism~\cite{chorowski2014end} architecture." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1506.07503" id="10973">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many state-of-art models, eg, connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1610.03022" id="10974">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many state-of-art models, eg, connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1211.3711" id="10975">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many state-of-art models, eg, connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1303.5778" id="10976">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many state-of-art models, eg, connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1801.00841" id="10977">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many state-of-art models, eg, connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1508.01211" id="10978">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many state-of-art models, eg, connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1706.03762" id="10979">
        <attvalues>
          <attvalue for="5" value=" Accordingly, many state-of-art models, eg, connectionist temporal classification (CTC)~\cite{dahl2011context,mohri2008speech,graves2006connectionist}, attention-based encoder-decoder model (AED)~\cite{miao2015eesen,chorowski2014end,chorowski2015attention,zhang2017very}, recurrent neutral network transducer (RNN-T)~\cite{graves2012sequence,graves2013speech,rao2017exploring}, Listen, Attend and Spell (LAS)~\cite{chan2015listen} and Transformer~\cite{vaswani2017attention} have been proposed based on the E2E models." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1611.01487" id="10980">
        <attvalues>
          <attvalue for="5" value=" To facilitate the use of the LAS model for streaming, HMA~\cite{aharoni2016morphological} is put forward to the direct modeling of monotonic alignment to perform hard attention." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="1901.02860" id="10983">
        <attvalues>
          <attvalue for="5" value=" To attenuate context leaking, Transformer-XL~\cite{dai2019transformer} proposes a chunk-wise method in which all frames are visible to other frames in a chunk and caches the previous chunk to reduce the computational complexity." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="2005.08042" id="10984">
        <attvalues>
          <attvalue for="5" value=" Augment memory transformer~\cite{wu2020streaming,yeh2021streaming} is presented to adopt an augment memory bank to capture long-range history context and reduce computational complexity, which has demonstrated comparable performance." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="2011.07120" id="10985">
        <attvalues>
          <attvalue for="5" value=" Augment memory transformer~\cite{wu2020streaming,yeh2021streaming} is presented to adopt an augment memory bank to capture long-range history context and reduce computational complexity, which has demonstrated comparable performance." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="2109.07327" id="10986">
        <attvalues>
          <attvalue for="5" value="~\cite{cao2021improving} proposes to use self-training to improve the performance of chunk-wise streaming model." />
        </attvalues>
      </edge>
      <edge source="2203.15613" target="2010.10759" id="10987">
        <attvalues>
          <attvalue for="5" value=" Emformer~\cite{shi2021emformer}, which is brought forward to reduce computational complexity and applies parallelized block processing training method." />
        </attvalues>
      </edge>
      <edge source="2110.00157" target="2006.00377" id="10988">
        <attvalues>
          <attvalue for="5" value=" Likewise, reading difficulty can be expressed in various forms such as age level, grade level, or a number from a certain range defined by a book publisher \cite{deutsch-etal-2020-linguistic}." />
        </attvalues>
      </edge>
      <edge source="2110.00157" target="1906.07580" id="10989">
        <attvalues>
          <attvalue for="5" value=" Through the years, this process has evolved from the use of handcrafted arithmetic formulas such as the Flesch-Kincaid Reading Ease \cite{kincaid1975derivation} and Dale-Chall \cite{dale1948formula} readability formulas to the use of supervised machine learning algorithms such as Logistic Regression and Support Vector Machines \cite{chatzipanagiotidis-etal-2021-broad,weiss-meurers-2018-modeling,xia-etal-2016-text,reynolds-2016-insights,vajjala-meurers-2012-improving}." />
        </attvalues>
      </edge>
      <edge source="2110.00157" target="2105.00973" id="10990">
        <attvalues>
          <attvalue for="5" value=" Despite the significant growth in research history, several problems still pose as open challenges for the task such as the (a) availability of corpora and tools for linguistic feature extraction, (b) extrinsic evaluation, and (c) interpretation of linguistic predictors used which is arguably the most important of all \cite{vajjalatrends}." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2303.08774" id="10991">
        <attvalues>
          <attvalue for="5" value="5~\cite{openai_chatgpt}, GPT-4~\cite{openai2024gpt4}, and Claude~\cite{claude}, demonstrate remarkable success across various languages, particularly in English." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2309.10305" id="10992">
        <attvalues>
          <attvalue for="5" value=" On the other hand, models such as Baichuan~\cite{baichuan2023baichuan2}, ChatGLM~\cite{zeng2022glm}, and Qwen~\cite{qwen} are mainly used for Chinese." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2210.02414" id="10993">
        <attvalues>
          <attvalue for="5" value=" On the other hand, models such as Baichuan~\cite{baichuan2023baichuan2}, ChatGLM~\cite{zeng2022glm}, and Qwen~\cite{qwen} are mainly used for Chinese." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2307.09288" id="10995">
        <attvalues>
          <attvalue for="5" value=" Remarkably, the Llama 2 model~\cite{Touvron2023Llama2O} series demonstrates that, when trained with massive data, smaller models can surpass their larger counterparts in performance." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2104.09864" id="10996">
        <attvalues>
          <attvalue for="5" value=" We incorporate advanced techniques such as ROPE~\cite{su2024roformer}, Group-Query-Attention~\cite{ainslie2023gqa}, and FlashAttention-2~\cite{dao2022flashattention} to expedite our training process." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2305.13245" id="10997">
        <attvalues>
          <attvalue for="5" value=" We incorporate advanced techniques such as ROPE~\cite{su2024roformer}, Group-Query-Attention~\cite{ainslie2023gqa}, and FlashAttention-2~\cite{dao2022flashattention} to expedite our training process." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2205.14135" id="10998">
        <attvalues>
          <attvalue for="5" value=" We incorporate advanced techniques such as ROPE~\cite{su2024roformer}, Group-Query-Attention~\cite{ainslie2023gqa}, and FlashAttention-2~\cite{dao2022flashattention} to expedite our training process." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2310.15777" id="10999">
        <attvalues>
          <attvalue for="5" value="3B\cite{yang2023mindllm} and TinyLLaMA-1." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2401.02385" id="11000">
        <attvalues>
          <attvalue for="5" value="1B\cite{zhang2024tinyllama} on general benchmarks, including MMLU\cite{hendrycks2020measuring}, C-Eval\cite{huang2024c}, and CMMLU\cite{li2023cmmlu}." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2009.03300" id="11001">
        <attvalues>
          <attvalue for="5" value="1B\cite{zhang2024tinyllama} on general benchmarks, including MMLU\cite{hendrycks2020measuring}, C-Eval\cite{huang2024c}, and CMMLU\cite{li2023cmmlu}." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2305.08322" id="11002">
        <attvalues>
          <attvalue for="5" value="1B\cite{zhang2024tinyllama} on general benchmarks, including MMLU\cite{hendrycks2020measuring}, C-Eval\cite{huang2024c}, and CMMLU\cite{li2023cmmlu}." />
        </attvalues>
      </edge>
      <edge source="2406.09900" target="2306.09212" id="11003">
        <attvalues>
          <attvalue for="5" value="1B\cite{zhang2024tinyllama} on general benchmarks, including MMLU\cite{hendrycks2020measuring}, C-Eval\cite{huang2024c}, and CMMLU\cite{li2023cmmlu}." />
        </attvalues>
      </edge>
      <edge source="2304.11985" target="1706.03762" id="11008">
        <attvalues>
          <attvalue for="5" value=" The Transformer architecture \cite{vaswani2017attention}, as an epitome of the attention-based encoder-decoder framework \cite{chan2016listen, chorowski2015attention}, has become one of the dominant end-to-end (E2E) ASR techniques." />
        </attvalues>
      </edge>
      <edge source="2304.11985" target="1211.3711" id="11009">
        <attvalues>
          <attvalue for="5" value=" Compared with other modelling strategies such as connectionist temporal classification (CTC) \cite{graves2006connectionist, graves2014towards} and recurrent neural network (RNN) transducer \cite{graves2012sequence}, Transformer sustains severe latency issues at inference time, since it requires access to the full speech utterance for decoding." />
        </attvalues>
      </edge>
      <edge source="2304.11985" target="2001.02674" id="11011">
        <attvalues>
          <attvalue for="5" value=" Besides, the CTC posterior spikes corresponding to the non-blank labels are also regarded as effective attention boundaries used in MLT, given that they sparsely appear around the endpoint of acoustic events \cite{moritz2020streaming}." />
        </attvalues>
      </edge>
      <edge source="2304.11985" target="2011.13834" id="11012">
        <attvalues>
          <attvalue for="5" value=" To facilitate streaming ASR, a number of online attention mechanisms have been proposed in literature, including monotonic chunkwise attention (MoChA) \cite{chiu2017monotonic, tsunoo2019towards, inaguma2020enhancing}, CTC-triggered attention \cite{moritz2020streaming}, decoder-end adaptive computation steps (DACS) based algorithms \cite{li2021transformer, li2021head}, as well as the recent cumulative attention (CA) \cite{li2022transformer}." />
        </attvalues>
      </edge>
      <edge source="2304.11985" target="2203.05736" id="11013">
        <attvalues>
          <attvalue for="5" value=" We applied SR-MLT on two streaming Transformer ASR systems that are based on MoChA \cite{chiu2017monotonic} and CA \cite{li2022transformer} algorithms respectively, in order to show it compatible to various online attention mechanisms." />
        </attvalues>
      </edge>
      <edge source="2304.11985" target="1712.01769" id="11015">
        <attvalues>
          <attvalue for="5" value=" MLT introduces the &quot;ground-truth&quot; token boundaries into the attention calculation, which can be obtained from the forced alignments produced by conventional hidden Markov model (HMM) based ASR models \cite{chiu2018state}." />
        </attvalues>
      </edge>
      <edge source="2211.07283" target="2105.12806" id="11019">
        <attvalues>
          <attvalue for="5" value=" Although classical parametrized models only require $n$ parameters to fit $n$ data points,~\cite{bubeck} showed that large over-parameterized models with at least $nd$ parameters are in fact necessary for smooth data interpolation (where $d$ is the data dimensionality)." />
        </attvalues>
      </edge>
      <edge source="2211.07283" target="2007.05558" id="11020">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{limits} found that the compute power needed to reduce error rates by a factor of $x$ was at least $x^{10}$ across a range of image classification and natural language processing tasks." />
        </attvalues>
      </edge>
      <edge source="2211.07283" target="2106.06955" id="11022">
        <attvalues>
          <attvalue for="5" value=" However, \cite{imp-fail} suggests that the winning tickets cannot be found without prior dense training and therefore requires much more time to train than the original unpruned model." />
        </attvalues>
      </edge>
      <edge source="2211.07283" target="1711.02017" id="11023">
        <attvalues>
          <attvalue for="5" value=" The additional time taken holds for other sparsification schemes like dynamic pruning, which regrows weights during training according to gradients~\cite{nest} or momentum~\cite{sparse-momentum} in each backward pass." />
        </attvalues>
      </edge>
      <edge source="2211.07283" target="1907.04840" id="11024">
        <attvalues>
          <attvalue for="5" value=" The additional time taken holds for other sparsification schemes like dynamic pruning, which regrows weights during training according to gradients~\cite{nest} or momentum~\cite{sparse-momentum} in each backward pass." />
        </attvalues>
      </edge>
      <edge source="2211.07283" target="2006.04558" id="11025">
        <attvalues>
          <attvalue for="5" value=" We apply our method to FastSpeech2~\cite{fastspeech2} and evaluate its effectiveness via naturalness, intelligibility, prosody, and training time, comparing it to both dense and constant-sparsity models." />
        </attvalues>
      </edge>
      <edge source="2005.00496" target="1909.02209" id="11026">
        <attvalues>
          <attvalue for="5" value=" Such representations can come in handy in tasks involving text understanding, such as coreference resolution~\cite{ponzetto-strube-2006-exploiting} and reading comprehension~\cite[eg,][]{berant-etal-2014-modeling,zhang2019semantics}." />
        </attvalues>
      </edge>
      <edge source="2005.00496" target="1804.08199" id="11027">
        <attvalues>
          <attvalue for="5" value=" For example, syntax can drive feature design~\cite[eg,][ and others]{punyakanok2005necessity,toutanova-etal-2005-joint,kshirsagar-etal-2015-frame,johansson-nugues-2008-dependency}, and can also be embedded into neural network architectures~\cite{strubell2018linguistically}." />
        </attvalues>
      </edge>
      <edge source="2005.00496" target="1810.02245" id="11028">
        <attvalues>
          <attvalue for="5" value=" The SRL literature is witness to a rich array of techniques for structured inference, including integer linear programs~\cite[eg,][]{punyakanok2005necessity,punyakanok2008importance}, bespoke inference algorithms~\cite[eg,][]{tackstrom2015efficient}, A* decoding~\cite[eg,][]{he-etal-2017-deep}, greedy heuristics~\cite[eg,][]{ouchi-etal-2018-span}, or simple Viterbi decoding to ensure that token tags are BIO-consistent." />
        </attvalues>
      </edge>
      <edge source="2005.00496" target="1907.11692" id="11029">
        <attvalues>
          <attvalue for="5" value=" We instantiate our framework on top of a strong baseline system based on the RoBERTa~\cite{liu2019roberta} encoder, which by itself performs on par with previous best SRL models that are not ensembled." />
        </attvalues>
      </edge>
      <edge source="2004.04418" target="1706.03762" id="11030">
        <attvalues>
          <attvalue for="5" value=" Since both dropout and ReLU have been core components of the transformer since its inception \cite{vaswani2017attention}, this loss of information is likely to be taking place and should be taken into account when selecting the number of transformer layers." />
        </attvalues>
      </edge>
      <edge source="2004.04418" target="1804.00247" id="11033">
        <attvalues>
          <attvalue for="5" value=" However, at the same time, transformer models remain difficult to optimize and require careful tuning of hyper-parameters to be useful in this setting \cite{popel2018training, nguyen2019transformers}." />
        </attvalues>
      </edge>
      <edge source="2004.04418" target="1910.06717" id="11034">
        <attvalues>
          <attvalue for="5" value=" In this work, we find that the current trend in the field to use very large models is detrimental for low-resource languages, since it makes training more difficult and hurts overall performance, confirming the observations by \cite{murray2019auto,Fan2019ReducingTD}." />
        </attvalues>
      </edge>
      <edge source="2004.04418" target="1909.11556" id="11035">
        <attvalues>
          <attvalue for="5" value=" In this work, we find that the current trend in the field to use very large models is detrimental for low-resource languages, since it makes training more difficult and hurts overall performance, confirming the observations by \cite{murray2019auto,Fan2019ReducingTD}." />
        </attvalues>
      </edge>
      <edge source="2004.04418" target="1611.01232" id="11036">
        <attvalues>
          <attvalue for="5" value=" Our intuition concerning the relationship between performance and depth stems from prior work on signal propagation theory in noise-regularised neural networks \cite{schoenholz2016deep, pretorius2018critical}." />
        </attvalues>
      </edge>
      <edge source="2004.04418" target="1811.00293" id="11037">
        <attvalues>
          <attvalue for="5" value=" Although the architecture of a transformer is far more involved than those analysed by \cite{pretorius2018critical}, the fundamental building blocks remain the same." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="1703.09902" id="11038">
        <attvalues>
          <attvalue for="5" value=" NLG has a range of applications~\cite{gatt2018survey,santhanam2019survey}." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="1812.08951" id="11039">
        <attvalues>
          <attvalue for="5" value=" From the perspective of neural text generation, various NLG applications have task-agnostic commonalities: (\romannumeral1) Unlike traditional systems, neural networks capture features without ad-hoc feature engineering \cite{belinkov2019analysis}; (\romannumeral2) Various neural generative frameworks mostly use similar encoder-decoder architectures, so they have common modules and training (or inference) strategies; (\romannumeral3) Evaluation metrics also have good generalization in NLG." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="2105.04387" id="11042">
        <attvalues>
          <attvalue for="5" value=" Existing survey papers usually summarize one of the NLG applications such as Story Generation \cite{hou2019survey,alhussain2021automatic,tang-etal-2022-ngep,huang-etal-2022-improving}, Text summarisation \cite{suleiman2020deep,el2021automatic}, Dialogue~\cite{ni2021recent,tang2022terminology,zhanga2023cadge}, Machine Translation~\cite{yang2020survey,dabre2020survey}, etc~\cite{loakman2023phonetically}." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="2210.15551" id="11043">
        <attvalues>
          <attvalue for="5" value=" Existing survey papers usually summarize one of the NLG applications such as Story Generation \cite{hou2019survey,alhussain2021automatic,tang-etal-2022-ngep,huang-etal-2022-improving}, Text summarisation \cite{suleiman2020deep,el2021automatic}, Dialogue~\cite{ni2021recent,tang2022terminology,zhanga2023cadge}, Machine Translation~\cite{yang2020survey,dabre2020survey}, etc~\cite{loakman2023phonetically}." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="2305.06294" id="11044">
        <attvalues>
          <attvalue for="5" value=" Existing survey papers usually summarize one of the NLG applications such as Story Generation \cite{hou2019survey,alhussain2021automatic,tang-etal-2022-ngep,huang-etal-2022-improving}, Text summarisation \cite{suleiman2020deep,el2021automatic}, Dialogue~\cite{ni2021recent,tang2022terminology,zhanga2023cadge}, Machine Translation~\cite{yang2020survey,dabre2020survey}, etc~\cite{loakman2023phonetically}." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="2002.07526" id="11045">
        <attvalues>
          <attvalue for="5" value=" Existing survey papers usually summarize one of the NLG applications such as Story Generation \cite{hou2019survey,alhussain2021automatic,tang-etal-2022-ngep,huang-etal-2022-improving}, Text summarisation \cite{suleiman2020deep,el2021automatic}, Dialogue~\cite{ni2021recent,tang2022terminology,zhanga2023cadge}, Machine Translation~\cite{yang2020survey,dabre2020survey}, etc~\cite{loakman2023phonetically}." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="1803.07133" id="11046">
        <attvalues>
          <attvalue for="5" value=" Only a few studies~\cite{lu2018neural,chandu2020positioning,jin2020recent,dong2021survey} discuss the development of the whole NLG area." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="2010.07279" id="11047">
        <attvalues>
          <attvalue for="5" value=" Therefore, neural text generation has similar challenges \cite{chandu2020positioning,thomson2021generation} and solutions to analyze." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="2112.11739" id="11048">
        <attvalues>
          <attvalue for="5" value=" Only a few studies~\cite{lu2018neural,chandu2020positioning,jin2020recent,dong2021survey} discuss the development of the whole NLG area." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="2002.12327" id="11049">
        <attvalues>
          <attvalue for="5" value=" Task-specific surveys are beneficial, but the survey of the whole NLG area could give broader ideas as inspiration for various applications using generative techniques, as the surveys like \cite{belinkov2019analysis,rogers2020primer} did." />
        </attvalues>
      </edge>
      <edge source="2203.03047" target="2108.05644" id="11050">
        <attvalues>
          <attvalue for="5" value=" Therefore, neural text generation has similar challenges \cite{chandu2020positioning,thomson2021generation} and solutions to analyze." />
        </attvalues>
      </edge>
      <edge source="2203.04045" target="2006.03533" id="11057">
        <attvalues>
          <attvalue for="5" value=" The task extends the DSTC-9 track 1~\cite{kim2020domain} from written conversations to spoken conversations, where cross-domain dialogue agents are built to answer questions that cannot be solved with only domain APIs." />
        </attvalues>
      </edge>
      <edge source="2203.04045" target="1907.11692" id="11058">
        <attvalues>
          <attvalue for="5" value=" Second, we use pre-trained language models (eg RoBERTa~\cite{roberta}, ELECTRA~\cite{clark2020electra}, and UniLM~\cite{unilm}), and design different ensemble algorithms for each sub-task." />
        </attvalues>
      </edge>
      <edge source="2203.04045" target="2003.10555" id="11059">
        <attvalues>
          <attvalue for="5" value=" Second, we use pre-trained language models (eg RoBERTa~\cite{roberta}, ELECTRA~\cite{clark2020electra}, and UniLM~\cite{unilm}), and design different ensemble algorithms for each sub-task." />
        </attvalues>
      </edge>
      <edge source="2203.04045" target="1905.03197" id="11060">
        <attvalues>
          <attvalue for="5" value=" Second, we use pre-trained language models (eg RoBERTa~\cite{roberta}, ELECTRA~\cite{clark2020electra}, and UniLM~\cite{unilm}), and design different ensemble algorithms for each sub-task." />
        </attvalues>
      </edge>
      <edge source="2307.08621" target="1706.03762" id="11075">
        <attvalues>
          <attvalue for="5" value=" Transformer~\cite{transformer} has become the de facto architecture for large language models~\cite{gpt3}, which was initially proposed to overcome the sequential training issue of recurrent models~\cite{lstm}." />
        </attvalues>
      </edge>
      <edge source="2307.08621" target="2005.14165" id="11076">
        <attvalues>
          <attvalue for="5" value=" Transformer~\cite{transformer} has become the de facto architecture for large language models~\cite{gpt3}, which was initially proposed to overcome the sequential training issue of recurrent models~\cite{lstm}." />
        </attvalues>
      </edge>
      <edge source="2307.08621" target="1911.02150" id="11077">
        <attvalues>
          <attvalue for="5" value=" However, training parallelism of Transformers is at the cost of inefficient inference, because of the $O(N)$ complexity per step and memory-bound key-value cache~\cite{multiquery}, which renders Transformers unfriendly to deployment." />
        </attvalues>
      </edge>
      <edge source="2307.08621" target="2006.16236" id="11078">
        <attvalues>
          <attvalue for="5" value=" First, linearized attention~\cite{linear-transformer} approximates standard attention scores $\exp(\vq \cdot \vk)$ with kernels $\phi(\vq) \cdot \phi(\vk)$, so that autoregressive inference can be rewritten in a recurrent form." />
        </attvalues>
      </edge>
      <edge source="2307.08621" target="2305.13048" id="11079">
        <attvalues>
          <attvalue for="5" value=" As a remedy, element-wise operators~\cite{rwkv} are used for acceleration, however, representation capacity and performance are harmed." />
        </attvalues>
      </edge>
      <edge source="2307.08621" target="2212.14052" id="11081">
        <attvalues>
          <attvalue for="5" value=" The third line of research explores replacing attention with other mechanisms, such as S4~\cite{s4}, and its variants~\cite{h3,hyena}." />
        </attvalues>
      </edge>
      <edge source="2307.08621" target="2302.10866" id="11082">
        <attvalues>
          <attvalue for="5" value=" The third line of research explores replacing attention with other mechanisms, such as S4~\cite{s4}, and its variants~\cite{h3,hyena}." />
        </attvalues>
      </edge>
      <edge source="2307.08621" target="2205.14135" id="11083">
        <attvalues>
          <attvalue for="5" value=" During training, \our also achieves 25-50\% memory saving and 7$\times$ acceleration than standard Transformer and an advantage towards highly-optimized FlashAttention~\cite{flashattention}." />
        </attvalues>
      </edge>
      <edge source="2404.06411" target="2308.11432" id="11095">
        <attvalues>
          <attvalue for="5" value=" Generative Agents~\cite{kiela2023saturation} are software systems that leverage foundation models like Large Language Models (LLMs) to perform complex tasks, take decisions, devise multi-steps plans and use tools (API calls, coding, etc) to build solutions in heterogeneous contexts~\cite{wang2023survey,weng2023prompt}." />
        </attvalues>
      </edge>
      <edge source="2404.06411" target="2308.03688" id="11096">
        <attvalues>
          <attvalue for="5" value=" For example, some benchmarks focus on specific capabilities and provide gaming environments, which we refer to as ``closed-box'' -- eg with a finite set of actions~\cite{liu2023agentbench, patil2023gorilla,chalamalasetti2023clembench} -- whereas other benchmarks provide open-ended tasks and access to general tools, like web browsing~\cite{zhuang2023toolqa, zheng2023judging,mialon2023gaia}." />
        </attvalues>
      </edge>
      <edge source="2404.06411" target="2306.13304" id="11098">
        <attvalues>
          <attvalue for="5" value=" For example, some benchmarks focus on specific capabilities and provide gaming environments, which we refer to as ``closed-box'' -- eg with a finite set of actions~\cite{liu2023agentbench, patil2023gorilla,chalamalasetti2023clembench} -- whereas other benchmarks provide open-ended tasks and access to general tools, like web browsing~\cite{zhuang2023toolqa, zheng2023judging,mialon2023gaia}." />
        </attvalues>
      </edge>
      <edge source="2404.06411" target="2311.12983" id="11099">
        <attvalues>
          <attvalue for="5" value=" For example, some benchmarks focus on specific capabilities and provide gaming environments, which we refer to as ``closed-box'' -- eg with a finite set of actions~\cite{liu2023agentbench, patil2023gorilla,chalamalasetti2023clembench} -- whereas other benchmarks provide open-ended tasks and access to general tools, like web browsing~\cite{zhuang2023toolqa, zheng2023judging,mialon2023gaia}." />
        </attvalues>
      </edge>
      <edge source="2301.04761" target="1907.11692" id="11102">
        <attvalues>
          <attvalue for="5" value=" In this work, we explore an orthogonal approach to efficiency: can we make masked language models efficient by reducing the length of the input sequence that each layer needs to process? In particular, pretraining by masked language modeling only involves prediction of masked tokens (typically, only 15\% of the input tokens; \cite{devlins2019bert,Liu2019RoBERTaAR})." />
        </attvalues>
      </edge>
      <edge source="2301.04761" target="2006.03654" id="11103">
        <attvalues>
          <attvalue for="5" value=" Pretrained masked language models, such as BERT \cite{devlins2019bert}, RoBERTa \cite{Liu2019RoBERTaAR}, and DeBERTa \cite{deberta}, have pushed the state-of-the-art on a wide range of downstream tasks in natural language processing." />
        </attvalues>
      </edge>
      <edge source="2301.04761" target="1706.03762" id="11104">
        <attvalues>
          <attvalue for="5" value=" Since the former sublayer implies quadratic time complexity in the input sequence length \cite{Vaswani2017AttentionIA}, many have proposed methods to make the self-attention computation more efficient \cite[inter alia]{katharopoulos-et-al-2020,performer,wang2020linformer,RFA,peng-etal-2022-abc}." />
        </attvalues>
      </edge>
      <edge source="2301.04761" target="2006.16236" id="11105">
        <attvalues>
          <attvalue for="5" value=" Since the former sublayer implies quadratic time complexity in the input sequence length \cite{Vaswani2017AttentionIA}, many have proposed methods to make the self-attention computation more efficient \cite[inter alia]{katharopoulos-et-al-2020,performer,wang2020linformer,RFA,peng-etal-2022-abc}." />
        </attvalues>
      </edge>
      <edge source="2301.04761" target="2009.14794" id="11106">
        <attvalues>
          <attvalue for="5" value=" Since the former sublayer implies quadratic time complexity in the input sequence length \cite{Vaswani2017AttentionIA}, many have proposed methods to make the self-attention computation more efficient \cite[inter alia]{katharopoulos-et-al-2020,performer,wang2020linformer,RFA,peng-etal-2022-abc}." />
        </attvalues>
      </edge>
      <edge source="2301.04761" target="2006.04768" id="11107">
        <attvalues>
          <attvalue for="5" value=" Since the former sublayer implies quadratic time complexity in the input sequence length \cite{Vaswani2017AttentionIA}, many have proposed methods to make the self-attention computation more efficient \cite[inter alia]{katharopoulos-et-al-2020,performer,wang2020linformer,RFA,peng-etal-2022-abc}." />
        </attvalues>
      </edge>
      <edge source="2301.04761" target="1905.00537" id="11111">
        <attvalues>
          <attvalue for="5" value=" In addition to pretraining, many downstream applications only use a single vector representation (eg, only the [CLS] token) for prediction purposes, which is much smaller than the number of input tokens (eg, sequence classification tasks as in GLUE/SuperGLUE; \cite{wang-etal-2018-glue,superglue})." />
        </attvalues>
      </edge>
      <edge source="2301.04761" target="1409.0473" id="11113">
        <attvalues>
          <attvalue for="5" value=" The second approach reduces the input length to the attention sublayers: queries are only computed for masked tokens in the attention mechanism \cite{Bahdanau2014NeuralMT}, while the keys and values are not recomputed for non-masked tokens, which leads to a greater than $2\times$ speedup in pretraining." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="1901.07291" id="11115">
        <attvalues>
          <attvalue for="5" value=" In recent years, advances in multilingual models such as mBERT \cite{devlin2019bert}, XLM \cite{conneau2019cross}, XLM-R \cite{conneauetal2020unsupervised}, etc, after being fine-tuned with annotated data, have enabled significant improvements in many cross-lingual tasks." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="1911.02116" id="11116">
        <attvalues>
          <attvalue for="5" value=" In recent years, advances in multilingual models such as mBERT \cite{devlin2019bert}, XLM \cite{conneau2019cross}, XLM-R \cite{conneauetal2020unsupervised}, etc, after being fine-tuned with annotated data, have enabled significant improvements in many cross-lingual tasks." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="1812.10464" id="11118">
        <attvalues>
          <attvalue for="5" value=" Some works aligned word embeddings between high- and low-resource languages through additional parallel sentence pairs \cite{artetxeschwenk2019massively,WeiW0XYL21,chi-etal-2021-infoxlm,pan-etal-2021-multilingual} or bilingual dictionaries \cite{Cao2020Multilingual,ijcai2020-533,Liu_Winata_Lin_Xu_Fung_2020}, so that high-resource fine-tuned models can be transferred to low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="2007.15960" id="11119">
        <attvalues>
          <attvalue for="5" value=" Some works aligned word embeddings between high- and low-resource languages through additional parallel sentence pairs \cite{artetxeschwenk2019massively,WeiW0XYL21,chi-etal-2021-infoxlm,pan-etal-2021-multilingual} or bilingual dictionaries \cite{Cao2020Multilingual,ijcai2020-533,Liu_Winata_Lin_Xu_Fung_2020}, so that high-resource fine-tuned models can be transferred to low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="2007.07834" id="11120">
        <attvalues>
          <attvalue for="5" value=" Some works aligned word embeddings between high- and low-resource languages through additional parallel sentence pairs \cite{artetxeschwenk2019massively,WeiW0XYL21,chi-etal-2021-infoxlm,pan-etal-2021-multilingual} or bilingual dictionaries \cite{Cao2020Multilingual,ijcai2020-533,Liu_Winata_Lin_Xu_Fung_2020}, so that high-resource fine-tuned models can be transferred to low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="2010.12547" id="11121">
        <attvalues>
          <attvalue for="5" value=" Some works aligned word embeddings between high- and low-resource languages through additional parallel sentence pairs \cite{artetxeschwenk2019massively,WeiW0XYL21,chi-etal-2021-infoxlm,pan-etal-2021-multilingual} or bilingual dictionaries \cite{Cao2020Multilingual,ijcai2020-533,Liu_Winata_Lin_Xu_Fung_2020}, so that high-resource fine-tuned models can be transferred to low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="2006.06402" id="11123">
        <attvalues>
          <attvalue for="5" value=" Some works aligned word embeddings between high- and low-resource languages through additional parallel sentence pairs \cite{artetxeschwenk2019massively,WeiW0XYL21,chi-etal-2021-infoxlm,pan-etal-2021-multilingual} or bilingual dictionaries \cite{Cao2020Multilingual,ijcai2020-533,Liu_Winata_Lin_Xu_Fung_2020}, so that high-resource fine-tuned models can be transferred to low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="1911.09273" id="11124">
        <attvalues>
          <attvalue for="5" value=" Some works aligned word embeddings between high- and low-resource languages through additional parallel sentence pairs \cite{artetxeschwenk2019massively,WeiW0XYL21,chi-etal-2021-infoxlm,pan-etal-2021-multilingual} or bilingual dictionaries \cite{Cao2020Multilingual,ijcai2020-533,Liu_Winata_Lin_Xu_Fung_2020}, so that high-resource fine-tuned models can be transferred to low-resource languages." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="1904.09077" id="11125">
        <attvalues>
          <attvalue for="5" value=" To disengage from the dependence on parallel corpora or bilingual dictionaries \cite{wu-dredze-2019-beto,pmlr-v119-hu20b}, some studies have found that syntactic features in high-resource languages can improve zero-shot cross-lingual transfer learning \cite{meng-etal-2019-target,subburathinam-etal-2019-cross,ahmad-etal-2021-syntax,Ahmad_Peng_Chang_2021}." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="1909.01482" id="11126">
        <attvalues>
          <attvalue for="5" value=" To disengage from the dependence on parallel corpora or bilingual dictionaries \cite{wu-dredze-2019-beto,pmlr-v119-hu20b}, some studies have found that syntactic features in high-resource languages can improve zero-shot cross-lingual transfer learning \cite{meng-etal-2019-target,subburathinam-etal-2019-cross,ahmad-etal-2021-syntax,Ahmad_Peng_Chang_2021}." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="2106.02134" id="11127">
        <attvalues>
          <attvalue for="5" value=" To disengage from the dependence on parallel corpora or bilingual dictionaries \cite{wu-dredze-2019-beto,pmlr-v119-hu20b}, some studies have found that syntactic features in high-resource languages can improve zero-shot cross-lingual transfer learning \cite{meng-etal-2019-target,subburathinam-etal-2019-cross,ahmad-etal-2021-syntax,Ahmad_Peng_Chang_2021}." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="2010.03009" id="11128">
        <attvalues>
          <attvalue for="5" value=" To disengage from the dependence on parallel corpora or bilingual dictionaries \cite{wu-dredze-2019-beto,pmlr-v119-hu20b}, some studies have found that syntactic features in high-resource languages can improve zero-shot cross-lingual transfer learning \cite{meng-etal-2019-target,subburathinam-etal-2019-cross,ahmad-etal-2021-syntax,Ahmad_Peng_Chang_2021}." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="2110.14782" id="11130">
        <attvalues>
          <attvalue for="5" value=" \cite{libovicky-etal-2020-language} found that the embeddings of different languages are clustered according to their language families, as shown in Figure \ref{fig:intro}a and \ref{fig:intro}b, which demonstrated that different languages are not aligned perfectly in mBERT \cite{Ameet-Mbert}." />
        </attvalues>
      </edge>
      <edge source="2210.09934" target="2104.08645" id="11131">
        <attvalues>
          <attvalue for="5" value=" \cite{huang-etal-2021-improving-zero} tried adversarial training and randomized smoothing with English synonym augmentation to build robust regions for embeddings in the multilingual models, as illustrated in Figure \ref{fig:intro}c." />
        </attvalues>
      </edge>
      <edge source="2009.11032" target="1804.05922" id="11142">
        <attvalues>
          <attvalue for="5" value=" Leveraging accurate CD coreference models seems particularly appealing for applications that merge information across texts, which have been gaining growing attention recently, such as multi-document summarization~\cite{falke-etal-2017-concept} and multi-hop question answering \cite{dhingra-etal-2018-neural, wang-etal-2019-multi-hop}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="1211.3711" id="11144">
        <attvalues>
          <attvalue for="5" value=" This work is motivated by the need for fast, simple, robust, and adjustable confidence estimation for the end-to-end Connectionist Temporal Classification (CTC) \cite{graves_connectionist_2006} and Recurrent Neural Network Transducer (RNN-T) \cite{graves2012transducer} ASR models." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="1610.02136" id="11145">
        <attvalues>
          <attvalue for="5" value=" The probability of the most-likely unit (the maximum probability) is a natural way of estimating confidence \cite{hendrycks2016iclr,park20d_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="1412.1897" id="11147">
        <attvalues>
          <attvalue for="5" value=" The effectiveness of this approach is limited by the so-called prediction overconfidence, when the probability distribution is skewed towards the best hypothesis \cite{nguyen2015}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="2002.07650" id="11148">
        <attvalues>
          <attvalue for="5" value=" To mitigate the overconfidence issue, one can use temperature scaling, dropout, ensemble of ASR models etc \cite{vyas2019icassp,malinin2021uncertainty, Oneata2021}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="2010.11428" id="11149">
        <attvalues>
          <attvalue for="5" value=" An alternative approach is based on dedicated neural confidence models ~\cite{li2020confidence,jeon2020,woodward2020confidence,qiu2021learning,li2021residual,qui2021multi,wang2021word}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="1907.09636" id="11150">
        <attvalues>
          <attvalue for="5" value=" An alternative approach is based on dedicated neural confidence models ~\cite{li2020confidence,jeon2020,woodward2020confidence,qiu2021learning,li2021residual,qui2021multi,wang2021word}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="2103.06716" id="11151">
        <attvalues>
          <attvalue for="5" value=" An alternative approach is based on dedicated neural confidence models ~\cite{li2020confidence,jeon2020,woodward2020confidence,qiu2021learning,li2021residual,qui2021multi,wang2021word}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="2104.12870" id="11153">
        <attvalues>
          <attvalue for="5" value=" An alternative approach is based on dedicated neural confidence models ~\cite{li2020confidence,jeon2020,woodward2020confidence,qiu2021learning,li2021residual,qui2021multi,wang2021word}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="2110.15222" id="11154">
        <attvalues>
          <attvalue for="5" value=" An alternative approach is based on dedicated neural confidence models ~\cite{li2020confidence,jeon2020,woodward2020confidence,qiu2021learning,li2021residual,qui2021multi,wang2021word}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="2101.05525" id="11155">
        <attvalues>
          <attvalue for="5" value=" The former requires significant computational resources \cite{zapotoczny2019}, while the latter requires careful selection of aggregation methods \cite{Oneata2021}." />
        </attvalues>
      </edge>
      <edge source="2212.08703" target="1909.09577" id="11156">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} The implementation of the proposed confidence estimation methods is available in the NeMo toolkit~\cite{kuchaiev2019nemo}." />
        </attvalues>
      </edge>
      <edge source="2401.14003" target="1612.03975" id="11157">
        <attvalues>
          <attvalue for="5" value=" As human-annotated CSKBs~\cite{singh2017conceptnet, sap2019atomic, mostafazadeh2020glucose} are usually incomplete and of a small coverage, reasoning over CSKBs, eg, CSKB reasoning, is a way for expansion." />
        </attvalues>
      </edge>
      <edge source="2401.14003" target="2304.10392" id="11158">
        <attvalues>
          <attvalue for="5" value=" We implement ConstraintChecker and conduct extensive experiments on a CSKB Reasoning benchmark CKBPv2 ~\cite{fang2023ckbpv2} as well as a synthetic discriminative version of ATOMIC$_{20}^{20}$ (in short, SD-ATOMIC$_{20}^{20}$), over two large language models: ChatGPT (gpt-3." />
        </attvalues>
      </edge>
      <edge source="2401.14003" target="1811.10830" id="11159">
        <attvalues>
          <attvalue for="5" value=" Expanding CSKBs via such a reasoning process can lead to better and broader commonsense knowledge as valuable resources to augment AI models in various aspects, such as visual reasoning ~\cite{zellers2019vcr}, text generation~\cite{zhou2021commonsense,ilievski2021story}, or building more capable knowledge models for further downstream applications ~\cite{yu2022cocolm,hwang2021comet,wang-etal-2023-cat}." />
        </attvalues>
      </edge>
      <edge source="2401.14003" target="2305.04808" id="11162">
        <attvalues>
          <attvalue for="5" value=" Expanding CSKBs via such a reasoning process can lead to better and broader commonsense knowledge as valuable resources to augment AI models in various aspects, such as visual reasoning ~\cite{zellers2019vcr}, text generation~\cite{zhou2021commonsense,ilievski2021story}, or building more capable knowledge models for further downstream applications ~\cite{yu2022cocolm,hwang2021comet,wang-etal-2023-cat}." />
        </attvalues>
      </edge>
      <edge source="2401.14003" target="2302.06476" id="11163">
        <attvalues>
          <attvalue for="5" value=" Recently, inspired by the emergence of Large Language Models (LLMs) that can perform well in many commonsense reasoning tasks~\cite{qin2023chatgpt, bian2023chatgpt}, ~\cite{chan2023chatgpt} attempted to use LLMs for a CSKB Reasoning benchmark named CSKB Population (CKBP)~\cite{fang2023ckbpv2}." />
        </attvalues>
      </edge>
      <edge source="2401.14003" target="2304.14827" id="11164">
        <attvalues>
          <attvalue for="5" value=" Recently, inspired by the emergence of Large Language Models (LLMs) that can perform well in many commonsense reasoning tasks~\cite{qin2023chatgpt, bian2023chatgpt}, ~\cite{chan2023chatgpt} attempted to use LLMs for a CSKB Reasoning benchmark named CSKB Population (CKBP)~\cite{fang2023ckbpv2}." />
        </attvalues>
      </edge>
      <edge source="2401.14003" target="2212.10403" id="11167">
        <attvalues>
          <attvalue for="5" value=" Nonetheless, they are task-agnostic and suffer from the inherent shortcoming of LLMs in inducing the rules in CSKBs (which we refer as symbolic reasoning ability), as current deep learning still struggles to deal with symbolic and high-level concepts reasoning tasks ~\cite{bengio2021deeplearning,huang-chang-2023-towards, pan2023logiclm}." />
        </attvalues>
      </edge>
      <edge source="2201.09146" target="1808.07036" id="11168">
        <attvalues>
          <attvalue for="5" value=" Conversational question answering extends traditional Question Answering (QA) by involving a sequence of interconnected questions and answers \cite{Choi2018}." />
        </attvalues>
      </edge>
      <edge source="2201.09146" target="2101.07382" id="11170">
        <attvalues>
          <attvalue for="5" value=" For this step, the rewritten question is used as a query to an external datastore, and thus the performance of the initial rewriting module can affect the conversational passage retrieval \cite{Vakulenko2021a}." />
        </attvalues>
      </edge>
      <edge source="2201.09146" target="2010.04898" id="11171">
        <attvalues>
          <attvalue for="5" value=" The Question Rewriting in Conversational Context (QReCC) dataset \cite{Anantha2021} brings these tasks together, supporting the training and evaluation of neural models for conversational QA." />
        </attvalues>
      </edge>
      <edge source="2406.00048" target="2307.15936" id="11174">
        <attvalues>
          <attvalue for="5" value=" However, there is no consensus on the mechanisms behind language {acquisition} by LLMs~\cite{arora2023theory, douglas2023large}." />
        </attvalues>
      </edge>
      <edge source="2406.00048" target="2001.08361" id="11175">
        <attvalues>
          <attvalue for="5" value=" As a result, empirical phenomena such as the scaling of the test loss with dataset size and number of parameters~\cite{kaplan2020scaling} and the emergence of specific skills at certain scales~\cite{ganguli2022predictability,schaeffer2024emergent} remain unexplained." />
        </attvalues>
      </edge>
      <edge source="2406.00048" target="2202.07785" id="11176">
        <attvalues>
          <attvalue for="5" value=" As a result, empirical phenomena such as the scaling of the test loss with dataset size and number of parameters~\cite{kaplan2020scaling} and the emergence of specific skills at certain scales~\cite{ganguli2022predictability,schaeffer2024emergent} remain unexplained." />
        </attvalues>
      </edge>
      <edge source="2406.00048" target="2304.15004" id="11177">
        <attvalues>
          <attvalue for="5" value=" As a result, empirical phenomena such as the scaling of the test loss with dataset size and number of parameters~\cite{kaplan2020scaling} and the emergence of specific skills at certain scales~\cite{ganguli2022predictability,schaeffer2024emergent} remain unexplained." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2403.08295" id="11178">
        <attvalues>
          <attvalue for="5" value=" For the language backbone, we use the recently released Gemma models \cite{team2024gemma}." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2401.02330" id="11179">
        <attvalues>
          <attvalue for="5" value=" Our work is inspired by the rapid progress in small but capable visual language models (VLMs), such as LLaVA-Phi \cite{zhu2024llava}, which have demonstrated remarkable efficiency and effectiveness in various language understanding tasks." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="1706.03762" id="11180">
        <attvalues>
          <attvalue for="5" value=" Recent advancements in (LLMs) \cite{vaswani2017attention} and multimodal foundation models (MMFMs) \cite{li2023multimodal} have propelled the interest and development of Large Multimodal Models (LMMs)." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2303.08774" id="11182">
        <attvalues>
          <attvalue for="5" value=" Notable models like GPT-4 \cite{achiam2023gpt}, LLaVA \cite{liu2024visual,liu2023improved}, and their derivatives have demonstrated significant performance in vision-language tasks such as Visual Question Answering (VQA) and image captioning \cite{hudson2019gqa}." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2310.03744" id="11184">
        <attvalues>
          <attvalue for="5" value=" \end{enumerate} We follow the LLaVA framework \cite{liu2023improved} with a few design modifications." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="1902.09506" id="11185">
        <attvalues>
          <attvalue for="5" value=" This dataset includes synthetic data generated \cite{liu2024visual}, as well as examples from established vision-language training sets such as GQA \cite{hudson2019gqa} and TextCaps \cite{sidorov2020textcaps}." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2307.09288" id="11187">
        <attvalues>
          <attvalue for="5" value=" This framework combines a pretrained vision encoder (such as CLIP \cite{radford2021learning}) and pretrained language model (such as Llama-2 \cite{touvron2023llama}) into a multimodal model using a MLP connector and a two-stage training procedure." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2003.12462" id="11188">
        <attvalues>
          <attvalue for="5" value=" This dataset includes synthetic data generated \cite{liu2024visual}, as well as examples from established vision-language training sets such as GQA \cite{hudson2019gqa} and TextCaps \cite{sidorov2020textcaps}." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2306.05685" id="11189">
        <attvalues>
          <attvalue for="5" value=" Whereas LLaVA uses the 7 and 13-billion parameter vicuña langauge models \cite{zheng2023judging}, Gemma offers 2 and 7-billion parameter versions." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2304.07193" id="11190">
        <attvalues>
          <attvalue for="5" value=" Correspondingly, we explore the use of the larger 1-billion parameter DINOv2 image encoder \cite{oquab2023dinov2} as the vision tower." />
        </attvalues>
      </edge>
      <edge source="2404.01331" target="2402.07865" id="11191">
        <attvalues>
          <attvalue for="5" value=" Related work on VLMs \cite{karamcheti2024prismatic} finds that skipping the initial pretraining stage improves downstream performance." />
        </attvalues>
      </edge>
      <edge source="2009.08560" target="1706.03872" id="11192">
        <attvalues>
          <attvalue for="5" value=" This type of text simplification is challenging as its natural language generation process potentially involves multiple sub-processes such as co-reference resolution, named-entity recognition, semantic role labelling, etc Split and Rephrase has two main real-world uses: first, to benefit systems whose performance improves with decreasing length of sentences eg entity extraction \cite{zhang-etal-2017-position} and machine translation \cite{koehn-knowles-2017-six} by acting as a pre-processing step; second, to benefit human readers, especially those less proficient with the language when reading complex documents such as terms and agreements, in understanding the meaning more easily and accurately \cite{Inui2003TextSF,Siddharthan2002AnAF}." />
        </attvalues>
      </edge>
      <edge source="2009.08560" target="1707.06971" id="11193">
        <attvalues>
          <attvalue for="5" value=" The task was introduced by \cite{narayan-etal-2017-split}, with the release of the WebSplit corpus." />
        </attvalues>
      </edge>
      <edge source="2009.08560" target="1409.0473" id="11194">
        <attvalues>
          <attvalue for="5" value=" Afterwards, \cite{aharoni-goldberg-2018-split} proposed the state-of-the-art model to date, a sequence-to-sequence model \cite{Bahdanau2015NeuralMT} with a copy mechanism \cite{gu-etal-2016-incorporating,see-etal-2017-get} with the observation that most texts are unchanged during a Split and Rephrase operation." />
        </attvalues>
      </edge>
      <edge source="2009.08560" target="1603.06393" id="11195">
        <attvalues>
          <attvalue for="5" value=" Afterwards, \cite{aharoni-goldberg-2018-split} proposed the state-of-the-art model to date, a sequence-to-sequence model \cite{Bahdanau2015NeuralMT} with a copy mechanism \cite{gu-etal-2016-incorporating,see-etal-2017-get} with the observation that most texts are unchanged during a Split and Rephrase operation." />
        </attvalues>
      </edge>
      <edge source="2009.08560" target="1704.04368" id="11196">
        <attvalues>
          <attvalue for="5" value=" Afterwards, \cite{aharoni-goldberg-2018-split} proposed the state-of-the-art model to date, a sequence-to-sequence model \cite{Bahdanau2015NeuralMT} with a copy mechanism \cite{gu-etal-2016-incorporating,see-etal-2017-get} with the observation that most texts are unchanged during a Split and Rephrase operation." />
        </attvalues>
      </edge>
      <edge source="2009.08560" target="1808.09468" id="11197">
        <attvalues>
          <attvalue for="5" value=" Apart from its series of limitations already reported, such as a small vocabulary, unnatural expressions, etc \cite{botha-etal-2018-learning}, we further show that its complex sentences systematically follow only 3 syntactical patterns marked by lexical cues (Section~\ref{sec:issues})." />
        </attvalues>
      </edge>
      <edge source="2009.08560" target="1810.05995" id="11198">
        <attvalues>
          <attvalue for="5" value=" Also, \cite{sulem-etal-2018-bleu} studied the problems of using BLEU as the evaluation metric for this task, while proposing a manually constructed test set called HSplit." />
        </attvalues>
      </edge>
      <edge source="2310.07328" target="2005.14165" id="11210">
        <attvalues>
          <attvalue for="5" value=" However, the closed source of LLMs (eg, GPT-3 \cite{brown2020language} and PaLM \cite{chowdhery2022palm}) coupled with the requirement for massive computing resources to build the exclusive LLM has deterred researchers from reaching the LLM training stage." />
        </attvalues>
      </edge>
      <edge source="2310.07328" target="2204.02311" id="11211">
        <attvalues>
          <attvalue for="5" value=" However, the closed source of LLMs (eg, GPT-3 \cite{brown2020language} and PaLM \cite{chowdhery2022palm}) coupled with the requirement for massive computing resources to build the exclusive LLM has deterred researchers from reaching the LLM training stage." />
        </attvalues>
      </edge>
      <edge source="2310.07328" target="2201.11903" id="11212">
        <attvalues>
          <attvalue for="5" value=" Subsequently, a series of &quot;API research&quot; based on GPT-3 and ChatGPT are constantly emerging, which stimulate the specific capabilities of frozen LLMs (eg, Chain-of-Thought \cite{wei2023chainofthought,wang2023selfconsistency,kojima2023large}) or guide them to complete specific tasks \cite{yang2022empirical,shen2023hugginggpt}, by calling OpenAI interfaces and carefully designing prompts without model training." />
        </attvalues>
      </edge>
      <edge source="2310.07328" target="2203.11171" id="11213">
        <attvalues>
          <attvalue for="5" value=" Subsequently, a series of &quot;API research&quot; based on GPT-3 and ChatGPT are constantly emerging, which stimulate the specific capabilities of frozen LLMs (eg, Chain-of-Thought \cite{wei2023chainofthought,wang2023selfconsistency,kojima2023large}) or guide them to complete specific tasks \cite{yang2022empirical,shen2023hugginggpt}, by calling OpenAI interfaces and carefully designing prompts without model training." />
        </attvalues>
      </edge>
      <edge source="2310.07328" target="2109.05014" id="11215">
        <attvalues>
          <attvalue for="5" value=" Subsequently, a series of &quot;API research&quot; based on GPT-3 and ChatGPT are constantly emerging, which stimulate the specific capabilities of frozen LLMs (eg, Chain-of-Thought \cite{wei2023chainofthought,wang2023selfconsistency,kojima2023large}) or guide them to complete specific tasks \cite{yang2022empirical,shen2023hugginggpt}, by calling OpenAI interfaces and carefully designing prompts without model training." />
        </attvalues>
      </edge>
      <edge source="2310.07328" target="2109.01652" id="11219">
        <attvalues>
          <attvalue for="5" value=" This open project verifies the important role of instruction-tuning \cite{wei2022finetuned, chung2022scaling} open LLMs in replicating the ChatGPT process." />
        </attvalues>
      </edge>
      <edge source="2310.07328" target="2211.05100" id="11220">
        <attvalues>
          <attvalue for="5" value="g, Bloom \cite{workshop2023bloom}, GPT-J \cite{gpt-j}) are shown to have significant improvements in instruction-following performance with instruction-tuning." />
        </attvalues>
      </edge>
      <edge source="2310.07328" target="2304.12986" id="11221">
        <attvalues>
          <attvalue for="5" value=" Besides, we consider the AGI (instruction-following) capability and professional knowledge reserve (human exams) of models, and correspondingly select two benchmarks Belle-eval \cite{BELLE} and MMCU \cite{zeng2023measuring} for comprehensively evaluation." />
        </attvalues>
      </edge>
      <edge source="2205.00415" target="2107.12708" id="11222">
        <attvalues>
          <attvalue for="5" value=" Benchmarks have been proven pivotal for driving progress in Natural Language Understanding (NLU) in recent years \cite{rogers2021qa, bach2022promptsource, wang2022benchmarking}." />
        </attvalues>
      </edge>
      <edge source="2205.00415" target="1908.05803" id="11223">
        <attvalues>
          <attvalue for="5" value=" For instance, $\sim36\%$ of the instruction examples for the \quoref dataset \cite{dasigi2019quoref} start with ``What is the name'', and this same pattern can be observed in $\sim59\%$ of the collected instances." />
        </attvalues>
      </edge>
      <edge source="2205.00415" target="1803.02324" id="11225">
        <attvalues>
          <attvalue for="5" value=" Such biases are often attributed to annotator-related biases, such as writing style and background knowledge \cite{gururangan-etal-2018-annotation, geva-etal-2019-modeling} (see more discussion on related work in \S\ref{sec:related_work})." />
        </attvalues>
      </edge>
      <edge source="2205.00415" target="2010.11982" id="11226">
        <attvalues>
          <attvalue for="5" value=" From a broader perspective, our findings also have implications on the recent learning-by-instructions paradigm \cite{efrat2020turking, mishra2021cross}, where crowdsourcing instructions are used in model training." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="1807.10543" id="11227">
        <attvalues>
          <attvalue for="5" value=" With the rapid advancements of artificial intelligence (AI) and natural language processing (NLP) approaches, there has been increasing interest in developing more AI-powered grading and feedback systems for educational purposes \cite{lu2021integrating,jia2022automated, baral2021improving,suzen2020automatic, zhang2022automatic, qiu2022toward}." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="2307.09288" id="11228">
        <attvalues>
          <attvalue for="5" value=" While access to top-tier LLMs including GPT-3 \cite{brown2020language}, GPT-4 \cite{openai2023gpt4}, Chinchilla \cite{hoffmann2022training}, and both iterations of PaLM \cite{chowdhery2022palm, anil2023palm} remains restricted, the release of models like Falcon \cite{almazrouei2023falcon}, LLaMA \cite{touvron2023llama}, and LLaMA-2 \cite{touvron2023llama2} under open-access terms has democratized access to powerful LLMs." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="2005.14165" id="11229">
        <attvalues>
          <attvalue for="5" value=" While access to top-tier LLMs including GPT-3 \cite{brown2020language}, GPT-4 \cite{openai2023gpt4}, Chinchilla \cite{hoffmann2022training}, and both iterations of PaLM \cite{chowdhery2022palm, anil2023palm} remains restricted, the release of models like Falcon \cite{almazrouei2023falcon}, LLaMA \cite{touvron2023llama}, and LLaMA-2 \cite{touvron2023llama2} under open-access terms has democratized access to powerful LLMs." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="2303.08774" id="11230">
        <attvalues>
          <attvalue for="5" value=" While access to top-tier LLMs including GPT-3 \cite{brown2020language}, GPT-4 \cite{openai2023gpt4}, Chinchilla \cite{hoffmann2022training}, and both iterations of PaLM \cite{chowdhery2022palm, anil2023palm} remains restricted, the release of models like Falcon \cite{almazrouei2023falcon}, LLaMA \cite{touvron2023llama}, and LLaMA-2 \cite{touvron2023llama2} under open-access terms has democratized access to powerful LLMs." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="2203.15556" id="11231">
        <attvalues>
          <attvalue for="5" value=" While access to top-tier LLMs including GPT-3 \cite{brown2020language}, GPT-4 \cite{openai2023gpt4}, Chinchilla \cite{hoffmann2022training}, and both iterations of PaLM \cite{chowdhery2022palm, anil2023palm} remains restricted, the release of models like Falcon \cite{almazrouei2023falcon}, LLaMA \cite{touvron2023llama}, and LLaMA-2 \cite{touvron2023llama2} under open-access terms has democratized access to powerful LLMs." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="2204.02311" id="11232">
        <attvalues>
          <attvalue for="5" value=" While access to top-tier LLMs including GPT-3 \cite{brown2020language}, GPT-4 \cite{openai2023gpt4}, Chinchilla \cite{hoffmann2022training}, and both iterations of PaLM \cite{chowdhery2022palm, anil2023palm} remains restricted, the release of models like Falcon \cite{almazrouei2023falcon}, LLaMA \cite{touvron2023llama}, and LLaMA-2 \cite{touvron2023llama2} under open-access terms has democratized access to powerful LLMs." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="2109.01652" id="11234">
        <attvalues>
          <attvalue for="5" value=" Fine-tuning LLMs involve adapting the expansive knowledge of the pretrained LLMs for a target task, leading to the successful impacts of LLMs across many fields and applications \cite{wei2021finetuned}." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="2211.10438" id="11235">
        <attvalues>
          <attvalue for="5" value=" To mitigate this, there has been a growing interest in quantization techniques \cite{xiao2023smoothquant, yao2023comprehensive}." />
        </attvalues>
      </edge>
      <edge source="2405.00602" target="2205.05638" id="11236">
        <attvalues>
          <attvalue for="5" value=" Such strategies fall under the umbrella of Parameter-efficient fine-tuning (PEFT), as discussed in \cite{liu2022few}, where the focus is on fine-tuning a small number of parameters while retaining the core capabilities of LLMs." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2306.15595" id="11237">
        <attvalues>
          <attvalue for="5" value=" Recently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models~\cite{chen2023position_interpolation,peng2023yarn,chen2024longlora,ding2024longrope,fu2024data_engineer,zhang2024soaring,an2024make_your_llm_utilize_context}." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2309.00071" id="11238">
        <attvalues>
          <attvalue for="5" value=" Recently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models~\cite{chen2023position_interpolation,peng2023yarn,chen2024longlora,ding2024longrope,fu2024data_engineer,zhang2024soaring,an2024make_your_llm_utilize_context}." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2309.12307" id="11239">
        <attvalues>
          <attvalue for="5" value=" We also mix LongAlpaca~\cite{chen2024longlora} in the training set, which contains 12K instruction tuning instances with 16K length at maximum." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2402.13753" id="11240">
        <attvalues>
          <attvalue for="5" value=" Recently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models~\cite{chen2023position_interpolation,peng2023yarn,chen2024longlora,ding2024longrope,fu2024data_engineer,zhang2024soaring,an2024make_your_llm_utilize_context}." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2402.10171" id="11241">
        <attvalues>
          <attvalue for="5" value=" Recently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models~\cite{chen2023position_interpolation,peng2023yarn,chen2024longlora,ding2024longrope,fu2024data_engineer,zhang2024soaring,an2024make_your_llm_utilize_context}." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2401.03462" id="11242">
        <attvalues>
          <attvalue for="5" value=" Recently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models~\cite{chen2023position_interpolation,peng2023yarn,chen2024longlora,ding2024longrope,fu2024data_engineer,zhang2024soaring,an2024make_your_llm_utilize_context}." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2404.16811" id="11243">
        <attvalues>
          <attvalue for="5" value=" Recently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models~\cite{chen2023position_interpolation,peng2023yarn,chen2024longlora,ding2024longrope,fu2024data_engineer,zhang2024soaring,an2024make_your_llm_utilize_context}." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2303.08774" id="11244">
        <attvalues>
          <attvalue for="5" value=" Specifically, we use GPT-4~\cite{openai2024gpt4} to synthesize 3." />
        </attvalues>
      </edge>
      <edge source="2404.19553" target="2305.14314" id="11245">
        <attvalues>
          <attvalue for="5" value=" We use QLoRA~\cite{dettmers2023qlora} to efficiently fine-tune the model." />
        </attvalues>
      </edge>
      <edge source="2212.10750" target="2106.09449" id="11246">
        <attvalues>
          <attvalue for="5" value=" We demonstrate the utility of our dataset and models through downstream use case studies on summary hallucination detection \cite{maynez-etal-2020-faithfulness}, and DocNLI \cite{yin-etal-2021-docnli}, through which we show that recognizing and decomposing entailment relations at the proposition-level could provide fine-grained characterization and explanation for NLI-like tasks, especially with long and compositional hypotheses." />
        </attvalues>
      </edge>
      <edge source="2212.10750" target="2204.07447" id="11247">
        <attvalues>
          <attvalue for="5" value=" We construct PropSegmEnt by sampling clusters of topically-aligned documents, eg\ documents focusing on the same entity or event, from Wikipedia \cite{schuster2022stretching} and the news domains \cite{gu2020generating}." />
        </attvalues>
      </edge>
      <edge source="2212.10750" target="2001.09386" id="11248">
        <attvalues>
          <attvalue for="5" value=" We construct PropSegmEnt by sampling clusters of topically-aligned documents, eg\ documents focusing on the same entity or event, from Wikipedia \cite{schuster2022stretching} and the news domains \cite{gu2020generating}." />
        </attvalues>
      </edge>
      <edge source="2212.10750" target="2005.00661" id="11249">
        <attvalues>
          <attvalue for="5" value=" We demonstrate the utility of our dataset and models through downstream use case studies on summary hallucination detection \cite{maynez-etal-2020-faithfulness}, and DocNLI \cite{yin-etal-2021-docnli}, through which we show that recognizing and decomposing entailment relations at the proposition-level could provide fine-grained characterization and explanation for NLI-like tasks, especially with long and compositional hypotheses." />
        </attvalues>
      </edge>
      <edge source="2305.16626" target="1805.04843" id="11250">
        <attvalues>
          <attvalue for="5" value=" Since QG can be applied in not only educational scenarios~\cite{kurdi2020systematic, steuer2021not, moon-etal-2022-evaluating} but also improving question-answering tasks~\cite{chen2021factuality,wang2018learning, yu2020based}, designing better QG frameworks and their automatic evaluations have gained more attention~\cite{ However, previous QG works mostly evaluate their methods based on how similar the generated questions are to the gold reference questions~\cite{chan-fan-2019-recurrent, DBLP:conf/nlpcc/ZhouYWTBZ17, du-cardie-2018-harvesting}, using n-gram-based similarity metrics, such as BLEU~\cite{papineni-etal-2002-bleu} and ROUGE~\cite{lin-2004-rouge}." />
        </attvalues>
      </edge>
      <edge source="2305.16626" target="1603.08023" id="11252">
        <attvalues>
          <attvalue for="5" value=" Given a single reference, these metrics do not account for the lexical and semantic diversity of questions~\cite{DBLP:conf/iclr/ZhangKWWA20}, showing poor correlation with human judgment~\cite{liu-etal-2016-evaluate, novikova-etal-2017-need, chaganty-etal-2018-price}." />
        </attvalues>
      </edge>
      <edge source="2305.16626" target="2004.04696" id="11253">
        <attvalues>
          <attvalue for="5" value=" Though prior works studied alternative metrics of leveraging language models, such as BERTScore~\cite{DBLP:conf/iclr/ZhangKWWA20} and BLEURT~\cite{sellam-etal-2020-bleurt}, such metrics are limited in that the diversity of gold questions is only implicitly represented in the embedding space, rather than data space (or, raw questions)." />
        </attvalues>
      </edge>
      <edge source="2305.16626" target="2005.14165" id="11254">
        <attvalues>
          <attvalue for="5" value=" To explicitly compare with the diverse gold questions in the data space, we propose to augment the single reference question for evaluating QG frameworks, which we call Multi-Reference Evaluation (MRE), by leveraging the few-shot ability of large language models (LLMs) like GPT-3~\cite{NEURIPS2020_1457c0d6} and ChatGPT~\cite{openai_2023}." />
        </attvalues>
      </edge>
      <edge source="2305.16626" target="2004.14989" id="11255">
        <attvalues>
          <attvalue for="5" value=" Though there have been efforts to augment references for improving evaluations, they are either limited in other text generation tasks, such as machine translation~\cite{bawden-etal-2020-study} and question answering~\cite{liu-etal-2021-language}, or the methods are hard to be applied in question generation tasks, as naive LLMs generate some negative (toxic or erroneous) questions~\cite{10." />
        </attvalues>
      </edge>
      <edge source="2305.16626" target="2108.08485" id="11256">
        <attvalues>
          <attvalue for="5" value=" Though there have been efforts to augment references for improving evaluations, they are either limited in other text generation tasks, such as machine translation~\cite{bawden-etal-2020-study} and question answering~\cite{liu-etal-2021-language}, or the methods are hard to be applied in question generation tasks, as naive LLMs generate some negative (toxic or erroneous) questions~\cite{10." />
        </attvalues>
      </edge>
      <edge source="2305.16626" target="2205.01730" id="11257">
        <attvalues>
          <attvalue for="5" value=" Experimental results on quiz design dataset~\cite{laban2022quiz} show that the performance of existing metrics can be considerably improved when MRE is applied." />
        </attvalues>
      </edge>
      <edge source="2406.14956" target="2003.08271" id="11258">
        <attvalues>
          <attvalue for="5" value=" Recently, large language models (LLMs) have shown impressive performance in a range of natural language processing tasks~\cite{llmsurvey}." />
        </attvalues>
      </edge>
      <edge source="2406.14956" target="1902.00751" id="11259">
        <attvalues>
          <attvalue for="5" value=" To mitigate this, parameter-efficient tuning (PET) methods have been developed to fine-tune a small number of (extra) model parameters instead of the entire model~\cite{serialadapter}." />
        </attvalues>
      </edge>
      <edge source="2406.14956" target="2106.09685" id="11260">
        <attvalues>
          <attvalue for="5" value=" Low-rank adaptation (LoRA) \cite{lora} is now the de-facto PET method." />
        </attvalues>
      </edge>
      <edge source="2406.14956" target="2101.08134" id="11262">
        <attvalues>
          <attvalue for="5" value=" HeteroLoRA leverages zero-cost proxies~\cite{zerocostproxy} to avoid the high cost of brute-force search." />
        </attvalues>
      </edge>
      <edge source="2203.17217" target="1908.10090" id="11263">
        <attvalues>
          <attvalue for="5" value=" Yet, many authors have noticed that maximum-likelihood decoding (or an approximation thereto) yields incredibly unnatural text \cite{pmlr-v97-cohen19a,stahlberg_nmt_2019,eikema_is_2020}\clara{citation from 2021?}." />
        </attvalues>
      </edge>
      <edge source="2203.17217" target="2005.10283" id="11264">
        <attvalues>
          <attvalue for="5" value=" Yet, many authors have noticed that maximum-likelihood decoding (or an approximation thereto) yields incredibly unnatural text \cite{pmlr-v97-cohen19a,stahlberg_nmt_2019,eikema_is_2020}\clara{citation from 2021?}." />
        </attvalues>
      </edge>
      <edge source="2203.17217" target="2004.10450" id="11265">
        <attvalues>
          <attvalue for="5" value=" Rather, the relationship between probability and quality appears to have an inflection point \cite{zhang_trading_2020}, eg, quality and probability are positively correlated only up until a certain point, after which the relationship becomes negative." />
        </attvalues>
      </edge>
      <edge source="1909.03759" target="1809.01494" id="11266">
        <attvalues>
          <attvalue for="5" value=" ShARC, a conversational QA task \cite{ShARC}, requires a system to answer user questions based on rules expressed in natural language text." />
        </attvalues>
      </edge>
      <edge source="1909.03759" target="1908.05915" id="11269">
        <attvalues>
          <attvalue for="5" value=" Several deep learning models such as BERT-QA \cite{BERT}, E3 \cite{E3}, and BiSon \cite{BiSon} perform reasonably well on this task." />
        </attvalues>
      </edge>
      <edge source="1909.03759" target="2003.04808" id="11270">
        <attvalues>
          <attvalue for="5" value=" We observe that the performance of the models mentioned above drops when they are tested on a perturbed dataset, suggesting that the underlying neural models do not generalize and are rather over-sensitive \cite{welbl2020undersensitivity} to minor textual perturbations." />
        </attvalues>
      </edge>
      <edge source="1909.03759" target="2005.09241" id="11271">
        <attvalues>
          <attvalue for="5" value=" By sensitivity we refer to a model's ability to generalize itself but not over-fit, while still being invariant to perturbations or text transformations \cite{teney2020value, szegedy2013intriguing}." />
        </attvalues>
      </edge>
      <edge source="1909.03759" target="1907.07355" id="11272">
        <attvalues>
          <attvalue for="5" value=" Our observations about conversational QA models designed for ShARC learning spurious statistical clues are in line with those reported by \cite{ProbingNN}." />
        </attvalues>
      </edge>
      <edge source="2203.07402" target="1901.04587" id="11274">
        <attvalues>
          <attvalue for="5" value=" The need for such inductive bias is justified via psychological experiments \cite{colors} indicating that humans do have the ability to generalize on such tasks." />
        </attvalues>
      </edge>
      <edge source="2203.07402" target="1910.02612" id="11275">
        <attvalues>
          <attvalue for="5" value=" This has led to the development of many specialized architectures \cite{prim_subs, permutation, ness, lexicon}, learning procedures \cite{lake_meta, titov_meta} and data augmentation methods \cite{geca, kim-rush-aug} to solve the task." />
        </attvalues>
      </edge>
      <edge source="2203.07402" target="1906.05381" id="11276">
        <attvalues>
          <attvalue for="5" value=" This has led to the development of many specialized architectures \cite{prim_subs, permutation, ness, lexicon}, learning procedures \cite{lake_meta, titov_meta} and data augmentation methods \cite{geca, kim-rush-aug} to solve the task." />
        </attvalues>
      </edge>
      <edge source="2203.07402" target="1904.09545" id="11277">
        <attvalues>
          <attvalue for="5" value=" This has led to the development of many specialized architectures \cite{prim_subs, permutation, ness, lexicon}, learning procedures \cite{lake_meta, titov_meta} and data augmentation methods \cite{geca, kim-rush-aug} to solve the task." />
        </attvalues>
      </edge>
      <edge source="2205.06910" target="1705.11168" id="11278">
        <attvalues>
          <attvalue for="5" value=" Computational explorations of this claim often study the extent to which models that learn semantic representations through text alone can capture conceptual knowledge \cite{lucy-gauthier-2017-distributional, forbes2019neural, da-kasai-2019-cracking, bhatia2020transformer}." />
        </attvalues>
      </edge>
      <edge source="2205.06910" target="1508.05326" id="11279">
        <attvalues>
          <attvalue for="5" value=" On the basis of the goals of the task, our framework focuses on reasoning where conclusions do not deductively follow from the premise, unlike the goals of the more commonly-used task of natural language inference \cite{bowman2015large}, and it therefore allows for testing of human-like inferences that are seldom studied in LMs \cite[cf." />
        </attvalues>
      </edge>
      <edge source="2205.06910" target="2001.03632" id="11280">
        <attvalues>
          <attvalue for="5" value=" Next, as we show below, our framework opens a new window into exploring how large neural network models of language generalize beyond their training experience, complementing inquiries of models' inductive bias with respect to syntactic structure \cite{mccoy2020does} and ``universal linguistic constraints'' \cite{mccoy2020universal}." />
        </attvalues>
      </edge>
      <edge source="2205.06910" target="1910.01157" id="11282">
        <attvalues>
          <attvalue for="5" value=" Additionally, this work advances research aiming to diagnose the nature and extent of conceptual knowledge in LMs \cite{da-kasai-2019-cracking,forbes2019neural,weir2020probing, bhatia2020transformer} by additionally focusing on how knowledge present in LM representations drives the generalizations they make." />
        </attvalues>
      </edge>
      <edge source="2205.06910" target="1908.02899" id="11283">
        <attvalues>
          <attvalue for="5" value=" Additionally, this work advances research aiming to diagnose the nature and extent of conceptual knowledge in LMs \cite{da-kasai-2019-cracking,forbes2019neural,weir2020probing, bhatia2020transformer} by additionally focusing on how knowledge present in LM representations drives the generalizations they make." />
        </attvalues>
      </edge>
      <edge source="2205.06910" target="2004.04877" id="11284">
        <attvalues>
          <attvalue for="5" value=" Additionally, this work advances research aiming to diagnose the nature and extent of conceptual knowledge in LMs \cite{da-kasai-2019-cracking,forbes2019neural,weir2020probing, bhatia2020transformer} by additionally focusing on how knowledge present in LM representations drives the generalizations they make." />
        </attvalues>
      </edge>
      <edge source="2408.00357" target="2307.09288" id="11291">
        <attvalues>
          <attvalue for="5" value="Open-source large language models, such as LLAMA \cite{2} and LLAMA2 \cite{3}, Falcon \cite{4}, Vicuna \cite{5}, MOSS \cite{6}, ChatGLM, and ChatGLM2 \cite{7}, have demonstrated satisfactory performance in general-purpose domains following pre-training on large-scale corpora." />
        </attvalues>
      </edge>
      <edge source="2408.00357" target="2103.10360" id="11292">
        <attvalues>
          <attvalue for="5" value="Open-source large language models, such as LLAMA \cite{2} and LLAMA2 \cite{3}, Falcon \cite{4}, Vicuna \cite{5}, MOSS \cite{6}, ChatGLM, and ChatGLM2 \cite{7}, have demonstrated satisfactory performance in general-purpose domains following pre-training on large-scale corpora." />
        </attvalues>
      </edge>
      <edge source="2408.00357" target="2303.08774" id="11293">
        <attvalues>
          <attvalue for="5" value=" Even the state-of-the-art GPT-4 model \cite{8} in the Chinese legal domain generates a significant number of fictitious legal texts, highlighting the prevalence of this issue." />
        </attvalues>
      </edge>
      <edge source="2408.00357" target="2305.15062" id="11294">
        <attvalues>
          <attvalue for="5" value=" \cite{10} proposed a method that retrieves relevant legal articles based on the user’s query and contextual information, which can serve as evidence for the query before generating each response." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2208.10099" id="11296">
        <attvalues>
          <attvalue for="5" value=" Text-to-SQL, the task of converting natural language to SQL queries, enables non-technical users to access databases with natural language \cite{deng-etal-2022-recent, katsogiannis2023survey}." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2307.09288" id="11297">
        <attvalues>
          <attvalue for="5" value=" Recently, Large Language Models (LLMs) have made significant progress on various tasks \cite{touvron2023llama,openai2023gpt}." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2303.08774" id="11298">
        <attvalues>
          <attvalue for="5" value=" Recently, Large Language Models (LLMs) have made significant progress on various tasks \cite{touvron2023llama,openai2023gpt}." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2201.11903" id="11299">
        <attvalues>
          <attvalue for="5" value=" Although researchers have proposed various methods to enhance the reasoning abilities of LLMs \cite{wei2022chain,yao2023tree,besta2024graph}, However, they are still facing challenges with Text-to-SQL tasks \cite{li2023can,hong2024next}." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2305.10601" id="11300">
        <attvalues>
          <attvalue for="5" value=" Although researchers have proposed various methods to enhance the reasoning abilities of LLMs \cite{wei2022chain,yao2023tree,besta2024graph}, However, they are still facing challenges with Text-to-SQL tasks \cite{li2023can,hong2024next}." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2308.09687" id="11301">
        <attvalues>
          <attvalue for="5" value=" Although researchers have proposed various methods to enhance the reasoning abilities of LLMs \cite{wei2022chain,yao2023tree,besta2024graph}, However, they are still facing challenges with Text-to-SQL tasks \cite{li2023can,hong2024next}." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2406.08426" id="11302">
        <attvalues>
          <attvalue for="5" value=" Although researchers have proposed various methods to enhance the reasoning abilities of LLMs \cite{wei2022chain,yao2023tree,besta2024graph}, However, they are still facing challenges with Text-to-SQL tasks \cite{li2023can,hong2024next}." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2304.03442" id="11303">
        <attvalues>
          <attvalue for="5" value=" The LLM-based multi-agent system leverages collective intelligence from a group of LLMs and have achieved exceptional performance across various tasks \cite{park2023generative,hong2023metagpt,xu2023language}, but little work explores using them on Text-to-SQL." />
        </attvalues>
      </edge>
      <edge source="2402.14851" target="2310.18940" id="11304">
        <attvalues>
          <attvalue for="5" value=" The LLM-based multi-agent system leverages collective intelligence from a group of LLMs and have achieved exceptional performance across various tasks \cite{park2023generative,hong2023metagpt,xu2023language}, but little work explores using them on Text-to-SQL." />
        </attvalues>
      </edge>
      <edge source="2403.04894" target="2303.12712" id="11305">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) are highly capable at a variety of NLP tasks when prompted with appropriate natural language instructions \cite{bubeck2023sparks,few-shot-learners}." />
        </attvalues>
      </edge>
      <edge source="2403.04894" target="2005.14165" id="11306">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) are highly capable at a variety of NLP tasks when prompted with appropriate natural language instructions \cite{bubeck2023sparks,few-shot-learners}." />
        </attvalues>
      </edge>
      <edge source="2403.04894" target="2010.15980" id="11307">
        <attvalues>
          <attvalue for="5" value=" Some rely on access to model parameters and gradients to optimize discrete \cite{shin-etal-2020-autoprompt} or continuous \cite{lester-etal-2021-power, qin-eisner-2021-learning} prompts given task-specific training data." />
        </attvalues>
      </edge>
      <edge source="2403.04894" target="2104.08691" id="11308">
        <attvalues>
          <attvalue for="5" value=" Some rely on access to model parameters and gradients to optimize discrete \cite{shin-etal-2020-autoprompt} or continuous \cite{lester-etal-2021-power, qin-eisner-2021-learning} prompts given task-specific training data." />
        </attvalues>
      </edge>
      <edge source="2403.04894" target="2205.12548" id="11309">
        <attvalues>
          <attvalue for="5" value=" Others involve revising the task-prompt with discrete manipulations, such as through reinforcement learning \cite{deng-etal-2022-rlprompt, zhang2022tempera, hao2022optimizing}." />
        </attvalues>
      </edge>
      <edge source="2403.04894" target="2211.01910" id="11310">
        <attvalues>
          <attvalue for="5" value=" Discrete mutations of the task-prompt can also be made via another LLM \cite{zhou2023llms_are_prompt_engineers,pryzant-etal-2023-automatic}." />
        </attvalues>
      </edge>
      <edge source="2403.04894" target="2309.16797" id="11311">
        <attvalues>
          <attvalue for="5" value=" To evaluate ConstitutionalExperts, we compare it to state-of-the-art prompt optimizing baselines, including ProTeGi \cite{pryzant-etal-2023-automatic} and PromptBreeder \cite{fernando2023promptbreeder}, across six NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2403.04894" target="2212.08073" id="11312">
        <attvalues>
          <attvalue for="5" value=" Our approach is inspired by the ConstitutionalAI workflow \cite{bai2022constitutional} used to create fine-tuning datasets for LLMs." />
        </attvalues>
      </edge>
      <edge source="2311.09709" target="2005.14165" id="11314">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) are gaining increasing attention given their strong performance \cite{Radford2019Language,brown2020language,workshop2023BLOOM,touvron2023LLaMA}." />
        </attvalues>
      </edge>
      <edge source="2311.09709" target="2101.00421" id="11315">
        <attvalues>
          <attvalue for="5" value=" While anticipating certain limitations such as domain mismatch \cite{bogoychev-chen-2021-highs,domhan-etal-2022-devil}, vocabulary shortlisting in LLMs poses a fundamental challenge: often LLM outputs are variable and open-ended, complicating the determination of the required lexicons." />
        </attvalues>
      </edge>
      <edge source="2311.09709" target="2205.06618" id="11316">
        <attvalues>
          <attvalue for="5" value=" While anticipating certain limitations such as domain mismatch \cite{bogoychev-chen-2021-highs,domhan-etal-2022-devil}, vocabulary shortlisting in LLMs poses a fundamental challenge: often LLM outputs are variable and open-ended, complicating the determination of the required lexicons." />
        </attvalues>
      </edge>
      <edge source="2101.00421" target="1409.0473" id="11317">
        <attvalues>
          <attvalue for="5" value=" Neural Machine translation (NMT) has achieved state-of-the-art performance in a variety of language pairs and settings \cite{bahdanau_nmt, vaswani_transformer}, but it is vulnerable to domain mismatch, where the test set differs significantly from the training data in terms of vocabulary, genre, length, etc This issue is exacerbated in a low-resource condition \cite{koehn-knowles-2017-six}." />
        </attvalues>
      </edge>
      <edge source="2101.00421" target="1706.03872" id="11318">
        <attvalues>
          <attvalue for="5" value=" Neural Machine translation (NMT) has achieved state-of-the-art performance in a variety of language pairs and settings \cite{bahdanau_nmt, vaswani_transformer}, but it is vulnerable to domain mismatch, where the test set differs significantly from the training data in terms of vocabulary, genre, length, etc This issue is exacerbated in a low-resource condition \cite{koehn-knowles-2017-six}." />
        </attvalues>
      </edge>
      <edge source="2101.00421" target="1911.03109" id="11319">
        <attvalues>
          <attvalue for="5" value=" This typically results in hallucinations in the output \cite{mller2019domain}, because the overly zealous language model component prefers a fluent translation, as opposed to an adequate one." />
        </attvalues>
      </edge>
      <edge source="2101.00421" target="1511.06732" id="11320">
        <attvalues>
          <attvalue for="5" value=" A number of methods have been proposed in order to tackle the issue: exposing the model to its predictions during training \cite{ranzato2015sequence, shen-etal-2016-minimum, zhang-etal-2019-bridging, rico-exposure}; tuning directly towards BLEU \cite{wiseman-rush-2016-sequence} or using minimum Bayes risk decoding \cite{kumar-byrne-2004-minimum, stahlberg-etal-2017-neural}." />
        </attvalues>
      </edge>
      <edge source="2101.00421" target="1606.02960" id="11321">
        <attvalues>
          <attvalue for="5" value=" A number of methods have been proposed in order to tackle the issue: exposing the model to its predictions during training \cite{ranzato2015sequence, shen-etal-2016-minimum, zhang-etal-2019-bridging, rico-exposure}; tuning directly towards BLEU \cite{wiseman-rush-2016-sequence} or using minimum Bayes risk decoding \cite{kumar-byrne-2004-minimum, stahlberg-etal-2017-neural}." />
        </attvalues>
      </edge>
      <edge source="2404.12845" target="1905.00537" id="11323">
        <attvalues>
          <attvalue for="5" value=" While there exist a number of benchmarks, such as GLUE \cite{wang2018glue}, SuperGLUE \cite{wang2019superglue}, or XGLUE \cite{liang2020xglue}, for evaluating the quality of embeddings and language models for modern languages, such benchmarks are lacking for ancient and historical languages." />
        </attvalues>
      </edge>
      <edge source="2404.12845" target="2004.01401" id="11324">
        <attvalues>
          <attvalue for="5" value=" While there exist a number of benchmarks, such as GLUE \cite{wang2018glue}, SuperGLUE \cite{wang2019superglue}, or XGLUE \cite{liang2020xglue}, for evaluating the quality of embeddings and language models for modern languages, such benchmarks are lacking for ancient and historical languages." />
        </attvalues>
      </edge>
      <edge source="2404.12845" target="2009.10053" id="11326">
        <attvalues>
          <attvalue for="5" value=" Large pre-trained language models, however, are predominantly trained on corpora of modern languages, with few exceptions such as LatinBERT~\cite{bamman2020latin}." />
        </attvalues>
      </edge>
      <edge source="2404.12845" target="1811.01088" id="11327">
        <attvalues>
          <attvalue for="5" value=" A useful feature of both the supplementary training approach of \cite{phang2018sentence} and the adapter-based training of \cite{pfeiffer2020mad} is that they provide a uniform framework that can be applied to different languages and tasks in a similar manner." />
        </attvalues>
      </edge>
      <edge source="2404.12845" target="2005.00052" id="11328">
        <attvalues>
          <attvalue for="5" value=" Our submission to the SIGTYP 2024 Shared Task on Ancient on Word Embedding Evaluation for Ancient and Historical Languages adopts the methods described by \cite{pfeiffer2020mad, pfeiffer2020unks} by stacking fine-tuned language and task adapters, and customizing the tokenizer and the embedding layers." />
        </attvalues>
      </edge>
      <edge source="2404.12845" target="1902.00751" id="11329">
        <attvalues>
          <attvalue for="5" value=" \cite{pfeiffer2020mad} developed a cross-lingual transfer-learning approach based on the adapters framework for parameter efficient fine-tuning of language models~\cite{houlsby2019parameter, bapna2019simple}." />
        </attvalues>
      </edge>
      <edge source="2404.12845" target="2012.15562" id="11330">
        <attvalues>
          <attvalue for="5" value=" This adapter-based method is expanded by \cite{pfeiffer2020unks} by adopting a custom tokenizer and an embedding layer." />
        </attvalues>
      </edge>
      <edge source="2310.06003" target="1311.2524" id="11331">
        <attvalues>
          <attvalue for="5" value=" With the development of machine learning technology, the overall performance of deep learning algorithms in fields such as face recognition, recommender system, and natural language processing has significantly improved \cite{girshick2014rich,G-Meta,NEURIPS2020_1457c0d6}." />
        </attvalues>
      </edge>
      <edge source="2310.06003" target="2401.04338" id="11332">
        <attvalues>
          <attvalue for="5" value=" With the development of machine learning technology, the overall performance of deep learning algorithms in fields such as face recognition, recommender system, and natural language processing has significantly improved \cite{girshick2014rich,G-Meta,NEURIPS2020_1457c0d6}." />
        </attvalues>
      </edge>
      <edge source="2310.06003" target="2005.14165" id="11333">
        <attvalues>
          <attvalue for="5" value=" Over the past few years, model size has increased from 110 million parameters for BERT \cite{devlin2019bert} to 175 billion parameters for GPT-3 \cite{NEURIPS2020_1457c0d6}." />
        </attvalues>
      </edge>
      <edge source="2310.06003" target="2110.14883" id="11335">
        <attvalues>
          <attvalue for="5" value=" In LLM training, to effectively utilize the computing power and memory of hundreds of GPU devices, a variety of distributed parallel training technologies have been proposed, such as data parallelism (DP), tensor parallelism (TP) and pipeline parallelism (PP) \cite{2023ColossalAI}." />
        </attvalues>
      </edge>
      <edge source="2310.06003" target="1804.05349" id="11336">
        <attvalues>
          <attvalue for="5" value=" However, since a complete model is copied on each GPU, significant memory redundancy occurs, especially when training large models \cite{proficz_improving_2018}." />
        </attvalues>
      </edge>
      <edge source="2310.06003" target="1910.02054" id="11337">
        <attvalues>
          <attvalue for="5" value=" \cite{ZeRO} proposed the Zero Redundancy Optimization (ZeRO) strategy set, which splits the model state (eg optimizer state, gradient and parameters) based on data parallelism and reconstructs them through the collective communication." />
        </attvalues>
      </edge>
      <edge source="2310.06003" target="1910.04940" id="11338">
        <attvalues>
          <attvalue for="5" value=" In high-performance clusters such as NVIDIA DGX-2 or DGX-A100 \cite{Blink2020}, NVLink/NVSwitch with a bandwidth of up to 4." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2208.11761" id="11344">
        <attvalues>
          <attvalue for="5" value=" In recent years, the efficacy of self-supervised representation learning (SSRL) methodologies has been empirically validated across diverse downstream tasks in the speech community \cite{yang2021superb, tsai2022superb, xuankai2021asru, peng2023study, evain21_interspeech, shi23g_interspeech, conneau22_interspeech,javed2023indicsuperb, mohamed2022self}." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2006.11477" id="11345">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, WavLM \cite{chen2022wavlm}, w2v-bert \cite{chung2021w2v}, and BEST-RQ \cite{chiu2022self}, predominantly rely on pseudo-labels generated from either mel-frequency cepstrum coefficients (MFCC) or the model's own representation features." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2110.13900" id="11346">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, WavLM \cite{chen2022wavlm}, w2v-bert \cite{chung2021w2v}, and BEST-RQ \cite{chiu2022self}, predominantly rely on pseudo-labels generated from either mel-frequency cepstrum coefficients (MFCC) or the model's own representation features." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2108.06209" id="11347">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, WavLM \cite{chen2022wavlm}, w2v-bert \cite{chung2021w2v}, and BEST-RQ \cite{chiu2022self}, predominantly rely on pseudo-labels generated from either mel-frequency cepstrum coefficients (MFCC) or the model's own representation features." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2104.00355" id="11349">
        <attvalues>
          <attvalue for="5" value=" This highlighted that the pseudo-labels inherently contain some semantic information, and this fact is also supported by the studies of \cite{polyak2021speech} and \cite{wu2023improving}." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2305.18096" id="11350">
        <attvalues>
          <attvalue for="5" value=" This highlighted that the pseudo-labels inherently contain some semantic information, and this fact is also supported by the studies of \cite{polyak2021speech} and \cite{wu2023improving}." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2204.09224" id="11351">
        <attvalues>
          <attvalue for="5" value=" ContentVec \cite{qian2022contentvec} and Spin \cite{chang2023self} are self-supervised methods that disentangle speaker information to impose a speaker-invariant constraint on a pre-trained HuBERT." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2305.11072" id="11352">
        <attvalues>
          <attvalue for="5" value=" ContentVec \cite{qian2022contentvec} and Spin \cite{chang2023self} are self-supervised methods that disentangle speaker information to impose a speaker-invariant constraint on a pre-trained HuBERT." />
        </attvalues>
      </edge>
      <edge source="2310.03975" target="2211.03025" id="11353">
        <attvalues>
          <attvalue for="5" value=" utilized unsupervised ASR to bridge speech SSL with text-pre-trained models in order to take benefits from semantic information from the textual representations \cite{shi2023bridging}." />
        </attvalues>
      </edge>
      <edge source="2304.01083" target="2005.14165" id="11373">
        <attvalues>
          <attvalue for="5" value=" In recent years, large language models (LLMs), eg, ChatGPT \cite{chatgpt} and GPT-3 \cite{brown2020language}, have exhibited remarkable performance." />
        </attvalues>
      </edge>
      <edge source="2304.01083" target="2111.06206" id="11374">
        <attvalues>
          <attvalue for="5" value=" It has been discovered that among all $2^n$ interactions, a DNN usually only encodes a small number of salient interactions, and all other interactions have ignorable effects $I(S)\approx 0$ \cite{ren2023defining,li2023does}." />
        </attvalues>
      </edge>
      <edge source="2208.01818" target="1712.01769" id="11375">
        <attvalues>
          <attvalue for="5" value=" At the same time, it has shown equivalent or better performance over conventional hidden Markov model (HMM) based approaches \cite{chiu2018state, pham2019very, guo2021recent}." />
        </attvalues>
      </edge>
      <edge source="2208.01818" target="1506.07503" id="11376">
        <attvalues>
          <attvalue for="5" value=" According to recent literature, we have observed two mainstream modeling directions for end-to-end ASR: attention-based encoder-decoder (AED) \cite{chorowski2015attention, chan2016listen, watanabe2017hybrid}, and RNN transducer (RNN-T) \cite{graves2012sequence, rao2017exploring, saon2021advancing}." />
        </attvalues>
      </edge>
      <edge source="2208.01818" target="1211.3711" id="11377">
        <attvalues>
          <attvalue for="5" value=" According to recent literature, we have observed two mainstream modeling directions for end-to-end ASR: attention-based encoder-decoder (AED) \cite{chorowski2015attention, chan2016listen, watanabe2017hybrid}, and RNN transducer (RNN-T) \cite{graves2012sequence, rao2017exploring, saon2021advancing}." />
        </attvalues>
      </edge>
      <edge source="2208.01818" target="cs/0010012" id="11378">
        <attvalues>
          <attvalue for="5" value=" However, $N$-best lists are insufficient for some downstream applications or post-processing stages, such as \begin{inparaenum}[(1)] \item language model rescoring~\cite{ljolje1999efficient, sak2010fly, rybach2017lattice}; \item downstream processing of ASR output (eg, translation\cite{kumar2014some} and keyword spotting \cite{kingsbury2013high, rosenberg2017end}); \item confusion network generation~\cite{mangu2000finding, hakkani2006beyond}; and \item sequence discriminative training \cite{normandin1996maximum, povey2002, gibson2006hypothesis}." />
        </attvalues>
      </edge>
      <edge source="2208.01818" target="2012.06749" id="11379">
        <attvalues>
          <attvalue for="5" value=" \cite{prabhavalkar2021less} further investigated hypothesis merging for recurrent neural network transducer (RNN-T) with a full-context prediction network." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="1905.05879" id="11381">
        <attvalues>
          <attvalue for="5" value=" There are several approaches in this category, including autoencoder-based approaches \cite{qian2019autovc, yuan2021improving, wang2021vqmivc, chen2021again, lian2022robust}, where input speech is mapped to a bottleneck representation with much smaller dimensions than the input and decoded back to that of the target speaker; ASR-based and TTS-based approaches \cite{casanova2022yourtts, levkovitch2022zero, li2023styletts, hussain2023ace}, where input speech is mapped to the same latent representation as the phoneme representation from the text; and large pre-trained speech language model (SLM) approaches \cite{choi2021neural, qian2022contentvec, dang2022training}, where speech is directly reconstructed from SLM representations in deep layers that contain less speaker information." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2112.02418" id="11382">
        <attvalues>
          <attvalue for="5" value=" Furthermore, it outperforms YourTTS \cite{casanova2022yourtts} and StyleTTS-VC \cite{li2023styletts} in terms of naturalness, two of the best-performing publicly available models for zero-shot voice conversion that rely on text labels, albeit no text labels are required during training." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2110.14513" id="11383">
        <attvalues>
          <attvalue for="5" value=" There are several approaches in this category, including autoencoder-based approaches \cite{qian2019autovc, yuan2021improving, wang2021vqmivc, chen2021again, lian2022robust}, where input speech is mapped to a bottleneck representation with much smaller dimensions than the input and decoded back to that of the target speaker; ASR-based and TTS-based approaches \cite{casanova2022yourtts, levkovitch2022zero, li2023styletts, hussain2023ace}, where input speech is mapped to the same latent representation as the phoneme representation from the text; and large pre-trained speech language model (SLM) approaches \cite{choi2021neural, qian2022contentvec, dang2022training}, where speech is directly reconstructed from SLM representations in deep layers that contain less speaker information." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2010.12788" id="11384">
        <attvalues>
          <attvalue for="5" value=" GAN-based methods \cite{zhang2020gazev, nguyen2022nvc, takahashi2022robust}, on the other hand, employ a discriminator to determine whether the converted voice is from the target speaker or not." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2206.02246" id="11385">
        <attvalues>
          <attvalue for="5" value=" GAN-based methods typically produce more natural speech \cite{levkovitch2022zero, yasur2023deepfake}, as the latent representations are not forced to be disentangled, preserving more linguistic information." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2306.07691" id="11386">
        <attvalues>
          <attvalue for="5" value=" Our recent work has demonstrated promising results by employing SLMs as discriminators for text-to-speech (TTS), where we show that leveraging SLMs as discriminators improves the naturalness of synthesized speech, specifically from paralinguistic aspects \cite{li2023styletts2}." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2206.04658" id="11388">
        <attvalues>
          <attvalue for="5" value=" We employ a neural vocoder BIGVGAN \cite{lee2022bigvgan} to convert the generated mel-spectrograms into waveforms, which are then fed into the WavLM \cite{chen2022wavlm} encoder to obtain the SLM representations of the converted speech." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2110.13900" id="11389">
        <attvalues>
          <attvalue for="5" value=" We employ a neural vocoder BIGVGAN \cite{lee2022bigvgan} to convert the generated mel-spectrograms into waveforms, which are then fed into the WavLM \cite{chen2022wavlm} encoder to obtain the SLM representations of the converted speech." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2011.00316" id="11391">
        <attvalues>
          <attvalue for="5" value=" Subjective evaluations show that our model surpasses two baseline models, VQMIVIC \cite{wang2021vqmivc} and AGAIN-VC \cite{chen2021again}, in terms of both naturalness and similarity." />
        </attvalues>
      </edge>
      <edge source="2307.09435" target="2212.14227" id="11392">
        <attvalues>
          <attvalue for="5" value=" Furthermore, it outperforms YourTTS \cite{casanova2022yourtts} and StyleTTS-VC \cite{li2023styletts} in terms of naturalness, two of the best-performing publicly available models for zero-shot voice conversion that rely on text labels, albeit no text labels are required during training." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="1707.07167" id="11393">
        <attvalues>
          <attvalue for="5" value=" Automatic speech recognition (ASR) systems have been used across various applications, such as video captioning~\cite{Liao2013LargeSD}, dictation~\cite{Li2015LSTMTA}, voice search~\cite{Wang2008AnIT,Shan2017AttentionBasedES}, voice assistant~\cite{Kepuska2018NextgenerationOV} and telephony~\cite{Hain2000THECM,Xiong2017TheM2,Tuske2021OnTL}." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="1708.06073" id="11394">
        <attvalues>
          <attvalue for="5" value=" Automatic speech recognition (ASR) systems have been used across various applications, such as video captioning~\cite{Liao2013LargeSD}, dictation~\cite{Li2015LSTMTA}, voice search~\cite{Wang2008AnIT,Shan2017AttentionBasedES}, voice assistant~\cite{Kepuska2018NextgenerationOV} and telephony~\cite{Hain2000THECM,Xiong2017TheM2,Tuske2021OnTL}." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="2105.00982" id="11395">
        <attvalues>
          <attvalue for="5" value=" Automatic speech recognition (ASR) systems have been used across various applications, such as video captioning~\cite{Liao2013LargeSD}, dictation~\cite{Li2015LSTMTA}, voice search~\cite{Wang2008AnIT,Shan2017AttentionBasedES}, voice assistant~\cite{Kepuska2018NextgenerationOV} and telephony~\cite{Hain2000THECM,Xiong2017TheM2,Tuske2021OnTL}." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="2008.06580" id="11396">
        <attvalues>
          <attvalue for="5" value=" Although domain adaptation has been widely studied for ASR~\cite{Bell2020AdaptationAF} with various paradigms including input feature adaptation~\cite{Gales1998MaximumLL,Fainberg2017FactorisedRF,Sainath2020ASO}, model-based adaptation~\cite{Li2017LargeScaleDA,Asami2017DomainAO,Manohar2018ATL,Samarakoon2018DomainAO,Sim2018DomainAU} and multi-task learning~\cite{Denisov2018UnsupervisedDA,Meng2018AdversarialTL}, our work pays particular attention to modularity." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="1808.05312" id="11397">
        <attvalues>
          <attvalue for="5" value=" To ensure the model performs well for all domains, one simple and effective approach is to mix all data during training to obtain a multidomain model~\cite{Narayanan2018TowardDS,Chan2021SpeechStewSM}." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="1712.01818" id="11399">
        <attvalues>
          <attvalue for="5" value=" Third, the final recipe was validated on three different domains with a large amount of data and minimum word error rate (MWER) training~\cite{Prabhavalkar2017MinimumWE}." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="2003.12710" id="11400">
        <attvalues>
          <attvalue for="5" value=" Although domain adaptation has been widely studied for ASR~\cite{Bell2020AdaptationAF} with various paradigms including input feature adaptation~\cite{Gales1998MaximumLL,Fainberg2017FactorisedRF,Sainath2020ASO}, model-based adaptation~\cite{Li2017LargeScaleDA,Asami2017DomainAO,Manohar2018ATL,Samarakoon2018DomainAO,Sim2018DomainAU} and multi-task learning~\cite{Denisov2018UnsupervisedDA,Meng2018AdversarialTL}, our work pays particular attention to modularity." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="1807.11284" id="11402">
        <attvalues>
          <attvalue for="5" value=" Although domain adaptation has been widely studied for ASR~\cite{Bell2020AdaptationAF} with various paradigms including input feature adaptation~\cite{Gales1998MaximumLL,Fainberg2017FactorisedRF,Sainath2020ASO}, model-based adaptation~\cite{Li2017LargeScaleDA,Asami2017DomainAO,Manohar2018ATL,Samarakoon2018DomainAO,Sim2018DomainAU} and multi-task learning~\cite{Denisov2018UnsupervisedDA,Meng2018AdversarialTL}, our work pays particular attention to modularity." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="1804.00644" id="11403">
        <attvalues>
          <attvalue for="5" value=" Although domain adaptation has been widely studied for ASR~\cite{Bell2020AdaptationAF} with various paradigms including input feature adaptation~\cite{Gales1998MaximumLL,Fainberg2017FactorisedRF,Sainath2020ASO}, model-based adaptation~\cite{Li2017LargeScaleDA,Asami2017DomainAO,Manohar2018ATL,Samarakoon2018DomainAO,Sim2018DomainAU} and multi-task learning~\cite{Denisov2018UnsupervisedDA,Meng2018AdversarialTL}, our work pays particular attention to modularity." />
        </attvalues>
      </edge>
      <edge source="2305.13408" target="2302.11529" id="11404">
        <attvalues>
          <attvalue for="5" value=" This brings many advantages of modularity~\cite{pfeiffer2023modular}: similar functions of the ASR model are encoded with the same module while allocating distinct functions to per-domain parameters; per-domain parameters can be constructed separately and updated locally; parameter efficiency is much higher than finetuning the entire model or have multiple single-domain models." />
        </attvalues>
      </edge>
      <edge source="2210.04107" target="1703.09902" id="11405">
        <attvalues>
          <attvalue for="5" value=" Traditionally, most data-to-text applications have been designed in a modular fashion as this facilitates reuse in different domains; going directly from input to output with rules has been simply too complex \cite{gatt2018survey}." />
        </attvalues>
      </edge>
      <edge source="2210.04107" target="1908.09022" id="11406">
        <attvalues>
          <attvalue for="5" value=" Recent empirical studies have demonstrated that a combination of template and pipeline systems produce texts that are more appropriate than the neural-based approaches, which frequently hallucinate content unsupported by the semantic input \cite{ferreira2019neural}." />
        </attvalues>
      </edge>
      <edge source="2104.08928" target="2010.03574" id="11407">
        <attvalues>
          <attvalue for="5" value=" For instance, service providers mine online consumer reviews to inform operational decisions on platforms~\cite{mankad2016understanding} or to infer market structure and the competitive landscape for products \cite{netzer2012mine}; Twitter posts are used to forecast TV show viewership \cite{liu2016structured}; analyst reports of S\&amp;P 500 firms are used to measure innovation \cite{bellstam2020text}; medical notes are used to predict operational metrics such as readmissions rates \cite{hsu-etal-2020-characterizing}; online ads or reviews are used to flag service providers that are likely engaging in illicit activities \cite{ramchandani2021unmasking, li2021detecting}." />
        </attvalues>
      </edge>
      <edge source="2104.08928" target="1310.4546" id="11408">
        <attvalues>
          <attvalue for="5" value=" Given the large number of words in the English language, to be effective in practice, embeddings must be trained on large-scale and comprehensive text data, eg, popular embeddings such as Word2Vec \cite{mikolov2013distributed} and GloVe \cite{pennington2014glove} are trained on Wikipedia articles." />
        </attvalues>
      </edge>
      <edge source="2104.08928" target="1709.07470" id="11409">
        <attvalues>
          <attvalue for="5" value=" Consequently, there has been a large body of work training specialized embeddings in a number of diverse contexts, ranging from radiology reports \cite{ong2020machine}, stock market prediction \cite{li2017learning}, cybersecurity vulnerability reports \cite{roy2017learning}, and patent classification \cite{risch2019domain}." />
        </attvalues>
      </edge>
      <edge source="2104.08928" target="1007.1771" id="11411">
        <attvalues>
          <attvalue for="5" value=" We build on prior work establishing error bounds for the group LASSO \cite{lounici2011oracle} and low-rank matrix problems \cite{ge2017no,negahban2011estimation}." />
        </attvalues>
      </edge>
      <edge source="2104.08928" target="1704.00708" id="11412">
        <attvalues>
          <attvalue for="5" value=" We build on prior work establishing error bounds for the group LASSO \cite{lounici2011oracle} and low-rank matrix problems \cite{ge2017no,negahban2011estimation}." />
        </attvalues>
      </edge>
      <edge source="2104.08928" target="1305.2436" id="11414">
        <attvalues>
          <attvalue for="5" value=" Furthermore, under a slightly weaker condition that can characterize all local minima \cite{loh2015regularized}, all local minima identified by our algorithm are statistically indistinguishable from the global minimum, implying that our estimator can be computed efficiently." />
        </attvalues>
      </edge>
      <edge source="2312.12343" target="2009.03300" id="11433">
        <attvalues>
          <attvalue for="5" value=" However, many widely used benchmarks are also largely constructed from web resources \cite{hendrycks2020measuring}, which are very likely to be unintentionally included in the pretraining stage." />
        </attvalues>
      </edge>
      <edge source="2312.12343" target="2303.08774" id="11434">
        <attvalues>
          <attvalue for="5" value=" Recent analysis has revealed that data contamination is widespread in model evaluations \cite{openai2023gpt4,sainz2023nlp}, which greatly undermines the credibility of evaluation results \cite{marie2023,li2023open} and prevents fair comparisons between models \cite{dickson2023}." />
        </attvalues>
      </edge>
      <edge source="2312.12343" target="2310.18018" id="11435">
        <attvalues>
          <attvalue for="5" value=" Recent analysis has revealed that data contamination is widespread in model evaluations \cite{openai2023gpt4,sainz2023nlp}, which greatly undermines the credibility of evaluation results \cite{marie2023,li2023open} and prevents fair comparisons between models \cite{dickson2023}." />
        </attvalues>
      </edge>
      <edge source="2312.12343" target="2310.17589" id="11436">
        <attvalues>
          <attvalue for="5" value=" Recent analysis has revealed that data contamination is widespread in model evaluations \cite{openai2023gpt4,sainz2023nlp}, which greatly undermines the credibility of evaluation results \cite{marie2023,li2023open} and prevents fair comparisons between models \cite{dickson2023}." />
        </attvalues>
      </edge>
      <edge source="2312.12343" target="2103.12028" id="11437">
        <attvalues>
          <attvalue for="5" value=" Moreover, the massive scale of training data makes decontaminating existing benchmarks extremely difficult \cite{kreutzer-etal-2022-quality}." />
        </attvalues>
      </edge>
      <edge source="2312.12343" target="2305.10160" id="11439">
        <attvalues>
          <attvalue for="5" value=" One potential solution to avoid contaminated evaluation is to create new test data constantly or use human evaluation \cite{liu2023evaluating,jacovi2023stop}, just like how examination for human works." />
        </attvalues>
      </edge>
      <edge source="2109.15144" target="1603.06155" id="11440">
        <attvalues>
          <attvalue for="5" value=" Language style should be a special consideration in current and future intelligent interaction systems~\cite{li2016persona} that understand, process, or generate speech or text." />
        </attvalues>
      </edge>
      <edge source="2109.15144" target="1605.06069" id="11441">
        <attvalues>
          <attvalue for="5" value=" Inspired by the success of the encoder-decoder models in other fields, including machine translation (MT)~\cite{cho2014properties, wu2016google}, text summarization~\cite{chopra2016abstractive} and dialogue generation~\cite{serban2017hierarchical}, a number of style transfer models are built upon this end-to-end model of learning~\cite{li2018delete, sudhakar2019transforming, xu2019formality, dai2019style, cheng2020contextual, fu2018style, dos2018fighting, prabhumoye2018style, john2019disentangled, zhang2018learning, prabhumoye2018style2, tian2018structured, liu2020revision, wang2019controllable, zhou2020exploring, lee2020stable, zhang2018shaped, hu2017toward}." />
        </attvalues>
      </edge>
      <edge source="2109.15144" target="1804.06437" id="11442">
        <attvalues>
          <attvalue for="5" value=" Auxiliary elements, such as style embeddings~\cite{li2018delete, sudhakar2019transforming, dai2019style, cheng2020contextual, fu2018style, dos2018fighting, zhang2018learning, lee2020stable, hu2017toward, shen2017style, logeswaran2018content, chen2018adversarial}, style classifiers~\cite{xu2019formality, dai2019style} and/or adversarial discriminators~\cite{shen2017style, zhao2018language} are also discussed." />
        </attvalues>
      </edge>
      <edge source="2109.15144" target="1903.06353" id="11444">
        <attvalues>
          <attvalue for="5" value=" Auxiliary elements, such as style embeddings~\cite{li2018delete, sudhakar2019transforming, dai2019style, cheng2020contextual, fu2018style, dos2018fighting, zhang2018learning, lee2020stable, hu2017toward, shen2017style, logeswaran2018content, chen2018adversarial}, style classifiers~\cite{xu2019formality, dai2019style} and/or adversarial discriminators~\cite{shen2017style, zhao2018language} are also discussed." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2303.18223" id="11445">
        <attvalues>
          <attvalue for="5" value=" Researchers have found that advanced AI technologies, exemplified by large language models (LLMs), are proficient in addressing a broad spectrum of challenges, spanning from everyday tasks to complex software engineering issues~\cite{chatgpt,LLMSurvey,codex,aprplm,codeagent1,wang2022usb,wang2023pandalm,wang2024exploring,kieval}." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2107.03374" id="11446">
        <attvalues>
          <attvalue for="5" value=" Previous multi-task benchmarks, like CodeXGLUE~\cite{codexglue}, have been critiqued~\cite{codex} because they rely on similarity-based metrics like BLEU and CodeBLEU~\cite{codebleu}, which do not involve running the code." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2304.01457" id="11449">
        <attvalues>
          <attvalue for="5" value=" Researchers have found that advanced AI technologies, exemplified by large language models (LLMs), are proficient in addressing a broad spectrum of challenges, spanning from everyday tasks to complex software engineering issues~\cite{chatgpt,LLMSurvey,codex,aprplm,codeagent1,wang2022usb,wang2023pandalm,wang2024exploring,kieval}." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2402.15043" id="11450">
        <attvalues>
          <attvalue for="5" value=" Researchers have found that advanced AI technologies, exemplified by large language models (LLMs), are proficient in addressing a broad spectrum of challenges, spanning from everyday tasks to complex software engineering issues~\cite{chatgpt,LLMSurvey,codex,aprplm,codeagent1,wang2022usb,wang2023pandalm,wang2024exploring,kieval}." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2308.12950" id="11451">
        <attvalues>
          <attvalue for="5" value=" The aim was to answer a couple of crucial questions: How good are these LLMs at coding? And how do continued pre-training~\cite{codellama} and instruction fine-tuning~\cite{wizardcoder} affect their programming performance? After running a slew of experiments on CoderUJB, we find that current LLMs still perform poorly in solving non-functional code generation tasks, especially defect detection tasks." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2305.06161" id="11452">
        <attvalues>
          <attvalue for="5" value=" In addition to these general-purpose LLMs, there are code-centric large language models (code LLMs), such as CodeX~\cite{codex}, CodeLlama~\cite{codellama}, and StarCoder~\cite{starcoder}, which are specifically designed to excel at software engineering tasks." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2302.00288" id="11453">
        <attvalues>
          <attvalue for="5" value=" Then, CoderEval~\cite{codereval} noticed that the questions in HumanEval are simple single-function generation tasks that do not match the actual development scenarios (eg, writing code in a software project)." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2102.04664" id="11454">
        <attvalues>
          <attvalue for="5" value=" Previous multi-task benchmarks, like CodeXGLUE~\cite{codexglue}, have been critiqued~\cite{codex} because they rely on similarity-based metrics like BLEU and CodeBLEU~\cite{codebleu}, which do not involve running the code." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2303.03004" id="11455">
        <attvalues>
          <attvalue for="5" value=" The recently proposed multi-programming task dataset XCodeEval~\cite{xcodeeval} focuses on questions from programming competitions, which do not accurately reflect typical real-world development scenarios." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2308.01861" id="11456">
        <attvalues>
          <attvalue for="5" value=" Because of this interest, different benchmarks~\cite{codex,codereval,codexglue,xcodeeval,classeval,aixbench} have been designed to measure the programming capabilities of these LLMs." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2009.10297" id="11458">
        <attvalues>
          <attvalue for="5" value=" Previous multi-task benchmarks, like CodeXGLUE~\cite{codexglue}, have been critiqued~\cite{codex} because they rely on similarity-based metrics like BLEU and CodeBLEU~\cite{codebleu}, which do not involve running the code." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2305.04207" id="11459">
        <attvalues>
          <attvalue for="5" value=" We extracted 238 functional code generation questions and 140 code-based test generation~\cite{chatteser} questions from these projects by analyzing the abstract syntax trees and test coverage relationships of the project source code." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2209.11515" id="11460">
        <attvalues>
          <attvalue for="5" value=" Then, we extracted and collected 451 issue-based test generation~\cite{issuetest} questions, 470 automatic program repair~\cite{aprplm} questions, and 940 defect detection~\cite{devign} questions from the projects by combining the detailed defect information and related issue reports from the projects." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="1909.03496" id="11461">
        <attvalues>
          <attvalue for="5" value=" Then, we extracted and collected 451 issue-based test generation~\cite{issuetest} questions, 470 automatic program repair~\cite{aprplm} questions, and 940 defect detection~\cite{devign} questions from the projects by combining the detailed defect information and related issue reports from the projects." />
        </attvalues>
      </edge>
      <edge source="2403.19287" target="2306.08568" id="11462">
        <attvalues>
          <attvalue for="5" value=" The aim was to answer a couple of crucial questions: How good are these LLMs at coding? And how do continued pre-training~\cite{codellama} and instruction fine-tuning~\cite{wizardcoder} affect their programming performance? After running a slew of experiments on CoderUJB, we find that current LLMs still perform poorly in solving non-functional code generation tasks, especially defect detection tasks." />
        </attvalues>
      </edge>
      <edge source="2304.11350" target="1706.03762" id="11474">
        <attvalues>
          <attvalue for="5" value=" We investigate the advantages of using Romanian monolingual Transformer-based \cite{vaswani2017attention} language models together with merging all the datasets for each language presented at the competition in a single corpus and then fine-tuning several multilingual language models on it." />
        </attvalues>
      </edge>
      <edge source="2304.11350" target="1911.02116" id="11475">
        <attvalues>
          <attvalue for="5" value=" Our experiments show that by employing these two algorithms, the results of the cross-lingual robustly optimized BERT approach (XLM-RoBERTa) \cite{conneau2020unsupervised} improve by 2." />
        </attvalues>
      </edge>
      <edge source="2304.11350" target="2009.08712" id="11476">
        <attvalues>
          <attvalue for="5" value=" Additionally, we report state-of-the-art (SOTA) results with the monolingual training of Romanian Bidirectional Encoder Representations from Transformer (RoBERT) \cite{dumitrescu2020birth} in comparison with the results obtained at the PARSEME 1." />
        </attvalues>
      </edge>
      <edge source="2305.15183" target="2211.05166" id="11477">
        <attvalues>
          <attvalue for="5" value=" Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text \cite{bryant2022grammatical}." />
        </attvalues>
      </edge>
      <edge source="2305.15183" target="1910.13461" id="11478">
        <attvalues>
          <attvalue for="5" value=" The first is treating GEC as a low-resource machine translation task \cite{yuan2016grammatical}, where sequence-to-sequence models like BART \cite{lewis2020bart} are used." />
        </attvalues>
      </edge>
      <edge source="2305.15183" target="2204.10994" id="11479">
        <attvalues>
          <attvalue for="5" value=" In CGEC, \cite{li-etal-2018-hybrid}, \cite{liang-etal-2020-bert} and \cite{zhang-etal-2022-mucgec} ensemble their models by majority voting on edits and achieve considerable improvement." />
        </attvalues>
      </edge>
      <edge source="2305.15183" target="1603.09727" id="11480">
        <attvalues>
          <attvalue for="5" value=" Besides, \cite{DBLP:journals/corr/XieAAJN16} adopt language models to improve neural language correction, following whom \cite{junczys-dowmunt-etal-2018-approaching} ensemble their GEC models using a language model probability." />
        </attvalues>
      </edge>
      <edge source="2305.15183" target="1706.03762" id="11482">
        <attvalues>
          <attvalue for="5" value=" Today, transformer-based \cite{vaswani2017attention} Pre-trained Language Models (PLMs) have been in predominant use in NLP." />
        </attvalues>
      </edge>
      <edge source="2401.09615" target="1802.05365" id="11483">
        <attvalues>
          <attvalue for="5" value=" The preference for LLMs can be attributed to their performance gains in a wide variety of NLP tasks, including question answering, textual entailment, sentiment analysis, and commonsense reasoning \cite{peters_deep_2018, devlin_bert_2019, sap_commonsense_2020}." />
        </attvalues>
      </edge>
      <edge source="2401.09615" target="2111.09543" id="11484">
        <attvalues>
          <attvalue for="5" value=" As LLMs scale up, their performance gains not only compete with but also exceed human performance on language understanding benchmarks \cite{he2022debertav3, chowdhery2023palm}." />
        </attvalues>
      </edge>
      <edge source="2401.09615" target="2106.15195" id="11485">
        <attvalues>
          <attvalue for="5" value=" Whether such performance gains are meaningful depends on the quality of the evaluation metrics and the relevance of benchmarking schemes \cite{marie2021scientific, bommasani2021opportunities}." />
        </attvalues>
      </edge>
      <edge source="2401.09615" target="1907.07355" id="11486">
        <attvalues>
          <attvalue for="5" value=" Recent works have shown that LLMs tend to learn shortcuts based on statistical cues eg the word ``not&quot; \cite{niven_probing_2019}, keywords \cite{moon_masker_2021, du_towards_2021} and cues related to language variations \cite{nguyen_learning_2021} to make predictions." />
        </attvalues>
      </edge>
      <edge source="2401.09615" target="2012.09392" id="11487">
        <attvalues>
          <attvalue for="5" value=" This behavior, also known as shortcut learning, leads the model to learn non-generalizable decision rules that do not perform well on out-of-distribution data \cite{moon_masker_2021, du_shortcut_2022}, but continues to give state-of-the-art results on independent identically distributed (IID) samples." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="1808.10792" id="11488">
        <attvalues>
          <attvalue for="5" value=" Generating abstractive summaries can help readers quickly grasp the main topics, yet prior work has mostly focused on short texts (containing hundreds of words), eg, news articles~\cite{gehrmann2018bottom,liu-lapata-2019-text,DBLP:journals/corr/abs-1912-08777}." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="1912.08777" id="11490">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art systems~\cite{lewis-etal-2020-bart,DBLP:journals/corr/abs-1912-08777} are built upon Transformer~\cite{NIPS2017_3f5ee243}, which uses attentions to compute pairwise relations between tokens." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="1910.13461" id="11491">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art systems~\cite{lewis-etal-2020-bart,DBLP:journals/corr/abs-1912-08777} are built upon Transformer~\cite{NIPS2017_3f5ee243}, which uses attentions to compute pairwise relations between tokens." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="1706.03762" id="11492">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art systems~\cite{lewis-etal-2020-bart,DBLP:journals/corr/abs-1912-08777} are built upon Transformer~\cite{NIPS2017_3f5ee243}, which uses attentions to compute pairwise relations between tokens." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="2006.04768" id="11493">
        <attvalues>
          <attvalue for="5" value=" First, when summarizing documents of the same length, Hepos attention yields significantly better ROUGE scores than a non-trivial comparison that projects attentions into low-rank space~\cite{wang2020linformer}." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="2004.05150" id="11494">
        <attvalues>
          <attvalue for="5" value=" Solutions have been proposed to reduce the calculation of encoder self-attentions~\cite{wang2020linformer,zaheer2020big} by selectively attending to neighboring tokens~\cite{beltagy2020longformer,child2019generating} or relevant words~\cite{Kitaev2020Reformer:,tay2020sparse}." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="1904.10509" id="11495">
        <attvalues>
          <attvalue for="5" value=" Solutions have been proposed to reduce the calculation of encoder self-attentions~\cite{wang2020linformer,zaheer2020big} by selectively attending to neighboring tokens~\cite{beltagy2020longformer,child2019generating} or relevant words~\cite{Kitaev2020Reformer:,tay2020sparse}." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="2001.04451" id="11496">
        <attvalues>
          <attvalue for="5" value=" Solutions have been proposed to reduce the calculation of encoder self-attentions~\cite{wang2020linformer,zaheer2020big} by selectively attending to neighboring tokens~\cite{beltagy2020longformer,child2019generating} or relevant words~\cite{Kitaev2020Reformer:,tay2020sparse}." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="2002.11296" id="11497">
        <attvalues>
          <attvalue for="5" value=" Solutions have been proposed to reduce the calculation of encoder self-attentions~\cite{wang2020linformer,zaheer2020big} by selectively attending to neighboring tokens~\cite{beltagy2020longformer,child2019generating} or relevant words~\cite{Kitaev2020Reformer:,tay2020sparse}." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="2005.00661" id="11498">
        <attvalues>
          <attvalue for="5" value=" However, training on curtailed content further aggravates ``hallucination'' in existing abstractive models~\cite{maynez-etal-2020-faithfulness}." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="1804.05685" id="11499">
        <attvalues>
          <attvalue for="5" value="4$k words) and summaries ($553$ words) than existing datasets, such as PubMed and arXiv~\cite{cohan-etal-2018-discourse} (see Table~\ref{tab:basic_stat}); (2) Salient content is spread throughout the documents, as opposed to cases where summary-worthy words are more heavily concentrated in specific parts of the document." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="1906.00318" id="11500">
        <attvalues>
          <attvalue for="5" value=" We further propose a new evaluation metric for faithfulness, inspired by APES~\cite{eyal-etal-2019-question}, a fill-in-the-blank QA metric for summary evaluation." />
        </attvalues>
      </edge>
      <edge source="2104.02112" target="1910.12840" id="11501">
        <attvalues>
          <attvalue for="5" value=" It is shown to be better correlated with human judgment than the original metric and an entailment-based scorer~\cite{kryscinski-etal-2020-evaluating}." />
        </attvalues>
      </edge>
      <edge source="2312.14345" target="1804.11192" id="11526">
        <attvalues>
          <attvalue for="5" value=" Explainable recommender systems have gained significant attention in recent years due to the need for transparency and interpretability in AI-driven decision-making processes \cite{zhang2020explainable}." />
        </attvalues>
      </edge>
      <edge source="2312.14345" target="2102.02503" id="11528">
        <attvalues>
          <attvalue for="5" value=" These challenges emphasize the necessity of addressing the limitations of generic LLMs to ensure reliable, personalized, and responsible explainable recommender systems \cite{tamkin2021understanding}." />
        </attvalues>
      </edge>
      <edge source="2312.14345" target="2005.14165" id="11529">
        <attvalues>
          <attvalue for="5" value=" In \ref{sec:model}, we provide more details about how we utilize few-shot prompting \cite{brown2020language} as a means for aspect extraction and how chain-of-thought reasoning \cite{wei2022chain} is used as one of the main components of our framework." />
        </attvalues>
      </edge>
      <edge source="2312.14345" target="2201.11903" id="11530">
        <attvalues>
          <attvalue for="5" value=" In \ref{sec:model}, we provide more details about how we utilize few-shot prompting \cite{brown2020language} as a means for aspect extraction and how chain-of-thought reasoning \cite{wei2022chain} is used as one of the main components of our framework." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2110.14378" id="11531">
        <attvalues>
          <attvalue for="5" value=" Developing models with multi-modal capabilities is well recognized as a path forward toward artificial general intelligence \cite{fei2022towards, huang2021makes}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2002.05709" id="11533">
        <attvalues>
          <attvalue for="5" value=" Contrastive learning-based models have demonstrated exceptional adaptability across a range of related tasks, such as image classification \cite{chen2020simple, he2020momentum}, natural language processing \cite{gao2021simcse} and speech processing \cite{ravanelli2020multi}, making them a crucial area of research in multi-modal machine learning." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2104.08821" id="11534">
        <attvalues>
          <attvalue for="5" value=" Contrastive learning-based models have demonstrated exceptional adaptability across a range of related tasks, such as image classification \cite{chen2020simple, he2020momentum}, natural language processing \cite{gao2021simcse} and speech processing \cite{ravanelli2020multi}, making them a crucial area of research in multi-modal machine learning." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2111.09734" id="11536">
        <attvalues>
          <attvalue for="5" value=" It stands out as a groundbreaking vision-language model, facilitating essential tasks such as formulating image captions \cite{mokady2021clipcap} and generating images from text \cite{rombach2022high}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2112.10752" id="11537">
        <attvalues>
          <attvalue for="5" value=" It stands out as a groundbreaking vision-language model, facilitating essential tasks such as formulating image captions \cite{mokady2021clipcap} and generating images from text \cite{rombach2022high}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2109.14084" id="11538">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2106.11097" id="11539">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2205.00823" id="11540">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2212.05051" id="11541">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2201.04850" id="11542">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2208.12415" id="11543">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2106.13043" id="11544">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2211.06687" id="11545">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2305.11834" id="11546">
        <attvalues>
          <attvalue for="5" value=" Similar work on contrastive learning has been extended to other multi-modal domains, such as video-language \cite{xu2021videoclip,fang2021clip2video,zhao2022centerclip,luo2022clip4clip,cheng2023vindlu,ge2022bridging} and audio-language models \cite{elizalde2023clap,huang2022mulan,guzhov2022audioclip,wu2023large,deshmukh2023pengi,wu2023audio}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2303.10667" id="11547">
        <attvalues>
          <attvalue for="5" value=" \cite{wu2023audio} reveal that current audio-language models (ALMs) are biased towards retrieving nouns and verbs, neglecting complete sentence context." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2102.12092" id="11548">
        <attvalues>
          <attvalue for="5" value=" These multi-context encoders integrate well with other downstream models, such as retrieval and open-ended generation models \cite{ramesh2021zero, li2022blip, yuan2021florence, singh2022flava}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2204.03162" id="11549">
        <attvalues>
          <attvalue for="5" value=" Recent studies highlight limitations in models like CLIP's ability to understand language reasoning despite extensive training datasets \cite{thrush2022winoground, ma2023crepe, yuksekgonul2022and}, suggesting that the focus of contrastive pre-training on retrieval tasks allows such models to excel in retrieval-based benchmarks without a robust compositional grasp." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2310.08753" id="11550">
        <attvalues>
          <attvalue for="5" value=" In response, \cite{ghosh2023compa} critique existing audio-retrieval benchmarks, revealing ALMs' superficial success without true compositional understanding." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2301.02074" id="11551">
        <attvalues>
          <attvalue for="5" value=" Inspired by the work of \cite{bagad2023test} on instilling time in video-language models, we aim to improve ALMs' understanding of the relationship between modalities." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2402.01383" id="11552">
        <attvalues>
          <attvalue for="5" value=" These methods have demonstrated limitations and biases \cite{gao2024llm,jones2023cognitive,stureborg2024inconsistent,wang2023notfair}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2309.17012" id="11553">
        <attvalues>
          <attvalue for="5" value=" These methods have demonstrated limitations and biases \cite{gao2024llm,jones2023cognitive,stureborg2024inconsistent,wang2023notfair}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="2305.17926" id="11555">
        <attvalues>
          <attvalue for="5" value=" These methods have demonstrated limitations and biases \cite{gao2024llm,jones2023cognitive,stureborg2024inconsistent,wang2023notfair}." />
        </attvalues>
      </edge>
      <edge source="2408.09269" target="1712.06087" id="11556">
        <attvalues>
          <attvalue for="5" value=" Previous models have been evaluated for their test of time \cite{Shocher_2018_CVPR, Bau_2019, Kundu_2020_CVPR, Huang_2020_NeurIPS, Sun_2020_ICML, Liu_2021_NeurIPS}, but these evaluations never try to test the model's capability of general language and time--understanding." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1606.09274" id="11570">
        <attvalues>
          <attvalue for="5" value=" To improve the efficiency of NMT training, most of the studies focus on reducing the number of parameters in the model \cite{DBLP:journals/corr/SeeLM16,DBLP:journals/corr/CregoKKRYSABCDE16,DBLP:journals/corr/HubaraCSEB16} and implementing parallelism in the data or in the model \cite{DBLP:journals/corr/WuSCLNMKCGMKSJL16,DBLP:journals/corr/KalchbrennerESO16,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1610.05540" id="11571">
        <attvalues>
          <attvalue for="5" value=" To improve the efficiency of NMT training, most of the studies focus on reducing the number of parameters in the model \cite{DBLP:journals/corr/SeeLM16,DBLP:journals/corr/CregoKKRYSABCDE16,DBLP:journals/corr/HubaraCSEB16} and implementing parallelism in the data or in the model \cite{DBLP:journals/corr/WuSCLNMKCGMKSJL16,DBLP:journals/corr/KalchbrennerESO16,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1609.07061" id="11572">
        <attvalues>
          <attvalue for="5" value=" To improve the efficiency of NMT training, most of the studies focus on reducing the number of parameters in the model \cite{DBLP:journals/corr/SeeLM16,DBLP:journals/corr/CregoKKRYSABCDE16,DBLP:journals/corr/HubaraCSEB16} and implementing parallelism in the data or in the model \cite{DBLP:journals/corr/WuSCLNMKCGMKSJL16,DBLP:journals/corr/KalchbrennerESO16,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1609.08144" id="11573">
        <attvalues>
          <attvalue for="5" value=" To improve the efficiency of NMT training, most of the studies focus on reducing the number of parameters in the model \cite{DBLP:journals/corr/SeeLM16,DBLP:journals/corr/CregoKKRYSABCDE16,DBLP:journals/corr/HubaraCSEB16} and implementing parallelism in the data or in the model \cite{DBLP:journals/corr/WuSCLNMKCGMKSJL16,DBLP:journals/corr/KalchbrennerESO16,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1610.10099" id="11574">
        <attvalues>
          <attvalue for="5" value=" To improve the efficiency of NMT training, most of the studies focus on reducing the number of parameters in the model \cite{DBLP:journals/corr/SeeLM16,DBLP:journals/corr/CregoKKRYSABCDE16,DBLP:journals/corr/HubaraCSEB16} and implementing parallelism in the data or in the model \cite{DBLP:journals/corr/WuSCLNMKCGMKSJL16,DBLP:journals/corr/KalchbrennerESO16,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1706.03762" id="11576">
        <attvalues>
          <attvalue for="5" value=" To improve the efficiency of NMT training, most of the studies focus on reducing the number of parameters in the model \cite{DBLP:journals/corr/SeeLM16,DBLP:journals/corr/CregoKKRYSABCDE16,DBLP:journals/corr/HubaraCSEB16} and implementing parallelism in the data or in the model \cite{DBLP:journals/corr/WuSCLNMKCGMKSJL16,DBLP:journals/corr/KalchbrennerESO16,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1707.09533" id="11577">
        <attvalues>
          <attvalue for="5" value=" Specifically, \cite{DBLP:journals/corr/KocmiB17aa} empirically investigated curriculum learning based on the sentence length and word rank." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1708.00712" id="11578">
        <attvalues>
          <attvalue for="5" value=" All of these criteria \cite{wang-EtAl:2017:Short3,wang-EtAl:2017:EMNLP20174,vanderwees-bisazza-monz:2017:EMNLP2017} are calculated before performing the NMT training based on the domain information and are fixed while performing the complete procedure." />
        </attvalues>
      </edge>
      <edge source="1805.00178" target="1612.06138" id="11579">
        <attvalues>
          <attvalue for="5" value=" \cite{zhang-EtAl:2017:I17-2} adopted the sentence-level training cost as a dynamic criterion to gradually fine-tune the NMT training." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1709.06671" id="11580">
        <attvalues>
          <attvalue for="5" value=" Given independently trained multiple word representations (aka embeddings) learnt using diverse algorithms and lexical resources, word meta-embedding (ME) learning methods~\cite{Yin:ACL:2016,Bao:COLING:2018,Bollegala:IJCAI:2018,wu2020task,He:2020,jawanpuria-etal-2020-learning,Coates:NAACL:2018} attempt to learn more accurate and wide-coverage word embeddings." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1804.05262" id="11583">
        <attvalues>
          <attvalue for="5" value=" ME learning methods use different techniques such as concatenation~\cite{Yin:ACL:2016}, orthogonal projections~\cite{jawanpuria-etal-2020-learning,He:2020} and averaging~\cite{Coates:NAACL:2018} after applying zero-padding to the sources with smaller dimensionalities as necessary to handle source embeddings with different dimensionalities." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1103.0398" id="11584">
        <attvalues>
          <attvalue for="5" value=" For example, context-insensitive static word embedding methods~\cite{Dhillon:2015,Mnih:HLBL:NIPS:2008,Collobert:2011,Huang:ACL:2012,Milkov:2013,Pennington:EMNLP:2014} represent a word by a single vector that does not vary depending on the context in which the word occurs." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1802.05365" id="11585">
        <attvalues>
          <attvalue for="5" value=" On the other hand, contextualised word embedding methods~\cite{Elmo,BERT,XLNET,ALBERT,RoBERTa} represent the same word with different embeddings in its different contexts." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1909.11942" id="11588">
        <attvalues>
          <attvalue for="5" value=" On the other hand, contextualised word embedding methods~\cite{Elmo,BERT,XLNET,ALBERT,RoBERTa} represent the same word with different embeddings in its different contexts." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1907.11692" id="11589">
        <attvalues>
          <attvalue for="5" value=" On the other hand, contextualised word embedding methods~\cite{Elmo,BERT,XLNET,ALBERT,RoBERTa} represent the same word with different embeddings in its different contexts." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1511.06438" id="11590">
        <attvalues>
          <attvalue for="5" value="\\ Source embeddings can be trained using different linguistic resources such as text corpora or dictionaries~\cite{tissier-gravier-habrard:2017:EMNLP2017,Alsuhaibani:AKBC:2019,Bollegala:AAAI:2016}." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1812.04224" id="11591">
        <attvalues>
          <attvalue for="5" value=" Prior work~\cite{Yin:2018,Levy:TACL:2015} studying word embeddings have shown that the performance of a static word embedding is directly influenced by its dimensionality." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1704.01419" id="11592">
        <attvalues>
          <attvalue for="5" value=" An ensemble typically helps to cancel out noise in individual models, while reinforcing the useful patterns repeated in multiple models~\cite{muromagi-etal-2017-linear}." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1502.03520" id="11593">
        <attvalues>
          <attvalue for="5" value=" Although there are some theoretical work studying word embedding learning~\cite{Arora:TACL:2016,All-but-Top,Bollegala:AAAI:2018}, the theoretical analysis of ME learning has been under-developed, with the exception of concatenated meta-embeddings~\cite{Bollegala:IJCAIa:2022}." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1709.01199" id="11595">
        <attvalues>
          <attvalue for="5" value=" Although there are some theoretical work studying word embedding learning~\cite{Arora:TACL:2016,All-but-Top,Bollegala:AAAI:2018}, the theoretical analysis of ME learning has been under-developed, with the exception of concatenated meta-embeddings~\cite{Bollegala:IJCAIa:2022}." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="2204.12386" id="11596">
        <attvalues>
          <attvalue for="5" value=" Although there are some theoretical work studying word embedding learning~\cite{Arora:TACL:2016,All-but-Top,Bollegala:AAAI:2018}, the theoretical analysis of ME learning has been under-developed, with the exception of concatenated meta-embeddings~\cite{Bollegala:IJCAIa:2022}." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1804.03235" id="11597">
        <attvalues>
          <attvalue for="5" value=" ME learning can also been seen as related to model distillation~\cite{46642,Hinton:2015} where we must learn a simpler student model from a more complicated teacher model." />
        </attvalues>
      </edge>
      <edge source="2204.11660" target="1503.02531" id="11598">
        <attvalues>
          <attvalue for="5" value=" ME learning can also been seen as related to model distillation~\cite{46642,Hinton:2015} where we must learn a simpler student model from a more complicated teacher model." />
        </attvalues>
      </edge>
      <edge source="2408.12226" target="2102.12971" id="11599">
        <attvalues>
          <attvalue for="5" value=" Recent advancements include the implementation of transformer-based models like BERT \cite{rama2021pre} for evaluating CEFR levels of sentences, and GPT variants \cite{gpt4} for assessing essays written by L2 English learners \cite{yancey2023rating}." />
        </attvalues>
      </edge>
      <edge source="2408.12226" target="2303.08774" id="11600">
        <attvalues>
          <attvalue for="5" value=" Recent advancements include the implementation of transformer-based models like BERT \cite{rama2021pre} for evaluating CEFR levels of sentences, and GPT variants \cite{gpt4} for assessing essays written by L2 English learners \cite{yancey2023rating}." />
        </attvalues>
      </edge>
      <edge source="2408.12226" target="2306.05685" id="11601">
        <attvalues>
          <attvalue for="5" value=" As a baseline, we first investigated the ability of leading LLMs, which are highly ranked on the LMSYS leaderboard \cite{lmsys}, to directly evaluate candidate responses by leveraging their intrinsic knowledge and prompt engineering." />
        </attvalues>
      </edge>
      <edge source="2402.07158" target="2303.08774" id="11602">
        <attvalues>
          <attvalue for="5" value=" Although more recent LLMs~\cite{openai2023gpt4,touvron2023llama,jiang2024mixtral} have the capability to do data analysis and even data summarization and representation, the ability to connect to external data sources, algorithms and specialized interfaces to LLMs~\cite{nexusflow2023} adds additional flexibility to LLMs by enabling it to perform tasks that involves analysis of domain specific real time data, or even the possibility to perform tasks that are still beyond LLM's capabilities." />
        </attvalues>
      </edge>
      <edge source="2402.07158" target="2005.11401" id="11603">
        <attvalues>
          <attvalue for="5" value=" However, in recent months many have witnessed a transition towards more sophisticated systems such as Retrieval-Augmented Generation (RAG)~\cite{lewis2021retrievalaugmented} and AI Agents~\cite{wolfram2023}." />
        </attvalues>
      </edge>
      <edge source="2402.07158" target="2307.09288" id="11604">
        <attvalues>
          <attvalue for="5" value=" Although more recent LLMs~\cite{openai2023gpt4,touvron2023llama,jiang2024mixtral} have the capability to do data analysis and even data summarization and representation, the ability to connect to external data sources, algorithms and specialized interfaces to LLMs~\cite{nexusflow2023} adds additional flexibility to LLMs by enabling it to perform tasks that involves analysis of domain specific real time data, or even the possibility to perform tasks that are still beyond LLM's capabilities." />
        </attvalues>
      </edge>
      <edge source="2402.07158" target="2401.04088" id="11605">
        <attvalues>
          <attvalue for="5" value=" Although more recent LLMs~\cite{openai2023gpt4,touvron2023llama,jiang2024mixtral} have the capability to do data analysis and even data summarization and representation, the ability to connect to external data sources, algorithms and specialized interfaces to LLMs~\cite{nexusflow2023} adds additional flexibility to LLMs by enabling it to perform tasks that involves analysis of domain specific real time data, or even the possibility to perform tasks that are still beyond LLM's capabilities." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="2001.08361" id="11606">
        <attvalues>
          <attvalue for="5" value=" Building upon the established correlation between language model performance and computational capacity \cite{kaplan2020scaling}, there has emerged an undeniable trend towards the adoption of ever-larger language models across a diverse range of NLP applications." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="1907.11692" id="11608">
        <attvalues>
          <attvalue for="5" value=" Despite the ongoing popularity of compact encoders such as BERT \cite{devlin-etal-2019-bert} and RoBERTa \cite{liu2019roberta}, there is a growing inclination to leverage the capabilities of recent, larger language models, eg, LLaMA-2 \cite{touvron2023llama}, even breaking from the conventional roles of encoders and decoders." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="2307.09288" id="11609">
        <attvalues>
          <attvalue for="5" value=" Despite the ongoing popularity of compact encoders such as BERT \cite{devlin-etal-2019-bert} and RoBERTa \cite{liu2019roberta}, there is a growing inclination to leverage the capabilities of recent, larger language models, eg, LLaMA-2 \cite{touvron2023llama}, even breaking from the conventional roles of encoders and decoders." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="1908.10084" id="11610">
        <attvalues>
          <attvalue for="5" value=" Consequently, the enduring challenge of finding a balance between performance and computational cost---a persistent issue in sentence representation learning \cite{reimers-gurevych-2019-sentence}---continues to be elusive." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="2106.07345" id="11611">
        <attvalues>
          <attvalue for="5" value=" In recent years, there has been a marked improvement in the quality of sentence embeddings, a progress primarily driven by the advent of contrastive learning frameworks (\cite{kim-etal-2021-self,gao-etal-2021-simcse,chuang-etal-2022-diffcse}; inter alia)." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="2104.08821" id="11612">
        <attvalues>
          <attvalue for="5" value=" In recent years, there has been a marked improvement in the quality of sentence embeddings, a progress primarily driven by the advent of contrastive learning frameworks (\cite{kim-etal-2021-self,gao-etal-2021-simcse,chuang-etal-2022-diffcse}; inter alia)." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="2204.10298" id="11613">
        <attvalues>
          <attvalue for="5" value=" In recent years, there has been a marked improvement in the quality of sentence embeddings, a progress primarily driven by the advent of contrastive learning frameworks (\cite{kim-etal-2021-self,gao-etal-2021-simcse,chuang-etal-2022-diffcse}; inter alia)." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="2210.07316" id="11614">
        <attvalues>
          <attvalue for="5" value=" However, since the performance of these embeddings is generally evaluated based on their ability to encapsulate the overall meaning of the corresponding sentences---as measured by benchmarks like STS-B \cite{agirre-etal-2012-semeval,cer-etal-2017-semeval} and MTEB \cite{muennighoff-etal-2023-mteb}, it remains uncertain whether they adequately capture information relating to the various aspects of the source sentences." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="2305.15093" id="11615">
        <attvalues>
          <attvalue for="5" value=" In the literature, three prevalent approaches have been established for constructing conditioned representations \cite{deshpande-etal-2023-c}, particularly in estimating their similarity (see Figure \ref{fig:fig_architectures4})." />
        </attvalues>
      </edge>
      <edge source="2403.09490" target="1609.09106" id="11616">
        <attvalues>
          <attvalue for="5" value=" In this work, we present Hyper-CL, a method that integrates Hypernetworks \cite{ha2017hypernetworks} with Contrastive Learning to efficiently compute conditioned sentence representations and their similarity." />
        </attvalues>
      </edge>
      <edge source="2403.08258" target="1508.04395" id="11617">
        <attvalues>
          <attvalue for="5" value=" There are three popular E2E models for ASR tasks: Connectionist Temporal Classification~(CTC)~\cite{graves2006connectionist}, Attention-based Encoder-Decoder (AED)~\cite{AED,LAS} and Recurrent Neural Network Transducer~(RNN-T)~\cite{graves2012sequence}." />
        </attvalues>
      </edge>
      <edge source="2403.08258" target="1211.3711" id="11618">
        <attvalues>
          <attvalue for="5" value=" There are three popular E2E models for ASR tasks: Connectionist Temporal Classification~(CTC)~\cite{graves2006connectionist}, Attention-based Encoder-Decoder (AED)~\cite{AED,LAS} and Recurrent Neural Network Transducer~(RNN-T)~\cite{graves2012sequence}." />
        </attvalues>
      </edge>
      <edge source="2403.08258" target="2005.08100" id="11619">
        <attvalues>
          <attvalue for="5" value=" The vanilla Conformer~\cite{conformer} encoder begins with two convolution layers with stride 2 respectively along temporal dimension to reduce the input sequence length by 4 times." />
        </attvalues>
      </edge>
      <edge source="2403.08258" target="2206.00888" id="11621">
        <attvalues>
          <attvalue for="5" value=" Squeezeformer~\cite{squeezeformer} combines downsampling with temporal U-Net structure." />
        </attvalues>
      </edge>
      <edge source="2403.08258" target="2208.07657" id="11622">
        <attvalues>
          <attvalue for="5" value=" A similar strategy was also used for deeper downsampling in Uconv-Conformer~\cite{Uconvconformer}." />
        </attvalues>
      </edge>
      <edge source="2403.08258" target="2210.16481" id="11624">
        <attvalues>
          <attvalue for="5" value=" For example, in~\cite{wang2023accelerating}, they proposed to do frame reduction in the middle of RNN-T encoder using co-trained CTC guidance." />
        </attvalues>
      </edge>
      <edge source="2403.08258" target="2104.02882" id="11625">
        <attvalues>
          <attvalue for="5" value=" As for RNN-T and CTC model, blank symbols can be ignored during decoding, which can accelerate the decoding process \cite{blankregularized,wang2023accelerating,FSR,factorized}." />
        </attvalues>
      </edge>
      <edge source="2403.08258" target="2110.01500" id="11626">
        <attvalues>
          <attvalue for="5" value=" As for RNN-T and CTC model, blank symbols can be ignored during decoding, which can accelerate the decoding process \cite{blankregularized,wang2023accelerating,FSR,factorized}." />
        </attvalues>
      </edge>
      <edge source="2204.05211" target="1706.03762" id="11627">
        <attvalues>
          <attvalue for="5" value=" Second, the 2017~breakthrough that was the transformer architecture \cite{VaswaniSPUJGKP17} has led to the so-called ImageNet moment of Natural Language Processing \cite{ruder2018nlpimagenet} and brought about unprecedented progress in transfer-learning \cite{raffel2020t5}, few-shot learning \cite{schick-schutze-2021-just}, zero-shot learning \cite{sanh2021multitask}, and prompt-based learning \cite{le-scao-rush-2021-many} for natural language." />
        </attvalues>
      </edge>
      <edge source="2204.05211" target="2009.07118" id="11629">
        <attvalues>
          <attvalue for="5" value=" Second, the 2017~breakthrough that was the transformer architecture \cite{VaswaniSPUJGKP17} has led to the so-called ImageNet moment of Natural Language Processing \cite{ruder2018nlpimagenet} and brought about unprecedented progress in transfer-learning \cite{raffel2020t5}, few-shot learning \cite{schick-schutze-2021-just}, zero-shot learning \cite{sanh2021multitask}, and prompt-based learning \cite{le-scao-rush-2021-many} for natural language." />
        </attvalues>
      </edge>
      <edge source="2204.05211" target="2110.08207" id="11630">
        <attvalues>
          <attvalue for="5" value=" To advance research in this area, an increasing number of datasets have been created to support the development and evaluation of NER approaches in historical text~\cite{Neudecker2016AnOC, ehrmann_extended_2020,ehrmann_maud_2022_6089968} In this paper, we examine the zero-shot abilities of T0---a prompt-based LLM developed as part of the BigScience project for open research \cite{sanh2021multitask}---on the challenging task of historical NER." />
        </attvalues>
      </edge>
      <edge source="2204.05211" target="2103.08493" id="11631">
        <attvalues>
          <attvalue for="5" value=" Second, the 2017~breakthrough that was the transformer architecture \cite{VaswaniSPUJGKP17} has led to the so-called ImageNet moment of Natural Language Processing \cite{ruder2018nlpimagenet} and brought about unprecedented progress in transfer-learning \cite{raffel2020t5}, few-shot learning \cite{schick-schutze-2021-just}, zero-shot learning \cite{sanh2021multitask}, and prompt-based learning \cite{le-scao-rush-2021-many} for natural language." />
        </attvalues>
      </edge>
      <edge source="2204.05211" target="2107.13586" id="11632">
        <attvalues>
          <attvalue for="5" value=" Third, the growing popularity of prompt-based methods \cite{LiuPengfei2021PPaP} has resulted in a new paradigm for training and fine-tuning Large Language Models~(LLM) as well as novel applications in Named Entity Recognition~(NER)~\cite{QaNER}." />
        </attvalues>
      </edge>
      <edge source="2204.05211" target="2109.11406" id="11633">
        <attvalues>
          <attvalue for="5" value=" However, applying NER to historical texts poses a number of challenges, including those due to errors in Optical Character Recognition (OCR)~\cite{Ehrmann2021NamedER,Hamdi2019AnAO,Boros2020AlleviatingDE} and domain transfer~\cite{Blouin_Favre_Auguste_Henriot_2021}." />
        </attvalues>
      </edge>
      <edge source="2204.05211" target="2104.05240" id="11634">
        <attvalues>
          <attvalue for="5" value=" To better contextualize the results of our experiments, we also run zero-shot prompt-based probing \cite{zhong-etal-2021-factual} to assess T0's broader ability of extracting factual knowledge about two key factors in our experiment, that is, language variation and historical variation in the dataset." />
        </attvalues>
      </edge>
      <edge source="2311.10804" target="2106.06103" id="11635">
        <attvalues>
          <attvalue for="5" value=" Significant progress has been made in the development of Text-to-Speech (TTS) systems, with models such as VITS \cite{Kim2021ConditionalText-to-Speech} achieving a mean opinion score comparable to that of genuine speech recordings." />
        </attvalues>
      </edge>
      <edge source="2311.10804" target="2006.11239" id="11636">
        <attvalues>
          <attvalue for="5" value=" In this paper, we present an exploratory study in which we enhance the VITS model with expressiveness control by adding a Denoising Diffusion Model (DDM) \cite{Ho2020DenoisingModels} conditioned on joint audio/text embeddings such as CLAP embeddings \cite{Wu2023Large-ScaleAugmentation} to alter the latent VITS encodings." />
        </attvalues>
      </edge>
      <edge source="2311.10804" target="2211.06687" id="11637">
        <attvalues>
          <attvalue for="5" value=" In this paper, we present an exploratory study in which we enhance the VITS model with expressiveness control by adding a Denoising Diffusion Model (DDM) \cite{Ho2020DenoisingModels} conditioned on joint audio/text embeddings such as CLAP embeddings \cite{Wu2023Large-ScaleAugmentation} to alter the latent VITS encodings." />
        </attvalues>
      </edge>
      <edge source="2311.10804" target="2204.03458" id="11638">
        <attvalues>
          <attvalue for="5" value=" We chose DDM because these models are known to be easy to condition and are currently state-of-the-art in many computer vision tasks \cite{Ho2022VideoModels, Hoogeboom2023SimpleImages, Rombach2021High-ResolutionModels}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="1412.5567" id="11639">
        <attvalues>
          <attvalue for="5" value=" For instance, chain model \cite{povey2016purely} incorporates Convolutional Neural Networks (CNNs) and Time Delay Neural Network (TDNNs), while DeepSpeech \cite{hannun2014deep} model utilizes Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs)." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="1706.03762" id="11640">
        <attvalues>
          <attvalue for="5" value=" Modern systems are leaning to even more sophisticated architectures such as Transformer \cite{vaswani2017attention} and Conformer \cite{gulati2020conformer}, coupled with sequence losses like Connectionist Temporal Classification (CTC) \cite{CTC} and Recurrent Neural Network Transducer (RNN-T) \cite{graves2012sequence}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2005.08100" id="11641">
        <attvalues>
          <attvalue for="5" value=" Modern systems are leaning to even more sophisticated architectures such as Transformer \cite{vaswani2017attention} and Conformer \cite{gulati2020conformer}, coupled with sequence losses like Connectionist Temporal Classification (CTC) \cite{CTC} and Recurrent Neural Network Transducer (RNN-T) \cite{graves2012sequence}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="1211.3711" id="11642">
        <attvalues>
          <attvalue for="5" value=" Modern systems are leaning to even more sophisticated architectures such as Transformer \cite{vaswani2017attention} and Conformer \cite{gulati2020conformer}, coupled with sequence losses like Connectionist Temporal Classification (CTC) \cite{CTC} and Recurrent Neural Network Transducer (RNN-T) \cite{graves2012sequence}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2303.01037" id="11644">
        <attvalues>
          <attvalue for="5" value=" From a system perspective, driven by the scaling law from language modeling research, large speech models have been developed such as OpenAI-Whisper \cite{radford2022robust}, and Google-USM \cite{zhang2023google}, pushing up the scale of ASR training by orders of magnitude." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2006.11477" id="11645">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, HuBERT \cite{hsu2021hubert}, WavLM \cite{chen2022wavlm}, and data2vec \cite{baevski2022data2vec}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2106.07447" id="11646">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, HuBERT \cite{hsu2021hubert}, WavLM \cite{chen2022wavlm}, and data2vec \cite{baevski2022data2vec}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2110.13900" id="11647">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, HuBERT \cite{hsu2021hubert}, WavLM \cite{chen2022wavlm}, and data2vec \cite{baevski2022data2vec}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2202.03555" id="11648">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec}, HuBERT \cite{hsu2021hubert}, WavLM \cite{chen2022wavlm}, and data2vec \cite{baevski2022data2vec}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="1909.09577" id="11650">
        <attvalues>
          <attvalue for="5" value=" Given the swift evolution of ASR technology, a variety of speech toolkits have been developed and open-sourced, such as HTK \cite{young2002htk}, Kaldi \cite{povey2011kaldi}, ESPnet \cite{watanabe2018espnet}, NeMo \cite{kuchaiev2019nemo}, SpeechBrain \cite{ravanelli2021speechbrain}, WeNet \cite{yao2021wenet}, and K2, offering comprehensive libraries and recipes to facilitate ASR research and development." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2106.04624" id="11651">
        <attvalues>
          <attvalue for="5" value=" Given the swift evolution of ASR technology, a variety of speech toolkits have been developed and open-sourced, such as HTK \cite{young2002htk}, Kaldi \cite{povey2011kaldi}, ESPnet \cite{watanabe2018espnet}, NeMo \cite{kuchaiev2019nemo}, SpeechBrain \cite{ravanelli2021speechbrain}, WeNet \cite{yao2021wenet}, and K2, offering comprehensive libraries and recipes to facilitate ASR research and development." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2102.01547" id="11652">
        <attvalues>
          <attvalue for="5" value=" Given the swift evolution of ASR technology, a variety of speech toolkits have been developed and open-sourced, such as HTK \cite{young2002htk}, Kaldi \cite{povey2011kaldi}, ESPnet \cite{watanabe2018espnet}, NeMo \cite{kuchaiev2019nemo}, SpeechBrain \cite{ravanelli2021speechbrain}, WeNet \cite{yao2021wenet}, and K2, offering comprehensive libraries and recipes to facilitate ASR research and development." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2104.11348" id="11653">
        <attvalues>
          <attvalue for="5" value=" adjcol However, the evaluation of ASR still remains challenging \cite{del2021earnings} \cite{gandhi2022esb}, because there exist various crucial subtleties and pitfalls that require non-trivial efforts to do right in practice, such as text normalization \cite{faria2022toward}." />
        </attvalues>
      </edge>
      <edge source="2403.08196" target="2206.06192" id="11655">
        <attvalues>
          <attvalue for="5" value=" adjcol However, the evaluation of ASR still remains challenging \cite{del2021earnings} \cite{gandhi2022esb}, because there exist various crucial subtleties and pitfalls that require non-trivial efforts to do right in practice, such as text normalization \cite{faria2022toward}." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2310.06825" id="11656">
        <attvalues>
          <attvalue for="5" value=" A significant trend in NLP involves the utilization of large language models (LLMs) such as LLama \cite{LLama}, Mistral \cite{mistral}, Falcon \cite{falcon}, GPT-NeoX \cite{black2022gptneox20b}, or Mixtral \cite{jiang2024mixtral}." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2311.16867" id="11657">
        <attvalues>
          <attvalue for="5" value=" A significant trend in NLP involves the utilization of large language models (LLMs) such as LLama \cite{LLama}, Mistral \cite{mistral}, Falcon \cite{falcon}, GPT-NeoX \cite{black2022gptneox20b}, or Mixtral \cite{jiang2024mixtral}." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2204.06745" id="11658">
        <attvalues>
          <attvalue for="5" value=" A significant trend in NLP involves the utilization of large language models (LLMs) such as LLama \cite{LLama}, Mistral \cite{mistral}, Falcon \cite{falcon}, GPT-NeoX \cite{black2022gptneox20b}, or Mixtral \cite{jiang2024mixtral}." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2401.04088" id="11659">
        <attvalues>
          <attvalue for="5" value=" A significant trend in NLP involves the utilization of large language models (LLMs) such as LLama \cite{LLama}, Mistral \cite{mistral}, Falcon \cite{falcon}, GPT-NeoX \cite{black2022gptneox20b}, or Mixtral \cite{jiang2024mixtral}." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2303.12712" id="11660">
        <attvalues>
          <attvalue for="5" value=" While LLMs offer impressive performance \cite{bubeck2023sparks}, their deployment is often hampered by hardware availability, cost, and latency bottlenecks." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2211.17192" id="11661">
        <attvalues>
          <attvalue for="5" value=" Several strategies, such as efficient decoding \cite{leviathan2023fast, ye-etal-2023-fid}, model recycling \cite{lester2022reducing}, and model size reduction \cite{dettmers2023qlora, ma2023llmpruner}, have been developed to streamline their use." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2208.05577" id="11662">
        <attvalues>
          <attvalue for="5" value=" Several strategies, such as efficient decoding \cite{leviathan2023fast, ye-etal-2023-fid}, model recycling \cite{lester2022reducing}, and model size reduction \cite{dettmers2023qlora, ma2023llmpruner}, have been developed to streamline their use." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2305.14314" id="11663">
        <attvalues>
          <attvalue for="5" value=" Several strategies, such as efficient decoding \cite{leviathan2023fast, ye-etal-2023-fid}, model recycling \cite{lester2022reducing}, and model size reduction \cite{dettmers2023qlora, ma2023llmpruner}, have been developed to streamline their use." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2105.10059" id="11664">
        <attvalues>
          <attvalue for="5" value=" Among these, knowledge distillation (KD) \cite{model_compression, distillation} a widely adopted technique \cite{distilbert, tinybert, small100, how_fish, token_not_all, cost_effective}, transferring the capabilities of large, complex teacher models into more manageable and smaller student models, tailored for specific tasks." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2106.06168" id="11666">
        <attvalues>
          <attvalue for="5" value=" In fact, recent research predominantly focuses on synthetic data fine-tuning \cite{tacl_a_00492, kramchaninova-defauw-2022-synthetic, instructedfinetuning}, rather than refining logits loss in the black box approach." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2310.19019" id="11667">
        <attvalues>
          <attvalue for="5" value=" Thus far, the primary method for KD with decoder models is to use the text generated by the teacher model \cite{how_fish, step_step}, and if possible, when students and teachers models belong to the same family, improving KD by employing output-logit distillation with Kullback–Leibler divergence \cite{small100, token_not_all, minilm}." />
        </attvalues>
      </edge>
      <edge source="2402.12030" target="2210.11621" id="11668">
        <attvalues>
          <attvalue for="5" value=" Thus far, the primary method for KD with decoder models is to use the text generated by the teacher model \cite{how_fish, step_step}, and if possible, when students and teachers models belong to the same family, improving KD by employing output-logit distillation with Kullback–Leibler divergence \cite{small100, token_not_all, minilm}." />
        </attvalues>
      </edge>
      <edge source="2303.01249" target="1905.11235" id="11713">
        <attvalues>
          <attvalue for="5" value=" With the widespread of end-to-end automatic speech recognition (ASR) frameworks \cite{dong2020cif, kahn2020self, miao2020transformer}, multilingual ASR has become a research hotspot." />
        </attvalues>
      </edge>
      <edge source="2303.01249" target="2112.06825" id="11714">
        <attvalues>
          <attvalue for="5" value=" The adapter-based modeling technique has been successfully used for domain adaptation in computer vision \cite{sung2022vl}, natural language processing \cite{pfeiffer2020unks}, and machine translation \cite{bapna2019simple}." />
        </attvalues>
      </edge>
      <edge source="2303.01249" target="2012.15562" id="11715">
        <attvalues>
          <attvalue for="5" value=" The adapter-based modeling technique has been successfully used for domain adaptation in computer vision \cite{sung2022vl}, natural language processing \cite{pfeiffer2020unks}, and machine translation \cite{bapna2019simple}." />
        </attvalues>
      </edge>
      <edge source="2303.01249" target="1909.05330" id="11717">
        <attvalues>
          <attvalue for="5" value=" In multilingual ASR, \cite{kannan2019large} investigated using adapter modules for nine Indian languages in an RNN-T model." />
        </attvalues>
      </edge>
      <edge source="2303.01249" target="2012.01687" id="11718">
        <attvalues>
          <attvalue for="5" value=" Furthermore, The Adapter-and-Adjust framework was introduced in \cite{winata2020adapt}, where both language-specific (LSA) and common adapters were applied to an encoder-decoder network." />
        </attvalues>
      </edge>
      <edge source="2303.01249" target="2006.11477" id="11719">
        <attvalues>
          <attvalue for="5" value="0 pre-trained model \cite{baevski2020wav2vec} in the training procedure." />
        </attvalues>
      </edge>
      <edge source="2303.01249" target="2107.05876" id="11720">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{zhou2022configurable} simply concatenated a one-hot vector to the input features of the encoder network, whereas \cite{zhu2020multilingual} concatenated LID to multi-head attention inputs in the transformer model." />
        </attvalues>
      </edge>
      <edge source="2303.01249" target="2101.00190" id="11721">
        <attvalues>
          <attvalue for="5" value=" This is inspired by the prefix-tuning method \cite{li2021prefix}." />
        </attvalues>
      </edge>
      <edge source="2404.08666" target="2211.15649" id="11722">
        <attvalues>
          <attvalue for="5" value=" The more recent work to examine the disparate sources utilized to construct new datasets is the study by \cite{yu-etal-2022-beyond}." />
        </attvalues>
      </edge>
      <edge source="2106.08686" target="1510.01032" id="11723">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art AWE models are trained using either contrastive objectives \cite{kamper+etal_icassp16, he+etal_iclr17} or reconstruction objectives \cite{kamper2015unsupervised, kamper2019truly}." />
        </attvalues>
      </edge>
      <edge source="2106.08686" target="1706.03818" id="11724">
        <attvalues>
          <attvalue for="5" value=" AWEs have been used in downstream applications including ASR \cite{bengio+heigold_interspeech14} and QbE search \cite{settle2017query, yuan+etal_interspeech18}." />
        </attvalues>
      </edge>
      <edge source="2106.08686" target="1611.04496" id="11725">
        <attvalues>
          <attvalue for="5" value=" Although previous studies have proposed to incorporate the pronunciation distance in the learning objective \cite{he+etal_iclr17, yang2019linguistically}, the reported performance showed no improvement on the word discrimination task, while the distance in the AWE space has shown only a weak correlation with orthographic similarity \cite{he+etal_iclr17}." />
        </attvalues>
      </edge>
      <edge source="2106.08686" target="2008.02888" id="11726">
        <attvalues>
          <attvalue for="5" value=" Since AWE models have been recently adopted as cognitive models of infant phonetic learning \cite{MatusevychSKFG20} and cross-language non-native processing \cite{matusevych2021phonetic}, we argue that more effort should be devoted to analyze and understand the emergent embedding space to make sure it behaves as expected." />
        </attvalues>
      </edge>
      <edge source="2106.08686" target="2101.11332" id="11727">
        <attvalues>
          <attvalue for="5" value=" Since AWE models have been recently adopted as cognitive models of infant phonetic learning \cite{MatusevychSKFG20} and cross-language non-native processing \cite{matusevych2021phonetic}, we argue that more effort should be devoted to analyze and understand the emergent embedding space to make sure it behaves as expected." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="1802.08232" id="11728">
        <attvalues>
          <attvalue for="5" value=" Large language models (LLMs) memorize many sequences from their pretraining corpora~\cite{carlini2019secret, lehman-etal-2021-bert, lee2023language}." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="2012.07805" id="11729">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{carlini2021extracting} show that GPT2 \cite{radford2019language} can leak some private contact information verbatim." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="2104.08696" id="11730">
        <attvalues>
          <attvalue for="5" value=" We systematically evaluate five methods on our two benchmarks, including existing localization methods (\Act, \cite{geva-etal-2022-transformer}; \IG, \cite{dai-etal-2022-knowledge}), a brute-force method that searches for the most important neurons (\Zero), and two methods we adapt from network pruning \cite{hassibi1992second,han2015deep}, \Slim\ and \HC." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="2202.05262" id="11731">
        <attvalues>
          <attvalue for="5" value=" Prior work on knowledge editing suggests that we can locate a small set of LLM parameters that store factual knowledge~\cite{dai-etal-2022-knowledge,meng2022locating}." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="2301.04213" id="11732">
        <attvalues>
          <attvalue for="5" value=" However, \cite{hase2023does} argue that editing success and localization are actually uncorrelated." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="2209.11895" id="11733">
        <attvalues>
          <attvalue for="5" value=" We propose the \BII, inspired by knockouts \cite{olsson2022context}, a reverse-engineering approach that removes a set of nodes from the computation graph to observe their importance for specific model behavior." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="2203.14680" id="11734">
        <attvalues>
          <attvalue for="5" value=" We systematically evaluate five methods on our two benchmarks, including existing localization methods (\Act, \cite{geva-etal-2022-transformer}; \IG, \cite{dai-etal-2022-knowledge}), a brute-force method that searches for the most important neurons (\Zero), and two methods we adapt from network pruning \cite{hassibi1992second,han2015deep}, \Slim\ and \HC." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="1510.00149" id="11735">
        <attvalues>
          <attvalue for="5" value=" We systematically evaluate five methods on our two benchmarks, including existing localization methods (\Act, \cite{geva-etal-2022-transformer}; \IG, \cite{dai-etal-2022-knowledge}), a brute-force method that searches for the most important neurons (\Zero), and two methods we adapt from network pruning \cite{hassibi1992second,han2015deep}, \Slim\ and \HC." />
        </attvalues>
      </edge>
      <edge source="2311.09060" target="2304.01373" id="11736">
        <attvalues>
          <attvalue for="5" value="9B \cite{biderman2023pythia} identified by \HC\ makes the model forget $57." />
        </attvalues>
      </edge>
      <edge source="2306.03959" target="2208.03188" id="11737">
        <attvalues>
          <attvalue for="5" value=" In prior work, retrieval-enhanced language models have achieved success integrating external knowledge from internet searches into conversational agents \cite{shuster2022,thoppilan2022}." />
        </attvalues>
      </edge>
      <edge source="2306.03959" target="2201.08239" id="11738">
        <attvalues>
          <attvalue for="5" value=" In prior work, retrieval-enhanced language models have achieved success integrating external knowledge from internet searches into conversational agents \cite{shuster2022,thoppilan2022}." />
        </attvalues>
      </edge>
      <edge source="2306.03959" target="2002.08909" id="11739">
        <attvalues>
          <attvalue for="5" value=" Rather than querying the open web, it's more suitable to perform retrieval over a closed set of documents, like in \cite{guu2020,lewis2020}." />
        </attvalues>
      </edge>
      <edge source="2306.03959" target="2005.11401" id="11740">
        <attvalues>
          <attvalue for="5" value=" Rather than querying the open web, it's more suitable to perform retrieval over a closed set of documents, like in \cite{guu2020,lewis2020}." />
        </attvalues>
      </edge>
      <edge source="2306.03959" target="2010.11791" id="11741">
        <attvalues>
          <attvalue for="5" value=" Following the lines of \cite{henderson2021}, which introduces a unique pre-training objective for slot-labeling, our method leverages custom objectives suited for action prediction tasks." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2006.11477" id="11742">
        <attvalues>
          <attvalue for="5" value=" Transformer-based speech self-supervised learning (SSL) models \cite{baevski2020wav2vec, hsu2021hubert, chen2021wavlm} have risen to prominence with great performance in various speech-related tasks\,\cite{yang21c_interspeech, feng2023superb}." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2106.07447" id="11744">
        <attvalues>
          <attvalue for="5" value=" Nonetheless, the downside of these models mainly comes from the requirement of substantial computational resources during the pre-training stage---HuBERT\,Base takes over 82 GPU-days for pre-training, and 32 GPUs en masse are utilized to shorten this\,\cite{hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2106.05933" id="11745">
        <attvalues>
          <attvalue for="5" value=" The above issues make compression techniques essential for the speech SSL models, and several studies have attempted to solve these issues by applying pruning\,\cite{lai2021parp} or quantization\,\cite{wang2022deep} technique." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2303.07592" id="11746">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation of the Transformer-based speech SSL models is being actively studied, with previous works of task-specific compression including automatic speech recognition\,(ASR)\,\cite{choi2023masked}, keyword spotting (KWS)\,\cite{lim2023lightweight}, and automatic speaker verification\,(ASV)\,\cite{heo23_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2305.17394" id="11747">
        <attvalues>
          <attvalue for="5" value=" Knowledge distillation of the Transformer-based speech SSL models is being actively studied, with previous works of task-specific compression including automatic speech recognition\,(ASR)\,\cite{choi2023masked}, keyword spotting (KWS)\,\cite{lim2023lightweight}, and automatic speaker verification\,(ASV)\,\cite{heo23_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2110.01900" id="11748">
        <attvalues>
          <attvalue for="5" value=" Some studies even inefficiently discard these trained linear heads after distillation, although they can convey the teacher's knowledge\,\cite{chang2021distilhubert, lee2022fithubert, peng23c_interspeech, ashihara2022deep}." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2207.00555" id="11749">
        <attvalues>
          <attvalue for="5" value=" On the other hand, approaches that realize task-agnostic compression via knowledge distillation include DistilHuBERT\,\cite{chang2021distilhubert} and FitHuBERT\,\cite{lee2022fithubert}, where the former suggests shallow and wide student model design, and the latter suggests deep and narrow one." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2305.11685" id="11750">
        <attvalues>
          <attvalue for="5" value=" ARMHuBERT\,\cite{jang23_interspeech} proposes to reuse attention maps across Transformer\,\cite{vaswani2017attention} layers and utilizes both masked and unmasked speech frames for masking distillation." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="1706.03762" id="11751">
        <attvalues>
          <attvalue for="5" value=" ARMHuBERT\,\cite{jang23_interspeech} proposes to reuse attention maps across Transformer\,\cite{vaswani2017attention} layers and utilizes both masked and unmasked speech frames for masking distillation." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2305.17651" id="11752">
        <attvalues>
          <attvalue for="5" value=" Some approaches\,\cite{peng23c_interspeech, wang23da_interspeech} jointly conduct distillation with $L_0$ regularization\,\cite{louizos2018learning} and structured pruning." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2203.15610" id="11753">
        <attvalues>
          <attvalue for="5" value=" It even surpasses LightHuBERT\,\cite{wang2022lighthubert}, which demands extensive computational resources for compression, in terms of overall score, number of parameters, and MACs." />
        </attvalues>
      </edge>
      <edge source="2312.09040" target="2110.13900" id="11754">
        <attvalues>
          <attvalue for="5" value=" Our model distilled from HuBERT\,Base achieves the best overall score\,\cite{chen2021wavlm} of 79." />
        </attvalues>
      </edge>
      <edge source="2309.12712" target="2305.05084" id="11756">
        <attvalues>
          <attvalue for="5" value="5B parameters), Nvidia FastConformers \cite{rekesh2023fast} range from Large (118M parameters) to XXLarge (1." />
        </attvalues>
      </edge>
      <edge source="2309.12712" target="2106.07447" id="11757">
        <attvalues>
          <attvalue for="5" value="2B parameters) and self-supervised models like Hubert \cite{hsu2021hubert} or WavLM \cite{chen2021wavlm} are generally available in Base and Large versions." />
        </attvalues>
      </edge>
      <edge source="2309.12712" target="2110.13900" id="11758">
        <attvalues>
          <attvalue for="5" value="2B parameters) and self-supervised models like Hubert \cite{hsu2021hubert} or WavLM \cite{chen2021wavlm} are generally available in Base and Large versions." />
        </attvalues>
      </edge>
      <edge source="2309.12712" target="1912.06670" id="11759">
        <attvalues>
          <attvalue for="5" value=" This is shown in Figure \ref{fig:res} (a), where the mean Word Error Rates (WER) of four Whisper models with different sizes on the test set of CommonVoice \cite{CV} are presented." />
        </attvalues>
      </edge>
      <edge source="2309.12712" target="2008.03403" id="11760">
        <attvalues>
          <attvalue for="5" value=" A few works \cite{ewer2, ewer3} have already attempted to choose among several ASR model versions using WER prediction." />
        </attvalues>
      </edge>
      <edge source="2309.12712" target="2303.06740" id="11761">
        <attvalues>
          <attvalue for="5" value=" Instead of saving computation by choosing between separate ASR models, these methods have attempted to make forward passes lighter by skipping some of the last transformer layers of an ASR model \cite{salahrobin, yoon2022hubertee}." />
        </attvalues>
      </edge>
      <edge source="2309.12712" target="2006.01659" id="11762">
        <attvalues>
          <attvalue for="5" value=" \cite{surprisal}, who propose to save computation cost at inference time by choosing between a large and a small ASR decoder." />
        </attvalues>
      </edge>
      <edge source="2309.12712" target="2106.04624" id="11763">
        <attvalues>
          <attvalue for="5" value=" \item The codebase, developed within the SpeechBrain \cite{ravanelli2021speechbrain} framework, is released for further investigations." />
        </attvalues>
      </edge>
      <edge source="2305.10684" target="1905.05879" id="11765">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, we find that existing pretrained few-shot VC models, such as AutoVC \cite{qian2019autovc} and FragmentVC \cite{lin2021fragmentvc} perform poorly in these conditions." />
        </attvalues>
      </edge>
      <edge source="2305.10684" target="2010.14150" id="11766">
        <attvalues>
          <attvalue for="5" value=" Unfortunately, we find that existing pretrained few-shot VC models, such as AutoVC \cite{qian2019autovc} and FragmentVC \cite{lin2021fragmentvc} perform poorly in these conditions." />
        </attvalues>
      </edge>
      <edge source="2109.12258" target="1802.05365" id="11784">
        <attvalues>
          <attvalue for="5" value=" As a fragmentary example, \cite{Fili:19} reports that a large ensemble of 6 BiLSTMs with BERT \cite{bert}, ELMo \cite{elmo}, Word2Vec \cite{word2vec}, and GloVe \cite{glove} embeddings showed only $\sim$1\% accuracy improvement from a single SVM model developed by \cite{Xia:16}." />
        </attvalues>
      </edge>
      <edge source="2109.12258" target="1906.07580" id="11786">
        <attvalues>
          <attvalue for="5" value=" As a fragmentary example, \cite{Fili:19} reports that a large ensemble of 6 BiLSTMs with BERT \cite{bert}, ELMo \cite{elmo}, Word2Vec \cite{word2vec}, and GloVe \cite{glove} embeddings showed only $\sim$1\% accuracy improvement from a single SVM model developed by \cite{Xia:16}." />
        </attvalues>
      </edge>
      <edge source="2109.12258" target="1103.0398" id="11787">
        <attvalues>
          <attvalue for="5" value=" Even though deep neural networks have achieved state-of-the-art (SOTA) performance in almost all semantic tasks where sufficient data were available \cite{colbert:11, Zhang:15}, neural models started showing promising results in RA only quite recently \cite{Martinc:21}." />
        </attvalues>
      </edge>
      <edge source="2109.12258" target="1907.11779" id="11788">
        <attvalues>
          <attvalue for="5" value=" \cite{Martinc:21} raised the SOTA classification accuracy on the popular WeeBit dataset \cite{Vajjala:12} by about 4\% using BERT." />
        </attvalues>
      </edge>
      <edge source="2109.12258" target="2006.00377" id="11789">
        <attvalues>
          <attvalue for="5" value=" Such a hybrid system is only reported by \cite{deutsch-etal-2020-linguistic}, concluding, ``(hybrid models) did not achieve SOTA performance." />
        </attvalues>
      </edge>
      <edge source="2408.06281" target="2307.03172" id="11800">
        <attvalues>
          <attvalue for="5" value=" However, these models often struggle when the input context is long, particularly when the relevant information is distributed across the document \cite{liu2023lost}." />
        </attvalues>
      </edge>
      <edge source="2408.06281" target="2105.08209" id="11801">
        <attvalues>
          <attvalue for="5" value=" More recently, narrative summarization research has focused on TV shows and books \cite{kryscinski2021booksum,moskvichev-mai-2023-narrativexl}, with less attention given to movie screenplays \cite{gorinski-lapata-2015-movie,papalampidi-etal-2020-screenplay}." />
        </attvalues>
      </edge>
      <edge source="2408.06281" target="2305.13877" id="11802">
        <attvalues>
          <attvalue for="5" value=" More recently, narrative summarization research has focused on TV shows and books \cite{kryscinski2021booksum,moskvichev-mai-2023-narrativexl}, with less attention given to movie screenplays \cite{gorinski-lapata-2015-movie,papalampidi-etal-2020-screenplay}." />
        </attvalues>
      </edge>
      <edge source="2408.06281" target="2004.12727" id="11803">
        <attvalues>
          <attvalue for="5" value=" More recently, narrative summarization research has focused on TV shows and books \cite{kryscinski2021booksum,moskvichev-mai-2023-narrativexl}, with less attention given to movie screenplays \cite{gorinski-lapata-2015-movie,papalampidi-etal-2020-screenplay}." />
        </attvalues>
      </edge>
      <edge source="2408.06281" target="2104.07091" id="11804">
        <attvalues>
          <attvalue for="5" value=" Notably, \cite{chen-etal-2022-summscreen} introduced a dataset of TV show transcripts which has gained considerable interest and was included in a long document summarization benchmark \cite{shaham-etal-2022-scrolls}." />
        </attvalues>
      </edge>
      <edge source="2408.06281" target="2201.03533" id="11805">
        <attvalues>
          <attvalue for="5" value=" Notably, \cite{chen-etal-2022-summscreen} introduced a dataset of TV show transcripts which has gained considerable interest and was included in a long document summarization benchmark \cite{shaham-etal-2022-scrolls}." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="2301.07597" id="11806">
        <attvalues>
          <attvalue for="5" value=" ChatGPT has shown strong capabilities as a dialogue system, providing clearer and more helpful answers than humans \cite{guo2023close}." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="1911.02116" id="11807">
        <attvalues>
          <attvalue for="5" value=" It remains unclear whether its performance on text categorization tasks, more specifically on automatic genre identification, can be compared to the existing large language models (LLMs), such as the XLM-RoBERTa model \cite{conneau2020unsupervised}, fine-tuned to the task." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="2302.06476" id="11808">
        <attvalues>
          <attvalue for="5" value=" \cite{qin2023chatgpt} analyzed its zero-shot performance on reasoning tasks, natural language inference, dialogue, question answering, summarization, named-entity recognition and sentiment analysis." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="2302.10198" id="11809">
        <attvalues>
          <attvalue for="5" value=" Similarly, \cite{zhong2023can} compared ChatGPT with fine-tuned language models BERT \cite{DBLP:journals/corr/abs-1810-04805} and RoBERTa \cite{liu2019roberta} on the GLUE benchmark \cite{wang2018glue}, consisting of sentiment analysis, linguistic acceptability, paraphrase, textual similarity, natural language inference, and question answering." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="1907.11692" id="11811">
        <attvalues>
          <attvalue for="5" value=" Similarly, \cite{zhong2023can} compared ChatGPT with fine-tuned language models BERT \cite{DBLP:journals/corr/abs-1810-04805} and RoBERTa \cite{liu2019roberta} on the GLUE benchmark \cite{wang2018glue}, consisting of sentiment analysis, linguistic acceptability, paraphrase, textual similarity, natural language inference, and question answering." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="2212.14548" id="11813">
        <attvalues>
          <attvalue for="5" value=" In contrast, when \cite{zhang2022would} analyzed ChatGPT's performance on stance detection, it achieved state-of-the-art (SOTA) results on this task." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="2302.09210" id="11814">
        <attvalues>
          <attvalue for="5" value=" The ChatGPT model was also shown to achieve competitive performance in machine translation of high-resource languages, while it is lacking behind the SOTA models on low-resource languages based on the WMT22 \cite{kocmi-etal-2022-findings} dataset \cite{hendy2023good}." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="2302.07736" id="11815">
        <attvalues>
          <attvalue for="5" value=" When \cite{huang2023chatgpt} examined the performance of ChatGPT on categorizing implicit hate speech and providing natural language explanations for the implicit hate speech detection, the results showed great potential of ChatGPT for this and similar tasks." />
        </attvalues>
      </edge>
      <edge source="2303.03953" target="2201.03857" id="11816">
        <attvalues>
          <attvalue for="5" value=" The models are compared on two test sets, English EN-GINCO dataset and Slovenian GINCO dataset \cite{kuzman-rupnik-ljubei:2022:LREC}, in three scenarios: English prompt with English text, English prompt with Slovenian text and Slovenian prompt with Slovenian text." />
        </attvalues>
      </edge>
      <edge source="2407.04416" target="2211.06687" id="11817">
        <attvalues>
          <attvalue for="5" value=" In particular, the development of language models~\cite{clap,t5} and diffusion models~\cite{dalle3,stable_diffusion} have enabled the creation of powerful systems~\cite{audioldm2,tango} on generating high-fidelity audio clips." />
        </attvalues>
      </edge>
      <edge source="2407.04416" target="2112.10752" id="11819">
        <attvalues>
          <attvalue for="5" value=" In particular, the development of language models~\cite{clap,t5} and diffusion models~\cite{dalle3,stable_diffusion} have enabled the creation of powerful systems~\cite{audioldm2,tango} on generating high-fidelity audio clips." />
        </attvalues>
      </edge>
      <edge source="2407.04416" target="2308.05734" id="11820">
        <attvalues>
          <attvalue for="5" value=" In particular, the development of language models~\cite{clap,t5} and diffusion models~\cite{dalle3,stable_diffusion} have enabled the creation of powerful systems~\cite{audioldm2,tango} on generating high-fidelity audio clips." />
        </attvalues>
      </edge>
      <edge source="2407.04416" target="2303.17395" id="11821">
        <attvalues>
          <attvalue for="5" value="5$K audio clips match with the caption `` Music is playing'' in WavCaps~\cite{wavcaps}), causing the system to avoid learning specific audio feature and lead to more instability in the generated outputs." />
        </attvalues>
      </edge>
      <edge source="2407.04416" target="2301.12503" id="11822">
        <attvalues>
          <attvalue for="5" value=" Using Sound-VECaps as the training dataset, our experiments with the audio generation model, AudioLDM~\cite{audioldm}, show substantial improvements over baseline models." />
        </attvalues>
      </edge>
      <edge source="2205.10517" target="1706.03762" id="11823">
        <attvalues>
          <attvalue for="5" value=" Transformer-based \cite{NIPS2017_3f5ee243} contextual models such as BERT \cite{devlin-etal-2019-bert} have gained success since the fine-tuning step is relatively inexpensive, while attaining state-of-the-art results in various syntactic and semantic tasks." />
        </attvalues>
      </edge>
      <edge source="2310.15075" target="1508.00305" id="11824">
        <attvalues>
          <attvalue for="5" value=" Question answering systems devote to answering various questions with the evidence located in the structured knowledge base~(eg, Table~\cite{pasupat2015compositional}, Knowledge Graph~\cite{lan2021survey}), unstructured texts~\cite{rajpurkar2016squad} or images~(eg, VQA~\cite{antol2015vqa})." />
        </attvalues>
      </edge>
      <edge source="2310.15075" target="2105.11644" id="11825">
        <attvalues>
          <attvalue for="5" value=" Question answering systems devote to answering various questions with the evidence located in the structured knowledge base~(eg, Table~\cite{pasupat2015compositional}, Knowledge Graph~\cite{lan2021survey}), unstructured texts~\cite{rajpurkar2016squad} or images~(eg, VQA~\cite{antol2015vqa})." />
        </attvalues>
      </edge>
      <edge source="2310.15075" target="1606.05250" id="11826">
        <attvalues>
          <attvalue for="5" value=" Question answering systems devote to answering various questions with the evidence located in the structured knowledge base~(eg, Table~\cite{pasupat2015compositional}, Knowledge Graph~\cite{lan2021survey}), unstructured texts~\cite{rajpurkar2016squad} or images~(eg, VQA~\cite{antol2015vqa})." />
        </attvalues>
      </edge>
      <edge source="2310.15075" target="1505.00468" id="11827">
        <attvalues>
          <attvalue for="5" value=" Question answering systems devote to answering various questions with the evidence located in the structured knowledge base~(eg, Table~\cite{pasupat2015compositional}, Knowledge Graph~\cite{lan2021survey}), unstructured texts~\cite{rajpurkar2016squad} or images~(eg, VQA~\cite{antol2015vqa})." />
        </attvalues>
      </edge>
      <edge source="2310.15075" target="2004.07347" id="11828">
        <attvalues>
          <attvalue for="5" value=" TableQA mainly contains three typical subtasks: Spreadsheet QA \cite{zhu2021tat}, Encyclopedia QA \cite{chen2020hybridqa}, and Structured QA." />
        </attvalues>
      </edge>
      <edge source="2310.15075" target="2303.18223" id="11831">
        <attvalues>
          <attvalue for="5" value=" Moreover, with the development of large language model (LLM) ~\cite{zhao2023survey}, LLM-based methods~\cite{dong2022survey, touvron2023llama} have gradually become new paradigms in addressing NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2005.13895" target="1910.09799" id="11832">
        <attvalues>
          <attvalue for="5" value=" Previous studies showed SANs can yield superior speech recognition results compared to recurrent neural networks (RNNs) \cite{wang2019transformerHybrid, karita2019comparative}." />
        </attvalues>
      </edge>
      <edge source="2005.13895" target="1412.3555" id="11833">
        <attvalues>
          <attvalue for="5" value=" Although gated structures, such as Long Short-Term Memory (LSTM)~\cite{hochreiter1997LSTM} and Gated Recurrent Unit (GRU)~\cite{chung2014GRU} are proposed to alleviate this problem, capturing temporal relationships across a wide time span remains challenging for these models." />
        </attvalues>
      </edge>
      <edge source="2005.13895" target="1409.0473" id="11834">
        <attvalues>
          <attvalue for="5" value=" In SANs, self-attention layers encode contextual information through attention mechanisms \cite{bahdanau2015neural,luong2015effective,vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="2005.13895" target="1706.03762" id="11836">
        <attvalues>
          <attvalue for="5" value=" In this paper we study Transformers~\cite{vaswani2017attention}, end-to-end SAN-based models with two components: an encoder and a decoder." />
        </attvalues>
      </edge>
      <edge source="2005.13895" target="1705.08091" id="11837">
        <attvalues>
          <attvalue for="5" value=" Previous work on attention-based RNN end-to-end models has shown that for speech recognition, since acoustic events usually happen in a left-to-right order within small time spans, restricting the attention to be monotonic along the time axis improves the model's performance \cite{tjandra2017local, kim2017joint, zhang2019windowed}." />
        </attvalues>
      </edge>
      <edge source="2308.00081" target="1910.12507" id="11838">
        <attvalues>
          <attvalue for="5" value="~\cite{DBLP:journals/semweb/GeseseBAS21}, which categorizes these methods based on their scoring function (inspired by the work by Wang et al." />
        </attvalues>
      </edge>
      <edge source="2308.00081" target="2306.08302" id="11839">
        <attvalues>
          <attvalue for="5" value="~\cite{pan2024unifying} describes methods that integrate LLMs for KG embeddings and KG completion, yet the role of schematic information is not considered in the anlaysis." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="1907.11692" id="11842">
        <attvalues>
          <attvalue for="5" value=" Some prominent examples of PrLMs are BERT \cite{devlin2019bert}, GPT \cite{radford2018improving}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ERNIE \cite{ernie2019sun, ernie202019sun}, ALBERT \cite{lan2019albert} and ELECTRA \cite{electra2020Clark}." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="1904.09223" id="11843">
        <attvalues>
          <attvalue for="5" value=" Some prominent examples of PrLMs are BERT \cite{devlin2019bert}, GPT \cite{radford2018improving}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ERNIE \cite{ernie2019sun, ernie202019sun}, ALBERT \cite{lan2019albert} and ELECTRA \cite{electra2020Clark}." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="1909.11942" id="11844">
        <attvalues>
          <attvalue for="5" value=" Some prominent examples of PrLMs are BERT \cite{devlin2019bert}, GPT \cite{radford2018improving}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ERNIE \cite{ernie2019sun, ernie202019sun}, ALBERT \cite{lan2019albert} and ELECTRA \cite{electra2020Clark}." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="2003.10555" id="11845">
        <attvalues>
          <attvalue for="5" value=" Some prominent examples of PrLMs are BERT \cite{devlin2019bert}, GPT \cite{radford2018improving}, XLNet \cite{yang2019xlnet}, RoBERTa \cite{liu2019roberta}, ERNIE \cite{ernie2019sun, ernie202019sun}, ALBERT \cite{lan2019albert} and ELECTRA \cite{electra2020Clark}." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="2102.05474" id="11846">
        <attvalues>
          <attvalue for="5" value=" \cite{DialoGPTLG2020Zhang} pre-trained GPT further with a conventional Language Model (LM) objective on a large dialogue corpus, Reddit, and get DialoGPT for response generation tasks; \cite{zhang2021kkt} and \cite{AnED2019Whang} pre-trained BERT with Mask Language Model (MLM) objective on the target datasets before fine-tuning on response selection tasks." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="1908.04812" id="11847">
        <attvalues>
          <attvalue for="5" value=" \cite{DialoGPTLG2020Zhang} pre-trained GPT further with a conventional Language Model (LM) objective on a large dialogue corpus, Reddit, and get DialoGPT for response generation tasks; \cite{zhang2021kkt} and \cite{AnED2019Whang} pre-trained BERT with Mask Language Model (MLM) objective on the target datasets before fine-tuning on response selection tasks." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="2004.06871" id="11848">
        <attvalues>
          <attvalue for="5" value=" Notably, \cite{wu2020tod} proposed a response contrastive loss to match the context with the corresponding response and distinguish from the randomly sampled negative response." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="1902.08654" id="11849">
        <attvalues>
          <attvalue for="5" value=" As it is pointed out by recent dialogue evaluation studies \cite{WhatMA2019See,UnsupervisedEO2020Mehri,mehri2020usr,pang2020towards,DesigningPA2020Zhao}, estimating whether an open-domain dialogue is informative is also essential for assessing its quality, which motivates us to improve dialogue-adaptive pre-training by considering informativeness to better simulate dialogue-specific characteristics." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="2006.12719" id="11850">
        <attvalues>
          <attvalue for="5" value=" As it is pointed out by recent dialogue evaluation studies \cite{WhatMA2019See,UnsupervisedEO2020Mehri,mehri2020usr,pang2020towards,DesigningPA2020Zhao}, estimating whether an open-domain dialogue is informative is also essential for assessing its quality, which motivates us to improve dialogue-adaptive pre-training by considering informativeness to better simulate dialogue-specific characteristics." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="2005.00456" id="11851">
        <attvalues>
          <attvalue for="5" value=" As it is pointed out by recent dialogue evaluation studies \cite{WhatMA2019See,UnsupervisedEO2020Mehri,mehri2020usr,pang2020towards,DesigningPA2020Zhao}, estimating whether an open-domain dialogue is informative is also essential for assessing its quality, which motivates us to improve dialogue-adaptive pre-training by considering informativeness to better simulate dialogue-specific characteristics." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="2004.04908" id="11852">
        <attvalues>
          <attvalue for="5" value=" As it is pointed out by recent dialogue evaluation studies \cite{WhatMA2019See,UnsupervisedEO2020Mehri,mehri2020usr,pang2020towards,DesigningPA2020Zhao}, estimating whether an open-domain dialogue is informative is also essential for assessing its quality, which motivates us to improve dialogue-adaptive pre-training by considering informativeness to better simulate dialogue-specific characteristics." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="1806.08044" id="11853">
        <attvalues>
          <attvalue for="5" value=" Inspired by \cite{ModelingLC2005Barzilay,CoherenceMF2018Cervone,DialogueCA2020Mesgar}, we propose coherence-oriented context corruption, including utterance ordering, utterance insertion and utterance replacement, to generate incoherent dialogues as negative samples." />
        </attvalues>
      </edge>
      <edge source="2009.04984" target="1908.08486" id="11854">
        <attvalues>
          <attvalue for="5" value=" Inspired by \cite{ModelingLC2005Barzilay,CoherenceMF2018Cervone,DialogueCA2020Mesgar}, we propose coherence-oriented context corruption, including utterance ordering, utterance insertion and utterance replacement, to generate incoherent dialogues as negative samples." />
        </attvalues>
      </edge>
      <edge source="2305.14556" target="2005.14165" id="11858">
        <attvalues>
          <attvalue for="5" value=" \\ Meanwhile, very large pre-trained language models (LPLMs), such as BERT \cite{devlin2018bert}, T5 \cite{raffel-t5}, GPT-3 \cite{GPT-3-reference}, LaMDA \cite{lamda-paper}, PaLM \cite{chowdhery2022palm}, and IstructGPT \cite{instruct-gpt}, have demonstrated unparalleled ability to generate high-quality text via prompting strategies \cite{prompting-survey}." />
        </attvalues>
      </edge>
      <edge source="2305.14556" target="2201.08239" id="11859">
        <attvalues>
          <attvalue for="5" value=" \\ Meanwhile, very large pre-trained language models (LPLMs), such as BERT \cite{devlin2018bert}, T5 \cite{raffel-t5}, GPT-3 \cite{GPT-3-reference}, LaMDA \cite{lamda-paper}, PaLM \cite{chowdhery2022palm}, and IstructGPT \cite{instruct-gpt}, have demonstrated unparalleled ability to generate high-quality text via prompting strategies \cite{prompting-survey}." />
        </attvalues>
      </edge>
      <edge source="2305.14556" target="2204.02311" id="11860">
        <attvalues>
          <attvalue for="5" value=" \\ Meanwhile, very large pre-trained language models (LPLMs), such as BERT \cite{devlin2018bert}, T5 \cite{raffel-t5}, GPT-3 \cite{GPT-3-reference}, LaMDA \cite{lamda-paper}, PaLM \cite{chowdhery2022palm}, and IstructGPT \cite{instruct-gpt}, have demonstrated unparalleled ability to generate high-quality text via prompting strategies \cite{prompting-survey}." />
        </attvalues>
      </edge>
      <edge source="2305.14556" target="2107.13586" id="11862">
        <attvalues>
          <attvalue for="5" value=" \\ Meanwhile, very large pre-trained language models (LPLMs), such as BERT \cite{devlin2018bert}, T5 \cite{raffel-t5}, GPT-3 \cite{GPT-3-reference}, LaMDA \cite{lamda-paper}, PaLM \cite{chowdhery2022palm}, and IstructGPT \cite{instruct-gpt}, have demonstrated unparalleled ability to generate high-quality text via prompting strategies \cite{prompting-survey}." />
        </attvalues>
      </edge>
      <edge source="2212.06346" target="2207.04672" id="11863">
        <attvalues>
          <attvalue for="5" value=" With \cite{MetaNoLanguageLeftBehind} and related work, we have seen progress in recent years on the expansion of machine translation into the domain of under-served languages both by advancing science as well as creation of corpora in machine translation." />
        </attvalues>
      </edge>
      <edge source="2212.06346" target="2204.08582" id="11864">
        <attvalues>
          <attvalue for="5" value=" First, we created and released the Multilingual Amazon SLU resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation, or MASSIVE dataset \cite{massiveDataset}, containing 1 million realistic, parallel, labeled virtual assistant text utterances spanning 51 languages." />
        </attvalues>
      </edge>
      <edge source="2304.08823" target="1911.02116" id="11866">
        <attvalues>
          <attvalue for="5" value=" MMTs' representation spaces are heavily skewed in favor of high-resource languages, for which they have been exposed to much more data in pretraining ~\cite{joshi-etal-2020-state,wu-dredze-2020-languages}; combined with the `curse of multilinguality' -- eg, limited per-language representation quality stemming from a limited capacity of the model \cite{conneau-etal-2020-unsupervised,pfeiffer-etal-2022-lifting} -- this leads to lower representational quality for languages underrepresented in MMTs' pretraining." />
        </attvalues>
      </edge>
      <edge source="2304.08823" target="2010.11934" id="11867">
        <attvalues>
          <attvalue for="5" value=" Massively multilingual Transformer-based language models (MMTs) such as mBERT \cite{devlin-etal-2019-bert}, XLM-RoBERTa \cite{conneau-etal-2020-unsupervised} and mT5 \cite{xue-etal-2021-mt5} have been the driving force of modern multilingual NLP, allowing for rapid bootstrapping of language technology for a wide range of low(er)-resource languages by means of (zero-shot or few-shot) cross-lingual transfer from high(er)-resource languages \cite{lauscher-etal-2020-zero,hu2020xtreme,xu-murray-2022-por,schmidt-etal-2022-dont}." />
        </attvalues>
      </edge>
      <edge source="2304.08823" target="2204.13869" id="11868">
        <attvalues>
          <attvalue for="5" value=" Massively multilingual Transformer-based language models (MMTs) such as mBERT \cite{devlin-etal-2019-bert}, XLM-RoBERTa \cite{conneau-etal-2020-unsupervised} and mT5 \cite{xue-etal-2021-mt5} have been the driving force of modern multilingual NLP, allowing for rapid bootstrapping of language technology for a wide range of low(er)-resource languages by means of (zero-shot or few-shot) cross-lingual transfer from high(er)-resource languages \cite{lauscher-etal-2020-zero,hu2020xtreme,xu-murray-2022-por,schmidt-etal-2022-dont}." />
        </attvalues>
      </edge>
      <edge source="2304.08823" target="2004.09095" id="11869">
        <attvalues>
          <attvalue for="5" value=" MMTs' representation spaces are heavily skewed in favor of high-resource languages, for which they have been exposed to much more data in pretraining ~\cite{joshi-etal-2020-state,wu-dredze-2020-languages}; combined with the `curse of multilinguality' -- eg, limited per-language representation quality stemming from a limited capacity of the model \cite{conneau-etal-2020-unsupervised,pfeiffer-etal-2022-lifting} -- this leads to lower representational quality for languages underrepresented in MMTs' pretraining." />
        </attvalues>
      </edge>
      <edge source="2304.08823" target="2005.09093" id="11870">
        <attvalues>
          <attvalue for="5" value=" MMTs' representation spaces are heavily skewed in favor of high-resource languages, for which they have been exposed to much more data in pretraining ~\cite{joshi-etal-2020-state,wu-dredze-2020-languages}; combined with the `curse of multilinguality' -- eg, limited per-language representation quality stemming from a limited capacity of the model \cite{conneau-etal-2020-unsupervised,pfeiffer-etal-2022-lifting} -- this leads to lower representational quality for languages underrepresented in MMTs' pretraining." />
        </attvalues>
      </edge>
      <edge source="2304.08823" target="2205.06266" id="11871">
        <attvalues>
          <attvalue for="5" value=" MMTs' representation spaces are heavily skewed in favor of high-resource languages, for which they have been exposed to much more data in pretraining ~\cite{joshi-etal-2020-state,wu-dredze-2020-languages}; combined with the `curse of multilinguality' -- eg, limited per-language representation quality stemming from a limited capacity of the model \cite{conneau-etal-2020-unsupervised,pfeiffer-etal-2022-lifting} -- this leads to lower representational quality for languages underrepresented in MMTs' pretraining." />
        </attvalues>
      </edge>
      <edge source="2304.08823" target="2012.15682" id="11872">
        <attvalues>
          <attvalue for="5" value=" Cross-lingual transfer with MMTs thus fails exactly in settings in which it is needed the most: for low-resource languages with small digital footprint \cite{zhao-etal-2021-closer}." />
        </attvalues>
      </edge>
      <edge source="2210.11255" target="2005.00870" id="11873">
        <attvalues>
          <attvalue for="5" value=" In NLP, prior work has examined the different yet related task of performance prediction \cite{xia-etal-2020-predicting,ye-etal-2021-towards}, surveyed and categorized LMs~\cite{xia-etal-2020-bert}, and used probing to predict LM performance specifically for dependency parsing~\cite{muller-eberstein-etal-2022-sort}, but has yet to extensively investigate how to rank the increasingly large number of pre-trained LM encoders across various tasks and domains." />
        </attvalues>
      </edge>
      <edge source="2210.11255" target="2102.05486" id="11874">
        <attvalues>
          <attvalue for="5" value=" In NLP, prior work has examined the different yet related task of performance prediction \cite{xia-etal-2020-predicting,ye-etal-2021-towards}, surveyed and categorized LMs~\cite{xia-etal-2020-bert}, and used probing to predict LM performance specifically for dependency parsing~\cite{muller-eberstein-etal-2022-sort}, but has yet to extensively investigate how to rank the increasingly large number of pre-trained LM encoders across various tasks and domains." />
        </attvalues>
      </edge>
      <edge source="2210.11255" target="2206.04935" id="11876">
        <attvalues>
          <attvalue for="5" value=" In NLP, prior work has examined the different yet related task of performance prediction \cite{xia-etal-2020-predicting,ye-etal-2021-towards}, surveyed and categorized LMs~\cite{xia-etal-2020-bert}, and used probing to predict LM performance specifically for dependency parsing~\cite{muller-eberstein-etal-2022-sort}, but has yet to extensively investigate how to rank the increasingly large number of pre-trained LM encoders across various tasks and domains." />
        </attvalues>
      </edge>
      <edge source="2210.11255" target="2102.11005" id="11877">
        <attvalues>
          <attvalue for="5" value=" Preliminary work by \cite{you2021logme} shows that the LogME estimator holds promise, including the first steps for encoder selection in NLP." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="1908.02265" id="11881">
        <attvalues>
          <attvalue for="5" value=" Inspired by this approach, VilBERT \cite{lu_vilbert_2019} leveraged parallel multimodal data for pre-training a visual-language model." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="2005.14165" id="11882">
        <attvalues>
          <attvalue for="5" value=" To solve these issues, GPT-3 \cite{brown_language_2020} proposed ``prompt tuning&quot;, an intuitive method to transfer a powerful pretrained model only with text interactions, called ``prompts&quot;, without any gradient updates." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="2104.08691" id="11883">
        <attvalues>
          <attvalue for="5" value=" This idea was later extended, with many variations \cite{lester-etal-2021-power, li-liang-2021-prefix}, to make these prompts trainable, now called ``soft prompts&quot;." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="1902.00751" id="11884">
        <attvalues>
          <attvalue for="5" value=" MAGMA \cite{eichenberg_magma_2021} extended this by showing that the addition of adapter layers \cite{houlsby2019parameter} in between the frozen language layers outperforms Frozen." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="2106.13884" id="11885">
        <attvalues>
          <attvalue for="5" value=" Our approach extends concepts introduced by visual-language models \cite{tsimpoukelli2021multimodal,eichenberg_magma_2021, alayrac_flamingo_2022} to include audio." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="2112.05253" id="11886">
        <attvalues>
          <attvalue for="5" value=" Our approach extends concepts introduced by visual-language models \cite{tsimpoukelli2021multimodal,eichenberg_magma_2021, alayrac_flamingo_2022} to include audio." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="2204.14198" id="11887">
        <attvalues>
          <attvalue for="5" value=" Flamingo \cite{alayrac_flamingo_2022} scaled up and optimised this concept by introducing a flexible visual encoder which can turn arbitrary sequences of images or even video frames to a fixed number of visual tokens." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="1805.08660" id="11888">
        <attvalues>
          <attvalue for="5" value=" The next innovation was the introduction of the attention model to create sophisticated fusion approaches \cite{gu_multimodal_2018, wang_words_2018}." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="1706.03762" id="11889">
        <attvalues>
          <attvalue for="5" value=" This naturally led to the incorporation of the transformer \cite{vaswani_attention_2017} as the central model for this task \cite{DBLP:journals/corr/abs-1806-06176, delbrouck-etal-2020-transformer}." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="1806.06176" id="11890">
        <attvalues>
          <attvalue for="5" value=" This naturally led to the incorporation of the transformer \cite{vaswani_attention_2017} as the central model for this task \cite{DBLP:journals/corr/abs-1806-06176, delbrouck-etal-2020-transformer}." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="1911.05544" id="11891">
        <attvalues>
          <attvalue for="5" value=" ICCN \cite{Sun2020LearningRB} introduced Deep Canonical Correlation Analysis for jointly learning representations." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="1811.09362" id="11892">
        <attvalues>
          <attvalue for="5" value=" \cite{wang_words_2018} and later MAG-BERT \cite{rahman_integrating_2020} proposed shifting methods." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="1908.05787" id="11893">
        <attvalues>
          <attvalue for="5" value=" \cite{wang_words_2018} and later MAG-BERT \cite{rahman_integrating_2020} proposed shifting methods." />
        </attvalues>
      </edge>
      <edge source="2212.00678" target="2102.04830" id="11894">
        <attvalues>
          <attvalue for="5" value=" More recently, many researchers turned their efforts towards intricate multimodal pre-training strategies, such as \cite{yu2021le, kim_cmsbert-clr_2022}." />
        </attvalues>
      </edge>
      <edge source="2212.00006" target="2005.14165" id="11895">
        <attvalues>
          <attvalue for="5" value=" \item Foundational Role: The same substrate models now form the basis of a number of applications \cite{foundation_models} and this is driven both by the utility of their representations as well as phenomena such as in-context learning \cite{gpt3}, which raise the abstraction level of interacting with such models." />
        </attvalues>
      </edge>
      <edge source="2212.00006" target="2206.11249" id="11897">
        <attvalues>
          <attvalue for="5" value=" We believe that modern generative models have had three main impacts: \begin{enumerate} \item Fluent Generations: The fluency of generation in both language and vision modalities has rendered existing benchmarks and metrics much less useful in diagnosing system errors \cite{gemv2, bigbench}, with previously widely used benchmarks and metrics on tasks such as machine translation \cite{salted} or summarization \cite{goyalzeroshotnews2022} becoming less useful in gauging system problems." />
        </attvalues>
      </edge>
      <edge source="2212.00006" target="2205.09988" id="11898">
        <attvalues>
          <attvalue for="5" value=" We believe that modern generative models have had three main impacts: \begin{enumerate} \item Fluent Generations: The fluency of generation in both language and vision modalities has rendered existing benchmarks and metrics much less useful in diagnosing system errors \cite{gemv2, bigbench}, with previously widely used benchmarks and metrics on tasks such as machine translation \cite{salted} or summarization \cite{goyalzeroshotnews2022} becoming less useful in gauging system problems." />
        </attvalues>
      </edge>
      <edge source="2212.00006" target="2209.12356" id="11899">
        <attvalues>
          <attvalue for="5" value=" We believe that modern generative models have had three main impacts: \begin{enumerate} \item Fluent Generations: The fluency of generation in both language and vision modalities has rendered existing benchmarks and metrics much less useful in diagnosing system errors \cite{gemv2, bigbench}, with previously widely used benchmarks and metrics on tasks such as machine translation \cite{salted} or summarization \cite{goyalzeroshotnews2022} becoming less useful in gauging system problems." />
        </attvalues>
      </edge>
      <edge source="2212.00006" target="2108.07258" id="11900">
        <attvalues>
          <attvalue for="5" value=" \item Foundational Role: The same substrate models now form the basis of a number of applications \cite{foundation_models} and this is driven both by the utility of their representations as well as phenomena such as in-context learning \cite{gpt3}, which raise the abstraction level of interacting with such models." />
        </attvalues>
      </edge>
      <edge source="2004.14979" target="1906.01753" id="11904">
        <attvalues>
          <attvalue for="5" value=" In the other direction (Section \ref{sec:using_chirps}), we incorporate data from Chirps, represented in the Chirps re-scorer feature vector, into a state-of-the-art event coreference system \cite{barhom-etal-2019-revisiting}." />
        </attvalues>
      </edge>
      <edge source="2004.14979" target="1504.05929" id="11906">
        <attvalues>
          <attvalue for="5" value=" Recent systems rely on neural representations of the mentions and their contexts \cite{kenyon-dean-etal-2018-resolving,barhom-etal-2019-revisiting}, while earlier approaches leveraged WordNet and other lexical resources to obtain a signal of whether a pair of mentions may be coreferring \cite[eg][]{bejan-harabagiu-2010-unsupervised,yang2015hierarchical}." />
        </attvalues>
      </edge>
      <edge source="2004.14979" target="cs/0304006" id="11907">
        <attvalues>
          <attvalue for="5" value=" These included similarity between argument distributions \cite{Lin2001DIRTD, berant2012global}, backtranslation across languages \cite{barzilay-mckeown-2001-extracting,ganitkevitch2013ppdb,mallinson-etal-2017-paraphrasing}, or leveraging redundant news reports on the same event, which are hence likely to refer to the same events and entities using different words \cite{shinyama2002automatic,shinyama-sekine-2006-preemptive,barzilay-lee-2003-learning,zhang2013harvesting,xu2014extracting,shwartz-etal-2017-acquiring}." />
        </attvalues>
      </edge>
      <edge source="2004.14979" target="1708.00391" id="11908">
        <attvalues>
          <attvalue for="5" value=" In some cases, the paraphrase collection phase includes a step of validating a subset of the paraphrases and training a model on these gold paraphrases to re-rank the entire resource \cite{lan-etal-2017-continuously}." />
        </attvalues>
      </edge>
      <edge source="2402.08968" target="2105.03023" id="11909">
        <attvalues>
          <attvalue for="5" value=" Previous attempts for safe response generation have been dedicated to making use of exemplary safe dialogues annotated by humans, by fine-tuning~\cite{xu2021bot, kim2022prosocialdialog, ziems2022moral} or training auxiliary safety detector~\cite{liu2021dexperts}." />
        </attvalues>
      </edge>
      <edge source="2402.08968" target="2205.12688" id="11910">
        <attvalues>
          <attvalue for="5" value=" There have been few early attempts to incorporate the human norms, namely Rules-of-Thumb (RoT), into dialog system~\cite{kim2022prosocialdialog, ziems2022moral}." />
        </attvalues>
      </edge>
      <edge source="2402.08968" target="2004.13637" id="11911">
        <attvalues>
          <attvalue for="5" value=" We demonstrate the quantitative and qualitative effectiveness of GrounDial with Blenderbot~\cite{roller2021recipes} where both response safety and RoT relevance are improved without additional training." />
        </attvalues>
      </edge>
      <edge source="2402.14890" target="2305.10403" id="11912">
        <attvalues>
          <attvalue for="5" value=" Increasingly large language models such as \cite{anil2023palm,xu2023wizardlm,taori2023alpaca, vicuna2023, chowdhery2022palm, bajaj2022metro, zoph2022designing, raffel2020exploring, brown2020language,devlin-etal-2019-bert} manifest a clear trend to develop large, foundational models and then fine-tune in on a variety of NLP tasks depending on the use case." />
        </attvalues>
      </edge>
      <edge source="2402.14890" target="2304.12244" id="11913">
        <attvalues>
          <attvalue for="5" value=" Increasingly large language models such as \cite{anil2023palm,xu2023wizardlm,taori2023alpaca, vicuna2023, chowdhery2022palm, bajaj2022metro, zoph2022designing, raffel2020exploring, brown2020language,devlin-etal-2019-bert} manifest a clear trend to develop large, foundational models and then fine-tune in on a variety of NLP tasks depending on the use case." />
        </attvalues>
      </edge>
      <edge source="2402.14890" target="1905.00537" id="11916">
        <attvalues>
          <attvalue for="5" value=" This method significantly reduces the resources required for the qualitative evaluation of large NLP models; \item we conduct extensive analysis of the GLUE \cite{wang2018glue}, SuperGLUE \cite{wang2019superglue}, CLUE \cite{xu2020clue}, RussianSuperGLUE \cite{shavrina2020russiansuperglue} using our approach." />
        </attvalues>
      </edge>
      <edge source="2402.14890" target="2004.05986" id="11917">
        <attvalues>
          <attvalue for="5" value=" This method significantly reduces the resources required for the qualitative evaluation of large NLP models; \item we conduct extensive analysis of the GLUE \cite{wang2018glue}, SuperGLUE \cite{wang2019superglue}, CLUE \cite{xu2020clue}, RussianSuperGLUE \cite{shavrina2020russiansuperglue} using our approach." />
        </attvalues>
      </edge>
      <edge source="2402.14890" target="2010.15925" id="11918">
        <attvalues>
          <attvalue for="5" value=" This method significantly reduces the resources required for the qualitative evaluation of large NLP models; \item we conduct extensive analysis of the GLUE \cite{wang2018glue}, SuperGLUE \cite{wang2019superglue}, CLUE \cite{xu2020clue}, RussianSuperGLUE \cite{shavrina2020russiansuperglue} using our approach." />
        </attvalues>
      </edge>
      <edge source="2010.07261" target="1506.06714" id="11919">
        <attvalues>
          <attvalue for="5" value=" Enabling chatbots to indulge in engaging conversations requires massive datasets of human-human conversations \cite{ritter2011data,sordoni2015neural,vinyals2015neural,zhang2018personalizing,zhang2019dialogpt}." />
        </attvalues>
      </edge>
      <edge source="2010.07261" target="1506.05869" id="11920">
        <attvalues>
          <attvalue for="5" value=" Enabling chatbots to indulge in engaging conversations requires massive datasets of human-human conversations \cite{ritter2011data,sordoni2015neural,vinyals2015neural,zhang2018personalizing,zhang2019dialogpt}." />
        </attvalues>
      </edge>
      <edge source="2010.07261" target="1911.00536" id="11922">
        <attvalues>
          <attvalue for="5" value=" Enabling chatbots to indulge in engaging conversations requires massive datasets of human-human conversations \cite{ritter2011data,sordoni2015neural,vinyals2015neural,zhang2018personalizing,zhang2019dialogpt}." />
        </attvalues>
      </edge>
      <edge source="2010.07261" target="1901.05415" id="11923">
        <attvalues>
          <attvalue for="5" value=" \cite{hancock2019learning} treat this feedback as a gold response to the wrong turn and use it as an additional training sample to improve the chatbot." />
        </attvalues>
      </edge>
      <edge source="2010.07261" target="1510.03055" id="11924">
        <attvalues>
          <attvalue for="5" value=" Naive modification of feedback using heuristics like regular expressions would lead to generic responses that are ineffective in improving the dialog ability of chatbots \cite{li2016diversity}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1508.04395" id="11933">
        <attvalues>
          <attvalue for="5" value=" S2S ASR models are designed for directly converting the input speech into transcripts \cite{chan2016listen,bahdanau2016end,chorowski2015attention}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1506.07503" id="11934">
        <attvalues>
          <attvalue for="5" value=" S2S ASR models are designed for directly converting the input speech into transcripts \cite{chan2016listen,bahdanau2016end,chorowski2015attention}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="2007.13876" id="11935">
        <attvalues>
          <attvalue for="5" value=" In this work, we adopt the speech chain reconstruction as a data augmentation method and focus on the FixMatch algorithm \cite{sohn2020fixmatch} which has recently been applied on S2S ASR \cite{weninger2020semi}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1909.09116" id="11936">
        <attvalues>
          <attvalue for="5" value=" In the consistency training paradigm for S2S ASR \cite{weninger2020semi,chen2021semi,zhang2020semi,masumura2020sequence,park2020improved,kahn2020self}, a teacher model trained on transcribed speech is used to produce pseudo transcripts for untranscribed speech." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="2106.08922" id="11938">
        <attvalues>
          <attvalue for="5" value=" Thus, many semi-supervised learning algorithms have been proposed to efficiently train ASR models with the help of untranscribed speech \cite{zhang2020semi,weninger2020semi,masumura2020sequence,chen2021semi,kahn2020self,park2020improved,higuchi2021momentum,xiao2021contrastive}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="2103.05149" id="11939">
        <attvalues>
          <attvalue for="5" value=" Thus, many semi-supervised learning algorithms have been proposed to efficiently train ASR models with the help of untranscribed speech \cite{zhang2020semi,weninger2020semi,masumura2020sequence,chen2021semi,kahn2020self,park2020improved,higuchi2021momentum,xiao2021contrastive}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1412.4864" id="11940">
        <attvalues>
          <attvalue for="5" value=" Consistency regularization \cite{bachman2014learning} is an important principle of semi-supervised learning algorithms." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1610.02242" id="11941">
        <attvalues>
          <attvalue for="5" value=" This principle was originally designed for semi-supervised image classification \cite{samuli2017temporal,sajjadi2016regularization,sohn2020fixmatch,berthelot2019mixmatch} and it has recently been extended to semi-supervised S2S ASR \cite{zhang2020semi,weninger2020semi,masumura2020sequence,chen2021semi,wang2020improving}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1606.04586" id="11942">
        <attvalues>
          <attvalue for="5" value=" This principle was originally designed for semi-supervised image classification \cite{samuli2017temporal,sajjadi2016regularization,sohn2020fixmatch,berthelot2019mixmatch} and it has recently been extended to semi-supervised S2S ASR \cite{zhang2020semi,weninger2020semi,masumura2020sequence,chen2021semi,wang2020improving}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="2001.07685" id="11943">
        <attvalues>
          <attvalue for="5" value=" In this work, we adopt the speech chain reconstruction as a data augmentation method and focus on the FixMatch algorithm \cite{sohn2020fixmatch} which has recently been applied on S2S ASR \cite{weninger2020semi}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1905.02249" id="11944">
        <attvalues>
          <attvalue for="5" value=" This principle was originally designed for semi-supervised image classification \cite{samuli2017temporal,sajjadi2016regularization,sohn2020fixmatch,berthelot2019mixmatch} and it has recently been extended to semi-supervised S2S ASR \cite{zhang2020semi,weninger2020semi,masumura2020sequence,chen2021semi,wang2020improving}." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1904.08779" id="11945">
        <attvalues>
          <attvalue for="5" value=" In the literature, SpecAugment \cite{park2019specaugment} is commonly used to perturb speech features due to its simplicity." />
        </attvalues>
      </edge>
      <edge source="2205.06963" target="1707.04879" id="11946">
        <attvalues>
          <attvalue for="5" value=" In previous work, machine speech chain \cite{tjandra2017listening,tjandra2020machine} was designed to jointly train ASR and TTS by reconstructing unlabeled speech and text data." />
        </attvalues>
      </edge>
      <edge source="2005.14709" target="1706.03762" id="11947">
        <attvalues>
          <attvalue for="5" value=" On the one hand, novel architectures \cite{Vaswani2017} enable efficient unsupervised training on large corpora to obtain expressive contextualised word and sentence representations for a multitude of downstream NLP tasks \cite{Devlin2018}." />
        </attvalues>
      </edge>
      <edge source="2005.14709" target="1508.05326" id="11949">
        <attvalues>
          <attvalue for="5" value=" On the other hand, large-scale datasets \cite{Bowman2015,rajpurkar2016squad,Williams2018} provide sufficient examples to optimise large neural models that are capable of outperforming the human baseline on multiple tasks \cite{Raffel2019ExploringTransformer,Lan2020ALBERT:Representations}." />
        </attvalues>
      </edge>
      <edge source="2005.14709" target="1606.05250" id="11950">
        <attvalues>
          <attvalue for="5" value=" On the other hand, large-scale datasets \cite{Bowman2015,rajpurkar2016squad,Williams2018} provide sufficient examples to optimise large neural models that are capable of outperforming the human baseline on multiple tasks \cite{Raffel2019ExploringTransformer,Lan2020ALBERT:Representations}." />
        </attvalues>
      </edge>
      <edge source="2005.14709" target="1704.05426" id="11951">
        <attvalues>
          <attvalue for="5" value=" On the other hand, large-scale datasets \cite{Bowman2015,rajpurkar2016squad,Williams2018} provide sufficient examples to optimise large neural models that are capable of outperforming the human baseline on multiple tasks \cite{Raffel2019ExploringTransformer,Lan2020ALBERT:Representations}." />
        </attvalues>
      </edge>
      <edge source="2005.14709" target="1909.11942" id="11953">
        <attvalues>
          <attvalue for="5" value=" On the other hand, large-scale datasets \cite{Bowman2015,rajpurkar2016squad,Williams2018} provide sufficient examples to optimise large neural models that are capable of outperforming the human baseline on multiple tasks \cite{Raffel2019ExploringTransformer,Lan2020ALBERT:Representations}." />
        </attvalues>
      </edge>
      <edge source="2005.14709" target="1803.02324" id="11956">
        <attvalues>
          <attvalue for="5" value=" Specifically, training and evaluation data may contain exploitable superficial cues, such as syntactic constructs \cite{mccoy2019right}, specific words \cite{Poliak2018} or sentence length \cite{gururangan2018annotation} that are predictive of the expected output." />
        </attvalues>
      </edge>
      <edge source="2005.14709" target="1907.07355" id="11957">
        <attvalues>
          <attvalue for="5" value=" After having been evaluated on data in which those cues have been removed, the performance of those models deteriorated significantly \cite{mccoy2019right,Niven2019}, showing that they are in fact relying on the existing cues rather than learning to understand meaning or perform inference." />
        </attvalues>
      </edge>
      <edge source="2312.00949" target="2205.01068" id="11958">
        <attvalues>
          <attvalue for="5" value=" Large-scale Language Models (LLMs) have shown exceptional ability in language understanding and generation~\cite{OPT, text2text, LLM_MTL, gpt3}." />
        </attvalues>
      </edge>
      <edge source="2312.00949" target="2303.08774" id="11959">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art models like ChatGPT~\cite{chatgpt} and GPT-4~\cite{gpt4} have garnered a great deal of interest from the academic and industrial communities." />
        </attvalues>
      </edge>
      <edge source="2312.00949" target="2308.10792" id="11961">
        <attvalues>
          <attvalue for="5" value=" Additional fine-tuning of LLMs on a dataset of instructions is called Instruction-Tuning; this technique has become ubiquitous due to its efficiency~\cite{zhang2023instruction_survey}." />
        </attvalues>
      </edge>
      <edge source="2312.00949" target="2106.09685" id="11962">
        <attvalues>
          <attvalue for="5" value=" Such methods are quite sensitive to the choice of hyperparameters~\cite{hu2021lora, Valipour2022DyLoRAPT}." />
        </attvalues>
      </edge>
      <edge source="2406.15809" target="1910.14142" id="11963">
        <attvalues>
          <attvalue for="5" value=" In {extractive summarization}, the algorithm selects a subset representative of the original text \cite{xu-etal-2020-discourse, zhong-etal-2020-extractive, zhang-etal-2022-hegel, zhang-etal-2023-diffusum}." />
        </attvalues>
      </edge>
      <edge source="2406.15809" target="2309.09558" id="11964">
        <attvalues>
          <attvalue for="5" value=" Plus, summaries generated by LLMs showcase high coherence and are overwhelmingly preferred by human evaluators over other baseline algorithms~\cite{pu2023summarization, liu2023learning}." />
        </attvalues>
      </edge>
      <edge source="2005.13978" target="1409.0473" id="11967">
        <attvalues>
          <attvalue for="5" value=" On the one hand, the introduction of neural machine translation (NMT) has significantly advanced the field \cite{bahdanau+al-2014-nmt}, continually producing state-of-the-art translation accuracy." />
        </attvalues>
      </edge>
      <edge source="2005.13978" target="1312.6114" id="11968">
        <attvalues>
          <attvalue for="5" value=" In this paper, we focus on improving Variational NMT (VNMT) \cite{zhang-etal-2016-variational-neural}: a family of LV-NMT models that relies on the amortized variational method \cite{DBLP:journals/corr/KingmaW13} for inference." />
        </attvalues>
      </edge>
      <edge source="2005.13978" target="1505.05770" id="11969">
        <attvalues>
          <attvalue for="5" value=" (1) We employ variational distributions based on normalizing flows \cite{rezende15}, instead of uni-modal Gaussian." />
        </attvalues>
      </edge>
      <edge source="2005.13978" target="1706.03762" id="11970">
        <attvalues>
          <attvalue for="5" value=" (2) We employ the Transformer architecture \cite{transformer}, including Transformer-Big, as our VNMT's generator network." />
        </attvalues>
      </edge>
      <edge source="2408.06186" target="1904.09751" id="11971">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{diversity1} proposes a set of metrics for evaluating the diversity of generated text, including self-BLEU, distinct $n$-grams, and entropy." />
        </attvalues>
      </edge>
      <edge source="2408.06186" target="1906.06362" id="11972">
        <attvalues>
          <attvalue for="5" value=" To the best of our knowledge, the only black-box diversity improvement algorithm was developed and discussed in \cite{ippolito}, who suggested oversampling, clustering the samples into much fewer clusters using an approach such as K-Means, and then only taking the centroid from each cluster - we compare to this baseline in our experiments." />
        </attvalues>
      </edge>
      <edge source="2408.06186" target="2404.16807" id="11973">
        <attvalues>
          <attvalue for="5" value=" More broadly, there has been work studying diversity for reasoning~\cite{naik2023diversity,zhang2024improving}, and improving diversity of recommender systems~\cite{carraro2024enhancing}." />
        </attvalues>
      </edge>
      <edge source="2408.06186" target="2401.11506" id="11974">
        <attvalues>
          <attvalue for="5" value=" More broadly, there has been work studying diversity for reasoning~\cite{naik2023diversity,zhang2024improving}, and improving diversity of recommender systems~\cite{carraro2024enhancing}." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="2101.09624" id="11975">
        <attvalues>
          <attvalue for="5" value=" Recent advances in SD systems are outlined in \cite{park2022review} and the independent module optimized SD systems typically consists of the following main sub-tasks: (a) segment the input audio into speech segments using a Voice activity detector (VAD), (b) generate speaker segments from the speech segments by either using a uniform window size \cite{32wang2018speaker,zhang2019fully,garcia2017speaker} or by detecting speaker turns \cite{yin2018neural,park2018multimodal,xia2022turn}, (c) extract speaker embeddings \cite{li2017deep,snyder2018x,32wang2018speaker,dawalatabad2021ecapa} for each of the speaker segments and (d) cluster the resulting speaker embeddings using clustering algorithms like Spectral Clustering \cite{32wang2018speaker}, Agglomerative Hierarchical Clustering \cite{garcia2017speaker} among others." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="1810.04719" id="11977">
        <attvalues>
          <attvalue for="5" value=" Recent advances in SD systems are outlined in \cite{park2022review} and the independent module optimized SD systems typically consists of the following main sub-tasks: (a) segment the input audio into speech segments using a Voice activity detector (VAD), (b) generate speaker segments from the speech segments by either using a uniform window size \cite{32wang2018speaker,zhang2019fully,garcia2017speaker} or by detecting speaker turns \cite{yin2018neural,park2018multimodal,xia2022turn}, (c) extract speaker embeddings \cite{li2017deep,snyder2018x,32wang2018speaker,dawalatabad2021ecapa} for each of the speaker segments and (d) cluster the resulting speaker embeddings using clustering algorithms like Spectral Clustering \cite{32wang2018speaker}, Agglomerative Hierarchical Clustering \cite{garcia2017speaker} among others." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="1805.10731" id="11978">
        <attvalues>
          <attvalue for="5" value=" There have been a handful of works \cite{park2018multimodal,xia2022turn, park2020speaker,shafey2019joint,india2023language,flemotomos2019linguistically} which leverage the ASR transcripts to infuse lexical information in the SD module." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="2109.11641" id="11979">
        <attvalues>
          <attvalue for="5" value=" There have been a handful of works \cite{park2018multimodal,xia2022turn, park2020speaker,shafey2019joint,india2023language,flemotomos2019linguistically} which leverage the ASR transcripts to infuse lexical information in the SD module." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="1705.02304" id="11980">
        <attvalues>
          <attvalue for="5" value=" Recent advances in SD systems are outlined in \cite{park2022review} and the independent module optimized SD systems typically consists of the following main sub-tasks: (a) segment the input audio into speech segments using a Voice activity detector (VAD), (b) generate speaker segments from the speech segments by either using a uniform window size \cite{32wang2018speaker,zhang2019fully,garcia2017speaker} or by detecting speaker turns \cite{yin2018neural,park2018multimodal,xia2022turn}, (c) extract speaker embeddings \cite{li2017deep,snyder2018x,32wang2018speaker,dawalatabad2021ecapa} for each of the speaker segments and (d) cluster the resulting speaker embeddings using clustering algorithms like Spectral Clustering \cite{32wang2018speaker}, Agglomerative Hierarchical Clustering \cite{garcia2017speaker} among others." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="2104.01466" id="11981">
        <attvalues>
          <attvalue for="5" value=" Recent advances in SD systems are outlined in \cite{park2022review} and the independent module optimized SD systems typically consists of the following main sub-tasks: (a) segment the input audio into speech segments using a Voice activity detector (VAD), (b) generate speaker segments from the speech segments by either using a uniform window size \cite{32wang2018speaker,zhang2019fully,garcia2017speaker} or by detecting speaker turns \cite{yin2018neural,park2018multimodal,xia2022turn}, (c) extract speaker embeddings \cite{li2017deep,snyder2018x,32wang2018speaker,dawalatabad2021ecapa} for each of the speaker segments and (d) cluster the resulting speaker embeddings using clustering algorithms like Spectral Clustering \cite{32wang2018speaker}, Agglomerative Hierarchical Clustering \cite{garcia2017speaker} among others." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="1907.05337" id="11982">
        <attvalues>
          <attvalue for="5" value=" \cite{shafey2019joint} modeled SD and ASR jointly but is confined to 2 speakers with specific distinct roles." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="1911.07994" id="11983">
        <attvalues>
          <attvalue for="5" value=" There have been a handful of works \cite{park2018multimodal,xia2022turn, park2020speaker,shafey2019joint,india2023language,flemotomos2019linguistically} which leverage the ASR transcripts to infuse lexical information in the SD module." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="2004.06756" id="11984">
        <attvalues>
          <attvalue for="5" value=" \cite{park2020speaker} made use of turn probabilities from lexical cues in the clustering stage by enhancing the adjacency matrix." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="1907.11692" id="11986">
        <attvalues>
          <attvalue for="5" value=" This SEC module makes use of the any of the readily available pre-trained LMs \cite{devlin2018bert,liu2019roberta,yang2019xlnet,brown2020language} to infuse the lexical knowledge to correct speaker errors while also leveraging speaker scores from the SD system to prevent over-corrections." />
        </attvalues>
      </edge>
      <edge source="2306.09313" target="2005.14165" id="11988">
        <attvalues>
          <attvalue for="5" value=" This SEC module makes use of the any of the readily available pre-trained LMs \cite{devlin2018bert,liu2019roberta,yang2019xlnet,brown2020language} to infuse the lexical knowledge to correct speaker errors while also leveraging speaker scores from the SD system to prevent over-corrections." />
        </attvalues>
      </edge>
      <edge source="2102.12060" target="1812.01193" id="11989">
        <attvalues>
          <attvalue for="5" value=" In particular, human justifications are used for three goals: (i) to aid models with additional training supervision \cite{zaidan-etal-2007-using}, (ii) to train interpretable models that explain their own predictions \cite{camburu2018snli}, and (iii) to evaluate plausibility of model-generated explanations by measuring their agreement with human explanations \cite{deyoung-etal-2020-eraser}." />
        </attvalues>
      </edge>
      <edge source="2102.12060" target="1911.03429" id="11990">
        <attvalues>
          <attvalue for="5" value=" In particular, human justifications are used for three goals: (i) to aid models with additional training supervision \cite{zaidan-etal-2007-using}, (ii) to train interpretable models that explain their own predictions \cite{camburu2018snli}, and (iii) to evaluate plausibility of model-generated explanations by measuring their agreement with human explanations \cite{deyoung-etal-2020-eraser}." />
        </attvalues>
      </edge>
      <edge source="2102.12060" target="1702.08608" id="11991">
        <attvalues>
          <attvalue for="5" value=" It complements other explainable AI ( XAI) surveys and critical retrospectives that focus on definitions, methods, and/or evaluation \cite{doshi2017towards, biran2017explanation, lipton2018mythos, adadi-berrada-peeking-2018, Ras2018, hoffman2018metrics, Gilpin2018ExplainingEA, Yang2019EvaluatingEW, clinciu-hastie-2019-survey, Guidotti2019ASO, miller2019explanation, Verma2020CounterfactualEF, BARREDOARRIETA202082, Murdoch22071, jacovi-goldberg-2020-towards, Burkart2021ASO}, but not on datasets." />
        </attvalues>
      </edge>
      <edge source="2102.12060" target="2011.03870" id="11992">
        <attvalues>
          <attvalue for="5" value=" Datasets and methods for explaining fact checking \cite{kotonya-toni-2020-survey} and reading comprehension \cite{Thayaparan2020ASO} have been reviewed; we are the first to review all datasets with textual explanations regardless of task, comprehensively categorize them into three distinct classes, and provide critical retrospectives and best-practice recommendations." />
        </attvalues>
      </edge>
      <edge source="2102.12060" target="2010.00389" id="11993">
        <attvalues>
          <attvalue for="5" value=" Datasets and methods for explaining fact checking \cite{kotonya-toni-2020-survey} and reading comprehension \cite{Thayaparan2020ASO} have been reviewed; we are the first to review all datasets with textual explanations regardless of task, comprehensively categorize them into three distinct classes, and provide critical retrospectives and best-practice recommendations." />
        </attvalues>
      </edge>
      <edge source="2311.08380" target="2302.01398" id="11994">
        <attvalues>
          <attvalue for="5" value=" Recent work \cite{garcia2023unreasonable, suzgun-etal-2023-follow, yang-2023-thesis} has shown that MBR decoding can significantly boost the translation performance of MLLMs \cite{lin-etal-2022-shot, muennighoff-etal-2023-crosslingual, zeng2023glm130b}, outperforming greedy decoding and beam search." />
        </attvalues>
      </edge>
      <edge source="2311.08380" target="2305.18290" id="11995">
        <attvalues>
          <attvalue for="5" value=" We propose a novel self-supervised fine-tuning method based on DPO~\cite{DBLP:conf/nips/RafailovSMMEF23}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2005.14165" id="12013">
        <attvalues>
          <attvalue for="5" value=" The large language model (LLM) has demonstrated its powerful capability in text generation \cite{brown2020language,openai2023gpt4,touvron2023llama2}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2303.08774" id="12014">
        <attvalues>
          <attvalue for="5" value=" The large language model (LLM) has demonstrated its powerful capability in text generation \cite{brown2020language,openai2023gpt4,touvron2023llama2}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2307.09288" id="12015">
        <attvalues>
          <attvalue for="5" value=" The large language model (LLM) has demonstrated its powerful capability in text generation \cite{brown2020language,openai2023gpt4,touvron2023llama2}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2001.08361" id="12016">
        <attvalues>
          <attvalue for="5" value=" It can auto-regressively generate expressive and diverse text sequences, especially when scaled to a larger model with more training data \cite{kaplan2020scaling}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2305.13009" id="12017">
        <attvalues>
          <attvalue for="5" value=" This breakthrough has also attracted widespread attention from the speech domain, inspiring the next-gen speech generation paradigm, eg speech language model \cite{hassid2024textually,wu2023speechgen}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2305.07243" id="12019">
        <attvalues>
          <attvalue for="5" value=" Then, we can combine speech and text together for LLM training and inference, and achieve conditional speech generation, eg text-to-speech (TTS) \cite{betker2023better,lajszczak2024base}, voice conversion (VC) \cite{wang2023lm,kuan2023towards}, and speech-to-speech translation \cite{huang2023speech}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2402.08093" id="12020">
        <attvalues>
          <attvalue for="5" value=" Then, we can combine speech and text together for LLM training and inference, and achieve conditional speech generation, eg text-to-speech (TTS) \cite{betker2023better,lajszczak2024base}, voice conversion (VC) \cite{wang2023lm,kuan2023towards}, and speech-to-speech translation \cite{huang2023speech}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2306.10521" id="12021">
        <attvalues>
          <attvalue for="5" value=" Then, we can combine speech and text together for LLM training and inference, and achieve conditional speech generation, eg text-to-speech (TTS) \cite{betker2023better,lajszczak2024base}, voice conversion (VC) \cite{wang2023lm,kuan2023towards}, and speech-to-speech translation \cite{huang2023speech}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2309.14324" id="12022">
        <attvalues>
          <attvalue for="5" value=" Then, we can combine speech and text together for LLM training and inference, and achieve conditional speech generation, eg text-to-speech (TTS) \cite{betker2023better,lajszczak2024base}, voice conversion (VC) \cite{wang2023lm,kuan2023towards}, and speech-to-speech translation \cite{huang2023speech}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2312.13585" id="12023">
        <attvalues>
          <attvalue for="5" value=" Then, we can combine speech and text together for LLM training and inference, and achieve conditional speech generation, eg text-to-speech (TTS) \cite{betker2023better,lajszczak2024base}, voice conversion (VC) \cite{wang2023lm,kuan2023towards}, and speech-to-speech translation \cite{huang2023speech}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="1711.00937" id="12024">
        <attvalues>
          <attvalue for="5" value=" Vector-quantized variational autoencoder (VQ-VAE) \cite{vqvae}, as the mainstream model for discrete representation learning of speech, has been well applied in multiple tasks, including speech coding \cite{garbacea2019low}, VC \cite{wang2021vqmivc}, and TTS \cite{Du2022VQTTSHT}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="1910.06464" id="12025">
        <attvalues>
          <attvalue for="5" value=" Vector-quantized variational autoencoder (VQ-VAE) \cite{vqvae}, as the mainstream model for discrete representation learning of speech, has been well applied in multiple tasks, including speech coding \cite{garbacea2019low}, VC \cite{wang2021vqmivc}, and TTS \cite{Du2022VQTTSHT}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2204.00768" id="12027">
        <attvalues>
          <attvalue for="5" value=" Vector-quantized variational autoencoder (VQ-VAE) \cite{vqvae}, as the mainstream model for discrete representation learning of speech, has been well applied in multiple tasks, including speech coding \cite{garbacea2019low}, VC \cite{wang2021vqmivc}, and TTS \cite{Du2022VQTTSHT}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2309.00169" id="12028">
        <attvalues>
          <attvalue for="5" value=" It also demonstrates great potential in speech tokenization over conventional k-means-based approaches \cite{huang2023repcodec}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2305.08842" id="12029">
        <attvalues>
          <attvalue for="5" value=" However, in practice, training a high-quality VQ-VAE-based speech tokenizer is also challenging due to the problem of ``index collapse&quot; \cite{huh2023improvedvqste}." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2210.13438" id="12030">
        <attvalues>
          <attvalue for="5" value=" Some works turn to exploit multi-sequence discrete representations, eg MSMC-VQ \cite{guo2023msmc} and RVQ \cite{DBLP:journals/corr/abs-2210-13438,wang2023neural,zhang2023speechtokenizer}, to avoid learning large codebooks, but they cannot adapt LLMs only modeling single discrete sequences directly, introducing more challenges." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2301.02111" id="12031">
        <attvalues>
          <attvalue for="5" value=" Some works turn to exploit multi-sequence discrete representations, eg MSMC-VQ \cite{guo2023msmc} and RVQ \cite{DBLP:journals/corr/abs-2210-13438,wang2023neural,zhang2023speechtokenizer}, to avoid learning large codebooks, but they cannot adapt LLMs only modeling single discrete sequences directly, introducing more challenges." />
        </attvalues>
      </edge>
      <edge source="2406.02940" target="2308.16692" id="12032">
        <attvalues>
          <attvalue for="5" value=" Some works turn to exploit multi-sequence discrete representations, eg MSMC-VQ \cite{guo2023msmc} and RVQ \cite{DBLP:journals/corr/abs-2210-13438,wang2023neural,zhang2023speechtokenizer}, to avoid learning large codebooks, but they cannot adapt LLMs only modeling single discrete sequences directly, introducing more challenges." />
        </attvalues>
      </edge>
      <edge source="1910.07973" target="1310.4546" id="12033">
        <attvalues>
          <attvalue for="5" value=" Since the introduction of pre-trained word embeddings such as word2vec \cite{Mikolov} and GloVe \cite{Pennington2014}, a lot of efforts have been devoted to developing universal sentence embeddings." />
        </attvalues>
      </edge>
      <edge source="1910.07973" target="1705.02364" id="12034">
        <attvalues>
          <attvalue for="5" value=" Recent work \cite{Conneau2017} has shown that models trained in supervised fashion on datasets like Stanford Natural Language Inference (SNLI) corpus \cite{Bowman2015a} can consistently outperform unsupervised methods like SkipThought vectors \cite{Kiros2015}." />
        </attvalues>
      </edge>
      <edge source="1910.07973" target="1508.05326" id="12035">
        <attvalues>
          <attvalue for="5" value=" Recent work \cite{Conneau2017} has shown that models trained in supervised fashion on datasets like Stanford Natural Language Inference (SNLI) corpus \cite{Bowman2015a} can consistently outperform unsupervised methods like SkipThought vectors \cite{Kiros2015}." />
        </attvalues>
      </edge>
      <edge source="1910.07973" target="1506.06726" id="12036">
        <attvalues>
          <attvalue for="5" value=" Recent work \cite{Conneau2017} has shown that models trained in supervised fashion on datasets like Stanford Natural Language Inference (SNLI) corpus \cite{Bowman2015a} can consistently outperform unsupervised methods like SkipThought vectors \cite{Kiros2015}." />
        </attvalues>
      </edge>
      <edge source="1910.07973" target="1706.03762" id="12038">
        <attvalues>
          <attvalue for="5" value=" More recently, Universal Sentence Encoder \cite{Cer2018} equipped with the Transformer \cite{Vaswani2017} as the encoder, co-trained on a large amount of unsupervised training data and SNLI corpus, has demonstrated surprisingly good performance with minimal amounts of supervised training data for a transfer task." />
        </attvalues>
      </edge>
      <edge source="2406.10325" target="2110.07592" id="12041">
        <attvalues>
          <attvalue for="5" value=" DeToxy~\cite{ghosh2021detoxy} and IEMOCAP~\cite{busso2008iemocap} are public monolingual datasets that are extremely small and lack real-world characteristics given their construction." />
        </attvalues>
      </edge>
      <edge source="2406.10325" target="2006.11477" id="12042">
        <attvalues>
          <attvalue for="5" value=" Examples of such strategies include developing a custom attention architecture~\cite{yousefi2021audio}, implementing multi-task learning~\cite{roblox_mtl}, or utilizing pre-trained speech encoders~\cite{baevski2020wav2vec,Duquenne:2023:sonar_arxiv}." />
        </attvalues>
      </edge>
      <edge source="2406.10325" target="2110.13900" id="12043">
        <attvalues>
          <attvalue for="5" value="0~\cite{baevski2020wav2vec, nada2023lightweight, adima_gupta} and WavLM~\cite{chen2022wavlm, roblox_mtl}as the base encoder." />
        </attvalues>
      </edge>
      <edge source="2406.10325" target="2401.05060" id="12044">
        <attvalues>
          <attvalue for="5" value=" For multilingual datasets, availability is much more limited~\cite{costajussà2024mutox}." />
        </attvalues>
      </edge>
      <edge source="2406.10325" target="2308.11466" id="12045">
        <attvalues>
          <attvalue for="5" value=" Examples of such strategies include developing a custom attention architecture~\cite{yousefi2021audio}, implementing multi-task learning~\cite{roblox_mtl}, or utilizing pre-trained speech encoders~\cite{baevski2020wav2vec,Duquenne:2023:sonar_arxiv}." />
        </attvalues>
      </edge>
      <edge source="2406.10325" target="2401.10653" id="12046">
        <attvalues>
          <attvalue for="5" value=" One approach even involves feeding speech and text into the classifier at inference time to improve classification performance~\cite{mandal2024attentive}." />
        </attvalues>
      </edge>
      <edge source="2406.10325" target="2205.02444" id="12047">
        <attvalues>
          <attvalue for="5" value=" Machine Translation architectures such as ConST~\cite{ye2022cross} have already benefited from this phenomenon to improve performance in their domain." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="1911.06311" id="12048">
        <attvalues>
          <attvalue for="5" value=" This task has attracted significant attention from the database community, and many solutions based on deep learning techniques, especially pre-trained Language Models (PLMs)~\cite{DBLP:journals/pvldb/ZhangSLHDT20,DBLP:conf/sigmod/SuharaL0ZDCT22,DBLP:journals/pacmmod/MiaoW23}, have been developed to improve overall performance." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="2104.01785" id="12049">
        <attvalues>
          <attvalue for="5" value=" This task has attracted significant attention from the database community, and many solutions based on deep learning techniques, especially pre-trained Language Models (PLMs)~\cite{DBLP:journals/pvldb/ZhangSLHDT20,DBLP:conf/sigmod/SuharaL0ZDCT22,DBLP:journals/pacmmod/MiaoW23}, have been developed to improve overall performance." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="2206.04853" id="12050">
        <attvalues>
          <attvalue for="5" value=" Semantic type detection is an important task in many data preparation applications, such as data cleaning, schema matching, entity resolution and data discovery~\cite{DBLP:journals/pvldb/ZhangSLHDT20,DBLP:conf/sigmod/SuharaL0ZDCT22,DBLP:conf/sigmod/Wang0HK22,DBLP:conf/deem/Wang022}." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="1605.07723" id="12051">
        <attvalues>
          <attvalue for="5" value=" We argue that a weak supervision approach, such as data programming~\cite{DBLP:conf/nips/RatnerSWSR16}, is a good solution to reduce the burdens of training data annotation." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="1711.10160" id="12052">
        <attvalues>
          <attvalue for="5" value=" \textsf{Snorkel}~\cite{DBLP:journals/pvldb/RatnerBEFWR17} proposed a probabilistic model to aggregate the user-written LFs." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="2303.08774" id="12053">
        <attvalues>
          <attvalue for="5" value=" The recent advances in the era of Large Language Model (LLM), such as GPT-4~\cite{DBLP:journals/corr/abs-2303-08774} and LLaMA~\cite{DBLP:journals/corr/abs-2302-13971}, have shown powerful capability in various tasks in different fields." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="2311.00739" id="12055">
        <attvalues>
          <attvalue for="5" value=" Some recent efforts~\cite{DBLP:journals/corr/abs-2311-00739,DBLP:conf/acl/ZhangYSSZ22} have been made to harness LLMs for automate the generation of LFs for NLP tasks." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="2106.07258" id="12056">
        <attvalues>
          <attvalue for="5" value=" For example, the number of semantic labels in the Gittable~\cite{DBLP:journals/pacmmod/HulsebosDG23} and TURL~\cite{DBLP:journals/pvldb/DengSL0020} corpus is 835 and 255, respectively." />
        </attvalues>
      </edge>
      <edge source="2408.16173" target="2109.11377" id="12058">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, the task with the largest number of labels in the WRENCH benchmarking~\cite{DBLP:conf/nips/ZhangYNWYYR21} only has 18 class labels." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="1704.04368" id="12059">
        <attvalues>
          <attvalue for="5" value=" The field of generic summarization \cite{see-etal-2017-get,gehrmann-etal-2018-bottom,liu-lapata-2019-text} has made significant progress in recent years, thanks to the development of generative deep neural models \cite{NIPS2014_a14ac55a,NIPS2017_3f5ee243} and the availability of large-scale training data \cite{nallapati-etal-2016-abstractive,narayan-etal-2018-dont,zhu-etal-2021-mediasum}." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="1808.10792" id="12060">
        <attvalues>
          <attvalue for="5" value=" The field of generic summarization \cite{see-etal-2017-get,gehrmann-etal-2018-bottom,liu-lapata-2019-text} has made significant progress in recent years, thanks to the development of generative deep neural models \cite{NIPS2014_a14ac55a,NIPS2017_3f5ee243} and the availability of large-scale training data \cite{nallapati-etal-2016-abstractive,narayan-etal-2018-dont,zhu-etal-2021-mediasum}." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="1409.3215" id="12062">
        <attvalues>
          <attvalue for="5" value=" The field of generic summarization \cite{see-etal-2017-get,gehrmann-etal-2018-bottom,liu-lapata-2019-text} has made significant progress in recent years, thanks to the development of generative deep neural models \cite{NIPS2014_a14ac55a,NIPS2017_3f5ee243} and the availability of large-scale training data \cite{nallapati-etal-2016-abstractive,narayan-etal-2018-dont,zhu-etal-2021-mediasum}." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="1706.03762" id="12063">
        <attvalues>
          <attvalue for="5" value=" The field of generic summarization \cite{see-etal-2017-get,gehrmann-etal-2018-bottom,liu-lapata-2019-text} has made significant progress in recent years, thanks to the development of generative deep neural models \cite{NIPS2014_a14ac55a,NIPS2017_3f5ee243} and the availability of large-scale training data \cite{nallapati-etal-2016-abstractive,narayan-etal-2018-dont,zhu-etal-2021-mediasum}." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="1602.06023" id="12064">
        <attvalues>
          <attvalue for="5" value=" The field of generic summarization \cite{see-etal-2017-get,gehrmann-etal-2018-bottom,liu-lapata-2019-text} has made significant progress in recent years, thanks to the development of generative deep neural models \cite{NIPS2014_a14ac55a,NIPS2017_3f5ee243} and the availability of large-scale training data \cite{nallapati-etal-2016-abstractive,narayan-etal-2018-dont,zhu-etal-2021-mediasum}." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="1808.08745" id="12065">
        <attvalues>
          <attvalue for="5" value=" The field of generic summarization \cite{see-etal-2017-get,gehrmann-etal-2018-bottom,liu-lapata-2019-text} has made significant progress in recent years, thanks to the development of generative deep neural models \cite{NIPS2014_a14ac55a,NIPS2017_3f5ee243} and the availability of large-scale training data \cite{nallapati-etal-2016-abstractive,narayan-etal-2018-dont,zhu-etal-2021-mediasum}." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="2103.06410" id="12066">
        <attvalues>
          <attvalue for="5" value=" The field of generic summarization \cite{see-etal-2017-get,gehrmann-etal-2018-bottom,liu-lapata-2019-text} has made significant progress in recent years, thanks to the development of generative deep neural models \cite{NIPS2014_a14ac55a,NIPS2017_3f5ee243} and the availability of large-scale training data \cite{nallapati-etal-2016-abstractive,narayan-etal-2018-dont,zhu-etal-2021-mediasum}." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="1704.08300" id="12067">
        <attvalues>
          <attvalue for="5" value=" Most of the available QFS corpora \cite{dang-2006-duc,dang2006duc,nema-etal-2017-diversity,baumel2016topic,zhong-etal-2021-qmsum} contain only a few thousand documents or less, which is insufficient for training a robust neural model." />
        </attvalues>
      </edge>
      <edge source="2305.13086" target="2104.05938" id="12068">
        <attvalues>
          <attvalue for="5" value=" Furthermore, it may necessitate a meticulous definition of the query scheme based on the domain of documents \cite{zhong-etal-2021-qmsum}." />
        </attvalues>
      </edge>
      <edge source="2106.10003" target="1703.10135" id="12070">
        <attvalues>
          <attvalue for="5" value=" Recent advancement of end-to-end neural TTS has demonstrated that it can synthesize very natural, human-like speech~\cite{wang2017tacotron, ping2018deep, shen2018natural, fengyu2019improving, guangzhi2020generating}." />
        </attvalues>
      </edge>
      <edge source="2106.10003" target="1409.3215" id="12071">
        <attvalues>
          <attvalue for="5" value=" The trained neural TTS models usually consist of an encoder-decoder neural network~\cite{sutskever2014sequence, bahdanau2015neural} which can map a text sequence to a sequence of speech frames." />
        </attvalues>
      </edge>
      <edge source="2106.10003" target="1806.04558" id="12072">
        <attvalues>
          <attvalue for="5" value=" Extensions of these models have shown that speech styles (eg, speaker identity, emotion and prosody), which are essential for expressive and diverse voice generation, can be also modelled and controlled~\cite{Ye2018Transfer, wu2019end-to-end, stanton2018predicting, skerry2018towards, liu2020expressive, lei2021fine}." />
        </attvalues>
      </edge>
      <edge source="2106.10003" target="1804.02135" id="12073">
        <attvalues>
          <attvalue for="5" value=" Currently most neural TTS systems~\cite{akuzawa2018expressive, xiaochun2019learning, hsu2018hierarchical, habib2020semi-supervised, guangzhi2020fully} are modelled by using a corpus of a single expressive style." />
        </attvalues>
      </edge>
      <edge source="2106.10003" target="1803.09017" id="12074">
        <attvalues>
          <attvalue for="5" value="~\cite{bian2019multi} introduce a multi-reference encoder to GST~\cite{wang2018style} and adopt an intercross training scheme, which together ensure that each sub-encoder of the multi-reference encoder independently disentangles and controls a specific style." />
        </attvalues>
      </edge>
      <edge source="2106.10003" target="1812.04342" id="12075">
        <attvalues>
          <attvalue for="5" value=" Recently, neural TTS model with global style tokens (GST)~\cite{wang2018style, li2021controllable} or a variational autoencoder (VAE)~\cite{zhang2019learning} has received interests for controlling and transferring speech styles." />
        </attvalues>
      </edge>
      <edge source="2106.10003" target="1910.11958" id="12076">
        <attvalues>
          <attvalue for="5" value=" Though ~\cite{whitehill2020multi} improves performance of style transfer, it suffers a limitation, similar to ~\cite{bian2019multi}, that can only transfer the style seen during training, and is inadequate to transfer the speech to a target style from a new speaker with an unknown, arbitrary style, thus narrowing down the applicable scenarios of neural TTS systems." />
        </attvalues>
      </edge>
      <edge source="2106.10003" target="1606.04934" id="12077">
        <attvalues>
          <attvalue for="5" value=" The main contributions of this paper are summarized as follows: \begin{itemize} \item To facilitate seen and unseen style transfer in end-to-end neural TTS, we first adopt an inverse autoregressive flow (IAF) structure~\cite{kingma2016improving} to improve the style representation, and then propose four different loss functions to together make sure the seen and unseen style transfer: 1) using a reconstruction loss to measure the distortions in both source and target reconstructions; 2) injecting an adversarial loss to ``fool&quot; a well-trained discriminator; 3) introducing a style distortion loss to measure the expected style loss after the transfer; 4) incorporating a cycle consistency loss to preserve the speaker identity of the source after the transfer." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1905.04226" id="12078">
        <attvalues>
          <attvalue for="5" value=" Neural network language models (NNLMs) play critical roles in automatic speech recognition (ASR) systems \cite{mikolov2010recurrent,chen2015improving,xu2018neural,irie2019language}." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="2103.05081" id="12079">
        <attvalues>
          <attvalue for="5" value=" For conventional ASR models, NNLMs are widely used in the second pass via $N$-best or lattice rescoring \cite{liu2014efficient,xu2018pruned,li2021parallelizable}." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1211.3711" id="12080">
        <attvalues>
          <attvalue for="5" value=" For end-to-end ASR \cite{graves2006connectionist,graves2012sequence,chan2016listen}, although linguistic information is implicitly learned, NNLMs can still further improve accuracy by fusion in first-pass decoding \cite{kannan2018analysis, kim2021improved} or second-pass rescoring." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1712.01996" id="12081">
        <attvalues>
          <attvalue for="5" value=" For end-to-end ASR \cite{graves2006connectionist,graves2012sequence,chan2016listen}, although linguistic information is implicitly learned, NNLMs can still further improve accuracy by fusion in first-pass decoding \cite{kannan2018analysis, kim2021improved} or second-pass rescoring." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1811.06621" id="12082">
        <attvalues>
          <attvalue for="5" value=" With the latest advances in mobile technologies, hosting an ASR system entirely on-device has important implications from a reliability, latency, and particularly privacy perspective, and has become an active area of research and industrial applications \cite{he2019streaming}." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1610.02527" id="12083">
        <attvalues>
          <attvalue for="5" value=" To resolve this privacy concern, federated learning (FL) \cite{konevcny2016federated, konevcny2016federated2, mcmahan2017communication}, a distributed learning technique, has been proposed and applied in many fields including recommendation \cite{chen2018federated}, keyboard suggestion \cite{arnold2016suggesting, ji2019learning}, keyword spotting \cite{leroy2019federated}, health care \cite{xu2019federated}, and more recently, ASR including hybrid acoustic models and end-to-end models \cite{dimitriadis2020federated, guliani2021training, cui2021federated}." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1802.07876" id="12084">
        <attvalues>
          <attvalue for="5" value=" To resolve this privacy concern, federated learning (FL) \cite{konevcny2016federated, konevcny2016federated2, mcmahan2017communication}, a distributed learning technique, has been proposed and applied in many fields including recommendation \cite{chen2018federated}, keyboard suggestion \cite{arnold2016suggesting, ji2019learning}, keyword spotting \cite{leroy2019federated}, health care \cite{xu2019federated}, and more recently, ASR including hybrid acoustic models and end-to-end models \cite{dimitriadis2020federated, guliani2021training, cui2021federated}." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1810.05512" id="12085">
        <attvalues>
          <attvalue for="5" value=" To resolve this privacy concern, federated learning (FL) \cite{konevcny2016federated, konevcny2016federated2, mcmahan2017communication}, a distributed learning technique, has been proposed and applied in many fields including recommendation \cite{chen2018federated}, keyboard suggestion \cite{arnold2016suggesting, ji2019learning}, keyword spotting \cite{leroy2019federated}, health care \cite{xu2019federated}, and more recently, ASR including hybrid acoustic models and end-to-end models \cite{dimitriadis2020federated, guliani2021training, cui2021federated}." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1911.06270" id="12086">
        <attvalues>
          <attvalue for="5" value=" To resolve this privacy concern, federated learning (FL) \cite{konevcny2016federated, konevcny2016federated2, mcmahan2017communication}, a distributed learning technique, has been proposed and applied in many fields including recommendation \cite{chen2018federated}, keyboard suggestion \cite{arnold2016suggesting, ji2019learning}, keyword spotting \cite{leroy2019federated}, health care \cite{xu2019federated}, and more recently, ASR including hybrid acoustic models and end-to-end models \cite{dimitriadis2020federated, guliani2021training, cui2021federated}." />
        </attvalues>
      </edge>
      <edge source="2110.10026" target="1812.07108" id="12087">
        <attvalues>
          <attvalue for="5" value=" Federated language modeling has been well explored in mobile keyboard suggestion where sentences typed by users provide instant labeled data for supervised learning \cite{ji2019learning}." />
        </attvalues>
      </edge>
      <edge source="2210.05675" target="2205.05055" id="12088">
        <attvalues>
          <attvalue for="5" value=" Transformer-based architectures have an impressive ability to use both information stored in weights during training (``in-weights learning''), and information stored only in the inputs provided at inference time (without any gradient updates to the weights of the model; ``in-context learning'') \cite{chan_data_2022}." />
        </attvalues>
      </edge>
      <edge source="2210.05675" target="2005.14165" id="12089">
        <attvalues>
          <attvalue for="5" value=" In-context learning on pretrained models enables learning efficiently from a few examples (``few-shot learning&quot;) \cite{brown_language_2020}, or even efficiently compressing a large dataset (``prompt tuning&quot;) \cite{li_prefix-tuning_2021, lester_power_2021, sun_black-box_2022}." />
        </attvalues>
      </edge>
      <edge source="2210.05675" target="2101.00190" id="12090">
        <attvalues>
          <attvalue for="5" value=" In-context learning on pretrained models enables learning efficiently from a few examples (``few-shot learning&quot;) \cite{brown_language_2020}, or even efficiently compressing a large dataset (``prompt tuning&quot;) \cite{li_prefix-tuning_2021, lester_power_2021, sun_black-box_2022}." />
        </attvalues>
      </edge>
      <edge source="2210.05675" target="1906.05271" id="12091">
        <attvalues>
          <attvalue for="5" value=" Exemplar-based generalization (that uses all available features) is useful in a low-data regime where there is not enough information to form an abstract sparse rule \cite{feldman2020does}." />
        </attvalues>
      </edge>
      <edge source="2210.05675" target="1907.02893" id="12092">
        <attvalues>
          <attvalue for="5" value=" One interpretation of these results is that the distribution of natural language is more compatible with rule-based generalization from context (rule-based generalization is in fact optimal in compositional domains like langauge \cite{arjovsky2019invariant}), and such patterns might present strong enough learning pressure to overcome -- and even reverse -- transformers' inherent bias towards exemplar-based generalization from context." />
        </attvalues>
      </edge>
      <edge source="2307.01381" target="1706.03762" id="12093">
        <attvalues>
          <attvalue for="5" value=" One branch of machine learning models that have been effective in SimulST is transformers \cite{vaswani2017attention} using block processing, a process that breaks an input sequence into segments which the encoder processes sequentially and individually \cite{dong2019self}." />
        </attvalues>
      </edge>
      <edge source="2307.01381" target="1902.06450" id="12094">
        <attvalues>
          <attvalue for="5" value=" One branch of machine learning models that have been effective in SimulST is transformers \cite{vaswani2017attention} using block processing, a process that breaks an input sequence into segments which the encoder processes sequentially and individually \cite{dong2019self}." />
        </attvalues>
      </edge>
      <edge source="2307.01381" target="1901.02860" id="12095">
        <attvalues>
          <attvalue for="5" value=" The concept of left context was idealized with the Transformer-XL \cite{dai2019transformer}, a model optimized for language modeling, which was later adapted for streaming automatic speech recognition (ASR)." />
        </attvalues>
      </edge>
      <edge source="2307.01381" target="2005.08042" id="12096">
        <attvalues>
          <attvalue for="5" value=" Memory banks were later introduced in the self-attention calculation of the Augmented Memory Transformer \cite{wu2020streaming}, allowing it to outperform the Transformer-XL in streaming ASR and also be state-of-the-art in SimulST \cite{ma2021streaming}." />
        </attvalues>
      </edge>
      <edge source="2307.01381" target="2011.00033" id="12097">
        <attvalues>
          <attvalue for="5" value=" Memory banks were later introduced in the self-attention calculation of the Augmented Memory Transformer \cite{wu2020streaming}, allowing it to outperform the Transformer-XL in streaming ASR and also be state-of-the-art in SimulST \cite{ma2021streaming}." />
        </attvalues>
      </edge>
      <edge source="2309.13029" target="1706.03762" id="12106">
        <attvalues>
          <attvalue for="5" value=" The transformer \cite{vaswani2017attention} architecture is an AED-based system that uses self-attention to capture long-range interactions." />
        </attvalues>
      </edge>
      <edge source="2309.13029" target="2005.08100" id="12107">
        <attvalues>
          <attvalue for="5" value=" For this reason, conformers \cite{49414} have been proposed as an approach for E2E ASR, which outperform RNN-based approaches and transformers since they can model the global and local dependencies of an audio sequence by combining CNNs with transformers." />
        </attvalues>
      </edge>
      <edge source="2309.13029" target="1910.11455" id="12108">
        <attvalues>
          <attvalue for="5" value="\ Notice that while the focus of this work is on offline ASR settings, the proposed MANN is also expected to complement streaming ASR approaches that address the long-form ASR problem \cite{narayanan2019recognizing, wu2020streaming, tsunoo2019transformer}." />
        </attvalues>
      </edge>
      <edge source="2309.13029" target="2303.00747" id="12109">
        <attvalues>
          <attvalue for="5" value=" Another solution is to segment the audio in advance using a separate voice activity detector (VAD) based approach \cite{bain2023whisperx}, or an E2E model that learns to predict segment boundaries \cite{51460}." />
        </attvalues>
      </edge>
      <edge source="2309.13029" target="2204.10749" id="12110">
        <attvalues>
          <attvalue for="5" value=" The E2E segmenter proposed in \cite{51460} relies on human-created heuristics to insert end-of-segment tokens in utterances at training time so that the model can learn to predict those tokens." />
        </attvalues>
      </edge>
      <edge source="2309.13029" target="1410.5401" id="12111">
        <attvalues>
          <attvalue for="5" value=" In contrast to the works mentioned above, we hypothesise whether adding a memory-augmented neural network (MANN) in between the encoder and decoder module -- like a neural Turing machine (NTM) \cite{graves2014neural} -- may be a convenient method to enrich the learning capacity of a conformer, contributing to increase the network generalisation for longer utterances without the need for any ad hoc pre-processing or optimisation in training or decoding." />
        </attvalues>
      </edge>
      <edge source="2309.13029" target="2002.06165" id="12112">
        <attvalues>
          <attvalue for="5" value="\ In particular, NTM has been used to perform unsupervised speaker adaptation in \cite{sari2020unsupervised} by storing i-vectors \cite{dehak2010front} and then reading from the memory to combine the resulting read vector with the hidden vectors of the encoder of the listen, attend and spell (LAS) architecture \cite{7472621}." />
        </attvalues>
      </edge>
      <edge source="2406.03673" target="1908.10084" id="12114">
        <attvalues>
          <attvalue for="5" value=" It is also a popular benchmark for developing tasks such as text embedding learning \cite{senteval,sbert,sbert-aug} and language understanding \cite{glue}." />
        </attvalues>
      </edge>
      <edge source="2406.03673" target="2010.08240" id="12115">
        <attvalues>
          <attvalue for="5" value=" It is also a popular benchmark for developing tasks such as text embedding learning \cite{senteval,sbert,sbert-aug} and language understanding \cite{glue}." />
        </attvalues>
      </edge>
      <edge source="2406.03673" target="2305.15093" id="12117">
        <attvalues>
          <attvalue for="5" value=" A new task called csts has been proposed to resolve those issues \cite{csts}." />
        </attvalues>
      </edge>
      <edge source="2112.05662" target="2201.04458" id="12119">
        <attvalues>
          <attvalue for="5" value=" Previous works scrutinizing BERT-based ranking models either relied on axiomatic approaches adapted to neural models~\cite{CamaraDiagnosingBERTRetrieval2020,sciavolino2021simple}, controlled experiments~\cite{MacAvaneyABNIRMLAnalyzingBehavior2020}, or direct investigation of the learned representations~\cite{JiangHowDoesBERT2021,FormalWhiteBoxAnalysis2021} or attention~\cite{10." />
        </attvalues>
      </edge>
      <edge source="2112.05662" target="2011.00696" id="12121">
        <attvalues>
          <attvalue for="5" value=" Previous works scrutinizing BERT-based ranking models either relied on axiomatic approaches adapted to neural models~\cite{CamaraDiagnosingBERTRetrieval2020,sciavolino2021simple}, controlled experiments~\cite{MacAvaneyABNIRMLAnalyzingBehavior2020}, or direct investigation of the learned representations~\cite{JiangHowDoesBERT2021,FormalWhiteBoxAnalysis2021} or attention~\cite{10." />
        </attvalues>
      </edge>
      <edge source="2112.05662" target="2012.09650" id="12122">
        <attvalues>
          <attvalue for="5" value=" This line of work has shown -- among other findings -- that these models, which rely on contextualized semantic matching, are actually still quite sensitive to lexical match and term statistics in documents/collections~\cite{JiangHowDoesBERT2021,FormalWhiteBoxAnalysis2021}." />
        </attvalues>
      </edge>
      <edge source="2112.05662" target="2104.08663" id="12123">
        <attvalues>
          <attvalue for="5" value=" The BEIR benchmark~\cite{ThakurBEIRHeterogenousBenchmark2021a} has shown that the only systems improving the overall performance over BM25 in the zero-shot setting have (somehow) a lexical bias, eg models like doc2query-T5~\cite{Nogueiradoc2querydocTTTTTquery} or ColBERT~\cite{KhattabColBERTEfficientEffective2020}." />
        </attvalues>
      </edge>
      <edge source="2112.05662" target="2004.12832" id="12124">
        <attvalues>
          <attvalue for="5" value=" The BEIR benchmark~\cite{ThakurBEIRHeterogenousBenchmark2021a} has shown that the only systems improving the overall performance over BM25 in the zero-shot setting have (somehow) a lexical bias, eg models like doc2query-T5~\cite{Nogueiradoc2querydocTTTTTquery} or ColBERT~\cite{KhattabColBERTEfficientEffective2020}." />
        </attvalues>
      </edge>
      <edge source="2408.16502" target="2305.12947" id="12126">
        <attvalues>
          <attvalue for="5" value=" LLM augmentation has been used in various domains such as sentiment analysis~\cite{ONAN2023101611, piedboeuf-langlais-2023-chatgpt}, intent classification~\cite{cegin-etal-2023-chatgpt}, news classification~\cite{piedboeuf-langlais-2023-chatgpt, cegin2024effectsdiversityincentivessample} and health symptoms classification~\cite{dai2023auggpt}." />
        </attvalues>
      </edge>
      <edge source="2408.16502" target="1511.06709" id="12128">
        <attvalues>
          <attvalue for="5" value=" In the established variant, paraphrasing is done through back-translation using a RNN~\cite{sennrich-etal-2016-improving}, while inserts and swaps use BERT-based approach~\cite{kobayashi-2018-contextual, kumar-etal-2020-data}." />
        </attvalues>
      </edge>
      <edge source="2408.16502" target="1805.06201" id="12129">
        <attvalues>
          <attvalue for="5" value=" In the established variant, paraphrasing is done through back-translation using a RNN~\cite{sennrich-etal-2016-improving}, while inserts and swaps use BERT-based approach~\cite{kobayashi-2018-contextual, kumar-etal-2020-data}." />
        </attvalues>
      </edge>
      <edge source="2408.16502" target="2305.14314" id="12130">
        <attvalues>
          <attvalue for="5" value=" We experimented with 6 different datasets (with tasks of sentiment analysis, news classification, and intent classification), 3 downstream classifier models (BERT, RoBERTa, DistilBERT), and 2 fine-tuning approaches (fully fine-tuned, and QLoRA~\cite{dettmers2024qlora})." />
        </attvalues>
      </edge>
      <edge source="2207.00883" target="1907.10726" id="12131">
        <attvalues>
          <attvalue for="5" value=" Context information plays an important role in ASR, especially in scenes that require inter-sentential information such as conversation since semantically related words, or phrases often reoccur across sentences~\cite{kim2019cross}." />
        </attvalues>
      </edge>
      <edge source="2207.00883" target="1701.04056" id="12132">
        <attvalues>
          <attvalue for="5" value=" Typically, traditional hybrid acoustic-language ASR models usually rely on rich language models to model contextual information~\cite{mikolov2010recurrent,mikolov2012context,mnih2007three, ji2015document,liu2017dialog, xiong2018session}." />
        </attvalues>
      </edge>
      <edge source="2207.00883" target="1808.02171" id="12133">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, there are also several researches adopting context information particularly in end-to-end ASR by adding additional context to the decoder or simply concatenate multiple consecutive utterances as the input of an end-to-end model~\cite{kim2018dialog, masumura2019large, masumura2021hierarchical}." />
        </attvalues>
      </edge>
      <edge source="2207.00883" target="1706.03762" id="12134">
        <attvalues>
          <attvalue for="5" value=" Transformer~\cite{vaswani2017attention}, as the most successful attention-based end-to-end model, has recently received more attention due to its superior performance on a wide range of tasks including ASR~\cite{wang2019learning, raganato2018analysis, dong2018speech, karita2019comparative, luo2021simplified, gulati2020conformer}." />
        </attvalues>
      </edge>
      <edge source="2207.00883" target="1906.01787" id="12135">
        <attvalues>
          <attvalue for="5" value=" Transformer~\cite{vaswani2017attention}, as the most successful attention-based end-to-end model, has recently received more attention due to its superior performance on a wide range of tasks including ASR~\cite{wang2019learning, raganato2018analysis, dong2018speech, karita2019comparative, luo2021simplified, gulati2020conformer}." />
        </attvalues>
      </edge>
      <edge source="2207.00883" target="1901.02860" id="12136">
        <attvalues>
          <attvalue for="5" value=" Several studies in natural language processing (NLP) have been explored to utilize the long contextual information for Transformer~\cite{dai2019transformer, rae2019compressive, beltagy2020longformer, zhou2020informer}." />
        </attvalues>
      </edge>
      <edge source="2207.00883" target="2102.07935" id="12137">
        <attvalues>
          <attvalue for="5" value=" Inspired by above studies in the NLP task, some approaches were also proposed to incorporate contextual information across successive input sequences in Transformer-based ASR~\cite{masumura2021hierarchical,hori2020transformer}, but these methods do not solve the problem of the high computational and memory cost, or have high model complexity." />
        </attvalues>
      </edge>
      <edge source="2207.00883" target="2012.11747" id="12138">
        <attvalues>
          <attvalue for="5" value=" Inspired by~\cite{he2020realformer}, we include a residual attention module in the encoder, which accelerates the convergence speed and well models the long-range global dependencies within each input sequence." />
        </attvalues>
      </edge>
      <edge source="2210.07093" target="2004.04906" id="12139">
        <attvalues>
          <attvalue for="5" value=" Our baseline model GAR~\cite{mao2021generation} trails behind its dense retrieval counterpart DPR~\cite{karpukhin2020dense} by a large margin when retrieving a small number of passages." />
        </attvalues>
      </edge>
      <edge source="2210.07093" target="2104.07186" id="12140">
        <attvalues>
          <attvalue for="5" value=" Despite the advent of dense retrieval approaches based on semantic matching for open-domain question answering such as DPR~\cite{karpukhin2020dense}, approaches based on lexical matching~(eg, BM25) remain important due to their space-efficiency and can serve as input to hybrid methods~\cite{gao2021coil,Formal2021SPLADESL,Lin2021AFB}." />
        </attvalues>
      </edge>
      <edge source="2210.07093" target="2107.05720" id="12141">
        <attvalues>
          <attvalue for="5" value=" Despite the advent of dense retrieval approaches based on semantic matching for open-domain question answering such as DPR~\cite{karpukhin2020dense}, approaches based on lexical matching~(eg, BM25) remain important due to their space-efficiency and can serve as input to hybrid methods~\cite{gao2021coil,Formal2021SPLADESL,Lin2021AFB}." />
        </attvalues>
      </edge>
      <edge source="2210.07093" target="2009.08553" id="12143">
        <attvalues>
          <attvalue for="5" value=" Our baseline model GAR~\cite{mao2021generation} trails behind its dense retrieval counterpart DPR~\cite{karpukhin2020dense} by a large margin when retrieving a small number of passages." />
        </attvalues>
      </edge>
      <edge source="2210.07093" target="1906.00300" id="12145">
        <attvalues>
          <attvalue for="5" value=" We evaluate our approach on two established benchmarks: Natural Questions~\cite{kwiatkowski-etal-2019-natural} and TriviaQA~\cite{lee2019latent}." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2004.04906" id="12146">
        <attvalues>
          <attvalue for="5" value=" Dense retrieval models have demonstrated impressive performance in ad-hoc information retrieval (IR) tasks, eg, web search, outperforming traditional retrieval systems such as BM25 \cite[inter alia]{karpukhin-etal-2020-dense, lin2021pretrained, Ni2022LargeDE, neelakantan2022text}." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2007.00808" id="12148">
        <attvalues>
          <attvalue for="5" value=" A major reason for its success lies in the availability of large-scale supervised training datasets in English, such as MS MARCO \cite{msmarco} or NQ \cite{nq}, and coupled with effective training strategies, such as custom hard-negative mining \cite{xiong:2021, lin:2023}, or teacher distillation \cite{hoeffstater:2021, ren:2021}." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2104.06967" id="12149">
        <attvalues>
          <attvalue for="5" value=" A major reason for its success lies in the availability of large-scale supervised training datasets in English, such as MS MARCO \cite{msmarco} or NQ \cite{nq}, and coupled with effective training strategies, such as custom hard-negative mining \cite{xiong:2021, lin:2023}, or teacher distillation \cite{hoeffstater:2021, ren:2021}." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2004.09813" id="12150">
        <attvalues>
          <attvalue for="5" value=" However, there is a limited exploration of dense retrieval models in multilingual retrieval, due to uneven and low distribution of human-supervised training data for other languages apart from English \cite{reimers:2020, feng:2022,wieting-etal-2023-beyond}." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2212.10726" id="12151">
        <attvalues>
          <attvalue for="5" value=" However, there is a limited exploration of dense retrieval models in multilingual retrieval, due to uneven and low distribution of human-supervised training data for other languages apart from English \cite{reimers:2020, feng:2022,wieting-etal-2023-beyond}." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2305.10403" id="12152">
        <attvalues>
          <attvalue for="5" value=" In our work, we utilize PaLM 2 \cite{anil2023palm}, a recent multilingual LLM (successor of PaLM 540B \cite{chowdhery2022palm}) for query generation." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2204.02311" id="12153">
        <attvalues>
          <attvalue for="5" value=" In our work, we utilize PaLM 2 \cite{anil2023palm}, a recent multilingual LLM (successor of PaLM 540B \cite{chowdhery2022palm}) for query generation." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2010.11934" id="12154">
        <attvalues>
          <attvalue for="5" value=" We develop synthetic multilingual (both monolingual and cross-lingual) dense retrieval models called SWIM-X, using mT5 (base) \cite{xue2021mt5} as the backbone and fine-tune on SWIM-IR." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2010.11856" id="12155">
        <attvalues>
          <attvalue for="5" value=" As shown in \ref{fig:overall_results}, on XOR-Retrieve \cite{asai-etal-2021-xor}, SWIM-X outperforms the best-supervised baseline (mContriever-X) by 7." />
        </attvalues>
      </edge>
      <edge source="2311.05800" target="2305.11938" id="12156">
        <attvalues>
          <attvalue for="5" value=" On XTREME-UP \cite{ruder2023xtremeup}, a challenging benchmark containing 20 underrepresented Indo-European languages, SWIM-X~outperforms mContriever-X by 11." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="1706.03762" id="12172">
        <attvalues>
          <attvalue for="5" value=" The key component of these model architectures is the attention mechanism~\cite{vanilla-transformer}." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="1809.01576" id="12173">
        <attvalues>
          <attvalue for="5" value=" However, the original attention design struggles to efficiently handle long sequences, which becomes particularly problematic in scenarios such as document-level translation~\cite{werlen2018document, kim2019and} and large-scale text generation~\cite{zhou2023recurrentgpt}, as its time and space computation costs increase quadratically with the sequence length \cite{tay2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="2305.13304" id="12174">
        <attvalues>
          <attvalue for="5" value=" However, the original attention design struggles to efficiently handle long sequences, which becomes particularly problematic in scenarios such as document-level translation~\cite{werlen2018document, kim2019and} and large-scale text generation~\cite{zhou2023recurrentgpt}, as its time and space computation costs increase quadratically with the sequence length \cite{tay2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="2009.06732" id="12175">
        <attvalues>
          <attvalue for="5" value=" However, the original attention design struggles to efficiently handle long sequences, which becomes particularly problematic in scenarios such as document-level translation~\cite{werlen2018document, kim2019and} and large-scale text generation~\cite{zhou2023recurrentgpt}, as its time and space computation costs increase quadratically with the sequence length \cite{tay2022efficient}." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="2006.04768" id="12176">
        <attvalues>
          <attvalue for="5" value=" Efficient architectures that provide an approximate expression of attention have been explored widely~\cite{wang2020linformer, rfa, peng2022abc,choromanski2021hybrid, zheng2022efficient, zheng2022linear}." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="2110.04367" id="12177">
        <attvalues>
          <attvalue for="5" value=" Efficient architectures that provide an approximate expression of attention have been explored widely~\cite{wang2020linformer, rfa, peng2022abc,choromanski2021hybrid, zheng2022efficient, zheng2022linear}." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="2001.04451" id="12178">
        <attvalues>
          <attvalue for="5" value=" The second line of work is to keep the calculation expression the same and use an external structure like hash function~\cite{kitaev2019reformer, daras2020smyrf}, clustering~\cite{routing-transformer, vyas2020fast} and memory selector~\cite{pietruszka2022sparsifying, transformer-xl, bertsch2023unlimiformer, expire-span, adaptive-span, child2019generating} to find the suitable subset of queries and keys in the long sequence for attention calculation." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="2003.05997" id="12179">
        <attvalues>
          <attvalue for="5" value=" The second line of work is to keep the calculation expression the same and use an external structure like hash function~\cite{kitaev2019reformer, daras2020smyrf}, clustering~\cite{routing-transformer, vyas2020fast} and memory selector~\cite{pietruszka2022sparsifying, transformer-xl, bertsch2023unlimiformer, expire-span, adaptive-span, child2019generating} to find the suitable subset of queries and keys in the long sequence for attention calculation." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="2009.05169" id="12180">
        <attvalues>
          <attvalue for="5" value=" The second line of work is to keep the calculation expression the same and use an external structure like hash function~\cite{kitaev2019reformer, daras2020smyrf}, clustering~\cite{routing-transformer, vyas2020fast} and memory selector~\cite{pietruszka2022sparsifying, transformer-xl, bertsch2023unlimiformer, expire-span, adaptive-span, child2019generating} to find the suitable subset of queries and keys in the long sequence for attention calculation." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="1901.02860" id="12181">
        <attvalues>
          <attvalue for="5" value=" Specifically, we focus on pushing Transformer-XL~\cite{transformer-xl} architecture to a better position by selecting higher-quality tokens inside its memory." />
        </attvalues>
      </edge>
      <edge source="2310.15494" target="1609.07843" id="12182">
        <attvalues>
          <attvalue for="5" value=" Through experiments on two language modeling benchmark datasets, namely word-level WikiText-103~\cite{wikitext103} and character-level enwik8~\cite{enwik8}, we achieve an improvement in the model's performance, as demonstrated by a 0." />
        </attvalues>
      </edge>
      <edge source="2406.11037" target="2106.07447" id="12183">
        <attvalues>
          <attvalue for="5" value=" Self-supervised models have shown to be highly effective in extracting meaningful representations from raw speech signals~\cite{hubert, chen2022wavlm, baevski2020wav2vec, mohamed2022self}." />
        </attvalues>
      </edge>
      <edge source="2406.11037" target="2104.00355" id="12184">
        <attvalues>
          <attvalue for="5" value=" This framework was shown to be effective in modeling multiple levels of the speech utterance: prosody, content~\cite{on_generative, kharitonov2021text, borsos2022audiolm, kharitonov2023speak}, speech compression and enhancement~\cite{polyak2021speech, wang2023selm, erdogan2023tokensplit}, voice and emotion conversion~\cite{kreuk2021textless, maimon2023speaking}, spoken dialogue~\cite{nguyen2022generative}, and speech-to-speech translation~\cite{lee2021direct, popuri2022enhanced, lee2022textless, wang2023speech}." />
        </attvalues>
      </edge>
      <edge source="2406.11037" target="2203.16502" id="12185">
        <attvalues>
          <attvalue for="5" value=" This framework was shown to be effective in modeling multiple levels of the speech utterance: prosody, content~\cite{on_generative, kharitonov2021text, borsos2022audiolm, kharitonov2023speak}, speech compression and enhancement~\cite{polyak2021speech, wang2023selm, erdogan2023tokensplit}, voice and emotion conversion~\cite{kreuk2021textless, maimon2023speaking}, spoken dialogue~\cite{nguyen2022generative}, and speech-to-speech translation~\cite{lee2021direct, popuri2022enhanced, lee2022textless, wang2023speech}." />
        </attvalues>
      </edge>
      <edge source="2406.11037" target="2107.05604" id="12186">
        <attvalues>
          <attvalue for="5" value=" This framework was shown to be effective in modeling multiple levels of the speech utterance: prosody, content~\cite{on_generative, kharitonov2021text, borsos2022audiolm, kharitonov2023speak}, speech compression and enhancement~\cite{polyak2021speech, wang2023selm, erdogan2023tokensplit}, voice and emotion conversion~\cite{kreuk2021textless, maimon2023speaking}, spoken dialogue~\cite{nguyen2022generative}, and speech-to-speech translation~\cite{lee2021direct, popuri2022enhanced, lee2022textless, wang2023speech}." />
        </attvalues>
      </edge>
      <edge source="2406.11037" target="2209.15483" id="12187">
        <attvalues>
          <attvalue for="5" value=" Although providing impressive results, the method proposed by~\cite{gat-etal-2023-augmentation} is based on a teacher-student paradigm with k-means being the teacher." />
        </attvalues>
      </edge>
      <edge source="2406.11037" target="2011.11588" id="12188">
        <attvalues>
          <attvalue for="5" value=" We evaluate NAST's invariance to signal variations (eg, time-stretch, pitch-shift, additive-noise, and reverberation), encoding capabilities (ABX), together with zero-shot sequence modeling evaluations, eg, sWUGGY, sBLIMP~\cite{nguyen2020zero}, and Spoken StoryCloze~\cite{hassid2023textually}." />
        </attvalues>
      </edge>
      <edge source="2406.11037" target="2305.13009" id="12189">
        <attvalues>
          <attvalue for="5" value=" We evaluate NAST's invariance to signal variations (eg, time-stretch, pitch-shift, additive-noise, and reverberation), encoding capabilities (ABX), together with zero-shot sequence modeling evaluations, eg, sWUGGY, sBLIMP~\cite{nguyen2020zero}, and Spoken StoryCloze~\cite{hassid2023textually}." />
        </attvalues>
      </edge>
      <edge source="2408.01623" target="2303.08774" id="12190">
        <attvalues>
          <attvalue for="5" value=" The widespread use of Large Language Models (LLMs)~\cite{openai2023gpt4} for chatbots, highlighted by their human-like conversational abilities across many topics, faces challenges in specialized domains due to their tendency to go off-topic." />
        </attvalues>
      </edge>
      <edge source="2408.01623" target="2212.09939" id="12191">
        <attvalues>
          <attvalue for="5" value=" This structure helps steer the conversation, keeping it within relevant topics, and also enables chatbots to adapt to new tasks or domains without prior training~\cite{zhao-etal-2023-anytod}." />
        </attvalues>
      </edge>
      <edge source="2408.01623" target="1905.05709" id="12192">
        <attvalues>
          <attvalue for="5" value=" However, the construction of precise dialog flows is challenging~\cite{huang2020challenges}, given the diversity of dialog in different domains." />
        </attvalues>
      </edge>
      <edge source="2408.01623" target="2106.07056" id="12193">
        <attvalues>
          <attvalue for="5" value=" The most prevalent approaches~\cite{mehri2021schema, zhao-etal-2023-anytod} use schemas that are carefully handcrafted by the dialog system developers." />
        </attvalues>
      </edge>
      <edge source="2012.06262" target="1806.03743" id="12194">
        <attvalues>
          <attvalue for="5" value=" \cite{gerz-etal-2018-relation} and \cite{cotterell-etal-2018-languages} find that morphological complexity is predictive of language modeling difficulty, while \cite{mielke-etal-2019-kind} conclude that simple statistics of a text like the number of types explain differences in modeling difficulty, rather than morphological measures." />
        </attvalues>
      </edge>
      <edge source="2012.06262" target="1906.04726" id="12195">
        <attvalues>
          <attvalue for="5" value=" \cite{gerz-etal-2018-relation} and \cite{cotterell-etal-2018-languages} find that morphological complexity is predictive of language modeling difficulty, while \cite{mielke-etal-2019-kind} conclude that simple statistics of a text like the number of types explain differences in modeling difficulty, rather than morphological measures." />
        </attvalues>
      </edge>
      <edge source="2012.06262" target="1508.07909" id="12196">
        <attvalues>
          <attvalue for="5" value=" Byte-Pair Encoding \cite[BPE;][]{shibata1999byte} is widely used in NLP tasks including machine translation \cite{sennrich-etal-2016-neural} as an unsupervised information-theoretic method for segmenting text data into subword units." />
        </attvalues>
      </edge>
      <edge source="2012.06262" target="1804.10959" id="12197">
        <attvalues>
          <attvalue for="5" value=" Variants of BPE or closely related methods such as WordPiece \cite{kudo-2018-subword} are frequently employed by state-of-the-art pretrained language models \cite{roberta, radford2019language, bert, xlnet}." />
        </attvalues>
      </edge>
      <edge source="2012.06262" target="1907.11692" id="12198">
        <attvalues>
          <attvalue for="5" value=" Variants of BPE or closely related methods such as WordPiece \cite{kudo-2018-subword} are frequently employed by state-of-the-art pretrained language models \cite{roberta, radford2019language, bert, xlnet}." />
        </attvalues>
      </edge>
      <edge source="2404.05825" target="2004.04906" id="12199">
        <attvalues>
          <attvalue for="5" value=" The Bi-encoder \cite{karpukhin2020dense} is a type of neural network architecture that is widely used in information retrieval." />
        </attvalues>
      </edge>
      <edge source="2404.05825" target="1706.03762" id="12200">
        <attvalues>
          <attvalue for="5" value=" It consists of two encoders, typically in the form of transformer models \cite{vaswani2017attention}, which encode an vector representation for user queries and potential documents or passages respectively." />
        </attvalues>
      </edge>
      <edge source="2404.05825" target="1901.04085" id="12201">
        <attvalues>
          <attvalue for="5" value=" Cross-encoders \cite{nogueira2019passage}, unlike bi-encoders, amalgamate the inputs at an early stage, allowing for a more intricate interaction between user queries and documents." />
        </attvalues>
      </edge>
      <edge source="2404.05825" target="2004.12832" id="12202">
        <attvalues>
          <attvalue for="5" value=" Late-interaction models, such as ColBERT \cite{khattab2020colbert}, ColBERTv2 \cite{santhanam2021colbertv2} or SPALDE++ \cite{formal2022distillation}, are model architectures that hybrids cross-encoder models and bi-encoder models." />
        </attvalues>
      </edge>
      <edge source="2404.05825" target="2112.01488" id="12203">
        <attvalues>
          <attvalue for="5" value=" Late-interaction models, such as ColBERT \cite{khattab2020colbert}, ColBERTv2 \cite{santhanam2021colbertv2} or SPALDE++ \cite{formal2022distillation}, are model architectures that hybrids cross-encoder models and bi-encoder models." />
        </attvalues>
      </edge>
      <edge source="2404.05825" target="2205.04733" id="12204">
        <attvalues>
          <attvalue for="5" value=" Late-interaction models, such as ColBERT \cite{khattab2020colbert}, ColBERTv2 \cite{santhanam2021colbertv2} or SPALDE++ \cite{formal2022distillation}, are model architectures that hybrids cross-encoder models and bi-encoder models." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2303.18223" id="12205">
        <attvalues>
          <attvalue for="5" value=" Recently, the rapid advancement and deployment of Large Language Models (LLMs) have transformed various sectors by providing unprecedented natural language processing capabilities~\cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2312.15883" id="12207">
        <attvalues>
          <attvalue for="5" value=" For example, ChatMed~\cite{zhu2023ChatMed}, DISC-MedLLM~\cite{bao2023discmedllmbridginggenerallarge}, HyKGE~\cite{jiang2024hykgehypothesisknowledgegraph}, IvyGPT~\cite{wang2023ivygpt}, and HuatuoGPT~\cite{zhang2023huatuogpt} are notable examples, demonstrating significant advancements within their specialized medical domains compared to generic LLMs." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2307.10512" id="12208">
        <attvalues>
          <attvalue for="5" value=" For example, ChatMed~\cite{zhu2023ChatMed}, DISC-MedLLM~\cite{bao2023discmedllmbridginggenerallarge}, HyKGE~\cite{jiang2024hykgehypothesisknowledgegraph}, IvyGPT~\cite{wang2023ivygpt}, and HuatuoGPT~\cite{zhang2023huatuogpt} are notable examples, demonstrating significant advancements within their specialized medical domains compared to generic LLMs." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2305.15075" id="12209">
        <attvalues>
          <attvalue for="5" value=" For example, ChatMed~\cite{zhu2023ChatMed}, DISC-MedLLM~\cite{bao2023discmedllmbridginggenerallarge}, HyKGE~\cite{jiang2024hykgehypothesisknowledgegraph}, IvyGPT~\cite{wang2023ivygpt}, and HuatuoGPT~\cite{zhang2023huatuogpt} are notable examples, demonstrating significant advancements within their specialized medical domains compared to generic LLMs." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2306.16092" id="12210">
        <attvalues>
          <attvalue for="5" value=" Models like ChatLaw~\cite{cui2024chatlawmultiagentcollaborativelegal}, LawGPT~\cite{zhou2024lawgpt}, and DISC-LawLLM~\cite{yue2023disc} exemplify this trend, demonstrating a deeper grasp of legal language and principles compared to their generic counterparts." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2406.04614" id="12211">
        <attvalues>
          <attvalue for="5" value=" Models like ChatLaw~\cite{cui2024chatlawmultiagentcollaborativelegal}, LawGPT~\cite{zhou2024lawgpt}, and DISC-LawLLM~\cite{yue2023disc} exemplify this trend, demonstrating a deeper grasp of legal language and principles compared to their generic counterparts." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2309.11325" id="12212">
        <attvalues>
          <attvalue for="5" value=" Models like ChatLaw~\cite{cui2024chatlawmultiagentcollaborativelegal}, LawGPT~\cite{zhou2024lawgpt}, and DISC-LawLLM~\cite{yue2023disc} exemplify this trend, demonstrating a deeper grasp of legal language and principles compared to their generic counterparts." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2306.06031" id="12213">
        <attvalues>
          <attvalue for="5" value=" Examples include FinGPT~\cite{yang2023fingpt}, DISC-FinLLM~\cite{chen2023disc}, and PIXIU~\cite{xie2023pixiu}." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2310.15205" id="12214">
        <attvalues>
          <attvalue for="5" value=" Examples include FinGPT~\cite{yang2023fingpt}, DISC-FinLLM~\cite{chen2023disc}, and PIXIU~\cite{xie2023pixiu}." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2306.05443" id="12215">
        <attvalues>
          <attvalue for="5" value=" Examples include FinGPT~\cite{yang2023fingpt}, DISC-FinLLM~\cite{chen2023disc}, and PIXIU~\cite{xie2023pixiu}." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2403.01063" id="12216">
        <attvalues>
          <attvalue for="5" value=" Additionally, in other fields, models like FaiMA~\cite{yang2024faimafeatureawareincontextlearning}, designed for Multi-domain applications, and Kuaiji~\cite{luo2024kuaijichineseaccountinglarge}, tailored for accounting tasks, illustrate the versatility and potential of fine-tuned LLMs in specialized domains." />
        </attvalues>
      </edge>
      <edge source="2407.07094" target="2402.13866" id="12217">
        <attvalues>
          <attvalue for="5" value=" Additionally, in other fields, models like FaiMA~\cite{yang2024faimafeatureawareincontextlearning}, designed for Multi-domain applications, and Kuaiji~\cite{luo2024kuaijichineseaccountinglarge}, tailored for accounting tasks, illustrate the versatility and potential of fine-tuned LLMs in specialized domains." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="1706.10006" id="12218">
        <attvalues>
          <attvalue for="5" value=" Automated audio captioning (AAC), a cross-modal translation involving transcribing audio signals into concise and meaningful natural language descriptions \cite{aac}, remains a particularly challenging task with a substantial performance gap between human and machine." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="1910.09387" id="12219">
        <attvalues>
          <attvalue for="5" value=" Furthermore, the scarcity of high-quality data, with the most widely used datasets, AudioCaps \cite{audiocaps} and Clotho \cite{clotho} containing only 50K and 20K captions, respectively, poses an additional challenge." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="2107.09817" id="12220">
        <attvalues>
          <attvalue for="5" value=" To address these challenges, prior studies have employed pretrained audio encoders trained on audio classification tasks \cite{mei, conette, beats-conformer}, leveraged the text generation capabilities of pretrained language models like GPT-2 \cite{gpt2, prefix_tuning, pengi} and BART \cite{bart, gontier}, and incorporated auxiliary loss terms, including keyword prediction loss \cite{koizumi_keyword} or sentence embedding loss \cite{sentence_embedding}, to improve the semantic quality of captions and provide additional training signal." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="1910.13461" id="12221">
        <attvalues>
          <attvalue for="5" value=" To address these challenges, prior studies have employed pretrained audio encoders trained on audio classification tasks \cite{mei, conette, beats-conformer}, leveraged the text generation capabilities of pretrained language models like GPT-2 \cite{gpt2, prefix_tuning, pengi} and BART \cite{bart, gontier}, and incorporated auxiliary loss terms, including keyword prediction loss \cite{koizumi_keyword} or sentence embedding loss \cite{sentence_embedding}, to improve the semantic quality of captions and provide additional training signal." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="2305.01482" id="12223">
        <attvalues>
          <attvalue for="5" value=" To address these challenges, prior studies have employed pretrained audio encoders trained on audio classification tasks \cite{mei, conette, beats-conformer}, leveraged the text generation capabilities of pretrained language models like GPT-2 \cite{gpt2, prefix_tuning, pengi} and BART \cite{bart, gontier}, and incorporated auxiliary loss terms, including keyword prediction loss \cite{koizumi_keyword} or sentence embedding loss \cite{sentence_embedding}, to improve the semantic quality of captions and provide additional training signal." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="2401.17690" id="12224">
        <attvalues>
          <attvalue for="5" value=" \cite{enclap} proposed the EnCLAP framework which integrates a set of pretrained models with an auxiliary training task." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="2210.13438" id="12225">
        <attvalues>
          <attvalue for="5" value=" Specifically, EnCLAP utilizes two acoustic feature encoders, EnCodec \cite{encodec} and CLAP \cite{clap_laion}, to generate timestep-level and sequence-level representation of the input audio sequence, respectively." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="2211.06687" id="12226">
        <attvalues>
          <attvalue for="5" value=" Specifically, EnCLAP utilizes two acoustic feature encoders, EnCodec \cite{encodec} and CLAP \cite{clap_laion}, to generate timestep-level and sequence-level representation of the input audio sequence, respectively." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="2303.17395" id="12227">
        <attvalues>
          <attvalue for="5" value=" acknowledge the issue of overfitting in larger model variants but do not investigate the use of large-scale weakly-labeled datasets \cite{wavcaps, beats-conformer}, which contain noisy and model-generated captions." />
        </attvalues>
      </edge>
      <edge source="2409.01201" target="2309.17352" id="12228">
        <attvalues>
          <attvalue for="5" value=" Furthermore, we adopt a sampling-and-reranking approach \cite{beats-conformer} as an alternative to beam search decoding and evaluate its effectiveness." />
        </attvalues>
      </edge>
      <edge source="2209.03316" target="1907.11692" id="12230">
        <attvalues>
          <attvalue for="5" value=" Pre-training (\cite[PT;][]{Devlin:2018uk,liu2019roberta} has achieved tremendous success in natural language processing fields." />
        </attvalues>
      </edge>
      <edge source="2209.03316" target="1905.02450" id="12231">
        <attvalues>
          <attvalue for="5" value=" While recent studies have empirically shown their benefit for the low-resource translation task where the labeled (eg parallel) sentences are limited~\cite{bart2020,song2019mass,Liu:2020mbart}, we are generally confronted with resource-rich scenarios, eg millions of parallel sentence pairs, in WMT evaluations~\cite{akhbardeh-EtAl:2021:WMT} and industries." />
        </attvalues>
      </edge>
      <edge source="2209.03316" target="1910.13461" id="12232">
        <attvalues>
          <attvalue for="5" value=" While recent studies have empirically shown their benefit for the low-resource translation task where the labeled (eg parallel) sentences are limited~\cite{bart2020,song2019mass,Liu:2020mbart}, we are generally confronted with resource-rich scenarios, eg millions of parallel sentence pairs, in WMT evaluations~\cite{akhbardeh-EtAl:2021:WMT} and industries." />
        </attvalues>
      </edge>
      <edge source="2209.03316" target="2001.08210" id="12233">
        <attvalues>
          <attvalue for="5" value=" For these resource-rich tasks, PT becomes less effective (sometimes, even worse) than their Random-Initialization (RI) counterparts, for example, as \cite{Zhu2020Incorporating,Liu:2020mbart} reported, the PT underperforms RI if improperly utilized or significant amount of bi-text data is given." />
        </attvalues>
      </edge>
      <edge source="2209.03316" target="1409.0473" id="12234">
        <attvalues>
          <attvalue for="5" value=" Inspired by BERT~\cite{Devlin:2018uk}, recent works~\cite{song2019mass,bart2020,Liu:2020mbart} attempt to leverage sequence-to-sequence PT for neural machine translation (\cite[NMT;][]{bahdanau2014neural,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}) by leveraging a large amount of unlabeled (eg monolingual) sentences." />
        </attvalues>
      </edge>
      <edge source="2209.03316" target="1706.03762" id="12236">
        <attvalues>
          <attvalue for="5" value=" Inspired by BERT~\cite{Devlin:2018uk}, recent works~\cite{song2019mass,bart2020,Liu:2020mbart} attempt to leverage sequence-to-sequence PT for neural machine translation (\cite[NMT;][]{bahdanau2014neural,DBLP:journals/corr/GehringAGYD17,DBLP:journals/corr/VaswaniSPUJGKP17}) by leveraging a large amount of unlabeled (eg monolingual) sentences." />
        </attvalues>
      </edge>
      <edge source="2209.03316" target="2002.06823" id="12237">
        <attvalues>
          <attvalue for="5" value=" For these resource-rich tasks, PT becomes less effective (sometimes, even worse) than their Random-Initialization (RI) counterparts, for example, as \cite{Zhu2020Incorporating,Liu:2020mbart} reported, the PT underperforms RI if improperly utilized or significant amount of bi-text data is given." />
        </attvalues>
      </edge>
      <edge source="2209.03316" target="1910.05653" id="12238">
        <attvalues>
          <attvalue for="5" value=" Motivated by this finding, we propose a simple combination method to reach \tcgray{{3}} by aligning the neurons and weights of PT and RI and then fusing them into a single model based on optimal transport~\cite{monge1781memoire,singh2020model}." />
        </attvalues>
      </edge>
      <edge source="2402.12913" target="2303.18223" id="12239">
        <attvalues>
          <attvalue for="5" value=" This revolutionary technology has elevated the capabilities of AI systems, enabling them to perform complex reasoning and problem-solving tasks with remarkable proficiency\cite{zhao2023survey}." />
        </attvalues>
      </edge>
      <edge source="2402.12913" target="2307.14283" id="12240">
        <attvalues>
          <attvalue for="5" value=" LLMs are not only limited to language-related tasks but can also function as generalist agents, collaborating with external systems, tools, and models to achieve a wide range of objectives set by humans\cite{triguero2024general}." />
        </attvalues>
      </edge>
      <edge source="2402.12913" target="2401.05778" id="12241">
        <attvalues>
          <attvalue for="5" value=" While prior works have delved into the roots of hallucination within specific, smaller-scale language models and tasks, there is still a notable gap in understanding the exact nature and prevalence of content that LLMs are likely to hallucinate\cite{cui2024risk, chang2023survey}." />
        </attvalues>
      </edge>
      <edge source="2105.03775" target="2105.14879" id="12242">
        <attvalues>
          <attvalue for="5" value=" Therefore, we have about 40\% improvement compared to the baseline, which is a Gated Attention (GA) model \cite{zheng-2021-semeval-task4}." />
        </attvalues>
      </edge>
      <edge source="2105.03775" target="2004.05150" id="12243">
        <attvalues>
          <attvalue for="5" value=" Since we use the long document transformer model (Longformer \cite{beltagy2020longformer}), no limitation is considered in context passage length." />
        </attvalues>
      </edge>
      <edge source="1911.00712" target="1606.05250" id="12244">
        <attvalues>
          <attvalue for="5" value="0 \cite{rajpurkar2016squad}." />
        </attvalues>
      </edge>
      <edge source="1911.00712" target="1806.03822" id="12245">
        <attvalues>
          <attvalue for="5" value=" Variants of this task include unanswerable questions such as in \cite{rajpurkar2018know}." />
        </attvalues>
      </edge>
      <edge source="1911.00712" target="1611.01603" id="12246">
        <attvalues>
          <attvalue for="5" value=" There is a leaderboard on SQUAD dataset which showcases lot of models built for this task \cite{seo2016bidirectional,chen2017reading,bert}." />
        </attvalues>
      </edge>
      <edge source="1911.00712" target="1704.00051" id="12247">
        <attvalues>
          <attvalue for="5" value=" There is a leaderboard on SQUAD dataset which showcases lot of models built for this task \cite{seo2016bidirectional,chen2017reading,bert}." />
        </attvalues>
      </edge>
      <edge source="1911.00712" target="1707.03904" id="12249">
        <attvalues>
          <attvalue for="5" value=" In NN approaches for Open QA, generally answers are extracted using a reading comprehension model on the subset of the retrieved documents or passages considered as relevant \cite{DBLP:journals/corr/DhingraMC17,joshi2017triviaqa}." />
        </attvalues>
      </edge>
      <edge source="1911.00712" target="1705.03551" id="12250">
        <attvalues>
          <attvalue for="5" value=" In NN approaches for Open QA, generally answers are extracted using a reading comprehension model on the subset of the retrieved documents or passages considered as relevant \cite{DBLP:journals/corr/DhingraMC17,joshi2017triviaqa}." />
        </attvalues>
      </edge>
      <edge source="1911.00712" target="1706.03610" id="12251">
        <attvalues>
          <attvalue for="5" value=" We report the performance of our model on different datasets and show that in some cases it outperforms the state-of-the-art systems of BIOASQ \cite{wiese-etal-2017-neural,kamath-etal-2018-adaption,DBLP:journals/corr/abs-1901-08746} in average." />
        </attvalues>
      </edge>
      <edge source="2010.05572" target="1901.02860" id="12254">
        <attvalues>
          <attvalue for="5" value=" Transformer-based pertained language models, such as BERT \cite{devlin2018bert}, GPT-2 \cite{GPT2pre,radford2018improving}, Transformer-XL \cite{Dai2019transformerxl}, XLNet \cite{Yang2019xlnet}, have revolutionized the landscape of natural language processing lately." />
        </attvalues>
      </edge>
      <edge source="2010.05572" target="1605.06069" id="12256">
        <attvalues>
          <attvalue for="5" value=" Most of the existing neural dialogue response generation models are based on recurrent neural networks(RNNs) These neural response generation system suffers from content or style inconsistencies, lack of long-term contextual information \cite{serban17} and blandness \cite{li-etal-2016-diversity,Zhang2019ReCoSa}." />
        </attvalues>
      </edge>
      <edge source="2010.05572" target="1510.03055" id="12257">
        <attvalues>
          <attvalue for="5" value=" Most of the existing neural dialogue response generation models are based on recurrent neural networks(RNNs) These neural response generation system suffers from content or style inconsistencies, lack of long-term contextual information \cite{serban17} and blandness \cite{li-etal-2016-diversity,Zhang2019ReCoSa}." />
        </attvalues>
      </edge>
      <edge source="2010.05572" target="1907.05339" id="12258">
        <attvalues>
          <attvalue for="5" value=" Most of the existing neural dialogue response generation models are based on recurrent neural networks(RNNs) These neural response generation system suffers from content or style inconsistencies, lack of long-term contextual information \cite{serban17} and blandness \cite{li-etal-2016-diversity,Zhang2019ReCoSa}." />
        </attvalues>
      </edge>
      <edge source="2010.05572" target="1810.11118" id="12260">
        <attvalues>
          <attvalue for="5" value=" We have evaluated DSRNet on the Ubuntu-IRC corpus (multi-interlocutor conversation) \cite{kummerfeld2018large} to generate response utterances which clearly indicate improved response text in terms of alignment with context utterances of the conversation topic." />
        </attvalues>
      </edge>
      <edge source="2010.05572" target="1604.04562" id="12261">
        <attvalues>
          <attvalue for="5" value="0 dataset (direct conversation) \cite{lowe2015ubuntu} (mainly pertains to the IT domain) and CamRest676 \cite{wenN2N17} which contains restaurant related conversations." />
        </attvalues>
      </edge>
      <edge source="2010.05572" target="1910.03771" id="12262">
        <attvalues>
          <attvalue for="5" value=" We built DSRNet upon the Huggingface Pytorch Transformer \cite{wolf2019huggingfaces}." />
        </attvalues>
      </edge>
      <edge source="2208.02578" target="1506.05869" id="12273">
        <attvalues>
          <attvalue for="5" value=" Recent advanced response generation models~\cite{zhang:acl2020demo:dialogpt, adiwardana:arxiv2020:meena, roller:eacl2021:blenderbot} can generate relevant and meaningful responses, which can resolve dull response problems~\cite{vinyals:icml2015ws:neuralconv, sordoni:naacl2015:gen-context-sensitive, serban:aaai2016:HRED}." />
        </attvalues>
      </edge>
      <edge source="2208.02578" target="2012.13391" id="12274">
        <attvalues>
          <attvalue for="5" value=" Thus, one practical technique for avoiding contradiction is to have an accurate contradiction detector that eliminates all contradictory candidates from the $n$-best list~\cite{nie:acl2020:i-like-fish}." />
        </attvalues>
      </edge>
      <edge source="2208.02578" target="2106.02228" id="12275">
        <attvalues>
          <attvalue for="5" value=" Nonetheless, earlier quantitative investigations of contradiction relied solely on $1$-bests from models~\cite{li:acl2021:addressing}." />
        </attvalues>
      </edge>
      <edge source="2208.02578" target="1908.04319" id="12276">
        <attvalues>
          <attvalue for="5" value=" Our results show that beam search has limitations in terms of avoiding contradiction and that the newer techniques, such as unlikelihood training~\cite{welleck:iclr2020:unlikelihood}, can help overcome these limitations." />
        </attvalues>
      </edge>
      <edge source="2108.13653" target="1703.01365" id="12284">
        <attvalues>
          <attvalue for="5" value=" We propose a method for explaining classes in text classification tasks using deep learning models and feature attribution techniques, such as the Integrated Gradients (IG) method introduced by \cite{sundararajan2017axiomatic}." />
        </attvalues>
      </edge>
      <edge source="2108.13653" target="2010.05607" id="12285">
        <attvalues>
          <attvalue for="5" value=" We focus specifically on IG as it provides a general framework for estimating feature importance in deep neural networks and has been shown to provide reliable saliency maps in text classification tasks among others \cite{bastings2020elephant,kokhlikyan2020captum}." />
        </attvalues>
      </edge>
      <edge source="2108.13653" target="1706.07979" id="12287">
        <attvalues>
          <attvalue for="5" value=" Given the importance of this endeavour, several different techniques have been suggested in order to interpret model predictions \cite[see][for recent discussion]{montavon2018}." />
        </attvalues>
      </edge>
      <edge source="2310.14771" target="2009.07810" id="12288">
        <attvalues>
          <attvalue for="5" value=" Constructing and completing these KBs at high quality and scale is a long-standing research challenge, and multiple benchmarks exist, eg, FB15k~\cite{bordes2013translating}, CoDEx~\cite{safavi2020codex}, and LM-KBC22~\cite{singhania2022lm}." />
        </attvalues>
      </edge>
      <edge source="2310.14771" target="1909.01066" id="12289">
        <attvalues>
          <attvalue for="5" value=" Our main results are: \begin{enumerate} \item For the long-tail entities of WD-Known, GPT models perform considerably worse than what less demanding benchmarks like LAMA \cite{petroni} have indicated." />
        </attvalues>
      </edge>
      <edge source="2310.14771" target="2107.13586" id="12290">
        <attvalues>
          <attvalue for="5" value=" Starting from the seminal LAMA paper \cite{petroni}, a throve of works have explored how to better probe, train, or fine-tune these LMs \cite{liu2021pre}." />
        </attvalues>
      </edge>
      <edge source="2310.14771" target="2009.11564" id="12291">
        <attvalues>
          <attvalue for="5" value=" For example, part of Yago's success stems from its validated $&gt;$95\% accuracy, and according to \cite{weikum-machine-knowledge}, the Google Knowledge Vault was not deployed into production partly because it did not achieve 99\% accuracy." />
        </attvalues>
      </edge>
      <edge source="2310.14771" target="2303.11082" id="12292">
        <attvalues>
          <attvalue for="5" value=" We evaluate by employing (i) a recent KB completion benchmark, WD-Known, \cite{veseli2023evaluating}, which randomly samples facts from Wikidata and (ii) by a manual evaluation of subject-relation pairs without object values." />
        </attvalues>
      </edge>
      <edge source="2211.00142" target="2005.00333" id="12294">
        <attvalues>
          <attvalue for="5" value=" Despite the recent increase in work focusing on creating multilingual and cross-lingual resources for NLP \cite{nekoto-etal-2020-participatory,ponti-etal-2020-xcopa,ruder-etal-2021-xtreme}, data-to-text datasets are mostly limited to English and a small number of other languages." />
        </attvalues>
      </edge>
      <edge source="2211.00142" target="2104.07412" id="12295">
        <attvalues>
          <attvalue for="5" value=" Despite the recent increase in work focusing on creating multilingual and cross-lingual resources for NLP \cite{nekoto-etal-2020-participatory,ponti-etal-2020-xcopa,ruder-etal-2021-xtreme}, data-to-text datasets are mostly limited to English and a small number of other languages." />
        </attvalues>
      </edge>
      <edge source="2211.00142" target="2004.14373" id="12297">
        <attvalues>
          <attvalue for="5" value=" Datasets frequently suffer from outputs that are not attributable to the inputs or are unnatural, and overly simple tasks fail to identify model limitations~\cite{parikh-etal-2020-totto,thomson-etal-2020-sportsett,DBLP:journals/corr/abs-2111-06467}." />
        </attvalues>
      </edge>
      <edge source="2211.00142" target="2111.06467" id="12298">
        <attvalues>
          <attvalue for="5" value=" Datasets frequently suffer from outputs that are not attributable to the inputs or are unnatural, and overly simple tasks fail to identify model limitations~\cite{parikh-etal-2020-totto,thomson-etal-2020-sportsett,DBLP:journals/corr/abs-2111-06467}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2303.08774" id="12299">
        <attvalues>
          <attvalue for="5" value=" Hence, the capabilities of LLMs to reason over natural language are essential for many of the recent breakthroughs in generative AI, such as NExT-GPT~\cite{Wu2023a}, OpenAI's GPT-4 Vision, GPT-4 Turbo, Google's Gemini, and Apple's Ferret~\cite{OpenAI2023,Deepmind2023,you2023ferret}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2005.14165" id="12300">
        <attvalues>
          <attvalue for="5" value=" With human-like text processing, generation, and reasoning capabilities, LLMs have broad applications ranging from creative content generation to complex problem-solving~\cite{Brown2020, romera2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2204.02311" id="12302">
        <attvalues>
          <attvalue for="5" value=" Similarly, models like Google's PaLM~\cite{Chowdhery2022,Anil2023} and Meta’s LLaMA~\cite{Touvron2023,Touvron2023a}, together with open-source variants like Vicuna~\cite{Zheng2023} and Alpaca~\cite{alpaca2023}, represent parallel advances in the field." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2305.10403" id="12303">
        <attvalues>
          <attvalue for="5" value=" Similarly, models like Google's PaLM~\cite{Chowdhery2022,Anil2023} and Meta’s LLaMA~\cite{Touvron2023,Touvron2023a}, together with open-source variants like Vicuna~\cite{Zheng2023} and Alpaca~\cite{alpaca2023}, represent parallel advances in the field." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2307.09288" id="12305">
        <attvalues>
          <attvalue for="5" value=" Similarly, models like Google's PaLM~\cite{Chowdhery2022,Anil2023} and Meta’s LLaMA~\cite{Touvron2023,Touvron2023a}, together with open-source variants like Vicuna~\cite{Zheng2023} and Alpaca~\cite{alpaca2023}, represent parallel advances in the field." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2306.05685" id="12306">
        <attvalues>
          <attvalue for="5" value=" Similarly, models like Google's PaLM~\cite{Chowdhery2022,Anil2023} and Meta’s LLaMA~\cite{Touvron2023,Touvron2023a}, together with open-source variants like Vicuna~\cite{Zheng2023} and Alpaca~\cite{alpaca2023}, represent parallel advances in the field." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2303.05398" id="12307">
        <attvalues>
          <attvalue for="5" value=" Despite serious challenges in areas like mathematical reasoning~\cite{Imani2023}, student error detection~\cite{bewersdorff2023assessing}, and mitigating hallucinations in outputs~\cite{Ji2023,azamfirei2023large,zhang2023siren,manakul2023selfcheckgpt}, LLMs have already had impact in diverse sectors such as healthcare~\cite{Chintagunta2021,Enarvi2020}, finance~\cite{Dowling2023}, journalism~\cite{Pavlik2023}, creative writing~\cite{Yuan2022, Sessler2023}, and, more recently, to scientific discoveries~\cite{romera2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2308.06088" id="12308">
        <attvalues>
          <attvalue for="5" value=" Despite serious challenges in areas like mathematical reasoning~\cite{Imani2023}, student error detection~\cite{bewersdorff2023assessing}, and mitigating hallucinations in outputs~\cite{Ji2023,azamfirei2023large,zhang2023siren,manakul2023selfcheckgpt}, LLMs have already had impact in diverse sectors such as healthcare~\cite{Chintagunta2021,Enarvi2020}, finance~\cite{Dowling2023}, journalism~\cite{Pavlik2023}, creative writing~\cite{Yuan2022, Sessler2023}, and, more recently, to scientific discoveries~\cite{romera2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2202.03629" id="12309">
        <attvalues>
          <attvalue for="5" value=" Despite serious challenges in areas like mathematical reasoning~\cite{Imani2023}, student error detection~\cite{bewersdorff2023assessing}, and mitigating hallucinations in outputs~\cite{Ji2023,azamfirei2023large,zhang2023siren,manakul2023selfcheckgpt}, LLMs have already had impact in diverse sectors such as healthcare~\cite{Chintagunta2021,Enarvi2020}, finance~\cite{Dowling2023}, journalism~\cite{Pavlik2023}, creative writing~\cite{Yuan2022, Sessler2023}, and, more recently, to scientific discoveries~\cite{romera2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2309.01219" id="12310">
        <attvalues>
          <attvalue for="5" value=" Despite serious challenges in areas like mathematical reasoning~\cite{Imani2023}, student error detection~\cite{bewersdorff2023assessing}, and mitigating hallucinations in outputs~\cite{Ji2023,azamfirei2023large,zhang2023siren,manakul2023selfcheckgpt}, LLMs have already had impact in diverse sectors such as healthcare~\cite{Chintagunta2021,Enarvi2020}, finance~\cite{Dowling2023}, journalism~\cite{Pavlik2023}, creative writing~\cite{Yuan2022, Sessler2023}, and, more recently, to scientific discoveries~\cite{romera2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2303.08896" id="12311">
        <attvalues>
          <attvalue for="5" value=" Despite serious challenges in areas like mathematical reasoning~\cite{Imani2023}, student error detection~\cite{bewersdorff2023assessing}, and mitigating hallucinations in outputs~\cite{Ji2023,azamfirei2023large,zhang2023siren,manakul2023selfcheckgpt}, LLMs have already had impact in diverse sectors such as healthcare~\cite{Chintagunta2021,Enarvi2020}, finance~\cite{Dowling2023}, journalism~\cite{Pavlik2023}, creative writing~\cite{Yuan2022, Sessler2023}, and, more recently, to scientific discoveries~\cite{romera2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2110.07356" id="12312">
        <attvalues>
          <attvalue for="5" value=" Despite serious challenges in areas like mathematical reasoning~\cite{Imani2023}, student error detection~\cite{bewersdorff2023assessing}, and mitigating hallucinations in outputs~\cite{Ji2023,azamfirei2023large,zhang2023siren,manakul2023selfcheckgpt}, LLMs have already had impact in diverse sectors such as healthcare~\cite{Chintagunta2021,Enarvi2020}, finance~\cite{Dowling2023}, journalism~\cite{Pavlik2023}, creative writing~\cite{Yuan2022, Sessler2023}, and, more recently, to scientific discoveries~\cite{romera2023mathematical}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2204.14198" id="12313">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2304.10592" id="12314">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2304.12995" id="12315">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2304.14178" id="12316">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2305.06355" id="12317">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2303.03378" id="12319">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2305.04160" id="12320">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2305.16355" id="12321">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2303.04671" id="12322">
        <attvalues>
          <attvalue for="5" value=" Building on the foundations of LLMs, multimodal generative AI models have further expanded the scope to encompass visual, auditory, and other sensory data~\cite{Alayrac2022,Zhu2023,Huang2023a,Ye2023,Li2023,Maaz2023,Driess2023,Chen2023,Su2023,Wu2023,Shen2023}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2309.05519" id="12324">
        <attvalues>
          <attvalue for="5" value=" Hence, the capabilities of LLMs to reason over natural language are essential for many of the recent breakthroughs in generative AI, such as NExT-GPT~\cite{Wu2023a}, OpenAI's GPT-4 Vision, GPT-4 Turbo, Google's Gemini, and Apple's Ferret~\cite{OpenAI2023,Deepmind2023,you2023ferret}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2310.07704" id="12325">
        <attvalues>
          <attvalue for="5" value=" Hence, the capabilities of LLMs to reason over natural language are essential for many of the recent breakthroughs in generative AI, such as NExT-GPT~\cite{Wu2023a}, OpenAI's GPT-4 Vision, GPT-4 Turbo, Google's Gemini, and Apple's Ferret~\cite{OpenAI2023,Deepmind2023,you2023ferret}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2110.01889" id="12326">
        <attvalues>
          <attvalue for="5" value=" Such advancements, coupled with other deep learning and explainability techniques~\cite{borisov2022deep,rombach2022high}, can potentially revolutionize science~\cite{wong2023discovery} and society." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2112.10752" id="12327">
        <attvalues>
          <attvalue for="5" value=" Such advancements, coupled with other deep learning and explainability techniques~\cite{borisov2022deep,rombach2022high}, can potentially revolutionize science~\cite{wong2023discovery} and society." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2311.07434" id="12328">
        <attvalues>
          <attvalue for="5" value=" A recent study~\cite{kim2023understanding} highlighted a frequent source of user dissatisfaction with ChatGPT: its occasional failure to understand user intentions." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2201.11903" id="12329">
        <attvalues>
          <attvalue for="5" value=" This assumption finds substantial backing in the recent advancements in natural language processing methodologies, notably the Chain-of-Thought, Tree-of-Thought, and Graph-of-Thought techniques~\cite{wei2022chain,yao2023tree,besta2023graph}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2305.10601" id="12330">
        <attvalues>
          <attvalue for="5" value=" This assumption finds substantial backing in the recent advancements in natural language processing methodologies, notably the Chain-of-Thought, Tree-of-Thought, and Graph-of-Thought techniques~\cite{wei2022chain,yao2023tree,besta2023graph}." />
        </attvalues>
      </edge>
      <edge source="2402.02136" target="2308.09687" id="12331">
        <attvalues>
          <attvalue for="5" value=" This assumption finds substantial backing in the recent advancements in natural language processing methodologies, notably the Chain-of-Thought, Tree-of-Thought, and Graph-of-Thought techniques~\cite{wei2022chain,yao2023tree,besta2023graph}." />
        </attvalues>
      </edge>
      <edge source="2409.11630" target="2005.14165" id="12333">
        <attvalues>
          <attvalue for="5" value=" The success of large language models (LLMs) in text domain \cite{brown2020language,openai2023gpt4,touvron2023llama2} has demonstrated their great capability in discrete sequence generation." />
        </attvalues>
      </edge>
      <edge source="2409.11630" target="2303.08774" id="12334">
        <attvalues>
          <attvalue for="5" value=" The success of large language models (LLMs) in text domain \cite{brown2020language,openai2023gpt4,touvron2023llama2} has demonstrated their great capability in discrete sequence generation." />
        </attvalues>
      </edge>
      <edge source="2409.11630" target="2307.09288" id="12335">
        <attvalues>
          <attvalue for="5" value=" The success of large language models (LLMs) in text domain \cite{brown2020language,openai2023gpt4,touvron2023llama2} has demonstrated their great capability in discrete sequence generation." />
        </attvalues>
      </edge>
      <edge source="2409.11630" target="2303.03926" id="12336">
        <attvalues>
          <attvalue for="5" value=" It also inspires the birth of a new text-to-speech synthesis (TTS) paradigm based on the neural codec language model (CLM) \cite{VALLEX, tortoise, lajszczak2024base}, which treats TTS as a next-token prediction task." />
        </attvalues>
      </edge>
      <edge source="2409.11630" target="2210.13438" id="12337">
        <attvalues>
          <attvalue for="5" value=" This framework usually relies on a neural codec \cite{encodec, hifi-codec, dac} to encode the speech audio into discrete tokens, which can be incorporated with the text sequence and generated by the LM, eg an auto-regressive decoder." />
        </attvalues>
      </edge>
      <edge source="2409.11630" target="2310.01427" id="12338">
        <attvalues>
          <attvalue for="5" value=" This long sequence length not only increases the complexity of TTS modeling but aggregates the ``recency bias'' of LMs \cite{peysakhovich2023attention, wang2024eliminating}, eg overly focusing on recent tokens during auto-regressive generation." />
        </attvalues>
      </edge>
      <edge source="2409.11630" target="2406.07855" id="12339">
        <attvalues>
          <attvalue for="5" value=" Although monotonic attention constraints \cite{han2024vall, du2024vall, wang2024attention} are proposed to fix stability issues, they still cannot solve ``recency bias'' fundamentally." />
        </attvalues>
      </edge>
      <edge source="2409.11630" target="2305.07243" id="12340">
        <attvalues>
          <attvalue for="5" value=" Some works \cite{tortoise, socodec, li2024single} turn to directly model shorter speech sequences with a larger frameshift to avoid this issue, but limits the fine-grained expression of LMs in TTS." />
        </attvalues>
      </edge>
      <edge source="2305.01633" target="2204.05961" id="12341">
        <attvalues>
          <attvalue for="5" value=" The work reported in this paper forms part of the ReproHum project in which our aim is to build on existing work on recording properties of human evaluations datasheet-style \cite{shimorina-belz-2022-human}, and assessing how close results from a reproduction study are to the original study \cite{belz-etal-2022-quantified}, to investigate systematically what factors make a human evaluation more---or less---reproducible." />
        </attvalues>
      </edge>
      <edge source="2406.03205" target="1703.04009" id="12342">
        <attvalues>
          <attvalue for="5" value=" While abuse detection in other modalities like text \cite{davidson2017automated, madhu2023detecting} and visual \cite{alcantara2020offensive, gao2020offensive} content has garnered significant focus and development, AAD has not received comparable attention, despite its critical importance in safeguarding digital spaces." />
        </attvalues>
      </edge>
      <edge source="2406.03205" target="2204.02263" id="12344">
        <attvalues>
          <attvalue for="5" value=" \cite{sharon22_interspeech} showed that AAD performance can be enchanced by fusing audio and ASR transcribed textual representations." />
        </attvalues>
      </edge>
      <edge source="2406.03205" target="2407.20808" id="12345">
        <attvalues>
          <attvalue for="5" value=" This is a difficult task as some languages require training on that language for better AAD performance and models trained in other languages will perform poorly on them also pointed out by Spiesberger et al \cite{spiesberger23_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2406.03205" target="2110.04621" id="12346">
        <attvalues>
          <attvalue for="5" value=" However, representations from paralingual PTM \cite{shor2022universal} which has shown SOTA performance in related tasks to AAD such as speech emotion recognition (SER), speaker identification (SI), and so on haven't been explored or given less attention for AAD." />
        </attvalues>
      </edge>
      <edge source="2406.03205" target="2206.05518" id="12347">
        <attvalues>
          <attvalue for="5" value=" \par We also conducted an investigative analysis by combining these PTM representations to explore their potential for complementary behavior, akin to observations in other tasks such as speech recognition \cite{arunkumar22b_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1906.00138" id="12348">
        <attvalues>
          <attvalue for="5" value=" Recent progress in abstractive summarization has been fueled by the advent of large-scale Transformers pre-trained on autoregressive language modeling objectives \cite{hoang-etal-2019,khandelwal-etal-2019,lewis-2019-bart,pegasus}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1905.08836" id="12349">
        <attvalues>
          <attvalue for="5" value=" Recent progress in abstractive summarization has been fueled by the advent of large-scale Transformers pre-trained on autoregressive language modeling objectives \cite{hoang-etal-2019,khandelwal-etal-2019,lewis-2019-bart,pegasus}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1910.13461" id="12350">
        <attvalues>
          <attvalue for="5" value=" We study two prominent summarization models, PEGASUS \cite{pegasus} and BART \cite{lewis-2019-bart}, fine-tuned on two English summarization datasets, CNN/Daily Mail \cite{hermann-2015-cnndm} and XSum \cite{narayan-2018-xsum}, to understand model behavior in each setting." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1912.08777" id="12351">
        <attvalues>
          <attvalue for="5" value=" We study two prominent summarization models, PEGASUS \cite{pegasus} and BART \cite{lewis-2019-bart}, fine-tuned on two English summarization datasets, CNN/Daily Mail \cite{hermann-2015-cnndm} and XSum \cite{narayan-2018-xsum}, to understand model behavior in each setting." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1910.12840" id="12353">
        <attvalues>
          <attvalue for="5" value=" Free-form generation in these models also leads to serious downstream errors, such as factual inconsistencies with the input document \cite{CaoEtAl2018fact,kryscinski-etal-2019-factuality,wang-etal-2020-qags,durmus-etal-2020-feqa,goyal-durrett-2020-factuality}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="2004.04228" id="12354">
        <attvalues>
          <attvalue for="5" value=" Free-form generation in these models also leads to serious downstream errors, such as factual inconsistencies with the input document \cite{CaoEtAl2018fact,kryscinski-etal-2019-factuality,wang-etal-2020-qags,durmus-etal-2020-feqa,goyal-durrett-2020-factuality}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="2010.05478" id="12356">
        <attvalues>
          <attvalue for="5" value=" Free-form generation in these models also leads to serious downstream errors, such as factual inconsistencies with the input document \cite{CaoEtAl2018fact,kryscinski-etal-2019-factuality,wang-etal-2020-qags,durmus-etal-2020-feqa,goyal-durrett-2020-factuality}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1602.04938" id="12357">
        <attvalues>
          <attvalue for="5" value=" Although the interpretability of NLU models has been extensively studied \cite{ribeiro2016should,ghaeini-etal-2018-interpreting,jain-wallace-2019-attention,desai2020calibration}, summarization models specifically have not received similar attention, with analysis efforts often focused on datasets and evaluation \cite{kryscinski-etal-2019-neural}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1808.03894" id="12358">
        <attvalues>
          <attvalue for="5" value=" Although the interpretability of NLU models has been extensively studied \cite{ribeiro2016should,ghaeini-etal-2018-interpreting,jain-wallace-2019-attention,desai2020calibration}, summarization models specifically have not received similar attention, with analysis efforts often focused on datasets and evaluation \cite{kryscinski-etal-2019-neural}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="2003.07892" id="12360">
        <attvalues>
          <attvalue for="5" value=" Although the interpretability of NLU models has been extensively studied \cite{ribeiro2016should,ghaeini-etal-2018-interpreting,jain-wallace-2019-attention,desai2020calibration}, summarization models specifically have not received similar attention, with analysis efforts often focused on datasets and evaluation \cite{kryscinski-etal-2019-neural}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1908.08960" id="12361">
        <attvalues>
          <attvalue for="5" value=" Although the interpretability of NLU models has been extensively studied \cite{ribeiro2016should,ghaeini-etal-2018-interpreting,jain-wallace-2019-attention,desai2020calibration}, summarization models specifically have not received similar attention, with analysis efforts often focused on datasets and evaluation \cite{kryscinski-etal-2019-neural}." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1803.00047" id="12362">
        <attvalues>
          <attvalue for="5" value=" While uncertainty in generation has been studied from the perspective of data \cite{ott2018analyzing}, sampling \cite{fan-etal-2018-hierarchical,holtzman2019curious}, and training \cite{correia2019adaptively,kang2020improved}, it is underutilized as a technique for analysis and inspection of generation systems." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1805.04833" id="12363">
        <attvalues>
          <attvalue for="5" value=" While uncertainty in generation has been studied from the perspective of data \cite{ott2018analyzing}, sampling \cite{fan-etal-2018-hierarchical,holtzman2019curious}, and training \cite{correia2019adaptively,kang2020improved}, it is underutilized as a technique for analysis and inspection of generation systems." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1904.09751" id="12364">
        <attvalues>
          <attvalue for="5" value=" While uncertainty in generation has been studied from the perspective of data \cite{ott2018analyzing}, sampling \cite{fan-etal-2018-hierarchical,holtzman2019curious}, and training \cite{correia2019adaptively,kang2020improved}, it is underutilized as a technique for analysis and inspection of generation systems." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1909.00015" id="12365">
        <attvalues>
          <attvalue for="5" value=" While uncertainty in generation has been studied from the perspective of data \cite{ott2018analyzing}, sampling \cite{fan-etal-2018-hierarchical,holtzman2019curious}, and training \cite{correia2019adaptively,kang2020improved}, it is underutilized as a technique for analysis and inspection of generation systems." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1506.03340" id="12367">
        <attvalues>
          <attvalue for="5" value=" We study two prominent summarization models, PEGASUS \cite{pegasus} and BART \cite{lewis-2019-bart}, fine-tuned on two English summarization datasets, CNN/Daily Mail \cite{hermann-2015-cnndm} and XSum \cite{narayan-2018-xsum}, to understand model behavior in each setting." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1808.08745" id="12368">
        <attvalues>
          <attvalue for="5" value=" We study two prominent summarization models, PEGASUS \cite{pegasus} and BART \cite{lewis-2019-bart}, fine-tuned on two English summarization datasets, CNN/Daily Mail \cite{hermann-2015-cnndm} and XSum \cite{narayan-2018-xsum}, to understand model behavior in each setting." />
        </attvalues>
      </edge>
      <edge source="2010.07882" target="1704.04368" id="12369">
        <attvalues>
          <attvalue for="5" value=" First, by comparing $n$-grams between the input document and generated summaries, we establish two coarse types for decoded tokens, copy and generate \cite{see-2017-ptrgen}." />
        </attvalues>
      </edge>
      <edge source="2212.05093" target="2107.02794" id="12370">
        <attvalues>
          <attvalue for="5" value=" Motivated by previous research in cognitive science~\cite{evans2003two}, \cite{nye2021improving} pointed out that the reasoning of a neural-based model should consist of two systems, eg the system 1 makes intuitive and associative responses, and the system 2 makes deliberative and logical decisions." />
        </attvalues>
      </edge>
      <edge source="2212.05093" target="1908.06177" id="12371">
        <attvalues>
          <attvalue for="5" value=" Typical examples of such task include story continuation with logical coherency~\cite{nye2021improving,sinha2019clutrr}, and recipe generation with step-by-step planning~\cite{marin2019recipe1m+}." />
        </attvalues>
      </edge>
      <edge source="2212.05093" target="1810.06553" id="12372">
        <attvalues>
          <attvalue for="5" value=" We comprehensively evaluate our approach on the recipe generation task using the widely-used Recipe1M+ benchmark~\cite{marin2019recipe1m+}." />
        </attvalues>
      </edge>
      <edge source="2212.05093" target="1909.05858" id="12373">
        <attvalues>
          <attvalue for="5" value=" For example, CTRL~\cite{keskar2019ctrl}, which trains a class-conditional language model, and PPLM~\cite{dathathri2019plug}, which re-ranks the language model predictions by an attribute model." />
        </attvalues>
      </edge>
      <edge source="2212.05093" target="1912.02164" id="12374">
        <attvalues>
          <attvalue for="5" value=" For example, CTRL~\cite{keskar2019ctrl}, which trains a class-conditional language model, and PPLM~\cite{dathathri2019plug}, which re-ranks the language model predictions by an attribute model." />
        </attvalues>
      </edge>
      <edge source="2212.05093" target="1704.06851" id="12375">
        <attvalues>
          <attvalue for="5" value=" Examples of control attribute include sentiment \cite{ghosh2017affect}, topic \cite{tang2019topic} and formality \cite{wang2019topic}." />
        </attvalues>
      </edge>
      <edge source="2310.06803" target="2106.00969" id="12376">
        <attvalues>
          <attvalue for="5" value=" In 2021, researchers proposed Com2Sense~\cite{singh-etal-2021-com2sense}, a reliable and comprehensive commonsense reasoning benchmark with strict pairwise accuracy metrics." />
        </attvalues>
      </edge>
      <edge source="2310.06803" target="1907.11692" id="12378">
        <attvalues>
          <attvalue for="5" value=" Initial works on the dataset revealed that neither general purpose language models \cite{devlin-etal-2019-bert}, \cite{DBLP:journals/corr/abs-1907-11692}, \cite{Raffel2019ExploringTL}, etc nor dedicated commonsense understanding models \cite{Khashabi2020UnifiedQACF}, \cite{2020unifiedqa} performed well on the dataset." />
        </attvalues>
      </edge>
      <edge source="2310.06803" target="2005.00700" id="12380">
        <attvalues>
          <attvalue for="5" value=" Initial works on the dataset revealed that neither general purpose language models \cite{devlin-etal-2019-bert}, \cite{DBLP:journals/corr/abs-1907-11692}, \cite{Raffel2019ExploringTL}, etc nor dedicated commonsense understanding models \cite{Khashabi2020UnifiedQACF}, \cite{2020unifiedqa} performed well on the dataset." />
        </attvalues>
      </edge>
      <edge source="2302.13273" target="2001.00854" id="12389">
        <attvalues>
          <attvalue for="5" value=" In recent years, it has played an important role in many fields, such as pronunciation guidance \cite{C2} and speech recognition \cite{C4,C48,C50}, so it has attracted many researchers to devote themselves to this field." />
        </attvalues>
      </edge>
      <edge source="2302.13273" target="2204.00873" id="12391">
        <attvalues>
          <attvalue for="5" value=" 1DCNNs were used to extract speech features in \cite{C42}." />
        </attvalues>
      </edge>
      <edge source="2305.12000" target="1907.11692" id="12408">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art deep learning models, such as BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta}, GPT-3 \cite{gpt_3}, and others, automatically generate, select, and rank candidate substitutions with performances superior to traditional approaches." />
        </attvalues>
      </edge>
      <edge source="2305.12000" target="2005.14165" id="12409">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art deep learning models, such as BERT \cite{devlin2019bert}, RoBERTa \cite{liu2019roberta}, GPT-3 \cite{gpt_3}, and others, automatically generate, select, and rank candidate substitutions with performances superior to traditional approaches." />
        </attvalues>
      </edge>
      <edge source="2305.12000" target="2302.02888" id="12410">
        <attvalues>
          <attvalue for="5" value=" These include relying on pre-existing lexicons, simplification rules, or engineered features \cite{tsar2022}." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1907.01749" id="12411">
        <attvalues>
          <attvalue for="5" value=" Most existing TTS systems implement these two components or each processing step of them individually using either rule based models (eg, syntactic trees based rules for PSP \cite{zhang2016mandarin}, dictionary matching et al based polyphone disambiguity \cite{huang2010disambiguation}), or statistical learning models (like CRF \cite{qian2010automatic} and LSTM/Attention based NN models \cite{pan2019mandarin,lu2019self} for PSP, \cite{cai2019polyphone,shan2016bi} for Mandarin G2P)." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1703.10135" id="12412">
        <attvalues>
          <attvalue for="5" value=" The aim of this work is to model PSP and G2P simultaneously in a unified manner and provide all the necessary prosodic and pronunciation information for the TTS backend (eg, the acoustic model Tacotron~\cite{wang2017tacotron} and Wavenet vocoder \cite{oord2016wavenet})." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1609.03499" id="12413">
        <attvalues>
          <attvalue for="5" value=" The aim of this work is to model PSP and G2P simultaneously in a unified manner and provide all the necessary prosodic and pronunciation information for the TTS backend (eg, the acoustic model Tacotron~\cite{wang2017tacotron} and Wavenet vocoder \cite{oord2016wavenet})." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1702.07825" id="12414">
        <attvalues>
          <attvalue for="5" value=" Deep Voice~\cite{arik2017deep} attempts to simplify the front-end pipeline by replacing all the components with deep neural networks." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1911.04111" id="12415">
        <attvalues>
          <attvalue for="5" value=" Pan et al \cite{pan2019unified} proposed a unified front-end structure that models PSP and G2P as a single sequence-to-sequence neural model in an auto-regressive way." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1912.10915" id="12416">
        <attvalues>
          <attvalue for="5" value=" Inspired by the success of pre-trained language model (PLM) in TTS front-end related tasks \cite{zhu2019probing,sun2019knowledge,du2019prosodic,talman2019predicting}, we proposed a novel unified Mandarin TTS front-end model based on Chinese BERT, and further compressed it by utilizing a knowledge distillation technique named TinyBERT\cite{jiao2019tinybert}." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1908.02262" id="12417">
        <attvalues>
          <attvalue for="5" value=" Inspired by the success of pre-trained language model (PLM) in TTS front-end related tasks \cite{zhu2019probing,sun2019knowledge,du2019prosodic,talman2019predicting}, we proposed a novel unified Mandarin TTS front-end model based on Chinese BERT, and further compressed it by utilizing a knowledge distillation technique named TinyBERT\cite{jiao2019tinybert}." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1909.10351" id="12418">
        <attvalues>
          <attvalue for="5" value=" \item We employed a two-stage distillation strategy \cite{jiao2019tinybert} to further distill the front-end model into a more compact one, which facilitate deployment on edge devices." />
        </attvalues>
      </edge>
      <edge source="2012.15404" target="1706.05098" id="12419">
        <attvalues>
          <attvalue for="5" value=" The contribution of this work can be summarized into the following aspects: \begin{itemize} &#09;\item We proposed a unified front-end model that employs BERT as text encoder and jointly models the PSP and G2P components in a multi-task learning framework\cite{ruder2017overview}." />
        </attvalues>
      </edge>
      <edge source="2103.04386" target="1606.08425" id="12420">
        <attvalues>
          <attvalue for="5" value=" Research to date has tended to focus on assigning readability levels to whole text rather than to individual sentences, despite the fact that any text is composed of a number of sentences, which vary in their difficulty \cite{schumacher2016predicting}." />
        </attvalues>
      </edge>
      <edge source="2401.07851" target="2303.08774" id="12421">
        <attvalues>
          <attvalue for="5" value=" Large Language Models (LLMs) have achieved remarkable proficiency in a range of downstream tasks~\cite{openai:2023gpt4, Hugo:2023llama, Hugo:2023llama2, vicuna2023, mistral}." />
        </attvalues>
      </edge>
      <edge source="2401.07851" target="1811.03115" id="12422">
        <attvalues>
          <attvalue for="5" value=" To accelerate LLM inference, an innovative inference paradigm, Speculative Decoding has been introduced~\cite{Stern:2018blockwise, xia:2022specdec, Leviathan:2023specdec, Chen:2023specsampling}." />
        </attvalues>
      </edge>
      <edge source="2401.07851" target="2203.16487" id="12423">
        <attvalues>
          <attvalue for="5" value=" For instance, how to design an optimal drafter to strike a balance between speculation accuracy and drafting efficiency~\cite{xia:2022specdec, Zhou:2023distillspec, Li:2024eagle}." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1601.01073" id="12424">
        <attvalues>
          <attvalue for="5" value=" The longstanding goal of multilingual machine translation \cite{firat16,johnson16,aharoni19,gu2018universal} has been to develop a universal translation model, capable of providing high-quality translations between any pair of languages." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1903.00089" id="12425">
        <attvalues>
          <attvalue for="5" value=" The longstanding goal of multilingual machine translation \cite{firat16,johnson16,aharoni19,gu2018universal} has been to develop a universal translation model, capable of providing high-quality translations between any pair of languages." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1802.05368" id="12426">
        <attvalues>
          <attvalue for="5" value=" Past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts \cite{gu2018universal,neubig2018rapid,lakew2019adapting,chronopoulou2020reusing}." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1611.07725" id="12427">
        <attvalues>
          <attvalue for="5" value=" This setting, dubbed in the literature as continual learning \cite{ring1994continual,rebuffi2017icarl,kirkpatrick2017overcoming,lopez2017gradient}, introduces new challenges not found in the traditional multi-task setup, most famously catastrophic forgetting \cite{mccloskey1989catastrophic}, in which the model may lose its previously-learned knowledge as it learns new language pairs." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1612.00796" id="12428">
        <attvalues>
          <attvalue for="5" value=" This setting, dubbed in the literature as continual learning \cite{ring1994continual,rebuffi2017icarl,kirkpatrick2017overcoming,lopez2017gradient}, introduces new challenges not found in the traditional multi-task setup, most famously catastrophic forgetting \cite{mccloskey1989catastrophic}, in which the model may lose its previously-learned knowledge as it learns new language pairs." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1706.08840" id="12429">
        <attvalues>
          <attvalue for="5" value=" This setting, dubbed in the literature as continual learning \cite{ring1994continual,rebuffi2017icarl,kirkpatrick2017overcoming,lopez2017gradient}, introduces new challenges not found in the traditional multi-task setup, most famously catastrophic forgetting \cite{mccloskey1989catastrophic}, in which the model may lose its previously-learned knowledge as it learns new language pairs." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1508.07909" id="12430">
        <attvalues>
          <attvalue for="5" value=" This situation is further complicated by the training procedures of standard tokenizers, such as Byte-Pair Encoding (BPE) \cite{sennrich2015neural} or Sentencepiece \cite{kudo18}, which necessitate access to monolingual data for all the languages considered before producing the vocabulary." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1808.06226" id="12431">
        <attvalues>
          <attvalue for="5" value=" This situation is further complicated by the training procedures of standard tokenizers, such as Byte-Pair Encoding (BPE) \cite{sennrich2015neural} or Sentencepiece \cite{kudo18}, which necessitate access to monolingual data for all the languages considered before producing the vocabulary." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1808.04189" id="12432">
        <attvalues>
          <attvalue for="5" value=" Past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts \cite{gu2018universal,neubig2018rapid,lakew2019adapting,chronopoulou2020reusing}." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="1910.13998" id="12433">
        <attvalues>
          <attvalue for="5" value=" Past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts \cite{gu2018universal,neubig2018rapid,lakew2019adapting,chronopoulou2020reusing}." />
        </attvalues>
      </edge>
      <edge source="2103.06799" target="2009.07610" id="12434">
        <attvalues>
          <attvalue for="5" value=" Past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts \cite{gu2018universal,neubig2018rapid,lakew2019adapting,chronopoulou2020reusing}." />
        </attvalues>
      </edge>
      <edge source="1909.12163" target="1609.05625" id="12435">
        <attvalues>
          <attvalue for="5" value=" Both English ASR MGB-1 ~\cite{bell2015mgb} and the first editon of the Arabic ASR in MGB-2 ~\cite{ali2016mgb} focused on using mainstream broadcast media (BBC in MGB-1, Al Jazeera in MGB-2)." />
        </attvalues>
      </edge>
      <edge source="1909.12163" target="1709.07276" id="12436">
        <attvalues>
          <attvalue for="5" value=" The MGB-3 ~\cite{ali2017speech} used YouTube recordings to extend the diversity of the challenge and deal with dialectal Arabic as a typical example of languages which do not have well-defined orthographic rules and not enough transcribed data." />
        </attvalues>
      </edge>
      <edge source="2408.04325" target="2004.11544" id="12447">
        <attvalues>
          <attvalue for="5" value=" Automatic speech recognition (ASR) \cite{asr0,asr1}, as a crucial branch in the field of artificial intelligence, plays a pivotal role in various application scenarios." />
        </attvalues>
      </edge>
      <edge source="2408.04325" target="2210.14515" id="12448">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{subsampling4} and \cite{subsampling4_1} opt for a subsampling rate of 4, while \cite{subsampling3} and \cite{subsamplingrate2} choose rates of 3 and 2, respectively." />
        </attvalues>
      </edge>
      <edge source="2408.04325" target="2002.00551" id="12449">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{subsampling4} and \cite{subsampling4_1} opt for a subsampling rate of 4, while \cite{subsampling3} and \cite{subsamplingrate2} choose rates of 3 and 2, respectively." />
        </attvalues>
      </edge>
      <edge source="2408.04325" target="2005.10470" id="12450">
        <attvalues>
          <attvalue for="5" value=" \cite{han2021multistream} suggests using different subsamples with different encoders, which can achieve various downsampling rates." />
        </attvalues>
      </edge>
      <edge source="2408.04325" target="2206.00888" id="12451">
        <attvalues>
          <attvalue for="5" value=" \cite{kim2022squeezeformer} and \cite{yao2023zipformer} adopt a U-Net-like architecture incorporating multiple sampling rates." />
        </attvalues>
      </edge>
      <edge source="2408.04325" target="2310.11230" id="12452">
        <attvalues>
          <attvalue for="5" value=" \cite{kim2022squeezeformer} and \cite{yao2023zipformer} adopt a U-Net-like architecture incorporating multiple sampling rates." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="1612.01744" id="12453">
        <attvalues>
          <attvalue for="5" value=" Recently, there has been a growing interest in end-to-end (E2E) methods that directly translate spoken source language to target language text using a single sequence-to-sequence model~\cite{berard2016listen,weiss17_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="1703.08581" id="12454">
        <attvalues>
          <attvalue for="5" value=" Recently, there has been a growing interest in end-to-end (E2E) methods that directly translate spoken source language to target language text using a single sequence-to-sequence model~\cite{berard2016listen,weiss17_interspeech}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2004.06358" id="12455">
        <attvalues>
          <attvalue for="5" value=" However, this approach is still less accurate than the cascade system~\cite{sperber2020speech,agrawal2023findings}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2104.06683" id="12456">
        <attvalues>
          <attvalue for="5" value=" On the other hand, if a segment is too short or does not contain a proper sentence, the translation may contain phrases not in the input, referred to as an insertion error or {hallucination}~\cite{raunak2021curious,lee2018hallucinations}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2104.11710" id="12457">
        <attvalues>
          <attvalue for="5" value=" Pause-based segmentation using voice activity detection (VAD) is commonly employed as a preliminary step for ST systems~\cite{potapczyk2020srpols,gaido2021beyond,gallego2021end}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2105.04512" id="12458">
        <attvalues>
          <attvalue for="5" value=" Another widely used strategy involves length-based segmentation techniques, where speech is divided into segments according to heuristic principles~\cite{gallego2021end,radford2023whisper}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2104.07868" id="12460">
        <attvalues>
          <attvalue for="5" value=" For cascaded speech translation systems, there are works on re-segmentation of ASR output text~\cite{cho2012segmentation,wan2021segmenting}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2002.00551" id="12461">
        <attvalues>
          <attvalue for="5" value=" Also, it is proposed to interpret predictions of ASR and ST models for fixed-size chunks as segmentation~\cite{yoshimura2020end,huang2023e2e,polak2023long,shu2023cif}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2211.15432" id="12462">
        <attvalues>
          <attvalue for="5" value=" Also, it is proposed to interpret predictions of ASR and ST models for fixed-size chunks as segmentation~\cite{yoshimura2020end,huang2023e2e,polak2023long,shu2023cif}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2309.11384" id="12463">
        <attvalues>
          <attvalue for="5" value=" Also, it is proposed to interpret predictions of ASR and ST models for fixed-size chunks as segmentation~\cite{yoshimura2020end,huang2023e2e,polak2023long,shu2023cif}." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2202.04774" id="12464">
        <attvalues>
          <attvalue for="5" value=" Recently, data-driven approaches for audio segmentation have been proposed~\cite{shas,fukuda2022speech,fukuda2024improving}, which consist of a neural network encoder and predict segmentation at frame level." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2203.15479" id="12465">
        <attvalues>
          <attvalue for="5" value=" Recently, data-driven approaches for audio segmentation have been proposed~\cite{shas,fukuda2022speech,fukuda2024improving}, which consist of a neural network encoder and predict segmentation at frame level." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2006.11477" id="12467">
        <attvalues>
          <attvalue for="5" value="0~\cite{baevski2020wav2vec}, whose computational cost would be a hurdle for deploying ST system on mobile devices." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2207.03169" id="12468">
        <attvalues>
          <attvalue for="5" value=" We propose that the ASR-with-punctuation task~\cite{nozaki2022interspeech,kim23_interspeech}, the joint task of speech recognition and punctuation prediction, is an effective pre-training task for the segmentation model." />
        </attvalues>
      </edge>
      <edge source="2406.10549" target="2306.01296" id="12469">
        <attvalues>
          <attvalue for="5" value=" We propose that the ASR-with-punctuation task~\cite{nozaki2022interspeech,kim23_interspeech}, the joint task of speech recognition and punctuation prediction, is an effective pre-training task for the segmentation model." />
        </attvalues>
      </edge>
      <edge source="2408.11845" target="2111.10746" id="12471">
        <attvalues>
          <attvalue for="5" value=" The field of punctuation restoration encompasses two distinct techniques: cascade methods, exemplified by models like BERT \cite{devlin2018bert}, commonly applied independently to Automatic Speech Recognition (ASR) outputs in spoken domains without punctuation \cite{puaics2022capitalization}." />
        </attvalues>
      </edge>
      <edge source="2408.11845" target="2307.09288" id="12474">
        <attvalues>
          <attvalue for="5" value=" Acknowledged for its effectiveness in various language-related tasks, LLaMA emerges as a compelling alternative that surpasses existing benchmarks across numerous Natural Language Processing (NLP) tasks \cite{touvron2023llama} \cite{touvron2023llama2}." />
        </attvalues>
      </edge>
      <edge source="2408.11845" target="2106.09685" id="12475">
        <attvalues>
          <attvalue for="5" value=" Additionally, with LoRA fine-tuning \cite{hu2021lora}, which demands significantly less supervised training data, we achieve comparable and even superior performance for punctuation restoration compared to traditional methods introduced previously." />
        </attvalues>
      </edge>
      <edge source="2204.07356" target="1908.02265" id="12477">
        <attvalues>
          <attvalue for="5" value=" Motivated by this, many cross-modal Vision-Language Pretrained Models (VLPMs) have been designed~\cite{vilbert,vlbert,uniter,oscar}." />
        </attvalues>
      </edge>
      <edge source="2204.07356" target="1909.11740" id="12479">
        <attvalues>
          <attvalue for="5" value=" Motivated by this, many cross-modal Vision-Language Pretrained Models (VLPMs) have been designed~\cite{vilbert,vlbert,uniter,oscar}." />
        </attvalues>
      </edge>
      <edge source="2204.07356" target="2004.06165" id="12480">
        <attvalues>
          <attvalue for="5" value=" Motivated by this, many cross-modal Vision-Language Pretrained Models (VLPMs) have been designed~\cite{vilbert,vlbert,uniter,oscar}." />
        </attvalues>
      </edge>
      <edge source="2204.07356" target="2011.15124" id="12482">
        <attvalues>
          <attvalue for="5" value=" Existing surveys in this area have only partially reviewed some related tasks \cite{mogadala2021trends} or focused mainly on systematical analysis \cite{metaanalysis}." />
        </attvalues>
      </edge>
      <edge source="2203.14222" target="1806.02786" id="12483">
        <attvalues>
          <attvalue for="5" value=" The existing UDA approaches such as domain adversarial training \cite{sun2018domain, sun2017unsupervised}, knowledge distillation \cite{li2017large}, and self-training \cite{khurana2021unsupervised} have shown effectiveness for mitigating data shifting and improving ASR performance." />
        </attvalues>
      </edge>
      <edge source="2203.14222" target="2011.13439" id="12485">
        <attvalues>
          <attvalue for="5" value=" The existing UDA approaches such as domain adversarial training \cite{sun2018domain, sun2017unsupervised}, knowledge distillation \cite{li2017large}, and self-training \cite{khurana2021unsupervised} have shown effectiveness for mitigating data shifting and improving ASR performance." />
        </attvalues>
      </edge>
      <edge source="2203.14222" target="2006.10726" id="12486">
        <attvalues>
          <attvalue for="5" value=" \looseness=-1 Test-Time Adaptation (TTA) \cite{wang2020tent, liang2020we, mummadi2021test, khurana2021sita, Hu2021MixNormTA, you2021test, fleuret2021test} has recently attracted growing interest since it effectively adapts models in prediction time with little target data (a batch or even a single instance) without access to source data." />
        </attvalues>
      </edge>
      <edge source="2203.14222" target="2112.02355" id="12487">
        <attvalues>
          <attvalue for="5" value=" Recently, SITA \cite{khurana2021sita} has presented a single-instance TTA method, which lifts the limitation about latency and distribution since it does not require the pre-collecting batch of test samples, and it allows test samples that come from heterogeneous sources." />
        </attvalues>
      </edge>
      <edge source="2305.14794" target="2212.06950" id="12489">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art methods mostly focus on designing sophisticated human guidance to obtain high-quality labels, through contextualized weak supervision~\cite{mekala2020contextualized}, prompting language models~\cite{meng2020text,npprompt}, clustering for soft matching ~\cite{wang2020x}, and complicated interactions between seeds~\cite{Zhang2021WeaklysupervisedTC}." />
        </attvalues>
      </edge>
      <edge source="2305.14794" target="2110.02591" id="12491">
        <attvalues>
          <attvalue for="5" value=" State-of-the-art methods mostly focus on designing sophisticated human guidance to obtain high-quality labels, through contextualized weak supervision~\cite{mekala2020contextualized}, prompting language models~\cite{meng2020text,npprompt}, clustering for soft matching ~\cite{wang2020x}, and complicated interactions between seeds~\cite{Zhang2021WeaklysupervisedTC}." />
        </attvalues>
      </edge>
      <edge source="2305.14794" target="1809.01478" id="12492">
        <attvalues>
          <attvalue for="5" value=" In this paper, we revisit the seed matching-based weak supervision (denoted as Vanilla)~\cite{mekala2020contextualized,meng2018weakly,tao2015doc2cube}, which is arguably the simplest way to generate pseudo-labels, and show that its power was greatly underestimated." />
        </attvalues>
      </edge>
      <edge source="2305.14794" target="2210.06720" id="12493">
        <attvalues>
          <attvalue for="5" value=" Recently, weakly supervised text classification, because of its light requirement of human effort, has been extensively studied~\cite{mekala2020contextualized,wang2020x,npprompt,meng2020text,Zhang2021WeaklysupervisedTC,meng2018weakly,tao2015doc2cube,Park2022LIMEWT}." />
        </attvalues>
      </edge>
      <edge source="2305.14794" target="2101.06329" id="12494">
        <attvalues>
          <attvalue for="5" value=" In practice, to select those correct labels, a common way is to use the confidence score of a classifier trained on pseudo-labels~\cite{Rizve2021InDO}." />
        </attvalues>
      </edge>
      <edge source="2305.14794" target="2205.12528" id="12495">
        <attvalues>
          <attvalue for="5" value=" However, those high-confidence pseudo-labels may not be correct in the weakly-supervised setting, likely because the classifier may fail to learn reliable confidence on these noisy pseudo-labels~\cite{Mekala2022LOPSLO}." />
        </attvalues>
      </edge>
      <edge source="2301.12568" target="2011.10819" id="12498">
        <attvalues>
          <attvalue for="5" value=" We build upon Semantic Accuracy~\cite{DBLP:conf/inlg/DusekK20}, which evaluates faithfulness in table-to-text tasks by recognizing textual entailment (RTE)." />
        </attvalues>
      </edge>
      <edge source="2301.12568" target="2004.15006" id="12499">
        <attvalues>
          <attvalue for="5" value=" To cover the 45 services in the SGD and the 225 service variations in the SGD-X without prohibitive labor, we propose a rule-based algorithm that constructs entailment references based on slot descriptions from service schema, which are provided in the SGD~\cite{DBLP:conf/emnlp/KaleR20} and other popular task-oriented dialogue datasets such as MultiWOZ 2." />
        </attvalues>
      </edge>
      <edge source="2301.12568" target="2007.12720" id="12500">
        <attvalues>
          <attvalue for="5" value="2~\cite{DBLP:journals/corr/abs-2007-12720}." />
        </attvalues>
      </edge>
      <edge source="2301.12568" target="2101.00190" id="12501">
        <attvalues>
          <attvalue for="5" value=" To address this, we experimented with prefix-tuning (PT) \cite{DBLP:conf/acl/LiL20} which was reported to generalize better than fine-tuning (FT) on unseen data \cite{DBLP:conf/acl/LiL20, DBLP:journals/corr/abs-2110-08329}." />
        </attvalues>
      </edge>
      <edge source="2301.12568" target="2004.06577" id="12502">
        <attvalues>
          <attvalue for="5" value=" Noting their complementary advantages, we used SGSAcc to implement a fidelity reranker similar to that from \cite{DBLP:conf/coling/HarkousGS20} to select faithful generations from an ensemble of PT and FT models, which further improved SER and SGSAcc on the SGD." />
        </attvalues>
      </edge>
      <edge source="2209.09746" target="2010.08824" id="12503">
        <attvalues>
          <attvalue for="5" value=" Neural conversational agents have achieved great successes in recent years, and various methods have been proposed to generate informative responses, eg, the use of knowledge~\cite{Zhao:emnlp2020:KnowledgeGrounded, Wu:acl2020:DiversInformativeDialogue}, personality~\cite{Li:acl2016:PersonabasedNeuralConv,zhang:acl2018:Personachat}, emotional considerations~\cite{Rashkin:acl2019:EmpatheticDialogue, Zhong:emnlp2019:PersonaEmpathetic}, and large-scale models~\cite{Zhang:acl2020:DIALOGPT, Adiwardana:arxiv2020:Meena, Roller:eacl2021:Blender, Thoppilan:arxiv2022:LaMDA}." />
        </attvalues>
      </edge>
      <edge source="2209.09746" target="1603.06155" id="12504">
        <attvalues>
          <attvalue for="5" value=" Neural conversational agents have achieved great successes in recent years, and various methods have been proposed to generate informative responses, eg, the use of knowledge~\cite{Zhao:emnlp2020:KnowledgeGrounded, Wu:acl2020:DiversInformativeDialogue}, personality~\cite{Li:acl2016:PersonabasedNeuralConv,zhang:acl2018:Personachat}, emotional considerations~\cite{Rashkin:acl2019:EmpatheticDialogue, Zhong:emnlp2019:PersonaEmpathetic}, and large-scale models~\cite{Zhang:acl2020:DIALOGPT, Adiwardana:arxiv2020:Meena, Roller:eacl2021:Blender, Thoppilan:arxiv2022:LaMDA}." />
        </attvalues>
      </edge>
      <edge source="2209.09746" target="1811.00207" id="12506">
        <attvalues>
          <attvalue for="5" value=" Neural conversational agents have achieved great successes in recent years, and various methods have been proposed to generate informative responses, eg, the use of knowledge~\cite{Zhao:emnlp2020:KnowledgeGrounded, Wu:acl2020:DiversInformativeDialogue}, personality~\cite{Li:acl2016:PersonabasedNeuralConv,zhang:acl2018:Personachat}, emotional considerations~\cite{Rashkin:acl2019:EmpatheticDialogue, Zhong:emnlp2019:PersonaEmpathetic}, and large-scale models~\cite{Zhang:acl2020:DIALOGPT, Adiwardana:arxiv2020:Meena, Roller:eacl2021:Blender, Thoppilan:arxiv2022:LaMDA}." />
        </attvalues>
      </edge>
      <edge source="2209.09746" target="1905.11553" id="12507">
        <attvalues>
          <attvalue for="5" value=" For example, \cite{Tang:acl2019:TargetGuided} proposed the task of Target-Guided Open-Domain Conversation, in which an agent is required to actively lead a conversation to a predefined target word." />
        </attvalues>
      </edge>
      <edge source="2209.09746" target="1907.03590" id="12508">
        <attvalues>
          <attvalue for="5" value=" Several studies have implemented these target-oriented task settings~\cite{Dai:arxiv2019:MultipleGenerative, Qin:aaai2020:DynamicKnowledge, Yuan:iccai2020:MultihopMemory, Zhong:aaai2021:KeywordGuided, Zhu:sigir2021:ProactiveRetrieval}." />
        </attvalues>
      </edge>
      <edge source="2209.09746" target="1902.00771" id="12509">
        <attvalues>
          <attvalue for="5" value=" However, these prior studies all lack planning, a crucial notion that has been intensively studied in the context of goal-oriented artificial intelligence (AI) agents~\cite[etc]{NorvigAndRussell:book1995:AIModernApproach, KuijpersAndDockx:ai1998:man-machine-ai-planning, Stent:acl2004:TrainableSentencePlanning, Walker:jair2007:IndividualAndDomain} and has also been introduced in neural conversational agents~\cite{Botea:deep-dial2019:AutomatedPlanning,Jiang:aaai2019:GeneralPlanningBasedFramework,Jiang:sigdial2019:Lookahead}." />
        </attvalues>
      </edge>
      <edge source="2209.09746" target="1908.05408" id="12510">
        <attvalues>
          <attvalue for="5" value=" However, these prior studies all lack planning, a crucial notion that has been intensively studied in the context of goal-oriented artificial intelligence (AI) agents~\cite[etc]{NorvigAndRussell:book1995:AIModernApproach, KuijpersAndDockx:ai1998:man-machine-ai-planning, Stent:acl2004:TrainableSentencePlanning, Walker:jair2007:IndividualAndDomain} and has also been introduced in neural conversational agents~\cite{Botea:deep-dial2019:AutomatedPlanning,Jiang:aaai2019:GeneralPlanningBasedFramework,Jiang:sigdial2019:Lookahead}." />
        </attvalues>
      </edge>
      <edge source="2103.00110" target="1712.05884" id="12511">
        <attvalues>
          <attvalue for="5" value=" Speech quality assessment \cite{sqa} aims to measure the quality of synthesized speech and has long been a challenge in speech synthesis areas such as text to speech~\cite{tacotron2,fastspeech,ren2021fastspeech} and voice conversion~\cite{cycleganvc,autovc}." />
        </attvalues>
      </edge>
      <edge source="2103.00110" target="2006.04558" id="12513">
        <attvalues>
          <attvalue for="5" value=" Speech quality assessment \cite{sqa} aims to measure the quality of synthesized speech and has long been a challenge in speech synthesis areas such as text to speech~\cite{tacotron2,fastspeech,ren2021fastspeech} and voice conversion~\cite{cycleganvc,autovc}." />
        </attvalues>
      </edge>
      <edge source="2103.00110" target="1905.05879" id="12514">
        <attvalues>
          <attvalue for="5" value=" Speech quality assessment \cite{sqa} aims to measure the quality of synthesized speech and has long been a challenge in speech synthesis areas such as text to speech~\cite{tacotron2,fastspeech,ren2021fastspeech} and voice conversion~\cite{cycleganvc,autovc}." />
        </attvalues>
      </edge>
      <edge source="2103.00110" target="1808.05344" id="12516">
        <attvalues>
          <attvalue for="5" value=" Although many methods have been proposed to train the MOS prediction model and improve the prediction accuracy, there exists two important problems when dealing with the MOS training data: \begin{itemize} \item To the best of our knowledge, all the previous works ~\cite{QualityNet,mosnet,choi2020neural,choi2020deep} leveraged the utterance-level MOS as the training target, and discarded the detailed judge scores." />
        </attvalues>
      </edge>
      <edge source="2103.00110" target="2007.08267" id="12518">
        <attvalues>
          <attvalue for="5" value=" Experiments on VCC 2018 and VCC 2016 datasets show that MBNet significantly improves the prediction accuracy compared with previous MOS prediction systems including MOSNet and its variants \cite{mosnet,choi2020neural,choi2020deep}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1610.03022" id="12520">
        <attvalues>
          <attvalue for="5" value=" Convolution Neural Network (CNN) based models for end-to-end (E2E) speech recognition is attracting an increasing amount of attention~\cite{zhang2017very,zeghidour2018fully,li2019jasper,kriman2019quartznet}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1812.06864" id="12521">
        <attvalues>
          <attvalue for="5" value=" Convolution Neural Network (CNN) based models for end-to-end (E2E) speech recognition is attracting an increasing amount of attention~\cite{zhang2017very,zeghidour2018fully,li2019jasper,kriman2019quartznet}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1910.10261" id="12523">
        <attvalues>
          <attvalue for="5" value=" This is a big improvement over previous CNN based architectures such as QuartzNet\cite{kriman2019quartznet}, and it outperforms transformer and LSTM based models~\cite{zhang2020transformer, wang2019transformer, zeyer2019comparison, karita2019comparative, park2019specaugment}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1610.02357" id="12524">
        <attvalues>
          <attvalue for="5" value=" Depthwise separable convolutions~\cite{chollet2017xception} have been utilized to further increase the speed and accuracy of CNN models~\cite{hannun2019sequence,kriman2019quartznet}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1904.02619" id="12525">
        <attvalues>
          <attvalue for="5" value=" Depthwise separable convolutions~\cite{chollet2017xception} have been utilized to further increase the speed and accuracy of CNN models~\cite{hannun2019sequence,kriman2019quartznet}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1912.05533" id="12526">
        <attvalues>
          <attvalue for="5" value=" The key advantage of a CNN based model is its parameter efficiency; however, the WER achieved by the best CNN model, QuartzNet~\cite{kriman2019quartznet}, is still behind the RNN/transformer based models~\cite{largespecaugment,karita2019comparative,wang2019transformer,zhang2020transformer}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1909.06317" id="12527">
        <attvalues>
          <attvalue for="5" value=" A major difference between the RNN/Transformer~\cite{karita2019comparative, wang2019transformer, zhang2020transformer} based models and a CNN model is the length of the context." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1910.09799" id="12528">
        <attvalues>
          <attvalue for="5" value=" The key advantage of a CNN based model is its parameter efficiency; however, the WER achieved by the best CNN model, QuartzNet~\cite{kriman2019quartznet}, is still behind the RNN/transformer based models~\cite{largespecaugment,karita2019comparative,wang2019transformer,zhang2020transformer}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="2002.02562" id="12529">
        <attvalues>
          <attvalue for="5" value=" This is a big improvement over previous CNN based architectures such as QuartzNet\cite{kriman2019quartznet}, and it outperforms transformer and LSTM based models~\cite{zhang2020transformer, wang2019transformer, zeyer2019comparison, karita2019comparative, park2019specaugment}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1709.01507" id="12530">
        <attvalues>
          <attvalue for="5" value=" To enhance the global context in the CNN model, we draw inspirations from the squeeze-and-excitation (SE) layer introduced in~\cite{hu2018squeeze}, and propose a novel CNN model for ASR, which we call ContextNet." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1211.3711" id="12531">
        <attvalues>
          <attvalue for="5" value=" For instance, we use a RNN-T decoder~\cite{graves2012sequence, rao2017exploring, he2019, tara2020} instead of the CTC decoder~\cite{graves2006connectionist}." />
        </attvalues>
      </edge>
      <edge source="2005.03191" target="1710.05941" id="12532">
        <attvalues>
          <attvalue for="5" value=" Moreover, we use the Swish activation function~\cite{ramachandran2017searching}, which contributes a slight but consistent reduction in WER." />
        </attvalues>
      </edge>
      <edge source="1908.09936" target="1806.01773" id="12533">
        <attvalues>
          <attvalue for="5" value=" In the Dialogue State Tracking task, the Slot Carryover (SC) paradigm \cite{naik2018contextualSC} involves tracking a key-value pair (a slot) throughout the dialogue." />
        </attvalues>
      </edge>
      <edge source="1908.09936" target="1805.01555" id="12534">
        <attvalues>
          <attvalue for="5" value=" This is used to generate the candidates to be tracked, but slot-to-slot mappings can be challenging, as the keys and values could have been unseen during training \cite{xu-hu-2018-end}." />
        </attvalues>
      </edge>
      <edge source="1908.09936" target="1703.05175" id="12535">
        <attvalues>
          <attvalue for="5" value=" Previous work exists on inferring labels with little to no information provided during training, such as zero and few-shot learning methods \cite{Snell2017PrototypicalNF}, and hybrid approaches such as in \cite{Akata2013LabelEmbeddingFA, yogatama2015embedding, Shi2018MultiContextLE}." />
        </attvalues>
      </edge>
      <edge source="1908.09936" target="1807.05127" id="12536">
        <attvalues>
          <attvalue for="5" value=" Leveraging the hierarchical nature of a corpus is not a new approach either, as hypernym classification requires some sort of hierarchical assumption, for example, \cite{Murty2018HierarchicalLA, Fu2014LearningSH, Nickel2017PoincarEF}." />
        </attvalues>
      </edge>
      <edge source="1908.09936" target="1902.09091" id="12537">
        <attvalues>
          <attvalue for="5" value=" Work has also been made on employing KBs to improve learning, as in \cite{Lee2015LeveragingKB,Yang2017LeveragingKB}." />
        </attvalues>
      </edge>
      <edge source="2409.00315" target="1901.08149" id="12538">
        <attvalues>
          <attvalue for="5" value="3295349}, language models trained on large-scale corpora have dominated the field of machine translation and other NLP tasks, including open-domain dialog generation~\cite{DBLP:journals/corr/abs-1901-08149,zhang2019dialogpt}." />
        </attvalues>
      </edge>
      <edge source="2409.00315" target="1906.01603" id="12539">
        <attvalues>
          <attvalue for="5" value=" Despite the success of Transformer-based dialog models, they were often criticized for not understanding dialog context~\cite{sankar-etal-2019-neural,saleh-etal-2020-probing}, which can lead to generic responses~\cite{li-etal-2016-diversity} or self-contradictions~\cite{kim-etal-2020-will}." />
        </attvalues>
      </edge>
      <edge source="2409.00315" target="2006.08331" id="12540">
        <attvalues>
          <attvalue for="5" value=" Despite the success of Transformer-based dialog models, they were often criticized for not understanding dialog context~\cite{sankar-etal-2019-neural,saleh-etal-2020-probing}, which can lead to generic responses~\cite{li-etal-2016-diversity} or self-contradictions~\cite{kim-etal-2020-will}." />
        </attvalues>
      </edge>
      <edge source="2409.00315" target="1510.03055" id="12541">
        <attvalues>
          <attvalue for="5" value=" Despite the success of Transformer-based dialog models, they were often criticized for not understanding dialog context~\cite{sankar-etal-2019-neural,saleh-etal-2020-probing}, which can lead to generic responses~\cite{li-etal-2016-diversity} or self-contradictions~\cite{kim-etal-2020-will}." />
        </attvalues>
      </edge>
      <edge source="2409.00315" target="2004.05816" id="12542">
        <attvalues>
          <attvalue for="5" value=" Despite the success of Transformer-based dialog models, they were often criticized for not understanding dialog context~\cite{sankar-etal-2019-neural,saleh-etal-2020-probing}, which can lead to generic responses~\cite{li-etal-2016-diversity} or self-contradictions~\cite{kim-etal-2020-will}." />
        </attvalues>
      </edge>
      <edge source="2409.00315" target="2001.09977" id="12543">
        <attvalues>
          <attvalue for="5" value=" For example, Meena~\cite{DeFreitas2020TowardsAH} limited the context to no more than seven utterances, while PLATO~\cite{bao-etal-2020-plato} limited the total length of the context sequence to no more than 256 tokens." />
        </attvalues>
      </edge>
      <edge source="2409.00315" target="1910.07931" id="12544">
        <attvalues>
          <attvalue for="5" value=" For example, Meena~\cite{DeFreitas2020TowardsAH} limited the context to no more than seven utterances, while PLATO~\cite{bao-etal-2020-plato} limited the total length of the context sequence to no more than 256 tokens." />
        </attvalues>
      </edge>
      <edge source="2011.00682" target="1409.3215" id="12545">
        <attvalues>
          <attvalue for="5" value=" \item Sequence-to-Sequence architectures: The performance of network models that transduce one string to another, used in machine translation and semantic parsing, has been greatly improved by the use of independent encoder and decoder networks \cite{sutskever2014sequence}." />
        </attvalues>
      </edge>
      <edge source="2011.00682" target="1406.1078" id="12546">
        <attvalues>
          <attvalue for="5" value=" \begin{itemize} \itemRecurrent units: More sophisticated recurrent units like LSTMs \cite{lstm} and GRUs \cite{gru} have been shown to better encode preceding context than SRNs." />
        </attvalues>
      </edge>
      <edge source="2011.00682" target="1409.0473" id="12547">
        <attvalues>
          <attvalue for="5" value=" \item Attention mechanism: The ability of a network to produce contextually appropriate outputs even in the context of novel vocabulary items has been facilitated by content-sensitive attention mechanisms \cite{bahdanau2016neural, luong-etal-2015-effective}." />
        </attvalues>
      </edge>
      <edge source="2101.03453" target="1907.11692" id="12549">
        <attvalues>
          <attvalue for="5" value=" The BERT family of models~\cite[and others]{devlin2019bert,liu2019roberta} form the backbone of today's NLP systems." />
        </attvalues>
      </edge>
      <edge source="2101.03453" target="1704.05426" id="12551">
        <attvalues>
          <attvalue for="5" value=" A RoBERTa-based model that scores $\sim 89\%$ on the Multi-NLI dataset~\cite{williams2018broad} identifies that the premise entails the hypothesis." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="1904.03240" id="12553">
        <attvalues>
          <attvalue for="5" value=" One of the main categories of SSL methods learns representations by reconstructing the signal such as full reconstruction with autoencoders \cite{chen2019audio, chorowski2019unsupervised}, future reconstruction with Autoregressive Predictive Coding (APC) \cite{chung2019unsupervised} and masked reconstructions \cite{liu2020mockingjay, ling2020deep, ling2020bertphone}." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="1910.12638" id="12554">
        <attvalues>
          <attvalue for="5" value=" One of the main categories of SSL methods learns representations by reconstructing the signal such as full reconstruction with autoencoders \cite{chen2019audio, chorowski2019unsupervised}, future reconstruction with Autoregressive Predictive Coding (APC) \cite{chung2019unsupervised} and masked reconstructions \cite{liu2020mockingjay, ling2020deep, ling2020bertphone}." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="2106.07447" id="12555">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} and HuBERT \cite{hsu2021hubert} pre-train the transformer based encoder using the self-supervised loss, add a randomly initialized output layer on top and fine-tune with the CTC loss \cite{graves2006connectionist}." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="1807.03748" id="12556">
        <attvalues>
          <attvalue for="5" value=" Another category of SSL technology in literature learns representations through a contrastive loss by distinguishing a true future audio sample from a set of negative examples, such as the Contrastive Predictive Coding (CPC) model \cite{oord2018representation} and wav2vec \cite{schneider2019wav2vec}." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="1904.05862" id="12557">
        <attvalues>
          <attvalue for="5" value=" Wav2vec \cite{schneider2019wav2vec} and vq-wav2vec \cite{baevski2019vq} build the wav2letter \cite{collobert2016wav2letter} acoustic model by using the pre-trained embeddings as input features instead of log-mel filterbanks." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="1910.05453" id="12558">
        <attvalues>
          <attvalue for="5" value=" Wav2vec \cite{schneider2019wav2vec} and vq-wav2vec \cite{baevski2019vq} build the wav2letter \cite{collobert2016wav2letter} acoustic model by using the pre-trained embeddings as input features instead of log-mel filterbanks." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="2006.11477" id="12559">
        <attvalues>
          <attvalue for="5" value="0 \cite{baevski2020wav2vec} and HuBERT \cite{hsu2021hubert} pre-train the transformer based encoder using the self-supervised loss, add a randomly initialized output layer on top and fine-tune with the CTC loss \cite{graves2006connectionist}." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="2108.06209" id="12560">
        <attvalues>
          <attvalue for="5" value=" In addition, w2v-BERT \cite{chung2021w2v} combines the two categories by optimizing two self-supervised losses simultaneously (the contrastive loss and masked language modeling loss)." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="1609.03193" id="12562">
        <attvalues>
          <attvalue for="5" value=" Wav2vec \cite{schneider2019wav2vec} and vq-wav2vec \cite{baevski2019vq} build the wav2letter \cite{collobert2016wav2letter} acoustic model by using the pre-trained embeddings as input features instead of log-mel filterbanks." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="2101.07597" id="12563">
        <attvalues>
          <attvalue for="5" value=" The initial UniSpeech work \cite{wang2021unispeech} demonstrates that representations learned during pre-training can be improved if the self supervised contrastive loss is combined with phonetic CTC loss, and the following Unispeech at scale work \cite{wang2021unispeech2} demonstrates better representations from the pre-training stage for the downstream ASR task when combining the contrastive loss and the transducer loss." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="2107.05233" id="12564">
        <attvalues>
          <attvalue for="5" value=" The initial UniSpeech work \cite{wang2021unispeech} demonstrates that representations learned during pre-training can be improved if the self supervised contrastive loss is combined with phonetic CTC loss, and the following Unispeech at scale work \cite{wang2021unispeech2} demonstrates better representations from the pre-training stage for the downstream ASR task when combining the contrastive loss and the transducer loss." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="2011.00093" id="12565">
        <attvalues>
          <attvalue for="5" value=" \cite{talnikar2021joint} alternatively minimizes an unsupervised masked CPC loss and a supervised CTC loss." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="2111.08137" id="12566">
        <attvalues>
          <attvalue for="5" value=" \cite{bai2022joint} explores the benefit of combining the supervised RNN-T loss \cite{graves2012sequence}, the self-supervised contrastive loss and masked language modeling (MLM) losses during different training stages." />
        </attvalues>
      </edge>
      <edge source="2210.12335" target="1211.3711" id="12567">
        <attvalues>
          <attvalue for="5" value=" \cite{bai2022joint} explores the benefit of combining the supervised RNN-T loss \cite{graves2012sequence}, the self-supervised contrastive loss and masked language modeling (MLM) losses during different training stages." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2005.14165" id="12568">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advent of large-scale language models, such as GPT models \cite{brown2020language}, \cite{chen2021evaluating}, has revolutionized natural language understanding and generation \cite{rae2021scaling}, \cite{chowdhery2022palm}, \cite{thoppilan2022lamda}, \cite{zhang2022opt}, \cite{smith2022using}, \cite{fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2107.03374" id="12569">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advent of large-scale language models, such as GPT models \cite{brown2020language}, \cite{chen2021evaluating}, has revolutionized natural language understanding and generation \cite{rae2021scaling}, \cite{chowdhery2022palm}, \cite{thoppilan2022lamda}, \cite{zhang2022opt}, \cite{smith2022using}, \cite{fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2204.02311" id="12570">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advent of large-scale language models, such as GPT models \cite{brown2020language}, \cite{chen2021evaluating}, has revolutionized natural language understanding and generation \cite{rae2021scaling}, \cite{chowdhery2022palm}, \cite{thoppilan2022lamda}, \cite{zhang2022opt}, \cite{smith2022using}, \cite{fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2201.08239" id="12571">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advent of large-scale language models, such as GPT models \cite{brown2020language}, \cite{chen2021evaluating}, has revolutionized natural language understanding and generation \cite{rae2021scaling}, \cite{chowdhery2022palm}, \cite{thoppilan2022lamda}, \cite{zhang2022opt}, \cite{smith2022using}, \cite{fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2205.01068" id="12572">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advent of large-scale language models, such as GPT models \cite{brown2020language}, \cite{chen2021evaluating}, has revolutionized natural language understanding and generation \cite{rae2021scaling}, \cite{chowdhery2022palm}, \cite{thoppilan2022lamda}, \cite{zhang2022opt}, \cite{smith2022using}, \cite{fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2201.11990" id="12573">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advent of large-scale language models, such as GPT models \cite{brown2020language}, \cite{chen2021evaluating}, has revolutionized natural language understanding and generation \cite{rae2021scaling}, \cite{chowdhery2022palm}, \cite{thoppilan2022lamda}, \cite{zhang2022opt}, \cite{smith2022using}, \cite{fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2101.03961" id="12574">
        <attvalues>
          <attvalue for="5" value=" In recent years, the advent of large-scale language models, such as GPT models \cite{brown2020language}, \cite{chen2021evaluating}, has revolutionized natural language understanding and generation \cite{rae2021scaling}, \cite{chowdhery2022palm}, \cite{thoppilan2022lamda}, \cite{zhang2022opt}, \cite{smith2022using}, \cite{fedus2022switch}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2212.14402" id="12575">
        <attvalues>
          <attvalue for="5" value=" For instance, studies have assessed the performance of these models in answering questions from the bar exam \cite{bommarito2022gpt}, \cite{katz2023gpt}, medical tests \cite{lievin2022can}, \cite{kung2023performance}, \cite{nori2023capabilities}, radiology examinations \cite{bhayana2023performance}, \cite{bhayana2023gpt}, and mathematical tests \cite{floridi2020gpt}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2207.08143" id="12576">
        <attvalues>
          <attvalue for="5" value=" For instance, studies have assessed the performance of these models in answering questions from the bar exam \cite{bommarito2022gpt}, \cite{katz2023gpt}, medical tests \cite{lievin2022can}, \cite{kung2023performance}, \cite{nori2023capabilities}, radiology examinations \cite{bhayana2023performance}, \cite{bhayana2023gpt}, and mathematical tests \cite{floridi2020gpt}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2303.13375" id="12577">
        <attvalues>
          <attvalue for="5" value=" For instance, studies have assessed the performance of these models in answering questions from the bar exam \cite{bommarito2022gpt}, \cite{katz2023gpt}, medical tests \cite{lievin2022can}, \cite{kung2023performance}, \cite{nori2023capabilities}, radiology examinations \cite{bhayana2023performance}, \cite{bhayana2023gpt}, and mathematical tests \cite{floridi2020gpt}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2111.02840" id="12578">
        <attvalues>
          <attvalue for="5" value=" While these studies have demonstrated the potential of GPT models in tackling complex and domain-specific questions, they have also identified challenges and limitations, such as the susceptibility of the models to hallucinate information that is not present in the input data \cite{bender2021dangers}, \cite{wang2021adversarial}." />
        </attvalues>
      </edge>
      <edge source="2308.11827" target="2107.13586" id="12579">
        <attvalues>
          <attvalue for="5" value=" In this regard, integrating relevant contextual information into the input prompts can significantly enhance the models’ ability to generate accurate and coherent responses \cite{liu2023pre}, \cite{borgeaud2022improving}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2205.10643" id="12581">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning~(SSL) for encoder pre-training has emerged as a foundational element in speech processing, outperforming conventional approaches across various applications~\cite{mohamed2022self,liu2022audio}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2203.01205" id="12582">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning~(SSL) for encoder pre-training has emerged as a foundational element in speech processing, outperforming conventional approaches across various applications~\cite{mohamed2022self,liu2022audio}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2203.06849" id="12584">
        <attvalues>
          <attvalue for="5" value=" Given the substantial cost associated with human annotation of speech data, SSL methods leverage unlabeled audio data to pre-train encoders, generating good representations for downstream tasks like automatic speech recognition~(ASR) and speaker identification~\cite{yang2021superb,tsai-etal-2022-superb}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2106.07447" id="12585">
        <attvalues>
          <attvalue for="5" value=" \cite{huang2022improving} proposes HuBERT-MGR via domain adversarial training to render the fine-tuned HuBERT model~\cite{hsu2021hubert} invariant to domain shifts." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2202.03555" id="12586">
        <attvalues>
          <attvalue for="5" value=" The application of SSL models has notably concentrated on ASR, aiming to mitigate the dependence on large transcribed corpora~\cite{hsu2021hubert,baevski2022data2vec,liu2023dinosr}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2305.10005" id="12587">
        <attvalues>
          <attvalue for="5" value=" The application of SSL models has notably concentrated on ASR, aiming to mitigate the dependence on large transcribed corpora~\cite{hsu2021hubert,baevski2022data2vec,liu2023dinosr}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2205.09872" id="12589">
        <attvalues>
          <attvalue for="5" value=" Thus, extracting content representations has become a crucial aspect of speech SSL research~\cite{tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2208.13191" id="12590">
        <attvalues>
          <attvalue for="5" value=" Thus, extracting content representations has become a crucial aspect of speech SSL research~\cite{tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2311.03389" id="12591">
        <attvalues>
          <attvalue for="5" value=" Thus, extracting content representations has become a crucial aspect of speech SSL research~\cite{tjandra2020unsupervised,chan2022content,peyser2022towards,williams2022learning}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2204.09224" id="12592">
        <attvalues>
          <attvalue for="5" value=" \cite{qian2022contentvec} propose ContentVec by disentangling speaker and content information, demonstrating promising results." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2305.11072" id="12593">
        <attvalues>
          <attvalue for="5" value=" Alternatively, \cite{chang2023spin} propose Speaker-invariant Clustering~(Spin) to produce content representations with minimal fine-tuning resources." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2104.01027" id="12594">
        <attvalues>
          <attvalue for="5" value=" While current methods perform well on clean speech datasets, they are vulnerable to out-of-domain data like distorted audio signals~\cite{hsu2021robust}." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2203.16104" id="12595">
        <attvalues>
          <attvalue for="5" value=" \cite{huang2022improving} proposes HuBERT-MGR via domain adversarial training to render the fine-tuned HuBERT model~\cite{hsu2021hubert} invariant to domain shifts." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2110.13900" id="12596">
        <attvalues>
          <attvalue for="5" value=" WavLM~\cite{chen2022wavlm} integrates denoising with the HuBERT pre-training framework, achieving state-of-the-art performance in many speech processing downstream tasks." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2210.15324" id="12597">
        <attvalues>
          <attvalue for="5" value=" Similarly, \cite{zhu2023robust} propose Robust data2vec, introducing perturbations to the input to predict the exponential moving average teacher model's representations." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2103.03230" id="12598">
        <attvalues>
          <attvalue for="5" value=" In deHuBERT~\cite{ng2023dehubert}, the Barlow Twins loss~\cite{zbontar2021barlow} is applied to encourage representation invariability to input perturbations." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2110.04934" id="12599">
        <attvalues>
          <attvalue for="5" value=" Although many methods have shown success in noisy speech recognition~\cite{wang2022wav2vec-switch,zhu2022noise,huang2022spiral,hu2023wav2code}, to our knowledge, none have concurrently addressed the disentanglement of speaker and noise while enhancing content information." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2201.10207" id="12601">
        <attvalues>
          <attvalue for="5" value=" Although many methods have shown success in noisy speech recognition~\cite{wang2022wav2vec-switch,zhu2022noise,huang2022spiral,hu2023wav2code}, to our knowledge, none have concurrently addressed the disentanglement of speaker and noise while enhancing content information." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2304.04974" id="12602">
        <attvalues>
          <attvalue for="5" value=" Although many methods have shown success in noisy speech recognition~\cite{wang2022wav2vec-switch,zhu2022noise,huang2022spiral,hu2023wav2code}, to our knowledge, none have concurrently addressed the disentanglement of speaker and noise while enhancing content information." />
        </attvalues>
      </edge>
      <edge source="2311.09117" target="2204.03240" id="12603">
        <attvalues>
          <attvalue for="5" value=" The pseudo-labels are generated by learning acoustic pieces~\cite{ren2022acoustic-piece} on top of the discrete units produced by a pre-trained Spin model, offering better training targets that closely align with phonemes and characters." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1907.11692" id="12605">
        <attvalues>
          <attvalue for="5" value=" Researchers have increasingly raised concerns about the tendency of recent NLU models~\cite{devlin2018bert,liu2019roberta} to quickly leverage spurious surface lexical-syntactic features~\cite{poliak2018hypothesis,gururangan2018annotation,dasgupta2018evaluating,ghaddar2017winer}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1805.01042" id="12606">
        <attvalues>
          <attvalue for="5" value=" Researchers have increasingly raised concerns about the tendency of recent NLU models~\cite{devlin2018bert,liu2019roberta} to quickly leverage spurious surface lexical-syntactic features~\cite{poliak2018hypothesis,gururangan2018annotation,dasgupta2018evaluating,ghaddar2017winer}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1803.02324" id="12607">
        <attvalues>
          <attvalue for="5" value=" Researchers have increasingly raised concerns about the tendency of recent NLU models~\cite{devlin2018bert,liu2019roberta} to quickly leverage spurious surface lexical-syntactic features~\cite{poliak2018hypothesis,gururangan2018annotation,dasgupta2018evaluating,ghaddar2017winer}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1802.04302" id="12608">
        <attvalues>
          <attvalue for="5" value=" Researchers have increasingly raised concerns about the tendency of recent NLU models~\cite{devlin2018bert,liu2019roberta} to quickly leverage spurious surface lexical-syntactic features~\cite{poliak2018hypothesis,gururangan2018annotation,dasgupta2018evaluating,ghaddar2017winer}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1912.11078" id="12609">
        <attvalues>
          <attvalue for="5" value=" These superficial properties, also referred as dataset biases~\cite{shah2020predictive,utama2020towards,moosavi2020improving}, result in significant performance drop on out-of-distribution (OOD) sets containing counterexamples to biases in the training data~\cite{mccoy2019right,schuster2019towards,zhang2019paws,ghaddar2021context}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="2009.12303" id="12610">
        <attvalues>
          <attvalue for="5" value=" \cite{utama2020towards} propose to use instead a model trained on a tiny fraction ($&lt;1\%$) of the training data for few epochs as a bias model; while \cite{clark2020learning}~and \cite{sanh2020learning} trained a low capacity model on the full training set." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="2010.12510" id="12611">
        <attvalues>
          <attvalue for="5" value=" These superficial properties, also referred as dataset biases~\cite{shah2020predictive,utama2020towards,moosavi2020improving}, result in significant performance drop on out-of-distribution (OOD) sets containing counterexamples to biases in the training data~\cite{mccoy2019right,schuster2019towards,zhang2019paws,ghaddar2021context}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1908.05267" id="12613">
        <attvalues>
          <attvalue for="5" value=" Both models are trained simultaneously in an end-to-end manner as in \cite{mahabadi2020end}, where the importance of training samples for both models are adjusted using the example reweighting technique of~\cite{schuster2019towards}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="2107.11610" id="12615">
        <attvalues>
          <attvalue for="5" value=" These superficial properties, also referred as dataset biases~\cite{shah2020predictive,utama2020towards,moosavi2020improving}, result in significant performance drop on out-of-distribution (OOD) sets containing counterexamples to biases in the training data~\cite{mccoy2019right,schuster2019towards,zhang2019paws,ghaddar2021context}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1908.10763" id="12616">
        <attvalues>
          <attvalue for="5" value=" Several debiasing training paradigms have been proposed to adjust the importance of biased training samples, such as product of experts~\cite{clark2019don,he2019unlearn}, learned-mixin~\cite{clark2019don}, example reweighting~\cite{schuster2019towards}, debiased focal loss~\cite{mahabadi2020end}, and confidence regularization~\cite{utama2020mind}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1909.06321" id="12617">
        <attvalues>
          <attvalue for="5" value=" Both models are trained simultaneously in an end-to-end manner as in \cite{mahabadi2020end}, where the importance of training samples for both models are adjusted using the example reweighting technique of~\cite{schuster2019towards}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="2005.00315" id="12618">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{utama2020mind} used a copy of the main model (110M parameters) as the bias model, while \cite{sanh2020learning} used Bert{-Tiny}~\cite{turc2019well} that has 11M parameters and has been pre-trained from scratch using the masked LM objective~\cite{devlin2018bert}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="2011.03856" id="12619">
        <attvalues>
          <attvalue for="5" value=" \cite{utama2020towards} propose to use instead a model trained on a tiny fraction ($&lt;1\%$) of the training data for few epochs as a bias model; while \cite{clark2020learning}~and \cite{sanh2020learning} trained a low capacity model on the full training set." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="2012.01300" id="12620">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{utama2020mind} used a copy of the main model (110M parameters) as the bias model, while \cite{sanh2020learning} used Bert{-Tiny}~\cite{turc2019well} that has 11M parameters and has been pre-trained from scratch using the masked LM objective~\cite{devlin2018bert}." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="2004.07453" id="12621">
        <attvalues>
          <attvalue for="5" value=" The idea of using intermediate classifiers has previously been explored to reduce the inference cost~\cite{schwartz2020right,zhou2020bert,xin2021berxit} for Transformer-based~\cite{vaswani2017attention} models." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="2006.04152" id="12622">
        <attvalues>
          <attvalue for="5" value=" The idea of using intermediate classifiers has previously been explored to reduce the inference cost~\cite{schwartz2020right,zhou2020bert,xin2021berxit} for Transformer-based~\cite{vaswani2017attention} models." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1706.03762" id="12623">
        <attvalues>
          <attvalue for="5" value=" The idea of using intermediate classifiers has previously been explored to reduce the inference cost~\cite{schwartz2020right,zhou2020bert,xin2021berxit} for Transformer-based~\cite{vaswani2017attention} models." />
        </attvalues>
      </edge>
      <edge source="2109.02071" target="1908.08962" id="12624">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{utama2020mind} used a copy of the main model (110M parameters) as the bias model, while \cite{sanh2020learning} used Bert{-Tiny}~\cite{turc2019well} that has 11M parameters and has been pre-trained from scratch using the masked LM objective~\cite{devlin2018bert}." />
        </attvalues>
      </edge>
      <edge source="2111.10157" target="1908.10992" id="12625">
        <attvalues>
          <attvalue for="5" value=" We employ minimum word error rate (MWER) \cite{hori2016minimum} training criterion which has been shown to improve accuracy of attention-based rescoring models \cite{sainath2019two,gandhe2020audio,hu2020deliberation}." />
        </attvalues>
      </edge>
      <edge source="2111.10157" target="1211.3711" id="12626">
        <attvalues>
          <attvalue for="5" value=" In \cite{sainath2019two}, a listen-attend-spell \cite{chan2016listen} based model was proposed to rescore n-best lists where the encoder is shared with the first-pass recurrent neural network transducer (RNN-T) \cite{graves2012sequence} model." />
        </attvalues>
      </edge>
      <edge source="2111.10157" target="1912.03363" id="12627">
        <attvalues>
          <attvalue for="5" value=" We employ minimum word error rate (MWER) \cite{hori2016minimum} training criterion which has been shown to improve accuracy of attention-based rescoring models \cite{sainath2019two,gandhe2020audio,hu2020deliberation}." />
        </attvalues>
      </edge>
      <edge source="2111.10157" target="1609.07730" id="12630">
        <attvalues>
          <attvalue for="5" value=" There has already been some work on representing lattice structures in recurrent encoders \cite{sperber2017neural,su2017lattice,ladhak2016latticernn} and transformers \cite{sperber2019self,xiao2019lattice} models." />
        </attvalues>
      </edge>
      <edge source="2111.10157" target="1906.01617" id="12631">
        <attvalues>
          <attvalue for="5" value=" There has already been some work on representing lattice structures in recurrent encoders \cite{sperber2017neural,su2017lattice,ladhak2016latticernn} and transformers \cite{sperber2019self,xiao2019lattice} models." />
        </attvalues>
      </edge>
      <edge source="2111.10157" target="1906.01282" id="12632">
        <attvalues>
          <attvalue for="5" value=" There has already been some work on representing lattice structures in recurrent encoders \cite{sperber2017neural,su2017lattice,ladhak2016latticernn} and transformers \cite{sperber2019self,xiao2019lattice} models." />
        </attvalues>
      </edge>
      <edge source="2111.10157" target="1805.02023" id="12633">
        <attvalues>
          <attvalue for="5" value=" Using a lattice encoder instead of the 1-best output has been shown to improve performance of downstream tasks like speech translation \cite{sperber2017neural,su2017lattice,sperber2019self,xiao2019lattice} and spoken language understanding \cite{ladhak2016latticernn,zhang2018chinese}." />
        </attvalues>
      </edge>
      <edge source="2111.10157" target="1503.00075" id="12634">
        <attvalues>
          <attvalue for="5" value=" In \cite{sperber2017neural}, LatticeLSTM was proposed for machine translation, which extends TreeLSTM \cite{tai2015improved} to encode directed acyclic graphs with weights." />
        </attvalues>
      </edge>
      <edge source="2305.19650" target="1003.1141" id="12635">
        <attvalues>
          <attvalue for="5" value=" Lexical information is generally represented either in online dictionaries or by embeddings extracted from corpora \cite{Turney2010,devlin-etal-2019-bert,peters-etal-2018-deep}." />
        </attvalues>
      </edge>
      <edge source="2305.19650" target="1802.05365" id="12637">
        <attvalues>
          <attvalue for="5" value=" Lexical information is generally represented either in online dictionaries or by embeddings extracted from corpora \cite{Turney2010,devlin-etal-2019-bert,peters-etal-2018-deep}." />
        </attvalues>
      </edge>
      <edge source="2305.19650" target="1912.00582" id="12639">
        <attvalues>
          <attvalue for="5" value=" Among the standard benchmarks, only GLUE \cite{wang-etal-2018-glue} and BLiMP \cite{warstadt2020blimp} cover adverbs, and then only marginally." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2307.09288" id="12644">
        <attvalues>
          <attvalue for="5" value=" In recent years, the rapid development of large language models has revolutionized the field of natural language processing ~\cite{devlin2019bert,touvron2023llama2openfoundation,openai2024gpt4,bai2023qwenvl,du2022glamefficientscalinglanguage}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2303.08774" id="12645">
        <attvalues>
          <attvalue for="5" value=" Due to the powerful capabilities of LLMs, a natural idea is to combine them with multimodal research, leveraging the strong cognitive abilities of LLMs to process information from other modalities ~\cite{openai2024gpt4,chu2024qwen2audiotechnicalreport, Maaz2023VideoChatGPTTD}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2308.12966" id="12646">
        <attvalues>
          <attvalue for="5" value=" Although~\cite{aghajanyan2023scaling} investigated scaling laws in the context of multimodal scenarios, they neglected the connected vision paradigm~\cite{zhang2024mmllms,bai2023qwenvl}, leading to the following challenges: C1." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2112.06905" id="12647">
        <attvalues>
          <attvalue for="5" value=" These models, equipped with an enormous number of parameters, have demonstrated outstanding performance in areas such as translation ~\cite{fan2020englishcentric}, question answering ~\cite{devlin2019bert, Raffel2019ExploringTL}, and text generation ~\cite{brown2020languagemodelsfewshotlearners,du2022glamefficientscalinglanguage}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2010.11125" id="12648">
        <attvalues>
          <attvalue for="5" value=" These models, equipped with an enormous number of parameters, have demonstrated outstanding performance in areas such as translation ~\cite{fan2020englishcentric}, question answering ~\cite{devlin2019bert, Raffel2019ExploringTL}, and text generation ~\cite{brown2020languagemodelsfewshotlearners,du2022glamefficientscalinglanguage}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2005.14165" id="12649">
        <attvalues>
          <attvalue for="5" value=" These models, equipped with an enormous number of parameters, have demonstrated outstanding performance in areas such as translation ~\cite{fan2020englishcentric}, question answering ~\cite{devlin2019bert, Raffel2019ExploringTL}, and text generation ~\cite{brown2020languagemodelsfewshotlearners,du2022glamefficientscalinglanguage}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2407.10759" id="12650">
        <attvalues>
          <attvalue for="5" value=" Due to the powerful capabilities of LLMs, a natural idea is to combine them with multimodal research, leveraging the strong cognitive abilities of LLMs to process information from other modalities ~\cite{openai2024gpt4,chu2024qwen2audiotechnicalreport, Maaz2023VideoChatGPTTD}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2312.14238" id="12651">
        <attvalues>
          <attvalue for="5" value="5} and image-text retrieval~\cite{chen2024internvl}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2301.12597" id="12652">
        <attvalues>
          <attvalue for="5" value=" Mechanisms such as cross-attention~\cite{li2023blip2} or linear projectors~\cite{llava} are utilized to connect the vision encoder (eg, ViT~\cite{dosovitskiy2021imageworth16x16words}) with the LLM backbone~\cite{zhang2024mmllms}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2010.11929" id="12654">
        <attvalues>
          <attvalue for="5" value=" Mechanisms such as cross-attention~\cite{li2023blip2} or linear projectors~\cite{llava} are utilized to connect the vision encoder (eg, ViT~\cite{dosovitskiy2021imageworth16x16words}) with the LLM backbone~\cite{zhang2024mmllms}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2401.13601" id="12655">
        <attvalues>
          <attvalue for="5" value=" Although~\cite{aghajanyan2023scaling} investigated scaling laws in the context of multimodal scenarios, they neglected the connected vision paradigm~\cite{zhang2024mmllms,bai2023qwenvl}, leading to the following challenges: C1." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2001.08361" id="12656">
        <attvalues>
          <attvalue for="5" value=" Apart from VLMs, previous research on scaling laws~\cite{kaplan2020scaling,hoffmann2022training} adopts an empirical approach to study the relationship between model performance, parameter size, and the amount of training data." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2203.15556" id="12657">
        <attvalues>
          <attvalue for="5" value=" Apart from VLMs, previous research on scaling laws~\cite{kaplan2020scaling,hoffmann2022training} adopts an empirical approach to study the relationship between model performance, parameter size, and the amount of training data." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2402.02054" id="12658">
        <attvalues>
          <attvalue for="5" value=" Additionally, scaling laws have been extended to various scenarios, including graph neural networks~\cite{liu2024neural}, data mixing~\cite{ye2024data}, data pruning~\cite{Sorscher2022BeyondNS}, and fine-tuning LLMs~\cite{zhang2024scaling}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2403.16952" id="12659">
        <attvalues>
          <attvalue for="5" value=" Additionally, scaling laws have been extended to various scenarios, including graph neural networks~\cite{liu2024neural}, data mixing~\cite{ye2024data}, data pruning~\cite{Sorscher2022BeyondNS}, and fine-tuning LLMs~\cite{zhang2024scaling}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2206.14486" id="12660">
        <attvalues>
          <attvalue for="5" value=" Additionally, scaling laws have been extended to various scenarios, including graph neural networks~\cite{liu2024neural}, data mixing~\cite{ye2024data}, data pruning~\cite{Sorscher2022BeyondNS}, and fine-tuning LLMs~\cite{zhang2024scaling}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2402.17193" id="12661">
        <attvalues>
          <attvalue for="5" value=" Additionally, scaling laws have been extended to various scenarios, including graph neural networks~\cite{liu2024neural}, data mixing~\cite{ye2024data}, data pruning~\cite{Sorscher2022BeyondNS}, and fine-tuning LLMs~\cite{zhang2024scaling}." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2301.03728" id="12662">
        <attvalues>
          <attvalue for="5" value=" Although~\cite{aghajanyan2023scaling} investigated scaling laws in the context of multimodal scenarios, they neglected the connected vision paradigm~\cite{zhang2024mmllms,bai2023qwenvl}, leading to the following challenges: C1." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2102.08981" id="12663">
        <attvalues>
          <attvalue for="5" value=" For the dataset, we choose image-text pairs from CC12M~\cite{changpinyo2021conceptual} and Laion400M~\cite{Schuhmann2021LAION400MOD}, known for their high-quality pairs." />
        </attvalues>
      </edge>
      <edge source="2408.00620" target="2111.02114" id="12664">
        <attvalues>
          <attvalue for="5" value=" For the dataset, we choose image-text pairs from CC12M~\cite{changpinyo2021conceptual} and Laion400M~\cite{Schuhmann2021LAION400MOD}, known for their high-quality pairs." />
        </attvalues>
      </edge>
      <edge source="1911.02656" target="1310.4546" id="12665">
        <attvalues>
          <attvalue for="5" value=" The setup subsumes some popular embedding techniques such as Latent Semantic Analysis (LSA) \cite{deerwester-lsa}, word2vec \cite{mikolov-distributed, mikolov-efficient}, and GloVe \cite{pennington-glove}, wherein the matrices $U$ and $V$ appear in a suitably chosen $f$ only through their product $UV$." />
        </attvalues>
      </edge>
      <edge source="1908.11020" target="1409.3215" id="12666">
        <attvalues>
          <attvalue for="5" value=" On the contrary, neural machine translation (NMT) advocates a unified manner to jointly learn source and target context using an encoder-decoder framework with an attention mechanism, leading to substantial gains over SMT in translation quality~\cite{sutskever2014sequence, bahdanau2014neural, gehring2017convolutional, vaswani2017attention}." />
        </attvalues>
      </edge>
      <edge source="1908.11020" target="1608.06043" id="12668">
        <attvalues>
          <attvalue for="5" value=" As a result, \cite{tu2017context} carefully designed context gates to dynamically control the influence from source and target contexts and observed significant improvements in the recurrent neural network (RNN) based NMT." />
        </attvalues>
      </edge>
      <edge source="1908.11020" target="1706.03762" id="12669">
        <attvalues>
          <attvalue for="5" value=" Although Transformer~\cite{vaswani2017attention} delivers significant gains over RNN for translation, there are still one third translation errors related to context control problem as described in Section~\ref{subsec:error}." />
        </attvalues>
      </edge>
      <edge source="2206.14982" target="1601.00710" id="12670">
        <attvalues>
          <attvalue for="5" value=" Multilingual Neural Machine Translation (MNMT), which enables one system to serve translation for multiple directions, has attracted much attention in the machine translation area~\cite{zoph-knight-2016-multi,firat-etal-2016-zero}." />
        </attvalues>
      </edge>
      <edge source="2206.14982" target="1606.04164" id="12671">
        <attvalues>
          <attvalue for="5" value=" Multilingual Neural Machine Translation (MNMT), which enables one system to serve translation for multiple directions, has attracted much attention in the machine translation area~\cite{zoph-knight-2016-multi,firat-etal-2016-zero}." />
        </attvalues>
      </edge>
      <edge source="2206.14982" target="1611.04558" id="12672">
        <attvalues>
          <attvalue for="5" value=" In our preliminary experiments, we observed that the complete many-to-many training is still as challenging as one-to-many training~\cite{johnson-etal-2017-googles,wang-etal-2020-negative}, since we have introduced more one-to-many translation tasks into the training." />
        </attvalues>
      </edge>
      <edge source="2206.14982" target="1803.05567" id="12673">
        <attvalues>
          <attvalue for="5" value=" Because the multilingual capability hugely reduces the deployment cost at training and inference, MNMT has actively been employed as a machine translation system backbone in recent years~\cite{johnson-etal-2017-googles,DBLP:journals/corr/abs-1803-05567}." />
        </attvalues>
      </edge>
      <edge source="2206.14982" target="1906.01181" id="12674">
        <attvalues>
          <attvalue for="5" value=" Similarly reported in the many-to-many training with zero-shot setup \cite{gu-etal-2019-improved,yang-etal-2021-improving-multilingual}, the complete MNMT model also suffers from capturing correlations in the data for all the X-Y directions as one model training, due to highly imbalanced data." />
        </attvalues>
      </edge>
      <edge source="2206.14982" target="2004.11867" id="12675">
        <attvalues>
          <attvalue for="5" value=" Recent work~\cite{gu-etal-2019-improved,zhang-etal-2020-improving,yang-etal-2021-improving-multilingual} pointed out that such MNMT systems severely face an off-target translation issue, especially in translations from a non-English language X to another non-English language Y." />
        </attvalues>
      </edge>
      <edge source="2206.14982" target="2109.04778" id="12676">
        <attvalues>
          <attvalue for="5" value=" Similarly reported in the many-to-many training with zero-shot setup \cite{gu-etal-2019-improved,yang-etal-2021-improving-multilingual}, the complete MNMT model also suffers from capturing correlations in the data for all the X-Y directions as one model training, due to highly imbalanced data." />
        </attvalues>
      </edge>
      <edge source="2206.14982" target="2010.10239" id="12677">
        <attvalues>
          <attvalue for="5" value=" Meanwhile, \cite{freitag-firat-2020-complete} have extended data resources with multi-way aligned data and reported that one complete many-to-many MNMT can be fully supervised, achieving competitive translation performance for all X-Y directions." />
        </attvalues>
      </edge>
      <edge source="2204.06514" target="2105.04663" id="12680">
        <attvalues>
          <attvalue for="5" value=" We focus on the breakthroughs in training efficiency achieved by using JAX~\cite{jax} to enable tensor and data parallelism through GSPMD~\cite{gspmd}, XLA, and Google Cloud TPU VMs~\cite{tpu-vm}, highlighting the use of recently released TPU~v4 Pods~\cite{tpu-v4}." />
        </attvalues>
      </edge>
      <edge source="2102.09786" target="1906.09821" id="12681">
        <attvalues>
          <attvalue for="5" value=" To this end, methods for grouping relevant arguments within a given topic by their similarities~\cite{misra-etal-2016-measuring,reimers-etal-2019-classification,chen-etal-2019-seeing} should be developed to prohibit redundant outcomes (argument clustering)." />
        </attvalues>
      </edge>
      <edge source="2102.09786" target="1910.14192" id="12682">
        <attvalues>
          <attvalue for="5" value=" To address this, various domain adaptation methods~\cite{li2019transferable,das2019learning,wang2019adversarial,cao2019unsupervised} have been recently explored." />
        </attvalues>
      </edge>
      <edge source="2102.09786" target="1908.09209" id="12684">
        <attvalues>
          <attvalue for="5" value=" To address this, various domain adaptation methods~\cite{li2019transferable,das2019learning,wang2019adversarial,cao2019unsupervised} have been recently explored." />
        </attvalues>
      </edge>
      <edge source="2102.09786" target="1911.06137" id="12685">
        <attvalues>
          <attvalue for="5" value=" To address this, various domain adaptation methods~\cite{li2019transferable,das2019learning,wang2019adversarial,cao2019unsupervised} have been recently explored." />
        </attvalues>
      </edge>
      <edge source="2102.09786" target="1907.11692" id="12687">
        <attvalues>
          <attvalue for="5" value=" In particular, several studies found that continual pretraining of a language model (eg, BERT~\cite{devlin-etal-2019-bert} and RoBERTa~\cite{liu2019roberta}) is effective with both unsupervised domain adaptation~\cite{ma-etal-2019-domain,rietzler2019adapt} and general supervised learning~\cite{howard2018universal,gururangan2020don}." />
        </attvalues>
      </edge>
      <edge source="2102.09786" target="1908.11860" id="12688">
        <attvalues>
          <attvalue for="5" value=" In particular, several studies found that continual pretraining of a language model (eg, BERT~\cite{devlin-etal-2019-bert} and RoBERTa~\cite{liu2019roberta}) is effective with both unsupervised domain adaptation~\cite{ma-etal-2019-domain,rietzler2019adapt} and general supervised learning~\cite{howard2018universal,gururangan2020don}." />
        </attvalues>
      </edge>
      <edge source="2102.09786" target="1801.06146" id="12689">
        <attvalues>
          <attvalue for="5" value=" In particular, several studies found that continual pretraining of a language model (eg, BERT~\cite{devlin-etal-2019-bert} and RoBERTa~\cite{liu2019roberta}) is effective with both unsupervised domain adaptation~\cite{ma-etal-2019-domain,rietzler2019adapt} and general supervised learning~\cite{howard2018universal,gururangan2020don}." />
        </attvalues>
      </edge>
      <edge source="2306.02405" target="2205.10643" id="12691">
        <attvalues>
          <attvalue for="5" value=" Self-supervised learning (SSL) for the speech modality is an active area of research that aims to develop models that build meaningful speech representations from raw audio without any explicit labels or transcriptions (see \cite{9893562} for an overview)." />
        </attvalues>
      </edge>
      <edge source="2306.02405" target="1807.03748" id="12692">
        <attvalues>
          <attvalue for="5" value=" These models can be further adapted for downstream tasks such as automatic speech recognition and speaker identification, and have become the state-of-the-art approach even when limited labeled data are available \cite{van2018representation, Schneider2019wav2vecUP, baevski2020wav2vec, hsu2021hubert}." />
        </attvalues>
      </edge>
      <edge source="2306.02405" target="2105.05582" id="12694">
        <attvalues>
          <attvalue for="5" value=" A few recent studies have investigated the discrete units from a neural network interpretability point of view \cite{higy2021discrete, nguyen2022discrete, wells2022phonetic, sicherman2023analysing}." />
        </attvalues>
      </edge>
      <edge source="2306.02405" target="2301.00591" id="12695">
        <attvalues>
          <attvalue for="5" value=" In \cite{sicherman2023analysing}, the authors concluded that there exists a strong correspondence between discrete units and phonemes, and attributed the lack of consistent phoneme-to-unit mapping to variations in phonological contexts." />
        </attvalues>
      </edge>
      <edge source="2311.04930" target="1307.0225" id="12696">
        <attvalues>
          <attvalue for="5" value=" In contrast to the information-theory-grounded approaches, which focus on predicting upcoming inputs \cite{Palmer2015-py}, this approach focuses on the internal representation states and on predicting future internal states." />
        </attvalues>
      </edge>
      <edge source="2311.04930" target="1807.03748" id="12697">
        <attvalues>
          <attvalue for="5" value=" For example, representations that are predictive of incoming input have been argued to be useful in both biological systems - across perception, action, and cognition \cite{Rao1999-bl,Palmer2015-py,Shadmehr2010-pk,hohwy2008predictive,jessup2010error,Shain2020-je,Frank2015-rf} - and in artificial systems across domains \cite{Van_den_Oord2018-kf,Radford_dl}." />
        </attvalues>
      </edge>
      <edge source="2311.04930" target="0712.4381" id="12698">
        <attvalues>
          <attvalue for="5" value=" The first approach leverages information theory \cite{Shannon1949-xp} to quantify the relationship between the past and current neural states and future inputs (eg, \cite{Bialek2007-qc,Tishby2000-id,Wiskott2002-up,Palmer2015-py})." />
        </attvalues>
      </edge>
      <edge source="2311.04930" target="physics/0004057" id="12699">
        <attvalues>
          <attvalue for="5" value=" The first approach leverages information theory \cite{Shannon1949-xp} to quantify the relationship between the past and current neural states and future inputs (eg, \cite{Bialek2007-qc,Tishby2000-id,Wiskott2002-up,Palmer2015-py})." />
        </attvalues>
      </edge>
      <edge source="2008.02897" target="1907.03540" id="12700">
        <attvalues>
          <attvalue for="5" value="23\times$ relative speedup gain over a manual search procedure~\cite{shrinkml2019}." />
        </attvalues>
      </edge>
      <edge source="2008.02897" target="1708.06519" id="12701">
        <attvalues>
          <attvalue for="5" value=" Alternatively, iterative approaches have been successfully used to manually compress image recognition models, often delivering better results than their one-shot counterparts~\cite{liu_learning_2017,molchanov_pruning_2017,frankle_lottery_2018,gao_dynamic_2019}." />
        </attvalues>
      </edge>
      <edge source="2008.02897" target="1611.06440" id="12702">
        <attvalues>
          <attvalue for="5" value=" Alternatively, iterative approaches have been successfully used to manually compress image recognition models, often delivering better results than their one-shot counterparts~\cite{liu_learning_2017,molchanov_pruning_2017,frankle_lottery_2018,gao_dynamic_2019}." />
        </attvalues>
      </edge>
      <edge source="2008.02897" target="1810.05331" id="12703">
        <attvalues>
          <attvalue for="5" value=" Alternatively, iterative approaches have been successfully used to manually compress image recognition models, often delivering better results than their one-shot counterparts~\cite{liu_learning_2017,molchanov_pruning_2017,frankle_lottery_2018,gao_dynamic_2019}." />
        </attvalues>
      </edge>
      <edge source="2208.08386" target="1908.10084" id="12714">
        <attvalues>
          <attvalue for="5" value=" Recent work has focused on fine-tuning pretrained language models with contrastive learning, either supervised (eg \cite{reimers-gurevych-2019-sentence, zhang-etal-2021-pairwise, yan-etal-2021-consert}) or unsupervised (eg \cite{giorgi-etal-2021-declutr, gao-etal-2021-simcse})." />
        </attvalues>
      </edge>
      <edge source="2303.03750" target="2210.08523" id="12718">
        <attvalues>
          <attvalue for="5" value=" For many South African languages there are still challenges finding easily available textual datasets \cite{marivate2020investigating} even if there are many speakers for those languages \cite{ranathunga-de-silva-2022-languages}." />
        </attvalues>
      </edge>
      <edge source="2303.03750" target="2004.09095" id="12719">
        <attvalues>
          <attvalue for="5" value=" There is a need to focus on development of local language \cite{joshi-etal-2020-state} NLP resources." />
        </attvalues>
      </edge>
      <edge source="2303.03750" target="1704.04154" id="12720">
        <attvalues>
          <attvalue for="5" value=" The parallel corpora were generated using LASER encoders \cite{schwenk2017learning}, facilitating the one-to-one alignment of tokenised sentence data." />
        </attvalues>
      </edge>
      <edge source="2303.03750" target="2010.11125" id="12721">
        <attvalues>
          <attvalue for="5" value=" We also provide NMT benchmarks for the parallel corpora by fine-tuning a massively multilingual model (M2M100 \cite{fan2021beyond}) building on the work of \cite{lafand}." />
        </attvalues>
      </edge>
      <edge source="2303.03750" target="2205.02022" id="12722">
        <attvalues>
          <attvalue for="5" value=" We also provide NMT benchmarks for the parallel corpora by fine-tuning a massively multilingual model (M2M100 \cite{fan2021beyond}) building on the work of \cite{lafand}." />
        </attvalues>
      </edge>
      <edge source="2004.12835" target="1605.07766" id="12725">
        <attvalues>
          <attvalue for="5" value=" \cite{nguyen2016integrating} improve the weights of feature vectors with a special method based on local mutual information and propose an extension of the skip-gram model that integrates the new vector representations into the objective function." />
        </attvalues>
      </edge>
      <edge source="2004.12835" target="1410.0718" id="12726">
        <attvalues>
          <attvalue for="5" value=" \cite{hill2014not} and \cite{hill2014embedding} show that translation-based embeddings perform better in applications that require concepts to be organized according to similarity and better capture their true ontologic status." />
        </attvalues>
      </edge>
      <edge source="2004.12835" target="1412.6448" id="12727">
        <attvalues>
          <attvalue for="5" value=" \cite{hill2014not} and \cite{hill2014embedding} show that translation-based embeddings perform better in applications that require concepts to be organized according to similarity and better capture their true ontologic status." />
        </attvalues>
      </edge>
      <edge source="2004.12835" target="1701.02962" id="12728">
        <attvalues>
          <attvalue for="5" value=" \cite{nguyen2017distinguishing} train a neural network model that exploits lexico-syntactic patterns from syntactic parse trees to distinguish antonyms." />
        </attvalues>
      </edge>
      <edge source="2004.12835" target="1710.06371" id="12729">
        <attvalues>
          <attvalue for="5" value=" \cite{vulic2018specialising} and \cite{vulic2018injecting} show one can inject information on hyponyms, hyperonyms, synonyms, and antonyms to distinguish the obtained embeddings using additional linguistic constraints, see \cite{mrkvsic2017semantic}." />
        </attvalues>
      </edge>
      <edge source="2004.12835" target="1906.05612" id="12730">
        <attvalues>
          <attvalue for="5" value=" Similarly to \cite{etcheverry2019unraveling} it is based on a siamese network, yet does not require a two-phase training and is more intuitive than the one proposed in \cite{ali2019antonym}." />
        </attvalues>
      </edge>
      <edge source="2204.13511" target="1503.02531" id="12732">
        <attvalues>
          <attvalue for="5" value=" Following the trend of distilling the knowledge from neural network models \cite{hinton2015distilling}, many types of distillation have been used to extract optimal parameters or extract the knowledge of larger language models into smaller ones \cite{sanh2019distilbert,dewynter2020bort,jiao2020tinybert}." />
        </attvalues>
      </edge>
      <edge source="2204.13511" target="1909.10351" id="12735">
        <attvalues>
          <attvalue for="5" value=" Following the trend of distilling the knowledge from neural network models \cite{hinton2015distilling}, many types of distillation have been used to extract optimal parameters or extract the knowledge of larger language models into smaller ones \cite{sanh2019distilbert,dewynter2020bort,jiao2020tinybert}." />
        </attvalues>
      </edge>
      <edge source="2204.13511" target="2001.06286" id="12736">
        <attvalues>
          <attvalue for="5" value=" In this paper, we distill the Dutch BERT model RobBERT v2 \cite{delobelle2020robbert}, and name it RobBERTje." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="2205.10643" id="12737">
        <attvalues>
          <attvalue for="5" value=" Self-supervised pre-trained models have significantly advanced the field of speech technology, offering robust solutions for learning general representations from vast quantities of unlabeled data\cite{sslreview,wavlm,wav2vec,hubert}." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="2110.13900" id="12738">
        <attvalues>
          <attvalue for="5" value=" Self-supervised pre-trained models have significantly advanced the field of speech technology, offering robust solutions for learning general representations from vast quantities of unlabeled data\cite{sslreview,wavlm,wav2vec,hubert}." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="2006.11477" id="12739">
        <attvalues>
          <attvalue for="5" value="0\cite{wav2vec} and HuBERT\cite{hubert} can be easily fine-tuned for a range of downstream tasks, such as automatic speech recognition (ASR), speech synthesis and speech tokenization." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="2106.07447" id="12740">
        <attvalues>
          <attvalue for="5" value="0\cite{wav2vec} and HuBERT\cite{hubert} can be easily fine-tuned for a range of downstream tasks, such as automatic speech recognition (ASR), speech synthesis and speech tokenization." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="2312.07338" id="12742">
        <attvalues>
          <attvalue for="5" value=" In \cite{ssladaptive}, unlabeled data collected from the target language of downstream tasks are leveraged to adapt the pre-trained model prior to the last fine-tuning stage." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="1612.00796" id="12743">
        <attvalues>
          <attvalue for="5" value=" When adapting an existed model to new data or new tasks, there is a phenomenon called catastrophic forgetting\cite{overcomingcf} which indicates that the model tends to forget previously acquired knowledge." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="1904.08039" id="12744">
        <attvalues>
          <attvalue for="5" value=" A multi-task learning framework is advocated in \cite{MTL} where reserving original knowledge and learning new knowledge are treated as two independent tasks." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="2308.03300" id="12745">
        <attvalues>
          <attvalue for="5" value=" For instance, \cite{weight} introduced a continual learning algorithm known as Regularized Adaptive Weight Modification (RAWM) for fake audio detection." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="1805.07441" id="12746">
        <attvalues>
          <attvalue for="5" value=" Alternative methodologies involve the orthogonal modification of weights\cite{weight,weightconsolidation,rotate}." />
        </attvalues>
      </edge>
      <edge source="2406.14092" target="2112.08352" id="12748">
        <attvalues>
          <attvalue for="5" value=" Specifically, the SSL model under study is mHuBERT\cite{mhubert}, while the existed language and new language to adapt are English and Mandarin respectively." />
        </attvalues>
      </edge>
      <edge source="2007.11314" target="1704.04565" id="12750">
        <attvalues>
          <attvalue for="5" value=" Early work on paraphrase detection relied on hand-crafted features, while state-of-the-art approaches for paraphrase identification are primarily neural networks \cite{gong_natural_2018,wang_bilateral_2017,tomar_neural_2017} and hybrid techniques \cite{pang_text_2016,wu_ecnu_2017,feng_beihang-msra_2017}." />
        </attvalues>
      </edge>
      <edge source="2007.11314" target="1702.03814" id="12752">
        <attvalues>
          <attvalue for="5" value=" Early work on paraphrase detection relied on hand-crafted features, while state-of-the-art approaches for paraphrase identification are primarily neural networks \cite{gong_natural_2018,wang_bilateral_2017,tomar_neural_2017} and hybrid techniques \cite{pang_text_2016,wu_ecnu_2017,feng_beihang-msra_2017}." />
        </attvalues>
      </edge>
      <edge source="2007.11314" target="1602.06359" id="12753">
        <attvalues>
          <attvalue for="5" value=" Early work on paraphrase detection relied on hand-crafted features, while state-of-the-art approaches for paraphrase identification are primarily neural networks \cite{gong_natural_2018,wang_bilateral_2017,tomar_neural_2017} and hybrid techniques \cite{pang_text_2016,wu_ecnu_2017,feng_beihang-msra_2017}." />
        </attvalues>
      </edge>
      <edge source="2007.11314" target="1805.03616" id="12755">
        <attvalues>
          <attvalue for="5" value=" Recent work successfully introduced topics in neural architectures for language generation: \cite{wang_reinforced_2018} used a topic-enhanced encoder for summarisation, \cite{chen_guided_2016} integrated topics in the decoder for machine translation and \cite{narayan_dont_2018} included topics in both encoder and decoder of their summarisation model." />
        </attvalues>
      </edge>
      <edge source="2007.11314" target="1607.01628" id="12756">
        <attvalues>
          <attvalue for="5" value=" Recent work successfully introduced topics in neural architectures for language generation: \cite{wang_reinforced_2018} used a topic-enhanced encoder for summarisation, \cite{chen_guided_2016} integrated topics in the decoder for machine translation and \cite{narayan_dont_2018} included topics in both encoder and decoder of their summarisation model." />
        </attvalues>
      </edge>
      <edge source="2007.11314" target="1808.08745" id="12757">
        <attvalues>
          <attvalue for="5" value=" Recent work successfully introduced topics in neural architectures for language generation: \cite{wang_reinforced_2018} used a topic-enhanced encoder for summarisation, \cite{chen_guided_2016} integrated topics in the decoder for machine translation and \cite{narayan_dont_2018} included topics in both encoder and decoder of their summarisation model." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="1303.5778" id="12758">
        <attvalues>
          <attvalue for="5" value=" Recurrent neural networks (RNNs) have been widely used given that they can effectively capture the temporal dependencies of a speech frame sequence \cite{graves2013speech,chiu2018state}." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="1712.01769" id="12759">
        <attvalues>
          <attvalue for="5" value=" Recurrent neural networks (RNNs) have been widely used given that they can effectively capture the temporal dependencies of a speech frame sequence \cite{graves2013speech,chiu2018state}." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="2005.03191" id="12760">
        <attvalues>
          <attvalue for="5" value=" By carefully tuning the receptive fields, residual connections and temporal resolution of the convolutional layers, deep CNNs can yield very competitive results \cite{han2020contextnet,han2021multistream}." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="2005.10470" id="12761">
        <attvalues>
          <attvalue for="5" value=" By carefully tuning the receptive fields, residual connections and temporal resolution of the convolutional layers, deep CNNs can yield very competitive results \cite{han2020contextnet,han2021multistream}." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="1706.03762" id="12762">
        <attvalues>
          <attvalue for="5" value=" Recently, the Transformer architecture with self-attention \cite{vaswani2017attention} is trending in ASR tasks due to its dominant performance in modeling the long-range dependencies \cite{wang2020transformer,moritz2020streaming}." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="1910.09799" id="12763">
        <attvalues>
          <attvalue for="5" value=" Recently, the Transformer architecture with self-attention \cite{vaswani2017attention} is trending in ASR tasks due to its dominant performance in modeling the long-range dependencies \cite{wang2020transformer,moritz2020streaming}." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="2001.02674" id="12764">
        <attvalues>
          <attvalue for="5" value=" Recently, the Transformer architecture with self-attention \cite{vaswani2017attention} is trending in ASR tasks due to its dominant performance in modeling the long-range dependencies \cite{wang2020transformer,moritz2020streaming}." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="1911.11423" id="12765">
        <attvalues>
          <attvalue for="5" value=" Despite the great success of Transformer, several research have found that the powerful attention mechanism can still be complemented by traditional neural components that are well-suited for capturing fine-grain local context \cite{merity2019single,lei2021attention}." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="2005.08100" id="12767">
        <attvalues>
          <attvalue for="5" value=" For example, Conformer \cite{gulati2020conformer} proposes adding a convolution module leveraging gating mechanisms in point-wise convolution and gated linear unit (GLU) on top of multi-head self-attention to augment Transformer, yielding faster convergence and state-of-the-art performance on various speech recognition datasets." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="1709.02755" id="12768">
        <attvalues>
          <attvalue for="5" value=" The basic idea of SRU++ is to leverage a highly parallelizable recurrent neural network structure called SRU \cite{lei2017simple} and perform attention before the recurrence." />
        </attvalues>
      </edge>
      <edge source="2110.05571" target="2010.13956" id="12769">
        <attvalues>
          <attvalue for="5" value=" We conduct ASR experiments on three popular public datasets and compare the results with the Transformer and Conformer encoders implemented and open-sourced in \cite{guo2021recent}." />
        </attvalues>
      </edge>
      <edge source="2409.02302" target="2309.07525" id="12770">
        <attvalues>
          <attvalue for="5" value=" Recently, the speech anti-spoofing research community has been increasingly focusing on this challenging issue, resulting in the development of related datasets~\cite{SingFake, CtrSVDD_interspeech, 10446271}, challenges~\cite{SVDD_challenge}, and models~\cite{chen2024singing}." />
        </attvalues>
      </edge>
      <edge source="2409.02302" target="2406.03111" id="12772">
        <attvalues>
          <attvalue for="5" value=" Recently, the speech anti-spoofing research community has been increasingly focusing on this challenging issue, resulting in the development of related datasets~\cite{SingFake, CtrSVDD_interspeech, 10446271}, challenges~\cite{SVDD_challenge}, and models~\cite{chen2024singing}." />
        </attvalues>
      </edge>
      <edge source="2409.02302" target="2407.02826" id="12773">
        <attvalues>
          <attvalue for="5" value=" Speech foundation models are large, pre-trained models designed to serve as the backbone for various speech-related tasks, including speaker verification, speech recognition, and more~\cite{lin2024sawavlms, 10446072, jiang2024target}." />
        </attvalues>
      </edge>
      <edge source="2409.02302" target="2110.13900" id="12774">
        <attvalues>
          <attvalue for="5" value=" Many of these models rely on self-supervised learning (SSL) to develop robust speech representations, such as WavLM~\cite{wavlm} and wav2vec2~\cite{wav2vec2}." />
        </attvalues>
      </edge>
      <edge source="2409.02302" target="2006.11477" id="12775">
        <attvalues>
          <attvalue for="5" value=" Many of these models rely on self-supervised learning (SSL) to develop robust speech representations, such as WavLM~\cite{wavlm} and wav2vec2~\cite{wav2vec2}." />
        </attvalues>
      </edge>
      <edge source="2409.02302" target="2204.05177" id="12776">
        <attvalues>
          <attvalue for="5" value=" Recently, many studies on speech anti-spoofing have adopted this approach and achieved state-of-the-art performance~\cite{10003971, liu2024neural, 10446331, 9747768, 10448049, 10448016}." />
        </attvalues>
      </edge>
      <edge source="2409.02302" target="1709.01507" id="12778">
        <attvalues>
          <attvalue for="5" value=" To address these issues,} inspired by Squeeze-and-Excitation Networks (SENet)~\cite{SE}, we propose the SE Aggregation (SEA) method." />
        </attvalues>
      </edge>
      <edge source="2006.14223" target="1711.00549" id="12779">
        <attvalues>
          <attvalue for="5" value="(\cite{Kumar2017JustAB}, \cite{Kumar2017JustAB}) In ASK work flow, the skill developer provides a set of slots (often catalogs of entities), and a list of intents, which can be mapped to actions, and a set of example phrases defining the grammar of an intent." />
        </attvalues>
      </edge>
      <edge source="2006.14223" target="1409.3215" id="12780">
        <attvalues>
          <attvalue for="5" value=" It has been shown to have comparable performance to the phrase-based translation systems (\cite{sutskever2014sequence}, \cite{sutskever2014sequence}), and it is very flexible and modular, allowing to reuse pre-trained components, such as word embeddings or other networks trained on different datasets." />
        </attvalues>
      </edge>
      <edge source="2308.12488" target="2303.08774" id="12781">
        <attvalues>
          <attvalue for="5" value=" While many researchers have concentrated on evaluating ChatGPT and GPT-4~\cite{openai2023gpt4} within their specific domains of expertise, a comprehensive review encompassing the assessments in multiple tasks and disciplines can offer a holistic understanding of the strengths and limitations of these GPT models." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="1706.03741" id="12782">
        <attvalues>
          <attvalue for="5" value=" Most previous work on RLHF focuses on in-distribution (ID) preference learning (PL) \cite{casper2023open}, eg, using the ID preference data for reward learning \cite{christiano2017deep} and then performing policy optimization using PPO (Fig." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="1909.08593" id="12783">
        <attvalues>
          <attvalue for="5" value=" As shown by recent research on LLMs \cite{christiano2017deep,ziegler2019fine,stiennon2020learning,ouyang2022training}, RLHF initially trains a reward model (RM) to capture human preferences from a pairwise preference dataset." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="2305.18290" id="12786">
        <attvalues>
          <attvalue for="5" value=" (ii) the distribution shifts result in policy drift during the policy optimization process with Kullback-Leibler divergence \cite{ramamurthy2022reinforcement,rafailov2024direct}: the policy tends to move towards the reference policy largely followed in the training distribution, exacerbating the model's deviation from the test distribution in PL." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="2310.12036" id="12787">
        <attvalues>
          <attvalue for="5" value=" Besides, direct preference optimization (DPO) \cite{rafailov2024direct} and its variants \cite{azar2024general,ethayarajh2024kto,park2024disentangling} directly optimize a language policy using maximum likelihood estimation on the ID preference data (Fig." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="2402.01306" id="12788">
        <attvalues>
          <attvalue for="5" value=" Besides, direct preference optimization (DPO) \cite{rafailov2024direct} and its variants \cite{azar2024general,ethayarajh2024kto,park2024disentangling} directly optimize a language policy using maximum likelihood estimation on the ID preference data (Fig." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="2403.19159" id="12789">
        <attvalues>
          <attvalue for="5" value=" Besides, direct preference optimization (DPO) \cite{rafailov2024direct} and its variants \cite{azar2024general,ethayarajh2024kto,park2024disentangling} directly optimize a language policy using maximum likelihood estimation on the ID preference data (Fig." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="2301.03652" id="12790">
        <attvalues>
          <attvalue for="5" value=" However, defining precise rewards for various real-world tasks is non-trivial \cite{mckinney2022fragility}, and obtaining high-quality feedback that accurately represents human preferences is challenging \cite{bai2022training}." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="2204.05862" id="12791">
        <attvalues>
          <attvalue for="5" value=" However, defining precise rewards for various real-world tasks is non-trivial \cite{mckinney2022fragility}, and obtaining high-quality feedback that accurately represents human preferences is challenging \cite{bai2022training}." />
        </attvalues>
      </edge>
      <edge source="2402.14760" target="2210.01241" id="12792">
        <attvalues>
          <attvalue for="5" value=" (ii) the distribution shifts result in policy drift during the policy optimization process with Kullback-Leibler divergence \cite{ramamurthy2022reinforcement,rafailov2024direct}: the policy tends to move towards the reference policy largely followed in the training distribution, exacerbating the model's deviation from the test distribution in PL." />
        </attvalues>
      </edge>
      <edge source="1910.04887" target="1603.06180" id="12805">
        <attvalues>
          <attvalue for="5" value=" Existing solutions take into account both the image and the query \cite{hu2016segmentation,Hu_2016_CVPR,shi2018key}." />
        </attvalues>
      </edge>
      <edge source="1910.04887" target="1511.04164" id="12806">
        <attvalues>
          <attvalue for="5" value=" Existing solutions take into account both the image and the query \cite{hu2016segmentation,Hu_2016_CVPR,shi2018key}." />
        </attvalues>
      </edge>
      <edge source="1910.04887" target="1804.09661" id="12807">
        <attvalues>
          <attvalue for="5" value=" Sequence models have been conditioned to complete text from a prefix and index \cite{jaech2018personalized}, however have not been extended to take into account an image." />
        </attvalues>
      </edge>
      <edge source="1910.04887" target="1703.06870" id="12808">
        <attvalues>
          <attvalue for="5" value=" Deep neural networks have been trained to segment all instances in an image at very high quality \cite{maskrcnn2017,Hu_2018}." />
        </attvalues>
      </edge>
      <edge source="1910.04887" target="1711.10370" id="12810">
        <attvalues>
          <attvalue for="5" value=" Deep neural networks have been trained to segment all instances in an image at very high quality \cite{maskrcnn2017,Hu_2018}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="1807.03748" id="12811">
        <attvalues>
          <attvalue for="5" value=" Frame representation produced by self-supervised models have shown to work well at distinguishing phone identities at the frame level~\cite{oord2018cpc,chung2019an,yang2022auto,yang21c_interspeech} and at the segment level \cite{chorowski21b_interspeech, bhati2022unsupervised, cuervo2022variable}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="1904.03240" id="12812">
        <attvalues>
          <attvalue for="5" value=" Frame representation produced by self-supervised models have shown to work well at distinguishing phone identities at the frame level~\cite{oord2018cpc,chung2019an,yang2022auto,yang21c_interspeech} and at the segment level \cite{chorowski21b_interspeech, bhati2022unsupervised, cuervo2022variable}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="2104.11946" id="12814">
        <attvalues>
          <attvalue for="5" value=" Frame representation produced by self-supervised models have shown to work well at distinguishing phone identities at the frame level~\cite{oord2018cpc,chung2019an,yang2022auto,yang21c_interspeech} and at the segment level \cite{chorowski21b_interspeech, bhati2022unsupervised, cuervo2022variable}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="2106.07447" id="12815">
        <attvalues>
          <attvalue for="5" value=" We introduce a loss function predicting pseudo-labels as self-supervised learning, similar to~\cite{hsu2021hubert,chiu2022self}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="2301.00591" id="12816">
        <attvalues>
          <attvalue for="5" value=" In this paper, we will focus on phone types, because phonetic properties are salient even after crude quantization down to 100 or even 50 codes \cite{hsu2021hubert, wells2022phonetic,Amitay2023}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="1712.04313" id="12817">
        <attvalues>
          <attvalue for="5" value=" It is also highly related to unsupervised phone segmentation \cite{kreuk20_interspeech,zero2015,zero2017} and lexical discovery~\cite{kamper2017segmental}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="1804.00316" id="12818">
        <attvalues>
          <attvalue for="5" value=" Much of the recent progress focuses on adversarial approaches to distinguish sequences of cluster IDs and phone sequences observed in data sets \cite{liu18g_interspeech,chen19e_interspeech,liu2022,baevski2021unsupervised,liu2023towards}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="2207.14568" id="12819">
        <attvalues>
          <attvalue for="5" value=" Much of the recent progress focuses on adversarial approaches to distinguish sequences of cluster IDs and phone sequences observed in data sets \cite{liu18g_interspeech,chen19e_interspeech,liu2022,baevski2021unsupervised,liu2023towards}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="2105.11084" id="12820">
        <attvalues>
          <attvalue for="5" value=" Much of the recent progress focuses on adversarial approaches to distinguish sequences of cluster IDs and phone sequences observed in data sets \cite{liu18g_interspeech,chen19e_interspeech,liu2022,baevski2021unsupervised,liu2023towards}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="2204.02492" id="12821">
        <attvalues>
          <attvalue for="5" value=" Much of the recent progress focuses on adversarial approaches to distinguish sequences of cluster IDs and phone sequences observed in data sets \cite{liu18g_interspeech,chen19e_interspeech,liu2022,baevski2021unsupervised,liu2023towards}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="1812.09323" id="12822">
        <attvalues>
          <attvalue for="5" value=" The problem of learning phone types has a long history and shows up under different names, such as acoustic unit discovery \cite{park2008unsup,zero2015,zero2017} and unsupervised speech recognition \cite{liu18g_interspeech,chen19e_interspeech,liu2022,baevski2021unsupervised,liu2023towards,yeh2018unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="2007.13465" id="12823">
        <attvalues>
          <attvalue for="5" value=" It is also highly related to unsupervised phone segmentation \cite{kreuk20_interspeech,zero2015,zero2017} and lexical discovery~\cite{kamper2017segmental}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="1606.06950" id="12824">
        <attvalues>
          <attvalue for="5" value=" It is also highly related to unsupervised phone segmentation \cite{kreuk20_interspeech,zero2015,zero2017} and lexical discovery~\cite{kamper2017segmental}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="2305.12464" id="12825">
        <attvalues>
          <attvalue for="5" value=" Recent work on analyzing representations of self-supervised predictive coding finds that phonetic and speaker information are largely represented in orthogonal subspaces~\cite{liu2023self}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="1710.04087" id="12826">
        <attvalues>
          <attvalue for="5" value=" This is reminiscent to the line of work in unsupervised machine translation~\cite{lample2018word,2018gromov,alaux2018unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2310.17558" target="1809.00013" id="12827">
        <attvalues>
          <attvalue for="5" value=" This is reminiscent to the line of work in unsupervised machine translation~\cite{lample2018word,2018gromov,alaux2018unsupervised}." />
        </attvalues>
      </edge>
      <edge source="2009.05485" target="1509.08062" id="12830">
        <attvalues>
          <attvalue for="5" value=" In \cite{heigold2016end}, a recurrent neural network (RNN) is used to capture temporal dependency and derive utterance-level embedding for SV." />
        </attvalues>
      </edge>
      <edge source="2009.05485" target="1506.07503" id="12831">
        <attvalues>
          <attvalue for="5" value=" Attention mechanism in DNN has been shown effective in various application areas \cite{chorowski2015attention,zeyer2018improved,vaswani2017attention,wang2018non}." />
        </attvalues>
      </edge>
      <edge source="2009.05485" target="1805.03294" id="12832">
        <attvalues>
          <attvalue for="5" value=" Attention mechanism in DNN has been shown effective in various application areas \cite{chorowski2015attention,zeyer2018improved,vaswani2017attention,wang2018non}." />
        </attvalues>
      </edge>
      <edge source="2009.05485" target="1706.03762" id="12833">
        <attvalues>
          <attvalue for="5" value=" Attention mechanism in DNN has been shown effective in various application areas \cite{chorowski2015attention,zeyer2018improved,vaswani2017attention,wang2018non}." />
        </attvalues>
      </edge>
      <edge source="2009.05485" target="1711.07971" id="12834">
        <attvalues>
          <attvalue for="5" value=" Attention mechanism in DNN has been shown effective in various application areas \cite{chorowski2015attention,zeyer2018improved,vaswani2017attention,wang2018non}." />
        </attvalues>
      </edge>
      <edge source="2009.05485" target="1803.10963" id="12835">
        <attvalues>
          <attvalue for="5" value=" In the case of SV, attention mechanism can be implemented in the process of aggregating frame-level features with learned attention weights \cite{bhattacharya2017deep,okabe2018attentive}." />
        </attvalues>
      </edge>
      <edge source="2009.05485" target="1807.05688" id="12836">
        <attvalues>
          <attvalue for="5" value=" This idea of collaborative attention was applied in video-based person re-identification\cite{zhang2019scan} and text-dependent SV\cite{zhang2019seq2seq}." />
        </attvalues>
      </edge>
    </edges>
  </graph>
</gexf>
